From 8b3f6a6693a0c789f7b8eaf29279872b539b433f Mon Sep 17 00:00:00 2001 From: Christian Convey Date: Mon, 28 Jan 2019 17:46:25 -0500 Subject: [PATCH] Merge in upstream `master` through 4fe546 (#543) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Support full convention in quantized pooling (#13260) * fix quantized pooling and enable it in INT8 SqueezeNet * add test * fix test * address review comments * refine the test for quantized pooling * Add utility slave (#13383) * A few operators on graphs stored as CSR (#13290) * edge_id op csr forward on CPU (#34) * add node subgraph generator. (#35) * create DGLSubgraph. * fix. * return old eids in node_subgraph. * accelerate subgraph construction. * Add neighborhood op (#37) * add csr_neighborhood op * update neighborhood sample * Update csr_neighborhood_sample-inl.h * Update csr_neighborhood_sample-inl.h * Update csr_neighborhood_sample.cc * add graph compact operator. * fix a bug in dgl_subgraph. * fix a bug in dgl_graph_compact. * Update csr sample op (#39) * add csr_neighborhood op * update neighborhood sample * Update csr_neighborhood_sample-inl.h * Update csr_neighborhood_sample-inl.h * Update csr_neighborhood_sample.cc * Update csr_neighborhood_sample-inl.h * Update csr_neighborhood_sample.cc * Update csr_neighborhood_sample-inl.h * remove space. * move to dgl_graph to contrib. * move code. * move edge id. * fix compilation error. * add test for subgraph. * cleanup. * fix. * fix. * fix compile error. * fix compile error. * fix compile error. * fix. * add operator doc. * remove graph_compact * update doc. * address comments. * retrigger. * address comments. * retrigger * fix a bug in test. * retrigger * add check_format * Fixes #13386 - Refer Warnings (#13387) * Updated the paths for images for java tutorial (#13361) * Updated the paths for images * Empty commit * Empty commit * Nudge to CI * Fix/env disable mkldnn cache map (#13324) * add flag to disable mkldnn cache * update docs * fix typos * update var name * fix ordering * set cache size * fix log message * update docs * fix lint * fix lint * fix comparison * update method name * fix missing * fix logging * remove random item when cache exceeded * update helper name * update hash namespace * make ophash template * udpate function params * fix return * fix return * update return for helper * chagne class to typename * add typename * fix lint * update doc * pass ptr to cache * retrigger * retrigger * retrigger * change env var name to MXNET_MKLDNN_CACHE_NUM * fix log env name * retrigger * Initial website documentation for Java API (#13289) * Initial website documentation for Java API * Changing paths to be relative * Refactoring Java API website landing page * Update Java web docs based on feedback * Minor formatting fixes * Update maven repo to nightly build so that java will be available prior to 1.4.0 release * Adding java tutorial index to test_sanity_tutorials whitelist * Fix link to javadocs * Fix javadoc for infer package and minor install doc fix * Minor path fix * Replace mxnetci dockcross with public dockcross due to missing image (#13402) * Replace mxnetci dockcross with public dockcross due to missing image * Remove source lists change * Disable Jetson * Move to mxnetcipinned * Correct shapes of images in cifar10 and cifar100 (#13348) * Correct shapes of images in cifar10 and cifar100 cifar10 and cifar100 have 3 channels * Retrigger build * Updated recommenders example (#13041) * initial modification recommender * Recommender updates * fix notebooks * Update README.md * trigger build * Update README.md * Retrigger build * Improving multi-processing reliability for gluon DataLoader (#13318) * improving multi-processing reliability for gluon dataloader I found some multi-processing-related issues in the Gluon DataLoader. 1) Each time a _MultiWorkerIter shuts down, it could leave some dangling processes. The shutdown mechanism could not guarantee that all worker processes can be terminated. As a result, after running for several epochs, more and more dangling processes will stay there. This problem barely happens during training. In this case, there is a decent time interval between the last-batch data prefetching and the _MultiWorkerIter's shutting down). But the problem frequently happens 1) when I stop the iter before the end of an epoch, and 2) when I use the DataLoader for a data loading service and load the data as fast as possible. In both cases, the time interval between the most recent data prefetching and the iter shutdown are short. I guess that the _MultiWorkerIter iter is unable to shut down properly during active data prefetching. To fix this, I explicitly terminate the worker processes inside the shutdown function. 2) When loading data fast (still mostly during testing and data serving), there seems to be a risk of data racing. The data iter uses a _MultiWorkerIter to cache prefetched data, but the dict does not seem to be thread-safe for concurrent inserting and deleting elements. So occasionally, the data can be missing from the dict. To prevent this, I use a scope lock to guard the dict access. * do not wait for the workers to join, and kill any alive wokers as soon as possible * Onnx multi output (#13390) * Fix ONNX export to support multi-output graphs * Add ONNX unit-test * Added multi-output shape inference. - Removed unnecessary forward_pass() call - Modified infer_output_shape to return multiple shapes for multiple outputs as well as output names. * Fixed pylint * Change docker login (#13408) * Fixing doc links and minor edits for Java API (#13405) Update the main website links * Fix repeated typo in mxnet_op.h (#13406) * Use dynamic omp schedule for sparse dot with large matrix (#13398) * dynamic omp for dot update heuristic * add doc * Update mxnet_op.h * Update dot-inl.h * Added proper default value in cpp-package for optional (#13415) * Fix infoGan Gluon tutorial errors. (#13416) - Update notebook to avoid divide by 0 causing a warning. - Add MXBoard dependency. * :memo: Fixes #13388 Adds Clojure to MXNet installation docs (#13393) * Minor fixes to documentation (#13412) * Minor fixes to documentation * Updated the Maven Repository URL to point to staging repo * [Example] fix cpp example inception-bn and training acc issue (#13284) * fix inception-bn and training acc issue * add parameter initialization, fix lint * fix comparison * change optimizer to sgd * update sgd and update model name * add inception_bn in jenkins build * make max epoch an argument * remove inception_bn test * trigger ci * remove ci test * trigger ci * [Example]Fix mlp_csv example (#13273) * add instruction to get the data and fix typo * fix typo * update file name * trigger CI * add unit_test for unit_test_mlp_csv * add mlp_csv to jenkinsfile * revert jenkinsfile to another PR * trigger CI * trigger CI * Java doc (#13368) * Fix scaladoc and javadoc errors * Stop on errors starting on scala 1.3.x build * Adding Java to ubuntu setup install page and minor fixes to docs (#13420) * Adding Java to ubuntu setup install page and minor fixes to other java api docs * Improving javadoc for java-api predictor class Mostly documentation changes * [MXNET-1029] Feature request: randint operator (#12749) * randint operator add along with add optional tag to params * register param * lint space issue * randn issue fix * uniform_int_distribution doesn't support int8, uint8 fix * dtype ftype * ftype to dtype - invalid template arg * fix template arg issue * test with int dtype for windows * removed int8,uint8 from test * gpu implementation * gpu engine state diff * removed gpu support * empty commit * temporary fix : batchnorm flaky test skip * removed randn symbol specific code since other PR is on it * revert ndarray/randn for compatibility * added unit test for checking extremes and uniform distribution for sufficient samples * increased the high val * int32 to int64 support, indentation fix, check for optype correctly based on type of random function * gpu support, revert finfertype using template specialization, remove defaults, prints, test other low high val * fix for invalid template arg by checking for int32,int64 * gpu randint in random_generator * sample_uniform issue and param, removed old flaky test skip line * replaced discrete_uniform function by rand_int64 for consistency * formula update and removed itype * change ctx to include gpu, randint samepl_op.cu typo * trigger ci * doc fix, check fix, whitespace remove * added the without dtype testcase * Java demo file-path fix (#13358) * fix on ubuntu * add readme instruction * fix intellij Tutorials * fix intelliJ tutorial * fix the document * update demo * revert the change on intelliJ tutorial * fix make process * fix documentation * Updated README and NEWS with 1.3.1 release information (#13423) * Be more explicit about the exit status of the container (#13425) * [MKLDNN]Add quantized concat (#13297) * Add quantized concat * Fix non-mkldnn build * Add size check for MKLDNNQuantizedConcatForward * use all capital for constant * Rename constant with Google C++ style. * Address apeforest comments * Address apeforest comments * fix lint * Add frontend interface. * Retrigger CI * Add ARMv7 builds to dev_menu.py (#13432) * Add ARMv7 builds to dev_menu.py * Add Python3 CPU Intel MKLDNN unittests to dev_menu * [MXNET-1110] find the path to include header files (#13359) * add find_include_path API * address reviewer comment * change return type from list to string * add unit test * address reviewer comment * address reviewer comment * address reviewer comment * address reviewer comment * add subgraph adjacency operator. (#13396) * add adjacency. * fix lint. * add GPU impl. * retrigger * address comments. * Update dgl_graph.cc * Java added to install page (#13404) * added java install option * update maven blocks * update maven button url to snapshot search for java * add version; remove formatting on dependency * merge clojure updates * merge clojure updates - give code some breathing room * merge clojure updates - give code even more breathing room * 1.3.1 website updates (#13444) * 1.3.1 website updates * Java added to install page (#13404) * added java install option * update maven blocks * update maven button url to snapshot search for java * add version; remove formatting on dependency * merge clojure updates * merge clojure updates - give code some breathing room * merge clojure updates - give code even more breathing room * remove redundant link (#13428) * remove redundant link * retrigger * retrigger * [MXNET-886] ONNX export: HardSigmoid, Less, Greater, Equal (#12812) * ONNX export: Comparison operators * ONNX export: Hard sigmoid * Correct Inception Reference for Pertained Model (#13360) I noticed that the symbols and parameters in the model zoo are infact from https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/symbols/inception-bn.py, which is not inception v3. It is inception + batch normalization. In this commit, I update the documentation and link to the correct research basis. * exclude the error folder from sphinx toc (#13354) * exclude the error folder from sphinx toc * bumping commit for CI * Update MKL-DNN to fix LSTM perf regression (#13417) * update MKL-DNN CI id * fix the reorder perf issue * bumped version to v0.17.1 * bumped to MKL-DNN v0.17.1 * pin to v0.17.1 * Mitigate #13341 (#13343) - KL never succeeds so it always goes exponential - Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges * parallelize NDArray::Copy when data size is large (#12926) * parallelize NDArray::Copy by OpenMP when data size > MXNET_CPU_PARALLEL_COPY_SIZE * code specification according to reviewer's suggestions * align with std::memcpy api * add descriptive error message * update MXNET_CPU_PARALLEL_COPY_SIZE doc * update MXNET_CPU_PARALLEL_COPY_SIZE doc again * fix property not updating bug (#13085) * [MXNET-1222] Scala Inference enable different shapes input (#13330) * init commit with Predictor Improvement * add predictor Example * change into dArr * add img config * add new line and fix code style important bug fixes * Fix deconvolution / PR 13421 (#13433) * add test case * revert refactor * use with seed decorator * retrigger * remove seed * remove iteration * remove old test * update deconvolution test to have filter length that triggers mkldnn reorder * Add DGL subgraph sampling op (#13392) * add csr sample op * fix compile error in some platform * update * update openmp * speedup sampling * update csr * update csr * update time seed * update * fix compiler error * update doc * fix ci error * fix quantize_graph pass error when there're multiple outputs from a single node (#13000) * fix quantize_graph pass error when there're multiple outputs from a single node that need to insert 'contrib_quantize', 'min' and 'max' nodes for these outputs. * fix lint * Make the single output align with multiple outputs when inserting contrib_quantize * Change op comparing from its name to itself * skip unsupported quantize_concat * retrigger ci * Get the correct include path in pip package (#13452) * add find_include_path API * address reviewer comment * change return type from list to string * add unit test * address reviewer comment * address reviewer comment * address reviewer comment * address reviewer comment * fix include path problem in pip package * add comment * fix lint error * address reviewer comment * address reviewer comment * Use ~/.ccache as default ccache directory so is not cache is not erased on reboot (#13431) * Skip flaky test https://github.com/apache/incubator-mxnet/issues/13446 (#13480) * Rewrite dataloader with process pool, improves responsiveness and reliability (#13447) * fix recordio.py * rewrite dataloader with pool * fix batch as tuple * fix prefetching * fix pylint * picklable function * use pickle * add missing commit * Fix errors in docstrings for subgraph op; use code directive (#13463) * [MXNET-1158] JVM Memory Management Documentation (#13105) * update train_mnist * Add documentation for JVM Memory Management * update doc * address nit picks * address nit picks * Grammar and clarity edits for memory management doc * Edits for scala memory management * Update memory-management.md * Update memory-management.md * Update memory-management.md * capitalization fix * Update row_sparse tutorial (#13414) Update row_sparse tutorial * Add resiliency to onnx export code (#13426) * Added resiliency to onnx export code - With previous infer-shape implementation, if input shape was list instead of tuple or if extra non-existent parameters were provided, the code would still work. The fixes in this commit make sure that behavior is restored to prevent any compatibility issues with existing export code. * Fixed name of net in unittest * Fix pylint * [MXNET-1185] Support large array in several operators (part 1) (#13418) * fix a few operators with large arrays (# of elements) * fix bug in broadcast_div and add tests * address reviewer comment * add unit test * add empty line * retrigger CI * [MXNET-1210 ] Gluon Audio - Example (#13325) * Initialized the example * Addressed PR comments, about existing synset.txt file - no overwrite * RST - docstring issues fixed * added README * Addressed PR comments * Addressed PR comments, checking Divide by 0 * Raising error if format is not supported. * changed a line for ndarray of labels * Trigger CI * Trigger CI * PR comments addressed around skip_header argument * Addressed PR comments around librosa import * PR Comments * Passing lazy=lazy from argument * Added PR comments, labels to README.MD * Trigger CI * Addressing PR Comments in README * Modified README.md * Added example under audio folder * Retrigger CI * Retrigger CI * ONNX export: Instance normalization, Shape (#12920) * ONNX import/export: Make backend_rep common * ONNX export: Instance Normalization * ONNX export: Shape operator * Clarify dependency on OpenCV in CNN Visualization tutorial. (#13495) * clarify ops faq regarding docs strings (#13492) * Add graph_compact operator. (#13436) * add graph_compact. * fix. * add doc. * add tests for graph_compact. * address comments. * update docs. * trigger CI * Deprecate Jenkinsfile (#13474) * update github location for sampled_block.py (#13508) Updated to https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/model/sampled_block.py * #13453 [Clojure] - Add Spec Validations to the Optimizer namespace (#13499) * ONNX export: Logical operators (#12852) * Fix cmake options parsing in dev_menu (#13458) Add GPU+MKLDNN unittests to dev_menu * Revert "Manually track num_max_thread (#12380)" (#13501) This reverts commit 75410210e07a5fab5e044348aee276d578d5857e. * Feature/mkldnn static 2 (#13503) * build mkldnn as static lib * update makefile to statically build mkldnn * build static mkldnn * fix static name * fix static name * update static for mac * rename mkldnn dep in ci * remove moving mkldnn dynamic lib * remove commented code * remove mkldnn dnaymic for unitest * force static for mkldnn lib * remove dynamic mkldnn bind * only link windows * add mkldnn.mk * try force linking * remove mkldnn dynanmic check * remove test mkldnn install * fix spacing * fix index * add artifacts * add comment about windows * remove static * update makefile * fix toctree Sphinx errors (#13489) * fix toctree errors * nudging file for CI * Disabled flaky test test_gluon_data.test_recordimage_dataset_with_data_loader_multiworker (#13527) * [MXNET-1234] Fix shape inference problems in Activation backward (#13409) * Provide a failing test for ReLU activation shape inference bug * Fix Activation backward shape inference fixes: #13333 * Add softsign Activation to test_gluon.py * Use activation in GPU if we are using CUDNN and not MKLDNN as it's happening right now * Don't disable MKLDNN * Docs & website sphinx errors squished 🌦 (#13488) * fix scala ndarray docs; remove interpreter style * fix docs error in kvstore * remove interpreter format in examples * remove python indicator for these non-functioning python code blocks; clears a sphinx error * remove old table that was not being used and was triggering a sphinx error * get rid of curly braces that was causing a pygments error * fix ambiguous reference causing sphinx error * nudging file for CI * [MXNET-1235] Add a test for AdaMax optimizer (#13467) * Add a test for AdaMax optimizer * Modify nested for loop with itertools.product and left tolerance to default * Trigger * Adadelta optimizer test (#13443) * adadelta test * comments * Update java setup docs for 1.4.0 (#13536) * Update java setup docs for 1.4.0 * Update Java-demo to 1.4.0 * Revert "Feature/mkldnn static 2 (#13503)" (#13540) This reverts commit 65edc9500b10a3404945d6d79acbae54a2833890. * doc fix (#13465) * [MXAPPS-1020] Clean up some Sphinx warnings. (#13539) * [MXNET-1110] Add header files required by horovod (#13062) * Add header files required by horovod * Add symbolic link and cherry picked required header * add python API to return include path * update link * fix windows CI * fix windows build * fix dlpack link * merge with master * exclude 3rd party header files from license check * exclude license check * exclude include directory * remove commented lines * Bumped minor version from 1.4.0 to 1.5.0 on master, updated License file (#13478) * updated to v1.5.0 * Bumped minor version from 1.4.0 to 1.5.0 on master * added Anirudh as maintainer for R package ... adding something useful and re-trigger PR check * Updated license file for clojure, onnx-tensorrt, gtest, R-package * Get the correct include path in pip package (#13452) * add find_include_path API * address reviewer comment * change return type from list to string * add unit test * address reviewer comment * address reviewer comment * address reviewer comment * address reviewer comment * fix include path problem in pip package * add comment * fix lint error * address reviewer comment * address reviewer comment * Use ~/.ccache as default ccache directory so is not cache is not erased on reboot (#13431) * Skip flaky test https://github.com/apache/incubator-mxnet/issues/13446 (#13480) * Rewrite dataloader with process pool, improves responsiveness and reliability (#13447) * fix recordio.py * rewrite dataloader with pool * fix batch as tuple * fix prefetching * fix pylint * picklable function * use pickle * add missing commit * Fix errors in docstrings for subgraph op; use code directive (#13463) * [MXNET-1158] JVM Memory Management Documentation (#13105) * update train_mnist * Add documentation for JVM Memory Management * update doc * address nit picks * address nit picks * Grammar and clarity edits for memory management doc * Edits for scala memory management * Update memory-management.md * Update memory-management.md * Update memory-management.md * capitalization fix * Update row_sparse tutorial (#13414) Update row_sparse tutorial * Add resiliency to onnx export code (#13426) * Added resiliency to onnx export code - With previous infer-shape implementation, if input shape was list instead of tuple or if extra non-existent parameters were provided, the code would still work. The fixes in this commit make sure that behavior is restored to prevent any compatibility issues with existing export code. * Fixed name of net in unittest * Fix pylint * [MXNET-1185] Support large array in several operators (part 1) (#13418) * fix a few operators with large arrays (# of elements) * fix bug in broadcast_div and add tests * address reviewer comment * add unit test * add empty line * retrigger CI * [MXNET-1210 ] Gluon Audio - Example (#13325) * Initialized the example * Addressed PR comments, about existing synset.txt file - no overwrite * RST - docstring issues fixed * added README * Addressed PR comments * Addressed PR comments, checking Divide by 0 * Raising error if format is not supported. * changed a line for ndarray of labels * Trigger CI * Trigger CI * PR comments addressed around skip_header argument * Addressed PR comments around librosa import * PR Comments * Passing lazy=lazy from argument * Added PR comments, labels to README.MD * Trigger CI * Addressing PR Comments in README * Modified README.md * Added example under audio folder * Retrigger CI * Retrigger CI * ONNX export: Instance normalization, Shape (#12920) * ONNX import/export: Make backend_rep common * ONNX export: Instance Normalization * ONNX export: Shape operator * Clarify dependency on OpenCV in CNN Visualization tutorial. (#13495) * clarify ops faq regarding docs strings (#13492) * Add graph_compact operator. (#13436) * add graph_compact. * fix. * add doc. * add tests for graph_compact. * address comments. * update docs. * trigger CI * Deprecate Jenkinsfile (#13474) * update github location for sampled_block.py (#13508) Updated to https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/model/sampled_block.py * #13453 [Clojure] - Add Spec Validations to the Optimizer namespace (#13499) * ONNX export: Logical operators (#12852) * Fix cmake options parsing in dev_menu (#13458) Add GPU+MKLDNN unittests to dev_menu * Revert "Manually track num_max_thread (#12380)" (#13501) This reverts commit 75410210e07a5fab5e044348aee276d578d5857e. * Feature/mkldnn static 2 (#13503) * build mkldnn as static lib * update makefile to statically build mkldnn * build static mkldnn * fix static name * fix static name * update static for mac * rename mkldnn dep in ci * remove moving mkldnn dynamic lib * remove commented code * remove mkldnn dnaymic for unitest * force static for mkldnn lib * remove dynamic mkldnn bind * only link windows * add mkldnn.mk * try force linking * remove mkldnn dynanmic check * remove test mkldnn install * fix spacing * fix index * add artifacts * add comment about windows * remove static * update makefile * fix toctree Sphinx errors (#13489) * fix toctree errors * nudging file for CI * Disabled flaky test test_gluon_data.test_recordimage_dataset_with_data_loader_multiworker (#13527) * [MXNET-1234] Fix shape inference problems in Activation backward (#13409) * Provide a failing test for ReLU activation shape inference bug * Fix Activation backward shape inference fixes: #13333 * Add softsign Activation to test_gluon.py * Use activation in GPU if we are using CUDNN and not MKLDNN as it's happening right now * Don't disable MKLDNN * Fixing a 404 in the ubuntu setup doc (#13542) * [MXNET-1249] Fix Object Detector Performance with GPU (#13522) * Reduce post processing time * fix ssd * fix the CI * add comments * [MXNET-769] Use MXNET_HOME in a tempdir in windows to prevent access denied due t… (#13531) * Use MXNET_HOME in cwd in windows to prevent access denied due to concurrent data downloads Fixes #13484 * Revert "Disabled flaky test test_gluon_data.test_recordimage_dataset_with_data_loader_multiworker (#13527)" This reverts commit 3d499cb3584919b767142c5596211a7f7fb18d50. * Add a retry to qemu_provision (#13551) Fixes #13504 * Fix #13521 (#13537) * fix pool release * fix * Simplifications and some fun stuff for the MNIST Gluon tutorial (#13094) * Simplify mnist Gluon tutorial and add mislabelled sample plotting * Add mnist Gluon tutorial images * Gluon MNIST tutorial: Use modern Gluon constructs, fix some wordings * [Gluon] Move to data loaders and improve wording in MNIST tutorial * Fix broken links * Fix spelling of mislabeled * Final rewordings and code simplifications * Fix things according to review - Apply hybrid blocks - Move outputs outside of code blocks and mark for notebooks to ignore - Remove images, use external link - Fix a few formulations * Change activations to sigmoid in MNIST tutorial * Remove superfluous last layer activations in MNIST tutorial * Updated docs for randint operator (#13541) * updated docs for randint * added randint in __all__ and reordered acc to categorical then alphabetical * Trigger CI * minus mxnet.symbol and alphabetical for ndarray,symbol.md * alphabetical order * Chi_square_check for discrete distribution fix (#13543) * check for bucket instead of index * enumerate instead of range(len()) * count instead of sum to solve attribute error * revert to sum * seperate discrete and continuous * Trigger CI * Revert "Bumped minor version from 1.4.0 to 1.5.0 on master, updated License file" (#13558) * Revert "Chi_square_check for discrete distribution fix (#13543)" This reverts commit cf6e8cbd035bf315b3e8280416468a629c780d03. * Revert "Updated docs for randint operator (#13541)" This reverts commit e0ff3c36ee171386fef01fb86c54c343e4b04c14. * Revert "Simplifications and some fun stuff for the MNIST Gluon tutorial (#13094)" This reverts commit 8bbac827742c21607a863137792f03bd09847419. * Revert "Fix #13521 (#13537)" This reverts commit f6b4665995f8f8ff32862a029b2074475d8467eb. * Revert "Add a retry to qemu_provision (#13551)" This reverts commit f6f840110d74111f98c20eab5b08d64a46ebf0cd. * Revert "[MXNET-769] Use MXNET_HOME in a tempdir in windows to prevent access denied due t… (#13531)" This reverts commit bd8e0f8356676749ecae16ec38a366b4cc00bf15. * Revert "[MXNET-1249] Fix Object Detector Performance with GPU (#13522)" This reverts commit 1c8972c3c8f832519364916865541f48597581c7. * Revert "Fixing a 404 in the ubuntu setup doc (#13542)" This reverts commit cb0db290adcfd0fce956d02c234f81d453e41013. * Revert "Bumped minor version from 1.4.0 to 1.5.0 on master, updated License file (#13478)" This reverts commit 40db61908000ee86d21aac847ff2225807d6c168. * #13441 [Clojure] Add Spec Validations for the Random namespace (#13523) * Adding test for softmaxoutput (#13116) * Add workspace cleaning after job finished (#13490) * Add workspace cleaning after job finished * Update Jenkinsfile_utils.groovy * Update Jenkinsfile_utils.groovy * Fix flaky test test_random:test_randint_generator (#13498) * updated seed, alpha value, comments * typo in comment fix * added nrepeat * removed unusued variable, added link for scipy alpha, rephrased the sentence for discrete distribution buckets * removed fixed seed, alpha * Update version to v1.5.0 including clojure package (#13566) * Update DESCRIPTION * update version to v1.5.0 except for clojure * update version from 1.4.0 to 1.5.0 - add utility script to help bump versions in future - fix README to correct to current maven versions * License update (#13565) * Update LICENSE * update license for Clojure, R, ONNX-TRT and location of 3rd party dependencies. * fixed typo * Fix use-before-assignment in convert_dot (#13511) * fix the situation where idx didn't align with rec (#13550) minor fix the image.py add last_batch_handle for imagedeiter remove the label type refactor the imageiter unit test fix the trailing whitespace fix coding style add new line move helper function to the top of the file * Update MXNetTutorialTemplate.ipynb (#13568) Fix typos * ONNX import/export: Size (#13112) * fix link for gluon model zoo (#13583) * Fix exception handling api doc (#13519) * Fix exception handling api doc * Update waitall api doc Co-Authored-By: anirudh2290 * add cpp example inception to nightly test (#13534) * add inception test * fix max iter for mlp * rename and add comment * rename epoch num * Add notes about debug with libstdc++ symbols (#13533) * Add imresize and copyMakeBorder to mx.image (#13357) * Add imresize API to docs * address comments * copyMakeBorder * [MXNET-1253] fix control_flow_op (#13555) * fix control_flow_op * change type for M * add test for sparse where op * Add Intel MKL blas to Jenkins (#13607) * add mkl blas to Jenkins * add mkl install script * fix bug in mkl script * remove python2 ut and add cpu-mkl node * #13385 [Clojure] - Turn examples into integration tests (#13554) * fix the Float not showing correctly problem (#13617) Merge this PR for 1.4.x * [MXNET-1155] Add scala packageTest utility (#13046) * [MXNET-1155] Add scala packageTest utility * Clean up utility * Safe change directory in Makefile for scala * mvn install file instructions with details * [MXNET-1224]: improve scala maven jni build and packing. (#13493) Major JNI feature changes. Please find more info here: https://cwiki.apache.org/confluence/display/MXNET/Scala+maven+build+improvement * [MXNET-1225] Always use config.mk in make install instructions (#13364) * Always use config.mk in make install instructions * Specify Cuda 0 for ubuntu with mkldnn * Scala install doc avoid build_from_source Minor doc fixes * Fix build_from_source CMake usage * CPP Install Instruction with CMake * Use cmake out of source build * Fix warning in waitall doc (#13618) * Optimize C++ API (#13496) * Optimize C++ API Pass parameter with reference instead of value. Add const as well as it is not changed. * fix docs/architecture/overview.md Fix BinaryShapeFunction typedef Add a right brace for SmoothL1Shape_ * fix quantize pass error when the quantization supported Op are excluded in the model (#13596) * Scripts for building dependency libraries of MXNet (#13282) * openblas script * ps-lite dependencies * USE_S3 dependencies * image libraries * license * add batch norm test (#13625) * add batch norm test * fix formatting * use out_arr as input * fix typo * remove const * use ptr * eval ptr * Set install path for libmxnet.so dynamic lib on Mac OS (#13629) * Fix the bug of BidirectionalCell (#13575) * Fix the bug of BidirectionalCell I did hybridize( ) and pass "valid_length" to the unroll( ) function of BidirectionalCell, then returned AssertionError in line 79. Because symbol.split( ) return a symbol but not a symbol list. Result in the length of inputs dont equal parameter "length" when call unroll( ) to compute r_outputs and r_states. * add a test for BidirectionalCell * Fix the bug of BidirectionalCell I did hybridize( ) and pass "valid_length" to the unroll( ) function of BidirectionalCell, then returned AssertionError in line 79. Because symbol.split( ) return a symbol but not a symbol list. Result in the length of inputs dont equal parameter "length" when call unroll( ) to compute r_outputs and r_states. * fix test_bidirectional_unroll_valid_length( ) Fix the error of parameter. * Fix the bug of BidirectionalCell I did hybridize( ) and pass "valid_length" to the unroll( ) function of BidirectionalCell, then returned AssertionError in line 79. Because symbol.split( ) return a symbol but not a symbol list. Result in the length of inputs dont equal parameter "length" when call unroll( ) to compute r_outputs and r_states. * fix test_bidirectional_unroll_valid_length( ) * Feature/mkldnn static (#13628) * Revert "Revert "Feature/mkldnn static 2 (#13503)" (#13540)" This reverts commit a3eca5f5c96eed0bc29bd4e58e470997091a1fb3. * include headers on mkldnn lib * retrigger * retrigger * build config for maven and pip (#13556) * config for pip * symbol whitelist * maven build config * Fix for import mxnet taking long time if multiple process launched (#13602) * https://github.com/apache/incubator-mxnet/issues/12255 doing import mxnet in multiple processes take very long time. Details : #12255 One of the reason we have OMP tuning code which iterates to find OMP tune overhead. We are reducing this iteration count to reduce the overehead of tuning code. Also, We added an environment variable where users can set the number of cores that should be used to determine tuning. * cpplint fix * Adding new environment variable: MXNET_USE_NUM_CORES_OPERATOR_TUNING to doc * fixing formatting in doc * Add reshape op supported by MKL-DNN (#12980) * Add reshape op supported by MKL-DNN * fix build issue * fix lint * fix lint * fix lint * fix lint * fix lint * fix lint * fix white space * add unit test * merge if blocks * Improve dev_menu usability, local build and virtualenv (#13529) * Improve dev_menu, add build command and virtualenv creation with local builds for easy testing * Update dev_menu.py Co-Authored-By: larroy * Cuda off by default, use ccache * address CR * [Clojure] Correct the versions in the README so they correspond to the latest maven.org release (#13507) * Correct the versions so they correspond to the latest maven.org release * trigger build * feedback from @kohr-h * Optimization of metric evaluation (#13471) * Change argsort to argpartition * Global statistics in metrics * Fix lint * Fixes from review * Trigger * Fixes from review, fix to F1, MCC and perplexity metrics, added test for global stats * Fix lint * Fix compatibility with Python 2 * Revert "Feature/mkldnn static (#13628)" (#13638) This reverts commit 5bcf2bd6e8b48fa27bfcfdafd06401ec2d28978b. * support mkl log when dtype is fp32 or fp64 (#13150) * support mkl log when dtype is fp32 or fp64 * remove macro * ensure data size less than or equal MKL_INT_MAX * code specification * fix indent * for retrigger * [MXNET-1209] Tutorial transpose reshape (#13208) * transpose tutorial * Adding Anirudhs comments * Update tutorial with some more examples * Adding links * Fixing the links, adding more examples * Update reshape_transpose.md * Fixing spelling mistakes * Updating image resolution * Adding Simon's comments * Small fixes * Update reshape_transpose.md * Update reshape_transpose.md * empty commit * empty commit * updated reference to Apache MXNet (#13645) * Complimentary gluon DataLoader improvements (#13606) * init * add tests * doc * lint * fix openmp * Improve CCache handling (#13456) * Remove gitignore entries * Modify Makefile * Modify user permissions * Add new ccache wrapper function * Change PATH rewrite to a different one to resolve CUDA issues * Add ccache to gpu cmake * Enable ccache for every build * Set permissions for arm dockerfiles * Disable ccache for ASAN * Remove g++-8 ccache redirect * Update Android Dockerfiles for user permissions * Fix ASAN compiler typo * Remove sanity for speed * Move build dir creation in android armv8 * Revert "Remove sanity for speed" This reverts commit e8386a774dafe96337930b9cac36cb24fc36585e. * Add ccache for NVCC in Makefile * [MXNET-918] Random module (#13039) * introduce random API * revert useless changes * shorter types in APIDoc gen code * fix after merge from master * Trigger CI * temp code / diag on CI * cleanup type-class code * cleanup type-class code * fix scalastyle * Fix incorrect delete in MXExecutorReshape exception handling (#13376) * Fix bad delete. Delete the pointed-to handle on cleanup, not the location of the handle itself. Also don't delete it if we didn't set it in the first place. * Remove unusued 'exec' var from MXExecutorBindEX. * [MXNET-1251] Basic configuration to do static-linking (#13621) * Basic configuration to do static-linking * update build script and place it in the install part * clean up the code further * revert maven into build-from-source * add curl to deps * [MXNET-1195] Cleanup Scala README file (#13582) * Updated the Scala-Readme with upto-date information * Updated the header * Removed redundant build status * Minor formatting changes * Addressed the PR feedback * Added section on Scala training APIs * Removed mention of deprecated Model API * scripts for building libmxnet binary and wheel (#13648) * add script for making all dependencies * tools for building pip package * build scripts for lib and wheel * [MXNET-1083] Add the example to demonstrate the inference workflow using C++ API (#13294) * [MXNET-1083] Add the example to demonstrate the inference workflow using C++ API * [MXNET-1083] Add the example to demonstrate the inference workflow using C++ API * Updated the code to address the review comments. * Added the README file for the folder. * Addressed the review comments * Addressed the review comments to use argmax and default mean values. * Update MKLDNN_README.md (#13653) * Support Quantized Fully Connected by INT8 GEMM (#12922) * add quantized fully connect support * disable qfc cpu case since s8u8s32 is only supported by MKL BLAS library * retrigger to ci testing * move implementation to cc file and add STORAGE_TYPE_ASSIGN_CHECK * fix typo bug * retrigger the ci test * fix typo bug * retrigger ci * retrigger the ci test * retrigger the ci * retrigger the ci test * retrigger ci test * fix indent issue * retrigger the ci * retrigger the ci test * add verbose message * update log message * using range for loop * using for auto range * enable MKL BLAS ci test * fix typo issue * use TYPE_ASSIGN_CHECK * retrigger the ci * add build fix for Scala/Java build (#13655) * Fix Jetson compilation (#13532) * remove omp which can cause ssd accuracy variance (#13622) * Revert "[MXNET-43] Fix Jetson compilation" (#13665) * Revert "remove omp which can cause ssd accuracy variance (#13622)" This reverts commit 655f1c6f7a0706dd622f73db9af2e6df895ca213. * Revert "Fix Jetson compilation (#13532)" This reverts commit 48e25c4cae355753dd96ea7afe004bf78e0719e4. * Fix Jetson compilation (#13666) * turn on Sphinx warnings as errors (#13544) * turn on warnings as errors * move warnings as error logic to build_all_version * fix typo in comment * add warning as error option for docs pipeline * bump ci to test again; use this chance to add notes on this feature * fix bugs in image.py docs * Update CODEOWNERS, add Pedro Larroy. (#13579) * Revert "Revert "[MXNET-43] Fix Jetson compilation" (#13665)" (#13672) This reverts commit 3433776dac7be75928082bbc1d552fca248fb8e8. * Accelerate DGL csr neighbor sampling (#13588) * Speedup and fix bug in dgl_csr_sampling op * Update dgl_graph.cc * simplify functions. * avoid adding nodes in the last level in the queue. * remove a hashtable lookup in neigh_pos. * reduce a hashtable lookup in sub_ver_mp. * merge copying vids and layers. * reduce hashtable lookup when writing to output csr. * fix a bug. * limit the number of sampled vertices. * fix lint. * fix a compile error. * fix compile error. * fix compile. * remove one hashtable lookup per vertex and hashtable iteration. * remove queue. * use vector for neigh_pos. * fix lint * avoid init output arrays. * fix tests. * fix tests. * update docs. * retrigger * retrigger * [MXNET-1252][1 of 2] Decouple NNVM to ONNX from NNVM to TenosrRT conversion (#13659) * fix unpicklable transform_first on windows (#13686) * Move the debug output message into MXNET_MKLDNN_DEBUG (#13662) * NEWS.md backport from v1.4.x to master (#13693) * merge NEWS.md from 1.4.x to master * NEWS.md backport from v1.4.x to master * Fallback to dense version for grad(reshape), grad(expand_dims) (#13599) * fallback to dense version for grad(reshape), grad(expand_dims) * add _backward_reshape gpu version * reshape test case comments * fix gpu test * remove mkldnn support for _backward_reshape * ONNX export: Add Flatten before Gemm (#13356) * Add Flatten before Gemm * ONNX export test: Allow multiple inputs in forward pass * ONNX export: Test for fully connected * [MXNET-1164] Generate the document for cpp-package using Doxygen (#12977) * Adding cpp-package directory to the Doxyfile. Updating the index.md file in c++ api directory. * Updating the link to classes in C++ API to point to correct html file. * Updated the links to use relative paths. * Removed the extra slash character in the url * Excluded the 3rdparty folder as per the review comment. * Update git clone location to apache github (#13706) * Add timeout/retry logic to docker cache download (#13573) * Added timeout/retry (linear backoff) to docker cache download * Units changed, as time.sleep takes seconds as argument * Improved error handling * Using retry decorator * Added retry decorator to _login_dockerhub method * Fixed wrong import * Fix NDArray ToDLPack Bug (#13698) * Added javadocs and improved example instructions (#13711) * Rearrange tests written only for update_on_kvstore = True (#13514) * Update test_gluon_trainer.py * Update test_gluon_trainer.py * test * Update mshadow to support batch_dot with fp16. (#13716) * fp16 dot * update mshadow * update mshadow * update mshadow * Fix the quantization script to support Python2 (#13700) * fix the quantization script to support python2 * Fix comments, fix similiar issue in imagenet_inference.py * ONNX test code cleanup (#13553) * ONNX test code cleanup * Make tests use the common test case list * Remove import test_cases * Make Gluon backend rep common * Partially enable broadcast tests * Common function to populate tests * Make backend common * test models * Test nodes * ONNX export: Test for fully connected * Edit CI scripts mxnet export test cleanup * Further cleanup backend tests * README * Some corrections * test case format for test_models * update social media section (#13705) * script for installing gpu libraries and build tools (#13646) * Port of scala infer package to clojure (#13595) * Port of scala infer package to clojure * Add inference examples * Fix project.clj * Update code for integration tests * Address comments and add unit tests * Add specs and simplify interface * Minor nit * Update README * update code owner (#13737) * AdamW operator (Fixing Weight Decay Regularization in Adam) (#13728) * tests * remove optimizer and move op to contrib * rename parameter * ONNX import/export: Add missing tests, ONNX export: LogSoftMax (#13654) * Logsoftmax, missing tests * Support multiple outputs in Gluon backendrep * Remove repeated unsqueeze test * Allow multiple output support * ONNX test code cleanup - part 2 (#13738) * Common test caller * Remove incorrect comment * Make corrections to CI * fix ci script * Update basic_layers.py (#13732) * ONNX import: Hardmax (#13717) * ONNX import: Hardmax * Fix lint errors * add github link for issue with reshape * gluon docfix (#13631) * Fixes for trainer with update_on_kvstore=False (#13721) * add clarification for param_dict * more tests for dist kvstore * more unittests * fix a bug * more dist exception test * revert optimizer list * fix bug and comment * fix doc rendering and lint * add invalid sched test * fix website * trigger * update doc * Reorder module import orders for dist-kvstore (#13742) * Reorder module import orders for dist-kvstore * more code comments * CMake: Enable installation of cpp-package headers (#13339) * Allow CMake based installation of cpp-package * Add installation of missing nnvm headers * Add documentation as to where public headers will be installed * disable error checking when building old versions (#13725) * Integrate MKLDNN Conv1d and support 3d layout (#13530) * add 3d layout support for MKLDNN Conv and Activation * fix lint * code refactor * add testcase for group1 conv and skip quantization for conv1d * fix lint * avoid conv1d quantization * code refactor and add activation ut * del todo * Making MKL-DNN default on MXNet master (#13681) * mkldnn is default makefile and explicitly turn off for buidls * add endif * retrigger * retrigger * build mkldnn as static lib * update makefile to statically build mkldnn * build static mkldnn * fix static name * fix static name * update static for mac * rename mkldnn dep in ci * remove moving mkldnn dynamic lib * retrigger * remove commented code * retrigger * remove mkldnn dnaymic for unitest * retrigger * retrigger * force static for mkldnn lib * turn of mkldnn on arm builds * remove dynamic mkldnn bind * update jenkins to use only mkldnn * remove last flag * turn mkldnn by default on mac * move mkldnn files for GPU MKLDNN build * copy lib mxnet in gpu build * only link windows * add mkldnn.mk * try force linking * retrigger * retrigger * remove mkldnn dynanmic check * use ifndef * remove test mkldnn install * fix spacing * fix index * remove cp of mkldnn since statically linked * add libmkldnn.a to list of files to pack * include mkl_ml * add mkldnn to pack * add libiomp to ci pack * move static libs * fix typo * pack mkldnn * retrigger * add linux artifacts * move libmkldnn in gpu cmake build * move libmkldnn and libiomp5 on gpu workspace * move linked files * fix typo * fix typo * add artifacts for tensorrt * move mkldnn lib in scala build * move mkldnn lib on cpu scala * create dir for binding * rename libmkldnn in scala * move mklml dep in scala builds * move mkl to another linked folder * move libmkl to another dir * add libmklml * move mkldnn * move mkldnn on centos * specify new dynamic path * retrigger * remove mkldnn dynamic lib * remove moving mkldnn artifact * add ld path * retrigger * Revert "remove moving mkldnn artifact" This reverts commit 16cca196e9e1ad92db74f4e8a01b3b052076d268. * Revert "remove mkldnn dynamic lib" This reverts commit d51043622d4ef7fcb95aff6a3e84d91ab71b48c9. * update makefile * Revert RPATH change and trigger CI * correcting use-mkldnn flags for two tests * mkldnn default on linux for starters * reverting naming rules of pack_lib * adding mkldnn=0 flags to centos non mkldnn builds * adding mkldnn=0 flags to ubuntu gpu non mkldnn builds * removing mkldnn binary operation for ubuntu gpu cmake non mkldnn build * removing mkldnn binary operation for centos non-mkldnn unittest * adding explicit USE_MKLDNN=0 flags for clang builds * adding explicit USE_MKLDNN=0 flags for cpu ubuntu builds * removing mkldnn binaries from non mkldnn builds scala gpu * adding explicit flag mkldnn=0 for tensorrt gpu build * adding explicit flag mkldnn=0 for ubuntu cmake asan * adding centos cpu mkldnn tests to CI * adding CentOS GPU MKLDNN build and unittest * not keeping mkldnn default for mac os * setting mkldnn default for x86_64 only * running docs with mkldnn=0 flag * removing CentOS CPU Scala MKLDNN test * setting mkldnn default for x86_64 only * not making mkldn default on windows * removing Centos MKLDNN tests from CI * retrigger * retrigger * retrigger * use relative links; update links (#13741) * [MXNET-1231] Allow not using Some in the Scala operators (#13619) * add initial commit * update image classifier as well * create Util class make Some conversion * add test changes * adress Comments * fix the spacing problem * fix generator base * change name to Option * fix bug in profiler tutorial when using cpu (#13695) try except approach only goes to ctx=mx.gpu() because test_utils.list_gpus() at least returns empty array and do not producing error * local docs build feature (#13682) * make ROIAlign support position-sensitive pooling (#13088) * make ROIAlign support position-sensitive pooling * add unittest for RoIAlign op * fix ccplint error * fix python3 compability for unittest * change OMP for better performance * delete blank line to trigger CI * add shape check when position_sensitive is true * fix the typo * typo: shuold -> should * remove private() clause in omp statement * add examples and fix the dependency problem (#13620) * add examples and fix the dependency problem * add Nightly run and optimized script * add explanation for the line * Update Adam optimizer documentation (#13754) * Less cudaGet/SetDevice calls in Gluon execution (#13764) * Remove unnecessary cudaGetDevice/cudaSetDevice calls * Fixes for the DeviceGuard * Retrigger CI * Fix for possible invalid device ordinal when using DeviceStore while driver is unloading * Fix for RTC when the driver API call is the first call * Added DeviceStore to pooled engine * Scope requests so it's not needed for dev_menu (#13771) * Fix USE_MKLDNN check in Makefile (#13775) * fix makefile * change make/config.mk * add comments * retrigger ci * fix c complier to clang (#13778) * Fixed mailing list addresses (#13766) * [MXNET-1255] update hybridize documentation (#13597) * update hybridize documentation * address review comments * improve doc * address comments * address comments * [MXNET-244] Work around likely compiler bug on nested inlines and temporary acces… (#13535) * Work around likely compiler bug on nested inlines and temporary access to stream * Don't compile khatri_rao tests if we don't have LAPACK * Address CR comment * Use curl to download sample data instead of wget. (#13761) * fix bipartite match memory corruption (#13727) * remove attributs clear on TRT nodes for GetOptimizedSymbol (#13703) * Add CPU test coverage and refine cmake builds (#13338) * add license (#13793) * [MXNET-862] Basic maven jenkins pipeline (#13450) * Jenkins Publish Nightly Maven Progress * Seperate Build, Test, and Deploy Stages with parallel * Re-organize Scala maven build (#13626) * Re-organize scala maven build 1. Automatically detect which platform to build for scala. 2. Remove platform dependend submodules 3. Fix cyclic module dependencies 4. Fix scalatype style check 5. Now mvn can be executed in submodule 6. Maven build can be executed from any directory not only in root project 7. Checkin javah header file, and use verify task to detect native API changes 8. Improve incremental build performance 9. Remove unittest and integration-test profile, use proper task instead 10. Delete generated scala file during maven clean. * Redo maven deploy related tasks. 1. Removed maven release plugin. 2. Make maven build friendly to CI, allow cli override version. 3. Moved gpg signing to deploy stage. 4. Created a separeated deploy module. 5. Updated Makefile to new maven build change. 6. Remove unused nexus-staging-plugin 7. Added nightly and staging profile for CI. * Support mkldnn for Scala. * Add extra header file to export for error checking (#13795) * add extra header file to include * fix sanity check * fix sanity * move c_api_common.h to include folder * fix build error * keep c_api_common.h internal * strip out error handling API into a separate header * consolidate comment into one paragraph per review * remove unnecessary include * fix redirection issues; set default version to master (#13796) * [MXNET-898] ONNX import/export: Sample_multinomial, ONNX export: GlobalLpPool, LpPool (#13500) * ONNX import/export: Sample_multinomial * ONNX export: GlobalLpPool, LpPool * Handle default p_value * Add tests for multinomial, lppool, globallppool * add a comment about shape test * whitelist symbols for using MXNet error handling externally (#13812) * fix for params with no dims in onnx (#13413) * fix for params with no dims * fix * fix * retrigger build * test added * retrigger CI * retrigger ci * Remove semicolon in libmxnet.sym file (#13822) * Remove semicolon in libmxnet.sym file * empty commit to trigger CI * Clojure example for fixed label-width captcha recognition (#13769) * Clojure example for fixed label-width captcha recognition * Update evaluation * Better training and inference (w/ cleanup) * Captcha generation for testing * Make simple test work * Add test and update README * Add missing consts file * Follow comments * Update LICENSE File with subcomponents (#13808) * Update LICENSE File with subcomponents * Fix JavaScript licenses * Dockerfiles for Publish Testing (#13707) * Add new Maven build for Scala package (#13819) * clean up build * fix minor issue and add mkldnn * fix mx_dist problem * fix clojure build * fix skip test * ONNX ops: norm exported and lpnormalization imported (#13806) * ReduceL1, l2 export, lpnormalization import added * fix * fix * fix * fix * remove useless code (#13777) * Fixing a symlink issue with R install (#13708) * fix minor indentation (#13827) * [MXNET-880] ONNX export: Random uniform, Random normal, MaxRoiPool (#13676) * ONNX export: Random uniform, Random normal * ONNX export: MaxRoiPool * tests for maxroipool, randomnormal, randomuniform * onnx export ops (#13821) * onnx export ops * retrigger ci * retrigger ci * fix * [MXNET-1260] Float64 DType computation support in Scala/Java (#13678) * Added Float64 as a supported datatype in NDArray * Added unit tests for Float64 in NDArray * Fix for failing Clojure unit tests * Added Float and Double as MX_PRIMITIVES for computation in Scala * Trying out second approach --> Private Impl methods with generic signature, and public methods calling the Impls * Fixed errors in *= method * Added Float64 in IO.scala and DataIter.scala * Added another testcase for IO.DataDesc creation * Fixed failing CI * Added Float64 in Predictor class * Added Float64 in Classifier class * Added Double as a possible return type to : classifyWithNDArray * Added unit tests for Classifier and Predictor.scala classes for Float64/Double * Approach 3 --> Using a trait to mirror Float and Double in Scala * Added comments on MX_PRIMITIVES.scala * Added Float64/Double support for inference in ImageClassifier APIs * Added unary- and compareTo in MX_NUMBER_LIKE * Renamed MX_NUMBER_LIKE to MX_PRIMITIVE_TYPE * Fixed linting issue * Now specifying dType from the available data in copyTo and MXDataIter.scala for creating a new DataIterator * Add primitives support handling to the generator for proper conversion * Reduced code duplication in classify method in Classifier.scala * Fix infer package for new signatures and address some bugs * Removed code duplication in getPixelsArray * remove debugging * Changed classifyWithNDArray method in Classifier.scala * Removed code duplication in predictImpl * Satisfying lint god _/\_ * Fixed failing PredictorSuite test * Renamed MX_FLOAT to Camel case * Revert "Renamed MX_FLOAT to Camel case" This reverts commit 9d7c3ce6f9c4d6ed2c46041a02e23c0f1df8dfe5. * Added an implicit conversion from int--> float to support int operations in NDArrays. (These ops were already supported in the previous versions) * Added Float64 as a training option to ImClassification Suite. Also added integration tests for it * Satisfy Lint God _/\_ * Added Float64 support in Java NDArray * Added Float64 support in Java's Predictor API * Added yours truly to the Contributors list * Added method comments on Predictor.predict with Array[Double] as a possible input * Added method comments explaining what MX_PRIMITIVE_TYPE is * Fixed errors cause by rebasing with master * Added licences to the files * [MXNET-1263] Unit Tests for Java Predictor and Object Detector APIs (#13794) * Added unit tests for Predictor API in Java * Added unit tests for ObjectDetectorOutput * Added unit tests for ObjectDetector API in Java * Addressed PR comments * Added Maven SureFire plugin to run the Java UTs * Pom file clean up -- moved surefire plugin to parent pom.xml * Renamed skipTests to SkipJavaTests * Fix scala doc build break for v1.3.1 (#13820) * Fix doc build break for v1.3.1 * ignore errors on v1.3.x during scala docs gen * Remove MXNET_STORAGE_FALLBACK_LOG_VERBOSE from test_autograd.py (#13830) * Add Local test stage and option to jump directly to menu item from commandline (#13809) * Removes unneeded nvidia driver ppa installation (#13814) * Improve license_header tool by only traversing files under revision c… (#13803) * Improve license_header tool by only traversing files under revision control * use HEAD instead of master for CI * Disabled flaky test (#13758) * change to compile time (#13835) * fix Makefile for rpkg (#13590) * fix Makefile for rpkg * update R and roxygen2 requirements * add roxygen requirement * add roxygen requirement * [CI] Prevent timeouts when rebuilding containers with docker. (#13818) * Prevent timeouts when rebuilding containers with docker. Increase timeout from 120 to 180 for pipelines * Increase docker cache timeout * Increase timeout also for docs * limit parallel builds to 10 * Code modification for testcases of various network models in directory example (#12498) * example testcase modified * rcnn file add * license add * license init * CI test trigger * rcnn modify give up * trigger * modify for better user experience * change the default parameter to xpu=None * Update bdk_demo.py * Update fcn_xs.py * Update test.py * Update train.py * Update bdk_demo.py * Update bdk_demo.py * modify review comments * refine * modify Readmes according to the changed code. * finetune READMEs * re-trigger ci * re-trigger ci twice * Add copyrights for third party licenses to license file (#13851) * Fix Tree Reduction on new instance type p3dn.24xlarge (#13852) * add fallback for gpu topology detection using CUDA 9.2 * add fallback for gpu topology detection using CUDA 9.2 * add log * update 3rdparty to master * add fallback for gpu topology detection using CUDA 9.2 * add log * update 3rdparty to master * bring 3rdparty packages to upstream/master * rebase to master * Update gpu_topology.h * [Clojure] package infer tweaks (#13864) * change object detection prediction to be a map * change predictions to a map for image-classifiers * change return types of the classifiers to be a map - add tests for base classifier and with-ndarray as well * tweak return types and inputs for predict - add test for plain predict * updated infer-classify examples * adjust the infer/object detections tests * tweak predictor test * Feedback from @kedarbellare review * put scaling back in * put back predict so it can handle multiple inputs * restore original functions signatures (remove first) * Modifying clojure CNN text classification example (#13865) * Modifying clojure CNN text classification example * Small fixes * Another minor fix * adding tolerance to flaky test (#13850) * adding tolerance * retrigger ci * retrigger ci * Julia v0.7/1.0 support and drop v0.6 support (#12845) * Fix cpp examples build on Mac. (#13826) This is a regression of addning @rpath name to libmxnet.so on Mac, example executable is not able to find libmxnet.so anymore. Add @rpath search path to fix this issue. * Fix launch bounds in spatial transformer (#13188) * Fix launch bounds in spatial transformer * Adding explanation in comment. * Update example scripts classpath. (#13849) * [MXNET-1177]Adding Scala Demo to be run as a part of Nightly CI (#13823) * Adding Scala Demo to be run as a part of Nightly CI * Addressed PR feedback : making a profile to fetch nightly jars only on CI * Changed name from scalacidemo to scala_ci_demo * Synchronized the scala-demo and java-demo for nightly CI runs * Pruned the maven command to simply maven install * changed running from ./.sh to bash .sh to be consistent * Add CODEOWNERS for Julia package (#13872) * fix ssd quantization script error (#13843) * fix ssd quantization script error * update readme for ssd * move quantized SSD instructions from quantization/README.md to ssd/README.md * update ssd readme and accuracy * update readme for SSD-vGG16 * Rename to avoid merge conflict with upstream. * Update submodule versions. - update mkldnn and mshadow to version used by upstream master - update ngraph-mxnet-bridge to current master Renames nGraph README to follow MXnet conventions. * Fix merge error for nGraph support in CMakeLists.txt * Fixes CMake file error. --- .gitignore | 6 +- .travis.yml | 5 +- 3rdparty/mkldnn | 2 +- 3rdparty/mshadow | 2 +- CMakeLists.txt | 43 +- CODEOWNERS | 33 +- CONTRIBUTORS.md | 20 + LICENSE | 552 ++-- MKLDNN_README.md | 14 +- MXNET_README.md | 1 + Makefile | 124 +- NEWS.md | 714 +++++- README.md => NGRAPH_README.md | 0 R-package/.Rbuildignore | 1 + R-package/DESCRIPTION | 14 +- R-package/LICENSE | 2 +- R-package/R/zzz.R | 4 +- R-package/dummy.NAMESPACE | 16 + ci/Jenkinsfile_docker_cache | 4 +- ci/Jenkinsfile_utils.groovy | 133 +- ci/README.md | 91 +- ci/build.py | 169 +- ci/build_windows.py | 3 - ci/docker/Dockerfile.build.android_armv7 | 7 +- ci/docker/Dockerfile.build.android_armv8 | 10 +- ci/docker/Dockerfile.build.armv6 | 7 +- ci/docker/Dockerfile.build.armv7 | 7 +- ci/docker/Dockerfile.build.armv8 | 11 +- ci/docker/Dockerfile.build.centos7_cpu | 2 + ci/docker/Dockerfile.build.jetson | 12 +- ci/docker/Dockerfile.build.test.arm_qemu | 6 +- ci/docker/Dockerfile.build.ubuntu_cpu | 3 + ci/docker/Dockerfile.publish.test.centos7_cpu | 38 + ci/docker/Dockerfile.publish.test.centos7_gpu | 38 + .../Dockerfile.publish.test.ubuntu1404_cpu | 39 + .../Dockerfile.publish.test.ubuntu1404_gpu | 40 + .../Dockerfile.publish.test.ubuntu1604_cpu | 39 + .../Dockerfile.publish.test.ubuntu1604_gpu | 39 + .../Dockerfile.publish.test.ubuntu1804_cpu | 44 +- .../Dockerfile.publish.test.ubuntu1804_gpu | 41 + ci/docker/Dockerfile.publish.ubuntu1404_cpu | 36 + ci/docker/Dockerfile.publish.ubuntu1404_gpu | 36 + ci/docker/install/centos7_adduser.sh | 5 + ci/docker/install/centos7_base.sh | 33 + ci/docker/install/centos7_scala.sh | 39 + ci/docker/install/ubuntu_adduser.sh | 5 + ci/docker/install/ubuntu_arm_qemu.sh | 5 +- ci/docker/install/ubuntu_base.sh | 36 + ci/docker/install/ubuntu_caffe.sh | 1 + ci/docker/install/ubuntu_clang.sh | 2 + ci/docker/install/ubuntu_core.sh | 11 +- ci/docker/install/ubuntu_docs.sh | 1 + ci/docker/install/ubuntu_emscripten.sh | 1 + ci/docker/install/ubuntu_gcc8.sh | 2 +- ci/docker/install/ubuntu_julia.sh | 28 +- ci/docker/install/ubuntu_llvm.sh | 3 +- ci/docker/install/ubuntu_mkl.sh | 31 + ci/docker/install/ubuntu_mklml.sh | 2 +- ci/docker/install/ubuntu_nightly_tests.sh | 2 +- ci/docker/install/ubuntu_npm_blc.sh | 2 +- ci/docker/install/ubuntu_nvidia.sh | 4 - ci/docker/install/ubuntu_onnx.sh | 1 + ci/docker/install/ubuntu_perl.sh | 1 + ci/docker/install/ubuntu_publish.sh | 65 + ci/docker/install/ubuntu_python.sh | 1 + ci/docker/install/ubuntu_r.sh | 2 +- ci/docker/install/ubuntu_rat.sh | 2 +- ci/docker/install/ubuntu_scala.sh | 38 +- ci/docker/install/ubuntu_tutorials.sh | 5 +- ci/docker/qemu/runtime_functions.py | 35 +- ci/docker/qemu/vmcontrol.py | 105 +- ci/docker/runtime_functions.sh | 405 ++- ci/docker_cache.py | 59 +- .../jenkins/Jenkins_steps.groovy | 1136 +++++---- ci/jenkins/Jenkinsfile_centos_cpu | 53 + ci/jenkins/Jenkinsfile_centos_gpu | 51 + ci/jenkins/Jenkinsfile_clang | 51 + ci/jenkins/Jenkinsfile_edge | 56 + ci/jenkins/Jenkinsfile_miscellaneous | 54 + ci/jenkins/Jenkinsfile_sanity | 48 + ci/jenkins/Jenkinsfile_unix_cpu | 74 + ci/jenkins/Jenkinsfile_unix_gpu | 73 + ci/jenkins/Jenkinsfile_website | 52 + ci/jenkins/Jenkinsfile_windows_cpu | 52 + ci/jenkins/Jenkinsfile_windows_gpu | 54 + ci/publish/Jenkinsfile | 107 + ci/publish/scala/build.sh | 29 + ci/publish/scala/buildkey.py | 165 ++ ci/publish/scala/deploy.sh | 41 + ci/publish/scala/fullDeploy.sh | 23 + ci/publish/scala/test.sh | 28 + ci/requirements.txt | 1 + ci/test_docker_cache.py | 4 +- ci/util.py | 2 +- ci/windows/test_py2_cpu.ps1 | 7 +- ci/windows/test_py2_gpu.ps1 | 11 +- ci/windows/test_py3_cpu.ps1 | 7 +- ci/windows/test_py3_gpu.ps1 | 11 +- cmake/AutoDetectF16C.cmake | 53 + cmake/DownloadMKLML.cmake | 74 + cmake/FirstClassLangCuda.cmake | 16 + cmake/MklDnn.cmake | 44 - cmake/cmake_options.yml | 53 + contrib/clojure-package/LICENSE | 2 +- contrib/clojure-package/README.md | 247 +- .../examples/captcha/.gitignore | 3 + .../examples/captcha/README.md | 61 + .../examples/captcha/captcha_example.png | Bin 0 -> 9762 bytes .../examples/captcha/gen_captcha.py | 40 + .../examples/captcha/get_data.sh | 32 + .../examples/captcha/project.clj | 28 + .../examples/captcha/src/captcha/consts.clj | 27 + .../captcha/src/captcha/infer_ocr.clj | 56 + .../captcha/src/captcha/train_ocr.clj | 156 ++ .../captcha/test/captcha/train_ocr_test.clj | 119 + .../cnn-text-classification/README.md | 38 +- .../cnn-text-classification/project.clj | 2 +- .../cnn_text_classification/classifier.clj | 53 +- .../cnn_text_classification/data_helper.clj | 195 +- .../classifier_test.clj | 48 + .../clojure-package/examples/gan/project.clj | 5 +- .../examples/gan/src/gan/gan_mnist.clj | 6 +- .../examples/gan/src/gan/viz.clj | 4 +- .../examples/gan/test/gan/gan_test.clj | 25 + .../examples/imclassification/project.clj | 2 +- .../src/imclassification/train_mnist.clj | 20 +- .../imclassification/train_mnist_test.clj | 39 + .../test/test-symbol.json.ref | 105 + .../examples/infer/imageclassifier/.gitignore | 12 + .../examples/infer/imageclassifier/README.md | 24 + .../infer/imageclassifier/project.clj | 25 + .../scripts/get_resnet_18_data.sh | 45 + .../scripts/get_resnet_data.sh | 44 + .../src/infer/imageclassifier_example.clj | 111 + .../infer/imageclassifier_example_test.clj | 58 + .../examples/infer/objectdetector/.gitignore | 12 + .../examples/infer/objectdetector/README.md | 24 + .../examples/infer/objectdetector/project.clj | 25 + .../objectdetector/scripts/get_ssd_data.sh | 49 + .../src/infer/objectdetector_example.clj | 120 + .../infer/objectdetector_example_test.clj | 65 + .../examples/infer/predictor/.gitignore | 12 + .../examples/infer/predictor/README.md | 24 + .../examples/infer/predictor/project.clj | 25 + .../predictor/scripts/get_resnet_18_data.sh | 44 + .../predictor/scripts/get_resnet_data.sh | 44 + .../predictor/src/infer/predictor_example.clj | 101 + .../test/infer/predictor_example_test.clj | 51 + .../examples/module/project.clj | 2 +- .../examples/module/test/mnist_mlp_test.clj | 29 + .../examples/multi-label/project.clj | 2 +- .../multi-label/test/multi_label_test.clj | 26 + .../examples/neural-style/project.clj | 2 +- .../neural-style/src/neural_style/core.clj | 24 +- .../test/neural_style/vgg_19_test.clj | 53 + .../examples/pre-trained-models/project.clj | 2 +- .../examples/profiler/project.clj | 2 +- .../examples/profiler/src/profiler/core.clj | 6 +- .../examples/profiler/test/core_test.clj | 31 + .../test/profile-matmul-20iter.json.ref | 271 ++ .../clojure-package/examples/rnn/project.clj | 2 +- .../examples/rnn/src/rnn/test_char_rnn.clj | 4 + .../examples/rnn/src/rnn/train_char_rnn.clj | 4 + .../examples/rnn/test/rnn/core_test.clj | 26 + .../examples/tutorial/.gitignore | 1 + .../examples/tutorial/project.clj | 7 +- .../tutorial/src/tutorial/kvstore.clj | 60 +- .../examples/tutorial/src/tutorial/module.clj | 169 +- .../tutorial/src/tutorial/ndarray.clj | 75 +- .../examples/tutorial/src/tutorial/symbol.clj | 138 +- .../tutorial/test/tutorial/core_test.clj | 27 + .../examples/visualization/project.clj | 2 +- .../test/visualization/core_test.clj | 28 + contrib/clojure-package/integration-tests.sh | 28 + contrib/clojure-package/project.clj | 7 +- .../scripts/infer/get_resnet_18_data.sh | 38 + .../scripts/infer/get_ssd_data.sh | 39 + .../scripts/update_versions.sh | 18 +- contrib/clojure-package/src/dev/generator.clj | 13 +- .../src/org/apache/clojure_mxnet/image.clj | 139 ++ .../src/org/apache/clojure_mxnet/infer.clj | 372 +++ .../src/org/apache/clojure_mxnet/ndarray.clj | 3 +- .../org/apache/clojure_mxnet/optimizer.clj | 52 +- .../org/apache/clojure_mxnet/primitives.clj | 46 + .../src/org/apache/clojure_mxnet/random.clj | 30 +- .../src/org/apache/clojure_mxnet/symbol.clj | 2 +- .../src/org/apache/clojure_mxnet/util.clj | 16 +- .../test/good-test-ndarray.clj | 10 +- .../clojure-package/test/good-test-symbol.clj | 10 +- .../org/apache/clojure_mxnet/image_test.clj | 79 + .../infer/imageclassifier_test.clj | 128 + .../infer/objectdetector_test.clj | 82 + .../clojure_mxnet/infer/predictor_test.clj | 77 + .../org/apache/clojure_mxnet/ndarray_test.clj | 2 +- .../apache/clojure_mxnet/operator_test.clj | 2 +- .../apache/clojure_mxnet/optimizer_test.clj | 10 + .../apache/clojure_mxnet/primitives_test.clj | 45 + .../org/apache/clojure_mxnet/random_test.clj | 17 +- .../org/apache/clojure_mxnet/util_test.clj | 20 + .../test/test-images/Pug-Cookie.jpg | Bin 0 -> 104323 bytes .../test/test-images/kitten.jpg | Bin 0 -> 110969 bytes cpp-package/CMakeLists.txt | 2 + cpp-package/example/Makefile | 17 +- cpp-package/example/README.md | 39 +- cpp-package/example/alexnet.cpp | 12 +- cpp-package/example/charRNN.cpp | 2 +- cpp-package/example/example.mk | 6 +- cpp-package/example/feature_extract/Makefile | 4 +- cpp-package/example/googlenet.cpp | 2 +- cpp-package/example/inception_bn.cpp | 26 +- cpp-package/example/inference/Makefile | 40 + cpp-package/example/inference/README.md | 41 + .../example/inference/inception_inference.cpp | 446 ++++ .../unit_test_inception_inference.sh | 43 + cpp-package/example/lenet_with_mxdataiter.cpp | 2 +- cpp-package/example/mlp.cpp | 10 +- cpp-package/example/mlp_csv.cpp | 2 +- cpp-package/example/resnet.cpp | 19 +- cpp-package/example/test_optimizer.cpp | 4 + cpp-package/example/test_score.cpp | 5 + cpp-package/example/utils.h | 2 +- cpp-package/include/mxnet-cpp/operator.h | 4 +- cpp-package/include/mxnet-cpp/operator.hpp | 4 +- cpp-package/include/mxnet-cpp/symbol.hpp | 4 +- cpp-package/scripts/OpWrapperGenerator.py | 6 +- cpp-package/tests/ci_test.sh | 5 + dev_menu.py | 239 ++ docs/Doxyfile | 4 +- docs/Jenkinsfile | 6 +- docs/Jenkinsfile-dev | 4 +- docs/README.md | 6 +- docs/_static/js/auto_module_index.js | 36 +- docs/_static/js/clipboard.js | 778 ++++++ docs/_static/js/copycode.js | 21 +- docs/_static/js/docversion.js | 22 +- docs/_static/js/navbar.js | 24 +- docs/_static/js/options.js | 22 +- docs/_static/js/page.js | 19 + docs/_static/js/search.js | 24 +- docs/_static/js/sidebar.js | 23 +- docs/_static/mxnet-theme/index.html | 6 +- docs/_static/mxnet-theme/navbar.html | 2 + docs/_static/selectlang.js | 21 +- docs/api/c++/index.md | 55 +- docs/api/clojure/ndarray.md | 3 +- docs/api/clojure/symbol.md | 18 - docs/api/index.md | 2 + docs/api/java/index.md | 15 + docs/api/python/gluon/loss.md | 1 + docs/api/python/gluon/model_zoo.md | 2 + docs/api/python/image/image.md | 8 +- docs/api/python/io/io.md | 39 +- docs/api/python/module/module.md | 2 +- docs/api/python/ndarray/contrib.md | 11 + docs/api/python/ndarray/ndarray.md | 28 +- docs/api/python/ndarray/random.md | 7 +- docs/api/python/ndarray/sparse.md | 3 +- docs/api/python/symbol/contrib.md | 8 + docs/api/python/symbol/random.md | 6 +- docs/api/python/symbol/symbol.md | 26 +- docs/api/scala/index.md | 18 +- docs/api/scala/kvstore.md | 98 +- docs/api/scala/ndarray.md | 186 +- docs/api/scala/symbol.md | 66 +- docs/architecture/note_memory.md | 15 +- docs/architecture/overview.md | 5 +- docs/build_version_doc/artifacts/.htaccess | 41 +- docs/build_version_doc/build_all_version.sh | 9 +- docs/community/contribute.md | 25 +- docs/community/ecosystem.md | 2 +- docs/community/mxnet_channels.md | 6 +- docs/conf.py | 3 +- docs/faq/add_op_in_backend.md | 24 +- docs/faq/develop_and_hack.md | 4 +- docs/faq/env_var.md | 50 +- docs/faq/index.md | 44 +- docs/gluon/index.md | 11 +- docs/install/build_from_source.md | 249 +- docs/install/c_plus_plus.md | 3 +- docs/install/centos_setup.md | 62 +- docs/install/download.md | 3 +- docs/install/index.md | 252 +- docs/install/java_setup.md | 123 + docs/install/osx_setup.md | 11 +- docs/install/scala_setup.md | 4 +- docs/install/ubuntu_setup.md | 58 +- docs/install/windows_setup.md | 2 +- docs/model_zoo/index.md | 2 +- docs/mxdoc.py | 53 +- docs/settings.ini | 39 + .../vision}/cnn_visualization/gradcam.py | 4 +- docs/tutorials/basic/module.md | 148 +- docs/tutorials/basic/reshape_transpose.md | 197 ++ docs/tutorials/basic/symbol.md | 8 +- docs/tutorials/c++/basics.md | 12 +- .../control_flow/ControlFlowTutorial.md | 12 +- docs/tutorials/gluon/hybrid.md | 110 +- docs/tutorials/gluon/info_gan.md | 437 ++++ docs/tutorials/gluon/learning_rate_finder.md | 28 +- .../gluon/learning_rate_schedules.md | 4 +- docs/tutorials/gluon/mnist.md | 554 +++-- docs/tutorials/gluon/save_load_params.md | 2 +- docs/tutorials/index.md | 11 + docs/tutorials/java/index.md | 8 + docs/tutorials/java/mxnet_java_on_intellij.md | 180 ++ docs/tutorials/java/ssd_inference.md | 186 ++ docs/tutorials/onnx/export_mxnet_to_onnx.md | 2 +- docs/tutorials/onnx/fine_tuning_gluon.md | 45 +- docs/tutorials/python/linear-regression.md | 2 +- docs/tutorials/python/predict_image.md | 1 + docs/tutorials/python/profiler.md | 8 +- docs/tutorials/r/fiveMinutesNeuralNetwork.md | 4 +- docs/tutorials/scala/char_lstm.md | 2 +- .../scala/mxnet_scala_on_intellij.md | 64 +- docs/tutorials/sparse/row_sparse.md | 2 +- docs/tutorials/unsupervised_learning/gan.md | 9 +- docs/tutorials/vision/cnn_visualization.md | 21 +- example/MXNetTutorialTemplate.ipynb | 6 +- example/adversary/README.md | 4 +- example/adversary/adversary_generation.ipynb | 343 +-- .../variational_autoencoder}/README.md | 0 .../variational_autoencoder}/VAE.py | 0 .../variational_autoencoder/VAE_example.ipynb | 1204 +++++++++ example/bayesian-methods/README.md | 24 + example/bayesian-methods/bdk_demo.py | 73 +- example/bi-lstm-sort/README.md | 28 +- example/bi-lstm-sort/bi-lstm-sort.ipynb | 607 +++++ example/bi-lstm-sort/infer_sort.py | 80 - example/bi-lstm-sort/lstm.py | 175 -- example/bi-lstm-sort/lstm_sort.py | 142 -- example/bi-lstm-sort/rnn_model.py | 73 - example/bi-lstm-sort/sort_io.py | 255 -- example/capsnet/README.md | 132 +- example/capsnet/capsulenet.py | 695 +++--- example/cnn_visualization/README.md | 17 - example/cnn_visualization/gradcam_demo.py | 110 - example/cnn_visualization/vgg.py | 84 - example/deep-embedded-clustering/README.md | 11 +- example/deep-embedded-clustering/data.py | 15 +- example/deep-embedded-clustering/dec.py | 25 +- example/fcn-xs/README.md | 27 +- example/fcn-xs/fcn_xs.py | 4 +- .../gluon/{ => actor_critic}/actor_critic.py | 0 example/gluon/audio/transforms.py | 205 ++ example/gluon/audio/urban_sounds/README.md | 100 + example/gluon/audio/urban_sounds/datasets.py | 179 ++ example/gluon/audio/urban_sounds/model.py | 33 + example/gluon/audio/urban_sounds/predict.py | 92 + .../gluon/audio/urban_sounds/requirements.txt | 2 + example/gluon/audio/urban_sounds/train.py | 157 ++ example/gluon/{DCGAN => dc_gan}/README.md | 0 example/gluon/{DCGAN => dc_gan}/__init__.py | 0 example/gluon/{DCGAN => dc_gan}/dcgan.py | 0 .../{DCGAN => dc_gan}/inception_score.py | 0 .../kaggle_k_fold_cross_validation.py | 0 example/gluon/learning_rate_manipulation.py | 63 - example/gluon/{ => lstm_crf}/lstm_crf.py | 10 +- example/gluon/{ => mnist}/mnist.py | 0 example/gluon/sn_gan/data.py | 2 +- example/gluon/sn_gan/model.py | 2 +- example/gluon/sn_gan/train.py | 2 +- example/gluon/sn_gan/utils.py | 2 +- .../super_resolution.py | 0 example/gluon/tree_lstm/README.md | 29 + example/gluon/tree_lstm/dataset.py | 4 +- .../gluon/tree_lstm/fetch_and_preprocess.sh | 4 +- example/gluon/tree_lstm/main.py | 1 - example/gluon/tree_lstm/scripts/download.py | 1 - example/image-classification/train_mnist.py | 1 + example/memcost/Makefile | 38 - example/memcost/README.md | 30 - example/memcost/inception_memcost.py | 107 - example/{ => module}/utils/__init__.py | 0 example/{ => module}/utils/get_data.py | 6 +- example/multi-task/README.md | 13 +- example/multi-task/example_multi_task.py | 159 -- example/multi-task/multi-task-learning.ipynb | 454 ++++ example/multivariate_time_series/README.md | 4 +- example/named_entity_recognition/README.md | 11 +- .../named_entity_recognition/src/metrics.py | 2 +- example/named_entity_recognition/src/ner.py | 2 +- example/nce-loss/README.md | 2 +- example/notebooks/README.md | 4 - example/numpy-ops/numpy_softmax.py | 84 - example/onnx/super_resolution.py | 86 - example/python-howto/README.md | 37 - example/python-howto/data_iter.py | 76 - example/python-howto/monitor_weights.py | 46 - example/python-howto/multiple_outputs.py | 38 - example/quantization/README.md | 43 +- .../quantization/imagenet_gen_qsym_mkldnn.py | 13 +- example/quantization/imagenet_inference.py | 7 +- example/rcnn/README.md | 3 +- example/rcnn/demo.py | 2 +- example/rcnn/test.py | 4 +- example/rcnn/train.py | 4 +- example/recommenders/README.md | 38 +- example/recommenders/crossentropy.py | 141 -- example/recommenders/demo1-MF.ipynb | 1694 ++++++++++++- example/recommenders/demo1-MF2-fancy.ipynb | 295 --- example/recommenders/demo2-binary.ipynb | 137 - example/recommenders/demo2-dssm.ipynb | 2210 +++++++++++++++++ example/recommenders/demo3-dssm.ipynb | 113 - example/recommenders/matrix_fact.py | 85 +- example/recommenders/movielens_data.py | 12 +- example/recommenders/negativesample.py | 180 -- example/recommenders/randomproj.py | 150 -- example/recommenders/recotools.py | 45 - example/recommenders/symbol_alexnet.py | 62 - example/reinforcement-learning/ddpg/README.md | 2 + example/reinforcement-learning/dqn/setup.sh | 6 +- .../restricted-boltzmann-machine/README.md | 52 + example/rnn-time-major/bucket_io.py | 264 -- .../rnn-time-major/get_sherlockholmes_data.sh | 43 - example/rnn-time-major/readme.md | 24 - example/rnn-time-major/rnn_cell_demo.py | 189 -- example/rnn/README.md | 5 + example/rnn/bucketing/README.md | 13 +- example/rnn/large_word_lm/data.py | 2 +- example/rnn/large_word_lm/readme.md | 66 - example/rnn/word_lm/README.md | 2 +- example/sparse/linear_classification/data.py | 5 +- example/speech_recognition/README.md | 4 +- example/speech_recognition/label_util.py | 2 +- example/speech_recognition/log_util.py | 74 +- example/speech_recognition/main.py | 6 +- example/speech_recognition/singleton.py | 32 +- .../speech_recognition/stt_datagenerator.py | 6 +- example/speech_recognition/stt_io_iter.py | 6 +- example/speech_recognition/stt_metric.py | 2 +- example/speech_recognition/stt_utils.py | 6 +- example/speech_recognition/train.py | 2 +- example/ssd/README.md | 47 + example/ssd/quantization.py | 11 +- example/stochastic-depth/sd_cifar10.py | 31 +- example/stochastic-depth/sd_mnist.py | 6 +- example/svm_mnist/svm_mnist.py | 110 +- example/svrg_module/README.md | 6 +- .../benchmarks/svrg_benchmark.ipynb | 111 +- .../linear_regression/data_reader.py | 24 +- .../svrg_module/linear_regression/train.py | 4 +- .../README.md | 0 .../convert_data.py | 0 .../vaegan_mxnet.py | 0 example/vae/VAE_example.ipynb | 1204 --------- include/dlpack | 1 + include/dmlc | 1 + include/mshadow | 1 + include/mxnet/base.h | 18 +- include/mxnet/c_api.h | 10 + include/mxnet/c_api_error.h | 56 + include/mxnet/engine.h | 1 - include/mxnet/ndarray.h | 36 +- include/mxnet/random_generator.h | 16 +- include/nnvm | 1 + julia/.gitignore | 3 - julia/README.md | 4 +- julia/REQUIRE | 1 - julia/deps/build.jl | 74 +- julia/docs/.gitignore | 6 + julia/docs/Makefile | 13 +- julia/docs/Project.toml | 7 + julia/docs/make.jl | 16 +- julia/docs/mkdocs.yml | 2 +- julia/docs/src/tutorial/mnist.md | 2 +- julia/examples/char-lstm/lstm.jl | 11 +- julia/examples/char-lstm/seq-data.jl | 13 +- julia/examples/char-lstm/train.jl | 2 +- julia/examples/char-lstm/visualize.jl | 4 +- .../Prediction with Pre-trained Model.ipynb | 2 +- julia/examples/mnist/mlp-test.jl | 26 +- julia/examples/mnist/mlp.jl | 2 +- julia/src/MXNet.jl | 20 +- julia/src/autograd.jl | 18 +- julia/src/base.jl | 66 +- julia/src/broadcast.jl | 29 +- julia/src/callback.jl | 2 +- julia/src/context.jl | 21 +- julia/src/deprecated.jl | 61 +- julia/src/executor.jl | 20 +- julia/src/initializer.jl | 8 +- julia/src/io.jl | 47 +- julia/src/kvstore.jl | 14 +- julia/src/metric.jl | 60 +- julia/src/model.jl | 72 +- julia/src/ndarray.jl | 341 ++- julia/src/nn-factory.jl | 2 +- julia/src/optimizer.jl | 10 +- julia/src/optimizers/adadelta.jl | 2 +- julia/src/optimizers/adagrad.jl | 2 +- julia/src/optimizers/nadam.jl | 2 +- julia/src/optimizers/rmsprop.jl | 2 +- julia/src/optimizers/sgd.jl | 4 +- julia/src/random.jl | 9 +- julia/src/symbolic-node.jl | 242 +- julia/src/util.jl | 34 +- julia/src/visualize.jl | 8 +- julia/test/runtests.jl | 18 +- julia/test/unittest/autograd.jl | 79 +- julia/test/unittest/bind.jl | 6 +- julia/test/unittest/io.jl | 8 +- julia/test/unittest/kvstore.jl | 14 +- julia/test/unittest/metric.jl | 6 +- julia/test/unittest/model.jl | 6 +- julia/test/unittest/name.jl | 6 +- julia/test/unittest/ndarray.jl | 463 ++-- julia/test/unittest/operator.jl | 6 +- julia/test/unittest/optimizer.jl | 10 +- julia/test/unittest/random.jl | 15 +- julia/test/unittest/symbolic-node.jl | 120 +- julia/test/unittest/util.jl | 9 +- julia/test/unittest/visualize.jl | 4 +- make/config.mk | 6 +- make/config/libmxnet.sym | 15 + make/config/libmxnet.ver | 19 + make/crosscompile.jetson.mk | 4 +- make/maven/maven_darwin_cpu.mk | 182 ++ make/maven/maven_linux_cpu.mk | 182 ++ make/maven/maven_linux_cu90.mk | 185 ++ make/pip/pip_darwin_cpu.mk | 182 ++ make/pip/pip_darwin_mkl.mk | 166 ++ make/pip/pip_linux_cpu.mk | 182 ++ make/pip/pip_linux_cu100.mk | 185 ++ make/pip/pip_linux_cu100mkl.mk | 169 ++ make/pip/pip_linux_cu75.mk | 182 ++ make/pip/pip_linux_cu75mkl.mk | 166 ++ make/pip/pip_linux_cu80.mk | 185 ++ make/pip/pip_linux_cu80mkl.mk | 169 ++ make/pip/pip_linux_cu90.mk | 185 ++ make/pip/pip_linux_cu90mkl.mk | 169 ++ make/pip/pip_linux_cu91.mk | 185 ++ make/pip/pip_linux_cu91mkl.mk | 169 ++ make/pip/pip_linux_cu92.mk | 185 ++ make/pip/pip_linux_cu92mkl.mk | 169 ++ make/pip/pip_linux_mkl.mk | 166 ++ perl-package/AI-MXNet-Gluon-Contrib/META.yml | 17 + perl-package/AI-MXNet-Gluon-ModelZoo/META.yml | 17 + perl-package/AI-MXNet/META.yml | 17 + perl-package/AI-MXNetCAPI/META.yml | 17 + perl-package/AI-MXNetCAPI/mxnet.i | 10 + perl-package/AI-NNVMCAPI/META.yml | 17 + python/mxnet/__init__.py | 9 +- python/mxnet/autograd.py | 6 +- python/mxnet/base.py | 12 +- python/mxnet/callback.py | 4 +- python/mxnet/context.py | 24 + python/mxnet/contrib/io.py | 4 +- .../contrib/onnx/mx2onnx/_op_translations.py | 1544 +++++------- .../contrib/onnx/mx2onnx/export_model.py | 5 + .../mxnet/contrib/onnx/mx2onnx/export_onnx.py | 135 +- .../contrib/onnx/onnx2mx/_import_helper.py | 14 +- .../contrib/onnx/onnx2mx/_op_translations.py | 79 +- .../onnx/onnx2mx/_translation_utils.py | 6 +- .../contrib/onnx/onnx2mx/import_model.py | 19 +- .../mxnet/contrib/onnx/onnx2mx/import_onnx.py | 9 +- .../contrib/onnx/onnx2mx/import_to_gluon.py | 5 + python/mxnet/contrib/quantization.py | 6 +- .../contrib/svrg_optimization/svrg_module.py | 17 +- python/mxnet/contrib/text/embedding.py | 84 +- python/mxnet/contrib/text/vocab.py | 12 +- python/mxnet/gluon/block.py | 6 +- python/mxnet/gluon/contrib/nn/basic_layers.py | 4 +- .../mxnet/gluon/contrib/rnn/conv_rnn_cell.py | 18 +- python/mxnet/gluon/contrib/rnn/rnn_cell.py | 13 +- python/mxnet/gluon/data/dataloader.py | 275 +- python/mxnet/gluon/data/dataset.py | 16 +- python/mxnet/gluon/data/vision/datasets.py | 41 +- python/mxnet/gluon/data/vision/transforms.py | 4 +- python/mxnet/gluon/loss.py | 72 +- .../mxnet/gluon/model_zoo/vision/mobilenet.py | 14 +- python/mxnet/gluon/nn/basic_layers.py | 19 +- python/mxnet/gluon/parameter.py | 10 +- python/mxnet/gluon/rnn/rnn_cell.py | 43 +- python/mxnet/gluon/rnn/rnn_layer.py | 130 +- python/mxnet/gluon/trainer.py | 130 +- python/mxnet/image/detection.py | 66 +- python/mxnet/image/image.py | 140 +- python/mxnet/initializer.py | 9 +- python/mxnet/io/io.py | 8 +- python/mxnet/libinfo.py | 35 +- python/mxnet/metric.py | 305 ++- python/mxnet/model.py | 5 + python/mxnet/module/base_module.py | 13 +- python/mxnet/module/sequential_module.py | 15 +- python/mxnet/ndarray/contrib.py | 108 +- python/mxnet/ndarray/ndarray.py | 77 +- python/mxnet/ndarray/random.py | 136 +- python/mxnet/ndarray/sparse.py | 92 +- python/mxnet/optimizer/optimizer.py | 106 +- python/mxnet/recordio.py | 49 +- python/mxnet/rnn/rnn.py | 6 +- python/mxnet/rnn/rnn_cell.py | 23 +- python/mxnet/symbol/contrib.py | 18 +- python/mxnet/symbol/random.py | 82 +- python/mxnet/symbol/symbol.py | 2 +- python/mxnet/symbol_doc.py | 13 - python/mxnet/test_utils.py | 83 +- python/mxnet/torch.py | 2 +- python/mxnet/visualization.py | 37 +- python/setup.py | 2 +- readthedocs.yml | 17 + scala-package/.gitignore | 5 + scala-package/README.md | 281 ++- .../assembly/linux-x86_64-cpu/pom.xml | 127 - .../src/main/assembly/assembly.xml | 28 - .../assembly/linux-x86_64-gpu/pom.xml | 127 - .../src/main/assembly/assembly.xml | 28 - .../osx-x86_64-cpu/main/assembly/assembly.xml | 30 - scala-package/assembly/osx-x86_64-cpu/pom.xml | 123 - .../src/main/assembly/assembly.xml | 28 - scala-package/assembly/pom.xml | 157 +- scala-package/assembly/src/javadoc.xml | 24 - .../assembly/src/main/assembly/assembly.xml | 57 + .../assembly/src/main/assembly/javadoc.xml | 15 + .../assembly/src/main/assembly/source.xml | 19 + scala-package/assembly/src/source.xml | 32 - scala-package/core/pom.xml | 142 +- .../main/scala/org/apache/mxnet/Base.scala | 58 +- .../main/scala/org/apache/mxnet/Context.scala | 2 + .../scala/org/apache/mxnet/Executor.scala | 9 +- .../src/main/scala/org/apache/mxnet/IO.scala | 7 +- .../main/scala/org/apache/mxnet/Image.scala | 185 ++ .../main/scala/org/apache/mxnet/KVStore.scala | 2 +- .../main/scala/org/apache/mxnet/LibInfo.scala | 3 + .../org/apache/mxnet/MX_PRIMITIVES.scala | 85 + .../main/scala/org/apache/mxnet/NDArray.scala | 232 +- .../scala/org/apache/mxnet/NDArrayAPI.scala | 13 +- .../scala/org/apache/mxnet/Optimizer.scala | 2 +- .../org/apache/mxnet/ResourceScope.scala | 6 +- .../main/scala/org/apache/mxnet/Symbol.scala | 2 + .../scala/org/apache/mxnet/SymbolAPI.scala | 12 +- .../org/apache/mxnet/Visualization.scala | 1 + .../org/apache/mxnet/io/MXDataIter.scala | 10 +- .../org/apache/mxnet/io/NDArrayIter.scala | 11 +- .../org/apache/mxnet/io/PrefetchingIter.scala | 4 +- .../org/apache/mxnet/io/ResizeIter.scala | 4 +- .../org/apache/mxnet/javaapi/Context.scala | 16 +- .../scala/org/apache/mxnet/javaapi/IO.scala | 11 +- .../org/apache/mxnet/javaapi/NDArray.scala | 463 ++++ .../org/apache/mxnet/javaapi/Shape.scala | 3 +- .../org/apache/mxnet/module/BaseModule.scala | 6 +- .../apache/mxnet/module/BucketingModule.scala | 4 +- .../module/DataParallelExecutorGroup.scala | 4 +- .../org/apache/mxnet/module/Module.scala | 4 +- .../mxnet/module/SequentialModule.scala | 4 +- .../mxnet/util/NativeLibraryLoader.scala | 82 +- .../apache/mxnet/util/OptionConversion.scala | 22 + .../org/apache/mxnet/javaapi/NDArrayTest.java | 100 + .../mxnet/javaapi/ResourceScopeTestSuite.java | 110 + .../test/scala/org/apache/mxnet/IOSuite.scala | 27 + .../scala/org/apache/mxnet/ImageSuite.scala | 100 + .../scala/org/apache/mxnet/NDArraySuite.scala | 422 +++- .../scala/org/apache/mxnet/SymbolSuite.scala | 22 + scala-package/deploy/pom.xml | 140 ++ .../deploy/src/main/deploy/deploy.xml | 33 + scala-package/dev/compile-mxnet-backend.sh | 7 +- scala-package/examples/pom.xml | 79 +- .../benchmark/run_image_inference_bm.sh | 17 +- .../benchmark/run_java_inference_bm.sh | 27 + .../scripts/benchmark/run_text_charrnn_bm.sh | 18 +- .../examples/scripts/customop/run_customop.sh | 2 +- .../scripts/customop/run_customopwithrtc.sh | 2 +- .../imageclassifier/get_resnet_18_data.sh | 8 +- .../infer/imageclassifier/get_resnet_data.sh | 8 +- .../imageclassifier/run_classifier_example.sh | 2 +- .../infer/objectdetector/get_ssd_data.sh | 13 +- .../infer/objectdetector/run_ssd_example.sh | 3 +- .../objectdetector/run_ssd_java_example.sh | 34 + .../predictor/run_predictor_java_example.sh | 31 + .../examples/scripts/module/mnist_mlp.sh | 2 +- .../scripts/module/run_sequential_module.sh | 2 +- .../neuralstyle_end2end/run_test_end2end.sh | 2 +- .../neuralstyle_end2end/run_train_end2end.sh | 2 +- .../scripts/profiler/run_profiler_matmul.sh | 2 +- .../scripts/profiler/run_profiler_ndarray.sh | 2 +- .../scripts/rnn/run_lstm_bucketing.sh | 7 +- .../examples/scripts/rnn/run_test_charrnn.sh | 9 +- .../examples/scripts/rnn/run_train_charrnn.sh | 10 +- .../scripts/run_cnntextclassification.sh | 2 +- .../examples/scripts/run_gan_mnist.sh | 2 +- .../examples/scripts/run_multitask.sh | 2 +- .../examples/scripts/run_neuralstyle.sh | 2 +- .../examples/scripts/run_train_mnist.sh | 10 +- .../examples/scripts/run_visualization.sh | 2 +- .../javaapi/benchmark/InferBase.java | 35 + .../javaapi/benchmark/JavaBenchmark.java | 129 + .../benchmark/ObjectDetectionBenchmark.java | 64 + .../javaapi/infer/objectdetector/README.md | 97 + .../objectdetector/SSDClassifierExample.java | 199 ++ .../infer/predictor/PredictorExample.java | 200 ++ .../javaapi/infer/predictor/README.md | 61 + .../imclassification/TrainModel.scala | 24 +- .../datasets/SyntheticDataIter.scala | 8 +- .../imclassification/models/Lenet.scala | 5 +- .../models/MultiLayerPerceptron.scala | 5 +- .../imclassification/models/Resnet.scala | 16 +- .../infer/objectdetector/README.md | 20 +- .../objectdetector/SSDClassifierExample.scala | 4 +- .../infer/predictor/PredictorExample.scala | 92 + .../CNNClassifierExampleSuite.scala | 3 +- .../IMClassificationExampleSuite.scala | 10 +- .../ImageClassifierExampleSuite.scala | 6 +- .../ObjectDetectorExampleSuite.scala | 5 +- .../predictor/PredictorExampleSuite.scala | 87 + .../multitask/MultiTaskSuite.scala | 3 +- .../profiler/ProfilerSuite.scala | 3 +- scala-package/infer/pom.xml | 165 +- .../org/apache/mxnet/infer/Classifier.scala | 54 +- .../apache/mxnet/infer/ImageClassifier.scala | 48 +- .../apache/mxnet/infer/ObjectDetector.scala | 31 +- .../org/apache/mxnet/infer/Predictor.scala | 77 +- .../mxnet/infer/javaapi/ObjectDetector.scala | 129 + .../infer/javaapi/ObjectDetectorOutput.scala | 71 + .../mxnet/infer/javaapi/Predictor.scala | 134 + .../javaapi/ObjectDetectorOutputTest.java | 59 + .../infer/javaapi/ObjectDetectorTest.java | 106 + .../mxnet/infer/javaapi/PredictorTest.java | 100 + .../apache/mxnet/infer/ClassifierSuite.scala | 47 +- .../mxnet/infer/ImageClassifierSuite.scala | 7 + .../apache/mxnet/infer/PredictorSuite.scala | 32 +- .../init-native/linux-x86_64/pom.xml | 106 - scala-package/init-native/osx-x86_64/pom.xml | 105 - scala-package/init-native/pom.xml | 174 +- .../org_apache_mxnet_init_native_c_api.h | 45 + scala-package/init/pom.xml | 107 +- .../scala/org/apache/mxnet/init/Base.scala | 20 +- scala-package/macros/pom.xml | 88 +- .../org/apache/mxnet/APIDocGenerator.scala | 346 ++- .../org/apache/mxnet/GeneratorBase.scala | 232 ++ .../scala/org/apache/mxnet/NDArrayMacro.scala | 352 ++- .../scala/org/apache/mxnet/SymbolMacro.scala | 325 ++- .../mxnet/javaapi/JavaNDArrayMacro.scala | 125 + .../apache/mxnet/utils/CToScalaUtils.scala | 39 +- .../scala/org/apache/mxnet/MacrosSuite.scala | 3 +- scala-package/memory-management.md | 118 + scala-package/mxnet-demo/java-demo/Makefile | 52 + scala-package/mxnet-demo/java-demo/README.md | 90 + .../mxnet-demo/java-demo/bin/java_sample.sh | 8 +- .../mxnet-demo/java-demo/bin/run_od.sh | 8 +- scala-package/mxnet-demo/java-demo/pom.xml | 71 + .../src/main/java/mxnet/HelloWorld.java | 32 + .../src/main/java/mxnet/ObjectDetection.java | 101 + .../mxnet-demo/{ => scala-demo}/Makefile | 8 +- .../mxnet-demo/{ => scala-demo}/README.md | 15 +- .../mxnet-demo/{ => scala-demo}/bin/demo.sh | 0 .../mxnet-demo/{ => scala-demo}/bin/run_im.sh | 0 .../mxnet-demo/{ => scala-demo}/pom.xml | 21 +- .../src/main/scala/sample/HelloWorld.scala | 0 .../sample/ImageClassificationExample.scala | 0 scala-package/native/README.md | 67 + scala-package/native/linux-x86_64-cpu/pom.xml | 106 - scala-package/native/linux-x86_64-gpu/pom.xml | 106 - scala-package/native/osx-x86_64-cpu/pom.xml | 106 - scala-package/native/pom.xml | 169 +- .../native/org_apache_mxnet_native_c_api.cc | 10 + .../native/org_apache_mxnet_native_c_api.h | 861 +++++++ scala-package/packageTest/Makefile | 91 + scala-package/packageTest/README.md | 72 + scala-package/packageTest/core/pom.xml | 39 + scala-package/packageTest/core/scripts | 1 + scala-package/packageTest/examples/pom.xml | 48 + scala-package/packageTest/examples/scripts | 1 + scala-package/packageTest/infer/pom.xml | 38 + scala-package/packageTest/pom.xml | 196 ++ scala-package/pom.xml | 204 +- scala-package/spark/README.md | 3 +- scala-package/spark/bin/run-mnist-example.sh | 4 +- scala-package/spark/pom.xml | 33 +- snapcraft.yaml | 19 +- src/c_api/c_api.cc | 11 + src/c_api/c_api_common.h | 42 +- src/c_api/c_api_executor.cc | 12 +- src/c_api/c_api_symbolic.cc | 8 +- src/c_api/c_predict_api.cc | 14 +- src/common/cuda_utils.h | 19 +- src/common/exec_utils.h | 8 +- src/common/utils.h | 21 + src/engine/engine.cc | 2 +- src/engine/stream_manager.h | 10 +- src/engine/threaded_engine.h | 2 +- src/engine/threaded_engine_pooled.cc | 8 +- src/executor/attach_op_execs_pass.cc | 6 +- src/executor/graph_executor.cc | 13 +- src/executor/tensorrt_pass.cc | 8 +- src/executor/trt_graph_executor.cc | 9 +- src/imperative/cached_op.cc | 57 +- src/imperative/imperative.cc | 57 +- src/imperative/imperative_utils.cc | 6 +- src/imperative/imperative_utils.h | 17 +- src/initialize.cc | 8 +- src/io/image_aug_default.cc | 17 +- src/io/image_det_aug_default.cc | 20 +- src/io/image_io.cc | 9 +- src/io/iter_image_recordio_2.cc | 4 +- src/kvstore/comm.h | 31 +- src/kvstore/comm_tree.h | 37 +- src/kvstore/gpu_topology.h | 89 +- src/ndarray/ndarray.cc | 31 +- src/ndarray/ndarray_function.cc | 14 +- src/nnvm/graph_editor.cc | 13 +- src/nnvm/legacy_json_util.cc | 8 +- src/nnvm/legacy_op_util.cc | 6 +- src/operator/bilinear_sampler-inl.h | 8 +- src/operator/c_lapack_api.cc | 72 + src/operator/c_lapack_api.h | 35 +- src/operator/contrib/adamw-inl.h | 125 + src/operator/contrib/adamw.cc | 71 + src/operator/contrib/adamw.cu | 35 + src/operator/contrib/adaptive_avg_pooling.cc | 8 +- src/operator/contrib/boolean_mask-inl.h | 134 + src/operator/contrib/boolean_mask.cc | 114 + src/operator/contrib/bounding_box-inl.h | 2 +- src/operator/contrib/bounding_box.cc | 27 +- src/operator/contrib/dgl_graph-inl.h | 68 + src/operator/contrib/dgl_graph.cc | 1631 ++++++++++++ src/operator/contrib/dgl_graph.cu | 29 + src/operator/contrib/index_copy-inl.h | 14 +- src/operator/contrib/index_copy.cc | 36 +- src/operator/contrib/multibox_detection.cc | 1 - src/operator/contrib/nnvm_to_onnx-inl.h | 42 +- src/operator/contrib/nnvm_to_onnx.cc | 34 +- src/operator/contrib/nnz.cc | 189 ++ src/operator/contrib/quadratic_op-inl.h | 2 +- src/operator/contrib/quadratic_op.cc | 6 +- src/operator/contrib/roi_align-inl.h | 8 +- src/operator/contrib/roi_align.cc | 63 +- src/operator/contrib/roi_align.cu | 28 +- src/operator/contrib/tensorrt-inl.h | 38 +- src/operator/contrib/tensorrt.cc | 28 +- src/operator/contrib/tensorrt.cu | 2 +- src/operator/cudnn_rnn-inl.h | 305 ++- src/operator/custom/custom.cc | 14 +- src/operator/custom/native_op-inl.h | 12 +- src/operator/custom/ndarray_op-inl.h | 6 +- src/operator/elemwise_op_common.h | 28 +- src/operator/grid_generator-inl.h | 8 +- src/operator/l2_normalization-inl.h | 2 +- src/operator/l2_normalization.cc | 102 +- src/operator/linalg_impl.h | 52 +- src/operator/math_functions-inl.h | 2 + src/operator/mshadow_op.h | 4 + src/operator/mxnet_op.h | 117 +- src/operator/nn/activation-inl.h | 12 +- src/operator/nn/activation.cc | 84 +- src/operator/nn/activation.cu | 30 +- src/operator/nn/batch_norm.cc | 7 +- src/operator/nn/concat.cc | 29 +- src/operator/nn/ctc_loss.cc | 3 + src/operator/nn/cudnn/cudnn_algoreg-inl.h | 66 +- src/operator/nn/cudnn/cudnn_convolution-inl.h | 484 ++-- .../nn/cudnn/cudnn_deconvolution-inl.h | 506 ++-- src/operator/nn/cudnn/cudnn_pooling-inl.h | 5 + src/operator/nn/dropout-inl.h | 5 +- src/operator/nn/mkldnn/mkldnn_act.cc | 19 +- src/operator/nn/mkldnn/mkldnn_base-inl.h | 55 +- src/operator/nn/mkldnn/mkldnn_base.cc | 105 +- .../nn/mkldnn/mkldnn_batch_norm-inl.h | 10 +- src/operator/nn/mkldnn/mkldnn_concat-inl.h | 83 + src/operator/nn/mkldnn/mkldnn_concat.cc | 83 +- src/operator/nn/mkldnn/mkldnn_convolution.cc | 291 ++- .../nn/mkldnn/mkldnn_deconvolution.cc | 121 +- src/operator/nn/mkldnn/mkldnn_lrn-inl.h | 10 +- src/operator/nn/mkldnn/mkldnn_pooling-inl.h | 18 +- src/operator/nn/mkldnn/mkldnn_pooling.cc | 50 +- src/operator/operator_common.h | 4 +- src/operator/operator_tune-inl.h | 5 +- src/operator/operator_tune.cc | 2 + src/operator/operator_util.cc | 10 +- .../mkldnn/mkldnn_quantized_concat.cc | 119 + .../quantization/quantization_utils.h | 2 + .../quantization/quantize_graph_pass.cc | 100 +- src/operator/quantization/quantized_concat.cc | 149 ++ .../quantization/quantized_fully_connected.cc | 159 +- .../quantization/quantized_pooling.cc | 27 +- src/operator/random/multisample_op.h | 8 +- src/operator/random/sample_op.cc | 195 +- src/operator/random/sample_op.cu | 48 +- src/operator/random/sample_op.h | 610 +++-- src/operator/random/sampler.h | 73 +- src/operator/rnn-inl.h | 84 +- src/operator/rnn.cc | 5 + src/operator/rnn.cu | 2 +- src/operator/spatial_transformer-inl.h | 8 +- src/operator/spatial_transformer.cu | 43 +- .../subgraph/mkldnn/mkldnn_conv-inl.h | 3 - src/operator/subgraph/mkldnn/mkldnn_conv.cc | 14 +- src/operator/subgraph/partition_graph.cc | 26 +- src/operator/tensor/broadcast_reduce-inl.h | 98 +- src/operator/tensor/broadcast_reduce_op.h | 4 +- .../tensor/broadcast_reduce_op_index.cc | 2 +- .../tensor/broadcast_reduce_op_index.cu | 4 +- .../tensor/broadcast_reduce_op_value.cc | 4 +- .../tensor/broadcast_reduce_op_value.cu | 4 +- src/operator/tensor/cast_storage-inl.cuh | 2 +- src/operator/tensor/cast_storage-inl.h | 2 +- src/operator/tensor/control_flow_op.cu | 2 +- src/operator/tensor/control_flow_op.h | 18 +- src/operator/tensor/diag_op-inl.h | 2 +- src/operator/tensor/diag_op.cc | 2 +- src/operator/tensor/diag_op.cu | 2 +- src/operator/tensor/dot-inl.cuh | 2 +- src/operator/tensor/dot-inl.h | 23 +- .../elemwise_binary_broadcast_op-inl.cuh | 4 + .../tensor/elemwise_binary_broadcast_op.h | 18 +- .../elemwise_binary_broadcast_op_basic.cc | 4 +- .../elemwise_binary_broadcast_op_basic.cu | 4 +- .../elemwise_binary_broadcast_op_extended.cc | 2 +- .../elemwise_binary_broadcast_op_extended.cu | 4 +- .../elemwise_binary_broadcast_op_logic.cc | 2 +- .../elemwise_binary_broadcast_op_logic.cu | 4 +- src/operator/tensor/elemwise_binary_op-inl.h | 2 +- src/operator/tensor/elemwise_binary_op.cc | 7 + .../tensor/elemwise_binary_op_basic.cc | 4 +- .../tensor/elemwise_binary_op_basic.cu | 4 +- .../tensor/elemwise_binary_op_extended.cc | 4 +- .../tensor/elemwise_binary_op_extended.cu | 2 +- .../tensor/elemwise_binary_op_logic.cc | 2 +- .../tensor/elemwise_binary_op_logic.cu | 2 +- .../tensor/elemwise_binary_scalar_op_basic.cc | 4 +- .../tensor/elemwise_binary_scalar_op_basic.cu | 4 +- .../elemwise_binary_scalar_op_extended.cc | 4 +- .../elemwise_binary_scalar_op_extended.cu | 4 +- .../tensor/elemwise_binary_scalar_op_logic.cc | 4 +- .../tensor/elemwise_binary_scalar_op_logic.cu | 4 +- src/operator/tensor/elemwise_scatter_op.cc | 5 + src/operator/tensor/elemwise_scatter_op.cu | 5 + src/operator/tensor/elemwise_scatter_op.h | 2 +- src/operator/tensor/elemwise_sum.cc | 2 +- src/operator/tensor/elemwise_sum.cu | 2 +- src/operator/tensor/elemwise_sum.h | 2 +- src/operator/tensor/elemwise_unary_op.h | 41 + .../tensor/elemwise_unary_op_basic.cc | 41 +- .../tensor/elemwise_unary_op_basic.cu | 16 +- src/operator/tensor/elemwise_unary_op_trig.cc | 2 +- src/operator/tensor/elemwise_unary_op_trig.cu | 4 +- src/operator/tensor/histogram-inl.h | 4 + src/operator/tensor/histogram.cc | 4 + src/operator/tensor/histogram.cu | 4 + src/operator/tensor/indexing_op.cc | 146 +- src/operator/tensor/indexing_op.cu | 127 +- src/operator/tensor/indexing_op.h | 158 +- src/operator/tensor/init_op.h | 6 +- .../tensor/{la_op_inline.h => la_op-inl.h} | 244 +- src/operator/tensor/la_op.cc | 35 +- src/operator/tensor/la_op.cu | 4 +- src/operator/tensor/la_op.h | 18 +- src/operator/tensor/matrix_op-inl.h | 229 +- src/operator/tensor/matrix_op.cc | 86 +- src/operator/tensor/ordering_op-inl.h | 2 +- src/operator/tensor/ordering_op.cc | 2 +- src/operator/tensor/ordering_op.cu | 4 +- src/operator/tensor/ravel.cc | 2 +- src/operator/tensor/ravel.cu | 2 +- src/operator/tensor/sparse_retain-inl.h | 2 +- src/operator/tensor/sparse_retain.cc | 2 +- src/operator/tensor/sparse_retain.cu | 2 +- src/profiler/aggregate_stats.cc | 13 +- src/storage/cpu_device_storage.h | 10 +- src/storage/gpu_device_storage.h | 12 +- src/storage/naive_storage_manager.h | 6 +- src/storage/pinned_memory_storage.h | 12 +- src/storage/pooled_storage_manager.h | 34 +- src/storage/storage.cc | 50 +- tests/cpp/include/test_mkldnn.h | 625 +++++ tests/cpp/kvstore/gpu_topology_test.cc | 36 +- tests/cpp/operator/activation_perf.cc | 26 +- tests/cpp/operator/krprod_test.cc | 3 + tests/cpp/operator/mkldnn.cc | 1646 ------------ tests/cpp/operator/mkldnn_operator_test.cc | 1356 ++++++++++ tests/cpp/operator/mkldnn_test.cc | 420 ++++ tests/jenkins/run_test.sh | 4 +- tests/jenkins/run_test_amzn_linux_gpu.sh | 4 +- tests/jenkins/run_test_ubuntu.sh | 4 +- tests/nightly/Jenkinsfile | 20 +- tests/nightly/JenkinsfileForBinaries | 20 +- .../apache_rat_license_check/rat-excludes | 14 +- .../JenkinsfileForBLC | 4 +- tests/nightly/dist_async_kvstore.py | 18 +- tests/nightly/dist_device_sync_kvstore.py | 19 + tests/nightly/dist_sync_kvstore.py | 26 +- .../JenkinsfileForMBCC | 4 +- .../train_mxnet_legacy_models.sh | 4 +- tests/nightly/test_large_array.py | 139 +- tests/python-pytest/onnx/README.md | 33 + .../onnx/{export => }/backend.py | 66 +- .../mxnet_backend_rep.py => backend_rep.py} | 93 +- tests/python-pytest/onnx/backend_test.py | 89 + .../python-pytest/onnx/export/backend_rep.py | 84 - .../onnx/export/mxnet_export_test.py | 251 -- .../onnx/export/onnx_backend_test.py | 147 -- .../onnx/import/gluon_backend.py | 75 - .../onnx/import/gluon_backend_rep.py | 71 - .../onnx/import/gluon_backend_test.py | 55 - .../onnx/import/mxnet_backend.py | 67 - .../onnx/import/mxnet_backend_test.py | 55 - .../onnx/import/onnx_import_test.py | 290 --- tests/python-pytest/onnx/import/test_cases.py | 120 - tests/python-pytest/onnx/mxnet_export_test.py | 121 + tests/python-pytest/onnx/test_cases.py | 131 + tests/python-pytest/onnx/test_models.py | 178 ++ tests/python-pytest/onnx/test_node.py | 278 +++ tests/python/gpu/test_device.py | 46 +- tests/python/gpu/test_gluon_gpu.py | 184 +- tests/python/gpu/test_kvstore_gpu.py | 18 +- tests/python/gpu/test_operator_gpu.py | 55 +- tests/python/mkl/test_mkldnn.py | 46 + tests/python/mkl/test_subgraph.py | 23 +- .../python/quantization/test_quantization.py | 239 +- tests/python/unittest/common.py | 11 +- tests/python/unittest/test_autograd.py | 17 +- .../python/unittest/test_contrib_operator.py | 1 + tests/python/unittest/test_dgl_graph.py | 245 ++ tests/python/unittest/test_gluon.py | 21 +- tests/python/unittest/test_gluon_data.py | 30 +- tests/python/unittest/test_gluon_rnn.py | 44 +- tests/python/unittest/test_gluon_trainer.py | 92 +- tests/python/unittest/test_image.py | 254 +- .../python/unittest/test_libinfo.py | 15 +- tests/python/unittest/test_loss.py | 18 + tests/python/unittest/test_metric.py | 63 + tests/python/unittest/test_module.py | 59 + tests/python/unittest/test_ndarray.py | 53 + tests/python/unittest/test_operator.py | 560 +++-- tests/python/unittest/test_optimizer.py | 151 +- tests/python/unittest/test_random.py | 243 +- tests/python/unittest/test_sparse_ndarray.py | 23 +- tests/python/unittest/test_sparse_operator.py | 42 + tests/requirements.txt | 1 + tests/tutorials/test_sanity_tutorials.py | 7 +- tests/tutorials/test_tutorials.py | 14 +- tools/build/build_lib.sh | 80 + tools/build/build_wheel.sh | 31 + tools/dependencies/README.md | 14 + tools/dependencies/cityhash.sh | 32 + tools/dependencies/curl.sh | 64 + tools/dependencies/eigen.sh | 34 + tools/dependencies/libpng.sh | 40 + tools/dependencies/libtiff.sh | 32 + tools/dependencies/libturbojpeg.sh | 47 + .../dependencies/libz.sh | 50 +- tools/dependencies/lz4.sh | 31 + .../dependencies/make_shared_dependencies.sh | 39 +- tools/dependencies/openblas.sh | 35 + tools/dependencies/opencv.sh | 192 ++ tools/dependencies/openssl.sh | 38 + tools/dependencies/patch/opencv_lapack.h | 23 + tools/dependencies/protobuf.sh | 43 + tools/dependencies/zmq.sh | 38 + tools/license_header.py | 173 +- tools/pip/MANIFEST.in | 28 + tools/pip/sanity_test.py | 32 + tools/pip/setup.py | 195 ++ tools/pip_package/README.md | 9 - tools/pip_package/make_pip_package.sh | 179 -- tools/pip_package/setup.py | 60 - tools/setup_gpu_build_tools.sh | 254 ++ 1056 files changed, 50819 insertions(+), 20101 deletions(-) rename README.md => NGRAPH_README.md (100%) create mode 100644 R-package/dummy.NAMESPACE create mode 100644 ci/docker/Dockerfile.publish.test.centos7_cpu create mode 100644 ci/docker/Dockerfile.publish.test.centos7_gpu create mode 100644 ci/docker/Dockerfile.publish.test.ubuntu1404_cpu create mode 100644 ci/docker/Dockerfile.publish.test.ubuntu1404_gpu create mode 100644 ci/docker/Dockerfile.publish.test.ubuntu1604_cpu create mode 100644 ci/docker/Dockerfile.publish.test.ubuntu1604_gpu rename example/bi-lstm-sort/gen_data.py => ci/docker/Dockerfile.publish.test.ubuntu1804_cpu (61%) create mode 100644 ci/docker/Dockerfile.publish.test.ubuntu1804_gpu create mode 100644 ci/docker/Dockerfile.publish.ubuntu1404_cpu create mode 100644 ci/docker/Dockerfile.publish.ubuntu1404_gpu create mode 100755 ci/docker/install/centos7_base.sh create mode 100755 ci/docker/install/centos7_scala.sh create mode 100755 ci/docker/install/ubuntu_base.sh create mode 100755 ci/docker/install/ubuntu_mkl.sh create mode 100644 ci/docker/install/ubuntu_publish.sh rename Jenkinsfile => ci/jenkins/Jenkins_steps.groovy (71%) create mode 100644 ci/jenkins/Jenkinsfile_centos_cpu create mode 100644 ci/jenkins/Jenkinsfile_centos_gpu create mode 100644 ci/jenkins/Jenkinsfile_clang create mode 100644 ci/jenkins/Jenkinsfile_edge create mode 100644 ci/jenkins/Jenkinsfile_miscellaneous create mode 100644 ci/jenkins/Jenkinsfile_sanity create mode 100644 ci/jenkins/Jenkinsfile_unix_cpu create mode 100644 ci/jenkins/Jenkinsfile_unix_gpu create mode 100644 ci/jenkins/Jenkinsfile_website create mode 100644 ci/jenkins/Jenkinsfile_windows_cpu create mode 100644 ci/jenkins/Jenkinsfile_windows_gpu create mode 100644 ci/publish/Jenkinsfile create mode 100755 ci/publish/scala/build.sh create mode 100644 ci/publish/scala/buildkey.py create mode 100755 ci/publish/scala/deploy.sh create mode 100644 ci/publish/scala/fullDeploy.sh create mode 100755 ci/publish/scala/test.sh create mode 100644 ci/requirements.txt create mode 100644 cmake/AutoDetectF16C.cmake create mode 100644 cmake/DownloadMKLML.cmake delete mode 100644 cmake/MklDnn.cmake create mode 100644 cmake/cmake_options.yml create mode 100644 contrib/clojure-package/examples/captcha/.gitignore create mode 100644 contrib/clojure-package/examples/captcha/README.md create mode 100644 contrib/clojure-package/examples/captcha/captcha_example.png create mode 100755 contrib/clojure-package/examples/captcha/gen_captcha.py create mode 100755 contrib/clojure-package/examples/captcha/get_data.sh create mode 100644 contrib/clojure-package/examples/captcha/project.clj create mode 100644 contrib/clojure-package/examples/captcha/src/captcha/consts.clj create mode 100644 contrib/clojure-package/examples/captcha/src/captcha/infer_ocr.clj create mode 100644 contrib/clojure-package/examples/captcha/src/captcha/train_ocr.clj create mode 100644 contrib/clojure-package/examples/captcha/test/captcha/train_ocr_test.clj create mode 100644 contrib/clojure-package/examples/cnn-text-classification/test/cnn_text_classification/classifier_test.clj create mode 100644 contrib/clojure-package/examples/gan/test/gan/gan_test.clj create mode 100644 contrib/clojure-package/examples/imclassification/test/imclassification/train_mnist_test.clj create mode 100644 contrib/clojure-package/examples/imclassification/test/test-symbol.json.ref create mode 100644 contrib/clojure-package/examples/infer/imageclassifier/.gitignore create mode 100644 contrib/clojure-package/examples/infer/imageclassifier/README.md create mode 100644 contrib/clojure-package/examples/infer/imageclassifier/project.clj create mode 100755 contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_18_data.sh create mode 100755 contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_data.sh create mode 100644 contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj create mode 100644 contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj create mode 100644 contrib/clojure-package/examples/infer/objectdetector/.gitignore create mode 100644 contrib/clojure-package/examples/infer/objectdetector/README.md create mode 100644 contrib/clojure-package/examples/infer/objectdetector/project.clj create mode 100755 contrib/clojure-package/examples/infer/objectdetector/scripts/get_ssd_data.sh create mode 100644 contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj create mode 100644 contrib/clojure-package/examples/infer/objectdetector/test/infer/objectdetector_example_test.clj create mode 100644 contrib/clojure-package/examples/infer/predictor/.gitignore create mode 100644 contrib/clojure-package/examples/infer/predictor/README.md create mode 100644 contrib/clojure-package/examples/infer/predictor/project.clj create mode 100755 contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_18_data.sh create mode 100755 contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_data.sh create mode 100644 contrib/clojure-package/examples/infer/predictor/src/infer/predictor_example.clj create mode 100644 contrib/clojure-package/examples/infer/predictor/test/infer/predictor_example_test.clj create mode 100644 contrib/clojure-package/examples/module/test/mnist_mlp_test.clj create mode 100644 contrib/clojure-package/examples/multi-label/test/multi_label_test.clj create mode 100644 contrib/clojure-package/examples/neural-style/test/neural_style/vgg_19_test.clj create mode 100644 contrib/clojure-package/examples/profiler/test/core_test.clj create mode 100644 contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref create mode 100644 contrib/clojure-package/examples/rnn/test/rnn/core_test.clj create mode 100644 contrib/clojure-package/examples/tutorial/test/tutorial/core_test.clj create mode 100644 contrib/clojure-package/examples/visualization/test/visualization/core_test.clj create mode 100755 contrib/clojure-package/integration-tests.sh create mode 100755 contrib/clojure-package/scripts/infer/get_resnet_18_data.sh create mode 100755 contrib/clojure-package/scripts/infer/get_ssd_data.sh rename ci/docker/qemu/qemu_run.sh => contrib/clojure-package/scripts/update_versions.sh (70%) create mode 100644 contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj create mode 100644 contrib/clojure-package/src/org/apache/clojure_mxnet/infer.clj create mode 100644 contrib/clojure-package/src/org/apache/clojure_mxnet/primitives.clj create mode 100644 contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj create mode 100644 contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj create mode 100644 contrib/clojure-package/test/org/apache/clojure_mxnet/infer/objectdetector_test.clj create mode 100644 contrib/clojure-package/test/org/apache/clojure_mxnet/infer/predictor_test.clj create mode 100644 contrib/clojure-package/test/org/apache/clojure_mxnet/primitives_test.clj create mode 100644 contrib/clojure-package/test/test-images/Pug-Cookie.jpg create mode 100644 contrib/clojure-package/test/test-images/kitten.jpg create mode 100644 cpp-package/example/inference/Makefile create mode 100644 cpp-package/example/inference/README.md create mode 100644 cpp-package/example/inference/inception_inference.cpp create mode 100755 cpp-package/example/inference/unit_test_inception_inference.sh create mode 100755 dev_menu.py create mode 100644 docs/_static/js/clipboard.js create mode 100644 docs/api/java/index.md create mode 100644 docs/install/java_setup.md rename {example => docs/tutorial_utils/vision}/cnn_visualization/gradcam.py (98%) create mode 100644 docs/tutorials/basic/reshape_transpose.md create mode 100644 docs/tutorials/gluon/info_gan.md create mode 100644 docs/tutorials/java/index.md create mode 100644 docs/tutorials/java/mxnet_java_on_intellij.md create mode 100644 docs/tutorials/java/ssd_inference.md rename example/{vae => autoencoder/variational_autoencoder}/README.md (100%) rename example/{vae => autoencoder/variational_autoencoder}/VAE.py (100%) create mode 100755 example/autoencoder/variational_autoencoder/VAE_example.ipynb create mode 100644 example/bi-lstm-sort/bi-lstm-sort.ipynb delete mode 100644 example/bi-lstm-sort/infer_sort.py delete mode 100644 example/bi-lstm-sort/lstm.py delete mode 100644 example/bi-lstm-sort/lstm_sort.py delete mode 100644 example/bi-lstm-sort/rnn_model.py delete mode 100644 example/bi-lstm-sort/sort_io.py delete mode 100644 example/cnn_visualization/README.md delete mode 100644 example/cnn_visualization/gradcam_demo.py delete mode 100644 example/cnn_visualization/vgg.py rename example/gluon/{ => actor_critic}/actor_critic.py (100%) create mode 100644 example/gluon/audio/transforms.py create mode 100644 example/gluon/audio/urban_sounds/README.md create mode 100644 example/gluon/audio/urban_sounds/datasets.py create mode 100644 example/gluon/audio/urban_sounds/model.py create mode 100644 example/gluon/audio/urban_sounds/predict.py create mode 100644 example/gluon/audio/urban_sounds/requirements.txt create mode 100644 example/gluon/audio/urban_sounds/train.py rename example/gluon/{DCGAN => dc_gan}/README.md (100%) rename example/gluon/{DCGAN => dc_gan}/__init__.py (100%) rename example/gluon/{DCGAN => dc_gan}/dcgan.py (100%) rename example/gluon/{DCGAN => dc_gan}/inception_score.py (100%) rename example/gluon/{ => house_prices}/kaggle_k_fold_cross_validation.py (100%) delete mode 100644 example/gluon/learning_rate_manipulation.py rename example/gluon/{ => lstm_crf}/lstm_crf.py (95%) rename example/gluon/{ => mnist}/mnist.py (100%) rename example/gluon/{ => super_resolution}/super_resolution.py (100%) create mode 100644 example/gluon/tree_lstm/README.md delete mode 100644 example/memcost/Makefile delete mode 100644 example/memcost/README.md delete mode 100644 example/memcost/inception_memcost.py rename example/{ => module}/utils/__init__.py (100%) rename example/{ => module}/utils/get_data.py (94%) delete mode 100644 example/multi-task/example_multi_task.py create mode 100644 example/multi-task/multi-task-learning.ipynb delete mode 100644 example/notebooks/README.md delete mode 100644 example/numpy-ops/numpy_softmax.py delete mode 100644 example/onnx/super_resolution.py delete mode 100644 example/python-howto/README.md delete mode 100644 example/python-howto/data_iter.py delete mode 100644 example/python-howto/monitor_weights.py delete mode 100644 example/python-howto/multiple_outputs.py delete mode 100644 example/recommenders/crossentropy.py delete mode 100644 example/recommenders/demo1-MF2-fancy.ipynb delete mode 100644 example/recommenders/demo2-binary.ipynb create mode 100644 example/recommenders/demo2-dssm.ipynb delete mode 100644 example/recommenders/demo3-dssm.ipynb delete mode 100644 example/recommenders/negativesample.py delete mode 100644 example/recommenders/randomproj.py delete mode 100644 example/recommenders/recotools.py delete mode 100644 example/recommenders/symbol_alexnet.py delete mode 100644 example/rnn-time-major/bucket_io.py delete mode 100755 example/rnn-time-major/get_sherlockholmes_data.sh delete mode 100644 example/rnn-time-major/readme.md delete mode 100644 example/rnn-time-major/rnn_cell_demo.py delete mode 100644 example/rnn/large_word_lm/readme.md rename example/{mxnet_adversarial_vae => vae-gan}/README.md (100%) rename example/{mxnet_adversarial_vae => vae-gan}/convert_data.py (100%) rename example/{mxnet_adversarial_vae => vae-gan}/vaegan_mxnet.py (100%) delete mode 100755 example/vae/VAE_example.ipynb create mode 120000 include/dlpack create mode 120000 include/dmlc create mode 120000 include/mshadow create mode 100644 include/mxnet/c_api_error.h create mode 120000 include/nnvm create mode 100644 julia/docs/.gitignore create mode 100644 julia/docs/Project.toml create mode 100644 make/config/libmxnet.sym create mode 100644 make/config/libmxnet.ver create mode 100644 make/maven/maven_darwin_cpu.mk create mode 100644 make/maven/maven_linux_cpu.mk create mode 100644 make/maven/maven_linux_cu90.mk create mode 100644 make/pip/pip_darwin_cpu.mk create mode 100644 make/pip/pip_darwin_mkl.mk create mode 100644 make/pip/pip_linux_cpu.mk create mode 100644 make/pip/pip_linux_cu100.mk create mode 100644 make/pip/pip_linux_cu100mkl.mk create mode 100644 make/pip/pip_linux_cu75.mk create mode 100644 make/pip/pip_linux_cu75mkl.mk create mode 100644 make/pip/pip_linux_cu80.mk create mode 100644 make/pip/pip_linux_cu80mkl.mk create mode 100644 make/pip/pip_linux_cu90.mk create mode 100644 make/pip/pip_linux_cu90mkl.mk create mode 100644 make/pip/pip_linux_cu91.mk create mode 100644 make/pip/pip_linux_cu91mkl.mk create mode 100644 make/pip/pip_linux_cu92.mk create mode 100644 make/pip/pip_linux_cu92mkl.mk create mode 100644 make/pip/pip_linux_mkl.mk delete mode 100644 scala-package/assembly/linux-x86_64-cpu/pom.xml delete mode 100644 scala-package/assembly/linux-x86_64-cpu/src/main/assembly/assembly.xml delete mode 100644 scala-package/assembly/linux-x86_64-gpu/pom.xml delete mode 100644 scala-package/assembly/linux-x86_64-gpu/src/main/assembly/assembly.xml delete mode 100644 scala-package/assembly/osx-x86_64-cpu/main/assembly/assembly.xml delete mode 100644 scala-package/assembly/osx-x86_64-cpu/pom.xml delete mode 100644 scala-package/assembly/osx-x86_64-cpu/src/main/assembly/assembly.xml delete mode 100644 scala-package/assembly/src/javadoc.xml create mode 100644 scala-package/assembly/src/main/assembly/assembly.xml create mode 100644 scala-package/assembly/src/main/assembly/javadoc.xml create mode 100644 scala-package/assembly/src/main/assembly/source.xml delete mode 100644 scala-package/assembly/src/source.xml create mode 100644 scala-package/core/src/main/scala/org/apache/mxnet/Image.scala create mode 100644 scala-package/core/src/main/scala/org/apache/mxnet/MX_PRIMITIVES.scala create mode 100644 scala-package/core/src/main/scala/org/apache/mxnet/javaapi/NDArray.scala create mode 100644 scala-package/core/src/main/scala/org/apache/mxnet/util/OptionConversion.scala create mode 100644 scala-package/core/src/test/java/org/apache/mxnet/javaapi/NDArrayTest.java create mode 100644 scala-package/core/src/test/java/org/apache/mxnet/javaapi/ResourceScopeTestSuite.java create mode 100644 scala-package/core/src/test/scala/org/apache/mxnet/ImageSuite.scala create mode 100644 scala-package/deploy/pom.xml create mode 100644 scala-package/deploy/src/main/deploy/deploy.xml create mode 100644 scala-package/examples/scripts/benchmark/run_java_inference_bm.sh create mode 100755 scala-package/examples/scripts/infer/objectdetector/run_ssd_java_example.sh create mode 100755 scala-package/examples/scripts/infer/predictor/run_predictor_java_example.sh create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/InferBase.java create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/JavaBenchmark.java create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/ObjectDetectionBenchmark.java create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/README.md create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/SSDClassifierExample.java create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/PredictorExample.java create mode 100644 scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/README.md create mode 100644 scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/predictor/PredictorExample.scala create mode 100644 scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/predictor/PredictorExampleSuite.scala create mode 100644 scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetector.scala create mode 100644 scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetectorOutput.scala create mode 100644 scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/Predictor.scala create mode 100644 scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorOutputTest.java create mode 100644 scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorTest.java create mode 100644 scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/PredictorTest.java delete mode 100644 scala-package/init-native/linux-x86_64/pom.xml delete mode 100644 scala-package/init-native/osx-x86_64/pom.xml create mode 100644 scala-package/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h create mode 100644 scala-package/macros/src/main/scala/org/apache/mxnet/GeneratorBase.scala create mode 100644 scala-package/macros/src/main/scala/org/apache/mxnet/javaapi/JavaNDArrayMacro.scala create mode 100644 scala-package/memory-management.md create mode 100644 scala-package/mxnet-demo/java-demo/Makefile create mode 100644 scala-package/mxnet-demo/java-demo/README.md rename tools/pip_package/MANIFEST.in => scala-package/mxnet-demo/java-demo/bin/java_sample.sh (81%) mode change 100644 => 100755 rename ci/docker/qemu/ansible.cfg => scala-package/mxnet-demo/java-demo/bin/run_od.sh (81%) mode change 100644 => 100755 create mode 100644 scala-package/mxnet-demo/java-demo/pom.xml create mode 100644 scala-package/mxnet-demo/java-demo/src/main/java/mxnet/HelloWorld.java create mode 100644 scala-package/mxnet-demo/java-demo/src/main/java/mxnet/ObjectDetection.java rename scala-package/mxnet-demo/{ => scala-demo}/Makefile (90%) rename scala-package/mxnet-demo/{ => scala-demo}/README.md (85%) rename scala-package/mxnet-demo/{ => scala-demo}/bin/demo.sh (100%) rename scala-package/mxnet-demo/{ => scala-demo}/bin/run_im.sh (100%) rename scala-package/mxnet-demo/{ => scala-demo}/pom.xml (87%) rename scala-package/mxnet-demo/{ => scala-demo}/src/main/scala/sample/HelloWorld.scala (100%) rename scala-package/mxnet-demo/{ => scala-demo}/src/main/scala/sample/ImageClassificationExample.scala (100%) create mode 100644 scala-package/native/README.md delete mode 100644 scala-package/native/linux-x86_64-cpu/pom.xml delete mode 100644 scala-package/native/linux-x86_64-gpu/pom.xml delete mode 100644 scala-package/native/osx-x86_64-cpu/pom.xml create mode 100644 scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h create mode 100644 scala-package/packageTest/Makefile create mode 100644 scala-package/packageTest/README.md create mode 100644 scala-package/packageTest/core/pom.xml create mode 120000 scala-package/packageTest/core/scripts create mode 100644 scala-package/packageTest/examples/pom.xml create mode 120000 scala-package/packageTest/examples/scripts create mode 100644 scala-package/packageTest/infer/pom.xml create mode 100644 scala-package/packageTest/pom.xml create mode 100644 src/operator/c_lapack_api.cc create mode 100644 src/operator/contrib/adamw-inl.h create mode 100644 src/operator/contrib/adamw.cc create mode 100644 src/operator/contrib/adamw.cu create mode 100644 src/operator/contrib/boolean_mask-inl.h create mode 100644 src/operator/contrib/boolean_mask.cc create mode 100644 src/operator/contrib/dgl_graph-inl.h create mode 100644 src/operator/contrib/dgl_graph.cc create mode 100644 src/operator/contrib/dgl_graph.cu create mode 100644 src/operator/contrib/nnz.cc create mode 100644 src/operator/nn/mkldnn/mkldnn_concat-inl.h create mode 100644 src/operator/quantization/mkldnn/mkldnn_quantized_concat.cc create mode 100644 src/operator/quantization/quantized_concat.cc rename src/operator/tensor/{la_op_inline.h => la_op-inl.h} (77%) create mode 100644 tests/cpp/include/test_mkldnn.h delete mode 100644 tests/cpp/operator/mkldnn.cc create mode 100644 tests/cpp/operator/mkldnn_operator_test.cc create mode 100644 tests/cpp/operator/mkldnn_test.cc create mode 100644 tests/python-pytest/onnx/README.md rename tests/python-pytest/onnx/{export => }/backend.py (57%) rename tests/python-pytest/onnx/{import/mxnet_backend_rep.py => backend_rep.py} (56%) create mode 100644 tests/python-pytest/onnx/backend_test.py delete mode 100644 tests/python-pytest/onnx/export/backend_rep.py delete mode 100644 tests/python-pytest/onnx/export/mxnet_export_test.py delete mode 100644 tests/python-pytest/onnx/export/onnx_backend_test.py delete mode 100644 tests/python-pytest/onnx/import/gluon_backend.py delete mode 100644 tests/python-pytest/onnx/import/gluon_backend_rep.py delete mode 100644 tests/python-pytest/onnx/import/gluon_backend_test.py delete mode 100644 tests/python-pytest/onnx/import/mxnet_backend.py delete mode 100644 tests/python-pytest/onnx/import/mxnet_backend_test.py delete mode 100644 tests/python-pytest/onnx/import/onnx_import_test.py delete mode 100644 tests/python-pytest/onnx/import/test_cases.py create mode 100644 tests/python-pytest/onnx/mxnet_export_test.py create mode 100644 tests/python-pytest/onnx/test_cases.py create mode 100644 tests/python-pytest/onnx/test_models.py create mode 100644 tests/python-pytest/onnx/test_node.py create mode 100644 tests/python/unittest/test_dgl_graph.py rename julia/deps/cpcblas.sh => tests/python/unittest/test_libinfo.py (75%) mode change 100755 => 100644 create mode 100755 tools/build/build_lib.sh create mode 100755 tools/build/build_wheel.sh create mode 100644 tools/dependencies/README.md create mode 100755 tools/dependencies/cityhash.sh create mode 100755 tools/dependencies/curl.sh create mode 100755 tools/dependencies/eigen.sh create mode 100755 tools/dependencies/libpng.sh create mode 100755 tools/dependencies/libtiff.sh create mode 100755 tools/dependencies/libturbojpeg.sh rename ci/docker/qemu/playbook.yml => tools/dependencies/libz.sh (55%) mode change 100644 => 100755 create mode 100755 tools/dependencies/lz4.sh rename example/python-howto/debug_conv.py => tools/dependencies/make_shared_dependencies.sh (58%) mode change 100644 => 100755 create mode 100755 tools/dependencies/openblas.sh create mode 100755 tools/dependencies/opencv.sh create mode 100755 tools/dependencies/openssl.sh create mode 100644 tools/dependencies/patch/opencv_lapack.h create mode 100755 tools/dependencies/protobuf.sh create mode 100755 tools/dependencies/zmq.sh create mode 100644 tools/pip/MANIFEST.in create mode 100644 tools/pip/sanity_test.py create mode 100644 tools/pip/setup.py delete mode 100644 tools/pip_package/README.md delete mode 100755 tools/pip_package/make_pip_package.sh delete mode 100644 tools/pip_package/setup.py create mode 100755 tools/setup_gpu_build_tools.sh diff --git a/.gitignore b/.gitignore index c8a813649..7eb8e7d6e 100644 --- a/.gitignore +++ b/.gitignore @@ -167,13 +167,11 @@ python/.eggs tests/Makefile tests/mxnet_unit_tests -# generated wrappers for ccache -cc -cxx - # Code coverage related .coverage *.gcov *.gcno coverage.xml +# Local CMake build config +cmake_options.yml diff --git a/.travis.yml b/.travis.yml index b5c323f3a..079293059 100644 --- a/.travis.yml +++ b/.travis.yml @@ -35,8 +35,9 @@ script: - mv make/osx.mk config.mk - make -j 2 + # Temporarily disabled due to https://github.com/apache/incubator-mxnet/issues/13136 # We ignore several tests to avoid possible timeouts on large PRs. # This lowers our test coverage, but is required for consistent Travis runs. # These tests will be tested in a variety of environments in Jenkins based tests. - - python -m nose --with-timer --exclude-test=test_sparse_operator.test_elemwise_binary_ops --exclude-test=test_gluon_model_zoo.test_models --exclude-test=test_random.test_shuffle --exclude-test=test_operator.test_broadcast_binary_op --exclude-test=test_operator.test_pick --exclude-test=test_profiler.test_continuous_profile_and_instant_marker --exclude-test=test_metric_perf.test_metric_performance --exclude-test=test_operator.test_order --verbose tests/python/unittest/ - - python2 -m nose --verbose tools/coreml/test --exclude-test=test_mxnet_image +# - python -m nose --with-timer --exclude-test=test_sparse_operator.test_elemwise_binary_ops --exclude-test=test_gluon_model_zoo.test_models --exclude-test=test_random.test_shuffle --exclude-test=test_operator.test_broadcast_binary_op --exclude-test=test_operator.test_pick --exclude-test=test_profiler.test_continuous_profile_and_instant_marker --exclude-test=test_metric_perf.test_metric_performance --exclude-test=test_operator.test_order --verbose tests/python/unittest/ +# - python2 -m nose --verbose tools/coreml/test --exclude-test=test_mxnet_image diff --git a/3rdparty/mkldnn b/3rdparty/mkldnn index 0e7ca7388..a7c5f5383 160000 --- a/3rdparty/mkldnn +++ b/3rdparty/mkldnn @@ -1 +1 @@ -Subproject commit 0e7ca738866d22cc700aa33b8de120b938f910d0 +Subproject commit a7c5f53832acabade6e5086e72c960adedb3c38a diff --git a/3rdparty/mshadow b/3rdparty/mshadow index 696803bd7..6dc04f7c7 160000 --- a/3rdparty/mshadow +++ b/3rdparty/mshadow @@ -1 +1 @@ -Subproject commit 696803bd7723ade8230af878460d96c68a550fbc +Subproject commit 6dc04f7c729cd5c6c6210d5d4d2026a26ce0bfbf diff --git a/CMakeLists.txt b/CMakeLists.txt index d4bda9f40..45ab12d14 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -21,7 +21,7 @@ mxnet_option(USE_LAPACK "Build with lapack support" ON) mxnet_option(USE_NGRAPH "Build with nGraph support" ON) mxnet_option(USE_MKL_IF_AVAILABLE "Use MKL if found" ON) mxnet_option(USE_MKLML_MKL "Use MKLDNN variant of MKL (if MKL found)" ON IF USE_MKL_IF_AVAILABLE AND (NOT APPLE)) -mxnet_option(USE_MKLDNN "Use MKLDNN variant of MKL (if MKL found)" ON IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) AND (NOT USE_NGRAPH)) +mxnet_option(USE_MKLDNN "Use MKLDNN variant of MKL (if MKL found)" ON IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) AND (NOT MSVC) AND (CMAKE_SYSTEM_PROCESSOR MATCHES x86_64) AND (NOT USE_NGRAPH)) mxnet_option(USE_OPERATOR_TUNING "Enable auto-tuning of operators" ON IF NOT MSVC) mxnet_option(USE_GPERFTOOLS "Build with GPerfTools support (if found)" ON) mxnet_option(USE_JEMALLOC "Build with Jemalloc support" ON) @@ -113,27 +113,17 @@ else(MSVC) endif() # For cross complication, turn off flag if target device does not support it if(USE_F16C) - check_cxx_compiler_flag("-mf16c" COMPILER_SUPPORT_MF16C) - if(CMAKE_SYSTEM_NAME STREQUAL "Linux") - execute_process(COMMAND cat /proc/cpuinfo - COMMAND grep flags - COMMAND grep f16c - OUTPUT_VARIABLE CPU_SUPPORT_F16C) - elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") - execute_process(COMMAND sysctl -a - COMMAND grep machdep.cpu.features - COMMAND grep F16C - OUTPUT_VARIABLE CPU_SUPPORT_F16C) - endif() - if(NOT CPU_SUPPORT_F16C) - message("CPU does not support F16C instructions") - endif() - if(CPU_SUPPORT_F16C AND COMPILER_SUPPORT_MF16C) - set(SUPPORT_F16C TRUE) - endif() + # Determine if hardware supports F16C instruction set + message(STATUS "Determining F16C support") + include(cmake/AutoDetectF16C.cmake) else() set(SUPPORT_F16C FALSE) endif() + if(SUPPORT_F16C) + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mf16c") + else() + add_definitions(-DMSHADOW_USE_F16C=0) + endif() set(CMAKE_POSITION_INDEPENDENT_CODE ON) set(CMAKE_C_FLAGS "-Wall -Wno-unknown-pragmas -Wno-sign-compare") if ("${CMAKE_CXX_COMPILER_ID}" MATCHES ".*Clang$") @@ -304,13 +294,17 @@ if(ENABLE_TESTCOVERAGE) endif() if(USE_MKLDNN) - include(cmake/MklDnn.cmake) + include(cmake/DownloadMKLML.cmake) # CPU architecture (e.g., C5) can't run on another architecture (e.g., g3). if(NOT MSVC) set(ARCH_OPT_FLAGS "-mtune=generic") + else() + set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /EHsc") + set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /EHsc /Gy") endif() - set(WITH_TEST OFF) - set(WITH_EXAMPLE OFF) + set(WITH_TEST OFF CACHE INTERNAL "" FORCE) + set(WITH_EXAMPLE OFF CACHE INTERNAL "" FORCE) + set(ARCH_OPT_FLAGS "" CACHE INTERNAL "" FORCE) add_subdirectory(3rdparty/mkldnn) include_directories(3rdparty/mkldnn/include) @@ -498,6 +492,7 @@ endif() # ---[ LAPack if(USE_LAPACK) + message("USE_LAPACK is ON") add_definitions(-DMXNET_USE_LAPACK=1) if (NOT MSVC) list(APPEND mxnet_LINKER_LIBS lapack) @@ -816,8 +811,12 @@ install(TARGETS ${MXNET_INSTALL_TARGETS} LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} ARCHIVE DESTINATION ${CMAKE_INSTALL_LIBDIR} ) +# NOTE: Public headers will be installed into ${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_INCLUDEDIR}, see +# https://cmake.org/cmake/help/v3.0/variable/CMAKE_INSTALL_PREFIX.html +# https://cmake.org/cmake/help/v3.0/module/GNUInstallDirs.html install(DIRECTORY include/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) +install(DIRECTORY 3rdparty/tvm/nnvm/include/ DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) if (INSTALL_EXAMPLES) install(DIRECTORY example DESTINATION ${CMAKE_INSTALL_DATADIR}/${PROJECT_NAME}) endif() diff --git a/CODEOWNERS b/CODEOWNERS index 5a88e89df..a9655ecf0 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -12,15 +12,19 @@ * @apache/mxnet-committers # Language bindings -/R-package/ @thirdwing -/scala-package/ @yzhliu @nswamy -/perl-package/ @sergeykolychev -/python/ @szha -/contrib/clojure-package/ @gigasquid +/R-package/ @thirdwing +/scala-package/ @yzhliu @nswamy @pllarroy +/perl-package/ @sergeykolychev +/python/ @szha @pllarroy +/python/mxnet/kvstore.py @eric-haibin-lin +/python/mxnet/optimizer/ @eric-haibin-lin +/python/mxnet/gluon/trainer.py @eric-haibin-lin +/contrib/clojure-package/ @gigasquid +/julia/ @iblis17 # C++ base /src/kvstore/ @rahul003 @anirudh2290 -/include/ @anirudh2290 +/include/ @anirudh2290 @pllarroy /src/c_api/ @anirudh2290 /src/common/ @anirudh2290 /src/engine/ @anirudh2290 @@ -31,15 +35,20 @@ /src/nnvm/ @anirudh2290 /src/operator/ @anirudh2290 /src/profiler/ @anirudh2290 +/src/kvstore/ @eric-haibin-lin /src/storage/ @anirudh2290 /tests/cpp/ @anirudh2290 -/cpp-package/ @nswamy +/cpp-package/ @nswamy @pllarroy +/src/ @pllarroy +/plugin/ @pllarroy # CMake -CMakeLists.txt @szha @rahul003 -/cmake/ @szha @rahul003 +CMakeLists.txt @szha @rahul003 @pllarroy +/cmake/ @szha @rahul003 @pllarroy # MXNet CI +dev_menu.py @pllarroy +/ci/ @pllarroy /tests/ci_build/ @marcoabreu Jenkinsfile @marcoabreu .travis.yml @marcoabreu @@ -50,16 +59,16 @@ Makefile @szha prepare_mkl.sh @szha # Docs -/docs/ @szha +/docs/ @szha @pllarroy # Submodules .gitmodules @szha # Examples -/example/ @szha +/example/ @szha @pllarroy # Tools -/tools/ @szha +/tools/ @szha @pllarroy # Github templates /.github/ @szha diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md index 5e5e76d52..5b5fdce71 100644 --- a/CONTRIBUTORS.md +++ b/CONTRIBUTORS.md @@ -58,6 +58,7 @@ who are willing to help maintaining and leading the project. Committers come fro New committers will be proposed by current committers, with support from more than two of current committers. + List of Contributors -------------------- * [Full List of Contributors](https://github.com/apache/incubator-mxnet/graphs/contributors) @@ -187,3 +188,22 @@ List of Contributors * [LuckyPigeon](https://github.com/LuckyPigeon) * [Anton Chernov](https://github.com/lebeg) * [Denisa Roberts](https://github.com/D-Roberts) +* [Dick Carter](https://github.com/DickJC123) +* [Rahul Padmanabhan](https://github.com/rahul3) +* [Yuxi Hu](https://github.com/yuxihu) +* [Harsh Patel](https://github.com/harshp8l) +* [Xiao Wang](https://github.com/BeyonderXX) +* [Piyush Ghai](https://github.com/piyushghai) + +Label Bot +--------- +* [mxnet-label-bot](https://github.com/mxnet-label-bot) + - mxnet-label-bot provides users with the functionality to manage labels for Issues/Pull Requests on the repository + - To use me, comment: + - @mxnet-label-bot add [specify comma separated labels here] + - @mxnet-label-bot remove [specify comma separated labels here] + - @mxnet-label-bot update [specify comma separated labels here] + (i.e. @mxnet-label-bot update [Bug, Python]) + + - Available label names which are supported: [Labels](https://github.com/apache/incubator-mxnet/labels) + - For further details: [My Wiki Page](https://cwiki.apache.org/confluence/display/MXNET/Machine+Learning+Based+GitHub+Bot) diff --git a/LICENSE b/LICENSE index a8b57e583..b73ba3740 100644 --- a/LICENSE +++ b/LICENSE @@ -216,18 +216,65 @@ The following components are provided under an Apache 2.0 license. 1. MXNet Cpp-package - For details, /cpp-package/LICENSE + Copyright (c) 2015-2016 by Contributors 2. MXNet rcnn - For details, see, example/rcnn/LICENSE - 3. scala-package - For details, see, scala-package/LICENSE - 4. Warp-CTC - For details, see, src/operator/contrib/ctc_include/LICENSE + Copyright (c) 2014, 2015, The Regents of the University of California (Regents) + 3. MXNet scala-package - For details, see, scala-package/LICENSE + Copyright (c) 2014, 2015, the respective contributors + 4. Warp-CTC - For details, see, 3rdparty/ctc_include/LICENSE + Copyright 2015-2016, Baidu USA LLC. 5. 3rdparty/dlpack - For details, see, 3rdparty/dlpack/LICENSE + Copyright 2017 by Contributors 6. 3rdparty/dmlc-core - For details, see, 3rdparty/dmlc-core/LICENSE + Copyright (c) 2015 by Contributors + Copyright 2015 by dmlc-core developers + Copyright by Contributors 7. 3rdparty/mshadow - For details, see, 3rdparty/mshadow/LICENSE + Copyright (c) 2014-2016 by Contributors + Copyright by Contributors 8. 3rdparty/tvm - For details, see, 3rdparty/tvm/LICENSE - 9. 3rdparty/tvm/dmlc-core - For details, see, 3rdparty/tvm/dmlc-core/LICENSE - 10. 3rdparty/tvm/nnvm - For details, see, 3rdparty/tvm/nnvm/LICENSE + Copyright (c) 2016-2018 by Contributors + Copyright 2018 by Contributors + Copyright (c) 2018 by Xilinx, Contributors + 9. 3rdparty/tvm/dmlc-core - For details, see, 3rdparty/tvm/3rdparty/dmlc-core/LICENSE + Copyright (c) 2015 by Contributors + 10. 3rdparty/tvm/dlpack - For details, see, 3rdparty/tvm/3rdparty/dlpack/LICENSE + Copyright (c) 2015-2017 by Contributors + Copyright by Contributors 11. 3rdparty/ps-lite - For details, see, 3rdparty/ps-lite/LICENSE + Copyright 2015 Carnegie Mellon University + Copyright 2016, ps-lite developers + Copyright (c) 2015-2016 by Contributors + Copyright by Contributors 12. 3rdparty/mkldnn - For details, see, 3rdparty/mkldnn/LICENSE + Copyright (c) 2017-2018 Intel Corporation + Copyright 2016-2018 Intel Corporation + Copyright 2018 YANDEX LLC 13. googlemock scripts/generator - For details, see, 3rdparty/googletest/googlemock/scripts/generator/LICENSE + Copyright [2007-2009] Neal Norwitz + Portions Copyright [2007-2009] Google Inc. + 14. MXNet clojure-package - For details, see, contrib/clojure-package/LICENSE + Copyright 2018 by Contributors + 15. MXNet R-package - For details, see, R-package/LICENSE + Copyright (c) 2015 by Contributors + 16. ONNX-TensorRT benchmark package - For details, see, 3rdparty/onnx-tensorrt/third_party/onnx/third_party/benchmark/LICENSE + Copyright 2015 Google Inc. All rights reserved. + Copyright 2016 Ismael Jimenez Martinez. All rights reserved. + Copyright 2017 Roman Lebedev. All rights reserved. + 17. Dockerfiles - For details, see docker/Dockerfiles/License.md + 18. MXNet Julia Package - For details, see julia/LICENSE.md + Copyright (c) 2015-2018 by Chiyuan Zhang + 19. Benchdnn - For details, see 3rdparty/mkldnn/tests/benchdnn/README.md + Copyright 2017-2018 Intel Corporation + 20. MXNet perl-package - For details, see perl-package/README + 21. MXNet perl-package AI-MXNET - For details, see perl-package/AI-MXNet/README + 22. MXNet perl-package AI-MXNET Gluon Contrib - For details, see perl-package/AI-MXNet-Gluon-Contrib/README + 23. MXNet perl-package AI-MXNET Gluon ModelZoo - For details, see perl-package/AI-MXNet-Gluon-ModelZoo/README + 24. MXNet perl-package AI-MXNETCAPI - For details, see perl-package/AI-MXNetCAPI/README + 25. MXNet perl-package AI-NNVMCAPI - For details, see perl-package/AI-NNVMCAPI/README + 26. Cephes Library Functions - For details, see src/operator/special_functions-inl.h + Copyright (c) 2015 by Contributors + Copyright 1984, 1987, 1992 by Stephen L. Moshier ======================================================================================= @@ -235,75 +282,111 @@ ======================================================================================= 1. Fast R-CNN - For details, see example/rcnn/LICENSE + Copyright (c) Microsoft Corporation 2. Faster R-CNN - For details, see example/rcnn/LICENSE + Copyright (c) 2015 Microsoft Corporation 3. tree_lstm - For details, see example/gluon/tree_lstm/LICENSE + Copyright (c) 2017 Riddhiman Dasgupta, Sheng Zha 4. OpenMP - For details, see 3rdparty/openmp/LICENSE.txt - 5. HalideIR - For details, see nnvm/tvm/HalideIR/LICENSE + Copyright (c) 1997-2016 Intel Corporation + 6. HalideIR - For details, see 3rdparty/tvm/3rdparty/HalideIR/LICENSE + Copyright (c) 2016 HalideIR contributors + Copyright (c) 2012-2014 MIT CSAIL, Google Inc., and other contributors + Copyright (c) 2016-2018 by Contributors + 7. ONNX-TensorRT - For details, see 3rdparty/onnx-tensorrt/LICENSE + Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. + Copyright (c) 2018 Open Neural Network Exchange + 8. ONNX-TensorRT - For details, see 3rdparty/onnx-tensorrt/third_party/onnx/LICENSE + Copyright (c) Facebook, Inc. and Microsoft Corporation. + 9. clipboard.js - Refer to https://zenorocha.github.io/clipboard.js + Licensed MIT © Zeno Rocha + 10. clipboard.min.js - Refer to https://zenorocha.github.io/clipboard.js + Licensed MIT © Zeno Rocha ======================================================================================= - NVIDIA Licenses + 3-clause BSD licenses ======================================================================================= - 1. Moderngpu - For details, see, src/operator/contrib/ctc_include/contrib/moderngpu/LICENSE - - /****************************************************************************** - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * * Neither the name of the NVIDIA CORPORATION nor the - * names of its contributors may be used to endorse or promote products - * derived from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL NVIDIA CORPORATION BE LIABLE FOR ANY - * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - * - ******************************************************************************/ - - 2. CUB Library - For details, see, 3rdparty/cub/LICENSE.TXT + 1. Xbyak - For details, see 3rdparty/mkldnn/src/cpu/xbyak/COPYRIGHT + Copyright (c) 2007 MITSUNARI Shigeo + Copyright 2016-2018 Intel Corporation + 2. gtest - For details, see, 3rdparty/mkldnn/tests/gtests/gtest/LICENSE + Copyright 2005-2008, Google Inc. + 3. Moderngpu - For details, see, 3rdparty/ctc_include/contrib/moderngpu/LICENSE + Copyright (c) 2013, NVIDIA CORPORATION. All rights reserved. + 4. CUB Library - For details, see, 3rdparty/cub/LICENSE.TXT + Copyright (c) 2010-2011, Duane Merrill. All rights reserved. + Copyright (c) 2011-2016, NVIDIA CORPORATION. All rights reserved. + 5. CUB mersenne.h - For details, see 3rdparty/cub/test/mersenne.h + Copyright (C) 1997 - 2002, Makoto Matsumoto and Takuji Nishimura, + 6. Googlemock - For details, see, 3rdparty/googletest/googlemock/LICENSE + Copyright 2006-2015, Google Inc. + 7. Googletest - For details, see, 3rdparty/googletest/googletest/LICENSE + Copyright 2005-2015, Google Inc. + 8. OpenMP Testsuite - For details, see, 3rdparty/openmp/testsuite/LICENSE + Copyright (c) 2011, 2012 University of Houston System + + + ======================================================================================= + 2-clause BSD licenses + ======================================================================================= + + 1. Sphinx JavaScript utilties for the full-text search - For details, see, docs/_static/searchtools_custom.js + Copyright (c) 2007-2017 by the Sphinx team + 2. blockingconcurrentqueue.h - For details, see, 3rdparty/dmlc-core/include/dmlc/blockingconcurrentqueue.h + ©2015-2016 Cameron Desrochers + 3. concurrentqueue.h - For details, see, 3rdparty/dmlc-core/include/dmlc/concurrentqueue.h + Copyright (c) 2013-2016, Cameron Desrochers. + 4. MSCOCO Toolbox - For details, see, example/ssd/dataset/pycocotools/coco.py + Code written by Piotr Dollar and Tsung-Yi Lin, 2014. + 5. PyBind11 FindEigen3.cmake - For details, see 3rdparty/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/FindEigen3.cmake + Copyright (c) 2006, 2007 Montel Laurent, + Copyright (c) 2008, 2009 Gael Guennebaud, + Copyright (c) 2009 Benoit Jacob + 6. PyBind11 FindPythonLibsNew.cmake - For details, see 3rdparty/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/FindPythonLibsNew.cmake + Copyright 2001-2009 Kitware, Inc. + Copyright 2012 Continuum Analytics, Inc. - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - * Neither the name of the NVIDIA CORPORATION nor the - names of its contributors may be used to endorse or promote products - derived from this software without specific prior written permission. - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL NVIDIA CORPORATION BE LIABLE FOR ANY - DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; - LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND - ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ======================================================================================= Other Licenses ======================================================================================= - 1. Caffe - For details, see, example/rcnn/LICENSE + 1. Caffe - For details, see, example/rcnn/LICENSE + Copyright (c) 2014, 2015, The Regents of the University of California (Regents) + Copyright (c) 2014, 2015, the respective contributors + 2. pool.h - For details, see, src/operator/nn/pool.h + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + 3. pool.cuh - For details, see, src/operator/nn/pool.cuh + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + 4. im2col.h - For details, see, src/operator/nn/im2col.h + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + 5. im2col.cuh - For details, see, src/operator/nn/im2col.cuh + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + + 6. deformable_im2col.h - For details, see, src/operator/contrib/nn/deformable_im2col.h + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + + 7. deformable_im2col.cuh - For details, see, src/operator/contrib/nn/deformable_im2col.cuh + Copyright (c) 2014-2017 The Regents of the University of California (Regents) + Copyright (c) 2014-2017, the respective contributors + + + COPYRIGHT + + Caffe uses a shared copyright model: each contributor holds copyright over + their contributions to Caffe. The project versioning records all such + contribution and copyright details. If a contributor wants to further mark + their specific copyright on a particular contribution, they should indicate + their copyright solely in the commit message of the change when it is + committed. LICENSE @@ -335,8 +418,9 @@ ======================================================================================= - 2. MS COCO API + 8. MS COCO API For details, see, example/rcnn/LICENSE + Copyright (c) 2014, Piotr Dollar and Tsung-Yi Lin Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: @@ -364,155 +448,15 @@ ======================================================================================= - 3. Sphinx JavaScript utilties for the full-text search - For details, see, docs/_static/searchtools_custom.js - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - * Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - ======================================================================================= - - 4. FindCrypto.cmake - For details, see, 3rdparty/dmlc-core/cmake/Modules/FindCrypto.cmake, - Redistribution and use is allowed according to the terms of the BSD license. - - ======================================================================================= - - 5. Googlemock - For details, see, 3rdparty/googletest/googlemock/LICENSE - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following disclaimer - in the documentation and/or other materials provided with the - distribution. - * Neither the name of Google Inc. nor the names of its - contributors may be used to endorse or promote products derived from - this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - ======================================================================================= - - 6. Googletest - For details, see, 3rdparty/googletest/googletest/LICENSE - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are - met: - - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following disclaimer - in the documentation and/or other materials provided with the - distribution. - * Neither the name of Google Inc. nor the names of its - contributors may be used to endorse or promote products derived from - this software without specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT - LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - ======================================================================================= - - 7. OpenMP Testsuite - For details, see, 3rdparty/openmp/testsuite/LICENSE - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions - are met: - - o Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - - o Redistributions in binary form must reproduce the above copyright - notice, this list of conditions and the following disclaimer in the - documentation and/or other materials provided with the distribution. - - o Neither the name of the University of Houston System nor the names of its - contributors may be used to - endorse or promote products derived from this software without specific - prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR - A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT - HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, - SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED - TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF - LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING - NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS - SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. - - ======================================================================================= - - 8. Semaphore implementation in blockingconcurrentqueue.h + 9. Semaphore implementation in blockingconcurrentqueue.h This file uses a semaphore implementation under the terms of its separate zlib license. For details, see, 3rdparty/dmlc-core/include/dmlc/blockingconcurrentqueue.h + Copyright Jeff Preshing ======================================================================================= - 9. blockingconcurrentqueue.h - This file is Distributed under the terms of the simplified BSD license. - For details, see, 3rdparty/dmlc-core/include/dmlc/blockingconcurrentqueue.h - - ======================================================================================= - - 10. concurrentqueue.h - This file is Distributed under the terms of the simplified BSD license. - For details, see, 3rdparty/dmlc-core/include/dmlc/concurrentqueue.h - - ======================================================================================= - - 11. ONNX Export module - For details, see, python/mxnet/contrib/onnx/_export/LICENSE + 10. ONNX Export module + For details, see, python/mxnet/contrib/onnx/mx2onnx/LICENSE # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file @@ -559,4 +503,206 @@ # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + ======================================================================================= + + 11. ONNX python bindings + For details, see, 3rdparty/onnx-tensorrt/third_party/onnx/third_party/pybind11/LICENSE + Copyright (c) 2015-2017 Wenzel Jakob , All rights reserved. + Copyright (c) 2016 Trent Houliston and Wenzel Jakob + Copyright (c) 2016-2017 Jason Rhinelander + Copyright (c) 2016 Klemens Morgenstern and Wenzel Jakob + Copyright (c) 2017 Henry F. Schreiner + Copyright (c) 2016 Sergey Lyskov and Wenzel Jakob + Copyright (c) 2016 Ben North + Copyright (c) 2016 Klemens D. Morgenstern + Copyright (c) 2016 Pim Schellart + Copyright (c) 2016 Ivan Smirnov + Copyright (c) 2016 Sergey Lyskov + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are met: + + 1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + + 2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + + 3. Neither the name of the copyright holder nor the names of its contributors + may be used to endorse or promote products derived from this software + without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND + ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED + WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE + DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE + FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + You are under no obligation whatsoever to provide any bug fixes, patches, or + upgrades to the features, functionality or performance of the source code + ("Enhancements") to anyone; however, if you choose to make your Enhancements + available either publicly, or directly to the author of this software, without + imposing a separate written license agreement for such Enhancements, then you + hereby grant the following license: a non-exclusive, royalty-free perpetual + license to install, use, modify, prepare derivative works, incorporate into + other computer software, distribute, and sublicense such enhancements or + derivative works thereof, in binary and source code form. + + ======================================================================================= + + 12. Clang + For details, see, 3rdparty/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang/LICENSE.TXT + + LLVM Release License + University of Illinois/NCSA + Open Source License + + Copyright (c) 2007-2012 University of Illinois at Urbana-Champaign. + All rights reserved. + + Developed by: + + LLVM Team + + University of Illinois at Urbana-Champaign + + http://llvm.org + + Permission is hereby granted, free of charge, to any person obtaining a copy of + this software and associated documentation files (the "Software"), to deal with + the Software without restriction, including without limitation the rights to + use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies + of the Software, and to permit persons to whom the Software is furnished to do + so, subject to the following conditions: + + * Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimers. + + * Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimers in the + documentation and/or other materials provided with the distribution. + + * Neither the names of the LLVM Team, University of Illinois at + Urbana-Champaign, nor the names of its contributors may be used to + endorse or promote products derived from this Software without specific + prior written permission. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS + FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE + SOFTWARE. + + The LLVM software contains code written by third parties. Such software will + have its own individual LICENSE.TXT file in the directory in which it appears. + This file will describe the copyrights, license, and restrictions which apply + to that code. + + The disclaimer of warranty in the University of Illinois Open Source License + applies to all code in the LLVM Distribution, and nothing in any of the + other licenses gives permission to use the names of the LLVM Team or the + University of Illinois to endorse or promote products derived from this + Software. + + The following pieces of software have additional or alternate copyrights, + licenses, and/or restrictions: + + Program Directory + ------- --------- + + + ======================================================================================= + + 13. MKL BLAS + For details, see, [Intel® Simplified license](https://software.intel.com/en-us/license/intel-simplified-software-license) and MKLDNN_README.md + + Copyright (c) 2018 Intel Corporation. + + Use and Redistribution. You may use and redistribute the software (the “Software”), without modification, provided the following conditions are met: + + * Redistributions must reproduce the above copyright notice and the following terms of use in the Software and in the documentation and/or other materials provided with the distribution. + + * Neither the name of Intel nor the names of its suppliers may be used to endorse or promote products derived from this Software without specific prior written permission. + + * No reverse engineering, decompilation, or disassembly of this Software is permitted. + + Limited patent license. Intel grants you a world-wide, royalty-free, non-exclusive license under patents it now or hereafter owns or controls to make, have made, use, import, offer to sell and sell (“Utilize”) this Software, but solely to the extent that any such patent is necessary to Utilize the Software alone. The patent license shall not apply to any combinations which include this software. No hardware per se is licensed hereunder. + + Third party and other Intel programs. “Third Party Programs” are the files listed in the “third-party-programs.txt” text file that is included with the Software and may include Intel programs under separate license terms. Third Party Programs, even if included with the distribution of the Materials, are governed by separate license terms and those license terms solely govern your use of those programs. + + DISCLAIMER. THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE DISCLAIMED. THIS SOFTWARE IS NOT INTENDED FOR USE IN SYSTEMS OR APPLICATIONS WHERE FAILURE OF THE SOFTWARE MAY CAUSE PERSONAL INJURY OR DEATH AND YOU AGREE THAT YOU ARE FULLY RESPONSIBLE FOR ANY CLAIMS, COSTS, DAMAGES, EXPENSES, AND ATTORNEYS’ FEES ARISING OUT OF ANY SUCH USE, EVEN IF ANY CLAIM ALLEGES THAT INTEL WAS NEGLIGENT REGARDING THE DESIGN OR MANUFACTURE OF THE MATERIALS. + + LIMITATION OF LIABILITY. IN NO EVENT WILL INTEL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. YOU AGREE TO INDEMNIFY AND HOLD INTEL HARMLESS AGAINST ANY CLAIMS AND EXPENSES RESULTING FROM YOUR USE OR UNAUTHORIZED USE OF THE SOFTWARE. + + No support. Intel may make changes to the Software, at any time without notice, and is not obligated to support, update or provide training for the Software. + + Termination. Intel may terminate your right to use the Software in the event of your breach of this Agreement and you fail to cure the breach within a reasonable period of time. + + Feedback. Should you provide Intel with comments, modifications, corrections, enhancements or other input (“Feedback”) related to the Software Intel will be free to use, disclose, reproduce, license or otherwise distribute or exploit the Feedback in its sole discretion without any obligations or restrictions of any kind, including without limitation, intellectual property rights or licensing obligations. + + Compliance with laws. You agree to comply with all relevant laws and regulations governing your use, transfer, import or export (or prohibition thereof) of the Software. + + Governing law. All disputes will be governed by the laws of the United States of America and the State of Delaware without reference to conflict of law principles and subject to the exclusive jurisdiction of the state or federal courts sitting in the State of Delaware, and each party agrees that it submits to the personal jurisdiction and venue of those courts and waives any objections. The United Nations Convention on Contracts for the International Sale of Goods (1980) is specifically excluded and will not apply to the Software. + + *Other names and brands may be claimed as the property of others. + + ======================================================================================= + 14. FindJeMalloc.cmake + For details, see, cmake/Modules/FindJeMalloc.cmake + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + + Copyright (c) 2014 Thomas Heller + Copyright (c) 2007-2012 Hartmut Kaiser + Copyright (c) 2010-2011 Matt Anderson + Copyright (c) 2011 Bryce Lelbach + + Distributed under the Boost Software License, Version 1.0. + Boost Software License - Version 1.0 - August 17th, 2003 + + Permission is hereby granted, free of charge, to any person or organization + obtaining a copy of the software and accompanying documentation covered by + this license (the "Software") to use, reproduce, display, distribute, + execute, and transmit the Software, and to prepare derivative works of the + Software, and to permit third-parties to whom the Software is furnished to + do so, all subject to the following: + + The copyright notices in the Software and this entire statement, including + the above license grant, this restriction and the following disclaimer, + must be included in all copies of the Software, in whole or in part, and + all derivative works of the Software, unless such copies or derivative + works are solely in the form of machine-executable object code generated by + a source language processor. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT + SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE + FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, + ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + DEALINGS IN THE SOFTWARE. diff --git a/MKLDNN_README.md b/MKLDNN_README.md index 2618d2338..ecb721e5f 100644 --- a/MKLDNN_README.md +++ b/MKLDNN_README.md @@ -1,9 +1,9 @@ # Build/Install MXNet with MKL-DNN -A better training and inference perforamce are expected to achieved on Intel-Architecture CPUs with MXNET built with [Intel MKL-DNN](https://github.com/intel/mkl-dnn) on multiple operating system, including Linux, Windows and MacOS. -In the following sections, you will find building instructions for MXNET with Intel MKL-DNN on Linux, MacOS and Windows. +A better training and inference performance is expected to be achieved on Intel-Architecture CPUs with MXNet built with [Intel MKL-DNN](https://github.com/intel/mkl-dnn) on multiple operating system, including Linux, Windows and MacOS. +In the following sections, you will find build instructions for MXNet with Intel MKL-DNN on Linux, MacOS and Windows. -The detailed performance data collected on Intel Xeon CPU with MXNET built with Intel MKL-DNN can be found at [here](https://mxnet.incubator.apache.org/faq/perf.html#intel-cpu). +The detailed performance data collected on Intel Xeon CPU with MXNet built with Intel MKL-DNN can be found [here](https://mxnet.incubator.apache.org/faq/perf.html#intel-cpu).

Contents

@@ -78,12 +78,12 @@ cd incubator-mxnet ### Build MXNet with MKL-DNN ``` -LIBRARY_PATH=$(brew --prefix llvm)/lib/ make -j $(sysctl -n hw.ncpu) CC=$(brew --prefix llvm)/bin/clang++ CXX=$(brew --prefix llvm)/bin/clang++ USE_OPENCV=1 USE_OPENMP=1 USE_MKLDNN=1 USE_BLAS=apple USE_PROFILER=1 +LIBRARY_PATH=$(brew --prefix llvm)/lib/ make -j $(sysctl -n hw.ncpu) CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ USE_OPENCV=1 USE_OPENMP=1 USE_MKLDNN=1 USE_BLAS=apple USE_PROFILER=1 ```

Windows

-On Windows, you can use [Micrsoft Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) and [Microsoft Visual Studio 2017](https://www.visualstudio.com/downloads/) to compile MXNET with Intel MKL-DNN. +On Windows, you can use [Micrsoft Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) and [Microsoft Visual Studio 2017](https://www.visualstudio.com/downloads/) to compile MXNet with Intel MKL-DNN. [Micrsoft Visual Studio 2015](https://www.visualstudio.com/vs/older-downloads/) is recommended. **Visual Studio 2015** @@ -123,7 +123,7 @@ cmake -G "Visual Studio 14 Win64" .. -DUSE_CUDA=0 -DUSE_CUDNN=0 -DUSE_NVRTC=0 -D These commands produce a library called ```libmxnet.dll``` in the ```./build/Release/``` or ```./build/Debug``` folder. Also ```libmkldnn.dll``` with be in the ```./build/3rdparty/mkldnn/src/Release/``` -6. Make sure that all the dll files used above(such as `libmkldnn.dll`, `libmklml.dll`, `libiomp5.dll`, `libopenblas.dll`, etc) are added to the system PATH. For convinence, you can put all of them to ```\windows\system32```. Or you will come across `Not Found Dependencies` when loading mxnet. +6. Make sure that all the dll files used above(such as `libmkldnn.dll`, `libmklml.dll`, `libiomp5.dll`, `libopenblas.dll`, etc) are added to the system PATH. For convinence, you can put all of them to ```\windows\system32```. Or you will come across `Not Found Dependencies` when loading MXNet. **Visual Studio 2017** @@ -177,7 +177,7 @@ cmake -G "Visual Studio 15 2017 Win64" .. -T host=x64 -DUSE_CUDA=0 -DUSE_CUDNN=0 msbuild mxnet.sln /p:Configuration=Release;Platform=x64 /maxcpucount ``` -9. Make sure that all the dll files used above(such as `libmkldnn.dll`, `libmklml.dll`, `libiomp5.dll`, `libopenblas.dll`, etc) are added to the system PATH. For convinence, you can put all of them to ```\windows\system32```. Or you will come across `Not Found Dependencies` when loading mxnet. +9. Make sure that all the dll files used above(such as `libmkldnn.dll`, `libmklml.dll`, `libiomp5.dll`, `libopenblas.dll`, etc) are added to the system PATH. For convinence, you can put all of them to ```\windows\system32```. Or you will come across `Not Found Dependencies` when loading MXNet.

Verify MXNet with python

diff --git a/MXNET_README.md b/MXNET_README.md index 23b9d329d..369df9b64 100644 --- a/MXNET_README.md +++ b/MXNET_README.md @@ -33,6 +33,7 @@ How to Contribute What's New ---------- +* [Version 1.3.1 Release](https://github.com/apache/incubator-mxnet/releases/tag/1.3.1) - MXNet 1.3.1 Patch Release. * [Version 1.3.0 Release](https://github.com/apache/incubator-mxnet/releases/tag/1.3.0) - MXNet 1.3.0 Release. * [Version 1.2.0 Release](https://github.com/apache/incubator-mxnet/releases/tag/1.2.0) - MXNet 1.2.0 Release. * [Version 1.1.0 Release](https://github.com/apache/incubator-mxnet/releases/tag/1.1.0) - MXNet 1.1.0 Release. diff --git a/Makefile b/Makefile index 50639cebf..cf05a0035 100644 --- a/Makefile +++ b/Makefile @@ -23,16 +23,14 @@ print-%: @echo '$*=$($*)' .DEFAULT_GOAL := all - ROOTDIR = $(CURDIR) TPARTYDIR = $(ROOTDIR)/3rdparty -SCALA_VERSION_PROFILE := scala-2.11 - ifeq ($(OS),Windows_NT) UNAME_S := Windows else UNAME_S := $(shell uname -s) + UNAME_P := $(shell uname -p) endif ifndef config @@ -69,6 +67,15 @@ endif # use customized config file include $(config) +ifndef USE_MKLDNN +ifneq ($(UNAME_S), Darwin) +ifneq ($(UNAME_S), Windows) +ifeq ($(UNAME_P), x86_64) + USE_MKLDNN=1 +endif +endif +endif +endif ifeq ($(USE_MKL2017), 1) $(warning "USE_MKL2017 is deprecated. We will switch to USE_MKLDNN.") USE_MKLDNN=1 @@ -80,12 +87,10 @@ ifeq ($(USE_MKLDNN), 1) export USE_MKLML = 1 endif - include $(TPARTYDIR)/mshadow/make/mshadow.mk include $(DMLC_CORE)/make/dmlc.mk include 3rdparty/ngraph-mxnet-bridge/ngraph.mk - # all tge possible warning tread WARNFLAGS= -Wall -Wsign-compare -Wno-comment CFLAGS = -DMSHADOW_FORCE_STREAM $(WARNFLAGS) @@ -123,6 +128,7 @@ ifeq ($(USE_TENSORRT), 1) CFLAGS += -I$(ROOTDIR) -I$(TPARTYDIR) -DONNX_NAMESPACE=$(ONNX_NAMESPACE) -DMXNET_USE_TENSORRT=1 LDFLAGS += -lprotobuf -pthread -lonnx -lonnx_proto -lnvonnxparser -lnvonnxparser_runtime -lnvinfer -lnvinfer_plugin endif +# -L/usr/local/lib ifeq ($(DEBUG), 1) NVCCFLAGS += -std=c++11 -Xcompiler -D_FORCE_INLINES -g -G -O0 -ccbin $(CXX) $(MSHADOW_NVCCFLAGS) @@ -325,7 +331,6 @@ ifeq ($(USE_ASAN), 1) CFLAGS += -fsanitize=address -fno-omit-frame-pointer LDFLAGS += -lasan endif - ifneq ($(ADD_CFLAGS), NONE) CFLAGS += $(ADD_CFLAGS) endif @@ -367,7 +372,7 @@ endif # be JIT-compiled by the updated driver from the included PTX. ifeq ($(USE_CUDA), 1) ifeq ($(CUDA_ARCH),) - KNOWN_CUDA_ARCHS := 30 35 50 52 60 61 70 + KNOWN_CUDA_ARCHS := 30 35 50 52 60 61 70 75 # Run nvcc on a zero-length file to check architecture-level support. # Create args to include SASS in the fat binary for supported levels. CUDA_ARCH := $(foreach arch,$(KNOWN_CUDA_ARCHS), \ @@ -427,18 +432,13 @@ PLUGIN_OBJ = PLUGIN_CUOBJ = include $(MXNET_PLUGINS) -ifeq ($(UNAME_S), Windows) - # TODO(yizhi) currently scala package does not support windows - SCALA_PKG_PROFILE := windows -else +ifneq ($(UNAME_S), Windows) ifeq ($(UNAME_S), Darwin) WHOLE_ARCH= -all_load NO_WHOLE_ARCH= -noall_load - SCALA_PKG_PROFILE := osx-x86_64 else WHOLE_ARCH= --whole-archive NO_WHOLE_ARCH= --no-whole-archive - SCALA_PKG_PROFILE := linux-x86_64 endif endif @@ -457,7 +457,6 @@ ifeq ($(USE_CUDA), 1) # Make sure to add stubs as fallback in order to be able to build # without full CUDA install (especially if run without nvidia-docker) LDFLAGS += -L/usr/local/cuda/lib64/stubs - SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-gpu ifeq ($(USE_NCCL), 1) ifneq ($(USE_NCCL_PATH), NONE) CFLAGS += -I$(USE_NCCL_PATH)/include @@ -469,7 +468,6 @@ ifeq ($(USE_CUDA), 1) CFLAGS += -DMXNET_USE_NCCL=0 endif else - SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-cpu CFLAGS += -DMXNET_USE_NCCL=0 endif @@ -484,6 +482,9 @@ else CFLAGS += -DMXNET_USE_LIBJPEG_TURBO=0 endif +ifeq ($(CI), 1) + MAVEN_ARGS := -B +endif # For quick compile test, used smaller subset ALLX_DEP= $(ALL_DEP) @@ -509,13 +510,17 @@ build/plugin/%.o: plugin/%.cc | ngraph %_gpu.o: %.cu @mkdir -p $(@D) - $(NVCC) $(NVCCFLAGS) $(CUDA_ARCH) -Xcompiler "$(CFLAGS) -Isrc/operator" -M -MT $*_gpu.o $< >$*_gpu.d + $(NVCC) $(NVCCFLAGS) $(CUDA_ARCH) -Xcompiler "$(CFLAGS) -Isrc/operator" --generate-dependencies -MT $*_gpu.o $< >$*_gpu.d $(NVCC) -c -o $@ $(NVCCFLAGS) $(CUDA_ARCH) -Xcompiler "$(CFLAGS) -Isrc/operator" $< %.o: %.cc $(CORE_INC) @mkdir -p $(@D) $(CXX) -std=c++11 -c $(CFLAGS) -MMD -Isrc/operator -c $< -o $@ +# Set install path for libmxnet.so on Mac OS +ifeq ($(UNAME_S), Darwin) + LDFLAGS += -Wl,-install_name,@rpath/libmxnet.so +endif # NOTE: to statically link libmxnet.a we need the option # --Wl,--whole-archive -lmxnet --Wl,--no-whole-archive lib/libmxnet.a: $(ALLX_DEP) @@ -554,7 +559,6 @@ $(NNVM_PATH)/lib/libnnvm.a: $(NNVM_INC) $(NNVM_SRC) bin/im2rec: tools/im2rec.cc $(ALLX_DEP) MXNET_RELATIVE_PATH_TO_RUNTIME_LIB_DIR := "../lib" - $(BIN) : @mkdir -p $(@D) $(CXX) $(CFLAGS) -std=c++11 -o $@ $(filter %.cpp %.o %.c %.a %.cc, $^) \ @@ -585,7 +589,6 @@ pylint: python_clean: $(RM) -r python/build $(RM) -r python/dist - doc: docs docs: @@ -620,87 +623,50 @@ rpkg: cp -rf lib/libmxnet.so R-package/inst/libs mkdir -p R-package/inst/include cp -rf include/* R-package/inst/include + rm R-package/inst/include/dmlc + rm R-package/inst/include/nnvm cp -rf 3rdparty/dmlc-core/include/* R-package/inst/include/ cp -rf 3rdparty/tvm/nnvm/include/* R-package/inst/include Rscript -e "if(!require(devtools)){install.packages('devtools', repo = 'https://cloud.r-project.org/')}" + Rscript -e "if(!require(devtools)||packageVersion('roxygen2') < '6.1.1'){install.packages('roxygen2', repo = 'https://cloud.r-project.org/')}" Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/')); install_deps(pkg='R-package', dependencies = TRUE)" - echo "import(Rcpp)" > R-package/NAMESPACE - echo "import(methods)" >> R-package/NAMESPACE + cp R-package/dummy.NAMESPACE R-package/NAMESPACE + echo "import(Rcpp)" >> R-package/NAMESPACE R CMD INSTALL R-package - Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package')" - Rscript -e "if (!require('roxygen2')||packageVersion('roxygen2')!= '5.0.1'){\ - devtools::install_version('roxygen2',version='5.0.1',\ - repo='https://cloud.r-project.org/',quiet=TRUE)}" - Rscript -e "require(roxygen2); roxygen2::roxygenise('R-package')" + Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package'); warnings()" + rm R-package/NAMESPACE + Rscript -e "devtools::document('R-package'); warnings()" R CMD INSTALL R-package rpkgtest: Rscript -e 'require(testthat);res<-test_dir("R-package/tests/testthat");if(!testthat:::all_passed(res)){stop("Test failures", call. = FALSE)}' - Rscript -e 'res<-covr:::package_coverage("R-package");fileConn<-file("r-package_coverage.json");writeLines(covr:::to_codecov(res), fileConn);close(fileConn)' + Rscript -e 'res<-covr:::package_coverage("R-package");fileConn<-file(paste("r-package_coverage_",toString(runif(1)),".json"));writeLines(covr:::to_codecov(res), fileConn);close(fileConn)' scalaclean: - (cd $(ROOTDIR)/scala-package; \ - mvn clean -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE)) + (cd $(ROOTDIR)/scala-package && mvn clean) scalapkg: - (cd $(ROOTDIR)/scala-package; \ - mvn package -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) -Dcxx="$(CXX)" \ - -Dbuild.platform="$(SCALA_PKG_PROFILE)" \ - -Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \ - -Dcurrent_libdir="$(ROOTDIR)/lib" \ - -Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a") + (cd $(ROOTDIR)/scala-package && mvn install -DskipTests) + +scalainstall: + (cd $(ROOTDIR)/scala-package && mvn install) scalaunittest: - (cd $(ROOTDIR)/scala-package; \ - mvn integration-test -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE),unittest -Dcxx="$(CXX)" \ - -Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \ - -Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a" $(SCALA_TEST_ARGS)) + (cd $(ROOTDIR)/scala-package && mvn install) scalaintegrationtest: - (cd $(ROOTDIR)/scala-package; \ - mvn integration-test -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE),integrationtest -Dcxx="$(CXX)" \ - -Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \ - -Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a" $(SCALA_TEST_ARGS)) - -scalainstall: - (cd $(ROOTDIR)/scala-package; \ - mvn install -P$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) -DskipTests=true -Dcxx="$(CXX)" \ - -Dbuild.platform="$(SCALA_PKG_PROFILE)" \ - -Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \ - -Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a") - -scalarelease-dryrun: - (cd $(ROOTDIR)/scala-package; \ - mvn release:clean release:prepare -DdryRun=true -DautoVersionSubmodules=true \ - -Papache-release,$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) \ - -Darguments=""-Dbuild\.platform=\""$(SCALA_PKG_PROFILE)\""\ -DskipTests=true\ -Dcflags=\""$(CFLAGS)\""\ -Dcxx=\""$(CXX)\""\ -Dldflags=\""$(LDFLAGS)\""\ -Dlddeps=\""$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a\"""") - -scalarelease-prepare: - (cd $(ROOTDIR)/scala-package; \ - mvn release:clean release:prepare -DautoVersionSubmodules=true \ - -Papache-release,$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) \ - -Darguments=""-Dbuild\.platform=\""$(SCALA_PKG_PROFILE)\""\ -DskipTests=true\ -Dcflags=\""$(CFLAGS)\""\ -Dcxx=\""$(CXX)\""\ -Dldflags=\""$(LDFLAGS)\""\ -Dlddeps=\""$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a\"""") - -scalarelease-perform: - (cd $(ROOTDIR)/scala-package; \ - mvn release:perform -DautoVersionSubmodules=true \ - -Papache-release,$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) \ - -Darguments=""-Dbuild\.platform=\""$(SCALA_PKG_PROFILE)\""\ -DskipTests=true\ -Dcflags=\""$(CFLAGS)\""\ -Dcxx=\""$(CXX)\""\ -Dldflags=\""$(LDFLAGS)\""\ -Dlddeps=\""$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a\"""") - -scaladeploy: - (cd $(ROOTDIR)/scala-package; \ - mvn deploy -Papache-release,$(SCALA_PKG_PROFILE),$(SCALA_VERSION_PROFILE) \-DskipTests=true -Dcxx="$(CXX)" \ - -Dbuild.platform="$(SCALA_PKG_PROFILE)" \ - -Dcflags="$(CFLAGS)" -Dldflags="$(LDFLAGS)" \ - -Dlddeps="$(LIB_DEP) $(ROOTDIR)/lib/libmxnet.a") + (cd $(ROOTDIR)/scala-package && mvn integration-test -DskipTests=false) jnilint: - 3rdparty/dmlc-core/scripts/lint.py mxnet-jnicpp cpp scala-package/native/src + 3rdparty/dmlc-core/scripts/lint.py mxnet-jnicpp cpp scala-package/native/src --exclude_path scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h -ifneq ($(EXTRA_OPERATORS),) -clean: cyclean $(EXTRA_PACKAGES_CLEAN) - $(RM) -r build lib bin *~ */*~ */*/*~ */*/*/*~ R-package/NAMESPACE R-package/man R-package/R/mxnet_generated.R \ +rclean: + $(RM) -r R-package/src/image_recordio.h R-package/NAMESPACE R-package/man R-package/R/mxnet_generated.R \ R-package/inst R-package/src/*.o R-package/src/*.so mxnet_*.tar.gz + +ifneq ($(EXTRA_OPERATORS),) +clean: rclean cyclean $(EXTRA_PACKAGES_CLEAN) + $(RM) -r build lib bin deps *~ */*~ */*/*~ */*/*/*~ cd $(DMLC_CORE); $(MAKE) clean; cd - cd $(PS_PATH); $(MAKE) clean; cd - cd $(NNVM_PATH); $(MAKE) clean; cd - @@ -708,7 +674,7 @@ clean: cyclean $(EXTRA_PACKAGES_CLEAN) $(RM) -r $(patsubst %, %/*.d, $(EXTRA_OPERATORS)) $(patsubst %, %/*/*.d, $(EXTRA_OPERATORS)) $(RM) -r $(patsubst %, %/*.o, $(EXTRA_OPERATORS)) $(patsubst %, %/*/*.o, $(EXTRA_OPERATORS)) else -clean: ngraph_clean mkldnn_clean cyclean testclean $(EXTRA_PACKAGES_CLEAN) +clean: ngraph_clean rclean mkldnn_clean cyclean testclean $(EXTRA_PACKAGES_CLEAN) $(RM) -r build lib bin *~ */*~ */*/*~ */*/*/*~ R-package/NAMESPACE R-package/man R-package/R/mxnet_generated.R \ R-package/inst R-package/src/image_recordio.h R-package/src/*.o R-package/src/*.so mxnet_*.tar.gz cd $(DMLC_CORE); $(MAKE) clean; cd - diff --git a/NEWS.md b/NEWS.md index 4dcf8bf0c..f06cc35d8 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,5 +1,675 @@ MXNet Change Log ================ + +## 1.4.0 + +- [New Features](#new-features) + * [Java Inference API](#java-inference-api) + * [Julia API](#julia-api) + * [Control Flow Operators (experimental)](#control-flow-operators--experimental-) + * [SVRG Optimization](#svrg-optimization) + * [Subgraph API (experimental)](#subgraph-api--experimental-) + * [JVM Memory Management](#jvm-memory-management) + * [Topology-aware AllReduce (experimental)](#topology-aware-allreduce--experimental-) + * [MKLDNN backend: Graph optimization and Quantization (experimental)](#mkldnn-backend--graph-optimization-and-quantization--experimental-) + + [Graph Optimization](#graph-optimization) + + [Quantization](#quantization) +- [New Operators](#new-operators) +- [Feature improvements](#feature-improvements) + * [Operator](#operator) + * [Optimizer](#optimizer) + * [Sparse](#sparse) + * [ONNX](#onnx) + * [MKLDNN](#mkldnn) + * [Inference](#inference) + * [Other](#other) +- [Frontend API updates](#frontend-api-updates) + * [Gluon](#gluon) + * [Symbol](#symbol) +- [Language API updates](#language-api-updates) + * [Java](#java) + * [R](#r) + * [Scala](#scala) + * [Clojure](#clojure) + * [Perl](#perl) + * [Julia](#julia) +- [Performance benchmarks and improvements](#performance-benchmarks-and-improvements) +- [Bug fixes](#bug-fixes) +- [Licensing updates](#licensing-updates) +- [Improvements](#improvements) + * [Tutorial](#tutorial) + * [Example](#example) + * [Documentation](#documentation) + * [Website](#website) + * [MXNet Distributions](#mxnet-distributions) + * [Installation](#installation) + * [Build and CI](#build-and-ci) + * [3rd party](#3rd-party) + + [TVM:](#tvm-) + + [CUDNN:](#cudnn-) + + [Horovod:](#horovod-) +- [Deprications](#deprications) +- [Other](#other-1) +- [How to build MXNet](#how-to-build-mxnet) +- [List of submodules used by Apache MXNet (Incubating) and when they were updated last](#list-of-submodules-used-by-apache-mxnet--incubating--and-when-they-were-updated-last) +### New Features +#### Java Inference API + +Model inference is often managed in a production ecosystem using primarily Java/Scala tools and frameworks. This release seeks to alleviate the need for software engineers to write custom MXNet wrappers to fit their production environment. + +Inference on a trained model has a couple of common use cases: + + 1. Real-time or Online Inference - tasks that require immediate feedback, such as fraud detection + 2. Batch or Offline Inference - tasks that don't require immediate feedback, these are use cases where you have massive amounts of data and want to run inference or pre-compute inference results +Real-time Inference is often performed and deployed on popular web frameworks such as Tomcat, Netty, Jetty, etc., all of which use Java. +Batch Inference is often performed on big data platforms such as Spark using Scala or Java. + +With this project, we had the following goals: +* Build a new set of APIs that are Java friendly, compatible with Java 7+, are easy to use for inference. +* Lower the barrier to entry of consuming MXNet for production use cases. + +More details can be found at the [Java Inference API document](https://cwiki.apache.org/confluence/display/MXNET/MXNet+Java+Inference+API). + +#### Julia API + +MXNet.jl is the Julia package of Apache MXNet. MXNet.jl brings flexible and efficient GPU computing and state-of-art deep learning to Julia. Some highlights of features include: + + * Efficient tensor/matrix computation across multiple devices, including multiple CPUs, GPUs and distributed server nodes. + * Flexible manipulation of symbolic to composite for construction of state-of-the-art deep learning models. + +#### Control Flow Operators (experimental) + +Today we observe more and more dynamic neural network models, especially in the fields of natural language processing and graph analysis. The dynamics in these models come from multiple sources, including: + + * Models are expressed with control flow, such as conditions and loops; + * NDArrays in a model may have dynamic shapes, meaning the NDArrays of a model or some of the NDArrays have different shapes for different batches; + * Models may want to use more dynamic data structures, such as lists or dictionaries. +It's natural to express dynamic models in frameworks with an imperative programming interface (e.g., Gluon, Pytorch, TensorFlow Eager). In this kind of interface, developers can use Python control flows, or NDArrays with any shape at any moment, or use Python lists and dictionaries to store data as they want. The problem of this approach is that it highly dependent on the originating front-end programming language (mainly Python). A model implemented in one language can only run in the same language. + +A common use case is that machine learning scientists want to develop their models in Python, whereas engineers who deploy the models usually have to use a different "production" language (e.g., Java or C). Gluon tries to close the gap between the model development and production deployment. Machine learning scientists design and implement their models in Python with the imperative interface, and then Gluon converts the implementations from imperative to symbolic by invoking `hybridize()` for model exporting. + +The goal of this project is to enhance Gluon to turn a dynamic neural network into a static computation graph. The dynamic control flows are expressed by control flow operators with Gluon hybridization, and these are exported for deployment. + +More information can be found at [Optimize dynamic neural network models with control flow operators](https://cwiki.apache.org/confluence/display/MXNET/Optimize+dynamic+neural+network+models+with+control+flow+operators) + +#### SVRG Optimization + +SVRG stands for Stochastic Variance Reduced Gradient, which was first introduced in the paper [Accelerating Stochastic Gradient Descent using Predicative Variance Reduction in 2013](https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf). It is an optimization technique that complements SGD. + +SGD is known for large scale optimization, but it suffers from slow convergence asymptotically due to the inherent variance. SGD approximates the full gradient using a small batch of samples which introduces variance. In order to converge faster, SGD often needs to start with a smaller learning rate. + +SVRG remedies the slow convergence problem by keeping a version of the estimated weights that is close to the optimal parameters and maintains the average of the full gradient over the full pass of data. The average of the full gradients of all data is calculated w.r.t to parameters of last mth epochs. It has provable guarantees for strongly convex smooth functions; a detailed proof can be found in section 3 of the [paper](https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf). SVRG uses a different update rule than SGD: gradients w.r.t current parameters minus gradients w.r.t parameters from the last mth epoch, plus the average of gradients over all data. + +Key Characteristics of SVRG: + + * Explicit variance reduction + * Ability to use relatively large learning rate compared to SGD, which leads to faster convergence. +More details can be found at [SVRG Optimization in MXNet Python Module](https://cwiki.apache.org/confluence/display/MXNET/Unified+integration+with+external+backend+libraries) + +#### Subgraph API (experimental) + +MXNet can integrate with many different kinds of backend libraries, including TVM, MKLDNN, TensorRT, Intel nGraph and more. In general, these backends support a limited number of operators, so running computation in a model usually involves an interaction between backend-supported operators and MXNet operators. These backend libraries share some common requirements: + +TVM , MKLDNN and nGraph use customized data formats. Interaction between these backends with MXNet requires data format conversion. +TVM, MKLDNN, TensorRT and nGraph fuses operators. +Integration with these backends should happen in the granularity of subgraphs instead of in the granularity of operators. To fuse operators, it's obvious that we need to divide a graph into subgraphs so that the operators in a subgraph can be fused into a single operator. To handle customized data formats, we should partition a computation graph into subgraphs as well. Each subgraph contains only TVM, MKLDNN or nGraph operators. In this way, MXNet converts data formats only when entering such a subgraph, and the operators inside a subgraph handle format conversion themselves if necessary. This makes interaction of TVM and MKLDNN with MXNet much easier. Neither the MXNet executor nor the MXNet operators need to deal with customized data formats. Even though invoking these libraries from MXNet requires similar steps, the partitioning rule and the subgraph execution of these backends can be different. As such, we define the following interface for backends to customize graph partitioning and subgraph execution inside an operator. More details can be found at PR 12157 and [Subgraph API](https://cwiki.apache.org/confluence/display/MXNET/Unified+integration+with+external+backend+libraries). + +#### JVM Memory Management + +The MXNet Scala and Java API uses native memory to manage NDArray, Symbol, Executor, DataIterators using MXNet's internal C APIs. The C APIs provide appropriate interfaces to create, access and free these objects. MXNet Scala has corresponding Wrappers and APIs that have pointer references to the native memory. Before this project, JVM users (e.g. Scala, Clojure, or Java) of MXNet have to manage MXNet objects manually using the dispose pattern. There are a few usability problems with this approach: + +* Users have to track the MXNet objects manually and remember to call `dispose`. This is not Java idiomatic and not user friendly. Quoting a user: "this feels like I am writing C++ code which I stopped ages ago". +* Leads to memory leaks if `dispose` is not called. +* Many objects in MXNet-Scala are managed in native memory, needing to use `dispose` on them as well. +* Bloated code with `dispose()` methods. +* Hard to debug memory-leaks. +Goals of the project are: +* Provide MXNet JVM users automated memory management that can release native memory when there are no references to JVM objects. +* Provide automated memory management for both GPU and CPU memory without performance degradation. More details can be found here: [JVM Memory Management](https://cwiki.apache.org/confluence/display/MXNET/JVM+Memory+Management) + +#### Topology-aware AllReduce (experimental) +For distributed training, the `Reduce` communication patterns used by NCCL and MXNet are not optimal for small batch sizes. The `Topology-aware AllReduce` approach is based on the idea of using trees to perform the `Reduce` and `Broadcast` operations. We can use the idea of minimum spanning trees to do a binary tree `Reduce` communication pattern to improve distributed training following this paper by Wang, Li, Edo and Smola [1]. Our strategy is to use: + + * a single tree (latency-optimal for small messages) to handle `Reduce` on small messages + * multiple trees (bandwidth-optimal for large messages) to handle `Reduce` on large messages + +More details can be found here: [Topology-aware AllReduce](https://cwiki.apache.org/confluence/display/MXNET/Single+machine+All+Reduce+Topology-aware+Communication) +Note: This is an experimental feature and has known problems - see [13341](https://github.com/apache/incubator-mxnet/issues/13341). Please help to contribute to improve the robustness of the feature. + +#### MKLDNN backend: Graph optimization and Quantization (experimental) + +Two advanced features, graph optimization (operator fusion) and reduced-precision (INT8) computation, are introduced to MKLDNN backend in this release ([#12530](https://github.com/apache/incubator-mxnet/pull/12530), [#13297](https://github.com/apache/incubator-mxnet/pull/13297), [#13260](https://github.com/apache/incubator-mxnet/pull/13260)). +These features significantly boost the inference performance on CPU (up to 4X) for a broad range of deep learning topologies. Currently, this feature is only available for inference on platforms with [supported Intel CPUs](https://github.com/intel/mkl-dnn#system-requirements). + +##### Graph Optimization +MKLDNN backend takes advantage of MXNet subgraph to implement the most of possible operator fusions for inference, such as Convolution + ReLU, Batch Normalization folding, etc. When using mxnet-mkl package, users can easily enable this feature by setting export MXNET_SUBGRAPH_BACKEND=MKLDNN. + +##### Quantization +Performance of reduced-precision (INT8) computation is also dramatically improved after the graph optimization feature is applied on CPU Platforms. Various models are supported and can benefit from reduced-precision computation, including symbolic models, Gluon models and even custom models. Users can run most of the pre-trained models with only a few lines of commands and a new quantization script imagenet_gen_qsym_mkldnn.py. The observed accuracy loss is less than 0.5% for popular CNN networks, like ResNet-50, Inception-BN, MobileNet, etc. + +Please find detailed information and performance/accuracy numbers here: [MKLDNN README](https://github.com/apache/incubator-mxnet/blob/master/MKLDNN_README.md), [quantization README](https://github.com/apache/incubator-mxnet/tree/master/example/quantization#1) and [design proposal](https://cwiki.apache.org/confluence/display/MXNET/MXNet+Graph+Optimization+and+Quantization+based+on+subgraph+and+MKL-DNN) + +### New Operators + +* Add trigonometric operators (#12424) +* [MXNET-807] Support integer label type in ctc_loss operator (#12468) +* [MXNET-876] make CachedOp a normal operator (#11641) +* Add index_copy() operator (#12810) +* Fix getnnz operator for CSR matrix (#12908) - issue #12872 +* [MXNET-1173] Debug operators - isfinite, isinf and isnan (#12967) +* Add sample_like operators (#13034) +* Add gauss err function operator (#13229) +* [MXNET -1030] Enhanced Cosine Embedding Loss (#12750) +* Add bytearray support back to imdecode (#12855, #12868) (#12912) +* Add Psroipooling CPU implementation (#12738) + +### Feature improvements +#### Operator +* [MXNET-912] Refactoring ctc loss operator (#12637) +* Refactor L2_normalization (#13059) +* Customized and faster `TakeOpForward` operator on CPU (#12997) +* Allow stop of arange operator to be inferred from dims. (#12064) +* Make check_isfinite, check_scale optional in clip_global_norm (#12042) add FListInputNames attribute to softmax_cross_entropy (#12701) [MXNET-867] Pooling1D with same padding (#12594) +* Add support for more req patterns for bilinear sampler backward (#12386) [MXNET-882] Support for N-d arrays added to diag op. (#12430) + +#### Optimizer +* Add a special version of Adagrad optimizer with row-wise learning rate (#12365) +* Add a Python SVRGModule for performing SVRG Optimization Logic (#12376) + +#### Sparse + +* Fall back when sparse arrays are passed to MKLDNN-enabled operators (#11664) +* Add Sparse support for logic operators (#12860) +* Add Sparse support for take(csr, axis=0) (#12889) + +#### ONNX + +* ONNX export - Clip operator (#12457) +* ONNX version update from 1.2.1 to 1.3 in CI (#12633) +* Use modern ONNX API to load a model from file (#12777) +* [MXNET-892] ONNX export/import: DepthToSpace, SpaceToDepth operators (#12731) +* ONNX export: Fully connected operator w/o bias, ReduceSum, Square (#12646) +* ONNX export/import: Selu (#12785) +* ONNX export: Cleanup (#12878) +* [MXNET-892] ONNX export/import: DepthToSpace, SpaceToDepth operators (#12731) +* ONNX export: Scalar, Reshape - Set appropriate tensor type (#13067) +* [MXNET-886] ONNX export: HardSigmoid, Less, Greater, Equal (#12812) + +#### MKLDNN + +* MKLDNN Forward FullyConnected op cache (#11611) +* [MXNET-753] Fallback when using non-MKLDNN supported operators (#12019) +* MKLDNN Backward op cache (#11301) +* Implement mkldnn convolution fusion and quantization. (#12530) +* Improve mkldnn fallback. (#12663) +* Update MKL-DNN dependency (#12953) +* Update MKLML dependency (#13181) +* [MXNET-33] Enhance mkldnn pooling to support full convention (#11047) + +#### Inference +* [MXNET-910] Multithreading inference. (#12456) +* Tweaked the copy in c_predict_api.h (#12600) + +#### Other +* support for upper triangular matrices in linalg (#12904) +* Introduce Random module / Refactor code generation (#13038) +* [MXNET-779]Add DLPack Transformation API (#12047) +* Draw label name next to corresponding bounding boxes when the mapping of id to names is specified (#9496) +* Track epoch metric separately (#12182) +* Set correct update on kvstore flag in dist_device_sync mode (#12786) + +### Frontend API updates + +#### Gluon + +* Update basic_layers.py (#13299) +* Gluon LSTM Projection and Clipping Support (#13056) +* Make Gluon download function to be atomic (#12572) +* [MXNET -1004] Poisson NegativeLog Likelihood loss (#12697) +* Add activation information for `mxnet.gluon.nn._Conv` (#12354) +* Gluon DataLoader: avoid recursionlimit error (#12622) + +#### Symbol +* Addressed dumplicate object reference issues (#13214) +* Throw exception if MXSymbolInferShape fails (#12733) +* Infer dtype in SymbolBlock import from input symbol (#12412) + +### Language API updates +#### Java +* [MXNET-1198] MXNet Java API (#13162) + +#### R +* Refactor R Optimizers to fix memory leak - 11374 +* Add new Vignettes to the R package + * Char-level Language modeling - 12670 + * Multidimensional Time series forecasting - 12664 +* Fix broken Examples and tutorials + * Tutorial on neural network introduction - 12117 + * CGAN example - 12283 + * Test classification with LSTMs - 12263 + +#### Scala +* Explain the details for Scala Experimental (#12348) +* [MXNET-716] Adding Scala Inference Benchmarks (#12721) +* [MXNET-716][MIRROR #12723] Scala Benchmark Extension pack (#12758) +* NativeResource Management in Scala (#12647) +* Ignore generated Scala files (#12928) +* Use ResourceScope in Model/Trainer/FeedForward.scala (#12882) +* [MXNET-1180] Scala Image API (#12995) +* Update log4j version of Scala package (#13131) +* Review require() usages to add meaningful messages (#12570) +* Fix Scala readme (#13082) + +#### Clojure +* Introduction to Clojure-MXNet video link (#12754) +* Improve the Clojure Package README to Make it Easier to Get Started (#12881) +* MXNET-873 - Bring Clojure Package Inline with New DataDesc and Layout in Scala Package (#12387) +* Port of Scala Image API to Clojure (#13107) + +#### Perl +* [MXNET-1026] [Perl] Sync with recent changes in Python's API (#12739) + +#### Julia +* Import Julia binding (#10149), how to use is available at https://github.com/apache/incubator-mxnet/tree/master/julia + +### Performance benchmarks and improvements +* Update mshadow for omp acceleration when nvcc is not present (#12674) +* [MXNET-860] Avoid implicit double conversions (#12361) +* Add more models to benchmark_score (#12780) +* Add resnet50-v1 to benchmark_score (#12595) + +### Bug fixes +* Fix for #10920 - increase tolerance for sparse dot (#12527) +* [MXNET-1234] Fix shape inference problems in Activation backward (#13409) +* Fix a bug in `where` op with 1-D input (#12325) +* [MXNET-825] Fix CGAN R Example with MNIST dataset (#12283) +* [MXNET-535] Fix bugs in LR Schedulers and add warmup (#11234) +* Fix speech recognition example (#12291) +* Fix bug in 'device' type kvstore (#12350) +* fix search result 404s (#12414) +* Fix help in imread (#12420) +* Fix render issue on < and > (#12482) +* [MXNET-853] Fix for smooth_l1 operator scalar default value (#12284) +* Fix subscribe links, remove disabled icons (#12474) +* Fix broken URLs (#12508) +* Fix/public internal header (#12374) +* Fix lazy record io when used with dataloader and multi_worker > 0 (#12554) +* Fix error in try/finally block for blc (#12561) +* Add cudnn_off parameter to SpatialTransformer Op and fix the inconsistency between CPU & GPU code (#12557) +* [MXNET-798] Fix the dtype cast from non float32 in Gradient computation (#12290) +* Fix CodeCovs proper commit detection (#12551) +* Add TensorRT tutorial to index and fix ToC (#12587) +* Fixed typo in c_predict_api.cc (#12601) +* Fix typo in profiler.h (#12599) +* Fixed NoSuchMethodError for Jenkins Job for MBCC (#12618) +* [MXNET-922] Fix memleak in profiler (#12499) +* [MXNET-969] Fix buffer overflow in RNNOp (#12603) +* Fixed param coercion of clojure executor/forward (#12627) (#12630) +* Fix version dropdown behavior (#12632) +* Fix reference to wrong function (#12644) +* Fix the location of the tutorial of control flow operators (#12638) +* Fix issue 12613 (#12614) +* [MXNET-780] Fix exception handling bug (#12051) +* Fix bug in prelu, issue 12061 (#12660) +* [MXNET-833] [R] Char-level RNN tutorial fix (#12670) +* Fix static / dynamic linking of gperftools and jemalloc (#12714) +* Fix #12672, importing numpy scalars (zero-dimensional arrays) (#12678) +* [MXNET-623] Fixing an integer overflow bug in large NDArray (#11742) +* Fix benchmark on control flow operators (#12693) +* Fix regression in MKLDNN caused by PR 12019 (#12740) +* Fixed broken link for Baidu's WARP CTC (#12774) +* Fix CNN visualization tutorial (#12719) +* [MXNET-979] Add fix_beta support in BatchNorm (#12625) +* R fix metric shape (#12776) +* Revert [MXNET-979] Add fix_beta support in BatchNorm (#12625) (#12789) +* Fix mismatch shapes (#12793) +* Fixed symbols naming in RNNCell, LSTMCell, GRUCell (#12794) +* Fixed __setattr__ method of _MXClassPropertyMetaClass (#12811) +* Fixed regex for matching platform type in Scala Benchmark scripts (#12826) +* Fix broken links (#12856) +* Fix Flaky Topk (#12798) +* [MXNET-1033] Fix a bug in MultiboxTarget GPU implementation (#12840) +* [MXNET-1107] Fix CPUPinned unexpected behaviour (#12031) +* Fix __all__ in optimizer/optimizer.py (#12886) +* Fix Batch input issue with Scala Benchmark (#12848) +* fix type inference in index_copy. (#12890) +* Fix the paths issue for downloading script (#12913) +* Fix indpt[0] for take(csr) (#12927) +* Fix the bug of assigning large integer to NDArray (#12921) +* Fix Sphinx errors for tutorials and install ToCs (#12945) +* Fix variable name in tutorial code snippet (#13052) +* Fix example for mxnet.nd.contrib.cond and fix typo in src/engine (#12954) +* Fix a typo in operator guide (#13115) +* Fix variational autoencoder example (#12880) +* Fix problem with some OSX not handling the cast on imDecode (#13207) +* [MXNET-953] Fix oob memory read (#12631) +* Fix Sphinx error in ONNX file (#13251) +* [Example] Fixing Gradcam implementation (#13196) +* Fix train mnist for inception-bn and resnet (#13239) +* Fix a bug in index_copy (#13218) +* Fix Sphinx errors in box_nms (#13261) +* Fix Sphinx errors (#13252) +* Fix the cpp example compiler flag (#13293) +* Made fixes to sparse.py and sparse.md (#13305) +* [Example] Gradcam- Fixing a link (#13307) +* Manually track num_max_thread (#12380) +* [Issue #11912] throw mxnet exceptions when decoding invalid images. (#12999) +* Undefined name: load_model() --> utils.load_model() (#12867) +* Change the way NDArrayIter handle the last batch (#12545) +* Add embedding to print_summary (#12796) +* Allow foreach on input with 0 length (#12471) +* [MXNET-360]auto convert str to bytes in img.imdecode when py3 (#10697) +* Fix unpicklable transform_first on windows (#13686) + +### Licensing updates +* Add license headers to R-package (#12559) +* License header (#13178) +* add url and license to clojure package project (#13304) + +### Improvements +#### Tutorial +* [MXNET-422] Distributed training tutorial (#10955) +* Add a tutorial for control flow operators. (#12340) +* Add tutorial Gotchas using NumPy (#12007) +* Updated Symbol tutorial with Gluon (#12190) +* Improve tutorial redirection (#12607) +* Include missing import in TensorRT tutorial (#12609) +* Update Operator Implementation Tutorial (#12230) +* Add a tutorial for the subgraph API. (#12698) +* Improve clojure tutorial (#12974) +* Update scala intellij tutorial (#12827) +* [Example] Gradcam consolidation in tutorial (#13255) +* [MXNET-1203] Tutorial infogan (#13144) +* [MXNET-703] Add a TensorRT walkthrough (#12548) + +#### Example +* Update C++ example so it is easier to run (#12397) +* [MXNET-580] Add SN-GAN example (#12419) +* [MXNET-637] Multidimensional LSTM example for MXNetR (#12664) +* [MXNET-982] Provide example to illustrate usage of CSVIter in C++ API (#12636) +* [MXNET-947] Expand scala imclassification example with resnet (#12639) +* MKL-DNN Quantization Examples and README (#12808) +* Extending the DCGAN example implemented by gluon API to provide a more straight-forward evaluation on the generated image (#12790) +* [MXNET-1017] Updating the readme file for cpp-package and adding readme file for example directory. (#12773) +* Update tree lstm example (#12960) +* Update bilstm integer array sorting example (#12929) +* Updated / Deleted some examples (#12968) +* Update module example (#12961) +* Update adversary attack generation example (#12918) +* Update Gluon example folder (#12951) +* Update dec example (#12950) +* Updated capsnet example (#12934) +* Updates to several examples (#13068) +* Update multi-task learning example (#12964) +* Remove obsolete memory cost example (#13235) +* [Example] Update cpp example README (#13280) +* [Example]update NER example readme on module prediction (#13184) +* Update proposal_target.py (#12709) +* Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training (#12362) +* Update the README with instruction to redirect the user to gluon-cv (#13186) + +#### Documentation +* Update ONNX API docs references (#12317) +* Documentation update related to sparse support (#12367) +* Edit shape.array doc and some style improvements (#12162) +* Fixed docs/website build checkout bug (#12413) +* Add Python API docs for test_utils and visualization (#12455) +* Fix the installation doc for MKL-DNN backend (#12534) +* Added comment to docs regarding ToTensor transform (#12186) +* Pinned dockcross to a tag with fixed ABI for RPi (#12588) +* Refine the documentation of im2rec (#12606) +* Update and modify Windows docs (#12620) +* update docs to list cmake required for build from source page (#12592) +* update the distributed_training document (#12626) +* Add docstring in im2rec.py (#12621) +* [Doc] Change the description for pip packages (#12584) +* Change dependencies documentation opencv2-->opencv (#12654) +* Add documents for two new environment variables for memory pool. (#12668) +* Scala Docs - Replace old Symbol api usages (#12759) +* add/update infer_range docs (#12879) +* Fix typo in formula in docstring for GRU cell and layer and add clarification to description (gluon.rnn) (#12896) +* Fix the operator API documentation (#12942) +* fix broken docs (#12871) +* fix mac r install and windows python build from source docs (#12919) +* Document the newly added env variable (#13049) +* Add documentation on GPU performance on Quantization example (#13145) +* Fix Sphinx python docstring formatting error. (#13177) +* [Doc] Fix repo paths in Ubuntu build doc (#13101) +* Fix Sphinx document parsing error. (#13195) +* Fix #13090, Add image.imread to python API doc. (#13176) +* Fix Sphinx docstring formatting error. (#13004, #13005, #13006) (#13175) +* Fix #12944, Fix Sphinx python docstring formatting error. (#13174) +* Fix #13013, Fix Sphinx python docstring error. (#13173) +* Fixed Sparse astype doc string formatting error (#13171) +* Fixed Documentation issues (#13215) +* update the doc (#13205) +* Fix Sphinx doc errors (#13170) +* Fix Sphinx python docstring error: initializer.InitDesc (#12939) (#13148) +* Fix Sphinx python docstring error: text contrib module (#12949) (#13149) +* Fix Sphinx python docstrings (#13160) +* Add Java API docs generation (#13071) +* Fix scaladoc build errors (#13189) +* Add missing documentations for getnnz (#13128) +* Addressed ONNX module documentation warnings and added notes for short-form representation (#13259) +* Doc fixes (#13256) +* Addressed doc issues (#13165) +* stop gap fix to let website builds through; scaladoc fix pending (#13298) +* Fix Sphinx python docstring formatting error. (#13194) +* Visualization doc fix. Added notes for shortform (#13291) +* [Example] Add docstring for test optimizer and test score (#13286) +* Fix descriptions in scaladocs for macro ndarray/sybmol APIs (#13210) +* Sphinx error reduction (#12323) +* Sphinx errors in Gluon (#13275) +* Update env_var.md (#12702) +* Updated the Instructions for use of the label bot (#13192) +* Added/changed file_name, brief description comments in some files (#13033) + +#### Website +* adding apache conf promo to home page (#12347) +* Consistent website theme and custom 404 (#12426) +* update apachecon links to https (#12521) +* [HOLD] 1.3.0 release website updates (#12509) +* add mentions of the gluon toolkits and links to resources (#12667) +* remove apachecon promo (#12695) +* [MXNet-1002] Add GluonCV and NLP tookits, Keras, and developer wiki to navigation (#12704) + +#### MXNet Distributions +* Make the output of ci/docker/install/ubuntu_mklml.sh less verbose (#12422) +* Fix tvm dependency for docker (#12479) +* [MXNET-703] Add TensorRT runtime Dockerfile (#12549) +* [MXNET-951] Python dockerfiles built on pip binaries and build/release script (#12556) +* Change numpy version to 1.15.2 in python and docker install requirements (#12711) +* Add mkl-dnn to docker install method (#12643) +* Fix docker cleanup race condition (#13092) +* Bugfix in ci/docker_cache.py (#13249) +* Update PyPI version number (#11773) +* update download links to apache distros (#12617) + +#### Installation +* Installation instructions consolidation (#12388) +* Refine mxnet python installation (#12696) +* R install instructions update for macOS (#12832) +* remove legacy installation of Roxygen2 5.0 and add R-specific clean target (#12993) (#12998) +* Force APT cache update before executing install (#13285) +* Make the Ubuntu scripts executable after download. (#12180) +* replacing windows setup with newer instructions (#12504) +* Updated download links and verification instructions (#12651) +* Remove pip overwrites (#12604) + +#### Build and CI +* [MXNET-908] Enable minimal OSX Travis build (#12462) +* Use jom for parallel Windows builds (#12533) +* [MXNET-950] Enable parallel R dep builds in CI (#12552) +* Speed up CI Windows builds (#12563) +* [MXNET-908] Speed up travis builds to avoid timeouts (#12706) +* Simplify mac MKLDNN build (#12724) +* [MXNET-674] Speed up GPU builds in CI (#12782) +* Improved git reset for CI builds (#12784) +* Improve cpp-package example project build files. (#13093) +* Add --no-cache option to build.py when building containers (#13182) +* Addressed sphinx build issue (#13246) +* Tighten up PyLint directives again (#12322) +* [MXNET-859] Add a clang-tidy stage to CI (#12282) +* A solution to prevent zombie containers locally and in CI (#12381) +* [MXNET-696][PYTHON][UNDEFINED NAME] import logging in ci/util.py (#12488) +* [MXNET-703] Static linking for libprotobuf with TensorRT (#12475) +* Remove regression checks for website links (#12507) +* [MXNET-953] - Add ASAN sanitizer, Enable in CI (#12370) +* Allow custom path and static linking for custom mallocs in make (#12645) +* Correct PR branch detection in code coverage (#12615) +* Update osx.mk - Added apple to USE_BLAS comment (#12819) +* [MXNET-953] Correct ASAN cflags flag (#12659) +* [MXNET-1025] Add Jetpack 3.3 support to Jetson (#12735) +* Fail the broken link job when broken links are found (#12905) +* Removed unused header (#13066) +* Maven Surefire bug workaround (#13081) +* Add Turing and Volta support to arch_name (#13168) +* Moves f16c autodetection to its own cmake module (#12331) +* la_op_inline.h to la_op-inl.h for consistency (#13045) +* [MXNET-793] Virtualized ARMv7 with Qemu CI integration (#13203) +* Remove unused variable `rotateM_` (#10803) +* Separate refactoring from #12276 in a prior PR (#12296) +* [MXNET-860] Remove std::moves that have no affect (#12730) +* [MXNET-860] Use emplace where helpful (#12694) +* Enable C++ coverage (#12642) +* [MXNET-860] Update to modern nullptr usage (#12352) +* [MXNET-860] Reduce redundant copies, check for regressions with clang-tidy (#12355) + + +#### 3rd party +##### TVM: +* Updated tvm submodule head (#12764) +* Updated tvm submodule head (#12448) +##### CUDNN: +* [MXNET-1179] Enforce deterministic algorithms in convolution layers (#12992) +* CudnnFind() usage improvements (#12804) +* Add option for automatic downcasting dtype for cudnn to allow using Tensorcore for fp32 (#12722) +##### Horovod: +* [MXNET-1111] Remove CPUPinned in ImageRecordIter (#12666) + +### Deprications +* Add a deprecate message (#13042) contrib_CTCLoss is deprecated. Added a message in command +### Other +* Updating news, readme files and bumping master version to 1.3.1 (#12525) +* Add new name to CONTRIBUTORS.md (#12763) +* Update contribute.md (#12685) +* Updated CONTRIBUTORS.md to include lebeg and gigasquid, moved mabreu to committers section (#12766) +* Update CONTRIBUTORS.md (#12996) +* Updated CONTRIBUTORS.md to include mxnet-label-bot (#13048) + +### How to build MXNet +Please follow the instructions at https://mxnet.incubator.apache.org/install/index.html + +### List of submodules used by Apache MXNet (Incubating) and when they were updated last +Submodule@commit ID::Last updated by MXNet:: Last update in submodule + +* cub@05eb57f::Jul 31, 2017 :: Jul 31, 2017 +* dlpack@10892ac:: Oct 30, 2017 :: Aug 23, 2018 +* dmlc-core@0a0e8ad:: Aug 15, 2018 :: Nov 15, 2018 +* googletest@ec44c6c:: July 14, 2016 :: July 14, 2016 +* mkldnn@a7c5f53:: Nov 7, 2018 :: Nov 5, 2018 +* mshadow@696803b:: Sep 28, 2018 :: Nov 7, 2018 +* onnx-tensorrt@3d8ee04:: Aug 22, 2018 :: Nov 10, 2018 +* openmp@37c7212: Nov 22, 2017 :: Nov 13, 2018 +* ps-lite@8a76389: April 25, 2018 :: Oct 9, 2018 +* tvm@0f053c8: Oct 10, 2018 :: Oct 8, 2018 + +## 1.3.1 + +### Bug fixes + +* [MXNET-953] Fix oob memory read (v1.3.x) / [#13118](https://github.com/apache/incubator-mxnet/pull/13118) +Simple bugfix addressing an out-of-bounds memory read. + + +* [MXNET-969] Fix buffer overflow in RNNOp (v1.3.x) / [#13119](https://github.com/apache/incubator-mxnet/pull/13119) +This fixes an buffer overflow detected by ASAN. + + +* CudnnFind() usage improvements (v1.3.x) / [#13123](https://github.com/apache/incubator-mxnet/pull/13123) + This PR improves the MXNet's use of cudnnFind() to address a few issues: + 1. With the gluon imperative style, cudnnFind() is called during forward(), and so might have its timings perturbed by other GPU activity (including potentially other cudnnFind() calls). + 2. With some cuda drivers versions, care is needed to ensure that the large I/O and workspace cudaMallocs() performed by cudnnFind() are immediately released and available to MXNet. + 3. cudnnFind() makes both conv I/O and workspace allocations that must be covered by the GPU global memory headroom defined by MXNET_GPU_MEM_POOL_RESERVE. Per issue #12662, large convolutions can result in out-of-memory errors, even when MXNet's storage allocator has free memory in its pool. + + This PR addresses these issues, providing the following benefits: + 1. Consistent algo choice for a given convolution type in a model, both for instances in the same GPU and in other GPUs in a multi-GPU training setting. + 2. Consistent algo choice from run to run, based on eliminating sources of interference of the cudnnFind() timing process. + 3. Consistent model global memory footprint, both because of the consistent algo choice (algo's can have markedly different workspace requirements) and changes to MXNet's use of cudaMalloc. + 4. Increased training performance based on being able to consistently run with models that approach the GPU's full global memory footprint. + 5. Adds a unittest for and solves issue #12662. + +* [MXNET-922] Fix memleak in profiler (v1.3.x) / [#13120](https://github.com/apache/incubator-mxnet/pull/13120) + Fix a memleak reported locally by ASAN during a normal inference test. + +* Fix lazy record io when used with dataloader and multi_worker > 0 (v1.3.x) / [#13124](https://github.com/apache/incubator-mxnet/pull/13124) + Fixes multi_worker data loader when record file is used. The MXRecordIO instance needs to require a new file handler after fork to be safely manipulated simultaneously. + + This fix also safely voids the previous temporary fixes #12093 #11370. + +* fixed symbols naming in RNNCell, LSTMCell, GRUCell (v1.3.x) / [#13158](https://github.com/apache/incubator-mxnet/pull/13158) + This fixes #12783, by assigning all nodes in hybrid_forward a unique name. Some operations were in fact performed without attaching the appropriate (time) prefix to the name, which makes serialized graphs non-deserializable. + +* Fixed `__setattr__` method of `_MXClassPropertyMetaClass` (v1.3.x) / [#13157](https://github.com/apache/incubator-mxnet/pull/13157) + Fixed `__setattr__` method + +* allow foreach on input with 0 length (v1.3.x) / [#13151](https://github.com/apache/incubator-mxnet/pull/13151) + Fix #12470. With this change, outs shape can be inferred correctly. + +* Infer dtype in SymbolBlock import from input symbol (v1.3.x) / [#13117](https://github.com/apache/incubator-mxnet/pull/13117) + Fix for the issue - #11849 + Currently, Gluon symbol block cannot import any symbol with type other than fp32. All the parameters are created as FP32 leading to failure in importing the params when it is of type fp16, fp64 etc, + In this PR, we infer the type of the symbol being imported and create the Symbol Block Parameters with that inferred type. + Added the tests + +### Documentation fixes + +* Document the newly added env variable (v1.3.x) / [#13156](https://github.com/apache/incubator-mxnet/pull/13156) + Document the env variable: MXNET_ENFORCE_DETERMINISM added in PR: [#12992](https://github.com/apache/incubator-mxnet/pull/12992) + +* fix broken links (v1.3.x) / [#13155](https://github.com/apache/incubator-mxnet/pull/13155) + This PR fixes broken links on the website. + +* fix broken Python IO API docs (v1.3.x) / [#13154](https://github.com/apache/incubator-mxnet/pull/13154) + Fixes [#12854: Data Iterators documentation is broken](https://github.com/apache/incubator-mxnet/issues/12854) + + This PR manually specifies members of the IO module so that the docs will render as expected. This is workaround in the docs to deal with a bug introduced in the Python code/structure since v1.3.0. See the comments for more info. + + This PR also fixes another issue that may or may not be related. Cross references to same-named entities like name, shape, or type are confusing Sphinx and it seems to just link to whatever it last dealt with that has the same name, and not the current module. To fix this you have to be very specific. Don't use type, use np.type if that's what you want. Otherwise you might end up with mxnet.kvstore.KVStore.type. This is a known Sphinx issue, so it might be something we have to deal with for the time being. + + This is important for any future modules - that they recognize this issue and make efforts to map the params and other elements. + +* add/update infer_range docs (v1.3.x) / [#13153](https://github.com/apache/incubator-mxnet/pull/13153) + This PR adds or updates the docs for the infer_range feature. + + Clarifies the param in the C op docs + Clarifies the param in the the Scala symbol docs + Adds the param for the the Scala ndarray docs + Adds the param for the Python symbol docs + Adds the param for the Python ndarray docs + +### Other Improvements + +* [MXNET-1179] Enforce deterministic algorithms in convolution layers (v1.3.x) / [#13152](https://github.com/apache/incubator-mxnet/pull/13152) + Some of the CUDNN convolution algorithms are non-deterministic (see issue #11341). This PR adds an env variable to enforce determinism in the convolution operators. If set to true, only deterministic CUDNN algorithms will be used. If no deterministic algorithm is available, MXNet will error out. + + +### Submodule updates + +* update mshadow (v1.3.x) / [#13122](https://github.com/apache/incubator-mxnet/pull/13122) + Update mshadow for omp acceleration when nvcc is not present + +### Known issues + +The test test_operator.test_dropout has issues and has been disabled on the branch: + +* Disable flaky test test_operator.test_dropout (v1.3.x) / [#13200](https://github.com/apache/incubator-mxnet/pull/13200) + + + +For more information and examples, see [full release notes](https://cwiki.apache.org/confluence/x/eZGzBQ) + + ## 1.3.0 ### New Features - Gluon RNN layers are now HybridBlocks @@ -211,7 +881,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Added support for distributed mixed precision training with FP16. It supports storing of master copy of weights in float32 with the multi_precision mode of optimizers (#10183). Improved speed of float16 operations on x86 CPU by 8 times through F16C instruction set. Added support for more operators to work with FP16 inputs (#10125, #10078, #10169). Added a tutorial on using mixed precision with FP16 (#10391). ### New Features - Added Profiling Enhancements -- Enhanced built-in profiler to support native Intel:registered: VTune:tm: Amplifier objects such as Task, Frame, Event, Counter and Marker from both C++ and Python -- which is also visible in the Chrome tracing view(#8972). Added Runtime tracking of symbolic and imperative operators as well as memory and API calls. Added Tracking and dumping of aggregate profiling data. Profiler also no longer affects runtime performance when not in use. +- Enhanced built-in profiler to support native Intel:registered: VTune:tm: Amplifier objects such as Task, Frame, Event, Counter and Marker from both C++ and Python -- which is also visible in the Chrome tracing view(#8972). Added Runtime tracking of symbolic and imperative operators as well as memory and API calls. Added Tracking and dumping of aggregate profiling data. Profiler also no longer affects runtime performance when not in use. ### Breaking Changes - Changed Namespace for MXNet scala from `ml.dmlc.mxnet` to `org.apache.mxnet` (#10284). @@ -242,7 +912,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Fixed a bug that was causing training metrics to be printed as NaN sometimes (#10437). - Fixed a crash with non positive reps for tile ops (#10417). -### Performance Improvements +### Performance Improvements - On average, after the MKL-DNN change, the inference speed of MXNet + MKLDNN outperforms MXNet + OpenBLAS by a factor of 32, outperforms MXNet + MKLML by 82% and outperforms MXNet + MKLML with the experimental flag by 8%. The experiments were run for the image classifcation example, for different networks and different batch sizes. - Improved sparse SGD, sparse AdaGrad and sparse Adam optimizer speed on GPU by 30x (#9561, #10312, #10293, #10062). - Improved `sparse.retain` performance on CPU by 2.5x (#9722) @@ -347,7 +1017,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Added `axis` argument to `SequenceLast`, `SequenceMask` and `SequenceReverse` operators (#9306) - Added `lazy_update` option for standard `SGD` & `Adam` optimizer with `row_sparse` gradients (#9468, #9189) - Added `select` option in `Block.collect_params` to support regex (#9348) -- Added support for (one-to-one and sequence-to-one) inference on explicit unrolled RNN models in R (#9022) +- Added support for (one-to-one and sequence-to-one) inference on explicit unrolled RNN models in R (#9022) ### Deprecations - The Scala API name space is still called `ml.dmlc`. The name space is likely be changed in a future release to `org.apache` and might brake existing applications and scripts (#9579, #9324) ### Performance Improvements @@ -393,10 +1063,10 @@ For more information and examples, see [full release notes](https://cwiki.apache - MXNet now compiles and runs on NVIDIA Jetson TX2 boards with GPU acceleration. - You can install the python MXNet package on a Jetson board by running - `$ pip install mxnet-jetson-tx2`. ### New Features - Sparse Tensor Support [General Availability] - - Added more sparse operators: `contrib.SparseEmbedding`, `sparse.sum` and `sparse.mean`. + - Added more sparse operators: `contrib.SparseEmbedding`, `sparse.sum` and `sparse.mean`. - Added `asscipy()` for easier conversion to scipy. - Added `check_format()` for sparse ndarrays to check if the array format is valid. -### Bug-fixes +### Bug-fixes - Fixed a[-1] indexing doesn't work on `NDArray`. - Fixed `expand_dims` if axis < 0. - Fixed a bug that causes topk to produce incorrect result on large arrays. @@ -408,9 +1078,9 @@ For more information and examples, see [full release notes](https://cwiki.apache ### Doc Updates - Added a security best practices document under FAQ section. - Fixed License Headers including restoring copyright attributions. - - Documentation updates. + - Documentation updates. - Links for viewing source. - + For more information and examples, see [full release notes](https://cwiki.apache.org/confluence/display/MXNET/Apache+MXNet+%28incubating%29+1.0+Release+Notes) @@ -418,15 +1088,15 @@ For more information and examples, see [full release notes](https://cwiki.apache ### Bug-fixes - Added GPU support for the `syevd` operator which ensures that there is GPU support for all linalg-operators. - Bugfix for `syevd` on CPU such that it works for `float32`. - - Fixed API call when `OMP_NUM_THREADS` environment variable is set. + - Fixed API call when `OMP_NUM_THREADS` environment variable is set. - Fixed `MakeNonlossGradNode` bug. - - Fixed bug related to passing `dtype` to `array()`. + - Fixed bug related to passing `dtype` to `array()`. - Fixed some minor bugs for sparse distributed training. - - Fixed a bug on `Slice` accessing uninitialized memory in `param.begin` in the file `matrix_op-inl.h`. + - Fixed a bug on `Slice` accessing uninitialized memory in `param.begin` in the file `matrix_op-inl.h`. - Fixed `gluon.data.RecordFileDataset`. - Fixed a bug that caused `autograd` to crash on some networks. - - + + ## 0.12.0 ### Performance - Added full support for NVIDIA Volta GPU Architecture and CUDA 9. Training CNNs is up to 3.5x faster than Pascal when using float16 precision. @@ -434,7 +1104,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Improved ImageRecordIO image loading performance and added indexed RecordIO support. - Added better openmp thread management to improve CPU performance. ### New Features - Gluon - - Added enhancements to the Gluon package, a high-level interface designed to be easy to use while keeping most of the flexibility of low level API. Gluon supports both imperative and symbolic programming, making it easy to train complex models imperatively with minimal impact on performance. Neural networks (and other machine learning models) can be defined and trained with `gluon.nn` and `gluon.rnn` packages. + - Added enhancements to the Gluon package, a high-level interface designed to be easy to use while keeping most of the flexibility of low level API. Gluon supports both imperative and symbolic programming, making it easy to train complex models imperatively with minimal impact on performance. Neural networks (and other machine learning models) can be defined and trained with `gluon.nn` and `gluon.rnn` packages. - Added new loss functions - `SigmoidBinaryCrossEntropyLoss`, `CTCLoss`, `HuberLoss`, `HingeLoss`, `SquaredHingeLoss`, `LogisticLoss`, `TripletLoss`. - `gluon.Trainer` now allows reading and setting learning rate with `trainer.learning_rate` property. - Added API `HybridBlock.export` for exporting gluon models to MXNet format. @@ -447,7 +1117,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Added `mx.autograd.grad` and experimental second order gradient support (most operators don't support second order gradient yet). - Autograd now supports cross-device graphs. Use `x.copyto(mx.gpu(i))` and `x.copyto(mx.cpu())` to do computation on multiple devices. ### New Features - Sparse Tensor Support - - Added support for sparse matrices. + - Added support for sparse matrices. - Added limited cpu support for two sparse formats in `Symbol` and `NDArray` - `CSRNDArray` and `RowSparseNDArray`. - Added a sparse dot product operator and many element-wise sparse operators. - Added a data iterator for sparse data input - `LibSVMIter`. @@ -457,7 +1127,7 @@ For more information and examples, see [full release notes](https://cwiki.apache - Added limited support for fancy indexing, which allows you to very quickly access and modify complicated subsets of an array's values. `x[idx_arr0, idx_arr1, ..., idx_arrn]` is now supported. Features such as combining and slicing are planned for the next release. Checkout master to get a preview. - Random number generators in `mx.nd.random.*` and `mx.sym.random.*` now support both CPU and GPU. - `NDArray` and `Symbol` now supports "fluent" methods. You can now use `x.exp()` etc instead of `mx.nd.exp(x)` or `mx.sym.exp(x)`. - - Added `mx.rtc.CudaModule` for writing and running CUDA kernels from python. + - Added `mx.rtc.CudaModule` for writing and running CUDA kernels from python. - Added `multi_precision` option to optimizer for easier float16 training. - Better support for IDE auto-completion. IDEs like PyCharm can now correctly parse mxnet operators. ### API Changes @@ -505,14 +1175,14 @@ For more information and examples, see [full release notes](https://cwiki.apache ## 0.10.0 -- Overhauled documentation for commonly used Python APIs, Installation instructions, Tutorials, HowTos and MXNet Architecture. -- Updated mxnet.io for improved readability. -- Pad operator now support reflection padding. -- Fixed a memory corruption error in threadedengine. -- Added CTC loss layer to contrib package. See mx.contrib.sym.ctc_loss. -- Added new sampling operators for several distributions (normal,uniform,gamma,exponential,negative binomial). +- Overhauled documentation for commonly used Python APIs, Installation instructions, Tutorials, HowTos and MXNet Architecture. +- Updated mxnet.io for improved readability. +- Pad operator now support reflection padding. +- Fixed a memory corruption error in threadedengine. +- Added CTC loss layer to contrib package. See mx.contrib.sym.ctc_loss. +- Added new sampling operators for several distributions (normal,uniform,gamma,exponential,negative binomial). - Added documentation for experimental RNN APIs. - + ## 0.9.3 - Move symbolic API to NNVM @tqchen - Most front-end C API are backward compatible diff --git a/README.md b/NGRAPH_README.md similarity index 100% rename from README.md rename to NGRAPH_README.md diff --git a/R-package/.Rbuildignore b/R-package/.Rbuildignore index 423105a79..d0da835e2 100644 --- a/R-package/.Rbuildignore +++ b/R-package/.Rbuildignore @@ -3,5 +3,6 @@ src/*.so$ \.dll$ ^.*\.Rproj$ ^\.Rproj\.user$ +dummy.NAMESPACE README.md diff --git a/R-package/DESCRIPTION b/R-package/DESCRIPTION index 0990c63e0..70aa66e36 100644 --- a/R-package/DESCRIPTION +++ b/R-package/DESCRIPTION @@ -1,17 +1,17 @@ Package: mxnet Type: Package Title: MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems -Version: 1.3.1 +Version: 1.5.0 Date: 2017-06-27 -Author: Tianqi Chen, Qiang Kou, Tong He +Author: Tianqi Chen, Qiang Kou, Tong He, Anirudh Acharya Maintainer: Qiang Kou -Repository: DMLC +Repository: apache/incubator-mxnet Description: MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to mix the flavours of deep learning programs together to maximize the efficiency and your productivity. License: Apache License (== 2.0) -URL: https://github.com/dmlc/mxnet/tree/master/R-package -BugReports: https://github.com/dmlc/mxnet/issues +URL: https://github.com/apache/incubator-mxnet/tree/master/R-package +BugReports: https://github.com/apache/incubator-mxnet/issues Imports: methods, Rcpp (>= 0.12.1), @@ -29,7 +29,7 @@ Suggests: imager, covr Depends: - R (>= 3.3.0) + R (>= 3.4.4) LinkingTo: Rcpp VignetteBuilder: knitr -RoxygenNote: 5.0.1 +RoxygenNote: 6.1.1 diff --git a/R-package/LICENSE b/R-package/LICENSE index 8f71f43fe..0ebc218b3 100644 --- a/R-package/LICENSE +++ b/R-package/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright (c) 2015 by Contributors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/R-package/R/zzz.R b/R-package/R/zzz.R index 6c63081af..db7573ab2 100644 --- a/R-package/R/zzz.R +++ b/R-package/R/zzz.R @@ -34,8 +34,8 @@ NULL .onLoad <- function(libname, pkgname) { # Require methods for older versions of R require(methods) - library.dynam("libmxnet", pkgname, libname, local=FALSE) - library.dynam("mxnet", pkgname, libname) + tryCatch(library.dynam("libmxnet", pkgname, libname, local=FALSE), error = function(e) { print('Loading local: inst/libs/libmxnet.so'); dyn.load("R-package/inst/libs/libmxnet.so", local=FALSE) }) + tryCatch(library.dynam("mxnet", pkgname, libname), error = function(e) { print('Loading local: src/mxnet.so'); dyn.load("R-package/src/mxnet.so") }) loadModule("mxnet", TRUE) init.symbol.methods() init.context.default() diff --git a/R-package/dummy.NAMESPACE b/R-package/dummy.NAMESPACE new file mode 100644 index 000000000..6225fbf70 --- /dev/null +++ b/R-package/dummy.NAMESPACE @@ -0,0 +1,16 @@ +# Generated by roxygen2: do not edit by hand + +import(Rcpp) +import(methods) +importFrom(DiagrammeR,add_global_graph_attrs) +importFrom(DiagrammeR,create_edge_df) +importFrom(DiagrammeR,create_graph) +importFrom(DiagrammeR,create_node_df) +importFrom(DiagrammeR,render_graph) +importFrom(jsonlite,fromJSON) +importFrom(magrittr,"%>%") +importFrom(stringr,str_extract_all) +importFrom(stringr,str_replace_all) +importFrom(stringr,str_replace_na) +importFrom(stringr,str_trim) +importFrom(visNetwork,visHierarchicalLayout) diff --git a/ci/Jenkinsfile_docker_cache b/ci/Jenkinsfile_docker_cache index 77f0122f9..35e6ff9e7 100644 --- a/ci/Jenkinsfile_docker_cache +++ b/ci/Jenkinsfile_docker_cache @@ -23,12 +23,12 @@ // timeout in minutes total_timeout = 300 -node('restricted-mxnetlinux-cpu') { +node('restricted-utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu') +utils.assign_node_labels(utility: 'restricted-utility', linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu') utils.main_wrapper( core_logic: { diff --git a/ci/Jenkinsfile_utils.groovy b/ci/Jenkinsfile_utils.groovy index e2b08555d..8291bae1f 100644 --- a/ci/Jenkinsfile_utils.groovy +++ b/ci/Jenkinsfile_utils.groovy @@ -26,8 +26,11 @@ def init_git() { // retries as this will increase the amount of requests and worsen the throttling timeout(time: 15, unit: 'MINUTES') { checkout scm - sh 'git submodule update --init --recursive' sh 'git clean -xdff' + sh 'git reset --hard' + sh 'git submodule update --init --recursive' + sh 'git submodule foreach --recursive git clean -ffxd' + sh 'git submodule foreach --recursive git reset --hard' } } catch (exc) { deleteDir() @@ -45,8 +48,11 @@ def init_git_win() { // retries as this will increase the amount of requests and worsen the throttling timeout(time: 15, unit: 'MINUTES') { checkout scm - bat 'git submodule update --init --recursive' bat 'git clean -xdff' + bat 'git reset --hard' + bat 'git submodule update --init --recursive' + bat 'git submodule foreach --recursive git clean -ffxd' + bat 'git submodule foreach --recursive git reset --hard' } } catch (exc) { deleteDir() @@ -58,9 +64,11 @@ def init_git_win() { // pack libraries for later use def pack_lib(name, libs, include_gcov_data = false) { - sh """ + sh returnStatus: true, script: """ +set +e echo "Packing ${libs} into ${name}" echo ${libs} | sed -e 's/,/ /g' | xargs md5sum +return 0 """ stash includes: libs, name: name @@ -75,9 +83,11 @@ echo ${libs} | sed -e 's/,/ /g' | xargs md5sum def unpack_and_init(name, libs, include_gcov_data = false) { init_git() unstash name - sh """ + sh returnStatus: true, script: """ +set +e echo "Unpacked ${libs} from ${name}" echo ${libs} | sed -e 's/,/ /g' | xargs md5sum +return 0 """ if (include_gcov_data) { // Restore GCNO files that are required for GCOV to operate during runtime @@ -85,23 +95,32 @@ echo ${libs} | sed -e 's/,/ /g' | xargs md5sum } } +def get_jenkins_master_url() { + return env.BUILD_URL.split('/')[2].split(':')[0] +} + +def get_git_commit_hash() { + lastCommitMessage = sh (script: "git log -1 --pretty=%B", returnStdout: true) + lastCommitMessage = lastCommitMessage.trim() + if (lastCommitMessage.startsWith("Merge commit '") && lastCommitMessage.endsWith("' into HEAD")) { + // Merge commit applied by Jenkins, skip that commit + git_commit_hash = sh (script: "git rev-parse @~", returnStdout: true) + } else { + git_commit_hash = sh (script: "git rev-parse @", returnStdout: true) + } + return git_commit_hash +} + def publish_test_coverage() { // CodeCovs auto detection has trouble with our CIs PR validation due the merging strategy - lastCommitMessage = sh (script: "git log -1 --pretty=%B", returnStdout: true) - lastCommitMessage = lastCommitMessage.trim() - if (lastCommitMessage.startsWith("Merge commit '") && lastCommitMessage.endsWith("' into HEAD")) { - // Merge commit applied by Jenkins, skip that commit - GIT_COMMIT_HASH = sh (script: "git rev-parse @~", returnStdout: true) - } else { - GIT_COMMIT_HASH = sh (script: "git rev-parse @", returnStdout: true) - } + git_commit_hash = get_git_commit_hash() if (env.CHANGE_ID) { // PR execution - codecovArgs = "-B ${env.CHANGE_TARGET} -C ${GIT_COMMIT_HASH} -P ${env.CHANGE_ID}" + codecovArgs = "-B ${env.CHANGE_TARGET} -C ${git_commit_hash} -P ${env.CHANGE_ID}" } else { // Branch execution - codecovArgs = "-B ${env.BRANCH_NAME} -C ${GIT_COMMIT_HASH}" + codecovArgs = "-B ${env.BRANCH_NAME} -C ${git_commit_hash}" } // To make sure we never fail because test coverage reporting is not available @@ -128,8 +147,9 @@ def collect_test_results_windows(original_file_name, new_file_name) { } -def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') { - def command = "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY} %USE_NVIDIA% --platform %PLATFORM% --docker-build-retries 3 --shm-size %SHARED_MEM% /work/runtime_functions.sh %FUNCTION_NAME%" +def docker_run(platform, function_name, use_nvidia, shared_mem = '500m', env_vars = "") { + def command = "ci/build.py %ENV_VARS% --docker-registry ${env.DOCKER_CACHE_REGISTRY} %USE_NVIDIA% --platform %PLATFORM% --docker-build-retries 3 --shm-size %SHARED_MEM% /work/runtime_functions.sh %FUNCTION_NAME%" + command = command.replaceAll('%ENV_VARS%', env_vars.length() > 0 ? "-e ${env_vars}" : '') command = command.replaceAll('%USE_NVIDIA%', use_nvidia ? '--nvidiadocker' : '') command = command.replaceAll('%PLATFORM%', platform) command = command.replaceAll('%FUNCTION_NAME%', function_name) @@ -138,14 +158,83 @@ def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') { sh command } +// Allow publishing to GitHub with a custom context (the status shown under a PR) +// Credit to https://plugins.jenkins.io/github +def get_repo_url() { + checkout scm + return sh(returnStdout: true, script: "git config --get remote.origin.url").trim() +} +def update_github_commit_status(state, message) { + node(NODE_UTILITY) { + // NOTE: https://issues.jenkins-ci.org/browse/JENKINS-39482 + //The GitHubCommitStatusSetter requires that the Git Server is defined under + //*Manage Jenkins > Configure System > GitHub > GitHub Servers*. + //Otherwise the GitHubCommitStatusSetter is not able to resolve the repository name + //properly and you would see an empty list of repos: + //[Set GitHub commit status (universal)] PENDING on repos [] (sha:xxxxxxx) with context:test/mycontext + //See https://cwiki.apache.org/confluence/display/MXNET/Troubleshooting#Troubleshooting-GitHubcommit/PRstatusdoesnotgetpublished + repoUrl = get_repo_url() + commitSha = get_git_commit_hash() + context = get_github_context() + + step([ + $class: 'GitHubCommitStatusSetter', + reposSource: [$class: "ManuallyEnteredRepositorySource", url: repoUrl], + contextSource: [$class: "ManuallyEnteredCommitContextSource", context: context], + commitShaSource: [$class: "ManuallyEnteredShaSource", sha: commitSha], + statusBackrefSource: [$class: "ManuallyEnteredBackrefSource", backref: "${env.RUN_DISPLAY_URL}"], + errorHandlers: [[$class: 'ShallowAnyErrorHandler']], + statusResultSource: [ + $class: 'ConditionalStatusResultSource', + results: [[$class: "AnyBuildResult", message: message, state: state]] + ] + ]) + } +} + +def get_github_context() { + // Since we use multi-branch pipelines, Jenkins appends the branch name to the job name + if (env.BRANCH_NAME) { + short_job_name = JOB_NAME.substring(0, JOB_NAME.lastIndexOf('/')) + } else { + short_job_name = JOB_NAME + } + + return "ci/jenkins/${short_job_name}" +} + +def parallel_stage(stage_name, steps) { + // Allow to pass an array of steps that will be executed in parallel in a stage + new_map = [:] + + for (def step in steps) { + new_map = new_map << step + } + + stage(stage_name) { + parallel new_map + } +} def assign_node_labels(args) { + // This function allows to assign instance labels to the generalized placeholders. + // This serves two purposes: + // 1. Allow generalized placeholders (e.g. NODE_WINDOWS_CPU) in the job definition + // in order to abstract away the underlying node label. This allows to schedule a job + // onto a different node for testing or security reasons. This could be, for example, + // when you want to test a new set of slaves on separate labels or when a job should + // only be run on restricted slaves + // 2. Restrict the allowed job types within a Jenkinsfile. For example, a UNIX-CPU-only + // Jenkinsfile should not allowed access to Windows or GPU instances. This prevents + // users from just copy&pasting something into an existing Jenkinsfile without + // knowing about the limitations. NODE_LINUX_CPU = args.linux_cpu NODE_LINUX_GPU = args.linux_gpu NODE_LINUX_GPU_P3 = args.linux_gpu_p3 NODE_WINDOWS_CPU = args.windows_cpu NODE_WINDOWS_GPU = args.windows_gpu + NODE_UTILITY = args.utility } def main_wrapper(args) { @@ -158,21 +247,27 @@ def main_wrapper(args) { // assign any caught errors here err = null try { + update_github_commit_status('PENDING', 'Job has been enqueued') args['core_logic']() // set build status to success at the end currentBuild.result = "SUCCESS" + update_github_commit_status('SUCCESS', 'Job succeeded') } catch (caughtError) { - node(NODE_LINUX_CPU) { + node(NODE_UTILITY) { sh "echo caught ${caughtError}" err = caughtError currentBuild.result = "FAILURE" + update_github_commit_status('FAILURE', 'Job failed') } } finally { - node(NODE_LINUX_CPU) { + node(NODE_UTILITY) { // Call failure handler args['failure_handler']() - + + // Clean workspace to reduce space requirements + cleanWs() + // Remember to rethrow so the build is marked as failing if (err) { throw err diff --git a/ci/README.md b/ci/README.md index 3737fc740..f56a6f6a7 100644 --- a/ci/README.md +++ b/ci/README.md @@ -5,6 +5,8 @@ Docker containers You need docker and nvidia docker if you have a GPU. +Also you need to run `pip3 install docker` as it uses the [docker python module](https://docker-py.readthedocs.io/en/stable/containers.html#) + If you are in ubuntu an easy way to install Docker CE is executing the following script: @@ -86,9 +88,9 @@ source 3rdparty folder of the parent mxnet sources concurrent builds of different platforms is NOT SUPPORTED. ## ccache -For all builds a directory from the host system is mapped where ccache will store cached -compiled object files (defaults to /tmp/ci_ccache). This will speed up rebuilds -significantly. You can set this directory explicitly by setting CCACHE_DIR environment +For all builds a directory from the host system is mapped where ccache will store cached +compiled object files (defaults to /tmp/ci_ccache). This will speed up rebuilds +significantly. You can set this directory explicitly by setting CCACHE_DIR environment variable. All ccache instances are currently set to be 10 Gigabytes max in size. @@ -98,5 +100,84 @@ To run the unit tests under qemu: ./build.py -p armv7 && ./build.py -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu ``` -To get a shell on the container and debug issues with the emulator itself: -Run the output of `./build.py -p test.arm_qemu --print-docker-run` +To get a shell on the container and debug issues with the emulator itself, we build the container +and then execute it interactively. We can afterwards use port 2222 on the host to connect with SSH. + + +``` +ci/build.py -p test.arm_qemu -b && docker run -p2222:2222 -ti mxnetci/build.test.arm_qemu +``` + +Then from another terminal: + +``` +ssh -o StrictHostKeyChecking=no -p 2222 qemu@localhost +``` + +There are two pre-configured users: `root` and `qemu` both without passwords. + + +### Example of reproducing a test result with QEMU on ARM + + +You might want to enable a debug build first: + +``` +$ git diff +diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh +index 39631f9..666ceea 100755 +--- a/ci/docker/runtime_functions.sh ++++ b/ci/docker/runtime_functions.sh +@@ -172,6 +172,7 @@ build_armv7() { + -DUSE_LAPACK=OFF \ + -DBUILD_CPP_EXAMPLES=OFF \ + -Dmxnet_LINKER_LIBS=-lgfortran \ ++ -DCMAKE_BUILD_TYPE=Debug \ + -G Ninja /work/mxnet + + ninja -v + +``` + +Then we build the project for armv7, the test container and start QEMU inside docker: + +``` +ci/build.py -p armv7 +ci/build.py -p test.arm_qemu -b && docker run -p2222:2222 -ti mxnetci/build.test.arm_qemu +``` + + + +At this point we copy artifacts and sources to the VM, in another terminal (host) do the following: + +``` +# Copy mxnet sources to the VM +rsync --delete -e 'ssh -p2222' --exclude='.git/' -zvaP ./ qemu@localhost:mxnet + + +# Ssh into the vm +ssh -p2222 qemu@localhost + +cd mxnet + +# Execute a single failing C++ test +build/tests/mxnet_unit_tests --gtest_filter="ACTIVATION_PERF.ExecuteBidirectional" + +# To install MXNet: +sudo pip3 install --upgrade --force-reinstall build/mxnet-1.3.1-py2.py3-none-any.whl + +# Execute a single python test: + +nosetests-3.4 -v -s tests/python/unittest/test_ndarray.py + + +# Debug with cgdb +sudo apt install -y libstdc++6-6-dbg +cgdb build/tests/mxnet_unit_tests + +(gdb) !pwd +/home/qemu/mxnet +(gdb) set substitute-path /work /home/qemu +(gdb) set substitute-path /build/gcc-6-6mK9AW/gcc-6-6.3.0/build/arm-linux-gnueabihf/libstdc++-v3/include/ /usr/include/c++/6/ +(gdb) r --gtest_filter="ACTIVATION_PERF.ExecuteBidirectional" +``` diff --git a/ci/build.py b/ci/build.py index 238b89ae7..3f420d5d3 100755 --- a/ci/build.py +++ b/ci/build.py @@ -92,34 +92,38 @@ def get_dockerfiles_path(): def get_platforms(path: str = get_dockerfiles_path()) -> List[str]: """Get a list of architectures given our dockerfiles""" - dockerfiles = glob.glob(os.path.join(path, "Dockerfile.build.*")) + dockerfiles = glob.glob(os.path.join(path, "Dockerfile.*")) dockerfiles = list(filter(lambda x: x[-1] != '~', dockerfiles)) - files = list(map(lambda x: re.sub(r"Dockerfile.build.(.*)", r"\1", x), dockerfiles)) + files = list(map(lambda x: re.sub(r"Dockerfile.(.*)", r"\1", x), dockerfiles)) platforms = list(map(lambda x: os.path.split(x)[1], sorted(files))) return platforms def get_docker_tag(platform: str, registry: str) -> str: """:return: docker tag to be used for the container""" - return "{0}/build.{1}".format(registry, platform) + platform = platform if any(x in platform for x in ['build.', 'publish.']) else 'build.{}'.format(platform) + if not registry: + registry = "mxnet_local" + return "{0}/{1}".format(registry, platform) def get_dockerfile(platform: str, path=get_dockerfiles_path()) -> str: - return os.path.join(path, "Dockerfile.build.{0}".format(platform)) + platform = platform if any(x in platform for x in ['build.', 'publish.']) else 'build.{}'.format(platform) + return os.path.join(path, "Dockerfile.{0}".format(platform)) def get_docker_binary(use_nvidia_docker: bool) -> str: return "nvidia-docker" if use_nvidia_docker else "docker" -def build_docker(platform: str, docker_binary: str, registry: str, num_retries: int, use_cache: bool) -> str: +def build_docker(platform: str, docker_binary: str, registry: str, num_retries: int, no_cache: bool) -> str: """ Build a container for the given platform :param platform: Platform :param docker_binary: docker binary to use (docker/nvidia-docker) :param registry: Dockerhub registry name :param num_retries: Number of retries to build the docker image - :param use_cache: will pass cache_from to docker to use the previously pulled tag + :param no_cache: pass no-cache to docker to rebuild the images :return: Id of the top level image """ @@ -145,7 +149,9 @@ def build_docker(platform: str, docker_binary: str, registry: str, num_retries: "-f", get_dockerfile(platform), "--build-arg", "USER_ID={}".format(os.getuid()), "--build-arg", "GROUP_ID={}".format(os.getgid())] - if use_cache: + if no_cache: + cmd.append("--no-cache") + elif registry: cmd.extend(["--cache-from", tag]) cmd.extend(["-t", tag, get_dockerfiles_path()]) @@ -209,7 +215,7 @@ def default_ccache_dir() -> str: ccache_dir = "/tmp/_mxnet_ccache" os.makedirs(ccache_dir, exist_ok=True) return ccache_dir - return os.path.join(tempfile.gettempdir(), "ci_ccache") + return os.path.join(os.path.expanduser("~"), ".ccache") def trim_container_id(cid): @@ -224,20 +230,21 @@ def container_run(platform: str, local_ccache_dir: str, command: List[str], cleanup: Cleanup, + environment: Dict[str, str], dry_run: bool = False) -> int: """Run command in a container""" container_wait_s = 600 # # Environment setup # - environment = { + environment.update({ 'CCACHE_MAXSIZE': '500G', 'CCACHE_TEMPDIR': '/tmp/ccache', # temp dir should be local and not shared 'CCACHE_DIR': '/work/ccache', # this path is inside the container as /work/ccache is # mounted 'CCACHE_LOGFILE': '/tmp/ccache.log', # a container-scoped log, useful for ccache # verification. - } + }) # These variables are passed to the container to the process tree killer can find runaway # process inside the container # https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller @@ -294,7 +301,6 @@ def container_run(platform: str, # noinspection PyShadowingNames # runc is default (docker info | grep -i runtime) runtime = 'nvidia' - container = docker_client.containers.run( tag, runtime=runtime, @@ -312,52 +318,59 @@ def container_run(platform: str, {'bind': '/work/ccache', 'mode': 'rw'}, }, environment=environment) - logging.info("Started container: %s", trim_container_id(container.id)) - # Race condition: - # If the previous call is interrupted then it's possible that the container is not cleaned up - # We avoid by masking the signals temporarily - cleanup.add_container(container) - signal.pthread_sigmask(signal.SIG_UNBLOCK, {signal.SIGINT, signal.SIGTERM}) - # - ############################# - - stream = container.logs(stream=True, stdout=True, stderr=True) - sys.stdout.flush() - for chunk in stream: - sys.stdout.buffer.write(chunk) - sys.stdout.buffer.flush() - sys.stdout.flush() - stream.close() try: - logging.info("Waiting for status of container %s for %d s.", - trim_container_id(container.id), - container_wait_s) - wait_result = container.wait(timeout=container_wait_s) - logging.info("Container exit status: %s", wait_result) - ret = wait_result.get('StatusCode', 200) - except Exception as e: - logging.exception(e) - ret = 150 - - # Stop - try: - logging.info("Stopping container: %s", trim_container_id(container.id)) - container.stop() - except Exception as e: - logging.exception(e) - ret = 151 + logging.info("Started container: %s", trim_container_id(container.id)) + # Race condition: + # If the previous call is interrupted then it's possible that the container is not cleaned up + # We avoid by masking the signals temporarily + cleanup.add_container(container) + signal.pthread_sigmask(signal.SIG_UNBLOCK, {signal.SIGINT, signal.SIGTERM}) + # + ############################# + + stream = container.logs(stream=True, stdout=True, stderr=True) + sys.stdout.flush() + for chunk in stream: + sys.stdout.buffer.write(chunk) + sys.stdout.buffer.flush() + sys.stdout.flush() + stream.close() + try: + logging.info("Waiting for status of container %s for %d s.", + trim_container_id(container.id), + container_wait_s) + wait_result = container.wait(timeout=container_wait_s) + logging.info("Container exit status: %s", wait_result) + ret = wait_result.get('StatusCode', 200) + if ret != 0: + logging.error("Container exited with an error 😞") + else: + logging.info("Container exited with success 👍") + except Exception as e: + logging.exception(e) + ret = 150 - # Remove - try: - logging.info("Removing container: %s", trim_container_id(container.id)) - container.remove() - except Exception as e: - logging.exception(e) - ret = 152 - cleanup.remove_container(container) - containers = docker_client.containers.list() - if containers: - logging.info("Other running containers: %s", [trim_container_id(x.id) for x in containers]) + # Stop + try: + logging.info("Stopping container: %s", trim_container_id(container.id)) + container.stop() + except Exception as e: + logging.exception(e) + ret = 151 + + # Remove + try: + logging.info("Removing container: %s", trim_container_id(container.id)) + container.remove() + except Exception as e: + logging.exception(e) + ret = 152 + cleanup.remove_container(container) + containers = docker_client.containers.list() + if containers: + logging.info("Other running containers: %s", [trim_container_id(x.id) for x in containers]) + except docker.errors.NotFound as e: + logging.info("Container was stopped before cleanup started: %s", e) return ret @@ -391,11 +404,15 @@ def script_name() -> str: """:returns: script name with leading paths removed""" return os.path.split(sys.argv[0])[1] - -def main() -> int: +def config_logging(): + import time logging.getLogger().setLevel(logging.INFO) logging.getLogger("requests").setLevel(logging.WARNING) - logging.basicConfig(format='{}: %(asctime)-15s %(message)s'.format(script_name())) + logging.basicConfig(format='{}: %(asctime)sZ %(levelname)s %(message)s'.format(script_name())) + logging.Formatter.converter = time.gmtime + +def main() -> int: + config_logging() logging.info("MXNet container based build tool.") log_environment() @@ -433,7 +450,7 @@ def main() -> int: action='store_true') parser.add_argument("-d", "--docker-registry", - help="Dockerhub registry name to retrieve cache from. Default is 'mxnetci'", + help="Dockerhub registry name to retrieve cache from.", default='mxnetci', type=str) @@ -442,10 +459,12 @@ def main() -> int: default=1, type=int) - parser.add_argument("-c", "--no-dockerhub-cache", action="store_true", - help="Disables use of --cache-from option on docker build, allowing docker" - " to use local layers for caching. If absent, we use the cache from dockerhub" - " which is the default.") + parser.add_argument("--no-cache", action="store_true", + help="passes --no-cache to docker build") + + parser.add_argument("-e", "--environment", nargs="*", default=[], + help="Environment variables for the docker container. " + "Specify with a list containing either names or name=value") parser.add_argument("command", help="command to run in the container", @@ -458,9 +477,6 @@ def main() -> int: args = parser.parse_args() - def use_cache(): - return not args.no_dockerhub_cache or under_ci() - command = list(chain(*args.command)) docker_binary = get_docker_binary(args.nvidiadocker) @@ -478,15 +494,18 @@ def signal_handler(signum, _): signal.signal(signal.SIGTERM, signal_handler) signal.signal(signal.SIGINT, signal_handler) + environment = dict([(e.split('=')[:2] if '=' in e else (e, os.environ[e])) + for e in args.environment]) + if args.list: print(list_platforms()) elif args.platform: platform = args.platform tag = get_docker_tag(platform=platform, registry=args.docker_registry) - if use_cache(): + if args.docker_registry: load_docker_cache(tag=tag, docker_registry=args.docker_registry) build_docker(platform=platform, docker_binary=docker_binary, registry=args.docker_registry, - num_retries=args.docker_build_retries, use_cache=use_cache()) + num_retries=args.docker_build_retries, no_cache=args.no_cache) if args.build_only: logging.warning("Container was just built. Exiting due to build-only.") return 0 @@ -497,13 +516,13 @@ def signal_handler(signum, _): ret = container_run( platform=platform, nvidia_runtime=args.nvidiadocker, shared_memory_size=args.shared_memory_size, command=command, docker_registry=args.docker_registry, - local_ccache_dir=args.ccache_dir, cleanup=cleanup) + local_ccache_dir=args.ccache_dir, cleanup=cleanup, environment=environment) elif args.print_docker_run: command = [] ret = container_run( platform=platform, nvidia_runtime=args.nvidiadocker, shared_memory_size=args.shared_memory_size, command=command, docker_registry=args.docker_registry, - local_ccache_dir=args.ccache_dir, dry_run=True, cleanup=cleanup) + local_ccache_dir=args.ccache_dir, dry_run=True, cleanup=cleanup, environment=environment) else: # With no commands, execute a build function for the target platform command = ["/work/mxnet/ci/docker/runtime_functions.sh", "build_{}".format(platform)] @@ -511,7 +530,7 @@ def signal_handler(signum, _): ret = container_run( platform=platform, nvidia_runtime=args.nvidiadocker, shared_memory_size=args.shared_memory_size, command=command, docker_registry=args.docker_registry, - local_ccache_dir=args.ccache_dir, cleanup=cleanup) + local_ccache_dir=args.ccache_dir, cleanup=cleanup, environment=environment) if ret != 0: logging.critical("Execution of %s failed with status: %d", command, ret) @@ -519,14 +538,14 @@ def signal_handler(signum, _): elif args.all: platforms = get_platforms() + platforms = [platform for platform in platforms if 'build.' in platform] logging.info("Building for all architectures: %s", platforms) logging.info("Artifacts will be produced in the build/ directory.") for platform in platforms: tag = get_docker_tag(platform=platform, registry=args.docker_registry) - if use_cache(): - load_docker_cache(tag=tag, docker_registry=args.docker_registry) + load_docker_cache(tag=tag, docker_registry=args.docker_registry) build_docker(platform, docker_binary=docker_binary, registry=args.docker_registry, - num_retries=args.docker_build_retries, use_cache=use_cache()) + num_retries=args.docker_build_retries, no_cache=args.no_cache) if args.build_only: continue shutil.rmtree(buildir(), ignore_errors=True) @@ -540,7 +559,7 @@ def signal_handler(signum, _): container_run( platform=platform, nvidia_runtime=args.nvidiadocker, shared_memory_size=args.shared_memory_size, command=command, docker_registry=args.docker_registry, - local_ccache_dir=args.ccache_dir, cleanup=cleanup) + local_ccache_dir=args.ccache_dir, cleanup=cleanup, environment=environment) shutil.move(buildir(), plat_buildir) logging.info("Built files left in: %s", plat_buildir) diff --git a/ci/build_windows.py b/ci/build_windows.py index 56769f7cd..b7d47fb1f 100755 --- a/ci/build_windows.py +++ b/ci/build_windows.py @@ -160,9 +160,6 @@ def windows_package(args): copy_tree('python', j(pkgdir, 'python')) logging.info('packing headers') copy_tree('include', j(pkgdir, 'include')) - copy_tree(j('3rdparty','dmlc-core','include'), j(pkgdir, 'include')) - copy_tree(j('3rdparty','mshadow', 'mshadow'), j(pkgdir, 'include', 'mshadow')) - copy_tree(j('3rdparty','tvm','nnvm', 'include'), j(pkgdir,'include', 'nnvm', 'include')) logging.info("Compressing package: %s", pkgfile) check_call(['7z', 'a', pkgfile, pkgdir]) diff --git a/ci/docker/Dockerfile.build.android_armv7 b/ci/docker/Dockerfile.build.android_armv7 index 799e29c99..a2e98cd2e 100644 --- a/ci/docker/Dockerfile.build.android_armv7 +++ b/ci/docker/Dockerfile.build.android_armv7 @@ -18,7 +18,7 @@ # # Dockerfile to build MXNet for Android ARMv7 -FROM mxnetci/dockcross-linux-base:08212018 +FROM mxnetcipinned/dockcross-base:11262018 MAINTAINER Pedro Larroy "pllarroy@amazon.com" # The cross-compiling emulator @@ -75,6 +75,11 @@ ENV OpenBLAS_DIR=${CROSS_ROOT} WORKDIR /work +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + COPY runtime_functions.sh /work/ WORKDIR /work/mxnet diff --git a/ci/docker/Dockerfile.build.android_armv8 b/ci/docker/Dockerfile.build.android_armv8 index 2c2c71c00..f7de86763 100644 --- a/ci/docker/Dockerfile.build.android_armv8 +++ b/ci/docker/Dockerfile.build.android_armv8 @@ -18,7 +18,7 @@ # # Dockerfile to build MXNet for Android ARM64/ARMv8 -FROM mxnetci/dockcross-linux-base:08212018 +FROM mxnetcipinned/dockcross-base:11262018 MAINTAINER Pedro Larroy "pllarroy@amazon.com" RUN apt-get update && apt-get install -y \ @@ -74,6 +74,12 @@ ENV CXX=${CROSS_ROOT}/bin/${CROSS_TRIPLE}-clang++ COPY install/android_arm64_openblas.sh /work/ RUN /work/android_arm64_openblas.sh ENV CPLUS_INCLUDE_PATH /work/deps/OpenBLAS -WORKDIR /work/build + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh COPY runtime_functions.sh /work/ + +WORKDIR /work/build \ No newline at end of file diff --git a/ci/docker/Dockerfile.build.armv6 b/ci/docker/Dockerfile.build.armv6 index 78071fa33..60e223b7a 100644 --- a/ci/docker/Dockerfile.build.armv6 +++ b/ci/docker/Dockerfile.build.armv6 @@ -18,7 +18,7 @@ # # Dockerfile to build MXNet for ARMv6 -FROM mxnetci/dockcross-linux-armv6:08212018 +FROM mxnetcipinned/dockcross-linux-armv6:11262018 ENV ARCH armv6l ENV HOSTCC gcc @@ -38,5 +38,10 @@ ENV OpenBLAS_DIR=${CROSS_ROOT} COPY install/deb_ubuntu_ccache.sh /work/ RUN /work/deb_ubuntu_ccache.sh +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + COPY runtime_functions.sh /work/ WORKDIR /work/mxnet diff --git a/ci/docker/Dockerfile.build.armv7 b/ci/docker/Dockerfile.build.armv7 index 9a23a5dbe..0b557d583 100644 --- a/ci/docker/Dockerfile.build.armv7 +++ b/ci/docker/Dockerfile.build.armv7 @@ -18,7 +18,7 @@ # # Dockerfile to build MXNet for Android ARMv7 -FROM mxnetci/dockcross-linux-armv7:09182018 +FROM mxnetcipinned/dockcross-linux-armv7:11262018 ENV ARCH armv7l ENV HOSTCC gcc @@ -38,5 +38,10 @@ ENV OpenBLAS_DIR=${CROSS_ROOT} COPY install/deb_ubuntu_ccache.sh /work/ RUN /work/deb_ubuntu_ccache.sh +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + COPY runtime_functions.sh /work/ WORKDIR /work/mxnet diff --git a/ci/docker/Dockerfile.build.armv8 b/ci/docker/Dockerfile.build.armv8 index 46cc229d5..ef9c95865 100644 --- a/ci/docker/Dockerfile.build.armv8 +++ b/ci/docker/Dockerfile.build.armv8 @@ -18,7 +18,7 @@ # # Dockerfile to build MXNet for ARM64/ARMv8 -FROM mxnetci/dockcross-linux-arm64:08212018 +FROM mxnetcipinned/dockcross-linux-arm64:11262018 ENV ARCH aarch64 ENV HOSTCC gcc @@ -27,8 +27,8 @@ ENV TARGET ARMV8 WORKDIR /work/deps # gh issue #11567 https://github.com/apache/incubator-mxnet/issues/11567 -RUN sed -i '\#deb http://cdn-fastly.deb.debian.org/debian-security jessie/updates main#d' /etc/apt/sources.list -RUN sed -i 's/cdn-fastly.//' /etc/apt/sources.list +#RUN sed -i '\#deb http://cdn-fastly.deb.debian.org/debian-security jessie/updates main#d' /etc/apt/sources.list +#RUN sed -i 's/cdn-fastly.//' /etc/apt/sources.list COPY install/ubuntu_arm.sh /work/ RUN /work/ubuntu_arm.sh @@ -42,5 +42,10 @@ ENV OpenBLAS_DIR=${CROSS_ROOT} COPY install/deb_ubuntu_ccache.sh /work/ RUN /work/deb_ubuntu_ccache.sh +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + COPY runtime_functions.sh /work/ WORKDIR /work/build diff --git a/ci/docker/Dockerfile.build.centos7_cpu b/ci/docker/Dockerfile.build.centos7_cpu index 076ef5df9..e2802aa2f 100644 --- a/ci/docker/Dockerfile.build.centos7_cpu +++ b/ci/docker/Dockerfile.build.centos7_cpu @@ -28,6 +28,8 @@ COPY install/centos7_ccache.sh /work/ RUN /work/centos7_ccache.sh COPY install/centos7_python.sh /work/ RUN /work/centos7_python.sh +COPY install/centos7_scala.sh /work/ +RUN /work/centos7_scala.sh COPY install/ubuntu_mklml.sh /work/ RUN /work/ubuntu_mklml.sh diff --git a/ci/docker/Dockerfile.build.jetson b/ci/docker/Dockerfile.build.jetson index 15518cd6f..07097887f 100644 --- a/ci/docker/Dockerfile.build.jetson +++ b/ci/docker/Dockerfile.build.jetson @@ -22,15 +22,15 @@ FROM nvidia/cuda:9.0-cudnn7-devel as cudabuilder -FROM mxnetci/dockcross-linux-arm64:05082018 +FROM mxnetcipinned/dockcross-linux-arm64:11262018 ENV ARCH aarch64 ENV HOSTCC gcc ENV TARGET ARMV8 # gh issue #11567 https://github.com/apache/incubator-mxnet/issues/11567 -RUN sed -i '\#deb http://cdn-fastly.deb.debian.org/debian-security jessie/updates main#d' /etc/apt/sources.list -RUN sed -i 's/cdn-fastly.//' /etc/apt/sources.list +#RUN sed -i '\#deb http://cdn-fastly.deb.debian.org/debian-security jessie/updates main#d' /etc/apt/sources.list +#RUN sed -i 's/cdn-fastly.//' /etc/apt/sources.list WORKDIR /work/deps @@ -77,10 +77,16 @@ RUN JETPACK_DOWNLOAD_PREFIX=https://developer.download.nvidia.com/devzone/devcen dpkg -i --force-architecture $ARM_NVINFER_INSTALLER_PACKAGE && \ dpkg -i --force-architecture $ARM_NVINFER_DEV_INSTALLER_PACKAGE && \ apt update -y || true && apt install -y cuda-libraries-dev-9-0 libcudnn7-dev libnvinfer-dev +RUN ln -s /usr/include/aarch64-linux-gnu/cudnn_v7.h /usr/include/aarch64-linux-gnu/cudnn.h ENV PATH $PATH:/usr/local/cuda/bin ENV NVCCFLAGS "-m64" ENV CUDA_ARCH "-gencode arch=compute_53,code=sm_53 -gencode arch=compute_62,code=sm_62" ENV NVCC /usr/local/cuda/bin/nvcc +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + COPY runtime_functions.sh /work/ WORKDIR /work/mxnet diff --git a/ci/docker/Dockerfile.build.test.arm_qemu b/ci/docker/Dockerfile.build.test.arm_qemu index fde105c3f..68891a72c 100644 --- a/ci/docker/Dockerfile.build.test.arm_qemu +++ b/ci/docker/Dockerfile.build.test.arm_qemu @@ -39,6 +39,8 @@ RUN /work/ubuntu_adduser.sh COPY runtime_functions.sh /work/ COPY qemu/* /work/ -COPY qemu/ansible.cfg /etc/ansible/ansible.cfg -CMD ["./runtime_functions.py","run_ut_py3_qemu"] +# SSH to the Qemu VM +EXPOSE 2222/tcp + +CMD ["./runtime_functions.py","run_qemu_interactive"] diff --git a/ci/docker/Dockerfile.build.ubuntu_cpu b/ci/docker/Dockerfile.build.ubuntu_cpu index 7c7e2240e..2df9f5887 100644 --- a/ci/docker/Dockerfile.build.ubuntu_cpu +++ b/ci/docker/Dockerfile.build.ubuntu_cpu @@ -54,6 +54,9 @@ RUN /work/ubuntu_clang.sh COPY install/ubuntu_gcc8.sh /work/ RUN /work/ubuntu_gcc8.sh +COPY install/ubuntu_mkl.sh /work/ +RUN /work/ubuntu_mkl.sh + COPY install/ubuntu_mklml.sh /work/ RUN /work/ubuntu_mklml.sh diff --git a/ci/docker/Dockerfile.publish.test.centos7_cpu b/ci/docker/Dockerfile.publish.test.centos7_cpu new file mode 100644 index 000000000..7d2844529 --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.centos7_cpu @@ -0,0 +1,38 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to build and run MXNet on CentOS 7 for CPU + +FROM centos:7 + +WORKDIR /work/deps + +COPY install/centos7_base.sh /work/ +RUN /work/centos7_base.sh + +COPY install/centos7_scala.sh /work/ +RUN /work/centos7_scala.sh + +ARG USER_ID=0 +COPY install/centos7_adduser.sh /work/ +RUN /work/centos7_adduser.sh + +ENV PYTHONPATH=./python/ +WORKDIR /work/mxnet + +COPY runtime_functions.sh /work/ diff --git a/ci/docker/Dockerfile.publish.test.centos7_gpu b/ci/docker/Dockerfile.publish.test.centos7_gpu new file mode 100644 index 000000000..e7f584683 --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.centos7_gpu @@ -0,0 +1,38 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to build and run MXNet on CentOS 7 for CPU + +FROM nvidia/cuda:9.2-cudnn7-devel-centos7 + +WORKDIR /work/deps + +COPY install/centos7_base.sh /work/ +RUN /work/centos7_base.sh + +COPY install/centos7_scala.sh /work/ +RUN /work/centos7_scala.sh + +ARG USER_ID=0 +COPY install/centos7_adduser.sh /work/ +RUN /work/centos7_adduser.sh + +ENV PYTHONPATH=./python/ +WORKDIR /work/mxnet + +COPY runtime_functions.sh /work/ diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu b/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu new file mode 100644 index 000000000..035837686 --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.ubuntu1404_cpu @@ -0,0 +1,39 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to build and run MXNet on Ubuntu 14.04 for CPU + +FROM ubuntu:14.04 + +WORKDIR /work/deps + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu b/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu new file mode 100644 index 000000000..854dd68a6 --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.ubuntu1404_gpu @@ -0,0 +1,40 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to run MXNet on Ubuntu 14.04 for GPU + +# Use CPU with setup_gpu script +FROM ubuntu:14.04 + +WORKDIR /work/deps + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu b/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu new file mode 100644 index 000000000..bbb7b6a0d --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.ubuntu1604_cpu @@ -0,0 +1,39 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU + +FROM ubuntu:16.04 + +WORKDIR /work/deps + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu b/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu new file mode 100644 index 000000000..660461dc0 --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.ubuntu1604_gpu @@ -0,0 +1,39 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to run MXNet on Ubuntu 16.04 for GPU + +FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04 + +WORKDIR /work/deps + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/example/bi-lstm-sort/gen_data.py b/ci/docker/Dockerfile.publish.test.ubuntu1804_cpu similarity index 61% rename from example/bi-lstm-sort/gen_data.py rename to ci/docker/Dockerfile.publish.test.ubuntu1804_cpu index 55af1b455..e3a8c193f 100644 --- a/example/bi-lstm-sort/gen_data.py +++ b/ci/docker/Dockerfile.publish.test.ubuntu1804_cpu @@ -1,3 +1,4 @@ +# -*- mode: dockerfile -*- # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -14,24 +15,27 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. +# +# Dockerfile to build and run MXNet on Ubuntu 18.04 for CPU + +FROM ubuntu:18.04 + +WORKDIR /work/deps + +ENV DEBIAN_FRONTEND noninteractive + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ -import random - -vocab = [str(x) for x in range(100, 1000)] -sw_train = open("sort.train.txt", "w") -sw_test = open("sort.test.txt", "w") -sw_valid = open("sort.valid.txt", "w") - -for i in range(1000000): - seq = " ".join([vocab[random.randint(0, len(vocab) - 1)] for j in range(5)]) - k = i % 50 - if k == 0: - sw_test.write(seq + "\n") - elif k == 1: - sw_valid.write(seq + "\n") - else: - sw_train.write(seq + "\n") - -sw_train.close() -sw_test.close() -sw_valid.close() +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu b/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu new file mode 100644 index 000000000..99f7e0d3e --- /dev/null +++ b/ci/docker/Dockerfile.publish.test.ubuntu1804_gpu @@ -0,0 +1,41 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to run MXNet on Ubuntu 18.04 for GPU + +FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04 + +WORKDIR /work/deps + +ENV DEBIAN_FRONTEND noninteractive + +COPY install/ubuntu_base.sh /work/ +RUN /work/ubuntu_base.sh + +COPY install/ubuntu_scala.sh /work/ +RUN /work/ubuntu_scala.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.ubuntu1404_cpu b/ci/docker/Dockerfile.publish.ubuntu1404_cpu new file mode 100644 index 000000000..04ce94f95 --- /dev/null +++ b/ci/docker/Dockerfile.publish.ubuntu1404_cpu @@ -0,0 +1,36 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to build and run MXNet on Ubuntu 14.04 for CPU + +FROM ubuntu:14.04 + +WORKDIR /work/deps + +COPY install/ubuntu_publish.sh /work/ +RUN /work/ubuntu_publish.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/Dockerfile.publish.ubuntu1404_gpu b/ci/docker/Dockerfile.publish.ubuntu1404_gpu new file mode 100644 index 000000000..9855986a2 --- /dev/null +++ b/ci/docker/Dockerfile.publish.ubuntu1404_gpu @@ -0,0 +1,36 @@ +# -*- mode: dockerfile -*- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Dockerfile to run MXNet on Ubuntu 14.04 for GPU + +FROM ubuntu:14.04 + +WORKDIR /work/deps + +COPY install/ubuntu_publish.sh /work/ +RUN /work/ubuntu_publish.sh + +ARG USER_ID=0 +ARG GROUP_ID=0 +COPY install/ubuntu_adduser.sh /work/ +RUN /work/ubuntu_adduser.sh + +COPY runtime_functions.sh /work/ + +WORKDIR /work/mxnet +ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib diff --git a/ci/docker/install/centos7_adduser.sh b/ci/docker/install/centos7_adduser.sh index ba72c9b92..f9d2402c9 100755 --- a/ci/docker/install/centos7_adduser.sh +++ b/ci/docker/install/centos7_adduser.sh @@ -34,4 +34,9 @@ then mkdir /work/mxnet mkdir /work/build chown -R jenkins_slave /work/ + + # Later on, we have to override the links because underlying build systems ignore our compiler settings. Thus, + # we have to give the process the proper permission to these files. This is hacky, but unfortunately + # there's no better way to do this without patching all our submodules. + chown -R jenkins_slave /usr/local/bin fi diff --git a/ci/docker/install/centos7_base.sh b/ci/docker/install/centos7_base.sh new file mode 100755 index 000000000..3b84aeb57 --- /dev/null +++ b/ci/docker/install/centos7_base.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# build and install are separated so changes to build don't invalidate +# the whole docker cache for the image + +set -ex + +# Multipackage installation does not fail in yum +yum -y install epel-release +yum -y install git +yum -y install wget +yum -y install make +yum -y install cmake +yum -y install unzip +yum -y install ninja-build +yum -y install gcc-gfortran diff --git a/ci/docker/install/centos7_scala.sh b/ci/docker/install/centos7_scala.sh new file mode 100755 index 000000000..5c43f011c --- /dev/null +++ b/ci/docker/install/centos7_scala.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# build and install are separated so changes to build don't invalidate +# the whole docker cache for the image + +set -ex + +yum install -y java-1.8.0-openjdk-devel +export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk +export PATH=$JAVA_HOME/bin:$PATH +# Build from source with Maven +wget -q http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz +tar xzf apache-maven-3.3.9-bin.tar.gz +mkdir /usr/local/maven +mv apache-maven-3.3.9/ /usr/local/maven/ +alternatives --install /usr/bin/mvn mvn /usr/local/maven/apache-maven-3.3.9/bin/mvn 1 + +echo "export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk" >> /etc/profile.d/maven.sh +echo "export M3_HOME=/usr/local/src/apache-maven" >> /etc/profile.d/maven.sh +echo "export PATH=$M3_HOME/bin:$JAVA_HOME/bin:$PATH" >> /etc/profile.d/maven.sh +chmod +x /etc/profile.d/maven.sh +source /etc/profile.d/maven.sh diff --git a/ci/docker/install/ubuntu_adduser.sh b/ci/docker/install/ubuntu_adduser.sh index 515a80f63..a7668bac2 100755 --- a/ci/docker/install/ubuntu_adduser.sh +++ b/ci/docker/install/ubuntu_adduser.sh @@ -40,4 +40,9 @@ then mkdir /work/mxnet mkdir /work/build chown -R jenkins_slave /work/ + + # Later on, we have to override the links because underlying build systems ignore our compiler settings. Thus, + # we have to give the process the proper permission to these files. This is hacky, but unfortunately + # there's no better way to do this without patching all our submodules. + chown -R jenkins_slave /usr/local/bin fi diff --git a/ci/docker/install/ubuntu_arm_qemu.sh b/ci/docker/install/ubuntu_arm_qemu.sh index c30dc4f13..79ab67bfd 100755 --- a/ci/docker/install/ubuntu_arm_qemu.sh +++ b/ci/docker/install/ubuntu_arm_qemu.sh @@ -31,6 +31,7 @@ apt-get install -y \ qemu-system-arm \ unzip \ bzip2 \ - vim-nox + vim-nox \ + toilet -pip3 install ansible ipython +pip3 install ipython diff --git a/ci/docker/install/ubuntu_base.sh b/ci/docker/install/ubuntu_base.sh new file mode 100755 index 000000000..b34c0b3e1 --- /dev/null +++ b/ci/docker/install/ubuntu_base.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# build and install are separated so changes to build don't invalidate +# the whole docker cache for the image + +set -ex +apt-get update || true +apt-get install -y \ + build-essential \ + ca-certificates \ + cmake \ + curl \ + git \ + ninja-build \ + libgfortran3 \ + software-properties-common \ + sudo \ + unzip \ + wget diff --git a/ci/docker/install/ubuntu_caffe.sh b/ci/docker/install/ubuntu_caffe.sh index 8c1361211..eaa8ab9b1 100755 --- a/ci/docker/install/ubuntu_caffe.sh +++ b/ci/docker/install/ubuntu_caffe.sh @@ -18,6 +18,7 @@ # under the License. set -ex +apt-get update || true apt-get install -y \ libgflags-dev \ libgoogle-glog-dev \ diff --git a/ci/docker/install/ubuntu_clang.sh b/ci/docker/install/ubuntu_clang.sh index cb0f234a1..19aada9b3 100755 --- a/ci/docker/install/ubuntu_clang.sh +++ b/ci/docker/install/ubuntu_clang.sh @@ -21,6 +21,8 @@ # the whole docker cache for the image set -ex + +apt-get update || true # Install clang 3.9 (the same version as in XCode 8.*) and 6.0 (latest major release) wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add - && \ apt-add-repository "deb http://apt.llvm.org/xenial/ llvm-toolchain-xenial-3.9 main" && \ diff --git a/ci/docker/install/ubuntu_core.sh b/ci/docker/install/ubuntu_core.sh index 8a2ea30cd..4382aa6ae 100755 --- a/ci/docker/install/ubuntu_core.sh +++ b/ci/docker/install/ubuntu_core.sh @@ -21,12 +21,11 @@ # the whole docker cache for the image set -ex -apt-get update +apt-get update || true apt-get install -y \ apt-transport-https \ build-essential \ ca-certificates \ - cmake \ curl \ git \ libatlas-base-dev \ @@ -41,3 +40,11 @@ apt-get install -y \ sudo \ unzip \ wget + + +# Ubuntu 14.04 +if [[ $(lsb_release -r | grep 14.04) ]]; then + apt-get install -y cmake3 +else + apt-get install -y cmake +fi diff --git a/ci/docker/install/ubuntu_docs.sh b/ci/docker/install/ubuntu_docs.sh index a709b3de7..5dc201c15 100755 --- a/ci/docker/install/ubuntu_docs.sh +++ b/ci/docker/install/ubuntu_docs.sh @@ -23,6 +23,7 @@ set -ex # Install dependencies echo 'Installing dependencies...' +apt-get update || true apt-get install -y \ doxygen \ pandoc diff --git a/ci/docker/install/ubuntu_emscripten.sh b/ci/docker/install/ubuntu_emscripten.sh index e3d72caf0..28ede755f 100755 --- a/ci/docker/install/ubuntu_emscripten.sh +++ b/ci/docker/install/ubuntu_emscripten.sh @@ -25,6 +25,7 @@ set -ex +apt-get update || true apt-get -y install nodejs git clone -b 1.38.6 https://github.com/kripken/emscripten.git diff --git a/ci/docker/install/ubuntu_gcc8.sh b/ci/docker/install/ubuntu_gcc8.sh index 0a9dc6057..cd31f8213 100755 --- a/ci/docker/install/ubuntu_gcc8.sh +++ b/ci/docker/install/ubuntu_gcc8.sh @@ -19,5 +19,5 @@ sudo add-apt-repository ppa:jonathonf/gcc-8.0 sudo add-apt-repository ppa:jonathonf/gcc-7.3 -sudo apt-get update +sudo apt-get update || true sudo apt-get install -y gcc-8 g++-8 diff --git a/ci/docker/install/ubuntu_julia.sh b/ci/docker/install/ubuntu_julia.sh index 62013e36d..13093acc4 100755 --- a/ci/docker/install/ubuntu_julia.sh +++ b/ci/docker/install/ubuntu_julia.sh @@ -22,16 +22,22 @@ set -ex -export JLBINARY='julia.tar.gz' -export JULIADIR='/work/julia' -export JULIA="${JULIADIR}/bin/julia" +function install_julia() { + local suffix=`echo $1 | sed 's/\.//'` # 0.7 -> 07; 1.0 -> 10 + local JLBINARY="julia-$1.tar.gz" + local JULIADIR="/work/julia$suffix" + local JULIA="${JULIADIR}/bin/julia" -mkdir -p $JULIADIR -# The julia version in Ubuntu repo is too old -# We download the tarball from the official link: -# https://julialang.org/downloads/ -wget -O $JLBINARY https://julialang-s3.julialang.org/bin/linux/x64/0.6/julia-0.6.2-linux-x86_64.tar.gz -tar xzvf $JLBINARY -C $JULIADIR --strip 1 -rm $JLBINARY + mkdir -p $JULIADIR + # The julia version in Ubuntu repo is too old + # We download the tarball from the official link: + # https://julialang.org/downloads/ + wget -O $JLBINARY https://julialang-s3.julialang.org/bin/linux/x64/$1/julia-$2-linux-x86_64.tar.gz + tar xzvf $JLBINARY -C $JULIADIR --strip 1 + rm $JLBINARY -$JULIA -e 'versioninfo()' + $JULIA -e 'using InteractiveUtils; versioninfo()' +} + +install_julia 0.7 0.7.0 +install_julia 1.0 1.0.3 diff --git a/ci/docker/install/ubuntu_llvm.sh b/ci/docker/install/ubuntu_llvm.sh index 09e13d3d1..afd881eae 100755 --- a/ci/docker/install/ubuntu_llvm.sh +++ b/ci/docker/install/ubuntu_llvm.sh @@ -23,4 +23,5 @@ echo deb-src http://apt.llvm.org/xenial/ llvm-toolchain-xenial-5.0 main\ >> /etc/apt/sources.list.d/llvm.list wget -O - http://apt.llvm.org/llvm-snapshot.gpg.key|sudo apt-key add - -apt-get update && apt-get install -y --force-yes llvm-5.0 \ No newline at end of file +apt-get update || true +apt-get install -y --force-yes llvm-5.0 diff --git a/ci/docker/install/ubuntu_mkl.sh b/ci/docker/install/ubuntu_mkl.sh new file mode 100755 index 000000000..36fc7b07f --- /dev/null +++ b/ci/docker/install/ubuntu_mkl.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# build and install are separated so changes to build don't invalidate +# the whole docker cache for the image + +set -ex + +apt-get update || true +# Install Intel Math Kernel Library (latest major release) +# https://software.intel.com/en-us/articles/installing-intel-free-libs-and-python-apt-repo +wget -O - wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2019.PUB | apt-key add - && \ + sh -c 'echo deb https://apt.repos.intel.com/mkl all main > /etc/apt/sources.list.d/intel-mkl.list' && \ + apt-get update && \ + apt-get install -y intel-mkl-2019.1-053 diff --git a/ci/docker/install/ubuntu_mklml.sh b/ci/docker/install/ubuntu_mklml.sh index 7e17295f4..862e28464 100755 --- a/ci/docker/install/ubuntu_mklml.sh +++ b/ci/docker/install/ubuntu_mklml.sh @@ -21,5 +21,5 @@ # the whole docker cache for the image set -ex -wget -q --no-check-certificate -O /tmp/mklml.tgz https://github.com/intel/mkl-dnn/releases/download/v0.14/mklml_lnx_2018.0.3.20180406.tgz +wget -q --no-check-certificate -O /tmp/mklml.tgz https://github.com/intel/mkl-dnn/releases/download/v0.17-rc/mklml_lnx_2019.0.1.20180928.tgz tar -zxf /tmp/mklml.tgz && cp -rf mklml_*/* /usr/local/ && rm -rf mklml_* diff --git a/ci/docker/install/ubuntu_nightly_tests.sh b/ci/docker/install/ubuntu_nightly_tests.sh index 406985ea3..80028d87c 100755 --- a/ci/docker/install/ubuntu_nightly_tests.sh +++ b/ci/docker/install/ubuntu_nightly_tests.sh @@ -25,7 +25,7 @@ set -ex # Adding ppas frequently fails due to busy gpg servers, retry 5 times with 5 minute delays. for i in 1 2 3 4 5; do add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || sleep 300; done -apt-get update +apt-get update || true apt-get -y install time # Install for RAT License Check Nightly Test diff --git a/ci/docker/install/ubuntu_npm_blc.sh b/ci/docker/install/ubuntu_npm_blc.sh index 30fcb5a1b..59caa9f88 100755 --- a/ci/docker/install/ubuntu_npm_blc.sh +++ b/ci/docker/install/ubuntu_npm_blc.sh @@ -22,7 +22,7 @@ set -ex echo 'Installing npm...' -apt-get update +apt-get update || true apt-get install -y npm echo "Obtaining NodeJS version 8.x" diff --git a/ci/docker/install/ubuntu_nvidia.sh b/ci/docker/install/ubuntu_nvidia.sh index 7b16ed16f..7012b897f 100755 --- a/ci/docker/install/ubuntu_nvidia.sh +++ b/ci/docker/install/ubuntu_nvidia.sh @@ -18,10 +18,6 @@ # under the License. set -ex -apt install -y software-properties-common - -# Adding ppas frequently fails due to busy gpg servers, retry 5 times with 5 minute delays. -for i in 1 2 3 4 5; do add-apt-repository -y ppa:graphics-drivers && break || sleep 300; done # Retrieve ppa:graphics-drivers and install nvidia-drivers. # Note: DEBIAN_FRONTEND required to skip the interactive setup steps diff --git a/ci/docker/install/ubuntu_onnx.sh b/ci/docker/install/ubuntu_onnx.sh index 329352efd..0dad3f9ee 100755 --- a/ci/docker/install/ubuntu_onnx.sh +++ b/ci/docker/install/ubuntu_onnx.sh @@ -27,6 +27,7 @@ set -e set -x echo "Installing libprotobuf-dev and protobuf-compiler ..." +apt-get update || true apt-get install -y libprotobuf-dev protobuf-compiler echo "Installing pytest, pytest-cov, protobuf, Pillow, ONNX and tabulate ..." diff --git a/ci/docker/install/ubuntu_perl.sh b/ci/docker/install/ubuntu_perl.sh index 4d868f7d5..e04141eee 100755 --- a/ci/docker/install/ubuntu_perl.sh +++ b/ci/docker/install/ubuntu_perl.sh @@ -22,5 +22,6 @@ set -ex # install libraries for mxnet's perl package on ubuntu +apt-get update || true apt-get install -y libmouse-perl pdl cpanminus swig libgraphviz-perl cpanm -q Function::Parameters Hash::Ordered PDL::CCS diff --git a/ci/docker/install/ubuntu_publish.sh b/ci/docker/install/ubuntu_publish.sh new file mode 100644 index 000000000..1ad6ab947 --- /dev/null +++ b/ci/docker/install/ubuntu_publish.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Build on Ubuntu 14.04 LTS for LINUX CPU/GPU +apt-get update +apt-get install -y software-properties-common +add-apt-repository ppa:ubuntu-toolchain-r/test -y +add-apt-repository ppa:openjdk-r/ppa -y # Java lib +apt-get update +apt-get install -y git \ + cmake3 \ + libcurl4-openssl-dev \ + unzip \ + gcc-4.8 \ + g++-4.8 \ + gfortran \ + gfortran-4.8 \ + binutils \ + nasm \ + libtool \ + curl \ + wget \ + sudo \ + gnupg \ + gnupg2 \ + gnupg-agent \ + pandoc \ + python3-pip \ + automake \ + pkg-config \ + openjdk-8-jdk +curl -o apache-maven-3.3.9-bin.tar.gz http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz +tar xzf apache-maven-3.3.9-bin.tar.gz +mkdir /usr/local/maven +mv apache-maven-3.3.9/ /usr/local/maven/ +update-alternatives --install /usr/bin/mvn mvn /usr/local/maven/apache-maven-3.3.9/bin/mvn 1 +update-ca-certificates -f + +apt-get install -y python python3 + +# the version of the pip shipped with ubuntu may be too lower, install a recent version here +wget -nv https://bootstrap.pypa.io/get-pip.py +python3 get-pip.py +python2 get-pip.py + +apt-get remove -y python3-urllib3 + +pip2 install nose cpplint==1.3.0 pylint==1.9.3 'numpy<=1.15.2,>=1.8.2' nose-timer 'requests<2.19.0,>=2.18.4' h5py==2.8.0rc1 scipy==1.0.1 boto3 +pip3 install nose cpplint==1.3.0 pylint==2.1.1 'numpy<=1.15.2,>=1.8.2' nose-timer 'requests<2.19.0,>=2.18.4' h5py==2.8.0rc1 scipy==1.0.1 boto3 diff --git a/ci/docker/install/ubuntu_python.sh b/ci/docker/install/ubuntu_python.sh index d6e66aa45..ee05058f2 100755 --- a/ci/docker/install/ubuntu_python.sh +++ b/ci/docker/install/ubuntu_python.sh @@ -22,6 +22,7 @@ set -ex # install libraries for mxnet's python package on ubuntu +apt-get update || true apt-get install -y python-dev python3-dev virtualenv wget # the version of the pip shipped with ubuntu may be too lower, install a recent version here diff --git a/ci/docker/install/ubuntu_r.sh b/ci/docker/install/ubuntu_r.sh index 0e95601ea..cefc4172f 100755 --- a/ci/docker/install/ubuntu_r.sh +++ b/ci/docker/install/ubuntu_r.sh @@ -34,7 +34,7 @@ apt-key add r.gpg # Installing the latest version (3.3+) that is compatible with MXNet add-apt-repository 'deb [arch=amd64,i386] https://cran.rstudio.com/bin/linux/ubuntu xenial/' -apt-get update +apt-get update || true apt-get install -y --allow-unauthenticated \ libcairo2-dev \ libssl-dev \ diff --git a/ci/docker/install/ubuntu_rat.sh b/ci/docker/install/ubuntu_rat.sh index b131a0bb5..2c905fc27 100755 --- a/ci/docker/install/ubuntu_rat.sh +++ b/ci/docker/install/ubuntu_rat.sh @@ -20,7 +20,7 @@ set -ex echo "Install dependencies" -apt-get update +apt-get update || true apt-get install -y subversion maven openjdk-8-jdk openjdk-8-jre echo "download RAT" diff --git a/ci/docker/install/ubuntu_scala.sh b/ci/docker/install/ubuntu_scala.sh index bee0e6bba..5bade4746 100755 --- a/ci/docker/install/ubuntu_scala.sh +++ b/ci/docker/install/ubuntu_scala.sh @@ -24,17 +24,31 @@ set -ex cd "$(dirname "$0")" # install libraries for mxnet's scala package on ubuntu echo 'Installing Scala...' -apt-get install -y software-properties-common -apt-get update -apt-get install -y openjdk-8-jdk -apt-get install -y openjdk-8-jre -echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list -# ubuntu keyserver is very flaky -#apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 -#apt-key adv --keyserver keys.gnupg.net --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 -apt-key add sbt.gpg -apt-get update && apt-get install -y \ - maven \ - sbt \ +# Ubuntu 14.04 +if [[ $(lsb_release -r | grep 14.04) ]]; then + add-apt-repository -y ppa:openjdk-r/ppa +fi + +# All Ubuntu +apt-get update || true +apt-get install -y \ + openjdk-8-jdk \ + openjdk-8-jre \ + software-properties-common \ + gnupg \ + gnupg2 \ + gnupg-agent \ scala + +# Ubuntu 14.04 +if [[ $(lsb_release -r | grep 14.04) ]]; then + curl -o apache-maven-3.3.9-bin.tar.gz http://www.eu.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz + tar xzf apache-maven-3.3.9-bin.tar.gz + mkdir /usr/local/maven + mv apache-maven-3.3.9/ /usr/local/maven/ + update-alternatives --install /usr/bin/mvn mvn /usr/local/maven/apache-maven-3.3.9/bin/mvn 1 + update-ca-certificates -f +else + apt-get install -y maven +fi diff --git a/ci/docker/install/ubuntu_tutorials.sh b/ci/docker/install/ubuntu_tutorials.sh index 9a236bbf4..404d4bbf6 100755 --- a/ci/docker/install/ubuntu_tutorials.sh +++ b/ci/docker/install/ubuntu_tutorials.sh @@ -21,6 +21,7 @@ # the whole docker cache for the image set -ex +apt-get update || true apt-get install graphviz python-opencv -pip2 install jupyter matplotlib Pillow opencv-python scikit-learn graphviz tqdm -pip3 install jupyter matplotlib Pillow opencv-python scikit-learn graphviz tqdm +pip2 install jupyter matplotlib Pillow opencv-python scikit-learn graphviz tqdm mxboard +pip3 install jupyter matplotlib Pillow opencv-python scikit-learn graphviz tqdm mxboard diff --git a/ci/docker/qemu/runtime_functions.py b/ci/docker/qemu/runtime_functions.py index 6cf01b6f9..8b8e5acb5 100755 --- a/ci/docker/qemu/runtime_functions.py +++ b/ci/docker/qemu/runtime_functions.py @@ -33,6 +33,8 @@ import sys import types import glob +import vmcontrol +from vmcontrol import qemu_ssh, qemu_provision, qemu_rsync_to_host, VM def activate_this(base): import site @@ -53,21 +55,24 @@ def activate_this(base): sys.path.remove(item) sys.path[:0] = new_sys_path + + + def run_ut_py3_qemu(): + """Run unit tests in the emulator and copy the results back to the host through the mounted + volume in /mxnet""" from vmcontrol import VM with VM() as vm: - logging.info("VM provisioning with ansible") - check_call(["ansible-playbook", "-v", "-u", "qemu", "-i", "localhost:{},".format(vm.ssh_port), "playbook.yml"]) - logging.info("VM provisioned successfully.") - logging.info("sync tests") - check_call(['rsync', '-e', 'ssh -p{}'.format(vm.ssh_port), '-a', 'mxnet/tests', 'qemu@localhost:mxnet']) + qemu_provision(vm.ssh_port) logging.info("execute tests") - check_call(["ssh", "-o", "ServerAliveInterval=5", "-p{}".format(vm.ssh_port), "qemu@localhost", "./runtime_functions.py", "run_ut_python3_qemu_internal"]) + qemu_ssh(vm.ssh_port, "./runtime_functions.py", "run_ut_python3_qemu_internal") + qemu_rsync_to_host(vm.ssh_port, "*.xml", "mxnet") + logging.info("copied to host") logging.info("tests finished, vm shutdown.") vm.shutdown() def run_ut_python3_qemu_internal(): - """this runs inside the vm, it's run by the playbook above by ansible""" + """this runs inside the vm""" pkg = glob.glob('mxnet_dist/*.whl')[0] logging.info("=== NOW Running inside QEMU ===") logging.info("PIP Installing %s", pkg) @@ -75,8 +80,20 @@ def run_ut_python3_qemu_internal(): logging.info("PIP Installing mxnet/tests/requirements.txt") check_call(['sudo', 'pip3', 'install', '-r', 'mxnet/tests/requirements.txt']) logging.info("Running tests in mxnet/tests/python/unittest/") - check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_ndarray.py:test_ndarray_fluent']) + check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_engine.py']) + # Example to run a single unit test: + # check_call(['nosetests', '--with-timer', '--with-xunit', '--xunit-file', 'nosetests_unittest.xml', '--verbose', 'mxnet/tests/python/unittest/test_ndarray.py:test_ndarray_fluent']) + + + +def run_qemu_interactive(): + vm = VM(interactive=True) + vm.detach() + vm.start() + vm.wait() + logging.info("QEMU finished") +################################ def parsed_args(): parser = argparse.ArgumentParser(description="""python runtime functions""", epilog="") @@ -95,7 +112,7 @@ def chdir_to_script_directory(): os.chdir(base) def main(): - logging.getLogger().setLevel(logging.DEBUG) + logging.getLogger().setLevel(logging.INFO) logging.basicConfig(format='{}: %(asctime)-15s %(message)s'.format(script_name())) chdir_to_script_directory() diff --git a/ci/docker/qemu/vmcontrol.py b/ci/docker/qemu/vmcontrol.py index 2262bc77b..d80e22b1d 100644 --- a/ci/docker/qemu/vmcontrol.py +++ b/ci/docker/qemu/vmcontrol.py @@ -42,6 +42,8 @@ # # The VMs are provisioned after boot, tests are run and then they are stopped # +QEMU_SSH_PORT=2222 +QEMU_RAM=4096 QEMU_RUN=""" qemu-system-arm -M virt -m {ram} \ @@ -55,17 +57,72 @@ -display none -nographic """ +QEMU_RUN_INTERACTIVE=""" +qemu-system-arm -M virt -m {ram} \ + -kernel vmlinuz \ + -initrd initrd.img \ + -append 'root=/dev/vda1' \ + -drive if=none,file=vda.qcow2,format=qcow2,id=hd \ + -device virtio-blk-device,drive=hd \ + -netdev user,id=mynet,hostfwd=tcp::{ssh_port}-:22 \ + -device virtio-net-device,netdev=mynet \ + -nographic +""" + +def retry(target_exception, tries=4, delay_s=1, backoff=2): + """Retry calling the decorated function using an exponential backoff. + + http://www.saltycrane.com/blog/2009/11/trying-out-retry-decorator-python/ + original from: http://wiki.python.org/moin/PythonDecoratorLibrary#Retry + + :param target_exception: the exception to check. may be a tuple of + exceptions to check + :type target_exception: Exception or tuple + :param tries: number of times to try (not retry) before giving up + :type tries: int + :param delay_s: initial delay between retries in seconds + :type delay_s: int + :param backoff: backoff multiplier e.g. value of 2 will double the delay + each retry + :type backoff: int + """ + import time + from functools import wraps + + def decorated_retry(f): + @wraps(f) + def f_retry(*args, **kwargs): + mtries, mdelay = tries, delay_s + while mtries > 1: + try: + return f(*args, **kwargs) + except target_exception as e: + logging.warning("Exception: %s, Retrying in %d seconds...", str(e), mdelay) + time.sleep(mdelay) + mtries -= 1 + mdelay *= backoff + return f(*args, **kwargs) + + return f_retry # true decorator + + return decorated_retry + + + + class VMError(RuntimeError): pass class VM: """Control of the virtual machine""" - def __init__(self, ssh_port=2222): + def __init__(self, ssh_port=QEMU_SSH_PORT, ram=QEMU_RAM, interactive=False): self.log = logging.getLogger(VM.__name__) self.ssh_port = ssh_port self.timeout_s = 300 self.qemu_process = None self._detach = False + self._interactive = interactive + self.ram = ram def __enter__(self): self.start() @@ -77,13 +134,22 @@ def __exit__(self, exc_type, exc_value, traceback): self.terminate() def start(self): - self.log.info("Starting VM, ssh port redirected to localhost:%s", self.ssh_port) + sys.stderr.flush() + call(['toilet', '-f', 'smbraille', 'Starting QEMU']) + sys.stdout.flush() + self.log.info("Starting VM, ssh port redirected to localhost:%s (inside docker, not exposed by default)", self.ssh_port) if self.is_running(): raise VMError("VM is running, shutdown first") - self.qemu_process = run_qemu(self.ssh_port) + if self._interactive: + self.qemu_process = Popen(shlex.split(QEMU_RUN_INTERACTIVE.format(ssh_port=self.ssh_port, ram=self.ram))) + return + else: + self.log.info("Starting in non-interactive mode. Terminal output is disabled.") + self.qemu_process = Popen(shlex.split(QEMU_RUN.format(ssh_port=self.ssh_port, ram=self.ram)), stdout=DEVNULL, stdin=DEVNULL, stderr=PIPE) def keep_waiting(): return self.is_running() + logging.info("waiting for ssh to be open in the VM (timeout {}s)".format(self.timeout_s)) ssh_working = wait_ssh_open('127.0.0.1', self.ssh_port, keep_waiting, self.timeout_s) if not self.is_running(): @@ -140,11 +206,30 @@ def __del__(self): logging.info("VM destructor hit") self.terminate() -def run_qemu(ssh_port=2222): - cmd = QEMU_RUN.format(ssh_port=ssh_port, ram=4096) - logging.info("QEMU command: %s", cmd) - qemu_process = Popen(shlex.split(cmd), stdout=DEVNULL, stdin=DEVNULL, stderr=PIPE) - return qemu_process + +def qemu_ssh(ssh_port=QEMU_SSH_PORT, *args): + check_call(["ssh", "-o", "ServerAliveInterval=5", "-o", "StrictHostKeyChecking=no", "-p{}".format(ssh_port), "qemu@localhost", *args]) + + +def qemu_rsync(ssh_port, local_path, remote_path): + check_call(['rsync', '-e', 'ssh -o StrictHostKeyChecking=no -p{}'.format(ssh_port), '-a', local_path, 'qemu@localhost:{}'.format(remote_path)]) + +def qemu_rsync_to_host(ssh_port, remote_path, local_path): + check_call(['rsync', '-e', 'ssh -o StrictHostKeyChecking=no -p{}'.format(ssh_port), '-va', 'qemu@localhost:{}'.format(remote_path), local_path]) + + +@retry(subprocess.CalledProcessError) +def qemu_provision(ssh_port=QEMU_SSH_PORT): + import glob + logging.info("Provisioning the VM with artifacts and sources") + + artifact = glob.glob('/work/mxnet/build/*.whl') + for x in artifact: + qemu_rsync(ssh_port, x, 'mxnet_dist/') + qemu_rsync(ssh_port, '/work/runtime_functions.py','') + qemu_rsync(ssh_port, '/work/vmcontrol.py','') + qemu_rsync(ssh_port, 'mxnet/tests', 'mxnet') + logging.info("Provisioning completed successfully.") def wait_ssh_open(server, port, keep_waiting=None, timeout=None): @@ -159,7 +244,7 @@ def wait_ssh_open(server, port, keep_waiting=None, timeout=None): import errno import time log = logging.getLogger('wait_ssh_open') - sleep_s = 0 + sleep_s = 1 if timeout: from time import time as now # time module is needed to calc timeout shared between two exceptions @@ -183,7 +268,7 @@ def wait_ssh_open(server, port, keep_waiting=None, timeout=None): log.debug("connect timeout %d s", next_timeout) s.settimeout(next_timeout) - log.info("connect %s:%d", server, port) + log.debug("connect %s:%d", server, port) s.connect((server, port)) ret = s.recv(1024).decode() if ret and ret.startswith('SSH'): diff --git a/ci/docker/runtime_functions.sh b/ci/docker/runtime_functions.sh index 43006f239..6c3f99948 100755 --- a/ci/docker/runtime_functions.sh +++ b/ci/docker/runtime_functions.sh @@ -23,6 +23,7 @@ set -ex NOSE_COVERAGE_ARGUMENTS="--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet" +NOSE_TIMER_ARGUMENTS="--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error" CI_CUDA_COMPUTE_CAPABILITIES="-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70" CI_CMAKE_CUDA_ARCH_BIN="52,70" @@ -35,35 +36,67 @@ clean_repo() { git submodule update --init --recursive } +scala_prepare() { + # Clean up maven logs + export MAVEN_OPTS="-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn" +} + build_ccache_wrappers() { set -ex - rm -f cc - rm -f cxx - - touch cc - touch cxx - if [ -z ${CC+x} ]; then echo "No \$CC set, defaulting to gcc"; export CC=gcc fi - - if [ -z ${CXX+x} ]; then + if [ -z ${CXX+x} ]; then echo "No \$CXX set, defaulting to g++"; export CXX=g++ fi - # this function is nessesary for cuda enabled make based builds, since nvcc needs just an executable for -ccbin - - echo -e "#!/bin/sh\n/usr/local/bin/ccache ${CC} \"\$@\"\n" >> cc - echo -e "#!/bin/sh\n/usr/local/bin/ccache ${CXX} \"\$@\"\n" >> cxx - - chmod +x cc - chmod +x cxx - - export CC=`pwd`/cc - export CXX=`pwd`/cxx + # Recommended by CCache: https://ccache.samba.org/manual.html#_run_modes + # Add to the beginning of path to ensure this redirection is picked up instead + # of the original ones. Especially CUDA/NVCC appends itself to the beginning of the + # path and thus this redirect is ignored. This change fixes this problem + # This hacky approach with symbolic links is required because underlying build + # systems of our submodules ignore our CMake settings. If they use Makefile, + # we can't influence them at all in general and NVCC also prefers to hardcode their + # compiler instead of respecting the settings. Thus, we take this brutal approach + # and just redirect everything of this installer has been called. + # In future, we could do these links during image build time of the container. + # But in the beginning, we'll make this opt-in. In future, loads of processes like + # the scala make step or numpy compilation and other pip package generations + # could be heavily sped up by using ccache as well. + mkdir /tmp/ccache-redirects + export PATH=/tmp/ccache-redirects:$PATH + ln -s ccache /tmp/ccache-redirects/gcc + ln -s ccache /tmp/ccache-redirects/gcc-8 + ln -s ccache /tmp/ccache-redirects/g++ + ln -s ccache /tmp/ccache-redirects/g++-8 + ln -s ccache /tmp/ccache-redirects/nvcc + ln -s ccache /tmp/ccache-redirects/clang++-3.9 + ln -s ccache /tmp/ccache-redirects/clang-3.9 + ln -s ccache /tmp/ccache-redirects/clang++-5.0 + ln -s ccache /tmp/ccache-redirects/clang-5.0 + ln -s ccache /tmp/ccache-redirects/clang++-6.0 + ln -s ccache /tmp/ccache-redirects/clang-6.0 + ln -s ccache /usr/local/bin/gcc + ln -s ccache /usr/local/bin/gcc-8 + ln -s ccache /usr/local/bin/g++ + ln -s ccache /usr/local/bin/g++-8 + ln -s ccache /usr/local/bin/nvcc + ln -s ccache /usr/local/bin/clang++-3.9 + ln -s ccache /usr/local/bin/clang-3.9 + ln -s ccache /usr/local/bin/clang++-5.0 + ln -s ccache /usr/local/bin/clang-5.0 + ln -s ccache /usr/local/bin/clang++-6.0 + ln -s ccache /usr/local/bin/clang-6.0 + + export NVCC=ccache + + # Uncomment if you would like to debug CCache hit rates. + # You can monitor using tail -f ccache-log + # export CCACHE_LOGFILE=/work/mxnet/ccache-log + # export CCACHE_DEBUG=1 } build_wheel() { @@ -105,6 +138,8 @@ build_jetson() { set -ex pushd . + #build_ccache_wrappers + cp make/crosscompile.jetson.mk ./config.mk make -j$(nproc) @@ -128,6 +163,7 @@ build_armv6() { # We do not need OpenMP, since most armv6 systems have only 1 core + build_ccache_wrappers cmake \ -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ @@ -158,6 +194,7 @@ build_armv7() { # file tries to add -llapack. Lapack functionality though, requires -lgfortran # to be linked additionally. + build_ccache_wrappers cmake \ -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE} \ -DCMAKE_CROSSCOMPILING=ON \ @@ -180,6 +217,7 @@ build_armv7() { } build_armv8() { + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ @@ -204,6 +242,7 @@ build_armv8() { build_android_armv7() { set -ex cd /work/build + build_ccache_wrappers cmake \ -DANDROID=ON\ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ @@ -224,6 +263,7 @@ build_android_armv7() { build_android_armv8() { set -ex cd /work/build + build_ccache_wrappers cmake\ -DANDROID=ON \ -DUSE_CUDA=OFF\ @@ -243,19 +283,21 @@ build_centos7_cpu() { cd /work/mxnet export CC="ccache gcc" export CXX="ccache g++" - + build_ccache_wrappers make \ DEV=1 \ USE_LAPACK=1 \ ENABLE_TESTCOVERAGE=1 \ USE_LAPACK_PATH=/usr/lib64/liblapack.so \ USE_BLAS=openblas \ + USE_MKLDNN=0 \ USE_DIST_KVSTORE=1 \ -j$(nproc) } build_amzn_linux_cpu() { cd /work/build + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ @@ -272,19 +314,17 @@ build_amzn_linux_cpu() { ninja -v } - build_centos7_mkldnn() { set -ex cd /work/mxnet export CC="ccache gcc" export CXX="ccache g++" - + build_ccache_wrappers make \ DEV=1 \ ENABLE_TESTCOVERAGE=1 \ USE_LAPACK=1 \ USE_LAPACK_PATH=/usr/lib64/liblapack.so \ - USE_MKLDNN=1 \ USE_BLAS=openblas \ -j$(nproc) } @@ -293,13 +333,14 @@ build_centos7_gpu() { set -ex cd /work/mxnet # unfortunately this build has problems in 3rdparty dependencies with ccache and make - # build_ccache_wrappers + build_ccache_wrappers make \ DEV=1 \ ENABLE_TESTCOVERAGE=1 \ USE_LAPACK=1 \ USE_LAPACK_PATH=/usr/lib64/liblapack.so \ USE_BLAS=openblas \ + USE_MKLDNN=0 \ USE_CUDA=1 \ USE_CUDA_PATH=/usr/local/cuda \ USE_CUDNN=1 \ @@ -313,6 +354,21 @@ build_ubuntu_cpu() { } build_ubuntu_cpu_openblas() { + set -ex + export CC="gcc" + export CXX="g++" + build_ccache_wrappers + make \ + DEV=1 \ + ENABLE_TESTCOVERAGE=1 \ + USE_CPP_PACKAGE=1 \ + USE_BLAS=openblas \ + USE_MKLDNN=0 \ + USE_DIST_KVSTORE=1 \ + -j$(nproc) +} + +build_ubuntu_cpu_mkl() { set -ex export CC="ccache gcc" export CXX="ccache g++" @@ -320,7 +376,9 @@ build_ubuntu_cpu_openblas() { DEV=1 \ ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ - USE_BLAS=openblas \ + USE_BLAS=mkl \ + USE_MKLDNN=0 \ + USE_INTEL_PATH=/opt/intel \ USE_DIST_KVSTORE=1 \ -j$(nproc) } @@ -329,6 +387,7 @@ build_ubuntu_cpu_cmake_debug() { set -ex pushd . cd /work/build + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ @@ -337,6 +396,7 @@ build_ubuntu_cpu_cmake_debug() { -DUSE_MKL_IF_AVAILABLE=OFF \ -DUSE_OPENMP=OFF \ -DUSE_OPENCV=ON \ + -DUSE_SIGNAL_HANDLER=ON \ -DCMAKE_BUILD_TYPE=Debug \ -G Ninja \ /work/mxnet @@ -350,13 +410,15 @@ build_ubuntu_cpu_cmake_asan() { pushd . cd /work/build + export CXX=g++-8 + export CC=gcc-8 + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ - -DCMAKE_CXX_COMPILER=/usr/bin/g++-8 \ - -DCMAKE_C_COMPILER=/usr/bin/gcc-8 \ -DUSE_CUDA=OFF \ -DUSE_MKL_IF_AVAILABLE=OFF \ + -DUSE_MKLDNN=OFF \ -DUSE_OPENMP=OFF \ -DUSE_OPENCV=OFF \ -DCMAKE_BUILD_TYPE=Debug \ @@ -376,13 +438,14 @@ build_ubuntu_cpu_cmake_asan() { build_ubuntu_cpu_clang39() { set -ex - export CXX=clang++-3.9 + export CXX=clang++-3.9 export CC=clang-3.9 - build_ccache_wrappers - make \ + build_ccache_wrappers + make \ ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ + USE_MKLDNN=0 \ USE_OPENMP=0 \ USE_DIST_KVSTORE=1 \ -j$(nproc) @@ -400,6 +463,7 @@ build_ubuntu_cpu_clang60() { ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ + USE_MKLDNN=0 \ USE_OPENMP=1 \ USE_DIST_KVSTORE=1 \ -j$(nproc) @@ -414,10 +478,12 @@ build_ubuntu_cpu_clang_tidy() { pushd . cd /work/build + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DUSE_CUDA=OFF \ + -DUSE_MKLDNN=OFF \ -DUSE_MKL_IF_AVAILABLE=OFF \ -DUSE_OPENCV=ON \ -DCMAKE_BUILD_TYPE=Debug \ @@ -443,7 +509,6 @@ build_ubuntu_cpu_clang39_mkldnn() { ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ - USE_MKLDNN=1 \ USE_OPENMP=0 \ -j$(nproc) } @@ -460,7 +525,6 @@ build_ubuntu_cpu_clang60_mkldnn() { ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ - USE_MKLDNN=1 \ USE_OPENMP=1 \ -j$(nproc) } @@ -475,7 +539,19 @@ build_ubuntu_cpu_mkldnn() { ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ - USE_MKLDNN=1 \ + -j$(nproc) +} + +build_ubuntu_cpu_mkldnn_mkl() { + set -ex + + build_ccache_wrappers + + make \ + DEV=1 \ + ENABLE_TESTCOVERAGE=1 \ + USE_CPP_PACKAGE=1 \ + USE_BLAS=mkl \ -j$(nproc) } @@ -497,6 +573,8 @@ build_ubuntu_gpu_tensorrt() { mkdir -p build cd build cmake \ + -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DCMAKE_CXX_FLAGS=-I/usr/include/python${PYVER}\ -DBUILD_SHARED_LIBS=ON ..\ -G Ninja @@ -511,7 +589,10 @@ build_ubuntu_gpu_tensorrt() { cd 3rdparty/onnx-tensorrt/ mkdir -p build cd build - cmake .. + cmake \ + -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ + -DCMAKE_C_COMPILER_LAUNCHER=ccache \ + .. make -j$(nproc) export LIBRARY_PATH=`pwd`:$LIBRARY_PATH popd @@ -530,6 +611,7 @@ build_ubuntu_gpu_tensorrt() { USE_CUDA_PATH=/usr/local/cuda \ USE_CUDNN=1 \ USE_OPENCV=0 \ + USE_MKLDNN=0 \ USE_DIST_KVSTORE=0 \ USE_TENSORRT=1 \ USE_JEMALLOC=0 \ @@ -549,7 +631,6 @@ build_ubuntu_gpu_mkldnn() { ENABLE_TESTCOVERAGE=1 \ USE_CPP_PACKAGE=1 \ USE_BLAS=openblas \ - USE_MKLDNN=1 \ USE_CUDA=1 \ USE_CUDA_PATH=/usr/local/cuda \ USE_CUDNN=1 \ @@ -566,7 +647,6 @@ build_ubuntu_gpu_mkldnn_nocudnn() { DEV=1 \ ENABLE_TESTCOVERAGE=1 \ USE_BLAS=openblas \ - USE_MKLDNN=1 \ USE_CUDA=1 \ USE_CUDA_PATH=/usr/local/cuda \ USE_CUDNN=0 \ @@ -582,6 +662,7 @@ build_ubuntu_gpu_cuda91_cudnn7() { DEV=1 \ ENABLE_TESTCOVERAGE=1 \ USE_BLAS=openblas \ + USE_MKLDNN=0 \ USE_CUDA=1 \ USE_CUDA_PATH=/usr/local/cuda \ USE_CUDNN=1 \ @@ -594,6 +675,7 @@ build_ubuntu_gpu_cuda91_cudnn7() { build_ubuntu_amalgamation() { set -ex # Amalgamation can not be run with -j nproc + build_ccache_wrappers make -C amalgamation/ clean make -C amalgamation/ \ USE_BLAS=openblas \ @@ -603,6 +685,7 @@ build_ubuntu_amalgamation() { build_ubuntu_amalgamation_min() { set -ex # Amalgamation can not be run with -j nproc + build_ccache_wrappers make -C amalgamation/ clean make -C amalgamation/ \ USE_BLAS=openblas \ @@ -613,14 +696,16 @@ build_ubuntu_amalgamation_min() { build_ubuntu_gpu_cmake_mkldnn() { set -ex cd /work/build + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \ + -DUSE_SIGNAL_HANDLER=ON \ -DENABLE_TESTCOVERAGE=ON \ -DUSE_CUDA=1 \ -DUSE_CUDNN=1 \ -DUSE_MKLML_MKL=1 \ - -DUSE_MKLDNN=1 \ -DCMAKE_BUILD_TYPE=Release \ -DCUDA_ARCH_NAME=Manual \ -DCUDA_ARCH_BIN=$CI_CMAKE_CUDA_ARCH_BIN \ @@ -636,15 +721,19 @@ build_ubuntu_gpu_cmake_mkldnn() { build_ubuntu_gpu_cmake() { set -ex cd /work/build + build_ccache_wrappers cmake \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ -DCMAKE_C_COMPILER_LAUNCHER=ccache \ + -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache \ + -DUSE_SIGNAL_HANDLER=ON \ -DENABLE_TESTCOVERAGE=ON \ - -DUSE_CUDA=1 \ - -DUSE_CUDNN=1 \ - -DUSE_MKLML_MKL=0 \ - -DUSE_MKLDNN=0 \ - -DUSE_DIST_KVSTORE=1 \ + -DUSE_CUDA=ON \ + -DUSE_CUDNN=ON \ + -DUSE_MKL_IF_AVAILABLE=OFF \ + -DUSE_MKLML_MKL=OFF \ + -DUSE_MKLDNN=OFF \ + -DUSE_DIST_KVSTORE=ON \ -DCMAKE_BUILD_TYPE=Release \ -DCUDA_ARCH_NAME=Manual \ -DCUDA_ARCH_BIN=$CI_CMAKE_CUDA_ARCH_BIN \ @@ -674,9 +763,9 @@ unittest_ubuntu_python2_cpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest - nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_train.xml --verbose tests/python/train - nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_quantization.xml --verbose tests/python/quantization + nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest + nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_train.xml --verbose tests/python/train + nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_quantization.xml --verbose tests/python/quantization } unittest_ubuntu_python3_cpu() { @@ -684,8 +773,8 @@ unittest_ubuntu_python3_cpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_quantization.xml --verbose tests/python/quantization + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_quantization.xml --verbose tests/python/quantization } unittest_ubuntu_python3_cpu_mkldnn() { @@ -693,8 +782,8 @@ unittest_ubuntu_python3_cpu_mkldnn() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_mkl.xml --verbose tests/python/mkl + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_mkl.xml --verbose tests/python/mkl } unittest_ubuntu_python2_gpu() { @@ -702,31 +791,8 @@ unittest_ubuntu_python2_gpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu -} - -tutorialtest_ubuntu_python3_gpu() { - set -ex - cd /work/mxnet/docs - export MXNET_DOCS_BUILD_MXNET=0 - make html - export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - export PYTHONPATH=/work/mxnet/python/ - export MXNET_TUTORIAL_TEST_KERNEL=python3 - cd /work/mxnet/tests/tutorials - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_tutorials.xml test_tutorials.py --nologcapture -} - -tutorialtest_ubuntu_python2_gpu() { - set -ex - cd /work/mxnet/docs - export MXNET_DOCS_BUILD_MXNET=0 - make html - export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - export PYTHONPATH=/work/mxnet/python/ - export MXNET_TUTORIAL_TEST_KERNEL=python2 - cd /work/mxnet/tests/tutorials - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_tutorials.xml test_tutorials.py --nologcapture + export CUDNN_VERSION=7.0.3 + nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu } unittest_ubuntu_python3_gpu() { @@ -734,7 +800,8 @@ unittest_ubuntu_python3_gpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu + export CUDNN_VERSION=7.0.3 + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu } unittest_ubuntu_python3_gpu_nocudnn() { @@ -742,7 +809,7 @@ unittest_ubuntu_python3_gpu_nocudnn() { export PYTHONPATH=./python/ export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 export CUDNN_OFF_TEST_ONLY=true - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu } unittest_ubuntu_tensorrt_gpu() { @@ -750,8 +817,9 @@ unittest_ubuntu_tensorrt_gpu() { export PYTHONPATH=./python/ export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 export LD_LIBRARY_PATH=/work/mxnet/lib:$LD_LIBRARY_PATH + export CUDNN_VERSION=7.0.3 python tests/python/tensorrt/lenet5_train.py - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_trt_gpu.xml --verbose --nocapture tests/python/tensorrt/ + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_trt_gpu.xml --verbose --nocapture tests/python/tensorrt/ } # quantization gpu currently only runs on P3 instances @@ -761,7 +829,8 @@ unittest_ubuntu_python2_quantization_gpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_quantization_gpu.xml --verbose tests/python/quantization_gpu + export CUDNN_VERSION=7.0.3 + nosetests-2.7 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_quantization_gpu.xml --verbose tests/python/quantization_gpu } # quantization gpu currently only runs on P3 instances @@ -771,19 +840,31 @@ unittest_ubuntu_python3_quantization_gpu() { export PYTHONPATH=./python/ export MXNET_MKLDNN_DEBUG=1 # Ignored if not present export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 - nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_quantization_gpu.xml --verbose tests/python/quantization_gpu + export CUDNN_VERSION=7.0.3 + nosetests-3.4 $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_quantization_gpu.xml --verbose tests/python/quantization_gpu } unittest_ubuntu_cpu_scala() { set -ex - make scalapkg USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1 - make scalaunittest USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1 + scala_prepare + cd scala-package + mvn -B integration-test +} + +unittest_centos7_cpu_scala() { + set -ex + cd /work/mxnet + scala_prepare + cd scala-package + mvn -B integration-test } unittest_ubuntu_cpu_clojure() { set -ex - make scalapkg USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1 - make scalainstall USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1 + scala_prepare + cd scala-package + mvn -B install + cd .. ./contrib/clojure-package/ci-test.sh } @@ -792,7 +873,7 @@ unittest_ubuntu_cpugpu_perl() { ./perl-package/test.sh } -unittest_ubuntu_gpu_cpp() { +unittest_cpp() { set -ex build/tests/mxnet_unit_tests } @@ -826,57 +907,67 @@ unittest_ubuntu_gpu_R() { make rpkgtest R_LIBS=/tmp/r-site-library R_GPU_ENABLE=1 } -unittest_ubuntu_cpu_julia06() { +unittest_ubuntu_cpu_julia() { set -ex - export PATH="/work/julia/bin:$PATH" + export PATH="$1/bin:$PATH" export MXNET_HOME='/work/mxnet' - export JULIA_PKGDIR='/work/julia-pkg' - export DEPDIR=`julia -e 'print(Pkg.dir())'` + export JULIA_DEPOT_PATH='/work/julia-depot' + export DEVDIR="$JULIA_DEPOT_PATH/dev" - julia -e 'versioninfo()' - julia -e 'Pkg.init()' + julia -e 'using InteractiveUtils; versioninfo()' # install package - ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet + mkdir -p $DEVDIR + ln -sf ${MXNET_HOME}/julia ${DEVDIR}/MXNet - # install dependencies - julia -e 'Pkg.resolve()' + # register MXNet.jl and dependencies + julia -e 'using Pkg; Pkg.develop("MXNet")' # FIXME export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so' + export LD_LIBRARY_PATH=/work/mxnet/lib:$LD_LIBRARY_PATH # use the prebuilt binary from $MXNET_HOME/lib - julia -e 'Pkg.build("MXNet")' + julia -e 'using Pkg; Pkg.build("MXNet")' # run the script `julia/test/runtests.jl` - julia -e 'Pkg.test("MXNet")' + julia -e 'using Pkg; Pkg.test("MXNet")' # See https://github.com/dmlc/MXNet.jl/pull/303#issuecomment-341171774 julia -e 'using MXNet; mx._sig_checker()' } +unittest_ubuntu_cpu_julia07() { + set -ex + unittest_ubuntu_cpu_julia /work/julia07 +} + +unittest_ubuntu_cpu_julia10() { + set -ex + unittest_ubuntu_cpu_julia /work/julia10 +} + unittest_centos7_cpu() { set -ex cd /work/mxnet - python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest - python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_train.xml --verbose tests/python/train + python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_unittest.xml --verbose tests/python/unittest + python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_train.xml --verbose tests/python/train } unittest_centos7_gpu() { set -ex cd /work/mxnet - python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu + export CUDNN_VERSION=7.0.3 + python3.6 -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_gpu.xml --verbose tests/python/gpu } integrationtest_ubuntu_cpu_onnx() { set -ex export PYTHONPATH=./python/ - python example/onnx/super_resolution.py - pytest tests/python-pytest/onnx/import/mxnet_backend_test.py - pytest tests/python-pytest/onnx/import/onnx_import_test.py - pytest tests/python-pytest/onnx/import/gluon_backend_test.py - pytest tests/python-pytest/onnx/export/onnx_backend_test.py - python tests/python-pytest/onnx/export/mxnet_export_test.py + python tests/python-pytest/onnx/backend_test.py + pytest tests/python-pytest/onnx/mxnet_export_test.py + pytest tests/python-pytest/onnx/test_models.py + pytest tests/python-pytest/onnx/test_node.py } integrationtest_ubuntu_gpu_python() { @@ -927,8 +1018,10 @@ integrationtest_ubuntu_cpu_dist_kvstore() { integrationtest_ubuntu_gpu_scala() { set -ex - make scalapkg USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_DIST_KVSTORE=1 SCALA_ON_GPU=1 ENABLE_TESTCOVERAGE=1 - make scalaintegrationtest USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 SCALA_TEST_ON_GPU=1 USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1 + scala_prepare + cd scala-package + export SCALA_TEST_ON_GPU=1 + mvn -B integration-test -DskipTests=false } integrationtest_ubuntu_gpu_dist_kvstore() { @@ -953,7 +1046,7 @@ test_ubuntu_cpu_python2() { cd /work/mxnet/python pip install -e . cd /work/mxnet - python -m "nose" $NOSE_COVERAGE_ARGUMENTS --with-timer --verbose tests/python/unittest + python -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --verbose tests/python/unittest popd } @@ -969,7 +1062,7 @@ test_ubuntu_cpu_python3() { pip3 install nose nose-timer pip3 install -e . cd /work/mxnet - python3 -m "nose" $NOSE_COVERAGE_ARGUMENTS --with-timer --verbose tests/python/unittest + python3 -m "nose" $NOSE_COVERAGE_ARGUMENTS $NOSE_TIMER_ARGUMENTS --verbose tests/python/unittest popd } @@ -992,7 +1085,6 @@ build_docs() { popd } - # Functions that run the nightly Tests: #Runs Apache RAT Check on MXNet Source for License Headers @@ -1086,7 +1178,7 @@ nightly_straight_dope_python2_single_gpu_tests() { cd /work/mxnet/tests/nightly/straight_dope export PYTHONPATH=/work/mxnet/python/ export MXNET_TEST_KERNEL=python2 - nosetests-2.7 --with-xunit --xunit-file nosetests_straight_dope_python2_single_gpu.xml \ + nosetests-2.7 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_straight_dope_python2_single_gpu.xml \ test_notebooks_single_gpu.py --nologcapture } @@ -1095,7 +1187,7 @@ nightly_straight_dope_python3_single_gpu_tests() { cd /work/mxnet/tests/nightly/straight_dope export PYTHONPATH=/work/mxnet/python/ export MXNET_TEST_KERNEL=python3 - nosetests-3.4 --with-xunit --xunit-file nosetests_straight_dope_python3_single_gpu.xml \ + nosetests-3.4 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_straight_dope_python3_single_gpu.xml \ test_notebooks_single_gpu.py --nologcapture } @@ -1105,7 +1197,7 @@ nightly_straight_dope_python2_multi_gpu_tests() { cd /work/mxnet/tests/nightly/straight_dope export PYTHONPATH=/work/mxnet/python/ export MXNET_TEST_KERNEL=python2 - nosetests-2.7 --with-xunit --xunit-file nosetests_straight_dope_python2_multi_gpu.xml \ + nosetests-2.7 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_straight_dope_python2_multi_gpu.xml \ test_notebooks_multi_gpu.py --nologcapture } @@ -1114,46 +1206,107 @@ nightly_straight_dope_python3_multi_gpu_tests() { cd /work/mxnet/tests/nightly/straight_dope export PYTHONPATH=/work/mxnet/python/ export MXNET_TEST_KERNEL=python3 - nosetests-3.4 --with-xunit --xunit-file nosetests_straight_dope_python3_multi_gpu.xml \ + nosetests-3.4 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_straight_dope_python3_multi_gpu.xml \ test_notebooks_multi_gpu.py --nologcapture } +nightly_tutorial_test_ubuntu_python3_gpu() { + set -ex + cd /work/mxnet/docs + export BUILD_VER=tutorial + export MXNET_DOCS_BUILD_MXNET=0 + make html + export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 + export PYTHONPATH=/work/mxnet/python/ + export MXNET_TUTORIAL_TEST_KERNEL=python3 + cd /work/mxnet/tests/tutorials + nosetests-3.4 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_tutorials.xml test_tutorials.py --nologcapture +} + +nightly_tutorial_test_ubuntu_python2_gpu() { + set -ex + cd /work/mxnet/docs + export BUILD_VER=tutorial + export MXNET_DOCS_BUILD_MXNET=0 + make html + export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 + export PYTHONPATH=/work/mxnet/python/ + export MXNET_TUTORIAL_TEST_KERNEL=python2 + cd /work/mxnet/tests/tutorials + nosetests-3.4 $NOSE_TIMER_ARGUMENTS --with-xunit --xunit-file nosetests_tutorials.xml test_tutorials.py --nologcapture +} + +nightly_java_demo_test_cpu() { + set -ex + cd /work/mxnet/scala-package/mxnet-demo/java-demo + make java_ci_demo + bash bin/java_sample.sh + bash bin/run_od.sh +} + +nightly_scala_demo_test_cpu() { + set -ex + cd /work/mxnet/scala-package/mxnet-demo/scala-demo + make scala_ci_demo + bash bin/demo.sh + bash bin/run_im.sh +} + # Deploy deploy_docs() { set -ex pushd . - make docs + make docs SPHINXOPTS=-W popd } deploy_jl_docs() { set -ex - export PATH="/work/julia/bin:$PATH" + export PATH="/work/julia10/bin:$PATH" export MXNET_HOME='/work/mxnet' - export JULIA_PKGDIR='/work/julia-pkg' - export DEPDIR=`julia -e 'print(Pkg.dir())'` + export JULIA_DEPOT_PATH='/work/julia-depot' + export DEVDIR="$JULIA_DEPOT_PATH/dev" - julia -e 'versioninfo()' - julia -e 'Pkg.init()' - ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet - julia -e 'Pkg.resolve()' + julia -e 'using InteractiveUtils; versioninfo()' + mkdir -p $DEVDIR # FIXME export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so' + export LD_LIBRARY_PATH=/work/mxnet/lib:$LD_LIBRARY_PATH - # use the prebuilt binary from $MXNET_HOME/lib - julia -e 'Pkg.build("MXNet")' - # build docs - julia -e 'Pkg.add("Documenter")' - julia -e 'cd(Pkg.dir("MXNet")); include(joinpath("docs", "make.jl"))' + make -C julia/docs # TODO: make Jenkins worker push to MXNet.jl ph-pages branch if master build # ... } +publish_scala_build() { + set -ex + pushd . + scala_prepare + ./ci/publish/scala/build.sh + popd +} + +publish_scala_test() { + set -ex + pushd . + scala_prepare + ./ci/publish/scala/test.sh + popd +} + +publish_scala_deploy() { + set -ex + pushd . + scala_prepare + ./ci/publish/scala/deploy.sh + popd +} + # broken_link_checker broken_link_checker() { @@ -1180,5 +1333,3 @@ EOF declare -F | cut -d' ' -f3 echo fi - - diff --git a/ci/docker_cache.py b/ci/docker_cache.py index bebcb25fb..f906b0eba 100755 --- a/ci/docker_cache.py +++ b/ci/docker_cache.py @@ -32,7 +32,13 @@ import json from typing import * import build as build_util +from util import retry +DOCKERHUB_LOGIN_NUM_RETRIES = 5 +DOCKERHUB_RETRY_SECONDS = 5 +DOCKER_CACHE_NUM_RETRIES = 3 +DOCKER_CACHE_TIMEOUT_MINS = 15 +PARALLEL_BUILDS = 10 def build_save_containers(platforms, registry, load_cache) -> int: @@ -47,7 +53,7 @@ def build_save_containers(platforms, registry, load_cache) -> int: if len(platforms) == 0: return 0 - platform_results = Parallel(n_jobs=len(platforms), backend="multiprocessing")( + platform_results = Parallel(n_jobs=PARALLEL_BUILDS, backend="multiprocessing")( delayed(_build_save_container)(platform, registry, load_cache) for platform in platforms) @@ -78,7 +84,7 @@ def _build_save_container(platform, registry, load_cache) -> Optional[str]: logging.debug('Building %s as %s', platform, docker_tag) try: # Increase the number of retries for building the cache. - image_id = build_util.build_docker(docker_binary='docker', platform=platform, registry=registry, num_retries=10, use_cache=True) + image_id = build_util.build_docker(docker_binary='docker', platform=platform, registry=registry, num_retries=10, no_cache=False) logging.info('Built %s as %s', docker_tag, image_id) # Push cache to registry @@ -99,24 +105,47 @@ def _upload_image(registry, docker_tag, image_id) -> None: :param image_id: Image id :return: None """ - _login_dockerhub() # We don't have to retag the image since it is already in the right format logging.info('Uploading %s (%s) to %s', docker_tag, image_id, registry) push_cmd = ['docker', 'push', docker_tag] subprocess.check_call(push_cmd) +@retry(target_exception=subprocess.CalledProcessError, tries=DOCKERHUB_LOGIN_NUM_RETRIES, + delay_s=DOCKERHUB_RETRY_SECONDS) def _login_dockerhub(): """ Login to the Docker Hub account :return: None """ dockerhub_credentials = _get_dockerhub_credentials() - login_cmd = ['docker', 'login', '--username', dockerhub_credentials['username'], '--password', - dockerhub_credentials['password']] - subprocess.check_call(login_cmd) + + logging.info('Logging in to DockerHub') + # We use password-stdin instead of --password to avoid leaking passwords in case of an error. + # This method will produce the following output: + # > WARNING! Your password will be stored unencrypted in /home/jenkins_slave/.docker/config.json. + # > Configure a credential helper to remove this warning. See + # > https://docs.docker.com/engine/reference/commandline/login/#credentials-store + # Since we consider the restricted slaves a secure environment, that's fine. Also, using this will require + # third party applications which would need a review first as well. + p = subprocess.run(['docker', 'login', '--username', dockerhub_credentials['username'], '--password-stdin'], + stdout=subprocess.PIPE, input=str.encode(dockerhub_credentials['password'])) + logging.info(p.stdout) + logging.info('Successfully logged in to DockerHub') + + +def _logout_dockerhub(): + """ + Log out of DockerHub to delete local credentials + :return: None + """ + logging.info('Logging out of DockerHub') + subprocess.call(['docker', 'logout']) + logging.info('Successfully logged out of DockerHub') +@retry(target_exception=subprocess.TimeoutExpired, tries=DOCKER_CACHE_NUM_RETRIES, + delay_s=DOCKERHUB_RETRY_SECONDS) def load_docker_cache(registry, docker_tag) -> None: """ Load the precompiled docker cache from the registry @@ -125,9 +154,16 @@ def load_docker_cache(registry, docker_tag) -> None: :return: None """ # We don't have to retag the image since it's already in the right format + if not registry: + return + assert docker_tag + logging.info('Loading Docker cache for %s from %s', docker_tag, registry) pull_cmd = ['docker', 'pull', docker_tag] - subprocess.call(pull_cmd) # Don't throw an error if the image does not exist + + # Don't throw an error if the image does not exist + subprocess.run(pull_cmd, timeout=DOCKER_CACHE_TIMEOUT_MINS*60) + logging.info('Successfully pulled docker cache') def delete_local_docker_cache(docker_tag): @@ -175,8 +211,7 @@ def _get_dockerhub_credentials(): # pragma: no cover logging.exception("The request was invalid due to:") elif client_error.response['Error']['Code'] == 'InvalidParameterException': logging.exception("The request had invalid params:") - else: - raise + raise else: secret = get_secret_value_response['SecretString'] secret_dict = json.loads(secret) @@ -213,7 +248,11 @@ def script_name() -> str: args = parser.parse_args() platforms = build_util.get_platforms() - return build_save_containers(platforms=platforms, registry=args.docker_registry, load_cache=True) + try: + _login_dockerhub() + return build_save_containers(platforms=platforms, registry=args.docker_registry, load_cache=True) + finally: + _logout_dockerhub() if __name__ == '__main__': diff --git a/Jenkinsfile b/ci/jenkins/Jenkins_steps.groovy similarity index 71% rename from Jenkinsfile rename to ci/jenkins/Jenkins_steps.groovy index 64ef1d982..e812c4e24 100644 --- a/Jenkinsfile +++ b/ci/jenkins/Jenkins_steps.groovy @@ -17,13 +17,17 @@ // specific language governing permissions and limitations // under the License. // -// Jenkins pipeline -// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ +// This file contains the steps that will be used in the +// Jenkins pipelines + +utils = load('ci/Jenkinsfile_utils.groovy') // mxnet libraries mx_lib = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a' -// for scala build, need to pass extra libs when run with dist_kvstore -mx_dist_lib = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a, 3rdparty/ps-lite/build/libps.a, deps/lib/libprotobuf-lite.a, deps/lib/libzmq.a' + +// Python wheels +mx_pip = 'build/*.whl' + // mxnet cmake libraries, in cmake builds we do not produce a libnvvm static library by default. mx_cmake_lib = 'build/libmxnet.so, build/libmxnet.a, build/3rdparty/dmlc-core/libdmlc.a, build/tests/mxnet_unit_tests, build/3rdparty/openmp/runtime/src/libomp.so' // mxnet cmake libraries, in cmake builds we do not produce a libnvvm static library by default. @@ -31,12 +35,8 @@ mx_cmake_lib_debug = 'build/libmxnet.so, build/libmxnet.a, build/3rdparty/dmlc-c mx_cmake_mkldnn_lib = 'build/libmxnet.so, build/libmxnet.a, build/3rdparty/dmlc-core/libdmlc.a, build/tests/mxnet_unit_tests, build/3rdparty/openmp/runtime/src/libomp.so, build/3rdparty/mkldnn/src/libmkldnn.so.0' mx_mkldnn_lib = 'lib/libmxnet.so, lib/libmxnet.a, lib/libiomp5.so, lib/libmkldnn.so.0, lib/libmklml_intel.so, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a' mx_tensorrt_lib = 'lib/libmxnet.so, lib/libnvonnxparser_runtime.so.0, lib/libnvonnxparser.so.0, lib/libonnx_proto.so, lib/libonnx.so' -mx_lib_cpp_examples = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a, 3rdparty/ps-lite/build/libps.a, deps/lib/libprotobuf-lite.a, deps/lib/libzmq.a, build/cpp-package/example/lenet, build/cpp-package/example/alexnet, build/cpp-package/example/googlenet, build/cpp-package/example/lenet_with_mxdataiter, build/cpp-package/example/resnet, build/cpp-package/example/mlp, build/cpp-package/example/mlp_cpu, build/cpp-package/example/mlp_gpu, build/cpp-package/example/test_score, build/cpp-package/example/test_optimizer' -mx_lib_cpp_examples_cpu = 'build/libmxnet.so, build/cpp-package/example/mlp_cpu' - -// timeout in minutes -max_time = 120 - +mx_lib_cpp_examples = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a, 3rdparty/ps-lite/build/libps.a, deps/lib/libprotobuf-lite.a, deps/lib/libzmq.a, build/cpp-package/example/*' +mx_lib_cpp_examples_cpu = 'build/libmxnet.so, build/cpp-package/example/*' // Python unittest for CPU // Python 2 @@ -89,126 +89,206 @@ def python3_gpu_ut_nocudnn(docker_container_name) { } } -def deploy_docs() { - parallel 'Docs': { - node(NODE_LINUX_CPU) { - ws('workspace/docs') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('ubuntu_cpu', 'deploy_docs', false) - sh "ci/other/ci_deploy_doc.sh ${env.BRANCH_NAME} ${env.BUILD_NUMBER}" +//------------------------------------------------------------------------------------------ + +def compile_unix_cpu_openblas() { + return ['CPU: Openblas': { + node(NODE_LINUX_CPU) { + ws('workspace/build-cpu-openblas') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_openblas', false) + utils.pack_lib('cpu', mx_lib, true) + } } } - } - }, - 'Julia docs': { - node(NODE_LINUX_CPU) { - ws('workspace/julia-docs') { - timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_lib) - utils.docker_run('ubuntu_cpu', 'deploy_jl_docs', false) + }] +} + +def compile_unix_openblas_debug_cpu() { + return ['CPU: Openblas, cmake, debug': { + node(NODE_LINUX_CPU) { + ws('workspace/build-cpu-openblas') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_cmake_debug', false) + utils.pack_lib('cpu_debug', mx_cmake_lib_debug, true) + } } } - } - } + }] } -node('mxnetlinux-cpu') { - // Loading the utilities requires a node context unfortunately - checkout scm - utils = load('ci/Jenkinsfile_utils.groovy') +def compile_unix_mkl_cpu() { + return ['CPU: MKL': { + node(NODE_LINUX_CPU) { + ws('workspace/build-cpu-mkl') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_mkl', false) + utils.pack_lib('cpu_mkl', mx_mkldnn_lib, true) + } + } + } + }] } -utils.assign_node_labels(linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') -utils.main_wrapper( -core_logic: { - stage('Sanity Check') { - parallel 'Lint': { +def compile_unix_mkldnn_cpu() { + return ['CPU: MKLDNN': { node(NODE_LINUX_CPU) { - ws('workspace/sanity-lint') { - utils.init_git() - utils.docker_run('ubuntu_cpu', 'sanity_check', false) + ws('workspace/build-mkldnn-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_mkldnn', false) + utils.pack_lib('mkldnn_cpu', mx_mkldnn_lib, true) + } } } - }, - 'RAT License': { + }] +} + +def compile_unix_mkldnn_mkl_cpu() { + return ['CPU: MKLDNN_MKL': { node(NODE_LINUX_CPU) { - ws('workspace/sanity-rat') { - utils.init_git() - utils.docker_run('ubuntu_rat', 'nightly_test_rat_check', false) + ws('workspace/build-mkldnn-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_mkldnn_mkl', false) + utils.pack_lib('mkldnn_mkl_cpu', mx_mkldnn_lib, true) + } } } - } - } + }] +} - stage('Build') { - parallel 'CPU: CentOS 7': { +def compile_unix_mkldnn_gpu() { + return ['GPU: MKLDNN': { node(NODE_LINUX_CPU) { - ws('workspace/build-centos7-cpu') { + ws('workspace/build-mkldnn-gpu') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('centos7_cpu', 'build_centos7_cpu', false) - utils.pack_lib('centos7_cpu', mx_lib, true) + utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn', false) + utils.pack_lib('mkldnn_gpu', mx_mkldnn_lib, true) + } + } + } + }] +} + +def compile_unix_mkldnn_nocudnn_gpu() { + return ['GPU: MKLDNN_CUDNNOFF': { + node(NODE_LINUX_CPU) { + ws('workspace/build-mkldnn-gpu-nocudnn') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn_nocudnn', false) + utils.pack_lib('mkldnn_gpu_nocudnn', mx_mkldnn_lib, true) + } + } + } + }] +} + +def compile_unix_full_gpu() { + return ['GPU: CUDA9.1+cuDNN7': { + node(NODE_LINUX_CPU) { + ws('workspace/build-gpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_cuda91_cudnn7', false) + utils.pack_lib('gpu', mx_lib_cpp_examples, true) } } } - }, - 'CPU: CentOS 7 MKLDNN': { + }] +} + +def compile_unix_cmake_mkldnn_gpu() { + return ['GPU: CMake MKLDNN': { node(NODE_LINUX_CPU) { - ws('workspace/build-centos7-mkldnn') { + ws('workspace/build-cmake-mkldnn-gpu') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('centos7_cpu', 'build_centos7_mkldnn', false) - utils.pack_lib('centos7_mkldnn', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'build_ubuntu_gpu_cmake_mkldnn', false) + utils.pack_lib('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib, true) } } } - }, - 'GPU: CentOS 7': { + }] +} + +def compile_unix_cmake_gpu() { + return ['GPU: CMake': { node(NODE_LINUX_CPU) { - ws('workspace/build-centos7-gpu') { + ws('workspace/build-cmake-gpu') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('centos7_gpu', 'build_centos7_gpu', false) - utils.pack_lib('centos7_gpu', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'build_ubuntu_gpu_cmake', false) + utils.pack_lib('cmake_gpu', mx_cmake_lib, true) + } + } + } + }] +} + +def compile_unix_tensorrt_gpu() { + return ['TensorRT': { + node(NODE_LINUX_CPU) { + ws('workspace/build-tensorrt') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_gpu_tensorrt', 'build_ubuntu_gpu_tensorrt', false) + utils.pack_lib('tensorrt', mx_tensorrt_lib, true) } } } - }, - 'CPU: Openblas': { + }] +} + +def compile_centos7_cpu() { + return ['CPU: CentOS 7': { node(NODE_LINUX_CPU) { - ws('workspace/build-cpu-openblas') { + ws('workspace/build-centos7-cpu') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_openblas', false) - utils.pack_lib('cpu', mx_dist_lib, true) + utils.docker_run('centos7_cpu', 'build_centos7_cpu', false) + utils.pack_lib('centos7_cpu', mx_lib, true) } } } - }, - 'CPU: ASAN': { + }] +} + +def compile_centos7_cpu_mkldnn() { + return ['CPU: CentOS 7 MKLDNN': { node(NODE_LINUX_CPU) { - ws('workspace/build-cpu-asan') { + ws('workspace/build-centos7-mkldnn') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_cmake_asan', false) - utils.pack_lib('cpu_asan', mx_lib_cpp_examples_cpu) + utils.docker_run('centos7_cpu', 'build_centos7_mkldnn', false) + utils.pack_lib('centos7_mkldnn', mx_mkldnn_lib, true) } } } - }, - 'CPU: Openblas, debug': { + }] +} + +def compile_centos7_gpu() { + return ['GPU: CentOS 7': { node(NODE_LINUX_CPU) { - ws('workspace/build-cpu-openblas') { + ws('workspace/build-centos7-gpu') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_cmake_debug', false) - utils.pack_lib('cpu_debug', mx_cmake_lib_debug, true) + utils.docker_run('centos7_gpu', 'build_centos7_gpu', false) + utils.pack_lib('centos7_gpu', mx_lib, true) } } } - }, - 'CPU: Clang 3.9': { + }] +} + +def compile_unix_clang_3_9_cpu() { + return ['CPU: Clang 3.9': { node(NODE_LINUX_CPU) { ws('workspace/build-cpu-clang39') { timeout(time: max_time, unit: 'MINUTES') { @@ -217,8 +297,11 @@ core_logic: { } } } - }, - 'CPU: Clang 6': { + }] +} + +def compile_unix_clang_6_cpu() { + return ['CPU: Clang 6': { node(NODE_LINUX_CPU) { ws('workspace/build-cpu-clang60') { timeout(time: max_time, unit: 'MINUTES') { @@ -227,8 +310,11 @@ core_logic: { } } } - }, - 'CPU: Clang Tidy': { + }] +} + +def compile_unix_clang_tidy_cpu() { + return ['CPU: Clang Tidy': { node(NODE_LINUX_CPU) { ws('workspace/build-cpu-clang60_tidy') { timeout(time: max_time, unit: 'MINUTES') { @@ -237,8 +323,11 @@ core_logic: { } } } - }, - 'CPU: Clang 3.9 MKLDNN': { + }] +} + +def compile_unix_clang_3_9_mkldnn_cpu() { + return ['CPU: Clang 3.9 MKLDNN': { node(NODE_LINUX_CPU) { ws('workspace/build-cpu-mkldnn-clang39') { timeout(time: max_time, unit: 'MINUTES') { @@ -248,8 +337,11 @@ core_logic: { } } } - }, - 'CPU: Clang 6 MKLDNN': { + }] +} + +def compile_unix_clang_6_mkldnn_cpu() { + return ['CPU: Clang 6 MKLDNN': { node(NODE_LINUX_CPU) { ws('workspace/build-cpu-mkldnn-clang60') { timeout(time: max_time, unit: 'MINUTES') { @@ -259,106 +351,130 @@ core_logic: { } } } - }, - 'CPU: MKLDNN': { + }] +} + +def compile_armv8_jetson_gpu() { + return ['NVidia Jetson / ARMv8':{ node(NODE_LINUX_CPU) { - ws('workspace/build-mkldnn-cpu') { + ws('workspace/build-jetson-armv8') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_mkldnn', false) - utils.pack_lib('mkldnn_cpu', mx_mkldnn_lib, true) + utils.docker_run('jetson', 'build_jetson', false) } } } - }, - 'GPU: MKLDNN': { + }] +} + +def compile_armv7_cpu() { + return ['ARMv7':{ node(NODE_LINUX_CPU) { - ws('workspace/build-mkldnn-gpu') { + ws('workspace/build-ARMv7') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn', false) - utils.pack_lib('mkldnn_gpu', mx_mkldnn_lib, true) + utils.docker_run('armv7', 'build_armv7', false) + utils.pack_lib('armv7', mx_pip) } } } - }, - 'GPU: MKLDNN_CUDNNOFF': { - node(NODE_LINUX_CPU) { - ws('workspace/build-mkldnn-gpu-nocudnn') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_mkldnn_nocudnn', false) - utils.pack_lib('mkldnn_gpu_nocudnn', mx_mkldnn_lib, true) - } - } - } - }, - 'GPU: CUDA9.1+cuDNN7': { + }] +} + +def compile_armv6_cpu() { + return ['ARMv6':{ node(NODE_LINUX_CPU) { - ws('workspace/build-gpu') { + ws('workspace/build-ARMv6') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_build_cuda', 'build_ubuntu_gpu_cuda91_cudnn7', false) - utils.pack_lib('gpu', mx_lib_cpp_examples, true) + utils.docker_run('armv6', 'build_armv6', false) } } } - }, - 'Amalgamation MIN': { + }] +} + +def compile_armv8_cpu() { + return ['ARMv8':{ node(NODE_LINUX_CPU) { - ws('workspace/amalgamationmin') { + ws('workspace/build-ARMv8') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_amalgamation_min', false) + utils.docker_run('armv8', 'build_armv8', false) } } } - }, - 'Amalgamation': { + }] +} + +def compile_armv8_android_cpu() { + return ['Android / ARMv8':{ node(NODE_LINUX_CPU) { - ws('workspace/amalgamation') { + ws('workspace/android64') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_cpu', 'build_ubuntu_amalgamation', false) + utils.docker_run('android_armv8', 'build_android_armv8', false) } } } - }, + }] +} - 'GPU: CMake MKLDNN': { +def compile_armv7_android_cpu() { + return ['Android / ARMv7':{ node(NODE_LINUX_CPU) { - ws('workspace/build-cmake-mkldnn-gpu') { + ws('workspace/androidv7') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_gpu', 'build_ubuntu_gpu_cmake_mkldnn', false) - utils.pack_lib('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib, true) + utils.docker_run('android_armv7', 'build_android_armv7', false) } } } - }, - 'GPU: CMake': { + }] +} + +def compile_unix_asan_cpu() { + return ['CPU: ASAN': { node(NODE_LINUX_CPU) { - ws('workspace/build-cmake-gpu') { + ws('workspace/build-cpu-asan') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_gpu', 'build_ubuntu_gpu_cmake', false) - utils.pack_lib('cmake_gpu', mx_cmake_lib, true) + utils.docker_run('ubuntu_cpu', 'build_ubuntu_cpu_cmake_asan', false) + utils.pack_lib('cpu_asan', mx_lib_cpp_examples_cpu) + } + } + } + }] +} + +def compile_unix_amalgamation_min() { + return ['Amalgamation MIN': { + node(NODE_LINUX_CPU) { + ws('workspace/amalgamationmin') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'build_ubuntu_amalgamation_min', false) } } } - }, - 'TensorRT': { + }] +} + +def compile_unix_amalgamation() { + return ['Amalgamation': { node(NODE_LINUX_CPU) { - ws('workspace/build-tensorrt') { + ws('workspace/amalgamation') { timeout(time: max_time, unit: 'MINUTES') { utils.init_git() - utils.docker_run('ubuntu_gpu_tensorrt', 'build_ubuntu_gpu_tensorrt', false) - utils.pack_lib('tensorrt', mx_tensorrt_lib, true) + utils.docker_run('ubuntu_cpu', 'build_ubuntu_amalgamation', false) } } } - }, - 'Build CPU windows':{ + }] +} + +def compile_windows_cpu() { + return ['Build CPU windows':{ node(NODE_WINDOWS_CPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/build-cpu') { @@ -370,9 +486,11 @@ core_logic: { } } } - }, + }] +} - 'Build GPU windows':{ +def compile_windows_gpu() { + return ['Build GPU windows':{ node(NODE_WINDOWS_CPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/build-gpu') { @@ -384,8 +502,11 @@ core_logic: { } } } - }, - 'Build GPU MKLDNN windows':{ + }] +} + +def compile_windows_gpu_mkldnn() { + return ['Build GPU MKLDNN windows':{ node(NODE_WINDOWS_CPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/build-gpu') { @@ -397,72 +518,11 @@ core_logic: { } } } - }, - 'NVidia Jetson / ARMv8':{ - node(NODE_LINUX_CPU) { - ws('workspace/build-jetson-armv8') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('jetson', 'build_jetson', false) - } - } - } - }, - // 'ARMv7':{ - // node(NODE_LINUX_CPU) { - // ws('workspace/build-ARMv7') { - // timeout(time: max_time, unit: 'MINUTES') { - // utils.init_git() - // utils.docker_run('armv7', 'build_armv7', false) - // } - // } - // } - // }, - 'ARMv6':{ - node(NODE_LINUX_CPU) { - ws('workspace/build-ARMv6') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('armv6', 'build_armv6', false) - } - } - } - }, - 'ARMv8':{ - node(NODE_LINUX_CPU) { - ws('workspace/build-ARMv8') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('armv8', 'build_armv8', false) - } - } - } - }, - 'Android / ARMv8':{ - node(NODE_LINUX_CPU) { - ws('workspace/android64') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('android_armv8', 'build_android_armv8', false) - } - } - } - }, - 'Android / ARMv7':{ - node(NODE_LINUX_CPU) { - ws('workspace/androidv7') { - timeout(time: max_time, unit: 'MINUTES') { - utils.init_git() - utils.docker_run('android_armv7', 'build_android_armv7', false) - } - } - } - } - - } // End of stage('Build') + }] +} - stage('Tests') { - parallel 'Python2: CPU': { +def test_unix_python2_cpu() { + return ['Python2: CPU': { node(NODE_LINUX_CPU) { ws('workspace/ut-python2-cpu') { try { @@ -476,56 +536,95 @@ core_logic: { } } } - }, - 'Python3: CPU': { - node(NODE_LINUX_CPU) { - ws('workspace/ut-python3-cpu') { + }] +} + +def test_unix_python2_gpu() { + return ['Python2: GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/ut-python2-gpu') { try { - utils.unpack_and_init('cpu', mx_lib, true) - python3_ut('ubuntu_cpu') + utils.unpack_and_init('gpu', mx_lib, true) + python2_gpu_ut('ubuntu_gpu') utils.publish_test_coverage() } finally { - utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_cpu_unittest.xml') - utils.collect_test_results_unix('nosetests_quantization.xml', 'nosetests_python3_cpu_quantization.xml') + utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python2_gpu.xml') } } } - }, - 'CPU ASAN': { - node(NODE_LINUX_CPU) { - ws('workspace/ut-python3-cpu-asan') { - utils.unpack_and_init('cpu_asan', mx_lib_cpp_examples_cpu) - utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_asan', false) + }] +} + +def test_unix_python2_quantize_gpu() { + return ['Python2: Quantize GPU': { + node(NODE_LINUX_GPU_P3) { + ws('workspace/ut-python2-quantize-gpu') { + timeout(time: max_time, unit: 'MINUTES') { + try { + utils.unpack_and_init('gpu', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_python2_quantization_gpu', true) + utils.publish_test_coverage() + } finally { + utils.collect_test_results_unix('nosetests_quantization_gpu.xml', 'nosetests_python2_quantize_gpu.xml') + } + } + } + } + }] +} + +def test_unix_python2_mkldnn_gpu() { + return ['Python2: MKLDNN-GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/ut-python2-mkldnn-gpu') { + try { + utils.unpack_and_init('mkldnn_gpu', mx_mkldnn_lib, true) + python2_gpu_ut('ubuntu_gpu') + utils.publish_test_coverage() + } finally { + utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python2_mkldnn_gpu.xml') + } } } - }, - 'Python3: CPU debug': { + }] +} + +def test_unix_python3_cpu() { + return ['Python3: CPU': { node(NODE_LINUX_CPU) { - ws('workspace/ut-python3-cpu-debug') { + ws('workspace/ut-python3-cpu') { try { - utils.unpack_and_init('cpu_debug', mx_cmake_lib_debug, true) + utils.unpack_and_init('cpu', mx_lib, true) python3_ut('ubuntu_cpu') + utils.publish_test_coverage() } finally { - utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_cpu_debug_unittest.xml') - utils.collect_test_results_unix('nosetests_quantization.xml', 'nosetests_python3_cpu_debug_quantization.xml') + utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_cpu_unittest.xml') + utils.collect_test_results_unix('nosetests_quantization.xml', 'nosetests_python3_cpu_quantization.xml') } } } - }, - 'Python2: GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/ut-python2-gpu') { + }] +} + +def test_unix_python3_mkl_cpu() { + return ['Python3: MKL-CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-python3-cpu') { try { - utils.unpack_and_init('gpu', mx_lib, true) - python2_gpu_ut('ubuntu_gpu') + utils.unpack_and_init('cpu_mkl', mx_lib, true) + python3_ut('ubuntu_cpu') utils.publish_test_coverage() } finally { - utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python2_gpu.xml') + utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_cpu_unittest.xml') + utils.collect_test_results_unix('nosetests_quantization.xml', 'nosetests_python3_cpu_quantization.xml') } } } - }, - 'Python3: GPU': { + }] +} + +def test_unix_python3_gpu() { + return ['Python3: GPU': { node(NODE_LINUX_GPU) { ws('workspace/ut-python3-gpu') { try { @@ -537,23 +636,11 @@ core_logic: { } } } - }, - 'Python2: Quantize GPU': { - node(NODE_LINUX_GPU_P3) { - ws('workspace/ut-python2-quantize-gpu') { - timeout(time: max_time, unit: 'MINUTES') { - try { - utils.unpack_and_init('gpu', mx_lib, true) - utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_python2_quantization_gpu', true) - utils.publish_test_coverage() - } finally { - utils.collect_test_results_unix('nosetests_quantization_gpu.xml', 'nosetests_python2_quantize_gpu.xml') - } - } - } - } - }, - 'Python3: Quantize GPU': { + }] +} + +def test_unix_python3_quantize_gpu() { + return ['Python3: Quantize GPU': { node(NODE_LINUX_GPU_P3) { ws('workspace/ut-python3-quantize-gpu') { timeout(time: max_time, unit: 'MINUTES') { @@ -567,8 +654,27 @@ core_logic: { } } } - }, - 'Python2: MKLDNN-CPU': { + }] +} + +def test_unix_python3_debug_cpu() { + return ['Python3: CPU debug': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-python3-cpu-debug') { + try { + utils.unpack_and_init('cpu_debug', mx_cmake_lib_debug, true) + python3_ut('ubuntu_cpu') + } finally { + utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_cpu_debug_unittest.xml') + utils.collect_test_results_unix('nosetests_quantization.xml', 'nosetests_python3_cpu_debug_quantization.xml') + } + } + } + }] +} + +def test_unix_python2_mkldnn_cpu() { + return ['Python2: MKLDNN-CPU': { node(NODE_LINUX_CPU) { ws('workspace/ut-python2-mkldnn-cpu') { try { @@ -582,25 +688,32 @@ core_logic: { } } } - }, - 'Python2: MKLDNN-GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/ut-python2-mkldnn-gpu') { + }] +} + +def test_unix_python3_mkldnn_cpu() { + return ['Python3: MKLDNN-CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-python3-mkldnn-cpu') { try { - utils.unpack_and_init('mkldnn_gpu', mx_mkldnn_lib, true) - python2_gpu_ut('ubuntu_gpu') + utils.unpack_and_init('mkldnn_cpu', mx_mkldnn_lib, true) + python3_ut_mkldnn('ubuntu_cpu') utils.publish_test_coverage() } finally { - utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python2_mkldnn_gpu.xml') + utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_mkldnn_cpu_unittest.xml') + utils.collect_test_results_unix('nosetests_mkl.xml', 'nosetests_python3_mkldnn_cpu_mkl.xml') } } } - }, - 'Python3: MKLDNN-CPU': { + }] +} + +def test_unix_python3_mkldnn_mkl_cpu() { + return ['Python3: MKLDNN-MKL-CPU': { node(NODE_LINUX_CPU) { - ws('workspace/ut-python3-mkldnn-cpu') { + ws('workspace/ut-python3-mkldnn-mkl-cpu') { try { - utils.unpack_and_init('mkldnn_cpu', mx_mkldnn_lib, true) + utils.unpack_and_init('mkldnn_mkl_cpu', mx_mkldnn_lib, true) python3_ut_mkldnn('ubuntu_cpu') utils.publish_test_coverage() } finally { @@ -609,8 +722,11 @@ core_logic: { } } } - }, - 'Python3: MKLDNN-GPU': { + }] +} + +def test_unix_python3_mkldnn_gpu() { + return ['Python3: MKLDNN-GPU': { node(NODE_LINUX_GPU) { ws('workspace/ut-python3-mkldnn-gpu') { try { @@ -622,8 +738,11 @@ core_logic: { } } } - }, - 'Python3: MKLDNN-GPU-NOCUDNN': { + }] +} + +def test_unix_python3_mkldnn_nocudnn_gpu() { + return ['Python3: MKLDNN-GPU-NOCUDNN': { node(NODE_LINUX_GPU) { ws('workspace/ut-python3-mkldnn-gpu-nocudnn') { try { @@ -635,76 +754,128 @@ core_logic: { } } } - }, - 'Python3: CentOS 7 CPU': { - node(NODE_LINUX_CPU) { - ws('workspace/build-centos7-cpu') { + }] +} + +def test_unix_python3_tensorrt_gpu() { + return ['Python3: TensorRT GPU': { + node(NODE_LINUX_GPU_P3) { + ws('workspace/build-tensorrt') { timeout(time: max_time, unit: 'MINUTES') { try { - utils.unpack_and_init('centos7_cpu', mx_lib, true) - utils.docker_run('centos7_cpu', 'unittest_centos7_cpu', false) + utils.unpack_and_init('tensorrt', mx_tensorrt_lib, true) + utils.docker_run('ubuntu_gpu_tensorrt', 'unittest_ubuntu_tensorrt_gpu', true) utils.publish_test_coverage() } finally { - utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_centos7_cpu_unittest.xml') - utils.collect_test_results_unix('nosetests_train.xml', 'nosetests_python3_centos7_cpu_train.xml') + utils.collect_test_results_unix('nosetests_tensorrt.xml', 'nosetests_python3_tensorrt_gpu.xml') } } } } - }, - 'Python3: CentOS 7 GPU': { + }] +} + +def test_unix_python3_integration_gpu() { + return ['Python Integration GPU': { node(NODE_LINUX_GPU) { - ws('workspace/build-centos7-gpu') { + ws('workspace/it-python-gpu') { timeout(time: max_time, unit: 'MINUTES') { - try { - utils.unpack_and_init('centos7_gpu', mx_lib, true) - utils.docker_run('centos7_gpu', 'unittest_centos7_gpu', true) - utils.publish_test_coverage() - } finally { - utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python3_centos7_gpu.xml') - } + utils.unpack_and_init('gpu', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_python', true) + utils.publish_test_coverage() } } } - }, - 'Python3: TensorRT GPU': { - node(NODE_LINUX_GPU_P3) { - ws('workspace/build-tensorrt') { - timeout(time: max_time, unit: 'MINUTES') { - try { - utils.unpack_and_init('tensorrt', mx_tensorrt_lib, true) - utils.docker_run('ubuntu_gpu_tensorrt', 'unittest_ubuntu_tensorrt_gpu', true) - utils.publish_test_coverage() - } finally { - utils.collect_test_results_unix('nosetests_tensorrt.xml', 'nosetests_python3_tensorrt_gpu.xml') + }] +} + +def test_unix_caffe_gpu() { + return ['Caffe GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/it-caffe') { + timeout(time: max_time, unit: 'MINUTES') { + utils.init_git() + utils.unpack_lib('gpu', mx_lib) + utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_caffe', true) + utils.publish_test_coverage() + } } + } + }] +} + +def test_unix_cpp_package_gpu() { + return ['cpp-package GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/it-cpp-package') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('gpu', mx_lib_cpp_examples, true) + utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_cpp_package', true) + utils.publish_test_coverage() } } } - }, - 'Scala: CPU': { + }] +} + +def test_unix_scala_cpu() { + return ['Scala: CPU': { node(NODE_LINUX_CPU) { ws('workspace/ut-scala-cpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_dist_lib, true) + utils.unpack_and_init('cpu', mx_lib, true) + utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_scala', false) + utils.publish_test_coverage() + } + } + } + }] +} + +def test_unix_scala_mkldnn_cpu(){ + return ['Scala: MKLDNN-CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-scala-mkldnn-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('mkldnn_cpu', mx_mkldnn_lib, true) utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_scala', false) utils.publish_test_coverage() } } } - }, - 'Clojure: CPU': { + }] +} + +def test_unix_scala_gpu() { + return ['Scala: GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/ut-scala-gpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('gpu', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_scala', true) + utils.publish_test_coverage() + } + } + } + }] +} + +def test_unix_clojure_cpu() { + return ['Clojure: CPU': { node(NODE_LINUX_CPU) { ws('workspace/ut-clojure-cpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_dist_lib, true) + utils.unpack_and_init('cpu', mx_lib, true) utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_clojure', false) utils.publish_test_coverage() } } } - }, - 'Perl: CPU': { + }] +} + +def test_unix_r_cpu() { + return ['Perl: CPU': { node(NODE_LINUX_CPU) { ws('workspace/ut-perl-cpu') { timeout(time: max_time, unit: 'MINUTES') { @@ -714,104 +885,204 @@ core_logic: { } } } - }, - 'Perl: GPU': { + }] +} + +def test_unix_cpp_gpu() { + return ['Cpp: GPU': { node(NODE_LINUX_GPU) { - ws('workspace/ut-perl-gpu') { + ws('workspace/ut-cpp-gpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('gpu', mx_lib, true) - utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_cpugpu_perl', true) + utils.unpack_and_init('cmake_gpu', mx_cmake_lib, true) + utils.docker_run('ubuntu_gpu', 'unittest_cpp', true) utils.publish_test_coverage() } } } - }, - 'Cpp: GPU': { + }] +} + +def test_unix_cpp_mkldnn_gpu() { + return ['Cpp: MKLDNN+GPU': { node(NODE_LINUX_GPU) { - ws('workspace/ut-cpp-gpu') { + ws('workspace/ut-cpp-mkldnn-gpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cmake_gpu', mx_cmake_lib, true) - utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_gpu_cpp', true) + utils.unpack_and_init('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib, true) + utils.docker_run('ubuntu_gpu', 'unittest_cpp', true) + utils.publish_test_coverage() + } + } + } + }] +} + +def test_unix_cpp_cpu() { + return ['Cpp: CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-cpp-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('cpu_debug', mx_cmake_lib_debug, true) + utils.docker_run('ubuntu_cpu', 'unittest_cpp', false) utils.publish_test_coverage() } } } - }, - 'Cpp: MKLDNN+GPU': { + }] +} + +def test_unix_r_gpu() { + return ['Perl: GPU': { node(NODE_LINUX_GPU) { - ws('workspace/ut-cpp-mkldnn-gpu') { + ws('workspace/ut-perl-gpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cmake_mkldnn_gpu', mx_cmake_mkldnn_lib, true) - utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_gpu_cpp', true) + utils.unpack_and_init('gpu', mx_lib, true) + utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_cpugpu_perl', true) utils.publish_test_coverage() } } } - }, - 'R: CPU': { + }] +} + +def test_unix_julia07_cpu() { + return ['Julia 0.7: CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-julia07-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('cpu', mx_lib) + utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_julia07', false) + } + } + } + }] +} + +def test_unix_julia10_cpu() { + return ['Julia 1.0: CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-julia10-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('cpu', mx_lib) + utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_julia10', false) + } + } + } + }] +} + +def test_unix_onnx_cpu() { + return ['Onnx CPU': { node(NODE_LINUX_CPU) { - ws('workspace/ut-r-cpu') { + ws('workspace/it-onnx-cpu') { timeout(time: max_time, unit: 'MINUTES') { utils.unpack_and_init('cpu', mx_lib, true) - utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_R', false) + utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_onnx', false) + utils.publish_test_coverage() + } + } + } + }] +} + +def test_unix_distributed_kvstore_cpu() { + return ['dist-kvstore tests CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/it-dist-kvstore') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('cpu', mx_lib, true) + utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_dist_kvstore', false) utils.publish_test_coverage() } } } - }, - 'R: GPU': { + }] +} + +def test_unix_distributed_kvstore_gpu() { + return ['dist-kvstore tests GPU': { node(NODE_LINUX_GPU) { - ws('workspace/ut-r-gpu') { + ws('workspace/it-dist-kvstore') { timeout(time: max_time, unit: 'MINUTES') { utils.unpack_and_init('gpu', mx_lib, true) - utils.docker_run('ubuntu_gpu', 'unittest_ubuntu_gpu_R', true) + utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_dist_kvstore', true) utils.publish_test_coverage() } } } - }, - 'Julia 0.6: CPU': { + }] +} + +def test_centos7_python3_cpu() { + return ['Python3: CentOS 7 CPU': { node(NODE_LINUX_CPU) { - ws('workspace/ut-julia06-cpu') { + ws('workspace/build-centos7-cpu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_lib) - utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_julia06', false) + try { + utils.unpack_and_init('centos7_cpu', mx_lib, true) + utils.docker_run('centos7_cpu', 'unittest_centos7_cpu', false) + utils.publish_test_coverage() + } finally { + utils.collect_test_results_unix('nosetests_unittest.xml', 'nosetests_python3_centos7_cpu_unittest.xml') + utils.collect_test_results_unix('nosetests_train.xml', 'nosetests_python3_centos7_cpu_train.xml') + } } } } - }, + }] +} - 'Python 2: CPU Win':{ - node(NODE_WINDOWS_CPU) { - timeout(time: max_time, unit: 'MINUTES') { - ws('workspace/ut-python-cpu') { +def test_centos7_python3_gpu() { + return ['Python3: CentOS 7 GPU': { + node(NODE_LINUX_GPU) { + ws('workspace/build-centos7-gpu') { + timeout(time: max_time, unit: 'MINUTES') { try { - utils.init_git_win() - unstash 'windows_package_cpu' - powershell 'ci/windows/test_py2_cpu.ps1' + utils.unpack_and_init('centos7_gpu', mx_lib, true) + utils.docker_run('centos7_gpu', 'unittest_centos7_gpu', true) + utils.publish_test_coverage() } finally { - utils.collect_test_results_windows('nosetests_unittest.xml', 'nosetests_unittest_windows_python2_cpu.xml') + utils.collect_test_results_unix('nosetests_gpu.xml', 'nosetests_python3_centos7_gpu.xml') } } } } - }, - 'Python 3: CPU Win': { + }] +} + +def test_centos7_scala_cpu() { + return ['Scala: CentOS CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-scala-centos7-cpu') { + timeout(time: max_time, unit: 'MINUTES') { + utils.unpack_and_init('centos7_cpu', mx_lib, true) + utils.docker_run('centos7_cpu', 'unittest_centos7_cpu_scala', false) + utils.publish_test_coverage() + } + } + } + }] +} + +def test_windows_python2_cpu() { + return ['Python 2: CPU Win':{ node(NODE_WINDOWS_CPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/ut-python-cpu') { try { utils.init_git_win() unstash 'windows_package_cpu' - powershell 'ci/windows/test_py3_cpu.ps1' + powershell 'ci/windows/test_py2_cpu.ps1' } finally { - utils.collect_test_results_windows('nosetests_unittest.xml', 'nosetests_unittest_windows_python3_cpu.xml') + utils.collect_test_results_windows('nosetests_unittest.xml', 'nosetests_unittest_windows_python2_cpu.xml') } } } } - }, - 'Python 2: GPU Win':{ + }] +} + +def test_windows_python2_gpu() { + return ['Python 2: GPU Win':{ node(NODE_WINDOWS_GPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/ut-python-gpu') { @@ -826,8 +1097,11 @@ core_logic: { } } } - }, - 'Python 3: GPU Win':{ + }] +} + +def test_windows_python3_gpu() { + return ['Python 3: GPU Win':{ node(NODE_WINDOWS_GPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/ut-python-gpu') { @@ -842,8 +1116,11 @@ core_logic: { } } } - }, - 'Python 3: MKLDNN-GPU Win':{ + }] +} + +def test_windows_python3_gpu_mkldnn() { + return ['Python 3: MKLDNN-GPU Win':{ node(NODE_WINDOWS_GPU) { timeout(time: max_time, unit: 'MINUTES') { ws('workspace/ut-python-gpu') { @@ -858,97 +1135,104 @@ core_logic: { } } } - }, - 'Onnx CPU': { - node(NODE_LINUX_CPU) { - ws('workspace/it-onnx-cpu') { - timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_lib, true) - utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_onnx', false) - utils.publish_test_coverage() + }] +} + +def test_windows_python3_cpu() { + return ['Python 3: CPU Win': { + node(NODE_WINDOWS_CPU) { + timeout(time: max_time, unit: 'MINUTES') { + ws('workspace/ut-python-cpu') { + try { + utils.init_git_win() + unstash 'windows_package_cpu' + powershell 'ci/windows/test_py3_cpu.ps1' + } finally { + utils.collect_test_results_windows('nosetests_unittest.xml', 'nosetests_unittest_windows_python3_cpu.xml') + } } } } - }, - 'Python GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/it-python-gpu') { + }] +} + +def test_qemu_armv7_cpu() { + return ['ARMv7 QEMU': { + node(NODE_LINUX_CPU) { + ws('workspace/ut-armv7-qemu') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('gpu', mx_lib, true) - utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_python', true) - utils.publish_test_coverage() + utils.unpack_and_init('armv7', mx_pip) + sh "ci/build.py --docker-registry ${env.DOCKER_CACHE_REGISTRY} -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu" } } } - }, - 'cpp-package GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/it-cpp-package') { + }] +} + +def docs_website() { + return ['Docs': { + node(NODE_LINUX_CPU) { + ws('workspace/docs') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('gpu', mx_lib_cpp_examples, true) - utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_cpp_package', true) - utils.publish_test_coverage() + utils.init_git() + utils.docker_run('ubuntu_cpu', 'deploy_docs', false) + + master_url = utils.get_jenkins_master_url() + if ( master_url == 'jenkins.mxnet-ci.amazon-ml.com') { + sh "ci/other/ci_deploy_doc.sh ${env.BRANCH_NAME} ${env.BUILD_NUMBER}" + } else { + print "Skipping staging documentation publishing since we are not running in prod. Host: {$master_url}" + } } } } - }, - // Disabled due to: https://github.com/apache/incubator-mxnet/issues/11407 - // 'Caffe GPU': { - // node(NODE_LINUX_GPU) { - // ws('workspace/it-caffe') { - // timeout(time: max_time, unit: 'MINUTES') { - // utils.init_git() - // utils.unpack_lib('gpu', mx_lib) - // utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_caffe', true) - // utils.publish_test_coverage() - // } - // } - // } - // }, - 'dist-kvstore tests GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/it-dist-kvstore') { + }] +} + +def docs_julia() { + return ['Julia docs': { + node(NODE_LINUX_CPU) { + ws('workspace/julia-docs') { timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('gpu', mx_lib, true) - utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_dist_kvstore', true) - utils.publish_test_coverage() + utils.unpack_and_init('cpu', mx_lib) + utils.docker_run('ubuntu_cpu', 'deploy_jl_docs', false) } } } - }, - 'dist-kvstore tests CPU': { + }] +} + +def misc_asan_cpu() { + return ['CPU ASAN': { node(NODE_LINUX_CPU) { - ws('workspace/it-dist-kvstore') { - timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('cpu', mx_lib, true) - utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_dist_kvstore', false) - utils.publish_test_coverage() - } + ws('workspace/ut-python3-cpu-asan') { + utils.unpack_and_init('cpu_asan', mx_lib_cpp_examples_cpu) + utils.docker_run('ubuntu_cpu', 'integrationtest_ubuntu_cpu_asan', false) } } - }, - 'Scala: GPU': { - node(NODE_LINUX_GPU) { - ws('workspace/ut-scala-gpu') { - timeout(time: max_time, unit: 'MINUTES') { - utils.unpack_and_init('gpu', mx_dist_lib, true) - utils.docker_run('ubuntu_gpu', 'integrationtest_ubuntu_gpu_scala', true) - utils.publish_test_coverage() - } + }] +} + +def sanity_lint() { + return ['Lint': { + node(NODE_LINUX_CPU) { + ws('workspace/sanity-lint') { + utils.init_git() + utils.docker_run('ubuntu_cpu', 'sanity_check', false) } } - } - } - - stage('Deploy') { - deploy_docs() - } + }] } -, -failure_handler: { - // Only send email if master or release branches failed - if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { - emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' - } + +def sanity_rat_license() { + return ['RAT License': { + node(NODE_LINUX_CPU) { + ws('workspace/sanity-rat') { + utils.init_git() + utils.docker_run('ubuntu_rat', 'nightly_test_rat_check', false) + } + } + }] } -) + +return this diff --git a/ci/jenkins/Jenkinsfile_centos_cpu b/ci/jenkins/Jenkinsfile_centos_cpu new file mode 100644 index 000000000..a47ab3de7 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_centos_cpu @@ -0,0 +1,53 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_centos7_cpu(), + custom_steps.compile_centos7_cpu_mkldnn() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_centos7_python3_cpu(), + custom_steps.test_centos7_scala_cpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_centos_gpu b/ci/jenkins/Jenkinsfile_centos_gpu new file mode 100644 index 000000000..cad77a9a7 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_centos_gpu @@ -0,0 +1,51 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_centos7_gpu() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_centos7_python3_gpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_clang b/ci/jenkins/Jenkinsfile_clang new file mode 100644 index 000000000..029c72081 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_clang @@ -0,0 +1,51 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_unix_clang_3_9_cpu(), + custom_steps.compile_unix_clang_6_cpu(), + custom_steps.compile_unix_clang_tidy_cpu(), + custom_steps.compile_unix_clang_3_9_mkldnn_cpu(), + custom_steps.compile_unix_clang_6_mkldnn_cpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_edge b/ci/jenkins/Jenkinsfile_edge new file mode 100644 index 000000000..9d8e01399 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_edge @@ -0,0 +1,56 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_armv8_jetson_gpu(), + custom_steps.compile_armv7_cpu(), + custom_steps.compile_armv6_cpu(), + custom_steps.compile_armv8_cpu(), + custom_steps.compile_armv8_android_cpu(), + custom_steps.compile_armv7_android_cpu() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_qemu_armv7_cpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_miscellaneous b/ci/jenkins/Jenkinsfile_miscellaneous new file mode 100644 index 000000000..dbf2a9e41 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_miscellaneous @@ -0,0 +1,54 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_unix_asan_cpu(), + custom_steps.compile_unix_amalgamation_min(), + custom_steps.compile_unix_amalgamation() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.misc_asan_cpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_sanity b/ci/jenkins/Jenkinsfile_sanity new file mode 100644 index 000000000..ed4d16ec4 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_sanity @@ -0,0 +1,48 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Sanity Check', [ + custom_steps.sanity_lint(), + custom_steps.sanity_rat_license() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_unix_cpu b/ci/jenkins/Jenkinsfile_unix_cpu new file mode 100644 index 000000000..ea3c06175 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_unix_cpu @@ -0,0 +1,74 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_unix_cpu_openblas(), + custom_steps.compile_unix_openblas_debug_cpu(), + custom_steps.compile_unix_mkl_cpu(), + custom_steps.compile_unix_mkldnn_cpu(), + custom_steps.compile_unix_mkldnn_mkl_cpu() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_unix_python2_cpu(), + custom_steps.test_unix_python3_cpu(), + custom_steps.test_unix_python3_debug_cpu(), + custom_steps.test_unix_python3_mkl_cpu(), + custom_steps.test_unix_python2_mkldnn_cpu(), + custom_steps.test_unix_python3_mkldnn_cpu(), + custom_steps.test_unix_python3_mkldnn_mkl_cpu(), + custom_steps.test_unix_scala_cpu(), + custom_steps.test_unix_scala_mkldnn_cpu(), + custom_steps.test_unix_clojure_cpu(), + custom_steps.test_unix_r_cpu(), + custom_steps.test_unix_julia07_cpu(), + custom_steps.test_unix_julia10_cpu(), + custom_steps.test_unix_onnx_cpu(), + custom_steps.test_unix_cpp_cpu(), + /* Disabled due to master build failure: + * http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1221/pipeline/ + * https://github.com/apache/incubator-mxnet/issues/11801 + custom_steps.test_unix_distributed_kvstore_cpu() + */ + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_unix_gpu b/ci/jenkins/Jenkinsfile_unix_gpu new file mode 100644 index 000000000..bd884904d --- /dev/null +++ b/ci/jenkins/Jenkinsfile_unix_gpu @@ -0,0 +1,73 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_unix_mkldnn_gpu(), + custom_steps.compile_unix_mkldnn_nocudnn_gpu(), + custom_steps.compile_unix_full_gpu(), + custom_steps.compile_unix_cmake_mkldnn_gpu(), + custom_steps.compile_unix_cmake_gpu(), + custom_steps.compile_unix_tensorrt_gpu(), + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_unix_python2_gpu(), + custom_steps.test_unix_python3_gpu(), + custom_steps.test_unix_python2_quantize_gpu(), + custom_steps.test_unix_python3_quantize_gpu(), + custom_steps.test_unix_python2_mkldnn_gpu(), + custom_steps.test_unix_python3_mkldnn_gpu(), + custom_steps.test_unix_python3_mkldnn_nocudnn_gpu(), + custom_steps.test_unix_python3_tensorrt_gpu(), + custom_steps.test_unix_r_gpu(), + custom_steps.test_unix_cpp_gpu(), + custom_steps.test_unix_cpp_mkldnn_gpu(), + custom_steps.test_unix_python3_integration_gpu(), + custom_steps.test_unix_cpp_package_gpu(), + custom_steps.test_unix_scala_gpu(), + custom_steps.test_unix_distributed_kvstore_gpu() + + // Disabled due to: https://github.com/apache/incubator-mxnet/issues/11407 + //custom_steps.test_unix_caffe_gpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_website b/ci/jenkins/Jenkinsfile_website new file mode 100644 index 000000000..acdd2be4d --- /dev/null +++ b/ci/jenkins/Jenkinsfile_website @@ -0,0 +1,52 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_unix_cpu_openblas() + ]) + + utils.parallel_stage('Deploy', [ + custom_steps.docs_website(), + custom_steps.docs_julia() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_windows_cpu b/ci/jenkins/Jenkinsfile_windows_cpu new file mode 100644 index 000000000..a8746db73 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_windows_cpu @@ -0,0 +1,52 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', windows_cpu: 'mxnetwindows-cpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_windows_cpu() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_windows_python2_cpu(), + custom_steps.test_windows_python3_cpu() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/jenkins/Jenkinsfile_windows_gpu b/ci/jenkins/Jenkinsfile_windows_gpu new file mode 100644 index 000000000..2319f2594 --- /dev/null +++ b/ci/jenkins/Jenkinsfile_windows_gpu @@ -0,0 +1,54 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +// timeout in minutes +max_time = 180 + +node('utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') + custom_steps = load('ci/jenkins/Jenkins_steps.groovy') +} +utils.assign_node_labels(utility: 'utility', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') + +utils.main_wrapper( +core_logic: { + utils.parallel_stage('Build', [ + custom_steps.compile_windows_gpu(), + custom_steps.compile_windows_gpu_mkldnn() + ]) + + utils.parallel_stage('Tests', [ + custom_steps.test_windows_python2_gpu(), + custom_steps.test_windows_python3_gpu(), + custom_steps.test_windows_python3_gpu_mkldnn() + ]) +} +, +failure_handler: { + // Only send email if master or release branches failed + if (currentBuild.result == "FAILURE" && (env.BRANCH_NAME == "master" || env.BRANCH_NAME.startsWith("v"))) { + emailext body: 'Build for MXNet branch ${BRANCH_NAME} has broken. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[BUILD FAILED] Branch ${BRANCH_NAME} build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/publish/Jenkinsfile b/ci/publish/Jenkinsfile new file mode 100644 index 000000000..9a360c6b5 --- /dev/null +++ b/ci/publish/Jenkinsfile @@ -0,0 +1,107 @@ +// -*- mode: groovy -*- + +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// Jenkins pipeline +// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ + +//mxnet libraries +mx_scala_pub = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a, 3rdparty/ps-lite/build/libps.a, deps/lib/libprotobuf-lite.a, deps/lib/libzmq.a, config.mk, scala-package/pom.xml, scala-package/**/pom.xml, scala-package/*/target/**, scala-package/local-snapshot/**' + +// timeout in minutes +max_time = 120 + +node('restricted-utility') { + // Loading the utilities requires a node context unfortunately + checkout scm + utils = load('ci/Jenkinsfile_utils.groovy') +} +utils.assign_node_labels(utility: 'restricted-utility', linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu') + +// CPU and GPU. OSX nodes are not currently supported by Jenkins +def nodeMap = ['cpu': NODE_LINUX_CPU, 'gpu': NODE_LINUX_GPU] +def scalaOSMap = ['cpu': 'linux-x86_64-cpu', 'gpu': 'linux-x86_64-gpu'] + +def wrapStep(nodeToRun, workspaceName, step) { + return { + node(nodeToRun) { + ws("workspace/${workspaceName}") { + timeout(time: max_time, unit: 'MINUTES') { + step() + } + } + } + } +} + +def toBuild = [:] +def labels = ['cpu'] // , 'gpu'] +for (x in labels) { + def label = x // Required due to language + toBuild["Scala Build ${label}"] = wrapStep(nodeMap[label], "build-scala-${label}") { + withEnv(["MAVEN_PUBLISH_OS_TYPE=${scalaOSMap[label]}"]) { + utils.init_git() + utils.docker_run("ubuntu_${label}", 'publish_scala_build', label == 'gpu', '500m', 'MAVEN_PUBLISH_OS_TYPE') + utils.pack_lib("scala_${label}", mx_scala_pub, false) + } + } +} + +def toTest = [:] +def systems = ['ubuntu1604', 'ubuntu1804', 'centos7'] +for (x in labels) { + def label = x // Required due to language + for (y in systems) { + def system = y // Required due to language + toTest["Scala Test ${system} ${label}"] = wrapStep(nodeMap[label], "test-scala-${system}-${label}") { + utils.unpack_and_init("scala_${label}", mx_scala_pub, false) + utils.docker_run("publish.test.${system}_${label}", 'publish_scala_test', label == 'gpu') + } + } +} + +def toDeploy = [:] +for (x in labels) { + def label = x // Required due to language + toDeploy["Scala Deploy ${label}"] = wrapStep(nodeMap[label], "deploy-scala-${label}") { + withEnv(["MAVEN_PUBLISH_OS_TYPE=${scalaOSMap[label]}"]) { + utils.unpack_and_init("scala_${label}", mx_scala_pub, false) + utils.docker_run("ubuntu_${label}", 'publish_scala_deploy', label == 'gpu', '500m', 'MAVEN_PUBLISH_OS_TYPE MAVEN_PUBLISH_SECRET_ENDPOINT_URL MAVEN_PUBLISH_SECRET_NAME_CREDENTIALS MAVEN_PUBLISH_SECRET_NAME_GPG DOCKERHUB_SECRET_ENDPOINT_REGION') + } + } +} + +utils.main_wrapper( +core_logic: { + stage('Build Packages') { + parallel toBuild + } + stage('Test Packages') { + parallel toTest + } + stage('Deploy Packages') { + parallel toDeploy + } +} +, +failure_handler: { + if (currentBuild.result == "FAILURE") { + // emailext body: 'Generating the nightly maven has failed. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[NIGHTLY MAVEN FAILED] Build ${BUILD_NUMBER}', to: '${EMAIL}' + } +} +) diff --git a/ci/publish/scala/build.sh b/ci/publish/scala/build.sh new file mode 100755 index 000000000..17f969afe --- /dev/null +++ b/ci/publish/scala/build.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -ex + +# Setup Environment Variables +# MAVEN_PUBLISH_OS_TYPE: linux-x86_64-cpu|linux-x86_64-gpu|osx-x86_64-cpu +# export MAVEN_PUBLISH_OS_TYPE=linux-x86_64-cpu + +bash scala-package/dev/compile-mxnet-backend.sh $MAVEN_PUBLISH_OS_TYPE ./ + +# Compile tests for discovery later +cd scala-package +mvn -B deploy diff --git a/ci/publish/scala/buildkey.py b/ci/publish/scala/buildkey.py new file mode 100644 index 000000000..8a1b7bf63 --- /dev/null +++ b/ci/publish/scala/buildkey.py @@ -0,0 +1,165 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import os +import json +import logging +import subprocess + +HOME = os.environ['HOME'] +KEY_PATH = os.path.join(HOME, ".m2") + + +''' +This file would do the following items: + Import keys from AWS Credential services + Create settings.xml in .m2 with pass phrase + Create security-settings.xml in .m2 with master password + Import keys.asc the encrypted keys in gpg +''' + + +def getCredentials(): + import boto3 + import botocore + endpoint_url = os.environ['MAVEN_PUBLISH_SECRET_ENDPOINT_URL'] + secret_creds_name = os.environ['MAVEN_PUBLISH_SECRET_NAME_CREDENTIALS'] + secret_key_name = os.environ['MAVEN_PUBLISH_SECRET_NAME_GPG'] + region_name = os.environ['DOCKERHUB_SECRET_ENDPOINT_REGION'] + + session = boto3.Session() + client = session.client( + service_name='secretsmanager', + region_name=region_name, + endpoint_url=endpoint_url + ) + try: + get_secret_value_response = client.get_secret_value( + SecretId=secret_creds_name + ) + get_secret_key_response = client.get_secret_value( + SecretId=secret_key_name + ) + except botocore.exceptions.ClientError as client_error: + if client_error.response['Error']['Code'] == 'ResourceNotFoundException': + name = (secret_key_name if get_secret_value_response + else secret_creds_name) + logging.exception("The requested secret %s was not found", name) + elif client_error.response['Error']['Code'] == 'InvalidRequestException': + logging.exception("The request was invalid due to:") + elif client_error.response['Error']['Code'] == 'InvalidParameterException': + logging.exception("The request had invalid params:") + raise + else: + secret = get_secret_value_response['SecretString'] + secret_dict = json.loads(secret) + secret_key = get_secret_key_response['SecretString'] + return secret_dict, secret_key + + +def importASC(key, gpgPassphrase): + filename = os.path.join(KEY_PATH, "key.asc") + with open(filename, 'w') as f: + f.write(key) + subprocess.check_output(['gpg2', '--batch', '--yes', + '--passphrase-fd', '0', + "--import", "{}".format(filename)], + input=str.encode(gpgPassphrase)) + + +def encryptMasterPSW(password): + filename = os.path.join(KEY_PATH, "encryptMasterPassword.exp") + with open(filename, 'w') as f: + f.write(''' + spawn mvn --encrypt-master-password + expect -exact "Master password: " + send -- "{}\r" + expect eof + '''.format(password)) + result = subprocess.check_output(['expect', filename]) + return str(result).split('\r\n')[-1][2:-3] + + +def encryptPSW(password): + filename = os.path.join(KEY_PATH, "encryptPassword.exp") + with open(filename, 'w') as f: + f.write(''' + spawn mvn --encrypt-password + expect -exact "Password: " + send -- "{}\r" + expect eof + '''.format(password)) + result = subprocess.check_output(['expect', filename]) + return str(result).split('\r\n')[-1][2:-3] + + +def masterPSW(password): + with open(os.path.join(KEY_PATH, "settings-security.xml"), "w") as f: + f.write("\n {}\n" + .format(password)) + + +def serverPSW(username, password, gpgPassphrase): + with open(os.path.join(KEY_PATH, "settings.xml"), "w") as f: + settingsString = ''' + + + + + + apache.snapshots.https + {} + {} + + + + apache.releases.https + {} + {} + + + + + + gpg + + gpg2 + {} + true + + + + + gpg + + '''.format(username, password, username, password, gpgPassphrase) + f.write(settingsString) + + +if __name__ == "__main__": + if not os.path.exists(KEY_PATH): + os.makedirs(KEY_PATH) + credentials, gpgKey = getCredentials() + masterPass = encryptMasterPSW(credentials['masterpass']) + masterPSW(masterPass) + passwordEncrypted = encryptPSW(credentials['password']) + serverPSW(credentials['user'], passwordEncrypted, + credentials['gpgPassphrase']) + importASC(gpgKey, credentials['gpgPassphrase']) diff --git a/ci/publish/scala/deploy.sh b/ci/publish/scala/deploy.sh new file mode 100755 index 000000000..4eb33907e --- /dev/null +++ b/ci/publish/scala/deploy.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -ex + +# Setup Environment Variables +# MAVEN_PUBLISH_OS_TYPE: linux-x86_64-cpu|linux-x86_64-gpu|osx-x86_64-cpu +# export MAVEN_PUBLISH_OS_TYPE=linux-x86_64-cpu + +# Run python to configure keys +python3 ci/publish/scala/buildkey.py + +# Updating cache +mkdir -p ~/.gnupg +echo "default-cache-ttl 14400" > ~/.gnupg/gpg-agent.conf +echo "max-cache-ttl 14400" >> ~/.gnupg/gpg-agent.conf +echo "allow-loopback-pinentry" >> ~/.gnupg/gpg-agent.conf +echo "pinentry-mode loopback" >> ~/.gnupg/gpg-agent.conf +export GPG_TTY=$(tty) + +cd scala-package + +mvn -B deploy -Pnightly + +# Clear all password .xml files, exp files, and gpg key files +rm -rf ~/.m2/*.xml ~/.m2/key.asc ~/.m2/*.exp diff --git a/ci/publish/scala/fullDeploy.sh b/ci/publish/scala/fullDeploy.sh new file mode 100644 index 000000000..69d674a97 --- /dev/null +++ b/ci/publish/scala/fullDeploy.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -ex + +./ci/publish/scala/build.sh +./ci/publish/scala/test.sh +./ci/publish/scala/deploy.sh diff --git a/ci/publish/scala/test.sh b/ci/publish/scala/test.sh new file mode 100755 index 000000000..5cef35ca3 --- /dev/null +++ b/ci/publish/scala/test.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -ex + +if [ -z "$JAVA_HOME" ]; then + source /etc/profile +fi + +# Test +cd scala-package/packageTest +# make testlocal CI=1 +make testsnapshot UNIT=1 CI=1 diff --git a/ci/requirements.txt b/ci/requirements.txt new file mode 100644 index 000000000..8f21ead27 --- /dev/null +++ b/ci/requirements.txt @@ -0,0 +1 @@ +docker==3.5.0 diff --git a/ci/test_docker_cache.py b/ci/test_docker_cache.py index 358d54985..0a3bc4640 100644 --- a/ci/test_docker_cache.py +++ b/ci/test_docker_cache.py @@ -135,7 +135,7 @@ def test_full_cache(self): """ platform = 'test_full_cache' docker_tag = build_util.get_docker_tag(platform=platform, registry=DOCKER_REGISTRY_PATH) - dockerfile_path = os.path.join(DOCKERFILE_DIR, 'Dockerfile.build.' + platform) + dockerfile_path = os.path.join(DOCKERFILE_DIR, 'Dockerfile.' + platform) try: with open(dockerfile_path, 'w') as dockerfile_handle: dockerfile_handle.write(dockerfile_content) @@ -196,7 +196,7 @@ def test_partial_cache(self): """ platform = 'test_partial_cache' docker_tag = build_util.get_docker_tag(platform=platform, registry=DOCKER_REGISTRY_PATH) - dockerfile_path = os.path.join(DOCKERFILE_DIR, 'Dockerfile.build.' + platform) + dockerfile_path = os.path.join(DOCKERFILE_DIR, 'Dockerfile.' + platform) try: # Write initial Dockerfile with open(dockerfile_path, 'w') as dockerfile_handle: diff --git a/ci/util.py b/ci/util.py index 4d68b57a3..9a8d52eb1 100644 --- a/ci/util.py +++ b/ci/util.py @@ -18,7 +18,6 @@ import os import contextlib import logging -import requests def get_mxnet_root() -> str: curpath = os.path.abspath(os.path.dirname(__file__)) @@ -89,6 +88,7 @@ def under_ci() -> bool: def ec2_instance_id_hostname() -> str: + import requests if under_ci(): result = [] try: diff --git a/ci/windows/test_py2_cpu.ps1 b/ci/windows/test_py2_cpu.ps1 index aa38b81e3..46e49baea 100644 --- a/ci/windows/test_py2_cpu.ps1 +++ b/ci/windows/test_py2_cpu.ps1 @@ -16,11 +16,14 @@ # under the License. 7z x -y windows_package.7z + $env:MXNET_LIBRARY_PATH=join-path $pwd.Path windows_package\lib\libmxnet.dll $env:PYTHONPATH=join-path $pwd.Path windows_package\python $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 +$env:MXNET_HOME=[io.path]::combine($PSScriptRoot, 'mxnet_home') + c:\Anaconda3\envs\py2\Scripts\pip install -r tests\requirements.txt -c:\Anaconda3\envs\py2\python.exe -m nose -v --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest if (! $?) { Throw ("Error running unittest") } -c:\Anaconda3\envs\py2\python.exe -m nose -v --with-xunit --xunit-file nosetests_train.xml tests\python\train +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_train.xml tests\python\train if (! $?) { Throw ("Error running train tests") } diff --git a/ci/windows/test_py2_gpu.ps1 b/ci/windows/test_py2_gpu.ps1 index 5f8de5ac4..d362c61da 100644 --- a/ci/windows/test_py2_gpu.ps1 +++ b/ci/windows/test_py2_gpu.ps1 @@ -16,15 +16,18 @@ # under the License. 7z x -y windows_package.7z + $env:MXNET_LIBRARY_PATH=join-path $pwd.Path windows_package\lib\libmxnet.dll $env:PYTHONPATH=join-path $pwd.Path windows_package\python $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 +$env:MXNET_HOME=[io.path]::combine($PSScriptRoot, 'mxnet_home') + c:\Anaconda3\envs\py2\Scripts\pip install -r tests\requirements.txt -c:\Anaconda3\envs\py2\python.exe -m nose -v --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest if (! $?) { Throw ("Error running unittest") } -c:\Anaconda3\envs\py2\python.exe -m nose -v --with-xunit --xunit-file nosetests_operator.xml tests\python\gpu\test_operator_gpu.py +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_operator.xml tests\python\gpu\test_operator_gpu.py if (! $?) { Throw ("Error running tests") } -c:\Anaconda3\envs\py2\python.exe -m nose -v --with-xunit --xunit-file nosetests_forward.xml tests\python\gpu\test_forward.py +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_forward.xml tests\python\gpu\test_forward.py if (! $?) { Throw ("Error running tests") } -c:\Anaconda3\envs\py2\python.exe -m nose -v tests\python\train +c:\Anaconda3\envs\py2\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error tests\python\train if (! $?) { Throw ("Error running tests") } diff --git a/ci/windows/test_py3_cpu.ps1 b/ci/windows/test_py3_cpu.ps1 index 0dd48de26..32da4885f 100644 --- a/ci/windows/test_py3_cpu.ps1 +++ b/ci/windows/test_py3_cpu.ps1 @@ -16,11 +16,14 @@ # under the License. 7z x -y windows_package.7z + $env:MXNET_LIBRARY_PATH=join-path $pwd.Path windows_package\lib\libmxnet.dll $env:PYTHONPATH=join-path $pwd.Path windows_package\python $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 +$env:MXNET_HOME=[io.path]::combine($PSScriptRoot, 'mxnet_home') + c:\Anaconda3\envs\py3\Scripts\pip install -r tests\requirements.txt -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest if (! $?) { Throw ("Error running unittest") } -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_train.xml tests\python\train +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_train.xml tests\python\train if (! $?) { Throw ("Error running train tests") } diff --git a/ci/windows/test_py3_gpu.ps1 b/ci/windows/test_py3_gpu.ps1 index 4a0feb1ed..b30b22ae9 100644 --- a/ci/windows/test_py3_gpu.ps1 +++ b/ci/windows/test_py3_gpu.ps1 @@ -16,15 +16,18 @@ # under the License. 7z x -y windows_package.7z + $env:MXNET_LIBRARY_PATH=join-path $pwd.Path windows_package\lib\libmxnet.dll $env:PYTHONPATH=join-path $pwd.Path windows_package\python $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 +$env:MXNET_HOME=[io.path]::combine($PSScriptRoot, 'mxnet_home') + c:\Anaconda3\envs\py3\Scripts\pip install -r tests\requirements.txt -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_unittest.xml tests\python\unittest if (! $?) { Throw ("Error running unittest") } -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_operator.xml tests\python\gpu\test_operator_gpu.py +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_operator.xml tests\python\gpu\test_operator_gpu.py if (! $?) { Throw ("Error running tests") } -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_forward.xml tests\python\gpu\test_forward.py +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_forward.xml tests\python\gpu\test_forward.py if (! $?) { Throw ("Error running tests") } -c:\Anaconda3\envs\py3\python.exe -m nose -v --with-xunit --xunit-file nosetests_train.xml tests\python\train +c:\Anaconda3\envs\py3\python.exe -m nose -v --with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error --with-xunit --xunit-file nosetests_train.xml tests\python\train if (! $?) { Throw ("Error running tests") } diff --git a/cmake/AutoDetectF16C.cmake b/cmake/AutoDetectF16C.cmake new file mode 100644 index 000000000..04331c3ac --- /dev/null +++ b/cmake/AutoDetectF16C.cmake @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Determines whether hardware and compiler support F16C +# instruction set +# +# The following are set after configuration is done: +# SUPPORT_F16C + +if(AUTO_DETECT_F16_CMAKE_INCLUDED) + return() +endif() +set(AUTO_DETECT_F16_CMAKE_INCLUDED True) + +set(SUPPORT_F16C False) +if(MSVC) + message("F16C instruction set is not yet supported for MSVC") + return() +endif() +include(CheckCXXCompilerFlag) +check_cxx_compiler_flag("-mf16c" COMPILER_SUPPORT_MF16C) +if(CMAKE_SYSTEM_NAME STREQUAL "Linux") + execute_process(COMMAND cat /proc/cpuinfo + COMMAND grep flags + COMMAND grep f16c + OUTPUT_VARIABLE CPU_SUPPORT_F16C) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + execute_process(COMMAND sysctl -a + COMMAND grep machdep.cpu.features + COMMAND grep F16C + OUTPUT_VARIABLE CPU_SUPPORT_F16C) +endif() +if(NOT CPU_SUPPORT_F16C) + message("CPU does not support F16C instructions") + return() +endif() +if(CPU_SUPPORT_F16C AND COMPILER_SUPPORT_MF16C) + set(SUPPORT_F16C TRUE) +endif() diff --git a/cmake/DownloadMKLML.cmake b/cmake/DownloadMKLML.cmake new file mode 100644 index 000000000..eabf861a4 --- /dev/null +++ b/cmake/DownloadMKLML.cmake @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script will download MKLML + +message(STATUS "Downloading MKLML...") + +set(MKLDNN_RELEASE v0.17-rc) +set(MKLML_RELEASE_FILE_SUFFIX 2019.0.1.20180928) + +if(MSVC) + set(MKL_NAME "mklml_win_${MKLML_RELEASE_FILE_SUFFIX}") + + file(DOWNLOAD "https://github.com/intel/mkl-dnn/releases/download/${MKLDNN_RELEASE}/${MKL_NAME}.zip" + "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.zip" + EXPECTED_MD5 "443e661bdfd32dbbc99b460b43afceee" SHOW_PROGRESS) + file(DOWNLOAD "https://github.com/apache/incubator-mxnet/releases/download/utils/7z.exe" + "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z2.exe" + EXPECTED_MD5 "E1CF766CF358F368EC97662D06EA5A4C" SHOW_PROGRESS) + + execute_process(COMMAND "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z2.exe" "-o${CMAKE_CURRENT_BINARY_DIR}/mklml/" "-y") + execute_process(COMMAND "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z.exe" + "x" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.zip" "-o${CMAKE_CURRENT_BINARY_DIR}/mklml/" "-y") + + set(MKLROOT "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}") + + message(STATUS "Setting MKLROOT path to ${MKLROOT}") + + include_directories(${MKLROOT}/include) + +elseif(APPLE) + set(MKL_NAME "mklml_mac_${MKLML_RELEASE_FILE_SUFFIX}") + + file(DOWNLOAD "https://github.com/intel/mkl-dnn/releases/download/${MKLDNN_RELEASE}/${MKL_NAME}.tgz" + "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" + EXPECTED_MD5 "95f887af332205b1d15b392260003952" SHOW_PROGRESS) + execute_process(COMMAND "tar" "-xzf" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" + "-C" "${CMAKE_CURRENT_BINARY_DIR}/mklml/") + + set(MKLROOT "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}") + + message(STATUS "Setting MKLROOT path to ${MKLROOT}") + include_directories(${MKLROOT}/include) + +elseif(UNIX) + set(MKL_NAME "mklml_lnx_${MKLML_RELEASE_FILE_SUFFIX}") + + file(DOWNLOAD "https://github.com/intel/mkl-dnn/releases/download/${MKLDNN_RELEASE}/${MKL_NAME}.tgz" + "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" + EXPECTED_MD5 "a63abf155361322b9c03f8fc50f4f317" SHOW_PROGRESS) + execute_process(COMMAND "tar" "-xzf" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" + "-C" "${CMAKE_CURRENT_BINARY_DIR}/mklml/") + + set(MKLROOT "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}") + message(STATUS "Setting MKLROOT path to ${MKLROOT}") + include_directories(${MKLROOT}/include) + +else() + message(FATAL_ERROR "Wrong platform") +endif() diff --git a/cmake/FirstClassLangCuda.cmake b/cmake/FirstClassLangCuda.cmake index 4b44077d2..8d79c2b63 100644 --- a/cmake/FirstClassLangCuda.cmake +++ b/cmake/FirstClassLangCuda.cmake @@ -120,6 +120,16 @@ else() list(APPEND CUDA_COMMON_GPU_ARCHITECTURES "5.2+PTX") endif () +if (CUDA_TOOLSET VERSION_GREATER "9.0") + list(APPEND CUDA_KNOWN_GPU_ARCHITECTURES "Volta") + list(APPEND CUDA_COMMON_GPU_ARCHITECTURES "7.0") +endif() + +if (CUDA_TOOLSET VERSION_GREATER "10.0") + list(APPEND CUDA_KNOWN_GPU_ARCHITECTURES "Turing") + list(APPEND CUDA_COMMON_GPU_ARCHITECTURES "7.5") +endif() + ################################################################################################ # Function for selecting GPU arch flags for nvcc based on CUDA_ARCH_NAME # Usage: @@ -185,6 +195,12 @@ function(mshadow_select_nvcc_arch_flags out_variable) elseif(${arch_name} STREQUAL "Pascal") set(arch_bin 6.0 6.1) set(arch_ptx 6.1) + elseif(${arch_name} STREQUAL "Volta") + set(arch_bin 7.0) + set(arch_ptx 7.0) + elseif(${arch_name} STREQUAL "Turing") + set(arch_bin 7.5) + set(arch_ptx 7.5) else() message(SEND_ERROR "Unknown CUDA Architecture Name ${arch_name} in CUDA_SELECT_NVCC_ARCH_FLAGS") endif() diff --git a/cmake/MklDnn.cmake b/cmake/MklDnn.cmake deleted file mode 100644 index acaf878b2..000000000 --- a/cmake/MklDnn.cmake +++ /dev/null @@ -1,44 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -#this file download mklml - -message(STATUS "download mklml") -if(MSVC) - set(MKL_NAME "mklml_win_2018.0.3.20180406") - file(DOWNLOAD "https://github.com/intel/mkl-dnn/releases/download/v0.14/${MKL_NAME}.zip" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.zip" EXPECTED_MD5 "8DD73E7D3F19F004551809824C4E8970" SHOW_PROGRESS) - file(DOWNLOAD "https://github.com/apache/incubator-mxnet/releases/download/utils/7z.exe" "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z2.exe" EXPECTED_MD5 "E1CF766CF358F368EC97662D06EA5A4C" SHOW_PROGRESS) - - execute_process(COMMAND "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z2.exe" "-o${CMAKE_CURRENT_BINARY_DIR}/mklml/" "-y") - execute_process(COMMAND "${CMAKE_CURRENT_BINARY_DIR}/mklml/7z.exe" "x" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.zip" "-o${CMAKE_CURRENT_BINARY_DIR}/mklml/" "-y") - set(MKLROOT "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}") - include_directories(${MKLROOT}/include) - file(COPY ${MKLROOT}/lib/libiomp5md.dll DESTINATION ${CMAKE_CURRENT_BINARY_DIR}) - file(COPY ${MKLROOT}/lib/mklml.dll DESTINATION ${CMAKE_CURRENT_BINARY_DIR}) - file(COPY ${CMAKE_SOURCE_DIR}/3rdparty/mkldnn/config_template.vcxproj.user DESTINATION ${CMAKE_SOURCE_DIR}) -elseif(UNIX) - set(MKL_NAME "mklml_lnx_2018.0.3.20180406") - file(DOWNLOAD "https://github.com/intel/mkl-dnn/releases/download/v0.14/${MKL_NAME}.tgz" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" EXPECTED_MD5 "DAF7EFC3C1C0036B447213004467A8AE" SHOW_PROGRESS) - execute_process(COMMAND "tar" "-xzf" "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}.tgz" "-C" "${CMAKE_CURRENT_BINARY_DIR}/mklml/") - set(MKLROOT "${CMAKE_CURRENT_BINARY_DIR}/mklml/${MKL_NAME}") - include_directories(${MKLROOT}/include) - file(COPY ${MKLROOT}/lib/libiomp5.so DESTINATION ${CMAKE_CURRENT_BINARY_DIR}) - file(COPY ${MKLROOT}/lib/libmklml_gnu.so DESTINATION ${CMAKE_CURRENT_BINARY_DIR}) - file(COPY ${MKLROOT}/lib/libmklml_intel.so DESTINATION ${CMAKE_CURRENT_BINARY_DIR}) -else() - message(FATAL_ERROR "not support now") -endif() diff --git a/cmake/cmake_options.yml b/cmake/cmake_options.yml new file mode 100644 index 000000000..a4323feb9 --- /dev/null +++ b/cmake/cmake_options.yml @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +--- # CMake configuration +USE_CUDA: "OFF" # Build with CUDA support +USE_OLDCMAKECUDA: "OFF" # Build with old cmake cuda +USE_NCCL: "OFF" # Use NVidia NCCL with CUDA +USE_OPENCV: "ON" # Build with OpenCV support +USE_OPENMP: "ON" # Build with Openmp support +USE_CUDNN: "ON" # Build with cudnn support) # one could set CUDNN_ROOT for search path +USE_SSE: "ON" # Build with x86 SSE instruction support IF NOT ARM +USE_F16C: "ON" # Build with x86 F16C instruction support) # autodetects support if "ON" +USE_LAPACK: "ON" # Build with lapack support +USE_MKL_IF_AVAILABLE: "ON" # Use MKL if found +USE_MKLML_MKL: "ON" # Use MKLDNN variant of MKL (if MKL found) IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) +USE_MKLDNN: "ON" # Use MKLDNN variant of MKL (if MKL found) IF USE_MKL_IF_AVAILABLE AND (NOT APPLE) +USE_OPERATOR_TUNING: "ON" # Enable auto-tuning of operators IF NOT MSVC +USE_GPERFTOOLS: "ON" # Build with GPerfTools support (if found) +USE_JEMALLOC: "ON" # Build with Jemalloc support +USE_PROFILER: "ON" # Build with Profiler support +USE_DIST_KVSTORE: "OFF" # Build with DIST_KVSTORE support +USE_PLUGINS_WARPCTC: "OFF" # Use WARPCTC Plugins +USE_PLUGIN_CAFFE: "OFF" # Use Caffe Plugin +USE_CPP_PACKAGE: "OFF" # Build C++ Package +USE_MXNET_LIB_NAMING: "ON" # Use MXNet library naming conventions. +USE_GPROF: "OFF" # Compile with gprof (profiling) flag +USE_CXX14_IF_AVAILABLE: "OFF" # Build with C++14 if the compiler supports it +USE_VTUNE: "OFF" # Enable use of Intel Amplifier XE (VTune)) # one could set VTUNE_ROOT for search path +ENABLE_CUDA_RTC: "ON" # Build with CUDA runtime compilation support +BUILD_CPP_EXAMPLES: "ON" # Build cpp examples +INSTALL_EXAMPLES: "OFF" # Install the example source files. +USE_SIGNAL_HANDLER: "ON" # Print stack traces on segfaults. +USE_TENSORRT: "OFF" # Enable infeference optimization with TensorRT. +USE_ASAN: "OFF" # Enable Clang/GCC ASAN sanitizers. +ENABLE_TESTCOVERAGE: "OFF" # Enable compilation with test coverage metric output +CMAKE_BUILD_TYPE: "Debug" +CMAKE_CUDA_COMPILER_LAUNCHER: "ccache" +CMAKE_C_COMPILER_LAUNCHER: "ccache" +CMAKE_CXX_COMPILER_LAUNCHER: "ccache" diff --git a/contrib/clojure-package/LICENSE b/contrib/clojure-package/LICENSE index 8f71f43fe..26c4e047b 100644 --- a/contrib/clojure-package/LICENSE +++ b/contrib/clojure-package/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright {yyyy} {name of copyright owner} + Copyright 2018 by Contributors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/contrib/clojure-package/README.md b/contrib/clojure-package/README.md index 8c71224a3..ba6160aed 100644 --- a/contrib/clojure-package/README.md +++ b/contrib/clojure-package/README.md @@ -1,154 +1,227 @@ # Clojure MXNet -A clojure package to the MXNet Deep Learning library +A Clojure Package Built on the MXNet Deep Learning Library ## Introduction -MXNet is a first class, modern deep learning library. It supports multiple languages on a first class basis and is incubating as an Apache project. +MXNet is a flexible and efficient deep learning library. While its core is built in C++ for maximum performance, support for multiple programming languages in intermediate and high-level APIs is a first-class feature. MXNet is currently an incubating Apache project. -The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support. +The motivation for creating a Clojure package was to give Clojurians access to a world-class deep learning platform, thereby building bridges for future development and innovation in the community. The Clojure package provides all the essential tools, including low-level and high-level APIs, dynamic graphs, etc., and enables building advanced architectures like GANs or LSTM to tackle challenging applications such as image recognition or natural language processing. -For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide. +To maximize leverage, the Clojure package has been built on the existing Scala package using [Java Interop](https://clojure.org/reference/java_interop). This approach has allowed rapid initial development and close parity with the Scala package functionality. It also leaves the door open to incrementally developing Clojure code that directly interfaces MXNet core using [JNI](https://en.wikipedia.org/wiki/Java_Native_Interface). -For a **video introduction**, see [Clojure MXNet with Carin Meier - Clojure Virtual Meetup](https://www.crowdcast.io/e/clojure-mxnet-with-carin) (setup instructions from 20:49) +For a **video introduction**, see [Clojure MXNet with Carin Meier - Clojure Virtual Meetup](https://www.crowdcast.io/e/clojure-mxnet-with-carin) (setup instructions from 20:49). ## Current State and Plans -The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package. +The Clojure MXNet package is currently treated as *user-contributed code* within MXNet, as can be seen from its placement under `contrib` in the source tree. This means that it should first undergo a stabilization period and receive feedback from users before it can graduate to a fully integrated and supported part of MXNet. -Help is needed testing and generally making the package better. A list of the pacakge status and contribution needs can be found here [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs). Please get involved :) +That said, because it closely tracks the Scala package, Clojure MXNet can be expected to have a similar level of maturity and stability regarding the low-level functionality. It is mostly in the hand-written Java interop part of the Clojure wrapper where bugs are more likely to be encountered. Such bugs tend to be fixed rather quickly once they are known and their origin is clear (see also [Getting Involved](#getting-involved)). -Testing instructions can be found in the testing.md. +For an overview of the development status and open problems, please refer to [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs). -## Getting Started +## Getting Involved + +By far the best way to get involved with this project is to install the Clojure MXNet package, run the examples, play around, build new things with it, and get back to the development team with feedback! Your input can not only help to identify current issues, but also guide the future development of the Clojure package by pointing out must-have features that are currently missing, or by identifying usability or performace problems of high impact. -The following systems are supported: +There are two main ways of reaching out to other users and the package maintainers: -- OSX cpu -- Linux cpu -- Linux gpu +- If you have a question or general feedback, or you encountered a problem but are not sure if it's a bug or a misunderstanding, then the *Apache Slack* (channels `#mxnet` and `#mxnet-scala`) is the best place to turn check out. To join, [ask for an invitation](https://mxnet.apache.org/community/contribute.html#slack) at `dev@mxnet.apache.org`. +- If you found a bug, miss an important feature or want to give feedback directly relevant for development, please head over to the MXNet [GitHub issue page](https://github.com/apache/incubator-mxnet/issues) and create a new issue. If the issue is specific to the Clojure package, consider using a title starting with `[Clojure]` to make it easily discoverable among the many other, mostly generic issues. -There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv). +Of course, contributions to code or documentation are also more than welcome! Please check out the [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs) to get an idea about where and how to contribute code. +For a more comprehensive overview of different ways to contribute, see [Contributing to MXNet](https://mxnet.apache.org/community/contribute.html). +## Getting Started -### Prerequisites +The Clojure MXNet framework consists of a core C library, a Scala API that talks to the core through [JNI (Java Native Interface)](https://en.wikipedia.org/wiki/Java_Native_Interface) bindings, and finally a Clojure wrapper around the Scala API. +Since the core contains native (compiled) code and is bundled with the language bindings, your hardware and OS matter to the choices to be made during installation. The following combinations of operating system and compute device are supported: -Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html -about _Prepare Environment for GPU Installation_ -and _Install MXNet dependencies_ +- Linux CPU +- Linux GPU +- OSX CPU +There are three ways of getting started: -#### Cloning the repo and running from source +1. [Install prebuilt Clojure jars](#option-1-clojure-package-from-prebuilt-jar) with the native dependencies baked in. This the quickest way to get going. +2. [Install the Clojure package from source, but use prebuilt jars for the native dependencies](#option-2-clojure-package-from-source-scala-package-from-jar). Choose this option if you want pre-release features of the Clojure package but don't want to build (compile) native dependencies yourself. +3. [Build everything from source](#option-3-everything-from-source). This option is for developers or advanced users who want cutting-edge features in all parts of the dependency chain. -To use the prebuilt jars (easiest), you will need to replace the native version of the line in the project dependencies with your configuration. +**Note:** This guide assumes that you are familiar with the basics of creating Clojure projects and managing dependencies. See [here](https://github.com/technomancy/leiningen/blob/stable/doc/TUTORIAL.md) for the official Leiningen tutorial. -`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.3.0"]` -or -`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.3.0"]` -or -`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0"]` +### Option 1: Clojure Package from Prebuilt Jar -If you are using the prebuilt jars they may have a slightly different dependencies then building from source: +If you are new to MXNet and just want to try things out, this option is the best way to get started. You will install release versions of MXNet core, MXNet Scala and MXNet Clojure. -*For OSX you will need:* +For reference, the Clojure MXNet jars can be found on [maven.org](https://search.maven.org/search?q=clojure%20mxnet). -`brew install opencv` +#### Installing additional dependencies -*For Ubuntu Linux you will need:* +Depending on your operating system, you will need a couple of packages that are not distributed through Maven: -``` -sudo add-apt-repository ppa:timsc/opencv-3.4 +- [OpenCV](https://opencv.org/) version 3.4 +- [OpenBLAS](https://www.openblas.net/) +- [ATLAS](http://math-atlas.sourceforge.net/) +- [cURL](https://curl.haxx.se/) library version 3 + +##### Linux (Ubuntu) + +As of writing this, OpenCV 3.4 is not available in the default repositories. Therefore, a third-party repository is needed. + +```bash +sudo add-apt-repository ppa:timsc/opencv sudo apt-get update -sudo apt install libopencv-imgcodecs3.4 +sudo apt install libopencv-imgcodecs3.4 libopenblas-base libatlas3-base libcurl3 ``` -*For Arch Linux you will need:* +Note: `libcurl3` may conflict with other packages on your system. [Here](https://github.com/apache/incubator-mxnet/issues/12822) is a possible workaround. -_CPU_ +##### Linux (Arch) -``` +```bash yaourt -S openblas-lapack yaourt -S libcurl-compat export LD_PRELOAD=libcurl.so.3 ``` -_GPU_ -``` +To enable GPU support, you will additionally need the CUDA toolkit: + +```bash wget https://archive.archlinux.org/packages/c/cuda/cuda-9.0.176-4-x86_64.pkg.tar.xz sudo pacman -U cuda-9.0.176-4-x86_64.pkg.tar.xz ``` -If you want to see the exact versions and flags that the jars were built with, look here: -[Scala Release Process](https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process) +##### OSX + +```bash +brew install wget +brew install opencv +``` + +#### Installing the Clojure package + +- Create a new project with `lein new my-mxnet` +- Edit your `project.clj` and add one of the following entries to `:dependencies`, based on your system and the compute device you want to use: + + + - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-cpu ]` + - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-gpu ]` + - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu ]` +You can find the latest version out on [maven central- clojure-mxnet latest](https://search.maven.org/search?q=clojure-mxnet) -Check your installation with `lein test`. If that works alright then, you can try some code! +After making this change and running `lein deps`, you should be able to run example code like this [NDArray Tutorial](https://github.com/apache/incubator-mxnet/blob/master/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj). -```clojure +### Option 2: Clojure package from Source, Scala Package from Jar -(ns tutorial.ndarray - (:require [org.apache.clojure-mxnet.ndarray :as ndarray] - [org.apache.clojure-mxnet.context :as context])) +With this option, you will install a Git revision of the Clojure package source and a [Scala package jar from Maven](https://search.maven.org/search?q=g:org.apache.mxnet) with native dependencies baked in. -;;Create NDArray -(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 -(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension -(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 +- Install additional dependencies as described in [the corresponding section for Option 1](#installing-additional-dependencies), -;;; There are also ways to convert to a vec or get the shape as an object or vec -(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] +- Recursively clone the MXNet repository and checkout the desired version, (example 1.3.1). You should use the latest [version](https://search.maven.org/search?q=clojure-mxnet)), and a clone into the `~/mxnet` directory: + + ```bash + git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet + cd ~/mxnet + git tag --list # Find the tag that matches the Scala package version + + git checkout tags/ -b my_mxnet + git submodule update --init --recursive + cd contrib/clojure + ``` + +- Edit `project.clj` to include the desired Scala jar from Maven: + + + [org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu ] + +- Run `lein test`. All the tests should run without error. +- At this point you can run `lein install` to build and install the Clojure jar locally. + +To run examples, you can now use `lein run` in any of the example directories, e.g., `examples/imclassification`. You can also specify the compute device, e.g., `lein run :cpu 2` (for 2 CPUs) or `lein run :gpu` (for 1 GPU). + +#### Experimental: Using Scala Snapshot Jars +**Note:** Instead of a release tag, you can also use a development version of the Clojure package, e.g., Git `master`, together with the prebuilt Scala jar. There is a repo of nightly built snapshots of Scala jars. You can use them in your `project.clj` by adding a repository: + +``` +["snapshots" {:url "https://repository.apache.org/content/repositories/snapshots" + :snapshots true + :sign-releases false + :checksum :fail + :update :always + :releases {:checksum :fail :update :always}}] ``` -See the examples/tutorial section for more. +Then you should be able to run with your dependency: + [org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "latest-version-SNAPSHOT"] -The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted. +In that case, however, breakage can happen at any point, for instance when the Scala development version adds, changes or removes an interface and the Clojure development version moves along. If you really need the most recent version, you should consider [installation option 3](#option-3-everything-from-source). -### Build from MXNET Source +### Option 3: Everything from Source -First, ensure you have JDK 8 on your system. Later versions may produce cryptic build errors mentioning `scala.reflect.internal.MissingRequirementError`. +With this option, you will compile the core MXNet C++ package and jars for both Scala and Clojure language bindings from source. If you intend to make changes to the code in any of the parts, or if you simply want the latest and greatest features, this choice is for you. -Checkout the latest SHA from the main package: +The first step is to recursively clone the MXNet repository and checkout the desired version, (example 1.3.1). You should use the latest [version](https://search.maven.org/search?q=clojure-mxnet)), and clone into the `~/mxnet` directory: -`git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet` -`cd ~/mxnet` + ```bash + git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet + cd ~/mxnet + git checkout tags/version -b my_mxnet # this is optional + git submodule update --init --recursive + ``` -If you need to checkout a particular release you can do it with: +If you have previous builds and other unwanted files lying around in the working directory and would like to clean up, [here](https://gist.github.com/nicktoumpelis/11214362) is a useful script for that task. However, be aware that this recipe will remove any untracked files and reset uncommitted changes in the working directory. -`git checkout tags/1.3.0 -b release-1.3.0` +#### Building the core library -`git submodule update --init --recursive` +Detailed instructions for building MXNet core from source can be found [in the MXNet installation documentation](https://mxnet.incubator.apache.org/install/index.html). The relevant sections are: -Sometimes it useful to use this script to clean hard -https://gist.github.com/nicktoumpelis/11214362 +- For Ubuntu Linux: [CUDA Dependencies](https://mxnet.incubator.apache.org/install/ubuntu_setup.html#cuda-dependencies) and [Building MXNet from Source](https://mxnet.incubator.apache.org/install/ubuntu_setup.html#build-mxnet-from-source) +- For Mac OSX: [Build the Shared Library](https://mxnet.incubator.apache.org/install/osx_setup.html#build-the-shared-library) +In particular, ignore all of the language-interface-specific sections. -Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html +The outcome of this step will be a shared library `lib/libmxnet.so` that is used in the next step. - Run `make scalapkg` then `make scalainstall` +#### Building the Scala jar -then replace the correct jar for your architecture in the project.clj, example `[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0-SNAPSHOT"]` +- Ensure you have JDK 8 on your system. Later versions may produce cryptic build errors mentioning `scala.reflect.internal.MissingRequirementError`. +- Build and install the Scala package in your local Maven directory using the following commands: -#### Test your installation + ```bash + cd scala-package + mvn install + ``` -To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package. +#### Building the Clojure jar + +- Enter the `contrib/clojure` directory and edit the `project.clj` file. Add the Scala jar that was just created and installed, e.g., `[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "latest-version-SNAPSHOT"]`, to the `:dependencies`. +- Run `lein test`. All the tests should run without an error. +- Run `lein install` to build and install the Clojure jar locally. +To run examples, you can now use `lein run` in any of the example directories, e.g., `examples/imclassification`. You can also specify the compute device, e.g., `lein run :cpu 2` (for 2 CPUs) or `lein run :gpu` (for 1 GPU). -#### Generation of NDArray and Symbol apis +## Docker Files -The bulk of the ndarray and symbol apis are generated via java reflection into the Scala classes. The files are generated as a compile time step (AOT) in the `dev.generator` namespace. +There are Dockerfiles available as well. -You may also run this manually with the repl functions: +- [Community Provided by Magnet](https://hub.docker.com/u/magnetcoop/) +- [MXNet CI](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/Dockerfile.build.ubuntu_cpu) and the install scripts + - [Ubuntu core](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_core.sh) + - [Ubuntu Scala](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_scala.sh) + - [Ubuntu Clojure](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_clojure.sh) -`(generate-ndarray-file)` -and -`(generate-symbol-file)` +## Need Help? +If you are having trouble getting started or have a question, feel free to reach out at: -These will generate the files under `src/org.apache.clojure-mxnet/gen/` that are loaded by the `src/org.apache.clojure-mxnet/ndarray.clj` and `src/org.apache.clojure-mxnet/symbol.clj` files. +- Clojurian Slack #mxnet channel. To join, go to [http://clojurians.net/](http://clojurians.net/). +- Apache Slack #mxnet and #mxnet-scala channel. To join this slack send an email to dev@mxnet.apache.org. +- Create an Issue on [https://github.com/apache/incubator-mxnet/issues](https://github.com/apache/incubator-mxnet/issues). ## Examples @@ -160,11 +233,11 @@ There are quite a few examples in the examples directory. To use. There are README is every directory outlining instructions. A good place to get started is the module example. -Do `lein run` for the cpu version or `lein run :gpu` for gpu. +Do `lein run` for the CPU version or `lein run :gpu` for GPU. ## Generating documentation -To generate api docs, run `lein codox`. The html docs will be generated in the target/docs directory. +To generate API docs, run `lein codox`. The html docs will be generated in the target/docs directory. ## Code Coverage @@ -182,12 +255,11 @@ before the submit a new pull request so we can keep the style consistent through ## FAQ - **Why build on the Scala package?** The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done. -**How can I tell if the gpu is being used?** +**How can I tell if the GPU is being used?** CUDA is finding a best algorithm... As long as a Context.gpu() passed in the code as a context, GPU should be used. @@ -197,30 +269,17 @@ This command can be very handy too timestamp, name, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]` **Supported APIs** -There are 3 high level apis supported in MXNet: (Model/FeedForward), Module, and Gluon. The Module api is supported in the Clojure package because of the existing support for it in the Scala package. The Module api is very similar to the Gluon api and examples of the usage can be found in the examples directory. The Model/FeedForward Api is deprected. +There are 3 high level APIs supported in MXNet: (Model/FeedForward), Module, and Gluon. The Module API is supported in the Clojure package because of the existing support for it in the Scala package. The Module API is very similar to the Gluon API and examples of the usage can be found in the examples directory. The Model/FeedForward API is deprected. -Gluon support will come later and may or may not be built on the Scala gluon api (when it lands there) +Gluon support will come later and may or may not be built on the Scala gluon API (when it lands there) ## Architecture & Design See the Confluence page: https://cwiki.apache.org/confluence/display/MXNET/MXNet+Clojure ## Building and Deploying Jars -The process to build and deploy the jars currently is a manual process using the `lein` build tool and `Clojars`, the Clojure dependency hosting platform. - -There is one jar for every system supported. - -- Comment out the line in the `project.clj` for the system that you are targeting, (example OSX cpu you would uncomment out ` [org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` but leave the linux deps commented) -- Change the `defproject org.apache.mxnet.contrib.clojure/clojure-mxnet "0.1.1-SNAPSHOT"` in the project to reference the correct version number and jar description. For example changing the line to be `org.apache.mxnet.contrib.clojure/mxnet-osx-cpu "0.1.2"` would create a jar with the group id of `org.apache.mxnet.contrib.clojure` and the artifact name of `mxnet-osx-cpu` and the version of `0.1.2` -- Run `lein clean` -- Run `lein jar` to create the jar -- Check that the jar looks alright in the `/target` directory. - -To deploy the jar to Clojars, you do `lein deploy clojars` and it will prompt you for your username and password. - -_Note: Integration with deployment to Nexus can be enabled too for the future [https://help.sonatype.com/repomanager2/maven-and-other-build-tools/leiningen](https://help.sonatype.com/repomanager2/maven-and-other-build-tools/leiningen)_ -You would repeat this process for all the build system types. +The release process for deploying the Clojure jars is on the [Apache MXNet developer wiki](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Release+Process). ## Special Thanks diff --git a/contrib/clojure-package/examples/captcha/.gitignore b/contrib/clojure-package/examples/captcha/.gitignore new file mode 100644 index 000000000..e1569bd89 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/.gitignore @@ -0,0 +1,3 @@ +/.lein-* +/.nrepl-port +images/* diff --git a/contrib/clojure-package/examples/captcha/README.md b/contrib/clojure-package/examples/captcha/README.md new file mode 100644 index 000000000..6b593b2f1 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/README.md @@ -0,0 +1,61 @@ +# Captcha + +This is the clojure version of [captcha recognition](https://github.com/xlvector/learning-dl/tree/master/mxnet/ocr) +example by xlvector and mirrors the R captcha example. It can be used as an +example of multi-label training. For the following captcha example, we consider it as an +image with 4 labels and train a CNN over the data set. + +![captcha example](captcha_example.png) + +## Installation + +Before you run this example, make sure that you have the clojure package +installed. In the main clojure package directory, do `lein install`. +Then you can run `lein install` in this directory. + +## Usage + +### Training + +First the OCR model needs to be trained based on [labeled data](https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/data/captcha_example.zip). +The training can be started using the following: +``` +$ lein train [:cpu|:gpu] [num-devices] +``` +This downloads the training/evaluation data using the `get_data.sh` script +before starting training. + +It is possible that you will encounter some out-of-memory issues while training using :gpu on Ubuntu +linux (18.04). However, the command `lein train` (training on one CPU) may resolve the issue. + +The training runs for 10 iterations by default and saves the model with the +prefix `ocr-`. The model achieved an exact match accuracy of ~0.954 and +~0.628 on training and validation data respectively. + +### Inference + +Once the model has been saved, it can be used for prediction. This can be done +by running: +``` +$ lein infer +INFO MXNetJVM: Try loading mxnet-scala from native path. +INFO MXNetJVM: Try loading mxnet-scala-linux-x86_64-gpu from native path. +INFO MXNetJVM: Try loading mxnet-scala-linux-x86_64-cpu from native path. +WARN MXNetJVM: MXNet Scala native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH), or specifying by Java cmd option -Djava.library.path=[lib path]. +WARN org.apache.mxnet.DataDesc: Found Undefined Layout, will use default index 0 for batch axis +INFO org.apache.mxnet.infer.Predictor: Latency increased due to batchSize mismatch 8 vs 1 +WARN org.apache.mxnet.DataDesc: Found Undefined Layout, will use default index 0 for batch axis +WARN org.apache.mxnet.DataDesc: Found Undefined Layout, will use default index 0 for batch axis +CAPTCHA output: 6643 +INFO org.apache.mxnet.util.NativeLibraryLoader: Deleting /tmp/mxnet6045308279291774865/libmxnet.so +INFO org.apache.mxnet.util.NativeLibraryLoader: Deleting /tmp/mxnet6045308279291774865/mxnet-scala +INFO org.apache.mxnet.util.NativeLibraryLoader: Deleting /tmp/mxnet6045308279291774865 +``` +The model runs on `captcha_example.png` by default. + +It can be run on other generated captcha images as well. The script +`gen_captcha.py` generates random captcha images for length 4. +Before running the python script, you will need to install the [captcha](https://pypi.org/project/captcha/) +library using `pip3 install --user captcha`. The captcha images are generated +in the `images/` folder and we can run the prediction using +`lein infer images/7534.png`. diff --git a/contrib/clojure-package/examples/captcha/captcha_example.png b/contrib/clojure-package/examples/captcha/captcha_example.png new file mode 100644 index 0000000000000000000000000000000000000000..09b84f7190fab4cb391f8a3927b10f44d609aaaf GIT binary patch literal 9762 zcmV+-Cf(VIP)002A)0ssI2ZV4tr001TFNkl$qd)Kg=JPmwuTSN6f=S^pBb7Zny2R zWJ{E6%91IIGHG!GH-H2PR$;5Ed-E>2oOvh`g0KQSx5g`fB>Z>tnC%C^w(T2xTUdOfJL=Hzse)#S6+cuN-_SaD$Q` zA;ZuM0bqH|Pj)(6f!O+^Ns+YV-1}EPY%k5B%eph^&%>uOK4VP(7tT2VoNb-!|Myh$ z`NwHCQL>fJ{p!~H0eMN0Jmw&m5$EwAzdmzo@pjL~EMt|R{BIY29r|I7RgUjJb}PSq zs`)%XefdQ_sp)JH+aXxV4<|pFOz%jMG0v;DmM96qqx706+5ctj`jDAaN_RUQKlF1` zDAWg|{YtA-%@-O${jtPX%a3NxwDW~PyNQx<7MJ|eBe%AUyeMYm>|JVEpSUj)mT7BR zy}25zOioVMYxT3Ob3YqU#=(%y7D%LG5V>u@t4QfaQdLZd6hHWuam9cJ5E1i=jC9r?K{(1A6`V%3T0XBo?Z98iNYZw_IAmok{vkWcTr2SI;xkM%XX+};6 zQ1A2o9k2^qA-I>tiIOeZK7Hqmq>MWv5P_?mY8tudZnwXco6xGIofOIE7r(f1@8%M8 zzFN89SLrSi1Jwu9uJOIs6C&XLpBoX7W)_Qm8%x9D_vOcf?P3xMEKs)`=UuJms$P7T%uhhQl3 zxA%_XLiW;L%e0M;hzJC6p2T_bM6|RaW3<*hMd=uu<}&aGT?Q96W^JI7?c+_Ea)GM~ldjl!7U9Ct7}f>#r$gOfVt}dC>3&e>nPsA#WgbLxya-{jJ@Elv0V7Grid*?8G3^ zX8_0mz!Cre-v6h-LV?O`-n$;}9US}7!MC^GsR3iaXt3NOAp$~N#7U7n5iK2f{@$U{ zvEk8jwcP9VF45JcqQf)diGMu7aUQqgcAG7v zEC+xR4+q`qKOOp6Giq#Ide&m)CKP8$H||}YzH(k&PACV#34jb)WxxUw5P-E71#r(H z4ia)x%sm@E(~@(DxRw1Z$r4$}Bu&6tTUc}*1`Irg9Pclm$3x%+#}6D!lB80p=m9g< z9LrUn7x~20)ob}=SHw9gX>$-v40=m>oMcyF0Y{C6YSLppZ%u0};As zTjwT|J3rk0y+&An=wOmeelhW7O1aJyN{qOt2f&G3nWxE<*oIx#<(&7a3OFx^B>>2a zyj~kRQJiSY`7@J$>F{1^v)n1HASLufenVVl?>zUmBb-OD8MlQETQ7=Yp|{x6aW9RT z0zV5*jvgQM2A&kVN-3GknWfn&dAp4Z0vrG&Ly_JV-1h!$$SAF8S zp67Z4wl)wB40;1!-8}zQ{CCC4I$#{?X7OCUIMupci7Njvf|ESCH8a^=?1D#7K#9UK ziz4TV$!=cc*QRfLHhDG=3j}T%fgw`X%iK5jzY&o?GB{F=s{M_s6L*|FVo7yAxq2qc zBs84FrtN(3W@i!rA`zCn4F^UH3eQjmXb}urG6bA}dMfVbNt`9aPr_MR7mTq=P!_=V zSH~86i`R?E%+bh^mD!T*NtI@K=KKEB>K*_#B3Nd!-JAdV-lddhE;u3gcb9gJ?H+xC z%PrCJ=KQUeH=jbr1S3Nx*S>P=;o$z-ShLyy07p3eheIzswC7-})4t+eP17{V3JOqC z`r-7MJ?)$C?SFG~f|oh7N-%;6+R z6>6q4b66jG+P$xD24{Kp#l)9gkr*C z%}lY7d0M=B;8ZoLI>K3I7XTc`aU7>ss{ZigTf0g-!`QPu9j8evX}6O0=7D8o$T;)3 z8?x|F?Z7MJF9vP^0Nmrqo;thz!BmY6km}oyl<91#=w&8*dj0m}!Rr&8=9xA+ zqXGmg`{l!9hZ>dolQsD+EL}9r7;Y_qAnqotDTek`Muy9cjojGiU@a0MA);`E5W;bsb)S4R#Cb9r?d}zc1polo@&n%3 z3-xE;IrLV^+sxz%{jgE3|MaCF{p{Hv4T%BYdZ1|&HCw3S{Edr6QABk9)$k zxdlSx%nAa8M9Kco6P(~$1j0YH_jzBrXbBjQz=YB-C%$}21jB=f;rav{v>oi)KBrG% ziIcTo-9B&oxe9{FjfN}D@y7lg)#38PY}&{Vo3+NCfsq#~&(w=r#G{hCp*I}oi32Ro z6M@djBPT1RvL{@o0kj3gl(LqbySX&^o5>H_>D*K0*mr!L>xf582xN#Hkt6VJM~b$N zek6GPJ=HQ}1F)GzU|j!@DP+9Nim0Rgtf)WKIV}%e0#tw`P zjBQ;{L{8WY8~bWwFYJFV+SpUFU}w~MHyJTJPTt<7upF$QY4wkN_iCd(bG=9=c>3kdnMJ9lts$`u&lz+ z4>u=xqtxnF(0_Oxf|XXrYGX7YlCdf+))L&yllk8KpDuoWE54PnEJ>wj+}n$HmU^A< zz4*o^1Y2u+DoIS5P>u|3VN|C8&(6=!54crFIIimk-ui#OaxDuXs-m*_43kWLaqXPU zB{*mj-4AXP01=S_AYiU=bEh#-|M}>9Pc{8ss3J~dD-FvKLm&b&CU*;CjJ1}r^%eEn z)aoNh^ZF9J*j+--0D-JU!PXKS7s+q0pPpX0-Gwd%Kt8D=)v3OiTpgPqW5^;e^u2Xl z_@qb{pX!qh%&HAsD`$T1~W+ zN^V!0*J-*aiYSPRq5uE{Mr+D85|y-Mdp2uzBPnXG@P!k(PQ@*1HZ7^&Uq1cim2)yr z*S2h}?G?!vv*!|%A|hE*z1*`-Ev)|e?%B^5zC7Q#^skqHbA9pFY_ApP@roYrujN)Y zw>;;Ef`F7T#DO70iwqb5l)UJTgRhjG5)lCcAboZBD>8PAZLUS4y2hmkDPFoC}(-r&3Y|0pYj}C#vMRJhxlH9 zcQveRBEBnAs{ruv^qKxI+cFn6k4zx~F^BaTmYHmI+rOLmXpRks7Gs87w3;|M`b@N`F$G^J% zyMH?L)8+%!x7}K`yvQe}uIE@Vz<~MB?R_?I?r#|c1jg80gLRQ{K)#W?_)x?}(l6GO zHv84h_xE_a-rW6~b`#aku>$-bJM$b=PfsFc&TAvS@$xkl)>1KQ@3l;YS5P9U1w%oWjH&&zZh&Z6n7fttr z$&ZiDAN<~t*Om#ETFzw)bNNDQ@5kpDXN1TZGljO9u>!d(fT0Lo4Ypm({p#v(-`V|x zX4EKoQ6t!1ty-KViIUP}$gv7#(6l^~k{?BGRCdY*8Q@H~&dQkO^473<$mRaSMe^zG zzruAqS?vcO?4QK+K?#z66 z>+KHpG`OnQToBm4^nL+7>5S6h)y6$uj~7o#5EMqv5v`7zl`fNB{`Zq^)A^ zR(I-RaAS*J9wkP9`>grqv)~|Wxv2bKZwYi-3VT{#l z_3t14UVPz$dwPM4Mb3aBJYblJv@%2%fdN1yfPPODkzUwb5E6;H+-npD+ zSz0LT0JxZFdA@ULpWGL@;TCmToX6K@uE#22jD;eI+#vY2Q(p4IDvu%_Bql?SnaDqy z{ggW>b&3wpZIP)=X7ZtN#K;Mo!O(v_{PR1VnUAK=WO=4hBS+u>0E{!n84#JmWJ@_{ zs8!1^H=ZAk8e7r-+r_#y=Nl626Fcqigb}3>5cx zfx1K8OVbmzpgOn(f@LP#owv-Oi#}&_U0w#ZpCAWT3Nx+F+6fZD1ZV2jpM<-gFE&= zy^z^jXL7?m7%Tx15g4$AE-u}=EGcujvojn%)EIA+>TB)$&zC2@0I+B1AFJ$xAL#@ySANKWjV5gPj!w=^ zn!*?jh8r#zN%B4Kh%>%#@7``N-Z9u5EY(V0>FL#Bt#zf=cfvw16rSTVpDdApdvw66CS`a$G{p$O1`Eli$fd79-#u9Q-R)?kq#X^Jp-dFZ+RX^wRf z_(8o=fA5)hMsdWk4geISG#Ld&|AD?P+9=1Ddd>kXC%-aPcZXDi2o*dBkdxzbTypWw(3e=z<=iARp*hFM*rJNZnP$9~{NLAc4uk-!UF$8oJkhL&23 zS8iR6vxFhz$T{Oc0GfiH8)3MSn(F7?kz=#m*i{?xnNQYoh9VNk7;GVIiO766;DI0j zpL@G%JJ(W`s6>*KB$rF~$&%GJU&;aujyeYq?K~K6Q;|k~X?)j#llx9Y7#5w(0-*zQ ze6f>uGM!qo+j5E?4UrS}f0jR4pWu>T8t`jJ!UMuGL{O-Ekt`eq0#+5stSW~H{ouLr zlfLhfwh74_y~#L_3sW$c<+dOqofM#8ZG3yVwPn#X%Y(2)b3Xvbad*aPgLobwB zi4!YObxm9us6D;XoOSV#nk)ct9hVV0nZqpO4Bf~B&ZJ3CPoL@2xsj^mzy)xD0syL5 z0lG6{PB|jRjG*65j+X+o>7&2zL{o?w$ zZuYe5>(-)A&FPslu!#h_t~%?lY8 zz$g^Kf#LCGqoyyrH+xepIiF|=09IS8jVd${0Yhs{$bvVAUtg&h^aE`D{W^{_5LE}f zYDhj>N}-sH?|gjWPjUXl?v2<8{W;$1P4%hSxMT;w_xyTgs4`M5^*TM0000Cb4^B0n zpUdYjE?fbCz2!Y;Tj$;!c_Uo0u`^<+%@CL+BQ#q|md;K7T{4LrK|_$bz4@-!Scl+M z@$@81WG<6D?c2DlFUt>%qKyucCz(u(G}9>)qU4rF>N`U(Oq5KtY^QU7X?@vB`|KhS zIhyIj2v-;m$bc+~lm|RmsrCz{?zZk$E7c$fwtg--@Pgy}kKOqDREHhauWK>7op^ajj(>aex%>l+19ze<G&!_TjtjZLqg6lcix z500%ou)0d0_l3Bpdl?l53^UG)A+?#J*ynX4_XI|L`!F~|n02vTPUf4_HXhkW{N~azG`Z3rcHZX@vh9l3YTnPn= zzO3Ku-EL*`{qj(2tSE3OEgz~LaD|t4Enz!@92rYZ{`;>_mrq6r&B%`cAlJoWy10-o zCOT=rHZod-gb{-TkRx*hA~Jdc2PsyP3LEgcQcfNcQaROp{$AX2m6P&<2^K4PW9C+4 z-;nFN_cy`-5+zSLVZY1IXaCxY+nHaWLX#UO5I5#-TCj{G5Fmmh`o~bV;DIpUWey1- z_6tVv$lgQC0s2+d4t;ekGkIQqt-%ly1MFnob~fL}#naPg5|sb|AQF*FBFx;8VCQoO zPI{hS?=}AK)Q73bkh4Tf8>0_GbqpU$0Kw4Fq!iIRXwnp_lBEIrpe_Uzsh1EpH%hp~x6 z>&+8^02Hiela7j$JOO~BD6ULSEYM=E_50U9`oZw)M|K~qiP}SnQAKg}=GDTe{tG4s zxufi+X9(8i6247mAAM3ce|yV#$YP9~u*{=s5iQaZ39Hb?lwQ6wF&GSZYvM8&dTAo*D270cGkCz5dgq~DJ;>3A?Uuk zK0bW-FWoPC5EB>+RBH}hN+-HD1^^;(EO(6vrF(Mt=(GDyG|P>?B8l8EWMN2QYBJ;q z9EshhA}?}nRi<-kbCs&I_rBaSJW_+&<}a-%ipAbiUKE+l``!m3k|n8SE1g?Q;WE~8 z2`lLAV=JVt#6wQ_0{}7{5(BeIJ9iX$nC4ll)9Um(eT!qIvW@d(cCi&}i9UursPfuK zurto%N**ooCG7z9s0_igW5;XdZ>-66XBFphOU_B1-p@B8NS(H``INGY3s;T}9$p*D zey#CRYoRTW0|zoxxcYqOBJ(*|bVxW>_?dIS866xN7_KyG!8hdtC9m}Q{+Ivl>)%K> zMUK^Swd|Dob-P5Mj1r#1gwYPgES>GNxMG1HF5jZER`Lo}T%Nu>nckrg5bsx?8D*r+ zHZO==#U*T?&b8S10QiH{@$u@BV|$M-UFzIY(@NNa)Iza%r!#YRes=G!v0AC>dw!fH ziArX=v!C7js|pPo^3Zwlz{$W1CTAy!A#es-A&Dvu?C^$nRGQ1ZEbredG23fhnVE>0 z0OMhQ+7XKHK3Bhh8wg_kfq* zuM1eUMwfbvEj_p7cKZ$mSX*+!s2WWrcTVlxFog2>T&|1t30~=C0$hH5@YLbqgWZL= z$Ft02ZNBikn;#BAZF+I~mBXjXL3uG<`smspXP}k(d9IZJrvy>es{nvs4ZBMT5uiqf zhzwVnlFR$|D(FH-PPCx9{X1EFO`+!X!&7168gZ+OP zq95>@?Y7Qb`?KMOTriH+d6CJygi)h8G!WJrL4BFFG)v<=Y30^xV+FDP>eTL_uC5*P zcY_THUX?l~qNSqa-SL2kxq>(uV>(4VBk!e4#f9Wj9%{z9wG=341e+)KpD20J?BYF{ zr4l3r0Gxx5UFvDvFzlBXd6s3jJJWw^eO3sC2EkBZyj?-Ft>^2dApm%5*Bfg$VAOE= z#udQ@LDBbxNLyC$dR{qL-9K8{v$s4Nh9LkXN?u&J(%0}S(-m)W;0G97$NtA?i z?riJa9)DMrmvg0*uo^WF9d5xX+ECTjtL7?j(gwg8_yS_=8k==h$0= zQO$F`&|EKX>^shrB$u&GAYuSO1Ugkt4{A~6nbBj-q2@|saNP>>Rd?;I+@#_>{?U+N{+ksf+_CF>+c&Yqscdr_gEh3i*0SFF=o&Ju&VBm>d zeEJC3P=eRK#F7{N_{b0c_2O^l^g>E0bAlt$oYXUfuu_S3n)Z7s@2WkU*YoqH9 z3;--p28Gs|k%NFPi8A*GqRNqxLqh{YQS^=0-FgN2@+S56p6^$9`TNJ-P?z+Sxsy_n znjASMN|Lp|k3TNDg(->|eHSy9c&RA6=x_iaZ2}&Qly~+|&HO{KjpCrPUqZxp_rLjn zzWVo$D%=`lf&*trK!(VehzHGJXt>l^KBXV|Vc_~6IiwA`0ubXoiFtCb+u{xffM`WE ztIXu*SI*AiqK+&wgMg^9B!ZHNBL5+mw1$4@yIuf6rixIM2HeW=%Atch4h)2~wP#tk zs4Z6sy2st0oOt)f;;j!RPbVs7E=SH&lO@(_RF>(?DsquIoH-m2p|HO726*+*?(y(H z1;GGN@}l9WF%s@9x=IzsY9ei25WetSatHm|+k3vh>h+N!YlzJKq^m-6CBEKRs$ZGD z{PLlfBR^6~C7G1DoLRivmh+OO=n7;=1U_?{eCUNkC)YiR`t5;*uOE12>B9hy9(s<-C+qmfB1jiHwQR0dN3-EmsO_VmVfH_|C~U z8gG$hJ>IkSpY-8 zF*o^QrhjwccO#V@I|g@UCRbWto4GMLKh@2;OF_(pAa?&9g~oes&xxV(Q1DQK{nAh{ zu*(@PL%4frWTdjQ7S`52wyguy>poL*qxTNH)wlEiaPu^26YC5aLym|D7D!u56j;Bi zqmzweJIceAp#0Fjo*uSOuwQbR^Sz_5eR=b%13UKzUg&}d{2=n8wcBc5XKckA` z@Sa2K-VyLP*CBi}efRz~#&MDS)4`v^9w9@*@A!`9FVo?{CvEi;=irKVpe25s}c$n`9d_#f?c^`^*Y|Gm9i?g7YPSalYU^GE^EAHhq<^W()Z(2lZ+tZg(_t2% zk3r$p+{#f$y_L~Ntiay7)o^HRe4=x$Qk3E)X-dQc2L%uSS!^hD)HL8tAHtrrD zs|y#ugKs`7iXzUFd%f0gu70o}Iw@p8E6S@0?pHM`!Z}lg^1-3;AHMKry)yJA?1A8zY{aSf5s{jB107*qoM6N<$f)Wn*ga7~l literal 0 HcmV?d00001 diff --git a/contrib/clojure-package/examples/captcha/gen_captcha.py b/contrib/clojure-package/examples/captcha/gen_captcha.py new file mode 100755 index 000000000..43e0d26fb --- /dev/null +++ b/contrib/clojure-package/examples/captcha/gen_captcha.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from captcha.image import ImageCaptcha +import os +import random + +length = 4 +width = 160 +height = 60 +IMAGE_DIR = "images" + + +def random_text(): + return ''.join(str(random.randint(0, 9)) + for _ in range(length)) + + +if __name__ == '__main__': + image = ImageCaptcha(width=width, height=height) + captcha_text = random_text() + if not os.path.exists(IMAGE_DIR): + os.makedirs(IMAGE_DIR) + image.write(captcha_text, os.path.join(IMAGE_DIR, captcha_text + ".png")) diff --git a/contrib/clojure-package/examples/captcha/get_data.sh b/contrib/clojure-package/examples/captcha/get_data.sh new file mode 100755 index 000000000..baa7f9eb8 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/get_data.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -evx + +EXAMPLE_ROOT=$(cd "$(dirname $0)"; pwd) + +data_path=$EXAMPLE_ROOT + +if [ ! -f "$data_path/captcha_example.zip" ]; then + wget https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/R/data/captcha_example.zip -P $data_path +fi + +if [ ! -f "$data_path/captcha_example/captcha_train.rec" ]; then + unzip $data_path/captcha_example.zip -d $data_path +fi diff --git a/contrib/clojure-package/examples/captcha/project.clj b/contrib/clojure-package/examples/captcha/project.clj new file mode 100644 index 000000000..fa37fecbe --- /dev/null +++ b/contrib/clojure-package/examples/captcha/project.clj @@ -0,0 +1,28 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(defproject captcha "0.1.0-SNAPSHOT" + :description "Captcha recognition via multi-label classification" + :plugins [[lein-cljfmt "0.5.7"]] + :dependencies [[org.clojure/clojure "1.9.0"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] + :main ^:skip-aot captcha.train-ocr + :profiles {:train {:main captcha.train-ocr} + :infer {:main captcha.infer-ocr} + :uberjar {:aot :all}} + :aliases {"train" ["with-profile" "train" "run"] + "infer" ["with-profile" "infer" "run"]}) diff --git a/contrib/clojure-package/examples/captcha/src/captcha/consts.clj b/contrib/clojure-package/examples/captcha/src/captcha/consts.clj new file mode 100644 index 000000000..318e0d806 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/src/captcha/consts.clj @@ -0,0 +1,27 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns captcha.consts) + +(def batch-size 8) +(def channels 3) +(def height 30) +(def width 80) +(def data-shape [channels height width]) +(def num-labels 10) +(def label-width 4) +(def model-prefix "ocr") diff --git a/contrib/clojure-package/examples/captcha/src/captcha/infer_ocr.clj b/contrib/clojure-package/examples/captcha/src/captcha/infer_ocr.clj new file mode 100644 index 000000000..f6a648e98 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/src/captcha/infer_ocr.clj @@ -0,0 +1,56 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns captcha.infer-ocr + (:require [captcha.consts :refer :all] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [org.apache.clojure-mxnet.ndarray :as ndarray])) + +(defn create-predictor + [] + (let [data-desc {:name "data" + :shape [batch-size channels height width] + :layout layout/NCHW + :dtype dtype/FLOAT32} + label-desc {:name "label" + :shape [batch-size label-width] + :layout layout/NT + :dtype dtype/FLOAT32} + factory (infer/model-factory model-prefix + [data-desc label-desc])] + (infer/create-predictor factory))) + +(defn -main + [& args] + (let [[filename] args + image-fname (or filename "captcha_example.png") + image-ndarray (-> image-fname + infer/load-image-from-file + (infer/reshape-image width height) + (infer/buffered-image-to-pixels [channels height width]) + (ndarray/expand-dims 0)) + label-ndarray (ndarray/zeros [1 label-width]) + predictor (create-predictor) + predictions (-> (infer/predict-with-ndarray + predictor + [image-ndarray label-ndarray]) + first + (ndarray/argmax 1) + ndarray/->vec)] + (println "CAPTCHA output:" (apply str (mapv int predictions))))) diff --git a/contrib/clojure-package/examples/captcha/src/captcha/train_ocr.clj b/contrib/clojure-package/examples/captcha/src/captcha/train_ocr.clj new file mode 100644 index 000000000..91ec2fff3 --- /dev/null +++ b/contrib/clojure-package/examples/captcha/src/captcha/train_ocr.clj @@ -0,0 +1,156 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns captcha.train-ocr + (:require [captcha.consts :refer :all] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [org.apache.clojure-mxnet.callback :as callback] + [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.eval-metric :as eval-metric] + [org.apache.clojure-mxnet.initializer :as initializer] + [org.apache.clojure-mxnet.io :as mx-io] + [org.apache.clojure-mxnet.module :as m] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.optimizer :as optimizer] + [org.apache.clojure-mxnet.symbol :as sym]) + (:gen-class)) + +(when-not (.exists (io/file "captcha_example/captcha_train.lst")) + (sh "./get_data.sh")) + +(defonce train-data + (mx-io/image-record-iter {:path-imgrec "captcha_example/captcha_train.rec" + :path-imglist "captcha_example/captcha_train.lst" + :batch-size batch-size + :label-width label-width + :data-shape data-shape + :shuffle true + :seed 42})) + +(defonce eval-data + (mx-io/image-record-iter {:path-imgrec "captcha_example/captcha_test.rec" + :path-imglist "captcha_example/captcha_test.lst" + :batch-size batch-size + :label-width label-width + :data-shape data-shape})) + +(defn accuracy + [label pred & {:keys [by-character] + :or {by-character false} :as opts}] + (let [[nr nc] (ndarray/shape-vec label) + pred-context (ndarray/context pred) + label-t (-> label + ndarray/transpose + (ndarray/reshape [-1]) + (ndarray/as-in-context pred-context)) + pred-label (ndarray/argmax pred 1) + matches (ndarray/equal label-t pred-label) + [digit-matches] (-> matches + ndarray/sum + ndarray/->vec) + [complete-matches] (-> matches + (ndarray/reshape [nc nr]) + (ndarray/sum 0) + (ndarray/equal label-width) + ndarray/sum + ndarray/->vec)] + (if by-character + (float (/ digit-matches nr nc)) + (float (/ complete-matches nr))))) + +(defn get-data-symbol + [] + (let [data (sym/variable "data") + ;; normalize the input pixels + scaled (sym/div (sym/- data 127) 128) + + conv1 (sym/convolution {:data scaled :kernel [5 5] :num-filter 32}) + pool1 (sym/pooling {:data conv1 :pool-type "max" :kernel [2 2] :stride [1 1]}) + relu1 (sym/activation {:data pool1 :act-type "relu"}) + + conv2 (sym/convolution {:data relu1 :kernel [5 5] :num-filter 32}) + pool2 (sym/pooling {:data conv2 :pool-type "avg" :kernel [2 2] :stride [1 1]}) + relu2 (sym/activation {:data pool2 :act-type "relu"}) + + conv3 (sym/convolution {:data relu2 :kernel [3 3] :num-filter 32}) + pool3 (sym/pooling {:data conv3 :pool-type "avg" :kernel [2 2] :stride [1 1]}) + relu3 (sym/activation {:data pool3 :act-type "relu"}) + + conv4 (sym/convolution {:data relu3 :kernel [3 3] :num-filter 32}) + pool4 (sym/pooling {:data conv4 :pool-type "avg" :kernel [2 2] :stride [1 1]}) + relu4 (sym/activation {:data pool4 :act-type "relu"}) + + flattened (sym/flatten {:data relu4}) + fc1 (sym/fully-connected {:data flattened :num-hidden 256}) + fc21 (sym/fully-connected {:data fc1 :num-hidden num-labels}) + fc22 (sym/fully-connected {:data fc1 :num-hidden num-labels}) + fc23 (sym/fully-connected {:data fc1 :num-hidden num-labels}) + fc24 (sym/fully-connected {:data fc1 :num-hidden num-labels})] + (sym/concat "concat" nil [fc21 fc22 fc23 fc24] {:dim 0}))) + +(defn get-label-symbol + [] + (as-> (sym/variable "label") label + (sym/transpose {:data label}) + (sym/reshape {:data label :shape [-1]}))) + +(defn create-captcha-net + [] + (let [scores (get-data-symbol) + labels (get-label-symbol)] + (sym/softmax-output {:data scores :label labels}))) + +(def optimizer + (optimizer/adam + {:learning-rate 0.0002 + :wd 0.00001 + :clip-gradient 10})) + +(defn train-ocr + [devs] + (println "Starting the captcha training ...") + (let [model (m/module + (create-captcha-net) + {:data-names ["data"] :label-names ["label"] + :contexts devs})] + (m/fit model {:train-data train-data + :eval-data eval-data + :num-epoch 10 + :fit-params (m/fit-params + {:kvstore "local" + :batch-end-callback + (callback/speedometer batch-size 100) + :initializer + (initializer/xavier {:factor-type "in" + :magnitude 2.34}) + :optimizer optimizer + :eval-metric (eval-metric/custom-metric + #(accuracy %1 %2) + "accuracy")})}) + (println "Finished the fit") + model)) + +(defn -main + [& args] + (let [[dev dev-num] args + num-devices (Integer/parseInt (or dev-num "1")) + devs (if (= dev ":gpu") + (mapv #(context/gpu %) (range num-devices)) + (mapv #(context/cpu %) (range num-devices))) + model (train-ocr devs)] + (m/save-checkpoint model {:prefix model-prefix :epoch 0}))) diff --git a/contrib/clojure-package/examples/captcha/test/captcha/train_ocr_test.clj b/contrib/clojure-package/examples/captcha/test/captcha/train_ocr_test.clj new file mode 100644 index 000000000..ab785f7fe --- /dev/null +++ b/contrib/clojure-package/examples/captcha/test/captcha/train_ocr_test.clj @@ -0,0 +1,119 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns captcha.train-ocr-test + (:require [clojure.test :refer :all] + [captcha.consts :refer :all] + [captcha.train-ocr :refer :all] + [org.apache.clojure-mxnet.io :as mx-io] + [org.apache.clojure-mxnet.module :as m] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.shape :as shape] + [org.apache.clojure-mxnet.util :as util])) + +(deftest test-consts + (is (= 8 batch-size)) + (is (= [3 30 80] data-shape)) + (is (= 4 label-width)) + (is (= 10 num-labels))) + +(deftest test-labeled-data + (let [train-batch (mx-io/next train-data) + eval-batch (mx-io/next eval-data) + allowed-labels (into #{} (map float (range 10)))] + (is (= 8 (-> train-batch mx-io/batch-index count))) + (is (= 8 (-> eval-batch mx-io/batch-index count))) + (is (= [8 3 30 80] (-> train-batch + mx-io/batch-data + first + ndarray/shape-vec))) + (is (= [8 3 30 80] (-> eval-batch + mx-io/batch-data + first + ndarray/shape-vec))) + (is (every? #(<= 0 % 255) (-> train-batch + mx-io/batch-data + first + ndarray/->vec))) + (is (every? #(<= 0 % 255) (-> eval-batch + mx-io/batch-data + first + ndarray/->vec))) + (is (= [8 4] (-> train-batch + mx-io/batch-label + first + ndarray/shape-vec))) + (is (= [8 4] (-> eval-batch + mx-io/batch-label + first + ndarray/shape-vec))) + (is (every? allowed-labels (-> train-batch + mx-io/batch-label + first + ndarray/->vec))) + (is (every? allowed-labels (-> eval-batch + mx-io/batch-label + first + ndarray/->vec))))) + +(deftest test-model + (let [batch (mx-io/next train-data) + model (m/module (create-captcha-net) + {:data-names ["data"] :label-names ["label"]}) + _ (m/bind model + {:data-shapes (mx-io/provide-data-desc train-data) + :label-shapes (mx-io/provide-label-desc train-data)}) + _ (m/init-params model) + _ (m/forward-backward model batch) + output-shapes (-> model + m/output-shapes + util/coerce-return-recursive) + outputs (-> model + m/outputs-merged + first) + grads (->> model m/grad-arrays (map first))] + (is (= [["softmaxoutput0_output" (shape/->shape [8 10])]] + output-shapes)) + (is (= [32 10] (-> outputs ndarray/shape-vec))) + (is (every? #(<= 0.0 % 1.0) (-> outputs ndarray/->vec))) + (is (= [[32 3 5 5] [32] ; convolution1 weights+bias + [32 32 5 5] [32] ; convolution2 weights+bias + [32 32 3 3] [32] ; convolution3 weights+bias + [32 32 3 3] [32] ; convolution4 weights+bias + [256 28672] [256] ; fully-connected1 weights+bias + [10 256] [10] ; 1st label scores + [10 256] [10] ; 2nd label scores + [10 256] [10] ; 3rd label scores + [10 256] [10]] ; 4th label scores + (map ndarray/shape-vec grads))))) + +(deftest test-accuracy + (let [labels (ndarray/array [1 2 3 4, + 5 6 7 8] + [2 4]) + pred-labels (ndarray/array [1 0, + 2 6, + 3 0, + 4 8] + [8]) + preds (ndarray/one-hot pred-labels 10)] + (is (float? (accuracy labels preds))) + (is (float? (accuracy labels preds :by-character false))) + (is (float? (accuracy labels preds :by-character true))) + (is (= 0.5 (accuracy labels preds))) + (is (= 0.5 (accuracy labels preds :by-character false))) + (is (= 0.75 (accuracy labels preds :by-character true))))) diff --git a/contrib/clojure-package/examples/cnn-text-classification/README.md b/contrib/clojure-package/examples/cnn-text-classification/README.md index 86a8abb06..19bb91373 100644 --- a/contrib/clojure-package/examples/cnn-text-classification/README.md +++ b/contrib/clojure-package/examples/cnn-text-classification/README.md @@ -3,19 +3,19 @@ An example of text classification using CNN To use you must download the MR polarity dataset and put it in the path specified in the mr-dataset-path -The dataset can be obtained here: [https://github.com/yoonkim/CNN_sentence](https://github.com/yoonkim/CNN_sentence). The two files `rt-polarity.neg` +The dataset can be obtained here: [CNN_sentence](https://github.com/yoonkim/CNN_sentence). The two files `rt-polarity.neg` and `rt-polarity.pos` must be put in a directory. For example, `data/mr-data/rt-polarity.neg`. You also must download the glove word embeddings. The suggested one to use is the smaller 50 dimension one -`glove.6B.50d.txt` which is contained in the download file here [https://nlp.stanford.edu/projects/glove/](https://nlp.stanford.edu/projects/glove/) +`glove.6B.50d.txt` which is contained in the download file here: [GloVe](https://nlp.stanford.edu/projects/glove/) ## Usage You can run through the repl with -`(train-convnet {:embedding-size 50 :batch-size 100 :test-size 100 :num-epoch 10 :max-examples 1000})` +`(train-convnet {:embedding-size 50 :batch-size 100 :test-size 100 :num-epoch 10 :max-examples 1000 :pretrained-embedding :glove})` or -`JVM_OPTS="Xmx1g" lein run` (cpu) +`JVM_OPTS="-Xmx1g" lein run` (cpu) You can control the devices you run on by doing: @@ -24,10 +24,36 @@ You can control the devices you run on by doing: `lein run :gpu 2` - This will run on 2 gpu devices -The max-examples only loads 1000 each of the dataset to keep the time and memory down. To run all the examples, -change the main to be (train-convnet {:embedding-size 50 :batch-size 100 :test-size 1000 :num-epoch 10) +The max-examples only loads 1000 each of the dataset to keep the time and memory down. To run all the examples, +change the main to be (train-convnet {:embedding-size 50 :batch-size 100 :test-size 1000 :num-epoch 10 :pretrained-embedding :glove}) and then run - `lein uberjar` - `java -Xms1024m -Xmx2048m -jar target/cnn-text-classification-0.1.0-SNAPSHOT-standalone.jar` + +## Usage with word2vec + +You can also use word2vec embeddings in order to train the text classification model. +Before training, you will need to download [GoogleNews-vectors-negative300.bin](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing) first. +Once you've downloaded the embeddings (which are in a gzipped format), +you'll need to unzip them and place them in the `contrib/clojure-package/data` directory. + +Then you can run training on a subset of examples through the repl using: +``` +(train-convnet {:embedding-size 300 :batch-size 100 :test-size 100 :num-epoch 10 :max-examples 1000 :pretrained-embedding :word2vec}) +``` +Note that loading word2vec embeddings consumes memory and takes some time. + +You can also train them using `JVM_OPTS="-Xmx8g" lein run` once you've modified +the parameters to `train-convnet` (see above) in `src/cnn_text_classification/classifier.clj`. +In order to run training with word2vec on the complete data set, you will need to run: +``` +(train-convnet {:embedding-size 300 :batch-size 100 :test-size 1000 :num-epoch 10 :pretrained-embedding :word2vec}) +``` +You should be able to achieve an accuracy of `~0.78` using the parameters above. + +## Usage with learned embeddings + +Lastly, similar to the python CNN text classification example, you can learn the embeddings based on training data. +This can be achieved by setting `:pretrained-embedding nil` (or omitting that parameter altogether). diff --git a/contrib/clojure-package/examples/cnn-text-classification/project.clj b/contrib/clojure-package/examples/cnn-text-classification/project.clj index f44144942..29ebefe5d 100644 --- a/contrib/clojure-package/examples/cnn-text-classification/project.clj +++ b/contrib/clojure-package/examples/cnn-text-classification/project.clj @@ -19,6 +19,6 @@ :description "CNN text classification with MXNet" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :pedantic? :skip :main cnn-text-classification.classifier) diff --git a/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj b/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj index 29ff36fe1..3c0288c9c 100644 --- a/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj +++ b/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/classifier.clj @@ -16,7 +16,9 @@ ;; (ns cnn-text-classification.classifier - (:require [cnn-text-classification.data-helper :as data-helper] + (:require [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [cnn-text-classification.data-helper :as data-helper] [org.apache.clojure-mxnet.eval-metric :as eval-metric] [org.apache.clojure-mxnet.io :as mx-io] [org.apache.clojure-mxnet.module :as m] @@ -26,30 +28,50 @@ [org.apache.clojure-mxnet.context :as context]) (:gen-class)) +(def data-dir "data/") (def mr-dataset-path "data/mr-data") ;; the MR polarity dataset path -(def glove-file-path "data/glove/glove.6B.50d.txt") (def num-filter 100) (def num-label 2) (def dropout 0.5) -(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size embedding-size]}] +(when-not (.exists (io/file (str data-dir))) + (do (println "Retrieving data for cnn text classification...") (sh "./get_data.sh"))) + +(defn shuffle-data [test-num {:keys [data label sentence-count sentence-size vocab-size embedding-size pretrained-embedding]}] (println "Shuffling the data and splitting into training and test sets") (println {:sentence-count sentence-count :sentence-size sentence-size - :embedding-size embedding-size}) + :vocab-size vocab-size + :embedding-size embedding-size + :pretrained-embedding pretrained-embedding}) (let [shuffled (shuffle (map #(vector %1 %2) data label)) train-num (- (count shuffled) test-num) training (into [] (take train-num shuffled)) - test (into [] (drop train-num shuffled))] + test (into [] (drop train-num shuffled)) + ;; has to be channel x y + train-data-shape (if pretrained-embedding + [train-num 1 sentence-size embedding-size] + [train-num 1 sentence-size]) + ;; has to be channel x y + test-data-shape (if pretrained-embedding + [test-num 1 sentence-size embedding-size] + [test-num 1 sentence-size])] {:training {:data (ndarray/array (into [] (flatten (mapv first training))) - [train-num 1 sentence-size embedding-size]) ;; has to be channel x y + train-data-shape) :label (ndarray/array (into [] (flatten (mapv last training))) [train-num])} :test {:data (ndarray/array (into [] (flatten (mapv first test))) - [test-num 1 sentence-size embedding-size]) ;; has to be channel x y + test-data-shape) :label (ndarray/array (into [] (flatten (mapv last test))) [test-num])}})) +(defn get-data-symbol [num-embed sentence-size batch-size vocab-size pretrained-embedding] + (if pretrained-embedding + (sym/variable "data") + (as-> (sym/variable "data") data + (sym/embedding "vocab_embed" {:data data :input-dim vocab-size :output-dim num-embed}) + (sym/reshape {:data data :target-shape [batch-size 1 sentence-size num-embed]})))) + (defn make-filter-layers [{:keys [input-x num-embed sentence-size] :as config} filter-size] (as-> (sym/convolution {:data input-x @@ -63,9 +85,9 @@ ;;; convnet with multiple filter sizes ;; from Convolutional Neural Networks for Sentence Classification by Yoon Kim -(defn get-multi-filter-convnet [num-embed sentence-size batch-size] +(defn get-multi-filter-convnet [num-embed sentence-size batch-size vocab-size pretrained-embedding] (let [filter-list [3 4 5] - input-x (sym/variable "data") + input-x (get-data-symbol num-embed sentence-size batch-size vocab-size pretrained-embedding) polled-outputs (mapv #(make-filter-layers {:input-x input-x :num-embed num-embed :sentence-size sentence-size} %) filter-list) total-filters (* num-filter (count filter-list)) concat (sym/concat "concat" nil polled-outputs {:dim 1}) @@ -74,10 +96,11 @@ fc (sym/fully-connected "fc1" {:data hdrop :num-hidden num-label})] (sym/softmax-output "softmax" {:data fc}))) -(defn train-convnet [{:keys [devs embedding-size batch-size test-size num-epoch max-examples]}] - (let [glove (data-helper/load-glove glove-file-path) ;; you can also use word2vec - ms-dataset (data-helper/load-ms-with-embeddings mr-dataset-path embedding-size glove max-examples) +(defn train-convnet [{:keys [devs embedding-size batch-size test-size + num-epoch max-examples pretrained-embedding]}] + (let [ms-dataset (data-helper/load-ms-with-embeddings mr-dataset-path max-examples embedding-size {:pretrained-embedding pretrained-embedding}) sentence-size (:sentence-size ms-dataset) + vocab-size (:vocab-size ms-dataset) shuffled (shuffle-data test-size ms-dataset) train-data (mx-io/ndarray-iter [(get-in shuffled [:training :data])] {:label [(get-in shuffled [:training :label])] @@ -89,7 +112,7 @@ :label-name "softmax_label" :data-batch-size batch-size :last-batch-handle "pad"})] - (let [mod (m/module (get-multi-filter-convnet embedding-size sentence-size batch-size) {:contexts devs})] + (let [mod (m/module (get-multi-filter-convnet embedding-size sentence-size batch-size vocab-size pretrained-embedding) {:contexts devs})] (println "Getting ready to train for " num-epoch " epochs") (println "===========") (m/fit mod {:train-data train-data :eval-data test-data :num-epoch num-epoch @@ -103,10 +126,10 @@ ;;; omit max-examples if you want to run all the examples in the movie review dataset ;; to limit mem consumption set to something like 1000 and adjust test size to 100 (println "Running with context devices of" devs) - (train-convnet {:devs [(context/cpu)] :embedding-size 50 :batch-size 10 :test-size 100 :num-epoch 10 :max-examples 1000}) + (train-convnet {:devs devs :embedding-size 50 :batch-size 10 :test-size 100 :num-epoch 10 :max-examples 1000 :pretrained-embedding :glove}) ;; runs all the examples #_(train-convnet {:embedding-size 50 :batch-size 100 :test-size 1000 :num-epoch 10}))) (comment - (train-convnet {:devs [(context/cpu)] :embedding-size 50 :batch-size 10 :test-size 100 :num-epoch 10 :max-examples 1000})) + (train-convnet {:devs devs :embedding-size 50 :batch-size 10 :test-size 100 :num-epoch 10 :max-examples 1000})) diff --git a/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/data_helper.clj b/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/data_helper.clj index 796652177..82ba13087 100644 --- a/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/data_helper.clj +++ b/contrib/clojure-package/examples/cnn-text-classification/src/cnn_text_classification/data_helper.clj @@ -21,53 +21,84 @@ [org.apache.clojure-mxnet.context :as context] [org.apache.clojure-mxnet.ndarray :as ndarray] [org.apache.clojure-mxnet.random :as random]) - (:import (java.io DataInputStream)) + (:import (java.io DataInputStream) + (java.nio ByteBuffer ByteOrder)) (:gen-class)) (def w2v-file-path "../../data/GoogleNews-vectors-negative300.bin") ;; the word2vec file path -(def max-vectors 100) ;; If you are using word2vec embeddings and you want to only load part of them - -(defn r-string [dis] - (let [max-size 50 - bs (byte-array max-size) - sb (new StringBuilder)] - (loop [b (.readByte dis) - i 0] - (if (and (not= 32 b) (not= 10 b)) - (do (aset bs i b) - (if (= 49 i) - (do (.append sb (new String bs)) - (recur (.readByte dis) 0)) - (recur (.readByte dis) (inc i)))) - (.append sb (new String bs 0 i)))) - (.toString sb))) - -(defn get-float [b] - (-> 0 - (bit-or (bit-shift-left (bit-and (aget b 0) 0xff) 0)) - (bit-or (bit-shift-left (bit-and (aget b 1) 0xff) 8)) - (bit-or (bit-shift-left (bit-and (aget b 2) 0xff) 16)) - (bit-or (bit-shift-left (bit-and (aget b 3) 0xff) 24)))) +(def EOS "") ;; end of sentence word + +(defn glove-file-path + "Returns the file path to GloVe embedding of the input size" + [embedding-size] + (format "data/glove/glove.6B.%dd.txt" embedding-size)) + +(defn r-string + "Reads a string from the given DataInputStream `dis` until a space or newline is reached." + [dis] + (loop [b (.readByte dis) + bs []] + (if (and (not= 32 b) (not= 10 b)) + (recur (.readByte dis) (conj bs b)) + (new String (byte-array bs))))) + +(defn get-float [bs] + (-> (ByteBuffer/wrap bs) + (.order ByteOrder/LITTLE_ENDIAN) + (.getFloat))) (defn read-float [is] (let [bs (byte-array 4)] (do (.read is bs) (get-float bs)))) -(defn load-google-model [path] - (println "Loading the word2vec model from binary ...") - (with-open [bis (io/input-stream path) - dis (new DataInputStream bis)] - (let [word-size (Integer/parseInt (r-string dis)) - dim (Integer/parseInt (r-string dis)) - _ (println "Processing with " {:dim dim :word-size word-size} " loading max vectors " max-vectors) - word2vec (reduce (fn [r _] - (assoc r (r-string dis) - (mapv (fn [_] (read-float dis)) (range dim)))) - {} - (range max-vectors))] - (println "Finished") - {:num-embed dim :word2vec word2vec}))) +(defn- load-w2v-vectors + "Lazily loads the word2vec vectors given a data input stream `dis`, + number of words `nwords` and dimensionality `embedding-size`." + [dis embedding-size num-vectors] + (if (= 0 num-vectors) + (list) + (let [word (r-string dis) + vect (mapv (fn [_] (read-float dis)) (range embedding-size))] + (cons [word vect] (lazy-seq (load-w2v-vectors dis embedding-size (dec num-vectors))))))) + +(defn load-word2vec-model + "Loads the word2vec model stored in a binary format from the given `path`. + By default only the first 100 embeddings are loaded." + ([path embedding-size opts] + (println "Loading the word2vec model from binary ...") + (with-open [bis (io/input-stream path) + dis (new DataInputStream bis)] + (let [word-size (Integer/parseInt (r-string dis)) + dim (Integer/parseInt (r-string dis)) + {:keys [max-vectors vocab] :or {max-vectors word-size}} opts + _ (println "Processing with " {:dim dim :word-size word-size} " loading max vectors " max-vectors) + _ (if (not= embedding-size dim) + (throw (ex-info "Mismatch in embedding size" + {:input-embedding-size embedding-size + :word2vec-embedding-size dim}))) + vectors (load-w2v-vectors dis dim max-vectors) + word2vec (if vocab + (->> vectors + (filter (fn [[w _]] (contains? vocab w))) + (into {})) + (->> vectors + (take max-vectors) + (into {})))] + (println "Finished") + {:num-embed dim :word2vec word2vec}))) + ([path embedding-size] + (load-word2vec-model path embedding-size {:max-vectors 100}))) + +(defn read-text-embedding-pairs [rdr] + (for [^String line (line-seq rdr) + :let [fields (.split line " ")]] + [(aget fields 0) + (mapv #(Float/parseFloat ^String %) (rest fields))])) + +(defn load-glove [glove-file-path] + (println "Loading the glove pre-trained word embeddings from " glove-file-path) + (into {} (read-text-embedding-pairs (io/reader glove-file-path)))) (defn clean-str [s] (-> s @@ -84,9 +115,12 @@ (string/replace #"\)" " ) ") (string/replace #"\?" " ? ") (string/replace #" {2,}" " ") - (string/trim)));; Loads MR polarity data from files, splits the data into words and generates labels. - ;; Returns split sentences and labels. -(defn load-mr-data-and-labels [path max-examples] + (string/trim))) + +(defn load-mr-data-and-labels + "Loads MR polarity data from files, splits the data into words and generates labels. + Returns split sentences and labels." + [path max-examples] (println "Loading all the movie reviews from " path) (let [positive-examples (mapv #(string/trim %) (-> (slurp (str path "/rt-polarity.pos")) (string/split #"\n"))) @@ -104,41 +138,68 @@ negative-labels (mapv (constantly 0) negative-examples)] {:sentences x-text :labels (into positive-labels negative-labels)})) -;; Pads all sentences to the same length. The length is defined by the longest sentence. -;; Returns padded sentences. -(defn pad-sentences [sentences] - (let [padding-word "" +(defn pad-sentences + "Pads all sentences to the same length where the length is defined by the longest sentence. Returns padded sentences." + [sentences] + (let [padding-word EOS sequence-len (apply max (mapv count sentences))] (mapv (fn [s] (let [diff (- sequence-len (count s))] (if (pos? diff) (into s (repeat diff padding-word)) s))) - sentences)));; Map sentences and labels to vectors based on a pretrained embeddings -(defn build-input-data-with-embeddings [sentences embedding-size embeddings] - (mapv (fn [sent] - (mapv (fn [word] (or (get embeddings word) - (ndarray/->vec (random/uniform -0.25 0.25 [embedding-size])))) - sent)) - sentences)) - -(defn load-ms-with-embeddings [path embedding-size embeddings max-examples] - (println "Translating the movie review words into the embeddings") + sentences))) + +(defn build-vocab-embeddings + "Returns the subset of `embeddings` for words from the `vocab`. + Embeddings for words not in the vocabulary are initialized randomly + from a uniform distribution." + [vocab embedding-size embeddings] + (into {} + (mapv (fn [[word _]] + [word (or (get embeddings word) + (ndarray/->vec (random/uniform -0.25 0.25 [embedding-size])))]) + vocab))) + +(defn build-input-data-with-embeddings + "Map sentences and labels to vectors based on a pretrained embeddings." + [sentences embeddings] + (mapv (fn [sent] (mapv #(embeddings %) sent)) sentences)) + +(defn build-vocab + "Creates a vocabulary for the data set based on frequency of words. + Returns a map from words to unique indices." + [sentences] + (let [words (flatten sentences) + wc (reduce + (fn [m w] (update-in m [w] (fnil inc 0))) + {} + words) + sorted-wc (sort-by second > wc) + sorted-w (map first sorted-wc)] + (into {} (map vector sorted-w (range (count sorted-w)))))) + +(defn load-ms-with-embeddings + "Loads the movie review sentences data set for the given + `:pretrained-embedding` (e.g. `nil`, `:glove` or `:word2vec`)" + [path max-examples embedding-size {:keys [pretrained-embedding] + :or {pretrained-embedding nil} + :as opts}] (let [{:keys [sentences labels]} (load-mr-data-and-labels path max-examples) sentences-padded (pad-sentences sentences) - data (build-input-data-with-embeddings sentences-padded embedding-size embeddings)] + vocab (build-vocab sentences-padded) + vocab-embeddings (case pretrained-embedding + :glove (->> (load-glove (glove-file-path embedding-size)) + (build-vocab-embeddings vocab embedding-size)) + :word2vec (->> (load-word2vec-model w2v-file-path embedding-size {:vocab vocab}) + (:word2vec) + (build-vocab-embeddings vocab embedding-size)) + vocab) + data (build-input-data-with-embeddings sentences-padded vocab-embeddings)] {:data data :label labels :sentence-count (count data) :sentence-size (count (first data)) - :embedding-size embedding-size})) - -(defn read-text-embedding-pairs [rdr] - (for [^String line (line-seq rdr) - :let [fields (.split line " ")]] - [(aget fields 0) - (mapv #(Double/parseDouble ^String %) (rest fields))])) - -(defn load-glove [glove-file-path] - (println "Loading the glove pre-trained word embeddings from " glove-file-path) - (into {} (read-text-embedding-pairs (io/reader glove-file-path)))) + :embedding-size embedding-size + :vocab-size (count vocab) + :pretrained-embedding pretrained-embedding})) diff --git a/contrib/clojure-package/examples/cnn-text-classification/test/cnn_text_classification/classifier_test.clj b/contrib/clojure-package/examples/cnn-text-classification/test/cnn_text_classification/classifier_test.clj new file mode 100644 index 000000000..744307e3e --- /dev/null +++ b/contrib/clojure-package/examples/cnn-text-classification/test/cnn_text_classification/classifier_test.clj @@ -0,0 +1,48 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns cnn-text-classification.classifier-test + (:require [clojure.test :refer :all] + [org.apache.clojure-mxnet.module :as module] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.util :as util] + [org.apache.clojure-mxnet.context :as context] + [cnn-text-classification.classifier :as classifier])) + +(deftest classifier-with-embeddings-test + (let [train (classifier/train-convnet + {:devs [(context/default-context)] + :embedding-size 50 + :batch-size 10 + :test-size 100 + :num-epoch 1 + :max-examples 1000 + :pretrained-embedding :glove})] + (is (= ["data"] (util/scala-vector->vec (module/data-names train)))) + (is (= 20 (count (ndarray/->vec (-> train module/outputs ffirst))))))) + +(deftest classifier-without-embeddings-test + (let [train (classifier/train-convnet + {:devs [(context/default-context)] + :embedding-size 50 + :batch-size 10 + :test-size 100 + :num-epoch 1 + :max-examples 1000 + :pretrained-embedding nil})] + (is (= ["data"] (util/scala-vector->vec (module/data-names train)))) + (is (= 20 (count (ndarray/->vec (-> train module/outputs ffirst))))))) diff --git a/contrib/clojure-package/examples/gan/project.clj b/contrib/clojure-package/examples/gan/project.clj index 06f623227..a326f7a56 100644 --- a/contrib/clojure-package/examples/gan/project.clj +++ b/contrib/clojure-package/examples/gan/project.clj @@ -19,6 +19,7 @@ :description "GAN MNIST with MXNet" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"] - [nu.pattern/opencv "2.4.9-7"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"] + [org.openpnp/opencv "3.4.2-1"] + ] :main gan.gan-mnist) diff --git a/contrib/clojure-package/examples/gan/src/gan/gan_mnist.clj b/contrib/clojure-package/examples/gan/src/gan/gan_mnist.clj index e2e336453..944791bce 100644 --- a/contrib/clojure-package/examples/gan/src/gan/gan_mnist.clj +++ b/contrib/clojure-package/examples/gan/src/gan/gan_mnist.clj @@ -157,7 +157,9 @@ (save-img-diff i n calc-diff)))) -(defn train [devs] +(defn train + ([devs] (train devs num-epoch)) + ([devs num-epoch] (let [mod-d (-> (m/module (discriminator) {:contexts devs :data-names ["data"] :label-names ["label"]}) (m/bind {:data-shapes (mx-io/provide-data-desc mnist-iter) :label-shapes (mx-io/provide-label-desc mnist-iter) @@ -203,7 +205,7 @@ (save-img-gout i n (ndarray/copy (ffirst out-g))) (save-img-data i n batch) (calc-diff i n (ffirst diff-d))) - (inc n))))))) + (inc n)))))))) (defn -main [& args] (let [[dev dev-num] args diff --git a/contrib/clojure-package/examples/gan/src/gan/viz.clj b/contrib/clojure-package/examples/gan/src/gan/viz.clj index 8b57b9432..67f78806d 100644 --- a/contrib/clojure-package/examples/gan/src/gan/viz.clj +++ b/contrib/clojure-package/examples/gan/src/gan/viz.clj @@ -22,7 +22,7 @@ (:import (nu.pattern OpenCV) (org.opencv.core Core CvType Mat Size) (org.opencv.imgproc Imgproc) - (org.opencv.highgui Highgui))) + (org.opencv.imgcodecs Imgcodecs))) ;;; Viz stuff (OpenCV/loadShared) @@ -83,5 +83,5 @@ _ (Core/vconcat (java.util.ArrayList. line-mats) result)] (do (Imgproc/resize result resized-img (new Size (* (.width result) 1.5) (* (.height result) 1.5))) - (Highgui/imwrite (str output-path title ".jpg") resized-img) + (Imgcodecs/imwrite (str output-path title ".jpg") resized-img) (Thread/sleep 1000)))) diff --git a/contrib/clojure-package/examples/gan/test/gan/gan_test.clj b/contrib/clojure-package/examples/gan/test/gan/gan_test.clj new file mode 100644 index 000000000..71b9126ca --- /dev/null +++ b/contrib/clojure-package/examples/gan/test/gan/gan_test.clj @@ -0,0 +1,25 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns gan.gan_test + (:require + [gan.gan-mnist :refer :all] + [org.apache.clojure-mxnet.context :as context] + [clojure.test :refer :all])) + +(deftest check-pdf + (train [(context/cpu)] 1)) \ No newline at end of file diff --git a/contrib/clojure-package/examples/imclassification/project.clj b/contrib/clojure-package/examples/imclassification/project.clj index ad0a28a76..5f77cf55c 100644 --- a/contrib/clojure-package/examples/imclassification/project.clj +++ b/contrib/clojure-package/examples/imclassification/project.clj @@ -19,6 +19,6 @@ :description "Clojure examples for image classification" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :pedantic? :skip :main imclassification.train-mnist) diff --git a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj index a43dc3b69..e61e9ebf6 100644 --- a/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj +++ b/contrib/clojure-package/examples/imclassification/src/imclassification/train_mnist.clj @@ -32,7 +32,7 @@ (def batch-size 10) ;; the batch size (def optimizer (optimizer/sgd {:learning-rate 0.01 :momentum 0.0})) (def eval-metric (eval-metric/accuracy)) -(def num-epoch 5) ;; the number of training epochs +(def num-epoch 1) ;; the number of training epochs (def kvstore "local") ;; the kvstore type ;;; Note to run distributed you might need to complile the engine with an option set (def role "worker") ;; scheduler/server/worker @@ -82,7 +82,9 @@ (sym/fully-connected "fc3" {:data data :num-hidden 10}) (sym/softmax-output "softmax" {:data data}))) -(defn start [devs] +(defn start + ([devs] (start devs num-epoch)) + ([devs _num-epoch] (when scheduler-host (println "Initing PS enviornments with " envs) (kvstore-server/init envs)) @@ -94,14 +96,18 @@ (do (println "Starting Training of MNIST ....") (println "Running with context devices of" devs) - (let [mod (m/module (get-symbol) {:contexts devs})] - (m/fit mod {:train-data train-data + (let [_mod (m/module (get-symbol) {:contexts devs})] + (m/fit _mod {:train-data train-data :eval-data test-data - :num-epoch num-epoch + :num-epoch _num-epoch :fit-params (m/fit-params {:kvstore kvstore :optimizer optimizer - :eval-metric eval-metric})})) - (println "Finish fit")))) + :eval-metric eval-metric})}) + (println "Finish fit") + _mod + ) + + )))) (defn -main [& args] (let [[dev dev-num] args diff --git a/contrib/clojure-package/examples/imclassification/test/imclassification/train_mnist_test.clj b/contrib/clojure-package/examples/imclassification/test/imclassification/train_mnist_test.clj new file mode 100644 index 000000000..2ebefc2fc --- /dev/null +++ b/contrib/clojure-package/examples/imclassification/test/imclassification/train_mnist_test.clj @@ -0,0 +1,39 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns imclassification.train-mnist-test + (:require + [clojure.test :refer :all] + [clojure.java.io :as io] + [clojure.string :as s] + [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.module :as module] + [imclassification.train-mnist :as mnist])) + +(defn- file-to-filtered-seq [file] + (->> + file + (io/file) + (io/reader) + (line-seq) + (filter #(not (s/includes? % "mxnet_version"))))) + +(deftest mnist-two-epochs-test + (module/save-checkpoint (mnist/start [(context/cpu)] 2) {:prefix "target/test" :epoch 2}) + (is (= + (file-to-filtered-seq "test/test-symbol.json.ref") + (file-to-filtered-seq "target/test-symbol.json")))) \ No newline at end of file diff --git a/contrib/clojure-package/examples/imclassification/test/test-symbol.json.ref b/contrib/clojure-package/examples/imclassification/test/test-symbol.json.ref new file mode 100644 index 000000000..ba1d2fad3 --- /dev/null +++ b/contrib/clojure-package/examples/imclassification/test/test-symbol.json.ref @@ -0,0 +1,105 @@ +{ + "nodes": [ + { + "op": "null", + "name": "data", + "inputs": [] + }, + { + "op": "null", + "name": "fc1_weight", + "attrs": {"num_hidden": "128"}, + "inputs": [] + }, + { + "op": "null", + "name": "fc1_bias", + "attrs": {"num_hidden": "128"}, + "inputs": [] + }, + { + "op": "FullyConnected", + "name": "fc1", + "attrs": {"num_hidden": "128"}, + "inputs": [[0, 0, 0], [1, 0, 0], [2, 0, 0]] + }, + { + "op": "Activation", + "name": "relu1", + "attrs": {"act_type": "relu"}, + "inputs": [[3, 0, 0]] + }, + { + "op": "null", + "name": "fc2_weight", + "attrs": {"num_hidden": "64"}, + "inputs": [] + }, + { + "op": "null", + "name": "fc2_bias", + "attrs": {"num_hidden": "64"}, + "inputs": [] + }, + { + "op": "FullyConnected", + "name": "fc2", + "attrs": {"num_hidden": "64"}, + "inputs": [[4, 0, 0], [5, 0, 0], [6, 0, 0]] + }, + { + "op": "Activation", + "name": "relu2", + "attrs": {"act_type": "relu"}, + "inputs": [[7, 0, 0]] + }, + { + "op": "null", + "name": "fc3_weight", + "attrs": {"num_hidden": "10"}, + "inputs": [] + }, + { + "op": "null", + "name": "fc3_bias", + "attrs": {"num_hidden": "10"}, + "inputs": [] + }, + { + "op": "FullyConnected", + "name": "fc3", + "attrs": {"num_hidden": "10"}, + "inputs": [[8, 0, 0], [9, 0, 0], [10, 0, 0]] + }, + { + "op": "null", + "name": "softmax_label", + "inputs": [] + }, + { + "op": "SoftmaxOutput", + "name": "softmax", + "inputs": [[11, 0, 0], [12, 0, 0]] + } + ], + "arg_nodes": [0, 1, 2, 5, 6, 9, 10, 12], + "node_row_ptr": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14 + ], + "heads": [[13, 0, 0]], + "attrs": {"mxnet_version": ["int", 10400]} +} \ No newline at end of file diff --git a/contrib/clojure-package/examples/infer/imageclassifier/.gitignore b/contrib/clojure-package/examples/infer/imageclassifier/.gitignore new file mode 100644 index 000000000..35491f1a0 --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/.gitignore @@ -0,0 +1,12 @@ +/target +/classes +/checkouts +/images +pom.xml +pom.xml.asc +*.jar +*.class +/.lein-* +/.nrepl-port +.hgignore +.hg/ diff --git a/contrib/clojure-package/examples/infer/imageclassifier/README.md b/contrib/clojure-package/examples/infer/imageclassifier/README.md new file mode 100644 index 000000000..a8328607c --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/README.md @@ -0,0 +1,24 @@ +# imageclassifier + +Run image classification using clojure infer package. + +## Installation + +Before you run this example, make sure that you have the clojure package installed. +In the main clojure package directory, do `lein install`. Then you can run +`lein install` in this directory. + +## Usage + +``` +$ chmod +x scripts/get_resnet_18_data.sh +$ ./scripts/get_resnet_18_data.sh +$ +$ lein run -- --help +$ lein run -- -m models/resnet-18/resnet-18 -i images/kitten.jpg -d images/ +$ +$ lein uberjar +$ java -jar target/imageclassifier-0.1.0-SNAPSHOT-standalone.jar --help +$ java -jar target/imageclassifier-0.1.0-SNAPSHOT-standalone.jar \ + -m models/resnet-18/resnet-18 -i images/kitten.jpg -d images/ +``` diff --git a/contrib/clojure-package/examples/infer/imageclassifier/project.clj b/contrib/clojure-package/examples/infer/imageclassifier/project.clj new file mode 100644 index 000000000..2d5b171d9 --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/project.clj @@ -0,0 +1,25 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(defproject imageclassifier "0.1.0-SNAPSHOT" + :description "Image classification using infer with MXNet" + :plugins [[lein-cljfmt "0.5.7"]] + :dependencies [[org.clojure/clojure "1.9.0"] + [org.clojure/tools.cli "0.4.1"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] + :main ^:skip-aot infer.imageclassifier-example + :profiles {:uberjar {:aot :all}}) diff --git a/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_18_data.sh b/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_18_data.sh new file mode 100755 index 000000000..1a142e8ed --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_18_data.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -evx + +MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd) + +data_path=$MXNET_ROOT/models/resnet-18/ + +image_path=$MXNET_ROOT/images/ + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +if [ ! -d "$image_path" ]; then + mkdir -p "$image_path" +fi + +if [ ! -f "$data_path/resnet-18-0000.params" ]; then + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -P $data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -P $data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -P $data_path +fi + +if [ ! -f "$image_path/kitten.jpg" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path + wget https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg -P $image_path +fi diff --git a/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_data.sh b/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_data.sh new file mode 100755 index 000000000..fcef59bac --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/scripts/get_resnet_data.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e + +MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd) + +data_path=$MXNET_ROOT/models/resnet-152/ + +image_path=$MXNET_ROOT/images/ + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +if [ ! -d "$image_path" ]; then + mkdir -p "$image_path" +fi + +if [ ! -f "$data_path/resnet-152-0000.params" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-0000.params -P $data_path + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-symbol.json -P $data_path + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/synset.txt -P $data_path +fi + +if [ ! -f "$image_path/kitten.jpg" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path +fi diff --git a/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj new file mode 100644 index 000000000..6994b4fad --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/src/infer/imageclassifier_example.clj @@ -0,0 +1,111 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.imageclassifier-example + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.string :refer [join]] + [clojure.tools.cli :refer [parse-opts]]) + (:gen-class)) + +(defn check-valid-dir + "Check that the input directory exists" + [input-dir] + (let [dir (io/file input-dir)] + (and + (.exists dir) + (.isDirectory dir)))) + +(defn check-valid-file + "Check that the file exists" + [input-file] + (.exists (io/file input-file))) + +(def cli-options + [["-m" "--model-path-prefix PREFIX" "Model path prefix" + :default "models/resnet-18/resnet-18" + :validate [#(check-valid-file (str % "-symbol.json")) + "Model path prefix is invalid"]] + ["-i" "--input-image IMAGE" "Input image" + :default "images/kitten.jpg" + :validate [check-valid-file "Input file not found"]] + ["-d" "--input-dir IMAGE_DIR" "Input directory" + :default "images/" + :validate [check-valid-dir "Input directory not found"]] + ["-h" "--help"]]) + +(defn print-predictions + "Print image classifier predictions for the given input file" + [predictions] + (println (apply str (repeat 80 "="))) + (doseq [p predictions] + (println p)) + (println (apply str (repeat 80 "=")))) + +(defn classify-single-image + "Classify a single image and print top-5 predictions" + [classifier input-image] + (let [image (infer/load-image-from-file input-image) + topk 5 + predictions (infer/classify-image classifier image topk)] + [predictions])) + +(defn classify-images-in-dir + "Classify all jpg images in the directory" + [classifier input-dir] + (let [batch-size 20 + image-file-batches (->> input-dir + io/file + file-seq + (filter #(.isFile %)) + (filter #(re-matches #".*\.jpg$" (.getPath %))) + (mapv #(.getPath %)) + (partition-all batch-size))] + (apply concat (for [image-files image-file-batches] + (let [image-batch (infer/load-image-paths image-files) + topk 5] + (infer/classify-image-batch classifier image-batch topk)))))) + +(defn run-classifier + "Runs an image classifier based on options provided" + [options] + (let [{:keys [model-path-prefix input-image input-dir]} options + descriptors [{:name "data" + :shape [1 3 224 224] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors) + classifier (infer/create-image-classifier + factory {:contexts [(context/default-context)]})] + (println "Classifying a single image") + (print-predictions (classify-single-image classifier input-image)) + (println "\n") + (println "Classifying images in a directory") + (doseq [predictions (classify-images-in-dir classifier input-dir)] + (print-predictions predictions)))) + +(defn -main + [& args] + (let [{:keys [options summary errors] :as opts} + (parse-opts args cli-options)] + (cond + (:help options) (println summary) + (some? errors) (println (join "\n" errors)) + :else (run-classifier options)))) diff --git a/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj b/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj new file mode 100644 index 000000000..4b71f845d --- /dev/null +++ b/contrib/clojure-package/examples/infer/imageclassifier/test/infer/imageclassifier_example_test.clj @@ -0,0 +1,58 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.imageclassifier-example-test + (:require [infer.imageclassifier-example :refer [classify-single-image + classify-images-in-dir]] + [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.test :refer :all])) + +(def model-dir "models/") +(def image-dir "images/") +(def model-path-prefix (str model-dir "resnet-18/resnet-18")) +(def image-file (str image-dir "kitten.jpg")) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/get_resnet_18_data.sh")) + +(defn create-classifier [] + (let [descriptors [{:name "data" + :shape [1 3 224 224] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-image-classifier factory))) + +(deftest test-single-classification + (let [classifier (create-classifier) + [[predictions]] (classify-single-image classifier image-file)] + (is (some? predictions)) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-batch-classification + (let [classifier (create-classifier) + predictions (first (classify-images-in-dir classifier image-dir))] + (is (some? predictions)) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) diff --git a/contrib/clojure-package/examples/infer/objectdetector/.gitignore b/contrib/clojure-package/examples/infer/objectdetector/.gitignore new file mode 100644 index 000000000..35491f1a0 --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/.gitignore @@ -0,0 +1,12 @@ +/target +/classes +/checkouts +/images +pom.xml +pom.xml.asc +*.jar +*.class +/.lein-* +/.nrepl-port +.hgignore +.hg/ diff --git a/contrib/clojure-package/examples/infer/objectdetector/README.md b/contrib/clojure-package/examples/infer/objectdetector/README.md new file mode 100644 index 000000000..921c53e04 --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/README.md @@ -0,0 +1,24 @@ +# objectdetector + +Run object detection on images using clojure infer package. + +## Installation + +Before you run this example, make sure that you have the clojure package installed. +In the main clojure package directory, do `lein install`. Then you can run +`lein install` in this directory. + +## Usage + +``` +$ chmod +x scripts/get_ssd_data.sh +$ ./scripts/get_ssd_data.sh +$ +$ lein run -- --help +$ lein run -- -m models/resnet50_ssd/resnet50_ssd_model -i images/dog.jpg -d images/ +$ +$ lein uberjar +$ java -jar target/objectdetector-0.1.0-SNAPSHOT-standalone.jar --help +$ java -jar target/objectdetector-0.1.0-SNAPSHOT-standalone.jar \ + -m models/resnet50_ssd/resnet50_ssd_model -i images/dog.jpg -d images/ +``` diff --git a/contrib/clojure-package/examples/infer/objectdetector/project.clj b/contrib/clojure-package/examples/infer/objectdetector/project.clj new file mode 100644 index 000000000..4501f14a3 --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/project.clj @@ -0,0 +1,25 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(defproject objectdetector "0.1.0-SNAPSHOT" + :description "Object detection using infer with MXNet" + :plugins [[lein-cljfmt "0.5.7"]] + :dependencies [[org.clojure/clojure "1.9.0"] + [org.clojure/tools.cli "0.4.1"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] + :main ^:skip-aot infer.objectdetector-example + :profiles {:uberjar {:aot :all}}) diff --git a/contrib/clojure-package/examples/infer/objectdetector/scripts/get_ssd_data.sh b/contrib/clojure-package/examples/infer/objectdetector/scripts/get_ssd_data.sh new file mode 100755 index 000000000..06440a284 --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/scripts/get_ssd_data.sh @@ -0,0 +1,49 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +set -e + +MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd) + +data_path=$MXNET_ROOT/models/resnet50_ssd + +image_path=$MXNET_ROOT/images + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +if [ ! -d "$image_path" ]; then + mkdir -p "$image_path" +fi + +if [ ! -f "$data_path/resnet50_ssd_model-0000.params" ]; then + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $data_path + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $data_path + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -P $data_path +fi + +if [ ! -f "$image_path/000001.jpg" ]; then + cd $image_path + wget https://cloud.githubusercontent.com/assets/3307514/20012566/cbb53c76-a27d-11e6-9aaa-91939c9a1cd5.jpg -O 000001.jpg + wget https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -O dog.jpg + wget https://cloud.githubusercontent.com/assets/3307514/20012563/cbb41382-a27d-11e6-92a9-18dab4fd1ad3.jpg -O person.jpg +fi + diff --git a/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj b/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj new file mode 100644 index 000000000..5c30e5db6 --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/src/infer/objectdetector_example.clj @@ -0,0 +1,120 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.objectdetector-example + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.string :refer [join]] + [clojure.tools.cli :refer [parse-opts]]) + (:gen-class)) + +(defn check-valid-dir + "Check that the input directory exists" + [input-dir] + (let [dir (io/file input-dir)] + (and + (.exists dir) + (.isDirectory dir)))) + +(defn check-valid-file + "Check that the file exists" + [input-file] + (.exists (io/file input-file))) + +(def cli-options + [["-m" "--model-path-prefix PREFIX" "Model path prefix" + :default "models/resnet50_ssd/resnet50_ssd_model" + :validate [#(check-valid-file (str % "-symbol.json")) + "Model path prefix is invalid"]] + ["-i" "--input-image IMAGE" "Input image" + :default "images/dog.jpg" + :validate [check-valid-file "Input file not found"]] + ["-d" "--input-dir IMAGE_DIR" "Input directory" + :default "images/" + :validate [check-valid-dir "Input directory not found"]] + ["-h" "--help"]]) + +(defn print-predictions + "Print image detector predictions for the given input file" + [predictions width height] + (println (apply str (repeat 80 "="))) + (doseq [{:keys [class prob x-min y-min x-max y-max]} predictions] + (println (format + "Class: %s Prob=%.5f Coords=(%.3f, %.3f, %.3f, %.3f)" + class + prob + (* x-min width) + (* y-min height) + (* x-max width) + (* y-max height)))) + (println (apply str (repeat 80 "=")))) + +(defn detect-single-image + "Detect objects in a single image and print top-5 predictions" + [detector input-image] + (let [image (infer/load-image-from-file input-image) + topk 5 + [predictions] (infer/detect-objects detector image topk)] + predictions)) + +(defn detect-images-in-dir + "Detect objects in all jpg images in the directory" + [detector input-dir] + (let [batch-size 20 + image-file-batches (->> input-dir + io/file + file-seq + (filter #(.isFile %)) + (filter #(re-matches #".*\.jpg$" (.getPath %))) + (mapv #(.getPath %)) + (partition-all batch-size))] + (apply concat (for [image-files image-file-batches] + (let [image-batch (infer/load-image-paths image-files) + topk 5] + (infer/detect-objects-batch detector image-batch topk)))))) + +(defn run-detector + "Runs an image detector based on options provided" + [options] + (let [{:keys [model-path-prefix input-image input-dir + device device-id]} options + width 512 height 512 + descriptors [{:name "data" + :shape [1 3 height width] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors) + detector (infer/create-object-detector + factory + {:contexts [(context/default-context)]})] + (println "Object detection on a single image") + (print-predictions (detect-single-image detector input-image) width height) + (println "\n") + (println "Object detection on images in a directory") + (doseq [predictions (detect-images-in-dir detector input-dir)] + (print-predictions predictions width height)))) + +(defn -main + [& args] + (let [{:keys [options summary errors] :as opts} + (parse-opts args cli-options)] + (cond + (:help options) (println summary) + (some? errors) (println (join "\n" errors)) + :else (run-detector options)))) diff --git a/contrib/clojure-package/examples/infer/objectdetector/test/infer/objectdetector_example_test.clj b/contrib/clojure-package/examples/infer/objectdetector/test/infer/objectdetector_example_test.clj new file mode 100644 index 000000000..2b8ad951a --- /dev/null +++ b/contrib/clojure-package/examples/infer/objectdetector/test/infer/objectdetector_example_test.clj @@ -0,0 +1,65 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.objectdetector-example-test + (:require [infer.objectdetector-example :refer [detect-single-image + detect-images-in-dir]] + [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.test :refer :all])) + +(def model-dir "models/") +(def image-dir "images/") +(def model-path-prefix (str model-dir "resnet50_ssd/resnet50_ssd_model")) +(def image-file (str image-dir "dog.jpg")) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/get_ssd_data.sh")) + +(defn create-detector [] + (let [descriptors [{:name "data" + :shape [1 3 512 512] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-object-detector factory))) + +(deftest test-single-detection + (let [detector (create-detector) + predictions (detect-single-image detector image-file) + {:keys [class prob x-min x-max y-min y-max] :as pred} (first predictions)] + (is (some? predictions)) + (is (= 5 (count predictions))) + (is (string? class)) + (is (< 0.8 prob)) + (is (every? #(< 0 % 1) [x-min x-max y-min y-max])) + (is (= #{"dog" "person" "bicycle" "car"} (set (mapv :class predictions)))))) + +(deftest test-batch-detection + (let [detector (create-detector) + batch-predictions (detect-images-in-dir detector image-dir) + predictions (first batch-predictions) + {:keys [class prob x-min x-max y-min y-max] :as pred} (first predictions)] + (is (some? batch-predictions)) + (is (= 5 (count predictions))) + (is (string? class)) + (is (< 0.8 prob)) + (every? #(< 0 % 1) [x-min x-max y-min y-max]) + (is (= #{"dog" "person" "bicycle" "car"} (set (mapv :class predictions)))))) diff --git a/contrib/clojure-package/examples/infer/predictor/.gitignore b/contrib/clojure-package/examples/infer/predictor/.gitignore new file mode 100644 index 000000000..35491f1a0 --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/.gitignore @@ -0,0 +1,12 @@ +/target +/classes +/checkouts +/images +pom.xml +pom.xml.asc +*.jar +*.class +/.lein-* +/.nrepl-port +.hgignore +.hg/ diff --git a/contrib/clojure-package/examples/infer/predictor/README.md b/contrib/clojure-package/examples/infer/predictor/README.md new file mode 100644 index 000000000..9ca71cf46 --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/README.md @@ -0,0 +1,24 @@ +# predictor + +Run model prediction using clojure infer package. + +## Installation + +Before you run this example, make sure that you have the clojure package installed. +In the main clojure package directory, do `lein install`. Then you can run +`lein install` in this directory. + +## Usage + +``` +$ chmod +x scripts/get_resnet_18_data.sh +$ ./scripts/get_resnet_18_data.sh +$ +$ lein run -- --help +$ lein run -- -m models/resnet-18/resnet-18 -i images/kitten.jpg +$ +$ lein uberjar +$ java -jar target/predictor-0.1.0-SNAPSHOT-standalone.jar --help +$ java -jar target/predictor-0.1.0-SNAPSHOT-standalone.jar \ + -m models/resnet-18/resnet-18 -i images/kitten.jpg +``` diff --git a/contrib/clojure-package/examples/infer/predictor/project.clj b/contrib/clojure-package/examples/infer/predictor/project.clj new file mode 100644 index 000000000..0bd1eaee6 --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/project.clj @@ -0,0 +1,25 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(defproject predictor "0.1.0-SNAPSHOT" + :description "Model prediction using infer with MXNet" + :plugins [[lein-cljfmt "0.5.7"]] + :dependencies [[org.clojure/clojure "1.9.0"] + [org.clojure/tools.cli "0.4.1"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] + :main ^:skip-aot infer.predictor-example + :profiles {:uberjar {:aot :all}}) diff --git a/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_18_data.sh b/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_18_data.sh new file mode 100755 index 000000000..cf85355fa --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_18_data.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -evx + +MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd) + +data_path=$MXNET_ROOT/models/resnet-18/ + +image_path=$MXNET_ROOT/images/ + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +if [ ! -d "$image_path" ]; then + mkdir -p "$image_path" +fi + +if [ ! -f "$data_path/resnet-18-0000.params" ]; then + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -P $data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -P $data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -P $data_path +fi + +if [ ! -f "$image_path/kitten.jpg" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path +fi diff --git a/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_data.sh b/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_data.sh new file mode 100755 index 000000000..fcef59bac --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/scripts/get_resnet_data.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e + +MXNET_ROOT=$(cd "$(dirname $0)/.."; pwd) + +data_path=$MXNET_ROOT/models/resnet-152/ + +image_path=$MXNET_ROOT/images/ + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +if [ ! -d "$image_path" ]; then + mkdir -p "$image_path" +fi + +if [ ! -f "$data_path/resnet-152-0000.params" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-0000.params -P $data_path + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-symbol.json -P $data_path + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/synset.txt -P $data_path +fi + +if [ ! -f "$image_path/kitten.jpg" ]; then + wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path +fi diff --git a/contrib/clojure-package/examples/infer/predictor/src/infer/predictor_example.clj b/contrib/clojure-package/examples/infer/predictor/src/infer/predictor_example.clj new file mode 100644 index 000000000..05eb0add3 --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/src/infer/predictor_example.clj @@ -0,0 +1,101 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.predictor-example + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.image :as image] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [clojure.java.io :as io] + [clojure.string :refer [join split]] + [clojure.tools.cli :refer [parse-opts]]) + (:gen-class)) + +(defn check-valid-file + "Check that the file exists" + [input-file] + (.exists (io/file input-file))) + +(def cli-options + [["-m" "--model-path-prefix PREFIX" "Model path prefix" + :default "models/resnet-18/resnet-18" + :validate [#(check-valid-file (str % "-symbol.json")) + "Model path prefix is invalid"]] + ["-i" "--input-image IMAGE" "Image path" + :default "images/kitten.jpg" + :validate [check-valid-file "Input image path not found"]] + ["-h" "--help"]]) + +(defn print-prediction + [prediction] + (println (apply str (repeat 80 "="))) + (println prediction) + (println (apply str (repeat 80 "=")))) + +(defn preprocess + "Preprocesses image to make it ready for prediction" + [image-path width height] + (-> image-path + infer/load-image-from-file + (infer/reshape-image width height) + (infer/buffered-image-to-pixels [3 width height]) + (ndarray/expand-dims 0))) + +(defn do-inference + "Run inference using given predictor" + [predictor image] + (let [predictions (infer/predict-with-ndarray predictor [image])] + (first predictions))) + +(defn postprocess + [model-path-prefix predictions] + (let [synset-file (-> model-path-prefix + io/file + (.getParent) + (io/file "synset.txt")) + synset-names (split (slurp synset-file) #"\n") + [max-idx] (ndarray/->int-vec (ndarray/argmax predictions 1))] + (synset-names max-idx))) + +(defn run-predictor + "Runs an image classifier based on options provided" + [options] + (let [{:keys [model-path-prefix input-image]} options + width 224 + height 224 + descriptors [{:name "data" + :shape [1 3 height width] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors) + predictor (infer/create-predictor + factory + {:contexts [(context/default-context)]}) + image-ndarray (preprocess input-image width height) + predictions (do-inference predictor image-ndarray) + best-prediction (postprocess model-path-prefix predictions)] + (print-prediction best-prediction))) + +(defn -main + [& args] + (let [{:keys [options summary errors] :as opts} + (parse-opts args cli-options)] + (cond + (:help options) (println summary) + (some? errors) (println (join "\n" errors)) + :else (run-predictor options)))) diff --git a/contrib/clojure-package/examples/infer/predictor/test/infer/predictor_example_test.clj b/contrib/clojure-package/examples/infer/predictor/test/infer/predictor_example_test.clj new file mode 100644 index 000000000..02f826fbb --- /dev/null +++ b/contrib/clojure-package/examples/infer/predictor/test/infer/predictor_example_test.clj @@ -0,0 +1,51 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns infer.predictor-example-test + (:require [infer.predictor-example :refer [preprocess + do-inference + postprocess]] + [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.test :refer :all])) + +(def model-dir "models/") +(def image-file "images/kitten.jpg") +(def model-path-prefix (str model-dir "resnet-18/resnet-18")) +(def width 224) +(def height 224) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/get_resnet_18_data.sh")) + +(defn create-predictor [] + (let [descriptors [{:name "data" + :shape [1 3 height width] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-predictor factory))) + +(deftest predictor-test + (let [predictor (create-predictor) + image-ndarray (preprocess image-file width height) + predictions (do-inference predictor image-ndarray) + best-prediction (postprocess model-path-prefix predictions)] + (is (= "n02123159 tiger cat" best-prediction)))) diff --git a/contrib/clojure-package/examples/module/project.clj b/contrib/clojure-package/examples/module/project.clj index 487ceece9..b667a2a4e 100644 --- a/contrib/clojure-package/examples/module/project.clj +++ b/contrib/clojure-package/examples/module/project.clj @@ -19,7 +19,7 @@ :description "Clojure examples for module" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :pedantic? :skip :main mnist-mlp) diff --git a/contrib/clojure-package/examples/module/test/mnist_mlp_test.clj b/contrib/clojure-package/examples/module/test/mnist_mlp_test.clj new file mode 100644 index 000000000..5fbcdd3c0 --- /dev/null +++ b/contrib/clojure-package/examples/module/test/mnist_mlp_test.clj @@ -0,0 +1,29 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; +(ns mnist-mlp-test + (:require + [mnist-mlp :refer :all] + [org.apache.clojure-mxnet.context :as context] + [clojure.test :refer :all])) + +(deftest run-those-tests + (let [devs [(context/cpu)]] + (run-intermediate-level-api :devs devs) + (run-intermediate-level-api :devs devs :load-model-epoch (dec num-epoch)) + (run-high-level-api devs) + (run-prediction-iterator-api devs) + (run-predication-and-calc-accuracy-manually devs))) \ No newline at end of file diff --git a/contrib/clojure-package/examples/multi-label/project.clj b/contrib/clojure-package/examples/multi-label/project.clj index a41c7fd7b..6e6a14340 100644 --- a/contrib/clojure-package/examples/multi-label/project.clj +++ b/contrib/clojure-package/examples/multi-label/project.clj @@ -19,5 +19,5 @@ :description "Example of multi-label classification" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :main multi-label.core) diff --git a/contrib/clojure-package/examples/multi-label/test/multi_label_test.clj b/contrib/clojure-package/examples/multi-label/test/multi_label_test.clj new file mode 100644 index 000000000..446a84626 --- /dev/null +++ b/contrib/clojure-package/examples/multi-label/test/multi_label_test.clj @@ -0,0 +1,26 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns multi_label_test + (:require + [multi-label.core :as label] + [clojure.java.io :as io] + [org.apache.clojure-mxnet.context :as context] + [clojure.test :refer :all])) + +(deftest run-multi-label + (label/train [(context/cpu)])) \ No newline at end of file diff --git a/contrib/clojure-package/examples/neural-style/project.clj b/contrib/clojure-package/examples/neural-style/project.clj index f8d6e28de..b6d29f7c0 100644 --- a/contrib/clojure-package/examples/neural-style/project.clj +++ b/contrib/clojure-package/examples/neural-style/project.clj @@ -19,7 +19,7 @@ :description "Neural Style Transfer with MXNet" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"] [net.mikera/imagez "0.12.0"] [thinktopic/think.image "0.4.16"]] :main neural-style.core) diff --git a/contrib/clojure-package/examples/neural-style/src/neural_style/core.clj b/contrib/clojure-package/examples/neural-style/src/neural_style/core.clj index 50f95c975..ac1f537f1 100644 --- a/contrib/clojure-package/examples/neural-style/src/neural_style/core.clj +++ b/contrib/clojure-package/examples/neural-style/src/neural_style/core.clj @@ -24,6 +24,8 @@ [org.apache.clojure-mxnet.random :as random] [org.apache.clojure-mxnet.shape :as mx-shape] [org.apache.clojure-mxnet.symbol :as sym] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] [mikera.image.core :as img] [mikera.image.filters :as img-filter] [think.image.pixel :as pixel] @@ -31,6 +33,9 @@ (:gen-class));; An Implementation of the paper A Neural Algorithm of Artistic Style ;;by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge +(when-not (.exists (io/file "input")) + (do (println "Retrieving data...") (sh "./download.sh"))) + (def content-image "input/IMG_4343.jpg") (def style-image "input/starry_night.jpg") (def model-path "model/vgg19.params") @@ -39,7 +44,7 @@ (def content-weight 5) ;; the weight for the content image (def blur-radius 1) ;; the blur filter radius (def output-dir "output") -(def lr 10) ;; the learning rate +(def lr 10.0) ;; the learning rate (def tv-weight 0.01) ;; the magnitude on the tv loss (def num-epochs 1000) (def num-channels 3) @@ -157,9 +162,10 @@ out (ndarray/* out tv-weight)] (sym/bind out ctx {"img" img "kernel" kernel})))) -(defn train [devs] - - (let [dev (first devs) +(defn train + ([devs] (train devs 20)) + ([devs n-epochs] + (let [dev (first devs) content-np (preprocess-content-image content-image max-long-edge) content-np-shape (mx-shape/->vec (ndarray/shape content-np)) style-np (preprocess-style-image style-image content-np-shape) @@ -193,7 +199,7 @@ ;;;train ;;initialize with random noise - img (ndarray/- (random/uniform 0 255 content-np-shape dev) 128) + img (ndarray/- (random/uniform 0 255 content-np-shape {:ctx dev}) 128) ;;; img (random/uniform -0.1 0.1 content-np-shape dev) ;; img content-np lr-sched (lr-scheduler/factor-scheduler 10 0.9) @@ -212,7 +218,7 @@ tv-grad-executor (get-tv-grad-executor img dev tv-weight) eps 0.0 e 0] - (doseq [i (range 20)] + (doseq [i (range n-epochs)] (ndarray/set (:data model-executor) img) (-> (:executor model-executor) (executor/forward) @@ -237,8 +243,10 @@ (println "Epoch " i "relative change " eps) (when (zero? (mod i 2)) (save-image (ndarray/copy img) (str output-dir "/out_" i ".png") blur-radius true))) - - (ndarray/set old-img img)))) + (ndarray/set old-img img)) + ; (save-image (ndarray/copy img) (str output-dir "/final.png") 0 false) + ; (postprocess-image img) + ))) (defn -main [& args] ;;; Note this only works on cpu right now diff --git a/contrib/clojure-package/examples/neural-style/test/neural_style/vgg_19_test.clj b/contrib/clojure-package/examples/neural-style/test/neural_style/vgg_19_test.clj new file mode 100644 index 000000000..a7c978607 --- /dev/null +++ b/contrib/clojure-package/examples/neural-style/test/neural_style/vgg_19_test.clj @@ -0,0 +1,53 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns neural-style.vgg-19-test + (:require + [clojure.test :refer :all] + [mikera.image.core :as img] + [clojure.java.io :as io] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.context :as context] + [neural-style.core :as neural])) + +(defn pic-to-ndarray-vec[path] + (-> path + img/load-image + neural/image->ndarray + ndarray/->vec)) + +(defn last-modified-check[x] + (let [t (- (System/currentTimeMillis) (.lastModified x)) ] + (if (> 10000 t) ; 10 seconds + x + (throw (Exception. (str "Generated File Too Old: (" t " ms) [" x "]")))))) + +(defn latest-pic-to-ndarray-vec[folder] + (->> folder + io/as-file + (.listFiles) + (sort-by #(.lastModified %)) + last + (last-modified-check) + (.getPath) + pic-to-ndarray-vec)) + +(deftest vgg-19-test + (neural/train [(context/cpu)] 3) + (is (not (nil? (latest-pic-to-ndarray-vec "output"))))) +; generated file different depending on the platform :/ +; (pic-to-ndarray-vec "test/ref_out_2.png")))) \ No newline at end of file diff --git a/contrib/clojure-package/examples/pre-trained-models/project.clj b/contrib/clojure-package/examples/pre-trained-models/project.clj index c61e9a6f8..11e002503 100644 --- a/contrib/clojure-package/examples/pre-trained-models/project.clj +++ b/contrib/clojure-package/examples/pre-trained-models/project.clj @@ -19,7 +19,7 @@ :description "Example of using pre-trained models with MXNet" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"] [net.mikera/imagez "0.12.0"] [thinktopic/think.image "0.4.16"]] :main pre-trained-models.fine-tune) diff --git a/contrib/clojure-package/examples/profiler/project.clj b/contrib/clojure-package/examples/profiler/project.clj index dfcfea7c2..cc50482d0 100644 --- a/contrib/clojure-package/examples/profiler/project.clj +++ b/contrib/clojure-package/examples/profiler/project.clj @@ -18,5 +18,5 @@ (defproject profiler "0.1.0-SNAPSHOT" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :main profiler.core) diff --git a/contrib/clojure-package/examples/profiler/src/profiler/core.clj b/contrib/clojure-package/examples/profiler/src/profiler/core.clj index e366c578c..67ba0feb8 100644 --- a/contrib/clojure-package/examples/profiler/src/profiler/core.clj +++ b/contrib/clojure-package/examples/profiler/src/profiler/core.clj @@ -27,9 +27,9 @@ (def profiler-mode "symbolic") ;; can be symbolic, imperative, api, mem (def output-path ".") ;; the profile file output directory (def profiler-name "profile-matmul-20iter.json") -(def iter-num 100) -(def begin-profiling-iter 50) -(def end-profiling-iter 70) +(def iter-num 5) +(def begin-profiling-iter 0) +(def end-profiling-iter 1) (def gpu? false) (defn run [] diff --git a/contrib/clojure-package/examples/profiler/test/core_test.clj b/contrib/clojure-package/examples/profiler/test/core_test.clj new file mode 100644 index 000000000..1173f0755 --- /dev/null +++ b/contrib/clojure-package/examples/profiler/test/core_test.clj @@ -0,0 +1,31 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns core_test + (:require + [profiler.core :as profiler] + [clojure.java.io :as io] + [clojure.test :refer :all])) + +(defn count-lines[file] + (count (line-seq (io/reader (io/as-file file))))) + +(deftest run-profiler + (profiler/run) + (let [new-file (clojure.java.io/as-file profiler/profiler-name)] + (is (.exists new-file)) + (is (> 10000 (- (System/currentTimeMillis) (.lastModified new-file)))))) \ No newline at end of file diff --git a/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref b/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref new file mode 100644 index 000000000..d6baa4211 --- /dev/null +++ b/contrib/clojure-package/examples/profiler/test/profile-matmul-20iter.json.ref @@ -0,0 +1,271 @@ +{ + "traceEvents": [ + { + "ph": "M", + "args": { + "name": "cpu/0" + }, + "pid": 0, + "name": "process_name" + }, + { + "ph": "M", + "args": { + "name": "cpu/1" + }, + "pid": 1, + "name": "process_name" + }, + { + "ph": "M", + "args": { + "name": "cpu/2" + }, + "pid": 2, + "name": "process_name" + }, + { + "ph": "M", + "args": { + "name": "cpu/3" + }, + "pid": 3, + "name": "process_name" + }, + { + "ph": "M", + "args": { + "name": "cpu pinned/" + }, + "pid": 4, + "name": "process_name" + }, + { + "ph": "M", + "args": { + "name": "cpu shared/" + }, + "pid": 5, + "name": "process_name" + }, { + "ph": "M", + "args": { + "name": "MXNET_C_API" + }, + "pid": 13841910479334118176, + "name": "process_name" + }, + + { + "name": "MXNet C API Calls", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51195258331, + "args": { "MXNet C API Calls": 1 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "MXNet C API Concurrency", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51195258338, + "args": { "MXNet C API Concurrency": 1 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "MXExecutorForward", + "cat": "MXNET_C_API", + "ph": "b", + "ts": 51195258348, + "id": 6902988396839073221, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "MXExecutorForward", + "cat": "MXNET_C_API", + "ph": "e", + "ts": 51195258357, + "id": 6902988396839073221, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "MXNet C API Concurrency", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51195258358, + "args": { "MXNet C API Concurrency": 0 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, { + "ph": "M", + "args": { + "name": "Device Storage" + }, + "pid": 13545698322897290393, + "name": "process_name" + }, + + { + "name": "Memory: cpu/0", + "cat": "Device Storage", + "ph": "C", + "ts": 51195543378, + "args": { "Memory: cpu/0": 8 }, + "pid": 13545698322897290393, + "tid": 5603937861270119161 + } +, + { + "name": "MXNet C API Calls", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51195258559, + "args": { "MXNet C API Calls": 2 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "Memory: cpu/0", + "cat": "Device Storage", + "ph": "C", + "ts": 51195857697, + "args": { "Memory: cpu/0": 67108872 }, + "pid": 13545698322897290393, + "tid": 5603937861270119161 + } +, + { + "name": "MXNet C API Concurrency", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51195258560, + "args": { "MXNet C API Concurrency": 1 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + + { + "name": "[dot]", + "cat": "operator", + "ph": "B", + "ts": 51195857671, + "pid": 0, + "tid": 5603937861270119161 + } +, + { + "name": "[dot]", + "cat": "operator", + "ph": "E", + "ts": 51196931353, + "pid": 0, + "tid": 5603937861270119161 + } +, + + { + "name": "WaitForVar", + "cat": "operator", + "ph": "B", + "ts": 51196931369, + "pid": 0, + "tid": 5603937861270119161 + } +, + { + "name": "WaitForVar", + "cat": "operator", + "ph": "E", + "ts": 51196931376, + "pid": 0, + "tid": 5603937861270119161 + } +, { + "ph": "M", + "args": { + "name": "operator" + }, + "pid": 10847949044720084585, + "name": "process_name" + }, + + { + "name": "[dot]", + "cat": "operator", + "ph": "b", + "ts": 51195857671, + "id": 5603937861270119161, + "pid": 10847949044720084585, + "tid": 5603937861270119161 + } +, + { + "name": "[dot]", + "cat": "operator", + "ph": "e", + "ts": 51196931350, + "id": 5603937861270119161, + "pid": 10847949044720084585, + "tid": 5603937861270119161 + } +, + { + "name": "MXNDArrayWaitToRead", + "cat": "MXNET_C_API", + "ph": "b", + "ts": 51195258561, + "id": 6902988396839073221, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "MXNDArrayWaitToRead", + "cat": "MXNET_C_API", + "ph": "e", + "ts": 51196931386, + "id": 6902988396839073221, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } +, + { + "name": "WaitForVar", + "cat": "operator", + "ph": "b", + "ts": 51196931369, + "id": 5603937861270119161, + "pid": 10847949044720084585, + "tid": 5603937861270119161 + } +, + { + "name": "WaitForVar", + "cat": "operator", + "ph": "e", + "ts": 51196931376, + "id": 5603937861270119161, + "pid": 10847949044720084585, + "tid": 5603937861270119161 + } +, + { + "name": "MXNet C API Concurrency", + "cat": "MXNET_C_API", + "ph": "C", + "ts": 51196931391, + "args": { "MXNet C API Concurrency": 0 }, + "pid": 13841910479334118176, + "tid": 6902988396839073221 + } diff --git a/contrib/clojure-package/examples/rnn/project.clj b/contrib/clojure-package/examples/rnn/project.clj index f411a5458..64f4c2907 100644 --- a/contrib/clojure-package/examples/rnn/project.clj +++ b/contrib/clojure-package/examples/rnn/project.clj @@ -19,5 +19,5 @@ :description "RNN example" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :main rnn.train-char-rnn) diff --git a/contrib/clojure-package/examples/rnn/src/rnn/test_char_rnn.clj b/contrib/clojure-package/examples/rnn/src/rnn/test_char_rnn.clj index d03b1a6b3..22a2982f2 100644 --- a/contrib/clojure-package/examples/rnn/src/rnn/test_char_rnn.clj +++ b/contrib/clojure-package/examples/rnn/src/rnn/test_char_rnn.clj @@ -17,6 +17,7 @@ (ns rnn.test-char-rnn (:require [clojure.string :as string] + [clojure.java.shell :refer [sh]] [rnn.util :as util] [rnn.lstm :as lstm] [org.apache.clojure-mxnet.context :as context] @@ -24,6 +25,9 @@ [org.apache.clojure-mxnet.module :as m] [org.apache.clojure-mxnet.ndarray :as ndarray])) +(when-not (.exists (clojure.java.io/file "data")) + (do (println "Retrieving data...") (sh "./get_data.sh"))) + (def data-path "data/obama.txt") (def model-prefix) (def start-sentence "The joke ") diff --git a/contrib/clojure-package/examples/rnn/src/rnn/train_char_rnn.clj b/contrib/clojure-package/examples/rnn/src/rnn/train_char_rnn.clj index 150cd94e6..41a764f7a 100644 --- a/contrib/clojure-package/examples/rnn/src/rnn/train_char_rnn.clj +++ b/contrib/clojure-package/examples/rnn/src/rnn/train_char_rnn.clj @@ -17,6 +17,7 @@ (ns rnn.train-char-rnn (:require [clojure.string :as string] + [clojure.java.shell :refer [sh]] [rnn.util :as util] [rnn.lstm :as lstm] [rnn.test-char-rnn :as test-rnn] @@ -34,6 +35,9 @@ ;;https://github.com/apache/incubator-mxnet/blob/master/example/rnn/old/char-rnn.ipynb +(when-not (.exists (clojure.java.io/file "data")) + (do (println "Retrieving data...") (sh "./get_data.sh"))) + ;; batch size for training (def batch-size 32) ;; we can support various length input diff --git a/contrib/clojure-package/examples/rnn/test/rnn/core_test.clj b/contrib/clojure-package/examples/rnn/test/rnn/core_test.clj new file mode 100644 index 000000000..b19857724 --- /dev/null +++ b/contrib/clojure-package/examples/rnn/test/rnn/core_test.clj @@ -0,0 +1,26 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns rnn.core_test + (:require + [rnn.test-char-rnn :as rnn] + [clojure.test :refer :all])) + +(deftest check-trained-network + (is (= + "The joke that we can start by the challenges of the American people. The American people have been talking about how to compete with the streets of San Antonio who the courage to come together as one " + (rnn/rnn-test "data/obama" 75 200 false)))) \ No newline at end of file diff --git a/contrib/clojure-package/examples/tutorial/.gitignore b/contrib/clojure-package/examples/tutorial/.gitignore index c53038ec0..338927e78 100644 --- a/contrib/clojure-package/examples/tutorial/.gitignore +++ b/contrib/clojure-package/examples/tutorial/.gitignore @@ -9,3 +9,4 @@ pom.xml.asc /.nrepl-port .hgignore .hg/ +filename \ No newline at end of file diff --git a/contrib/clojure-package/examples/tutorial/project.clj b/contrib/clojure-package/examples/tutorial/project.clj index 4910886ca..58a10f04f 100644 --- a/contrib/clojure-package/examples/tutorial/project.clj +++ b/contrib/clojure-package/examples/tutorial/project.clj @@ -19,4 +19,9 @@ :description "MXNET tutorials" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]]) + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"] + + ;; Uncomment the one appropriate for your machine & configuration: + #_[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-cpu "1.4.0"] + #_[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-gpu "1.4.0"] + #_[org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu "1.4.0"]]) diff --git a/contrib/clojure-package/examples/tutorial/src/tutorial/kvstore.clj b/contrib/clojure-package/examples/tutorial/src/tutorial/kvstore.clj index 558b21f0a..f35d4a069 100644 --- a/contrib/clojure-package/examples/tutorial/src/tutorial/kvstore.clj +++ b/contrib/clojure-package/examples/tutorial/src/tutorial/kvstore.clj @@ -16,35 +16,44 @@ ;; (ns tutorial.kvstore + "A REPL tutorial of the MXNet Clojure API for KVStore, based on + https://mxnet.incubator.apache.org/api/clojure/kvstore.html" (:require [org.apache.clojure-mxnet.kvstore :as kvstore] [org.apache.clojure-mxnet.ndarray :as ndarray] [org.apache.clojure-mxnet.context :as context])) -;;Basic Push and Pull -;;Provides basic operation over multiple devices (GPUs or CPUs) on a single device. -;; Initialization -;; Let’s consider a simple example. It initializes a (int, NDArray) pair into the store, and then pulls the value out. +;;;; Basic Push and Pull -(def kv (kvstore/create "local")) ;; create a local kvstore +;; Provides basic operation over multiple devices (GPUs or CPUs) on a +;; single device. + +;;; Initialization +;; Let’s consider a simple example. It initializes a (`int`, +;; `NDArray`) pair into the store, and then pulls the value out. + +(def kv (kvstore/create "local")) ; create a local kvstore (def shape [2 3]) -;;; init the kvstore with a vector of keys (strings) and ndarrays +;; init the kvstore with a vector of keys (strings) and ndarrays (kvstore/init kv ["3"] [(ndarray/* (ndarray/ones shape) 2)]) (def a (ndarray/zeros shape)) (kvstore/pull kv ["3"] [a]) (ndarray/->vec a) ;=> [2.0 2.0 2.0 2.0 2.0 2.0] -;;Push, Aggregation, and Updater -;;For any key that’s been initialized, you can push a new value with the same shape to the key, as follows: - +;;; Push, Aggregation, and Updater +;; For any key that’s been initialized, you can push a new value with +;; the same shape to the key, as follows: (kvstore/push kv ["3"] [(ndarray/* (ndarray/ones shape) 8)]) (kvstore/pull kv ["3"] [a]) (ndarray/->vec a);=>[8.0 8.0 8.0 8.0 8.0 8.0] -;;The data that you want to push can be stored on any device. Furthermore, you can push multiple values into the same key, where KVStore first sums all of these values, and then pushes the aggregated value, as follows: +;; The data that you want to push can be stored on any +;; device. Furthermore, you can push multiple values into the same +;; key, where KVStore first sums all of these values, and then pushes +;; the aggregated value, as follows: -;; using multiple cpus instead of gpus +;; (Here we use multiple CPUs.) (def cpus [(context/cpu 0) (context/cpu 1) (context/cpu 2)]) (def b [(ndarray/ones shape {:ctx (nth cpus 0)}) (ndarray/ones shape {:ctx (nth cpus 1)}) @@ -53,22 +62,33 @@ (kvstore/pull kv "3" a) (ndarray/->vec a) ;=> [3.0 3.0 3.0 3.0 3.0 3.0] - -;;Pull -;;You’ve already seen how to pull a single key-value pair. Similar to the way that you use the push command, you can pull the value into several devices with a single call. +;;; Pull +;; You’ve already seen how to pull a single key-value pair. Similar to +;; the way that you use the push command, you can pull the value into +;; several devices with a single call. (def b [(ndarray/ones shape {:ctx (context/cpu 0)}) (ndarray/ones shape {:ctx (context/cpu 1)})]) (kvstore/pull kv ["3" "3"] b) (map ndarray/->vec b) ;=> ([3.0 3.0 3.0 3.0 3.0 3.0] [3.0 3.0 3.0 3.0 3.0 3.0]) -;;List Key-Value Pairs -;;All of the operations that we’ve discussed so far are performed on a single key. KVStore also provides the interface for generating a list of key-value pairs. For a single device, use the following: + +;;;; List Key-Value Pairs + +;; All of the operations that we’ve discussed so far are performed on +;; a single key. KVStore also provides the interface for generating a +;; list of key-value pairs. For a single device, use the following: (def ks ["5" "7" "9"]) -(kvstore/init kv ks [(ndarray/ones shape) (ndarray/ones shape) (ndarray/ones shape)]) -(kvstore/push kv ks [(ndarray/ones shape) (ndarray/ones shape) (ndarray/ones shape)]) -(def b [(ndarray/zeros shape) (ndarray/zeros shape) (ndarray/zeros shape)]) +(kvstore/init kv ks [(ndarray/ones shape) + (ndarray/ones shape) + (ndarray/ones shape)]) +(kvstore/push kv ks [(ndarray/ones shape) + (ndarray/ones shape) + (ndarray/ones shape)]) +(def b [(ndarray/zeros shape) + (ndarray/zeros shape) + (ndarray/zeros shape)]) (kvstore/pull kv ks b) -(map ndarray/->vec b);=> ([1.0 1.0 1.0 1.0 1.0 1.0] [1.0 1.0 1.0 1.0 1.0 1.0] [1.0 1.0 1.0 1.0 1.0 1.0]) +(map ndarray/->vec b) ;=> ([1.0 1.0 1.0 1.0 1.0 1.0] [1.0 1.0 1.0 1.0 1.0 1.0] [1.0 1.0 1.0 1.0 1.0 1.0]) diff --git a/contrib/clojure-package/examples/tutorial/src/tutorial/module.clj b/contrib/clojure-package/examples/tutorial/src/tutorial/module.clj index 3cef342f0..e19498111 100644 --- a/contrib/clojure-package/examples/tutorial/src/tutorial/module.clj +++ b/contrib/clojure-package/examples/tutorial/src/tutorial/module.clj @@ -16,6 +16,8 @@ ;; (ns tutorial.module + "A REPL tutorial of the MXNet Clojure API for Module, based on + https://mxnet.incubator.apache.org/api/clojure/module.html" (:require [clojure.java.io :as io] [clojure.java.shell :refer [sh]] [org.apache.clojure-mxnet.eval-metric :as eval-metric] @@ -24,12 +26,26 @@ [org.apache.clojure-mxnet.symbol :as sym] [org.apache.clojure-mxnet.ndarray :as ndarray])) + +;; The Module API provides an intermediate and high-level interface +;; for performing computation with neural networks in MXNet. Module +;; wraps a Symbol and one or more Executors. It has both a high level +;; and intermediate level API. + + +;;;; Prepare the Data + +;; In this example, we are going to use the MNIST data set. If you +;; start, we can run some helper scripts to download the data for us. + (def data-dir "data/") (when-not (.exists (io/file (str data-dir "train-images-idx3-ubyte"))) (sh "../../scripts/get_mnist_data.sh")) -;;; Load the MNIST datasets +;; MXNet provides function in the `io` namespace to load the MNIST +;; datasets into training and test data iterators that we can use with +;; our module. (def train-data (mx-io/mnist-iter {:image (str data-dir "train-images-idx3-ubyte") :label (str data-dir "train-labels-idx1-ubyte") :label-name "softmax_label" @@ -47,11 +63,13 @@ :flat true :silent false})) -;; The module API provides an intermediate and high-level interface for performing computation with neural networks in MXNet. Module wraps a Symbol and one or more Executors. It has both a high level and intermediate level api -;; Preparing a module for Computation +;;;; Preparing a module for Computation -;; construct a module +;; To construct a module, we need to have a symbol as input. This +;; symbol takes input data in the first layer and then has subsequent +;; layers of fully connected and relu activation layers, ending up in +;; a softmax layer for output. (let [data (sym/variable "data") fc1 (sym/fully-connected "fc1" {:data data :num-hidden 128}) @@ -62,7 +80,7 @@ out (sym/softmax-output "softmax" {:data fc3})] out) ;=>#object[org.apache.mxnet.Symbol 0x1f43a406 "org.apache.mxnet.Symbol@1f43a406"] -;; You can also use as-> for easier threading +;; You can also write this with the `as->` threading macro. (def out (as-> (sym/variable "data") data @@ -75,10 +93,15 @@ ;=> #'tutorial.module/out -;; By default, context is the CPU. If you need data parallelization, you can specify a GPU context or an array of GPU contexts. -;; like this (m/module out {:contexts [(context/gpu)]}) +;; By default, context is the CPU. If you need data parallelization, +;; you can specify a GPU context or an array of GPU contexts, like +;; this: `(m/module out {:contexts [(context/gpu)]})` -;; Before you can compute with a module, you need to call `bind` to allocate the device memory and `initParams` or `set-params` to initialize the parameters. If you simply want to fit a module, you don’t need to call `bind` and `init-params` explicitly, because the `fit` function automatically calls them if they are needed. +;; Before you can compute with a module, you need to call `bind` to +;; allocate the device memory and `initParams` or `set-params` to +;; initialize the parameters. If you simply want to fit a module, you +;; don’t need to call `bind` and `init-params` explicitly, because the +;; `fit` function automatically calls them if they are needed. (let [mod (m/module out)] (-> mod @@ -86,29 +109,46 @@ :label-shapes (mx-io/provide-label train-data)}) (m/init-params))) -;; Now you can compute with the module using functions like `forward`, `backward`, etc. +;; Now you can compute with the module using functions like `forward`, +;; `backward`, etc. -;; Training, Predicting, and Evaluating +;;;; Training and Predicting -;;Modules provide high-level APIs for training, predicting, and evaluating. To fit a module, call the `fit` function with some DataIters: +;; Modules provide high-level APIs for training, predicting, and +;; evaluating. To fit a module, call the `fit` function with some data +;; iterators: -(def mod (m/fit (m/module out) {:train-data train-data :eval-data test-data :num-epoch 1})) +(def mod + (m/fit (m/module out) {:train-data train-data + :eval-data test-data + :num-epoch 1})) +;; => ;; Epoch 0 Train- [accuracy 0.12521666] ;; Epoch 0 Time cost- 8392 ;; Epoch 0 Validation- [accuracy 0.2227] -;; You can pass in batch-end callbacks using batch-end-callback and epoch-end callbacks using epoch-end-callback in the `fit-params`. You can also set parameters using functions like in the fit-params like optimizer and eval-metric. To learn more about the fit-params, see the fit-param function options. To predict with a module, call `predict` with a DataIter: +;; You can pass in batch-end callbacks using batch-end-callback and +;; epoch-end callbacks using epoch-end-callback in the +;; `fit-params`. You can also set parameters using functions like in +;; the fit-params like optimizer and eval-metric. To learn more about +;; the fit-params, see the fit-param function options. To predict with +;; a module, call `predict` with a DataIter: + +(def results + (m/predict mod {:eval-data test-data})) -(def results (m/predict mod {:eval-data test-data})) (first results) ;=>#object[org.apache.mxnet.NDArray 0x3540b6d3 "org.apache.mxnet.NDArray@a48686ec"] (first (ndarray/->vec (first results))) ;=>0.08261358 -;;The module collects and returns all of the prediction results. For more details about the format of the return values, see the documentation for the `predict` function. +;; The module collects and returns all of the prediction results. For +;; more details about the format of the return values, see the +;; documentation for the `predict` function. -;;When prediction results might be too large to fit in memory, use the `predict-every-batch` API +;; When prediction results might be too large to fit in memory, use +;; the `predict-every-batch` API. (let [preds (m/predict-every-batch mod {:eval-data test-data})] (mx-io/reduce-batches test-data @@ -118,23 +158,33 @@ ;;; do something (inc i)))) -;;If you need to evaluate on a test set and don’t need the prediction output, call the `score` function with a DataIter and an EvalMetric: +;; If you need to evaluate on a test set and don’t need the prediction +;; output, call the `score` function with a data iterator and an eval +;; metric: -(m/score mod {:eval-data test-data :eval-metric (eval-metric/accuracy)}) ;=>["accuracy" 0.2227] +(m/score mod {:eval-data test-data + :eval-metric (eval-metric/accuracy)}) ;=>["accuracy" 0.2227] -;;This runs predictions on each batch in the provided DataIter and computes the evaluation score using the provided EvalMetric. The evaluation results are stored in metric so that you can query later. +;; This runs predictions on each batch in the provided DataIter and +;; computes the evaluation score using the provided EvalMetric. The +;; evaluation results are stored in metric so that you can query +;; later. -;;Saving and Loading Module Parameters -;;To save the module parameters in each training epoch, use a `checkpoint` function +;;;; Saving and Loading + +;; To save the module parameters in each training epoch, use the +;; `save-checkpoint` function: (let [save-prefix "my-model"] (doseq [epoch-num (range 3)] (mx-io/do-batches train-data (fn [batch ;; do something -])) - (m/save-checkpoint mod {:prefix save-prefix :epoch epoch-num :save-opt-states true}))) + ])) + (m/save-checkpoint mod {:prefix save-prefix + :epoch epoch-num + :save-opt-states true}))) ;; INFO org.apache.mxnet.module.Module: Saved checkpoint to my-model-0000.params ;; INFO org.apache.mxnet.module.Module: Saved optimizer state to my-model-0000.states @@ -144,20 +194,22 @@ ;; INFO org.apache.mxnet.module.Module: Saved optimizer state to my-model-0002.states -;;To load the saved module parameters, call the `load-checkpoint` function: +;; To load the saved module parameters, call the `load-checkpoint` +;; function: (def new-mod (m/load-checkpoint {:prefix "my-model" :epoch 1 :load-optimizer-states true})) new-mod ;=> #object[org.apache.mxnet.module.Module 0x5304d0f4 "org.apache.mxnet.module.Module@5304d0f4"] -;;To initialize parameters, Bind the symbols to construct executors first with bind function. Then, initialize the parameters and auxiliary states by calling `init-params` function. - +;; To initialize parameters, bind the symbols to construct executors +;; first with the `bind` function. Then, initialize the parameters and +;; auxiliary states by calling the `init-params` function.\ (-> new-mod - (m/bind {:data-shapes (mx-io/provide-data train-data) :label-shapes (mx-io/provide-label train-data)}) + (m/bind {:data-shapes (mx-io/provide-data train-data) + :label-shapes (mx-io/provide-label train-data)}) (m/init-params)) -;;To get current parameters, use `params` - +;; To get current parameters, use `params` (let [[arg-params aux-params] (m/params new-mod)] {:arg-params arg-params :aux-params aux-params}) @@ -178,20 +230,57 @@ new-mod ;=> #object[org.apache.mxnet.module.Module 0x5304d0f4 "org.apache.mxnet. ;; :aux-params {}} -;;To assign parameter and aux state values, use `set-params` function. +;; To assign parameter and aux state values, use the `set-params` +;; function: +(m/set-params new-mod {:arg-params (m/arg-params new-mod) + :aux-params (m/aux-params new-mod)}) -(m/set-params new-mod {:arg-params (m/arg-params new-mod) :aux-params (m/aux-params new-mod)}) -;=> #object[org.apache.mxnet.module.Module 0x5304d0f4 "org.apache.mxnet.module.Module@5304d0f4"] -;;To resume training from a saved checkpoint, instead of calling `set-params`, directly call `fit`, passing the loaded parameters, so that `fit` knows to start from those parameters instead of initializing randomly: +;; To resume training from a saved checkpoint, pass the loaded +;; parameters to the `fit` function. This will prevent `fit` from +;; initializing randomly. -;; reset the training data before calling fit or you will get an error +;; (First, reset the training data before calling `fit` or you will +;; get an error) (mx-io/reset train-data) (mx-io/reset test-data) -(m/fit new-mod {:train-data train-data :eval-data test-data :num-epoch 2 - :fit-params (-> (m/fit-params {:begin-epoch 1}))}) - -;;Create fit-params, and then use it to set `begin-epoch` so that fit() knows to resume from a saved epoch. - - +;; Create `fit-params` and then use it to set `begin-epoch` so that +;; `fit` knows to resume from a saved epoch. + + +(comment +;; FIXME +; Caused by: java.io.EOFException +; at java.io.DataInputStream.readInt(DataInputStream.java:392) +; at java.io.ObjectInputStream$BlockDataInputStream.readInt(ObjectInputStream.java:3182) +; at java.io.ObjectInputStream.readInt(ObjectInputStream.java:1032) +; at org.apache.mxnet.Optimizer$$anon$1$$anonfun$deserializeState$1.apply$mcVI$sp(Optimizer.scala:84) +; at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) +; at org.apache.mxnet.Optimizer$$anon$1.deserializeState(Optimizer.scala:83) +; at org.apache.mxnet.module.Module$$anonfun$loadOptimizerStates$3.apply(Module.scala:594) +; at org.apache.mxnet.module.Module$$anonfun$loadOptimizerStates$3.apply(Module.scala:589) +; at scala.Option.foreach(Option.scala:257) +; at org.apache.mxnet.module.Module.loadOptimizerStates(Module.scala:589) +; at org.apache.mxnet.module.Module$$anonfun$initOptimizer$4.apply(Module.scala:407) +; at org.apache.mxnet.module.Module$$anonfun$initOptimizer$4.apply(Module.scala:406) +; at scala.Option.foreach(Option.scala:257) +; at org.apache.mxnet.module.Module.initOptimizer(Module.scala:406) +; at org.apache.mxnet.module.BaseModule.fit(BaseModule.scala:407) +; at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) +; at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) +; at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) +; at java.lang.reflect.Method.invoke(Method.java:498) +; at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) +; at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) +; at org.apache.clojure_mxnet.module$fit.invokeStatic(module.clj:551) +; at org.apache.clojure_mxnet.module$fit.invoke(module.clj:538) +; at tutorial.module$eval1787.invokeStatic(module.clj:250) +; at tutorial.module$eval1787.invoke(module.clj:250) + +(m/fit new-mod {:train-data train-data + :eval-data test-data + :num-epoch 2 + :fit-params (m/fit-params {:begin-epoch 1})}) + +) \ No newline at end of file diff --git a/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj b/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj index 858316eef..d18bb53da 100644 --- a/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj +++ b/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj @@ -16,42 +16,53 @@ ;; (ns tutorial.ndarray + "A REPL tutorial of the MXNet Clojure API for NDArray, based on + https://mxnet.incubator.apache.org/api/clojure/ndarray.html" (:require [org.apache.clojure-mxnet.ndarray :as ndarray] [org.apache.clojure-mxnet.context :as context])) -;;The NDArray package (mxnet.ndarray) contains tensor operations similar to numpy.ndarray. The syntax is also similar, except for some additional calls for dealing with I/O and multiple devices. +;; The NDArray API contains tensor operations similar to +;; `numpy.ndarray`. The syntax is also similar, except for some +;; additional calls for dealing with I/O and multiple devices. -;;Create NDArray -;;Create mxnet.ndarray as follows: -(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50 -(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension -(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3 +;;;; Create NDArray -;;; There are also ways to convert to a vec or get the shape as an object or vec +;; Create an MXNet NDArray as follows: +(def a (ndarray/zeros [100 50])) ; all-zero array of dimension 100 x 50 +(def b (ndarray/ones [256 32 128 1])) ; all-one array of given dimensions +(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ; array with given contents in shape 2 x 3 + +;;; There are also ways to convert an NDArray to a vec or to get the +;;; shape as an object or vec: (ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0] (ndarray/shape c) ;=> #object[org.apache.mxnet.Shape 0x583c865 "(2,3)"] (ndarray/shape-vec c) ;=> [2 3] -;; NDArray Operations -;; Arithmtic Operations +;; There are some basic NDArray operations, like arithmetic and slice +;; operations. + + +;;;; NDArray Operations: Arithmetic + (def a (ndarray/ones [1 5])) (def b (ndarray/ones [1 5])) -(-> (ndarray/+ a b) (ndarray/->vec)) ;=> [2.0 2.0 2.0 2.0 2.0] +(ndarray/->vec (ndarray/+ a b)) ;=> [2.0 2.0 2.0 2.0 2.0] ;; original ndarrays are unchanged (ndarray/->vec a) ;=> [1.0 1.0 1.0 1.0 1.0] (ndarray/->vec b) ;=> [1.0 1.0 1.0 1.0 1.0] -;;inplace operators +;; inplace operators (ndarray/+= a b) (ndarray/->vec a) ;=> [2.0 2.0 2.0 2.0 2.0] -;; other arthimetic operations are similar +;; Other arthimetic operations are similar. + -;; Slice operations +;;;; NDArray Operations: Slice (def a (ndarray/array [1 2 3 4 5 6] [3 2])) (def a1 (ndarray/slice a 1)) @@ -62,7 +73,8 @@ (ndarray/shape-vec a2) ;=>[2 2] (ndarray/->vec a2) ;=> [3.0 4.0 5.0 6.0] -;; Dot Product + +;;;; NDArray Operations: Dot Product (def arr1 (ndarray/array [1 2] [1 2])) (def arr2 (ndarray/array [3 4] [2 1])) @@ -70,23 +82,42 @@ (ndarray/shape-vec res) ;=> [1 1] (ndarray/->vec res) ;=> [11.0] -;;Save and Load NDArray -;;You can use MXNet functions to save and load a map of NDArrays from file systems, as follows: + +;;;; Save and Load NDArray + +;; You can use MXNet functions to save and load a map of NDArrays from +;; file systems, as follows: (ndarray/save "filename" {"arr1" arr1 "arr2" arr2}) -;; you can also do "s3://path" or "hdfs" +;; (you can also do "s3://path" or "hdfs") + +;; (ndarray/save "/Users/daveliepmann/src/coursework/mxnet-clj-tutorials/abc" +;; {"arr1" arr1 "arr2" arr2}) -;; to load +;; To load: (def from-file (ndarray/load "filename")) + from-file ;=>{"arr1" #object[org.apache.mxnet.NDArray 0x6115ba61 "org.apache.mxnet.NDArray@43d85753"], "arr2" #object[org.apache.mxnet.NDArray 0x374b5eff "org.apache.mxnet.NDArray@5c93def4"]} -;;Multi-Device Support +;; The good thing about using the `save` and `load` interface is that +;; you can use the format across all `mxnet` language bindings. They +;; also already support Amazon S3 and HDFS. + + +;;;; Multi-Device Support -;;Device information is stored in the mxnet.Context structure. When creating NDArray in MXNet, you can use the context argument (the default is the CPU context) to create arrays on specific devices as follows: +;; Device information is stored in the `mxnet.Context` structure. When +;; creating NDArray in MXNet, you can use the context argument (the +;; default is the CPU context) to create arrays on specific devices as +;; follows: (def cpu-a (ndarray/zeros [100 200])) (ndarray/context cpu-a) ;=> #object[org.apache.mxnet.Context 0x3f376123 "cpu(0)"] -(def gpu-b (ndarray/zeros [100 200] {:ctx (context/gpu 0)})) ;; to use with gpu +(comment + (def gpu-b (ndarray/zeros [100 200] {:ctx (context/gpu 0)})) ;; to use with gpu +) -;;Currently, we do not allow operations among arrays from different contexts. To manually enable this, use the copyto function to copy the content to different devices, and continue computation: +;; Currently, we do not allow operations among arrays from different +;; contexts. To manually enable this, use the `copy-to` function to +;; copy the content to different devices, and continue computation. diff --git a/contrib/clojure-package/examples/tutorial/src/tutorial/symbol.clj b/contrib/clojure-package/examples/tutorial/src/tutorial/symbol.clj index bec71dee8..e88260069 100644 --- a/contrib/clojure-package/examples/tutorial/src/tutorial/symbol.clj +++ b/contrib/clojure-package/examples/tutorial/src/tutorial/symbol.clj @@ -16,79 +16,66 @@ ;; (ns tutorial.symbol + "A REPL tutorial of the MXNet Clojure Symbolic API, based on + https://mxnet.incubator.apache.org/api/clojure/symbol.html" (:require [org.apache.clojure-mxnet.executor :as executor] [org.apache.clojure-mxnet.ndarray :as ndarray] [org.apache.clojure-mxnet.symbol :as sym] [org.apache.clojure-mxnet.context :as context])) -;; How to compose symbols -;;The symbolic API provides a way to configure computation graphs. You can configure the graphs either at the level of neural network layer operations or as fine-grained operations. -;;The following example configures a two-layer neural network. +;;;; How to Compose Symbols +;; The symbolic API provides a way to configure computation +;; graphs. You can configure the graphs either at the level of neural +;; network layer operations or as fine-grained operations. + +;; The following example configures a two-layer neural network. (def data (sym/variable "data")) (def fc1 (sym/fully-connected "fc1" {:data data :num-hidden 128})) (def act1 (sym/activation "act1" {:data fc1 :act-type "relu"})) (def fc2 (sym/fully-connected "fc2" {:data act1 :num-hidden 64})) (def net (sym/softmax-output "out" {:data fc2})) -;; you could also combine this more dynamically with +;; This can also be combined more dynamically with the `as->` Clojure +;; threading form. (as-> (sym/variable "data") data (sym/fully-connected "fc1" {:data data :num-hidden 128}) - (sym/activation "act1" {:data data :act-type "relu"}) + (sym/activation "act1" {:data data :act-type "relu"}) (sym/fully-connected "fc2" {:data data :num-hidden 64}) - (sym/softmax-output "out" {:data data})) + (sym/softmax-output "out" {:data data})) net ;=> #object[org.apache.mxnet.Symbol 0x5c78c8c2 "org.apache.mxnet.Symbol@5c78c8c2"] - -;;The basic arithmetic operators (plus, minus, div, multiplication) - -;;The following example creates a computation graph that adds two inputs together. +;; The basic arithmetic operators (plus, minus, div, multiplication) +;; work as expected. The following example creates a computation graph +;; that adds two inputs together. (def a (sym/variable "a")) (def b (sym/variable "b")) (def c (sym/+ a b)) -;; Each symbol takes a (unique) string name. NDArray and Symbol both represent a single tensor. Operators represent the computation between tensors. Operators take symbol (or NDArray) as inputs and might also additionally accept other hyperparameters such as the number of hidden neurons (num_hidden) or the activation type (act_type) and produce the output. - -;; We can view a symbol simply as a function taking several arguments. And we can retrieve those arguments with the following method call: - -;;We can view a symbol simply as a function taking several arguments. And we can retrieve those arguments with the following method call: - -(sym/list-arguments net) - ;=> ["data" "fc1_weight" "fc1_bias" "fc2_weight" "fc2_bias" "out_label"] - -;; These arguments are the parameters and inputs needed by each symbol: - -;; data: Input data needed by the variable data. -;; fc1_weight and fc1_bias: The weight and bias for the first fully connected layer fc1. -;; fc2_weight and fc2_bias: The weight and bias for the second fully connected layer fc2. -;; out_label: The label needed by the loss. - -;;We can also specify the names explicitly: -(def net (sym/variable "data")) -(def w (sym/variable "myweight")) -(def net (sym/fully-connected "fc1" {:data net :weight w :num-hidden 128})) -(sym/list-arguments net) - ;=> ["data" "fc1_weight" "fc1_bias" "fc2_weight" "fc2_bias" "out_label" "myweight" "fc1_bias"] +;;;; More Complicated Compositions - -;;In the above example, FullyConnected layer has 3 inputs: data, weight, bias. When any input is not specified, a variable will be automatically generated for it. - - -;; More complicated composition - -;;MXNet provides well-optimized symbols for layers commonly used in deep learning (see src/operator). We can also define new operators in Python. The following example first performs an element-wise add between two symbols, then feeds them to the fully connected operator: +;; MXNet provides well-optimized symbols for layers commonly used in +;; deep learning (see src/operator). We can also define new operators +;; in Python. The following example first performs an element-wise add +;; between two symbols, then feeds them to the fully connected +;; operator: (def lhs (sym/variable "data1")) (def rhs (sym/variable "data2")) -(def net (sym/fully-connected "fc1" {:data (sym/+ lhs rhs) :num-hidden 128})) +(def net (sym/fully-connected "fc1" {:data (sym/+ lhs rhs) + :num-hidden 128})) (sym/list-arguments net) ;=> ["data1" "data2" "fc1_weight" "fc1_bias"] -;; Group Multiple Symbols -;;To construct neural networks with multiple loss layers, we can use mxnet.sym.Group to group multiple symbols together. The following example groups two outputs: + +;;;; Group Multiple Symbols + +;; To construct neural networks with multiple loss layers, we can use +;; `group` to group multiple symbols together. The following example +;; groups two outputs: (def net (sym/variable "data")) (def fc1 (sym/fully-connected {:data net :num-hidden 128})) @@ -96,56 +83,51 @@ net ;=> #object[org.apache.mxnet.Symbol 0x5c78c8c2 "org.apache.mxnet.Symbol@5c78 (def out1 (sym/softmax-output {:data net2})) (def out2 (sym/linear-regression-output {:data net2})) (def group (sym/group [out1 out2])) -(sym/list-outputs group);=> ["softmaxoutput0_output" "linearregressionoutput0_output"] +(sym/list-outputs group) ;=> ["softmaxoutput0_output" "linearregressionoutput0_output"] -;; Symbol Manipulation -;; One important difference of Symbol compared to NDArray is that we first declare the computation and then bind the computation with data to run. +;;;; Serialization -;; In this section, we introduce the functions to manipulate a symbol directly. But note that, most of them are wrapped by the module package. +;; You can use the `save` and `load` functions to serialize Symbol +;; objects as JSON. These functions have the advantage of being +;; language-agnostic and cloud-friendly. You can also get a JSON +;; string directly using `to-json`. -;; Shape and Type Inference -;; For each symbol, we can query its arguments, auxiliary states and outputs. We can also infer the output shape and type of the symbol given the known input shape or type of some arguments, which facilitates memory allocation. -(sym/list-arguments fc1) ;=> ["data" "fullyconnected1_weight" "fullyconnected1_bias"] -(sym/list-outputs fc1) ;=> ["fullyconnected1_output"] +;; The following example shows how to save a symbol to a file, load it +;; back, and compare two symbols using a JSON string. You can also +;; save to S3 as well. -;; infer the shapes given the shape of the input arguments -(let [[arg-shapes out-shapes] (sym/infer-shape fc1 {:data [2 1]})] - {:arg-shapes arg-shapes - :out-shapes out-shapes}) ;=> {:arg-shapes ([2 1] [128 1] [128]), :out-shapes ([2 128])} +(def a (sym/variable "a")) +(def b (sym/variable "b")) +(def c (sym/+ a b)) +(sym/save c "symbol-c.json") +(def c2 (sym/load "symbol-c.json")) +(= (sym/to-json c) (sym/to-json c2)) ;=>true -;; Bind with Data and Evaluate -;; The symbol c constructed above declares what computation should be run. To evaluate it, we first need to feed the arguments, namely free variables, with data. -;; We can do it by using the bind method, which accepts device context and a dict mapping free variable names to NDArrays as arguments and returns an executor. The executor provides forward method for evaluation and an attribute outputs to get all the results. +;;;; Executing Symbols + +;; To execute symbols, first we need to define the data that they +;; should run on. We can do this with the `bind` function, which +;; returns an executor. We then use `forward` to evaluate and +;; `outputs` to get the results. (def a (sym/variable "a")) (def b (sym/variable "b")) (def c (sym/+ a b)) -(def ex (sym/bind c {"a" (ndarray/ones [2 2]) "b" (ndarray/ones [2 2])})) +(def ex + (sym/bind c {"a" (ndarray/ones [2 2]) + "b" (ndarray/ones [2 2])})) + (-> (executor/forward ex) (executor/outputs) (first) (ndarray/->vec));=> [2.0 2.0 2.0 2.0] -;;We can evaluate the same symbol on GPU with different data. -;; To do this you must have the correct native library jar defined as a dependency - -;;Note In order to execute the following section on a cpu set gpu_device to (cpu). - - -(def ex (sym/bind c (context/gpu 0) {"a" (ndarray/ones [2 2]) "b" (ndarray/ones [2 2])})) - -;; Serialization -;; There are two ways to save and load the symbols. You can use the mxnet.Symbol.save and mxnet.Symbol.load functions to serialize the Symbol objects. The advantage of using save and load functions is that it is language agnostic and cloud friendly. The symbol is saved in JSON format. You can also get a JSON string directly using mxnet.Symbol.toJson. Refer to API documentation for more details. - -;; The following example shows how to save a symbol to a file, load it back, and compare two symbols using a JSON string. You can also save to S3 as well - -(def a (sym/variable "a")) -(def b (sym/variable "b")) -(def c (sym/+ a b)) -(sym/save c "symbol-c.json") -(def c2 (sym/load "symbol-c.json")) -(= (sym/to-json c) (sym/to-json c2)) ;=>true - +(comment + ;; We can evaluate the same symbol on GPU with different data. + ;; (To do this you must have the correct native library jar defined as a dependency.) + (def ex (sym/bind c (context/gpu 0) {"a" (ndarray/ones [2 2]) + "b" (ndarray/ones [2 2])})) +) diff --git a/contrib/clojure-package/examples/tutorial/test/tutorial/core_test.clj b/contrib/clojure-package/examples/tutorial/test/tutorial/core_test.clj new file mode 100644 index 000000000..0e5169c5c --- /dev/null +++ b/contrib/clojure-package/examples/tutorial/test/tutorial/core_test.clj @@ -0,0 +1,27 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns tutorial.core_test + (:require [clojure.test :refer :all]) + (:require + [tutorial.introduction] + [tutorial.kvstore] + [tutorial.module] + [tutorial.ndarray] + [tutorial.symbol])) + +(deftest if-this-goes-here-then-tutorials-have-loaded-properly (is true)) \ No newline at end of file diff --git a/contrib/clojure-package/examples/visualization/project.clj b/contrib/clojure-package/examples/visualization/project.clj index 18882d4ff..d91ace318 100644 --- a/contrib/clojure-package/examples/visualization/project.clj +++ b/contrib/clojure-package/examples/visualization/project.clj @@ -19,5 +19,5 @@ :description "Visualization example" :plugins [[lein-cljfmt "0.5.7"]] :dependencies [[org.clojure/clojure "1.9.0"] - [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT"]] + [org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT"]] :main visualization.core) diff --git a/contrib/clojure-package/examples/visualization/test/visualization/core_test.clj b/contrib/clojure-package/examples/visualization/test/visualization/core_test.clj new file mode 100644 index 000000000..1b10695cb --- /dev/null +++ b/contrib/clojure-package/examples/visualization/test/visualization/core_test.clj @@ -0,0 +1,28 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns visualization.core_test + (:require + [visualization.core :as visualization] + [clojure.test :refer :all])) + +(deftest check-pdf + (visualization/test-viz) + (let [new-pdf (clojure.java.io/as-file "testviz.pdf")] + (is (.exists new-pdf)) + (is (> 10000 (- (System/currentTimeMillis) (.lastModified new-pdf)))))) + \ No newline at end of file diff --git a/contrib/clojure-package/integration-tests.sh b/contrib/clojure-package/integration-tests.sh new file mode 100755 index 000000000..6e5868712 --- /dev/null +++ b/contrib/clojure-package/integration-tests.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -evx + +MXNET_HOME=$(cd "$(dirname $0)/../.."; pwd) +EXAMPLES_HOME=${MXNET_HOME}/contrib/clojure-package/examples +#cd ${MXNET_HOME}/contrib/clojure-package +#lein test +#lein cloverage --codecov +for test_dir in `find ${EXAMPLES_HOME} -name test` ; do + cd ${test_dir} && lein test +done diff --git a/contrib/clojure-package/project.clj b/contrib/clojure-package/project.clj index 3779c82cf..c4428ce6e 100644 --- a/contrib/clojure-package/project.clj +++ b/contrib/clojure-package/project.clj @@ -15,8 +15,11 @@ ;; limitations under the License. ;; -(defproject org.apache.mxnet.contrib.clojure/clojure-mxnet "1.3.1-SNAPSHOT" +(defproject org.apache.mxnet.contrib.clojure/clojure-mxnet "1.5.0-SNAPSHOT" :description "Clojure package for MXNet" + :url "https://github.com/apache/incubator-mxnet" + :license {:name "Apache License" + :url "http://www.apache.org/licenses/LICENSE-2.0"} :dependencies [[org.clojure/clojure "1.9.0"] [t6/from-scala "0.3.0"] @@ -26,7 +29,7 @@ ;[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.2.1"] ;;; CI - [org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.3.1-SNAPSHOT"] + [org.apache.mxnet/mxnet-full_2.11 "INTERNAL"] [org.clojure/tools.logging "0.4.0"] [org.apache.logging.log4j/log4j-core "2.8.1"] diff --git a/contrib/clojure-package/scripts/infer/get_resnet_18_data.sh b/contrib/clojure-package/scripts/infer/get_resnet_18_data.sh new file mode 100755 index 000000000..601f362c4 --- /dev/null +++ b/contrib/clojure-package/scripts/infer/get_resnet_18_data.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -evx + +if [ ! -z "$MXNET_HOME" ]; then + data_path="$MXNET_HOME/data" +else + MXNET_ROOT=$(cd "$(dirname $0)/../.."; pwd) + data_path="$MXNET_ROOT/data" +fi + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +resnet_18_data_path="$data_path/resnet-18" +if [ ! -f "$resnet_18_data_path/resnet-18-0000.params" ]; then + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -P $resnet_18_data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -P $resnet_18_data_path + wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -P $resnet_18_data_path +fi diff --git a/contrib/clojure-package/scripts/infer/get_ssd_data.sh b/contrib/clojure-package/scripts/infer/get_ssd_data.sh new file mode 100755 index 000000000..96e27a12d --- /dev/null +++ b/contrib/clojure-package/scripts/infer/get_ssd_data.sh @@ -0,0 +1,39 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +set -evx + +if [ ! -z "$MXNET_HOME" ]; then + data_path="$MXNET_HOME/data" +else + MXNET_ROOT=$(cd "$(dirname $0)/../.."; pwd) + data_path="$MXNET_ROOT/data" +fi + +if [ ! -d "$data_path" ]; then + mkdir -p "$data_path" +fi + +resnet50_ssd_data_path="$data_path/resnet50_ssd" +if [ ! -f "$resnet50_ssd_data_path/resnet50_ssd_model-0000.params" ]; then + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $resnet50_ssd_data_path + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $resnet50_ssd_data_path + wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -P $resnet50_ssd_data_path +fi diff --git a/ci/docker/qemu/qemu_run.sh b/contrib/clojure-package/scripts/update_versions.sh similarity index 70% rename from ci/docker/qemu/qemu_run.sh rename to contrib/clojure-package/scripts/update_versions.sh index 53a6487aa..607e3f357 100755 --- a/ci/docker/qemu/qemu_run.sh +++ b/contrib/clojure-package/scripts/update_versions.sh @@ -17,15 +17,11 @@ # specific language governing permissions and limitations # under the License. -set -exuo pipefail +# Run this from the main Clojure project directory with 2 arguments +# old-version and new-version +# Ex: scripts/update_version 1.5.0-SNAPSHOT 1.5.0-SNAPSHOT -qemu-system-arm -M virt -m 1024 \ - -kernel vmlinuz \ - -initrd initrd.img \ - -append 'root=/dev/vda1' \ - -drive if=none,file=vda.qcow2,format=qcow2,id=hd \ - -device virtio-blk-device,drive=hd \ - -netdev user,id=mynet,hostfwd=tcp::2222-:22 \ - -device virtio-net-device,netdev=mynet \ - -nographic \ - -display none +set -evx +echo "Replacing $2 with $2 in the directory $PWD " +find ./ -type f -exec sed -i '' -e "s/$1/$2/g" {} \; +echo "Done! Check the changed files" diff --git a/contrib/clojure-package/src/dev/generator.clj b/contrib/clojure-package/src/dev/generator.clj index 1f0418951..ca93c3421 100644 --- a/contrib/clojure-package/src/dev/generator.clj +++ b/contrib/clojure-package/src/dev/generator.clj @@ -212,11 +212,11 @@ (.write w "\n")))) (def symbol-gen-ns "(ns org.apache.clojure-mxnet.symbol - (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten load max - min repeat reverse set sort take to-array empty sin - get apply shuffle]) - (:require [org.apache.clojure-mxnet.util :as util]) - (:import (org.apache.mxnet Symbol)))") + (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten load max + min repeat reverse set sort take to-array empty sin + get apply shuffle ref]) + (:require [org.apache.clojure-mxnet.util :as util]) + (:import (org.apache.mxnet Symbol)))") (defn generate-symbol-file [] @@ -307,7 +307,8 @@ (def ndarray-gen-ns "(ns org.apache.clojure-mxnet.ndarray (:refer-clojure :exclude [* - + > >= < <= / cast concat flatten identity load max - min repeat reverse set sort take to-array empty shuffle]) + min repeat reverse set sort take to-array empty shuffle + ref]) (:import (org.apache.mxnet NDArray Shape)))") diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj new file mode 100644 index 000000000..e2e87ed47 --- /dev/null +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/image.clj @@ -0,0 +1,139 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.image + (:require [t6.from-scala.core :refer [$ $$] :as $] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.util :as util] + [clojure.spec.alpha :as s]) + (:import (org.apache.mxnet Image NDArray) + (java.io InputStream))) + +;; Flags for conversion of images +(def GRAYSCALE 0) +(def COLOR 1) + +(s/def ::input-stream #(instance? InputStream %)) +(s/def ::color-flag #{GRAYSCALE COLOR}) +(s/def ::to-rgb boolean?) +(s/def ::ndarray #(instance? NDArray %)) +(s/def ::output (s/or :empty nil? :ndarray ::ndarray)) +(s/def ::decode-image-opts + (s/keys :opt-un [::color-flag ::to-rgb ::output])) + +(defn decode-image + "Decodes an image from an input stream" + ([input-stream {:keys [color-flag to-rgb output] + :or {color-flag COLOR to-rgb true output nil} + :as opts}] + (util/validate! ::input-stream input-stream "Invalid input stream") + (util/validate! ::decode-image-opts opts "Invalid options for decoding") + (Image/imDecode input-stream color-flag to-rgb ($/option output))) + ([input-stream] + (decode-image input-stream {}))) + +(s/def ::filename string?) +(s/def ::optional-color-flag + (s/or :none nil? :some ::color-flag)) +(s/def ::optional-to-rgb + (s/or :none nil? :some ::to-rgb)) + +(defn read-image + "Reads an image file and returns an ndarray" + ([filename {:keys [color-flag to-rgb output] + :or {color-flag nil to-rgb nil output nil} + :as opts}] + (util/validate! ::filename filename "Invalid filename") + (util/validate! ::optional-color-flag color-flag "Invalid color flag") + (util/validate! ::optional-to-rgb to-rgb "Invalid conversion flag") + (util/validate! ::output output "Invalid output") + (Image/imRead + filename + ($/option color-flag) + ($/option to-rgb) + ($/option output))) + ([filename] + (read-image filename {}))) + +(s/def ::int int?) +(s/def ::optional-int (s/or :none nil? :some int?)) + +(defn resize-image + "Resizes the image array to (width, height)" + ([input w h {:keys [interpolation output] + :or {interpolation nil output nil} + :as opts}] + (util/validate! ::ndarray input "Invalid input array") + (util/validate! ::int w "Invalid width") + (util/validate! ::int h "Invalid height") + (util/validate! ::optional-int interpolation "Invalid interpolation") + (util/validate! ::output output "Invalid output") + (Image/imResize input w h ($/option interpolation) ($/option output))) + ([input w h] + (resize-image input w h {}))) + +(defn apply-border + "Pad image border" + ([input top bottom left right + {:keys [fill-type value values output] + :or {fill-type nil value nil values nil output nil} + :as opts}] + (util/validate! ::ndarray input "Invalid input array") + (util/validate! ::int top "Invalid top margin") + (util/validate! ::int bottom "Invalid bottom margin") + (util/validate! ::int left "Invalid left margin") + (util/validate! ::int right "Invalid right margin") + (util/validate! ::optional-int fill-type "Invalid fill type") + (util/validate! ::output output "Invalid output") + (Image/copyMakeBorder input top bottom left right + ($/option fill-type) + ($/option value) + ($/option values) + ($/option output))) + ([input top bottom left right] + (apply-border input top bottom left right {}))) + +(defn fixed-crop + "Return a fixed crop of the image" + [input x0 y0 w h] + (util/validate! ::ndarray input "Invalid input array") + (util/validate! ::int x0 "Invalid starting x coordinate") + (util/validate! ::int y0 "Invalid starting y coordinate") + (util/validate! ::int w "Invalid width") + (util/validate! ::int h "Invalid height") + (Image/fixedCrop input x0 y0 w h)) + +(defn rgb-array? + "Returns whether the ndarray is in the RGB format" + [input] + (util/validate! ::ndarray input "Invalid input array") + (let [shape (ndarray/shape-vec input)] + (and + (= 3 (count shape)) + (= 3 (shape 2))))) + +(s/def ::all-bytes #(= dtype/UINT8 (ndarray/dtype %))) +(s/def ::rgb-array rgb-array?) +(s/def ::to-image-ndarray + (s/and ::ndarray ::all-bytes ::rgb-array)) + +(defn to-image + "Convert a NDArray image in RGB format to a real image" + [input] + (util/validate! ::to-image-ndarray input "Invalid input array") + (Image/toImage input)) diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/infer.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/infer.clj new file mode 100644 index 000000000..09edf15b4 --- /dev/null +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/infer.clj @@ -0,0 +1,372 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.infer + (:refer-clojure :exclude [type]) + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.io :as mx-io] + [org.apache.clojure-mxnet.shape :as shape] + [org.apache.clojure-mxnet.util :as util] + [clojure.spec.alpha :as s] + [org.apache.clojure-mxnet.shape :as mx-shape]) + (:import (java.awt.image BufferedImage) + (org.apache.mxnet NDArray) + (org.apache.mxnet.infer Classifier ImageClassifier + ObjectDetector Predictor))) + +(s/def ::predictor #(instance? Predictor %)) +(s/def ::classifier #(instance? Classifier %)) +(s/def ::image-classifier #(instance? ImageClassifier %)) +(s/def ::object-detector #(instance? ObjectDetector %)) + +(defrecord WrappedPredictor [predictor]) +(defrecord WrappedClassifier [classifier]) +(defrecord WrappedImageClassifier [image-classifier]) +(defrecord WrappedObjectDetector [object-detector]) + +(s/def ::ndarray #(instance? NDArray %)) +(s/def ::number-array (s/coll-of number? :kind vector?)) +(s/def ::vvec-of-numbers (s/coll-of ::number-array :kind vector?)) +(s/def ::vec-of-ndarrays (s/coll-of ::ndarray :kind vector?)) +(s/def ::image #(instance? BufferedImage %)) +(s/def ::batch-images (s/coll-of ::image :kind vector?)) + +(s/def ::wrapped-predictor (s/keys :req-un [::predictor])) +(s/def ::wrapped-classifier (s/keys :req-un [::classifier])) +(s/def ::wrapped-image-classifier (s/keys :req-un [::image-classifier])) +(s/def ::wrapped-detector (s/keys :req-un [::object-detector])) + +(defn- format-detection-predictions [predictions] + (mapv (fn [[c p]] + (let [[prob xmin ymin xmax ymax] (mapv float p)] + {:class c :prob prob :x-min xmin :y-min ymin :x-max xmax :y-max ymax})) + predictions)) + +(defn- format-classification-predictions [predictions] + (mapv (fn [[c p]] {:class c :prob p}) predictions)) + +(defprotocol APredictor + (predict [wrapped-predictor inputs]) + (predict-with-ndarray [wrapped-predictor input-arrays])) + +(defprotocol AClassifier + (classify + [wrapped-classifier inputs] + [wrapped-classifier inputs topk]) + (classify-with-ndarray + [wrapped-classifier inputs] + [wrapped-classifier inputs topk])) + +(defprotocol AImageClassifier + (classify-image + [wrapped-image-classifier image] + [wrapped-image-classifier image topk] + [wrapped-image-classifier image topk dtype]) + (classify-image-batch + [wrapped-image-classifier images] + [wrapped-image-classifier images topk] + [wrapped-image-classifier images topk dtype])) + +(defprotocol AObjectDetector + (detect-objects + [wrapped-detector image] + [wrapped-detector image topk]) + (detect-objects-batch + [wrapped-detector images] + [wrapped-detector images topk]) + (detect-objects-with-ndarrays + [wrapped-detector input-arrays] + [wrapped-detector input-arrays topk])) + +(extend-protocol APredictor + WrappedPredictor + (predict + [wrapped-predictor inputs] + (util/validate! ::wrapped-predictor wrapped-predictor + "Invalid predictor") + (util/validate! ::vvec-of-numbers inputs + "Invalid inputs") + (->> (.predict (:predictor wrapped-predictor) + (util/vec->indexed-seq (mapv float-array inputs))) + (util/coerce-return-recursive) + (mapv #(mapv float %)))) + (predict-with-ndarray [wrapped-predictor input-arrays] + (util/validate! ::wrapped-predictor wrapped-predictor + "Invalid predictor") + (util/validate! ::vec-of-ndarrays input-arrays + "Invalid input arrays") + (-> (.predictWithNDArray (:predictor wrapped-predictor) + (util/vec->indexed-seq input-arrays)) + (util/coerce-return-recursive)))) + +(s/def ::nil-or-int (s/nilable int?)) + +(extend-protocol AClassifier + WrappedClassifier + (classify + ([wrapped-classifier inputs] + (classify wrapped-classifier inputs nil)) + ([wrapped-classifier inputs topk] + (util/validate! ::wrapped-classifier wrapped-classifier + "Invalid classifier") + (util/validate! ::vvec-of-numbers inputs + "Invalid inputs") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.classify (:classifier wrapped-classifier) + (util/vec->indexed-seq (mapv float-array inputs)) + (util/->int-option topk)) + (util/coerce-return-recursive) + (format-classification-predictions)))) + (classify-with-ndarray + ([wrapped-classifier inputs] + (classify-with-ndarray wrapped-classifier inputs nil)) + ([wrapped-classifier inputs topk] + (util/validate! ::wrapped-classifier wrapped-classifier + "Invalid classifier") + (util/validate! ::vec-of-ndarrays inputs + "Invalid inputs") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.classifyWithNDArray (:classifier wrapped-classifier) + (util/vec->indexed-seq inputs) + (util/->int-option topk)) + (util/coerce-return-recursive) + (mapv format-classification-predictions)))) + WrappedImageClassifier + (classify + ([wrapped-image-classifier inputs] + (classify wrapped-image-classifier inputs nil)) + ([wrapped-image-classifier inputs topk] + (util/validate! ::wrapped-image-classifier wrapped-image-classifier + "Invalid classifier") + (util/validate! ::vvec-of-numbers inputs + "Invalid inputs") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.classify (:image-classifier wrapped-image-classifier) + (util/vec->indexed-seq (mapv float-array inputs)) + (util/->int-option topk)) + (util/coerce-return-recursive) + (format-classification-predictions)))) + (classify-with-ndarray + ([wrapped-image-classifier inputs] + (classify-with-ndarray wrapped-image-classifier inputs nil)) + ([wrapped-image-classifier inputs topk] + (util/validate! ::wrapped-image-classifier wrapped-image-classifier + "Invalid classifier") + (util/validate! ::vec-of-ndarrays inputs + "Invalid inputs") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.classifyWithNDArray (:image-classifier wrapped-image-classifier) + (util/vec->indexed-seq inputs) + (util/->int-option topk)) + (util/coerce-return-recursive) + (mapv format-classification-predictions))))) + +(s/def ::image #(instance? BufferedImage %)) +(s/def ::dtype #{dtype/UINT8 dtype/INT32 dtype/FLOAT16 dtype/FLOAT32 dtype/FLOAT64}) + +(extend-protocol AImageClassifier + WrappedImageClassifier + (classify-image + ([wrapped-image-classifier image] + (classify-image wrapped-image-classifier image nil dtype/FLOAT32)) + ([wrapped-image-classifier image topk] + (classify-image wrapped-image-classifier image topk dtype/FLOAT32)) + ([wrapped-image-classifier image topk dtype] + (util/validate! ::wrapped-image-classifier wrapped-image-classifier + "Invalid classifier") + (util/validate! ::image image "Invalid image") + (util/validate! ::nil-or-int topk "Invalid top-K") + (util/validate! ::dtype dtype "Invalid dtype") + (->> (.classifyImage (:image-classifier wrapped-image-classifier) + image + (util/->int-option topk) + dtype) + (util/coerce-return-recursive) + (mapv format-classification-predictions)))) + (classify-image-batch + ([wrapped-image-classifier images] + (classify-image-batch wrapped-image-classifier images nil dtype/FLOAT32)) + ([wrapped-image-classifier images topk] + (classify-image-batch wrapped-image-classifier images topk dtype/FLOAT32)) + ([wrapped-image-classifier images topk dtype] + (util/validate! ::wrapped-image-classifier wrapped-image-classifier + "Invalid classifier") + (util/validate! ::batch-images images "Invalid Batch Images") + (util/validate! ::nil-or-int topk "Invalid top-K") + (util/validate! ::dtype dtype "Invalid dtype") + (->> (.classifyImageBatch (:image-classifier wrapped-image-classifier) + (util/vec->indexed-seq images) + (util/->int-option topk) + dtype) + (util/coerce-return-recursive) + (mapv format-classification-predictions))))) + +(extend-protocol AObjectDetector + WrappedObjectDetector + (detect-objects + ([wrapped-detector image] + (detect-objects wrapped-detector image nil)) + ([wrapped-detector image topk] + (util/validate! ::wrapped-detector wrapped-detector + "Invalid object detector") + (util/validate! ::image image "Invalid image") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.imageObjectDetect (:object-detector wrapped-detector) + image + (util/->int-option topk)) + (util/coerce-return-recursive) + (mapv format-detection-predictions)))) + (detect-objects-batch + ([wrapped-detector images] + (detect-objects-batch wrapped-detector images nil)) + ([wrapped-detector images topk] + (util/validate! ::wrapped-detector wrapped-detector + "Invalid object detector") + (util/validate! ::nil-or-int topk "Invalid top-K") + (util/validate! ::batch-images images "Invalid Batch Images") + (->> (.imageBatchObjectDetect (:object-detector wrapped-detector) + (util/vec->indexed-seq images) + (util/->int-option topk)) + (util/coerce-return-recursive) + (mapv format-detection-predictions)))) + (detect-objects-with-ndarrays + ([wrapped-detector input-arrays] + (detect-objects-with-ndarrays wrapped-detector input-arrays nil)) + ([wrapped-detector input-arrays topk] + (util/validate! ::wrapped-detector wrapped-detector + "Invalid object detector") + (util/validate! ::vec-of-ndarrays input-arrays + "Invalid inputs") + (util/validate! ::nil-or-int topk "Invalid top-K") + (->> (.objectDetectWithNDArray (:object-detector wrapped-detector) + (util/vec->indexed-seq input-arrays) + (util/->int-option topk)) + (util/coerce-return-recursive) + (mapv format-detection-predictions))))) + +(defprotocol AInferenceFactory + (create-predictor [factory] [factory opts]) + (create-classifier [factory] [factory opts]) + (create-image-classifier [factory] [factory opts]) + (create-object-detector [factory] [factory opts])) + +(defn convert-descriptors + [descriptors] + (util/vec->indexed-seq + (into [] (map mx-io/data-desc descriptors)))) + +(defrecord InferenceFactory [model-path-prefix input-descriptors] + AInferenceFactory + (create-predictor + [factory] + (create-predictor factory {})) + (create-predictor + [factory opts] + (let [{:keys [contexts epoch] + :or {contexts [(context/cpu)] epoch 0}} opts] + (->WrappedPredictor + (new Predictor + model-path-prefix + (convert-descriptors input-descriptors) + (into-array contexts) + (util/->int-option epoch))))) + (create-classifier + [factory] + (create-classifier factory {})) + (create-classifier + [factory opts] + (let [{:keys [contexts epoch] + :or {contexts [(context/cpu)] epoch 0}} opts] + (->WrappedClassifier + (new Classifier + model-path-prefix + (convert-descriptors input-descriptors) + (into-array contexts) + (util/->int-option epoch))))) + (create-image-classifier + [factory] + (create-image-classifier factory {})) + (create-image-classifier + [factory opts] + (let [{:keys [contexts epoch] + :or {contexts [(context/cpu)] epoch 0}} opts] + (->WrappedImageClassifier + (new ImageClassifier + model-path-prefix + (convert-descriptors input-descriptors) + (into-array contexts) + (util/->int-option epoch))))) + (create-object-detector + [factory] + (create-object-detector factory {})) + (create-object-detector + [factory opts] + (let [{:keys [contexts epoch] + :or {contexts [(context/cpu)] epoch 0}} opts] + (->WrappedObjectDetector + (new ObjectDetector + model-path-prefix + (convert-descriptors input-descriptors) + (into-array contexts) + (util/->int-option epoch)))))) + +(s/def ::model-path-prefix string?) +(s/def ::input-descriptors (s/coll-of ::mx-io/data-desc)) + +(defn model-factory + "Creates a factory that can be used to instantiate an image classifier + predictor or object detector" + [model-path-prefix input-descriptors] + (util/validate! ::model-path-prefix model-path-prefix + "Invalid model path prefix") + (util/validate! ::input-descriptors input-descriptors + "Invalid input descriptors") + (->InferenceFactory model-path-prefix input-descriptors)) + +(defn reshape-image + "Reshape an image to a new shape" + [image width height] + (util/validate! ::image image "Invalid image") + (util/validate! int? width "Invalid width") + (util/validate! int? height "Invalid height") + (ImageClassifier/reshapeImage image width height)) + +(defn buffered-image-to-pixels + "Convert input BufferedImage to NDArray of input shape" + ([image input-shape-vec] + (buffered-image-to-pixels image input-shape-vec dtype/FLOAT32)) + ([image input-shape-vec dtype] + (util/validate! ::image image "Invalid image") + (util/validate! (s/coll-of int?) input-shape-vec "Invalid shape vector") + (ImageClassifier/bufferedImageToPixels image (shape/->shape input-shape-vec) dtype))) + +(s/def ::image-path string?) +(s/def ::image-paths (s/coll-of ::image-path)) + +(defn load-image-from-file + "Loads an input image given a file name" + [image-path] + (util/validate! ::image-path image-path "Invalid image path") + (ImageClassifier/loadImageFromFile image-path)) + +(defn load-image-paths + "Loads images from a list of file names" + [image-paths] + (util/validate! ::image-paths image-paths "Invalid image paths") + (util/scala-vector->vec + (ImageClassifier/loadInputBatch (util/convert-vector image-paths)))) diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj index e37a8bc8c..651bdcb3f 100644 --- a/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/ndarray.clj @@ -17,7 +17,8 @@ (ns org.apache.clojure-mxnet.ndarray (:refer-clojure :exclude [* - + > >= < <= / cast concat flatten identity load max - min repeat reverse set sort take to-array empty shuffle]) + min repeat reverse set sort take to-array empty shuffle + ref]) (:require [org.apache.clojure-mxnet.base :as base] [org.apache.clojure-mxnet.context :as mx-context] [org.apache.clojure-mxnet.shape :as mx-shape] diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/optimizer.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/optimizer.clj index f18ff40f5..672090a89 100644 --- a/contrib/clojure-package/src/org/apache/clojure_mxnet/optimizer.clj +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/optimizer.clj @@ -17,7 +17,19 @@ (ns org.apache.clojure-mxnet.optimizer (:refer-clojure :exclude [update]) - (:import (org.apache.mxnet.optimizer SGD DCASGD NAG AdaDelta RMSProp AdaGrad Adam SGLD))) + (:require + [clojure.spec.alpha :as s] + [org.apache.clojure-mxnet.util :as util]) + (:import + (org.apache.mxnet.optimizer SGD DCASGD NAG AdaDelta RMSProp AdaGrad Adam SGLD) + (org.apache.mxnet FactorScheduler))) + +(s/def ::learning-rate number?) +(s/def ::momentum number?) +(s/def ::wd number?) +(s/def ::clip-gradient number?) +(s/def ::lr-scheduler #(instance? FactorScheduler %)) +(s/def ::sgd-opts (s/keys :opt-un [::learning-rate ::momentum ::wd ::clip-gradient ::lr-scheduler])) (defn sgd "A very simple SGD optimizer with momentum and weight regularization." @@ -26,10 +38,14 @@ momentum 0.0 wd 0.0001 clip-gradient 0}}] + (util/validate! ::sgd-opts opts "Incorrect sgd optimizer options") (new SGD (float learning-rate) (float momentum) (float wd) (float clip-gradient) lr-scheduler)) ([] (sgd {}))) +(s/def ::lambda number?) +(s/def ::dcasgd-opts (s/keys :opt-un [::learning-rate ::momentum ::lambda ::wd ::clip-gradient ::lr-scheduler])) + (defn dcasgd "DCASGD optimizer with momentum and weight regularization. Implementation of paper 'Asynchronous Stochastic Gradient Descent with @@ -40,10 +56,13 @@ lambda 0.04 wd 0.0 clip-gradient 0}}] + (util/validate! ::sgd-opts opts "Incorrect dcasgd optimizer options") (new DCASGD (float learning-rate) (float lambda) (float momentum) (float wd) (float clip-gradient) lr-scheduler)) ([] (dcasgd {}))) +(s/def ::nag-opts (s/keys :opt-un [::learning-rate ::momentum ::wd ::clip-gradient ::lr-scheduler])) + (defn nag "SGD with nesterov. It is implemented according to @@ -53,10 +72,16 @@ momentum 0.0 wd 0.0001 clip-gradient 0}}] + (util/validate! ::nag-opts opts "Incorrect nag optimizer options") (new NAG (float learning-rate) (float momentum) (float wd) (float clip-gradient) lr-scheduler)) ([] (nag {}))) +(s/def ::rho number?) +(s/def ::rescale-gradient number?) +(s/def ::epsilon number?) +(s/def ::ada-delta-opts (s/keys :opt-un [::rho ::rescale-gradient ::epsilon ::wd ::clip-gradient])) + (defn ada-delta "AdaDelta optimizer as described in Matthew D. Zeiler, 2012. http://arxiv.org/abs/1212.5701" @@ -66,10 +91,15 @@ epsilon 1e-8 wd 0.0 clip-gradient 0}}] + (util/validate! ::ada-delta-opts opts "Incorrect ada-delta optimizer options") (new AdaDelta (float rho) (float rescale-gradient) (float epsilon) (float wd) (float clip-gradient))) ([] (ada-delta {}))) +(s/def gamma1 number?) +(s/def gamma2 number?) +(s/def ::rms-prop-opts (s/keys :opt-un [::learning-rate ::rescale-gradient ::gamma1 ::gamma2 ::wd ::clip-gradient])) + (defn rms-prop "RMSProp optimizer as described in Tieleman & Hinton, 2012. http://arxiv.org/pdf/1308.0850v5.pdf Eq(38) - Eq(45) by Alex Graves, 2013. @@ -80,18 +110,21 @@ - wd L2 regularization coefficient add to all the weights - clip-gradient clip gradient in range [-clip_gradient, clip_gradient] - lr-scheduler The learning rate scheduler" - ([{:keys [learning-rate rescale-gradient gamma1 gamma2 wd lr-scheduler clip-gradient] + ([{:keys [learning-rate rescale-gradient gamma1 gamma2 wd lr-scheduler clip-gradient] :as opts :or {learning-rate 0.002 rescale-gradient 1.0 gamma1 0.95 gamma2 0.9 wd 0.0 clip-gradient 0}}] + (util/validate! ::rms-prop-opts opts "Incorrect rms-prop optimizer options") (new RMSProp (float learning-rate) (float rescale-gradient) (float gamma1) (float gamma2) (float wd) lr-scheduler (float clip-gradient))) ([] (rms-prop {}))) +(s/def ::ada-grad-opts (s/keys :opt-un [::learning-rate ::rescale-gradient ::epsilon ::wd])) + (defn ada-grad " AdaGrad optimizer as described in Duchi, Hazan and Singer, 2011. http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf @@ -101,15 +134,20 @@ Default value is set to 1e-7. - rescale-gradient rescaling factor of gradient. - wd L2 regularization coefficient add to all the weights" - ([{:keys [learning-rate rescale-gradient epsilon wd] + ([{:keys [learning-rate rescale-gradient epsilon wd] :as opts :or {learning-rate 0.05 rescale-gradient 1.0 epsilon 1e-7 wd 0.0}}] + (util/validate! ::ada-grad-opts opts "Incorrect ada-grad optimizer options") (new AdaGrad (float learning-rate) (float rescale-gradient) (float epsilon) (float wd))) ([] (ada-grad {}))) +(s/def ::beta1 number?) +(s/def ::beta2 number?) +(s/def ::adam-opts (s/keys :opt-un [::learning-rate ::beta1 ::beta2 ::epsilon ::decay-factor ::wd ::clip-gradient ::lr-scheduler])) + (defn adam "Adam optimizer as described in [King2014] @@ -125,7 +163,7 @@ - wd L2 regularization coefficient add to all the weights - clip-gradient clip gradient in range [-clip_gradient, clip_gradient] - lr-scheduler The learning rate scheduler" - ([{:keys [learning-rate beta1 beta2 epsilon decay-factor wd clip-gradient lr-scheduler] + ([{:keys [learning-rate beta1 beta2 epsilon decay-factor wd clip-gradient lr-scheduler] :as opts :or {learning-rate 0.002 beta1 0.9 beta2 0.999 @@ -133,11 +171,14 @@ decay-factor (- 1 1e-8) wd 0 clip-gradient 0}}] + (util/validate! ::adam-opts opts "Incorrect adam optimizer options") (new Adam (float learning-rate) (float beta1) (float beta2) (float epsilon) (float decay-factor) (float wd) (float clip-gradient) lr-scheduler)) ([] (adam {}))) +(s/def ::sgld-opts (s/keys :opt-un [::learning-rate ::rescale-gradient ::wd ::clip-gradient ::lr-scheduler])) + (defn sgld "Stochastic Langevin Dynamics Updater to sample from a distribution. @@ -146,11 +187,12 @@ - wd L2 regularization coefficient add to all the weights - clip-gradient Float, clip gradient in range [-clip_gradient, clip_gradient] - lr-scheduler The learning rate scheduler" - ([{:keys [learning-rate rescale-gradient wd clip-gradient lr-scheduler] + ([{:keys [learning-rate rescale-gradient wd clip-gradient lr-scheduler] :as opts :or {learning-rate 0.01 rescale-gradient 1 wd 0.0001 clip-gradient 0}}] + (util/validate! ::sgld-opts opts "Incorrect sgld optimizer options") (new SGLD (float learning-rate) (float rescale-gradient) (float wd) (float clip-gradient) lr-scheduler)) ([] diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/primitives.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/primitives.clj new file mode 100644 index 000000000..0967df228 --- /dev/null +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/primitives.clj @@ -0,0 +1,46 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.primitives + (:import (org.apache.mxnet MX_PRIMITIVES$MX_FLOAT MX_PRIMITIVES$MX_Double + MX_PRIMITIVES$MX_PRIMITIVE_TYPE))) + + +;;; Defines customer mx primitives that can be used for mathematical computations +;;; in NDArrays to control precision. Currently Float and Double are supported + +;;; For purposes of automatic conversion in ndarray functions, doubles are default +;; to specify using floats you must use a Float + +(defn mx-float + "Creates a MXNet float primitive" + [num] + (new MX_PRIMITIVES$MX_FLOAT num)) + +(defn mx-double + "Creates a MXNet double primitive" + [num] + (new MX_PRIMITIVES$MX_Double num)) + +(defn ->num + "Returns the underlying number value" + [primitive] + (.data primitive)) + +(defn primitive? [x] + (instance? MX_PRIMITIVES$MX_PRIMITIVE_TYPE x)) + diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj index d6e33789a..0ec2039ba 100644 --- a/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/random.clj @@ -16,8 +16,18 @@ ;; (ns org.apache.clojure-mxnet.random - (:require [org.apache.clojure-mxnet.shape :as mx-shape]) - (:import (org.apache.mxnet Random))) + (:require + [org.apache.clojure-mxnet.shape :as mx-shape] + [org.apache.clojure-mxnet.context :as context] + [clojure.spec.alpha :as s] + [org.apache.clojure-mxnet.util :as util]) + (:import (org.apache.mxnet Context Random))) + +(s/def ::low number?) +(s/def ::high number?) +(s/def ::shape-vec (s/coll-of pos-int? :kind vector?)) +(s/def ::ctx #(instance? Context %)) +(s/def ::uniform-opts (s/keys :opt-un [::ctx])) (defn uniform "Generate uniform distribution in [low, high) with shape. @@ -29,10 +39,18 @@ out: Output place holder} returns: The result ndarray with generated result./" ([low high shape-vec {:keys [ctx out] :as opts}] + (util/validate! ::uniform-opts opts "Incorrect random uniform parameters") + (util/validate! ::low low "Incorrect random uniform parameter") + (util/validate! ::high high "Incorrect random uniform parameters") + (util/validate! ::shape-vec shape-vec "Incorrect random uniform parameters") (Random/uniform (float low) (float high) (mx-shape/->shape shape-vec) ctx out)) ([low high shape-vec] (uniform low high shape-vec {}))) +(s/def ::loc number?) +(s/def ::scale number?) +(s/def ::normal-opts (s/keys :opt-un [::ctx])) + (defn normal "Generate normal(Gaussian) distribution N(mean, stdvar^^2) with shape. loc: The standard deviation of the normal distribution @@ -43,10 +61,15 @@ out: Output place holder} returns: The result ndarray with generated result./" ([loc scale shape-vec {:keys [ctx out] :as opts}] + (util/validate! ::normal-opts opts "Incorrect random normal parameters") + (util/validate! ::loc loc "Incorrect random normal parameters") + (util/validate! ::scale scale "Incorrect random normal parameters") + (util/validate! ::shape-vec shape-vec "Incorrect random uniform parameters") (Random/normal (float loc) (float scale) (mx-shape/->shape shape-vec) ctx out)) ([loc scale shape-vec] (normal loc scale shape-vec {}))) +(s/def ::seed-state number?) (defn seed " Seed the random number generators in mxnet. This seed will affect behavior of functions in this module, @@ -58,4 +81,5 @@ This means if you set the same seed, the random number sequence generated from GPU0 can be different from CPU." [seed-state] - (Random/seed (int seed-state))) + (util/validate! ::seed-state seed-state "Incorrect seed parameters") + (Random/seed (int seed-state))) \ No newline at end of file diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/symbol.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/symbol.clj index 58b1d6d49..095165582 100644 --- a/contrib/clojure-package/src/org/apache/clojure_mxnet/symbol.clj +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/symbol.clj @@ -18,7 +18,7 @@ (ns org.apache.clojure-mxnet.symbol (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten load max min repeat reverse set sort take to-array empty sin - get apply shuffle]) + get apply shuffle ref]) (:require [org.apache.clojure-mxnet.base :as base] [org.apache.clojure-mxnet.context :as mx-context] [org.apache.clojure-mxnet.executor :as ex] diff --git a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj index 6f22b0eb3..43970c0ab 100644 --- a/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj +++ b/contrib/clojure-package/src/org/apache/clojure_mxnet/util.clj @@ -19,6 +19,7 @@ (:require [clojure.spec.alpha :as s] [t6.from-scala.core :refer [$ $$] :as $] [clojure.string :as string] + [org.apache.clojure-mxnet.primitives :as primitives] [org.apache.clojure-mxnet.shape :as mx-shape]) (:import (org.apache.mxnet NDArray) (scala Product Tuple2 Tuple3) @@ -36,7 +37,8 @@ "byte<>" "byte-array" "java.lang.String<>" "vec-or-strings" "org.apache.mxnet.NDArray" "ndarray" - "org.apache.mxnet.Symbol" "sym"}) + "org.apache.mxnet.Symbol" "sym" + "org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE" "double-or-float"}) (def symbol-param-coerce {"java.lang.String" "sym-name" "float" "num" @@ -66,6 +68,9 @@ (defn ->option [v] ($ Option v)) +(defn ->int-option [v] + (->option (when v (int v)))) + (defn option->value [opt] ($/view opt)) @@ -141,6 +146,8 @@ (and (get targets "int<>") (vector? param)) (int-array param) (and (get targets "float<>") (vector? param)) (float-array param) (and (get targets "java.lang.String<>") (vector? param)) (into-array param) + (and (get targets "org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE") (instance? Float param)) (primitives/mx-float param) + (and (get targets "org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE") (number? param)) (primitives/mx-double param) :else param)) (defn nil-or-coerce-param [param targets] @@ -174,8 +181,15 @@ (instance? Map return-val) (scala-map->map return-val) (instance? Tuple2 return-val) (tuple->vec return-val) (instance? Tuple3 return-val) (tuple->vec return-val) + (primitives/primitive? return-val) (primitives/->num return-val) :else return-val)) +(defn coerce-return-recursive [return-val] + (let [coerced-val (coerce-return return-val)] + (if (vector? coerced-val) + (into [] (map coerce-return-recursive coerced-val)) + coerced-val))) + (defmacro scala-fn "Creates a scala fn from an anonymous clojure fn of the form (fn [x] body)" [f] diff --git a/contrib/clojure-package/test/good-test-ndarray.clj b/contrib/clojure-package/test/good-test-ndarray.clj index 8cdfce76f..b048a819c 100644 --- a/contrib/clojure-package/test/good-test-ndarray.clj +++ b/contrib/clojure-package/test/good-test-ndarray.clj @@ -1,6 +1,7 @@ (ns org.apache.clojure-mxnet.ndarray (:refer-clojure :exclude [* - + > >= < <= / cast concat flatten identity load max - min repeat reverse set sort take to-array empty shuffle]) + min repeat reverse set sort take to-array empty shuffle + ref]) (:import (org.apache.mxnet NDArray Shape))) ;; Do not edit - this is auto-generated @@ -26,11 +27,12 @@ (defn div - ([ndarray num-or-ndarray] + ([ndarray ndarray-or-double-or-float] (util/coerce-return (.$div ndarray (util/coerce-param - num-or-ndarray - #{"float" "org.apache.mxnet.NDArray"}))))) + ndarray-or-double-or-float + #{"org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE" + "org.apache.mxnet.NDArray"}))))) diff --git a/contrib/clojure-package/test/good-test-symbol.clj b/contrib/clojure-package/test/good-test-symbol.clj index 0f7479ad4..947d9262d 100644 --- a/contrib/clojure-package/test/good-test-symbol.clj +++ b/contrib/clojure-package/test/good-test-symbol.clj @@ -1,9 +1,9 @@ (ns org.apache.clojure-mxnet.symbol - (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten load max - min repeat reverse set sort take to-array empty sin - get apply shuffle]) - (:require [org.apache.clojure-mxnet.util :as util]) - (:import (org.apache.mxnet Symbol))) + (:refer-clojure :exclude [* - + > >= < <= / cast concat identity flatten load max + min repeat reverse set sort take to-array empty sin + get apply shuffle ref]) + (:require [org.apache.clojure-mxnet.util :as util]) + (:import (org.apache.mxnet Symbol))) ;; Do not edit - this is auto-generated diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj new file mode 100644 index 000000000..38ab11c86 --- /dev/null +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/image_test.clj @@ -0,0 +1,79 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.image-test + (:require [org.apache.clojure-mxnet.image :as image] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [clojure.java.io :as io] + [clojure.test :refer :all]) + (:import (javax.imageio ImageIO))) + +(def tmp-dir (System/getProperty "java.io.tmpdir")) +(def image-path (.getAbsolutePath (io/file tmp-dir "Pug-Cookie.jpg"))) + +(defn download-image [] + (with-open [in (io/input-stream "https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg") + out (io/output-stream (io/file image-path))] + (io/copy in out))) + +(defn delete-image [] + (io/delete-file image-path)) + +(defn with-downloaded-image [f] + (download-image) + (f) + (delete-image)) + +(use-fixtures :once with-downloaded-image) + +(deftest test-decode-image + (let [img-arr (image/decode-image + (io/input-stream image-path)) + img-arr-2 (image/decode-image + (io/input-stream image-path) + {:color-flag image/GRAYSCALE})] + (is (= [576 1024 3] (ndarray/shape-vec img-arr))) + (is (= [576 1024 1] (ndarray/shape-vec img-arr-2))))) + +(deftest test-read-image + (let [img-arr (image/read-image image-path) + img-arr-2 (image/read-image + image-path + {:color-flag image/GRAYSCALE})] + (is (= [576 1024 3] (ndarray/shape-vec img-arr))) + (is (= [576 1024 1] (ndarray/shape-vec img-arr-2))))) + +(deftest test-resize-image + (let [img-arr (image/read-image image-path) + resized-arr (image/resize-image img-arr 224 224)] + (is (= [224 224 3] (ndarray/shape-vec resized-arr))))) + +(deftest test-crop-image + (let [img-arr (image/read-image image-path) + cropped-arr (image/fixed-crop img-arr 0 0 224 224)] + (is (= [224 224 3] (ndarray/shape-vec cropped-arr))))) + +(deftest test-apply-border + (let [img-arr (image/read-image image-path) + padded-arr (image/apply-border img-arr 1 1 1 1)] + (is (= [578 1026 3] (ndarray/shape-vec padded-arr))))) + +(deftest test-to-image + (let [img-arr (image/read-image image-path) + resized-arr (image/resize-image img-arr 224 224) + new-img (image/to-image resized-arr)] + (is (= true (ImageIO/write new-img "png" (io/file tmp-dir "out.png")))))) diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj new file mode 100644 index 000000000..e3935c31e --- /dev/null +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/imageclassifier_test.clj @@ -0,0 +1,128 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.infer.imageclassifier-test + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.test :refer :all])) + +(def model-dir "data/") +(def model-path-prefix (str model-dir "resnet-18/resnet-18")) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/infer/get_resnet_18_data.sh")) + +(defn create-classifier [] + (let [descriptors [{:name "data" + :shape [1 3 224 224] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-image-classifier factory))) + +(deftest test-single-classification + (let [classifier (create-classifier) + image (infer/load-image-from-file "test/test-images/kitten.jpg") + [predictions-all] (infer/classify-image classifier image) + [predictions-with-default-dtype] (infer/classify-image classifier image 10) + [predictions] (infer/classify-image classifier image 5 dtype/FLOAT32)] + (is (= 1000 (count predictions-all))) + (is (= 10 (count predictions-with-default-dtype))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-batch-classification + (let [classifier (create-classifier) + image-batch (infer/load-image-paths ["test/test-images/kitten.jpg" + "test/test-images/Pug-Cookie.jpg"]) + [batch-predictions-all] (infer/classify-image-batch classifier image-batch) + [batch-predictions-with-default-dtype] (infer/classify-image-batch classifier image-batch 10) + [predictions] (infer/classify-image-batch classifier image-batch 5 dtype/FLOAT32)] + (is (= 1000 (count batch-predictions-all))) + (is (= 10 (count batch-predictions-with-default-dtype))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-single-classification-with-ndarray + (let [classifier (create-classifier) + image (-> (infer/load-image-from-file "test/test-images/kitten.jpg") + (infer/reshape-image 224 224) + (infer/buffered-image-to-pixels [3 224 224] dtype/FLOAT32) + (ndarray/expand-dims 0)) + [predictions-all] (infer/classify-with-ndarray classifier [image]) + [predictions] (infer/classify-with-ndarray classifier [image] 5)] + (is (= 1000 (count predictions-all))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-single-classify + (let [classifier (create-classifier) + image (-> (infer/load-image-from-file "test/test-images/kitten.jpg") + (infer/reshape-image 224 224) + (infer/buffered-image-to-pixels [3 224 224] dtype/FLOAT32) + (ndarray/expand-dims 0)) + predictions-all (infer/classify classifier [(ndarray/->vec image)]) + predictions (infer/classify classifier [(ndarray/->vec image)] 5)] + (is (= 1000 (count predictions-all))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-base-classification-with-ndarray + (let [descriptors [{:name "data" + :shape [1 3 224 224] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors) + classifier (infer/create-classifier factory) + image (-> (infer/load-image-from-file "test/test-images/kitten.jpg") + (infer/reshape-image 224 224) + (infer/buffered-image-to-pixels [3 224 224] dtype/FLOAT32) + (ndarray/expand-dims 0)) + [predictions-all] (infer/classify-with-ndarray classifier [image]) + [predictions] (infer/classify-with-ndarray classifier [image] 5)] + (is (= 1000 (count predictions-all))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + +(deftest test-base-single-classify + (let [descriptors [{:name "data" + :shape [1 3 224 224] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors) + classifier (infer/create-classifier factory) + image (-> (infer/load-image-from-file "test/test-images/kitten.jpg") + (infer/reshape-image 224 224) + (infer/buffered-image-to-pixels [3 224 224] dtype/FLOAT32) + (ndarray/expand-dims 0)) + predictions-all (infer/classify classifier [(ndarray/->vec image)]) + predictions (infer/classify classifier [(ndarray/->vec image)] 5)] + (is (= 1000 (count predictions-all))) + (is (= 5 (count predictions))) + (is (= "n02123159 tiger cat" (:class (first predictions)))) + (is (= (< 0 (:prob (first predictions)) 1))))) + + diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/objectdetector_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/objectdetector_test.clj new file mode 100644 index 000000000..e2b9579c7 --- /dev/null +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/objectdetector_test.clj @@ -0,0 +1,82 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.infer.objectdetector-test + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.test :refer :all] + [org.apache.clojure-mxnet.ndarray :as ndarray])) + +(def model-dir "data/") +(def model-path-prefix (str model-dir "resnet50_ssd/resnet50_ssd_model")) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/infer/get_ssd_data.sh")) + +(defn create-detector [] + (let [descriptors [{:name "data" + :shape [1 3 512 512] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-object-detector factory))) + +(deftest test-single-detection + (let [detector (create-detector) + image (infer/load-image-from-file "test/test-images/kitten.jpg") + [predictions-all] (infer/detect-objects detector image) + [predictions] (infer/detect-objects detector image 5) + {:keys [class prob x-min x-max y-min y-max] :as pred} (first predictions)] + (is (some? predictions)) + (is (= 5 (count predictions))) + (is (= 13 (count predictions-all))) + (is (= "cat" class)) + (is (< 0.8 prob)) + (every? #(< 0 % 1) [x-min x-max y-min y-max]))) + +(deftest test-batch-detection + (let [detector (create-detector) + image-batch (infer/load-image-paths ["test/test-images/kitten.jpg" + "test/test-images/Pug-Cookie.jpg"]) + [batch-predictions-all] (infer/detect-objects-batch detector image-batch) + [predictions] (infer/detect-objects-batch detector image-batch 5) + {:keys [class prob x-min x-max y-min y-max] :as pred} (first predictions)] + (is (some? predictions)) + (is (= 13 (count batch-predictions-all))) + (is (= 5 (count predictions))) + (is (= "cat" class)) + (is (< 0.8 prob)) + (every? #(< 0 % 1) [x-min x-max y-min y-max]))) + +(deftest test-detection-with-ndarrays + (let [detector (create-detector) + image (-> (infer/load-image-from-file "test/test-images/kitten.jpg") + (infer/reshape-image 512 512) + (infer/buffered-image-to-pixels [3 512 512] dtype/FLOAT32) + (ndarray/expand-dims 0)) + [predictions-all] (infer/detect-objects-with-ndarrays detector [image]) + [predictions] (infer/detect-objects-with-ndarrays detector [image] 1) + {:keys [class prob x-min x-max y-min y-max] :as pred} (first predictions)] + (is (some? predictions-all)) + (is (= 1 (count predictions))) + (is (= "cat" class)) + (is (< 0.8 prob)) + (every? #(< 0 % 1) [x-min x-max y-min y-max]))) + diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/predictor_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/predictor_test.clj new file mode 100644 index 000000000..e1526be61 --- /dev/null +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/infer/predictor_test.clj @@ -0,0 +1,77 @@ +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.infer.predictor-test + (:require [org.apache.clojure-mxnet.context :as context] + [org.apache.clojure-mxnet.dtype :as dtype] + [org.apache.clojure-mxnet.infer :as infer] + [org.apache.clojure-mxnet.layout :as layout] + [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.shape :as shape] + [clojure.java.io :as io] + [clojure.java.shell :refer [sh]] + [clojure.string :refer [split]] + [clojure.test :refer :all] + [org.apache.clojure-mxnet.util :as util])) + +(def model-dir "data/") +(def model-path-prefix (str model-dir "resnet-18/resnet-18")) +(def width 224) +(def height 224) + +(when-not (.exists (io/file (str model-path-prefix "-symbol.json"))) + (sh "./scripts/infer/get_resnet_18_data.sh")) + +(defn create-predictor [] + (let [descriptors [{:name "data" + :shape [1 3 height width] + :layout layout/NCHW + :dtype dtype/FLOAT32}] + factory (infer/model-factory model-path-prefix descriptors)] + (infer/create-predictor factory))) + +(deftest predictor-test-with-ndarray + (let [predictor (create-predictor) + image-ndarray (-> "test/test-images/kitten.jpg" + infer/load-image-from-file + (infer/reshape-image width height) + (infer/buffered-image-to-pixels [3 width height]) + (ndarray/expand-dims 0)) + predictions (infer/predict-with-ndarray predictor [image-ndarray]) + synset-file (-> (io/file model-path-prefix) + (.getParent) + (io/file "synset.txt")) + synset-names (split (slurp synset-file) #"\n") + [best-index] (ndarray/->int-vec (ndarray/argmax (first predictions) 1)) + best-prediction (synset-names best-index)] + (is (= "n02123159 tiger cat" best-prediction)))) + +(deftest predictor-test + (let [predictor (create-predictor) + image-ndarray (-> "test/test-images/kitten.jpg" + infer/load-image-from-file + (infer/reshape-image width height) + (infer/buffered-image-to-pixels [3 width height]) + (ndarray/expand-dims 0)) + predictions (infer/predict predictor [(ndarray/->vec image-ndarray)]) + synset-file (-> (io/file model-path-prefix) + (.getParent) + (io/file "synset.txt")) + synset-names (split (slurp synset-file) #"\n") + ndarray-preds (ndarray/array (first predictions) [1 1000]) + [best-index] (ndarray/->int-vec (ndarray/argmax ndarray-preds 1)) + best-prediction (synset-names best-index)] + (is (= "n02123159 tiger cat" best-prediction)))) diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj index 79e94412d..9ffd3abed 100644 --- a/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/ndarray_test.clj @@ -97,7 +97,7 @@ (is (= [1.0 1.0] (->vec ndhalves))))) (deftest test-full - (let [nda (full [1 2] 3)] + (let [nda (full [1 2] 3.0)] (is (= (shape nda) (mx-shape/->shape [1 2]))) (is (= [3.0 3.0] (->vec nda))))) diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj index 1b4b2ea2f..c97711b5f 100644 --- a/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/operator_test.clj @@ -462,7 +462,7 @@ test (sym/transpose data) shape-vec [3 4] ctx (context/default-context) - arr-data (random/uniform 0 100 shape-vec ctx) + arr-data (random/uniform 0 100 shape-vec {:ctx ctx}) trans (ndarray/transpose (ndarray/copy arr-data)) exec-test (sym/bind test ctx {"data" arr-data}) out (-> (executor/forward exec-test) diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/optimizer_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/optimizer_test.clj index f6461b10f..599a0672b 100644 --- a/contrib/clojure-package/test/org/apache/clojure_mxnet/optimizer_test.clj +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/optimizer_test.clj @@ -44,3 +44,13 @@ ["sgld" optimizer/sgld]]] (doseq [opt opts] (test-optimizer opt)))) + +(deftest test-optimizers-parameters-specs + (is (thrown? Exception (optimizer/sgd {:wd 'a}))) + (is (thrown? Exception (optimizer/dcasgd {:lambda 'a}))) + (is (thrown? Exception (optimizer/nag {:momentum 'a}))) + (is (thrown? Exception (optimizer/ada-delta {:epsilon 'a}))) + (is (thrown? Exception (optimizer/rms-prop {:gamma1 'a}))) + (is (thrown? Exception (optimizer/ada-grad {:rescale-gradient 'a}))) + (is (thrown? Exception (optimizer/adam {:beta1 'a}))) + (is (thrown? Exception (optimizer/sgld {:lr-scheduler 0.1})))) \ No newline at end of file diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/primitives_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/primitives_test.clj new file mode 100644 index 000000000..1a538e537 --- /dev/null +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/primitives_test.clj @@ -0,0 +1,45 @@ +;; +;; Licensed to the Apache Software Foundation (ASF) under one or more +;; contributor license agreements. See the NOTICE file distributed with +;; this work for additional information regarding copyright ownership. +;; The ASF licenses this file to You under the Apache License, Version 2.0 +;; (the "License"); you may not use this file except in compliance with +;; the License. You may obtain a copy of the License at +;; +;; http://www.apache.org/licenses/LICENSE-2.0 +;; +;; Unless required by applicable law or agreed to in writing, software +;; distributed under the License is distributed on an "AS IS" BASIS, +;; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +;; See the License for the specific language governing permissions and +;; limitations under the License. +;; + +(ns org.apache.clojure-mxnet.primitives-test + (:require [org.apache.clojure-mxnet.primitives :as primitives] + [clojure.test :refer :all]) + (:import (org.apache.mxnet MX_PRIMITIVES$MX_PRIMITIVE_TYPE + MX_PRIMITIVES$MX_FLOAT + MX_PRIMITIVES$MX_Double))) + +(deftest test-primitive-types + (is (not (primitives/primitive? 3))) + (is (primitives/primitive? (primitives/mx-float 3))) + (is (primitives/primitive? (primitives/mx-double 3)))) + +(deftest test-float-primitives + (is (instance? MX_PRIMITIVES$MX_PRIMITIVE_TYPE (primitives/mx-float 3))) + (is (instance? MX_PRIMITIVES$MX_FLOAT (primitives/mx-float 3))) + (is (instance? Float (-> (primitives/mx-float 3) + (primitives/->num)))) + (is (= 3.0 (-> (primitives/mx-float 3) + (primitives/->num))))) + +(deftest test-double-primitives + (is (instance? MX_PRIMITIVES$MX_PRIMITIVE_TYPE (primitives/mx-double 2))) + (is (instance? MX_PRIMITIVES$MX_Double (primitives/mx-double 2))) + (is (instance? Double (-> (primitives/mx-double 2) + (primitives/->num)))) + (is (= 2.0 (-> (primitives/mx-double 2) + (primitives/->num))))) + diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/random_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/random_test.clj index c4e919807..6952335c1 100644 --- a/contrib/clojure-package/test/org/apache/clojure_mxnet/random_test.clj +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/random_test.clj @@ -26,9 +26,9 @@ (let [[a b] [-10 10] shape [100 100] _ (random/seed 128) - un1 (random/uniform a b shape {:context ctx}) + un1 (random/uniform a b shape {:ctx ctx}) _ (random/seed 128) - un2 (random/uniform a b shape {:context ctx})] + un2 (random/uniform a b shape {:ctx ctx})] (is (= un1 un2)) (is (< (Math/abs (/ (/ (apply + (ndarray/->vec un1)) @@ -52,3 +52,16 @@ (is (< (Math/abs (- mean mu)) 0.1)) (is (< (Math/abs (- stddev sigma)) 0.1))))) +(defn random-or-normal [fn_] + (is (thrown? Exception (fn_ 'a 2 []))) + (is (thrown? Exception (fn_ 1 'b []))) + (is (thrown? Exception (fn_ 1 2 [-1]))) + (is (thrown? Exception (fn_ 1 2 [2 3 0]))) + (is (thrown? Exception (fn_ 1 2 [10 10] {:ctx "a"}))) + (let [ctx (context/default-context)] + (is (not (nil? (fn_ 1 1 [100 100] {:ctx ctx})))))) + +(deftest test-random-parameters-specs + (random-or-normal random/normal) + (random-or-normal random/uniform) + (is (thrown? Exception (random/seed "a")))) \ No newline at end of file diff --git a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj b/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj index ee7710317..c26f83d5a 100644 --- a/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj +++ b/contrib/clojure-package/test/org/apache/clojure_mxnet/util_test.clj @@ -20,6 +20,7 @@ [org.apache.clojure-mxnet.shape :as mx-shape] [org.apache.clojure-mxnet.util :as util] [org.apache.clojure-mxnet.ndarray :as ndarray] + [org.apache.clojure-mxnet.primitives :as primitives] [org.apache.clojure-mxnet.symbol :as sym] [org.apache.clojure-mxnet.test-util :as test-util] [clojure.spec.alpha :as s]) @@ -54,6 +55,16 @@ (is (instance? Option x)) (is (= 1 (.get x))))) +(deftest test->int-option + (let [x (util/->int-option 4.5)] + (is (instance? Option x)) + (is (= 4 (.get x))))) + +(deftest test-empty->int-option + (let [x (util/->int-option nil)] + (is (instance? Option x)) + (is (.isEmpty x)))) + (deftest test-option->value (is (= 2 (-> (util/->option 2) (util/option->value))))) @@ -123,6 +134,9 @@ (is (= "[F" (->> (util/coerce-param [1 2] #{"float<>"}) str (take 2) (apply str)))) (is (= "[L" (->> (util/coerce-param [1 2] #{"java.lang.String<>"}) str (take 2) (apply str)))) + (is (primitives/primitive? (util/coerce-param 1.0 #{"org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE"}))) + (is (primitives/primitive? (util/coerce-param (float 1.0) #{"org.apache.mxnet.MX_PRIMITIVES$MX_PRIMITIVE_TYPE"}))) + (is (= 1 (util/coerce-param 1 #{"unknown"})))) (deftest test-nil-or-coerce-param @@ -161,6 +175,12 @@ (util/convert-tuple [1 2])))) (is (= [1 2 3] (util/coerce-return (util/convert-tuple [1 2 3])))) + + (is (instance? Double (util/coerce-return (primitives/mx-double 3)))) + (is (= 3.0 (util/coerce-return (primitives/mx-double 3)))) + (is (instance? Float (util/coerce-return (primitives/mx-float 2)))) + (is (= 2.0 (util/coerce-return (primitives/mx-float 2)))) + (is (= "foo" (util/coerce-return "foo")))) (deftest test-translate-keyword-shape diff --git a/contrib/clojure-package/test/test-images/Pug-Cookie.jpg b/contrib/clojure-package/test/test-images/Pug-Cookie.jpg new file mode 100644 index 0000000000000000000000000000000000000000..56f5dc16ed7a8b5e3c5168906c67be5bd2a02463 GIT binary patch literal 104323 zcmb4pWl&tfw(j7r!96fI!QI{6-Q8V-OJEq>!r<=imH=UJ_h2DF@WFxv33;4Tb?g3l zRj=OK{i}QR>fU=V?XUl>|Jw!ND$6U$1K{A`0E%x9;NL4iL`GXqN>@W$n8v~OgR`fd z1C2iyJ0}g3pO2Hf1C5fh0*$h!oC1wMKQ}WQjhCO5tCMdKjTns(ANs#dfD8Zu9{xY= zjS$}&5-JiBA|etx3JNkRCORf2208`?7B)UE7B(I>1_mw>F5Wu=LPA1J9AXk80up=z zLW2JYfkSwkgNTHNgoH+bg@Hxz|4sh}0eGnJl868VI9dQa9vlK5+`kb3B>?cYv;Vy7 z{}2KqJQ5r-3IG-DZCDi-01tfP{pMfC7&K2afwgymyop0Z zLPmKT{DAkilQ)7xd}|2*Wr2goLwJwRg(yXf^p0E0%8LM*PCAW8JF?{81_1Mo%Ol4;lv4iR3fe5YkjqTYT6}dq6^hXRa#n&kneCYcF$eh^B)oTo~)K>cz zoCLQ{&=2^#hXr8W3VXHYv<4xPeFCP^`${bzuP4`~e6wbV z7Ou|>mVz_)3TUl4VU(%19~9LD4=RJN2Q?=sNabtlfVHyAtB1ij6@>C3^Muu)E$XY< zW&+Hj3Q|X=z|y7)fE+2WB$tfL;gc2r=66SP72<#40fR10 z{5e9~l6ptoc7HzYf zhMOgL9^S+KmS-n^xKwg9^jI&NjunJwt}aL6%_K{8OOrsMxo({R4y|Qejlf7AjhV_& z#bsg>9BTj&6L1TS4UooS#R7}4t#@8QjFHh9RAG3xW2Y6Zq}X9Y zyO_bwqX(3it6(rC3x%?BABnfqG^GVq*qJk*$g$GSr{8b+S6Q0}Rutb>S(Qk|P!2d% z=Rgpp^AhBlj8KxCx^ub+NOU32?89%ivS+@$Yy-ih5JTVz9AL5u-Yz7EiKwL2@?a)o+EwCwEds4<5D?jM{^78= zG&g}ic?CJ<%|_oxIC@2>@zzQhXIf9li-a=mT1Q4<5s9m*eZoT4XdtK1oVQOd17&?L z&}qYf5+VoHBazfpI2S+|t#GZuQE{c?7dN;ieO0`G_cbVK5oYLS_&)D=!Yb37lkg*jE#R_b>S@>k5M)D_xkAMTg1BW zP#Ztl-;xerTn0l{S!6ok;t0`mXJi`yjMdaqCko6Ea&{&-ysi$+h)||qVOj76)w&p% zK4;QaMaq^{Rn7_OB4O<`rdKYQxz7ufV$KW^MxV}Pte%$eG5-M?&~j250PsR3oJS|B z!t?ndngAgxTVav&|eMUCMXyE<^c?Vu5LvNj@{^C?=jF<>T zCA5-EbIK)(+8;3Z&ZM=P<5zh=QgJ|CVSI~NPW{S+nM<&hC}XAT4(gI-@8(6reSnJ5 zKJd0hDb;#YfFAX8QmyNOQZOaO{-v~fvxHvzOyA#R&3?s-Q6PD5kJ4UW7)V?8p$?7s zM0tGRWV|%vSXvi>&w|^!f!-9P>;?V@sL;(k$x27pwyXMO?xdt&?Tf0|T~SVX^4$P& zu@xQui``I#Ed^PaA4vt!oC%{ZOtNa<6u?aG1=frC2qLt>1MlT0g=k-;1iP~!@}TcQ z5`v25)^%{bmuY3Z7@eJnA(*<8ja)L!DDh#wi<-1lh0D_)_d#FM&|Iy#bOfPRsAH?; z&T&t)Nce2gS_p-spy&>CBy%-F1p+>T5gT55VTL-eeXbM~O^9H`rI3jn3m?E#TACOM z%GE|PEj=GUOil!$V`|MG5$YxtNzHr7X=zHyYw2?Wi7M;NwV>|^1GCMjq!f~i>&EZP zmz^k8RgA1{vVw+aFX%^+N?0GVzK`<9KI}=Rt-}z&s!AuMarvdI*+G2 zvDem?ua$)rl<~WS2(dxIfp;j8;xN0r$57E6Gv?n|`%DUa2=D~)`lyr6(S z@L|k1f^XQ4w!y*k7{a^rrq;QBGiZwyaQYNv??85Of+d_a@aIfy%yFiFnbbvAj7t64 zzCkcRc4NH6gh;Vwa_oC~Ubn@HEX8V2c)7|g?VArqa7siuK9L|d=m(m++Ey0moKUgw z9>f{5|JtavsYMBip4;-Up2Bv~r-=}8tQX%)(p`=FOfsWGk4y<|GXLQ%A}~)Fh%PSL zu7G3cO}&(-bCHX311soDFzX7qW*BK#vW6G&L2+(WF?QyMnW>@7k0`Wr8Jexc-%N;Jui{NHvAQV)q2v-ynOgXlbcJr_Wa`B~xA+kvDK%EIV1)xIzYH#&T<4&Hs;XFjP^`*`9LEp^91-74u5QeBu%JyZS4 zQoFrsumX0!@iW5wQ zL&)i@DMbVlA}vACF(9b{vfNjvdTsPKq?xm$G{EJ1X`3U99fvX9mNIB8M$o1xr-LbU zr7+HPU^8&+*#(!~B>0b|r__u2pE?DgZ3A~} zOzFLdc?tb|n(k5e_M<-5f%7yPf`@k~tL!8uQvG7;DVMsWP9e`yNlA6#wdAi|ln zJRTZUfI@*{wa;TkpIfy$V^rRjK601E!6!5h=ft#Yj~K>LaqKf;W5;H1ImSlLz)NBK zR|)nOYaH0pQf$B#iz1mhTC#d7;Oq_L5J+uNUHiDD)6OYhq?8BYXCxMpPK_$}u#RgD zi`P{YO%4}FbG|En)1VZVZ}bXc&3Z}~gW_CYdOQqX7mZ`jJ>1^^pi5&JE_c;=0ZnZp zWTagDbwLZ_(vY7*94HF*a~PwfuJZj#A$a{ub#sUgDadC~P%LujCC>S4dGM`^|5Q*A zn-^N#o!)OjX-`zDioD6PwT$tm8I{+8mEI~N0fPq%5g#CE_J)e|6T&9ID?+R;#3Q*2 zV%NHEG6x*yMU|$E(n=jz)tbB4( zt1LnY^!ziEoi>0vCRxG~oH58!ZOE<)NFg+`S%zH(+sfapLp@7POu(2;IOw!%LnJ{pHX#H8kS2{8OB7@g<~q*$KO)xfL7GR zK}%-YYSfESisO2kj|raJ1mdl6^Jcb07Xpm)FDQWpwc?umI6inkH^P$u@C z+MKtv9U+-H#~8El%L*cBoZCyKdoR~ZGR-lb&`V`ftfmHueILXv*LfJt;-i`ddWX5{ z&Gh`vYlef{y>iLf#XwM(jHUSUu|T?$`p87~Nn+L?Ran!-oH+!+HY zDZGM`iCI2zdY#YKeklZp+De}Km!>hT=Je-XgF3na6$-u5d#MPfx{YEa++h6<$yADk zvxZNPrC*P$qzQj zn+{d8q*#?I<|KyjIyhE)P&-4Kp#aIvTB|E;M@3JK4vhgg&VK;o&m#W-EU1@q8peP` zsG@&$%71*Y*84a2drEU{WPZ!0JYZhG`>aXQr0-?uN!;_~BTZhmLX^tbpl5z3mXvkQ zpFT}wlxL0O|2p?0?FqD0{HLk%)PmCnNJ5~v(DLnkmSXog;2FWFKh3Q5lsiZ97%Q-B zSs&}>ut%&5Dk3J$+z6iXpuktcpc7z|EfK(3_^Q0}>u*_mo{HgN5F828xTRhIm)=3J zKVRaFSC#w}vV8re_Ku~)(oS_pVSURIk&KrgMF^)+0A|YbP$Oae>yrOd=jI1o;0|w# z^zWVxf;L=@O@yzjUwl`;eAsgFQSmf)iDc8*BNjAZ39wTSl~m!J6uh*u3`r1~1|^bb&tWBj1@(WR^R%h%5C_Z3~;SDw-0?viq2 z-iB@@7gEn_-}W09?@uZil=hGX_fyjmEvvDWD0Tq^a1-zi!3K2)$*2DS21{0H#>}!m zFIoB?i@D6R$=H{2EurTrguLw^Nai>SBs@dw}o1MH##O@?R+k=4CmPoQ6a zL!RY^94oQ-R$Aenor~}|>^mW6BiSJZ<(qvJL}~f>-?m_|V*%_p zRrfRK?RE+e0^XlaP+)^J)#2@aOxe;6yyvQ*aQX9G4xWst;_8UHS)j*XTEnG-=F*@GS{MIT;fRcy|i@?Emm&Fjj;X z_TK5NZlvpR5ys8|M>Nj8FnXRhJ}S7bkolY{vNli2he+JnTv<@dChWOXI?#VLdri>l zK7{eI&MmL-g9Sq{E%VjcObf&98aiaCP{5{skwj0OYti^+;__ z^j)eH2>V3iO&bNN%8PD_H=KL@i{2@dhs2H#A*MW!-93R8*o<7ei543N?XpNY42SswU-}E#aBli`xxqzGQ~F; z#&%{PW>b3la3B4PjjhXMj5+?R9Tsc#EZX>0&a11Wo~NYTp4ix;n+BHSFXr}lB=;rPm7|w?^o7y; zA2aovhup-b!6u7I>0%wH|3hT?O0J#Yt#Yd!$-GAgC& zd&W2(ht+L|YjgJe425>2EOr}kwpZNM!^n5>F)G@~+b0SqF^%h4l>!MZ7qOokT&#lo z&Nf0QXU%}H7z3PBs*Ytp@tTvV0p$~0ys6*X@ncsDvi0!TK`$R7f1tYr&9`rcH7pkC zW%EjTXX|&~D9WRJB9G$QI@CFQVy^qjueE@8W%-Oz7q&!9Cf0EJ9$$Is4;LJf{HUym^dL-0MxbT<3>NjT%4jiG}`b#PE2xxgOmf3qjVkS&i+Jd)K}NOD7b;S9e0b1=5WbL*2EHwU7yRZM} zlf!cbEXKsptnq*mu1sIzV^YuGu&gpaff0-b(l4_6{P)+BfXHh{;hDDUbICRma2LEKJy`Z~ z%u8txOYlvB$%T6EQ7Y&bUMNZa18mb(ZGyo4(ad7}e` z#ybE-8O0u|^HeNrZ!X_hzR5Mc;Xu-# zZ45#12(pe{Wk00Yo%9VCd3Dx1u(1O;>uE18qG_94D1R*vtFOEGMj)`U&mu*&-=P-O zsraW@*f!L8qI3pcad3_hn3c}B7;$o2N2-&&+&13*Vwf}4dg&Sk2i1aswU@Wom3>Hspv}SOJ zf+j9i6q$P@X|scCXnB&Ilth-67Pa+^1rpoY|DHI-Ips1tNe$J=d+S0ewA(m>-IdL) zph{ZIQnhEXb9@f;?}Z=sN^(BgeXLWS>?iY9HW~}Asg9{rQEWdbgXl}`L^yM)jdC&v z4lsA0vZ2J-7sP9)1rQuqm&Xa}mv6OT$1KT?<4mj*m2`yKGV)xH zOQ>8YGss*JXTI_K62vldIqqoqSTY}oAPI5>X_tpQ3B~dX&uho!h_kPQvTg!XX$4j& z(|Auc5R&Y8U5#inAf-TAT?a+yrxelBjwRja)f43Np|u&%KrSug)nu8-aJ&JfC%cf2 zIe%`*W9b75!ncb2Hk@^#t(_7>(k_u_W%({j7HQ5(mnsI8!-H~0{07l#M$9tBOY&J4 zAdx)OUsSu3HaQekC^Gi;2Uf3j1&15M+Z4LIZ_z|97b?aCJAxch6?q4#8{W_wLOvjz zrmB6VAbc@mz5C!{bBK1ATvk{nkY~F#-+}Q&MxXCg`!6^-sawSe@N*3 zNvXp{5rLFZZcH>^Pi$-?33=P7@LJa99{~4e;QAjRyyBzEauH>dZ%-p?&vi+dZELQg ztxjaBy4T$p;3!4;$mRO*LS+I7F)C1PRS?B|_OHYA&bVTyial5sEgePdJBpM%&m5~aTtM}>nOJx+1&B6GTHr8 zMf<~@Oeg|&L9<4!bDd_^Q0(NpWr=pN*P6fweBn>MXw|GX03^(YuFX^pLLHB5bV`W^ zKl0}UoM>Kr+;<}AlQD4!XPZ>Ha}{1N=lf9m;YX|QZa1sanrXO0BcQtX`;Xo#aK|iH z4QD4CdVja@H7Ht?h@>4{mD<72KZBmh#u9VvCzZu$$x_TMoFJH91Iy`KDtov}vjXWe zk2m4QE4(I&Gkz7FLAG{3aAh?PRd<^o24|TI{E(|`h_+jVTWi_hDj6hkYlU!ruaqW& ze}Hc!A6$Fqs%#2yCA(Oe)8$0v?e)o9@&RZlm&VmVupR$;VNu|MlYi%V^5~&`nyTyM z`Nx*(=P;hlFJYOXtR~&bQ(l(WqzvbIj9;GlIuQ=0=K6z+G zJ;=o^D(P>~!v_?~95Tkg3>Lz}%*FD~)ST9ELdp^_wi)t=owUhaqeatX%X1z!05(N3 zn+)w{-ml;5-Kn4l1;9ufd;zwxhyV_&>>p!RAgZi-4Zj-0sc^qtlZul~N3dC0k~Fbv z@cmoW0-9;EZwaNVJa%onebF>kOw%BV|FNCXW2t{r&Xsa>^U3D|_t`t7F+Wd!>9&SX zrOk2jTzv9VHYI-D7&2sak`{>C3Vv#SFsPBqOH1iZ!Fh1#FP%g z@w_UbA?6Dy{m9aBvWf9px$0+lXI>03NY~xL^q!QAU<3s+m4E$=nQgg++W{4}F=|`# zb<;6i<*;t==vs1XClAb)CUNNSjjRlMCtBP&~ zDq#502FT8(^NtTI&>c-;`9Q^W-58Y@YWxhO&;9{$*k+qbPqEw((*JWnn^CN_T*8Y$36z33F_lrn5` z(;(_dsb#5@s~iH}RJFw6K^>Y*RlrC(S71b#TI=}Wqq|byq0)ekYzRM)UF1Q3O;R7$WY`&E8S;*q1I;u4YCn8Jt0LXsv= zf*mFdPslo|<79wqv{LT1#bYFt4?i{Any0L}hO>+-<{N^GDuv_J7+{c77yFRFi%T#1 zfp*6~5J}RP8xN+>v-96Ew`cTNbhRoN9hSr-$^?v^1432mp&tfHP3(7=hZN)P4e;1S zM>^k)SWN-wNVH}X>py;24&JO`oOk_VA~}D%=}6bp-5vT{1fhq@>}yQ1kYcN+X`4H{ zRLIXc=sE{4IV;8od9`p|Bh*+cE4H>i{-b;tpK6f&nGY8L`BG)Hto)i zko^y&a&w>u+oudL0xQLSe$JNCFfMr+d6xpktBAv>1R>1LVVS<=cuJ^|cYQCL$&cRx zb1v&@=VQ9GEV7o+gt%)VXOr$-$C|S5Xo}@W48Mdld(akEOKXdDaau-pT8=;9XUpxA z$KF@vbzi?ICu99SqSzoma`#LwHz_=+!IpI0-*+?pWQ3O!}A@ zd*Xh{Ch*Ru?x3?9Th7c5=ZcIYIA_nI@+}tU*JSB@bR*AE^80a#p z4WIl5ZWebJNsZufSQEc1y>l?$bh?=YrOWwn(35}It;(#$SLY6(c=em%{ZbJ^$q(DH zL03$~)AZ^A3~D_LKL4#VA2`KmkiIwCk?B;2KGzm${S7yU#i4HJr%yleo9XO**h&}c zXSYiZc%`wK25+iRPaoYiNI%p9dpq>vzb5`MEtLN#J=pZ-AfNPXf1_Tg$e-fGlP^U> zn!3Z$y?#y|f+MPgVC#QfI&ndQjQqpSOQLJ|eMCRHyDcn({sFl3)|j+cki9QdLi^zK z{oA1QXS{>lgzpI~0sgxVPhpw$U0$r*BJNw?f>6Y@jwLP^yUML6&q5gA8$}|}gsU#y zoHSkyN}@o4NGM&TM|Wl7P1xS2$pRNFfXjb?eS)8ByU7~}B*_8^t=$yOLzMpj62@I4 zjnIa6?M`iEt^RQWPSKF?A3IfU)-l{%dSPBgQz_RK;X%+n{9FtAyp zbsO$@fAl%Wp|l7QD#)!yOxRvqVjoROx%eAS*ZuJUKcjc%=qNSfeP$^|h&7j_R`#A0 z5omRr#jNOWft`!}15DP8kkwoRZ>oWX!zmwn+BeCs{>+Mfx$|9BL3|$4A4hZ(8hdT8 z?{x+5h(7At(}d5wMLgqAl#ZDO8{gnq&xz5G2QiwgT?p-B}57YnV$t`TR7lFE(Ne&WqoeEGg`*UsYp}sJb`{L73yqJCM z9J&<3`Ejpv9_Yv1iXRh|6MOGPuUwp#o@(rM2wIR4uMBlr&B(m8&8Y2ek6;B31g!k6 z55X0OK`wFde6Q!G9CtGw!Va*NMFDy(SVaoW<1e@3FlF)=)?GE_8T&4yeNuVSaiO7F zrL3Qbk==jQS>@7QzUQ|n+=k^!J=^>8?bnVx=m^nR3tmak1W6Q%ue3W-QwMw0*h>-a zjdkzJPM3U%h)|M9C4A)0dB|S8v|vEF>2wJ`DFO%YfKSj0@NvGyCKw>x5SGqK*yG?! z7}wvCbO5(e3KSZ>{)#M^bNCc>e576o@35QABP)j-qzx0dJ|>5LY?LH@HZWiO>Y5yF zm}+q<&^i;DzHm8lw~Nc_2N9n(=MTlBA*JK3j}K~eSSo72l=S0;VRe>?dNU!`8Fz;4 zw;p!+o?)@yqD0xJ;W%tYgzFgBr{$Q%{%5sXux00mP+nbqrl=(!k z{OGPmmVNR>9_s_m!IDq18Y^Riy>EiDPzCTc^Pp^im1Zx(8#rERLx9=_a^})gij4n6 zo0ps&tAU)C^aib9+KVqApddi>`6~B~Nx>uDa$)HqK(@GRZW_Ql&&_s>2nT(jfVPi_A-!TAn})FF-fy_Xem21~mq|-Y~FO z8bmD*FQ+Pr1e7Ao4PaukvO{&186dCJWL)jsvdB;tJz7$0^Yf`l4*q zf?gJcgnduIKZ%TQV5CF}Lib0>ThTyU#1`|?oSQX=G5x5T((uI&Yr5ymyiqv++Y!Sx z;sc^sv$T}8L}sIMI(I*&|8Asl$H3go>0{qOI(au*I)Ad~cP+3xf{^}(yu5BY9KvF{ z5$-^eX9sRrjd4?mxGvNjANP&<&U&7n+w?B?hXS4+GI zHJ^YumjS7UT(Ctv^C-XYJ!}FoGD?Jcwey2vO@FuHLH?#$5g9V0yVRgtzvu2%s0hNA z?uQY`m%N|m8}%-9wp-;^z+_bzb~@&xVb&x?(lqK3`cfA1u{TDXLTg7E zFQ3qG>q3uB0Udd-JQ+pkT+WTUue5W4PX(h+B+)E*a(~qDGYoIM3crliC!cv#2t}y& zJ9|ZMNA`}14%XJDec-lq{p1#DPC=Qc=#zsJNgd_e8RkR1RHM10-~B*Z;o;9NzOf8o z3!O-GWzYpxy=MFT#Vst*M8ST}!%NhHj_F5!d;JUvz*}C0QT*FHCq}b;})Ss zgGlubC9UpN0Rx&H`2;Q25a9|v-Pb*e=sQFMVVB27_b{sj{j@A)e`<}hSHcxx8oo? zIcpD@7j`h|Z)+1V7+SfD2gVJ%*1`<@cg{yBj@bH=ISKM3`p=991`hnY!iOW|{QPHs z8oF~dr%GP7I}~~A2J8F5bdEay>@)XnyvZ&}7Dc+Uk*wvq7{n~np1E-LHI5`@KmS05 zfJq-2I%K|@Sx8fS4zCqj?g6OpxORQRp?c|SE%gLqn z&s8~Ny%{tS&Fy(o;;{GQRHePh2}>|1r^@X!e(mXvT;1;GHb4A$h0NkPbUQ6>pH(A4DP<^SREMA?ro^pgb+=So;vR9PPaZnu^AL~xRLc59K3jMpf zDeox!YAHq*ZoqKz9}>*P;4A(Ri%)CYDvhTCJkCU&M4Gv_Jc<>TnvFXm@@0Z3!>P2G z2XGh6Ut<24EmHRvScz1w=)DkXVG<--#REetJOH^Y;22qP4T9Wv8$@mB~xAs`6NZV zWY1L^Q)YMex<*9kD*Jc_Z3-+4YZ}nA*VulQ z25uKKhdK{gqE*gKSydttNe~C74f@9tItrLpep|Lz*RgQNnugQbLqn5_cvJ08+`yN% zq+i|LJycHo_}J_7cA~r^pB@kOrc$zd9cuYR%na1JmuW{!!d}Jlo2da~*rGh#xI82q zeRzZ8q|ApK>VKC8TxyJzYQDQ%D-<8cZ2>b#K1yVDZQ!tIel3vnK<7hBWIuO5;{S!e zLh9Q4XJG!2$!^sqeNXA|mA){FRv^cl{b$DyhpUD)!JEc=V3x$Rb127J?`jL-mxLOQ ziwDot#>{#mLaE3Xh3$54lq`kJ6lnqjOAU~KnatIlA0@4d8S*6K6&pLBA=uk&OAsBn z$+&N;{-XGII5PFa=%`8_eMDag|q+2=W{7nj#ruk zbY&Ljr1wH{o8qablscLV?SDDmeAsG7vVl{W`|y?=84LXR4(o2*?Cwnq$BTUG}zd)7km4@w}_<^X# zt;&gW0~M>^77)cJxE>jZ6{RR{`sACk0}(HvA5AMi$*HO;&G-j!y%R58$G7o8B?2NM zvY}i(vU^&*SM%(5XFCDIoXGwG6kVua*P|dxq)@sc>WQ#X8M*;X-Ij%%-gRpk7U^o@RfJ|MqMxRDsuMC zQ^tF6C$H4LxG(t5Hk`0F5LeWH4cb&{aaeBr8ton}M6QDO6$n4@2N2y3FI1%dG5jCE z(T_2GYiCO}o%N4!8n7+pMP3d-ESgH!rfOVuff{$uDWVM3Eo~$%9<}g$cew9%1C0&d#`*yI>Ohzv=&^WYb7SpRQ97>Z< zq}A1}eGnz+p<^^v=FGjxn>>Uq&`UD)Tu_xoEfal#;{5mBpPq)zlK~0rys|%OCY$(1 zws03745K{z%sZwIm@PkbMDq`d;)9P$J9dhfqm#s?xQj3)8}>9J^O4c{W&6YMz&2^- z*+P|GcF}c0pshNaX(`y8a~>4<2BhzMV7=6SADnw!>K#*^TWY-C?x*7ZDJ{qT_xqi; z1nyI^7!RZ_WWuKF3L#^=3M$Rr%%_qV zZq!7bcB7bThou3N`UD^5oHewYbN$sNxxQxEs!P!=9fxK4PGHwMANz7r;?n@DZ z?_g>~6%BsUaS2H0*(tNFI{%(HVzWtLC8iR-{_$7Id74H)iZ)+r>RBJIX7 zUF;csrk1Uu%yQwb%z-hI=Gtuzxq*Y`ok{&Qc-UuFL_g2o94}Y5_1#v8{)s~^)XTBi zNd6Mnv{g}Rs1{OX-{I#TIjbOEu3g6>tG;~jyfRv2Z)0m^4hoZzUY zVh@nEYBb1QeYw40q8>b79?g)E>*@3;vZxbKZJ&0o^@+DXf?|mq9RU-QjCM&{`x>I{ zE>%@Uy_%k;j%Bj&8T<)iY&Tw6uA?%^U0LFo{>5vCnGv}#KcM{Ju9o~gvplmB3c+n@ zdZw4OLg_j4=P~$55-1-D6fh#wA%wozoV>hcyOJaC-BSGq$kiwgoOIWg2bX+4S&wBz zJs=C_g2I!4breOk7r>TF@j@m3`0Q8$BzxI%hq+QP>BzE6)#AfQvRxe}CU|sb!s6mx z9p;TVw;0Mg=AC8*3f=uVKKs99L;AzjWmN=(r6o|^c|p@9uE?%4mG_4!R7y2+^z&Am zpwA@rvR>~Pw%vH%cxtQY!Mm~B$z6?VIzM3$_!pq=j zY!*rD(yeQ0j$2uN)KXU4`P{bqdStpTXXMdnxBAPMlkj)q9W*~0jOh_B>*I=%hwAfU z%+FD_RtDGIS2%YAYIE-rO*{i;qp5NaJ7!$=)RK$b=Ln+Q1}5|OT75vUY%Py+k5b1Q zDNC-uENn$$e6e=@%vgm#XrH!Ov-#+wyD-n zwYfFikJw2SH)mzow`x#|zNO`Rs(ISq>}Aol5hpQr`msmP8ydAY4zpHR&X6~MbXZT` zU#31Y5bn|r`V6@P>6&(@6}5!X$9alzix?bPjNl{(&yD<00X*IT(N|v?UzAqFX+Em8MhcN4Zcw~a0H z!#w#JT3l;SMABX#Q1^YR5Cw1CS>%IUsnm4-)aDqdv0@|!?Oiv$r>T>k9`r!8io}Ma z@X4nN6hoa$@L721d-8u!i88dM)yyzf9#XA~>i*>k|LZ!R(A|^V-|xgaef9JdV=5P4 zKzAE4*)U4|*PM7ao#*b6c=>#X8TW;Gan7h-;fp+_O)A`bN$!Te$JXxm1E_TK+8d_A zl@*kG+oQ&ZB)IsxyaBlVX_1i=g5#D7@3XifA-#&DO6x^>Gl);F#=4sAATvySwp^9e zAVBnZnzsSnH9WGQ@v3mN`G6U{L)_8wPp>raf|lpx(o*^g?ILsEDC$0kdl73IL1y$qEU>8w)2-lQ1% z25oHTz*mTxG7~<37U7TwvQ7`B7^ez>hl8btPa;(b|Hdcv2?0=2Xr)ZMxqKekHJ$4i zJK7v)LDd(s_eD_STb@nF`vnW>y*@^d6 zkQ_xE0NG+8E)<8)xfQGR*G0$rVV0@|kB1xZm(7s2H{RSKWl^$_ct^D#b`O7v2gVE& zO^f9&MdQ!z-&>IxPkKMT9eLGneOJ`yAXokpNV=B5<$gmOI$>7)oMx=wV8>Um)oGVI z!owT#*=8o?cZm|K+-h2>QV>BCMU<0+fg|N7aRsy1loGe88Tak?O-|VZLKW@JN}~G0 zSBqMMsVTF8uz@1`NlQ9w>BOPl5Pk^y4ZlzU*VB$Hlh|&py5W% zUeDkpMNyT_#Nc7WdeZ!Toqyu;1LwMka*K$R8yzN3=_?A|*24?bV{V?`Z=hh?$$L2w>_(f!?c+t(~RK#%`n(t!@tdL<Xm zPif`8>P{VMwf?N=?qLyy5)EP4Y?yzC@+?r|uL&P1It9n>8Os3Ka){xj*VEvmM(!=J zajQ8w(OZ8d;+Zj4@=>auEKP~!`!5`?&dL^ID_ec+JnkmZ+$@_4W^)2tS1p*4dsO@S zy$~&Is<{xk-5&K?w@m(&Wh$-mx-mg7q35UV3nFb_qeUrJ`5^WyxsS6+e;`xUr4OctbWD6 z$greAqs&jXohUIAJuwId_Ue5Tn@})}88;6}tYVp=a}XDs{+bkdBa7}qW2I`Gv|u*F zdhAh3WuARZ8K8cOx}WB*M?*U;s^O`N*D5&1OyDKoGWuD6J^2r*5Vg!sAR1MLkIL&3%@IxN|GKbGryd6GE{9RPYY0Y>JDEzNtGey0OMETP)CH5o4dg_YaG~ zVUczoHR~rjJ1Q*I8uw9BuI^dtCwA$aLp$$>*3VmxkP`8+Tp&5hXAe*3#=o+oUFvpD zCtbDOv_}pNmD6pKBoJ{Q6Wr27a5)2P`6SxZ4Hm9pMbMq{6}~B#2KjBe9JP$WB{#W# z?}0C?N9p{malb{py#eE+E;YVCM*bSVuXYT!+ukW+U;nid9B=lAn1_^{MHTD`eonNT z$^b`pV;1eF3}J`~leGW!(C0i&pW91l*tTp>P_m%!=ej8qQAqN>#^lIk+Ut(XYB~)^ z=BpF>(Eb^*H-tQry*XXkwunNp1rvW~BsHviqE?-C&2FF7QFMuW$5G+F z$>}??$wVi(&URhkJ7D^_4{yx7qX5jY*8OH!y+!$xk6Pyd2uqkaHLGix9cgB&j-Nu# zT>uBKmEb!6$`e_>W#LCV-yD zN@MS&jR#w=-1bg!1wKX}d3^hk;MPZlbSbgdg(Al>QLiyx*w?fLjO5ix7x&(iw~x5s z=PRtAhJ2U)YrT9iYa9-QgV*1qSqmVK0?7M>)cX-uS^OUWjzDq0*v)IC>QP6c*v3&U z?px*Cb^_o9JD@?9hAYmW>0A%`Ss_h|b8E!6C+C-9Tc&{1@8$iP)N`S5g2p(0>xf$J9$`a`l8NuzG zV$0x7roSAp+4T*!op0l`mwKcMmZ>U?c@yox89WlaDd5!VeG$}sKXq#-OX_gnO3)i< zo$lMhu!T?jMGQ$?06>wplEmy)#@hN5s4r~ndUH$YjShRQCcxtvXuC zPQ9O1(l2MfxSmZRhlC!=AsgyN3WFZ=nH@s_HYq%-7vz9ydfTHk%UvpK{a;do-dPy6 zo|777;~UEOjlGyg6l3vjE08dZYrg6I0i$Skr&_hY(e5rGjM~KTun3Ge*ryC%XBozP z*xCWZR~q<(@ZMgZnp?eFSGKg&ZX{7*WpM}0vhqiZJGd*j1D%0bjD-M^iqbh6W^zhL z$lY7ew=%rZY4)vpZsZcjG-@WA=(lERNjO~b_`MEIan7OWi~St8&OAvjNHOmOf;)~s z!@nGXUweE<=^HB@63X6rW_zofe;-o5y`A8bVu6_C4n&|6#?8tPlx;)BeEZb>F>P*U zIwif7uDen1uOk8&IN~%8JU!XYNAVvc3JrRgvbK4daU)!S6cgaT z46HJz+O|D^LYmJ}o8~u5CVv~NDRJ%Fox6Ks*lZ`Qw2=Fct1GODv)QuNCdQRLa&>YAlwCTgSb&pS3tN z-Q0otP_38FzS*s(OtCj6JdO$Gmu`#3DfVkqY3`#A58kV%vfK2{b=#howw~O`tP#|3 z39U}|K|)T<@_UZeJ4IYqqeL+jH71Z181Gg$_9g~7AFWl&yHC=zTkOfI2^{4<)n#LT z$S|$PwNh?1Byv4y;zvf`9IJB~^MQfK{M1+>!3S>e51DF$%JrcmpDuY7ZyofOlBAKh zJ;r&@(y@`EkX!5J-c~WSM;JfhP}A?Fg6cV9lfrd85lUW5^AZ^`g5B}kH5+Z>c=nJR zIRy5qi`mhqnk`nvr3wYYC?^9Mt!nxZz=ASIAlBE?8q#aJy|fK!1UIa}F&rb0UMUW* z(&p23dx+(O!3Bv>IZ{4{+Ovyk(Yscx)b^PBRN}ajC)^tt^sh}G*2wkrp`Z-dCkN|W zq-0scevZlvP2zj0?tay445n*%injgDf*6g!{k!6|=;hgr0i5_je`H-QYgktH%Hwo$F!50a@H55=_Z<3<&lUDOgtT3NIP~pbApXgEE~%xo zQmlz3jH%*yWP`X7_}AA1fN}ZySJ%#RF^=B#(Sxfwax0aU^DQeUtvfjNq~sU|y2{X; zz3F*7(=rMKgA@X9#Ra?5FG~@EXi$TST2M1hKO&ZtRz(GzU{Z6fJs{$gm~d(tc=jbF zF8tFn&>V63)lFYQlMD)4LG-628O|DF!L$ z29uO`sv9nZfkGJ6W3@RCCX(g^rJ67dKkXRdr=}40gT|$Q-fBcXu3g>YG76)=83Zu zYKi1dG3iXkDRIUzO-6a`LL?anYL!8IRL3+ku&BAIZLA=`sZ;tYN0Czc*MQAr_N|LL zG2XpiuV)o06(UKfo<8-Hz^D_!tI;M&7@?NItQX>;yC$fR37=}i&9QQLsBQ{5t#7J2 zmJJ@#NhE!tMovu|QC3+i9M(FGY-3^_GBd$7#(Ll5RqmS434}(x;GW0xtsCMG##?)r zmM@k`S3F_Lu>9+kTkBuGH55oL3W*OeTsQ>t*;}{xSDlwX=d@TX}SIYvs$lw{{Rx;j`^=Pc~kA4j4;;d>Hh!?{vAJ4 z+e3GDmYTMKb^J2#GOvLCf<=5_=Zpp!KZVJ!s{S5yY&|QZjXos1vA9yn@|SlKFfr#F zS8*ZNj^lCu?Q)KU_*{WHkN0={|DI&x8U9sdASE%52P>y68I0Fjr* z-GSs$_<`W40ggD%F3)>mA`}mMZ=JwgVdv`ljG05D}XR~rg z0PW6d<5lU6YQo!Dyy`flv6k>I-9lwyX)?(eA%mbQ* zO>PZD?6vt=TG}aw_5w&B9#Y#%l~u-9ET?WyT=;v_7M*|7ho|kePu`Hl9EzGoqT(x~ z7UAW844vu-1!Y267-X*8rFQN|JA75M=Uq49oL3fhi>q8cqTC^p&rxN-SIz?lR@~?a z^AVh=-lTwBIVZq4uT7e2x8J(9*~vAwp=+7r+NA=5qy98`Duk#E2~}ha5nH2mH^A*< z;mlXogH6(}G&|MtQpZ)dSnY$Ixp*$2!b15Ak_xWjxbw8p^jWndq3T**t8;ay%^pjO zut?w#jO~Nr&Z<`vV2L>0xUojJOlhqgtEM_;D@|_KLFyRtVncIftXaw>LWVua@kjyj z7lPrIbb_)m2e zg~U^8x;50-cAV~-HQGhY0Kb_-q45`R&Q-xBky!GssOUW*HB0LjN$hVfrw$}4WdGeC}Oy>!JPBU7m(MJn71kwDpfwsA~EQ}G?{#=Ymqv{B6&{4W??Cc==2M#Tb*Li#xGvVH<{50aUzKn6OsIdSTQ4vAFXK} zO=k_X_t4m*cv1)^1y0nD`5Y10`PdxSL}Z!IDA3KGvL1Mq<9M|z8y1cWxLlTbBRi8A z_ly|$a{{V=%0G$3u z@*vfudS$e_u9ZF0tlB*2w#|1E5U9c~a-d|8dF|%GBfVd;fpt9+JwvKBv1F|sjkVPK zq--AxGD)y77#y4edwIq^E2?On38(bGP4w0M$~D%tsIAS@Lu)Fr%R5DEaD5Krp7X<>!Sj z_D$mf3T$~|80VG_NdR}-^{jGB^lb?4$Mj7G(#uNe&YP0j-pbm0TkS^kf@F>ciZnt1 zz&bh@DEtGo2<=w)3@PUe;vyJ7HdB6IP8jqs0FQCppaXhfi z)RRN8wYz|_#s=uE?TW_21?&zMHHhQzsCGCRt!gWdozZRe*6H*6>rjT=ug>f+tgO5s z=LHZbk@MJu$oFm+E!p0qNa`~6fV;QTE~2ceR_1x`-KB$3z01d9apUYeVzY}Q$rN$CYC?>X?!aV&T>k*4ufN)D2U^rM z+onimaBr@JBX5ezqBvIooyrzX9UIkl@@hJzg}v>xTW!EnW3=%<8--Qm?eeL>0D34s9emT&{{XZi==ZjEx&*%~ zq*_B9QQgRMbV~F5A`g&~GF%c#$ITbCjL6aWqBjc~MF8Zs@Is>Gt7VCI0z=aD zpNjo4(Kh14L6XW{Zq)wnx3e(@iaAyAQ)EG;nnZ2{V>_JiIo)5MwjUWao}HR`EN^GD znJt#W)>V-BkVJOoC1cq0hi=`#;;s4@<0^E8!rLO9Hp0r$;drc|Gf3{)RR?hdmIrSq z06XV7V0DaT7k!jzIPs(H-5S$U>RX%1BGzwUwbkb*Rn%u;^ADT>3pbi56^JSjJ4oH| zIf4}=}WzLS+`ZT60wa^$##z+5-=9yXa{c4a6bXB&j>$gyPJo2ul39O zZtwE;mp&A^BWaAw{Hj!PP-g@nDmfj4q(5kxZnR~2{63fI=#(X%m!{i65t|1tiE9#tF2EJbC{R0sdySJx_=l`42S}FM-&iQZ*4_oTF-<+xawZ1zn4p0$r|YKnBkI6a%7WivfMByC63i6c5#~RJqc%H)5P0D zWZKTB{{R~6SBevK>oCX(A1<+?4=N{MGEZ_&=d1Mosi^7xC8nu$^HSi>H7!DDyd}qf z;1Cdw{kwWB_C&&yp);HI&rT|TpIr_C2n+uY?Rlq``(fwCC8ZrXb#cKGTQ+1-X9=>jV%(Q`l|lai&m!ZV`K~pjYZ|Xd>T9HF{X45`Ftmkb>U)7B>G8%6 zOo~<>6l1e8?etvodhL%`SZH?EuO_7|>;ajixHFcK7mdj^%f_e)<9eV2o_+b8r2BMU zP>-c$)wjdVC#&v+n%0Dux{REx7Re!vvOnT6knWHS<0OPW{Lvxl*)L>UDB`>r_**9HrF#xX&s`51lt>jXxcz{W($n4I6H{WK=Wd`-&R>zY5Id&+ysps z_pEld%p_nz2+bzV;9#8Koc{o81d4|j4#|YMoQ5BTGIb5w85-YA(>x}5;|JPW;zF>A97Y+H_Y22<TO4?NMZEmhD7TvA@I6hs>0;vo*VmLYNn&p@F zY|_rV;wqU8u&{g|?+-5r>tA^MX83tEyni#Q++MA_g7RtJJ+KqDFs4@uP?Lef4WJGW z09VFcTc^Qosmi`JzUE@W2y#{>_8bMkuR;^OGm|7WXw#ogh10^cgXbC1K> z_swg%lwt`QCUlgq8<6>P^sVaeSut5lVPgbw#>uzDV0Vx?Zg2vEPb3e{v zjl&rS&{nOe>2g@ON5;_QH+nG~Ha?l-@~iza@K_{koN#*{L8`9EU!!waZB@<6 zl*$g@`gx#U=r=lu-SRVdSn|A3Wk~v9=lWDM%}Fn9VvOnm7!80lNKgL&%sKpxHEpHC zX)G#kO}j~*z>^YWcHu{Fu4`o_(<;t;u4ttnDmcjmNTpl!HLGeGbdaED{l-!(le zF(g;XJ1#P^yq+>`^*gX~KRkVFUBBo^uLf-pzD7Z{^&b8i$Uk)Aul*Ei&L1R2ImZ+} z)w*k4HuiT73ohmy?b=a?-B<%!&CQ5H*vKIEG`-U@YJfo@JZBi@tss}hhyrD&qJ`P3$D+~~85-<}J9zA4(nIbu+*YL}xd5`9 z`qkqk%E@sg*3PkKXetg11O8Ms-w_9oDa{@knY)d|56q9HPE|v+smhbvwHqqBKIr|8 zHA}fP;ig>2y4eAqC72TN9QQ`=x6qE=kF9-V!^wePEB?sKcMgjcz3sRPp=5?ro#0@F zV0$u=k)KQuc(1CPf%LAN4RA7_Vl%F3IeX%)CdDHt&2MIWkeq`DwrCDG=e;)|rKMbR zM9QgHqpx}kj&nz2%>rM-O-m*~7>Zs|MWl>B7^P-jKPqZoGfGOtipqrr8xkl;sTn8QKHLiW+BZmzVy692ANud%wjd-m_R5A+K(T46&ONr zDTv1yrPtn=i=5(sZXQ6U2Rzcza0MPSlZvu3pqps1uWE|0Et-fNcB}wTG`^vG6@(2WvMg!`;&Xs1tUXPS3p2LW1Co6!=C6$(5t4bH)+s@VaLwtOwfML3b|}1SST9;d0ESq`6#oFNe81G% z>{^A(Gvx2*W^?*ipPME8QP+wd*vz*YcCC8NHO?P60XVIzr9K?~{*qZ>b`eHK=5#Cv z{Wd*D-rDNY z?b1Ufyn2*(T3oTWnL-1&1GTZ`*m4hiS1eJY-3E{z?R+c#AQ~8x1mGh@o^>0;7l4+V<*d=E0oSQQ4>UaS`LNWDLHAyRU zx+fpSa$>e>^XiuK*!2V3K|H<$j;kcr(}Tzp$0p%}oaBt-och<$T`6H<)jD9)^=R#F zuI7ycFYoA?XCnZzNxe?q0od{|_5&ELJD|Q2b*#Ef6X=oJ+$8QUrLfd(q_>lHKM>l? z!@mYIpQUzP3rW+UnkVYL15&isVs_4?v8z7PxFAUu$r&-}A^WX#LM+Q9lG-b^zk_=A zoztH&qt6^R7Ywt_ZFwq2ml6;T^^}%YVSq!4KRO7~^zVrGT4b7}H~l-+R*kyft8{GL zTHUBpyG+nYyM6-Xk~#I{*4<&K+ga)}#i2#3U4{X^TJFt}Tgpk=Q3}2Q9IhC9oE&p~ zZ&c|%n7sMR9Ws8LxL{+M?PW-#o%t=eD}7J$kM9G`Z)r0G+`l5~X}%;&r`*kJHIBCo zO6E;pO}<#JJX{^7;v5+Bz&Ui?f-u{LPJz_grPaJvh`Lp!mZ6^sEiNqH%fux4!a@#V zVi|};EgYkE*z@MAZBE~*Z|tJfI#*bSsv$4qM|q~aHxgPfIol(NJa~v12haHq)vd2z z+4Zfp&XKL_I+sstA_E-P4{k5C`;@`vXh1BWgXUlzPBK`LM&Cq6D(xE9g{xfj=9_t< z^Q zf~t$haI6n1GPbK@rz}!kbcNsUoi@rvo*x!cUdmI0v~L+`_hjs1$2n8yz-v|M`C_!w zHIAK`9>ZN|p_fSj1kRTP_yxm>Rs<^p;0wzYibUE^q_*BK9l#7fmbOjyQcy?89evYHqfe$xtUr&S+1)HLoHDtyfU)D> zc|Vb>9-GpXd^5AYgTs*)`^1qOmWtvbBSgwQH-*}LfXDL++5Z4&D?0_T&@~M{8IaFq7`|JlkJq}7)E)+^XjYpoK&>?_@TDxiS+5=g8DmH=4iIC{1!e0szSm)a(s4wbKI4IRz2KP_{2=E8v2lcvr)c_2_o&pG3hT(WNTbV4iHy8LzO zSmW_JKc{0urfEDRqisfcV8GQO-;@yfW zvE&@AY)(5V_NN(kQ<6;hg4f`#pJk+tacwSI;y7+6wziE!?wNN8Ezyq$Vaby@077>h zwhQ{7!L-r!JreRuOKY20BN7W}rH$cv^5`Z$vf+|rp3Jbj432B)uZeS}htU>Vo}CxT z^tO+r+gXOVZ@R4jflTp+VAC$L-E2kvn#wEajO8kR7 z-;Zd{Jj_l8O+CNE-lI)KMPU@v%>)tL#S9TM3s{cf9FPA1)F5sP*+X-K#cX|7@lx|u z>ZqiTPisqA=9cCwNz898%&(Zqf8^RiEJGe$q<})W6|=?hM^IZ2_CG|9&pMizuJySa zWwpb&BIf8Z9(~RTS0LmfgiI=f4kYrXQFQNGbry%K&3JVwEd)^EZ(dFEvJ4kszz-(k zFo*)N3OwEGYqa>G)*5$MTJ<)ut>4?*#@{JoFQXUIfHs4&9E1gc8DZ^!a5GkCqWXVT z(^cWT5OhA5h+JC~k5TG7^Okqqa_bT)&f}1vNmA3FB+=?C6GA%W^u$Mt;-Uwwa_f*P7%VMI^4tX<;PL z+JN6(s&=aSZRglvbG-Dg$GtaK)o#~WpHbEIhCUltnu_X~%J%9BV=LlWL*$9@1GUt0Mx zPmH?fP2JgD>M&kt`dVy?+SbwuEeogzglW#=q;a^Y7(Y{8Z%F;8v_!IJweeo!TnOHK zt?(&0$}Z&y7-zd@w1(R$ZXi+xJ^T`x+zgsVKsc`7mt;Bpj* zk0kp0b6Re-(mG~|J)Wa`41HT9Opk1)=#U#6Iox--^}>Z5oMMfk{ANp=sFU}kV&2;l zvcqVXQwBT%8C)WF<90EgMQknc5b7^%)7V9(TdS5b+}uDUU`J!&ou#l(10+^b$&IdF z%R(7q__d>7>8(oN;uldtdv~IswzGftc%umtc$xfcp-BgU_>ST+-#D%AJ#$glbsw2B z&bn+Ng2Um19E>~Uu>tXneh|4m%~0KHIv-M6JTuv~+}5p?7q?Nc#(ha*6le4H6}EKx z6Ju`^HKM^F4H&e8c?1tq6e{D;o@+Q{a(ZQ_1d_G?0K$4yx~;-V6qYwO)};!F;xQtx zAB)StKbaJzyjK^nHNDl0+I6MBlrfaX^BiEd<=75Az-;^Wt)@Aa-FzW9d3R-dlgEDC ze}!Nyen0V+E3aP-JkCGr^cB$;eY2gadPg?8(xO|2k2l(tx`xDGFL4?Yc*%EcHy9aQ zap~z*7rO4XcO|x;qUic|gEGb^5}%NfV0+y;66P&@ZEWS%#nYHNL*g7@IYgZs)! z_5E2o-&R%uY;5h+`JC=z$M~7a1o^Um=Jc+0@r&T(H@Zaj_WDdQ-WA@8zm$US%fF*xnX`qfun={NfBn{g$ct!1S^ z7c=S_+{_;!(pUFofJFB`pTKJsJ$}i-8LVQl>$@)hA=E`gPXf5B(B#AoQyq`kR z?o-d8Y<+7}=^Ywd+tVeWGEE;b9>ag3B=*OD(!RC${qTpSZ8Z6vdXG!I)WW$DXci98 z-~dKva!3AL`+fdw(lmSRChe`3LuiIfZQc~QIL2I_2X1Sw6XcBN8g_Lj=;-=qrLFYg zA-&Y^ki77~eD0?KVN7$DAP=IKJmWNY`ce%#?s#QB09fRCN_kNsi};RDuz9ie3WJXLyOX@c_GA^5AZp|jEcJh41*35jWuj2{F4jG9)%){$k z7vg@B_8mp2x$7ONj?O&Q-1o5^+ z%91}YLHk!eis9AwWbO9NY+5d(I?4oiZ&T_`T&ijIbiSIizp|P&w~G)K%PKJ_Ti;&d6W$E?^@%Z7E)^ZBh=ByeW#f3^y#jH0tB() zB#CkAxW{wbXy8{K*Yq1=rJ)`#w)F7aepw*?wc0hEX8I^%7WW1`VZ3Ev%y2trJ-(Hs z^$x2xp&PEHX=Qp84bof0V37#)BN+PRam{Tx#j7)nS;Q~wOe{#`cJ0j~Jf}6gYdTy= zrQwcf)b?3C!2OMCcUIdR@rubm*+!2=CP=Fd;0mfhR-B70quQw?EM_)WU9=>IHpcPLk?#{CI+fIOLJWYXzos{!uhpLn8Tex7wSxgia$V z=4|DMuYYPB@e>(PGHO~&G&gXz2|LDD*S%Soqx8@AVumZvhdOMCM6h6#J@y{Ndyd_` z^X*-hW4Dgxxi{?RWei;nV|!$)922MUB3{Bl!T$hX%vV>KhBe!Td{N6S%vw|FNyurb zKD3m%{Ht0iRsu%mmX)#%8L4+Xnv{u11NWfFJhW!-LX(k66CqN0pgZ@YK0q|9)*2xCxR7z)CccJY6EwwngDKTNtZdM+dh=6rmk~d#3c5mWL_%5RRmHp zkO37GP1uB-98%XDnrcDLX<6!~(G-%Ek#m}AN?phqs=F*P9WxhtRw`;P^(YO-3F4iB zIHe+Oq*IZVITC?QM@qu^9MdRyt12inA18`&DOfxGX}I|Yt9EM;=sQygywd5A zMK+?KG#M!9c@z~G9`s~6B85g%flNh9r)pv`fI0P`8cx*2V*`p^4-{R-r5UAcWiT;J zFKSjIO)`e88f&oY=|w||3LYqrQ5c-ti=;7j2(vUr;s!Dsp)oc>Nj2yX-RGiM!Cn|Cbu0rI}Vnyl4&j$ zCsUCc9j^ZXOyJj@l69ndSn=Nd8$FCG(~vxtv%RN=G2^_UKt9|6FmOBOlAlwuyiXfd zvYsT!A2$MSick9TcK-lc<$Z0{!&1J2OK63wCjp5L$L`n$u6hrmId%3C2T$mdE~yrcskEhhmsghd+Kq_G7;KG< zgdcJbr+V#In&(QiLVapYD$4pISLSr=#T#VhNmAT-iO6m+=v`M(&|gnF_LJ0B z*OrJ7I@vTAFfKAi*$X4D93vbN_)7LQe-Qp6X}xO<8e{3U`ji(k2;zGr7P7cM+kzqn z{C697eQPPcEY=a^W3PI?T}NA*2(?KrH1#1_E;dE94lpDfdyMB5W6;`uovvQM*Iz$x z8!%s$wT{lkatPt(mQWuf`zw>qF;y{X@9_57vRlkxQF4J%t} z+U}8Z7cj?TsDws6v$hf!>z%pBuWIUYOGhhzyO17Upjsr_IDpUQIrDr4k+q zA!fv61KW^rNXV{VfALPkMbmDjvb30MHp6Ha8k4dGPc~!oWSnCl1IBx26~=EqAYSWw z!{x6o9(URe#LX;$+arVZ$@JiyfGca$z6k1n5H00sWxlnzx3(9NGEO9oTfW60lNlo` zl1k(rdsS<8*=76*^*6@54PRG%VjU{(*Hc#7T;7|M101I*oD32-ZuQBY8+Ea9hP$VT_OB9AV>MdXs}0Ix|#zNoA9!7HF1Lm_aNC8Hx<$6f%r$Vfadb z*gF`NInjC>LX!3)6`X!N1hJGT63Gc89wx(p;DF$R!>&+}LYnCe+S}@O@M!VeHm_)^ zBe1rdZonCd?cITAU{q}4K@={d~$0VGFQmhVc`ci1M zOQ>}_CbIqWC|9`&5*V)c<-AF4{)F@w>o^74Vaktg^+f^gzrAV$8IT(o!x2h+gH#hw0SRLSk*Vjw8lizA_9F1$$2^4 zdCGH0&!{G)r^j`75=!4O4ZOK<7sGg6WMF%MvB>IwxRtW?yt3+6iySdr$8B?|JKkI_ zS9C(+=3T3vFaz!^t33TvVbiyFaF>%%SJ1S(atTz}ZDI~cUNB|BBal_` zTCc<@bi2zgqnAZdHk)U7-X-fCc+sns@l^*395QY67;Jjh#n$&N(!E!r#bqmLr%Hm( zA>k7sg62;qYvT{3QnzALk zD71Yp<6pQ%js%tF0@eis_;942Sc?5KwCCHsebex(qcpyo=o?9|qFrmJb=^I0?|{qr zJ_@Yx;YM2?48}IVBo!Qtir}BJmq-5qwmocZwWo_n)AdU!Y%HZ>SHyBlK&VQPBvyt+ z8@*T5fDhKc1+Oi1YimJ%QXNhw)oqMMEJd{-n&2vN=8U06Q=U{0Y9-24jOQ*f9qX-9 zWcX{R*!1?9V+^`<+EiBVzGxj5Ye_-}<1wPhoadZYwbuUt5UjPIhY%vz-9(*XuG_g+FvDwuA!^RYi`kk65b1?F)2Cx zMB*j`-v^A0=~_O$qm6d=s4r||u+y|{Qs!F%{&rirw%W!NFByv2U|rpVTnu_wO>u1H zUn-#wiTz-8y<*A^qLSWiH%_>Ha_Yj?GV!T~Vz#!bR0fc|hAaR#FklA+V}pLs-EXON zM^9)L+Ww_Jn{hXfHTupPLZ~D%@~T3zt1MuVo$?|mJ0k`6WQNbEWWUt)JGZh>zc694 z1hP6Q@WR9o7!n0Y*9BC>qaHGv<^KQ|XN#__dT?pBcJbW&o7_Pyrd{ES#3-0<&dPEW zuLY5rwxn@^l5g3%OQYqjb?&t7OzI6C;#nDd7nPK&9f|_Ol01Yi20N&3++wVHU#j(O za$gAyaM)bRE!riIa-eg9BTvM{CmGIg$Bboxp*? zVo7&n$t~yx2Q$;D@QD!8Bz4rh3&4(YdBT4J13S!x3`90 z32z?-gEOe+H6X~u02>@&D%a8X{ZDB%u8;fL{dH|l#%LwA(5@f-+6i0mUEP7m#s`NC zhs1s`imR~vN6{L=Xk-^REu?5Nk@FG>0?BiR$b?A8DviNnGEUKua0PZgp7`;7q-$&Y zuGdN$Es*dmw2MgQOK{Q*JcJM8%Oig@jN6F-7Vn8x}S^yg2SrlUoBMiNVhg)w^{6ISHieMQyKi!r#X#_)=YUJ)Cw8JM9HxR=L9EK{+KzIIe7%qV-2sJd<1Gds6sy z)%u>Fr|Fh9Hh(M=dyn2z-Wa#wIgSV-juk!fSTO+e$)~zI;N|a0>8U22qSXYhm&4e!KNjCh7s&wL{{XwK?Cw$o-N|IIZJ-ujNncKC z^IN&I)+`?9{g=Adw0oA=@1v?t$Orx-!qNu7K8UP%^{#B5EM$$L=1Mw9(r)Lx31?*_ zc9TsKi4q+)A>hJ5$=diO`y8Q*sjklEY@5R71u zPni1Tm(EGAE`Byh{JkA9&k1}{w}S%PESJ+Qm7{{n- zHZ|)KyCRY?8vR~I0evy$=kG{lr&LyW?HkNm<4ZJ_^K4kqVIyYF^YlT%D0vytO*mXV|DQt5Wrw)&LjEj<+YhFilL@Hqf@g9FpBHC=Pn zT`kly#L#FhdvMHVl1rOY9AUG|?nwLd-0_;(wTIEJ?GEu}5q8DA@uYG{#0=p~ig0-J ztq)GNvb4LJX=9c2= zw0S*z#l)%!9>Iwp+3#KJB8&K(>g1|S{L{A8Wa(8JYw2mF1~_|*nGcGn2bhp>2JH6l zi~*lr??Ke{U2{}@OHmIr2^uw;%rh#e17`!U9znO*s_|QQ#;%apni-@{nZLbTJipCO zU{yxXd_+JX59^%gIj#p~uij{u@*PpHV=_29LWDB7?7NOIgZ((J*iFqG^N$yQ#VX|_;L zEy!uL2_pgaE4WzScx@AjVxt z7~{|%^{!&a6seHn+Os_$ z@Dr=L#F1mA%132qTms&MHPc7zXV*GsgxZFUI>5k#JSW6J+j0jS{#dWPe`T(&(t5W< z5=9_;wE)ZDh|$+4kjOESliUI;=?2s6%LzOP!NxL!Zb#)^aDtp}&UY>jsQfwWzlC@G zDQc5A@f09tz&T$rTzz>JqebOv{et+J@C!`q3kfZatb4BDk}Qr2H;AsJeqq%A0C-pC zPsXo<`j5jNqP>nwc^1++{31&5jC05T0Ls-8#?maBmeHlyOz;@mH!C7ADnaaPrRX<9 zR3BD$0O$J7etHX)h!FY@5l<)!M+z`{80E0v?F4??p*BD|6&;gzwE z@T;DX(c{1BOCciou&jZkT;S~oC;C$Q%mR6m^L8Aaj6W_7U;HJs`HrZ9;#ivI)+7G$ z6lV|N_GTYp%|a^A(vO5)GMzQkkwFBaSR_?&z+gWdHa!U(AFwsLOrU+LgQ1t@x_-(r zBFz|xY?VgP>*z@IIIFqek8@sxbdkX;v00Bal&#*Hm8m&<{VM2%*u00D1M)edy#@PX zu4n;|G$%Y}h~vEiQ^Be}ic4q|gb-;blvG@RT6P+vmNN>9nA+LV}7 z8lg!kSxq%5rDdTbQS*vwA&OQ$)ZAUo0MtCwaV{xnk7{l{^eICzj?~m+kxNIM{V9m( zRgOS^#+*(mID1o(u>erA8BveYnMzd-X_)&87a2~}WS}A9jGP)2V=>b)b4td0QxKeS zLZgxPqQ}lCN!p7ada^0d!f{7aKtbAyrlbc$iZUK)WNJEdKrjsvC;_&uOKHuY;hmU{bq7)lz-P-T#{^*USWa}Qc64Kf?esY%xBD6z z)8^t?jx#*A#NhjYdHl_CUcKuXZ!elNeI)9q@!$Y{!_u|hrtAGjS(Us?h6==;r-762 z@A=laZXRR&SwnwXW5n0WLndYT$D+<1Q|9Df5#?-l0ZV7}uc-e3WuB?h^u1{z)Yne9 zvbbHFu2N0$eTs$y)04+F!hRfSmRha+aD381C_A$`Bl1R`ZfzB13Q@1B3ym@5V?5L!U z6VTQ-y+?U}YiDknR;dy3B9<$`GpGX$>mV7GKs!pSF9!@)BEF63>)7=^kCNY7ib#4 zc#2F^IWYeV4UIcc90$$=>r0}v~i2(Pe3oi8-;H1NS`A!#mA zOBa?87k2&u&fNRsxvk&f??+m7LdNJ}a$#8~j$6Q5-58bLqy~;y&JI>Y2hihiHI(>@ zJ{N2E8a2Um&|zh3tu{4zqmhRq?kHs>!!ocL@Yw*hKuW(4vo964IM-%bC(=gs@RQ(n zjc&SC?xhqOj*)(R2fhd+n$}jzm?coe4oVe~bAswr;e!s}(;Xt+AJwd~xxBKH&7Ldk zm{2|4nNhhUZ|0IR$;)SQ5*u(R)vf$6wr;YrYx@Z%>a8wd$$Ofj#`bZ&JXGOhkdR9N-We*~s9}TL*Uvu?KWD8oTfKg7LQ8j> zTUCiKZ7il#xLH5qE!DRhmPcSnAZNtfuCC99-+HO+X82d3YWjWj`*a@)?@!gNV<{t| zFjhVm2ZtiA;jn@iJ`wWBx#7J>Z!O$cTK0u*^XYnm{_*e?w@4r?JYNm-{{RvfLOlZo z4Z#N+_d)B>%g|b$t*YvGR}%zz=Z0;_Zh<4kX5H$dLzck>8A%=UqRU9nhdzyg*GB06 zu-CQCBzW2l^2htq+Rb^je5%aR#xhh0-6oAeBp8?s<0kt~)!N&qwfnnwxBclI>fWeB z6r}b3&wvl#L+jCpU*$q9cfmox;nIM?@lPqALV9W=xRX(+y?6)30oNumeq?4@l?I6J4A*rE` zI3sEVojJ=O*OfH@@y$qibc z#eGiV{{XA(b!#P%35~qaAY!bJB2ZL&dEUh@b29lldZafqxq5UE*y>tF|K$u%Wm%NuWfY;^@baL zQaBnkmff(EIoMWJ9Bl>2TpmzM7$9vmZPGnRodMPzJ*){)*3(soQnXm)Bgcy6)6U%F z6>ZAOc?B?gRe0ZJuk>3clsfxN)isOjnU?l@ivFfttLhhC6cPndksC%8 zvw%p99D|d9d)AGtYwL5>7S_6q7Ute1iGoHLxuF-W9sGnj#k7;@ayBxlh6wM8K{9 z2V_~o@Glqfx-rPD{{TgPqt;W|X`c*yc$V);w$OTiSiaQnAH*@o1;p(u5PR*3u*cWJ zs*bzsNo^w6w7n)#EczCusL7|^i2neHRZs&Ygz>pnMU=Kb@K9s9txKhDueEIxD&8xL1@G?tj#27r<7)o#wx4^*T5Lb6`|^#cXO=SYjA3iD_^y(+@_`dYnd#1)QX?~5Ar2UQQA0PfD=<7YN`%S6atnmeTWVvwb5d(#9#HvT~sQ~+PTb6_SN8gKk z4L$8`t|zs9$$fhiQsIv5vj&X0Y_|lt001y>D@5ttFHUV%Hr6jz#R9CxH?tRP%1~js zI626|pD5sxJ$8<(>WwPbt{~MkeOFM`QrM(|F^g$sl|~2r0U3)Jz}bQTK5tseQBw44 z2{@f!PWaP*q-gBDO>Jv`e-;c|5+f{0__sHZt{dqfIL3Pl>v~7+x6<)x*E)U7HhPu3 zlO&BL{o}NtkO5>^u)S!TCAv~&NTXgz4#^Mk8ya?TaezOSj3u&hk3h(I^4C8~nzLy}>BXO*mz^i9*i0>w>qpdWRBj=(^Ay{AaNw0E z>$G!P-2GK;Ykl&NKy=B_vLvdgp@wh*HW-uX6OcmuKTo_d!{>v|pLn3L(8!G}1}DQp02VuO?m7JHo@lyk5Zgv`=%Cr|3sTu-DV6Sf3Cyq(rcCXJLiodhV+-e$y=8RzO|S>xJewo6rT zMn{!o860t)xjyE)zUyw{&UFNHo}ShEM^};wH(EzADJ1I=LW}CC7y`b7HM9+F zMuif^e;gNpW-%Ouk+_Y{UpWN&k(NIBHTB2E-`RUdlG5h(%EPH5njT(D*r9M=l5;B} z9k~9^1$>LuUkf#j3saWjJA9a&$OMKzG5V_Kx3@f3Zam6bW-`XsXfjJ<19O&a_SwaAyNS!6qiUW9 zve!yx6l&T9(n~au!5@yS009uQ9(_CG?^n;!=Tvz_%8M%gJc1khig_G*lj}$6fKBxF z))v=m9u_Y&NgAT(mTa8l`wVl8*83ikZ^V(>PGYu=K{F{=k2ZRn>bS;yTa6 zi@RqmtNbyO&ma%IYFGL+mQT-bMgZW~)Ot@uF1T6()#wx zMV9vF+`5%qxspFRf%PRo&-};UyHn4O$27Cc_CD3D&Adw%`K~;e>uBnPi`vmZT#z=X}XYDUNt382XCRSq3|LTbMX^QGDfnaN~*|54P32Oja&A9 zlyuAqYoPN z=9PEvQdfS3$JUpVnu3CI`qJ)cnW<;0 z8?AzKie6K}qrERDY7#3gF4V-7yq%~3J7SfS*EGYiBA1h#nz@5iEhymeNllDXmjs2R zr92uCCe);-2Ah!cN=@9=4UnY2YF1imPg+)578{SXH6L?JM@&Y1)B}+o)a*T}C{8g> zLVUCXahhf|+($IDbj)4GDMnAR$f>kE(nxz#NOMCU~~V(oq~hk-^SB z)z6nF9V19Z%d>a*&HGDg+U3l)dPMI9x48UFGwOXS=PgsI?lmixHxj&=K5sEOq~@C9 z`Li^Paq4M1nW2g!8tutHjQ;?YaJ}uFsVJ&OB$cOdv6K4M-j!|{M?`LV{cF83F08h1D_4ie_v8db_+rzb1+Z;-LE1g|F z_9UOkdQ%=de3@L?DRDsFQz0bsezl0#dja@!>q9OF7|E=ptQ#Ybt}C(ha%;1%d?M+M zKU&r8qQ1DhSlPxJ*_8GkptXIg@WVizb+n3j;gdw+5pJG2JidT=gDLvwgYRDwd=t^* z)^(WHc{LRZcSgE$A${JTiI@Cq^fTc1O6a#Xt0mMOO7|?Ao?E1jIL91-12F!AzJJY^ z7JUbU8;R5G`fB-x%{mP;O}BCc0@=)xf!LB%GZFQ{A6!!U`{_RlE#6!GGRs!H1ghTa zQXoc|`rFg<^q8sL3H@jt0L zyHQ;`Q<~Nb0#yjU@uCTwa?K0w9sLjGUQ{Y?;(Bt6kK{A)58@@)SGCkyW~9+C;_$65 zJUC$=+5i9)WcNRaDTZmv)xI#<#QXK6iK-zl&*m5vCcEH=Lb#AjHnu}URKamb2 zir!7p4s(>jAY}8wrn*AJK|QRK1=8mnrPQd>LoN@hfOlY?ag*Dz@7$rSoE+PgBkjN0 zZcP_o(^~gKxf%t|lO%IGUWQ2EaKx#Rz!`wPcyBx`#G^7~53AFvMDXTYAeBDtVhb)%=0KIN!u*i%MvdYmb$1F_43gTGIh{RwKBFX>*%ddPp>g$QD zwQmYW)HRJJ#_||tnD1E#nG9XXz-bZGXB@`)5Nt!T&1(4mk8+Jej9ZKwY7!T zk99lRXtMaz#L7#Er7@lO0~?i^JdhhZQz}7FO~#$oo-O^w<*k$=%F-yU?x3?}R`AZ) z)T-wMsgX!;!qKn<0HjwR_!sdS=S`M30^oYi+>Jf$B418{{Hf$W0r0~!KQRu z5qo_msTG-)-c3XkVGERINmR_L8@K#F5ji;pRP(ge);fD5outv!x+?EW)bw3KnITNGH@>KBR{ z=&^J_cKD3EpTgVlN7ve|_R&dXYe%~MMxlO^*h-P43SwB|Co02hD-8btc!Tu~TBfV4 zMX2fV!*K$w+Dy+J=r973S)<3IhCB~U@!Zvg=AC&x_s>1Fh_W~ZTo4rF%edpd7j`I8 zbiMYbkwp{_EYhkAes=>SF73Jc;Ea2Q_Bic|;;7-1ZJjIO#zfnug_UuQ#j1G_J{%Kg-8FH{oIb05I;LS4rv(ac}ioEppi|bi1Ws z)4>$NAWDVtk~7&CJBlwLE9@BoYOPmH)-u-6=TM?+-3ikE4U*d5NGG&1H<4V0V-Uw; z3f)2jcRm|e6Fo@XL9WLw`Ze9mjp4TgbjWZ*@Hk+32OZ6Q5z(4EbvBQs^y*#3HMRWG zM`b*+D7m$j5?4sFK`I(g41w6>fw)L>^WR zNbgK!ZG4gS2gA*NK7;6c*>Cj5*I<;3Yc<^Vvq2-qU2WeO<%3FMd>-;_IAWsy->s#w zp5Lu*FD1LvwJQhGwKkh_3h_R^@gnurDZL>#pJPTS8z(MMaSWn$3^6pLQTLKB%xqC zS2gQxPRB!+sC3|v+}qsQUBLp{NLVJv@sc(_%oU*u2=2V*whddVZZ#bvr}do%3t}`I za}3&i&aLt>s}zwiPED~AGLld5!}(6dQaPtw-Or}m!{P%S?wIB&ZFXSF5Abs8J21dj z_uLqF<5d$mG;^PdD)4n>^g=k{R&m{5warp7uvE`Ki0K;#nJdk~c zeLwtY}`T4!bb!O7O&MBz2k|luWs!kTVoV4 zyzEJXasCn$&eOQ(J^aMxy{2fb9GBOZusrrKyaH(v)NPI2bI%;$jyoF2iOr?_m+u&# z<`O9~ptCD0oyxfdy)b(Wj@(pOAJjE%X2(gk(~ZoZDxy2zm0WzgNJh{~;|xNAJ$sNb z%{$b3{#?Xx^|&yY|+hkFU*wwGQ!c5 z;k;Xfaj-qpfJk28+56+mk)B7m^u-H*BezY~0$j`4H zUa+~;bmX$WiVLJdc1mrd@s~gSUAxb2*~M7dYmcT_#cdlrmd}RU6#gMhp5Yka9CzdM zsyudW9JF?>j&o_NTElC3b*0_-+grwgWk6HSj}T$FIP#SwAMJLpyM7FOU(oKf9X1*M zAWIIZ3p76~egrKtcks)uQ+9Gxl~O^*%=xSHLQjZO^$hlHtIcuL3vR6xGL543V;lt~ zfh?eY7|CTFfWfYt)87<-dRh3eUEQXkX1gGnm%?k9LBWin+!W){2v9~wF{W|KNsJYN(@vP?>j)!vz332kD`>W#tUDb+?cq<|xApR~6Y45b?`l!O!ojV20 zh{_Gj+Hvur0OYKQGqmu35-7%Tky~lH8{S#n6&F@HAvYXy z<#++nmOyy!R=aK2T3g)O!kV8@{Fa+@923CTQMJR`mLLEmP_h&!)j(xncYxV7=jDo; zwr`C}4@X|o?DT6pPY(A^w2jPcBa+eFEWW$~I6mB;r3C1mIrEKmZE7aDkYG2BJlnpm zqD*>naaAzulA8!#d)L;Y7>zDfl1;$z!F3s7k5C&Q%=oK2u8P%Uy^B?fIOMTZB@M$B z4WH(G2F^#g^zU4aIO!`ji;TA21ZxdLOmvbfTM=Me%@>Cyx(pBHSvbkZa7Jh_TE(k) z&8g{^7ZP?5!c2eJmk+8mb_sv{%7j1p$I**3s)gvRG%k8E@4Na(WaV*Xg- z((SL1FleTZBO8J0%!L$ZKTK3g3fJhecAa)x^hUL!EwO^uTX>?0n=vC0@$ER|93Bom zD4MP9mA%7Dr^{)0N!mfWR&_bx?E~xDl)u+*ui9&S!=cM63_KU?{zdj769?~{R3g>1 z9V*pVP`T;YWjFzBpb-#1yyRxKPgUr{-LUNq#W@V>MgdKkMB)t<{^k=wq=cQNXFtpS3Gl&eznwPy6LTLO})MN)KxP0mS>e0{{Sc; zdR0$Rf9*Lfp@v8~}Sw%0$p^o5Gl zgW}!AdE-pj;Foz&@yGV-=6GYs-Jv;?dkq?=*DVC4B+B@DP^on zeR`{S_IF!&RJx1s0mtD01Nn+8&1Yjgg)dq!C>V;_f{y)$!{6S%ed*m>_iSqbv7O6c z{{Rjmq=W8P3<3I9!rGMjm|{PKxzGFDvIWnu@iz*)(M7I~g@4rAJeOW2)zUnQKv6tl zWbgfz!St<1tn`g1sBNddlGg4;M)J=MvZo)}+@q2{yqaAvCY*e;z_P1g6tlYXAE_9@ z$J&{W+UnW~K2rMNN0J@~k&}b&20^9JG!B{3U>!pYaZd6eaB^ZLm*_y|zfS%I>o8q( z)ud6xvWU7m-UxZhaM`Z-}PTnG_ zpqeSXRKOS+&UXEQA8Nig^RpbP(cX*TP#R3;g&o5*3S5oEcLN!yG=rM;R?ZNP{*}xA zG<+7&{w?Uf73H*I`W|7NFPrMWn8>bHWmzZ%(Jpxlg7^4adynlrq*kYEI zs`W-KwktRUcBH1@Q9J!TVBk9FtAS$)x7)YNo>SQj&0KsVP}~=t$IN znv14htMrUt0CGw@7I-T~8>CD&%&rg8njmchq{{RFW}t!rfbd4`J<8$&oE0>Ml&$ zems0|i(a~t(@kK*kk6hg=S#U-;#PSayB@@C3hV$D-mzBf|t!lL7aL5$Tq6O->p zo=o%WOOwrYB}n3y&m#9H^)y>(M2;6d{VNhC%#7z8Q~fo)wOCdtqMky)1({T-{dlQ* zLMq)KUjE8`9rIr=DnX^hr#>W_jnRlO?~G#y)48v=ulkQ$>0X+)8dBH`hhHVOoaM8CpP7Ax?PToQ~D;+2r!(Yti;NVH|joMW)_$H(6^JQ(mTzWoZNET}N1v zqK+3J;3;vmwmHVq!@esT*Tm7$T{x>Qm1Td8(c0ZbdvO{wVC11#5sxGsmjj*#IIRz_ zJ~z!3&CEJn*P0%e8mbFhYqKIhx_DqLYKI?#X&BCTcdh}cYxnw&zdR^z4xupFx4}D< z4nnFfG3mzKSEYl;OSF0U{EIq2S$@zn>UNir>9IDMd$gB*E=GYi@EFK9VZCvXdB;50 z0Mw+Ev6adpCnvy>NN-)huOD2VXu6fYr8Ms)^Gx`Co+ZE`cY6{?eh+%6)Nb9Qg4y)z zdr6gxc%ckWDxXH@X&%I2{cF~Y66o?LHr~ah3E}N&5H<)+m#g6?>j7@R(MDi0)9oJ^Fd+4Ofz z{7H{Z)Szo?X>M7bIPdiNRT^8Um@gwEW1J8{$DbhQI3v5+bv~b~-CEyhH}cN~)xytr z@-hZ%8B3sw<_O5j#_g~&a-vX(Kn2zQG0}B2>buaIhNUf~^kgEIRYi3q?ghBm**VV& z#GS=SEPIE;EpASd)MA>-_Qz4No++Z5*5ciyx07;VVvvwLIHfydI4V=bP{4R&ILZ|5 z9g8KZXM6a2)fcwiOKm=-YiAhQccI?c+bJ=K{06KrNfh1?Ba8vu8r-_u_rF+O z+1M3|D^8-)CcKfii5Pe#6(7DZ=_Eb)CLG|An&90N@h4E9tF48^&8a@AC9b0l#mfUU z3mwFOiBAN}56nziznhcmTZWy~T5gf5d^_s9h%V;SA<>0|nsl=iQ8BlkK^Q#wLrHI( z4*dPGakgxvhwgQ&xGeQsjZapNYq%|DmN2;5<||8v$9`AEkPP~qpGx_&;?G!I!_l`| zRMN>T_hKUy@~96bw}~X1n2*F{Y1&Nl+<#X%gLE zaRWOQ)>D#4n4s)P9AD#Rt#zw)Hl3x+L{9=mZ1&K$_~8+;j7B~7l_2|(nye$VGEd?| z;C`03x}=XS%(kdteXSDiCE1cV@yC?zV}gCKJAir@Pg={O{3H44Nq?WzqPupCFfgnP z(W-;(x>P?+ow%X=9O%2d8{d(+iR812RkfRtHp*p;MhW!unA`1~d)CX=`Zc2T{LtCm zOu7WRl>Y#`qSJ?r8ZE`qrYC4p7G*mZCo^&eD${oKPBhZl9Sh;y+U|mFbqHkBt@W8M zFOzGKW>{fYZHh~6fgPI+cmr|C+ln{CZk*6>dPeT=P`UpA`cP37tE!f|m6k>!UHFXx z@^g)dS&m3kK-N0SORHJ6Ym2LgvbD4Q&!*Z%E>bHS6haxs(S&(qi+5n&azkVa6|x%- zg_m*3rpmTAx^%Lv_Xor{oU<7MgMeeVjAVCdSORf@R8eo)qdC7?U&j|s^xd7TJ}txV z)h?r%3aOfD1aZc><%Dr489kZg_04uJoA_yA@gt`#v|Se42(PIkAA8id&FZ~~SxoDXdMYiam9 ztlVjRKYd}U*<1B8Y1Z=ITv>m@x=@VidWg%*9fnC2l#RX_Kgx4<^ie8ey05~u)ilJ_ zbsZ{gI^ya`t!|TI{Gptv#L$#2_{c5@1xqZL-O3Hjd^owezPW~79BJasQVY0kt+&Xs zHr0+)hC+E^B#KlM-9|w`^``3Ge`ya)K81f}KAbfb5n0~a!Gn1piW`4*DtvIwfPe7{ zoPxE4_b>D~=TAq<-ajQ*ZYoxXNYW5mUn-bCd!=Y{oz$lEsXSgRM zOBvvrl+M%u{W`bXq{inlDw7Uk-_%_5y2I{Xde@wU(#-L`}@mn zHfh|b7Xxfp0!Ibold^>j1Mc#bATT*lM$4yko|mocxRwYO(QTue=&Y+2>0}vY2s7nC zCHC^gpH4XruhQ*xHQn8!m~7TrHY5Hu0Ay|+nfLU?Z+F?x#jT@$Yi&*|yNgpTt*y&k zGOfeRi28llJ2p!rU zKh(Fq2d=4>7#$r97Z$fr36R9#ZdXI!-y9G?Ac8YhN2o&GM=hL$Ttkp2kwFfazy)!G z#~k;-tx_|TudugM%hgx6dtivXizU!d_z{It@)XJSu)zVhW3pCn z$Jh`96a7Mp*C31;6-F|9R^b+%x>zSC_y7i-v;P1Kk^ny341wRhE{Te=cDhVbtU@^g z$aY8>kd;-zW-LGD1CjTqF66SaOXQN_Y-O}%Yj(kq&gyoMS0s6q2G8LjV0Wuql#fT! zp}AumHwHs`q%x9@GqFg|90U2|y=Z+!soZr2u=cjhFBGtfCWbaZ>JJ-G{v7e;`g&Cs z&X=SYR_kf*QLU|HWV9-Zv4z3f0K)hDAod@hdIp)&{YTap(lwo(-TlnWS=m*c*puAg z5_$gs8v0lEd;O9VR&><5k`Etg3~Y(Qh3}U>yk@^fJ`?;P&^k^8g4WIpW{eE12&jZ- zvB2ky*3pb_Y@>rwxgVCjANvvfPU^*qR_X0x7^GJ@j>r}VY1%)C=nh)|^T5gHwL8>5 z{-6H<7xbxaq_OFpO76}#ZwcgzC6&|AvMjHI^aTCu_XkeC(z<@Ysx(7t6gSG{{T?E4s@CuOMf46E3uO*J54) z=_HKu@^Eptk2j~GrtKt>GOPG_;j(xYZK(8hjnH{?9Z4?0{{Z3T@=1@Td)BqOSf_ND zGRhmwAA_3l@?w@ZO!YD3jQ101H&1Yg^3H{nU~Cv-20P$?kF{()9o3qhmaA_ixRX@j zzudsfvZw{JyoSn;W578aR*z!a*oKxkP~$t#0CxMDvzaH;ExbhADBK+;ie2hGJrCQS z;}y#&r>1o&Q?z|8(p^~s$Zs@j(`f??!#&)JW+@`#NKQ%d6Osr!fyQvYpFn6D{-e_} zX_oq|S6YnidGglIfM)PI1;V%tOQN#!tlNVC(d33d5qvuMi84WPV>A}GD(t75a!B#Ilw)NB`Mxzja!b|+!Z%$4IKETMtijerRZapz>i zPR~moYg<0a_-)iM+up@vr(7ki)Z<{YydX$?bvSX#K9Wj3| z*|qgQhHRsn;!8OguZ(4mX`zt)JgjnkFms$^gT-ssTIPXpk;AII+Fj!|Q7)q^1CyM$ z@J4w50H1(!THIY{<4!Kp?@#c7{{VP^+cbV2{tt@`j1Xr4WG1vFcf-v4RK$ zmWqY!eqQDTM!+GIA2M;iTRJ`cxLs+r5IyxkfMN3W{<*Q=zVLGbe@JSbLYZptSnrif!dv~lyrO9x!KAopN zFk*&4hmTU;8f_!(Ro7aI!y3ao*|am0%#%rQWFPsP^x}`DX6X(!VRYTH)cE58WIoc7}bup^B9Gm6tX=i$}9o|=<~k5RgW zZWi#yMZKOzcQGX7k3i(-@gHj1x=*MqAnG#h6agP{xtXvzJn*^4+mG_CuU+Y@KDTv> zCopJ{$O_w9i5;e!oRN+iMkEXjo_m3kJFv;OM;26kFReA^kE%*i@XYA^$c>Soy-9l7 z{^2nz?Q_9$2;cX|2lTI?z9jrVk5ja}hegqBEN!d?+ll13MTXq-gbqj&O5EoeKAAWs zzIKyJmrT51vLh;-FhjxmeQVW=w0Uxii#JZ0>lv9qXi?blxm8;~m2Jt?F~*`o+^b1vk%sk zmAN!o7y@Z#IW-K@D?}L3oZx?*O^*~8HB!%@PQ*~1jw(_g%7r1o%?iZ(vo*H+?1q;zuty|?mN=c4=psunqEv1gIPB01&s13sQc2fj1f)6+*XUAHx)Mz zYECoSoQK!iqz%S$DVW-nN@hMNX4vFMwKWmWKGdXMl-zjYh0wwpW+BZj7~+|W+M$v% zz3AycN;>}lN|G34G|Y6e`cnx*A%r=jp{0=Gm`VWa+K8u`8qwoiim`rFgU3uIFC8%% zG%@xbKT0}K5cZ?TQb^20CYRPVXmnjVYpL4_8k0;X0=a+3sit0zx(PEx3Z3M8dsSPc z*(qB-KK-P=INx<%y)~p4E|y7^@X@&Z>*v^RPAIWPc!l`Nwz_pOnWGJUgKCqO{0)E3R=4zH24kbkE?rEAL;;sGH+^)=Ie6LgR7 zZCwdwUif8gq+n#6;NW9C)_G-6W$?!4k@gov>Y8PTPNST{2uY`zUnH zSd+f)DW1;cZd~O~P;xjRlbqyNI(=41X1bE)7I~)1KaC24r`>Wer|ZQZ{h>O9;t?#X zWU5^va35@d9#ugBdE?8DJ?qz4_?|!9zRk|tP_vn1x4(JsrJ7T3c?_u2OBPscmN^jR zn~{LQw-S4kT1L8)K@vvO8RJ(d7DXV18Nm!jFmQO|7(5ENUy0Y((uLMm3-{WQi`I!& zR|v(Idx4VUKaBZ^;G9ij@5=!<_0~PA-3oNBUA}Pq)Gc!WFcSLcy za(_is5;PiIAZo&PwkDu+Di4 zlis)8N35c@iLLcrQc0|TdR|Rz@PgoP@Z)Ai0Dv}`nEwEC1Cv}WwF~_c%JN9$Pm#MZ ztcWF3;JUFUFqzHKU;w0fvDR*Z3 zd4I#+sd{rkmsIK-Z9UJ1(#o;zd?w~B6t9~DBX>U7HL>UxV@v7XZsOt=ifAp6$7a#8 zxK@=nI6i~~Bc3_^4wj26eVe~XE~7U}X!kdfn|_{Iu991g#a4O5GD^FF<(TaX2m`i1 z7i&f8dy}e53SG3C73}SI7KX}vGV#W)4~3Dp%azFF@5gn>(0Y5QuJo-oT?^)Yt2KR7 zSGk{E)E;Y#g`9Y6IaVtv-bfk8C)TW?ZpyrYb*NVJQ_`VAu^AlVr5>Vf=bqX3*87Oc&2Xx+N~{oDB%g?S<8LP!{69ln;x3`WO=~N&Kn_%T5S5JmpsakimiBz&& zIW7n%B={yo3;mZPf!$c-{q56Qg!XQlp1F50yS?N1HYoC;f-K_lj%ja!K(nC%Bciz@QwL^72=)%L!6(Msk8@4U!hOlWdq%jFFl=a5G2}F^P^q zjPOs=x)!t0-UXDf#<0le&+`y7Fky^@9$bIaI43wHVzli?rY|(GvO2hl{{RVPAd&C? z0OGG1reiE_C#-b#zLFJ;F7g4(HqqZ5&u_hUYdcY;^v0QKc#=h9s88dqDCxX#>+{zT%nN#N<-N2%@=Y+6WXiaG7RO@c)P-XLSW zNMp31kOa02 zI(OX+I70q?b{-U>YFGi2neL0deZ{{Y0rdM|tGd!@E%f_? zq}UO)RU0lxxj$OwcY2ilMvU5p!`!o+w}`~x{{Y!F$n7a9`-}5mD&c{~S3hrB z=(9~oMuP-iJ*!69BRC3u>blzD__vZq z@S_I)$_sQA2wj!y6lk4(O6aE#;TFF|j-l;O%cx z7?t{4p}r?vbhe#!ZKl~zso59^YiXx@d#Dtzh!tCwLdpZK3&c)Jl>~wL{ipRgyOr1n zGm?bO3o1N=7(KDc03V|QGC{4^L-@geZKqnv! z2DvifwWG5ZW$gP?O8A)ZX#uo=x}Kd3f(w<5tasL+@~+Uyovbocm85mo*UA>OSkxmqjks0OwlwGS}MU4OjOAf zdCD&J#sZv-fTP?40M|e$zk^+0rF!6=pN=buXScqFe-*R~$V0}hoHIh$GNydZf^o=gX)yDgMK*uy57&yw6Ss4afh6kK}AS-FsH8gED z-7c=WYGldTZmz;icP$b!oyVT)-K&mheL*dYXGpr1-rSXQ8l;{ojOEZtDmV->e3EgD zVTS^`W}Tww`u&VLl+oN4w{aZOLu4S9-MyM9fblcCIS6}>2YTmQ=+a7^wmW4Wmj$)G z%y;Re$l1@%p?hV=b}S-g0RI5Zim&QErtk#IAMRaN-2frEqfO0Z&KGbID zjWum2mK%H91&o)L^g@qmF+YTm;TSRdpQdP*+HXo}sSprP73>+1hAkI~>w&qLG4;xk z*z-qyRTS0VX1<+G2UA!{WX9&}U>J_uNtqZQY^;HCocH#wbJtxr4gQ;RJ;s2I0uUO5m zY4B~!aXYUSCY*P zR{Bg0A)7z^+2jLkDsng{e;>+dQ(Q~{cb6tyjj*)+G5o7*&>a}JdDMW z`QTTnjqZ`<%`VP2(7z9Fde2dhPqw?YidG?2rDOdt4r}b+grBj#jiK7Zd(=Cx^)$l_ zxG^u-4*vk1b<27*)$h8sWIfxnAHu)G0g?dO^pUa ziWPkjbCW=j@!E(_)`0DrlCiK(aA+=Qj%X}<)EWYMQd8cTm9QySGyq)wl)S?tj^l$% zvF3=*OjbcaCnNl+iMt$9QXFQj(Vt)#`G}?FYAc)?T2tPtYY}-{icRUMds40Sp>PdA zm$9SBX;~X56^&a1NyQ#B%`YEeM?+VO1F>^W!g6UiPXe2W`KsV?DVWA-Sp6xeO$xL( z9|Th|cBN3~6x38DA(-iyyV9t8Q!#g;BS}VHc=n(r9X}Ncv53VP4-^$NWI5-x3bjGk zzA1?5XwD5C4G0=hfkuh_GeB0zq9MVn5|yE~J+VhaOCh5n`qY_?CnQr5(vh_@8O;FX zeXEv!(K9Wl!`(_|89N97S5S=CKKR+B9c!bm;=y%D+1Wk#;C{79{w%b6KPNRyhkLo1 zkMke#s{a5R%nN>%bJo^Rru7x{5-)~Cgzf&dt})AZ;<;neIwjeCZ*XN@z;Hb)H+P$e zMr@JkRk2=p2tXM=gW907g>A&;>q4d7yX;kF0i28l&*7^ZYpEb?L{9bxFZ4a}TGCwa zVcnb5)zlYDwE1%(04N*``_wc_?6T@C%caNV{1^%|{cE4yvpnZn54>ZJbcjGsH8 zB>nMPP2H$T2ppVpezjP-EhJV4p697<7}4$`IQM%jF34z@y>7sap!7Z6Su}(ZbqZkJ$I;GYO_xb!$8G> zQyx_A9;BZ5^c>ccsp>Zu@)_>;w1@(3E3^!HWA)G0pXx1I;#;|7a&|)aj8@5S@C&SEgT)$afsEjg2>!K2)ISaSkE94~OHH}74Uw0OsBh1)tnGU)lkB~* zx{6C{h$n_SfMS*zE%HcNHZrTJQ_rCugs2CK)aTM}ucErMR`c22VJ&u&>Ku^79>4|& z2q!sVk6af?)aSCF2M7rV`#pv|K9#ifJT_W|zL%(3NgQ{Q#Md_xs=nxESKF|XNWluO zyFlhbJt|`9mXft-<@W8eRYF8$u=A0UzgpR)zOcRmSgd|dL2_qkp&_HeQRPtGfNk7z zFxd3vZn)7?N9qhmG&99;qTx&j%DayvA48mdF;mgB$nRxDYm*9+0cDA#!21DLlcP0u z5*zt1uVpfUA)T`CD&)3X>FviA(Ykx5?X5b7-p>66j-w2*h+g6U0E6QP1S>lBM_=du z>G!Km1K`LyW@NLH`gTW{{{Xnm@H|7E={z~u!x%Zrk2ZT^w)WF)x+KkSs*8)6q=2vP zjaFBIf^(S6keNTIe})3G zkalehyABwSUO>ZWnueq#I;Fm;boy?jBw;1Hp=hSb^DnN`yUl_z^&C~-S=@B?rGKYO zqxAgRWQu_qCYvVUcPE&@lbns-qZtCUeNO(&cq?nBLYA>aHwn=0Z#GYuIr@M`IP~^K ze#^H(ZnbMmJDY2mAY_ff6U5AR?f(FiJmC4U_s6D7F={b2jMjuKmbVKVhvc+!1t8-* zAH+xaRcw~2acs7;Ot(26Bx@TuNc_2A6yO7#WPm>Ot-n%u7jZ`_?kEBQFC!hW2VqyF z%-qYd9J0^M05g<-8OorI1Z6fPABJ4der(zh)xXePF|b0k?KD;uFB z*eS{N?fF#>vX2OqF7VNZW+<5e_QppZpQTi*U~CstF&Bz&0ubOd`TQ(9bLsCxzR}tr z_q&;-nnpP+NhO$^XY0-@bLk5~2K$@C;Ky?s>;-aJTx};Ef{f<|wH<%0EtgS}d(A(@ zfXvbqwv)kd+ZZTT843X*yB<_?-lbd758|w-oOmTsE{SXvbtgMWIKlh>0Id~oF6`l* z9bQQ8cCH%)p83H)8(W_Hd~%SnmJHY*iGVzx)c&=+e7tk3{{Un;Dsm45dSi?l zHN}MQW~|ywH;Wy-GH#iRHvaj*_p04O8?72bCbtG8^0R@t2qS`hxcA0t>sPjmPXdtqxiR{}^KOkr?&WHFW7+dLdq8fNNNbzN^$yVTS7)~91Fyc4QT zZ>VcuIA^?Nz$+8GF@3ydP7kmg3Zc2T&}CR9k}G{e<(Zo2b59Y$&eT%ipD80JkH4)U zVING(tlS+}R#gl!ZzRhwUA~zqxPnKxVTx;0(d?oTOj7Ob7j`GGx<`s;-f+dw;Ny%H z2aZ7Iu+^I_qe<7gdiP7Xp7I-MZJ}2x%81IUGm+|h5%0xp<5hicrk+^|lI;06wBRuW zl0wIv;{bzz1_<>7E=6o~``G2xbUCIMfcOn_JiDa?XL2wYCy~gM$+$d$o5}~6XC9n>Bl_2J>3*RW z*84#b5Tk?0C+%EPYiM-O3+1DdJGCrlk)#D?^PQ}#k1rSmWaNzToL0#Ov`vg5hw(P| zUjG0>eJc#>md{7Umkz{!HgzO=n6$QxwtKc4AcN0p<_)RHK#jRkz~Zm^@tO4)oc758 zepRT*`(-)zuCE<&ibtL0Ny=@Ln&_m2S4a2<@e`(gAACpC^}ep@SUSr@fn-^sAKIW7E=Hv6=kc_{DoTogOAln{UN#uF^WUuC=XKK+$9A z%^L3Eq_xtakjR$Np2!&TfA1Vts~nrqu1`JiX(nzy)i@*(j!jxpy_AYk(RB{4@kDdU z6;&n2%$8=xTRFu6@0DER%~-dS#Loa4;>dSndM1{Qu1|uW;;U)NRlVo$LNJ)cSkjUr>Bi>MpCBM2-m| z>Fp%zSLSIJJX^=HTMXQ14V)9sE1p|vBcU&5qpgc&)^!OiA`sa%)0oyXltm|nUQR&l z4tW@@cT=-5PvbB1dC2v_>_^hN*Tt<~JKmw`{TfX&Jsotd9@kD`C9t=d@V+UMP^>}b zQ?>93K%0PU*C^@;{#tbicLCYG{{SkdQlurg?46QLaPb>NW|l873Ub*8IUmxtO)FGu ztHim9qek2eo-P$xk716$=QuxKO4DsM18XEwBfMo>c2M#Uzt*JpO4km2qj4w6_;7u` zoYb{w$w|eXtD}Ye_Nh8qMPqR)m)#S@5@0s-<;F*^xCDANgxg3e>d3MTD}#;I@J~QKp>xy_L1tQOnHvid`I*=b0QEh+IpV&< z_&wCFl4+4&UERTJBejb7jUu+6HZt)?A~0n~K5&XhJcTF9e6J!E(d>Bg<)(Ih7hAKs zmRoxbF)pqa;K3S~OKA!L-b;Cxz>xm{E>I6nJ!^I7`@0*f`-nbv_ouH0ViEkou(?Lg z2#Ng2;F3I~Z0&=b3J#mr&1a`a4C33XZTHCWN+ylO;0!1Xm}0~5k^?xw;~6zWeX84a zL=xJB)@?0BMW0W1iv40ijix^n5-1Wv#XnHpYahb! z*{&j0oR8)VhBl4CuAjW$bb2mRL|SRZPLwEIZXJmh6pYTyt(8Bh3Dn`NYE z+NO%u+Kd{F;)CJ~n3h0|j&}xJs+GtDD}BefuO(d`#!HpdFDg(Mu$#nZw zxY5Q4p5jTT2_l>j0_JY-Iqn0Un@mASmX$CvFETo)vJf3P&=@LU` z(@(2IXKxV9i50sBPCK)zr~}XsUMonxsH>|vwz;V38dzACORJgPyvl&P5$H;<9CCJ+ zQOA1a`ku0vnv}M77Lyj$!hU%J$yk}DhiLh2PK#GX)MQoa37 zPI26v@mufgz3~1k+la3xg=DyS1c>efEJ!_v_KEdA+*fuu@@X7dr^OcE3_oPqI)_J# zQR*9+w80{S6^ttuob$B^$%@0Up)=K6L8~%k$*3`?lZ@0iZmL-vK;PV(duC(-rD75_ZpDefAE}}- z=8;Akjr&Rt>w9KoPiK>m$NrvI?pm89nv*jIg(Fmn!0$nj=QRwpRkAMrv<7U7oQJgq zja1f=7HS(FDosOk#YzvLY;i$zlg&-J=7htXnu42T9E?)!MUe6IpfTcw&^MsDqR40s z>q0lasYn(uJix=~)dkEh{0#2?G9DqvDJ_w^J=9i7Q zqoMrhK*Aa_98$_?sAyHOr8L5xKGd=tulUhZnic43HpdiICYD3mm`2FOSp}vu8~Rbz z=`}c_#^;V`9DhUTeze4GNh4rW5jPYoO%F!dq`HR0Ug-BxV=VEN0-A`?+;5DIYJ1Q> zCVyyNjJxTsx4V{fWl7|0arLfB!v?=Vziv;5!sn;0^-E|A+-@u79e}UT*~vgb7-8I3 z7aDYLcd`vGDkf5B1<5(A7dYdJ)`0vkj2z~zY-KV?gLVP#eT8euDF>5Q+I7YK%&@26rHD>pK8?Z?s8il&0bwhXN;)8%H#5_e)1y>rC4CrX;Eo07w^gMSrPG) zYb0|{TEMd5NXb2`Peo0Gqi$te*wG6^8075oll3R&$GCoZt*_#)opEh$An<2522bMN-4>Em7#NQyJ3aR2KKQN~ zNqxkQ=%%rpZ&;nTGZ5f6DkW^TKgOuvb6|2wJPNkHXw$}76{Md(@frB~@s595u7-b? z4K@QvPm~`|t#`!XH;;#%F{AwaQ|s3W9AJj<*!Qor{u=6i8EIt|zMR&v`-O0O`d8=7 zD?7+qS7YjOd8<23Qs%&hhE|QU{HF&s*Bpv5;+D>SEz6U7KT#5UUlKL9SlF{Y z>y&?eP%ZAu2U)Ga}cqL+fCFysFH>-3L7*1UK~+EqesWymYv7|)>k{{W?Y z;rm+jOj@n&wza4DcK6W4={j%?!UNq@44eXTYbfMQ>CNM$tsf^{T1l(Dw6aFi2o+-7 zDVgFdGBT652x>D*hUB!EK5f;|g`h2$n=MDTKgDK*!@J2(i z${r6HPEF|?nBb$fY&M+r7XXo>8s_ zWOwARhiMQex1-?X^MP8%xu<^kf-6&~w3lP%nmypJmr z_?ssK^v!2nj?#T#E}0W8ga9mNJ(ySY5;6e0vyyO6-1n))btSFCK_U%~!0(vna0i2& z{eP`*vzf}v*!1O$7?vR`hHb8?lO|4iu65MO8NPOn3Ke462Zk9Pt9-V*abnY4c#6c+_9-f7LJ!{%?oi9pTaMDD9K-{D9{{ULn_3P6vw`)4H z9F|UbCuzYW=~Sy_-j10bs+M{+<4(%WxD|bw;{{oLH~@@dxF=CueAUrVbGiL{;<~1< zEQdg2;v7c1w&a}sD~{jD)*7UuWLX(f(bNN-EwlY0Yd!*b;HQHTH=4nfY zEg?BQy?;vGwA+%`ASI(+yt`Fk0G9DHfwT^Njso5N%@o%2G;+s+4&drRQ^yCrORL#f^`Xw7vYI!vmSm1bo)IK%6RSms8?Z`~ z<{3Ft%HsraRMryBZlXtN8U`yFK*R8Uz>8TFKpz5rdJz6?1Q@#`{)g!CY*R3~e6VdQx|=c}PT1g^w5+ z{LNR%43Uqgy;;RFiMX8uNtWgvK3mgm2BB`AS!0=)?BrpN)&v8b2JMlI=M{R^si*4k z39PR)i_M8V0WKq%2qQU1?7-w>8Sm>{zfIQeY=K0Pxf^^u_XKv&`BnCptu@lLiE|=I zk_la88BFAPkEU~h&w9>KX7QRkgkKPJt5GGr&7|#TD(#jjZQdaa`*2unaE!$Bk^%eG z8(z+_XmPvR#_*JgNply56Et|x^MQ&Cp6L27q4mS>af~TCEn&)sx>QlWJ-ASU^ zS~+1Rq-Dl&tAdqVuJc4P44q zN(DLGyM`<1euAFbU)ks4UcMd}FYfOx^t*L}?Vy#KNmfF9HA1pRHsB^f1I#;D80j9J zk#t+lQstw!v$q0C4bn^D+{@i970?_=>$reQ@UC((T~FdKP+D|X!EGC%ZQza_UsJsO z`K?VItWl~gkt;!LoNpUdgB;)jKp%&KRq|%nYLUdQb8V>09EuWTVvef7jl+Y;;P&+5 zs`aw6y133j#xf{@ij%>ii*Mc&tP1`c5O4}JmOg+51m>}iWXou(Q}5oggUk;d`KF`O zos)Ms_andBvGT5^PSdpV4l$a!jWljrHk@?(i(8TwSl`SiLzM(}VBbt~rwl!_gI{O- zE9(6;^vD-eM*6#1j~-o4F(gw31o$@&Jn@DbcFI-I=Pi+dK0WASFKwjK@1>D6=_E#Z z?d_I%ZN@+>s$B`dB<;=)ImdeJUkhM|PqddtzrDECZLQS&rPL%WX~{lhlx0Zoj2~Pa zSD&7$+3EQAXVRKqQrS09F?d?u?)p8CWew6pFQ?!;U1ZJ|gSdZWjukKU31ZVHklR~X zTWTn)9G2S4GPEpkRktYF$-wiXkWMk{T>jqEPSa1$MWf4WW@AJ~g_h#fKm5^>6>u_h zg&^k`&1~8RuLZw{Wpl12iz4hqw!BMWC@0LSq;62zK2o?mitzF@dvS}=cCqWaw8<8s zf1v4xbj~ANcx7f${{V*AP|P_#TsPjUW$CyrE^aO_noU(9QM&siF768DnND~haBJQ9Sml&bwVA(aiGc zcN&4RJ5{bP!^VmZL0E=(;PP>wzcg!Ewg;?6!-J5WPk;gso z00WPueM#_Q7U|BFhH(i?i>yO!81Eyn!l)j=a(?ymHk)Y^U5Ty-i78;)Jqvpup!OB@ zr@@^*;^xXdPX7RRj_I=18~*?TA6EYJcKY*O_&+nx@)bha*EDNik+ew5Z*J_z)YGiw z(4kHM?Sa_(Rv6&-ucU`Z&);SpvqL8}mued|ETeWtL(K(`^r^KC&OWtPUM-QT{X%IS z($3kMIi(qSm;V3;KUV(n?7v=n15>}$4XH?BPa5uEd?<5{Jz2e!dY`BRB;vhkYx1$N zXEMzp-z=nlHv4)HVdybXi?HxR9j&}@#DOA>LZS}90l=>@#bb6m)E-Rcq|1-!^B(jD zJXD#*1>f4B(JCP$WYE~uiJ;s{utg+{aHyq^_jMA~UYC0ML$j{c8M&^@^xu%fuKrjv{%YjR& z;*6$+Xlgddqp8L{DM-(nnNvboc>QxmLra6r6&ogmD(GQbwG|Ex0ZwR&Xjo}aG}I>p z-jj^veQBu9KhmWXmqT&34K{>Q2yu#PCaW15&&3{*hpc*TOF2+5=LGt3Uz*>vH^8gz zfV-1Wju!bzWWd1p=QaASFw256J*z_aq43JbuQax`xr{oTk;h|-#&N%rG>i0pHIso* z-Op<79~-}Col~GS3!8{-+}Vdb1>g^HTtZlyDGInMz~F=FRZ>jR=#0hylk$s8x^sS%cJ;na1W`V8QU`I1aEQ2maHjUWr&mU3;u&**{{n(zA zANx$;m(p){Yyf_gT`tFbEqN&Z1Z}l**tRqNpRESrLAa3{(2-Zy(F8~*0L6OIJi@2u6{N;+Sn_?U&^4{jp)Bx4;oC$4LX2`w1~)g#MAsj2yRL2 zdzwAZl0Av|LGCMfsyXE*#w5Hu&RF-Yzu~W4UG%QMZyL_aX)Em`b^?>Zu6WLJPkNZR zV4hQkTo7tmp1mGbtoO}2IL_agJ0@tcTm$A|jl(C7NC()~gV0@C&(pTk7ZINfGL}5C#{<|5`kD%A_qRss=|A7C z;gL2)3?-mu-ZqAAsa$B9xzqYG>U*dl`G&uiQ*Uo( zi{{=71Lo&}xD%Wx&24FI)wl4ADY}H7&FVRvt%$PL8Y2Rqm?E=VAgEQ&uMD^t|( zE-m!i2ztG4udio$C9u-$=0y+&0JOy5ki=tdN`)A0ti92&n^D)%9^+75G)V63Ug3i* zdqzvRCE4U805HuXU<{GRLXWe}a`ZW@Xff%XJv_QxD|K}>%R)69nInYVMzP_dfJ&lG zv9LR(@(gf7u?t-OXtX#j?vql#H=2#g0rJ{A?Sd8H;K)z$SmPP
a_rR$Aad;MVP z@hqtcjyPgTN{Hb^?HYlARY4;Q#!1dkdj9~Iu+pvCXf4!2W5St;XJe1V4hJOj^v)}L zF_y}0M#jKgywOCueY>CG+oZ+XA?R6%%Xj4a(vazlmh;UpWx{_A3KlF+13ZEUInR11 z)E?&NBo>UY$UKj?>06GEJ)5zL=gy@703XChYFjd+nKxK!nq`Ilo{a#4_B4uoz>dTz zRvTG?C~^lYf_WXds~&;TWxlw5N>h}Dwx;8Zx2Yd$+&ZJ9ZFP9}+aOomJ72vZC#>bL~wvs7Gs6anXo+p zVo&E$1sZMkUQOA$BBrgSn3-41qn~fBXlLQMeM)z@pB9P8DaPaMD`e7dn${%)G6FCM zZ_1jwd0DaIFP%4FH{m$^C+m)Cic}p-LVv*0?c#>f1Xu%U1Dtj|(t4bGVrFsx+B1{f zRIg-F9^LyJXmxT(9`u#OvR+%OZY6=sWburi)HJG8P)&45EO31(Ww?!3%m(cKFK@Lu zai)}rJO2O=lYx$EqrkYAl!=)~ZL6BPc3|xdz2}J`!OjPN@~DYRE=yvA9IF(5Wy@n6 z)Yc}NE)EZ{t5E(=H26)vgwa-HaT*m0M{<2?zGY-MCm?#%uzVunfFOVeuik|lu$T5K z_W@v15R8D$lS@R89`7;19fdfT5|o7tlAx21zj`EwGb<3dMg(mc)HKR|u{&?M7|!u>b&LxKulP4o?Ig8@+N4it1Bk63BHQ3gr}}WgU`Ol>RQ* z+@xe}2iNi~>4C0wQ@Mi(n=$NT7$-f)W*xqV-xbW^XJlq={ZW5sZ((4JQr%{di8Q@U zT3B<82F^;b#s)!BFgWaMM(W;^(PX&3ytUJxLYnJ~D;RFW*~KRH-TX;OZx6%WfLNWx zWv~dY#nJi}pRF;HXL6?4q-m#01oDzLvV}NFl=HbV83s0Fb`w-id$uP(L%GBhisbIG`%FHtatjS z@X#OkcnI@Ieevy0>D?uB6t>RtLm!OT8!*lX89r_}-QUpF#8PQm0^44s$*A7!luFY- zh!o? zw>LnRdwY2Xaq6#+!_GM*V~YBB;a;59mKI5ZFa)~k^T|A+XS7XIT`w6;g>_`+Z|xh;f8d5 zyKj};K*x+lY;qJFU^8cSGnUPHS(}ri)5lQSM>Mt5ZnWP8r@Bop<}$=c*G_IE$jJdq zxPM;b^X*smu)#gUC6<`_uA?d)^Xgaj_dXXKZjWvTN{&3Iu*Z7tx1S3vCsTKI4T{9c zi9UB>5*+p+fdG969lKV6eQ3WAwF#!MgFv;gb;|e!R(W!8L_}aK9^4DbP)Kgwo7iL0nR)%eBk3Guj$^JyXlLaW2WJ;NZ(Gg8E@T|rbu4( zh(E=4A)8?%>T%m>;>+Kj$C0q%hS8;Rtd$Lwmus&u}a>8_cE z&fi$K)MSN~{!ZTFGz8;=;h`B~Pxf>7t4AAYvpHo}k;iO$c4W4jz`3`!ff&18e5`V$ zK#Y?P@XX%4j1D<&4miieJv!K#)(fO~%l`lgofbBmpTreM;pF*`e@yyzPOh`MxM{8= z7Tr7@w+6XMR#ArG+jdC7#xsoIb6hJ_&{{1*Bxy*7R>Dg^aHG79oc{ostss?Xw&caM zew#ykZIg)OWn8Z%-1c9o&-ho?odu_;GQPBEXb6~Ov0O&ZTr+T2G8hfA|k zEx{PwkHsPP%YTscuch=vk_!k!40(9PdR`=Mc6r`xsZnsbQG?&q`_x^rQO}fTlR=V0 z+P%v>E;f1O6ef`8gHdyuTKf9mO@PmLGF!q3UGg>oC$Jvm`T@zPviwkHe_E@()tt*E zrGh2R^6>l5@ZiYEG7-;iLXJ<+auzb*>!0xaT`I`%WMQ`EQU3raKk&!Y`M!dzEbMLc z>t(mLiD89uLa6RfB$Mh%lwm=_3GKe z7%7rhJ0gg8k-Y|eJ<0R{niZY7wuJbMB%VXQSpdfNKEG}N1JD6LFi&$qleJ0T=|Oj? zvnI9;&pc41KJ-?lrA<&BsH1?iAO_9#CR0K9w@s55&BbUNg?D@ahlu?LV2drl8E;E(`aZ0 zqBy3bG^B0KGKY!*$c-?a=}6BbG}7^k0faQvWa5;J(=oLG%8|ttO#w)wDWDFGOrvMD zD3o}cClmuH)MYffnkrBSQ$$uY1vEu814#7BNse>PBC-V>@%O1MMOTD6(DyU}wTlwd;&uZm#f%0mxub4T5M-$b=o*Q_BHxD9Ds_c*1BfP zU9*zj;uVowJ;rM}MLnX*RDL6B8hqMy#E?#gIS}B3N~`VqSKq%JKV*3CZ)dpE;RU8A zX)s_vRee7I0DG3-6h zonp;U52X_q%`o`a0_&tc{+zUN@)O7xX7(QsyXxw`r z6tMn{P`H`&Jx)|qSz?fFk|!QlL+SMQ&v1Da4OZX9wi4U#lFag~--%d`Rk6Tq56>C; z4@xfSqk*loRn2f>5f$L$AkfX!mpEK=^re`PaC28$bSZ8D2QI^tU1_FrHAPbs>t2t8 z_op=_l)m{F1F$)yqXne_MoBd?(kmX#{{Wz(Yui%uIQjrc$v)N7#A(sC zmimvZa^IOw;@9-DfXi3)Fl;#|7mmSzAp6fQ6b@{*@+MjZJPS1UcpN}!Rl>)lDz zT~pQ49d=I^NTX>*i#oZshNoci%dF~7&m=L- z&l(p5V}9atidAvwcPacNQ(B(6YaO`x8*c+ob&NYm#7!^&_w9yqSg~AZb_WBRC~RK- z`R*<@35Y5pNb$kr01N^7R+=-DGotj>)}d^=hNyvHMiIi&zC1vxFfpIQjAPTk)~R~4 zRWe>1V2YwIDv-p2KAHajL03BFvMgYGYdG561Ljt2{{T-z?^-0%n~S3)gCv~bRika0 zjo6tBlB9v^D*7aE30Vm;U=Rm=I*8&o&3#~_c? zQo6TL!*zIJhxdKV7WU#cCIwTHpd11`(kUf(^S}q1qOrWaS*|YQn`>Yqyn}NU!5^v5 zda_BCcV)hy0^8dvU;hB@0Xf|PkIG_IEMyOy5;2Xv$;La@t*c#&javR;EUr95v91{J zDmWXn&T;Qs2D7OayVG@PEnu@)N~=d>DG#{^KoTe+jpj_8 zgOiT@cB*S}^3gDN<0l6k56Ygi-Ennu8FPnS!#K(R09vWlBujVN*oXyEsyxNB{MBFJ z)1o@9yT@&8d_~(gnQ(a=U9Djvk?th-r#t^7e&iiEV>N@-UBP`Sqcz9VUT{~){&K-Xoi13adT~R5b-U9K#TohWdo0TG&+otT?c8TNr}jjSc9LvWYv=v(e!4O))xT{+_J%YcJgexOW4s; z>@w}coMD(AbKlhG)1CD{RP_d;?+e`A4K~@@&1YFmo){-41e6oFV<#CPdR0!4Z@RM9 zWEa;zEf7^;WG?fvjuYPj17V230~}W1bW2MOD%#@0;Ud=~VzTHDZMg&lWnD`gFge2m z$nK{oySc}JT{Ugq@@A~zzmysIQ8#NVS7HAX{QHdv$c6;iWg$) ztn8$P&mnw>ah~9wmpm zH)6xV+*AX|QIptmJ?hEXl$92+^*#N%U6U-2Ebe4!Ly}aLWh7^uki-+-jjMImqkpH` z+B~-(!jHmf-d8OwTerr96M(8q5C#D!BxjnSk5OBI!JNe^^SMSz$0Ltgt83;53^!xe zv{q3U7n+sj#ujV0k|{If5)v_<$DZEbl_m3HHI0$a{OZFvByxW`MBN7%{69leRCbTV z7-Y^44Ew%7HYKH?=C8%?nn5*H@{Rh776SOqJSlj&71m#|ut zR$|^LH61S7LbQ>~MFd;qzL}IArdD!FFFC@6&lm%^w$U$Uw3AY~v%R#I;`Vj3wz-wx z$B{+}0zKJC&JYscMF%;c!xDeHLyfB(YT17gf!mg5b_~o;-NYHHC@*a-g@Ia83xxC!TQIfbLOLXMyw&!aXg%8Fc6G4N3n1cXbW=u(Z-VLbnKn1@W#< zkRg=rs=Qr%U5Y6lmV28KPl!IR$v>q86ci{y3;j`Tj=^9;Rxt{o3yyq zdaGD%Q>a=-Yl!Z0T{+qwcHNl3$pDu?2N(bmT~ndHIO`hniw!fQ8q&y0(ZMH|Pl|KN zEwqCn;~msz>s-s=_rf=`5o-FarIn14i0*X#2G&$*Aq0RXa>0a3!FGu}rH^6{C+Mz+ z_pczd6OADfyp_+CpG5JmDEn8;a4K!+fN#lUojwt8;4N7 zAPYRIeL@z>r`%-yt4!;jlY%RmZz8yW-bR~*_gm#~y|82n2fm$q1Q8D|fGPHcbr1eR+ z)>}@o(=PQW=3>%GWRge;TBMY>BhKTnwL%iOUlvs8fqx?G8qcD z^kZKwe$YK;oh_)uI%@t+I_y5^?yXTH@dg0h8AWd?J==Cj?Z-3@iRuXNp@#DM+S{m? z9yFKwl!EGE?S_?!iASRnzf9I^{k2T!oOv|Svvs$`doG}Pjn(WwEpEHoO(_l3akp*E z$(AR+NyjJaTsHSnw~lEp-rrE2rzFKC+)NqCIo?BKjB*Fp*1MNdUR&y-7_KdBwRwE9 z2yJ9`g&!Pm&RNJD?Z^QBmCY|^xYp#jxfZQ|G*h&3^zhpQ{+BhyItyIUdM8IR>pHEb zojl~7moo*DR_(PD@jp4>1Dfbse_3j}q-r!inZCJ*sKw^5{{XY$1pffa;LJzbEFXIH z{549?C(CyuZ#tX+dD)@kC&rDF`j7=9t#uxmp(IVHUqNhU;uzVpB*)nhWt{yK8mw-f z*L7^RvD8g(3GZj1$MVaMV>=d<{m`e{oYDGINa@2X>9!Er%{LQHgEGzk0Njw}oc&mu z^%22o2#r@+)^YL{AG)oN{{R)LC-G1BU)hI>>NF`9ujFG__0;&D(Wq-qV`fQju;eBz{g(9TPG&=4n{DJ?@% zi%*S9?s+6TVun6mR`yZ;eSqYUIi>D(mY!KHw6sghge7E=n1~@hyV!Rl`wP#$JJBrk z)wPkQmP2!T<7?bF!Nzm7M`d52`<$L=2wGY=m*(!aHRCSO7nOxi@~`MJ-Pn&n01Z1< zNud&ZP^+sj$3JQdVy4aMLUTfB9F4^RjTS-;0gVYD^d~hoG$(2T4slAzd(osml(WzS zdIX1qMRQBYXaKS1mX*0Q!#6af-RM?8S+kl1hKk+jGW(hkD>r&hQ^BU?Jeo>V#S0(} z-h%%C;YDN31^BGyNC`b@Sv_fqTZ&oDRRO5K2&NO7S~KMo!Ww`bDY&>4qEj&%*#WrC zHinao@l8Z>DnQIs zEg9mDjX*G_iiZ>gG*vVMYG|ryXxo}PngGgZi$PPGGMWI0Gs&W<%?W6OgFrBi#weRZ z3@`*AdIB~m$la(94K~{$gh3noQRPqcqNS|yjKy~2*h&MB^{FFqMn@H68Cx+O_1(Ta zYhgP70BYU2{{YH)r)>3mTbVpHlWeE|01iR5ze7Mq2N>p|eLaYbt@FTBw? zlf#LoBLYL&e#Vjgrur*R=`N$Vv$2*LEvF(xhB*PwPxE7(ag$u@_8RNkT`RA~lYXdT zxyOdgISrmMUrhemeHyp6b{hSix&n$~npHuZZSsFQlI z&E3qBC*UWrzy_)Gy+O3enY<*ph18_5OZa48@dS4SatH)g`PBANNdSh#GOD%|>PR9C|ry#e1HBBZ-ZuDsip_W9$!6_LrNsa;F;Eos!jB*8K zwB*JC?z~f$8jZ%0aAkxm#~=$eyr7upOoO%59!5vp5C|ZEYBO0wB0bfZZ9YYa2?fAS{{51B{Bl&?2-N zw6dZ3D_!KFbG?ysz|X12sr&m>OD2ef7bA7IlSz+9f?00NNb!)?aUHYV$TCYv{NZ!h z0DFL-kVqx2xpx+kdm={uHPaa8GRlVmKsfGLgXIUXzzc#n6jMPQx3=vXGOTK^zEjKn zbLsRSZca(9lUCHj+d*$_dk)-fsb&bE5 zKAy+ku9cYw)%B_MOUIqx#B<4Fpb`2CrM;Y%LO5?+RQyuU+ZnGVv3Ws0Z)(vhE7)i* zil6{(Vfxm=YZb(ZhXyixU<`hAm4=fP(*}@&ys2D}SL<687S7NtB|Le9?OnJ8k4`_% zse3hw40PLhq+ge4qM4&mRY#PZa&miQ(${yEw#_r!ME?LT>|GV#b13`c-mV6miE((L zO2H->R7str`EXc{InR7mPNf79TVOK@R1t9zDgZ!1LWG=>dB~}@%S)qC)URb@&R25h z1apkmH1;US%iQ*2c%<&;G8SWzkTIWXYfpw50Qo=$zoQ_Xryl;bc60X4x57zm;Awy3 za*}z$8*zpF^G3MRzG?^-)>K)e+}T{=K>PRLcc{SnUY(`saS594^5)s)m16R&FSv-% zmL2#dv5|%tJ@K0>LL`z{S=_fcEuEvHDvq4+5sjX&yHUsuXeS?NygC#izryqC1(V}y<_;P>Qz=xKZ1Doa%iK`6wk=Zu=z zbj@_y#U-!`RgQDmeMvuROHyy1pJYI71mT;|XO=W0iYlz3*o2oMsfx~ z_Q~|m*0y~gqhZuE87}0!jwD0Ic_-{EUTcfZFUN)%WkisHA}e_xX~@X+!Q^J8GTdmd zK1MsR2iBFjQ7*%?2s^jAt6OO!kZhL#`;27NbvU7D^BSrV!E#A2{*@`2wMdLQ?D3>= z#GV?)F&~4KE6+G1-}SDQ(wBCcy9*sP%))ext%d3%Ol~qVu|EXlle8`ykT89D??h|% zoh#KA8m^% z+3FG`cRG}56ju)I3&em&xSX<*IVZOV(;Q&dbxhT}Mxi4JTm21M^xlhW)cTBeGir9a zGsqil-SK6BFa)S}0PO_fcM*loNdqJ)EueIf(l#i7Zwts7I0rv^gc~goGsogWPcFO= zJ$bFH=ci`HdGPCAx9R2Ae{{ztlLL!un~NzHb&1Y)nL;r!?j4E1ZKoL^(z-60X!jQ} z39W8(31#u8Yj!x~gk$9|=mP=|VUk5&>C);NR)uV5)HNU6(urbANG2)*x8f)WDCkOn z%bs!0I5fmPDQT+1aeD@-U~VRD^61YPwVh)mVWK$j-A;cBHthcZB_Hh4jaI_e*8J>e z@MOqD*rohZZ~}z@WamD z*7$R#+>pK(AgjDiTy9*e4a0`#BB_3jD@O0son6&^N2lFbSm{lt*aa6CR&v6UN&={g zBPdAYL<7uYj4BM1U3skO z6GLJwr6U4$jm}xN_H*0;<|8?^=uU&UhW`LhePc*Bmzsh3t6Lk% zmf2;CZ~RHK=J1bfah$LlxfMxDnRLHHy*aGeopj4*6#8a~YasbHx7=rP812|IpAoS8 zaB`%Bkk~cZ`U|0=)b+a|YpBZ}sUnMmr)kzdHf`Gcw$(M*aH=&bWW9R ztLoQJcd1?Kvdr!Kx$JFT7;tj0aIt~7sKY9+lmXwIp!*l${f?v1noN2ZMVNG{?vY;m zSD8^`h9bFQRgUL;m%_1B2PB3$TIG|rbSsKC&WGtORkSN;+QMk9#Q6C;(pFoUTOiAH z%*@g;#uz&{pa=jM>e@xTpD@<1c9&OCAZaaaUn?6CoXH~$AU>W@eT8bB8oeZRr@LKh zO-fkUTTQ6F+ep$m^AOSisyP|Q(00aaa_M2GsNr!C}Lnt5&+Pw?P~Ip;n0rzh)@0rji(yMP3Y zz-0uHxyK)ur5fYqY@%gWj@8(6v?xDrG0kfg?8eWKemz+0_c|t^Ep#i1nL$bQ9a`24 zbPfl2QT&g_0NQsEoVEcK^AAaUNz!zFqqa_^ypL6y7Tl3u#P4p00YHKn%Wpn|h8%)1 z#eMJbZ4K3w-WSL#s<(?J?xM*uW1b>0IBq{W_%7yKPM_;Zby#&90Go(7mJ@&vC^$yQ zAdXdiJNom?<4050z9VY)mZICIx(en&Brk4mb){V<%HAdeJ0OqvgMr5Vrw5)5 zXft%4yFG+4YI^O(y7I8u=_+h(k$d@AW=8w(8SZO)>Fl z%Q2A@y7Dq`dHrg*^*ooWET1e^Klp|fh)3M_X8QLzW9?Z^U!zz@(JzKwGp6+BwK`ep zw)TQMbl)UVe->swtmn+xJo0P0ME)F5sbQ8p{Y`OyhgYWN@@wgtb8vUI-*4p~kgl(7 zGf#N{ECf-F%n!NkUH%-jaCwnRdM;a)p|HrJla(H{CdX{oteNIe7w$z2jwr3^Lp|!I zgeoVBl|=G?tpNkxu)x(6onvivE5oC~F1b3Ltk!t>bUvY{Zy+Po0?q#bEl2l8rK_%W zd+!Dx{{Vr~*`th)`KIDW{I&z?4I5s7TkfQ_w!D(c1(Q+K6Nk050h%&6%>W*Se@DQ<-76Kl}}w5U2kDxhD+YVYGXOt7|(OT`tirtnK2pwvX^d3Is*_0mVs; z00^Ou+8E-7HHs+IDk2Ojf!F{$3iF(O>P%>CXb80lvMOZcniEipAjXF0gHmHbcJ!ej zoY0uj9N^GefQvVJ3mTiAO$m>B05Y`!%@xi~1F#ZPddBP}{a zh26p+?#yvXuAz5+acKyG5B~s(xE!DEr?1@Nve2x~8c#4Bs-DB^Or!CFXhjR12`<1XwGs$`)0nu{iSqGbEf)=-&@x7OLuwP z7u9qoZVE`4^7Pq*n;y)~i9sH?L0VO5n`<~vb2$3FSzuqObVVzt{@g9To9j#&EV(zjhQ z+85KLyShuHbN~j4KvpDj8028^53YS`vM&zCucC)Yjge;(Is7ZhP(3m2+}5kAYBv)& z7X_Gs!9KL^tExTC%Ogx?RN-Y|$sbWwrX{)H23!xw)mFA@-HA>-Lx4s;wItL0Fr|YH z+=Gl}n!1POv60`lE7ntsd6HI*Wl_jR>}yN1qIXd&D5NR-=e;{?aE%;stVT!1GC0K! z_D9t(USAKwi*|510-dz9RNCdTK;zib!a<^!Xza?zzI|%)FX0~%)5k%G)?~;Wjy}YH zl~Wm@AcfBSjy;86Sj!sNBV@=MlAv~fN~1>UQLyPZ+6I@ce(=;ZPu^M#fu4OPE9qeQ zkO9Ud8DLbhu+DiL){m(`w^na=3ozOOvdBi_#2-&?Ju7sJN4B@sZDYOEZET`3_|x#x zJYcedr2yr^lE87>xUF~MG?qWVZ(-6eqSP$Xd3-ycl{W72hbpA0B!X2{isXP>zH?e6 z=!tZ5XvQ%N4L_p zoeuKS+6h#ohJAx;gMhyIAE)VBJ-qmc$RrDb7XbN=exB83rt2aGXytNZZt6#^X3^0W zZ(2`L-my!mRz_P^RE)QnD9IpW^Cp2ZMR#*?9O}})hF{3TB%X6#=_EJ$jPB(Y)q*H2 z-~=bvpQd@QPo}+<*;SPr49Xb}?lX_i{{UK{p;KpSY^0J`!pS0$_a925)GoYB)|9ek zMgbj(_pQfLX0`Fn+(u@S5gcUqL+Ia@D^}NZu-jTcD}W0wIp@};5cC?(<>a)Dx9uc= ze>$|%q>5cqFBCU~r)zI)=a2j1wcC4n5sDGY0OV)Vooz{A3{g(!!6CbMr4pEask)7C zne9|tBMkU)5lg9JidtB0N&=IKC`mLqH<(*V#SYap7e~D1)Q@D{{U*kg_%M0`p{m1 zs}+62*RUALqCgzXvRwJc@~G@8+d`TN)hQ#h%LMZ9e647Vh3gMu-qcamemR=4jLGMpR2x>B#Oc z(s3hN*<485ZVGQi^ufW;em(iBJ8MhGVv}Ncan8GP;B9a*ouvBxt93pkEhG?|R};iG z{JVm)ah_MVp#*RTu;A46DOxL*fi&oK4LFf3x^v27O~i6=aukovcpOw+8*x%7wA*N{ zwB&scP`-y*4rP|z4kKh-700^+yVQ(hw>29rVoO^~tv>Bu$~*M}TavOV5t3K~8l9}0 zjz$+eaC_6*-$|fRb}i)7Cb*FJI&6?S#>=0_9H%N&eNG3xZTAv5*B0=} z9l5xcF*TG#10RAIhFFOtw+sl$TpZ?k1rn=H>Km?->GacYbnAUvPQH(IoUeOu$b@Ik zu&k<}=OYf8RN9v(pLS-Dh;#VRlOPftm=V)~lxnXfO0~VEBo!gnb ztCB}!xDacnbl*{Gx_#N!H5{duwK805cV+e;1(bzBo>(^TGvfz)W17~wTFM=3u5EQ4 zPSRNPi+hxNjh4>sD{#!JlN?C-7%d?=IAWv%D|6E|4G&xD?Ppi&OI=3J$(fmL{9W?A zqa*mvahEJd90E=@kOCa1Bx@M7j>{eNJy$dr5LzwFcdo}uhAVlM?^FU7Tm=PSUksou z3*a0os2JROdrQ_u)EYg^OR3KL9(!xc*#u3$pD#Fw>FlyleCh+k>^RXkHELKUA(V`oT8rK zGi9@r+zRN1Yh58y)-(`pz}Ty}pJ9dn05ep5Cv~Fqu8VGqG<&Etb=LIxOX+s-)sw8IGU&VqE$Le$XR|5F0(OSLbptgodZWAPOKo+H2pStQTM3S^#CD{9hqV*ia$f? znxeStKAv+k>ODuGTd6!lZ>U6woabt&&PUqlUM6K6GprF zJDo-&e0x~igEh;2qCj8zQVYT%L$Och4P0rOMTUoJZFJi! zeLmQo#v53cBq!er0L@u8flm6~y{dV1-4to@AO0^{eT03WLR|frr9rwmH9R)IeLl8$ zCe}~kS_A(8*j^Za_Y0q`Y)o!k+4UGYYKUyDuy-1oH9x6eG;Lw?>WwAqJP*qqqG#Nc8ta4~FcQh;o=uBvj zLwW#DL6fnfy#_iUN93l+XrRC_EYpnj#!f45{~`DWD^9 zMN>c>NX1Mx4GCz9e_9bil;l*Fgr^ihXh=|VQd$_PuX+ILcm|53W1eYv`{If_Pz)|H zMoux!ERDq$H_v*tnMiG47^9@4wnl#RhW`L+^nsiLkG0p3PjAkaR*gU6kHURF;zpk& zH%L4}REEd5did+CKVz%wZl$<}+APA&nB*J=2EP5ayIGl#jgqf)J^tg|QQjnz8`Lo$ z+7(GbH)Z20K4AE3`yAS6nq`&Qovig`WtDCe2G~cTQ^N}H*8O#bs%fRRQyARZ0}r`nuUcM#i? z*}%Rv-)J3wsad9vD82-4lI0{V8j=41E^z+) z2LSu=x9gG*9sdALwbOLIveo8c1QzlW6WgR42H<}ZfIP3qBx642xwq{B)0%dnX)VT) zcDMGD1C%91k-wo*xmFx=oPEKs114AZk9QsK zoe}S4BFqNWSyToMh2^jp_ml zM!&o}!eLGXPH@4Q= zTxDj5#xia|i=F_%2b_%Y!5?2rQd^eO1B8+BBrO{5%9Rbl#yf%!+nU#}E+a-|+BXg| z3GZ4}ktuG=8=X<4K1qA>+??mvlh4wDavE!28Zy!W#Ee*HcO?CBk6NZiFDTqdO!61F z6*WO2J9~`dJ?NJ9X6JeZG8S$!PqhtlxfN{$5?S4&0$iaC8WMBw!Spm)^qKBlG&Yld z9FRsyKJ_Ma*+-HZiYJkAlls$RO_~;31J*JHtNtxA{I8w=24CT4eOqGs+}WKYa%ALlVQg=JpF3kJ!VPmS}CLwrsIT@ zxX-9HN8XF*(r48snlTX|BKJ@+n%sIu_xB7U@eyR1Ngi+PmFBd)Y9S7$M0^JQ4PR*{ zaT^yUqhJ+sdlSdyR%pg9<+E_O)#9GT46b}L4gr5cd;b7h*JH3Z(5Y1tDQ&zQ;QCcm znxoo4VP-0hzQgL9GrXASPV(X zAdV{XC5_mIkfQpwYahE;F2T2)H(tCBe>FTEbiA%oJVVP!B$oA32lK${5iqebF}gR zR$*Kk*@`#~fIUyOZv7*4%`MHmI)YirEG9ONU`!VYkajn>AP`Tlw|dR0H-xW8bkO?l z+h5af^etxA=SeVJM=j)H${Cg`frWNo6B5Q-3JUJq%WwuMZF8sfJN*_*PMOfA>bp(e zBi(8G5(}F(?;!CbMrT8S!_ETZf-6Pn-m<;u9abKp(xB4ky|eK44OYe}o;FEVMQF>0 z^9Du3fEWTtY}Vn^J$J6^I{p5k(~wPPWcMcGf8Fvmh_W&Rl|@$0)g+LrSdh3K>KavO zw<#x0Xw|ek-kQ3#xVF@7bgLNvz>+Jeqq=fmkW>axM;=nijCVQ3HEE+*Yub!d7?CCL)60=B_4Z z%Mgcl8F(9{B}-$3<2CfRNozK?LfSh!H?q{X<|dWCVnrO`rI8e>kOo38U=$Vu%meUy zLz?eUmr^!*UXo;IS??AR;=>yZi2h@Z(>BgbAiwDY9udrO(@G<_2DPrcT42-IBa zl4eC%^Y~Q*VNQ73OCIE$l4}UBxw7+@2VIx_0jKIST1zeFt9N-Tk}d3}J`!yWoDi8v zhH;Jw!0vgi`%&sg8cy2_n@Q&KY>wTx#GssD$V!8fF`WKJwEnBqZTbev;tP2#^t*>q z9G9Ak?{*xO+@l~V!3(&t=KyvGSJU;aLsHX>`n)$5x@6842DoDk*DQRymL%9lc^L=@ z9kX3cDCpt*pC)v@Humm&aMw4|Th8*B&7^ULU7+#jTmhePky~Vz>}Y3%P;DIAT84You!Z9?D+U2C%r0{{ZhXT1f|q z@8&4KPW4FI(-MvB+wV19J~(_7vyWEKB_fF=$j|q|KYnXQ>nmG>6zw&%`dy<9-Xs#j zh^Nz%ScCQ;*2{0yR(d_1%$7C~O>8-T1=dne_Z5jBrE^ZO*L54oo_~~;Cv+#nc8Mh0 zjGxD~)O%D;N~l6K&zw5bq+!&$l6SJ}3)ZwRx4^i#-@BeWHr!`Ef~(i1wA)9SEQOWa zsJmlEw^JjxzSZsa$@|w&*1R2N-UxMT3t3>d18v-nj>LEJxd$UY)uHS9e0D1>#lEPi zwmh+4F2%{lPjmLJcXf7k#-53MI6-fz+%GFE9$YUOZ1Onz*JZPnosLFIk;yr(HSmVr zCuuH9x`kq62aKHh*KN?@4PuPk?j+~e6|aZ6Y|G5j(Cm%PM72s>7U1MioPl3Lj|_`Y znADx9WvW3F-mrUCY9+X!FnAOuZ|hlh&)SCG^adE_piO1FR1azc%t6VaF{w2Tk9rYA zN&b{FYHV>sY7#-H4rs1;A6g5V08D)-Zfb03PIEvB&S)~60Y!QQhJXtnY7_d=7|>kz zpam^1{iw1YXbxxqN!ZfPXwtW(F6Mv?l%$8fF)6^MWIgBw>ttYBC(s(YK`?XalLoG(~C(nwpvcg=#W3MJ%R@jgPee%9<+B6~9Uip=+CO{{V=$T(Xb(kaPL2e$?c7ZUD*l6|L(3 z02I1cM7(P&jcN(?74KnU8t&wMJSc-O{KZdY5;mJ_`&~lRPi;JR5T4RTcMs`Wf5Z!0 zYwb39q>b(%P$NDQcF!m6>Frwnr_>kP-P2s?9TC*4cHjJKh;+@~+q_v7n0*-!+O<3D zPm1&0kDcirpPgB-lTD}rGUxkV8WH+pvvnq!U3x3<$Qtl+h{M2*!*+u@chNB5YV zpTEC)<^KR3v;lgs-PpNEr^~m6fhoZ5Gu-3Q5`Mj_VZFBaiJ;1HX`#6-krBF{Qm5D* ztL}KNd)2MgTHckY+3FWMMXW3fY?RF&;0}0fZ3G_Mh$g;bo1NM8TH<`E)OvZfzmP`k zPDm_M1KS7w^G4~*7Rx)P*^qfuf&u)!t7X^hCAYe1UgF;FK-qPQa@(=oHyrl&ss_8B zJAAIhiALYY<^%dyvj?Nho|`kT5#SNwqey@mKsZ&w#(mHJwXWUC18qpA9J7GG2A-hm+savRJK017Js5LID> zK7)#dx`vQ&c8|r5IIW57m8>n|Z#{v}l-2A?G8_;e@eXRdGVA~j2V+EA<|Ut!MGs$ew3Zotp>$k2Xb#LTs9ep7#_nEgx6^E08Z672kBK; z5~DEzhu)<`hRZ99)RZVl!XlG~##oQ3^~F@}X=FV2HD>)gsM+audZen6*<8&eF`pGT z??9!pncSlexFi5`#&Bt?T@<7$#Ne^vM?ZSkqQ8)hhMv;6Rm&1NUwXQ<(UlogVa|QH z6@6!<$EU#nbis1F(oZQMjQ6X^Y@o8Yb+(b3IanM6=E3X_zc@9MpXk*=F3PkgPag~8LgGA7%*Vvsx!Au? zttF$%vcj+(p~(QA#D?R)^{QR=UXewn*c+?K1fRl;sb8k zVEP5CSuNP=Gg&I`p4%bYKZ~CK0AEUe!$@EbhEv+TX@`CJKBw$lNd)fCaq)&0kGyFErJbW`ack zJagyHydU-LO4(@9+};Qp>P)KhBJuzukLr6>l)q$gOy-yU8!figUEneDu^wO6sy;$n zX(UO33=ap;*I;W>A)ei`_R$d&6&=oUPCs61gQ#>=43a=rM0}N8=VFgvwOusK-apZz z#Uz%9PEQ;YOvR`+;&)*7$f)Z2YumX`H$0zeLOtg`Q<3dTMt0MtF>@b_c^M77anH4F z7MJl`__4YNjp5mkWA9q3xFB)OSy}24L1{MiCQ>%5XOGg1(Q9LOa4xB62mUe-DeaM( zq_mQKBKj%9hIrd{))@w@p}O!?7E)Uugp6cRZWvx%>`{g6qN+n_gS*xmM_1tIB!THq z>AL#u@Xz8PfE%#ls;9A1j?2McLrMr6AYiFHfPHFe$x>yF<=J*TocfA(!fR7%1HxsH zcTvHr6XGjjff%NuM;*O>wG~8Jo9mTHN~)9|fQoBKk_MO=)G}unZr{>|G{G~gZpl7` zin6n`vay*&v%bb2W{Vjgk>J%`1Z)p)E~Tf4tTgqNBR(seNLiVEGUwaV+O?G4;-MPSVg}sX zP70_Xas!O!k82_MxY=HbIPV!2Q3$&bAWS<=Qyr|r|CESAoE>x_N@1ou7sJQ zg_b!YI95p%f)M1WWI0C2!yJs^B}Y_Us%J*hemrX$owPbn#EG>Iq0%p-!VOL1x=_Kk zmr*J?g~vUxc;JlWS!wo?SqGn|^|&sy@h}Z!OCzLYfN-%+WK?a~+t`3k1}mA|X>Y3N z(IAUX)X)aU%SyJlma?79L&W&sp4sR5Kwxkv9WB(idJu}t%+~Oh@mXe8KnsD%Rsi7o z_fx^D%GxaoS~>;Jhj?u6&D3^xaJ0fiySuwK=oOR(DdTY9fN%y5N3JW@hp4YLyIYpj z=am!)VSBF$Vz!7d+GB|Z$iM@zZ?Dptk5tk1xR&bbD|SnP{CfdzL~uxRiI7HjCmV7A zBO|f&zt%2nb&YDnPB#{PLW>H<5@|0aw}1i8c9uEpFbAh^ZYjz1cSRgpDDJwd8V1pY;i z!aY9LCeG8g=25>SU~a;%UeVOX*B&o?-B)79N-m@qWXE1cd9tL4f{5` z{a)Js^{rCo>KWWOFd|4DhJ6`MdGEB2)y{g_H-7+G4L?!0jFBTGDw6~16rV95o<>h$ z#Sd2MZ4KlN1Xi=e(qYyMm@WdgFr-Nk5&_5aC>)M^)O9YSU1}L+-6it?s%_(T3gMp(Z6u3LQ?Q35 zXFiqNt>?RYZwW@|@qx#sU&NmoJ1ONt&<`8SpL&6lbq?v78y=Nu>y)7eW zG{lDjl#taB0_K;H_oGQmNN^264I6!Fh|MP(Q!%wl1F;%q4I>%kQ*pLvz{=AR8dVQ8 z$}j*_w`dHZrlWdA4KR)EMc5xhOh$R6;(X$njk8+;)HK2#)SPD^QxTph2N2QJ(y^T4 zj*UPthaA%=MJT42M&f{BO%Y86PH2LFGSOCoJJC{rVLuHOO$I)~h_nHe@kAb!C8B|v z0E<#uii(&Rpd_ZIy{aH_Sy}+8+KLY!S{imVp0lmm=oViwbkFB;98koirJ6(dQZ*d2 zA7S@60Ga~ENmAY?mRDG!MJ(>@6;ux7lh{_7s{BTt<4n7~>FXt*PPU1qlUl#rmW4pi z@Kkx{C$QUvC;94gX)CL=cimw%ycWnq!=lMD{JclFX?-Sf_3|m~3G)>Cm#p-?XHwF1 zt0tdSK{&j*O9)zAtVA*{dvhU=*Rq;gU6poF$?&VKI=aHv{{X3a$Hu*M4=#ZAIMX91 z{{Ri+jzoDe*p_lK4h3u)Mz^heIJHYl-hzksRDG_xDlsZA)Y=fTuh(aZR}$;pMRN?2 zTWEILZMrFTxz+B%1X0OmRL%e#oRNyup}yM)5-+FiAhe~yXcR#wd>lvoD1l-!(N#I;tf#1c6@aPBB3g(?l@z0|!^+_x=D=0dT zQzyguy9p3Lhqs#L@Pi-nvmd2fOZzf(uTspneQPEE0AFgG3JZ;Db}1M>k;H@qAF2A* zyVjixW$>R)llM1Y-}H^D^Btb4bq(ddf3mAM@j&_qtg_pQY%ahR?6@2EpIcZ>>^V{9Ni- zX7D2#UtL7bF*Fg}M91}66n*nd^{-#+x=ox&dOW0)k2HQP1xK5MoQ5CybK1UsP{^f9 zk?z89t7i=B-xe9m_-e!t55-Tz$6?0+asL3Vb4#FOnL?8^nI2%-=iA=5zlfbhb*Kgx zDweJW9tA=PIrSs)dt}!*o=v3faJlSH73pG4Mw#bjlD!Muw3EoifdC%Ix#!lCmCArJ zap_FnT$rNp(Xy?Uz~d~$i~(I|6JU24o40>jZUPkyWpR#sa(?s(2)A*S zz~i{>Q+tJLfK+*XhjUgmh{Mks&Zh^%1Y{}a9<&*zY19_y+b6L0p~$3>k6zR!46GH2 z@cl^Ul|;P?Ok`H`4WQ7BW?2G7=PgsZo5Um34hCuM9ukB$%*fni`cTa)Wv+{(Z@5&E z$!_JBlUog(N|y|wM6H~z(UAWDl_;*3mca%aa`}rOJ5M|VQL?;PhFOXeoE&yFlWm;&m<*lG)%Nzo%NdEw_RMs|0d8urXmIUyb*9QZDNd14# zr*%6)rQ5P?@zn0%e=ZMyPhQmtTP$yvxB4a5P9RZp``J{;*kuv6JB*)R+z(?~cAJ@Q z-2nv1!)a0V?fx}!(|V`R*vkuivmJ;8l12u=KBu;OW~=OVrH1O^zFnnI{xV342LprO zoPDU(5(diXL9fjv!ZpNuyfDSF$7Sc!irj8?t&+r&8I|zGu#E5du!u zC2&1gCcUE1Z)Y7NqR9nOAx*4x{msc*#AhSk-MlT6SCj zM|>VDdDhlOm`EG~2H!6O>G!RE?!-!2oaB9ZC;ck%bjnU9sS_5Jfyfxm729%!=Bo8pVq7GE+mgC ztU|T}0B|_%ny_|ePZM(-a(VQnlA3*tT>QPSf))s39F5?XargHXS11XYK3#A}4bDFF z^scw}2#R2fBX8l@KQegy>bCv_f>u>BD&w;s^s3Pf7C*h3{dU|*6lkfi1Hix^L6h6o zgQx2&r>m@X_H%fg6LD@F{zbce#ZpA#GI$*N;;i1z*q}DoMo%a&9m0r@{XTCsL1?ji zqZ4kCSrG2xBn{;r`83v_sL8owB$p&@Z(59zNd$36G;v209l^|M#EMOf$pt2U$dWXzC zv1p){>D45P1Q|&DZ^I|{S2*+>WD3&cn)Y>VHs&epH7gfpW|l@mzyLNhMBrl`*Xl4u z4#TVCI!Lv;ia8>VAd2f4Mr8z@k}%}=;rSzvYBr73)_OLraIc`QW&=WGkKlMnbtO>%(K|TQB=a^4-fF z^`*1vv9ydD-X=}$k0~gu6$bA6M^SzW0H-}t~%P%@I{MTH=ZN0U2U!rz~G(ax0tCM zWVaP(($QLKx^9~#)}^I~VCQVI+#|%G<7=o~F!oS$$n9L#4PwsX?A+XVmlyI4;3@fa zR?ny)miOR}^xWYixbbcl8J*(F#}n;U=Yk1tcRTIkgeKzN#S5Fp@^}eG$phi}* z5v{Y5aD1WI@=pMjYz$;JGg=kSq0;&kjQVz)s@S}aFb(_%Yxn;ElyT^GkD#u5qkL29 z9-4-IKTOp17_|8Zp3WiZwj^v*Hzm<0VBd+?dM7M(Q>$exWZ0bZZ ztj>uiV0Niw$?xuY<2Ca>m#*sZT8nKyEge=6SLbE1w~^WV7{$bKu_OD$b6rEAej{p{ zW}YR8=}fK!cN%r1f-}MVV`v#X_u9vSp4H2e*GEh#=Pmtr_Au&_*kew!7ZL~m0E*Fq%6p7s{QGy}meq8-ty1JiW-cxzW+bd?;fVAb za!(zybM!UV`aAY{>dj9@mF}&hwsl?c+1>e4VUFdOEZF|Z!u!=_B(%a#!t`=UYp0@0 z=`4p~k<;*J9Dsd$_Z)reu-ULnBzA1G4C5V#YNqM$gBp&Vrd!E%e3v&2;pd5?aO&T1 z2;#Rl_+<4Bk-WP<*6!R|OV77_#SJY!ZH-TCukHdSrBY z+_jm<@lz=2$MX;LqG-n0%WR)8VZGdbUg;5C8=M%Nb|Cw3N1?6nQ0Z+-H?^Kw);nf9 z<%AQp6moxF)R&VvFN4uVh7o&5KELR@dlMv)T&(f9AyvrSr#yY>3A%O&i?c}4p}|v= z)pqeIaR@)B=OH1sUywWgvO-J-h|Ct8JH*Zs1sf$pvHjlIH0rS zR$aX)M9>jqLt|27nj0DbY-$W~Qf|~XGytS-XbxzP>q2^11m=M!YBSREA4&i+Qu3Y+ z9z#n?*chM}l9rc%X!06fW5J*ol9iW^)We+6T!1Q$KvC!qMsv<+E_kZQ0GF|)Bs9#Y zidIhb2mqFqo3W-`($iHzn2j?TrD7Qr)O=KdggK_;G@Nw86UVPLEDfVk)Y8$@3s8W< z8ZtI%WnytmL^E1NP#l2MXlYm$nv7Mz%tY;`kmiq<8|4L z3}m<3tNvaO{n3B%vw!NsDB_9*CN&8A1%3|FHugToi{jir5!?6mb^^L%B`w9-&eM-* zXmN^{ttM^nq~OgQL;gj{{Y<+fr<4iTVMFs5ypLsbNW)lUq>JA zwzoHDwquVA`kV@C*G#sx$AxbipX~+b^{0_=J5UDJk|6U&51p@$|$C(v&%{{WO#HnY|p zMSY~*UFdp1(yY=lQ&86tmE}0Xi0y_SlYl%zo%oFQq!1 zyWrfU1q%SgN;k+`J4+pdB=bO4#qk$Q_>FKbR>xB6tvV+UXQ1g8hG-b6%FB5(5f1Le zL|+NU;9KzSiFffAU;hAuXQFhSyAB@qN7Sxy>E(~Zj34@#R_Pju(IY{%5Cv5ofE|FN zAm)J4Pr`eRYySZ5KCS=~^dO z+UhEj71FG3?cdBUyj0vUF~JN@JMe!>=${sC{zG4qSzIJ%J`54A@7LS<*D&jidg}W~ zQ1QVMzDt#b18x5Ros{G#9*2SLn)%F;`LuloIVX)DK72^HOUGEGxIjmjhA+1qascH0 ztC;T%<;Y?i$~NSh-g>LSj(dl93Rs^86dq(xChp>>&vS@ zoNtv!@Ru8WPzy2j&N!;wKUn@54RGD%d-LA63%5(nhBhV1>_#XFq_n{ARa2h%HDEt7{KqD zt>i4jCnWGW^`?c#hjin1(%*;<(357Aa~DtFf&9}L73(%*|^*> z?Nw^aav^bW(N04Sf0!OAS+10VGI9FQZf+V$36PTfccstLt3!})7!$#!BbR}ZkUc8D z80=_?F5dm>IF?MXNt!2TfsA0}V|O?sx7M3&g(H$UjCdb_Hh(ed2lS*OaIU1b-hC*y zmouir;*%Qkjs2guwNSc|a~0L4?WA_iaOF;1i6JDTXO71Yk?sXi1d~sgEUL|KCz}I| zue#=%wz##sx&HurPYZHw4+z_a#|%ef^`qTbM;wu}TnS^4f5bt`pJVU$1m>WJvDdkZ zduZ;hd@(9TiB>#<*yDKzu_M2xdG+DjOY2DFwR@{{zmYjbnVW3z{uYPmK;#k1pUWEc zyhhi>ag{1g*2e_qoxih+50f%X%WZVZ>d3h$joFtZXR|3Ba)f%+qRK;>owcpi(3ee; zTnE^O?qJO(1~_a5IO8}zqtmsXg!*46SF^qJ?SMcdBi5@WeHTpK zEOkw8<4=Q6(k$kdK&%?HV)5>#;RkYTU(rK+z4o4!c zwRMrTS!TDfk_hiu;}Ur8p}h>N#HfGg8-_8K^yJERLvKaho~V{mOKk)a*h;6vds!lB zS_b~k$K{tf!Q_nbire}};dY_amoX*P^i~6Y=4)l~3C|yd3HIIfL3bh%ptmL~! zWE>{tBYJyczM9f{GHYu%;Ao>!xGJvPg%#sw@;H7crH>yj&vcJW>7NcXRQU;#E2&if z03H}oS-(Z~_RcGN>78m5YNG0OEZmO~vPrF8Zkif!A>{-Cg&y@;Do+dU@d6?YDv!N* zbMrEC+20=;4mxLB0@}_3L?FigVow=ATIQbrd2Uyy|<~7fZP%F~Q@-;cwf+IZnmBH&;fw)E`iuHcbx9~dsd0nJy3Kh(pXuf@jRt5_(!Sx`&Rg}OA}2P&L49} zT)m3NSh;&SZB;FzQX+i$i6HJH+tU@m`hrfM>5FYiVuIRK)8kk~b7U4f@aNxlNI4bJ zw7#r?5nUZ*iV&vYmLJ0mxexUNkIueM{i8a<9Y+4@eJfU)-Xz3RJSZ8v22eh%#FO+Q zy|n)FIN=A4(PORky|kD1mNv}Ibw20*ArFiLfyZIT-mmnnUJX{&C5LgICB(p#Z{5+xn008n7dSbqo_&Er@c1e;r*5WbeV~jA*A79G3)bjXp z<&B+FhA$!&G(Ke@kjhnuq2i)kFwJjs=mHr|=s zV?1pK+P&W&>Z>J3BBppb{+{8yZqYzmhRu(?F+Hrd!Bjd>KprFuB=OeAMLs1HV%^rjIrUP%6w%0zvtK?IARdS)p8oYdlk=e08wRcN~e zqD%qBGa3_KiV{z5DcEhv9jf+$7~`hmkKcn*ir*c@IFRGGs<{Ah!;^tqE|b#NSBM=6 zj1iNOjE`J%Q?ZW6rhZ!IQY$U&%f1U-a5%^=Pi*^A-w}Q~>0KJ{Zu0==D2bIunCH09 zryic;HRE|cYmdVpXRYISoW5BL`Lt_2HFGkVBnJQ!$u*>PWboNY%_^!z*zV3NSznHN zRPjL`oqD>J+o3HgKGaX`Hhbfo4_eeN`pWyO^;L?(B%P4$E)}rN^yjdzp3N-TKPG$l zp^hCRxqLVl+O*#(I;7UqM}s0inm+h%$l|u!?}iu9BbzxA@zCU8?cTpk`j67M9+lM> zJvXV^E};Y2-9sX}Bu~Vt>$g1T+#juX4w3lFYlxIg7`pEnz|S}w`V9V=!St@r6E<8? zt2q3u5X$W%QPy1Psf(asJ|@gC$2rITm1)xUOLKN)lY~Ux0UwJ#ziL0lj;hl2OCOzs z!|`M&#T##xk9?{0K3}m1-nEW}>I=JzYoT{0%eZ5}VBV#3f&F>=R~MftXOfh?9bOa` zC~-$#w~E5v-F|9N#xcvdAJf{aG~GQl-}l62TUl9*PT(gaw;rCqY*!QbiSfHj(YkhI z)HOeov}uVg9GRAIfCC(O;ZkLbc;M%p z4*2a_Hox%$P0~7GY;PPjs&2r`gd2QA{o>djb68uJssjag#u6TD(6!s%g4-9z}C`FdeqWM$!4XQ~v--qF*So*R<=V;u%rw*}WBs z44M0tEB7Pbo6v2(a`RBsPa;R~BL4vG$C3S8ajScsJ{!wr&M2bRLYfhgsHb+rzuPnTW8{o7!hyUP1o=%1<};ijHVj znae+DjVDaf>{C;}z80)sY*2>F&fWH+4qKl<57xd?>Wy0SRO$KFXZ_P;SZ#qL{ye9! z-1y_a;RoyVuetRfg!Z~VpQ&oPo1UoZS>Y=tsi-RCDH!nKjBy%gvM+uzNw18)C)~b| zCB~O`97}wpvRg+G%w>lGus9%gUrg7Lnmi|@dcHg58f^LJT!tG-wj-7~#vAc>#!uAu z`w?2cqMIe;j(w|O)zy<#o;FYdwj1e;XZo7g@7zet!;#omOg$LnqPA!7+atSnp`?X< zYZnp*BzGc*9Ow~)gS*hzLPsF%(}s)@fJg(1kBPTw2XcG+A8JR$&uW`o9|T9~yLDMB z1A_igPSz`&@+dPSNpJ`m6d2#VGLr;7`_-5%ZRIF-v2O^GxG#FJn)W#Dj4A}m8*;ed z_Wf$27aN0jUc`HfCVWK!TN&@oWl8MOJqDgwWK=P$ZX?WFwtke{DzQMN(Xdx5jsZVf zuaZ9kATjv3%|aHL)s$qXf3+wkhc^K-hdIx;dS*zNEJqTAR1T$oG4-mb z?cA@Bf(Ybht5!%gnd8n66_Yar8SU>wWGnBE+|fg*0Z8VihyyI_F@iv-$#~8eocmEH ziCusk8Wc8U%oK7>D5cS1rOeMVuH(t=n%%A7eJmu*=+Ux<9OZ}HS3ZVgZ6IOJ4Oz#m z$#i3doQ2ODkF8P0&Eqk>cp@>xo-A%NfPHIExP&S7t%p-*GO%&6*UDC^XHa|9)+f51J1U?hBq-K7gKn{r&2$?&%%`J|5XYJER2pzJwoNuk@;Bi94iQcW{m1>6fb7|0#PS)#Vf?JKD6w3}(I;${f$va2JNWRZUhkEr+H z`eui1EzR7fHQOxG40)Au-oB&1^`L1wEykd609SF(hFEStVOvI*ph*Rvjdf^cg`@~0 zAcC#zeMjVYtlO^5;G0I%@Yd_)Y~*X*J?|_oM}amru}`v)Mn*}&!S8}Ot;0>}X(!a+ zI(5Ch;ed$(!bQTAUqYuZ8{@j?9N_vIy7)^Q>26ZzP-V8ew%K-(?Zwn;MlcS4iy7n+ z0l@VH*7J9Bp=$!l?@5U?uBd>kqeMJ7@7MWcMqGGuelAx4^N_rXN{sv=^m06mn-Cr?woIt$ovcF2eoX|+sGtnuy+H4$Lm+kEslq0G|W|2 zAUsJi$v%Vs08gz3JILmY%FE$MSIwT_o^kiDFE%7^42|s6^~~E^$(0SqZ(7jx%jscI zgrYiNxjYQ}3f{Vo`{XUBxpm&(h9vcKTnnl%?{!^b8+(#tMqu(s0M60sbMN2nUOe#J zk53%*SM>*6SV5>N$*CU~Bp(kM81%*ut#!|a_Or>QM=Hu9idHU`$rxN``IwIFUp3w9 znogtDcG^ahr$K39CCbLP3cICYo=GRv_x}K~Us81QE|;fSgajTCsU3!R{{SlMlsF|6 zQOlOi>r2w(vyMp}ce7yR(>4fq$((K=wtQUnr^A|7FW~|(KxP0D{cEXwLb+MA$?c&b zQB~qpfNaM-tk!a*Mx~P<9tw5>NMt;Cl>J&W+WdDStK8(Uz63 z#278)JZ%~F=YvG}pP@%ClYe<-YdT-s;pC0H#4ZTmf!qx6YrhsXh3w9A=CpH(7XGcHT?8K=1(0z#XyVXOrj;)}s=&va;9WYg(^O zT^S$e!rexpb2M{f1pfdo)7bra&1rY`2FmhVbCOi>ap_%(-&)hP_3-7A-dR}z+DGGF z;QdE6q;;-{`egqAFfHVoIf-#PWQ6DNW7EIdy{{L@cx6wc%<^!otNApxgV50008iGf z#i(`4ki+RilGg(wzMe8tNu%dnTwOAq;Gn!DRl8f9_Mo++=e20v5LI|2R2n>0n?gr% zK$0J&WZke=v~(7a2A_DFnj2fW8479vTO@hrmvlX<$?e=9!$5myK9o%`R=jkikaB9} ztmHkGmy*@jy3>;M3P~WPnggEIc=l}MfGK$_Tpr(QaweHd@qH;-Vk0$oHG={8O(!Lk zjgheal}<=gDCCbP6s)lQD*5!ubNCGbwDujrry>d$gFF#SJ-0jx*k4aMFtix zhyug0r^ z!L)?^L8h(#Hr%QI0NkQs^vJ2==pAk=KYDfx5C%xDPjB(f#y4r{`88O<@!~ZjcANryZj4flWnik5B1b${&wbryu(@fd2J3pB^kR&Ydsh)TR!S5n?DCjpu6TjGt=P z`irY=G)s?`eeXG3szfu4=Oq1rC-SXga`RB?E1Qdn zZ-U7pG)klSV>}Q~^AbC;?_9Z~DJn+zoMS7Ix@%oSsM?{uzj-1y;zrzj_?N?M{{VZF z{VV6*po>!Yg>`v@-;|SA|2n7BZ$8Ow{T|206FLk)&jU`B9V1Y+ITp(Qi_$TwN zY7UCHAOmw50D3KOL&ua|wX@cKMmZ#`)4EHipwcxc?xVPA=7{blETs;!D3E&Mulif1EOc0vt~Xsa<0qZ+K_@KP zJeEDP=qo&)H0JH2Tz*CjiXy!>~r)y*2mO1*V&|L3{puKg(QQy8#ex( zjy>xPz7*uuEgHkk#+KQ|wK(m%>hP=jm-Pr|qq8 zEg<<^_04WwT71CY^y5VO+ToLGIoG{26Agq>m6g|warRsw#fHUczc-V!jC=v zjKl-^R{os>WdLF?CG!9aW6*kisTuwmX*07Q8e&-PVq5Vab5@6qD7!iGc`%ZsYrhNn zzfse5nBLamBl&WSmYaVn~+Dai@~3~|^G zuodv~d^pnWBw1p4!zvW~To8NLL+L)fpQ3(K|f}$^9}dyC)Cpx8!f_+0rKRZxjpKPe$m})YZ5_gdmIp;z(F5xdh3Tb`6HT2 zEAeO9qR%{kl_#sDmU?v0AN*oAMm^l{ztq>q6MoWNc-^&4zuk;gE&l-R2iMn(f%S$S z8;4z}1_K%L4TD-7Hf34rnU~w#4<|HGI%m-jcy+J4FH_PjP6ET;&=}wbz zo=JmK)EDd7t)oAbgdgPf-?SYUNa>v_-%Zltw9~9D*JM`K;ZYEM0RI3U=ef;(UB6y+ zUCSKn`izA4&BJH>X*TMfpmshcuc#CCxPSGaKUDQ-?QY=0-N77-pEF7ZKRpY1j6%hAbmv^a8s3=?yv^z$qQ5YoP2M zvr!ywD4EZfqlQama0w0dPw7vS2ND`R`R#2cf4fG9N4;VT z!0Y!F*W$C@agXU)YXxt_p#D^L4py3V%Ax)feK@09XxDQ%Q5jL{#BeL3K9~y}E)UwI zG~fmY>rp`GlWA8fa8$4v?NCiN>o{1S-nuerJm7_4p43TnatJLKhQ9Rx=O<0O-VcJ= zJm3mq8%vasG?6j%5?Mxz z2LSt0j}dtZDnSR2t#lJ@7&zEJN`qKN&7FhawJDP&?9oJ9yx75RaqV7^cu51PNW!xZsi8~JD4-Dd`sSkl z0ASPdYsJlNjnQXsk+&bc0s1av&R?3INE-AWX2n?Hql^$v7-Iw5+MJK2ZWQHt`_YDh zc9Cr;4iCLifk=TwBytDh9^##r@8@@J{{WAiz6Jp&`G?$h`ijJQP27AKi3DP$6@|<$ zhr#n>Yh-s6EUk7_-GZ{Ro-xN1y9|$itzBPf*H;hXYTG#4AEvq_^W z%d%2HIUT_ri1a)I=Eg}C3CwJL52k(nkFT|KSZ21gv{Vs7(Kg~O=0M&F?ST9->@dNy zG1%5Jf1`Lsdp0@xA);|5_NAuJ6P1x}5wQ)x3zE)9;d~5aFeLXFsJ4}70!aQmaahWu z!-h#W#N-}VcsS2~FbMk^rcG37@yz;Ly|J5i`*~qTkCBWKj1l(-(}RvHt&uneogOw}6X06$+ErheGeS|TpxPdwEK~! zO(y9)Sj%owtQ#EU5%=}?u31Id(4?8P+UhdNcMKNou_TZtGtND8k8@vnbR)yldTeP7 zxiG}G%EN^m0@xm$;8()m4lXp^L#o7@tjlX{Yafh+oNjP;cJ;}{b)L2O=-Nh&Wn-ht zCS$ruBtw(I?f#X^k~Z{?oOLf}(VF*OTlf*$$$cD=#W9vgW039+Pv~$5+PAKh*S>$J z8tIrxEzFGIFFRK_1Xt$0$5m^ZxUK_uiercn249&3#z(oTfpL3e~Lrc6GpH8_(#HCwmgV~00UmScq z(%(+%iFI<|!)*>>iM+)S@DI?pEnNfR2ghjs80Znbp6x>R=1<|;j?7ygziia`82RN% z6OGhcXLhvsk#D8gTD9%U8I~g)yPE(%;m&=@t=~u1V!LQ8t>jVRMzXRg=N{+tugcc= zztkO3)nemQki}&80(Q4v*hk!|Kf=D)_&e5jJzdnguCHkC9i7Fjb`ZzAJ4VvX+4tua z(+nSPvmBCAR*#@M+ToL`HSLY2S(^!QJ&~s%|58D5pTz^ zU=n{Ye_Hx`sEwwH(NOKgmrrbw$GH9?ybS#_&3t3=(^85)p?hYXQq>?q6l=-IZOq^G z6J628$7i0hGm~pN-Lzb)0(yn2*tY_7%!y&NhA zVlv-M@P3B5Pf}aG-&bmPFw7)f{vlzUo->N9v%b5v(e5B*F78Pg&PIK|Fe_Jq*9S7Y>|V(KTd1l`MS?h z)bxuTJ)DcGnR}J0rUyZi+v@Bz}Nr?_8eyQ-Ub^i&F$fBvz5oNtoRLZOR9>(7F1PTC9H; ze|*PyvM_=Ng<*>a8%aKI{{WX2%yqt`l4qV++^M-OZjlBdQXq0k9_G4m`14((o_V0U zX6K=NT^$puy`{>$x<(bFnkEiaSY$H&21Z49TMoOo)p|l}$fS<))_BoayM6%Q%s*b< z)$rF+$9M9}1&RiZwu6Sjj~<}b+3?Ha7Om0t*EV-bFKlHSd^c=u9^?Hhu3jv=iaCGD zgLK*Tj260ex*?6(Nl}=;kaOG`v2LGWr^*l8n)%mG_@xJh95*)-;YkT80F_+!`qxCz zKWGl0(Ci|CvnwV*au{*-_pZMhS#kU%v~l?n#Wl(=XF`)fW7$8Y3Hp7{bU6D}44<^$ zNGwW7oW40wwOc>!SJF|b2?fKqpxa)pGt1Gn9-Kad2iMC&ZkZ=12l86c?mue%2p{2T z(;R!8RXykJYi6eqXvM$0^$Ig@LrCd|rZP#|e@cX%FwQs_9@WRMe``yo2f)$ckLt~u zq`v*Db?H}ZR+h1o>$a-lE7^4f+B&Dc`2PUa z)2`F0l=`B9Qn9!^Bu-#hg?QIGzhs!P9!eJ^eZZB_=y_Iaa88ZZ^;l zi8;*~XZEVqpCc4SmmahQs@$;0`O(toU}u9=!sF8>h_ht!XbZ@vE66-kNqK?K6|bzN zdodJrxhEN*FQdHNcx)PBZr?q{QAKdL#W`y{aj`B74`65ug}p2QClp1vE7_0hRQ~`n zrH%nn>6#&!pIQRtH5pvvaTue;cJep@^8VFB71q;(+uoaseTQ+qflu8$>yeJV!w!=&Z;AnhR=E{PRG|21cO7h=0$iN``&<%LJ{wfW? zS3)z*0qy|Dy#&f1oX`_)2Hx~P0w>9D@vJ0-=OZ zQb_|80z%k#;MN|>HwP#W+L$~Fz$ZUS0TW=k7|5YDf~e#M6#FnO#Y#JQXKY0qNPU!M zp|raTeH6mM41lH9Ciuc=6)-%gpy_1lWQkKpfA++%HtL+s6p=a-X; zE}WPBn@0v&bry4TXaU1<_NcX&sQ$In?!F22MWwq-Et>86HfvgLjMf#(wTy#l#YB9Z zGwMtJ$(9VU?kv?lMOy;{CcHGzoMRQT+;o{QVHMOP#O-%0~nHhCB;iiDbWRA6vDt7$f>RE9zM ziV(F_QUcH#>S^fQD&ecsTS^AkUBa)8s3$nAHMV_u`&LxYHPLFP>GrIhIYMM^VtuP( zlE8R8uS*ZWIKW5-la8=X5ip`YU*1Z>{~d`wGC{Sej6lI&^7Os z0FXsttQFXnVm`vQC(}?5Am{5@OCr6%%~%@ZSo!doZ>g!ioCqO+&%JCW&O)T350F!3#v*X#G8^OJ*G8j1Ow9gIZlN6^RHlUY^*f2X4_@9QH&IV^NP&n#6W%-c|&A zW}?BZjXTGPf&kOGfmQ;Nuw zw;cT{_*xCP0A`}FM-ES|2z``orIwRBe_AW)J{kfX6Y4QnAyA$HB-CEn0M9x4)Tc$0 zNgC|>c~%4vR)W@BvBq=nTOqene-9P!mQ&nOyC`d`WI(_mA8McO!bV68-+I3`v5w1= zMT*^+{v3U30js`~FZMsJMRd2_%Hp>NwZQM1h_qJVf=yj?StnMFRdl9ffG`aiG+sR8 z?Zs^3z$K4punNYZU{URRitOgqI+Qx;Gd;VGE`&Qu1$zk zCusYcQQ$KiDuMMB;)fw|AzWb7G2gl1v8)?yoOi4Xd^SL;xS~{;(%woK9}$P=OfIu> zhaz_UYPF*yIPF4U-)gS|U%1P-dam{d1#TFT>P>F_HTy$#cT3xwYh6-Bw>ZpfatixCx-ga^$(0EsFF0fv{-gW;pGpZJpTZv z+PGkrL_ShQ8VjjGIbv%+@u7D`*ONcl-gJ+ORvKjY(@hTZJE2I_@DvOYT9;Gn7W&q$ zJ@9f7n{-jLXTtzsV!7weU<{L*jF-;i8;(BplkpM8zP+C1ugiPj>qX-K-@lfVg(Lq4!1df6w zRz*eZ0T?6x{p*-({{RtW>Rm;zZr?MI6dYgxJu8|Cq#WdmlE_{A^I9w@=6R&exiy%Q zR$xKh$>x|E%x91k{p(Ls;epO6xXEBK-nLkhGv_94E3Tcx$`Dg$UzFpJSLs@!O}GU= zikpBi_2RTpiro}#vuFMFG63Eo$KIwjNt|T4Z2Q)_@;1N)Hw>-ejQ6WjAh%6ISkDfA z)iu47k2O^W<0hvJ$2`=~ETp++Jd>J(+&l2jNcx&toSsEM+-_jB118~sz+;MtW8BbL zz`*sa8Nd_)o>DlWw3HT?amlEeLG3^lrarVLkdwtkxHaOtK{=oz(zm@pv2ID`qVq3d z%>n1d09SG-B<6%>&w7NDg%klKd{9}a5>8KQ3(iLr0h(M=@;N-v%()%tQe0wyleRr* zCG$gR_fX8hde9SjG0g>>oYWdj`q1QjiUZ7e&S)<6hkvaFnt;N~kF^v|DK&bUoSFgE zl$s!H(x}*?tI!6{Dr)A24P;OZBh>eCkwA=}dM@!mH4LEed(pRnNDSvAieV+n;k(cc zq?x8-pBx|7tD|BM(wGviFlY-EVBCr#$&TidjzNwND4sHp<`e_U!G>yLLOpn-(q1s8 zh^-+!;}ipFHh9l8L+2RX#Ufk--dpOX_`3cCc8)&y)DDaam-P zV!27^v|{AyC>XynTKGXh8O~@iytgt1jy7}X6cb(Vp}%EaOH^p&)FmN!C2@>bZ_$5e z9Sf(TjI>Jf0h8;KUq9sibH%~3Q=0T0T6}J|Ankyo#|{$N}dpF~x5B5A5}< zYEA{lVHnPFoc>kzR)NrV77%6|WI^_)F0_bkLrb+*zFWxu08#l9{{Y1rN56;Z9y~wt zJ7>yvf3p;eaEU8!2f41f(BHF;nWpVBM65@vb6pvFhfT6w&g#s*^qBa!r&`EkMpAR$ zyiC7W@-nK)lm3rOAAtyO90zpXg46U#fLmC&{{WYPPhV-#M1j&cln+3lSHIe;YwJ%C zI1b$LTCTUR+yy=xWeN1JCVBaP%Ff7QmF*T>cCXS+NTuzY3Di2U!3Mr^_|foY-ulvQQ%*o8P+Q15 z{SAG$(?8TJEdI}t{{Uyrd7nt*AGwGAkD5agDl7Q2nj0k1Na-Lb>Tp!il`=D5cNUt_ z^6gfcGRj2=GcRsyBx0lo=T{){9w#k9WB}31)C%)JLu!$Vi$fo`KYE!$4;Y?~bVRFcW0J5(CD7P-b~$r8jcDNGfx>8TjQASdfnI(LbHfGGa}3bGLy zeuL{qNPrkMEoc>MrNpPtmZ^(njl75{*#;XJqDJT!k%LmY0)jCj;gnDs%|WKzcfrL7 zeQKoL05OCZ0-++Tia{LvifqTYa4Ba}jmX7800blsf29>>ju`OR`%y4T@HwT{*D^D| zZ)%jb0GDg2I0P@XOT!2-Ii^!AcB1lqs&8yyOt<1RYjg_u;xax0@t#Sh9uNiqrfs5Q z-+}e16dTVusd4=Q0cDYpwFj^y@G(}Qc&`Y_G~&TV_C#kRB9wbyrEIcW{&XX4$19Oj z#rp=g^37h@SJ2h-*^K@XLN+<zkoov~TafPEtYM_IT)oRxJ1_w3ew$G;(U{?!#w`yuQLXOnm z%j43Bh(9_37%jhxxbsh{;BiwBlTs=`=791hK_qiojj(w*6@?%Q6pp#TGy`oecv{Rd zfC%-VM%Y^MlLt7dOrt|jHo>2I93-s?0Bh1KGBMTQlY7508|!T)U1-Qk-#)2kPv7IJfLWO4yw0Q6pk?EV z1I%b_YsLM|L8KUe-XiaK#h14T(F29*0$ z!)_F?dLp=>8bgm-DCgFb%}pf50N!e9+;c!l0}(`^8I1AGNVh*q1a0X=$@iccS$w?I z!{?k*@js?0!GNF`j!;fJQFt`6xb&hdn4k=$knnk?qM4X-X)z|@Swpq5C zRKDKSXxUvk1R4ViH#o->c@Q$L(n&s)xMf$xZoLiC&8*!d&Zodrn&$E|$H#-5`bOAr z%RJN}?ASgPd?K5x?xT|BSzyRNO8T>=ejaIk9cYVZ68MR^Ft+eyx;K!P( z)vn8@hY`zqk?BcTIJZ`5AA#y?z;oXdy+L+vS6xSMrdbxqI2F(J-mJRSgQN$KaaBE4 z)QKT6BOGUw^r+~{t!KRmB%Td(Pb+SYyjXayOhIkrVBRYe_c^NTy&e{A`||nstBb2! zw^uQ#EPE5{TJq=`tdp4~+#?4hXBEvAyFCQ|0FviQlHp`Xqmh6o7|#{HX}V&vug!VI za{E13%ezjRPn63+3jIXe*V_LYl zGVBlAxT_AE)_e#IQnp*ttmPJ{!-GoES*}}LSckZUSTgcfsdP@czQ5E(ywNmdd-krG zZ#A_0gZ>lfMQVE2?)xj!@d3CV_~xvnlA}np^wEEC#%l})BP*UNn%c%I$R)O%?ncP( zgG#ql+pW2bzc}EY^{HxIRQ6X}l#(;vqA%sk*+&M|9Ao2`!teHW`LAHWjgC$@;<)fT z0k5GxE%hPSG@~1m#~C^Pwe!;`jo1;`;=g4250{h3@t53tN9Uhd@#BLv#db0ANFdis=j0T}b%qk&M2?(b9DJ%vaz04B2MVB(|OkSZ=%<062VbBcq$ zYanH4Og5bQP#NBFijMLB0HsM>H>Fq`xK~avn6l(!x39ecSUpIra!-0Hrbo44pkG=7 z4IWoL%@1Kb3L!nQQ$e(IKmnr1dWb=eMHQPZ^sJqraX7j8FhE@9$Ia!0l7TPB`;G4PXT1)bIxYVvIlnh7^-P zc?Z2q0MynpN7As-?G;w&8Bw%xQeQ1adsLNs5^8|SZ2C|flj({(xQt?e#JTN2VMrbM ztTKXf1rSG3nm~gXpdpY4)|Qo8Mxv{Z^WeFe(kcbXXt$8EkH6x#XD&WS_PXdb??fcYLOfE6a zWfYT|i$L=Xn!#Vvro#rJP6bE?W6z~|q5yZSm-ypsxfTwK4?_i9M(c?hD2$ zDmNc$@m8#ifj~=v-n>{=7B*wp)QWLHM}OyDJbmf~N}kmvLj8pSF*ySiCXtD$8*;qX z6*nAE9ujdwMe~wq;t-sk^eG$AP!f(@J~5h{YbbB+lW0C3!z0z_a8)EJ*i%#Hx>Kxv824-^3k`kEPVoz5{sX?FKCtkP~d&S(QY z&gSjK4oMX9C_#V}7nCue591DKGE7e%lrttfP>h6{0Nb!Rp*+`D;Z8Uh_w=@G=vFpg9M=ohX{LvAI=Ri0OO-(8%z@kMdZv0RTXEo&+tSJ?iZ)yU0 zRUFn9eDSyX)J6|_CKXqKKu##k(kKHZh#Xhb9}hHJdmfZVbpl3QAM~%9^y`lX>NauZ z1RDA~qcyvG{VFJxhnMSLA^xFrcvNH2^j}6x?l#Tarf%Z0icnYspL)L1`hn-U@Wh^6 z_6D^XbcZK&Ex32Z7f91Ceq;s8{{VKsFq|!)ZyB#=LpL%cysB1JP8a();;A*> zqidVhLg#2cweR+tFo_y4=YlAs&DuH_D5_@frD|cEB#J(SRqgaiEsJAnmLB!Wk5U_E z5(XHjk6*mOPz}T1J*qfLF6cGOrdofoK;8?I;~?jn+wAUO)2!Q^f#2&|b+=JTaR$xZ zyVAOkQpaZlc#;9ZtlPJ;^xm0ecdU&=P<2&1PvJDYw}VfaUCst-zo+#pIfn5%O;Jp2+d~17=soKi_fe>ZD9Ag{;^Vy|2U0B81jvH~!I6bHc z$94rx+=>9=YziE$zp$aJ`_#NaKwv9YB#f^lb6EwCAlI0d??6RHK9%f~#bgjG-n0fb zp7j&OWt``Vjtd;n8Njb%vPLW0@-Qd}5$2*4;-j~m)cq&`n9g{wcxTp)#y+*+Ge84o zc&vCB1m>991w;(c5JmKGQY%rO&Ad$dRRyl z))|@3eJVg^muqBlD04T9=~!0xsTSSXP##T!JPc(^0r*A+v41KC# zkiF=CIN1=A`p^u(h50 zyNUw~m*W*Si)S=8lf(Qqly-5+pe4VIR5p^;mRwYKpGpF52G6YyId+cKl~zB9@j{b6 zlmyP{y>Kc?AB1vh5wTkIjLpRXiKkXL6%e`Ru&{09;;>)ypeE#P#Q-o`%Q-xHRB*nu z228o2mL&06b{>MDJ#)nY={&2N0za71q+QEP5w|#?F*AK=IK@mUW0GqQI5YsAfCht| z)Iy`ygv>oC4>kvWC=VtN6($9{)Czzzo@fXXO+buROJ<&2pm+76dDO^ zngTfYuNTK^@$c=QLGuj`ZvYjE`RQVJX+Ki_oicGL0ikx$@ zoF3EzU8-nzeW_NPH6@%UuQUTL!7Bd%qk7S_YxGBA5S(|dT1$<~-n9Os*B-_~W8jZc zYlF=3*O3PXbolw@rS`D)z;7LA#LHhC1`_R-elDQLl>hhr| zJ${sm-bZeQ%i!Z9K9#y^9Y?22q`pdcn|K5s3wb}CI_O5wj@e!=ifzAi%Szf*xwEIsO?UPkla(6`@NdC@=Wv-@} zySJS7;)~byiFByIKQn$|d9>t=3g?#nAi-5R;$ z9qP*0uWdSVHanLi)Q;86^}mQV`fZ~rbynxgyEVhQ-{QXD=?tZu@eQ4MT+-PZ##sc? zP;}wQ(cmiiNx81}=zN=0Zwn3vnyaX#!D8a>pF|RfePmii#FZmQ3ng+!B{U z(lYa!qb-_KYU?=n^8fkkyr1{H=l!1dJkOuF_g~LP4$cpWSG(H0tuf2j4d;IAyZuaF z4ny~v%|he;wMVo^Ah=%_+PXP2k11h5jwW0w9?f0yLC+wuNu8+g=P%6pEBkvL7JrYb z{K-Bt>L&BzhvVw_b371>cz1B|NwH&5w%*#!@!m)a>fR6$p}y46E3wFpMFDrKa@yq3 z9BcW0GmZy4d85f`{v*|xS)yj~08o>*Wct3+l9O!UGmB`Hv{U$G%SjkJX|>I3P>4n& z9W6E(Hff5oQ*>kvUi$U2u{(8AOH9Mq(qdMBONs#&d;2sfG7l(De;ZBE#0BAngxY?b zRw~_5CuNS$J+n#-+)_D;(gphmpasg**A9xf2RhK>s9oFGKJSP-+ZuJR?q?J&Syx5* zCX*j3ju88C7CTNrnsf){!KV6+UrhrG9y!O@^;(g@dw;+q*g<*mm&^#O(LqyhiN2NG z<>bDta-(T^X6o*1OrmrCvg%0LcyJ)u|J3Fb63C@y(UkgE6C<+#x9J(XTevm7csGY< zY9TVZ#|jM6{@B%5q7R~L7ATf2Z5 zz_WeL4ZxUYYHrJk9+2sYr-qxV1T%LY(Uk$A{Nn5}oz>9y z$e+gDo0;=FI%KQ)3OOD<#v0KNmyrQb)qs*or2J4P zMt5^b_!Xj2sc#t=2S2A!>CaEQV35UZsmJ_ZaQ#4#c7v-tEdHt1*lI|sRITd9u8s3V#P?g`$6uLdh{0#-^>l#M1AQc%szj{IPRZ8}&w|?llbx`Gr z7X7-eNSJOJO9jqThFJlKC(>HWm~~Mz zc&3`-5tlu|c_)vOV@mlR#|_hoZsl(SR>F8Hqk|+w3Evs(7U8B4>pG$cWXj=GUr?Po z?jQ*^LNB(*Bsj=Jb}RiUP{HqA2G8}{6S0@=-UorB!uI)4mpGqq_VfyN29(@BRO|6_jNY^h$;y>qjL~8s`>q> ztIJAs=&h1(hb}B8)Bd^iI`nz)O~dBQvMcM2ZOSOK?YG|B{doDNa{apeDf;@RvheT@#ey{}raawz!J*BMu> zLJ;}EkLZ-3xqW?{jbjp*g`gk2gUxRYmhaSW9L){Z9BoNF7^rox2s)?Rm?U0W+ZcV_ z8oY{(9~#s=XxiblT&~btl&pt!pPk<8Q<$}*-s6j%#koUgYDODQw_S31VH;b!qc!Ws zI->6k-e_Oao2>Fs-0;}7u zezWR~vz>9&sk~?P_co8nsOQO04NZ$4MUs*r<}+cNnNV!Qd~R z^#EJkGr+>8-aoPIRe;e4tr^FMcryi~)PtVrF$)v+1=Gyh+V1UdPyOMAB5<%^sI9b> zx*(z&)0_6TGJJJIH~SZUlMc+!$5*+*>Bs|ac^cnMuB3`<8(5o61?V(LV}+!E^;}6K z5n$UtrLD=LShYZpj=cMeys}%Ol}=6S{2u+r1O!~#r3AwF^a~+=L7WXWnFnGqOB^NJB6NZ1lEyY4bTCKO*JYS@FZP0u^vc2k=$t*2yBV zI%=rXRK2_9s*nB+K2-G~k5sF%C>Gaj&>>);)@rOrYdNVW$JN6n(3o&U!Vp`tE-n_r@<}#XfpE>{b!K zKk=>S-kyiZiR~|eJ6j%CuA{$Z#?NCUE{jjUQ!K&Dlt+%E+3~>hYZam!fA*|NT4cEN zPg6^<&|~^|+rS(-R8I4y_>C7!F6}gYqFvN^>`n18L3cAwZgcW3b!q;eo*hime1x8F z7Eg6(auSby!3=tOSNh-SUxM0NQXcEXNvAK7kBHt%;}Y(h&r+6eHuW2ff19wsfw-W@ zQy!!Wwr#v?clvw@6+S3dON+y{s)hJf7Tvh>=liLyZ-h9i`$PLXJbrI0TjCD#vQmq(hit6s))!OB(@Yf&V<+HOAw~JDm^0l{TauSJ zBuD7dECA5u?U(~6F)<-Yptx*nw-4fGb~{kEx1e-iN;WYb;{OJ;z=$$ywXes;RnB}m=$iPG1a8DNb zte3b*nR&A59=IxNO54n5T31`_@|vrEsB>6A9l;Ww$rwpOzfd1>p30~iL`UG3GkORB&x&N8PIBh=;FrlBa9y!U0^<@1KBw^4k3rye zbHuC|eP1ifTEaw4cS>NmrQSGL0_TUL1_^#>FH;>^cRv@7)^!fQqyG96T^ej6#gH(V^l;unSb$Azk&{Fo}Ig0LH9!I4X8rwUmfTk&9(3d z%+`bCmAB>A%+kI4vR;Ux<(e+K^=^bBls2%ysqn%%(Z*PUfEZ=m1NbTT1nm{qYm_mX zsv;S`Sr(KUeO;l%==d!Jh84u7yhP6YbiL2)*JlcsB=w>2D_pJUMipXL#! zKVn}snDp#=zrx?k6!F;eiBMYG32M>F{$|*EN8vq&<>j0AcGisrpx( z@sEe&DdrLzKuq>M`#2r9J7asHuCD_QWSi{_-uCayc<{3S!JqcYHyL*l(lzbJyUfHP z%2M{O-AYhGslh(|)ae}xc7ziwQw!TCO#+QyTB4i40L@ zUMjGFoK6+jD8Utmx$6O)ZHUD>46-V#dqMUDr1dtiy$HEg+-&Fo^|<%*l=R9ay`m3{ z5Ix#lem_u|I#aad8Z$LD0o_{qd0EvMd{B+)`ymppQnBPJSEyuS}q9 z3~QDBV6wT_AZG(}80)e1(vp9G2 zWqwwQ1Z}5?*bJ(fJ%@R`#iwvPaajq;PoPQh6Rr;T(DK^J9|2ytMuHhzX zCt;#cW~f3#K1Kw)lpqbTVNnfk{Rq^;$UZqta2{7vRTNxwE^DesTr?=DfYGn#AL#8+ zaY=y|D+$cu^biR6GNc$751c`JY;0z8ws6mr=3~SDD@^;c-a^ z#`TB$0Zeyd(QKbgd$OJmC`QO>2Z%dmS{(Y(k(8FhE;UM`*B5=3a+L?*=%0bY@HXJ) z9@9LMxfir<&Az8C^K`h742W&E;%vF3mjO@@#3dR9)w$3$1gfw5n*>Kw+(be{>j`76 zBtG-`&HL>OHQ$n>wu~yilcf-NlBK@VVt>>(Je6(wtIah}hIKMbY$=ZT+nFIf+MgfV7wB@jOl8 z7R5Nkh-3F^f)nUVUi|7^EJmI}zMV(~b$IUr@Yim60s zBSp}#lo^{`pJSY+M4wRzg!}HQDE# e(@Bh>$@f_EXT|k*NaoIb6M1N<)-L{kQ~w8%epme-{AEaF`wpKypRmB;o+T z-))jgBNuOfZ$B4rA3mAek^qQ?o&h;I;EG6+fh1M{GA=uj~Ml`T+!8{QZ3t@7(dU3-8h z{zoFI%UEPz8L7bZSc` zmSbApr)s;zbVXyiE1kDM91J%TuHXL{3(26dP%uz`P`Gd{IlLP2%DGe6u!Y^SZJG^T zO#?M;)PjBaIdLeSw|#OI>0G(pkwX`0+DM{cCEl_3F9Kd z2_jFPyfMp=0x4a%L}yYJ%E^6?&dHyena;(dM%XkHyFgLk8r$86Mx6YRqcW)ieM z=|3!Uv}IMUYPWzEoSWmEaT;cNSYs$=S5LUzr}7%c9BTBQ^g5$_AE7X4KG(yqSKQVp zjJmd6)oKm@fyu7dKMWwVt@|yFJoWrBTCEQNDUEGbo5iP2Da^&h$oPkm5g&5yZraSA z9M_R-8k&mP@2>}*jYZiy;*s*(4d*6_^M0H6;=P86GaPSDZGY(-woW0Vx>oiPe1nKJ z4|63I)rkT=zuur8fU&ecKe&P{g!`gO&m*M)QO0F^v0gaU<#BTP!CXU~3bRZ8yK2Z`AD5xc45F}G2BIHMvH>Z_#CNLhhz$7ziXAQs#(o|Xo!Tac-6th3m z?#2*P6UsE4hlJhbGG-KTh8&kaQ|MFOwvd;qfXsw0ApQbQDs)HORxzO_MKvBIn}{Ol z$(!irS!7mJK&T0S7K5X0xXAjcfa8Ln@z*;k&FuWZH7Z$+Hlali5(9%C{}|E~^Tj)g ze$C3S-9-0-YQ6_IMu=U!J+6A`tJd|^%p!=K;WZ3;I_03WRN=(-+*%sP=nBu245}3R zS*7ywr0k8InYuu<^}T>w>b?m9OvCT1tTNS9vo-&8i3(hz_ktbvF9YuCdO1HbG6t#M z8%+h7Q|7e9B=SGVh1H~_h345Scz8r*sDF6!=~q6%?P<_g#w7&x%(jVdqi^7|wYLQ* zT@LQbgiNe8Eq5>+!L+H{toNQOE1tIQD_9KbJm!{*UgTRXa{QeWRAS~gWJbEMVzj{l zH1_fwCDPw z1!TsP)XQ8K;6`{F8_w1%_zfeZ)R#G15NgPh@g}@tZFtW>PklG38WCf zdcv^}EOr2uEeCoj8v}jHsE2y9L8aI7=dF-IN)y#n;y$FxVB4_p>O_EdJ4D%~l`8BcNZ)d(zc3q; z%$_vxd@Q8`=fh-&6K9-BgUbr03OINY)k5(ky@6OjfXcj^`An`SEeA(mWEtcxq=v*5 zune%raM9ygrACPGJ{NMM`Oro6K0;|J_wh*?GwQO*GUqAv2(O#fwRRD{Y%egOCABX~ zTM$I%jgtUVHIxZ{sIWU6s9>EsntuB&y*qaR`Z(9b0-C&h_U%2TOe&Zb76e7k++4q= z!(E+=&^s(vQa+vgz_N{VqvZr~l(JyY{j}JG)7(6-EE$6o$4n_^Wnqb#7#{FX>WP z`k(S6tx$J(t@0v;em0s8%mbEXn?7b_li{?B{v7xrf)}JC-2X^>P>t)Z7tmrt#)^;ov=q#(aKYze(8Wm>V?nma4bs^S+2rhGEy0FJl zHHdOeHe_Hohd?`ET|`(!cCH-e{L*?7iP=sI_2K)Kw4pXbQ%ZKNLN*Kg7a+a-AbB{DKD@7HwZ%4DUw8boC?v{X#YQPoYnqEYN* zt9^By&higmQgfFYl9Ec!0M}|%s4TkZ;qTcl_E=SI(w*ZwX?*Ae^^aR`XrE<5X4u3= zM@c&CM$y_HBRkhKV$sRt(!ZT(X>Y>(A?N&3hQ_yiG^VmC~xuuI5>hMgu0H@r=)O@{ELH1B6rE zoQ|8;+)_f?FSI#cJKR#%4BZ2bIp|xd|G97#p!YQA1Vx=(D7(*kh*d$LlYzshRRgRV zlDeEqp74HKSL?uSJ*8j9IiJ#-wjgDWEJw@jf}k0;RhiBzy^M6}Zz-FVG zi^@I?ELEf%N7AdUlENj<(?u*Om6ZDk^^;u2 zJRHCzBQ-KilB1aIB`@@*vFwn3;QCE4rE(^5f)*vB{X`0 zF|e>yvdbHql{`u3knCrevDwQP1s${eKo2g2Mg(-v&Cv2OLw9#|jj14Oc-$+J8lce* z#usXw3J4;N4G|rl&o=hp+njGG&&dqAa9zrGf3NKNo?1R8sMk-I?JVzTd)FYnXa+Q$ zu*FsMw!T+bPuOoR2Nv|EOyGDfcU?t%({?*oAv%gGl$~t;S234>52WTRr!imm{_MGN zO{=Ix8PB1$7}Oz2oseky9*RF2HF&q16e2c{6XTRjU*vj>)&ImjOb}}5bMKI+Qu5R7 ztDoltd7!4>JJS0Ze~#olIV%6WmhR>uuYZ@1p>uf-!%CtcRGjx@JF6YS3yQ_*&Bs3K z5^!AhxHKJ_qsq#qA0SY!=9$d+ipbK@TK5YJ9~I6;a+c4AfoIwoaha;aN%tE)ldOBS z#%|YrEFbR<+tx$Ay`RLAEWV5RwQa!NKY{fNb;P!dR&gvEL5}L&t^y#&K)D}EN#nZx zlp9p|XPyd6c>w2JiKnSL9(Wmx`nexpgJo3+4yeI{rTlw{sVaF=2#7SZd?-~})WBGN zS)uevJyoopKj_wTu4T@{y6fyjrC$EUU1%$h}JhN)K(jT`}m65=;!JvoD&32)G1R)0?ppk7RJ-l7CkDKuP-VMsviIP zWi7OPj+$TRj35pBrd9A*pDn#<22J zUX#Ci)8k=*Qi8%3q4S2($+;}vu$(%Fk~rjiMmEyKwnj>fIE;;Bh5I$M~O;h7Dwzq z(GoW^vBH^RLOi#ash*f@XEXqB+L4))$!d_e>KQ5Q$jndPdL|N1HM-+UI6o;3(fCj@ z6vS%?52*)R7;Wm2FN|%sv`(kvp(KaJqm)yZ+49}Pf*eyx!FSUsr5>tLdF&dJ$Ihr) zVWQH9qR7(lnLd#ITBX(bi98amG&h=Ld7TcFLG`-G#BxTfVM9oqK$(;gO;mSm?^W0sCrJ$m!nUr+7C%-2HpX~MmKF^Q^yroQ0ppanQz(ec$%*5u{~Qw%9JJ> z8x%71%9o=V2m20xg^&TmtHx+m;g={;ww{A)o4smUZTt@VNe>b`*CInNq!alEGoyx& z$mGq`Iak}{7o_#fRgNBEQB#Yi9LOo)_Ha%O=9qUz?roy)Qz>i1E%752&|Ei`)i5 z=jioTRt)O*Bzdt7*ywZh%!P<>cDk?YBpd8|5m0GE7N`Cq&r)fNa{Bc z1ICw(qG!*URXf3-ra}}W;Yxs|FT}uXR0-SAHk`NHUH5(h)%i#KOTUU=e|WRKNcSIm zwOO6wbw*9>Cst26zq)w&6Gu+`zY%96duls}g+kl>qHIY;0y=%X3%9L6&5W0vhtsj4 z-8z3{T3SMtEIzoD!V);%8_SJ1Whzzg;mU;nsD(7_Nb3|;UH7PVI+XomDqgvvuFJ^( zX>Fe`Mow5MdBt?P2Dsr7yFuv&`|9*^Iq~Ty?>tX^h@FVlcugvjl{Timb3Qi1QpZWE z{D*YIoidD8j6-wzL7>LlmeJ)jqS`z6%x@?QB$Q!Q+xe z?At@LH$viL+PXX%)ZA4Q(-xh8rCL96xZBRp`mRu=nv8|_(|N{x3{k4ReS{f#&%Wk? zVVKlatmU$pe}n2?Y-$3lFH#rEvLTY*{WdOnc?M_L;vj67oTMTb?QX=w*>${avM}`l z-iCT8?3kOG;DtH>T;;YqJ!)qx z+fFho)axoI<+%-@VJ9eb%#QXxtkN@snz^WGrWawdPKxvV=5td$wXT*B{J?Z}Zcj|= z3Nj$0`6}eaB_mDS-}V}mSnNM?)AR9_`i-P*su(SJ1c!adpGC%n^>$68Ur0(uS~;NB z%wWpeD6MoFvkks5{CK<~-DKs);nKUT*+{H(P045Mul_O#F%Mn^Vczb7N40lYYK)J7 z=IR2=N@pheD63~k|8sM?1die?vg=E2`?;K6Bm-pa zZW1McLykt$pO^g&6A@{^7bT)wIZG;PkTV?aMo*fae4ui;VJX~9_+>!vOm0k4a}x%* zLq>3r#eeS$th&L~H`l&y&I;3T)v5+PbZ%IRv9?q~A>Wihm*Pzva@|V8pY?MJ3me|b zGjY6ji4L8fdu2tr*6vHq+AwqWA(@HI zq2uW6QR~aOnG$Vm0GWc>yj9(U+ovOWs-bZ&2sA!!ny&^bX%+~Ce$fl}96t#TirF2-XvMz(tBQw$8{@6JgB8L* z{0ijEooD6P?W`w-V3x4x3caDkxgtk#c0WVCOd8OwC~NHbxw&|f$hv0I@b{tzARol( zE&}3*{ngG`tF=Inm*Ib*I-H&|K*ftyEFV

J66l>z7T20v&BrMUK^shcTlD$)2!& z5`D6QGPrTQv=zN=2yoX7`b0JZ<7eEw>PtzYQ0@7e|0=C)kk-{g3ZpRi)C^3!!<8%g z!&_L7FU-EiMfl4VX6cSGvAF_JKmka$3~V94p7gGvDe+H2qRvIIESKyy_tU1ewy`ntT`-A zm_UBg*vY8U?MMe_7Uy?s^ax1=NK1B&p>4w-mJ22v;h` zu6QF1;AMlTUPtg=DTq5Q4ilqNvW=7qt*aJlMz>MUgUQ|C^x8!=q3pL5qt;qSa6ad= zO~+w8y<)J~iAeegu7H(*wqRl~a#k}@5H;HBI|}R4%EV;+88jwmJneP3u)fGM!qqW~9qqvtrMC&5c@Vt-e><%2E{hc8W2j9nXpE zSeH}W-kCKTdV}0yR39>}Kj3j%KR=+&h_lT)O_fwROc@d*hRVb_S5>QM+7Fd`d0WkU zYwk~22hfs(={SaTx6~nW4$)&#akaWUu{^j#??-=lh&%1B9Tmi?Z=fS_4Vsrt>)O59 z(LXb<{S*b=v_Zhs##=bP*@r*%*c99$tFP!umy=wW6}C$c1}V&X+N4!1zOXU8d#7Sy zTY_c3cEAD5At+4#?VD99TXwoMbh4~9#SLl7Sa{=TE3&XMF^Dhvyb&h<=CsQ!QKXo7 z)18YnNKVe3!*{@?Mh=0pG? z`$qW`Q?+c*<3ErSX;!QKZb{JbFFErH3D zj84IjTP*^;F8_m-4kk8NGcMkh@6x~Jv1Z5Z5QTXzljP=KpH2 zs+M92lq;2S`>B4&>(!q4eW~aEBGP3_XWh&_3}83oV_ovBHkN(0P8GDi2Ap?$kAmne7&I#_)dNHkkzC{`1%QPk&O1rsd zUbhYT{U*5;6!oAJ38VB$mkvCFoSE$Dd6qQtqX7PHDcW&uOp6fXlPXNO43~`fl_`%8p2gNG+Bi8I#aoU!$;gt*+9#YT8Lt%Mj$frL@qyQWp1$eUI&FU ze1rAjve}byr0*pzW%4Rg-!i*vJm2ifV=Dd0aPS=|A}iy!sbiv(CI2I=Hs=uX-PPz$ zlFdIxZX*5aE14im+3%Ez0I_ zg_#CFg@mxU5*9Wi5TY>-3vrm)vKN*JUX+L;;71CMeL>KL)bn(n~$&c=3jD za3-?Ms&--&a5M0zm(DpMlZ5kVx^`mSa_lGDkSwh-?F@$Abk^E-=GOCA05hX#pa3%p zrC8^9ivn)qmU@PNx)%1^kQ#kxY^DyTVh)(y_=At=1RqduHofQ-E?SZ1P>V4A(w84*aYEy*c3EQAqM&)!X&O#eqbu6~>NBPS*(f#gxmnZonDAlT>-6WKU(Vsw zpi^(1Q-Y}}%XdLX4<=j6FeZ3bGdK_Er}*3P@eI(R$3$O9W8~BtiH-}avO?vrx$W_7 zUkHC|-A`Jvve^`S+&tYujkF4Sy2850t0hR+Sv=)jdh64}?q0c<)G*EY_ePG4zP0Iq z6S8g1Pu?ZGF+Ya)etxd7N-wN`67WW|q(LC++VKoN=9NRkgE#kv8=fdCYjT3c>9Tt& zM@w7ghcd7N1wTn1S$&*166H7Ewi2vq4~S@ld|7??Jm9T-nf&Z?daOPf2GUI^tp@%! zV_#=tFS2<_e3bO?4#$s2_X5z>D|eC$%Z!hhcINds;Q=J*b4MR-2boaks@af1Whr~& zfH#1-vPG*=u9`2oYT^1vdp2t5QrY`QRiV{ok1LK(4AhzovP+e3&Py@gg2}{0*OX?; zpxM^GX)zSf)^`{;@rd23Wy|}+SV&YU_N!3Lj*|i39F6C$nBa zjtTIE#>iX-9NsXR2@P6$ged;3@Rk)mnoyfL>%o+jp6T;_1WRMqrK>;=EZt7}eXz|# z#AjAm*Di{jKz64%R#cmv9p+$quw6B-RPfQ!=O2hZtEm5h7ZE!VYLH1uj=%4SM&$i2F09$2^!%e zl&?%SzAd>eqtriRrXM%I0*iWEOrSA96CpoD^X9$E-;7<>z;Ke{+&wqN?e!v2b+g}7 zqg!)1wd(NQ=8MjU(<+Om8Gi0o?=gN99wG>UgAvz9*JpOa0y)-y^A3#QCbFSYc_T7{=A+G4@;e?D5J zxRJ*n9^@=^$3}-?_Y4pN&n1q(p!;ayjnjwQF+S>lHhA`CC$=JQkA30}HIy1l=cT+{B?K5AHkwM0C(Fxu;Uvm{ZT)>Dr_!t&ny4`v-_d;l%`#h$k z(YI9UeOilLnMf0;Z%E+f;v~ApgDKQ~MSZp+ZPU!stbnAcNN2dcPsH_`MNG`}^=$Fv zETYL^qMl2^O)+1iFHbOTW~3%Za(5iw-r!=Y;jMyHE}(1omIm;#^Y5x@1y#PliW{+n zqHG7kzD=Nql3A>l?}DHgn0hOyXhof)FRi}notWEA73*y0;kx^wNKkhmiZCN)gZ)q` z%$toGlJJmof;l&gJ^I*|6efEdpQ z(|O%PXVZWqVjC0u`+Vz2q=4*@uNp9GovPL=lH@O-5%Z>_rD2lW>!3DVX&q%iQQ5Q= zSDG#xL(&CZ8x5PpPYkrv?B&c!Tu2h={NB!Z)*o0~<)g}v87iZz4QWBPdh^1eI@vlU zZB&IT?+nj4cGV&s5Lpdz_fES)GDIyQZX?bpQ7cmglO5YKhyn=+dLg!MlSAyl64MEY zM51nBpp_o1s8Ff&WN1LlQeigh-L9!#0ecPo3(uB_=D;7EdH|iMG5T6ixFHprOeR9v z5Br+uJAs#>toh^AH}2N=v^fgbXAEFXnNi$#7SrV#k8u|r8ae#jR~O#SpQbpocUmZ(m+J5 zps3yn=iyuxxKB}+jxD#@@-G0_uVLkFrtkcTF^RY^^=xDLL4NXg8u)cgg-B=rOO|v6 zNFCaOLXQH3z05wL(ucPkebf$6)&9Z8_Ci8GyvZO0$>G_jlN8UF48edo`V65!&qr** ztj;Qn`iUkkBwk8&qOAG7k5&}6b&yFezm(ljnI5jxNPA^rq?a{D$KKD#j^6fn^!$cGmB1SQBA0;IBVtO@mRTyK&}sj z1ejVmIoSId3SLz!DLuCMC>R;PTUmiIdQY0B3vWqTqAs2ohoT zO8UVZAHR=3HS0XqaznX zjj3E$t^Kqitc?nKUg1L2E5Q^+Yh)wxbbPa&;jpXNs;mDEO3l1*l^PT|YMfFcyf@k~ zz5J_WT5l;yYek?!_gJ~X2BA^)7r>y?BDd0KWX`6SdUus8>*{Pea{9xJ1>33G;fdT! zac3&X_NJkZ3;64=<5Ki-37>#_e<=+d>ra@}#QKh!;TvA<3KKw>?Tp0^M? zPpZk4Apnx3VvYK8dq%a2PsV-V3AKK%PhO5r z9t*!O?=kZ9#A5FUs2y1$G;T0_Zi%+Q_dK^e!D2z?|1ifgq>&*BjtT=&$IwA9Ki_P zklm8QjZtK4-!b?&Y@sD9Qdy7xSkOaGMcuVx>mUGNcybKx(D^n@mzv_%^@R94r`hX= z_E^0aD=(rPYyH}KlN()p#%dTosPes7FhKkDpCwD}n7y--W+dAut=iUgVEV6I=IBH7 zxrTj0TGe6sBbzTqw6xiZaDtmR?log za$=*l(hJ!Od1!{x-iU-82TO`G^$@*cTUryzYp0&2#}~~z=bR;-JLn;fglS~&LgtRi zM*jksz0H1$UE}}$hUJNY{tM6aCqjfbtecK;6~dnCiZX+jqN8UY%W4bQ-)q3_)k6h` zL`@dUoJgER1)?!nHB6NHRJ_3C$N|VkLfXvw)NU+rW%+`L?nmLCrmj%HR1osUh9Exf zUW+rCCFRw*&o}gy%HLF_{slZ82r0WOm|k}|!p964R;vC%$HkmYk{TMEe+CxKZ9O*I zX$LP#fUb-ct|Ev+T0I-kfz#Ykt( z`#?|X3~!uaii`*9b`N|%f9t>1>b~{Zuxc}|fk(!%wJDx4=r3S`5Jh;Ih5t0LQ-5Bj zqFuo`-gTkv#ymw_^nLp$(^7*sZsNKr<^3{I$jLzl@-Kj05&Y2Z{BEU$FXyutp2QGM zkdcWqObxhjqUyHOvssc&1ilIv>#G@DQ48z}Ot^)Fl|Q5i+d?())`DM3wEH|9Y~sJy zifcAvq=lw@REoHNSFGY^ME5Jy`_F`z?)o;8ikU6!b&%cDOw3=vFgWi?zynY{8Zv`K zk38=fGV8poQuwpkFsO`~wcgV?Tn0DK{?x<_fGt&66N;beY+ks|tguPMS5Rd)>aUyC;{(Ln?hWj7HDc=ZuN_-mQU+wx-f2z_I}bhU61tdcC1XqF8H zwp1IPWUkSk$>8L}tlAtj5RA``#Mr1`yV5ZRt~iaS8#9iEW5vXyyD!<2+^~W&4I8rF{lS^Bb8iFU>0M7|U!-|5Ekl z0qaw1WJYYY?!W->5;Mwr`|S%+(O!tZ+9E=ff=3hCn`UD zgbkQyZZK=tuXSVEE0nNQ*49bW2|E0$QlPf9R5?6X07f*G#pXnD_T$Ku*WfuN)uMWX zhqV#~31j25nne?I5$1`Oivk%JcPn2hD<(=!%uvG#bA|VZ5KOA@QURF$O3{qn0BT|; zfE;9GED$!Hzp?^x6LXHFyBYNJ;RsRJcTz=6y*S3SG~G0#leSMR^>BL$ZoS9fx`2#A zq$6NcmrpAou!mWwj%UBO-&dXGq!5;puFY(OzqX&s7IZUBz)-t#3W|Za9#<5sS2w-W z^Oe29YQm6_nJHUkIe50^o--G6gs@#^)YN z9lWT?3;7ctFNXFPj`#AHEbYqiW;h8gK)Qn5;frpjt;w;6oZinTc&{&r8qBhu%eMf; z>(ig)j_ivN$CCrOQZ_#CjV#vb{h?{PO9)!&$4;cyI|#a^vwo4(LmqBiiUZixglS0l zR&u|nNc^$Aao%>U(C#5Y(OaY79b7qty7_EkX_{P;n(J->b9Ay$tF$W1pNqeMuJ6P> zkq*OO$Fc>9yd0rcdcG4sjtKlTI#2W5l*~;5eeo3@g2Fy=Q3Xf?hj8e%()ZJ+8`ur~ zw#GS8_kL?pXAPZ2jXaB?7OpsPN8X0=ilSSe^Lu{`{M`nQ{Kg_JW! zAQhoTrY9_k(1Oc}P#XtNwO8IbQ@@IGUH~NsYdK?OB#YSkX@ZUqBO(8X3Q}>~oa3!& z!RTFmC4+GOps4d9-Te~%H2&rXeY1kKx@Nx8u&jHC!CKc#`gMjO8ILaLlm1s)8PsBKHh+^7ey)}GC!d06-kUGNBF)fQC^g!SSzVtvD znPpsnZ?>V)FNwGu3H&TOvz+{-XD~xvEPd8<3+)Q#7YHJErm_wC`XpuFZ4yPiYu8V4b!IjVv!YD+v#z1E@apxts32jk4fim5j!oB=(FZO?ou!UZY}d

hPAkbbTr(^R-tMK2 zKN0>_fp0py#L1`FRr&*5;-DDr4LLD|sj;tZQ3zKN@*cwl?-923bSUpcacn1JL{nYv zZtft-cac}hZN}n*gHxCHDFv?0CxyQfV9AWc=f(?@7if+U5rSJOQNp>2UTVfCY}SbF zqaxj|>0{M`iOe$Rt9u3RW`~8f`k|HzD zF^%opSyX9+jYUn z&%KiTR$^Vi!830A>~toy)LJOz!?S86`uLBg2Ua4rSk58ug@-Kly(2<+7h z$4T(;B=Ewob?78uEe=6);xm2g<$@*#FTG96J_{Tve(iRE2dF!0pqX(Mqorx|Jm0x? zn;B>Y0=^EuFo+Rx8l(n?tgXnM=cN6S#$h3#tgPlvKPyb0qr#omugkC3hSZIPlx`Qm zvZDfran~yU0tQVc^qq>L(4W@B9>`>is0P||d53}ZJl!nJr>ead>u!vlnM1|`e*Xms zx4$4%27eqmmG(+M4H<{0|CsP_*Ls={{hFIIzEq{!#hDC7*rt6dNA6FgHCE_&kKXSgyWf0T6hkPnzXgD zq4eo@6Lk4bK|I1#K+_75c0QPlwJo>K*xk=|54;NMt!uuNsV{Bfxmf#738k{MdPf2n z`^po}=l7!Cwd+$7`6niNYFh^%l#vXpwrONCp6VgnR?Ol9Ha@A@J)5MHD>i}aKH!!% z8#3+sK|2b=6#N{XdVRy>j|Qf&G6?aq;T^Z7iC#t(IdC_9tV-0pTutTGQtR4vtZYDU zbIrRyLZ|hq*bmh6^7hgK0&R>QPR;_(NVA&k=7q^Y0SYmUS;0Q<;zexTndS5N4(}la z?oNwrZo7FJt+-SD9b^q4b~Arrav(?Sd4|`mhi?prte4`S9DS(ncm7Ez)ZbnkPRDnN z0ZZ*Hz~hg!|1fTZ(Cb@S&`1mgxM1n)e3!5{XI)|f`M`@)BCM(`Rb!v<;VKU$ z$y09BH;tcS-v<~`(RqLOVg;Ue;R4Tq$+Q0{8%JPe!goy?$K<(N@mC(*`}#p zCa7nmM7qGa9Ny3?c8=&X=-Bn#I2G@oWbv&XBjuz)PFBW!xPRCuGn6O?cdqhk+}9xLsyJI|)dcQ7%PI=(?BoP&&tt2<9Udg<0YlLFU&$)A<=b{O_@mAR-C8?(@HZo= z(<%gUV5$Ho8siaRU9!-@v!|ZZgpF6OhegI_&2r5?H0q;?3?zPIjLab8y=$MZJVo$*EQ>u`JXy`1j@@{YN;(mL1~|&H)xKF6gTz{uF;2> zlL#8K8u5TZ0Y4w1R;%MP2d{s-sr_9T9M0c8=4u2ZH8R6T&m+xfBR;|_pJ)&h*J^~O zA3t*oxnV-%*;pn5=I7j(4pyT8EBv%daJZN26^m<-IHEHPgRlr}1a6>nI=lTzgH>Xc zOdpn?p1n9%*+jR1DYQF-!RDYza>CX={rqF7ujf0siB^p)mLvf$KbC5e62j9Ry>T1; z08;FyJIURCr|gsXCQG-TRi;$ESu*DyDUV6fUXMNCGxnuI~dAUJzO8#6NUx;9&QfT9Mg;f}KN> znh8dNpshkl)u9(Ip%;NKf)ibmGjBCzt7M008mx$5cwqO_a4>(~d%E{CXU4r{SJDYB zyeL}rZmBX_5Md{4x5A6TX2(EleG_d{KzzFdYm@zcX9t%pt~iB6`_&7G0X4U-Bv*V^6#fu1z zWdqkq}J(~TaA&G>H&zxkVtNSv~L%nAXbjFuN~x6wrrARi#y%DRM#cF&C30@RoX-OezPAI8PYw!62fhHh{vM*0a-Kb=qX zru|7^P<>Cusc1?9(gQ3jLoUl~7(+VlXa_QMZEj86vN^jo;`*-J{PSPH2ki$=5Nw1a zxE`6k2HIl{8h{;DTdr`ywHuEG-ZJ!*${#kbS(nO&=2@%7i8@Bp?AW?WM5r$g9`gnW;+R)l-b*$*?EEt+alshA9LgB)?M*w zf9@(tQX6ps&?#fF19D!_^*p!4%}b#H#$9h;Mb!u2&V;1zypoCKfEJ% zbO@y~wwB;gtTT%a*8C!&i{$GVDoP8d_7>nFTB&ozHZwfTzEOOZ_zVk;%)UjjsvD}@ zU{1Z*C8Ij0eMcHyrmyJBykxXmnjFunS85YV>3CPW`wQ&ppe3s(g7m@?rp) zS)Xb0-APx0`>mn4LACDGYF7Du?;nN)X?{)DQ~EDXrByui*{ z2sB27i8Tm!H%;3&KAn269Dy&8Y;MXN<=io=eRRnX@*BPLYTbS*yZC7Dt7}SVi-J!! zN}z)fG-JmBMF7`0w^N%u7UNsM8&kDkO=Zo4LA(tk&5q zy;B30AVhVuPE+@ur8Kji8QTsPMLWa+-2QmOXlZf69?_@PS#jb8~fa*B$!H1=7l9ZCcOK6$(h$eWkcsgxh z>1*)FVwr#6)D-kDU^;ur)hW^$JX%hAP5vK$`%)zg`bvwjA_aefac zvCVT?nnH>hhRyjjTdq&XTer1AP=FXePj_*vm4%#BO9!`FpUnN7EFzM5MIxRw>mHW+ zvj&&&6O*DsSdX_A4|#Ke?$v>(n>xQr)5gnOauaC2wsd*Jj5iTU$FY3j5`?6}N!E~j z%D_Z44H*9)0HHu$ze|FNdv0)OCl!N`MkyMhdM2Q}Mlw$IYPCvGBL=9DW zloCM%Qku;!(K}8ut;c$~+O9fqb$ixXY@P-@!_ZZ#-Et!XIQ!8-HH%%$5Io-+2c}*TocSG^Q>?ABv4?;0`?1MU8g{0INTGkZS4({6ik1m{yCD zXa4|;^{10K9rX{+stCGw3a8O?2YlABQEx=;@D)y?aQGa%(byl!ZY7>X8 zDO3Ld#Z`~mDIBVXe)5LFA<7mSEEfx)Z3eTRtG^u!4V)`}4Lfw|6(|&?!}c{xyGn9M zQ6%7ZpxDV2`?S}j&CsPSNm93MESEL?vN83|YaiLxS4mrMS4)dUah{KEac-m>rvs3s zg(wk?@!GN(jK+Dwi7|qi2ltEluM2GZ_63r!VG|aWhbR{V{ z3IPGWGmiP7*U~yv#{Mt$E6aFJ>6`p(a8OIBQd%oZs5=pm2`AVL8lkZw`EhMcLFya4 zr{re}jKi*!H~e}^lcW>4{J`R^eHnS_nYNPTE|tVMsv)I4DW^3k^0ctWh_=tVmXqz; ztq=Tn=a`jQWC9?09WDr-pxp$6sDQ>HrT)gLmTKaY=UZ)t9*o zMzv^za%)n6;W`*gWAj<9DZwwKon&w7C3=cSS)wP($La}fg%A<*YA8YyJ7KpFQ}G0E^UDb;imX{gB&>N>Vq|hcu*toazZ2@ljl* zx+z?ATkW$S>n0ty+~l&kPpwWxkbs{ROZ-c3$_h6bQa{96HljQsVd0okMgh+wvq)5U*02a9kg>+#xoX0b$qefu*59d5o2Dl)!R z=9aG$!cjW~9DWiBKwog64i3kB_;nuBZtCl0_SFo=L$m%yU@2M{OJ+H3dONb>Tqq3aXaOvB#sJFXxgqLH=@eT$Qq@=A!a)hjG;M^qPK!I0FO{L~9Qf4I_{#AaQh+H+ zu$&Rx6(t~UHaXnySZBjjT`tmBYLwaehHwV5b+}Rq(~+E=haJLwYw2%_HydA5baI>$ zVvB@CONjh-jx#siuwNlr!6;&4_3u+H0HNdRNBBg0+OKSMov!S zoPVts`dtSfdo@m-wxzoqj*Gd(L^a%GG^R^R%Sw!t7YYi21T4Ad83S~Y-l`o+r+R?e z;kLscwB%e~>3Odqpnz0`1x8xH<1DF1ZgihYyxAQV+x+{4>1E*!r^FN}1f|DR;$KT> zC!gLWzz3ltgIVWNNtGY3t<&v64@S5_i+qk2bdx?i%2-Q@7(2DCvY*5Pxvt+!CsuRj zNwZ=2UkW|<;cdp@c9@w@+?$^I8Cgj1U5zCLwPYNEqUNj|WOg-_b#3Y+vfMCU`PE0) z4Kf@9QcsShX0qI{V zLYr}!bdO?f(1|l8w$c!_X$ez{01|$R&2=e0$wgAOT9I=y%c3}OPfG#9p^wA~)wiJp z{RiHqrgc@h_dZ?R2HJ=V2xVWpTGF(Flb_+xSApE(p!!bnYJ=34%VegO!qhL>PQv7Pw9Ay=o$8w8f`sA2mRVBI!}hnk>Uit0#7Mfkfno+D1dY7 z7fpOTvUE>VUEj1xTUPsWx@{4h8dSGlh?%cGgFehW$;z(w_c!| zVYKuevGlZ=n%3DZig6$#OI%ta#gkjLlrA^ z3v@Dbp*oaGQJzLqfPIa5`La1WM`kD`vZ}tV7p|ph!PCllw&oUKsh|p!Ji1O*?USEx zTKWdlDbAbtTuq`u`N{9bw0`#**6K<;RRj2Pp^#OObfEV&iF`HbbMz(SiD>&1oDM3mzwxuZJ%UN6toO{Ht z+A?Ake$RCdmKP~3$B31jl>@@xJ83EBggcKl!UAqEUWhiHsj81;(30 zUSirXe|mgkfPaLkWR&PNgohcQaFWavKy8SByDC^KZ7W*Bm6QBSP)W+WBm}3qt7C*# zJ6(~sM^x7oUyF}X10+g;6tsSY(R)>)Sw zc^Pf?T2>YnjX}m1q@Bt^&XEbxy;R%@ETY>c-7`XplO4GaLbwLAf)X7{!=oI8sc75y zM;f!e8PneoVy6E9Ze^)UbssG1SzByO0RALvLW@gQMpKmX2mIC9iN;zvWpH--GI@>l zEQpdmc`_wPN{+da-;T6gLNHU|$w>)JgbZN5&?g(@f@(hN5*JvgmZrSVQ6WKUYCuE< z&RWsu3oYtw7g9XMAOf&R!Agp)mmYxlzta;-{SVc#CdG6Tt9AO?xKjd~-xwez)cXYm zf=-nbkSUig&visODZBLi2p701E=aYh_L56^8Fn+SfTZC;Ckq;P2O#Wd<61+)OSE8< zEH7GRWxI{lO)DX@G1SUV`2__@C0Pe4K3>$jb-vTd zHt!ADw%90s_YjjRJ4^`ET^P!P@rfvLc)`@1>c)H3KFqe_hRcj4hnD74*wv>TZj-tX z({F0>Jx)Jrc`8mG^JN1EIRxYU>*s&%P}f?^_^)@B z)Sx*TE}xTIZwFEt&J&Z8QhkZz7_X*&70A8V`fhJfD%T~}BSUfa(-jGj>eY?6Dgzkx zP_K;orYrYL^!wY*mV~#~@)IG%m$Z|R2+2EDeKTv*?)G>;>dkxHSAw13t2+Z;4G>15 zQ-2}b6oUi4nihEt5ka0dz@p>CQq%^V`cd)iOK^-Nnw+=T)N$neg(RBrG-G5`dt%lx zq|xnJ)u(jTrX7gj5|7TMnXzDF!^rwl6;LQf#(B|Axn*8IyZhFZn+KQheqxaAgkY#Q z`cX)37txkIXIeoTXeVJTa+=d-SjY#+Dgt$!VE*qPYBVc>NO6KTl?=j;=6h_E*8lsl9-dVC3(;NU}B=%|tNgn0{3#$)C;~!d=alCe*-ETQ3*1n+jlOmjKjj4qo=QSeg1q$PShZL4F0x31% zlby3f7QyZBUhs(oID zLU1xFh+77cs1kouK{`pvsTWbUJMw4;k%CThSXRcGbxo^0VARAp;UEm)gHTRwk_Bk({mXCb|4>sRg$0PL_+xN987pXWNCPgv< zuD5$n^PkR%YW19DRhBW-VB-#cw0k{6fIRryn#R$mvw0X8P{-DbMbr_EXZ-6jV*Uxo zh)2?-qu+K1#LWQqUsrWd8y-FC8s~Ue1Ov@Oi_!{nuOfw!a%s`A$@ihrg)Zu%?gF_K z6#2lQk2dt$NK%UNkbd-uEEgLpIZ35#bgZad*0ngIr00FJS_k%ZhULeuoVP3~EXso+ zl>r$}f>eT>5&;Msoyi~NR9mV;e~TI4nwL)|%RB197BsZXiM~!%-s}ZJPUAuDTvYjk}ATJ0S>Ehh2Tv+*TWN#86Qw#zK7AI5|#9^{w^WEH>SGz&1)&E8AF z8hk`UVND@m<0(*az!W=qnQ*T}~p@rPsp zwf$1q9`$I)f~GvAvO8tH{-eJK-=EQN>@Gn_D{$J(ha|EO1=inMo$SznNK;GZGV zq-zN|@iwl77&U1e^P5^IRw(^8!pq~R?TtuuDJoMkS1G}Plj0>hqpSY_mg2wat=IcJ zU;1Lg^~%9}hQk(X>e-0>&MbzP9yG8?W#z<5))10+Iv3dCKAGQ9bQGIo&0OtmH+YQG z{tA9ou^ky!OKrTDKvPOr;Q?Mns{smGHDvfoi)>4q-A~g!0V+Mt4rBElqGH{7^}tXI zIuPQD7C;5W;1I1F5TK@&MUG4MIh^9w%}e6P!-zd04Od>Ev8E(Yr_`U9I^#0pP9JZ^ z8>J4mpyO&7NlKNe0Zr9*Erq*7@NJXtGFn&4i-bFXT3l{qc#&Kzc%RIqvb2IfiLkH< z(Tdw$l-=W@jkW1Au2La3WlX2DoOLC)+2K1bXa3vtQ{=k z1ARRvRQn&Cywi}@*06>#f;4!_C9~zilt!GK%4sj98ZEuD)zgpIZ#J0n9JL^!W9R%` zO8xsK#D|mS+$lOi0bGYwQj(y$ZmwI@l!?;})VS+yfTWm_)Jk-y?p2{fZ$M2^EsK$L zd2m@vFrivTQietpRii$N-_oyt6K~ebC8}ih zBRa#*-eU*6OzPB?6@{xA&IE-3PrtQ%+QstWH{wA@%)1iohmf`(Nb^QpCsK|F7;Ply zw})Mr<58qdzE0V_{{Uz<=g8^%W$VXGseM5aH#trr1NW}7*NZ9TD}oAS0kFrD9O9_I z55HwxI$x-IfY_4t#SxoC2RTqu+(RxnpK+9s2L*ohru;y*C#bqAP0nzNi<8!;8MLys z97Hcd4=F0ZAvr=pDFCO)S04}_3 zx|TgFDLF^(Z6-#>Tqk<^H-jU=Oje7YAzrMy%VA+4o&3v2Gs(`>&&3uW>D%mUqM;@@ zx-Cz%>OsSaZKj(fpI;RZZGd?vIjHwwy=&=bZgODn1=ue$Icw76?z%sEfI9#ZoO`Q1 z>f9r<@i(RHod?@cPVO0Kxx>nI8wmS}QP%bO7-Uz~3ffm{yZD9%E7as^G9HeJ(1$^=F zR^blk))v+;&M7wOueTCv3xPR~>d+U39{Att1xt;_*`n2-T-z^DM^m$4xJhxxWkYf4 zO(VrPk}?TD;oNGf?^X02 z(Q;d#5@4BeOqW6nSD&Dl2aK;We&EaE5ld!E;fZFu%b8fHUh9enC;%9 zde5nthiQe?-9Tc;xX8JvIVELIGadsB68h9vooXl!0Ca(^cmSMIX7#PHH<_2o*C&j7 zW39|^vW-boib53Qji)#&Q#oxTI|c2Hs*`qRFIZh<^%d&XA|FU}i|$<~q4x-CT*W$xdixZQnbS&~Ev|6Y2eB1v3S4#MM5!rFrw%U-?+k&HnDldwV zw+O+&KfEDW8+d3)8nNMnTX^zsAt(5A<2^{D7Ql0k} z(`@*X{pRN?+~g~5&Bmd|F`FTGSM()r6el{LC<-G=e1T4HsEwh%DRmRQJq_Q6{{Rha z;@;aTDZAVnPBgl-oW}}l%;TgFF~|e}k%Z+&-q!R-+;v7|!zZK`s3>{qGJyX8ew51! zR-ZTb0ci?I1BGeIlz){w84G_~^fOm5i;|-?af$#ANQzSIbd$mqk_g}UicU5gRSqG4 z!<5HNPFK#P_)_Ow<0?mhjD8fYg`l!de-gRutZrdniPaFnRqk!oFj!7om1~w?2qa)> zFCkSrwc){Yd@hNqoqc*!fr|I8`-8FTFyPORz9l$4~n0UWofLp&sGnwpPs>#mud@)sn`a%>jD)o+m6jjiMydE&`R zlR}YPn2@ga_{mrAV7!Sj7poyjTw|}n$mc@9--`Iprhm1^#7lIvS~}~cZC7}0BgoA5 zi^;}tmA2_6dyHg-xGO}n{Ac)0)IqDFpmkSB-qeL4Na>62QH{s`?)J)1&dDTa81rpi zD3jlAcggRL-R6Ter=k zcZUk#!a!M;!CGHd1^`kV30~*;Navqgwb{DYr25X-!)y45XzIsetL2d@d_~!>ER1C# zdJY^p1QHTA^{#clP2FC!1;K7r`4PW-BCABlkPi=g&zIRR!kGXT+luQ;XTnm#zk~u#kaJ%h77`Rf5E46A(|v1nzjUXgQcF3_D5sz<5d6+&Sz#m5dZYay< z*vInr^s3F$&7)V6wfWYxfN+n`imF}hX~8-`$m12bf>82ex4T&xP)Qi)ZR(#oI-6cF zd98Z=Z2>Dfd-_&+dV&EY3Ur7{1yyP6GAX4ed}ALii+)ij`5x&4u zGKqo|0qASG2cY7fDDU}VxKKtxAFUv4V*xwq#X8=@Z>=R`JbKrf=bB(K^m-bRia6VP zk8cG@+r3IfP6B-?tU%WG98-x%$l{p?1A0(&0x6IvK0XF0v5a%IMkVl`4HVP*)11{% zjach|VBk;^5;KegMk_eT=7xA@YHAUYj*d?M07?zscI`*V8b|f0_3m&^$F&M1*~Ca& zV>vWqS3-fwpb7`lnRg*42OmmFAX`kD{j-6)nETU=-}Ec-ezj40LXnY@2fYC0$!)9g zdsMEOOEn1nMZfo)d(rXwlS$cilQVoHZAbH?n7$r-Pu8fNg&O>RrpO+By>rb;vDAZM z;(lharZ41U_mw9XsJO$2erdyI#>Y^ed`+rJi`v}b%zf)3jnqjv^X_V5CsDW*07+Q( zJWys~yHC-{cjzQVI5_x*bfjeYd?a zJ|hgsN|urU^5=d*{*)EP=F9m5JMU69xoSg*TS)UHg#*|hZ%VI$6Lx)d)KeLrmSkJo z@mTOig=tX)5rQ_x-)vXT9}v+ccRAlGsID$SIk((e3qb&RwC7?;0D!EIL&a$w9n}{J z$qSs*h*57spDu?QQ6tdqKEzcot!>dIDnyICHm0nVArcvP3qo*kr6e4jdvEKJUp?xR zSr1PIBz%+A##hdn%aeFZZjs`{@my&PHh=KI!Af!2I}CA0o|A4(lr8q;LW43}P?IT5 z6$Twn=~IVep=2X-woY@tOZ{~Lgvoihw|@Y5o@Ki|LO4NANYSL69&iV9wS5)+p6)ie zeJs_VtzC?N){=#UoL~?Xl_y|DYRr5;=y2;7--vxRGAzjDUq0evpq5=) zbQjg5`^HoO1!&czoTvC<;lBs7MKUTCS%l$gQcO(quSGdnhRhc}D{n)Cvh8WO1oE+Nzqz(Fsaf9k45>z`Jvn6#l}L%g7C< zrWD?$QnfS`vH?nx(aJ(qry)o>c2FP@Sx;T{ z!?zpKE>;QGG&gaG&yNpKC|8bD60IkP8&YwoY_I}Vns0e@Bymdi4}bW`__xB3Wwyq) z^*mV3++j4*RAa3H#Vz)8&Im5Jlq>?31%bY#(jwlVS*`+mEJLGFqP9qG(Q5qg( z9Oo%?v~kad;z-I>q)khHwbLFX8mX}}qD_lr|)T=V4 z+TzEOZP6h)9uwuvw&aJ~@mT->Hp)VO=5a%{Uu+3w-jqj8rtamOSbCD!_1X6x327{} z#DADbKf;B!9S4vy6Tztdn3&z?qHQ+|laB`Nw@2SWQ#eXAy8NJAN#6}R(ndkpI0s>q z`o7g>wI&^w$8C9N>lR|fej!ha7EsvwZfYc;PhLA%%CZ5A+@Q^hBBlnDv!Q7Q1v3J+J-;e3+jE74L0b`e9B7XCm278 zC$&!WUrW8$cl(vn@UPjkEmk=%E^d(L%e2xMOK8$g>se@nz5?7=(zR}ZgKv{}wavB| ztXuAOFZOvZqspk!(IVnnZ6IX$g;CL|Kx6lS5;>>IJ}IOuV%xHj@#n5OcdI&r>(P;| zZNZNMBJHu-OPFq3Sq``pqi-^aM#Ex|dsoi=A^RpLTiUKJvK)gh0a0x$C=JSCK?+82 z2+EL?#@;CU)#Lj_^p)^l6VijHM&8n@V+igr3z^AEQEDI%nP&E~&Iy-nZW!nwF%^jTzYu z6`^jn>KR+wRtdsgB09kYlAU!_z2DxE8o|`PM{kxIgK=rqLx_@3%wVkzA$Z*&A)=GQ zd^3P|6&=&<*}CFOjzV=Xo4#L@F{tBCrkx8)T2gRw3fxK)jtU6pQKx#I@p@IED}g1Z z=Ank1@S?V%8T6_|h-?rOpW-1)^yiA_;F$K(q7SGnuE(@Yi6OKpl4|0I`cBV3tU7J!@-J5vTq9aguo0=ZLroV#kC7>Gy!y(r1`=15 z6QsL*Yn+>`X9Q(8G)`r!dns^)FykSl$du}pokU>1oRTmUui+V;>DZkW8L3@QbcEE1 zw&gi)>HA!l6j0<=;s+ZJl@JzEg!qe3a^qxzNfhwV`yLFW7 zYfgjbOj|XfASmfS@S9LcIsWm|Qb`HiR-tq=-w~dXV}o>JJ(?s%E%l*~4Wx&OQr&m( zB_+h<=qgi%v>+`&Y6c$H)l=pOPxT~8jH>LnZa3c%M^w2AD8R_UEVkE*tvqRM9o zpBe2oOEi-z(jT?NWyuVil9!pU6{L9>NZlZ52g(AKfH56HDF_8=0R#^=3u&|bMCo}-g51T2<;ZCU+=dvqC&D>W zl=hLM%ruQWl6TF0hMukKkBe}jbj&M=wZf5YYDC$qj*?t))2l;qf|M50H2_PeO4c&3 z6954vbQev+YJHcY`lI$+ExZ*zU&x{P4?e7FenYyDTy;uNT3aMG+DJYb+bzm6Ku%FU zS-R%cX-;qm&IM^d8onK?sqPl8fwB}r>FY#9Ou9}eNpX}P z?((#R;lkQhu#lXRuymZ7%x~Qta7BP$8b!w2V}P{Bv_ewWkmAltQGvhV%9N$#lB}F$ zR*Fdv7Tcn}j;FgL#CqS=x0!b39e=YfvKHw|iOKNZ*4VB zP*Eg>IKqh8D+6T?Frl)&0m!9Iha%wPCRyc#k>4Z{w|d5Y7JN917I1N^QSf3k9Z*E1 zdCouLx!8Y&Cw?pH9*Ik@!C~n#nr)_(l%ce2pTawBx1qoT70(NEq)XIIaVQ_!XQ>%( z&td7O7R0Fxu%8g2B@ZJB0H~9cC>(K&``67YQ=TC>UjG2i>00OZfW4-7TR$1cK4DRl zp(-c|DMukW0B7s(St0QpR&YQV{cBnLHMvG0N5QH4!BvYPyG$QCeNJ0r+XC;?JO@=V719xTOk0 zm&QFi)Y;D@Iz4?U=8taP;+R3tU@7*8eg3qplM6`$1M;Gpa~q5us5A}vr`iNyjke83 zNESu90sjCh4b=C?){k{K&P53FK^)i7Gu4YD+)3Xw!jLnZedxzi*kYL}89Q@G(MYgW zCw<0g3P#*fD@Q+{(vVTMPTbNA*dmdiT1s0W=ZY^t+PG1;q6Tq>RHbJdpK6wj%7FV& zkkSe7QtaTQ4slG7sA#sV9gQ;LPI=mnKcS{pfJxe&L?bO6jFK~)QSnqb%>n>P-`9ap zvf@I48~tjL><%P}$QzyLXR;1+k9v4{hH00G!5QP$s*sL91bm#})N`csQV)`H1q8O# zPShFCIE@7XkbkW-?mJV9LWbj-X+c=r;EG1YS|a8k?gteIEwTy5)n$;zpiZ7DB3xoK zw-p&tM9~IRq3SEil;TGx=}x-N2_RDoY=O8m6oX^QxhXu8MX`7w{vtm04zvTcAR%cv z6-zeDqzkZF8CM?lY_iVQ;2xEg#apO22N|m!nfab70uD~xR@hO$bj#75)33;VNUOv| zwyoP(AbM73(a|LqQny|uKg-^)uvu}Hr+^P}No7*eYHTOL$ymUsJFK$abSUrdQZ6Yi zM$}dLnyOsqqs9YYC-bI`#+45@48500B>Eb;biJ7ql9V9;IgEPMe!>wTgt#{6aaw0g zL20$6*1}VcO*?cV?381pt~R(0-yL6skQ5T6x$le=t8ROWqV?;-U8Pc^v`f$zt8aIwjD zIU(g3uOVTTXUtQPuYW=^2-wxfr}_usUs7A2LM6^>#e$?^p2OD{{TW?BPLC@7*n?EqjE5aq+!Og5L|4d@iwqYNE_|7 zVUaqA;v$gZBf9K}6ZA}N#@udW4wNWiTAL+$c?C(_9D)XLXF7)E)K@h^YpCx9*|w)r zyI)c=+LJL2qjvmk>RM7pbqu(Jm1D|jR_l{q1D!v9k#39?bJZ0{9u#KWaVa{Gw#iFM z)T9p)wFD?Qm47m5@<{@NXWTlS`;-9KVf!@ThupPJXvjlsD32Jn=5msS6=VW3k`J+R zOIGA{1-@%9NsiQ~T(`ds)hTId+@nu|<}#3=Tv9z zbIhsv9{vC)i~8=4Op5A$LFvwyLJ1xkl(N`q#o&2P_#EytxT{t3rsj1sFEMVBF!efC z%QM?b^s4t6rj+l_wd(RPmR`Zin@DtIrF9u^izBEYe1-ikMsXyv)9QapWU}c?-UGQF zVt|vS!RAh!`GML_Me5yQHduGq_c)iHl4TRwkje_)Ub5?He<|#etb>v=Qa2lhjJUe( zeo(D8j`ch#SBVxm4&~bqQpJO6>Iiz!!VOwECeM-)xcsh@YkTx2WG?F&SNynPE zPMNv2)v}}8rJ+iO4&jN!ytlvYvRv@*T&Dpp#td@*->uI zy>FJwYe+B0aXXJO8B$76v|!{Z0~yVIukg`}H^b{ao>PnYrpo?(_X<4tURE4H-1$pO zLbI?gBaP`ZW8%kz?G?_rVZM;~WoYX-j4k?@?eN0A%Wyea@mdaF?HLE)j zkoO!NWaAj22Nr_cx+`A|?bjAMC*p5Xbk|cdtK#FO$d46%?ZiIv-+8n*z!_6$!5bY| zHQt~2gK@N8Q4-rZw=GaFjzwkFq$Mj2$Y?m?fZM|HR4oG{{ZH9-lw`7;o2>|OwUr@5~r<=!I>Hju}B5K0u*JH zzO@06ubFX50Y@iEC$%%8lc+QyvLVi78$Z zSPwsq&ZpCoPD%2o03ZS*CbbZ>p4n3PASQLBewmAHakvf(t>oMqVUG>hm}Qn0G&pu1 z8Au5~@YX@^R~BKwhQ)^S9z4e4;7oB3Ct6vz{L7A{g%gFLBq_nGB>3nX5m8?WM@@A2 zf`4r|wW;TA<=E>_gjFC&aqzK_NC-}~{{R-CgpI}t*1nsa)*U46rgYntzDHEtU!3#u zn^P{^`hF?UrrT=+!3%Y!nQgJUu-L%P*=|+i{{W)POJeQwFZOiLyeYul-h9U)*i3>% zcA^9&Z%kTF{u6QAN;0fuH2(l5t5;k7rtc=-)3zBh-eOcXQ)FT1JWRHyb(dG14Os=( z3Xa6AZ~zvMh!*)bXm-w>vATvEx})Xix|hnt3lt@xpxJlvWzxfnQ=278ZPfrno7evU zXa3H%*`>D5FBb@3B!p!6%8ulR6qKZ3k>VsQ>c(`Se>G9HLM7zo=c*f+^nRPRnJ99= z(Q)EN9G1(gTFx*J_lnY`e+X=)CYfD^={7qkk#hNKbI458!)c%r>rE*DscQUKQrF~4 zyt&+rBnmI1m__zw=ch|;N4i`SHQ92Jr&3yEYdXlrkV(K#a@o$wz=%bbW6)>XS95k$ zsS}w>173Ga5!9dEB?Ek^Wwhk^Q;;+GjZHn$9<*3`FROkbvuSkOuaXwK%0$PI%H1oJ z2_c4*r2t!Pv@0Yhk^G>OT9-|~WVb+y%>Jih;`?l-NJ~U07j&q&@k%mMl0m|p#z4s6 zDznF-TP+qrV7JXBnT*PiyOXip2$ZA{(NOav{oTBTpdarvq~HQG2R`Dv1V-#_B6Foq zh^T=*%?>!0LP=0z8j5#L$s;4WhXknd6G-|`r&zYhj_tbW-L1(=Sep7KyMkU&3PxKQ z*e4&4QqBQ57y`ZVzR_Tn)9aHcyK-2I;l@L4t%%W*4p!nD1R!sp-nvK_*yjm3=fiJ{ zms=-=G{&;DWtB4?)XOPfjUb#Y=@0pwBO`>AgQZ^7N5=mEjonVgzr?qA_jjf&G?gX8 z9s>^ny1y1zft(}~Kjq$xewRk$Zqf6nP(IbF?DSYDGi6P{FG>ngKs;c%q@A|}gzh*S zRlniIg5>L;v?jUZ&bK^SM071Jq1`;IK4mC@u>m_1&1P3y@?_q`xw4&pV+Cn&tbwV) zp8N#)j=*-}weN%yXI?H5Zf-_mL1ivA9mbThl9EyswQiJxZ~)YOcFjZ7N!25-9VNcc zq`nzJyWAUH-u9^{=A7}Ff*B(K0urYZqp}A2QMVa2KWDkR$5cs6w?pBCFc|lKW~0xw)WQQLXqOQ>1-)rdIPdS*lpUg6#PVqV`6RIOHCQ~Ql$8K@c>XK zAm;}aEL~`v_V>{V@e$9P>np4O05n#WK1vsWay>TZ>)xuQYIG#H7me~pKRWaB-kS2B zWhp>$MByN5-?!43LxB$Nq?4cF?N_6cTXtEk2uLXy`qk#n+D?;}tH)4eLuRN`KiVv z4LsG5loaOTqUK{5qt^Jr-l6U?01DoTh)a^v0nZgny||JU?^YYMu$<)8BJ(>|q7M}| z_BB~h>LrXT2D1L4SOsA3S_S$}vafp1@0d_l{Blh+Xh|9M-9);g2Hw>SK_ru$b5=W? zZEI1+NT~A&o)Sf87e_SXWJ-qnn&y=dFe+7WGq@P0%oV4EoK$2&Q3GowTmFPUevrl>h)0BEws6`qY~yR~-Y}y$Iq+Ab>_F z*E{-9%NgdrjP#-g;&~X&Fi?KfV$qMTDZK|fQZlK6sk7;dMvPHq2apXUwtYFI88A&9 zx%*cN4mjqD@W~?pd)JW4iN-4e&cS;qIxEwBm~MVxLQzoZ&S-idh7v$TW}uASh98vOypwhEr-Csnmj){B$6kovO)a zb+(;KjtCpoYB*3k=B91r_^BSdaZH%$(YSO+Q^C@f9?}jrC;e+ljU}lmNolZhIUTDK z==hj<3P2o;1s$h1wE^l z80m1GK#+X}JQ<`*bmg)2AQp(uz>E-aSG!~jvod2fZkn$Z;t*T;ojV?RPx;b%TG*ct z0U;UOX1jHIMYc*7r&vR2BL@q~KWZ;#iK=x)<#tY;jT+?L%3O3ZH6*wkPDbZZ0YHFw z&0`mSpoMar<7i>Tpn?{qqab8{6@+CV@A$UItzNpS^k>*3&TihPVg!U2SaHOy2OE_E zl%IUn8Vsv_!SvdgmV`)cUPnsG0*5+D$DbpyA46VGsBO-Uyh$Zw0JO`t%WOls8HJ@J zswxl?^KE1CC&M6+KbRekGwD{zy+L+Gp~Xw-L3w5?XXTe7D@l~n$R2jzK+hfbJPksf zYR?!rBHQjV7<9N@Ojyka769<{_UuWCyvn^#QLgBjChWiO8PtM0)>HujW#HS?$ z9YaY5t+t+(xgzYjJu>uLf}uxTt6Ygpr6FiY0c?kW!jZVtf;8amgUyGcA-i|zSn#d) z*jJWe$Wsv(^hCD%ZRjd#w;n;%Qlpfi%{}xeCz_Ip(qXhFS4PH+W8BoQoawo83f9zu z4!;bb6samstQ28sP~#esLwC`ZyD0H`j>US8^c#v-CRL{G8RKhkCAmgA(&9@2Ux2!d zl%sT&DYWTP){#^8Xx9_g_GBdtbpv)7NL)LJK#r-f-!3b`aU>0GsYxK^AhhsC!z38_ zjP#UQk*gEOjdpCmYiJ;;ECnURyvh*NfD|}!9s{@s8B2f+ zqzbup2TVwlW1ZAChoVQ4-O4S-Gj2PTq0p^sV{l3wNiGyPPvHTzgsCLg&$Ip@l)o9f z-O4Mdi25vd0`4;PzEad#dBlu>rX&!R4gz!(k&3E=m(ov??G+w|_)XK-JFvFJY1GbC zhW=J-MJ_BlhJ&oEdJh!r!5b;}sDmq`skjK{X1Na-z z;?;xbJw;r;2lT`{z2Z!jxa4tZMxceDt#M^G;PF@_l1`F^Cnxw->5N(aIrJpnsp!6< zm2FO|q_>!Y(*eK03^c}ZEY@!_KIUVruGz+fV~ePjRhrmRu=b&G$Y;C?`46PI<4UQD(f{V0ARF z4l?3Hfc&d`xz85-1-#o~Lv<1Rv@K&%{{V^N;~P^+H1sBsj|+_Ze_vRxvykGXwCI&2 zvciI;uv|-MQC0~+R+p8Xu#!i5)_UIeaocmD2k02#UXcVbyDfH19V|)!xD+6;`_r))amrjF$ zeCeBGNQ~t9483V}nBE#trM%%wq~irCZ)6U|YF;vPLJ7}H!KBhB>>YnD4_QmFT;7ry zxCB69E;w8vO*XfWjilr{mzUxxqH-0dSUN}$-`ZQLA8V)jYTX7&ZdL9Sc?>dMEw>6H zJj<5&In#|^8+x}SKIY{}HE8$|@KLY!H>P^7?h$ffyPV0Aknm70NJ7b8H_6CJBpr$t zu5&{Cc=&w=&ueneHhFP{Om_J#*H_{=`whnb0CR}92nPw>eiC*bKmo=>rwB%68uRfpc#CT^7o#!WfqV z;^pJ1UV@^ez=a+HtQTm z5|uOar3KgBP{v&F)Z$cj0I5L($_A~1&C^mkL!~Sq4_TiP9YWywi>$dP#aga!l_<1Q zjo)zyTjxnu3c}o3N)nKx2)g9_nr&YcReE-O85b_5oqLGemYbv~p~w-`W47j}63j>q ztw1a#O~z7A;I|YwwH<2|CF8Et)Aufd(+mr-8}TWLs8Q0k%@9Nyr|8iQtpyf<4}UYMJ}mph}wY-hLV7WThzB4eO@3_i#k^3%1AR-_xplBU>Pk!==HVJ zy)d^CbDq>e!EL-{)ta^H47j%*%G?X}*O*yN> zLr8HskEvQqgso#E1xr?}fB^<$tDtz@MQ^${bdCE~ET0?{f{VyJSu)2Z;$-b!hN8b)l2zgLEM$)V`87d)eq%RttJE&{Y5(!X~*F!Dn zg}imN9-Ml_^oYrN@TDc;9Z||(ZZC*d*jONj&H&P#&Z4ydkc4737gKdVQoh$n+2=@YhFVjpXXMshxM?Q}0idANbp9d<^AVR8 za1trTx)J@%X4K8<$nxK}^u5ws(p(|9tbl^<`bw4%Jcvp_N>CN0ARWmh*0Fu*Tc!U1 zrQT($YtS3;v*E*vqT}vhX>A1g+!^5OLJ0(&fTeYV7JituL7UROO53;h)Bft@>#GTl z(hkaeAt@OrASDSICz5xgI&-OKHGONR(KgA?zT)qY0*@oVz#cN>DZUcgA<}9g*2}to;*_>EU!5WR>T91^zYBE)g3#cH8v*O z_(>*y0h== zwL^SP=%y|GJj9E5Y0m{j0);09B^;NG7BqsBO4T_ zkJ6ZpaRGi&fx$WF>t48!>I;r6B`19$q0anwsDB+ubR-o4wn3uN6GyU5*QW|UonLJ7 z5_FvVb6EYBZj}!;XckH2=^BTvO{P*T)h%*{5;OJ9TrG1jK*`#(D_rd9IsIzcZn)_K zCwiZf>WwB0%p@F&n__U%Jk>&jaoZ(TGR4RQ?y7Ym8Z9o*0Y8eNE;j%M=74_bfsE8F z`<|eok@?Yb0z(&h`hoL)wM@H7$vf1Y%6_bne_E^DWa`ci)QL70(O$c~t5*EyV^r&+ z3d(>RRjzp|Bx63Jpk+rcDEsYKNRwqr_=QV5AIi9kW=|O%jad_HQq+9Ad(my$PCpGZ zS|-(%GidVcpVFCj(yV?CYV?aiC-DB6t~+GF&WvWc=BP&mXAjzr{!>CC-mvJ{?Y9-6 z$+PLm2;5Xtmd!3?pbq~4YJ`@Kc|!UkBf=mLE`8|cVT6pS8T!=JTX-lQUFflFDg|0b zYnirkxa`T+GtD-%_9BT=H^wvml#z%^`8X#?*c?}}w?oNZFdjurXSYm5d? zF;wJ><76X@jm{`il9D#S^{KaGk4lezImjSj6In>Q=onk&HsY(4hv zUJRUUIjOlEQ;H+qV>}JLDmp}Bb|S1Lf_9-^iIKsm%48mjq=*A@Y93q;a-Y_&k3<8J z?NPEIg>QkqMaP6$EoN=cT0#dQ)KsB#f$T1Rhgl)h%tlOg?Je8wT zQd5rOiqo2H5o3)=*p5M}r)NwE%j=Dl;Ac6ax<*10vXyU8*L8TxA-w(1CZ~EL++L7? z{>jeuR2dpAmDeAAJ{ZDJJ*d8+5zI9t;4e-Bi6c+mr6IL!g4$G-q$>ieJzXR`(u*oi z2jV%~+irg$LwB=8_e(Cqh%#81>;5~5!)r^(TX`AfK!TsRrtl&>89{xu=PkO!=tv;? zo(Lwr(KD4H=NYwIVT7Rr#gXCos4Dj^0i5r)KJ>fgx)XN+Jy8_X>GNu5LfTh+08f-{ zzBlx*FV(NA+1-ZdJ;?XV#le)zA)Ap`io}K#rAkV1_?%ct@lZWRahw{kxeDt1CtJQY zsIysdd3WU;a7jXvHwsRjlr|wd?ZK#CoVvBucNpltNKzS!vRe6dG?ErElz^h)+#c!0 zXjjN$-x)>?ruO4HJR(`OB12CS0Z4Qw0X~6V+#SVB8gdNUj*Noo*7B`ZsWNQVYqUk3 zY{GR0L#*l^Qd5$U&`{th3C0f~;;MZ@<*%q6Y5t|XrVRUoi1FPFINOoc{5knxhnr9s z0ZMI2SSP({(`NOBvDVhb$EVLz4eGyU=(3aZ>n6!l-??;4` zxU{I`mfS|%g(XzaTl5W{)g7p;b*;JEjm4odGf%T8Q;SIet(P?nF8~tS(vN)-@RE94$($wC&xq!l)K1Wi`&NE5a zp|!BIB0- zHFv*s_ruh@;{IaPt58F(Eep4)))TEKTG=5=QgCyrwT0kr5(x&edo`ASL-kB4&>M$u zwaS*J;Jy;<%Ot{~%3W{wtQ8{!rv)cYbf*N4h)Kn4n@U|DMqaGV!H0Lb!ffTlEZ^Y1 z+FN}D>2bMiDGDvRk1n93Y38mREgJw zNV~D&lH$^+<&`_DxN}=hE3hPyoZx*ccelNDu*I}nTTy-Y&AAFwX(>onu;YsCbm78A zdyMl~hxSzIxv+X>(PNa_K4vQ^NA468;xhIY(q05*Nz^pz17H)_*8K9_QMgJWOvGt& zJRLYn9PwN!3dZL-B}cF-$<9S_MqH&Q_b-ujesBDGwc4RwtQT7w%6YqVnJH>0N|4Kp zG*aA&t}Q?fDp*4efChXb5)RdTjq$Zid^;9vg6A#CscGqGYB*VS)*S>ftZa~_wvQ28 zkbsnp*NSWVRoE?%I(x!)EVtWs;}s|^C0{Wq)V5q-h@}tRlH-dhNf}BKwQ0)9s((OO z+5Z69gytqT`%>hkF10D$+hE9ZOHGb5;ipp6zv5c4{1s&=@n5*DREX|?>PPMot`|Ae zd-`U&Y)y{v# z`SIk#dQ1|D_Q}D*2?^Y5QBLX!Ep8LObnY`w z{SfHhoA`!{G^ZjrZA`gsVJ^KXNn39^GN%^dL0Snvz=W0ga52d1p5~~JQ8K%?M>E#< z_0OD`_In|gwwG=bOLH&N0S-I!4y7f;w2XqLk_z$?6dMRg6>gxt^h_DjdPdy>G9raJ zn2p?qQqyS|irWaoY_tFxmgAqsqC(W8LjM40sqesu*Gb6uy30}$(Ta;(EijE9UXh$BZK?L7bZf|b&W(Q>IqVgkenx(&Zmpi z7dt#q44pJSsOc{bZ5%coCXrnHh{vv;@$ za&d)~HshWUc;O@|(!A-wNx%Z<5-(KQS!PxF_;+U<%MhE6^IssyCk0NGr@=z>fD(kQ z!lROKHmOVM`i=T@jCAxki2^a7wFPugR$h@U*6S7N*eRw#jWPP$wtQ zeJi&Zks(SB%9845B{^+^^GZt`=}USOr)=;Bt@)}o(%~@|C9{8&kuAVNO#C5BPIekj zb$eo^#a@TERc3+H*DJ(l79q-u;4tPA6P;=s{{RZoG#(E1eCjKUZhSMl-Y?d-kjr>2 zWxo)%;TYAeD#lKEAd1LrZvic(yviCw%Su5F2K4^`<~xtIXrC0P!GqBa+ime$dC3VY zd%ty%L0*h~pBS}l<7=_r!f$od12eT`R~ktNZk#qqy98kv(Z z!s5c2E5YAeVLwhP9#T-JO1D8LI!MU;=tP5S#N^lkSp;R@N_pZZ9^BRYlbQ_Icz3LJBzS@emsEJoyM2%VA zia>HDCfq`aIr~&Ri8}MW1mxngoaT@mD&K!fv{5$BJT>;Fk#AF{AbqIlvCGGU_Nn+5 zu%Hf{?YF&EAv+->*^b^%(vND)0l-PCR9is)UHxi7yE)hdewDi#tfwN*Mn;3Zc~*Q3 zXsgm~8-5W@xoQ9$Bz{#nM$JZZD$1S6N8i?<<=Ki?l&2W&T8yhu9H;*P%}}##ARK@z zno+G2akf#|x2Xg0??k3Lf_SR|w*-^0A8Hm%aD^NVzLm!y>YcVBkh7JQkoe^j}gXq2A)K2!8CA;amF{N z*^Rd1qVf&M4H1FCz|S00sfslI6TL_xHvAfSF~%@Qed&F`a8;oAvW9@{!o#c?h9wg!$2?TX?UI)G5!!o2kT2Lz7AQDVT|vC=}C4j zMsZUqh^%VC_O7lXvA733*F3Tuk}A(d=gUDo7di5u%B>GVIXh(WnhkJ?BoVbiy6{X+uMq(+HFf={YX!gsR=)QJ{yia5K&`S@&Eymx(}cPxSu)S{JadlfMrOHwTpp zd`6xK!AMlM8943bN#7OQf^l~cpi&SKutxzsvDbS z>INhLpBhU`j2R#S<{HvLD#mvl58=&hx5=F;aI?QGLo?$(G`Zr$`Nrd9=Tg>$1*JLQ ztn7Eqe328X?)Jynyg1XyLqzCX(zU4P@gYPcD;eyeMl=1Qp_uMDbz<1bXDuk1GCXIG z8PeLb^f>wu4JJ6tq>VJ6X5!aQ%1LSY_WMj&vICwK-rH_Zi50K_8E`X|XWaeG4{+(J zc4iVKK)6n_q-8%c(@AST#I);98_uDyImsBuHIc;l`4aJHofQ_PyJu{r)fp4k$_AY3 z0mW&?Nf=4aPaxECem~f{owm@cauXqf6A~zn%f*Zxma-5MJLv;fPUlZGr;dLHi(e+& zU}vXovf5h+>Q5QvtHUtR?1>0K!3`H$ASoSya4#9}+faupWP5x_mZT&);PIM|axXC; zC@b-(w!k?~dkmlE2C;sc_>Xas8R-z`-mPh6Ki;k6wzrfA;te!{vz_@V2OQ&SrO*D; z9}BGCFd0?bsXC8XOod#eB(~dOSqf2cejAD;0u%etFbQo$U^WHIt%E9Q(>GqF==+t< z_M64)rEXSYwJ4(D9zxKQg#HEih+154Cm<~<89M>Fr{{b!v|V8v)EKdpw1utd7RUS_ z2m!vPRFW2VI}%5*8&}3%bMbSj`hxJQr%qhDo=w9@2%6>P_Srscg{80>(}Tv8B|8r; zC_6vJU+o1YZ(eNHHm)}dQjsQ2&N*?p#0BY2rx2v2%%~7GA!S?S5yg1&`cfxccG>n% zPv2z3LSBl>E>M_nN>zYPoTJ-gUp4;#XfBSn+xpntBijC6ETuN< zR|xL-iHx#$3rHAJPI3Ih1D+48f7_+m3T=;w{TSqVtsM?dfSpQE&V22a44i?u)87KL zE}iR7hF=?D{>Y0dhsu$qEV$x<3nyYm#RoXS;1HaR^ISRSlhZqrbai2ba`g6X-wh)& zlOAV_6qKO|WpS34(E104vY>(dszA~>^AX=QZbk0bY=a%U>fD!bag@3tv=p_r+P?;r z5w=_*I>^RUH~`}~={}o(rC=f(M5XA-W;+UaYQv#v)U)ZVX&D;|ExxNPGTZWH>K8e3 zJVz4w{o`S2OMZL3AsCNfpdv*Okp4<{WVJqOl$zM z+BA+Af`FxY)I7vv7|Cun)!n`wUN8MaW?i1d#HFn-IaUV z5Se|pnZlcG(4?}Kk&Nn7XI8b0<0~1?anlyLDiOC-W*-6iP08}5E*p=-i++k>C7DN4 zo*`jDD?)pymqrP|2}s=5*YIPbru5fWKW}~%c;V}Wx1i5$(613CYFnfMwxo~}b$M}& z9kW$`hnEQb94#(iVW;+$jh3RZH7nQiPI&kX5yq_qDJvP+0o!};Nc9B!=S$iqwGg6x zy|A^y()01ac!_?;5xHZmR!=0;3?E|uUczNBP@CtAFtVdMo^ z0|)A^y@7n{nfCsgu*Xz}p(N!oUSVzxK!TON`<@F21f6JU#c~i6_^Tu6D}9&Y{fQRI zH!={@Npj-y){v)_&VoQXwI4ChlmMS8$OeA%{h~j!_d>MlBtL4nO&KAjT$)HjgdZ}> zjtaNnAw&;P!&Jwc+|vx;)8vT-d>*lMOU_(ww>v$NX-Yf^R|)29w^B2fS!gB2D;<)7 zw)`E$=S@k!^(=I(S?;%6bCDOQ+$~xm#%5|kH z(5gp;zX?bwK3o8Ju9zq#S4TXfk-KWmK3`6Bdunp@rF5H>)2)uE1=x|*;9xfFcE~tVlD4os7LF7WalQ@-&MNs6;&;QouY(MA zeTfm{OhR6RB16bQ(g7d==52fvq%5ya`bG{1>RXM8HiI_795{y%rkZgpdH4>Xa@l#M zVL98#>Jmml0E(kZv`Rd#q>A;nRfnnCL9=3jAN2B zRqLPa_${NS<+IXKZLhLg1iL9-I!dstJlmzDWB_reZuzI#lW@C6a@ix{q$gm=N|&MA zv!*d#z@timEpNY-AY*)IG*y<*)AkuJ+^lFs3xhG#+AZ-WL~$W!gee8&sbFLH^n;wA zO6F7K&W=1*i92fN~(>s_O!tmz-dY=9NE z9CKRvD>s?j-o7s4fjDQDH;vm8*TTi zA{PcPO)b{mY?l$C2}?w&Iq#m?{ObM@m-lY8CdypR6mJo{{SICA^fOh=8=fWINXgnsfcHB zzV!zcK}Q7bQ;@;P$KMq;z=`-ysXBM5_>#1MGrd4VNjTUHcBxqDQ8?3{{L(anPPK4S ze~5uZe$-M^l7A|QLR2>u(vys23N#xeS9H~|S85T-gy#qKpp~d_bMKnsNhi!{k`=JD zJ(XaZl!no8=_=-8%}FLO#&AC>>5LBer1x3{k)9b(+w`w#{{Rh17|PBKaL2|lKTau@S~ZcYZ#e`| zuE#r&Fa=B&FgV-vrk23Y^+@Iln;_g{{52aYHl6XGTCzI9PI1T5fRMo>7^=rT9hmYm z05lFpJLaEeKBJ8^X^b<14G`3Dk_a2)nxgd@D)J366-0g-kY+akjwwvGI3kT{p`41# z)DfKQLL+BzGf#+sk)A%3V+cvYig*-a8b;Wx^BnS#8X;h310%H|$fE}dCZ-hFBS`JV zGFnd@=bDuAk#5LGA_Z@c(wG+VPU#zZ)ahU-@%dL%7!$CrXOJ-=k!}MA6tCH7Jf|kC z4Z_a)M*h_4hqgB3(ucS@R8|*i1abOP4qEUrq*buP17Ik3p<|J^(y1JjG9hi2@S+e7 z6;hI|#kK}ie?93;=w(?0j`T|3OX(>I;BlJteKuPQHNF$2S`aQD3Uwhw;1R!`tdnCt z+o#5G=^Nsz_X%m2G)8eaNauk-bPdvWxHl!Z+~D@49HYWf3M6;iWBF12Qymuw4v1=P z#^R4LBsAtv8d9U?r;#Qtsn&{HVn|5}(wB7U`(vNI9n>9D7Uu~LL5L#CXca5Yf`jN(+7mlYMZ4yP1b3r+^<)YcR91RC$crP`$2V$6}#_Jx$bmQ^84 zr7L$(&Xp~*qfqRk5PMYNCkQr*O(`@~dQ;%F`u^*zqBxN3R|ACs#$HlA2yD0xah_5~ zt}E#8guem$TGMc0i{wj-swq%H-)RUrUdl-f>rqawFhYKuRjruTo|t&7Hi&llX!182 z5?%4VRFV>(IOk4IeR&nEbjMQk)t1(j+pe+Qj^Ng{7u;pdB<`glSRo^xPAi)xX=axh zJrs^Ljy$5a6yL)>owjuK$}B5vR$DMug)6}1gdGP~axihgHTj9vy%y_TNn`4V;uM>m zu)6A6I+R^Y%Vl65#1NeS0L&}(>dL5BSjv$O=K^~YTP>;9W-WLa#s&wJobF0%^TXp` zN4wIpZ#_QrH>1hvxzi;u;q$olvQbLXusGDqt+a4C8nc65pP|OLiFsN$^GW`<;D@nv zyQpM#q`#h*>Xm~HjTSS}^K?>X46Wi(Fti0Y~)| z7!Yg^D%l)Qb|7Uw3oWFm`7)I_q=SK_Dp+h_6+6dz6585bpDto1I~7sVqFCcz31m$Yy%yf@H z>GGjk;>hlav|R2iMrFw8mej>KsEAu(3WAhIWwaef%%8-Pq5&sK@0yRFe7N+?w(N0k zQdAg>%xMonP)f_JIzV+G{uL;UC;{A^ytQ|;Po9?Le8f1@Au#jG(HXx1#}e=uY-%|H zDNBb>Do%oT=Cf;oBK_CV;=d9*vTl+PjHHYtHWZzCJd#{GhTA6_k(y-H#Q^l(A~N%$}0}75u4VK+iocQZp( z(iD>Bq~k{5q=Uf}M^tpAIM)Fqqs;iNMN&}>{FE#xek}zgtqqgDl>$9LtZU&7wNd)6 zY@B8rOt#Q2Em$gAfyVrh0*~IlsflA|d;uX!LKoqZm8&`c;O86f+iI`+m#KG2B)-e- z{YiQ#admC}?8UC15jvK{;$M*(8dUb7a(XLC@aHy^5}%liKxM;_qO1fGe90}jmle5u z9k_KLSi_%rwOgICGW#rormje73wKYMD?`9|$=nQ_x|{%|xy@wv4xF{#y5b#@I)c*L zR_w*NtTxy|)5sB(pxU0cKecLAOA_0jY- z^GC)%gsN+#nK_NNY0Y@Yb_^sYA#QvD-t7B8*-kXm(NJ^CzprkaE zf^?^CIA5VTPC(+nJbKFMZM929zSm@3sm7E^9tsd#17M-3c%Hf7RF_BmM!ZCIq_-tX z#!^zS3KgEpPB3excy)0mvR;|?KgA2=exQQMcjj*HFar`Jt+Y0@0CEc4@dmNn0H8VB zH9+YbG)Y&MS%D?yn$~uEDAAtYLVvAPU%kqm3&amP(pH_)!%0hc$8`FANgqnT{47e| zWphkGP|)EFrA0({&}uWa5u#dp^~elFMX9MQ$CTS(1uT`NUbym;`JYP1Zpf0$YQ%&q zPWVblKc>}w>T9R%NeV;XGL=RJ4YsK5wh|8`%4;^i$dHmkrRhl+(v=aB?geJJ#Lic; zyCvnV%&7?{JM)j$nq*~r(=M%UJ)9g5QcXz1jHQ4``sSk{8~F@?!BHD#tgs@Dwg+*| zM_b{Lp|K*ZcF0-MHlzXxiyH#~f2B&tkAdabQ_&!#Bm+`R!brfzDQLh+kKdE;O}fzo zH8b*u8x81b5y==SKT10_5N{2;(`Y$W9Q1Gw>HR6Tc%iJRtsZA zBx5_&R3s<=07j?n=*a-~JXI2)7CTA?pbcE@Bqtl@srLT>FMrakmgn@5zZFr7)w$Gw z1x%sDY2;N>1c87-six!tk>;ad4SB$GKu93u*PVQ0ni1&Fq=zy||}a zY<3jtTgLdVFLzr!-vc3{2NdEO$-o&D@*Bv;1w5AZRArbcv>~=}+jH$iw$TGLrrRTt z^rG5lSMc%eNV3>$Lv3fa^zVbe6jL!2NZ8Rr8Ne7k(BLj*xh#-^Yq*7>jphn{03=OH_7#?xONTMDF zam6|EqqR3CQlEk(!x-F9l4D^cV~%tBRfNaD;LxweAH)Z>JR4&osJQDc=}HfpkB@Fr zzY}1C>qZs|To^~^UVZ6pqX__vSFPz@&WI`)a?pe+E7UjMq^;7G3>0MXwLtju+OhtA9tsQk_$a-10IxsK1Es zFoIL92sr2P(sFiDX%!oLZb+5#&CH71s>XbGG^CHND)#S8v~0?VyR@$nO_eREZ6vfw z8){fkJNwpEWuH1k{{RK1*m1VisH?_XYAMeNR^vZv`Uj=DX}6-Hd}-=zV+z)^zNHb^ zsHhS40+Th-NNJxQb^idv`}BLtybElUG6_&a%2qZQ$x=WE?fKM~!`_~~+x!<~O|!Wk zQOcuh!|6J=0a{CeM<30D`d8K6MbH!O0ZNRZ@mTVqN%05QNhAuO-8wTu!;P^GwYG7o z=a7`6d=QX?rw7<%``4cH$v!P;>R8j*b-fWAkd~$1Os5%W1xJhEIWdm<9Nj7-7~G8i z09v!!y3#B;X|p(DA|jRI;q@eysBxVNDs5h>&fcc7gC64*GoOT^mq$;lby;-xsrZ4{3erf^G8k!0vk!l0~tsg4DYp0bT7vKyXnh{ zEi>fDy4jo>5E^x1Nk|@CD`biBhtxhW{VO%H>pmG2h3dX|#;{es#KgNE8>U6f+{$?YpNCqUf!j#zq@TI_Tq~{y% zYw3Q!_zCd);`|mmucha)Yw9;z3Pgxbq@mn)T0qjQZ@NZ%8u`!S7xrQJlhZ4HOum+f zdW#rpT)1E|;**VvkdwNPY>My5{V%O*b9^WLnavfK57QOOKiw7wJ}2~#OWGQCWz)ex z%ZMzsl@*kflpJR%!9BgrWY2XS^RI61Pd+2WS$;P#)Y?5<jq(^z zlpKGDBQ$Y(Ur*_uxhba2DdoxZ#x#SvU#sFk_~+A?IBu8OwM;Oikdfgh#2P$3c_BpM zN7Ey0<2CwXBsFsBf1X;vLYqp`NWmZ=AH(mrKBs|SmkOT)p}IyFMk+f^OTau93PK_fuJEgn<*a4h+dh9&!^-Pgp z`94)0UXL8-uZC4Fkmh{-(TF}ANUbaw=}E$zfA^YIfq()^%9Ft=NLQe$z0O9fJZYvA zF^poE?YJ0Ef|ggQVZ8E;nQAwf7%{5Ti`U_AUeKAoL;F|Krq<;EjS zw5F`dY+NKSEvar4G}4kn8fcxtQjmv^LCGn_N_1o?i1?|~kBKp#M5kSF4-AhGtt6Et zJf!8-0(Ss~cBz+ba_!bwRwoh+ZAmI=andxolC4Ww?Uby9@}iF%kuUniv| zJ-H64wKUldX*k~$CNmx$GCuE%GCmWvQn*8k+M0F*l`^a?`>y4sZfMz7X#v8m%1~9YPvHP= zR0Vq<>DeQZxqKdfs>LjEQGQJa;744wVpr@t!>DzpR?|peyi$BnqNKZ#z6inR8w&ad zsw}ZDmdA^PHzCrJoG-@6T6YII7|w8>PIs@4JZp?=nxyqTr7CrCl9r!>Q*L!3Ng7r| zXj4uMauk7`@!kfW=TM>*r4YO(dr*4NR!KFlc(6Fo^~sPF@REQu>~ z@E|siF(mf%8TYJ$C&a&q_ooayZ1(KPRucN(Uy9ntBzeG2c-*LQR$hg%{KB7aqbuFTswN zVQG7;dgb30_mEN>fo{mw;y^vY(2RaBIT;QEU!a!mn}yL`Cj+OgF~mkKaZwGIT^g1W zl!T2bISJz=>^bJK?}UF3<9sB8E3JCA`wCxDU4MBpWXF>nwgN^HHLDFJBVmEC$I^r9 z--|Ze)PyX)TY%9~MrNTbNm7Oa{%oZQZ*2EbJ8_z}T=8jA`d37uiQRd9m#(GiNmkIL z-z7qf9Z7AaGj5WE6^)Lk6f}dl-zV%V9qSK>_L&Sj5-dr{kdRwT%t*oVk*N7l;Xa&k zRr{w|!RgtEW(@nNOweDkNd#vFK_j*{ ztb)%D99yGoNlS}xAN)W9PuOqzR*fN zDwqqIdH_}LPxE7^Kj`73Io${YVs|WfIwS4Q-7fBasEIAe+aZf8sw%ljx2lG3M=4z!(veL6Nz0Hd{bFM^!<0w+jTO)pU864uO zciU`qFczlJl`DK4X0OxYH!(-RC|;bCDsySH%4xd z+v+MA-+n4CKs=05&M4`{0V_L@c>cA+lcSzJ#S4-zV0kp-0q^aY4n7R&WJat?<=>k+G`c ze7Xu&afq46mw$ROxIOgqQgJQiKoSlhhX zr61wr+J`ohKZ3M%SaaKo9d;eR4|*T%*|?eUvy;L|q#II!$rYfg!;Q)~q$15PM= z3@K4WZq`OJj`ZqnXBa3ym0d-MpWxf+PNv2&_&*_A*kMsA3ALPl98;*Z?}D6fwOxN= zV15pJ;*^U%4oY$Kt?V`qET*R0ry5hYu%}T4Tol#7+46Qrw)M~0;DO{nTvC6xVc=EU zw{M&pDX1z00Fp6Q1lmac98>Mu$T-uQKETQhfUNM7L18FCbOgn8TABR4a@xwzTQI~EdUcR-}TQw&tJ5`!Y5L7@$2==J? z5u_!x5~2nxzXUkDDwP$bGa4iNr#s_*X!)%NA?s-%Z9qQ4%#0Dis|-7A+oWYiw#8HH zLUk;RB@L(fmbGChPr8T|=yG&XXs+4sYlQh{&CWAWop%m(?bJuN#eQsuPwv=h3R{jJ zK$L<{+OP0>H>ct$6K~feDGNSS#&_l76Y29SNn77W8&~||r24n1a}- zDJ-_8T37tTQFZVBtA#({tzRbP@MV1?@VBX(Z7+35vvll4I0Dx$V~UWB`n1s@KSv5r zwQP1Cr|1rtjq*0B^Dgf;Fyx6#G1*uAxdB9bWQzIH*>j)NokoFjxj!YwloHBZAU4Q9 z!rN&92p{r+T0E|rCfZqQ=(ws^D{*Hz_N2)w(nTP-ZL>_c^?YlbcXT1o45I-EkcMOF z_tF6Sl4~-)-kEV4U49cwa|mWSA8Gybx0TET$id_5Qe8aD%*S#A1yT|bhb20nDuy;b zbztEC0EV-!v4?rI$dDtdN@6ICx?E@_N8bTBA8*dQ%)hGScENo#jV8r6sBI0kZjo>k z{$qcZVmnR@_6r1{Ap8DawE@*}qg%W;WZ31}+6yjdinkE8C>@j=LX-jLjFa1X%HTM} zCS1BiR2eGC(x+hSLKU6ztb`o;l?t?3`irFBf&=zh3!SA!*Wuh?G|IF1fIuqJIQ}9@ z?NhDUN+p}M+2u@vmhSCGo?QuJ!hOU^aB`#;(`m>wHm{DnJ(Of=z2IBN>p*>{n1L)K-)Q2 z)3r%jFVgyDO|^2f^uwqT40sZ*&Ce1`qzx^%p)HoicilwqkWE(>g~@(F`ZKQYT{PUN z?Dc4Ax00n4DfS!kqzo-6DoTR1gPbiu5Dw(l#qj&%lM#WG$$=G5CP>Kx zQt{z~@2rzpwEm%k)M$80mu_NGG$KkZ;ciD$gE)B&tpm)Vk(8?>gOGEYkAC5Cf*HBY zn|HZH0WCb_)}2xr2{;}Sf^?vqfrVg>{*=;bx)DbEKE1R2Rrq6iNRJVldmb}MQX5c5 zh*EzMI6AOB>pA%2eChLJTXvSx+=pB!Y%QdrF6487HtaPW_xc*gVoqXgUk=`=vRqSb zC2dGlW^j{}mt1izww8e0ft9HL01!Ae$$e;|!}jK_(3_GHmK$;mbviLOPGCwFI_^@Lp0>g)KTpPBFIHv+7UYwx5O8 zz}mvnOBw-bBr9};x%5O+QZvKEdeV0nTauh>vQF z1ZK-AZ3!w`Op>i?BL^c;JY*2Ae%KdFgovxPqSVaQwBbq;mk^9N0Mt&Bk_p)=PW)#) z*Po6{gG{90`KF3xy3pgWSeo2fDp?9chC^Yr5LB%V>eYo5lC>u}0373UQQbti@7r96 zqar)ewRm78lC4R?&=jPcCzG7_BDD0mY?Ekh*QdB~YcSzg=xfmPP zL)LF5(ENyY_Fqsa@YJI44U&+Y0#xP{p`G`}TOb^nlY8O&yzkn-ES@1oU+{% z@5{Dx8DU;j60?E;I7!j~$m9y3SU+jaLs8>IOoA2TXnCOEQifnK&4SA9s}7R16)d6L@STn^zS?=L)1qfP7U12=&woryfo;sTnF&_D zE7CTIk>jN{^oA$Ss#r z8*_9mDH_sFoQ^6o;d_Gg*OfD+MYJi9w|+_*qq6ZMNpoL_3RQ%R6(`r6=DJ*ydwqQD z=Hlpq>OY7bZPZuRAYQtuaidA_-Yqe2T4Ow<`I3byLc9+ElO|zL?+#8IAz{$6 zGx$v%+A5c8Ght}7qBC+IdnHQ?2z3DQg|ecPo_Idupg;Ck0&~i4dRlR0 zoN{8I@T7990C*4A=ww7bZK_EEF6XB3K1uESq{{S!0Ri$Y{S@GSD6{P7d zke4(cMIFwcQ^@o*lgT9M1~FB#6X7SqJA4<|l>@_0h3s9Bk~I(E!mvM6>}zwrS^APJ z9|5GzWsM)o5|!=zD%=o#Gg18m5$LeoT*zo?r47)eY0`H5KzUEpbDYtWXWk%4b#~=V zf)ANrv!({}J#YZe(B_*GQg-_nCC65065>8mTAvv#xCfQTeERqH9+ev9v9(|}TXjHS zlmm~}v{iLElH!Hh*Jrwn;F)m)GBfVBi~vv;yPz|s*uKj-_mK>I4G zC?-PLoyN0ec__|JO~SF`PnUYT#^^*nwVwHCxRbbT5^d5j~S&T5$s68`x;+w>>EXT6`YJ6CZ{6XFamT`lvqFy!%~sr3=ODp84(D!r_!Hp z+%x!d_NlbQ0B(^>5yzztA+jkJ@Blh^q`d(+kJWPzUhQky{sY*FHB9g4tC+f=I& z(g!pX1ScF~q#9VG#KCPN0*q-pXYWBF+;^fHdm9x}@c>Xj?kTmWBWmx8F}*b7*v>Kc zrE&NGJchHdH5Dc(!60r$OD({$oQe_Q*bD3kBz)C;>Z;JPQ zC#P*}c%UejzyQK(UNyHk%703eiv$g{;-%w28+mKr!+@$(M6+t(Y5gdqTJiil)cY_} z0Lo~V;HT2N5nM1wprG&?iixwQjTE719-jcyKrFwdT8ts+ zKPusc+$2`^4Mkt}Kt!mYjYc(z*LF9n<<%m*mIt z@$Fs_2cZ0@dmm#_TK$wUvTLH*zv1WFtY#QHX|H^qPH|h)>K+cNTAwxL_{HgRb=gF^0CvFL$-(%Gj-k7n8 zooFLlTL>XQt$l@ScZjp>tn0Rj+L8uV@!eldv-YYdPO|N`>}0f7l2lFyCarEmpFUHv zF0Xic##;(bcJ>uHzXM~@TXbBcfcx>-QoO_~!ql#L8ncS0bz4`9(pien&L!k*RHZf$ z4tC%vK>LweTqOP`B+be=8rlon*j0Y_I@;8Qwt%-%kKtP=&^^Kc-|t*mBAMBaTufR` zj^?M-+_hZgsRweUwxj|(`O}Z{se2uSNOh!7ea1^(WUUs&4FmpmlrnMaT3VX6U6}`4 ziE|!T9%&(IKA|HzXsaJXS`(^U5hBcRkTUzL)K%(2Ml<@?BhM&ybo**(sQjnww-a); z-|kVv4X6Ui#-e`~ol8nUJ;uMT!l9#e9fDNVLz>IcBlutMOLjt=QqDZ=#-mkC~aA9jdwBolGGhtE_ zCDpo=ZLrIUXdze|4PX#918TD|@DHcb>uthq;Gp1HX(5#%D($0E0^8>vQ<||#>Dw?>7TC!+Bphc>K%&yf?IS5DF3g3d?enC@W*i~#0A*aMxb{{RviVmjQi^vhQW47L*CM8zReJVcUI^Ti7#z<-BP zR@oypyWROI{`?1Fm2?fLkR?GDGES1R&tT4TalFZ&L#8_arqJ4d^)uh zx~R|^Y-?JMuM9MuKmeR)&Pqppsiu@&kA>f7&lXEWi)xiB>guExoKn^h6{t1Fl#PKW zow>*ct^Ff?m1`F0*9%kWeo{$Fn^KT+yaAPNMiLI=a7Y{0vp3<3whLFz%D0^A3scT% zcS2M_&nh`cCv_blas_2wQ5toPF&(|IGV{xIuMbK>aIeIHoPa$BKD7ztkEnlht|{V4 z_%|Mvy0s;dAt`yaINO1=f(l%0DL;n=L?~l?ng&I8MPm zj!DVef30F232=zpEV$!MAt^{uQb-GMa6v!B0KhxsVAaiu-X+yWG`MaMu$GjCD6Pjq zI$Bmo;Lb+FvC66t#VG7)&N9<4MpGg26T%@%QdSr!)U7B4D?Z0yduJ7ud{pZTbXae> z4W^wS5TvO?Iyt}~jX5OozDeG!^6u@m8cAtL&Z6=ZwE|a!A1Fe%DFZm*V;$=j_``X9 z>v4vudZjwBoRh0Y2mpD3&@gg9!OD(!uBc*iXy$WBt6aLK<8ik+JX%Dgkn)2FN}N{X z@TXu$aezGr{pwRJQK#7=bv3mpisFz|!)he2Ac3Hc5<$+LvPe8uk?`lHZZFME@!8C? z0%8*!r%DuD3BfB$9KuN786@+vPT18O_&;cp+V34M#CIgTrFaW5QKuyS5wK2Fa7YQj zPQx1terptykAIp69&xF3Xa4{QCd-(n;9TR)eYGo6j94gJPC7sscZUW-0R-$yLB^!- za(>ThX~<05`lU_Dwy9nRooa}n6|2J~$FSH0t1Yb~2O|T^RfakVeKFLI#=1mPjwL~n zsV^ZRE*ynN@ZC;)`{_D&$E{k!QqggZ9&CA?M;$12D?>5X!;xD;ib{}QZwfk-g>R?~ zCm?~Il+>Na$;mY1*b;8HMj^D^?yHSxpW--tD_y`%Y^6&it3Y3f!8`1bq=G=kDwiP_ zmT!xc$&q+*DpJ~H#&IfbymFAFw1N-Oe5+&4Snto@Al_EbNm;`!f~S<`A%FsaDJXLa zI7s$E=Q^`k-NsK>i^8T!4o?LqGGt0p!-HN%>On#j4`GaYZ%+%0**P@VWs>REKM+?k zQfGAYa+`FPn_Dpw!hjhFNpA}Z#`xnQBXLs}uAQ|_3hF!0PuykC3?w@L0L-)_FrWYl z($lMO9D(If81(H>cHW`tGRmDrhSwqGf`x*^DocYH^D7}rBXTz%oic}Jl7&4PxQ?Yj z>TWKT9rnt~$^QUe^-1T7jbHR79B%Ams~@|q#m9?lgoYL5*A1mK>wiO?{^_dinkDKi z#IyTICA8p_$$6}e_8~yz9sP}5uDxq~wnuFUeqm)u^7v?H83Sx;2Pf(JRVq#Pbm$KP=`)xXKCswM=tTjf&#aqeHZHt1vDKRneB*lF9={Qczi2i35VPjSh4C zX;yuE)yJZxE2u6ikS%4l+F5M4&`3+D?d_jYpUbv0V(3e??&IR6wL@rtqy@M&0qL*+ zkG9pl^evL%)5=6qZfR=gQe5!$D0c@sPJYIw+MB42$BMRW_U4&|YiioTaytAXmXh*z zPy=Sk_4M`_^r|$weEXBdrN{xmbNiP_C+Ym(l}X(0@)FV+S|me#pa>w8IN2M56}Tgg z#*y+3+!|{09fXG5SMS?w1o&@bay0FwUx!8E}3jI8P)mbNSQgXaV#cskn&FHytYgDIkwRGAeS>cf4AW;M?w!S!D9hiK(OY zZCOG5sx`b0-KqA9L86r!iz62Z6A1mKWsj)x^ds`?`sv-X`3jR}+i#Eow_TenJTomm z1D`;wlq>%LFsi(Fa6lA$Pk=cUpWGE)64I^Gp({+d&N6#zP67HJ4KSnxjfHY0=42%J zQKh(s)ysLyXX=xY`cl_(i6KBqj+T^v;xL>Kx;Xr?K}81Nkaeg5x93bPzcObEuIqEudf& zRP-Bg)Coz+^`e-!okPpP%}m3IjBQVg(eI&@ens1%07sNnmJB{Np&-Qoi~+S+Au)gk zE7DE!(6jA{6 zu9SiXV)n&0sGij)pc?UP8r)1nsu}JnwL|u)l{7sm3S@T_IE02VHsX?E)T){SkXjy; zD8yln?MeA>K~iP0TrGjxw-gdA#?IKJ7{fWmNv!~OriuZf#K0df>t6X$&eV5?4JacX zmAIIX`EOBO5rO1;RF8%aYEWy`P~s9eMQ0JzuN<2(Pn9-6p zjHMq{uW!f?^QIqd#U!BtEhPT{3u)7zspS53GvEW-nst$+?LvT&Q<((`Nx}#DRa*5F zoTPE+D*13=kZ>xcc|kxs5^8c(hmU4?H7IG)l@W~8Wx?p>IZ8%-sE(vlh;)&jY9*Fd z>t|Ny-nrX4BApSg$5f|sk%WWIKCc2Y0||2~R@D6)n4 zn{^FaBQ}6G30#8OLO|dqxt=$d!+)Q#bv_-n`TqbVhTdO_?}m#hLr!29Buil`Qgg-% zvPty)D(P*TEFkHC^9a-eP|{ieaq6%=^H~&oC4$u@Nx1w*=*W%}rAmSGS4V;Gg)qrv zeV#vBA_qwH71iAq7rO7NrWIhDH%zmP`#AHi1b@t=ZA%|Yqb;re>;C|wODmH90H^-| z!{~cd6x`N=A}PlZHDT`{2R!OXKc_vbw`dk8QTrnE1xZV*IMWomganL-6PC`BJCdXN z;;KIlBFO2<$hIDlMXAC8D!p|~SEealvePJUNk05miFUHexR+mM?8_1sgSEUV#W$15 zSyDjHAB#29?bLL?Ke5fr>hJUWn6Fu}(y%q!;o5pp z>ohs>l8I1KUTh@*KHEu73XogxosObN{(P*DDdFq{#n z;4CRboj4@p9QLZ>dIzum4D@RjE{}%Q+FO#smYEU+e+i`@Hrfz00lrjLcmM%hv&5fr z{{RPMGA@!dPO`MmzCvOQRwlCupffP#D=U*ThCVGH2_xcETP$QS z+v;{g76SAf4>#h-J7gWW=Lfx8Z2tfntonvy%`0S+wO(;*Glf@%^UmA2hZa@LP&fmQrw3>Q|5nQndHA6+Cs_PJcrg)^mZC*aLqS8#l`TVAR&o4XdR1|#EPR}5{`>xfb3PN| zrFHf+%Bw{4b~&6v?@D`S#K%Mq_{DRV2>&Cf)$Kyn%K-~R%mS{qV!1d z$XXQSu(0ZXn@C8)a!$%4-0fE>)@Ljh0|ws>w%m^yC}<(Xl&eS@TS{_~gN=bYM{i2Z zWx4Txs5)ljDT_KRQkqJQxlF7$*;w-a@TFJ;@8X86jt%x)dFJm#Nlc<04yoP-pd0iIRH z-1e)lO7wrm$=xFN`(Y*v*A`eUtcc@69M9s=Fx#a=azP{>IjIhe_%rbXqk3XuWy`l- zuFs*xGcr;nC(KS#j3v?vaoD77^s2@Wd|&b;vPFKPmd8t6>}@7QzM8d8MAvi&u52R& z9mxtwSBMp{Bk8qQEt#Vpk20hv}=_hY}RrcwIyCt~hwY66tK`BVaT1 z&0aWVB~AYTCnT~*mXU-x)9_*?FD5D;PMm7g0*C%#zvoeM3?&sXWiZOY)1hceNIw4n z_N$LkUphi7%5SZH1$DPa$j;78e&X9JAK`fg#He%s0101uiGgeQTYVwBY+K}nugn^W z*AP~;dj_Xl*k9QysY}u|{{ZO}j!j)Z?fyzc`#rqK@TEjS*M)+N9bWlQKT;`8^ixvQ zhStN#3Lv!Z6M{X+l${(p5v&$KDRPJ{5 zOe<`aDJb7Ufdpf?P^T}StHIdDiP%~f}=q!Z`= z04?YTud2><3MkfFm2pmg6(LtNox-@PQtxE%u9;QrjNy65tq@sua^gAFz@(<)Kbolu zz8svdk9z36wdH?M_YrD#zZlX0N9phcmye8rM*qX;Ykc&1wyrA}$CR z^8rsHdwl92oo9&ZEO4x5iEpS1j&*I#0~g<(MsgMU*FJD4ZmOheBk?TOEaz;0E8)E z(lfs_F@N9nr1hmK69nDj^g{^5&B9>hSCP#e)twVA_ z^r+YfQ6w)Uind(s?dnk_KSF9U6~0ri6vMX4l0n=@j<-KOAlD4Lqp5UP?yyjzJVE6& zixfL}m=Z!%2?uJn%jo{Byf~N;x^7{Rv8fFm2w5JwRG>f}ybkn^p6P21iEl5fddk-s z_Eb`+R>gkMm!9CMpjmIzb*TRUhMhu9bO$7~3)NAfTcXCB9j_fgf)EY?J+&Ug)`(9| z^;x{Jb6)#_t5wmhS_ z(!B9i3unZxs_K|avpac{37|6jP?DHTY4q^e2ub$HHC&Nsb*qz-qOc0c(%M^20#Z8+ zbLvkd0M=YvbNwK7+~33U-yjkH%$Dxh{qo>rZ#rq zmlsuG)E^J#NL|8bPha|qKshryo2V73N7uuSkfHwo-z)1?`|O5oP|BpZmA>yK!^hgG zf%jy(x7qTr*vgzfRF7JgPgN-@ul@f3XQI%<71#Lw4QZaW{hWnA)15if7Kb;|VA+~= z<$Yh3<2(5gQF8j%uCDEPy;0V;sWG2)Hq=om=TfqEOKr3XN|bmBN-6$<_)FJaM-X1F zO{&La@Vs=&nATl+pXX(j6y*p102S4=&;I}uwP>9?d+956(XN{KX58D=+reJ9tq-6{ zTVQ_iZ{3uTh|Gcb#5U4^HdK`j=_8Lk)ydRZu?jXDN00#Sf6bInmlSbRX}`FA$u*qEIH6QO8z z^%OX;2P*R@=LBIR+dI{zpNKbW;|5KxA`CR7DQYKDmPx`gdDKoeIX{&uRm%S4H{-c( zW-DzdHl?csrPY2V41W?vLQXOcrDtr{25+hTCtfGV_3W>)@%ChhZal7xk@GF6c#O8o zZ@SUkrLqG50K7hhNyc|p#~LS1d_=h3bjM?+BtStxFUXejX+wD%r6D=W^RXj1<29b! zZoU}Mo||+b4mgB|7lo->K{!G{8|+R_*-lTan;m9iup3R0Ek{WhaY`g86M}i)f(LJE zhAgs-xi(WQAt>=kE8_nEK=kj#Yg*#mn+~lAP-VomkkRne0z*_aQtTOx3?WF=3X*mhDJMzW8Q_}z0_#4!xZD|i7RQ~AQqt6)6|dcF zG86`ND0zGHfym8#XIB<2b_Qj+yb-d$}{O+-=)dQS1yk; zG+s6IvbX(`eifgI9dxd<>UFgm&0q-d+J7NWy1heK2ZW6~k`I4M{SCF4mbih7a|lrl z0+$F5cz_NFPvSm=bL*P;$KkvbTx`rH-0MkoJ{!fxmK#uT;YAy8Hd0Q<8qUWX?cF;! zsI4tVh^cK#X$ZhBtcO;E=15QnN{HNNVhGrfK&_f`Qqt%Uik90n`JH*zD@>+Yu{K0R z>OvIxNRZw#trA;Qj|PG7d9V z74`M?FQOz!DU{YTC4MPRj187Ex`-TujE?(M9nxP?+$W{Tkz+d*u!K7u}B<0Y=AKfDD9SDhz2Xz>Wncg6%fNK#=)OzG%!$%eGDR~!Yp z$SD9UAzoLLk1CQ>l2fNi%5Z1yvwEJ@W=B!F+zY*`HAw(QKz&!clxCP~DG$rTk%rq9Je1|5{{T21_^dnPO`_G*)0>6Y z@uc*1&PEji=C(w;HOVA?FBPCp{{YUA;2Q;CR+^uUZOG~2arvjw9IxdQaPgFnlzfba z-cSlHr9f&)Qe13+a+Ls}D@(sYZFd_pDmJ%}HQphyZA7SUp+iz+PO_4wooPuR{L}?- zb$~O0-wbicyZ)yOl%V(hiWD19#7~9s*^)2UyHp3xn3r2M-PK!U{{V-?Wo43mlmdrw z=_H=F>IpiL)pqL-Q#O!F&DO@kq!jvxqlTS7T}0ITOf%CBvwrEGqmR+7-am5O#%3z% z(pQiEAIuc3OdnPzHoZO8kxbnMnlth3zYuylUqnxklH@C`m9aL@IlufwG8S@=@kMoM zANWIxe`ax7Zod6~e(3?_Pqq2^@W$Q zny}jKeI3*-w-VXZwlyWy7bEMQ?d!&)?RvTYZ-uiAMQSDttJ{d}IK3+oGN&282 zKc*=?2P*w+Kx!S@WM^61EWV_NTOF2>(6K)J%trnbT#K>q-t%9XxbdNtL$#CzkmNoha(z2zUeC!7gPf)n*|w2$#t zdDT{H-%h;JuNK!MGc0P0Bg*7?DLvYG1h(h>Dn9tDCj5(0@L2_es%`8ZqO`Eh`qvzi z?At@aTM16!rR1mgtmI@L6$r*^8dSzxaV|u2#EkkwtS>p#gTOfMI6mU5ur70LliIr5 zCrN`nkM88fc}%>6?6eS}^vztNb*vtz)eEkyNS`4a9(|1%~EbL&s`%M;P!p{O!a!Q)^SSx7u zUUsN2no!|il`B+bvT#ko$xyB;j7YBN;3Vp%9J?z{>-MPEUO3qO=vNk_zH>z&GC|9#xk}9h^~q{K zi|f4v${!uYCXpZCnQkLdgiqP$wgiWBxblTOVzrz zs6}cgpU%9T_rdR>Wp zSucs3)gwh_>!~`mY~LqKvdUaVtBKr?Z`#S(@`r8<-faVZ{~)2N`FX|O5u;C7^O#dHHqv|;qDaZe@9 z^O_JPM}8^54;#~1C^@%>8K=_qMx|s^sB_zjHfki2@4YL;CWAtWr^jl*p~AZ1a+8g* zTq!4N#-odKYGrPArxcS-mi=o4ZAzw=s(iLbB9L0h$E6DhHotRS1e47;rU>myODEES zhW9+zfIQI&XFCvSVV`Oy8A#uXbpZ2AOFQvS2CzS^Ycb@VyVA3tt#d{zpd4{qW;zL_ zjGk#b*FmFTqsv2OB&jMV&=5abuU=H!Xi-Es-T*4&i| z4ui*ut0|6?>ZFyBe5n2*XuVuhXi|ldDiU}%2UCYr{{YEP4S&tMo`pPCdojN{+*l<< zl@gvh7>pCFuH!K*#G<%cCfY(KcqXa~0U1`+Js3<$3 zu@)nP`7##ML?7bBhrK>sTY@j27-inAc$JG?p$!(%6)qt^xgWx^grNZ+{5H~thw_t^ z7`csf&7M0KUbcj{Uy#xyTl#eiQ!mJALnwAqhnh5X3=o~3bRQ@}4~bKZmN`~6Z{_sA z}Z5=r+_L_Pua+x;0*o_9-j*aw2j>>$Dw4KQV005<^lc`lJ)px6ZO4?oM&V#zQIpD(J z+^?)3Bf)s_*oLmnIG_WM4c$r*+Hlf;c%JH&&9`1#?fsX+8-tBOE&l*ybErCi!*Q;G z=wz_-?yKbFIc+9GdYmB&i4l0z`j&L1=_kzku-Y`R0#+AMO^?%#tD=~cnI<}r?eXZu zI3-I#%3-#o=Tj`Jk~VeiL}e&$o3b?T!`l>opi=a8A5cbL?2rZ(ZnrBaa&P!udkSff z zQmwWKC`+O{gz~VLRQts4xAm27lTnwBw0<8^T&1b7({^aGZxWD3&CHUprkE$yDt;?i z^p2s$bWSm@2n(>|1;;VC zX*`^NF)Hn_$8(B{!vxzZvP!7AOK+%xA5$!)3M)!l2~Gy*d;!V8$DqinHP52&*Tugg z;KG#J-au`=T2GYWI6bm;9lcMjY&LuR8SaS1Ne{Asr8Y>)bH-0@DETbIZ8}y*mr8)n zH`@dXXemMcB;_RgG2Wf{e{iuRrHsb#3oa$lai9UGQTlm)!mLo?b!D;SCSREea0R7) zV3UEt?l#BTwEKHdq#;DOg(b$yR5{rH0Nj4H6L)clK_T`2?I=h&86yCIe^2XFe44hz z^&(MJ1Hn_ML-`>f%${lq8HVkHL+3}Rxpr|t`4J>1d-T< zf;}ov8g9xd$tq@v8trgQhNre%kcr`K?sqHD;VqHahZ%W3#RnT=phv~K_fExks_d;s z@gqq@S&XSayOzVKTAL1_`BaqzxP<~RNcF}O(%oRhxV=juw1WMHRDP*t19DqpOKwX@ z9)*=SulkmI)iz8gPhIs#!b}Z!bWOI}mUuJSTAzlK{{TALEf4rw30z3JGK-bYbaBViU-24j5g~H%@$kz z=3T1Eba|IoagCDwl5=2)*EWZBGE}6vj1cP!d9n|e8^L)!MLVeNeL>Rj(bTM^4E3i@ zLwPO5y|9cHZg5;aR5$~PafiqEWz?n3QQy|zbQE$cT4y|zpz8Ey&D{PYbeB?GZ7D#q zbz`AFGYq!V*3PG4Nz{<*Bq}q`Ek0)5NG-ew(o*3CcF%{sSrY3V);dm1PMLmdnVHhJ zQ}Z1;b9C+dw7yTtyiUM4p}xp{2MDOSNVIgU3wJ?vyytHn4LP?Y^q*3*6<;-MY`8;# zEgv?7ItpaKQ3`3$r8H8kc!2enOy2DZMZ#f>?8h5IoS6~8>|4B7K@87fz%c3@(xfk7 zb*Ib{wJ22FvPZAa&*;XezW)F}K0F$Ym0dj5;ZwRw=_ld$RWh|O^a*noh>kc8l{Va^ z`I_7j;K~IZyhB7@E)W%TcEj-xrlj=s=VdN+raDO|V$q6ysC-|FbM|Ud2@c#g*Ztw8=tFU@d7x)Tg*?rwEddU;@2S-h{Z!>F%Hh z_JT#Ceo92>j1b9Jqw&-DcjZlS-3F@|sJ}WZ@ov-PRWUBxUM@7EB)pYl853*{{ZI4a4Dxgr8)E>v>%ApKB2O?8c)2(t`(Guwa?;0!44fKWJ6gW98xDc`3J?aQ#aCfOo z4^dsZb@$%3Jl}^c0jaShHXe&LvBPh&e6A?S2pi1?^!=e=sYTi=5=F~iJ0jVne*LC0`L;EaL}K_hw%PGYB^bwh4jnCb4QjP1(O!gRz^ z2oo)!{{Z^4irAw&mKJ<7{5~4I)g67aJ!lL`YFkt7vvn#Q_5lsP6Zn!+e9gF=jcqH; zl1Ctg5ZAzLjasrXs)s4xf6koMk&o#~JfG5mlJYiDk?ZuP98WmS6h_P*^Ms$N#IIQG? zp||t3=tP9Z#PP6o5upKXs(Poa-$WDr!e zPik>2o@l=SdQ!tf(ue@VL{|z;7+dzFmU4KkU_AuWt+uT3PlkP|*4qPYRtSV7fGN}k zZLy~ijP|FJ-?d;3fNQ!*{ORfWX^fwQ!`9#TVdvQ;3Y6J|mga*bY>EJ%_zy07OOw z1=miN!PA^>qv(MwYkBm|n*{t6AzRa&oMv^Q_CLJQ%A{Bm-V|s9QTQy#PmG z_=wMQ&TB-3B0_}Qg?QU0ve9xXPz||&E0_MImm1PoY2_6WjlasLifKA2IN7L5y+>c* zl9HceOkzFhl}K_zV?k-eu;i7YP6Fz`3oW5OXP#H^w2jXvNZzsTm3HkdEl<8W`e4d& z#^A+e-raDU8 ztA)-|l!ha+q^D%0sY)tQ2|^H%RHdkdp(!d+024*G@lwOmF`T#bwe6YL7m`|^eq^X2 zNl?O=gOC)Kak8HvE#1OG8#rGy@#LP~qQ5_>cuFwR+bk~hqkRz|IP0+PeNNi)&{|OZ z8zRT?X*ooxVDX(yvU@tQnvv{PnzIjELH&2wPQE~fB*!Na1xM7AOdQ~bL$S8wMCKAT`_WE zdy`bxKve;%bN>D&j!01jn z>CdTKxiVUnBI2@tn59<~{{YfGt1Rjp%-MZAHcYdb{YiaxOqKQ_hJ%G8*gG%(0KguV zZD#1?Z-&z^Q{Er6dfxo}*#|f}Lzz(eYD|&za8*wE3^Q!$_?YvC=E?&c_>z>^KXDvX z@<}DHi~hg#2q$~}zo-3`Lj0T1aMQqGfcY^l`{O-N|MM=Cj^vZ^v-D)zOH)ZB`>u4 zv^^b4KjAtv`2{CGb5)X}rdASo0B7G6G$okqElU|o4Y&X!v2A1g=&X;SeYMy%%50WH zvl=JHaDvx;EaZ5pKjB)El>Ie!G*iAbYav4p5*?DcNm74_8ZeSR>N|Soqg+eUd?J9S z9B_DHBa(On{Hj@?(TZX=;#@=|h=F5;r&uLJ{E zMRk`X$zeq+NK>t-e+tvONc(|T&+Q1NM^&cs!vQH>!s8v#ld(*X0^*O>;Q0Rlrn@lC z@k{K^dBrS??u-xPA56q&;!VcyVUDQQUaV?ARN4?YWr^tV%On2)l<|F^V$Eqk4En17 z0MtD(3M@Gej=T7A4as&%2?1_g>`et1;S`;naw-(!(1V{6DxYfvG>};88#{eN@UGQy zV+((!F0w64Kg1zT)h;nU$&iW39-7CkQfxPiw@=^g{Xx^Rmv6k;ab;Vk*&GE?qD&zxlWTNy;icfytgG<` zU1@1Lka25|($aJ3T3=TfE`_B%ks&7-OZ#;>-q(N6 z{XgV_jcN7#fAB@BD?J(1P~EyT-1`;cBG9R@fmCayvBDDK+s@%>AS*(ETW++FKqGm% zT{#}lCPk&^lBc4qwj4>>)bf?2{{SkIQladjLG)v-^x9?g&F!fYY_g;mortu6+D265bpSqb zl#R(q4xoNC5$85X=iGXG=Ao_(OO2{(f~H$!l`1rnI9HVFZPENLpoOQ@jHJ?flc$@4 z!g|-DC(T{9B!7F9nQk!`TLx?2{LcvOs zpm0*4cLWeY2e>qUV5jmP)R$Rx53=ixI?~ePZZ@)&wzQn3X;2`gK!8C9Al0kIXRnl1 zp|(1$M5y>_N*cW$_(;NAm)`>jd$3EJE|cRAT5$_W7zgQtjD7o1uxu$xBRu47#twd+ zsM|g9cE1`Vh^d`yno=i0TJ#5W{udCf{5O-6p~1M|g(RsYsZXA%jP1=wRq8TJvqslfD{{Y2E{`7wk{ovoks0F&5@+fpQl9!txwxOBOC4B9wX;;ciD9}ky;%U_9_irO3 zQaIcxB&35;dD!NTvtQF`ThXP)Gc9wpqI5laONmZ&xP>3U@-j*n?ox7{VlP_17HU=+H7srU)TZbRo42$;N&PCx6*xf1A4-Y0 zwv{Cy?TVXWDFEQ)W18{(MAe+LXW0Tbd~Zuj0CG5^!&%tZd=F~&k=6*7y{W`Cj?_w7 z7^S7JYGVxsfv+4C%GF}(Oq8aY{hWxur>1!U0f3E z*Bc-llg)DGha;mCqN5(=B+3!n)JgB2YeTejqYv7J`>5D)*eN7OhL;K#J=F4mp}HFf z@whi1g+61H#q>u@M!2};=I9oSY*i#YTR3s|2+Cu$f1N-QRl*aMAK?nsQ3h4Xg-x1< z+rkplOtPFb!zma;NC$+ZeaR%8l1)iLEtPU~VZRWhUEdPp!&+UDHI~?&EtdQyiwMG- zgB830)JDi2TxG>16XgIL#uvlRqU#%jcF3%$a$YJ8x2zxx0;MTSSCt4z1xi|ul%x_! z0=}?UM@G6VG~9JMwJ0G811eIEzz=g#y-U*YtxzAZS@i0W<+)32{{ZheAchiu@DiX2 z{{X~L3O-bzRPyAPZ=k_0muJeIS7S#?+GTWSM9EXpC!`|%dI>3V^MwTn?Sq9i0SW+y z&J_BR9Cu*bZPH@&mr&j&vj*P+Oqmj0LlN0w1x`HT)B;e1j#Nni4|@7i>C#N)UMndf zKg4(*^ogstd@_op_-al5;K>ds3@X_X#Qp*NvK>M59$Rf%c2d=+pZ(0TKe8?DDC^|N z@0NMp180-a-6btAUT#eo-A{aBVQDBQ&1;6>{{ZaiAP3$b->hgKxj^wM?2GSDBUTjS zfOFV;)`Ksp>?&V|P==D6X-Yr<3YY0+^!w;#9r7Iq!@HPVgU;$SD0EC@CZD#8ysOJST%rvS44xr&_dFCBlhZ1`! zm@P|KJ(ji)aY(YfnHD&zMI2Nh=HALr#Y8o3wOD=_^rP!~%V}k|Q1DI+meb+}9#{CJ zE!mTZJSJ>({{X@mOa`V}x(4ZX!nLyz`y*=3;mMA+w@4npD;@Ob{{V?oGh`zF0D6^k zkr}q0nChM_pkXa%@U}e?goL3Z(ITdmXL0LqOZ37ckN*HoTo*OpMJ30R5wFP&JbsQ; z=j!E`E0S8OU&0E)p#-00cj85+WxgLtbeM4Xwo6J(fH*HZq`CF;rO*7;A9ronquAuE z;Zdy7nm_X>DMbGO<}M$oH8=4h=O*XXChvlzy&5}!j@r@U+uZ~a;ky*} z!jR!{ztPe^I!u#pa!ar9Xr!YXD~-17a^q`8rkntS=tusI7>p&RkgblBkD*bc8e}ab zDInx`G}_uvh!UizB_l}o-2VWrBx2WNxi16O<)>Ln&I#D>&%f5BTy0GyL0l0~%4NvT z4B({h4r);njEI^Ra#As$YJMVEhNUGbQ9LNnlcgaspKUz3r67Isf29v4YH1kH98jus zI8;21S|o7PMZN~fT0zJ1sU#ohKPsw0d2d{`5>Dc- zuvUlSNRCHzAJ_i$i?O7rZKjhVIcSYbI7k@hCm-ilUy3a%?Pa;m)TKC?mU*ew=;V2^ z$^7FsKEk6cF#Jo@1{L^zX{ETX-Wn7ZvVDS5qxPfjZ+z{JLvEip>-uff>x{A>vJanD}=-=;=cJ(V^ZIAwO9qPK>iHU_0 zqkxr=6c4B)Bl8tKV13@1>Igkk5g3tamdxUj{6&?A2j&t--lRI_)f=aJs`X%k<{N`G z)Wuh2$5GbF`?jPXy%@f5UvJOyD_ms${{WxhnD|$AwpuPuboWt95v+H&qb5tr84xBk zyu|?aKncH{kNz5zq%Rz@>!Yx%0gom#H{OAq%H!Q^@+D7)82%)_$i&Fw8=Xxy9Zoq^ ze2>LgJNGN%XEZ_zu<8=Dox*`o1QFh~#%@FO9hBSL7crsy9qQ{>&BhR-juw6zpoz#VlQ++dVdL7#7WR}B=f+oDvt)$?M4k;*c9*Y2Mcq3!6 z+G6P$aMokG=ykQEEvG>8xY^-D`VrWI4;1PsOT8Jr_R`YA++{^Rsm^OYb<2-WT|a4C zJRs(Uvg=PH#AZt>#>wMtMVM|E7O3c*t^ZLJ`KhY+DhGbMi&3vi{ng!Q^n zqBNDLC;g!b9#&l^Z=L{Q4jg#k1grPDRI-@3P3dcklOx1tTzM?2Ecg;L^7Epog0>yT z;1U9oq2@x;m8S(FtG!#(Q7y<~`P1?qnbWr=sSw_K9z1ta0bEHMc1!K#9YX~3I1DN2X#;!o zR@0R}kU5yz23!Mokdol@rOi|$m0+iEcLW`=^{rzi@=CFFdL!*ii2B)mm4_R32MR)m z1b^%FqwWSZ(i@lCTP?6!Df#$l*f6CK1FepH6&qeo96#nJFHa!dwIKBi#w8fYP?ujCZFJ!nUEv z4OsNHW!vSw0&89ri0du5jOPj-r7YumY_@s92iB#&LmTO&E*aw$y})oHf;Z!OQC2wL zYBa-9Ja0`o6+;_V zlCH;W*Upmc@_LV`Q*VgtZR*zR&%2)rGbtLJd3gR5Cx23Kk-CLzpAqk^w9BPVfB+ zaM^GTcu~sTa~pWB5EP@^%8}(jtIFC}hw{it8)Bk*bw4Qqu!Iqj%~`E+S&-tMlA;)B zc&{jO5IYZYNJ#fS)i~@jbTpWWvfF-W9Wo^(hR^;RMfIyIYCXkK?^kH3Qkx|? z;8MOM{6ddwOU}OP5~U504gx?JN=Kjo0rVAw#rTE()b#>jMU+})C~`!(Sk#D-n;M;X zj`~x;{v{M7ZlO@kA93+Tas4Zljdg@Zg2CzuNNr^cEy7{Jl)U)HR-xUt@(CZrqLlHf zs69_-wq71wi;SS_?x{{V%Kqs6AEj8uxpi;^4%3EGEzyi5*8JmQ$T6 zKC1pD9_mt!K({NYXQx}LaXV~sO>pJJmXkWM&GyNAufwKpGh^DU%!yOxzKGG8O0nWP zwFIPRu*n%chDqY0z9L#*nu~W$^mMIu9-KnsMR@}sIQrBTDsdjKIFYB(me^15)Rl$% zy@Jtp>Zvce^O7@SwC+Jd-EUOK$GRh^G!JD<2b9q9Z!27O(w4$PHrRHhZdjv9_A*N( zwq+efD)G{B)bo$Kge6NPWbiU;QaxdC>04#jq$MbP&qZ{CziORWKew*ZA|^^ctgy62 z{{Z}Fed=n>Vzf)2{?uO~wJzP$ZGPB&mq00f#@3Mq-;uoi=Xuv{y8l&}8)9u@bmioP9Nu2&C< zy)k5vhY-};+$it*cqQ}~GwtDp*3aNt4n> zKU80p{f!*%T4ssVV-v#>@Z)HQ2{`8fbT@S(L6>}Jx?OysRy^O~NEE9KvK=OIBO?Wj~N z6`>E66R-xIdU_Cz^yOp@YFr;>6uwK3&W53>5f!Rr^r|9A{{V)C;V1s3k`wHl{{S}9 zT|uMfy6I8Zskp4eipd;!apthXasL2HDhH#SYDZ979Fn2(@n8Ap$vXz4q`t4wtbOWI z+NHT*zP})9T!vvZ;V1Z)BshM%v}&+!zpwHC0A%Eq>+Ssi0MS!L#5k~`09tSqlfsAn z#acSLDZfVP>*H!6`L-y@S?;GIsc{$g#B^utREwcZIZ5#GjPLZTY>DYtKv?IpwFO(Q z*C;^t0mMg;k@>0L%++uv(?qiArY6nPzdW|w1t^fxhi}D=#a#M9q0+Z;GF&qv8|^s` ze01&e5gZtGvHmTB(t9lkJ*t47rA#-PxiGR4tZPrFtzRwuOA6%`AH6_QfYPjuzO~Vd z9J?VoBUguIMXKE0sJh&w!Jg;J%0h6NF{Cov%{rfDgcW14Q5}s*$GE_^q4*b+EjHPx z2#~a=C6x^5d=BLVlkSqV_oz;kxIb{{mtzoC_fE`?u)OY=Zx>P!eL@~yN1%9)!lz&P zYrIs%aUeM68wZS&wm(|*VyUK%8NF!+{Ao^AB$f;js70@`ti;W?Y9Bkl)at&23j3Hp10DbEFHaWXT@+(xxh&p{RV^ zAo)hkyQv?AP3C$GBwqS){+g1c36`Ms!qByxYHa!CM9KbOJaiA|L!fTpsH+>#S*MAO zM@)uXZBgY$Q6UneFCHT*^a=nFe}zdxhUpYN%Ke$RU+#)`$nW?~EfB^Jmado1mu}TD zD2V#xz?RZwL6EfIyuyh9tI=s8NLcK&D0)yIvq=6N9MZYgm{f0M3Sr~`Qho7Gw*(wv zJ5=f|#EfXoKHao}HWa?-coDE8qBH4nXewxehY~-DwQQmJZau)KU9@-(siH!HPRavi zr;x9>!TZ$P7Tz+R^m^dz+q-RPPg@OTvU+k$yWp!2%*($+XW_@-az)sJGS)=vHDX&+@h@D zlk}^jmVHM$4)mm1JLx-nQ+vSRR2sIBc%u<%ByI&-k!DJF(@(W)#>FO;<~IVMZA5Yj z`%v!NKp%&xoXnltwG-N}&dl zq?1ns`ikg4CV;^D52bfUwGrW0ygJ4P(d}Lc_M!`Tr1)pjv4+A(?Ow$>;h&{*PAC{o z_3lM?9+kogqG1XmkV(eWvXVV(pIXKSbw`RBWaoM{#=^K9(+XyRfy#w*C$&4^9w|v- zUX_Azutzli0NRq4jXajm6ijdvN*VO7=sl@nCV&nBj(DyfaZYK-;*q3^CJ?QuvZGF@ zW|>gM24N~Y^GPZmw3UK4rj(;b%|h0_BOm;4oe7vvL$8a}Bj8k9egwbTblEpu*0Wak$l z_ncF%y1mrniaAn`xg?sebw%ywUWwC+P!6)8HhNDYOQ{7dpLW^MoOV*(s|QR>L~tEb zhyYhUT`DV&do}ZPVm-SV!4((Q9eB@7^!ss?y$?FrmiW%&Q2o+A$U3p==e1W`yvLt1 zJZUd!jg>heWB6D?O5HzXC0~42JJ!8baO*3a$IyinA>}xvjuWWlC+G$M`cp&nno&uW z4z=oA_gdXuN(+f+RyOZfUW)j5^|z;5g0z&&>JJKR(mxU*Az6&i{c8pF4@9x)RzHWo z4N-FWf15@;bpW%D6Z}U$$;qvL4W=sPyD?26CBu}b0|`I@C`kGwBz>!&Ea^TaA#Lo+ z&%>z8DG6-uF`RdyuKgz(*989Lt3b|2e`?h(R|{>mId#U;bL&;dx{}8(^9>KW+SrU& zIq*jZb-G?A6{{fbjFVsz^$A-g1B0eVKpQ-{Tpudj%$A4Zeu zS>s5!Ta>mQWT^#v5G!6sR9Ym^?Gz< zpQcikexj+}D!^*r)y--krp(ic3hj%OERWW@px*?)bDio;(t2&hxoow}=dBWL)3WQl|Kup z+D}VMa67b{#8{(#%}0I3ey=xQJ;P@5w^Z|KR;M{2d)Ao^sZsi0r)}>8;!K%yxw*ga zm9rVM{{Z@rRU?ibeg%}}-~+5WklXId3qQIfV|tmibwZ|mGrO|VQn1HTTb31_(q4r% zIQ#e`@~ZD$^!t;Nmy^DvCvodWbbXQbdJn53s2?ob1cB^Q~aC@4a*SHsSrhi_6@pC~mJ=^+-0YFe-abvC_f&#P*6moxqfnzD5x(XMiA_jH!0 z+7{rFk}#w^(u#QQJWLAta^dkYn~N+*Z3{x;rD4PlRHdA)M0ydVZP_)WJ=D`Wj@4xN zg$?yPCioF-x&S=Lp8+XLs!;y`<21NWd#OwGHPWE?{>MAqjn}3xR|`DGpLbN2+c;VW z4z!_#p#K2Ol1H|4+O4lC5hN}vd3CHMY&w79G@+>q`=pHf=hCyznwqC#vb7+jl#NOB zKGmQ_@F!Z3px`cBy-xlD)&BtGte>-<)uYhOQ#{JvL&%WAi)(E`Xi()U2V+j%Ev&i| zK09Hy-zQUP;kO6kNcP_&*pEsfhLBP;kaO)u#$hTeisl1yFp^IRSo|vc6NBtcYP}n+ zqKiJxmkA{`T)A}mPDw^{T4alI(xoM6AwZBuI3tl!vTeyyPE%ajB3cZ`E*4D+Id3D} zP=edO$<1i@Yf1r82&($v43JY^blZzMBCP=1VT+Ze?w@fQ-j1ctw{5Tmvqg(v6*XFgH=jH67dqS`nE^rw+;18qO0Pa@xdFgyBF>9=piP0V!y2@t&#L^B`5 z40V&7YrHpO=015o* z#^UD#PTrWJvWnnCCJ5t6rHCW}<%kwopbw zc&k%Z%y60|HuRCe{HRzZ0)XwC93uzmL$1}6jXwVXTE0lOq#ii^YB}3}GBNoINj+%i zo-&+mz#a(tiYb;fU=dRCAQYWJG{}sk`U>Tgju}IXKFdxiM-(TDplhMiUtwsc5RB52 zMGU7;Cc1|+Ld`*7?~GH40QRO3plQ^fYQ~{jL9P@ESQ~e)5NMchM|$CJ+MY^9D5kX- z6t+EU+7q5B@{x`y1ORphia;7#0qso`Z^aTu+*blqS)hX!^SvcZjPXW<3gK-hwF3br zIP|56^h#R6;+;!Fy<-6Uxf`A7VX82-?MN-2!m)w$54|T&4Lqa~%`#5>&FILfd@IL%e1tx%86HM!z5KYvg8EB^qdpGs8ijMJj5yZp)52ksN47oUr zmG>l?wApV@xTf1m)xW-KiEDPe+FJstjErN$`~Zp4nQ+&;C>< ztTGVXD{)d16QmQcsi&$LfJ9Bdr|t4qjVKK@;Uxb6iDw`F=v`2HpWCN zuId%0D>0jKMROl@?0Zb`>KdeJYDW6roP zHm@|R9HAxQ*(TKI<9c9juJj1*9;g{QXSw~vQE6O^??jc|8l;RimL+x;t#)Ovm! ziELSU>Go@Pdm`ubZ%_u(ONn9`7#x8pAM1+A`gU83{{U%iavH~9u2l!(lw-Oq6ka5o-*4Sev=^V{Q^3p$kp6-A@ydUDn;vN407jukr zpgMMCo>J|TK@uUjsP5!s>p>}3>YCVX;n?X)kfkALAxTnB2qV5}Cs7(I%Ejrj*!uUX zqsm%Jw>6{n1XivM7n*Gi%Kjt)gs=Qa^ggNly#)q*MP)z(kaq-Pp>)e7@z4((q>_}H**ccqU2LFc zst2Y5m7N1BJK$Agkt8dFWj&^xmFHL`AnjAuTLZF7X)T{Ox2;;OeHE>3>TAoKn!VZj zVVLeAMW6yXspQXkiAdtDvI5!B>1l?N)J<-y32m@9QjW<_)RG6@tQY>8jMd1-kA*dA zWxH!j8-@5Fg{*(f>YZ2pRdcmP5uR1y2pVmTDoO$4#v~=5{{R@ItNw#glbA$kC`)4? zjDDQgGHjzt%2bOelNh$7=NSB`yG^wb>{m-Vq3bslL_Q{e2Y|J#ZQnZmcE@_KTbpfo z@c`e}r|p)=s+0l8n1ROGtHM%EGBd7>)>lcpEk0Bctacsf7dlqs6_S^l)ufc2eP2ch(@M--YkrhMxwqKnc}q%@|M50s8`>03v^j-QzC zPh2`~djZggP4YHIKYqbOPLKNIEqV3uQ|U^L)5(1PDYaZ zVQsX%E8B9~fl9~SDphr?kD@i?L@RuD5`cshq;fqCPFm!~gE^-j$&9dCkTL%N31dqC z03qfc=@f0o#<;FK6(d)IwKtL92Os58egSD(7x@~=yB|!NX}93ZcHbj+W4NZ4M+Ae& z_B@YcQ?Z&$sNSIi!Q?`kvz4uo!-V%+UO;~6@ao5IT2|Nsop(ZPT&SV+CP5x!sL2oxG80x-SxU7wcv$^=AN_jJ@)%AC#ZM$`F?3Ka(69(1s&XPEI7jrYD(M+I zRg~2-d|W$_JJ*+<6m{ZcGGc?;k7_V}(&Gx5QD>oK7*g!SC#8V5c0?F^?gHfyH@D z4gjU4c;38KkxJNTYvqzp0Ml!XIK>nWPc_Bq!KIT!A~8|MbAh!swm}DKkx~IUq$!;3 zik*%7kzyuKA9^9ULqGc7jYA_mQBAUwz~Y%PUVz$)2Wl<2V2#BR+dyzB^q>z)su8ec z#m4xjQxs#0d}BDpDF;k8OL_WYtBCHRdb5XJo2*}5!D>GI=$!R4dU}u^I zY=TBd{{Wh$rMH8ypjMHq1XnIaOytWNKA$e1THtfg(Ec#sisrrcudN~wrw^?<%88)@ zv)_s(kWMQYIjSKXP|a;mp;X8X1SY&eq@iQIbhsgNzT>~3e0pEJ=p;*JADaCjJN!5W|)&*k@fCVTk9jTD&abEF3tYPA? zYl`2D(>ld@PJlV1W;pHu#RBVzN{;mV&R}m*a+XheRun#Eqt?Ci0bTP^3Rx@U)9kN2 z&>F@6F7=xed2Unzy022qgai;U@||cRWRZ z)kAe{{ZI8suII7ua-lE!t4d^iu}U+xMk?iO_)8s&5oP9) zwaqd%PjbOe`BnDS(h={R{Rx8Uh>0b`* zPNXunQNGy#RBu`Q8@D}Tvs*-F%S=I%;KFt;JuTI!{YXn={*hWk(xa_OaX1+5Qxn~f zF4=F$8Tm2KZ{9gjj#iVVQNr3kKfWUW0B6~jpzQm5Bo^?XF|iez$B8=6H%ic?qh%!H zA6oq`xOCi0wY_Y-%2pJd9H+H>@9{U`1Q;wkEfC5^qvlB4e&(%<)j53=B|IuGlt6w9 z^-y%D5#9L%rXiyHyD3tim z2*C`b{{YffzxvZq9Z7MA8Kxrzf~6hBXU%oY+lPyKp7~njtxGE4D3jQB9)qv|Re9d8 zcIA9R3W}11twZg#ReD>IH=2>A#k|BXp^$Nb-1_lT%-*85fa+ZtNY(!UugboBM_4?L zq)bhC1SJH1RbZ2Tl*0v0x=x&&XOW%7bjQ;r+0QJtZBshN8;Eh0tYO69FF637GHMDJ zP(LPHkyoq5N!SzkhqY!_yX)?_li?TQpDU?PAt$){RqDjUZKyc%JVbJb2d82x+*B5c zHs0B%bn0Avg!to9xZl#X?wE#_VWfkMjBmbcBIz}$_uE5n6@afQx%(Q|pu6(SrKb{7 zR6B86J=h;a)!s^rsgtEYzh$>Rr%siB_(?TcnJZHF0DDn)c`y8fN<*0NoLhWvg#fYo z>#C%gt|@p?&%If*MN-*mizCFfx;w{`hX>JVDL<+aQ&y?OU>;3Utn*Y+GB_vqkruJ; zTPbjUi8VWCm=r)HXV)}9(R+^}OGrA7Y1dsV)S?FU9|gctH_bllxK1-c^kf5Z2~tf{ z?oAD3%D1{|i+!|3LhD(_p>Cmq9s5Cx} z#{5xj;(={oZ`jdDUi{M`SXdxZ!iFeaW7d>i82Z(s$w+*w2X58kyeB!{g)eV_X)AaH z^HfZmJt6PTfLptBT#)|f27z0^pY))`u~HNWGz!ShPkMdVLGM90;K@0ze3VNj&!!w` zfyFXVt_i7Cuy@Eb!lR7i74>H&g0_RsXvTIO>#2>7DbzF*_N;3jWYJ0EiX9q9dUQCS zYGh%|9tCiME1oKQ9qWa_Xn;bNQ?(^d6W+R@4|;5;zH1ovbGB(?RUs!Hl==#BzSV+w zd)KuAwJQld=_^WckyzavQ-d7R6yd=%g%r{ht+q4_c~2VPkg?vIBpPi*gMnLN3lHbL zbi0ww0j!OwV5p6bD;RmlGnxtDPH{wOIL>Hi2K%b9fkSy$G)ru7X(1cti(w$*v4>-_ zeJG;Y18i5CMLM+MG)!0T$tjM!9@X<7#?`OoUjxwB)3?;+g=3so&A%BSI_!6f5>Cg8 zl$h~80QiQp`$f>MbslPv9auRB9+jwkLFu`RxjL^5kWSw9N|`fd^L;?5uA;fz9=Rp|mv>W^w_?@S`_^GP@DS&O+OVAPJu9;uEfJKcwn)EC>M7{ALkUAk&Q!H&LQ*$UlzSy3B%eW0 zom01YH6VKxxLMyJ-OyABRyaQX z)pfSbEGRZe4Xcg#{{Xd0L+Fs?0@BcRBxzPo=sy0UpwniZ3g9i$Qm`^SHDhmK^rnkX zY)NZIlkoZ+#9h8&hlp)hI!7N`+!&U^n@e(10RUjub@0cg-Ap{as1%F4F zkYhcE6-BVs5}h|BF7X+HbK{p8$F8>4pXxY zEM}x&rwR!{;NTx@(V%%W%T>umAZq^rI+8N=C(Lu2r&?TfAS9^o$F*1)C28E6S9BC6 ze1w2-p;cRytgJ-8P~G2=HE5I_WuC)pq_BHyc5kM z$34HTc}(nG60S)~+GzltND2q(oR9OS!=8upphLMEovG9~In|uh+ZiC>;XIM;UMoJd zI#;&mYVN)>O2!EecCPCr{pfVBea$alAPP{}QWu~5Q^{WZ(2c9yQ=wb)Kq0NGenk?6 zPz|rt(I{CQ_oA4gI<~JB9CN)i7m>|*YVz|`ixNokoFbWV)26t!ZM`t#gBzNvAs~4U z9OIGRg;Ky7Bx08(eX=t_y6H$c`%rF(H(NUU-~*B>E$K>9%5%5YyD8JS+;-Z8NGn%= zTI8IYAt|{ZXsilhKxcY)UZJ_B$j^HF6iDVk?_9_`is5W}n&Pkq)I NZ`_ZLr78E zY|;YARnN-W(W$&;+7OWc&1X| z#`WT{*iisNRB>EWyJHm3h(_X)r$1U26U%T@N>Lr@KoedGBtqe zpOz$TSi=QrSJsr5QU)_h`9tYm`BDyOGaU~PTIO3k(`$~QzVwu~q0VbqLXeip+Z0P} zp*w>?Hj|n$wCCEg07_Pric61-Qgob9?j>8C(Pjg<;nWa)D=q6@nimF~8Z(i_YHzZ# zDmq22FPtek%_CM6==lEtss0e6Q|on6!QXAFu;0N?-vVL-pYYe(Qo2otvzCFO6VWk( zfh>SNl|Br9NuM?J{{X?ZmmSv(pbTRa)Mr_TX01=8j zK1`)cMNSokxa{tp)f0SMQ2eUKF$h%uAeB7r$dUs1A$!f zi;-wFi%(Hr8=Z953B~wR+7E=|Y^!AtvciYjt#8?G(*vF_55y!%K}jTbAL&QL>NwXV{G7Cd z*b;NKbmGXanGBfSSa(e!R97tB=PyYb0*9Xvry3*H^laZR-IzQo?1g1oY`8~2YtqG2W zhK2!C2_B%Pp_A3qq*5N9*3X@7T^l)YbS<(5lZ~jZp6L0^zWZuH-*7Rp74!-^PBI@u zEu>(a9k{6o{T&cG5uqt_1aZwldZ{vx3QWg(U#8~UCM{kXl*^ogKM%cMx(JIj%BPFFalq?(*j>m&VTSS;h5*cilW0B&tIVt{O>s5@UAsk}5FE+@g zLrHNfN*_unhGTmgi&XJ;HDDhIS!mm!`Rb`v!j-e(oC@SnYKHplCwcS-R3o0Wql1=FMP!s{>_zVh)qFR&K3p zNx-Yk!Y`K?s)gMFv=f@-&l@j^lpC7#ICjAl8eRA8>rdYo`_6mRV{TWCJu}dcM4DYV z-BkN-9P?4AU%eF4{l3*EG9(baxBRQRjya|Ru${NB7YN3GI$90TQdTpurG<{>giEK_ zYFb>$-;B^|fS7Ab8TLKt;H~I3-1GvUO7q15tTDCca(JQ=p_~o=v=eK_H`xCGT70PD zfr3KwN-7w^p%SWatmA5AD@U9l=LUs!SOl7JZULOo?kK1Y`Kra7*)$IN_T+8-XeGI} z^i+@ncIVoooK^|v)9F)_ws1~+2Y<|}>nx=GX@#jpSR2u?h7qe)^%u4XK1}-PoE@v$ z_uO+!4H9;){ISk)n)?w($;~B7{Hfx?yNc(_sO&~)u!5aL>?yUR<35yUfOCvj6&&M= z1|!a@X-ZbD#YicG!A&H`8~JyvU`EL_LII_sIn|9cN+XU1V1g8@(!2uB!jAcD@+nDU z8{)Bn5QT6>I+jWP8Z8V1+Mh!h?M8qA8Nl3Eybj`|iy0hOK3_bF)M7L>*wT{9hXSMo zap_$V#S;MrGBc*SnEC!1kxYMTSdJ?&A}LXldr<0(0ph0>8`_y%Vcb@r*%Bk~_-KaO zz}+T|OCHqnSQU~30q`<56op0hsZktKOnvsFMzSm9f$d3Eykm-&OnvvIi22}FqcRjg z2eo*>AIthw;EX*hpDXQKiL8Z8B!T#8VkjM|TjhpxUHNg?P@`E2^e~`up4D0E3o2iZ zr0CrHRrzcuABLhXaT2D~H1_nRlx!){iYnBIG77*NfyE7YbxDk+1q`Hjt6xz@3_7EY zvsw37Piibz3fF<4k4p3Mz8X5Qu0|d4F8J%x3rsw9l@$K~$|?h*Iz9Q*(=pkjKqzLt>aj-_RnJfl1KdGGIvvHVi%RnA|?A!^sSPM^}V-kRyFjo#$sNyu3XP*GO- z(kgLDZI0XN7fzdjI`kQk$t5z7G^;rCbNbbe*U;Bvb`GqmD?Ud#!pCD*Ux(cv76b%a zpDGQhK0}Yelm5Tfv^ek!ASL!vTxA(?KyIKvi`>^N?etYiBvox43d@KBOJxfq3OV2F zUc7WH`?`HRA&$q>wNU1sNG-PWZSR zBOq;2!nSiZSetNit^t0AZ0WwI!XLZ#eZM)9XnbqR4x3R*zVqSNb8T}{&~mg%|M#vC8sH=q4;U2&zOnpQx{ zxHS@5l7XQvel+?YTAb+S6+Y?|f=Nlv2>nG=*=W6RZak>vrvVv4{XJ4^=pTmt7Qz^6 zTASCX1FIM!se7a^g41Pf(83Vm-yWOOw;1}+Q{pG-R$*>whCnIPw>4e8#7NXov-Hhf zCQj{@IjrcQq#61}b_(K#VA#y+*CHrCg1vtUZybkF8k7!}F;qYy>DJG>uKP za!C78Fq~=EfCh<~WU~b%gM*s7L{dvYfrC|>LrFSCTH{AjTLzg3$oc9TNjR!a{nRNc z7{++3)y0COn$Nn81+=X3jMqMHiel`|FKa1~pnKGFOXCWnua0YYpL&jE&wc%C!^zp{ zN1{mz%0_6mR0n#GLXHOXOH1^rG(G5$3)=$~?Rne$sf4KMuIs?u)}btYE5~Y3cihmS z&p}=eWZ?SIp=>=4@H0=P-OhLewMMq+2eAg9L!&?CO5znJN;K%$`q4>VF}+1H=f2cp z9PiB|R8cU!LvBSVxG*SSv&L~sQufYhFR*MHUU3ML2zcel>?KVg*tjUb1|7f zV`0TcJU~wsD#0)-mJ zP|zHTa5gK7UY)T>`GG{+AMnSeaHHCihe7X7gP!$)V`Vb9(EJQLcK5Z6Gp&zuco@y2@%$PjeNQBRwCu0%^235#d-dzJ4apz z*--dj(rKG7#9LuRt5TFldepk1NVC8#pC2VoIr ztXj8BL2bb2%VdxKA`c_-{*^+%{7HUqPsuGfTYW0R6gkd&`+u!a*y;As3d@)oMXLG zdXCDaYSN68?OGGaTznzGagB`uHX3Co1v~nIU6G2EPF`GDh+H~(*9ir@o%>WPofNkf zkkJ|6AY!*Cx_!5Q6MObG);e9R?%q3bMmSqwc~UW0T@th`E5bjOT5Y{PVK=O!aqY0J zMUIqcVOj>>)DKa^jcbCnl%*#X)$CHXL*2JX$}di(prokbTvY<~czL-V3Qj!Rf2~@t z62tNqM*y5v2LAvZ^*E=o<>tKB?Va&jC*j}POMYc=nwz&yW=jPsUZJ26JDSDOcLMGs zsm~^KpRV<8iGD0O7CS(cs05Nb8ZRB~8On`19jY|=YG~uju=*PFJuoH1Q!OneKosx& z4GP`ihFU|PhgtwVx2dj+y+wB9{9<0o17_qug;;OJ%vY z8bMN0r6gd2IrghmhoDGku(FayKt2A3p`x&cW#&4;N!!}2pySa8J5)#AdIsNieVGij z1A##7Fl$}t%PF_OLuEO_am7RQ>}F)Vj}g2mW850jVMYmMBVmE*U09=T$Vy%adz5le za5xoBx<^`+cdZI~Q>2B8sodkEB|`?T&=azgDU4*E1w%_f1dZy`IVn1TLGcJIsgtaZOl0|3M z7t#twuWG+rRD#-Ycf~wJXQsAF57Mb!MGY%ydB$;9xko7?shvcXcxuLKd1Gj3Ix%jh zhFW=QN3VLXHtrRS_p6Umq=m0idCxUMb&?s|$_{BPxGJDcA6v-x>$s5s0a|iK#sx_7HGnyqU;}telF!Vk4 zr5AS@B7hEaOYs34aoT`vd2Wsl{`9muQk?3})gJTMJb|Cik@G4Ez*f|<=;Y28`~Iea zanz{ep7hJIbqs^f#)U5cc0ZL(v&qW#78kxg)bnc6GI#bg2aE{XpN`>BIO3W=Cr$|; zbp(O9t}JKXyp@ji(nWoc2EupZloP)+ps%GXn#K}EHdBsi-0w(GqRb>$Kpwl-G~oBH z;L?QdoYz7Cr6Yq&3Ga$10zjmxIqy#P;Xi-Sm zQwT{E@CUzoY5=tyjj6IuE4jxsg0V_udb|j(1&a1TG#at2R-s2HD^Z1jDVLF`5GmY>;R6+g=%aOvQmD(o?O!@| zb*Rajw*LT_@3nnD)lH>QR&kw;V7JJL7fnQe7Wu9(Qxm5PB06@~n`OU35&>7?{{T8& z!do`B3$Lp|&Qbu})Xad=7)Wg+DC{bWer|R0=$6uO$}y~*_C2eG(v>r*bz*M)Upnfz z?LrGnSPIA;{WI-Q-D6~!C&0dMrw86NLpfp}L#K z*!Lclb?K>A&k2bEy+1I{wXI*oqqaXfkwv}du5ZqprLwr_`DG;FrNoo@AL=P2oQC9) zceO|HA~unLvYiU`){&5Y;|KGp87h?0T6qccHg`|4-)e=oIOMxhUAZ!-?;&YRjx=^r zyJxBiPimsI1)HE;og(Nhcnnm1Km7juu^S1uGw6{+rW9K80+y z#*U@hlDR|@oG53w#^BUrOwF`iT9*~&DTVAlpI>UQ+U~a5P5#bOe7m+!^5T^m1OUB2 zLce6oTsIC7Rjs}ArCy&yREm`d+D9YxT&ErlJo=KPZJ$xt{VGy5)d-S_4csuL5uY~M z^dQwAtszgeJS9XLk&LB4>_??mWA#}onn0ZB+1r5O~t_SNb- z0sB)CtpXMb9ej_ zD3i8pdg?1G-%85U$i{KSV_jQgax;rsLD>3Nm!BKEI$@HuLZ)8Bekisj6OS>)S5)tY zbdii2A&H5|%D+ANH)BYy*`!i%}28EKhRU?btPHI%_Gprl4Z=CB~(gO0R$6K@zuE| zjY$-Yst8K9t~f{;PB-*Rh&}(p{5UxGzN)~eE zwuLz<2}v2_^r4j{JDh<*NOa)wLMlNUf@(ZZA1kM_VKPD6R&(h`#e5D`o+`G~jmAl- z7}7STOThSYQ$GF+rFfK$CXk@8afypVn#b%NP;rXEyB5(So#=#<*ig;n z8Y#Lq1Eid172u^BSQ)Mqk-)9kEbvk9OFN1{80NdSD1)*Z{Lox=0y1$}hgL=`%~J8U zYk0aNQ!3A;WZheC@trtEbL~*fth5~!3u7ii$X-fF=8+QL9xR9TAS(w@9)s4bwu>^_ zK|UG!;;P+T?I%$39cNF;`uY866K^+dW))Cm34B<0!D{$K*93?&V8yn z94m~7??|~2h{!tfu1M?;xv3^BP;63Fb(Dm;HPjV6r4HVM(y85H)OT5O)sU!`sO=d} z;Ey;@BkR}?dRnwUMV5Ta6{hJ9G=!*3G2!jcshv2VqnL6 z7*4W5&V4hsed>*S>ZD9zyVS;(!X-2dD0FBlLU#%&JCWNILN7NPjB7+04Mm0XGQe%` z#1Ilvq^SC2=k%p(Lw0QaJJj~qV$HT0YSk&V_zG8)6{D1I_*4kbrgKvLIeKF9(jm=l z#$~XxzZ~aL;MS|j%yli^oQ+<)4tJxvN+JW7qIB?`_5s5IY9Ur;Xp052;c4jvD5pY%1+ z6>8BJrd(a>by^nOpD-&pSN-Zv_tp-ymZ!yDRE;Yr&-aM$kMgcJVuyN+xP>6P%Dk!^ z&k{epJI0mt{Q<|K#^n1@qY4kVjq@LgQucST?^c*P4{d099H$5=o!i9>soptthP0JQ1+X8Ajd;VLQ{d*Y-&?Vv7GH!f~7>8vLa2p zP^h<}6p|F0vpNz+Flbj|YDvmzj8&yF&>OZ%_o&CBg)7S51zKdn&+yO=vWD^pG(ph> zZQ&y&12hbnDcwV0H^oj!*dEmtF+&7re@e#ai_@W75nW2oa1P?UwsL#t=}t8im1#N6 zWwR1C_gy$ji8VIN511L#$f)~eyn+-(TP?9l6Zl3ckgb(_pHC%C9Z2n2H(T_>dK^)2 z7|(7i==o7waf7G~;oI80HIcYi0f?* ztfc-ES>sc3paJ}=#mi8YBC*I?nV5DN+*F)dTDd#ss&S-)$SRPNc$1dQ3| z2+8-MA5K9zr=BN5_M*Edi2V^r_ob9-q1#ufG+Sy&=DxLT?mUJ^aZCAh z*L*>WNqil%MvxB{h}hRg+fz$_o#};0IoVW9*kZGaZ0*f9)8vmeJET4)pJgF^tq9ltIpE=4NXg(M-(;zLYFv9q3<{1zFYhq^e3l z-!;TY3Rc;m=*TUmf@q{|wrF-$sUtM_+icb_=Dj?c=x`cgYvg8{E+`&rT^g(@OHXns zmlQFzGFDA<3OpUCIapLrD3jj2g=ys1h#*oio@fjbmxt23tvqIi)DuZBfOoBCpwMYt zb3&o)jg2qGIHIHmhg3nP9RiZKh&vi+$Qj&<&6y!-(kbLD6N5pQ(s(u0ypg`D7G{YM zKo#tyjpzX|W12{s0l`%jBnl*Qk=~M3?ZpV0N%a-gO2-C`qj4LathPofr7qZIK1ok{ zw7l?fB_Pu*#RYlA1s+V&d{G$|$lEOwq;XllRxuttmt9hP+47ujeT8gtbfK*y^sJAs zp*p3L9BNp~hE4#e`l!@&H>ZDA%yRp8a%|IKZM2tPB}ylK2+y@={X|@d>f$YL$F|1Q zRbe!b>09xTqMQrV>dNL+1R*s)+b$!z7aG4>ze2jTag_jYZ#2Y*AI41)YX>I%O zRY7N{y0Q#d5D@5GCfP@XqMQ(vsQ&;8K{|=fp464Q(X*_dDJgOzPn-h-!dOuAC@V_RIMP+I8)F1jBdI#R z$J5sKI7^n>g`RYyJwYJLdNRvJ8Q`~# zh#AQnW3@+hbW)M0^jlq3Xt%v%zt04cP>icqTEGb-?Su9erC1WL8HZB#LX)9jcTnL& z-mtE$mDJ2U-O31{|z$`Y*QCnJyh z)gdJoiOMcW*XA|gup}uuNKhpEQL$Y?Y%4nvOm$Q3x6`X}vJ_U3Qla&zn_TiEAh@sL z9M`Fd@i-;2H?JSWHvuOY?O!}~04_STgSO`#>wD`dEx#EkA5yrK%5bFY4r_ zx3y}Y8&aE2usq(QJk=9%=uZ18Jv=1DOHYNAr~nRqJu5|n^0$kA9Bn0O*-6C)I&Lb` zw#MmJ_#0sMBAK?wl?BHT(NeS>$MYtS88inZDVtD_DL*QNsUYB>jq&MUM*JwVw%it- zD8V2TSdF=m{FWR_NGl_b_4Kd9_?{K9t%k`Rsj)6oC6UxiJA^g>#}t7vBg#!VB!RJ@ z7UDv-O?nh_hqmsm>13-OS`D{UcA}fyXEmt6@1XO6nu4Abl2wqOTCqNmQg;V61wATO zQ`)r{uF95EkWPNp8sfr^NCPx{=7>?>dWm!`D+&t6c@${YDMW>AGfuF)FC>9MzavUA z4)hYP3PIR= zdU0=@WF4v@m-3&*p8U}sI4M4A97pC5r}o_bdsmD8Pxb9V_@_VjuC45&KWbTZL^4Pk zO3IXIbmtf*o_#=!;~lA-jQ*U|S;3QRSUBzeRO4iUjwy|2JJBtq6qB9>G#AJr9iO5> z1aK)pC|7|b(z|B9tmzQ7C=O|owS{nL#VU}M@4XaZSk>6mN=m`}G)e||rD)9%Rf3?9 zcN^2mUMXq-cBB$BLd9SqA!Lr6hWyK0o>CL zq!F>KVlqOMG6Al{E-3D4Pc}1(b|Xm}={18yjC_RdY0~zcjSd!4FcXSp$4)jU@~qt% zAkAR$N)76$^gc&=Et!KDCwdxYsvRxq+)_a~rd^f+{8Y)94&czBhLr)&6w2K|IMqPO zn}M;Y=Ow85fui(fvTem@gG0Em2F8M7QkF5OntYIbP)|BR$fdIdDpG*vkf}@KRYxuIdCeomQijHr z+ZiH6%2wnPfm{<6t7DpuNprqw#DxKpcA%3prb;4OxF(im;BI-SpBRqy;`uuhOb1Ao zW%9M*B$}3EWagpmXe+@YrW6Vho-3mgrDSI9&^)IUuu`6Ch1L;VLKNvBkOu_n+ zj42*3lxN#Ked={FX2m5Im<%+dk>aQfkEJsv-+CIPPr1Zalw>7Lc#8Uj*NJ=jk5ZFb zC~tHRK-(NHz0(Hb^X}<(QzuwR&cP(|bK4bJyxJzkzDz}{rqKzP8%r^e>O!1Io?NW!v6-mexuhqOqzEjEcwC8C0c z60Pv|kVaB=ByN2O$l9w8u_bCvx)ce>ZZZq5Hj)$K^Q#`Gll2s_)IJ1gw_9z^Z8ZTw ziEOPoN|JvHjy%ac9FOv)w?2mAqH03J8e2=T+}mZT2xz3G1zreH$MO%7k7H+X?bp2)}W%cRj4O@wykHUZ_Uqbw`C=%DY&wd+sbsWOR4RvJAyl(&aRLx z7RRBvX_V{lu=);^zN4+4`cke)^v-cs#=H`nr)aKxE$i!tRP^)KE}?T&#=ij|jHav- zu1>%}Mt=wZ8WCM965S;-;l$Q*yAb>ec0q7q5U$os{~a-%CM zJBrBqjbTnWow9TJR+n^+xbakzk+H1K_@p$YEoX9S8Lo2KTy+#wrll=3rx4l5$=a&d zd!y}Wl)yU-C{^m>lzfrE0IN9r)CBuG5+ect1zh8d)gu;=u1d<~m#F2frwh%MCvo%@ za_O7H^45j2RpK_BDLfSqQR`66bm}31qL7~6x%Cw(3eob6H!=&lleTbi{{ZHpUnX+V zq}XpMb<)Gg8j-=^5$RTTtlHf9#z4{rckVqaC2{+b5}EK@aIFLPfDQ5Mio0|_SzINw zVM=RG{#D@MdlA~Gqw-C=E|w?_eDVuOz}1~6jQdy6T`qhZdtoC3NUSEt`=(gK8cv~{ z9gS{%F0i-=2W*UwYNj>DMCY_ix}vW1K9j3B%^Bez;=vWt+D37WtEiO&YQWn&Q_U3- zPG|&fE1E)cf(2t9S~YsqbhzLs`WhkmVT7Ejqn@~j6V62o3RYL|Fa85m`(H3BT6uMYH73CqU z;2dOgUo3T1oiL`5P(cU1ec#v7P^K0GX~qstE938p@d9o~N->oZ1$lnA7m~Qv%=MN> zru8iqttkL{Q0y`7Qn1>_HFh4A=T#k1R4tI8I5b;pB;`l)uO%tQkSp48xH!cdptcYm z2PrBbQXP^L{9m0gr$TTB*ye(fCpj7Rskdi5=UoPggd7EJ{b+_|hl*m&RoPq?Fy=>tFISy_0=l72cyuU zeQGr?g&Y%0*TxP-ERige332^E=k=taY3xRGO%srEX&{uVwM$1gY%v8n$s;sM6rDTl zPu`iT*|XwP(MeOqE9u%1rx{RODV{u zAQ80{6Ii4xM>OblCWd)hMrpRzrFN}#qflFm9jIlwNp(jZ_Muj_6UnR)q`AH`y(?Mm zK_N;flSQ_LE0bEyNKsOaGUC>O>qLf$?kTrfDi{h&~Nj%U7deW=|nl;jwK?HnXtWhf-0iZqyr~i*4Qa{b}7` ziWOY(O{vEcMgYLA$UH-KJ7Sz|qE9)X+K_Sqz%+9Y4P!W_TM_7;w>Fm04l!Q1PeVY0 zp-duL5Z~g))l=$gs*$=C!K=<{5hn<9kz{4$$zDO&b3$I4ZRn05p4Aayx`n3-S2ZDV zn1wK)oyhdAcRon38fI10R-U7AZ?R*PSSlC=C&1Cx6NK+*$8#oYn8OR`USKAH7 zTt!D}W6uT@d`4Px3R_D4R4beg!19yys2e?{uNDe?ccVVTdKi?Sz~f~86?%Y1G04F7 zt6cj8_)NDY^ft64rAW{YS#)wo_!Kjd$A3{>DIzWSC2r0tG$QMM8>}r}Vw1aMoq?$6 zP9{TSl4(~rR~{>YDJ``jpd3LU;F=yHGaV~IAme{}zBWooul0kT8%ooP$nPzQGL(R@ zG36b`Yews-{{StNE0bA7o4X5;RE3RMDg)N1Dsp8PL=R9;T3b^AT0jSXOjLEds9Rcm z2?PR0Njy`R*v?E#szQh=1SI+zq$M(%aJKqLP&ng}REtt3?K1v)jv7j%%SZ0h_(99>Q$+{d9qN7;Q+%=#H7P(HxT-2>MwzHtBccEb3i6)bm1<_>*hWSUD?PH_ z)igGgsDY0AZ&qX?T7gn=fx!ZrKVoeHb$s@uL|Q@ui3Yw__?K#Rb6k?7;Uu0o8`sox z?at0Z(4Yz9$}1xHiPCSOw+dNSFi#c9>Z8gfF`qoQ89?Bt9+{%rMv}spQ8@i5W?Cy50Pjbz&iiNYSqs61#!(nMWOv0JhqV9)dKtF( z^8IPn&_d3`oYPqGw1SYF?gj-u+eDIf-_TbW9+>>-lp{#um7+4(*+%=1%A0%?0!Yto z=&%NN*m{~`W&?Qj8`9CDtOP!F>CeznjGXVswMM4}c+T}8$U1U8C>MlJ(hmq~+nRJI z0OpoKY-FeJOs({Tj^@6jNYvQsPI;y3NIOvB^y8eGZPyk?Q?+Xi1(w5rns3924CGhIGpv;dr0oxSHnQxp^&bShGBAZrLoTn5RPBMo5Hy-q3X+g)eMXA-atm29zZzqngb4@ZTj;<$yy1lof=Wnh4Kek{=q#2ZP$2LDbnK0bG76bp@S|wJE1i zgy~So?_0=FK82Hv=DMiCTak(=7Sw#BZ(2*rDId~@?qELvr{5IoXaI9u4y=vG>0Bxx z!B3;Y06U(pTq6{OOfLTF$Ll6jWxCkW;8_LW5*ufe)^``$tQ}U^+#AV&`?sp4>fd=Y(vPx zK;EdmQP3{WDk)Y0=7wH!jUn;zi?##P{{Rspc$>Tu0py*k^9!jX-B5%f(S>eLwPa)A zT=cqJKu}0I02;V-?TTfk=2Y`*QW7zbQ&5AN(?v3HdqkTrnKyY8%x2RXI?5Q-cR1x( z{*@r5z`5!1)U)ab;ryxQ;WEpG=bQ;75R_y5sfQ*)Zc}L+YV;V*GL7h+(8cQ5h|m;^ zb6Sv7<_Z22>ME%|8qK`CmOtJI^AItd3>wt!%n3>eLIF9?9jayigNZStmV$K*f&~}p zT(*Jl@zXwPzgyKl$sw3dgh*QlAdkf6Gy2mX4eav2W;*1Ck`km8NI~04&cyp2^ZC}T z)jtenK}!4puL~r$us;ucesvYlaGfklhdHke)VQ=94U{?igUwDjnw2r$MAf2KrL56S zyF`{$veSnsS?~Qf;QHdFrL^O0=!FTw*&0An_WuBs`+adobe+0PGN`YB<2flp$j*`2 zE7d>Jr7rLtODm4hG2i7nPExO?YVQ)F3nt96=EEKy5fLmpwdqoDa+6YVZRu(_Ndr1+ zVrNgE2gh|mOF_!E0Gf@PC&N$f#x~-SEf9I#8Q;*?&0Rr#m2|kcl`TQMk(~Ff6q{jvSw7!W!;>Kq(!tUW3Ws`b z>gAb}^9~S_p`4xbiixwqaZ5&q&JN)E*Lu3#@}M~9gh&bW-y^rusIiFViKKM`WI++F zYQnVOf$2=!EfP(`qrR zqROMBM_!_=I5KgXlf>6!b2JNO+Up(0Nr<-^J-|FxnPtB^(i;jVNg3X&of9?%#gs-{ zU@QPIwAGU7U~#=FYw#-s5OIolDMM+ADJP~_oe2#1@s!NTHD#sj- z^z{6K>_g6}#FQV>qHYn60?X<^9nv?V6w{^Btt%rtW15&|E03}RjD-`yJkYk?29heH zrqMA0Zb(vr>^99qPWXQmEV!0azo^Y=k>H`C3y44>y0(}km4S~+bhIUzLV6@bA;m`u zNjvSmao(S4r;xJ3$nF5GN%TWX2`WzAsvY2$7pbQkSqIcnrhr0OA?R?FMky&ze)VH$ z%qSow#HZV8HqmNZ>q#mA_p1wbkbrfZ{i%~^4Wc&M!FdS|4DHPU)>amz+t#AkBTs$m z+a5DX000hXKB0u;kQ_o%Kq8G=rVFNgS=Y9QWAp zNvz4$V4Rxp^2Vg->rT)!%PGhiBk57kyrZ4)eWVNs%O#@M4#6obVP352N$P{!l$MJcWQH>Ii?f!bt_ zKT25|RtXvHMK;g_KE3P9Xlons??+5JI~bhN=ou6_>nR1?k?L#cT(7~RsAReT*ysGK$$u-w z5dQ#xq7;np&e);dXaW?2LBD~A%RxgRZ(Ktv3P0;eWo{_wD$P9FR)u+v)stqUii#Xi z3eV?7e6T_?38u9uD;w90m0+Z()~%Rn%x4Capn%o~CyE@YZZ89RcLaDkDa8|w zK#}W;HMV?RnmY<<=*wwif30dYiPF*$sGqeCpcnW-#T4scpapMBjLKWWl%9QSH13Rm z($lFx-x<2JJzZN*fTb-4ZIa``cq4W5aBsF zuRl6kAMnPkdyS~HIEAD4hjZ;qac@I3&@-){30gNesPyxD;DAIn_8}lnr(xhPhHZ_|_$uBoypD#c^)D z!Ujg6?McZFqSy?i2+~qrNWzZ9b6k^YdQ0hXxa6frIRc+iT*aj583!Y5){AC}!l1U7 z8%{vpgfThEvO*+~gK|;VIidkb) z*i_s~%XTJWvRp;)R$)8xwz^8NC#{TRZ{JLVZ^w~ z-AGn98RochRkU{FPLn=;_`!8U@3|Q`DnF%UQEkYS;cH%%m2HaDek()?oVfsOt--2q zM_LTan`>8?C4Xv$JF>HSvr*_ihK{HRT$WO;fl0?S=U#L&6Kiy^!a{;FkbC-yyW1X* zZiLYoNJ=*q3GoxEqsOpL?=i|Pxz`+r`VTplG>D^jX;7=;vT;A zty?!pzjT5fj-;*B=^*0+oc$|UVdmyULRZKgt1pTKMI~+=fw}gpeX{J5lqsh&2;zc! zpM?ytSLm3S#^kVo3PIn6$o-$@#fr02L9s5VtAIG+`qCqZy~ zcA`K_n};xaq|q@MG6(Nc;tAtwZd5lLUy(t2eQREWRL=&Il!JlpYsN#Z<+QeNcvdNK z+gZk>eQB}K3u2!3ij_N{`tev+s!w8jcZ(5krP;fv_b6Nw|r$)+boNh8Hry(?ilr4~= zeMJkSRgHAT!L=hVF@kr>s?52OmYktU$E8*ngNJm4gahlfSYp9!M;XR_F;!D^O{7N4 zlDN_}sd^h5MHwYg?f=4<${;l zR-k)iccpUp8;xw3gFZogrQzkq*rMgmf59%tMzVpp2YMOnd15<9#Vc`1*e07_h=#lu z(zJzt5ES=X7QBr))OQCMC5S329N_PoiFWnW@tjJb3L_;d8;Uwue51wi&H%}vt+Hln%a1Um zuXOs0@4X=##|9L_d83i9QpwfWG`${DkWzO^Cx2>Mjr11V?3_z0Pzc7L1q&|M zh4_J3_RV>z&$fWduv{JeXeo=h$P0D1z$o0O=}9TO(j7%r*mDYTpTcr|>9?O$hYLXA zdB**!yBFdey_ZW5Fr_$yf&j?Ku1@Pfw^UlUt)&#|1w#P))iqs;kbT1N{gLL|l-Y4B zI0Ox-_!r2s_yGj=Jc`aPy=d&BTbhBl89UT8j;fNOQNG0l2c2Hh&54k9xZ^I&00TwY88_up*sFGId2}&wjZ{oZz`Y zQd>%}m2wZ(t!i8s(dAYTrx>Q;An019Mo6VJ+8tR5V?qhs_M}Ng4&NYIaY1QLwy=;l zJJd4zg4Eg+mnEeuPTbXo=hM<;M{z95N>it7RzG>@d*q33AuAXMIX&sB9Y|E)XtRr& zUVONj=qLv`EA8~6lh%==Vlt$qOD7l@#Zq8cA2Hh*ixS%+r z_;qp(5219z+@EN<#*p-uJX8#Y4DZ^9ORhH{rNnGb#+4>}EF-~&JTs3dsD)3txZ;~s ze6O)1WA96@>?xK6hMgKywF^173QKK`L~tlSom(1Qe1_Bqm}k8)AWka4QH5>t_gP;n;*d>XbnG){5k zYj92`(oi-mMKyLcma1-nyeMNat~;NC_!Q^r+&A4j_#|LwbXn z9DO?2SPL8A_o&ypkuXwQo`tkW3dTYEiae6+9B-n*agfSdQdF%waak8ubtlNmaw}36 zKkC~a)hpEfaG#Z0BQIL9fJI|Bx$|Ft#c@heaGVl&`c!k};=Z2CVaL0WRp$I&DkVo& z;FDCVjFifE#SiJych7n@=)~956&28}yufaL{*_C-+V3t%NNFwv=X`gsJB+05owpc( zd_iHE_Vp0xE}(k$svkg1O{!yvmT`q`lUC{dJ9(bsnQ67Cft;SzpWb?FsT!2yabqkk z8NmXrjX1kxa!o#sjXg~oLQ@qlCv%gE&Hgd<18Wg2$#4{p#e-Bhtb9eb$y=`5j*RN~bfuk30aGkk#pC`#V|1oM%RQ$0V{Fz#1{Jst3- z+d5ABU>dDZlRWOqVs(iD|~Go@Ydqv3~_gK~*}`Pi*F9l}UBQO1q8``zZ;r zKGRC%sRwU*(q()ZzV!{Ao8i8=+7F+>KAxXSMB{e0GrBcSpX*pozzb#IqDI&`sOz6v z#6*N9OF?03(h1upsvQ&4^7?XWVLqQ8KRG*Kj^9eZKcXVje+5XA3@C&)94$$jl_7e z$kfjG^Ca(@ad+fHQhaEn0&sTiOgkD{RhHGA9H5QpYn9?N5jDt$R563)&h&XTqf%ti z>KkIn3dkqgo-yLi!-&SA`qq5i;5RKUA~DGYK`K_n3Vz#tyxi{YGRs~IEhk_>8~WzA zxSOVp!j+l|VzPwcNCf&3QFj@SMP<~XqvhNmK~PZowqc&*HzoK!3bf-H)O~6fsQRy| z8@YHbHpj|12-KoMH2l|rc)uf&y5@SWD~uw#<6vMEifgE(OSv*t;kwe3!71D6nv97q z_SBCR2#lefif~SSD_7~qAl&U?iLA8a>jf(EV2UwrOO2cKZjYYslG3TVSa1?@KNW zeX&Ytk4#p-2V`<+IuQ*4njyz>ayX#SzSuunS#T$QKJ+i-hDnQgV+X!{Dl$XG3M86+ z`2#sB;-Y0d6`lV8?L`siXPulu$ReMIaZy+SUrL2@T9#6kbL&J#W!&+#Z`hQY@?4?Z z)Di(I`qa!+xhu_rK&s866U!iPT3=6E0$K`i9G&X;Zcm7uh;P{MxRNyEYLA_0eUN;- zR*oVzk%BR`0XhRtAc6*Q=~l^sHQ8KwDW7G;l^r0#kPwuelDuY?%ONRRK+Ak7L(|s1 zZ;xu*=8A%V2>~M|WcH~Bt-&pY>vaSYl(sRP1Kd}nx1+nFDmA7vflevZEDeGFA?aO_ zrYaP%2s!}R4oykRvz52=t(2um9rh%1$9iRLy@2C-U0wzXcii#~8lYX2b@eAw%elJE zvdW0)ZNJ^P>E!3I^{Z8-S0|vNMdg&48nCpWU;*nzx|fjiZK1TOOA5#%2~IXAG`3@0 z8r0!R)RiQqI!06vuhNRJ1he5Sm0$vcum{qGn`5`mU^4r@MV~T2+*1s(cSAZRfHf~j zQz$tn-_nRnGZCseVMjT{)Dy5um5kBuNf1h! zao|86xunRB%%_y8I@RG3q@azDtqm)ul_xF5kmD=-YD&?9K9oA#P<#DM0)J9KkK!o{ zLf&t15_cwn{{Y%sv0{OBwuFTfj4U57trB`Ir8webnCerUtSd^1p$EeIbABS^g}vir z1x^(JDOWYHzC8gi;w7$knT6;mRy>>jbv0?~gj@!;SAsTCP{j_%!dpvBA*hH;X+4p# zKYsM|K8v--35fKUBODZ@DZrh7_pIEXMqGT7v8LOfDpI7Ow^Dq-(k5Q*47IImZAwb9 zoQ>!S-46ybRODF_6y`MQ3&;b$MN&WZbJU2r%U$x}^OlCsh{~HdQc3J_=|Rd-@nK0n zv7*PgB6`y%D%x5xf|RIaRHhPJ@fD3Y+*LNu(>*a{Vm-n`&cj|p5T&|EDm-Sa(rw?p zWhk6R6^tivywgSA&>gPF7dk7-`Bh0vdU)R%r+;o)t#Mp-M~1XCGOhAzHch75?6#PA zEw|84vPQ@2NRw-W5ou}Qg(*PkZ11f;wAR+r1t-{3w>xK01V3wN=xNTE zwY-vIp$#~)PP}(D8#dh;nT^}-4k5)If|K5-T9q;*FA1@CWcgB&sYm^4q}Rn;a~HV~ zFWo6{h=#O;K4X<7L0$4VrKqs0M7nwwz{8EKwj3%thONajcKCm3nIRGzX{PbWN&RS+ zx|gPI5>TkqooJxop=rQ5U#xc74*bofh%dfwVz0Gnc!4g?E$bqhj6 zU;?eNMn`lQf#~iuweko%8lv?)l=U2B8)fzjR^3V}@SMPK)p@(aHrPtHwJu3{k`@#j zR-K4A)yEZOjnOAGp3P*pCRhuQlGsn7r7S$o5YCmfp~|&$xBGw0!n~B^f=dJU%>iXli?M%g%pjH0iYDja-}-^M4sDXrXk32 zX>&r8fwub(y?W_36c}GOa#~wLQdFbdVy5Hf##?leO>*CIb;Z2fEg)yO^ASikIWM0W zqaY7TwL!kyCM7CmI?|vBI&+YCKSM)au5HF|a)@DjB~G0nn$8|>!ldqqMqw7#kuEqS z{{VKbHxvcR>kiuYg*C)9zf9ER*z>2_()F6>4cExlQlY6oP(@|;FNR%XdY{;uHc7xLJ%(|;1&-asXDd|AHOYNF zF{dIq2?|yUK>q+btRtWx6+I$N$;Q&=N*hYglu<6uh7{VHVZ)u$ruPa5y+1A^$b3KZ zn$o0$pqwAdr)s*k6&9btS#I9D8vg*@<~Zs#V=6ovm5bFfW0IFi)9XP@OC&u10EYU? zHdlmuA8M^m>L_-13v!!#LawukPZe0YF^!IVoj_jYiL}6cX27)p82aw{Uvd^JsP!+R9DgTR2w9745VCvjKq znuiRj`-RHp;AHI6Xh86rRxnQciX!U{N?G~4Y=pSkSn(F5fE0tVDbJ+`QdSzKRom%H zVwnw?%MT?WWhgs)=hBO}^rV7IanOg6f(XvvdY^@Hj{HaLrxDw42wbL~3R=cJKgy-L zHq&m3;-Lp5w%9tgtYoX-dNFERXcA64B3iROZ;jzUZ)qqeK2f+dMX;^6{3qc!{6YAV z*y64dm1&0=^~6+((h!2Qwj1+oI|`@UdZpB`<;mD$HV+4E4&t=pgBIBB`qON23b;aL z)^^W^r0Rxx~a*Ed4`>3yy^P}3j z;*~(+W)#nHGr-x|IY@Y37bUyU5Oe zp{apY*6(Z^YJa4LG@e zGkU>NToH~6efP*eN}7$kvH582%l6cF%I3~*L!^R3eq@o~J?JT2Lc4%0tqsX(r06Tt z=09qW>prdOCLzU(4m<#j1i&wN$K;rGHtTF#55-dvf*Bgb_QYeR$+f}D>| zcIK%F)9jq4t3~>KwB7AqB&K}IPMiasK>aJ>KkW(d_DzER_0}D!^g?mNsqYMg=j&g6 zIEC5cr7;uYAwE?HK=SYS_Qt}nzwJY*9}VXG!ces-#3>6wI0Msuf6ANHVp(yv`&&6{ zlcV8~l7cy>kZ?u?IFzLCkAG_DXef<2RbaDtDH&fhV-dGitP08inhVQfT}Jtk;MTrDFBp=nwoY?C0Gg@ z)Me6R4Kk%_8OGEyQrOXs&#GFw2bLtF$WoZV>RN~%=K%6{sTStMNNMKOLdx}Fc^J;> z89vkipKE#F3NjQ>+EfRNsk5kdIUMhjYAx3uVYcGAa1Y2+p-!!~kB6ZrJ{Tvq<0Ny% zS}%{>^_su2W>wxAirW_yq`8L>($kgaAA}LLHRD`U6}MftlaQdW&?h_g$6_iuj;@Iu zE;DfmWwofS<;DRiJ^aU!xZb9%y=vO!s@oUB)!`+;ogicrwh89}v@1*0{^G3~eo3_B z;BBd|I_@2}wuB=9XbL-?%2DI7B8%7JG^NC4Db#g1q^n>%k@-}2?a7lNtx{TXl}K9K zOK1hp3?%u~t+AZ#o;En3(H{29b+IPv9W1Fn8)|uihcHOSNFZY!@sevfS3ptuC6yXn zcF_79b@Z!s1IqpJ+PQM+#NAYPT_Hj}Yer1Ji0M z=i*3h4Y@U>ty+?Yd}5M)h6X4Z3w=@Ao`#=7m8C$2fpDZ8lc$V!;-O|iyPz^-+g?yw zlpAmrBsNGSWFGk(4CbcpG0^F!EtWBDVTlo;+}v(ks*bk;pFmfKr9&B3*e5@wBlgdN zr%`V365srWxG9y&PSVMd`K%(#zLIR65sA^lbZ}&Vu*WRRCSfY_-wZ> zn3EPG>pCTQdtH5{L_DgOAEV%pV4dqHw@ZO+!(kOeo#?;~=TcgU53>y1-OrEP$&VljCnxs^{LzK#j8EATrMd@WU!EbbCsx{K#(${^r#!-y=m3Y{>isR zx?1BQ;AFiS2}0d$BO$F$Y=BQH0Y?B0vS_5)I;9MzTh}ht8e931K)~1O^m-PHO&&g-ogqH zLVSvEAfLCmsA<+_Ww|ANOVqaL(4IzLW*9jw={?-fhBA9|M0EAfqbxSYI(pwd;>;$T zZM8n?oJkyjRISE$#{+y-rx zY^7;A!R!DuJlA3645DaZz%?J;l_yZ_4l$g6N`BXMmKkYJjcAkSL0e}yJRfB7liGr= zfbEV&wlOM0nxhzsQo@4Pqm>+ICZcX}82RZboeh;V+QADe+02c`YGJj`W;2uIN-Il_ zD6N;U5Z};t7~Z2MrqgnemdqKgI#rKFsf_>MsvesxX=o0BOGd@kwc;l^q_WCohY zgq-s4YsA+TB)?wKJh4`<(4Zyt`v7ObPD=uSNpZg#t%!erdCUMNNpgup(6n&_=(Ow{`7s% zsUwuBxwc0_6OCs=FrQ4C7LpFzDswOIu@O$rwntI|f(TkQC}iU}#}$!vx5B8se1%;_ zurUR)RPU1C znGs(xXl>;_l9s%=DObO0+HH61>K_K%ku_%qZGNAaf!GM6Q4P(C8iT1h(*fuHoNB!7lBxOP;| z>NKI$u%8iXKv{4B$n`vNTfA#zn2Tm(5rer{Z6VQ(3G7q>-yGCe#ICBi*xI+VVd&G; zQtm5TOJ$s;q=AFyN_@j#0G~rhqSD(Q)R{rB^wp040Mlxj7F(!s4i=e&{6#dj(l!Uq z27PKi4@bvj_!2IGD)9hP+Kjb?kaM;+C)%Hr@nfy6Eh6Q6_(A&~+Rm`zB!`Jtn*m63 z7|7zaE}fNRz1Si3N9_0Q%}y!9t`Zwr#@a|br_hl?SlpGSGLnaAt}4^h@3yDy@*8{Q zj#O6BN!ELTjEvO9yQksJj$z7E#rX~DLQv5u#y85lVx(u<^g?yVCxoGuha7N(q$mTj zb>olpp;j(0Ee}MP!>oj*X>7iR{6rIwpsn?7`R!Vtk!^S(Bw21S-1$XA`Q~FB_*(R- ztPF($ds7WG{G~7#;kBW#;`BSLpyzDk*NU^;AqbkT7dF-J!5&@k4R#H( z*B)EnvXoo#)17RTgs%i+R!-wN+qNiD=@~zfrQR+woO%;oK2){krz*xa$2(MQ<~6=e zyge}wIUU!Vc}q!INeD(Y0)du_v${`uX**Xgz`NH)YIHeoT-k;a9&^E4o(rHB!5odZ zCZ!W*m2VJNSp`M7`mhRCf15q` z^u-*~Ubwc}%vm2Ub*#9{%Slp+%8FB{1B2;NaJJ~8b=a##sgWa7P@}3-g+a!=EDkov zmiQM43M9mzydU_d!!Z zrVEY$dt_}!Na@R6t6bP|V>2PQf*g?MgoU^{2@VtM>%9|iw@Q%X>1}R945+?b2|+<# z2}n+o6^wTGrcrN97NfKTh&SkR{&L-LRA$&A9s}|64tx2qM)Wk+7MP|>mewZRE8OH;ELNm7Xsnac!87TSZRPLsf zg+^^kSSrp6w44;`6dG5ctF45x?XqgyX_dr;meZGyiW0VzdDIVYtwVJ+@-$!n07O~j zL%2K}{6Hxh#&-ck^Xv^uTDpEbYn;2a&N7@#rcm3>1eX)7CuNOXWO6-fBdD!*dpwY- zw#m;#xDeEI!fCOoTM!OYf%WIm&~2r36;1YMm-|)bEwSTgzx=t9qPOEZrMNWk0ZR5N z2Vi`swHrR-%+8vHXzF`Zcm7&39*=RhGQ2pBQWTVr5eJ@lJ?iF;qh?*%7W=R5II~fy zQQUdPmg3qfz|SM`4*kYQHO2IWsvUx2?Jc?#JQ#?|xnhSxURd0+cQax$VOVv^g*g8IWKDd4!2k+|F*af-5hKkF`~gK&vr=(&!w z4fQ}gn$adk}`@8!-m`cmy+Met#9$@4 zB#ilRdmm6w^{J`%yK@L)E0PdjeJWZTTZH(DNXaS2GC<@F!1k`1Oog#-GLokw_&_1V zge64esaWA&n-9H5O)aGjHubHiSxE?5Uv0G^dD!4DW9dOny)?`QWtkCMLyJgpjb%z& z4u1&)bq;a$s=rs<7beOjvh=|fw)G^XwXL@rBpqrA1zMB7K;E;GqbRRE9~}PxYAX*< z*uF1Zp!CmAHD!)jX~!cn-iE`f1gS|#*aNo7+OkaxN#oMB{{ZbB0^=XzOS0v@!|`Rq zW!P^rwJ8cf*x;+p&tftwCvY74*Ny68mRTtDcH&9F4W-{Mk|_>kf}%IB>ECMe#3eZU z*O`@W$;}}d8guDQtmJL^Qgvr@T?p6TA8O^B*~d7Z@{Ne!+P#CQC{uXz5$I^PThC)g z6K&BI3uS|*-%3`UTW?w%ILx}*9ce?{R%LDqXy+Ii+cjy0b#b<|>&6D#*Q@F0s%KU_ z8nWaRhJgxD`cRUtYiTO*$=i+Uz58k6cUHoX_04YWwm7b-WJyvT;UrD^LQbN*V~S#3 z=}$C5oF_f14qTP$!c7KN=?6QGcCIP6BSQ)L2D8+YXLzWzuw(dvy&>>)HV#UPkK$G^ za0WYOr@v=1rI9Imkl{K&KsX(Tpab=;w&ELF&3M|yz`WDT{qKo@ zlpuNv_L7S1qjHvzWR9A)&p?FiH8~h7P*&grw&UMF(v>@;*pSOnU_F4+w4{NoyYp$CNRj1O+~RA9KxsiY}MF6ALeIOo!xDE1?A8HsI`5tkV8guO%s z6yqr)vBxJo(9+WaEW)|7&|XkNv^j-iAA}#FB#Jf1B}8FJVL>`lMv<|}_okNtE-)NZ zZ8ml)2?J7z_4Tact42-U!R95`VQE_3eWffE4Py!*=Wq0;U^5g>b#BB_a|10cN?U1H z0FK!n)hJ=1wu5SYL4S!m4TpWZQzitj3D5VS2ILHGN%x`0$axX(E^JwXN}6BGIG}{3 zCt8Wd{P2*FHXX_KqLB5*VplB^-D3$M0mON|j{9##n`#t-+wtAXz;DMZ^J5$1(uR)0 znjwhtlC&^Y`1T~7vPtyCZFx2P49@BV+Z}1Eq-R}0bHhs72?;7YW7~pBG+36~>+w|6 zjHSXbLbUmPN%sE$N)gy|ZeA_EN63L7jSgU_j2z_Vj{$In6zGR~#W${{Vt1{-Bzo zx89V)VdJS@3mGK#$;R|VOZC~;K4LB9)sS#QfayPD>+MIP(bIEK1QqH~PIIqw>S{S>odidt?UcqHcl_5&nizG)>HCa4Ngr?IyMj#IzyjkKp$vEn+GvQ@fx8}c-P z^rJ)83r)FA+HF27!P2GaWhY+brzJ{jbWd3Wo(t7YW3(ej>`sXy-V-zz{*C0QdFPC@rM98;Emh|JxZ z(dSN4kaL@ae;zH4l|R!^`eV4t)N4TdLdrQB>9d{Vt4OQ&h%eO!j`#BMM-rB zDp(2yY8>OSr*_vsQe70wkL)SZN>MgLQ(jsS17xftCnIC@JNwXgc2#F@Lb`?M;l&cB z(v_?YKqM3Vpb|L8>slNb78Jaawp%ZS0EL~m9DQre+oIi^ek*d2zZ@rCmPT@&!QYde z{RRaUIYbR6R;(zLy$m&Tf5OqNZ#>ePDuTSuzW>&`{JyrjC}4k3C96QgVZ zK|ARi=Clv?W$4iS7tkGHr0GgRcE&MP3yYC0i*cmB+ersxsNf+Uhi|CG9#R6Q(1qXb zO{fOhjIsgMl^t%WSOob`+XK>?W2kPi(#!9%r<)Hg)hMk&PI+<=2LKOBIWu}%LyWf~ zeYhPsO0p6Zka7oXeWZfHvv&DJ0 zp`aAEB;##GCw|0iK8CL|SX9^00HSitJ4knM`WsGa-igv z*(eE8R)-VE@TDN~HpkMU-kg?nIO`H7A(W{_rm&Q_NhM)9!Oq;Bw#I5^Yf@7kGT6D0 zK-8AjfW1h@3C9V@-xV3u5~Rgy8+OQzsX-1rveHOEQb_WpImjF5(!HihLe+E)DMCmeSfr>Dn^w7S7vdpvb`Hq$8@+Jbdw;CQ@8R8*K>%NgyQWAaY0owZGgnyFGIWwMA*D z2vXDn8wm*jEBs2|IKesgpb)O8i2KUAp@}h^bm?0_0HlmzBmw22E;HqY!GUWNC+K`4 zKqx410HSyH8*`2*vi0?roVWhRbxg-}INDS<3mM5O2aU+bxb&rW;44dE4O!pti?ZP* z<&fjr7nEfj;NyS+7*$BfL{FD)Sy*#AT2o<>fsKIA{!#YcyKiaM8jTs|08E4^FF-hK zl$?MG-z4K1?M*(^`0<;zMRk-Z!+K*GR3n`e=Tu}`WT#(L6a6F^{Gmt$!sAyJkB6MZBJ>?b>w2dpme9F#9`c#Y1 zB!;r<wbra70>T)b6VlQ-OLEDFwKTcnG*={ z4vi~l05p#_*ubM0wkkB|B{v~@p9l{v3IVW$j3)r@HzXbC_TWY`o0A}=Dnd(P#VgEG za85wmw?AqrxqZOt6-F-Cq2ot3-`XgYgslunQeQ(zC+HGRaq4#VqHRO0Zr1r-KJ*}% zt&o?QX&@zp?4S}ewn6PvW2e&@9xb`ThEz)N_*+OPH~{WPYGTPQY)g!{ZMO*SA=idk zQZQAGv!Ay#A3M-j7QW>zuAS=Gw^MXt?7(3$$AtmFml`SubrNz;Kmg-$n!2-aQcF!q zf#Dz^Eoku2RLYb#8O91m<8Nw{op^yU#OPrm=!6oMl8|>J&Y$KVr!@h7yveeq7U(e_ zQ;!`%_fS-jNF?NgfO2@_w|XgY1-VjS9ZhMEcbb$vbf0Stv*D^kZK2XZ<5HA110!M0 z1#P#xZMB;)>fhQCwKX0)veMU-i~`q?1rIPjyB>Sgn-inBpO>EVD9T2sT`I$h0|1@& zBB(bxH&jfTfXhvmkRB3Z!`V%za8kSw4&)Gdthqjd)fyC5Au!rLPC^_FtxX>Z3Ri%z z0N*)YoyHHgC=R2Oc6L&tTkX;=_V!fTq$##^w(1Tt*&$k$zJG*c1DsT4t?S27!AmyQ zqgzZ>mmPTuSb4>T94iUi8)tu|NUF`$w#&oP?|n?ll_9jGhGGGh^>BS~PwHsXUrP}* z-H^AK9Za<{EapX~-s+aO5~zYg##8{(0ZB>JNx{x)JEv9EQsz9&`7$HNg(pNgYfgd2 z4l;441n1P8cB{Mh_)*e}EXZ@;EbAdnwpI|XkIf(E?H+gqKhx6gYLHEa2oRy@}r&?hQz)G$^W!#hAv21-zLI zw9-zMV`U?JkapPdiW2b#)RNL+Jh-WBmy*K|C@IC0xd4NYTyQDl7OueMszf`G`jY1v zHrqY>_GEtuw#xi;2?T(ZYC2M_N*Px9!PtydzTat^Z*nSbcR!i|r3RdF(}bx+5{)2| z6R?KWB*&)ZCb`K4NL{ye}evdA#G|Pz4av@wtPnN>0O02pqa}MYkHBMh9+c`AUuD(XB{r8lGRw_vfG~IdoYZxa z4bsnYo^1>@COIAvY3~smfvF=pcfs2{;L~jMZMsY8Ri5ftWy~c>)2S^aZ>2rK=LeDR zML|kpt~7Hx?ECQ+wV^UlQXbMgrALJUw>#q%R@GctHFvaeQ$)uS8jXiz5BJ8CDJx4Z z6!GD>HykG*6pNHAv$GhNZdhtiY^13Tv3r-8iKM(9E@ihAEiW_L>rT>%~O&jA~@a05TJ~xk_ZQJr(vEwYAP)I>{&6;&7;n+ zzZWiLB_t>Pa~q`{t(Rnxe6f z7EUV8)CV1DxTuo3ua^{B!Es7Q;oOms4y=whrYtfp5M?S`_Tv@FGCUVsX_S$tI~8)I z0CT?mxHaha#UeB27f!z&D|7}>$jX%2<$L4;JC69L?kS$t!W?->E+M?Fmk;rIch8~Y zYAJjQ_1Wv>x7~>v)Y}poU=Zu!%YL}-#mNe5DIrK-2U2*dGyM%3Y&A7%Y8u~l_T)CT z%34t2M*%?KYAG5Hz~ejmT&7$dacP8saZie%U1kjIGpcP$ zISL6<)O6q|9rT^)q?!U~i?m;^GTvMVIvCrZL#{fh3`}Wp(#}Xi0NH14lt=^-+-_&A zoh3F)?xN{@THRssDl(h_N@RkqBLti%B`vkVa_W;-lS=?zeykU=1dw#GykT~lS1xY9t% zin2#xiheX0@rqLgT3b>{DM|p!yOZgk>s=l6LhG~TFYQh6&#W%;&s5tjZO?}RhaQyk zLl3Zke8WjPl-SOD>EE{X^2*VXo@?|=s+PsX4!yt7O;|82< zU?&?;Za=4DcFt>veObpf$&v>=kaI&*n~wD@8aOy43=Y*94=PZ=sm2J>J66RCV`IjL z+CVD61J;Vpl;bqhL(7n9(ZBHxtI&p#+X!lpC2B~d1t_bYX~zI4+L>$>L zskqSGg1|x$R6y@Z64cVN>pN$9_Y{6kPEWxO*>W;lN~5GD0|O-UM+*a(U#psb#h2sN$Xiu7XxW!J5OCxYSXp)sA zsE3XR2^1UakjhG$Z^R>f8XYru7#F#%AQ>$jr|=^r@7pzBUA}yo#!QQwr^HcFYzzPo zYLgc!0PB>}QgfVQjcP)k{rZ{+N$-wmxG5@-RNP&ml)Cv~#%VbyBP;$OJ*b#79F;XT z#JsXqzL0hwS{TS#dBO<7h5!`&M$*Uo(>hS_{vbCrwAVq?J$UGL9+assIHx?M6zaeO z+*MBOu9&mJlM+)ThZT~Blag^(ONIL8j}u8J#2621(3aosL57%cEjMrYy<<1KvbokL|BZi zu;-c0@K1VnlXVhq(zXdUJLrC0S})C%JxLE|7%P9gW#hn~x4k&0xrBYPq)3vyG9Or0rZEu?5{55x7Z zPJTMrN|hyODf|GOfDbh|?XJ}GJ{m@q5HfSkN8CDEG?9S4cpE1>QB8i9X4Biz32t&5 zPb4!dTFE4Mz#aQyoQz=Gt}Z(+VUV>eDo#cQPaAfoWOP!N>R)K7ryDp<4&Rj&&qluF zZxL^xyU94_mBQ5_o<(+DVqThiN(0CV=NYIjw~cJNDmwx*YI)^jP#=VPnutY((J`=O zl#no@I|?H0X?i;a__CB#k({fVdXrs~`ct%ub*(Oquc5T%RagD`49_Ky65ajnkR@!xQLf%xS9ZH@GDkG40syHt!%2VjJLAS!T zKuj2jM}Uph}uqb-$dkh0^cSC|E3x1gZl#!^YpY&iE9BDz$| zno80V6p}~yjY%;H^{EWEAHz|=1+;0@alX{VFNqdQgfyZv=|-{wytpbk;;iwl@-8T5 zY7}yv(2%SEDR{V#?q6kTrM(8=q$T%KmldIB83z?54m^fuxiOhV#;~l32W65!7CWBQ zW#TjJge|WDT27!Ddq~&5D)F5GHx>v>5kGgsd~~KR_Y#=Hkb*1Dpe8Cf>I6?ov~F* zoV&%kYNJMm+R{IZj(^g!sO_0PT12#HvXTjQC}jmo-2;8;O1E3tOV3P>9L^k0dCoW{ zg|^&fy2xf@psAs;Pj!3!sn>1}w;jhGM~*RMZbDZ;-oCxqj*J+C5u!iQAa8Ik)wSAZ0j$U}&48c8|9^%*>7Gq7TKu9^%SK5Vpq#9y6P#;kNc{n*3tvJSU1Z3kbh-U?= zYhD@=t>*zz+sKjhG?}*ub1m*$V<^K>K44lX4gfKv_NlpjDxlI(!cLMis2mYOLFsup zLlTv!DB!EmR;YFxZ%mcwtId`Ly)fi>4aoyju7v(9_RqCUyVPAa^SWPbD3gTc1A46v zl7NX1u0SOT7)T1iKK1KMeJqJ34>IX-uym7?y=igq1e@&%yWLv(sV+r;rAH{#*eJ+S z`wvm|rmc4OWwWnBV93;!tteKu)~tRU`|xPV4gBNmn2ZFaCkjbC`;U5-W&~KsQlt`e z`GP>(H7eNmNQ1gsX16HIi7iQ55}dc;qX3S5>OWI$rKc{ef})@Uji>O8{t$f)6@8iW zPOZqWB&p4b$RQ^KZRj>FcI&Hg+A`!YwK=J5g`8>~y{J7s47%BAdD!|A@*Yx@?vQ-g zC`sSSMG16%)Z6jdFw*O*)ZsbFPuJYgeMIHPD-6Z9+w~BfI^wv-Hak=dAKPVX(7kRs zk1<9ORbruC_Nmj$5z20?MW#w{{ST;jn>ByxVGj}(u*y$ zP7o30!1wo|ZfzMMiBY9K`n4z_q>na$GH`uPDT~}>+?-`@L`Q&=k`ws7shb_K#?>G( zA>hhRwzV&G9^iXYa#If;9!YCN%RQz~&!PgN>p*2*Fw}F%KGb6RvAd8=1Vc-W$Wp>y zX$J#I-%;(J)B>P2p~ob;>w^HODCaxWi;K0zjM7_dG?i{bM{lKCH_<2ZD%RDRFTZX~ z2G^taiv>El?~~0#KW%9;*CNisQtPCnLh>3`HyIwHr(wBnN!HMD)e-zX^IQvn+=f|w zm9nKP0Vn#=tx^}wkv9m}$&E&C5sTuof=aRgKj9yKCMt&>SgGky`SS zpP=L3qpsIkv2F>QaDAmjcm~8kfC@ny9QssM-s>{ZhZ>UVFb-Ql-9S_X%k7foi4se$ zx`d2?qB4*-7@@YbGF7%+=HKGPf90Bt;uKN?j-^A(>ACi$T`m)%MO51qS0;SQ0+r&s z`@gCG0JUeAA#Cc1iD9SZ#f;hW1T2D~{HMMuwIoiky7jq>thS&?1R(+9cRjtUe&MBB zHr|=I&zm7hmda4#8&8O>I0W_y&m$C+N}F?@^|_ZCEAc$Z5y(cLbHP61unAXgyXhAp zT&?pJAv(iu1v*u&8QUP#>;C`}di@A)Q)@&xs&Hybhwn4ok5N=$*N^VmjC&ONq z1N^|$bO>EV@a+t#cP|~4CqjWonS1h2uJrbhotBQT)76$mTr945&mjs4XsD#2c*b+= zM}9%C&a0x`jc}67>C)qGEh|XdN`j6?KDGKiW$FtYxc>k)&X}lH6r`Z;Ro`y)@E^w* z@gaOs>IR?@^6QkR5rKi`+~8LauHH2M27{|9tCBe1Gz;(JB<8wTf%QJLE5}*;*O1d? z5^`fMKdoEs=})zdhS@YPhM=qfcRZSXuEQJsDmf!TaqvvRR*{^Fm1X?nd}5%OBmqYu z>;OqQ_ol}Nwo(zIWw|4a?N5gQ?@;YLLBXQK!%sP@ts?87)jl!K1u)#82?rczgj?Cj z-v?~d0q~QJTl!F)Bu05SlFE^*kN*HQH{%kWhQ@&sPIo() folder. +The examples in this folder are built while building the MXNet library and cpp-package from source . However, they can be built manually as follows From cpp-package/examples directory @@ -18,24 +19,24 @@ The examples that are built to be run on GPU may not work on the non-GPU machine The makefile will also download the necessary data files and store in a data folder. (The download will take couple of minutes, but will be done only once on a fresh installation.) -## Examples +## Examples demonstrating training workflow -This directory contains following examples. In order to run the examples, ensure that the path to the MXNet shared library is added to the OS specific environment variable viz. **LD\_LIBRARY\_PATH** for Linux, Mac and Ubuntu OS and **PATH** for Windows OS. +This directory contains following examples. In order to run the examples, ensure that the path to the MXNet shared library is added to the OS specific environment variable viz. **LD\_LIBRARY\_PATH** for Linux, Mac and Ubuntu OS and **PATH** for Windows OS. For example `export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/ubuntu/incubator-mxnet/lib` on ubuntu using gpu. ### [alexnet.cpp]() The example implements the C++ version of AlexNet. The networks trains on MNIST data. The number of epochs can be specified as a command line argument. For example to train with 10 epochs use the following: - ``` - ./alexnet 10 - ``` +``` +build/alexnet 10 +``` ### [googlenet.cpp]() The code implements a GoogLeNet/Inception network using the C++ API. The example uses MNIST data to train the network. By default, the example trains the model for 100 epochs. The number of epochs can also be specified in the command line. For example, to train the model for 10 epochs use the following: ``` -./googlenet 10 +build/googlenet 10 ``` ### [mlp.cpp]() @@ -44,7 +45,7 @@ The code implements a multilayer perceptron from scratch. The example creates it To run the example use the following command: ``` -./mlp +build/mlp ``` ### [mlp_cpu.cpp]() @@ -53,7 +54,7 @@ The code implements a multilayer perceptron to train the MNIST data. The code de To run the example use the following command: ``` -./mlp_cpu +build/mlp_cpu ``` ### [mlp_gpu.cpp]() @@ -61,7 +62,7 @@ To run the example use the following command: The code implements a multilayer perceptron to train the MNIST data. The code demonstrates the use of the "SimpleBind" C++ API and MNISTIter. The example is designed to work on GPU. The example does not require command line arguments. To run the example execute following command: ``` -./mlp_gpu +build/mlp_gpu ``` ### [mlp_csv.cpp]() @@ -69,7 +70,13 @@ The code implements a multilayer perceptron to train the MNIST data. The code de The code implements a multilayer perceptron to train the MNIST data. The code demonstrates the use of the "SimpleBind" C++ API and CSVIter. The CSVIter can iterate data that is in CSV format. The example can be run on CPU or GPU. The example usage is as follows: ``` -mlp_csv --train mnist_training_set.csv --test mnist_test_set.csv --epochs 10 --batch_size 100 --hidden_units "128,64,64 [--gpu]" +build/mlp_csv --train data/mnist_data/mnist_train.csv --test data/mnist_data/mnist_test.csv --epochs 10 --batch_size 100 --hidden_units "128 64 64" --gpu +``` +* To get the `mnist_training_set.csv` and `mnist_test_set.csv` please run the following command: +```python +# in incubator-mxnet/cpp-package/example directory +python mnist_to_csv.py ./data/mnist_data/train-images-idx3-ubyte ./data/mnist_data/train-labels-idx1-ubyte ./data/mnist_data/mnist_train.csv 60000 +python mnist_to_csv.py ./data/mnist_data/t10k-images-idx3-ubyte ./data/mnist_data/t10k-labels-idx1-ubyte ./data/mnist_data/mnist_test.csv 10000 ``` ### [resnet.cpp]() @@ -77,7 +84,7 @@ mlp_csv --train mnist_training_set.csv --test mnist_test_set.csv --epochs 10 --b The code implements a resnet model using the C++ API. The model is used to train MNIST data. The number of epochs for training the model can be specified on the command line. By default, model is trained for 100 epochs. For example, to train with 10 epochs use the following command: ``` -./resnet 10 +build/resnet 10 ``` ### [lenet.cpp]() @@ -85,22 +92,22 @@ The code implements a resnet model using the C++ API. The model is used to train The code implements a lenet model using the C++ API. It uses MNIST training data in CSV format to train the network. The example does not use built-in CSVIter to read the data from CSV file. The number of epochs can be specified on the command line. By default, the mode is trained for 100,000 epochs. For example, to train with 10 epochs use the following command: ``` -./lenet 10 +build/lenet 10 ``` ### [lenet\_with\_mxdataiter.cpp]() The code implements a lenet model using the C++ API. It uses MNIST training data to train the network. The example uses built-in MNISTIter to read the data. The number of epochs can be specified on the command line. By default, the mode is trained for 100 epochs. For example, to train with 10 epochs use the following command: ``` -./lenet\_with\_mxdataiter 10 +build/lenet_with_mxdataiter 10 ``` In addition, there is `run_lenet_with_mxdataiter.sh` that downloads the mnist data and run `lenet_with_mxdataiter` example. -###[inception_bn.cpp]() +### [inception_bn.cpp]() The code implements an Inception network using the C++ API with batch normalization. The example uses MNIST data to train the network. The model trains for 100 epochs. The example can be run by executing the following command: ``` -./inception_bn +build/inception_bn ``` diff --git a/cpp-package/example/alexnet.cpp b/cpp-package/example/alexnet.cpp index 3d6e6855b..7564d4361 100644 --- a/cpp-package/example/alexnet.cpp +++ b/cpp-package/example/alexnet.cpp @@ -234,15 +234,6 @@ int main(int argc, char const *argv[]) { * initializer to call*/ xavier(arg.first, &arg.second); } - /*print out to check the shape of the net*/ - for (const auto &s : Net.ListArguments()) { - LG << s; - const auto &k = args_map[s].GetShape(); - for (const auto &i : k) { - std::cout << i << " "; - } - std::cout << std::endl; - } /*these binary files should be generated using im2rc tools, which can be found * in mxnet/bin*/ @@ -258,7 +249,7 @@ int main(int argc, char const *argv[]) { auto val_iter = MXDataIter("MNISTIter"); setDataIter(&val_iter, "Label", data_files, batch_size); - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("momentum", 0.9) ->SetParam("rescale_grad", 1.0 / batch_size) ->SetParam("clip_gradient", 10) @@ -275,7 +266,6 @@ int main(int argc, char const *argv[]) { train_iter.Reset(); while (train_iter.Next()) { auto batch = train_iter.GetDataBatch(); - LG << train_iter.GetDataBatch().index.size(); /*use copyto to feed new data and label to the executor*/ batch.data.CopyTo(&args_map["data"]); batch.label.CopyTo(&args_map["label"]); diff --git a/cpp-package/example/charRNN.cpp b/cpp-package/example/charRNN.cpp index ad564f665..54b8eea7a 100644 --- a/cpp-package/example/charRNN.cpp +++ b/cpp-package/example/charRNN.cpp @@ -465,7 +465,7 @@ void train(const std::string file, int batch_size, int max_epoch, int start_epoc mx_float learning_rate = 0.0002; mx_float weight_decay = 0.000002; - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("lr", learning_rate) ->SetParam("wd", weight_decay); // opt->SetParam("momentum", 0.9)->SetParam("rescale_grad", 1.0 / batch_size) diff --git a/cpp-package/example/example.mk b/cpp-package/example/example.mk index 4914b31ba..ef99d7426 100644 --- a/cpp-package/example/example.mk +++ b/cpp-package/example/example.mk @@ -18,7 +18,7 @@ CPPEX_SRC = $(wildcard cpp-package/example/*.cpp) CPPEX_EXE = $(patsubst cpp-package/example/%.cpp, build/cpp-package/example/%, $(CPPEX_SRC)) -CPPEX_CFLAGS += -Icpp-package/include -Ibuild/cpp-package/include +CPPEX_CFLAGS += -Icpp-package/include CPPEX_EXTRA_LDFLAGS := -L$(ROOTDIR)/lib -lmxnet EXTRA_PACKAGES += cpp-package-example-all @@ -30,8 +30,8 @@ cpp-package-example-all: cpp-package-all $(CPPEX_EXE) build/cpp-package/example/% : cpp-package/example/%.cpp lib/libmxnet.so $(CPP_PACKAGE_OP_H_FILE) @mkdir -p $(@D) - $(CXX) -std=c++0x $(CFLAGS) $(CPPEX_CFLAGS) -MM -MT cpp-package/example/$* $< >build/cpp-package/example//$*.d - $(CXX) -std=c++0x $(CFLAGS) $(CPPEX_CFLAGS) -o $@ $(filter %.cpp %.a, $^) $(LDFLAGS) $(CPPEX_EXTRA_LDFLAGS) + $(CXX) -std=c++11 $(CFLAGS) $(CPPEX_CFLAGS) -MM -MT cpp-package/example/$* $< >build/cpp-package/example//$*.d + $(CXX) -std=c++11 $(CFLAGS) $(CPPEX_CFLAGS) -o $@ $(filter %.cpp %.a, $^) $(LDFLAGS) $(CPPEX_EXTRA_LDFLAGS) cpp-package-example-clean: rm -rf build/cpp-package/example/* diff --git a/cpp-package/example/feature_extract/Makefile b/cpp-package/example/feature_extract/Makefile index f598183bd..193eaa7e8 100644 --- a/cpp-package/example/feature_extract/Makefile +++ b/cpp-package/example/feature_extract/Makefile @@ -27,12 +27,12 @@ LDFLAGS=$(COMMFLAGS) -L ../../../lib -lmxnet $(BLAS) $(CUDA) -lgomp -pthread all: feature_extract prepare_data_with_opencv feature_extract: ./feature_extract.cpp - $(CXX) -c -std=c++0x $(CFLAGS) $^ + $(CXX) -c -std=c++11 $(CFLAGS) $^ $(CXX) $(basename $@).o -o $@ $(LDFLAGS) -rm -f $(basename $@).o prepare_data_with_opencv: ./prepare_data_with_opencv.cpp - $(CXX) -c -std=c++0x $(OPENCV_CFLAGS) $^ + $(CXX) -c -std=c++11 $(OPENCV_CFLAGS) $^ $(CXX) $(basename $@).o -o $@ $(OPENCV_LDFLAGS) -rm -f $(basename $@).o diff --git a/cpp-package/example/googlenet.cpp b/cpp-package/example/googlenet.cpp index ad9212c75..4bd3be27f 100644 --- a/cpp-package/example/googlenet.cpp +++ b/cpp-package/example/googlenet.cpp @@ -144,7 +144,7 @@ int main(int argc, char const *argv[]) { auto val_iter = MXDataIter("MNISTIter"); setDataIter(&val_iter, "Label", data_files, batch_size); - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("momentum", 0.9) ->SetParam("rescale_grad", 1.0 / batch_size) ->SetParam("clip_gradient", 10) diff --git a/cpp-package/example/inception_bn.cpp b/cpp-package/example/inception_bn.cpp index c499df77e..5b444e467 100644 --- a/cpp-package/example/inception_bn.cpp +++ b/cpp-package/example/inception_bn.cpp @@ -91,7 +91,8 @@ Symbol InceptionFactoryB(Symbol data, int num_3x3red, int num_3x3, Shape(1, 1), name + "_double_3x3_1"); Symbol pooling = Pooling("max_pool_" + name + "_pool", data, Shape(3, 3), PoolingPoolType::kMax, - false, false, PoolingPoolingConvention::kValid, Shape(2, 2)); + false, false, PoolingPoolingConvention::kValid, + Shape(2, 2), Shape(1, 1)); std::vector lst; lst.push_back(c3x3); lst.push_back(cd3x3); @@ -143,8 +144,8 @@ Symbol InceptionSymbol(int num_classes) { int main(int argc, char const *argv[]) { int batch_size = 40; - int max_epoch = 100; - float learning_rate = 1e-4; + int max_epoch = argc > 1 ? strtol(argv[1], NULL, 10) : 100; + float learning_rate = 1e-2; float weight_decay = 1e-4; auto ctx = Context::gpu(); @@ -172,7 +173,13 @@ int main(int argc, char const *argv[]) { auto val_iter = MXDataIter("MNISTIter"); setDataIter(&val_iter, "Label", data_files, batch_size); - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + // initialize parameters + Xavier xavier = Xavier(Xavier::gaussian, Xavier::in, 2); + for (auto &arg : args_map) { + xavier(arg.first, &arg.second); + } + + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("momentum", 0.9) ->SetParam("rescale_grad", 1.0 / batch_size) ->SetParam("clip_gradient", 10) @@ -182,9 +189,12 @@ int main(int argc, char const *argv[]) { auto *exec = inception_bn_net.SimpleBind(ctx, args_map); auto arg_names = inception_bn_net.ListArguments(); + // Create metrics + Accuracy train_acc, val_acc; for (int iter = 0; iter < max_epoch; ++iter) { LG << "Epoch: " << iter; train_iter.Reset(); + train_acc.Reset(); while (train_iter.Next()) { auto data_batch = train_iter.GetDataBatch(); data_batch.data.CopyTo(&args_map["data"]); @@ -200,10 +210,11 @@ int main(int argc, char const *argv[]) { } NDArray::WaitAll(); + train_acc.Update(data_batch.label, exec->outputs[0]); } - Accuracy acu; val_iter.Reset(); + val_acc.Reset(); while (val_iter.Next()) { auto data_batch = val_iter.GetDataBatch(); data_batch.data.CopyTo(&args_map["data"]); @@ -211,9 +222,10 @@ int main(int argc, char const *argv[]) { NDArray::WaitAll(); exec->Forward(false); NDArray::WaitAll(); - acu.Update(data_batch.label, exec->outputs[0]); + val_acc.Update(data_batch.label, exec->outputs[0]); } - LG << "Accuracy: " << acu.Get(); + LG << "Train Accuracy: " << train_acc.Get(); + LG << "Validation Accuracy: " << val_acc.Get(); } delete exec; MXNotifyShutdown(); diff --git a/cpp-package/example/inference/Makefile b/cpp-package/example/inference/Makefile new file mode 100644 index 000000000..5efe6cfb6 --- /dev/null +++ b/cpp-package/example/inference/Makefile @@ -0,0 +1,40 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +CPPEX_SRC = $(wildcard *.cpp) +CPPEX_EXE = $(patsubst %.cpp, %, $(CPPEX_SRC)) +OPENCV_CFLAGS=`pkg-config --cflags opencv` +OPENCV_LDFLAGS=`pkg-config --libs opencv` + +CXX=g++ + + +CFLAGS=$(COMMFLAGS) -I../../../3rdparty/tvm/nnvm/include -I../../../3rdparty/dmlc-core/include -I ../../include -I ../../../include -Wall -O3 -msse3 -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas +CPPEX_EXTRA_LDFLAGS := -L../../../lib -lmxnet $(OPENCV_LDFLAGS) + +all: $(CPPEX_EXE) + +debug: CPPEX_CFLAGS += -DDEBUG -g +debug: all + + +$(CPPEX_EXE):% : %.cpp + $(CXX) -std=c++0x $(CFLAGS) $(CPPEX_CFLAGS) -o $@ $(filter %.cpp %.a, $^) $(CPPEX_EXTRA_LDFLAGS) + +clean: + rm -f $(CPPEX_EXE) diff --git a/cpp-package/example/inference/README.md b/cpp-package/example/inference/README.md new file mode 100644 index 000000000..79831b40b --- /dev/null +++ b/cpp-package/example/inference/README.md @@ -0,0 +1,41 @@ +# MXNet C++ Package Inference Workflow Examples + +## Building C++ Inference examples + +The examples in this folder demonstrate the **inference** workflow. +To build examples use following commands: + +- Release: **make all** +- Debug: **make debug all** + + +## Examples demonstrating inference workflow + +This directory contains following examples. In order to run the examples, ensure that the path to the MXNet shared library is added to the OS specific environment variable viz. **LD\_LIBRARY\_PATH** for Linux, Mac and Ubuntu OS and **PATH** for Windows OS. + +### [inception_inference.cpp]() + +This example demonstrates image classification workflow with pre-trained models using MXNet C++ API. The command line parameters the example can accept are as shown below: + +``` +./inception_inference --help +Usage: +inception_inference --symbol + --params + --image ) downloads the pre-trained **Inception** model and a test image. The users can invoke this script as follows: + +``` +./unit_test_inception_inference.sh +``` diff --git a/cpp-package/example/inference/inception_inference.cpp b/cpp-package/example/inference/inception_inference.cpp new file mode 100644 index 000000000..7005e745b --- /dev/null +++ b/cpp-package/example/inference/inception_inference.cpp @@ -0,0 +1,446 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* + * This example demonstrates image classification workflow with pre-trained models using MXNet C++ API. + * The example performs following tasks. + * 1. Load the pre-trained model. + * 2. Load the parameters of pre-trained model. + * 3. Load the image to be classified in to NDArray. + * 4. Normalize the image using the mean of images that were used for training. + * 5. Run the forward pass and predict the input image. + */ + +#include +#include +#include +#include +#include +#include +#include "mxnet-cpp/MxNetCpp.h" +#include + +using namespace mxnet::cpp; + +static mx_float DEFAULT_MEAN_R = 123.675; +static mx_float DEFAULT_MEAN_G = 116.28; +static mx_float DEFAULT_MEAN_B = 103.53; +/* + * class Predictor + * + * This class encapsulates the functionality to load the model, process input image and run the forward pass. + */ + +class Predictor { + public: + Predictor() {} + Predictor(const std::string& model_json_file, + const std::string& model_params_file, + const Shape& input_shape, + bool gpu_context_type = false, + const std::string& synset_file = "", + const std::string& mean_image_file = ""); + void PredictImage(const std::string& image_file); + ~Predictor(); + + private: + void LoadModel(const std::string& model_json_file); + void LoadParameters(const std::string& model_parameters_file); + void LoadSynset(const std::string& synset_file); + NDArray LoadInputImage(const std::string& image_file); + void LoadMeanImageData(); + void LoadDefaultMeanImageData(); + void NormalizeInput(const std::string& mean_image_file); + inline bool FileExists(const std::string& name) { + struct stat buffer; + return (stat(name.c_str(), &buffer) == 0); + } + NDArray mean_img; + std::map args_map; + std::map aux_map; + std::vector output_labels; + Symbol net; + Executor *executor; + Shape input_shape; + NDArray mean_image_data; + NDArray std_dev_image_data; + Context global_ctx = Context::cpu(); + std::string mean_image_file; +}; + + +/* + * The constructor takes following parameters as input: + * 1. model_json_file: The model in json formatted file. + * 2. model_params_file: File containing model parameters + * 3. synset_file: File containing the list of image labels + * 4. input_shape: Shape of input data to the model. Since this class will be running one inference at a time, + * the input shape is required to be in format Shape(1, number_of_channels, height, width) + * The input image will be resized to (height x width) size before running the inference. + * The constructor will: + * 1. Load the model and parameter files. + * 2. Load the synset file. + * 3. Invoke the SimpleBind to bind the input argument to the model and create an executor. + * + * The SimpleBind is expected to be invoked only once. + */ +Predictor::Predictor(const std::string& model_json_file, + const std::string& model_params_file, + const Shape& input_shape, + bool gpu_context_type, + const std::string& synset_file, + const std::string& mean_image_file): + input_shape(input_shape), + mean_image_file(mean_image_file) { + if (gpu_context_type) { + global_ctx = Context::gpu(); + } + // Load the model + LoadModel(model_json_file); + + // Load the model parameters. + LoadParameters(model_params_file); + + /* + * The data will be used to output the exact label that matches highest output of the model. + */ + LoadSynset(synset_file); + + /* + * Load the mean image data if specified. + */ + if (!mean_image_file.empty()) { + LoadMeanImageData(); + } else { + LG << "Mean image file for normalizing the input is not provide." + << " We will use the default mean values for R,G and B channels."; + LoadDefaultMeanImageData(); + } + + // Create an executor after binding the model to input parameters. + args_map["data"] = NDArray(input_shape, global_ctx, false); + executor = net.SimpleBind(global_ctx, args_map, std::map(), + std::map(), aux_map); +} + +/* + * The following function loads the model from json file. + */ +void Predictor::LoadModel(const std::string& model_json_file) { + if (!FileExists(model_json_file)) { + LG << "Model file " << model_json_file << " does not exist"; + throw std::runtime_error("Model file does not exist"); + } + LG << "Loading the model from " << model_json_file << std::endl; + net = Symbol::Load(model_json_file); +} + + +/* + * The following function loads the model parameters. + */ +void Predictor::LoadParameters(const std::string& model_parameters_file) { + if (!FileExists(model_parameters_file)) { + LG << "Parameter file " << model_parameters_file << " does not exist"; + throw std::runtime_error("Model parameters does not exist"); + } + LG << "Loading the model parameters from " << model_parameters_file << std::endl; + std::map parameters; + NDArray::Load(model_parameters_file, 0, ¶meters); + for (const auto &k : parameters) { + if (k.first.substr(0, 4) == "aux:") { + auto name = k.first.substr(4, k.first.size() - 4); + aux_map[name] = k.second.Copy(global_ctx); + } + if (k.first.substr(0, 4) == "arg:") { + auto name = k.first.substr(4, k.first.size() - 4); + args_map[name] = k.second.Copy(global_ctx); + } + } + /*WaitAll is need when we copy data between GPU and the main memory*/ + NDArray::WaitAll(); +} + + +/* + * The following function loads the synset file. + * This information will be used later to report the label of input image. + */ +void Predictor::LoadSynset(const std::string& synset_file) { + if (!FileExists(synset_file)) { + LG << "Synset file " << synset_file << " does not exist"; + throw std::runtime_error("Synset file does not exist"); + } + LG << "Loading the synset file."; + std::ifstream fi(synset_file.c_str()); + if (!fi.is_open()) { + std::cerr << "Error opening synset file " << synset_file << std::endl; + throw std::runtime_error("Error in opening the synset file."); + } + std::string synset, lemma; + while (fi >> synset) { + getline(fi, lemma); + output_labels.push_back(lemma); + } + fi.close(); +} + + +/* + * The following function loads the mean data from mean image file. + * This data will be used for normalizing the image before running the forward + * pass. + * The output data has the same shape as that of the input image data. + */ +void Predictor::LoadMeanImageData() { + LG << "Load the mean image data that will be used to normalize " + << "the image before running forward pass."; + mean_image_data = NDArray(input_shape, global_ctx, false); + mean_image_data.SyncCopyFromCPU( + NDArray::LoadToMap(mean_image_file)["mean_img"].GetData(), + input_shape.Size()); + NDArray::WaitAll(); +} + + +/* + * The following function loads the default mean values for + * R, G and B channels into NDArray that has the same shape as that of + * input image. + */ +void Predictor::LoadDefaultMeanImageData() { + LG << "Loading the default mean image data"; + std::vector array; + /*resize pictures to (224, 224) according to the pretrained model*/ + int height = input_shape[2]; + int width = input_shape[3]; + int channels = input_shape[1]; + std::vector default_means; + default_means.push_back(DEFAULT_MEAN_R); + default_means.push_back(DEFAULT_MEAN_G); + default_means.push_back(DEFAULT_MEAN_B); + for (int c = 0; c < channels; ++c) { + for (int i = 0; i < height; ++i) { + for (int j = 0; j < width; ++j) { + array.push_back(default_means[c]); + } + } + } + mean_image_data = NDArray(input_shape, global_ctx, false); + mean_image_data.SyncCopyFromCPU(array.data(), input_shape.Size()); + NDArray::WaitAll(); +} + + +/* + * The following function loads the input image into NDArray. + */ +NDArray Predictor::LoadInputImage(const std::string& image_file) { + if (!FileExists(image_file)) { + LG << "Image file " << image_file << " does not exist"; + throw std::runtime_error("Image file does not exist"); + } + LG << "Loading the image " << image_file << std::endl; + std::vector array; + cv::Mat mat = cv::imread(image_file); + /*resize pictures to (224, 224) according to the pretrained model*/ + int height = input_shape[2]; + int width = input_shape[3]; + int channels = input_shape[1]; + cv::resize(mat, mat, cv::Size(height, width)); + for (int c = 0; c < channels; ++c) { + for (int i = 0; i < height; ++i) { + for (int j = 0; j < width; ++j) { + array.push_back(static_cast(mat.data[(i * height + j) * 3 + c])); + } + } + } + NDArray image_data = NDArray(input_shape, global_ctx, false); + image_data.SyncCopyFromCPU(array.data(), input_shape.Size()); + NDArray::WaitAll(); + return image_data; +} + + +/* + * The following function runs the forward pass on the model. + * The executor is created in the constructor. + * + */ +void Predictor::PredictImage(const std::string& image_file) { + // Load the input image + NDArray image_data = LoadInputImage(image_file); + + // Normalize the image + image_data.Slice(0, 1) -= mean_image_data; + + LG << "Running the forward pass on model to predict the image"; + /* + * The executor->arg_arrays represent the arguments to the model. + * + * Copying the image_data that contains the NDArray of input image + * to the arg map of the executor. The input is stored with the key "data" in the map. + * + */ + image_data.CopyTo(&(executor->arg_dict()["data"])); + NDArray::WaitAll(); + + // Run the forward pass. + executor->Forward(false); + + // The output is available in executor->outputs. + auto array = executor->outputs[0].Copy(global_ctx); + NDArray::WaitAll(); + + /* + * Find out the maximum accuracy and the index associated with that accuracy. + * This is done by using the argmax operator on NDArray. + */ + auto predicted = array.ArgmaxChannel(); + NDArray::WaitAll(); + + int best_idx = predicted.At(0, 0); + float best_accuracy = array.At(0, best_idx); + + if (output_labels.empty()) { + LG << "The model predicts the highest accuracy of " << best_accuracy << " at index " + << best_idx; + } else { + LG << "The model predicts the input image to be a [" << output_labels[best_idx] + << " ] with Accuracy = " << best_accuracy << std::endl; + } +} + + +Predictor::~Predictor() { + if (executor) { + delete executor; + } + MXNotifyShutdown(); +} + + +/* + * Convert the input string of number of hidden units into the vector of integers. + */ +std::vector getShapeDimensions(const std::string& hidden_units_string) { + std::vector dimensions; + char *p_next; + int num_unit = strtol(hidden_units_string.c_str(), &p_next, 10); + dimensions.push_back(num_unit); + while (*p_next) { + num_unit = strtol(p_next, &p_next, 10); + dimensions.push_back(num_unit); + } + return dimensions; +} + +void printUsage() { + std::cout << "Usage:" << std::endl; + std::cout << "inception_inference --symbol " << std::endl + << "--params " << std::endl + << "--image " << std::endl + << "--synset " << std::endl + << "[--input_shape ] " << std::endl + << "[--mean ] " + << std::endl + << "[--gpu ]" + << std::endl; +} + +int main(int argc, char** argv) { + std::string model_file_json; + std::string model_file_params; + std::string synset_file = ""; + std::string mean_image = ""; + std::string input_image = ""; + bool gpu_context_type = false; + + std::string input_shape = "3 224 224"; + int index = 1; + while (index < argc) { + if (strcmp("--symbol", argv[index]) == 0) { + index++; + model_file_json = (index < argc ? argv[index]:""); + } else if (strcmp("--params", argv[index]) == 0) { + index++; + model_file_params = (index < argc ? argv[index]:""); + } else if (strcmp("--synset", argv[index]) == 0) { + index++; + synset_file = (index < argc ? argv[index]:""); + } else if (strcmp("--mean", argv[index]) == 0) { + index++; + mean_image = (index < argc ? argv[index]:""); + } else if (strcmp("--image", argv[index]) == 0) { + index++; + input_image = (index < argc ? argv[index]:""); + } else if (strcmp("--input_shape", argv[index]) == 0) { + index++; + input_shape = (index < argc ? argv[index]:input_shape); + } else if (strcmp("--gpu", argv[index]) == 0) { + gpu_context_type = true; + } else if (strcmp("--help", argv[index]) == 0) { + printUsage(); + return 0; + } + index++; + } + + if (model_file_json.empty() || model_file_params.empty() || synset_file.empty()) { + LG << "ERROR: Model details such as symbol, param and/or synset files are not specified"; + printUsage(); + return 1; + } + + if (input_image.empty()) { + LG << "ERROR: Path to the input image is not specified."; + printUsage(); + return 1; + } + + std::vector input_dimensions = getShapeDimensions(input_shape); + + /* + * Since we are running inference for 1 image, add 1 to the input_dimensions so that + * the shape of input data for the model will be + * {no. of images, channels, height, width} + */ + input_dimensions.insert(input_dimensions.begin(), 1); + + Shape input_data_shape(input_dimensions); + + try { + // Initialize the predictor object + Predictor predict(model_file_json, model_file_params, input_data_shape, gpu_context_type, + synset_file, mean_image); + + // Run the forward pass to predict the image. + predict.PredictImage(input_image); + } catch (std::runtime_error &error) { + LG << "Execution failed with ERROR: " << error.what(); + } catch (...) { + /* + * If underlying MXNet code has thrown an exception the error message is + * accessible through MXGetLastError() function. + */ + LG << "Execution failed with following MXNet error"; + LG << MXGetLastError(); + } + return 0; +} diff --git a/cpp-package/example/inference/unit_test_inception_inference.sh b/cpp-package/example/inference/unit_test_inception_inference.sh new file mode 100755 index 000000000..4f40b496b --- /dev/null +++ b/cpp-package/example/inference/unit_test_inception_inference.sh @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Downloading the data and model +mkdir -p model +wget -nc http://data.dmlc.ml/mxnet/models/imagenet/inception-bn.tar.gz +wget -nc -O model/dog.jpg https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/dog.jpg?raw=true +wget -nc -O model/mean_224.nd https://github.com/dmlc/web-data/raw/master/mxnet/example/feature_extract/mean_224.nd +tar -xvzf inception-bn.tar.gz -C model + +# Building +make all + + +# Running the example with dog image. +if [ "$(uname)" == "Darwin" ]; then + DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:../../../lib ./inception_inference --symbol "./model/Inception-BN-symbol.json" --params "./model/Inception-BN-0126.params" --synset "./model/synset.txt" --mean "./model/mean_224.nd" --image "./model/dog.jpg" 2&> inception_inference.log +else + LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:../../../lib ./inception_inference --symbol "./model/Inception-BN-symbol.json" --params "./model/Inception-BN-0126.params" --synset "./model/synset.txt" --mean "./model/mean_224.nd" --image "./model/dog.jpg" 2&> inception_inference.log +fi +result=`grep -c "pug-dog" inception_inference.log` +if [ $result == 1 ]; +then + echo "PASS: inception_inference correctly identified the image." + exit 0 +else + echo "FAIL: inception_inference FAILED to identify the image." + exit 1 +fi diff --git a/cpp-package/example/lenet_with_mxdataiter.cpp b/cpp-package/example/lenet_with_mxdataiter.cpp index 9869356be..4df6fbee9 100644 --- a/cpp-package/example/lenet_with_mxdataiter.cpp +++ b/cpp-package/example/lenet_with_mxdataiter.cpp @@ -102,7 +102,7 @@ int main(int argc, char const *argv[]) { auto val_iter = MXDataIter("MNISTIter"); setDataIter(&val_iter, "Label", data_files, batch_size); - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("momentum", 0.9) ->SetParam("rescale_grad", 1.0) ->SetParam("clip_gradient", 10) diff --git a/cpp-package/example/mlp.cpp b/cpp-package/example/mlp.cpp index 595d75c67..cc16f53cf 100644 --- a/cpp-package/example/mlp.cpp +++ b/cpp-package/example/mlp.cpp @@ -144,13 +144,13 @@ void MLP() { grad_req_type, aux_states); std::cout << "Training" << std::endl; - int max_iters = 20000; + int max_epoch = 15000; mx_float learning_rate = 0.0001; - for (int iter = 0; iter < max_iters; ++iter) { + for (int epoch_num = 0; epoch_num < max_epoch; ++epoch_num) { exe->Forward(true); - - if (iter % 100 == 0) { - std::cout << "epoch " << iter << std::endl; + // print accuracy every 100 epoch + if (epoch_num % 100 == 0) { + std::cout << "epoch " << epoch_num << std::endl; std::vector& out = exe->outputs; float* cptr = new float[128 * 10]; out[0].SyncCopyToCPU(cptr, 128 * 10); diff --git a/cpp-package/example/mlp_csv.cpp b/cpp-package/example/mlp_csv.cpp index 8aec4b76d..43a14c84e 100644 --- a/cpp-package/example/mlp_csv.cpp +++ b/cpp-package/example/mlp_csv.cpp @@ -72,7 +72,7 @@ std::vector getLayers(const std::string& hidden_units_string) { void printUsage() { std::cout << "Usage:" << std::endl; std::cout << "mlp_csv --train mnist_training_set.csv --test mnist_test_set.csv --epochs 10 " - << "--batch_size 100 --hidden_units \"128 64 64\" [--gpu]" << std::endl; + << "--batch_size 100 --hidden_units \"128 64 64\" --gpu" << std::endl; std::cout << "The example uses mnist data in CSV format. The MNIST data in CSV format assumes " << "the column 0 to be label and the rest 784 column to be data." << std::endl; std::cout << "By default, the example uses 'cpu' context. If '--gpu' is specified, " diff --git a/cpp-package/example/resnet.cpp b/cpp-package/example/resnet.cpp index bc86c0b66..0bb77a1a1 100644 --- a/cpp-package/example/resnet.cpp +++ b/cpp-package/example/resnet.cpp @@ -184,7 +184,13 @@ int main(int argc, char const *argv[]) { auto val_iter = MXDataIter("MNISTIter"); setDataIter(&val_iter, "Label", data_files, batch_size); - Optimizer* opt = OptimizerRegistry::Find("ccsgd"); + // initialize parameters + Xavier xavier = Xavier(Xavier::gaussian, Xavier::in, 2); + for (auto &arg : args_map) { + xavier(arg.first, &arg.second); + } + + Optimizer* opt = OptimizerRegistry::Find("sgd"); opt->SetParam("lr", learning_rate) ->SetParam("wd", weight_decay) ->SetParam("momentum", 0.9) @@ -194,9 +200,12 @@ int main(int argc, char const *argv[]) { auto *exec = resnet.SimpleBind(ctx, args_map); auto arg_names = resnet.ListArguments(); + // Create metrics + Accuracy train_acc, val_acc; for (int iter = 0; iter < max_epoch; ++iter) { LG << "Epoch: " << iter; train_iter.Reset(); + train_acc.Reset(); while (train_iter.Next()) { auto data_batch = train_iter.GetDataBatch(); data_batch.data.CopyTo(&args_map["data"]); @@ -211,10 +220,11 @@ int main(int argc, char const *argv[]) { opt->Update(i, exec->arg_arrays[i], exec->grad_arrays[i]); } NDArray::WaitAll(); + train_acc.Update(data_batch.label, exec->outputs[0]); } - Accuracy acu; val_iter.Reset(); + val_acc.Reset(); while (val_iter.Next()) { auto data_batch = val_iter.GetDataBatch(); data_batch.data.CopyTo(&args_map["data"]); @@ -222,9 +232,10 @@ int main(int argc, char const *argv[]) { NDArray::WaitAll(); exec->Forward(false); NDArray::WaitAll(); - acu.Update(data_batch.label, exec->outputs[0]); + val_acc.Update(data_batch.label, exec->outputs[0]); } - LG << "Accuracy: " << acu.Get(); + LG << "Train Accuracy: " << train_acc.Get(); + LG << "Validation Accuracy: " << val_acc.Get(); } delete exec; MXNotifyShutdown(); diff --git a/cpp-package/example/test_optimizer.cpp b/cpp-package/example/test_optimizer.cpp index ee1201228..42547ea6c 100644 --- a/cpp-package/example/test_optimizer.cpp +++ b/cpp-package/example/test_optimizer.cpp @@ -15,6 +15,10 @@ * KIND, either express or implied. See the License for the * specific language governing permissions and limitations * under the License. + * + * The file is used for testing if the optimizer could be created more than 1. + * By running: build/test_optimizer + * It return 0(means no error) if it succeed otherwise 1(error). */ #include "mxnet-cpp/MxNetCpp.h" diff --git a/cpp-package/example/test_score.cpp b/cpp-package/example/test_score.cpp index f92560fe8..9252701e2 100644 --- a/cpp-package/example/test_score.cpp +++ b/cpp-package/example/test_score.cpp @@ -19,6 +19,11 @@ /*! * Xin Li yakumolx@gmail.com + * The file is used for testing if the score(accurary) we get + * is better than the threshold we set using mlp model. + * By running: build/test_score 0.75 + * 0.75 here means the threshold score + * It return 0 if we can achieve higher score than threshold, otherwise 1 */ #include #include "utils.h" diff --git a/cpp-package/example/utils.h b/cpp-package/example/utils.h index 98b647268..2ed5c4c11 100644 --- a/cpp-package/example/utils.h +++ b/cpp-package/example/utils.h @@ -42,7 +42,7 @@ bool check_datafiles(const std::vector &data_files) { return true; } -bool setDataIter(MXDataIter *iter , std::string useType, +bool setDataIter(MXDataIter *iter , const std::string &useType, const std::vector &data_files, int batch_size) { if (!check_datafiles(data_files)) return false; diff --git a/cpp-package/include/mxnet-cpp/operator.h b/cpp-package/include/mxnet-cpp/operator.h index 4d4bedac8..9f289f0e2 100644 --- a/cpp-package/include/mxnet-cpp/operator.h +++ b/cpp-package/include/mxnet-cpp/operator.h @@ -86,7 +86,7 @@ class Operator { * \param symbol the input symbol * \return reference of self */ - Operator &SetInput(const std::string &name, Symbol symbol); + Operator &SetInput(const std::string &name, const Symbol &symbol); /*! * \brief add an input symbol * \param symbol the input symbol @@ -133,7 +133,7 @@ class Operator { * \param ndarray the input ndarray * \return reference of self */ - Operator &SetInput(const std::string &name, NDArray ndarray); + Operator &SetInput(const std::string &name, const NDArray &ndarray); /*! * \brief add an input ndarray * \param ndarray the input ndarray diff --git a/cpp-package/include/mxnet-cpp/operator.hpp b/cpp-package/include/mxnet-cpp/operator.hpp index f4ce43d58..edc396f14 100644 --- a/cpp-package/include/mxnet-cpp/operator.hpp +++ b/cpp-package/include/mxnet-cpp/operator.hpp @@ -158,7 +158,7 @@ inline void Operator::Invoke(NDArray &output) { Invoke(outputs); } -inline Operator &Operator::SetInput(const std::string &name, Symbol symbol) { +inline Operator &Operator::SetInput(const std::string &name, const Symbol &symbol) { if (symbol.GetHandle()) { input_keys_.push_back(name.c_str()); input_symbols_.push_back(symbol.GetHandle()); @@ -166,7 +166,7 @@ inline Operator &Operator::SetInput(const std::string &name, Symbol symbol) { return *this; } -inline Operator &Operator::SetInput(const std::string &name, NDArray ndarray) { +inline Operator &Operator::SetInput(const std::string &name, const NDArray &ndarray) { input_keys_.push_back(name.c_str()); input_ndarrays_.push_back(ndarray.GetHandle()); return *this; diff --git a/cpp-package/include/mxnet-cpp/symbol.hpp b/cpp-package/include/mxnet-cpp/symbol.hpp index b82e060ca..aed963949 100644 --- a/cpp-package/include/mxnet-cpp/symbol.hpp +++ b/cpp-package/include/mxnet-cpp/symbol.hpp @@ -281,8 +281,8 @@ inline void Symbol::InferExecutorArrays( auto iter_req = grad_req_type.find(arg_name); if (iter_req != grad_req_type.end()) { grad_reqs->push_back(iter_req->second); - } else if (arg_name.rfind("data") == arg_name.length() - 4 - || arg_name.rfind("label") == arg_name.length() - 5) { + } else if (arg_name.rfind("data") != std::string::npos + || arg_name.rfind("label") != std::string::npos) { grad_reqs->push_back(OpReqType::kNullOp); } else { grad_reqs->push_back(OpReqType::kWriteTo); diff --git a/cpp-package/scripts/OpWrapperGenerator.py b/cpp-package/scripts/OpWrapperGenerator.py index 1b5f8b56b..ca430ec99 100644 --- a/cpp-package/scripts/OpWrapperGenerator.py +++ b/cpp-package/scripts/OpWrapperGenerator.py @@ -138,6 +138,8 @@ def __init__(self, opName = '', argName = '', typeString = '', descString = ''): self.defaultString = 'Shape(' + self.defaultString[1:-1] + ")" elif self.type == 'dmlc::optional': self.defaultString = self.type + '(' + self.defaultString + ')' + elif self.type == 'dmlc::optional': + self.defaultString = self.type + '(' + self.defaultString + ')' elif typeString.startswith('caffe-layer-parameter'): self.defaultString = 'textToCaffeLayerParameter(' + self.MakeCString(self.defaultString) + ')' hasCaffe = True @@ -221,14 +223,14 @@ def GetOpDefinitionString(self, use_name, indent=0): if arg.isEnum and use_name: # comments ret = ret + self.GenDescription(arg.description, \ - '/*! \\breif ', \ + '/*! \\brief ', \ ' * ') ret = ret + " */\n" # definition ret = ret + arg.enum.GetDefinitionString(indent) + '\n' # create function comments ret = ret + self.GenDescription(self.description, \ - '/*!\n * \\breif ', \ + '/*!\n * \\brief ', \ ' * ') for arg in self.args: if arg.name != 'symbol_name' or use_name: diff --git a/cpp-package/tests/ci_test.sh b/cpp-package/tests/ci_test.sh index 57007f3a8..4a17d8d34 100755 --- a/cpp-package/tests/ci_test.sh +++ b/cpp-package/tests/ci_test.sh @@ -36,6 +36,9 @@ cp ../../build/cpp-package/example/lenet_with_mxdataiter . cp ../../build/cpp-package/example/resnet . ./resnet 5 +cp ../../build/cpp-package/example/inception_bn . +./inception_bn 5 + cp ../../build/cpp-package/example/mlp . ./mlp @@ -50,3 +53,5 @@ cp ../../build/cpp-package/example/mlp_gpu . cp ../../build/cpp-package/example/test_score . ./test_score 0.93 + +sh unittests/unit_test_mlp_csv.sh diff --git a/dev_menu.py b/dev_menu.py new file mode 100755 index 000000000..7329b446e --- /dev/null +++ b/dev_menu.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# -*- coding: utf-8 -*- +"""Tool to ease working with the build system and reproducing test results""" + +import argparse +import os +import sys +from subprocess import check_call +import shlex +from ci.util import retry, remember_cwd +from typing import List +from collections import OrderedDict +import logging +import yaml +import shutil + +DEFAULT_PYENV=os.environ.get('DEFAULT_PYENV','py3_venv') +DEFAULT_PYTHON=os.environ.get('DEFAULT_PYTHON','python3') +DEFAULT_CMAKE_OPTIONS=os.environ.get('DEFAULT_CMAKE_OPTIONS','cmake_options.yml') + +class Confirm(object): + def __init__(self, cmds): + self.cmds = cmds + + def __call__(self): + resp = input("This will run the following command(s) '{}' are you sure? yes / no: ".format(self.cmds)) + while True: + if resp.lower() == 'yes': + handle_commands(self.cmds) + return + elif resp.lower() == 'no': + return + else: + resp = input("Please answer yes or no: ") + +class CMake(object): + def __init__(self, cmake_options_yaml=DEFAULT_CMAKE_OPTIONS, cmake_options_yaml_default='cmake/cmake_options.yml'): + if os.path.exists(cmake_options_yaml): + self.cmake_options_yaml = cmake_options_yaml + else: + self.cmake_options_yaml = cmake_options_yaml_default + logging.info('Using {} for CMake configuration'.format(self.cmake_options_yaml)) + self.cmake_options = None + self.read_config() + + def read_config(self): + assert os.path.isfile(self.cmake_options_yaml) + with open(self.cmake_options_yaml, 'r') as f: + self.cmake_options = yaml.load(f) + + def _cmdlineflags(self): + res = [] + for opt,v in self.cmake_options.items(): + res.append('-D{}={}'.format(opt,v)) + return res + + def cmake_command(self) -> str: + """ + :return: Cmake command to run given the options + """ + cmd_lst = ['cmake'] + cmd_lst.extend(self._cmdlineflags()) + return cmd_lst + + def __call__(self, build_dir='build', generator='Ninja', build_cmd='ninja'): + logging.info("CMake / {} build in directory {}".format( + generator, os.path.abspath(build_dir))) + cmd_lst = self.cmake_command() + os.makedirs(build_dir, exist_ok=True) + with remember_cwd(): + os.chdir(build_dir) + cmd_lst.extend(['-G{}'.format(generator), '..']) + logging.info('Executing: {}'.format('\t\n'.join(cmd_lst))) + check_call(cmd_lst) + logging.info('Now building') + check_call(shlex.split(build_cmd)) + +def create_virtualenv(venv_exe, pyexe, venv) -> None: + logging.info("Creating virtualenv in %s with python %s", venv, pyexe) + if not (venv_exe and pyexe and venv): + logging.warn("Skipping creation of virtualenv") + return + check_call([venv_exe, '-p', pyexe, venv]) + activate_this_py = os.path.join(venv, 'bin', 'activate_this.py') + # Activate virtualenv in this interpreter + exec(open(activate_this_py).read(), dict(__file__=activate_this_py)) + check_call(['pip', 'install', '--upgrade','--force-reinstall', '-e', 'python']) + check_call(['pip', 'install', '-r', 'tests/requirements.txt']) + +def create_virtualenv_default(): + create_virtualenv('virtualenv', DEFAULT_PYTHON, DEFAULT_PYENV) + logging.info("You can use the virtualenv by executing 'source %s/bin/activate'", DEFAULT_PYENV) + +COMMANDS = OrderedDict([ + ('[Local] BUILD CMake/Ninja (using cmake_options.yaml (cp cmake/cmake_options.yml .) and edit) ({} virtualenv in "{}")'.format(DEFAULT_PYTHON, DEFAULT_PYENV), + [ + CMake(), + create_virtualenv_default, + ]), + ('[Local] Python Unit tests', + "./py3_venv/bin/nosetests -v tests/python/unittest/" + ), + ('[Website and docs build] Will build to docs/_build/html/', + "ci/docker/runtime_functions.sh deploy_docs"), + ('[Docker] sanity_check. Check for linting and code formatting.', + "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh sanity_check"), + ('[Docker] Python3 CPU unittests', + [ + "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_openblas", + "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python3_cpu", + ]), + ('[Docker] Python3 GPU unittests', + [ + "ci/build.py --platform ubuntu_gpu /work/runtime_functions.sh build_ubuntu_gpu", + "ci/build.py --nvidiadocker --platform ubuntu_gpu /work/runtime_functions.sh unittest_ubuntu_python3_gpu", + ]), + ('[Docker] Python3 GPU+MKLDNN unittests', + [ + "ci/build.py --platform ubuntu_gpu /work/runtime_functions.sh build_ubuntu_gpu_cmake_mkldnn", + "ci/build.py --nvidiadocker --platform ubuntu_gpu /work/runtime_functions.sh unittest_ubuntu_python3_gpu", + ]), + ('[Docker] Python3 CPU Intel MKLDNN unittests', + [ + "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_mkldnn", + "ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python3_cpu", + ]), + ('[Docker] Python3 ARMv7 unittests (QEMU)', + [ + "ci/build.py -p armv7", + "ci/build.py -p test.arm_qemu ./runtime_functions.py run_ut_py3_qemu" + ]), + ('Clean (RESET HARD) repository (Warning! erases local changes / DATA LOSS)', + Confirm("ci/docker/runtime_functions.sh clean_repo")) +]) + +def clip(x, mini, maxi): + return min(max(x,mini), maxi) + +@retry((ValueError, RuntimeError), 3, delay_s = 0) +def show_menu(items: List[str], header=None) -> int: + print('\n-- MXNet dev menu --\n') + def hr(): + print(''.join(['-']*30)) + if header: + print(header) + hr() + for i,x in enumerate(items,1): + print('{}. {}'.format(i,x)) + hr() + choice = int(input('Choose option> ')) - 1 + if choice < 0 or choice >= len(items): + raise RuntimeError('Choice must be between {} and {}'.format(1, len(items))) + return choice + +def handle_commands(cmds) -> None: + def handle_command(cmd): + logging.info("Executing command: %s",cmd) + check_call(shlex.split(cmd)) + + if type(cmds) is list: + for cmd in cmds: + handle_commands(cmd) + elif type(cmds) is str: + handle_command(cmds) + elif callable(cmds): + cmds() + else: + raise RuntimeError("handle_commands(cmds): argument should be str or List[str] but is %s", type(cmds)) + +def use_menu_ui(args) -> None: + command_list = list(COMMANDS.keys()) + if hasattr(args, 'choice') and args.choice and args.choice[0].isdigit(): + choice = int(args.choice[0]) - 1 + else: + choice = show_menu(command_list, 'Available actions') + handle_commands(COMMANDS[command_list[choice]]) + +def build(args) -> None: + """Build using CMake""" + venv_exe = shutil.which('virtualenv') + pyexe = shutil.which(args.pyexe) + if not venv_exe: + logging.warn("virtualenv wasn't found in path, it's recommended to install virtualenv to manage python environments") + if not pyexe: + logging.warn("Python executable %s not found in path", args.pyexe) + if args.cmake_options: + cmake = CMake(args.cmake_options) + else: + cmake = CMake() + cmake() + create_virtualenv(venv_exe, pyexe, args.venv) + +def main(): + logging.getLogger().setLevel(logging.INFO) + parser = argparse.ArgumentParser(description="""Utility for compiling and testing MXNet easily""") + parser.set_defaults(command='use_menu_ui') + + subparsers = parser.add_subparsers(help='sub-command help') + build_parser = subparsers.add_parser('build', help='build with the specified flags from file') + build_parser.add_argument('cmake_options', nargs='?', + help='File containing CMake options in YAML') + build_parser.add_argument('-v', '--venv', + type=str, + default=DEFAULT_PYENV, + help='virtualenv dir') + build_parser.add_argument('-p', '--pyexe', + type=str, + default=DEFAULT_PYTHON, + help='python executable') + build_parser.set_defaults(command='build') + + menu_parser = subparsers.add_parser('menu', help='jump to menu option #') + menu_parser.set_defaults(command='use_menu_ui') + menu_parser.add_argument('choice', nargs=1) + + args = parser.parse_args() + globals()[args.command](args) + return 0 + +if __name__ == '__main__': + sys.exit(main()) diff --git a/docs/Doxyfile b/docs/Doxyfile index cee3942bc..bf6344d4f 100644 --- a/docs/Doxyfile +++ b/docs/Doxyfile @@ -770,7 +770,7 @@ WARN_LOGFILE = # spaces. # Note: If this tag is empty the current directory is searched. -INPUT = include src/common +INPUT = include src/common cpp-package/include/mxnet-cpp # This tag can be used to specify the character encoding of the source files # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses @@ -805,7 +805,7 @@ RECURSIVE = YES # Note that relative paths are relative to the directory from which doxygen is # run. -EXCLUDE = +EXCLUDE = 3rdparty # The EXCLUDE_SYMLINKS tag can be used to select whether or not files or # directories that are symbolic links (a Unix file system feature) are excluded diff --git a/docs/Jenkinsfile b/docs/Jenkinsfile index e20d984f0..676204291 100644 --- a/docs/Jenkinsfile +++ b/docs/Jenkinsfile @@ -21,14 +21,14 @@ // See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/ // timeout in minutes -max_time = 120 +max_time = 180 -node('restricted-mxnetlinux-cpu') { +node('restricted-utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu') +utils.assign_node_labels(utility: 'restricted-utility', linux_cpu: 'restricted-mxnetlinux-cpu') utils.main_wrapper( core_logic: { diff --git a/docs/Jenkinsfile-dev b/docs/Jenkinsfile-dev index 169ebe13e..760a2f953 100644 --- a/docs/Jenkinsfile-dev +++ b/docs/Jenkinsfile-dev @@ -23,12 +23,12 @@ // timeout in minutes max_time = 120 -node('mxnetlinux-cpu') { +node('utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu') utils.main_wrapper( core_logic: { diff --git a/docs/README.md b/docs/README.md index c21836edd..80463cc68 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,9 +17,13 @@ git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet cd mxnet/docs/build_version_doc ./setup_docs_ubuntu.sh cd ../../ -make docs USE_OPENMP=1 +make docs USE_OPENMP=1 SPHINXOPTS=-W ``` +OpenMP speeds things up and will work on Ubuntu if you used the `setup_docs_ubuntu.sh` script. +The `-W` Sphinx option enforces "warnings as errors". This will help you debug your builds and get them through CI. +**CI will not let a PR through if it breaks the website.** Refer to the [MXNet Developer wiki's documentation guide](https://cwiki.apache.org/confluence/display/MXNET/Documentation+Guide) for troubleshooting tips. + For more information on each API's documentation dependencies, how to serve the docs, or how to build the full website with each legacy MXNet version, refer to the following links: * [Dependencies](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#dependencies) - required before you build the docs diff --git a/docs/_static/js/auto_module_index.js b/docs/_static/js/auto_module_index.js index 7f4e18565..8527a71ed 100644 --- a/docs/_static/js/auto_module_index.js +++ b/docs/_static/js/auto_module_index.js @@ -1,3 +1,23 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Customizations to the Sphinx auto module plugin output */ function auto_index(module) { $(document).ready(function () { // find all classes or functions @@ -10,15 +30,17 @@ function auto_index(module) { var html = "

    "; for (var i = 0; i < targets.length; ++i) { - var id = $(targets[i]).attr('id'); - // remove 'mxnet.' prefix to make menus shorter - var id_simple = id.replace(/^mxnet\./, ''); - html += "
  • " + id_simple + "
    • "; + var id = $(targets[i]).attr('id'); + if ( id ) { + // remove 'mxnet.' prefix to make menus shorter + var id_simple = id.replace(/^mxnet\./, ''); + html += "
  • " + id_simple + "
    • "; + } } html += "
"; li_node.append(html); }); -} \ No newline at end of file +} diff --git a/docs/_static/js/clipboard.js b/docs/_static/js/clipboard.js new file mode 100644 index 000000000..75b6af353 --- /dev/null +++ b/docs/_static/js/clipboard.js @@ -0,0 +1,778 @@ +/*! + * clipboard.js v1.6.1 + * https://zenorocha.github.io/clipboard.js + * + * Licensed MIT © Zeno Rocha + */ +(function(f){if(typeof exports==="object"&&typeof module!=="undefined"){module.exports=f()}else if(typeof define==="function"&&define.amd){define([],f)}else{var g;if(typeof window!=="undefined"){g=window}else if(typeof global!=="undefined"){g=global}else if(typeof self!=="undefined"){g=self}else{g=this}g.Clipboard = f()}})(function(){var define,module,exports;return (function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f}var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1][e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o 0 && arguments[0] !== undefined ? arguments[0] : {}; + + this.action = options.action; + this.emitter = options.emitter; + this.target = options.target; + this.text = options.text; + this.trigger = options.trigger; + + this.selectedText = ''; + } + }, { + key: 'initSelection', + value: function initSelection() { + if (this.text) { + this.selectFake(); + } else if (this.target) { + this.selectTarget(); + } + } + }, { + key: 'selectFake', + value: function selectFake() { + var _this = this; + + var isRTL = document.documentElement.getAttribute('dir') == 'rtl'; + + this.removeFake(); + + this.fakeHandlerCallback = function () { + return _this.removeFake(); + }; + this.fakeHandler = document.body.addEventListener('click', this.fakeHandlerCallback) || true; + + this.fakeElem = document.createElement('textarea'); + // Prevent zooming on iOS + this.fakeElem.style.fontSize = '12pt'; + // Reset box model + this.fakeElem.style.border = '0'; + this.fakeElem.style.padding = '0'; + this.fakeElem.style.margin = '0'; + // Move element out of screen horizontally + this.fakeElem.style.position = 'absolute'; + this.fakeElem.style[isRTL ? 'right' : 'left'] = '-9999px'; + // Move element to the same position vertically + var yPosition = window.pageYOffset || document.documentElement.scrollTop; + this.fakeElem.style.top = yPosition + 'px'; + + this.fakeElem.setAttribute('readonly', ''); + this.fakeElem.value = this.text; + + document.body.appendChild(this.fakeElem); + + this.selectedText = (0, _select2.default)(this.fakeElem); + this.copyText(); + } + }, { + key: 'removeFake', + value: function removeFake() { + if (this.fakeHandler) { + document.body.removeEventListener('click', this.fakeHandlerCallback); + this.fakeHandler = null; + this.fakeHandlerCallback = null; + } + + if (this.fakeElem) { + document.body.removeChild(this.fakeElem); + this.fakeElem = null; + } + } + }, { + key: 'selectTarget', + value: function selectTarget() { + this.selectedText = (0, _select2.default)(this.target); + this.copyText(); + } + }, { + key: 'copyText', + value: function copyText() { + var succeeded = void 0; + + try { + succeeded = document.execCommand(this.action); + } catch (err) { + succeeded = false; + } + + this.handleResult(succeeded); + } + }, { + key: 'handleResult', + value: function handleResult(succeeded) { + this.emitter.emit(succeeded ? 'success' : 'error', { + action: this.action, + text: this.selectedText, + trigger: this.trigger, + clearSelection: this.clearSelection.bind(this) + }); + } + }, { + key: 'clearSelection', + value: function clearSelection() { + if (this.target) { + this.target.blur(); + } + + window.getSelection().removeAllRanges(); + } + }, { + key: 'destroy', + value: function destroy() { + this.removeFake(); + } + }, { + key: 'action', + set: function set() { + var action = arguments.length > 0 && arguments[0] !== undefined ? arguments[0] : 'copy'; + + this._action = action; + + if (this._action !== 'copy' && this._action !== 'cut') { + throw new Error('Invalid "action" value, use either "copy" or "cut"'); + } + }, + get: function get() { + return this._action; + } + }, { + key: 'target', + set: function set(target) { + if (target !== undefined) { + if (target && (typeof target === 'undefined' ? 'undefined' : _typeof(target)) === 'object' && target.nodeType === 1) { + if (this.action === 'copy' && target.hasAttribute('disabled')) { + throw new Error('Invalid "target" attribute. Please use "readonly" instead of "disabled" attribute'); + } + + if (this.action === 'cut' && (target.hasAttribute('readonly') || target.hasAttribute('disabled'))) { + throw new Error('Invalid "target" attribute. You can\'t cut text from elements with "readonly" or "disabled" attributes'); + } + + this._target = target; + } else { + throw new Error('Invalid "target" value, use a valid Element'); + } + } + }, + get: function get() { + return this._target; + } + }]); + + return ClipboardAction; + }(); + + module.exports = ClipboardAction; +}); + +},{"select":5}],8:[function(require,module,exports){ +(function (global, factory) { + if (typeof define === "function" && define.amd) { + define(['module', './clipboard-action', 'tiny-emitter', 'good-listener'], factory); + } else if (typeof exports !== "undefined") { + factory(module, require('./clipboard-action'), require('tiny-emitter'), require('good-listener')); + } else { + var mod = { + exports: {} + }; + factory(mod, global.clipboardAction, global.tinyEmitter, global.goodListener); + global.clipboard = mod.exports; + } +})(this, function (module, _clipboardAction, _tinyEmitter, _goodListener) { + 'use strict'; + + var _clipboardAction2 = _interopRequireDefault(_clipboardAction); + + var _tinyEmitter2 = _interopRequireDefault(_tinyEmitter); + + var _goodListener2 = _interopRequireDefault(_goodListener); + + function _interopRequireDefault(obj) { + return obj && obj.__esModule ? obj : { + default: obj + }; + } + + function _classCallCheck(instance, Constructor) { + if (!(instance instanceof Constructor)) { + throw new TypeError("Cannot call a class as a function"); + } + } + + var _createClass = function () { + function defineProperties(target, props) { + for (var i = 0; i < props.length; i++) { + var descriptor = props[i]; + descriptor.enumerable = descriptor.enumerable || false; + descriptor.configurable = true; + if ("value" in descriptor) descriptor.writable = true; + Object.defineProperty(target, descriptor.key, descriptor); + } + } + + return function (Constructor, protoProps, staticProps) { + if (protoProps) defineProperties(Constructor.prototype, protoProps); + if (staticProps) defineProperties(Constructor, staticProps); + return Constructor; + }; + }(); + + function _possibleConstructorReturn(self, call) { + if (!self) { + throw new ReferenceError("this hasn't been initialised - super() hasn't been called"); + } + + return call && (typeof call === "object" || typeof call === "function") ? call : self; + } + + function _inherits(subClass, superClass) { + if (typeof superClass !== "function" && superClass !== null) { + throw new TypeError("Super expression must either be null or a function, not " + typeof superClass); + } + + subClass.prototype = Object.create(superClass && superClass.prototype, { + constructor: { + value: subClass, + enumerable: false, + writable: true, + configurable: true + } + }); + if (superClass) Object.setPrototypeOf ? Object.setPrototypeOf(subClass, superClass) : subClass.__proto__ = superClass; + } + + var Clipboard = function (_Emitter) { + _inherits(Clipboard, _Emitter); + + /** + * @param {String|HTMLElement|HTMLCollection|NodeList} trigger + * @param {Object} options + */ + function Clipboard(trigger, options) { + _classCallCheck(this, Clipboard); + + var _this = _possibleConstructorReturn(this, (Clipboard.__proto__ || Object.getPrototypeOf(Clipboard)).call(this)); + + _this.resolveOptions(options); + _this.listenClick(trigger); + return _this; + } + + /** + * Defines if attributes would be resolved using internal setter functions + * or custom functions that were passed in the constructor. + * @param {Object} options + */ + + + _createClass(Clipboard, [{ + key: 'resolveOptions', + value: function resolveOptions() { + var options = arguments.length > 0 && arguments[0] !== undefined ? arguments[0] : {}; + + this.action = typeof options.action === 'function' ? options.action : this.defaultAction; + this.target = typeof options.target === 'function' ? options.target : this.defaultTarget; + this.text = typeof options.text === 'function' ? options.text : this.defaultText; + } + }, { + key: 'listenClick', + value: function listenClick(trigger) { + var _this2 = this; + + this.listener = (0, _goodListener2.default)(trigger, 'click', function (e) { + return _this2.onClick(e); + }); + } + }, { + key: 'onClick', + value: function onClick(e) { + var trigger = e.delegateTarget || e.currentTarget; + + if (this.clipboardAction) { + this.clipboardAction = null; + } + + this.clipboardAction = new _clipboardAction2.default({ + action: this.action(trigger), + target: this.target(trigger), + text: this.text(trigger), + trigger: trigger, + emitter: this + }); + } + }, { + key: 'defaultAction', + value: function defaultAction(trigger) { + return getAttributeValue('action', trigger); + } + }, { + key: 'defaultTarget', + value: function defaultTarget(trigger) { + var selector = getAttributeValue('target', trigger); + + if (selector) { + return document.querySelector(selector); + } + } + }, { + key: 'defaultText', + value: function defaultText(trigger) { + return getAttributeValue('text', trigger); + } + }, { + key: 'destroy', + value: function destroy() { + this.listener.destroy(); + + if (this.clipboardAction) { + this.clipboardAction.destroy(); + this.clipboardAction = null; + } + } + }], [{ + key: 'isSupported', + value: function isSupported() { + var action = arguments.length > 0 && arguments[0] !== undefined ? arguments[0] : ['copy', 'cut']; + + var actions = typeof action === 'string' ? [action] : action; + var support = !!document.queryCommandSupported; + + actions.forEach(function (action) { + support = support && !!document.queryCommandSupported(action); + }); + + return support; + } + }]); + + return Clipboard; + }(_tinyEmitter2.default); + + /** + * Helper function to retrieve attribute value. + * @param {String} suffix + * @param {Element} element + */ + function getAttributeValue(suffix, element) { + var attribute = 'data-clipboard-' + suffix; + + if (!element.hasAttribute(attribute)) { + return; + } + + return element.getAttribute(attribute); + } + + module.exports = Clipboard; +}); + +},{"./clipboard-action":7,"good-listener":4,"tiny-emitter":6}]},{},[8])(8) +}); \ No newline at end of file diff --git a/docs/_static/js/copycode.js b/docs/_static/js/copycode.js index f9ebd64ab..d42e99277 100644 --- a/docs/_static/js/copycode.js +++ b/docs/_static/js/copycode.js @@ -1,4 +1,23 @@ -/*Copy code to clipboard*/ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Copy code to clipboard */ LANG_GP = {'default':'>>> ', 'python':'>>> ' , 'scala':'scala>', 'julia':'julia> ', 'r':'> ', 'perl':'pdl>' , 'cpp':'', 'bash':'$ '}; function addBtn() { diff --git a/docs/_static/js/docversion.js b/docs/_static/js/docversion.js index f87c4587b..1119f4ec1 100644 --- a/docs/_static/js/docversion.js +++ b/docs/_static/js/docversion.js @@ -1,3 +1,23 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Set the version of the website */ function setVersion(){ let doc = window.location.pathname.match(/^\/(api\/.*)$/) || window.location.pathname.match(/^\/versions\/[^*]+\/(api\/.*)$/); if (doc) { @@ -11,7 +31,7 @@ function setVersion(){ $( el ).attr('href', versionedDoc); } }); - } + } } } diff --git a/docs/_static/js/navbar.js b/docs/_static/js/navbar.js index 0384194fa..5dde7d8ff 100644 --- a/docs/_static/js/navbar.js +++ b/docs/_static/js/navbar.js @@ -1,3 +1,23 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Custom navigation bar formatting */ var searchBox = $("#search-input-wrap"); var TITLE = ['/install/', '/gluon/', '/api/', '/docs/', '/community/' ]; var DOC_TITLE = ['/faq/', '/tutorials/', '/architecture/', '/model_zoo/']; @@ -25,7 +45,7 @@ function navbar() { $(this).hide; } else rightPos = $(this).offset().left + $(this).width(); - + if(isCovered) { plusMenuList.push($(this).clone()); $(this).hide(); @@ -38,7 +58,7 @@ function navbar() { } else $(this).show(); }); - + if(plusMenuList.length == 0) { $(".plusIcon").first().hide(); return; diff --git a/docs/_static/js/options.js b/docs/_static/js/options.js index e9225aafc..f4fde4e1f 100644 --- a/docs/_static/js/options.js +++ b/docs/_static/js/options.js @@ -1,4 +1,24 @@ -var versionSelect = defaultVersion = 'v1.3.0'; +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Installation page display functions for install selector */ +var versionSelect = defaultVersion = 'v1.3.1'; var platformSelect = 'Linux'; var languageSelect = 'Python'; var processorSelect = 'CPU'; diff --git a/docs/_static/js/page.js b/docs/_static/js/page.js index caba7dd1b..425998d6d 100644 --- a/docs/_static/js/page.js +++ b/docs/_static/js/page.js @@ -1,3 +1,22 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + /* Generate url tracking for each page */ var protocol = location.protocol.concat("//"); var host = protocol.concat(window.location.host); diff --git a/docs/_static/js/search.js b/docs/_static/js/search.js index 9df970222..6a70b4ef5 100644 --- a/docs/_static/js/search.js +++ b/docs/_static/js/search.js @@ -1,9 +1,29 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Display functionality for the search feature */ $(document).ready(function () { var searchForm = $("#search-input-wrap").children("form").first(); searchForm.append('
'); searchForm.children("div").first().addClass("searchBox"); $(".searchBox").addClass("searchBoxNorm"); - + $('#searchIcon').click(function () { if($('#search-input-wrap').is(':hidden')) { $('#search-input-wrap').show(); @@ -16,4 +36,4 @@ $(document).ready(function () { $('#searchIcon span').addClass('glyphicon-search'); } }); -}); \ No newline at end of file +}); diff --git a/docs/_static/js/sidebar.js b/docs/_static/js/sidebar.js index 3e7ad41bf..65899d5dd 100644 --- a/docs/_static/js/sidebar.js +++ b/docs/_static/js/sidebar.js @@ -1,5 +1,24 @@ -/*Preprocess*/ -var LANG = ['python', 'c++', 'clojure', 'julia', 'perl', 'r', 'scala']; +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/* Customizations to the sphinx theme */ +var LANG = ['python', 'c++', 'clojure', 'julia', 'perl', 'r', 'scala', 'java']; var TITLE_WITH_LANG = ['/get_started/', '/tutorials/', '/faq/', '/architecture/', '/community/']; for(var i = 0; i < LANG.length; ++i) { TITLE_WITH_LANG.push('/api/' + LANG[i] + '/'); diff --git a/docs/_static/mxnet-theme/index.html b/docs/_static/mxnet-theme/index.html index a6cae4de0..302b17322 100644 --- a/docs/_static/mxnet-theme/index.html +++ b/docs/_static/mxnet-theme/index.html @@ -23,9 +23,9 @@
-

MXNet 1.3.0 Released

-

We're excited to announce the release of MXNet 1.3.0! This release includes Clojure bindings, Gluon package enhancements, ONNX export, TensorRT integration, and more!

- Learn More +

MXNet 1.3.1 Released

+

This release includes bug fixes, performance improvements, and documentation updates.

+ Learn More

A 60-minute Gluon Crash Course

diff --git a/docs/_static/mxnet-theme/navbar.html b/docs/_static/mxnet-theme/navbar.html index 977d005d9..50c3debba 100644 --- a/docs/_static/mxnet-theme/navbar.html +++ b/docs/_static/mxnet-theme/navbar.html @@ -27,6 +27,7 @@

  • Perl
  • R
  • Scala
    • +
  • Java
    • @@ -80,6 +81,7 @@

  • Perl
  • R
  • Scala
    • +
  • Java
  • diff --git a/docs/_static/selectlang.js b/docs/_static/selectlang.js index 25337abcb..c0075f956 100644 --- a/docs/_static/selectlang.js +++ b/docs/_static/selectlang.js @@ -1,3 +1,22 @@ +/*! + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + function changeLanguage(langSelect, langSelectLabel, rootpath){ langSelect.change(function() { var lang = langSelect.val(); @@ -22,4 +41,4 @@ $(document).ready(function () { langSelectLabel.text($("option:selected").text()); changeLanguage(langSelect, langSelectLabel, getRootPath()); -}) \ No newline at end of file +}) diff --git a/docs/api/c++/index.md b/docs/api/c++/index.md index 9a68cdafd..6bb7a410b 100644 --- a/docs/api/c++/index.md +++ b/docs/api/c++/index.md @@ -1,8 +1,55 @@ # MXNet - C++ API +The MXNet C++ Package provides C++ API bindings to the users of MXNet. Currently, these bindings are not available as standalone package. +The users of these bindings are required to build this package as mentioned below. + +## Building C++ Package + +The cpp-package directory contains the implementation of C++ API. As mentioned above, users are required to build this directory or package before using it. +**The cpp-package is built while building the MXNet shared library, *libmxnet.so*.** + +### Steps to build the C++ package: +1. Building the MXNet C++ package requires building MXNet from source. +2. Clone the MXNet GitHub repository **recursively** to ensure the code in submodules is available for building MXNet. + ``` + git clone --recursive https://github.com/apache/incubator-mxnet mxnet + ``` + +3. Install the [prerequisites](), desired [BLAS libraries]() and optional [OpenCV, CUDA, and cuDNN]() for building MXNet from source. +4. There is a configuration file for make, [make/config.mk]() that contains all the compilation options. You can edit this file and set the appropriate options prior to running the **make** command. +5. Please refer to [platform specific build instructions]() and available [build configurations](https://mxnet.incubator.apache.org/install/build_from_source#build-configurations) for more details. +5. For enabling the build of C++ Package, set the **USE\_CPP\_PACKAGE = 1** in [make/config.mk](). Optionally, the compilation flag can also be specified on **make** command line as follows. + ``` + make -j USE_CPP_PACKAGE=1 + ``` + +## Usage + +In order to consume the C++ API please follow the steps below. + +1. Ensure that the MXNet shared library is built from source with the **USE\_CPP\_PACKAGE = 1**. +2. Include the [MxNetCpp.h]() in the program that is going to consume MXNet C++ API. + ``` + #include + ``` +3. While building the program, ensure that the correct paths to the directories containing header files and MXNet shared library. +4. The program links the MXNet shared library dynamically. Hence the library needs to be accessible to the program during runtime. This can be achieved by including the path to the shared library in the environment variable **LD\_LIBRARY\_PATH** for Linux, Mac. and Ubuntu OS and **PATH** for Windows OS. + + +## Tutorial + +A basic tutorial can be found at . + +## Examples + +The example directory contains examples for you to get started. For namespaces, classes, and code files for the MXNet C++ package, see the following: -* [Namespaces](http://mxnet.io/doxygen/namespaces.html) -* [Classes](http://mxnet.io/doxygen/annotated.html) -* [Code Files](http://mxnet.io/doxygen/files.html) -* [MXNet CPP Package](https://github.com/dmlc/mxnet/tree/master/cpp-package) +## Links to the documentation + +The classes and functions in MXNet C++ API are available under **mxnet::cpp** namespace. The links to the documenation are as follows: + +1. [Namespaces](../../doxygen/namespaces.html) +2. [Classes in mxnet::cpp namespace](../../doxygen/namespacemxnet_1_1cpp.html) +3. [Code Files](../../doxygen/files.html) +4. [MXNet CPP Package](https://github.com/dmlc/mxnet/tree/master/cpp-package) diff --git a/docs/api/clojure/ndarray.md b/docs/api/clojure/ndarray.md index 814df8b2c..b0e5c991f 100644 --- a/docs/api/clojure/ndarray.md +++ b/docs/api/clojure/ndarray.md @@ -91,8 +91,9 @@ You can use MXNet functions to save and load a list or dictionary of NDArrays fr ```clojure (ndarray/save "filename" {"arr1" arr1 "arr2" arr2}) ;; you can also do "s3://path" or "hdfs" +``` -To load +To load: ```clojure (def from-file (ndarray/load "filename")) diff --git a/docs/api/clojure/symbol.md b/docs/api/clojure/symbol.md index 85e197736..1ec841f15 100644 --- a/docs/api/clojure/symbol.md +++ b/docs/api/clojure/symbol.md @@ -7,7 +7,6 @@ Topics: * [Group Multiple Symbols](#group-multiple-symbols) * [Serialization](#serialization) * [Executing Symbols](#executing-symbols) -* [Multiple Outputs](#multiple-outputs) * [Symbol API Reference](http://mxnet.incubator.apache.org/api/clojure/docs/org.apache.clojure-mxnet.symbol.html) @@ -128,23 +127,6 @@ _To do this you must have the correct native library jar defined as a dependency (def ex (sym/bind c (context/gpu 0) {"a" (ndarray/ones [2 2]) "b" (ndarray/ones [2 2])})) ``` -## Multiple Outputs - -To construct neural networks with multiple loss layers, we can use mxnet.sym.Group to group multiple symbols together. The following example groups two outputs: - -```clojure -(def net (sym/variable "data")) -(def fc1 (sym/fully-connected {:data net :num-hidden 128})) -(def net2 (sym/activation {:data fc1 :act-type "relu"})) -(def out1 (sym/softmax-output {:data net2})) -(def out2 (sym/linear-regression-output {:data net2})) -(def group (sym/group [out1 out2])) -(sym/list-outputs group);=> ["softmaxoutput0_output" "linearregressionoutput0_output"] -``` - -After you get the ```group```, you can bind on ```group``` instead. -The resulting executor will have two outputs, one for `linerarregressionoutput_output` and one for `softmax_output`. - ## Next Steps * See [NDArray API](ndarray.md) for vector/matrix/tensor operations. * See [KVStore API](kvstore.md) for multi-GPU and multi-host distributed training. diff --git a/docs/api/index.md b/docs/api/index.md index eff680767..9e7a58f77 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -1,11 +1,13 @@ # MXNet APIs + ```eval_rst .. toctree:: :maxdepth: 1 c++/index.md clojure/index.md + java/index.md julia/index.md perl/index.md python/index.md diff --git a/docs/api/java/index.md b/docs/api/java/index.md new file mode 100644 index 000000000..b2ae035b1 --- /dev/null +++ b/docs/api/java/index.md @@ -0,0 +1,15 @@ +# MXNet - Java Inference API + +MXNet supports Java for performing inference on a trained model. The MXNet Java Inference API is an extension of the [Scala Infer API](../../api/scala/infer.html) which provides model loading and inference functionality. + +The goal of the MXNet Java package is to provide an efficient and easy to use inference API. +The MXNet Java package makes it easy to quickly deploy an existing model into a production level Java ecosystem. + +## Installation +* [MXNet Java Inference API setup instructions](../../install/java_setup.md) + +## Tutorials +See the [tutorial page](../../tutorials/index.html#java-tutorials) for detailed tutorials and examples using the Java Inference API. + +## Java Inference API Reference +The [Java Infer API javadocs](docs/index.html#org.apache.mxnet.infer.package) provides detailed API information. \ No newline at end of file diff --git a/docs/api/python/gluon/loss.md b/docs/api/python/gluon/loss.md index 3747a0f89..948f4983d 100644 --- a/docs/api/python/gluon/loss.md +++ b/docs/api/python/gluon/loss.md @@ -25,6 +25,7 @@ This package includes several commonly used loss functions in neural networks. LogisticLoss TripletLoss CTCLoss + CosineEmbeddingLoss PoissonNLLLoss ``` diff --git a/docs/api/python/gluon/model_zoo.md b/docs/api/python/gluon/model_zoo.md index e6ac795a6..288cf96f1 100644 --- a/docs/api/python/gluon/model_zoo.md +++ b/docs/api/python/gluon/model_zoo.md @@ -26,6 +26,7 @@ In the rest of this document, we list routines provided by the `gluon.model_zoo` ```eval_rst .. currentmodule:: mxnet.gluon.model_zoo.vision .. automodule:: mxnet.gluon.model_zoo.vision + :noindex: ``` The following table summarizes the available models. @@ -232,6 +233,7 @@ The following table summarizes the available models. .. automodule:: mxnet.gluon.model_zoo.vision :members: :imported-members: + :noindex: ``` diff --git a/docs/api/python/image/image.md b/docs/api/python/image/image.md index 11fff4f43..299c54570 100644 --- a/docs/api/python/image/image.md +++ b/docs/api/python/image/image.md @@ -16,8 +16,11 @@ images provided in .. autosummary:: :nosignatures: + image.imread image.imdecode + image.imresize image.scale_down + image.copyMakeBorder image.resize_short image.fixed_crop image.random_crop @@ -56,7 +59,7 @@ Iterators support loading image from binary `Record IO` and raw image files. We use helper function to initialize augmenters ```eval_rst - .. currentmodule:: mxnet +.. currentmodule:: mxnet .. autosummary:: :nosignatures: @@ -163,8 +166,11 @@ and a list of augmenters specific for `Object detection` is provided .. autoclass:: mxnet.image.ImageIter :members: +.. automethod:: mxnet.image.imread .. automethod:: mxnet.image.imdecode +.. automethod:: mxnet.image.imresize .. automethod:: mxnet.image.scale_down +.. automethod:: mxnet.image.copyMakeBorder .. automethod:: mxnet.image.resize_short .. automethod:: mxnet.image.fixed_crop .. automethod:: mxnet.image.random_crop diff --git a/docs/api/python/io/io.md b/docs/api/python/io/io.md index 698083483..a069eae61 100644 --- a/docs/api/python/io/io.md +++ b/docs/api/python/io/io.md @@ -3,7 +3,7 @@ ## Overview This document summarizes supported data formats and iterator APIs to read the -data including +data including: ```eval_rst .. autosummary:: @@ -68,8 +68,7 @@ A detailed tutorial is available at ## Helper classes and functions - -Data structures and other iterators provided in the ``mxnet.io`` packages. +### Data structures and other iterators ```eval_rst .. autosummary:: @@ -83,7 +82,7 @@ Data structures and other iterators provided in the ``mxnet.io`` packages. io.MXDataIter ``` -Functions to read and write RecordIO files. +### Functions to read and write RecordIO files ```eval_rst .. autosummary:: @@ -95,7 +94,7 @@ Functions to read and write RecordIO files. recordio.pack_img ``` -## Develop a new iterator +## How to develop a new iterator Writing a new data iterator in Python is straightforward. Most MXNet training/inference programs accept an iterable object with ``provide_data`` @@ -133,7 +132,7 @@ Parsing and performing another pre-processing such as augmentation may be expens If performance is critical, we can implement a data iterator in C++. Refer to [src/io](https://github.com/dmlc/mxnet/tree/master/src/io) for examples. -### Change batch layout +### How to change the batch layout By default, the backend engine treats the first dimension of each data and label variable in data iterators as the batch size (i.e. `NCHW` or `NT` layout). In order to override the axis for batch size, @@ -151,10 +150,36 @@ The backend engine will recognize the index of `N` in the `layout` as the axis f +### mxnet.io - Data Iterators + ```eval_rst .. automodule:: mxnet.io - :members: + :noindex: + :members: NDArrayIter, CSVIter, LibSVMIter, ImageRecordIter, ImageRecordUInt8Iter, MNISTIter +``` + +### mxnet.io - Helper Classes & Functions + +```eval_rst +.. automodule:: mxnet.io + :noindex: + :members: DataBatch, DataDesc, DataIter, MXDataIter, PrefetchingIter, ResizeIter +``` + +### mxnet.recordio + +```eval_rst +.. currentmodule:: mxnet.recordio + .. automodule:: mxnet.recordio :members: + ``` + +```eval_rst +.. _name: mxnet.symbol.Symbol.name +.. _shape: mxnet.ndarray.NDArray.shape + +``` + diff --git a/docs/api/python/module/module.md b/docs/api/python/module/module.md index 5a874ac6d..7a86eccd8 100644 --- a/docs/api/python/module/module.md +++ b/docs/api/python/module/module.md @@ -176,7 +176,7 @@ additional functionality. We summarize them in this section. .. autosummary:: :nosignatures: - BucketModule.switch_bucket + BucketingModule.switch_bucket ``` ### Class `SequentialModule` diff --git a/docs/api/python/ndarray/contrib.md b/docs/api/python/ndarray/contrib.md index 97f38a53a..d7c9021b5 100644 --- a/docs/api/python/ndarray/contrib.md +++ b/docs/api/python/ndarray/contrib.md @@ -55,6 +55,17 @@ In the rest of this document, we list routines provided by the `ndarray.contrib` foreach while_loop cond + isinf + isfinite + isnan + index_copy + getnnz + edge_id + dgl_csr_neighbor_uniform_sample + dgl_csr_neighbor_non_uniform_sample + dgl_subgraph + dgl_adjacency + dgl_graph_compact ``` ## API Reference diff --git a/docs/api/python/ndarray/ndarray.md b/docs/api/python/ndarray/ndarray.md index 01a154405..6419c4ed4 100644 --- a/docs/api/python/ndarray/ndarray.md +++ b/docs/api/python/ndarray/ndarray.md @@ -587,15 +587,17 @@ The `ndarray` package provides several classes: .. autosummary:: :nosignatures: - mxnet.ndarray.random.uniform - mxnet.ndarray.random.normal - mxnet.ndarray.random.gamma - mxnet.ndarray.random.exponential - mxnet.ndarray.random.poisson - mxnet.ndarray.random.negative_binomial - mxnet.ndarray.random.generalized_negative_binomial - mxnet.ndarray.random.multinomial - mxnet.ndarray.random.shuffle + random.exponential + random.gamma + random.generalized_negative_binomial + random.multinomial + random.negative_binomial + random.normal + random.poisson + random.randint + random.randn + random.shuffle + random.uniform mxnet.random.seed ``` @@ -656,6 +658,7 @@ The `ndarray` package provides several classes: log_softmax relu sigmoid + erf ``` ### More @@ -706,9 +709,14 @@ The `ndarray` package provides several classes: :members: :imported-members: :special-members: - :exclude-members: CachedOp, NDArray + :exclude-members: CachedOp, NDArray, save, load + +.. automodule:: mxnet.ndarray + :noindex: + :members: save, load .. automodule:: mxnet.random + :noindex: :members: ``` diff --git a/docs/api/python/ndarray/random.md b/docs/api/python/ndarray/random.md index 4341a3ce2..60c565dd5 100644 --- a/docs/api/python/ndarray/random.md +++ b/docs/api/python/ndarray/random.md @@ -31,12 +31,14 @@ In the rest of this document, we list routines provided by the `ndarray.random` exponential gamma generalized_negative_binomial + multinomial negative_binomial normal poisson - uniform - multinomial + randint + randn shuffle + uniform mxnet.random.seed ``` @@ -51,6 +53,7 @@ In the rest of this document, we list routines provided by the `ndarray.random` .. automodule:: mxnet.random :members: + :noindex: ``` diff --git a/docs/api/python/ndarray/sparse.md b/docs/api/python/ndarray/sparse.md index 2ade059a7..608a9501a 100644 --- a/docs/api/python/ndarray/sparse.md +++ b/docs/api/python/ndarray/sparse.md @@ -582,7 +582,7 @@ We summarize the interface for each class in the following sections. :members: shape, context, dtype, stype, data, indices, indptr, copy, copyto, as_in_context, asscipy, asnumpy, asscalar, astype, tostype, slice, wait_to_read, zeros_like, round, rint, fix, floor, ceil, trunc, sin, tan, arcsin, arctan, degrees, radians, sinh, tanh, arcsinh, arctanh, expm1, log1p, sqrt, square, __neg__, sum, mean, norm, square, __getitem__, __setitem__, check_format, abs, clip, sign .. autoclass:: mxnet.ndarray.sparse.RowSparseNDArray - :members: shape, context, dtype, stype, data, indices, copy, copyto, as_in_context, asnumpy, asscalar, astype, tostype, wait_to_read, zeros_like, round, rint, fix, floor, ceil, trunc, sin, tan, arcsin, arctan, degrees, radians, sinh, tanh, arcsinh, arctanh, expm1, log1p, sqrt, square, __negative__, norm, __getitem__, __setitem__, check_format, retain, abs, clip, sign + :members: shape, context, dtype, stype, data, indices, copy, copyto, as_in_context, asnumpy, asscalar, astype, tostype, wait_to_read, zeros_like, round, rint, fix, floor, ceil, trunc, sin, tan, arcsin, arctan, degrees, radians, sinh, tanh, arcsinh, arctanh, expm1, log1p, sqrt, square, norm, __getitem__, __setitem__, check_format, retain, abs, clip, sign .. automodule:: mxnet.ndarray.sparse :members: @@ -590,6 +590,7 @@ We summarize the interface for each class in the following sections. :exclude-members: BaseSparseNDArray, RowSparseNDArray, CSRNDArray .. automodule:: mxnet.ndarray.sparse + :noindex: :members: array, zeros, empty .. automodule:: mxnet.ndarray diff --git a/docs/api/python/symbol/contrib.md b/docs/api/python/symbol/contrib.md index c0a4da54c..a0253216f 100644 --- a/docs/api/python/symbol/contrib.md +++ b/docs/api/python/symbol/contrib.md @@ -55,6 +55,14 @@ In the rest of this document, we list routines provided by the `symbol.contrib` foreach while_loop cond + index_copy + getnnz + edge_id + dgl_csr_neighbor_uniform_sample + dgl_csr_neighbor_non_uniform_sample + dgl_subgraph + dgl_adjacency + dgl_graph_compact ``` ## API Reference diff --git a/docs/api/python/symbol/random.md b/docs/api/python/symbol/random.md index 22c686ff2..1ecaf3883 100644 --- a/docs/api/python/symbol/random.md +++ b/docs/api/python/symbol/random.md @@ -31,12 +31,13 @@ In the rest of this document, we list routines provided by the `symbol.random` p exponential gamma generalized_negative_binomial + multinomial negative_binomial normal poisson - uniform - multinomial + randint shuffle + uniform mxnet.random.seed ``` @@ -48,6 +49,7 @@ In the rest of this document, we list routines provided by the `symbol.random` p .. automodule:: mxnet.symbol.random :members: + :noindex: .. automodule:: mxnet.random :members: diff --git a/docs/api/python/symbol/symbol.md b/docs/api/python/symbol/symbol.md index 7c78cbd59..9eba26180 100644 --- a/docs/api/python/symbol/symbol.md +++ b/docs/api/python/symbol/symbol.md @@ -297,8 +297,8 @@ Composite multiple symbols into a new one by an operator. Symbol.take Symbol.one_hot Symbol.pick - Symbol.ravel_multi_index - Symbol.unravel_index + ravel_multi_index + unravel_index ``` ### Get internal and output symbol @@ -577,7 +577,7 @@ Composite multiple symbols into a new one by an operator. broadcast_logical_and broadcast_logical_or broadcast_logical_xor - broadcast_logical_not + logical_not ``` ### Random sampling @@ -586,15 +586,16 @@ Composite multiple symbols into a new one by an operator. .. autosummary:: :nosignatures: - mxnet.symbol.random.uniform - mxnet.symbol.random.normal - mxnet.symbol.random.gamma - mxnet.symbol.random.exponential - mxnet.symbol.random.poisson - mxnet.symbol.random.negative_binomial - mxnet.symbol.random.generalized_negative_binomial - mxnet.symbol.random.multinomial - mxnet.symbol.random.shuffle + random.exponential + random.gamma + random.generalized_negative_binomial + random.multinomial + random.negative_binomial + random.normal + random.poisson + random.randint + random.shuffle + random.uniform mxnet.random.seed ``` @@ -657,6 +658,7 @@ Composite multiple symbols into a new one by an operator. log_softmax relu sigmoid + erf ``` ### More diff --git a/docs/api/scala/index.md b/docs/api/scala/index.md index 8b32c9fe9..f7a150019 100644 --- a/docs/api/scala/index.md +++ b/docs/api/scala/index.md @@ -19,6 +19,7 @@ See the [MXNet Scala API Documentation](docs/index.html#org.apache.mxnet.package symbol.md ``` + ## Image Classification with the Scala Infer API The Infer API can be used for single and batch image classification. More information can be found at the following locations: @@ -32,20 +33,19 @@ The Infer API can be used for single and batch image classification. More inform You can perform tensor or matrix computation in pure Scala: ```scala - scala> import org.apache.mxnet._ import org.apache.mxnet._ - scala> val arr = NDArray.ones(2, 3) - arr: org.apache.mxnet.NDArray = org.apache.mxnet.NDArray@f5e74790 + val arr = NDArray.ones(2, 3) + // arr: org.apache.mxnet.NDArray = org.apache.mxnet.NDArray@f5e74790 - scala> arr.shape - res0: org.apache.mxnet.Shape = (2,3) + arr.shape + // org.apache.mxnet.Shape = (2,3) - scala> (arr * 2).toArray - res2: Array[Float] = Array(2.0, 2.0, 2.0, 2.0, 2.0, 2.0) + (arr * 2).toArray + // Array[Float] = Array(2.0, 2.0, 2.0, 2.0, 2.0, 2.0) - scala> (arr * 2).shape - res3: org.apache.mxnet.Shape = (2,3) + (arr * 2).shape + // org.apache.mxnet.Shape = (2,3) ``` diff --git a/docs/api/scala/kvstore.md b/docs/api/scala/kvstore.md index 2157176d2..e195c4d7e 100644 --- a/docs/api/scala/kvstore.md +++ b/docs/api/scala/kvstore.md @@ -16,13 +16,13 @@ Let's consider a simple example. It initializes a (`int`, `NDArray`) pair into the store, and then pulls the value out. ```scala - scala> val kv = KVStore.create("local") // create a local kv store. - scala> val shape = Shape(2,3) - scala> kv.init(3, NDArray.ones(shape)*2) - scala> val a = NDArray.zeros(shape) - scala> kv.pull(3, out = a) - scala> a.toArray - Array[Float] = Array(2.0, 2.0, 2.0, 2.0, 2.0, 2.0) +val kv = KVStore.create("local") // create a local kv store. +val shape = Shape(2,3) +kv.init(3, NDArray.ones(shape)*2) +val a = NDArray.zeros(shape) +kv.pull(3, out = a) +a.toArray +// Array[Float] = Array(2.0, 2.0, 2.0, 2.0, 2.0, 2.0) ``` ### Push, Aggregation, and Updater @@ -30,10 +30,10 @@ a (`int`, `NDArray`) pair into the store, and then pulls the value out. For any key that's been initialized, you can push a new value with the same shape to the key, as follows: ```scala - scala> kv.push(3, NDArray.ones(shape)*8) - scala> kv.pull(3, out = a) // pull out the value - scala> a.toArray - Array[Float] = Array(8.0, 8.0, 8.0, 8.0, 8.0, 8.0) +kv.push(3, NDArray.ones(shape)*8) +kv.pull(3, out = a) // pull out the value +a.toArray +// Array[Float] = Array(8.0, 8.0, 8.0, 8.0, 8.0, 8.0) ``` The data that you want to push can be stored on any device. Furthermore, you can push multiple @@ -41,13 +41,13 @@ values into the same key, where KVStore first sums all of these values, and then pushes the aggregated value, as follows: ```scala - scala> val gpus = Array(Context.gpu(0), Context.gpu(1), Context.gpu(2), Context.gpu(3)) - scala> val b = Array(NDArray.ones(shape, gpus(0)), NDArray.ones(shape, gpus(1)), \ - scala> NDArray.ones(shape, gpus(2)), NDArray.ones(shape, gpus(3))) - scala> kv.push(3, b) - scala> kv.pull(3, out = a) - scala> a.toArray - Array[Float] = Array(4.0, 4.0, 4.0, 4.0, 4.0, 4.0) +val gpus = Array(Context.gpu(0), Context.gpu(1), Context.gpu(2), Context.gpu(3)) +val b = Array(NDArray.ones(shape, gpus(0)), NDArray.ones(shape, gpus(1)), \ +NDArray.ones(shape, gpus(2)), NDArray.ones(shape, gpus(3))) +kv.push(3, b) +kv.pull(3, out = a) +a.toArray +// Array[Float] = Array(4.0, 4.0, 4.0, 4.0, 4.0, 4.0) ``` For each push command, KVStore applies the pushed value to the value stored by an @@ -55,22 +55,22 @@ For each push command, KVStore applies the pushed value to the value stored by a control how data is merged. ```scala - scala> val updater = new MXKVStoreUpdater { - override def update(key: Int, input: NDArray, stored: NDArray): Unit = { - println(s"update on key $key") - stored += input * 2 - } - override def dispose(): Unit = {} - } - scala> kv.setUpdater(updater) - scala> kv.pull(3, a) - scala> a.toArray - Array[Float] = Array(4.0, 4.0, 4.0, 4.0, 4.0, 4.0) - scala> kv.push(3, NDArray.ones(shape)) - update on key 3 - scala> kv.pull(3, a) - scala> a.toArray - Array[Float] = Array(6.0, 6.0, 6.0, 6.0, 6.0, 6.0) +val updater = new MXKVStoreUpdater { + override def update(key: Int, input: NDArray, stored: NDArray): Unit = { + println(s"update on key $key") + stored += input * 2 + } + override def dispose(): Unit = {} + } +kv.setUpdater(updater) +kv.pull(3, a) +a.toArray +// Array[Float] = Array(4.0, 4.0, 4.0, 4.0, 4.0, 4.0) +kv.push(3, NDArray.ones(shape)) +// update on key 3 +kv.pull(3, a) +a.toArray +// Array[Float] = Array(6.0, 6.0, 6.0, 6.0, 6.0, 6.0) ``` ### Pull @@ -79,11 +79,11 @@ You've already seen how to pull a single key-value pair. Similar to the way that pull the value into several devices with a single call. ```scala - scala> val b = Array(NDArray.ones(shape, gpus(0)), NDArray.ones(shape, gpus(1)),\ - scala> NDArray.ones(shape, gpus(2)), NDArray.ones(shape, gpus(3))) - scala> kv.pull(3, outs = b) - scala> b(1).toArray - Array[Float] = Array(6.0, 6.0, 6.0, 6.0, 6.0, 6.0) +val b = Array(NDArray.ones(shape, gpus(0)), NDArray.ones(shape, gpus(1)),\ +NDArray.ones(shape, gpus(2)), NDArray.ones(shape, gpus(3))) +kv.pull(3, outs = b) +b(1).toArray +// Array[Float] = Array(6.0, 6.0, 6.0, 6.0, 6.0, 6.0) ``` ## List Key-Value Pairs @@ -92,14 +92,14 @@ All of the operations that we've discussed so far are performed on a single key. the interface for generating a list of key-value pairs. For a single device, use the following: ```scala - scala> val keys = Array(5, 7, 9) - scala> kv.init(keys, Array.fill(keys.length)(NDArray.ones(shape))) - scala> kv.push(keys, Array.fill(keys.length)(NDArray.ones(shape))) - update on key: 5 - update on key: 7 - update on key: 9 - scala> val b = Array.fill(keys.length)(NDArray.zeros(shape)) - scala> kv.pull(keys, outs = b) - scala> b(1).toArray - Array[Float] = Array(3.0, 3.0, 3.0, 3.0, 3.0, 3.0) +val keys = Array(5, 7, 9) +kv.init(keys, Array.fill(keys.length)(NDArray.ones(shape))) +kv.push(keys, Array.fill(keys.length)(NDArray.ones(shape))) +// update on key: 5 +// update on key: 7 +// update on key: 9 +val b = Array.fill(keys.length)(NDArray.zeros(shape)) +kv.pull(keys, outs = b) +b(1).toArray +// Array[Float] = Array(3.0, 3.0, 3.0, 3.0, 3.0, 3.0) ``` diff --git a/docs/api/scala/ndarray.md b/docs/api/scala/ndarray.md index 3d4bc37a1..9e87d397c 100644 --- a/docs/api/scala/ndarray.md +++ b/docs/api/scala/ndarray.md @@ -14,13 +14,13 @@ Topics: Create `mxnet.ndarray` as follows: ```scala - scala> import org.apache.mxnet._ - scala> // all-zero array of dimension 100x50 - scala> val a = NDArray.zeros(100, 50) - scala> // all-one array of dimension 256x32x128x1 - scala> val b = NDArray.ones(256, 32, 128, 1) - scala> // initialize array with contents, you can specify dimensions of array using Shape parameter while creating array. - scala> val c = NDArray.array(Array(1, 2, 3, 4, 5, 6), shape = Shape(2, 3)) +import org.apache.mxnet._ +// all-zero array of dimension 100x50 +val a = NDArray.zeros(100, 50) +// all-one array of dimension 256x32x128x1 +val b = NDArray.ones(256, 32, 128, 1) +// initialize array with contents, you can specify dimensions of array using Shape parameter while creating array. +val c = NDArray.array(Array(1, 2, 3, 4, 5, 6), shape = Shape(2, 3)) ``` This is similar to the way you use `numpy`. ## NDArray Operations @@ -30,77 +30,77 @@ We provide some basic ndarray operations, like arithmetic and slice operations. ### Arithmetic Operations ```scala - scala> import org.apache.mxnet._ - scala> val a = NDArray.zeros(100, 50) - scala> a.shape - org.apache.mxnet.Shape = (100,50) - scala> val b = NDArray.ones(100, 50) - scala> // c and d will be calculated in parallel here! - scala> val c = a + b - scala> val d = a - b - scala> // inplace operation, b's contents will be modified, but c and d won't be affected. - scala> b += d +import org.apache.mxnet._ +val a = NDArray.zeros(100, 50) +a.shape +// org.apache.mxnet.Shape = (100,50) +val b = NDArray.ones(100, 50) +// c and d will be calculated in parallel here! +val c = a + b +val d = a - b +// inplace operation, b's contents will be modified, but c and d won't be affected. +b += d ``` ### Multiplication/Division Operations ```scala - scala> import org.apache.mxnet._ - //Multiplication - scala> val ndones = NDArray.ones(2, 1) - scala> val ndtwos = ndones * 2 - scala> ndtwos.toArray - Array[Float] = Array(2.0, 2.0) - scala> (ndones * ndones).toArray - Array[Float] = Array(1.0, 1.0) - scala> (ndtwos * ndtwos).toArray - Array[Float] = Array(4.0, 4.0) - scala> ndtwos *= ndtwos // inplace - scala> ndtwos.toArray - Array[Float] = Array(4.0, 4.0) - - //Division - scala> val ndones = NDArray.ones(2, 1) - scala> val ndzeros = ndones - 1f - scala> val ndhalves = ndones / 2 - scala> ndhalves.toArray - Array[Float] = Array(0.5, 0.5) - scala> (ndhalves / ndhalves).toArray - Array[Float] = Array(1.0, 1.0) - scala> (ndones / ndones).toArray - Array[Float] = Array(1.0, 1.0) - scala> (ndzeros / ndones).toArray - Array[Float] = Array(0.0, 0.0) - scala> ndhalves /= ndhalves - scala> ndhalves.toArray - Array[Float] = Array(1.0, 1.0) +import org.apache.mxnet._ +// Multiplication +val ndones = NDArray.ones(2, 1) +val ndtwos = ndones * 2 +ndtwos.toArray +// Array[Float] = Array(2.0, 2.0) +(ndones * ndones).toArray +// Array[Float] = Array(1.0, 1.0) +(ndtwos * ndtwos).toArray +// Array[Float] = Array(4.0, 4.0) +ndtwos *= ndtwos // inplace +ndtwos.toArray +// Array[Float] = Array(4.0, 4.0) + +//Division +val ndones = NDArray.ones(2, 1) +val ndzeros = ndones - 1f +val ndhalves = ndones / 2 +ndhalves.toArray +// Array[Float] = Array(0.5, 0.5) +(ndhalves / ndhalves).toArray +// Array[Float] = Array(1.0, 1.0) +(ndones / ndones).toArray +// Array[Float] = Array(1.0, 1.0) +(ndzeros / ndones).toArray +// Array[Float] = Array(0.0, 0.0) +ndhalves /= ndhalves +ndhalves.toArray +// Array[Float] = Array(1.0, 1.0) ``` ### Slice Operations ```scala - scala> import org.apache.mxnet._ - scala> val a = NDArray.array(Array(1f, 2f, 3f, 4f, 5f, 6f), shape = Shape(3, 2)) - scala> val a1 = a.slice(1) - scala> assert(a1.shape === Shape(1, 2)) - scala> assert(a1.toArray === Array(3f, 4f)) - - scala> val a2 = arr.slice(1, 3) - scala> assert(a2.shape === Shape(2, 2)) - scala> assert(a2.toArray === Array(3f, 4f, 5f, 6f)) +import org.apache.mxnet._ +val a = NDArray.array(Array(1f, 2f, 3f, 4f, 5f, 6f), shape = Shape(3, 2)) +val a1 = a.slice(1) +assert(a1.shape === Shape(1, 2)) +assert(a1.toArray === Array(3f, 4f)) + +val a2 = arr.slice(1, 3) +assert(a2.shape === Shape(2, 2)) +assert(a2.toArray === Array(3f, 4f, 5f, 6f)) ``` ### Dot Product ```scala - scala> import org.apache.mxnet._ - scala> val arr1 = NDArray.array(Array(1f, 2f), shape = Shape(1, 2)) - scala> val arr2 = NDArray.array(Array(3f, 4f), shape = Shape(2, 1)) - scala> val res = NDArray.dot(arr1, arr2) - scala> res.shape - org.apache.mxnet.Shape = (1,1) - scala> res.toArray - Array[Float] = Array(11.0) +import org.apache.mxnet._ +val arr1 = NDArray.array(Array(1f, 2f), shape = Shape(1, 2)) +val arr2 = NDArray.array(Array(3f, 4f), shape = Shape(2, 1)) +val res = NDArray.dot(arr1, arr2) +res.shape +// org.apache.mxnet.Shape = (1,1) +res.toArray +// Array[Float] = Array(11.0) ``` ### Save and Load NDArray @@ -108,18 +108,18 @@ We provide some basic ndarray operations, like arithmetic and slice operations. You can use MXNet functions to save and load a list or dictionary of NDArrays from file systems, as follows: ```scala - scala> import org.apache.mxnet._ - scala> val a = NDArray.zeros(100, 200) - scala> val b = NDArray.zeros(100, 200) - scala> // save list of NDArrays - scala> NDArray.save("/path/to/array/file", Array(a, b)) - scala> // save dictionary of NDArrays to AWS S3 - scala> NDArray.save("s3://path/to/s3/array", Map("A" -> a, "B" -> b)) - scala> // save list of NDArrays to hdfs. - scala> NDArray.save("hdfs://path/to/hdfs/array", Array(a, b)) - scala> val from_file = NDArray.load("/path/to/array/file") - scala> val from_s3 = NDArray.load("s3://path/to/s3/array") - scala> val from_hdfs = NDArray.load("hdfs://path/to/hdfs/array") +import org.apache.mxnet._ +val a = NDArray.zeros(100, 200) +val b = NDArray.zeros(100, 200) +// save list of NDArrays +NDArray.save("/path/to/array/file", Array(a, b)) +// save dictionary of NDArrays to AWS S3 +NDArray.save("s3://path/to/s3/array", Map("A" -> a, "B" -> b)) +// save list of NDArrays to hdfs. +NDArray.save("hdfs://path/to/hdfs/array", Array(a, b)) +val from_file = NDArray.load("/path/to/array/file") +val from_s3 = NDArray.load("s3://path/to/s3/array") +val from_hdfs = NDArray.load("hdfs://path/to/hdfs/array") ``` The good thing about using the `save` and `load` interface is that you can use the format across all `mxnet` language bindings. They also already support Amazon S3 and HDFS. @@ -128,29 +128,29 @@ The good thing about using the `save` and `load` interface is that you can use t Device information is stored in the `mxnet.Context` structure. When creating NDArray in MXNet, you can use the context argument (the default is the CPU context) to create arrays on specific devices as follows: ```scala - scala> import org.apache.mxnet._ - scala> val cpu_a = NDArray.zeros(100, 200) - scala> cpu_a.context - org.apache.mxnet.Context = cpu(0) - scala> val ctx = Context.gpu(0) - scala> val gpu_b = NDArray.zeros(Shape(100, 200), ctx) - scala> gpu_b.context - org.apache.mxnet.Context = gpu(0) +import org.apache.mxnet._ +val cpu_a = NDArray.zeros(100, 200) +cpu_a.context +// org.apache.mxnet.Context = cpu(0) +val ctx = Context.gpu(0) +val gpu_b = NDArray.zeros(Shape(100, 200), ctx) +gpu_b.context +// org.apache.mxnet.Context = gpu(0) ``` Currently, we *do not* allow operations among arrays from different contexts. To manually enable this, use the `copyto` member function to copy the content to different devices, and continue computation: ```scala - scala> import org.apache.mxnet._ - scala> val x = NDArray.zeros(100, 200) - scala> val ctx = Context.gpu(0) - scala> val y = NDArray.zeros(Shape(100, 200), ctx) - scala> val z = x + y - mxnet.base.MXNetError: [13:29:12] src/ndarray/ndarray.cc:33: - Check failed: lhs.ctx() == rhs.ctx() operands context mismatch - scala> val cpu_y = NDArray.zeros(100, 200) - scala> y.copyto(cpu_y) - scala> val z = x + cpu_y +import org.apache.mxnet._ +val x = NDArray.zeros(100, 200) +val ctx = Context.gpu(0) +val y = NDArray.zeros(Shape(100, 200), ctx) +val z = x + y +// mxnet.base.MXNetError: [13:29:12] src/ndarray/ndarray.cc:33: +// Check failed: lhs.ctx() == rhs.ctx() operands context mismatch +val cpu_y = NDArray.zeros(100, 200) +y.copyto(cpu_y) +val z = x + cpu_y ``` ## Next Steps diff --git a/docs/api/scala/symbol.md b/docs/api/scala/symbol.md index 5b73ae5d6..c10d5fb60 100644 --- a/docs/api/scala/symbol.md +++ b/docs/api/scala/symbol.md @@ -20,14 +20,14 @@ You can configure the graphs either at the level of neural network layer operati The following example configures a two-layer neural network. ```scala - scala> import org.apache.mxnet._ - scala> val data = Symbol.Variable("data") - scala> val fc1 = Symbol.api.FullyConnected(Some(data), num_hidden = 128, name = "fc1") - scala> val act1 = Symbol.api.Activation(Some(fc1), "relu", "relu1") - scala> val fc2 = Symbol.api.FullyConnected(some(act1), num_hidden = 64, name = "fc2") - scala> val net = Symbol.api.SoftmaxOutput(Some(fc2), name = "out") - scala> :type net - org.apache.mxnet.Symbol + import org.apache.mxnet._ + val data = Symbol.Variable("data") + val fc1 = Symbol.api.FullyConnected(Some(data), num_hidden = 128, name = "fc1") + val act1 = Symbol.api.Activation(Some(fc1), "relu", "relu1") + val fc2 = Symbol.api.FullyConnected(some(act1), num_hidden = 64, name = "fc2") + val net = Symbol.api.SoftmaxOutput(Some(fc2), name = "out") + :type net + // org.apache.mxnet.Symbol ``` The basic arithmetic operators (plus, minus, div, multiplication) are overloaded for @@ -36,10 +36,10 @@ The basic arithmetic operators (plus, minus, div, multiplication) are overloaded The following example creates a computation graph that adds two inputs together. ```scala - scala> import org.apache.mxnet._ - scala> val a = Symbol.Variable("a") - scala> val b = Symbol.Variable("b") - scala> val c = a + b + import org.apache.mxnet._ + val a = Symbol.Variable("a") + val b = Symbol.Variable("b") + val c = a + b ``` ## Symbol Attributes @@ -54,7 +54,7 @@ For proper communication with the C++ backend, both the key and values of the at ``` data.attr("mood") - res6: Option[String] = Some(angry) + // Option[String] = Some(angry) ``` To attach attributes, you can use ```AttrScope```. ```AttrScope``` automatically adds the specified attributes to all of the symbols created within that scope. The user can also inherit this object to change naming behavior. For example: @@ -71,7 +71,7 @@ To attach attributes, you can use ```AttrScope```. ```AttrScope``` automatically val exceedScopeData = Symbol.Variable("data3") assert(exceedScopeData.attr("group") === None, "No group attr in global attr scope") -``` +``` ## Serialization @@ -83,14 +83,14 @@ Refer to [API documentation](http://mxnet.incubator.apache.org/api/scala/docs/in The following example shows how to save a symbol to an S3 bucket, load it back, and compare two symbols using a JSON string. ```scala - scala> import org.apache.mxnet._ - scala> val a = Symbol.Variable("a") - scala> val b = Symbol.Variable("b") - scala> val c = a + b - scala> c.save("s3://my-bucket/symbol-c.json") - scala> val c2 = Symbol.load("s3://my-bucket/symbol-c.json") - scala> c.toJson == c2.toJson - Boolean = true + import org.apache.mxnet._ + val a = Symbol.Variable("a") + val b = Symbol.Variable("b") + val c = a + b + c.save("s3://my-bucket/symbol-c.json") + val c2 = Symbol.load("s3://my-bucket/symbol-c.json") + c.toJson == c2.toJson + // Boolean = true ``` ## Executing Symbols @@ -101,25 +101,25 @@ handled by the high-level [Model class](model.md) and the [`fit()`] function. For neural networks used in "feed-forward", "prediction", or "inference" mode (all terms for the same thing: running a trained network), the input arguments are the -input data, and the weights of the neural network that were learned during training. +input data, and the weights of the neural network that were learned during training. To manually execute a set of symbols, you need to create an [`Executor`] object, -which is typically constructed by calling the [`simpleBind()`] method on a symbol. +which is typically constructed by calling the [`simpleBind()`] method on a symbol. ## Multiple Outputs To group the symbols together, use the [mxnet.symbol.Group](#mxnet.symbol.Group) function. ```scala - scala> import org.apache.mxnet._ - scala> val data = Symbol.Variable("data") - scala> val fc1 = Symbol.api.FullyConnected(Some(data), num_hidden = 128, name = "fc1") - scala> val act1 = Symbol.api.Activation(Some(fc1), "relu", "relu1") - scala> val fc2 = Symbol.api.FullyConnected(Some(act1), num_hidden = 64, name = "fc2") - scala> val net = Symbol.api.SoftmaxOutput(Some(fc2), name = "out") - scala> val group = Symbol.Group(fc1, net) - scala> group.listOutputs() - IndexedSeq[String] = ArrayBuffer(fc1_output, out_output) + import org.apache.mxnet._ + val data = Symbol.Variable("data") + val fc1 = Symbol.api.FullyConnected(Some(data), num_hidden = 128, name = "fc1") + val act1 = Symbol.api.Activation(Some(fc1), "relu", "relu1") + val fc2 = Symbol.api.FullyConnected(Some(act1), num_hidden = 64, name = "fc2") + val net = Symbol.api.SoftmaxOutput(Some(fc2), name = "out") + val group = Symbol.Group(fc1, net) + group.listOutputs() + // IndexedSeq[String] = ArrayBuffer(fc1_output, out_output) ``` After you get the ```group```, you can bind on ```group``` instead. diff --git a/docs/architecture/note_memory.md b/docs/architecture/note_memory.md index 6b752bdc4..8ec77c395 100644 --- a/docs/architecture/note_memory.md +++ b/docs/architecture/note_memory.md @@ -312,18 +312,9 @@ that are already optimized for big operations, you can reduce memory consumption roughly *by half*. You can reduce memory usage even more if you are optimizing a fine-grained computation network -used by symbolic libraries, such as Theano. - -Most of the ideas in this article inspired the design of _MXNet_. -We've also provided a [Memory Cost Estimation Script](https://github.com/dmlc/mxnet/tree/master/example/memcost), -which you can use to see how much memory you need under different scenarios. - -The script has an option called `forward_only`, -which shows the cost of running only the forward pass. -You will find that cost when using this option -is extremely low compared to others. -This is simply because there's more memory reuse -if you run only the forward pass. +used by symbolic libraries, such as Theano. Most of the ideas in this article inspired the design of _MXNet_. + +Also, you will notice that memory cost, for forward pass only execution, is extremely low compared to running both forward and backward pass. This is simply because there's more memory reuse if you run only the forward pass. So here are two takeaways: diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index a7632d4a6..6a37f8830 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -567,8 +567,8 @@ let's check input data shape consistency and provide output shape. ```cpp typedef TShape (*UnaryShapeFunction)(const TShape& src, const EnvArguments& env); - typedef TShape (*BinaryShapeFunction)(const TShape& const TShape& rhs,lhs, - + typedef TShape (*BinaryShapeFunction)(const TShape& lhs, + const TShape& rhs, const EnvArguments& env); ``` You can use `mshadow::TShape` to check input data shape and designate output data shape. @@ -597,6 +597,7 @@ Written explicitly, it is: inline TShape SmoothL1Shape_(const TShape& src, const EnvArguments& env) { return TShape(src); + } ``` ### Define Functions diff --git a/docs/build_version_doc/artifacts/.htaccess b/docs/build_version_doc/artifacts/.htaccess index d553ce5a8..6bf3a659c 100644 --- a/docs/build_version_doc/artifacts/.htaccess +++ b/docs/build_version_doc/artifacts/.htaccess @@ -1,28 +1,29 @@ RewriteEngine on -RewriteRule ^get_started/why_mxnet.html$ /faq/why_mxnet.html [R=301,L] -RewriteRule ^get_started.*$ /install/ [R=301,L] -RewriteRule ^how_to.*$ /faq/ [R=301,L] -RewriteRule ^api/python/symbol.html$ /api/python/symbol/symbol.html [R=301,L] -RewriteRule ^community/index.html$ /community/contribute.html [R=301,L] +RewriteRule .* - [E=default_version:/versions/master] +RewriteRule ^get_started/why_mxnet.html$ %{ENV:default_version}/faq/why_mxnet.html [R=301,L] +RewriteRule ^get_started.*$ %{ENV:default_version}/install/ [R=301,L] +RewriteRule ^how_to.*$ %{ENV:default_version}/faq/ [R=301,L] +RewriteRule ^api/python/symbol.html$ %{ENV:default_version}/api/python/symbol/symbol.html [R=301,L] +RewriteRule ^community/index.html$ %{ENV:default_version}/community/contribute.html [R=301,L] # Navigation bar redirects to latest info -RewriteRule ^versions/[^\/]+/architecture/.*$ /architecture/ [R=301,L] -RewriteRule ^versions/[^\/]+/community/.*$ /community/ [R=301,L] -RewriteRule ^versions/[^\/]+/faq/.*$ /faq/ [R=301,L] -RewriteRule ^versions/[^\/]+/gluon/.*$ /gluon/ [R=301,L] -RewriteRule ^versions/[^\/]+/install/.*$ /install/ [R=301,L] -RewriteRule ^versions/[^\/]+/tutorials/(.*)$ /tutorials/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/architecture/(.*)$ %{ENV:default_version}/architecture/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/community/(.*)$ %{ENV:default_version}/community/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/faq/(.*)$ %{ENV:default_version}/faq/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/gluon/(.*)$ %{ENV:default_version}/gluon/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/install/(.*)$ %{ENV:default_version}/install/$1 [R=301,L] +RewriteRule ^versions\/[0-9.]+\/tutorials/(.*)$ %{ENV:default_version}/tutorials/$1 [R=301,L] # Redirect navbar APIs that did not exist -RewriteRule ^versions/0.11.0/api/python/contrib/onnx.html /error/api.html [R=301,L] -RewriteRule ^versions/0.12.1/api/python/contrib/onnx.html /error/api.html [R=301,L] -RewriteRule ^versions/1.0.0/api/python/contrib/onnx.html /error/api.html [R=301,L] -RewriteRule ^versions/1.1.0/api/python/contrib/onnx.html /error/api.html [R=301,L] +RewriteRule ^versions/0.11.0/api/python/contrib/onnx.html %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/0.12.1/api/python/contrib/onnx.html %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/1.0.0/api/python/contrib/onnx.html %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/1.1.0/api/python/contrib/onnx.html %{ENV:default_version}/error/api.html [R=301,L] -RewriteRule ^versions/0.11.0/api/clojure/.*$ /error/api.html [R=301,L] -RewriteRule ^versions/0.12.1/api/clojure/.*$ /error/api.html [R=301,L] -RewriteRule ^versions/1.0.0/api/clojure/.*$ /error/api.html [R=301,L] -RewriteRule ^versions/1.1.0/api/clojure/.*$ /error/api.html [R=301,L] -RewriteRule ^versions/1.2.1/api/clojure/.*$ /error/api.html [R=301,L] +RewriteRule ^versions/0.11.0/api/clojure/.*$ %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/0.12.1/api/clojure/.*$ %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/1.0.0/api/clojure/.*$ %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/1.1.0/api/clojure/.*$ %{ENV:default_version}/error/api.html [R=301,L] +RewriteRule ^versions/1.2.1/api/clojure/.*$ %{ENV:default_version}/error/api.html [R=301,L] ErrorDocument 404 https://mxnet.incubator.apache.org/error/404.html diff --git a/docs/build_version_doc/build_all_version.sh b/docs/build_version_doc/build_all_version.sh index 5f857996f..3c432bbed 100755 --- a/docs/build_version_doc/build_all_version.sh +++ b/docs/build_version_doc/build_all_version.sh @@ -43,6 +43,9 @@ set -e set -x +# Set OPTS to any Sphinx build options, like -W for "warnings as errors" +OPTS= + # $1 is the list of branches/tags to build if [ -z "$1" ] then @@ -117,6 +120,10 @@ function checkout () { git checkout "$repo_folder" || git branch $repo_folder "upstream/$repo_folder" && git checkout "$repo_folder" || exit 1 if [ $tag == 'master' ]; then git pull + # master gets warnings as errors for Sphinx builds + OPTS="-W" + else + OPTS= fi git submodule update --init --recursive cd .. @@ -160,7 +167,7 @@ for key in ${!build_arr[@]}; do echo "Building $tag..." cd $tag/docs - make html USE_OPENMP=1 BUILD_VER=$tag || exit 1 + make html USE_OPENMP=1 BUILD_VER=$tag SPHINXOPTS=$OPTS || exit 1 # Navigate back to build_version_doc folder cd ../../../ # Use the display tag name for the folder name diff --git a/docs/community/contribute.md b/docs/community/contribute.md index 0f40ba86a..1b83638ed 100644 --- a/docs/community/contribute.md +++ b/docs/community/contribute.md @@ -62,13 +62,24 @@ To join the MXNet slack channel send request to the contributor mailing list. ### Social Media -Keep connected with the latest MXNet news and updates on [Twitter](https://twitter.com/apachemxnet) and [Reddit](https://reddit.com/r/mxnet). Also, subscribe to the [MXNet YouTube channel](https://www.youtube.com/channel/UCQua2ZAkbr_Shsgfk1LCy6A). - -
    -

    - -

    -reddit r/mxnet +Keep connected with the latest MXNet news and updates. + +

    + + Contributor and guest blogs about MXNet +

    +

    + +reddit Discuss MXNet on r/mxnet +

    +

    + + Apache MXNet on Twitter +

    +

    + + MXNet YouTube channel +

    ## JIRA diff --git a/docs/community/ecosystem.md b/docs/community/ecosystem.md index 54f8c8993..100ed973c 100644 --- a/docs/community/ecosystem.md +++ b/docs/community/ecosystem.md @@ -62,7 +62,7 @@ Community contributions to MXNet have added many new valuable features and funct ## Model Zoos -* [Gluon Model Zoo](https://github.com/awslabs/mxnet-model-server) - models trained in Gluon and available through Gluon's model zoo API. +* [Gluon Model Zoo](https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html) - models trained in Gluon and available through Gluon's model zoo API. * [ONNX Model Zoo](https://github.com/onnx/models) - ONNX models from a variety of ONNX-supported frameworks. diff --git a/docs/community/mxnet_channels.md b/docs/community/mxnet_channels.md index 98cce941e..8bc8471df 100644 --- a/docs/community/mxnet_channels.md +++ b/docs/community/mxnet_channels.md @@ -3,8 +3,8 @@ Converse with the MXNet community via the following channels: - [Forum](https://discuss.mxnet.io/): [discuss.mxnet.io](https://discuss.mxnet.io/) -- [MXNet Apache developer mailing list](https://lists.apache.org/list.html?dev@mxnet.apache.org) (dev@mxnet.apache.org): To subscribe, send an email to user-subscribe@mxnet.apache.org -- [MXNet Apache user mailing list](https://lists.apache.org/list.html?user@mxnet.apache.org) (user@mxnet.apache.org): To subscribe, send an email to dev-subscribe@mxnet.apache.org -- [MXNet Slack channel](https://the-asf.slack.com/) (Channel: #mxnet): To request an invitation to the channel please subscribe to the mailing list above and then email: dev@mxnet.apache.org +- [MXNet Apache developer mailing list](https://lists.apache.org/list.html?dev@mxnet.apache.org) (dev@mxnet.apache.org): To subscribe, send an email to dev-subscribe@mxnet.apache.org +- [MXNet Apache user mailing list](https://lists.apache.org/list.html?user@mxnet.apache.org) (user@mxnet.apache.org): To subscribe, send an email to user-subscribe@mxnet.apache.org +- [MXNet Slack channel](https://the-asf.slack.com/) (Channel: #mxnet): To request an invitation to the channel please subscribe to the dev mailing list above and then email: dev@mxnet.apache.org Note: if you have an email address with apache.org, you do not need an approval to join the MXNet Slack channel. diff --git a/docs/conf.py b/docs/conf.py index 656a1da96..c6c8204d9 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -107,7 +107,8 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = ['3rdparty', 'build_version_doc', 'virtualenv', 'api/python/model.md', 'README.md'] +exclude_patterns = ['3rdparty', 'api/python/model.md', 'build_version_doc', 'error', 'README.md', 'tutorial_utils', 'virtualenv'] + # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None diff --git a/docs/faq/add_op_in_backend.md b/docs/faq/add_op_in_backend.md index 2fcc678b8..c44a0aa05 100644 --- a/docs/faq/add_op_in_backend.md +++ b/docs/faq/add_op_in_backend.md @@ -135,7 +135,7 @@ The last line of the above code snippet is a tuple of three lists returned by `d.infer_shape()`. The first list contains all the argument shapes of `a`, `b`, and `c`. The second contains the output shape of `d`. The third one represents the shapes of auxiliary states, which is not used -in this case, and thus is empty. +in this case, and thus is empty. In this example, we only specified values for variable `a`'s first dimension and `c`'s second dimension. The `0` in shape `(2, 0)` indicates that the size of the second dimension is unknown, same meaning for shape `(0, 3)`. @@ -437,10 +437,13 @@ NNVM_REGISTER_OP(quadratic) where :math:`x` is an input tensor and all operations in the function are element-wise. -Example:: - x = [[1, 2], [3, 4]] - y = quadratic(data=x, a=1, b=2, c=3) - y = [[6, 11], [18, 27]] +Example: + + .. code-block:: python + :emphasize-lines: 1,3 + x = [[1, 2], [3, 4]] + y = quadratic(data=x, a=1, b=2, c=3) + y = [[6, 11], [18, 27]] )code" ADD_FILELINE) // 4 .set_attr_parser(ParamParser) // 5 @@ -474,8 +477,11 @@ NNVM_REGISTER_OP(_backward_quadratic) of `Op` type and save it in the operator manager and return a reference of the just created operator object. - Lines 3-4: Add description as an operator attribute -including examples of the operator. The documentation engine would extract +including examples of the operator. The documentation engine will extract this description and display it on the documentation web page. +`emphasize-lines` is optional. +For more examples and troubleshooting with doc strings, refer to the [MXNet +developer wiki's Documentation Guide](https://cwiki.apache.org/confluence/display/MXNET/Documentation+Guide). - Line 5: Set parameter struct parser for the operator. It is used for parsing the parameters `a`, `b`, and `c` input from frontend. - Line 6: Set the number of inputs for the operator. @@ -489,9 +495,9 @@ in the computational graph. MXNet would add the missing argument with name `quadratic0_data`, where the prefix `quadratic0` is the operator name appended with an index and the postfix `data` comes from the return value of the user defined `FListInputName` function. -Users still can generate an executor for the `quand_func` like the following: +Users still can generate an executor for the `quad_func` like the following: ```python -quand_exe = quand_func.simple_bind(ctx=mx.cpu(), quandratic0_data=(1,)) +quad_exe = quad_func.simple_bind(ctx=mx.cpu(), quadratic0_data=(1,)) ``` - Line 12: Register shape inference function. - Line 13: Register type inference function. @@ -630,7 +636,7 @@ python tools/flakiness_checker.py test_operator.test_quadratic_function Please note that for `check_symbolic_forward` and `check_symbolic_backward` we pass both the operator symbols and expected results for comparison, for -`check_numeric_gradient` we only pass the operator symbol, as the +`check_numeric_gradient` we only pass the operator symbol, as the `check_numeric_gradient` computes the expected value using finite difference method. Which is why it is highly recommended to add `check_numeric_gradient` test for every operator with backward function implemented as it eliminates diff --git a/docs/faq/develop_and_hack.md b/docs/faq/develop_and_hack.md index 8b7dd672e..da53fd010 100644 --- a/docs/faq/develop_and_hack.md +++ b/docs/faq/develop_and_hack.md @@ -4,5 +4,5 @@ - [Set environment variables of MXNet](env_var.md) # Other Resources -- [MXNet System Architecture Overview](http://mxnet.io/architecture/overview.html) -- [Contributor Guidelines](http://mxnet.io/community/contribute.html) \ No newline at end of file +- [MXNet System Architecture Overview](/architecture/overview.html) +- [Contributor Guidelines](/community/contribute.html) diff --git a/docs/faq/env_var.md b/docs/faq/env_var.md index 0464c73fa..98057d0d7 100644 --- a/docs/faq/env_var.md +++ b/docs/faq/env_var.md @@ -37,6 +37,12 @@ $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 * MXNET_CPU_NNPACK_NTHREADS - Values: Int ```(default=4)``` - The number of threads used for NNPACK. NNPACK package aims to provide high-performance implementations of some layers for multi-core CPUs. Checkout [NNPACK](http://mxnet.io/faq/nnpack.html) to know more about it. +* MXNET_MP_WORKER_NTHREADS + - Values: Int ```(default=1)``` + - The number of scheduling threads on CPU given to multiprocess workers. Enlarge this number allows more operators to run in parallel in individual workers but please consider reducing the overall `num_workers` to avoid thread contention (not available on Windows). +* MXNET_MP_OPENCV_NUM_THREADS + - Values: Int ```(default=0)``` + - The number of OpenCV execution threads given to multiprocess workers. OpenCV multithreading is disabled if `MXNET_MP_OPENCV_NUM_THREADS` < 1 (default). Enlarge this number may boost the performance of individual workers when executing underlying OpenCV functions but please consider reducing the overall `num_workers` to avoid thread contention (not available on Windows). ## Memory Options @@ -67,6 +73,10 @@ $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 * MXNET_GPU_MEM_POOL_ROUND_LINEAR_CUTOFF - Values: Int ```(default=24)``` - The cutoff threshold that decides the rounding strategy. Let's denote the threshold as T. If the memory size is smaller than `2 ** T` (by default, it's 2 ** 24 = 16MB), it rounds to the smallest `2 ** n` that is larger than the requested memory size; if the memory size is larger than `2 ** T`, it rounds to the next k * 2 ** T. +* MXNET_GPU_MEM_LARGE_ALLOC_ROUND_SIZE + - Values: Int ```(default=2097152)``` + - When using the naive pool type, memory allocations larger than this threshhold are rounded up to a multiple of this value. + - The default was chosen to minimize global memory fragmentation within the GPU driver. Set this to 1 to disable. ## Engine Type @@ -95,10 +105,10 @@ $env:MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0 * MXNET_KVSTORE_REDUCTION_NTHREADS - Values: Int ```(default=4)``` - The number of CPU threads used for summing up big arrays on a single machine - - This will also be used for `dist_sync` kvstore to sum up arrays from different contexts on a single machine. - - This does not affect summing up of arrays from different machines on servers. + - This will also be used for `dist_sync` kvstore to sum up arrays from different contexts on a single machine. + - This does not affect summing up of arrays from different machines on servers. - Summing up of arrays for `dist_sync_device` kvstore is also unaffected as that happens on GPUs. - + * MXNET_KVSTORE_BIGARRAY_BOUND - Values: Int ```(default=1000000)``` - The minimum size of a "big array". @@ -162,7 +172,7 @@ When USE_PROFILER is enabled in Makefile or CMake, the following environments ca * MXNET_CUDNN_AUTOTUNE_DEFAULT - Values: 0, 1, or 2 ```(default=1)``` - - The default value of cudnn auto tuning for convolution layers. + - The default value of cudnn auto tuning for convolution layers. - Value of 0 means there is no auto tuning to pick the convolution algo - Performance tests are run to pick the convolution algo when value is 1 or 2 - Value of 1 chooses the best algo in a limited workspace @@ -186,12 +196,28 @@ When USE_PROFILER is enabled in Makefile or CMake, the following environments ca * MXNET_HOME - Data directory in the filesystem for storage, for example when downloading gluon models. - Default in *nix is .mxnet APPDATA/mxnet in windows. - + * MXNET_MKLDNN_ENABLED - Values: 0, 1 ```(default=1)``` - Flag to enable or disable MKLDNN accelerator. On by default. - Only applies to mxnet that has been compiled with MKLDNN (```pip install mxnet-mkl``` or built from source with ```USE_MKLDNN=1```) +* MXNET_MKLDNN_CACHE_NUM + - Values: Int ```(default=-1)``` + - Flag to set num of elements that MKLDNN cache can hold. Default is -1 which means cache size is unbounded. Should only be set if your model has variable input shapes, as cache size may grow unbounded. The number represents the number of items in the cache and is proportional to the number of layers that use MKLDNN and different input shape. + +* MXNET_ENFORCE_DETERMINISM + - Values: 0(false) or 1(true) ```(default=0)``` + - If set to true, MXNet will only use deterministic algorithms in forward and backward computation. + If no such algorithm exists given other constraints, MXNet will error out. This variable affects the choice + of CUDNN convolution algorithms. Please see [CUDNN developer guide](https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html) for more details. + +* MXNET_CPU_PARALLEL_COPY_SIZE + - Values: Int ```(default=200000)``` + - The minimum size to call parallel copy by OpenMP in CPU2CPU mode. + - When the array size is bigger than or equal to this threshold, NDArray::Copy(from, to) is implemented by OpenMP with the Recommended OMP Thread Count. + - When the array size is less than this threshold, NDArray::Copy(from , to)) is implemented by memcpy in single thread. + Settings for Minimum Memory Usage --------------------------------- - Make sure ```min(MXNET_EXEC_NUM_TEMP, MXNET_GPU_WORKER_NTHREADS) = 1``` @@ -202,3 +228,17 @@ Settings for More GPU Parallelism - Set ```MXNET_GPU_WORKER_NTHREADS``` to a larger number (e.g., 2) - To reduce memory usage, consider setting ```MXNET_EXEC_NUM_TEMP```. - This might not speed things up, especially for image applications, because GPU is usually fully utilized even with serialized jobs. + +Settings for controlling OMP tuning +--------------------------------- +- Set ```MXNET_USE_OPERATOR_TUNING=0``` to disable Operator tuning code which decides whether to use OMP or not for operator + - Values: String representation of MXNET_ENABLE_OPERATOR_TUNING environment variable + - 0=disable all + - 1=enable all + - float32, float16, float32=list of types to enable, and disable those not listed + - refer : https://github.com/apache/incubator-mxnet/blob/master/src/operator/operator_tune-inl.h#L444 + +- Set ```MXNET_USE_NUM_CORES_OPERATOR_TUNING``` to define num_cores to be used by operator tuning code. + - This reduces operator tuning overhead when there are multiple instances of mxnet running in the system and we know that + each mxnet will take only partial num_cores available with system. + - refer: https://github.com/apache/incubator-mxnet/pull/13602 diff --git a/docs/faq/index.md b/docs/faq/index.md index 1b4a95d3f..fe91f7ca4 100644 --- a/docs/faq/index.md +++ b/docs/faq/index.md @@ -15,72 +15,72 @@ and full working examples, visit the [tutorials section](../tutorials/index.md). ## API -* [What's the difference between the Module and Gluon APIs for Python?](http://mxnet.io/api/python/index.html) +* [What's the difference between the Module and Gluon APIs for Python?](../api/python/index.html) ## Modeling -* [How do I fine-tune pre-trained models to a new dataset?](http://mxnet.io/faq/finetune.html) +* [How do I fine-tune pre-trained models to a new dataset?](finetune.html) -* [How do I work with variable-length input in MXNet (bucketing)?](http://mxnet.io/faq/bucketing.html) +* [How do I work with variable-length input in MXNet (bucketing)?](bucketing.html) -* [How do I visualize neural networks as computation graphs?](http://mxnet.io/faq/visualize_graph.html) +* [How do I visualize neural networks as computation graphs?](visualize_graph.html) ## Scale -* [How can I train with multiple CPU/GPUs on a single machine with data parallelism?](http://mxnet.io/faq/multi_devices.html) +* [How can I train with multiple CPU/GPUs on a single machine with data parallelism?](multi_devices.html) -* [How can I train using multiple machines with data parallelism?](http://mxnet.io/faq/distributed_training.html) +* [How can I train using multiple machines with data parallelism?](distributed_training.html) -* [How can I train using multiple GPUs with model parallelism?](http://mxnet.io/faq/model_parallel_lstm.html) +* [How can I train using multiple GPUs with model parallelism?](model_parallel_lstm.html) ## Speed -* [How do I use gradient compression with distributed training?](http://mxnet.io/faq/gradient_compression.html) +* [How do I use gradient compression with distributed training?](gradient_compression.html) -* [Can I use nnpack to improve the CPU performance of MXNet?](http://mxnet.io/faq/nnpack.html) +* [Can I use nnpack to improve the CPU performance of MXNet?](nnpack.html) -* [What are the best setup and data-handling tips and tricks for improving speed?](http://mxnet.io/faq/perf.html) +* [What are the best setup and data-handling tips and tricks for improving speed?](perf.html) -* [How do I use mixed precision with MXNet or Gluon?](http://mxnet.io/faq/float16.html) +* [How do I use mixed precision with MXNet or Gluon?](float16.html) ## Deployment Environments -* [Can I run MXNet on smart or mobile devices?](http://mxnet.io/faq/smart_device.html) +* [Can I run MXNet on smart or mobile devices?](smart_device.html) * [How to use data from S3 for training?](s3_integration.md) -* [How to setup MXNet on AWS?](http://docs.aws.amazon.com/mxnet/latest/dg/mxnet-on-ec2-instance.html) +* [How to run MXNet on AWS?](https://docs.aws.amazon.com/mxnet/latest/dg/whatis.html) * [How to do distributed training using MXNet on AWS?](http://docs.aws.amazon.com/mxnet/latest/dg/mxnet-on-ec2-cluster.html) -* [How do I run MXNet on a Raspberry Pi for computer vision?](http://mxnet.io/tutorials/embedded/wine_detector.html) +* [How do I run MXNet on a Raspberry Pi for computer vision?](../tutorials/embedded/wine_detector.html) * [How do I run Keras 2 with MXNet backend?](https://github.com/awslabs/keras-apache-mxnet/blob/master/docs/mxnet_backend/installation.md) * [How to convert MXNet models to Apple CoreML format?](https://github.com/apache/incubator-mxnet/tree/master/tools/coreml) ## Security -* [How to run MXNet securely?](http://mxnet.io/faq/security.html) +* [How to run MXNet securely?](security.html) ## Extend and Contribute to MXNet -* [How do I join the MXNet development discussion?](http://mxnet.io/community/mxnet_channels.html) +* [How do I join the MXNet development discussion?](../community/mxnet_channels.html) -* [How do I contribute a patch to MXNet?](http://mxnet.io/community/contribute.html) +* [How do I contribute a patch to MXNet?](../community/contribute.html) -* [How do I implement operators in MXNet backend?](http://mxnet.io/faq/add_op_in_backend.html) +* [How do I implement operators in MXNet backend?](add_op_in_backend.html) -* [How do I create new operators in MXNet?](http://mxnet.io/faq/new_op.html) +* [How do I create new operators in MXNet?](new_op.html) * [How do I implement sparse operators in MXNet backend?](https://cwiki.apache.org/confluence/display/MXNET/A+Guide+to+Implementing+Sparse+Operators+in+MXNet+Backend) * [How do I contribute an example or tutorial?](https://github.com/apache/incubator-mxnet/tree/master/example#contributing) -* [How do I set MXNet's environmental variables?](http://mxnet.io/faq/env_var.html) +* [How do I set MXNet's environmental variables?](env_var.html) ## Questions about Using MXNet If you need help with using MXNet, have questions about applying it to a particular kind of problem, or have a discussion topic, please use our [forum](https://discuss.mxnet.io). ## Issue Tracker -We track bugs and new feature requests in the MXNet Github repo in the issues folder: [mxnet/issues](https://github.com/dmlc/mxnet/issues). +We track bugs and new feature requests in the MXNet Github repo in the issues folder: [mxnet/issues](https://github.com/apache/incubator-mxnet/issues). ## Roadmap -MXNet is evolving fast. To see what's next and what we are working on internally, go to the [MXNet Roadmap](https://github.com/dmlc/mxnet/labels/Roadmap). +MXNet is evolving fast. To see what's next and what we are working on internally, go to the [MXNet Roadmap](https://cwiki.apache.org/confluence/display/MXNET/MXNet+Roadmap). diff --git a/docs/gluon/index.md b/docs/gluon/index.md index 4f6d3c10f..96e8e36db 100644 --- a/docs/gluon/index.md +++ b/docs/gluon/index.md @@ -43,7 +43,7 @@ The community is also working on parallel effort to create a foundational resour Use plug-and-play neural network building blocks, including predefined layers, optimizers, and initializers: -```python +``` net = gluon.nn.Sequential() # When instantiated, Sequential stores a chain of neural network layers. # Once presented with data, Sequential executes each layer in turn, using @@ -59,7 +59,7 @@ with net.name_scope(): Prototype, build, and train neural networks in fully imperative manner using the MXNet autograd package and the Gluon trainer method: -```python +``` epochs = 10 for e in range(epochs): @@ -76,20 +76,19 @@ for e in range(epochs): Build neural networks on the fly for use cases where neural networks must change in size and shape during model training: -```python +``` def forward(self, F, inputs, tree): children_outputs = [self.forward(F, inputs, child) for child in tree.children] #Recursively builds the neural network based on each input sentence’s #syntactic structure during the model definition and training process - … ```
    **__High Performance__** Easily cache the neural network to achieve high performance by defining your neural network with ``HybridSequential`` and calling the ``hybridize`` method: -```python +``` net = nn.HybridSequential() with net.name_scope(): net.add(nn.Dense(256, activation="relu")) @@ -97,7 +96,7 @@ with net.name_scope(): net.add(nn.Dense(2)) ``` -```python +``` net.hybridize() ``` diff --git a/docs/install/build_from_source.md b/docs/install/build_from_source.md index eff66662a..e807fb44b 100644 --- a/docs/install/build_from_source.md +++ b/docs/install/build_from_source.md @@ -1,93 +1,84 @@ # Build MXNet from Source -This document explains how to build MXNet from source code. Building MXNet from source is a two step process. - -1. Build the MXNet shared library, `libmxnet.so`, from [C++ source files](#build-the-shared-library) -2. Install the [language bindings](#installing-mxnet-language-bindings) for MXNet. MXNet supports the following languages: - - Python - - C++ - - Clojure - - Julia - - Perl - - R - - Scala +This document explains how to build MXNet from source code. + +**For Java/Scala/Clojure, please follow [this guide instead](./scala_setup.md)** + +## Overview + +Building from source follows this general two-step flow of building the shared library, then installing your preferred language binding. Use the following links to jump to the different sections of this guide. + +1. Build the MXNet shared library, `libmxnet.so`. + * [Clone the repository](#clone-the-mxnet-project) + * [Prerequisites](#prerequisites) + * [Math library selection](#math-library-selection) + * [Install GPU software](#install-gpu-software) + * [Install optional software](#install-optional-software) + * [Adjust your build configuration](#build-configurations) + * [Build MXNet](#build-mxnet) + * [with NCCL](#build-mxnet-with-nccl) (optional) + * [for C++](#build-mxnet-with-c++) (optional) + * [Usage Examples](#usage-examples) + * [systems with GPUs and Intel CPUs](#recommended-for-Systems-with-NVIDIA-GPUs-and-Intel-CPUs) + * [GPUs with non-Intel CPUs](#recommended-for-Systems-with-Intel-CPUs) + * [Intel CPUs](#recommended-for-Systems-with-Intel-CPUs) + * [non-Intel CPUs](#recommended-for-Systems-with-non-Intel-CPUs) +2. [Install the language API binding(s)](#installing-mxnet-language-bindings) you would like to use for MXNet. +MXNet's newest and most popular API is Gluon. Gluon is built into the Python binding. If Python isn't your preference, you still have more options. MXNet supports several other language APIs: + - [Python (includes Gluon)](../api/python/index.html) + - [C++](../api/c++/index.html) + - [Clojure](../api/clojure/index.html) + - [Julia](../api/julia/index.html) + - [Perl](../api/perl/index.html) + - [R](../api/r/index.html) + - [Scala](../api/scala/index.html) + - [Java](../api/java/index.html) + +
    -## Prerequisites - -You need C++ build tools and a BLAS library to build the MXNet shared library. If you want to run MXNet with GPUs, you will need to install [NVDIA CUDA and cuDNN](https://developer.nvidia.com/cuda-downloads) first. +## Build Instructions by Operating System -You may use [GNU Make](https://www.gnu.org/software/make/) to build the library but [cmake](https://cmake.org/) is required when building with MKLDNN +Detailed instructions are provided per operating system. Each of these guides also covers how to install the specific [Language Bindings](#installing-mxnet-language-bindings) you require. +You may jump to those, but it is recommended that you continue reading to understand more general "build from source" options. +* [Amazon Linux / CentOS / RHEL](centos_setup.html) +* [macOS](osx_setup.html) +* [Raspbian](raspian_setup.html) +* [TX2](tx2_setup.html) +* [Ubuntu](ubuntu_setup.html) +* [Windows](windows_setup.html) -### C++ build tools -1. A C++ compiler that supports C++ 11. -[G++ (4.8 or later)](https://gcc.gnu.org/gcc-4.8/) or -[Clang](http://clang.llvm.org/) is required. +
    -2. [Git](https://git-scm.com/downloads) for downloading the sources from Github repository. +## Clone the MXNet Project +1. Clone or fork the MXNet project. +```bash +git clone --recursive https://github.com/apache/incubator-mxnet mxnet +cd mxnet +``` +
    +## Prerequisites -### BLAS library +The following sections will help you decide which specific prerequisites you need to install. +#### Math Library Selection +It is useful to consider your math library selection prior to your other prerequisites. MXNet relies on the [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) (Basic Linear Algebra Subprograms) library for numerical computations. Those can be extended with [LAPACK (Linear Algebra Package)](https://github.com/Reference-LAPACK/lapack), an additional set of mathematical functions. MXNet supports multiple mathematical backends for computations on the CPU: - * [Apple Accelerate](https://developer.apple.com/documentation/accelerate) * [ATLAS](http://math-atlas.sourceforge.net/) * [MKL](https://software.intel.com/en-us/intel-mkl) (MKL, MKLML) * [MKL-DNN](https://github.com/intel/mkl-dnn) * [OpenBLAS](http://www.openblas.net/) -Usage of these are covered in more detail in the [build configurations](#build-configurations) section. - - -### Optional - -These might be optional, but they're typically desirable. - -* [OpenCV](http://opencv.org/) for Image Loading and Augmentation -* [NVDIA CUDA and cuDNN](https://developer.nvidia.com/cuda-downloads) for running MXNet with GPUs - - -## Build Instructions by Operating System - -Detailed instructions are provided per operating system. -You may jump to those, but it is recommended that you continue reading to understand more general build from source options. - -| | | | | -|---|---|---|---| -| [macOS](osx_setup.html) | [Ubuntu](ubuntu_setup.html) | [CentOS/*unix](centos_setup.html) | [Windows](windows_setup.html) | -| [raspbian](raspian_setup.html) | [tx2](tx2_setup.html) | | | - - - -## Build - -1. Clone the MXNet project. -```bash -git clone --recursive https://github.com/apache/incubator-mxnet mxnet -cd mxnet -``` - -There is a configuration file for make, -[`make/config.mk`](https://github.com/apache/incubator-mxnet/blob/master/make/config.mk), that contains all the compilation options. You can edit it and then run `make`. - - -## Build Configurations - -`cmake` is recommended for building MXNet (and is required to build with MKLDNN), however you may use `make` instead. - - -### Math Library Selection -It is useful to consider your math library selection first. - The default order of choice for the libraries if found follows the path from the most (recommended) to less performant backends. The following lists show this order by library and `cmake` switch. @@ -122,21 +113,21 @@ https://software.intel.com/en-us/mkl It has following flavors: * MKL is a complete math library, containing all the functionality found in ATLAS, OpenBlas and LAPACK. It is free under - community support licensing (https://software.intel.com/en-us/articles/free-mkl), - but needs to be downloaded and installed manually. +community support licensing (https://software.intel.com/en-us/articles/free-mkl), +but needs to be downloaded and installed manually. * MKLML is a subset of MKL. It contains a smaller number of functions to reduce the - size of the download and reduce the number of dynamic libraries user needs. +size of the download and reduce the number of dynamic libraries user needs. - + * MKL-DNN is a separate open-source library, it can be used separately from MKL or MKLML. It is - shipped as a subrepo with MXNet source code (see 3rdparty/mkldnn or the [MKL-DNN project](https://github.com/intel/mkl-dnn)) +shipped as a subrepo with MXNet source code (see 3rdparty/mkldnn or the [MKL-DNN project](https://github.com/intel/mkl-dnn)) Since the full MKL library is almost always faster than any other BLAS library it's turned on by default, however it needs to be downloaded and installed manually before doing `cmake` configuration. -Register and download on the [Intel performance libraries website](https://software.seek.intel.com/performance-libraries). +Register and download on the [Intel performance libraries website](https://software.intel.com/en-us/performance-libraries). Note: MKL is supported only for desktop builds and the framework itself supports the following hardware: @@ -150,6 +141,32 @@ If you have a different processor you can still try to use MKL, but performance unpredictable. +#### Install GPU Software + +If you want to run MXNet with GPUs, you must install [NVDIA CUDA and cuDNN](https://developer.nvidia.com/cuda-downloads). + + +#### Install Optional Software + +These might be optional, but they're typically desirable as the extend or enhance MXNet's functionality. + +* [OpenCV](http://opencv.org/) - Image Loading and Augmentation. Each operating system has different packages and build from source options for OpenCV. Refer to your OS's link in the [Build Instructions by Operating System](#build-instructions-by-operating-system) section for further instructions. +* [NCCL](https://developer.nvidia.com/nccl) - NVIDIA's Collective Communications Library. Instructions for installing NCCL are found in the following [Build MXNet with NCCL](#build-mxnet-with-nccl) section. + +More information on turning these features on or off are found in the following [build configurations](#build-configurations) section. + + +
    + +## Build Configurations + +There is a configuration file for make, +[`make/config.mk`](https://github.com/apache/incubator-mxnet/blob/master/make/config.mk), that contains all the compilation options. You can edit it and then run `make` or `cmake`. `cmake` is recommended for building MXNet (and is required to build with MKLDNN), however you may use `make` instead. For building with Java/Scala/Clojure, only `make` is supported. + +
    + +## Build MXNet + ### Build MXNet with NCCL - Download and install the latest NCCL library from NVIDIA. - Note the directory path in which NCCL libraries and header files are installed. @@ -183,53 +200,103 @@ nosetests --verbose tests/python/gpu/test_nccl.py **Recommendation to get the best performance out of NCCL:** It is recommended to set environment variable NCCL_LAUNCH_MODE to PARALLEL when using NCCL version 2.1 or newer. +
    + +### Build MXNet with C++ -### Build MXNet with Language Packages -* To enable C++ package, just add `USE_CPP_PACKAGE=1` when you run `make` or `cmake`. +* To enable C++ package, just add `USE_CPP_PACKAGE=1` when you run `make` or `cmake` (see examples). +
    ### Usage Examples -* `-j` runs multiple jobs against multi-core CPUs. Example using all cores on Linux: + +For example, you can specify using all cores on Linux as follows: ```bash -make -j$(nproc) +mkdir build && cd build +cmake -GNinja . +ninja -v ``` -* Build without using OpenCV: + +#### Recommended for Systems with NVIDIA GPUs and Intel CPUs +* Build MXNet with `cmake` and install with MKL DNN, GPU, and OpenCV support: ```bash -make USE_OPENCV=0 +mkdir build && cd build +cmake -DUSE_CUDA=1 -DUSE_CUDA_PATH=/usr/local/cuda -DUSE_CUDNN=1 -DUSE_MKLDNN=1 -GNinja . +ninja -v ``` +#### Recommended for Systems with NVIDIA GPUs * Build with both OpenBLAS, GPU, and OpenCV support: ```bash -make -j USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 +mkdir build && cd build +cmake -DBLAS=open -DUSE_CUDA=1 -DUSE_CUDA_PATH=/usr/local/cuda -DUSE_CUDNN=1 -GNinja . +ninja -v +``` + +#### Recommended for Systems with Intel CPUs +* Build MXNet with `cmake` and install with MKL DNN, and OpenCV support: + +```bash +mkdir build && cd build +cmake -DUSE_CUDA=0 -DUSE_MKLDNN=1 -GNinja . +ninja -v +``` + +#### Recommended for Systems with non-Intel CPUs +* Build MXNet with `cmake` and install with OpenBLAS and OpenCV support: + +```bash +mkdir build && cd build +cmake -DUSE_CUDA=0 -DBLAS=open -GNinja . +ninja -v +``` + +#### Other Examples + +* Build without using OpenCV: + +```bash +mkdir build && cd build +cmake -DUSE_OPENCV=0 -GNinja . +ninja -v ``` * Build on **macOS** with the default BLAS library (Apple Accelerate) and Clang installed with `xcode` (OPENMP is disabled because it is not supported by the Apple version of Clang): ```bash -make -j USE_BLAS=apple USE_OPENCV=0 USE_OPENMP=0 +mkdir build && cd build +cmake -DBLAS=apple -DUSE_OPENCV=0 -DUSE_OPENMP=0 -GNinja . +ninja -v ``` * To use OpenMP on **macOS** you need to install the Clang compiler, `llvm` (the one provided by Apple does not support OpenMP): ```bash brew install llvm -make -j USE_BLAS=apple USE_OPENMP=1 +mkdir build && cd build +cmake -DBLAS=apple -DUSE_OPENMP=1 -GNinja . +ninja -v ``` +
    + ## Installing MXNet Language Bindings -After building MXNet's shared library, you can install other language bindings. (Except for C++. You need to build this when you build MXNet from source.) +After building MXNet's shared library, you can install other language bindings. + +**NOTE:** The C++ API binding must be built when you build MXNet from source. See [Build MXNet with C++](#build-mxnet-with-c++). The following table provides links to each language binding by operating system: -| | Linux | macOS | Windows | -|---|---|---|---| -| Python | [Linux](ubuntu_setup.html#install-mxnet-for-python) | [macOS](osx_setup.html) | [Windows](windows_setup.html#install-mxnet-for-python) | -| C++ | [Linux](c_plus_plus.html) | [macOS](c_plus_plus.html) | [Windows](c_plus_plus.html) | -| Clojure | [Linux](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | [macOS](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | n/a | -| Julia | [Linux](ubuntu_setup.html#install-the-mxnet-package-for-julia) | [macOS](osx_setup.html#install-the-mxnet-package-for-julia) | [Windows](windows_setup.html#install-the-mxnet-package-for-julia) | -| Perl | [Linux](ubuntu_setup.html#install-the-mxnet-package-for-perl) | [macOS](osx_setup.html#install-the-mxnet-package-for-perl) | [Windows](n/a) | -| R | [Linux](ubuntu_setup.html#install-the-mxnet-package-for-r) | [macOS](osx_setup.html#install-the-mxnet-package-for-r) | [Windows](windows_setup.html#install-the-mxnet-package-for-r) | -| Scala | [Linux](scala_setup.html) | [macOS](scala_setup.html) | n/a | +| | [Ubuntu](ubuntu_setup.html) | [macOS](osx_setup.html) | [Windows](windows_setup.html) | +| --- | ---- | --- | ------- | +| Python | [Ubuntu guide](ubuntu_setup.html#install-mxnet-for-python) | [OSX guide](osx_setup.html) | [Windows guide](windows_setup.html#install-mxnet-for-python) | +| C++ | [C++ guide](c_plus_plus.html) | [C++ guide](c_plus_plus.html) | [C++ guide](c_plus_plus.html) | +| Clojure | [Clojure guide](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | [Clojure guide](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | n/a | +| Julia | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-julia) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-julia) | [Windows guide](windows_setup.html#install-the-mxnet-package-for-julia) | +| Perl | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-perl) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-perl) | n/a | +| R | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-r) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-r) | [Windows guide](windows_setup.html#install-the-mxnet-package-for-r) | +| Scala | [Scala guide](scala_setup.html) | [Scala guide](scala_setup.html) | n/a | +| Java | [Java guide](java_setup.html) | [Java Guide](java_setup.html) | n/a | diff --git a/docs/install/c_plus_plus.md b/docs/install/c_plus_plus.md index 6078877c2..6ad67e280 100644 --- a/docs/install/c_plus_plus.md +++ b/docs/install/c_plus_plus.md @@ -6,7 +6,8 @@ To enable C++ package, just add `USE_CPP_PACKAGE=1` in the [build from source](b For example to build MXNet with GPU support and the C++ package, OpenCV, and OpenBLAS, from the project root you would run: ```bash -make -j USE_CPP_PACKAGE=1 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 +cmake -DUSE_CUDA=1 -DUSE_CUDA_PATH=/usr/local/cuda -DUSE_CUDNN=1 -DUSE_MKLDNN=1 -DUSE_CPP_PACKAGE=1 -GNinja . +ninja -v ``` You may also want to add the MXNet shared library to your `LD_LIBRARY_PATH`: diff --git a/docs/install/centos_setup.md b/docs/install/centos_setup.md index f63099bcf..e5efe42a6 100644 --- a/docs/install/centos_setup.md +++ b/docs/install/centos_setup.md @@ -1,12 +1,12 @@ # Installing MXNet on CentOS and other non-Ubuntu Linux systems -1. Install build tools and git on `CentOS >= 7` and `Fedora >= 19`: +Step 1. Install build tools and git on `CentOS >= 7` and `Fedora >= 19`: ```bash sudo yum groupinstall -y "Development Tools" && sudo yum install -y git ``` -2. Install Atlas: +Step 2. Install Atlas: ```bash sudo yum install atlas-devel @@ -15,7 +15,7 @@ sudo yum install atlas-devel Installing both `git` and `cmake` or `make` by following instructions on the websites is straightforward. Here we provide the instructions to build `gcc-4.8` from source codes. -3. Install the 32-bit `libc` with one of the following system-specific commands: +Step 3. Install the 32-bit `libc` with one of the following system-specific commands: ```bash sudo apt-get install libc6-dev-i386 # In Ubuntu @@ -24,7 +24,7 @@ sudo yum install glibc-devel.i386 # In CentOS 5.8 sudo yum install glibc-devel.i686 # In CentOS 6/7 ``` -4. Download and extract the `gcc` source code with the prerequisites: +Step 4. Download and extract the `gcc` source code with the prerequisites: ```bash wget http://mirrors.concertpass.com/gcc/releases/gcc-4.8.5/gcc-4.8.5.tar.gz @@ -33,7 +33,7 @@ cd gcc-4.8.5 ./contrib/download_prerequisites ``` -5. Build `gcc` by using 10 threads and then install to `/usr/local` +Step 5. Build `gcc` by using 10 threads and then install to `/usr/local` ```bash mkdir release && cd release @@ -42,49 +42,49 @@ make -j10 sudo make install ``` -6. Add the lib path to your configure file such as `~/.bashrc`: +Step 6. Add the lib path to your configure file such as `~/.bashrc`: ```bash export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib64 ``` -7. Build [OpenBLAS from source](https://github.com/xianyi/OpenBLAS#installation-from-source). +Step 7. Build [OpenBLAS from source](https://github.com/xianyi/OpenBLAS#installation-from-source). -8. Build OpenCV +Step 8. Build OpenCV To build OpenCV from source code, you need the [cmake](https://cmake.org) library. * If you don't have cmake or if your version of cmake is earlier than 3.6.1, run the following commands to install a newer version of cmake: - ```bash - wget https://cmake.org/files/v3.6/cmake-3.6.1-Linux-x86_64.tar.gz - tar -zxvf cmake-3.6.1-Linux-x86_64.tar.gz - alias cmake="cmake-3.6.1-Linux-x86_64/bin/cmake" - ``` +```bash +wget https://cmake.org/files/v3.6/cmake-3.6.1-Linux-x86_64.tar.gz +tar -zxvf cmake-3.6.1-Linux-x86_64.tar.gz +alias cmake="cmake-3.6.1-Linux-x86_64/bin/cmake" +``` * To download and extract the OpenCV source code, run the following commands: - ```bash - wget https://codeload.github.com/opencv/opencv/zip/2.4.13 - unzip 2.4.13 - cd opencv-2.4.13 - mkdir release - cd release/ - ``` +```bash +wget https://codeload.github.com/opencv/opencv/zip/2.4.13 +unzip 2.4.13 +cd opencv-2.4.13 +mkdir release +cd release/ +``` * Build OpenCV. The following commands build OpenCV with 10 threads. We - disabled GPU support, which might significantly slow down an MXNet program - running on a GPU processor. It also disables 1394 which might generate a - warning. Then install it on `/usr/local`. +disabled GPU support, which might significantly slow down an MXNet program +running on a GPU processor. It also disables 1394 which might generate a +warning. Then install it on `/usr/local`. - ```bash - cmake -D BUILD_opencv_gpu=OFF -D WITH_CUDA=OFF -D WITH_1394=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. - make -j10 - sudo make install - ``` +```bash +cmake -D BUILD_opencv_gpu=OFF -D WITH_CUDA=OFF -D WITH_1394=OFF -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local .. +make -j10 +sudo make install +``` * Add the lib path to your configuration such as `~/.bashrc`. - ```bash - export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig/ - ``` +```bash +export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig/ +``` diff --git a/docs/install/download.md b/docs/install/download.md index b61b815e1..d7b9440e0 100644 --- a/docs/install/download.md +++ b/docs/install/download.md @@ -4,7 +4,8 @@ These source archives are generated from tagged releases. Updates and patches wi | Version | Source | PGP | SHA | |---------|-------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| -| 1.3.0 | [Download](https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz) | [Download](https://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.asc) | [Download](https://apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.sha512) | +| 1.3.1 | [Download](https://www.apache.org/dyn/closer.cgi/incubator/mxnet/1.3.1/apache-mxnet-src-1.3.1-incubating.tar.gz) | [Download](https://apache.org/dist/incubator/mxnet/1.3.1/apache-mxnet-src-1.3.1-incubating.tar.gz.asc) | [Download](https://apache.org/dist/incubator/mxnet/1.3.1/apache-mxnet-src-1.3.1-incubating.tar.gz.sha512) | +| 1.3.0 | [Download](https://archive.apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.asc) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.3.0/apache-mxnet-src-1.3.0-incubating.tar.gz.sha512) | | 1.2.1 | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz.asc) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.1/apache-mxnet-src-1.2.1-incubating.tar.gz.sha512) | | 1.2.0 | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz.asc) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.2.0/apache-mxnet-src-1.2.0-incubating.tar.gz.sha512) | | 1.1.0 | [Download](https://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz.asc) | [Download](https://archive.apache.org/dist/incubator/mxnet/1.1.0/apache-mxnet-src-1.1.0-incubating.tar.gz.sha512) | diff --git a/docs/install/index.md b/docs/install/index.md index fc136978a..319e72a32 100644 --- a/docs/install/index.md +++ b/docs/install/index.md @@ -1,12 +1,31 @@ # Installing MXNet +```eval_rst +.. toctree:: + :hidden: + + amazonlinux_setup.md + build_from_source.md + c_plus_plus.md + centos_setup.md + download.md + java_setup.md + osx_setup.md + raspbian_setup.md + scala_setup.md + tx2_setup.md + ubuntu_setup.md + validate_mxnet.md + windows_setup.md +``` + Indicate your preferred configuration. Then, follow the customized commands to install MXNet.
    -
      -
    • v1.3.0
      • +
    • v1.3.1
    • v1.2.1
    • v1.1.0
    • v1.0.0
      • @@ -34,6 +53,8 @@ Indicate your preferred configuration. Then, follow the customized commands to i
      + + @@ -79,13 +100,13 @@ Indicate your preferred configuration. Then, follow the customized commands to i
      -
      +
      ``` $ pip install mxnet ``` -
      +
      ``` @@ -144,7 +165,7 @@ $ pip install mxnet --pre MXNet offers MKL pip packages that will be much faster when running on Intel hardware. Check the chart below for other options, refer to PyPI for other MXNet pip packages, or validate your MXNet installation. -pip packages +pip packages **NOTES:** @@ -213,13 +234,13 @@ To build from source, refer to the MXNet Ubuntu inst
      -
      +
      ``` $ pip install mxnet-cu92 ``` -
      +
      ``` @@ -271,7 +292,7 @@ $ pip install mxnet-cu92 --pre MXNet offers MKL pip packages that will be much faster when running on Intel hardware. Check the chart below for other options, refer to PyPI for other MXNet pip packages, or validate your MXNet installation. -pip packages +pip packages **NOTES:** @@ -382,7 +403,7 @@ $ make rpkg

      -You can use the Maven packages defined in the following `dependency` to include MXNet in your Scala project. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. +You can use the Maven packages defined in the following dependency to include MXNet in your Scala project. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. maven badge @@ -397,7 +418,7 @@ You can use the Maven packages defined in the following `dependency` to include

      -You can use the Maven packages defined in the following `dependency` to include MXNet in your Scala project. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. +You can use the Maven packages defined in the following dependency to include MXNet in your Scala project. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. maven badge @@ -412,6 +433,81 @@ You can use the Maven packages defined in the following `dependency` to include
      +
      +
      +
      + +You can use the Maven packages defined in the following dependency to include MXNet in your Clojure project. To maximize leverage, the Clojure package has been built on the existing Scala package. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process that is required to use the Clojure dependency. + +maven badge + +```html + + org.apache.mxnet.contrib.clojure + clojure-mxnet-linux-gpu + +``` + +
      +
      +
      +
      +You can use the Maven packages defined in the following dependency to include MXNet in your Clojure project. To maximize leverage, the Clojure package has been built on the existing Scala package. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process that is required to use the Clojure dependency. + +maven badge + +```html + + org.apache.mxnet.contrib.clojure + clojure-mxnet-linux-cpu + +``` + +
      +
      +
      + + +
      +
      +
      +You can use the Maven packages defined in the following dependency to include MXNet in your Java project. The Java API is provided as a subset of the Scala API and is intended for inference only. Please refer to the MXNet-Java setup guide for a detailed set of instructions to help you with the setup process. + +maven badge + +```html + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-gpu + system + 1.4.0 + /system/path/to/jar/mxnet-full_2.11-linux-x86_64-gpu-1.4.0-SNAPSHOT.jar + +``` + +
      +
      + +
      +
      +You can use the Maven packages defined in the following dependency to include MXNet in your Java project. The Java API is provided as a subset of the Scala API and is intended for inference only. Please refer to the MXNet-Java setup guide for a detailed set of instructions to help you with the setup process. + +maven badge + +```html + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-cpu + system + 1.4.0 + /system/path/to/jar/mxnet-full_2.11-linux-x86_64-cpu-1.4.0-SNAPSHOT.jar + +``` +
      +
      +
      + +

      @@ -439,7 +535,6 @@ Refer to the Perl
      -For more installation options, refer to the       MXNet Ubuntu installation guide.
      @@ -450,13 +545,13 @@ For more installation options, refer to the MXNet Ub
      -
      +
      ``` $ pip install mxnet ``` -
      +
      ``` @@ -511,7 +606,7 @@ $ pip install mxnet --pre MXNet offers MKL pip packages that will be much faster when running on Intel hardware. Check the chart below for other options, refer to PyPI for other MXNet pip packages, or validate your MXNet installation. -pip packages +pip packages **NOTES:** @@ -597,17 +692,18 @@ MXNet developers should refer to the MXNet wiki's

      -Install OpenCV and OpenBLAS. + +To run MXNet you also should have OpenCV and OpenBLAS installed. You may install them with `brew` as follows: ```bash brew install opencv -brew install openblas@0.3.1 +brew install openblas ``` -Add a soft link to the OpenBLAS installation. This example links the 0.3.1 version: +To ensure MXNet R package runs with the version of OpenBLAS installed, create a symbolic link as follows: ```bash -ln -sf /usr/local/opt/openblas/lib/libopenblasp-r0.3.* /usr/local/opt/openblas/lib/libopenblasp-r0.3.1.dylib +ln -sf /usr/local/opt/openblas/lib/libopenblas.dylib /usr/local/opt/openblas/lib/libopenblasp-r0.3.1.dylib ``` Install the latest version (3.5.1+) of R from [CRAN](https://cran.r-project.org/bin/macosx/). @@ -633,7 +729,7 @@ Will be available soon.

      -You can use the Maven packages defined in the following `dependency` to include MXNet in your Scala project. Please refer to the       MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. +You can use the Maven packages defined in the following dependency to include MXNet in your Scala project. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process. maven badge @@ -653,6 +749,77 @@ Not available at this time.
      +
      +
      +
      +You can use the Maven packages defined in the following dependency to include MXNet in your Clojure project. To maximize leverage, the Clojure package has been built on the existing Scala package. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process that is required to use the Clojure dependency. + +maven badge + +```html + + org.apache.mxnet.contrib.clojure + clojure-mxnet-osx-cpu + +``` + +
      +
      +
      +Not available at this time.
      +
      +
      + + +
      +
      +
      +You can use the Maven packages defined in the following dependency to include MXNet in your Java project. The Java API is provided as a subset of the Scala API and is intended for inference only. Please refer to the MXNet-Java setup guide for a detailed set of instructions to help you with the setup process. + +maven badge + +```html + + org.apache.mxnet + mxnet-full_2.11-osx-x86_64-cpu + system + 1.4.0 + /system/path/to/jar/mxnet-full_2.11-osx-x86_64-cpu-1.4.0-SNAPSHOT.jar + +``` +
      +
      +
      + +Not available at this time.
      + +
      +
      + + +
      +
      +
      +You can use the Maven packages defined in the following `dependency` to include MXNet in your Clojure project. To maximize leverage, the Clojure package has been built on the existing Scala package. Please refer to the MXNet-Scala setup guide for a detailed set of instructions to help you with the setup process that is required to use the Clojure dependency. + +maven badge + +```html + + org.apache.mxnet.contrib.clojure + clojure-mxnet-osx-cpu + +``` +
      +
      +
      + +Not available at this time.
      + +
      +
      + +
      @@ -688,13 +855,13 @@ For more installation options, refer to the MXNet macOS
      -
      +
      ``` $ pip install mxnet ``` -
      +
      ``` @@ -746,7 +913,7 @@ $ pip install mxnet --pre MXNet offers MKL pip packages that will be much faster when running on Intel hardware. Check the chart below for other options, refer to PyPI for other MXNet pip packages, or validate your MXNet installation. -pip packages +pip packages **NOTES:** @@ -757,7 +924,7 @@ All MKL pip packages are experimental prior to version 1.3.0.
      -
      +

      Docker images with *MXNet* are available at [Docker Hub](https://hub.docker.com/r/mxnet/). @@ -800,19 +967,25 @@ mxnet/python 1.3.0_cpu_mkl deaf9bf61d29 4 days ago **Step 4** Validate the installation. -
      +
      + +
      +
      +Refer to the MXNet Windows installation guide + +
      -
      +
      ``` $ pip install mxnet-cu92 ``` -
      +
      ``` @@ -864,7 +1037,7 @@ $ pip install mxnet-cu92 --pre MXNet offers MKL pip packages that will be much faster when running on Intel hardware. Check the chart below for other options, refer to PyPI for other MXNet pip packages, or validate your MXNet installation. -pip packages +pip packages **NOTES:** @@ -886,7 +1059,7 @@ Refer to [#8671](https://github.com/apache/incubator-mxnet/issues/8671) for stat You can either upgrade your CUDA install or install the MXNet package that supports your CUDA version. -
      +

      @@ -894,7 +1067,7 @@ You can either upgrade your CUDA install or install the MXNet package that suppo To build from source, refer to the MXNet Windows installation guide. -
      +
      @@ -915,6 +1088,8 @@ options(repos = cran) install.packages("mxnet") ``` +To run MXNet you also should have OpenCV and OpenBLAS installed. +
      @@ -941,6 +1116,22 @@ MXNet-Scala for Windows is not yet available.
      +
      +
      +
      +MXNet-Clojure for Windows is not yet available. +
      +
      +
      + +
      +
      +
      +MXNet-Java for Windows is not yet available. +
      +
      +
      +

      @@ -1006,6 +1197,9 @@ Follow the four steps in this [docker documentation](https://docs.docker.com/eng ## Build +**Please use a Native build with gcc 4 as explained below, higher compiler versions currently cause test +failures on ARM** + The following command will build a container with dependencies and tools and then compile MXNet for ARMv7. The resulting artifact will be located in `build/mxnet-x.x.x-py2.py3-none-any.whl`, copy this file to your Raspberry Pi. diff --git a/docs/install/java_setup.md b/docs/install/java_setup.md new file mode 100644 index 000000000..0075e9205 --- /dev/null +++ b/docs/install/java_setup.md @@ -0,0 +1,123 @@ +# Setup the MXNet Package for Java + +The following instructions are provided for macOS and Ubuntu. Windows is not yet available. + +**Note:** If you use IntelliJ or a similar IDE, you may want to follow the [MXNet-Java on IntelliJ tutorial](../tutorials/java/mxnet_java_on_intellij.html) instead of these instructions. + +
      + +## Maven + +### Setup Instructions + +**Step 1.** Install dependencies: + +**macOS Steps** + +```bash +brew update +brew tap caskroom/versions +brew cask install java8 +brew install opencv +brew install maven +``` + +**Ubuntu Steps** + +These scripts will install Maven and its dependencies. You will be running the Scala scripts because the MXNet-Java project has a dependency on the MXNet-Scala project. + +```bash +wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh +wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_scala.sh +chmod +x ubuntu_core.sh +chmod +x ubuntu_scala.sh +sudo ./ubuntu_core.sh +sudo ./ubuntu_scala.sh +``` + +**Step 2.** Run the demo MXNet-Java project. + +Go to the [MXNet-Java demo project's README](https://github.com/apache/incubator-mxnet/tree/master/scala-package/mxnet-demo/java-demo) and follow the directions to test the MXNet-Java package installation. + +#### Maven Repository + +MXNet-Java can be easily included in your Maven managed project. The Java packages are currently available as nightly builds on Maven. Add the following Maven repository to your `pom.xml` to fetch the Java packages : + +```html + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + +``` + +Also, add the dependency which corresponds to your platform to the `dependencies` tag : + +**Linux CPU** +```html + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-cpu + 1.4.0-SNAPSHOT + +``` + +**Linux GPU** +```html + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-gpu + 1.4.0-SNAPSHOT + +``` + +**macOS CPU** +```html + + org.apache.mxnet + mxnet-full_2.11-osx-x86_64-cpu + 1.4.0-SNAPSHOT + +``` + + +The official Java Packages will be released with the release of MXNet 1.4 and will be available on [MXNet Maven package repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.mxnet%22). +
      + +## Source + +The previously mentioned setup with Maven is recommended. Otherwise, the following instructions for macOS and Ubuntu are provided for reference only: + +**If you have already built mxnet from source using `cmake`, run `make clean` and then follow the appropriate guide below*** + +| OS | Step 1 | Step 2 | +|---|---|---| +|macOS | [Shared Library for macOS](../install/osx_setup.html#build-the-shared-library) | [Scala Package for macOS](http://mxnet.incubator.apache.org/install/osx_setup.html#install-the-mxnet-package-for-scala) | +| Ubuntu | [Shared Library for Ubuntu](../install/ubuntu_setup.html#installing-mxnet-on-ubuntu) | [Scala Package for Ubuntu](http://mxnet.incubator.apache.org/install/ubuntu_setup.html#install-the-mxnet-package-for-scala) | +| Windows | | Call for Contribution | + + +#### Build Java from an Existing MXNet Installation +If you have already built MXNet **from source** and are looking to setup Java from that point, you may simply run the following from the MXNet source root: + +``` +make scalapkg +make scalainstall +``` +This will install both the Java Inference API and the required MXNet-Scala package. +
      + +## Documentation + +Javadocs are generated as part of the docs build pipeline. You can find them published in the [Java API](../api/java/index.html) section of the website or by going to the [scaladocs output](https://mxnet.incubator.apache.org/api/scala/docs/index.html#org.apache.mxnet.package) directly. + +To build the docs yourself, follow the [developer build docs instructions](https://github.com/apache/incubator-mxnet/tree/master/docs/build_version_doc#developer-instructions). + +
      + +## Resources + +* [Java API](../api/java/index.html) +* [javadocs](../api/java/docs/index.html#org.apache.mxnet.package) +* [MXNet-Java Tutorials](../../tutorials/index.html#java-tutorials) diff --git a/docs/install/osx_setup.md b/docs/install/osx_setup.md index 4e9293efc..7d90d3d45 100644 --- a/docs/install/osx_setup.md +++ b/docs/install/osx_setup.md @@ -83,7 +83,7 @@ After you have installed the dependencies, pull the MXNet source code from Git a The file called ```osx.mk``` has the configuration required for building MXNet on OS X. First copy ```make/osx.mk``` into ```config.mk```, which is used by the ```make``` command: ```bash - git clone --recursive https://github.com/dmlc/mxnet ~/mxnet + git clone --recursive https://github.com/apache/incubator-mxnet ~/mxnet cd ~/mxnet cp make/osx.mk ./config.mk echo "USE_BLAS = openblas" >> ./config.mk @@ -96,7 +96,14 @@ The file called ```osx.mk``` has the configuration required for building MXNet o To build with MKLDNN ```bash -LIBRARY_PATH=$(brew --prefix llvm)/lib/ make -j $(sysctl -n hw.ncpu) CC=$(brew --prefix llvm)/bin/clang++ CXX=$(brew --prefix llvm)/bin/clang++ USE_OPENCV=1 USE_OPENMP=1 USE_MKLDNN=1 USE_BLAS=apple USE_PROFILER=1 +echo "CC=$(brew --prefix llvm)/bin/clang" >> ./config.mk +echo "CXX=$(brew --prefix llvm)/bin/clang++" >> ./config.mk +echo "USE_OPENCV=1" >> ./config.mk +echo "USE_OPENMP=1" >> ./config.mk +echo "USE_MKLDNN=1" >> ./config.mk +echo "USE_BLAS=apple" >> ./config.mk +echo "USE_PROFILER=1" >> ./config.mk +LIBRARY_PATH=$(brew --prefix llvm)/lib/ make -j $(sysctl -n hw.ncpu) ``` If building with ```GPU``` support, add the following configuration to config.mk and build: diff --git a/docs/install/scala_setup.md b/docs/install/scala_setup.md index 0dadd8bca..98e752b21 100644 --- a/docs/install/scala_setup.md +++ b/docs/install/scala_setup.md @@ -79,11 +79,13 @@ https://mvnrepository.com/artifact/org.apache.mxnet The previously mentioned setup with Maven is recommended. Otherwise, the following instructions for macOS, Ubuntu, and Windows are provided for reference only: +**If you have already built mxnet from source using `cmake`, run `make clean` and then follow the appropriate guide below*** + | OS | Step 1 | Step 2 | |---|---|---| |macOS | [Shared Library for macOS](http://mxnet.incubator.apache.org/install/osx_setup.html#build-the-shared-library) | [Scala Package for macOS](http://mxnet.incubator.apache.org/install/osx_setup.html#install-the-mxnet-package-for-scala) | | Ubuntu | [Shared Library for Ubuntu](http://mxnet.incubator.apache.org/install/ubuntu_setup.html#installing-mxnet-on-ubuntu) | [Scala Package for Ubuntu](http://mxnet.incubator.apache.org/install/ubuntu_setup.html#install-the-mxnet-package-for-scala) | -| Windows | [Shared Library for Windows](http://mxnet.incubator.apache.org/install/windows_setup.html#build-the-shared-library) | Call for Contribution | +| Windows | | Call for Contribution | #### Build Scala from an Existing MXNet Installation diff --git a/docs/install/ubuntu_setup.md b/docs/install/ubuntu_setup.md index 804887aee..bf964182b 100644 --- a/docs/install/ubuntu_setup.md +++ b/docs/install/ubuntu_setup.md @@ -14,6 +14,7 @@ The following installation instructions are for installing MXNet on computers ru * [R](#install-the-mxnet-package-for-r) * [Julia](#install-the-mxnet-package-for-julia) * [Scala](#install-the-mxnet-package-for-scala) + * [Java](#install-the-mxnet-package-for-java) * [Perl](#install-the-mxnet-package-for-perl) * [Contributions](#contributions) * [Next Steps](#next-steps) @@ -78,20 +79,8 @@ Alternatively, you can use the table below to select the package that suits your #### pip Package Availability The following table presents the pip packages that are recommended for each version of MXNet. - - -![pip package table](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/install/pip-packages-1.3.0.png) + +![pip package table](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/install/pip-packages.png) To install an older version of MXNet with one of the packages in the previous table add `==` with the version you require. For example for version 1.1.0 of MXNet with CUDA 8, you would use `pip install mxnet-cu80==1.1.0`. @@ -100,7 +89,7 @@ To install an older version of MXNet with one of the packages in the previous ta ## Build MXNet from Source -You can build MXNet from source, and then you have the option of installing language-specific bindings, such as Scala, Julia, R or Perl. This is a two-step process: +You can build MXNet from source, and then you have the option of installing language-specific bindings, such as Scala, Java, Julia, R or Perl. This is a two-step process: 1. Build the shared library from the MXNet C++ source code. 2. (optional) Install the supported language-specific packages for MXNet. Be sure to check that section first, as some scripts may be available to handle all of the dependencies, MXNet build, and language bindings for you. Here they are again for quick access: @@ -108,6 +97,7 @@ You can build MXNet from source, and then you have the option of installing lang * [R](#install-the-mxnet-package-for-r) * [Julia](#install-the-mxnet-package-for-julia) * [Scala](#install-the-mxnet-package-for-scala) +* [Java](#install-the-mxnet-package-for-java) * [Perl](#install-the-mxnet-package-for-perl) **Note:** To change the compilation options for your build, edit the ```make/config.mk``` file prior to building MXNet. More information on this is mentioned in the different language package instructions. @@ -162,24 +152,35 @@ If building on CPU and using OpenBLAS: ```bash git clone --recursive https://github.com/apache/incubator-mxnet.git - cd mxnet - make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas + cd incubator-mxnet + echo "USE_OPENCV = 1" >> ./config.mk + echo "USE_BLAS = openblas" >> ./config.mk + make -j $(nproc) ``` If building on CPU and using MKL and MKL-DNN (make sure MKL is installed according to [Math Library Selection](build_from_source.html#math-library-selection) and [MKL-DNN README](https://github.com/apache/incubator-mxnet/blob/master/MKLDNN_README.md)): ```bash git clone --recursive https://github.com/apache/incubator-mxnet.git - cd mxnet - make -j $(nproc) USE_OPENCV=1 USE_BLAS=mkl USE_MKLDNN=1 + cd incubator-mxnet + echo "USE_OPENCV = 1" >> ./config.mk + echo "USE_BLAS = openblas" >> ./config.mk + echo "USE_CUDA = 0" >> ./config.mk + echo "USE_MKLDNN = 1" >> ./config.mk + make -j $(nproc) ``` If building on GPU and you want OpenCV and OpenBLAS (make sure you have installed the [CUDA dependencies first](#cuda-dependencies)): ```bash git clone --recursive https://github.com/apache/incubator-mxnet.git - cd mxnet - make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 + cd incubator-mxnet + echo "USE_OPENCV = 1" >> ./config.mk + echo "USE_BLAS = openblas" >> ./config.mk + echo "USE_CUDA = 1" >> ./config.mk + echo "USE_CUDA_PATH = /usr/local/cuda" >> ./config.mk + echo "USE_CUDNN = 1" >> ./config.mk + make -j $(nproc) ``` *Note* - USE_OPENCV and USE_BLAS are make file flags to set compilation options to use OpenCV and BLAS library. You can explore and use more compilation options in `make/config.mk` and also review common [usage examples](build_from_source.html#usage-examples). @@ -189,7 +190,7 @@ Building from source creates a library called ```libmxnet.so``` in the `lib` fol You may also want to add the MXNet shared library to your `LD_LIBRARY_PATH`: ```bash -export LD_LIBRARY_PATH=~/incubator-mxnet/lib +export LD_LIBRARY_PATH=$PWD/lib ``` After building the MXNet library, you may install language bindings. @@ -207,6 +208,7 @@ After you have installed the MXNet core library. You may install MXNet interface - [Perl](#install-the-mxnet-package-for-perl) - [R](#install-the-mxnet-package-for-r) - [Scala](#install-the-mxnet-package-for-scala) +- [Java](#install-the-mxnet-package-for-java)
      @@ -348,7 +350,9 @@ $ sudo apt-get install -y libopencv-dev ```bash $ git clone --recursive https://github.com/apache/incubator-mxnet $ cd incubator-mxnet -$ make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas +$ echo "USE_OPENCV = 1" >> ./config.mk +$ echo "USE_BLAS = openblas" >> ./config.mk +$ make -j $(nproc) ``` *Note* - USE_OPENCV and USE_BLAS are make file flags to set compilation options to use OpenCV and BLAS library. You can explore and use more compilation options in `make/config.mk`. @@ -397,6 +401,14 @@ Further information is in the [MXNet-Scala Setup Instructions](scala_setup.html) If you use IntelliJ or a similar IDE, you may want to follow the [MXNet-Scala on IntelliJ tutorial](../tutorials/scala/mxnet_scala_on_intellij.html) instead.
      +### Install the MXNet Package for Java + +To use the MXNet-Java package, you can acquire the Maven package as a dependency. + +Further information is in the [MXNet-Java Setup Instructions](java_setup.html). + +If you use IntelliJ or a similar IDE, you may want to follow the [MXNet-Java on IntelliJ tutorial](../tutorials/java/mxnet_java_on_intellij.html) instead. +
      ## Contributions diff --git a/docs/install/windows_setup.md b/docs/install/windows_setup.md index 8e612454a..b34936140 100755 --- a/docs/install/windows_setup.md +++ b/docs/install/windows_setup.md @@ -108,7 +108,7 @@ To get further enhancements for deep neural networks, you may want to enable MKL Check the chart below for other options or refer to [PyPI for other MXNet pip packages](https://pypi.org/project/mxnet/). -![pip packages](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/install/pip-packages-1.3.0.png) +![pip packages](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/install/pip-packages.png) ## Build from Source diff --git a/docs/model_zoo/index.md b/docs/model_zoo/index.md index f7d147902..034d360b9 100644 --- a/docs/model_zoo/index.md +++ b/docs/model_zoo/index.md @@ -41,7 +41,7 @@ For instructions on using these models, see [the python tutorial on using pre-tr | [SqueezeNet v1.1](http://data.dmlc.ml/models/imagenet/squeezenet/squeezenet_v1.1-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/squeezenet/squeezenet_v1.1-0000.params) | [Iandola et al.., 2016](https://arxiv.org/pdf/1602.07360v4.pdf) | @jspisak | | [VGG16](http://data.dmlc.ml/models/imagenet/vgg/vgg16-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/vgg/vgg16-0000.params)| [Simonyan et al.., 2015](https://arxiv.org/pdf/1409.1556v6.pdf) | @jspisak | | [VGG19](http://data.dmlc.ml/models/imagenet/vgg/vgg19-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/vgg/vgg19-0000.params) | [Simonyan et al.., 2015](https://arxiv.org/pdf/1409.1556v6.pdf) | @jspisak | -| [Inception v3 w/BatchNorm](http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params) | [Szegedy et al.., 2015](https://arxiv.org/pdf/1512.00567.pdf) | @jspisak | +| [Inception w/ BatchNorm](http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/inception-bn/Inception-BN-0126.params) | [Szegedy et al.., 2015](https://arxiv.org/pdf/1502.03167.pdf) | @jspisak | | [ResidualNet152](http://data.dmlc.ml/models/imagenet/resnet/152-layers/resnet-152-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/resnet/152-layers/resnet-152-0000.params) | [He et al.., 2015](https://arxiv.org/pdf/1512.03385v1.pdf) | @jspisak | | [ResNext101-64x4d](http://data.dmlc.ml/models/imagenet/resnext/101-layers/resnext-101-64x4d-symbol.json) | ImageNet | [Param File](http://data.dmlc.ml/models/imagenet/resnext/101-layers/resnext-101-64x4d-0000.params) | [Xie et al.., 2016](https://arxiv.org/pdf/1611.05431.pdf) | @Jerryzcn | | Fast-RCNN | PASCAL VOC | [Param File] | [Girshick, 2015](https://arxiv.org/pdf/1504.08083v2.pdf) | | diff --git a/docs/mxdoc.py b/docs/mxdoc.py index 7092b9ee9..52f050e0a 100644 --- a/docs/mxdoc.py +++ b/docs/mxdoc.py @@ -40,11 +40,12 @@ for section in [ _DOC_SET ]: print("Document sets to generate:") - for candidate in [ 'scala_docs', 'clojure_docs', 'doxygen_docs', 'r_docs' ]: + for candidate in [ 'scala_docs', 'java_docs', 'clojure_docs', 'doxygen_docs', 'r_docs' ]: print('%-12s : %s' % (candidate, parser.get(section, candidate))) _MXNET_DOCS_BUILD_MXNET = parser.getboolean('mxnet', 'build_mxnet') _SCALA_DOCS = parser.getboolean(_DOC_SET, 'scala_docs') +_JAVA_DOCS = parser.getboolean(_DOC_SET, 'java_docs') _CLOJURE_DOCS = parser.getboolean(_DOC_SET, 'clojure_docs') _DOXYGEN_DOCS = parser.getboolean(_DOC_SET, 'doxygen_docs') _R_DOCS = parser.getboolean(_DOC_SET, 'r_docs') @@ -58,7 +59,8 @@ # language names and the according file extensions and comment symbol _LANGS = {'python' : ('py', '#'), 'r' : ('R','#'), - 'scala' : ('scala', '#'), + 'scala' : ('scala', '//'), + 'java' : ('java', '//'), 'julia' : ('jl', '#'), 'perl' : ('pl', '#'), 'cpp' : ('cc', '//'), @@ -85,10 +87,10 @@ def generate_doxygen(app): def build_mxnet(app): """Build mxnet .so lib""" if not os.path.exists(os.path.join(app.builder.srcdir, '..', 'config.mk')): - _run_cmd("cd %s/.. && cp make/config.mk config.mk && make -j$(nproc) DEBUG=1" % + _run_cmd("cd %s/.. && cp make/config.mk config.mk && make -j$(nproc) DEBUG=1 USE_MKLDNN=0" % app.builder.srcdir) else: - _run_cmd("cd %s/.. && make -j$(nproc) DEBUG=1" % + _run_cmd("cd %s/.. && make -j$(nproc) DEBUG=1 USE_MKLDNN=0" % app.builder.srcdir) def build_r_docs(app): @@ -101,21 +103,50 @@ def build_r_docs(app): _run_cmd('mkdir -p ' + dest_path + '; mv ' + pdf_path + ' ' + dest_path) def build_scala(app): - """build scala for scala docs and clojure docs to use""" + """build scala for scala docs, java docs, and clojure docs to use""" _run_cmd("cd %s/.. && make scalapkg" % app.builder.srcdir) _run_cmd("cd %s/.. && make scalainstall" % app.builder.srcdir) def build_scala_docs(app): """build scala doc and then move the outdir""" scala_path = app.builder.srcdir + '/../scala-package' - # scaldoc fails on some apis, so exit 0 to pass the check - _run_cmd('cd ' + scala_path + '; scaladoc `find . -type f -name "*.scala" | egrep \"\/core|\/infer\" | egrep -v \"Suite\"`; exit 0') + scala_doc_sources = 'find . -type f -name "*.scala" | egrep \"\.\/core|\.\/infer\" | egrep -v \"\/javaapi\" | egrep -v \"Suite\"' + scala_doc_classpath = ':'.join([ + '`find native -name "*.jar" | grep "target/lib/" | tr "\\n" ":" `', + '`find macros -name "*.jar" | tr "\\n" ":" `', + '`find core -name "*.jar" | tr "\\n" ":" `', + '`find infer -name "*.jar" | tr "\\n" ":" `' + ]) + # There are unresolvable errors on mxnet 1.2.x. We are ignoring those errors while aborting the ci on newer versions + scala_ignore_errors = '; exit 0' if '1.2.' or '1.3.' in _BUILD_VER else '' + _run_cmd('cd {}; scaladoc `{}` -classpath {} -feature -deprecation {}' + .format(scala_path, scala_doc_sources, scala_doc_classpath, scala_ignore_errors)) dest_path = app.builder.outdir + '/api/scala/docs' _run_cmd('rm -rf ' + dest_path) _run_cmd('mkdir -p ' + dest_path) + # 'index' and 'package.html' do not exist in later versions of scala; delete these after upgrading scala>2.12.x scaladocs = ['index', 'index.html', 'org', 'lib', 'index.js', 'package.html'] for doc_file in scaladocs: - _run_cmd('cd ' + scala_path + ' && mv -f ' + doc_file + ' ' + dest_path) + _run_cmd('cd ' + scala_path + ' && mv -f ' + doc_file + ' ' + dest_path + '; exit 0') + +def build_java_docs(app): + """build java docs and then move the outdir""" + java_path = app.builder.srcdir + '/../scala-package' + java_doc_sources = 'find . -type f -name "*.scala" | egrep \"\.\/core|\.\/infer\" | egrep \"\/javaapi\" | egrep -v \"Suite\"' + java_doc_classpath = ':'.join([ + '`find native -name "*.jar" | grep "target/lib/" | tr "\\n" ":" `', + '`find macros -name "*.jar" | tr "\\n" ":" `', + '`find core -name "*.jar" | tr "\\n" ":" `', + '`find infer -name "*.jar" | tr "\\n" ":" `' + ]) + _run_cmd('cd {}; scaladoc `{}` -classpath {} -feature -deprecation' + .format(java_path, java_doc_sources, java_doc_classpath)) + dest_path = app.builder.outdir + '/api/java/docs' + _run_cmd('rm -rf ' + dest_path) + _run_cmd('mkdir -p ' + dest_path) + javadocs = ['index', 'index.html', 'org', 'lib', 'index.js', 'package.html'] + for doc_file in javadocs: + _run_cmd('cd ' + java_path + ' && mv -f ' + doc_file + ' ' + dest_path + '; exit 0') def build_clojure_docs(app): """build clojure doc and then move the outdir""" @@ -125,7 +156,7 @@ def build_clojure_docs(app): _run_cmd('rm -rf ' + dest_path) _run_cmd('mkdir -p ' + dest_path) clojure_doc_path = app.builder.srcdir + '/../contrib/clojure-package/target/doc' - _run_cmd('cd ' + clojure_doc_path + ' && cp -r * ' + dest_path) + _run_cmd('cd ' + clojure_doc_path + ' && cp -r * ' + dest_path + '; exit 0') def _convert_md_table_to_rst(table): """Convert a markdown table to rst format""" @@ -404,7 +435,6 @@ def add_buttons(app, docname, source): # source[i] = '\n'.join(lines) def setup(app): - # If MXNET_DOCS_BUILD_MXNET is set something different than 1 # Skip the build step if os.getenv('MXNET_DOCS_BUILD_MXNET', '1') == '1' or _MXNET_DOCS_BUILD_MXNET: @@ -419,6 +449,9 @@ def setup(app): if _SCALA_DOCS: print("Building Scala Docs!") app.connect("builder-inited", build_scala_docs) + if _JAVA_DOCS: + print("Building Java Docs!") + app.connect("builder-inited", build_java_docs) if _CLOJURE_DOCS: print("Building Clojure Docs!") app.connect("builder-inited", build_clojure_docs) diff --git a/docs/settings.ini b/docs/settings.ini index b8e486e58..7de3268ab 100644 --- a/docs/settings.ini +++ b/docs/settings.ini @@ -1,68 +1,107 @@ [mxnet] build_mxnet = 0 +[document_sets_tutorial] +clojure_docs = 0 +doxygen_docs = 1 +java_docs = 0 +r_docs = 0 +scala_docs = 0 + [document_sets_default] clojure_docs = 1 doxygen_docs = 1 +java_docs = 1 +r_docs = 0 +scala_docs = 1 + +[document_sets_1.3.1] +clojure_docs = 1 +doxygen_docs = 1 +java_docs = 0 +r_docs = 0 +scala_docs = 1 + +[document_sets_1.3.0] +clojure_docs = 1 +doxygen_docs = 1 +java_docs = 0 +r_docs = 0 +scala_docs = 1 + +[document_sets_v1.3.x] +clojure_docs = 1 +doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 1 [document_sets_1.2.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 1 [document_sets_v1.2.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 1 [document_sets_1.1.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_v1.1.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_1.0.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_v1.0.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_0.12.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_v0.12.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_0.11.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 [document_sets_v0.11.0] clojure_docs = 0 doxygen_docs = 1 +java_docs = 0 r_docs = 0 scala_docs = 0 diff --git a/example/cnn_visualization/gradcam.py b/docs/tutorial_utils/vision/cnn_visualization/gradcam.py similarity index 98% rename from example/cnn_visualization/gradcam.py rename to docs/tutorial_utils/vision/cnn_visualization/gradcam.py index a8708f787..54cb65eef 100644 --- a/example/cnn_visualization/gradcam.py +++ b/docs/tutorial_utils/vision/cnn_visualization/gradcam.py @@ -249,8 +249,8 @@ def visualize(net, preprocessed_img, orig_img, conv_layer_name): imggrad = get_image_grad(net, preprocessed_img) conv_out, conv_out_grad = get_conv_out_grad(net, preprocessed_img, conv_layer_name=conv_layer_name) - cam = get_cam(imggrad, conv_out) - + cam = get_cam(conv_out_grad, conv_out) + cam = cv2.resize(cam, (imggrad.shape[1], imggrad.shape[2])) ggcam = get_guided_grad_cam(cam, imggrad) img_ggcam = grad_to_image(ggcam) diff --git a/docs/tutorials/basic/module.md b/docs/tutorials/basic/module.md index 191e3baaa..f7a4d6e25 100644 --- a/docs/tutorials/basic/module.md +++ b/docs/tutorials/basic/module.md @@ -39,11 +39,16 @@ training examples each time. A separate iterator is also created for test data. ```python import logging +import random logging.getLogger().setLevel(logging.INFO) + import mxnet as mx import numpy as np mx.random.seed(1234) +np.random.seed(1234) +random.seed(1234) + fname = mx.test_utils.download('https://s3.us-east-2.amazonaws.com/mxnet-public/letter_recognition/letter-recognition.data') data = np.genfromtxt(fname, delimiter=',')[:,1:] label = np.array([ord(l.split(',')[0])-ord('A') for l in open(fname, 'r')]) @@ -64,7 +69,7 @@ net = mx.sym.FullyConnected(net, name='fc1', num_hidden=64) net = mx.sym.Activation(net, name='relu1', act_type="relu") net = mx.sym.FullyConnected(net, name='fc2', num_hidden=26) net = mx.sym.SoftmaxOutput(net, name='softmax') -mx.viz.plot_network(net) +mx.viz.plot_network(net, node_attrs={"shape":"oval","fixedsize":"false"}) ``` @@ -135,11 +140,17 @@ for epoch in range(5): print('Epoch %d, Training %s' % (epoch, metric.get())) ``` - Epoch 0, Training ('accuracy', 0.4554375) - Epoch 1, Training ('accuracy', 0.6485625) - Epoch 2, Training ('accuracy', 0.7055625) - Epoch 3, Training ('accuracy', 0.7396875) - Epoch 4, Training ('accuracy', 0.764375) + +Expected output: + + +``` +Epoch 0, Training ('accuracy', 0.434625) +Epoch 1, Training ('accuracy', 0.6516875) +Epoch 2, Training ('accuracy', 0.6968125) +Epoch 3, Training ('accuracy', 0.7273125) +Epoch 4, Training ('accuracy', 0.7575625) +``` To learn more about these APIs, visit [Module API](http://mxnet.io/api/python/module/module.html). @@ -172,34 +183,36 @@ mod.fit(train_iter, optimizer='sgd', optimizer_params={'learning_rate':0.1}, eval_metric='acc', - num_epoch=8) + num_epoch=7) ``` - INFO:root:Epoch[0] Train-accuracy=0.364625 - INFO:root:Epoch[0] Time cost=0.388 - INFO:root:Epoch[0] Validation-accuracy=0.557250 - INFO:root:Epoch[1] Train-accuracy=0.633625 - INFO:root:Epoch[1] Time cost=0.470 - INFO:root:Epoch[1] Validation-accuracy=0.634750 - INFO:root:Epoch[2] Train-accuracy=0.697187 - INFO:root:Epoch[2] Time cost=0.402 - INFO:root:Epoch[2] Validation-accuracy=0.665500 - INFO:root:Epoch[3] Train-accuracy=0.735062 - INFO:root:Epoch[3] Time cost=0.402 - INFO:root:Epoch[3] Validation-accuracy=0.713000 - INFO:root:Epoch[4] Train-accuracy=0.762563 - INFO:root:Epoch[4] Time cost=0.408 - INFO:root:Epoch[4] Validation-accuracy=0.742000 - INFO:root:Epoch[5] Train-accuracy=0.782312 - INFO:root:Epoch[5] Time cost=0.400 - INFO:root:Epoch[5] Validation-accuracy=0.778500 - INFO:root:Epoch[6] Train-accuracy=0.797188 - INFO:root:Epoch[6] Time cost=0.392 - INFO:root:Epoch[6] Validation-accuracy=0.798250 - INFO:root:Epoch[7] Train-accuracy=0.807750 - INFO:root:Epoch[7] Time cost=0.401 - INFO:root:Epoch[7] Validation-accuracy=0.789250 +Expected output: + + +``` +INFO:root:Epoch[0] Train-accuracy=0.325437 +INFO:root:Epoch[0] Time cost=0.550 +INFO:root:Epoch[0] Validation-accuracy=0.568500 +INFO:root:Epoch[1] Train-accuracy=0.622188 +INFO:root:Epoch[1] Time cost=0.552 +INFO:root:Epoch[1] Validation-accuracy=0.656500 +INFO:root:Epoch[2] Train-accuracy=0.694375 +INFO:root:Epoch[2] Time cost=0.566 +INFO:root:Epoch[2] Validation-accuracy=0.703500 +INFO:root:Epoch[3] Train-accuracy=0.732187 +INFO:root:Epoch[3] Time cost=0.562 +INFO:root:Epoch[3] Validation-accuracy=0.748750 +INFO:root:Epoch[4] Train-accuracy=0.755375 +INFO:root:Epoch[4] Time cost=0.484 +INFO:root:Epoch[4] Validation-accuracy=0.761500 +INFO:root:Epoch[5] Train-accuracy=0.773188 +INFO:root:Epoch[5] Time cost=0.383 +INFO:root:Epoch[5] Validation-accuracy=0.715000 +INFO:root:Epoch[6] Train-accuracy=0.794687 +INFO:root:Epoch[6] Time cost=0.378 +INFO:root:Epoch[6] Validation-accuracy=0.802250 +``` By default, `fit` function has `eval_metric` set to `accuracy`, `optimizer` to `sgd` and optimizer_params to `(('learning_rate', 0.01),)`. @@ -225,12 +238,17 @@ It can be used as follows: ```python score = mod.score(val_iter, ['acc']) print("Accuracy score is %f" % (score[0][1])) -assert score[0][1] > 0.77, "Achieved accuracy (%f) is less than expected (0.77)" % score[0][1] +assert score[0][1] > 0.76, "Achieved accuracy (%f) is less than expected (0.76)" % score[0][1] ``` - Accuracy score is 0.789250 + +Expected output: +``` +Accuracy score is 0.802250 +``` + Some of the other metrics which can be used are `top_k_acc`(top-k-accuracy), `F1`, `RMSE`, `MSE`, `MAE`, `ce`(CrossEntropy). To learn more about the metrics, visit [Evaluation metric](http://mxnet.io/api/python/metric/metric.html). @@ -252,22 +270,27 @@ mod = mx.mod.Module(symbol=net) mod.fit(train_iter, num_epoch=5, epoch_end_callback=checkpoint) ``` - INFO:root:Epoch[0] Train-accuracy=0.101062 - INFO:root:Epoch[0] Time cost=0.422 - INFO:root:Saved checkpoint to "mx_mlp-0001.params" - INFO:root:Epoch[1] Train-accuracy=0.263313 - INFO:root:Epoch[1] Time cost=0.785 - INFO:root:Saved checkpoint to "mx_mlp-0002.params" - INFO:root:Epoch[2] Train-accuracy=0.452188 - INFO:root:Epoch[2] Time cost=0.624 - INFO:root:Saved checkpoint to "mx_mlp-0003.params" - INFO:root:Epoch[3] Train-accuracy=0.544125 - INFO:root:Epoch[3] Time cost=0.427 - INFO:root:Saved checkpoint to "mx_mlp-0004.params" - INFO:root:Epoch[4] Train-accuracy=0.605250 - INFO:root:Epoch[4] Time cost=0.399 - INFO:root:Saved checkpoint to "mx_mlp-0005.params" +Expected output: + + +``` +INFO:root:Epoch[0] Train-accuracy=0.098437 +INFO:root:Epoch[0] Time cost=0.421 +INFO:root:Saved checkpoint to "mx_mlp-0001.params" +INFO:root:Epoch[1] Train-accuracy=0.257437 +INFO:root:Epoch[1] Time cost=0.520 +INFO:root:Saved checkpoint to "mx_mlp-0002.params" +INFO:root:Epoch[2] Train-accuracy=0.457250 +INFO:root:Epoch[2] Time cost=0.562 +INFO:root:Saved checkpoint to "mx_mlp-0003.params" +INFO:root:Epoch[3] Train-accuracy=0.558187 +INFO:root:Epoch[3] Time cost=0.434 +INFO:root:Saved checkpoint to "mx_mlp-0004.params" +INFO:root:Epoch[4] Train-accuracy=0.617750 +INFO:root:Epoch[4] Time cost=0.414 +INFO:root:Saved checkpoint to "mx_mlp-0005.params" +``` To load the saved module parameters, call the `load_checkpoint` function. It loads the Symbol and the associated parameters. We can then set the loaded @@ -299,16 +322,25 @@ mod.fit(train_iter, assert score[0][1] > 0.77, "Achieved accuracy (%f) is less than expected (0.77)" % score[0][1] ``` - INFO:root:Epoch[3] Train-accuracy=0.544125 - INFO:root:Epoch[3] Time cost=0.398 - INFO:root:Epoch[4] Train-accuracy=0.605250 - INFO:root:Epoch[4] Time cost=0.545 - INFO:root:Epoch[5] Train-accuracy=0.644312 - INFO:root:Epoch[5] Time cost=0.592 - INFO:root:Epoch[6] Train-accuracy=0.675000 - INFO:root:Epoch[6] Time cost=0.491 - INFO:root:Epoch[7] Train-accuracy=0.695812 - INFO:root:Epoch[7] Time cost=0.363 + +Expected output: + + +``` +INFO:root:Epoch[3] Train-accuracy=0.555438 +INFO:root:Epoch[3] Time cost=0.377 +INFO:root:Epoch[4] Train-accuracy=0.616625 +INFO:root:Epoch[4] Time cost=0.457 +INFO:root:Epoch[5] Train-accuracy=0.658438 +INFO:root:Epoch[5] Time cost=0.518 +........................................... +INFO:root:Epoch[18] Train-accuracy=0.788687 +INFO:root:Epoch[18] Time cost=0.532 +INFO:root:Epoch[19] Train-accuracy=0.789562 +INFO:root:Epoch[19] Time cost=0.531 +INFO:root:Epoch[20] Train-accuracy=0.796250 +INFO:root:Epoch[20] Time cost=0.531 +``` diff --git a/docs/tutorials/basic/reshape_transpose.md b/docs/tutorials/basic/reshape_transpose.md new file mode 100644 index 000000000..999b22ca2 --- /dev/null +++ b/docs/tutorials/basic/reshape_transpose.md @@ -0,0 +1,197 @@ +## Difference between reshape and transpose operators + +What does it mean if MXNet gives you an error like the this? +``` +Check failed: shape_.Size() == shape.Size() (127872 vs. 25088) NDArray.Reshape: target shape must have the same size as current shape when recording with autograd. +``` +This error message tells you that the data being passed to your model or between layers in the model is not in the correct format. Modifying the shape of tensors is a very common operation in Deep Learning. +For instance, when using pretrained neural networks it is often necessary to adjust the input data dimensions to correspond to what the network has been trained on, e.g. tensors of shape `[batch_size, channels, width, height]`. This notebook discusses briefly the difference between the operators [Reshape](http://mxnet.incubator.apache.org/test/api/python/ndarray.html#mxnet.ndarray.NDArray.reshape) and [Transpose](http://mxnet.incubator.apache.org/test/api/python/ndarray.html#mxnet.ndarray.transpose). Both allow you to change the shape, however they are not the same and are commonly mistaken. + +```python +import matplotlib.pyplot as plt +import matplotlib.image as mpimg +import mxnet as mx +from mxnet import gluon +import numpy as np +``` + + +```python +img_array = mpimg.imread('https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/basic/transpose_reshape/cat.png') +plt.imshow(img_array) +plt.axis("off") +print (img_array.shape) +``` + +(157, 210, 3) + +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/basic/transpose_reshape/cat.png) + + +The color image has the following properties: +* width: 210 pixels +* height: 157 pixels +* colors: 3 (RGB) + +Now let's reshape the image in order to exchange width and height dimensions. + + +```python +reshaped = img_array.reshape((210,157,3)) +print (reshaped.shape) +plt.imshow(reshaped) +plt.axis("off") +``` +(210,157,3) + +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/basic/transpose_reshape/reshaped_image.png) + + +As we can see the first and second dimensions have changed. However the image can't be identified as cat any longer. In order to understand what happened, let's have a look at the image below. + + + +While the number of rows and columns changed, the layout of the underlying data did not. The pixel values that have been in one row are still in one row. This means for instance that pixel 10 in the upper right corner ends up in the middle of the image instead of the lower left corner. Consequently contextual information gets lost, because the relative position of pixel values is not the same anymore. As one can imagine a neural network would not be able to classify such an image as cat. + +`Transpose` instead changes the layout of the underlying data. + + +```python +transposed = img_array.transpose((1,0,2)) +plt.imshow(transposed) +plt.axis("off") +``` + +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/basic/transpose_reshape/transposed_image.png) + + +As we can see width and height changed, by rotating pixel values by 90 degrees. Transpose does the following: + + + +As shown in the diagram, the axes have been flipped: pixel values that were in the first row are now in the first column. +## When to transpose/reshape with MXNet +In this chapter we discuss when transpose and reshape is used in MXNet. +#### Channel first for images +Images are usually stored in the format height, wight, channel. When working with [convolutional](https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Conv1D) layers, MXNet expects the layout to be `NCHW` (batch, channel, height, width). This is in contrast to Tensorflow, where image tensors are in the form `NHWC`. MXNet uses `NCHW` layout because of performance reasons on the GPU. When preprocessing the input images, you may have a function like the following: +```python +def transform(data, label): + return data.astype(np.float32).transpose((2,0,1))/255.0, np.float32(label) +``` +Images may also be stored as 1 dimensional vector for example in byte packed datasets. For instance, instead of `[28,28,1]` you may have `[784,1]`. In this situation you need to perform a reshape e.g. `ndarray.reshape((1,28,28))` + + +#### TNC layout for RNN +When working with [LSTM](https://mxnet.incubator.apache.org/api/python/gluon/rnn.html#mxnet.gluon.rnn.LSTM) or [GRU](https://mxnet.incubator.apache.org/api/python/gluon/rnn.html#mxnet.gluon.rnn.GRU) layers, the default layout for input and ouput tensors has to be `TNC` (sequence length, batch size, and feature dimensions). For instance in the following network the input goes into a 1 dimensional convolution layer and whose output goes into a GRU cell. Here the tensors would mismatch, because `Conv1D` takes data as `NCT`, but GRU expects it to be `NTC`. To ensure that the forward pass does not crash, we need to do a tensor transpose. We can do this by defining a ```HybridLambda```. +```python +network = gluon.nn.HybridSequential() +with network.name_scope(): + network.add(gluon.nn.Conv1D(196, kernel_size=2, strides=1)) + network.add(gluon.nn.HybridLambda(lambda F, x: F.transpose(x, (0, 2, 1)))) + network.add(gluon.rnn.GRU(128)) + +network.hybridize() +network.initialize(mx.init.Xavier(), ctx=mx.cpu()) +a = mx.random.uniform(shape=(1,100,1000)) +network(a) +output = network(a) +print (output.shape) +``` +(1, 999, 128) +#### Advanced reshaping with MXNet ndarrays +It is sometimes useful to automatically infer the shape of tensors. Especially when you deal with very deep neural networks, it may not always be clear what the shape of a tensor is after a specific layer. For instance you may want the tensor to be two-dimensional where one dimension is the known batch_size. With ```mx.nd.array(-1, batch_size)``` the first dimension will be automatically inferred. Here is a simplified example: +```python +batch_size = 100 +input_data = mx.random.uniform(shape=(batch_size, 20,100)) +reshaped = input_data.reshape(batch_size, -1) +print (input_data.shape, reshaped.shape) +``` +(100L, 20L, 100L), (100L, 2000L) + +The reshape function of [MXNet's NDArray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html?highlight=reshape#mxnet.ndarray.NDArray.reshape) allows even more advanced transformations: For instance:0 copies the dimension from the input to the output shape, -2 copies all/remainder of the input dimensions to the output shape. With -3 reshape uses the product of two consecutive dimensions of the input shape as the output dim. With -4 reshape splits one dimension of the input into two dimensions passed subsequent to -4. Here an example: +```python +x = mx.nd.random.uniform(shape=(1, 3, 4, 64, 64)) + +``` +Assume ```x``` with the shape ```[batch_size, channel, upscale, width, height]``` is the output of a model for image superresolution. Now we want to apply the upscale on width and height, to increase the 64x64 to an 128x128 image. +To do so, we can use advanced reshaping, where we have to split the third dimension (upscale) and multiply it with width and height. We can do +```python +x = x.reshape(1, 3, -4, 2, 2, 0, 0) +print (x.shape) +``` + +(1L, 3L, 2L, 2L, 64L, 64L) + +This splits up the third dimension into ```[2,2]```, so (1L, 3L, **4L** , 64L, 64L) becomes (1L, 3L, **2L** , **2L** , 64L, 64L) The other dimensions remain unchanged. In order to multiply the new dimensions with width and height, we can do a transpose and then use reshape with -3. +```python +x = x.transpose((0, 1, 4, 2, 5, 3)) +print (x.shape) +x = x.reshape(0, 0, -3, -3) +print (x.shape) +``` + +(1L, 3L, 64L, 2L, 64L, 2L) + +(1L, 3L, 128L, 128L) + +Reshape -3 will calculate the dot product between the current and subsequent column. So (1L, 3L, **64L** , **2L** , ***64L, 2L*** ) becomes (1L, 3L, **128L** , ***128L*** ) + +#### Most Common Pitfalls +In this section we want to show some of the most common pitfalls that happen when your input data is not correctly shaped. + +##### Forward Pass + +You execute the forward pass and get an error message followed by a very long stacktrace, for instance: + +``` +*** Error in `python': free(): invalid pointer: 0x00007fde5405a918 *** +======= Backtrace: ========= +/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fdf475927e5] +/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7fdf4759b37a] +/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7fdf4759f53c] +/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXExecutorReshape+0x1852)[0x7fdecef2c6e2] +/home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c)[0x7fdf46332ec0] +/home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d)[0x7fdf4633287d] +/home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce)[0x7fdf46547e2e] +/home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x12865)[0x7fdf46548865] +python(_PyObject_FastCallDict+0x8b)[0x56457eba2d7b] +python(+0x19e7ce)[0x56457ec327ce] +python(_PyEval_EvalFrameDefault+0x2fa)[0x56457ec54cba] +python(+0x197dae)[0x56457ec2bdae] +[...] + +``` +This happens when you your data does not have the shape ```[batch_size, channel, width, height]``` e.g. your data may be a one-dimensional vector or when the color channel may be the last dimension instead of the second one. + +##### Backward Pass +In other cases the forward pass may not fail, but the shape of the network output is not as expected. For instance in our previous RNN example, nothing is preventing us to skip the transpose. + +```python +network = gluon.nn.HybridSequential() +with network.name_scope(): + network.add(gluon.nn.Conv1D(196, kernel_size=2, strides=1)) + #network.add(gluon.nn.HybridLambda(lambda F, x: F.transpose(x, (0, 2, 1)))) + network.add(gluon.rnn.GRU(128)) + +network.hybridize() +network.initialize(mx.init.Xavier(), ctx=mx.cpu()) +a = mx.random.uniform(shape=(1,100,1000)) +output = network(a) +print (output.shape) +``` +(1, 196, 128) + +Instead of ```(1, 999, 128)``` the shape is now ```(1, 196, 128)```. But during the training loop, calculating the loss would crash because of shape mismatch between labels and output. You may get an error like the following: +``` +mxnet.base.MXNetError: [10:56:29] src/ndarray/ndarray.cc:229: Check failed: shape_.Size() == shape.Size() (127872 vs. 25088) NDArray.Reshape: target shape must have the same size as current shape when recording with autograd. + +Stack trace returned 6 entries: +[bt] (0) 0 libmxnet.so 0x00000001126c0b90 libmxnet.so + 15248 +[bt] (1) 1 libmxnet.so 0x00000001126c093f libmxnet.so + 14655 +[bt] (2) 2 libmxnet.so 0x0000000113cd236d MXNDListFree + 1407789 +[bt] (3) 3 libmxnet.so 0x0000000113b345ca MXNDArrayReshape64 + 970 +[bt] (4) 4 libffi.6.dylib 0x000000010b399884 ffi_call_unix64 + 76 +[bt] (5) 5 ??? 0x00007fff54cadf50 0x0 + 140734615969616 + +``` + diff --git a/docs/tutorials/basic/symbol.md b/docs/tutorials/basic/symbol.md index 7ebcadfc1..5e1e3cd8c 100644 --- a/docs/tutorials/basic/symbol.md +++ b/docs/tutorials/basic/symbol.md @@ -89,7 +89,7 @@ f = mx.sym.reshape(d+e, shape=(1,4)) # broadcast g = mx.sym.broadcast_to(f, shape=(2,4)) # plot -mx.viz.plot_network(symbol=g) +mx.viz.plot_network(symbol=g, node_attrs={"shape":"oval","fixedsize":"false"}) ``` The computations declared in the above examples can be bound to the input data @@ -108,7 +108,7 @@ net = mx.sym.FullyConnected(data=net, name='fc1', num_hidden=128) net = mx.sym.Activation(data=net, name='relu1', act_type="relu") net = mx.sym.FullyConnected(data=net, name='fc2', num_hidden=10) net = mx.sym.SoftmaxOutput(data=net, name='out') -mx.viz.plot_network(net, shape={'data':(100,200)}) +mx.viz.plot_network(net, shape={'data':(100,200)}, node_attrs={"shape":"oval","fixedsize":"false"}) ``` Each symbol takes a (unique) string name. NDArray and Symbol both represent @@ -211,7 +211,7 @@ def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0),name=None, su prev = mx.sym.Variable(name="Previous Output") conv_comp = ConvFactory(data=prev, num_filter=64, kernel=(7,7), stride=(2, 2)) shape = {"Previous Output" : (128, 3, 28, 28)} -mx.viz.plot_network(symbol=conv_comp, shape=shape) +mx.viz.plot_network(symbol=conv_comp, shape=shape, node_attrs={"shape":"oval","fixedsize":"false"}) ``` Then we can define a function that constructs an inception module based on @@ -237,7 +237,7 @@ def InceptionFactoryA(data, num_1x1, num_3x3red, num_3x3, num_d3x3red, num_d3x3, return concat prev = mx.sym.Variable(name="Previous Output") in3a = InceptionFactoryA(prev, 64, 64, 64, 64, 96, "avg", 32, name="in3a") -mx.viz.plot_network(symbol=in3a, shape=shape) +mx.viz.plot_network(symbol=in3a, shape=shape, node_attrs={"shape":"oval","fixedsize":"false"}) ``` Finally, we can obtain the whole network by chaining multiple inception diff --git a/docs/tutorials/c++/basics.md b/docs/tutorials/c++/basics.md index d3231e7a1..aa73a7363 100644 --- a/docs/tutorials/c++/basics.md +++ b/docs/tutorials/c++/basics.md @@ -8,9 +8,9 @@ The following contents assume that the working directory is `/path/to/mxnet/cpp- Load Data -------- -Before going into codes, we need to fetch MNIST data. You can either use the script `get_mnist.sh`, +Before going into codes, we need to fetch MNIST data. You can either use the script `/path/to/mxnet/cpp-package/example/get_data.sh`, or download mnist data by yourself from Lecun's [website](http://yann.lecun.com/exdb/mnist/) -and decompress them into `mnist_data` folder. +and decompress them into `data/mnist_data` folder. Except linking the MXNet shared library, the C++ package itself is a header-only package, which means all you need to do is to include the header files. Among the header files, @@ -36,14 +36,14 @@ The digits in MNIST are 2-dimension arrays, so we should set `flat` to true to f ```cpp auto train_iter = MXDataIter("MNISTIter") - .SetParam("image", "./mnist_data/train-images-idx3-ubyte") - .SetParam("label", "./mnist_data/train-labels-idx1-ubyte") + .SetParam("image", "./data/mnist_data/train-images-idx3-ubyte") + .SetParam("label", "./data/mnist_data/train-labels-idx1-ubyte") .SetParam("batch_size", batch_size) .SetParam("flat", 1) .CreateDataIter(); auto val_iter = MXDataIter("MNISTIter") - .SetParam("image", "./mnist_data/t10k-images-idx3-ubyte") - .SetParam("label", "./mnist_data/t10k-labels-idx1-ubyte") + .SetParam("image", "./data/mnist_data/t10k-images-idx3-ubyte") + .SetParam("label", "./data/mnist_data/t10k-labels-idx1-ubyte") .SetParam("batch_size", batch_size) .SetParam("flat", 1) .CreateDataIter(); diff --git a/docs/tutorials/control_flow/ControlFlowTutorial.md b/docs/tutorials/control_flow/ControlFlowTutorial.md index 9e4c66f85..4b6a23136 100644 --- a/docs/tutorials/control_flow/ControlFlowTutorial.md +++ b/docs/tutorials/control_flow/ControlFlowTutorial.md @@ -15,13 +15,13 @@ from mxnet.gluon import HybridBlock ## foreach `foreach` is a for loop that iterates over the first dimension of the input data (it can be an array or a list of arrays). It is defined with the following signature: -```python +``` foreach(body, data, init_states, name) => (outputs, states) ``` It runs the Python function defined in `body` for every slice from the input arrays. The signature of the `body` function is defined as follows: -```python +``` body(data, states) => (outputs, states) ``` @@ -243,13 +243,13 @@ res, states = lstm(rnn_data, [x for x in init_states], valid_length) ## while_loop `while_loop` defines a while loop. It has the following signature: -```python +``` while_loop(cond, body, loop_vars, max_iterations, name) => (outputs, states) ``` Instead of running over the first dimension of an array, `while_loop` checks a condition function in every iteration and runs a `body` function for computation. The signature of the `body` function is defined as follows: -```python +``` body(state1, state2, ...) => (outputs, states) ``` @@ -297,13 +297,13 @@ print(state) ## cond `cond` defines an if condition. It has the following signature: -```python +``` cond(pred, then_func, else_func, name) ``` `cond` checks `pred`, which is a symbol or an NDArray with one element. If its value is true, it calls `then_func`. Otherwise, it calls `else_func`. The signature of `then_func` and `else_func` are as follows: -```python +``` func() => [outputs] ``` diff --git a/docs/tutorials/gluon/hybrid.md b/docs/tutorials/gluon/hybrid.md index f9f2c112f..f11622bd6 100644 --- a/docs/tutorials/gluon/hybrid.md +++ b/docs/tutorials/gluon/hybrid.md @@ -1,6 +1,9 @@ # Hybrid - Faster training and easy deployment -*Note: a newer version is available [here](http://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html).* +*Related Content:* +* [Fast, portable neural networks with Gluon HybridBlocks](https://gluon.mxnet.io/chapter07_distributed-learning/hybridize.html) +* [A Hybrid of Imperative and Symbolic Programming +](http://en.diveintodeeplearning.org/chapter_computational-performance/hybridize.html) Deep learning frameworks can be roughly divided into two categories: declarative and imperative. With declarative frameworks (including Tensorflow, Theano, etc) @@ -125,6 +128,7 @@ with other language front-ends like C, C++ and Scala. To this end, we simply use `export` and `SymbolBlock.imports`: ```python +net(x) net.export('model', epoch=1) ``` @@ -136,4 +140,106 @@ to gluon with `SymbolBlock`: net2 = gluon.SymbolBlock.imports('model-symbol.json', ['data'], 'model-0001.params') ``` - +## Operators that do not work with hybridize + +If you want to hybridize your model, you must use `F.some_operator` in your 'hybrid_forward' function. +`F` will be `mxnet.nd` before you hybridize and `mxnet.sym` after hybridize. While most APIs are the same in NDArray and Symbol, there are some differences. Writing `F.some_operator` and call `hybridize` may not work all of the time. +Here we list some frequently used NDArray APIs that can't be hybridized and provide you the work arounds. + +### Element-wise Operators + +In NDArray APIs, the following arithmetic and comparison APIs are automatically broadcasted if the input NDArrays have different shapes. +However, that's not the case in Symbol API. It's not automatically broadcasted, and you have to manually specify to use another set of broadcast operators for Symbols expected to have different shapes. + + +| NDArray APIs | Description | +|---|---| +| [*NDArray.\__add\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__add__) | x.\__add\__(y) <=> x+y <=> mx.nd.add(x, y) | +| [*NDArray.\__sub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__sub__) | x.\__sub\__(y) <=> x-y <=> mx.nd.subtract(x, y) | +| [*NDArray.\__mul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mul__) | x.\__mul\__(y) <=> x*y <=> mx.nd.multiply(x, y) | +| [*NDArray.\__div\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__div__) | x.\__div\__(y) <=> x/y <=> mx.nd.divide(x, y) | +| [*NDArray.\__mod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__mod__) | x.\__mod\__(y) <=> x%y <=> mx.nd.modulo(x, y) | +| [*NDArray.\__lt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__lt__) | x.\__lt\__(y) <=> x x mx.nd.lesser(x, y) | +| [*NDArray.\__le\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__le__) | x.\__le\__(y) <=> x<=y <=> mx.nd.less_equal(x, y) | +| [*NDArray.\__gt\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__gt__) | x.\__gt\__(y) <=> x>y <=> mx.nd.greater(x, y) | +| [*NDArray.\__ge\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ge__) | x.\__ge\__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)| +| [*NDArray.\__eq\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__eq__) | x.\__eq\__(y) <=> x==y <=> mx.nd.equal(x, y) | +| [*NDArray.\__ne\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__ne__) | x.\__ne\__(y) <=> x!=y <=> mx.nd.not_equal(x, y) | + +The current workaround is to use corresponding broadcast operators for arithmetic and comparison to avoid potential hybridization failure when input shapes are different. + +| Symbol APIs | Description | +|---|---| +|[*broadcast_add*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_add) | Returns element-wise sum of the input arrays with broadcasting. | +|[*broadcast_sub*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_sub) | Returns element-wise difference of the input arrays with broadcasting. | +|[*broadcast_mul*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_mul) | Returns element-wise product of the input arrays with broadcasting. | +|[*broadcast_div*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_div) | Returns element-wise division of the input arrays with broadcasting. | +|[*broadcast_mod*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_mod) | Returns element-wise modulo of the input arrays with broadcasting. | +|[*broadcast_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_equal) | Returns the result of element-wise *equal to* (==) comparison operation with broadcasting. | +|[*broadcast_not_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_not_equal) | Returns the result of element-wise *not equal to* (!=) comparison operation with broadcasting. | +|[*broadcast_greater*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_greater) | Returns the result of element-wise *greater than* (>) comparison operation with broadcasting. | +|[*broadcast_greater_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_greater_equal) | Returns the result of element-wise *greater than or equal to* (>=) comparison operation with broadcasting. | +|[*broadcast_lesser*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser) | Returns the result of element-wise *lesser than* (<) comparison operation with broadcasting. | +|[*broadcast_lesser_equal*](https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.broadcast_lesser_equal) | Returns the result of element-wise *lesser than or equal to* (<=) comparison operation with broadcasting. | + +For example, if you want to add a NDarray to your input x, use `broadcast_add` instead of `+`: + +```python +def hybrid_forward(self, F, x): + # avoid writing: return x + F.ones((1, 1)) + return F.broadcast_add(x, F.ones((1, 1))) +``` + +If you used `+`, it would still work before hybridization, but will throw an error of shape missmtach after hybridization. + +### Shape + +Gluon's imperative interface is very flexible and allows you to print the shape of the NDArray. However, Symbol does not have shape attributes. As a result, you need to avoid printing shapes in `hybrid_forward`. +Otherwise, you will get the following error: +```bash +AttributeError: 'Symbol' object has no attribute 'shape' +``` + +### Slice +`[]` in NDArray is used to get a slice from the array. However, `[]` in Symbol is used to get an output from a grouped symbol. +For example, you will get different results for the following method before and after hybridization. + +```python +def hybrid_forward(self, F, x): + return x[0] +``` + +The current workaround is to explicitly call [`slice`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice) or [`slice_axis`](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.slice_axis) operators in `hybrid_forward`. + + +### Not implemented operators + +Some of the often used operators in NDArray are not implemented in Symbol, and will cause hybridization failure. + +#### NDArray.asnumpy +Symbol does not support the `asnumpy` function. You need to avoid calling `asnumpy` in `hybrid_forward`. + +#### Array creation APIs + +`mx.nd.array()` is used a lot, but Symbol does not have the `array` API. The current workaround is to use `F.ones`, `F.zeros`, or `F.full`, which exist in both the NDArray and Symbol APIs. + +#### In-Place Arithmetic Operators + +In-place arithmetic operators may be used in Gluon imperative mode, however if you expect to hybridize, you should write these operations explicitly instead. +For example, avoid writing `x += y` and use `x = x + y`, otherwise you will get `NotImplementedError`. This applies to all the following operators: + +| NDArray in-place arithmetic operators | Description | +|---|---| +|[*NDArray.\__iadd\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__iadd__) | x.\__iadd\__(y) <=> x+=y | +|[*NDArray.\__isub\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__isub__) | x.\__isub\__(y) <=> x-=y | +|[*NDArray.\__imul\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imul__) | x.\__imul\__(y) <=> x*=y | +|[*NDArray.\__idiv\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__idiv__) | x.\__rdiv\__(y) <=> x/=y | +|[*NDArray.\__imod\__*](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.NDArray.__imod__) | x.\__rmod\__(y) <=> x%=y | + + + +## Summary + +The recommended practice is to utilize the flexibility of imperative NDArray API during experimentation. Once you finalized your model, make necessary changes mentioned above so you can call `hybridize` function to improve performance. + + \ No newline at end of file diff --git a/docs/tutorials/gluon/info_gan.md b/docs/tutorials/gluon/info_gan.md new file mode 100644 index 000000000..8b2668ab2 --- /dev/null +++ b/docs/tutorials/gluon/info_gan.md @@ -0,0 +1,437 @@ + +# Image similarity search with InfoGAN + +This notebook shows how to implement an InfoGAN based on Gluon. InfoGAN is an extension of GANs, where the generator input is split in 2 parts: random noise and a latent code (see [InfoGAN Paper](https://arxiv.org/pdf/1606.03657.pdf)). +The codes are made meaningful by maximizing the mutual information between code and generator output. InfoGAN learns a disentangled representation in a completely unsupervised manner. It can be used for many applications such as image similarity search. This notebook uses the DCGAN example from the [Straight Dope Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html) and extends it to create an InfoGAN. + + +```python +from __future__ import print_function +from datetime import datetime +import logging +import multiprocessing +import os +import sys +import tarfile +import time + +import numpy as np +from matplotlib import pyplot as plt +from mxboard import SummaryWriter +import mxnet as mx +from mxnet import gluon +from mxnet import ndarray as nd +from mxnet.gluon import nn, utils +from mxnet import autograd + +``` + +The latent code vector can contain several variables, which can be categorical and/or continuous. We set `n_continuous` to 2 and `n_categories` to 10. + + +```python +batch_size = 64 +z_dim = 100 +n_continuous = 2 +n_categories = 10 +ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu() +``` + +Some functions to load and normalize images. + + +```python +lfw_url = 'http://vis-www.cs.umass.edu/lfw/lfw-deepfunneled.tgz' +data_path = 'lfw_dataset' +if not os.path.exists(data_path): + os.makedirs(data_path) + data_file = utils.download(lfw_url) + with tarfile.open(data_file) as tar: + tar.extractall(path=data_path) + +``` + + +```python +def transform(data, width=64, height=64): + data = mx.image.imresize(data, width, height) + data = nd.transpose(data, (2,0,1)) + data = data.astype(np.float32)/127.5 - 1 + if data.shape[0] == 1: + data = nd.tile(data, (3, 1, 1)) + return data.reshape((1,) + data.shape) +``` + + +```python +def get_files(data_dir): + images = [] + filenames = [] + for path, _, fnames in os.walk(data_dir): + for fname in fnames: + if not fname.endswith('.jpg'): + continue + img = os.path.join(path, fname) + img_arr = mx.image.imread(img) + img_arr = transform(img_arr) + images.append(img_arr) + filenames.append(path + "/" + fname) + return images, filenames +``` + +Load the dataset `lfw_dataset` which contains images of celebrities. + + +```python +data_dir = 'lfw_dataset' +images, filenames = get_files(data_dir) +split = int(len(images)*0.8) +test_images = images[split:] +test_filenames = filenames[split:] +train_images = images[:split] +train_filenames = filenames[:split] + +train_data = gluon.data.ArrayDataset(nd.concatenate(train_images)) +train_dataloader = gluon.data.DataLoader(train_data, batch_size=batch_size, shuffle=True, last_batch='rollover', num_workers=multiprocessing.cpu_count()-1) +``` + +## Generator +Define the Generator model. Architecture is taken from the DCGAN implementation in [Straight Dope Book](https://gluon.mxnet.io/chapter14_generative-adversarial-networks/dcgan.html). The Generator consist of 4 layers where each layer involves a strided convolution, batch normalization, and rectified nonlinearity. It takes as input random noise and the latent code and produces an `(64,64,3)` output image. + + +```python +class Generator(gluon.HybridBlock): + def __init__(self, **kwargs): + super(Generator, self).__init__(**kwargs) + with self.name_scope(): + self.prev = nn.HybridSequential() + self.prev.add(nn.Dense(1024, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu')) + self.G = nn.HybridSequential() + + self.G.add(nn.Conv2DTranspose(64 * 8, 4, 1, 0, use_bias=False)) + self.G.add(nn.BatchNorm()) + self.G.add(nn.Activation('relu')) + self.G.add(nn.Conv2DTranspose(64 * 4, 4, 2, 1, use_bias=False)) + self.G.add(nn.BatchNorm()) + self.G.add(nn.Activation('relu')) + self.G.add(nn.Conv2DTranspose(64 * 2, 4, 2, 1, use_bias=False)) + self.G.add(nn.BatchNorm()) + self.G.add(nn.Activation('relu')) + self.G.add(nn.Conv2DTranspose(64, 4, 2, 1, use_bias=False)) + self.G.add(nn.BatchNorm()) + self.G.add(nn.Activation('relu')) + self.G.add(nn.Conv2DTranspose(3, 4, 2, 1, use_bias=False)) + self.G.add(nn.Activation('tanh')) + + def hybrid_forward(self, F, x): + x = self.prev(x) + x = F.reshape(x, (0, -1, 1, 1)) + return self.G(x) +``` + +## Discriminator +Define the Discriminator and Q model. The Q model shares many layers with the Discriminator. Its task is to estimate the code `c` for a given fake image. It is used to maximize the lower bound to the mutual information. + + +```python +class Discriminator(gluon.HybridBlock): + def __init__(self, **kwargs): + super(Discriminator, self).__init__(**kwargs) + with self.name_scope(): + self.D = nn.HybridSequential() + self.D.add(nn.Conv2D(64, 4, 2, 1, use_bias=False)) + self.D.add(nn.LeakyReLU(0.2)) + self.D.add(nn.Conv2D(64 * 2, 4, 2, 1, use_bias=False)) + self.D.add(nn.BatchNorm()) + self.D.add(nn.LeakyReLU(0.2)) + self.D.add(nn.Conv2D(64 * 4, 4, 2, 1, use_bias=False)) + self.D.add(nn.BatchNorm()) + self.D.add(nn.LeakyReLU(0.2)) + self.D.add(nn.Conv2D(64 * 8, 4, 2, 1, use_bias=False)) + self.D.add(nn.BatchNorm()) + self.D.add(nn.LeakyReLU(0.2)) + + self.D.add(nn.Dense(1024, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu')) + + self.prob = nn.Dense(1) + self.feat = nn.HybridSequential() + self.feat.add(nn.Dense(128, use_bias=False), nn.BatchNorm(), nn.Activation(activation='relu')) + self.category_prob = nn.Dense(n_categories) + self.continuous_mean = nn.Dense(n_continuous) + self.Q = nn.HybridSequential() + self.Q.add(self.feat, self.category_prob, self.continuous_mean) + + def hybrid_forward(self, F, x): + x = self.D(x) + prob = self.prob(x) + feat = self.feat(x) + category_prob = self.category_prob(feat) + continuous_mean = self.continuous_mean(feat) + + return prob, category_prob, continuous_mean +``` + +The InfoGAN has the following layout. + + +Discriminator and Generator are the same as in the DCGAN example. On top of the Disciminator is the Q model, which is estimating the code `c` for given fake images. The Generator's input is random noise and the latent code `c`. + +## Training Loop +Initialize Generator and Discriminator and define correspoing trainer function. + + +```python +generator = Generator() +generator.hybridize() +generator.initialize(mx.init.Normal(0.002), ctx=ctx) + +discriminator = Discriminator() +discriminator.hybridize() +discriminator.initialize(mx.init.Normal(0.002), ctx=ctx) + +lr = 0.0001 +beta = 0.5 + +g_trainer = gluon.Trainer(generator.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta}) +d_trainer = gluon.Trainer(discriminator.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta}) +q_trainer = gluon.Trainer(discriminator.Q.collect_params(), 'adam', {'learning_rate': lr, 'beta1': beta}) +``` + +Create vectors with real (=1) and fake labels (=0). + + +```python +real_label = nd.ones((batch_size,), ctx=ctx) +fake_label = nd.zeros((batch_size,),ctx=ctx) +``` + +Load a pretrained model. + + +```python +if os.path.isfile('infogan_d_latest.params') and os.path.isfile('infogan_g_latest.params'): + discriminator.load_parameters('infogan_d_latest.params', ctx=ctx, allow_missing=True, ignore_extra=True) + generator.load_parameters('infogan_g_latest.params', ctx=ctx, allow_missing=True, ignore_extra=True) +``` +There are 2 differences between InfoGAN and DCGAN: the extra latent code and the Q network to estimate the code. +The latent code is part of the Generator input and it contains mutliple variables (continuous, categorical) that can represent different distributions. In order to make sure that the Generator uses the latent code, mutual information is introduced into the GAN loss term. Mutual information measures how much X is known given Y or vice versa. It is defined as: + +![gif](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/info_gan/entropy.gif) + +The InfoGAN loss is: + +![gif](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/info_gan/loss.gif) + +where `V(D,G)` is the GAN loss and the mutual information `I(c, G(z, c))` goes in as regularization. The goal is to reach high mutual information, in order to learn meaningful codes for the data. + + +Define the loss functions. `SoftmaxCrossEntropyLoss` for the categorical code, `L2Loss` for the continious code and `SigmoidBinaryCrossEntropyLoss` for the normal GAN loss. + + +```python +loss1 = gluon.loss.SigmoidBinaryCrossEntropyLoss() +loss2 = gluon.loss.L2Loss() +loss3 = gluon.loss.SoftmaxCrossEntropyLoss() +``` + +This function samples `c`, `z`, and concatenates them to create the generator input. + + +```python +def create_generator_input(): + + #create random noise + z = nd.random_normal(0, 1, shape=(batch_size, z_dim), ctx=ctx) + label = nd.array(np.random.randint(n_categories, size=batch_size)).as_in_context(ctx) + c1 = nd.one_hot(label, depth=n_categories).as_in_context(ctx) + c2 = nd.random.uniform(-1, 1, shape=(batch_size, n_continuous)).as_in_context(ctx) + + # concatenate random noise with c which will be the input of the generator + return nd.concat(z, c1, c2, dim=1), label, c2 +``` + +Define the training loop. +1. The discriminator receives `real_data` and `loss1` measures how many real images have been identified as real +2. The discriminator receives `fake_image` from the Generator and `loss1` measures how many fake images have been identified as fake +3. Update Discriminator. Currently, it is updated every second iteration in order to avoid that the Discriminator becomes too strong. You may want to change that. +4. The updated discriminator receives `fake_image` and `loss1` measures how many fake images have been been identified as real, `loss2` measures the difference between the sampled continuous latent code `c` and the output of the Q model and `loss3` measures the difference between the sampled categorical latent code `c` and the output of the Q model. +4. Update Generator and Q + + +```python +with SummaryWriter(logdir='./logs/') as sw: + + epochs = 1 + counter = 0 + for epoch in range(epochs): + print("Epoch", epoch) + starttime = time.time() + + d_error_epoch = nd.zeros((1,), ctx=ctx) + g_error_epoch = nd.zeros((1,), ctx=ctx) + + for idx, data in enumerate(train_dataloader): + + #get real data and generator input + real_data = data.as_in_context(ctx) + g_input, label, c2 = create_generator_input() + + + #Update discriminator: Input real data and fake data + with autograd.record(): + output_real,_,_ = discriminator(real_data) + d_error_real = loss1(output_real, real_label) + + # create fake image and input it to discriminator + fake_image = generator(g_input) + output_fake,_,_ = discriminator(fake_image.detach()) + d_error_fake = loss1(output_fake, fake_label) + + # total discriminator error + d_error = d_error_real + d_error_fake + + d_error_epoch += d_error.mean() + + #Update D every second iteration + if (counter+1) % 2 == 0: + d_error.backward() + d_trainer.step(batch_size) + + #Update generator: Input random noise and latent code vector + with autograd.record(): + fake_image = generator(g_input) + output_fake, category_prob, continuous_mean = discriminator(fake_image) + g_error = loss1(output_fake, real_label) + loss3(category_prob, label) + loss2(c2, continuous_mean) + + g_error.backward() + g_error_epoch += g_error.mean() + + g_trainer.step(batch_size) + q_trainer.step(batch_size) + + # logging + if idx % 10 == 0: + count = idx + 1 + logging.info('speed: {} samples/s'.format(batch_size / (time.time() - starttime))) + logging.info('discriminator loss = %f, generator loss = %f at iter %d epoch %d' + %(d_error_epoch.asscalar()/count,g_error_epoch.asscalar()/count, count, epoch)) + + g_input,_,_ = create_generator_input() + + # create some fake image for logging in MXBoard + fake_image = generator(g_input) + + sw.add_scalar(tag='Loss_D', value={'test':d_error_epoch.asscalar()/count}, global_step=counter) + sw.add_scalar(tag='Loss_G', value={'test':d_error_epoch.asscalar()/count}, global_step=counter) + sw.add_image(tag='data_image', image=((fake_image[0]+ 1.0) * 127.5).astype(np.uint8) , global_step=counter) + sw.flush() + + discriminator.save_parameters("infogan_d_latest.params") + generator.save_parameters("infogan_g_latest.params") +``` + +## Image similarity +Once the InfoGAN is trained, we can use the Discriminator to do an image similarity search. The idea is that the network learned meaningful features from the images based on the mutual information e.g. pose of people in an image. + +Load the trained discriminator and retrieve one of its last layers. + + +```python +discriminator = Discriminator() +discriminator.load_parameters("infogan_d_latest.params", ctx=ctx, ignore_extra=True) + +discriminator = discriminator.D[:11] +print (discriminator) + +discriminator.hybridize() +``` + +Nearest neighbor function, which takes a matrix of features and an input feature vector. It returns the 3 closest features. + + +```python +def get_knn(features, input_vector, k=3): + dist = (nd.square(features - input_vector).sum(axis=1))/features.shape[0] + indices = dist.asnumpy().argsort()[:k] + return [(index, dist[index].asscalar()) for index in indices] +``` + +A helper function to visualize image data. + + +```python +def visualize(img_array): + plt.imshow(((img_array.asnumpy().transpose(1, 2, 0) + 1.0) * 127.5).astype(np.uint8)) + plt.axis('off') +``` + +Take some images from the test data, obtain its feature vector from `discriminator.D[:11]` and plot images of the corresponding closest vectors in the feature space. + + +```python +feature_size = 8192 + +features = nd.zeros((len(test_images), feature_size), ctx=ctx) + +for idx, image in enumerate(test_images): + + feature = discriminator(nd.array(image, ctx=ctx)) + feature = feature.reshape(feature_size,) + features[idx,:] = feature.copyto(ctx) + + +for image in test_images[:100]: + + feature = discriminator(mx.nd.array(image, ctx=ctx)) + feature = feature.reshape((feature_size,)) + image = image.reshape((3,64,64)) + + + indices = get_knn(features, feature, k=10) + fig = plt.figure(figsize=(15,12)) + plt.subplot(1,10,1) + + visualize(image) + for i in range(2,9): + if indices[i-1][1] < 1.5: + plt.subplot(1,10,i) + sim = test_images[indices[i-1][0]].reshape(3,64,64) + visualize(sim) + plt.show() + plt.clf() +``` +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/info_gan/output.png) + +## How the Generator learns +We trained the Generator for a couple of epochs and stored a couple of fake images per epoch. Check the video. + ![alt text](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/info_gan/infogan.gif) + + +The following function computes the TSNE on the feature matrix and stores the result in a json-file. This file can be loaded with [TSNEViewer](https://ml4a.github.io/guides/ImageTSNEViewer/) + + +```python +import json + +from sklearn.manifold import TSNE +from scipy.spatial import distance + +tsne = TSNE(n_components=2, learning_rate=150, perplexity=30, verbose=2).fit_transform(features.asnumpy()) + +# save data to json +data = [] +counter = 0 +for i,f in enumerate(test_filenames): + + point = [float((tsne[i,k] - np.min(tsne[:,k]))/(np.max(tsne[:,k]) - np.min(tsne[:,k]))) for k in range(2) ] + data.append({"path": os.path.abspath(os.path.join(os.getcwd(),f)), "point": point}) + +with open("imagetsne.json", 'w') as outfile: + json.dump(data, outfile) +``` + +Load the file with TSNEViewer. You can now inspect whether similiar looking images are grouped nearby or not. + + + + diff --git a/docs/tutorials/gluon/learning_rate_finder.md b/docs/tutorials/gluon/learning_rate_finder.md index 661a01709..b571a53f6 100644 --- a/docs/tutorials/gluon/learning_rate_finder.md +++ b/docs/tutorials/gluon/learning_rate_finder.md @@ -80,7 +80,6 @@ We also adjust our `DataLoader` so that it continuously provides batches of data ```python -from multiprocessing import cpu_count from mxnet.gluon.data.vision import transforms transform = transforms.Compose([ @@ -109,7 +108,7 @@ class ContinuousBatchSampler(): sampler = mx.gluon.data.RandomSampler(len(dataset)) batch_sampler = ContinuousBatchSampler(sampler, batch_size=128) -data_loader = mx.gluon.data.DataLoader(dataset, batch_sampler=batch_sampler, num_workers=cpu_count()) +data_loader = mx.gluon.data.DataLoader(dataset, batch_sampler=batch_sampler) ``` ## Implementation @@ -143,7 +142,7 @@ class LRFinder(): self.learner.trainer._init_kvstore() # Store params and optimizer state for restore after lr_finder procedure # Useful for applying the method partway through training, not just for initialization of lr. - self.learner.net.save_params("lr_finder.params") + self.learner.net.save_parameters("lr_finder.params") self.learner.trainer.save_states("lr_finder.state") lr = lr_start self.results = [] # List of (lr, loss) tuples @@ -156,7 +155,7 @@ class LRFinder(): break lr = lr * lr_multiplier # Restore params (as finder changed them) - self.learner.net.load_params("lr_finder.params", ctx=self.learner.ctx) + self.learner.net.load_parameters("lr_finder.params", ctx=self.learner.ctx) self.learner.trainer.load_states("lr_finder.state") return self.results @@ -231,10 +230,10 @@ As discussed before, we should select a learning rate where the loss is falling ```python -learner.net.save_params("net.params") +learner.net.save_parameters("net.params") lr = 0.05 -for iter_idx in range(500): +for iter_idx in range(300): learner.iteration(lr=lr) if ((iter_idx % 100) == 0): print("Iteration: {}, Loss: {:.5g}".format(iter_idx, learner.iteration_loss)) @@ -247,9 +246,6 @@ Iteration: 100, Loss: 1.6653 Iteration: 200, Loss: 1.4891 -Iteration: 300, Loss: 1.0846 - -Iteration: 400, Loss: 1.0633 Final Loss: 1.1812 @@ -262,10 +258,10 @@ And now we have a baseline, let's see what happens when we train with a learning ```python net = mx.gluon.model_zoo.vision.resnet18_v2(classes=10) learner = Learner(net=net, data_loader=data_loader, ctx=ctx) -learner.net.load_params("net.params", ctx=ctx) +learner.net.load_parameters("net.params", ctx=ctx) lr = 0.5 -for iter_idx in range(500): +for iter_idx in range(300): learner.iteration(lr=lr) if ((iter_idx % 100) == 0): print("Iteration: {}, Loss: {:.5g}".format(iter_idx, learner.iteration_loss)) @@ -278,9 +274,6 @@ Iteration: 100, Loss: 1.9666 Iteration: 200, Loss: 1.6919 -Iteration: 300, Loss: 1.3643 - -Iteration: 400, Loss: 1.4743 Final Loss: 1.366 @@ -293,10 +286,10 @@ And lastly, we see how the model trains with a more conservative learning rate o ```python net = mx.gluon.model_zoo.vision.resnet18_v2(classes=10) learner = Learner(net=net, data_loader=data_loader, ctx=ctx) -learner.net.load_params("net.params", ctx=ctx) +learner.net.load_parameters("net.params", ctx=ctx) lr = 0.005 -for iter_idx in range(500): +for iter_idx in range(300): learner.iteration(lr=lr) if ((iter_idx % 100) == 0): print("Iteration: {}, Loss: {:.5g}".format(iter_idx, learner.iteration_loss)) @@ -309,9 +302,6 @@ Iteration: 100, Loss: 1.8621 Iteration: 200, Loss: 1.6316 -Iteration: 300, Loss: 1.6295 - -Iteration: 400, Loss: 1.4019 Final Loss: 1.2919 diff --git a/docs/tutorials/gluon/learning_rate_schedules.md b/docs/tutorials/gluon/learning_rate_schedules.md index dc340b799..88b109e7f 100644 --- a/docs/tutorials/gluon/learning_rate_schedules.md +++ b/docs/tutorials/gluon/learning_rate_schedules.md @@ -12,7 +12,6 @@ In this tutorial, we visualize the schedules defined in `mx.lr_scheduler`, show ```python -%matplotlib inline from __future__ import print_function import math import matplotlib.pyplot as plt @@ -20,6 +19,7 @@ import mxnet as mx from mxnet.gluon import nn from mxnet.gluon.data.vision import transforms import numpy as np +%matplotlib inline ``` ```python @@ -134,7 +134,7 @@ batch_size = 64 # Load the training data train_dataset = mx.gluon.data.vision.MNIST(train=True).transform_first(transforms.ToTensor()) -train_dataloader = mx.gluon.data.DataLoader(train_dataset, batch_size, shuffle=True) +train_dataloader = mx.gluon.data.DataLoader(train_dataset, batch_size, shuffle=True, num_workers=5) # Build a simple convolutional network def build_cnn(): diff --git a/docs/tutorials/gluon/mnist.md b/docs/tutorials/gluon/mnist.md index 5b8a98a3d..35fb40521 100644 --- a/docs/tutorials/gluon/mnist.md +++ b/docs/tutorials/gluon/mnist.md @@ -1,24 +1,22 @@ -# Handwritten Digit Recognition +# Hand-written Digit Recognition -In this tutorial, we'll give you a step by step walk-through of how to build a hand-written digit classifier using the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset. +In this tutorial, we'll give you a step-by-step walkthrough of building a hand-written digit classifier using the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset. -MNIST is a widely used dataset for the hand-written digit classification task. It consists of 70,000 labeled 28x28 pixel grayscale images of hand-written digits. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits). The task at hand is to train a model using the 60,000 training images and subsequently test its classification accuracy on the 10,000 test images. +MNIST is a widely used dataset for the hand-written digit classification task. It consists of 70,000 labeled grayscale images of hand-written digits, each 28x28 pixels in size. The dataset is split into 60,000 training images and 10,000 test images. There are 10 classes (one for each of the 10 digits). The task at hand is to train a model that can correctly classify the images into the digits they represent. The 60,000 training images are used to fit the model, and its performance in terms of classification accuracy is subsequently validated on the 10,000 test images. ![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/example/mnist.png) **Figure 1:** Sample images from the MNIST dataset. -This tutorial uses MXNet's new high-level interface, gluon package to implement MLP using -imperative fashion. - -This is based on the Mnist tutorial with symbolic approach. You can find it [here](http://mxnet.io/tutorials/python/mnist.html). +This tutorial uses MXNet's high-level *Gluon* interface to implement neural networks in an imperative fashion. It is based on [the corresponding tutorial written with the symbolic approach](https://mxnet.incubator.apache.org/tutorials/python/mnist.html). ## Prerequisites -To complete this tutorial, we need: -- MXNet. See the instructions for your operating system in [Setup and Installation](http://mxnet.io/install/index.html). +To complete this tutorial, you need: -- [Python Requests](http://docs.python-requests.org/en/master/) and [Jupyter Notebook](http://jupyter.org/index.html). +- MXNet. See the instructions for your operating system in [Setup and Installation](https://mxnet.incubator.apache.org/install/index.html). +- The Python [`requests`](http://docs.python-requests.org/en/master/) library. +- (Optional) The [Jupyter Notebook](https://jupyter.org/index.html) software for interactively running the provided `.ipynb` file. ``` $ pip install requests jupyter @@ -26,308 +24,420 @@ $ pip install requests jupyter ## Loading Data -Before we define the model, let's first fetch the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. +The following code downloads the MNIST dataset to the default location (`.mxnet/datasets/mnist/` in your home directory) and creates `Dataset` objects `train_data` and `val_data` for training and validation, respectively. +These objects can later be used to get one image or a batch of images at a time, together with their corresponding labels. -The following source code downloads and loads the images and the corresponding labels into memory. +We also immediately apply the `transform_first()` method and supply a function that moves the channel axis of the images to the beginning (`(28, 28, 1) -> (1, 28, 28)`), casts them to `float32` and rescales them from `[0, 255]` to `[0, 1]`. +The name `transform_first` reflects the fact that these datasets contain images and labels, and that the transform should only be applied to the first of each `(image, label)` pair. ```python import mxnet as mx -# Fixing the random seed +# Select a fixed random seed for reproducibility mx.random.seed(42) -mnist = mx.test_utils.get_mnist() +def data_xform(data): + """Move channel axis to the beginning, cast to float32, and normalize to [0, 1].""" + return nd.moveaxis(data, 2, 0).astype('float32') / 255 + +train_data = mx.gluon.data.vision.MNIST(train=True).transform_first(data_xform) +val_data = mx.gluon.data.vision.MNIST(train=False).transform_first(data_xform) ``` -After running the above source code, the entire MNIST dataset should be fully loaded into memory. Note that for large datasets it is not feasible to pre-load the entire dataset first like we did here. What is needed is a mechanism by which we can quickly and efficiently stream data directly from the source. MXNet Data iterators come to the rescue here by providing exactly that. Data iterator is the mechanism by which we feed input data into an MXNet training algorithm and they are very simple to initialize and use and are optimized for speed. During training, we typically process training samples in small batches and over the entire training lifetime will end up processing each training example multiple times. In this tutorial, we'll configure the data iterator to feed examples in batches of 100. Keep in mind that each example is a 28x28 grayscale image and the corresponding label. +Since the MNIST dataset is relatively small, the `MNIST` class loads it into memory all at once, but for larger datasets like ImageNet, this would no longer be possible. +The Gluon `Dataset` class from which `MNIST` derives supports both cases. +In general, `Dataset` and `DataLoader` (which we will encounter next) are the machinery in MXNet that provides a stream of input data to be consumed by a training algorithm, typically in batches of multiple data entities at once for better efficiency. +In this tutorial, we will configure the data loader to feed examples in batches of 100. + +An image batch is commonly represented as a 4-D array with shape `(batch_size, num_channels, height, width)`. +This convention is denoted by "NCHW", and it is the default in MXNet. +For the MNIST dataset, each image has a size of 28x28 pixels and one color channel (grayscale), hence the shape of an input batch will be `(batch_size, 1, 28, 28)`. -Image batches are commonly represented by a 4-D array with shape `(batch_size, num_channels, width, height)`. For the MNIST dataset, since the images are grayscale, there is only one color channel. Also, the images are 28x28 pixels, and so each image has width and height equal to 28. Therefore, the shape of input is `(batch_size, 1, 28, 28)`. Another important consideration is the order of input samples. When feeding training examples, it is critical that we don't feed samples with the same label in succession. Doing so can slow down training. -Data iterators take care of this by randomly shuffling the inputs. Note that we only need to shuffle the training data. The order does not matter for test data. +Another important consideration is the order of input samples. +When feeding training examples, it is critical not feed samples with the same label in succession since doing so can slow down training progress. +Data iterators, i.e., instances of [`DataLoader`](https://mxnet.incubator.apache.org/api/python/gluon/data.html#mxnet.gluon.data.DataLoader), take care of this issue by randomly shuffling the inputs. +Note that we only need to shuffle the training data -- for validation data, the order does not matter. -The following source code initializes the data iterators for the MNIST dataset. Note that we initialize two iterators: one for train data and one for test data. +The following code initializes the data iterators for the MNIST dataset. ```python batch_size = 100 -train_data = mx.io.NDArrayIter(mnist['train_data'], mnist['train_label'], batch_size, shuffle=True) -val_data = mx.io.NDArrayIter(mnist['test_data'], mnist['test_label'], batch_size) +train_loader = mx.gluon.data.DataLoader(train_data, shuffle=True, batch_size=batch_size) +val_loader = mx.gluon.data.DataLoader(val_data, shuffle=False, batch_size=batch_size) ``` ## Approaches -We will cover a couple of approaches for performing the hand written digit recognition task. The first approach makes use of a traditional deep neural network architecture called Multilayer Perceptron (MLP). We'll discuss its drawbacks and use that as a motivation to introduce a second more advanced approach called Convolution Neural Network (CNN) that has proven to work very well for image classification tasks. +We will cover two approaches for performing the hand-written digit recognition task. +In our first attempt, we will make use of a traditional neural network architecture called [Multilayer Perceptron (MLP)](https://en.wikipedia.org/wiki/Multilayer_perceptron). +Although this architecture lets us achieve over 95 % accuracy on the validation set, we will recognize and discuss some of its drawbacks and use them as a motivation for using a different network. +In the subsequent second attempt, we introduce the more advanced and very widely used [Convolutional Neural Network (CNN)](https://en.wikipedia.org/wiki/Convolutional_neural_network) architecture that has proven to work very well for image classification tasks. -Now, let's import required nn modules +As a first step, we run some convenience imports of frequently used modules. ```python -from __future__ import print_function +from __future__ import print_function # only relevant for Python 2 import mxnet as mx -from mxnet import gluon +from mxnet import nd, gluon, autograd from mxnet.gluon import nn -from mxnet import autograd as ag ``` -### Define a network: Multilayer Perceptron +### Defining a network: Multilayer Perceptron (MLP) -The first approach makes use of a [Multilayer Perceptron](https://en.wikipedia.org/wiki/Multilayer_perceptron) to solve this problem. We'll define the MLP using MXNet's imperative approach. +MLPs consist of several fully connected layers. +In a fully connected (short: FC) layer, each neuron is connected to every neuron in its preceding layer. +From a linear algebra perspective, an FC layer applies an [affine transform](https://en.wikipedia.org/wiki/Affine_transformation) *Y = X W + b* to an input matrix *X* of size (*n x m*) and outputs a matrix *Y* of size (*n x k*). +The number *k*, also referred to as *hidden size*, corresponds to the number of neurons in the FC layer. +An FC layer has two learnable parameters: the (*m x k*) weight matrix *W* and the (*1 x k*) bias vector *b*. -MLPs consist of several fully connected layers. A fully connected layer or FC layer for short, is one where each neuron in the layer is connected to every neuron in its preceding layer. From a linear algebra perspective, an FC layer applies an [affine transform](https://en.wikipedia.org/wiki/Affine_transformation) to the *n x m* input matrix *X* and outputs a matrix *Y* of size *n x k*, where *k* is the number of neurons in the FC layer. *k* is also referred to as the hidden size. The output *Y* is computed according to the equation *Y = W X + b*. The FC layer has two learnable parameters, the *m x k* weight matrix *W* and the *m x 1* bias vector *b*. +In an MLP, the outputs of FC layers are typically fed into an activation function that applies an elementwise nonlinearity. +This step is crucial since it gives neural networks the ability to classify inputs that are not linearly separable. +Common choices for activation functions are [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function), [hyperbolic tangent ("tanh")](https://en.wikipedia.org/wiki/Hyperbolic_function#Definitions), and [rectified linear unit (ReLU)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)). +In this example, we'll use the ReLU activation function since it has several nice properties that make it a good default choice. -In an MLP, the outputs of most FC layers are fed into an activation function, which applies an element-wise non-linearity. This step is critical and it gives neural networks the ability to classify inputs that are not linearly separable. Common choices for activation functions are sigmoid, tanh, and [rectified linear unit](https://en.wikipedia.org/wiki/Rectifier_%28neural_networks%29) (ReLU). In this example, we'll use the ReLU activation function which has several desirable properties and is typically considered a default choice. +The following code snippet declares three fully connected (or *dense*) layers with 128, 64 and 10 neurons each, where the last number of neurons matches the number of output classes in our dataset. +Note that the last layer uses no activation function since the [softmax](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#mxnet.ndarray.softmax) activation will be implicitly applied by the loss function later on. +To build the neural network, we use a [`HybridSequential`](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.HybridSequential) layer, which is a convenience class to build a linear stack of layers, often called a *feed-forward neural net*. -The following code declares three fully connected layers with 128, 64 and 10 neurons each. -The last fully connected layer often has its hidden size equal to the number of output classes in the dataset. Furthermore, these FC layers uses ReLU activation for performing an element-wise ReLU transformation on the FC layer output. - -To do this, we will use [Sequential layer](http://mxnet.io/api/python/gluon/gluon.html#mxnet.gluon.nn.Sequential) type. This is simply a linear stack of neural network layers. `nn.Dense` layers are nothing but the fully connected layers we discussed above. +The "Hybrid" part of name `HybridSequential` refers to the fact that such a layer can be used with both the Gluon API and the Symbol API. +Using hybrid blocks over dynamic-only blocks (e.g. [`Sequential`](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.nn.Sequential)) has several advantages apart from being compatible with a wider range of existing code: for instance, the computation graph of the network can be visualized with `mxnet.viz.plot_network()` and inspected for errors. +Unless a network requires non-static runtime elements like loops, conditionals or random layer selection in its forward pass, it is generally a good idea to err on the side of hybrid blocks. +For details on the differences, see the documentation on [`Block`](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.Block) and [`HybridBlock`](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#mxnet.gluon.HybridBlock). ```python -# define network -net = nn.Sequential() +net = nn.HybridSequential(prefix='MLP_') with net.name_scope(): - net.add(nn.Dense(128, activation='relu')) - net.add(nn.Dense(64, activation='relu')) - net.add(nn.Dense(10)) + net.add( + nn.Flatten(), + nn.Dense(128, activation='relu'), + nn.Dense(64, activation='relu'), + nn.Dense(10, activation=None) # loss function includes softmax already, see below + ) ``` -#### Initialize parameters and optimizer +**Note**: using the `name_scope()` context manager is optional. +It is, however, good practice since it uses a common prefix for the names of all layers generated in that scope, which can be very helpful during debugging. -The following source code initializes all parameters received from parameter dict using [Xavier](http://mxnet.io/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initializer -to train the MLP network we defined above. +#### Initializing parameters and optimizer -For our training, we will make use of the stochastic gradient descent (SGD) optimizer. In particular, we'll be using mini-batch SGD. Standard SGD processes train data one example at a time. In practice, this is very slow and one can speed up the process by processing examples in small batches. In this case, our batch size will be 100, which is a reasonable choice. Another parameter we select here is the learning rate, which controls the step size the optimizer takes in search of a solution. We'll pick a learning rate of 0.02, again a reasonable choice. Settings such as batch size and learning rate are what are usually referred to as hyper-parameters. What values we give them can have a great impact on training performance. +Before the network can be used, its parameters (weights and biases) need to be set to initial values that are sufficiently random while keeping the magnitude of gradients limited. +The [Xavier](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.initializer.Xavier) initializer is usually a good default choice. -We will use [Trainer](http://mxnet.io/api/python/gluon/gluon.html#trainer) class to apply the -[SGD optimizer](http://mxnet.io/api/python/optimization/optimization.html#mxnet.optimizer.SGD) on the -initialized parameters. +Since the `net.initialize()` method creates arrays for its parameters, it needs to know where to store the values: in CPU or GPU memory. +Like many other functions and classes that deal with memory management in one way or another, the `initialize()` method takes an optional `ctx` (short for *context*) argument, where the return value of either `mx.cpu()` or `mx.gpu()` can be provided. ```python -gpus = mx.test_utils.list_gpus() -ctx = [mx.gpu()] if gpus else [mx.cpu(0), mx.cpu(1)] -net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx) -trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.02}) +ctx = mx.gpu(0) if mx.context.num_gpus() > 0 else mx.cpu(0) +net.initialize(mx.init.Xavier(), ctx=ctx) ``` -#### Train the network +To train the network parameters, we will make use of the [stochastic gradient descent (SGD)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) optimizer. +More specifically, we use mini-batch SGD in contrast to the classical SGD that processes one example at a time, which is very slow in practice. +(Recall that we set the batch size to 100 in the ["Loading Data"](#loading-data) part.) + +Besides the batch size, the SGD algorithm has one important *hyperparameter*: the *learning rate*. +It determines the size of steps that the algorithm takes in search of parameters that allow the network to optimally fit the training data. +Therefore, this value has great influence on both the course of the training process and its final outcome. +In general, hyperparameters refer to *non-learnable* values that need to be chosen before training and that have a potential effect on the outcome. +In this example, further hyperparameters are the number of layers in the network, the number of neurons of the first two layers, the activation function and (later) the loss function. + +The SGD optimization method can be accessed in MXNet Gluon through the [`Trainer`](https://mxnet.incubator.apache.org/api/python/gluon/gluon.html#trainer) class. +Internally, it makes use of the [`SGD`](https://mxnet.incubator.apache.org/api/python/optimization/optimization.html#mxnet.optimizer.SGD) optimizer class. -Typically, one runs the training until convergence, which means that we have learned a good set of model parameters (weights + biases) from the train data. For the purpose of this tutorial, we'll run training for 10 epochs and stop. An epoch is one full pass over the entire train data. +```python +trainer = gluon.Trainer( + params=net.collect_params(), + optimizer='sgd', + optimizer_params={'learning_rate': 0.04}, +) +``` + +#### Training -We will take following steps for training: +Training the network requires a way to tell how well the network currently fits the training data. +Following common practice in optimization, this quality of fit is expressed through a *loss value* (also referred to as badness-of-fit or data discrepancy), which the algorithm then tries to minimize by adjusting the weights of the model. -- Define [Accuracy evaluation metric](http://mxnet.io/api/python/metric/metric.html#mxnet.metric.Accuracy) over training data. -- Loop over inputs for every epoch. -- Forward input through network to get output. -- Compute loss with output and label inside record scope. -- Backprop gradient inside record scope. -- Update evaluation metric and parameters with gradient descent. +Ideally, in a classification task, we would like to use the prediction inaccuracy, i.e., the fraction of incorrectly classified samples, to guide the training to a lower value. +Unfortunately, inaccuracy is a poor choice for training since it contains almost no information that can be used to update the network parameters (its gradient is zero almost everywhere). +As a better behaved proxy for inaccuracy, the [softmax cross-entropy loss](https://mxnet.incubator.apache.org/api/python/gluon/loss.html#mxnet.gluon.loss.SoftmaxCrossEntropyLoss) is a popular choice. +It has the essential property of being minimal for the correct prediction, but at the same time, it is everywhere differentiable with nonzero gradient. +The [accuracy](https://mxnet.incubator.apache.org/api/python/metric/metric.html#mxnet.metric.Accuracy) metric is still useful for monitoring the training progress, since it is more intuitively interpretable than a loss value. -Loss function takes (output, label) pairs and computes a scalar loss for each sample in the mini-batch. The scalars measure how far each output is from the label. -There are many predefined loss functions in gluon.loss. Here we use -[softmax_cross_entropy_loss](http://mxnet.io/api/python/gluon/gluon.html#mxnet.gluon.loss.softmax_cross_entropy_loss) for digit classification. We will compute loss and do backward propagation inside -training scope which is defined by `autograd.record()`. +**Note:** `SoftmaxCrossEntropyLoss` combines the softmax activation and the cross entropy loss function in one layer, therefore the last layer in our network has no activation function. ```python -%%time -epoch = 10 -# Use Accuracy as the evaluation metric. metric = mx.metric.Accuracy() -softmax_cross_entropy_loss = gluon.loss.SoftmaxCrossEntropyLoss() -for i in range(epoch): - # Reset the train data iterator. - train_data.reset() - # Loop over the train data iterator. - for batch in train_data: - # Splits train data into multiple slices along batch_axis - # and copy each slice into a context. - data = gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0) - # Splits train labels into multiple slices along batch_axis - # and copy each slice into a context. - label = gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0) - outputs = [] - # Inside training scope - with ag.record(): - for x, y in zip(data, label): - z = net(x) - # Computes softmax cross entropy loss. - loss = softmax_cross_entropy_loss(z, y) - # Backpropagate the error for one iteration. - loss.backward() - outputs.append(z) - # Updates internal evaluation - metric.update(label, outputs) - # Make one step of parameter update. Trainer needs to know the - # batch size of data to normalize the gradient by 1/batch_size. - trainer.step(batch.data[0].shape[0]) - # Gets the evaluation result. +loss_function = gluon.loss.SoftmaxCrossEntropyLoss() +``` + +Typically, the training is run until convergence, which means that further iterations will no longer lead to improvements of the loss function, and that the network has probably learned a good set of model parameters from the train data. +For the purpose of this tutorial, we only loop 10 times over the entire dataset. +One such pass over the data is usually called an *epoch*. + +The following steps are taken in each `epoch`: + +- Get a minibatch of `inputs` and `labels` from the `train_loader`. +- Feed the `inputs` to the network, producing `outputs`. +- Compute the minibatch `loss` value by comparing `outputs` to `labels`. +- Use backpropagation to compute the gradients of the loss with respect to each of the network parameters by calling `loss.backward()`. +- Update the parameters of the network according to the optimizer rule with `trainer.step(batch_size=inputs.shape[0])`. +- Print the current accuracy over the training data, i.e., the fraction of correctly classified training examples. + +```python +num_epochs = 10 + +for epoch in range(num_epochs): + for inputs, labels in train_loader: + # Possibly copy inputs and labels to the GPU + inputs = inputs.as_in_context(ctx) + labels = labels.as_in_context(ctx) + + # The forward pass and the loss computation need to be wrapped + # in a `record()` scope to make sure the computational graph is + # recorded in order to automatically compute the gradients + # during the backward pass. + with autograd.record(): + outputs = net(inputs) + loss = loss_function(outputs, labels) + + # Compute gradients by backpropagation and update the evaluation + # metric + loss.backward() + metric.update(labels, outputs) + + # Update the parameters by stepping the trainer; the batch size + # is required to normalize the gradients by `1 / batch_size`. + trainer.step(batch_size=inputs.shape[0]) + + # Print the evaluation metric and reset it for the next epoch name, acc = metric.get() - # Reset evaluation result to initial state. + print('After epoch {}: {} = {}'.format(epoch + 1, name, acc)) metric.reset() - print('training acc at epoch %d: %s=%f'%(i, name, acc)) ``` -#### Prediction +#### Validation + +When the above training has completed, we can evaluate the trained model by comparing predictions from the validation dataset with their respective correct labels. +It is important to notice that the validation data was not used during training, i.e., the network has not seen the images and their true labels yet. +Keeping a part of the data aside for validation is crucial for detecting *overfitting* of a network: If a neural network has enough parameters, it can simply memorize the training data and look up the true label for a given training image. +While this results in 100 % training accuracy, such an overfit model would perform very poorly on new data. +In other words, an overfit model does not generalize to a broader class of inputs than the training set, and such an outcome is almost always undesirable. +Therefore, having a subset of "unseen" data for validation is an important part of good practice in machine learning. -After the above training completes, we can evaluate the trained model by running predictions on validation dataset. Since the dataset also has labels for all test images, we can compute the accuracy metric over validation data as follows: +To validate our model on the validation data, we can run the following snippet of code: ```python -# Use Accuracy as the evaluation metric. metric = mx.metric.Accuracy() -# Reset the validation data iterator. -val_data.reset() -# Loop over the validation data iterator. -for batch in val_data: - # Splits validation data into multiple slices along batch_axis - # and copy each slice into a context. - data = gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0) - # Splits validation label into multiple slices along batch_axis - # and copy each slice into a context. - label = gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0) - outputs = [] - for x in data: - outputs.append(net(x)) - # Updates internal evaluation - metric.update(label, outputs) -print('validation acc: %s=%f'%metric.get()) -assert metric.get()[1] > 0.94 +for inputs, labels in val_loader: + # Possibly copy inputs and labels to the GPU + inputs = inputs.as_in_context(ctx) + labels = labels.as_in_context(ctx) + metric.update(labels, net(inputs)) +print('Validaton: {} = {}'.format(*metric.get())) +assert metric.get()[1] > 0.96 ``` -If everything went well, we should see an accuracy value that is around 0.96, which means that we are able to accurately predict the digit in 96% of test images. This is a pretty good result. But as we will see in the next part of this tutorial, we can do a lot better than that. - -### Convolutional Neural Network +If everything went well, we should see an accuracy value that is around 0.968, which means that we are able to accurately predict the digit in 97 % of test images. +This is a pretty good result, but as we will see in the next part of this tutorial, we can do a lot better than that. -Earlier, we briefly touched on a drawback of MLP when we said we need to discard the input image's original shape and flatten it as a vector before we can feed it as input to the MLP's first fully connected layer. Turns out this is an important issue because we don't take advantage of the fact that pixels in the image have natural spatial correlation along the horizontal and vertical axes. A convolutional neural network (CNN) aims to address this problem by using a more structured weight representation. Instead of flattening the image and doing a simple matrix-matrix multiplication, it employs one or more convolutional layers that each performs a 2-D convolution on the input image. +That said, a single number only conveys very limited information on the performance of our neural network. +It is always a good idea to actually look at the images on which the network performed poorly, and check for clues on how to improve the performance. +We do that with the help of a small function that produces a list of the images which the network got wrong, together with the predicted and true labels. -A single convolution layer consists of one or more filters that each play the role of a feature detector. During training, a CNN learns appropriate representations (parameters) for these filters. Similar to MLP, the output from the convolutional layer is transformed by applying a non-linearity. Besides the convolutional layer, another key aspect of a CNN is the pooling layer. A pooling layer serves to make the CNN translation invariant: a digit remains the same even when it is shifted left/right/up/down by a few pixels. A pooling layer reduces a *n x m* patch into a single value to make the network less sensitive to the spatial location. Pooling layer is always included after each conv (+ activation) layer in the CNN. - -The following source code defines a convolutional neural network architecture called LeNet. LeNet is a popular network known to work well on digit classification tasks. We will use a slightly different version from the original LeNet implementation, replacing the sigmoid activations with tanh activations for the neurons. +```python +def get_mislabeled(loader): + """Return list of ``(input, pred_lbl, true_lbl)`` for mislabeled samples.""" + mislabeled = [] + for inputs, labels in loader: + inputs = inputs.as_in_context(ctx) + labels = labels.as_in_context(ctx) + outputs = net(inputs) + # Predicted label is the index is where the output is maximal + preds = nd.argmax(outputs, axis=1) + for i, p, l in zip(inputs, preds, labels): + p, l = int(p.asscalar()), int(l.asscalar()) + if p != l: + mislabeled.append((i.asnumpy(), p, l)) + return mislabeled +``` -A typical way to write your network is creating a new class inherited from `gluon.Block` -class. We can define the network by composing and inheriting Block class as follows: +We can now get the mislabeled images in the training and validation sets and plot a selection of them: ```python -import mxnet.ndarray as F - -class Net(gluon.Block): - def __init__(self, **kwargs): - super(Net, self).__init__(**kwargs) - with self.name_scope(): - # layers created in name_scope will inherit name space - # from parent layer. - self.conv1 = nn.Conv2D(20, kernel_size=(5,5)) - self.pool1 = nn.MaxPool2D(pool_size=(2,2), strides = (2,2)) - self.conv2 = nn.Conv2D(50, kernel_size=(5,5)) - self.pool2 = nn.MaxPool2D(pool_size=(2,2), strides = (2,2)) - self.fc1 = nn.Dense(500) - self.fc2 = nn.Dense(10) - - def forward(self, x): - x = self.pool1(F.tanh(self.conv1(x))) - x = self.pool2(F.tanh(self.conv2(x))) - # 0 means copy over size from corresponding dimension. - # -1 means infer size from the rest of dimensions. - x = x.reshape((0, -1)) - x = F.tanh(self.fc1(x)) - x = F.tanh(self.fc2(x)) - return x +import numpy as np + +sample_size = 8 +wrong_train = get_mislabeled(train_loader) +wrong_val = get_mislabeled(val_loader) +wrong_train_sample = [wrong_train[i] for i in np.random.randint(0, len(wrong_train), size=sample_size)] +wrong_val_sample = [wrong_val[i] for i in np.random.randint(0, len(wrong_val), size=sample_size)] + +import matplotlib.pyplot as plt + +fig, axs = plt.subplots(ncols=sample_size) +for ax, (img, pred, lbl) in zip(axs, wrong_train_sample): + fig.set_size_inches(18, 4) + fig.suptitle("Sample of wrong predictions in the training set", fontsize=20) + ax.imshow(img[0], cmap="gray") + ax.set_title("Predicted: {}\nActual: {}".format(pred, lbl)) + ax.xaxis.set_visible(False) + ax.yaxis.set_visible(False) + +fig, axs = plt.subplots(ncols=sample_size) +for ax, (img, pred, lbl) in zip(axs, wrong_val_sample): + fig.set_size_inches(18, 4) + fig.suptitle("Sample of wrong predictions in the validation set", fontsize=20) + ax.imshow(img[0], cmap="gray") + ax.set_title("Predicted: {}\nActual: {}".format(pred, lbl)) + ax.xaxis.set_visible(False) + ax.yaxis.set_visible(False) ``` +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/gluon/mnist_wrong_preds_train.png) +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/doc/tutorials/gluon/mnist_wrong_preds_val.png) + +In this case, it is rather obvious that our MLP network is either too simple or has not been trained long enough to perform really great on this dataset, as can be seen from the fact that some of the mislabeled examples are rather "easy" and should not be a challenge for our neural net. +As it turns out, moving to the CNN architecture presented in the following section will give a big performance boost. + +### Convolutional Neural Network (CNN) -We just defined the forward function here, and the backward function to compute gradients -is automatically defined for you using autograd. -We also imported `mxnet.ndarray` package to use activation functions from `ndarray` API. +A fundamental issue with the MLP network is that it requires the inputs to be flattened (in the non-batch axes) before they can be processed by the dense layers. +This means in particular that the spatial structure of an image is largely discarded, and that the values describing it are just treated as a long vector. +The network then has to figure out the neighborhood relations of pixels from scratch by adjusting its weights accordingly, which seems very wasteful. -Now, We will create the network as follows: +A CNN aims to address this problem by using a more structured weight representation. +Instead of connecting all inputs to all outputs, the characteristic [convolution layer](https://mxnet.incubator.apache.org/api/python/gluon/nn.html#mxnet.gluon.nn.Conv2D) only considers a small neighborhood of a pixel to compute the value of the corresponding output pixel. +In particular, the spatial structure of the image is preserved, i.e., one can speak of input and output pixels in the first place. +Only the size of the image may change through convolutions. +[This article](http://deeplearning.net/software/theano/tutorial/conv_arithmetic.html) gives a good and intuitive explanation of convolutions in the context of deep learning. + +The size of the neighborhood that a convolution layer considers for each pixel is usually referred to as *filter size* or *kernel size*. +The array of weights -- which does not depend on the output pixel location, only on the position within such a neighborhood -- is called *filter* or *kernel*. +Typical filter sizes range from *3 x 3* to *13 x 13*, which implies that a convolution layer has *far* fewer parameters than a dense layer. ```python -net = Net() +conv_layer = nn.Conv2D(kernel_size=(3, 3), channels=32, in_channels=16, activation='relu') +print(conv_layer.params) ``` -![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/conv_mnist.png) - -**Figure 3:** First conv + pooling layer in LeNet. +`Parameter conv0_weight (shape=(32, 16, 3, 3), dtype=)` -Now we train LeNet with similar hyper-parameters as before. Note that, if a GPU is available, we recommend using it. This greatly speeds up computation given that LeNet is more complex and compute-intensive than the previous multilayer perceptron. To do so, we only need to change `mx.cpu()` to `mx.gpu()` and MXNet takes care of the rest. Just like before, we'll stop training after 10 epochs. +`Parameter conv0_bias (shape=(32,), dtype=)` -Training and prediction can be done in the similar way as we did for MLP. +Filters can be thought of as little feature detectors: in early layers, they learn to detect small local structures like edges, whereas later layers become sensitive to more and more global structures. +Since images often contain a rich set of such features, it is customary to have each convolution layer employ and learn many different filters in parallel, so as to detect many different image features on their respective scales. +This stacking of filters, which directly translates to a stacking of output images, is referred to as output *channels* of the convolution layer. +Likewise, the input can already have multiple channels. +In the above example, the convolution layer takes an input image with 16 channels and maps it to an image with 32 channels by convolving each of the input channels with a different set of 32 filters and then summing over the 16 input channels. +Therefore, the total number of filter parameters in the convolution layer is `channels * in_channels * prod(kernel_size)`, which amounts to 4608 in the above example. -#### Initialize parameters and optimizer +Another characteristic feature of CNNs is the usage of *pooling*, i.e., summarizing patches to a single number, to shrink the size of an image as it travels through the layers. +This step lowers the computational burden of training the network, but the main motivation for pooling is the assumption that it makes the network less sensitive to small translations, rotations or deformations of the image. +Popular pooling strategies are max-pooling and average-pooling, and they are usually performed after convolution. -We will initialize the network parameters as follows: +The following code defines a CNN architecture called *LeNet*. +The LeNet architecture is a popular network known to work well on digit classification tasks. +We will use a version that differs slightly from the original in the usage of `tanh` activations instead of `sigmoid`. ```python -# set the context on GPU is available otherwise CPU -ctx = [mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()] -net.initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx) -trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.03}) +lenet = nn.HybridSequential(prefix='LeNet_') +with lenet.name_scope(): + lenet.add( + nn.Conv2D(channels=20, kernel_size=(5, 5), activation='tanh'), + nn.MaxPool2D(pool_size=(2, 2), strides=(2, 2)), + nn.Conv2D(channels=50, kernel_size=(5, 5), activation='tanh'), + nn.MaxPool2D(pool_size=(2, 2), strides=(2, 2)), + nn.Flatten(), + nn.Dense(500, activation='tanh'), + nn.Dense(10, activation=None), + ) ``` -#### Training +To get an overview of all intermediate sizes of arrays and the number of parameters in each layer, the `summary()` method can be a great help. +It requires the network parameters to be initialized, and an input array to infer the sizes. ```python -# Use Accuracy as the evaluation metric. -metric = mx.metric.Accuracy() -softmax_cross_entropy_loss = gluon.loss.SoftmaxCrossEntropyLoss() - -for i in range(epoch): - # Reset the train data iterator. - train_data.reset() - # Loop over the train data iterator. - for batch in train_data: - # Splits train data into multiple slices along batch_axis - # and copy each slice into a context. - data = gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0) - # Splits train labels into multiple slices along batch_axis - # and copy each slice into a context. - label = gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0) - outputs = [] - # Inside training scope - with ag.record(): - for x, y in zip(data, label): - z = net(x) - # Computes softmax cross entropy loss. - loss = softmax_cross_entropy_loss(z, y) - # Backpropogate the error for one iteration. - loss.backward() - outputs.append(z) - # Updates internal evaluation - metric.update(label, outputs) - # Make one step of parameter update. Trainer needs to know the - # batch size of data to normalize the gradient by 1/batch_size. - trainer.step(batch.data[0].shape[0]) - # Gets the evaluation result. - name, acc = metric.get() - # Reset evaluation result to initial state. - metric.reset() - print('training acc at epoch %d: %s=%f'%(i, name, acc)) +lenet.initialize(mx.init.Xavier(), ctx=ctx) +lenet.summary(nd.zeros((1, 1, 28, 28), ctx=ctx)) +``` + +``` +Output: + +-------------------------------------------------------------------------------- + Layer (type) Output Shape Param # +================================================================================ + Input (1, 1, 28, 28) 0 + Activation-1 0 + Activation-2 (1, 20, 24, 24) 0 + Conv2D-3 (1, 20, 24, 24) 520 + MaxPool2D-4 (1, 20, 12, 12) 0 + Activation-5 0 + Activation-6 (1, 50, 8, 8) 0 + Conv2D-7 (1, 50, 8, 8) 25050 + MaxPool2D-8 (1, 50, 4, 4) 0 + Flatten-9 (1, 800) 0 + Activation-10 0 + Activation-11 (1, 500) 0 + Dense-12 (1, 500) 400500 + Dense-13 (1, 10) 5010 +================================================================================ +Parameters in forward computation graph, duplicate included + Total params: 431080 + Trainable params: 431080 + Non-trainable params: 0 +Shared params in forward computation graph: 0 +Unique parameters in model: 431080 +-------------------------------------------------------------------------------- ``` -#### Prediction +![png](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/image/conv_mnist.png) + +**Figure 3:** First conv + pooling layer in LeNet. -Finally, we'll use the trained LeNet model to generate predictions for the test data. +Now we train LeNet with similar hyperparameters and procedure as before. +Note that it is advisable to use a GPU if possible, since this model is significantly more computationally demanding to evaluate and train than the previous MLP. ```python -# Use Accuracy as the evaluation metric. +trainer = gluon.Trainer( + params=lenet.collect_params(), + optimizer='sgd', + optimizer_params={'learning_rate': 0.04}, +) metric = mx.metric.Accuracy() -# Reset the validation data iterator. -val_data.reset() -# Loop over the validation data iterator. -for batch in val_data: - # Splits validation data into multiple slices along batch_axis - # and copy each slice into a context. - data = gluon.utils.split_and_load(batch.data[0], ctx_list=ctx, batch_axis=0) - # Splits validation label into multiple slices along batch_axis - # and copy each slice into a context. - label = gluon.utils.split_and_load(batch.label[0], ctx_list=ctx, batch_axis=0) - outputs = [] - for x in data: - outputs.append(net(x)) - # Updates internal evaluation - metric.update(label, outputs) -print('validation acc: %s=%f'%metric.get()) -assert metric.get()[1] > 0.98 +num_epochs = 10 + +for epoch in range(num_epochs): + for inputs, labels in train_loader: + inputs = inputs.as_in_context(ctx) + labels = labels.as_in_context(ctx) + + with autograd.record(): + outputs = lenet(inputs) + loss = loss_function(outputs, labels) + + loss.backward() + metric.update(labels, outputs) + + trainer.step(batch_size=inputs.shape[0]) + + name, acc = metric.get() + print('After epoch {}: {} = {}'.format(epoch + 1, name, acc)) + metric.reset() + +for inputs, labels in val_loader: + inputs = inputs.as_in_context(ctx) + labels = labels.as_in_context(ctx) + metric.update(labels, lenet(inputs)) +print('Validaton: {} = {}'.format(*metric.get())) +assert metric.get()[1] > 0.985 ``` -If all went well, we should see a higher accuracy metric for predictions made using LeNet. With CNN we should be able to correctly predict around 98% of all test images. +If all went well, we should see a higher accuracy metric for predictions made using LeNet. +With this CNN we should be able to correctly predict around 99% of all validation images. ## Summary -In this tutorial, we have learned how to use MXNet to solve a standard computer vision problem: classifying images of hand written digits. You have seen how to quickly and easily build, train and evaluate models such as MLP and CNN with MXNet Gluon package. +In this tutorial, we demonstrated how to use MXNet to solve a standard computer vision problem: classifying images of hand-written digits. +We showed how to quickly build, train and evaluate models such as MLPs and CNNs with the MXNet Gluon package. diff --git a/docs/tutorials/gluon/save_load_params.md b/docs/tutorials/gluon/save_load_params.md index d8eac88d8..ebc8103e7 100644 --- a/docs/tutorials/gluon/save_load_params.md +++ b/docs/tutorials/gluon/save_load_params.md @@ -243,7 +243,7 @@ One of the main reasons to serialize model architecture into a JSON file is to l Serialized Hybrid networks (saved as .JSON and .params file) can be loaded and used inside Python frontend using `gluon.nn.SymbolBlock`. To demonstrate that, let's load the network we serialized above. ```python -deserialized_net = gluon.nn.SymbolBlock.imports("lenet-symbol.json", ['data'], "lenet-0001.params") +deserialized_net = gluon.nn.SymbolBlock.imports("lenet-symbol.json", ['data'], "lenet-0001.params", ctx=ctx) ``` `deserialized_net` now contains the network we deserialized from files. Let's test the deserialized network to make sure it works. diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 07f32b501..9457a409a 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -3,11 +3,13 @@ ```eval_rst .. toctree:: :hidden: + basic/index.md c++/index.md control_flow/index.md embedded/index.md gluon/index.md + java/index.md nlp/index.md onnx/index.md python/index.md @@ -57,6 +59,7 @@ Select API:  * [Logistic Regression](/tutorials/gluon/logistic_regression_explained.html) * [Word-level text generation with RNN, LSTM and GRU](http://gluon.mxnet.io/chapter05_recurrent-neural-networks/rnns-gluon.html) External link * [Visual Question Answering](http://gluon.mxnet.io/chapter08_computer-vision/visual-question-answer.html) External link + * [Image similiarity search with InfoGAN](/tutorials/gluon/info_gan.html) * Practitioner Guides * [Gotchas using NumPy](/tutorials/gluon/gotchas_numpy_in_mxnet.html) * [Multi-GPU training](http://gluon.mxnet.io/chapter07_distributed-learning/multiple-gpus-gluon.html) External link @@ -75,6 +78,7 @@ Select API:  * NDArray * [NDArray API](/tutorials/gluon/ndarray.html) ([Alternative](http://gluon.mxnet.io/chapter01_crashcourse/ndarray.html) External link) * [Advanced NDArray API](/tutorials/basic/ndarray.html) + * [Difference between reshape and transpose](/tutorials/basic/reshape_transpose.html) * [NDArray Indexing](https://mxnet.incubator.apache.org/tutorials/basic/ndarray_indexing.html) * Sparse NDArray * [Sparse Gradient Updates (RowSparseNDArray)](/tutorials/sparse/row_sparse.html) @@ -154,6 +158,13 @@ Select API:  * [MXNet-Scala Examples](https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/src/main/scala/org/apache/mxnetexamples)
      +## Java Tutorials +* Getting Started + * [Developer Environment Setup on IntelliJ IDE](/tutorials/java/mxnet_java_on_intellij.html) +* [Multi Object Detection using pre-trained Single Shot Detector (SSD) Model](/tutorials/java/ssd_inference.html) +* [MXNet-Java Examples](https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer) +
      + ## C++ Tutorials * Models diff --git a/docs/tutorials/java/index.md b/docs/tutorials/java/index.md new file mode 100644 index 000000000..87d728944 --- /dev/null +++ b/docs/tutorials/java/index.md @@ -0,0 +1,8 @@ +# Tutorials + +```eval_rst +.. toctree:: + :glob: + + * +``` diff --git a/docs/tutorials/java/mxnet_java_on_intellij.md b/docs/tutorials/java/mxnet_java_on_intellij.md new file mode 100644 index 000000000..ef2c009f6 --- /dev/null +++ b/docs/tutorials/java/mxnet_java_on_intellij.md @@ -0,0 +1,180 @@ +# Run MXNet Java Examples Using the IntelliJ IDE (macOS) + +This tutorial guides you through setting up a simple Java project in IntelliJ IDE on macOS and demonstrates usage of the MXNet Java APIs. + +## Prerequisites: +To use this tutorial you need the following pre-requisites: + +- [Java 8 JDK](http://www.oracle.com/technetwork/java/javase/downloads/index.html) +- [Maven](https://maven.apache.org/install.html) +- [OpenCV](https://opencv.org/) +- [IntelliJ IDEA](https://www.jetbrains.com/idea/) (One can download the community edition from [here](https://www.jetbrains.com/idea/download)) + +### MacOS Prerequisites + +Run the following commands to install the prerequisites on MacOS. +``` +/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" +brew update +brew tap caskroom/versions +brew cask install java8 +brew install maven +brew install opencv +``` + +### Ubuntu Prerequisites + +Run the following commands to install the prerequisites on Ubuntu. + +```bash +wget https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_core.sh +sudo ./ubuntu_core.sh +wget https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_scala.sh +sudo ./ubuntu_scala.sh +``` + +Note : You might need to run `chmod u+x ubuntu_core.sh` and `chmod u+x ubuntu_scala` before running the scripts. + +The `ubuntu_scala.sh` installs the common dependencies required for both MXNet Scala and MXNet Java packages. + +## Set Up Your Project + +**Step 1.** Install and setup [IntelliJ IDEA](https://www.jetbrains.com/idea/) + +**Step 2.** Create a new Project: + +![intellij welcome](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-welcome.png) + +From the IntelliJ welcome screen, select "Create New Project". + +Choose the Maven project type. + +Select the checkbox for `Create from archetype`, then choose `org.apache.maven.archetypes:maven-archetype-quickstart` from the list below. More on this can be found on a Maven tutorial : [Maven in 5 Minutes](https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html). + +![maven project type - archetype](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/project-archetype.png) + +click `Next`. + +![project metadata](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/intellij-project-metadata.png) + +Set the project's metadata. For this tutorial, use the following: + +**GroupId** +``` +mxnet +``` +**ArtifactId** +``` +javaMXNet +``` +**Version** +``` +1.0-SNAPSHOT +``` + +![project properties](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/intellij-project-properties.png) + +Review the project's properties. The settings can be left as their default. + +![project location](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/intellij-project-location.png) + +Set the project's location. The rest of the settings can be left as their default. + +![project 1](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/intellij-project-pom.png) + +After clicking Finish, you will be presented with the project's first view. +The project's `pom.xml` will be open for editing. + +**Step 3.** The Java packages are currently available as nightly builds on Maven. Add the following Maven repository to your `pom.xml` to fetch the Java packages : + +```html + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + +``` + +Also, add this under the `dependencies` tag : + +```html + + org.apache.mxnet + mxnet-full_2.11-osx-x86_64-cpu + 1.4.0-SNAPSHOT + +``` +The official Java Packages will be released with the release of MXNet 1.4 and will be available on [MXNet Maven package repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.mxnet%22). + +Note : +- Change the osx-x86_64 to linux-x86_64 if your platform is linux. +- Change cpu into gpu if you have a gpu backed machine and want to use gpu. + + +**Step 4.** Import dependencies with Maven: + + - Note the prompt in the lower right corner that states "Maven projects need to be imported". If this is not visible, click on the little greed balloon that appears in the lower right corner. + +![import_dependencies](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tutorials/java/project-import-changes.png) + +Click "Import Changes" in this prompt. + +**Step 5.** Build the project: +- To build the project, from the menu choose Build, and then choose Build Project. + +**Step 6.** Navigate to the App.java class in the project and paste the code in `main` method from HelloWorld.java from [Java Demo project](https://github.com/apache/incubator-mxnet/blob/java-api/scala-package/mxnet-demo/java-demo/src/main/java/sample/HelloWorld.java) on MXNet repository, overwriting the original hello world code. +You can also grab the entire [Java Demo project](https://github.com/apache/incubator-mxnet/tree/java-api/scala-package/mxnet-demo/java-demo) and run it by following the instructions on the [README](https://github.com/apache/incubator-mxnet/blob/java-api/scala-package/mxnet-demo/java-demo/README.md). + +**Step 7.** Now run the App.java. + +The result should be something similar to this: + +``` +Hello World! +(1,2) +Process finished with exit code 0 +``` + +### Troubleshooting + +If you get an error, check the dependencies at the beginning of this tutorial. For example, you might see the following in the middle of the error messages, where `x.x` would the version it's looking for. + +``` +... +Library not loaded: /usr/local/opt/opencv/lib/libopencv_calib3d.x.x.dylib +... +``` + +This can be resolved be installing OpenCV. + +### Command Line Build Option + +- You can also compile the project by using the following command at the command line. Change directories to this project's root folder then run the following: + +```bash +mvn clean install dependency:copy-dependencies +``` +If the command succeeds, you should see a lot of info and some warning messages, followed by: + +```bash +[INFO] ------------------------------------------------------------------------ +[INFO] BUILD SUCCESS +[INFO] ------------------------------------------------------------------------ +[INFO] Total time: 3.475 s +[INFO] Finished at: 2018-11-08T05:06:31-08:00 +[INFO] ------------------------------------------------------------------------ +``` +The build generates a new jar file in the `target` folder called `javaMXNet-1.0-SNAPSHOT.jar`. + +To run the App.java use the following command from the project's root folder and you should see the same output as we got when the project was run from IntelliJ. +```bash +java -cp "target/javaMXNet-1.0-SNAPSHOT.jar:target/dependency/*" mxnet.App +``` + +## Next Steps +For more information about MXNet Java resources, see the following: + +* [Java Inference API](/api/java/index.html) +* [Java Inference Examples](https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer) +* [MXNet Tutorials Index](http://mxnet.io/tutorials/index.html) diff --git a/docs/tutorials/java/ssd_inference.md b/docs/tutorials/java/ssd_inference.md new file mode 100644 index 000000000..3a20329f9 --- /dev/null +++ b/docs/tutorials/java/ssd_inference.md @@ -0,0 +1,186 @@ +# Multi Object Detection using pre-trained SSD Model via Java Inference APIs + +This tutorial shows how to use MXNet Java Inference APIs to run inference on a pre-trained Single Shot Detector (SSD) Model. + +The SSD model is trained on the Pascal VOC 2012 dataset. The network is a SSD model built on Resnet50 as the base network to extract image features. The model is trained to detect the following entities (classes): ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']. For more details about the model, you can refer to the [MXNet SSD example](https://github.com/apache/incubator-mxnet/tree/master/example/ssd). + +## Prerequisites + +To complete this tutorial, you need the following: +* [MXNet Java Setup on IntelliJ IDEA](/java/mxnet_java_on_intellij.html) (Optional) +* [wget](https://www.gnu.org/software/wget/) To download model artifacts +* SSD Model artifacts + * Use the following script to get the SSD Model files : +```bash +data_path=/tmp/resnet50_ssd +mkdir -p "$data_path" +wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $data_path +wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $data_path +wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -P $data_path +``` +* Test images : A few sample images to run inference on. + * Use the following script to download sample images : +```bash +image_path=/tmp/resnet50_ssd/images +mkdir -p "$image_path" +cd $image_path +wget https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -O dog.jpg +wget https://cloud.githubusercontent.com/assets/3307514/20012563/cbb41382-a27d-11e6-92a9-18dab4fd1ad3.jpg -O person.jpg +``` + +Alternately, you can get the entire SSD Model artifacts + images in one single script from the MXNet Repository by running [get_ssd_data.sh script](https://github.com/apache/incubator-mxnet/blob/master/scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh) + +## Time to code! +1\. Following the [MXNet Java Setup on IntelliJ IDEA](/java/mxnet_java_on_intellij.html) tutorial, in the same project `JavaMXNet`, create a new empty class called : `ObjectDetectionTutorial.java`. + +2\. In the `main` function of `ObjectDetectionTutorial.java` define the downloaded model path and the image data paths. This is the same path where we downloaded the model artifacts and images in a previous step. + +```java +String modelPathPrefix = "/tmp/resnet50_ssd/resnet50_ssd_model"; +String inputImagePath = "/tmp/resnet50_ssd/images/dog.jpg"; +``` + +3\. We can run the inference code in this example on either CPU or GPU (if you have a GPU backed machine) by choosing the appropriate context. + +```java + +List context = getContext(); +... + +private static List getContext() { +List ctx = new ArrayList<>(); +ctx.add(Context.cpu()); // Choosing CPU Context here + +return ctx; +} +``` + +4\. To provide an input to the model, define the input shape to the model and the Input Data Descriptor (DataDesc) as shown below : + +```java +Shape inputShape = new Shape(new int[] {1, 3, 512, 512}); +List inputDescriptors = new ArrayList(); +inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); +``` + +The input shape can be interpreted as follows : The input has a batch size of 1, with 3 RGB channels in the image, and the height and width of the image is 512 each. + +5\. To run an actual inference on the given image, add the following lines to the `ObjectDetectionTutorial.java` class : + +```java +BufferedImage img = ObjectDetector.loadImageFromFile(inputImagePath); +ObjectDetector objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); +List> output = objDet.imageObjectDetect(img, 3); // Top 3 objects detected will be returned +``` + +6\. Let's piece all of the above steps together by showing the final contents of the `ObjectDetectionTutorial.java`. + +```java +package mxnet; + +import org.apache.mxnet.infer.javaapi.ObjectDetector; +import org.apache.mxnet.infer.javaapi.ObjectDetectorOutput; +import org.apache.mxnet.javaapi.Context; +import org.apache.mxnet.javaapi.DType; +import org.apache.mxnet.javaapi.DataDesc; +import org.apache.mxnet.javaapi.Shape; + +import java.awt.image.BufferedImage; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +public class ObjectDetectionTutorial { + + public static void main(String[] args) { + + String modelPathPrefix = "/tmp/resnet50_ssd/resnet50_ssd_model"; + + String inputImagePath = "/tmp/resnet50_ssd/images/dog.jpg"; + + List context = getContext(); + + Shape inputShape = new Shape(new int[] {1, 3, 512, 512}); + + List inputDescriptors = new ArrayList(); + inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + + BufferedImage img = ObjectDetector.loadImageFromFile(inputImagePath); + ObjectDetector objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); + List> output = objDet.imageObjectDetect(img, 3); + + printOutput(output, inputShape); + } + + + private static List getContext() { + List ctx = new ArrayList<>(); + ctx.add(Context.cpu()); + + return ctx; + } + + private static void printOutput(List> output, Shape inputShape) { + + StringBuilder outputStr = new StringBuilder(); + + int width = inputShape.get(3); + int height = inputShape.get(2); + + for (List ele : output) { + for (ObjectDetectorOutput i : ele) { + outputStr.append("Class: " + i.getClassName() + "\n"); + outputStr.append("Probabilties: " + i.getProbability() + "\n"); + + List coord = Arrays.asList(i.getXMin() * width, + i.getXMax() * height, i.getYMin() * width, i.getYMax() * height); + StringBuilder sb = new StringBuilder(); + for (float c: coord) { + sb.append(", ").append(c); + } + outputStr.append("Coord:" + sb.substring(2)+ "\n"); + } + } + System.out.println(outputStr); + + } +} +``` + +7\. To compile and run this code, change directories to this project's root folder, then run the following: +```bash +mvn clean install dependency:copy-dependencies +``` + +The build generates a new jar file in the `target` folder called `javaMXNet-1.0-SNAPSHOT.jar`. + +To run the ObjectDetectionTutorial.java use the following command from the project's root folder. +```bash +java -cp "target/javaMXNet-1.0-SNAPSHOT.jar:target/dependency/*" mxnet.ObjectDetectionTutorial +``` + +You should see a similar output being generated for the dog image that we used: +```bash +Class: car +Probabilties: 0.99847263 +Coord:312.21335, 72.02908, 456.01443, 150.66176 +Class: bicycle +Probabilties: 0.9047381 +Coord:155.9581, 149.96365, 383.83694, 418.94516 +Class: dog +Probabilties: 0.82268167 +Coord:83.82356, 179.14001, 206.63783, 476.78754 +``` + +![dog_1](https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg) + +The results returned by the inference call translate into the regions in the image where the model detected objects. + +![dog_2](https://cloud.githubusercontent.com/assets/3307514/19171063/91ec2792-8be0-11e6-983c-773bd6868fa8.png) + +## Next Steps +For more information about MXNet Java resources, see the following: + +* [Java Inference API](/api/java/index.html) +* [Java Inference Examples](https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer) +* [MXNet Tutorials Index](/tutorials/index.html) diff --git a/docs/tutorials/onnx/export_mxnet_to_onnx.md b/docs/tutorials/onnx/export_mxnet_to_onnx.md index b838bae94..3f925c7b5 100644 --- a/docs/tutorials/onnx/export_mxnet_to_onnx.md +++ b/docs/tutorials/onnx/export_mxnet_to_onnx.md @@ -49,7 +49,7 @@ Let us describe the MXNet's `export_model` API. help(onnx_mxnet.export_model) ``` -```python +``` Help on function export_model in module mxnet.contrib.onnx.mx2onnx.export_model: export_model(sym, params, input_shape, input_type=, onnx_file_path=u'model.onnx', verbose=False) diff --git a/docs/tutorials/onnx/fine_tuning_gluon.md b/docs/tutorials/onnx/fine_tuning_gluon.md index 07d8bdf0a..750a67572 100644 --- a/docs/tutorials/onnx/fine_tuning_gluon.md +++ b/docs/tutorials/onnx/fine_tuning_gluon.md @@ -23,19 +23,23 @@ We recommend that you have first followed this tutorial: ```python -import numpy as np +import json +import logging +import multiprocessing +import os +import tarfile + +logging.basicConfig(level=logging.INFO) + +import matplotlib.pyplot as plt import mxnet as mx from mxnet import gluon, nd, autograd from mxnet.gluon.data.vision.datasets import ImageFolderDataset from mxnet.gluon.data import DataLoader import mxnet.contrib.onnx as onnx_mxnet +import numpy as np + %matplotlib inline -import matplotlib.pyplot as plt -import tarfile, os -import json -import multiprocessing -import logging -logging.basicConfig(level=logging.INFO) ``` @@ -122,7 +126,7 @@ We need to transform the images to a format accepted by the network EDGE = 224 SIZE = (EDGE, EDGE) BATCH_SIZE = 32 -NUM_WORKERS = multiprocessing.cpu_count() +NUM_WORKERS = 6 ``` We transform the dataset images using the following operations: @@ -152,18 +156,18 @@ ____image4 ```python -dataset_train = ImageFolderDataset(root=training_path, transform=transform) -dataset_test = ImageFolderDataset(root=testing_path, transform=transform) +dataset_train = ImageFolderDataset(root=training_path) +dataset_test = ImageFolderDataset(root=testing_path) ``` -We use num_workers=Number of CPU cores, which means the dataloading and pre-processing is going to be distributed across multiple processes. This will help preventing our GPU from starving and waiting for the data to be copied across +We use several worker processes, which means the dataloading and pre-processing is going to be distributed across multiple processes. This will help preventing our GPU from starving and waiting for the data to be copied across ```python -dataloader_train = DataLoader(dataset_train, batch_size=BATCH_SIZE, last_batch='discard', +dataloader_train = DataLoader(dataset_train.transform(transform, lazy=False), batch_size=BATCH_SIZE, last_batch='rollover', shuffle=True, num_workers=NUM_WORKERS) -dataloader_test = DataLoader(dataset_test, batch_size=BATCH_SIZE, last_batch='discard', - shuffle=True, num_workers=NUM_WORKERS) +dataloader_test = DataLoader(dataset_test.transform(transform, lazy=False), batch_size=BATCH_SIZE, last_batch='rollover', + shuffle=False, num_workers=NUM_WORKERS) print("Train dataset: {} images, Test dataset: {} images".format(len(dataset_train), len(dataset_test))) ``` @@ -183,7 +187,7 @@ Let's plot the 1000th image to test the dataset ```python N = 1000 -plt.imshow(np.transpose(dataset_train[N][0].asnumpy(),(1,2,0))) +plt.imshow((transform(dataset_train[N][0], 0)[0].asnumpy().transpose((1,2,0)))) plt.axis('off') print(categories[dataset_train[N][1]]) ``` @@ -251,7 +255,7 @@ We pick a context, fine-tuning on CPU will be **WAY** slower. ```python -ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu() +ctx = mx.gpu() if mx.context.num_gpus() > 0 else mx.cpu() ``` We create a symbol block that is going to hold all our pre-trained layers, and assign the weights of the different pre-trained layers to the newly created SymbolBlock @@ -282,8 +286,9 @@ We add the SymbolBlock and the new dense layer to a HybridSequential network ```python net = gluon.nn.HybridSequential() -net.add(pre_trained) -net.add(dense_layer) +with net.name_scope(): + net.add(pre_trained) + net.add(dense_layer) ``` ### Loss @@ -321,7 +326,7 @@ We measure the accuracy in a non-blocking way, using `nd.array` to take care of ```python def evaluate_accuracy_gluon(data_iterator, net): - num_instance = nd.zeros(1, ctx=ctx) + num_instance = 0 sum_metric = nd.zeros(1,ctx=ctx, dtype=np.int32) for i, (data, label) in enumerate(data_iterator): data = data.astype(np.float32).as_in_context(ctx) @@ -330,7 +335,7 @@ We measure the accuracy in a non-blocking way, using `nd.array` to take care of prediction = nd.argmax(output, axis=1).astype(np.int32) num_instance += len(prediction) sum_metric += (prediction==label).sum() - accuracy = (sum_metric.astype(np.float32)/num_instance.astype(np.float32)) + accuracy = (sum_metric.astype(np.float32)/num_instance) return accuracy.asscalar() ``` diff --git a/docs/tutorials/python/linear-regression.md b/docs/tutorials/python/linear-regression.md index f96568440..fd336ad2a 100644 --- a/docs/tutorials/python/linear-regression.md +++ b/docs/tutorials/python/linear-regression.md @@ -147,7 +147,7 @@ model = mx.mod.Module( We can visualize the network we created by plotting it: ```python -mx.viz.plot_network(symbol=lro) +mx.viz.plot_network(symbol=lro, node_attrs={"shape":"oval","fixedsize":"false"}) ``` ## Training the model diff --git a/docs/tutorials/python/predict_image.md b/docs/tutorials/python/predict_image.md index a9a0d2901..8be98d991 100644 --- a/docs/tutorials/python/predict_image.md +++ b/docs/tutorials/python/predict_image.md @@ -69,6 +69,7 @@ def get_image(url, show=False): img = mx.image.imresize(img, 224, 224) # resize img = img.transpose((2, 0, 1)) # Channel first img = img.expand_dims(axis=0) # batchify + img = img.astype('float32') # for gpu context return img def predict(url): diff --git a/docs/tutorials/python/profiler.md b/docs/tutorials/python/profiler.md index d99bb19ee..7dcda10f1 100644 --- a/docs/tutorials/python/profiler.md +++ b/docs/tutorials/python/profiler.md @@ -94,10 +94,10 @@ Let's define a method that will run one training iteration given data and label. ```python # Use GPU if available -try: - mx.test_utils.list_gpus(); ctx = mx.gpu() -except: - ctx = mx.cpu() +if len(mx.test_utils.list_gpus())!=0: + ctx=mx.gpu() +else: + ctx=mx.cpu() # Initialize the parameters with random weights net.collect_params().initialize(mx.init.Xavier(), ctx=ctx) diff --git a/docs/tutorials/r/fiveMinutesNeuralNetwork.md b/docs/tutorials/r/fiveMinutesNeuralNetwork.md index a2ce5ecd3..6d79cd288 100644 --- a/docs/tutorials/r/fiveMinutesNeuralNetwork.md +++ b/docs/tutorials/r/fiveMinutesNeuralNetwork.md @@ -1,7 +1,7 @@ Develop a Neural Network with MXNet in Five Minutes ============================================= -This tutorial is designed for new users of the `mxnet` package for R. It shows how to construct a neural network to do regression in 5 minutes. It shows how to perform classification and regression tasks, respectively. The data we use is in the `mlbench` package. Instructions to install R and MXNet's R package in different environments can be found [here](http://mxnet.incubator.apache.org/install/index.html?platform=Linux&language=R&processor=CPU). +This tutorial is designed for new users of the `mxnet` package for R. It shows how to construct a neural network to do regression in 5 minutes. It shows how to perform classification and regression tasks, respectively. The data we use is in the `mlbench` package. Instructions to install R and MXNet's R package in different environments can be found [here](http://mxnet.incubator.apache.org/install/index.html?platform=Linux&language=R&processor=CPU). ## Classification @@ -88,7 +88,7 @@ Note that `mx.set.seed` controls the random process in `mxnet`. You can see the To get an idea of what is happening, view the computation graph from R: - ```{r} + ```r graph.viz(model$symbol) ``` diff --git a/docs/tutorials/scala/char_lstm.md b/docs/tutorials/scala/char_lstm.md index e5f071b19..4d6a5aee9 100644 --- a/docs/tutorials/scala/char_lstm.md +++ b/docs/tutorials/scala/char_lstm.md @@ -129,7 +129,7 @@ To prepare the data: ```scala scala> // Build a vocabulary of what char we have in the content scala> def buildVocab(path: String): Map[String, Int] = { - val content = readContent(dataPath).split("\n") + val content = readContent(path).split("\n") var idx = 1 // 0 is left for zero padding var theVocab = Map[String, Int]() for (line <- content) { diff --git a/docs/tutorials/scala/mxnet_scala_on_intellij.md b/docs/tutorials/scala/mxnet_scala_on_intellij.md index 497b1cd01..a0bf24e34 100644 --- a/docs/tutorials/scala/mxnet_scala_on_intellij.md +++ b/docs/tutorials/scala/mxnet_scala_on_intellij.md @@ -73,7 +73,6 @@ The configuration you should update is in the pom file's dependency for MXNet: org.apache.mxnet mxnet-full_2.11-osx-x86_64-cpu - 1.2.0 ``` @@ -158,7 +157,7 @@ The project's `pom.xml` will be open for editing. **Step 3.** Replace the pom file's content with the following code. Changes include: - Project properties: `scala.version`, upgrading from `2.11.5` to `2.11.8` - - Project dependencies: adding the MXNet package from Maven and updating the dependency for JUnitRunner (specs2-junit_) + - Project dependencies: adding the MXNet package from Maven and updating the dependency for JUnitRunner (specs2-junit_) and logging - Build options: removing '-make:transitive' @@ -204,19 +203,25 @@ The project's `pom.xml` will be open for editing. UTF-8 2.11.8 2.11 + 1.7.7 + osx-x86_64-cpu org.apache.mxnet mxnet-full_2.11-osx-x86_64-cpu - 1.2.0 org.scala-lang scala-library ${scala.version} + + args4j + args4j + 2.0.29 + @@ -237,6 +242,18 @@ The project's `pom.xml` will be open for editing. 2.2.4 test + + + + org.slf4j + slf4j-api + ${slf4jVersion} + + + org.slf4j + slf4j-log4j12 + ${slf4jVersion} + @@ -292,11 +309,24 @@ The project's `pom.xml` will be open for editing. Click "Import Changes" in this prompt. -**Step 5.** Build the project: +**Step 5.** Setup log4j configuration + +Create a folder `src/main/resources` and a new file in it `src/main/resources/log4j.properties` with the contents: + +``` +log4j.rootLogger = info, stdout + +log4j.appender.stdout = org.apache.log4j.ConsoleAppender +log4j.appender.stdout.Target = System.out +log4j.appender.stdout.layout = org.apache.log4j.PatternLayout +log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n +``` + +**Step 6.** Build the project: - To build the project, from the menu choose Build, and then choose Build Project. -**Step 6.** Run the Hello World App: +**Step 7.** Run the Hello World App: ![hello world app](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-project-hello-world-app.png) @@ -306,7 +336,7 @@ Navigate to the App included with the project. Run the App by clicking the green arrow, and verify the Hello World output -**Step 7.** Run Sample MXNet Code in the App: +**Step 8.** Run Sample MXNet Code in the App: ![run hello mxnet](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/scala/intellij-project-hello-mxnet.png) @@ -347,6 +377,28 @@ Library not loaded: /usr/local/opt/opencv/lib/libopencv_calib3d.x.x.dylib This can be resolved be installing OpenCV. +### Using MXNet from source + +If you chose to "Build from Source" when following the [install instructions](https://mxnet.incubator.apache.org/install/index.html) (or the detailed [build from source instructions](https://mxnet.incubator.apache.org/install/build_from_source.html#installing-mxnet-language-bindings)), you can use your custom build instead of the build from maven. Use your build by editing the `pom.xml` file and replacing the `org.apache.mxnet` dependency with the following: + +``` + org.apache.mxnet + mxnet-core_${scala.version}-${platform}-sources + system + /PathToMXNetSource/incubator-mxnet/scala-package/assembly/osx-x86_64-cpu/target/mxnet-full_${scala.version}-osx-x86_64-cpu-1.5.0-SNAPSHOT-sources.jar + + + + org.apache.mxnet + mxnet-full_${scala.version}-${platform} + system + /PathToMXNetSource/incubator-mxnet/scala-package/assembly/osx-x86_64-cpu/target/mxnet-full_${scala.version}-osx-x86_64-cpu-1.5.0-SNAPSHOT.jar + +``` + +Note that you have to edit both of the `systemPath` properties to point to your generated jar files. + + ### Command Line Build Option - You can also compile the project by using the following command at the command line. Change directories to this project's folder then run the following: diff --git a/docs/tutorials/sparse/row_sparse.md b/docs/tutorials/sparse/row_sparse.md index 27cc0d3d9..46a5edad0 100644 --- a/docs/tutorials/sparse/row_sparse.md +++ b/docs/tutorials/sparse/row_sparse.md @@ -459,7 +459,7 @@ Note that warning messages will be printed when such a storage fallback event ha ## Sparse Optimizers -In MXNet, sparse gradient updates are applied when weight, state and gradient are all in `row_sparse` storage. +In MXNet, sparse gradient updates are applied when gradient is in `row_sparse` storage and the optimizer is created with `lazy_update=True`. The sparse optimizers only update the row slices of the weight and the states whose indices appear in `gradient.indices`. For example, the default update rule for SGD optimizer is: diff --git a/docs/tutorials/unsupervised_learning/gan.md b/docs/tutorials/unsupervised_learning/gan.md index 1556bf609..0efdc5565 100644 --- a/docs/tutorials/unsupervised_learning/gan.md +++ b/docs/tutorials/unsupervised_learning/gan.md @@ -63,7 +63,7 @@ The MNIST dataset contains 70,000 images of handwritten digits. Each image is 28 ### 1. Preparing the MNSIT dataset -Let us start by preparing the handwritten digits from the MNIST dataset. +Let us start by preparing the handwritten digits from the MNIST dataset. ```python import mxnet as mx import numpy as np @@ -75,7 +75,7 @@ mnist_test = mx.gluon.data.vision.datasets.MNIST(train=False) ```python # The downloaded data is of type `Dataset` which are # Well suited to work with the new Gluon interface but less -# With the older symbol API, used in this tutorial. +# With the older symbol API, used in this tutorial. # Therefore we convert them to numpy array first X = np.zeros((70000, 28, 28)) for i, (data, label) in enumerate(mnist_train): @@ -394,7 +394,8 @@ As a result, we have created two neural nets: a Generator, which is able to crea Along the way, we have learned how to do the image manipulation and visualization that is associated with the training of deep neural nets. We have also learned how to use MXNet's Module APIs to perform advanced model training functionality to fit the model. ## Acknowledgements -This tutorial is based on [MXNet DCGAN codebase](https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dcgan.py), + +This tutorial is based on [MXNet DCGAN codebase](https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py), [The original paper on GANs](https://arxiv.org/abs/1406.2661), as well as [this paper on deep convolutional GANs](https://arxiv.org/abs/1511.06434). - \ No newline at end of file + diff --git a/docs/tutorials/vision/cnn_visualization.md b/docs/tutorials/vision/cnn_visualization.md index 940c261ef..5ded6f158 100644 --- a/docs/tutorials/vision/cnn_visualization.md +++ b/docs/tutorials/vision/cnn_visualization.md @@ -1,16 +1,21 @@ # Visualizing Decisions of Convolutional Neural Networks -Convolutional Neural Networks have made a lot of progress in Computer Vision. Their accuracy is as good as humans in some tasks. However it remains hard to explain the predictions of convolutional neural networks, as they lack the interpretability offered by other models, for example decision trees. +Convolutional Neural Networks have made a lot of progress in Computer Vision. Their accuracy is as good as humans in some tasks. However, it remains difficult to explain the predictions of convolutional neural networks, as they lack the interpretability offered by other models such as decision trees. -It is often helpful to be able to explain why a model made the prediction it made. For example when a model misclassifies an image, it is hard to say why without visualizing the network's decision. +It is often helpful to be able to explain why a model made the prediction it made. For example, when a model misclassifies an image, without visualizing the network's decision, it is hard to say why the misclassification was made. Explaining the misclassification of volcano as spider -Visualizations also help build confidence about the predictions of a model. For example, even if a model correctly predicts birds as birds, we would want to confirm that the model bases its decision on the features of bird and not on the features of some other object that might occur together with birds in the dataset (like leaves). +Visualizations can also build confidence about the predictions of a model. For example, even if a model correctly predicts birds as birds, we would want to confirm that the model bases its decision on the features of bird and not on the features of some other object that might occur together with birds in the dataset (like leaves). -In this tutorial, we show how to visualize the predictions made by convolutional neural networks using [Gradient-weighted Class Activation Mapping](https://arxiv.org/abs/1610.02391). Unlike many other visualization methods, Grad-CAM can be used on a wide variety of CNN model families - CNNs with fully connected layers, CNNs used for structural outputs (e.g. captioning), CNNs used in tasks with multi-model input (e.g. VQA) or reinforcement learning without architectural changes or re-training. +In this tutorial we show how to visualize the predictions made by convolutional neural networks using [Gradient-weighted Class Activation Mapping](https://arxiv.org/abs/1610.02391). Unlike many other visualization methods, Grad-CAM can be used on a wide variety of CNN model families - CNNs with fully connected layers, CNNs used for structural outputs (e.g. captioning), CNNs used in tasks with multi-model input (e.g. VQA) or reinforcement learning without architectural changes or re-training. -In the rest of this notebook, we will explain how to visualize predictions made by [VGG-16](https://arxiv.org/abs/1409.1556). We begin by importing the required dependencies. `gradcam` module contains the implementation of visualization techniques used in this notebook. +In the rest of this notebook, we will explain how to visualize predictions made by [VGG-16](https://arxiv.org/abs/1409.1556). We begin by importing the required dependencies. + +## Prerequesites +* OpenCV is required by `gradcam` (below) and can be installed with pip using `pip opencv-python`. + +* the `gradcam` module contains the implementation of visualization techniques used in this notebook. `gradcam` can be installed to a temporary directory by executing the following code block. ```python from __future__ import print_function @@ -22,7 +27,7 @@ from matplotlib import pyplot as plt import numpy as np gradcam_file = "gradcam.py" -base_url = "https://raw.githubusercontent.com/indhub/mxnet/cnnviz/example/cnn_visualization/{}?raw=true" +base_url = "https://raw.githubusercontent.com/apache/incubator-mxnet/master/docs/tutorial_utils/vision/cnn_visualization/{}?raw=true" mx.test_utils.download(base_url.format(gradcam_file), fname=gradcam_file) import gradcam ``` @@ -151,7 +156,8 @@ def show_images(pred_str, images): for i in range(num_images): fig.add_subplot(rows, cols, i+1) plt.xlabel(titles[i]) - plt.imshow(images[i], cmap='gray' if i==num_images-1 else None) + img = images[i].astype(np.uint8) + plt.imshow(img, cmap='gray' if i==num_images-1 else None) plt.show() ``` @@ -181,6 +187,7 @@ Next, we'll write a method to get an image, preprocess it, predict category and 2. **Guided Grad-CAM:** Guided Grad-CAM shows which exact pixels contributed the most to the CNN's decision. 3. **Saliency map:** Saliency map is a monochrome image showing which pixels contributed the most to the CNN's decision. Sometimes, it is easier to see the areas in the image that most influence the output in a monochrome image than in a color image. + ```python def visualize(net, img_path, conv_layer_name): orig_img = mx.img.imread(img_path) diff --git a/example/MXNetTutorialTemplate.ipynb b/example/MXNetTutorialTemplate.ipynb index 851a87f18..0b0b19381 100644 --- a/example/MXNetTutorialTemplate.ipynb +++ b/example/MXNetTutorialTemplate.ipynb @@ -13,7 +13,7 @@ "source": [ "A brief introduction to the tutorial that describes:\n", "\n", - "- The problem that that the tutorial addresses\n", + "- The problem that the tutorial addresses\n", "- Who the intended audience is\n", "- The expected experience level of that audience with a concept or tool \n", "- Which environment/language it runs in \n", @@ -239,7 +239,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If appropriate, summarize the tasks required to create the model, defining and explaining key concepts." + "If appropriate, summarize the tasks required to fit the model, defining and explaining key concepts." ] }, { @@ -342,7 +342,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To *fperform the task*, *provide explanation here.*" + "To *perform the task*, *provide explanation here.*" ] }, { diff --git a/example/adversary/README.md b/example/adversary/README.md index 51d295d30..5d5b44fb9 100644 --- a/example/adversary/README.md +++ b/example/adversary/README.md @@ -1,7 +1,7 @@ # Adversarial examples This demonstrates the concept of "adversarial examples" from [1] showing how to fool a well-trained CNN. -The surprising idea is that one can easily generate examples which the CNN will consistently -make the wrong prediction for that a human can easily tell are correct. +Adversarial examples are samples where the input has been manipulated to confuse a model (i.e. confident in an incorrect prediction) but where the correct answer still appears obvious to a human. +This method for generating adversarial examples uses the gradient of the loss with respect to the input to craft the adversarial examples. [1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." [arXiv preprint arXiv:1412.6572 (2014)](https://arxiv.org/abs/1412.6572) diff --git a/example/adversary/adversary_generation.ipynb b/example/adversary/adversary_generation.ipynb index b8804bd81..0b4536650 100644 --- a/example/adversary/adversary_generation.ipynb +++ b/example/adversary/adversary_generation.ipynb @@ -6,8 +6,7 @@ "source": [ "# Fast Sign Adversary Generation Example\n", "\n", - "This notebook demos find adversary example by using symbolic API and integration with Numpy\n", - "Reference: \n", + "This notebook demos finds adversary examples using MXNet Gluon and taking advantage of the gradient information\n", "\n", "[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. \"Explaining and harnessing adversarial examples.\" arXiv preprint arXiv:1412.6572 (2014).\n", "https://arxiv.org/abs/1412.6572" @@ -15,7 +14,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "collapsed": false }, @@ -28,290 +27,312 @@ "import matplotlib.pyplot as plt\n", "import matplotlib.cm as cm\n", "\n", - "from mxnet.test_utils import get_mnist_iterator" + "from mxnet import gluon" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Build Network\n", - "\n", - "note: in this network, we will calculate softmax, gradient in numpy" + "Build simple CNN network for solving the MNIST dataset digit recognition task" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 17, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "dev = mx.cpu()\n", - "batch_size = 100\n", - "train_iter, val_iter = get_mnist_iterator(batch_size=batch_size, input_shape = (1,28,28))" + "ctx = mx.gpu() if len(mx.test_utils.list_gpus()) else mx.cpu()\n", + "batch_size = 128" ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], + "cell_type": "markdown", + "metadata": {}, "source": [ - "# input\n", - "data = mx.symbol.Variable('data')\n", - "# first conv\n", - "conv1 = mx.symbol.Convolution(data=data, kernel=(5,5), num_filter=20)\n", - "tanh1 = mx.symbol.Activation(data=conv1, act_type=\"tanh\")\n", - "pool1 = mx.symbol.Pooling(data=tanh1, pool_type=\"max\",\n", - " kernel=(2,2), stride=(2,2))\n", - "# second conv\n", - "conv2 = mx.symbol.Convolution(data=pool1, kernel=(5,5), num_filter=50)\n", - "tanh2 = mx.symbol.Activation(data=conv2, act_type=\"tanh\")\n", - "pool2 = mx.symbol.Pooling(data=tanh2, pool_type=\"max\",\n", - " kernel=(2,2), stride=(2,2))\n", - "# first fullc\n", - "flatten = mx.symbol.Flatten(data=pool2)\n", - "fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=500)\n", - "tanh3 = mx.symbol.Activation(data=fc1, act_type=\"tanh\")\n", - "# second fullc\n", - "fc2 = mx.symbol.FullyConnected(data=tanh3, num_hidden=10)" + "## Data Loading" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, + "execution_count": 3, + "metadata": {}, "outputs": [], "source": [ - "def Softmax(theta):\n", - " max_val = np.max(theta, axis=1, keepdims=True)\n", - " tmp = theta - max_val\n", - " exp = np.exp(tmp)\n", - " norm = np.sum(exp, axis=1, keepdims=True)\n", - " return exp / norm" + "transform = lambda x,y: (x.transpose((2,0,1)).astype('float32')/255., y)\n", + "\n", + "train_dataset = gluon.data.vision.MNIST(train=True).transform(transform)\n", + "test_dataset = gluon.data.vision.MNIST(train=False).transform(transform)\n", + "\n", + "train_data = gluon.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=5)\n", + "test_data = gluon.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create the network" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "def LogLossGrad(alpha, label):\n", - " grad = np.copy(alpha)\n", - " for i in range(alpha.shape[0]):\n", - " grad[i, int(label[i])] -= 1.\n", - " return grad" + "net = gluon.nn.HybridSequential()\n", + "with net.name_scope():\n", + " net.add(\n", + " gluon.nn.Conv2D(kernel_size=5, channels=20, activation='tanh'),\n", + " gluon.nn.MaxPool2D(pool_size=2, strides=2),\n", + " gluon.nn.Conv2D(kernel_size=5, channels=50, activation='tanh'),\n", + " gluon.nn.MaxPool2D(pool_size=2, strides=2),\n", + " gluon.nn.Flatten(),\n", + " gluon.nn.Dense(500, activation='tanh'),\n", + " gluon.nn.Dense(10)\n", + " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Prepare useful data for the network" + "## Initialize training" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": { - "collapsed": false + "collapsed": true }, "outputs": [], "source": [ - "data_shape = (batch_size, 1, 28, 28)\n", - "arg_names = fc2.list_arguments() # 'data' \n", - "arg_shapes, output_shapes, aux_shapes = fc2.infer_shape(data=data_shape)\n", - "\n", - "arg_arrays = [mx.nd.zeros(shape, ctx=dev) for shape in arg_shapes]\n", - "grad_arrays = [mx.nd.zeros(shape, ctx=dev) for shape in arg_shapes]\n", - "reqs = [\"write\" for name in arg_names]\n", - "\n", - "model = fc2.bind(ctx=dev, args=arg_arrays, args_grad = grad_arrays, grad_req=reqs)\n", - "arg_map = dict(zip(arg_names, arg_arrays))\n", - "grad_map = dict(zip(arg_names, grad_arrays))\n", - "data_grad = grad_map[\"data\"]\n", - "out_grad = mx.nd.zeros(model.outputs[0].shape, ctx=dev)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Init weight " + "net.initialize(mx.initializer.Uniform(), ctx=ctx)\n", + "net.hybridize()" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "for name in arg_names:\n", - " if \"weight\" in name:\n", - " arr = arg_map[name]\n", - " arr[:] = mx.rnd.uniform(-0.07, 0.07, arr.shape)" + "loss = gluon.loss.SoftmaxCELoss()" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "def SGD(weight, grad, lr=0.1, grad_norm=batch_size):\n", - " weight[:] -= lr * grad / batch_size\n", - "\n", - "def CalAcc(pred_prob, label):\n", - " pred = np.argmax(pred_prob, axis=1)\n", - " return np.sum(pred == label) * 1.0\n", - "\n", - "def CalLoss(pred_prob, label):\n", - " loss = 0.\n", - " for i in range(pred_prob.shape[0]):\n", - " loss += -np.log(max(pred_prob[i, int(label[i])], 1e-10))\n", - " return loss" + "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1, 'momentum':0.95})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Train a network" + "## Training loop" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train Accuracy: 0.92\t Train Loss: 0.32142\n", + "Train Accuracy: 0.97\t Train Loss: 0.16773\n", + "Train Accuracy: 0.97\t Train Loss: 0.14660\n" + ] + } + ], "source": [ - "num_round = 4\n", - "train_acc = 0.\n", - "nbatch = 0\n", - "for i in range(num_round):\n", + "epoch = 3\n", + "for e in range(epoch):\n", " train_loss = 0.\n", - " train_acc = 0.\n", - " nbatch = 0\n", - " train_iter.reset()\n", - " for batch in train_iter:\n", - " arg_map[\"data\"][:] = batch.data[0]\n", - " model.forward(is_train=True)\n", - " theta = model.outputs[0].asnumpy()\n", - " alpha = Softmax(theta)\n", - " label = batch.label[0].asnumpy()\n", - " train_acc += CalAcc(alpha, label) / batch_size\n", - " train_loss += CalLoss(alpha, label) / batch_size\n", - " losGrad_theta = LogLossGrad(alpha, label)\n", - " out_grad[:] = losGrad_theta\n", - " model.backward([out_grad])\n", - " # data_grad[:] = grad_map[\"data\"]\n", - " for name in arg_names:\n", - " if name != \"data\":\n", - " SGD(arg_map[name], grad_map[name])\n", + " acc = mx.metric.Accuracy()\n", + " for i, (data, label) in enumerate(train_data):\n", + " data = data.as_in_context(ctx)\n", + " label = label.as_in_context(ctx)\n", + " \n", + " with mx.autograd.record():\n", + " output = net(data)\n", + " l = loss(output, label)\n", + " \n", + " l.backward()\n", + " trainer.update(data.shape[0])\n", " \n", - " nbatch += 1\n", - " #print(np.linalg.norm(data_grad.asnumpy(), 2))\n", - " train_acc /= nbatch\n", - " train_loss /= nbatch\n", - " print(\"Train Accuracy: %.2f\\t Train Loss: %.5f\" % (train_acc, train_loss))" + " train_loss += l.mean().asscalar()\n", + " acc.update(label, output)\n", + " \n", + " print(\"Train Accuracy: %.2f\\t Train Loss: %.5f\" % (acc.get()[1], train_loss/(i+1)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Get pertubation by using fast sign method, check validation change.\n", - "See that the validation set was almost entirely correct before the perturbations, but after the perturbations, it is much worse than random guessing." + "## Perturbation\n", + "\n", + "We first run a validation batch and measure the resulting accuracy.\n", + "We then perturbate this batch by modifying the input in the opposite direction of the gradient." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation batch accuracy 0.96875\n" + ] + } + ], + "source": [ + "# Get a batch from the testing set\n", + "for data, label in test_data:\n", + " data = data.as_in_context(ctx)\n", + " label = label.as_in_context(ctx)\n", + " break\n", + "\n", + "# Attach gradient to it to get the gradient of the loss with respect to the input\n", + "data.attach_grad()\n", + "with mx.autograd.record():\n", + " output = net(data) \n", + " l = loss(output, label)\n", + "l.backward()\n", + "\n", + "acc = mx.metric.Accuracy()\n", + "acc.update(label, output)\n", + "\n", + "print(\"Validation batch accuracy {}\".format(acc.get()[1]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we perturb the input" + ] + }, + { + "cell_type": "code", + "execution_count": 10, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Validation batch accuracy after perturbation 0.40625\n" + ] + } + ], "source": [ - "val_iter.reset()\n", - "batch = val_iter.next()\n", - "data = batch.data[0]\n", - "label = batch.label[0]\n", - "arg_map[\"data\"][:] = data\n", - "model.forward(is_train=True)\n", - "theta = model.outputs[0].asnumpy()\n", - "alpha = Softmax(theta)\n", - "print(\"Val Batch Accuracy: \", CalAcc(alpha, label.asnumpy()) / batch_size)\n", - "#########\n", - "grad = LogLossGrad(alpha, label.asnumpy())\n", - "out_grad[:] = grad\n", - "model.backward([out_grad])\n", - "noise = np.sign(data_grad.asnumpy())\n", - "arg_map[\"data\"][:] = data.asnumpy() + 0.15 * noise\n", - "model.forward(is_train=True)\n", - "raw_output = model.outputs[0].asnumpy()\n", - "pred = Softmax(raw_output)\n", - "print(\"Val Batch Accuracy after pertubation: \", CalAcc(pred, label.asnumpy()) / batch_size)" + "data_perturbated = data + 0.15 * mx.nd.sign(data.grad)\n", + "\n", + "output = net(data_perturbated) \n", + "\n", + "acc = mx.metric.Accuracy()\n", + "acc.update(label, output)\n", + "\n", + "print(\"Validation batch accuracy after perturbation {}\".format(acc.get()[1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Visualize an example after pertubation.\n", - "Note that the prediction is consistently incorrect." + "## Visualization" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's visualize an example after pertubation.\n", + "\n", + "We can see that the prediction is often incorrect." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 16, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "true label: 1\n", + "predicted: 3\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADpJJREFUeJzt3V+IXeW5x/Hfc9JsNbbMmLbGkAQdgxwZAxoZY+EMJy1tgo2F2AuluSg5IE0vIrbQi4q9qJeh9A9eSHGqobG2ScVWDConsaFgS0p1FI/G8VRNSWmGJGOxpCnIjJk8vdgrZYx7r7Wz1989z/cDw+xZ715rPbMmv6y997vW+5q7C0A8/1F3AQDqQfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwT1sSp31mq1fNmyZaVs+/Tp06Vs97yhoaHa9p0lrbYmq/O41X3M0n73rNref//9rm1nz57V/Py89VJDrvCb2W2SHpS0RNIj7r4r7fnLli3T+Ph4nl129eyzz5ay3fPS6i5731nKOqZlq/O41X3M0n73rNqmpqa6tk1PT/dcQ98v+81siaSHJH1R0qikbWY22u/2AFQrz3v+DZLecfc/u/ucpH2SthZTFoCy5Qn/Kkl/XfDz8WTZh5jZDjObNLPJubm5HLsDUKTSP+139wl3H3P3sVarVfbuAPQoT/inJa1Z8PPqZBmAAZAn/C9Jus7MRsysJekrkvYXUxaAsvXd1efuZ83sHkkH1O7q2+3ubxRWWQd1dg3V3Z3XVE0+LrfffnvdJTRarn5+d39O0nMF1QKgQlzeCwRF+IGgCD8QFOEHgiL8QFCEHwiq0vv5szS5z7jJBvW4NbkfPuuYlll71r5HRkYK2Q9nfiAowg8ERfiBoAg/EBThB4Ii/EBQ5u7V7cysup1doMndSnmldQ1l/d51dhPW+TcZ1O5RKb2rb3p6WrOzsz0N3c2ZHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCatQtvVkWc199HoN6XOq8bbZueX63tFl6LwZnfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IKlc/v5kdk3RG0ryks+4+lvb8oaEhjY+P59klOshzb3revvQ8ffXPPPNM6rr79u1Lbd+7d29q+/z8fGp7HovhGoQiLvL5nLv/rYDtAKgQL/uBoPKG3yUdNLOXzWxHEQUBqEbel/3j7j5tZldKet7M/t/dX1j4hOQ/hR2SdNlll+XcHYCi5Drzu/t08n1G0lOSNnR4zoS7j7n7WKvVyrM7AAXqO/xmdrmZfeL8Y0mbJR0pqjAA5crzsn+FpKfM7Px2fuHu/1tIVQBKV+m4/cPDw74Y+/kHeQz4vIaHh1PbDx482LXtlltuybXvO++8M7X9ySef7HvbTe7HT7ufn3H7AWQi/EBQhB8IivADQRF+ICjCDwQ1UEN3lylyd10eGzduTG1fvnx517ajR4+mrjs7O5vanqcrr8my/i2mTdF9MTjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQld7Sa2bV7QyFWLp0aWp7Vp/0tdde2/e+d+7cmdp+4MCBvrc9yNL6+bmlF0Amwg8ERfiBoAg/EBThB4Ii/EBQhB8IqtL7+bOm6Oae+s7KHEY665ivW7cutX3Tpk2p7Wn37J87dy513aj9+FXhzA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQWX285vZbklfkjTj7uuSZcsl/VLSNZKOSbrL3f+et5g8/dlNvkagzume8x6XLVu2pLZnjb2f5sUXX+x7XeTXy5n/p5Juu2DZfZIOuft1kg4lPwMYIJnhd/cXJL13weKtkvYkj/dIuqPgugCUrN/3/Cvc/UTy+KSkFQXVA6AiuT/w8/YggF3H5jOzHWY2aWaTc3NzeXcHoCD9hv+Uma2UpOT7TLcnuvuEu4+5+1ir1epzdwCK1m/490vanjzeLunpYsoBUJXM8JvZXkl/kPSfZnbczO6WtEvSJjN7W9IXkp8BDJDMfn5339al6fMF15JLnX3pdSvzGodbb7011/rz8/Nd23btynfOiPo3n5qaKmQ7XOEHBEX4gaAIPxAU4QeCIvxAUIQfCKrSobtRjjxdXuvXr09tHx0dTW1fu3Ztavvs7GzXtquvvjp13ax25MOZHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqrSf//Tp06Xdfhr19s68brzxxtT2rH78LIcPH861PsrDmR8IivADQRF+ICjCDwRF+IGgCD8QFOEHguJ+/uCuv/76XOtnTcH2+OOP59p+Hkzbno4zPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EldnPb2a7JX1J0oy7r0uWPSDpa5LeTZ52v7s/V1aR6N/NN9+c2n7DDTfk2n5WP/+pU6dybR/l6eXM/1NJt3VY/iN3vyn5IvjAgMkMv7u/IOm9CmoBUKE87/nvMbPXzGy3mV1RWEUAKtFv+H8saa2kmySdkPSDbk80sx1mNmlmk33uC0AJ+gq/u59y93l3PyfpJ5I2pDx3wt3H3H2s3yIBFK+v8JvZygU/flnSkWLKAVCVXrr69kr6rKRPmdlxSd+V9Fkzu0mSSzom6esl1gigBJnhd/dtHRY/WkItudR573YT7s3uZnh4OLXdzHJt/6233sq1fh6L9X79rN9rZGSk720vxBV+QFCEHwiK8ANBEX4gKMIPBEX4gaAYursAWV0zdXYFbt68ObX96NGjqe1r1qxJbX/iiScuuqaiZB3XtL9Lk7tnq8KZHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCqrSff2hoSOPj413bm3yLZh55rwPIWv+qq67q2nbppZemrpvl8OHDqe1HjqSP45Lnb0pffLk48wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUNzP3wB5r29Iu3Yi79DcBw4cSG0v89qMJo+TsBhw5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoDL7+c1sjaTHJK2Q5JIm3P1BM1su6ZeSrpF0TNJd7v73PMWUOa3xYjY0NNT3umfOnEltf/jhh/vedpa8/fR5/uZNvoYga99TU1OF7KeXM/9ZSd9y91FJn5G008xGJd0n6ZC7XyfpUPIzgAGRGX53P+HurySPz0h6U9IqSVsl7UmetkfSHWUVCaB4F/We38yukbRe0h8lrXD3E0nTSbXfFgAYED2H38w+LulXkr7p7v9Y2OburvbnAZ3W22Fmk2Y2OTc3l6tYAMXpKfxmtlTt4P/c3X+dLD5lZiuT9pWSZjqt6+4T7j7m7mOtVquImgEUIDP81r4t7FFJb7r7Dxc07Ze0PXm8XdLTxZcHoCy93NL7X5K+Kul1M3s1WXa/pF2SnjCzuyX9RdJd5ZSILGm39GZZvXp1avvGjRtT2z/44IO+951X3iHPmyqr7pGRkUL2kxl+d/+9pG43hX++kCoAVI4r/ICgCD8QFOEHgiL8QFCEHwiK8ANBLZqhuwd5GOesft0lS5aktqf1+65duzZ13ZMnT6a219mPn6XJw4YPwjUGnPmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKhF088/yLL6jLP6+fMM3T0z03EApp4NQn92J3n76Qf1916IMz8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBGXtmbYq2plZ6s4G+Z78Ol155ZVd2+69997UdR966KHU9kceeaSvmlCetPEbpqenNTs7222o/Q/hzA8ERfiBoAg/EBThB4Ii/EBQhB8IivADQWX285vZGkmPSVohySVNuPuDZvaApK9Jejd56v3u/lzGtqq7qKBCZV+fkGcM+cVw3/kgKvPfxNTUVNe2i+nn72Uwj7OSvuXur5jZJyS9bGbPJ20/cvfv97IjAM2SGX53PyHpRPL4jJm9KWlV2YUBKNdFvec3s2skrZf0x2TRPWb2mpntNrMruqyzw8wmzWwyV6UACtVz+M3s45J+Jemb7v4PST+WtFbSTWq/MvhBp/XcfcLdx9x9rIB6ARSkp/Cb2VK1g/9zd/+1JLn7KXefd/dzkn4iaUN5ZQIoWmb4zcwkPSrpTXf/4YLlKxc87cuSjhRfHoCy9NLVNy7pd5Jel3QuWXy/pG1qv+R3ScckfT35cDBtW4uyqw+4WHm6Aivr6nP330vqtLHUPn0AzcYVfkBQhB8IivADQRF+ICjCDwRF+IGgKh26+5JLLvFVqwbznqDR0dG+181zS27Zyr7lN8/txmlDVKMzhu4GkInwA0ERfiAowg8ERfiBoAg/EBThB4KqeorudyX9ZcGiT0n6W2UFXJym1tbUuiRq61eRtV3t7p/u5YmVhv8jOzebbOrYfk2tral1SdTWr7pq42U/EBThB4KqO/wTNe8/TVNra2pdErX1q5baan3PD6A+dZ/5AdSklvCb2W1m9icze8fM7qujhm7M7JiZvW5mr9Y9xVgyDdqMmR1ZsGy5mT1vZm8n3ztOk1ZTbQ+Y2XRy7F41sy011bbGzH5rZlNm9oaZfSNZXuuxS6mrluNW+ct+M1si6S1JmyQdl/SSpG3u3n0w8gqZ2TFJY+5ee5+wmf23pH9Keszd1yXLvifpPXfflfzHeYW7f7shtT0g6Z91z9ycTCizcuHM0pLukPQ/qvHYpdR1l2o4bnWc+TdIesfd/+zuc5L2SdpaQx2N5+4vSHrvgsVbJe1JHu9R+x9P5brU1gjufsLdX0ken5F0fmbpWo9dSl21qCP8qyT9dcHPx9WsKb9d0kEze9nMdtRdTAcrFsyMdFLSijqL6SBz5uYqXTCzdGOOXT8zXheND/w+atzdb5b0RUk7k5e3jeTt92xN6q7paebmqnSYWfrf6jx2/c54XbQ6wj8tac2Cn1cnyxrB3aeT7zOSnlLzZh8+dX6S1OT7TM31/FuTZm7uNLO0GnDsmjTjdR3hf0nSdWY2YmYtSV+RtL+GOj7CzC5PPoiRmV0uabOaN/vwfknbk8fbJT1dYy0f0pSZm7vNLK2aj13jZrx298q/JG1R+xP/o5K+U0cNXeq6VtL/JV9v1F2bpL1qvwz8QO3PRu6W9ElJhyS9Lek3kpY3qLafqT2b82tqB21lTbWNq/2S/jVJryZfW+o+dil11XLcuMIPCIoP/ICgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBPUv5DLnMbZADooAAAAASUVORK5CYII=\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ - "import random as rnd\n", - "idx = rnd.randint(0, 99)\n", - "images = data.asnumpy() + 0.15 * noise\n", - "plt.imshow(images[idx, :].reshape(28,28), cmap=cm.Greys_r)\n", - "print(\"true: %d\" % label.asnumpy()[idx])\n", - "print(\"pred: %d\" % np.argmax(pred, axis=1)[idx])" + "from random import randint\n", + "idx = randint(0, batch_size-1)\n", + "\n", + "plt.imshow(data_perturbated[idx, :].asnumpy().reshape(28,28), cmap=cm.Greys_r)\n", + "print(\"true label: %d\" % label.asnumpy()[idx])\n", + "print(\"predicted: %d\" % np.argmax(output.asnumpy(), axis=1)[idx])" ] } ], "metadata": { "kernelspec": { - "display_name": "Python 2", + "display_name": "Python 3", "language": "python", - "name": "python2" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.13" + "pygments_lexer": "ipython3", + "version": "3.6.4" } }, "nbformat": 4, - "nbformat_minor": 0 + "nbformat_minor": 2 } diff --git a/example/vae/README.md b/example/autoencoder/variational_autoencoder/README.md similarity index 100% rename from example/vae/README.md rename to example/autoencoder/variational_autoencoder/README.md diff --git a/example/vae/VAE.py b/example/autoencoder/variational_autoencoder/VAE.py similarity index 100% rename from example/vae/VAE.py rename to example/autoencoder/variational_autoencoder/VAE.py diff --git a/example/autoencoder/variational_autoencoder/VAE_example.ipynb b/example/autoencoder/variational_autoencoder/VAE_example.ipynb new file mode 100755 index 000000000..964e13725 --- /dev/null +++ b/example/autoencoder/variational_autoencoder/VAE_example.ipynb @@ -0,0 +1,1204 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import mxnet as mx\n", + "import numpy as np\n", + "import os\n", + "import logging\n", + "import matplotlib.pyplot as plt\n", + "import matplotlib.cm as cm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Building a Variational Autoencoder in MXNet\n", + "\n", + "#### Xiaoyu Lu, July 5th, 2017\n", + "\n", + "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "To complete this tutorial, we need following python packages:\n", + "\n", + "- numpy, matplotlib " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 1. Loading the Data\n", + "\n", + "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "60000 784\n" + ] + } + ], + "source": [ + "mnist = mx.test_utils.get_mnist()\n", + "image = np.reshape(mnist['train_data'],(60000,28*28))\n", + "label = image\n", + "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n", + "label_test = image_test\n", + "[N,features] = np.shape(image) #number of examples and features\n", + "print(N,features)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACWCAYAAAA7UIUvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAFI5JREFUeJzt3X+wVfO/x/H3u9+JfikVnb5FaYTEnLmqe5mLDDKJwfGVaYzfVJRBP+5FgzHjd6YoMkyFcbvUJDToxnW7GDql6YpSFwnpVKRfqDM+94+2O+f9Waf94+xfa+3P8zHTnF6rtfd6n/a73afde62lzjkBAAAAQtSs3AUAAAAA5cJiGAAAAMFiMQwAAIBgsRgGAABAsFgMAwAAIFgshgEAABAsFsMAAAAIFothAAAABCuvxbCqnq+q61V1o6pOLlRRAAAAQCloU+9Ap6rNReQrETlXRL4XkRUicqVz7otDPaZLly6ud+/eTToe4uXbb7+V7du3azGemz6pLCtXrtzunOtajOemVyoH7ynIFu8pyEYu7ykt8jjOP4jIRufc1yIiqvpvIjJSRA65GO7du7fU1tbmcUjERXV1ddGemz6pLKq6qVjPTa9UDt5TkC3eU5CNXN5T8hmTOEZENjfI36e2Gap6o6rWqmrttm3b8jgcKhl9gmzRK8gGfYJs0Sso+gl0zrnZzrlq51x1165F+V8NVAD6BNmiV5AN+gTZoleQz2L4BxGpapB7prYBAAAAiZDPYniFiPRT1T6q2kpE/i4iiwtTFgAAAFB8TT6BzjlXr6rjROQdEWkuIi8459YWrDIAAACgyPK5moQ455aIyJIC1QIAAACUFHegAwAAQLBYDAMAACBYLIYBAAAQLBbDAAAACBaLYQAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECw8roDHYDiue2220yeMWNGZJ8OHTqYvG7dOpO7d+9e+MIAAKggfDIMAACAYLEYBgAAQLBYDAMAACBYzAwXwdatW00eNGiQyUOHDjV5wYIFRa8J8bdq1SqTn376aZObNYv+23X37t0m19XVmczMMJBcf/zxh8nz5883+Zprrkn7+OXLl5s8ePBgk/ft2xd5TJs2bUw+cOCAyf77UOvWrdPWACQBnwwDAAAgWCyGAQAAECwWwwAAAAgWM8NFMHv2bJP9GeJFixaVshzEVH19vcnz5s3L+TmOP/54k3v16pVXTQDKY+/evZFtU6dONfnJJ580WVXTPuftt99u8mmnnWbyc889F3nMZZddZvL7779vcu/evU32z4Hxz1O48cYbTe7UqdOhCwbKhE+GAQAAECwWwwAAAAgWi2EAAAAEi5nhEnDOlbsExNAnn3xi8owZM3J+jsmTJ5vcsWPHvGoCUB6XXnppZNvSpUtNzjQj7Fu5cqXJtbW1GZ/vtddeS/ucO3bsyOk53333XZOXLFkSeU6uVZzehg0bTO7fv39knxEjRpg8a9Ysk48++ujCF+bZuXOnyX6v+OdTZTJu3LjItqqqqtwLywKfDAMAACBYLIYBAAAQLBbDAAAACBYzw0Wwdu1ak/0ZqksuuaSU5SAm/vzzT5OnTJmS0+NXrFgR2XbyySfnVRMyu+WWW0z+4osvTL7qqqtMHjhwoMmDBw8uTmFItLlz55r8wQcflKmS4vr6669N/uOPPyL7MDOc3vz5801ubNb7jTfeMNk/J8WfGT7qqKNMHjVqVNoaPvvss8g2v2d//PFHk7dt22ayf/5Uphn48847L7KNmWEAAACgwFgMAwAAIFgshgEAABAsZoaLwJ/F8TV2P3hUPn/m6sMPP0y7/xFHHGFyY9eJbNmyZf6FwfCvs/rss8+m3X/58uUm+3NwLVpE32b9uWL/epp9+vRJ+xz79u0zediwYWlrRPn512CdOnWqyQcOHChlOQXjnwMzcuRIky+//HKT27RpU/Sakqa+vt7k1atXm/zEE0/k/Jz+vK6fff41rXOd7y2GIUOGlOxYfDIMAACAYLEYBgAAQLAyLoZV9QVVrVPVzxts66yqS1V1Q+prp+KWCQAAABReNjPDc0TkKRGZ12DbZBFZ5px7SFUnp/KkwpeXDPv37zd548aNZaoEcfbNN9/ktP8pp5xicvfu3QtZDg7Bn+dt27atyb/99lvax/uzdo3Ngq5cudLka665xuRmzeznFP68nn+Mww47zOQuXbpEjvnRRx+ZTD+V1po1a0z2/97wX1OR6LXJ/b7IpF27dia3b9/e5MauLeu/7/j8GeBWrVrlVBOidu3aZfLpp5+edv+amprItuHDh5vsn6OyaNEikzdt2pRLiSXhfw+NnW9RLBn/ZDnn/ktEfvY2jxSRv64YPldELi5wXQAAAEDRNXVmuJtzbkvq5z+JSLcC1QMAAACUTN4n0LmD/7cT/f+dFFW9UVVrVbU206U9EC76BNmiV5AN+gTZolfQ1MXwVlXtISKS+lp3qB2dc7Odc9XOuequXbs28XCodPQJskWvIBv0CbJFr6Cp08mLReRqEXko9fX1glWUQNu3bzf57bffNrmxEyMQnrfeeiun/R955JEiVYJ0jj/+eJP9P98zZ8402T/5xffMM89EtmX69Mk/cSqTPXv2pM0i0e9r4cKFJnPjjtKqq7OfITV2UwP/hDn/RMkrrrjC5NGjR5t87LHHmlxVVZVznSi8vXv3mnz33Xfn9Pjrr78+su2cc84x2e+FBx980OQtW7ZIvt59912T77zzTpMznWzsn5zsv1eW8qZS2Vxa7RUR+VhE+qvq96p6nRxcBJ+rqhtEZFgqAwAAAImS8ZNh59yVh/ilcw6xHQAAAEgE7kAHAACAYJXuisYV7OOPPza5sdkvhOfXX381edmyZTk9vrq6upDloIn8ubY77rgjp8dPmhS9H1GmmeBXXnnF5Ew38vEvqL9hw4bIPv4ccaZ5PhSWPyfaFL169TJ5xowZJvu9inj6/fffTZ4/f77J/nlGN998s8n+fHA2/N7w58kzWbduXWTbmDFjcnqODh06mPzhhx+afMwxx+T0fIXEJ8MAAAAIFothAAAABIvFMAAAAILFzHABnHrqqSb714Js1aqVyW3atCl6TSg//9qRP/zwQ9r9u3WzdzVn9rwy+O8H2bjhhhty2v/ee+81ee7cuZF9xo0bZ/LkyZNNHjFiRE7HRG78vyeOPPJIk3fs2JHxOdavX2/yrFmzTB45cqTJxx13XC4lokTat29v8oIFC0wePny4yY8//njRa/K9+eabJtfU1ET2yfR3lD+n7M8IDxgwoInVFR6fDAMAACBYLIYBAAAQLBbDAAAACBYzwwXw2Wefmbxv3z6T+/TpY3JTZggRf/X19Sb7830+f0b4pZdeMrlZM/6tiuy0a9fO5FGjRkX28WeGJ0yYUNSaYHXv3t3kfv36mZzNzLBv4sSJJk+dOtXkRx991GT/erUoj5YtW5o8ZMgQk9977z2TS3H96Hfeecfkiy66yORszmHxr8E+fvx4k8t5HeFM+NsWAAAAwWIxDAAAgGCxGAYAAECwmBkuAP/aef59xS+//PJSloMymT59usnLli1Lu/+JJ55o8tlnn13wmhCmZ555JuM+/gwrSuu1114zubq6OrKPf23yTOcR+OerjB071uS77rrL5BUrVkSeo2/fvia3aMEyodhat25t8uDBg4t+zO+++87kiy++OO3+Rx11VGTbLbfcYrI/w56keyrwyTAAAACCxWIYAAAAwWIxDAAAgGAxDFQA/uydfz2+QYMGlbIclEk2c5oNnXDCCUWqBKHZuXOnydOmTYvs48/vVVVVFbUmpOf/vTFv3rzIPqNHjzZ569ateR3Tnyk+6aSTIvv417g988wz8zom4sG/jrA/77t//36Tjz76aJM//fTTyHP6+yQZnwwDAAAgWCyGAQAAECwWwwAAAAgWM8MF4M/n+dcZXr16tckjRowoek2IP/oAheLPq2/bti2yj99vnMsQL41dZ9y/hr1/DeBiuOKKK0xesGCByUOHDi16Dcjfrl27TPZnwb/99tu0j9+7d6/J/nkJIswMAwAAABWBxTAAAACCxWIYAAAAwWIxDAAAgGBxAl0B+DfZ8DMqz6ZNmyLbfvzxx7SPad68ucmnn356QWtCOH7++WeTp0+fbnLPnj0jj3n55ZeLWhMKz78xyubNm01euHChyffff7/JO3bsyPmY/smXY8eONfmjjz4yuW3btjkfA8W3Zs0akx977DGTM61TZs2aZfKAAQMKU1hM8ckwAAAAgsViGAAAAMFiMQwAAIBgMTNcAP6Fp3/66SeTL7roolKWgxLwX2MRkd9++y3tY8aMGWNy+/btC1oTKtf+/ftNfvjhh032+/HCCy+MPMfhhx9e+MJQVP55Bj169DC5Y8eOJvs3WiiE1q1bp60J8VBbW2tyrjd1qqmpMfmCCy7Iu6Yk4ZNhAAAABCvjYlhVq1T1fVX9QlXXqur41PbOqrpUVTekvnYqfrkAAABA4WTzyXC9iNzhnBsgIoNFZKyqDhCRySKyzDnXT0SWpTIAAACQGBlnhp1zW0RkS+rnu1X1SxE5RkRGisg/p3abKyL/KSKTilJlzK1atcpk//p9nTrxoXmlefbZZ3N+zFNPPWXyTTfdZPIJJ5yQV02oXFu2bDH5kUceMdm/1usDDzxQ9JqQ3u7du02eOHGiye3atTO5W7dukedwzpn86quvmuz/3ZPJn3/+aXKzZpk/DzvssMNyOgaK7/XXX49su/baa03OND/euXNnk6dNm2Zyhw4dmlhdMuU0M6yqvUXkVBH5RES6pRbKIiI/iUj0TzIAAAAQY1kvhlX1cBFZICITnHPmnxzu4D9f3SEed6Oq1qpqrX9nG+Av9AmyRa8gG/QJskWvIKvFsKq2lIML4Zedc3/d/3GrqvZI/XoPEalr7LHOudnOuWrnXHXXrl0LUTMqEH2CbNEryAZ9gmzRK8g4M6wHB2CfF5EvnXNPNPilxSJytYg8lPoaHWIJRN++fU32r/nZq1evUpaDmPLn//w+YWYYh7Jz506T/fMSHnvsMZMHDRpU9JqQ3ubNm01+7rnncn4O/z3Df939nIk/I9zY4/33IX8+vVWrVjkdE/nbtGmTyffcc09kn19++cXkTL0xbNgwk0M/tymbm278o4iMFpH/UdXVqW3/IgcXwf+uqteJyCYRqTnE4wEAAIBYyuZqEv8tIof6J8Y5hS0HAAAAKB3uQAcAAIBgZTMmgQz8a4D6szr79u0zmes2Jl+PHj1yfow/a3fWWWcVqhxUmPr6epOnTp1qcsuWLU0eOnRo0WtCbvzruF533XUmP//886Usp1Ht27ePbFuyZInJVVVVpSoHKXV19noEp512msn+fLBIdL7cd8YZZ5j84osvmtyiRdjLQT4ZBgAAQLBYDAMAACBYLIYBAAAQrLCHRApkz549JvvzpMwIV55bb701sm3mzJkm+/d2nz59elFrQuVYu3atyYsXLzb5wgsvNJnrCsdP9+7dTX7wwQdNfvPNN03eunVr0WsaP368yaNGjYrsw4xw6flriP79+5u8a5e56W9W15e+7777TJ4wYYLJoc8I+/hkGAAAAMFiMQwAAIBgsRgGAABAsBgaKQD/PuH+rBgqT2OvcWPXfgSa4oEHHjDZn+/z508Rfx07djT5q6++MnnMmDGRx7z00ks5HaO6utrkadOmmTxkyJCcng/FsXfvXpMvu+wyk/0Z4Wz4M8KTJk0y2b/OPSw+GQYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAgWJ9AVwP3331/uEgBUsDPOOMPkgQMHlqkSNJV/EqSf586dG3lMY9uQfO3atTO5rq4up8fX1NREtk2ZMsVkbqqRGz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZKAAAAymTx4sUmjxs3zuS2bduaPGfOnMhzMCOcHz4ZBgAAQLBYDAMAACBYLIYBAAAQLIZMACBm/JnBhQsXlqkSAMXWs2dPkxctWlSmSsLFJ8MAAAAIFothAAAABIvFMAAAAIKlzrnSHUx1m4hsEpEuIrK9ZAduGmpM72/Oua7FeOKE9YlIMuqkV8qPGtMrRZ+I8DoUSqX3Cq9B4ZSrzqz7pKSL4f8/qGqtc6665AfOATWWX1K+vyTUmYQa85GE748a4yEJ3yM1ll8Svr8k1CiSjDoZkwAAAECwWAwDAAAgWOVaDM8u03FzQY3ll5TvLwl1JqHGfCTh+6PGeEjC90iN5ZeE7y8JNYokoM6yzAwDAAAAccCYBAAAAIJV0sWwqp6vqutVdaOqTi7lsdNR1RdUtU5VP2+wrbOqLlXVDamvncpcY5Wqvq+qX6jqWlUdH8c6CyWOvUKfxE8c+0SEXokjeqXJ9QXVJyLx7JW490mqnsT2SskWw6raXESeFpELRGSAiFypqgNKdfwM5ojI+d62ySKyzDnXT0SWpXI51YvIHc65ASIyWETGpn7/4lZn3mLcK3OEPomNGPeJCL0SK/RKXoLpE5FY98ociXefiCS5V5xzJfkhIkNE5J0GeYqITCnV8bOor7eIfN4grxeRHqmf9xCR9eWu0av3dRE5N+51Vlqv0Cfx+RHnPqFX4vWDXqFPKqFXktQnSeuVUo5JHCMimxvk71Pb4qqbc25L6uc/iUi3chbTkKr2FpFTReQTiXGdeUhSr8T2958+iZ3Yvgb0SuzE8jUIoE9EktUrsX0NktYrnECXBXfwnzOxuOyGqh4uIgtEZIJzblfDX4tTnSGK0+8/fRJvcXoN6JV4i8trQJ/EW5xegyT2SikXwz+ISFWD3DO1La62qmoPEZHU17oy1yOq2lIONtjLzrmFqc2xq7MAktQrsfv9p09iK3avAb0SW7F6DQLqE5Fk9UrsXoOk9kopF8MrRKSfqvZR1VYi8ncRWVzC4+dqsYhcnfr51XJw9qVsVFVF5HkR+dI590SDX4pVnQWSpF6J1e8/fRLbPhGJ2WtAr9Ar2QisT0SS1Suxeg0S3SslHqYeLiJficj/isi/lntgukFdr4jIFhE5IAfng64TkSPl4FmPG0TkP0Skc5lr/Cc5+F8La0RkderH8LjVWcm9Qp/E70cc+4ReiecPeoU+SXKvxL1Pkt4r3IEOAAAAweIEOgAAAASLxTAAAACCxWIYAAAAwWIxDAAAgGCxGAYAAECwWAwDAAAgWCyGAQAAECwWwwAAAAjW/wEgPmufEARJLAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "nsamples = 5\n", + "idx = np.random.choice(len(mnist['train_data']), nsamples)\n", + "_, axarr = plt.subplots(1, nsamples, sharex='col', sharey='row',figsize=(12,3))\n", + "\n", + "for i,j in enumerate(idx):\n", + " axarr[i].imshow(np.reshape(image[j,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "model_prefix = None\n", + "\n", + "batch_size = 100\n", + "latent_dim = 5\n", + "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n", + " batch_size = batch_size)\n", + "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n", + " batch_size = batch_size)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Building the Network Architecture\n", + "\n", + "### 2.1 Gaussian MLP as encoder\n", + "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n", + "\n", + "\\begin{align}\n", + "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n", + "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n", + "h &= \\tanh(W_1x+b_1)\n", + "\\end{align}\n", + "\n", + "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n", + "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n", + "\n", + "\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "## define data and loss labels as symbols \n", + "data = mx.sym.var('data')\n", + "loss_label = mx.sym.var('loss_label')\n", + "\n", + "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n", + "encoder_h = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n", + "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n", + "\n", + "## define mu and log variance which are the fully connected layers of the previous activation layer\n", + "mu = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = latent_dim)\n", + "logvar = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = latent_dim)\n", + "\n", + "## sample the latent variables z according to Normal(mu,var)\n", + "z = mu + mx.symbol.broadcast_mul(mx.symbol.exp(0.5 * logvar), \n", + " mx.symbol.random_normal(loc=0, scale=1, shape=(batch_size, latent_dim)),\n", + " name=\"z\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.2 Bernoulli MLP as decoder\n", + "\n", + "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n", + "\n", + "\\begin{align}\n", + "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n", + "\\textit{ where } y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n", + "\\end{align}\n", + "\n", + "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "# define fully connected and tanh activation layers for the decoder\n", + "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n", + "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n", + "\n", + "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n", + "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n", + "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3 Joint Loss Function for the Encoder and the Decoder\n", + "\n", + "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n", + "\n", + "\\begin{align}\n", + "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n", + "\\end{align}\n", + "\n", + "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "# define the objective loss function that needs to be minimized\n", + "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n", + "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n", + " + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n", + "output = mx.symbol.MakeLoss(sum(loss),name='loss')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 3. Training the model\n", + "\n", + "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n", + "\n", + "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "# set up the log\n", + "nd_iter.reset()\n", + "logging.getLogger().setLevel(logging.DEBUG) \n", + "\n", + "# define function to trave back training loss\n", + "def log_to_list(period, lst):\n", + " def _callback(param):\n", + " \"\"\"The checkpoint function.\"\"\"\n", + " if param.nbatch % period == 0:\n", + " name, value = param.eval_metric.get()\n", + " lst.append(value)\n", + " return _callback\n", + "\n", + "# define the model\n", + "model = mx.mod.Module(\n", + " symbol = output ,\n", + " data_names=['data'],\n", + " label_names = ['loss_label'])" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Epoch[0] Train-loss=373.547317\n", + "INFO:root:Epoch[0] Time cost=5.020\n", + "INFO:root:Epoch[1] Train-loss=212.232684\n", + "INFO:root:Epoch[1] Time cost=4.651\n", + "INFO:root:Epoch[2] Train-loss=207.448528\n", + "INFO:root:Epoch[2] Time cost=4.665\n", + "INFO:root:Epoch[3] Train-loss=205.369479\n", + "INFO:root:Epoch[3] Time cost=4.758\n", + "INFO:root:Epoch[4] Train-loss=203.651983\n", + "INFO:root:Epoch[4] Time cost=4.672\n", + "INFO:root:Epoch[5] Train-loss=202.061007\n", + "INFO:root:Epoch[5] Time cost=5.087\n", + "INFO:root:Epoch[6] Train-loss=199.348143\n", + "INFO:root:Epoch[6] Time cost=5.056\n", + "INFO:root:Epoch[7] Train-loss=196.266242\n", + "INFO:root:Epoch[7] Time cost=4.813\n", + "INFO:root:Epoch[8] Train-loss=194.694945\n", + "INFO:root:Epoch[8] Time cost=4.776\n", + "INFO:root:Epoch[9] Train-loss=193.699284\n", + "INFO:root:Epoch[9] Time cost=4.756\n", + "INFO:root:Epoch[10] Train-loss=193.036517\n", + "INFO:root:Epoch[10] Time cost=4.757\n", + "INFO:root:Epoch[11] Train-loss=192.555736\n", + "INFO:root:Epoch[11] Time cost=4.678\n", + "INFO:root:Epoch[12] Train-loss=192.020813\n", + "INFO:root:Epoch[12] Time cost=4.630\n", + "INFO:root:Epoch[13] Train-loss=191.648876\n", + "INFO:root:Epoch[13] Time cost=5.158\n", + "INFO:root:Epoch[14] Train-loss=191.057798\n", + "INFO:root:Epoch[14] Time cost=4.781\n", + "INFO:root:Epoch[15] Train-loss=190.315835\n", + "INFO:root:Epoch[15] Time cost=5.117\n", + "INFO:root:Epoch[16] Train-loss=189.311271\n", + "INFO:root:Epoch[16] Time cost=4.707\n", + "INFO:root:Epoch[17] Train-loss=187.285967\n", + "INFO:root:Epoch[17] Time cost=4.745\n", + "INFO:root:Epoch[18] Train-loss=185.271324\n", + "INFO:root:Epoch[18] Time cost=4.692\n", + "INFO:root:Epoch[19] Train-loss=183.510888\n", + "INFO:root:Epoch[19] Time cost=4.762\n", + "INFO:root:Epoch[20] Train-loss=181.756008\n", + "INFO:root:Epoch[20] Time cost=4.838\n", + "INFO:root:Epoch[21] Train-loss=180.546818\n", + "INFO:root:Epoch[21] Time cost=4.764\n", + "INFO:root:Epoch[22] Train-loss=179.479776\n", + "INFO:root:Epoch[22] Time cost=4.791\n", + "INFO:root:Epoch[23] Train-loss=178.352077\n", + "INFO:root:Epoch[23] Time cost=4.981\n", + "INFO:root:Epoch[24] Train-loss=177.385084\n", + "INFO:root:Epoch[24] Time cost=5.292\n", + "INFO:root:Epoch[25] Train-loss=175.920123\n", + "INFO:root:Epoch[25] Time cost=5.097\n", + "INFO:root:Epoch[26] Train-loss=174.377171\n", + "INFO:root:Epoch[26] Time cost=4.907\n", + "INFO:root:Epoch[27] Train-loss=172.590589\n", + "INFO:root:Epoch[27] Time cost=4.484\n", + "INFO:root:Epoch[28] Train-loss=170.933683\n", + "INFO:root:Epoch[28] Time cost=4.348\n", + "INFO:root:Epoch[29] Train-loss=169.866807\n", + "INFO:root:Epoch[29] Time cost=4.647\n", + "INFO:root:Epoch[30] Train-loss=169.182084\n", + "INFO:root:Epoch[30] Time cost=5.034\n", + "INFO:root:Epoch[31] Train-loss=168.121719\n", + "INFO:root:Epoch[31] Time cost=5.615\n", + "INFO:root:Epoch[32] Train-loss=167.389992\n", + "INFO:root:Epoch[32] Time cost=4.733\n", + "INFO:root:Epoch[33] Train-loss=166.189067\n", + "INFO:root:Epoch[33] Time cost=5.041\n", + "INFO:root:Epoch[34] Train-loss=163.783392\n", + "INFO:root:Epoch[34] Time cost=5.168\n", + "INFO:root:Epoch[35] Train-loss=162.167959\n", + "INFO:root:Epoch[35] Time cost=5.019\n", + "INFO:root:Epoch[36] Train-loss=161.192039\n", + "INFO:root:Epoch[36] Time cost=5.064\n", + "INFO:root:Epoch[37] Train-loss=160.307114\n", + "INFO:root:Epoch[37] Time cost=5.180\n", + "INFO:root:Epoch[38] Train-loss=159.591957\n", + "INFO:root:Epoch[38] Time cost=5.440\n", + "INFO:root:Epoch[39] Train-loss=159.109593\n", + "INFO:root:Epoch[39] Time cost=5.119\n", + "INFO:root:Epoch[40] Train-loss=158.463844\n", + "INFO:root:Epoch[40] Time cost=5.299\n", + "INFO:root:Epoch[41] Train-loss=158.037287\n", + "INFO:root:Epoch[41] Time cost=4.856\n", + "INFO:root:Epoch[42] Train-loss=157.598576\n", + "INFO:root:Epoch[42] Time cost=5.227\n", + "INFO:root:Epoch[43] Train-loss=157.097344\n", + "INFO:root:Epoch[43] Time cost=5.237\n", + "INFO:root:Epoch[44] Train-loss=156.594472\n", + "INFO:root:Epoch[44] Time cost=4.783\n", + "INFO:root:Epoch[45] Train-loss=156.177069\n", + "INFO:root:Epoch[45] Time cost=4.834\n", + "INFO:root:Epoch[46] Train-loss=155.825302\n", + "INFO:root:Epoch[46] Time cost=4.902\n", + "INFO:root:Epoch[47] Train-loss=155.318117\n", + "INFO:root:Epoch[47] Time cost=4.966\n", + "INFO:root:Epoch[48] Train-loss=154.890766\n", + "INFO:root:Epoch[48] Time cost=5.012\n", + "INFO:root:Epoch[49] Train-loss=154.504158\n", + "INFO:root:Epoch[49] Time cost=4.844\n", + "INFO:root:Epoch[50] Train-loss=154.035214\n", + "INFO:root:Epoch[50] Time cost=4.736\n", + "INFO:root:Epoch[51] Train-loss=153.692903\n", + "INFO:root:Epoch[51] Time cost=5.057\n", + "INFO:root:Epoch[52] Train-loss=153.257554\n", + "INFO:root:Epoch[52] Time cost=5.044\n", + "INFO:root:Epoch[53] Train-loss=152.849715\n", + "INFO:root:Epoch[53] Time cost=4.783\n", + "INFO:root:Epoch[54] Train-loss=152.483047\n", + "INFO:root:Epoch[54] Time cost=4.842\n", + "INFO:root:Epoch[55] Train-loss=152.091617\n", + "INFO:root:Epoch[55] Time cost=5.044\n", + "INFO:root:Epoch[56] Train-loss=151.715490\n", + "INFO:root:Epoch[56] Time cost=5.029\n", + "INFO:root:Epoch[57] Train-loss=151.362293\n", + "INFO:root:Epoch[57] Time cost=4.873\n", + "INFO:root:Epoch[58] Train-loss=151.003241\n", + "INFO:root:Epoch[58] Time cost=4.729\n", + "INFO:root:Epoch[59] Train-loss=150.619678\n", + "INFO:root:Epoch[59] Time cost=5.068\n", + "INFO:root:Epoch[60] Train-loss=150.296043\n", + "INFO:root:Epoch[60] Time cost=4.458\n", + "INFO:root:Epoch[61] Train-loss=149.964152\n", + "INFO:root:Epoch[61] Time cost=4.828\n", + "INFO:root:Epoch[62] Train-loss=149.694102\n", + "INFO:root:Epoch[62] Time cost=5.012\n", + "INFO:root:Epoch[63] Train-loss=149.290113\n", + "INFO:root:Epoch[63] Time cost=5.193\n", + "INFO:root:Epoch[64] Train-loss=148.934186\n", + "INFO:root:Epoch[64] Time cost=4.999\n", + "INFO:root:Epoch[65] Train-loss=148.657502\n", + "INFO:root:Epoch[65] Time cost=4.810\n", + "INFO:root:Epoch[66] Train-loss=148.331948\n", + "INFO:root:Epoch[66] Time cost=5.201\n", + "INFO:root:Epoch[67] Train-loss=148.018539\n", + "INFO:root:Epoch[67] Time cost=4.833\n", + "INFO:root:Epoch[68] Train-loss=147.746825\n", + "INFO:root:Epoch[68] Time cost=5.187\n", + "INFO:root:Epoch[69] Train-loss=147.406399\n", + "INFO:root:Epoch[69] Time cost=5.355\n", + "INFO:root:Epoch[70] Train-loss=147.181831\n", + "INFO:root:Epoch[70] Time cost=4.989\n", + "INFO:root:Epoch[71] Train-loss=146.860770\n", + "INFO:root:Epoch[71] Time cost=4.934\n", + "INFO:root:Epoch[72] Train-loss=146.604369\n", + "INFO:root:Epoch[72] Time cost=5.283\n", + "INFO:root:Epoch[73] Train-loss=146.351628\n", + "INFO:root:Epoch[73] Time cost=5.062\n", + "INFO:root:Epoch[74] Train-loss=146.102506\n", + "INFO:root:Epoch[74] Time cost=4.540\n", + "INFO:root:Epoch[75] Train-loss=145.828805\n", + "INFO:root:Epoch[75] Time cost=4.875\n", + "INFO:root:Epoch[76] Train-loss=145.571626\n", + "INFO:root:Epoch[76] Time cost=4.856\n", + "INFO:root:Epoch[77] Train-loss=145.365383\n", + "INFO:root:Epoch[77] Time cost=5.003\n", + "INFO:root:Epoch[78] Train-loss=145.101047\n", + "INFO:root:Epoch[78] Time cost=4.718\n", + "INFO:root:Epoch[79] Train-loss=144.810765\n", + "INFO:root:Epoch[79] Time cost=5.127\n", + "INFO:root:Epoch[80] Train-loss=144.619876\n", + "INFO:root:Epoch[80] Time cost=4.737\n", + "INFO:root:Epoch[81] Train-loss=144.399066\n", + "INFO:root:Epoch[81] Time cost=4.742\n", + "INFO:root:Epoch[82] Train-loss=144.220090\n", + "INFO:root:Epoch[82] Time cost=4.810\n", + "INFO:root:Epoch[83] Train-loss=143.904279\n", + "INFO:root:Epoch[83] Time cost=5.176\n", + "INFO:root:Epoch[84] Train-loss=143.734935\n", + "INFO:root:Epoch[84] Time cost=4.921\n", + "INFO:root:Epoch[85] Train-loss=143.499403\n", + "INFO:root:Epoch[85] Time cost=4.692\n", + "INFO:root:Epoch[86] Train-loss=143.304287\n", + "INFO:root:Epoch[86] Time cost=4.778\n", + "INFO:root:Epoch[87] Train-loss=143.096145\n", + "INFO:root:Epoch[87] Time cost=4.962\n", + "INFO:root:Epoch[88] Train-loss=142.877920\n", + "INFO:root:Epoch[88] Time cost=4.815\n", + "INFO:root:Epoch[89] Train-loss=142.677429\n", + "INFO:root:Epoch[89] Time cost=5.127\n", + "INFO:root:Epoch[90] Train-loss=142.499622\n", + "INFO:root:Epoch[90] Time cost=5.463\n", + "INFO:root:Epoch[91] Train-loss=142.300291\n", + "INFO:root:Epoch[91] Time cost=4.639\n", + "INFO:root:Epoch[92] Train-loss=142.111362\n", + "INFO:root:Epoch[92] Time cost=5.064\n", + "INFO:root:Epoch[93] Train-loss=141.912848\n", + "INFO:root:Epoch[93] Time cost=4.894\n", + "INFO:root:Epoch[94] Train-loss=141.723130\n", + "INFO:root:Epoch[94] Time cost=4.635\n", + "INFO:root:Epoch[95] Train-loss=141.516580\n", + "INFO:root:Epoch[95] Time cost=5.063\n", + "INFO:root:Epoch[96] Train-loss=141.362380\n", + "INFO:root:Epoch[96] Time cost=4.785\n", + "INFO:root:Epoch[97] Train-loss=141.178878\n", + "INFO:root:Epoch[97] Time cost=4.699\n", + "INFO:root:Epoch[98] Train-loss=141.004168\n", + "INFO:root:Epoch[98] Time cost=4.959\n", + "INFO:root:Epoch[99] Train-loss=140.865592\n", + "INFO:root:Epoch[99] Time cost=5.155\n" + ] + } + ], + "source": [ + "# training the model, save training loss as a list.\n", + "training_loss=list()\n", + "\n", + "# initilize the parameters for training using Normal.\n", + "init = mx.init.Normal(0.01)\n", + "model.fit(nd_iter, # train data\n", + " initializer=init,\n", + " # if eval_data is supplied, test loss will also be reported\n", + " # eval_data = nd_iter_test,\n", + " optimizer='sgd', # use SGD to train\n", + " optimizer_params={'learning_rate':1e-3,'wd':1e-2}, \n", + " # save parameters for each epoch if model_prefix is supplied\n", + " epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n", + " batch_end_callback = log_to_list(N/batch_size,training_loss), \n", + " num_epoch=100,\n", + " eval_metric = 'Loss')" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=12.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYXFWd//H3t6t676S7kw5kX4AAsgbMBMQNFQUZZ9AZxsFlwJVxfjjO4vxQ0BkUxVHHGZfRB0WG38AMi7tEh0cEFRjHYQmrbEIgW4ck3Ul3J7133brf3x/3VlPp1K0Kla6uTufzep56UnXurapz63bO957lnmPujoiIyEtVU+0MiIjIwUkBREREyqIAIiIiZVEAERGRsiiAiIhIWRRARESkLAogMqnM7Jtm9veTve9MYGaNZvYTM9ttZt+rwvdfbmbXHui+ZnammXVObu4S87HRzM6aiu+Sly5d7QzI9GFmG4EPuPud5X6Gu3+oEvvOEOcDhwNz3T2Y6i93989VYt+XwsyWAxuA2mr8BjK5VAOR/WZmh9QFRwWOdxnwTDkF56H228vBQQFEADCz/wCWAj8xswEzu9TMlpuZm9n7zWwz8Mt43++Z2fa4KeYeMzs+73P+3cw+Gz8/08w6zeyjZtZlZtvM7L1l7js3bv7ZY2YPmNlnzezXRY7nVWb2GzPrM7MtZvaeOP0uM/tA3n7vyf+c+HgvMbNngWfN7Goz+9KEz77VzP42fr7QzH5gZt1mtsHMPpKQn08D/wD8afz7vt/Maszsk2a2KT7mG8ysNd6/4G8/4TNzv9mleb/ZW83sXDN7xsx6zOzyvP0/ZWb/OeHzLzKzzWa208w+UWjfIr/x5fH7NprZu/LSf9/MHo7P1RYz+1Te2+6J/+2Lf4dXxO/5oJk9ZWb9ZvakmZ2a955VZvZY/Pf2HTNryPuut5jZI/F5/o2ZnZS37WNmtjX+zN+Z2RuKHY+Uwd310AN3B9gInJX3ejngwA1AM9AYp78PmAXUA18BHsl7z78Dn42fnwkEwJVALXAuMAS0l7HvLfGjCTgO2AL8OuE4lgH9wDviz5oLrIq33UXUTJfb9z35nxMf7x3AHKAReE38XRZvbweGgYVEF2APEgWGOuAI4Hng7IR8fQr4z7zX7wPWx+9rAX4I/Eex337C5+V+s3+Ij/ODQDdwU3x+jo/zumLi9+d9/rfj4zwZGAVeViivCd/7L/HfwGuBQeCYvO0nxr/PScAO4K0Tvjed93l/AmwFfg8w4ChgWd7f5P3x7z0HeAr4ULztFKALOA1IARfF+9cDx8TnbWHe9x5Z7f9jM+2hGojsj0+5+6C7DwO4+3Xu3u/uo0QFzcm5K+cCMsCV7p5x99uAAaL/3Pu9r5mlgD8GrnD3IXd/Eri+SH7fCdzp7jfHn7XL3R95Ccf7j+7eEx/vfxMVeK+Ot50P/K+7v0BU4M1z9yvdfczdnycqkC/Yz+95F/Av7v68uw8AlwEXTGiu2uu3LyADXOXuGaIA2wF8NT4/TwBPEgWHJJ9292F3fxR4tMS+E/29u4+6+93AfwFvB3D3u9z9t+4euvtjwM1EQSbJB4AvuvsDHlnv7pvytn/N3V9w9x7gJ8CqOP1i4Fvufp+7Z939eqIgeDqQJQokx5lZrbtvdPfnXsKxyX5QAJH9sSX3xMxSZvZ5M3vOzPYQXfFBVHAVssv3bvMfIrrafin7ziMa8LElb1v+84mWAAdSWIx/trs7UcH8jjjpncCN8fNlwMK4+aTPzPqAy4k6yvfHQiC/oNxEdJz57y92nBD9Ztn4eS7I7MjbPkzy7w2wPe95sXMzUa+7D+a93kR0PJjZaWb2q7hZbzfwIZL/PqD0+UrK4zLgoxN+/yVEtY71wF8TXeB0mdktZrZwP49N9pMCiORLmpo5P/2dwHnAWUArUdMARE0PldJN1GSyOC9tSZH9twBHJmwbJGoGy5lfYJ+Jv8PNwPlmtoyoueQHed+zwd3b8h6z3P3cInnL9wJRIZizlOg48wPAdJ0uu93MmvNeLyU6Hoia0NYCS9y9FfgmL/59FDqeYuermC1Eta/837/J3W8GcPeb3P1VRL+xA18o4zukCAUQybeDqD2+mFlEzQS7iAriigz3zBdfYf8Q+JSZNZnZscCFRd5yI3CWmb3dzNJxB3yu2eMR4I/izzkKeP9+fP/DwE7gWuB2d++LN90P9MedtY1x7ewEM/u9/Ty0m4G/MbMVZtZC9Ft+xw+e4a2fNrM6M3s18BYgd2/LLKDH3UfMbA3RRUdONxCy99/ZtcDfmdnLLXJUHKxL+TbwobjGY2bWHHfgzzKzY8zs9WZWD4wQ1cTCAzxemUABRPL9I/DJuDng7xL2uYGouWIrUfv6vVOUtw8T1Xi2A/9BVPiOFtrR3TcTdcJ/FOghChq5tv0vA2NEwfJ6XmyOKuUmolrXTXnfkyUqOFcR3duQCzJJ/UETXRcfyz3x+0eAv9zP91bbdqCXqNZxI1HH9tPxtv8DXGlm/UQd/N/Nvcndh4CrgP+J/85Od/fvxWk3EQ1++DFRh3lR7r6OaODA1+O8rCcaFAFR/8fnic7JduAwoj4mmUS5kSUiBxUz+wIw390vqnZeRA5VqoHIQcHMjjWzk+KmijVETU8/qna+RA5lurtVDhaziJqtFhI1P/0zcGtVcyRyiFMTloiIlEVNWCIiUpYZ3YTV0dHhy5cvr3Y2REQOKg8++OBOd59Xar8ZHUCWL1/OunXrqp0NEZGDipltKr2XmrBERKRMCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbLM6PtAREQOBkNjATv7x+gbHqOjpZ7DZzeQqonW4BocDdi2e4TB0YCxbMhoJmTPSIaewTH6hsYIQqcuXUNdqoa6dA2pGiNdY8ybVc/rj93fxTHLowAiIhIbC0I6e4foHwnIZEPGsiFhCKE7oTtmRo1ByozhTJau/lG69ozSP5IhlTJSZoQOA6MZBkYCRjIhqVRUoNeYMZLJMpzJMjSWZc9wht3xY2gsu1c+alPG/NYG9gwH7B7OlHUspyxtUwAREUni7uwaHGPTriG69oxgFhXWZjAahAyPRQX2SCbL8FiWoUyWgZGAPSOZ8SCRDZ0gdLbtHmZr7zBhGfPLNtamXgwyGC0NaVrq0zTU1hCETpCNtjXWpmisS9FYm2LJnCZOaKyltbGWuS11dLTU09ZYS/fAKFt6hnmhb5jWxloWtDWwsLWRWQ1p6tMp6tI1zG5M095UR1tTLemaGjLZkNEgJMg7nlwNppIUQESkYsaCkL6hMXqGxkiZ0VyfprkuTf9ohm27R3ihb5gg6zTXp2iuTzM0lmXzriE29wyxbfcw3QNj7OyPrvBDj2oC7kRX9DVGJhvuc/VeTLrGmNWQZlZDLbMa0lGTj0WftWpJO29btYhlc5tpb44K5nTKqE3VEJXFUYEcupONm40On93AvJZ66tLV7U5O1aRoqE1N+fcqgIjIPsLQ6R0ao3coM15oB2HISCa6qh8aC+jPu5Lfndcc0zs4xq7BMXYOjNI/Ut7y7rMb0ixqb6KjpY4jO5qZ3VhLjRmpuJzOhpANQ1I1NSyZ08jSOU0saG2Mt0VX+w210ZV+Q11N9G9titqUxg1NJgUQkRlmJJMlCJ3muhRm0VXz0FjAroExegbHxgv6gdGA0UyWkSBk93CGLT1DdPZGTSe7BsfIvoS2nOa6FLPj5pg5zXWcsKiVuc11zG2uo725jvamOkJ3BkcDBkYDWurTLGhrZGFrA3XpGgZGAwZHszTU1rBsTjOtTbWV+nlkEimAiEwz7k7P4BizGmrHm0bC0NnaN8z6rgH6hscYHI1qAcNjISNB1L7f2TvMs139bO4Zwh1qDJrr0wRZZzhTvJmnNmUsbGtkcXsjZx4zj3mz6pnXUk97cx2puAO4xqChNkVTXZrG2hSzG9PMbqilpSGtK/tDlAKISAXlgkHv0BhjgZPJhgyOBWzri9r/dw6MjnfajmSyPNc9wLM7BugfjZp+OlrqmNNcx9beYQYT2vprU0Z9OsWC1gZOWNjK205ZRFNdiv6RqJkpVWN0tNQzt6WOOU11tDZFNYWokzdFQ20NDekUNVPQ6SoziwKISAm5kT4DIwFB3L6+s3+UzT1RZ2+und8MRjMhuwZH2TkwRnf/KF39I2SyyU1BsxvSpOOr91SNcURHM287NerI7R/JsGPPCDsHxjjjyA6OPnwWRx/ewtyWeprrX6wJTMVoG5FCFEDkkDQ4GjA4Go3TH8pEN2p19kbDOHcPZ8bb6l/oG2ZLz1Di1X9uVA+AA3WpGua21NPRUseKjmYOn93A4bPrmdNcR326htpU1KG7oK2RBa0NVRk5IzJZFEBkRtqxZ4SHNvXy9Pb+8fHxQ5ksG7oHWd89QHf/aMH31aaM1sba8eGmC9saOf2IuSyb20RbUy2pmmjYZ1tTbTzyp2G8BiFyqFEAkRlhcDTg1+t38qunu/j1+p109g4DUbNSbaqGdI1Rn65h2dxmXnv0PI6Y18zshtrxPoD5sxtY3N7EYbPq1Rcgsp8UQOSglMmGPNa5m/99bif/s34XD27qZSwb0lKf5pVHzeU9Zyzn5cvaOX5ha9Vv8hKZqRRAZNpzd17YPcKjW/p4ZEsfD2/u5bHO3YwGIQDHLZjNRWcs43XHHMbq5XMUMESmiAKITBuDowEPbOzh3ud72NI7xMBI1JG9adcQOweiPou6VA3HL5rNu09fxsuXtXP6EXOZ01xX5ZyLHJoUQKQqBkcDfvF0F09t28PmXUNs3DXI77b3E4RObcpY0t7ErIY0zfVpXnN0B6uWtHHS4jZetmAW9WmNXBKZDhRAZMqMBSG/eW4ntz7yAj97fDvDmSzpGmNxeyNL5zbzwdfM44wj5/LyZe001elPU2S60/9SqRh3p7N3mIe39HHnkzv41dNd9I8GzG5I89ZTFvG2UxZx6tI2DYMVOUgpgMikCkPnrme6uOm+LTy8uZddg2MAtDfVcs4J8zn7+Pm8amWHbqATmQEUQGS/BdmQrv5Rtu8ZYdfAGLsGRtkzkqEuVUNTXZqB0YAb79vEc92DzJ/dwOuOPYyTl7Rx8uJWjlswWzUNkRlGAUQS7R7KcNczXfziqS7WbexhR/9oySm+T1g0m69esIpzT1ygGVpFZjgFEBn34KZebvjfjWzbPcKOPdHcUNnQmdtcxxlHdbBsThML4zmcOlrqmdNSR2tjLWNByHAmSxg6i9sbx9egEJGZTQFExn3xZ0/z+NbdHL+wlZMWt3HeyQt57TGHsWpJW/EZX+unLo8iMn0ogBxCdg9n+PkT2/mv327jpMVt/O0bjx7ftnNglAc29vDh1x3F377pmCrmUkQOFgogM1AmG7Jh5yDP7Ohn066h8Rv1Ht7cx1g2pKG2ht+s38V7zlg+fhf3HU/uIHQ454QFVc69iBwsFECmscHRgK19w4xmQjJhyFgQsnNglK49o+wcGCWTDcmGkA1DeocydPWP0NU/ypaeob0WMepoqWfpnEbeffoy/uDkBTTVpTn7K/fww4c6+cCrjwDgZ49vZ9ncJl62YFa1DldEDjIKINPIWBDy40e28qOHtvJc9wBdCWtWQLR6XV2qJl6vGtqb65jXUs+x82dx9vHzOfrwFlYeNosj5jUXvKv71KVt3HT/Zt7/qhXsGQn4zXM7ed8rV6gDXET2mwJIFWzYOcj1v9nII1v6OOqwFo6dPwt3uO5/NrBt9wgrD2vhNUfPY0VHM0vmNNFYmyKdigLGnOY6Dp/dQFtj7QGtW3HBmqVc+v3HeGBjL529UY3lnBPmT+JRishMpwAyhe7f0MM3736OXz7dRW3KWLWkjbuf6eb7D3YCsGb5HD73Rydy5tHzKl4TeMtJC/jMT57klvs3MzAaMH92Aycvbqvod4rIzKIAMgUe6+zjSz9/hnue6aajpY6/esNK3nX6Ug6b1QBEI6B2D2c4cl7LlOWpqS7NH65aOB683rFmqVbiE5GXpCoBxMz+CfgDYAx4Dnivu/fF2y4D3g9kgY+4++1x+jnAV4EUcK27f74aed8f7s4zOwb41e+6+OXTXdy/oYf2plouP/dY/uz05TTW7T0PVEdLPR0tU38zxTvWLOXG+zYDcPbxar4SkZemWjWQO4DL3D0wsy8AlwEfM7PjgAuA44GFwJ1mlrtZ4RvAG4FO4AEzW+vuT1Yh7/vYM5Lh1oe38tDmPp7fOciG7gH2jARAtFre/z37GC58xTJmNdRWOad7O2FRKycuauWFvmHWrJhT7eyIyEGmKgHE3X+e9/Je4Pz4+XnALe4+Cmwws/XAmnjbend/HsDMbon3rVoAcXce2dLHzfdv5iePbmM4k2VhawNHzGvhvFWLOH7hbM485jDmtzZUK4v75V/fcQr9I0HxO81FRAqYDn0g7wO+Ez9fRBRQcjrjNIAtE9JPK/RhZnYxcDHA0qVLJzWjADv2jPD9Bzv5wUOdPN89SFNdireespB3rlnGiYtbJ/37Km15R3O1syAiB6mKBRAzuxMo1LD+CXe/Nd7nE0AA3DhZ3+vu1wDXAKxevbr41LEvUVf/CGd/5R76hjKsWTGHP3/NEZx74oJp1zQlIjIVKhZA3P2sYtvN7D3AW4A3uHuuoN8KLMnbbXGcRpH0KfPZnz7F0GiWn/7lqzhh0cFX2xARmUxVWbAhHlF1KfCH7j6Ut2ktcIGZ1ZvZCmAlcD/wALDSzFaYWR1RR/vaqczzPc90s/bRF/iLM49U8BARoXp9IF8nmgT8jviGuXvd/UPu/oSZfZeoczwALnH3LICZfRi4nWgY73Xu/sRUZXYkk+WTP36cIzqa+Yszj5yqrxURmdaqNQrrqCLbrgKuKpB+G3BbJfOV5F9/+Sybe4a46YOnaS1vEZGY1hwtIQydb//3Bv7g5IWccWRHtbMjIjJtKICUMJaNplE/dr6mORcRyacAUkIQRgPEalO60U5EJJ8CSAlBNgQgXaOfSkQkn0rFEnIr+6kGIiKyNwWQEoIwroGk9FOJiORTqVhCENdA0ppsUERkLwogJWRyfSBqwhIR2YsCSAm5UVjqRBcR2ZtKxRICdaKLiBSkAFLCeCe6aiAiIntRqVhCbhiv+kBERPamAFJC7kbCWg3jFRHZi0rFEl7sRFcNREQknwJICS8O49VPJSKST6ViCRqFJSJSmAJICRqFJSJSmErFEjSZoohIYQogJWgyRRGRwlQqlpDRZIoiIgUpgJTwYie6fioRkXwqFUt4sQlLNRARkXwKICWMd6JrFJaIyF5UKpYQaD0QEZGCFEBKyE1lklInuojIXhRASlAnuohIYSoVSwjCEDPVQEREJlIAKSGTdXWgi4gUoJKxhCAbqgNdRKQABZASgtB1F7qISAEKICVksqE60EVEClDJWEKQdTVhiYgUoABSQiYMtRaIiEgBKhlLCLKutUBERApQACkhCEOtBSIiUoBKxhIyWY3CEhEpRAGkhECjsEREClLJWEIQahSWiEghCiAlZLKhpjIRESmgKiWjmX3GzB4zs0fM7OdmtjBONzP7mpmtj7efmveei8zs2fhx0VTlVfeBiIgUVq1L639y95PcfRXwU+Af4vQ3Ayvjx8XA1QBmNge4AjgNWANcYWbtU5HRTOiaiVdEpICqBBB335P3shnw+Pl5wA0euRdoM7MFwNnAHe7e4+69wB3AOVORV3Wii4gUlq7WF5vZVcCFwG7gdXHyImBL3m6dcVpSesVlNZmiiEhBFbu0NrM7zezxAo/zANz9E+6+BLgR+PAkfu/FZrbOzNZ1d3cf8OdpMkURkcIqVgNx97P2c9cbgduI+ji2Akvyti2O07YCZ05Ivyvhe68BrgFYvXq1F9rnpdAwXhGRwqo1Cmtl3svzgKfj52uBC+PRWKcDu919G3A78CYza487z98Up1VckHVNpigiUkC1+kA+b2bHACGwCfhQnH4bcC6wHhgC3gvg7j1m9hnggXi/K929ZyoyGjVhqQYiIjJRVQKIu/9xQroDlyRsuw64rpL5KkRNWCIihaltpoRMVuuBiIgUopKxBK0HIiJSmAJICVoPRESksJIlo5mdYGY35O6tMLPrzeykqchctbk7maxTqxsJRUT2UTSAxDf9/Yjonov3xY+7gR/kbgicybJhdBuJaiAiIvsqNQrrSuCN7r4xL+0xM/slcGv8mLGC8QCiGoiIyESlLq3TE4IHAHFabSUyNJ1ksiGA1gMRESmgVMkYmNnSiYlmtgwIKpOl6SPIqgYiIpKkVBPWFcCdZvY54ME4bTXwceBjlczYdJAJoxqIZuMVEdlX0QDi7j82sw3AR4G/jJOfBN7u7o9WOnPV9mINRE1YIiITlZzKJA4UF05BXqad8VFYqoGIiOyj1DDeDjO7wsw+YmYtZnZ1vKbHrWZ21FRlslrGO9FVAxER2UepkvEmoJ5ojfL7gQ3A+UTrmF9b2axVn4bxiogkK9WEdbi7X25mBmxy9y/G6U+bWcFZc2eSXA1EkymKiOyrVMmYhfFp1ndO2BZWJEfTSK4TXZMpiojsq1QN5AgzWwtY3nPi1ysqmrNpIMgN41UfiIjIPkoFkPz5rr40YdvE1zNOJlcD0SgsEZF9lLoP5O6kbWb2HaKJFWcs3QciIpLsQErGV0xaLqap8TvR1QciIrIPXVoXMd6JrlFYIiL7KNqEZWanJm3iEJiNN8iqBiIikqRUJ/o/F9n29GRmZDrKhBrGKyKSpFQn+uumKiPTUaAbCUVEEpWaC+vSvOd/MmHb5yqVqelC64GIiCQrdWl9Qd7zyyZsO2eS8zLt5EZhaTJFEZF9lSoZLeF5odczTq4GktKNhCIi+ygVQDzheaHXM05uNl4N4xUR2VepUVgnm9keotpGY/yc+HVDRXM2DWgYr4hIslKjsFJTlZHpSOuBiIgkU9tMEeMrEqoJS0RkHyoZiwiyTo1BjTrRRUT2oQBSRCYMNROviEgClY5FBFnXWiAiIgkUQIoIsqqBiIgkUelYRCZ0TaQoIpJAAaSIIBtqIkURkQQqHYsIsq57QEREEiiAFBE1YeknEhEpRKVjEVETlmogIiKFKIAUkcm6RmGJiCSoauloZh81Mzezjvi1mdnXzGy9mT2Wvya7mV1kZs/Gj4umIn9BGGoUlohIglKz8VaMmS0B3gRszkt+M7AyfpwGXA2cZmZzgCuA1UTTyD9oZmvdvbeSeQyyrrVAREQSVLMG8mXgUvZeV+Q84AaP3Au0mdkC4GzgDnfviYPGHUzBioiZbKiJFEVEElSldDSz84Ct7v7ohE2LgC15rzvjtKT0Qp99sZmtM7N13d3dB5TPbKhhvCIiSSrWhGVmdwLzC2z6BHA5UfPVpHP3a4BrAFavXn1AqyZmQqdJnegiIgVVLIC4+1mF0s3sRGAF8KiZASwGHjKzNcBWYEne7ovjtK3AmRPS75r0TE8QZENNpigikmDKL6/d/bfufpi7L3f35UTNUae6+3ZgLXBhPBrrdGC3u28DbgfeZGbtZtZOVHu5vdJ51Z3oIiLJqjYKK8FtwLnAemAIeC+Au/eY2WeAB+L9rnT3nkpnRuuBiIgkq3oAiWshuecOXJKw33XAdVOULUDrgYiIFKPL6yK0HoiISDKVjkVoPRARkWQKIEVoPRARkWQqHYvQKCwRkWQKIEVkwlDrgYiIJFDpWESQda0HIiKSQAEkgbsThFoPREQkiUrHBEEYTaOl+0BERApTAEkQZKMAohqIiEhhKh0TZMIQQH0gIiIJFEASZMdrIAogIiKFKIAkGK+BqAlLRKQglY4Jcn0g6kQXESlMASSBOtFFRIpT6Zgg14SlyRRFRApTAEkwXgPRZIoiIgWpdEyQyeY60VUDEREpRAEkwfid6AogIiIFKYAkCHI1EDVhiYgUpNIxQUY3EoqIFKUAkiAYH4Wln0hEpBCVjgleHIWlGoiISCEKIAlyo7BUAxERKUylY4LcKCz1gYiIFKYAkiCjUVgiIkWpdEygPhARkeIUQBJk1YQlIlKUAkiCjIbxiogUpdIxgZqwRESKUwBJ8OJkivqJREQKUemYQJMpiogUpwCSQJMpiogUp9IxQW4yRdVAREQKUwBJEIQhqRrDTAFERKQQBZAEQdY1AktEpAgFkASZrOseEBGRIlRCJgjCUHehi4gUoQCSIJN1jcASESlCJWSCIBtqBJaISBFVCSBm9ikz22pmj8SPc/O2XWZm683sd2Z2dl76OXHaejP7eKXzGISuJiwRkSLSVfzuL7v7l/ITzOw44ALgeGAhcKeZHR1v/gbwRqATeMDM1rr7k5XKXCYbUqsmLBGRRNUMIIWcB9zi7qPABjNbD6yJt6139+cBzOyWeN+KBZAg66Q0jFdEJFE1L7E/bGaPmdl1ZtYepy0CtuTt0xmnJaXvw8wuNrN1Zrauu7u77MxFo7BUAxERSVKxEtLM7jSzxws8zgOuBo4EVgHbgH+erO9192vcfbW7r543b17ZnxOErk50EZEiKtaE5e5n7c9+ZvZt4Kfxy63AkrzNi+M0iqRXhO5EFxEprlqjsBbkvXwb8Hj8fC1wgZnVm9kKYCVwP/AAsNLMVphZHVFH+9pK5jGTVROWiEgx1epE/6KZrQIc2Aj8OYC7P2Fm3yXqHA+AS9w9C2BmHwZuB1LAde7+RCUzGIROQ60CiIhIkqoEEHf/syLbrgKuKpB+G3BbJfOVL8iGpOun2yA1EZHpQ5fYCaLJFNUHIiKSRAEkQRCGmgtLRKQIlZAJgqymMhERKUYBJEEmDLUeiIhIESohE+g+EBGR4hRAEmSyrvtARESKUAmZIAi1HoiISDEKIAkCrUgoIlKUSsgEGa1IKCJSlAJIAq1IKCJSnAJIAe5ONnRSasISEUmkErKAIHQAajWMV0QkkQJIAUE2CiAaxisikkwlZAGZMARQJ7qISBEKIAWM10DUhCUikkgBpIBUjfH7Jy5gxbyWamdFRGTa0opJBbQ21vKNd51a7WyIiExrqoGIiEhZFEBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIiImVRABERkbKYu1c7DxVjZt3ApgP4iA5g5yRl52BxKB4zHJrHfSgeMxyax/0mnpXeAAAFlUlEQVRSj3mZu88rtdOMDiAHyszWufvqaudjKh2KxwyH5nEfiscMh+ZxV+qY1YQlIiJlUQAREZGyKIAUd021M1AFh+Ixw6F53IfiMcOhedwVOWb1gYiISFlUAxERkbIogIiISFkUQAows3PM7Hdmtt7MPl7t/FSKmS0xs1+Z2ZNm9oSZ/VWcPsfM7jCzZ+N/26ud18lmZikze9jMfhq/XmFm98Xn/DtmVlftPE42M2szs++b2dNm9pSZvWKmn2sz+5v4b/txM7vZzBpm4rk2s+vMrMvMHs9LK3huLfK1+PgfM7OyV89TAJnAzFLAN4A3A8cB7zCz46qbq4oJgI+6+3HA6cAl8bF+HPiFu68EfhG/nmn+Cngq7/UXgC+7+1FAL/D+quSqsr4K/MzdjwVOJjr+GXuuzWwR8BFgtbufAKSAC5iZ5/rfgXMmpCWd2zcDK+PHxcDV5X6pAsi+1gDr3f15dx8DbgHOq3KeKsLdt7n7Q/HzfqICZRHR8V4f73Y98Nbq5LAyzGwx8PvAtfFrA14PfD/eZSYecyvwGuDfANx9zN37mOHnmmjZ7kYzSwNNwDZm4Ll293uAngnJSef2POAGj9wLtJnZgnK+VwFkX4uALXmvO+O0Gc3MlgOnAPcBh7v7tnjTduDwKmWrUr4CXAqE8eu5QJ+7B/HrmXjOVwDdwP+Lm+6uNbNmZvC5dvetwJeAzUSBYzfwIDP/XOckndtJK+MUQAQzawF+APy1u+/J3+bROO8ZM9bbzN4CdLn7g9XOyxRLA6cCV7v7KcAgE5qrZuC5bie62l4BLASa2beZ55BQqXOrALKvrcCSvNeL47QZycxqiYLHje7+wzh5R65KG//bVa38VcArgT80s41EzZOvJ+obaIubOWBmnvNOoNPd74tff58ooMzkc30WsMHdu909A/yQ6PzP9HOdk3RuJ62MUwDZ1wPAynikRh1Rp9vaKuepIuK2/38DnnL3f8nbtBa4KH5+EXDrVOetUtz9Mndf7O7Lic7tL939XcCvgPPj3WbUMQO4+3Zgi5kdEye9AXiSGXyuiZquTjezpvhvPXfMM/pc50k6t2uBC+PRWKcDu/Oaul4S3YlegJmdS9ROngKuc/erqpylijCzVwH/DfyWF/sDLifqB/kusJRoOvy3u/vEDrqDnpmdCfydu7/FzI4gqpHMAR4G3u3uo9XM32Qzs1VEAwfqgOeB9xJdRM7Yc21mnwb+lGjE4cPAB4ja+2fUuTazm4EziaZt3wFcAfyYAuc2DqZfJ2rOGwLe6+7ryvpeBRARESmHmrBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMqiACIyTZnZmbnZgkWmIwUQEREpiwKIyAEys3eb2f1m9oiZfStea2TAzL4cr0XxCzObF++7yszujddh+FHeGg1HmdmdZvaomT1kZkfGH9+St4bHjfFNYCLTggKIyAEws5cR3en8SndfBWSBdxFN3LfO3Y8H7ia6MxjgBuBj7n4S0QwAufQbgW+4+8nAGUSzx0I0Q/JfE61NcwTRXE4i00K69C4iUsQbgJcDD8SVg0aiSetC4DvxPv8J/DBek6PN3e+O068Hvmdms4BF7v4jAHcfAYg/735374xfPwIsB35d+cMSKU0BROTAGHC9u1+2V6LZ30/Yr9w5g/LnaMqi/7MyjagJS+TA/AI438wOg/F1qJcR/d/Kzfj6TuDX7r4b6DWzV8fpfwbcHa8G2Wlmb40/o97Mmqb0KETKoKsZkQPg7k+a2SeBn5tZDZABLiFasGlNvK2LqJ8Eomm1vxkHiNyMuBAFk2+Z2ZXxZ/zJFB6GSFk0G69IBZjZgLu3VDsfIpWkJiwRESmLaiAiIlIW1UBERKQsCiAiIlIWBRARESmLAoiIiJRFAURERMry/wGCuVFXyLXDyQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n", + "plt.plot(ELBO)\n", + "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. " + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "arg_params = model.get_params()[0]\n", + "nd_iter_test.reset()\n", + "test_batch = nd_iter_test.next()\n", + "\n", + "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n", + "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n", + "# assert sym.tojson() == output.tojson()\n", + "\n", + "e = y.bind(mx.cpu(), {'data': test_batch.data[0],\n", + " 'encoder_h_weight': arg_params['encoder_h_weight'],\n", + " 'encoder_h_bias': arg_params['encoder_h_bias'],\n", + " 'mu_weight': arg_params['mu_weight'],\n", + " 'mu_bias': arg_params['mu_bias'],\n", + " 'logvar_weight':arg_params['logvar_weight'],\n", + " 'logvar_bias':arg_params['logvar_bias'],\n", + " 'decoder_z_weight':arg_params['decoder_z_weight'],\n", + " 'decoder_z_bias':arg_params['decoder_z_bias'],\n", + " 'decoder_x_weight':arg_params['decoder_x_weight'],\n", + " 'decoder_x_bias':arg_params['decoder_x_bias'], \n", + " 'loss_label':label})\n", + "\n", + "x_fit = e.forward()\n", + "x_construction = x_fit[0].asnumpy()" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmUXWWZ7/HfAxnIPJC6SQgJCRAwYQzUihG4t72SjggyZNnQAkKiMngXAi7TKtpqR1qQq40IF5cNNpNCqyBCEIIKNAi0QFPYQYaQhEwmUCEVMlUChCS894+zo4e8z0l21Rn32d/PWrVy6tnTu8956q03u/azXwshCAAAAMijPerdAAAAAKBeGAwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtxgMAwAAILcYDDcAM/s3M/tavdsBlMPMxppZMLMeJZa/ZGYfrnGzABf9LrKOPrdyjOcMS2a2qejbvpK2SNqefH9hCOGO2rcKeWZmyySdF0J4uN5tScvMxkpaKqlnCGFbfVuDRke/i0ZCn5tv7v8m8iaE0H/H6zQ/EGbWg8RD1pHHqCf6XeQNOdy4uE0iBTP7tpn9wsx+Zmadkj5lZreb2eyidaYmHfqO7/c1s3vMrMPMlprZRbvY/1/2tWM/ZvbVZNvXzexkM/u4mS0ys7Vm9uWibT9kZk+b2Xozazez68ysZ9Hyj5nZQjPbYGb/z8z+08xmFi0/z8xeMbN1ZvagmY2u0NuGKklyYV7ymf/BzA4vWnaZmS02s04ze9nMphctm5l8/teY2ZuSZiexJ83sX5IcWGpmHyvaZpCZ3ZTk1mvJz8KeybI9k+3WmNkSSSftpt3LzGxq8nq2md2V5H6nmb1gZgcleb/azFaY2bSibT9tZvOTdZeY2YU77fvLSRtfT3I6mNmBybLeSTv/bGZvmNm/mlmf8j4FVBv9LhoFfW7z97kMhtObLunfJQ2S9ItdrWhme0i6X9KzkkZJ+ltJXzKz41Mea18VPpt9JP2zpJskfVLSJEkflnS5mY1J1t0m6VJJwyQdK+kESRcm7fgfku6U9KVk+VJJk4va+Ylk2amSWiQ9k5wjGpSZTZJ0swqf8d6SbpB0n5n1TlZZLOl/qpCn35J0u5mNLNrFByUtkTRc0hVFsQUq5Mh3Jd1kZpYsu1WFHDtQhfybJum8ZNn5kj6exFsl/V0XT+dkST+VNETSf0v6rQp5P0rS5cm57bA6OdZASZ+WdI2ZHZW8JydI+qKkqUk7P7zTca6SdJCkI5PloyR9s4ttRX3Q76Ku6HNz0ueGEPgq+pK0TNLUnWLflvQfO8VulzS76PupkpYlr4+VtGSn9b8h6ccljvmXfSX72SRpz+T7IZKCpKOL1n9e0sdL7OsfJN2VvP6MpCeKlpmkdkkzk+8fkjSjaHkPFe7bG1XvzyHvX14eJvEfSfrnnWILJP1Nif3Mk3Rq8nqmpD/vtHympFeLvu+b5NsIFTrvLZL6FC0/U9Kjyev/kPS5omXTkm177O6cJM2W9FDRspN3yvsByb4Gl9jXvZIuTV7fLOk7RcsOTLY9MMn5zZIOKFr+IUlL6/0Z87XrfKff5aveOZjE6XND8/e53DOc3oourLufpDFmtr4otqekx1JuvyaEsKOQ5O3k3zeKlr8tqb8kmdkHJF0t6WgVfqh6qHClQSpc4fhLu0MIwcxW7tTOH5rZtUWx91S4QvJayraitvaTNMPMLi6K9VLhs5aZnavC/9jHJsv6q3D1YQcvj1fteBFCeCu5QNFf0lBJPSW1//WihfYo2sf78kvS8i6ey8457eV9f0nrkz8j/pMKVxv2UCHXXyhqR1vRvorb1JKs+1zROZgKP49ofPS7qDf63Bz0uQyG09v5sRubVfjAdxhR9HqFpEUhhAlVb1XhzxpPS/r7EMImM/sHFf60IRWuRhTfA2Qq/LmiuJ3fCCHs8s+PaCgrJF0RQrhi5wVmtp+kH0s6XtJTIYTtZjZPhY5oh648PmaFClcphgW/6KNdUvG9jmOcdcqW/DnybknnSpoTQthqZvfqr+fVrsJAYofiNq1RoZM/JITAQCN76HdRb/S5OehzuWe4++ZJOsnMhiT3B11StOwpSe+a2Swz2yu56f0wMzu6Cu0YIGmDpM1mNkHJfWuJ+yUdZYVCkB4q3OPWUrT8XyX9Y7KdzGywmXX1HiRUT88kf3Z89VCh4/2cmX3QCvqZ2UlmNkBSPxU63g6pUAAh6dDuHjyE0C7pd5KuNrOBZraHmR1gZn+TrHKnpEusULQ0RNJlZZzrrvSS1FuF89qWXLGYVrT8TkmfNrMJZtZXhT+N7ziH91R4z65J7uWUmY0ys49Wqa2oLvpdVBN9bkHu+lwGw913q6T5KvyZ4jeSfr5jQfI/uhNVKJpYpsL/lG5Q4Ub0SpslaYakzuQYf7naEEJ4Q9LfS/q+pDclHaDCTfNbkuV3JcvuMrONkv4kqaETNmfmqvA/7B1fs0MIbSoUUVwvaZ2kV1W4B00hhJdV+NPtUyr8OewwSf9ZZhvOVaFjfDk53i8l7SgO+bEKBRjPS/qjpF+VeSxXCKFThUHPnUkbzpJ0X9HyByVdJ+lRFd6Pp5NFW5J/v7IjnuT5w5IOrkZbUXW3in4X1UOfq3z2uUy6kSNWeDzL65L+LoTwRL3bA1RDcsXtRUm9S/ypEagZ+l00u2boc7ky3OTM7ITkz3C9VfhTxlZJ/1XnZgEVZWbTrfBsyyGS/q+kX2e1U0b20e+i2TVbn8tguPkdp8IzDjtU+FPc9BDCll1vAmTOhSo8F3OxClP6/p/6Ngc5R7+LZtdUfS63SQAAACC3uDIMAACA3CrrOcNWmJLvWhUepvxvIYSrdrX+sGHDwtixY8s5JKDnnntuTQihZfdrVg65i3ItW7ZMa9assd2vWTnkLSqBPhdZlTZ3uz0YTipkf6jC/O8rJT1rZvcljxpxjR07Vm1tbaUWA6mYWVdn3SkbuYtytba21vyY5C0qgT4XWZU2d8u5TWKyCvNrLwkhvKvC8x5PLWN/AAAAQE2VMxgepffPR71S759yUpJkZheYWZuZtXV0dJRxOKC2yF1kEXmLrCJ3US9VL6ALIdwYQmgNIbS2tNT0liOgLOQusoi8RVaRu6iXcgbDr0kaXfT9vkkMAAAAyIRyBsPPShpvZuPMrJekT6po7moAAACg0XX7aRIhhG1m9nlJv1Xh0Wo3hxBeqljLAAAAgCor6znDIYS5kuZWqC0AAABATTEDHQAAAHKrrCvDAAAA+KsQQr2b8D5mNZ34MpO4MgwAAIDcYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3OJpEgAAAN1QjSdH8PSH2uPKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAjoAAIBdqOUUy2mPRaFd5XBlGAAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlVVgGdmS2T1Clpu6RtIYTWSjQKQP7UskCl0ihkAbIpbb9Taj0vvn379lSxUrz+pEePeLjmrefFSrX9vffeS7XuHnvE1029WFf6wUbrMyvxNIn/HUJYU4H9AAAAADXFbRIAAADIrXIHw0HS78zsOTO7wFvBzC4wszYza+vo6CjzcEDtkLvIIvIWWUXuol7KHQwfF0I4StLHJF1kZv9r5xVCCDeGEFpDCK0tLS1lHg6oHXIXWUTeIqvIXdRLWfcMhxBeS/5dbWb3SJos6fFKNAxoduUUjHWlmMMrkvBUo6DBa081iizqXbhR7nki5r2nb775ZhSbM2eOu/0ll1wSxd56661Ux+7Xr18Uu+6669x1zz333CjmFTuh/rycShsrVQC3devWKLZhw4ZUsVL9eO/evVPFevXqFcW8fqfU7wAv3rNnzyjWt2/fVMf2iupKtanRdPvKsJn1M7MBO15LmibpxUo1DAAAAKi2cv77OlzSPcmIv4ekfw8h/KYirQIAAABqoNuD4RDCEklHVLAtAAAAQE3xaDUAAADkVm7u8n/66aej2LXXXuuuO2rUqCjWp0+fKDZjxowoNnToUHefpeLIB69QwYt5RRpbtmyJYp2dne5xNm3alGr7rhSIpG3TO++8E8XefffdKLbnnnu6x/EKMrxijr322iuKjRgxIooNGzbMPY63z7QzOaE6lixZEsW+/e1vR7HbbrutrOOUKvDZ2dtvvx3Fzj//fHfdhQsXRrErr7yy28dGZaQtjEvbN3v9myStWLEiij377LNRbOXKlVHM6/MkaeDAgVFs7733jmLeuMIr3vT6Ycnv271+c/To0VFs8ODBUSzLOZ7dlgMAAABlYjAMAACA3GIwDAAAgNxiMAwAAIDcYjAMAACA3MrN0yS8Jz8sWrSorH1eccUVUWzQoEHuulOmTCnrWLUwduzYKPbVr37VXXfMmDFVbk02lZr2ctu2bVHMm8Zz3bp1UeyVV16JYk8++aR7nD//+c9RbOPGjamOU2qq2s2bN6eKlTPVreRXUHtPfvB+xiZPnhzFZs6c6R7Hy12mz62NUtPPfulLX4pi9957b8WP7z3JxIuVqr73fO9734tixxxzTBQ75ZRTUu8T6XVlavq0T5Pwnpbj9a2SdOedd0axF154IYp5fZn3O1fyn9Tg9ZvePr0noZRqu/d7YMiQIVHMe+qF155S/WgWnszDlWEAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkVm6qRrxijHnz5rnrHnLIIVHspZdeimLPPPNMFJszZ467z9/+9rdRbNy4cVFs6dKl7vZpeTewjxw5Mop5U0h6St3g/5WvfKVL7cqLUsUc3rSXXsGZV+jwu9/9Loo9+uij7nHa29tTHSftVNCleNt75967d+8o5k2nLElr166NYhs2bIhiXju9IsHjjjvOPY433ToFdJXn5cMNN9zgrpu2WK5Pnz5R7NBDD3XXnTVrVhT7yEc+EsW8aW4vu+yyKOYVypVyxx13RLGPfvSjUcz7+UD1eP2WV9zsFZbNnTvX3ef9998fxbwitkmTJkWxAw880N2nl9NeUZ1X2Ob1owsWLHCP4z1EwOufW1paotg+++yTatus4MowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcmu3VSNmdrOkj0taHUI4NIkNlfQLSWMlLZN0RgghvuO8gUyYMCFVrJTDDz88ip155plR7KqrrnK3X7ZsWRTzCuiWLFmSuk0e74Z6r4DOO3ZHR0cU+8AHPlBWe1CaV8yxxx7x/0+9Ipthw4a5+/SKlrziMK8YwyuSkPwCI2/WLm+WIW+2OG+GI8kvaH3ggQeimFcgkrZIsFqyMMNSrXnv/0UXXZR6ey/HvvnNb0axL3/5y11rWAqzZ8+OYt5MY5K0fPnyKPbLX/4yinkz0J199tldbxxSSTvbnDfboPf7+vHHH3eP4xU9jx49OopNnDgxinkzZ0p+/963b98o5vU7XSmEXr16dRTzZt8bMWJEFPMKAgcMGOAex/u91mh9Zporw7dKOmGn2GWSHgkhjJf0SPI9AAAAkCm7HQyHEB6XtPOlmFMl3Za8vk3SaRVuFwAAAFB13b1neHgIYccDTVdJGl5qRTO7wMzazKzN+zM80KjIXWQReYusIndRL2UX0IXCjTn+TAOF5TeGEFpDCK2l7kkEGhG5iywib5FV5C7qpbvTLr1hZiNDCO1mNlJSfBd2TpWagSVtIVpXivrS8mbKW7NmTRT74Ac/GMWmTZtW8fbkkVdA4BVEeLOjHX/88anWk6TNmzdHMa8Yw8vHUr98vHam5RXvebMzSdLWrVuj2G9+85so5r2Xw4fHf5zyij6k9MV/qK/LL788ilWjWM7j9eN33323u25ra2uqfV555ZVR7NRTT41i/fv3T7U/7Fran2mv3/EK2UvNDtuzZ88otv/++0exD33oQ1HM67ckPwe8fittsZxXFCdJ69evj2LerJ/eeKGzszOKlfod4v0e8Aoc69kPd/fK8H2SZiSvZ0jy5yAGAAAAGthuB8Nm9jNJT0k62MxWmtlnJV0l6W/NbJGkqcn3AAAAQKbs9jaJEEL8MN2C+G+3AAAAQIYwAx0AAAByi8EwAAAAcqu7T5NAg/KeJjB9+vQo5k1L+YMf/CCK9enTpzINywnvSQeSXwnsTZ3tTX3sPc1hzJgx7nG86uKBAwdGMW/aTK/iV0pf4Zt2+tNSlc0rV66MYl4+e+/HUUcdFcW8acgl/zx5mkTlvfjii6nX9abovvjiiyvZnC7xcvSss84qa5+vvPJKFPvud78bxbynaKB6vD7Km/J927Zt7vZeP+7lsxfr3bu3u89Sv0fStGnjxo1RzHtqhCStW7cuinlT23uxd955J4qVerqF97vBi3lq1TdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5RQFdk7n11luj2KpVq6KYV6i13377VaNJuVLqZv+0BRFeoZ0XK1XY6BUleAVj5U5JnLb4wSuoKDWt6R/+8Ico9u6770axww8/PIqdeOKJUWzw4MHucSiWq42XX3459br1LGr0iotmzpwZxRYuXFjxY3v99ezZs9110/YhedOVPPHW9fpCr6jOW0/y+7hNmzZFMe/3sFfcLPn9u9cmb5rk+fPnR7H29nb3OB0dHVHM69u9Yjmvby5VZOi9R+X+fFe6f+CnCwAAALnFYBgAAAC5xWAYAAAAucVgGAAAALlFAV1GLV682I1/8YtfTLX9U089FcVGjBhRVptQmnezf9qCGG+9UsUcaY9dDV6Bx4YNG6LYnDlz3O2XLFkSxbwCE69Y7uCDD45iPXv2dI+D2mhtbU29rlfI85Of/CSKfe5zn0u9T68QyCs4+vrXvx7Ffv3rX6c+TjnOOOOMKEaBZ/V4763XTwwbNiyKlSp282are+mll6LYgw8+GMVeffVVd5977bVXFNu6dWsU6+zsjGLez9Ly5cvd43jFo9579Pbbb0cxb3bQUrOLejPteb/XvBgz0AEAAABVxmAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjO7WdLHJa0OIRyaxGZLOl/Sjju1vxZCmFutRiJWqsDDu8n+9NNPj2L7779/xduErimnMKBUAV3ameEqva3kzz70wgsvRLHHHnvM3d6bpWjy5MlR7OSTT45igwYNimKlChQpUKqNcvuYW265JYqNHz8+ih1wwAHu9tdff30Uu+aaa8pqUzm8HD3nnHOiGPlZPV6f0KtXryh26KGHpopJfgGdV6jp9XsLFixw9zlgwIAo5hWh9e3bN4p551NqZjhvZjkv/7y+2Suq88Yfkl9cXe7vm0pLc2X4VkknOPFrQghHJl8MhAEAAJA5ux0MhxAelxT/1wcAAADIuHLuGf68mf3JzG42syGlVjKzC8yszczavOffAY2K3EUWkbfIKnIX9dLdwfCPJB0g6UhJ7ZKuLrViCOHGEEJrCKG1paWlm4cDao/cRRaRt8gqchf10q0Z6EIIb+x4bWY/lnR/xVqEiHdT+j333OOu691k/53vfCeKdWUGM5TPK0rwCgjSFs+UKj5Ie5y025ba3iuIWLVqVRS7/fbbo9iKFSvc44wcOTKKfeITn4hi48aNi2LeLFIUItWXV6x09dX+dZNZs2ZFsba2tig2bdq08hvWTUcccYQbf/7551NtP3Xq1NT7RPnSzvrZo0c8DBo7dmwUO+2009zjeL9zX3nllSjmFat5hWml2uT952D06NGp2vPmm2+6x0k745s3I16fPn2iWJbHFd26Mmxmxb+1pkt6sTLNAQAAAGonzaPVfibpw5KGmdlKSf8k6cNmdqSkIGmZpAur2EYAAACgKnY7GA4hnOmEb6pCWwAAAICaYgY6AAAA5BaDYQAAAORWt54mgdq66ab4rpQnnnjCXfess86KYky93JjKedpBV578UM56pdZ96623otgdd9wRxR5++OHUxzn++OOjmFeB701BmrYqGrXjvf+XXHKJu6431e3Pf/7zVDFvWthSxx8+fHgU86b87spTUA455BA3vrNLL7001XqoHi8nvCcgDBw4MIpNmTLF3aeXU4sWLYpi3hMdSvVRw4YNi2LeU3S89To7O6PY0qVL3eP069cvinnvx5Ah8VQS3nvkPdWn1D4brX/myjAAAAByi8EwAAAAcovBMAAAAHKLwTAAAAByiwK6BjNv3rwodvHFF0exwYMHu9tffvnlFW8Tmk9Xpm3etm1bFHvuueeimFdAt379+ih29NFHu8c555xzopg3BWkWijHg8wodJb9Q0ot94xvfiGKvvfaau09vSluvWC6tMWPGuPEDDzwwim3YsCGKHXbYYd0+NqrHy0mvEGzQoEHu9hMnToxi3nTOW7ZsiWLetPaSP/1x2sJhbz2vAE7yC+i86Zy9ftgroPO2LdXORit65sowAAAAcovBMAAAAHKLwTAAAAByi8EwAAAAcosCujryZk4688wzo9j27duj2Nlnn+3uk9nm8i1tAYJXLFeqmGPVqlVR7Prrr49i3ixHXiHI9OnT3eMcdNBBUcwrgvLOkQK6fNhvv/1SxaqhVIGp1z97RUxewREak9efeH2RlL4Ar1T/mvb43nHS7tMr3it1HC9PvVifPn2iWLkz0NWzb+fKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHJrtwV0ZjZa0k8kDZcUJN0YQrjWzIZK+oWksZKWSTojhLCuek3NNu9G95NOOimKLViwIIpNmDAhin3rW9+qTMOQS14xkFfQKUl33XVXFPv9738fxbZu3RrFvNnmTjnlFPc43uxF5RQEUlSHSlq8eLEb9wpHb7jhhmo3BzVWqj9JO5NaVwro0vL6vc2bN0exdev8oZlX8Ja2WC5tcXOpeKP1z2muDG+TNCuEMFHSFEkXmdlESZdJeiSEMF7SI8n3AAAAQGbsdjAcQmgPIfwxed0pab6kUZJOlXRbstptkk6rViMBAACAaujSPcNmNlbSJEnPSBoeQmhPFq1S4TYKb5sLzKzNzNo6OjrKaCpQW+Qusoi8RVaRu6iX1INhM+sv6W5JXwghbCxeFgo3rrhPJA8h3BhCaA0htLa0tJTVWKCWyF1kEXmLrCJ3US+pZqAzs54qDITvCCH8Kgm/YWYjQwjtZjZS0upqNbIZrF27Noo99thjqbb96U9/GsWGDh1abpOQE16RhVfsNn/+fHf7uXPnRrFNmzZFscGDB0exT33qU1Fsn332cY+TdpYioB66UhT3wAMPRLHzzjuvks1Bg0jbR3mFdl3hFeB5sXfeeSeKlWqjV7Ts9cOetIWDu4p3d71q2O2nY4XW3SRpfgjh+0WL7pM0I3k9Q9KcyjcPAAAAqJ40V4aPlXSOpBfMbF4S+5qkqyTdaWaflbRc0hnVaSIAAABQHbsdDIcQnpRU6tr18ZVtDgAAAFA7zEAHAACA3GIwDAAAgNxK9TQJpLdhwwY3PmXKlFTb33777VFs0qRJZbUJ+eE9OWL79u1RzMvTxx9/3N3nihUrolj//v2j2LRp06KYN/Vyr1693ONUupLYey9K4akVqKQnn3wyiq1fvz6KeU9gQfaV25+UM7289zSIUk+I8J5G4U2z7D05oiv9aznnUytcGQYAAEBuMRgGAABAbjEYBgAAQG4xGAYAAEBuUUBXYbfccosbX7JkSartjzvuuCjWaDeao/5KFS94BRFbtmyJYsuXL49iCxcudPfpTdl52GGHRbHTTz89ig0ZMiSKlTstaTWKMbJQ4IHsWLt2bRTzfr4mT55ci+agAZTbn3h9u9dvbd26NYqV6nP33nvvKDZw4MAo5hXgecfuSlFdo+HKMAAAAHKLwTAAAAByi8EwAAAAcovBMAAAAHKLAroyLFq0KIrNnj279g1BJqUtNuhKoYJXPOHNfPX8889HsXXr1rn79IrgjjjiiCh28MEHR7GuzFzkFZhQ2AYA6fXs2TOKjRkzxl1348aNUaylpSWKjRs3Lop5hdWlCvWy0GdzZRgAAAC5xWAYAAAAucVgGAAAALnFYBgAAAC5tdsCOjMbLeknkoZLCpJuDCFca2azJZ0vqSNZ9WshhLnVamgjeuKJJ6KYd0N6KRMmTIhiffr0KatNyI60BWNebNu2be4+N2/eHMVWrFgRxRYvXhzFNm3a5O6zb9++Ucybucgrnkg7a1K9ZaHAA/U1ffp0N37PPfdEsbfeeiuK7bvvvhVvE/LD66N69IiHcMOHD49iU6dOdfd51FFHRbH+/ftHsdGjR0exAQMGRDFvpjrJb3uj9blpniaxTdKsEMIfzWyApOfM7KFk2TUhhH+pXvMAAACA6tntYDiE0C6pPXndaWbzJY2qdsMAAACAauvSPcNmNlbSJEnPJKHPm9mfzOxmM4sfRlrY5gIzazOzto6ODm8VoCGRu8gi8hZZRe6iXlIPhs2sv6S7JX0hhLBR0o8kHSDpSBWuHF/tbRdCuDGE0BpCaPUe5gw0KnIXWUTeIqvIXdRLqsGwmfVUYSB8RwjhV5IUQngjhLA9hPCepB9Lmly9ZgIAAACVl+ZpEibpJknzQwjfL4qPTO4nlqTpkl6sThObwzHHHBPFHnrooSjG0ySwM6/qttS0l95UnN50ygcddFDq4/fr1y+KHXvssVFs8ODBUaxUOz1pp14ud4rmRqtiRjaUqsh//fXXa9wS5JHXb3lPb/Ce/rP//vu7+/SmWfaeUOEdx4uV6luz0OemeZrEsZLOkfSCmc1LYl+TdKaZHanC49aWSbqwKi0EAAAAqiTN0ySelOQN63P1TGEAAAA0H2agAwAAQG4xGAYAAEBupblnGCV85jOfSRUD0ko7bWWpggRvikyvWG78+PFRzJs6WUo/fXK5bS93ewBoVmkL6EpNiVzpYzcbrgwDAAAgtxgMAwAAILcYDAMAACC3GAwDAAAgtyxtcUxFDmbWIWl58u0wSWtqdvDq43xqZ78QQk0nri/K3UZ+X7qD86mdeuat1NjvTXdwPrVD7lYW51M7qXK3poPh9x3YrC2E0FqXg1cB55MPzfa+cD750WzvDeeTH8323nA+jYfbJAAAAJBbDIYBAACQW/UcDN9Yx2NXA+eTD832vnA++dFs7w3nkx/N9t5wPg2mbvcMAwAAAPXGbRIAAADILQbDAAAAyK2aD4Zu3yMbAAACn0lEQVTN7AQzW2Bmr5rZZbU+frnM7GYzW21mLxbFhprZQ2a2KPl3SD3b2BVmNtrMHjWzl83sJTO7NIln9pyqhdxtLORuOlnPW6m5cpe8TS/rudtMeSs1d+7WdDBsZntK+qGkj0maKOlMM5tYyzZUwK2STtgpdpmkR0II4yU9knyfFdskzQohTJQ0RdJFyWeS5XOqOHK3IZG7u9EkeSs1V+6Styk0Se7equbJW6mJc7fWV4YnS3o1hLAkhPCupJ9LOrXGbShLCOFxSWt3Cp8q6bbk9W2STqtpo8oQQmgPIfwxed0pab6kUcrwOVUJudtgyN1UMp+3UnPlLnmbWuZzt5nyVmru3K31YHiUpBVF369MYlk3PITQnrxeJWl4PRvTXWY2VtIkSc+oSc6pgsjdBkbultSseSs1wedM3u5Ss+ZuU3zOzZa7FNBVWCg8qy5zz6szs/6S7pb0hRDCxuJlWT0ndE1WP2dyF1n8nMlbZPVzbsbcrfVg+DVJo4u+3zeJZd0bZjZSkpJ/V9e5PV1iZj1VSOw7Qgi/SsKZPqcqIHcbELm7W82at1KGP2fyNpVmzd1Mf87Nmru1Hgw/K2m8mY0zs16SPinpvhq3oRrukzQjeT1D0pw6tqVLzMwk3SRpfgjh+0WLMntOVULuNhhyN5VmzVspo58zeZtas+ZuZj/nps7dEEJNvySdKGmhpMWS/rHWx69A+38mqV3SVhXuYfqspL1VqKBcJOlhSUPr3c4unM9xKvxJ40+S5iVfJ2b5nKr4XpG7DfRF7qZ+nzKdt8k5NE3ukrddeq8ynbvNlLfJ+TRt7jIdMwAAAHKLAjoAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADkFoNhAAAA5BaDYQAAAOQWg2EAAADk1v8HLgAFn4S3bUIAAAAASUVORK5CYII=\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# learning images on the test set\n", + "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n", + "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax1.set_title('True image')\n", + "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax2.set_title('Learned image')\n", + "ax3.imshow(np.reshape(image_test[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax3.set_title('True image')\n", + "ax4.imshow(np.reshape(x_construction[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax4.set_title('Learned image')\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[('loss', 140.17346005859375)]" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# calculate the ELBO which is minus the loss for test set\n", + "metric = mx.metric.Loss()\n", + "model.score(nd_iter_test, metric)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 4. All together: MXNet-based class VAE" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "from VAE import VAE" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "One can directly call the class `VAE` to do the training:\n", + "\n", + "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n", + "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n", + "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n", + "\n", + "The outputs are the learned model and training loss." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Epoch[0] Train-loss=383.478870\n", + "INFO:root:Epoch[0] Time cost=5.075\n", + "INFO:root:Epoch[1] Train-loss=211.923867\n", + "INFO:root:Epoch[1] Time cost=4.741\n", + "INFO:root:Epoch[2] Train-loss=206.789445\n", + "INFO:root:Epoch[2] Time cost=4.601\n", + "INFO:root:Epoch[3] Train-loss=204.428186\n", + "INFO:root:Epoch[3] Time cost=4.865\n", + "INFO:root:Epoch[4] Train-loss=202.417322\n", + "INFO:root:Epoch[4] Time cost=4.606\n", + "INFO:root:Epoch[5] Train-loss=200.635136\n", + "INFO:root:Epoch[5] Time cost=4.711\n", + "INFO:root:Epoch[6] Train-loss=199.009614\n", + "INFO:root:Epoch[6] Time cost=5.159\n", + "INFO:root:Epoch[7] Train-loss=197.565788\n", + "INFO:root:Epoch[7] Time cost=4.588\n", + "INFO:root:Epoch[8] Train-loss=196.524507\n", + "INFO:root:Epoch[8] Time cost=4.905\n", + "INFO:root:Epoch[9] Train-loss=195.725745\n", + "INFO:root:Epoch[9] Time cost=4.426\n", + "INFO:root:Epoch[10] Train-loss=194.902025\n", + "INFO:root:Epoch[10] Time cost=4.685\n", + "INFO:root:Epoch[11] Train-loss=194.026873\n", + "INFO:root:Epoch[11] Time cost=4.622\n", + "INFO:root:Epoch[12] Train-loss=193.350646\n", + "INFO:root:Epoch[12] Time cost=4.712\n", + "INFO:root:Epoch[13] Train-loss=192.737502\n", + "INFO:root:Epoch[13] Time cost=4.618\n", + "INFO:root:Epoch[14] Train-loss=192.338165\n", + "INFO:root:Epoch[14] Time cost=4.763\n", + "INFO:root:Epoch[15] Train-loss=191.888625\n", + "INFO:root:Epoch[15] Time cost=5.168\n", + "INFO:root:Epoch[16] Train-loss=191.170650\n", + "INFO:root:Epoch[16] Time cost=4.809\n", + "INFO:root:Epoch[17] Train-loss=190.307264\n", + "INFO:root:Epoch[17] Time cost=4.622\n", + "INFO:root:Epoch[18] Train-loss=188.988063\n", + "INFO:root:Epoch[18] Time cost=4.543\n", + "INFO:root:Epoch[19] Train-loss=187.616311\n", + "INFO:root:Epoch[19] Time cost=5.154\n", + "INFO:root:Epoch[20] Train-loss=186.352783\n", + "INFO:root:Epoch[20] Time cost=4.661\n", + "INFO:root:Epoch[21] Train-loss=185.428020\n", + "INFO:root:Epoch[21] Time cost=5.193\n", + "INFO:root:Epoch[22] Train-loss=184.543097\n", + "INFO:root:Epoch[22] Time cost=4.519\n", + "INFO:root:Epoch[23] Train-loss=184.029907\n", + "INFO:root:Epoch[23] Time cost=4.732\n", + "INFO:root:Epoch[24] Train-loss=183.643270\n", + "INFO:root:Epoch[24] Time cost=5.011\n", + "INFO:root:Epoch[25] Train-loss=183.246912\n", + "INFO:root:Epoch[25] Time cost=4.706\n", + "INFO:root:Epoch[26] Train-loss=183.065233\n", + "INFO:root:Epoch[26] Time cost=4.673\n", + "INFO:root:Epoch[27] Train-loss=182.680542\n", + "INFO:root:Epoch[27] Time cost=4.628\n", + "INFO:root:Epoch[28] Train-loss=182.428677\n", + "INFO:root:Epoch[28] Time cost=4.772\n", + "INFO:root:Epoch[29] Train-loss=182.219946\n", + "INFO:root:Epoch[29] Time cost=4.571\n", + "INFO:root:Epoch[30] Train-loss=182.070927\n", + "INFO:root:Epoch[30] Time cost=4.603\n", + "INFO:root:Epoch[31] Train-loss=181.837968\n", + "INFO:root:Epoch[31] Time cost=4.559\n", + "INFO:root:Epoch[32] Train-loss=181.624303\n", + "INFO:root:Epoch[32] Time cost=5.069\n", + "INFO:root:Epoch[33] Train-loss=181.534547\n", + "INFO:root:Epoch[33] Time cost=4.654\n", + "INFO:root:Epoch[34] Train-loss=181.239556\n", + "INFO:root:Epoch[34] Time cost=4.776\n", + "INFO:root:Epoch[35] Train-loss=181.098188\n", + "INFO:root:Epoch[35] Time cost=4.571\n", + "INFO:root:Epoch[36] Train-loss=180.820560\n", + "INFO:root:Epoch[36] Time cost=4.815\n", + "INFO:root:Epoch[37] Train-loss=180.828095\n", + "INFO:root:Epoch[37] Time cost=4.455\n", + "INFO:root:Epoch[38] Train-loss=180.495569\n", + "INFO:root:Epoch[38] Time cost=5.096\n", + "INFO:root:Epoch[39] Train-loss=180.389106\n", + "INFO:root:Epoch[39] Time cost=4.797\n", + "INFO:root:Epoch[40] Train-loss=180.200965\n", + "INFO:root:Epoch[40] Time cost=5.054\n", + "INFO:root:Epoch[41] Train-loss=179.851014\n", + "INFO:root:Epoch[41] Time cost=4.642\n", + "INFO:root:Epoch[42] Train-loss=179.719933\n", + "INFO:root:Epoch[42] Time cost=4.603\n", + "INFO:root:Epoch[43] Train-loss=179.431740\n", + "INFO:root:Epoch[43] Time cost=4.341\n", + "INFO:root:Epoch[44] Train-loss=179.235384\n", + "INFO:root:Epoch[44] Time cost=4.638\n", + "INFO:root:Epoch[45] Train-loss=179.108771\n", + "INFO:root:Epoch[45] Time cost=4.754\n", + "INFO:root:Epoch[46] Train-loss=178.714163\n", + "INFO:root:Epoch[46] Time cost=4.457\n", + "INFO:root:Epoch[47] Train-loss=178.508338\n", + "INFO:root:Epoch[47] Time cost=4.960\n", + "INFO:root:Epoch[48] Train-loss=178.288002\n", + "INFO:root:Epoch[48] Time cost=4.562\n", + "INFO:root:Epoch[49] Train-loss=178.083288\n", + "INFO:root:Epoch[49] Time cost=4.619\n", + "INFO:root:Epoch[50] Train-loss=177.791330\n", + "INFO:root:Epoch[50] Time cost=4.580\n", + "INFO:root:Epoch[51] Train-loss=177.570741\n", + "INFO:root:Epoch[51] Time cost=4.704\n", + "INFO:root:Epoch[52] Train-loss=177.287114\n", + "INFO:root:Epoch[52] Time cost=5.172\n", + "INFO:root:Epoch[53] Train-loss=177.122645\n", + "INFO:root:Epoch[53] Time cost=4.678\n", + "INFO:root:Epoch[54] Train-loss=176.816022\n", + "INFO:root:Epoch[54] Time cost=4.819\n", + "INFO:root:Epoch[55] Train-loss=176.670484\n", + "INFO:root:Epoch[55] Time cost=4.568\n", + "INFO:root:Epoch[56] Train-loss=176.459671\n", + "INFO:root:Epoch[56] Time cost=4.450\n", + "INFO:root:Epoch[57] Train-loss=176.174175\n", + "INFO:root:Epoch[57] Time cost=4.579\n", + "INFO:root:Epoch[58] Train-loss=175.935856\n", + "INFO:root:Epoch[58] Time cost=4.552\n", + "INFO:root:Epoch[59] Train-loss=175.739928\n", + "INFO:root:Epoch[59] Time cost=4.385\n", + "INFO:root:Epoch[60] Train-loss=175.579695\n", + "INFO:root:Epoch[60] Time cost=4.496\n", + "INFO:root:Epoch[61] Train-loss=175.403871\n", + "INFO:root:Epoch[61] Time cost=5.088\n", + "INFO:root:Epoch[62] Train-loss=175.157114\n", + "INFO:root:Epoch[62] Time cost=4.628\n", + "INFO:root:Epoch[63] Train-loss=174.953950\n", + "INFO:root:Epoch[63] Time cost=4.826\n", + "INFO:root:Epoch[64] Train-loss=174.743393\n", + "INFO:root:Epoch[64] Time cost=4.832\n", + "INFO:root:Epoch[65] Train-loss=174.554056\n", + "INFO:root:Epoch[65] Time cost=4.375\n", + "INFO:root:Epoch[66] Train-loss=174.366719\n", + "INFO:root:Epoch[66] Time cost=4.583\n", + "INFO:root:Epoch[67] Train-loss=174.160622\n", + "INFO:root:Epoch[67] Time cost=4.586\n", + "INFO:root:Epoch[68] Train-loss=173.981699\n", + "INFO:root:Epoch[68] Time cost=5.149\n", + "INFO:root:Epoch[69] Train-loss=173.751617\n", + "INFO:root:Epoch[69] Time cost=4.495\n", + "INFO:root:Epoch[70] Train-loss=173.548732\n", + "INFO:root:Epoch[70] Time cost=4.588\n", + "INFO:root:Epoch[71] Train-loss=173.380950\n", + "INFO:root:Epoch[71] Time cost=5.042\n", + "INFO:root:Epoch[72] Train-loss=173.158519\n", + "INFO:root:Epoch[72] Time cost=4.817\n", + "INFO:root:Epoch[73] Train-loss=172.970726\n", + "INFO:root:Epoch[73] Time cost=4.791\n", + "INFO:root:Epoch[74] Train-loss=172.782357\n", + "INFO:root:Epoch[74] Time cost=4.377\n", + "INFO:root:Epoch[75] Train-loss=172.581992\n", + "INFO:root:Epoch[75] Time cost=4.518\n", + "INFO:root:Epoch[76] Train-loss=172.385020\n", + "INFO:root:Epoch[76] Time cost=4.863\n", + "INFO:root:Epoch[77] Train-loss=172.198309\n", + "INFO:root:Epoch[77] Time cost=5.104\n", + "INFO:root:Epoch[78] Train-loss=172.022333\n", + "INFO:root:Epoch[78] Time cost=4.571\n", + "INFO:root:Epoch[79] Train-loss=171.816585\n", + "INFO:root:Epoch[79] Time cost=4.557\n", + "INFO:root:Epoch[80] Train-loss=171.643714\n", + "INFO:root:Epoch[80] Time cost=4.567\n", + "INFO:root:Epoch[81] Train-loss=171.460581\n", + "INFO:root:Epoch[81] Time cost=4.735\n", + "INFO:root:Epoch[82] Train-loss=171.284854\n", + "INFO:root:Epoch[82] Time cost=5.012\n", + "INFO:root:Epoch[83] Train-loss=171.113129\n", + "INFO:root:Epoch[83] Time cost=4.877\n", + "INFO:root:Epoch[84] Train-loss=170.947790\n", + "INFO:root:Epoch[84] Time cost=4.487\n", + "INFO:root:Epoch[85] Train-loss=170.766223\n", + "INFO:root:Epoch[85] Time cost=4.723\n", + "INFO:root:Epoch[86] Train-loss=170.602559\n", + "INFO:root:Epoch[86] Time cost=4.803\n", + "INFO:root:Epoch[87] Train-loss=170.448713\n", + "INFO:root:Epoch[87] Time cost=4.636\n", + "INFO:root:Epoch[88] Train-loss=170.273053\n", + "INFO:root:Epoch[88] Time cost=4.562\n", + "INFO:root:Epoch[89] Train-loss=170.099485\n", + "INFO:root:Epoch[89] Time cost=4.567\n", + "INFO:root:Epoch[90] Train-loss=169.934289\n", + "INFO:root:Epoch[90] Time cost=4.905\n", + "INFO:root:Epoch[91] Train-loss=169.768920\n", + "INFO:root:Epoch[91] Time cost=4.636\n", + "INFO:root:Epoch[92] Train-loss=169.620803\n", + "INFO:root:Epoch[92] Time cost=4.429\n", + "INFO:root:Epoch[93] Train-loss=169.448189\n", + "INFO:root:Epoch[93] Time cost=4.985\n", + "INFO:root:Epoch[94] Train-loss=169.295794\n", + "INFO:root:Epoch[94] Time cost=4.649\n", + "INFO:root:Epoch[95] Train-loss=169.143627\n", + "INFO:root:Epoch[95] Time cost=4.602\n", + "INFO:root:Epoch[96] Train-loss=168.989410\n", + "INFO:root:Epoch[96] Time cost=4.904\n", + "INFO:root:Epoch[97] Train-loss=168.841089\n", + "INFO:root:Epoch[97] Time cost=4.602\n", + "INFO:root:Epoch[98] Train-loss=168.694906\n", + "INFO:root:Epoch[98] Time cost=4.589\n", + "INFO:root:Epoch[99] Train-loss=168.527604\n", + "INFO:root:Epoch[99] Time cost=4.560\n", + "INFO:root:Epoch[100] Train-loss=168.385596\n", + "INFO:root:Epoch[100] Time cost=4.835\n", + "INFO:root:Epoch[101] Train-loss=168.246526\n", + "INFO:root:Epoch[101] Time cost=4.558\n", + "INFO:root:Epoch[102] Train-loss=168.093663\n", + "INFO:root:Epoch[102] Time cost=4.609\n", + "INFO:root:Epoch[103] Train-loss=167.938807\n", + "INFO:root:Epoch[103] Time cost=4.599\n", + "INFO:root:Epoch[104] Train-loss=167.814916\n", + "INFO:root:Epoch[104] Time cost=4.394\n", + "INFO:root:Epoch[105] Train-loss=167.676473\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Epoch[105] Time cost=4.724\n", + "INFO:root:Epoch[106] Train-loss=167.560241\n", + "INFO:root:Epoch[106] Time cost=4.316\n", + "INFO:root:Epoch[107] Train-loss=167.424132\n", + "INFO:root:Epoch[107] Time cost=4.646\n", + "INFO:root:Epoch[108] Train-loss=167.284482\n", + "INFO:root:Epoch[108] Time cost=4.472\n", + "INFO:root:Epoch[109] Train-loss=167.184511\n", + "INFO:root:Epoch[109] Time cost=4.768\n", + "INFO:root:Epoch[110] Train-loss=167.037793\n", + "INFO:root:Epoch[110] Time cost=4.717\n", + "INFO:root:Epoch[111] Train-loss=166.916652\n", + "INFO:root:Epoch[111] Time cost=4.803\n", + "INFO:root:Epoch[112] Train-loss=166.796803\n", + "INFO:root:Epoch[112] Time cost=4.617\n", + "INFO:root:Epoch[113] Train-loss=166.655028\n", + "INFO:root:Epoch[113] Time cost=4.420\n", + "INFO:root:Epoch[114] Train-loss=166.561129\n", + "INFO:root:Epoch[114] Time cost=4.333\n", + "INFO:root:Epoch[115] Train-loss=166.434593\n", + "INFO:root:Epoch[115] Time cost=4.526\n", + "INFO:root:Epoch[116] Train-loss=166.322805\n", + "INFO:root:Epoch[116] Time cost=4.310\n", + "INFO:root:Epoch[117] Train-loss=166.195452\n", + "INFO:root:Epoch[117] Time cost=4.458\n", + "INFO:root:Epoch[118] Train-loss=166.073792\n", + "INFO:root:Epoch[118] Time cost=4.333\n", + "INFO:root:Epoch[119] Train-loss=165.967437\n", + "INFO:root:Epoch[119] Time cost=4.459\n", + "INFO:root:Epoch[120] Train-loss=165.876094\n", + "INFO:root:Epoch[120] Time cost=5.070\n", + "INFO:root:Epoch[121] Train-loss=165.748064\n", + "INFO:root:Epoch[121] Time cost=4.782\n", + "INFO:root:Epoch[122] Train-loss=165.656283\n", + "INFO:root:Epoch[122] Time cost=4.640\n", + "INFO:root:Epoch[123] Train-loss=165.540462\n", + "INFO:root:Epoch[123] Time cost=4.522\n", + "INFO:root:Epoch[124] Train-loss=165.448734\n", + "INFO:root:Epoch[124] Time cost=4.858\n", + "INFO:root:Epoch[125] Train-loss=165.347751\n", + "INFO:root:Epoch[125] Time cost=4.842\n", + "INFO:root:Epoch[126] Train-loss=165.230048\n", + "INFO:root:Epoch[126] Time cost=4.495\n", + "INFO:root:Epoch[127] Train-loss=165.147932\n", + "INFO:root:Epoch[127] Time cost=4.766\n", + "INFO:root:Epoch[128] Train-loss=165.036021\n", + "INFO:root:Epoch[128] Time cost=4.526\n", + "INFO:root:Epoch[129] Train-loss=164.977613\n", + "INFO:root:Epoch[129] Time cost=5.091\n", + "INFO:root:Epoch[130] Train-loss=164.881467\n", + "INFO:root:Epoch[130] Time cost=5.223\n", + "INFO:root:Epoch[131] Train-loss=164.785627\n", + "INFO:root:Epoch[131] Time cost=4.165\n", + "INFO:root:Epoch[132] Train-loss=164.707629\n", + "INFO:root:Epoch[132] Time cost=4.527\n", + "INFO:root:Epoch[133] Train-loss=164.598039\n", + "INFO:root:Epoch[133] Time cost=4.167\n", + "INFO:root:Epoch[134] Train-loss=164.502932\n", + "INFO:root:Epoch[134] Time cost=4.354\n", + "INFO:root:Epoch[135] Train-loss=164.422286\n", + "INFO:root:Epoch[135] Time cost=4.387\n", + "INFO:root:Epoch[136] Train-loss=164.344749\n", + "INFO:root:Epoch[136] Time cost=4.662\n", + "INFO:root:Epoch[137] Train-loss=164.264898\n", + "INFO:root:Epoch[137] Time cost=4.671\n", + "INFO:root:Epoch[138] Train-loss=164.178707\n", + "INFO:root:Epoch[138] Time cost=4.776\n", + "INFO:root:Epoch[139] Train-loss=164.109071\n", + "INFO:root:Epoch[139] Time cost=4.787\n", + "INFO:root:Epoch[140] Train-loss=163.993291\n", + "INFO:root:Epoch[140] Time cost=4.726\n", + "INFO:root:Epoch[141] Train-loss=163.956234\n", + "INFO:root:Epoch[141] Time cost=4.337\n", + "INFO:root:Epoch[142] Train-loss=163.845638\n", + "INFO:root:Epoch[142] Time cost=4.787\n", + "INFO:root:Epoch[143] Train-loss=163.790882\n", + "INFO:root:Epoch[143] Time cost=5.563\n", + "INFO:root:Epoch[144] Train-loss=163.723495\n", + "INFO:root:Epoch[144] Time cost=4.529\n", + "INFO:root:Epoch[145] Train-loss=163.634262\n", + "INFO:root:Epoch[145] Time cost=5.028\n", + "INFO:root:Epoch[146] Train-loss=163.552854\n", + "INFO:root:Epoch[146] Time cost=4.933\n", + "INFO:root:Epoch[147] Train-loss=163.501429\n", + "INFO:root:Epoch[147] Time cost=4.912\n", + "INFO:root:Epoch[148] Train-loss=163.444245\n", + "INFO:root:Epoch[148] Time cost=5.034\n", + "INFO:root:Epoch[149] Train-loss=163.348476\n", + "INFO:root:Epoch[149] Time cost=4.600\n", + "INFO:root:Epoch[150] Train-loss=163.256955\n", + "INFO:root:Epoch[150] Time cost=4.704\n", + "INFO:root:Epoch[151] Train-loss=163.216139\n", + "INFO:root:Epoch[151] Time cost=4.670\n", + "INFO:root:Epoch[152] Train-loss=163.144691\n", + "INFO:root:Epoch[152] Time cost=4.678\n", + "INFO:root:Epoch[153] Train-loss=163.050236\n", + "INFO:root:Epoch[153] Time cost=4.595\n", + "INFO:root:Epoch[154] Train-loss=162.991225\n", + "INFO:root:Epoch[154] Time cost=5.307\n", + "INFO:root:Epoch[155] Train-loss=162.907200\n", + "INFO:root:Epoch[155] Time cost=4.684\n", + "INFO:root:Epoch[156] Train-loss=162.838075\n", + "INFO:root:Epoch[156] Time cost=4.686\n", + "INFO:root:Epoch[157] Train-loss=162.759286\n", + "INFO:root:Epoch[157] Time cost=4.750\n", + "INFO:root:Epoch[158] Train-loss=162.725998\n", + "INFO:root:Epoch[158] Time cost=4.637\n", + "INFO:root:Epoch[159] Train-loss=162.635852\n", + "INFO:root:Epoch[159] Time cost=4.498\n", + "INFO:root:Epoch[160] Train-loss=162.563777\n", + "INFO:root:Epoch[160] Time cost=5.048\n", + "INFO:root:Epoch[161] Train-loss=162.527387\n", + "INFO:root:Epoch[161] Time cost=5.040\n", + "INFO:root:Epoch[162] Train-loss=162.395881\n", + "INFO:root:Epoch[162] Time cost=4.764\n", + "INFO:root:Epoch[163] Train-loss=162.353654\n", + "INFO:root:Epoch[163] Time cost=4.561\n", + "INFO:root:Epoch[164] Train-loss=162.285584\n", + "INFO:root:Epoch[164] Time cost=5.051\n", + "INFO:root:Epoch[165] Train-loss=162.204332\n", + "INFO:root:Epoch[165] Time cost=4.455\n", + "INFO:root:Epoch[166] Train-loss=162.147100\n", + "INFO:root:Epoch[166] Time cost=5.021\n", + "INFO:root:Epoch[167] Train-loss=162.051296\n", + "INFO:root:Epoch[167] Time cost=4.551\n", + "INFO:root:Epoch[168] Train-loss=161.978708\n", + "INFO:root:Epoch[168] Time cost=4.744\n", + "INFO:root:Epoch[169] Train-loss=161.927990\n", + "INFO:root:Epoch[169] Time cost=4.821\n", + "INFO:root:Epoch[170] Train-loss=161.883088\n", + "INFO:root:Epoch[170] Time cost=4.365\n", + "INFO:root:Epoch[171] Train-loss=161.785367\n", + "INFO:root:Epoch[171] Time cost=4.448\n", + "INFO:root:Epoch[172] Train-loss=161.716386\n", + "INFO:root:Epoch[172] Time cost=4.622\n", + "INFO:root:Epoch[173] Train-loss=161.656391\n", + "INFO:root:Epoch[173] Time cost=4.500\n", + "INFO:root:Epoch[174] Train-loss=161.598127\n", + "INFO:root:Epoch[174] Time cost=4.677\n", + "INFO:root:Epoch[175] Train-loss=161.518613\n", + "INFO:root:Epoch[175] Time cost=4.958\n", + "INFO:root:Epoch[176] Train-loss=161.418783\n", + "INFO:root:Epoch[176] Time cost=4.607\n", + "INFO:root:Epoch[177] Train-loss=161.407767\n", + "INFO:root:Epoch[177] Time cost=4.427\n", + "INFO:root:Epoch[178] Train-loss=161.319552\n", + "INFO:root:Epoch[178] Time cost=4.930\n", + "INFO:root:Epoch[179] Train-loss=161.234087\n", + "INFO:root:Epoch[179] Time cost=4.240\n", + "INFO:root:Epoch[180] Train-loss=161.187404\n", + "INFO:root:Epoch[180] Time cost=4.484\n", + "INFO:root:Epoch[181] Train-loss=161.123118\n", + "INFO:root:Epoch[181] Time cost=4.937\n", + "INFO:root:Epoch[182] Train-loss=160.999420\n", + "INFO:root:Epoch[182] Time cost=4.489\n", + "INFO:root:Epoch[183] Train-loss=160.955369\n", + "INFO:root:Epoch[183] Time cost=4.894\n", + "INFO:root:Epoch[184] Train-loss=160.908542\n", + "INFO:root:Epoch[184] Time cost=4.269\n", + "INFO:root:Epoch[185] Train-loss=160.846908\n", + "INFO:root:Epoch[185] Time cost=4.998\n", + "INFO:root:Epoch[186] Train-loss=160.765964\n", + "INFO:root:Epoch[186] Time cost=4.467\n", + "INFO:root:Epoch[187] Train-loss=160.687773\n", + "INFO:root:Epoch[187] Time cost=4.609\n", + "INFO:root:Epoch[188] Train-loss=160.652674\n", + "INFO:root:Epoch[188] Time cost=5.327\n", + "INFO:root:Epoch[189] Train-loss=160.551175\n", + "INFO:root:Epoch[189] Time cost=4.267\n", + "INFO:root:Epoch[190] Train-loss=160.477424\n", + "INFO:root:Epoch[190] Time cost=4.798\n", + "INFO:root:Epoch[191] Train-loss=160.501221\n", + "INFO:root:Epoch[191] Time cost=4.695\n", + "INFO:root:Epoch[192] Train-loss=160.370335\n", + "INFO:root:Epoch[192] Time cost=4.640\n", + "INFO:root:Epoch[193] Train-loss=160.279749\n", + "INFO:root:Epoch[193] Time cost=4.653\n", + "INFO:root:Epoch[194] Train-loss=160.242415\n", + "INFO:root:Epoch[194] Time cost=5.044\n", + "INFO:root:Epoch[195] Train-loss=160.197063\n", + "INFO:root:Epoch[195] Time cost=4.684\n", + "INFO:root:Epoch[196] Train-loss=160.132983\n", + "INFO:root:Epoch[196] Time cost=4.460\n", + "INFO:root:Epoch[197] Train-loss=160.083149\n", + "INFO:root:Epoch[197] Time cost=4.713\n", + "INFO:root:Epoch[198] Train-loss=160.025012\n", + "INFO:root:Epoch[198] Time cost=4.779\n", + "INFO:root:Epoch[199] Train-loss=159.945513\n", + "INFO:root:Epoch[199] Time cost=4.659\n" + ] + } + ], + "source": [ + "# can initilize weights and biases with the learned parameters as follows: \n", + "# init = mx.initializer.Load(params)\n", + "\n", + "# call the VAE, output model contains the learned model and training loss\n", + "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) " + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "# encode test images to obtain mu and logvar which are used for sampling\n", + "[mu,logvar] = VAE.encoder(out,image_test)\n", + "# sample in the latent space\n", + "z = VAE.sampler(mu,logvar)\n", + "# decode from the latent space to obtain reconstructed images\n", + "x_construction = VAE.decoder(out,z)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADACAYAAADhh27FAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xm0XFWZ9/HfQxJCyETm4ZLkQgiahOYNcknC8NpR0jSDLMAWFbUNk9AuURHpBmmlozRL+qVB8JWlQAeChrZBZQgsWhtBRpkueYMEAoQkF5NwM0IGQshA9vtHnWh591Pk3NSt4dT5fta6K3WfOsM+Vfvu2jl1nvNYCEEAAABAHu1V6wYAAAAAtcJkGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWS4DpjZf5jZZbVuB1AOM2s2s2Bm3Us8/5KZTatyswAX4y6yjjG36xj3GZbM7J2iX/eVtFXS+8nv54cQbq9+q5BnZtYm6dwQwm9r3Za0zKxZ0lJJPUIIO2rbGtQ7xl3UE8bcfHP/N5E3IYQ+ux6n+YMws+50PGQd/Ri1xLiLvKEP1y8uk0jBzP7VzO4ws5+b2SZJXzCzOWY2s2iZ6cmAvuv3/c3sbjNbY2ZLzewrH7D9P21r13bM7FvJum+a2clm9gkzW2Rmb5nZPxWte6SZPW1m682s3cx+aGY9ip4/wcxeM7MNZvZ/zexJMzuz6PlzzewVM3vbzP7bzEZ10cuGCkn6wvzkPf+9mR1a9NylZrbYzDaZ2ctmdlrRc2cm7/8PzGydpJlJ7Akz+/ekDyw1sxOK1ulvZrOSvrUi+VvoljzXLVlvrZktkXTSbtrdZmbTk8czzewXSd/fZGYvmtnBSb9fbWbLzOy4onXPMrOFybJLzOz8Dtv+p6SNbyZ9OpjZQclzPZN2/tHMVpnZT8ysV3nvAiqNcRf1gjG38cdcJsPpnSbpPyX1l3THBy1oZntJul/Sc5KaJP2NpH80s2NT7mt/Fd6bkZKukDRL0mclHSZpmqTvmdnoZNkdkr4uabCkoyUdL+n8pB1DJd0p6R+T55dKmlzUzr9LnjtF0hBJzyTHiDplZodJukWF93iQpBslzTWznskiiyX9bxX66XclzTGzEUWbmCJpiaRhkq4sir2qQh/5P5JmmZklz81WoY8dpEL/O07SuclzX5L0iSTeIulTnTyckyX9TNIASf9P0m9U6PdNkr6XHNsuq5N99ZN0lqQfmNlHktfkeEkXSZqetHNah/1cJelgSZOS55skXd7JtqI2GHdRU4y5ORlzQwj8FP1IapM0vUPsXyU93CE2R9LMot+nS2pLHh8taUmH5b8j6eYS+/zTtpLtvCOpW/L7AElB0uFFy78g6RMltnWxpF8kj8+W9HjRcyapXdKZye8PSppR9Hx3Fa7ba6r1+5D3H68fJvEfS7qiQ+xVSX9dYjvzJZ2SPD5T0h87PH+mpNeLft836W/DVRi8t0rqVfT8GZJ+lzx+WNI/FD13XLJu990dk6SZkh4seu7kDv2+b7Kt/Ups6x5JX08e3yLp+0XPHZSse1DS5zdLGlv0/JGSltb6Pebng/s74y4/te6DSZwxNzT+mMs1w+kt68SyYySNNrP1RbFukh5Juf7aEMKuRJItyb+rip7fIqmPJJnZhyVdI+lwFf6ouqtwpkEqnOH4U7tDCMHMlndo5w1mdn1RbKcKZ0hWpGwrqmuMpBlm9tWi2N4qvNcysy+q8D/25uS5PiqcfdjF68crdz0IIbybnKDoI2mgpB6S2v980kJ7FW3jL/qXpDc6eSwd+7TX7/tIWp98jfgvKpxt2EuFvv5iUTtai7ZV3KYhybLPFx2DqfD3iPrHuItaY8zNwZjLZDi9jrfd2KzCG77L8KLHyyQtCiGMr3irCl9rPC3pMyGEd8zsYhW+2pAKZyOKrwEyFb6uKG7nd0IIH/j1I+rKMklXhhCu7PiEmY2RdLOkYyU9FUJ438zmqzAQ7dKZ28csU+EsxeDgJ320Syq+1nG0s0zZkq8jfyXpi5LuDSFsN7N79OfjaldhIrFLcZvWqjDITwwhMNHIHsZd1Bpjbg7GXK4Z3nPzJZ1kZgOS64O+VvTcU5K2mdk3zWyf5KL3vzKzwyvQjr6SNkjabGbjlVy3lrhf0keskAjSXYVr3IYUPf8TSf+crCcz28/MOnsNEiqnR9J/dv10V2Hg/Qczm2IFvc3sJDPrK6m3CgPvGqmQACHpkD3deQihXdL/SLrGzPqZ2V5mNtbM/jpZ5E5JX7NC0tIASZeWcawfZG9JPVU4rh3JGYvjip6/U9JZZjbezPZV4avxXcewU4XX7AfJtZwysyYz+9sKtRWVxbiLSmLMLcjdmMtkeM/NlrRQha8pfi3pv3Y9kfyP7kQVkibaVPif0o0qXIje1b4paYakTck+/nS2IYSwStJnJF0raZ2ksSpcNL81ef4XyXO/MLONkv4gqa47bM48oML/sHf9zAwhtKqQRPEjSW9Lel2Fa9AUQnhZha9un1Lh67C/kvRkmW34ogoD48vJ/n4paVdyyM0qJGC8IGmepLvK3JcrhLBJhUnPnUkbPidpbtHz/y3ph5J+p8Lr8XTy1Nbk30t2xZN+/ltJH6pEW1Fxs8W4i8phzFU+x1yKbuSIFW7P8qakT4UQHq91e4BKSM64LZDUs8RXjUDVMO6i0TXCmMuZ4QZnZscnX8P1VOGrjO2Snq1xs4AuZWanWeHelgMk/Zuk+7I6KCP7GHfR6BptzGUy3PiOUeEeh2tU+CrutBDC1g9eBcic81W4L+ZiFUr6frm2zUHOMe6i0TXUmMtlEgAAAMgtzgwDAAAgt8q6z7AVSvJdr8LNlP8jhHDVBy0/ePDg0NzcXM4uAT3//PNrQwhDdr9k16HvolxtbW1au3at7X7JrkO/RVdgzEVWpe27ezwZTjJkb1Ch/vtySc+Z2dzkViOu5uZmtba2lnoaSMXMOlt1p2z0XZSrpaWl6vuk36IrMOYiq9L23XIuk5isQn3tJSGEbSrc7/GUMrYHAAAAVFU5k+Em/WU96uX6y5KTkiQzO8/MWs2sdc2aNWXsDqgu+i6yiH6LrKLvolYqnkAXQrgphNASQmgZMqSqlxwBZaHvIovot8gq+i5qpZzJ8ApJo4p+3z+JAQAAAJlQzmT4OUnjzOwAM9tb0mdVVLsaAAAAqHd7fDeJEMIOM7tA0m9UuLXaLSGEl7qsZQAAAECFlXWf4RDCA5Ie6KK2AAAAAFVFBToAAADkVllnhgEAAFB9IYQu36ZZVYtk1g3ODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt7ibBAAAQJ1Ie5cIbzkvVq07RGT5ThScGQYAAEBuMRkGAABAbjEZBgAAQG4xGQYAAEBukUAHIFMqUYLUk+VkEADVkXY82rlzZxR7//333WV37NiRallvm9641b27P9Xba6/4fGi3bt1SbTNtrJRy1+9qnBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpWVQGdmbZI2SXpf0o4QQktXNApA/pRTdamc5aT0iRsk1WXbhg0botjNN98cxebMmRPFXnjhBXebvXv3jmILFy6MYqNGjUrTRNSBtJXdJD+JzUuAe/fdd6NYe3u7u82XX345inl9av369VGsR48eUWzw4MHufpqbm6PYmDFjUq3ft2/fKNarVy93P14CX9rkPW85qevH4q64m8THQghru2A7AAAAQFVxmQQAAAByq9zJcJD0P2b2vJmd5y1gZueZWauZta5Zs6bM3QHVQ99FFtFvkVX0XdRKuZPhY0IIH5F0gqSvmNlHOy4QQrgphNASQmgZMmRImbsDqoe+iyyi3yKr6LuolbKuGQ4hrEj+XW1md0uaLOmxrmgYkEflVlerRHJZ2mQSL5Ek7XKd2Y+XOOElWZRKsPCWTRsrhcS62inVn7yEt/POi7/AnDdvXqr9lOoPW7ZsiWLLli2LYiTQ1ae0406panHbt2+PYhs3boxiixYtimIPPPCAu81HHnkkii1dujTVvvv06RPF+vfv7+5n7NixUezggw+OYhMnToxiH/rQh6LYyJEj3f3069cvinmJfp5SY2tXj7l7fGbYzHqbWd9djyUdJ2lBVzUMAAAAqLRyzgwPk3R3MjvvLuk/Qwi/7pJWAQAAAFWwx5PhEMISSf+rC9sCAAAAVBW3VgMAAEBudUXRjUx4+umno9j111/vLtvU1BTFvMoqM2bMiGIDBw50t1kqjnwoJwnNq2a0bds2dz9bt26NYu+9916q2ObNm1Nv00sa8ioseTFve6W26SWIeIkTw4cPj2Jecojk/33vu+++UcxL8CBRrra8222deOKJ7rJpE+PSKpVA5/3Ntra2RrGjjjoqinl/h9ddd527n1tvvTWKvfTSS1HMq/aFgnISjEslanpj1Lp166LY888/H8UWLPDTrFavXh3FvDmEVy3OS6Ar9Xnh9Wmvqt2bb74ZxYYOHRrFBg0a5O6n1GuXRrlJ5WlxZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5xWQYAAAAuZWbtFPvzg9eecTOuPLKK6NYqbKHU6dOLWtf1dDc3BzFvvWtb7nLjh49usKtyaZSma9py3t6Wb8bNmyIYl5WveRnIb/++utR7I033ohiK1eudLe5atWqKOZlHHsZwz179oxi++yzj7sf7zXy9uM56KCDotjRRx/tLjt9+vQo5pXKTVsuFJWxadOmKHbEEUdEMa/0cSn77bdfFDvnnHOi2LRp06KYV35Wkm644YYo1tLSEsUWL14cxc4666wo9uSTT7r78Tz22GNR7OMf/3jq9VE+bxz3+u4777wTxXr37u1u0+vnH/7wh6NYqT7ZkTfeS/5dL7w75nh3KPHGx1J3MunWrVsU8+5kUcu79XBmGAAAALnFZBgAAAC5xWQYAAAAucVkGAAAALmVmwS6e+65J4rNnz/fXXbixIlRzCt7+cwzz0Sxe++9193mb37zmyh2wAEHRLGlS5e666flXcA+YsSIKJY26cRLqpOkSy65pFPtyru05T290p5vv/12FHvllVfc/XjlPRcuXBjFXnvttSjmJd9JfvlkLwmuX79+UWzYsGFRrG/fvu5+vLLTS5YsiWJeUp33uh144IHufkqVJu2I0svV4/19XHHFFVHMG7e85BxJOvPMM6PYhRdeGMUmTJiQooWlXX311VHMG+8vuOCCKPbCCy+k3s/pp58exbxEP5Tm/U17fc9brtR4kDYR2tuPV2JZ8hN6jzzyyCg2ZMiQKOYl6nljeKm497nk3RjAG++9svaSPy+pt6Q6zgwDAAAgt5gMAwAAILeYDAMAACC3mAwDAAAgt3abQGdmt0j6hKTVIYRDkthASXdIapbUJunTIYQ4y6eOjB8/PlWslEMPPTSKnXHGGVHsqquuctdva2uLYl4CnZcw1Bl77713FPMS6Lx9e1XNvKo3qBwvWcBLKvCSzSTp3XffjWJegseAAQOi2ODBg91teklwI0eOjGJesuXQoUOjWKkKdF6lPK8qnlcRz0ugK7UfL8nDS+boTBINyvPtb387il1zzTVRzHufvGQ1SfrYxz5WfsOKeIlFknTrrbdGMS9Rb8uWLVHM+5u78cYb3f2cfPLJUcwbG9A5lfib9vqKl0DXp08fd33vM9tLYvOqwHmfDRs3bnT3431e9OrVK4oNHz48ig0aNCjVupKfQJc2Wa6eEuhmSzq+Q+xSSQ+FEMZJeij5HQAAAMiU3U6GQwiPSXqrQ/gUSbclj2+TdGoXtwsAAACouD39jmVYCKE9ebxSUvw9asLMzjOzVjNr9b6GB+oVfRdZRL9FVtF3UStlX3AUChfCxBfD/Pn5m0IILSGEFu8G0UC9ou8ii+i3yCr6LmplTyvQrTKzESGEdjMbIckvXZVDpZJ20iaidSapLy2vUt7atWuj2JQpU6LYcccd1+XtyaO0iXFeokHv3r2jmJdgIUlNTU2ptuklkXnrStLBBx+calmvIpGX8ORVSCoV9xI/vOQUL5lj3Lhx7n68RBTvNUL1/PKXv0y1nFdVrqsT5SS/6uPZZ5/tLjt37txU2/Qmd88991wU86qPofZKJXKlraTmxbykOknatGlTFHvrrY5Xq/oJcIsWLYpib7zxhrsfbyz1EqG9PulVz+vZs6e7H298TZsYV08JdJ65kmYkj2dI8msQAwAAAHVst5NhM/u5pKckfcjMlpvZOZKukvQ3ZrZI0vTkdwAAACBTdvvdYAghvpluwbFd3BYAAACgqrhjNwAAAHKLyTAAAAByixTqBrN58+Yodtppp0UxL4v0uuuui2Klyiuic7yMWC8L2cu69d4D744IkjRmzJgoNnr06CjmZQd7pZMlv3Szd9cU7xi9UtBeprQkzZs3L4otX748inklxw855JAoNmHCBHc/3t050pZjRm194Qtf6PJteiW/r7jiiiiW9q4RknTCCSdEsR/+8IdRjDtHZEdn7ibhjeOdubPOggULoph3hxNvLFy2bFkUK1WOeezYsVHMG0u9uwd5n0ul7sqThdL2nBkGAABAbjEZBgAAQG4xGQYAAEBuMRkGAABAbpFA12Bmz54dxbwEEa98rZd8hc4plRRQquxmR2lLNPft29dd30vI6dOnTxTz3n8vsazU/r3j9I5x27ZtUWz+/Pnufu65554o5iWYeCXLp06dGsX2339/dz9e0gmy4Y477ohiRx55pLtsjx49opiX0HnJJZdEsTlz5qRuk5dIdPnll0exAw88MPU2UX86k0Dnlbv3xlcv4V2SlixZEsW8MsveWOaNmV4pcMkvqTxy5Mgo5n2GlFNiuR5xZhgAAAC5xWQYAAAAucVkGAAAALnFZBgAAAC5RQJdRi1evNiNX3TRRanWf+qpp6LY8OHDy2oTOidtsoGXoOElNEhSz549o5iX4JO2gpzkJx15y+7YsSOKrVixIordfvvt7n7a2tqimJcgMm3atCjmJVF5SSySn6SI2jrggAOi2Ouvvx7FfvKTn0SxF1980d3mT3/60yh29dVXR7G0yXKlKjQ+++yzUYzKcvnhjc9egrOXxFYqmddLglu/fn0US/sZUqqSrDcWep8h3nJZTpbz8KkAAACA3GIyDAAAgNxiMgwAAIDcYjIMAACA3NptAp2Z3SLpE5JWhxAOSWIzJX1J0ppksctCCA9UqpGI3XfffW58+/btUez000+PYlRDqq60Fds8XqWfUslh3jbTJj/s3LnT3WbaZTdt2hTFHn744Sj26KOPuvvxtulVm/vUpz4VxUaMGBHFvApkUuMlfjSCO++8M4oNGDAg1bpPPvmkGx87duwet+fcc8+NYldddZW7bNp2IttKjRve+OolOHtjlFcJVPIrw3m8MdNLZN6yZYu7/rp161It632ueLEsj61pzgzPlnS8E/9BCGFS8sNEGAAAAJmz28lwCOExSW9VoS0AAABAVZVzzfAFZvYHM7vFzEp+T2Rm55lZq5m1rlmzptRiQN2h7yKL6LfIKvouamVPJ8M/ljRW0iRJ7ZKuKbVgCOGmEEJLCKHFu+k0UK/ou8gi+i2yir6LWtmjCnQhhFW7HpvZzZLu77IWIeIlxd19993usl71mO9///tRzKuag9rzkjG896pU5aK0SXmeziQ/bNu2LYotXbo0is2dOzeKedWVJL8CopfIdPjhh0cxr8JSZyrNZTnxoxF4Fbu8KpmXXHJJFHviiSfcbZZKCE3j1FNPjWIkyuVbqbHVGzu8Cp/9+/ePYqWqGo4bNy6KeZ8DXnXQP/7xj1HMq14nSStXroxi3hn5ziQoZ9UenRk2s+JX5jRJC7qmOQAAAED1pLm12s8lTZM02MyWS/oXSdPMbJKkIKlN0vkVbCMAAABQEbudDIcQznDCsyrQFgAAAKCqqEAHAACA3GIyDAAAgNzao7tJoLpmzYqvSnn88cfdZT/3uc9FMUov1ycvC9mLeeWYS90poZwM+lK8jGUv49i7c8S8efOiWKlS0p/85Cej2Gc+85ko5pU65e4o2eb1+8mTJ0ex2bNnR7GJEye62yxVgjaNyy+/PIq1tLS4y3ILsGxLW2q4lHLG8f3228/dpnc3iZEjR0Yxb2x+7rnnotjChQvd/bz33ntRbMOGDVGsEp8r9YYzwwAAAMgtJsMAAADILSbDAAAAyC0mwwAAAMgtEujqzPz586PYV7/61ShW6sL7733ve13eJlSPlxjnJXOUKh9cTiKZl4wh+aU8vVK59913X6r9TJkyxY1/+ctfjmJeCVyS5fLLKx9bKlHu5JNPjmJeCdm77rorinmJn4cccoi7H6/8bc+ePd1l0XjSJuB546uXDCxJTU1NUWzUqFFRbMeOHVFs7dq1UWz16tXufrzPm+3bt0exziQUZhVnhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbJNDVkJf4ccYZZ0Qx78L7z3/+8+42qTbXeNJWOOoMLyGiVCLSq6++GsXuvffeKOYlN40ZMyaKnX766e5+vL5Lslx+vfPOO1Hs7LPPjmK9e/d21/cS4zzf+MY3otiPfvSjKOYlJkl+da9Jkyal2jfqU2eq0qVNlvOS3QYOHOhu06tq6CUTb968OYrtvffeqWKlpD32Rkuq48wwAAAAcovJMAAAAHKLyTAAAAByi8kwAAAAcmu3CXRmNkrSTyUNkxQk3RRCuN7MBkq6Q1KzpDZJnw4hvF25pmbbzp07o9hJJ50UxbxkpfHjx0ex7373u13TMOSSlyzX1tbmLnv//fdHsaeffjqKeQkeH/3oR6PYtGnT3P10JskDjc9LoHvllVeiWKkEOq+6lufaa6+NYocffngUO+uss9z1L7rooijmVWMs1U5UTznJYd5nuOQnxm3YsCGKvfvuu1GsVKXCffbZJ4p5SXlvvfVWFGtvb0/VHslP4POqNJabsJ0FaUaLHZK+GUKYIGmqpK+Y2QRJl0p6KIQwTtJDye8AAABAZux2MhxCaA8hzEseb5K0UFKTpFMk3ZYsdpukUyvVSAAAAKASOnXNsJk1SzpM0jOShoUQdp2PX6nCZRTeOueZWauZta5Zs6aMpgLVRd9FFtFvkVX0XdRK6smwmfWR9CtJF4YQNhY/FwoX2Lh3YA4h3BRCaAkhtHg3kgbqFX0XWUS/RVbRd1ErqSrQmVkPFSbCt4cQdpX1WWVmI0II7WY2QtLqSjWyEXgXuj/yyCOp1v3Zz34WxUpVrgE62rp1axTzqsX9/ve/d9f34l6y22GHHRbFTjnllChW6kMubcIT8sFLOBs9enQUW7Vqlbu+d2bR63telcNhw9wvOl0vv/xyqm0iO7xkue3bt7vLrl+/PoqtWLEiVaxU0rC3fy8Bz0tkfuqpp6KYN/+QpKFDh0axfv36RbE8jM27PUIrpBHOkrQwhFCcdjtX0ozk8QxJcX1WAAAAoI6lOTN8tKS/l/Simc1PYpdJukrSnWZ2jqQ3JH26Mk0EAAAAKmO3k+EQwhOSSt1k7tiubQ4AAABQPY1/IQgAAABQApNhAAAA5Faqu0kgvVJlD6dOnZpq/Tlz5kQxL0sf8HilQd9+O66S7mXAP/vss+42N27cGMUOPPDAKDZ58uQo5pUS98p9Svko+Yn0+vbtG8UuvvjiKPa1r33NXX/kyJFRzMuKP/roo6PYY489lqaJkvw7VHjldJFtpe4msW7duij22muvRbEFCxakWleSNm/eHMWWL18exZYuXZpq3TFjxrj7aWpqimLenVTKLdGchbGdM8MAAADILSbDAAAAyC0mwwAAAMgtJsMAAADILRLoutitt97qxpcsWZJq/WOOOSaKZeHic1SXV65T8pMnli1bFsXmz58fxdrb291tDhgwIIp5CXRTpkyJYl4SVGdKe9L3UWzGjBlRbN68ee6ys2fPjmLe382jjz5aVpt+/etfl7U+sqHUWOSVVPYSzrZu3RrFvERmSVq8eHEU8xKhvbHUK1k+bdo0dz/Tp0+PYl4CXffu8VTR23eWx2vODAMAACC3mAwDAAAgt5gMAwAAILeYDAMAACC3SKArw6JFi6LYzJkzq98QNDQv6cdLxpCk9evXR7GVK1dGMa/y0b777utu06uwNXHixCjmVTPyEi86I4QQxbKcpIHy9OnTJ4rdeOON7rLNzc1RbNasWVHMSzD1+vLDDz/s7mfEiBFuHPUn7djhJYd5iXKSNHTo0Ch2xBFHRLHBgwdHMa9KouQn0HljvrfNQw89NIp5yc2Sn2znfQ5069YtijXaOMyZYQAAAOQWk2EAAADkFpNhAAAA5BaTYQAAAOTWbrNbzGyUpJ9KGiYpSLophHC9mc2U9CVJa5JFLwshPFCphtajxx9/PIpt3Lgx9frjx4+PYr169SqrTcg2L2HMS6DbsWOHu74X9xI/vMSNnj17utscOHBgFJs0aVIU86rNeUkW3jGWindm/TTrovGUStL8zne+kyqGfPPGCS9hzItJ/rjZv3//KHbQQQdFsWOPPdbdpjfmpx33vOS/Um3PQ2JcWmlSvXdI+mYIYZ6Z9ZX0vJk9mDz3gxDCv1eueQAAAEDl7HYyHEJol9SePN5kZgslxfedAQAAADKmU9cMm1mzpMMkPZOELjCzP5jZLWY2oMQ655lZq5m1rlmzxlsEqEv0XWQR/RZZRd9FraSeDJtZH0m/knRhCGGjpB9LGitpkgpnjq/x1gsh3BRCaAkhtHg37wfqFX0XWUS/RVbRd1ErqSbDZtZDhYnw7SGEuyQphLAqhPB+CGGnpJslTa5cMwEAAICul+ZuEiZplqSFIYRri+IjkuuJJek0SQsq08TGcNRRR0WxBx98MIpxNwl05GX39ujRw1120KBBqZb1SoiWuhOKly3trb/PPvu461dDXjOgAXS9zownjD2NIc3dJI6W9PeSXjSz+UnsMklnmNkkFW631ibp/Iq0EAAAAKiQNHeTeEKS91+fXN1TGAAAAI2HCnQAAADILSbDAAAAyK001wyjhLPPPjtVDEirWqVB999//yiWttxnZ3jHU27CCQkrAICuxJlhAAAA5BaTYQAAAOQWk2EAAADkFpNhAAAA5JZVImmm5M7M1kh6I/l1sKS1Vdt55XGxTVv2AAADPUlEQVQ81TMmhFDVwvVFfbeeX5c9wfFUTy37rVTfr82e4Hiqh77btTie6knVd6s6Gf6LHZu1hhBaarLzCuB48qHRXheOJz8a7bXhePKj0V4bjqf+cJkEAAAAcovJMAAAAHKrlpPhm2q470rgePKh0V4Xjic/Gu214Xjyo9FeG46nztTsmmEAAACg1rhMAgAAALnFZBgAAAC5VfXJsJkdb2avmtnrZnZptfdfLjO7xcxWm9mCothAM3vQzBYl/w6oZRs7w8xGmdnvzOxlM3vJzL6exDN7TJVC360v9N10st5vpcbqu/Tb9LLedxup30qN3XerOhk2s26SbpB0gqQJks4wswnVbEMXmC3p+A6xSyU9FEIYJ+mh5Pes2CHpmyGECZKmSvpK8p5k+Zi6HH23LtF3d6NB+q3UWH2XfptCg/Td2Wqcfis1cN+t9pnhyZJeDyEsCSFsk/Rfkk6pchvKEkJ4TNJbHcKnSLoteXybpFOr2qgyhBDaQwjzksebJC2U1KQMH1OF0HfrDH03lcz3W6mx+i79NrXM991G6rdSY/fdak+GmyQtK/p9eRLLumEhhPbk8UpJw2rZmD1lZs2SDpP0jBrkmLoQfbeO0XdLatR+KzXA+0y//UCN2ncb4n1utL5LAl0XC4V71WXufnVm1kfSryRdGELYWPxcVo8JnZPV95m+iyy+z/RbZPV9bsS+W+3J8ApJo4p+3z+JZd0qMxshScm/q2vcnk4xsx4qdOzbQwh3JeFMH1MF0HfrEH13txq130oZfp/pt6k0at/N9PvcqH232pPh5ySNM7MDzGxvSZ+VNLfKbaiEuZJmJI9nSLq3hm3pFDMzSbMkLQwhXFv0VGaPqULou3WGvptKo/ZbKaPvM/02tUbtu5l9nxu674YQqvoj6URJr0laLOmfq73/Lmj/zyW1S9quwjVM50gapEIG5SJJv5U0sNbt7MTxHKPCVxp/kDQ/+Tkxy8dUwdeKvltHP/Td1K9TpvttcgwN03fpt516rTLddxup3ybH07B9l3LMAAAAyC0S6AAAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBbTIYBAACQW0yGAQAAkFtMhgEAAJBb/x851DqC1HpbSAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n", + "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax1.set_title('True image')\n", + "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax2.set_title('Learned image')\n", + "ax3.imshow(np.reshape(image_test[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax3.set_title('True image')\n", + "ax4.imshow(np.reshape(x_construction[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + "ax4.set_title('Learned image')\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=15.0 to DejaVu Sans ('/usr/local/lib/python3.5/dist-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXuQJFd15r9T1VXqqepBgmx5tUbT2RIWbLQxL7URhLyWlsK2GMkrO0wIyy0Y1oTbKtiNMQFhHh1reTeiYQkcYFgZRIM1zKpqWWQEy2MF2EiyMCC09IDQogeyzHSPhBCaaUkITQ2a0fTZP7KylZWdj5uZNx+VeX4RJ2a6KjPrZlXmd0+ee+65xMwQBEEQykMt7wYIgiAIehFhFwRBKBki7IIgCCVDhF0QBKFkiLALgiCUDBF2QRCEkiHCLuQKEa0R0avzbocglAkRdmFsICImol/RdKwLiehBHccShKIhwi4IglAyRNiFwkBELyei24jocSL6CRFdTUTN4XtfH272fSJ6koheN3z9EiK6Y7jPt4joRY7jrRHR24noTiL6GRF9mogmiagN4MsAfnl4rCeJ6Jc92rObiO4mop8T0Y+J6O3D1y8kogeJ6N1EdGT4OQuO/S4mou8R0RNE9AAR/aXruL8xbOvjw/ffOHz9FCL6KyI6REQ/JaJriGiH1i9ZqAQi7EKROAngrQCmAbwSQAfAmwGAmX9zuM2LmXmKmT9NRC8FcC2APwVgAPgYgC8Q0SmOY14G4CIAZwF4EYA3MvNRAK8B8NDwWFPM/JBHe/4WwJ8y804ALwRws+O9M4btfC6APQBWiOgFw/eOAngDgNMAXAygS0S/BwBEZMLqVP47gNMBvATAHcP9/huA5w9f+5Xhsf9C8bsThC1E2IXCwMwHmPnbzPw0M6/BEuoLAnZZBPAxZr6dmU8y834ATwF4hWObDzPzQ8z8KIAvwhJNVU4AmCOiZzHzY8z8Xdf7/5mZn2LmWwH8H1idCJj5H5n5/zHzJjPfCeBTjvP4IwBfY+ZPMfMJZt5g5juIiIbn81ZmfpSZfw7gPQD+MEJ7BQGACLtQIIjo+UT0JSJ6mIiegCVs0wG7mADeNgxpPE5EjwPYBcAZVnnY8f8BgKkITfoDALsBrBPRrUT0Ssd7jw09f5t1+3OJ6DwiuoWIDhPRzwBc6TiPXQD+xeOzTgfQAnDAcS5fGb4uCJEQYReKxEcB3AvgHGZ+FoB3A6CA7R8AsMzMpzmsxcyfUvis0LKmzPwdZr4UwC8B+N8Arne8/exhrN5mBoAdzvmfAL4AYBcznwrgGsd5PADgeR4fdwTAMQC/6jiXU5k5SkckCABE2IVisRPAEwCeJKJ/A6Drev+nAM52/P1xAFcOPWQiovZw4HKnwmf9FIBBRKd6vUlETSJaIKJTmfnEsF2brs3+y3C7fwvgEgB/5ziPR5n5F0T0cljhF5s+gFcT0WVENEFEBhG9hJk3h+fzQSL6pWEbnktEv6NwLoIwggi7UCTeDksEfw5L5D7tev8vAewfhiouY+ZVAH8C4GoAjwG4H8AbVT6Ime+FFfv+0fB427JiALwewNowLHQlgAXHew8PP/MhWGJ95fCYgDXg+1+J6OewBj+3PH1mPgQrvPM2AI/CGjh98fDtdwzP4dvDz/waAHtAVhCUIVloQxCiQUQXAugx85l5t0UQvBCPXRAEoWSIsAuCIJQMCcUIgiCUDPHYBUEQSsaEjoMQ0WkAPgFr2jUD+GNmvs1v++npaZ6dndXx0YIgCJXhwIEDR5g5dNKaFmEH8CEAX2Hm1w6LNrWCNp6dncXq6qqmjxYEQagGRLSusl1iYR9O8PhNDPOHmfk4gONJjysIgiDEQ0eM/SwAhwHsG5Yq/YRrqjUAgIgWiWiViFYPHz6s4WMFQRAEL3QI+wSAlwH4KDO/FFbJ0ne6N2LmFWaeZ+b500+XukaCIAhpoUPYHwTwIDPfPvz7M7CEXhAEQciBxMLOzA8DeMCxyEAHwN1JjysIgiDEQ1ce+38C0CeiO2EtZPAeTccVhBH6/T5mZ2dRq9UwOzuLfr+fd5MEoXBoSXdk5jsAzOs4liD40e/3sbi4iMFgAABYX1/H4uIiAGBhYSFoV0GoFDLzVBgblpaWtkTdZjAYYGlpKacWCUIxEWEXCo0z9LK+7j0349ChQxm3ShCKjQi7UFjs0Mv6+jqCitXNzMxsbS/x92Ijv1FGMHPmdu6557IghGGaJsOqPeRrrVaLe70e93o9brVanu8J+uj1emyaJhMRm6YZ6fuV3yg5AFZZQWNF2IXCQkS+gu4WFr9OwDTNfE+iRCQVZvmNkqMq7LnUY5+fn2cpAiaEMTs76xlXN00Ta2trI6/VajXPcA0RYXPTvQa1EIcov4cX8hslh4gOMHNoBqLE2IXCsry8jFZrtFBoq9XC8vLytm3tOLvq60J0/AapVQev5TfKDhF2obAsLCxgZWUFpmmCiGCaJlZWVjxz1qN0AoJF1IHMpMIsv1GGqMRrdJvE2IU0SDKwVzX84uXdbtf3O/Tap9FosGEYyt+5/EbJgMTYBUHwwy9eTkQjcfBWqzXylNTv97F3715sbGx4Hte9vaAXibELguCLX1zc7eh5zew9duyY73EHgwH27t0rueo5I8IuCBUkyoDl+vr6ljh7lXVws7GxsTWpzK7nI+KeLSLsgpAiecy0dH7m9PQ0pqent33+7t27QUQj+7n/dmKLs19ZhyAGgwH27Nkj4p4lKoF43SaDp0Ke9Ho9NgxjawDQMIxUBvHSnGnpNwjp9Zlum5yc9JzwNTc3F7ifPUgatE2QOc9dBlHjAZl5KpSdOOLQ6/W40WhsE51ms6ldXNKaaRnUYaiUYfCzer0ee19Vs38nKS0QDxF2odR4CXSj0QgVhyDhs0XH7iwMw/BN5VPpVPy8WyLadi5ROqigDiNtYU5q9jmm0eFVARF2oRCk9cjtDKU4zTCMwP3CQglBYYyoBcf8BMwwjJHOw91BhYUskoRDwkQ3bWG3z8Xv84VgRNiF3NH9yK0aaggiaH+VUETQNm6P029CT7PZVBJAv++v3W5rF9xGo8ETExOpinpYuEg89nBE2IXc0XkDqwwKuoXdKSK2IPt5+jrMK8TiHqRV/fygkEUa7Y7SrrifEzTAKzF2NUTYhdzR+citKnJ2KCZKR2Bb0sFDZxjI6/OjiGJQyEK3dTqdzMIwdskC5/ctWTHqiLALuaPTY1cRHmdmS1xvN4nAOYU9ibdte69TU1OZCHueJp56NFSFXSYoCamhs5pf2EzJer2Oa6+9dqtGSdx1UJl5a6KOYRgwDANEhHq9HrrvxsbG1kSgKBN5ms0m2u321t87duwAABw9ejRi68cPWYw8JVTUX7eJx14ddGXFBIVWnF5ft9tVCqmEeeb2U0XS3HAV63Q6kcNGZTLJhlEHEooRyoaXaDs7i263q/z474z1BgluFrHnWq2Wu7jmaZINo46qsEsoRhgL+v0+9u/fj5MnT269Zod17PDLysqK0rH27NmD888/P3S7m266yfJ+UqbMy8IF1Z8BZKGN1FBRf90mHnt1iRuaURmI9XrfywzDqHToI2tz/t7OhTyCZvYK3kBCMULRSJK/rJI6mVV6oNh28xvX8AuzSC57PDIXdgB1AN8D8KWwbUXYq0mS9EeVfU855ZTcBa6qNjc3t61jDRJqmX0aD1Vh1xlj3wvgHo3HE0pGklXuw1In+/0+nnrqqeSNFGJxzz33jIxHEBH27NkDANvq0QfVdY+bpiq4UFH/MANwJoCbALwK4rELPiT10oLi8+NQ2bBqVqvVPCtwBtXKEY89GGTssf81gD8H4Du8T0SLRLRKRKuHDx/W9LHCOJF0wtLCwgLW1tawubmJtbW1kQWT46zsI6TL5uYmTpw4MfLaiRMncPz4cc/tJUNGH4mFnYguAfAIMx8I2o6ZV5h5npnnTz/99KQfK4whCwsLWFlZgWmaICKYpqltRXuVmaFCsdF1LQgAsSMuFusARO8F8HoATwOYBPAsAJ9l5iv89pmfn+fV1dVEnysITsLypYV0ISIk0RLTNLG2tqavQSWFiA4w83zYdok9dmZ+FzOfycyzAP4QwM1Boi4Iuun3+yLsOaMq6o1GA81mc+Q1CcHoR2aeCmPP0tJSIm9R8Ea1+Jl7HyfNZnOrkJppmti3bx+uvfbaVMJxwjMkDsXEQUIxgk5qtZoIu2ZM08Ty8jL27t2LjY2NyPseOnQIMzMzIyUfhOSohmImsmiMIKTJzMyMZMVopNVqYffu3VhcXMRgMIi0r8TKi4GEYgQAVpzaPZFkXPBKoxTiYU8suvHGGz1FPWgsQ2LlxUGEXUC/38fi4iLW19fBzFhfX8fi4uJYibu9OIWQDGbGjTfe6DsD1C/kFRQrH2enYWxRmcWk22TmabHwm7VZr9cLX3kvztqmYuEWZdHvoIUypNiXXiDVHQVVVKoi6rwZda2qxCylBNKyZrOpvABIUBkA1TISOq+JMiPCLnjidQOpiqOOOh66PTgp1ZudTUxMbKv9EvbbqZRb9rsmnLXbRewtRNgFZh4VcsMwthVgsm8g1XBGWh523E5DPPZszTCMbWIbpzib8/f22yZKGeCqIMIuKMef7ZtRNa6ahoftF6cNe0RXPceqryuqy9y/U9gTmMoTWpSnrqpXfxRhrwhu4XM+vvqtauNlUbz2JDdYFI9dNWzT6/WUzjXK9yHm/7s7rzmVlZPCOucoT11BA7VVACLs5UdnRkicWLWuNvs9AYR1AmFhJqfV63Vut9u5C+M4mVcoRNUBiCLAXteE3/UoHrsIe+nJO74cJRzjFmGVRYyDOhuvsJGItz7zG7xMMtAe5Ll7PXlKmuR2IMJeLrxuirwzQgzDUG57nKwH1UE1d5tUQzNi/qIcp7N1/rZe4TKvJ6put8vMzN1ud+s3q9fr3O12JQXSA4iwlwc/YYwyiSQt82uv84b0a2dY1kOUR3R3m/Lu9MbRGo1GqHiGeex2x+om6FrtdDqer9uiLzwDRNjLg9/NNDk56XuzvATgpwE+nLIYtNvtkbBK1EFYt7kf4d2dhMoxgkIGdrpe3iJaRHM/gXl5zGHjOkQ0IshRwjduq9frWdxeYwVE2PND9yNkVO+zXqvxNwB+mCh1YddtYYNuKk8ptVrNM9Zuf4/2b5P3uRbNVCYNOfPWg45l/05Jv2cJv4wCEfZ8SKM2RlSP5wqAD05M8Ad27Bg7YQ/LeihC+GlczTCMkVh20HevkpaaRecoA6ajQIQ9H6LkaavS6/W40WiwCVg/mYddMPycKYB/DPClAF+F9EMxOs1+jA962hFPO7q5s5BUMk6Cjhd2rafRfsECisIuZXs141fu1O91VYgIPwHwCpf9HYBjAB4YbvcXAO4B8PmQYxURZsb+/ft9ywf3+33UanLJRsEwDBw7dgwbGxtb3+n+/fuxZ88eGIaxtZ277LHf9+xcKm95eTnRtTQxobbOz8bGhpT6jYqK+us28dhH8Ur1UjnmxQCfBPgNw7+fD/BRgF84/PsqFNNj9wun+IUIbI8z73aPmwVlThmGERhDDzquTa/X46mpqUzOpeoTk2wgoZh8iBpj73a7nhdyp9MJFLNzAH4M4Ksdr30Z4L9x/H0ViifstVrN95zF9FncuQ6maQaGWJyzfr2yY9KaIGYP7Lo7E3cWTtmBCHt+RMmKiTORZgrguwD+BsATw9cuAvgJgJ8H8KlDey/AR4b/b6YsJFfBO/b/O67tOp2O9tisePPeIsisPw5uX8t+37kdw8/6nKempnzvM3eBO79c+3EAIuzFIEzkwy7YywE+CCvkcnD49w0APwTwGY7t9sJbWG1bSvnGugrWE8R5LnuWazt7EE/nZwfl81fRnGELnTOU7UHMsFBN2PtpmZf37tfJNJvNsRR3iLDnT9Djqi30QeVkLwf4SYwK9FMAnwD4/OHFaW/7XFiZMU7bB/Djw//PpHxTXQX1sI942OnZxMREZOdBxZzhxLBwS56zoolopGRwUKc2jnF7iLDnj8pjcFAo5iC8ve8nsN0z3umx/1UIF1vVUJDp0xaG1XGofJbz5svjpi+7OSdgxS3f7Gd2+CKKN57X72wLdljnMo4lgCHCnj+qF3a73d66+YiIJyYmGLDCL6xoF3gc9yoEiy2z/6Oq25rY3plcD/AA4LOHn3Vi+HnHAf4uwL+fw01dNbPDe7Kg9zNme+0q3924ARH2/FEduHKO+Dtv0IPwFvGDmm6ATqcTWxDcqZYLAL8V4AsB/l2AvzRsq4h7upakFktZTWW1LImxi7DHRtUb9itM5RVjf3L4ep43jleqpZd9C+DvFeBGL6u1221mjjYbVzx7S9THdaFsZCXsAHYBuAXA3QDuArA3bJ+yCrtqudoo5pUVk+dN4ZVq6WdvH7a7VoCbuYxme52qHrvUqrdsYmKCG43GyGvjkg+PDIX9XwN42fD/OwHcB2AuaJ+iC3vQOqLu3t15Y1VhUNAr1dLP3gardLAIe3rmHJsJ29YW9rzbXHQrsgePvEIxsMqU/FbQNkUWdpWBKOfU6zI/2rqfFq6HlW55vuL+3wL4QAHOoypWBcciKytqVUkoCjtZ2+qBiGYBfB3AC5n5Cdd7iwAWAWBmZubc9fV1bZ+rk9nZWai0zTAMTE1NKW07jlwO4OMA2o7XGMA/AniXa9u7AXwRwA0A7h3u8ycALgLwe8P3hGwgIui8p6uMaZpYW1vLuxkjENEBZp4P3U7XRUBEUwBuBbDMzJ8N2nZ+fp5XV1e1fK5uarWa3BgADgKYVdz2QgCvB/DvYMXlNgF8F8B7AHwlhbYJQhYQETY3N/Nuxgiqwq5WNzP8wxqwHLZ+mKgXnZmZmdJ64VGY8Xl9E0Dd4/VbU2yLkC31eh0nT57MuxmpMTk5iV/84heh2z3nOc/JoDXpkLi4NVkFmf8WwD3M/IHkTcqX5eVltFqtXNtQhHrpftXjk1WVF4pOq9XC/v37R+qulw0VUR93dKxacD6sJ/FXEdEdQ9ut4bip0O/3MTs7i1qthtnZ2W0F/BcWFrCysgLTNEFEME0T7Xbb52jp4BcKIqLM2vJuAEddrx0dvi6Ul8FggKWlJSwuLubdlNx59NFH825CfFRGWHVbXlkxKgv0+qU0qsxmy8I6nU5mn1W0HHqxbMyekl+Uaz4vK2LJAcjM01GCJmYErSZjU4SLPI2St2JibgtbbKMKNu7pjpUQ9rj55s7VYvK+0AApdyuWvtneehUdCGd1zCKKOrMI+whxvQ/7Ii/KJKQq3mxi2ZlzWn2Ue8a5LsA4W7fbjbT6WR5AhP0ZggQxaFEAeSQVK7NNTk5uiZi9YLizjEbe7cvaarXatk6qaDVkIML+DH7iXK/XfUsD2DG2cfOSbS9DwjZiYWYLutd7rVZra12AqptzVaa8gQj7MwQJt3Mbr0cwv06hCIOpfhdhULvFxFRt3JyaNK1erxei1C+qKOxe4uysvmhnxUT5Ufw6hbB1H/Mye8FhuSnFxNK1PDJnUDVhVx3kjPJj+HUKRY4/Rq3RLSYmFt+yznVH1YQ9ipCp/BhB4ZtxEE3DMAobLhIbbytLFoyfnQvwPoDvhTU5b1/AtlkviA1FYddRUqAQHDqkXsUkaFu75MAVV1yBwWAw8t5gMMCePXvGokjYxsZG4SrTCeNNvV5Ht9vFU089hV6vl3dzUuN8AL8B4DsAHg7ZdmbGr1xeCN/8JnDeecDkJHDWWcCHPxzvOH6oqL9uK4rH7rVSkko4R+LXYlW1RqMRmlgw7kaO/38H/h67agLGNv75n5nbbebXvY75ppuY3/te5nqd+eMfD9U5VC0Uo5qa2Gq1uNvtJk4HFHEXq6rZA/RFmZEd1UzAkj4Pu8C1rZ+w12q1LeH2Sy/2Hc9bXGQ+5xzmEyeeea3bZT7zTObNzUCdQ9WEfXjSoRekqleudIGU1GMREwuzcRlr8rImwOe57HqABwCf7drWT9idnZtbT0z4dxx8yy3Mu3Yxv+Mdo+J1663W+3feGaZx1RP2sIlIQdtENTucU6QMmbIPaomJpWEXwxokfYPHe0GhGD89cXYcrwCYb7uN+bWvZZ6cZP7BDyzZ3bdvVLweecR6/frrAzUOVRs8BYDdu73LwC8uLmJhYQEAtAx8tlotLC8vAwA+8pGPJD6eLt70pjfl3QRBGCvOAdAD8FEA/yPivv1+3zMR4ziA24f2E9MEjhwBbrgBuOYa4LTTrI3sf22e/Wzr38cei9gKH1TUX7dl7bE7B0vjxMYNwwgcFCnKI2lR2iE2XqZjzCmKERF3Op3cr9cpgO8C+BsAT/hsE+Sx+5UBt63VavHn3/9+5lNPZX7zmy2xePBByzP/3OdGxevECev1j30sUONQxVBMkGgTUegP4bdfWBGgoFrvYmJFNjtMmbXI2gOLWd437oVjbgf4IYDPCNgnSNjDbArg+xoNfuScc5iPH7fE4sknLdn95CdHRURzKKZUwp7WxakyW1WKbomNq+ksSx3lPsha1J8ELMkb2ibAfxmyXxRh9+s4zt6xY1Q/du1ifuc7RwXk61+32iWDp9tJu3Z6UG6qpD+KiWErZJl3O9x2EKOibttDGM2O2QnwNMB/MLT7Ab7Z8bff8cM6jpHZ7ouLzC94AfPTTz/z2lveYgm+pDt645wkkMYF4ue95x0vFBMrihXxXjgJb2F32wVD83vf7/gHfbZ3dhx8223MP/vZMxOULr+c+eabmd/3PuaJCZmgpEpQ+qM9EDo1NRXrwrUZx3xeSYsUS9M6nU5hVh2z7SC8hfegpuOrdhx8yy2WcPzTPzH/+q8zn3IKs2kyf+hDSpoGEXa1OuxxPHu78H5RS/eKiemyRqMRecENe0A277Y7zStU8uTwdR3HP+g6tm0HHdvoACLsFmH1G+J424ZhcKPRyP1ijdPuvNsgNj5Wq9W40+nEutaZOff2u809uKlL1O1jh3UcOmq3Q4RdjagDrkFrpBbZ5ubmCjmoJVYc8yrzHPeJdtzCkzosrOOwyxAkASLs6ji9eq9Ffd0evwikWNksjsPiNz4la6X6W1Igwp4eVfRGxMprdkw8isNie5/dbncrH71er8dKRqiSJV0zFSLs3ijXTA45hsw0FSuLxVkA3S+sIE+z6hZnzVRkKewALgLwQwD3A3hn2PZ5CbtKlowquuLs4xivFyuX2VleUceb/BaNF1O3qGumIithB1AH8C8AzgbQBPB9AHNB++Ql7GFFwsLQPfnJMAyJ2YsVwuyVkeIIdLPZHMsssSJY1DVTkaGwvxLAVx1/vwvAu4L2yUvY/QRU5cvVXa7A+aRQpJruYtU123NnlnEk3eY356XIHvtrAXzC8ffrAVztsd0igFUAqzMzM5FORhdJPHadF7rtqTuRkIxYUUyuRf3Wbre1hIFRNGF32rjF2KPMogvrAPwGncKeCBqNRiFKATSbTe50Orm3Qyxdk/CgftORuAEJxXgT9uW634+yRqpTtL3EL6wTCcqnd8Y/7fe9JpRkYXLTV9ckGyy+6QAZCvsEgB8BOAvPDJ7+atA+Rc1j9/Kao4hYs9n0zS7wW7AjSS8usXkxsfEwHbNOmTMUduuzsBvAfbCyY5bCti+qsOuIowd5NO5YftL0SxngEhMrvtVqNS11YpgzFvaoVlRhTzvEYGcd2B66XyegMpirEvefm5vL/aIOu+AlTU6szOaVKJEEiLBHx88D1iX4qkLmTL/0CtWopF7aC3DnfWGrXPgSsxcrm8Wd+BgGRNij4xcasQuBAdkMHDpF2f15KsWa7Fj/uAjmuLRTTCzMkmS8qAAR9ngEDWZm4QEnTWt0PvqNg8cuJlYmSxuIsOsnzLOM63naS/UlTWFUGZwVExNLx6LOIo0DFIW9BkGZmZkZ3/darRauvPJKmKYJIoJpmjAMI/SYrVYL+/fvx3XXXYdjx45hc3MzdJ92uw0i2nac5eXlrb/7/T6WlpYwGAxQr9cBAKZpotPpbP0tCIIe3Pdf7qiov24bN4/dWRjJyyv3G/n28pibzea2iUfM6mETv1DNKaecsvX/dru9bRt7rEA8eDExfZZ2TN0NJBSjh6BJSyo/qOoEJJUwjixkoM8WAf57gB8G+HGAvwHwbxWgXWLjY1mEXtxAhF0PSUv9Jv0csXRsHeCPAXwpwK8G+JOw1qr83QK0Taz4Zmee2eioA6MCRNj1kKTUbxRkoDNbMzxe+ybANxegbWLFtlqtNlIeROcCPmFAhF0PWXnszNYFIiVT9ZgJWJe3h13gs8/VAN9bgLYX3ewxJXeN8SotYu0U7iw1AiLsekirN/Z7dFMNyeRV2XFcrAnweS67HuABwGf77HMA4C8UoO1FNr9BePdEviqYLdxZPdUzi7D7EicWFmUflW2DZrjmfbGW1S6GFUN/g8/7/wGWN39hAdpaZAt6orRnTBPR2Dkecdobtgi4eOwpoFpjxcv79tvXS7DdNdLdtWDcx+/1elLTOmM7B+DHYIVavN5/GcBPAvzBArS1zGZ7tvb1X4T7wA4rBWlDmHBLjD0jYff7ov28DWfP6rWvX+64V7541AtALJ51Af4OwI8CfBTgO4evubebAvguWOmMEx7vnwXwTwD+PMC1ApxX2cyeVW0OF61JEn6MY41GYyREZHcmXk/TQc5bmHBLVkwGwh71QnHGwtK4yMIe2cSi27uHdgnArwJ4GVaoZWP470GALwf4BoAfAvgMj2OcDvB9AH8b4B0FOKcymn3tB4lj2ES+uJ69zpK5WQl3GKiysEet2eL02NOoNBg2yCKW3C4H+ARgXdJDe2r42vke27dhefz3ATxdgPaPu/ld2/bKQSrhDD/h7PV6nuWu7ZRDdzaZ7hroRQJlE/YoPabfRWQYRugjlW6vWiUtSszjhseoSDvtAo/tD/ps+wS2Z8fsBPirsIT/co/38z73Mlm73WZm9cwRv/Etd8jTPUGoKqBMwh51cCLssS+og4gSYw8z9zqnEmNXN9V0xTos7/sk/DsCt10Q8n7e5142C3JobI/ebw6H6thYVUCZhD1OOlGSmJif1+B1gTUajcAUMHcbJaUxnnmlK/4rhIv4wQK0Xczf7MHNOA5PGnniRQdlEvYsJwCE4ddhqLZRwjG5FRJlAAAR4UlEQVTRzS9dsQ7wubA88M8AvIlRUX8SVqgl7/aLBVvcnHfx2Mdc2LOcAOCFivev2sY8bhx3RcharcadTmfknLrd7kjpXxXLovxBWLqi066H5dU7s2LyEisxfaYyNlYVUCZhz3ICgMpn2+GXqJOfirQO6cTEhGdd+F6vp7R/rVZLtISfqgWlK7rtYlie+vMK8P2K6THVsbGqgDIJO3N+eaQqoRP3xQd4T4QochgmSvZOs9lMpYO6HJanbXvc18PKXPFKV/Sy9wB8DJKTXhYrc9piXFA2Yc8LVQELml1qi2beN0qSc0jbLocVE2eHbcIqo+uVrvh/AX4LrFrqrwH4AwAfB/i9BfgexZKbnS0jjAIRdj2oetkqBYGKUBsjzGyCamSkYQcxKupBdgHAKwD/EFY5gcMAfwvghQJ8f2L6TNgORNj1oOppF2V2qdcMPVUjom2Pvlmdj18e+skCCIxY9lav17fdhxJjF2H3Jc4FEpb9oRqfTmu9UvcgaBJP253Fk5XXfhDewn6wACIjlo8579kk6w6XCWQh7ADeD+BeAHcC+ByA01T2y0vY42bXRLmwgjx8ItIeu/ZK+UwSI7e9drvzc6+Sk5Z5xdglD7265vTYw5yLKqU+IiNh/20AE8P/vw/A+1T2y0vYk+TDq9Rjt18PughVPGrTNJVyxMPKKsQRZ6+c4azMnRUjol5ts+8nlXBgVSYrIetQDIDfB9BX2TYvYdc9g9XvCcBvJp3TCwm6SP2O7TT7M9yLFng9mkYJp0xOTuZ+Q4uJ2RZUK8ZpVSkvgByE/YsArlDZdhw99ijH8/OQnQXB/C5WZ5qX07uPMojp9OTHIc1STCzIVJ4ixWOPKOwAvgbgBx52qWObJVgxdgo4ziKAVQCrMzMzsU8saXEvnTNYg54Aut3uiDftFHW7Le4MlkajEalcgcoNMQ4plmJiQeYc97H/dr4vMfYUPHYAbwRwG4CW6j5xPXYdwqwzbSrpE4BqW/JOoRQTy8L8nBD3/VTl1EdkNHh6EYC7AZweZb+4wq47lJKUtGvYZD1JSEwsb3M7Me41DaoOMhL2+wE8AOCOoV2jsl9cYS9S+V6btLyHLKb1y5OAWNYWdM35vVelUEsYUBR2srbNlvn5eV5dXY283+zsLNbX17e9bpom1tbWNLSsOPidKwDUajXs2LEDR48eTfQZzWYTjUYj8XEEQQUiQq1Ww8mTJyPvW8Z7PA5EdICZ58O2q2XRGF0sLy+j1WqNvNZqtbC8vJxTi9Lj0KFDvu9tbm5CR4d8/PhxTE9PwzTNxMcShDCYOZaoA8H3g7CdsRL2hYUFrKyswDRNEBFM08TKygoWFhbybpp2ZmZmAt8fDAbKx2o2m77vHTp0CLt371Y+liDkQdj9ILhQidfotnEqApYXXiuzxzE77h93+TExsSKY1Ga3gGKMfaw89iqxsLCAnTt3JjpGq9XC7t27sbS0hM3NTU0tE4Ts2djYwBVXXIHp6Wn0+/28m1N4RNgLQr/fx+zsLGq1GmZnZ9Hv97GxsRH5OHa8vF6vYzAY4JprrvEdhBWEIkJEvu9tbGxgcXFxm7h73T+VRsWt120SihnFLx8+ajqiYRiZLDAtJpa3Oeeu5LkmctagjOmOZWV6ejqWd+6k2WyCmXHixAlNrRKE4kJEW+HFKqVBlzLdsYzEDbk0Gg0YhrGVHbRz504RdWHsqdfrSts5s2T8Qo1VTpEUYc+ZpaUl3/cMw9iWt2+/vm/fPhw5cgTXXXcdACh3Do1GA+12O15jBSFlTp48GZiea3PkyBFMT08HxuMrnSKpEq/RbeMcY9ddQgABccRerxf4eaplB9yrPWVRfybJ2qti1TXDMLZdO7VaLfKykl7r95YByJqn+kljkMavol2tVgvdV0WgvfJ/064RY3cgUjK4mhZ3ScWgRTXiXEtlBCLs+kmjumSSCzNIoIOeJtL02JvN5sgiH3ktsxckHlmt41plsx0e1e1rtZrWjK6yLrwBEXb9pFFdMkln4bdvvV4PDBWpriMZZu70Snt2q2ma3O12t9pXFM/dfnrJu8OJGlYYV8sr9basqY7MIuypkIbHniS8oyJQfsfqdrvKN4qXh9tsNtkwDCYiz7ioVzt03OhEFLmjqNfr276DXq+Xi/BURdSzNvc4UlmBCLt+0poIkXS5P3tf1RVonPuGiZthGJ41a+LUnlHpAILMPo+oHnfQE5WquOuo2yOWjlWpjgxE2NMhqghnuYxXklCRX6el28OMK5B2Bxonq8cOTdmhI+dvoRKScoaVxNRtcnIyk88pc+jFDUTY8yfrqc5prMGq8waMW2HSbku329Wa0dNoNEKP1263mVnWnS26lXWw1A1E2PMn6zVa0+hIdN14Ojx1VXG1t9MxaGsYBjOzxMbHwKrgtUOEPX/yWKNVd+gnSgzaGeawwxf230kGKk3TjBwKsT87qVjYE13yFi2xcKtCSAYi7PmTtseeRfy+1+ttG/BsNBrbhNsr68T5ftKbNo5I6/DY43QqYslscnIydjpq2UMyEGHPnzRj7FnG7+MMGKtmvxiGoSScUUM5umLinU4nd6GrmnW73dhPSWk+DRcBiLAXg7S86qzj91FQDbs44+c6a8tkPdCpa8KXnzWbzUjzDopojUYj8nUR1OH7DcQX4fpPE4iwl5s84vdeRM2k8erkdNaVUQ2b6FoDtt1upx6qabfbY72AirP9qh2gfX34zaHodruVWVzDCUTYy00RPHa/cFDQDWsLuFPYdQmj3amFHU9SF/VaGpO37N/SPYnOORmp2+1uXU/1ep273W5m135eQIS93BRhOTA/AVX1hu32Rk1j9DP7xh7HLJZxLXNsC63usYgwB6UI138eQIS9/ITVak87YyZIaFVDK6qpkK1WayQTp91ub3UgtVqN2+32SKplEbzyiYkJ5e9gHDsjpwDrDEcRUaj3XYQn1jyACHt1ycqbCbqZ7bx2lRs5zMO3i3g5OytnZUlVEW80Gtpi6zpFTOX7TNOSfifM0Wfm2tlQROQ5+Svsei3KGFPWQFHYZWm8ErK0tITBYDDy2mAw8F2Gr9/vY3Z2FrVaDbOzs+j3+0qfs7y87Ll0HwAcP34cAAKXLgOAWq22tSixH/v37wcALC4uYn19HcyMjY2NreUAres9nBMnToR+Vhj1eh1EpLw2ZxjO5dt2794d+n2lQavVQqPRSHwMVYgIl1122dbfx44d27bNYDDAnj17fK9Fv2XvKr0cnhMV9Q8zAG+D1WNOq2wvHnu6RPFmonr37hBP2ml4uhdgSGLO70VHqKfRaBR6URJVi3INEBF3Oh3lMQW/a1Fi7CmHYgDsAvBVAOsQYS8EUeKPUbbNqgJkUc0pGjrCJs7VpsZ1dmutVouUqtrtdiN31H5x8ywrpxYFZCjsnwHwYgBrEGEvBFG8mSjevZ/4JPFe2+32WGSE2MXAgr7jOGaPH+ga7C3KalV+FqcDK3vcPArIQtgBXArgQ8P/ryFA2AEsAlgFsDozM5PBV1BtVLyZoIlBXl5SGpkm9qzKooRb/MxrMQddqzDpWl1qHCzONVT2TJcoQJewA/gagB942KUAbgdwKisIu9PEY8+fII/Tz7v387aSeon2jRslVuteb9XP5ubmtImS+3vRGRc3DGNsY+xRf+sonVgV4uZRQNoeO4BfA/AILEFfA/A0gEMAzgjbV4Q9f4Ieif2WGvML8XhN745idt6yijfnvtGz9nSd3qPuuHgVVmqy01ZVZqtWJW4eBWSdxw7x2AuDShgmTESDshH86r3EFSW7jnvUG1138TAVc8Z7dYemnEXRyui9O8cpwq4XCb94AxH2aqI6cKoiwnFuLlVvzNk2Fa87SkZPFtbpdFL5fPs8k3SUft9z3mMZkraYHMjM02qimr6o4hWGZSN4ee9Ra7GrZIREzejJyubm5rR71u7vPMmx7EW8nb+NfX2kMQO33W4H/tZRriPBGxH2ihJ1clLcx2E/T0vFI3TH8MM8U7+6ITo82qSdg87BWa/vPO7AtH1eTlFPowqjiokHrg8R9ooSpzhSnMfhOKLqV1o17OkhaILKOOTAq5p73dgohcGctXncnVWj0cjt6cbO0xf0IMJeUeLGLKM+DscViqBBWb99/EJCUbzQoPYWoTBYrVbb1kmpPgE5xVNnXF7H9yKTi/Qiwl5hsohZJhEQPw886tNGlDaE5ci7OwidYm8/qcQJq6hkDDnFU1eb3QuWB7U9KDddslv0IsIupEpY+CRICII88ChPG1EW6AiaJWrn0Ts7Q93ZIyrfWVDbgs7VKZ46Swo4jxv220h2SzaIsAupExYDjlKuwHlM1aeNKB67fSw/gXS3SWdMul6vb52bs8NQeSpwpj96dTZu8dTZGbk74LDfRrJb0keEXciEqCGZqBOfgojiAdsipSpiqudlL2Yd1hHEnXTkHmwOW+dTZ4xdwijFQ4RdyAQVwbLzqe36Ll6zVuM+xrs7hLBYb5Q8f5WMG2eH4PfZ9mpBScVV5Xvy2sYr28a9GpXXwK143MVDhF3IjLB8eDvG7SdKUQdNg7x7nbFgleqNzjYGCXvc0I7K0nlenVKcpx8JoxQfEXYhc4KEJ+i9KJOqvAYSvbxW3bFglQ4h6Dz8zj/Mm3fO2KzqOp/CM4iwC5kTJH5xRM/LE1Ud/AxrZxzvNGy/oPNQeZLwysl3Lp8X9clGKB8i7EIu+IlfEtGzCQv3qLYvrbQ8FfEO6hjCxgckpVAQYRcKRVgIRcWLVs3lDiJtrzdJrFol1CKx8Gojwi4UBi9P0554EwU/UbYHZ1UocpxaQi1CGKrCXoMgpMzS0hIGg8HIa8yMG2+8MdJxlpeX0Wq1Rl4jIlx55ZVYWFhQOsbMzEyk17PE6/xarRaWl5dzapEwtqiov24Tj71a6PSSk4YiksbB0ybvzxeKDSQUIxSFooUY/MRTBieFoqMq7GRtmy3z8/O8urqa+ecK+dDv97G4uDgSjmm1WlhZWVEOoWTB7Ows1tfXt71umibW1tayb5AguCCiA8w8H7adxNiF1FlYWMDKygpM0wQRwTTNwok6ABw6dCjS64JQVMRjF4Qh4rELRUc8dkGIiGSlCGVBhF0QhoxLyEgQwpBQjCAIwpggoRhBEISKIsIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZuWTFENFhANtngpSLaQBH8m5Ejsj5y/nL+evHZObTwzbKRdirABGtqqQllRU5fzl/Of/8zl9CMYIgCCVDhF0QBKFkiLCnx0reDcgZOf9qI+efIxJjFwRBKBnisQuCIJQMEXZBEISSIcKeMkT0NiJiIprOuy1ZQkTvJ6J7iehOIvocEZ2Wd5uygIguIqIfEtH9RPTOvNuTJUS0i4huIaK7ieguItqbd5vygIjqRPQ9IvpSXm0QYU8RItoF4LcBVHFttX8A8EJmfhGA+wC8K+f2pA4R1QH8DYDXAJgDcDkRzeXbqkx5GsDbmHkOwCsAvKVi52+zF8A9eTZAhD1dPgjgz2GteF8pmPnvmfnp4Z/fBnBmnu3JiJcDuJ+Zf8TMxwH8LwCX5tymzGDmnzDzd4f//zkscXtuvq3KFiI6E8DFAD6RZztE2FOCiC4F8GNm/n7ebSkAfwzgy3k3IgOeC+ABx98PomLCZkNEswBeCuD2fFuSOX8Ny5nbzLMRE3l++LhDRF8DcIbHW0sA3g0rDFNags6fmT8/3GYJ1iN6P8u2CflBRFMAbgDwZ8z8RN7tyQoiugTAI8x8gIguzLMtIuwJYOZXe71ORL8G4CwA3yciwApDfJeIXs7MD2fYxFTxO38bInojgEsAdLgaEyZ+DGCX4+8zh69VBiJqwBL1PjN/Nu/2ZMz5AP49Ee0GMAngWUTUY+Yrsm6ITFDKACJaAzDPzJWpdkdEFwH4AIALmPlw3u3JAiKagDVQ3IEl6N8B8EfMfFeuDcsIsryY/QAeZeY/y7s9eTL02N/OzJfk8fkSYxfS4moAOwH8AxHdQUTX5N2gtBkOFv9HAF+FNXB4fVVEfcj5AF4P4FXD3/yOofcqZIx47IIgCCVDPHZBEISSIcIuCIJQMkTYBUEQSoYIuyAIQskQYRcEQSgZIuyCIAglQ4RdEAShZPx/7EEwhkbXQPkAAAAASUVORK5CYII=\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAACPCAYAAAAfidZ8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXmsXdWV5r8FMYHEmHjCGGP8jHHwQAzGDjghjkgYhEMqgNLEga4qSk11ulsVpao7ippUq1v1T3VXSqpqtVrd6oqqUpCESlIRICBBIsSYyQNgYwYP2MZ4wMYDZjSEEAi7/3jXO99e9e72uffd6Zzz/SSLdd8+7979znf2uZu7vruWhRAghBBCCCFEHTmu3xMQQgghhBCiX2gzLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK2aDMshBBCCCFqizbDQgghhBCitmgzXBAb5jtm9krj33fMzPo9L9E6ZvY5M1tpZm+Y2a5+z0e0j5l9y8w2mtkRM9tpZt/q95xEe5jZfzSzF8zsTTN7ycz+p5l9qN/zEu1hZieY2RYz29vvuYj2MLO/MLP3zOwt+ndWv+fVDbQZLs7XAFwD4DwACwD8HoB/19cZiXZ5G8D3AGjjVH4MwB8CGA/gSgBfN7Ov9ndKok3uBnBBCGEcgHMxfK/9Rn+nJEbBtwC83O9JiFHzkxDCWPr3Qr8n1A20GXaY2XL3f0HvmtmDAG4E8DchhL0hhH0A/gbAH/VzriJPMy1DCI+HEH4AoJKLuopktPzrEMKTIYT3QwhbAdwF4OJ+z1c0J6PljhDC60cPA/ABgLP7OFWRIfNeCTObCeD3AfyPvk5SFCKnZV3QZtgRQoj/FwTgdAxvmH4EYD6Ap+nQpxs/EwNKRktRMopo2bAtLQWwqQ9TFAXJaWlmN5jZmwAOY/iT4b/r30xFjmOsyf8N4M8BvNOv+YniHEPL3zOzV81sk5n9h/7NsrtoM9wEMzsOwD8BeDCE8HcAxgJ4gw55A8BY+YYHnxG0FCXlGFr+BYbvaf/Y63mJ1hlJyxDCPzVsEh8H8P8AHOzjFEUBvI5mdi2A40MId/Z5aqJFRliT/wxgLoDJAP4tgP9mZtf3cYpdQ5vh5vwlgJPxO8/aWwDG0fg4AG+FEEKvJyZaxmspysuIWprZ1zHsHb4qhPBuPyYmWqbpugwhbMfwJ/z/t9eTEi0TdTSzjwL4a+heW1aSNRlC2BxCeCmE8NsQwmoA/wvAv+rnBLuFvqk7Ao0v4FwP4JMhhPcaP96E4bTd443H50Hp2IGniZaihDTT0sz+DYCbAXw2hKBvrpeAguvyQwBm9W5WolW8jmY2H8AQgEcaSdMTAJxiZgcALAkh7OrXXEWegmsyYNjPXzn0ybDDzBZi2O90TQiBvwn7fQD/ycymmdnpAL4J4JY+TFEUpJmWZnacmZ0IYMzwQzvRzE7o1zzFsclo+a8B/HcAl1f1W85VI6PlH5vZqY14HoBvA1jRn1mKY9FEx40ApgM4v/HvjzFsdTkfwIv9mKc4Npk1ebWZjW+Ulr0Qw58Y39WveXYTfTL8L7kaw2WaHiU78CMAvgDgLADPNn7299CXOwadZlp+B8BKOu4dAA8BuKSXkxMt0UzLOQAmAniCfv7DEMK/7/kMRVGaaXkAwF+a2VgMl+T6KYD/2pcZiiKMqGMIYdnRB2b2KoAPQggH+jA/UZxma/J1DJch/TCAvQC+E0K4tS8z7DImy6sQQgghhKgrskkIIYQQQojaos2wEEIIIYSoLdoMCyGEEEKI2jKqzbCZXWlmW83seTO7uVOTEkIIIYQQohe0/QU6MzsewDYAl2P4W4ZPALg+hLC5c9MTQgghhBCie4ymtNqFAJ4/WtvTzH6M4fIcTTfDkyZNCkNDQ6N4SdEuu3btwuHDhztWLFta9o9Oaikd+8v69esPhxAmd+K5pGV/kZbVQVpWg1beK0ezGZ6GtIj2XgAX5X5haGgI69atG8VLinZZvHhxR59PWvaPTmopHfuLme3u1HNJy/4iLauDtKwGrbxXdv0LdGb2NTNbZ2brXn755WP/ghhYpGU1kI7VQVpWB2lZHaRl+RjNZngfhtsuHuWMxs8SQgjfDSEsDiEsnjy5I1kH0SekZTWQjtVBWlYHaVkdpGX5GM1m+AkAs81sppmdAOCrAO7uzLSEEEIIIYToPm17hkMI75vZ1wHcB+B4AN8LIWzq2MyEEEIIIYToMqP5Ah1CCPcCuLdDcxFCCCGEEKKnjGozLIQQnSBX79yP8WOOzdIKOscdd1zTMdFZiurnj/vggw9G/B3Wzj+Wlp2l3V4D7T5/M809Xmd+nLsGdH38jk5r67XLre2cDkXXcy+1VDtmIYQQQghRW7QZFkIIIYQQtUU2CSFEX8il2PixT80VTa0PSvqtihS1rgCpXrmxdvWSlsUomjLvRGq9nfXrydkkBjHN3i86rVdu7Le//W0yxo9zNomcha1Z7Om2lvpkWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWeYZR3O/2m9/8JsaHDh1Kjtu7d2+MP/zhDydjs2bNivHYsWOTseOPPz7GKh0zeor6UN97770Yv/baa8lxe/bsifH777+fjM2YMSPGEydOTMbGjBkTY3nahimqR27d5TyGufOc88G141uso47NNGpFy6Lll3LnupXSe0Wog5Y5HTrxnP752D/67rvvxvj1119Pjnv11Vdj/NZbbyVj/H74kY98JBmbNGlSjE855ZQY+/dbfg5PWbXtxPXfyv2Q3/fefvvtGPt9z4EDB2Ls30dZl/Hjxydjp59+eoz5ffTEE09MjvvQh5pvUTutpT4ZFkIIIYQQtUWbYSGEEEIIUVtkk2gBTq1v2bIlGbvvvvti7FMCy5cvj7FP/eRSOs3IpTvKmgZql1ZSf81sEjt27EiOu/POO2PM1hgA+NKXvhTjCy+8MBnjlE5RjaqgZdGyaLmyPLkxb1Xh88zWFL+WciWBeF6+nA8/D8e50m2tlHUbZIraVXLlslrpUtXs/LaS0uVji2pU1TJ8ne44lrse/Lp84403YvzCCy/EeO3atclxzz333Ii/41+PrRAAMG/evBh/9rOfjfHs2bOT4z760Y/GuEzaebptjcjdY1955ZUYb9q0KcYPPfRQchy/d3o7DK+xU089NRlbsGBBjJcuXRrjc845JzmObaW5+3sndNYnw0IIIYQQorZoMyyEEEIIIWqLNsNCCCGEEKK21NIz3K7P9Ne//nWMn3zyyeQ49kF534sYDJp5hrdu3Zoct23bthh731rR9rFVJ9dulf1nHHv/NfsF2aMGAC+//HKMubQPAIwbN27E2Pvx2U/steIx/3v+cTOKelwH/TrJeaubacnrB0hLZPlyWfzYexNPOumkGPN5z5VYyrV29WW2TjjhBBShaCnEMpErY5c7Lre2uWTaSy+9lIyxn3TNmjUx3rx5c3Icr+133nknGePX89cA36e5xNd1112XHMfeYn89lJVOeMH92ub7sddyxYoVMX744Ydj7Pc9fA/39wSe88knn5yM7d+/P8Z8DXi9eC/VzverWkGfDAshhBBCiNqizbAQQgghhKgtlbJJdLqsjH9OTtdyKh0ADh48GGMuGwKkKYJcSZ9OpOeKpsbKRLu2llz6jy0v3ibBXXW46xEATJkyJca+O06VtWylxBKnyzgFxp2nAGD79u0x5vI9QKoBWxqAVIPTTjstxj6tymk1rxXr6sv+cGo91wGprOQ6h/lUJ6fFjxw5EmNOdQOplrt3707G3nzzzRh72wLrwLp6exL/nk+XciktX9ayWaeyKtwXgfbtOUU7yf3qV79KxjZu3Bjj22+/PRl74oknYsxpcJ+ez9laOHXv7TZ8jfFzslUKSLub8TVVB4p2BQTSsmh33HFHMvbAAw/EeOfOnSM+H5DanHx3XX5f8K/9/PPPx5jX6LRp05Lj+LG/VtSBTgghhBBCiA6hzbAQQgghhKgtpc8B9rLjDqd+nnnmmeQ4tlBwmgZI0wfeJiFGphO65rTklL3vJsjpOZ9m45S6T9c2S9v4n5e58sBRiqZZeV3s3bs3Oe7ZZ5+NMadAgdSecNZZZyVjs2bNivGECRNi7KtVcIWKXKcr/01nTrvmrC9l0Qoorpe3SbA1gr9xzilxIK0Y8NprryVjfP8788wzkzE+95xy9Vryc/oxtkn46gecWs11K6wiRTuY+dQ33/8ee+yxZOz73/9+0zG2VLDmvF6B1BrjK7ewrcpbEfft2xdj7nC3evXq5LiLL754xNcCqm978tc/rxXfafXWW2+N8b333puMcbUO1ogrdQDpeuZ7MZBq6V97165dI475yiMXXXRRjL0FqtNaamcmhBBCCCFqizbDQgghhBCitmgzLIQQQgghakvpDTTs2+tGaTX2U23YsCHG3v/I3qT58+cnY1wWqNu+wzL5GHsNa8mlXXxpNT6HH//4x5OxnP+7ndJqueMGWcuiZZvYN+ZLbvEa8l63mTNnxvjTn/50MnbGGWfEmL2f7G/1r82lD4HUSzd16tSm889RFT8x43Vgzzd32Xz66aeT49jP6b19Z599dowvuOCCZIw9+Ozp9WW1uBQil90D0nXty+tNnDgxxr70UxVo5fsIPMZlr3yHxwcffDDG7BEGUp8wawKknlH2euZ8pr4s2uuvvx5jvy5XrlwZY76XeJ/p2rVrY7x48eJkrCqe4WY+Ye+n37NnT4x/+MMfJmNcTo09wkBa7ozvv4sWLUqOmzt3boz9uufvaaxfvz4Z4/s2+4f9vorv2957zs/RifvtMT8ZNrPvmdkhM9tIP5tgZveb2fbGf8fnnkMIIYQQQohBpIhN4hYAV7qf3QxgRQhhNoAVjcdCCCGEEEKUimPmDEIID5vZkPvx1QAuacS3AngQwH/u4LzaouhH5a3YKbjU0OOPPx5jn8Y755xzYuxLQdWhjE+nKZqGbwXWctWqVTH23bQ4HeO7CeYsL0Upawq9GblygWxd8Ckw7kzmSyCde+65MZ4xY0Yyxim8XIqQux5xWTAgTZdyGt8/J1M13UbC/+2sH5ez8lYFLos2Z86cZGzJkiUxnj59ejLGZdF4XftrivXyKV0uk8jPBwBDQ0MxbqZrVfH3SbZGcBk0thUAwG233RZjXz4tV3Zy2bJlI8b+/ZA18jrz83st2YrDNgm2VgCp7c2/T/N1Wqb1nCsTyu9rfl3+5Cc/ibHvGMj3Y29xuPTSS2N85ZW/+yzUvx+yNcZbUPjc+5KNL774Yoz5vsLla/0c/XP4jnSjpd0v0E0JIRyd9QEA9ep5KIQQQgghKsGoq0mE4f9lafpRnZl9zczWmdk6/wmcKBfSshpIx+ogLauDtKwO0rJ8tLsZPmhmUwGg8d9DzQ4MIXw3hLA4hLB48uTJbb6cGASkZTWQjtVBWlYHaVkdpGX5aLfOyN0AbgTwV43/3tWxGfUZ781hXyO3IPUtLNknx+V8gNQXJZ/p6ClaTs+PsbdwzZo1MWZvKZB63KZNm5aMsZa50mo5qqBlrqwY+9m4ja73g3HpM9/Kk8sq+ZJY7MFnjdkHCaQle3bu3JmMcYtR36q5bt7S3PXIvj/+hMvf/1g/X1qSS+F5nXkNsafVw9eR939z+SUu1Qak1xjr2o0ynIOGv475PrdxYywOhR/84AfJcewT5vMHAKeffnqMly9fnoxdc801MWavtvd25u7fvC75OYDUo8z3AD9Hvh68Z7gqG1Nef+yZvueee5LjfvrTn8aYfbpA2gr9iiuuSMa+8pWvxPj888+PsS+Fxz5hryXr7PdErDP/Lb6VO++//D2n0xQprfYjAGsAnGNme83sJgxvgi83s+0ALms8FkIIIYQQolQUqSZxfZOhS5v8XAghhBBCiFJQjXYsHcR/1M8pOS7zwSW2AOAzn/lMjH0XpKI2iSqkzwcJn1bZsWNHjLds2RJjb3dYuHBhjDmVBBTvelNnLbkEzuHDh2Psv0jCtgZ/nrmskj+XnE5niwN3FQRSjbdt25aMcQrWd9Li66Yq6fSi1iKfWue0Jevlj/vYxz4WY06lA6nNxZeZ5LlwaTxvXeFOWl5nThP7jpGcQs/pyo9z6d4y4e9/zbqRsWUMSHX2tpMbbrhhxBhIdffvj83InWu+poA0Rc/XkbfXcEc9X46rrPjzxJYXLvnKXeWA9D3P70u4fNr116efeXLnPj7vuTKa/p7AFgq/hnitc5x7/m6vw1FXkxBCCCGEEKKsaDMshBBCCCFqizbDQgghhBCittTSM+y9J+zH8b6XdevWxZi9SL5F7KJFi2Ls2xLmylCJ7uG9ZI888kiM2ZPoPY7s//bet06UySsjrbTHZj8b+4RfeeWV5Dhea74EEvtAvY+MfaDsiVu9enVy3DPPPBNjLrMGpCW+fEk29rCx79J7MMu6rnNz9X8j68D3P1+OkH/Pt8X2jxl+Hm6zzH5vIPUJc/tWIP17eI7+MXvDfbmv3Lrm63vQdeY1xaUkgbQd7y9/+csY+3bG3O78qquuSsa4nBqXzAOAMWPGxLjoefL3DtbBv4/ytcL3du9DZ28sz6nM+PcyLhXJ5dP4ngek54b3KEBaPu2iiy5Kxvg7HDkfb25t8Jhfl7zW+d7vvzvC16LXudPok2EhhBBCCFFbtBkWQgghhBC1pZY2iRw+/bdq1aoY88f+S5YsSY7jTmXtdiYTncWn/1hLTutyKTUAmDVrVoxzlpc6k7MWcUqPY58u57Jdzz33XDLGKTFfEoh15a5KW7duTY7j1Lov1cWdjXJdj7hTkl/XVSyZ6G0Szbo++fPJOnCpJyA9n3498bnn5/AdAzn9y+X6gFQjn45la86RI0difNJJJyXH8fWWSwsPOrzenn766WTs3nvvjTGXDPVWggULFsT42muvTcbYGtFtO6C3L3Fqna9Lb3lh25vXuayWF15DAHD//ffHeOXKlTH21z/bOb/4xS8mYxdffHGMfZfPduyAuRKY3trE92bec40fPz45bvr06TGWTUIIIYQQQoguoc2wEEIIIYSoLbW0SfhvsHI64uDBg8kYp/w4ncQVB4A0VTfo6Zcqw+kz36lq8+bNMWYtP/WpTyXHcapGWrYOnzNOv/kU2P79+2O8YcOGZIxT5j4dy6lATqX6tCqn6fya5/S8t0nwt/D52828xkd6zrLCenmLwMSJE2PMWvq/ndeWrwyycePGGHstuTII/x5bGoDUluFtOVwlwl8DbKngVLPvbuatOGXB68BrgytGAKmNiM8ZawykFkDf0a8TFSNyY5wy3759ezK2e/fuGHPKfMqUKclxbHvjTpaDjj8X3D2P/3YAeOCBB2LMexZfjYGtENxxDkjXQCesQX5dsp1t06ZNyRhXGeK1N2/evOQ4trx0276kT4aFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbamlZ9jDPlNfjob9g+x59B1b2vFSidHjfVZcvmvFihXJGPsQJ02aFGPv/5aWo4NLHbHn0PvB2C/K/mEg9Xr68kj8/OwX9CWWWEfvuWN8OUV+nPPTVuXa4L/Dd1zkkpGsn/cFsz9/7969yRh7sr2HkzXi2PtA2f+9b9++ZCzX/Y6vMcaXaSpzOTWGS8n5bmTeh32U2bNnJ4/5OxS5kltFyX1Hh73LQFpi8ec//3kyxvcI9jmzLxYAli5dGmN/PZcJPjePPfZYMsba8vk888wzk+MuvPDCGPtOq7wG/L2sWZk87wvmtefLLT766KMxXrt2bTLG99izzz47xqwdkL5Pq7SaEEIIIYQQXUKbYSGEEEIIUVtqY5PgVE0utc7dXIA0PccpX+7EA1QnZVoGct3POE3oUzPMueeeG+OZM2cmY6ylv1ak8zA5+wCnVvncLlu2LDmOU2C7du1KxjiN5kuajRs3LsannHJKjP21wL/nS25xWSH//BMmTBjxON+pq6ypdX8N89/hS4zxfY5TmHzegbRTlO/8yM/vSz/xuefn9OltLs3kSybmOtyxfny9eetNOx23BgF/zbO9yNtV+Fj+++fMmZMcxxYVf40XLSfIx3GJMCC1zaxZsyYZu/POO2O8fv36ZIx1Wbx4cYy//OUvJ8dx1zWfWh9kbf25ZduBL03GlhdeQ/y+BgBz586NsV/bufe5Zjp7GxLbSLncGwD8+Mc/jrG/v/O6vOyyy2LsO/vyddpt7cp5NxdCCCGEEKIDaDMshBBCCCFqizbDQgghhBCitlTWM9xKC0j2MHFpFyD1oC1YsCDGZWrzWEaK6seebgDYsmVLjPfs2ZOMsWdq0aJFMfY+RtEa3pfH5/nUU0+NsT/P7E3k9stAWrqLPcJA6pHj9Xno0KHkOH5O9gH7eXH5MD8v/lty/sNcaaJBJ1dajc8b+/d86bPzzjsvxtxqFUjXq9eSy+GxluwDBoADBw7E2Hsf2ZPKvmAg/Z4H/y05/3eZtMy18PXw9cta+vcyfg5fdpBfz58XHuPf8x7ve+65J8a/+MUvkjEugeh1Zj/p8uXLY8z3cv97ZfL151pr+5bx/HfxdzR8a21eU/57E7n7F5dM43l47+99990X45/97GfJGOvu7/1XXHFFjK+77roYn3baaU3nL8+wEEIIIYQQXUKbYSGEEEIIUVsqa5PIwSkAANi5c2eMfScsTifNnz8/xr6ED6NyXO3RTtken8Z7/PHHY+y7T3E5l/PPPz/GOS3FyBTtzMapWZ+C53QmawOkJXx8+Sgm18WO072+Ox2n032ZRE4bd7vr0SDQTC8g1ZatBb4cHZ9Pn47NlUJs1o3Md7jzncoYtl74Lltsh+H5567ZQb9f5+6T/H7FfzuQ2sb4vL/00kvJcc8++2yMfSktTnf7MS6px2XRHn744eS4J598MsY+/c9rz3d5vfHGG2PMpdV8Cj7XWW3QyJV85cf+/sX3Tj7OlzXctm1b09dme4VfX2xL4m53/P4KABs2bGj62tyx9+qrr07GbrrpphhzBzr/d/bS5nLMVzKz6Wa20sw2m9kmM/vTxs8nmNn9Zra98d/xx3ouIYQQQgghBoki2+73AXwzhDAPwBIAf2Jm8wDcDGBFCGE2gBWNx0IIIYQQQpSGY26GQwj7QwhPNuIjALYAmAbgagC3Ng67FcA13ZqkEEIIIYQQ3aAls6SZDQFYCOAxAFNCCEdNegcATGnyawNBzmfKHiY/xqVKuNVhrqzMoPuUyob3UrHnm9svA6n/2+tw5plnxnhoaKjpcaI1cucv5y1m2F8G5Ms7sSeVvW7cvhRIr5NcaTXvV27HJ1yVayhXViznLWbfvfeGsw6+FCJry7p63zEfx15HIC3z5v3f7K2sikaM/5v4uuZSoACwffv2GPNaWbVqVXIcn3u+TwKp79r7urmkHntVfcnDXCm8z33uczFmXymQfmeHvdFlarncCnydex24dTl/V2L16tXJcayRb2/NeO82t/LmEpX+Hsvnmr2/AHDNNb/7fPSGG25Ixnidsk+4FS9/p3Uu7E42s7EAbgfwZyGEpAhkGN6tjOjqN7Ovmdk6M1vn60+KciEtq4F0rA7SsjpIy+ogLctHoc2wmY3B8Eb4thDCHY0fHzSzqY3xqQAOjfS7IYTvhhAWhxAWT548uRNzFn1CWlYD6VgdpGV1kJbVQVqWj2PaJGz4s+h/ALAlhPC3NHQ3gBsB/FXjv3d1ZYbHoGg5Lk7V+RIg3PXGl9nij/59Skf0Bq8xp1p9WSDuXOXLP7GWnJbPpVty11dV0nGdptl58eeSU2J+3eXSY5xm5RSpL8uT05g7HfluZLk5N5tjHeC/N6elPy885tOgXOKLrwGvJdtcuKucH/N2m9ycmx1XJl39XPnvv+yyy5IxTnevWbMmxocPH06Oe+ihh2LMpbOAf3lPZbjUGp9r3xVt9uzZMf785z+fjC1btizGU6dOTcbYfpOzMpW1M6SfG9u3uPseAOzYsSPGK1eujLF/P+QSaV47Xm/evsT7JT7vvlvnJz/5yRhfe+21Tcf8uuTXzt07ekkRz/DFAP4AwLNm9lTjZ3+O4U3wP5vZTQB2A/hKd6YohBBCCCFEdzjmZjiE8CiAZtv1Szs7HSGEEEIIIXpHZVtv+U5H/LH/q6++moy9/fbbMfbfLJ8xY0aMuTuOUuujJ3eecl2rOB3HaSD/e96rNWvWrBizlr3sclM3iq6FVr4Rzt+y5vSeT+HxN6L983GVAU+zOWvtNqdoajqnA997fSe5s846K8ZshQLS9K+32/B9P9ftq0zkrB9cZcF3cOMqEXzOuOMckJ7f3DXP91AAmD59eozZkrZo0aLkOJ5XrvpHK9apZscNOrm/ibU877zzkjGuEsE2L18xgi0wbC8D0vc97uIIpBYVfu1LLrkkOY619R0P2V7RbsfHXmqpXYAQQgghhKgt2gwLIYQQQojaos2wEEIIIYSoLaXzDBf1mebGfNke7kzmPSoXXHBBjLmET87jWKZyLoNEM09fTldfLoa9TtyZCkg7Mp188skxlpadpdN+zKIeVO9L43XO/jsgX6aJHw9K2Z9BoF1di66nXBc79jT6Mk2sl/eZNnutnIexTPh587nwJc2uuuqqGM+dOzfGTz31VHIcd/Hk72cA6f3W31/nzJkTY/6ujdcrt/Zy662sGhXF/318LfuyrpdffnmMuVQdd9MF0hJsvmMg+7N9GbuFCxfGmP36/jtVOS07/T7abf31ybAQQgghhKgt2gwLIYQQQojaUjqbRFH8R+pcfmRoaCgZ+8Y3vhFjX8aLU61FS4WI9mh2Dv255pT30qVLkzHueuN/j1N80rL/FO0OloPXtS/1xKlatsUAaRkoX4KP1zyn/uqWtgU6Y3kp+hx8Pn1XQO4YeOTIkWSMNfKluthewWnnqmiX+zu8ZYTXAFvG5s+fnxzH5ej8+yFrWbTTYCvp805TJp1zc/Xrga/rT3ziEzGeN29echyXnvTrkB/798Bm+rVbIq1dVFpNCCGEEEKIHqDNsBBCCCGEqC3aDAshhBBCiNpSKc9wzl/CvhfvYfKl1tp5ftFZcq16+bEvwcSteos+v+gsRc9tUf+hp5nf218LvrQUw9eJL7vG/ry6l1Zr52/23sScN5zPL5933x6Wy1/6MlN8//ZZ5+vrAAAESElEQVRlvJq1aq7q9wSK/h25MnPenyr6Qzttiv17pb8niubok2EhhBBCCFFbtBkWQgghhBC1pXQ2iVzqoBNlgKqSLisDOtf1pmgaMGeh4NS3tztxKancc7STjhTN8ecsV8Kpmc6+syR3y/K0U8ZLugohGH0yLIQQQgghaos2w0IIIYQQorZoMyyEEEIIIWpL6TzDOeQDE6J61LENcpVoxxvuvb/tPr8QQhRBnwwLIYQQQojaos2wEEIIIYSoLdaJcmSFX8zsZQC7AUwCcLhnL9ycOs1jRghhcqeerKHl26jP+StKqbTUmmxKr+YhLbuPtBw9dZtHp7UclPdKoF5aFtaxp5vh+KJm60IIi3v+wppHRxmUeQ/KPIDBmksrDMq8NY/RMyhz1zxGz6DMXfMYHYM070GZy6DM4yiySQghhBBCiNqizbAQQgghhKgt/doMf7dPr+vRPEbHoMx7UOYBDNZcWmFQ5q15jJ5BmbvmMXoGZe6ax+gYpHkPylwGZR4A+uQZFkIIIYQQYhCQTUIIIYQQQtSWnm6GzexKM9tqZs+b2c09fu3vmdkhM9tIP5tgZveb2fbGf8d3eQ7TzWylmW02s01m9qf9mEcn6JeWg6Bj4zWl5ehfV1p2kLrfXxuvKS1H/9p917IqOgK6v5ZFy55ths3seAD/B8AyAPMAXG9m83r1+gBuAXCl+9nNAFaEEGYDWNF43E3eB/DNEMI8AEsA/EnjHPR6HqOiz1regv7rCEjLTnALpGVH0P01Ii1Hzy3ov5al1xHou5a3oP86AmXRMoTQk38APgXgPnr8bQDf7tXrN15zCMBGerwVwNRGPBXA1h7P5y4Al/d7HmXTctB0lJbSst//+q2jtJSW0nHwtBw0HQdZy17aJKYBeJEe7238rJ9MCSHsb8QHAEzp1Qub2RCAhQAe6+c82mTQtOzr+ZOWHUVatseg6QhIy3aRlkSJdQQGT0utySboC3QNwvD/nvSktIaZjQVwO4A/CyG82a95VJFenz9p2T2kZXWQltVB75XVQGsypZeb4X0AptPjMxo/6ycHzWwqADT+e6jbL2hmYzB8QdwWQrijX/MYJYOmZV/On7TsCtKyPQZNR0Batou0RCV0BAZPS63JJvRyM/wEgNlmNtPMTgDwVQB39/D1R+JuADc24hsx7GXpGmZmAP4BwJYQwt/2ax4dYNC07Pn5k5ZdQ1q2x6DpCEjLdqm9lhXRERg8LbUmm9Fj4/QXAGwDsAPAf+nxa/8IwH4A72HYt3MTgIkY/hbjdgC/BDChy3P4DIZTAc8AeKrx7wu9nkeZtRwEHaWltBzEf3W/v0rL6mhZFR37qeUg6FgmLdWBTgghhBBC1BZ9gU4IIYQQQtQWbYaFEEIIIURt0WZYCCGEEELUFm2GhRBCCCFEbdFmWAghhBBC1BZthoUQQgghRG3RZlgIIYQQQtQWbYaFEEIIIURt+f+UtoefRFpU0wAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "z1 = z[:,0]\n", + "z2 = z[:,1]\n", + "\n", + "fig = plt.figure()\n", + "ax = fig.add_subplot(111)\n", + "ax.plot(z1,z2,'ko')\n", + "plt.title(\"latent space\")\n", + "\n", + "#np.where((z1>3) & (z2<2) & (z2>0))\n", + "#select the points from the latent space\n", + "a_vec = [2,5,7,789,25,9993]\n", + "for i in range(len(a_vec)):\n", + " ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro') \n", + " ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n", + " xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n", + "\n", + "\n", + "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6, sharex='col', sharey='row',figsize=(12,2.5))\n", + "for i in range(len(a_vec)):\n", + " eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", + " eval('ax%d' %(i)).set_title('z%d'%i)\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right." + ] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.2" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/example/bayesian-methods/README.md b/example/bayesian-methods/README.md index ec9e8be86..fc35b9421 100644 --- a/example/bayesian-methods/README.md +++ b/example/bayesian-methods/README.md @@ -11,3 +11,27 @@ and *Bayesian Dark Knowledge (BDK)* [(Balan, Rathod, Murphy and Welling, 2 **bdk.ipynb** shows how to use MXNet to implement the DistilledSGLD algorithm in Bayesian Dark Knowledge. **bdk_demo.py** contains scripts (more than the notebook) related to Bayesian Dark Knowledge. Use `python bdk_demo.py -d 1 -l 2 -t 50000` to run classification on MNIST. + +View parameters we can use with the following command. + +```shell +python bdk_demo.py -h + + +usage: bdk_demo.py [-h] [-d DATASET] [-l ALGORITHM] [-t TRAINING] [--gpu GPU] + +Examples in the paper [NIPS2015]Bayesian Dark Knowledge and [ICML2011]Bayesian +Learning via Stochastic Gradient Langevin Dynamics + +optional arguments: + -h, --help show this help message and exit + -d DATASET, --dataset DATASET + Dataset to use. 0 --> TOY, 1 --> MNIST, 2 --> + Synthetic Data in the SGLD paper + -l ALGORITHM, --algorithm ALGORITHM + Type of algorithm to use. 0 --> SGD, 1 --> SGLD, + other-->DistilledSGLD + -t TRAINING, --training TRAINING + Number of training samples + --gpu GPU 0 to use GPU, not set to use CPU +``` diff --git a/example/bayesian-methods/bdk_demo.py b/example/bayesian-methods/bdk_demo.py index 145dac10e..cd39bfd2a 100644 --- a/example/bayesian-methods/bdk_demo.py +++ b/example/bayesian-methods/bdk_demo.py @@ -156,34 +156,34 @@ def get_toy_sym(teacher=True, teacher_noise_precision=None): return net -def dev(): - return mx.gpu() +def dev(gpu_id=None): + return mx.gpu(gpu_id) if gpu_id else mx.cpu() -def run_mnist_SGD(training_num=50000): +def run_mnist_SGD(training_num=50000, gpu_id=None): X, Y, X_test, Y_test = load_mnist(training_num) minibatch_size = 100 net = get_mnist_sym() data_shape = (minibatch_size,) + X.shape[1::] - data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'softmax_label': nd.zeros((minibatch_size,), ctx=dev())} + data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'softmax_label': nd.zeros((minibatch_size,), ctx=dev(gpu_id))} initializer = mx.init.Xavier(factor_type="in", magnitude=2.34) - exe, exe_params, _ = SGD(sym=net, dev=dev(), data_inputs=data_inputs, X=X, Y=Y, + exe, exe_params, _ = SGD(sym=net, dev=dev(gpu_id), data_inputs=data_inputs, X=X, Y=Y, X_test=X_test, Y_test=Y_test, total_iter_num=1000000, initializer=initializer, lr=5E-6, prior_precision=1.0, minibatch_size=100) -def run_mnist_SGLD(training_num=50000): +def run_mnist_SGLD(training_num=50000, gpu_id=None): X, Y, X_test, Y_test = load_mnist(training_num) minibatch_size = 100 net = get_mnist_sym() data_shape = (minibatch_size,) + X.shape[1::] - data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'softmax_label': nd.zeros((minibatch_size,), ctx=dev())} + data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'softmax_label': nd.zeros((minibatch_size,), ctx=dev(gpu_id))} initializer = mx.init.Xavier(factor_type="in", magnitude=2.34) - exe, sample_pool = SGLD(sym=net, dev=dev(), data_inputs=data_inputs, X=X, Y=Y, + exe, sample_pool = SGLD(sym=net, dev=dev(gpu_id), data_inputs=data_inputs, X=X, Y=Y, X_test=X_test, Y_test=Y_test, total_iter_num=1000000, initializer=initializer, @@ -191,7 +191,7 @@ def run_mnist_SGLD(training_num=50000): thin_interval=100, burn_in_iter_num=1000) -def run_mnist_DistilledSGLD(training_num=50000): +def run_mnist_DistilledSGLD(training_num=50000, gpu_id=None): X, Y, X_test, Y_test = load_mnist(training_num) minibatch_size = 100 if training_num >= 10000: @@ -214,10 +214,10 @@ def run_mnist_DistilledSGLD(training_num=50000): logsoftmax = LogSoftmax() student_net = get_mnist_sym(output_op=logsoftmax, num_hidden=num_hidden) data_shape = (minibatch_size,) + X.shape[1::] - teacher_data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'softmax_label': nd.zeros((minibatch_size,), ctx=dev())} - student_data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'softmax_label': nd.zeros((minibatch_size, 10), ctx=dev())} + teacher_data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'softmax_label': nd.zeros((minibatch_size,), ctx=dev(gpu_id))} + student_data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'softmax_label': nd.zeros((minibatch_size, 10), ctx=dev(gpu_id))} teacher_initializer = BiasXavier(factor_type="in", magnitude=1) student_initializer = BiasXavier(factor_type="in", magnitude=1) student_exe, student_params, _ = \ @@ -231,17 +231,17 @@ def run_mnist_DistilledSGLD(training_num=50000): teacher_learning_rate=teacher_learning_rate, student_learning_rate=student_learning_rate, teacher_prior_precision=teacher_prior, student_prior_precision=student_prior, - perturb_deviation=perturb_deviation, minibatch_size=100, dev=dev()) + perturb_deviation=perturb_deviation, minibatch_size=100, dev=dev(gpu_id)) -def run_toy_SGLD(): +def run_toy_SGLD(gpu_id=None): X, Y, X_test, Y_test = load_toy() minibatch_size = 1 teacher_noise_precision = 1.0 / 9.0 net = get_toy_sym(True, teacher_noise_precision) data_shape = (minibatch_size,) + X.shape[1::] - data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev())} + data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev(gpu_id))} initializer = mx.init.Uniform(0.07) exe, params, _ = \ SGLD(sym=net, data_inputs=data_inputs, @@ -253,20 +253,20 @@ def run_toy_SGLD(): burn_in_iter_num=1000, thin_interval=10, task='regression', - minibatch_size=minibatch_size, dev=dev()) + minibatch_size=minibatch_size, dev=dev(gpu_id)) -def run_toy_DistilledSGLD(): +def run_toy_DistilledSGLD(gpu_id=None): X, Y, X_test, Y_test = load_toy() minibatch_size = 1 teacher_noise_precision = 1.0 teacher_net = get_toy_sym(True, teacher_noise_precision) student_net = get_toy_sym(False) data_shape = (minibatch_size,) + X.shape[1::] - teacher_data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev())} - student_data_inputs = {'data': nd.zeros(data_shape, ctx=dev())} - # 'softmax_label': nd.zeros((minibatch_size, 10), ctx=dev())} + teacher_data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev(gpu_id))} + student_data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id))} + teacher_initializer = mx.init.Uniform(0.07) student_initializer = mx.init.Uniform(0.07) student_grad_f = lambda student_outputs, teacher_pred: \ @@ -284,21 +284,21 @@ def run_toy_DistilledSGLD(): student_grad_f=student_grad_f, teacher_prior_precision=0.1, student_prior_precision=0.001, perturb_deviation=0.1, minibatch_size=minibatch_size, task='regression', - dev=dev()) + dev=dev(gpu_id)) -def run_toy_HMC(): +def run_toy_HMC(gpu_id=None): X, Y, X_test, Y_test = load_toy() minibatch_size = Y.shape[0] noise_precision = 1 / 9.0 net = get_toy_sym(True, noise_precision) data_shape = (minibatch_size,) + X.shape[1::] - data_inputs = {'data': nd.zeros(data_shape, ctx=dev()), - 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev())} + data_inputs = {'data': nd.zeros(data_shape, ctx=dev(gpu_id)), + 'teacher_output_label': nd.zeros((minibatch_size, 1), ctx=dev(gpu_id))} initializer = mx.init.Uniform(0.07) sample_pool = HMC(net, data_inputs=data_inputs, X=X, Y=Y, X_test=X_test, Y_test=Y_test, sample_num=300000, initializer=initializer, prior_precision=1.0, - learning_rate=1E-3, L=10, dev=dev()) + learning_rate=1E-3, L=10, dev=dev(gpu_id)) def run_synthetic_SGLD(): @@ -350,21 +350,22 @@ def run_synthetic_SGLD(): help="Type of algorithm to use. 0 --> SGD, 1 --> SGLD, other-->DistilledSGLD") parser.add_argument("-t", "--training", type=int, default=50000, help="Number of training samples") + parser.add_argument("--gpu", type=int, help="0 to use GPU, not set to use CPU") args = parser.parse_args() training_num = args.training if args.dataset == 1: if 0 == args.algorithm: - run_mnist_SGD(training_num) + run_mnist_SGD(training_num, gpu_id=args.gpu) elif 1 == args.algorithm: - run_mnist_SGLD(training_num) + run_mnist_SGLD(training_num, gpu_id=args.gpu) else: - run_mnist_DistilledSGLD(training_num) + run_mnist_DistilledSGLD(training_num, gpu_id=args.gpu) elif args.dataset == 0: if 1 == args.algorithm: - run_toy_SGLD() + run_toy_SGLD(gpu_id=args.gpu) elif 2 == args.algorithm: - run_toy_DistilledSGLD() + run_toy_DistilledSGLD(gpu_id=args.gpu) elif 3 == args.algorithm: - run_toy_HMC() + run_toy_HMC(gpu_id=args.gpu) else: run_synthetic_SGLD() diff --git a/example/bi-lstm-sort/README.md b/example/bi-lstm-sort/README.md index 3bacc8686..f00cc85ca 100644 --- a/example/bi-lstm-sort/README.md +++ b/example/bi-lstm-sort/README.md @@ -1,24 +1,14 @@ -This is an example of using bidirection lstm to sort an array. +# Bidirectionnal LSTM to sort an array. -Run the training script by doing the following: +This is an example of using bidirectionmal lstm to sort an array. Please refer to the notebook. -``` -python lstm_sort.py --start-range 100 --end-range 1000 --cpu -``` -You can provide the start-range and end-range for the numbers and whether to train on the cpu or not. -By default the script tries to train on the GPU. The default start-range is 100 and end-range is 1000. +We train a bidirectionnal LSTM to sort an array of integer. -At last, test model by doing the following: +For example: -``` -python infer_sort.py 234 189 785 763 231 -``` +`500 30 999 10 130` should give us `10 30 130 500 999` -This should output the sorted seq like the following: -``` -189 -231 -234 -763 -785 -``` +![](https://cdn-images-1.medium.com/max/1200/1*6QnPUSv_t9BY9Fv8_aLb-Q.png) + + +([Diagram source](http://colah.github.io/posts/2015-09-NN-Types-FP/)) \ No newline at end of file diff --git a/example/bi-lstm-sort/bi-lstm-sort.ipynb b/example/bi-lstm-sort/bi-lstm-sort.ipynb new file mode 100644 index 000000000..085117674 --- /dev/null +++ b/example/bi-lstm-sort/bi-lstm-sort.ipynb @@ -0,0 +1,607 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Using a bi-lstm to sort a sequence of integers" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import random\n", + "import string\n", + "\n", + "import mxnet as mx\n", + "from mxnet import gluon, nd\n", + "import numpy as np" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data Preparation" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "max_num = 999\n", + "dataset_size = 60000\n", + "seq_len = 5\n", + "split = 0.8\n", + "batch_size = 512\n", + "ctx = mx.gpu() if len(mx.test_utils.list_gpus()) > 0 else mx.cpu()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are getting a dataset of **dataset_size** sequences of integers of length **seq_len** between **0** and **max_num**. We use **split*100%** of them for training and the rest for testing.\n", + "\n", + "\n", + "For example:\n", + "\n", + "50 10 200 999 30\n", + "\n", + "Should return\n", + "\n", + "10 30 50 200 999" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "X = mx.random.uniform(low=0, high=max_num, shape=(dataset_size, seq_len)).astype('int32').asnumpy()\n", + "Y = X.copy()\n", + "Y.sort() #Let's sort X to get the target" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input [548, 592, 714, 843, 602]\n", + "Target [548, 592, 602, 714, 843]\n" + ] + } + ], + "source": [ + "print(\"Input {}\\nTarget {}\".format(X[0].tolist(), Y[0].tolist()))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For the purpose of training, we encode the input as characters rather than numbers" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0123456789 \n", + "{'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, ' ': 10}\n" + ] + } + ], + "source": [ + "vocab = string.digits + \" \"\n", + "print(vocab)\n", + "vocab_idx = { c:i for i,c in enumerate(vocab)}\n", + "print(vocab_idx)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We write a transform that will convert our numbers into text of maximum length **max_len**, and one-hot encode the characters.\n", + "For example:\n", + "\n", + "\"30 10\" corresponding indices are [3, 0, 10, 1, 0]\n", + "\n", + "We then one hot encode that and get a matrix representation of our input. We don't need to encode our target as the loss we are going to use support sparse labels" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Maximum length of the string: 19\n" + ] + } + ], + "source": [ + "max_len = len(str(max_num))*seq_len+(seq_len-1)\n", + "print(\"Maximum length of the string: %s\" % max_len)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "def transform(x, y):\n", + " x_string = ' '.join(map(str, x.tolist()))\n", + " x_string_padded = x_string + ' '*(max_len-len(x_string))\n", + " x = [vocab_idx[c] for c in x_string_padded]\n", + " y_string = ' '.join(map(str, y.tolist()))\n", + " y_string_padded = y_string + ' '*(max_len-len(y_string))\n", + " y = [vocab_idx[c] for c in y_string_padded]\n", + " return mx.nd.one_hot(mx.nd.array(x), len(vocab)), mx.nd.array(y)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "split_idx = int(split*len(X))\n", + "train_dataset = gluon.data.ArrayDataset(X[:split_idx], Y[:split_idx]).transform(transform)\n", + "test_dataset = gluon.data.ArrayDataset(X[split_idx:], Y[split_idx:]).transform(transform)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input [548 592 714 843 602]\n", + "Transformed data Input \n", + "[[0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n", + " [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n", + " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n", + " [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n", + " [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n", + " [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n", + " [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n", + " [1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n", + " [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]]\n", + "\n", + "Target [548 592 602 714 843]\n", + "Transformed data Target \n", + "[ 5. 4. 8. 10. 5. 9. 2. 10. 6. 0. 2. 10. 7. 1. 4. 10. 8. 4.\n", + " 3.]\n", + "\n" + ] + } + ], + "source": [ + "print(\"Input {}\".format(X[0]))\n", + "print(\"Transformed data Input {}\".format(train_dataset[0][0]))\n", + "print(\"Target {}\".format(Y[0]))\n", + "print(\"Transformed data Target {}\".format(train_dataset[0][1]))" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "train_data = gluon.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=20, last_batch='rollover')\n", + "test_data = gluon.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=False, num_workers=5, last_batch='rollover')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Creating the network" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "net = gluon.nn.HybridSequential()\n", + "with net.name_scope():\n", + " net.add(\n", + " gluon.rnn.LSTM(hidden_size=128, num_layers=2, layout='NTC', bidirectional=True),\n", + " gluon.nn.Dense(len(vocab), flatten=False)\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], + "source": [ + "net.initialize(mx.init.Xavier(), ctx=ctx)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "loss = gluon.loss.SoftmaxCELoss()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We use a learning rate schedule to improve the convergence of the model" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "schedule = mx.lr_scheduler.FactorScheduler(step=len(train_data)*10, factor=0.75)\n", + "schedule.base_lr = 0.01" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate':0.01, 'lr_scheduler':schedule})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training loop" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0] Loss: 1.6627886372227823, LR 0.01\n", + "Epoch [1] Loss: 1.210370733382854, LR 0.01\n", + "Epoch [2] Loss: 0.9692377131035987, LR 0.01\n", + "Epoch [3] Loss: 0.7976046623067653, LR 0.01\n", + "Epoch [4] Loss: 0.5714595343476983, LR 0.01\n", + "Epoch [5] Loss: 0.4458411196444897, LR 0.01\n", + "Epoch [6] Loss: 0.36039798817736035, LR 0.01\n", + "Epoch [7] Loss: 0.32665719377233626, LR 0.01\n", + "Epoch [8] Loss: 0.262064205702915, LR 0.01\n", + "Epoch [9] Loss: 0.22285924059279422, LR 0.0075\n", + "Epoch [10] Loss: 0.19018426854559717, LR 0.0075\n", + "Epoch [11] Loss: 0.1718730723604243, LR 0.0075\n", + "Epoch [12] Loss: 0.15736752171670237, LR 0.0075\n", + "Epoch [13] Loss: 0.14579375246737866, LR 0.0075\n", + "Epoch [14] Loss: 0.13546599733068587, LR 0.0075\n", + "Epoch [15] Loss: 0.12490207590955368, LR 0.0075\n", + "Epoch [16] Loss: 0.11803316300915133, LR 0.0075\n", + "Epoch [17] Loss: 0.10653189395336395, LR 0.0075\n", + "Epoch [18] Loss: 0.10514750379197141, LR 0.0075\n", + "Epoch [19] Loss: 0.09590611559279422, LR 0.005625\n", + "Epoch [20] Loss: 0.08146028108494256, LR 0.005625\n", + "Epoch [21] Loss: 0.07707348782965477, LR 0.005625\n", + "Epoch [22] Loss: 0.07206193436967566, LR 0.005625\n", + "Epoch [23] Loss: 0.07001185417175293, LR 0.005625\n", + "Epoch [24] Loss: 0.06797058351578252, LR 0.005625\n", + "Epoch [25] Loss: 0.0649358110224947, LR 0.005625\n", + "Epoch [26] Loss: 0.06219124286732775, LR 0.005625\n", + "Epoch [27] Loss: 0.06075144828634059, LR 0.005625\n", + "Epoch [28] Loss: 0.05711334495134251, LR 0.005625\n", + "Epoch [29] Loss: 0.054747099572039666, LR 0.00421875\n", + "Epoch [30] Loss: 0.0441775271233092, LR 0.00421875\n", + "Epoch [31] Loss: 0.041551097910454936, LR 0.00421875\n", + "Epoch [32] Loss: 0.04095017269093503, LR 0.00421875\n", + "Epoch [33] Loss: 0.04045371045457556, LR 0.00421875\n", + "Epoch [34] Loss: 0.038867686657195394, LR 0.00421875\n", + "Epoch [35] Loss: 0.038131744303601854, LR 0.00421875\n", + "Epoch [36] Loss: 0.039834817250569664, LR 0.00421875\n", + "Epoch [37] Loss: 0.03669035941996473, LR 0.00421875\n", + "Epoch [38] Loss: 0.03373505967728635, LR 0.00421875\n", + "Epoch [39] Loss: 0.03164981273894615, LR 0.0031640625\n", + "Epoch [40] Loss: 0.025532766055035336, LR 0.0031640625\n", + "Epoch [41] Loss: 0.022659448867148543, LR 0.0031640625\n", + "Epoch [42] Loss: 0.02307056112492338, LR 0.0031640625\n", + "Epoch [43] Loss: 0.02236944056571798, LR 0.0031640625\n", + "Epoch [44] Loss: 0.022204211963120328, LR 0.0031640625\n", + "Epoch [45] Loss: 0.02262336903430046, LR 0.0031640625\n", + "Epoch [46] Loss: 0.02253308448385685, LR 0.0031640625\n", + "Epoch [47] Loss: 0.025286573044797207, LR 0.0031640625\n", + "Epoch [48] Loss: 0.02439300988310127, LR 0.0031640625\n", + "Epoch [49] Loss: 0.017976388018181983, LR 0.002373046875\n", + "Epoch [50] Loss: 0.014343131095805067, LR 0.002373046875\n", + "Epoch [51] Loss: 0.013039355582379281, LR 0.002373046875\n", + "Epoch [52] Loss: 0.011884741885687715, LR 0.002373046875\n", + "Epoch [53] Loss: 0.011438189668858305, LR 0.002373046875\n", + "Epoch [54] Loss: 0.011447292693117832, LR 0.002373046875\n", + "Epoch [55] Loss: 0.014212571560068334, LR 0.002373046875\n", + "Epoch [56] Loss: 0.019900493724371797, LR 0.002373046875\n", + "Epoch [57] Loss: 0.02102568301748722, LR 0.002373046875\n", + "Epoch [58] Loss: 0.01346214400961044, LR 0.002373046875\n", + "Epoch [59] Loss: 0.010107964911359422, LR 0.0017797851562500002\n", + "Epoch [60] Loss: 0.008353193600972494, LR 0.0017797851562500002\n", + "Epoch [61] Loss: 0.007678258292218472, LR 0.0017797851562500002\n", + "Epoch [62] Loss: 0.007262124660167288, LR 0.0017797851562500002\n", + "Epoch [63] Loss: 0.00705223578087827, LR 0.0017797851562500002\n", + "Epoch [64] Loss: 0.006788556293774677, LR 0.0017797851562500002\n", + "Epoch [65] Loss: 0.006473606571238091, LR 0.0017797851562500002\n", + "Epoch [66] Loss: 0.006206096486842378, LR 0.0017797851562500002\n", + "Epoch [67] Loss: 0.00584477313021396, LR 0.0017797851562500002\n", + "Epoch [68] Loss: 0.005648705267137097, LR 0.0017797851562500002\n", + "Epoch [69] Loss: 0.006481769871204458, LR 0.0013348388671875003\n", + "Epoch [70] Loss: 0.008430448618341, LR 0.0013348388671875003\n", + "Epoch [71] Loss: 0.006877245421105242, LR 0.0013348388671875003\n", + "Epoch [72] Loss: 0.005671108281740578, LR 0.0013348388671875003\n", + "Epoch [73] Loss: 0.004832422162624116, LR 0.0013348388671875003\n", + "Epoch [74] Loss: 0.004441103402604448, LR 0.0013348388671875003\n", + "Epoch [75] Loss: 0.004216198591475791, LR 0.0013348388671875003\n", + "Epoch [76] Loss: 0.004041922989711967, LR 0.0013348388671875003\n", + "Epoch [77] Loss: 0.003937713643337818, LR 0.0013348388671875003\n", + "Epoch [78] Loss: 0.010251983049068046, LR 0.0013348388671875003\n", + "Epoch [79] Loss: 0.01829354052848004, LR 0.0010011291503906252\n", + "Epoch [80] Loss: 0.006723233448561802, LR 0.0010011291503906252\n", + "Epoch [81] Loss: 0.004397524798170049, LR 0.0010011291503906252\n", + "Epoch [82] Loss: 0.0038475305476087206, LR 0.0010011291503906252\n", + "Epoch [83] Loss: 0.003591177945441388, LR 0.0010011291503906252\n", + "Epoch [84] Loss: 0.003425112014175743, LR 0.0010011291503906252\n", + "Epoch [85] Loss: 0.0032633850549129728, LR 0.0010011291503906252\n", + "Epoch [86] Loss: 0.0031762316505959693, LR 0.0010011291503906252\n", + "Epoch [87] Loss: 0.0030452777096565734, LR 0.0010011291503906252\n", + "Epoch [88] Loss: 0.002950224184220837, LR 0.0010011291503906252\n", + "Epoch [89] Loss: 0.002821172171450676, LR 0.0007508468627929689\n", + "Epoch [90] Loss: 0.002725780961361337, LR 0.0007508468627929689\n", + "Epoch [91] Loss: 0.002660556359493986, LR 0.0007508468627929689\n", + "Epoch [92] Loss: 0.0026011724946319414, LR 0.0007508468627929689\n", + "Epoch [93] Loss: 0.0025355776256703317, LR 0.0007508468627929689\n", + "Epoch [94] Loss: 0.0024825221997626283, LR 0.0007508468627929689\n", + "Epoch [95] Loss: 0.0024245587435174497, LR 0.0007508468627929689\n", + "Epoch [96] Loss: 0.002365282145879602, LR 0.0007508468627929689\n", + "Epoch [97] Loss: 0.0023112583984719946, LR 0.0007508468627929689\n", + "Epoch [98] Loss: 0.002257173682780976, LR 0.0007508468627929689\n", + "Epoch [99] Loss: 0.002162747085094452, LR 0.0005631351470947267\n" + ] + } + ], + "source": [ + "epochs = 100\n", + "for e in range(epochs):\n", + " epoch_loss = 0.\n", + " for i, (data, label) in enumerate(train_data):\n", + " data = data.as_in_context(ctx)\n", + " label = label.as_in_context(ctx)\n", + "\n", + " with mx.autograd.record():\n", + " output = net(data)\n", + " l = loss(output, label)\n", + "\n", + " l.backward()\n", + " trainer.step(data.shape[0])\n", + " \n", + " epoch_loss += l.mean()\n", + " \n", + " print(\"Epoch [{}] Loss: {}, LR {}\".format(e, epoch_loss.asscalar()/(i+1), trainer.learning_rate))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Testing" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We get a random element from the testing set" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [], + "source": [ + "n = random.randint(0, len(test_data)-1)\n", + "\n", + "x_orig = X[split_idx+n]\n", + "y_orig = Y[split_idx+n]" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": {}, + "outputs": [], + "source": [ + "def get_pred(x):\n", + " x, _ = transform(x, x)\n", + " output = net(x.as_in_context(ctx).expand_dims(axis=0))\n", + "\n", + " # Convert output back to string\n", + " pred = ''.join([vocab[int(o)] for o in output[0].argmax(axis=1).asnumpy().tolist()])\n", + " return pred" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Printing the result" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "X 611 671 275 871 944\n", + "Predicted 275 611 671 871 944\n", + "Label 275 611 671 871 944\n" + ] + } + ], + "source": [ + "x_ = ' '.join(map(str,x_orig))\n", + "label = ' '.join(map(str,y_orig))\n", + "print(\"X {}\\nPredicted {}\\nLabel {}\".format(x_, get_pred(x_orig), label))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can also pick our own example, and the network manages to sort it without problem:" + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "10 30 130 500 999 \n" + ] + } + ], + "source": [ + "print(get_pred(np.array([500, 30, 999, 10, 130])))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The model has even learned to generalize to examples not on the training set" + ] + }, + { + "cell_type": "code", + "execution_count": 64, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Only four numbers: 105 202 302 501 \n" + ] + } + ], + "source": [ + "print(\"Only four numbers:\", get_pred(np.array([105, 302, 501, 202])))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "However we can see it has trouble with other edge cases:" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Small digits: 8 0 42 28 \n", + "Small digits, 6 numbers: 10 0 20 82 71 115 \n" + ] + } + ], + "source": [ + "print(\"Small digits:\", get_pred(np.array([10, 3, 5, 2, 8])))\n", + "print(\"Small digits, 6 numbers:\", get_pred(np.array([10, 33, 52, 21, 82, 10])))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This could be improved by adjusting the training dataset accordingly" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/example/bi-lstm-sort/infer_sort.py b/example/bi-lstm-sort/infer_sort.py deleted file mode 100644 index f81c6c0ec..000000000 --- a/example/bi-lstm-sort/infer_sort.py +++ /dev/null @@ -1,80 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=C0111,too-many-arguments,too-many-instance-attributes,too-many-locals,redefined-outer-name,fixme -# pylint: disable=superfluous-parens, no-member, invalid-name -import sys -import os -import argparse -import numpy as np -import mxnet as mx - -from sort_io import BucketSentenceIter, default_build_vocab -from rnn_model import BiLSTMInferenceModel - -TRAIN_FILE = "sort.train.txt" -TEST_FILE = "sort.test.txt" -VALID_FILE = "sort.valid.txt" -DATA_DIR = os.path.join(os.getcwd(), "data") -SEQ_LEN = 5 - -def MakeInput(char, vocab, arr): - idx = vocab[char] - tmp = np.zeros((1,)) - tmp[0] = idx - arr[:] = tmp - -def main(): - tks = sys.argv[1:] - assert len(tks) >= 5, "Please provide 5 numbers for sorting as sequence length is 5" - batch_size = 1 - buckets = [] - num_hidden = 300 - num_embed = 512 - num_lstm_layer = 2 - - num_epoch = 1 - learning_rate = 0.1 - momentum = 0.9 - - contexts = [mx.context.cpu(i) for i in range(1)] - - vocab = default_build_vocab(os.path.join(DATA_DIR, TRAIN_FILE)) - rvocab = {} - for k, v in vocab.items(): - rvocab[v] = k - - _, arg_params, __ = mx.model.load_checkpoint("sort", 1) - for tk in tks: - assert (tk in vocab), "{} not in range of numbers that the model trained for.".format(tk) - - model = BiLSTMInferenceModel(SEQ_LEN, len(vocab), - num_hidden=num_hidden, num_embed=num_embed, - num_label=len(vocab), arg_params=arg_params, ctx=contexts, dropout=0.0) - - data = np.zeros((1, len(tks))) - for k in range(len(tks)): - data[0][k] = vocab[tks[k]] - - data = mx.nd.array(data) - prob = model.forward(data) - for k in range(len(tks)): - print(rvocab[np.argmax(prob, axis = 1)[k]]) - - -if __name__ == '__main__': - sys.exit(main()) diff --git a/example/bi-lstm-sort/lstm.py b/example/bi-lstm-sort/lstm.py deleted file mode 100644 index 362481dd0..000000000 --- a/example/bi-lstm-sort/lstm.py +++ /dev/null @@ -1,175 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint:skip-file -import sys -import mxnet as mx -import numpy as np -from collections import namedtuple -import time -import math -LSTMState = namedtuple("LSTMState", ["c", "h"]) -LSTMParam = namedtuple("LSTMParam", ["i2h_weight", "i2h_bias", - "h2h_weight", "h2h_bias"]) -LSTMModel = namedtuple("LSTMModel", ["rnn_exec", "symbol", - "init_states", "last_states", "forward_state", "backward_state", - "seq_data", "seq_labels", "seq_outputs", - "param_blocks"]) - -def lstm(num_hidden, indata, prev_state, param, seqidx, layeridx, dropout=0.): - """LSTM Cell symbol""" - if dropout > 0.: - indata = mx.sym.Dropout(data=indata, p=dropout) - i2h = mx.sym.FullyConnected(data=indata, - weight=param.i2h_weight, - bias=param.i2h_bias, - num_hidden=num_hidden * 4, - name="t%d_l%d_i2h" % (seqidx, layeridx)) - h2h = mx.sym.FullyConnected(data=prev_state.h, - weight=param.h2h_weight, - bias=param.h2h_bias, - num_hidden=num_hidden * 4, - name="t%d_l%d_h2h" % (seqidx, layeridx)) - gates = i2h + h2h - slice_gates = mx.sym.SliceChannel(gates, num_outputs=4, - name="t%d_l%d_slice" % (seqidx, layeridx)) - in_gate = mx.sym.Activation(slice_gates[0], act_type="sigmoid") - in_transform = mx.sym.Activation(slice_gates[1], act_type="tanh") - forget_gate = mx.sym.Activation(slice_gates[2], act_type="sigmoid") - out_gate = mx.sym.Activation(slice_gates[3], act_type="sigmoid") - next_c = (forget_gate * prev_state.c) + (in_gate * in_transform) - next_h = out_gate * mx.sym.Activation(next_c, act_type="tanh") - return LSTMState(c=next_c, h=next_h) - - -def bi_lstm_unroll(seq_len, input_size, - num_hidden, num_embed, num_label, dropout=0.): - - embed_weight = mx.sym.Variable("embed_weight") - cls_weight = mx.sym.Variable("cls_weight") - cls_bias = mx.sym.Variable("cls_bias") - last_states = [] - last_states.append(LSTMState(c = mx.sym.Variable("l0_init_c"), h = mx.sym.Variable("l0_init_h"))) - last_states.append(LSTMState(c = mx.sym.Variable("l1_init_c"), h = mx.sym.Variable("l1_init_h"))) - forward_param = LSTMParam(i2h_weight=mx.sym.Variable("l0_i2h_weight"), - i2h_bias=mx.sym.Variable("l0_i2h_bias"), - h2h_weight=mx.sym.Variable("l0_h2h_weight"), - h2h_bias=mx.sym.Variable("l0_h2h_bias")) - backward_param = LSTMParam(i2h_weight=mx.sym.Variable("l1_i2h_weight"), - i2h_bias=mx.sym.Variable("l1_i2h_bias"), - h2h_weight=mx.sym.Variable("l1_h2h_weight"), - h2h_bias=mx.sym.Variable("l1_h2h_bias")) - - # embeding layer - data = mx.sym.Variable('data') - label = mx.sym.Variable('softmax_label') - embed = mx.sym.Embedding(data=data, input_dim=input_size, - weight=embed_weight, output_dim=num_embed, name='embed') - wordvec = mx.sym.SliceChannel(data=embed, num_outputs=seq_len, squeeze_axis=1) - - forward_hidden = [] - for seqidx in range(seq_len): - hidden = wordvec[seqidx] - next_state = lstm(num_hidden, indata=hidden, - prev_state=last_states[0], - param=forward_param, - seqidx=seqidx, layeridx=0, dropout=dropout) - hidden = next_state.h - last_states[0] = next_state - forward_hidden.append(hidden) - - backward_hidden = [] - for seqidx in range(seq_len): - k = seq_len - seqidx - 1 - hidden = wordvec[k] - next_state = lstm(num_hidden, indata=hidden, - prev_state=last_states[1], - param=backward_param, - seqidx=k, layeridx=1,dropout=dropout) - hidden = next_state.h - last_states[1] = next_state - backward_hidden.insert(0, hidden) - - hidden_all = [] - for i in range(seq_len): - hidden_all.append(mx.sym.Concat(*[forward_hidden[i], backward_hidden[i]], dim=1)) - - hidden_concat = mx.sym.Concat(*hidden_all, dim=0) - pred = mx.sym.FullyConnected(data=hidden_concat, num_hidden=num_label, - weight=cls_weight, bias=cls_bias, name='pred') - - label = mx.sym.transpose(data=label) - label = mx.sym.Reshape(data=label, target_shape=(0,)) - sm = mx.sym.SoftmaxOutput(data=pred, label=label, name='softmax') - - return sm - - -def bi_lstm_inference_symbol(input_size, seq_len, - num_hidden, num_embed, num_label, dropout=0.): - seqidx = 0 - embed_weight=mx.sym.Variable("embed_weight") - cls_weight = mx.sym.Variable("cls_weight") - cls_bias = mx.sym.Variable("cls_bias") - last_states = [LSTMState(c = mx.sym.Variable("l0_init_c"), h = mx.sym.Variable("l0_init_h")), - LSTMState(c = mx.sym.Variable("l1_init_c"), h = mx.sym.Variable("l1_init_h"))] - forward_param = LSTMParam(i2h_weight=mx.sym.Variable("l0_i2h_weight"), - i2h_bias=mx.sym.Variable("l0_i2h_bias"), - h2h_weight=mx.sym.Variable("l0_h2h_weight"), - h2h_bias=mx.sym.Variable("l0_h2h_bias")) - backward_param = LSTMParam(i2h_weight=mx.sym.Variable("l1_i2h_weight"), - i2h_bias=mx.sym.Variable("l1_i2h_bias"), - h2h_weight=mx.sym.Variable("l1_h2h_weight"), - h2h_bias=mx.sym.Variable("l1_h2h_bias")) - data = mx.sym.Variable("data") - embed = mx.sym.Embedding(data=data, input_dim=input_size, - weight=embed_weight, output_dim=num_embed, name='embed') - wordvec = mx.sym.SliceChannel(data=embed, num_outputs=seq_len, squeeze_axis=1) - forward_hidden = [] - for seqidx in range(seq_len): - next_state = lstm(num_hidden, indata=wordvec[seqidx], - prev_state=last_states[0], - param=forward_param, - seqidx=seqidx, layeridx=0, dropout=0.0) - hidden = next_state.h - last_states[0] = next_state - forward_hidden.append(hidden) - - backward_hidden = [] - for seqidx in range(seq_len): - k = seq_len - seqidx - 1 - next_state = lstm(num_hidden, indata=wordvec[k], - prev_state=last_states[1], - param=backward_param, - seqidx=k, layeridx=1, dropout=0.0) - hidden = next_state.h - last_states[1] = next_state - backward_hidden.insert(0, hidden) - - hidden_all = [] - for i in range(seq_len): - hidden_all.append(mx.sym.Concat(*[forward_hidden[i], backward_hidden[i]], dim=1)) - hidden_concat = mx.sym.Concat(*hidden_all, dim=0) - fc = mx.sym.FullyConnected(data=hidden_concat, num_hidden=num_label, - weight=cls_weight, bias=cls_bias, name='pred') - sm = mx.sym.SoftmaxOutput(data=fc, name='softmax') - output = [sm] - for state in last_states: - output.append(state.c) - output.append(state.h) - return mx.sym.Group(output) - diff --git a/example/bi-lstm-sort/lstm_sort.py b/example/bi-lstm-sort/lstm_sort.py deleted file mode 100644 index 3d7090a9a..000000000 --- a/example/bi-lstm-sort/lstm_sort.py +++ /dev/null @@ -1,142 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=C0111,too-many-arguments,too-many-instance-attributes,too-many-locals,redefined-outer-name,fixme -# pylint: disable=superfluous-parens, no-member, invalid-name -import argparse -import errno -import logging -import os -import random -import sys - -import numpy as np - -import mxnet as mx -from lstm import bi_lstm_unroll -from sort_io import BucketSentenceIter, default_build_vocab - -head = '%(asctime)-15s %(message)s' -logging.basicConfig(level=logging.DEBUG, format=head) - -TRAIN_FILE = "sort.train.txt" -TEST_FILE = "sort.test.txt" -VALID_FILE = "sort.valid.txt" -DATA_DIR = os.path.join(os.getcwd(), "data") -SEQ_LEN = 5 - - -def gen_data(seq_len, start_range, end_range): - if not os.path.exists(DATA_DIR): - try: - logging.info('create directory %s', DATA_DIR) - os.makedirs(DATA_DIR) - except OSError as exc: - if exc.errno != errno.EEXIST: - raise OSError('failed to create ' + DATA_DIR) - vocab = [str(x) for x in range(start_range, end_range)] - sw_train = open(os.path.join(DATA_DIR, TRAIN_FILE), "w") - sw_test = open(os.path.join(DATA_DIR, TEST_FILE), "w") - sw_valid = open(os.path.join(DATA_DIR, VALID_FILE), "w") - - for i in range(1000000): - seq = " ".join([vocab[random.randint(0, len(vocab) - 1)] for j in range(seq_len)]) - k = i % 50 - if k == 0: - sw_test.write(seq + "\n") - elif k == 1: - sw_valid.write(seq + "\n") - else: - sw_train.write(seq + "\n") - - sw_train.close() - sw_test.close() - -def parse_args(): - parser = argparse.ArgumentParser(description="Parse args for lstm_sort example", - formatter_class=argparse.ArgumentDefaultsHelpFormatter) - parser.add_argument('--start-range', type=int, default=100, - help='starting number of the range') - parser.add_argument('--end-range', type=int, default=1000, - help='Ending number of the range') - parser.add_argument('--cpu', action='store_true', - help='To use CPU for training') - return parser.parse_args() - - -def Perplexity(label, pred): - label = label.T.reshape((-1,)) - loss = 0. - for i in range(pred.shape[0]): - loss += -np.log(max(1e-10, pred[i][int(label[i])])) - return np.exp(loss / label.size) - -def main(): - args = parse_args() - gen_data(SEQ_LEN, args.start_range, args.end_range) - batch_size = 100 - buckets = [] - num_hidden = 300 - num_embed = 512 - num_lstm_layer = 2 - - num_epoch = 1 - learning_rate = 0.1 - momentum = 0.9 - - if args.cpu: - contexts = [mx.context.cpu(i) for i in range(1)] - else: - contexts = [mx.context.gpu(i) for i in range(1)] - - vocab = default_build_vocab(os.path.join(DATA_DIR, TRAIN_FILE)) - - def sym_gen(seq_len): - return bi_lstm_unroll(seq_len, len(vocab), - num_hidden=num_hidden, num_embed=num_embed, - num_label=len(vocab)) - - init_c = [('l%d_init_c'%l, (batch_size, num_hidden)) for l in range(num_lstm_layer)] - init_h = [('l%d_init_h'%l, (batch_size, num_hidden)) for l in range(num_lstm_layer)] - init_states = init_c + init_h - - data_train = BucketSentenceIter(os.path.join(DATA_DIR, TRAIN_FILE), vocab, - buckets, batch_size, init_states) - data_val = BucketSentenceIter(os.path.join(DATA_DIR, VALID_FILE), vocab, - buckets, batch_size, init_states) - - if len(buckets) == 1: - symbol = sym_gen(buckets[0]) - else: - symbol = sym_gen - - model = mx.model.FeedForward(ctx=contexts, - symbol=symbol, - num_epoch=num_epoch, - learning_rate=learning_rate, - momentum=momentum, - wd=0.00001, - initializer=mx.init.Xavier(factor_type="in", magnitude=2.34)) - - model.fit(X=data_train, eval_data=data_val, - eval_metric = mx.metric.np(Perplexity), - batch_end_callback=mx.callback.Speedometer(batch_size, 50),) - - model.save("sort") - -if __name__ == '__main__': - sys.exit(main()) diff --git a/example/bi-lstm-sort/rnn_model.py b/example/bi-lstm-sort/rnn_model.py deleted file mode 100644 index 1079e9099..000000000 --- a/example/bi-lstm-sort/rnn_model.py +++ /dev/null @@ -1,73 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=C0111,too-many-arguments,too-many-instance-attributes,too-many-locals,redefined-outer-name,fixme -# pylint: disable=superfluous-parens, no-member, invalid-name -import sys -import numpy as np -import mxnet as mx - -from lstm import LSTMState, LSTMParam, lstm, bi_lstm_inference_symbol - -class BiLSTMInferenceModel(object): - def __init__(self, - seq_len, - input_size, - num_hidden, - num_embed, - num_label, - arg_params, - ctx=mx.cpu(), - dropout=0.): - self.sym = bi_lstm_inference_symbol(input_size, seq_len, - num_hidden, - num_embed, - num_label, - dropout) - batch_size = 1 - init_c = [('l%d_init_c'%l, (batch_size, num_hidden)) for l in range(2)] - init_h = [('l%d_init_h'%l, (batch_size, num_hidden)) for l in range(2)] - - data_shape = [("data", (batch_size, seq_len, ))] - - input_shapes = dict(init_c + init_h + data_shape) - self.executor = self.sym.simple_bind(ctx=mx.cpu(), **input_shapes) - - for key in self.executor.arg_dict.keys(): - if key in arg_params: - arg_params[key].copyto(self.executor.arg_dict[key]) - - state_name = [] - for i in range(2): - state_name.append("l%d_init_c" % i) - state_name.append("l%d_init_h" % i) - - self.states_dict = dict(zip(state_name, self.executor.outputs[1:])) - self.input_arr = mx.nd.zeros(data_shape[0][1]) - - def forward(self, input_data, new_seq=False): - if new_seq == True: - for key in self.states_dict.keys(): - self.executor.arg_dict[key][:] = 0. - input_data.copyto(self.executor.arg_dict["data"]) - self.executor.forward() - for key in self.states_dict.keys(): - self.states_dict[key].copyto(self.executor.arg_dict[key]) - prob = self.executor.outputs[0].asnumpy() - return prob - - diff --git a/example/bi-lstm-sort/sort_io.py b/example/bi-lstm-sort/sort_io.py deleted file mode 100644 index 853d0ee87..000000000 --- a/example/bi-lstm-sort/sort_io.py +++ /dev/null @@ -1,255 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=C0111,too-many-arguments,too-many-instance-attributes,too-many-locals,redefined-outer-name,fixme -# pylint: disable=superfluous-parens, no-member, invalid-name -from __future__ import print_function -import sys -import numpy as np -import mxnet as mx - -# The interface of a data iter that works for bucketing -# -# DataIter -# - default_bucket_key: the bucket key for the default symbol. -# -# DataBatch -# - provide_data: same as DataIter, but specific to this batch -# - provide_label: same as DataIter, but specific to this batch -# - bucket_key: the key for the bucket that should be used for this batch - -def default_read_content(path): - with open(path) as ins: - content = ins.read() - content = content.replace('\n', ' ').replace('. ', ' ') - return content - -def default_build_vocab(path): - content = default_read_content(path) - content = content.split(' ') - - words = set([x for x in content if len(x) > 0]) - words = [x for x in words] - words = sorted(words) - the_vocab = {} - idx = 1 # 0 is left for zero-padding - the_vocab[' '] = 0 # put a dummy element here so that len(vocab) is correct - for word in words: - if len(word) == 0: - continue - if not word in the_vocab: - the_vocab[word] = idx - idx += 1 - return the_vocab - -def default_text2id(sentence, the_vocab): - words = sentence.split(' ') - words = [the_vocab[w] for w in words if len(w) > 0] - return words - -def default_gen_buckets(sentences, batch_size, the_vocab): - len_dict = {} - max_len = -1 - for sentence in sentences: - words = default_text2id(sentence, the_vocab) - lw = len(words) - if lw == 0: - continue - if lw > max_len: - max_len = lw - if lw in len_dict: - len_dict[lw] += 1 - else: - len_dict[lw] = 1 - print(len_dict) - - tl = 0 - buckets = [] - for l, n in len_dict.items(): # TODO: There are better heuristic ways to do this - if n + tl >= batch_size: - buckets.append(l) - tl = 0 - else: - tl += n - if tl > 0: - buckets.append(max_len) - return buckets - - -class SimpleBatch(object): - def __init__(self, data_names, data, label_names, label, bucket_key): - self.data = data - self.label = label - self.data_names = data_names - self.label_names = label_names - self.bucket_key = bucket_key - - self.pad = 0 - self.index = None # TODO: what is index? - - @property - def provide_data(self): - return [(n, x.shape) for n, x in zip(self.data_names, self.data)] - - @property - def provide_label(self): - return [(n, x.shape) for n, x in zip(self.label_names, self.label)] - -class DummyIter(mx.io.DataIter): - "A dummy iterator that always return the same batch, used for speed testing" - def __init__(self, real_iter): - super(DummyIter, self).__init__() - self.real_iter = real_iter - self.provide_data = real_iter.provide_data - self.provide_label = real_iter.provide_label - self.batch_size = real_iter.batch_size - - for batch in real_iter: - self.the_batch = batch - break - - def __iter__(self): - return self - - def next(self): - return self.the_batch - -class BucketSentenceIter(mx.io.DataIter): - def __init__(self, path, vocab, buckets, batch_size, - init_states, data_name='data', label_name='label', - seperate_char=' ', text2id=None, read_content=None): - super(BucketSentenceIter, self).__init__() - - if text2id is None: - self.text2id = default_text2id - else: - self.text2id = text2id - if read_content is None: - self.read_content = default_read_content - else: - self.read_content = read_content - content = self.read_content(path) - sentences = content.split(seperate_char) - - if len(buckets) == 0: - buckets = default_gen_buckets(sentences, batch_size, vocab) - print(buckets) - self.vocab_size = len(vocab) - self.data_name = data_name - self.label_name = label_name - - buckets.sort() - self.buckets = buckets - self.data = [[] for _ in buckets] - - # pre-allocate with the largest bucket for better memory sharing - self.default_bucket_key = max(buckets) - - for sentence in sentences: - sentence = self.text2id(sentence, vocab) - if len(sentence) == 0: - continue - for i, bkt in enumerate(buckets): - if bkt >= len(sentence): - self.data[i].append(sentence) - break - # we just ignore the sentence it is longer than the maximum - # bucket size here - - # convert data into ndarrays for better speed during training - data = [np.zeros((len(x), buckets[i])) for i, x in enumerate(self.data)] - for i_bucket in range(len(self.buckets)): - for j in range(len(self.data[i_bucket])): - sentence = self.data[i_bucket][j] - data[i_bucket][j, :len(sentence)] = sentence - self.data = data - - # Get the size of each bucket, so that we could sample - # uniformly from the bucket - bucket_sizes = [len(x) for x in self.data] - - print("Summary of dataset ==================") - for bkt, size in zip(buckets, bucket_sizes): - print("bucket of len %3d : %d samples" % (bkt, size)) - - self.batch_size = batch_size - self.make_data_iter_plan() - - self.init_states = init_states - self.init_state_arrays = [mx.nd.zeros(x[1]) for x in init_states] - - self.provide_data = [('data', (batch_size, self.default_bucket_key))] + init_states - self.provide_label = [('softmax_label', (self.batch_size, self.default_bucket_key))] - - def make_data_iter_plan(self): - "make a random data iteration plan" - # truncate each bucket into multiple of batch-size - bucket_n_batches = [] - for i in range(len(self.data)): - bucket_n_batches.append(len(self.data[i]) / self.batch_size) - self.data[i] = self.data[i][:int(bucket_n_batches[i]*self.batch_size)] - - bucket_plan = np.hstack([np.zeros(n, int)+i for i, n in enumerate(bucket_n_batches)]) - np.random.shuffle(bucket_plan) - - bucket_idx_all = [np.random.permutation(len(x)) for x in self.data] - - self.bucket_plan = bucket_plan - self.bucket_idx_all = bucket_idx_all - self.bucket_curr_idx = [0 for x in self.data] - - self.data_buffer = [] - self.label_buffer = [] - for i_bucket in range(len(self.data)): - data = np.zeros((self.batch_size, self.buckets[i_bucket])) - label = np.zeros((self.batch_size, self.buckets[i_bucket])) - self.data_buffer.append(data) - self.label_buffer.append(label) - - def __iter__(self): - init_state_names = [x[0] for x in self.init_states] - - for i_bucket in self.bucket_plan: - data = self.data_buffer[i_bucket] - label = self.label_buffer[i_bucket] - - i_idx = self.bucket_curr_idx[i_bucket] - idx = self.bucket_idx_all[i_bucket][i_idx:i_idx+self.batch_size] - self.bucket_curr_idx[i_bucket] += self.batch_size - data[:] = self.data[i_bucket][idx] - - for k in range(len(data)): - label[k] = sorted(data[k]) - #count = len(data[k]) / 2 - #for j in range(count): - # data[j+count] = data[j] - - #label[:, :-1] = data[:, 1:] - #label[:, -1] = 0 - - data_all = [mx.nd.array(data)] + self.init_state_arrays - label_all = [mx.nd.array(label)] - data_names = ['data'] + init_state_names - label_names = ['softmax_label'] - - data_batch = SimpleBatch(data_names, data_all, label_names, label_all, - self.buckets[i_bucket]) - - yield data_batch - - def reset(self): - self.bucket_curr_idx = [0 for x in self.data] diff --git a/example/capsnet/README.md b/example/capsnet/README.md index 49a6dd107..500c7df72 100644 --- a/example/capsnet/README.md +++ b/example/capsnet/README.md @@ -1,66 +1,66 @@ -**CapsNet-MXNet** -========================================= - -This example is MXNet implementation of [CapsNet](https://arxiv.org/abs/1710.09829): -Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017 -- The current `best test error is 0.29%` and `average test error is 0.303%` -- The `average test error on paper is 0.25%` - -Log files for the error rate are uploaded in [repository](https://github.com/samsungsds-rnd/capsnet.mxnet). -* * * -## **Usage** -Install scipy with pip -``` -pip install scipy -``` -Install tensorboard with pip -``` -pip install tensorboard -``` - -On Single gpu -``` -python capsulenet.py --devices gpu0 -``` -On Multi gpus -``` -python capsulenet.py --devices gpu0,gpu1 -``` -Full arguments -``` -python capsulenet.py --batch_size 100 --devices gpu0,gpu1 --num_epoch 100 --lr 0.001 --num_routing 3 --model_prefix capsnet -``` - -* * * -## **Prerequisities** - -MXNet version above (0.11.0) -scipy version above (0.19.0) - -*** -## **Results** -Train time takes about 36 seconds for each epoch (batch_size=100, 2 gtx 1080 gpus) - -CapsNet classification test error on MNIST - -``` -python capsulenet.py --devices gpu0,gpu1 --lr 0.0005 --decay 0.99 --model_prefix lr_0_0005_decay_0_99 --batch_size 100 --num_routing 3 --num_epoch 200 -``` - -![](result.PNG) - -| Trial | Epoch | train err(%) | test err(%) | train loss | test loss | -| :---: | :---: | :---: | :---: | :---: | :---: | -| 1 | 120 | 0.06 | 0.31 | 0.0056 | 0.0064 | -| 2 | 167 | 0.03 | 0.29 | 0.0048 | 0.0058 | -| 3 | 182 | 0.04 | 0.31 | 0.0046 | 0.0058 | -| average | - | 0.043 | 0.303 | 0.005 | 0.006 | - -We achieved `the best test error rate=0.29%` and `average test error=0.303%`. It is the best accuracy and fastest training time result among other implementations(Keras, Tensorflow at 2017-11-23). -The result on paper is `0.25% (average test error rate)`. - -| Implementation| test err(%) | ※train time/epoch | GPU Used| -| :---: | :---: | :---: |:---: | -| MXNet | 0.29 | 36 sec | 2 GTX 1080 | -| tensorflow | 0.49 | ※ 10 min | Unknown(4GB Memory) | -| Keras | 0.30 | 55 sec | 2 GTX 1080 Ti | +**CapsNet-MXNet** +========================================= + +This example is MXNet implementation of [CapsNet](https://arxiv.org/abs/1710.09829): +Sara Sabour, Nicholas Frosst, Geoffrey E Hinton. Dynamic Routing Between Capsules. NIPS 2017 +- The current `best test error is 0.29%` and `average test error is 0.303%` +- The `average test error on paper is 0.25%` + +Log files for the error rate are uploaded in [repository](https://github.com/samsungsds-rnd/capsnet.mxnet). +* * * +## **Usage** +Install scipy with pip +``` +pip install scipy +``` +Install tensorboard and mxboard with pip +``` +pip install mxboard tensorflow +``` + +On Single gpu +``` +python capsulenet.py --devices gpu0 +``` +On Multi gpus +``` +python capsulenet.py --devices gpu0,gpu1 +``` +Full arguments +``` +python capsulenet.py --batch_size 100 --devices gpu0,gpu1 --num_epoch 100 --lr 0.001 --num_routing 3 --model_prefix capsnet +``` + +* * * +## **Prerequisities** + +MXNet version above (1.2.0) +scipy version above (0.19.0) + +*** +## **Results** +Train time takes about 36 seconds for each epoch (batch_size=100, 2 gtx 1080 gpus) + +CapsNet classification test error on MNIST: + +``` +python capsulenet.py --devices gpu0,gpu1 --lr 0.0005 --decay 0.99 --model_prefix lr_0_0005_decay_0_99 --batch_size 100 --num_routing 3 --num_epoch 200 +``` + +![](result.PNG) + +| Trial | Epoch | train err(%) | test err(%) | train loss | test loss | +| :---: | :---: | :---: | :---: | :---: | :---: | +| 1 | 120 | 0.06 | 0.31 | 0.0056 | 0.0064 | +| 2 | 167 | 0.03 | 0.29 | 0.0048 | 0.0058 | +| 3 | 182 | 0.04 | 0.31 | 0.0046 | 0.0058 | +| average | - | 0.043 | 0.303 | 0.005 | 0.006 | + +We achieved `the best test error rate=0.29%` and `average test error=0.303%`. It is the best accuracy and fastest training time result among other implementations(Keras, Tensorflow at 2017-11-23). +The result on paper is `0.25% (average test error rate)`. + +| Implementation| test err(%) | ※train time/epoch | GPU Used| +| :---: | :---: | :---: |:---: | +| MXNet | 0.29 | 36 sec | 2 GTX 1080 | +| tensorflow | 0.49 | ※ 10 min | Unknown(4GB Memory) | +| Keras | 0.30 | 55 sec | 2 GTX 1080 Ti | diff --git a/example/capsnet/capsulenet.py b/example/capsnet/capsulenet.py index 6b44c3dfc..67108757b 100644 --- a/example/capsnet/capsulenet.py +++ b/example/capsnet/capsulenet.py @@ -1,348 +1,347 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -import mxnet as mx -import numpy as np -import os -import re -import urllib -import gzip -import struct -import scipy.ndimage as ndi -from capsulelayers import primary_caps, CapsuleLayer - -from tensorboard import SummaryWriter - -def margin_loss(y_true, y_pred): - loss = y_true * mx.sym.square(mx.sym.maximum(0., 0.9 - y_pred)) +\ - 0.5 * (1 - y_true) * mx.sym.square(mx.sym.maximum(0., y_pred - 0.1)) - return mx.sym.mean(data=mx.sym.sum(loss, 1)) - - -def capsnet(batch_size, n_class, num_routing,recon_loss_weight): - # data.shape = [batch_size, 1, 28, 28] - data = mx.sym.Variable('data') - - input_shape = (1, 28, 28) - # Conv2D layer - # net.shape = [batch_size, 256, 20, 20] - conv1 = mx.sym.Convolution(data=data, - num_filter=256, - kernel=(9, 9), - layout='NCHW', - name='conv1') - conv1 = mx.sym.Activation(data=conv1, act_type='relu', name='conv1_act') - # net.shape = [batch_size, 256, 6, 6] - - primarycaps = primary_caps(data=conv1, - dim_vector=8, - n_channels=32, - kernel=(9, 9), - strides=[2, 2], - name='primarycaps') - primarycaps.infer_shape(data=(batch_size, 1, 28, 28)) - # CapsuleLayer - kernel_initializer = mx.init.Xavier(rnd_type='uniform', factor_type='avg', magnitude=3) - bias_initializer = mx.init.Zero() - digitcaps = CapsuleLayer(num_capsule=10, - dim_vector=16, - batch_size=batch_size, - kernel_initializer=kernel_initializer, - bias_initializer=bias_initializer, - num_routing=num_routing)(primarycaps) - - # out_caps : (batch_size, 10) - out_caps = mx.sym.sqrt(data=mx.sym.sum(mx.sym.square(digitcaps), 2)) - out_caps.infer_shape(data=(batch_size, 1, 28, 28)) - - y = mx.sym.Variable('softmax_label', shape=(batch_size,)) - y_onehot = mx.sym.one_hot(y, n_class) - y_reshaped = mx.sym.Reshape(data=y_onehot, shape=(batch_size, -4, n_class, -1)) - y_reshaped.infer_shape(softmax_label=(batch_size,)) - - # inputs_masked : (batch_size, 16) - inputs_masked = mx.sym.linalg_gemm2(y_reshaped, digitcaps, transpose_a=True) - inputs_masked = mx.sym.Reshape(data=inputs_masked, shape=(-3, 0)) - x_recon = mx.sym.FullyConnected(data=inputs_masked, num_hidden=512, name='x_recon') - x_recon = mx.sym.Activation(data=x_recon, act_type='relu', name='x_recon_act') - x_recon = mx.sym.FullyConnected(data=x_recon, num_hidden=1024, name='x_recon2') - x_recon = mx.sym.Activation(data=x_recon, act_type='relu', name='x_recon_act2') - x_recon = mx.sym.FullyConnected(data=x_recon, num_hidden=np.prod(input_shape), name='x_recon3') - x_recon = mx.sym.Activation(data=x_recon, act_type='sigmoid', name='x_recon_act3') - - data_flatten = mx.sym.flatten(data=data) - squared_error = mx.sym.square(x_recon-data_flatten) - recon_error = mx.sym.mean(squared_error) - recon_error_stopped = recon_error - recon_error_stopped = mx.sym.BlockGrad(recon_error_stopped) - loss = mx.symbol.MakeLoss((1-recon_loss_weight)*margin_loss(y_onehot, out_caps)+recon_loss_weight*recon_error) - - out_caps_blocked = out_caps - out_caps_blocked = mx.sym.BlockGrad(out_caps_blocked) - return mx.sym.Group([out_caps_blocked, loss, recon_error_stopped]) - - -def download_data(url, force_download=False): - fname = url.split("/")[-1] - if force_download or not os.path.exists(fname): - urllib.urlretrieve(url, fname) - return fname - - -def read_data(label_url, image_url): - with gzip.open(download_data(label_url)) as flbl: - magic, num = struct.unpack(">II", flbl.read(8)) - label = np.fromstring(flbl.read(), dtype=np.int8) - with gzip.open(download_data(image_url), 'rb') as fimg: - magic, num, rows, cols = struct.unpack(">IIII", fimg.read(16)) - image = np.fromstring(fimg.read(), dtype=np.uint8).reshape(len(label), rows, cols) - return label, image - - -def to4d(img): - return img.reshape(img.shape[0], 1, 28, 28).astype(np.float32)/255 - - -class LossMetric(mx.metric.EvalMetric): - def __init__(self, batch_size, num_gpu): - super(LossMetric, self).__init__('LossMetric') - self.batch_size = batch_size - self.num_gpu = num_gpu - self.sum_metric = 0 - self.num_inst = 0 - self.loss = 0.0 - self.batch_sum_metric = 0 - self.batch_num_inst = 0 - self.batch_loss = 0.0 - self.recon_loss = 0.0 - self.n_batch = 0 - - def update(self, labels, preds): - batch_sum_metric = 0 - batch_num_inst = 0 - for label, pred_outcaps in zip(labels[0], preds[0]): - label_np = int(label.asnumpy()) - pred_label = int(np.argmax(pred_outcaps.asnumpy())) - batch_sum_metric += int(label_np == pred_label) - batch_num_inst += 1 - batch_loss = preds[1].asnumpy() - recon_loss = preds[2].asnumpy() - self.sum_metric += batch_sum_metric - self.num_inst += batch_num_inst - self.loss += batch_loss - self.recon_loss += recon_loss - self.batch_sum_metric = batch_sum_metric - self.batch_num_inst = batch_num_inst - self.batch_loss = batch_loss - self.n_batch += 1 - - def get_name_value(self): - acc = float(self.sum_metric)/float(self.num_inst) - mean_loss = self.loss / float(self.n_batch) - mean_recon_loss = self.recon_loss / float(self.n_batch) - return acc, mean_loss, mean_recon_loss - - def get_batch_log(self, n_batch): - print("n_batch :"+str(n_batch)+" batch_acc:" + - str(float(self.batch_sum_metric) / float(self.batch_num_inst)) + - ' batch_loss:' + str(float(self.batch_loss)/float(self.batch_num_inst))) - self.batch_sum_metric = 0 - self.batch_num_inst = 0 - self.batch_loss = 0.0 - - def reset(self): - self.sum_metric = 0 - self.num_inst = 0 - self.loss = 0.0 - self.recon_loss = 0.0 - self.n_batch = 0 - - -class SimpleLRScheduler(mx.lr_scheduler.LRScheduler): - """A simple lr schedule that simply return `dynamic_lr`. We will set `dynamic_lr` - dynamically based on performance on the validation set. - """ - - def __init__(self, learning_rate=0.001): - super(SimpleLRScheduler, self).__init__() - self.learning_rate = learning_rate - - def __call__(self, num_update): - return self.learning_rate - - -def do_training(num_epoch, optimizer, kvstore, learning_rate, model_prefix, decay): - summary_writer = SummaryWriter(args.tblog_dir) - lr_scheduler = SimpleLRScheduler(learning_rate) - optimizer_params = {'lr_scheduler': lr_scheduler} - module.init_params() - module.init_optimizer(kvstore=kvstore, - optimizer=optimizer, - optimizer_params=optimizer_params) - n_epoch = 0 - while True: - if n_epoch >= num_epoch: - break - train_iter.reset() - val_iter.reset() - loss_metric.reset() - for n_batch, data_batch in enumerate(train_iter): - module.forward_backward(data_batch) - module.update() - module.update_metric(loss_metric, data_batch.label) - loss_metric.get_batch_log(n_batch) - train_acc, train_loss, train_recon_err = loss_metric.get_name_value() - loss_metric.reset() - for n_batch, data_batch in enumerate(val_iter): - module.forward(data_batch) - module.update_metric(loss_metric, data_batch.label) - loss_metric.get_batch_log(n_batch) - val_acc, val_loss, val_recon_err = loss_metric.get_name_value() - - summary_writer.add_scalar('train_acc', train_acc, n_epoch) - summary_writer.add_scalar('train_loss', train_loss, n_epoch) - summary_writer.add_scalar('train_recon_err', train_recon_err, n_epoch) - summary_writer.add_scalar('val_acc', val_acc, n_epoch) - summary_writer.add_scalar('val_loss', val_loss, n_epoch) - summary_writer.add_scalar('val_recon_err', val_recon_err, n_epoch) - - print('Epoch[%d] train acc: %.4f loss: %.6f recon_err: %.6f' % (n_epoch, train_acc, train_loss, train_recon_err)) - print('Epoch[%d] val acc: %.4f loss: %.6f recon_err: %.6f' % (n_epoch, val_acc, val_loss, val_recon_err)) - print('SAVE CHECKPOINT') - - module.save_checkpoint(prefix=model_prefix, epoch=n_epoch) - n_epoch += 1 - lr_scheduler.learning_rate = learning_rate * (decay ** n_epoch) - - -def apply_transform(x, - transform_matrix, - fill_mode='nearest', - cval=0.): - x = np.rollaxis(x, 0, 0) - final_affine_matrix = transform_matrix[:2, :2] - final_offset = transform_matrix[:2, 2] - channel_images = [ndi.interpolation.affine_transform( - x_channel, - final_affine_matrix, - final_offset, - order=0, - mode=fill_mode, - cval=cval) for x_channel in x] - x = np.stack(channel_images, axis=0) - x = np.rollaxis(x, 0, 0 + 1) - return x - - -def random_shift(x, width_shift_fraction, height_shift_fraction): - tx = np.random.uniform(-height_shift_fraction, height_shift_fraction) * x.shape[2] - ty = np.random.uniform(-width_shift_fraction, width_shift_fraction) * x.shape[1] - shift_matrix = np.array([[1, 0, tx], - [0, 1, ty], - [0, 0, 1]]) - x = apply_transform(x, shift_matrix, 'nearest') - return x - -def _shuffle(data, idx): - """Shuffle the data.""" - shuffle_data = [] - - for k, v in data: - shuffle_data.append((k, mx.ndarray.array(v.asnumpy()[idx], v.context))) - - return shuffle_data - -class MNISTCustomIter(mx.io.NDArrayIter): - - def reset(self): - # shuffle data - if self.is_train: - np.random.shuffle(self.idx) - self.data = _shuffle(self.data, self.idx) - self.label = _shuffle(self.label, self.idx) - if self.last_batch_handle == 'roll_over' and self.cursor > self.num_data: - self.cursor = -self.batch_size + (self.cursor%self.num_data)%self.batch_size - else: - self.cursor = -self.batch_size - def set_is_train(self, is_train): - self.is_train = is_train - def next(self): - if self.iter_next(): - if self.is_train: - data_raw_list = self.getdata() - data_shifted = [] - for data_raw in data_raw_list[0]: - data_shifted.append(random_shift(data_raw.asnumpy(), 0.1, 0.1)) - return mx.io.DataBatch(data=[mx.nd.array(data_shifted)], label=self.getlabel(), - pad=self.getpad(), index=None) - else: - return mx.io.DataBatch(data=self.getdata(), label=self.getlabel(), \ - pad=self.getpad(), index=None) - - else: - raise StopIteration - - -if __name__ == "__main__": - # Read mnist data set - path = 'http://yann.lecun.com/exdb/mnist/' - (train_lbl, train_img) = read_data( - path + 'train-labels-idx1-ubyte.gz', path + 'train-images-idx3-ubyte.gz') - (val_lbl, val_img) = read_data( - path + 't10k-labels-idx1-ubyte.gz', path + 't10k-images-idx3-ubyte.gz') - # set batch size - import argparse - parser = argparse.ArgumentParser() - parser.add_argument('--batch_size', default=100, type=int) - parser.add_argument('--devices', default='gpu0', type=str) - parser.add_argument('--num_epoch', default=100, type=int) - parser.add_argument('--lr', default=0.001, type=float) - parser.add_argument('--num_routing', default=3, type=int) - parser.add_argument('--model_prefix', default='capsnet', type=str) - parser.add_argument('--decay', default=0.9, type=float) - parser.add_argument('--tblog_dir', default='tblog', type=str) - parser.add_argument('--recon_loss_weight', default=0.392, type=float) - args = parser.parse_args() - for k, v in sorted(vars(args).items()): - print("{0}: {1}".format(k, v)) - contexts = re.split(r'\W+', args.devices) - for i, ctx in enumerate(contexts): - if ctx[:3] == 'gpu': - contexts[i] = mx.context.gpu(int(ctx[3:])) - else: - contexts[i] = mx.context.cpu() - num_gpu = len(contexts) - - if args.batch_size % num_gpu != 0: - raise Exception('num_gpu should be positive divisor of batch_size') - - # generate train_iter, val_iter - train_iter = MNISTCustomIter(data=to4d(train_img), label=train_lbl, batch_size=args.batch_size, shuffle=True) - train_iter.set_is_train(True) - val_iter = MNISTCustomIter(data=to4d(val_img), label=val_lbl, batch_size=args.batch_size,) - val_iter.set_is_train(False) - # define capsnet - final_net = capsnet(batch_size=args.batch_size/num_gpu, n_class=10, num_routing=args.num_routing, recon_loss_weight=args.recon_loss_weight) - # set metric - loss_metric = LossMetric(args.batch_size/num_gpu, 1) - - # run model - module = mx.mod.Module(symbol=final_net, context=contexts, data_names=('data',), label_names=('softmax_label',)) - module.bind(data_shapes=train_iter.provide_data, - label_shapes=val_iter.provide_label, - for_training=True) - do_training(num_epoch=args.num_epoch, optimizer='adam', kvstore='device', learning_rate=args.lr, - model_prefix=args.model_prefix, decay=args.decay) +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +import mxnet as mx +import numpy as np +import os +import re +import gzip +import struct +import scipy.ndimage as ndi +from capsulelayers import primary_caps, CapsuleLayer + +from mxboard import SummaryWriter + +def margin_loss(y_true, y_pred): + loss = y_true * mx.sym.square(mx.sym.maximum(0., 0.9 - y_pred)) +\ + 0.5 * (1 - y_true) * mx.sym.square(mx.sym.maximum(0., y_pred - 0.1)) + return mx.sym.mean(data=mx.sym.sum(loss, 1)) + + +def capsnet(batch_size, n_class, num_routing,recon_loss_weight): + # data.shape = [batch_size, 1, 28, 28] + data = mx.sym.Variable('data') + + input_shape = (1, 28, 28) + # Conv2D layer + # net.shape = [batch_size, 256, 20, 20] + conv1 = mx.sym.Convolution(data=data, + num_filter=256, + kernel=(9, 9), + layout='NCHW', + name='conv1') + conv1 = mx.sym.Activation(data=conv1, act_type='relu', name='conv1_act') + # net.shape = [batch_size, 256, 6, 6] + + primarycaps = primary_caps(data=conv1, + dim_vector=8, + n_channels=32, + kernel=(9, 9), + strides=[2, 2], + name='primarycaps') + primarycaps.infer_shape(data=(batch_size, 1, 28, 28)) + # CapsuleLayer + kernel_initializer = mx.init.Xavier(rnd_type='uniform', factor_type='avg', magnitude=3) + bias_initializer = mx.init.Zero() + digitcaps = CapsuleLayer(num_capsule=10, + dim_vector=16, + batch_size=batch_size, + kernel_initializer=kernel_initializer, + bias_initializer=bias_initializer, + num_routing=num_routing)(primarycaps) + + # out_caps : (batch_size, 10) + out_caps = mx.sym.sqrt(data=mx.sym.sum(mx.sym.square(digitcaps), 2)) + out_caps.infer_shape(data=(batch_size, 1, 28, 28)) + + y = mx.sym.Variable('softmax_label', shape=(batch_size,)) + y_onehot = mx.sym.one_hot(y, n_class) + y_reshaped = mx.sym.Reshape(data=y_onehot, shape=(batch_size, -4, n_class, -1)) + y_reshaped.infer_shape(softmax_label=(batch_size,)) + + # inputs_masked : (batch_size, 16) + inputs_masked = mx.sym.linalg_gemm2(y_reshaped, digitcaps, transpose_a=True) + inputs_masked = mx.sym.Reshape(data=inputs_masked, shape=(-3, 0)) + x_recon = mx.sym.FullyConnected(data=inputs_masked, num_hidden=512, name='x_recon') + x_recon = mx.sym.Activation(data=x_recon, act_type='relu', name='x_recon_act') + x_recon = mx.sym.FullyConnected(data=x_recon, num_hidden=1024, name='x_recon2') + x_recon = mx.sym.Activation(data=x_recon, act_type='relu', name='x_recon_act2') + x_recon = mx.sym.FullyConnected(data=x_recon, num_hidden=np.prod(input_shape), name='x_recon3') + x_recon = mx.sym.Activation(data=x_recon, act_type='sigmoid', name='x_recon_act3') + + data_flatten = mx.sym.flatten(data=data) + squared_error = mx.sym.square(x_recon-data_flatten) + recon_error = mx.sym.mean(squared_error) + recon_error_stopped = recon_error + recon_error_stopped = mx.sym.BlockGrad(recon_error_stopped) + loss = mx.symbol.MakeLoss((1-recon_loss_weight)*margin_loss(y_onehot, out_caps)+recon_loss_weight*recon_error) + + out_caps_blocked = out_caps + out_caps_blocked = mx.sym.BlockGrad(out_caps_blocked) + return mx.sym.Group([out_caps_blocked, loss, recon_error_stopped]) + + +def download_data(url, force_download=False): + fname = url.split("/")[-1] + if force_download or not os.path.exists(fname): + mx.test_utils.download(url, fname) + return fname + + +def read_data(label_url, image_url): + with gzip.open(download_data(label_url)) as flbl: + magic, num = struct.unpack(">II", flbl.read(8)) + label = np.fromstring(flbl.read(), dtype=np.int8) + with gzip.open(download_data(image_url), 'rb') as fimg: + magic, num, rows, cols = struct.unpack(">IIII", fimg.read(16)) + image = np.fromstring(fimg.read(), dtype=np.uint8).reshape(len(label), rows, cols) + return label, image + + +def to4d(img): + return img.reshape(img.shape[0], 1, 28, 28).astype(np.float32)/255 + + +class LossMetric(mx.metric.EvalMetric): + def __init__(self, batch_size, num_gpu): + super(LossMetric, self).__init__('LossMetric') + self.batch_size = batch_size + self.num_gpu = num_gpu + self.sum_metric = 0 + self.num_inst = 0 + self.loss = 0.0 + self.batch_sum_metric = 0 + self.batch_num_inst = 0 + self.batch_loss = 0.0 + self.recon_loss = 0.0 + self.n_batch = 0 + + def update(self, labels, preds): + batch_sum_metric = 0 + batch_num_inst = 0 + for label, pred_outcaps in zip(labels[0], preds[0]): + label_np = int(label.asnumpy()) + pred_label = int(np.argmax(pred_outcaps.asnumpy())) + batch_sum_metric += int(label_np == pred_label) + batch_num_inst += 1 + batch_loss = preds[1].asnumpy() + recon_loss = preds[2].asnumpy() + self.sum_metric += batch_sum_metric + self.num_inst += batch_num_inst + self.loss += batch_loss + self.recon_loss += recon_loss + self.batch_sum_metric = batch_sum_metric + self.batch_num_inst = batch_num_inst + self.batch_loss = batch_loss + self.n_batch += 1 + + def get_name_value(self): + acc = float(self.sum_metric)/float(self.num_inst) + mean_loss = self.loss / float(self.n_batch) + mean_recon_loss = self.recon_loss / float(self.n_batch) + return acc, mean_loss, mean_recon_loss + + def get_batch_log(self, n_batch): + print("n_batch :"+str(n_batch)+" batch_acc:" + + str(float(self.batch_sum_metric) / float(self.batch_num_inst)) + + ' batch_loss:' + str(float(self.batch_loss)/float(self.batch_num_inst))) + self.batch_sum_metric = 0 + self.batch_num_inst = 0 + self.batch_loss = 0.0 + + def reset(self): + self.sum_metric = 0 + self.num_inst = 0 + self.loss = 0.0 + self.recon_loss = 0.0 + self.n_batch = 0 + + +class SimpleLRScheduler(mx.lr_scheduler.LRScheduler): + """A simple lr schedule that simply return `dynamic_lr`. We will set `dynamic_lr` + dynamically based on performance on the validation set. + """ + + def __init__(self, learning_rate=0.001): + super(SimpleLRScheduler, self).__init__() + self.learning_rate = learning_rate + + def __call__(self, num_update): + return self.learning_rate + + +def do_training(num_epoch, optimizer, kvstore, learning_rate, model_prefix, decay): + summary_writer = SummaryWriter(args.tblog_dir) + lr_scheduler = SimpleLRScheduler(learning_rate) + optimizer_params = {'lr_scheduler': lr_scheduler} + module.init_params() + module.init_optimizer(kvstore=kvstore, + optimizer=optimizer, + optimizer_params=optimizer_params) + n_epoch = 0 + while True: + if n_epoch >= num_epoch: + break + train_iter.reset() + val_iter.reset() + loss_metric.reset() + for n_batch, data_batch in enumerate(train_iter): + module.forward_backward(data_batch) + module.update() + module.update_metric(loss_metric, data_batch.label) + loss_metric.get_batch_log(n_batch) + train_acc, train_loss, train_recon_err = loss_metric.get_name_value() + loss_metric.reset() + for n_batch, data_batch in enumerate(val_iter): + module.forward(data_batch) + module.update_metric(loss_metric, data_batch.label) + loss_metric.get_batch_log(n_batch) + val_acc, val_loss, val_recon_err = loss_metric.get_name_value() + + summary_writer.add_scalar('train_acc', train_acc, n_epoch) + summary_writer.add_scalar('train_loss', train_loss, n_epoch) + summary_writer.add_scalar('train_recon_err', train_recon_err, n_epoch) + summary_writer.add_scalar('val_acc', val_acc, n_epoch) + summary_writer.add_scalar('val_loss', val_loss, n_epoch) + summary_writer.add_scalar('val_recon_err', val_recon_err, n_epoch) + + print('Epoch[%d] train acc: %.4f loss: %.6f recon_err: %.6f' % (n_epoch, train_acc, train_loss, train_recon_err)) + print('Epoch[%d] val acc: %.4f loss: %.6f recon_err: %.6f' % (n_epoch, val_acc, val_loss, val_recon_err)) + print('SAVE CHECKPOINT') + + module.save_checkpoint(prefix=model_prefix, epoch=n_epoch) + n_epoch += 1 + lr_scheduler.learning_rate = learning_rate * (decay ** n_epoch) + + +def apply_transform(x, + transform_matrix, + fill_mode='nearest', + cval=0.): + x = np.rollaxis(x, 0, 0) + final_affine_matrix = transform_matrix[:2, :2] + final_offset = transform_matrix[:2, 2] + channel_images = [ndi.interpolation.affine_transform( + x_channel, + final_affine_matrix, + final_offset, + order=0, + mode=fill_mode, + cval=cval) for x_channel in x] + x = np.stack(channel_images, axis=0) + x = np.rollaxis(x, 0, 0 + 1) + return x + + +def random_shift(x, width_shift_fraction, height_shift_fraction): + tx = np.random.uniform(-height_shift_fraction, height_shift_fraction) * x.shape[2] + ty = np.random.uniform(-width_shift_fraction, width_shift_fraction) * x.shape[1] + shift_matrix = np.array([[1, 0, tx], + [0, 1, ty], + [0, 0, 1]]) + x = apply_transform(x, shift_matrix, 'nearest') + return x + +def _shuffle(data, idx): + """Shuffle the data.""" + shuffle_data = [] + + for k, v in data: + shuffle_data.append((k, mx.ndarray.array(v.asnumpy()[idx], v.context))) + + return shuffle_data + +class MNISTCustomIter(mx.io.NDArrayIter): + + def reset(self): + # shuffle data + if self.is_train: + np.random.shuffle(self.idx) + self.data = _shuffle(self.data, self.idx) + self.label = _shuffle(self.label, self.idx) + if self.last_batch_handle == 'roll_over' and self.cursor > self.num_data: + self.cursor = -self.batch_size + (self.cursor%self.num_data)%self.batch_size + else: + self.cursor = -self.batch_size + def set_is_train(self, is_train): + self.is_train = is_train + def next(self): + if self.iter_next(): + if self.is_train: + data_raw_list = self.getdata() + data_shifted = [] + for data_raw in data_raw_list[0]: + data_shifted.append(random_shift(data_raw.asnumpy(), 0.1, 0.1)) + return mx.io.DataBatch(data=[mx.nd.array(data_shifted)], label=self.getlabel(), + pad=self.getpad(), index=None) + else: + return mx.io.DataBatch(data=self.getdata(), label=self.getlabel(), \ + pad=self.getpad(), index=None) + + else: + raise StopIteration + + +if __name__ == "__main__": + # Read mnist data set + path = 'http://yann.lecun.com/exdb/mnist/' + (train_lbl, train_img) = read_data( + path + 'train-labels-idx1-ubyte.gz', path + 'train-images-idx3-ubyte.gz') + (val_lbl, val_img) = read_data( + path + 't10k-labels-idx1-ubyte.gz', path + 't10k-images-idx3-ubyte.gz') + # set batch size + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('--batch_size', default=100, type=int) + parser.add_argument('--devices', default='gpu0', type=str) + parser.add_argument('--num_epoch', default=100, type=int) + parser.add_argument('--lr', default=0.001, type=float) + parser.add_argument('--num_routing', default=3, type=int) + parser.add_argument('--model_prefix', default='capsnet', type=str) + parser.add_argument('--decay', default=0.9, type=float) + parser.add_argument('--tblog_dir', default='tblog', type=str) + parser.add_argument('--recon_loss_weight', default=0.392, type=float) + args = parser.parse_args() + for k, v in sorted(vars(args).items()): + print("{0}: {1}".format(k, v)) + contexts = re.split(r'\W+', args.devices) + for i, ctx in enumerate(contexts): + if ctx[:3] == 'gpu': + contexts[i] = mx.context.gpu(int(ctx[3:])) + else: + contexts[i] = mx.context.cpu() + num_gpu = len(contexts) + + if args.batch_size % num_gpu != 0: + raise Exception('num_gpu should be positive divisor of batch_size') + + # generate train_iter, val_iter + train_iter = MNISTCustomIter(data=to4d(train_img), label=train_lbl, batch_size=int(args.batch_size), shuffle=True) + train_iter.set_is_train(True) + val_iter = MNISTCustomIter(data=to4d(val_img), label=val_lbl, batch_size=int(args.batch_size),) + val_iter.set_is_train(False) + # define capsnet + final_net = capsnet(batch_size=int(args.batch_size/num_gpu), n_class=10, num_routing=args.num_routing, recon_loss_weight=args.recon_loss_weight) + # set metric + loss_metric = LossMetric(args.batch_size/num_gpu, 1) + + # run model + module = mx.mod.Module(symbol=final_net, context=contexts, data_names=('data',), label_names=('softmax_label',)) + module.bind(data_shapes=train_iter.provide_data, + label_shapes=val_iter.provide_label, + for_training=True) + do_training(num_epoch=args.num_epoch, optimizer='adam', kvstore='device', learning_rate=args.lr, + model_prefix=args.model_prefix, decay=args.decay) diff --git a/example/cnn_visualization/README.md b/example/cnn_visualization/README.md deleted file mode 100644 index 10b914926..000000000 --- a/example/cnn_visualization/README.md +++ /dev/null @@ -1,17 +0,0 @@ -# Visualzing CNN decisions - -This folder contains an MXNet Gluon implementation of [Grad-CAM](https://arxiv.org/abs/1610.02391) that helps visualize CNN decisions. - -A tutorial on how to use this from Jupyter notebook is available [here](https://mxnet.incubator.apache.org/tutorials/vision/cnn_visualization.html). - -You can also do the visualization from terminal: -``` -$ python gradcam_demo.py hummingbird.jpg -Predicted category : hummingbird (94) -Original Image : hummingbird_orig.jpg -Grad-CAM : hummingbird_gradcam.jpg -Guided Grad-CAM : hummingbird_guided_gradcam.jpg -Saliency Map : hummingbird_saliency.jpg -``` - -![Output of gradcam_demo.py](https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/example/cnn_visualization/hummingbird_filenames.png) diff --git a/example/cnn_visualization/gradcam_demo.py b/example/cnn_visualization/gradcam_demo.py deleted file mode 100644 index d9ca5ddad..000000000 --- a/example/cnn_visualization/gradcam_demo.py +++ /dev/null @@ -1,110 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -import mxnet as mx -from mxnet import gluon - -import argparse -import os -import numpy as np -import cv2 - -import vgg -import gradcam - -# Receive image path from command line -parser = argparse.ArgumentParser(description='Grad-CAM demo') -parser.add_argument('img_path', metavar='image_path', type=str, help='path to the image file') - -args = parser.parse_args() - -# We'll use VGG-16 for visualization -network = vgg.vgg16(pretrained=True, ctx=mx.cpu()) -# We'll resize images to 224x244 as part of preprocessing -image_sz = (224, 224) - -def preprocess(data): - """Preprocess the image before running it through the network""" - data = mx.image.imresize(data, image_sz[0], image_sz[1]) - data = data.astype(np.float32) - data = data/255 - # These mean values were obtained from - # https://mxnet.incubator.apache.org/api/python/gluon/model_zoo.html - data = mx.image.color_normalize(data, - mean=mx.nd.array([0.485, 0.456, 0.406]), - std=mx.nd.array([0.229, 0.224, 0.225])) - data = mx.nd.transpose(data, (2,0,1)) # Channel first - return data - -def read_image_mxnet(path): - with open(path, 'rb') as fp: - img_bytes = fp.read() - return mx.img.imdecode(img_bytes) - -def read_image_cv(path): - return cv2.resize(cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB), image_sz) - -# synset.txt contains the names of Imagenet categories -# Load the file to memory and create a helper method to query category_index -> category name -synset_url = "http://data.mxnet.io/models/imagenet/synset.txt" -synset_file_name = "synset.txt" -mx.test_utils.download(synset_url, fname=synset_file_name) - -synset = [] -with open('synset.txt', 'r') as f: - synset = [l.rstrip().split(' ', 1)[1].split(',')[0] for l in f] - -def get_class_name(cls_id): - return "%s (%d)" % (synset[cls_id], cls_id) - -def run_inference(net, data): - """Run the input image through the network and return the predicted category as integer""" - out = net(data) - return out.argmax(axis=1).asnumpy()[0].astype(int) - -def visualize(net, img_path, conv_layer_name): - """Create Grad-CAM visualizations using the network 'net' and the image at 'img_path' - conv_layer_name is the name of the top most layer of the feature extractor""" - image = read_image_mxnet(img_path) - image = preprocess(image) - image = image.expand_dims(axis=0) - - pred_str = get_class_name(run_inference(net, image)) - - orig_img = read_image_cv(img_path) - vizs = gradcam.visualize(net, image, orig_img, conv_layer_name) - return (pred_str, (orig_img, *vizs)) - -# Create Grad-CAM visualization for the user provided image -last_conv_layer_name = 'vgg0_conv2d12' -cat, vizs = visualize(network, args.img_path, last_conv_layer_name) - -print("{0:20}: {1:80}".format("Predicted category", cat)) - -# Write the visualiations into file -img_name = os.path.split(args.img_path)[1].split('.')[0] -suffixes = ['orig', 'gradcam', 'guided_gradcam', 'saliency'] -image_desc = ['Original Image', 'Grad-CAM', 'Guided Grad-CAM', 'Saliency Map'] - -for i, img in enumerate(vizs): - img = img.astype(np.float32) - if len(img.shape) == 3: - img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) - out_file_name = "%s_%s.jpg" % (img_name, suffixes[i]) - cv2.imwrite(out_file_name, img) - print("{0:20}: {1:80}".format(image_desc[i], out_file_name)) - diff --git a/example/cnn_visualization/vgg.py b/example/cnn_visualization/vgg.py deleted file mode 100644 index b6215a334..000000000 --- a/example/cnn_visualization/vgg.py +++ /dev/null @@ -1,84 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -import mxnet as mx -from mxnet import gluon - -import os -from mxnet.gluon.model_zoo import model_store - -from mxnet.initializer import Xavier -from mxnet.gluon.nn import MaxPool2D, Flatten, Dense, Dropout, BatchNorm -from gradcam import Activation, Conv2D - -class VGG(mx.gluon.HybridBlock): - def __init__(self, layers, filters, classes=1000, batch_norm=False, **kwargs): - super(VGG, self).__init__(**kwargs) - assert len(layers) == len(filters) - with self.name_scope(): - self.features = self._make_features(layers, filters, batch_norm) - self.features.add(Dense(4096, activation='relu', - weight_initializer='normal', - bias_initializer='zeros')) - self.features.add(Dropout(rate=0.5)) - self.features.add(Dense(4096, activation='relu', - weight_initializer='normal', - bias_initializer='zeros')) - self.features.add(Dropout(rate=0.5)) - self.output = Dense(classes, - weight_initializer='normal', - bias_initializer='zeros') - - def _make_features(self, layers, filters, batch_norm): - featurizer = mx.gluon.nn.HybridSequential(prefix='') - for i, num in enumerate(layers): - for _ in range(num): - featurizer.add(Conv2D(filters[i], kernel_size=3, padding=1, - weight_initializer=Xavier(rnd_type='gaussian', - factor_type='out', - magnitude=2), - bias_initializer='zeros')) - if batch_norm: - featurizer.add(BatchNorm()) - featurizer.add(Activation('relu')) - featurizer.add(MaxPool2D(strides=2)) - return featurizer - - def hybrid_forward(self, F, x): - x = self.features(x) - x = self.output(x) - return x - -vgg_spec = {11: ([1, 1, 2, 2, 2], [64, 128, 256, 512, 512]), - 13: ([2, 2, 2, 2, 2], [64, 128, 256, 512, 512]), - 16: ([2, 2, 3, 3, 3], [64, 128, 256, 512, 512]), - 19: ([2, 2, 4, 4, 4], [64, 128, 256, 512, 512])} - -def get_vgg(num_layers, pretrained=False, ctx=mx.cpu(), - root=os.path.join('~', '.mxnet', 'models'), **kwargs): - layers, filters = vgg_spec[num_layers] - net = VGG(layers, filters, **kwargs) - if pretrained: - from mxnet.gluon.model_zoo.model_store import get_model_file - batch_norm_suffix = '_bn' if kwargs.get('batch_norm') else '' - net.load_params(get_model_file('vgg%d%s'%(num_layers, batch_norm_suffix), - root=root), ctx=ctx) - return net - -def vgg16(**kwargs): - return get_vgg(16, **kwargs) - diff --git a/example/deep-embedded-clustering/README.md b/example/deep-embedded-clustering/README.md index 90803d2ed..3972f90bd 100644 --- a/example/deep-embedded-clustering/README.md +++ b/example/deep-embedded-clustering/README.md @@ -1,9 +1,18 @@ # DEC Implementation This is based on the paper `Unsupervised deep embedding for clustering analysis` by Junyuan Xie, Ross Girshick, and Ali Farhadi +Abstract: + +Clustering is central to many data-driven application domains and has been studied extensively in terms of distance functions and grouping algorithms. Relatively little work has focused on learning representations for clustering. In this paper, we propose Deep Embedded Clustering (DEC), a method that simultaneously learns feature representations and cluster assignments using deep neural networks. DEC learns a mapping from the data space to a lower-dimensional feature space in which it iteratively optimizes a clustering objective. Our experimental evaluations on image and text corpora show significant improvement over state-of-the-art methods. + + ## Prerequisite - Install Scikit-learn: `python -m pip install --user sklearn` - Install SciPy: `python -m pip install --user scipy` +## Data + +The script is using MNIST dataset. + ## Usage -run `python dec.py` \ No newline at end of file +run `python dec.py` diff --git a/example/deep-embedded-clustering/data.py b/example/deep-embedded-clustering/data.py index 9fd472e6a..3649990db 100644 --- a/example/deep-embedded-clustering/data.py +++ b/example/deep-embedded-clustering/data.py @@ -19,20 +19,21 @@ from __future__ import print_function import os + +import mxnet as mx import numpy as np -from sklearn.datasets import fetch_mldata def get_mnist(): """ Gets MNIST dataset """ np.random.seed(1234) # set seed for deterministic ordering - data_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) - data_path = os.path.join(data_path, '../../data') - mnist = fetch_mldata('MNIST original', data_home=data_path) - p = np.random.permutation(mnist.data.shape[0]) - X = mnist.data[p].astype(np.float32)*0.02 - Y = mnist.target[p] + mnist_data = mx.test_utils.get_mnist() + X = np.concatenate([mnist_data['train_data'], mnist_data['test_data']]) + Y = np.concatenate([mnist_data['train_label'], mnist_data['test_label']]) + p = np.random.permutation(X.shape[0]) + X = X[p].reshape((X.shape[0], -1)).astype(np.float32)*5 + Y = Y[p] return X, Y diff --git a/example/deep-embedded-clustering/dec.py b/example/deep-embedded-clustering/dec.py index 44e582d9f..d7594703c 100644 --- a/example/deep-embedded-clustering/dec.py +++ b/example/deep-embedded-clustering/dec.py @@ -19,9 +19,6 @@ from __future__ import print_function import sys import os -# code to automatically download dataset -curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) -sys.path = [os.path.join(curr_path, "../autoencoder")] + sys.path import mxnet as mx import numpy as np import data @@ -33,14 +30,14 @@ import logging def cluster_acc(Y_pred, Y): - from sklearn.utils.linear_assignment_ import linear_assignment - assert Y_pred.size == Y.size - D = max(Y_pred.max(), Y.max())+1 - w = np.zeros((D,D), dtype=np.int64) - for i in range(Y_pred.size): - w[Y_pred[i], int(Y[i])] += 1 - ind = linear_assignment(w.max() - w) - return sum([w[i,j] for i,j in ind])*1.0/Y_pred.size, w + from sklearn.utils.linear_assignment_ import linear_assignment + assert Y_pred.size == Y.size + D = max(Y_pred.max(), Y.max())+1 + w = np.zeros((D,D), dtype=np.int64) + for i in range(Y_pred.size): + w[Y_pred[i], int(Y[i])] += 1 + ind = linear_assignment(w.max() - w) + return sum([w[i,j] for i,j in ind])*1.0/Y_pred.size, w class DECModel(model.MXModel): class DECLoss(mx.operator.NumpyOp): @@ -87,9 +84,9 @@ def setup(self, X, num_centers, alpha, save_to='dec_model'): ae_model = AutoEncoderModel(self.xpu, [X.shape[1],500,500,2000,10], pt_dropout=0.2) if not os.path.exists(save_to+'_pt.arg'): ae_model.layerwise_pretrain(X_train, 256, 50000, 'sgd', l_rate=0.1, decay=0.0, - lr_scheduler=mx.misc.FactorScheduler(20000,0.1)) + lr_scheduler=mx.lr_scheduler.FactorScheduler(20000,0.1)) ae_model.finetune(X_train, 256, 100000, 'sgd', l_rate=0.1, decay=0.0, - lr_scheduler=mx.misc.FactorScheduler(20000,0.1)) + lr_scheduler=mx.lr_scheduler.FactorScheduler(20000,0.1)) ae_model.save(save_to+'_pt.arg') logging.log(logging.INFO, "Autoencoder Training error: %f"%ae_model.eval(X_train)) logging.log(logging.INFO, "Autoencoder Validation error: %f"%ae_model.eval(X_val)) @@ -160,6 +157,8 @@ def refresh(i): def mnist_exp(xpu): X, Y = data.get_mnist() + if not os.path.isdir('data'): + os.makedirs('data') dec_model = DECModel(xpu, X, 10, 1.0, 'data/mnist') acc = [] for i in [10*(2**j) for j in range(9)]: diff --git a/example/fcn-xs/README.md b/example/fcn-xs/README.md index 145aa31cb..49c57fc08 100644 --- a/example/fcn-xs/README.md +++ b/example/fcn-xs/README.md @@ -40,14 +40,33 @@ this is the fully convolution style of the origin Once you completed all these steps, your working directory should contain a ```.\VOC2012``` directory, which contains the following: ```JPEGImages folder```, ```SegmentationClass folder```, ```train.lst```, ```val.lst``` #### Step 3: Train the fcn-xs model -* Based on your hardware, configure GPU or CPU for training in `fcn_xs.py`. It is recommended to use GPU due to the computational complexity and data load. -```python -# ctx = mx.cpu(0) -ctx = mx.gpu(0) +* Based on your hardware, configure CPU or GPU for training by parameter ```--gpu```. It is recommended to use GPU due to the computational complexity and data load. +View parameters we can use with the following command. +```shell +python fcn_xs.py -h + + +usage: fcn_xs.py [-h] [--model MODEL] [--prefix PREFIX] [--epoch EPOCH] + [--init-type INIT_TYPE] [--retrain] [--gpu GPU] + +Convert vgg16 model to vgg16fc model. + +optional arguments: + -h, --help show this help message and exit + --model MODEL The type of fcn-xs model, e.g. fcnxs, fcn16s, fcn8s. + --prefix PREFIX The prefix(include path) of vgg16 model with mxnet + format. + --epoch EPOCH The epoch number of vgg16 model. + --init-type INIT_TYPE + the init type of fcn-xs model, e.g. vgg16, fcnxs + --retrain true means continue training. + --gpu GPU 0 to use GPU, not set to use CPU ``` + * It is recommended to train fcn-32s and fcn-16s before training the fcn-8s model To train the fcn-32s model, run the following: + ```shell python -u fcn_xs.py --model=fcn32s --prefix=VGG_FC_ILSVRC_16_layers --epoch=74 --init-type=vgg16 ``` diff --git a/example/fcn-xs/fcn_xs.py b/example/fcn-xs/fcn_xs.py index 53244a175..5b799f32e 100644 --- a/example/fcn-xs/fcn_xs.py +++ b/example/fcn-xs/fcn_xs.py @@ -28,9 +28,10 @@ logger = logging.getLogger() logger.setLevel(logging.INFO) -ctx = mx.gpu(0) + def main(): + ctx = mx.cpu() if not args.gpu else mx.gpu(args.gpu) fcnxs = symbol_fcnxs.get_fcn32s_symbol(numclass=21, workspace_default=1536) fcnxs_model_prefix = "model_pascal/FCN32s_VGG16" if args.model == "fcn16s": @@ -85,6 +86,7 @@ def main(): help='the init type of fcn-xs model, e.g. vgg16, fcnxs') parser.add_argument('--retrain', action='store_true', default=False, help='true means continue training.') + parser.add_argument("--gpu", type=int, help="0 to use GPU, not set to use CPU") args = parser.parse_args() logging.info(args) main() diff --git a/example/gluon/actor_critic.py b/example/gluon/actor_critic/actor_critic.py similarity index 100% rename from example/gluon/actor_critic.py rename to example/gluon/actor_critic/actor_critic.py diff --git a/example/gluon/audio/transforms.py b/example/gluon/audio/transforms.py new file mode 100644 index 000000000..8b76d131c --- /dev/null +++ b/example/gluon/audio/transforms.py @@ -0,0 +1,205 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# coding: utf-8 +# pylint: disable= arguments-differ +"""Audio transforms.""" + +import warnings +import numpy as np +try: + import librosa +except ImportError as e: + warnings.warn("librosa dependency could not be resolved or \ + imported, could not provide some/all transform.") + +from mxnet import ndarray as nd +from mxnet.gluon.block import Block + +class MFCC(Block): + """Extracts Mel frequency cepstrum coefficients from the audio data file + More details : https://librosa.github.io/librosa/generated/librosa.feature.mfcc.html + + Attributes + ---------- + sampling_rate: int, default 22050 + sampling rate of the input audio signal + num_mfcc: int, default 20 + number of mfccs to return + + + Inputs: + - **x**: input tensor (samples, ) shape. + + Outputs: + - **out**: output array is a scaled NDArray with (samples, ) shape. + + """ + + def __init__(self, sampling_rate=22050, num_mfcc=20): + self._sampling_rate = sampling_rate + self._num_fcc = num_mfcc + super(MFCC, self).__init__() + + def forward(self, x): + if isinstance(x, np.ndarray): + y = x + elif isinstance(x, nd.NDArray): + y = x.asnumpy() + else: + warnings.warn("MFCC - allowed datatypes mx.nd.NDArray and numpy.ndarray") + return x + + audio_tmp = np.mean(librosa.feature.mfcc(y=y, sr=self._sampling_rate, n_mfcc=self._num_fcc).T, axis=0) + return nd.array(audio_tmp) + + +class Scale(Block): + """Scale audio numpy.ndarray from a 16-bit integer to a floating point number between + -1.0 and 1.0. The 16-bit integer is the sample resolution or bit depth. + + Attributes + ---------- + scale_factor : float + The factor to scale the input tensor by. + + + Inputs: + - **x**: input tensor (samples, ) shape. + + Outputs: + - **out**: output array is a scaled NDArray with (samples, ) shape. + + Examples + -------- + >>> scale = audio.transforms.Scale(scale_factor=2) + >>> audio_samples = mx.nd.array([2,3,4]) + >>> scale(audio_samples) + [1. 1.5 2. ] + + + """ + + def __init__(self, scale_factor=2**31): + self.scale_factor = scale_factor + super(Scale, self).__init__() + + def forward(self, x): + if self.scale_factor == 0: + warnings.warn("Scale factor cannot be 0.") + return x + if isinstance(x, np.ndarray): + return nd.array(x/self.scale_factor) + return x / self.scale_factor + + +class PadTrim(Block): + """Pad/Trim a 1d-NDArray of NPArray (Signal or Labels) + + Attributes + ---------- + max_len : int + Length to which the array will be padded or trimmed to. + fill_value: int or float + If there is a need of padding, what value to pad at the end of the input array. + + + Inputs: + - **x**: input tensor (samples, ) shape. + + Outputs: + - **out**: output array is a scaled NDArray with (max_len, ) shape. + + Examples + -------- + >>> padtrim = audio.transforms.PadTrim(max_len=9, fill_value=0) + >>> audio_samples = mx.nd.array([1,2,3,4,5]) + >>> padtrim(audio_samples) + [1. 2. 3. 4. 5. 0. 0. 0. 0.] + + + """ + + def __init__(self, max_len, fill_value=0): + self._max_len = max_len + self._fill_value = fill_value + super(PadTrim, self).__init__() + + def forward(self, x): + if isinstance(x, np.ndarray): + x = nd.array(x) + if self._max_len > x.size: + pad = nd.ones((self._max_len - x.size,)) * self._fill_value + x = nd.concat(x, pad, dim=0) + elif self._max_len < x.size: + x = x[:self._max_len] + return x + + +class MEL(Block): + """Create MEL Spectrograms from a raw audio signal. Relatively pretty slow. + + Attributes + ---------- + sampling_rate: int, default 22050 + sampling rate of the input audio signal + num_fft: int, default 2048 + length of the Fast Fourier transform window + num_mels: int, default 20 + number of mel bands to generate + hop_length: int, default 512 + total samples between successive frames + + + Inputs: + - **x**: input tensor (samples, ) shape. + + Outputs: + - **out**: output array which consists of mel spectograms, shape = (n_mels, 1) + + Usage (see librosa.feature.melspectrogram docs): + MEL(sr=16000, n_fft=1600, hop_length=800, n_mels=64) + + Examples + -------- + >>> mel = audio.transforms.MEL() + >>> audio_samples = mx.nd.array([1,2,3,4,5]) + >>> mel(audio_samples) + [[3.81801406e+04] + [9.86858240e-29] + [1.87405472e-29] + [2.38637225e-29] + [3.94043010e-29] + [3.67071565e-29] + [7.29390295e-29] + [8.84324438e-30]... + + + """ + + def __init__(self, sampling_rate=22050, num_fft=2048, num_mels=20, hop_length=512): + self._sampling_rate = sampling_rate + self._num_fft = num_fft + self._num_mels = num_mels + self._hop_length = hop_length + super(MEL, self).__init__() + + def forward(self, x): + if isinstance(x, nd.NDArray): + x = x.asnumpy() + specs = librosa.feature.melspectrogram(x, sr=self._sampling_rate,\ + n_fft=self._num_fft, n_mels=self._num_mels, hop_length=self._hop_length) + return nd.array(specs) diff --git a/example/gluon/audio/urban_sounds/README.md b/example/gluon/audio/urban_sounds/README.md new file mode 100644 index 000000000..c85d29db2 --- /dev/null +++ b/example/gluon/audio/urban_sounds/README.md @@ -0,0 +1,100 @@ +# Urban Sounds Classification in MXNet Gluon + +This example provides an end-to-end pipeline for a common datahack competition - Urban Sounds Classification Example. +Below is the link to the competition: +https://datahack.analyticsvidhya.com/contest/practice-problem-urban-sound-classification/ + +After logging in, the data set can be downloaded. +The details of the dataset and the link to download it are given below: + + +## Urban Sounds Dataset: +### Description + The dataset contains 8732 wav files which are audio samples(<= 4s)) of street sounds like engine_idling, car_horn, children_playing, dog_barking and so on. + The task is to classify these audio samples into one of the following 10 labels: + ``` + siren, + street_music, + drilling, + dog_bark, + children_playing, + gun_shot, + engine_idling, + air_conditioner, + jackhammer, + car_horn + ``` + +To be able to run this example: + +1. `pip install -r requirements.txt` + + If you are in the directory where the requirements.txt file lies, + this step installs the required libraries to run the example. + The main dependency that is required is: Librosa. + The version used to test the example is: `0.6.2` + For more details, refer here: +https://librosa.github.io/librosa/install.html + +2. Download the dataset(train.zip, test.zip) required for this example from the location: +https://drive.google.com/drive/folders/0By0bAi7hOBAFUHVXd1JCN3MwTEU + +3. Extract both the zip archives into the **current directory** - after unzipping you would get 2 new folders namely, + **Train** and **Test** and two csv files - **train.csv**, **test.csv** + + Assuming you are in a directory *"UrbanSounds"*, after downloading and extracting train.zip, the folder structure should be: + + ``` + UrbanSounds + - Train + - 0.wav, 1.wav ... + - train.csv + - train.py + - predict.py ... + ``` + +4. Apache MXNet is installed on the machine. For instructions, go to the link: https://mxnet.incubator.apache.org/install/ + + + +For information on the current design of how the AudioFolderDataset is implemented, refer below: +https://cwiki.apache.org/confluence/display/MXNET/Gluon+-+Audio + +### Usage + +For training: + +- Arguments + - train : The folder/directory that contains the audio(wav) files locally. Default = "./Train" + - csv: The file name of the csv file that contains audio file name to label mapping. Default = "train.csv" + - epochs : Number of epochs to train the model. Default = 30 + - batch_size : The batch size for training. Default = 32 + + +###### To use the default arguments, use: +``` +python train.py +``` +or + +###### To pass command-line arguments for training data directory, epochs, batch_size, csv file name, use : +``` +python train.py --train ./Train --csv train.csv --batch_size 32 --epochs 30 +``` + +For prediction: + +- Arguments + - pred : The folder/directory that contains the audio(wav) files which are to be classified. Default = "./Test" + + +###### To use the default arguments, use: +``` +python predict.py +``` +or + +###### To pass command-line arguments for test data directory, use : +``` +python predict.py --pred ./Test +``` \ No newline at end of file diff --git a/example/gluon/audio/urban_sounds/datasets.py b/example/gluon/audio/urban_sounds/datasets.py new file mode 100644 index 000000000..51c040c8f --- /dev/null +++ b/example/gluon/audio/urban_sounds/datasets.py @@ -0,0 +1,179 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# coding: utf-8 +# pylint: disable= +""" Audio Dataset container.""" +from __future__ import print_function +__all__ = ['AudioFolderDataset'] + +import os +import warnings +from itertools import islice +import csv +from mxnet.gluon.data import Dataset +from mxnet import ndarray as nd +try: + import librosa +except ImportError as e: + raise ImportError("librosa dependency could not be resolved or \ + imported, could not load audio onto the numpy array. pip install librosa") + + + +class AudioFolderDataset(Dataset): + """A dataset for loading Audio files stored in a folder structure like:: + + root/children_playing/0.wav + root/siren/23.wav + root/drilling/26.wav + root/dog_barking/42.wav + OR + Files(wav) and a csv file that has file name and associated label + + Parameters + ---------- + root : str + Path to root directory. + transform : callable, default None + A function that takes data and label and transforms them + train_csv: str, default None + train_csv should be populated by the training csv filename + file_format: str, default '.wav' + The format of the audio files(.wav) + skip_header: boolean, default False + While reading from csv file, whether to skip at the start of the file to avoid reading in header + + + Attributes + ---------- + synsets : list + List of class names. `synsets[i]` is the name for the `i`th label + items : list of tuples + List of all audio in (filename, label) pairs. + + """ + def __init__(self, root, train_csv=None, file_format='.wav', skip_header=False): + if not librosa: + warnings.warn("pip install librosa to continue.") + raise RuntimeError("Librosa not installed. Run pip install librosa and retry this step.") + self._root = os.path.expanduser(root) + self._exts = ['.wav'] + self._format = file_format + self._train_csv = train_csv + if file_format.lower() not in self._exts: + raise RuntimeError("Format {} not supported currently.".format(file_format)) + skip_rows = 0 + if skip_header: + skip_rows = 1 + self._list_audio_files(self._root, skip_rows=skip_rows) + + + def _list_audio_files(self, root, skip_rows=0): + """Populates synsets - a map of index to label for the data items. + Populates the data in the dataset, making tuples of (data, label) + """ + self.synsets = [] + self.items = [] + if not self._train_csv: + # The audio files are organized in folder structure with + # directory name as label and audios in them + self._folder_structure(root) + else: + # train_csv contains mapping between filename and label + self._csv_labelled_dataset(root, skip_rows=skip_rows) + + # Generating the synset.txt file now + if not os.path.exists("./synset.txt"): + with open("./synset.txt", "w") as synsets_file: + for item in self.synsets: + synsets_file.write(item+os.linesep) + print("Synsets is generated as synset.txt") + else: + warnings.warn("Synset file already exists in the current directory! Not generating synset.txt.") + + + def _folder_structure(self, root): + for folder in sorted(os.listdir(root)): + path = os.path.join(root, folder) + if not os.path.isdir(path): + warnings.warn('Ignoring {}, which is not a directory.'.format(path)) + continue + label = len(self.synsets) + self.synsets.append(folder) + for filename in sorted(os.listdir(path)): + file_name = os.path.join(path, filename) + ext = os.path.splitext(file_name)[1] + if ext.lower() not in self._exts: + warnings.warn('Ignoring {} of type {}. Only support {}'\ + .format(filename, ext, ', '.join(self._exts))) + continue + self.items.append((file_name, label)) + + + def _csv_labelled_dataset(self, root, skip_rows=0): + with open(self._train_csv, "r") as traincsv: + for line in islice(csv.reader(traincsv), skip_rows, None): + filename = os.path.join(root, line[0]) + label = line[1].strip() + if label not in self.synsets: + self.synsets.append(label) + if self._format not in filename: + filename = filename+self._format + self.items.append((filename, nd.array([self.synsets.index(label)]).reshape((1,)))) + + + def __getitem__(self, idx): + """Retrieve the item (data, label) stored at idx in items""" + filename, label = self.items[idx] + # resampling_type is passed as kaiser_fast for a better performance + X1, _ = librosa.load(filename, res_type='kaiser_fast') + return nd.array(X1), label + + + def __len__(self): + """Retrieves the number of items in the dataset""" + return len(self.items) + + + def transform_first(self, fn, lazy=False): + """Returns a new dataset with the first element of each sample + transformed by the transformer function `fn`. + + This is useful, for example, when you only want to transform data + while keeping label as is. + lazy=False is passed to transform_first for dataset so that all tramsforms could be performed in + one shot and not during training. This is a performance consideration. + + Parameters + ---------- + fn : callable + A transformer function that takes the first element of a sample + as input and returns the transformed element. + lazy : bool, default False + If False, transforms all samples at once. Otherwise, + transforms each sample on demand. Note that if `fn` + is stochastic, you must set lazy to True or you will + get the same result on all epochs. + + Returns + ------- + Dataset + The transformed dataset. + + """ + return super(AudioFolderDataset, self).transform_first(fn, lazy=lazy) diff --git a/example/gluon/audio/urban_sounds/model.py b/example/gluon/audio/urban_sounds/model.py new file mode 100644 index 000000000..af23cb946 --- /dev/null +++ b/example/gluon/audio/urban_sounds/model.py @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""This module builds a model an MLP with a configurable output layer( number of units in the last layer). +Users can pass any number of units in the last layer. SInce this dataset has 10 labels, +the default value of num_labels = 10 +""" +import mxnet as mx +from mxnet import gluon + +# Defining a neural network with number of labels +def get_net(num_labels=10): + net = gluon.nn.Sequential() + with net.name_scope(): + net.add(gluon.nn.Dense(256, activation="relu")) # 1st layer (256 nodes) + net.add(gluon.nn.Dense(256, activation="relu")) # 2nd hidden layer ( 256 nodes ) + net.add(gluon.nn.Dense(num_labels)) + net.collect_params().initialize(mx.init.Xavier()) + return net diff --git a/example/gluon/audio/urban_sounds/predict.py b/example/gluon/audio/urban_sounds/predict.py new file mode 100644 index 000000000..0c3631173 --- /dev/null +++ b/example/gluon/audio/urban_sounds/predict.py @@ -0,0 +1,92 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +""" Prediction module for Urban Sounds Classification""" +from __future__ import print_function +import os +import sys +import warnings +import mxnet as mx +from mxnet import nd +from model import get_net +try: + import librosa +except ImportError: + raise ImportError("Librosa is not installed! please run the following command:\ + `pip install librosa`") +sys.path.append('../') + +def predict(prediction_dir='./Test'): + """The function is used to run predictions on the audio files in the directory `pred_directory`. + + Parameters + ---------- + net: + The model that has been trained. + prediction_dir: string, default ./Test + The directory that contains the audio files on which predictions are to be made + + """ + + if not os.path.exists(prediction_dir): + warnings.warn("The directory on which predictions are to be made is not found!") + return + + if len(os.listdir(prediction_dir)) == 0: + warnings.warn("The directory on which predictions are to be made is empty! Exiting...") + return + + # Loading synsets + if not os.path.exists('./synset.txt'): + warnings.warn("The synset or labels for the dataset do not exist. Please run the training script first.") + return + + with open("./synset.txt", "r") as f: + synset = [l.rstrip() for l in f] + net = get_net(len(synset)) + print("Trying to load the model with the saved parameters...") + if not os.path.exists("./net.params"): + warnings.warn("The model does not have any saved parameters... Cannot proceed! Train the model first") + return + + net.load_parameters("./net.params") + file_names = os.listdir(prediction_dir) + full_file_names = [os.path.join(prediction_dir, item) for item in file_names] + from transforms import MFCC + mfcc = MFCC() + print("\nStarting predictions for audio files in ", prediction_dir, " ....\n") + for filename in full_file_names: + # Argument kaiser_fast to res_type is faster than 'kaiser_best'. To reduce the load time, passing kaiser_fast. + X1, _ = librosa.load(filename, res_type='kaiser_fast') + transformed_test_data = mfcc(mx.nd.array(X1)) + output = net(transformed_test_data.reshape((1, -1))) + prediction = nd.argmax(output, axis=1) + print(filename, " -> ", synset[(int)(prediction.asscalar())]) + + +if __name__ == '__main__': + try: + import argparse + parser = argparse.ArgumentParser(description="Urban Sounds clsssification example - MXNet") + parser.add_argument('--pred', '-p', help="Enter the folder path that contains your audio files", type=str) + args = parser.parse_args() + pred_dir = args.pred + + except ImportError: + warnings.warn("Argparse module not installed! passing default arguments.") + pred_dir = './Test' + predict(prediction_dir=pred_dir) + print("Urban sounds classification Prediction DONE!") diff --git a/example/gluon/audio/urban_sounds/requirements.txt b/example/gluon/audio/urban_sounds/requirements.txt new file mode 100644 index 000000000..d885e0bee --- /dev/null +++ b/example/gluon/audio/urban_sounds/requirements.txt @@ -0,0 +1,2 @@ +librosa>=0.6.2 # librosa is a library that is used to load the audio(wav) files and provides capabilities of feature extraction. +argparse # used for parsing arguments \ No newline at end of file diff --git a/example/gluon/audio/urban_sounds/train.py b/example/gluon/audio/urban_sounds/train.py new file mode 100644 index 000000000..c88f9fb55 --- /dev/null +++ b/example/gluon/audio/urban_sounds/train.py @@ -0,0 +1,157 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""The module to run training on the Urban sounds dataset""" +from __future__ import print_function +import sys +import os +import time +import warnings +import mxnet as mx +from mxnet import gluon, nd, autograd +from datasets import AudioFolderDataset +import model +sys.path.append('../') + +def evaluate_accuracy(data_iterator, net): + """Function to evaluate accuracy of any data iterator passed to it as an argument""" + acc = mx.metric.Accuracy() + for data, label in data_iterator: + output = net(data) + predictions = nd.argmax(output, axis=1) + predictions = predictions.reshape((-1, 1)) + acc.update(preds=predictions, labels=label) + return acc.get()[1] + + +def train(train_dir=None, train_csv=None, epochs=30, batch_size=32): + """Function responsible for running the training the model.""" + + if not train_dir or not os.path.exists(train_dir) or not train_csv: + warnings.warn("No train directory could be found ") + return + # Make a dataset from the local folder containing Audio data + print("\nMaking an Audio Dataset...\n") + tick = time.time() + aud_dataset = AudioFolderDataset(train_dir, train_csv=train_csv, file_format='.wav', skip_header=True) + tock = time.time() + + print("Loading the dataset took ", (tock-tick), " seconds.") + print("\n=======================================\n") + print("Number of output classes = ", len(aud_dataset.synsets)) + print("\nThe labels are : \n") + print(aud_dataset.synsets) + # Get the model to train + net = model.get_net(len(aud_dataset.synsets)) + print("\nNeural Network = \n") + print(net) + print("\nModel - Neural Network Generated!\n") + print("=======================================\n") + + #Define the loss - Softmax CE Loss + softmax_loss = gluon.loss.SoftmaxCELoss(from_logits=False, sparse_label=True) + print("Loss function initialized!\n") + print("=======================================\n") + + #Define the trainer with the optimizer + trainer = gluon.Trainer(net.collect_params(), 'adadelta') + print("Optimizer - Trainer function initialized!\n") + print("=======================================\n") + print("Loading the dataset to the Gluon's OOTB Dataloader...") + + #Getting the data loader out of the AudioDataset and passing the transform + from transforms import MFCC + aud_transform = MFCC() + tick = time.time() + + audio_train_loader = gluon.data.DataLoader(aud_dataset.transform_first(aud_transform), batch_size=32, shuffle=True) + tock = time.time() + print("Time taken to load data and apply transform here is ", (tock-tick), " seconds.") + print("=======================================\n") + + + print("Starting the training....\n") + # Training loop + tick = time.time() + batch_size = batch_size + num_examples = len(aud_dataset) + + for epoch in range(epochs): + cumulative_loss = 0 + for data, label in audio_train_loader: + with autograd.record(): + output = net(data) + loss = softmax_loss(output, label) + loss.backward() + + trainer.step(batch_size) + cumulative_loss += mx.nd.sum(loss).asscalar() + + if epoch%5 == 0: + train_accuracy = evaluate_accuracy(audio_train_loader, net) + print("Epoch {}. Loss: {} Train accuracy : {} ".format(epoch, cumulative_loss/num_examples, train_accuracy)) + print("\n------------------------------\n") + + train_accuracy = evaluate_accuracy(audio_train_loader, net) + tock = time.time() + print("\nFinal training accuracy: ", train_accuracy) + + print("Training the sound classification for ", epochs, " epochs, MLP model took ", (tock-tick), " seconds") + print("====================== END ======================\n") + + print("Trying to save the model parameters here...") + net.save_parameters("./net.params") + print("Saved the model parameters in current directory.") + + +if __name__ == '__main__': + training_dir = './Train' + training_csv = './train.csv' + epochs = 30 + batch_size = 32 + + try: + import argparse + parser = argparse.ArgumentParser(description="Urban Sounds classification example - MXNet Gluon") + parser.add_argument('--train', '-t', help="Enter the folder path that contains your audio files", type=str) + parser.add_argument('--csv', '-c', help="Enter the filename of the csv that contains filename\ + to label mapping", type=str) + parser.add_argument('--epochs', '-e', help="Enter the number of epochs \ + you would want to run the training for.", type=int) + parser.add_argument('--batch_size', '-b', help="Enter the batch_size of data", type=int) + args = parser.parse_args() + + if args: + if args.train: + training_dir = args.train + + if args.csv: + training_csv = args.csv + + if args.epochs: + epochs = args.epochs + + if args.batch_size: + batch_size = args.batch_size + + + except ImportError as er: + warnings.warn("Argument parsing module could not be imported \ + Passing default arguments.") + + + train(train_dir=training_dir, train_csv=training_csv, epochs=epochs, batch_size=batch_size) + print("Urban sounds classification Training DONE!") diff --git a/example/gluon/DCGAN/README.md b/example/gluon/dc_gan/README.md similarity index 100% rename from example/gluon/DCGAN/README.md rename to example/gluon/dc_gan/README.md diff --git a/example/gluon/DCGAN/__init__.py b/example/gluon/dc_gan/__init__.py similarity index 100% rename from example/gluon/DCGAN/__init__.py rename to example/gluon/dc_gan/__init__.py diff --git a/example/gluon/DCGAN/dcgan.py b/example/gluon/dc_gan/dcgan.py similarity index 100% rename from example/gluon/DCGAN/dcgan.py rename to example/gluon/dc_gan/dcgan.py diff --git a/example/gluon/DCGAN/inception_score.py b/example/gluon/dc_gan/inception_score.py similarity index 100% rename from example/gluon/DCGAN/inception_score.py rename to example/gluon/dc_gan/inception_score.py diff --git a/example/gluon/kaggle_k_fold_cross_validation.py b/example/gluon/house_prices/kaggle_k_fold_cross_validation.py similarity index 100% rename from example/gluon/kaggle_k_fold_cross_validation.py rename to example/gluon/house_prices/kaggle_k_fold_cross_validation.py diff --git a/example/gluon/learning_rate_manipulation.py b/example/gluon/learning_rate_manipulation.py deleted file mode 100644 index be1ffc290..000000000 --- a/example/gluon/learning_rate_manipulation.py +++ /dev/null @@ -1,63 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - - -# This example demonstrates how to manipulate the learning rate of an optimizer -# in gluon. The example uses linear regression as a case study. - -from __future__ import print_function -import numpy as np -import mxnet as mx -from mxnet import autograd -from mxnet import gluon - -# Generate synthetic data. -X = np.random.randn(10000, 2) -Y = 2 * X[:, 0] - 3.4 * X[:, 1] + 4.2 + .01 * np.random.normal(size=10000) - -net = gluon.nn.Sequential() -# The output dimension is 1. -net.add(gluon.nn.Dense(1)) -net.initialize() -loss = gluon.loss.L2Loss() - -# Initialize the learning rate as 0.1. -trainer = gluon.Trainer(net.collect_params(), 'sgd', - optimizer_params={'learning_rate': 0.1}) -net.initialize(mx.init.Xavier(magnitude=2.24), - force_reinit=True) -train_data = mx.io.NDArrayIter(X, Y, batch_size=10, shuffle=True) - -for epoch in range(5): - train_data.reset() - for i, batch in enumerate(train_data): - data = batch.data[0] - label = batch.label[0].reshape((-1, 1)) - with autograd.record(): - output = net(data) - mse = loss(output, label) - mse.backward() - trainer.step(data.shape[0]) - # After the second epoch, decay the learning rate of the optimizer every - # epoch. - if epoch > 1: - trainer.set_learning_rate(trainer.learning_rate * 0.9) - print('Epoch:', epoch, 'Learning rate:', trainer.learning_rate) - -for para_name, para_value in net.collect_params().items(): - # Print all the parameter values after training. - print(para_name, para_value.data().asnumpy()[0]) diff --git a/example/gluon/lstm_crf.py b/example/gluon/lstm_crf/lstm_crf.py similarity index 95% rename from example/gluon/lstm_crf.py rename to example/gluon/lstm_crf/lstm_crf.py index 3e95c054f..9c2218577 100644 --- a/example/gluon/lstm_crf.py +++ b/example/gluon/lstm_crf/lstm_crf.py @@ -21,8 +21,8 @@ import mxnet.optimizer as optim import sys -# This example demonstrates how LSTM-CRF model can be implemented in Gluon to perform -# noun-phrase chunking as a sequence labeling task. +# This example demonstrates how the LSTM-CRF model can be implemented +# in Gluon to perform noun-phrase chunking as a sequence labeling task. mx.random.seed(1) @@ -208,7 +208,9 @@ def forward(self, sentence): # dont confuse this with _forward_alg above. # Make sure prepare_sequence from earlier in the LSTM section is loaded for epoch in range(300): # again, normally you would NOT do 300 epochs, it is toy data - for sentence, tags in training_data: + + neg_log_likelihood_acc = 0. + for i, (sentence, tags) in enumerate(training_data): # Step 1. Get our inputs ready for the network, that is, # turn them into Variables of word indices. # Remember to use autograd to record the calculation. @@ -223,6 +225,8 @@ def forward(self, sentence): # dont confuse this with _forward_alg above. # calling optimizer.step() neg_log_likelihood.backward() optimizer.step(1) + neg_log_likelihood_acc += neg_log_likelihood.mean() + print("Epoch [{}], Negative Log Likelihood {:.4f}".format(epoch, neg_log_likelihood_acc.asscalar()/(i+1))) # Check predictions after training precheck_sent = prepare_sequence(training_data[0][0], word2idx) diff --git a/example/gluon/mnist.py b/example/gluon/mnist/mnist.py similarity index 100% rename from example/gluon/mnist.py rename to example/gluon/mnist/mnist.py diff --git a/example/gluon/sn_gan/data.py b/example/gluon/sn_gan/data.py index 7ed4c38a3..782f74ffc 100644 --- a/example/gluon/sn_gan/data.py +++ b/example/gluon/sn_gan/data.py @@ -17,7 +17,7 @@ # This example is inspired by https://github.com/jason71995/Keras-GAN-Library, # https://github.com/kazizzad/DCGAN-Gluon-MxNet/blob/master/MxnetDCGAN.ipynb -# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/DCGAN/dcgan.py +# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py import numpy as np diff --git a/example/gluon/sn_gan/model.py b/example/gluon/sn_gan/model.py index b714c7587..6040adb4e 100644 --- a/example/gluon/sn_gan/model.py +++ b/example/gluon/sn_gan/model.py @@ -17,7 +17,7 @@ # This example is inspired by https://github.com/jason71995/Keras-GAN-Library, # https://github.com/kazizzad/DCGAN-Gluon-MxNet/blob/master/MxnetDCGAN.ipynb -# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/DCGAN/dcgan.py +# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py import mxnet as mx from mxnet import nd diff --git a/example/gluon/sn_gan/train.py b/example/gluon/sn_gan/train.py index f4b988481..5faf3a2a0 100644 --- a/example/gluon/sn_gan/train.py +++ b/example/gluon/sn_gan/train.py @@ -17,7 +17,7 @@ # This example is inspired by https://github.com/jason71995/Keras-GAN-Library, # https://github.com/kazizzad/DCGAN-Gluon-MxNet/blob/master/MxnetDCGAN.ipynb -# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/DCGAN/dcgan.py +# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py import os diff --git a/example/gluon/sn_gan/utils.py b/example/gluon/sn_gan/utils.py index 06c02300b..1a77a6e90 100644 --- a/example/gluon/sn_gan/utils.py +++ b/example/gluon/sn_gan/utils.py @@ -17,7 +17,7 @@ # This example is inspired by https://github.com/jason71995/Keras-GAN-Library, # https://github.com/kazizzad/DCGAN-Gluon-MxNet/blob/master/MxnetDCGAN.ipynb -# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/DCGAN/dcgan.py +# https://github.com/apache/incubator-mxnet/blob/master/example/gluon/dc_gan/dcgan.py import math diff --git a/example/gluon/super_resolution.py b/example/gluon/super_resolution/super_resolution.py similarity index 100% rename from example/gluon/super_resolution.py rename to example/gluon/super_resolution/super_resolution.py diff --git a/example/gluon/tree_lstm/README.md b/example/gluon/tree_lstm/README.md new file mode 100644 index 000000000..e14ab4c70 --- /dev/null +++ b/example/gluon/tree_lstm/README.md @@ -0,0 +1,29 @@ + +# Tree-Structured Long Short-Term Memory Networks +This is a [MXNet Gluon](https://mxnet.io/) implementation of Tree-LSTM as described in the paper [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](http://arxiv.org/abs/1503.00075) by Kai Sheng Tai, Richard Socher, and Christopher Manning. + +### Requirements +- Python (tested on **3.6.5**, should work on **>=2.7**) +- Java >= 8 (for Stanford CoreNLP utilities) +- Other dependencies are in `requirements.txt` +Note: Currently works with MXNet 1.3.0. + +### Usage +Before delving into how to run the code, here is a quick overview of the contents: + - Use the script `fetch_and_preprocess.sh` to download the [SICK dataset](http://alt.qcri.org/semeval2014/task1/index.php?id=data-and-tools), [Stanford Parser](http://nlp.stanford.edu/software/lex-parser.shtml) and [Stanford POS Tagger](http://nlp.stanford.edu/software/tagger.shtml), and [Glove word vectors](http://nlp.stanford.edu/projects/glove/) (Common Crawl 840) -- **Warning:** this is a 2GB download!), and additionally preprocess the data, i.e. generate dependency parses using [Stanford Neural Network Dependency Parser](http://nlp.stanford.edu/software/nndep.shtml). +- `main.py`does the actual heavy lifting of training the model and testing it on the SICK dataset. For a list of all command-line arguments, have a look at `python main.py -h`. +- The first run caches GLOVE embeddings for words in the SICK vocabulary. In later runs, only the cache is read in during later runs. + +Next, these are the different ways to run the code here to train a TreeLSTM model. +#### Local Python Environment +If you have a working Python3 environment, simply run the following sequence of steps: + +``` +- bash fetch_and_preprocess.sh +- python main.py +``` + + +### Acknowledgments +- The Gluon version is ported from this implementation [dasguptar/treelstm.pytorch](https://github.com/dasguptar/treelstm.pytorch) +- Shout-out to [Kai Sheng Tai](https://github.com/kaishengtai/) for the [original LuaTorch implementation](https://github.com/stanfordnlp/treelstm), and to the [Pytorch team](https://github.com/pytorch/pytorch#the-team) for the fun library. diff --git a/example/gluon/tree_lstm/dataset.py b/example/gluon/tree_lstm/dataset.py index 02c57c0cc..5d6b76604 100644 --- a/example/gluon/tree_lstm/dataset.py +++ b/example/gluon/tree_lstm/dataset.py @@ -115,7 +115,7 @@ def load_embedding(self, f, reset=[]): self.add(word) if word in self.tok2idx: vectors[word] = [float(x) for x in tokens[1:]] - dim = len(vectors.values()[0]) + dim = len(list(vectors.values())[0]) def to_vector(tok): if tok in vectors and tok not in reset: return vectors[tok] @@ -154,7 +154,7 @@ def __init__(self, path, vocab, num_classes, shuffle=True): def reset(self): if self.shuffle: - mask = range(self.size) + mask = list(range(self.size)) random.shuffle(mask) self.l_sentences = [self.l_sentences[i] for i in mask] self.r_sentences = [self.r_sentences[i] for i in mask] diff --git a/example/gluon/tree_lstm/fetch_and_preprocess.sh b/example/gluon/tree_lstm/fetch_and_preprocess.sh index f37239283..a9b9d2861 100755 --- a/example/gluon/tree_lstm/fetch_and_preprocess.sh +++ b/example/gluon/tree_lstm/fetch_and_preprocess.sh @@ -18,8 +18,8 @@ # under the License. set -e -python2.7 scripts/download.py +python scripts/download.py CLASSPATH="lib:lib/stanford-parser/stanford-parser.jar:lib/stanford-parser/stanford-parser-3.5.1-models.jar" javac -cp $CLASSPATH lib/*.java -python2.7 scripts/preprocess-sick.py +python scripts/preprocess-sick.py diff --git a/example/gluon/tree_lstm/main.py b/example/gluon/tree_lstm/main.py index ad5d59f7a..53af3fa01 100644 --- a/example/gluon/tree_lstm/main.py +++ b/example/gluon/tree_lstm/main.py @@ -152,7 +152,6 @@ def train(epoch, ctx, train_data, dev_data): net.embed.weight.set_data(vocab.embed.as_in_context(ctx[0])) train_data.set_context(ctx[0]) dev_data.set_context(ctx[0]) - # set up trainer for optimizing the network. trainer = gluon.Trainer(net.collect_params(), optimizer, {'learning_rate': opt.lr, 'wd': opt.wd}) diff --git a/example/gluon/tree_lstm/scripts/download.py b/example/gluon/tree_lstm/scripts/download.py index 7ea930370..6537ef1ff 100644 --- a/example/gluon/tree_lstm/scripts/download.py +++ b/example/gluon/tree_lstm/scripts/download.py @@ -24,7 +24,6 @@ """ from __future__ import print_function -import urllib2 import sys import os import shutil diff --git a/example/image-classification/train_mnist.py b/example/image-classification/train_mnist.py index c0da09136..196cfaf4e 100644 --- a/example/image-classification/train_mnist.py +++ b/example/image-classification/train_mnist.py @@ -72,6 +72,7 @@ def get_mnist_iter(args, kv): help='the number of training examples') parser.add_argument('--add_stn', action="store_true", default=False, help='Add Spatial Transformer Network Layer (lenet only)') + parser.add_argument('--image_shape', default='1, 28, 28', help='shape of training images') parser.add_argument('--with-nnp', action="store_true", default=False, dest="is_nnp") fit.add_fit_args(parser) diff --git a/example/memcost/Makefile b/example/memcost/Makefile deleted file mode 100644 index f6d52db59..000000000 --- a/example/memcost/Makefile +++ /dev/null @@ -1,38 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License - -.PHONY: no_optimization with_inplace with_sharing with_both - -no_optimization: - @echo "Estimating the cost with no optimization..." - @NNVM_EXEC_ENABLE_INPLACE=false NNVM_EXEC_MATCH_RANGE=0 python inception_memcost.py - -with_inplace: - @echo "Estimating the cost with inplace optimization..." - @NNVM_EXEC_ENABLE_INPLACE=true MXNET_EXEC_MATCH_RANGE=0 python inception_memcost.py - -with_sharing: - @echo "Estimating the cost with memory sharing ..." - @NNVM_EXEC_ENABLE_INPLACE=false python inception_memcost.py - -with_both: - @echo "Estimating the cost with all optimizations ..." - @python inception_memcost.py - -forward_only: - @echo "Estimating the cost of forward only ..." - @python inception_memcost.py 'null' diff --git a/example/memcost/README.md b/example/memcost/README.md deleted file mode 100644 index 4c4e1fa97..000000000 --- a/example/memcost/README.md +++ /dev/null @@ -1,30 +0,0 @@ -Memory Cost of Deep Nets under Different Allocations -==================================================== -This folder contains a script to show the memory cost of different allocation strategies, -discussed in [Note on Memory Optimization](http://mxnet.io/architecture/note_memory.html). - -We use inception-bn as an example, with batch size of 32. - -How to See the cost -------------------- -The possible options are gathered together in the [Makefile](Makefile). -Type the following command to see the allocation cost. Look for the -```Final message Total x MB allocated``` -- ```make no_optimization``` - - Shows the cost without any optimization. -- ```make with_inplace``` - - Shows the cost with inplace optimization. -- ```make with_sharing``` - - Shows the cost with memory allocating algorithm for sharing. -- ```make with_both``` - - Shows the cost of memory allocation with both inplace and sharing optimization. -- ```make forward_only``` - - Shows the cost of when we only want to run forward pass. - -Notes ------ -- You can change the symbol in the [inception_memcost.py](inception_memcost.py) to the net you interested in. -- You will need to install mxnet or type make on the root folder before use the script. -- The estimation is only on space cost of intermediate node. - - The cost of temporal workspace is not estimated, so you will likely need more memory when running real nets. -- The estimation does real allocation on CPU, the plan is the same on GPU. diff --git a/example/memcost/inception_memcost.py b/example/memcost/inception_memcost.py deleted file mode 100644 index c539e73b3..000000000 --- a/example/memcost/inception_memcost.py +++ /dev/null @@ -1,107 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: skip-file -import sys -sys.path.append('../../python/') -import mxnet as mx -import logging - -def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0), name=None, suffix=''): - conv = mx.symbol.Convolution(data=data, num_filter=num_filter, kernel=kernel, stride=stride, pad=pad, name='conv_%s%s' %(name, suffix)) - bn = mx.symbol.BatchNorm(data=conv, name='bn_%s%s' %(name, suffix)) - act = mx.symbol.Activation(data=bn, act_type='relu', name='relu_%s%s' %(name, suffix)) - return act - -def InceptionFactoryA(data, num_1x1, num_3x3red, num_3x3, num_d3x3red, num_d3x3, pool, proj, name): - # 1x1 - c1x1 = ConvFactory(data=data, num_filter=num_1x1, kernel=(1, 1), name=('%s_1x1' % name)) - # 3x3 reduce + 3x3 - c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce') - c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), name=('%s_3x3' % name)) - # double 3x3 reduce + double 3x3 - cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1), name=('%s_double_3x3' % name), suffix='_reduce') - cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_0' % name)) - cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), name=('%s_double_3x3_1' % name)) - # pool + proj - pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(1, 1), pad=(1, 1), pool_type=pool, name=('%s_pool_%s_pool' % (pool, name))) - cproj = ConvFactory(data=pooling, num_filter=proj, kernel=(1, 1), name=('%s_proj' % name)) - # concat - concat = mx.symbol.Concat(*[c1x1, c3x3, cd3x3, cproj], name='ch_concat_%s_chconcat' % name) - return concat - -def InceptionFactoryB(data, num_3x3red, num_3x3, num_d3x3red, num_d3x3, name): - # 3x3 reduce + 3x3 - c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), suffix='_reduce') - c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), stride=(2, 2), name=('%s_3x3' % name)) - # double 3x3 reduce + double 3x3 - cd3x3r = ConvFactory(data=data, num_filter=num_d3x3red, kernel=(1, 1), name=('%s_double_3x3' % name), suffix='_reduce') - cd3x3 = ConvFactory(data=cd3x3r, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), stride=(1, 1), name=('%s_double_3x3_0' % name)) - cd3x3 = ConvFactory(data=cd3x3, num_filter=num_d3x3, kernel=(3, 3), pad=(1, 1), stride=(2, 2), name=('%s_double_3x3_1' % name)) - # pool + proj - pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(2, 2), pad=(1, 1), pool_type="max", name=('max_pool_%s_pool' % name)) - # concat - concat = mx.symbol.Concat(*[c3x3, cd3x3, pooling], name='ch_concat_%s_chconcat' % name) - return concat - -def inception(nhidden, grad_scale): - # data - data = mx.symbol.Variable(name="data") - # stage 1 - conv1 = ConvFactory(data=data, num_filter=64, kernel=(7, 7), stride=(2, 2), pad=(3, 3), name='conv1') - pool1 = mx.symbol.Pooling(data=conv1, kernel=(3, 3), stride=(2, 2), name='pool1', pool_type='max') - # stage 2 - conv2red = ConvFactory(data=pool1, num_filter=64, kernel=(1, 1), stride=(1, 1), name='conv2red') - conv2 = ConvFactory(data=conv2red, num_filter=192, kernel=(3, 3), stride=(1, 1), pad=(1, 1), name='conv2') - pool2 = mx.symbol.Pooling(data=conv2, kernel=(3, 3), stride=(2, 2), name='pool2', pool_type='max') - # stage 2 - in3a = InceptionFactoryA(pool2, 64, 64, 64, 64, 96, "avg", 32, '3a') - in3b = InceptionFactoryA(in3a, 64, 64, 96, 64, 96, "avg", 64, '3b') - in3c = InceptionFactoryB(in3b, 128, 160, 64, 96, '3c') - # stage 3 - in4a = InceptionFactoryA(in3c, 224, 64, 96, 96, 128, "avg", 128, '4a') - in4b = InceptionFactoryA(in4a, 192, 96, 128, 96, 128, "avg", 128, '4b') - in4c = InceptionFactoryA(in4b, 160, 128, 160, 128, 160, "avg", 128, '4c') - in4d = InceptionFactoryA(in4c, 96, 128, 192, 160, 192, "avg", 128, '4d') - in4e = InceptionFactoryB(in4d, 128, 192, 192, 256, '4e') - # stage 4 - in5a = InceptionFactoryA(in4e, 352, 192, 320, 160, 224, "avg", 128, '5a') - in5b = InceptionFactoryA(in5a, 352, 192, 320, 192, 224, "max", 128, '5b') - # global avg pooling - avg = mx.symbol.Pooling(data=in5b, kernel=(7, 7), stride=(1, 1), name="global_pool", pool_type='avg') - # linear classifier - flatten = mx.symbol.Flatten(data=avg, name='flatten') - fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=nhidden, name='fc1') - softmax = mx.symbol.SoftmaxOutput(data=fc1, name='softmax') - return softmax - - - -softmax = inception(1000, 1.0) -batch_size = 32 -softmax = inception(1000, 1.0) - -if len(sys.argv) == 2: - grad_req = sys.argv[1] -else: - grad_req = 'write' - -texec = softmax.simple_bind(ctx=mx.cpu(), - data=(batch_size, 3, 224, 224), - grad_req=grad_req) -# We extract the memory cost from the execution plan -print(texec.debug_str().split('\n')[-3]) diff --git a/example/utils/__init__.py b/example/module/utils/__init__.py similarity index 100% rename from example/utils/__init__.py rename to example/module/utils/__init__.py diff --git a/example/utils/get_data.py b/example/module/utils/get_data.py similarity index 94% rename from example/utils/get_data.py rename to example/module/utils/get_data.py index 861d16cdb..2a585ead3 100644 --- a/example/utils/get_data.py +++ b/example/module/utils/get_data.py @@ -17,6 +17,7 @@ import os import mxnet as mx +import zipfile def get_mnist(data_dir): if not os.path.isdir(data_dir): @@ -28,7 +29,7 @@ def get_mnist(data_dir): (not os.path.exists('t10k-labels-idx1-ubyte')): import urllib, zipfile zippath = os.path.join(os.getcwd(), "mnist.zip") - urllib.urlretrieve("http://data.mxnet.io/mxnet/data/mnist.zip", zippath) + mx.test_utils.download("http://data.mxnet.io/mxnet/data/mnist.zip", zippath) zf = zipfile.ZipFile(zippath, "r") zf.extractall() zf.close() @@ -45,7 +46,7 @@ def get_cifar10(data_dir): import urllib, zipfile, glob dirname = os.getcwd() zippath = os.path.join(dirname, "cifar10.zip") - urllib.urlretrieve("http://data.mxnet.io/mxnet/data/cifar10.zip", zippath) + mx.test_utils.download("http://data.mxnet.io/mxnet/data/cifar10.zip", zippath) zf = zipfile.ZipFile(zippath, "r") zf.extractall() zf.close() @@ -56,7 +57,6 @@ def get_cifar10(data_dir): os.rmdir(os.path.join(dirname, "cifar")) os.chdir(cwd) -# data def get_cifar10_iterator(args, kv): data_shape = (3, 28, 28) data_dir = args.data_dir diff --git a/example/multi-task/README.md b/example/multi-task/README.md index 9034814c3..b7756fe37 100644 --- a/example/multi-task/README.md +++ b/example/multi-task/README.md @@ -1,10 +1,13 @@ # Mulit-task learning example -This is a simple example to show how to use mxnet for multi-task learning. It uses MNIST as an example and mocks up the multi-label task. +This is a simple example to show how to use mxnet for multi-task learning. It uses MNIST as an example, trying to predict jointly the digit and whether this digit is odd or even. -## Usage -First, you need to write a multi-task iterator on your own. The iterator needs to generate multiple labels according to your applications, and the label names should be specified in the `provide_label` function, which needs to be consist with the names of output layers. +For example: -Then, if you want to show metrics of different tasks separately, you need to write your own metric class and specify the `num` parameter. In the `update` function of metric, calculate the metrics separately for different tasks. +![](https://camo.githubusercontent.com/ed3cf256f47713335dc288f32f9b0b60bf1028b7/68747470733a2f2f7777772e636c61737365732e63732e756368696361676f2e6564752f617263686976652f323031332f737072696e672f31323330302d312f70612f7061312f64696769742e706e67) -The example script uses gpu as device by default, if gpu is not available for your environment, you can change `device` to be `mx.cpu()`. +Should be jointly classified as 4, and Even. + +In this example we don't expect the tasks to contribute to each other much, but for example multi-task learning has been successfully applied to the domain of image captioning. In [A Multi-task Learning Approach for Image Captioning](https://www.ijcai.org/proceedings/2018/0168.pdf) by Wei Zhao, Benyou Wang, Jianbo Ye, Min Yang, Zhou Zhao, Ruotian Luo, Yu Qiao, they train a network to jointly classify images and generate text captions + +Please refer to the notebook for a fully worked example. diff --git a/example/multi-task/example_multi_task.py b/example/multi-task/example_multi_task.py deleted file mode 100644 index 9e898494a..000000000 --- a/example/multi-task/example_multi_task.py +++ /dev/null @@ -1,159 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: skip-file -import mxnet as mx -from mxnet.test_utils import get_mnist_iterator -import numpy as np -import logging -import time - -logging.basicConfig(level=logging.DEBUG) - -def build_network(): - data = mx.symbol.Variable('data') - fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128) - act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type="relu") - fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64) - act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type="relu") - fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10) - sm1 = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax1') - sm2 = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax2') - - softmax = mx.symbol.Group([sm1, sm2]) - - return softmax - -class Multi_mnist_iterator(mx.io.DataIter): - '''multi label mnist iterator''' - - def __init__(self, data_iter): - super(Multi_mnist_iterator, self).__init__() - self.data_iter = data_iter - self.batch_size = self.data_iter.batch_size - - @property - def provide_data(self): - return self.data_iter.provide_data - - @property - def provide_label(self): - provide_label = self.data_iter.provide_label[0] - # Different labels should be used here for actual application - return [('softmax1_label', provide_label[1]), \ - ('softmax2_label', provide_label[1])] - - def hard_reset(self): - self.data_iter.hard_reset() - - def reset(self): - self.data_iter.reset() - - def next(self): - batch = self.data_iter.next() - label = batch.label[0] - - return mx.io.DataBatch(data=batch.data, label=[label, label], \ - pad=batch.pad, index=batch.index) - -class Multi_Accuracy(mx.metric.EvalMetric): - """Calculate accuracies of multi label""" - - def __init__(self, num=None): - self.num = num - super(Multi_Accuracy, self).__init__('multi-accuracy') - - def reset(self): - """Resets the internal evaluation result to initial state.""" - self.num_inst = 0 if self.num is None else [0] * self.num - self.sum_metric = 0.0 if self.num is None else [0.0] * self.num - - def update(self, labels, preds): - mx.metric.check_label_shapes(labels, preds) - - if self.num is not None: - assert len(labels) == self.num - - for i in range(len(labels)): - pred_label = mx.nd.argmax_channel(preds[i]).asnumpy().astype('int32') - label = labels[i].asnumpy().astype('int32') - - mx.metric.check_label_shapes(label, pred_label) - - if self.num is None: - self.sum_metric += (pred_label.flat == label.flat).sum() - self.num_inst += len(pred_label.flat) - else: - self.sum_metric[i] += (pred_label.flat == label.flat).sum() - self.num_inst[i] += len(pred_label.flat) - - def get(self): - """Gets the current evaluation result. - - Returns - ------- - names : list of str - Name of the metrics. - values : list of float - Value of the evaluations. - """ - if self.num is None: - return super(Multi_Accuracy, self).get() - else: - return zip(*(('%s-task%d'%(self.name, i), float('nan') if self.num_inst[i] == 0 - else self.sum_metric[i] / self.num_inst[i]) - for i in range(self.num))) - - def get_name_value(self): - """Returns zipped name and value pairs. - - Returns - ------- - list of tuples - A (name, value) tuple list. - """ - if self.num is None: - return super(Multi_Accuracy, self).get_name_value() - name, value = self.get() - return list(zip(name, value)) - - -batch_size=100 -num_epochs=100 -device = mx.gpu(0) -lr = 0.01 - -network = build_network() -train, val = get_mnist_iterator(batch_size=batch_size, input_shape = (784,)) -train = Multi_mnist_iterator(train) -val = Multi_mnist_iterator(val) - - -model = mx.mod.Module( - context = device, - symbol = network, - label_names = ('softmax1_label', 'softmax2_label')) - -model.fit( - train_data = train, - eval_data = val, - eval_metric = Multi_Accuracy(num=2), - num_epoch = num_epochs, - optimizer_params = (('learning_rate', lr), ('momentum', 0.9), ('wd', 0.00001)), - initializer = mx.init.Xavier(factor_type="in", magnitude=2.34), - batch_end_callback = mx.callback.Speedometer(batch_size, 50)) - diff --git a/example/multi-task/multi-task-learning.ipynb b/example/multi-task/multi-task-learning.ipynb new file mode 100644 index 000000000..6e03e2b61 --- /dev/null +++ b/example/multi-task/multi-task-learning.ipynb @@ -0,0 +1,454 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Multi-Task Learning Example" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is a simple example to show how to use mxnet for multi-task learning.\n", + "\n", + "The network is jointly going to learn whether a number is odd or even and to actually recognize the digit.\n", + "\n", + "\n", + "For example\n", + "\n", + "- 1 : 1 and odd\n", + "- 2 : 2 and even\n", + "- 3 : 3 and odd\n", + "\n", + "etc\n", + "\n", + "In this example we don't expect the tasks to contribute to each other much, but for example multi-task learning has been successfully applied to the domain of image captioning. In [A Multi-task Learning Approach for Image Captioning](https://www.ijcai.org/proceedings/2018/0168.pdf) by Wei Zhao, Benyou Wang, Jianbo Ye, Min Yang, Zhou Zhao, Ruotian Luo, Yu Qiao, they train a network to jointly classify images and generate text captions" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "import logging\n", + "import random\n", + "import time\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import mxnet as mx\n", + "from mxnet import gluon, nd, autograd\n", + "import numpy as np" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Parameters" + ] + }, + { + "cell_type": "code", + "execution_count": 99, + "metadata": {}, + "outputs": [], + "source": [ + "batch_size = 128\n", + "epochs = 5\n", + "ctx = mx.gpu() if len(mx.test_utils.list_gpus()) > 0 else mx.cpu()\n", + "lr = 0.01" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data\n", + "\n", + "We get the traditionnal MNIST dataset and add a new label to the existing one. For each digit we return a new label that stands for Odd or Even" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "![](https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "train_dataset = gluon.data.vision.MNIST(train=True)\n", + "test_dataset = gluon.data.vision.MNIST(train=False)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "def transform(x,y):\n", + " x = x.transpose((2,0,1)).astype('float32')/255.\n", + " y1 = y\n", + " y2 = y % 2 #odd or even\n", + " return x, np.float32(y1), np.float32(y2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We assign the transform to the original dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "train_dataset_t = train_dataset.transform(transform)\n", + "test_dataset_t = test_dataset.transform(transform)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We load the datasets DataLoaders" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "train_data = gluon.data.DataLoader(train_dataset_t, shuffle=True, last_batch='rollover', batch_size=batch_size, num_workers=5)\n", + "test_data = gluon.data.DataLoader(test_dataset_t, shuffle=False, last_batch='rollover', batch_size=batch_size, num_workers=5)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input shape: (28, 28, 1), Target Labels: (5.0, 1.0)\n" + ] + } + ], + "source": [ + "print(\"Input shape: {}, Target Labels: {}\".format(train_dataset[0][0].shape, train_dataset_t[0][1:]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Multi-task Network\n", + "\n", + "The output of the featurization is passed to two different outputs layers" + ] + }, + { + "cell_type": "code", + "execution_count": 135, + "metadata": {}, + "outputs": [], + "source": [ + "class MultiTaskNetwork(gluon.HybridBlock):\n", + " \n", + " def __init__(self):\n", + " super(MultiTaskNetwork, self).__init__()\n", + " \n", + " self.shared = gluon.nn.HybridSequential()\n", + " with self.shared.name_scope():\n", + " self.shared.add(\n", + " gluon.nn.Dense(128, activation='relu'),\n", + " gluon.nn.Dense(64, activation='relu'),\n", + " gluon.nn.Dense(10, activation='relu')\n", + " )\n", + " self.output1 = gluon.nn.Dense(10) # Digist recognition\n", + " self.output2 = gluon.nn.Dense(1) # odd or even\n", + "\n", + " \n", + " def hybrid_forward(self, F, x):\n", + " y = self.shared(x)\n", + " output1 = self.output1(y)\n", + " output2 = self.output2(y)\n", + " return output1, output2" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can use two different losses, one for each output" + ] + }, + { + "cell_type": "code", + "execution_count": 136, + "metadata": {}, + "outputs": [], + "source": [ + "loss_digits = gluon.loss.SoftmaxCELoss()\n", + "loss_odd_even = gluon.loss.SigmoidBCELoss()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We create and initialize the network" + ] + }, + { + "cell_type": "code", + "execution_count": 137, + "metadata": {}, + "outputs": [], + "source": [ + "mx.random.seed(42)\n", + "random.seed(42)" + ] + }, + { + "cell_type": "code", + "execution_count": 138, + "metadata": {}, + "outputs": [], + "source": [ + "net = MultiTaskNetwork()" + ] + }, + { + "cell_type": "code", + "execution_count": 139, + "metadata": {}, + "outputs": [], + "source": [ + "net.initialize(mx.init.Xavier(), ctx=ctx)\n", + "net.hybridize() # hybridize for speed" + ] + }, + { + "cell_type": "code", + "execution_count": 140, + "metadata": {}, + "outputs": [], + "source": [ + "trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate':lr})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Evaluate Accuracy\n", + "We need to evaluate the accuracy of each task separately" + ] + }, + { + "cell_type": "code", + "execution_count": 141, + "metadata": {}, + "outputs": [], + "source": [ + "def evaluate_accuracy(net, data_iterator):\n", + " acc_digits = mx.metric.Accuracy(name='digits')\n", + " acc_odd_even = mx.metric.Accuracy(name='odd_even')\n", + " \n", + " for i, (data, label_digit, label_odd_even) in enumerate(data_iterator):\n", + " data = data.as_in_context(ctx)\n", + " label_digit = label_digit.as_in_context(ctx)\n", + " label_odd_even = label_odd_even.as_in_context(ctx).reshape(-1,1)\n", + "\n", + " output_digit, output_odd_even = net(data)\n", + " \n", + " acc_digits.update(label_digit, output_digit.softmax())\n", + " acc_odd_even.update(label_odd_even, output_odd_even.sigmoid() > 0.5)\n", + " return acc_digits.get(), acc_odd_even.get()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training Loop" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We need to balance the contribution of each loss to the overall training and do so by tuning this alpha parameter within [0,1]." + ] + }, + { + "cell_type": "code", + "execution_count": 142, + "metadata": {}, + "outputs": [], + "source": [ + "alpha = 0.5 # Combine losses factor" + ] + }, + { + "cell_type": "code", + "execution_count": 143, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Acc Digits 0.8945 Loss Digits 0.3409\n", + "Epoch [0], Acc Odd/Even 0.9561 Loss Odd/Even 0.1152\n", + "Epoch [0], Testing Accuracies (('digits', 0.9487179487179487), ('odd_even', 0.9770633012820513))\n", + "Epoch [1], Acc Digits 0.9576 Loss Digits 0.1475\n", + "Epoch [1], Acc Odd/Even 0.9804 Loss Odd/Even 0.0559\n", + "Epoch [1], Testing Accuracies (('digits', 0.9642427884615384), ('odd_even', 0.9826722756410257))\n", + "Epoch [2], Acc Digits 0.9681 Loss Digits 0.1124\n", + "Epoch [2], Acc Odd/Even 0.9852 Loss Odd/Even 0.0418\n", + "Epoch [2], Testing Accuracies (('digits', 0.9580328525641025), ('odd_even', 0.9846754807692307))\n", + "Epoch [3], Acc Digits 0.9734 Loss Digits 0.0961\n", + "Epoch [3], Acc Odd/Even 0.9884 Loss Odd/Even 0.0340\n", + "Epoch [3], Testing Accuracies (('digits', 0.9670472756410257), ('odd_even', 0.9839743589743589))\n", + "Epoch [4], Acc Digits 0.9762 Loss Digits 0.0848\n", + "Epoch [4], Acc Odd/Even 0.9894 Loss Odd/Even 0.0310\n", + "Epoch [4], Testing Accuracies (('digits', 0.9652887658227848), ('odd_even', 0.9858583860759493))\n" + ] + } + ], + "source": [ + "for e in range(epochs):\n", + " # Accuracies for each task\n", + " acc_digits = mx.metric.Accuracy(name='digits')\n", + " acc_odd_even = mx.metric.Accuracy(name='odd_even')\n", + " # Accumulative losses\n", + " l_digits_ = 0.\n", + " l_odd_even_ = 0. \n", + " \n", + " for i, (data, label_digit, label_odd_even) in enumerate(train_data):\n", + " data = data.as_in_context(ctx)\n", + " label_digit = label_digit.as_in_context(ctx)\n", + " label_odd_even = label_odd_even.as_in_context(ctx).reshape(-1,1)\n", + " \n", + " with autograd.record():\n", + " output_digit, output_odd_even = net(data)\n", + " l_digits = loss_digits(output_digit, label_digit)\n", + " l_odd_even = loss_odd_even(output_odd_even, label_odd_even)\n", + "\n", + " # Combine the loss of each task\n", + " l_combined = (1-alpha)*l_digits + alpha*l_odd_even\n", + " \n", + " l_combined.backward()\n", + " trainer.step(data.shape[0])\n", + " \n", + " l_digits_ += l_digits.mean()\n", + " l_odd_even_ += l_odd_even.mean()\n", + " acc_digits.update(label_digit, output_digit.softmax())\n", + " acc_odd_even.update(label_odd_even, output_odd_even.sigmoid() > 0.5)\n", + " \n", + " print(\"Epoch [{}], Acc Digits {:.4f} Loss Digits {:.4f}\".format(\n", + " e, acc_digits.get()[1], l_digits_.asscalar()/(i+1)))\n", + " print(\"Epoch [{}], Acc Odd/Even {:.4f} Loss Odd/Even {:.4f}\".format(\n", + " e, acc_odd_even.get()[1], l_odd_even_.asscalar()/(i+1)))\n", + " print(\"Epoch [{}], Testing Accuracies {}\".format(e, evaluate_accuracy(net, test_data)))\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Testing" + ] + }, + { + "cell_type": "code", + "execution_count": 144, + "metadata": {}, + "outputs": [], + "source": [ + "def get_random_data():\n", + " idx = random.randint(0, len(test_dataset))\n", + "\n", + " img = test_dataset[idx][0]\n", + " data, _, _ = test_dataset_t[idx]\n", + " data = data.as_in_context(ctx).expand_dims(axis=0)\n", + "\n", + " plt.imshow(img.squeeze().asnumpy(), cmap='gray')\n", + " \n", + " return data" + ] + }, + { + "cell_type": "code", + "execution_count": 152, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted digit: [9.], odd: [1.]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAADeVJREFUeJzt3X+MFPX9x/HXG6QGAQ3aiBdLpd9Ga6pBak5joqk01caaRuAfUhMbjE2viTUpEVFCNT31Dxu1rdWYJldLCk2/QhUb+KPWWuKP1jQNIKiotFJC00OEkjNBEiNyvPvHzdlTbz6zzs7uzPF+PpLL7e57Z+ad5V7M7H5m9mPuLgDxTKq7AQD1IPxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4I6oZsbMzNOJwQ6zN2tlee1tec3s6vM7O9mtsvMVrSzLgDdZWXP7TezyZL+IelKSYOSNku61t1fSyzDnh/osG7s+S+WtMvdd7v7EUlrJS1oY30Auqid8J8p6d9j7g9mj32ImfWZ2RYz29LGtgBUrOMf+Ln7gKQBicN+oEna2fPvlTR7zP3PZI8BmADaCf9mSWeb2efM7FOSvilpYzVtAei00of97n7UzG6S9JSkyZJWufurlXUGoKNKD/WV2hjv+YGO68pJPgAmLsIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCKj1FtySZ2R5J70galnTU3XuraApA57UV/sxX3P1gBesB0EUc9gNBtRt+l/RHM9tqZn1VNASgO9o97L/M3fea2emSnjazne7+/NgnZP8p8B8D0DDm7tWsyKxf0mF3vz/xnGo2BiCXu1srzyt92G9m08xsxuhtSV+TtKPs+gB0VzuH/bMk/c7MRtfz/+7+h0q6AtBxlR32t7QxDvuBjuv4YT+AiY3wA0ERfiAowg8ERfiBoAg/EFQVV/WhwaZPn56sL1++vK3lb7755mT97bffzq3deeedyWUffvjhZP3o0aPJOtLY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUFzSOwFMnTo1WV+xYkVurWgcftq0acl69n0NuTr591M0zr9s2bJk/ciRI1W2M2FwSS+AJMIPBEX4gaAIPxAU4QeCIvxAUIQfCIpx/i4oGqe//PLLk/Vbb701WZ8/f/4nballQ0NDbdWnTJmSWzvrrLNK9TTqySefTNafe+653NoDDzyQXHYinyPAOD+AJMIPBEX4gaAIPxAU4QeCIvxAUIQfCKpwnN/MVkn6hqQD7n5+9tipktZJmiNpj6TF7p7/Be3/W9dxOc5/0kknJesPPvhgsn7DDTdU2c6H7NixI1m/5557kvVt27Yl6zt37kzWZ8yYkVt76qmnkstecsklyXo7zjnnnGR9165dHdt2p1U5zv8rSVd95LEVkja5+9mSNmX3AUwgheF39+clffQ0rgWSVme3V0taWHFfADqs7Hv+We6+L7v9lqRZFfUDoEvanqvP3T31Xt7M+iT1tbsdANUqu+ffb2Y9kpT9PpD3RHcfcPded+8tuS0AHVA2/BslLcluL5G0oZp2AHRLYfjN7FFJf5X0BTMbNLNvS/qRpCvN7A1JV2T3AUwghe/53f3anNJXK+5lwrriiiuS9XbH8Q8ePJisr1u3Lrd2yy23JJd97733SvXUqp6entq2jTTO8AOCIvxAUIQfCIrwA0ERfiAowg8E1fbpvVGkprJevnx5R7f9yCOPJOsrV67s2LZPOCH9J7Jo0aJk/aGHHsqtnX766aV6atUzzzyTW9u7d29Htz0RsOcHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAY52/RHXfckVu79NJL21p30Tj+3Xff3db6U84999xkfenSpcl6X19zv6Ht3nvvza29++67XeykmdjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPO3qJPXnq9ZsyZZLxqTTk03XTROv3jx4mT9tNNOS9aLpnjvpNR3BUjSs88+251GJij2/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVOE4v5mtkvQNSQfc/fzssX5J35H0n+xpK939951qsgk2b96cW7v++uvbWveGDRuS9SNHjiTrU6dOza2dfPLJpXoa9f777yfr1113XbKemlNg7ty5pXoa9dhjjyXrTAGe1sqe/1eSrhrn8Z+6+7zs57gOPnA8Kgy/uz8vaagLvQDoonbe899kZi+b2Sozm1lZRwC6omz4fy7p85LmSdon6cd5TzSzPjPbYmZbSm4LQAeUCr+773f3YXc/JukXki5OPHfA3XvdvbdskwCqVyr8ZtYz5u4iSTuqaQdAt7Qy1PeopPmSPm1mg5J+KGm+mc2T5JL2SPpuB3sE0AHWzeuxzay+i7/bNGlS/kHS448/nlx24cKFVbdTmRdeeCFZv+uuu5L1ovMIisbiU4p6mz9/frI+PDxcetsTmbtbK8/jDD8gKMIPBEX4gaAIPxAU4QeCIvxAUHx1d4uOHTuWW7vxxhuTy+7fvz9ZL7osdufOncn6E088kVsr+nrrw4cPJ+snnnhisl40HGeWP+qUek0ladOmTcl61KG8qrDnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGguKQXSWeccUay/uabb5Ze9/bt25P1Cy+8sPS6I+OSXgBJhB8IivADQRF+ICjCDwRF+IGgCD8QFNfzI6m/v7+t5VNTfK9du7atdaM97PmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKjC6/nNbLakNZJmSXJJA+7+MzM7VdI6SXMk7ZG02N3fLlgX1/M3zKJFi5L11JwAklT093Pffffl1m677bbksiinyuv5j0pa5u5flHSJpO+Z2RclrZC0yd3PlrQpuw9ggigMv7vvc/cXs9vvSHpd0pmSFkhanT1ttaSFnWoSQPU+0Xt+M5sj6UuS/iZplrvvy0pvaeRtAYAJouVz+81suqT1kpa6+6Gxc7C5u+e9nzezPkl97TYKoFot7fnNbIpGgv8bdx/9BGi/mfVk9R5JB8Zb1t0H3L3X3XuraBhANQrDbyO7+F9Ket3dfzKmtFHSkuz2Ekkbqm8PQKe0MtR3maQ/S3pF0uicyis18r7/t5I+K+lfGhnqGypYF0N9DfPSSy8l63Pnzk3Wh4aS/+S64IILcmuDg4PJZVFOq0N9he/53f0vkvJW9tVP0hSA5uAMPyAowg8ERfiBoAg/EBThB4Ii/EBQfHX3ca7ostnzzjsvWR8eHk7Wb7/99mSdsfzmYs8PBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0EVXs9f6ca4nr8j5syZk1vbtm1bctlTTjklWd+6dWuyftFFFyXr6L4qv7obwHGI8ANBEX4gKMIPBEX4gaAIPxAU4QeC4nr+48DSpUtza0Xj+EX6+/vbWh7NxZ4fCIrwA0ERfiAowg8ERfiBoAg/EBThB4IqvJ7fzGZLWiNpliSXNODuPzOzfknfkfSf7Kkr3f33Beviev4SrrnmmmR9/fr1ubXJkye3te1Jk9g/TDStXs/fykk+RyUtc/cXzWyGpK1m9nRW+6m731+2SQD1KQy/u++TtC+7/Y6ZvS7pzE43BqCzPtExnZnNkfQlSX/LHrrJzF42s1VmNjNnmT4z22JmW9rqFEClWg6/mU2XtF7SUnc/JOnnkj4vaZ5Gjgx+PN5y7j7g7r3u3ltBvwAq0lL4zWyKRoL/G3d/QpLcfb+7D7v7MUm/kHRx59oEULXC8JuZSfqlpNfd/SdjHu8Z87RFknZU3x6ATmnl0/5LJX1L0itmtj17bKWka81snkaG//ZI+m5HOoR2796drB86dCi3NnPmuB/FfOD++xmsiaqVT/v/Imm8ccPkmD6AZuMMDiAowg8ERfiBoAg/EBThB4Ii/EBQTNENHGeYohtAEuEHgiL8QFCEHwiK8ANBEX4gKMIPBNXtKboPSvrXmPufzh5roqb21tS+JHorq8rezmr1iV09yedjGzfb0tTv9mtqb03tS6K3surqjcN+ICjCDwRVd/gHat5+SlN7a2pfEr2VVUtvtb7nB1Cfuvf8AGpSS/jN7Coz+7uZ7TKzFXX0kMfM9pjZK2a2ve4pxrJp0A6Y2Y4xj51qZk+b2RvZ7/R3c3e3t34z25u9dtvN7OqaepttZs+Y2Wtm9qqZfT97vNbXLtFXLa9b1w/7zWyypH9IulLSoKTNkq5199e62kgOM9sjqdfdax8TNrMvSzosaY27n589dq+kIXf/UfYf50x3v60hvfVLOlz3zM3ZhDI9Y2eWlrRQ0vWq8bVL9LVYNbxudez5L5a0y913u/sRSWslLaihj8Zz9+clDX3k4QWSVme3V2vkj6frcnprBHff5+4vZrffkTQ6s3Str12ir1rUEf4zJf17zP1BNWvKb5f0RzPbamZ9dTczjlnZtOmS9JakWXU2M47CmZu76SMzSzfmtSsz43XV+MDv4y5z9wslfV3S97LD20bykfdsTRquaWnm5m4ZZ2bpD9T52pWd8bpqdYR/r6TZY+5/JnusEdx9b/b7gKTfqXmzD+8fnSQ1+32g5n4+0KSZm8ebWVoNeO2aNON1HeHfLOlsM/ucmX1K0jclbayhj48xs2nZBzEys2mSvqbmzT68UdKS7PYSSRtq7OVDmjJzc97M0qr5tWvcjNfu3vUfSVdr5BP/f0r6QR095PT1f5Jeyn5erbs3SY9q5DDwfY18NvJtSadJ2iTpDUl/knRqg3r7taRXJL2skaD11NTbZRo5pH9Z0vbs5+q6X7tEX7W8bpzhBwTFB35AUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4L6L4bahh5ke9v1AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "data = get_random_data()\n", + "\n", + "digit, odd_even = net(data)\n", + "\n", + "digit = digit.argmax(axis=1)[0].asnumpy()\n", + "odd_even = (odd_even.sigmoid()[0] > 0.5).asnumpy()\n", + "\n", + "print(\"Predicted digit: {}, odd: {}\".format(digit, odd_even))" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/example/multivariate_time_series/README.md b/example/multivariate_time_series/README.md index 704c86ae7..87baca36d 100644 --- a/example/multivariate_time_series/README.md +++ b/example/multivariate_time_series/README.md @@ -3,6 +3,8 @@ - This repo contains an MXNet implementation of [this](https://arxiv.org/pdf/1703.07015.pdf) state of the art time series forecasting model. - You can find my blog post on the model [here](https://opringle.github.io/2018/01/05/deep_learning_multivariate_ts.html) +- A Gluon implementation is available [here](https://github.com/safrooze/LSTNet-Gluon) + ![](./docs/model_architecture.png) ## Running the code @@ -22,7 +24,7 @@ ## Hyperparameters -The default arguements in `lstnet.py` achieve equivolent performance to the published results. For other datasets, the following hyperparameters provide a good starting point: +The default arguements in `lstnet.py` achieve equivalent performance to the published results. For other datasets, the following hyperparameters provide a good starting point: - q = {2^0, 2^1, ... , 2^9} (1 week is typical value) - Convolutional num filters = {50, 100, 200} diff --git a/example/named_entity_recognition/README.md b/example/named_entity_recognition/README.md index 260c19d5f..c914a6985 100644 --- a/example/named_entity_recognition/README.md +++ b/example/named_entity_recognition/README.md @@ -11,9 +11,16 @@ To reproduce the preprocessed training data: 1. Download and unzip the data: https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus/downloads/ner_dataset.csv 2. Move ner_dataset.csv into `./data` -3. create `./preprocessed_data` directory 3. `$ cd src && python preprocess.py` To train the model: -- `$ cd src && python ner.py` \ No newline at end of file +- `$ cd src && python ner.py` + +To run inference using trained model: + +1. Recreate the bucketing module using `sym_gen` defined in `ner.py` +2. Loading saved parameters using `module.set_params()` + +Refer to the `test` function in the [Bucketing Module example](https://github.com/apache/incubator-mxnet/blob/master/example/rnn/bucketing/cudnn_rnn_bucketing.py) +and this [issue](https://github.com/apache/incubator-mxnet/issues/5008) on Bucketing Module Prediction \ No newline at end of file diff --git a/example/named_entity_recognition/src/metrics.py b/example/named_entity_recognition/src/metrics.py index 40c5015e8..d3d73782c 100644 --- a/example/named_entity_recognition/src/metrics.py +++ b/example/named_entity_recognition/src/metrics.py @@ -27,7 +27,7 @@ def load_obj(name): with open(name + '.pkl', 'rb') as f: return pickle.load(f) -tag_dict = load_obj("../preprocessed_data/tag_to_index") +tag_dict = load_obj("../data/tag_to_index") not_entity_index = tag_dict["O"] def classifer_metrics(label, pred): diff --git a/example/named_entity_recognition/src/ner.py b/example/named_entity_recognition/src/ner.py index 561db4c43..7f5dd8452 100644 --- a/example/named_entity_recognition/src/ner.py +++ b/example/named_entity_recognition/src/ner.py @@ -34,7 +34,7 @@ parser = argparse.ArgumentParser(description="Deep neural network for multivariate time series forecasting", formatter_class=argparse.ArgumentDefaultsHelpFormatter) -parser.add_argument('--data-dir', type=str, default='../preprocessed_data', +parser.add_argument('--data-dir', type=str, default='../data', help='relative path to input data') parser.add_argument('--output-dir', type=str, default='../results', help='directory to save model files to') diff --git a/example/nce-loss/README.md b/example/nce-loss/README.md index 70730b477..56e43525a 100644 --- a/example/nce-loss/README.md +++ b/example/nce-loss/README.md @@ -29,7 +29,7 @@ The dataset used in the following examples is [text8](http://mattmahoney.net/dc/ * word2vec.py: a CBOW word2vec example using nce loss. You need to [download the text8 dataset](#dataset-download) before running this script. Command to start training on CPU (pass -g for training on GPU): ``` -python word2vec.py +python wordvec.py ``` diff --git a/example/notebooks/README.md b/example/notebooks/README.md deleted file mode 100644 index 27ff7fabb..000000000 --- a/example/notebooks/README.md +++ /dev/null @@ -1,4 +0,0 @@ -Moved to -https://github.com/dmlc/mxnet-notebooks/tree/master/python/moved-from-mxnet/ - -This folder will be removed soon. diff --git a/example/numpy-ops/numpy_softmax.py b/example/numpy-ops/numpy_softmax.py deleted file mode 100644 index 88d247349..000000000 --- a/example/numpy-ops/numpy_softmax.py +++ /dev/null @@ -1,84 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: skip-file -import mxnet as mx -from mxnet.test_utils import get_mnist_iterator -import numpy as np -import logging - - -class NumpySoftmax(mx.operator.NumpyOp): - def __init__(self): - super(NumpySoftmax, self).__init__(False) - - def list_arguments(self): - return ['data', 'label'] - - def list_outputs(self): - return ['output'] - - def infer_shape(self, in_shape): - data_shape = in_shape[0] - label_shape = (in_shape[0][0],) - output_shape = in_shape[0] - return [data_shape, label_shape], [output_shape] - - def forward(self, in_data, out_data): - x = in_data[0] - y = out_data[0] - y[:] = np.exp(x - x.max(axis=1).reshape((x.shape[0], 1))) - y /= y.sum(axis=1).reshape((x.shape[0], 1)) - - def backward(self, out_grad, in_data, out_data, in_grad): - l = in_data[1] - l = l.reshape((l.size,)).astype(np.int) - y = out_data[0] - dx = in_grad[0] - dx[:] = y - dx[np.arange(l.shape[0]), l] -= 1.0 - -# define mlp - -data = mx.symbol.Variable('data') -fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128) -act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type="relu") -fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64) -act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type="relu") -fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10) -#mlp = mx.symbol.Softmax(data = fc3, name = 'mlp') -mysoftmax = NumpySoftmax() -mlp = mysoftmax(data=fc3, name = 'softmax') - -# data - -train, val = get_mnist_iterator(batch_size=100, input_shape = (784,)) - -# train - -logging.basicConfig(level=logging.DEBUG) - -# MXNET_CPU_WORKER_NTHREADS must be greater than 1 for custom op to work on CPU -context=mx.cpu() -# Uncomment this line to train on GPU instead of CPU -# context=mx.gpu(0) - -mod = mx.mod.Module(mlp, context=context) - -mod.fit(train_data=train, eval_data=val, optimizer='sgd', - optimizer_params={'learning_rate':0.1, 'momentum': 0.9, 'wd': 0.00001}, - num_epoch=10, batch_end_callback=mx.callback.Speedometer(100, 100)) diff --git a/example/onnx/super_resolution.py b/example/onnx/super_resolution.py deleted file mode 100644 index fcb8ccc88..000000000 --- a/example/onnx/super_resolution.py +++ /dev/null @@ -1,86 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Testing super_resolution model conversion""" -from __future__ import absolute_import as _abs -from __future__ import print_function -from collections import namedtuple -import logging -import numpy as np -from PIL import Image -import mxnet as mx -from mxnet.test_utils import download -import mxnet.contrib.onnx as onnx_mxnet - -# set up logger -logging.basicConfig() -LOGGER = logging.getLogger() -LOGGER.setLevel(logging.INFO) - -def import_onnx(): - """Import the onnx model into mxnet""" - model_url = 'https://s3.amazonaws.com/onnx-mxnet/examples/super_resolution.onnx' - download(model_url, 'super_resolution.onnx') - - LOGGER.info("Converting onnx format to mxnet's symbol and params...") - sym, arg_params, aux_params = onnx_mxnet.import_model('super_resolution.onnx') - LOGGER.info("Successfully Converted onnx format to mxnet's symbol and params...") - return sym, arg_params, aux_params - -def get_test_image(): - """Download and process the test image""" - # Load test image - input_image_dim = 224 - img_url = 'https://s3.amazonaws.com/onnx-mxnet/examples/super_res_input.jpg' - download(img_url, 'super_res_input.jpg') - img = Image.open('super_res_input.jpg').resize((input_image_dim, input_image_dim)) - img_ycbcr = img.convert("YCbCr") - img_y, img_cb, img_cr = img_ycbcr.split() - input_image = np.array(img_y)[np.newaxis, np.newaxis, :, :] - return input_image, img_cb, img_cr - -def perform_inference(sym, arg_params, aux_params, input_img, img_cb, img_cr): - """Perform inference on image using mxnet""" - metadata = onnx_mxnet.get_model_metadata('super_resolution.onnx') - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - # create module - mod = mx.mod.Module(symbol=sym, data_names=data_names, label_names=None) - mod.bind(for_training=False, data_shapes=[(data_names[0], input_img.shape)]) - mod.set_params(arg_params=arg_params, aux_params=aux_params) - - # run inference - batch = namedtuple('Batch', ['data']) - mod.forward(batch([mx.nd.array(input_img)])) - - # Save the result - img_out_y = Image.fromarray(np.uint8(mod.get_outputs()[0][0][0]. - asnumpy().clip(0, 255)), mode='L') - - result_img = Image.merge( - "YCbCr", [img_out_y, - img_cb.resize(img_out_y.size, Image.BICUBIC), - img_cr.resize(img_out_y.size, Image.BICUBIC)]).convert("RGB") - output_img_dim = 672 - assert result_img.size == (output_img_dim, output_img_dim) - LOGGER.info("Super Resolution example success.") - result_img.save("super_res_output.jpg") - return result_img - -if __name__ == '__main__': - MX_SYM, MX_ARG_PARAM, MX_AUX_PARAM = import_onnx() - INPUT_IMG, IMG_CB, IMG_CR = get_test_image() - perform_inference(MX_SYM, MX_ARG_PARAM, MX_AUX_PARAM, INPUT_IMG, IMG_CB, IMG_CR) diff --git a/example/python-howto/README.md b/example/python-howto/README.md deleted file mode 100644 index 29652408e..000000000 --- a/example/python-howto/README.md +++ /dev/null @@ -1,37 +0,0 @@ -Python Howto Examples -===================== - -* [Configuring Net to Get Multiple Ouputs](multiple_outputs.py) -* [Configuring Image Record Iterator](data_iter.py) -* [Monitor Intermediate Outputs in the Network](monitor_weights.py) -* Set break point in C++ code of the symbol using gdb under Linux: - - * Build mxnet with following values: - - ``` - DEBUG=1 - USE_CUDA=0 # to make sure convolution-inl.h will be used - USE_CUDNN=0 # to make sure convolution-inl.h will be used - ``` - - * run python under gdb: ```gdb --args python debug_conv.py``` - * in gdb set break point on particular line of the code and run execution: - -``` -(gdb) break src/operator/convolution-inl.h:120 -(gdb) run -Breakpoint 1, mxnet::op::ConvolutionOp::Forward (this=0x12219d0, ctx=..., in_data=std::vector of length 3, capacity 4 = {...}, req=std::vector of length 1, capacity 1 = {...}, out_data=std::vector of length 1, capacity 1 = {...}, - aux_args=std::vector of length 0, capacity 0) at src/operator/./convolution-inl.h:121 -121 data.shape_[1] / param_.num_group * param_.kernel[0] * param_.kernel[1]); -(gdb) list -116 } -117 Tensor data = in_data[conv::kData].get(s); -118 Shape<3> wmat_shape = -119 Shape3(param_.num_group, -120 param_.num_filter / param_.num_group, -121 data.shape_[1] / param_.num_group * param_.kernel[0] * param_.kernel[1]); -122 Tensor wmat = -123 in_data[conv::kWeight].get_with_shape(wmat_shape, s); -124 Tensor out = out_data[conv::kOut].get(s); -125 #if defined(__CUDACC__) -``` diff --git a/example/python-howto/data_iter.py b/example/python-howto/data_iter.py deleted file mode 100644 index 81c8988a8..000000000 --- a/example/python-howto/data_iter.py +++ /dev/null @@ -1,76 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Create a Cifar data iterator. - -This example shows how to create a iterator reading from recordio, -introducing image augmentations and using a backend thread to hide IO cost. -All you need to do is to set some parameters. -""" -import mxnet as mx - -dataiter = mx.io.ImageRecordIter( - # Dataset Parameter - # Impulsary - # indicating the data file, please check the data is already there - path_imgrec="data/cifar/train.rec", - # Dataset/Augment Parameter - # Impulsary - # indicating the image size after preprocessing - data_shape=(3,28,28), - # Batch Parameter - # Impulsary - # tells how many images in a batch - batch_size=100, - # Augmentation Parameter - # Optional - # when offers mean_img, each image will subtract the mean value at each pixel - mean_img="data/cifar/cifar10_mean.bin", - # Augmentation Parameter - # Optional - # randomly crop a patch of the data_shape from the original image - rand_crop=True, - # Augmentation Parameter - # Optional - # randomly mirror the image horizontally - rand_mirror=True, - # Augmentation Parameter - # Optional - # randomly shuffle the data - shuffle=False, - # Backend Parameter - # Optional - # Preprocessing thread number - preprocess_threads=4, - # Backend Parameter - # Optional - # Prefetch buffer size - prefetch_buffer=4, - # Backend Parameter, - # Optional - # Whether round batch, - round_batch=True) - -batchidx = 0 -for dbatch in dataiter: - data = dbatch.data[0] - label = dbatch.label[0] - pad = dbatch.pad - index = dbatch.index - print("Batch", batchidx) - print(label.asnumpy().flatten()) - batchidx += 1 diff --git a/example/python-howto/monitor_weights.py b/example/python-howto/monitor_weights.py deleted file mode 100644 index 929b0e7bf..000000000 --- a/example/python-howto/monitor_weights.py +++ /dev/null @@ -1,46 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: skip-file -import mxnet as mx -from mxnet.test_utils import get_mnist_iterator -import numpy as np -import logging - -# network -data = mx.symbol.Variable('data') -fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128) -act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type="relu") -fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64) -act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type="relu") -fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10) -mlp = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax') - -# data -train, val = get_mnist_iterator(batch_size=100, input_shape = (784,)) - -# monitor -def norm_stat(d): - return mx.nd.norm(d)/np.sqrt(d.size) -mon = mx.mon.Monitor(100, norm_stat) - -# train with monitor -logging.basicConfig(level=logging.DEBUG) -module = mx.module.Module(context=mx.cpu(), symbol=mlp) -module.fit(train_data=train, eval_data=val, monitor=mon, num_epoch=2, - batch_end_callback = mx.callback.Speedometer(100, 100), - optimizer_params=(('learning_rate', 0.1), ('momentum', 0.9), ('wd', 0.00001))) diff --git a/example/python-howto/multiple_outputs.py b/example/python-howto/multiple_outputs.py deleted file mode 100644 index 7c1ddd220..000000000 --- a/example/python-howto/multiple_outputs.py +++ /dev/null @@ -1,38 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Create a Multiple output configuration. - -This example shows how to create a multiple output configuration. -""" -from __future__ import print_function -import mxnet as mx - -net = mx.symbol.Variable('data') -fc1 = mx.symbol.FullyConnected(data=net, name='fc1', num_hidden=128) -net = mx.symbol.Activation(data=fc1, name='relu1', act_type="relu") -net = mx.symbol.FullyConnected(data=net, name='fc2', num_hidden=64) -out = mx.symbol.SoftmaxOutput(data=net, name='softmax') -# group fc1 and out together -group = mx.symbol.Group([fc1, out]) -print(group.list_outputs()) - -# You can go ahead and bind on the group -# executor = group.simple_bind(data=data_shape) -# executor.forward() -# executor.output[0] will be value of fc1 -# executor.output[1] will be value of softmax diff --git a/example/quantization/README.md b/example/quantization/README.md index 7f68dd795..b77537d4f 100644 --- a/example/quantization/README.md +++ b/example/quantization/README.md @@ -34,7 +34,7 @@ The following models have been tested on Linux systems. |[Inception V3](#7)|[Gluon-CV](https://gluon-cv.mxnet.io/model_zoo/classification.html)|[Validation Dataset](http://data.mxnet.io/data/val_256_q90.rec)|76.49%/93.10% |76.38%/93% | |[ResNet152-V2](#8)|[MXNet ModelZoo](http://data.mxnet.io/models/imagenet/resnet/152-layers/)|[Validation Dataset](http://data.mxnet.io/data/val_256_q90.rec)|76.76%/93.03%|76.48%/92.96%| |[Inception-BN](#9)|[MXNet ModelZoo](http://data.mxnet.io/models/imagenet/inception-bn/)|[Validation Dataset](http://data.mxnet.io/data/val_256_q90.rec)|72.09%/90.60%|72.00%/90.53%| -| [SSD-VGG](#10) | [example/ssd](https://github.com/apache/incubator-mxnet/tree/master/example/ssd) | VOC2007/2012 | 0.83 mAP | 0.82 mAP | +| [SSD-VGG16](#10) | [example/ssd](https://github.com/apache/incubator-mxnet/tree/master/example/ssd) | VOC2007/2012 | 0.8366 mAP | 0.8364 mAP |

      ResNet50-V1

      @@ -208,42 +208,9 @@ python imagenet_inference.py --symbol-file=./model/imagenet1k-inception-bn-symbo python imagenet_inference.py --symbol-file=./model/imagenet1k-inception-bn-quantized-5batches-naive-symbol.json --batch-size=64 --num-inference-batches=500 --ctx=cpu --benchmark=True ``` -

      SSD-VGG

      +

      SSD-VGG16

      -Follow the [SSD example's instructions](https://github.com/apache/incubator-mxnet/tree/master/example/ssd#train-the-model) in [example/ssd](https://github.com/apache/incubator-mxnet/tree/master/example/ssd) to train a FP32 `SSD-VGG16_reduced_300x300` model based on Pascal VOC dataset. You can also download our [SSD-VGG16 pre-trained model](http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/ssd_vgg16_reduced_300-dd479559.zip) and [packed binary data](http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/ssd-val-fc19a535.zip). Extract the zip files, then rename the directories to `model` and `data` respectively. Then, rename the files in directories as follows. - -``` -data/ -|---val.rec -|---val.lxt -|---val.idx -model/ -|---ssd_vgg16_reduced_300.params -|---ssd_vgg16_reduced_300-symbol.json -``` - -Then, use the following command for quantization. By default, this script uses 5 batches (32 samples per batch) for naive calibration: - -``` -python quantization.py -``` - -After quantization, INT8 models will be saved in `model/` dictionary. Use the following command to launch inference. - -``` -# USE MKLDNN AS SUBGRAPH BACKEND -export MXNET_SUBGRAPH_BACKEND=MKLDNN - -# Launch FP32 Inference -python evaluate.py --cpu --num-batch 10 --batch-size 224 --deploy --prefix=./model/ssd_ - -# Launch INT8 Inference -python evaluate.py --cpu --num-batch 10 --batch-size 224 --deploy --prefix=./model/cqssd_ - -# Launch dummy data Inference -python benchmark_score.py --deploy --prefix=./model/ssd_ -python benchmark_score.py --deploy --prefix=./model/cqssd_ -``` +SSD model is located in [example/ssd](https://github.com/apache/incubator-mxnet/tree/master/example/ssd), follow [the insturctions](https://github.com/apache/incubator-mxnet/tree/master/example/ssd#quantize-model) to run quantized SSD model.

      Custom Model

      @@ -320,4 +287,6 @@ the console to run model quantization for a specific configuration. - `launch_inference.sh` This is a shell script that calculate the accuracies of all the quantized models generated by invoking `launch_quantize.sh`. -**NOTE**: This example has only been tested on Linux systems. \ No newline at end of file +**NOTE**: +- This example has only been tested on Linux systems. +- Performance is expected to decrease with GPU, however the memory footprint of a quantized model is smaller. The purpose of the quantization implementation is to minimize accuracy loss when converting FP32 models to INT8. MXNet community is working on improving the performance. diff --git a/example/quantization/imagenet_gen_qsym_mkldnn.py b/example/quantization/imagenet_gen_qsym_mkldnn.py index 9056f7904..938890bb7 100644 --- a/example/quantization/imagenet_gen_qsym_mkldnn.py +++ b/example/quantization/imagenet_gen_qsym_mkldnn.py @@ -225,11 +225,7 @@ def save_params(fname, arg_params, aux_params, logger=None): rgb_mean = '123.68,116.779,103.939' rgb_std = '58.393, 57.12, 57.375' calib_layer = lambda name: name.endswith('_output') - excluded_sym_names += ['squeezenet0_flatten0_flatten0', - 'squeezenet0_pool0_fwd', - 'squeezenet0_pool1_fwd', - 'squeezenet0_pool2_fwd', - 'squeezenet0_pool3_fwd'] + excluded_sym_names += ['squeezenet0_flatten0_flatten0'] if exclude_first_conv: excluded_sym_names += ['squeezenet0_conv0_fwd'] elif args.model == 'mobilenet1.0': @@ -277,7 +273,9 @@ def save_params(fname, arg_params, aux_params, logger=None): logger.info('rgb_std = %s' % rgb_std) rgb_std = [float(i) for i in rgb_std.split(',')] std_args = {'std_r': rgb_std[0], 'std_g': rgb_std[1], 'std_b': rgb_std[2]} - + combine_mean_std = {} + combine_mean_std.update(mean_args) + combine_mean_std.update(std_args) if calib_mode == 'none': logger.info('Quantizing FP32 model %s' % args.model) qsym, qarg_params, aux_params = quantize_model(sym=sym, arg_params=arg_params, aux_params=aux_params, @@ -298,8 +296,7 @@ def save_params(fname, arg_params, aux_params, logger=None): shuffle=args.shuffle_dataset, shuffle_chunk_seed=args.shuffle_chunk_seed, seed=args.shuffle_seed, - **mean_args, - **std_args) + **combine_mean_std) qsym, qarg_params, aux_params = quantize_model(sym=sym, arg_params=arg_params, aux_params=aux_params, ctx=ctx, excluded_sym_names=excluded_sym_names, diff --git a/example/quantization/imagenet_inference.py b/example/quantization/imagenet_inference.py index 7d380d3e1..0725165b0 100644 --- a/example/quantization/imagenet_inference.py +++ b/example/quantization/imagenet_inference.py @@ -93,6 +93,7 @@ def score(sym, arg_params, aux_params, data, devs, label_name, max_num_examples, if logger is not None: logger.info('Finished inference with %d images' % num) logger.info('Finished with %f images per second', speed) + logger.warn('Note: GPU performance is expected to be slower than CPU. Please refer quantization/README.md for details') for m in metrics: logger.info(m.get()) @@ -180,6 +181,9 @@ def benchmark_score(symbol_file, ctx, batch_size, num_batches, logger=None): logger.info('rgb_std = %s' % rgb_std) rgb_std = [float(i) for i in rgb_std.split(',')] std_args = {'std_r': rgb_std[0], 'std_g': rgb_std[1], 'std_b': rgb_std[2]} + combine_mean_std = {} + combine_mean_std.update(mean_args) + combine_mean_std.update(std_args) label_name = args.label_name logger.info('label_name = %s' % label_name) @@ -205,8 +209,7 @@ def benchmark_score(symbol_file, ctx, batch_size, num_batches, logger=None): shuffle=True, shuffle_chunk_seed=3982304, seed=48564309, - **mean_args, - **std_args) + **combine_mean_std) # loading model sym, arg_params, aux_params = load_model(symbol_file, param_file, logger) diff --git a/example/rcnn/README.md b/example/rcnn/README.md index ab3c8fb88..5e6127ccb 100644 --- a/example/rcnn/README.md +++ b/example/rcnn/README.md @@ -9,7 +9,7 @@ For a gluon imperative version, checkout https://github.com/dmlc/gluon-cv. ### Out-of-box inference models Download any of the following models to the current directory and run `python3 demo.py --dataset $Dataset$ --network $Network$ --params $MODEL_FILE$ --image $YOUR_IMAGE$` to get single image inference. -For example `python3 demo.py --dataset voc --network vgg16 --params vgg16_voc0712.params --image myimage.jpg`, add `--gpu 0` to use GPU optionally. +For example `python3 demo.py --dataset voc --network vgg16 --params vgg16_voc0712.params --image myimage.jpg`, add `--gpu 0` to use GPU, not set to use CPU. Different network has different configuration. Different dataset has different object class names. You must pass them explicitly as command line arguments. | Network | Dataset | Imageset | Reference | Result | Link | @@ -32,6 +32,7 @@ Make a directory `data` and follow `py-faster-rcnn` for data preparation instruc ### Training and evaluation Use `python3 train.py --dataset $Dataset$ --network $Network$ --pretrained $IMAGENET_MODEL_FILE$ --gpus $GPUS$` to train, for example, `python3 train.py --dataset voc --network vgg16 --pretrained model/vgg16-0000.params --gpus 0,1`. +use `python3 train.py --dataset voc --imageset 2007_trainval+2012_trainval --network vgg16 --pretrained model/vgg16-0000.params --gpus 0,1` to train on both of voc2007 and voc2012. Use `python3 test.py --dataset $Dataset$ --network $Network$ --params $MODEL_FILE$ --gpu $GPU$` to evaluate, for example, `python3 test.py --dataset voc --network vgg16 --params model/vgg16-0010.params --gpu 0`. diff --git a/example/rcnn/demo.py b/example/rcnn/demo.py index 2315bb8af..b0a4ddbea 100644 --- a/example/rcnn/demo.py +++ b/example/rcnn/demo.py @@ -92,7 +92,7 @@ def parse_args(): parser.add_argument('--params', type=str, default='', help='path to trained model') parser.add_argument('--dataset', type=str, default='voc', help='training dataset') parser.add_argument('--image', type=str, default='', help='path to test image') - parser.add_argument('--gpu', type=str, default='', help='gpu device eg. 0') + parser.add_argument('--gpu', type=str, default='', help='GPU devices, eg."0,1,2,3" , not set to use CPU.') parser.add_argument('--vis', action='store_true', help='display results') parser.add_argument('--vis-thresh', type=float, default=0.7, help='threshold display boxes') # faster rcnn params diff --git a/example/rcnn/test.py b/example/rcnn/test.py index 3c047d222..e964c9080 100644 --- a/example/rcnn/test.py +++ b/example/rcnn/test.py @@ -35,7 +35,7 @@ def test_net(sym, imdb, args): logger.info('called with args\n{}'.format(pprint.pformat(vars(args)))) # setup context - ctx = mx.gpu(args.gpu) + ctx = mx.cpu() if not args.gpu else mx.gpu(args.gpu) # load testing data test_data = TestLoader(imdb.roidb, batch_size=1, short=args.img_short_side, max_size=args.img_long_side, @@ -94,7 +94,7 @@ def parse_args(): parser.add_argument('--params', type=str, default='', help='path to trained model') parser.add_argument('--dataset', type=str, default='voc', help='training dataset') parser.add_argument('--imageset', type=str, default='', help='imageset splits') - parser.add_argument('--gpu', type=int, default=0, help='gpu device eg. 0') + parser.add_argument('--gpu', type=int, default=0, help='0 to use GPU, not set to use CPU') # faster rcnn params parser.add_argument('--img-short-side', type=int, default=600) parser.add_argument('--img-long-side', type=int, default=1000) diff --git a/example/rcnn/train.py b/example/rcnn/train.py index 0739069af..7b1f2f7f3 100644 --- a/example/rcnn/train.py +++ b/example/rcnn/train.py @@ -33,7 +33,7 @@ def train_net(sym, roidb, args): logger.info('called with args\n{}'.format(pprint.pformat(vars(args)))) # setup multi-gpu - ctx = [mx.gpu(int(i)) for i in args.gpus.split(',')] + ctx = [mx.cpu()] if not args.gpus else [mx.gpu(int(i)) for i in args.gpus.split(',')] batch_size = args.rcnn_batch_size * len(ctx) # load training data @@ -127,7 +127,7 @@ def parse_args(): parser.add_argument('--pretrained', type=str, default='', help='path to pretrained model') parser.add_argument('--dataset', type=str, default='voc', help='training dataset') parser.add_argument('--imageset', type=str, default='', help='imageset splits') - parser.add_argument('--gpus', type=str, default='0', help='gpu devices eg. 0,1') + parser.add_argument('--gpus', type=str, help='GPU devices, eg: "0,1,2,3" , not set to use CPU') parser.add_argument('--epochs', type=int, default=10, help='training epochs') parser.add_argument('--lr', type=float, default=0.001, help='base learning rate') parser.add_argument('--lr-decay-epoch', type=str, default='7', help='epoch to decay lr') diff --git a/example/recommenders/README.md b/example/recommenders/README.md index bf5d7a666..628182c84 100644 --- a/example/recommenders/README.md +++ b/example/recommenders/README.md @@ -1,41 +1,24 @@ -# Recommender Systems with Sparse Data +# Recommender Systems + This directory has a set of examples of how to build various kinds of recommender systems -using MXNet. It also includes a set of tools for using sparse data. +using MXNet. The sparsity of user / item data is handled through the embedding layers that accept +indices as input rather than one-hot encoded vectors. + ## Examples The examples are driven by notebook files. -* [Matrix Factorization part 1: linear and non-linear models](demo1-MF.ipynb) -* [Matrix Factorization part 2: overfitting and deep ResNet](demo1-MF2-fancy.ipynb) -* [Binary classification with negative sampling](demo2-binary.ipynb) -* [Deep Structured Semantic Model (DSSM) for content-based recommendations](demo3-dssm.ipynb) - -## Prerequisite - -The plotting functionality in the above examples requires ```0.12.2``` version of ```Bokeh``` package. The plotting functionality throws following error when a different Bokeh version is loaded. -```bash -ValueError: PATCH-DOC message requires at least one event -``` - -## Re-usable code +* [Matrix Factorization: linear and non-linear models](demo1-MF.ipynb) +* [Deep Structured Semantic Model (DSSM) for content-based recommendations](demo2-dssm.ipynb) -These examples use and demonstrate a number of layers and other tools that can be used outside of these examples. They are all available from the [`recotools`](recotools.py) package. ### Negative Sampling -* `NegativeSamplingDataIter` - -### Loss Layers - -* `CosineLoss` -* `CrossEntropyLoss` - -### Sparse Data Projection layers - -* `SparseRandomProjection` -* `SparseBagOfWordProjection` +* A previous version of this example had an example of negative sampling. For example of negative sampling, please refer to: + [Gluon NLP Sampled Block](https://github.com/dmlc/gluon-nlp/blob/master/src/gluonnlp/model/sampled_block.py) + ## Acknowledgements @@ -43,4 +26,3 @@ Thanks to [xlvector](https://github.com/xlvector/) for the first Matrix Factoriz that provided the basis for these examples. [MovieLens](http://grouplens.org/datasets/movielens/) data from [GroupLens](http://grouplens.org/). - diff --git a/example/recommenders/crossentropy.py b/example/recommenders/crossentropy.py deleted file mode 100644 index ff4480828..000000000 --- a/example/recommenders/crossentropy.py +++ /dev/null @@ -1,141 +0,0 @@ -#!/usr/bin/env python - -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Cross-entropy loss layer for MXNet. -""" -import os -import time - -import numpy as np -import mxnet as mx - -# ref: http://mxnet.io/faq/new_op.html - -class CrossEntropyLoss(mx.operator.CustomOp): - """An output layer that calculates gradient for cross-entropy loss - y * log(p) + (1-y) * log(p) - for label "y" and prediction "p". - However, the output of this layer is the original prediction -- same as - the "data" input, making it useful for tasks like "predict". - If you actually want to use the calculated loss, see CrossEntropyLoss op. - - This is useful for multi-label prediction where each possible output - label is considered independently. - Cross-entropy loss provides a very large penalty for guessing - the wrong answer (0 or 1) confidently. - The gradient calculation is optimized for y only being 0 or 1. - """ - - eps = 1e-6 # Avoid -inf when taking log(0) - eps1 = 1. + eps - eps_1 = 1. - eps - - def forward(self, is_train, req, in_data, out_data, aux): - # Shapes: - # b = minibatch size - # d = number of dimensions - actually_calculate_loss = False - if actually_calculate_loss: - p = in_data[0] # shape=(b,d) - y = in_data[1] - out = y * mx.nd.log(p+self.eps) + (1.-y) * mx.nd.log((self.eps1) - p) - self.assign(out_data[0], req[0], out) - else: - # Just copy the predictions forward - self.assign(out_data[0], req[0], in_data[0]) - - - def backward(self, req, out_grad, in_data, out_data, in_grad, aux): - self.approx_backward(req, out_grad, in_data, out_data, in_grad, aux) - #self.exact_backward(req, out_grad, in_data, out_data, in_grad, aux) - - def approx_backward(self, req, out_grad, in_data, out_data, in_grad, aux): - """Correct grad = (y-p)/(p-p^2) - But if y is just 1 or 0, then this simplifies to - grad = 1/(p-1+y) - which is more numerically stable - """ - p = in_data[0] # shape=(b,d) - y = in_data[1] - grad = -1. / (p - self.eps_1 + y) - self.assign(in_grad[0], req[0], grad) - - - def exact_backward(self, req, out_grad, in_data, out_data, in_grad, aux): - """grad = (y-p)/(p-p^2) - """ - p = in_data[0] # shape=(b,d) - y = in_data[1] # seems right - grad = (p - y) / ((p+self.eps) * (self.eps1 - p)) - self.assign(in_grad[0], req[0], grad) - - -@mx.operator.register("CrossEntropyLoss") -class CrossEntropyProp(mx.operator.CustomOpProp): - def __init__(self): - super(CrossEntropyProp, self).__init__(need_top_grad=False) - - def list_arguments(self): - return ['data','label'] - - def list_outputs(self): - return ['preds'] - - def create_operator(self, ctx, shapes, dtypes): - return CrossEntropyLoss() - - def infer_shape(self, in_shape): - if in_shape[0] != in_shape[1]: - raise ValueError("Input shapes differ. data:%s. label:%s. must be same" - % (str(in_shape[0]),str(in_shape[1]))) - output_shape = in_shape[0] - return in_shape, [output_shape], [] - - -if __name__ == "__main__": - print("Simple test of cross-entropy") - data = mx.symbol.Variable('data') - labs = mx.symbol.Variable('labs') - net = mx.symbol.Custom(data=data, label=labs, name='ce', - op_type='CrossEntropyLoss') - rand = np.random.RandomState(seed=123) - for i in range(20): - sz = (6,4) - d = mx.nd.array(rand.uniform(0.01,0.99,sz)) - l = mx.nd.array(rand.randint(0,2,sz)) - e = net.bind(ctx=mx.cpu(), args={'data':d, 'labs':l}) - e.forward() - print("D:%s" % d.asnumpy()) - print("L:%s" % l.asnumpy()) - print("out:%s" % e.outputs[0].asnumpy()) - out = e.outputs[0].asnumpy() - if np.abs(out).max() > 1e20: - raise ValueError("output too high!") - - print("performance test") - sz = (6,4) - d = mx.nd.array(rand.uniform(0.01,0.99,sz)) - l = mx.nd.array(rand.randint(0,2,sz)) - e = net.bind(ctx=mx.cpu(), args={'data':d, 'labs':l}) - tic = time.time() - for i in range(5000): - e.forward() - e.outputs[0].wait_to_read() - print("5000 tests costs time: %f s" % (time.time()-tic)) - print("Done with test") diff --git a/example/recommenders/demo1-MF.ipynb b/example/recommenders/demo1-MF.ipynb index 96e07431d..e4d545ec9 100644 --- a/example/recommenders/demo1-MF.ipynb +++ b/example/recommenders/demo1-MF.ipynb @@ -4,249 +4,1717 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Matrix Factorization (MF) Example\n", - "Demonstrates matrix factorization with MXNet on the [MovieLens 100k](http://grouplens.org/datasets/movielens/100k/) dataset. \n", + "# Matrix Factorization (MF) Recommender Example\n", + "Demonstrates matrix factorization with MXNet on the [MovieLens 100k](http://grouplens.org/datasets/movielens/100k/) dataset. We perform **collaborative filtering**, where the recommendations are based on previous rating of users.\n", "\n", - "You need to have python package pandas and bokeh installed (pip install pandas bokeh)." + "We are trying to learn embeddings for users and movies, based on user partial ratings of movies, to estimate future movie ratings\n", + "\n", + "![](https://i.imgur.com/twyWChh.png)\n", + "\n", + "\n", + "For more deep learning based architecture for recommendation, refer to this survey: [Deep Learning based Recommender System: A Survey and New Perspectives](https://arxiv.org/pdf/1707.07435.pdf)" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "DEBUG:matplotlib.backends:backend module://ipykernel.pylab.backend_inline version unknown\n" + ] + } + ], "source": [ + "import matplotlib.pyplot as plt\n", "import mxnet as mx\n", - "from movielens_data import get_data_iter, max_id\n", - "from matrix_fact import train" + "from mxnet import gluon, nd, autograd\n", + "import numpy as np\n", + "\n", + "from matrix_fact import train\n", + "from movielens_data import get_dataset, max_id" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Config" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ - "# If MXNet is not compiled with GPU support (e.g. on OSX), set to [mx.cpu(0)]\n", - "# Can be changed to [mx.gpu(0), mx.gpu(1), ..., mx.gpu(N-1)] if there are N GPUs\n", - "ctx = [mx.gpu(0)]" + "ctx = [mx.gpu(0)] if len(mx.test_utils.list_gpus()) > 0 else [mx.cpu()]\n", + "batch_size = 128" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "(944, 1683)" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "train_test_data = get_data_iter(batch_size=50)\n", + "train_dataset, test_dataset = get_dataset()\n", "max_user, max_item = max_id('./ml-100k/u.data')\n", "(max_user, max_item)" ] }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "train_data = gluon.data.DataLoader(train_dataset, shuffle=True, last_batch='rollover', batch_size=batch_size, num_workers=0)\n", + "test_data = gluon.data.DataLoader(test_dataset, shuffle=True, batch_size=batch_size, num_workers=0)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "[38.]\n", + " \n", + "[508.]\n", + " \n", + "[2.]\n", + "\n" + ] + } + ], + "source": [ + "for user, item, score in test_data:\n", + " print(user[0], item[0], score[0])\n", + " break" + ] + }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Linear MF" + "## Linear Matrix Factorization" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": { "collapsed": false }, - "outputs": [], + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "plot\n", + "\n", + "\n", + "user\n", + "\n", + "user\n", + "\n", + "\n", + "linearMF_emb_user_fwd\n", + "\n", + "linearMF_emb_user_fwd\n", + "\n", + "\n", + "linearMF_emb_user_fwd->user\n", + "\n", + "\n", + "\n", + "\n", + "linearMF_relu0\n", + "\n", + "linearMF_relu0\n", + "\n", + "\n", + "linearMF_relu0->linearMF_emb_user_fwd\n", + "\n", + "\n", + "\n", + "\n", + "item\n", + "\n", + "item\n", + "\n", + "\n", + "linearMF_emb_item_fwd\n", + "\n", + "linearMF_emb_item_fwd\n", + "\n", + "\n", + "linearMF_emb_item_fwd->item\n", + "\n", + "\n", + "\n", + "\n", + "linearMF_relu1\n", + "\n", + "linearMF_relu1\n", + "\n", + "\n", + "linearMF_relu1->linearMF_emb_item_fwd\n", + "\n", + "\n", + "\n", + "\n", + "linearMF__mul0\n", + "\n", + "linearMF__mul0\n", + "\n", + "\n", + "linearMF__mul0->linearMF_relu0\n", + "\n", + "\n", + "\n", + "\n", + "linearMF__mul0->linearMF_relu1\n", + "\n", + "\n", + "\n", + "\n", + "linearMF_sum0\n", + "\n", + "linearMF_sum0\n", + "\n", + "\n", + "linearMF_sum0->linearMF__mul0\n", + "\n", + "\n", + "\n", + "\n", + "linearMF_flatten0\n", + "\n", + "linearMF_flatten0\n", + "\n", + "\n", + "linearMF_flatten0->linearMF_sum0\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "def plain_net(k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " score = mx.symbol.Variable('score')\n", - " # user feature lookup\n", - " user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) \n", - " # item feature lookup\n", - " item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " # predict by the inner product, which is elementwise product and then sum\n", - " pred = user * item\n", - " pred = mx.symbol.sum(data = pred, axis = 1)\n", - " pred = mx.symbol.Flatten(data = pred)\n", - " # loss layer\n", - " pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)\n", - " return pred\n", + "class LinearMatrixFactorization(gluon.HybridBlock):\n", + " \n", + " def __init__(self, k, max_user=max_user, max_item=max_item):\n", + " super(LinearMatrixFactorization, self).__init__(prefix='linearMF_')\n", + " \n", + " # user feature lookup\n", + " with self.name_scope():\n", + " self.user_embedding = gluon.nn.Embedding(input_dim=max_user, output_dim = k, prefix='emb_user_') \n", + "\n", + " # item feature lookup\n", + " self.item_embedding = gluon.nn.Embedding(input_dim=max_item, output_dim = k, prefix='emb_item_') \n", + " \n", + " def hybrid_forward(self, F, user, item):\n", + " user_embeddings = self.user_embedding(user).relu()\n", + " items_embeddings = self.item_embedding(item).relu()\n", + " \n", + " # predict by the inner product, which is elementwise product and then sum\n", + " pred = (user_embeddings * items_embeddings).sum(axis=1)\n", + " \n", + " return pred.flatten()\n", "\n", - "net1 = plain_net(64)\n", - "mx.viz.plot_network(net1)" + "net1 = LinearMatrixFactorization(64)\n", + "net1.initialize(mx.init.Xavier(), ctx=ctx)\n", + "mx.viz.plot_network(net1(mx.sym.var('user'), mx.sym.var('item')), node_attrs={\"fixedsize\":\"false\"})" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--------------------------------------------------------------------------------\n", + " Layer (type) Output Shape Param #\n", + "================================================================================\n", + " Input (128,), (128,) 0\n", + " Embedding-1 (128, 64) 60416\n", + " Embedding-2 (128, 64) 107712\n", + "LinearMatrixFactorization-3 (128, 1) 0\n", + "================================================================================\n", + "Parameters in forward computation graph, duplicate included\n", + " Total params: 168128\n", + " Trainable params: 168128\n", + " Non-trainable params: 0\n", + "Shared params in forward computation graph: 0\n", + "Unique parameters in model: 168128\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "net1.summary(user.as_in_context(ctx[0]), item.as_in_context(ctx[0]))" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, "metadata": { "collapsed": false, "scrolled": false }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Training RMSE 6.1854, Test RMSE 5.2134\n", + "Epoch [1], Training RMSE 2.9043, Test RMSE 2.1358\n", + "Epoch [2], Training RMSE 1.3456, Test RMSE 1.3472\n", + "Epoch [3], Training RMSE 0.9293, Test RMSE 1.0726\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[3126]: Change learning rate to 2.00000e-01\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [4], Training RMSE 0.7585, Test RMSE 0.9467\n", + "Epoch [5], Training RMSE 0.6742, Test RMSE 0.9301\n", + "Epoch [6], Training RMSE 0.6587, Test RMSE 0.9139\n", + "Epoch [7], Training RMSE 0.6449, Test RMSE 0.9023\n", + "Epoch [8], Training RMSE 0.6324, Test RMSE 0.8886\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[6251]: Change learning rate to 4.00000e-02\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [9], Training RMSE 0.6210, Test RMSE 0.8793\n", + "Epoch [10], Training RMSE 0.6100, Test RMSE 0.8764\n", + "Epoch [11], Training RMSE 0.6080, Test RMSE 0.8744\n", + "Epoch [12], Training RMSE 0.6059, Test RMSE 0.8747\n", + "Epoch [13], Training RMSE 0.6039, Test RMSE 0.8717\n", + "Epoch [14], Training RMSE 0.6020, Test RMSE 0.8688\n" + ] + } + ], + "source": [ + "losses_1 = train(net1, train_data, test_data, epochs=15, learning_rate=1, ctx=ctx)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[(6.185443237304687, 5.213418274168756),\n", + " (2.9042590438842772, 2.1358377728492592),\n", + " (1.345566930294037, 1.347152523554055),\n", + " (0.9292820259094239, 1.0726493737500185),\n", + " (0.7584892754554748, 0.9466582980884868),\n", + " (0.6742267098426818, 0.9300613839914844),\n", + " (0.6587229638576507, 0.9138735935186885),\n", + " (0.6448600271701813, 0.9023025612922231),\n", + " (0.632410079240799, 0.8885752661212994),\n", + " (0.6209696002960206, 0.8793287337965267),\n", + " (0.6100408156871796, 0.8763645693754695),\n", + " (0.6079610646724701, 0.8743740775782591),\n", + " (0.6059287497997284, 0.8747020732065675),\n", + " (0.6039103961467743, 0.8717364558748378),\n", + " (0.6019688241481781, 0.8687770996883417)]" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "losses_1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The optimizer used for training and hyper-parameter influence greatly how fast the model converge.\n", + "We can try with the [Adam optimizer](https://arxiv.org/abs/1412.6980) which will often converge much faster than SGD without momentum as we used before. You should see this model over-fitting quickly. " + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, "outputs": [], "source": [ - "results1 = train(net1, train_test_data, num_epoch=15, learning_rate=0.02, ctx=ctx)" + "net1 = LinearMatrixFactorization(64)\n", + "net1.initialize(mx.init.Xavier(), ctx=ctx)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Training RMSE 1.2345, Test RMSE 0.7134\n", + "Epoch [1], Training RMSE 0.6484, Test RMSE 0.6597\n", + "Epoch [2], Training RMSE 0.5852, Test RMSE 0.6618\n", + "Epoch [3], Training RMSE 0.5195, Test RMSE 0.5936\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[3126]: Change learning rate to 2.00000e-03\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [4], Training RMSE 0.4531, Test RMSE 0.5900\n", + "Epoch [5], Training RMSE 0.2978, Test RMSE 0.4903\n", + "Epoch [6], Training RMSE 0.2770, Test RMSE 0.4891\n", + "Epoch [7], Training RMSE 0.2710, Test RMSE 0.4920\n", + "Epoch [8], Training RMSE 0.2654, Test RMSE 0.4949\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[6251]: Change learning rate to 4.00000e-04\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [9], Training RMSE 0.2613, Test RMSE 0.4922\n", + "Epoch [10], Training RMSE 0.2311, Test RMSE 0.4868\n", + "Epoch [11], Training RMSE 0.2284, Test RMSE 0.4876\n", + "Epoch [12], Training RMSE 0.2278, Test RMSE 0.4886\n", + "Epoch [13], Training RMSE 0.2274, Test RMSE 0.4898\n", + "Epoch [14], Training RMSE 0.2272, Test RMSE 0.4899\n" + ] + } + ], + "source": [ + "losses_1_adam = train(net1, train_data, test_data, epochs=15, optimizer='adam', learning_rate=0.01, ctx=ctx)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Neural Network (non-linear) MF" + "### Visualizing embeddings" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(944, 1683)" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ratings = nd.dot(net1.user_embedding.weight.data(ctx=ctx[0]), net1.item_embedding.weight.data(ctx=ctx[0]).T).asnumpy()\n", + "ratings.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "# Helper function to print the recommendation matrix\n", + "# And the top 5 movies in several categories\n", + "\n", + "def evaluate_embeddings(ratings):\n", + " plt.figure(figsize=(15,15))\n", + " plt.xlabel('items')\n", + " plt.ylabel('users')\n", + " plt.title('Users estimated ratings of items sorted by mean ratings across users')\n", + " im = plt.imshow(((ratings[:, ratings.mean(axis=0).argsort()[::-1]])))\n", + " cb = plt.colorbar(im,fraction=0.026, pad=0.04, label=\"score\")\n", + " \n", + " top_5_movies = ratings.mean(axis=0).argsort()[::-1][:5] # Highest mean projected rating\n", + " worst_5_movies = ratings.mean(axis=0).argsort()[:5] # Lowest mean projected rating\n", + " top_5_controversial = ratings.std(axis=0).argsort()[::-1][:5] # With most variance\n", + " \n", + " with open('ml-100k/u.item', 'rb') as f:\n", + " movies = f.readlines()\n", + " \n", + " print(\"Top 5 movies:\")\n", + " for movie in top_5_movies:\n", + " print(\"{}, average rating {:.2f}\".format(str(movies[int(movie)-1]).split(\"|\")[1], ratings.mean(axis=0)[movie]))\n", + " print(\"\\nWorst 5 movies:\")\n", + " for movie in worst_5_movies:\n", + " print(\"{}, average rating {:.2f}\".format(str(movies[int(movie)-1]).split(\"|\")[1], ratings.mean(axis=0)[movie]))\n", + " print(\"\\n5 most controversial movies:\")\n", + " for movie in top_5_controversial:\n", + " print(\"{}, average rating {:.2f}\".format(str(movies[int(movie)-1]).split(\"|\")[1], ratings.mean(axis=0)[movie]))" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=10.0 to DejaVu Sans ('/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n", + "DEBUG:matplotlib.font_manager:findfont: Matching :family=sans-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=12.0 to DejaVu Sans ('/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf') with score of 0.050000\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Top 5 movies:\n", + "Schindler's List (1993), average rating 4.18\n", + "L.A. Confidential (1997), average rating 4.16\n", + "Star Wars (1977), average rating 4.16\n", + "Titanic (1997), average rating 4.15\n", + "Shawshank Redemption, The (1994), average rating 4.13\n", + "\n", + "Worst 5 movies:\n", + "Homage (1995), average rating -0.00\n", + "Bird of Prey (1996), average rating -0.00\n", + "Promise, The (Versprechen, Das) (1994), average rating -0.00\n", + "Fear, The (1995), average rating -0.00\n", + "Window to Paris (1994), average rating -0.00\n", + "\n", + "5 most controversial movies:\n", + "Pulp Fiction (1994), average rating 3.68\n", + "Independence Day (ID4) (1996), average rating 3.15\n", + "Clockwork Orange, A (1971), average rating 3.29\n", + "Big Night (1996), average rating 3.27\n", + "Apocalypse Now (1979), average rating 3.54\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5YAAAHzCAYAAABbrYK+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXm0b9lV1/uZa+3m15z+tnVv3brVpQLEhCaAAYMPnmAwgtijYoOKkNf5VBDF5hkYojyfA7F9Nu8pQpRGnjikcQQYNCIhIQTpQiWpSvXNbc49/a/ZzVrz/THX/p1fXe65VZUU1C3Y3zHOOOf89t5rzdXsved3ze9cP1FVevTo0aNHjx49evTo0aNHj48V7tU2oEePHj169OjRo0ePHj16vLbRE8sePXr06NGjR48ePXr06PFxoSeWPXr06NGjR48ePXr06NHj40JPLHv06NGjR48ePXr06NGjx8eFnlj26NGjR48ePXr06NGjR4+PCz2x7NGjR48ePXr06NGjR48eHxd6YtmjR49fN4jIfxGRP/Vq23EriMiPi8hX/BqW/zki8uFfq/I/FojI7xORp0XkSEQ+9RbHj0Tk/lfDttcCROSdIvKuE459rog88+tt02sNd+J90aNHjx49Pjb0xLJHjzsIIqIi8uBNn53ovN7JuJXdqvq7VPXf/hrUdW/qu+yVLvtjxc1jqao/qaqvfzVtugX+PvC/quqKqv73mw+mzx8DEJFvFZG//etu4a8hROQJEfn8V9uO30x4jdwXPXr06NHjY0BPLHv0+E2IO4mAvRbxG6j/LgMffLWN+PXGb6Dxu6Pwm61ff6O09zdKO3r06PHqoyeWPXq8hiAip0Xk+0VkT0R2ROQnRcSlYxdE5P8Tkesi8riI/Pml694pIt8jIu8SkQPgy0XkM0XkZ0XkQESuisg336beLxKRn0/1vkdE3rR07K+IyLMicigiHxaR3yEiXwj8NeBLk5zyF9K5C7mpiHy5iPyUiPyDVO5jIvLZ6fOnReTasmxWRH63iPz3ZO/TIvLOJRP/a/q9l+r7rHTNnxGRh0VkV0TeLSKXl8r7AhH5kIjsi8g/AeQ27T+p/3462f68iPwTESnS+Z09v5Ds+dKbpZEpWvY1IvKLyYbvEpHB0vGvTeU+JyJfsRzpEZG3i8ivpD5/VkS+5gS7nYj8DRF5MvXnt4nIuoiUInIE+GTjR0+4XkXkQRH5SuDLgK9N7fm+dPzF5tx/SH12KCK/JCIPicjXJVueFpHfuXT+l6c5cJjK+rITbDpx3orI7xGRD6Yx+XER+cSb+vuviMgvAhMR+Q7gHuD7Upu+Np33ljTH90TkF0Tkc5fKuE9EfiLZ+MPA6VvZeJO9f01EtlP9X5Y++4xku1867/dLuk9uUca3isg/E5OSH4ndN+dF5FvS3P6QLEmZX2RcTpy36biKyDtE5JF0zj8VkVveG/LavS8eEJEfFZEbaWz+nYhsLB2/JCL/MfXfDbHnw83PrBvAO+WEeyydP0h9cyP1x/tF5NxSWS9lvr9AKXCL/vpVz9/0uRORvyoiH031f7eIbKVjncLjz4rIU8CP3s7WHj169HjJUNX+p//pf+6QH0CBB2/67J3Au9Lffxf450Cefj4HI0QO+ADwfwAFcD/wGPC2pTIa4Pemc4fATwN/Ih1fAd5ygk2fClwDfitGRP4U8ARQAq8HngYupHPvBR642e6lsn4c+Ir095cDLfCnU7l/G3gK+Kep7N8JHAIr6fzPBd6Y7H8TcBX4vUv1KpAt1fUlwKPAJwIZ8DeA96Rjp1PZfzD1419MtnzFCX1wq/57M/CWVPa9wMPAXzhpLJP9zyz9/wTwM8AFYCtd/4507AuBK8AbgBHwruXygOeBz0l/bwKfdoLdfyb1wf1pjP8j8O23m28nzUfgW4G/vXTspcy5OfC21EffBjwO/PXU538OeDydOwYOgNen/+8C3nCCTbect8BDwAT4glT+16a2F0v9/fPAJWC49NnnL5V9EbgBvD217wvS/2eW6v5mbH7+dmwOvesEOz8Xm1Pd+f9Dsq9r468Av2vp/O8FvvqEsr4V2Mbm3AD40dSXf5Lje+fHXuK4vJR5+/3ABka8rwNf+BvsvngwjW0JnMEWpr4lHfPALwD/AJuXA+CtNz2z/rfUviG3uceArwK+L9nqU9+s8fLm+7fywvtu0V/c/vn7vwPvBe5O7fwXwHfc9Lz8tmTL8CRbX+yd1f/0P/1P/7P800cse/R4baHBnJDLqtqo5Scp8BmY8/sNqlqr5cX9K+CPLF3706r6n1Q1quoslfWgiJxW1SNVfe8JdX4l8C9U9X2qGtRyJCvMeQyY0/JJIpKr6hOqesvo1wl4XFX/jaoG4Lswp/8bVLVS1R8CaswJRFV/XFV/Kdn/i8B3YM76SXgH8HdV9WFVbYG/A3yKWNTy7cAHVfV7VLUBvgVzWG+HF/Sfqn5AVd+rqq2qPoE5brez51b4R6r6nKruYE7dp6TP/zDwb1T1g6o6xRz4ZTRYn6+p6q6q/twJ5X8Z8M2q+piqHgFfB/wReWWkby9lzv2kqr479f9/wJz4b0p9/p3AvUuRogj8FhEZqurzqnqSRPekefulwA+o6g+n8v8+5jB/9tK1/0hVn07z/1b448APquoPpnH+YeBngbeLyD2pzX8zzc//io3Zi6E7/yeAH8DGFuDfpvpIkaS3Af/+NuV8b5pzc4yEzlX125bunS5iedtxeYnz9ptUdU9VnwJ+jON5eSu85u4LVX00zZNKVa9j5L+z8TMxUvuXVXWiqnNV/W9Llz+nqv84tW/G7e+xBjiFEd+Q+uYglfNS5/vtcLvn7zuAv66qz6hqlfrqD950778ztbF7H5xka48ePXq8JPTEskePOwsBi7YsI8de+gD/F7Y6/kNJRvVX0+eXgQtJwrQnInuYFHVZyvT0TeX+WSzK86Eke/qiE2y6DHz1TWVfwlbJHwX+Aua0XBOR7xSRCy+jvVeX/p4BqOrNn60AiMhvFZEfS/K0fcxxup0U8TLwD5ds3sGiuxcxx3HRH4mc39w/N+MFx8Vknd8vIleSDPDvvIg9t8IymZ2S2nqzfbew7Q9g5PhJMWnmZ51Q/gXgyaX/n8QiLa+ExO2lzLmbx3I7EaHuf7CI9AQjhu8AnheRHxCRTzih3pPm7QvaqqoR67eLS9e+2BhfBv7QTW16K7aYcwHYTbZ2ePJWhSzhVud398e7gC8WkTFGmH5SVZ+/TVk39+Ut7xNeZFxe4rw9aV7eCq+5+0JEzqVn1bPJxnct2XgJeDIthtwKN9d5u3vs24F3A98pJt39e4kAvpz5fiJe5Pl7GfjepTnwMPZ+OemdcEtbX65NPXr0+M2Nnlj26HFn4SlMprSM+0iOi6oequpXq+r9wO8B/lLKqXkai/5tLP2squrbl8rR5UJV9RFV/aPAWeD/BL4nObk342ngG28qe6Sq35HK+feq+lbMkdFU1q+q7xXAvwf+M3BJVdcxSXCX+3Wrup4Gvuomu4eq+h5MMnepO1FEZPn/E3BzHf838CHgdaq6hjnvJ+Zpvkw8j0nYOrzANlV9v6p+CTZ2/wn47hPKeQ4blw73YFK+q7c+/ba4uf0vZc699MItsvkFGIn7EBZlu9V5J83bF7R1aUyfvU0bbtWmb7+pTWNV/SZsTDZvukfueZFm3er851I7nsWktb8f+BOYY/9K4MXG5ZWet6/F++LvYHa/Mdn4x5dsfBq45zZR/Zvbe+I9pqYq+XpV/SQscv5FmHz5Jc93TD49Wvr//AuMOfn5+zQmtV6eB4M0735VW25na48ePXq8VPTEskePOwvfBfwNEbk7bb7w+cAXA98Di010HkxO8z62Ah2xnKTDtJHDUES8iPwWEfmMkyoSkT8uImdSZGcvfRxvceq/At6RIoYiImOxjXRWReT1IvI/ikiJ5dPNlsq4ikkdX6nnzCqwo6pzEflM4I8tHbue6l3+zsV/DnydiLwhtXddRP5QOvYDwBvENkzJgD/PTQ7bS7TnADhK0Yb/6abjV2+y5+Xgu4E/LSKfKCIj4G92B0SkEJEvE5F1NcnnAbceNzC58F8U23RmBXOov+s20Zjb4eb2vOw5dxJSBOlLEgmrgCNOaNNt5u13A79bbPOoHPjqVNZ7Xkabuiji21J7BmKbpdytqk9istivT2PwVuzefDF0538O5qz/h6Vj34blgr4Ry817JfBi4/Ji8/bjxWvhvljF5ti+iFwE/vLSsZ/BCOw3pWfdQER+221sOvEeE5HPE5E3im3SdIApT+LLme9YXvDbRWRLRM5jEcquzbd7/v5z4BslbVgmImdE5EtOasRJtt6m3T169Ojxq9ATyx497ix8A+YI/zdgF/h7wJep6i+n468DfgRzRH4a+Geq+mNJXvhFWC7S49hGH/8PsH6bur4Q+KDY7qD/EPgjeovcM1X9WWyjlX+SbHoU28QCLL/nm1J9V7BIwdelY50DfUNETsoBfDn4n4FvEJFDbGOSRTQi5Vt9I/BTSfr1FlX9Xmz1/jvF5G6/DPyudP428IeS7Tewfv2pl2nP12Dk9hAj39910/F3Av822fOHeRlQ1f8C/CMsv+1RbBMOMCcULML1RGrXO7A8r1vhX2ORsP+KzYs5tvHIx4L/F8vl2hOR//QxzrmT4IC/hEV/drB8t5MIzy3nrap+GIs8/eNkyxcDX6yq9W3q/bvYQs6eiHyNqj6Nbfr017DFiqcx0tG9K/8YtonVDvC3MGJ4O1zB7pnngH+HbULzoaXj30uSLKY5/HHjJYzLi83bjxevhfvi64FPwxbnfoAlUp/674ux3O6ngGcw2epJuN09dh5bFDzApKg/kc59OfP927HNhJ4AfogX9uftnr//EFN4/FB6Zr4Xm7sn4SRbe/To0eMlQyy1qEePHj163KkQ+9qMXwbKjzHa2OMOhdhXvXyVqv7Iq23Law39fdGjR48edxb6iGWPHj163IEQkd8n9n2Tm1jk9ft65/k3FkTkD2B5bj/6atvyWkF/X/To0aPHnYs7iliKyBeKfcHvo3K822WPHj16/GbEV2HfH/pRLJf2lc6F6/EqQkR+HNvo5n9J+aI9Xhr6+6JHjx497lDcMVLYlDD+EexLi58B3g/8UVX9lVfVsB49evTo0aNHjx49evTocVvcSRHLzwQeVfuS4Rr78uwTdzDr0aNHjx49evTo0aNHjx53Bk76nqZXAxd54Zf1PsPtdzCj3BhodnYTFFbzilYdQQUvSqvGmTOJzIM1UwRUIUTHMG9ooyOqEKJDRAnBMSpq2ugpfUsVMwSlcIF5yGiqHHxEBDIfaIOnyFqa4CmygCooQtN6Mh9QFRBQFUZZzaQpWC/m7NUDnCiqgndxYZcTNZuiQ6OQZQFJ5zmnC9tjtP87qAreR9rKk5et1Qu0wUEQXG4qK+cUL5GqyRABkeMyYhQyb+cFFbT2lMOaqEIbrD2Zi8ybHG2FvGxp6owsD4Rphh+3tLW3wgQQtT8CoILkkcwHmiqnGDS0wREbh2QKomhwOB9RQJfaJ6LE6Oza1i/6qsgCdZ0xGDTMZzl46L6SK/ORdpZBEY/HPfVnW3tbTklNF69oFIiQFYEQHapms3WGgDcbiQKtkI8amiZD0pjkWaCpMmt3FMgUGoE8mi3B5mKRBeomg9bmBV5BQWpBM+szl0eYeHQUcS4SqgxXBGLjzCYFCcAwoo0Dp8lWfUFfSi1ooUhj7dAiInNn9ThNYwRE6wMqZ8Vkau1aKltS9pI6u0YCaGb1ae0QBfVme6w8eH1Bm4ipL33qdBUki2gQO28Qkdqh2fG5rgjo3L/gs25eSWNtI1p/vKAfCz3+ZjZJdYWl9hfWdj8VYpbaEcTanisEWbTPVVZvLNJcbJb6NFWNt7kkrdWnAhJB89R3SeAoDeBSH6qdg0IswKU9S2MBLvV1zNMcdWmMQrqlgJjxwm/v9NYXEpZsSOerLI2bWlmaK6629ks029WDlgqt2HUOsim0K6m/0xLkwr705pDA4n7yFYTCrhWsD/3s+Nrojx8LnR1+JouyNLO+UJds92a/a9NnwfrFNXaMVJak/gml2RwG6ZbwNoaL8sRslZDmsDc7YnHcX4vPsd/SWv1x6WviO1uskcmGZCPO/o6lHXPNcR917ddUdleuq5falaU2BzvfhTSe3ZzWpXmhx7Z3xySaHerSHEs2dmO/sCMe19G1WZ19tlyOBNJ9ksY+pls6zU8Jdm47BN8c2xLz42Pqjvu6s7vrE+tfpR2KjbOkfu/q1+Myu3tnMZfTPO3GiphuC11q/2LQjsvs0LV/cUpmY4Aej4fVoyBi4xHT3I9Wp68U9en5kfrahe7eUGIux/YkG2KW7pdGiZk9I6K3foi5LPrN5pUSSjunuy/VpbFQq9u1SvSS+kpTfwuuOa7f15GYOyQq6uyaUNixrl5X6+K58oJ7RuS43EzskdOd293PgDr7Q6IunkHqBVFdlGMPTU19n8rNBNeYfa7V9MwSXBtT38qiL2zOaDo3LuokXSNtelar2nUdbioTQJqIeuuTzma8W4y3Lu6d7qY9nr84gdi9z+z8RX90/3e/nSBNsDLd8eeLOpOBEs0eFipCWWqL9etyf3c2Lp4NMYJzx3937e/K6/pfl8au+7y7SZYRu0m4PNAKqi/494VG6dIvveWhm9GpJuXkU2510Us981fhkN1tVT3zMRfQ40VxJxHLlwQR+UrgKwH81gbnv+YvgVOkDMnRF9zVkrDZQOugiGw+m9OupAdjGckOPO2KPf391CGtmIMZhVgocRSQyrE+ccSBorkyziL5jcyc5fRS6Rw010A7VvxcaNYC+Z5HHcRSUW+OKwKr6SU5GCqD5z0IVFuRYtfRrEdcI0hympqtQLbnaTcC/tCj5+f4ZwaEUikOHPWand+9/F0APxOaFV28dIt9IQwgDOxB0mwEBlcyqtMBP3H2ok2Om6uF+qEZulMwes4TSnu5hIGa4+ug3BEmFyPDa47ZebM75kp+JMzPRPzc7HFNevnVQhgq42eEagvaoSanHfxcCKXZlR/YeZ0D3DkF6iHft6fS7K6Aa4TsyOHn1t/ZRGhWlcENoRl3zpHadYfW582qnUdyVIsDaEdWR7kD9bo5gX5uTnGzau3xlTmqg22oNjubj53bzrlXb/PGVXZOzCA/hPxImZ+x8Sh3zVmoNhXXCMOrVm82h3YAxaHZ1DmSANMLkfKGwzVWpoRunOx3s8qC8BUHSrUlC4dFoo1BKCE/Mkc7Zub4uSo5vZgT3owhDGFwDdoVmw+xsOP5odVXHCqTC2LtT3MrlPbju03+08uj2Ddb8kNzWGanZeHEZlNrL87qzSeQHyrzLSGbHTu2fq7sP6SMn3HE7Hg+SSIBfpZs7N6XMTm3IxheVZo1mw/Z1Potmyn1ulDsK/WGEHIY7CiuhWpTyI5s7NqhtUcCtGP7e7CtTC5a231lTp6rrb3Z1NqqYnWpTzZcs/qyqRILwVdKcaA043Tu3BzCZmz2uyZ95oRQ2FjFXBb3QjbVhbPtK6jXxJzauTlPYWhzc7ATmZ8yxyJmUO4qsYBQmAOXT63P5lvHhNfXdk+GIg2gS86pCMWRMj9lc9sFtbmYmx31qs3pck9pVoXBjYhrlWbkaFbMaYq59Xm3GNEOj59VKMxPC+WOEgZWTrUuZHMWZLgdC36uZHM4uiiMn9PFc8k1Vk47sD5xrZWfT5VmJIu51I3VgtTq0u9oc9glchO9UB5G5utuMdektT6KuTnjKMRMyKeRduiQoAsH3Mozh74Z25zOp9bR6m3edURp2cbBXmC+7slnSrVmz6p8psfESaA4jDQjc7yzmc0rdTZuzcieVwi4xvo7mynt4LiObJbsjRByO7fcj8TM7G2HsmhPPlMjRZlQ7LfMzuSEUiiOIiEXsrlSrzjymTn2vopUa55sHq1fBo52IBSTY2bXDN2iHzsyo17IppGY5p6fRZpVTyjsfgjp3lkmyOpS/0/igiTF3Pogq2wuu0bTs1jIJolILDm3EnSh1QpFIjMOmrGjOIwLZ9jVEV9bPSF3hKEnlEI2i2TTQLWRM9iuaccZMTM7smlAMyE/aGnHGerAzyNh6JAAfh4IpXuB028kzyXCbuPROc7ZPBAzRyjcghz6KixIUywc0qaF46C4KhBLT/RCftRQrxe4VvGzFs0d0irt0FPs14RRRkzEKhaO4vqMOMhwbSQMMiORTvDT9KJxQhh4G7fDZkHeYunJ9+Y0GwO7H6Li6kDMHbH0ZEcNAO3YVmfUCX7eIm1EFMIgIzuY064NcFULToiZszIKnwi14BrrCz9vibnH1S3q7TzrCzvXTWviMDfi5QU3b83OUY4/nKNFZsSubtHcI7XZopkD55AmEIsMsmOSJlUAL3aeN7LVkUVU7doIUjeQJ7e6adFBjlQtUtXgnBFHOB5/EaRpjRSHiObZghRK06Y2OCREaMMx8fUe6gYdFHZMBKYzGA2TAzGDLDNbZnP7rCysjBDs+pjuz2UyugSt04qn8xADkudGAqv00s8yqJv0IvFo2yLeoSEel7dMABPx1WZpn62XmVquHZmP4fYn3gY/ot/z5Md8cY+XhDuJWD4LXFr6/+702Qugqv8S+JcAKw+d1wv3bvP8tQ0euvsqjzx3lvHKnKNpRj5uCK3jM+5/kveFB3DDFvYK3LjB3fD49QaeGxC2GsQr+bDBPbyCf/CAEBxF3nJxfZ+HH7lItpMRVkDvnREax2i1YnJtDHkkBoFMWdmcMtkfIkcZ5998hd0fuYvpm+ZWZyXIvRMuntrn+d012idXaMeJ6G42VFuK1p7xmSNmv7JBGCn5rqcdR/yhJ44i/vkBw0/c4+jxdeYXGoiCNoK0QrnjaD/liObxMe1GwE0dcaNFs5yYKXEUkVagiFRnAkRoz9YMHysJpdKsKr6C7IkB9dmWMPD4GUwvBaQVwticlZg54kqgqoU4iFSnNJHrDD1b4R4dUt1X4a8VhEFk/SOeo7thfgaalYivbFU1XJ7R7BdGcu+rCMOcwTXH9IEad5jhKqFdixRnprQfWUEzjJwr1FsBBPzEHobNqZYw9JQ7jmacXrAXZxQ/Nebw3kgszZkZXHdMLra0K552NZDvemb3NQyfKKgfmCFXBoT1lnw7IwyUelPJD4R6zcirRCNo7SiSHzpCAcWeEEpo1wLDZz3tSGk2A/XlQPlEScyVdqtleC2nHZhT7ufmVHWLAvWmUu4Z0Z5dUvKJEAZKHERCIVRbirSJ2MyFel0ZXDciH1YCMvdMnOInDl+DRCF4W+RoV5VyT4gBqnNmt08OdjuEyRvmFE8MEIXpXfbAHmynOoPVNz+jFB9OUYuR2bv1y3D9zUqx5xar6PPzgfKaOR3VlvVdKIV2bES9WVGOHmjJ9jL83IhNM8acXw/u0Mg2FUzPgausze2W2etqmJ5X8kMhb2Fyt1LsCeWefe6CUOzD4f0p0tBAdcrONzJmjnIXEVInzM7YwklH6Jr1SLFnCxfN2IgGp22cmlXr23YERWuLDa5NJDA7JjExT/NkCPNTKYJ2Rth4xEgRCtN1RzuGtccju58g3PWelmbs2L/fMbymiOsWSQCB2VmhOLB6ZqfB13bcVbIgsypweNktoh8IuNbIrTojo/nU7uN2BCvPGGGeXoDTP6/Mt6AZ23wuDsUiNrtGTgZTu4easbU5mypH9wVWHvc0Y6FeBRVHNlXCwMhgR+5dYyQ+lMLsjDC6oszOCMWBkh8aAY+5LY40K/b35KKy/lFbnAmltS2bLUWcxPq5HXcLWUo7NKe8LWHvE5TNh4V2KORBj+dYYyR4eF1xQZmdOp6/rrG+2vkEz/C6kh8os7Ejr404tgNQ78hmSnGkTM95iGmRZ0VsoSIRyjAwQt2sgK+F6TlHuWv/L4hVk54pGTQjsyMUVpZrjaD62uqabwq+drYw4YVyX6lXrE0xE+ZnYPUJtSgSFgnav98x2Dbb6xFUa55yXwkFzE458iMlK82WZsXhGjumXqjWHaPtSLUuxDxPpFPIJ1BtCNlVpR3BcSaNs3kfhWacoQKD/YifK/WqwwVrV6iNnEIXkVOqDXtuNSPBl0JxaCRVopHytnQ0I3teZHO1hbuZzbN6xebfYMcIbjNKhD+RPF8rvo60I28LAnkXPReKw0DM7Rk1O+UY7BppVgftwFGvCcWhI59Eit0aSouGZVGpVz3NyBEzoTqV4+dGzGMugE/3pkuRRcXPW2Znh+RHER15QumoVxwrz9X2TBKIpaSouxH16ZnMFn0UQumIuZAfNsTcUW3k5JNjxzoMrU4JkAPt0JMfNMTSU69nlDsNEpV6lOGa2DlShNyhXij2G0Q9cZQT8yQ3cPaMzCYN0kbm54b4uZGqduSQNiPfnVOdHzG4MqVdL1Ev1OsZEqHcroilJxSO6tKI/CggqS8QIQwzXCLKMRf8PLOIXCKVftbQbAzIDmvCKCdmsohWhmGOhEgsM9ysNZtWywXBjsUQf1ShRYZUgTAucLUR/nZziJ8YGdLcG7mtWsK6ETLNHG4maOmRKiCJ5OkwhzaiQ4+b1miREcscUUWmFVrmuKpCiyVpw6gkDjKkyHAhoIMSHea4/akRwUTwdGCET51bRDJpGnQ0MCIZoh0rciOheYZM55BnSNUcR0bzHJrWCFyWQZFbHVlmv0OEzB9HU52DGNG2k6HocSRW4+JaWRmhh0fH5/lOmebs70QOJctAI5JIszoHTZPsao6vW4a4Y3K5/PcJkBShfq1vdfa2zxvrjZ2PnRyfhA/8YvVuVf3CV7zgl4k7afOeDNu853dghPL9wB9T1Q+edM2FN2zol77rbTx8cJ7MRc6UR3zK6tN8/5U3cvd4j/1mwH3jG7zv+r1c3VvlrZcfYzWfk0vgPdfu46GN60zagg9tn+ULLn2Yg3bIB67ezaeefZZJKHj4+jkub+6SSeC3n3qE9+7dz889dYnPue+jAGQukEvkfVcv8yfvex8/duMh7hvf4Eeefj2XNvaoQsZbT3+U/7b9AKVvOapLnnzsLOcv32BzMOPhRy7y4ANXODWYEFX40PZZMhe5e32fe0a7OIl8eP8cuQ+cGRzx3qfv5S2XnuAnP/ogMQh//s1/EzX+AAAgAElEQVQ/yg9d+yQ+ce0K77t+L2XW8vjDd/G1v+P7+c9XP5kqZKwXMx65cYZh0XB+5ZAP/ty9fOqnP8pHbpxhPitAlPGwpo2OzdEMJ8oz1ze57/w2z+2tsTaa40Wp2oyD/36K+mLDxqkjRmXNle11Puv+x/mZp+7h7MYRa+Wchx+/wP33XCN3gSpkPPHEWcgio7U5o7JBRLn+5CYX79/mDVvP88PvfROb9+0ynZdcPrXDM3sbTJ9b4a1vfpiDesiVySp16zk4GhIqz7lz++wcjDi7ccT+bEBdZ8Tg+NwHHuGJoy2uHKySuch0XrC1NqENnqNZybCs2b22CsDGmSPa6Di6ssLW3XvsPLuBX6txLhKDZzCsqT+8xvobb7D91AZbl/Y4mg7IssB8VuCzQFEEQnBsrEzZnwwJwRFaT5hmIEq+UuO9MhpUjIuGp588jRu1XDyzx7PXNsiKQEzyX4DPve8RPrx3jiefPQWVx6/VrK3M2D8Y4X2kmeRsnTtg76NbDO855MFT2/zCI5cYbsyZ3RgiZeTyxW12JiMOD4ZcOLvHwbykqnLaxhNnGYOnc3jjISEI3ivVc2PkVEWsPS4P6E4Jopx+YIftx7Yozk1pm4xwlNlLvXK4SnjLZ32In/rQg7g8IF75hAtXeWz7FLOdIW7YMl6dc7gzhsqclMGpGSJK+WNr7H1qzXhrRpk37FxZJ9/OiPfMCQc5MmrJypZ2nrOxdcTh0ZDQODjKIYCOA5cvbfPUc6fQxrFyesLkcIAeZVBGk4NOzSnVUzUc5rDaINsFcRyQYUBbhy8DedEyvzE81t4oZBs17UFhiob1lvGZKc2vrNHcXeOvFLSbLf7IE081uOsFen5OnGemltgu8JXQnGpNTtoKeJBaiBsNcpShZUQKkzDL1MNmjX++JJyrcds5cahoGTh/cZerj5w22W3lOPfQda5eX0dnnmI7o76rgdohgwCpXAB/4E22qyTJryKNKTLiMOAPMtQrca1F8kj+TGmLNWda1s4dMXt4g5grYbO1ftys4VqJhER2Bhb9ry805Fdzmk1TUywUD5uBfN8RRrZw0g7VFCJ7nljataFTBwwgbDW4Q1toIHbRWVMvtOOIFoqbOVwjFLvC7EIgO3SUe8LsjBJWTVmSHwquTfV5ZXDd0awp0RsB9zWEUhlsOyavr5BJRr7rCEMlbLUUz+fUZ1vyGxnFri3aZBNHfSrY+I0i+V5yiqIpUfJDt5A3x4yFwkEi1OsRCbJQTSzkms6is8WB0I6UdjVSblu50dui0vSC9eFgW2jWLGIahseS4mZdGV6xRZdYWJ31hvXt7HJD+XyGnwk+LXK0IyWbGmlpVk1hIwGqLWV43VQOroZ6XReLEcMryuy8LAh9taUUu+bQGUm0hatYQDtQin1J0kwWUe12nGTN3sYbsQh6vS60Qztvcm/L8LlsoRIwYgvlblJ0TJIKICkJXGNqi4WkWEz10A5Z1N/V1c0z6wMWigj19nc7TPf8vJOhGtk3pQGLCKlEXZxvfWgLVdlUqTaE4bYSchsL9Z3U0OSLzYr1b3FgiyQ2F5W2TNHGnIVkNQzs2jCE+Sll/VFTU+QHiq+hPIhMznuyqSkHwsDmImKR/5VnIm0qo1OclPuaFg5szLuFBxWLJlerthDlG1t88ZUdD4UR9GZkUWYXoFqzSLVJeDvJu51XrTlG1wLVhrPFuC4angvVqrD54RnzsyUSlHbgOLjXsfZkJJsnQpL4Sr3mbPHAm00hl2SLLhQV6mCwGxaLBBIhmwTCwBFKRyiE8bNzmvWcmAsH92RsfqQhpD7PjwLNil/Iq03JYQa0A5/Gwt5boXD4ecQFRVq1CK0TJudzhjcCvgpGjgtHs5LhZ4FsFiyCmaLD1VbB6MkDwri0+2vWQuZMyltZxBRYkFgtMpMst9Git6rEQW7EWMTKDZHZhTHjj+7Rbo5w04Y4yMj2Z7TrQ1Ms7M3MDu8X5JgYjVx2RNQ7pGqIK4M0DolEJ2Lo9g4t4tlFGLP0DOzKgOOy2tYilW1rRLZtjf2Js0hmF6H13iKhy7zjZsnuSViKqMb5/Pbn3gY/ot/zAVX99I+5gFcAn/7JA/2Zd9/zipfr73rkVW8b3EHEEkBE3g58C5a59K9V9Rtvd/7ggYv6wDd/BVWV4Zxyem3ClZ01iqKlmuc4p2yuT9g7GNFMcoYbc+aTwnLqanec84WRjVmVU+0M8asNcafAbdUWkdwtyM5NCa0nL1ra1hPmma1Yl/aE0t3CnLkyQOPw45ZwlOEPMnOg9jP0dI1sF2huUai40SDTDFmrYbskDrqVG+ylt17THuXIzCQesYxIygfUYUAqb/l6LmnZu1yqOr1gxpag42YpkjFQ3FwW0koJKYJYKpryMEWtfLdVo9dLpDEnpVmN+JlL0lhZyJNCqYuyYhlxlXlRvjKH1R95y2FzatLglZbiuYKYQRhHsiOT04ZRJDu0lWVfpRd1csiyqclmw0AXsmU/E9qVSL7v0KVFQnv5H/eTa4RsJjQrkTiwfsoOzalEIJtY7oxm5iy2Q5NJ5YfmqMYckJSTVijZxKFeF5LoRR6PkiKrqV+K1AdJdgxQ7DrqdSWMA8WuSb66vCKX8pzyA5NatiOoTkWyqdDlJFlOj0UBu5y+/FCYn44U+9YmPzc5dRhqkpwdy5NjzsLBh+RoViwk4NlUiLmRguM5YnaUuyb1AZMh54cW/QuFSYF9teRgJzVOZ2MYpPEamWx5fsrstMiTRR7rTSU7EnNck4x1IWNOMsOun0KSwbrWnOpyV6hXrUzfsMjTWuRjpXbU60px0GnizN5FW5JkUN1x3lU2Z5HX5WtzLjtJcRcV7HIlO+n28TiZg1juCM3K8bzopJi+snYOti36Wu6YDLReZSFl9lWyJ+/KS8faJL3tVFWJlHUOtq/s784eX3FMoLtEltTOMDRpsWtSfllXZjiOPtbrQn5kn7UjWfRtlqS1Me8idyafJvW/RcGSQ5h1Echj6WVxYBHEbJqkoykymk11QdgsJ80IaZev6IJFk00umc4prNwuZ0mzFDUc2/FmxciY5dFZO8pd+50fWiQMZxEjsyPNmTVJUltLCYjpvo55N1cshywUQjY7bp+k/mlWUr8KxzmOSd7dzZ12nGwo0hxvjse8WbFyu7EOhZBPdCFJ7p4f3Vzs+qwjMJ1MXJ09U1wDxZHZ241PO0qEDDuvPDDZbEfMs9lxekUol6S62Bwy2atSrZocdSHHT4SkPDgeJ9emZ2Od7r+Btacjfd190tndkciO7HQ5dfasOZYoh06q3EV8sHkj4fgd72uT30afJKcpT7H73DUm8fW1kk8i802/SE0IuZGletUtZNDFoR2sVzxZFZNcU5PaIKVfpHEzKbmpJpZh94IzkpXmsaVbCL6OhNKRTaPNqy5/MLBQX9icl6SCMSlxl7fqmrjIwYu5S/VFu4+H3ghTkq6GIkXLOklokujaYkG0aN/I42cxzfdUXmtETAJkRw1h4Be5nqE0abVvoklb20j0jmza0I5z/Kwllh4/a2lHuUVTVcHL8b3iHX7WIqrE3C+iln5/TrsxMDl07ixf0okRsNybrFTs3uxyIf2kJowLFjmPQCxNFuvqYNE7b2WR5K4LKa0XpLKyu7x3YCEHliZA5mwsppVFFVVNemobMSz6VZLcF+eO5a2dfDSEY2mqd0bqnLNI5bKMtos4LhE8aVojbi7JaLvjXR84sahhRxSX8zBTGS8oP0Y0kTkRWeRCEjpW7o4lqd1nsOhvnBjZXJbYLp+3qHZZLvsifCSFKhfR048BdwKxfPMnD/R97777FS83v+ujr3rb4M6SwqKqPwj84Es9f5A3vPHs80zbgjp6dmYjRqOKw50xFy/soMD1vRUAts4dMJmVlKMG7yProxlH8xInynReMJ0XZFmAPBIqDystGkFnnuzMnFPrEzYHM65PVpg3GaFoaVvH+sqc/cMhTaGMzkxo6oyQOYqyYTax7vWDQL5WkecthzPP6PSUpvG46AhOcV5pV1uk8qzfvc/B4ZCibKmujdi8tMf+/gjxynjQMJsUxMqT3cjtBXN5TmwcOvXIKG14Mo4WNSmjyVLP1qgoG5sT9p7asPzKyxOao7S0GYW1s0e0wdG2Ho2CPj0ibJnkNg6s3LhZw15BON3gy0CoPMWzBeM37XBwNITGEYJDJh4u1Dx4fptHnzhHfjWneP0h090hLos0qxF3qkaPMkIpxKGt8IdBRFdbWhX8jZx4rkIbRywy4kpLNmpp9wqTewroKOB2PM3IolVho7XNV2pBNxo0CBxmVAOFDYtg+amjXY9IJfjzM+pZbtfMHfPVhmy1IUahulaixTHhDgIkkhZHcfHic+OGOMnt/yKSX8sXG+TE3Gzxc6G+UBNmBbFQZNzSzh1xrSXOjHhLys+NmQMV4rkK/1xJO44WOdouLdpUCPFsjbtWWB5RrrhWmJ9vIVOaIAv5dLdxkhYRP3UMrzgmWyaxdbWQTYVm1doiAebnUt7xEnn2M7O/ev0M3S2MWCey064oMVNwSu2MfIfWHKp2lKIsKQ+5WTFJbLWlxELxlZHZ+rRJr2OpKd/pmPChQjuOJiXHSGyx76jOBPI9RxBoVyMSHOWuRbGYSMpRTFGV0pyjmEE2F6YXLBLmJ45saiRpftqiAp2zkE3NaWtHVme7orYosBUZPeeoNpVQWhvMiUttHlveprRCvWaENQyOoynNuvVJyJV6zRYFYgntmk0m3bcoULNiZLtZMVJe7liUJxbH+drTu9JCQMr91UR0q3WlvGFkQQLEobHrZcfdNTA/Y+MBx+S8za0vBokMu9qI0+ysSbEkmCy83DVSWK2bPC3kifAnuXeXCxoKecFnobB+ldaikqE06Wv09lszI1jqYXqXLPJ3Q8kix1jFIlrNmkWPwNptiw4WbYp5twggizxYX1kUrKt/cMPqqVfTAtCqkbByT6nXzJ6OcLcjiyzlh+bIz84aaW0HNhaI2T25Wyj2oM2PJcrZRJneZTJty50zu6qNRMxJY7fabZAiaUMeI90kUpwfGQk2Uswi11fUCK/Jzq2f80Nd5FzGlLNb7ivzkcPPoBnaefNNi8zEJA/3iWhWzuSnlmd43L9dnm8oZJFjHk2gwXToFoS1W4yJeYp4rspiQShmkkjv8ZxtB7bAGT3oIJFOD+0w5Viqkcj5hkW0mrHdb36xmNDlWCaSe2i/bcHD8jHbocPXgXpsOZXN0Bah8qnJdWNmZQz2ItW65Yg2Y1tU7chhtWHEocvF7CTMEpW2dLhgxDQUtujREdCYQYyCDk3eu8hn7uaut/zWas0xvBFQD9W6Q6KjOIomVU75y2AE2TWKYvLb+bpndL2lWbHcSnGJmAqEgT2vQymUu4F25NMilJHZLr/aBSU/tMgfQChsdS2bWX5oM3IUByFdmy3Is6sVPw/4KtKsWZQwmwRLT9mtqE6VxMKRH7Y0q9ki0qiZQ8Vks/VGiauj5aQmGa6ruvw/I3cx96m+SLOaGzFOG/1EL2StWm5p6dOCol8sKmQTI7JxmOGnLSH3tgePswhg9M7+9kJ2VBOHKQczWK5pl+8pPiaCm6UcZJMS+3lLLAvLw3QQ1oeW9xiUsDbATyp7twRFQkCHxTHhk8xIIFhEMJFHdc4ijd5b5LHIj2WqIRzncYa4kMcuy3A1N2lxtzkRnQq1u1YVohihvRVRFUFjRJzlnWrdIF3EsiOZIYDzlleZ8jDtuC4IpGTZcYRSErG9iVxKkshqCEiXY3oinNX9cRDLHr/2cC9+yp2LqMLjB1v8wsOXeXD1Olef2+DPve49XLp4g3Fe89xTp/i0S8+wvjrjcDIgBsc9W7v8lnPP87svfJDpRzZsR8+rQ+ppzsZ4xtqpCfmw4U9+2nuJRzm//9M/wGBYM6kKogp7v3yKo2tjVoYVze6AcVGzsTblwr3blHnL5z3wEeRayWx7xOkL+1x40xUQpW08R8+ska9XnF074sFz2wyGNZ/9usc4t3XAb/ukR/l9b3k/h5MBn3X/48QoDM5N+IzzT3Hm1CGx9sQojFfn9vAaKG/5vA+CKKfPHHDv664ieznjsxPytYo3v/ExVn6pZPTQHuvvL9nYnAAg6zXtqYby51bIRi2SKflaxWRacnlrlywLXD67YxGsUcsbHnqGlbUZZy7uEWsPUcifL9BrA1wWed1bn2D3+TV+5+s+hNaeUz+dk5+dEYMwzBrcQcaFn2qJ0XHv5eusvn/I6O4jRuM5btziamF8doKWwaRvWSS7njN44IDVtZmRua0amXraaYastLbR0pkKokUV83smPPjmp9g8d8DgSoZs1WjjyEdN2jwJ8kG6LgCrjUUyPzgmHzas/UqOq4Xz9+zgfOSz7n/coqyDAFG4cPkGUgbwSVY2bClPzyivZqytzbhw77ZNyMbISn4oRpTP1eQHjmYjsvGBkvquhmJf0NZx9mcBAT9zxEHk0qc/y+iZDFcLxZ6QPVeSHwlaRj7/9R/Cn59x+gOOOFTctYKw0XLXTyphLaAX5/i1huHmDL9q+SNuLgwvHiFbFQwCKBw90OK2atz5OWE1svqEEZWw0XL250y2OHjeclCH1xxbD+7QjiPtaoTtkmziGD0ryKUpg21oT6W6KkFHgThUVh8zsqcjy4Ud3n8AmPxvcEPTbotCs2ZOzen3ZYSN1ghoC81m4KFPf5KwEhk/Y6v3xYGRWy2UeiPiZo7RG3epN9OqcwaHD7QMdiwqql5ptsxBq883ZFOh2E+7256pkPWa8dNCfgDdLo7FXorADpXBDWXlaYuQn/4lJZaR6oE52WHaGbK2vOZuI6rqTGB4XRleE7Y+aPWrN+LtGiN+03taQqFUpwKuFtqtFsSIzeqjns2Hsdy/2jbJ0rTBiq+E6SfPmN0VKXct/FBt2ph0u9rOLgZmF1tGzynNRmByb6B93TSpHGDtiUizpuSHtlFUvZ42FWuNODUbMeUQGvkmQvGmPZpVZX7WosbVphGawbaRplAI89O6iJaFwhYTzvx8a6T6SAlDjsdjxXK5pbX6Yw5rv/0qYaQcvb6hWVPqdcXP4fA+2+irHSl85j7jZ5W7frpaEJVsZuQ7/+wdkwQmtEMgwuCGUuzB7Jwe554dKvkRzC62tunPUKg2heZNEyNvq9b+0fUWiSb/rDaNSLXjLlqqrD3ZLHZV7TaM6iKQ+QGL3WnDAGZnbCEom1i0NmbC6jNhYXOzBnXaGGx2xhzddmh9Odixvm1XMOe9VrKJRY9da1FEzRKhHFmUrv3Mw7SoYxGs6XlhvmUEae3RCc0YqlMwuh6Yn5ZFn3Wqi8N7rM/mp9ICQDQbbaxNllgcKu3Y8n6rTWG4o+QTpd6AZmR1o1CvC6PtaJt+JRWBa1PUNCkZ6jUjx0XaiKc8jJarmTYX80ki2skuZ2fEiFOEvU+yKOjwRmC+ZW5MteYWixE7b5st6sinkejh6C5PvSrks2gqnDHME4kMJQx34iLa2oyF6V1GXLO5svJsQ/Q294/uduRHLZPznuIg0IytjLYU6lXr+8mFTv9sJHaw05o0dN2c6OF2w3zTUey31Ct2rQSYb3pmW55sZhHD/KBlesbhZ3ERZZ2c8+x8Yo6rlWK/pdoS6lXPfMOnPGPh8JKn2vDMNy0C24yEYq9eRNSH1xuaoWN62jPYsef4/FRukcik9Og2NJqeyWiHjvLajGYtY3rGoqLNSMgntsdBvZFTrXvqVUezllFtevxhRTO2fM96PSeUjvmmT7sIO+an8zRG3qKh84BrokXShz7tWK2EUYaKRUzr9Rxf2cZArom0A0+5PaMdZ0llYRHObNLYBmmrmV3vLP83ZiYl7fIxCUp59ShtgJPIpneLjYvqjcwIb+Zwh7a5Ucw9YWDErl3J0cwRBhkSI25aH+/A3EbCyJRtWloOK41tJKTOIVVrn6+U4G3RIg5zI5KlR50jrg2Jq4MF4dM8QwclcVDc5AxHpD0mbBLSDrPdJj0xEtbHaFnYhjsxouPhC8voyO7Nn4kgmUdXx+jKKEUhdZFTqfO5kd6UNyqjYapX0bq26GKKjMpynmWKzGrT2oY+Ue3vYHmfi9/dT/d/t6nQaxpK0PiK/9wpuKOksC8Xd71hU//0d3weXiIfPjrHWzYe47HZGS6Wezw2O82pfMLZ4oCojndf/STevPUUj07OcLqccNAMFl9DctSU/NZTT7DfDpmFnMNmwLQtuDDaZ68ecnW2SulbDqoB47zm3OiAnWrMm9af5SNHZ7kyWeOe1V0A5iHjnvEuk7bkRjUic5HdasRhVXLP2i7TtuDxG1ucXTvi1GDC85M1Htq4zgdvnGe1rDiYD3hgc5sb8zH3r27zc9cuUWQthQ/sTEZsjadkLnJYlZwfH/Lk3iZb4ym5C9yYjokKn3b2WZ6ebLBWzFnNKt5/5RJelMPJgM21KZkP7ByMGQ8rZlXBxsqUSVUwnQwI04wLl25wajjlo9unaOqMtspY3bDr5nXO3Rv7PPr8GbY2Jlxc3efD184yPyxZ3ZrYrryihOioq4yH7rrGXcMDfuLxBwDI84AIrAwqtoZTogo7sxGFD0zrfJFTOBzWTKYlg0Fj+XjPbODXGsbjOZc3d/mlR+7m9PkDiqxlVueoClVjkmgR5WhvyNrmdBGFzbLAqGwWX/0RVZhOzYNpJzmrZ46YTUt4eki72dJ9JcfmmUPKvOXKc5v4QWAwrJkelQzHFdPrY4rNOW3tyQr7api2yVhfm7Ba1jz59GlOnzuwHNHrK/hRi88Ca+M50yonBEe1P2C4OWNtNGdnf0y7MyA7NaPZHTA8M6Wa58iVknimZmNzQhO8fc1K2XCwOwLg3ru3eeKxs+RrNc1hgQyCfV1I7XArDXK1JGy0lGsVqkK9XyKDwGBUM9segVc2zh5yOLGvwWkmOa4MxMMct9bYV520DoKQrZk83OeBZm8ARWTrzAG7N1bRIJSrFdXegHzVvqomtuaoZGWLPjFGc7XIcvo6k8HmnPDoCs2Zxr4aZ9wyGNZUT6wS1luTrDuM6Hul2Jyb/ZVFVWUUcDdywigaga68ScG7r4zIU+5l5SgvTGgfXzGn6LTlQgJItEWEbCcnFopmyuD8hNm+MQC/l1nO4NjKQhTZrBeRaj9zi7zLOIpoZptuhQ3LVdRxIBu2hOsDdGgLFH43g7sqso8OqC42Nt8qb3mTkwypzIHOjxzz861F4UcBguAPMotGpl2n8wOLPNVnW8rnM5o1Tc6MWhu9SeCBF+zI3O1YnU2EZt3yCqU1+fboqYzpPS3Z3vHGIBJtZ2lXOeLQcie7r+8IA5NQA2RHQr0Z087LLkWfbZfqWIA0LHZztugyVGdtA6x6M8mmx9HahbWzi0KXNyx/sotG+9SudqSL/MjR887y8mqrx8+E+dlAue3TLpTHUtdiT5JzawS6XUnR9Nyk2baxWeo7r0bQ9x2htJzImL6GQ53l0lWbJtfPj1Ie4SWLPrcj6+tYGLkeXRGmFyyqbTllxzmG2eRYtmr1WrS6y1v1lSwkzflhiqgleV4YsFjwiOnrXDQj7dKdIst5knOnXaTVs9hYrNthNwzs/CZFdLMjs6nbyKjctUULFbs2nxhRs93QobyRIv5Du5Ykx+/y6bryY9rl18/snFBY1DiUsti8ydVGZMs9I5qdpHr561ZcrYsUimbFNmgKJQup72In55kuIueuMvIqqb3FoTI7bRs0udpIKfz/7L3Jr2xbnh70rW430Z7+3O41WVn5ylmNH1W2QWLGhDFjBBNP8Mj/ARIzxIABExAj/ggkqhBC2ELgAW6oTpXO7uV7973bnD5ONLtZHYNvrbXPTTurrKq0sl7hkK7uvedE7Nh7x46I37e+LjGEI0rSbL0JOFxKzN6HxCzG4u1EBPTAcKVgyLgGzcWQXIOTJfalSkdlmTAl3llur8YssQ6wcwYkVVvOa95MNT25kkan5+JnWromtyElB8tUWRKfSIgBxJgCgyL6E1kWQPRAQFztY5HTRknmWPUBw5EqTLKdke1rbx36I1WOZVxI6CGifvAYjhTqe49QE7BRTRAwriTaK4txrT9IDVZ9gEvssnCA7nyR+YpIz6NdKPpGI0qyLy0pTOf1VWLrkjxadR5unkODREkmfirdFGk7MrGdwpLR8y2BkhooC5aDTzJhX8KEcnqrHBxTZZH3lfLikFJm5eiLPJmhQild2iY5rklsawJ6sdIlwVb2yZcZQknvjjkhV08JwTHJicUwkiEFKMHVZEDhfGEGs3Q8J9OWW2YynefjMnDNybRP2M7Y9wVMIqS03CehQAVnhACh1Ify1yKtFROw/XNYyxgjYk6m/Uvc/npIYev4f//By1/6dpsXX/zKjw34lgPLo791EX/3v//PcdrscVHv8NXhGL+xfI9//PbXcdIeEKPAj765wOnJDt87vsYfvnuB75zeoZIOM23xsn3A73/5fXx6fI+f3JziYrVD7zT+3vlX+J//n9+FWlmcHO1QpQ7Hk2aPrW3w5mGFF0eP+OZ+jZPFAbV2+PLqBALA89MNXn91hsuX99h2NY7mHXyQWNYDvrw+hhs1/tZH7/BnP32Bdt1DqYDd+wVm53sc7mYQtUfVWPz6xQ3+9Acf4cWnN+itxrIecX9oMatH3G9nGG5bzC72OJp3DAL6+hk+//hrfHF/iu2uBUREvGpgXuzR1hZSBvzW2Tv8ky9+DVIF/Nbzt3i7X+EwVOi6CkJGLGY9TmYdfvLNOUztsF50GKzGb5xd4c+uLwEAfVfB7QzmZwfs7yl/vfz4DqNT8EGyI7RiSM+n6zv8sy8+xvxftDD/0Q36kVKNoecHnR8JVhanB+xu5oCM+OzTd/jhj59DLS18p6FbB6kCjpcHvP/qBEfPH/HwzQqL5zuGD71u4ZYecmUhZKSVKQh8dHGPr2+OEL3A+ekW77864YC9p1QmHltgY/Dx99/hq7cniKOCelQwHzI5yCIAACAASURBVO9RGYftpkXVWgyPNRAEZqcHhCAxHAziQOb22Se3uL5bFSmyWnMSDDc1v4hPRsjrCn4eIBeU5prGwb6bEfC0Hqa18G9miCcW0Qkgha1EE4EqQFbJw3tbF0lrqCNi4yH2lBIJD4SLEXGvMb/cY+gNQhAIOwM5tzg53uPm3QqiS5KTGbfZflGh/6wHItD8sEF/wcoYdzEWQAcA0AFwEtVqgP9qDrzogTcN8LKDP2hUbw3sOiA2AfKg6LXtZUqmDYh1gLlm2m6sYpKZC8hHBrfYMwv1QOZSbTTUQWC8TODeSai9hG/p381VQLENgBcwDwr2iCm/vo6o7yW674zswbytpx691qP5xqB/RRBXvTWJ8YkISwdzbWAvCW7VRqG+leheeTRvFLpP+HO9JZsrO4nFVxL9KcNYwiygvlbon1vIvaKvVRGoMLkz+eUSSFEHgfEkFGZAWEoFhaPnt3vlCxD0xw716wp2TRluUBxQM1ubfc8AsP6XEo/fC9A7Sk3lmCSGSa5oNgRKbhEQdcTqX2qGhZxTbhwqDlWzNxL7TzyaK5UAEQqDofeC4OpRFN+qdASNOU3YzQkAs4d2PAqYvVGwC7LAaiCY8ksy5L6NZbv1LfcRUZSqIzWmhOA506XNTiJKArXsN9adKJ5cX1NZ4GZ8rqgo3/U1QdC4iph/w/Nr56Cnekf2ePF1xOGSwLA/J/DjMZLhlpaptfM34oMhHqA8ub0ScA3g03mrNgJPQ210xzTY7SeTFFoOKPuqDzzO5ZdkHPNz1PexABXXEBSoMabk1gkMVg8oXm/VM0DG7GMJnim9eUneigxu5CR3zX2aIdUziUBAR2BB9vXwjDVb2feMkCpzkkf08JyJzbP3HrsXCWxY1v2ofgryUUPye7qIxVuP7kQlCSr3nQsScZLsSlFYz/YmJLZRlN+bPetYuvPk+9vHcry5KioH+ZR+zlyj4yfvqwgEXNU2YDiSBejmPtPZtcfuuSLYrUR5bbM82uxjqZ9RA/fftkzFpQyVibXDKn0fZTCTPhPMIZKBTV7Z+ZWDr2SRAefamlzrQwDL10gPfIwe+H52KWE4BxS5WaoY0TzvOY1W9wE5sTjKpOAwIl2zMclY03UwxlK9kj2e3ZlCveGbIVfV2JWCPgTIMaA7N8XPqveeFScPFvuXNep7B99KqJ5AzLeKnZTJbxsSWMz1KqGSkGNIlTW++DGjBKqHAeMRF41DJfl7Adi5Rn0/0gfqI3yjIC19rGbrSpVIrjcBQCZ1CJAj2dSQJMJTPQg9maXyZfRwi4rpt7mfMQFA2ZNpizVZTrkfEJqKnsx6Yjazl5N1KpTIh0pD9iOiUpCHHmE1g+hGMoXOTz7MpiLgy2BNKUptQyBb6T1EPzIxNkagH1hDkgN5st8zh/0AlL5aC1ElltTaSQorJDAMEPMZYtfx/wAQyWrGvB9SIo4jGcsnoLFsJ+3rv86D+fQWc2LtXwG3/HUAlr/3eR3/rz948Uvf7uzFz37lxwZ8y4Fl/eqj+NF/+1/APlbQ9xpqYCy/XQLDqcfJH0rc/3YEzgbM/qhFd05pWg4vEYHSt6DTKvYsYPETjbM/tnj7H2oMFx6n/1TBLij7UgOHlvHEo7nSqFJ6XR6gRPL4jMcciMyWXqT9C4HmFuguYpGS2TkmucmKnZAi+bTqOw4z1YbDm2sojeou05CSlADtDf1Aw3HE/Btg/wqYfw0cnvO5d991aN5omB3v7xtg+WXA3e8INFcp3rxJkvj0hVnfA7uPI3wbsfyphNlzqMi+JcqcIhavBTbf95j/TOHwKmD1Q4n+nF92+sDV7/5sYiL0gQPI4RkT7x4+A6V9+XWoufquxojdx8DsLWsMcsCG7iIefw04+VNg95HA7G3E7iOB1U8jdh+LEpSh95TCuZlIlQ4eh3MFtwDMY5JWJanVuI5Y/4jnwM4p8xqXrA7ozjkA9RdkIObfcJvdZcTpH0U8fspqiP6Mq+P9GQNkqscpdfHoX7IzMA9+1YahJbuPeQ7GVZI/nXMglh5ornm9zN7xuMwemL2N2H4q0NwwiZC9f3wtZlcB248l9J4DQXPPL/ztJwKrLwIePhOQVmD5ZcTt71Bmafap3mE29cSZLuJwQcYqVxxID6y+dLj+XCOXwLfX9KzVtxH1NuLxE4nuMmD5U4nmIeDutwTO/0XA1d+RkCOfq7mOSWbIwTFKoL0POJxJHJ4JnPwgYPuKlQwu1Vec/5HFu3/fYP2TVLcwxMLg9GcC87eUwvVnAssvuQK9eyVw9CPPASslPdqZQH/K16i7YGCL2UXc/bZA+15g8Y3H4Zyr+e0tQzJ8RVbi5m8rSAusfxJQbzxufttg/bNpUB6XwPqLAFdTFog49TqaLRmTeRqsu0v673IHag4KOv0Th5u/TQZy+TqgP5KwS2D9U49cZD4uJYZjgXHN9377PjIsZc1ht72hv8rOJZp7SvP648yyCczfeXhDmZ6vBVZfUcYnUo9nVMDsXcTsyuFwoeEbDu03n0u8+j8G3PxOw55Kz6oK1Uesvhyx/bhKfsp8XQusfxQ4DLessBjWKZBpJrD+0pUexMO5wvy9Q3eiC/Cdv7XYfmxw/MMe15+3WH/h4GuB/oh9rnYusPyGrIjueb0e/diiO9fYfixw/v86HC4U1j8d0J8ZmK3H/rlh72Nim0QA6q3H7pnC/Crg/jOFl/9oh4fvzSEd97e5p49v+bXF7fcr6I6gZ0j9lrn30s7Se33Nzko1Tp2LzZ2DcBH75wbdOc9Ze8uBbVwIzN85dGcas/cWm18zOPrRiM2vVVh+7VDf9Lj7rQXmVw7jQiXwRwYoy27HFQfg0z/pIWKEa1gtwcqNiNmbHve/0WL5tUV/olE/emxfalS7iKM/3aB/vsDhXKO9ddi+0qgfp7qSKFDYpnEusHptoXcW3bOG19FK4PSP93Azg/7MkJm+d/CNxLBWECFi/saiuzBwtcDs2qG+G9A9a1DfWyAAD7/eYH7lsPmOgeoijn464HBRIUpg+WUHtzDQO1ZmHC6rIiWtHixljSuN7kRj/ZND6Vys3+0wXMwBKVKQjGZ/pCPTFLSAsvSUd5cVAds7C18z/VP1DuNxhSgFZl8+4ubvHqN+ZACOrwQWb0YcLiusfrKHbzRUZ7H53gLtrYN5HDlIGwU31xjWCu21pYe0VahvekQtMR5XaN53GE4bDEcai697/v76gMPHS7Rf72GPG0oxBdCfaKx/sIU7qpnCuhkAH+GOaoxrjfZdj/GohtkyOGdcabRXZHPs0qC+PgBSYvudOeZfd7CrCtVdj+G0QXPTAyFgOJ+h+XqL/tWyBB7pnoxfdT/ALRPj5SOqt4+AkhieLRGMQH03wC4rSBtgbnawZ4sEHC3sqoJ56MmkLyqYqy1iQ0AzPFug+WaLwycrzH50i+HjY1Tv9xgv51C9h77ZYXyxgt5ZyHsOMO5yDbXtMZ7PUb/dIsxriNEhtAbqsYdIgMsvagLBjnJY4QLgPIHXrIHoBkpDT5esI1nWiCmMyLzZMGin0pCbPSs/Rsuqj8c9Jauzhs+1nkPs+1TrERI7WBPoGU2AFgJENyCuFwSXuwPCYgZ5/0gWMET+nUCh2B0IsCoDdD1ijAR01rKjskvMoBCAVojbHUTTIHYdxHKBuO+A4CEWc16Puz3EasH7Pw3tSV2Y8dBNYC55OGOMTHFtG+7fMExAVAjkWpDY9RBtA/hABlEIspHOTSAxyWOjcxB1nbouP6wdyeBTCDF1YP55txi+9eE9v/d5Hf/PP3j2S9/u4sVXv/JjA77lwHLx2bN4/l/9Q6yWHZbNgO8fv8Pbbo3n7Qb//Ooj/N7Fa3x//hb/6OY38Ic/+JiR+r3BYjag0h63DwtUtU21EWTnvnN8i83YIkSB11cn+L1PvsI//+ojSgQ3NeqTDsNdi/qkwyen97jez4s8MQYBpQPOVzt01qDWDiftAT94ewHvFMKgcHy+xeanxxDnAy5PN3j77hhCB4RRwcxGeKfItv3xM/zm3/0Zfnx9hm5bo1mMGDqD6Chx1KsRq+UBd+/WgCd79u6LU3z03Wu8/uIc7dkB3X3L5FsrgCAYyvJsILP3dQt3QibGrAfYffry6BVi42FmFiFIsoqdQn3WwQ5cPg97g+qWbMbiOxvYf3qM/rmH7ChPi1XgduoAKFZv+Lct4okFdhpn37nD9ZfH9EseNORBQr88wPYaeKjIqEWQvfp4j3FXQfSUOMaTEXFUPCYVcf7qATc/O4GwCSjPA37js2/wwz95RcZtaRE7zcqE4wF+U8Ec97C7CuaGPZ/SCthjD9Qe8sEgzDzMrWaAy4kFnISwBA7tO4XDKwdUAc3rCsOZR5x5iE4x/TUC8jt7jPcNZCdRf7TD+OUCoYkQqxFhZ8jEBQGYAPWgU1AIWUrhRfKgRGBtgUcN4VKdwpLnRe9Yh9G8Mew0BaDvNfwysMO0InM3nHpUdwr1HbD9noe5l7Br9lnaVUD97AD/4wU9lCsL+b6GP6UvVm/JIMWZh7nWqO+5gGGPPVnCY+5L81ZD90B3GRCqiOZKYThNab06FiYkNJRetm8U7JILF3IQU/dmk4NNgFAzLRhikgOKIOCamMC3wPa7HtV9Dqbg4kS1ERhOOQjqA2WNekepJ2sKyFwFQ5nkuAb6S4fFFxrjiqwUg1AA970DFv9khlAB4wqFgezPA2bvCKabK0l5YACCTqxYk/YnAkNKBJYjgMT2Zcnh8quAm88FzE4URjB32y6+JOMDJHnlEashxjU+SJ2lny6dHwe4ZYTZTMwSkOR3cpI4xiQ3NDvg8fsOsy81E2YV0D1Lss2Wi1oAF8mqDX1uecHG15R8mi1ZqXHN13j5FRc+WEPB87J4HTEciwKo9YHyRwBlkazaRtz/FgOSqg0ZwZM/TecgoHQ8ZlCXqzbkMEkh9SHCt6L0XkbFxZMMJsc1ygJb9kYiAMuvp4UZIEk/O+6jeeQ226uAPgFqu5gCXUTgcdJPymqI4TilyNrEUi0ETn5AID+uUkWCR2ERc/JuZsTsnIwcBKbO27QAYLYR+jDJEnNAUZRky+gbJDAYjlOSb5Il2gUXVaIAbJJ4NrdMzc3pmbm6o97Ess3DBSsk3Gyqr8gLgSIAzR2TUgE+d72JhQ3TPRfqpAX2LwTmb2Kp9sjBNuZAr/Xjpwr1XerdXPFcZGau2gXsXijU91z4qR/ZOWkOAduXGuYQC/MpIplTb1AWZlwzpeNKF9EfS7S3gf8+kgSPSpSUX+l5nubvLIZjjaAFTArRqbYe/bFCfyIxf+tR3zvcf1ZD94mh3ofCHo4LifrRw9cSLqe6JpCqxojqMTAERwnYVmD23qI/M5A2otp6qI6P9S0XjRhkxARcZRm2058aVI8ew1qhuXcISqT9TMFBRsDsPMyjRX9RQ/XhScJ2Dkzj8UYtCgsoPEGEm0k0VwOikXAzxYTmSqB6JGhgZ2fAcKxhdqEkNgPsR603rOUYjgxUHwqDKF2Aa9mrGZWA6nx5/lCxjkN1ZCJdq2EexxRsxRRZ6QLsKs0sIRbvreod73OwTGEF4FtWeNTvD/CLiqE2WkA9jvCrCmo3Jhkq4I5byMEln2SAsGQPQ63hW00f6OghRvoGcwWI2g0INVfn5WApV81hPbXhe8x7uHULc7NjwI9jwI9I8ld5GCcAqFXp2ZzSbQNEPxZwHGsDcegZ1JP7LZNXkSm0STKbpb7ZVzmM9GP6QAB86PhxnABlxgQip9MCQFMT1OaP7kiPZQF4zgFSTTUj+ZZltj6l44ZQvJU/L3f9+ef/hbcQEe1f3mf574Dlv/3btxpYXvzmafyP/6f/BAvNi2ype2xdg6Xu8cX+FGvT4223wvP2Efdji+1Iz1StHbZjjeP6gDe7NU7aA2Z6xLv9CmftHp8ffY0/fXyOLzcneL58xHascbuf4XR+gBQRd/sZls2AbV/jbLEvnY13+xm0Cjid7bEbayyqAauqx+vtEWrlsR8rnM722I41Gu3gg8Sm4z5JGdAah03XwCiPh4c5vvvyGi5I3O5nMMpDyYjHfYOjRVf8ie+37HlczXrshwonsw7f3Bxhueiw2zdYzHv4IKFkQD8anCwO2PY1nJfQKiBEgRgF+q6CNg5tbbHvKhwtCbTbysIHiRAB6xUEgLayGKzGdt/g/HiLm/slewH3FdoFJ9qYt3vL+paqJkIYeoOPL+/w1bsTxCiwWHXoDjW08VAqIEagqSxGp9EdaqxXewzWwDmJsTM4Ot5jt2+wXHTYbGc4P95i19c47DmtKu1hjKc3dFQ4Pd1hs2vgBo3ZckDfVZAylP0zlYO1PK6mHTGOGkoFeC8Rg4Ab0oJB7Wl5cPQGAmDqbBClciZYCSEj1kcH7PZN8lwqxEFBNg4i1dv4TQVzNDBBV0aEUSF2iZlYWEQnIU2A32sGBrUObmegFxbeSUpxRaS/TUb6KQGIUQILC6Eiwl5jdnbA4XoO1PS1Ricoca0CZJP2eVRcXFgOsA8N/Zle0JtXBUgTEO+YxBtVBJYOsVeASVJOJ1FK5fL3ROuBLvnvRoG48MAosbjcYfd+wcWOXPXTc39ErxB1/gIEFwVi+pO/31IvozpIuFML0SvIXsAfu5Lsi6Xj/g8S0Lx/7nuLOjIxtw0QViA0AaL1EHcGLATnOY2GCyJqoyGsgFtxwBFWsOanU5DJ5xYM61ZiktpFE9MihwAkuxbhBWRPBjf7GgE+NswC6ivFJNx1gLmTECGl9QoGBbklwbwIKH65XB9Sqm8iQ5R8ywFXd6L430T+iE8yu6Aj1Ej1hXmUBdQXr1eWbR6Y5ltt+DfrdgT8LCI0EWqbrru06QzssyexPGd6DXN1BAEmjVu+mthbhlkJDGce7RsFt4jlWoiCktgoY5Lt4oPuSGmRElBZ2yFHpPM9SVWzbDxKFKmwGgjczS6FEj1J+4WcAHwO3HnanVjqY5KnUThMNUQRxR9och/j7Mn9s+Q0VbcQyCWP4Hyq48neVt+mFNl0PmSq6/BtPjcJEBsUr2JOV2XdChc+GLzEn+l+eo18w+371I1pdijXVU4Uzn5Ak2picg1PqQFK6b0Z5OX9z7JaNUzHm32BGYBnySgTUqek4yzfzP2IWYaZAXpOjlUjk32zHDeqD2We2YcHEIzntFnXTj7KD2qZjECTFAx5EcHOCGrtnCFCylJ2a2dTXVHuR8wsfknnVdMx+4bnQo2pfiS9J8xhktpm6XDQoiSpFrlrqvLRfSzHz1AeFJ+i8NNroMaYvJAEkQT9oUhb+Tz0m1LiGcs1WryIUpTEX2lT+nVkL6XuyQZH8fT58wJYLEC6VPF4QPUedqXJJKf7iydS4Czvza8lPwvJ+EsbWJ+SGbQcbCNZ0cJkWfolAcphERNrXdHPSCaTv5c2JKmvnFhtgL8PkWxmIPAGANl7QIoihxUhlm0WEIh0PCEUv6Uc6b2kxDX1WErJahNgChRKNSDZewkAcKFUm/C7iPJXymDF5BN9Os8ryfOTK0+Awo4WsAlMjOXPV31IQfCZa06ehAMhJ9bm/sryc8/tZHCZgeVTyevTbUg5yWX/ols5xm8/Y/m7n1fxH//+Lx9Yrl++/pUfG/AtB5aLz57F/+B//E/x5mGFYTAIo0I1GzHctfSFBQHZOoQ0qIdOM0yjCtNgKyNEpzgwHhT0nkyJPaKXi8Mph+bQhJTiGUvwR6yZUhkWHrACwkoydRHsZUSaueuAaCLMRrHIu/Gorhnb7eaBw0KfvsTi5GWi74pMUw6qCAowyUsUdOpcXPC+woPpo28N7AkZrDwQZCZH9fT0lIkw+VNyz6Wf8Tij4vCl94mFmMXSewnNwVL3rLow96qwS8A08NV3ArvfHFB9U5XnUj0HUL1n2qeb0xM1HoXyXCoVpecVcuGn/krfcN/lkEIxQqqISAOQ8EghLCh1EL7hQApMLFiWqOaB0C5iWvVMIComIBCQiq6TjFaQ/dKHLDdEGXry6yfstC95sC3dbbNpYM2DdqlMSPvmZhHzN1y9z7LocmxmGk7Nnv/OfX2hSttIA2F+3qiA9or+saevkdmn6zN1Q4YaqDYoPZYcwtPgu0c5frNLMuqAUnieAy4y65drNpDPi+PP1cCficSgyST3ZtplCglJ13Kop2NRY2Lm5um4+klaqtO/S78igFyUDvBxIZWm5+2FmvfNX95ulutDULoIc7l4CQop3ZxpHyPZMm6Tw2aWXZo9GRuzI2OUuwDzwPV0eMsBIvn1z0Nn7pIj6BFTJ2L+jpUow2IGkXLkPrmWssig6K8LhixeGcrnAip5/qSbzpV0oHwteddyR6u0sTA/SCCXyY1pwLexDNB8g/E1MQe+vtzXBAhTyEyuECkhOGq6bvJniGtFKZjP58bVojB35pB8hzYNu3X2vJG9yoyVrzObJQrwy/tfnjsN8zJVf4gQi98qs3jZU5eZZtZ5kCHKwIcMSupN1Bm4iMKuSBdLYE9mkPTAfaP/j/ugRtoUOPDG4rHk5waH+Qx2CmBNUt0MxjKoysEo5bMvA1sAZh9SqAqS5xGp35DPJdMclwGL6UJhzrwRYD1DfMJ2JWYugUXdheQ/nHon83BbbQPGlSrXcv5cjALls7feeNg5ZbZPPXd2wQOQGQw9IVn4mmVghMTMiXLMBDkJEMjMCk1AJioymGpIQCFGVmk0qjB1dslaBx5nuqZtYB1GAkD5GpK5a9ETsJCJnypLcsqpGkM5h/naY70FQ3Lyvj99/wMJTA9kEoPhHCNCRK64eerhFO7njjsS2MQn+5vlxDmBNRgJ6cO08AdQ3jn6qdYCgNoN7IuMgOx54YSWyay5K1PEXK2jpscnjyPBo2d9R6r0QIjJ08gXT6Tfh0VFQJaOmc9dQ1pfgGdmHTODWBizGKefhzABufDkGANf96gl8KQT9QOQ11s+V2YZbQJXSpJRrCtuU6sPAm8+6K98yizmfdPs4ywA+inDl/snc4jO021l32WuB3FJkirE9LskUYWURRpbmMqndSIAASQwAbswHR98ALT+AFDG0U77JyW3k/Y9+gChpjdpkd3+a/79592+7eE9f9OB5V+rHsu/zO3tZoVZbbGa9bh9WCBGgfaMwT2ZAesPFZT2iHWqiqgdRFrGd04itAr+0QArB9tKQAcIFRGjJoMUBVkaFeFlAjl1gGgdw2IkWR8RBeLCAUGwHuPMQjUO/qAhdAQ6BXvioOYOfmNgj5kgKQYJrCy8pmQzNB5Ba0BGhIVHaBSwcLAHVT7Q/SoBwyAQGnYVlnLzQWE891A7CT/3DJepA4QTGNahfLnACchBIhomYUJFhBkgekk2ZcHH2ksPsdOIFQNUVCcQcmT+gsDanjqoR6LX0AYGzWw0huOI+qsa40lifQaB8cxDP5KRsGsuAPia+zBc8HwEzclChNTzFxOQnEeoQwoaEYBbeai9YhIm0sA8i4CM0DsJt2AXpU7yT9UTuDKxUqC/CJBWkFhREaEmmxFTLyNi6mCrI/NsxgRAIwM6VJ+SGsdUbK0B88gvHIa18HrJx2d2sqyKF/YoeXelE7DrWEDY/hVZLQAlCTMvEMT0zt2/iJi9TcEliWHKTILwHCRtulbyoOna9KEtgCGlMta3Am7BbY/r9OYK9B6rnkN7d8lBVx0ojcshLmZHCaFrmZTp5iigIgNMfeB95IjSgal69vsdnscCwHOBuFvm4BVMATy55sGze5GeK2B4wVoH1XMgPTxLCaUigd0EXlVPHy6Q2CFFH7De8zz55CGm1FFAdQxAGY5FkXH6WT63BLN2AQxH9JKOS6Dacv/HNYEqh1hRAIi0aaCXAnbO56xvE9tmUNgR34qyWm92BCAldMSgBLREkcBvINjvT7mfXE3nvmVgqg+YAGjI4ELAJ6CbWahqEzEe07c8HHHxIVQ8jgKmDChFbRNQppoKUQmM8wnAS5/lkATX0qaKCw+4Nc9hZqSyrNi19JTnnkq9j+X199VUd8IgEgHfsE/P1wBDf+jJHldJGpuYx2AA76fjz6E642rqz8wsKIapBxHIixexSK8zU5fBuJ3JsjiTgWzQMklUOYRyUYVAx84FwshjrR4jA2IOMSWPpmstJX9OC1apt1FRPgsQZO4v+Zki8wKITkxTSkIl8BDT9hL4FCE+AW/039rZBNBFYDqpbQVsWgxTaeGjO1YJjMe0CMBrI+/DuJBc1EhS3f5YFXCkxg+ZtacLBlGyw1F3ObmU568/1mQYa1lCaaRlqA4A+CgKE2f2Af0xN2oOMQVoCbhGJvmnQL1hZ2Nm2cYFAV619QhmSvvNgDwza6GScDN6Mt1ClZ/bBT3F0ECUEiKm1NLkYae3n/8PmRkV4He2J8BEWojwlYTuUqXGGODmCqoPZMlSDUbUrCJxcw049lPK1OXoK1kWRoIWUL1P73f6TaMWVLoIINSSYLmWUBnIpkUyNzdk+0JEZrxcq7mYs3fl/U52TcBVBMWYV1PITTU9p2s1hAskwBwZQtU72IWBPrgSnIMQ4GYGKoOMiqBOps9lERhmI4xCMAoqA1KA4TYJaJYFoVrz3NWyLC5n4MOFkOTxcwGh1ul9QzAprC+9lbGhNBUhIEoFuICY5K9icIAkMA3zushaY+6MzKmq2XuZQZ9W9HAqhSjI4GVZK/fdFCCYE5WfhuqQ9UtAtQDQOAFJkd7gMT7prwwTWAQmUBkiz1MGdSnlVeR+y3yOQ2JJRULeOQU2XcvFa/mkExPZe/l0++XpM2BNyb4/z5z+/O2vUa3GX+UW8Bcc57f49q0GljEK7N/PsRfAycsHKB2SjFKhu2+h5xZCBoRBERyOEu1xB+cUnKXEERFQJsAcDbCbGicvH+C8wtlij59+cYnFskfXVfAiIgaBaFPNQevo9RMR9VGPtrbYPMygTYDbG8SZRzUfIUSEh4bQAXJXQX+yw3DXQq0tghfAKOnRi2Ay7NUcQhG4xCXrFsTRiNAroAmlPgACbulBFAAAIABJREFUrHm4WyAqxZ7F+wridEAIGs1Jj7GbAxUTICEjIiSa4x79XcNZd+YR50zoFAdNQNl6QAGhCpCtg1p52F0FfdHBPtT0zSmBULGYvvpkB/9nS/hLi7hnlHWcO8iNYRrjlcThOdlaqAgxKrRnBwzdAqEKgIkEklWSELYO4lBBvOjhBgVVBcRBQWw1awg+7eFjDZwP8Nc15MrCV+mDZkzSvJmHuDcpkMlDjBJuFgBF5WZmod1MwC88fBRQO6atqr2EO3HQDxq+DRAW7NKUBFlRCoSlh9wrDv5z9hxKqzhUaaB75WAeFPzKAVuFKAmSmMYJmE0C7vU0VJERDghNQPNOw7dkZ4OO8PMA7CXccUCsIkHykUd9reCXHrtWQHYS/XnS4AlAjRLDhUeV0jndLEB1CnbJGoiYjtXPAsQoYFdI8fcC/XOGPvXPHJn09xpqTyBuVxF2HQmE7SQzs8vAeoNBob/0mL1WCMnL111EDKcxDfKscohVRNxKMvPHHu03GsMxe+yiAvwsQH+jpmRPybClnG6aKyO4QgKMRwHrHwrc/z0LdUsfq5uTpc6Jpdk/GGqUagzfRCBwULMLLiZ4SwAg6gSUVnytpJMY10lSKMgSuzaB75Esz7hEYYmC4SJEMCntM9WGVY/0beoDE0qba9Bn2gL1HV+H4TgtlASy+9IK6K+4KDKuKb/NDJVdpsURmRZiFhMDTQ8XkPsVpQP604jmmr7J7FccjwV8Rd/k4RmVAM0NF3KaW4Jkd8zgILukl1SZ6TmCBrAVsIv0nDKDseRbbfiYYFI1yJbnFRuBmADnuGZwWX+WFAyrBMLTEKi7SKB7EAnUEaxVW57X4TgFaS25qDKcpEUDCWBLcHd4lsLFhsmH2J3Hcu4ymPU1Fx3aa3B2qlLvYmL5+nrybCICYQnIMbFWcQpFIhvF69cuEkBLXkizIzAORhSwLQKKJ5WsGEGerwjCDucK0lOK6Gr2S1YP9HgGR5+kr8nyNl1Ady7Tghw9lUFPx1E/pNCyHUF4lmWyh1UgzgUQuN9R0QsbJb2NYw4ysk+kwZEhRtUwAWQRgXFB0Fo9kqn2TQrZmwtUW8DOFfpjLs6IgNJbCaSFkLxfMj2+4mJf0JJAODHO4yJ1ECoGDNUbAkZfP1GpaAHTUcIpAjCuCC7NIYNaVlh0p5Kdo4kZD4aVHRDA/kLy+l0KrF47uEYWySzAbWUWcVjLsiAhIuBmEt7wPqony42YXq+7iGEtqeCYERxKI9CdalQ79ju6lo/XQ4Rwiiy5QpK5AmrgIuy4VNB94PM1Au27AcOqgj4E2LlC9egwHjE0TA6B/klDQOvrSTIblAAUATU/L2Va/NJQA8G5CBHjml7UeBBAo+hhHAKEi3ArBX3wPD9aYmwV6tsB41rDSG47OHZZVvcjok49kgm02oUGlSgyLfgxZKm6H4EY4VuDmPZV1qowrdlLGZMsNktifZPA8SMtVGRPJSCBUGuypzEStPYOQUvI0cHPc70Ut6/2OQBAIMxqQEuI0ZVqEACITUv/5G4EDEN3RGJKRTcyzEhzdYq9lhqx1mRZVSISlKJXM0YygFmiGiP9lqMFjKbvsqn4HIc+PX/Fb8gSvCOmFFityTrWT+4zDdf0ej4FoZnB1KlKRWtCI+cmBtsHhg15T+bziWQ1h/b8K9LXp0Dy/yegMgLwf4OB5bdaClt/51X83n/397F7mEHeGg7pSbIKFWHu+AESNQdGt2CoiEzskm9j8r5QmqrvNSDp85EWGE4CqkeZ5If07viaYEU/aH6QH8hG2AUH+moj0F0GmC1lnmrgwKQPAt0LD/MgJ/N8YlV8w5CRUHFf9T5F3Keo/XJcFVme8SigvpWlkyyvtLsZv+T7Ew6Iw3lAfcMeOTmmlegsLwRK+EVmkYLhwOdmQJTc5zw8+ApkuI5ikYiEOqK+kegvPRY/UxjXwHDpMP+ZJuiSYKjP8Yj6T1oWmyfWajzK/W4c0HKsvu7ABN7+SddYks6N64jZO4aY6ENiDvrkWUmeHDeLaN/xseMSxUvkkszQLqcBrn/msfyxwnCMcm4Apu32p8kzMpCVaq85fI3HEfOvRfG5uRmTaw/PCQIB7k+VmLz+jP16ALc1HvHc2WVEc5fYuRlrAkIa9LuLmK4flO6ycc1BXPVpMcTydV58LdCdp145zfsGLSb2JLFA+gDsPmIdQvY65bThKIH9S3b65Y617jKiuaJnyc4FhgRGosqsLQfV/oxgs7nioLx/BbRXAv1JYkXSOaUMkSyh2ZH9GdYC/XnE0Q840AtPwCUdC+3tKqX9zjmUybzfLbB8nV6PJbD4JsI1QHcuYA4pTMVkKQ6PUw3AcML0UxGA/UvWRzR3iZFUfBzAQVoNwHBMVnrxMybldueSAS0nkyy5euT7dP8yDcsLXkv1HT1FwfB11R3KeyIqlGqInMCbGcjsd2vuCMR1z365oJMEODIlOMuHRUTp0OvOBNqbmHxy02vFEKRYht9SC+ES25qkuos3AXYmMJwILF4HHJ5JzN8EdKcMOBkXopyb7EejLDp1BqYhvL5nv58aU0Lujsxac8dOvcxA148Bh3MOkWZLZupwITG7Cti/4HMDBEL7FwLte6B5CNg/kwwSaoEm1T10ZwLztwF2IVIKLP1w3TE/b91cQB+mZGHHRiZ4I9DeBgxH/HzsTwVm72MZ3r2Zwp5yWE79EItfstpReptrO7KEvNoFLug0Av0xv0OaBx6v7iPam4DDhcT8HZOAZ9ce+0sFPdC3t79kdQOBFmWMrp3ASzAEfbObNGAmiaWdkTXJjOAEuPjZIDwXZ4IiMGoeAsY5wQkrMFJlBnhuTBdLd6FdSNg5gVjzwNfGzvkcVQqpsXMBVzNRtzvTXFTpInTPLsbqkQzfuGTAjJ0RBC7eeoxzfqfVD7748QBgOCLTJ32EHKd6jf2lxuzKlfe56lkd0Z1qLF8P8DWvLTtXMDtf2C6m/6qkYuBj1JBer5ZhRfWDw7gmw6NsJFPaB9iZxOzKYjjSqDYOh2cG87c2XVMKqvdkMjv2HWZVgfSxgBsRWPkRkkSYqgSH7sygvmdwTHdmSkpvcz1gPK4gHOWyOgXddOcVqkefFq7IcCJJheVIya7eO8RKwqb+xlzP4WYKZucgrEd/2aK+TQE9iVWUnknD9W0PmxJpVe8hBw/VWQxnbWJrCbZV5yA7B3vSJMuNm8CdEpAuQD/0QIwIjWE67cOA8aRBdd3BrWuog4WfGUqOH3q4o4bH3FtEJckmDlNfYzSK0tdZBekCxOjgZxWfb/SsAXkicRXdiLBqIYZUC9YYbs8oMqxCQO2GxDQqiMHyeRID/bQnMvdLyscDotGIbQWRQntEl+o8AhnNXAsSE0CElASBubKkMkyU1QrC+SmEx5IFLb2RKWk1Gl2OAU9lpTpViwDlviXIRwpguyf7WFcTaEzHAqBIdJ9KYiEEovMEdElqK+qK98mBPTEgDiMlroLS1xLWk7dpHURiYf+VupF/09sT0PltD+/59z6v4v/++xe/9O2evfzmV35swLecsYQT2F4vIPcKwlK6hgioO83V79sEMNZcuZZWQngmGbKagl+gIgDxoKEPZELMnoPtuOY2s49tXBMIhUGjvkk+njT8ywdR5FZ6L1A9TJ6qKv0uvlNM+ksDmWspHdIHfjnHbgKboRIwW8rAfPIPZSlZVJKr9BHQSWKnO8r/pAPmb9KKfkMQ0tzwb4DeK8a1p3LtrIiIvH9OR4xSoL6LmRCivE4grZynQSutOLu5RP2QpE1vNLvOHrlqPR4LzP5ZC31IBdZaJJ+XoOfvGVe5hWPVRH2fPDlJRjV/5zEuJFd0b6bV++aagKB65OukDzxPwqWAhZmAUUxqZOpqGsLH5PuRQJQKzW1MwDVVRCQZXRRcQbZzUXxr1SaiPwPm73wZFrliH9FecSiNGqiuCaCqh4jqIcnKkv9H9wyBaG6nYR+B11swHJSHI8n6giEijphKq3dpMPL59eYQDkjIMfI8RrISNrI+5aB5TeQ+t/ohAZYuFrYiCiAKfnnKkaA1Sh5/9mo1N6mq4T4FPYwcqn3NxFqzj6g3EXbBffKVLMEVZps9ThNr194GCCchokCz8dADQWzfEUTN33tslUb9wEWaDLohUKo0qk1Mpe7JByeA+ZuUHLkJlAQmH9HsOkBEFoDTT8hV7ObBw7UKxqbjSUNr/cguNbMXmF/5tMIdyWzoiemrHziszd5xm+0V5cJmHzAuyDq4mtdFTl9lzQtfA3MIcA2Hg+ae/w6GoCsHhETBvkyzJ6CoH5NXyyaZcc//qyF53yqJ/jhJF4fp3EfF9061oz9OpvCT+jGi2kUmOI4S9YZBHqpLksGary8ljAQgAErKrHQRzhG81A9J+ikj6keP3lGipw8R1T4gdpPEt9o4+MrANcDs2iHLSOsNh+X6gfJ56SSaO5RewXHJcyCdQLXxcHOJcVAwXUAwiq+RoFTQzkySo/L6qB8CPyeCKL7S2fsR1VbBLlTqQgxokq+uP2YRfL0JMAcuHpl9gB7YKaqHqeez2nq4VsI1AtWWQ104MwRsPV/v5de+dPz5mkmZzQap4iGi2gVUGwcIQHcheeMkmluLw4XhED8wzVQPEc2tpYxSTDJxCEDvPfoTjerRQzpZvJGsPWI1CIRC9UCDuYiA6giugiIrqjueg+rRQfUekIYBNSFC7zyUDRDB0I/XebJdVYV6ILtW7fm3tBF67yAtj5dyboNq4+Baw2t45yEHJsqarYVsNdTBUdrZSJhHSiV1T4AYlEC1DzCHiRGRgwdgUO1YHSKcRDQScgjQB4fhtGZfoQ9TwM/IjkU10OMmBwVIAfMwYFwptDeWElIPmK1DMAZmM0CO7C5UqZtZb0eoTnIQTz5TNYQUhJMHeEAEBb23kKOBmytUDyOrVR4HVLWC2VkgRMyylNUIqM6iEgKqd5Qu24DQauhD4H6eNGTP5kwfNduxgAW1HxCdgTcSZmcRjIIcHEwgYBODR3XPFFW3rKfU00AgLEKE3qbajhCh7/ZAjNAHk35HP6W0AXKw0HvFz6KDTdsIUz+k9WmfRoRWQ3YW6sB+RtnoAiCl9RCHAWJeQe3H8jgRIsHjsoHa9gn8OSghILqRoMpHxEZD9I5MHgCR5aZCUKI6jAR3ShV/pBQCQUuI3kJYhzBrIPqB94tMaZWPPeITICccvYNitGTonIfoBdNYLXsshXWUttZV+v1IVrHrCfKEgAgEbSIlwooUthNHC0gBYQxBYgXEQw+R+ihjSI8xBrCW/07AMCe6Ruch0u8iwN/vk29BCp6XHP6TVAIlcOeDUKAnYLBPbKgPEMozvCcEgkmlSl9lTKxledwvCunJP081Jr/oFv8iNvNbdvubLIX9VjOW7a+/iB//N/8A/baGauh3nM0GdL1BDBLuoPHs5T3e/eyUXsIji9grVjLM6DlEABAF6hd79Dct6tMOUkZ09y3UnKX2uK6TTIJF90ICvlPQMwf3kEpjW18SN4WKkNcV/NqxqsIzzVC0HrFXEI7SS3fKNEuoCPWoEC8GiKsafuEBL2COB7i7BkjhNWHp6HVcOpgrA/d8gLxmYqc7dpC7qZw+zD3MnYabB6heUro4CvilhzxIFrUP3C83o7/PLSiLpM8h6el1LMl+5pFevgySQhMg5g7mdV18S/bIQx1SiMUgWGURATEK6I7gqHvhmcjYTX5LIDGm95SkulkaXseJIaF3Eghzj+a1gVvEclwlGEVygJ69k+guAvzSo3lrEEyEa1PwQx1R3SkMlw71e42oKZuMOsJsOESojuyk2Qh0H1tUVwxagkRilxmo058TqOVwmHwOxCBRPUgMl46VHB3Z3szCSk9JaE6QzN5PfUgySIHiFzXbKeETggsF3UUCMFuB/acOzTvNNMmRixPCUR5aPUgmJq5iCc6RI2thfB3RXMskrwwEEvdkhCl5BoQFmjsyg3kQrx7Z05nDpNyMwVEmeRXdLLHCMfk5qfZK/kLKTvWBLGHUXODICy1uEeHmAe0bhnnk8yAdt+sbPjZ7w3xNaapdpaE6TD4wes1SQE+g5LG5feLbTN95vp7Y+pKiuWIQTXslprClzDiKpBJILHn2K5K5RjqPEf15lglTVipdYl/T+ynvVxRkk4YjUTpqcyCUHNlzqvqJ6csMfw44yWxnDoLJ+8mkRS5eBZV8peZDht+na64oB6pJCZB9iFFlGaooIS75PrpL+xonxl1aLhK5lr/PacHZs5YZMemyXJjXVKldSV7XesPETbK2fO7mhgtKuiMbqnd4ksKIybcnU3jSmgsFw5FAtaEXVyYmNZ/rekP2WwS+PvVDGqzE5LnMHi5fc0EjP166WNjh9oZMLcNmYmEvS4BU4IJhMGT7D5e8ToDs3xRJEkp2Unf8bCkl8YYy2eZ2YlKzmqG+j6XiJTPV+fqwS1HCosqiYJt9lryPmyVVxRhRbbn9HFAFTN5JN8seYSTpZUr9tNMCSlDA/IqguD/OklS+jnlR1baigNRhRUZcjcCwFGg27IbVAxe12C3Lc5TTUnPfrBpjWeDIITvjklLOvADiGjEtPgElcKj4mVMYkXQRtpUp3TWWpFrp+LzjMrGMibnMwLnIdk3ywD7Zrh5SaElKW80JqNkrXG1Def/k7leRFljIiBPwujbVjaQAGV9PYTmulTC7UFJ8fS1h9r6oRSj/5OMzuykHMppPb1HLCZCOAeNKw2w9QiVTHY5HTlP1Df2fUZPBZuCgw3hk6GF94EKAnVMGK3yE7nxh1+3KwOwdXJNksprBQ6FWRe7qZjq93jwHdqGTvDawrmRmoHrWfuSbr1lHInyYQKDgNuAIyOXIxYkMqNR+hJ9XlKyOPGk5VEh0FrFW+CDE5wnDJ3rKQENjCNR9nIJzYoRbt9CPfaolobRVdinsx5G9jE1iNdM2IQXZz7T/sa4KyEUIZByHcQoWSz8XORiovKARpd4jP18O8CmLHROrWP4OP/e79O/4NBwImEDnMEw/AybWMgcAPUnuzcDzg9tfBliGXwBS/w1uf10Yy//t989/6du9ePnmV35swLccWDbffRlf/df/AHZT03doJaAj5FYhNAHmUcGu6bHL6a36ID4YygACjdAEqB1lsr6J0J2AXbOWoHqQTzx2QDRp0POAcKJIbYcTDvJ2xch94SmVtSmIxKbhPRekI3DQdgt2Cw4nHuZRMlCjiUWyGzW9Yn5GyWz3LJAp202smc/pmUmOKRwls2ZLc0cwBETjSUhhJfw2kCnx1s84xJidQH8aUzImv7CrjSh9eG42Ba0wll5gXAe0VzIBm/TlnQbTcR0RPtvD/OECWfabj0UlhlgfREnoLBUJefAXk4fHtRH1A0NPcsJqTkIVbhqYqgcOz6HiOSa7yqHKLgguQg6MGEBQYrktNZDV7Z4JSi9Tsmwe6u2SQCanszKYhoNrVMB4zA5G1X94rZZYfjvJU/Pg7hv+Hkl27OYoaZlPawmyp44hLIkB7egFK+xzTLLeNGAGPUkxt9/1WHyhysKAa1GY73HFfcgD6bhkmidlu+k49yjJo1GkMJzldN6CmY7RLlG6FJ8mfuZznmXYdklp53AkimRZxMTSphRL32RWPisHCDCCFkl2TK9aHnyrTZIGOjLQmbEe1xzoc2CLtBwy7XzqAswAIqdv9mfA4qspACR7SQpoAqC6iOGU14u0CTTuuU/SpuPI388CyNUb9X0sw6CbEVD4FFhSznGSuLtZ8k82fFwGPAxaydcmAYZdiAnMJaYq1x9wAOSg7Gb0RDZJtls9RtgZ2fv6PpY+QdeKMqTnJFVficJo0TMIREGAQU9lkh8fUTqagejTJGo1Ug6NyPe49BH7ZxKz96EEtyBSbsmBOEIPsUhhC7ixEd2ZRH2fqgtCYjc8SniMa7nfuuNz+1TrEDXS/okUXCUwe0/WO6ei5vCfpwmq/TEZeXYe8jps7kOSc+bPT0pT7ZyPN6mawlfA7DowSXdESrTldah7suLd8RR+w8UVXtPZv5r/VDsyqK6RKUk4A5pYAo7sTKZk11j2P3crmkMoQIwM+PS+y9+TuqcE1TeiVFaYfUhglwDP7HjOXMvPRpPY1gz+MpgyewIg21IKO6zpm2seqEzRfZySYX1MUlbKUEUConzhkMDNpKpRnYeb03dYbci6BSM/+OxUiRW1c1WqThBBps5HDMcmXY8e3alOjHQsnwm+Fqk/kmzw4cKgvXEl6TUnrTL1epIoPwWEagglpCcnsuaOR2nJcvpGFRCmOw+70ARmyN/zOZiHj1MDwRI/MwIX81ICa0mJBRKbbTEc1zAHBzEG+Bn7Gd2cFha9p9w0V3y4VkHZgKAk9G4k8F0YAkuBct/sbeQ+8jxDiCIRVkNi+FyAO2qgDg52WaG662CPGui9pSQ1REjr4RuCQKase7hFBdU7+NYQrElJqWulp15JLcnKWs/k2CSZRSTbGasUYCMlz0+SyubOSGHZURlrU4AV/Y55WMTUGen9VMGhud/QijJXJVHSZ5WcgntCoFzZsnIk70v+v0hgs4C73DVpHUoKq9GFdX0qhX0KaD8ICnJu2hcA0TkIKcl46rRPT2WrT9NhnSPgAyYwl9+DSnLbKjG0OSwpy3HzNhPIfMpePr39Qgzy5zCU33Yp7OefV/F//V/Ofunbff7q7a/82IBvObCcf/Y8nv+X/xAxCByf7rDdtThe73H9fk154ZzUhX2syBbumfIqVGQo6tYAOpI11Kwd0acd6tpBy4CHb1Zozjt4J2Efat7nwESGaCLkwiLsDYvuWwc/qCll+dYgXg44Odrj5vURg2s6hbhwELmzsBPwczKnce7Rrnq4Hy/hno0QdwZh7gtzyPvLxOJEiOMRpnYYr2YQTkBc9BBftXBH6c1fBbKaS7KQoWaia1h4wAmoPZlBYQVizZoRIA/wiYmaB8Q57w8vYDaKAS0dQz2qjUT/akTzukL/8YjqHb+UnwJyFl4/YQofJIZPB4h7hvuIkIbgJ585ek8mEwEp8EdC9hL1ncDhE4fqWmE89aivNYZLx1RdEaF6rp76Iwd9qzlALgJkJ0pojAhkG+UgSiqs8CLVWJDBG4+4ABANCqsxDaYE7HIUpW7CzSP0VrAbsCHb2lzJEroiPI+pvwyo7iXMNrEyKeBE9fScqn4C2q6lB9MukvQ0JceqntLh/pQA9/CCixlRTSvllPNNASxBExBXjwIupfnmhYT8XDnMRo5MZK3v6Ol084jFV7IwV3ZJoG2XfHx9T7AyrpNs7l6gv4iYvxYFELJSh4siMYFP35KJbd9HbD+lJ9Uuk7dWcP9WPwEOl6L4aXPHW7UBumdceGlu6EmFBNr3AocXqVYmITP6IJP0/ZDZJ4bk+JagLS8CuHl6vSPZZ/NIsNmdcyFk/pqBMNUmL6qQPePiywTmgZQ+mz5aKXdHCbRR6TkgmGB79ocBw0piOKEUXbrk8U3eabtk+m69IfDMqb3FK5sY4iy15eufFkQigYLP1SqBbFu1SczfnscxrhjwopLHNhgCzv1LYP1j4HCRA18oy+3ORZJrk/WUDmivA/oTWUCPbxMITnaB/pQgc0wAuD8jmM03u+D/dx8Byy8I8kSkBDmza9lvmwNdfEPWTvVcLEDk61jf0wurdzzmkuKaQJzueAyZKdYJ2GbQrMYEqPf5ReTjswfZzpOX00wLPTotRuS6mexrzKA8S7jzOQa4uFI9RgzHEvXDxHg2iYHMCwcASgWNCNymrwSGFUG3Scm51Y4/H1cC9UMoKa9coOPv+lPev7njdVdvKbsOBsUvWm8m9qM/5nM0dyElr0Yczghk6Z9kSI50lOlLy//Xj/TUBk3JdZW8tmqg5Lc/UphdO/hGYJzTC1vtAg5nTJwlaJ+YbTWS6az2IaXLRozzBGp6sqKUSEf0RwpmH1LIT1JHpIWILFuOQhQvMPMKyDAGxZ/P37ri3VQHh+5ZA7N1OFya0iHZ3lD+nDsjeW0ROAadPZQE4EFxnzPQ1Xt6HdUYMC4V6geH4Uina94hGAmzc+jPKpi9Z5COZNiR7gKq+xF2ZQrQzJJju+A2hGOirO4myaxwBFLmkRJcyrItQaUUSRbsS2ejNwkIa5GSjWWy6fgU3qMK0+gbSW8pv44hxwSWHaX6PndIagmz4b5nv2e1GRmWcxgRGlMqToQPGNcV04H7JIlNIFMfLPd5cAiNKZ8jcpz8faFShS0LFfdV+IBQKQLdGCEfO/oQtaSst8s+xQS0UpVKMKz/kCOvHbnvEeZNkeoCIFCVecGAPkwEAlYm1Uqyj3UFZD9lrv0YxsJG8ovOTGyfTSxsqmxBjAShT9lG5/FB56TR9F12A0oqLEAQmAdVIQgOgYlVTLfMUIrKEIzmGpJ0f2EM4jBOHtC07QwiYz6uzFqm//9CSexTOe4v+n3et78BwPL3/y0Ay5f/Dlj+1W/z7z2Pf+d/+M9wsAZKBqyqAY9jjeu7FZp2xKweUSmPq4cF7LaGmlss5z0e7ubQtS+l83VjUyJ0YBrsj5/ho+9co3ca11crCAEcnewwWA2lAmrtcfN2jZcf3eKb16dQM4dnpxvcbuewo8Z6tcfd2zVOnm9wf79gob0VOH61wf3NEubKwK480ATGfqcydVV7hLsace6ga48YBcJDBSwtopOAFdBLC7xu0Xy2wf6bJWLj+YVWeeC2RqgD0KQPYAEIFRC9BLaaYLiXmH+6wfb9AupRwx9bsgcPTF4LVYRwAuHIQT7owoaENsA8sC4FMsLcGNgzAnf1oOEXlLxWJz3sdUvAKoH21RaH6znUVjG85lnKWY+AaDwTafesMsngPqaOUbUaEd81iHWEfmQSqkgASDiBaJiQ6uYcPEMTEVWEWQ+Ir2eltB7gcQGgLPlZjzAoBjy9rWBPCU7Nlgywb1AYHz+j70hvCcT90qN+azAee8y+UVPa6UjwFBUQ5w7q3kAfBIZLB9lJmEd6KV5+AAAgAElEQVQJNw/FQ8teSFGqS6KK8GuH9osK3a+NEDuN2RuJ7lnA7K3E4VmSVFWRCwwmpvTGCDRcRIigTHY440BW3wv05wHmUcI3key9F9CJmQ8myZpTcBOCQHNLQBn1/8fem8XKlqVnQt8a9hTTiTPemzfvvZk1l+2qtFtuut1y0xJICAMS3RI8IzXwBBLDA1K/IHjjEfEAD61uWbwhBEiNZCEh8QCW6Da0i7ara/KQWZlZmXc6Uww79rAmHr611o6TLpfdUpXsbDmk1M1zTkTsISJ2rO//phhiZScgmaVVAhgvKSNmdQh7T8s7gsMgCRSZbBkgHJNmZx9RTpyqVVwVYBce9SsF18T3XRXZCk+2P8RaixSQJQ0Xyq5O9yPAMfMQS95jpcWcwNUXyEMDNXKIIS2BcJKXkjGkSoF1F6ljjgyX6mMXYdwPgOdDtyIDucTamkVAsefrqoap8iGlfBa7KQgrVeeIgAwiXYVckRI02ffhDDlsDJ7HLMcoXa2Q5dQEzARU44oDIN3F53QEaK5i+qzqp+5T3cX3fEhDDqC+AYYIYKt77n+xpye62E/Ho2LvqHR8new87vNp9OFuk5+ZkmLdIrO0KWDGq7QQpYR2OKWqgN2o3OfhnIm0lNfx2G0zvSbHrLQ0ExtM/3UcSgREWTXfO+OJiNLx9HkM6B5xuwmcJ2mstCC7KukZTgwyU2ujFDqC5qRiCJq+0CT5Tt2cKTArJcMiKxlSjQgls6qL6aqKgKk/ZcJpFQOLKCPnviQZaGJXpZ3CfkRkffWBAJv1OGSWEyuZwOe4YGplueW5ToFF0hKMuyjHT/2hw1rm4cBxoFl9R+YxhRCZWRxERGAlbUx/FUy2TZ78oiUD7AsywK5iGBCADObUGKWpC/a0Jpb/WBVRbRwTXiPLyJRmgi81TnLT9N5MQUGp75HdriHXkaS/DSuJeuPz8UpLRliNIQ8kfAR540KiueVFMwFPr8jaDicSRRdifQwy25g6Lcf5BNzV6GEWCvpA0JYCh3w1LcIJCkP02BP4uVpOCoxYX3LMdKZuyqAE1MjnTAoL1bm8yC+2I7pHNaQJKBJwdQSu7LaMva1x3xlK5NCfF6hvDVylciCQVwKuVlkaa+cKunW5FzKx1Ko1sEuG8riCQ4vk80zSY2mZOptYU90a2EURvZg+15TQ2hMimylR7EYedyEJNC292ccST+F8ltMGxeEFBEFsYjGFcfDNQ2ZTGEcGtCpYR1JIyAPXSaFQzFuIwTuhZFASYiqtSEzjsRw1+jAR/YvHwTe549JzCJRDf0JgIFAEhokZTccjUn1M6sA8Sm7lA458mpE9Dd5PtSPHHZqIjOM4lZgHayGUpAfzuBMz7f+PAZZ/KhySgKcPfwEs/5jbXwDLn8Kt+fKT8NX/+t9DoR2skxiMxjAUcLsCzXkH5yTGXQlReBS1xXgo6IOUgGws/KjIVspAGa0KvLK7dIGZJGyylwRtpYdoFSPkC4ZUAEBxr+gn9IjSWEkP4iARCp8ldAgAVIDcK1ZwxMVx+luaqvq4sHcLDziB0FDSm9i9sLBQN0X2oameQMEtPMo3iguSWOHhy5C3HzSBiW/81OnkeQ5SMmvy/CWvIADIMR5vEdnNWNGg94JdkTJKcmc+eyZDrMqYfaQxnPsYjsT9CjqgvJWThFcd+XgWHsW9nLxwCjDn9JCyU1HQOzoIymqbqQvSF7F/UiZAMDF5Ccy5mou59PzSxECRWFGhenpDi3v6D7NvycTXL75mQRIgJV+nJ0maWU6EKfjHzunZK7cip/QCEfB6MoxBIicPkx2J3lw7VRekBN9ij5xMmcBwksDSSzr5Tm3s2xzPWQPiqriAifIwLp4iuAABQn6+6Adkfx/BVPJaZhnZMDEwx116GUAfEBfZyCAodX4C8fxGyfEUNDN9HkSIct4WuQcxPY4Jl+m9Hf2C0WsmXVz0l8jsclq8yXECXMlXebwwFfEY1Ri9i4G/S+xJAiNp8AJwu9lzmPYvgg8T5c25lD4NbOK+QfL4kveNBfN8/iIm/aZjjG8t3i9tX0zy6XS/9Pr6CFqlYS2FHJCvBz56OfPrnIYxElkKnv2Ufcj1DsLRr5hk1dKG6f2kRPR08j4JOKaBAc/NFEqWfGkAcvCJ8MgMjTSRuUqv1dE5T3LXXPIeP6tJCp28uCIwNKmIgVNq5P0RwVB+nY9Y58lfiuzxYxgUsucx7QfwEJTk60SYXu/0e3H0+cj7fHQ86Za+B1JgT/JdBiVy0vG07emzx88h91W6CeiQ7Yzy6CgZViZkP58e6JP0sQ4jf/6i5PVYhhqUyH7AxFAffyZ8vJ7nDtp4no8l4Qx5ikObyBp6LeAK/py6MYF47rN/dnpeFTtQgbgdFf2U6T1gQgQlYL9jlL8mgOLVxATbmsFWtpbQAwOudBdD4dx0bMcya3G0fkrnOAFY7hNZSzuL2w7IYW62kdDdBFJT2m1iFVOabGLnde+Y7mp8vB6FnHyrkpczASDjow8TQJjkxezTJeslrI8DtCiVNj6/V46PNRyxSEFxYa+MZ9hNlPkmCXD+HAZ+r6XfJ3ZT+EAJbepeBMgMus+sQ6OXMwfJaBmBFHI6q3A+eh4BaaIUOP5MAOf4+ZYyX1fS9T8dR5LrJtmx8AGit9lXmRLwuT8+SlkBHHstj/ZdeA9fFQSZib10rA4RMUDoAauYuyiPLiQJ9KX/x/Q6iMHkxwetpmAoAA/8lMcAMAHQtN308/FjjgDrA8D4427HwPD4Ps4h+SsBPGQmQ/gjYT4/dht/moCe4B/UmPyz3v48AMv3frEIv/EzAJbPn778Mz82YII7n8tbegte3yzhvIRzEmVpcfZkg25TYzXvUSxG6NIhvD9HsxxQn/UolgOUpsSzOe2gGgt4YHHRoliMEKNEddoDOqBY99CrEeKqJxNoBcTpCD9zqB+1EDML0TjMf/4OYeagrzqgoGdzdtlCnA3snxRxRRGBqJ/HD9/CkdW56uFXXP26E4tQeYh3DuyZrHyWxIY5fZiwEm5tEc4MfO1hziwlnaOAeT7APDLsP3zeIVQEhPBAmFuylIHSVDdnwA1CXJg2js+3dlDt1FXmG092zAqCzOj7NM8HfqHPLf1K1wpuEeU/dUCxUQSTdcigEh4Qo8BwSf+EXTlABJhzC7N20DuJ8cLBLik5NWcW5Wsa+e08YDxz8GWAq7mIo+/OM4Qo1sq4MsBGWWoCJin8xtXI4UDwie0KEfCQNatuVJbCuiZkvyOiTHM8dZCOINHOfQaK6sBkWgQCXVcGhvFEr6dZEGAm6WpiMA9PHPpLD1cxCEcfBM6+4+hDXHmCcRlglgFeB/TnIXs67Zw9i64OGNcB0gkEFVDdEGSqCFCXf0CGdVwzMdXVDDRyMx6f1yH6JqNPriLQFpaMZrmlD7K/4peQLzggcM3E1tpZrNyI1/0UXjKsQwa+XlP6C8lFuzkJ+dzqmLZqllxoJk+pbievpS/5GFdPIDPV5tgZH5M8uCmwKIXZuCYyaIrvh7wADsnLOAFNYFowky2dmD2GbfA/JkyH7LlkfUfcrwjsEmhNj0+MXwLFZaywSKDD1siLWTOPLHdSMs1i9UVBabKdxfMSzz+rgxAZwpDPqRrJaPqSQMnO4mvkI+gskf3T9FNH5nGgrDWliqbAlLxQLRD9SohF8MfDGN5X9Xyf+NRvGEG/6iZPHwI/j+MJF+cZ6NqjYUSsK/Gxn87H4BgZk4yTbLTcJn8kGRo5IoaXIDNeZiFyCI6rpmO3tZgkrYrdk8IBw0pGxj1eB8qp/kMEZM9pqpVJqdz6QDCTQnzUEDKoTOc0+bWVmVi5BMgSuGI9DBmH9F5L76P8Xose6JQGbJsEulIXL+WkQdEHOcZgn2rnYzcoWcMExumlJ3NJRikdOzIYS/5JDjfitVKKB9UlaciS2Mf0vIlp9Jpy2AQabS2nL3gRAXZMBHcl3x9eC9iKFShkH+NzhxhupDlMYNAP+x8TS5kGBGRlCTxY4UM/p48/J1CvRnpKVR8TlSNwVB3vm7oj6aEkCEgBO2Yu4+crgs4IAlXqmayYJKtbymp9JXNQXAr8UQO9jpS2Sx5PTSYtBQgl4Ezvucx+WfY3IgNqs2QNiS8lZajbmIw6ev4+Dh107GlMwDG9Hq5W+Vy7msccFI83RFDrlXxQRZL2wTYKZqF5/1rFwQQBnS9V9qIClLSGUsI1ehqEVfRP+lrlsBppPbssFeWzkASBflbw94LH4GYaiMzwxGAqwHoG6oCgNSSfqJZkIAuyhkGwMiQUKoYY0d8JTbICWhL0evo6E0MJ5yl31WrySTrPnwH8Ea8kgNw5KeUEKo1FqEsErchO+nAUUCQmcJpYTgChiF2V8TEhyWElf/fgMem/tG0AQgiCQR8gtKYnU0hk3+XRfY9De6bOzYnlFLH/Mv39x4LKFKp0/N9f3D53t881Y1k9fRae/4f/aQ7EcFUMCVg6lG8m2V11J+FKSt3Sl3BikRC4CNctv0j6sxCZSKC6lpQAAtC7tHCiRGpc8TG+oKewupY5gMbWBG3qEMMW8n4waTQxaQAyAwREn1og01ZuuZhLMqtiBxyeBMw/EZM0rqI0sNgxSbRoBYZ1wPr3gPYJJWtyBNpn9O0VW6B9Tp9frn6IK7GUvpoCXcyCgEuAzOnxZD6l8BV7LmyrG0rgdDstcil74gLv9V+3qH9U8IvS0MNXv4kpjDNMcj07sXFmzgX/cBq9eXqSCuqW2yv2wO5LFrOPecFKrEOupgAyAynHxGQc1T7EknqziL7RWcjVL/pAWV2xiyEjp7Fu44aewPpGYFzTh5cWjPUNpXzdFatuzJLHJk08jgMvkuU9A3fSQlZ30f9UCeyfBpx9B+jPKB2sbwPaJwLV3cMAkiAmAJeSHl1M9CwOUdZ3yU4+hFgefx2DahQwrPmZoQcNGM4D5h8D/SVZpHLD+9o5w3VSuujuOUvoXZKndUyRPDySGJeUTY5L9kWaFcvbxxVfv903BlQfVSg3kcGQBDrdlcDsJdMtUyG8Tx1wamL1klxUmjAt2qMHUneUP26/RAlwqgtK3YUp3EcfKB0sd5Qt+pLHahsxdW1GH+jJBw67ZwqZUY8hUPV1yAXwib1hKMaULqp7Lm7tgu/F+QuHcTkx9KnUvbsSWH5IL9rhEbsk1Riwf6IIxmIyZdGG6Nvi66wGwM55TIlddiWinzOgufHoT/ihdVXsOIweRrMQqK95/EAKNOF7JPkhdR9Q3U+BMqYRLFq/8fT1lWQUVGQ1g5jSVXHEFgfNawAEMKTk1bgf0iL3RdpaYFjH9+hMYPmpRXulcoetV5Q9+iPfc9nyvAXBYJzDhUTRMshHd3z+csfaF2DaTlr4ixBQ3XtsvqhzDc+4EmiuKdvrTySaOwIJM5tSSnXvY9gPwZMy3Ocg+DcRgHEho0/VszakFBGkEjxJF9C8sbFYHjl9lGE/0V8bWWJpA7pzifqe/j1bi7wgH5bcL+GB+sZiXHHR358KLD9hII4yrCdZvLCwNaubhD/ybEbgoMYA1cUU0rmCHENOvR2XU6poc21xuKQ8o7pnv6UaeA7K1kP17Glk2jFrT3h9JuhMzKBwgO6YGLp9p0J9zyTT4USivncxUdRP7LgHfXtJGikBM5coWh/BrsyAHUDudswS2J6smZ1JVPeWFVczFcOqLFwVmaDIgM4+3qN/NJsYOwFUtwO2X5hR+m0Dk1IFKFXtfZZzlpsRrtaUlzqmupZbVur05wWqewvVWZhVCQT2W1bXA+xc55AlOTqYVQlXEXSqgRVIXgsULYfhSdrpZpoD28TqlTKGStFLqPcG42kFvUtg0TMMJ3oOXa2hDrHewoWciupmmp5DKXJiaxDsiVQD96/Y8lgJjil9DQVTY4WJ99lH4CYEZG8hBwNzOYfesK4kSUjlwBCbUCiodiTQ03xdxEAWMEtQfRwizEv6LAsFGVNaISXEaFlPctLws9oOCErlZFi5OSDMqomJNBZhXkf5LIGnaPscguPnNaAEH1fFFNm2J+DTiuCt1JD3ewI4yTAev5qx71IKoNAQ/Qg/byB37QTkEkuo+LMwlnLW2DcZygKijxc+JYF+AOpqYhuPPZbp/4EHzGcYDbsmlSLjF4HcjwvwAfAguEeUBUIbu+i0nu7rfPZfApj6K71H7tKMvsosi/0sK/kTkmAf3u3ocZ/zVNj33ivC//ozYCy/8OxPZiyFEGsAfw/AN8Cr5b8bQviHP839+FwDy9OvX4Z/6e//2zivWizUABMUfnRYwweJUllIEeCDwFyPuBtmuO1nOKsP2JkKjTbwQWBwGo022A41fBBY111+3Hao4YKAALAoB7ggYZyC9RJaehxMgYtZi84W+fc+CBiroJVHpS1cEHBeolQO7VBm2W5VcP+cl+jGAlo5jFZjXo3Y9xWMUSgKh0erHbZ9jd5orJoe+77CMGo8Xu/wertAXRo4L9GUvHCn5wOAcWSY0Greozcap7MO910NDuwEhAgwRkOIAKU8pAioCov7XYOydBAiQIiAddPjvqsxKw22hxrWSpSlQ9eWkNEPOZsNGEYNazR0YSEEsGgG3N7PETYllm9vj5VQaA8VlPKx9zfAOQGtPVazHpu2iWoOgbK08F7AWgUhAorCoTtUaGYDhp7jedsXEMpDlw7OKlyc7rDrKgxDgeAEdOlgBo2yNrBGwzsBpacWoaoy6LsS3kjUixF1aTAYjXHkgsi1BdScZirvBERknZWOq+e4D8ELBh1pMruijPKkAFTzEWbQHOr1CvAConRQhYfdM4AhVdWEvUZx1kOpgP6+xupyj+31PE4Z4klUAXAConbAfYGwZLWNrC28UTg+2bJ09PHuCnpZJQcq0J7fI15AHBQThat4TKMEtJ+kQKMk8z5GqZEVCDMHGJmTkVE5iINGSNU78SacABzZ9rydJAN0gj7heGyy5eTHz/k7yoBD9tQCYF3QMsq0nWAg1kEh1fqEMqoDnIAw1HSFMuTAqqBDlniHuYW612QvqsD91CF7Ucky8jwIK5ge3UrK1I+YbNUJuCWDooQXcA0Xu6qVDIuKcmtIytPJ9HGIpVtJj2scKgUNuJL7kBKnj2W6ZLJCltAHyfOpDlFmbQg4k3T0WEIKIMufg6LvNIBS7HFNZUSxo1e22MdgJ1YdcrgQA6qOJZt5WGcmFlal9PzIniMcPU5yUMALEVi9E+ttUpdwGvRIw9dURdl+qt9JlSk+VXlYstE6hlYlyalwPNdpe6lCJygOEVyF6FvD5P2rkGs2gkb2UQLIydMpBRmI7KyYJNJJAp4GCFNw0CRfNUsGPqVqFT4RQbMvpyFaen7h47mKQ70kLU7ydYKuifnMnlI/DWUgkJO2c2JzPAY58lh99DQnm0C6jiRZeLrv8fOn85reA6mqIyVUc+gbB2lRHaCiXxLgIMQ1IktF7XwaBB57YpMcPEvwRw4aErOfmK80jEiy2OPBUPJEJ8adnaZR2lpMScdJ6YCjz4xMNUbxdfY6+j5jVUti1vnZmyTRrkxMKs+NrQmSyzbAxQT0FEZ0bA/Ism4xscdUlpClT0nBSS6bZK/Hkn9bCaj4mhxLiNPjklw4nZMk03WlyInHSQKdzr/XIkuFMzuuwA7gcep1TecuAfs04OL7IDKxvX+Q7puOMUlX8/cPkNnIdGwphTdd1yih5z4fD8KPb/mYoqcTcVsqVaFEeS+fjx5JABCGgP3Yh8mBomACbUzf5b7zMTIC1uRrzAm11mdpbWI+j2tChHHwpY7vA5+l0cK4LK3NTOGRtzGDwgT6EhuZQKNS8fmPZKSxQ/PB4xImSGyjkg/B4LEMNTGP6e/e8f+DxwNZbAKbx3jjWH4b75PB50+6/XPgsfwzBpb/PYDfDCH8PSFECWAWQrj/ae7H5xpYrr72KPzLf//fQmsq/N4PH1MuavmFptYj3K5AuR4wHgqoysEdNPRNES8mAW7GRFB1kHCPB8hXVfQJxoXNyqO4jReWAAwXjt619QjxqoKvAvReQhguBsiGCQxXFsW9omRUhux3DI8GPk5PISOUpUU2NS5O9IELSlfyC0wNDD/RG3oMh8cGsw8JRtJCYjyhFLLYCphVTCW9YLVIktD5IqC64fEkOVqW6FWAWbKexEcvnSujjLKYQjrMysNXPG8IwPxjhe6RR3XLNMP+3RGz3y/RPfZoXkocnlmoE4PiB02WJCYpanUr0L3lUN0q6D0wnlJeO1w4zD9WsHNEEMNF43jhMPtQY7jwaF6xoHxckwlMjJ5ZBZR3aXFO5lgOky/Ql3GhPsZ+wVv6lfL5tkD9RqB7zPdAsSdLpQbuR38R0LwSD7yXwrGug4EUXOwULdmf4TTeP75OSarpC/ZgJrlmuUFmvDdfoX+0vEvA4TMLSnDb3WXA4mOB1EdY7AGIh12JZAW4XVeSUUyvf3cVMP8RH394MjGq9ZuAwxNg9ikys9M+Ze3MsA44+25A+5bMtRLjCdl4FRfqumfwixrJhJZ3IvsAgyTjR0kyq21O/oDpr8D0nlRDyJUgKa1V9chVLAz7mKok9IGvQXMdZb2xLy4l70pLlnj2ggvNLqbN6gMXGIlxT+xT6q5Mr43uYsroIQbg1LF/M3Bx2T4RaF5zn3WPHDrjiwhCNHJQS+qwIwscciVGnroEMtjHC2CzZDWI6pG7DxPrqwaCuf4yVp5E1hKYgEQKmxGxCoPycYL5Yh+4+IydgnYGLH/kMc4ZbtOfxQV3fB59YK9k6nvMQANcpJXbkBNa2SdIZYJK+9axB7DaMpU0yZhT/Ui5I6uauiATm1ltAop9QH8mUezDAy+r18gstO55HLNrH/sSQ5ZgNndkObszmY8pMXH0vfF9y9CUSY7qtaBsNP5tXIrM4if/JZBCcoBqy4oC0zChlceHvOCtdh79mucggSQzlzANg3qGE8lr/zhVoQRF76ONvZlBANUuxNcpoGw9hpWKFSNkMss9zzW9hUlSyWCY1BGaQ2cUpanjgkmxrpjCUsqdg6simyrJWqYanezhLBi0Y2uB5ScW/anK55feb4Fq48AKIwU1+FxXUt07jEsVJcW0UiRw74uHHtrjoJ08oIqACCFEeaeFbdQkzY3BLww1kvlaozuXk05TsE7ydvqC0tVy6/he8/yM6IND96hE82bE/u0Ks5cm+qUlyo2BWegMMvIxNhL6QNmsGjzMkq+TcJTZElArqM5DOg8bOyaDFKjuBvSXDM9Rg4NIvY5LpoYW25Fy0UCW08cuR9ZXeIgQMK6rXEkiRw7KVMt6ELMqUN5xoe5mmn8H2Ie5HeHmRfycBcAF6P0Is67JaNbR8+kD1KbHeDWHGhzkwcCtyuyTTGykHCyClDBnNcqbDsPVDMX9AN9o6M0As655Tg4jXFPweMdUf0LQ40sNvelg1w0B20C5qjBkM92yguoMX/PBTZJU6xGaArIdACHgmyKDO19qhgENJgMq0RtkD2JkUUVPVhfWsV/ysyCtLB6mrsbQGjEahLqcGEZjed/RRHbVPPRMfja8JwTKX5NPM9WPHAPLugK6fvr/FM6Tqk+6no9NDKVzgNYIff8QVCaQqPVDKWraVpS08g0aAaQZ+a8UCH2cWiUw6/2DVNnPgsc/FZjM5zhwG8PwJ9/3j7n9eQCW33yvCP/gZwAsv/QnAEshxAmAfwLgi+FnCP70n3yXP783LT2ez+9wO86weavGSdXj9Z4snhQB4gz4yvoNrqod/qfv/CU8e3aD6/Uc3gtIGWCNgpAB3kn82le+j398/gzGKtSlwZvbFdbLA9qLCnZUKGuLVWQZn63v8f7sHM5JhCtAyoCTZsDtZg5oh6+e3+GH12fQhUOhHHZtDddrPD7f4o1YUalRWHgnYQeNrz5/iT94cQVdWHijIUoLrTy+dn6D3399SfZS8AvTOoHZqsewKCGlh9mXkI2FUh6l8hiaGvVpj/FCYzHvEQDsXi8AHSBLh15V8EsyPKGIY/0iMmyVw6hKYD2iWAwwny4wNg6wkszYVkOcjlgue+yu55ifdQhPAbep0VVkqqrFgO4pk137nx+wWvJCtz+tuAipPMTcAl7gcCqwvGixFyuMXx7hO345Li5btCc1k3ADIPrIwKmAw5dGlMsRbdEAJwZhUHCPPPrHXMSImcX4SELfFtm7KUaJMHMQmgm5xWzEcCihaot2UTHgaObINA4SrpFwjYdrJIveTy3URsPXHjgx6EOF8cyheq0xPBuhbgq4MwM4AbVTCI8GWOVhPpxBvHPAftZEpseTWXQC8ALDY0D0ks/Z8aPYfKIRVgbWCbiKabnFRjGQ52xg4JQXSAFT+3ck/LmBuCvQASg3EuPKc19lZGw8oA4S4p0W3V2N8pbP52cerWCvpXk8whgJMUiMKwk/8zArgWJDZGJOHOCZwHvzDQF7OaJ4VTDV9UmL9tM5ynuJ8ZR+U7d0UDvW05gV32P1JxyGbL8Uga8KKL+wQ9uu0F/6zND52qN+pdFfWZS3CubEo9hI9FdTyEuxYChVCnrqLwDfeJgTsolMNyZYlwaQg0D/bIQcSlZfvG1Q3OnoYQuUqt9IiMBUUncxAoOCaiXGE4H6VqC7pLfNzulRlecidmty8IOg0V95At+NxHDlUGwkzNojiID6tYaLwVXCA+bUYv3tArt3PYTnIMHOCaaqOxE7OYH2qYduAXPqQQ+iYtJpHGal+hNXk2QeT5H9kkGxazYDpAqZkfSai/HuMZlSJgLTQyudRPskYP6pwLAGbC9iumhAuAJc5eEridS9CkRfbExvHM44PJFGQMRhXfNKROZIYVxRxtc+iYv4KqB5TTn1cJZqe2SWfI9nHnYuUV9zH10tclgSBBlA/xJonwKzTyXGNSAtE0d1x4oW1QHdlYrdr2Q+7QyYvQTaJ0B1L/j7LcFTfRMrWXoQ3M+5reqOMndgYoC8nu4vT+glkwY4PJ7YE+kmAG5uFfpzwL+S6M8mCTAZckAOldoAACAASURBVMlKkB0yQ5N8wbsZh115EBEHFa4C+nMdmVw+ZjgTMFtBL2kcUuqDnLo3GybPmuXU1ZnYpnGlKAnXPG/JZ2ljDyeg4QrANdxWlSTvcciljIatKa+ub6fQFFdQ6jmcyMjsT15W+rMFmtfxvR9Dq1IybdHGALT9JOmXI6L03GcfKQc1Oqbi8vFT0i+HCcOKclo1qsyIukqgu+B3TnPj0Z0z1MY0iUZE7CBV6M4khC/hSoH2SZGBqKs5HHBlrMo5TDUow1pz0BH7LSGA4uAxaJ37RosDBx9mJlDtyG5KF3C4IOhubmK/asXtCA+0j2fQfWACblnwuGaK5yf6ecelgjKRSYvsYVkpVqusFYQrMK4UEBlK3Xr0ZwpVrZA6fFMNSihqdBcFpGMCq624zbqUGNcarixQ3WmMJ5rprwcPXwnogwJQxddEAqFBf0YgaxasO2HiLK8h41pDDoF9m8ZjXJeUA5cEMOO6gOo9MCvi54uBQkGAHkzjEVZVZl31foRZlihCgK90rkoBkP2eqo9yXyWg9wq+omzWlwT6stSsKmlHso9awqXtj3wuoSV8pSPjqSAPI/yyJjj2gQA4MpIoVGQji6gMMshdlABBeVNmSW4QAqGKabKOSa2hLAhMm4qSVOcQZjXlsN5Typs6P6uS+2gsICjDFqnUN1WDpFsRfx/BdRZWlGUErQ6QRQSO0ScaAkRTk9VMwUtSZn9lgqniM5hGpO38hBt9m+In3ufzcxNw+Jkcy4UQ4h8f/fx3Qwh/9+jnLwB4A+DXhRC/COC3AfzHIYT2p7kTn2vG8ovfnIdv/Hd/G5/uV/jm+QuYIPGvn30bhXD49U9/Fb98+hH+39t30NkCl80eo9P4aLPG8xPSNjZILIoB27GG8xK///EjfOXZK/zKxQf44HCO3/wnX8ff/CvfQiUt/q+XX8JhKNEeKkjp8Veef4RvffoUX768xvs35/jX3v0ufn93hVeHBUIQqLVFO5ZYVAP2Q4Vnqzv84e0FfuXJD/F7myv8tYsP8A/e/ya08phVI1blgNEraOlx085w93qJd55fo9YGP7w+wxcvb/B7L64gpYdSlKgumgG1tvj4k3P8whc/wXc+eIKLyx3+lbe/j//z1Zdxt59hHApcnO5QKofbdoa/8ewP8b9965uYXx5QKId5NWLXV2hKg03b4OnZPe67Bjc/PMUvvfc+NmODN/s5ZqXBuye3+NbHT1FVFt+8eoFvffIUF6sWL65P8LW3X+F7P3wL67MWPgg8Xu7ww+sziO8v8Nf+1W/jOzePcVp3+L0fPMG/8N4f4lsfPocbFDBIzK5aHO4aLC9afOPyJX7rg3fx7OoO277CX7r6BD+4v8KmqzEMGovZgH4s8N5bn+K3/umX8Fe/8Yf43ptHBNCvFlhctdDSY7dv0MwGvH2ywYvtCidND+MlLmctvvPxW/CDgmgVHn3lGoPRuP9wjbe+8gYvXq+xXrfYbGdYLHpsXy2AIFCe9nBWIbyp8Mt/+ffx2x88x2rVYbtrMF/0OBwquF5Dlg7np3vcf/sCl7/0CtuuxqGtgABUDVfg4wdLrL52i/v7OYKRKN4UTL2tLX75Cx/hd/+Pr2F4YmI/J/DkS2/w8tuPoN/ZwzsJ8f4Ml7/8Cq9uV3wf7/kFUJ4MuDjZ424/Q/inK+AXdhheziCcQPGkRfV/L7H9eQNReqzP9jj0Jcau4HCljV9sKqA67TG0JerFiPXigDd3S+BFDWEF7BUBFzxQnvcYb2umDM88Lp/d4c3LExTzEe7FjAx+J1B+dRvl0xZ9V8Je11g+22L7ZgF9q1F9ZYvDiwXKywOG2wbCCpy+c4f2WxcY3h4phxIBqvTwVkDclvCNj7JSdnjCA6e/q1D+m2/w6qMzQAaouaV0+aYELgaoj2vYlUcoPMrXGq4JDJMSBIfV0z26XQXR0j+2eF+hfdsjXIwoawP//gL+eQd80jDx2QmEsxHqRQV9IJg1J57y3JXB4nsVul/sEF5XqG4lhlOPUARUbxTGU4/T7wjc/OqIs39YstPxr95i/71TSAOMj+yD+h1hBJpPmfZsFlQnSEM/uVoauJ5DGVgOLWYfaYwnAboX3O6CAw19q2EvDPR1AV8GLD6UsHPg8AUDPTfARw1srKU5+a5C+zf2KH5nwdqapy3Ed5dRMosYihVQv1YYTj3W3xfYfDVEL6zIKcgM3QH6x5bSYA0OIc4cpbcR+NoF/d/jqUf1RuU+XABwc4/mhUL3xKH5RGE8jYqHxx5nv0uQsfkycPE7Aa9+JfblPraoXhRR2kwliI8Jx+UO2H3RoflUof+FDvJTakqrW0ppXUWwe/J9AjOzYuhRktH6MqC6ldknzZAvMsf1bcD+ObdT7Ag4gxQ4PA5wDft/2ZkrMJ4QyA5n7Jy1M2Bc+1hPFNC8JpAbzqhOQGD9zPJD9m9uf3HA+rcr9Gdk7aleIUisrwPap3yMawLOvk3/6f4dj+pOorojyz//lEyxWdCf7aMvdlgT3B7e5vkrN+wftTXrXxDI2uuOvau6i0xvT0WAcNyPcYkpsTlW3egDYBfA4kce1Z3D7c8XWZlhUp9qlCaXG9ZuDKccPKghdpcOQPsW1Shy5ACivo01K4qeZ4Jk7tvstcfhSuZu0jL6cVMKqqvpCYcA9s8Ezr/j0D5WqO7pB+3PJBafOmyfRzZRke3XsbqHYUTcDsLE4IqoLEpSZrNkOvHstcXmCwWqjce4EJi/9tg9VTFwi4y5GgPax9xeOn++YEdptfE4XCqsPhrRvlVAWmD+YsDhqoxSVmQf+Pylwf7tkgx2Q49rd6ZYQ9R6mLnE4kcjxhMN1ZPFK3dkX9srjflrC1dKHK7oYy7agHJj0b5VYPbGwswVxqWE7gPqG4PusoDu2CnaXmlU29SBytTd+s6iP9MoWnqgq42DWSjUNwbDmtff+UcHdG812SO9+HTEsC4w/7jF9ssLLD7sMJ6W0AeH/qKIn18y6QwukhmgJR98sbUwS/pBq5sBrtFQnYXsLEIhYdYVJbegZ1cNPntHh7OKQLm1cLWC7sgAq95G/7sEFKtSIAS9wJ1F/3iG5kULXxKoqgNBaUqpBSi1lYOBPWkgnI8hbjIy8bH2ZtMhNBEMpuTgQkHuevhFBXkYEUqNICVk22fPpzgMURrLVNkHPs6mIqsZWdgssU09mcYiVEWuSOGCI7K0VUmvZ6oRkZLsp1QIfU8/pz8K6Tn2cQITeD5md4/TcX/cLYJP3/c/+X4/4fbng7Esw//yM2Asv/rsJ9eNCCH+MoB/BOBXQwi/JYT4bwBsQwj/+U9zPz7XwLJ+8iw8+i/+I6hWonktUewY6hFklP3dUfbVPqVMT3VRshSnk0ku6MooPYsei9kbh36tKMe6o5Sl3HOCmTq90gQ0qKnvjNH5QH/BL+AkL1KGU0uIKEuMXw7lZpJaBQXU95RHCcf9TL6FFM6QUgi9AmY3LOkdlzJLg/pzieaa4RrVhnLFYheOotkRu7F8fNwkr/QaebLXn0lUGz5ntXXozhTqO4/9ExW/4PnFGCTQXHtc/5LE5f/HUAnbUMrWr8WR/IlMRpJxFZ3H7m2N5ScWXgv0pxL1LXvaZtcOu7c1Fi9cPtem4TkYlhJ1LBEvWn5BFZ2HrRgaMS54TOWeHWZe83VL561oKQNLEfPbdxRWH3mUG4vhVMdJMTB/7bF/iwsKryl7bW6Ymtg+Ulh+amErTpdTBP64oEys2kTmQQKzaxc9RSzsru84DTxc8QuVUloGcDS30ZPXe9z8fIHmOqDc+Rj0wfdBUOxnMzOJcsdEwVTNYBuJcmNj2p6ErQTqO4f7LxZQQ8DsxmH3RGP+mmETKUa/2DGZtz/ll7nwQHVr0F0VLBgfA8xC4nDFL8L6nq+lbvmFa+aShfcbj9mLAdsvNlj8aMTmC2VO8mxuHCVLnu97ACjvRoynJdorhcWnFmYhUd1Z2Dlft+Z6RHdZctpdMHGw3DoECXTnGvWdiwmJInt2ds80Vj80cLWEbh0Qkxm7M4XFC4PuXKPaOqje43BVwFUCJx/06M9KLoJi3H5QlIsdHhXwhcD8paVM9ESxH29N1sVWAs0NQ0BcLdGfKtT308RXDiwIDyr2390xfXlYq8gweOjOYf+khDIJ8DguvG5NrlFgHYvIUsHqzsCVrBmAB8rNCDtj56yrYhqkYsBOCv0wS80C88Gj2BiYVZELzIMUKO9HQJAxaF60GM8ajGuN5tWA4bSENJER0nzO6tUe3dMlghQoDhb9WZmvQ7pL7zGHcV0wFCUA5ZbnMfnlir3BcFpRomgCit2I/rJGuTHorio0L3sWtyuBYmcwnpQotyamYyIGlxiYZQEzl5i9HDCuC1Rv+hwY4upY/G54Dso7Bp24SkL1PP7ytkcoFOxcZ7mgsAFqPyJUCnZeZHkgJKKH66h2pJRQvcuhL0HwNfFawlcKwykZnWJnyTQggXL26CVvoFkWqG56wAXYkyrXQfhSZlZHHxzGkwKuFtCtR31NGZtrGI5mZhq6c3khXN4NsMsS6mBzr5/eU2ZolwVU5xC0gDCev58VMKsySy5DIVHccRFnF2X8N+4nuPill9FBdAbjoznZod2IcV1BOqaNqk0Pe95AdhZydBjPGuj9CLvkcxb3fUzblNB3B/hZCUSPnWsKFLcH2HUDvenhS40Qg2RSwIuwPnvY3LyE2vU5mVMeRkDKmDpKP5uvNOAD9KbLj5GdJatTaojBwJzPyGjFVMygJdRh5AK+UBDdiP4LZyhve8jO0LMHkMVqCkorO4PQsH4iCAFI8PFVwfCcmLwq2x72fAH9ZktmqlD5OiIOAxftZZSkRhmkW1RQ9weEuoDoTQYdqVPRzyvIbQdIAXs2h369hV809OjJKZjGNwXktsuBNJBgCE7qRJzXQPT6wVgI6xCaCn7G0B0xWsAyBIifHUfZ5lEVhp/VkIf+wc+iHwhOYlCN3LXwJ3MC8P2BATjHvY8VuxnDrCazdyzfFILhNlLAL+cQ/cDtp2AbgPLVpgK2ewgp4c/XkIc+A61QFpCb/bSdQxdTUAXZwF1LKafWBEFFMYEja3P3Y5g3EG3HkBxjyFIu55SihgAUBUI/QGiFYB07H+PzhmNpazjyJKZtCokcdqMU9zF2RsrVEn674/tpMee2rYWoax7DOAIyvq+iRzG4KIk9rjyJ4E2U5VHQz/T3xD76wwGyqfkcRz7OLIUF8rEFYyFyCNYfAx6TrDb9/0+4fd7rRr7xXhn+59/46QPLrz//E4HlYwD/KITwbvz5XwTwd0II/8ZPcz8+18CyeudpeOvv/CeQ65HScCMBKzH1UALCSOiDgLmcpvQqBlD4meME/8xCbdlNaRcOUAH1iwJmMcnzKK9jNUb9UmNc+ehZmv6eYtFz2AIAV0fwWAZUrxTMysPNPMpblT1SyRcpYugC02bpu+wvKS30JRdk+iBincDU15jub+chetjICqie+9G+ayE7BpGEyqP5RKO/8AxasOIoYCHA1wHVa365DBesHHENvZepomPxocTuSw71S97PNon1QQ7eSGEnqmc1hC8YRpJCIopNLLpvJslcEPRwFjsZQVaqeRCUFd5SPqU7eiuLLc9FdUdgf5zAKzwDVew8Ttu3lDgWe5E9b6mEPrEHCDE191bCLsiCLT4SMCvkUnt9oF+x2E9yuqQREZ5yQN3Sf9Y+BaQR0C0ZBwZpCFT3MQXX8LnK+8nD5yomzQ7nIfr7KFus7tjvVt3Ts2QWIldt1LcE+8dDinKDHAThakSGSzw4V8n3CZDdSN7B4965VDejRrIbTKpErrZQIzB75XF4JNFfBCw/oASv2PPxZhGHGiMlnbNP+dqmCo5iT2ZE90c+RDAtVY5TSqtXPOch1mqk8616YFzz53JDZszM+LgUuOFjdH8qkw+SCb6sQeG2hWO1hm6Rvan1zXSehaPcbzilh67c8vG2iQOjA+V5kDF5thbwsf/RF0DzOuSkY2FjkusIDOcC1S0/P2bO941XlBCmIVS1CblDM9WAJH9Z6lMUlsOx4VRAdZQLpsoSsxRZrio8GZ3k07RzeiK95rmgXJLpxrPXHu0jma8P8KlPkPuR+1olt1+2AcNSwMQKi2LP1yB5YJPkkq/b1MvoyiQVJABlEXtMrRWRPTuRR95DBpSU24A+SlWlCRjWEosXZJTSvqWKkdzxGD+r0sb3tgsYVmSLhpXMQSRqCGRguskDWxy4SB3nHBamQZwyQL+m/zL5SocTprHO3rjcGeljBYYauK/VPQd86VhTMjR99lPHYwpVAYBy7zHOZZRZehzOVU6Eru+jX0whDvj4c/KX1hvui6um5GL2mHKAl6pX6luXvY/NtcF4otGfKOhhGip1l5qv8YHDGFdNnZFyDOjPFNQY8jAvncvks+a+8X3jY7hNuSErNqwVJaKraZvJq8haDfoe5egxnnDIk6plki829Wum3spUw5ErSTwHrGZOryO/NySKnYMao9wS7I50lcqeaF5PY2ejC9knmas/Yj9qGoCwfiLAzBVU7+OwhRJPNfhYTxL/HX0M8YlDHAGo3tFLKcGB0AmHXUUcsokASMt6D3ZE0m/oY/WGnXEIU9xzwKB39Ewm72xKeLWLAtV1l8+PLxVUZ+DrguxcfJwvZGTtDIcKhYTsHaRxsIsSuqW3MSh6Kl2t6fk8WPgoPU2MYJKU2kUJ1fO+aiDIcbMSaj/CJxkyQFaxUrmfMtV9iNHCNwUHSJ2FHC3DbyQI6Kso11SxCzOF0qRwIEQgHUL2XQJ4GOSTBgNKQHQj/LyG7Ecc+xJDoaMXODz0WRYaGEae85pA+oGv0rqpNuSzPZQhTHUjQ5TJ2pi8WiZje7yoGTtVmKRAnuP7p33NPZNkKnMqbLodMYkZ+H4msCfffKB/MgFfKSawGmW5UJT2Pui1PA4DSiBSyIf3+Um3z3kq7J8VsAQAIcRvAvj3Qwg/EEL8lwDmIYT/7Ke5H59rYLn++lX4tV//W9iaGtZLzGL6a295MUrproX0uDs0eLza4WAKCAC7voKSHlp59EZDS4+nJxt856O3UDWGiaBBYD7v4YN4kKiaZKN1aWAc00r3dzPM1x2ECBgGjeW8x66tYfsCzbJHt6v5eXQC9ckApTyqwmCznaMoLYauALzA/KTLj+9juusw8F+3L6AXBnZfYH5xQHvXQFYOZWUxtGX0EApIHeB6hflph74vKJNLUXFOQBQeQnmEwIRTIQPK0kLKgK4tUTUGSnnsb2dIyY1QgdelXlF612ruy7ZEddZhuKtRn/XorxugdhAyYL7q0XUl3C56EOYWi0WP/Q9PEIoAdTrA7gqg9CgaA9OWQEzxlJ2iTzDehBMIBWWGKDxgJdTSILypENYGIUoAhYmL07MBri0gughG3zrAXDeQMdQJhYdoFUIZKOVsPGQv2SXaS0osKw+9GrmPYQKQD1JZo4cxlB5ilBCG0r7hLQ4rAMCtLcHH6wJu7iHWI8JthVA7+kdPR4RBAUZAxjTVIAPCwkG0yV8KwAHCC/iFhborIltCwOhqDwR2jIoA2LWFvtOUNVoBtZdMMZ05DlFiyJWPIJygnIuw4l7moKNiK2CejcBe83kOAubMoXqtYBa8PwTreEJBUK9bAXPqEWSA3iqGQe0Eyp3AuKQcUMbtu5rhSnbtUNwrloIbAIG1NDKmgaY+SWkExjN6F4MCzIlDeauY7qk4OBAxZVR4dpKyn1NkqR2AXNGRuwIDoPfT0EaNfI5Ub6FGAbPgsMc2AamHVB8IflQXZY276BONaZVyJKM6nvL1SYvTohXoL2LA1TsWai9R3TBwSzgB1aUAkTgIq/jmS6msTI/la8/O0fjWDBNwTbcEsuySALW6o08x3d+V3O8knUuA3M74Wo6nfP10h9zPOXsV0F2IPETzGux9HSYJo+qjOqPCFFalkJ8nhT0VBw4DigNBcrkhQObnfkpTdeUE1IXn71KFUBpspeCjFLSTXufUn5n8iUn26UoOZsxM5JoiXzAkqrpjCJXwVKV4Pb1nEqhP3kUfk2RZNB+HKT2HRSmRVhoOkOibQ+zXoyRUdRyyqZ4STjuLao0lB1NB8djS8U41QwH6QJnocVLr7DUVNqm6QjgOfFRPQO6LeE4Kkfc1JbyK6AVlf2sc8HQhJ6WOaxHVQBwO+DTAia+L7mKAWKDKRh8IvtNrkK/pISaJDoGDqxOCtWI/pcimJFuVknkFjzFV4rBmJ/oto4pADxwm2lrkiiKv02tD/14aFqVaGlfGLsWYbCoNw5TS9UEaBnu5guqYVNOSAC1VRSEDwlTpkvap2vncnVl0rCUZF1TcuELkf9npyW15FUH0jnU1tmbYkhxZv5IUPUFwO8KzfiWBaw6iPUIE/qm+SDr6OauNi9dgmZUEHPrFOpoIeOm/ZQgh3yf0eNqa+2YbGQcK/L5mHVQaEnEAn6pcfCWg944s/xCrY0ZuR5oo74y+d2lCVgMI6+lzFFT5qIMFkvwUmJjlEBAK+SDNVg5UE0ibfLjxb1GuGpSg4kDQJ5kY8KDk9DghIEd+efhSQ5qYIqtkDiZ6kKiasJJhRYqvYt3a4Nh7aT1Crfn3BGSFgIgMa07FTSE9D4DYZ+SkQkzMbHh4vwddl/FvfK3D9Bj3Y0AlgNy5ae3EmB7fPgNAM9gzdvJCCkmmNN2OgOYfuf0Y1vTH3UIIn/vwnm+8V4b/8Tcuf+rP+wvPP/3TAMtfAutGSgDvA/jbIYS7n+Z+fK6BZfXu0/D2f/UfwI0SclMgyAB1kGTRVg76XsE1ATgxkNcFfBVQbCT9P5qL6pQ6Oj4xqH5Uwpch9mGKvFCVAxdzbkavkz23KN7o+DwspPex+1J1It6P+1js6ZsJMmA8d1CdRLGRcPXE8tkZg0PsnF8OuuUXolmwJB4+LmyXAeW9wOHZ1N2YvmQzcxEXekx9DDl9Ni9IxunvviSTkErnXc0F5rhmvUOxn5iVxHAFFTCe0gPkSjKBQQLllovIcRUwexUDF0pgPPGoX/NiUW65oEvno74R2H3FoX7BvjobOy27S/qn0pQ4Rfv3Fzx+s4p+pMjYJYbOVWlhmrwsmICKRK4RSAvf/pyBIvThEOzoQ5IzB5TbmNpac7siAIdH9DpxERpyyIZZxQXQEFmzOTtHxxPuT1ogmeW0SDx+HeobMmvslARSVH9aPIuYPunLtKDk+Zp/zB5INfLYjuPZ7Syen+gp2z8LmL0gY8uET7J4QQj0VxGABC6ihzX/1S0XS8Np7M+UZBXNYgIawylfj8SYsVeV21ADF80AstcoSdLHpUB/GXD6PXqdIKYqh3I7+bbGFc9h6rQEyCwyLTV6ywzPQ32NGCCTFsgid6T250yFDVrk40vH7CtK513Jx9jZtLit7tnV2T6WmbFOlQ7lJgbgXAmUO+QFa30T0F9MtQmunhjQrGpwPM5xKXLfI2spKJPnApOgy0c2yZcx2VZMAT0qbmM4IfvoamQ20Gsu7BEl9K7mzwzjSP9yIdfckAG0NZlTOwNmbzzaRwrVxsM29BuKyJzqPkTJK89VSowt2tgLGhfrqUe0uSbrlK5Z1Zbsm50l8BZ7Igf2Nlb3lIcnG0EQlIwPK5GvF/Vt7FttBOo7DzPnv+0VJev9iYpS8QjsosVgOJFobph829x5dKcSyvAc1nc+9mZSbq97Lp7T+0F3k00hJT0ndUgaKDSR9RtWEraO7PyBwCRIoN54HC5pXTBzifrWZca2OIQHzGwK//GxxzQxlBCU6APIzOa4iIv8yJrWd7QopPRW3bF3EwC6M1oLbC0yIEtAjR9YPm9976EP7KtMtoL6zkUGVmTvnCspwQ+Snr7ukr7f5BccF5SDBwUcLjTqDWXfphGYXfP5bM3k2LT4D0LAziR0R0uD7jwBtqYNpL7lAl+4kKXgrpbs44znLrGBQbJWIigBs1ARBPO4dOuzVDxtz8xVZktdRfDltUB5z/7RckdfIJNc46LdBpiliooZLrQTiKJaQOV6ESodaPlQA9nX1HU5npARdpVAdW9iqA2gO0umTQm4SgE+gS8fFTiS94k2ixQoM5xVKFrLz9dhYgeFcRhPKxRbA9fouK+Ox7HSKO8j86oEdO/IOPYW5rSO59Oy31IIlLcd+sdzCOuhD/QsZoDlaDVICa92VUHvKIXW9wPcggylW5VURuwG2JOK2+sMgpTsnxwsA3U6+0d+Fo7spC9k3hYwMY/CxITYyD6GmmwnhICblRAhQO2G/De5jV9cx92QKWHVOvhZBbnvKFOuNEQXGczj+xd6YgwTwIs1IUkqnMN3InuZwWIClXF7oakgumFiIY99kSFwW7H3MrOZNvoWpcgJtJlJBAiOLdlaSIUH8tskmwXituy0/8DkzQQo7VWKPydfZZLHep/TYNMtfIb1/Ge6Bf+5l8L+wntl+B9+4+qn/rzvPf/kz/zYgM85sJx95a3wi//tv4Ntx/G8lh7LeoCSHp9cr9E0I37u8hV+59O30e8ZRmKMglIezqrM6rWv52ThRgl4gcXZAUNfoKz45lXSox8KjLc1WamzAbp08I4XCrsvsLzaY38/gywd3K6AnFv4UWFxesD+bkY20Ur2+5UeQnuI6xJ+ZbG6aNH1BcNhYu+ivamhLzo4q+APmomlY6RWFP9fr0cGtzjBAJzrGcQocfHFW1xfLxGMnJi2xDzGBFh9q2HXjH6HJvuG2uUkTF8FMoQA4AUQuxtlR1YJ6xHqVQX/Vg/9YY3xykLtFMpbie6pJQv5uoAaBJ789R/hg+++BZwYqJesaQkqQF92MLc1xMwh9IrHFYDytcZ4wX0Tc4swKMiDIrN1Ei+IIkBvNFY/d4Ptd84ZSHJbQFjAPhkhrwu4hSe7GVlMCMTYTLKYxT0TO1XsEQwXI3BfMDlxZIhG/UrBLFlNow4S808ENt80THMViImyHnqrMjOSuuPGc8fz7QSlyJLMZnmtYs8aGSEI3lcdyMKt2U8pUAAAIABJREFUvyew+wJgLg3KFwXM2qO8UQiSDJXugfaZQ3knYVbhiN0C3IygPwHR6p6BFPt3gfX3gPuvx2HGnB2KxWZiOhhAwkVzeScRdIBtQFDtuaDdP/eo30hKiGcElGbOIY1d8tjGNetg7DzkFEzgmMkRGC4dyluJYs+0VSCyXIOAraMk+w1ZVkonCayGc4fqmhJsXwTAx+GHA5qXBKppEAREkHctMJ5OA5KUpGpnlHGn0JThjMMENXC4sfyAg4FU1yMsZd3NK5E91eMp2UN9iEzbWmSADCD3UBY7Hp+r+ZE8HkboA/d/9w73hQvkyPjF4UB1h9zdOJ7EAUkM5qD0WyB10UgjUL8hq8HpP8G5XTAMZjglE51qXPSBz5UHUWMaRgTsnwKzVwLjku+v+gYxHTSyU4f4uQKHFK4SefjRX1BiXuzZU5gAVnofuIYMp+4Cq08EWDu05vlJ20gyXt0RmCcwXrSx3qUFZtce+7eSlJWSbldRguxqqgooX42XNM3BVbllVYzw/P/klwcQ2SIyWiYPaUQ+XjVE8D9gsj5UQLUNlBA7DmbqjYeZCQJkIIPkVI/S3Hr0axGPIVZuHMhaqgj2hsgSJlZVBIJ3GzMFdB+mblAR738b0z/3BF7SsdfQLAXkGFBtCeDrO4bTdJcS5TagvnUw89i9F4D+hJUwxT4yjmI61uaGA9XhJMpt73hOxoXIuQUJ2KshDvocZbz9qcT8pUOxs9i+U+Vz42LtUpL/1jcO7Vs6Ph8yw9hcW+zf1rlnNMl7EyMnXMgMZ0iDgkrkepWim9Y+Lga+2IbBOGkwMy4lqi2B1rhSKPYew1rBzATKNqC6d7AzmcG4NAHN9Qiz0LA1Qb6ZRVbNEmC7UmSPdX+mUW55vps3Bu2TEsIDzWuDoOkNHlc6M6i65/Pp3sd0V4nZiw7d4xr64KH3BuNpSbnszlEK6wJUz4qTYu/gC7KAtlGwM/q5XaUyIPWlzOCMAFigvKVv2c6ZIl60Nkq1J9Y0aDl5kROhJsh8jusC+uDiUEk8CMCxjYLqmbyapLxBkpV0M50ZSmk9XM1aEt9oBCXpvd+MMKsSXgtUrzu4ecEqlJTWasjaBpGCdsgAytFmya1wDmKwsOf88GfAOk4ADFJOHZrWQ46WPt7oaUUK07lv6ZPVEuIwwC9nkLsDQl0hVApyPyDUBVInpXARhBmLMKsghgiIa3o4hfUM5Nm19JceV5IIMXlbo384+1wTUM1fwHICewBBp1bT830WCyTwmepFkoTWe8CHSSKr1OT/9DHw5whIBufycwfnCEDTLbGUPwZ8Ht8+i1M+74zlP+/A8nNdNyIAzAqDy6bFq8MCq3LAvBhQK4vqkcVmqPH//OG7+Pqzl7ie8YLRG43z+QHGKfzak+/if3/xc1i+0+Pli1N88flrvP/BI7xzeofvffwYB1Ph4nyHi1mL73/8mAufqw7mUGB12mKza/CNtz/F7378FMZoyMLjYr3HMNf40tk1vvX+c3zt4jV+gCu8c3qH7/zgKZaPd1jWAwrp8Xq2gHMS+12Ny/MdXn10htlli74rUT06wBqF9UmLrZxBaQdn+YH0XsCPErPZADnvMVqFbl9BLSzcoBCCgCw8ROFRVgbjUEAISl6rymD3ZoHFz91BiIDRavR9lHoCQOkhTzyJryCglMfYllCVw6PzDV78wSXO373D+azFD+xbqCqL6hv3cH9wAjzp4dsG6mREVRkcnIDfKbz/8SXUxYCytBifBoSbEvACZltBLCyqxqA3NdZXO+z2DcYnAUJ5qBcVrASKdQ83UxA/qoHKQRYewQv4mcW66XF7YTA77XAQAIxE8bKEf95BAfBWoj7p0bcldOmAD2dcaJ87FqpfdLBuhjC37Cg9G2GKAnJpIALQNRqicpAqwJYFNhce5XLE6GvU5x2G1zOImcX/z96b9Ni2JehB32p2e9ro4/b3vnzZVKUrTWHhAtkyMESCkpAQljwHJIb8ghwwsQDJAxCSjSxgwBzbPwGJCQOMbbLkqsqmXr737rtdRJxmd6tj8K219rkvs6pMkVLmK3Gkqxtx4py9127ixPrW19nCE3wbCXlQ8NcWsnJo/88Gxx8MwCJAaw87KbhWwl+yNX2qFeTIxYbQS+hHHfb9Anbp0W57jPea98JxCdd46IPMCXcApZGbfwl8+B3Wh9w8/4CvPjuDGBSDq75tcJwkhBPYfYvg1twYQASo9wStoSALoHYa8mkHc19jEohpqwL9K4PqiwLdcwfZS/SvJjQ/LWHXDkNQcI2HGiSwNBhbC/lQoHthAB1QflnANQHt56wLMVsHO0qEwsMueBBuRS9vWglt3ggcPvEYf6tHuCs5DoBS2TJgeMxVfdVJ6IOEW1n4UWL3PYfijsC23EnYhrLb3RWrTxA9jpR2CfRPHGQvcHjJFFFfefRPAooPCm7l0N/o3IMaVIBfAa7xOLwQaF5LHL/FBQY5KrgSuP9eIJOKKPGMdRa+iJ2UhqqDUARII2FWVAb0jyhXps9LYDr3UAPBq2tiJ+0ZGf/UP2tWAebSQt/HGHdLUGhWXLRxTZS7hlgDYgBzxs6+BFzLewLi4ZrjsG30aL+PiwUlpcTdDSf0buUBoeBqen69BoYrSlxtGyssamC8pJdWTgTm3aPYnxqA7jL2pcaFgfZL9jy6GtmHnbpjh0uP8p7nafNHwN33wI2EBPjpO3YlYFZcyCj2s9pDODKZ04YLCmoQURIaoDsCpbQAIByTTMPbWTZpVoC0TEv1mscwbvmHp30dQbROSo7I5h8DDo8lXPK3LgRsO6eIlg9kWM0KgBBRwcDrG6QimHZkz/trdiumhQ2mm8ZKk6hmsAtuc1qJLIPOyo/Y1coeUrLGSdrtqtivaTnGcUMgGCRwvFUYt3Nnp3AEwoen7Inl5w7fJw3/p78QON7yHHePA+QkcfYHHuNWZJnycBUXOqWCPgIPrzSqe4XxnMB5WnMhwmvAtjJWc2gU3eynpRc4YFopMsAi5JCzoRWRAQ4IJUFkCts73qqcAOtLAdtGX2tUA+iBxzmtJXuCL7lQcbjV+frymvPc2ErA3mhIx21Ue4/hXCLIMoPX4y3rbXSUoTIoTWL1uUV/yfoRQLGH9rLIcl9pPEyjMa0Kylb3DsKTkTZLAUACDcFqf1Pzvt4QKMopUAlQk3mdlgrYEhAyyCrAbTTGtWL4Xa0wnCuUx3hsqdfVC9ha0iNsS4znRax74XvMSqH6QA+u6sn0lj5A2IBxq1HdW/hKQhkfP4dUXhCTRsE1scu0IqNmFgoiKEp2a/4tMCsFOQaEZQF9tLCtQlB1DP+y8IWEXRRR6SHhFgWCljCtzmmqQSoEzU5SNViyvwEo7zxco+CqGsXBIKzqzBYGJeHqeXocCgl9PyA0BXxkNK2q6ClNi1UNgai9XNGzqSVBohLwm5bPWQ931nKh5zgyLMk7ekUjm+iXFT2YUW5Lea9ieJMQOTwJk0FoWacC74HRZjCKIfZdnkphhYh2IRtZzACME8KyjZ+BfmYjATKcpzJbFetorAVkZCmrkiFBWueKkTBNuaoknPhFU5flR/tI4UDAx32Wsa/yl7KZf06wzzfl4cMvB9F/GR7feMbyb/0P/zE6U+L9sYUxCuNdDUigPe8wdFE3J0AfY6wyCDKQNSw9QqfJYln640I5s3Qi+vaSnw4yINQeEAHyoOmz0vToSRt9XQHwNZkU+NkzAUk5GgJZKzHwl114+uJyz1ftoTsmlwYVJ6GDhFs5CCPoA2w8gmbIjm34BxUAvA45eEQfRf6ZL2JpfEN27tSnBcT9lmCFguVxq5EMBwIgR3qCbMOOQdVJ+CZAdhLlg8B44aFGsgLm0qJ6reFKsjVuQTYvecsoqeW+1SgoVQ5Jtgj4MiCUAcUHiXDCdpm1gzAyeuiQ/T5kZOgdcjUrC+TEiQ59dcknR5+a7gkSUshP+SAxbTxZv0AvnN5L2KVH9UHBK3rcbBOy/FGN7Obz0feGwP2EOGnzJcOJfAEkb5Vt4oQsBhilkCO7CFBjlNPGnjzbsk6iek9W1CwDiqPIgUy+4MRxPOc26nfAcDmH6qR7IElAzYK1E16TsUqMjK9mT1x+TxlQ7qOfsOMkMPUfApHhOkYmTcwyPWkjGxGlxq6az4U+YmbxJL8XHrB1nGTEQCdhyYqqkRLbJA1OY01esyRvTp5AFtTPTEeSy6bx2Ha+VxJblpgX4eI5jWFKSV6rBuQKg9RjmCavXnMsvuTz0tAn50qOX0TwmryrqkeW5akxJbbO3r8kMydLg7m3MJ4bacleJdl0Efv9sgctBYsE5O0CZE9sy2so+LGVJZNAlIfaWd7pqpnhlJYMmKtEvk/URFCUmFcVz4sIBJXShTltW2GeoFrKj9OYgPkc5YL7eGxBxGAcObNbruAYygdKlYPi5FhNyJ69xGTxdy2xIiEzeEEilt2H7PWTdg7RSiFCKVBNjfFeWDAIKV2zxLqlffJGizLY8oRhjR8NIsT7OH3vItsYA3WkO2E7C6DoQw77SZJ5rwE98j3lYU4YV9N8nzNcKGQglcJ0uP/UnYicEJ7Oa2LQhCebZhsJs6D8m6Fbc3jR6ZjUFCJbEnJnY/ob5hUl3LaO0lUghwqpzC6G7MGkn+80IV1k366aeOLUGL2p2cowhyclBjh9NqTPo/T+dMzZP6rT3z0eWwr2cZVAEdOuk0cxeQvTOUzXKt33riKQS7+HCRTJk/2mcB9pTvzsMfzn9D7Jn1nRa5hDWywBUHGw7EedKL0NkgngavBzbUUiDKNs9/Q5XvcYClVQMsvFIMXrPzKoSPVkNblYENk+mTyoEsIGSDf7FeUUWTAVw4CMzyxaULQmwYXsXZTWwxdzQm9mATFfv9SnCR/idaJEN9V6qKNBiFLgXM3hPITx8DUTaeU0B+5kgJRYu5P/vZb0T0bQk+SxHyXKJn+ii0BPJ4mqpy8yMZpxX8LG504DdayjRDUyfpkhPA3WOa3gyPuMwUM+zPLYX1bbESWz/IWJf5hPf34K0k49jM79ojT1pC8yA76vg7xTgHfi08wy1a8FAqVgno8A5Onja/v9sySywUx/6s/+vMdvCmP5v/yTm1/5dv+1Fz//tR8bAMg//yW/uQ8pAv7gs1t8/m6LdTNAKY8nL95jcdGh29W4OD9AqgDxusLF+gjIgOqmw+L2iPasx3Z7hF5PEDrS9CpANA5/7bd/gmI7QhiB7as7LJ/uIK8GVLcdpbBHDVyO7IarHfyZgb0w8CuH6uUexb1EKALCykI97RAah83zB4hRQF/2qLYDoAiIcDEymOPxAHczQk4C5sLy/VWAPJ/gr0cUmxGhcfCtR6i4Aih+6wDxpIddefjnPf091wPcwmN8atC8FTBbB7d2wPUItZlw9f23sBsPv7awT0aYc4fp0sKtHCCA8ukxhg4IiOsB5dMj3KMR0+MJvmWwyOJzCbmdUL8T6J8bFDsJ/emeHs0PGuONhX8yAALY/EjTF3lj4J4PDDY5M/S83Vi4MsA9HVj7svSon++x+ImCeTbBLj3s0sNsHUTryOI8GzkZfzqQubi2CDqge2FhFwSV5szBXFiMZx7TizEyNQF2TZaS4S5A/VZi+NYIt+Gxp8ATc2OgO4nh1jJwRIPnaetRvxPA9/ewiwDzaoAvgOnlQHD4wMmnPgg0bwOm7/RwdUB/4+ErwJx7TBufAZIvE3ME9LcEk/0zg/q9yKEwEEB4NsSJDCet1T3Q3xAMjxceZsVFgONzh+G3ewxXLoI6dspJB5iVx+qnnChPZwHDlYerAvonDv1Ty944yTCawycWwlEyeXxhyVwcEhsSMETpaveUzOHmxx7jWUD3nRHlQ8DxlUH7JVkhBCbcQrDHcDzzGC8C9r81wbUBF//Cofv2RDndecig7/jdEdU9x5pk3HZBX6x0wPG5Q38dmKx65XF46dB8FdDfBvRXAf21x3geMjs1ngV28j3yOD712PzEQ1qgv/WwC05odt+1OD7jJM2VrNJp3gYMlwGHlw7lLqB7RFZSGbIg/aOA7nHA4nPKCeGjXHVNGfG04cJG89Zn/6aKpeXlPuD43KN+H9B+FWKgDSf4/Q09jsdnlH8GwUCW+l3AcC7Q3wZMG/pGd991ODznNg8vPYYrnsPEcFHmSyAvJ2D3qWfgyhR7+GKi7f6lj0E5AfsX9EDuXwWonvvqr0WuIUohPr7iftLxCAt0t5xAT5sZ3FR3HuMG6C8F+msBVwvsPuX9keSzh+ec+B4fc6GovxU4PmJibP2edUXDFSce04bn5PiUrGT7xuP4REQGh8Bk/4pjnFZ8j+4C+iuB4VxkH2j1ELD7hGMIGli+5qRImoDdtwjkvCa7x4UPyp0TIE2Jx3mRJS4g7F8I7F/x+BdvLL2FkhP+8uAxbQWUAbpbAozdS3ocuyuJ3ScC9b3DtBJ5AYehMZR5Sge0bx2atxbdjUB/JTBt43FFL2J/ze0P5+zCSyDR1gL75xLdlUR5JJtY3zuoKaC7UTAtr6OM7Fb14GAWAsdHkmB9CGjeWexeKBwezfVb3ZXMQLK+dzg8IVuWfKrDGQNJ+guB/kqiOHiYVqJ9bbD4csRwRnlnuff0+VZkUYcz2him6O8cthz7sFEojqwF6y4VuiuJh5cqpuoK3H9bwSwkTJSpTksyb8drBdsweVUN6e8+WU6vOEbdMfioODLFtdg5VHcxgK1z2D3T8IXA7plGdWcAAfTnimO5phdSOC6qmKWKYUGSvk7LtFtpAuq3A6aNgi8Fjjca+mDQ3WjYpcoy1PLDgN2LkvVJAKaVwuFpiXFb5DAaNTi4WlHiOjq4RmI8L6AGh/FMY9pqlO87+id7Ju9W73p01yX66xLVVx2GMw1hA4rdRFAkYgWM5sKU6lir1F9o9FcFEGtkvBYZ5HQ3Fb/3AdOmhOoM7EKzdqdUMCsNsyr4s56SXX3XYbiuIDuDaVNA70eo0cGsNORg4GuF8azCeFEC3sOsS+j7AdNFHT9/mGhqVwXGyxq+YU+k8KzsccsSblHAtRr9owV8ITGd1bBLpr/6QkLtR8j9APXhAFdruEVJ5vpyAbco+b1SCErBrmtWE20aIAS4VYUgKYF1qyTnFjPbV2j4RQXRDRk4imGCbytAzRUyCSyGRY2wbCilXdTwqxZBK4S6IgBctQTjBWtQ/GaBUOhY0+IR1gv4s/jhHAKlswDgPBNp5Xy9UnJrWC2A1QJi0fJf/BoAWdTIPoqqhKjSqq3PtSOYImMpBZ+PoFakkJ5YQyKEIKh0EVj7wDGkf8H/4vfBRynt/HWwJ5Us39BHAOAgfuX/flMe32jGsnr1NDz7u/8ZzKGE3EW5Su0grAQcyCRVHrKXcUU8MHSmQmSAQl5lTp6s9nOJ8YzF6dKSXRKWkjCzirUbC/qmUliKjmzSeGPR/kzTd9XHVLjI2ngdMN1Y1D8vyEKtPKr3lEzZBX1xqaw9bTsUZJBU+j6uGidWS3ezxMi2DBFSAyfo1QemVFZ3nAQFQebFtoBXZMASU5CkY+UDJ4hjnMxX73mjJs+Uq4D6AyeOifVMNR7Vncjl19Jwwl4cJIQB+mcWzWcaKlUeWNaFyFFkJiet7qZEwJS6aFsyPsURuP9ti80f6I9CaXQfQ0xOmIKg+P503OWJX2vckiFMnsvkj0oMiRo5AU2eMBWl/GpI0riYBGm5qp+CMcyCY+TACIJUDLcRgfdBdUcGadpE5iJ61FyDjxIrpzW3lRInpZvZO2mQgVBKYHQ1GaXMDmJOmGQ1CtnW8YzHnbxwdsGakaDYvaqPMUBGcuKuRuRqE5cYw+jr09GflOR1qVg8MVauQa7PSQFMeiCAKfaRBYghOc1XAd0t711I5EAdllyLnAoKz5+7KtZ9FDPzQKZKzOxbPReK817heW/eRkB4zeOVkTUQIW4fPEazoNzNLBm6kxJMeU9y3HLkfZA8gvX7kKWRTGgUqD9QpuhLvh4iMc4hp3ume4/pjRxD+RDytm1DEDVcxYTOLuSQKDnNaZXDRZIRRiaqEvl6kmkMSJU5Sb7pY2pmiFUvlGkKtK8Z6pJ+B1PoV/7dPLHuFLFeJQVNJZYrs+eZ6ed51j3BhppmD24K3xnXMnsnq4fI7AmyeQDQfGBFR/59NTGJU7KndlpJlLGyRk0M4UmhMImVVRNTYBOjWnTs3U3MpB5iafuOATCmjSE9NYGiPmXPTlj77CuLgAwCsDUBFwII5FaUWTYfHMe69/l/08p4Hlz253lN5iZ9z20SbNtazCylCfl6Jbk3ABQHShQT6JU2ZHYxbZvnwOc6jtSNy6ClEBdRQk4INa1E89bCl3ONCjtBKaU0rUTzjj8fNgrVPrKWAigODOZxDRm8xLCqkT2xygQmmUrk47ctJan8e0VwHQTgGgk1pC7bAN07eCVysE4QiKmeIdd6kK0Tefv6mEAZg318KeFLMo7peHUXg2cCMsM4bTTqdxOBWQSdAMcXIkBNoM+29BL6Qp4wbCKzdUVn48KZhhpc/IyIfj4XoDsG5NhGodgZMnAAXMFjSoxgDn6P3sWgBcTkIUKAWRbZz5j6XZPf0BcSqjMER5WKbDIZYX2YYDYV74nYq0r/ogY8xwkAqreQg81VJsIy2CXJMIMWubOTf58KqNQHaj18paH2A1yUo6bxeC2hekPQVM2eQwBkJq2fQ3kmi1BGa8Bk4duS8tP069lPcJsG6jgBxsIvaspNJTL7J6xnr2fcZ6r/CFpCDGYO05nMHMwjBFnJ+JwYpvw5gLKA6EcErfi+GLwjjEWoS4hhohdynFhXkraXgoLS194DVUnm0ro5vOe097KugGEkkKsr/j8ZdoUCCP1AMCgleze9J9BL4T1fT5VNwNEHwLvZc1lVfM5G9lQpdnRG72SYpnk7SgHG/Jn+yfT4MwN+vv7ab7jH8rd+UIX/+Z88+pVv96+/+Nmv/diAbziwbD59HD75b/4ThCDQdyWEAKRy8E7BW4HFesDhQwsxKITKQVYOwUnIgrowNyoIFRB6BX2n4Z8OwNtqnigAZAfTJ7YKgKUcNeiA8qajvHaMenVBYIsp1lUU/NAXk6CkNUpm7bmBumd9g12zLoEVCooT09YjlB5qp3LEvWs9qncK46VD+V7BLmdwR4lmiJ2UAm7hCdosw27kwFVfaSgDTayo2qu8wg5Q4mnOHGQnc3emPlKuqjv6qFTa7pI+MP5BDPkPq+4oOXVLBrl4PXdIEkQGlHcSdhXgykDW8Maj2AlMZx6qlyjvOcEWgZPxaRNI3llkQCitgFccly95/BBRAhylpNKKLLlj+EuUMoU58de2DMRJINtsQmaeKGONcl0fAdJA+SrA8ak+yopTLYYC6jcMWPEVIAyyRyrVPiRp4an8xy65X9XP0kPpouQ0MpUZWNbIkipfANUHBsEkSV6SUVbvY/ek4cS8ekAOYVETcr8jgSKBbGJg5soJLrroo5j/bxiu4soIFLo58ESmaoXI4pjFLFFM0s0MvtxJtUR8T5LzTps5aVj3s8RTOD6XQFmSqyZJZ0qsTZJV+rn4s/Ta4hi3k9JdVeyA3PJ9KcU27SONNwXVpLGLMKfApiAW3fMeG84JtFiREmWmCnNSa0/JnI0LyqnD0Cx4vUQ8Z77gAsZpd6RN1Rh9BG4eeUHFNsgdoqdgMWie65TImh7Szu9N8kk1RUAeYgem5XjmNGC+JkjeiwloSlp3Y9VHmOWkS1YdJDkvwVC81+L8YE7xna9lOubyQGAzLWX0Z3G/eogLAEPIUlER5ns3XX8RkBM2pYmS6WGWlppWxH5OgfJAcCsjYE2yVRUBPH1ukXUG8jaBuBDk5ns2LdrJE/ArbeC+G35esIqC/Z96TECLqavTkmFeKUE4gfb0GuEIJhOgBLj/FCqUrouIwUVJ6usjMCk6spvKhBz0Mi1lrOsggEvA3CtuWw9cFMgVFElWGz2FKZzF1jyXrhB58SbJXBEAPcbKjhNpaWZVG5mrIlh9EaJkOMpyRTzOgaA5HY/wIX7WcLEsLV7oztF/V4q8SKeHed6TgnmUYaBOlssKzDUc8T4SdpappkTWHIwXFSbChsxSqtjD6Yv4+995yktjCFQC89LFPsxKRsBL2etcyeFgFzpPRaQNczhROoWC96srZU7JnSX8IYNiVnuE/NriYAlqO8tzXCkEKaCPJvsMU59jlrSnYwCAEHKwjrAeamQQTwIrcrSwy3Iea6DcNwfoLMtYCUI5a4h1G2qw8IXK7/uFQB3BZFy9G+Bbgic5WEqpjYNbVhF4Ely7pqBc1nler1hVAu+ZPFvQQxiUgBxslNQSYKeqEBH9jvAeYjD0NAL0Umo+H8qYEBvBpnCezGSq2ogS0VCVEMYSZEqBXLVxKokFCCQTwFWSfZbFSffk12W06XHqU0ydlqnfEvg45fXr4Thf31aqHAE+SpTN7z/ZXpjMfJxSzq8/Cej5f407viaL/aZLYf+yA8tvtBQWALwXCAFYLAcsFgMuNkdI6SF1wPGhwfriiKA9oAL8sUC9HFFWBgJAUVtI7QFP4LdYDvAbC+GA3/6rP0NoHPR6wuWTB1w9uwNUiB5NIBQBZtKoL3rIy5FS182E85sdRBBQVwNWj/fQ6wlh6SBXBr71rEFpLdyZBZ73EK3DdGWhthPEJ0f41gMLi3I9wl8YhKVlD2LlMZ176O1EZvJRB3s9QZyP8Jfch5oE7NYilD5LW1Fx5dZX3LfbWPjnA/S95urmysEvHIQTCC84o/a1B573CDpguuJ2pnMPd2ZZvt1GRmAQmG4sV61XFqEMKO8EE1T3Cub5iOIgssfUbii5nc49XM0U1ukswC8cxiuHoALspWFNyZnD+GrAcONht44s57WFOeeH6XjlYDYew7WHawLsiwFm5Vn1sQocazH3eZk19ym8mGsvRsCsA6aNx3hJ6WQAMF3SnO6agOGRJfBbe4YZLALMuYM543mdth5m48nK6Dm106wJbpmgt16dAAAgAElEQVS2Gcd8niajITONZknWfLpwufOPibScCNg4hmmTai0C7CLALkP0E8ZJVMPn+icOtg2wbcB4TqA8nAceT8lglenMM73ynmOZNpQwMimW+1Y90N+EODGO/s4+JrNeuugrJEDob3hcZhXZzlWSgc4T/DEej9c8D9MmwKwJVMYLD9vwe9ZWzPdYSpZ1BUHhcBlTKC8900kXQCi47+mMFSW+nFM8E7ttazLOvmTiq1lyom7aODGu5yTRJAeFjEmkay5umBW9rMmTOm54rsZzTvBZd8PJsoue3MSKc/GDstG0b9tGOXQdvWaO9RlyRD4PIUpa2cknYBdkXm0bfZ0VFwvGLVB/oJSViwHx/opMpa3JSgNko4dLgfGck/zj4znkxqwEw1y2ZC5tmxh8wC4+9u8liSnZb94b7BPlOZ/W0edtQz7+dDy+EBjPRD7fZiky45wWlforBqx0VzJPzNN5ClGe6qq4zSVfm0C2rbkfAoyZceyvuR9bcbL48IkkAx2vn/C8J4dziWlD2Sa7TQHTiOzP5KICv6enc/YPFgeC62kdF5viBNwsCB5tIzBuCYhcLTCuef5sFe+dAugvKOs0raDHNILkaSEj0GE6qKsIipOUn+MRaN47uELkYB0EoLuWMeQnZJVHArhmpTCuZWaxbS3gSolxIzEt47mO9SIIBHymlTALmWWr41rOHs+CTC/PkcgMnStFPJcS04L/eC9FNnTB+hHTyNxPSR9fyEEyQQr057FWZAzzfuLxCk/20zY8f7ZVTGYVIt8LrqSn0lVkRHM4T8kpEet9eA6Sv3BaSbhGYdwogtGtRnG0OTQnxHPimtiNGBlxsyDQ85rbc7XiokyjCEwRwZ6NfZWNZJVIfN6sFMGe4nkMWmA809G3Sl+jL5iQ6ioG1ZgVQcB4puHqyERGT6gvmfhqWwXXxOPVAnZZzGAx3rfjeQE5OobcNCozwq5SeR++5PjMKiahGrLArtG5UkOODq6UGM4LejqdJ+sp0vgp5RUuIJSUziJ+7Rruxy1L+ErBV5os6mgQCkmWEyD7WpORs+uastaYDBsKRWBuPINwKgUxmtiNyXvTVxx/UOzBhBDwtaaXU0qG8JTxNYUClIRbVpmxDKdsW1vRh1noXwSVgiBSxAqSoORcD5K8kWVBplFKSmFTtUhFOWvQKv+bAZznPpT6mPHMAUCarG+UwKKuCBbTe1JAj1IQUpI9jIBOlCVEUUAUxQxitcZHXlGA30fJa05/TQA4bRP4mL1Mstg/6x+AJIf9yxTe86v+95vy+EYzltvvXYe/+Q/+NpZ6xPthgd1UYV2O+NC3aAqDySncHxpslz2ECNjHWpKqMCiUh5Ye+6FCAHC4b7E9P6AfS0jp0e1qYFQozwZIGeC9QFlajKOG9xICgFSe1SVOYuoKSO1RVBbWzpUmAOCNhK4snJXwVqJdDxACaEqDd29XEJKJrWFXQp6NCF5gvepx/+UacmEQgoCQAX5SUJWDG+Iq3iQhF4aJpZ1GsTAwgyYLOyqIkgwtAiAKDyECvFEQB4Xipsd0KCErBz8pCB1X0SYJ3VrYY5FDjWRt4TuN6mzA9LqFvJjghvie+xKL5zscXi+hNxPsvoCoWKdSVhbDoQQGxUoTAKJ0wL4gs2slUEZGeJIE7jogOAG50/ANZcx+6SAmCdjIGgMIOgCFhzgyKTQHLAFkYh8PcLuC7289024fNHzlM+MsvIDfGMj7gtvzAqF1SLUsolcQ6wnhoUSQDFHyZwZiVwAqIMgAEQQZaRUiQyCgOnpbVSdR3kmMFz7vT46RUR4or4YXPDc2jr0XsFeG6bK95LEBEEZADWSDgzqpCYm+yPHcQR8l7NoDjgsfeqdgL0mnyAcN3/pce6J7ekKHa5/ZVtuEmDZJhsJXlHSbs7jwoAPUkRKt8l7CbOKijObrUtKnPgpMFzxGOYns3SToPpGAKsBsHNRRAbFKBYihS4JBTCKel1TbARBMJxl1XpE3glL1g5iTFSPrnoOeVGS9MYMfNfJ9qp8Z51QfMp2FzJAnwDOz3MihUOnhmhD7DAVDkHaCwUI9faZypJTVx3qQxLgPVywRL+9Y6xB0gIpMfJDcjllG5jZWwyS5dwo8crEOwqwi26dnlks45GsjYniUWVO+jpD8p/HcFif7ATLDmZgvX3CxoHlLptnFQBsXmcv0HjWchDBV8zi9QmZE1UQQla6N7k6Yy3Ler224CGKWs9w9sUOJTeIvSdx3BPKnoSWqJ6tfHMiiJzbVF7E/U4uZ4V0mWRxygm5i2xPjqYb5Pkr3s61j12sKnQpx4aCc91ccQl7EUMN8H5LN48+T6iCFr+T7ViRGb1ZYSEdm2EQWOzGZ5Y7HkSTLwiN3kOqen1VmKXKYEhDZaw+4Jh0/FxESQ52qXkwMw0q+0nTNkk1ATRyPCCGrNBIb6wtkC4mwBLYpFCfdRyloC0BmJ/UQMC1Elvl6zesmAjIgFz5kVjL5SVmTlAKbRD5nXsXxRcaevsv5XKiYrJrk3wzlij3TUZbLhSkZ2UPkvtO0gCajTYXe+CgXN2lRlvUhaSwqBgip0efPG1/E6xO4eJKAizRcsEyy5ZT0mj4T5BQZUkkvqSsllUN7nxdobCVRdMxqcFHinc5hYj75mTcvTFCx47LU15cCKoUzyaQ68VCDg1kVUJPPDLYafQTS9OGeBhf5MlbbuJPAIRdi4FTqwp0rQ3wZA4QmB18xzCgF3CBEqXEZ5z0u5CAhKAFhTkJmTmS6KaRIHQ18oSCNy4xo8rLKyQEukOkMgcC3opQ3yXNhPXxb5OAgYfgLSCDrICKjmcGWB8RkyCaeModJApu+14rBPVKS4dSzfzGFPKX3i5gES2bW87VI7HsCafHrX1Y1cgIEP/r+9BECvZKJEc09mYLjPH1pqhv5+na/tu1/JSwSQeZfBsbyf/zHj3/l2/03X/70135sAKB++MMf/rrH8Bd+/Nf/3d/94e/+R68wBY2HscHvXf0MvS+wLCd8e/0WWnv87vXP4YXAT99c4j/49J/jGGgSCgD2Q41/49Fn+OKwwSc37/Bv3fwUf3I8w/6rFX73uz/D33z1R3hrl5iswvdvX8MGhUebHZ6d3QEa+Ndvfw4jFKQKKCuLv/L4SwQp0DYTLldHnC07fHhY4Ppqh34s8R/+1j/Fe7ToxxLTWODJ2QNWqwFPzh9w17W4enSPurK4XB9xu9zj8fUd3uzX2Gw6TFOBi4sDNsseRWPgJaAbCyEDLrcHOCUgJFipAYHltsf3Hr1B2Vr0TsMbhdBriMLj4skDDrsGunbQhcNm0+Fs3eFwrCFkwPXlDutNh/1DC+iA0GvUFwPGDw3KywHPr+6w+9kGL7/9FcLCoe9LBAk8unrAFLuegpXQhcfvf+//wr88XEKWHmFSOL/eox8qfOeT15ALi36ghPns6oChr1B+UcJvHLBwKFoD5xSElQgy4MV3vsL9rsWn3/0SD3+ywfUnH3C8a1Ff92hXA1TDAminA8Kg0V4fMVkNYSXK7QirgGI9ob3qYLXAy0/e4P7LDUIVsHqyJ5D6oiIQNQS6jx7f4ebmAR+OCwYWtYZdqBsDdBryfIQXAsX5ALXkokLz4oAfvPo5vnh7hovffYuj1AiGgU5+5fB7v/PH+MIvsLjsMJoC20c7BjNVAd5LFNsR6ssK7sJCLizKz0uIl0eYUuDi+T3q7YB+IeCbAC+A7//1n+D1YYXzT+7RPzQQTqC86WCkhCg95EMB4YGLT+5Qng0YgoZTZHEROKFefPoAoyLrVAJ+y8TAUALF+QDUHmGSCI2HPJvgLw1w0Ag3I+TKwhaCPZxPRtgggZWF3BpYLRAg4B6P8F4R6K2YLGzOHdaP98CfNDDnBK/iWQ/1aIC/K2EvKREKmvovd0aZk68DhJVwrUf56gD5WQ2z8QhXE9AzPMMuIpO7SX/ABcLzAVYKCC+hPjlgggKcgL82cGuH4oOGawOggOGFQf3siNEWUL2AtGTn9UFhunAQj0fYJkBMCqEA/KseOGpKvG8mqJ3G8Jhgv3tuIa5HmEWAuzKQtyMmr+DWDJsSLztYryCtQPU79+gLCTkogjhHQORLoPjBA/pWwCFNiJKXlddw+qtHiHclzDJgejrBXVnYGjwmL6CPrCPRPZlEXxLwuoWHOfPQR/q8h++MEEcF870e5ecFum9NKHaKr7+1ZGZLieHaQyCOrwD8pz3k+4J+3ouAaQvYdYDXAWYdt58Yyo2H7gWmLdnM4ZGD7llJoyaB4zOP8l7EgKsAu+JrigdJT3IBdM8tynuJ8SJgvKb/vXvqsfgiBhd1kU0PAv0ThjL1tyEuaHARaNoG1B+A/QsgKIHDC0ro+0ceehDobkNO/AUE9t+yUD3HMJ0RJI4XZLUhuADQ3/jc/Xp8FjDeOkxnvGeHWwYpVXcCh1dcCOkfEeT3N9xOuQd2n3Dxxyx4nSFZ7XL41EEfJPobnt/ygQFG0lL2Pm0CXEsZ9uE5xzReEAT0t1GGPQDjeQoGEuiveD685uRbmoAhBiwN1wHKcEJvF2SRh6voAY8MsFky4Xs8F5jOBfavAhZfcHvH54B0InqxuW+7ZNAQFFny8Zyv6SJz3rwPmLaSCw81a1O6a9Zf0A/Hfe9fglVAkaWtDgHDBdntJIW3y1QJI7J1wNYCdinQX3JxyrYistBkqKc1LSX9DY8tFILsvACOTyV0H8cfRGTfkwSaftrhUpLB12RgmX7Nif9wLnOoUncr2Zsamf/DcxkXCSS6W4YP9VcSemB40bgVKAbKgseNiunAAofHiim8jcDupcK0VrEGRkZ1jsDxMT9fjrcK5YFyY68FBEBf8xCwf6phG4nxnEmph8cKeiAQ764UAf4UEAqB7ooBRsoC/aWGi9Lnw5MS3W0BKIH+QmPaKMgQfa87+gJdLXF4XKA8ehwflWje06sqPTBu5+0eH5VA9DzbRuF4W/I+37DXs7stoSagvyogQBa2e1zDNYqLYLWMDKRAiKmy01mZ02p9FRk9KTBc0r+oJjKpdlkAksA/KEGWUwpASUzbEtIF2BUlvAz4iRHwUSZt1xUgBcxFA32c4FY13LIEFNlYEYBQU5rrlxUEBHxTUEZdawJEgIsKdUEW+HxBkBuDf0T0bIampLy0KQFJJjXUBZlH6xBWDVDqzJSGNnokteY+kk9S6/lrFX/RUoqtUgS3SjHMR5MBF1XJcfQ9nxeS20my2SjhRXqtjhr9BC5jxQoiwymEIFv69X9A/Flc6Ejy2r/A48f4v7/84Q9/+Pf/whv4FTz+2//+v/zh7/+dNQLEr/TfP/x797/2YwO+4Yzlk+9vw9/7X7+Fe9fif3v4No62hBYeB1uhjJn6nS3xO9svsLc1fnR/A+cl2mLCw1ijLQz2Y4VPtu9hvcSXxzX6qcDFooMSHutywE/uL7A71lg0IzbNgONUoi0MBqvhg4CxClJ6rOsRzkv8/Ksz/ODF5/hst0GpHZyXCEFgUU646xos65G/F0Hgq7cbNIsRWnn4IFBqjvnYVzCjRllbbBY9+qnINHffVVgsBrw4u8Mfvb2EEJTkLhcD1vWI98cWzkl4L3C26jAajbY06I3GoatRlhabZsB+qOCDwNCXECJEVtRDKbKzV5sD3tytALCqxUwam1WfmWAlPe4OLaxRuDrb482HNQBgu+7wsG9QNxMOuwZSBTy9usOfvD4HAlC1Bs7GDxUREOL5CQGoaoN+X+Hyao+HfQNrFNrlyIU1J7Fd9nj7nmPSpYOdFIQElHaYPtRA7cj8RlCIIFAtRzirYA8FUHgUjUHwUX4Ux6G0hx0VpPbs+L0r0Tw6YOhLeCMhDhph4WZWN7HBRrKmxgpgbRBGhfa8wzgW8FYiDAqytfC7AsXZCHNXAZWH6BUB6V2F0FrIwsMfNUTlEQaF6rznmDvNlf0DWVm1MvBGIngBmMjeCgAjj1ftNFN3WwcUARgkIAG5MPDHgv/vC8hJwtcey5sDDl8tIUwcf6e5ymskxPmI8KFCqDz0vYJ/OsDHc4hJAoWH3GvuS5HdlZOAv5rISJfzSqQoPEKvIBoH7DVBXkV5OmSAutdwDaXGXGVmfYtwsRqmDFzZdYLPReYordD70qPYsV/Rl1zxV0cZGVcJ13iE2kOMEupAFsJcG4iRK7n6QUYGgKyp2RDkykHCX00QHwpO8lcE4tBkjdUxeuCMgK/m41WRFRae3mhhkkQzhZ3EYytYV1S9U5g2Hn7pUL7RZAtPQlgS+5gZOszeT+miD7gOJ1H9QLHjcxBkCQFEplySVZME33KkRzDoKG3feuhOQB8Y/qXGJB0PcyWIRAxCE5nBJGNKn3R5L5l0HcD7L9YKiTjWxGKpKfqno9JATrFCZ4ivi55N21JaXd7H3sV1rCuKFUNBIndrmmWUu0dPcmLVAN4v5Q4xnIr7VePMAjN8KmT5frmjzDiFjrnoTaZU/Rd9w5mxG5GrQJKE1RfzsaheoNzP+02dl2TLkSXZupvH7hrkELPhfPaz0sPL/ZZ7+liDQA6QUiOiZDygfpuky8i+4xxKlbyZZQzq6k+Y6hN/YxoHfZ+xXsbPHtbkvSQAiNVKTQwpi5UxAMFcek/at20peU2BO+ma2IYpvim8i4FrZG5TcNFH1xGsbTFNBI51qlPBXHMkGXxEnzDHkvy5AHJYHuTMkvsCqHaBnZNnEvW9x7gWKLr5WhddyNfcLAXqu5DDm1I1Sgr2Ko4cY3mcj1VN9PQGSWbUtBLlwcO0ia1ErvKhX5VA08ZQqsyATpTuqpHPAWQp0+eDtCEHH3kdw8Si1Dz5WYUlgD9l/hHoE/UlJ/imkageXGYW+ToyqXYhUd5bmLWOYw/RMxrl7IKAOyXo6qOL7CVyYmuqpUmhSCIGMaVjSs9z+4mBJZsnLb2fPlZ1SONhlhrFIdpblkVmYBMrmdlaGyCNi+NQEVzKzI6K0WUWMjOYkeEULtDrWUTWMC62U4Ybj8k4hCiHDUpkD2uuL4lez/w8QPZREdClGhPRDQz9MRahrSH6ka9bNHlMubZkmCjL9dFn6hzfe+w/em3CBKKpZ0ZTCmBM8oMIGk+wQ0ghQlLMQT5JsurnqpFfWpNycnx/Gpv59Ycfhj/1Z3/e4zeBsfzeD6rwD//xk1/5dv/Gy5/82o8N+IYDy/V3b8K//z/9Pj6MLUanMViNq+aIoy1hnIIQAYtiwroY8L//wbfw177zU/yzLx5j6go8eXSHz39+jhfP3+HtfoG2Mng41HhxdYc//tFjLJ/uIACcLzr89MfXgBMQrYOuLKra4HDf4Pxyj/2hQQgCl2d7vP7yDPVqRFFYHHYNqobsljsWgAxYnPWwVmJ60wIrQ8mqF5DaQ0oPc18DMkAvDcIXNa6+/xavvzgje1ZSXgovoEoH/6YGLkb4Y0FgszIIvSZ4sBJCBlxcHDAYjcPrJf+YdJSFlu/JhrjaAylJt1OcIC8c6vMBw0MFWElZqWAYDwoP1VgEJwg6Vha6trC7EiJ2R4aCfyTlSJaxvO0wfbngcR0kJ/+NJ3j6yRJ2ww8ctVeQTzuoHy0wPDWsdZkEqvcSw43Lk3WIgOrVHtMfrWG3FiLKbGVPiamYJELpUdwp2A0n9Tnts2SAkls57u/ZEepHS+geOD6jx1MOMWBp6VDcK5hbg+VZh/6PNmSPNgxbMluP5nOFMXo+hRewGwt5VAgXE4KVKN4WMJcG5ZcFJXNpgn8zQb6j1lBaAXNpoN8VOSTJbB3UQaF+JzCeh9jFOMuvVB8LuhecpE6XDnrHIKY08RAeUJ3A8Nih/kLRv7hgaIEaJN935lC9VwgqcEIeCBjKOxkntnNwUQ5DijI9V83Jx65h32r1RsGsmQZsFgRNwgo0bxneJKfYxVgR0PgqwFXs6/QlJ5/TJmC6dFj8REevp8iJrakn1DZMfS0O7BctDpS8MtlYwuuA9rXAeM5xMvRIYNpwbDJOQs3Go/xA8J2CZFKBvC9j0vIkmFRcAimxlinPBFvlTuTE3pRwDHBb05pgtTgKjFumLgfF86UPZFpUT+ZLHxja5KpAAJVCnyI4Yv0G7wF9FLm79lSWOW2YBh00fa7SiVz54vUsNU1BUOOW57S84/tT32eScZoVgYxZ8nkGBxF0wMevY9hS8iEWh5NxxU7S077SJIf1FVOSg4gpuw2BnNcxyfhIb6dZzhPbnDodJ/1mSU8qvXDIfaQpEOo0OThJdYt9TNmN/aHl/QyYAbJHaojSTkNGsjjM6c9JwpskkkmqWz6Q2ZlWHG/zlj5GE32WyX+dgEV17zFuKNdT40k/YgSptpklqLaOzHXL+6e6D6wTGU8CmcooLV0QnKkRmLbs8gwqXTtuM1W8pG7LtF/dI9/LKWhqOJdMFlbIsk7bIIcuZWAdw33YMwo07wmQbBtl0VHCm4KObBNDnGLqcwp/8gVQ34cI0CltTWFFEJTEBoEMYqWN0tKRAMs0SQJKQDUtZZapjmuJ8kjgl+5xZQjEKD/lvZS2l2SuXs/S0HReXMUkXdOSpWw+OEyLGHzkEWW0MXQogp8kh60efA4XKo4eZsGQodwRGrsz8/lNoBTIHsjUwQlEqWny2Z6EKLnUIxrAZN0YWOQLgeLIBF1I5O5JX/B4zYLjEpadngRMBN/pdQgJ2KZ7QqJ5M8KsiizNtI1EsbcoDgbjeYViZ+Banf2davBwlczP575bHwj4YmdmkremPs003vRwlYyf8T56pDXKhymDxCSRlYYBQ3K02UspnM/H5CsFOTiCTMFzLo8j02nbcu6mFIKSWCViGu/H8tJQRE9mTKwV3uf3ByEgnMuhQMK5HBqEEBgCpCRyH2V6CLKl4tgTQI6GlSOOwT7COqbLGhtBqfwYuCWAGZNs0+tyLcrXU2G9R7AOInk/Tx+xtzM4VosE5xnck+S4J0m1OazH2l/Y/ulr/pUeJwE+33Qp7Pd+UId/8I+e/sq3+7de/fGv/diAbziwbB49C8/+8/+Ck7cD/ziofp7ASzsnSKYUQwjkUvhUz+CL2ZNT7ALlOjVw/1c8Lv8PGYMl+Ic3hYGkSoQ0YUp+JxHiRCfWM6QEQJZ/cyLg0yQoTmamNSdBcuLEgBN5+m2KY8jemGkbvSQWWLx2GNcyBmLwD8vxqcDqJyFXLNhFXJFXcyVAdR9gFvTR6OMcUZ4S+9JKcerb0/08qXKViH+kkFmPcs9qgPoDJ0oi9qydpk+6Mv1R4jaWrx12zyjXkXaevNHjweCLckfwgPgcWQnBVfkaqO88umsF3Ydc+O4q7jdF8Cd/m22RvXVqQk6bdJVgdcMUMG4ZIiFjV51pU0gI5gTS6K+q72JJOXjuUlG7XQioOJ7+RmD9U49xFUFgO9c82GYuak/H7DWv+eIrj2EjkTxQagho38UV68CJhStnFqU4egxRNqbHkGsZ+N4EeGKNQkrOjL4jaUMOTPko3TFO6KqdY6hDLXKqqTRzqmdiHIZzifaNhxoD+nOJJtZGpGRPPXACqwdeY9OIfM36S4nFa5fvreQtqx8cjteUbSXGJBWqA3OhfZogqzFgXEssvrIwSxXL2ONEKk4c0wRNBGBcSVbG7FntkK4lV7l5PUwrUfQhru57HG81qgeuuLuCvig1kcFwMTk0PXTnczXLtJSxZiPWvFQCRc/rXz04HK81ipgmq3uPYavQfOAJsvW8Oj+uFYrOR7aCQSNq8pAjJ2TjRqK656RVWiY+pq9FlOrRS8VV/2LvYBcyflYGyDHALFl9ERQ786o7S2ldTMrk77zjpLCW0fPElfVxo1DuPYqjy1K7BKTU4POYpPGwC4Vix2N0NXv7pA2wrUJ5b3B8XKG+iz+vZK7c0J3Paco+sh1myYCV6p7XXvfRExV9WQCyX0za2UNW3htMa3rBXC2zH5G/05yk2lbl3w81epiVigwbq0ykY6JouYteqtgTJ1yAMp4hJ0v2FkoToA8GZlPQExfBDMDt2ZYpoCIEmIUmO+KRqzJcLePfOJ6/oIDqvaH0L4LEVEKfAIGcOIHjOeekXg1kmGzchy/oxcsT9hDgWh3rRoBiZ3LqJ89HQLHnB5BdFoAPuQqFKbJMGvWFnAFACAQKsbYjgYRpwy7IxCr5UkIf4mQ3MnwIQChnkMEFNpFZHTkxhVR1DGORk4MwntLD6DkD8FEFRgqKUYNlgAsA2Rm4JQGAOo6YrhaRuQvzsYQAvR/hFiXUYUSoipxCCg+CDSHgtSRrFUN48v0YAYuLiaMikL0S1sOuKuj9mGWXQcXrvRsBJZg2OsVqCCnhFiVkx+uQ6kF8o6GOUw7R4YHxtb/wfHyEQkF20wx+CgVYT0Yujo3diQryoePXleb3A0EaXCBYUXKuECk15GgAY5mA6j2ZttjBCGMp8xwMQluxIqQp6EUcJ4QFAVT2GlYlEJ+X+x6hqWZAdAqQymIOr3Ge4wJmD2ZMRg1lgdPgHWEsxGSyfzGDL8eqk7QdVmjYXB2S9sNzLeYTm8aTQGJ6faz7QOp5BHK4T7D2l9dypJ9JmWtCIASrPaQCvCOTmPyMKVjH2jkYyDlQkuXmsX3ND8lf1fj7EmXC+ZGAXfRVBmshtP4FNjJYC6EUgnNzAuxpQu3Xt5mYTSHxZwXzBB8ohQUIVP+Cj98EYPndH9Th7/+j57/y7f47r/7w135sAKB/3QP4//JYbHs8+Rs/x3EqsakGKOnx47cXAICm4gfutppQKAcpAn782RXOLg7YNAO+vFvj0dkOh7HC/a7FKgb8fPhiAxQBt4/uUO5bvP23S2zPj/h3n/wxfvRwi8/eb1GVFlo5fHi7xvbigLo0eHe3wrdu3+IPv7hG3UxY1BMOfYX7ty3Wj/aQIqAuDSar0PUVlPIotUM3lHBO4nAssLjo4L3AcKggtEe7GNE7iWks4EaFop1g+gLoFep/7wN2//QK4pMjnJOQ0sNOGvJ3RvQ/XyEsLWAk5YPrCd5K1MsJ+88XePS9N/j8i3OUiwnOSWtcwYAAACAASURBVCjlURQO1kqYSTNMCIAdFf2cPm5/KBAmifNHD7j78TlC6yB6ibBw2JcOftAQlUMYFIrVhEU7YrdvoH9aYzp3aK47PDu7x2d3WwyvFyiuerjPWrJAL3v4dxX01QClPe7fNyi2A4rCoXvXQowS5W2H+2NJBtcKoJr4f+khdgXCwkJWDnU7oT9WCIMig1l46JWBPRQc30iWU0gyxtuLA+6/XDMMaFBoHx3Qfb4EJFBdd/B/sIR5MSLJrIKTEJqeQ3jB93UacJRBypdHmHcNukcC4ZxhTPVywvjFIrIBnivFFVeTxSQZ/lM73DkBWU3wnUbSNn7QnsFETkAsLaQKDGL6qoa4niiPHcm8wQkII1h1owLkUUFeD8DPG7J2twbyQXMSWHG5mR69eHACZMB9oL/yjsAj1dCEkkykOki4tYcwEoDD/qWA7iSmcwc5JWkQYvgPWfLpgmnBvvKQI/186qrD7nWNoCl51bs4kRecOIcCs+z02QD1kxqqFxhuHD2D5xbFnUaxk5i2AXffV6zD2ToUH9ScEroIDBxa8ZjDzQD5ZQ27CVB7nov6dZTTVpSw1l/FJNY6QE3xvWCIUnjRA581AGKSaRvgVxb6XQHhAHMOtD/TkI5MohqYfptY3iBiWMy+iLIsQB/oa1ODwMOnDJQiqKZENRQeZey+VQOvEwM+WKXTvBV4+FQCUW6rppgmCuTKkBDDc8bzALMWqF8rmI1H/YYqBspfydq7KqB+V8KseCvqLjJ1tcz9uklSK2z8WSNQvynz8/ooYgKymtngyBbqTmE6o0RzuCAjrgZABA05ArtXJYoDcHxK1lwfBSdRIYWpIKbOJra2jOeG4WaJAUNk4JLHrogyVN0VGC6B5i1iom8MkKmY1tx+KdBfz/2yviCrKqcAu9AnIVGAvFW5eiaFByWZK4CcjixNvD/WQHXH1wxXQHkPjGesDhJh7rJN8sOZIWTHblrMfHjBCgo1UMaaFgnbtx7DeVzoMyc+Qp+YYy7occEuHsNE0GMX6R5FBHZFlACHvPCphwK6p+eSgV9k0NLiXJCxV1qAvZAxhIaLWnExU1OGuiglhrNYUbILgChyn2uSG6eAoCQRdSVB72kNifAVTAss3rCH1BWUxALAuBY4+8MJ3XWRA3+SXNjrFBzUoDhE1jm0qO8CxjX3A8mFXjKaDaqdw/hpCzVykTEt1lFqG+t2ppDTiPU4V6UATDFuPvg5VCjKPfVZCVcyXTaxrNWONxH9sbOcFABcVXPRIdaUcNGuQblzsfJEZJkrLrntovPozzXqezJn7O1suFAUWWAufHDRaPEnR3RPWt7rVzXvM5lky8jX1ld8vji4nKqrO1bCpMWv9Fx6uIodqGmBJXV8zswsA4MApugWnYVZagi3BALPxbTVKPYOqrPwNUE5z1UEPB5wjYI+2o8qUrgDJuiq0UN1EaxIXifVG5hNDd0Z/j1UMblXS6j9CLeucqhPqjQJSsy+QiADdTEaylidi+qbFKhz0sNpPWAdgvcIC/Z5in5CTooVguxgYHo9tKKMNbHZTQUxThxrXcbPC59DivJxC8EKkwTK++Ej8JcXYhIg9Z5A9bTSxPvM1AohEIYxMq0ElSh03g7GEdDxvDiX2coQ4v5U7Nl0Ln/90SO+PsHRECJg/v8fv7GPbzRjufrubXjxX/2nOHy+xqNvvcVXH9aoa8pPp2OJop1wvu5wHEscXi8hWovgRUxW1cAgobcT7LHAp5+8xo//2ROE8wlSz2mvq8WAh4c2y1qrf95i/H4P7wTEuxK4GuEHjfPbBwDAh8+3AAB1kAg3I1NZj5oyyyhT1W9KqFcHmM8XgAAWLx9weGigKwv/2QJu5dD+TEP/3h127xYQo4KYBPzSQT0ouDMLUXiUtYH5YkEvxLlF+WUBs/GobjqmsU6SklIj6NE7Sti1o6ToQeV00qABX/noKxPZixV0QGgokw1loCwWAGRcbfeAv56gv6hgzi3Ktzp3XPqCMsLmtcTxBaWsPiachrMJoeNr5SA50ZeArz3kJDmGxyNCnCCKXkJHb5xdUkZprwyKrwrYlZ+BR/LEXA8o/kWLaRvgNhbFBw2zdRCjRKg8ZbtBRBaSQU7SANOtRfmVhm0ZcJKSUfUxTsoAIFAOKyeZvWSuSv2ePD3TFbtGXRXgthbiqPPPpSVQSAms0hIUmJWHGiRc7VHeqXmbHpjOPPRe8Nq+Uwg6es4eBMZLHr/e8/wGSYmoNALDpUfzOvpJG3rQupu5vzOlw4ZYo5NkqtOG/rmUvFl9oN+uiL67BBDsgmmi09ZDGoFiJ+P58dAHgmU5kelMclbbUu47XlKGSyY+5O7MNPn0CmhfC0zreE1Hvk4fBap7VoaIlEg7cPKnBuD4zKN+I+duUUHlwLSlf1JY5E5MVmPwvLgqSos7ysNcSaCWAEcCJNOWx3x86tF8NXsp1YAsbXX1LKkNOvrfHpCTThPI0D3HsP6ZR38pKZ0VBBS2Qe4IPT4OqO4FzIL3JdNpkRUTrubXcuJ7p1X0AtpYFRL7MXUPZNmipU9QOKD5KmA8T/64ebL9kT8uSRzfBEwbgWnLfepjPJ4SqN9TolnsKEm1MeGzfeMxbmXs2GSvpu4Au0jSVo7R1TzH4zlQvwuxTgFx0n0ip5xmySIEj9FrAhj6wvhciF4zr9MElu9LrLRpZ0ksEGWxAVmhkZ6vPgRWrzSUocLTP5jBng9UMSwYUqO7EBUqc+KpcJjTZiMYSsoOALNctaEkGhGolLsQK2Mi61pzgSD5b1OqakqYTRUXSTbs6vl+qu5DrmXRfcgMOuLnWlLtsPc3nndPJYAIAdV99FGKuF/H/RYdgZyKoFL3lHemzs9TeX5a6Cl6j3FFCejiS4PdyxK2FqjvfN6HdDPbnthmVxAYpf7QBF71GDCuKEWVFpgWAvWD5zmKx6jHmKAaVTlq8hjX6iOVQpAEnNNSoH3n2XV69PmzHQIYN/Qu2lpS8VDJrLTQY0C5c3OaKuYFHd4rs+qi3Lv/h70367EtO9CEvjXt4Qwxx40735ycaZcxpbagaQEPLZCallC3aBASL7zwivgJvPCIBAIhQEI8wD+giwd+AK1+ANRdk6vKWbbTmc47x42IM+5hTTx8a619btrlVrcs2VnqI4Ui4gx7Puesb30T+zOTqqm+degvTNqeCJmAqp1L1LcO9kgV1YoaA4SLsEtVPI8ySVULIxQi3EJBDhHV3YjxtIIaQvHu+UqmZO1YgF3u6BQ+Qg6eTKlINSeK1S3s2fSF0SfrnVQfSqC6G1mPkhQgwpEtzwm1AKB7z8RVy/3guAIQY0CsqISQI+Ww3lC1IWxAqBT03sLNDPTecj8aXfyYOSlWby0QAnzqt4xKQK8GxFoVoMVOy0gWGECUEn5uoDrL50hBsJdlrzGWHs0si41aQu5H1njkW/YKHtZsjI5gMsldY62RLTrIjHdKfc3gLSZgJ7wn8+qoNEBd8bnJiylyb2R+nZKJPWYdicjeyEMm9JBF/GYybPZappRXoXWRvwJI4DZwnd5DGIPYdUBd01+ZV+E9hNGIh6yo978kfc21JNE6CPMNruvQZ5mOaYwRcRjwL3r7XWEs/+c/ePYbX+6/++Hnv/V9A77ljGUtHf7+B3+KfzT7GC9ujvDk8ha1crhodvjR2/u4Wm4w0yP2TYWfdBU+uf8Wf/n8HozhTFaQCu1swGaTPhTOByAK/GvPvsKfvn6Apra4Wm5gvcJu1UDsNfaPHU4WHfZ9hfknO2y2LYITGKxBCAKQEfVJT5N45XA07/E2MthmsewhRcRKRvjeMAQkJYnGIKB1QN8EVKc9Otvi907u8KObOQNMjj0wSvhFwMNHN3jx5TmOzje4ljPEBmiWA/wrhqYY4zBIA3MyYNaMuHuzJJPlAaSQkSgJUEKFUmkRZfI4SHZLyj23y2wk7ElAmAXE2qM96TGOGr7XWCx7dKaCviUgC7Wnv7KKqN4obL8/4OpqhesfXXLwOwg4KyFaMpthOTIExwqokxHy5y3BohcQnUKcO0CA95kI6ABx6YENPYmLx2vstg280lBrRd/e24ZdwMcOzWmPoZsTOJ0NkIq1LXGQ8AJQbzRVOU9GHJ3s0b09gRzpP1SdhN4LjOe+1GwEk0eC9MPlm3/cI64qBrW0DvGjAeFdA9V6hEFBv5PsvJwRtLdPN9g/X0DtJdzSAxJQdxwIukXyxSjA7MhgiSAQFx6jF2Sq6oDuSYC+1ZRcCgBRlOqGGIC4dAg3FfSOg/jt0wCzSSmK88jj8ectnAFO/tYbXP/5BfyJg7rVGC896tca44VHUGRDVc++TXOjMF7Qp+orILYBGDREAPorTlx4H+GOCOZziAoCWBtiJVQnCW41PZZ2GdPgm8BTJoCQfZgiBXGMZyGBA4GgE6CYR0DSqyhCYosEyBD2IoVJCHSPHBY/0xA9GTs/D6jfch8oi0+9hIsIfzXAHlUQAWjeSAa2HBNg+UsCELvk+qqVwPYjj9nXCv0l+zjrayaUIgLRRPhWorpjCikANG8ImoOJ2DxlHUB/j+9NP0usaEzMTqq08A2Pn1mL99gwBIK7/Scj2i+YCitW9LcFTcDm2ohQk23Kks9gIvRAOXF/SUAeBRNd29fv922qARgXwPYpmUz3eID5qmaFx1qkPk1RQCZ7THNhvcRwThAaVQoHiTzPLvVj5gmHnoIT9Bf0tkbBAT8DiATDay6SxF/nihUOmOTAntFqDfTnAu1b9pqaLc9dM5LxsQuB7pjPb97xOb4lQzOcRbRvUCTucqTU3814DrZPgPlzHpMMFnQnsHtAa0Hu+5y9iggqYv9AvFeV4mY8X8dfRKw+kmjfxALofc3tr2/T8lM9BgT3Rw2sSqnWgNVZZg509wSaa4K84YR+5fEYEJbHJ7N/2UOcl23nKcBGTNJ55TPYTO8lNYFh17J7FKClw1cCUkb0p0wuHY4YgmXnEovnBLEZ5AOczMjbsE/eR2WB1UcVP+NnwJjqcKICWa4EDO1SwGxQZNDS8XrLNhexnwB8d07LBL2/qbvSA/ulQHMXsb+gDF46mVhkUYJ1ohDl+OQeUGUlYOjXzMB180Rh8TygP1YEPWkyRnS5k1OkqpYAPz+oQElVLr4SZQID4H4Npzr1tNK6IGPEcEJpd3fPwNW0COQE25g6NsNhv2jFzs5qE5LFhICgv6rhagmxUDAbXySFAODmBKc51dYkX6a9MqhXfqo9AVKolYebU94dJTs1dRdhtg7dpYGbNTAbj1CzZ9NsPcZjjebaolqNGI8rStCHwLTYO7KQZufgjigtJhNZodTIaAmRrsWomPA6nNWsZwFtAbqjJNq1iqBYqAQGCcihBNw8BfZoTi5zokEitKlTU0v2WUoG+ojeITa6yKX5uclzHhsNte4RmooBPglQihRAJZC8qEIgtpTd+kZDr3tASkRDQBlaQ7tDWx2E+PjCCJLpDJT+WgJUYV3ysgKxqckEGgMxWN5fV8WrGeuKAHQYuY8p7IczRwfA8qDzEiGQscxgVCnAOXomc1Jsfo0+gBGVmdjEbODOIDtJuIvU9xDIAhCHvZv59ivCfIQQOHjGv7z9Dt6+1Yxl/fRJvP9f/hcwtxpuFqD3EnrLEvvseQxp0G22TAwkq0XmQwTK3JjeyPtyOMZwEUqCIT2YEQiCDM1pgN4yYt3XlM4BaSbaZUmVAMTE8lBKxpCRKDijbza5r4+DqijpXdOJNcnBHUEzAMTNchciv3h9zf2Ujvdl72YulRdJPhYFw1dEEGXWuAQYHNxyAqBvMqNDlkd3BCZIMqbxKKZEQlGYNYBsTV6vtGm2fJ22NQU0ZObL1ygdYtKhJH26Oe9DAnEEBskrmaRmObwjb7/Ps/MpTCN3yeWwG+EYRFKtUPyODIDI7AEHPDENYLPMLqoprCOvL0oUFiUHueTtP9zvfCwLKGgZkhJzwXpab5bnlbRPjxIuk1mdKNN5qfm4XRJc5F5BcQAWcjBLvj/KtO8poMWnrr2cXlnCNZJfMrMWJaVyAPQuor/gdYAk06s2vB5CzR7ESfKYfMtJ0WK2sRTJ624KWsmMC5CkaMmzzGsE7w0imZYZk4wrMTiJDbRzggMAKZCG2yEtJV45SfK9pNUE1gh6phAMSusSmzFkgMT3j0lBMs07epQzY6U6DiLMLqI/SwPa9D1brSOG4zTb7lGYM6aDch+rDdnnfK50xwGz6if2LJ8TnVIsKV2LKV1x6j3MSaH5WiGbw22jNHUKc8nddcCUilltyU7VqwA346C6ueFnUr5WESgvlDZXFkyfHVniyvWR0XINwZxrEsMkpx/1jU7IQx+snYvyXg85yX+f+9rS6RKTj9a2ZIuy/JWfcdxHsltJipj85vkzcFzkdNDU5RdRwIx03L5qOyV9muztNQQjmYEyHYEMQ3BSaEyIrIFIbFvuXaXcL71f3bTtUQD12qM/ZRG92YeS+JnDofJnT/YsBy3QvhnRXZqS6CltLJ2KMcn68muKBx4ogTsEJVNnIEvr32dVM/ACOIhn6uYUMpTPn7QEGONRAknpM7ra+Pe8z+yR5DZUm5A873JiNlM/oxrol1W9h2/o76SiQKYuy1COaWZfdUdmrIQTJcBIfy2ZyvFI0/fr6TdWnS+sW/7uDVUOOGIaKc+th2tV+W02DqHKfuIJrGVvqWuTf9jG6fqOiX1LCaL5ORlEhUrSP9374kszW7J0zIiYmER+foryuRWlKIyetImdjDkohuxhrOSUlJoYPuFikU1mRrN4UrWAN2l7chfkQeCL3FuERlMGqgTUdkRoDK+RzhF0pfernxvIMUD2FqHSxWMaDZlEOfrJz+qSvFTL6ZrM22gU1KoHtERoNGRnSz9kNAoIEXJvi080p6mK0fF5B4E7USmIEIr/VPgUwJOBTGZzhaBPNN8OuyaNLseDJz5/IYuJbdRq8nAClKMmr+R7MlMp0zXn32cWAwEfDlk9IQpzCR8m2Wx+bfZ1CjH5Og/ZR+cgjHkfzKVbLExmYicPk18BHIbp5PqPmMKECvBMIDXGSGYy/f3e63/V8n7dLYZvvcfy0x+08X/8gw9+48v9Ox/9xW993wBA/rOf8jt80/TO2WOPOPewxx7jCS9oe+rg5hEP/sYrNJ+uMJ4EuBNH0BYExpTm6eeM1w8Lx6j5Teq7qmNhCVQn6I3yBJihDuyuqyJL0TuRvojIAFJuygHceBQToyDQfTSWwXROwfNtRHjQYzgN7N4zkWmUAzBceQwXHtJyZl9aMYEVTbCs95SeDecBWb41nrD0PjMx9S1fG+rIlEgA7WsO6HSXJHsSJUHTzXKIzVSwnQf9lBIyxdJXHMjpLWfH1UgJW073VAmQsvibi6lvcoF9nimfUhlzObY9DtD7BIoVy9EzYDLb5N3qeN9wxtAj1RGIqQHoz7lunySPAEHqeBIxnBAI5SJ43ad0Sk15ZQZxGcQKjyn103EdMkmb8vp8PYGkfKx0iuR3c3BAskWRYMq0nJz2KZM/zez4vKjpudt8HAoDJyIwexNh1sDsOY9Lcx1hNskzlZQhuktl3fv0excxexFLcA7Pea4DSNI+cIY+A+lqzdeabURzE+DmDFeiBI5yyCzDNNsp7VU4HsscwiMc0F3wfFdrnjO9j6hXEaqLJW2yfRsxnMZyvKVFGdCqngN2NaAE/hCg8rFqwwFljrCvbxl45doUCBUZq1/dpWu65/q53XxetaEfqtry2p1dB3RXfE8370J5vuoYHKV6lCCuqBKYNEworTaxAKXhVKLapJTII3b4VRsGDS2/YghPtUkyu8BjxNqOWEKL2puA+o6yxeEkebEcUz/z88yOgWNMYWXolQhcl+4j6k0orwN4f3sd0F1wIs3sIxavfAJyEdUuwOwig8b0BILMLpaQITuTnJwYD853YKhWljAORwL1OtDjlWRx9Tpg/sol5ofezGobKO/dhwIU63VaV/rcye8T0wU+b5ySVEXgPtR3gaFTd6H40eq1R73iE6WLMPsIswsYFymcrGdC6xTcxGMtPUFe+y6guXGotgG6J6Cu1h7KRlTbUGTewkfUd7546KptQHPjUyhW2lcfYfaBrNAu7YOPaG881BhLvURz61FtQ9m/HLQlbYSyEbrjMfWJSRpOTUpBRQFSvpaUhu4D9BDQXNsD2XaAGtJxHAjqTApFki6DPc8JgQTW6M0kkLWtQPvWQo3cR9Wn7x5JhnB3nwxktaFktVqxXxiR99W3FjqtW0TAbCwBXJK1inS86lubvIuhgErpKcOULqK9thyAR4Irs/EwW0/gkq51OQZUtyP9i52DTvUSamSwkxwDqrUtsl81hMISF6/ezsNsXWHHqpVlQNHaIupJHqp6D721qG56BjaNAWbnCEzSfoVKFvYsA7tqbaEGppPSsxdQ3Y0wWwu9c6juBrg5VVV6ZxPIElApuCd7B1XvUsKpKnJN4QLUekyptgSVejVQMbCzrNDoPZNSpaBXcyQoiVUCWmNA9a7n9g6pWsQT0MuBwDFqCb0ZCCSFgNoOXA7ADsaZRlQSejNC7Ub4WQW1G2AXSe7pAsOQVPYs+hJ8JHoH0VkC2jz0u9kxrbXRkPvc7RMg9yOEC1OoUQ5VkunvGCF2/SRxBSg1FQKys8XnCBcoWVUEZlEIiH6A6EcCuJGhRFCSQG+0fMw6YBghhpGs4rYjQMwVIrMmTUiRVYyV4eRPCgyCD3x9qilhUJLh/Tl1FZiAY9fzsX4ArGW4UdcjWsvXDyNBmHPsmTQaMXscfaCMdRwRrUUcxvd+YEe+Tkjk3ktozX3M8lbvJ8ZSpTTcDC7zfYdDI8/gISEEGUopyv/0W8pf+ZMfgxS/tMxv6y1A/MZ/fldu327G8qPH8dP/7j+DVh6j01g0A0anEIJEiKTLu30NbTyGzkAbz77IysFZhap2cE4ieAm7qrG42mK/rSFVZO+hjFAzB609n7OpoBcWrtMQJkDIiNl8wPZmxkCZAFTnPWyvsTzusLmbQdeu9CWGHcvM5NwhDAqypkwSVgKSXYA4HQniUrCMUBGh04XFg4iQa40w94CKEDpysmlQgA6Ak2n6Nl1k5uBvOc2aFi2BIhOLIIAqUDI7MPAGXkBEkUBELNsYsx8zVXyUKpM2wKwU3DJJAFtfAoSiiFy+itBvKyAC7oJBMmGWZ7eQlpv8eVYU76XwAnIU7AV0ySvXRiYCrxR9j4Okn0UDYZ4qVNK+Rk0ApDoyHm6WQmtULLJKaQX8gh5M1dP7GLLcNQLCCagx9QMGsL6kz2mpfL60E/sVFdlcd1C9wVAASu/omyKAiQeflWZHJj3UsWyHdJMXNLNtqmeNR7WmZzMYILQBei0L28iaEpFkaNz23KNoNpPXMiffZjZVHk4IJhkrDkB3tRKwy1gALSLZ+Jyki8SiqWHyQeZzk8NeyCSklydvKAdg3O7MwKs+e1JFmfBhqAePjRpSd1+dJjzSczNzlxnYnFILMR0bLiv3TGLqZWxSlUieHE5AGpjAb06Izt5Zu+B5yiqHnBacg3Nyl2FQnNBwM06I9BdULGRp4OE6c4VHZpze+8me1IgyaWDnaZJBoshIARQ2eVJgTJMbvpp6BgEy+75BYdsPAVzpskvJ2ZkhLEnQ+mDZyBNWaaIlewTz211M2+1mia3PrKZACSnK6boAynWd+xKjEuWazc/LqcY51MY3B8E1bppwUmMsbF5WN+g9r+/s1yznPAEr3/CCZZALSmdgVh1kRYibJU91Cs8pHrs0UZcTn4MmK8b1ERwGNfkic2iN8Dyn5bM7K1rkdNxlYo2l5eSLm4kyUab7lGQtUjp1Oh75eo8aKdWcx8/ORZnMA6Zro1xvCcSqMaaeyPS+GslGZi+osglQJW9rloKqPl1bCZRn1jcnRucka2U5OZJTsZmSmydEQkkePkyzFoGsaK6aySDzPSVHOu9ZbRIVCrOuxlium5JifTDBSFlrSgV2qZbEp/3UAnpgwm5OWkfkhEDQooQclVTlg/ewHKeaETlO25wZd6p4RHkv5PejSOvOCe+QE1tPxZMovt7MBOeaEZESfX09JUZHmapcxgDfpqRiz2Tc/Hkt/ZTwDPB4uEYxMCeiLI9hPb4Aat+wC1JZhuDonS0JuT4xt5T0Uu7qUwWKHP3EpkYm9UYB5MoPAPQ7JjAbk30DPu1zYsiF9Qi1pr/SBfjE7Moh+yzTRIpPqbapegTABGxT8i9c4DoEJa05kZkLSucv/R+lJNuZk1M1g7UQAoGhVpMk9VASmgFc7qysDNlPo8vvaV3+fTY0s6DZt5lBaU5xTV2WqMz7bGHGBKkiJIYAcShJjanr8jBlNi9XyF9Kti1VI3m9OFzUwXoPE2N/HXv514Sx/B/+4MPf+HL/7kd//lvfN+DbDiw/fBwf/Ff/OXTlEYKEHyWEihDXFT2CnSKTOAuQcwv5ouGgOgChISiRg0ygCRBJ0plrFcLlCKwMcsl3ng0G+AGuctDNwLAUt2BoSVAoKZsy9Tv6GZ+HQDADJBBlKcNV+ynkhANVMoKhSsEqAXBHAWYlkySW23QoF4wJNHJAMAEduwwF8Ph5QPtCYTw6ADYgCMoDVpEYVrsMTJZMQMYtuP6SkLg6AGme2xxlLN/aoQLUnmAgGAIqgOxh1Cw8Z2hKSP2DKCAkD4jHk5BkyyiDPzVmtlQU+Wne/6hTUl3qSczHjl9WByE7gmyyGvKXLpcRNNMuc3l8kbxqlGOdmW3fTmxvZl3lSKmq7lKQSDt5yXIiIzANCmJKJJQWsDNuV7UCunscFOgdAVx9I9gruJ0AYB4kHxbAl4FGlke66X4AJZExM7bABBZycmZJhEQewL8vrdU9CpOVly9tCmuxEzjNg9Y8KC7HKe0z5dwoNR0czKQ3t3h/4JTL2n2NqWsR037nQJwssZaJBcxgI7Ph7wVpRD4nv6eDmQaaGVyXdMp0fwZN0qVycYMS7PEe0PPT//Rv5o42VkbZUwAAIABJREFUFJmw6pnwqTo+5lqy0AUQpG3LMrxcBJ4H0PnY5O5Dvu+5LNXxMy0olIEkN5I/ynJyxKdidXY0igTw00B/N1UXZWCbgaTueL4LWKsn9j77H3PAzqHEtMjTkwIg7yPDTyaAmm9Ffq4Bk8FvmMBgnojIKayZrS7XQZKrl2OVQJkcOTjOjFwOj1JjLFUyQYtJupkG5wX0i6w0iOX9kmW6ua9Q+inIJgOKfMvvnTy5IlOvnSyyu0niGAXPUZ5MiFLAm0kKq2wsXsNYjqWA7gm8skeuAGiZwJagqoXvr6n/MJ9vXlPxPXCTa3UOAQaAIm1lvUc+p9Nr8j6V3skk5c5gK29TPkZF9hu4jkOQn9dXuh+BIgstE6f573yLZGpj8SIKqC6k2pP3P0/yJFJeBzB9fk4XJgpYyxLUzLqKdGxlYkUPz3vZx8jX5moWXxNMiBDLoLpUx2g5AS4j0vs7lO2KigmoIdWTFPAoJgCdAWNmT1mLc3B/zNdfeA+05cAcAn2Zzs1UwQIAqnNFBgsgVd8ISJvAYNoGnv9QzlVmJwEU+arw3N98Hvg9MB0TXpeySGghBIJRieETBZwIn0BV9iwmAIgkpWYy6sG68vOASe5aJJt8vXChAMTyW0uI3hIYHoK3Q1lrDtgJSVKauyetIzDMUtac2JoAZu6pLOfosNsyg9EQ8J5U9vCWg3kOfYtZOpulqzFOlSD59s3l5b8Pq0C+KdE9eCz6gPdCdw7lsN9c1+E6/zlu3/Yey+/8oI3//R98/Btf7r//0Y9+6/sG4NsthdXa48HVHeazAX6rUc0sEIH2gw2qkwHRUFp6+eQWYWsw++wOUaei+AigDghHDoiAPusRL0aoXmD589T/llimzCipjsApNAGxolQppgARe+QR65A8UBG4GIp3zzcB5k4iXIwEcDoiLDzCFemeeDayW1FFuAUlsXYRYC8dqpUsM6vw9ImKq57gq+Lg0B4TfPkZt8meBj720Q7juS8ArL4RaF6qMugKFTCeevg2wi5TouhGwD0aMDzmsUDgfcIDsmdvpvrOFmYrIP7GCnYRYS8cqjsxyXklQaU99ugfOtgn3E/G7HNb3Ylj0X3LY9xfUBbVvBHono2FFTJrsofS0WtWrQSGMw/dCQz3HKKK2D913I4lZYCz52ICJYmtGq48pBMYjwNsSjZ19yykBeYvI8aTgP6BI5A2MTGAEcOHPfbPbGEuh/OA6o7ngVJfsnbdI4/hkscyKkD98I7nVk1yX2mBUAPdZwP6ewHdQ8p0th85ynA1BzSbDwOqVaqRqJmOOlyQwdw9Dlh/6hAMma76NqK7PzG+WfprjyKGy4DhLBbQMZ7wfzfjczPIkiPQX/LYdVeB12zL7TE7YPeBx3AeYBdMjO2uYpJmMyV198kIX1Gqu38YUK2B/VOP7pEr7JjP3aGZLUrgtL/yMOtY5MTbDzyTXa8jdo8okdzfjxiP4tQlO0dKvwS2z7jvwQD7B5wsqdaxsCHBMIxG75gYm1nX7bOA7l6SmKZjUq1jmhQiEN18wGOSJbS7x2k5jyPuvhuxfcxBpRojNh8HMl0G2D2aUh0J2IC73wvoz/ja/cOI8YRdp2YdsfmI2x4lcPt7QHeV+jErejjdXGA8Flh/EtBdUlZLWR29u7rj87dPKe30NTCcCuzvCwxnnAAJmmmx3VXq/DxliJNvge0Tgf29KSl0f58D3Lvvsqx+OOPgeDwS6C4FhhMOnoZTkR5jgM7t9/hbBBRmr7sHDGdcPtlSfpYMp4Iey4Z/d5ei+EurbfL09rx2u0sGodx9CgJzw+t2f18kOXxavgVWH0k0dwGuZcKoXYgyILdLUXyBu4f0SfqW6+nPBIZjge0jCTcDdg9kST8djiXsjPu/e8jJBF/RzzscSayfSdg5H7ez1Eu843G++0Rh80xg9RFTcX0D7B7SUzkeE7gMZ9yG8UgU5m3zVBbQ5Cv2vfZnEvsrCTtjgNHmqSygNwoBOxPYX/F5eojY3VdM262YgNtdSIzLKVV1/ZRerv2VRH/OflVlWbexu899Wj9TqQYkYFxKuJqP6Z6y42Ep0Z2RVXKNwP6ewvYRa366c4HtQwU7E6XnEyDoXT/VcA23aX9PYlxKbB8qbJ4ySMabadJ0nEvYhcS4kASwCQjvrlQBkWoIqNaOQT8zejPlSAbUNUnmP5fJ00457/6eLv9nhs81Av0Zk1TtTGLzWGHzSGF3X2FcSOwvFNnGiszb7kpjXKoibwxawM0VunOF8Vhjf6lhF4qMWQD6M10merYPDUSM6C5ZgdJdUO4btYRvee729w2ZtVpiOGPAjN758r9vFD2ZtYJvJfZXptw3nOhS1bF5UiNowXWFyG1KbO/ugQEiw4P29ypsnzSwS4PuooJdGvTnBuORgV0qqM5BrwfYpYZd8gLt79VMnN2O8EbCLg3cTKG/bEpaq09+UoDA1B5X3OeHLeTgsL/PGbz94xnskqExbk75rHCpU3ZZ8T055zF0yxq+NQz5iSjeSDc38E06zkqUJFh1s0U0iQ0UAr6lDNW3BvasgT1tIDz9nqGeZjFEZv0Ez1uYVRAxwh81ZCobQ9+mpPczaokwq6ZKEevgT2YIJ3Ne/8czQEmG8YTA8B0AqEwBmznZNXtKhfN8fgrfgWbiK6Sc7m9rgk2tWDeSQGysDeKsKf7L2NZMlq2r0ocppITQmr8VA3aEVgzq8Z6/lQKUgjCGvkwp+Dt3ZgoJiCxfTfueE1+zxxLgevKPUkUWK4wmUP1n/fzL2+/87VvNWJ597zL+vf/972Fja3y5OoPzEtYrGOVxMutws5thGJn6KSV7I7e7Bkp7CAEs2gHrXfPeMp1VCFayqqMOVBKYgLDTgI6QladUdlAwjYPdGUpNK05bS8PHvZWIg4KoOL0evYQ0HmFUkFWSwAIo3YMmsJakoZTEzC3s3hSZbHvWoXszg5ilL+g1+y9iHQAZIbdJHusFa1X2mumknvsCfSBzbQK3a6cRdQACv2BEw6RW1B7oFKW2Y5qltIJAfc7tgxeQMwe8rYGLAXhXI7Qe5miEXdUQjQc2GtXVHsNNCzGmZFMTAY8kI5Wl5kRYgdAyMTXMPOROIfcg+oYptWKQkAMBtNopsrlJkipGMsGxiqjeqSJzZBl35Pa6aaa9pPEFpD7AHOIUYO6SnDekmdXGQ20VEAT80sPcKQQ9ySgBkJVteK7lnl2KEEnCKg8YyjR762uGGAVF9lzYxI5awD4aId8ZsskxSUFnpMLkKIoMzrcR1WpKq82MRA4lUsPEVpd1blIYjEWa3Z4krLlSRabXmY1M6awE2pQKionVyIP8xGRJy+2sbwQTWj23S3WcGBAOsMf0IIogEBSBlxxTj2Yky5zZe9/EIskNOqW21gQb3A8GUuUZ5fE4YPZcwrfT+zkqwM4j2fEmQnoBYZMc1wPVRtAHjUldkAG6PSZTklnuLM21S55XJLYus5LBJKA5pPOWgqL0nr9ZmUGQnRk8NXB5rBlhpYj0KaAryROlB4YTsuul1y+rJzJDesCaZrannKPq/cdzGNUvsVMHLK/qmWSq+hSO1aGEDqkBhfnMkt+idsiT/Enum0OyMnuYpayHgVW5pgWS66xvE+jqJ6Yos+6sueExPew2zGFXuYsys5++RpH/Apw8sMtpsjAHOEUx9Rrm6zmqg88Jh8IA52NW0nV1ktUqUY5rZrpcM7HFqp/qLlizw2oS6UDZZmIbs4c4Ky18nQKQWj4/M8tqjOWYlrCyxD6VYDF7oBD4hqSYAUEx1WxkBo4yXd1FZOlpTMFbOYAps/8A3mPslSUDHNNEVg4Iy/Lawlym86X6NOlUCdQrSkujnAKSMruWJaeFWUvHMCgCdE5aTYyhrwSqTZiYwsQuuyb3bJItLYmyKWQKQAn3kj5NgmWprEQJUmpWPl3jAv0JJzKKTBwoyoLJmxyKNDtfByEFQDHIKDHlST4KZGmpTIqcWFjnYAias4+W7/H8+S0OZL+x+EN9o9I5iRNTGTkpDSR5qo3InZaFic3sb/6cyUrNgVUhfF8EhASyRUSRx5LllsW3GtW03mAYdmR2Dq5RJVQpHzvhY5HKMlCNfxcmFiCDm2mRAxkwv+9T2NJhIFUEQckBc5cDqjLLGKWAHByXkcKRZJaypmXDhUkKG+MkfQVKuA9TW8mmlgCiwuZOj78njz24lb5LmySkWYYqBfswZbovM6CHwT5Z9hoIxvN+FSY0+Ul53NK6lfzVrGdmHVNFSJHD5uCfwxCeGIoPNI6WzOSh1zLLYPM2Hi4ffwVz+qtueZ++9YzlLP63//CT3/hy//7Hf/Jb3zfgW85YCkT8fy+f4Iu7M3x48g5SBvxbj3+GfjR4t5vhYrHDs/NbeCcRo8B63aJpR4QgcLbcAQBmzYiq4jejfdPi+09e4t/53o9x/+kNdGPx2bNXWCx6qKWFah3CliOU+TGLvszcQi8t/vb3Poc0HmenOwhBf+Tlozv825/9JYSM+OTZa8Qo8OTxOwgJ1Ef8ZjathTAB5xcbnD9jdKeaObh3DWTlcfboDvOLPebNiOayA2SEbizkxYAPv/cSZjkCTuLR915D1B6Lqy3q1uLo/gbihqkvzXmH9qTHw2fv8ODTtxBVgDQB1eUeamkh5mRtVeVRnfbQbyrorYJaONz7+B3ah1vUT7eICwdV0zf57IO3iDcVzj99B/V1g/kHK4hewfYaYuagKk8w+uMFgfXcsyN0L/H4szcEkk1Ac9EREKoIfTTi7JMbVMcDLj+7hrjXwz/ugcuB+wnAPNtBBIH2kxXUIPDo916TxTix9DE2Hvq7a4Q2INYB8w9WqO7R8BXOLaJJADASSITLEebjDVnoqwEwrFLAkUWcOzTXEuadRrgYEaoIuZewVyMHZrMAd+YQn3Rkve/4ZRtOLMxGQD7ew554RBXhq4jq4zX8sUP7mt9Svo5wJx7VjUwywojx0nHS4EEP1RFIye+vCRCSpLi+FrBXFhDA+HFHmfRxgD0JEJ9uEe4NjK9/OpQkYnvPor7mjL2bBYz3HcYrh+Gegz13rNhoOdhw39nz9fccumcj5l8D9a0kK97Gsr7xgU0Dc8CeO4wf9KjWAt1nA6pbQbZ5ELBHEeNpQPfxCHc5kl3+TscQpr1A/4zHc7hyhYkPn21h1gS4dsmBynDpMV7x2HYfjqyzGIH+2Yjh0qO+lth+4tA9dOge8Pd4TPa3v0/mfjwO6J44svBBYPuhK3Lw/sMBw1mAW0TsnzjUNwLVnUB/5bH/ZIQcgd2HDnIk8JMW6C/JgDOYSBSgbY/IoA+XHnbBrkg1crBabfj5ZTbA9kOHsz+LmL9g6NTia6C+YfjVcBownkZ0V4nNf8fEXbtgEJVrySDvPrLYPfVkRC4DussJkEtHED2chzKY335sEcVhHQiBxu6xx3DGQd7+QUS1ArYfOzRvI7fjMmL2kgPh7XdHuAUAQTDoK4YXuRmXm8OwAKb2VquI7kGEPQL2DznQ234QSjWJWwDbp/Qj3vzQsZP0EZnd3P9Z30X0F3xtrv/IkvHmXUR3n/uzfxhhl2T+qRhIoCmS/cySZ3tE4Ln6DmtCYpJwuzkACewfEQDqXYRbkPHNQN1XKMm79GoKZJkpBCtAtk8YKtS+JQM/HqUKF5HY1pE1IghAf5kSgmfA+mMOUvvEFEdNFjQz4GYXMXvDgLPdY4HuHtlYCMpjoxTorhKbe8pKhfFkqiLaPOPy1BAxHAvUa6ouhlOyrlmG7msyr8OJwPpDCV+ziqO5Jau9eyCKLHk4FeysdLzGxiORApACXANsnpItHU5FCcyyc4HZW4/5a4/+VMLsA/QQsX2YnntCpjgk4OhmQHfOx3xKbN1d8f/MMnvDbdg+VLBzgkfbcttcS5Z5XHLCzOy5//2JLIx8lj7v7ktU24Bq41GvPGavLXwl0N54bB5rjAvJypGXDr4S2F+Sce1PZPLIeqghFrDPsCuylpnFba8thiX9mft7Cmbr0Z2z8sO17Pisby329xS7LwWZ5919DdcymMnVlMWy4gSo7iyyx1H1AbsHFcYjifrWwjUSaiBbXb8bYJcKw4mC2Xn2jkbAbDyqFfshVcdqJ06g8b3cnypsH1UJMLJ2xOwcgiFbyg8dAd9KqJEyTDcjYzucati5xnjENN3+rIJ0ZGzN1sI3CnrnUpq0Sr7RiP6swnisi1dXDR7DuSlhZCJEjCcG9tjQm7lnEFGoFOtXlgZurtHdqxEqBd9q2AVRf5QC+q6DHBz0ZoDPbKeSCLVC0GSDRQTc3MAd12Q5Z/Q2+tYASVIbGl3Y0CyxjUbBHTcQA0OcwqwiGJ9VENYh1IYsZ5ICh1mqLhECqCuE4xliYxBmNcRgEWYNw4JSqm+sNcFm9lpqVVJjo5IIixQW5ChLfi+FVoq0rPQaKQvjSc9nAoNtQ2YyMZZl/G3S+c4BPzpJd3OYTw7yAabXSYnoffoJ5Qchlt+/9icGvCfJ/ZbeIoAA+Rv/+V25fasZy6PPruLf/l//Y7zaLdGNBk9O7tA5g95pdKOBVgEXsx1ero+wWs1werrFYA3GQTPcxwso4xG8wsf33+LLd6cwxuOo7TE6jV1fwWiP0WoY49B3FbTxmDcjVpsWi3mPXVdDaw9rFSeoRoXzsy1W2xZKBYyjRgwcdJjawTsFtzGQcwdTOYydgTIBbl1BH41wg0I1s3CjwmwxYPtuVuC/0AREQkVEJ6EbC7utgCBgliPsbQ0x84ijhKg9l9snc+AoSxiPOBoZCOQFWck6T9NGMpbxgL0UYGDP0vG3Dji5v8HdzRxSB4YL3dYoYTXHI/y6YlCNjmQ/o4DYK3palzYdD4E4Sq5/pxBnDCOSK4OwcICTEG2aLt6kD8qWz8GgyvL1EScK4qqiv7Pih47oFGIVABMgNppMq5WAIcOLQUHMHOTbioEwrQdk5HNNhNonz2fry/KgI6IJkDuF0ATotWKQj4iIMw8x0NMrz0be96pBOHGQa26/8GREIUA2udOFbZN7WdimqOLBbCwgB1JB/tzyPAoATgCth7wzCDWZ3qjYqUffTUS4GKFeU2YTBQiog6AUec5tNu80Qwo/2mJ8OUesAtR+kuFFGaG67N2hH1Y4gVgTZAOAP3KQnYJZs4pHWPZGZr8sZbWJlfWZtUrhSFWEuWP/HbeRkwzmTiImViskP62vUghQYuQEUOpuIpBCe1DCfEogSfoesscBbZKC2+NQ2MgS3tGlap46wi8DZMf9YCqpgFvEwgC5pYfeqOT3nGqLsg9OdyjevqiSXzewWkh4sBt2nljrFKZEmTxSwjLv13t2TJqVhJ/FMiFS+krzTHqkTLu+SeckddNKLwpzLNLnUA5j8k06dpL7XN9xYO1nMYH6lJJ94NV1MwLPaHhMZPbbZSVAPKgGymzXbupxzD5MgH7h7h6PgZuT2S61GIUdIaDLYCdX2WTPXWY6Mzs2Hk3MW3NDiatO7Kveo+yLTWFB2XeZA4j6s4jmRkwBS0kKncNuoiKjOpxM1U3Zk6d3ZHej5HOCEhiPMdUjZTWB4+TCuCRYtkuB+oZAK/uSx2N6rbMP01cTG5x93YfsrBoIOn0zecbzsoCpViiHSOXjqXoyibxmJ/8fr51YQoiy/9LXCXi67D0lu1OtCZwBQLhYroFQibRclLTk7C3Pn4n01icmapyUJjkAys7fP1c6bbOdiZL0S4/3FCKUgRAiwViuWDlkTgFMNTL75KONlAIzMfwgTCgcsE7puJhtYqXTcxCRkn2Z+pvB4SGLnStvgqJ/Nq9D2YnxBDLjTzluvfFwTfIzjmQAgyGjdrisLBFGTOnDLZlIaSODgtL7lDYCBggJH+HmiiCynljUsiwAeh9gc+/oGJJaRpaJjsxEMtiN3xWqD3AzCb1LzGJEYZ0P/bmulTBbn9JoyWz6RrIHMzGhxdZwwMjKkfJg1sUoqN6X1N2oBFlTN52zKAX0dkRodQGkQBr2jAz0iYq+yyxHVr07YFLle4ynHBzDfUZXwFquTYELiEaSITz0dSamTgwWueIk153I0RW2tnguAcCH0jsZaw3RjYh1RUYzxMJAZsksPc6J2f1GtYiwDlAKJdUX4OOH3s18H6btzX/HkZP7IrOiOGAabY7fjxPwSyxo/CZT+k3McejH/DW3Q6wSh+HXPPPX334XGMtPfjCL/80//PQ3vtz/4OM/+q3vG/AtB5aPv38c/73/7R9g5ypIEbF3BqcVE0mGoPHHbx6g1h7daPD09BYv10f4mw++xE/Wl7jrGvwb97/CP71+BCMDOmtwOd9ib5msIUREZw32o8HlYofdWKFSHkoG+CBRawfrFVZdg3k94uFihTf7JWZmxNvdAr3VaIzDSdthbw1Gp7HtakgZcLkkq1lJj8FrfPXyDFf3VjAyYNU1GK3GDx68wE9vz7HZNTg72pd1+iiwGyrsNg3OTneotIORAdfbOQFWFFjOelTK48WbEyyPupRuzvO8Xrc4P9tCyYB3qzmaxiJGgW5f4+p8hf1QwXqFWT3ibtPi7GgPJQNutzP0uwqL4w7dvsbpMRnf/VCh0g4uSMwqi9Ep/vYKg9Xo+sTwtiMGq2FHDakCnFVQKqBuLEIQ6HY1Hty7w7prsNs0mC97LJoBb2+XmM8GbLcNohc4PtnDRwHvyUI7J3G86MuxWe1a9NsaqvIwlYP3EnbQECJSfqwiTMNUYKkpX65qy6LvrsJs3iMEif2mxsXFBu9uF1gsenS9gesNHj+4wS++uIRaWvhBoT3q4ayCGxWkjlDaY9xU0DMHbyXU6xr+/gD5poI/tzAtJwPEoAhSZ47pvzJCyIjgJOrWwlkFf11DXgyYzwbs9zXcuoI56RE8EWgMAmFDf4yYeUQnoFqPYCVU5RECC8xjDlN62wDH3IZxVUP0ErFN7O1OUW7cKeBkBNYGs0db7K5nELVHdJyYQBDc97VJIP3gDTlIyrA7zcfSfTCRUu+thrnsMG4qyK0mgFxJjFcO1RtNtlbGsh7RekQvuAwJTjy0DmFUJTUZfXpMRIjWAxtDwCsjQpO+/GoPsdOcaMjy28YhriqYO5nWi2ndEWi/MnD/yg7hecvliAQQk1TbXGvYEw9hJWJ7INOeB+iVKinEvk3bkCcLIsjWn9r3JgVUShcGAL8InDhQEbKT8AuP+dUO/c+WJQk3tAnAB1FAu1lLjKdkZvN2lOCqOk6vTfJztZOco0ieczmQPbFnDnAE1XrDgSKDiwhEQxVR3UrYZTqWKdAn1xDloCUAZcIgyx55X/Kug+B0eDJi/nnFPtS9SAApy6y4LL0X6O6HkkastwJumbqHgWnwnmTjWZptj1i9pPqpz1cNTIEOhse6ea2S7zOWxNn6lv3D0goM5x7NawV7FFGtpmAvyrjThEaq3JGOFUh6J0oPcA7woswVJWQqKoLpEj6VvKNmy+eEmpMMkFNqb5YQ5+erkQDuUG6dQVnucM59lVnaqXq+himuE0g/fF6+HYY1+SoFfgmUtOUpNAolKVn1KGDY7PLrEtOc0mbHYx67HP5W3YkyeWHWEfZIFCA5Hk+hViWoLQV/1TdxColKgMHN0rnIoWBqCgtCQJH/2sW0Lc11LDLtUg+05zaLQO+w7gj+1RBLR62IcZJgj9ymLOPNEw8MyUMJocrnLOgkRVZg32mLIrlWPdCfiZLwmycYsjzYJylvvYnoT8gAd5eyVEVlGWSeDIlSQHcEm4isw8nHTYRYEppzim35G2kiraH3WY2sRMlJuUWinBJxMwiUltUwQTFEihPPXN6h7FiOqW8YB3+n12dp+BTgJVLNUEyfHaEwsznkx7VkUF2rketvdEeQ6eYKeufL59lhABgiwbJZuxToxPClvEy9pwQkVARmWZ4rBw85eviZgd6OgBCU8CYQGioFOfgC5OTg4OcVZOcgR1e6PLNENBoFkcCqGB2ilJTeHqS7CusRWgO57lgfkvo3fylVNafCZpYSYIgQMlCW5f/3wF6WwyqJklR7eDsEgiEgZmB6mA57sKzofdkOoRQTXb8hg31/+X8FJomBzOhfg1TYT34wi//1//HZb3y5/9Enf/hb3zfgWw4sT757L/6b/8t/gje7Bb579ho/W11gbkbsbAUfBRbViKt2gz958wD7fQ1TOSgVUGmHi9keP319gauzNV5dHyPsNM4erjBYg901ewTU3EEbVyZamspis2uK9eB4ucd6M4MQEW5UUCZAaU9WstcwrYUbNIQOiE5CVh5+ZzjIFhFSxzINrbXHcNNCzi1MzTfN2BM0REtmD6OEWlrIL1u4hwOwNmT6vEB90tPL2LoCUBDIQqJTZPJcYuwEIGQEthqx9dCtg1QB9rqFOBohJICXNcLlCPGuQjyziIMEqgB5Z6C39Mz1DxyrQXSEOB0RRsWBvk4AwElIHWC+aDBcOQKFUZK9rCJgAvQ7A39vRHT8tFczB/lVw4HVLKC6VhjuOcheTgmYVYTaSYQmQm8otYwNj4NeK4hne9g70iLVjYI9CYgN60fo5QPkKFHdSozHlBmqjkmSoQ0EBT2rS/wRPaXCSSAAeidhTzxgAtSNQajIduobzb91RPtSIdQolSH0huWBNBmh6lZS8pkGf+6IX756w9Tf7DuMhoNYdkFKMlECxRNnjwlucvKqWXM/IID2tcD+QYDZSthFQHUn0V8GqJ6sZV5/VMDRT4HNB2lQkLbVzTjQHi88zC19p2ZFqZfu0qB9L9DcCOyvODAzaaAfDNnF8YjshepFGTDbI76f8mDLN+xurd8RJNpFSF5OsngZPNgjAopo0ralbll63zh4VyPZqcz2ZTAjHDB7GbH+BLx+PUpQT2FEGy5T75EGGhyg2wUQdES1EYWxHI9ZR+JrDlLywN/seOx3j+lRrd9x/VmyWa1F8ekBKBUReVDbn5ElPLzpDiXciJ5MpAFnAgvpOFYrDr5z5UdOGy7sVgIFqufzXIvkf0vyxTTYz2wTkBhJrYB2AAAgAElEQVS4BEDMBqVGotpE2EVmOydWrr6L6C5EOucEeMO5KFUoZpuY7yRPzRJa1m8kgKqZTBsVt9FsY/JLHlSLHHgv2ZM5sW5uJlCtyKBFjeJDDMkrmN+Hdpk8dwngENRFDKcM2qnvyD6JkNYhCUr6M7I1amTYkLRk7zLzGQxlq9LhoLaD+6N6TjB4I1Jlw8T2UX4KtNdpMkJOTKUa0t9poqCkkyeGXPgElEauq9py22XyUje3AftLWQbTauTvfG3M3nrsrhSiQuleFYGeQjunT013BCTdOT9v6k0sjJ9dMEVYBL6vXCvQ3AXsLyTaG7JyGYiqkWFTvmKwTrVLgLDOPkleV9WGnZ96iClYCBiWfL5tBXtEJaW1UfDc+CqFEx0sWw38n9eLKOm0ZCiZ3DsuZWFDAYIYVqCEsky9D+hPNcw+kDFrGH5k59z37IvUfYCdK0piU3prUFye7iPMzsO1fG3xH2K6ngmKA3wrUb+zlJKmVOL8nPzakEBQTo+OKQU4g62SuivoS4ya4MQ3srC3epfYwdS1mr2buvOlYqQE7wTALgnQYvK+FrBbpdflepF0vJkSi8JG1neW1SR7Jp6GigAt1KrsA7+jPXyj30uqjUamCSSHKChJ1ZuxVIhkL6UcfZGwytEjpK5NmQBY8VHKDGRTSmrxZEZEJeFnBmpvU4L4xGYKFyC6Ee58zt7N0aXvtEiwaD0lrS5AjBaxSn2dB5UjoTWQvQOGkZUfwMQe5pTZDBC1oucyxqlWRKkJGJYL6GA8LwT7LSuGIcXKcB/6kdLXwy7K/NpvJsjGSJbTuSJ9jcOBvzEmn2eIBJYh8rnZtwkQWOblZ4/lr2Ap/3l8lt92j+Vfd2D5uyPK/Re4daPBj/7iCe7WM/zh60e43XE0tOlrvPryHF/fnOAf/5PPsP3ZMaQK6F/N0XUV9n2Nn72+wP2zNV58dY4wKFTXGjdvj7B7O4M5GgATEG4qBC9xvtxh6Aw2PzlBuK2h/nIGv9e4eXEM12m4USEOCv62xrCrEJ+3kMbDe4nZUY/oBZrFAPlVyxCZyiPuNZd/U8N8PsOwrclAvGow3LQY384I6pwErISuCZz81pSBhTwfICveP9y0lNNZCfmygdhpzH5SQd4agtLUs1kv+ZroBGWeAnDrCv4XMzI6ApC/aDjQ6hXCsSNrJAAMCnJgl2P/yGL+BWcF9VYidBryTqN5o4EkTUWvsPx/2pQuS+mxXFNq2rzUkGtNALEyUCsNuVPwI784wixAbyTscfqCrEOa/Q+Y/0LBnTqoPUFl/U5CjBJ6oyiPfN0ChiE84zmTV/VbA7WXWPxUY/YLDb0RaK6BWEeyYz6HhRDs56CY5muD9iuDKCiPNSuCdX1tEFqCNdEpqEGgupMwKcVXbyl7rd9JshkR8AuPYID5VxK+icXb174WMLcS5k7CHQU012Tclj/j8kKVQnokGZPmmgP35ppA+vjH9D/qrcBwz5M52qeOy1sJu+QxnL+IaF/JlD4c0VwLQIK+04q9lqxRSQE7noPO5ecazbVA80pBb6eKl6DJJJk1ZZTSA2ZNoDh7KVCtgPnzPCvORFJ7RJA2XLA8fv41k5DbVzL1UAKLLyXG01CkcNIi1XzwOqxW6QNARNS3lHFKn6SmzcR2Mfqf5yJUcerynMVS2wLB2ptqhcJouTn3u7mOaN8wHdi3EXKgD1J4YPELLjcqENDvMvihJ89sJGYv6N0TATj+CdC85TU2e800X71nKE/7NmL+nIOvkx8TaGVWJiaGQ++B5VcRy69DCbXJEwjDCQf49d1UDzJ7FWE2ETqxM8ITgJptRH9JwFWvgOVXlARHSS8m2bNYAI6bcdsIFoD2DffTzRJ4sihJvEc/97DzPIGSGKLEts1ehwKc7JIg2C6YyBySlNIuU8CNYrVIf47SbRnU1B2pRm5DfZeOkyCbksNq7Dz5Nudg7Yrga6tNCunR2QvIc9BdccAfBVCvODht3hHs1XcxVYUk2WlKJa02vA7qVURzFwqIrzY8t3bBxNzZm4DjLy1m1wFqoGyzfUdvqdmmmhDLbW9uA6p1xP4qh9OgMG9RcNntjYfpIuavHNobD98QsATNazBvS1AC7U1I9S8Mp6k2MYXqCLgGqHaUGc5f+8Ke6T2vtxw+NH/toDsCkCxTXn7t4BuBYclBuUqBM1EAZhtQr0OaBBA4+gWTWhnKE+AbMla6J4BcvHKobx1cIzB/7VJwD/e7O5fls6Nekz3KQUdmH1GtfWGqRcyS11gSavP2+lqWc54ZMLMPqFc+Sd9lmhzgscr1O0c/79G8s6jWAaoLGI4V6juH/oT+PzuXaK5HAtqavsgsUZ29HsgKKlHYO2UjzMbBLhR0F9DcjBiPFFTn4WsBs3EF6Oidg96S6RpONVQXIC0Zx3GhiscREahWll2TIaK+HmC2BH16z/MatEB9mzIdNhahkpg93/P9KQCztQhGwKxH6M2I6naEWVsCttRDqfcOkALjsSZA7jx07wsoVp2DmytUqxGqJ2BRe8fOys7BrEeE1EnpWo3m+RZurqFvdnCtIvgzEqqzUDsL1yqo7UDGsFHlOSJEmLsebm6gdgOkCym8jPJTANA3O7KGOZRH8HPSzTXEQLmqbzRBnRBQnYVc7SH3I5k8T1YwKvZsApiqTSoyitHQhyhcgOws169UAWUiRojRMbinMhDDCLndA6MFLJlIuekJFLNkdrR4r8NSSv6fl7nvk28x2X1yXYkQBJojvZYxg8YQEBczgtVhhHC8nqK1XFfuotSaP9lDmf6PIUwyVoC/vSeAdA4lsKfrv/F4QLQOMR6wlZgAZrTuPX9ldI4MpPfl91/1w9d+Q7r7LbxFCHjI3/jP78rtW81Yzr7zIP7wf/pP8eL6BBenG1gvsfrxGfBgwNX5Cq/eHuM//P4f4o9vH+HF//UU3/8Hf4E/f3uFzfMjREVws/ybb7HrK3x2+QY/ubkAAMR/dIp7f/drLM2AP/rRM7QvNOLvb1AZh7ay+PT0Df7vH32K03sbrDczSBlgKocQJIbOYL7ssf/iCOHEof2iQvfBiOYXFc7+1iu8/MtL1NcK9iggXIzQlcfl6QYvXp4CAJNmew0MCk8+fIuXf3Qf7sShfmngvrOHFBF2VdNTl+QrYe4ht4p+vpVGPB8RvcD3PniJHz+/QtgY6JVCqCNmLyT2/2oHvK3pf2soq1NHFuGW9I4+77GY91j/+Axmy+f4hmDLPia7ePSnFdq/8wavX5wATnJ7zkbEvYbcE+iM5x4fffoKP//Th4gqon6nIH+wgv38CMc/eIfrl8cQA4FknNEDOf+5xu6Zg9pL+FOH5ssqJU5G+I96qJ81EJ9tIX60xPBxj9mfNdg9c2TUHg3AXQVxNqD+vEV/6dG+UoiSoSYiAmHpMD/tsFs1uH//Dv3/eYXNRwHhwkJem5LymkEJPtxjMe+x/8MzjCcBOLGY/3GD7XdHzH5aoXvsoddkGePcQV8buCMPfTLC/GgG8cMVuucLAMDiS4X9w4DHP3iF5//kAfyCfkVxr4e/qwAV0bwwGE8CzFrCfszz1LyR2H9ooW81+1DPB8RXDXBvgP6ygfjOFu6rOUITsfwJzYfrzxxmX2nsnzjMv9QICug+sBC1h35ew6wF9t8ZUT03UIPAcBEoL1xQnuobytNEBMbf38G95WRD/ZbXbtTA7IXE7lEATnneFz/V2D3hMe8e+uQtlJh9LdHfi6huCT7tHGjfsIpiPAk4+qnE5sOA5o1Efy8gzDyOf2Sw+oHF7AtDqdxawM6Bas2aFb2j5HJ4PGLx5xXcnFU2sxeyACk3R2KRKH/sHjs0LzR0B+yeek4UJCAdFdC+EgXQBQ3sn1mojcLJjznYW30CLH4hsH8QYY8CRBA4+lyiv6QcUm8EzJ4Aun0lsP6+xckfGqw/DpwoWSu4hYcIgp8By4D5LyTW33UwdwqzFwKr3x8h9gqLL1SRj/aXZGrt729hNzXqFwb13YE/LLF7q399wPH/WyNKYP1dD9kLqE6geZcAwEDP4ew1y993jyLslUX9tYHeCTTXlCCOx0B1B6y/63DvHytc/zBi+VMOCrdPgfod/WXdPbK7IUkst9+xOPmnrFDoL8hQ2zkZ0P4ion0tirRyPOGkQncpJsb1jjUo1Zqg8ugL+t1YLwMMp0Bzzc/+HNxz8pfA9jGZ6fZVRH8uMH/B5NT2XcD6QwnVJQmpJECtbyO6K4H2Dbfz7M887j5RqNYRq08jFj+XcHOC87vPOCGQWWMkWW8JBQqU0FYrHmOAEtajnxJIrD+QGI/J5DVv+dkyHnHfdg8F5i/5+/Rzj5vPFOq7iPZdxO2nEouvCfCDEqUGZjgPOPsTge0TTg6c/6nH9gG33bVkYbMccn81HQs1ROwf8Nguv/YYjhh6U60jNh8KzL+OhXmcvwrYPpKchLgQaF9H6NTD2Z/zfXj6OQe324cMHmtuA4Yj1rcEBZz/mUV/qrD+QOL0c3oEfQ0sXhKYMTyIIUaIwOJ5xP/P3pv0XJol6EHPmd7pjt8c8UVEZmRmDVljtxsDbVkYIXfDyiskhNoWSFgWXiAWLAGxQBawAYRYmAUL/gBu75BY2KjBaqqru+2uLtecQ0RGRsQ33/mdzsDiOee9X1RlqxurrKps9d18w733ve98z3OeafmeQPUqoLry6MYSxZLvaw5YN5MvwsAcBwVsHktMnvuB/cwXDrsTTVZwR6knfZjx3D9ggmtf8Ti0cwkTWddiQeawnXHiIF95bB6qmK7Nyp/5T3os3jOYfmK5fncOu2NGaZfXDMBRXUA75T0o24TBmysCkC0ddmcao1c96hONdipx9P0G24cZihgKVN5wErCZUYJuC4GDH7WwI0WQt/XQtUM/0aiPFIqFQzNTyFd83/qRQnnjkS8cPY41J0q2DzTKW4e+kihuLHZnBqOX3RCeU1122D6kysfmAqYm4z153sFWCt1Uwmw9ylc17CSG6fgA1QSyvQAmH++wfVLBFgLV6x71qUG2cgSTlUT1YgdXaPhMYf1WhvmPa6yeFjj4owW2702RLS26mYbsAoqrGut3RihuIyjNFLqZgVlbtEcG1bMtmvMK2W2Hbp7BbCzUpoOd5ejHGvlty5Cc6KkUzkNfLNG9dQjRe6hdj+6ohN72cIWGrThZkN0RzLrSQNU9RGMRcoXBQykl3IiyVzvLoVYdn/cBctcj5AqidXCzArLpASmhrpawpzPKXJ2H6B3kukGocoi6gx/nkJsWocwgts1ethoZxaAVK0sOJlC3K/jJKPolJUTdDs+HzOylrjHAR0Swyd7OyHQCBJhSQNTtm57K+x2eKZE2Abl7XZsIkZnUXLbfbCHyLCbNRhbTOUpXnUPoOoiyHHyaw2f8WR6fwW76pvmzvfczHr8MjOV73xiF/+4fvv9zX+6/94U//IVvG/A5B5b520/Cg//iP4W5i2mcUX7XzzxUTblOP3fIbhSyhcDmCwQgZkV2w6yB+ozv6x9wgNUeO1QvNLp5gNdkw3TNwaLPKU0MhUP2yiBo9i7uHjJwhHI/MkX5rYArgMnHAeun/EJfv2eR3SjonRg69oKMNQmvJZpDzsTLNprys1STgCH2X/QY6gAADi5TXUDQey+PywEb5Y/JB8MvWMrtstXeP3G/jD2Z1VO1gq75JW4LBk5sngSYrYBZccAre0Q/EZe/e8ti/KGGWXMgBBEGVjFbcqAZFAFFfiNjNyW9OWbDfZKCP9KALb/jOm7eIsvWjxmwYUsOgJtT/rQxOEPaxIikGVkGZCQfFX08As2Zx+QDye2I4SrZKkkOyVjIngClvOSgNUgyOd1UQNUc7JbXfC4VpfcTngv5HbD8Ro/p9w0HtAvuE59R6pqi7H3Gz7UlB7PNIdkFs2Wf3fhFwOZR7M+Lnj29FajPLaY/0tj+qzVG3y7Jhr706CaCcfbRB5U8UPUJAVd5FWIKIqWbZkOWxGxi6bqJz62j5FByED36NO6TjimWk485KE77vLrg9ultOkcpS3QZfWdmK1C95CBN1ayU6KcBsx/vPUf9hCE08x9wMCks2VTVEmTkd3x/tgoxZZKD9GGQP+cANIVksD+U29SPeV4A7D4sL0L0tTFVMl8EwFNyqeuA7aM4KK0ps+smAsUNz6V2RsZP1WSb2nkECQ95zlSXMczER6+cSxI2MlDtEVBccd/bKspPiygd82SMU7Lr+m2B6jVBWGJss3WUDxZkqlwmsHvIgXmQ3L7kD8sXUQoXGVoAg9+rn5BFdQUw+8ixl/BEoIrn++S5QzOXyNcB3Zgdltk6DKxT2i7ZMXE1JZbuTiQ7NTOG4HRTAbPeBxx5Q5Bx8xUD1VJqWiz4PlZaCExeWHQjCVsKrN8GJh8D2ZbHwWzpd8tWvJ67sUC+DHsg0RHYbc8kzCauR8v3lNcefSUH2W2+CqiPJIpbj9VTidGrAFuQjSQDxaCRvhJRVhqwO2X9QToPpY1hKyOC7Wzr4/FhImmQBBq7Uwm9C6iuPZq5RHXtUB8qjD8l2NANGVyCJjJOKaHVKwI/rj/vB9W1HySSANBNCCZk9Ce6jNdRNxHIF36QelJmK+I6R2bQ8ftH10lKyqRR3QRkG4cgRPQVJpbaYXumka8pE+0mZFuCBMbPanTzDOsnGsUdwZ8tJYpbS5lpIZFCX5q5QnVl0U4VVAybSZLtIMHORYFhHdgZSbBUXdn9Odh6QIro64syTimYQrp1ZPAMJXr9RA+eTJcL5HcWrpBwOSdsdb2X2Zqth8/I0LucUs7myCC/5XuEowy2H2uYjUVzaJCt3VC9kTosdW3RjzX0zqE5NDBbD9U4dHMNs3LoJwrZiqzl8mmG8o7Sy+KqRXOaQ7YBunYwiwb9QQFbKKgUYmMDZM9+TRHIFvKgCsjewY7MIFkVNkR2kIyaHRmCtygXdbmErh1sSQbS5Qp2pCN4axG0RD8vYFYd+kkG2TlACehFg/ZsBJdJ5LctuhkBXz/VyK8avi7ErslxBr1s0TyoUH6yRn9cQW97BEWPol41cOOc9R8uwBca3ijodUuWVAh4o6C2Ley0gF41vNeOM9aEdHYfhCPpQRR1CzcbMXAn9lCqNZlcN2LSrYhsq68M1OZeQExv4WcVJa8e7LtUCqLt2BuZG4hdi1AYyNUOflpBtBa+yiG3HKyFMiPz2VsCwQQIxxXEth7+ZgrknulDZshGlkyGDVUBsWsQioyy1gT2gD8ZrAlBcGfMz7wm+RUHGWry5yrJ56KvcfBbJhZVazKfAAGjZP8ljEZo2jdAaXAOIssIMn/Kg/lnftxLg/28eyzf+8Yo/De//dWf+3L//S/+/i9824DPObA8++ph+G//4VfwrD3Gx/URvjH+FLd2hNoZfH/1AIf5Du+NrnBqVvj7P/hr+Hff+2cAgG/fvo1Nl6MyHQplUekOf2n2HJfdFH+8OMdvnX8Lf/+jfxO/9dbv44PmBB9ujvGbx9/HP775Eq7rMf7WW9/Cq26Of7p4gkp3GOkOF80Ej6sFPlgd49PlDP/6+TN8vDnExLTQ0uHZ8hAPJyv8+PIEh5MtHo2X2PQ5amvwm2c/wP/5+iswyqF3CuOsxa/OX6D1Gj9enyJTFt95eY6vP3yFZVfim/NP8Wx3iHdGN/jdy3fw5fklvvXybZxN17hYTfDNs5d4vj7A1w5fofMatTP44O4YR9UWy7bAOOuw6TKcVFtIBNw2Fc7H1Bf+8PoURjv8x+/+3/h7v/M38IUvvMbz6wMczza4WoxRFj2kCHg0W0ILj7FpMdIt/vGHX8Tf/Mrv43/71l/F1770Aj96fYJHR0s4L/G33voW/pcf/xsY5x3emtzhn3z3i3j76RWOyw2erw7QW4WvHF/g2foAr69mAICvvfUKH94cwXsBIQK08kNw0ulkAyMdjosNvvX8KYIXeP/8AoumRGM1FusSk1GDR9MVVm2By9UYTw4WuNqO8NXjC3z/5hSPpitk0uKPPnmMo/kG86LGi8Ucs6rG1YKdAEXZYf1qgvyoRggCs3GNQltcLCbIMotMW4yyHi9eHeL8wR0WuxJNncFtNH7jV7+H33/9BEIEfPX4An/48jHq2xJ/6f2PUSiLm2aEkWnxo+tTPJyt8Hi0wHeuHtLjezHC6GyLx/MFfvjsAUxhoY3DwXiH6+UY46rBal1hPGqweDVFdtCgyHvkxsJ5gdtP5vhXvvEhni0P4QOwWI7gdxrT0w3KrMfduoLWDlp5WCfR1Bn+2hd+gm+/fAt9rzCuGuyaHH2n4RqF6dEWubG4/ugQIfd48tY1Xl7P4RqFw9MVTkZb/OT1CSt2DteoO4Nx0WK5K9E2BuNRg822gN0YTE436DrOpnaNwXy+xeJuxATjnYbQYUgLnh9vsLgdoZy0qNc5pPYYTRrMywYvfnQKddTivQdX+OEH5xC5w+nxClc3E/itgSgtpA7wTiDsNJB5fO3dT/HPf/AEonA4OFpjuRrBdRIq83CdhNhphNxhdFhDSo8QBM6nK/zkO49Zg/I2/5/nFptFieOTNa4/nUGNLSbjGovXE8iRRTVqsf10gre+dIHnPzrDV7/+HN979hChUzg4W8F5idVdhaePr/H8jx/i5P1r3CzGcLc5yocbzEc1Xn56CLnU8GPHTtiVwem7N1huS7S1AVYG2ekO7V0B0TDMqTjfor6qAE2ALEcW4S5jd6wI7J0VgG8U8mmL7qLC2ReucXVL06t8VsBOPPRxA/+iAs4bqI8KuHcauKWhT7uTyI9rOKvgOoXQKMg4iZe/s8buakQVggoIucfbb1/h+Y/PEHIHfW3gRkwddjFdWDjw99hz6yuH6iOD+isNxG3GicJ5TKQxAerGwE0c32sp/+4PPIIIEL1kONUnGfqZR3Gl0B4ReLkJfdBuwlRLfVJDfDBCd+BQfaKx+1ILET3naqGHSQk3cyjmDdrLaBWIMngAcCW3xWw4ySN6Cb2Lfakdq2raQ+57nweYFQFcd9ajfJaRiV8KbJ9aVM80+lmsJVrQAqAaMQTmpIRhPK4hPyphS/qC81uB3TnTgO2I/lxb8vPgOdEpLH3dQQCQZOa7GWtrZM8JSLWjv1lvOEFy+/XY87rkxKqwQHsU0I/J/pcXYkjg7Q49Rs8ltxVcV7NmirO0YkimRfSKdjPEfcQJDWkpc+8n9DhXr5m6q6I31uWALbltaV8gyqpVx0mN1EfaHnLCsbzia5rj+JyNk5erJMfmJGzqltVb7ovF+wxumjz3WL4ruQ6Cnu7RS5CJX3Mb8lugPQTyOwwy75TYe79/spsD40/ICLuC/lNdcwJRddEvvQvR80tWnaE/AdtzYPKM+64+pdogTSav3uVkSz8m+54Cicw2RAk0mdfmQL4xsSPc3oNtdgG7YzK57XSfnhsEg5aKGzLhlFfTVyv7gPpYDmFUq3cFJh8FZBtOSLRzTvIAQDsTKO44KVPeOlZ1ZCJOBHGCp7q2qA81dMPJiWztIS3QHEi4jAy77D3auYZ0IeYNONQnGsWdQzdW0E28zmNXaYgSZJftE3OzRY/mOIt+dAtXqBjOs+8LHby3scczCMp+vRbI7lr08xwuk4OPNFu0sCMz+HWDAPLLGt1RMUxc6G0PV2l4JSEjUyijjFmEQCC9buFLw2CeXEN2jp7QXYeQaQLeCPxTlYm628GPcwzJszoyjh5ATLiF9QPjmqS3flJA7hg0JJpuYDd58fo3QaoQBM9KDZ7MkEe/6LYGipzS3rqGMGYPTBPwU3IAnz/dk7nvwfyMgKDPwiXxdSGEz30q7LvfGIW/99tf/7kv929+8fd+4dsGfM6BZf70cXj8n/xncIUHphahUSiOajQ3JdRGIpy1kK8KyA7oTizMHZMcAQCZh6gVZOzNy29YHdHP2QW5fT5FfivRvN0BjYReUUqqt4L+tSP6U2QrOHALuFcQL+F1QD/zrHS4MrATj+qFQv2AN0DR78umi2uB+sxDxoGSHcd0RM/uPlVLuJyDEBZZ83PciHI+Dr48shsVZZRkOcfPBXaPuG6prNlNLbIrzYCPIkT/477+wY4DigvOenZzSmCDwFASD9ATuH0cMHopUJ+SVd09cpS69gw2AQCzobzO5QJ1DHdJgTLlpRwYRg66yES6Igzsrdnw2LicX6T5rUBzHJDfMCGyfC2HAYJL2xIY8gKQ1eun6UuF+yQlBTLcJmD5RW6brmOYi+KgJ/U/lpf0LzVHHFDlC/brmRWPgdkyPbK4JpPXTck+qjZg/Q5QvRJDWmM35XKLmz2bmsJL2gO+n6waWdL8loONbh4oo4teIr3jwEvvyByWl/feP6V3M7+LYSnbyIpPySi6nFK2JOmzFdnF/Cb6Axugne8Z1H5Cv6TwYehSS8l/3YTrOrrw2D6U6KYcQG0fRWmdALoZ95/sYmplZFDbA7LOk08crn5Forzi8TFr+qjWb9OXWJ9QdpnCN7whQ5dkmOVVwOYxB4jVa/oHXayOAGIYTRy0Va/JqAYJzH/i0BxKNIdkamUfZZl2P1hNrODujCqE+Q+B7TkHcWbHQePulOs1/tTTXyji9s0Z1NKPebxVH31fjgXwqRph9VTg+LsOzgjUJxJ5LFvfnLNrz2f0cqo2YHRJKVtzmJiUOHCdcOBcXFO6KPv9fhSe54E3YgjB2Z2S1WU6JFkhWwg0JwLZgtu2O5WYf2hx81WN6Uce9TFZkXzJY9GPyTontYS0DICpjxTlrMswBKrYgtK67ZkcOh/zlcf2XGL0kgEtquM+K689NucSkxcMiYEAqksf0yeB3bGE2ZGpzVcc4Jp1QHVtsXxqUNx67E4lyhuuc3FLr6LZ8Xz0isx4tvVopxLlrcPybY3RhY8+VY/mQFHSOBMYXTrYXA5Jt7phGqjq9myt2XnYPFb0RF9fPyJjXl15smGZRHOooBt6+Oojyhd3xxKjC4f1I43xK3pU26nE/IOOvYCBTGsKgpHdvqIBQmD9SN9LHw3Il8hQOk0AACAASURBVKx56MYC5Y1DO1WUhWqBbMPQmOZAkiG+dWgOyBZ6JdCP6RGkh9TFzsCAdq7QjQTydYDeeZitxeLdHLoJKO7o02/mCmYXoBp6AbfnGcorC9URECSG2xa8xoqbHnasUFy08IVCc2iQL/gFs31oIjjiPS9b8zO8EUNAjOo88rse9Uk2pJL2I4nilq9tZ5L7ugkxIIbex5S8ly0thA9oDwxDcOLzuvZQrUN9kqF61Q7VDbK1aE8KZLcddo8KmA1B0Ohli35i4GLSKROM6QV0uYJqybK6nMBd1QzlUbWHWXdoTgpKW0caxXWD9iiHajy7GBUrMnaPx8jWPeABWym4gqy0WbQMl2ks7MhAWg+9bGEnORnJ1qEfk5nNLraw8wKyc+inGfKrHbqjCj6TKF5v0R6XyJYdGTWAUtBdh/6ghLAequ4RlER3QCCTX+0AKYdt9Eaim2UorgleXGnYC1lGEOI8+mkG1XvABehFg/64Qna5Qf1kiuL1Fm6UQa1aeiBHBnrRIBQabsRl6JsafpxBLWvYg4rsaW4gNw38uCCYbGzsdSSLCOsHwOWqDGrTMWUVIPsoJcSuHZhCNxtBeE8fZU52L0TGE5EpFXWHUBiIbZSztjHl1EQGz+h9CE8IQJ7RH+k95arWMUjHuSGAJyjWk4Qig2jvJbdqRfAXZa/Is33ozv1Qn+ixhNYEcF1PMJgZeiDjeiB6LEUCj0nW+plAjoE8whgyhKmLMgFFJQFLv6RI2+Pf9FXel9mG3v4MuOR2fAawfOPpz1q3f3Gf5V8Ay3/5j881sCzOn4Tz/+HvIlwUUG38ElpQAtgeeYw+pa/DjsLgJzQrDiSTdDMlSDJ0Q8QZ3ehB+td2GP1eNQyKXQ50B55exRe8yJIMMA3g8gXlovkt+9Py2zgIroHVewRD2YrLtwUH6P04YPQC6KccFGcryl6Bfdrk6JXH3fuCMfgxFbK6CLGom4PJfsIQDZdz0Lp7SGAUJGecB2mgJaDV2zgwNCJGw3NGM8k687s4eBKcCWSCJ/eZ2RB0FFcC7REwfhbQzQX6itLQ1E929ysOxSuN4oayv/qMYHPzWKC8ipLSGE8vPDB67bF+Qn8PAw3CUEreHHJQv30kMPvAY/NEoryM0rgod6WMKURfFwfKSXoX1N5DFySw/lqH0//LoDmMVQBx8FneBKwfSxQ3lHpCELR0E4F+Csw+8DF2ngNc1XIg2R4RkKRKgNmHHrszyf9llCR6tZ/1BsAZ01H0t404i7s740x5iv8vrzy2D+QgY3M5B+7NMUHL6h2B8Qs+lxIddzF6vj3gzK3eAYsvM3TGsCkG3VSgiJ6ezbkc3qt3AduHEtNnbliWHfE4qYbAozmSqC4cdicK7QGln8Wdx/qxQnnlsXsgB8/ZALrAfSM7YHRpsXpLY/cQOP0Dh92pQnHr0RxK9GOB2Yd8Pl/se+0AyvWaI4nRaw+vgN0DrqfXAqu3JSafJI8If/hYLm+2PH+K2wBdE7R4IzB6TYCJwBCTvmJBeZAMqOknwPF3KBnbnNMD1cxlBIYBxR0HrrszenRSAml6Xb4mSFNtYPiMZvE9BK+v6prbKRxBXgo6KSLABChZDJLb6jJg9qGL25Z6/Agc1o80Zh/3sJXE9kwReEZQxXRLib4kc9CPOEu/+ILE+BN66MzaoZspni+1x/aBxvTjDosvZKiuGMjSHAj+HoM/UkJrkPSsZauA6sqiPmLVSfL3BSVQ3lAG6Y2AzQUmz1vsHmRDjYDZeuxONMobi80jjflPWthCoR9L9JWE6gPyBeWXxZKAtLroUZ8Y1IcSs2c96iON4pZhKMVNj27GfrpmxiqVbMOQk76SyJcO7Uxh8nGN5pRdvLsThfFLC1tKZEuLzaMM2dYPILIbS1QXlECmbbNllAV7DGxHfheBy9ygPuL1XN5wuS4TKG752eV1j/ZAo3zdYvOkQHFrKZU8yaB3njaEUqK46bE5z3jtb6LEdxsw+6hFP9ID8GYQDc/JbqqQ31n0ExXXk88V1x2CFqhPMpgN10N1gbUQAPpKxuAbnl/lqwaQAv3EoDnk9974kzam9qoY/OXgCol+JGFLidkHNbq5gS0l8jsLaT2awwzFNUNc+qlCtrDYPuTxH3/aoZ8ogqDbDiGTEJ1HyCRsqeM5QjDlcw2XSzTHBuVlD9U6uELBLFt0hwW6qcL44y1caZim6gMZnZRQKoB+YmiTWFuGyNQWamfRnBQcW1zWWH5xjOKW0lRX8HxoDzRGn+zQTzOYVYfmrES26qF2Fj5jeIvPFbwSMBsChmQvEZ7P6U0POzboR/QCulLD3DZozisUr3eAEGjOSk5AW4/spiHrFO9/ou0RcoPmpEB+26KfZsjuWngj4UqmpArn4TMdAaFAc1ohv2spJ1016A9L6FUL2Vj0hxX0skZ/WMXqC9CPrCXy1xu4UTYAWLnpIJyDm5XD+ngjKd+/2cCeTBCMhF623B+pisRI6Ks1AZB16M9nMJcbuEkBfb2GPZ5AbTvYGcGvXNXoz6bQqwai5mywmxNM2oMK+mrNvxc7+GkJ0VgCtDKDG+fQi5qgprcD0BNdD38woY8xBIRxCdFZ+NLQ/2g95LYFrEPIM8hdQ09iBI1ivdunt4ZAaepqC2SGMtboZxSWwT8ACBybdkiFHVjAzW6QuCbWD0YzxTX5E+MjeE8gmGcI2x3EqELYNRBa8TljCBazjJJUHyCKnOtY1xAlj1VoGnoih8qPALTtHjAm0JeAnrgXHpQ6MwGuJ4BQNxBlQcayI6splBpksUMCLIDQx6RY4GdBbAKOnwU69zvhzT8/51LYd74xDv/1P/j5A8v/4Evf+oVvG/A5B5aTLz8I3/if/0M8GK1Q6Q6vdjPkymLVFmisxnG1xcc3hyiyHo9nS/zg1SlO5hsAQKEtdr1B22vkxuLiaobRpEGZUerZ9hqbbYEs7zEqOqy2BSZVi22ToV4UKGYtvBcYlS06qxEC0DYZ8qJD2zAjPs97TMoW14sxZpMdbi+myKctlOJF0uwyKOPQbzKMDmvs1jl05tjPuOGNaHq6QV1nEDLAWUre3CpDdbJF1xpo49A1mt2HVkIIoG80hArsOCwcTGEBEWAMbwy7RQldWgSAia8AfM+by2heY3tXAr1EdsCOxqODDW4XY+RFh92yhNgpyIMO4TKHHztAe0wOdlhfjwArIUo7bId/XTAV9ekWCILBQ+MefmOQegPVrINbG4jCoRh1qO/KmGQrIAoHZTxcLxFqDTnqEW5z6NMa/V0O6ADRykHuJzoJddQiBMGuRRkgcg+sNULhIWoJKCBIVorgpIXfsPsw5AFi1rEj0Qn4TkGuNb0bUwtYCbWRcIcWiHJN0UqEykEu9eAfk72AbIHuxJHp1gF+xJl0uVFDnUcwAa70MEvFqhKFodfQVw5qw/j/MO8hbs0gK7vfh9ad9cg/zdAdO8idhBs5QGKQJ3qDIe03u1YMA6liB2Iv4CtKZ/IrhX7CwbErAnQtYCsPKLDv0AN24iFrShaFY9WJWSmYJSV/vvTIrxTaI4fsTjFwqcDQrZdCdLwBXOGhGobn1I8s5E5G3zMlVt2RQ3YryTK3QIhgHeCkSnfkobaSbPlj1tFkC8oBk7RSWrLxTGsNKC8k6ge89opLst12xGAc1Qm0h57namRV9ZaArj3y8EVA8ZpdhmZNfy19zBzQlxdkNVVDOVlzFFUHIJBmvP2+4y8F77RHHsUVmbluygmpoPl7KrnPFmJgUb1hXYvL9x2IQe/Dh+oH/D9rJ/bsssuj19ZFr3acDEuhNKqhj1w1TEvePqIiYfM4+pqnlF6azX7CI3nEk3Qvv+NEhq1YzcLKgJjO2lABIfs4+dMA3YSMf/IIuowTVs0JJ8S6KfdZtsDAdCbpYT8B9JZMvOrJ6rYH9L22B/TF2jF9wt5w3frRvmdPxyRP2VPOWFyHCPgicx297HrLyQBIIHUgpjoVIF2HBM7eYJB99lTTc5IteltTN6LqyHxXr8Mw+dUcRRYZ9L2WF/QgAxiUItKy/idJE9MkYOp2FJ7MNrD3CadaE4Q4CaA5YaLi5Fs/jkx2TFK1IypTzC7WJMVuyn4soLcBuuF6NAcEqbrh5ybfq7Rkyuvj1KlI4O0KMvhpvbJNGKpDpA1oJxLZlqCvOdz7fimTjUnERazqiZOQ+ZKsYTr/XMHzR/UB7ZT7TjcYajWEj95rQa8uwn4S1hb8f7Hk/aGdRkmmQ+yZDOhGMkpLyQLbghOB3TimzsZKDtUmGSWGYCF6B/eVKGnyJiXE9hXDcdL6JJ+p2VjUxwa65mSBreTgjdVbMoXCBficxy15I9Mksq14nLKl5aSDC+gmir5Rw0oWTj5omLUdsha8FtBbOwBw1Xp4w0kDr4H8zsJnnJzLlpww6mYaxW0f5aUaetPDZwo+Yz2HHSved1yAWffopwb5bYv6rEB+28dOSjKDttT0feZq8I2m5TEdV0FvbfR+9rAVQ8Nk58maKcFakHtA2eVM4E29nVD8fhDWMyFWkqkVAZCtjT7OmNLqw9BFKVz0bXaxkzLVjUQw5QszvB+eabHCe4gmAs/AyhK4AEhAtBYh16wyUYIMqFZDNUz6W9Tt4Kt8o4fyflfl/SAegIDXxqTWe4E+vHnxGvgT60ZUBNVKUdoaAegboC7VjGjN5YRAoPnT1SHevwGW+XGfATDv93D+SY8/Bz2Wf96Bpf7TX/LL+5CRxvh0M8Oj8RISAYXqsQglpAiorcHTo1u8Xk/w0e0hiqJHoS3udiUKbbFtMygRsK4LjKf00VWmR6F7bGSOttc4GNXYthnKvEdrFcHkYQ0hAgpjUXcGSnqMixZ3QWBUdLBWwXYKubHwQXBd2gyycJAyoKkzaOOQFRZCBGDUQ4iAvOrRNRpFxRk6k1k4LyGVh5QBzknkeY+6lGibDGXVom0MlHFQyqNeFMgmHbKyh9Ye251GVvXoawNpPIL26DsNmTt4J+CthJAB0nigB6Rx6FqC0iA8nOPzi3UJIQKaXQZ4IKgA30uEmUU+btFdVejHmmAQHtJ4eC/5mqmFujSwLdPzAEBpx9lMAGoUbxAmEEA6yUqW0kLkgGsUgvEIraJHR3vYecfX6QAY9k4KxeScILj89q6gJ0xE8Jx5COMRnIAoLdAqBBXYEQoQ+AEEr6Ul8FUxrCmLqTyB8mFYARjPXszCQxoHCM0SehvL15MPaOSZjrpV8BMHX/BLSTaUHgMMaXJjBzj6xoYv95yhUHACQu6ZddkJuIlDqCXQ0Y8VBIGdCPFmLiJzKgJkLeELVhz40gOW+wkm7I9nHghqW+7XIAUQJzKHGfeYmOt1gEjPSaA7CNwuxKoPwW0KgssGd12ULGPYbkRPinAEgd6w3sLne/l2UAEhyvNchqFKJQiGa/UTMJFY7WtGZFQvsKAeQJw86WYBqdvVxwlk9oUCVvHYJalcEARCqVNS9HwdEOXSZVy/uI0uF5COHZsuptK6PAyVBwDoIZzEjlLF/SD7CEgsYpdmBAlh79XqpwE9CDC9AXzapzoM64MQQ696rk8/ifUsvCTh9b3P1QEyUOqNQLDlJ1wGAsEaQEAB8DwSlvvBFvHcxP74p33WT1JpeuzSNICJITI+4/anAvgkzU/HVcTBX3rYEsP9gscq7v9AoBoEwVs6jxABo09pzlGlMShTRnsPVACZVJ+lbSPYCoJy3HRu+AwMi3FcHoOsxOD7Syymy9OKRrAdmVwBfq6uGcjDsCD+ZMqtiNdMqsLgOtErFo+rBxDPJ68pqydQSufUXonCCQOuH8BajqAAL6PfrBDDue2NgHdxMssg1mUQQNObFtU0URGT2PMkmx56CfOwr6speOxTR6ZXQMj2+4brGxnDGD5kc4FQcT37UgznclI2hChddhkGtYmwcYJlJAf7wv2ye2fEAIqD4Pre38/CA84AQcZgO7c//2whhve5TCDEr6d0vrZTOQQfpc8JEex6dQ+8qnjfFGK4fyJwe1OHoq3kAG6DBHzGYyAcB2YM1tM8L+I1gMDzOQUgse6DgFsoAgmvoppAYvAKutib6bQYjmNIE0QZJ7ZcoYaAHwDop2boyLSleqPLMr1H9gG2YmKt8EA31QTFAOzYxOMVWc04wRKUgB0TsPZjEyfQuC3ByIEBdYVGSD2dke0lyIzAMcqiffQchggEhd2zW14TeIt7oTQJVLJfU/A5LfcVJd7Dm6RI4/GCwAB6vCI4xL3XINOsKUG8d+s9Y5nO/VAYBCmBWF0CGSgRve9xjBUjIfkmJYZOy5AbSlmNJpPo/JDyChDwDqmvUcL9RiclgKHGJG4zbASFIfwssBwWLLgdKa02AkYRg7p4Q4nv94HMZfA/Cw4j2E3gN8lxB4D5ZwGVf44e/peoHuTn/fhcM5b5u4/Cu//930Gzy+BrDdFIDl56Qf9hLdntqMhi6BVnzIaBa07WismRHEiYVSxVFyHWeXDAqzrOxKcAAg4oOQOcitpdkeRQqYA6lkKbMFQYZCsyNPml4uy9xMAsDQxDZI1UzcGfqjmoSq8NOvqXtmJgdERgwIDsI0sT/VlJkuoNgxSAxCLEgvkktyu4XGEFgt77FdPAUW/5GvoC+eUjLfv/1k89ygu5r+gAl5/6G9P/U2hAkvcmBiXoPXjhzDQTaF2sMWB4AmK4BIZBYwogSNLTNIDXOzIaKaWVX8QYUjHTgCXN/ANxu0w8B7ZxkNTvByxpMJgGPOmeIFwCEfG4p20SccBi4wAt1i6k16b9nnxf0vJzAAZDAEBxEz2QKjIhgedVki27TOw9lwd7z65sAZ9Hpk/vJdou575JfrFhMBYH/mmWPSVukhXZJ8iqGJIBT68ZZ+K5fqrd9+XJPgZRxNcOQRZpkAkM3Xi2IKvAnkBeX9JxcJgG88Jzm1VLKW43F3sZdyzTVi3/ny3DMDhN5xZLwCk5lh3fZwsOmJKXS3ZhGKSzTDwCBsEZ9pRUquuw72rswjDQs4WAjx2W6XxL25UqB5J01BZkN1Ighs/2xzbdJ1KlBRCPRxyIJzY1PZfCONI50k3IZqRBtnDxuN7bl2ZHRietE68Znnxknu6tRwf0FX8O15NL4GMPxNO5nAb0IuyXpyKrlc43Ecg+Skfpr4jnU6pXMFsWzYs4wGAAB2JYSBiudwLJCBoySoldTuldkAK6DYM3WHiCGbMji5eOny3p9/QRIOpmPyBLx9EWZHqSrzJdJykxNtkh0j2GXZsYBmjdWELHfkSey2LYZxz48/PS4Dzt29SnmEru6X/cdzQKz23gNcDlJDbZbGL9RR9i2Iin/DnwOXY7xn1Q0DOqOsqlub+4gqoPg40iBa8A9JUi8PpK68fOwT3gSAnHACDbeP2LCORjgE1i1HRNBs1rAdV4Tj6YCIhzrn+Sw3rFewOv8Xi8AEgXu1PDm4BbtQ4uV28MmEMEX7L1DERJasC4v1TjYEeUhQd5bzs6StSl8whCoJ9o6K0b7hMEgRGMpHRNvz9npAsQMdSlH2uo1sfP9WTf1mSTfCaHTkvZewILLZFSXfkmAI6AS+0sQhYBCUBwpSI4AUGj3vI1CejJLkrqIwvIxFYJ0fs3tplJv+ma4Pp6LYeJGuEDZGsH+adwHkFx+a40rNgIewAoYw/kAACFiEydYz2IJ0AZ7A/R9xmMgujdUAUyHEcPCBfBWlzW/fcNTOD9n0IQWMX9KizPA9nYzwRWQYg905dAWpKJWrcHZDEVFW03rI/o+sFbyc9T+7oP5zEE5qQU1fvLSkmsiPeTCCjf+N8br0/hAvJnQeK9KpE9kxh/hp9a5r1t/5n/pfcFz9mL9P4E3K39zHV6A2vcB4//fwHlnwPG8unXx+G/+ge/+nNf7t/+8j/5hW8b8HlnLCVZtPl8i7rM4JyE7RSk9oCXkIcOrlfATQ4za2H7gn2JXgAqIBt16O4KmHmDbpVDtBLyaQ1jHLavR2TgosSyrxVv5MbDWgF4ATGywNLAzjxZtkaSNaos3MaQrckd0Cgg95BLjfqJhejYbRZGPcQujmRjkiME+DNw1t+NHZmiWY+w1VA7BkmE0sEtNdzIQa8V7MwRRBsP0VDuqdYMDLBzJhYGFSDHPXCVw1cOslbDZ8pewM4shJUwtxL9OMDOHGQtoxwwUEraSRSvFeq3e4x+YlCfMmSnfuCgdxL9jD19csebruoE2ncayKssgjKB5pwBQt2M26dXKgJjjzByMBcG2xgARFaF+yNbCmzfZsejHbFvcfvlFsXHOYIKEF6gH4VB3qm3ErYk41i+1Ng+YNKirvk6s2aiYjjqoD/NoWoBO+LUtc8DrAB0TVmrKyNb6BjcpDdkCrM7rkt2x5vj9gknNMbPBTZPQhzYMJgIMkoNrygdVTWTEPPbKLk85MQAJUz05gZN9iC/I7DoAwNzdg840UEJIpMQhY1yQwdkd4I+3cCB/vZRGIKEXM792s3oOQ4GULt9emN7EMFdR9llfs3C+r7iOlYvBepTTriYNSsf6mOB9iSguBDwOeto4CO7pAD4uN+2AqoR2J5zEFfcsK5HbwS6GUOHnIjdjdt9ZQhn9JmMmC2B+gGPS3EbsH0ECEff6vYRGePEjhoqsNHNWIPiI4uTx6TC7SMur5vwWOhtHKRaekM3jziz4nKC89V77F8MgoO+FPiTrTiorU/pHW4OuT+7Ofsb2fMnoW2g51AT6HQzhi/ZEatTyksCzvpM3GNmRAQEBLWpkieBmcTkZauA9gjwq1jDIYB+Jt6YeNE7YPOEUsvmiCCSA3wxVMVIR/CVL1m/Ub2OsswoKxYuoD2gv9lle5atuGWIE6suuP350g+JifUxf8peRFYjAryWMlFbEuC3B4r7/lzGjk6gK9mvuH2wT3Asb1hPYjb0iNYnEvmdx/otiXwR0BxHr7bZJ0O6TAyAqjkQKBYBi3cVRq8IWNoJJ/GKuxQ2FCdRoi8UcULRq70clVJKDAyf6gPqQ4GgJKpLh+rKwWcC9SGBgdkwfKu8jTUni4D6UKG4I8CtjxTyJX3Asudu6sZi6IvsR3KYCNmeyjg5JlAsfWTjBBZfMJi8YMehbgLWjxWqK4K35bsGZs0Uz24iB9lrO1P7RNPIQiLQO+sMga50GHyunByJvvy5ggj0X8s2oDmkbzNbe3RjhT7W36g4sZMv6MlULf2ru1ON8trCtB6rtwzyVRjk4tnaDz7QbqJZp9EGmDU9moPcVLLmJU0m2lIgW3m08wyyJ7B3uYiTZx6y9einOtaIRNCbg9UpM41sGcNGQMAYJK+NbqbhMhOlyR52pOKEBmXCsvOQnR+AKXxAO+d6662DPeD/i6sGu4clVOPRTxlk1JzlUA2XwdA8h35iEJRAtu7Rj8hgUtIJuLFEftOiPWToj9508CW7BeHpJRU+IL+s0TyokC069FNDAF1quEwiW3boJwbZsoOMNSUuJ/BzlYFsHaTzcKVGN+Ky9aYnyJtk0NsedpIP4FX29Bh24wyqtrCTDKqhD1Z2Hk5nUHVPgNhY+lWXLdyY4TrBSPgsBh8pATsuIGxgGNAog2z4XmljlYv18HlG/+hdDcRKDV9lkA3TXwHA5xpq18EXEeC2Fj7XQ7ek6CzcrISse8CFoZOS/ZMObl5BOM+k1tzQn1lkkE1Hj2SecX16R3ApJcS2Zv/kksEGflRCtB1CkdHLaXQMG9IEoKPyXjhOZB97G+tMGr4OluDSRKbR+X2IT57twW6Uy6LrCSqTz7ONtSPWAkXGn0H+jHxVSEnvZp4D3iOkQKJUT1IUCG23X9c0EzAkwqohoGiQ3Ca5630/5Z8ie/2Lx+fv8blmLL/yzTz8+v/6W5Ai4MfLE3z14DU+WB3j148/wh8vz1HpHmf5Cv/v1VN89eACF80Eq7bAPK9xXY9wWO5QqB7f/u57+Bt/+Z/i//jR1zCf7vDN45e4aCb43sfnePjgDg9HK3z/8gyTssXl9RQPTxfYtRneO7zGj25OoKVHZzW+efYS37k4R5n12NQ5jLHQ0mOzy2GMw3vHNwCA7370CAdHaxxWNRZ1iW2TodlmmM13WLye4OzJHa5uJ/gr736E3/2994GjFt946yX+6IMnqGY1jsY7fPLiCIenK9zdTKBzi0fHC3zyz85x8JUbHFVbPL89QN9pVFWL1e0IQnuIm4ySxcJBbDXCyELsNPRhQ/B8WwASODhf4u56AlN1CF7Ctory2F4CAdCVRXhZwJ92CL2EuTToH3RAL3HwYIXFYoSDgw1uX89w/HCJ64spgXjmEDqFX/nSc3x0d4jVXYXxnLLi9csJJudrNHUGPKvQn9HrqHML22nIqwxu7PD03UvUvcHFiwMgAPMHaxjtcLccwTYaKnesfrgacz0BeCsROglZOJwerXCzGKPfGsiNppzQCeCgg9Qe+R9X6L6xg90aoBc4fLLAts7Rva4QsoDJgzXWV2OI3CFsNEQQkAct5wMuCw5qThogCLilQXFSo90ZVN8r0Jx6uKmDyB0lvcssStcCzGEDe1kijBxUaREuCwQToNcSdhzZgdJB5pRQF1mPKu/w8vkRZblbjVA51jEc9dCXBu68hbjOUF5K7M4dzFqinxIU27GHmHdQLwq4IsDP+WVWTlq0L8ZkxXWAmnZwrYK+yMhMvrVFd1dAVJRxh1UGvZLoz3qgFwP1LHdkAYLxkJ1E0IHy3zIAluAy6IDsjucUJFA/7QiENxrZQkL09CAmqZvPPdS0h/lRieahhawl9FagO7FQKx5L4UHvaB5iH2mcvfb0hGbX9B/1h5zQ0DvWM7gx/aOqifUVFeXS2ZVCYDgnA5Uy/m6P6W1NUt/uvIN5lXEi5YBeTXfQQ18bSEvPqFmqwQ+m2qgSkOx47Q44sVR8eYn2B7MoVeNz7SGl0rKVEH309wURB8/8fNUINA/Z05smpMydQrbkpEY/85h8pNAehMG3WT/wrJPY0hMqPBOq1+/3KF6YQUGxZ6SBdk6gF57WkB+wO9Ws9ui6vgAAIABJREFUxdB7CxB8djN+zugTTnC42MGa+n29AYpr7gOXA82xh66Zip0Si1OHqnAEY9tHVIf4nB7KfsyJK0PbPCWd2R5oFzfsbs0XVAEUVxhAOUBVxegFsH3MbQgyMXAYwrXSciEoSc4WYpBJ57d7UF1ekv3sx9ymyTMmAq++7FC8jJI5zeXkNzyft48Cytd7lrI9EOhnAaMXDGhLAU+JIU+qBhE4GSAs5ce784DRJwRuLnYA7x4GmCX3jWr3rJ5uAhbvA9VLMSQOD2A5TjPbkl5Vs+Y6dXOus97FxOAzOTD1AAY/YJKc+hworrj9tuLkjjfYqwn8XpliohLGxd7d6pJBXymgLPlW2wMGuXVzgdFLVmikOgevBUYXDrsTiXwV2X1NBlt4gvLElGYbBsZlK4JRW0alQJTBzp71WL2lsXkCnP6Bx/aM6b0IZPqbAzmoTqSlx9Vs2Dur+gBdB3QTOShSpN2rmIJM/k6PbiQZJBWiBFgKmJ1n4nDLtF0bg8SoFOF2mW0YAK5ZOeweGIL3iYzsOc/j6pqeStV6qNqjm2v0Je8ZZsvkYLOmH9+WZISbA07wqi76XmOicLawaA/NECbFTmE5+CxdzsnnbO2weM+guvRxEoDbki8sbEVQKVyAzxjyZDYeQQvIjv5N2fvoByerKiJbS5ksoBp2dmYLsoHdjIBbtWTLVMvleSWQLVr00zxed9Ei08Wqj9j9qXc2st6eTGnn4QsF2bDqQzU2MqKRjR9nZFmFGABwUJHlRGRIJRAydm5K6wkaMz2kzHotobZMvvWZZqdmfIgokRVNT6AI7L2YALqzEfJntwSUsetSJHDa7cEp36cYOFSzTzIkdvUeAzkw0nXz5sA6YYIiJ/OaGYK/xHgmVlRrAs+yQFiu+TprI+sqB1DJoKB7701AE8Dg4Qxx+Uku+1mPe7Ukvmn+5Nf9KY9fDsZyEv7LfwmM5d/58v/zC9824HMOLPN3H4X3/6f/CFJ6LNcV3MbQ/wZAVA7YaohOwM8sdGE58Jf0zQlPQOBvcjJxO4VQOOSvDdmqkjc2s5ToJ7yhphCUFMiiOoF+Fis/akH5ZysGqa1qBWRHFuz+gNrOHETL96R6AhbKC3QzTw+ZYqdYe+ih2r1nI0mxIMhKdQecidcbgebcQa8lZMvwDjsmK+aqmIyakZGh/I6sjuw5IOmnPrKlrE9JATOy45dothLD4M8VYQjcKF9J7H61hv6o2A/KtgL9xMNnAaIX7Fgbh5iuG6C3kmzXhp/N52KIQhXgSlae+IyfEaK/sD+2KJ/HWTdBD1t+x/oM1UbfmwpxXbmtSRKcpJH7IBmgH1HiKC3ZIWnB99VcD1VTciXj/qeEkssfjkdkBL3hQNWsoswtHisZZWTdjHUx8AwVkS1fkzxe0g2YDO0Rj7neiv262z2TYMu9zFLXlIc2RwQcNg4svd4HofQHDuMPNdqDOBjWe2mjN+BAL6YKqyZKKufcp3rHZXYzsmzFNc+ffMGBoyu4bqpFZFr3n6vqeA71BAau5GeTxeVy7ShWmxRRYtnuB6zdDPd8VhyECrdfzyR1ReCgtzkRqF6RXRN+/95+TCl3SJJUz/9JFyXesfNN2r28N0lkbcXl6Jrbq3fcxkHuuqUUzBuB5oRBNwj83CQv9WZ/HNO5kpQJqqVsGOC+Fo7spdlEWR+A4oYVGkOYTxwUB81BfZJoE/jFHruU2urflG8GTVlwe8ifAINZ8jve77oZB/T0tIlBZq1iaEtKQi0WlLHaai/NTd4uppLGcyoyrLrZrweiTFHdS1BO14qtgHwZ0MzJeKZtTnLflJYMYO/7U2Q8y+swyI5TGu7AUPZR0ttikPebLYFCHpk+W96TygrKPduJHK7lNOC2pRyqM4AobY9+Tl1z2Qn0AJFt7HisZZTGdyOeo6onYEgTDfmCLIwzAqrn9jgjBum52Xm0c8lzp+d+lZbHwuv9MU/BNqplAq6JATHCBWRbssh9xdoKW1AyqxsG0QAYJJXShjfSqBEwrId0lI1S9s3vvFSjkyTIrErhtjgjYrgP96HqAmXCtR+SjVPFCYGeHN6XEn6TZHa4v/b76yTJ9tN5luTEA3CJibD0gIvI/PkBXAlH8GIr1rm0cx2lxoj7guBExG1AANoDzWoWJSgnlYLpuzYMqa7eUErNxGAMPtOguD9FBEX9WCNbdAhaUgobX6MaP3glEwADEPerhzMEkC6nT5GfHYbUaBEC+pGGWcXwG+fRVxpmQ9mnqzRUTXbQFWQKhyTdJC0V6TpliE0w9CQKT2mviOwhhIBq7OB1DEoOkuhBstwxYyHJaoOS8ViQiRTJrxe/R2UdvxDu14YkyWoCSp6sY1AKwUhAUlrL88IPclcIMopBCIQyMpvZXpYr2/1yRdMjFHwNpGQNiN/LUX2VQzjH98a6EggBX+WQ24avs26oEUEfl32vjuS+FHYI2rknl33D+5jkpZGNDEYzXTYxgwkIpmqRn6n1iH/fC+sJzgPe4Y1k1gQME6D8k1jD+xUkKTVWCtaKpEeS93q/Z0Lvh/rcfyQ280+TxP65kML+BbD8pX0U7z0KX/wf/zZWV2PIjWKJueEAWHZiSE68P5jTO87gqwawUfomWwKIEGfK7ThA7QSyNZMSXQYUNwwQAAjKVBN7DNdiMO8DiB5OQPTgF9YoxIRGJkkCwO7co7iUQ9mzcHtAIOMAhf6VMPgyu2kENIIDENXsB1yD9y9K57xBlLTtZ/EBDsz7cVp2BDfAkBqZPJypUJohBdwOs8YAJn0WEx03HDD1Yw6+WbvBwV9zRJlj8mYGtV9mWr8BNAUO0gdPpdx7H5OvzFZMh+Tgi7POerv3UqZ+yuQ96iYc5HWzPahIiZZJXplen+Rf3YTHPr8Vw4DWrHls03bpbRxot1yO2USmYkS5aErChEzrHGLQAoakw3xBmatZcdlvDMxGnEVHAKYfe7QzgX7CbbVjDLUkqmUYh9fA5BOH7QM1+LCG8I5sXz8jbUB15dGXEv0IQ32M2dD75BUGqZjZ7Jctu70XLR0nXacKF/qkkgc1+TeHczF6zlxGgNLN+TnZmoPN1IuY5I2Juck2/F+24ufSZ8XBtexihUUcdOsasdMzMpSRfQAwBHO4jNsEYAgrSRU6Zh2GzssBIBRJqkkA34/2bJHsCcSkDQP4S8xe8g1LSxBjtgRq/IyUBolh3YMQ6KaCYEKK4Vq7X5Wiuv3vuiHYaWb7/cUJlgjet+x11DUliGndbL4PpEnBKtIFlpSv9gN+P4SexBUQfK/qMcxumx2BEOWlfFlKAlWdRztRvM82HBD1FQEEgMi+cH8nUMF9vH+eB2kP8qUjSEXgfgtSDBJJ3TKpUzoyRfSSi8HzSH9cvCfE8yBfRiYoSh91s2eC2LdJoKUbdlNmGxawCx8GTyM9yXuQlQBnN+a2+igb5b5jofxQoC45EeQz8UY6qK1i8nJMJ9XRbyziYN5lkW1aWzRHBvmSPZPeCLQTAl3dxGMeQbiuCTZkF+L9dw8EIcj0JA+hitLLfswglnzBAa7eObRHhiCmJUjxhqyQCICq/R40CAzeRh9DV1THQWm639pSxvOQ57xZk1kTAQxm8fd8ij4M9zTZcfA9gKXPAGchgkrVOtiC52HyRBKMiuHeoNK+iqd6EGDh/V0HO2bhfX7XDeExZLkUVGPRzTKYdc9aEZ32neDnRyAnW4dulg2g0eUSqvOQbfriYdBMkALynmfOq1jdsSMIZAcnP9eVOnqTuRxhPdzIDFUnHL9Y9NOcbFvyroHnEFKYixCQnYvbTICXZKRuVhAICUGpqSDwS75oHyWygz+z1FA7C7mLVSF9BErR9xhiqA0k3khLFX2UmNY93CQfalF4QyGolk2PkJJIcwW56QafJQ+iGHyRPiP6l53lpIjkNg+Mn7oHzO4/XGCHpXODlJW5ABGM1pGxa1mnkcDi8FOrfe9kquEwGnKz24NeG6WxyUsZH6m7clin+2msbn+e7AGhf9Nj6RzX7af/n9JhvWcwTwKuqb9SCISU0qrUzwK9BPCSVzKtWwKc6by6/3cEliEEoO8x1I0Yw78BgtjPetxfXgKqP70uPwU2Q9/hX/TxywAs3/76JPzn//uv/dyX+3ff/51f+LYBn3OP5ThroaTHb3zze/irsx/jR/UD/OHdE4QgMDIt3hnd4A9u3sLdrsRXTi7w3YuH+Hfe/j4AYKobPK8P8Xx7gIfVEt9+8TaeHC7wa4ef4B99+iXMyxo+CLy8m+GvP/0xXBBY9wVu2wovFnMcj7dYtxnOxhs8Hd/iH330RYzKFmfjDT66PkJZ8PdPlzO8c3CLdZ/j5d0MWjvMlUP55R6/8fCH+J3LL+D1Yor5eIddm+Gdg1tc1SPM8gbf//Ac8+MNqrzDWbVG4wzGpsW3f/gOfuULn+B7r87wpQdX6J3C6/UEXzm5gA8C3339ENPxDi+fHQEqYH6ygRAB7x9d4oe3JyiNRWl65MqiUD2u6zGsl3h3do1Pt3NcrsfYrgo8fnAHHwSeTm9xVY9RW4Pr9Qj1ogC8AL7UYntTwBy0+PWnH+GD5TF6pzDJW7xeTvBv/+b38dv//FcRnMBXnr6Clh6vNxMU2uL17RQnB2tc3DKC8vxoiU9eHuKvfOlD3DQj/Oj5GUazBtZKPDpcwnmJ568OMZ7VuH01wfx8hbo1aNc5IAOEChhPa2Ta4cn0DoWy+N3vvwehA44e3GFZF1iuC4RGQZYWyji4lxXmX7zFtGjx4uoAdqcxP9ngi0dX+ONX53BW4vhoiWfPTvCld1/h4+tD9I1G6BQmJxu0rQHyHoWxWH06Ry0DzKiH/7Rkeuy0R/tpDvekRpZbwFisX0+wm/QQnxaov94w7fc6R8gC1KSHjwmmwUrUDxVmby/QbAsUZYfuuzO0v7aF93Jg56uzLV5ejTA6XWJ5MUJ+VA/fBVJ63N6VkLnD4cEWN98+Rvteg9GkweZqBH2nYf6tGzin4L51gM1fbhDuMoSxg740CE8auJ2GyBywNFDHLdxtTsZJB5SnOyxvSmRXGv2BByY91Cv2AbrKI79VaE8cZCPgyjjADAJbKxBGPeRGk619vIHt9CAPhg4AHD8XADYGQQSgoE9ZtoB6d4N2UQBWQE56hGUGtSG7ZE96FJMW7esYaxqAYAJGH2s036w5Af1pAdkFrL5q6aW2aacBov3/2HuTXdmyBEto7e401t72df7cPdrMyMqoSqQEhGDCiAFDfgAJJFRSiaZAUBJ/AGLIHzAriikgJARCJRiUVFSizMqsCHeP8OY1t7f2dLtjsPY+x55nQCYQKMJTadLTfXav2enNzl57dRLmKfnh9gLHHw9YXR2xe7eECAJ6K0eFgZtF+AsHfWfYk5tkoe56wOxnJUI5dZyGgnLgcsPak6iYGKx3rJup7uRYZ9K+8FQOLC3UN+zW0wdqFruXHrHyKG4M3IwKDARg9lahu6aPOEoJs6PcVR8T0Oq5bXYdEGoPtWf1jb3wWHyucfgha1/MQaD/yKJ8YzBcBpiNxPDSwbw3kFbC7BNLeEbVQ/eMXuNQKPhUm6OPAmZPsGRXSP5oXgsEAhLhZQfzi4oApOexZC9uhGp5LAB6acsNg576q4DFlxLdZYQa6AUPhvuepb3dM4/6rUL7ysNsCFoWXwP77wGyZ51A+UhAPqwTKN9zQnD2TuDp9wHhVQpPEyPzHjUlr/tP00Rjmowjkw+0zyLKJ7KJ7fOIUETU79U4UdddAasvIvavBfY/9jj/I4H99wSKrYJdcOLQ7Nn1a/ZcbkhKiVxZU2yo5BhWGt5QpgqAHvlaotiDA+MIqF7Sy3nHc5IDoHSTQGap6cMO9BM3z+UoE9atHEGiqzEGsFWPVHm012SxdSvhS5E6lJHqVNiDWz6xTsbOBdprgWJH/3b1EKEbmXzCCuUuYvcJu2ldSZ8ye5I5Kbh469G/MJOCIKk96rswVsj4ehqwL7+W2L+m5L3YknG1c9a5dOfslzZHsrNDmpTi5JhAca7GShM3q9g7fCHHShZlDbozgcVbgf1rhfltwPYH1MpnGS4nWTjZkQPW8rHPE6bS0hcMEIS7iiwwkJQwgeDWNAHthYLuCug+pIkUXnfmSL+tbotxUsMcuYxhzRCc7ozdoOWOTCsk0F7qMWzJzllzIiIgLw30wSMUBi6x2e2FRrXxUyBaoAe4vqdvWPgIVSm4j2fo1hKLt5ay3SV9pNnHWr/vsPt+hSgE5jcWcgiU1x7JDFdPZkx8lTbCzhWKLfuDh5WG6gLk3KSAKIYNSRfh5gr66MmClgqIJWTv4Wd6DEtiwrhAKDlx4CtNYK3JpgYtoBtPz2WhPkiRddfz1I3tPwgwipIM9XBeQh8s1HEYk2CjUbBXM+ijpQT2JKhIDA5hTomusB5BCMhjh1gXfF4ayGOHsJ4T3EpJgK7UyIKKNvViGg1xbEfZLIrEroYAaHZr5vAgxIh4eQk0PZ8XZvpbP5CQPk16BSBmM4JCYya/JoB4bCgPrmvEjp2goioJXH34kOkMESgSs/FBTckEMuMpoP5VbOWfY17/avgu/V+nwv52PqpXH8eP/u7f5eCsmeSJoYwoNhLtS4cX/1Di7g+B4lGi/dSiemvgUk+fcICfxdHnpnqRoufJtrmarJqb0ffSX6SZ15hkexZjDLoa+DzH8QMYZ84BnJQk55t1RLkhg9E+J3NaPcSxPN0cyKBxlnxi8VyNlI7IIJVcKWD29NXUd7z5Cp+YRJfYTU2mrHkJLH+BFL7Cbcp+It1SRugrkE2TmaFJ667o+Tl+BBT7HDVNFq7YYWS4clKoL8mwRclgkvlb9hpGIdBdAfUdQ0PcjOvLPX/DGQcUxY61HQz1IGMxJNlgZkDNjgyibjBKHWc3HrvvqdEjVD1yv8oH3lhz6qFdisQMerRXMjEAHBzZZWKYEyOUJY9RZ+lNZgX4+enPsn+F0sTDRwxGKbYRriYLlhk4u5gYPt1E2BW3n7I1Mcr0Vl9aHF6ZsZJhOGPnXX8usPw6oEvM6bAmm6CbHPSS2R6+T3iMEj8RJ0nW4p1n0fqBy4iJccjJm6qfWGS7EOO++sQm5PRW9sdFlDsyZmbP8+aNQLEn4w+Rmc4TZi4tK8vd8gBQDVP3H082Rsmx7hmMUhyyl4ryxZCYyPwwx+nc5TAPVrVgTCd1NQNq+jOJ6ok3KzubUlX7M8lBZ2KVo6JMc1hMLKXu8uc8jh6qYDjI1E3A4SMF3VCqWT0GuDolierpcxkVxv3J6gruE7+DdAtUT35M9bR1lv5Nr/UmfY4OcTxulK1OssysSMi+s9XXFnauUichvV+uJFvVXXKwksNjgk7L6/i68sni+LKALwSKQxgH5pkFFSFCdRHtpRq/B3UXKBO0SUooJmk0fYMhSVe5nGrL56on40u2UsAcySIqy/UMc0lPWi1hZxLlzifmcpptN4cAu5Aotx7tpWZirGcXYHeZuuscz9mwUmPAT7Fh+IevUqhWLUYGNifk+kKg2HvYORmmzBSOaa0gs6YGsleZ8WQ9BUZG3hzJ/vVretvGZM4A+Eqmz3VKSC3ExETaOF4bOUAmP9jVGRN7d1KvcLJM3SRmM9dEpHOUuzHlwEG8r+h987UcQQYioJInTQ70uCHEsT4iS3OLxwHDWQGflp1ZRrOzsEtDL10h4Ss59iJm2afZT32IAEa2FAmEAJRXurn+oBJDRFZhjAx8Omc5/ZbBQTF58MgE8vsp/c0F+JmmMuPoMZxpVHfDdF1JTOcoyUZzF2NMlRmuZsCK6gNZPymgWge172EvZny9DWNSqz7aBHb4ftk6+FUB/dDCXdbIvYs5eTWzOVlWChcALRm+4yPkwICaLDVFAPyqgGocfKVH2akIEbJziFqOqatZ3mrPKqjGjUnPshnIJGo5nqNsC8lsYqg0l5cZQOspPU3smWwGhHnJ/keAIToZQKRzKDsylPz8qPG5sB5hUXEZlWbwjvWs40j7jwzkvs0GnjBwYV5CHvtJWpq8kB88bJJ7asUQnMwgpg7JqE/8gkrxd3VJCWyWhDrP/knryHxW5cRefpvNPA3iOZH5IiRmU+tx+1kDkpjTXCGS9zFQLiqMSRLVDxNnow8TK6m/xS+NoTp+2qbTBNjxy4XS1+ynjKc+SXsSQJRTYb/NWH6bifxV6zh9nADK77oU9tOfLuPf+29+/Zvwd37yP//G9w34KwAsv/dv/4esiEgzvNkTqHqM9QoQnP0sNrzpDisO+rsLoHpM/qTkoSqfCEJmtyxAFp6Dwjwwtqf+p+Q7kC6i2HG5ud7D15x9ZmojJTIygdL+jAPuXBSeB85RTsESo9QtDRbcbJLzuRmSXCmOaYTCp6qBll6pYh/HXrDsO5I2rWOWBjQujnUYYy9a9gWmAbJM+2OOBBmshcA4a1rsA44vFAFE6kMzTdquNHPePGMZeJa+6paSvvZS8jgoATvjNrsZb0RFGiDbmRjrWEY5Yh7EJ5lbLqLO3iXdE4Dy+JycpxwYkQbiJpWk50Fklg1miV8eiBdHyg+BSaKWkxqL/SSDy37KLO3L3pygxPj7oAV0y5CGfK6yVywPsl0tUOx5o3GlQLXxGJZTMTZBWEi+oOm6UpbyuzwgzPs8JFAo3QQGs88sH7tcqSAdJUR2Jkff01g/IDjzW+zD2NWGSNDjk4zRlXm6k+cpaIFhzkqH7LkCMAZNBJNSSdP7pScAER70hCUAO8zpExPj9ZgG78lflQfP2WeVJYo5JdM0TGoMBQfFdi7H7ZOO16A5MKkyf+6LwwQWcjItgLHmxOynZMthNSVq5vOp+gkc5PNabn2S4cpx3T75vfLAisBejMvTbRh9WxnE5gdlgjGdSzECjlzLEEqRAP5U56DblCyaklIzOOF5SddqKl+nRSAf4wCfEhWD5vEK5qSEHIBdKugmYKxFSJJp1U3+NjmE0QuWX+8LCd16+ugaDurzwCOUBIzFPozHzhw99N5iOE9pSiBgsEsFmYCotIHAIpW7QwJma0c2gymdLIGXQxiPd5ZIqsZR7jj6xVLNh2MACb17ArrL0ksmVCobxv688XVJCusLCXNgmEmWiUZJHxuZWz9OULA+I4G5FMQiBw+7MiODIjz/Jl2EPlr4VNcQlBwTQvk5k2PBPL/nk3Q0vTbLK0OpRpCESPDtazI8Yy1EOt/S0pvlZ+YD6eYHQSHp+skMD3c4kjE6kViyg4+fzSyJzMmgsnfwcwPZujFkRQ6OqaSLcvwsZBmtCBGi93zP4Edwog49g1WKEzmlPKm5OLnm6B1kKmqUWRaawVlg4uhhgF9Xo7wTIUxVFkn6qY4D9+ekw0/4+IEMVFqfJJl5m5AG9HGUdIbaTM97j1wlEmsDuWtHxguRr5WtJdt1WjFR6JE54/Z3lJ2m2o0PpKMAmTKTGLy2R5hVPPc5MCYEhNIQ2GXwkpNDAUArHouU3MobhJ8AzmltBzBJOp3n/uQQG+dHhitfX2KwiLNqlGCO5z5Vf4wexXDS7dhbpqvmWgwlx20Qg51SVTOIO5GtAuA+d8N4PPnGBO66fmL2nB8BW+wHCCX5txE8pmOVakliCExgDWGUfo6SVef5fgDRWkCqCWBl0CYkQaBU/Al8+Dpges34XBAoG839Pt3P7IXUepLknv7+tMfyNIQnT6Kl94w9lRnEnizrg4eQQAyIIUJI8atfc7p9IX64L/8PH78NwPKTn67if/IPfv2b8O/+3v/0G9834DsuhZ2tO/zoX/sCn8yecD/MUUiH19UGf7x7hcvyiL0tmdjqFRpHOv5ZvcduqFEoh4dujnXRwkWFXV9h3xf4g+u3uOsX+PHiFv/tF7+P3312CwCotMVNs0RrDXqr8XK1gw0KEhFDUPib52/xj+8/xvdWj/jF7gIhCigR8c9ff4V/8vAapXL4/OYK1+d7BKuxrHo8NjWerXb44uYKf+v1G3y5vcBM80O+70pczBuUysGnEe39YY6Xqx3+2TfPsVq1eH62wdFyv2ptcRhKHIcCZ3WL3mnc3J3h4uyIVTGgsQafrJ7wj3/xCX7n9Q1+cXeJV0liuu9KHJsSP3l1g95pfP7+GuvVEd4raONwWbW4b2aIUaAuLO63CyxmHZ6eFnh2tcPm/Rmun28xWI22KbFc0EyqVMDjhi3m+l/oMC8H9FajDxJHLzGvBtxv5yhLC+cUXjy7x81hAecVyqrH5lijNCl9NApsDzWWixbb7Qwvrre42yywmPXYbmcoa4u6HLA/Vvj+9SPumxme2hJa88v1uKmhSg/fKZTLHiEI2EMBUQSUtUUIAs4q1LMBZ7MW7x9XKEuL9lhCyAilPbxLs6dBoKw5I3cIAmVp0Rwr+L0BCq5P7jTqT/boOoOytLCDRlUPsFaje6yglz1co2HmFvZoYBYDvFMoSofuocb82RHOSQw3M1x+f4OHhwXioJi8aiKgA2AlzHKA+GIG+9EAWXgIAN7SMKwKj+AFwtFAzimpdK96JvwGgWglhAnQxsNuS0BFzC4aeC8x3M4Qaw+hA+KgIHqJWHmIVkFfdbD7AmbVw24qiCpAFh5KBQxPFcqLFv1jzZF7EYBecXujAGQEnIBZDbCtQfHGwH5sERvFwK29hmoV5KdHDJuSAKUIiC09U2bdw7+voV72sK2BaDTMVYvhqQJKD9FoJrp6wX+BKgRxZhGOPD9CRogniWgALAfIe6bzusvIPhcRARUhj4o1KFcBkHweVg6iUTA7Abtm2i0UoHaUI46dtWW6DjoB4QWE4wSHW0fojYLqBYazgFgF6CfFFF6ZwrQAuKWHsBLQHKiajRzTWYfzwACygcuMKYDJbBTsGd+nejWCPUjAzwBhU5jYeYA8CECKMcxLRMAtA2QnoI8Sw0WqT5oFlLcK/ZWHPprRx+4LIBoB2U2VMLl8frgQUCmgq9jltMrI7txhKq8fLj1mXzFUSnjArYDZNwr9hUCxNeiuwqgiKR8F7FKx4ucVcFJ0AAAgAElEQVRR0pMYDXI/Z3VPSa5qszSSINzs0qRgqpEZzjRVIF1KgZUmKU4EomTAVndJ9YovqRbQrUgThPRfNy+nmh7hMfq73WySOtpFCn/bi9HjnUPGEA3cPKlS1rmvOAVJ1ZQ3V/di/FueqBCBjDT9jCZNUvL8k+EvxyAtSnwpSzcNknyd94pRZZO8+VRjYKwnysqYPCnrZtNzTjzxOYOk+F47r8hKN9zWgQ4HmAPGZGHuBMbaGzejouNUSiscJ3hzr7S00/H1FdU7rkaS4caUISBGVp2pqWIMOMuTGcw+mI0haGNA0PzEa54mQCA4CTR/5zEsJklwVujYVVIXnS9Q3we015xUDGnyUHVcp7QRrqpHhYT0SFJOXovFPo6dtnYmqIZYcWK12nJSKiiqUMwxjuFMp2FjnICYj35dldhgMutI0txUcZQYcm84QRr0fDxmLiXGSpvCfmqZJn9ZfZMnCm1N6XP15McJjXGydEXlQJ7AosqE4T5RTx2orqJSJMtw6XX3nGRKstzscw96WpZuOJGj2lRdYsmQqy7A1ZMP2uzJguccgNOJuDzJlIOWAE6YyTTJ48vkze3JOmcZreoyYJ3Y7fz9Sp8kCAhT1UrUkhMOaR3TpAFBfDCKjG7qDYVMn9HULxqMInua/JBRKciThNlQGtacZC/jaW8kMPotY2UI0HNoUAg47cHMYUGnfZji2x2aeR8yK+r8yASLBChjkg4jpEkGvgGIYQLyJ6FBp1LY8SaVwOqvepwSYEIBsf9/Dyz/+vH//+M7LfLV6Y772f4Kd+0CjSuw1i1WpsOfPj7HTA/YDRX+9OYFzsoW+6HEn9y/wF07x5vDGlJE7G2FrzdnaKzBbj/Dn22e4V+9/Bn+0f2n8E7i1WwLFyX+6d1zvHlY42GzQIgC980c267C7WGBu/0C/+ThNY59gc82V9gca3SDQWs17vsF7vZzAARa67KDCxL7vkDTlHizXcMNCl88XaLUDu8f1rjbLrB7RxD7brfCm6c1GmswOIU//cUrrNcNXJAopMPdYY4vvr6GFBFffvEM7WCw7So8HmcIvULTG9zuFji0Jf743Uu8fv6En799BtsabNsKbx/WOLYF7KHAn719jm82a/i9wePXZxishpIBn727hvMK290MD7s5hsbg+eIA9b5EiALm1mB3rLC/XcAdeBz3hxqP//QK+rMaP3p5i6YpsT3WeHqzRgTQvl/gcbPAfNajawsMTxW2fYX9ocbu3RKHrkR7KNF0BQ5Nhd2hRrit0FvOhbSDgbur8TuXd4hBYBg0Ht+cwbYGv7y/wNPjAkNT4GLeoOsM0Eso7YEg0N/XBJWdQnQS3gvYTkOpgOPtHG/eXsBvC7SHEuptCXFTwvUa4WCA+xJlbdG/n2HoNNybGdqmhN8bFPdphC7opQMAvzfo2gLuvsIwaPTHAtVbgxgFRKNgNyXMvYF/P0N8LCBEhHlUON7MIf5kCdkLtIOBeCwgBgnVSpRvDcFaEPA+fYT3Gv5gICSBYPFlieAFis9rVO81glUoNgL6bYmw46C0WAyIjYZtDGbXR4hWormbY+gM4AVEqyB1RHGjUd4qzD4vEE1E/JrexeAVzEah/GUJvy3gBoXiQXEC4hsN86Ch7gtKoXqFYtWjXPYQg4SQEegkJdW9hNkpxFbB7CRkD9heo/7GQAwS2BrIXkJ2EvZQsBKkNRB7bpftNETlUX5dQJ4NEAcN9cT3CitQPEnEDT2YcqOBrcHiKwnVSMSO6+TJSjdQx2qc2VsOLNSeg43qjs/LewXVEOwBgNpomK1E8UQAqfcEVrKVCDMP1QksvhYoNgLVO41iy4Hz/I0EgsD8jcD8m+Tp3AnUNwR1chCsQDmyFmX2jn4/dZSpKoXXA1SEHCTTmpNXtHgS7Bjd5oENE6lNqvOobyVUJ6DScmUCVuZAsGx2EqvPkcAWO2EB1lPIXgAftdAHdsKao4DuBBZf8zCqRtJq4OmPrO/oSVQDJe3Spu17oky42HE5sksD9CKifAT3tRVjCnP5BKiG+6Nb9sHKAahvBfs/82s1Q9pUK1A+ETzzd6xZMbvcaRoxf0sQKnvAHMQYllbfEhjO37A3VlpAePoqq7vE6jaAainZVwMwexdTgjCBrHD0CpabCLMn4KByIQGjhmnY1SOVI/VdZkEYEiZTtUxOmqXPD1R9HCN0k1QggWCvuouo72OSU0dUDzGBBfpHyyfK9N0i9d9u+Pf6IYyKlgwyq3uCkfIpjuAuS/fruzD6Ruv7AH2MqRKDwUTS8j3VY0T1FMZANt1MacHmwPCr5RuHxTufQGmkAiSFO0mbk3TjGJZF+TLfX6SQsRzeJuIkL8/L023aHs/jlZl503DbzG46N6qjH3MMZmsCimNEtfWoHz2kBWb3/DmsxLid5VNKRg0EsKblegnGCNyy1FoN6Vw2EdWDZU1PAs7lxhHotan/eAioHt3IyJsclJRUJuXWAwIotp4yaxtR3w1jkq85elokuojZu54Ko52H7vi6rIYqnyxESH9rA8zOozgEFDuH6onVIcXOp+TjJE0fIspHTq6ao0+TMzHVf/B4m8MJAxsn9UVxCCj2FspGmKNLQJ9BUqZxSVorYA4Ous2MKyXXIkQU2wGq88kXSVAoIgOxVJ9Cp46O18zeEwAGMuc5lTcYCbMbKLV+bKGOA9S+GxUVmaHWhwG69ZQap9uEPgwEl5ZsumzdFMIjBdSxh2oGdl62liCyGZgQrCVEricJgaAS3D7Z23R+BURr6QHVMnmmE6CzHlFKyG4gA6tlArxylM/GLHtND2EdZbiJlc3PRT8ktYQc/ZiQkkxoBqxZ2ioEl28d4BylqFIiOp+SZeMIQKMP/F3kzxEUJgAbk2w2/xsfIX7w+/+r1/yFqbHfkYeH/LX/+215fKelsOUPPoqf/Gd/G94phE3qlAMAQzYH4GAgqsifRYBo+XvZS3Y6hpR2ubYQRw29l7AvBogml3rxyyinyfLuxYEaAtLsloCfsbsOUSDUgZUflh15rAuJ47YAgK8DiicGBfgFmYLcJyZsqskwMSXQSkTFgQYCmYXiUcGuwjijnWeEhWPfWl5nrg2JWREqGQLi63yT54Ay+yJlzzTd7HeExOhBDQWPQ3/pEasAtUsrjWngKabeMV9N3XD9JTsUVSeSZzPddAfBfeh4rHyRmII6VRMEDvpUJyCdgF2QwQgqp1sK9jE6esnyoDrPhoeC22x2lFyGMnn3IiA918ebWmKa0vEvdgwSAThAPGUixpqRNJjL8kg/S5UpyRvoKz5XA0Z/rfRkLaLksqXjwDgk7x8TVDGm++bz6WsGurgaH9TNRDml1ubXZ4kfE/8wVjYgSa3HhMwyeQE3/L1d4oNKE28wynTHdUZun0rhGdkPaFO6cpbGSo8xAEOEk/23SZ7eTb9jnQvG/j1xuu9pvb4AlOXr3AwfJPyefpfmZaqe63Lz6W9yIItSPmH0/0qX2BAxndvsnc0eXldzearl8vK5yRJSkQbuIvK9wWCsGvG1GGtUyIpgTPzN3utJxkmZvU7r0Q3GeqFcu5Ol7PS/iXE7cjqsnefBG5JcU4yVDK4W43X2oVyX0upQpkqY9H/VJWYs5mNBmXpe3+gNLFK9SbrWxmszXZ/5ugkpGfhU0p737bS2J8vPh1VOIU3bms6LGjD6UvPv8/WWqzyCxlj9ka/D8Zina9LOpucfXD+C7xtlu6nOIvspg8JoD8hJp5k1zQAqH5/8+1yNATG9P5+THPCiEhhztZg+M/l7JrFcMtmWQpHYuiTNNQ3Tnvmhmd47yuMz+LK5UzFNNsQ4ft6zbP80CyBoMcqls2RbhGx3CGNS7Sjb1WLsaT2tFMnLG78HHD2hBI85zZmS3qDIpsWT88L06Q99oKdS/sy0+VJMCbuJHYxq8o4HM6U952Ou2oBQTis7Pad5fzO4cTUDbaIUsHOJYs+KDtV61oNkOXteXPpc5kRYXsO8EQuXGcSYKjwo+c7BS5MUWozJvaxiCWO1SVSTfzbvp+opu86D/Cyt9uVJgEy6pk6vfdpxGIyTU275nRMwrA10Q1AXCgIyNzMEOyfDR+ECmUQfR98pLT5JQu0SYEv1Ilk6yfovSZYwBeqoLqW7CkxgK/lKc1rt9D6mwopIOXEo9OQ5HS+iMLKF+TiM6bEpoZZS3ok9zFUisaBPdkyXzdJ/LUdWkQuMk/w4J8eGOHkmM3iL8UMPpWKVSZQiBdgkD60U4+uF86Okl9dp+HNM4wQG4yQvPk2V7foJNJ56FBN4+3Y67CjPzY/UkTn5PE/+5hxGSWuuUDlZfn4eY4RI+zj9+S+JQU4A5Xc9FfaTn67if/QP/sVf+3L/g9/7H3/j+wZ8x6WwdWHxb/2N/w1ftFf4R+8/gVYBndXQMuDFco9vtmsY5XE9P+Krx3M8X+/x5mENrQOcVSiMRwgCMQq8vtzgOBS4e1wCnUKsPS6ud3h8cwa97lFUFgUAozxmhcX9bg7bawgJuF6hXnWwg4a3EuWM0sqhoVRPPG/gjgWunu9w/34FVXuUxqGvSkBEzFYdmm0N9BKLFwccHmaYXR1x3FcQAJyT0KWH3RuU5x3EoNFXGmqjYS8d9NLCP5SoXx3QvlkwvfOhxOX3n3D/fgVheFeXhQfe1HA/aSiXPBjE2gODBEyAmVn0uwJ6aRGshG811F7BX1vIo0K8GOC8xB/++Jf437/8GK8/vcNXP3+O9estdp+fIV4OePl8g3efXUOsB7hPPH70/B6f315hKEuIXgIrB/FoIF90GLYFB1ulQDQBslVw64DnP7jH7Z9dI5xZoFe88Uvg5Q/vcPtHz+GfD8D7Av5FD/NNgeGlhT9oxNJDzBzClyXsmnJB2bFgPl4NwN4AC4voJNBLlFcthpsZbBCIZYCw/BkKBb+ipLB/Rlmk2ioOiq96mJ/X6K48VCvhFwHFg8TwzKE669C9myEuHaQJUF9V6F47iE4ilgHFvaZE7qJHcBJir4EoIC57soiSckb1wwO6p4rpqL1Eeatx/NgjLh2lnkWA3FOWGd8U6J85qKOE6slA9WcRfumhDhLhylIFc1NA/2iP9mZOFqxjEmd/zi9rVwN+4SFbiWIn0X1koR81io1A9yIAL3r4o4beaFhwcKIage6Vw49+/A5f/a+voRuB5nd6zH5eov3dDrFXgBecxCkC5l9qhDKiWwJ6L2CXEc/+uRts/pcXaD/yUEcJPw8Qg0B5L9F94qF3EvbMk83qBaKO8IsA/TON4TxiuHaYf07ZU/M7PWY/K2GXDMcyexbOuwWTTe157rMTUH+wxfGXS8zeSXTXEVFF1O8lfJm6VluB4w97xF5CbzWqO4HjJx6zbxSaj/wIaMt7BTkA7UtuY0iTFcVGJlaMEz32ysE8aASdJnE0O2/nX0vsfuIgvIB5kugvAbcImL1Ro0ec6aURfuWgDgrlQ5JopWCmcpPOX0nwmutkfBlH2SfA6qMogcVXAvsfBJQPElHwGEUBLH/BUK1QUop5+DRg9XOJ40cR1SMnWIY1k1erWyaDhoIA2s7zJBBlq8NqYsqCiQg6Yv5mSsZ0NRm0/adIfbpMCz2+Fpi/idj/IGD1OQ9y+zzCVxH6IFE+cYIgd7bm0LDuOmL5S6C7EFi8YfhWFEDzUUT5QKlpnogZZakd0D2LWP8MaF5yUijLen0dMf8GOL6SSb6KMQVWt1wGB/JIgWQTQCv2BObCA+1zgf6CbKrZp3MW6PdvXwjM3gLdpcDsPXB4zXCk6jHi8IlA+ZgnKnJwECbQrvidOP8m9SkvCATtktu0/oXH9nsK5gD051y3rxlIVu4ITprnDNpy9cTASjtNIITUZWr2ZELtQsDVDLFbf8ZraveJZMrqnt8jdslwrvn7gOaZGLMAyk1MtUAE1k1KvrVLiWCA83/mcXzBpNjVL8OYF5BlqYDEMCfrDfA49GuBxVtPaWkC6f1awRcC9YNHfyYnL36aXFMJxLbXBHHVJgVu6Wm5vgBmdwHNM5VAr0TOQbAzhfOfp/oEJXB8rlBuc/0RWfphyddDcOLGVQLFkefNl5SBNs812e+nxKDuPfYfaSy/dhiWcgTerhJYfjXg+NJA+ojiABQbh2Gtx55SiAlgDmsGGOWJgOoxwtUKzTOF+Y1Dv1JjrVT1SLC4/8hgfsPalyyJVckXrPrU51vrlHRLwDOcceioktffHDwUADdX4ySGXTBoKgNg1QMuhTDZuUT1YNE+KzB716M/Nyh2lLsHLVDfRNiVBiJgdgzNGS4KmINDe1nSY6zoMXYLw+7N1sGtSrgqhWg1fI3sPUQUCfjqsYrFnpWU1ZYMIYpaoH7fJjmsgGoG+GU1BSWFyK7NRQl16OFrAykEYsk0U5HksVGbNClIMCt6i1jXlMceBsRZAXmI9Lm2NlWyBPpKU71JEAKyYQBHrEtEkfymiRmMBohKQXSc8RaBr0NPpjTWJdnJ/H+RfJshIBYGwusJ8DlaaCCyfzqBvrKgBzqvo8y9dUyfjdbSh4kTcCg0hJjYSqH1FLQjBNeTWVGAIJoL+IsH/OIEjNq/+OW/zY8IIGRW4q/g47vNWH7ycXz9n/9thPcVGbF+ksH05xzIHD4JKJK8bjiLKJ94c3UzDmTaa0pa2mcB9S1nTGc3Ec3L5AUaUpfdIsuhGCtf33A2LbNjUZKJqx4ZRFNueSMpduwsLLYRh0/pBRp7NSPG5Fmznzw6cmBEv3AMzckelTwL7ioOINlBeCJPsZxlHdZTAmx1nzoZ08z97DZg/7HE7H1MMfIipdcJtM8iVr/ggFFaLgPA2JkZJQdglHHRx1PfRbRX9Jy4VIZebCJ0xzCh48dksxZf5RsfmYv9p+BAU09F5lEBs5uA40sG2/TnSeI2RPRnEt0lsP6MnhZz4ACo2DJsSR858EE65lHQP1NsObjuz1Mx/CXleSyDBy7+1KO5lik9kV/QxZ7sEdlfbuP8bUjHSLBfciVgmhwQFdFdyHHQmAeZxS5yAHUgw5BDWY6vuF1ZvtVdcZvMka9pXsjEYEayT56Dzzx48yVTUbtLgfm7gOMLOaaB1g8BQQH9irH0w4rvM23E4+/xvBd7Xiv9WU4qlXAVUgfelO5a3weYY8DhpUJ/LjC74fsyOy4t93s4owyw2HPAVt8FHD6S9PjMgcU3AXZBZqtfSRQHyuyGBTs6179wjNRPv4PggMunsCe7IHOWmU67FKjvA7wR6M8FFm88fCGw+57E+oswdSC6PCjiug8fcf+VjWgv2P03u+UA1BuB+omz+8Oc/qL9awVIYPENpXr9mp4gDvw4WKofmUjbnUuYJjNCvKaHhUC1CWPKqWmmQK1cITG7czi84s158c5h94mmZPMYxgFav6JM9viSA9b5+5A8Ysk/lboTuzOJakMmqV+T/VRDKr6fTzex4sCuRlcLdBcCsxvesHOyar+SqJ88js8VVl867F/rURJ3fKZQbRhs5guB4kh2whfcx/qB29ZdqJFFKvYB3ZnE7MEnJhRj6FG/UmO6su4CmmuN9Wcttj+qMX/v0vmT4/d6ufE4PtcojgH9UnJAXEm0lxKLtxw413cW/blGufXo14rhUDN6PZUlO5QH7u2lxMWftDh8XEH3+fok8Cr2Hs01E2R1x2MmLf9v5wo+M4daQKegJlcyoMrsKMXrLwy6M6buVhtep64SWH7T4/iiRH1v0Z9p1LcDjh+V0B0Td3efVKgfmKoZFaAPHt2VSaABaC/4+Vp+06dwMIFQyDHMSXVMu62e3Mh6DUt23ZaPPeyqoMftwLoH3YekWEjpryFC9kyANTsHZQO8kXALhWEhMX9nCbbriR0EADvn+SRYKDCsJGY3Fvpo0T6vUN0PCIVEd2lQbBzsUhMI3lt0FzxnqmUolOr5czhjoFMUQLEdEDSrbYa1xvzrI/zcICgJsx/g5gauVjAHx30pVWKymPCq96xEGM5L5J5N6UIKSCKQdAuF6rZH+7xKoUhkJM3eorsqUN32CKWCHDwOH9eoHizMboCfGeijhV0VU0WFQAqvSWxaraG3Pex5BVcrFE8DQqWgNz2G6xp6zxGznxEg+kqhvGngFwXVHa2D7AhuhnWBYjvA1zqlySbW1E/hV6KnH264rNM2aqijhVsUMJsOCAH2Ygbz2MAvygl4KTKH+pjCiioD2dixJzIsCiBEyN7BXtTQmx7y2MFdLyGSRNTPCgYT5e7MXUufX4zoXyxRvt9jeLZA8XYLfzGH2vfwc4IXtW3h1zXk4NkTKQT8ooTsHWKhGViU/INhUVJq2luEWYWwKKD2fUp0lSPTJfqBgT8JcIVFRf+h0QxHEgJy2xBYlQUTXPNPKSknzcFA1jHMJwXfRKMZKJR+F+uSgDIFDUFKArqUJCuajsvO3Y39wGNzEn4UU7JqBmeirhDbDqIqyR5mhhLgMT7tvwRG9vNX/g3A2Gd5AvxO/ZdjamtmJnNi67frPrRGbFqm4iqJmNNz8755ymWF0ZPP8v/LI8TvPGP58U/X8d//r/+lX/ty/+O/8T/8xvcN+I4zllARv/vqBp/pKwytQWw1zEbBScCfWxyURrywmP14j8evzyCXFk1VwhwE7FmAW0jYlafX6nJAKwpIK9BIyl6v/tYt7v/oGUMNQNYgaiBeDLBNBV/FUWLla0ofu8sIuw6AUBjOAspHAg5fCAzXFvmQ5xlh4QD3aYfwZYWoJ/leTnKNmsCmvgUOn3B9/sUA97aAm1MCaheA8GSAzAHwZUQoBfqPe9iVQXknR5/DdslBzu6HlHlKRzO9W0SYA+WW3TOycSLw781zAsn+gkDn+JMe9Rcl2h/0iLLEcBFgGon+jHLd4UygS+EQ/tmA+v8o0T7j7Gl/RgmKnwUMK4HmexbmkTcbfaR+bv8Dj8UvOXBzlUB/RrARCvajNa8Dzv9YoH1OcNldB5T3koxJS9kswGALgHUYwzJCnvO86paS3/aVw2HLTq/mZUzsj4AvBXY/HbD6kwL7H3jIgTHyugXaVx7VvUR3LRAfCE7NkbP4/YVH+aCgj8DuDztc/MOSfh1JYEsmIKJ9FqAbiWHNc9xdRhQbzu4jAsfXgXLlip6w2U1MExBilAFmdsTWrHLxFUF4ey7ha4H2Ok1WzIH2GpjdSAzf76Cb1BvYcTIhV024OVkmEXKFCX1stlZon3OSZfO7SF4/gvnqnmxN+7GFORgU+4juihLo9lmkV28vMSzFCIKki+jXAvP3DGLY/9CjelA4vhaobwl2fQ1UjwHHlwq6Ze1OsQcGTblWv6b/Lipe+7kqZlhzhj8nNoeUtNlfAItv+Fo3A2Iv0L5IqcytRPM8122QFQkFELRE8xHPy7CQ0H3E4WN6IbsLbqNqxeiPcgue49ktAzhyRQsg0V8INC8Czv5MYFgJDGf8rFF2p9Fe5yRfDrLtAtBfCdjVJH/sLqfexFw+T1mygPQew5xgt9gD/UpgSMwRIFA/IC2XXsPqyaP9SI5yuC6BbN9IdOcEp8Oc+zm/ESlFW6C9IhgVkfUsQQtW3nRkM5sXIvUfMmHVzrhOX/B9fidGBqlfSSzeBvRrMU6I6SbAzgTa5yW6S4FqI+AqiX4tyCBdSeiO58cPPB6+JKjszwVmdwLNM4lyK8dgkPZSotoQzHZnEusvLfqVQnslsXjjE/NLlqQvknTZZglemlAbmFAsHTAsOLjyht9j3Rn30RwIsvNxppc14vBCpc8SEKVCuScbZ+caw0pA9wRqutNoLyTqB6AM/FwRUHOyzhsBWwvYJSeR+nPKYBdv2c/nSwLnfsWwmWoDTsA1Mk2IEETT38nv2+5MjhLsKHNdCcF17mLsLiVWqSbFLRS6tYJdCMzfIdXLqFTD4+ELifaKTGCx07ALJiy7mZqSphOb1K8kdCtxeEmwW9/xut5/ZLD6MsKuFNQdQ1qCFrDzFJwmS5iDQ3+uYWcS6roeE4BDoeDqbM+IGNYGuS4lVAr9mWaycWFglwre8J4UtCKzlVKjma6scHwuUe6SDNkBvipga4nZ4DCcFSi7BF6r5HduUuKwEuguNIqtHKtNpA0QlcKw0qgHKif6NUPAYmKK7ELBbHpEwyRiCU76lQAgAVcpFC29baFUlN+DclBf6fRdp1E+0qfo5hrlfQtvNNorA9V52IWG7D3cXEE1CiJI2IWG3iu0L6qUAiwTc80gGbssuR9lheKhYRpvpREKieKRY5LhukbVWwL5QpJVKyQ9coqMoAYoHQ2cBIk6VfIoCbsiGLSrgpNlxwF2VaDY9Awp8zy/srHoLyuua1FClBpRSybl6gBoCbtIEzANr4NYKojOIhaGoLQbEI2GX1ZQAEJl4Of0XxQNPxuh0FBNR+AlJcK8gtoeecBzUq2SiFF9APCED/DnS8imZ/1JAqNhyaoYeE8ACRBg5p/RsJLk0I5AXBQFcujOyCgaM0lojUY8HCGqCrEj4IyDBYKHWHAgGY89RF2P0ujxvRlM+pTSmz2SYfI0iipF+0Pig1RZkba/HyCK5FtRagreEQJCKS4nB/6cyGNFZivT4zRF9rtMdP31Y3p8t7nYIPDl0zn6+xryvoA68uZKs7ZAdSdhvi6w+fwC9VsN3Jcwe4HyQaC8UQyz2JANEE/F6C80e/rybj67YjBFQw+g2ZMxiI0eQwjKTZIVJo+kdAJ6L1NSITsCETkwlgeGfhRbhmXQ/8SAFXMQ0AcGTphdirvvJ79YTuvTrYC6LVDfsnA6y3yqB8rXVEfAqA+AekigEhi9hcWGh04f+VqfwgOyDxER0AcyZvrIY1k9ki0qH1M1xwPDX8RRk8HSDFaQXsDNkxRwz8GRLt0IoIDshyGwK3ZA9cagvuEx4fsiiieVM3Bo8D/kAAlKqfQ+9d85MqXlA2fvhScYUy1vdsLzn9mx77N8SiEiLUuy5dJCH1lNojqB6oEeIWmB2ResMahuFYTne8cQhYaBJrnihEEa3B5pgeE8Qr8tRx9QKMhOqS550vTx8toAACAASURBVBKAzwy48BglYlHz2OiGgS8mJTMCmPxLiem0yzhWm6iWQCb7+4odGVBfUiqn+oh41PS8pO91BjtMLLhukSYmuK+s7Ejl8UVMwTJkhlWH0c8mj+rPJT8yoEOO1TTBJMBzlj8nnDRRDWsv6psE+B29lGNdio1pUJ6WnSSg0k/HtnpiAEeoeCykRQr+wFiQnR8x+dyiYFiLGmKSwiU5kUqerp5+XLuIY9qpOSRfnuT5yn7B7JcUgSz9eJ2ksBXhgNk7prpKT3AeFI93uQ9JMpeW5TECLZHCQXyVPHDi5BhoJB8YK16U5bUQJdlZym2ZUjl5xTh50q8ZRqTT5591QyLJ/oB+KcdBq0whKjKnn+ZrUU9S3PFcpcROXyTJa55UFtl/KRJASlJezevLzcTIKopIBjYfw7zsJk0e+IIgMic+qiFMNUplrvwR6fMfR3+rnYvUg6uSV41Ah0A5pnRLbkuelMgVP/0Zkyn7pUiSYzKEbsbjW+ymWpt8XwhGcMDextHnbFp2bnJ7eSxGr27yMYoQ4WuFU090FOy4jHr6Doo6r2fy9uV+2hycQk+rGP150uZ1kGkMhhN3rN/J7+U+saaK3bu5miR7OtVAEMR1Y0xLVUMYPebCs/pIWiSJaEzfAxJ2rnn9pA7a/kJQQhl5jFTrUlCMAiJ9v7lX0zQOdqUhPSfbTOPoT1QiSQ+B9lojGgnVcfLDzdSYUBoUQ2HGii8l4CuZAmA4eUfvdBhrsURIctx0nFwGWjUngnI/pVsWCFrCzhV0G0YfpAgRypIZVT2ZxCy9jZqMrJ8ZTiKkzklXyyQLBcKMjKy0ESH1YiLycxwNE5Kl/dCDys8O/x8LOfo2+bmTSbIpUycm60/00fPz7HjOVb639IHey561KFHK8ZwGQzCvGnZg5tqPKAUQwPqaGCF7D9mQ+YuGkxH0NQpAcvtDbUa/eP7sh0LRd1kZQAnESqeJAkPGtCErmL2PoVBQqVczH18OeNQot4xVMTGZmYxzqSYmAxtNhjEqRWDnybzmkJtoNEKVgF+MiGWBqBWiTumtCTyiLBLwPDknQgCJ9YR1yEmryLUcBWtL4jCQtTytcHHJm6nJboq65rKqioCuLBKQ5P6JXIGSwa9LLG4O6okRKMsP/omymGSvSnG78nrz7zJ7KQT/FgIBYupF5f+/5bWUcpTj5n/jezKgDf83/zII/jZj+h18ZCnsr/vfb8vjOy2FffX7Z/Hf+K/+dYQo0XqDp6HG28Ma67LD89kOt+2SNRV9hYu6wWM7gw88+Fp5PJ8d8PawwrrqsO0qOC9RaI9jX+BqcURrDfZtiaqwUDKitxqzckA3cJbGp28vJSJckFjXHR4PMygV4JyClPy5mndQMmBwCrPCQoqIm80SUkYUxqFpSijtMasGxCiwP9QoqwF9b1BV9MiVxiFGgbY3KAoHJSK2mxmq+QDvJbyXqCoLozx2+xpCRgQrEZ3E2dUB1nN7vJdQKkCJiLZnOmnwEqZwmFcDOqvRd0xumdcDQhQIUaBN23i1PuDuaYmq5nqbTY1iMUCICNtrvLze4t39GkXp0LcG9bxH3xsei0FD6YB61qNpSvhGwywG2KZgrYWTMLVFUTq0TQGlWA1he43n16xGub9bIQYBVTn4XkEavi/2EqL2KCpL9joI+vtKz/7eVkGWHsFJIAgIFWFqC9saVIse3aEEZITSAa5XMJVDiAK+0VzO3HE9nu+NnYJKXtQYBISavuykjvA7A1QeqgjwewO1GhCcROwVxCARSw84eltFqyCcYJhUESB04MAyCIZQ6Qikugp4MQVSzRwnOaxANGkwLiOgIhNvTYCwEnHm+X4kEFhERB0AFcfli5YTLNGkkKkZ/Zws5BYIM8+aEyugdhqhTmZ8HRlkFfP/T77cZPygFqN6eUS7LyG3humyKtJ3q/masPAQrYTqZKrboKeSvW6RKa8ujxgwsvChDFCtHIOvoiFbCgCyEwh1BDz3g71600THNFDPoFQw5GmgnyvoOIXk5OOeQY+bQGQoImLBgYrsiQikTRMgkWDSVymUKmLyGs6ZwOpL7mf+O9UQZL1UywkbmY5zDpERafuzukFEjBMjOZCK25aAi50Aaq5zQEyTTgBCiTTg5LpURw9lXmaW42cm7zTohEFI3E7VT/5OkcLPcgjVWJ3heG4ZDEa5ffaAjkFQPO0AADsny0tAye1jgiz3IYej5YAoAGOwzGlglerSeRdJNWII0nKaaE6vzLYDXhMYO32BaZk+TSzk12UFSgazBFuYQozyqtN6M2geA47MdI5i6gvOgVB5G8xx8lpmT2g+TqeWhSgx1l5klYNuMAZT5TChD5aRQG6+NsnoU84cU1gRO39T+JWdzn3e3w+OrU7vr+hjhDyZnEjAP7923IaTkB1XM9yM74uTUiNNyvhqugbHrl1M5ymoKWgsL3vsos32kLzdguDR7NPnPab9E0jdv3yfstN5Hz/buYdX5EkhgnMGrHESY1imgLkUKHQaUpRB36jqOFlflAxQykmwvhRj13CU/H7J/cinn5fceZsnA1jdgQ+Oz2moEYCRXc0duHzt1HWbO24Z6BbTdZzqPCKS7/jkebrOmNZKhYrs03FJ/sh8jESeAEzBOeN5SRMA+ZoQjn2n3PY4hmtlth3A2A8sPKszQupHnT6A6Weczkd+LS8m9nmKFEIUpWBAUKlHmXHUktUgrRtfA4AgF2B4UAayUkzpryfhP+K0JiSH8Jw+z9+Vuev11P/owygr/pX+xBx0M8pZ+V7hf8U6AYyVJXnZ+X358avWkwN5cmBQPNk24APp7QcS2NPuy289/lJ45K+AFPb1T9fx3/v7//Kvfbl/7/f/+9/4vgHfcWBZ/+hV/Jv/5b+Ju0eCNFM4tPsS1892uH9Yoqgsnq0PuN/P0d3MoS86uF6jWvTjAFc9b2G3JeTCQhuPYVtCdJSORh2BBABE6REbmslRBMBKiNoh5vTZo+YA3guo1QB/TJUNaeBsHiXTWjt+CdprC1V5xLsS8WKAeGCqbZx5yMojHAz0agC+qscBp/CAPQswTxLDJcNl8oAv1AHqkGaQLyxw0Azm8QLFrSZ7safk1S3D2Ivny0iQoCL0RkO1Av1Ly/1oJFQjIL2AncfxZuuriPnXEvsfe4ieN1TdMAFXH1Oqq0uD0ipArS3Mz2qmyZYB5Y1GKHgMym8K2GVAWHqoLfv9ZFpmMGl2vQTKJ5GSTCN8HVE8SQznTOLN/XuZBbbnDICRjiwwJNC9tChuNYZnDqLjKKbYig+CTcgKMy1WNxzEtt8foB/MOMARnjdku4qobgnGfJ1SdFsxDprcnDJKX0UM65j67Pj39qVnBUY6bv1VQCgi9F4y/XUWUexT6EcnYHZAfxnHDkG7iigekzRvHVOMvEgeJA5wh3WSNs8YXpLl3P1FRPE07bNuRAJnaXCj2NVHxoLvaV97VG/pNwzmZPCfMaRMA2pHqaluMcqiM7sKAOUjk0rdjAPMqFM/3hHorsi4+4rX5PIroL2apKY5jVcNZFdz2u6wjqjvBIblFKqiWg6O86Aagr5fuyTjrTpKeX1iaYPhILPYcl9yuqocctcekpeZA/uggOZlTGoFbl8GCsWGnurcXafbdAyfR+iDGGfk9ZFy49lNxPEVz0GxBZpXEbP3nO0XCfT5iv+3C4zJvfQRpxqNlCw6rKlcCJr7lgdlquUJsEuej2JHua6bYwQ/OvX/5bAOen0ZvJI78XSTB8gxSfDFyIC5On0PpGoFf5ICm7en2McRWFC2S59wfUef8Ow2qwcIsqXlsfYlWVi7mJaZwd9p2qoaeIzMMQ18IzAsqE6waV+L1AM5LEQamJIlo2d2ApfDmv7l9poBKbqdwFRmrkzD96k+hSQNZOmU5UBeuoj9xwrFdgIruuG2FAeeE+GR0oBjkhwj9QdS8pqTWbMcM3eOZkBijmlwLsg+5n5KmRhDNcTxWPkyfUccwwhKoiBLWz/Qm3s66W2OAc21ouqj4Xqko8928d6P22Dr6VgIj7GvERE4PldYvnXwhRwHshkAQFBiXO6SjPZCoNwlz7AWKHYeoSAY6S5VSh+m93VYkRXMIC3vpy/5GVM9GTwe4zix2MkPPE7ORAbn+FrC7D26C53YeQIWO6dUtdzTH0yWncmzZu/Rn2t4I1Du/AgEVB/QXTHgJPvqVfLgMjTHJuZWwRzpa9ZHP1lflIBO1Rqu1tAHi1CSGdSNh/QBdpYsNRGQA/dVdx5uptL3jh9fn9Nex87JxtMretcjajl6Uoe1GV8jIsNvMtAEEsgd+x6TdDqBKp88qOxv9Ailgq8UzN4CIcKuDIrtALvgT7cwkJ2nv/Rgx+dRCXpSbUDuhkSIDNJJ/ksIAf14hDufJTCeJJYuAD7CLwqoxI6O4E5LqMcjwnoGuJDk+gSHUUnIwY3XifBxTIeNRjE1NiXDCuvJvA2OgTtGIxoFeaB/NMrkqdQKUUrII28AYVlP73MeEILe0EOOm49p+anGI7GKwvkRtIrM2mXvpBBcVk6dLcyU3JpBZ/aJ5gqSfEwySyrE6OX8do2HKMzkj9QKMTOnWVabvataIbbt5Ke0bpTCjp7L9IjWQWQ2NwNLyc9p9P5DmeyvAJ48VBGx73/l3/4yj98WYPl3/v6/8mtf7n/6+//db3zfgO84sFz8zov4/f/i32GXovKoC4ufnN3irl/gZzfXeLY+YPAKt59dQl93sJ2GLj2U9gip/y8kBtO3CqIIOD8/4PnigM/eX8M7iU9ePOJ2t0B7LMk2JYCplhbr1RGbzZwMmglwrYaZWYTIO70pHNbzFrd3K0gT4DcF5Moiev49RgBBQO41wtJB1Q5+U0CfDXBHg2LVY9jzy1SWHqEje4YIqPWAuh5weJpBHBSKFw3clwvgVQe/MxBzB/W2hHtmIWRE9AJyp7H4/ha7xzkwSDIwJkAUAdhrsku1g7wtEcqAOPeAjJAbg7B0TI/VEcWdxvBqQPGmgP9+B/llBXvtyIYlcCxaBXXkzefyD25x88UVsLSQdwX8hYW5KWDPHfRWwy08t2XhgF6h/kaj/d5A1k3n2S1A7xXchYPaaPiFhzoohOsB8t7Arz30A1NH47kFdpqA52xAfCgJiFceoqN3FEhVInVEmAXIJpXdJxa62NKzVz5I2FUYB4bFExNT1VaPUf12GaAP9DblNNrqnUb30pGxGShtHS485CBR3Uq0LzzMViZmQsBVZNmiAspHyfTNjYRdRvg5JxPcgmyQ2dKnhwi4tYdJ0mE3IwuoOgmzS57bPQd77YuA2TsJu2Dg0ylA9BXZNvMkoQaB/pIez2C4/PqNHkvK83qjBoKOKDYS5YaBVvYsoH6j0HzisPiFRlAEZHbJ4xdNgD4ojDUyvUB1L9Bes5g+sxZRAnYVMHsjMZxTgpsraljonSTXNjGBNWVx1b1A+5yVNKoV8DPOqhfb5K09Th7VYpc9lzxGoQTsIrB+J80PzL8iWPN1hC+B2VuB7jqi2KUbvSMgVj2DcuQADGcp0CuxSm4WYbYC9X1Ed0GQMz4i0L6ImH/D5R1fEazqhsBVDkmunsC36smGZFYqM3Z2zhRbEcDKnoGBUUGnbTqfgL9uCHKrWwJLORDwtc/4WSgfk5y8JwDdfR9YfM200bEOpmA4Wn0rRvYNAObvAroLCTfDyPzJIQENR19gZhTLLY+/SlJsEVJSa8/Jj/VnIEB33L7uQqB8SgFhiTVTHcFm+UTJcXcpUGyYLFs+YiySj+rDBFmAYLy7oi+5eSlQPmC8vjIjZFPwFesSJmYt7zMnTehrH9aCgWJpAseX3M5yw1ArXxLMuhkl0lERlLuKctNhTVl/+4zrKjapLmZksMRY3QExTSb052KcHGC/ILfNzgkWu3OG60DSt+wqBjZJz6C0YUEw503y5MrUu5j2UbcRzTMJX/HYikiGrHlGwF1uU0VNkmYXB36H2ZqeU1fyw9Sfcf98yWNcPwSCyj23aUjhXsqCvt3+5BryqZIlJaxKz/OTw5YYyjQx0r7EKEnOAV65oidLJItjSMFY9HxmVi9PqtiZRP3gRjaPKaes4ujO+EUlXUT16Cmv1kjS7NQhefBoniUptI9wCYjpNozBUaoPGBbqJMHVoXlmuG0Dv+/MwaG9Lri/KYHVG8G+zY5VKaP8ODBoaDjTnIBKfkkRIqrbHt11CXMg+Cx2FsPScB17h+FMo9i6sQYlaEFP5tpwPYbLGZZJsnzwUL3HsDJQLUGxXaqxuzK/PktmCZJSKrdnz+SwMqjuWvSXFczBwc01zKZHKDV8rVhrMjeUKvdhDJAqNj18pZNEW0JaD7timJD6P9l7t1Db1gQ96Ptv4zbva+21b+fsc06dqjJtd5too+ZFJEHykhfFBx9EMCgoiAEVoX0QRVCJDz70o+2DGNCYQKKgPooKghHbRtJtqk111amqs+/rPm9jjP/qw/f/Y659uqpb0lV2nTYDNnutueYc9znn//3fzUXIkfLK0BnKkSUnl8s4ImU21twN8PMKejcSPGoJv2khbIQcPMLMQB0cPZohITQEa6p32YNpyWy6QMmsEkBIkEOW/GqJJElSFBlwbDTkbkDq6iwHxYkpzSAxCcGQn+xfFBkQiuNAiW0OxpkSX7PfUwyWr5GCPytJ76N1J5bwIVvpT35Pfh99BQv4LGctgPErQE9oTdD4UJoaIp/7QffkAwZTyp8c4PPA5/n7Limekmb/LpafB2D50S+t07/y1/6xn/p6/+1f/u//yI8N+JoDy/qzj9M/8p/+s3j5boPqB/U0S+9XrE2QY+6oNCz+Hp54dD9iTYHec5DXX2SpR+6MnH1ZZvxOYTksh84sTpWmQSr7FbNEaY/JS5gEAUb3ij4Wu2KKrF3xeeNZ/tLYZlbsaURzKXMMfvY0lX4wx8Gb7jH1Ek7sTnWSxenhgTeu4UD08ElgHULex/oGUxy9GoFxg6mLL1ZkGQtIgMDEbNV3meFoH0iWEiZ/1XARsfhCYlxh8qMVtsd3CatfuUL/P17wtYIJovuPGZld3fK4CjNU35G1KEs55tBkiV9P5onSudNAIikeM5MjI3afykkOxuAhDizHM24nVARJ3WuB0gFYzuXyB5EhMnn2OzSY/Kj1XcoJs6fHleWA8AOZVqkmGE7dhfNXTNC1qxN7Ud1z8F7fCdQ3eWAzsLz7eCEnuVnxT6qcUNy9pTdJHxnjX99xZtzsmHo6nMkP2Cc1MH223M8QvBe6NxzsHZ+TGU2SNRBkozhYtSvea+17DuzdDFP5um/FNKArDGRzzX2K1eleAzIzqDmwLoyXb4H5KyYBz14zPdauBBY/5KCX7zm+T6p7yrn2nwicfYchKP0jgdlr3hMFSOmeg05f54CXVmD2OmL7mURzQxahz+e2uk/TgLrI/0KDDBIEtt8ENt/hYLo/I6Nkc1Jw0qxZYMCQyImjBBn1Dd8zAAfqbpalcjbBrplMLR0wex9w9y01VT2UdGI1JNS7hGAwMYb9RfZdjjm5uMveyYHeMbvi4Nsc+BlS0oKLVPHwXKC+5zEDfGx4xITiJIF6S9lrkgLVIeL+GwqrL0IOzcmD61kJHToteuC+hFagvo2ZQeMg+/BUos5s8exNgJ1LVIdcZZBTWeu7eArbeSSnJNr2hv2MJ7aabNZwJrMfl9dyXDIgrbmJORTIob/Q0MPJ+xhqMYGRmAPRAO5ncxs+SLFt7rifxYcZNSs6ChOrMqApLJnpmRhbfJGl47EwaAyRArr3Hoen9DkvvrQY16xwGNYK1T6SoWpy8vCS2/etQLVnem5JhHUzvh/NgUmz+hAwnOspiE0GAOmUGlyYpSKXLHUVEwuawbT0CeaQwUvFpFsAMNucLLrM4VId03h9R6bNLhWqHaWOUWef9yFi3KgJpLfvRvRPako0VU4U3oYJtKshwq401BhRbT2ikdNAk3UQAqqPqLYObq5zbyRQ3zEwJmU2NVT0JppdgO6ZpKt7DuBLxyoiMJwp1NuI+oYzZqFWUINH/6SBshHN+wGHF12WkIqJgeZ3MoOM6luH/lGFahem2gqRgYLv1OTpjLUgaIsJUTHtlqnlBEd+plBfZ5CS+xKTkVOSLf2fmb3s/cRgublGdWspybRxuubFV5ikgLkdEFsN3+l8LyTI3sMvKqghPOgtzSE7VX6tEtD7nMZaK+ijmwb88ujgz9ssESU7qXoP2TvE1uRryn7J0jsZOg193SPVCrJ38OuWvkojofYj3FkHfSCTiwTIwSOZLCsdHAFhayaQIgePsKghB4/Y6FNH5+Dhz1uovYVw4cQ2AhCHAXHRQcQIYT09nll+Wno2hY8nltIoMowlhKZ0UYZIn2KITIAt/ZRaIXYV1Ps7QCkmwvYjUPGcoDITOBQhMrSnJMkOdgoFSm3N10nxYUpr2Q8lyURqTSCqFZlNJSEmVjEzlEUyKwTQ51nNzGymGOl5fOid/CpgEwLJWkDmv0lx6rH0no8DDA8q1SOFTYyR1STOnXoyczrsB8vDjksh8JOWh1jl685Y/nEHll/rVFjZC7z9zaeoB4HZy4ThgoOkca05oM1hEnYJzL5MEF6jfZ8Zh8SZ8O5tHmCds0Nr/iUTDe2CM4Hnv8UUS3NkRUL/SGL2MkvMBs5QlwQ9NVDeNa4kRJLo3keGYawEqm2EOXJWe/PdiNtvczbTHDjwXfwo1xac8zlJswoCibO0JsvZ+jOJ5ffJAHRXAVELHC+Y0ugbhniU2eLQKqx+N+ZQB1ZOJMFqBN0nNDeUipXS8WofUd0HJMFOxfYywhwTpE2otxLbTyWWPwi4/BWJ9e8Au08FVt+PqO4E1t+32H1sKM3Ki50LDBcC+//1AuuXcWImqm3AuDEw+4Rqz1oG3XPgqYcIuzRY/66DmysMazGlOo4rgcVLj2FDWdT+GasPXA7/KIXnzY2H9Bp2LrB46WCXChBMH0TiYKu6z6FEx4jZW4ftJxVKsXe1jwiVwuxdwOGJZLdcTsHUY8LsbcKwUWh/6GHnigMwkdMSX/OcXf+iwvnfDkyBPaMhrr3y0IPCbSux+JLgq7uM/OLuOfuvbJjKqfcvgIv/M6G+8xg2+uQ32/IY5q8S5l8OuP7lFouXjIGvdg52aZAE99U3QHvpCSTnmlK7Q5w60up7SrHMUXIQVAGrH9gHqYkeV79cY/ODMDEMSCmHljj0FxW2nyisv+8wrhT6c4nN7/S4/RNtrpFAlroJ1LeeJeCJs+TjmcH9NzRWf+eAYOZo7gKS1BA+YfZqAD5qoMeIYaVg+oRqGxAaAZEUZi8HmE0F6ThAbO4S+nOFxY84UPRzRTallhhWEtU+oHsnMHvjoGyE62o0dwnd6wF2w6Amc/CTtyg0NGDNvpRoLx2qrYNvWnSXHoCeJLbdZaDMTRl0lx7SJeyjQXfloaxCc+PhZwr7Fwqb34mUUG6B1Q8dDo812kuLw9MW0iV07wmMlCMTAgDV3YjhosmDVA1zTDDHCH0MsEsNcwgwW4tYK9x9s8HmO0f4hclMFweTxdcUDcO2lj8aMG4MhAfqnZiSZru3liE1Zwb1jUM7k2iuLUJVo9qGHNZhUN8HtG962LOGQUeHgMMzg2qfMHs9AinBrQykS6j2TPSUIaF7O6KaUdpXEh+la1DfWBye11h8cYD0M3Tfu8W4eoT2nUVdSdiVRve6x/FZi+bKor0S8C2TPOsbCzUYhFaie91D+gb1+x5JdTBbh/FRhe6dyzJEiWqbZYhzjeqO9QtmawHRQPgEt1Co7j2CMZi9HjFcVKju+d5Sg8dwUaPaejTXmEJxyncBEkGc2fIeFGPA/S8sUO94LfUQMH+dYJcK5mYAUgO9txCxQX05INYalRaoXt1Dfn4G3fsJTDB9PEGOkcnVjWRi6I2FvjlAjTOyQQsCEX0M7Ah8O5I9yoN7NQbo2x66bzCeVTA7Vn6Yvc+hORG6NVnentA/qaH3Fmo3QMQ5hE84PqvRfv8GcdFk3xkZKEjBFFcjofcOaQKyDnpv0cUEaQkmhyctqnt+Vpl9gN6NSKrjuvZkWOS2R1ySAhY+IswqqN0AdQ+ERY3YMDRG7S2lz4cBYTNjpdh9j9hUaIBcxSEQjYI6WsRKA7KirHT0ENZDDgpy8Ggzo6Mu72E2da4MCxCWEyaxVqje7jC8WMG8vkeSa57vux7VcUBcdoAQUL0mONISblVPxyQqNQEupSXM1RFVoyHvjwibGdTdkTUaxxEiJrinK+irPfyjOYx30LdHAJRoyqGC3g4EI/WJwUNgEmpSEup2B+E6JCOhr3uEJTsYk1HQd0wgDfOa9R6bjuxtoGcuKQnZE1Cq+56g6WYPOA8tgTCrYV7fIFysWDFyf4AYKsSuJjCrFfTlFnHZQfkIMVqIIUH4ACUlOxhXHeR+gOoqyLsDsKK8VV3dIy06grfR8hhtQyDXNRDbA1Q+bnXfE4iVa5fPZ5GSiqHIbRxUkZPGBCEZwgOjAUMljbjfUwIaud20WRIMzlqI3YGgymjgvgeaGhhGprEeeojKQB0N0jBCaE0gdThCiFlxyPB4RAD6AaJrWX2SuyJFUwPOQTgC+uJdFEqybqSu2SmZWT2hNdJ2B1FVSNYyHdZ7wHuG9miNeHsHMZ9BSIYB8csxB+CkhFTOB4AfmwrrPdddpK2lRgRAso7nqjCaAB4G9iTvIbLMFYGsbgoBctZ+wErGDBLLcfzY5Q9iMb+GS/g5Ctv5aS9fb8bykxfp+b/5r0EfBL1zVUDqNUQTgK1GqiPMrYY786jfGIwfWQamAEzw7FlozgcoBxRNgLijBCE21PkLKz8I+5iSNSMY2JE/NUoAi9ophEWAyAmxJZkTkf5EfaDXMZ1Z6Nc13LmHvqGMUwQgdBFylFnOQ6li805ieMKZ1xJc4Vb0FibFigcGW5AJsY/I2pYie7IrAscXHs07DbeIYOx3yomEgqmvBhg3cWJK6VkT3Jblc4ZzgvPhcUBq7jVnIgAAIABJREFUA8x7w3MiycKO5yxRHzeRMfz3rAYp0lMmgp4kcEmSMa3u5BRe4ueUOipLX6HqKTc8PuX1KpMG+ijgZ3G6pnoQEA6ApCyQEieBcRNR32ZWIcvZ7Irrai/JnEWdYA6UkLL+Axge85zrnsnAIntMIZBrFlL2sfE6Sc8JifFRRHUjJ4ZaBDGFdfg55Z12Q7lraHIwS77Hqh2Zt9CQndQHyiGLd5Opp5Q7Jgnsvu3RfqkROqbA1jcnlqf49PItDrvO/ZmBqbx2BcjxdE5EPLHHdk0WuyT5lvvcLunNLD5CXr/MWqqTfzJU7Dn1nZgeZ/UOphoWAFNqrnSYpKfNezFJPvXAY/AznoviB0X2WM5eiklWaddUBZS/FzamXPfiR2uuyRyKwImnImUtjCuTP0/HXEJPyn77ju9/nY9Z7/mYdMhyx+xJXJ2OgwxvYYceHHPe/kOpJQMukNMzuT41njyqiJlhz6EqJaGXfj4+p4T6FH+k7/I9JouCgY+FmtuT40k+6BaUDZrDifmyS8p0RUhobsk2lmAJPPjsKfJNft6VlE3uf+m65fs1Pz+mHPBBZt13ZEwKW4QE1PcRwzlTL9WAXAmT5afZ1+s6esPUgFwPkT+jNaaQGqYXk+kvCadFylmutfRkpekd52dENFQY2CUVBeW9EnP6rswhPcryeHx38iuqkc+THjmZNnsRsyol5endon4oCa2FGZ7OMbj95jbCtVmiGThRFGox+TuTBKodWeAiBy6J0tFQrlokqlM4TZWl0n3Kkkc5yWHNIftoGzGlUBe/ZmEgJ98ksme1zSEz+fOheF5d7hMFGF4zKXy6k3S2MNoilkqgBDcvoSgfMs4i5BCcfK59c6rombpHHyRDR8OE3SJxLZ9f9NEG9BcG5sBeSz/LtRuNhB4jU4fzcUpH+Wp9H+BmlNNKmzBuFOr7MPWJurmCGuMkxS3BMvQ0avq9F5xMKj2iADKDRhmtmyvUN449pfpUi1I8o9EwVRUAk7aHyC7SXO/kOwXhE/QQ8iQDmUu70tD7B97OzAqXlFzp+T0XMospXZqSbss/dWRaqFtqvjfHgGBYL2L2uYdVCkgbMvvJ/loRIitTSqiNEB8chwjZYyn4t0kmmcerhSVNgrUmam8ROqa0ipj/5dCj8t1XpKexy4mylvsuXKBkVWcPpBBMrhV5P/L+ycOIOKun8B7h4+TF5AnKry+LyFUy2c8JgCC5MJzW8R43+kM/odFkR/N5KKnPCNkTqVX2MCcm3GavpLAOKae5itFiqvlQctqfIpstSbPJOoiuwQdLuQeHkaA2RP4/he4U6S4TZtNoIXLK7uSrLPUiDwOHHvZuPlgmWeuP69p8+Lyy3QyK/26XnxfG8l/+q//4T329/+4/8N/+kR8b8DVnLIWJ+Ojvf4ebQwe3r/Htj97juy8f4x/91g/ww+0G1/czPP3sCu/v54g3Gi9eXOP11RoAmPa5sZAiIdxVaC56zNsRVy/XQB6QP/r4DlfvlxBzhxQk1LsK6UWP6CWkjpAyQqmE4b4mYJ17SB3RPtnDOYVxV8OZiHozwF520GcD0nUD22SJURXgnzFRNXQSUABWFiIKNE9GNJXD3XfPkEzC8YUH6kgf5mUDX0fAREQopC7A5dFJqhPVEzpCPR3hthU/sEdKh8y9gv1Wz3CgO4VUR6Q6IFoJv6QfMLURejnCv+kghwwwZYKIAsfnAVg7uFhDX/TwVsMvIsN0zgPchgBvqAMHab+4x+6yg7ASoYnA2kH+qCYYlGSdkQTiLGBoIhbf1egv+IUmwEEH5cQRUIkpnxcW9XcajM85+5zq/GEVgdhICA+YrYS9YLKo3kmkOmE8j9NAAgCwtqi/15ANfcwPtNAJeh8lMJ5FxDZg/viA4xdLIHv21ChgzwOa1wxFijLBnvP18khPhXp6RNrOsPtlx9TXkdJptwkQbYA/FxC9QqgS7IWHsBKyp88ztII+wT95D/e9JdQguC9NhOwl3FmA7BWGR8DslWTNjgLsJqB5r5EUMFwE1sZ4wPT0ZErHtE4kgVQn2HVCc5l9nC3l4W4dUV8r9E95rGbIYNSdJlP8MmapFgfFbh1QXbOWpf/Yo/uhxvgoIswikmJ0IsEF7yXfEWj7jr2wzRWBkEiAn8UcvCNw/NSjulTwC4L4qAhAWAmUPaDzCD9TlPO+CFh8VxGoZhCXJGBn9HAWybkagftvAwAl6rFKsDU7QyGA4zP6YSGA5i3DS9rLhO3nnIToH6cp4VVccYC3/yxPFqxykJMV2D8iEOufcQJI9wJ2xevYfalx/Cxi/R2B/nmkZN0Bxxce6ihRX8kJfAKcYNj+QkTzljPrpWtUWo5rVA8cn9DvmQSl2MU/6ltKPkNNUF7dEkj0j3OK68BJkeqWILBfCXRvebzme8Dus4T5lwR4/RN+ofsZFR7KFg8a97G5JBAdlxlMBeRuSKC9ykDGsZu0e0/WPLSUdibFLtX5q4jtZwLVloPr0ApsP+dE1erv5MqamL2nPUOJhkeUhw9nAlWiemXxo4j9c1YRHT6jJ7S8//snQPtOYFzRi9hf0IoQaqC5Irit7xL6i9PxASl7TTnQCzXXF2sCV7OnCsHPBJorSqT3H4vpvVPdsRfSd8Dyi4jDRxLtu4RxKbD4MmJYC+w+lVj+gLLt2etchVNxgBqaLL+3rEBRI/dx/1yh2vI8HZ8X6bnE/iP2qo5rAvDhEV+TjgTjSQrU9xHHxwSRyuZk3ZArYI4pB7xlkF8B/Zz7P3sdkUzuS23oIbbz/P55QrVOsRPUtwnIFT4pT5gM5/TBDo8EzI7bPj6S7Id9ydepkZ8Jh6f0eNZ3kec+JKRKZt9m5MSAIZhOHS0Zs/cBvqGMmcE2BK2mp0eSQXA5BEsJJBkRGgbx2KcKszdUWdgMznutUW8jjk8U5q/CJN0ezvRkRQEIUMeVyiFyWQovJESdpboZ0PYXFY6PKftOSkD0VFzU2zhNhsWWHa6zN/RN+prKi1hJTjA1lGOPC4nuigB4XEqqGg4Rdq3QXDsyz480qj0lxfWNQ6gUq4RCQv/IoNoxEdZ1rK9RjnJo6RL0MSK0/NLUhwA9BIwrk8GbwnBuUG0D9N5hPK+nTtrDRzWUTehe9xguGjSXA/fdSNhHFZorhviYewu7MZA2Yjw3eZsBvqEqobqzEFJgeFTRw5kAsx1hz3LvZiMngBpyqJHqA9PbJWtoohQwtxZ+1RAwZsAZWgPMqPAItYLZAYjszETKDLuivDjVuQ1gVrFnc9OgertDbGp2bPaegTxHh5TlubFSZKGXFTea18UQoHpi8qaEWEu28oMgHqWQRESaN5D3B/oUp3TZhLieQd4dThUnKU0/P/RlllAhSDmBPPGww1I8kN6mdOqoDPEDmazI1S3Jg/JXIQFn6e0UBIdCSc4LlG7KEjpUgofyNrg+nQGpwu8HF4UC9zWJP5TH8udhSQDiw6jwP2bL15qx3PzCRfpz/9k/DR8l3h6WAICQBD6a3+PNYQklEgav8dnqBr/99hmW3QAtI5b1gKvjDCGHuByHGueLA3yUuLqbo64dxtFgtTjifjvD2XqPwWmEIOGcgjEBTeWw27doWgslEqxXsKOBVBGbxREhStzvGzSNQwgSKQnY0UCIhJQEZjN+Ex0ODZaLI45DBTdqzBelmkTDe4nxUEHVgRUmTsFUHsEr+PsKZj0iJQFtApxTmHUjdncd6pllpYiMGEYDN2gG+EQB7A3Mox4hSEjBJN0QJGIUSFHCDxrVzMINGag6CaETZBUgRYI7GDx6usXV5QJPn97h6nYBbQLGQwWhIqRKDHOTcaLMlEoYdjVU45GCQHQKuvGIUSBaavSFjkijAmRCuxrQX+cUjLztODKVFAkMMtobiI4fLu1sxPG+BUSCkAm4N0gNgTciK0ekTPDbCjCR6xs0hJUMFHpfIcwi1NIh9IqTBJnZEQ3poOQkH9cR4qggH42ItzVSlQ34TeBzEge9ZmnhLxuImOsvRIJIAqmKZNaj4L7uDLBwZA8GxXCmJkDsMmAWgLnWcCsCHdEGCB0RjxpiZP1GzM+TPUFt0gmpCUAUEE5yssAkqKOEX3smFScOkJPM9SISE0AXIweaqSKQB4Awj1OVh/CZ+crTwakJBMaDRHw8Qr+q4c7I2AsvpkmJ6lYyfReYKiP8giFMSWe5ZhMBkaYEX3lQSHWC7MXEiAGYUofLPkpHZrtUipSFs868luBpyqxxhBwkzIFstrTi1EcZMO2ntGLy6tpHAdWVgp8nxHlgL+0g8uD/lCQc2gQ55r7SOd8PYRGht2Sn6UEW8HMCRbsuPjjkCoQEc8/nPozdpxeNxzlVbSAn9IrMhO8ehCAJINZpSvB1czKJKgNc3TPkyG4iu3iPDDxKEtPjzfvTNTvVb5zAUqx4rJBM85VWTKxuqSRJmus0e3Gq6mgIhkvQSqiZYlwST908JxaDwE1aPmZ2p0TPWOegpNzjTZDJ9QInFrewfw+rMR4yw4h83ZRgnE4APFZ8Tspv7Um5gtPrWatxOj/Ff87+TEzXlvcTTmnEmQH3HaZe2MLuukVmMEeex+IxL4qOwhCrvrAxfE35udTJfFVFwAAmWi1Kd22pSaEKg4C5vNdCw9eaPdnmUttS7QiSJra1gEZNpi5J/nxKSs3sV0OQa5enQJ0SVBNyh6Q5ZC/jg37Y0rfK/IMHXl+Z2fCc4sr7s3QpYnovlOtVKnnK48XDPlXMqOylz4FGbs5JGT2kKSRI2RwmZEtab/pgXfx8Exl45YTkzKoWbzEnTE8/K5eDnPaZZc19lK7LnmQF+FYSPMvTcUyfc4XVEmJ6bTQEeCm/Vo30qJe6kdINaucS7VWA7yTXKZh8G4w4VXFEQIaUWcsc4hNP6wAye509s6zPEbnPM00dmx/U0gAITa430TmRusk+8j5MQT/0FucuzVyJUpjyqJiaXNJqSwdtqR8p52ViTfGQFc7fl+J0nHIMgOTnlBrDgxCiwphHSowPFqE1kC4QVNowMZncSAYNmSWFLoylI0AtwT2Z1SO7eqoUSUpNoT4fKAJc7pMsSbH9iFQZyna7BmJ0BKtNzfX6QA+okvzbw3ulsInjh/LTSQprSudNIhAtYO6BvLUcZ3ogr52ksoXpLOxlWddDcFkee7j8hDTYh/v3dfdYPv+lTfoX/6s/81Nf77//J/+bP/JjA77mjGVIEpX0eHfc4NWX55CtR9wbvFVn7EfcVahWI37z7hOkBLzf11DvK7yuycCEeeQMbS9w93nE8dUc+iBhZYtYJ9yMCvpdhdtXDUooECIQVw7DlwvEKiK8aaGPAm4Zmax5p/D+qQF2GnovMeAkx0svBuB1C5GA3ayC7FmRcfeRgHpTA3XC7r4iS+gF/CpA36tTPx2AGAD/xEHvFNSbGU+EAESbcJQtzAgMFwr1e4XDE4/qSrPPFif2A9czmNwvlwAIDcR5hPACszuB/qmGdJSeiigesCYJjRW4UgvIO4O3/gzNKwO7jpi9ZuLo+Cig+1Kjfx5QXysMTzxWL+6A315A2gq+S6wvmWvoUcAtIqpbRUlyoly3f67QvVEIDT2s9W3FWe7HAe0bBbtK6C4F3FLBdwk21uhu86C+y7UOScEui0y1glsk6ERGUtwYVANgLwLa/7vhwMsJhF6i2UqYHUOd9CAQtZqqWlj5QBlwfNeh2oLF3hEYz3LnY8zBPXca3S3TSFWWJOsBcJ2CX2jKDPc8t+OZJFsGSun2n0joXiD1p8fNTrOu5q1GUhmUzRLaNwJuKaeKDX3MEkCoaVCohzxYXwDVraGvtwGG84TFFwKxEth9FlG/4brb9wLHJwnVa4a22AUQDjIHhyR0b5goavZkBY8fJ7SvFesedg2qHTi9mGWFapAPZIIC1V1h3AR6KXH2fwHbzwlyYsVz1Vwm6CMLs+2abGAZLCeVg4QaSQnsyPM+boDmhgFGbiEgRx6nnwH1LcO4Ft8DkIDd5xJ6z7Cq6l7mSpuUz1Opqsm1MldkXfZHzfCmtYTNsnDpgOYmYfeZRHWXAVVFr7ebUUIdNVOnqy0HMb7LnYLXCvUd3w++Bcwuy4Idk0JDLdC9Z6CN75jEKh3QvcEUupIk/cVJA8cnOXmz4mO+43EgZvAQOWhrrnmN7Yr3X5FOM72U12X+MmL/kUT7npUg1Y7rLXLfEiQVdfaVbyh5lw6obyPGtUSRWuojz2lzk6b+Pjs/pZYikSFrbpgq29xEjBuG+iRJdo7hTXxO/0iiOaTcF8j73rcC3Tuyb+016zrmrwPGJe+nEmxU3zFoyK7EBGDMMcHtRJZb89xHxcd9IybQUiTNZk8fve7TJFdWFhPoUpbS3aSA/lxO8sypnmQAZu8Cj+Mmwi4kmruIcSERGqC9ijg8UfS3Z8ll7Aso5D6KkZMo3fs4SXzb6xwilWW5UXM/3ExOklc1ZvVApCS32maPe5a4Sg/YOQOZ2NXJUKoi+fSNgIvA/JXP4IL36SS9zbU8yy899k+pnqi3aUrAnb/JSatJoTpE2Bnvk/o+YlhLyAA0N4GdkgfKLn0rJ+8+J0YizFFO4G7qLIwEM+NSYvbOUcKZARyAqXuRMn0JJB5XqOTUx5hkBoJDQppz4qnahxNIrAWqO4/+wqB973B4blDfR0jLe7VIRUu4UAHPwAnQC58Qa4E0Cph9yAmsEXKpoI8R+pilutnzX19b2HUFZRnUVICsb3M6bd7mVKsCsCokJ6vGWgFgiFGoJOrrEXZTobqzOSDH0CMLnfsiGSQltUB17+AWrE7hd2mA2lsk0U77USZczO0Ae95kL3Bg8NAYmJYaE8zeTXUddlOjuh0xnjeoL4+wmwbV3Qi7rulTvhvgFzWki5CjzyDJMJDHa+jdCL+qoX2RR7LjUiTAz5nmKhKmEB4AkPsBYdVOCa1Jywk0TQzl4AkCfYRwgay9EJA58ZVf0gLCeigAoreQUmCq8YiZjSwS1MIGAgwRiuD2nScgzCyiGCzBXwaCcrAncKokvaWVmepHxOCnbYrCbPowSWhLaiycp/IrJYb3SAlRV2RDvQeMOQHJwi4WyWkBiDnkp9SSFMCZct+lkJmxzFUh6WGPpRAEyp4VJFNNyo/t3owfsKk/aRFC/L7M5t9b/uiXrzVjWX/j4/T03/mL0K1HDAIpCVZ59ApQCaIN/B0ARnUqVwdONRb5PznKSaYVMouCmEvrJVgCH0EQ5wW7H/MMqijl9cDkkyo/h2WAPCrEJqJ+r+Dm2ctpM6ORPVehTWQMcg2DyEmufu2hdorMw16euvxmuVw8D9jdkvtT6hJKYm0SgD0PUNlPKm2uYmjz7KU9MQCQTMY1d1kW1iYIJxC7CLWT076pXsCuYx6Qko3yTZoSYQEOyPWO8j960jj4LTPYus9l854siRwpxwtthD6SMdN7geEx6yOKN7VUZNgN5bduwYqEUiMBAPaM0sMSSoTMwISagKx0CZod5XtJkR2Jmt5IvReT77G+5oW1KwIAswOGiywbXfL/WOV0yCzPLJ7E4Zw3l8pS1FLWbXY50TeXnpeuw1CRYapvCexCmzB7KXF8FjF7JZnGesNjcjNed7tKWHwB9E9OUfrF8wXk2elEAFjfYUpA9R0m72qSp/L08YwgbmJmRLlemX3J3rDixWQvJM8juz0F3BJMmFWndFxzBMY1k3kBrksfuV67zDP9IqfNWmD3DUxVKaEmU1DeEyoDRjIVlGdW95SdRkMGqywlkda3BIClb7H4ztyC50UESgXNlufPzYHZSw6afcfjVQOmZGeVWRnf5tTmzPQU5sctTqCK28sl2gKQNhGwL9m/2V6RHfH5nCSJqfIj1AQyhR0ze6aiApiYi3Lu1JDBpKNEskhQ7VJMjG3xaZo9gYU5Zo+1O6XOmn2amKXhkThNWvSUdxYWRyROBhUmqd4m2JmAmxPEyXx9QsW/P0waLmBS92nyS1P+mMHeARn4pskjq2zCuJFTl2V9T58nQ9ByJck9gVKs8kRNT9Z3YqYKy6i4b9InDJscftYK1FsCPBlOzy++Ppk9a3ZBCSJSBtb7CDuX0/WQnuBVjfzZ9Hw9fYdMtrULOYEx3WdwN2appKAP0s7EBz5QZdP0mgKojhecwEkSqHdp8h6OS4Fqfzp3SRLQx8xQ6TFN8lD6NAkcY+5FLb5ONUTYpYLr8r0ST4BTpHwtQW8jJ88i643OFZRNU0JuCZYrjHW1DdOxKxvhZpz9FCFN+xdMXucY6ZF9AAjNMUL17IssgDCqXPfRR4SG3YolzbUk4gL5PrOs6yhMXtKAayXquwDpItxSw+R02sJwAYCyTK8tY4eSvF0+J2UBuhlEsi+U96MaItQY6XkMCebg4Wb69/wvPQOMSsLrlNSaAUrIKbOFnTM5DIu1H5R0lnMZajmFK/mFgTp6gihgSgpWY0RoFfSOIUciJsRasWeyVkyX3VG2Gio5BVSFhudBjkwNDq2eAqBilX2AOQQoVRLBSKiBQFOOAVFLqMEjtAaqdwiNnhJzY61zrUcZTAmo3YDYVQRugkAvtprs4bwGYoI6MMSndFQWthIgQynLa0vlRk5FFb2DcJSyApgAcPFIIiQC5JSmMUypBBEpMWW1rk7VIZlZFNaxEqT4KfPP0IoATkn+XoDpw75JILOZZEmnBNnCWmbQ9hC8TqCw/P4Veevkr4yRgDFv9/eAywdLSXOdlvRgfWUbBZCmiFTA7cMlJdaMlDCiHwssHxzHj/vbw9W5nxDy8/9i+XlgLJ/90ib9C3/lz/7U1/sf/qn/+o/82ICfIWMphHgB4C8DeAJCsF9PKf2aEOIMwF8F8BmAHwD4Z1JKt4I5w78G4M8DOAL4Cyml3/x9tyETVOvxJ56/wxfXZ5AyYRwMvADqmYUdNGQdcbY6YHdsML7rINYWQiYYE9DWFreXC3TrHsebDu2mR3/VQXYen37zHb74nWdImoE+cZQQViI1EXJHb6IwEavNAXdvlhCdR+oVzNJi1o3YblvEJLBeHXH/cgW1tLBjQ9ngoFC9OGDsDVIQCEmgno8YbxsOAC880n0FBMCsRjhTQVYBzhjARIj3huFAg0TqAoKXEE5ABIEwixCthxsU1FYjLCknjBrA0iHuDPscj4oMUZ0QBeh523jIrYY7i8DcIR01hBETqDRPj0jfm8GuI1KdAyPmEWnh2XVZAfW1xPCMzK59FJHuFCVJMsvmMlMMqRA6ykFSlSjjfDQiHQy8JHvq24RkElwXoO81Qs1ORySF2CTYTUTYeCCZLDel/yiphLBICF5kjxnBitsEzL+vMZ4l+vy8hF1HyFHAdxH6IBGrCLc8DRT6Z2kC6gjA/rMAOQoMT+iL811C6BjCM24Y0hQrMBBHAt0bMQGf4SKiuhdTLYyfpxy+wsAgNYocNMR1MoAIiCsPt60gR2B4lGWBq4TqJfsxq0PEIXsPfe7UZOCLwPE5Jxz0nn6oEnhTBpwc4HNyoEgNQ3sKmFJ9/lKWBI5mKzCuEppLgm/hWT8STYKfJYznPG/DYw7KywQHfYpkFRFP7E5J8g0Nj59sBuW5scL0Je47BveM52kKiKqvMpMRCVTHs1PoUpHM+hkycwAcPmIoUZFBhoYSzGgEhCCocsuT5DCpHJqU5bR+XiSyAuoaGM6y9C5gAlz6eJLuAZRUqp49fmRvCfSqXZx6LQ/PMiu9z6BkJgjEZgxySiKD9X3C8Sk9onaJLNc6ATuzB4Zzjn9cR6Ym1NnnaIDYApACwxkBNsEZj6OwlSVYK9RkXt08S1azbLK/IKCQY7mGBEtuTuABmaWrXsBXHLz6NofRpCwtzUWhvqPPklVGp/sE+fNiXHNyQXqgu6QHsUgk3Vzg+FjkCQ6RmTqye6ERk+yVky/055bXFfYuGvqz1EA2LUlM/6eYJbYJAMTUxxkqbqu9TjkVFlNXZFI8L9O6unz/58mdcn5DJfI5kXAzATc7TUgpl4N1Ur4vAxAWIvdT8v3hcjCQSDlgaeBxl/5HNSaYI3I3JPLkY8KwJjssA8NykC9FFMC4JBB0Cwlfc/2mTxjWepIPjyuB5g4IOtfvnEvUkZ2VUzhVoyapc6zE5MENFSeHCos9bKj0ODyTqO/IMnbXmY1JyHK9/B5uJEwG7wUM+0Zj9iYnQGdQW+Slfac+kEAWaabrKKEsANfNqMIItZiCiUIrEWvu83BuYI4RCSdAl44EironQ+ky4xoVmc3ivQxzso9Fjmv2AW6uGKzVSlT3Ab7VfB+0egq5KRMiQAaGDQNvhE8ZAMocrKTJ5rmE8awmIF8ZmD0H9dFIxFbkJGc5MZqhUYCkhJQSbwHlIiegz6pcbcZQnFgr+E5BDRFuWeVANEHAPzNwcyYo+xmHkaHhJIc+ZFBcrqXAJCWNuRoGIRE0KgHX6Qm8x4pVHyKyN/JUOcLakNBqCJPDkkZWkqQCjpSYKkSSlgS0QAa3Yuq85MkVrD1ZNWQ6lUDo2il5HTiFyMTWIBoFc30gy6kJOGNn8r65k1suRsSmgsxy1om1qyt2PgKngB03ItUackhIlSLAVBlYlsCdDCoB0CspJR9r61OwUK4+gZKZbS1a5wJWy6yHOgFNpcggpnTyK8aE9IALfAgARdkfACngBC5/XD1Ikbvm8/FhgA/Z3YfnawKZPw5U/jFMg/3/w/IzYyyFEM8APEsp/aYQYgHg/wDwTwH4CwBuUkp/SQjxbwHYpJR+VQjx5wH8RRBY/mkAv5ZS+tO/3zbqTz9Oz/+Nfx1Jpcn7leoI4STMrYR96qCvDUJL039SiT6ySKmp3mcfUx54lxTI0CTEvB6o/MayBG71TS5Tn9ETFOoEP49Q/ak3UVkBNzsV3qtBwC8j2pcKw0WkPA0MC3FLDozLl19h2aTjYKKUs9O/lVC6uPwsIckEfZSYkg/zl2t1R4ZM9wKhSoiZqXvorQoNQZpwAnKkZM5s8+sO/DvAwRcy8GSKo8iJiwQdvgSjPInQx1xJMINlAAAgAElEQVR4//jE4DVXZD18R+masgBSPhYJqCMBxORLUkXuRQAR2pTDXABzz3NvdsDxORnLUKfpfHIGGVPKpQgEs0mRNRTh5H9LGmguWUSvjxz8FUZu8jqFzMiNyMck0FwB+xf5HLXcx1gDiEBzneWmd+y1jCVkRvF86V5MDE8JJSqdpX7GQalbJbRvxeT1OUlbMSW8lgGqGinZNftcGp/TXctgJknkWgSCnuqe+8frmrfb8ToWFjJUmf24Iwhw3SmVlaEemQXOx1H8V3Z12odQZ6lqHgAX9s3N8/E6yjqrLe+P3TeA+pYsrhq4nf5xwuwlQcHkkVOAPpAxmVI790D/lOe5uU44PD8lbkadWdGB/aElyATIrFwlJp+e6nnNdH9ikpmImrs6G953vj356aTjwI9y7TRdn1hhYjFEBksysNMuab6OiZsCwwWw+i5ZHbvOiZYRU0eqz8CuymyUOSbsP2IFTkkxjZkRBDIzp3kPlkTQkvBa2MLQ5GCWzHSW4y0D9MljeE/AqA+8vvqY0NyRSQtNBtWeAKgkuoYaU9cjshxTZxmnXfL+l54TTOOa8lj6sU4MZmHQS2+u2RNwkakXUy9oYV0BsG7mTE7slDnwWje37AJtbym7LL5Plf1e1T5i+0JP3Z96JNtYWMdql9mwzJ6pMX8ecX5hqvUpyabFS+e6zP72ZNd8Laf3XPFrNrcB/blmUuwsV0QZ3mvVNk3puWoki1fv4nTfmAN7E8elnDyCypKFLBVU7TW9efWWgTX1LsDXkvcmTmmyyjKgxc3VBMZKiqsIWQKdmT2RyG7253rq8SQzdmILhU84PjaotwHSJgxnBK0AGV4I5LAYAjSzdTg+q7NUN6I/06gOcQq6Ka9lAmrpcqVh2s0UGcKYptTW+tZj3Gje6/leLMcsEuBrwX0bI9xC5xTR02e+CAm+lWivLMFfyqynkajuRuw/6Rho0wiYfZzk78VXq8ZTeioZvuzxc/QChkay6sRF2JWZUlzN1k3dlrrPzF2l2HN5Y2FXZurq1D3rfyAIuqa+y8wSuqUhEM2MZXVPxlGOAW5hUN1buCUBjt6TLZQ2QPqIaFROZWYVlkgJ6uAQOkOAJshYFiZS5H7PWCuY+xHRKDKeY5g8iKUjU+a+SX3dw5+30Pcj3LqBGjyiYRVLAZXq4IBcE5OU4O+SbGJYNJTIPgA2UxdlTnlNRpLtM4osqlETEy9dQJHPFo8jnEfqagLQ3rEfM6RT8mtJdnUM9BGDO8ldgVNgTgnEAQiqtDoxlgU8ZVA3sY7luWUbxRuZEhnKzGQmo9lVWUBlSZQt6ylMZ2EyC+gs6y/ny/sPf//qktIkdZ2Wktgq5IesZWYqIViL8tVwnmnJILJ0Vf5E1vIPWlL8Q4X3/Lwwlv/8X/knfurr/Y/+1F//A49NCPEDADtw9sT/LM7Fz4yxTCm9AfAm/7wTQnwHwEcA/kkAfyY/7T8H8D8B+NX8+F9OvNv+phBiLYR4ltfz4xeVIJ4MiIPG2cUWx6HGNy+u8Op+hf2mwdPNDuZFxMvffQxsRiBIVK2D1gFCAIfLDmY1IkUJ21aASnj+4hoxCby/XiLdV1i/uMPt1QJiFRB6jf5JQNproIqoP+8x7mvo2iMGhXBvkJ45PLm4w5dfnqNZjRAiob/sYNYjxr5FemT5vDrCLwVQRahrg9hGJBPhJSAbD3cwQBUBS31VWAPqTiPMybr5TwZIleB6DXFU2Hx2i9sfbJDaAL+UlAFfGYRzB4yKd1AnkGYBdg1q/fcasY6ICw9cG7gFEB5b4B2/cNLHA+JVDqipItnMJw7mneEXqADcEwfVV4hrh+gq7D4PSFWEPKqJdXHrAH2Q8E8swkEjmQh9p+FXARUU3ILhKWEVACfQvWRYje9YYh+6CL1X9Fk+9XBzmQumCeqLJFj3An4OyE8P8K+6abBQBnIAB7Bxw2AbNUrEGhjmZBwLcA+5msXPE+yKwJMy1QgkiVgn2PoEDuw6wtxL2CUwXEQcPomYf6ExdEwq1Vt62Pp1gN5JtJdkjfSRKaLVllLicB5RXyrYNQGfPgr0K4LYaFL2UPJY7YagNn3riOF1C72nLLi65zrVSJA0nhHcjGf8UispoH6eJm9M1GkKZFGWKao2cTvjeUS6ZW2K7Qg4oibjK62AOZAVcXPkkBaB8SKi2slcL4ApKdXsBcQAHF7EEyBCnvzIUmbf8fECzqJmIuvwiNeA3qvMulrKNFVPxre9JKsqIlDf8FgJCCi3Fv7EXlZbglA3471hF1wHIJBmyGEeHAyWChR9EJM/jUwdj7sAAN+IqfrCd6f7rbnKjOeC8s0CVkslh8uSR7sCqjsC2vGcbGOZYADohxvOCCQKE8l1cVvmgMkH2D8me+Vnp/oTZHA5Zp/kcMZ7xOwJoopfldJbkaXRZIrLxMOwOVW0mH1mLWdkbOr7CLuiXzU0PD/NNf2SHKwDZpfDYRTX2T8WqG8ySyt4zQ/PE85/K6c95kms41P6PXlPiEnGF1qB9jJOgSx6DxyeUZYbKwGXr8PuY6ZgMlCE8tvdC4nZazH5IkXg8QG8PknynExVJQoTS+k6AltOvfPeNEeyryJiqk8heMrv2cUpJZX3h+L6t0BoBSwy8Gnpw4yG59xnUORaOUlR7UJN4FcPKU8IiZMc9Jglt2cSvslJyUFOzKN0rIxxuVLDLqgs8ZnFq3dk5EIt0Z+RSWuvElJiEIzr8uSNAnS5D2tOUErke80yjVRZ3recHBFZIh7gZgYhScSqypUe2Tveko3Vw6kKhmA6wXWUAUdN9tJ1pdKB29QDMJzpfI4otR1XWbosBKI8XcekyBZLn4NbckUNpdknpUY0EuLA7yM1EojGKjNyPmJ8ZMgMdmKSBicl0Z9rVPsIfYjwc/o4VZ5AtivNvtSGbKJv+FkaWqaylr5fJqk2BKBjZH+rZuiNuXdwKwO0ZGg1CijOuQyVhJ+RORXWwzcNTAG8bQ7ny8xiaPK9NQbAZMbYBYS1QX05wM+rzK5y+2qkVHVYV2jeHxHmFWtKvKFsVABuVUEN4eQxDYnsZAKUUVNoDvuhA8LcQN86xM6QGcwsY+nlLROyCpjYuqglpKWctlSAaB/5WqP4WEyIhkxtwTJFdhs69o2KYQCUROgMk2CFQKw0pGd2Z9ISdl2juhtZ/aMlRG8nhlT4DGALI6pkBrcKcnRkLHPvptz1SCVxVQjAgz2kvT29Np1AYTIaqAzEoc9AWQPFyiWzvAOAsJndFIK9q0o+8H7y/k9tDXEcTqmz+gEEeAg0U4KAJxCUkp2aoshzee6FMeydlAIQGuzopAyZhyYofy1LZip/j1S2gNeSHPv3WMr/L5Y/m1K6+lmtXP7BT/nDL0KIzwD8QwD+NwBPHoDFt6BUFiDo/PLBy17mx766rn9JCPEbQojfwGGP6CREr7Dbt/jmxRWetVuczY4IVuH99RJfvjxH96WCeF8jjRJu1DhezjB8sYBZjQhvOwQvoZcW8AKXtwtYr6B0QJIJQiTU85FflAcFcV1BWglhIoZjhXTQ8NctZrMBcuFQNw7v/+Yz1MsRw77CcKyAKsLd15SA3lZQB0k2FIDYsY5BrC1gEpovDWWwOkIahvdAAvVqQNg4qKXD/pse+ocNHq33kHUAJLD7rXMsv6sgBoX6vGcCrMjrrwNE9prKqsgkgCQTzGYAdgzr8c8s0qDY57cICDsDbCzWv62BkJnNe41qK6C/uYfvEqqZpd/zoOEXgTP5iRJNt4lorgSwdPCzCHlvyHSNJ6a4pI8iCcjWMwXWAWYzIs09QheZZnrG5FThBGavJNbfvgEApCpywD/3iJoSUveu5eCuThCPRwao1PQs1jcC5k4SyFZkZJMhUEKWe4aW6Zl2E9BcSoyPApJOUyJn7AKqWwmzfeBbHfnauPJkrxUQPx6gDhLu3KN7TZbcPbME25+OBDISmEJuDhLDU4/uDUGl2ZIRHJ6EqTMz6sJkBQzPApr/fYbmveQ123gcv20BQTBzfE5JsgwAZEL3nscwXgREQ6YTANxZgJ9FAhMF+E8H+C6h/9hn8JwZ6EPut3zmCHabzM7MGcJkz8Lk66UEi/5BASCZhPE8YP+LlgmuCz63e0tGHZnBdQte//higJvT87r/BFNyql1H2FVmU5cR/RPWILh5wv03Qb/sTuD4jIxGrBPGDcHZ9ltcz/EjdiKWcznpcj4/4Pipn9h6uyCrhgSILI8dz8lsisiqBLtKObBJTDLQWOHk124S7JLbcMuE3Wc8X4UNArieUIuJqTw+pmIgVgRvdh3RP2aIDhLXYzcEvG4B9J+6LOMkW1nA5XCRYHaUKA/nCcM5ZbfuH97B5jCdEvKz/WWH4UlAtUtwS4FxTaBsV3FiHodzAd8JHD4S+X3C80OvIoGFCOkDsH14RqarJMUen5W6FmD3eURzlZniIaF/ynvG7FgDsvssws8x1Xq4ucBwTlBrVwL9EwLUccXB+HhWjjvi8FxmWS+Z4NCclAG7T3MCaZar9k850TGcE2j4hmDl7lcs7FJgOBfoL+gdPTznz7Hm/gBZ9mnIyKYs9Rw3wP4Fr4nrRGZBi/yawGX/QsIuxRSKVOcQpHHDyQK75PERFGY2uD4xp6Hic8e1yJ5CMuPjisc7LiX6i5MUl5NIrHQ5fAQcHmeLhOBxHx/LPGGRMKwUDo+zR00TTNtF9mPuI9xCYPdCwi4E7EJiXDFMJxgBO1cYNgL33zAwPb2rJQ3VdcDhucD20wp99gofzxV2nwgEw+sSK3ZyDmuuc1wT/A5rhWCA/XMJ30r052qakIpV9izPBPYfKRweK0hXJkwoIR4Xgt7THVNIk+Z+hYrHVapVkhJZbps7KG1Cf2GQBHD799XYP1cYFxLHx6zOCBUlsXYu0J9LjEuF+nJAvQ3YPyPbNqWr+oTDU0pf3VzDNwLDRjEQbGFw+23Nc6AFfCcxnjfYfqrhW06m9o8U7FwiVBKHj5rJmwpB6W6pSnOz09Du8EQjdBVTXyvF+pWdxfGx5mO1YkWKj3CLCqGWPDetRhICdlPDd5TF2oXMrDf3TY0Rw9OOz5srBEOm0DeKgUZKQPqYvZ1MgLULiTAz8DOFWGvEWiLMK4wbjbCsKbvNnZVubmCXiudfCxw+pnR1PGOfJAQQWoPQKNiVphwXZHpjJeGWFWKtEFoNuzSQjq91SwPRO0AKBuUoidjxuRACft3AnlUIjYZf1UhSoLoeYM8axM5gPKvhHy0mNtSvGoSZQcq+09iaKQnWr1pgtJOUNXUNYldNQTupMTwWAHHWIKxaAr+YkBpDxjMzmKkyBIizBmKwcE+WSF0N/5itCP7JCu7JMvsVJWLXnBjVtoboxxPgzKAyni0Qz5dIqzniao64mSOuFwSoUk6yVtHUDP4py1d7Lb/CbqYQTs954McUSn7wDyGcQGUo6bK/91/K60x/DIBngkBMP/1/Py/Lzzy8RwgxB/A/A/gPUkp/Qwhxl1JaP/j7bUppI4T47wD8pZTS/5If/x8A/GpK6Td+0rrrzz9Kz/69fxUpSGCUZBgGAUQGqjRXEsO3BiSroLZkvBApTdU5gEbaDBYuLPSrmoPEJk3S1OqOqYJumU6M1tpDXxv6z0SCm58qBKQTsJsAfZTQh+zbmlMGaZdklCCA4RG3bXYMotFHkQN0MEkTS3x/SUxNhumkdsNAGyRuE+Dg351Rbms3KdcFBNQ3apLJ+VlC90rg+DxLNDMoQiIj45YR7VvFAf6c+1pCcyav2Hli3YEVCC07D/2MyZluzplzs+dg0835PPeLR9R/q2NAxwNJaNmn+oagSTrOTtsNE2VDw9frHnmWt+w3B0lka3it6hsOfqZ6hfweK6xlrLJ0N+9TSQc0W7J50hcfE2V4UxhQOvnY2H8Xsfi+hF1l6WjDwX+RdoY635wxB5fk8JoiXbXLk/+v1BjEiuf6Yf1ACakpNRPTsWSptj4KjGcJix+WDkB8UJfhZic5ZTTcx0lCmysICkOle0pPyYyenqss5bl2hRzYgCkVNFYnX2Fo+LzC1KkBGB7zupbaFsTCxjDAp8gwXcfEVrfEBLCTYPjM7jN6VIdzgtpJfqjyNcoeyJKWWVJw63syYNJmdkJz/8YzYPaanr/xDKhvcjjOIUs+jynfZ+KBZJcgXVkCoGqbAZzi8VR3/EwYzshEhYb3cHNTKhXwQXKoXfBYkuJ6Zm+47VBRThsNck0AgaW0wJBDdZLiMZs9GddS2aAHrt8uc5psCa5xaZKOJgHYdYny5zXxLab6geouob6nfNQuKBvuHwnMX0WylJMMO8u0m8JkpimgZzjn+dE9PwfYJSrQ3EaMKwmzPxXZRwOmwq4pzdRDSfXk/ROa03uvvxD58zWH2szFJA3WY5Fq8ryW9x/l3AQWJWiqvBeau4TjYzlJfNurOIXt9I8pqS7yZTvnZ5myDPmp7+jRs/MiBT71dKpShRNTlvyTYfNtlpPn4CDfUtqcBGXaw5lE957+vCJFPj7mtoqM0rf5c6io0RKPu9rFU9BQBOysBN9gApQlrCgpnmeVE0yHlUR1SBgXlOHSf8ntFSkvP1cSmtsAO1eTDLi5DYjVCZjV9wHDhsyZsuxRHFdk3fRQUmHpP0z6FP7jOrLZpqd3skh6+ZnJVNxxpeh1LJUQnnLGIgVn9QSm75RQS+ieEsdoZGbP01Rj4WfZg5nHPsEINDcM6vFdDsjL9Si+OYXbVbuYZd8RvpPQx8h928dpglANOZwnpUnimqRgmuoQ4JZmklULzx7JqAXrMqSY/IAxB93YtUb7zrLf0VIiWySVvlNk8kIC5MlPWV8PU58jIiBHj/G8YXWHT9BHhv3oPeWZhYnzHcEbA3SYbCvdqeYjCaC6J6tmV9UpFTYD5+p2zOebIT6h1dBHRxYtpUmmK2JiVcfoMT5iv6WfG+i9g5+TnVS9m54jAqWeSUuIMSB2Bvr2SHAWElKtII92quhIQnzwGDJjKncDUl3kqmmqBklSIrYasvcQIZzYR+u5zixtTTX7IBEoT43LFnLbnwakKSFVBvI4IHYNJawPxtepMafgoRygww9qBTGMBFbOn/yVhUEsKalS8u9F/gpwPT8uZbVIcEP8ELSVpUhmhUD6St3IJHOdPKISojJIw3B6Tkwn1lMKJOtOEtcCKEsKbA7uEUp9yGB+dflxIUQ/Yfm6h/c8/aWz9M/9l3/up77e//gf/Gs/BPCQifz1lNKvP3yOEOILALfgJ+Z/8tW//zSWn2ndiBDCAPjrAP6LlNLfyA+/KxLX7MN8nx9/BeDFg5d/nB/7iYvsJbq/3bAoe88v1jJ4Gc8k6puE9n2NUFPSVN+eQkuYKinRXmU50BfNg/Q/SrvUyEFE+fLiDL5A/bdM7khLp0Gu5helbxJmr2UOAMnekAQEk9BccpArfML8RyURNmH93eJ54/ZFiDn2O6f8CUx+qSQS6nsOisyxSKCA/4e9N4mVLkvQg74z3Ski3viPmVmZWZXdlGi6BbjUXhnLEhIyg4SQkJHYWEjsQGLBBrFkxQpYWGJhvKBlWsACC8QCqzESbWQ33eCmTXdmVWVVVmb+85tjvMOZWHznnPv+6qpy266iKlsO6dd7/4uIO8eN851vMgdG/EcRcfR58pw0/LbLdSUM4Ihor3NUf5xN8xpcUeQMM1MU0/ES9M5MS8nwE8kS5mAkC7YfSixfeUxLWQrHvRFYvIjYvSfw+H+o4askfzQC3kToMaLaBfRnCnrgc8rSdzOcKDS3vLlJxy8V17Aaor7j9t1PRwwqHzsOBLorh2nJZEc1RdiWA6ppmdMUObCaVkxO7C5Q/FbekMkYzmUZcAqPMrBRg8TqpQNeZmkaB/D+OVmeeh0xnDCdUrpYOsO84QDs5Pss7hY+lhANaelbyp1w6w80qxmSzy/72HLfWR5Y+YpeKxF5rgvYbgRWzxLgOZZwnUB34XH7TymcfeIotZtmX1GUAovXvOY5KCY70N54AqBrBlSMRwJHXzJEIvfPTUcSQXNgmUMs1BjR36jig6o2IflmOGiLEqjXHuOxgl1ItNe+gOco2b0GAGavUO08Fm94/vPgFuD5KGmNgdKw8Uhi8drCLhW6i1iWN60k2kuHYa1Qr7lse50qYJJXqr0GqjsH3yq4Gwmz57q6C3rRRGTFjXRMWjS7JI0bYvJKaQzHEt2lT+yu5zlLfXrBCL7WzX4vNXBbpiMFXwl0l66EatDnKKB3Hsc/iOgfVUkKJ7F4McItNLrLDKo5KJVTKBUEoZ6BQT6m4nNKHus7SvDIZs+JmSIwzCcPXNtrpGoHBdV7ytJayWqE1C/na/oaXSuxeEP/KCLSAJWDUdcpNDcWuvfJZ0omTI0e1ValMBR+UOxSo9pY2KUux6e7THK21FlXtxJm65PfjoOR8USjvrEJPDEYRY0sWZc2lGTN7Ds73nkO/FsJs3MQU4BbaCxe876rBoeoJMNLRt4vVl8QNEgXsXgVGEgSCWSQPLcIHLDqHVGmW+g0QOf+ihCREybtgvtYrVGknPW1BWJEdyHL9evr2X+npgC7nEvg6bUL8C2rHeoqySIPFm5hCGZW/FmSbS39V2ajoQ8OnZE87rf0zdmVRnOd5G4hQg703pmNLrJGlQBDSAmkcvCobgC3MohSwGwmqF5DDT6FS0VEI8lcGb5P7yzGsxpRCTQXPYFgY6D2E4NdJJljsxEEGl1VkkJzZ6A6WAKEPKgVgFtWMJvx7QF3ShCVk0d9ReAQjCRwcQFuVUPvLZ9Lg1u3rNC+nhB0Cr9J8lJz06MGvwflVMPcDWndIgFKk7x+AUalaoi0DDXOLE4wCuow8fNgPUJXQW0ZYx0qDREjqjv6/dRQQa4PiF1NwGMUzK2Y/XXpmoqVhpgczIEyTTFaRKXQ+AAxsnpDDhOE9azWcB61JZgyPhIATo4gtdKQw4QqyRtFTlK1Ds0hd5C5ORnVOojKQN/xXOr1vfeFQJCTtlfYGuIwottPwGRRHQxgHap98h56ynfF6GZgBIA1GjXgPOR+hBgSExgCpHX0IEoJ7OYajuJLTB2LIr9upKQ2VoasJUBGD0nE4gOEFCjJscM4J7hqBek9a0Ly/mkNrLfAooO8TfHnGRBqBbHd8/dU3fGW11BKQHoCNH0vVEfIMuEgVPIw9kjAUc9g7P7PGCC0nrseM8C7zyBmgBcSAMxgMv9NCp6vGBB9QDwcGPYDzBMgu4lgsU/1LXlfE+jNKbDl9f5Hg8HMQAopEO0/gOi6v53/5PGjHld/AtD852KML4QQjwD8lhDi2zHG3/5pbsTPMhVWAPhrAD6JMf5n9576nwD8ZQD/afr5P977+78vhPhvwfCe9U/0VwIIbUD35y9RKY+Xr0/x3tMbnDQ97oYWlVc4TAbb2w6IArLycMbDTQpKB8QoIFXAFsC0q6BuDL72z73EF99+UgZ2saXMVOgAqQP81qA6GbFZ15CdQ1U7jPsK2KU0k5WF1BG+V3j8zh3evDmGOGhg4RAPCu2jA4ZXC0QTARmhlhZhUsDaIFYecpCIpxNiBI5OD9g+O2LRPbh4MUrg2EK9qinztAL+2AEmAKOCSl1fvgtMsDUeovPAVhcZaVh6CBOAdQLHCw94yjRlz/qQ0BIk6DsFAST2UyCK5M87clB7CX/qIHYasbHYva/hHljIrcbml4BoPNB6dEcDXnyjS1JaiaiTj/A0IqwCxJ7JtmIgeBEO0NuI3YcpiXGS9BsOAtNHPdTzBm7FRNbhiUPzmmmx9oGDWiuoETg81ejfcYAEzA1vhrnKQY0SLhXPhyrNYh9SCNFGzWX3e2D3fkqYXXpUVzoluEbsPpAMQxoZ0iRCGnQYHv/qRmD7Icrf7SqgfS0pldUSIcmA5TQHSPlWQPXsr3RdxHhGnxVTSkUK8hGwx2kEGhIg3WqMD+gRHR6SKZcjv7DqO2D7DbK/4xkTbF/8hTTAHiSaS6aNum4OcjHbJN9ZRKydghroPYwqQvXA4YnEdAY0aeBrj1iTI52GXRLMisCJDOkTMx05+KzuBFNWI6AmVu+EChguNYZzMr52SWo4V8yIoEqdz8ZLuEWEryOaK538hxHNdZL+nUVMx+w8nVZkQsdTrn73rqFMVdHLFAxSXYmknNUJbtMiVc9Uc32EPnBfgyLz5FpABEXGz3D9wzlZ5+2HigEggb2euax+PE8M1UGUMKOgeY42HwWYjcDuPfatCi9KBY3eM1jm8Dj5oCKw/VrLazFNDOie+5IZaF9TgZCreaIC1KgKSx8UPXftG05YjGdkAzObnYOexrO0nQYwOw3hGXaleoX2ihNtOYAqajJ39ToxeCm0KCpgOgGqOwU1aNij1A+6YhhSTpR1rUF7yd+rjcJ4ygk+OQKQPJ8A2W67AtSgqRbYILG8gDmvYJciBUfxNaHipN94wmOd/anZUyldhJw0U1pjTOcOGM9q6D2BlV1xm6WjHJne0TlcR/jki0wsF3sJ+dWaK1ZCBdS3mjLoFF40nYgyESodJ5+272k0NwmQNylh2gF4mvYleURZS6ILs83JUlGSaZevNfaPZWHMq43C4bEsIUVBIXk4KwbGGSC+pxle5SLsQpfzp4Z0HJZzcFK9rmBT2m9WRmTW8PBEoL3U5bhUu5h8kUAUDdclAD1UPDenAs0DBqW4TqBeNyUEh7JQQISG9xZfFxaaFSsMZYoKWLxxGI8VhhOJxYUmG5rYY1bHxNJ9yQlO1ssEQ+lrc+spu3VAfedxeKihrJnDygJVAYs3GtnGYRcSeFyxvmQM7PZMQDoHDYmQql2auaJFuIjxVEG6tnS7MmiohZq4TXmSjuy0hHzSFsbWdRLeMDzIN7JMZAUtUvBOTAyqp0x0JdHcOE6kbBsyiynROQPioMVbvZsAIEIHhgsKqMFDWU6o+FqmCQwywKr3KfQugaUkZQ2VRHVnObkycltEmgg0W6bS5VoWAPPkR5LR+kZB711Kl1bpu24zNFYAACAASURBVMsDx21J0s29mwip8mSgdGc6qVCt2Z2ZU2JFiIU95cRYOl8J3MimQmjorYyKybhi4sSi7AlUQ63LcrDqUkosICfPoJ8QEDsm/MdKkym9F+wTjSrHVAycSIoAmUjnEYyGCAGiHxG1KnUjUQhASf491Y7EtiYAzOE9qcYkhkCmd7KITVUmXkpVCUD203ugrmZ29H6PZalOuddfmUF0jIBzEJr+zwxCM8sZfZiTX326qUr9x9nVkhobkyz2JwDHGAAoAtH4E5jPr8AjRsD/nKSrMcYX6eeFEOJvAPizAH6qwPJnmQr75wD8bQD/L5BLA/Afgz7L/x7A+wC+AOtGbhIQ/SsA/iJYN/Lv/CQZLADUH74Xn/xH/wElsKdTAXRXHz8AAHoaqwB1o8uXXlh41nCcMjE2mDm5NXurRACijtAPe9hDBX1p4LsAOcgUOZ9mHLsAvVaswAAAQZ9cliy6lYccJSWYK9ZZ5AqGLMt1XSwppEAKjenSAMcLRJ37JoFokiTnjnLYYDhwFAGwpwF6LcvAPqerZmmYmpCS25JXUGPuONPcJuGB6TjQO1inG+3E5NsoUcJGqjU9VllaqneUZUpLSep4nro4LQfi9IWlk5YG2rnXUY6pk3JN0KD3grUJZ6zu4IAplu3Lki+3SDLZIe1Xkv5TzpYSY1MXZR5835fhkd1Og/EhM7YosrKcrCpyEnedQO+B0tAcdJM7InONhV2ya3JaYZbgJQlbDo4ByKy7ThTp7XSSWNMN1zOdoHS3qZG/5xRW6djZV625LVkGmztOg07L2vJ3s+eA2Owoxc2DQHqu5vVmWaSv5/32TfpOsCipr65DkQarniy4XeYApXQMrmLyTM5JoyyUF6mzkMDC7FPgiQdZn4nvp0+Ug3qzjcjBLtxHetHUCKienkCeG/rBctVHTpPVPYHZ8jmPgxpZyD6tOAjL/Y45FTYnpkbFgT+BzyxvrTaxfH7tEfe7vYwpjj9dEx0H+tU6FhCY1Qf5OnUNgXZzxYF3/5BKhKhRPGm+St2HqXdTj5Rk5mOaX5MTLbOsNydpmh23OWjui/QEBlky62sG3biGvkY9RMpsE5t/eCRRrVMy9cBrL6d1Aii+QeESmIxJ0hhRUmxzIuZ4wuMrPKWzoQIWbxIwkDMwcZ1AcxPK+a7ShFkGW/lemwOgpEUJcimBOUMsFSZZ4lnYYKDUVughKR8SKJQ2Fk9nMLOEMq8vdzhC3E8r5bnJVSGZpdZDTMwuj0UGNVGm5UZKV6s9lRGUEHMDq20oAUIicrsy811tfVKjJE+iyYBgTlv1tSiftyxZDOkzb7uUnnpH1YAekiJlRTbF7H3Z/1zXwXAZHiM1UAKqUx9pBlTKxgJo7FKiubL0z9UyKXGokgCA+s5iPDGJuY6wy5n9tyuVElQJoPIxFI51IDIdd2mZZBtTv2A+H2ogQ11SXu/VjvD8yrlTslWljzX/3+wdXKfSfWFmboMmY+7auc6kWlsMDyqoIZRzpUbWpvgkJQXIdOZ15MkHvXdMaM1s9OhhO9aI6N7DtQRVuR8yVgQXriWLLuKsOOLFEIvsk6E2oSgKcupqZo71ZoRfVghKQu8mhFZzfw8TQpJ8Ck8QKZJ8NdQ8Jr6WqG4HhMYUkEbgA6ieTHZUZCpFYkP9oirAjtJgyxTYwZFJHqkQEDbJitNYgRPAGsInoJQYsagk5GDJyvoEhjN4mhwQgLCsSkosYiwdmFlJVrp4XaDc1QfErgZCSpY1ClGR4S3ryevIwC8DOwBvdWPmR+6NzMdJqwLuohT35KaiAEiReyardHyd/9Ey2PsgEGDSaw7jyb8nlrYwnFoh9gMBYQKAf6zmoyiBEnhLry3hPfcTWWOYWc/EgpJp/REA8t7jLdxxfx/+BIxkDHHetn+Exy+CFPbxr5zFf/s3/6Wf+nL/i3/+v/uJ+yaEWACQKVB1AeC3APwnMcb/5ae5HT/LVNj/A4D4MU//iz/i9RHAv/cPsw6hIp58/RpvLo7hR4VqNeHyeoX4cEIcidS64x79fkmQdjYBkyLLl9iU0AWYGwV7xgtVOLJ3oQqw+wrwAn4ZyOTJgDgyeERtFDsmQw0R5voMd+RZTRIEIFlj4rvI1M5Tj/pCYTqh9NMvA1nCKkLtGARjlwkQA4i1h7lMzGLyzIRVQOgl/JKgNjNNIvk17SoF0UQgnAQIL1Bdyzk1VaIwJ4IVkHBVJFgWgHQC4yPuQ3VHIFwNItWb8H3jgwQqU4UIFrEwG9k3GGWEO2JdAtlSrstXEe0bmRIII6ophdos5sqY7IN1SyafSkvv1vCA2+DbiPa1xHhO4JgrWdjhJiAnDiKiAZBZmFUkkF9GVGsOOnIya9C5n5DA3lcEZb4GhnMOhjKAn47TTPUqQl2Kss8++e6k5fuYUspQDrtiv6LZ8TX9Y7KclEKTBYqa68gBLqXeo07sQsOBPZkbkcB5Wl8Ch4E5UFA9WZ0c5iKn2T/qWq47vyYD7uFhYodAwJ0DYHi8ef6qbVpGRyCKwIH94YT7pqa5UuTwdJZXZhatvp39o2bP9w4NPWyuoyeOvW0E5zlgJVdbuBalpkKkSY7hnBJp3wDjKYGLGmbfFetA+Jrd1wTMloPE/qEswJjeXHrrfJ26NNP63EJA73mcpiMeR/YuzpMEIjJNVKbBLwS3HckzmAFc9iL6NtWxLMh87d4TqLaUyvaPRWHyEGegowb6AW3MgyeCOLKR/OlahrBkdikDGt2na3JJFtzsGfDiFryOhjSplas1CICA/lxCp6RTs+PnYzhNYTyZ5UpjAtfyM2d2aXJGIHnouI7xWKYaBh6fqAXqVBGSa27qNYFVc0twJDwABfRnBLl2IaAP6R4j0j1xQqmQ4HEgqAxmThWVnnL43JkqApNRhQcmI9Hc0utJv+R83Ko9a0DyxFTuYlQTgTdTLdM1kNjPKEV5r3QRw7EskzD3+yDbmwjb8FxNS3oPi19xE9CfK5g+TWB4FLA7rQTZ8vTe7F8UgezZeEypbnPjsX+soYcIQEBNAeOZgu4jqrQu1/EzMKXE0nzOfCULWB1Omayaa2CiBKZzjeaOsl67UPAVa0pcLSAS6NY9qzyGUx5fevQSQyaB/mHFfV8R2ApPWb20orB7aozlmPqW1SQyWQHGEwU1ZfaRn2lped6zfBlIoUFjZg/5HUBmlIBXJuldYcyMwHSkU/ibKMBWTvw8uqWA2XjYlSIYT55JvZ+9jkEJ+FaVapWoyOz5WpZakag0hoc1qrXDdGSgB0/2LNBi4tL2TCcVzNbCd5qS6L0r/xc2FqCpew9oBgKpKRS/pRo4rhnPagLmVKExnbdQBwchInxH6W6o+aUZaoIooQWCTl7PYwN9IEjUIcIek21Uo0/VJh6yd/BdVdg8X7NzUlqfAFMoNSRuWUEdLPyygtpN8MsKciTIjEZC9g4wc/KuHB3cqqY1RhGQRsPwnygF9HYsCa1+WZM9BAjYEkiUu6kAb6aegu9RChiBuGwgRktAK3j/FoJ9lFEpJsdWKZRn9DMTmD2Rzs++RpNBXShJrrFOwD5GguS6AsZpZvnugcpSOzLe67bMy8xJr0IAPoHNXElyH7ApyeWn44DsdU1yXCjF5NbSeylmABmTtNf9kBT5/kNJMrSVmcFm6cyU8+sS43lfJvu2DPg+TPgR6wHeAp9CAfEfA1j+ojx+TmE7jwH8jXT8NYDf/GmDSuBnyFj+//E4/6cfxH/hr/4lTEHjyAx4t72DFBF7V+NiXKJRFoM3WI8thIholIOWHjtbQ4oIGxRq5bAZGygZMHmFj46v8LDaYedr/M7LD/HPPn4BFxQ2tsHt0GJVjbgbWqzqsVwYtXKwQeFgDU6bHtd9h4fdHjd9h0fdFq/2R2i0w9VugY/OrvFiewwlA4SIMDLg9tCi0g6V9jiMFZYNl33a9Hi9XaHSHkJETG7+0E1O45sPLvBidwwjA5QMeH55itWy574ah5vtAst2LO/r6gm7oYb3EkoFHLWzGXvTN7BWYdWN2OwbSBnRNSNW9YSbQ4uzrseLqxMsFwNTc1XA0Ff42sNbfPbZYxw92sF5mbzmAf2hRowCVW3hnISUEYt2xO7Q4HjZ4zBW/FLzfM4ojxAFfJAYBoMHJzscJgMtAyanIUTE7nKB0ycbTE6jrSw2+wZPTze43neotMMwGSgVYJTHZttBGw8pA7yXCF5C6YBp0GgXE6xV5TgIEeGshjYOU2+wPO6xvV5AVh7RS6jKIwLQ2mPqDdrliOFQoW65b1XlMQ4G3knESWL1YI/9tkHd2rKOft3g5MEO1ivsb1tUS0qeY5BwOwPR8By3iwmHZyvEVZr8CGB1jJWQtUcMAtFJmNbCWYW6tbCTRrD8m+0NICNir9A9OOBw0wFWQCwdJ1tkhFARqvJwewPVOcQANN2Ew22L5nhECALTbQO5tFAqwDsJcVFzMmTFG7o5HuEmjXhIEp82Jdz1GvVZj+l1x4kCz25Z2TiEUUFUyTurIoITlGSfTcBVjbBMtKsXUCvLVGIRIaxE1BFQEaLywMYgLjwHAE5SCi4AudHAgxFhr4EoIDqHOEnW6iwdq3skABNgLgzsiWdsezrO0BEIHMCLhUPsFeReIdYRsQoQB8UqnYFSXZcngFL0u75TcCfzBFU0SboAQG2T97ELUPv5SzfLsVmzkJiZiKIYgADXuVXsn61jCvPKzB27ecUoEU2E2ik+LyNUL+GWHiIKiJFl9UHxHOqNKjqSqIDQBiAAei9hTzyEE6huFOxx4ORYDUTDDlgIzB24gyh/z+XiruMEUWjY70u/NjeZXag5qCupEpAmMLqA5o3CeM51ui49nyexi0xdzDLUgJLKPJ5FyDzBBdDG4PmPbBFTfKXlxFWoqbaYTmKR7uaeSyb9cjJOeKomokBRt6RDXxQYWdGQw7aYpMxjonezXDe/J9RxTuqekLz36XVVRH0rYFd87n73bVFfJHbcLTiJJ6fEovZcVq4ecu2sPlBJVmwXs+LCp0me8TRPyMzHmunO87bnEDOZJoI48ZEmPNNEFHMAsp8fhXXyFdeTA6PMnq+pb5MioME8aaBmhQcnymgVyAFgUaVjJufzxc9hOl5pcjMzwPnv8d5YN7ObruNyc/9nvhajArqLwKoXwW1mjgInv3IFTXPDpFsG2CAB33T52XkyLAeT5fCmfKyzUiYrAvJko0x4gP7luZs3qvlzlKW9JbQtxDKp4ytR5LXCM9PAthJ6yEmtBO88f3xO+pxBkY5NI5MXf2bhqEaIaTIgT2i+rSLInaZqyuuYmWqA4N3syMirBI4z613t6AXOagFEgv7c3RolX+urpM7yOfRLFDWTPvj5XGoxM5NI13N6zuxdkhsnljSxsdnvnG+xOfQos85zF2ag1zezpRIQU4C0nr2gLvmwAQLi5L+Wo0OoGRaEe0wrXCiy2sJE+1i2HYkBLhUnk0NoUyCQlOX9eVki+2/vJ7Nmz2MCk8IH9mXm5+89RPaTZiYWeJtdTMt/K/wnA76y/YnFzBLa1Mf51uMem/kT8ci9dX/Vw3se/cp5/Lf++l/8qS/3r3zrN3/u+wb8jMN7ftYPAUDLgI+Wr/G/PvsmXrbHWFYjPlzc4Nn2BN968By/88WHePd8jYWZ8MXdKXwU+PD0Fq+2R1hUE9ZDA6M81vsOSgb8nc8+wr/yzT/E77z8ELttg2eLUyzMhO+8fAzvJN6oiOAF+qMeIQoIETEMBjHwA/06HOP8dIcvbk8xWY1/5uwV/vDFOzhe9eg/O8JnIsI5hW88uMbHXz6FEBFKB3gjsd4YrJY9rFe4fH6C3aM9pIi4WTc4XvW4ec14TrOcYA8VvjBnuNt0CEHgaNXDO4ntroWpHPqxwrCrEIJA8BJCRGyuF6iWE6Z1DbVwGAeDEPjtHSYFoQJurYYf6InzXuAwUEv7/OoExnis1x2EiDg66mHvanwZz4AI7HYNQgJI0gQ07YTDqyXGscbTX7nAixdncE5BPGux+yhg/HKJeD5B6ggpA/aHFkfnewyDQZgUrm5XCLcVHn90hclp9H0FuVMYJoP+sgMe72GvWjQPrzH0FXZ9B7FXrC4xAbL2GO4aLM4PcFbD7zRcQ4A2yArBCciLGv7dHuGyQawDsAJ05bF9vQJMQAwC6jV1lf7EwwdwsP2RR9ga9JOC3Grsjy2kDtAva7hVwOHTE8Q6YAgC6nUN2xDM3V0vgQBUrwz8Rw7+roKwEjICsVeAB/ZOQo8CXigIL+hxPAeqS43pVEIOEtARbstZ9elBRP3tln2TQwUsCDTUTuJQNdA3moNuCSy/rzGd0KPozgQB6JsakMAAQPQKg2shrIDuJVwEQh1QvTSQjsAHKsJcGbgE9uoLBTkJTCeJRd8ojKJBfUdvLHtBBXBZQ5iIaCWqK4XpiYPcKbRvJA6ooEaBqBTUQcJsBYanAs2Fgj0KQCB4oRdWkWUeJCAj6iuJ/psjsNOobiWGVkOMkr2ijyPUmsdJjqbIqYNhMnEUCtHE1GFKGbgamJbspcLRtw36RxHRCWDPShd7KqB6nhd14PZJJ+kztYBIftJgItBrhMcj6s+aAm70K/ob1UA2PGigvuIgyZ541K/o38oMOgc59M22F2QG3QKIKklvpUDoZQE0UbNGpnvBjtbpiMtjfUmE3lOmVt3S69le5GRfBdeSwVcjZXHVmqx2fUPwJfYSIiUCI8oEXCL0WiJqYPllxO59KgyEBc/nNAOimLxq7ZZAcFqJJGMlE68GSWn1XqC5pK8bYe7LFImxU5ZApblK0mbL30Xg612HMlDUB/4uRw7G28skp5WA9QLLFxHX5xH6Tha2P4MoXyOx2ECoZtARNI9BdZd+rjn4z7UsBF6iLGf5krLr8ZggpVonFnhPkCJHqg+WL/j36UiguiMoMDuCleEBGfV6HXF4LNHc8vo4SE4YZJm9GgmQhgcR9S2tBtLxPC5fUHLrax5/s41QlUD3hqnkWW3hFkw4znLr6XhevtnR0yk896+7yExrBnMEbYfH/Lw0t6z2oQWE7GO1iZTAL5nIu3gdsX1XEeQ4yufVEAtAMfsIN6Ew2yr9Xm0Ddk8pwaUPWVJCnaXhAewurRK4lijAR/cEk36X2N5OQFsysxBI2+ZKD6caI/ZPFI4/m7B5v2La8U2AdASw41EKixsjzCEUFri7CKyvSdJZ6QERBMwhwGw9du8YNCmQrrnx6M8VpAeaK5vk5ALTsS5SXDEC04LhdboPsEuJ6s5hPGPPZX09kQndhZS2THVUfT0BT2p2aqZEW92zk9RsPaSL0DsP36rSGdncOEzHHCJWa0tf5rliN2cfEBQZ3ebWkY2tBfQ+FNm59DFZAdjPCTAszbUS9c0InNeobyb0j2vIKaC5jSWR1i0UqjsLXyv4lsqJ+mbEdFwxwXZVpfuaQn3VY3jU0oqTEnHVkIDgBAQjCxsvXITe5/A5D2Ek1GFiAJqPwHmLKAT0nkFgYvQIrS5Msho95bQRBHlKvCXLjUlubG4OCLWBuT3An3bsyhwtwqJmaFIKNYoRlO6CywuLGuqQQ4fY9Sksj4mYLGB0kvoSqKlhKtdWrNl1CYAy2pCY0yTVFYcBMGR4hQ9kOVM/5g9LVREjYAwZ0cwuDinYKHdwAgxDqgyic2Q/AwOIyhhdScptAUpkgTkEKD9UYnJTcuyPH/DfA6D2x7/snzx+/o+vNLDcjg1+74++gd8zX0d7NODNtMKtadE7g9fPzvC3dh1sb/D5548401QFnJ5v8fHzpzCVw/UVEyFk5RFua8TWozvu8T9//GvQxiNMCs8uziBlgDtoiIOC12QCNlbyfRNZoJOzPdbPjyFPJlw+O6Wk9mTCb3/xS7B3Na4GDRWB7c0CcqPx8asFByKth3zZYHx3QNgabF828CcOYlDYvUkSXhlxta0gegUcW4jvLyCWEddqyb7KUWHtJMSNgesCnNZQrYe6qmAfAvFAkCIEMPma8t61QQiJVQGBSzCAO3NQdxqhinA7hWBTX6EO8KGC7CXqa4n1kwrNtcLYelQXGtN7ZGLkrYJ94HC4qSBHymm3f/MJxEce3koYDwyvF2RPbiqwID7A3CpssABGieZCY3jiIK3A1R88gjt3MFca+iDQL1rIXmJ/sYDeSXz3k/egNxKiS12LVxLTA0DcGCgA/X6FKCPqWwW3VNB7BoVIybCY0TQwO4F4UMAtw3vEIsBcmcJY2FUoQUbSCUyvWkgJqDVvgtFo1C8VogGEp6cWULCDTHJJAX2QsEccdOteQP1Rl2byuR3DwyR9vKxQ3QqESqV6kojqQkN4ge6Zhm95/Y3nAWYr4camyGilTdKrXsHsBMR1zcHtBLidhNnGJOUVqF6a2UuqIxb/V8sqEUfJcXUnoA+awUl7kXyiAtW6ooT5hp13qheUtRpJsNUzuKR9M3frxVsCD2RZbMtjV60lulf0NnavKemSE1kYtZcpPEXCbJFmsbmfZg/s3uf+6x4wz6viE0U08InpMtcaesdBpD0CmkvKkNXEgBfpKFvLrAtrCRhgJN+w5/LoM6B/IGCPI7pXAn2QqK+T/7EWEFHC1xHLZxHTCftbzSFi/45kbctNw8HnAUW2TlmhQHMtMNnUJ3kAotKob7JskrJb9gqy2kMfInQPDDIVqvcJ5NQETvTcztJiaVmvEhQH69WaDMjuPQmzJ2ukBsrYfcN+xvqGktjdu5S/6gO3L2qCtmoz+08BlIodlZZdXxP46rRt3QVlpmYfcXgiSp+lHAFVA4sX9At3bwjOqnVMjAYlnFEB3ZsU5GF4HdR39CfnblX2ZJI1rTYRvRZoLiPG8ySdBWtKsh86B9gIRzalvZBoLglAqi1lwvpAybOyyROWKjBYH0XgBRAU0888J/4Gxc98MJT3ml1gpYFK2zslP+QmYtAC3RVn8oWPKXU71X9cznLr5TMmlActsHgVUugKUN/legwkrzwZzONPU7jXgaDL7HOVBwGxGiO6y4DhVMIcAvQIDMcEd93rQEZPcMAqooRdpJqbHeWo2/cUqi0QBUFOfy5Rr8mMCc97Wnvt03VOsMckZ14LegxAlInNYmUOXw9ESXmsOcyVLJlJk1bM40vB/cs1I9Ud35P9tfSG32NARKqh2kZUW24b2lSlsyMzpcYI1XvQRwG0V64wTe11QDBMRO8uWQ/TvZkwPDBo1slj6Vkz0limYGcAF8zshTUpFEpNAe2NT35bhuPokZLxzMLpg4dvCR6jyr5N+iaFC6hvA+ToYfacXMoe2aAEqp5AkX5Vj+rO8RptJZqLA8aHHaSNqG5H2FhD9x7VeoJbUBrLZGRWpeSwm/aa26UODgoMBSNzF9O9mhUmIaUTy8nDbDzc0jB1PPLaQwRU7yFHBzUYmM0EvzBQB65T+Ai1Z7BE1KZ4VaWLEINDPKphNgOEZ/BMtbbIwTJqMyVWsSLDnlKNWbEyFZ+nCBHRkXUUhxGxriBHyn2FC4Aj08kU5QBzm8IcU1K9mBxibZgO6yOEs8BhQqw003C9B6SA3E30VYbAACDrIHeBMtnUUZnlrLKXc2iP83x+ZPqqmCxltPdlryFARPo4xWQLCKQvUxBUptqT2HCSXPTjzJIe+pkl/KG6DwHLEB6tEPO2AHPaLPiZzN7PkgILFHbybZDJdNq3KkdSii1fIH9sciyAPxX9lfcf/sc6Bb/6j6+0FLb+xrvxl/7zfxcxCuxvWyAIyF1KHTshQPIdL1p9PCG8blJiW5Lu1GRYgolFzqUO9OwJJ+BXAdWVoiwtS5w0v9yytIUDffr63CKmqhJun+4p98oz2UCadW3Yx4hIORY7D9NrQVlcDqwJmjPxWTInvIA9Yo+l2dDPE3Us6y+BJYcUAhRESSv1FYN/IFGCbbJEyDexeJaylAiYfUF6T4bDLUJil7ie+lrg8A7DcqJJ+7YRJYhnPOXgNMu9slzK1yk0pyL7Iq0okiZfzwPxnDo5nXAW1awpbcoyHIDnM3u+csBM0CBTYWcvlq+zlC1tS/793v6GmozPcMafedY+y7KyZw1A6acDZhZDOJTwpuLpSp4t6d6edQdQ0klFRFmXTQmY42nA4rksgATgccl9lAS+lAPZhSgz+a6d9ydL2fRAT+niJVJy4Px+VlKghCBVa4b8VGsuwx7NrJPw93ooMXtL9f7eNeMpt8vPVxsCGLsQ5ZqSE1Ji5+wBzCFBMg2gQ42STOkTM+RbUbyRpeOvm+V53ZsUipQYCwY9kZ1xXeqqBAqroQYUCRdDKsg25eueryEQyV5PAMVLWdbREmAx5VMUCVuW4+UAFuEA37Kq5b50LEv3KHXjbH8eQKuRxzMY3ndyh+P9MKio0/nT9wJn8kPO13jU9IPlfQsVj0n+PGWmJ/cfSpvAah+LPDQfm/y6++FjeXt8heItzK/Jfkj6v0T5nNDbTZbDN0B7FYonM9c45fvvWxK+XMMjUxhUkpzmbslql+SNcWYaWdFE0Fdtk5dvyvs2B93ogcdoWsgStHP/mJZJmdSHGIUo94hcT5WlgBlQyOTDpO8TKWV6PpaseWKnYw4Iyt8b2Z+r+4hpRTCpE6Cm3xIpeVWkeygH53l/MyjlOYpFwqimOeQo11u5NvVAps9yfn1UTE+Vnh7Kcg/IXkWBEuyUq7NKwFHkPuk+hVMpPjctJbs1h8DJi/z5CrliaU7rZeJt+twIzBJpiTlUJaWxymkGD/naIigg81Q+8z6mChdVgGi+Zl2XQJ7nxIpwMbHfDAdSY8B4qmG2viSL5joiTkbMDEsO8sly8cyiqYmMovRULeXrL9epuFbB7FjRUgBRritR+T7B/9Pzm0KSUqDN/fAgEZKEc/L0Og6e11it3jouckwpqjJ3WlLKGWoFeILUkrLqA6KRlIC6wJqYBMb4nSHTOWFFjMzBOp2BHGwJ5mENikVoDNm0EBmYY31ZHw9kRGg09GaAX9UEeUXmSqAV6yQPzbUXpYsSiDWXWf4eqklswwAAIABJREFUCYIZDhRn5i4AIoSyDezQTL2UGZRpRVAlBOWdUtJ3KESpJRHWUWp6X5IqBISjp5LbzX3Ix3P+EIi3mcT8/5xIm1NUc0BQBql528I9+Snw9rK8fxtQ3g/Nyfuo1NvP/XAwUQ7+ya9JIT5xvFf1U5JmQ1mfuLeMeD+AKC/rH/SIobCf/yiPXxQp7L/51//ln/py/8tv/Tc/930DvuKMpVIBXz+7QYgCn+MMXT1htBq/fH6J27HDN371Gn/3xYf4C1/7Hv63L34Zv/pnP8XvfZfs5lE34OLiGM3jHYzyeOdog0+eP8HDB2vsxwrWK0yTQvvehLtXR/RPHTu888E1Lm9XcFYBmpHK06CxPD1gfL5COJ3wa197ie9cPEIIArY3iKMETISQEauTA/bfO8bqz1xje6hhbxssH+2xu1wAXuCDjy5wve8gf/cE8VsbVCpgd7mAWjj4jQFUhGwd5A8aHN7nTVodJOI3BkgZ6VOcJKwOqFuLsTfAdfKvOYnxqYX+vGENQBdSUFFMAwsCwuaDLfqXS4iTCfFVA7f0mB5GLB/t4foK7z++xvc+e4LF+QHDpODvGgwGqG8lzC9vMHy5QmgCpnc8FscD9nctMErUF5rR9b++xnjTwlwahA96+IOmD67zgBN49xtXePm9hww8ejwBo4K5UfDvD7CPFI7O9ui/fQJ75CEWDkJGVJ+2cF2EO/EQ4xyzH1r6zyApEZV7BfdLI/Cqhvlgj8NFh/aFQv+uoyfKsvIidAGAgntkARGh31TF5xQMpYrNlUT/OBQJYmw8fX6PRoStgbljxUf7WqJ/6inVi0A4coBN3ouDRDhy6L5flaTd4R0HOUh0zyX6J3niAZhOQpoASV6PlDDsvz5Af7+Bt5xosEesNxlPeTzUVmFQEbGK2NaUG/o2wh15tC94CxieeHq1eokoJcazALfgJEbzRsGuItOPIyCCQv+EkwmhjrBLToLk5Fg9CEwngfKzBHbcihUu6kAAJmtWlUxnHkff1ugfxZS4DPhTi+V3KkxHcziTGgQGzYmYYJBkraIEI/mGx2P9ywBERPdSYP/unLgctMB4HtBcyDlEJU16ADyn3WtgPOP/x44+t/5xhEhgKRimHasxe6FESRfOQOrwhBM6qy8IXG/+jMPyewa7bzgsf6AJeOMckFOCoAaBwztA94qDb9cKTEdcdjyZJ2T6R/ysuhSKc3jKdN1gyDQ310ifab7HriKaS1FAYV43JY9kWg+Pma7sUgKw2RH0Z7CPwOPnG8AuI9oLUf5ujzipMJ5xO1ybKkGWnARpbiM2HxIkZTYWCYS1V3N9R/8QWH3BwfL1rynIkWFPvk0MoEh9wanCIS7S8c9dwSFN7iRgOR2LxBKhhLcAgOnntOLpWKC+iTg8oWyTYTRk1PZPJeW1EWjuPHZPWZ8xnoo5jCdNwrl2DoWaTgS6V1zftBKYTsiOD2dpgkimiYqJ+5aTJkWIqHYRzZ3H+kOD7PHKAUTTMc+L6siK+gYAyMaffnfC+uuUejP9OR8TAbvkdojIiRlpAX86y16jkqg2EYdHlKX6ilLc+o5es8NjsrPmwHAeNUUcziXqu1Qlsk0TJ9U8WSM8z/u0Sv6xNPE3rSj/rXYBtiPTrwfKSHVKAB7OJOq7wAmbdL/N4Dj3EOdeZukAu6BCIANsk/qWi69XogQ5lYkMoFwX0gu4hSyJvTkxN4PS/pyAs956bD4wOPqcXbf7xwrLl45px6e6pDPn2pH+TBefYa5MUVZhOBFob7h/qmf/bU4UFg6ICVjapUFzZaFG9pZOR4osqxJlUkv3HtORhtl5TMcaqmfSrV1pSBuTzJUMYEigNQpANAz9k0amXln6DEOu/ehYHyJ9hG8UU3rHCHNwsMcGZifgDb+/fGvQXA7YfKND99oytdfG9D0pUa0tpqWm17h38JoMY6gUO14T+A1GwiQ/ZGgV9GEGDvakSaDUI1YEafZBB/iI2GrIiUBfuAh1mDA8aFDdceZWDg5+YSAi+1EzUPSLuvTpZs9j9k9mUK42A8RgEboa0Uj4sw56OxLMAvRJDrRwyP2A2JDljkYx2FYIxBAQli0nNaxn/UhvEZKcVISAsNDFExmFIFgFCATrFOufQKJwYMjQ7oBwvIBc7/kZ0wqxriDCSJBamdk/mbtDhwmxqQhePftvYTTQD4gelK3eA3pCpwRaaXDfSylyWFGtZoDnHKAJIsVigTgMlMZ6z05N5ziXk4OAEugUMc+o83HfB/1jH+FP8qJf7Afnof70MpZf6TPkrYILEh9/+RSHZytcvjzB4Tsn+PvP38UPXj7A3/o/fxX72xa/9dk34b67wieXjwEvMOwrXHzvHOZ5hf22webTU3zy7AnabsL1330C/7+fwX58BHvXYP9Hp5AHCbzfAwJ4+eIMIQrEvUZ43cDdVRBbjfE7x4gmImwM/uDjDzBct8CnCxz93zUQBRbfrRCthP+dUwgvcP35KaY3HcQosbtrodYa1ZXCF88fYP9yhf6bI4Z9heHTY5gbDfNpC3OngCCgP2+gBoHuCw21V/DLgOaPWuBZi/b/aYFeQb1oMPYGYWvQvpKoX1SoLxROfruBPQnQewG9S9KUgaym61j70W8bRBPRfNyifSNw9KlG/drA/f4JzHc7fO+zJ1h+1+CwrYHPFhCtQ3MhMZ57uG8fQe8F6kumDDgnoS8MqiuOqA8fONR/8whyo2FPPJq/36F6ZTiAuNY4/kTj4vcfY/Elw0WO/qBG96VGeyGAqxrLTypsXq+weMbk3bO/XUO8qQnsTETzUmP5hYTe05e2+q7G6jP2cy6+b8gUv2wgvKAH8wcKeg+YO4X2hYLZCjRXEme/r9DcCIi9AgaF+k4UNqa54g1BDYDeSbSv6emTW432QmLx9+hT7F4J1DcElWYrcfpHAvW1hDBkws2dQvdaAl7g8DXHAdgINK80jj6V9OL1AkefkfnrXioy6McWzaWE2bKLMlzVJQm3fQOc/iErX7rXAmonYXYCR9+TqK/IzksLLJ4LHH1CoG92QPtcobpRqG8lqjVQX9ND9+DvsX/TLwLaVwrH31YYzyOWn/N1CEDsPKo7geNPgVBFdC8jYmLa9Y7L6V5IPPz9gHrNQWZzRdmkHGSRui2eMZVXXxm0F7Gkx5oNU4HVBLRveK22F5SFuhOHo88DutcR03HE8afA8XfI7Jx8h4y6WQs0V3PwjOoZ9KL3BAEE6EB3GWA27EbsXgL7dwmkl8+Bs4+ZhHry3ZBqewjM6lsg114AQHPNfSD4Bc5/lx5PtZfwhoDersh8TMcR539oMT518DVw8m1g/17A9uuUSa6+jLzG9tz3DJRzuitAsLZ4HbB4ScDXXEb6AhvuV3VHmWl9F5Nfj167UAPdS4HDY1YHAcD5J44Sz8hQleE84uR7HtNxhBoi2jcMnJE24uhzV5hqaSO61xGHpxHtVYTZcRtyzUtm0ruLgPouorsIhWHLLFFzJVCvA9QY8fTvTOx93ERUd7F4+6ZjlOVS7gi0NwQB4wnlo66jN669iFi9cMnPR1WA7sma5wqS9jJieCjw+HcPpbKn2sQi4623AYdHAjffNASuFbejWnP/sgQ3+/XojUvKikNAdxmgBmD/lLUy7XVAexFmqSt4zYUKWL3wsAuB8Ujh+DMLaSMWrz19Y5uAo88pT7UrkZJeeX0df2bp5bsMmFLSrbTA2XcGsrKbiPYy4uxjW3yfD/5gj3rNa7vaRAynIlXscFJj+cIX1YO0QHMX0L2xZb3TCmiuPboLD28oUdY9wWdWwEjHc2QXBFJHn/VYvvKotpTSTkcEnPtHlNSbfcB4wr8tXlnokXJVc2CnY5bStpcWZh+QfZDLFxbVlizN8vmY7qF8b7XlddZee+g+wOw8qq1Hc20RpWDCcU9ZcbUNlJtKpuQuP99jPJJF2XF4qLB86bF7R0HvPY5/MMFsHQNuxojqzqG5stApYCYmsJjZ1dXzEfWdQ3MbsPziALNx2D8lKIQQqK8mTCsJ3bsyuB5PDfbvGJidQ3tl4VqJamPRvtoT9D+qYHYe/QOD6s4xnfqBQZYGSxthNhP0gd5N3TtELSAnVuzIKaDaWKoltpaTgEbAtRJuoWCXmqxzH4oUvrkYoDYTQiXhG4XqdsLhnRbthYVZD4XVV1OAtAGuU6ivB1ao3PYEbCNTdc3dyAmnywPVIi7CdayPEYEdk/aogrk5wNwcYI+YwhqM5LK0TOyqgL4boTcDosmJwIllTeymHCzErocYyVzq6x1kYkP9wiAqidAogslUSRKlRFg1EIFMc/X8lp2UlYYYbFFfRCMRFk0BnHI3Ug4rBGJTQ12tyUBXGupmR0A7jASb/QjRT5C7A6WxuwNlrEJAWAex2fM1u6Tpt45S2cpA7oYk005JtPfVh6lmROz7OQhIScpl4/z/eE8K+xZ7CMyM76FHHCcIJWdQCSDu9wDIQMbJkqUNEXEYyvJEVSFOE5eVUmjfDhNKjGpOwU2v+XH//vQ8BEKUP/V/vyiPr7QU9uibj+Of/6/+El7vVohRYHQKh0ON1WKACxIupZf2hwpSsoTWHzTgJFMnR8UEyoWFlAF2W2NxfiDDBkDIiOgFvY0RkOcjjPEY9xXioKCPJrheA1YygbJXnJHuHMJdhWgC5EEhNGlGpkrTsF5AjApRRUBGLl9F+h29gJwkoowp+ZLy3gychBXlZ32lMJ4F1ogsGdjiHlioO01pnObrwsJzuaNEaAKqa0VwuZGl25J9lhGxTUbvXIOy5brlKOAe2rTtkgExW5bX+0WAmATUKGAfOOgbDd+m6WEdycq1ATKlKYaGUuPqlt2VIQWOTOdMmhRWpOoKDjxzx2iO7A9VZNfmvQRHNZApioLpkmbLGXGRWCURgOmxI5vZczbenTpUbzTcKs7sWs19DVUsvaX22MPcqTkVMvnE1DTLjIUHt38UCG1AdcNrxq4iulcSwzlTGn1m3FKKZn6/2TCxUu/J7vk6pYVOAu4ooL5kuIq0lDn7NsItIsxaktEb+cXPVEXKkVkNQi+hbzjIUYMonqz+HY/2hUrsBxKLwWAbe5Sk2CrSO2qzpJZhI1memxMw5cTO1SxppnyV71U9j6c9CWifM7xG7ylRDRVl4Zy1JxORQ17uy8pjYvrUkMJZkHx8Kskw03dOlh3nBEomeqZzpriPb7FbCfRkOXDuA9UpxCXLbEWY5cdZ2nm/m5SAjwAm96NmX+l9GaUak/xbcFA/nsw1GCFdW9Kl5MiJ7JbrRJLQpvetk091QOmpzceIPtT5mKgkdc6MbpbKqzGif8jPWXvJDshQc3ulT9uSZJJRJWm64vHQ+7QvuUNT8u/tVeqARNqO5H/zjSgy2xl4JHn0SiQ2MlWd1JTNq4nXv2tTEmeqXIm5izIkuWVFaW/u7tQHYDhLaawHhuOU9FvNYx5Mvm5SYuW968p1ZPxyumnuds1dpLmzMqZ/99Mus6xZuliuzWwryK/LstR8PQQzV/5k1jt/fu5PaGcprMqSfoEiua52rFOpN5QQl6TPThSZee7LzdeZ9PM20T7AyZ0M+OpNTIE8osh39Zgk2ILnwOxDSSzNn618T86dnjmdNGjed0yfZbjJ+928LQPPzGNOOs3HIae25mCYbEvIPZX88HPQmiW4pd9ToqSlAij+xdz9GaUoSar1hifG17IwiTlwpoTSOLKB1dbDtTIB8lCWGUxKLq0lgaYWiQlF8ZTaBetKyEgq6EMoUmU1eLiFSvcLyh7tkoylT5LWfA8CUIJpMqAP9xjILF2UNhRfdJYFqyRnlj7CV5KS22oenMopyVm1ZC9oku7G+1JGTaDIBQO+JduZnxM2wK7MDPYm+h3dwkBmqXBMaamW4ChXklD+Lsp5zfvL+3ySyyaJbK4zkdbDHtUw24nHxQYmpRZZrCwpt/l4RCkge4fQUUIrbAAUa07YJ8maD+HolRbWl/RVSotndjVvj7C++CxjlQJ3hECoDeQwzb7KUhtyT1aaei3fksLmY56ZvmFC7BqI+4mswLwc4EdLYHMfZl52lvHewwFFsiolYgj8aS1K5Uj2SOZakrxdWZL9wzLVEGapcAaS/zBJsD9CIvtVT4V9+CsP4r/xG//qT325f/XXf+Pnvm/AVxxY1h+8F7/+l/9DDI885CRKimaOdPctZ8LtChieOHRfaIQKZYA+vWvRfr+CXXEwnQvpfUNJYnvJgZ9bcuCsU/deHtBluY6v7vn28uc/DSpj+qKNgl+8rksgIEn09AGlVD5KeuFCGgi2lxww+QqwxxH1FUGSXXEfmqskiYqpZ+949q+VQWSaac8Dl8y6MdkRc1y8yv42IPsC7ZLbWyUvZ5aXlt65Oy5jOuYgISqCQSZLppMUuS9mlzoMFc/L4jnlaFGShRhPGdzRPxRlncC9QZYiG2BXHPiNpyJJOlFm6aOgHHD5DOgfijLQtYt5wC4tz71dMujj8ITbUN+mxEPDnkQkH5+vE7NxLpL/LknSRl4barjn40xAw7VA95rP1+uYuueA8QSo15SkDeez77QMwNt0rldkorrLgMMjCbPjYMs3GVzNaZI5BIRMTIRbMLq/uY7Fb7f5iBJLCP59WiV5muXvvGY4IMrbe9/DVO1i2S9glqDl6yf70YLhQEANsYAFPaauQo/UETl7wXIgShSUyI1nBAlmz/3NAJMgNfXa+dmjC6RKgSR5zJ8j4QkQdB+LPFf4+fxQGpv8qs28Dt1TDuqbFFzT85jkgXyeUCjexykWT5OvKa3MYThB8/wNJ7LI8HzqWtRjQH+milyv2qXwk83sq3SNKP2RamTi5eqZLwmTGehkX58IQHPLQJEMAqLgclxLaSLEPJDnIBGo9gHTgt4qn4Oe2pTuaXke6k2gXDH1DUbJbWtuA2wriwdOeuDwQKLekJXLnYM5sMN2DERCBgrqXihOkujm+4yy7IBs1kzVtK1As6ZUMyhKHqWNGI8VzJ7SSpV8q64mu0WfIUoXYq4niIpAwC4oi/aG58BXInXHcvm2lekeOgOXXAUgU8ovgNIfe9+jF4x4y+9KBkame43DcKZh+lCAgq+47ObWoz/XJQQney+FB/QYoHce/SODakNGyK4U9D7AN/T8FT+gJzCwRwrCAdNKorkhYzUt51oHkWorXMcuzaiA8ViV3krdE1TpgcBJpeNujzhxVryEaf+L1LTm92m1tRhPKPtjQiv9j9InWWaSMvuGXsL8ez6GMn3GzJ4+Q19R0qn3fK3rVGGOfM0ApFxTkQGea9kPmcGWsvOg1tcKavQImstWI4+l8BH6wLCWXBmBCKjBFeBjV5QIms0EtzRA8mgjXSP5+mAIjiw+R9co6MFDr0fY06YAqBxcU4BUHqgn2Sl7Ig0DZUayam5hiidShAg5pB5GlWStkvJStR3hjxv6GGtdgIrwrMQooC1SGpol2tASYrSIlUaodfEnQgrIA6WlCEBoNCAIHHOoTf6sRC0hDxP8oi4eQrUbEbqKoF5LqMTyFTBXG8pDEzCDEKVjkumrTGLNn+ccqBOVgBgsYldDHEZKRDV9j1FLhucACK3h7zF5ObO/MgOvzMwVj2BEbAxEP7NvhXED6JmsK3oqa1PkrCLQO0lgmgCcTP7U0dILOYxAU5PxS7LW3IHJGpMErGIEnCMTmGWq92Slb3kg87ZVhstNz0Xn56TWGGYfJcD/3weLeTn3jwPwNshL3srZr3kfaCYwH2OSxSbQ+8N1I8AfX+9PeMQQgX+MHstfFGD5r//Gv/ZTX+5f+/X/+ue+b8BXXAoLL9B/zUJagWrNQYLZkTXwHZmQ/lGEXQUsPtewR5xJbq4F6jvKU6Xn/+1RACQH9PU1ExR379Mv1FyK5O/hQG54GAtoK2mUSMzZvXS+agu0bziLXm0on2uuCEb0wL+JAByecrvMjgNbNXFw1T+Ykw8Xz0QJfJEWWH3OxMHmOiagw3U2V4kB21M6lr1Aquf6utd8fX3Lkvb6NkLv5hAaleLZXcftX33OAXh7mdiynvtvDkD/aN7v9oIdcmbHWPwcSjMd86dwXJeagNUPkpyvByD4O8N1COb1gdJFyuyA+oaSuOwfGs4Futdphrvn+jOI6N4QKC1exgJGmut5tnM65qBYOvqH2gv6zHwNDOeM4l+8opdBeJ6L4QG9RLqnNLG54Q2wuaLcyyRZGzvLeC53X2N65XicCro1cPQ5B36Hx3NQUQao7FqjTLBac90X30pgys4MjtnxC9/sOIhur0ICAkxWXLwMWLxKoSeBx+j4uwSUruH1Va8DzJ5pod2bgPaKUic1URaX96/eBHRXAXbBc5Q9ONKyv23xmjKy7Qfz34JiKIpdCOyfSuyeqgLGmjvKA3UPLF956J4gfvHGw3U8z2S/CCKCESlcSBRJI0KqArgO0IfIkJd0zMdTSgLrdUR7GVBtYpkMqfY8F9Uuor4LcB0BY33Ha1KNBBfdVUD3hhK6/oGkZ3AbcfQl5armwGvbtQJ2MXfERQmYJKMkQCAYU0n2t39CX+y0FNi8rwuT2txRXhe0oJduKTCcEhgtXvsE0ihN3b2jKAnsA5q7AGUj6m1Ad+lg9oFVBTYSBKjEnvmI9obXyrQSKanVJ7BLVgsAEIHlSwdlI9rrgObOY1oILF5ZArxbhnhMK5lqJnwKoUnpnQoYVwLd5f/H3pv8WpLl932fM0XEHd+cLzNr7uqq6pYoUxQpWrKpBReGBQPyAPtfMLww4L/AS+8M/wGGJ8LwwtDOgOGtAcMwQEiERVHu7qru6soacn7DnWM6gxe/E3EzuylRBmiQRegCDy/z3XsjTgzvvvM93ylQruIIpqWKQnG4Esal2EcmN7KtyU1Pt5A/QZL2GlBBft4uNbOXfgRqRV5cKdchgy55T3WffWcpUWwC0cDi245oFdNXfpTcDp9TLlcltCeGciXjmL3oCaWMw1dy7qNRTF73+fhi9stFVEwCwBTZH8QIOHUG4igoVp7Z05ZoFO2Jpp8JaJHfd8v0VY8vNeVKxjh9IVLTUGmWX9WQZEFBpeP+o1W0506YxYWmPbOU9x7TxRFUdidWjiUl6gcOuwty/2+FGQOYvuwkLMcIKPRTPbL2uk8ZtMr++5kwc+Vtm+soNIdrR/W6o1z1+fpIeqmwTYpYKIqVJxaK+qrA1pFyJVLL8q7H1mEcS3vu8HOTqyQszYXD7j2mjVQ3HbYOmEZSTomJYttjd3K9+oXFtBG79yLrvO+wjXywuk0rYwJsEwT8KvElAnRLRz+zAlBzAI7N6ae2DrhNT7+0dAs3hmIlDd1pgV21+LmjvG0yMNG4TUex7nCb9sjEqQG0O/nczGMr1lIR0VxPMbVH9xF31xAmFlP3xwAXpegXDnsraXFh5rD7HntfQ0yEqUV3Ic8JpAbDL0vi1I1AymxbdOPpHszEWzgvMAepDjHbRsCVUehDPx5LmBbEyhKWuWprXo7AQrcec7eT/c/LUQqqO4+92cmiV+UEjA4sYuOJlcNuGsy6Rh96/OlEZKVWY19vCbMSfehkv7MSvWsEUGuN3tToTT2G7WAUZnUQgGmPMkq8nIN4Ms2g68geqrrD3GwE8HQ9en0YASdKEQsrgLDtxgAe1Yq8U7UCRvW2FimpMSNTqNoMBrV+62cSBhRFutpJUm5yVsaqlCSzDv7HSSXgb2AwfRBQGQL0Ii/FZ5bRGJTLdR/WCpOo1Fvf3+yVTLu97NNa0pAU+9YcOsi2vZd9Dv8eWMeB0RzGk5NuR4CodAan+Z4deioHgKv126AyCih+8wtj5DljZHt/xte/sJLkXz3+Ujy+14zl4rOH6ff/2/+I68mG23bGwrVcFjue1qfsfMm2K7nZzfjg7J59Lx+S19MthfY0wWFV5OALmmB5cnPO3//4p/wv//Q3WZwdcCYwK3qcCXw4v6MOjj9+8ZhZ1bFvCj65vOGL11f8zUdPWXUTfvHykpN5w8V0z6vdnN988Ix/dvOIwnrmrsMnzdO7E4I3fHh9y8x2NMFyV0+ZFR19MNzvJ3x8eUsXDMui4Y+fvsPj8zVdMKwPE6Zlx9T1PL05pax6Tqc1zgQmtufnzx9wujywq0u0jpQur5ipxO5QURSeWdlRd47eG2ZVR0xwPq059A6nI33UdF7CXCau53Y3xZpI0zpO5jUhKu5eLnGzHm0ixkS0Trx3uuLFdsGyavnmi2uuP77hbjPj4wc3/Oybh5AUJ2d7TiYNjbe8/PYMPfOcLA/c3yxYnu+ZFD03q7lIkmuHqzy+NyidmEw7utYSvMa4QPAG6wLzacNqPcO6wLsXK17vZuzWE1JtuX7/js4b1usp2iSWiwN3r5dS6KgTahJIjeH0esvq2RI970lRMZm39L2h35S4ZSsS6qSwNtB3ln5bcPZwQ9M5VEar3htC0ESvSQeLuzPwgz2hN3Bb4h7v6VuLGhdBE37vmF/u2a8noBPTecthV1JOesKXc/wiok46Yq/lM1on0sGiZz1F5Wm2JebWERYB3WjSWYervEi/7ybMr/bi3S0ixga6VYnZGnjUjuPWv5zgP2iIXqPXjnTekVqRZeuNRT1oCOsiS7iBXqEaw/yDNfUXp/gzCY9SvTBezPw4TlXlCc+tGxOa7V7kzmmaw4u8QnkFpz2pNahOy4r0JKBaA/Oe1MjPk5FIfgC17Ig7CTihiOiNJeYx6ia/NqijfHzYDxAneVwHYVxQEBcee+eILhEnIimHPNHuFH4ZMDtDOOvRWyudmnuNP/eQK17sxhByGFYqkvh49joHiGQWq4qoXPGja0OcBYoXlvBxDc8riKK28OciZ0dBmEXKVxICNaRR+4UwRGkSUbVGd1lCV0v4lsn/D5V4Xc1OGAm/iJhDBlDnPdNfFvhponunw9y5McjLbTT9IpKs9F6GKo2MlPaKUCQWTzT1dRrDWnSraC8Dxb3Bz47BTt2pXBdTa+xeqntMI+NEIT7vrKoYtlNs4XCdRln6IHXvlxEPKhOmAAAgAElEQVTdKdxWpNrSUyjycgnHkNdJKrFs3+0U/WxIdZVgLp1Dr+xBApzKOwmBSgYmL5QoNeZyvpOC7ixQ3hncNqfzFhLslAxj+vDAhLudLBh0pyJrV7nUfii4N60stg1qlWjJHlbprDwmFL+hMslhU0PgzpjUXct5063sU7ors9f6lSgPhrTe8j5xeCzjmb6URZY35aTRMfZ46h7qa1molEU7RrnzoNqxewluGoOshsTfeFxQlGoTkXIPC1nt2TFxedjv5EYWOEwr4UfFSsBqzMc+pLcOyazRqZw2LZ9jEkDzhox2UJD4LIOvM8uck521T3lR6I36nKxgkEXHRHWfOFwKO9oth45aWfwDWdTtc+XNcB7dflhIPKp0+pmwupNbkQ0PCcWTu8DhyrytLLJHebmr5XwN7yk3mWHvjym+ScuCQXMmVSihOEqPu5n0u5pW7ut+Kh7msWYjVxYNPucxRTwxSmsHaTdJEogHOeogww6lVM0M25GQJAEYvpIFiX5mKNaeUIlkeEjKNp1IhU0Tx0WdY2BVfq7NbPle2Pkwkc/sIQ3ctHEM5TJ1lK7MJIsH/cxmyStS2VLI/t3O46eyAGcaP7LOuovCBL8RBBQLYbGHxQqz70UWXJqjiiEi9Ew4nhudvZaDZFzXnjiRRQB96AjLCjK7nJwR36cZdNrqbda6FdaciLC3zhwZ2ZxWi1LH8J/coympuJk1HBJih/oRgCiLZHT9sYbkVxNZB2ZxSJ6FI3MK4rssnPgrtTqC35SOoDNF0hBGBL8mg32T2ZT//hl4JCfQft9TYS9/fJn+wf/4D/7ct/sHv/sHf+HHBt9zYLn87Dp98F/+J0zLjuvpDp80tXes6oqUFB+e3vGzV9fU64oP3rvh6+8uefz4DqcjMSlOq5qfPbtmPmtYrWa40nN9umVie2rv+O7lGbNFw+V8TxcMdefYHUp8aymnPdOqpQ8G7w2l8+zrgqvTHc+en2HKwNXZFq0Sr1dzHpzuePr8jHLW4VzgsCtRJlGWPYebKZOLGt8b+oPDTDzOBbrWMpnK6lPpera7Ca7wHF7NmD/c4b1hWrXsDpUsdDWOciYgrKx6wucL/AeSFqt0ZDbpcDaw2kwpq579tkJb0S+FnQWTOL3asXq6RAVF9WiPMZEQNCFoSbg9GOzGwPs16skE/26Lfl1w8ukddy9OUC5iqx5jEl1rMd9UuK2i/RtiPg87R3na4L+dEa860t5mf6qApgeXG15+cYV7eCD47DOxgRA06WVF8e6e/usZ5r0D/cvJmPQaTwXkoGB2VnPYlqS9RS9kpTzdlqSFp3ju6E+iVM3cW+xHO4LX+FcTmSAsPMW8o389IU0D9sZJDcxVS+wMZmVRjxrCfUmqAvQas+zQTybCaLtE0onJC8Phhx32xuHnAdVr1Lmkxc6+sbTnAj78aWDyraW9jJJgGxS6UYRJ9sKeR5j36FflmN5rGkV3EShuDd3jjuJpQX8qQEBNPCkq7OuCMI2SrHsawEZmPy/oThL9hRgJ3b2hv+rBJMpvZPt2r6g/7HCvnbxPJybfCohrzyKmU/hJZqMm0u9ZbBRNHv/0G8vhw575LxzR5rCa+RuTF5XP0SSgt5bpd5rDb9aYpxV+HiUEqVf0n9S4Lya0V4HytSFU4jtNLlG9khRf3WiqV5rmgTBIg5c1FuJX1u1QzC7AYfZUs38/gILpN4Z+Lr5Pu5MJSX8exN970DkwRuTvm08CySWm31iaq0h5pyk2MsmvH8ofxfnXmsMjkfdVd4rDw0j1WtOfJOxO+iHra1FEgMidTQeHh5HpC/EvhUqeD6X0baIkAGnwJc6/FeakuYR+IUFb0Sb6s4iuNadfwObjDK46xklsyNerWMvkZfd+pFhJEur0RaK5lEl8eyGAsLyH/buJ+deK9WeBxZdGlAfpWOnxZq1PmMjkvbqTdNEwyb5ZK9JvSYpN7N9BkpGRSfzhcWT+tR69rs2FBBAd3knMvpVUX9PKuZOu1sT+XSjvRN5rD9A8kH2VdwKcqtcScjR5raivBDQOIVBJiVQaBdPnWU7fSdLuIBUv76N0kG4Th0eKySs5j0P1je5FYWAPR0m3pG1mIJTl7e0ZJJNE8ZH9gc2FSKWLTWL9iaa8k9dNXokkf/pCJsz1Azj7WWT/SO5fAcgC7nUQkFCtIirA5kN99GUqAWlJi1S9XIm1wO1Equ32osxoTyUFePoy0p6I/N1XAjTCREBOeZ/G61ZfCXNfrrL6ZhdZ/0Ck3LoTFr89zYmvGagdHuoctCTMfXdy7B8FmL0IHC619H8GaE4Vs5cCjg4P9AhUVPaQJq1GoJWMnNNiK+B58LL6qSgbpP83v3cr4EJ8h0d7gsm1Jd0sM+bZGjIkzB4eaObPgkhxcz1TN5MOy8MDzexlYH9tOP1FR33lRpA5+EXdQeTZKCg2kX4uybPlOozpttWdp7lwwjQvNNNXnsMDm1UHIvGsbnt27xSUaxlLe2ayjD1R3ntJpO2PvlO3y75Po7AHYeaTgcWTmu6swNRRejdvepoLCeeZvuw5XDsmryWFdvBZai+JtG7n6eeismjOjChlXvVj2qw9yGfq/pFjchMEnA0y5KmAQdsEmnM3VgRNnzW05yWTlzX796ZMnje0lyXlTUsyin7pcJueWBr8VBQd1V2Hn1jcpqM7K3AbkUeb2kvCrBMw6tYNYeKkSzOHFQF0J5bytiNUwpDZ3ZFVHYJ//Ekp/u0+ECo71r7owUea5b2xNJhdJ8mwrfRKRqsxe5H46rofWWMAfRBfZarEx5lKkdWmSSHfKyeAcVJkYJZGsDn4NfWhIU4rCf4pCwGFZSEsqrMS6jMRj4iqczrspEQ1XZYXi1R3BIwDkHS5HGLwXA6PpmUM1ul7GJjSvn9b3tp3kl7rf33bY2flsA2Tk2T/RbUif4Yk9q+CFPavOrD8XteNhKRpOzmEta2oe8fE9XTe0tQFPw9XAqpc5PV2Bgl2TYkPwgK9Ws8BODQSxpMKz/PbE0DAjNKJw77kaSv7SFETOkNqNa2CENQIfrrOEjvD69UcoiIFxeu7JdoE+trxIi4hKtp9QRsVdBpMou41qtfyfq9RB0O0kWbnUL1mv7fgEgdbklpDb4RZ2N1PUTrRNZbYWMzUk3pNsykhKhqvYZ7Aa7yXSdO6yR8MKtHv5d+h03nFTUFQrO9n6FaYlnotLApBgY3g5bUK6FcF1kI6WKKF1WqGajVqZ+hnmj5Xd0QHzYNI3BTCYgVFezeRft9eCxtVKVSniNHxMp5ga0V3W8mHeK8IpVSGaK9obifYqOjuK0ynCFa8teogQUzawz5NoJPzmu4LYZg6RdwZ8ZI2itQb3EZR30iCq2ny6uKtoz8YYWq8rPArpYh3JTozJ92qEPbIW1kFjwWuE6ohOGFnANReJgG61cLSqALb6DExNBmwazOyFPqgs08YdCf3lW4U1CLZtith40yrJIDIK2EaNZh9niDVWsZZK9w2x8q3Fj879s6ZjUwSSGDWdmRURnnu2kq40K0AumE1vdhooknYKKv9IWhsLXJVYc3k5rBrYdsgr0AfZNVbeuSESfSdkp7QBOpVierBrfTYkcrzUiat9zqDJAmHikaClszWjL5Z0yghH6PsK3qI1uD2Ku8fubYduLWwoqYBlRTJSGouQCyE/XM79ZaXurwxhKlIjd1W4/Y5aCVIvytk9mCV+ypb6bBFSe/qIMcdrvkgb49WtqezH3nspvRQbIRlHXpctf+V59e5IsMr1J0kIPczsNuBWcnfewi9GuXIuodiJXURSYs03O0h1ZC0lrTZBMW9XDO3kUmw9gKyVQITIM7k+E0OHjIduRpCwKPu8z3ls389yLUZgJJuoXqppR8zZPZom0HnXvyZ5UrJtWoTMbO5bpu3Vct5cBu5/+Q52bbbSsqp22ZQVcn2ANgLANBZ5u52aeyPVNmX53Yp13MwVjSY3F2rvVgElB98loPcWu6PaEF1It8eakSG3zt7yAssRlHeCsOVjDCVxTrfb5WA4aTz8+rIaA1fRa4TiS7X0Riyb5HRoztIiO1BgJY7ZC9YHBKLj53HcNym6WUcOuQKqiAdsLoXMK2zJ9JtRUKuPdmPKAsNZLbLHlLuuhTmrNhK9UZSjH2exU4AlHhu1cjY2SaNHk+Q6x8KcO0bxxdE8u1LMypBdCcdoHLswjbq7CEdQPnAcgrTKK8nJWI/pAXLOMvMmo5BYl3CKZHMDnaEYivS5cH+MLCXthZFwVA3QpZJD9dvGONwnVQcEnUjbp9l1+2gBhCQKj2oIsMeak2S1SKTbgTAKS/9kUYrUW34iG2HACM1eubdPkhKa67e0SGOoUVybrK/MuT+UKWw+0AsRBIfqiEQL6FSRGefZrmO2Nrnz0Odr13ut7Ti4QUwjQAYYS1ziqtCQn+G+yfLknUbMEa896qPGCWhPvYQRGYLqBAxvdgydCshMboTFlC1RwDjdlnqnKtMdB+FcOx89opGVB9RXr4b5Nh1ZsfGaqM+jBJfYeBA9f2oFNG7TlhVo1BNFICYeyZVSMIAZiZR9eHIMKYkIT8jqyjVX/iAMpkx7kRmm9pOGMa2E3/nEJAzyGAHkNj18vOBvRy8m0a//Rrk3n/LSzkwjwiTOPSnpjw+6XvNvsyY5EtpqRZR6m32cQjvCUGee9OE96sgU2l+7fEG2FRa/cvaMf9SP/4ypbj+eT++10eWEuh/umB/KCl0kA7HqNlvhbHs+6wDWjv+zjtf41473jlZM6s6zmcHPr1+LWmy9xX6YPA3E85O9vzeD77kd9/7WmosguIHD2756OoOYwMpKopXltmiITyfMp21nCwPnCwOpNpwcbJn/rOC+aIhesXvvPctpgw8vljjXjqur9e4aQ9VQJWBk9MDZq+ZTDoWyxqWOYUmCtiaXR24ul7z6bsvKU8a3KQnzT2mDFxcbpnOW957/4brizWYxPz8ACZhS8/sW41aOXQVUC7y7/6NP4atsK1ouH68olh0FLMOikhx0XB9vZLJZK2YndX81mdP+Nf/xi/44P0b6e2ceMo7hV70UrthEuf/VPFbH34rqbYJAZBlQJ93lPeKky8U5XnN7//oc3St+fDjlzJRmXhSGQWIT0XS+Xd++BX9SWDxeMvses+nv/Ed7qTl8uM7UInLd1coD599+pRYJK4+uCe5xOTRDs46wonnBx++wp21YOUDa/loKz66y1YyaSbS6Th9mbh6/575+xuSSfgTj18Efu93fkp/EohLj2kVk1cK+6AmnvbYnUIve5JLPP7sFe6DPdcf3tHPhc0r7oX9KjZQXB9IVuSVScP7P3pJerfOnldhHcOjlupWJh/JJuwHO+bfQX/h6R56VK9YfnpPqNKYzBmKhHl0AJVwj/cUa4U/Cfgzj3rQCqNaJdxv30tQ0TISH3QsvhXvYDj1Iu/sFPpxjX10oLoFlPRZzj5e47YQPmpI1y3daWL2XADdIDHrzyLpYUvSwnzEKnH1oxuKFVSfrLE7OQfJymTbzxL9MtKdBbqzQPXBVhJzX0TKD7foXhF/UI8AbPHJStjOs4ifJcI00V4Ewse1AMXHDd2l1DFMfrTCn3nK20R3FuivesofbGg+aOlOo3ToPeyZvYj077Ysf+tWJi97iB80wkg5SNctfhFpzxLdUu7l0y8DzYcd8f0aP4H2YU90cPrLIEE300T7yI8MpK9EItgvI/U7IXecMibIJgPzZzGHq0B77UcvqMm1GipAcynvbx572vOEn4mf1u2FAW4vA/0i4ReJ/qOG+l2PrRPtZaQ7STlkKrF/V17vpyKt65bQvtPRnooEs9gmuqVI9pp3evwssf0o0v5GTbmJdOfie2zPI80DqdroZ3D4qKefC5PXXKScsiuT18P7nlDB7mMJI/GVBPO0Z4nuNHF4HOmX0P5GjZ8q2jPpwdz9uKW6jzQPAyrC7kPP4bHUaJhWxr/7gc/nQI7l8DhK3+c+sfswYOpEe57YvQ/1jxuaS+mALDYClLY/FD/v4Lvslor9h37se7SNeLhJwhpHp8aKl36RGc4r8aQPATMDoDhcCziJTmpj9u+K17VcRSY3sg9JO05sPpVjbC6FGWyu5DOgPVUcHgvgOTyUQKhQSh+lpM2KR3T2rMPtE82FMK7D+IcOwcNj2U+/PMphfSWgsr4S5jApCVDTXvyx9ZWiPRE2uJ+KNxSkm3X3QZKxOEW58vQLqarxOVDs8EgqX4bApsNDqa2o7j1+Avt3JHSpX8o4TS3BZOW9p1jJa9xevKLNhXiNd+8IyxutdJI2Z4r2XH7WT1UO6QI/k/N0eCSsa7GNbD9QtKeafir9l36aPZILOcYBbB2uNM2ZxleK9UfHjtvtB5ryvkd3EVuLj1eqRzS792ThdfOBpnzdyNguNM2ZjLs90ZIXcNPl8C/5inkZf/fYcLgyuE0ni4KteNvRMub6wuTgLVmokzEa8ZXONIcHhm6ucetOWFGQfU70mNjanBlsLXLbbm4we+mYlPAgYen6maZbaFQXaU4N0WlUiNhtR7KK9qKgz8Feg5ezOZMOz5QTbFVIRCOKi8OlVJTUVwV+ZjCHni77VHUnHvBQKtpTi9l3tCcGU/fU5xZde7qT7BlNifrSESorfZdTTf3AjWya7jx+YkY2cwhE8hMjPtVXK2JpiVbTnhf4mcVPDLvHBRI2WNCdFJkJzIA7RvR6LyynM2AlYKc7K2Sb25ZQGkJlCbOCMHPEiSNMrbzWmWMIkIY4FXlpmjj8spLJap/DmYyW530gTnL3ZSE3R5iX8vy8JE4LwqIkVY5YFaTCkQonTOR8IjUjQ9DPEMLj7NuBPoU7gsrcnTkCx8LJ660ZfaUYLf+3Jr9UtqmsPb4+RUhRvJ4pSZ1IEg9l8h41dHQaPUpW3wwbGmtN8lcKQZjNFMd//9pXTMevPy3853v2SChi+vP/+svy+F5LYcsP302P/ov/FAD7rMwrKDBo9XUQD090UqMg4T7iz9Bt9ra0apScjVH+lUyKD9cSJBOqYwn5WG2QfUGmZawwGDwo3emxMD7kfrshHMFtZRIFwqSY5o1kUy8TvCFhEeQ4hih8nQNl/MAWNLwVT9/PZWU9OllVb0/lOLplXt3f5tfk7Q2BPUSZIPazDAayd2hIiY2lJJm2Z2oMHRkK26tbSR2dP5E/8KHM6ZyzLJuzsPusZ/65w2YGyO4lvXT0H2VvQixEgtWeqjEIJkzkg/9ND9CwHJKUnDvdDQxN9nkcZBKk/DHFNVm55n4Odifb6BZyvVUSVkMCTyT4p7mSSeDQFzhInIY6geExMEPNuRorAdz+mNoqq7q5xmCTBGg5mby5LAPrluIrQst575cydvE4pSy/EXlbsZZwpXItkqxiLeEx/Uy8Pf1UXvsmS/OmR2s4FqmmUJQrYRUOD7IPykraZ3Omx1CipBkDj6KTa6TyNSHlgvjdG8eyEa/RUB8wVIeESn7HyrVsNzj5WbnOQTetnAPdyaR4kM2FQr4PgUX9TAKGhhTRoQKmvpLwp4GNGCZZ3SLv41S8SCSoryT5sVyncfsq5uN5o2KkX6jMzojsbkiklWChlNM6c/LvoJaKwkSYntFzJZ9HMrY+J9bKvZSvh87+ufxZ8mbgjO5ljH6Si94HtZETRsx0+Vzmqg/bJMp1oF0aTJ9Tc21mNBu5j4dJrqRkpvHYh4UD28Qs2ZOU1OE+TLnU2rTHahFbp3Fs3VxTbI+sinTzpWOarsq+rzyh7mdS4yCMUqJdGKqVyPnUG+drOBfuEOkWcq8MqbyhUFluF3NIjoynupPtDPdwdIrZ847m3Mm579NY9TJMuI+sWhoTSd+svxjYJ6mPOJ43FXNgViKzVfJ7lQyEwQeWr4WKMtZoh8qLY+m9n2rsIdItpY5iYHh9ZonsQWSVSUtw0lBzkZQE6PQLc2STMluXjBrP95t/I4ck2aRFLqj9cD4l6Vf3eeKXqyBCqWVfWWYpn3/ibRu6DqPL4LmLdCcW20g9hWkCYSKsmoopqwOEMTmmh+bzlIOFojn66FT2sA3nelB5DMoIucjCBodSU6w6/NSO99zwGNJjQa7jwGzFsYYCkXfuJUHXT2xmpWV8di8yPr9wuFVLe1nhtgOjFY/zD60IUytJr3a4XvEtL14shkqRHDozyCoh+zY1aIVu/dHb5+Wa6D4QJk5CYnwcX6/7QHQG3XjCzGH2PVhNqCw6A6lB3mkO4vELlYQGERKxsnIcWknoTgZauvWyXR9RdS8BQVqN21O9ePwGvx9WS1JtTpFNNjN7Uc5BckY8gbmSI1ZOkmJnWcrZh3E7wiSGMbwnWS3Js8Ycn88ptqr3xGmJCpKayxBYo7WweaUAOlISkDbeF3n7vT96ETNDmMribSYvSjDPWF0yMIFayzZjHLso6XoBawPYy68jRpLN+4OR0Rw9jH9a0msIsp2hHiTLSsdaEGtHv6PKjPFQF4LWpLZFwm+0gLvBB+n9kXmUN8l3m8OEBn/mMNYhPGd4nTHH54DYtsfOSedgAI4DM/lmgNCw7V/9968+/orVjVz8+Cr9O3/w7/25b/d/+jv/3V/4scH3XAqrbSTtLf/gb//fPPnggt89e0Kpe/5o/QFdMJwUDc/2J3y3OuHDsxWfP3nE1eM7nAlYHfls+Yr/6/mH/LXLl/zh//Fjfvvvfc6X95f8m49+yR/fvcPu9Rln5xvu9xNmk5ab+wUfPbzhy58+5oPPXlDowElZs+tLfvr5u5hFjzcB5wJ//foFT9bnfHr2ms/vHnA6qfny2wd0RcCYyPXplgfTLZ/fPKD+eom5rvGt5dP3X/B6P2N3qOAXM65++yV1byEY5lXL/W5KfTvhBx+/pPGWQ1swLTsWhejp1m3Fy19e0p839OuS2iQuH62530wp5jUTnei8YeY8ISra3lFYT4oaOguFp/n8lMWP77AmYnSkNAEfNX1T0jSOvnZU84762xn7T+QDZf9RJ37RFwtOH21o1lPssqb9ySmP37ulf2w4q2q+fHmJ1hH/csrigzVNXRCjZrk4sK9LiklLu5tQR0VaFzz++DU+atresnq5oDpr6L+ZcfnjG+63U37znaf8oz/5mPnDHbu7KcokHj2858z1fPXikhgUDx7d8vUXD9GnHfG2IM2CsKVPJyz+3mtWO+nR6A4FykTU3+o4rCZUy5bdfYW7saiP9/jeUPx8QvjRgX5VUl7UxKiwNtJ1FvPlhO4qYLaGcNpjZj3pZUWcBdyy5ex8w8v1gvbllFRGzh+u2Tcl3dMZh0cQqwTLnuqLivrTjvfeu+Xp61Mm0479qxn6oNn+KIg8uQzQGIrzBvtP5hze96CgPK/xvSUGhbGRcF+CV9jrA/afzcXLedIzWbTU64py0dJ3luWiZvPFGeGkZ53A3SvCOw2zRUN9KCn+ZMr2Uw8ussu9rqmI2JVl8VVi9RkUn2xonyzgYQsvSlQviyv9mYTazD9csXmxQHWKNA/QaspXltXf8jlUSUJwVAC/TOizhrAqjr/vrSZOA+7Osv7bPews868M6fdW7F7NKF849h9GYc4LYcLNSmS/61nC3Sd270nv6vSpALHD7+9IX8xlYebCi1IgKPRZy/QfT/EzWP8okoqYZdsBd+NYfJVYfyKMpSwaRapXhvYijj2zcelRe4M9aFSfvYiNyinKmtVv9gJCNgaSIpx4yucSIuTnefZpE8Wtxm1FIl1fSf1RMkh37V6Lp1SDoCsJ65k+d/gpHN7z0q2Ze02TgnAi59A0ksx8eDdS3hiaxz1mY0kmoXvD9IVi+3Fg8kzTXoj/lqRoH3r03mAPSsKNpsOinXjxunMob+WYkxWPpdtJT2uYBczegBYZXvVaYw+w+yCge83kpeLmtzUnP1Pc/03xGUcnkuJ+kdDBMPsO9p8pio38LDqp09m/I5J2P5f7TfeWIr8PZAy3v1GKZC3KAli5Sqw+hTJLfyWdmbFLtFzJAh0weiKTFW+qz/UpbpuoH0jw0fSF4nCdUFFWzMo7xfxZpF0q2rMje9idas5+Atv31Vg7M3ktlTuhNFQ3ifYjO4b07N+NzL/VlHdaFlI6AeV3PzaEQnqATXdMYG4uEouvGStrmnM1Ssf72SAhl7oOexgW7WQxp1zJ8djDcdGlX4pSorpVzF5Edg8d3Ykswu0fS4p6dyoLddFCdwbVa5g/h+27hsNjw/wJxEIqleZPI/WVlYWxJrH6oebsC/Ef7t5TzJ7K4qDpRIY6qMbS0Ie5T5TryPpjg9tmZnmQ93cJHttxUWisg8mqgd17cPpz+V3oZ+q4sJHl5qIesFR3Ut/TnuTUXAXz55LqHEpFsXHMXnhWP6zoTiVNvM8e0HKTaBcKV1uikWPWQRZhVp9olk8i7iALMbZJ1OeK6U3EVyXRKKY3XpJojaKfVIRSan9CoWiXinIjEtlhoWKo0CHJOU4K+qkm6YlIb730itomjf2doaiIVmU/aIltE3YfMkMvtpFQKIp9xB4sTfZzdnORxA4VP/YQ6JaSWL34uqF5UI4+UBWHxd5IfWGzrD4xeV5TX88o7jt27y+YvuzpzkrxRLaRfmFx+5B7NIffVwGpIm1O+JmVQJ4sbe1nFt1Fypua5tFcemo7Acl+qseqnCEYyG07urNSWOldh942+Mu5yLdrCepJRo/g2c+sVNXkxYFYaEn61UpqXkAA8rYhlgIwh3CfWDnszQ5/MUN3AT8vKF7tCKcz9L6VRNrcmamHfWdwqDsv9Si53kVva/Fa5rTWgRxKKcFsIttR6ui7NMJEM8npvr1IVUcAO4A2749g8g0AjNbZE5o9GwNjOPglux41nci/U0IP4NcYUtsKWwmkmNnMN6tJnDmG+/zzqkh+5ZH+Rf7M79Ej8s8B0X8FHt9rYDlzHaePN/yT23eJSfEP73+LRdXidOT5/ZJ3L1YANHXBXTWFCC/ulpRlz7Ts+d++++tcXOz44v4Kf+H5w3/2MYvrHf/r//47/I1t8ZoAACAASURBVO7f/ZxvXj7ixgZiVLzaLUmd4RdfPsTUmiffXbI4O/DFLx+hK8+HH7/k6e0J/aFgfrnlH/3kB5h5zxfA7c2CWz0ntZpq2XB4PufWBr77+QOYe6lLKT2zacs3d2ekpNA/mdNeBG63M9p9gS0922+XMulrNd++PsPfVrDo2T0/g7/5kuefP8A9PKDPWrSJ4CJ6bbmrZsTGcr9zIg/ttKRYPujElznrCY1FbS3tJMBJYP3lmaSE5sRPXQRSUqROUzx3NNeacqdopxp3bygv9rR/cgpXgfV3J6QyUH9+im0Uu6Zk+82S2/MO/bRi8ddu2dYzNi8W6EZjDorVoiCVkgibogKvwEU2Tcn2dobaG3RQNGGC6xQvn55RvLD8rHyArjX1L06wEZJJxGvFlz95jOoVGHjSX2G3Gm/Fr4pOhPuS4qB49eUFaRIoXjrSRcBsCg5Lh2403cZiO4U9KNpnUyAzzlliHX85J1SJfhbQWwmxCRNNmAUu/tBy91uaNIlUzxz9xvD1rsC9csxWin6u2b26ENlsl+VoZ4lgrfSori33nz/CniYOywJbizy5cwm31nTXCbs26EuZWLt7CWlQzxbEi8jil5p+CWWeJO/0hPMnkbuZwm0Kuplj/oMN7ecnpFmk/mWFUaCSlcnZQRFflmz3jtkT8ZJWzy3dqYTXNNcCoCcvhQ3cODisJkxvNI0pWTzRhCpPzm8MzQPPdjWluDW4rcjW7F4CS0LpcmKj1PF0JwkiLP7PKc0DiEY8fqFKTJ9J3cJ2bsWT2cD66QK305x8mXh9KR5P00C/kHNla+h6zeJrmVS6vZEqmw723804/VYm4ElZ6WwtoOsrTr4KrD4x2I0mTBTLLzWHR5rJC8X+MSyeJHbvivfT7bIqYC8r87uPIsUzBxrm32TW90RTvU65TzNx9keW3Yfw4B9H9g81u/csbgfTF7D+RCYVbqOzh04YFrdTxFJBC5MXhu4UogflBXjGUl5rmuyL/MJmVlk+M8tVoj1zzL9NvP6dyOlLxfwPFcFJN2F1k/vvyP2qQTF9LgzT5IVMSqfPLbsPJERHe9A5cffs88DmQ4NpzKiuOPkq4CeKdimM7PS5HdNTu6Uw7trD9JmAItOA8jB7Fel/aTOIEInxyS8Tu/fEO1fdqNw7q5h/nfIkVlHeRzYfadn+ae6TbQVgJCsKlPI+oj2sfiisvGmkIklFUQyohLBlDVK9ckvupYTJLwSkhUK2GQrxeC6/SlkFkDj5UsCbzwE6bh9IWibe7TnMv8uVSnee5swyfRU5PNQUOxnD4VoYZrfL1wCRYFY3idnLns17jmInPZumViyeSApuucngoU2QNKZJOC3+P9vA4huRdN79SFOsYfHMs3tomNxGio2Ag+g0/QyWTwKmS/iJpr5QhF4x+47Ri6dDolwJeLj4qQTZzJ4y1uP090dPna0T1SsBcGabmO0i5TqgomX+nUhC+ulEgMm3Hf1cQI66Z2SUh/TSbiYBO7aNcnwbLVVBW0ldre5zpcuJ/Hzo30xKQoe6uWbxtWJy61E+0c9NVnAIYFt8KzUpu0eG5Vc9oSw5+arDNIH2rGDybE99eUKxjaIq0uKZNL0ksU7uI/Yg+9S9orzv6edWJKdZBn3ypVgS5l9uqB/PM6NqmD5tOLxTCYCKSJ1L7akfT4itgDPdK6I1lPeB8rahOysp7lu2H82wh4BbNXRnFTokiq2mvrAUa0/1dMf+4yVuGwiVzqBN0VxIoI1p7NgNCkjw0c2e5tF87BedtZK2qktJVy1vAu1FSfViD8xESptg9u2BfiELgu6+oX0wIRqRKOvMWA9fyQqwdpsOP3PY204Ycp9w9zVJa5rHU/GE5rAcs2tpHs0pb2phZQ+e9rzE1gG7aYmVpbxtRgY4TJz4KZ14L92mlS7PGHGrVvowW6kFSQphYDuxoaAUYeIw+xY/s6guYrYtWI296YiLSjyXTWYmBzC4qUmlRbWeuJxgb3ZyXmNCbxtcH97o7RQgpzI4HatICif+zN6jY4SQmVBAr3ekqkAdmvGaUTjYHY4S2BBJ0wq1F4O4CsKSjkxo25HmU1ht5f0pZjY3+7GdE8azOgYAAaPHUlkrnsmqlLoTY6Rnc/SkhjekrxHl7K/7L0NA5fCg1HtUcVxIfqvC5I3kWAXE5vsvh/2r/Ph+S2Hfey89/M//M9xK/jiEqaR9YhP2tSOWiXTWMfuTShiJhaxeVjeK9lxWk9tzCXZRQeR0h3+tJt2KHGP6THN4L2C3EupRrMULM6QuHt73TL+x+Kl4wWZPDN2ZSDjdVibCYZKoXsuHja3h8DgxfSayND/Pk5rXIjMdUgX9XJIkuxNJNxxCO0wrMtgh8r2fJdxeJkC7DyJXfwSrTxWmVsQyF8CX8l4nn2tjfL3OXZn9Mk9K7oXREDmqTOSiY1zlDpVM+LpTSW7s5/L+5iJx/v8k9u8I83C4ThSbY2ri/lEGBM/F62RrkSBPXmXPy5Ixyr09zQmP9xI7H4osY82r8CJNlDFJQiPs3pNV9GECmgwcHso5jo5RRipJgVnKHGWiFQpJZGyu1DiptPUxKl4KvPM5r2GIvfdTNQZURCfnwu5l0hBKNSYohmqQvMrkqrkQ1sDtRLrq9rKtIUhjkH7aOqcXrlJOC1VjuXhSjIEkh0fCIlV3ecVSSXhKP5OEzvpS5ITzF4H1h5Zik8a491BAcyXArp8q/Ex6ObtlDuPxIu3zM5nsT26ivC4nL0rCpYxFe5nIhUqk0fUDWb0fPF3i0cty1u1RNpiM+DPry2NcvhTCyzUvV8dagEGCPkh6u4WcP92LLy1MpAdTglhkgrt/lIFLZoQEBMk1q+4Du0dmrIAQeasca3sm16m6k2NGC7MxfxalYzJLcqM5+qaGbsVhvAKqFSg5r+6Q5X3Zb2kbSeLs52o8h/WFgOBqdRybrUVmO7lLqJBozkSeShJppaR5Zn/nVGFqkcZGK88PLM1wXlydJIHzVWD72FJsE9Uq0JwZurli8dTTnBqZ7B0im/csi2de5LROPHaml3GU6zjKv6VSwRNKTbsQD5okd0ZhLUtNN1dMbwPtwuTzJb8vSZGljzL53z80nP28lwRKhOnxpTAn3Vz8XEMASrTSTan7yOHaUa7l/cLMvNFLmXtaZWIrTExzrll869m+a5nciR+yvtCU6zTe/6ZPmEbGGQqVA3JSlgcf5bntMrM4c5G8DYmxIr9tac8cvtKjJFXlz5KBzUlaJMb9zBzlqggwUkGu37C/Qf46bH+Q5ps6yqIDwla5vdxzKib6iaZcyWRs/8hS7KT300/ES5iMIlRSr2Aa8TqaxuOnlvbcERxUdxL6Il5Aiz2I/zBaRbIiQbUHkWZGJ1JhP9XYXaA7tdgMNCWwyBMqg1t3tBfl+LlkD4H23GIyeyZsVRorGIauzUE2F3KCqWkCfm5wW5EDS3+nSGmHZNP23I2pqeVtB0bRLZx0XGZ57hAa0y0c0+92+EUpQCNEwsxBglCJV9DuA27T0VxPcgKqnEsA1Ue6k2LsrOwXbmTcJBRIUklJ4GdWZM2blubBFLfNvYldQIVILCx+ZilWrSSVRpFHqhyaY3cd/WmFaXz+nRLJalJK0ksbjz50+LMJuhZZrc4hO4OUNFZOWLcImNxX3Hr8ssIcOpncpyTeRSf7VI0nLErZ15Bg6uMIXEJlpaKjNONzyYq/c2DEVOcFYLW9sHV7kcMqH8efhblUa+iDbEtvG1LhRBbrDPhInBfoxr+9PZPltoV9G6T4CBpJYy0L9O4wPheXU+mmHKSswNDlGBdTAZGdJL6q3pOGrsz8emE3g3RWwiilHV6bqhK92krFh1JHyewwDx96I2M6+h4HmeyQ6Fq4LLHN8trMKhLjW9LX4/HmQKUslwXG18qkcPBdvSG7TQm0gRjkO0j6Kxwls0a2nUIcpa+jxPaNMY2S3SFF9lcf/5JJPG8xld/zVNjzH1+lf/t/+A/+3Lf7P//d/+Yv/Njgew4sl59dp7//B/8+L/ZLpq7j0WTD63bOaXHg6f6Uq8mOgy/46dOHlFXP5XyPM4E+GPZdwemkxqhIHw1PXlzw2Tsv6aOhzV2OMSkmrmdiewrt+ZNnj3l0tiEmReMts6JjXVdMih4FbJqSq/mep3cnXCz3hKjpg2biPKvDBKVyuqECpRKV8/ig2R9KXOFZTFoKE1jXFaXzHNqCwsqHwnY3oSh7rIns9xWPL1fcHyZUzmdvtma3r6gmHW3jKKuepi6wLuCcp2kczgVOZzUxKVa7KSlBWXgKG9jsK2LQFGXP4X7C6dWOfV1QVT1dZ7E2sl9NsJUnAc4Fml3B6fme1fMlxVlDDJoYFNN5S30oCZ1GmcSDyw136xnVpCMlJVUrLypOfnRL2zvqQ8HJ8sBuX2FdoL6f4BYtxiT6zmKdp28tsTWUy5YYNMHLtk9yP+XpxY71WljFatoRo6KrHdrlyUwcPjAh9oZi1qFUInjDcnGg85b9ekIKCnRCrxzpoht7FMvrA11jUS8qzPt7uk1JsWzpa4etevpVJZNBk4Rt1eBOWvpVyeLRlr63+N4Qeo26L6RPMcs16TXohCoipgjwzUTkjUUiDR2SOqFqQyqzJLOIUu1Sekn4zem+etaPHY/Ki4RIeUWcRqpnluaxlzHaCL1GFZI8om0i3hfy2rlH76SvMbmIPhhJ0nWJMIljn2RSCVPrsZMunASpfll43EuHP4mYrZYFnyJhFj0pQtw72X9QmI0lPmhhJ34Xs5fk2zCTJOBUBXRtcuedeDCHnsrheqIgTQPm1hGnEdUK8A/TmOW4Im8l/wwkHbZfyPGZvRGJ5kFLsJNO6EZTrDXdUvxLYSbVKn4pNS+6V3SnInuFI7PoZxHdiizSHjT9MuA2kkDcLxM6gz4gBxvJsbi9GheKSNBehCz9jdi1zv5sWaQZ5Lf5Isgx52Tj6BLmIMccJrIwgRZ/8eAJf7MTc/ArKy+BQ8VKH9M4a0V3LhUkUvUinvVkhXGVSb7UzwwS3zDJIKdRI9M8JMT6KSSbxm2HKiccZ+Dfz2WhxC+SSF+Xst1oU5ZqZjlsr8bU1u5Etu02Uj1i6qO8yM9yl2SWMQ49i/m0jUFRfpb9x4Nnb1CGTXOibQZ0gw/WNEfZpEry/36eFy6yL705l3PrNpKI2s+OMtjBx1ve54W1DP7tXhYkhgW1WIp31jay0Dd8d4fj38BQMnrVh1TWaGShwbTCrhZr8ZYrL/dcUvJa08ii2uSVMK8py0DHbr4ozOQw9uEcmS570nP3Y8wkQ7TixY9WFnqGRNqhImQI/5IMANmnpK9KOvGYXqqOYxgSZ/t5TkLVImE37XGBQEVZUDJv2K5MK37WQeYKWd5cqfH5lM9T0oyZDP1UWM9oFa7OPm6TQX/u42xPpHbEV7B4GmhPNL4cQEiulxkWkeIbEts3chNsK4sX3VyP91a1CjQnJr8ne0phZKG1l3OmvdwXg1ezWgW6uXlr0WTwMg8+9GIfCYU+pgU3kW4hloZyHagvjFStdMdFC7cTz7I95M/AShaWUALYh9yIpGSBI2ko1kEWQI34Yge137DANdTzmEa8uW8uBPi5wTRy0ENdSSiPf1+0F2ZP+i1N9hG/seigc+IsjH5d4tGbO0hgQyWMqN1LCJCMx6P6QJgX2RPs8TOHacLIdCYjf2NVH4mFQffZa5lBfnLZM7vvjr7aNgPdzo+dkwP4JSTxnI7SUGQbRl4zgHnV5/7KfSMe0UGqCsRKQoiG9yTzxvtDBvmDZ/QNMKyGZFoYQTC/ggVG/+fgfXyjw/JNOWvqOgGfKR/LAFSVfitVVj6wwvi+4fGnYpA/o47k+95jef7jq/Rv/ff/4Z/7dv/hv/Ff/4UfG3zPgeX0k0fp4//qP+Zk0tBHjVGJ1WHCYVeibWQ5r9ntK3xrWZwe2LxYjJNbZRLTactuNUXZOAIOvGL5YMfm9Zzl1Y7N7Qwz8cRek6JCmSRgI0/IbRHoD24EByhk0pwUSks6a187JsuG+mYqaakuEndurF4YJzT6CEp0LX4yVUbZX+4npDbgEmbWy5gaI2Xr8/yLpkDtDRjQB02cRJj7cRsoUHcF6bSXuo9cKA/Ih2YVMC8KYpGIiyBjSoDX4o2zCbM3xKsO87IgVLl4epm9hRPpD4yzILUVOTkzVlHAUadJLlG8NvhFnvhakT6iIFUB1cgfPLPX+NMgAGnu0bdunOCHecRuNf2Fx91YARwg4z7x6I1FeUU49eAVbm3ozzzuVqR4SWeWZ5akKmAnk+c4jL+KY09iqBLtVRjBRnsepI6kTGOvo61V9tHlQKgk2x7K3UMl/Yq6VZS3Wgreg0yyqluVQ3vSCCDa85jrGgSkFBuNnybcJm97njIYSMy+Ew/cEHKETmMJ/fSZSNv6ZWTyQv5I+6mkrAr7N/wBlonjMPEcqgiSgyqz5gMwcFth05PO1Q61TJBDmZi+VBweJqrXObhkIhUKdi/HVKyFcfJTAVG6FSXBMLkqV8Jg1ddR6k1skiqTN+wIJr9nUA7UDyOmVRQrYUj95Dgp0528JlTil+vnwwRYftaei7SXxFtAprsMnPxE7pXuFEg5mOtE5Kgq11W0p0cwMZy7wZ9HnhBFK72JfpZBVD6/YZLZ5RsZ65seNZ9DmKIVQGFqcpm6JH1Gc5yo+uo4hlDJ66pbYbijfdtTB8L6F2sZLwjoRMk1FCZ+kCWL1LfY5n34zOiX0J6LvFEl+b/bCQBoLtV4fMPvmJwrUWUIWMlAdi7jGMBcvzgGf1V3aQyMcrs8qd+J784e5NiKtfQ0mk5ATL9QORVWjeFLUqlwvHmGMByiMLq6g/paVCNDxcwQ3hTKHGzkBRwN3YeDF3O4tgPjPYLNyBhOVq7TOMHu5nI8poPmQsK8wkSqFPqForqNtGfi26zuBLgA472s/FEaqr1IiA9XohQwDZnBTzl5UzG5EWXC8PfF7eVctPkcFnthXk0OMfJZYWEaAT6+UlSrQH1m8BNFsRM1TrkVhlZqc6Iknhr5HbX18VjdQeoxurmmOddU90dVQrmOOXBMqkW6pR77PtuleAH9RFQRtslhUL1s17QSjOX2gf21HcOhfKVEltqlERC6WjpuB0AzsLy2jpnxF1VHtAKmin0cg5xs9uQNrK+f6NzLeWTSi7UnVHoMiTKdXG/TCICBrLSZ6sx+C6AbvIl+anBbT7+0mFreIwxqlk32IlEfOlT9RAKUik3Ivj8JeQqFvE/3EV8J6JIANEkqHfZh90GY5SbgK0l3NXXMXtsglRKJEeAPTKRpA/3UjuNzO0+0wnAPTKyfGOwgUUzCUIbKoIMA7VAd96GbIGmrtR9DjmIpYx39jE0Y2VMAU8s+RxltDg8ybcDPXQ6w6oUZ1VIlJbUfAtDGsJ3MzOnuCBolgCjgTyfCSubtqwHgvMEoqj4I89rK+IBjIFEGfao/Mp7JKFQr/sM0KaSv0lmRiuY6kRH8GT0CuIGVFa9mehsU/mmPP2UuLz2d4VcY0TcA4MBswltg763tWTsyk28F7QwP72XcQw3Jm6BvYC6B1HW/XkPyq9v6sx7pT9nH/8fHvwKW//8/9J/9kr+8j5QUP756yXcvz3h9v+Dp8zPqQ8F8WeNcYL2ZMpu2VPOW/h+dsXy4Jd4XpKQIrWH/1Qm6CHBTcnq6Z3m+B2D73ZKzP7ICRKtA2MiSbDnrSK3h8fu3oBIpaPrGCvPTKwgKVQRmJ42wS60mPBdzU/+LBWbZ4Z4XxL0sdadpQLWa6YM9+qAxWyNXxEZOf3jH5Ttr0sFIZ9/LErWzqF6jtwb3xRR1V2C2RsDe2mJvHMUzR5pEkkrwuCG5SPlViWo15S8r3Dclbi01JNV3Dt0qYaA0qFZjXxSExy3aK9ytZfKkoPpG9lPeGapXlnDiKb4p8Fc9aRIIZ57ipR2ZGBUkkCSceLrHnXib1obyhXjOpl9bAUa9hH8Ut9LDp1tF9W0hPry1eDtPfmIpbwyTz0uKO035yuAve05+avDTxPInDj+PhEo8VCpC9U0h5fFFwt5bZk8s/mFH8dqOQSfaQ3flKW81s2/zKnGnsBvD5IVm+TNLca9pHkTCD2vK1wZTK/5f9t7s59Ysv+/6rLWecc/vfMY6dWrq6sHloe04trvdTpxABoUgQQhXXBCJK27h3wCJSYBBIC4REQHLRgkmsePEkd22u9t2ubumU2c+77znZ1xrcfFbz7Pf6rSJhRzZFWVLpXrP3vuZn733+q7vVO058pcGF3tGjwUk5acC2Jp9mVCwCdQHlsFLGSR2fYDjT4TpKu5ZbCpfpumVYv1mQ3nisKknPxUAlp9qkrnGZSKldrHHDqRKorxjpV/TQ3ahWb9b92xAeqXIT6Urc/BCU+57ULKO8tBT3Lbi6b3W5K802ZkiPw3djTEkCyVBJkoYk8FzRTOG7V3b9zaWR076A7fCim1/rMAbz+iZYv2akyCV+456L0gxEXB5+DuK5FrYruGLEBzzVs3gZQi0OZVKis3Dltn3AmDfhN7CjTCW0Ubk19mFgLTtXcvRNyE7C4PiAkZPYfwpjB7LYBwE2BXHAn7yU6losClMP5RtNmM599EW8jPF5IOI5TtWpNcljJ5I5cf0I2FoNq9btrc88SZ0UnasTSqMU7SF5HqX3Ly5K0Cq2hM5/PIdSz32nHyzZf2aVH4kc3mtuOVFPruCwancr9UerN50bO7K/ZSs6JnA4XPp3TO1SLvTuWd7R2o8XCwgJd6E78wIRk891b4A3HLfUx0IQB4/ccJAxZCdSyDT5JGnOPLk58I+VXuy3PQDemlyHDoQVw/lmmRX8t50LlUazRi2x+LVTBee7FLO/eSxo7gVgkQy2b5qYfxUKicGp554KaCyOvBs7ioGZw4bQN72loCn9Frk14NXAnJmHznyc8/ehxLvH23k/MpEike1AiKbgWL1QHH82xXlgTAe1SwwxvviHW5GiuJEBlPtUJJ5IVR3RASZ407CLkmiIgUfvrJUM8X8C4rNLd3XlmxvKfIzR3Es6cb1RDE4lW15DeNnlmqqQs8jfVhNO1CsXpO6jvU9zfq2Jr9wTB5b8ktHtafY3JbqjMPvVNQjkZe6RMDs9ljkyJMnrUii9wRUFoeazS0dQJuwUNVEvsdWdyRZdfKk7VN6r982vcy5Hglbly6lg7EZKNpUkV+58N6IqPLMPm56ljFZOdZ3DV4pVvcNywcRuvEU+5r1Xc3opXxoRy9akrWjzXTPTCYrR7J2VGPF8n7E4Ey8oPHWM3na9AnKydJJd6cXUBkFJiyqPMMXNdVUU+wbolK8knHhRMqcK4p9w/BlxeaWYXsU9f/Hw/pOxOC0IVk5ssuGai/CpiIzTha2Z12Lo0jqNRKRUGfXLcpJ7+XwRUVUONZ3Y0zlKI5FolscRgJIE015YCiOIqr9iPS6QVlPM9LEq5bh867iRFQei4dxsGFotseJ1Ip4CeBJrmt07djcTdEd02c925NUfIzLlmpPpMI+UtRTkT67RAeG0+GMYnssX3D5aUl2VlLtxdhUXm8HBq8V6WVJeRDjYo1LNM00FmBZCxjIX25kmdpRnqRE65p6lpBcbMW/2Mq66llCtBGgaWNNtKqJz7dU+6nUj4zivipEmFSR3JpClqn3s5BkLCyjLhuU88RnK3TZYDY1ZlNLKq8Hl0a0o4Ti3hizqYWRTCOZ0FEhRdeIhFeVEqSjWteDSh8b7DQT9pEg+U0ieX6YoFcFPk9w04HIb+NI2MCbqbFBEtt7JpVCr7eopXxxq7qRFNlSlveR6QG3z0K1SeefhB5Mq6KS8J04km3Ark4keCT7R5f22o+vBYD7qhZQ6V3fQ4lWAii1QqXJLlnWO5TRvTRWxRE00qWpIqlCUUr1/3XvUXEk6zL6//M/jOEH9lx+3h7/EqpG/izVjXyuw3v20i1/7eA7fH3vQ/7B+ZdITMvT1Yz3Dl7w3ugZ/+0HXyONW75+92PObo8B2O5d91LX/I2G2hkOH665KEfsp1s+iI84P52S/s0L3ogbPn15QDwrybKGQdIwPpjTWMN7rz/n/Zcn3NlfsixTJlnFJC355PKAe7M51XjNs/M9GDe8feucJ5M97u/NWe6laOV5Z3ZOpByfrvdZVSmzLxVYp/mZk0/4pU++zP3Jgu88useth5dkUcvVZsCb+xdcFCMGcS0hP5uE6d6Go+GGD5+eMNtfsylS3jm8YlmnfGnvlH/86E3y20tipxl/oeLF831mh2tmcUtiLLGxnK1GHAy3KOX59NkhP/HGYy7vDHn04pB4WGFbwzBtGGUVZRPx9mTJ6mHKzx5/xP/64Y/yVx++z9/L3+NH7j3nohixKDKs09yeLsmjhkhZLooRV5sBqXashgPu3Lrm9GrC3nTD+VMB/VUdUV7l/Bs/9nv8o0dvo4HtA7nWcdZwb7rgDz+4y8mdOVejAW8eXfPo8JA7BwtWZco4q9jWMcsP9/DHFaNxyepshHqtIo8t1UNHFLdEkaNpDO8cXvHx8IgsrxmYXQ9qVcYc7q1oNjmvTVcsy5T5G8JAv3v7jO+q+8QnBctZwsmdOacvZjx47YJlmZJElvPLMQ9OrnmcHKKXEe2DUnpVjcd7+NK9V3zw8pjpZMv1YsjrR9c8fnmArwzbr5bsT7ZcHQ4ZjUt8E1EeRNw6WvDik0PuvPeKNGr55OUhUWypnw2g0dI56WF7IKz78HCLexMmSUNRJZRFzO2jBeeLEeqOo5hnxMNGJkasAuNR2tM8bKkXKfG4Joocq3jI7S+eUbUR69sp20VKNGjZjmPMyhDd33Ay2fCiNNTHiruvXXJanfDejzzi8XyP1SbDXae0ueXsSJFMK5rTHB9pmv2Wwaxg9UaEm7bUBwazVxErz+LtnHvvveLx00NObs+Zr3OabUJ9aBjfWrEcjsF4klnF2U/l6MKTPFyxvhhg1non14w8g5MN12dD9u4umF8P8UXE7PaSuZPr6wAAIABJREFU+ekYm0b4WyVNbfA6pp1YMB5dGMb3l3AfVk8m1DNN8s6C68GE7J05rDLyewWrJxOUVUzfvOb6dAKNIt6rqE5lQkmXivawYXy4YXU9IBk0YDWR8tRZwqufjIgerDFvWdYfT9D3tiKr1wNc6jCFxqYOP2qJhyIl35zloL3IoitNci0TGM1RQ3QV4VKPGzXQaMykYXuWygz4QU06aCg+HmHvlJTrCCJPPK5pNjE2i2kmDnNUsj7PyO+tWC2nuAcFV8MUn1pRTpSG6iCi2RMAEM0N7X5LOitZPx32xe66UjQntcjyspbrYU47lmNqJy3FicEbT/16BfOYempoxo7JR5rtWzXtIMamYPdroosY/YU1V9kQH4lc2R1XuCSjPLGoWU07zLCpoxlrqkNHeilBWJKG62kmMjlhCul9ja809Z2Gq8uU4rZjk8g5TV9FtEPxCTYjT7Nvia9lMkk5YV3jDWwmIidtxz5Ie7vkWJH5icTT0x7X1KsIey4JrvWtFptK+u/qvmZ71+ISQ3koQTteGVZvWsojLZLmrKsMEXXIego2cygnYWGmDEmsUy9VWhZckrJ6wzJ8Zqj2hPVvhw5TKKLCUJwotvcsw08N27sOXcvkUhtAfDsSSbfXntFTTZsblm8JW28TYTqXr+8UENWBDNLLI7k2w+eGzR1PeiVANFmY3h/fjAzzr7SMP4jY3pHJuHZgxKsbwfpWhM1U8Ijvcgk29wz5qe/9pts7Hm+i4LVVZHNZptqH4TNRTLSDLj/B9xUznCQ0I5mIageRMOfzXWdpPQavMjZ3RX4t+Q2ewUtDNQXdJFIZVBqWD2H4HKqxAOhmIBMNXsmkULkfFBqZyEOrqWJzImms5SHoNqIeKfARqwcK0xjaVFFPZL9Xr0GbphTHWuqyMgU+Fn/2WDF8Kce4OYmCN11AfXsYUc4EkNtUsb6r8ToWP/u5E196LEFo2xMBouVM+j6jrccdalyiSOeGcl+jWshiBVpAzPZAQEh+7Sinkg6bLgRIr+7GUosUWNdqlhFvPTYbin91ElPsGZTNQyfmmGLfkF9ZVncN6dIDCeW+IS5Ehum1otwzZFdQzQwuHsj1aiUFt0t1rvYEZDejTCS8zuNHEShFFEsXaCeP7kBvtReFyidFvIyxuUwWJAuPSw3OJLJMYEFtatCRpjpMGDwvpNsy1agmwg5idG2l3qW2+FjjoxntQBJlvRlSHmXkrwphNQMAVZXGZTFqkEptWmIwC8DoIKFNsZMMvb5RkeI9vhYfKk2LH2ZQNf3fPgQC+SQWL23nzewqV7QSdVjHaJouejkwiMN8FxgUgKpsWKThaC3gNPgo0WbXXWkdqvNOxjE0zS6Y5waAVezYR5UkP1Aqe/OhguT5Xz/+bD8+11LY0Tu3/Nv/2d8hj1uWZUpsLEZ7vFeUTYRWHqMd2zIhSVpJNfWKttU4p/pZAwUMBhWr+YDBpGSUVWRRy4urCXFsaRpDW0fglHjwFKjIYSKHC16vJGupCjH3e6vFPzZoiZOWcpUSZS3tJiadllTLFJU4/DaSgVqrUZETqWsVJKvrGEYNvjbipcsamlWKSi1Ke7xVDMYVm6tcPHNeQa1RgxZfGVRi8W2Q54KAhw5AxA5fRPKaC8+VIlEldlL9kIjBnUb1fhuMeNw6bZbaRKhZDWcpbmRFxtt5AhstVQipFzlnFmbzKg3TRmTFW4PXsk5VK3zm0GuDTz0+s7K/Ohxb97eiT4xVhcGPWvEXetV7aZRTqEpkuz4XT6IuFHYkM9gSwqNQjbxH1wqX7bx1PpG/+/cYdn2D3bkOBhddi/fQazlOZRV2ZGUA6ulTfDtGsevDc6nHbERm28kGdS1+OV3KwMQNHPE8VHuEcCpd6tAh2XncQLeqT7NUTiSpupFaDbMOMpTIyzbXOkicwvK16l/vzocwuiFIZ+RIrmRw1PW4SmfazutmE7+T1Hp6v5eLkM+KD2mbbvd655PsWM9oEwbQeiejjDZKArhUWI8V5jNeyXnzCqIgM+7U5D4cW7SVGgxTyQDNJUGaWYRjDCFO0ger8LEAUeXEp9id8667U4X0U5f4IBsOvrFwvqMwABUWT0CVS4UtUg1o20lyZRu6kvXagSeZd2X1YX+Dd6m7V13aVX2EU3wjyEjZIGkN+9vVKqhwv3brcrHIO+UDEu5nFTxmCqpDCQzrvFDS1xiknnVYR7TzSvlItiGdobKOju3s+li7dYjvSzatG3lPJ6ntZJ7CLAVQZHaST2dECtzJxvtr0vUJh3u+kw3rdsce+3COTCXb6STRvS/XCMPdBZEpvztnnby1k592UlcBzbsQtJvnqwuZ6u/zzqta78KqenlwCPrqzp+cG9/7Ebv6i44Bc/ENHyhBJmlDd2mwEXTdlS6S8Lautxa6z5N8fzbDHcOqvLyvGUlAWBd21t17aHrVhW7COUe2q6wPx/fZ+7WTaNtkd/9191oXkkb4TJtK9q37fuiuo/q+cWV3X6jg+wR6+a10mErIVbwN51oHr2NEHzjV1W24SLyX3bnqezHZXcM2VaShqqTrA24zkbO6KCxnhQHt/LrOiBfSGT7TWdrJKrUN79Uq3Mey3psezG7fTfBUtpnaeYO7+zZcI1O7sP+EFFq/+35AtqPsrs/UpapncOO1DSmuYZkAFnr5Zzj/Lgn3RRgDdN2n3XF116DzMfcaOIdUhzT+hr9Z1ALtwGBKu/NhQi+x1N0xdfcf3e+ukm5YJdJXm5ndsnTXzn9mnVJ1Eo7x5v0UmLiud1QHaas3OgT2+f68qMpCpPtOUfFsBm+nQsBjF3gTPME+0qGSJHQph+fxHq9Dt2c3pgJhPm+8v2M+UUoAVidfNVoCfLq+S612LOTN3subj676A+R4u/d02/h+qe/Nv7+vu/MHSm27upIuTMf5f95X+Uct33k8/7gY5IZk9vPeY7n37rH/i//jv/snvt6/+zP/zZ/6scHnnLFsmoj9QcGzqxn182EfTqAsuMOa5HEq6YyzFjuPaEeW5FokjaobjOUeH3tWqfTulRcx+sWUxQjsviM+02Rl8BUdOhnoxo7kaYZLPNlSQg9s5okUUk1xIIEXysYStDCBZJmwfLdBf3tMZuhDLJSL+jAFrwLgKCOqPU90kREVnY8nRY9AuZh66hi+1Og6Y5SK/ylZirwv/SgKnpuYzQNLci2Jtt2ALF6Jl8p0nqzkxqDNyHo2dz26NgxO5QuzG9S0mQxAm5klfx5R3G8Y/F7O5p2a/d+MWd+XDrvRI/Gm2RT4yorq1YDRxxHZpcjXkg9Tlm86Rp9q6j2I1sggMjIMXnmuvwLDjxKaAQy6YAkDq4eOvfcVy4cwfmzY3IXJNxNWD8QH1gxlX2cfOsp98RbGazHn11NF8rF4j1wiA9v1Fyv2fyOhHSjqmeyHj8SftrmjiFfit2pGnskjmRlevWU5+XUojoxIDk8Ug1ewvq+pZ+KBbKwBBZOPpcZCBpIyWGgHis1dT3YtKbooxeauY/RYAF96pbn46Zbsaczge7K/g1fSv5jMZb0ugeGHsH5NtrF8Q5EtxGeWrGQbxaEi/USzeghRoRi8gPnPVqRPc0wpUkqbhEG7g+VDLamha0l9XT2QUI9kpSiOFOsHjvRc9+nC3ogscflQAmpGTyPGTy0XP2SYfOK5/rJ8FnAyc1/NlKTOTkWeN3rpWL4mctPbv6bY3FJy792We27/u5ardw2DU0mA7fxyXmuKE0kirkeK8gCO3pcB2MWPeg6+rVBO9V7HeiyyxvEjz+ILIhs1FVx9BdKFYu9Dy/I1AyVMH4l3rB5LqMjlDwuLsf8HMkhdvaaYvC/ded0AePRMQOjyzeCPNAAhFbeTTQ6kZiKZywCwDfdpsoDRdz3XX5BB6vQjCaNxkfT82USJ7G+iqaciWwV5n/K7AX9+Jb6p7ZGk67aZojhW6FL8k3GorHCxgIrxM8vmliFei4RUNzB8qpg+aoT5GCoGp5brLxhOfqvm6t2Ewbl05m1vh/tiKVJK5UWmWU005aF0MeaX8lq88WxPpPYBD+Onth9U21QxfNmwfD3GpiIJza4ti9djRi8t128bDv6gpR1oin2FS4SdmH3asngjYvjSURxrpp+0FPuG7S3F/nel7uTg9xuKg4jhy5rl6wnxNgTQKMgvBRhUU6kaqfYUt/9ZyfKBJIEXh3J8LobJY8vmWOwJXUKxhKzId1KbC5huB3J8fRCMhsmnNS7RbE4iNncUUSnXtBmIhHb8tGXxZszk05btkWH8tGbxRsLohSO9ali8mZBfigzRpuJl3B5rAUWNpGbrBg5/rxbfXfAO1iOR8A7OW1b3YvIrS7EvwSz1UPoPZx9V1NOI9R1DfuYoZxpTeLK5DPSqyc4b6GKRwcaLhu3tlGaoaQZyr6CgmkqK7eBMehfXtyOageLwDwqWr2W0uSRK52c1mzsp2aV09C0fRIxeWLZHcu2Of6ehOBB53/77W+r9hGhjwzmMya9a2kyTXTbYzOAixea2Ye+7Zc9CTT5pqA4yikPD5JMSlxhsLv5pXYduxFWLbj3lYYyLFMPThjbX6FqCa+qJdFTuPyqYv50Hz6hcs3Ql4TqzjwqacUy8rLn+woDRy5Z4JfulW0d5IDMbydphCgFwnQ+xGUehJiShOIgYPa9pRobhs4rlGznTj7e42FCcJH3P3ejxluogQzmP2bZE85L2IGd7nDA4qykOE/JTqcOQxFZLtG2ppwnZ2RaXxaxeyxg/Lqj2U/J5TTOIyE8LaB2b10cMn6ypDnN8It/ZphZJbnZWYQcRzdiQXjbkTwuUtZT3p+LdLGzvz4wvNzRHQ5Gvblpc8HqqELoTXW/xSURypSjuDsmfbyhuD8mfrqhuDUnPt1QnQ1TrSV+tqG6PiTYNZlnitaY5HBDPS5r9jPiqxA5jolWFHSSYokGtC+zhGBcbdNWK7NUhYTatQxUV7eEYM5dKjuZoRDQv8bmwlC7WZE8XImONNHqxhVjYThUbqQNxrmf+/DBDL7f4NBbWcS1gR0WGPhE1idDXG9x0CN6jV1v8IEUvNuKhLGt8nkrf5ChHr7ay/qbFl5XIP+NYPI7DXKStSbxjGUOlB9biRwOpFvFe/q4aaAvIM5HGlqGHMk2EudQaXzf0oTs96AwALolRWuPLauetBFQsrKkvS0hT8E7WA6go6utFsFb23VrxV3aM5R/BRv6Rj39Feiu///FnSbr6J/34XDOWky+c+J/7hb9F7Qwn+YrzcsQkLjktxmybmLdn5zzfzKjaiNoaWqs5GG4ZxyV7ScGTzR4az9PrGcVVzk99+SN++9l9rNUY49ifbGit4f7kmsRY3j8/YbPKMLElzxrGmXxQiyZivc2YjgoSY3l1OWU62RAbh/OKbZWQxg2L1QC8Ih9UPXMaRQ6lPLERT6gxjum4YJjUvLiakKYtTWPwXlFvYyZ7W1aPpxy8ecX1Ykia1UTGYZ2m2AQfxLCmaQz1Nma8t2V1McTkFpQniixNFZHmXdqrJY4tbWsolynZpMJaRbNMMcNWgo40qDCt7s4k/dTl4cshFWZxsFfQNAb3bIC+t5VE18ZgK0P+cUrxRk06qmjqCH+V4BPP8HhD8WRMem8tz3uF1g73PMdOJDjILCLU7ZJ2LdJLOwzb1R6Mx1yL9E8fVLjawDpC78kXvC2NBBlNWkxiaZeJBBoNHMSO+CzG3i9Rr1LwCpc6/NCijId16FbKhYmNxg3tNiI+j/EPCpxTuELeY4YNvMhkP2pF/lKzvefChIHC3qrwQXZoFhE+8iRzTT0TuZkC3P0S1ypYSoemN8Im2JHFrA0u8cRzjUuFLfRG/JXF6zV6FeFS189oK6vwqSN7HlPPXD8Ln51qtvdblBdGN7vQbF8TCnX8QcT6dUkiRQtD1gSJn0uEXbQDR7TWNDNJRq0PLPHCYLaKduR7mWHXu6i8hNjo4M1Mr6E8kHCddmKJloaoVJS3Wulh3Ircyw66GglhHbNzjUtkuY4JbYcStNOxQl1Ikikl2Mim4fVIGEMfCXhafMERFRL004x8kMyGlNJUmIt4GRJLSwEE1X7oCl2qXQLuIMgLB7JMfqooDyTYSHnYvN6SP436apB2KExqslASzNPuaoWacfCSevrk2O43x+ae/FTv0kgb2N6WbUZb9RmGN7sQP2NXV6HbHVsswSqyTheFjs+1LO9i8bBWe8K8JkvxUpoCykPfM1XJXM6nKQVoFccyCG0HArDkQggraAq55+INVFNJOW27EKHAErZDCXmKSpl46IJ+OmbUZvRBNqaSCYauCqiXVQ7kuLoqpi7VVDcC0rv9UmE/O/CXLDrG0PdeJRRMHrfM34okfXUTmMom3M8hYbIZSXAQIIFBpQ9JnYF1sTKBYoruvAuLZTNFfiHBItsTmcCpx4p04WUiaB4YDgWDM0kbtYkwSV36qgvJrd7IsTVDYZWUkyTa/CywiNUNlr6EZOMp9qXmQoX00HjjqWYCsLuKIxfJxE/PHpruOoivNZtLXU33Pm3lXisP5HjirRzr5raWyZFU9ddU2zDRVvge7OsW8kvL9tD0rORNVs5HkF1LZ6e2niaXEJ8OsDZD3Xe0dgx7dh2eD0m6yUr+3QUbdexfx/pGpWN7ILJLr0MAEdAMNfHWha5S31e72BCe04wN8cZSzmTfdXMj8OggkkTbKtxf3tMMNMna9RUyLlYCahv5Po8KR5tLmmh61dAODcmypZrFpPOGehrRBet0ya420X3AjG7ke8EmOiQe74KrTOn69zUj0y/rYkV2Kes2pUPXDpsL5ZksG6pZTFRIeIyPZALDpor0uhVAGepmgD6IKCoszTgW9jFWxMuGei/BJYo20yRLS7SV5eJlTXWQkSyl79PUsp/NMCKZ1+KntF58mt5jBxG6EeZS1xLak8wrbB71bCYIO9cOJB3dbMVr3Uxi4uWOxhcWV5jPaFVJum0WidKituK9HEZE6wZvNNGqCt/JMbqx6FVJezCU87ttgqS16atbfBKhyhZlrVSf3KgG8UbJ641FbSsJ9WlCmI4PrGeoRlGBpVRFJT7LpsXn6Q5cdswmfIZlFOZZuiuV8/hUAGlfcWKtAMwbFSSfqRu5EbzTg0StoJLxlW/Csj172sp90rYCiMPzvgkJt99fQ/IDHn8sPOL8vxKM5c/9D3/rT3y9//vX/us/9WODzzmwzN6863/mF/42rdM8fnlAlLS0dYTSnjhpaVuDnSfcenjJ2eWEOGmprnIZeEdB6hmkoum4olqmEsCTOthEpMdbyqsMXRiilaI+CBURxqOWu7zvaKNo9i16q0VmN7CSotqITNKsxYvi98OsznWMSx26lvfYWYsqJcBGWZHi6UZ8QFK/gEhGFahGsf/OFcvfPZCB/dyIlHCtsIMAAIYip7T7DThFdBmLdLGVxMtkqWlGDpdIXYSkTYrUL9rC5oFFNUqqBYJc0sU+eH0gub2hPh+g9yrsPAENw0cR1b6nnVrS0wibi8SPry7YXgxIziKySwmCqafC/MYLjXpnTX06QJcCaOKlpn67QL/IRJrnxcPTJTHqCuqTluxFTDN2wtoYT3IlPjOXeAYv5ZzVY7+TkKkgo1xLhYSpFM3Mkp5FJEvY3naoVhEVimgD6y/W5I8S2oGnnTqylyJJLe61jB5F2FgGnvVMBnjVvqeZWKK1+MbsrCV7mqBDGIYppQO0S4v1RsCWKaW2Ir2S6wCwfdCSnkYiXdTCdFV7AmZwgXleSxdqvNrJGKOtDCTbkQyEXRwAWBHSKH9sjvvNWQ9Getmkg819F6SNArqqAwkQ6gaFfYptJOE221vSsdoMobzbkL6KSefCqiVLJV4rI5LQZKF6hq7aE2CgWxkUb19r2fuOoTiW/a/2BNDFa6gOdsDLVAHwzEQGnJ+FBNgAalwExW3L4IUJ0i+5X2wnN72WQKHRE5ES9+s5FaAHsn/tUO6VauZ7mfL4U0mOLPcV8Ub2u5NNRoUApc196cV1sYCRwcvQTZsKsK0OHJNPZBDcyyrDte0CbOJVUAgE1txm4neyiXSlogRcD58LY9z1tJowwN8eK0bPBaR0susOeHcy1WYk6bgu2tVgwC4FVt4jIKGahU7Ziep7X5uJVGi4EFDU5gGkhc5SXQu724zUZ+SgAMNXjnoUpHtRSBmNAiPtIV6FJFgr98nglYAUH+R2Lpb3lAeiJrC5nKc6BM1kl57tbcX4iaM4Cl6yqe7TQFGQn8trUunhe0WIi+VzUE8Uycr3jLfXkKw91VTOU7WnyC538mwQgIAXxYDUXnjSpcfGUBzpPtQsnfteKpusPKvXNPmp7+WcHfjqPIPdvzvZbTtQvazQprLM8KX7TCJqm+0mC2wioLUZBvBrd2BRWyj35fqC1H0ky9BJGxjMjhXPrp2k+k4NLsiq8ytHM9glqkZlqA0ZC/DILxz1eDdZEJUSwmOC3LY40AzOLDZRbE+0sNlGUc4Uo5e2l0N28tfufJja99LrNtN9DYcpBcR4o6gmmvwiJJUORN5oGs/2UPehQl0PabKSeowuVdUm8v5kZdkeRX3oD8j5dDEMT1uK/Yj8sqXcN6RzJ764IEsFuR7d/dPVh0jNjEhJde2pZoZkHeSYrZzfZGmlTmMo/j9vBMx2AFQ5BFgZRTMymMr1UugeLLciCbWJ7sFZMwy9piosH/oktfWU+zHpdUs9C92LzhNtHfXEkF00lAcSLqQ84DzJvKIdJ7SZId62tLmwvNlZRT1LgsTZ9ZNjXbBQvG776ox2GGEC25ksauqpBPY0Ywk0irYN9TSRTk/PZ+o8bGoC8DOYQkKRXGrQAVR2abOqlU7Vdpxiiia8FmPWNT41uEhjyhasp52lwghvmn6iSVdt371pB6En82bYjVISxpMnu6fKRv7duh5IemP+OemrqkN4T9PuGMguxCcyUDfSh5kIO9j3YZaVMJ2BIRSQKkmxPk9RZS3bzFNhLJ3Dp7J/alsK0EtiYT5vSmLDuj4jWVVKgGMj2+o6KFWoPPFliUoSAZhtKwyElmPq+yw7oGqteD6V+mxIkPf4Tqp7MzH2X8BSft6B5ezdY/+NX/j3/sTX+398/b/6Uz82+JwDyztfnvm//j//W+SmweDQynMYrxnomj/c3uYkWfKs3GPRZCTa0nrN0NRo5XFecV6NGEUVpY3YtgnrOuXHj55wHK/4/dUdlk3GF8annNcjTrcTKhsRG8u2idnLChLd0nqDxjOIaj5ZHHA8XJOZhstyyEG24a3hOf/0/A1GScVlMeBksGbVpJ+hwa3TDOIa6zVFE3NntOCyHLKXbnm5mRBrR2pa1k1CrB2bOsFoxxf3T/l4cYjzinvjOZ8u9kmMxXrF3dGC710cc3e64GI7xGhhNRNjqdqIPG4ompgkaom143IzIE8aBnHDq/mYKHLkSUNkLFp5sqjlYj1kkNZcLoZMRyXXywFvnlzwwZMT3rh/zvl6iAL2h1uuNgOKQr7QxqOCuo2Y5CWvLqbcP77mepsTGUtsHFUTYb1if1CwKDI225Rb+0s2dZBcBG/s/HzE3XtXXG9yhlnNYp3zxVunPF1OiY1juZV8/r3RlsvlkChyjPOS1hqulwPyvMZaTRxZrNNs5jmT/Q11E+E9TIYl81VOmraURYLSDtsa+X6NxaPb1oZssPtSa1tDljYUZUxTxAwmpTC31hAnLUp5yk2CbzSHt5asi5TyIifZL2mqCBM52k2MGTZoJf7gdhMTj2qadYIZtLhWif/WK+K8oa0NOkip80HFdi0yPm+19HYqj2s1w3HJ+nQEHrLDgvr5EDdpUcYRJRbb6t2ARIPdRqjESkXOMoXYMZyWbC4G4sGNPGbaYFcxg6MNVRVjlwmqVjCTAYAvDeleSX06AKekh3LQynHWpvcMp8OauhL2Wh/UuMsE37Hg2hPlLfYqxedWvLiRh0x+gPR16Bh1SvpG92t8q9GLCI4rGaRvI8y4wa5j9NrgJq0wu6H2JrqOsCPXB+CgQE1qYaGDV1iVRhjavRai0COayf6IT1LRzuQ1at1X2qhKgwGvPGpgYR31XtebtSkopBfzXH5o6yOZSOp8ld4E0GE8ZikSfpdKcEvvS44kcAYbujq762mVhNRMpNeT4OFVLTR7YcJKg6mVpPYOxRutKyXXLfYymXZkyV5Fn5mI8sZLvdBCkhM7plo3qk9dlokcR7TVqAbQMrGja9V7IXuvbfDzukT8s+VJS/4iohn53hvZ+Vx9LPvoQ0WRD2C277hsoZ458eEG1ro7LgFkoVuzgmYqEw7FiUw0dQnCdEqDzPeTa67LzFDBhxu82Tt/JL2XtHuv11JHFK8EiDejEP7TvVYGmetKJoOitYDlLg3ZdX2WEbseSbWrjXGRLGcTYcpvepvrqSe7UrSZsLRd56Q3od6nlUmSzhvbDtl5VG/4asVXFyZRwnZNsevflGCc4F1FJv4AXCosuDBo9B2q3USfKQXwRlsBMtWe2Bm8ksmPqKQHTLoO5yf0fHa2EWUDc13vGO5ozQ7k+Z1vtZukk3MvkwqyLbUDftAzgi6RSYg2uwEWc/Fwtjm9ry5ddvJmFbx3u+vU+XF1EyYC2KkLlAsTMLnqfcFRIeekqzLCi4ezCZL6bj+6SaOoELAOYfKy9v2kROd7bDMB/VHhqMe6DzFKVi5IqEUq33V+dpObMjkhvZXJxvUAv7OlRIUP1o4AyEP3ZectNXXol/Q32O9wf0lqr5XgqcJJ2u0mVK0UcpB97QoygdB5Sr1SRIWVKpdwrUwpkunOU2oqF/b/xu+bD6x7WKfrPZv0MmWvEEa07tg/drUjracdRqG7MqgCSqn/6JbtvZ3+pgqiOwdSQeKSCF23wm7eYCj7fsvYSOIs9FUv4udUslxXbwK7SpOOpexAqfc7T6YOvsyO2QygtV9GKWEw4QcwliJhJo526+w6NJ37LOi82VN5sw7lBhD3Xb3JHxHQ0wPPH/D4fpziq+oHvu+P8/ga+FNwAAAgAElEQVTXwPJf/uNz7bG8XI355i/8CNfvOeKjgqaKODlacPbdI9JLTfbnLil+50AYhQcto08j6q+uZWBuPHcP5/zeR/dQxnP0/ySc/3zFL//2CdmX51R1hP9gxEfvHPIjd59zth5R/u6+SKs8vLxvUXs1fpnIACdxmJVh8SAn+adj1vcdT28X3Hp7yZMPT5jeW2B/dZ9vv3MAxhNdCoO3fqth8n7M4i9cszwbMfoo5vkbh5A4pt9K2P7UhrY2DMYVzfsTmokjvTC07275J4sh1mrcOuZ5sk98FtMOHBh4Odzj1v8V870/Nw1dmYrB/RWVdixPR8RT+XVua4MvIqKFYT22qGHLwT9KKQ8Vl69Z8jtrtudDSBzxaQynir1LT/nvNBz9YsaHP3OX2fuG870hxfdmTD6GT398hKo1h9/UIeAgZ/2e4sXxkOxZzONGc/SrCedfazALkffoRvE0n3Lw9iXxd4Y8/5LGXyeSinmrwlvN3m/HvKiOOPwtzeKvbhj86ojv/pwn/7Uxp+81JKcy0H1xNIDUwbOYV8dDVN6SfZCxPcg4+U0oZ4riAbzxf9c8/g8z7Dpi+vsxth7h3vFsUk98rbG55/6vWMoDw9mfT4ivNfuPYPvXa/y3phRvVCQvElZ3a8a/mzJdecr9lKPHlvy84eP/IOX4H8U0X4LBqaLYj7Efjnjz71ec//CINPhQX/vNihdfG4ik7H7D0a9HbO7E7D9xLB/GFA9rZr+ThBCQmNnHDU//UszBdzyn34h4+39qePxXRA9m3yxwVymjRwbdZBxsPYt3oFmPuP8PW9Z3YuZf9LQRzL6rmH9J5L+Hvx6zuasYPYk4/4Zm8gcxxS1PdZbw4FdbdGu5/GKCthE2Bf/xFPuw5fifGaYfbnn280OKhzV3fsXw8usD3v5ftlSHGcVBxPYkJr2WMvbtHcvoD2Pan64YfDvn3t+/5oP/JGPwwlDta279hpMo/v94i/mljPk7MdmFCkFAEfVEqj+ufiwmXmhu/4bl6V+OUQqOfwtWr+W0Q+kdVc9y8pVi/NizfCPhzj+uePKXE7yBe/+wYXscc/1lzfCp+AHPv6FRhWH4TKpZHvxyhUs0p19NKG5b7v8Dz/Ofizj+TRi+qigOEy5+WDzSr/+fa1799Jjhs5jhK8vzv+yYfSemnkSMnzr2fn/B1XtT8suWZN5IyubDhPkXNA9+eYvXissfypl+0rC5HbN8qPHa0w49h99SrO8qHvy9C+wo5dnPj6mnwgzjYfvAk51F3Pm1ihdfTzEl7H1gBfxYmL8pg7DZx5aocHz6bysmHykGF47hsy3r1wa0meb8JxzDJ5pk5Vm+qbj/KxWP/yPH8S/C2Y9H5Kee2Yc1mzsJmzsxo2cyqK5nmr3vtQw/XnLxE3ss34DJI0lvnH3SYkpP/mTB4795CMigefzE8uIveB7+XUszkZ+hi/cMsw8cV7Hh6FstT/4aTD6IOPx2RTM25KclT39+xPCFSE2HLx3X72rGn3qmnxS8+HrO7CPH1ZcM40eeq/dg9oF0H04fNbQD8aFqKz7D+Zspg/OW5yeak99ytKli77fPOf/aMflly+lPGPY+tKFkXpFfis9vc2KYPpJO4HagGTwvWb+WS09h4WjGhqsvCrq+9RsV+ccX2MMJlz80Il1aJt8649N//zazDy3ztwyHv9/w/OciTn6rZXsUsXwDXv+751z85GEPLuKN+PWirbA6+ceXADz/G7clNKaQ36Wjb20oTjI2tzSzDyuWryeMnzZ4o0jmNZt7OZdf1gxfeo5+a8H5V6cc/+or/Chn/qUJ9UgxemFl8J5qBh9ecvnnT1i9JnUx42cV8eWWZ//mPqaC0XNLet1y+ZWUwZlj9LjADiKe/nzC8Tdrso/PWPzEHVZ35V6YvxGRrDyH37zm+r0ZB//4OSjF+TfucvDtBT42nP7kmPzCUe7JcSUrR/TI4VJFsWdwCQzOHOM/uOD8a8d9sJhNFMMzy+B5yfmPDjGVZ3BpSRYtxVHSV8Qka8/wWYE3muI4Ib1u2dwRz+XgvCVet5T7CcMna8zVuu8/LN44IHu55vRn9jn6zpqL94Yc/JOXuMmAyx+ZhXAexeRJKeFMmRHGsRYGsh4bRo8Llm/kzN5foZ+8ZPlzbzN4UbJ8M2f/N885+/oxgwvL6LtXuFGKOVtw8Rfus/+dBXYQUx2kbG4ZRo9r4nlJvZeRPV1QPNxDN47se69o7h9SHabkLzas3hyjnGf0Tz6m+uHXSZ/OWX/xgPG3X7H46m1AMfndV8x//Bajx1v0uoTW4qYD9HxDc3dGNC/Riw3trRnLhwNcpBh/b45PTQ9QvNFc/tCI6Xc26GVBc2tM/GJBczIhWpa0k4zNvQxVixR8+L1zNu8eMXz/FRffuMfs1x+zfe+e3Ndas31rn8GjOXaa404yqVV6/4zyjUPSx1esv3LE6IM57TQnfvSK4iv3UB6S0w3q/AoOZjQHQ5F+Nw7l4eK9nONvFtjUoBJD+nKJHWdEL6+hafDbAvWVhwIei4b2IKfrE41WFWZdCwvZOuwkJXp2SXP/ELOpUNsKNxui5xvcdEB0tRbWcFPg9saoZ6egNGY0gLrBHU7Riw12b4x+dgazMWpbYm/vC5vpPUprqTbZFFLHcXaJOj6A0ws43IfLazjaRy3Wvb/ST4bgvHgtAbc3QW8K8UDOxnB2KYxhxwhqBXmYFapq8V+CyE3XG4giweZlicpljOHLMiwv4M7XNXo8klqSqvqMhLZnI5NEXlO6f657KK3wYX/+WDSXd//i93wOHv/aY/ln9DF799i/85//HbZ1zFuhiuPOaMHj5R5XyyFvHl+wrlOKJqa1Gq0djTUMkgalPNsqITIW7xUHwy0fPz9iOC750VvP+NbpXbxXfPn4Fe+fn1AWCSZylOuEwUSmbkXarnBOMRmWrAJjBlAVMQf7a5yHxUr6NNfrjDRrKNcp6bCmXKYo48nHsj7vVV8DwacD0ncXRNqx2mQY42gbg2s0cd7gneb2wYKXl1PaRcLszpLFfECUtsSxHFO5TYjTlqaM0LHDNZoHdy759OkROKRuYpFCJGwFTUiVbbSkuo4btJbtJllDU0W4jUzHJ3sl9umA/M0lmycT9FGJO8/QtcIeBkavMgweR5QnDjduwSmwivxwS/lqKGxRYtGRxy5i9KSRCo3LjOyooDobYPYqbKPFe1grovsbqnmGGba4ywR9WGEXQepRa5HFHm+prjPwinS/oN4m+EqjMitpuA7QEC0M7VGDKoQx8sbvPJjhPaqW5Fc7Fi9ptDL4+wXuIg0yZ4Pda9DLCFMr2rEDizBdx42k2TrFzXTe5MrQzFyfdGu2kp7beVfTM0O954g2wrr4vYb4WUJ9YEkuDLpVlHcakvOI+rjtfZs+8pKSWxjS84hm7CR1NQWXObIzI2yFg2Zm+zRgtEisdRm8MhPbp8/aWUv2JOnZHhcLQ4WSfU0upZtT5M2O7CyivCVSXm/Ctsa7lFZJUxWpcHwWkyyUdGS2MsseBd9fs2eFtRp6FPQpq16LZLQ8sZitJlqLlBgNw6ea9RviiTWFsG6qlRqFeib+UK9Fjjh4pSiOpb8yOw89pFMHTuR7zcgzfiQBUO3I0w5EYr29Z8mfm555qfY9phR5cD3dySfFDyfLRVtFspTkXhe8kt1vSrXvGT9S2FzCt/JT1UtxlVN9eq/0hnby5vAlE7xo7dBJwu4mJI9GgcHSO7bEheReXQc260JRz8TzqVphe6o92dfhS8/mtiK7gtXrjuFTTTOW7WXnAurqPU96EeSvYZyQzD3FkepZNRcL49ZJGptJYB2D97IeiywXOq9gFx7mGH8qYUC6EvZK18JyFSeq9xsmwZuYrOiDiJKFsF+mlPOUXtPXWEQb+nTbLuk1vfKsH0j/a5fC2vkC66m8biqRiuoWmUSM5HzFK99XHigvXtRmKF7EZqx6L2e6FIlqPRaZr6mCZLYSGW+y8H1vp02hPBQQ14wD0xaAk1RiyPmMCvm+2NzRsh/BB9gl33YBVM1YEa9l+/FaUoqLQ5GVp4sgd56H9NSUAISkP7PzujajIGVfyjZNHWo0tKxDtxKkBXJdo0pkp8lKmJo2lxoTE+Sy2oo/sjjQZPMwkXRL6jqUg3JPmDUxn4s/tOuxrMeq74ZNF471bROYOpExZ1cy6KzHAkq1DeFvSgLrOglzVArj1vk8bwR9Y2rP9lgzOHM94xUVTuouri2bW53XEUavRBbbhTaZxpMuXEiW1X366032yaYSyhUvLet7CenCUk0Mw5c1m9sJUSVSVGUlUGhzKyabS/BVm4kvU5ZvqacRycpSjw3R1oXj1cHCIoyibj3pVSMget5iM5HVNiODV4gMdhoRFcGHGQb5pnG0eRTSty1tLuBYt570upUQrtwQr2V5rxTRNkh58yDT7ZSQid6lxBpk32exeCuHEfFGJLXxspZKj9xIAFCsaYdib4iXIuuNlzXNJCFe1rjUgBUQ72JNsmx211KLUsMr1TOIpmjFKxqLDNZrJcE/VhjF+niIKcQr6BKDqWz/Xd0lx5ptjUsiTNHgEpHl4n3vmfRGCZsYrrfLY/RWxkRumKKXBW6Q9sxlB8RUUeNT6fPs1ue1RtdBSlHVfZhQnwbbJcUaIzLZNBEWsqykuzJNUOutAMU0DtsJYTzWCYMpg8/dwLoDnW2Ldw6VpTu2kh1bqOIYX9egb0hbuz7LDije9Gp2Hsvu+e5nrOvetO4zIUF/5ONfAY/l7N1j//X//m//ia/3F3/2v/xTPzb4nAPLW1/a9//p//ZVXtYzChtzVcuISwfNyFU1JFKOg3TDZTXEodg0idSQKEdmWlqvaawRyWIbU7YRRR2TJw2zrOByO2SSlWybGOs03iuGSU3rNNs65mC45WqbM8tLqjaiaCIOB1uWdUprDXVrMNoxzUu08mybGB30U0Ut6xxnFUUT0VhDFrcUQQIaaQn/aa3GWk2atHjAOR0CweS5qo5IkpYsblkXKdNhwWKTM8orijru12OdxmhH3UQSkuMEbHdhRZFxNK0hSxrKOkZrT1XGRLGlbQx70w2rbcZstGVTJSigKGOOZmtOryZoHWQmTjMYVDinqWtDktgQQARp2lJsU9Ks7oFvFDmKbUKchF688O/u0ak2srwWgG9kn5X2/d99DYwC10h9S1fLooy8r60NynhsGaGMI85a2trgaoNOLEnW9oFDw2lJWcZo7fBO6mlcbUiGNU0VoZTHOyXVMsqjI4cysj0I0tLCoCJ5zmQWW5oeZCrj8e1OukgIXMEqolEj8l8HrowE+HfVMY3egVRFAK6IHNMqkYs2QfPllFTXgADqSIAT7sYoqvPndOuLHKoyIkHdGPH8pg41kNGXtwoajS51LyvVeSvMfavw4xa1NvihDZMVgn5UpfGx+Fg7v4mPPOQWtRFgrGz3Cy5A16XS1SeMu8g7sSocK7vXoK+00UGq2ckfvQGfeJG3Qu8plXRn8T+7VKpW+mvQXY/utISAEpcIUPUGmRRplIQOVcGDPHCYdZikiIKcMUJqfLzsY1f7ohrVy1A7SahIzFTvgfPhmnQSOJuFihcrz7tOatdVooRr2ckEO0+yJC7LMZuik53u1uv5bKVIMpcKGBfT12lIrcxOdtj3KyqR04rMtUsa3g0OXHftggS08/XelF12z+ta/NQi+RWpqUgpBWR3y3ZSwm7nBcjfCI5qd6CoC4ABmRDpJKZd/UV/jAFwmFrOq492ksvunPaf1f7GCPdFqE/prklXlYKT13xEYD522+zvq+4CKPqAJhXkp7ol9PR99joAu3qhll1Fi915L6PtTlJ6M+BJPHa769kFAekQcNVVqfQ1KOGc3wyB6jzjhHu3/8yG4+0qR2xCX2ciDJ4KFS2+r+uweQDoUQABmr42o6+S8Tf2q9uO4bP3g+++U+gDi7rz1p3fm55jF+3On9RZhPva7s5Tf50cfQCaqW4ARS8eY2UFtPaySed39R2dtK/7n/P9PeRNSK+2vq/Q6eqo+tqasH2CvLf/9433SCUNoIIH94Y/rbtfbspTO8+xsmGCMJLX5Lyofh+1lTqRvuInEuDcnU9nus+k3+2TlqAgkabSBwr1/lP/fecpyFe7WhOXqPCd5gMT+tnKlpvP6Ua6ZjvfaHdu+q5K5/tqGqnxCL8V4bp0cmlvROoKoG/IOFX7ffvZ3njtxrVV1vXn2SsBoBIwJK/Jm25ITJWsq7tH+of30FWoBBkxLmyrA6udD9J5AYU/qG6k65iEnbz1pvS1acN5dzs5qvu+/bi5zzefu/l356H0flc74p2AyxsPf3M7f1QtSrev/4KH/z5w+//n8WcBWE7fPfFf++/+5IHlL33jv/hTPzb4zPDp8/dQynPdDnlZTrifXXE7W5DqllRbnqz2OclWAHy8FAnWvMy5PVhilGNVp2zahKKNyaOGt8YXXG9zvn7rY/7Kgz/kx46eMUlKfuLkCSf5itRY7oyWDJOa1LS0ThJmF2XGLC+5O5xTNBHDpOG6zIm1Yz/f8pfuf49RWlO2Ec+vppwM1sw3OfNNzmIxwGjHqkyZ5SWv711zcTkWgDjP+fLRK9K4ZZA2TIYlg1Q8grenS754fMrfeOv3SWP5EGdxy/nplMmgZBA37I+2AjLzknFWYbR4Jn/o+CWjXD6UeVpjjCNPBcis1xnDrGZ+PURrj1Kew70VdRVxvL/kajEkjlvO3z9if1CwfDHmrZMLXrzc497hnLaKSBJLFFlWV0M2p0OabdKDMK0lKfbe0TXWarJBHTyMniyvxRM5qNhcDHj7zhlp1jAZF+SDitl0w3aesz/b0KwSXju5wm5i4tgKkAygwESW27euMcbhreL4cIlWnnqVMJkU2G1EnDcMJ6UAUq84uT3HbWLKRYrWAjjrKsI1GvtqgK3FZzmYFtSLlOl0K4DSKyg1JnG4TYyrDSa27O9t0E8zxocbkkGNySxunmDylnRaotYR2aiSHlDj0etIPHLGk+8X2Asx5CQf5ahCy+9CpRnMClStUKVGJQ4VO/LDbQCEHj0KDGnw5AGoy0RA5f/L3nv8yrbd+X2ftdaOlU4ON737EvnIVieqA9o2JBtSwzIg2DDgiYM08djwwH+A/wIHyIO2rZntgeGRp4YHQgsyHBots6UOJF+Tj3zpphMr7rSCB7+1dp37mna3YAIkG13AxT2nTtWuvdfeVfX7/b6pdOh1JgilCqjKMbvYQOnJDnrJBe00WE19KZkiYeJQZx1mY8helJjXBWon280e7VC1RfUa80WFWvTos5b8Tc702Zryi4LsOsdsjDRthz16PhDmllA7issdamqpf1CKi2+AoAK6lWYtf7IlW0nOaXZvxiYu5F5MsmYOXzvRGeYeTjuKGwOnHeq4J5SecNHha49uFBwMkjVZBNxM6OSogDtwmEajrSIcDvjaoy46/MFAvkyiLHAzQXx9LShwcW3I76WJtdEt1+x0RL2kwOovB+yhwy0c1bWOxZCifmkwvRg32dOB+o2iuNfiGtxExPTQ4SspyoYjNyK1BHGKdUU0UyoD/ZMeV4kTro9NKnrfELhKvoyzbUTiHg3SPFaynfJOUO3+yI16OVfI6/WnTsyXMkCJ5hCg/0YzaiNtJTE71ZVEYNg6yL9oulXeiZ7P1ZL32p14QQEPPVkTC3cLzYVoo5oPu9HUyueiFfRFYPJCqMGuSvmCe81evhbzsqyF3VMbm5YgcVN1RMqivhOkQegXgWwLzRMnqKiVSKW0zeYsjIZbIds3wSDIc8rzzDeM+ZhJX9rPA+2p7M/0y0CxitfEQRDdZHRbTa7J/YEgoyETI62H2sqUITk2UZaI0op5VXsScLU8rn4tDrKbd+S1+7mcSxsbOG1h89zTHUUEEkZXWG/kmFFyn3JCdfW5oOmpCS3vAu2pILpyXclQxBfyGvkmxGY7yGAjCAqbGs9hJqZCwhyQ3MLUEE7eeIZoAORqiZkhCBIszr7qLd1jv1AS6xOguVBj89We7ItiO5HmqF8IcupKGQTkG0F0R6MrE9FtBcNCEM/JtadcBoptoNjI2mwfSyRRe6yo7uW9ZivYPBUnWuWh2PqxgVQ+7AcoTii7KMh3fmx0+5mYLzVnGldGA6ZMYmbaI2nIs6hLbU7kSUmvCKJDDEoeH4sjzBDopxpbKqYve4apOLLaSpGvHT7qIou1w8aMS9N7ydB0IQ4ofDRMkoZgd2ZoD8XILt84hloMk4KGfiauscPMjEitq3TMYVWj9lMcer240zaOEHM17VRT3g7iGBuNmACGmaY7zKJjrca0Pg7jFHrwmM7THefYqRHqqxcXWW2lMbaV3G+nBlsbhmkmKGhsdPP7jmzTYxrL7lwG2kEr7FROmivNaBA0LPJ4TeVou0fYglG4SSYaSb/P7gxG44uIQrqAr3JUK6hpahiDUYJcukAoDaGS1/WTAjcvZR1Sz2XF/CghlaEuxJCnLsBoQpETJqXsl1b4w+nbjZzRhIRSgpj/FDmhKiQ2pYw/Vyl4WAlK+vCWGs6HJjw+SMzKj2kYlVKo9NjkRuul2Qxv/fxnNZdf/feX6eZRP/F/Pyu3n2vEsvrgSfjN//bfZzfkXN/MKaqBflegCycoVO5xOynaUQFMGJEfNWiJkkhT2cbA3KKvCjGRWCn6S4veGkFmQFCaIhpnOKGp4ZHCtFPYmZfiJcYWqF6+AN3UY7YaNxXqYH/sUIMeC9A0mZcvzji9LILETuRSbPtCvsB0REf81FFcZ29NcV2ZEBWJEugPJD7ATj1mp8epq49FaXKZTfehAqaVx9nJnk43ToaNFKkqBsWbVpGvFbt3hPo4LMRGXfdqDG33ZaB4ssX+YCYGHrWnfilUwuSG6oqAj+YG2Wa/7RTtYGdhDM3Omr1xhqvEHEPF9XrIZ3J1GNc3mXJop3BFpGVOJR6jvNEMB3Ies2gAUtypaFiRqJuCdiSkJjmCJlQixU+kNcq2sm8JgfO57H+xlP2SMG15PT1ElKZLX1Cyryq5okdUJOSyP6aRbeRrOb96UBRLQSdS4Wl6KZrzrRqn77A38UgIhp2FMUKjP5B1SdN3OxFTkYRi2Vr+nvYvFZlpyZNhhWSHSiGLkrU3kc6XYgH2WZgR4YroSjLvUA7Q++D6FHSfRQrpECmUIW4zGV0Mc6ivHhxj3M7D10iZov0Bo/GITghaPIZ+IT9njbjc1ldSdPnigRFH2CMY8n7Zv09gvy7pnIwh6JF2qnvZ96xhRNBSY5Oa0xSNke3kmmmP5XrRwwNkJdJdxaJfrk2JJRHKnekEKRpjPIJQ5FwptMx+Lm64D51TXbV3WzXR4VW56CI7e3CdRpQR9hP8hPyIAU4Y4zhSgHowsk8JjRrRwDgYspPoRDuXwljZROWN12YrdE4C5NEN2BVSCKZj1TbEInuPRg1ToeEVm4AtBbVMhi5ZE8Zgejm4+HXRCVU0aHHnfWgEIk6damz60uuMyE63v8bT30wbG8l4/hKCklxVUzZmHqmfpguju2hqTLJWcmpBruOslf0Q59VI7UzoWUSnElo5XrMRlU2GM64QMxf5rFKxYZHHaStNuTfSlKdrJGWjmt7THpho8CL75UpF1khMR4ojGWueEe0iOsRK0auCoGoPjXQemsEImhzeQrmC3iM+Pl5XrpQmplg57ES0xUn7mNbblXKilQ8jOpqaGHlPSdzHMM/25zA2bYKWyfViWnFOzXd+RK8EzZTvhWROk65dFcKIghFfR66nEFFHtaeiahVRU8nAFBfoGBUTiGYyZnRfTcig8kKFzVqHreLfY2OXngcQMoWJTZ0rxKk2mc7I4zWmdW+dt2Bk3ZX1+MKM75dgFPnGyvvYKEHdlETbBKUwrcVVmVBPc41uLL7MBDFLSF90qU3Zl7joShwpq2qQ+A9lvdBRtx1uXo2IYFAKHemnvszQvVBY1eDByDHpdsBPCnGLHdxIbQ0m0k1TX6MZ0cFkqCP0Uz9CMcr6vQFPQu20JpQG3QwErVHe7yNBUjPngzSFCVXsB5K5z+juGv+XA3uAQI6OqQ8asAfIZSglFxMQp1nv36a72tjE5dl+v7wnJARUq7eQS5WMiRJF9aEuEwSZTDTYoSe5woa2e7ANI83jVx1oYWwURyrsX7QX+UtAhT34xkX4l//hv/sT3+7/8q/+Vz/1Y4Ofc/OeWdFxt6v5N9/9I36/foe/e/FH/MH6GVkU/bxsDvjjzx/x5OwerQKvl3PeOb6j94Y3qxnfevQlH9+d8duPv8f/9Me/xt/7pd/jf/zOr3F5uOF2M0E3OflBi/18inm641eefEmmPL/3o3f5+uPXXO+mXM7WfLlasFxPOJi2uKDYbSt+4elLvn91ijGe09mW18s5AN0s5/Roy6zsOCp3fLE+5Pa7J0w/WLK6m/D40R0vXh5RzzuaVcXloztB3GzGyWTL6/Wc1aomzx1uPnB2tKbKLD/69IzpcUNdDFy/WmAvLTooqmnLyXTHy7sFAH2bU087Lg/WrNoKpQKbpsQ5jbOG2axheT/h3/iFP+F3P/2QqhjYNiVZ7siNY1L2vHh5RDnt8V6x2xZ8+O5rPj045sPzG75cHoyZl9+6/IJ/8idfZxgMv/gvfcL33pxzWHdcTxao1lBdbGnqCflhi+syLs6XLLc1/bpE556wLCjOdkxL6bLWmxpnHEOT8/zJDdu+4OaTI8LUkU97oe7eV+iJ5fxkxc1yim1zlPFiBHTa4e5KhlOPmQ24NmN4KkhpCIr+synVe2uUCjTrisPDLXfXc3QR3Uh1oG+yPdUU0Zr2y1L0iltDftHQvpjAWSeRN5/OcAuLKj3NNOP0/VsGa9g2BTp3eK9RAZrrGvJANh1QP6oJFej3tgxNztNHt3zx8pjQGurTHZNi4P5uSlY4utsKn2n6C4sqHVnhaJocU1n6VUF+2DKsSnRt0S8r3GXHfNGwXtZkpaXzGttmmNIxvFQsSTcAACAASURBVCpxtUdNHKE1mF/e0rY5z89v+eT7l/QmYJaGp7/8ii/++SVu6lG1o/64ZPOBpT7d0d3VdA/otfl1RvOORQ2a5x+94kefnNNZLYOWSnSk/QH42mOnWhxGO9Gqzp6uWF/NUIUjOD2ipWFq6Z8D20yGLicDLHPIAv0RYNhTXwOEw4EQ9bej5rXw5K9yum9KjmkwoM47XGdEo7nM6A9FbypZZ6JlDJk4vc6/XbL6hV4GUla6SBPjhhLi5U8H1K1MuP3CMj/dsnw1RzmFbhT+TAIrs6iDpteo0stAbJWTLfcT4ZAHgpJBkSvD2LAkRrPc7/EzR/E6k9zPw4HsNsMeWnGbRTSzANlVLrrPApbfdDIo017ifh4P7O4yfKTzutpL/NGdoj33FE+3DD+akTWiRZVBCqA9+VLTPrIUt4bJS8XuImCPLGYVM1/f2TBc1eNQq7iNea5xWJWtBW3KGlid711d8xXs3h0wq4xiGYcqLjboU0HG0kBwOJChTnkjzWJ3KgOvfKlYfSAXhXJKhgq9FN3lnaBywYjekjjA0IOiO3OUV2bMpewOA/UbPQ4+TJ/ihjz1C0N7LusolGehCuOl2eyOZUAj+lNZN8kx1WSNIJ35Vo96TVfHY4pZp/lG9Lj5Cihg+1SNVF4ZJqaBSUTiBmnWhplcR1kjf+9O9vrjYQb52pA1okvNV4HmUo2xPflGGs3uODB5KXEszbGiP1Qoq+lOA9PPDK5mHCwEbXAFLD71rJ9pUIKm2qmin8t1m20F+XW5rE2xDnGAo8emXluJBpq8DFEbKL+Xyz3d2fRhRIt9ERHJnR6HQTIQfDBgyWVY4wuN7vc6VEHABHHdPCnEsXcX6A40zTkcfqxozmM8l92/brfQuEriWoaZIKOzLzz9QqG8lsc+oMqmzFJXQnUnuaAPKb8pC9Tniu5Qke3MqGd1ZTo+GQBNrmB3ZiiXgX66b8yzzrA701S3gWGiooPvftggjWK2pybnGVmThp5y/etBx9gSuXZk3/yo9RxmivLeo51kY0qEkBO3WSNDAdN6hkWJ7gLdUSVDpUUuje6DWBOI56hQ0sAnimypMK0Zm3XREksD6o0CLTmKIdOxuROTqu64FGfZVqigwUC+yRgmGcGIFjRFltjakG9M/EzyuMpgYlZmtpXYE9M6tPW4OhuHI8Mso7jrxybaNFY0n0hzm4YG+EDIDWbbE3KDqzI59rsdYVpJc6fUvoH14vGgu0Ea7kFyMEMtzaJe7giTMmoyi7GZVyHgq3Ic8inn3m5kk8srQB4be+f2aOJX6a6Rdqu+giSOt0z0nqFpIC/AS5Ot8phtaUzUYGpxj9Vaji9tN6LgIUanqDyTzMs/x5wn+L9gA/qzfAv8lXnPz+qtfP40PP2P/hMxCXGJdy9fJhCRozJ+kSdUYNjrGGwddTR6j9ANixADsffmF75E8ggzQQWFMrK3p3d11AMF+QJxJaO+B6LOJr62IGlS1IwaFx3DvmOhku1EewJRK5LtqVZZG9GRqK1KiEhC0BLykb5A862E0qfAehHix2NvGXUaQcXX9HtdzjAXhCxZvrtSkLNk+677iBZ5ea7poLkIVG/UqCUKap9HaDrZh2Eqf5t9EYuHaNWvvCAoJg6jHppuJMMS84Cylkw80v3yYR9t7pHfx2y4PobU6/3fXS3ndVgIajOiDpFiJ2jRfl3T9gT1EvQmIVauYtQkDTMxIQFBv+qriCr4/XlKmisJnI/6tFr0dVm7NxBJUQfZVgqgrJHnBgPdMUxfxMJhn/ss2rqIWAnFLSKNSl5LzEDUWGClokxQb/UWepWKs2wbovZLiqJ+Hh+XS9GcEBxt5eeEWqVr006kQMm3jCiBzwVlUF7OT3e0R4bSsYuGLV2DIRY88ThHZEAm+aaX19Ej0iXFTYjHSxA06GF8g7Z7JNXngq65UlEuvSBSMKLlCQFN7z0XKXQoxjB2V8pjTReRzXiOx4zRiFK6Utwp+7miuttnAGY7WdOEBiSb/ocaphTgnsLPk+bMTqKByyBaqESTE8Rpj4AGzZh5mO4fNWdOGrvyLoW7S6GZTFiM9ML7SArNuMYpjmBEzhJTYNgXrEm3pW3A9BIan3Racr2oMa9Pji3meE4V5SrsUdoHhisJ2QMpfLsDoerp+B7NdgEzyBpA3F+txuI9rWnQMEzkGPOdoFPe7B+fzvdoCKPlveozxcMaYYyu4AFS7SQ7MV13yktRZXqPLSVuwT6IlRgmMaNx2B9z0BGBTE1UagAiKmkaiWyQ61FR3gvSJYU+kpkIEUmMaIFjr90LeyRZebkOBd3WI6rmSkG3EsqVtHHyflTj+SaEUWeXjkmOPYwo1V6jGrcRNXkpszEhZIJyS7MxahIHzzDNxsiIRM9TPsR4BsYmytZG0DfkuT7XQt3zCCXSBrLWMcyySMOUnTadi5mJkpGZ4i6EfSHbDEaRtQ5vdPzckjXKtlHbnkuD6yp5vEShCGU0mduMKOLDeIlMUENl5Vh9odGdI8T7Et0y5HJ/am5w4c/oOLVNaHSMsniAQurBSaPjAtp6fBbROy0GN7qx+EkuzVFCLGOmpN4N+DobtYfKB9lWYMyr1L2sodkOEHWCQetRQxhyg24tvjCiVbRenFCjgU2K2gilQXVOmqVE7fTyeRxMiu5Ix2xQzu1jOzLJW036vqBU/Bzfv0/TfXhQnTRsakiC14CyDj+p9iikUiQDHGXdXouZZ5I5WRViqhP1jWqI0SRa77eb9I/J2Obhzw8RxIfRJYMdG7O3kMdEMU3aSuei+2tEJB/eF/8fcyOTNhIEdXz4s7XSmIKgq7DfH2Nkf/gxaOPDSBH9Z51g5W8/pvf4Czq+hq/oOP9Fbj8TiOVHF+G3/uG/9xPf7v/6r/2Dn/qxwc85YpkVjsO/dkOZWd7czbGD6PWUluzE+1vpRuwyx1zuGNYlD41O8nlPtylAB1RrGM68mI3EJsJsteSyzSz0kVNYeNTO0OcevBqpp7rX+DJSOzQxG0++LEMlI2XVCaKRH7a0V/VoPGK2Gl+J/ojKo9YZYW7JrnLskaA9zAe4zwWNCRCMOJL6Wl5TDUrcTAOYjZG/dzLBHg7deEzJQAQQqm2ke/oikJ01gm7tpDDzE4fqNaHwZMsMV3vJt1sZ7EJQjvxOi8NpRCA47diWpWjVejV+2A9zxIDDCTVUD4r180jTbYXeq6zCzfx+TXIpSM1UYc8GTG1Rn9exCQsMByoaSkTaaYCQyWR6mMnxuzKiE9GcJFEpgfiFL2YqKUjbHjjye4MvgiAg93o0ukhTZZ+lRlX2PTmmjrl0W0EBhkUsCo1QbwVZQgxxokunK2Mj0KpIDQ24ep/9FXcTdSIF6TCPhimdYph7to80+Raak/2HdJrky8AFMUKJGXIJpfJGaMT9QqjFxVphJ/J8ZWV/pViV/WrP0nAkjEHtIYWWt6CCYpiGuB5qpEwLJZiR8it5fXEfkplKbMBcJc/PGqHmahvpYKUcW7ZOFDJZ/0Q9dFW8/kvRr7k8fRHLWrs6jPl63dEeyUxDnbdMarzQuF2tR0OcfiHDKWAcDmgrJyY1fYmqGLI9PTQNcLKdNJ7JkMVV8ng7FerjMN1TzvuFGqnS6RjyTaTxdmngs9cK7gtIOQ+bZ4psK4OvYPbU2tHgJDYOrVZjQ9+cPxhSZYwNlx6i1quQ69NnjMWX6WUgI4Yjieonr9OevG20MxbMXg5IBnIy2EjDg6zd68qKNTHig7H5cLUMCkZjm2HfUCdKrfIR5YtUV+Xk+nGlDB6E5inTtaCluRqm8tlgK0FDTCvN/jDVo8SgV/v9l6GWGqnItlZ7qmie6JpqpJ7m8dx/1XBlbJKswWdIM6z215YrFa4lFtGMFPRiDd1CjdefrE/6nNX7UPoB/Im4laZB5zARXZnQS6URMl2iP+6vkWQik3IgXfmgaw7APDVR8f9oVjNShSMyl7SLkBquaCCj1DiQAHm/5rt905uQQ1vtDWeyRo3UWqH7x4FZJu6rqVlOAwdg1HkGo8ZjePg+SEY1Kii6QyMU2izSRl0Ycx5V7I47k437rHxgWGTR6ElhKz3qEl2p8YUeh3PikqpHxDRdB8PMSHZkdBV2ZUTYIGrviM2fxtYa05lIv39gvqOBKukn9p8ZDw2pHiKV3kgWY6KeBpNLA+sD2c5JA9wZXCHXvz8oxm2YXqJ0RqOc0ozrKfrHqOu3ARVzJV1pxgGYq0wcjOyzIdGiV0w04Ycurrp3IxoXco2KDrBpQP7QBEjlQmmVGsfjpvnYBI+3RKGOAyn/4LIet+k8oZIvwqR1RClUN4ir6uAgN9Icx+Y0xAYxxPMR6kIa3bLYD0eS/lApfJFLg5xuiWpqonYy7XNqNsf3XhCk0TrAEDKDepCDKR+G8fHhwb4/NOjxHvJ83J4azYAerFOIyGLan3S32v+c1nOksqf9jc1mSChobG5V/P+tJvOrDrA+SDP7593+EsSNBP4KsfyZvR1/8yx89A/+Q+ZFR+ey0f101UrsR50PbPucKrcMzhCCwnnF4AzOaYY+o6p76mIgN47r5YzZpEUr+Oj4Df/7n77P0fGGu7sZSgXOT1fcrqY4Kw6n9aSna3OU9tTVgA+KYTAY42nW1RgjsltVmNKhgNm0ZbWuR5fSrLTYPsNknrywNLc1Zmpxq5zypBE30uhaaxv5sNCFUB6DjzmUVpNPBoY2o5r1dNtC6HStQZfyOKUDvjOozBM6A16hp4O4juoQ6XxIELwOFJOeblOicze6nyodhLbX5HGyZ2A2SOh94WS7iW4YQ+OBMUoiNcTFcSturE18bYXQAGtH2MVtFNK468LJPg5KsinjY/VswK+F/oiK4fW5uG2GqYNBaFpi9mIEwZp5yNOHtpjghCyMX8jKyaDAl/Fxvd4X0k0cMuQyKVVOjfrUdMyilQ24Y4teZbIdHTCrGL0Rv7x85VGDjohlGBt+FaM4pCnXEVFTDAsxmEkGKkJbjMVjJa+dXBG1lWY1OZXuNXhKhhQ7g+kUQYuJTbYSy3k3k4iRVJSkb11fewmXj4wAVzxcr/26+TJEDZwaIzZcEffXSvSHL4QuOhbKOpBtZF1hX3TrnjEGZJhFJEsxul9mjaI/kqY+OYkSBLV1E7kGkhlHovP5XPS9xf0eRQEZUORLHZH0QNZKow2ylj661fqozR1miZ6ZGogQw+elyFWW8fhScz5GrfT7BiUF2XfHnskrHUPk5ThMJ5rWxIqws4BJ+tbYsICsk89BkPS4ToZYqAJBmnSfMbqmgjAwinvojmTQMbIR4v4nvbNpGVHmUQ8aWQhpGDA22IVsM5naJD1parJtLfueXDhNF/c1HktyHE0xIPlKtpka5oRgjyh2pDgmbayO5/gh+i7nMF0zxKicfcNLHFDYWpr3tH3lGVkBKuzZKol1Ekxs1sL+XKRb2r+Hg6wR5Y0siHy7R6iSm2tyIE0a3nR9JIQ+IcKC8qo9bTE6Wyamg2yYMapEkHZBbiX+Q5Bx5ffo/Yj+uj1rIu3jQwZDcg5NLAE9pCEXIyPi4fGi9tdLuqUYk5Gp00f0N9ah+fZBBIjer2HSP4/OomF/rokDr2GqRu2nLQU5T+6taViTIkfMIJ9XI6qf6uBIb+0X+xiWfCefo66QNehn+8/J0dUzIq/pvKT91G6PaD90nJXImD0qHrQi6wS9Ts6rSSeb9OLjtR+1p/pBk5saW2UZ3X0TS8B00hS6Usf1FPQVGLWiIVN7jWc8xw8bt9R8pvVMOuZg1LhtNaLgex1pthPEMrFLHmp/E4Mhub3qYY/OpjXVQ9pvoa/6QgsltRSkWTuhjbpSYwYf37dqdIxV1uMj6kwII3IsJ1uh+kj1jNrSEJslPQhqqJwfaa2phkkI8BijYf2efhoiLTdSU4m/jxEiuSCvaBjdWoFRSxkdYx9mQY6oZDLReYiaej+eJ4wZ0dCxkX3gHKucl/u9l8el1x8kN/Otm/dvm/M8bIRTM5rQVdgjiBJXELcRP+fSfV91g/0xmsu/qN7y590VdvHRRfiN/+Y/+Ilv9x/9rf/yp35s8HOOWO6WFVd/cMHybj9914N8EWwfK8IbyT1rfaRdniryVWDWyJdZ1gSa0wlsAnfPFCff97RHU9QO/ujglFkO6r7k0Vqmp92kYj7A7lIxfRHweUVJMvlQFG2g6mVif/HC088LinVgFqeP20cKdVdz3EnWl8sVwZRjlpmdKBZLL1NFG2jO5tRX4pqXiqKsgfakYPJaqFPlvdB0XJ5HalZOdxBNYWKmWh7d7LxRlGsfKXCKrE1FiexfH7UmPlNol0cKXE4ejRjynWf9pMTnUN5L1l19raNFvsEMsHpHc/BDh60ldP7u64b6KlCuPNnW0x0a+vkUO1HMv3BsHhvK+0DWBrqFoVx6dhdZzGaDYpWhXaBbaOwko7rxtEea+lY+dIdadEqTNzIVtqUi6xTzH7VsnlW4Yp+P5jOpJodaUa4DzYli/rnDF2rMmSuXgWIV2J3lFFuPywXVq689ysP2wrD43NLPNcXS0Z4Yslb2Q9ArFTUZgelrWL6bc/R9oZD0MxO/2GWam6bxPlOUKzdS4ppTafbyXSCYMKIetpRqTVuZzHcHiukbTz9TY4RActRzhdD6ypW4+rlSsT3Pmb1yMl2uZMpe3UrQ++Yyi68nBWo3V0yuXCzaPNvLjOrejc9ZP8mpbx3DRI+5eLOXA81JNha/0oAqyqU8Lm8CQx3Id7KWthKUc/rKMkx1RICEBpd1AVt5oVXWsUiMdEAz+Ghpr2S7W9HQdYeafBuEohbpfQnJcIWmPdTMv7T4TOiBysm+uCKMVDfdy5d0e2wEyd3IJD/bOdrTnPLe0h2YWNzJNQ3QnGZMriz9Qqb49dVAd5iN7zttA1kjToiEEBGhgO4CKrjRSMVWUkRV1wOukn3sDjO09REdU9RXFtP7kbKX6IHby5zZFz3Dwsi1tHNy/JFql28cdmJG0xJXaLmO156QQXnd0x8VdAea6cuB/iCjvB1oT3LKO4srNcNMi+nPVArgfCuujIBk6TWefDXQnpWRfhyobizdkeTtmc7hM40rNPnGYqcZttZkTSy8tIqGIorqZhA918TQzzV54ynvJPdO9+IEOXnd4UqDKzX52oqhSSOB9MVdj4tojqvjcXcSDTDMjNA8a01xbxkWhnzt2J3nVHeRNjlIHl950zEscuxEEJfifmBYSG6f0G1lGyDUSm0D5U1HUOIs2R/Ie0Led/Ler990bJ5VTL9o8aUhv2+xixJXGvL1QHNeUt4P6NZhpxkmUj99oSjvBvqFfN6XtwPDLCPfWTF0qc0YhyGmQE5yAtd2pFMmUxRf6DETMDUQehCa60O6uh4CppXj648KlA3kqwEiIqWdR/WiP0v01/rLDXZRYaeSTQjgS0O26vBFhptEl89SMgurqxY7zXGFZv7ZEBtWOQ/DvBBUykq2ox6EejgsCrKtlSZgNFBRgkwNbkTqfJEJ5XOaY1b9aOYi1MSoj9sOuDoXOmdpWPxwiLTOgG5S3mDA1RmzH7W4Oheq7CQju2sIxuDrDLOL23FS8PvCYHYx73A34GfFSP10k5z8eifNRgjYg5LJzoL3cRsBtCK7b7CLCj0IvVM1PRiNL3N8lYFWVK18x/giw2w70SfGXEUVAm5eUsR1SpmKhADWMVwekL9Z4+cVyaRGOYc9qsnfbPDTUo5j0+2potNKzGziLeQGc7/BXhyINvFmg1/U4ALFriNUOWrbRkSwkDzH+y32fEH+aomfVaIZnJSyj6sd7miKXjXksakKxkSaaU6xafGTCr1tCFVBPqJxMeex64V+Gn9XzkPXE6b12FSFTLbHYAmHc3k/bxuhj04qyXvseozW0qClhkopcF7McjY70RJOKnTT7RvFLCJ3dYnetfLc1Ow5H/MexelUVSWh7VBZNjZgwTpB5oxBZRmhH+Rxmy1qOiEMw0iTDQ/osm/pD7WK24vfEyljMi8I/iGl+KvTsYiaD1b0kunuGBuiHjScYwP4UEOZIkHeQmXdvlF9SL39yu3nF+b6q9vD2891Y1ktOn7zb36HRd5S6553qxt+o/6Ef7z9Bp80pywyQQy/bA55uVvw3mSN9ZpMezSB+77m/XLHzua82c25/DsSTfG3jr/L//zyW+TG8TdOvs/OF/xwd8LHd+cc1zsOi4bDouG+rymN5a6b8MO7Y+qqY1G23DYTfuP8U140B7w3veFle0DvDS824s8+K7oxM/O43tG5DOc1uXHshpyn0xXXzYxaBS4ma+7aCXdtzWm9Y92XPK63eBRfm73hn968Q2ksR9WO17s5ISgWxnLX1rg+52i2pTSWzklO5q+cfMk/v33Mpis4W6xorIzKtQo0Q85h1fCdLy5ZzBseLVZk2vNmO+N0smU7FDTRBOj85J6rz855+tuv+f7LMx6dLlm3JdtVzexvL1lta/LM0bc5X3/8ktt2yiTvuVseUBcDfsioZ1uuro84OblnN+T4IeNgvubFx0949u41r24XvH9xzaYvyYLi5uqAi6evuXl9ysXFNZ/eHPPe6Q3f/fySo8fXvLk+pCgcTw/v2Q0FJqyYZIJW3+1qqtyy63Kc0/SFZbuusBFVTtEjnQ6EAO2uoJr07K6mlMcNQ2GZVx33f3rGwb/zhtevjrk8W7LZ1WgtRs9NG/NHM0/f5dQnS+bAesjYNiWTquf9oxv+70/e4fnjG5ohZ9OW7NYlRT0w9BnPzu/Y/aPHbN8fuHx2y5vrBYtFw/2bOZdPb3l9dUDoDLPTLV2XYauBzaZC6cDBfAfA/XJK8Ipq0nN9W0PuyUqHXRXc/7onqyzvnN9yc3fAzWDIcses7rhaTtHaM5u2LFcT1oXlZLHlejnDvjAyPZ9a8smAc46itLT3FdmdYvOkJT9ZcnVzwMXJkjefnKKmltAbqsOWoc/IcsvQZ8KyySQbNdwX3B4JxBacIOOmcCjtGXaFUNe9ElQ4QH3Y0m5KQeEzR+g1ZmJxncHcavyJReeeLLc4a/BeEe4K1FFHWBbc/jqQO9RG0GE1dbDRqAHCcZz29prZyYbt53NCQsm1Qu0CHHq4N5TXmv7Y4xcRLVdOEPLKCeOgNcyO12yXNaZw8EWNnQcorRj+AKH06NLBjUCE5dMN7YupUOdzUDuhoWMs5i4DH52otVCxwtQJUj6x+G0G+cAbp2EA04BP9L+Y46mcbEOfdvjbgmyrGSKlPeQBTI7qFGqANzoXM5hzQ36tcZVc26ZTDBc9WelwNyXKapTXmF2in2rsQqJXXC1rZ7ZFjD1RBG3AgN5p3AJ0Y8i2iv7MC+q8Nfi5wyw17sBgVkaMg+ae/NZgZzoa7giiS6jEHbVRtKca0+/p8UFXQjHPxJm7fJ1hJ2HUOYLElDSPhJFgOhPRpEwQ5zpEmr5Y/iZHZmWrMavTG3ENbn9JonFMzE61E/lqzVeKchl15QW0p57yTuNNTTBw80u16PlNgWnElEiFjOpK0R8KSyFfiyFRvpQhWtZko4a3PzC4KpBtMrJGjZVZdxyobmRwk+0UKuRjJqatBW0XA5hi1N87Ifow/TIIndnu9f7dcSEU9jjctBOhSFbXoovXVrT/5V081t84eqAHz8Vte6XwWYXPxcinO1ZMXgnSePe1GVkjA7D2LKO8VSOamm0D/aGK1zQkJ3fxJShH7X5yAE9ux9kuonbRAVi0xPXoD5A10fm4A9NVMuyq45DOFTL88Yn6HEbktT8sSW7A5V1gmNV7tgJCVa3uA91C0EZXRWfkIJr25lQzuRJtaP+NMtLjFdWtZ3dWy3ZXnuQa2x9MxkEryD7mG5FQuCIZA1Xj/g71ZFy7h8jqMJUBqc9n4tsQHYCzLjC8X8nwwwkaKYNSRfaoxJYyoLTv1mPUi+keMEksMdpksdd2fzgh33kZCG19RHsXUWurxLTn6zOqG8vu8UWkPstjfaFwxUJo2mb2FsLrCk2xtLgnc1yt0cNi1LUm5FrcbvWoHYY9e0U0t1piVVoZHOVbK9fTbqB5Psc08rnlin0ephn8qI1Ngxuzs2AO38osDUaR37cSK+LiOWQuAwEfxuGGmxWYXS/NfxxCSBxJ/D0w0mhTo6ZaCweziGKLFjbpUUOm0c2ALzJhYjSDDCwS+vnjTHyiMU+oCt4y9ok3nZDMJAVIqOoDqm6i0gbvRbNpLUmbqQDCgxiRt5rLh9Tb8Gfv+zG3t1DM/x8ay5+FW0D9FRX2Z/U2/+gynP6n/zEAl2dLlruaX754wR+8fEJzW3N0ueJsuuXj7z3m8MmK5WcHXHx4TQgKoz27rqDpcoYuQ7+oCE9bLk+WBOCs3vJHv/8e0w+WZMbR9jnDYHAvJ0KtOe2ZLxq2W3FW/dqjN3z/9SknB1te/+kpoQhMz7d84+w1//Q777E437D7+BD1bIcPCrfLUDuDPukxP6qY/OId212J+v6E4Z1upF2wzmExUE87mi/m6LMW84Ma9dGGy8M1N9sJu03JYtGw/ZMjhsue07M196sJ2fcmmF9Zsr2ZSPC9Dujck+eOdiPfUjrzKBNQX4h2sX/Sk10VUhTOpUj3Vr5BQmtQvWL2I8Pmuad+pbG/usH84Qz/y2u625ryjcF92OAGTf5ZSXWr2DwXbaZa9HBV4qeO7C5DPd/hXtWEowE2GfqoJy8s2e/N6f66OKLq+4yDD++4/+ER1RstGXfXBfrZlvCjKfbQYTYa/aRhWJbonSZ/uqW7qzAbg5t6yDz5VS4UzW2M/MgDp7+vufpNJw6hV3ksVhTtux3FiwI7DWRrhekUzTsDeiM00uF8gEGj2xgzczow+14hRcChFKD5GlZ/bWDyw5zmGy3TP67YfDBQvs6o36gxdqA5D0y/UCx/qyVsMlRQ5Lca5RX1a9g8j3RWJxpDNwkU94rtM0f90rD7esfsOyXNhehrfRkIWaC4NVIIRuOp9txz+F3F+jnYmce0QrXtTxx4OPhORnsqPQgfBQAAIABJREFUtMfNu5bF9zJ2jwN27ihujezzROiiw6HE19i5o/4yo1xKbp6beGY/NKw/sBz9oeh0h4UUyz6HyUtFeyp5i+5koPii4OSPAy//zsD0uyXdibhOugK271mO/plh8xzql4phLnTO/iAw+0yz/mggu8uYfapYfU0Qy8UPFKsPhLo66gqd5Cg2Z4H5p7B+V9Z9/ik0p2qMXEHB7qnDbLU4VMbsxvIusH2i6M4tp79nWH0AkxeCNg9TRfNIsiAPvyfaLDHHCtx/I1BeyxpnbaQjzhWmleLSRSOXzTuew+9IU5bcUE0H26dhpI1mOylQD78vBdryAwg5+6iYQ6EFH34Mq3flWIoV4COz4IloXes3UqhsnkF1I3rgyStptLojxe6RZ/apHnMspy8CN3/dcf5/Gu6/Jvrc+krQ6P6AMYMyGYFNX3t2Z5r2TK5RWwtTRJgHgbtvaoo79nrQ54HjPxbkXTnoD6G6Dqw+hNM/CCw/EEfN+k2gWnq6uWb3SNBxV+zNwqZfQHUfuP9QU90IW+XgB4H7r0N1LcV5fSPsg/ZUXmv6ynP/oWb2RWDzjmL6pTQOk2vP8j1DcS9sl/J27xbaHyhmX3p2FzpmSYbRoTNRhVHS8HWHUjSUt4HJjcMVms0jcenMGy9MjTvZ19kXntX7mumXwlrp53DyJ5bNYzNSiVOOocSrKGYvLT5X3L+fRRMuoUDOXjj6qaY/UEzeSAZiMiDLGjFg2j5SZC0sPnXsTjXT18JM2J1IDFSxDiNDQFtYvWvEwfRKjsW0gduPcrSV82p6yV8sVoFiI0Xy8r2MxaeCWPRTTb9QzL+0bC/kmMqVp59p6lt5zOaxYXIljIHthRmP2WexyYo63fZQ3iPaBqpbx+ZxxrBQFPeSWVqsQmwshZE0ahq1mFslffD09YArNN2BHvMgTR8Ebb633H9QcPBJP2qqTefF3Kd1LN8tmb6xbC8y5p/1NGf5qCVWPlBfWVwVdZF9wDRCNR2mOjqeyudScd+zfVJR3Vl25xkHH2+5/2iGGQKzz1vsNCNbD2yf1VTXAyoE+kU2si0mb3r6eS4uqKcFykJ519MvcnwpzVt3YNAOZp+saR5NyRrH7rJg8YMtu8c1rlDMP9mwfWeGaT3FsicYMe7J71raxxPyVUTXck13mGMrxeyLLhp6RYRcK5qznOmLDt1ZhsMS07lRSxkyYR64iGjPPtvRnlVUr3c0j6ZMvtjQPJpSv9wSckN7WlG92eFmBf0iR/eyb8OioLhq6E8qijtB+YvXG/oLsT3WnSO7WuMPJrhpHs2I5JpsTwvKG3FxTccXSoPe9CPyOZzOUM6T3TcMxxO03Tf4otH0YyNnbja4oylqiPElVYZet4KSAmRatJm5EZRZa0KZCTJbF2Mdm5BatW0J0dUV7wlFBiGgOguZQW128veuF0R1vSPU5YiiKuvGjEoV6bOhyAWh9V4cZL1HdcI2GJvJB0ZE8mT1NqXVGGnkshi/M0QXV6WkaRx6cYW1FpyT5jI1kA+3EZ/31bzKH2sc9GNuIxob/J+l7v4L3H4WqLDzjy7Dr//XP3kq7O/+7f/ip35s8HPeWJ79wkn4t//7v8svTl/wWXfMaQzIK+M4+nevP6IwlqtmxjcPX/Pt6yd86/RLrrsp06znUbnkk90pH81e8z98+7f4t37pn3Hdzfjnbx7xN598wvdW5zRDzvsH11yUa/6vq3f55eMX/B+vnvOrZy942SzItOek3PLx/RnP5ve0Nue2nfDBwTUvdwterhb8+uXndD5jNVT0zvBses9VO+MXFi952R3wv33yAU9O7zHa882D13z75gn/+qPv8t/9wW/x/PENT6ZLfrg65oODa6w3/OGbR3zt5Io/evGIv/HuJ3gUf3T9iEfzFYW2fPtHz3hyfs/drkapwIfH16yHimnWY4Pm3ektn+2OsF7zbHrHTTdl3VeUmUyBbpoJJxEdLY3FBY0mcFA2LLuaq62YIs3KnrtdzXvHt6y6isvpirt2wjTv+HR5xK+evWA5VFw3M2ZFxyzvsF7z5eaA+82Ew9mOdSNj8tPZluvNlKeH99w2E6zTTIqBdVtyULf4oHg8W/Ld63NCUBSZ5Wy65Yv7Q4bB8LWLK7ZDQWszdl3B+XzDsq24uZvx6HTJsqno+0w+j4KirAaq3JIZx/PFHZ+ujri6mzOftkzKntv1lINpw+A026ZkPukw2uO80EqaPmdetyyKjlfrOV2foXXAGE/X5mgjDXzTFBzMdxgdKIxj1Za0TSHIcrx/uakJAapqYFr23NzP8F5zdLDFaM+mLTme7vj8xTHffPcln90doVRAq8BgDUOfkReWPHMyMGkLssyz25Scn66Ylx2v1zPKzNEOGbOqY91UbG8mnD5a0lvD+n5CXg/0u5xq1mMHw2K+Y72tMCagVCAERbsqmR3v2NxNUJmX7Ng3E0LtWJyI3e5mXVFNeoY+Y2gzstIRgNPDDXfrCd5pbJNRzjuMkaiX5rZGVVELogLBK6aLlrYpUFq+bIwJ2MGQF5a+y/eDzkY0uOawl4FN1NEGJ3rOrLaiVe4M1aLDDgaTObrrGkqPqcR5xjUZprZ4q6XJn1pM7gX1dNKwBStuFeYmJ1x0+MGgjKeoB+wgmu3dqsIUDrssRnMv5RTqsBe9cObRmccPBnpNtugJgFvnkq+be8iD6IQrh35TSjTJNiOUTrJ4c4/Ko146gBo0+b2mP7ViElbJ1F1FLbbaZIJKAuiAnljUqxJfB8rzHe1dJXpnGHXFWI3q1fg8NSjZrg4Ur3PRv2oZ0iinYGphk+11zV40wKFymPtMNL9RTysQmRKdauHRG0OIjtv5nZac2cpjGsn/JZfBiWqMaDWjHlm3YpoWStElm23UTYeYh9spXOVH/bQ7tOiNQXcqxiQEGSIURKMvxoxcHSNr7MJh1kYanTwIbfhGCxrYSHNvp4K4Fvcy/EkRDRIpIuhOtlWRdg0hD+QrLUOWiQywJJ9WtNfDPBpuReOdpCtMGcLwQKfYCCrrizBqDtMwRnSjcT1KcZ8u78RczEaHcmA0VMuaOPA6i+jVEJHCgnEwl143NdG+FA1w0nkn/WsacPSLvdY26UF9GV3YZ3Kn7iTypT9kzHFNxlOuiJEnlTTvw5TRcKmIWbvDBKpbcRlHRcfwqD9O8SBZI1rT5PqdkMykaR1RTy+RK66SRrFfqOiMzWhIVNzHTFctTtsSkxFiRIwMFBINXvkwIszJeTtpS0cjnujCW6zDaApFSMccnarvw7h/rhR9ZLkUKUS+jTppszf5SprMpEnOt3sn62T+lbTA4/UVHWWFKixrX0QpUL6Txruf6fE85k0YHaKHWlFsveTEZntHblsp6ls/DkWyZBilRNNqK3msj3rSJBFR0X1ZpCyBodajW7PLRZOqrKxVsZZmY5gbyjtLe5yhnTglp8+05Pqd8kCTDtNONFnrR3dgW5nRqdqVOiK0D5xfo1GVKzXZzo3n2bQuOt96cQOGsbElBDEjguikGyR+xKixUQ1Gjz4MwWhMM4x041A9cN+NzepIu00RJfAVum4QOnCZkRx0VWffairVYPd6zHhLOtsxTzO5zabXS68VHWpD0+ybT22k8Uu61eQk+9CJ9iFFNgT53Yd93Mj/1y14aTD9j3GZ/QveflYay1/7nb/3E9/uP/7t//ynfmzwc95YLj66CN/6nb/PSbXlB7enbLYVk0mHjbTG49mON/czjhc7qsxyu6vpunxsAEKAInPcX8/E/OaulC+lg5560rG+maJMoJz2dE1OWQ+0y1IMcYIiDBpTOVwjJjmpUFQTizYBt8zR84FwW3Lw7j3bXUlZDWyupqjCo69z3NyhaifUmutiRNiymxw39UKFC2DmA26bC/VtOqC/rLCHUhArq9GHwmcKbyrUWYdb5WDCvsgEzHTALYvR1ZUsRPdaCJUjv8rFQdZD/TKjeZKKVLGYyxY9fF7jLnqyasBe11I460CopDj0E0+2NIRnDf6qIuSefGWk8PJicOMOLPmiw72ayHNnjuLLnOHQx4I4yDq6B0Vtp1HHHeG2FNOO2pOtDfbAordGzGIqKfpMo7AzP9LCQhEwGx1jO4RWp3sptELpMfcZpoXh0EvBWopDb35vSEY1AUYX0uHUYpYmFlQau3DoTtDL4l7RXHqKSFtzZSDfRJfVWCCixEjG7DTKiuOhncjO2rmTNS091QuhH/ZHnvJaMywEXesPwt50KEB5q+kXYW/woKPbajI4yaWwKpaCOgr1KuxdAz2j0Y6vA2atxcdprbDzQHGn6I6jPf8kMHmp6Q/3xh0ShyP7n3IW843kvaXCPZmPDHMpKLtjKVzVgzgGV8XCehDkt1jFzMZ4TkHMdspbxTCV/c22iubCU78RR8zUICB/RndSLO8e7417QsxntFMpptUgBWt3JNRHbaG5EPTOTmKhbJEiMjmzRgdc0aTuCzU9xPWIRXtaI6HBSVGcoku0FcpevpXX6BdyzHYC2UYKRbRE4aQoHu2Q4j2PRXrMzBR0OBoLedmn0cgmCE3RNIzxKokml21igT1IA5AKU5Sgtc2pGg11kitqtkUQy22MSLKRDtfFKKRoHjTMhBqZqGwuXo+jUUq7N5zZXcjrlPfSHIxxI3ZPW9VWivzUQKRrO+nqbaXGNUkup+WtZCdmUT+cYn+KpTR4poP1O1KY59tAvg20x5KjmEx/iq0gjCkqw5WSWZhcbX0eaYwXojdPYfXJ8GZ//PJ7vg0jSitIojRGyaBmmCvqN57uSFPeeYapGq/nYhWjfuKa2BqJ/IiRKNlO3lebJ4r6zT62RYxsZP03TzXVdRgNjCTDUTTc+sGapjgVlyfHW8b4F1kHua9Yi+7ddEKJtJV6QD0kNh4xcsnKz/kuiL53I7rJ7iBpfxMdNYwoZR51zOncjcY4VraRsiyHiaLYeKFlzvT42LF56aWJs6U0QCbF6WSKfiafDdW9Z5iIxjf5EWRNGKOVipWjOc2iaVV4YKgktOWsDUITBYmSmejoGJwMkxzD1GA6T3k/sDsvyDeeYabHhkr3e+fUbOtE63wvmmciRbK8HQhGUMDirmc4yNGdH/XkPtdkjWOYie45NYq683sjnkyiN1SMXtGD39NHU/bjSnTQ8l70DHOJeDGNl8iXqO8FcHlEOndWGirrcbnE0thpNjYsPlMUywFbG7KdpT8oKFaDaImjBlv2KUiTZjSu1BQ37ej2aue5uNsqhWktwzwXjXznCEajO4uvM3AhHktqduNjHtBXH8as+DITHX8ziE61ytCtFUMeFeNSUmyKMdJIJrppbOb8JEe3D/K/tEbtxGwmOcaGIq7p4MRcJ9FUU2MYnxcyjV4LmorzI4V1dGN1HtpOHGjLQjSjCam0shbpfKouoqZJW/pwe6kPeEsj6VFFTsqhTL2CyjJBQIdBflZKUEylpeFzbkQlx0iTr8SPvEV7jSZFIWVq/jmUWADftn/uY/7fbj8rjeW3fufv/8S3+09++z/7qR8bkMIgfj5vgzP86LMzvv3pM85nG6q6p8wtmfEY41k2FbbPuPruKS9uDtisamaTFqUCs6oTpKTL+eb7LxjWBWpq+Vd+5WPOj2MI4aA5OV3jrCZ04vZ6fLHCNxk682S1pax6ykVHVjjUOuPdD16DV7jOUJ/vODzckl/usF7jv6zp2pzF+YbgFPnzLYeXa1jl1LOO6r01Zj5gakf5tRUh95SHLfMLefzh+Zr8oCM4jT1wnD69Z3GxQS16XGdwTUb9zhq3zDl6skRvDOqwJ5sO5LOeuu55/2uv8GXg+Ok95UFLebHDHAn1djixHD1eQuFpnliqk4bZ8yX5Qcf8co1tM+zZQP1xifeCkBRPthAUR5crwmmP6jTFhyv8dUmYOCafywfo/NmK4tEWX3vMbCB8NiVkQXRiVklTOXFUhy3F65zipJWYFx3ABLKLHeXHNdnFTjQEs0Ga44gGhQ+3uGOLLzzuaYs67AmFNI4UEsDOwQBq7/CaraTqzt/d0J9F59VDK9EuCobzAW9gOLbY0wF/1kuDHJ1s9XGPPbaQe3QrSELzkXyBKK/Qz7eUN3p06uSyw808plGU12LkY09E35E1e2Rj8qWheJPh6kB37jAXDe2Fw7SM/4M0fv5wkIiSqcOeDvCkwU09rhDKqyukibVPO7qTgKsC/Td39JcDbhKwj3r6RwP5WpFtNZPPDe5Rh2kVzfs97knL9l3L5KUUGvUrze6xw068RLLkMP0iFi2XrTTUMy8Fdi/6sP4w0J57dl/vcFVg90GPn3h8Hqmvjzp0rzBNzO7MIVy2DFOhGuZLaQJtLUX69omne9KLHs2Dcort+wPFCtrHgzT+vTi4dudOmtheXDK7c0t/aimWMmTojrzs32lgeNbTvDNgp4Hpl1rC5oM0x80zG6nLsqanfygGPL4IbJ86ipUU0P1BmmbEbR957DRgGqG3JsShPZMCtHlmGWbSfGVbQYdMK43wMA90J57NO14o0y+l2bFTaM88w0xoxdv3B9pzT3GvaM9c1PIFuuNAex7YXYZRa+Yq2D23o0tjfyANQnMuurzkjLp5z9Iey5AEhGbpKkG80v76UuI0XHSCtTEXdvPcS07nvTSr3ZGgS91JYPuOZ/PcYydw/62B1Xtw/3VBntpTabi6Y0F42rPA7nGITbQgSN2pNFa2liZ2WAjV1laK7bNAtgtsn8g+yxoKDXqYShbm9kmgO5S12F0odo8i6tRJ47O70OMwYP0+DDPF7TcNrmKMQRgWcr+r1BilsL3UpHD69lSxeUeovio28Kn5t7WiOdFsn0hDunlHHEg374jZ2jBVdEfSYNgaNs80w0yK4qChOdNvGTK1p/L6/YEanWWTk2/6e79QbJ5omhPN5qk0v825GqOTxERLYadCk3WFYpgo2hPFUMt+7C4UzZnC1orqztOeyONdAc2Jxk5lkGVLFQcmKmoFHcrB9pE0gP1CsX2iaY80tob2QLO9NGJgt/Fj5I6tZL+HqdBk20Mta3OgWL4nx9KcaPq5DPC2l0L/bU412wvD9lJM2vqZojkW5sLuVLN5rMe4l36u2Z0ZdmfRvTxAPxOTMZAs0cmraLw2F33d7iwbH7+9FKOubqGxtUSabB8Zto8Mw1SosLsTM8aQeAO20qNOsT0u0A6as4xhIue1OdL0C017aGiPDK7SbC8020fFSG3dnWpW75YMM3n95Yc1w0TTnub0BxkhU/QLja0N3cKwfio6PFvJfvcLQbuak4zuQFMsB5pTeS3T+dEFWNlAd5xjJ4Lq+VKe3x4ZXK2x0yw2bJphkrF9JLrb7jCnXwgq52rD7lGJaaWh9tE4yef/D3tv1mtLlucH/dYU057OeO+5N7MqszKzquk2dLcBuWUJhBACAwLEAxJ+YnjxGw/wBXhk8BvTF+CLgAR2W24sU93tpp2VVTnd4dwz7immNfHwWyvi3Orqdksuqyttb+nonrvP3hErVkTsvf7/3yRhVzqhspJjXrHjpDoHVysEJeErjeHMwK40xssadmXYcAg0w5Kjn8zI7ELT0Gk/INQaQUmEUk3usG6pIGzAcF7Crg2iklPzQo40aIomobGlRn+1AELEeF6nwlzAnlSISuL4gzWgJUJB7wG/LOGXJfM2bUBUClEp+CWNj2JlJmOhaJKB2+OB+1dyKlyjUQjLiq8BWCTW5YxKGs1CzLBQ9xdrhPMT+MsNxGjhz5bwF+sUSQIWm86TPluVLCq7YdZcCkHjotM14maFuF7y382KRSVA5DFGiKKAKApE5xCdp6mQDwi7PabMS5s0lgCEkojjODvCpqiT6D1iMi6KPiD0A0LPMYV+QBjtP/Lnnz9+tR/facSyef69+Ov/4X9NKsQustPfcuETNJ8b19Sa5NBogB3GLBbP9teumm3TM01FWoZK+1KgP6U+Kndic15dDvwG+DfTzh1d01JPMa4kO+rJul0mm3xv2CEWgeHu3tDZsnoIGFccv0joRO6cRiUYpl3RKTMYfgmrYaaN5IVHUALVg0f7TE37X1w77L+noVvqaHK+V1D8Ii92MYWMy6n7OuWHCS4YclD5uBCoHwKCYjfeF2LS9ojA4/MlFzeLV6kYS2He2elTujiFmlOsP1vw25SXWBzZje7PBDZf+ml+J1t3YEYCE70od5izbTvnUWBccL85k1K6iP5UotzG2V7d08GzP1UojoE0l9T1FgHQfcC4JF3GlVwQuXQ83pAmwzy7iMOVQnPLjrG0PI5M81EW7Dom+stTG/lgBPpT6pZ0T/dERBoP5OPpTyQ2X1rYpZqC1osDXWFdWgQoG6GPqXPdBzp3pqw028gpPw7I6Ga+Hub5sTWdQNVInZrukotqz1wzgMYKmSLFsHcJ3bJjz31xftQYodLxsIsfYfbsAtulmizrqWEkRck2vBbzHOkhQB883JKaHd2ye+4rOaFKoRDJeZddbV+KFPUQpkXl4u3IBYzJ2XV01XULOoYyLiFdg1sHu9bT2HNkie7Z5bcLLkK7SzM5wAaT7suK5z6kjDwaeRBZMEef9sEOum04h6qfg85josqNSwVlOX/Z8RKg/isqge5Co9gHqM4DksVIDrSP2TVQA3KMCIWYYzfA+6bcku61+77G5qcjfK1gDo6uph1jALK+K9/HwidULEa6jo6B1/maET8i5c4FzQgACJEy7+gKO66pbc45ecFImO0IuynoChrptDpsFHQXUezcdNwhRR3wfEtIG+ieW3HhrHqiPRmxebpfX/K6dbWEOfD+YGajTK8HzN7BrvSE5vi0j7x/CAHdOXjDhSE1iZGL3ifB9nGirNGNVgSg2FkEJaFbi+G0RHXTwS8MXK3pCvu8RPlgqU9Tgu62MUINYYq48CZFLAAT/c6bhNRFLryFD3ALw/um9XRUTQhO1HSJDZWadHBy8FCHAe605pgDiH61FsFwEc3mXEKpai5wZfc+QqQOI3x2m825iT5CtURN3ILFDhSvfTl6hLQoz2hVvidCzi9MuYQiAmJ0cKtycqsNRkH2DpBANNTNxdoAIcA3BRGoTCtMx/E0e1Nan2IlElK4HxClpMlKz+tF7wf4RQF1GOBXFdRhQKgN92VI0Za9g1+VkO1IF1OfkC0tIUbH/EMl4BcGsnNEtlJkRVgWk+lKSE6xQUuYmwPcxRLCesjeTmhZrA1/T/M06eVi5L6yeUuMCFUxjUWOROBEl9x6Nw3krktzoyFComYaRUQtXevC+imaIqwayGMPaIVYUDcI7xEXFY1l2gFhXUN0lsfeDnOxYzRiqSH3HcK6gTj2QGG4bc2CTPTD7PCaC4nCQPSJU501g4mm+R5S54lyTqY3WdOXkD3e+H4urLSaY1as4/NScr8Z8RtGxGVDvaJS/H9dskiTkq/rByA7ty4X3EeO+8jRH3k/jjTSyQV2HCGMQXaEFUpOER65gBNaMTMzucAKIRCHcaa2Ps3AzON+ikSORB2FppNwfsRfVKiFAFGVdLZNMSBTpEjSTwolZ+rqlE2ZzHx+vq7Ixj0/70AL4M+TSxl//n3fcSrs8kdX8bf/t//sl77d//vf/pt/4ccGfMddYX0JqP/kBrdfn6F+RQ2P6ti5tSu69Q0nEbsfRhSPpEKaHc1YYsrSyuHk4wnd+ugWxy5o9zygecUFjV0I9GcABFGj5luVFlQpYLtmAVhsBbrn1PYUjwrFnloMVwu0LyOKB8kA+DUmqs54QkOQvI2gScEbTmkC48tsIiIhB+DxRxKrr0jJ8iWSJTzSAoQmIMWjwP6TgOJRp4D1tBgrNILmcY9rMekpXJNMRmLE/W8oIACL1ywIMnXLVdSPbD+jKcZwFuFfKxw+jLj4MZ0Eg+F7mJ0nsP84QFqgfpdoWC8lmuuI9oWYNDZqVFMmX30b0T4XyQWP1K4xjd1XLKZy3Ev3jK+zK8bIxNTVr28YT9KfCzTvWDzbRQo/P8l5k0B7FXH++xzr9hN28s0xotgDd58YVHcRDy8ZGVG/i3AV0L4E1l/MtCfXACKyU56D26MgkrL5Calgx2csmosD5727JP2VOZURw6lAsU0Njy1w95do7OELomxB02SC4dF0ECzvI0LJ7nF3KWEOIrkCMs4gU+2GM4lix8L59b+qsfwKKA7s4HcXEou3LC4ef02iugWy0cnxhURznZCf50QXygcxhcbT6IRIQfcsYvUzhXIXsPuBwuIVsPs4uYWOQHVHpA0ROFwpmKNEc+vQn2rsP5J4/nsR+w8U6ruA7oIL4+Urh2Ej4I2CXSKdP95nu1ONzU9ZpHUXEs014AuFw4cCy2/jpMNhU0djOGGzY/sDic3PWOS1VxLt8wrrrzz6E6LKi2sPu1IY1kRX+nNe48//LvPTjs8lqns2OKKgMVLzlgVzeykxroqUiyjR3HB+zZEIUqZwBkNjF33ElC939xs0/1h+S9pjVIzPAQQbHCcKQWO6v05+kimtYi6kSzYiTBvQPjcTfVKNNKSC4L0jYkRzw8ZUlPwMMAfSqIudTwYjzF7bfaRx/ocO3ZlCfUdNUXcuYI5zwwaC90x3JmFXAs11gOnYDJAO6M4EigM/P1dfx1TwS4xLBd0yhibr9ap74Hilse49dh9prL9iQdqdqxRHJKF6ieMLjXIXYGuJxbVFf6bRnbPJsnteYP3ViP5co+5TLMWCcSXS0lymuywwrCSqbcC4kIzsaAz0EHF8JrF8k3RYQaM/VampQgQsKo3qgfMEzEhWRiuZ80djlSiAw8sCdpkMg6791IQwe4f2uUHzjs6uui+w/7BEfe+AkBp3az1dx9JF2EZNkTuZOnvyBRkS9oTmYa6RLPo9MHwgaSRTs7gdTjT0EFA8OvhKoX2mUd8xPkd4Nl1kqbD7pOZniedn5+I1aYH2nMYxrhLY/EMLtyoYQbOQqG8EEbITNmrWPwOOV2Wi73qY3Yj+WYnKkRbZXxjUNxbHFwZRCCxfjxhONaSNqN90cMsCZjfA1wYqaMJYAAAgAElEQVTj2kANbPAV25FI1qbAcKrRvE004RABJWDXJfoLg+WXvIZdQ0diSBbtunOIQmDcsGFQ3o8IpYSvNVRPWibAptr++xVMGyBHjVAKhFLBLhVqFzBuDKrRY/dJg/q2gHkcWBCmKBV/WsIcWOgGLaFaByE03LKA6hyCkhif1ygeFHyjUdx1GE9KZEsXtzJQnce4MdCHgtFCCU0DAL8oMZyXqG57jv3IgtTVCuZgIQoNX2mYuyPCskL/rErxOxrFXYvxvIF5kBDeY9wUKEeH4bKZjKpEYOZjedPCNwVdUHsHtR8gugGhMXCbEvowwi0LyMZA3+zhl2VqliapRcUlpnu2gnm3n9Zu9qxB4QLcSQXTDrDnC5j7FvasgXABuudz6mghU2ZjWJQQUsKvS+jbA/xpA7XtGFUyOprgVAZ2XUHvBxbVzs+mNSEgbBYsgrVCXPB9oSpIeXUBctuyOI2RPQfNbMh4toE4djOVNBoWw/3IKBStIPLfVksWtlqxqNWKhj65kFSK2zL0BohVQQpoYYCuh1jUgHWYSj8hILQCyoJ01sqwsE1RJRO1tK6ArmcxWVcsCvueRa4QiG0HKQRps33KgfR+RiZ/Pn5EKqDWLNqlgMjFaFnyNcMAUVcs6Luex6Uko1HwPrU1ItFiJf4E3TUmd9SJNvsLHkLhvaI0/mMUlv/88U/+8Z1GLMtPPohX/+1/BXiBHOguumSyUHLRI0aB6kaie+lhtilwXTFEPawc9XUDaXh2Tf2Xa1KQ9KlHeaNhV2GyqM9mB1GmfQRMuiy+LyEB6R7UvYCrGQ4vAlA8SoybAH2kcYOIczGStUfCMxS9upEpvzKjMEi6KFIDiy3pfllzwjdzIRk0pkUtLfJF0vnEJGRnMDxAzVlUCb2IpORFDUQ163LkQMpYLgBz0HixJ0Lsy6Q36bjgtssA3bJ4dIv5GKIGynuiQONphD5QRyeTyUmUdJ/MGp6s0fEl4wVi0iQBfD43CPJ4stlDnov8uow2S0daatYR2VV6aUINM2qdg8qFz8U/tYa0uE/7SppJt+B1g0hr/uZ1RH8pEsWVlL3sBJjzJhEwWeernvTGrL1j8Pn8nuz6mPV9ocBEqRSBGim75Ie2XSbNXkJ9zT4h8g2w+cJj/6GazTWAifoox/lY8znPdvJI14Wv0mv8bIShesyLnZJzm48jFHMRqntSGCGp9ctIge7YGMjIWdZguYaF97hiwyMH0yNkjWhiDFgW5tn5MmvfxhWfm7VE83aREMGQ0OUcXj7pCAekcZJJMG5yAwaTAyKdTanfkwMmFEk6jtGlEPuMpAvPQlBZLtb1kTRfn4Pc5fumGk81QOYYpyiI7ECadXXZDIRozmwMorvZqCO/N89vPi8iYHKezdpLETmv+b5yNabPl0nfhnSv5M8CjUl/Oc8TJmpmdjQdVmLScAadGkn3ZFIwP5Bjn+65pWCgvaSOb1izWJV+ZmXYRqS/z8wKX2KK0FAjWQTZHEX3fI9p4xTknscfJfWUuSjODJNc+KqBbJOJLZL0lUTCY6JR8v2cpMRcSEU4s4OJ5gc9u7RmAxc1RvQnEuUu8DhMzhnGxGzJFFhE7jsoNs1MGycdM4AJgckxDVGIqRgWERNTIl+fxSOR6Wxgw22wuMzMnvycdPxuzWYmviA6JD0mJN0bXutEukXKRk73jpx1qN6knNvl+xpWxnuIhLwmhN0hIe2J/TLGiWGR9atBi8R84fyYA108faUSIp5iNDoyDZj3KKfPUzmGGWFP9/qEfMVcbMuJuWAODr4mxdMcHOySCK7uPVytUx6xg13oxGIi+inSHNI0JyaTn/xZESf9ou490dKjQzBsGubXCJdMYtK4cnYv73Oiu9FIahbTNjMqzcaYSvMYJraBsD6hlRKMiiGaHKVIesLM6kksABcmnaJvNMQY0nNI80itJQD+3cWUMUpUUfYWflVBDsyWzK/ntczmAL/E0zmwRLnh44ROi5TvGUo9u7ZKHoPwAWLwE6Kc9Y3CegjrEUqDHM/BHST2iE86yLQf4SPRzOz4GiOilBAhTCY27zmspmKYSK+bdZEAi84c4ZHNcfK+czGatyXEvN+M+nUJ4U1F64RKjpbPA8AwAlpTBzlla+rZ0VVKop/4Mwo6n5BWqd5/PudhPnVxFXI6DkaOzPdNLiIn5PPn9/WLUMz8+NPyLv8x4kZ+VRDL3/pf//Nf+nb/1r/zP/6FHxvwHUcsl+WIq6tH3D0u8dvf+xYhCvxo+Q7Xwxq/f/cClXbwUeD18hxqYXH26R5GBtggcfe4RGk89IXHumL35tWrM/xH/9bfw6Nt8Krd4Hq/wr/xOz/B/3v/Ab65OcVHz+7xdstKZN30dNM81DjbHPHZyS1+/O4FSu2x3ddokt7zrG6xHSo8HhqMvYb4eIS0Csp4KAB21AgiYlGPOFu0+NnrCyzXHexjg9WvbbFrKyzLEYOlC+muraBVgI4C/+b3Psf/+e1nOBwqLJY9Drsazy52uN81+N7FI758c44YBT58/oCHtsbhocHv/NpP8YfvrhCCwKIiraS3GsNg4EeF5arH7maJ1eUB46hxsT5i35dYVwNevT6DqS38dY3Fx1vs367w6b/wDf7o73+Ek0/v8XC/hFARZydH3L5bAy8d5D9ocPI719h3FaxVCF5i/ds7vL3dQMiIbm8gFw5SRggZUFUW7U82+NG/8jW+eHeBs80B1/drSBXQv23w7Ie3ePfFOV788AavX5+h2XRoDyVOTo94uKPl+AcvHvD2boMYgQ8uH/H69gRDpyAM3TuHA3MDRaeApcWzyx2urzeQOmCwCqYZ4d80UFcd7L4AZISqPYYPaY60+HiL/fUScuEQegVZeQy9gmgVmg8POP6mxPhqgfMf3uFhu0BI+YyQEc1yQPvtEtXLI7p9SXfQTkOMdLNcXe3h/v4phjMPfZRwi4DyRYvjuwbqdIDbFZC9RDwbp/tAiIjQaZQnPYzxONw1kAcN8/KI7qslVB9hvz8g6Artx5bH0zgEKxFHBVF6RCdh3hrYM4/FsyO6r1cQz3qsFj0e75YovzUQXmA8CYjnI4SMCFZC7AzKB4n+owHNuof9fA35yQHhp0sEE6EPEsMzB3MyYNEMaPsC475AuR4wbCss/9ig+5db4E1F4yoAYpBYffSI/U83CAsPMXLxFEv+3TxoRr4AUA8afmMBJ1HcKrh/fYf2oYYYFB1UrYTZKthzB7lXiCYgLl2KkwkINR0/AaD+dIf9fQOMEnKQNC9aJJ3uysJ8XWJ8aaFvzeRyadcesYjQO4XyXqC/pAMpi84Iv3EQvUL1VmE8DfBnFsWrguyGRwH/l/cQ/9cK4xroX3iYRy7oxucWoidCFz/sIL+sEyVToH/uIHs5ua6qswHuYJhVabkQ0x0dSNUgYDfMqkRgA81dWjRfFDRgUmRzdB84QEUU1xp2FQHJyJrx3GPxtcJwGuHWAaufKBw/DPArD3lQdEUVmBxFg6bZlOoEVM/5Kx9YaIxnXDCGU4vymwLDhcfhkVpjc5DoPh5RfVOg/8Ci+alB/5wxKnIQEIGshuGcDSsW9hKupjlW1AK7TwIW30rsf+RQv9IY1xEmNfDUIKEPQH/JgrnY0oCq2AocP/Kov2XUhq85bjkC7ScW9VcmmQFFiCCnIju7qKrUkMkNHzUo6rI31AQvvpFTniI1oXSnlW42c9It/1bfRBxfAg8rgfJGznKL1OQyLabcwPKRxkr9M+pe5ShR7FIDYwH0ZxGLV9SU1u8UxpOI+i0bTP0FG6D1DTWSugXwQk1GTGrAZIRV3pMZks9jFBzn7lNABCLv2S01lHRTzc0Y4XmO7EpMDq6+4rE2b9iYqt9FtM8Uxg0dZHVLxopKgAodYtl0kCOPDWDDLCRToWw0lKUwABt81Q2ZBHYloA+piabZOCp3lGscX7CoXLxmoekaxcilDwQu/sBNpjxsQHF/x5dkmNgVsP5SoT/lPRe0IYK/j1i+oWawfSZRPurJ3XU4UejPJUyKdBmXyZ14IcgUeU7GQG4iqlGjvZKobk0yJeIYzDG5p6Yi3VWShfzIRXpufnSnRDmbdx79mUKxpztrsfM4vjCQNqJ68BhO+LfcQCDtmwV9UIZFpKccJGjmWqqeEpbiwIJ9WEnUdx7jmvNhjpxjV0uYvYevyJhwpUR1Z9E+N1i8oSmR6g2GE4Xm2mIsqbnMFP5hQ8ZGde8RSoHqZkR/WaDYunROiOjmZluxs/ClSuZ0YmoasBkWMJxoMoi2dood8csSkALDaUkzogDYtUZxP8ItNIrHAfZqieFUo77mc/pIWrDe9XDrivRiF6B3Pfxpk+jd1Lzquw6x0ghGQbUj6dlH0q1DkSngjBiJSpASrWWi+ZJCLbMb7LoBXICwDu5sAbUfEJYF1O0e4ZQ5l/rdDrE0LKL3ieasJNBUNPXJDrcZTc2Pp+6w+3Qz6YQw/jzNNiOf3rOwLAwwWrq7jpbopQ+JypsyLwGIp0Y+mBtZ0+NpoSm+0zYw/8w+vtOIZX31vfjDv/7fzA5yyYyifIhT6HB/xiy1/OGSH66iVbdd0OI76/HsIiFRSAHKq7RgSs6aysbJoGA4oQV4Rs4yWsMPfqIjRODi+/pOPyMTQSWrbU1toR5mBztf8Msh6wnVkJ3pAo7PFZTll4oaI7Y/UFi8TZ1Bk7V6mI0epu4ob2TThkkbahs60uUOsunSF0J2CdR8r/T5y47zqfoIu5QoHxy6Cz0dmy/SsfQR0ke0F6Q5jis56Z0AJFfGmDqfdPhTQ0SxD/CVSDpX/j0HJwct5u54iBgX1K+SDsxrub0k9TM7ngJzB9/WpIoR1eEXoi/EpMF8qg/M3XDp4vSavMgJWiTKJQ0ZdBeo1Vqp5CrH8fKYk1a1ZDc9684ATFbz2T2w2Hn4mrS7chumfek+THMhHfWHOdDbtGFCHnIXPyOS1Ps59GcFqpsB/WUJ1QdAAsOaiwmilmLSy4kn+jBXy0mzihjhFgqqT+ci6X2JNpFOqtuAca1Q7DykDRhOzNSRV8PcNfelhByTrk4RyYySiwlXc1Fj9kQApI8QjvsGAHNw6M8MdMfFzXiiITz1d65RE+pLVEBAH6mVUz33LdzsuugaNWnughFzuLYRKHYW8BF2ZRAKwWDuSkH1Hq5Rkx5TpGtRHyx8pSFtwHhCeqs5+DkLLT2iENP8+kKieKTtvG/4oaD3Fm7FzrO0gYjFwcJXKi3uFKQNDO1WEtSCzpqiqAR052dXRSNhFxq68zTGaAzU4NFfFNBdgN6njLxNMd1D0lKfN54WKB7p2KgPKYdM0XhDDQGqc/AJmcl2+1m7FpSA2Y3JqVXS/fFxnMYoArPxhGMenhqS9nelUb86wK3K9zrfGanKcyki4CsFvR8hbIBfcM6iIfoSjJyOIxg1aajl4BAVzUd05yekR6ZIgWgk4OOExohED/MLA70j5dFXmkiOkpN2T3WWYwYgBzcdt+wcoCXPb7r3M5ISGgN1HLm4jIBbGMYSZKQ7cN/RqPR8TNuISYtHgxLZZ9QD04I2xxvABUDTlEh4ag6l9ZDtCL8sIUcPue/hN9RV5lw+ESNCU3DcAVCHkewFSdMTMRD1gRLUJ0oxO2jGCNlZuNMG6jjC14bIlQsQ3kN0I8KmIYXRR/hVyc+fzqbfQ5oXBr5DiERbNNPzcrAIi3JGxdK2EcDjNYqZfZiRLQiOVbaW1MimmOYzzy0kEEsDuevo4tmNEKOl/tA6xKaatxcj53BIi/9+TOdLMyNx35MqmQxbsmYxFhpye0RsKlIFjZo0h4gRoqe+L2v4Ys3rCqOdUDPRj3QLtW5a1NPq3hAxCwGxKiHbftYkZiTMJ9Qw6QSn/MPRznTPfkRcNRDHDrGg/i9vUwwjC5TCcG5ULhoCsvPnexmKWS/5ZN9PETzRP9EKGmpdsw4ymkTHzMibD1OuYiyT7jK/J8bJ7TTW5XvjmYom6+axSDHNWzao4WvS/rSe9ZBGc0wA4BzRwLqmtvQp+uccdYz58z65ogqtOeauo54y00RDmDMhn0R1QOsntFTxfpwHN8xjyQ/JbUwIZUY58zaUYlE4Wv6eUMc4jvP2nj6yXjfnV+a/p8JQaM33GvOe62vM48+Pp4ViGvtTzWSm2MYQZ7rtL3i8p7P8zmssX8R/6X/55SOWv/vX/oe/8GMDvuOI5fnFDr/xn/4R/urJT7H1NSQiGjVgJXv8re1nODEtfvfmY/zl81eo1YhH2+Bl9YhSpLxGu8DBl/iseYf//Sd/BX/9k7+H33v8Pv6/d8/xH3/6Y7wZNji6AiEKnJdH/PjuJQBgt13gtz98hYMtUSmLpRmwG2sszYAQBf72H32Kv/rrX+DgShTSYWN6XPcrvN6tcbJocVkdcNMvoUTAs3qPv/P1x/gPPvsD/NHuCoPXMNLDR4nXj2tcro64WtCltncGlba4ble4MiNCFLioDjgxHf6Pbz/DZrWHUR7vjktcLXfYjhU+WGxx0y3hosTgNEav8KPTG9wPDX59/RY2Knx9PMPrwxqnVYdV0eNgSyzNgFeHDU7rFnddg03ZQ8uApRnwe199H6frFt1oUBcWddXjygxo9Ijf/fwT/Gu/9jn+4OYFinLEzW4JYxw+u7wGAHx+f4G2L7GsB7xY7fBHr65wujnCeYnSOFy/22Cx7lFoh340GAeDT57fQoiIz18/AwDUzYjvnz6gtQXubk8gVcDl5oBV2eNnt+ewVuH52Q6PxxohSJwsW9zvFlAqoNtWkIXHYtVjf7/A2eUWj48L5i5ahaqyWFUDrncLLOoB/ch4mkI7WK9w3FcwpcOqGSC1w3EoMFiNoTdQKmDRHPH4ag1zNuDZyQHXX53j8sNHPO5rVJVF3xXQxqM/lCgXI8ZeQxcel5sDdm2Nk6bDq6/PIY8KoQbKsxZ20Li63OL17z9H88NHHPcV8xM7BSzIrS1qCztoLFc9uq4ARIRtC6jSI1iJxWaP47crFM9bfHJ5hz9+9RzhCFx++IgYBe5/coaw9IAAqvWA/qGCrB2U8fCWlKlgJcQROP94i9ubFYrGwo0K4m2F8HyA0oEOyiNS1iKgqpHRNyLi5MUOh2MFNyisTg/YPzQw1wX0Z3t0dzUX0wUgpIc0AX5QkMYhHA3kwiJYAdEpxFrCpGxT32tIMyKMCrAaOaMxWAlYCagIBA1ZOeC2xObTBxyOFexDyczItYU4GEQdIRYOpnQY7yvonYJvWPAiANhYxIMGygAEBX2v4U4BRKA66zFcN4nDnOiLG8usSyWhBtLh49JB7kiJgwRCw/lGKCDbNMdLB3moERYe8qiAiwF4pyCiYhajB/z3aAscBkWUsgzAUUIMAqEmYio7jahSIWYSFd4pRFUgNB5i0CjvFWn+QrO5sHIQe8Pxrx3EsUAsA2RbIZ6OUNcNpkzOkHMQGYske4moNIRPEoNtWoxIFtj6KOA2AdEUs1zhYsTiDyscv+9R3hDRFS5pQuMJfBXhm4DqnYYcgP55QDQRxW0atyZq6BcKeqvga8YIiZBo1Oce1WsNt8wRNXGSE2TJA3XecaIhmz116iIyUka3jC5SraCztKhglwHFg2SkzJibT3QP1keiqW4ZEUxEeScnxJGaakwZisUDcPxeQHnfYNxwjNT7M1boKU0dmUqfqNa+ShT6ZfoyTDTKjBpLRyfeqLktaYkoesPXjJtM048pymc9UZl1DwwbTJE0qic6F3RNVPFIqn9M0ggRqCkv78XUWHUVsyfLh4jjhw10m6QHi/n7W9oUixOIJBePMx1cWmA4w5OoGf6r24hxQwp+NsYKKjdeE2rcURJQ7OlIXN5jyrTMiG9u0mV5BuLsDCwdG8O2WU0NXl9TchAU3W/LRxoD5mYzpQtJgtIminMfAbGepQFPIonY8NxMTWnu/3SiZRdHZqH6glTn6oFNjqDpl6BbLrJNl8wDBaZcSelIqRZ+biQXhzxGYNhIrL51aC+JBDY3YTrubFIHkH5+fK5QbsPclBegLnkfYGsB05FaLh33X986uIWazPmA1IhODVjp4iRLyKZs2dgt65OLHU23zMHTtC5Tf5M8gw6wkkZsKjXq3BPZTqL2ZnM4xnxF2KWCOXg2e1yiEisWmDrpXnPDUfg4NRFDweaX6t17+ZQisOEJiYkiTLptmAptaf1kJCQcXWzd8pLU5JEFrkhxJ5zfJxTS3FCzyaAo0ZRFbxFLnejPiVbrkqmXEuxJeRbZsdAQYzJcWlaQh57FcSq0RTYryqf3afGW3VzzeKrE9Gr7qYgXAOIwTAZDkAIi6S2nbSAfTpxQSvHk+HKupSzU+xmXwHtFqciM3Bj+PH4/v/KPrC39p/HxnUYsV792FT/+m38Dp02Hl4stxqDggsLRFdj2FX50eoMvtqSDahlwHApUhYWR81XZOw3nJQ7HCkJGnK5a/NbFK/zu649xOFT47MUNhIi43q/SawJikFguSIWtCosQZAqSZ8yJHTXKykLKgJfrHb59PMGyGnD96hTLiyO8lyxy2hJSecQgoY2HHTWk8qTVdwamttgsO+zbClp7DL2B9xJhVHj58h7daND2BYKXqOsRh0OFsrLwXsI7Bb83gAmo1gO84z5K47A/1NCG9FNnU6dSRvi8nbdL0jYvWhTaQ8qAfjQIQaLfl5BbDf2yRfhyAX81QNwXqD/ao92XwN5AbEbeNBGQtwUpKZcDc0APJepVj/augVo4+J2BaByEjBASqJsBh7dLmJMB9lCgOunhnYTSAfbLJaNBXi1w8uk97r89gTnt4e5qxNpTaysj1udHdF0BezSQFefTJyps7NOnUxGgbwr45wOEiogPBaKJUOsRUkTYLlELewndSriXA6mEbwuoTw4Y7muUZx2Gh2rKGJV7NSHDahRwVyPiQNMdWTkubLyAelvCnTrqglWEvtfwdeAi7nyA/uMGww8GxCCgHjTEix7xVQ1/4lBcaxo2NQFiEJBXPdTnDYZLGkXABI7bSUQTWABtGMvSfGkwbhg5Ehdp/xFAFJAd6aZhydeqewO/8EDtIVqN8p2atMYigGHsSwd9U6C8E2hfBmBjod4WcM8sqq8L6mrBxascBHzDoieqiFgHiFahvpZoPx0hd5qh8Y8S0gn0Hw0wbwtGXAxiyu1ktppIRYiAPgiMz5llWt5K9M+Y/SqCQNQBaqcZ3n0S0Hyj0J9HhDKgulZwiwh76lHcEi0azwJkJ5PmOKJ+w/M5nDMapbxXGE6ZUZpzQu0qLfC2AnZF6mmUEW7FLFTdcl6LLY228kIdYHEwJKqpLxlVYo6JRbAKQOBxq577Uz3nvr+Ic8GLWUOdNdfSkgqbF82ujklbnArCJQPtfc3CKaPb/bMAfeSc2jWplPsfMEPVLXhs5QPQnwG+YRGV96F6ZkMOp6RNqlZMz0OQzTGcUScdFI9zOCGN1exZ6LgFi5PxJKK+FuiuuA85zhTN9nlE+UhKZ3kH9BcsTsqHiMOHAtUtaJ7WM75HH+fCjxRWzqHuqMVmxmlEdcPzWTzShEz3c96nHDFlfWY6puoTXdQA5ZZsBNfM+txxzdfqlgt3WwsM5yzwdEsaa3XHIsnsaeCWKaW+Apq3nMusy86a3qxrVT21pocPkxut5XNqZKHRX3AbrmEREgoB1bEQG9cpn3PHv5NFQRZPMJhcs6kJJJ3ULVLh50m9zWMzh+TyvWKRVW6pUTy+SHTPpDG3S5pE5SzK5ppGXfVdgO5DMjhjkXa8Istjzv1kQZwZGtKTKqoG5pnmc+tq6pqnnNA469KzjjpKHmexj1CWplPmGFPUCCmeiHRnX772pDgbMWlH1cD50D2L58U7j35DGvVcmJMRQw0s761c5BSHgO5UoTjSDbs/VxzvmoZm7YWGshHllm7Rcojoz0k39cm92heJitrRlVt3ZLbI5Bhtlypd83FiI1V3dKwudollEWY3e3PwGE40qgfHosoF+FpNDAEAUF2ASyZYAGmpeW5Uz3M+nGiYvZ+KWdUHMkiSu7WvZ61ueWcxnhhU1x26FzXKu5HsiAei93ataDJVEh0PRqC6pYu37jx8xfEFTffnUKqkr005lcVcAGZnbeaCcptIv2dWQ0aw7Vk9of+kooLshdFPrAIk0zI5OCLnCS2PT8yVJmZBQqOF9RC9RViWkL1DqM3k3Ct7NyPwpZlYAJlaKyw1oqIdZuQ1u81qNSOc2bEWRJ6jVnjPMVfTLVs8RTkzgg3M28mPhHpDP0GDgdmtNqOsGWkGiKoCf6KgFMn59k9kWD55RB8gnhS5v/A1eQzD8Ge+7s96/Koglv/i//xf/NK3+3f+3f/+L/zYgO94YVl+8mH8+G/+DYy9Rtibme5pJeLKwVwb2DMuoGUnEaoAvVPJ8S6HrycK6IpaK2nZ8YEEoko6micZg0FH+GWAfqRboBxF+oClaYjuuCChDihOaIXuBLqXDmar6F7bi4mylwPlXRVT2Di769w+JlMdgFqm4dJDHem4KQcuEFUv4JZchAfD/Y0bap1EENOxCMcFnG6ffICkBbuv4pQZ6Eua6ORFSzbvGM4jgo4odhLDaUB5L+GaiOqWC0K7CqjeqamDfPy+A3TE8nOen2wGkhd9wykXk6FMC6MB6C75N1/xOHJA+7gJqK+pjdIHMZmQMF8vBaYXHAs7pJy7aZFoiQbkuc/j9uUTQxOXguGvBEQy2gk66YMKvr95wwVY1iEVj0D/LM1lyAt9FhPDGUPvsw4o63XoKsvn7IqoAs0jgP5ZMpGqOSflY0R/luiiCcGQI1GE8h4YTxl0zwVzWuQmI5bczS8fge1vjlj+wwKqn8ehBo65v0iL06Tp6i94Xkh3FnBL0KG44Lh9CZgdF8R2zUJAtxHdJRfKwxn3L1J3nkHiycm5TzTqNc//6mdceHKByjGZHR189TGZJUUiEaHkMTXXHJddJMfehgtmc5jnIMcCBcP3dlcRi2953Ifv0dk4G6jkogLAZErUP9F4NEkAACAASURBVON1X91yAd1fzBFGOZvUHPme7lJMZlki0AnXNWlhKfHefRQKpNB4bjcv6DPiEhSLFWoXuWi3CZmKgtt+ek1n8ypX0+gnH0/OUBTZVEWL5Fwdp3GOa+oEfQEsXwWMa7reLr9lgbV4E9BdMvbGF7O7dDbJUgMLpWBmJMUcaZ5EijfnzC4F6lvSnrP5jjlGdJdc+Ok2Qg8R3YVEdR/Qn0m+XvMc92cC5RaobwMOLyWqOzrNllu+ZjgVqG8Y05Rp98WBhkHKZiOgmMymIoYTCdXRfGnxNqA/zWZRAtVDhDdEf8bk6Jqdv3kvcd9sILEQjJJFpC84N8WBKM+wkROyVW45vyLwOI5XCou3dCWu7wPaZxJq4OsOLxSqhzDF3QCYjJIATHTgxbWfIppEIMJFp16iUGpkLFR2c1ZDhB4YnZILqXEhkwYQKTIpfSekBafpwhQdNC6oOawfSIG3NRGlYhdgF3KK9Wne0W3WNgLFkbID20iUW6JQw1qiOPA5VwPL1x7jiu9v3rlJL5ejgaSNCVFLVDrBrMb61lEegSxZoEttLpIYuZMMfTQgh1RErlgE6OQgnV2OXToec6B+kOZ6MRm68e/1jcW4ZoxNf0ZdIM8BCxu7zhmEmKnlY5KXNAkFKyVsI1HsSMVWXcC40Sh2PB5fE3H0hUDzdsBwWkAEFo6qJ/3c1cmQKOsRBRBKCZmyKLO5UBSA3RhKGIyE2VvYlSFNPUT0FxWKxxG+UjTxcYyqcbWC2ZMGz3lg0SY7B3tWTaY8Gb3TuwHjaTUVbW5BWUBeu+g9i4GoJOy6QPHQYzyvUb5rMZ7VMLsBbllAxAh1pOtw3h8kqEtM0S+cAwN1GJkd6cJElY4JeZQpiiZqORd265r0aCEmNC9qSSo3AHUYZqp3MgXKhj5Iz2XTnimyJVGdxZgos85PBVpUioWdVnxfQghFP7LQC4FuqqkQFNZNlOusaYylgRgs/33692n9llBFpWZqbKbBOk9qtBA0/QFm2nR+ZBrwzxnrZEptdKlgzUVjpvlmbWXw1FQKASg1G+s8QTyj9zNFd9rtn1J7/FlmPvm9dvxHvuZPe/wqFJaLH72If+l/+i9/6dv9u//ef/cXfmzAd5wKCw/0+xIQNM+gvoRf/k4yR808aKheJPoSC0fhwQKgBMxesjC5TR8sAxcBOVC82Iq0CE83WQ2Ye5WQibnwU32iVu1EMnoAzDFnxfGLq7hVpGkNs9GFCKRfyRF0qi0j9JHbLB9oegAkylJPzWN5k7LQtsmpM4p03CJ1yUWiEEku8k8xUbR0B3Lx5Wx0gNR5z4sTsxeTe6p+0iF3NRep5ZZ6uuJRoroBuis+T6QDsybVA3r/5Es2sS5ITUlOnjHNRUIGIDhWcwREEBPaEauEjAw0PZGONHuiCHx9MIDzAnKMNOEYMHUQyweeT7PjYisK7jsXH/qIaYEtIgu1vCgvRiRkL0VueEB5TIsxEVKhHgBIopWMrYlJm8eFte54Xn2J1HFPBcaQXUKTo+chZagmmpl0LJiimgvdjAToLjkRB44jX4863Qsz/Sqi+aKA2ccJAbGLtOCSbCJIx3nRbYRO3f9cEIYBia7FAiy7noaS0SmMe+E4zSGSEuNmalowaW46oiAIRHeqW44B4PFXt3EyxtFHjtuka9e0Ed6KNMcRMdHrIOkKGlN2bZRp7jXPo2tY5JR3nN8ouF9zjKnYi4hDQqdSE0P3EfVbvjdnNuojGwb6yCYTF+qcl1zQCpfcWn1ylU3NrnyfIQJo599FKqoyjUulc5ZpbCIA1X2g03OivBV77l93YioWfKARS0x/Fz4hJun8Cw8guWXqnnTWrP9WuXmVrhk1UutdbPM1ld1IBfyIdL7SNZmohXkcURJ1UjYVnYkOZ/acQxljQnZY4JSPicrmYzK0iclghOeKZiT8vDHpuIod0Sb1GKfr3hwBZYHqgcWISmOW6XyUO7qM5nzZ8jEkh1mOpdzy868Ary01AHqguyQpqBHKztEfxQGTJp6O1tSU8v/5viVCJa2YChRzFKlIium8RpiOWmNzYBHM44nTMXBhy887085OuJNLrI+Qkp97IlGxIRJ6GCKipJGJHmIq1gA1BBSR916BgNjO5ypTnaN8klea9OWmDSlnNSRDFOoTpWPervRzkadshL6P0/WnLM+fGgLKPVEwEQA9UHNsWqDcpdfkzGgXk0adBYrMObeDA2BIKQQmh1dpA4pDhGod7MpA9R4iUHML8P2+VDDHJ9sbWSQw6zMgO6UWeyBn4WYNv+oD5Ehttz46mDIhZkZCSAE1eIT+6f5yjq6E9AG6pfY5lDTQkS5C9h7CBmgjoDoHKQVEpL5OOBZFamRWL+mXYcoRlS5AtfNCPOeaysFDeDqMCoD7ze/3HHvW7OqWOnDRRgSnJhdY0xKpMxFJn5tzRCPnNVEzlaUTK88bEUs5OKisZ9ZEBREAEfg63ToI66EPlvdl53hcLaMt4OP0nPAeUSpmkY4OKuV0quTeCoupCMz7Ez6SHhoC4JJ7K+bXTVTOlLsqO8eIERcAESEs5iItO3Qn7aoAkTwhkmbVOhZ6MT7RNfI+EDnDcuR5mCM9yEgRzifNqJ/iRaZtZSpsP3IMPvCaGCyLvJ8vEAtQG5ofeT85P9TaqYh9r5h03N90FaWczJg1njFyf0/mY9p+nDWcMUaIGN5HJH+uePyFxeSfo5Dkm/8p4L/+M/L4TiOW1aeMG9mcHgEARnvc3q7w4vkj7vcLGONw2NbAwaC4bDE8Vjh7sUWhPQ59SXrq9Ql06eBuKpx/8oDdscJ4LGBqC9trwEr86NM36JzBN9+c003ztoQ4G6FUgBsVTOUw7kqIwk+/qwWpsCFIlCW1deFAaqowAfGoUZz1pKzuDOTSIgYBoXg+Xlxs8fYPnyFcjJAmQMpAauhmgP9iCfGDI+x1jeKqpRPoTbbLC0CnoDYW4hs6bcbas4gqPYSIuDzf4+ZuhWAVTG3hrELsFfTSwnUa4qDphFl6auUAiIJ6MHFXQB8F7EmA3km4VYCwAng2ILSaNMyFA3YG4nREfChQ3ir0H1rIylEL5wXkQUFc9fCPBQvIlYW8KSA/6ODf1ohFRKzSB3TOArECMBHFW43xIjtdAqqTcGcO8GKiWorKQzwYSCvg1p5jFCzK7SrArz3KNwbjD3pEJyF3GqoVcKuAWAUU7zTsmh/+qhfwSy5EzIPEeOU4FhWJjhcB+oGaNhEE6YWO7pjSJlfOIBAbB3VvoHox0eggiFYPF2mMkY2BUCS6qY5QR8l4mD1dYtkIoJunXbGoDQZTXAtpOhyHL+nmKZxAldBeXwfIQaK8F+heeMQ6YPNjg+6KqPa4JiXSLdjMUD0Xq+M6LfgMt89mB6b9Dxce+iBTHIlImXtgt7liUSZHAd9ERsyMzFvtPvDQOzk1MEQkCqqPCcnreE55IfL/dkV6lTkSoc8FunCAW0W4JkAf5NQACiaifscs2alBEJDietgMyjosObJRxHOB6T15X7oVU7SDXbNYy1RYadnkaF8ENK/kRB0WYW4oiICklcK0gAa4L0YCPRmXjnSPLYkYQ5Bm6eusjUPS+NBps71i42SKOGkSbTJr1xxpp3Tspbtlf8lx2RXnXLczNXM8YUGeNYI0aMJkRuVLUkOLHQtpu5wzgqOiq+hwlvJmF2LSkzFSiYh7ptPmWKBMpRtO074SQq5bItq6nampjn4znEfNxkV2GB1OMDWXWLyzUZddVRljJCa9IgRQPpCiSZMgTPo4NgrJ5BjXAvUNEdwoZ4olCzYeT3/Jc1PdxglBz1oy6YDjCzEhn+aYonuGhOTn+BCJydwrN91yM0pEFm9jQvqpCU3NoNRwY5YwpiieoJE0aUj5pkTL9cCi29Xz9Z+bGiI5gfpkKJe1feNaTEh5sScCnJsCUQm0lxLVAz8zo+R1okbAprxk3dMELxfP44oxK1EKZqUe5ripchdgm6Tf1HM0ke7ZJPKFmKKhlKUxXUZCy8RCyNpH2/B4ARrVZTM6XyYzsi4xBfwc7ZIpo5kSG4xA+eAxnCo0by2GMw1XChTHwGgWH6fok7yNqHj+i32YzNXUSORYxDhHoBhSNjNKmg3ydBemMdDMjxTU7O6Z41ikJSoJcPt2SSpqsXXcl6cJmrRx2pc5ehrWdWGKQQlGTqZy1bsBvqK7aVACrpEotg5IhbRPmsX+3KDYugktBTBRR4nk6hTlQefWnLk5nJcoHkaiqAcizb5WibaqpnnPTQU5eNh1QTQ1mX6Rlkqqp7QewSiijjFCdi7Raw2qdx1CkQ32LHxNNE/vekaUJPQzN3N8k5oXQ6J/PjEdEi4gVCy+JqOsTH1NGsfJAGdMCKQQgJJT8RuVgDz0NGeybjZDqsxkNgXrAE0jqqgVzZaqAtlMCWCTCdYxE9OHRIuV82tCMjOyDugHFqUhvl8E5jiT/JxSfyKuBABin5DPhFbyd5FMk9J7Q5yLyLy9n/8/nhSaqZj9cxWZMXzn40YWP3wRf+OfAGL5e//+rwZi+Z328i20w4urBxhN7aCSAX/lsy/hg4QxDi9We/zmx69w8r1HhCBQnfY4diW2xxpaBry7XeP0fI8QBdTFgIddA2M8Pnh5j+WCN8/HP3iHn92c4fXtCYQJ8K2GvuwhBLBZdSwkW4PlxRFCRXinUJ928INCWTpcnTH+IAYBvR6haofYaqjNCDcq+EEBlYeQgDQBoVcoSofXn19CftBC6gjfabhBAwLodyXciwHjoUD9wQHDtsLhzRJnL7aMZQBgTgfEALgzB3E6sjj0AvGgEYPA9atT4KYEegl3VyGmOAe3LSBawmJikBBt5l0B0UnEo0bUEeMLi6gj3CpAno2Qo0A4asijgmolYqtZzL4rgSDQfzxAtArxsYB61BCVJ/V2UFAnI2IREFsNvBhgdwVCSZ0cBonJlOJeQ/Y0YxkvUpFrBeoPDtQQegHZSsAJyIUFjtTsuQs7IYlRRvRXDv7EQYw03xB3BdSdQWg87IsR+igBKyZdIAT1dvCAOkpEA4hRQliJ4kZD9gIIRJpVzw/g4TRgOAtTQaMOCuZ+ns8oALdMnVGf9tULGoEIFg2+CdAHAXWU8E2YNGMAizrhiEaFeqYayTFRuhL6O26ICDTfqpkWGQGzk4gqMtt1L6HvNba/NcLVkRRsJ5DzUN06TAt93VGryBgJGqmokQUVdXsyxVuEVHBEhJImJkFhOtYcIaF6FtAmIeCZGh4VoFoxIbdRJ1RIsuiashItj9VXPL9IBakceYwAqeu647H0FyzSzIHzNKaiUHWAcKS8q47Nie7K0+G4JoIugNmAI5KSO+W+7jk2UmszE4LU8OGUBYxLuZs05JjpzG5JtE060sxtk5D9dC2ZHZHIbL6S6a9mTx0g0X+O5fAR58EtMOnKyApgEZuzaoUj6s/zwyJMjpyDnD+qD5iQY5WQcyAxFxrg8H1uE2kfruLvSPMUBRE9V7Oo9GVuEszIrD5iLtAXmFDg44epOEr7DUkLyYX2/LpMFdYtqcOczydU7UMqTDsWHxntlJaO3sNZzpDk6xGA4VROx8B5TsWUYDHlS2oFx3W6HsuZ3iVHHq9bCJT3McWJiKkADyUR8HFJFkPOzZUjx6gTjTn30XyRnJRdoiDHuZAWntuOUsAuSYU1h1TESGogo5qvjawtFWlbvmCREopp+MgZtSHdY2rkeHzFzwOyfARcJSbX9XKb8jmPnMdxSYqtObIZ4wsx7X863oSCQswoadDAsJJwVboGDQtBaZlDGlJRQyr+nHVqlzzOcpeosgNp1mwExemzi7EVPLf9ho7aMpvbpAzJHNUhIseCSBdxEalDZCYlb4TuUqdoF0WzG0F3deEZIxI054m5wCIh8IAv6f6dc28zpT4kV3tXyfQZHqFbIoEyu8abzKZKhZZL9F4Xp4JNuuSQnmjBmXHAOA5JZ+kxTqhrzurMjvO+Ig00inR/jRFuaeaMzITYx+wov6Brt12RGkw2VKJeR0xjIi2WiLKIYByIEvA1HYejkdC9T3R9IpwssDnfygZmWKbvOt05ZnKmPE+3LEhVTfmYIpvZ+IhQa4RCwexJnxUhXReGVGe97VjoJeOeYFSK8wgwDx1UotSGQk0ZlMIRmSOimhBQl1DZp4hodgFWCjAaYVURHVSKTsFCINZloto+MdFJuZ4xu9EKMe07NqSxidFCOE8EMwQgu+d6ai9j+oGULEyzG279JNjYaL43ayKTw20MYS4qk5stnOOPUkQ+tQKkosMtwNcIORWHQgi+P+tCw/toZi4qhRDUVoYIoeQv/IEU8082BvqOPwLEL/3nV+XxnS4srVd4/eUFbu9WKJTH9ljj8/sL3O8a9L3BV3en+OL+HN3/cw6lIuzXC+YligijPZQO2B9qhFGh/HED32v0vcGr12fohgLlNwW+en2OqrIQIiKOvKDVHy/gR4m72xXGQwFTORy/WSHsDOpmwPjVEhglDtdLvHtYJQOZCH9Twe8KVOcd/LZAcBKmZq5dDIAUEaII8F6gvGqh/4DCKll4IpmWXzrqTQkIoL1ZQC0soCIeHxf8oi498GWDGAWanxnoryqIQQJFwOLqyAWCT+6RCwesLXTjoJYOcmWhzgeiWQ8SWDjUpx3UwkGYMNFa1z8uIKxAca8Qb+gUdvnhI2KR3CcbBzEoalo7AXNdINYeF5/ccyEpmB8HK6G+qFHcsGg2n9fQK4vyVnHMZYAYJeRewz8fIa0ABonqTbK/7wXauwblOwW5sHS/9EBMCCgCoG8NYCKNbnoJ86hQvDXQB4n6rUDYOMSrHsU7DXVnYNcecmVR3tEUo36jUGwlYh3g1h7VO25bRMB+f4A/dZBLm8xV+MVf3Uqc/QHdMfUhFTgFgKWFHAXKR4H1T+YPx+pWQFrSZ0MZUL+V0DtJva+OiI2HWwY0rwVps+1c+MGEhBhG2BPPQlMgOWOKSccaNanTcmBBAQDFg4TdBLgTh9U/KKAGgWIn4KuA8p7a3Vyo6paFa3XNcRePkvE6Dnjxtz31QC8GNG+IRJb3wPqnfA9RT2A884iaaGUoaW6z+ipivOL8UZvIhaU99TM195B0ZoLU9Kg5T9GwQAFYxCzexAntdU2ckA3SkoHNT2hQ0196VHcRq68wFRXZ1CYUpEHWbxWatxHFY4RdB9iNh26JxJojUL+NjDWy1DYX+4RARxZMbkFEsXwgMrX6JiZ9IzWSEPyX5yiiuolYfk0tbLFNBksyYjiPE+rWvCE91TURds1iPRo2EIKJWH0JjCeBxVdEihvivKme48pU4ahYMAeVdLNtnJDfoIDuReACWXGOXMPtFTsWZ/rIhbLZx8mBE2CBN25Ir9XdvAhl7iC345acq+4qwDW8N6q7SJTWU+dtjhG+oqGRbnkd6j5yHGOcZARukVCRSKSyfsfxqIFIsxpYPOei6/AR52P5OtA9NOkPiz0pnc07Hnfel4gs0hCovV68oaaxugtE/UbSVoVn0VVuifocPxDoz4H6LsC0EfVtQPkQk6EN9brVY5zGP25m5NPVnDs1psgqwSKLVN8AlSip9W2gljbJLYbTJKnwLKqqR15n0pFOnYu5IhVh5S6hHZ4FU0aNs27YG+pV6cJJRFj3EfUD9ZtEZdJ38YpNoHIfUOxZeBfHgMW1g/Qp5sUnvXYtJq2k6gOKnYfugOUbx6xCw0JnSIU8jXF4LGqkDpZyAF7rwgPdGXWiviQq3J8zOzUooEyU0+oxbXskrTcbyWRqt22ILklLva/wQPPOQqaC0yREb1wJLF6PGFdiyoEkcphdXIng1Xd+0mYixlT8+9TMIi3W7H2KlJpRYgiwQVRLSE8092mElSvF1ChR3Yx6zsZxIbmmOthaTjFKaggwB5dMjQJ8zULebEe4RkJ1AcXWwjwO02uiFFCtgxwD9MHy+JcJEe08Y40cEchxpaAGPyHtxU0H1yj4UlJzWUuMG03Ka+8Yi3SwGNcK6mhhGw1hA9TAOKcc4wMArtapcIyTi2qm9krr034ZFaQfmduIRHMVNkB6RkCp3hF9lQKyTdEwKpnrHHvqSFMxGBqD4fkCvmF0z//P3psEWbLl6V2/c46Pd74xR2RGDi/fWFPPEy1kgBoM6BawYMUSwY4dC/YgtsLYscdMBjuMBTIZwkwmIYmeu6pe1Xv1psx8OUTGfCef/ZzD4u9+I19LamMorKuwvmbXMjLiDu7H/fo93/mm3lNK67BDYQxdaLrFbwWBhj5wpwOZLuqqYaoaWqn12QKvSuqsVFYI6OurTAKzZRtVX58CIpt1DlV2wTU9KI3DO8YzjvBJLNLa/vWaFupGpLx1A1W9rWTBOogjSXwNAogjVBKjoujOc9lLYcPwrlqkZwydxeeFAMqmFsbSmLuQnbq+81pqLf5V7yX0p69SattOjuu2P//5O9YK6G1a/Nv+0r+6/Uzefq6BpXMK1Si8VWy6xFeAJGlosog0lv83E0d5ncoKs9V4r1hlCW1taBsjtQ8zjw4tWsmXZRrXNI9LTOiIA8vefA1WEVyFtInHxBbfaMJBg/cKP24x04aiiIQtiRx61KC0l3TP0AqY8lBdDKS2oNG0jaHesduqBqU8TRZR3iZUO06SXRVSLWA8OrYEpSJIWqJ5KWA0drhWWCjbatp7cuFpB5524GFWQ63JlglR0mDGDWiPDjzpUBJjbWFkdRuZ4FS7ViTCjcFuAnTgsIeVfHn1zIRDwGOuKOpQJKGlEmDXKtAweKWw90ownqyM8LHHLyORRg1bTK2o91swXuSgWmSbPrWozMCkQR+UxMNaAogqSeQbn6yFPXIde1cbAbUehpMS32hUKyBJhQK2+qAmF0IzEzBEq3CV6SSLEnTkslDYnMHdxE2FffE96KCTeOUBZhFs31v8q8I+VHORynrjsfMGmzp8K2XtzbArKW9lIHUjx8omQORILyT5sjxshYWsJJnWdXKvPuHTlAg4LxEfV9MdGM822RRg8FqknsmNE0AXdXfTjUkkq8jRUibQPrqTo6hGSSl9dsfQNeNOjhsLSClnRhg2JYxMz0Q0w7vQop5RhM7vO2+6fZeT7o7dRBhF47fSPdf5Xk0hjKf4WjupUgR24LpKDQnC6vdfwfZ4FweuY01FOv02C2QqmbC2O802lVOAi0yIJTlVAG89ldcRSakSuXInWQs3bL2+vbxTugxFgik1FML0BHk3yR50YUtDRbEvyaLtQOSZuhEJsI07sJp2/tSOdewDeryR98wPlbDRSkrsfdAFysR9+JXsc7VnZbGgZOszbSaKZscKo9T5lpWTsZWKDAFBkjIrbGnPegWFgNEw460UzE562TEpyss54CLx1uLBTluSa79NK5XjLeew78LTer+f+EaF9Qsz7nyorYDVMHe4yG8Zuv4a5SLZFt0xu8FG2EPbgZUwEyDcs0vKCeNVj6ViQgKC5LrRpgJcmlHHFvYJyZ331QcCxqLNXW+yV+Klezu4TJgctn7ffsx05yeUa6swgaq9893eBbix7ZbtPxt9WFpYSOhbNVXYUMYzqKSeIVq7rZy0TdX2Gg4CeLae30KAVd9t3MuCxWcnwL5nI4X5k/2QFGBPtBFw0fZySCfHqP+8b/uJFfhAEV+X1FNFNTXbz7u8p3zGe6+3ct01QcnY6ebuet4fN9spCXq2XYCe5C+0iQQP9bUbppTn9D7ksPAd8Oy8n4V4JyXcrpOvVqJEKXfD7lrWUo+NzC1CJYsomduCVRf0tTGdLNbTsY/CJvYVGOLrb2i78KNw026PiVd3sldn5ByUWheNTfW2G1Z16aW9/1UApwBUk9ciq/ZdsFl3frlA4RKzXbjDeewgFBlvZXGR2gIrmwTU024RuAv1EYZP40yXTly0hKtazo9JRHxTi0e1S2kNM3vX7aqAVlKBfV+34RGWz4EpulCkUK5psj0aXbXo1m3ZTACbGNnfxnVjLSmt3gjT18uLXaBFpttLMLUSwFo2qLLumF3ThfNICJFq3J10tQOPsq0CasXkKxJYH2ho7TZRVueVsI59iE3X99p3s2KUAD4tfyfQIsnV0sPqIklk9YEWgPhWH2jPYG5vPSDVSn7u3zcMtt2nvMVgfuO5cMeY9v2exqD613ibJTSGbVKsNm+9drAFijI8Xh7bv0/HWCpjvslGdgyoPFb/hXdlzF/YdfnzcvMguOGnfP9Zuf1ceyyPvrXj/6e/N+F/vPkNik7T82yzwy/PX7AXrvl7b77D4WDFL0++5kW5Q6xbPl0fsp9seJHNGYUV07Dk45sjfvfej/h0c8SPLo/YGeY8fbnP733nB/zx1SnzpGAUVuxEOdYrZmHBokn5Bz/5kF975zmt03xycciTvWtuigFHwxXTsOSfvXxEsUr49juveLWc8lsnz/jD8wfspDknwyXjoORPrk75N48/5e9++qtMhiX/2snn/OM3T3gwuQXg1WaKdZrz1zPefXzOpo44HS/4+M0xTR2QDiqOx2teryY82bnmi+s9xmnJukj4zuEZP748pG0No7QiMpaLxQitPWFoSaOG1gqY/WDnks9v91luEn7nyWf8+PaI60yYz9mgoGoDlPLcLof87gcf8798/i3+7Xc/4R88+4DvHJ3xJ88fcH//llUZU9Yho7Til/Zf4rzmh9fH1K0hDlsOBxt++OKEo70liyzFGPGh7o83PHu5x3BaMh0UnN9M0MozHFQsrkYo4zk8XHC7HvDh4QXf/8kDvv3+S3787IQnpxcsipTWyoVxtU4ZDCumacnrN3NMZPnu/Vd8cn7Et4/O+Px6Hw/81skz/umrx1RVwHhYohWs8xhjHIFxFEXErz98zk9uDradnZOk5OnrPaKkIQgcSnmKPOZwd0lkLK+up4Sh5btHZ/z+p+8QpC2TcU4Strx+sUs8LfnO8Rk/enPMyXzJ69sp93cWnK9H5HmMd4o4aairkF988IIfvj7BtoY4qWlqQSrvH13y6etDchqWPQAAIABJREFU+V7OA7797is+OzsA5WnLEF9p5scrjO6lJp7LF3MevnPBbZ6yuhqKP7STLKvcoKY1YdwyTCtuXs0Ip9LJ6i8SvvtLT3mxmlJUEXUdSB3LUpjq0eGGk8mKz744RrWa8b0VqzdjvvvR11xkI27XA7xTNKsIVRr8wEo/5ouE5qDhtz/6gn/2+x8S3suo1jHJpCIILM0Pp4x/+ZqrNxPiSUW1iTGxxRaG4U5B+XSMHThOHl/x5scSyWuOC9qbZNvnqCuNG1rCUY19k2KOc5pFgrKK4cmasojwZwnxI/k5/DKlnnV1JVYxONlIxc9tjFlr0g8XZE+nsFeRDqX3tLlM0aVi/N5COjpXEcODjPKrMXYoKdScFswmOVfnE8JBQ5w0pFHD5dmUwVcR0W/c0FhD+dUYf1jJd+irROpaekA2aZnsZDStofp6hK7FO6xrTbjUtCOHHTrCW4ONPeYkp8kiwkGDe5WKDHmvwSSW+Ecp1Xdy3HWM3qlJ0pr8bETy2lAeSBck2uMPKsZ/mLL5zRz9dUqz10gQRqukd3LmtknbduTYe3TD9ee7Ig0OBIy6PVlAUMaR/DgVb6yGZqclfR5SfVhI1U/kiS+MVG28Vqy+V5N+FWFjkd4H1yHtXkP0JqRNRYJtY090K1Uo7U5L8iqknjlGzzXZiSe5llTmXnKMQ2TeDuzIMXxu2DxpmX/fsHq3Y+c7j3Q9cwxea4oD1wEVtQVZQS4LNV4LmHJdKJpNBDQr1/slYfPYwrSB24j4WpK88/uW8FbjA0guFZt3LKOnhs1jS3xlSN94Ng/pAps61rHp/NIjOT4u8WAVwxca3UlHq5nfgt30QlEcdP7ctPcKS0r49DNZHFi957YVPZIurIivFdWex0YC5nwAg1dakrL3RcVQzx0739cd+yiLYumFAO524NFWvLvVXIBvfCWsfbHfBapZkX2Ha0V+7IiWeusPtoln8qUsdPThY7Lw1o19BspLSrYPYPi6B3Cyj9VMEsbDtSwK9eMBogToA+iKg05CeyP7qGth2ZuhLOrEt578+M6fDaIsqacwfu63nuP8UBZWoqXfpgj31TPDM0c1FZl/ciOJyH3tigug3BOvbpsKM7y+r5k8tzRDLanKXhYjRq+krsamivjWEWWOfM9gky6leqyIFxJ4Vewr0qu7upgok9+vHmlGLxzNUDF63XLzUcjwtSMsJIF4cOkodkWmaxr5rLQDxeDSSaqwk/ClaCM1J5sT6bsNM9el9kJ605IdBndhV7XfgmBlIVpbbKIJVy3ZSURya2kGkrK7Po1IFlYWKWth5OuxxlSe5FpAaH4YktxYyrlhcNFQ7obCvMaaeNGCE/9o72ENcmHseml5fFPRjKNO0eNFxts4yp1guy993YkLBVTW84ho1QUMlS0u0LhYkn2bUUC4bsTbCR0LqzsWVMC7MKYWl3bhkJn4Ok3RSKhR2eKSoKscMei8EYAaanR9F5CE99+oJlHWIlUi7TaR1keBgPLusTjQ6ww3Hgp4Lmp6jye2l/LarWz3G52VIOmzvV+zrzXx/g6AVrUwmN7ji2ILLNFKGE0nj/VVvQWUfWAQ3Elh/2X1I38++Gd7cw7X+zz/H9x+FjyWg/eO/Yf/7d/6qb/un/7uf/2Xvm/wc85YKuX5s/IhGxtzU8uS6LenZzTecFFPmCc5h/Gai3rCPzl/h1Wb0nrDF6t9Khvw+fU+X2/mAhxNyapJ+Dfuf8bvHn3Mk9MLrqoRjyc3vF5N+Ox6ny9Xe6ybhNZpLssRv/DwJas64Syb8GTvmqyJ8MD7ows+XRxwMNnw2x9+wUU2QmvH63zKTppzmQ359PaAP7h4iPWKTzZHJHHDLC34h6/fYyfNWVYp8yhnVYgefrSbo/HkVUSgHGUW8bsffIz3iqINORhv+Ho5oypD5knBg/kttTU8mt+itaRGXq8l4CcM5YK73KRkRYxzmh+cn7DOY9o64B+9eMKqjKmqDkxmqYCTlzOOd5d8tdljmNZ8ne3w26dP+eJmD4CbPOVovCYILIt1yv/6yUecl2N+ef8lZR3y5s2M739+SpLWTGO5MHxr/1y2pUjYP1iRvRmyLmNOdpcMBwJukklFELdc3kxI44bSBoTjWsD06QVGOaxTOK8YxjVKQfZyzLqMOT66xXvFpxeHeA+fXBxyOlsAcFZMybOYJo/QCnbSnGoVk18OWV0PcV7xTz97Qho2ZFcDVlnCqkzweUBTB2QXQ5xTuEZTtwHLIiGOWz7Yv6B1GmU8u7MNi8WQsy/2iSYVcdTy8dkxs1HO2WJC2xjerMbUtSzFu0LGHOAHL+/JtfUspshidmcbosjy9HoHYxxh1KIiR2MNbW1wTkMWgPEsV/J5uLkdcvV0B5W2xKYlyxKwimjQEB/k4mP18CuPv6YuQm5vRlum3TUadVDy/U8fsFwPKM+HAioXMYcPbsBDnsXEQUs4rdCFomkNZtzwxeUeN8shrgP7e/eW8qEN5EukmYu8+/vnJ3jjqc+GUGuSqCEKWuqpLGLQaqpFAq3CloZwVFPkkbAMsePsfCasxcjSVgHBTolLJFDKTVrQAoJ96JmOSszaoBrF5mqIrYVpKDaxMLHTnk6C5Chjb5TdJeVp2CwGwjjUhnyVSNepFdayagLU8xQzqamrQIKEUos9rpiMc64uJuhlQFsbsnXC1dUYvPROLs4m5K9H2KnFZSG+Y7q8EVlx+mANtWZ1PaRtxR+GFgADbCXLw4NMti+RoC9VGJoy2II8vQywWUBxZHG3sVQVVYb85Qg9rySUqQHuFQJeioBi38NZIimJVqTXPnRUjyvcyG593TrXXD6fb8FZO5KALJSHZYjLQqkUqqXOSW86dmod4kMv/aEdq71+R57ng44FWwTS++rZ9lLSqSW2iaUbmUi5UUu52yXPAl53rFbits9VTp6fHzvChaHcU9hJF7B1r8RGImNuUwgy6VUVttVvt6OeOWHKGrbVRdGtEmnyzN0FHa01+iIS1jfyFEdOAplWEurlA4hu5JjGV7IoUO52DKiSwKje8+y1qAlMIR7lHvgVh37LJppKWNzsvtuG69QTR37kiG40wUaRH6kusEm8kvFN5+Ouuh7LTBGutfSnroXRa4ayDVIzpCn31Bbs9UnrPXNqE7/1AffMYbXTeYLfmkO6EOJrWRiodrokciuMvqQNy3FshxLa1O8v0EnPBWD2vkybdHLpLqAp6Dya9USOd3ZPfIrt8A74u+iuIsgr8VTbmK0fOFpBeq5IL9S2TswFUi/UV0aFGwHQxU7P0Mr+5wciiTVdUFEvzbeJ7KOoGYRprEdyjlUzAZV97VCQ3e13z9A2qVyzcbLt7bDzshoBxMLmC/h1gYxltPBbP3AzMoxeua03c1sn1gr7rBtRRvjudfrj+jbbCnyj1unteppqLseyTbSELnXeeUnWtsL2+l72LYm5faCSqf02QVmqiaSipU1Nd86I1NmFwj7r2gsL2m2XjfU2GElZj4s0baq33s12YDq2XZhoZT2jFyXRWl7DJvKd4CIJ2IlvarwSZreexXcsaNClO3dpsm1q3mJG7Zalq2dSf+ICLenAQ/F4utCgqq4qqA8ach0gNJKmixP21cXBndzWe1wSCIjsalN6memWGe1lpyB+TKP4Rn9lH+jTM4mdRHV7V289/m3vZ9Ol0b7NYmp1J3XdfrC7b03voG2/ASRVEOCtxVt757nsgW1/72Sv27939/55P89k2N1N5qs/7fvPyu3nm7H89o6P//P/gvnBmqPxmkFQE2nL62zK+XLMBwcXfP/LU+JRxb/68Cv+8fN32JtkPJrc8MVij50052w1QWvHYjkkCC1J3HA6W6CV54df3WO+uyEwjqyMOJqueXE1o1nGTA43RIHldjkkTmrisOVovOYyG3H9bE56vEFrj1aeuglI45pNHrM7lQTb2/WAtjVo7WivU5LDjDCwrM5HhJOa5jZGjxuipMUYxyCuuTybomORzaZpTVUHRFFLXQekcUPVBBjjqGv5t3oxQh2WW+ZvkyXMpxnrPME5xaO9G55fz2nqAFdIoquZ1rhG4ytDOK7ZmWYE2nG9Hgo7t0zkSyh0qOsIv9PAJmB0f8X6egiVJt4rmI0KLq4mqKsIN2sZzgpheC5SDt+94s3ZnNE8J1t1RnKv8KVheJCRXQ0kobYMOL13zU02IH8xlknQozX5KmE0K9icj1CDFt9ocCJrREEwbATwbULSvZy6CrGrECKHSYT1otVEV4b5L19ysxrQXqd444muDP5Jjj1LcbMWSk18ZahOGpG2LAPUQYVdhQTTmnYdohKLOY87T4XHDS3JWUj9boHbSOKv24h0SllFeCtprj4W2a/JNO1+A6VGTxr8dUyQS+F99rhjfLoJvE8c4XWAvVeizxLa/VqSfa9TvPaYSSM2h1cJ7VCSUduBvFf6IqA8sfhAZlG61AK+ai2eVjpf3lzSa13qiA9z2q9GMqkvRUYpYQte/J03obAuj6VrLLo21Pst6YtQ/Iupp521kp5rvAQwAW7colcB0VJT3mswiwC7V6NWoUge5w2jH8dkD6SzFQ3tSJKAkwtDuS+LI0GmaSeO+NIQrmH9rRoqjRpY1CLspK+Kes+SvA4oj1qUU4RLTXSr2LzbEl0ZkcglIh8Ou4TaaCETvHricNOW9GlEcdKSngW0Q098pcgeW1Tdh/7QSaEV9dyRXEgKr6m63tTOdxmuO09dDeWRZfZjTTVTlAeOcCUVQcWx27JUulG4wDN6IRPK4sh3nbOSEuxiCZqKrzXVTleNUkgPK0jaq3KShGsTaIZ3FS7JlbAibSqAKCjEm1vPHKOnmvU7junnms2p+FejlQT11F3/bDPsOv5aiK8V2QMBcLqSMRg97zoSM8ge+K10t++hjVZdcA4yPvGNYvNAnteDCd11oeoKyv27Pt4wh/xYKp7SC2GY7uSnkN0TJs0FkF5JCEw7YtsFmh8pJk8dV7+oGL6U7Ukv5Pe6k9gO3ohcUibvbLtK2wEk1z34kzEQ36FMjstdYd/iW6nQsYl4A70RmXKxLx2t1RwGF+LJTK5kHMo9GL7wFAeqS2+V90yu5X2qmfSLulCxfqC2/bNB3rGzA9newZknO5HzzcXyfBvC+pEwmrqR9xi9FC+npPX2E/u7VNZmpLb7qawE5SyeaKmb0ZK8Wu2IvLsPV6on4qs1tWf9UIvfthJQFW7E/1lNFeOvG6pZcJdo66SzVAKdZDyH59IzaqpOau9EChyvHPmBoe+g7KudZPy1AL6u9qqaK0YvHfVIjm1YiJS4GuutdLqvVDG1Z/3AsPNpQz0y25qUamoIc0exI/tTTxSjM4szXS9mz4yu5QcbCTDqa7yaodom2WoLw1cV64cxpoZqIkxiNTXbqhYXKMK1JTsOSRZWgn24kwInNw3FfsTgvCI7jqU6ZtXSjHrGzdEOBWglNzXlTtQxddIn2qYaGyoG5w3VPBCmcNOKTLdLLO1rVnQtUtxm0ieqOkl6HRuiVUs9CWgTYQ4l1VWYSRcKqAtySzWXgJ820aTnFcVhTLxoKXcC0itJeI2WDTYxNCNDvBCmsJ4KCxotG+n8zBrWj4ekV913Y5cW6yItXtLOR2lTs/Ve4gWghlm3fZEW8Nk4qVqphB1sZzIf0b2P08vrBpuGdhRuPaBegS4b7DhBV1KL4gN912XZz6u9p50khDe5AL9Ao6oGl4jCTvWhP10nphvF0r/Zs44d4ykX8i419u06Eq0FhGmNaq2E9YD8rBVEodSVOIcfJPJz93hau+3G/EYqa7/tqmMe++CeuEv6KisBiWEobGR/64Dk28mu3naMqDFbsPgX9lf+y2Sub6fF/v8gFXbw3ol/77/56TOWP/ibf/svfd/g5xxYTj449L/x3/1H3BQDWqtZLgfowDGf5ORVSFWF/Lvv/4hn2S4/+vqYKGlp6gBbGIJBS7uMMNMa7xRHe0teP98l2SlJooZpWvLiYk46qMluUlRlukAYxXx3zc3VGKU7jQUSsGNXERjP/HDF7ZsJOEW6l1O+GuEnDTSa0X5G9vVE/m9lWfrw3i2XN2NMYGlWMcGogZcpweON9HQ2SmSEkwa9CBk9XtL80ZzyvRIWshquTwpJeE0tOrKiuW40aI9ahfjQgZHtDZamqzUQD5MdWQhEWmXWBjuT6g6UlwTUVuESRziraC8T8eX1wSjDBn2WdOmbijb1nffSC8iYOfyoJTwX32K0VNSPKqk3WQTMvnXN9dUYtQhxQwEtqu58LYFHpVJdYnItPsXIyz7VGl0qTn/hjGfP9wmuw26C4LETS3QZ0MyEuRJvpsId1PjKoGKLzwN03rERQDuVACN3GxFsNM3MEi5Mt4rtiBa6q/SAdmzRlazoOwN2aKW/sBK/Y70v1SG6EjYrWsjqqg8EeKpuhV/ChtSWQTG1hPDgJN3U7JcEPxlQ7VviS2Ezen+djZFxaCSxFH+XJOpCCVepZ7Lin1woqh1hEkwpUf7Vrqc9qkm+iLGpBNSYXGpboougS4gUYAPie2xGnmbHMvs4ID+WCb6uZFtt7LsVc02bdOdAKExRuNDbffMd8HIRXU/eW3Iz1cn/QtnOPsHUDvwWuNVTAUVtKuebKe/YmXAlIFp6PxXFkSNcikwLJcBl+FLRDIV5KPY7dvJKtqevtAD52+CNoppLuiiebdJtfCtslnnrOxXe8nV1QCl/3GCWgXg0vYC4pgM28cLTjOX86SefLob4uku57RgoYcfuKk7agSe56pnKOwbIa3mcrroO1bIbu1RketVcjl+06oDWpWf1RFJn4wWo1tN2vsG+2kJX4gntfbI2km3VDZS7AgJ9cHc+hhuZ8Lddz6JNIFrIpL+aS+hRnz4pbMOdDLFnOYJCEmJ70GJT2Y5t/UbfzenZspUg7FafttkHvgSZMDAuFDCiu7AZ3QjQCHLx65a78hkMctmO5EYAmk0632LuKWfd+di9r42VMFEdI6MbAWRt55+VNFC+MZ427vdPJIw2Ybs9LpJrgOyM/Ny+VR/SjOT9+p5VXcvvwq5T9e3zD2DxoUiKo7WExvTXDdQdwOnPr7t9ujuX+uClZqC2Hk9T9768nhlUxGs5Lm2itiFYfbVL74eMsq6yJBJQ2vtPQc57SZ8VgBeUwt6Fud9W2kQrR7lzByDD/K6SpZjrrgfUUw+1hBBpkaMGhdvKYU0toTNh7qm75/aVItIz2X2PN9LHWewG287PZCFexzaVmo1iP6SaKobnDhvfVQ+hRE4pYUeaaC2T+z58J6jc1tcYVA5T3tVDiZfbUu4EhLm8hmo9NpHvQhd221dLBUjfGassxLc1xWFMkN0xRz0glBqTjo1L9RYQtqnp+jm1dG8aJV2eodqG9vT720zMtq7EddUiqnWg75jlZiyP6eWlpgOiunuPehpIFUkXuGMTAUBephndvslJ4Y2kwdrYYIp26780eY0dRRLa0zracSSgsHaSOms94XVOvTckXFXbsWhHobx2J/dUtgvXsR4fm60MVTUON5AaDl233wgP8p3XEbgDj9224kCXtbCGgZZAni4sR2cFdjZC5xUEhnaSECxySYntfZYg4NE6kZx2zKJ4L7u/9aCrkf5KlRXfBJlKiXS1bkSKG/cL2e4uyTYvBVz28lbXvV8HDre9lv1+dvvY+yv7mhHV1468jR3q5g6A9gmzbz3G17VIYa2V57/93E4K+7ZM9s/f3k6Q9d7jq+pf+Lj/K7e/Apb/399+rqWwHsWr5RSlPHujjNGk4HT/lt2BsIIPD274+19+xMfPTjjYW1FXAeNRwenpNbNJzsMnF8wnOdNJzptPDtAD+UC3TnO5HuKyAKMdv/LBM5LDTJJhc8PNqxmDSSnAUnuUkZ5JlbZEk4rll3MJphk37I0zhg9WHB0tJNAGmD2+xcQWlQXE05KLp7vszDKc0+i0xVmFPaipXwxJJyXxvCS9v5YgoHnDepliv7shHjSMHy5xe1I8pncqwkEjrF3UojYBWEV8kpEe5ESzio8+eEn4ZI16nDF774b03SXxbkE4qiFwpO8uUYUmWBqSnZL0ZEN8uiGcy0XFJ47o1jA+WhMsDN4LeJq9d0O91+IGDn0/J9wvaMaOcK2g0fhHBeOHS6oDYVzNMqDdr7l+Nsc3WkBWrdm7t0S1inC/EDmcU4QHBebxRkBlbAluA+LDHOUUr2+mmEWAPy2wIyt9kwraB6WE1+xWJMcZduTwhZFKlNcxyiriGw33C9SDjGAR4G4kvbbZFblbc9AQrhTRQlOd1jQHDcFGsf/gFq896YcL7FBSZPEd8NmzmLVh/KUherRBN4rquKU5aPD3C9zAkl4q2pndhgv1/Zn1zDH87k0XxKHgZUo79LJNY4cdSImxKRRu4MAp7ElFsJHnl4ct9bsF9UFLO/Db+oD1u5bqsCW9kG/xzZOWZq9BLUKKRzXN/Yr0zEhX31VAc6+mTaHacTSHDeFabUG12WjWj5x4hA5b2pFn+ELO6+Awx3ZywWilCNcSJmNTCZGq7jXU+y3N2NM8KTqGCtS31iIzHPdJkYr6wwIUVHsiOaxnjmpX2Lk29diBp9yXhEWCu9RbF0sdTX5q8dp34TieatcxegaLb7XkDyz1pJNsDjzVXHoji9OGzSMrgJOOKfMimdy812AaeX0XCCMGUO45qrkkt5b7nmpHnpuftgyehVv2Z/jSU81AahDE46UryB+0xNcSuFJPJJnTG/HDFYeezROLrjtQuIbBmaI49GT3PdXMUxw4Nk8ayj1Heq6258nWl+Zhc8r2fPBK+lWLA+kQDbtQoHJfvH71WABPfigLBdkD2+2np5l6wpXvOgO7DsukA7Q9U7enyO67bb2ETcCm4iUrDjybU2EZAVbvOmzSA6hu/GayeGBjYRzrjm0t9gT85cfik2vGcmyKQ0lv9Ur+5pXqWE1YfmglrGjWbUckyanlnnj+JHwJ8nvdNWOoGL1y1GPxwm3uC9janGhZBAil8qLYEwakr/AINzJWzUDSTNGwfqRYPbkDpm8nt0ZrYSjxkB8JYN2cCrisdhTrB2xlks1QSZhTF9IjSb8CjAaXjvxIUU0V5Y6i6RjRZixqhzDzFHvqzgda+rsQp4Ewnc1IwKtufFdDche41QyERWuG4iWsJgL+oo1nc6plLGeSUlrsyfUvXnbnf7dfyUK8e/VEtm1zosn3Nc4ISxdtnAQvBWyDi+qJohkqNvcM9VhRzYSp90ZY3809I2AwlH2rJorswFDNZRttrKROZipSTBcoqs6vV+yobW+sV1CNpS6kGahtCm2batpOVhuvXAeouuMfa8pdzfDCsTkRdjEsxLvYDGQ7mqEmKB31xFDNZBFX6lRMt7Ain0tdO/L9ABtril3ThSppqokwjs1Y+hA3xwF992s1M1Rj2Ufd92+GeivfDLK2Y891N16aYi8kyFoJ5Gk85V5IUFjyg5BiV8ay3AsxlSXIrVR9GEguK2zSp3/7rstYC8jvqkBcJMcmWrRkxxFBYTG1o54EIvGdhthEYypHPQ2opyHFQYwpLfUkIFzV4pUsWsr9Oxav3IswZUszCalnIeV+hEtCbGzQtaU4SQW0dmE1PtDY2NBOU4JNjR2GtCO5y8KBIT9J5Po3jahnMS4JRIYaGgkEqhoZS6PwoaHeG2BHMe1EKkJsGkqibGhop1IR4qJAZPKDiHaa4OIAn4b4KJTwnckAHxsBUq3FrCsBoLHZMq2qqIXNBHnOMMGnkQT5dKDODWJUUW0ZSN+xh246xMcRbiLWFzce4uajLWDtq0nwHj8Zduyku2MGncePh/jRADVI5d+h3O8CdUTuuv19D4Z7EGit9GK+PTf3XY9lH9gTRfKvUpL02staO2mrPNZvf/bWfePeM5beum8A15/n21+F9/yM3pIn9/zv/fd/k9ebKbdZyslsxRefHzM+kuz7tguMeTi75fufn/Lo4SXX2YBB1HC9GNEuI5K9gqoIGYwqvnXwhj/8+IkwVQ8KDndWvHq+i0os42lBlscMBhXeKzbXA4JBK9UlUUv1dIw5zanzkHgoPr8qizDnEe5eifk6ob1XYc5jbOJg2jCd5SwuR+wcrrj9ei5gZT+nbTVRZKlrQ3uV4gNPeGtoRw6zW+HPEsz9HPtyIIzYYU7zcoibtKgswA8sNIrxyZrNMsWcx3gtKaRBqajfLeBKfGc+lsRCFUqXZHRl0B9sGCQVy5/soOuOfRgIWxgsDPH7K9T/McX++orq6xHqqMQ8TWkflJgXCe3YYQqNOymZTTNuXs5EzqggfJCh/3RM8WFJMqgpXw/xoWd+b8nq8znJpSZ70jB4GlLNO+YrFtlfM5Vux3pfgjqaDwrCn6RU71SoW+miDJYBXnvsUMJbknOR4jVPCvTLhHZmJQXViGYuOIsIV4r8YSv+u867AgLuxr9yxdXZFLMS74UPPcm5oXhckzyPtv62ZioXu6Rj+/QvLeH3p9gEygeSyjv4OsBFMPq1K5Yf79LMW6KrAP8kwzYG9SYhWiqqHUd8qyneqTA3IcOvNasPW+JLQzPxhCcZ+odjQFjn+rAluAkwtUjcmiHkH1VEX0dUe5bo1hCuFdkjSd+NLgJMV1niImEe62lXN5A4Jp8E1LNu0ryG/Ndz3Bv5ggpyqSsJVx07O/XYo5rk85h4AcuPLJOfGJbfFn9jcBNIRYiB4SuZzNczSC6h3JWxmfwoYvVhw/BpSLnrcbGEg9x+xxPfdF6azn8VdPLH5FpSL4uHDeOfhNgYimPL+Cv5Iu77Hm0kjGp6ocjuOdILTZDB5oFM9pNrRT0V5nj6ZefZSu8ApSkUw1eS6Lj40DP9TJMfe+pdiy40g9e6Sw8VcDV4rdg8EGbehZCeC9iqPyrQzxNJDk0d6WtD/l7N3v8ecvs3StwiYvqJYfGdlmBlGJx1vrEuIROEmQ1y6TuNVp0PrBSJYz1W5CdSOWITRbEvY9amnuErSWYtO5Ax/cKxfFdS/YXcAAAgAElEQVSjGpEq29QzfKmJllIT0bNoq/cdu3+mWHwA8Y3IDDcPJXBm/MKyfGI6wNRNnHcdwxeaoLhj8poRXZKsZ/KUbf9ofqQYvJFtim9Exjp56ij2NMM3jtv3NcMzAfHVTDE4FzA2eC0TcxdKKM3gzLN+KNs8eiks7O7HAiCShSc70oSZsHbNWN7LBwJURy+FoZx/3rB4Em77E3UtcuPdjz03H4k0eVvBEQkjKYy72vZK6pZtT6RynmgjlSU3H0mHrEhGJcRlc08zuJB06GglIG383LG5rxm8kfqK5TsBybWjmsl43vnRBMhWcwGA889bSTsNodgVyXiQe9Jrx/KxYXDuOwbSs36gCXKYfVFTTwPyA40pRQ6bXkpCrgvkc9UHxaweaaZPXScBFV9sfqjY/34rQH0i19fkRs6dvgYmWouvsNxTjF9ISmrb+QqVg2JHEa88+YEcn2jtyY41uobRayvJw4XIkE3jO/mp33Zb9hLY9FbYNhC2MN8PiDLpaKzHBhsrkltLPepAceZo0g5wRrLdppHzI1o78oMAF8DkWc3ycdT5k+WYRxthO8evpFqjl6I2qWJw1WIKRzsw+ADyfUO0Fka0HmoGV8L41SPN6HWNjTXr+wGzL2uqWUBy05AfRAxfVzSTzkjqoNgLmDwrBXzuh1J/UlnqWUQ70EQrCfsJCvEb5/uB1Ks0kjg7OK9wRrM+jRi/FBAXLVs2JyHTZ8L69HLa9f1YAnciJYnFRlhfYW5ljONbYbfqWUgzlNcvdyM5FuclNg2wkdSk9Iynsl6CbjYt7TAgWlRUuzHRoqHaiUhfZuQPhiSXFdVOLOnIm4ZmHBKUlmBdo8qW6mhIuGmodmLS1xnVwWAr2Y0upWLEJQHNNCIoWlQtXZa9TNXcZLR7I5GuNhY7lu+0ZhR0igNPeFtuw3J02dJOEnQt/ZfhbSEAcxRjloWAPSfVI8GyW3jvg3a6IB1V1Ki6oT2aSWrvphT5aVHjxgk6E6ApfkqDzrvXiYU5VVWN7zoofRLf/d95+TcrhIXc5PjJCJUVUssxHgqQ2+Tyc+eZ9IGA3G0VSVXjm+abE+ueHUxT2ZeyRCXJHaAzRra3rqXD0hipHOmktL5thZW0FsIAX1ZbVlV+9xYI/fPBPX8RHnnrsT//4T0n/t2/85/81F/3h//ef/WXvm/wcw4spx8c+oO//Z/x3uElizKVsJbJis++PsK3isG8oHgj/rCDd645fznn9OEVjdNc3Y557/iCLy/2SOKG1eWIdF5Qvhnyq7/4BT86P6LIYkaTguPxmovNiKIKqa5SmZzPKmxrsFWXvpVYvFMEUYv+yQj3vjCQ3zt9yY/OjiVA5/MRzX7L6YMrXrzYJbwMaQ4bVGGY3l+yyRKZwB9UaO1pVxGz4xVFFXK6u+CLz48hkPqMeNCwP9lwsRzRNgE2D1CRVKCMdnM2i5TgPMKeVMRpQ1MLW6uUp7GGqgzZmWZcLUbYVYRKLMdHt1yvhrgvR/iHBe5NwuidJVke41oN6xAfW+Z/HMK/c0P5B7sU9wWsDHdzmsagfjKkeacUz6ZVDJ8GVL+Q025CwrF4R9EQLA3BOxuq8wFqVuNqA6Vmdm9F/oM5wbdW5OdDglmNbTRh0lLnISZyJD9MGf31Cy4/3cMNHPF5QPOkgIt42/04ee+WxbMZZq+iLQKC6xCbiqcxyBX1XsvRPzS8+Z2WaFRjvx7K7x9W+ELi6dWgZfRn8uWzfr9FtYro2tC+l8PrhPjxmuJ11zVaCbNZHlt2/0hjY1j+tZLgaUJ90BJdBtQ7Ek0+eB5Q7gtIsIknPdds3m0IlgHtyDJ6FlAcOoZfa6pdT3XcEp8FKKu2srp6KlLQeubY/yO4+E2PqhXqqMSfJ9sky6CQx5pSAFI1l8oMlIR2lPsWHzsGz0KasTyufFATXoYiXTVIj6cSUOdCARDBRtHMReY6/RxWT6B+UDP+s5jVd2sGX0RbFqoZC8ttSukujG819sMNPBuy/8eO1/+WwywNLvaMnhqGZ47Vf7iGP5pS7Yh8s56I9LI4tgQbmUijPekbGTuzCroETN/1UYq011TC3hQHjvFTTXbqaYeO4QvZns1DR3KpMQWsvtMQ3AYi684Uw9eS3OgCaEeO0TPN6kNL+tKIZy8WEOmNF4DZSUVNKR2KPWMcrgUYbe6L3DO5FHBU7QjY3/3jjjUZyRiX++Kh7EF8dCuT4uGZ+ATzY4/vg2O6oBzdKNI3ivUTS5BpBq/v/Ib1TBYgTCnnw+aRJX1jRELfdDLc8M4D6kORKadvFMsPLdPPDNm9zst45ckPRGadXqpt4E56IcC62BNGNbkSls0HIgE2tQDDfmyTG8/ttzw7H3cs3kL+Hm7ouhQhuy+dloM3UkeS3HSAuJNwDs4960cSNpNcy/PTcwGp8Q2UB15Y3KGMXZsIQ6ctBFknkfWyyDB5KnLO6bOW5TsBQeYpDsUH2Ut60ZI2WhwKyO493dHKU+50Us5cpKmbUznWk6+kx7JNFOVcE5QCUFaPBMg3I0V8K4B4/Ey2odqBvR9Y1qem6z7tgk2qO3nq+GVDMzIdqyd+U9XC5EVLvmfI7ilmnzvyfU286Oo6vGxHsS/nQXop7OzoTCaMm2NJGh2euY65g/lnFbfvxdJL2fVfJreO1QODtrLvpvKUcxnXeCkewtUjAcnptSU7NrSppKH2vsnxq5bs0DB+KX2G1VQqgKK1ZXUqi2T1RMBlkAswN7WnnEtnY7zwDC4asuOQetR5MkeKwYWA2PxA0kUlkEbqZXofpXKdZLUVRi/MHc3gzleYXkm6abxy2yoQkaELQ5kdGuKV9GymNw5de8q5+LSDystr176TfEp4TFAK8xlu7LZiJCgst+/HTJ63VDNDctWwuRcRFp70ot6+X7EXkF401NNAxmqsSBZO6mA8RIuaajdCN55wJYCsHXQ+xqMYZSG9qCj3I6JlSzUPSa5rqllIm2hGLwrKg5hoIcfC5MIUKg821rhAMTgraIehbIOG5LoRyXEif49vGrJ7MYM3tSxYGkW0qKhnwk66yFDNA6K17H96XlHPQtJXGfnpiPimphmHRIsK1ViK4yHRsumAeseGX+ZUeynRbUUziYSFjQ3RZUZ1OEI5T5A16LymnSbiibTiD1XOUxzEpOelpK82TtJaI0OwqaF1qKqmOp1LZUvWYCeR5AkEivCmwEeByHPLFhcagqs17cEEnTcoK35FWofqA2kQGasPDXqVC6iLwjv/JIg3MitFktpa3CgVWa1S2/oSvS63Ula3O0HfrgVMrjL8MEUVFT6JtvJbQLyUgTCmerHZymNVa0X6Gglw7d/3bU+ovIAEBfm2FfDYtneS2E6GqsJQeiqVAtWJH50VINkxsD1DqYIAX9f/vL/ybV+ltd+sNXn77+6bz/PNn/Oh/N+4/SwAy/TdE//u3/lPf+qv+/G//1/+pe8bQPCXvQH/b26JaTjZWaKV57cOnpLZmJ1QwmY2dcxeuiE4OuPzmz0Ohhuy/Yij4Yq8jfhwfsHLbMZ0VGC0Y3R6TdkEfPi9cz4av2FZpVyGQ47Ga5xXHIw2mInjTTKmqCJO5kuskw/PKKpYlLJ0PUsKvjSe2aBkEDbcVgOSuBHAW4TcO1hIeulOwez+DVUbkFchWjsOd1acOcV0klM1AbP7OdYpxoOKTR0xOtwQaIf1CqM8obGMBxVKlQwPal5dzTg5vuEmT9nZW3NTzdjb2chnM60w2hFox8VtwnAg/59PcupBRVFELPOUg+mGN480eMWD756hlSeNGtnXHSjqkNvvacIsoXmnZjgryG5SRknFTTWkeVgxGNSoYYVSnk0zwRcB44MNs7TkzE+ZjHOWk4F4z3cqjnZWXCxGmLFImr/Ym2C8IpjVzCY5dWvwQH2dsH+04PK7MNMOdVhhlKcetASBxR5UWKu4f3TL6+speq/CBJZ43pApGM4KqjLEK4+2mstfMYx2cvIsxu026NMa6oDhYUa+SgiilvX7LYROwKfV1C4mCiz1vtRNMG1IhxXFmxFlIsE4V7/pwCrCwNI+LtFAHXhM2uKtorivULWiOZULdK4j9KCl7Xyu9dhjR5biNwuaPCQa1VSNEo+sVcIG7zbYVQCzhvO/FkBsYezBKdRBhV9E+J2K5jrGlAp7WLMOQ9xIGGoCR5FKgivKUxwZfOhoQo+KHDb2uHmDDh3l2GAWAc1YGE0CTxNqmDQ0PmT1jqE6aAnjlvw4QoWOZuop96ULlNBRBR5Cjyo05f0a4zRu7Lj5lsEMK1xu8EPL5jFkDxQ+izFzjx1bKqPxgae4Bz52+EJL0E8uAFwPW2zocXlEfdiZ+rXHhh7bKlxgsDstWRvSjl0XXmQoDsSPWxgB4Sp0tCOp8Gimjk0ggLM6bCG2FHkkYzTWuEhAkh06vBIGaVsNsnPH8OtcUi/radep2gqAVFYWFVTbSx9FttsOJQW0SR3WKuzAUSae+NJQ7ggL6qKOadYK8LjES+LoQdfdGnmKQ7Y+2Xbgt4sE0kUpjFs78He+3Eh8tiiNriV4J1rIAku503mLh54s7HzA2ndy4k42OZYe1DZFUn4ndLUNauvHbYZdZ+bY046AwFPsiS+2msmYhRtNtefQrUjwmqknR4Jb1g8l9Cq+FmlwftR5um3XVziSxRYXe+q5jG89VVuWDUTa6gPf+fCEkYcuMMbD8nFA06Vs2vgOjFdzZMIcSCepDZVIo7teQUngZMvUeuPxXpEfSoiM19BMxJNVT+U1dK0ksdRLGJEAX0Qiu6+pO7lvPe37HOnSI2H5OEQ5kbX6zr+pQtgciYxSjpuwif0Y6LrvVJTHV1NNM4FVJFJLYWGF+dRWtvn6o4R2IMeybYTJaobCFvtalAHNQEk9SGd76vfXhZAdmW39hmr7GhoBdC5U5PvSi1hPBKy3SUAzUfjOYys+Ukc91LRpLz/upLcu3NZ5qLZb9JoIeG/TblyNlx72vv4BOY5e6a2f04ZS/SKeQKjHhnaoiDJFsdcV3Pe+2lrqQVzXEeo1bE5EzaItNFrhlYBkG6nt+7VdV6TXRt4zMoS5nBvl3HThRWbL/tfTQNjxLlG22A+7bko5tm2q0FbYUBfGtKnIsG0Sd+eDotyP5fEe2qEwie3AYENFNQ+xkYxdtRvTDDRehYSZpd6JxJuqBVjq1lPtxNhY08Zquz+uS4RVDqrdEK+gGct4iXRYrpPt0Gz7U5uREe9npGkTTbWXdj2+gXyO0gA77ZjiSG+DdpTzNNNEejPjAB9o2mEgCbUHQ1wk8mBvFCYSMIrzXUaCAecJcunidJHuZMPgQ007jTFZswV7PtCSvKpF/mwaRztJ5PwxChdJAqybyJzPpQG6keRX7ZzIWt9iIV0SbIGWSwJ0XuOSEF0L6GSYdH2/VuSvUedh7Dss0y40x2hoHX4o7+tnolrqPZQetp2SHgS0lgI2lXUQhXitUVrGVdmOAQzeAnNvA7/e89j/3fSpxx3jqLWAy7ZFGS0yVUAl8Zbd9NYJo9kFDf1zYk1DlxDrOyb0X0B0vZ0y+1e3n8pNKWWAPwJeee9/76f++j/PjOXuR3v+P/4f/nWeZrv88OyEXzh5xXfGr1nbhH/05gl//ehLAK6qEX/05pT/4PEP+J+ffZedYc7lZoi1mid71/z41RG/ePqSi3zMi9c7/K1f+Sf83c9+FaU8f+PhZ/xvz9+nzCNcJsmiR8e33K4HvLN/zavllLIK8R6Rr1YBv/zgBX/26h7jQcXp5JbvP7+PNp75NGOVJVTLhL3jJctNQlsFTGc5y+WAIGppq4DH96746rMjfu17X/KnX5/SFgEmbXGXCX7UMtmVGoR3d6740x8/BuD08SUvvtrHTDsjtlPszjeM44qvPjuCwDN4FpK/WxMNa5rzFN8FyeidGtv5MVWjYVYzm2csns8kQMdJb5vJJJb+l77zlD/95BHRrGI+zrl4ukuw0jRTS7Ax2MOK0bTA/sGc/L2Kh/euefUnx9ixVBWgQK8N6qjk/t6C5y/2UIHjwfENN3//RLrfFARL+eJrJ5bkPKAZOdx+zXSWs/5sjp20/Mq3nvJ0scPNxQS9kgmSD2SJ2Q8s+ycLrr7cgWmDzwP2ThdcX4/wrYZaE6wNw5eK/F+RKo/6JxNMobYSQTdqwUmokWpl0uoV+HFLeBESLiUkRu1UBF8n1Dsikdx5/4br53M5FqnF3AYS5hPLpCu6n1FepYS3Bv+owFklKa8ji5k2+MsYN22IXkcEG0W573AjS3ATYAeOINcyqfbCLoUrjbJ322cTmfwHa83wlaI4gHYoYx/dCrMa7BaYT4Rx/fbvfMYff/oYGkVyEVBPJdW0OHKkbzTNVFb/7TsF/qILcPIQLjT1QUuwCEiuFdmplfGqFO1Og14F25RcGwt46nsSdSOTnmipqGZeUmfnIj9NbiTYph0Ke2eqzjPSxd5vQ1s6wOJSObfMKpBeuYWi2u3DWoQJW3+7ZvbH8iVdHPR+WC3VDyORjArA8fh3ctSXAykrX/dhOeI/VC00xzXhubyWriQgKsgkddVFwu5VOx47EbmzCyUkSBJVHZMvNIvvNUQXMrEOMkW157Ajy+B5uC0s1w1bKXAz6hJioy6Rdcy2KgAlvs74UpiTcC19fvENHcgTJs2UClPI46tdCamJr6WfL8gRYGyE4ctO5Pgo30s87wKM6h1LdGMwlQQotSnEtwLAoiXdOIis1NTQDAQYmYqt3C5eOrIjTbT2bO53Mu6x7G9yJcfaawk1MqW89vBVl3S6EXmqjUTSGK7ls3DzXcfsE/H0Dc9EIhxmntVjRXoBwzeW7NCwfscRZJrdjy2b+4ZmJPUSzbh7by1BQ6YS72UvSW4HnZT4iSa+FaAeroVRTS9F5qocDF9LeMnifRi9lLAgU7ENuJl9btn8n+y9Sa9tWYIe9K1ud6e93etfNJmRWa1LBdgjZOEpJURn7BEzhG0YlpjSDRBQYH6ABQMLhgyZ8QsQsjxwVlVmVGZERvua259md6vz4Ftr7/siKyttyJQzSrWlp/vuuefs7uyzz/rW1z1VMG3yrDqep913ybpmhjB3AB5fCFSXPM+6m99z3VLKag6Uuw7bdz8n5sB+xnEjoLo4gRIRE2ubpL12QUZRD3FKTq2vPfYveH0W95QlhyKFD93HSUraXUjUV5Qx58UcYmLLuV3dk22NklLbUMz+UwD0aibfZ7Ejo1/dBfjkbexPBeor/l7feAxrNUtk+4jyniyrdEB3xv5FBu7QY8l7CRnN4xP2cubuQoBMcXXr0Z8qSkEHSpIPzzVUD9TXDodnGs0lv5v2zzWWX3v0JzLJ9PlemSQdbS802sdMop3CaSzloVmqOi4JutXAICGz97Arpq7WlyOEDeielNBH3rf7E4XqloFAOTTLNUxitUsF3dIbOawVynuPca0mBtEm4BYlj1kNIVUaCYKMyHVlBYIaI1THapDy3kMN9EQCQHk9YtwWUGOAa5gsG5XA4tUAVxMMiQj0JxrFwaO4GXF8UaG+sgSJPqbQIHZQSsuAnyKxrYiUOzPpV0K4CLMf4RaG8tyjhXABw1mVrCuRx6MFZO/hFhq6TYE8LiCUBEbF5RHDkyX0wc4eRC2grzvEBCRj8myq4whfG+ibI2KpYU8biMh6EUgB2Y5zr2RtIHr2STIki2xoLDTEoQOMRliU0zblcUAoDUKtoY7j1EOJALKdIOMY65LJrUbTG5nYxVgZiHZgMI8UiFUJ2faTZBYiH4d8pwokh/dASohhZICP0WQs8/LNUJ2c5loWiPsD11MkoJvXrzV7LKUCYkBsO4jsuVQMI4qpRiR6D5FZySxtzb/ntNqftzwI+fm2h/fUHz2L3/mHf++Xvt4/+ff/23+hYxNC/CGAvw5g/VfA8hvL4ntP43t/9PexqAimCuXRjga7fYPFomeZeMcOte3THe6/3AALhzhKiMqjrC36a3bP+fsCkBFqZVFVFsf7igXylWfoThD05pkA3Bv6GEkYAABTWxUH8/BiSllF6QnYdAQOLKSXpUc4sKYi9gr6XsGt/JTampNMVSsRLkbETkPUDtFJiFYhVtyP2CtARFZFpC8EbCxrQHREcaMYXrJNN63KI3oJ/daw3sJm+QJngWPFYzKXBAHxxEKoMFWRyFEiLB0lvE9GiL1OVQgET/KogPMBcVcg6gB1UBCex+KbABEEZM/OuOItB9u+oowvl1DHMkxJtIhAqFlWL08HiC9qls4fJIN13hjYCwt1ayaAFcrIIKCBzItfsaJCHSRCmaR+KYlu8o6qCHOnCCRXjgnAgeezvFYIJsKeBMiO/W7jllJQt5zTH1XPege74fGYncRw4WHuONAMBYGdcALVJVksBi7QP2oOZB9cEyefpfCp9mARYPb80s/ddrnKQwTA3AvYNUFUMATWZidZsfGavqhxTQmqcGSM8msRAUSRJJxiAqsiJZVm0MOeOEowzT1BSzCp5uIOZP9KYPG1QH8eoQ+ZCZt9nP2LkRUjO8n3O5LR6h4TlOU0WEp9CTBz/1kONIqCYMU33LbwDI9RY6qwSMmWPgFs3YkphVKOSOmvlEj6iiXuJvUbuia+U0pfvxEp7CRO58HXvIYyCLNLDvBkYkyCmcGg7oFxzWMrdgR4uk3SvLSfwzaiusmJrymN0yaGLDGgmQkqbzkwt6vkrxwwXQfSUraZE2T1kecsFGTCcmUHB5lMtgXmzwGASYobJV9bvyErqDskRofbdHVOCU37kOpGdJfYrDL/Pl9D2W8JYAJLLqW/AlzPsOHvvgbKG1ZYIHI9viLw89WcUmuOBI5qIJDpHjGUpzsXU3qt6ngO8qIGglERkdbN7TSvCcpUP9enZBaNic4MS8lT7nkATrCeAEwC0tJR4grB92xKJX6Q9puBY/YS2oVAeU+mzVes6uB+xokVFp7BObbhfYEySTmdU3Mge8trnB5Jm8rtfSEmoMawI4Hi8LMJrVHNxxQlaznGJbsipSNApBeRoS3VHX2H+ZyoMSWwrsnC1jcsvXdNqmoxvJayZ9W0PE5bc/9cJabJALtMfZNDTCnmIrF+DNbRfUS/JbCMiuAwB27xnMSUniqm3zPjlwEtjysxz4WgP3Ngqqs5BggXEUqCr6AF1BgYwmNTiNc+1XA0vOZyIEuxDw+SbgOGjZpCkoaNRHFIHaw+wpeSgUzHwOuhJ6jKYwtXyyS59qwj8YAaAhCAULL6IyTApjsPlxNXFY+Vk2MevpJTSqvuPLsh03XlaonylvLWDOikDVNarLRhSpaVNkJ1gfcJzcAg9kUqmIObzre0BHu+VNCdo5dxCFCWiaxZkpr9l8GQmcznRLeO3ZCpg9LsRvhKQ44zWOL3gUcoyUSq1kGODqHUPIeB28lyZoZi8VypztITGSLXGSNBok+Tx0pCOnZJisHPfZAP1i9ckr3GOAXtCO/TOshChqqAHFMvpWFoT9RkE6Ek02ABriNLWTPIyuArxIkFFA8qQWJVEBhKSQ9lToBN8taoJJnJPM5/mDybfY+5p/KbbGGu9Mhg7xsdkkIIMpLDAJQlYEdKYh8+L8tbM6h8kDo7Ldm7KeTESsYQIaRA/PMYTAAI/98DfH5dgOWH//MvH1j+6X/wi4GlEOIFgH8M4L8D8Id/BSy/sax/43H82//7H6DzBndjjdu+xtvrNcrK4veffoV/8uVLFIXD2aJFiAL7vsToFLpDieWmQ1NY3Nwv8OSUEY9fX2/w7OweX74+QdlYbBYd3rzeQsjICo8gsF51CFGgbUu4VsMs7JTIJJWH6w10ZVmeHgRg5SQ3bJoBw2Bge+awqyLdKERMzweK1YjxpoI56WFvKqAK0JWDazWWZy2G3qAoHYHvzgAbC106+Fc1zIsj7KARWq7fbHu4qxrqdIB3cpJsYmUhdYA/GIJZKyFqB1062F0JUTFeXaiIMCqo0sPv2cdY1SOkjOjakvUob2vUzw7TepfrDvvLJYHo0uLR+Q5X/+wR3DnZK3ExwA8KzabD8NMVwsUI3BZTpyMCsHi5x+HtgkC40xCjgH7UwVuFeE85IqLgfh40ok79iGcDTOkwvmkASTma3im40xlEiyAQBWWZ779/ic9+/AgoA4RmeBHKAAwS9UWL7rKBWlv4XgGDpJf0osXw8Zp9lSvWo6iDgl95VCc9+vsSZjnCDRripiArPArgYuBkghWIZQCcgIgCsXGAk/T2FemzeD4gBoGTkwNuPz1hn2ETABWh9vQiyk7Cbx3MJTsVAQ6GM7iETl88jUO1HDF+tcDpRze4/dMzRDUD4lxujZCAYB0gRoGwdtBXhvKuZUSsPMx6hH9dJ0kbgZ3beuj1iPCmguoF7Dn9oMNTVtYIJ1Dc0BNpUxIuAJSXCuNpQDwdob9mB2iUQKjIjJq9RP/UTZMuIrDvMyqyGeZAcB0MgZTwAsN7A6qflrArzvALx4FmZlTd2kPvUoXMivtSXip6TkEvadAE3cWdRPfCQVgBfZRkFM/mCYXMapeXEtFgYtPlVUFG+k4SUI8MRpJWvAMEo0xF9Aeu1zcB1SsmDLrkrwwmJgDEbeXrIwdShZKglb2iMwAFCIYzkJSWrKqvmKZb3HJiQ/VkaqUHEAjCfRFR7Ak+cvBP+zQm/yoQVIQaZ5YyV2MEEyfQbQ58PUFoujY9wXhQCaSK1M9Y8xpUPa9Hu2DNS/uMEywAr79cUSPHGVT7Eij2wLAF7Ipe2VCyZoRMIpIEM4HlBxUnCHxtdxHRvCGTHAquW3fcn2BmsM3UUkwVIexUnIH29HufgQ+Py5ecyJEjE3gBTAE3wXDdDPEB2icRzSuCgeGUx5EBeWaDdZ+lkjw/xX1Mg2WCcF9h6vx0S8DsyEpm9ZpwDNbxJUOR6isGVkWVko4jWduYrxsPFHcRuaYkFFy/6gjY3IL3HdXRf2lX3A+zjxjO6B8NiutyCwJmnlcypMMpAWixS8m5iucyT26IgHf8mVUGy54e5Q/SZC4AACAASURBVOZNIJjWBERkKHmMuXQ+aDFVl0jL6902YpoEynLOXC/jaoL0/kySPdVzRYxb0OvL91bANsnf6rnemGS95hinShbd8adKEyJBc3+jBMp7+jtNF9BvJeprJuTaRk77VN2GqbvSVRLFgZ2W+XW2liiOnDTI4Fh4Plf37Dt1pYDp6NHWfcC4IrhVXUB3zsAjX0h4Q9AOQbl3cSC49rVMEzG8V/qKYFsfPcFmBv+5tmWM9EaGmMKPmOwpfWR6a8NgH9eoJFGV03akpWx13OipHsVXKnmY0+N9AISAPjqMG8NQoRjhlmaSPRf3I0LJ1FsRI4RN1RsxphRZgmK7SmAuAmY38jpqDPQhsYk+IAoBmfokc/2JrzX03cB74AMQ6GvDGhTHfYQLU60JQqAMtrcQg0M0D9xoSjBoSAgC/t5i6qpUElFKMpP5NVIQKGZwmlNaQ5j3J78emAN7gJkd7Jk0O4FbIfi6h7Uf30xhzTLfFN4TXQKp73Rhhlkam3yW76zzAe6IGTCnOpJfuIT4l8Jj+SsClp8BuHrw0D+KMf6jh88RQvyfAP57ACsA/8VfActvLNV3n8e/+b/+Xdx0Da6+2JLhM2SsEAGYCHOtWWOwtMDOsDB+QWASU8AHwIGRaiW/HCTZDwjALVlvYHYM0sishW7F1EHFLybKReUophAUiFSG7NIAzrKr0FURAmR+MhNAudHssRGBkjOWfTN8pLzmzGWWwfDLOrFwkgNNaTmArK4EhhP+HhPIEk5MheOZpWCoAwd8romQPg0u7zmI0C2Zlsy4TLPbRUR1yYH77iPKJgEOeKobJm0GRZljZgcelm67BY8XmAdW3aM0oC7mQVWWn2XAZA6YqhRyXx8HaBx45IFhdzEnc2YpXx6wQHI/fBqESjefj+GUbI5dcN0AX5uPwTV8fu6MywOekMIh8rqkxTv9ZnZB9gKgD8tXTPN0DWVaKoEC4Xk+y1uJ7Z+FSSKmRmBcAcuvWK5eXbNgHILr6S4Su5ISHZHe53HD9RZ7vsfmOJeGZ89QUGn/0+y12XPAZw4zq3l8QTChhjTgHGdWJhgem+6AYcOB1LjmtZF7FEWa/HQLMXUZZrlnHlBSDsjkx3GZgG4anGQWJ3ubAKQycyTWigmjzSue09wlGDQHu+aQHk+9gK6i/LN5/S5rA3DQW+wZ9hKVmFgImY4hA/K8z/k9z2EjU8/ekcdR3eU+wFQDUs8sKuV687Uj8uStn7djOt4Hhi0H1Gp4dx9y4mZxCOhP5SQ3DYqJmlGk0Jo0iJaevq2oyNwETbbELvghqG5zeXr6rKcOPTXOg3Q+DvrSHAesxZHVElmqycASShnzsapUQJ+Zpyi5j65K/rU0ALb1PBDPhfI5XIWvI9uYz2UUYhpk6yHCG0wD6dxf6UuB8s6jPyHrMzGBIck0BWAOrGHI3YXcd243D8rJes3BRazgCKmvMLN2ZHuk5znjdUymSXcBdjn3+GVp8MNj0h0BIWJmh8ni1JcjDs8LXhctAU7USMeZWMM+dR2OYUqpDYWYEnwpfUzf+3G+DqVlqml3KrF8zcol1nzw+WRgOPjP0k/hYkqoFFA92bAokCZI6NXMQGE4MQy18Yn5TYy8SBJZEbK0P39niWn9uicTlz+LkPybtAF2pScmTo0ESNIGhJIgJTNQIgWx5EXaMB33cFIwFfRAZg0B0H3qd1S8xoMWE1ihyoe1GNKSKQyGjBmUQFD0kOrep/sYP1uqp58slGTrxrWB2duJZZ2WNEgX1vNnYviCkZCDh4iR61RikmdGISB7B78wDJgp1OwrBaZeRr8wEDZAjvQc8n5DBs/XhvfkECn3LDRy3yMA+EpDtRbCB4RC8zhGD3UYpu5Eed8iVmZi5+ACYqX5mUnrzgBKWM+/xfw+EVQBIEuoZ6+iHB1CpacOSihBz2DyNQrvGcizrCCGB0mnAWQbfQS0hOhGQAqERUkJqhD0IlZmAjxicDPAipHgy/mZQcxsn1bcVghkG5MnUvR8TAyW743R0/+hVAK3au6hTI+9k5IqEnB0bgZ/DxlOIQBr5/7HskBse14vRcHX5c5IIeb1OP8uq+jcu2xlpNd+YjIzaMxLAucTG5n3M70uDsN0HKIoCCrDDDR/ZvmLwnv+vOUvCbD84H/6+7/09f7wP/xv/sJjE0L8OwD+IMb4nwsh/hZ+RcDyWx3esywG/NtPfoCP2yf4pDmiUg43fYNaW9Ta4tPbU2ze6yFFxOAV1OOI22ONQgDOSZSFg1YeIUgsyhE+CtwdGvSHAmbdY1UPuN03aIyHUR4uPS9EgXYoMPQGdTMgBImuLXC6PeLmboHt5ggpgLtdA3cwWF4c4XuDJ6c7XO6WWFcDjl0JqAA3aigd0O9KiN/tIURE3xVYLHv4UcONGh2AMCjIDzq0hxJVM0KIiONVA1F7CBn599IzTfZoMF4Ai/MWx+sG1baHGzWZNxmhSo9gCVKlDkyzNR5ae3T7CtVygEz/74NAHBSG5xGy9Cgqh6YacHe3gPvNEe3lAqfP73BTbSFXFqZwOJxUgAD00kI3A56ud/jRT58CYIDO4apBddJjd5c6lrxA3zjEVsP1EvF0xLA3lC0HAdEqYG2x2bS4/3KDWHuITkFsRoQDfa9wApBxlipXAePzCIySUuGNRewVROMQDxowESePd7j/yQnCmtUYQlIvYzepssUL9oo6CQw8d7J2lPqqNHkh2O0YLka+D71Gte3hnYS7qiC2I+KuABYO/a1BNBHyZESwEq2VgBUQS8fEXRHJYlYe3VKgewnuuwBZThnRfkdAFB7tZQG/JQu8+x0P0ZPNkz1TM/3KT/sHETHsFeTzDru7EnLgF0ZUYWYBL0bgqBGLAH2r4bYW8kAPnd2wqzPelIgFJbrCkmnwKw9RO/RvSzIBpxbqXsOfjBw0dgqtFay2qT3fSy9gdoA9DdCnPcQnDSc17BxwYw7srpQjPYXRRMhBQrUi1bQoMoVbj+Ka1QPjY4vhVBF4JaBGWWAgu3piWe8iAPt4hGgV3ELODOI0mRQYeFRGhMbBXGvojsyi2XHiKNQBwgpKnSMDfsQgp4kcfcygi7454YFcKJ5fK4dU/7Mmw0qPJj3NciDLJxwH2FFFhgh5kRJ203M9oHpWstgNJbCIKSRF018KIPVOJun0Xk4st91wskyOEqojG2uXEvVrieE0YvE1GUv6TBkEJBwDhlxNXyxCeo+UnFJwXQNIqzCcBhQ7sq88vzMwLHZAfxYZ1BMI0u2aCbvDGRlH9m7Sm6s6yUmfVWIw00STL5lUXL+h5FK3YvJ8+ioxRCXoE+8NclCNbikdrl+LSabrFhLFvcD+PZ2uQUyTYgAQJWXroZiZSk5Kzd4g6QE5cJLSLnhMuSLG1TIlvaZ9OJCJrC+T7LjnY+1zTtwFPa/TVcD9d6oUSBIhgkZ5nfeL+2jXmHyKw0bPLCs4CSEtJ3pcTbZRt5QgS8dzmeWiN79ppkmOOYyHMmyAUmUROUETDLfnS04KBgXUV5z4CkVigDsNX1NqHAwn17iPSYbfRfhaTBNbeQJQuNS12Sfp7DEmTx8fr66Tzyz1dA4ptZRMY6o/SeA5Hw8EPb8icBIgV8xECcSnGq6mR3P3vp5AeFScnNp9YGAOnBhTA6XIxS5CWZWklhq2yXJWIMq0Dh/TBJSZ2OrMxLaPyBgOK4nqjveCYc3QHHo/wyThlS5NWmgB21AG7IsyVcVg9my6EsEQECMC45LsZp5EGldySq/tTxX7OsuKEyAZvKNONTqc1ECMKPZkD8cVr28ACMrAHAtKwBcK4lkKl1FkiKdJw57gW/czG+krMpfDqYE5eLiGzGiepBCBoJ1M+gL66OEaBXN0TGgdPOxSQx/9NDESjIS0JdQYpgkHCKQKkwIiVFMFCVDw2ABACJi95cSIEhBjmCdAEiCeliSxVb0DmmIC0aFQkL1FaIppIizLaBkO5SmDfQCsoSXgAkHvYGdADlCyqnKnkSQoVHL2R/rA75Wc8Jr6LaOixFdYNwF+0Y98vCwgrKOqP8l432Ub43Q+IAUwWgK+LI3V6aaUk2JDQIwjIDQgJEQO2cmBPBmsG/MucP4mqSXELLv9Zg3JgyXKn/unv1p+8fJvAvh3hRB/AKACsBZC/B8xxv/4l7mRbzVjWb7/Ir73D/4Q4xOLk4s9XJDo/2QLfNjCWQUEgfKTkob25yOqzwuI39shRoHhVcMqir3B8tER7p9u0b8/oloP6C9rqI2F/LzijO9vHOCsZprpfYPixzX65xayVcAZZYvYGcoxn/Uoflijf8KCdpgAWXmETqN4q2HXAfq8h39dA+cDgpOof1Kie98CVgBlgNxrhJVD9UWB/ombwJDaWPhOoXhjMF44iF6ienbE0BvozyqMj0mTFW817MsBzZ9UcDVYoTFKqKVF9ALBUt4KAEJEuL2BcBJRM2HTfFIxNGXroGomro6dAXaUcjZfSxxfMtRlOKd0Z/HeDscvVwyQWfgpnKf8QcMo+H/tANtrRM+7QvlFgeEZw0tcHRGqgPJKofn9Gxz++BT2JH1rFYEAr3LsyHw+YPHHFbrfbyG+rBFlki0+5wy77CXiKb2h5tOKaYZrj/KtIkv4loPe4SzA7AX8hz2qeoT94zVEEBieWsjGQX1ZwW48qte8ifZPHEQQKN+oWaL5eCTgjED9lU7F1GSjV58CN/+6w/LPDIazyDRSFSFGieqtTEEwEaGOqL+kD9I1EfbUY/1DzZTJgh49f2Kh7jTKaw72hedgSQ0C/VOL5U8M2mcEK+FihLw2UH1iQkdgOKeUdP2xQncRMT63kHca1aVE+x0LOIH1x3pOcvwe60LaDyzEIFHeKOgDpXVypLRPH1g5YfYC608jbn+bIG/9Q43ji4jlF8lrZ1Ny5CbCXtAXS8kUgeTqp8Dd3+yhP0s9mUd6+na/RYBK7xhZ9OyzlCPQX1BeWt4wpEdaYPk5sP8w+StNhN5JqJHeV7eMWHwhcP9bHrEIWP+JQdSpG/Io6AF96iA7ieKOHZ3Fjkxpf8YKk+qNhFtGFLesD+ke8VoaNwGLL9N74yh57B5T1uhqgqPlVxGHlylk5RjRn/F57bMZQIhI5ndcc5u6E+jPKfHUHVBdkQntz3mtmD0H7nbNY1h/Atx9H2hei8mLOJySuVfJJwkA+w8CijuJ6pqD7fYZk1b7C76vxQ7Yfd9j9WOF4Syi+Rrozzh4r64TgzYQCABkxatLMsjjhkqJ5jUHlb7ic8w+pbgued1XbyS6px7llYK0ZN33HwCbPwMO7zPA5vA+oPc8Z9KT2Tq+4HO6C4Iw1wC+YJ/l4X2g+RrYfwAsv6CctvlKoH8EbH9ENrc/J5hafhVS8itwfCaw+inloeUupkAWqgDqt6mTM/B9U2Pq4HzDHspcKcPPKsNaXCNweCHgFgGP/584MexjYo31EPH2rwOn/0ykahSy7dU1Wf/9S4HNpwH9iZxChHSf2M3EQBdHMqK799XkH5YuTsoHX5KN7s4kqlsyuGrk+QqK7HBmR9XA9Y5LAVeTrQ+GAHHxloAnV5SoPmL1lcPNb5oJbOuO50539BaqMeL+A436KqC+djg+MejPKFvtTyl5bK643vrWI2hWsSy/dohCTB2ex6cM2alvmQobJT8j/SlBqe4IHgloxKQSqK8DDk8VdE/1QGbCi0PA8RFBVLFjUI5t5HQ9u0qg3AVOCNQCzaWDcBHjVpNRT0z98bHC9hOL3Xsap3/aY9wY9lYe6TdVNmLxyiIqAddIyMQ2ByNg9p71IdduAiyukVQc7An8MstNtZLH8alBc+mgW4/2MTsj1UgWOPsZQ5Kg6lSP4guG47RPCCpWn/VwC81AnvOCrHk6p9W1g10plHcWYiTgCimJ1ReUi/pGU6paSXZ4XttJEmp2BCy7Dyp2btYKQQmU1z3cqoBdKdRfd+gf1xNgqt7y9/rLA44frlC/7jGclijuKEUdTksU9wR5bkFwbvaWDO3gMW5LblcKSBcQtEQ0EsIG6P0At63IUGbGUQnYRqO66mHXBVnfB8y5Og6UD28qekOVmCS0dmlQXraIRhFMthZ+UZBtVZL+SZD1nbyPiUGFSB7VdiRo1Io1IKWBut0zsXUYEauSAK0pyZ4aPakEZJu6L9ueXZaHlnUj1s2vL8y7dSOjJbNXs/sSPvA1bT/LYKUErGWv5EMG8oE/MsbIFFnnmfYaI+Iw4mFCa+x6iLoCnEP0IXkvGdITx/FdsCglos3Sn9lPCeAv9lQ+XGL4WVD6L7H8OjCW1UfP4wd/9MtnLH/0t//rf+Fj+yvG8ucsdT3i6d94hfOaCRCvj2v89r/1I9z2DW66Bh9ur/HD7SPsbxb4/vuv8eniDC+3O4QocPbkFXpvsBsqlNrhk5M1Ti726EeD9bM9tPK4OTFYP9njpOlwfWxw7EoUlUP/1KE5a7F+2WPfVfBeAo2FtQon6xZXHypI41GUDufrI673C2zPdngtT6Brx/7IIBGsxOqkRbs1ePz8Fle3K0jlUZ626LoCdqnx5P1r7NoKxabF/lBjcdrhGAREESAah7ocMY4a5rd2iINGVY84lDV04dCfB4Rzi3rVw3uJRT0gRoHdoUZR8MPtvUSxJTgOUeB0c8TlU4b1lKsBIdWd3AwaxZMW/W2F8USgfHbE4BbwiwBRO1TG4bi2CEcNsxqwetLjfreYQmgK7dFsR9xdLllnUUVAB4xnyaN3X8B+p0eMHJBBU66MAOiFpQ/1kYUyAd1j/s03AVg62FBA1OzwxAAIHVA3I45PNaAiTDPC9g3C0qHVemIAY8vwo3ZXIV7QL6mXFkp7DGcOsnLwFcN75NIidBpRKZRnHQZRw9QWdlAwJz36sSGLUkZUbyWGE0AuHMYNpdix8mRTrZli/bOULhRMChUeQOlxfK4QmsQmmkjP5lFPHiCAQCloAEWAHIBQBEhIQLKqRHjN1MBFhLCSjGcEogEwSoQqoHsWKelquJ829SGKXsFu4uQdDYYMkj1zUDuVvGoCfu2gBs3BquZ67FIj1AF2oSbQI62A6gSsZ+ei6iSvm3uC8uhT52FitXwJvm+HFGKUPHxyJEvnGtDfuldkzk4txKDQ9QpyjHArph27ZUBsJWTy9PkKZEESW29XSQqZmEHhyAr4miDWHMXU7xhqD+klfYqlwLDl3+yKLLcckyy14zr9IqA/ywM3hnvk95sBHRlwe5Q3GtIxyAchsY1lhEtss24z84okhWdIlXdI9RFkUMaVgNs6DKOC2fE8VlcEt+OTCNUKmBZQHcGEXVAiXV1l2XJMIT+stFFjxHgWYPYKbkk7QPa+jRuyS9m7l32AiNz3kJIipaWEW/W8ZoVjKjFluQLVFScsCN4ixm0KaBGUOrplRHlPabMvuP7hREyfhXFD9nIO1SGgna7JSFvCuBaJFeY1YBs+VuyzN4y+MXnDwaZIsmTdUbYPZOYrTimjDLR6IBkHWTRXk6nWrcCY2DZfpMmDJEctr9nHmGXpQfPYuvNZCkipMd+PPvn+TAd0p9SaymQVyKFRwtN+AWDah+z7UylQKgqe79hxv+xKwLRMMgXE5AEUHjDJL5q9p+gzK8gaGl+SmQwqbW+MyXdID7AvBCXVaYLJ1jxvMrwrg5dpkJiZI15XyZNsyLZJF+GMQJCzJN20Ae2FSnYCWi6k5fOjJpuaZc5BC/RbTuD1p1QZZFm37oH+JMs0OUHiN3KqxkAEih2BWHXr0J0TJEoLjFsz+QpdmdjXIWLcaB5rukZsncN/EnBMILM/NxOrqPoAe6ohegJKX/C4RQAQwLCdOFeNZFCpBiafZq8o+y7TvW4KCFLJfsD3To4BopFwhSDTVxkUQiCWTKV1tUJ5PWBca0RpoDuCWPZW8jKL4L3A10xjl46TKPpgMW7InAXN47TbckrtlWOEXxiCZ0NwXymZvp8kYgKfwge4paF8Oc7HoVqXrh8B32jIuwE+BRbpIQXjuAhXsweUwT3cd1/piXkUMQX7jAxCypJR32jo+wFuyR5L6eZQHxGSXDftXygUmLKa6kqCQKg19B17L4WdPYNRSQLAdL2HdTN7ITUjsKOUwANQGQ2rTaJRBGqlhuiTjH7yWkoC6AwK8/9lJLuZ5apCkOXU6Xk+zOzjw/XkRSmm1Go9S+eFYLVInJ8nqpJeyxiZ+GrdDCZN9lOE+WeWviKFHSlMUlvxi5SwMQBQlNd+y5dvL6X3i5dvNWPZfO9Z/N7/8p/AeoXj2wVDUhR4kxCA7Bke4rbpwx0pURJpQIaLAfLrakoGlYOEPpC9EAHonjomqy4YVJI9LG4doA7pSzBye/TYxSSh4//1kf17UWFKJMshGtlToVqRqiH4f4ADx1xroKbkS6ZkQpKtyJ5GSqIogcuMg1vEVLfAQT0Et8tBKgdldhmg0/ZyDcbwxKF8TbYsqojyJvlPdZKNbeLkCY0qQniB5rXA7rctlh8bjNuYZn8xffnIkftj9qmrLaWejhuGhIxrnjPhKRnLNRHltZiCHHwe2CSJYTCYAFn2SRV3BAqhiGheAd1jMV0HuSj+YeBHDuvI6ZluQQbNLVjlMK7oNwVmVkK3wOH9gOVn7FQDyAIIz237IibWicmoxS0HH6HgNsyO4MZX3J/MQuQQCabLAvvvepQ3CtXbOdwkFExUNAfK+syerNf6p2EeJO/4WPayqiGdI598X30aFBdzOEkOQ8nnN1cUDFsGb+R0TFZTiCm1VA5zpYRdRNSX9HgeXgosv4w4vKAvNCY5W7Gn52/c8PznQZyrUxKpwhTKkpM5x/X8mQtm7hx0dfK/xiRzTCEou48CTn8gpsTV/L4PJ2SD3EJAHwkc+jNec/VbekWz9zDoWdo3nIj0vvN9HVciBaUgfW7I1uTQCtfwMXPg51m3ad2B7E/ucgM4WPEV98euOOBSQ5yCPXTLz1r2RY4bgu9iH6f711QpkcJlgNnDbI70KWa/py9475CWFSM5hAbgNlRin3K1iXQR/alEeRd43GE+p/nz8DAQJ0+WuJrJrDk11tXpPpY8x1HOSaDSkU3qtxLlnkxcfyKw+SmrHfK5JQvE+wArNSg/lJ7XlC8JTOsr7qvueBzVXUB3ImE6joJtI7B469GdKjJ6xzTAb8k0jEv6M20tUN/mqgsCFNPGKUF08nomBtEXIqWAZnRLRklEgg1fzgDHHOMkV/QlZYfZX+gLHqfwEcNawrQp7KRI7G+ZEmSbJI8XQHXPIBfKOgNcJd+59lWubdAp2VRhkkS6iuAyKAE9hOm6GtYckNOHmiSohwC7TP5JpOPzQEjeUGnppRzXavKSDhvWY/hCoLxz6E81wXACR8U+YNjISeaaQdDkQY2Yj7sN9JwCqYoi30Po7wxaQHesr+C9lpJJrnh+z+SYWBKdPImWQTehlJNUNsr0+cwSyZiuc8tj1keHHFp1eFmivPMpTCY9N/tDfZz9kYlZ5fdQgFukjkItIIcA3Xn4SkG6MPk3g2HvpNnThxdKnhM1EOSNWwPVeYI+gQQu6TvNck7VOvhKMyTn6BCNTPcuP4GIUMjpvPhSQfq0Dz4Q6AmyhKFSUEf6K31TTOfA1wr6YKftyoHVG6HUkK0ls1eb5A+lpzGUGnJwcMsCqrUIpYZqR/hlATEGsoBS8jxmcJa8kiHVfGSJcig09K5PntzAao8QJk/p5N9UaUwAQISAUDMwMR9HNBLq+MC7F+MsPZWSktDc6ejDBBJFN/C9NJohO0qRrQRYN6IVmUqQSczM4dQvCcws4kNfZ2bvstcxJ7qmxNfpefln3ueH4PDhY9nDmT2dD7smp+c+9HjKmR18uI285P2TgutKctrQ9XOlSN5OrgrxD4ElfnZ9+W+/gLn8tnssq4+ex/d/BYzlx/8SjOWvcvlWq5W9k2jKEd2xAEREXHjofQJzC4dQ8eKUPZkc4UQCMWm2fW8QdERMf8sz8C7JFEWci6UFkHxMPnnh+EWYJX3VVapGyLPG6Ud5PYMTu/aTn4uDYqY0RgmyAZpeIuHzQIRAiUE5EnbN/YwyVyNgWrcIQHEr5qRIcL/NPvm7EpLtzyKEJTsjnGCnoOAAsHxlWKItAX2UqSqCMfEhxcTnoKC83aABMWSPVAqo6AR0l5P7OGDxKdEyz35LnwYkYwbTnOEXce4NjJKDSTUmsF2m6P/kFcrMQh7Y5uoMXxHEQiaJWBo0u0X2ufD5/fvs+Bq36T1JkrKQZntVR6YsJDA9bpGSNN9NLsx1BqHk+9+fpfcprScYgi9fzQAqlHECt0ACVCkls3qrIIe5csIuMQ3oMsAMhsxWfyonNibLFVm/gEn6aFdJwvV8/uy4moxVLiwfzsheZp+OrznocQ0ZL9XnHZiBnnQZsPO9zOEy4zpdoymhUrc8jv58Zt7MIcItEmuoeNy6nX1C+RqbfoqZ1clhTiJEhgPtI0SMKG9lCnOZ6wGinkvhbcN1Z7aBPjkCkwnA6/x+Jcnt+cwYiEDQO8kfy3QtjpiSOkVkd58vBIYTMhvjmte1r8TElI4bgkckJnVc5+tdTNfnFHCTQCmAqUpj+gxGTHLbccXjRmTJvV1SYuhqyvMQkyeuAHJ3nV0m5rXJ1xXX5QsyhlGQgVMJyNsFz6evMLHuwTA8JjNAUxVHJOALBhPYj5rXiS+RWCTeO2zD7UQ1s1xBi8lPl4OEcuF73qZ0SXqW6iKE5z1j8nEeCMz6M3ob+xOCSk4kpPOYBqj0RAqU+5lhdJWYQCmZR+6HtDxOn0KobCPJSguRXjv/DAaz3LLm43Yxs1hks8k4B5WYtFRn4UomeorIbbk6ya9X6TNWyimkKssno+Ax+TKtv2adxXTfSANPn4CabQT6jYKvxOQBzRUgTP5kRyRBvCB4FZhkzraWk+wzKEp+Y/ITZ1A9rvWU0uqqB8FHgu8xwPtuLqUng4qJiQ4lz5tOvjxXSwwnKlVH5M8uwTLrTdJAPnUiZpBKv51gAmqqEfGVS+z3/AAAIABJREFUmgKWghGwSzUxnLzn8hwCgF1K2LWGXRCsZTAaCp4DX0u4JRki6eIUfMVrNPVFLvRc6TEwfMhX3Gdv6GUcTgomoSoB1yhEw+qLUMh0TxNTgNK4NtM2fKXI3gmBoCUDf9K+hYJspOw9xrR+JAbOlyr9kwhKwi003NJMQUG+ISsZao1QaO5DrnFJn8FgJNyqgF+WBI6jh9uUGB8t4Et2SrIWgyqYKMUUBMRzqKfzFBXZxVyfEY1CmLom0/XbmAkwhuRzDIsSbmEwJ6Ey+CdUms8tFUFuaSZZbij1BLjJEBKsRqMQC4NYaLKgD5JXY10g1vQMRq3YOWmoaCJQTfeluqQ8NbGBseKNMAf9ICbmWcr5WAtDQKYVgVZOc801ITk8B2AYj1ZkIgvDf0bzXwakmaXMS/I6xhgRnYMoDIRW/GcMhNYQWnNSRGtA5gmaxHpKxX/e452qknw9aP2z4DYtQkkIIaZ/7yw/D3D+ZVwipjaJX+a/X5flW81Yrn/jcfzOP/xP8Wy9w21f4/nyHs/qe3zdbfDFfounix2+Pmxwu2+gtUeMAiEIOKuhtMeyoVzzZHOEEBFXl2v87odf4XfXX+P/+ux34JzCRxdXeHtcoh0NjocKRengrEJVM0Cna0uUlYVzEmNnIGSELhiEUxUWp02Hr+/WKLTHsSuwXXW4uVsiBkAZDzdqxMSMVcsR/b6ErhxDbY2HdxJSBRjjIUTE/maBi8f3cF5iWY746nLL19YjukOJzbbF7idbiCc9fM8b9enFDve7BXyrcfrkHoe2ohLCKSjtGS7WGejSQekA98kS7sJCFh5lZTF0BqZ0GHYllmctjrsKi3WP41crVE+OGD9dwZ+PgBe4eHqP69slqzU8k3K3z3a4u1lAqIjYK5w82eH2qw0ggOKkx3hdsfLDBMRBASqiXA4Y0z7ZY4H12RG71yuolUW4LqAfdbD3JS5e3OHy6y1lpkcNfdrTX7s3iDpg+2SP+/sG2BuI9YgwMjwGAActOkDqMPlO/d5Ms5qoAsxrA7cmow0doXYK4cxCXhn4DQN+hGPgjugYMrTcdNhfLyALj2Y5oPtkDd8EmO0AqQLij5Zofu8Wd69XgIoQRw1zEBhPPOTaAm9LhC2jaHNXqPAC5Uc7HG9r6EsDvwgMJGr1NEkQTUz7GaCvDcR7LcRPGk6WXFhglFB7RSnllr8Ly4mY5uKI/rMVQdjaAk4SaDcO9Q/qqb5iPPcwtwr+ZY+wN0xVDpwUCY1n2NDWwnxWwjdJhvjIQZiAzckRd6/WXO/KIvQaYpSIIqJ6qzGcesQiQt8puFMHdZ8qWFQEvGDNjASKK8XBb0W5rF3xcdkLdrZayfUWAbKTlCSeef7/KGC3gZNNAPzSw9yr5Hf1kJ1CqALUeoT8vEYE4LYespVTkJE6KgItzIoBX5LhH7/XQbwpGYSzYBcsDhrLTxXskpNJAKb0ZbcMWHypYBc8t9XXlOvZVIEih1Rrcp16THWc5Lu5fsQtKC8Ohr7MqIDiLvk7daQH80C2SYDhROWlmlhHaYH2PTedK7um39kcma5cXZMttauUwruJiCsH89ZADgwWUoOA2TPZOQcL5fCVPNFF5p7grLxLXaxVnPpJ6RWVsKuA5rXEuKECI2qge8+i/txMjFBOx/Vl9l9SskopIMOXVC8mUG6XVCJkllWk3tHqCth/GJl43AuMpwHFjYSy9DcXd2QphxOy/OOWnlhfEtQ/7B3VxxlAAwS7Zo+JnQwl0D3m+nOa9ZTSLcigHp/xsyR8kiynifmcUj3VY6RJkBymo0YG6viCwNPX9OSOG06yDecR65/wNe1THqvZ5/MY0T1+kPqcmLVQcPLDrgiOc49o/Tbi+FykdGhM/ldfzZMlOalXpfRoVzOsSfg4pS4HQ5XDuGaQUHmTqj3WAvrA+7O0iW1VyctZUXL7kInPigL2XsaUjJtu8YF+3PIuTtdC9p6T/U6ToUmazs5hAdVFLN56jKsk0VRpssnmRHFuq0rn3C4Iik1LefPilcfhmSLr7OaJCTVEHJ8oqD6iTPJa3dObaVItSU5t1wMBMcE8UKbU7ygZwAMBjAs5seO6p6Q5p/E+TCOuri2OTw3DdwqB8tZh3OpJJhyUQLHzE3vHCStOSJT3Hr7KkzwEzFQ+RLiFnNhq9mSyA1QlWbR0ZMSLvZ8YY7dQKG8t7JKeT18r/qwUmdtCzixzOgaqrtJxHh3sikBXd2Sms3IpSsFOzJQMDJCFRYzTdT15Q/eWacapViRKAV9rssbguvTBTiFC44YJwKp3yP2QsmMKsK8Ncr+k7BxiAtJ+YYCUuCt8RKg0ZDsiNAXUfpjAtHhQDfJOb2auC4lxSrCNqbMSAGJlyAhr+U59yfR3o6fkWTHaCcDmnktIOU0QcD8eMJ8ZxAIEjjmBNstnBZnKaB8k8GY8kbefEmujZxItjKGvEw9f8nOY0J+3xPD/Swr7a8FYfvd5fO+P/sEvfb1/9h/9V//Kjw34lgPL5ntP49/63/4OPrs9gbUKUkZIGdHuS0QnIUzAs0d36KzGzevN9Dq5Vxy4jxw8w0TohYW/rCDPB0jlYdsCQvHmLa4KxNwLaCLUZoTfFUDpuY7cGfmm4SB4zWqTWPIGpXYKfu0hjwrRsNcw6sRqjgy7EYPk4Lij/00fJOwjCwwyyXcF/NJDtcmfNsipLkV1HLyrnZpYWuEEQhEIXEbJAWDy2OhuDnbJCYNqEJNnQkTArTzUQU3sK/CANasCRBSAE4hlgNqr2fuXGNdoIvQ+sxuJRRWJsdRx6iG0J2k7APzWTX2O9Dhlpoc+KjkKjFt+66mWPhn72EJdGw4k+FbArTlwdouIYHg+VJ+SKyXPjRo52CzuZ4YoGLLPwpFRVD1l0UFniS8ZYTkw5ZLpl6lQ3RJE59qRUEaY+5TUmtYhknQzh65w1p/rUz0ZVd0SkPiS51GOfJ5uGZYjBw7KpxCf9JxQcnCUZbbSpXWP6XiK5E1KTLivgf4FA62kA/pHDHTJg4qJKUxex9w/6Ku5ekdabtuu+X/dkqXkBYgkLyfIyDUyk4QtMUbZY4aIJJ1OlS5JGpzZKd0lJirJd32ZmFCRrrmQWOrEYuue3r7MVuf6nupKzAz0hte9bjHX0ngkaWUKx8nSUMvjtylJM6sbZBrk5wFnloBSFoppkG0OmCSoUVIOPK6TZzYNXF3NdNBcxTL3+KVeRJdY6JjewxITM8+QF0yhK+aYJZR8jWsSiByRZLpMAzVteixi6kBUPWWQmbXTDyTUuuVgPB+TSLI9yjoxqwmQZLU58bMUSZXAwXIoZmAoAqa6kSgSyLid5bJuwTTRKMQkf5U+vvP6DDhznyNl85HXnEv+SUtA4kqB4ZSfuRxUk2XVvqY8OfdFTunCZr5WdUdWPku2M5tY7iiHjYJyXyAlfyZZctQzINTd/DkL6oEUOZEDWcI+VS4lOe7D6yID7Mxm55qWzNDn+p6oyIYWO97L8/Wf36dsF8hhMpkFRkwscMDksYry3boZ3c2y3SwFztJWbwT0QA9mZiFz+FC+XjI764tUedPIia3NICjLSHOisO7CVGGTt53rXXJ1Vj5ugKxvrnRRNk7VLA/DWx76ZzNbmv26+TmhEPQzaoFxJVHeB35uS+5zrnqRNiQZq4Q5+hkYjbP3mN9JYWIeoxRQvYdb8AJQQ2Ti6UJPj+suTN5CEfg5gEjgKvkJ8/e0TEE7cpilh7nfUfgIZZlKmveFn9c5EXb6u0z77gJczaoPmWpBgiGDxeRVnhuRGDg5Mj1WDh7RyJTUypRWMnT8LgoFmVj4JFctklS3cwhlBl2Raaoq74+AHFinkq/ZLCeXQ3pdBKT1ieEkCBQ2ENj1BIMT45oAKJSYmbYYISzrSyDIYArvKXMN3N/cO8mTFx6wlZif2w+sAUlBQKHQkH2aLXJJ/porRHJCK5BkyPqdQB1h3cRqTp2UWarrA5nJXF+SGdEQZu+ldWQuhaCsN8tTv5nA+kDiKjJLqRRiz9kaoTXBoPeAkIAdMclmgXeltSnAaJLd5moUYPr9HUntnwcqY/jZh/4yAMv/8VcALP/Orwew/FaH92gZ8PGrR1g0A0QR4ZyClAHff/kGP359gdWix1dfnEGUHsvzI477CkVlcf6SgTp1OWKwGtYq+FcNFh/scLhtENqCDNp9gbByePrbb/H2Zg3Xa2CQiK8ryEcDQjJRi8YhBAmcjFA6ILytEFYOCAKLsxZ2reD3BULDD5c5H+CsQtgZxLWH3GssP7zH7rbhbJ2KcCsHtLxRhZWDbhzk2wr+xAG9hDwf4A8a8lkH5yTQarI1o4TajvC9hmgVYp38F6uA2CsydE8H2F0JuRxhjIc7lrDHdCnoBEQ7CX9iCZQBAmgAYpTQBwW39AxhAOAXHvGJRTgaguazETEIWMF0UvF4wDhKSBMQbwrEIsI7Bf94gLqkjNkvAuSRj+FgEBKwghfAxsJpzQRJByBIuHMmpYq9RjQR49azuiMKiFGgf24ZNGMi2bSakmhhsxwporqS6F5aCCth7iQAAsawDDD3amJuhBfwq4CQAPJw4SCcgD2NEwOY+/CiJlipriUO37MovzboX45Qdxqhiqneg4Xuqk+gsIgYHnvIXsK+N0C+KVHeSOgWOL5kVUb3OKQ6CmA44WBNKMA1nmxUkg2Hgt48fRSpVJ5yUV9ELD+X6B7HBOYlqs8LDBcewgssP5PoLlLH4yqiuJVwTZyAWu5NBCgPVSMwNvTpmp3AeBLplx1ygqYAxnmQO24J8tVIJqm/8ChuOSnSvfDQ9zJ5TEVi2Qims291SME7+iAm5ms4JUjPATZ6EJSLB2A84T64JsnWnUB5LXB4n+cdgeE8BJFZ6p0mEHqeO2Bmu3wdUb+Vs7dQZXaM50ym485/Oz4P2PxIwq5n8GfXZLAyAyYcgXiWmosI9OdMcM1MVAb244aMmG6B44uYAGBiBiPPldnP9Q1ZgglgqpuQ2ZspgWGbwI1L24rzRItPNREysZPNKzF1zEpLD2OWgLsKEJFBK5RBZ3A6b9OuGXDTbSgBzfLoYTv7jFVPxgngdZzrMBDThEVKL+1POSnkwffOLjCllbpmDhcq7iMO77PWRkRKr4MChi2Z21zvoQaec8P8N3bCnpNhC4Ys0XAyd6/KGNGfpnOcZLGseEnAKgDaAu1jCV+krtokgQ5KAMm/258Q6I1bsnbtE4HqiseQpeK+IIDOXbj5HpO91iKw7iKDJNeQXQ0KqSYCGE4kdEtvp7LsU3XNDA7tgoBYJBmwdLPcWESCVYYF8ZjNgZ+RYUsWxzYEvP2pQHmfmKtC0Ae753PHFcE1BHB8oqawo6DZmepLrouBQwTHIjB9NTOLvhCTZ9XWin2oAz9741JAeAWI2fs5bOTsq7SJRRvoT2UdBwFmZkXZ65q8r4WYWEFf8f8ZhLqKCa71lUd3ppjO2hMQiwD4NVOVQ8HJOFfJqQN5XCd2MZDtEyGFVnkxTaJElSZGEOEadl32ZwamDXC1nIAefbwEdLEXyVsaUdyOcEsDV6dU14rnpdhZhAXloMOpQXlj4ZaKEyL3ZAl169OEIKWxZjdiOC3pv0zeTV9I+JrAUQ1hSruVLmI4Maguh3RfNImd1InhDHCVAlIAUe4LVUPAuDIo7i3sxkD1lN4OZxXK2wGuMZxE8ApqCIiaLGJ/UaO4G+FrDdVldk4iBjWBy1DqaTI8KgV3UsLsLEJN6asYZ8AiR4/ogVjyukGM8IsCuXtTDZ5spg3EOUZA9I7psBEQoyOTGPnBiUrz92AIJgeLKCXksU+SWQGYJA/O1SHO8yc4gQpglr0G3mMgBAGm0QSfNZNaYRIYrXhTyoxkTNUiABBXTUqFDXNVCSbeYGYbHy4ZEFoLUacamTEBY0UpNUxB8BfFO37K6TkPwXGS4D5cGAYU35HTvrs8kPGG+OcCzW/j8uskXf1lL99qxnL5/SfxD/7xv4ePFpf4wf0zBAj85OoMAPDXnrzCP/nsPfwb738OAOidwY+vz3G+POI4FhicQlNY3B1qfPToCl/db/B4tcfHnz8BAPzuh1/haX2P//v//T0gAtv37tCPBh9dXOGur7HvS+wPNU42R2zrDj/54TOsnu+wu17gr330JS7bBW52C7hXDRYf3GNdDYgA3t6sEbyALjxOVi2EiGiHArurBZZnLV5u7/DDL56gqCz8p0vgZYfVssPd3QIvHt/icreEHTXOT/Z489kpzEmPurI4HCr8jQ8/wz/98gWlpVXA8qTF4baZqkUuTnd4/dkZfuc3v8DHry8AUAIrVERRWRjjcbhtoCuHDx5f4/OrE/gE1rXxCEGgKByawuLN11usz4/Yf7nGi++9xVeX20km7J1EbDWWjw8otMP9boHgBKSOaBY9Dvc16iX7P72TsK2Zal5QBrx8cY0vPr0gCK78JBX+3ou3+PiTp0kyK1Ge9BhbAlPcFRCnA6KXVHjogOgl1OsC4v0W7rIis6OY8CpVRAQQrIQuHWKQ8JcVYpXkkBsLWIliM8C+rclUr5gY6z9fEHQ7Sda6V0BF9lpUHrFXKLYD/JcN/JKxjHo9wt0XgIrQS4v4qoJ574gYwWM4JmnPXsI9GxA7jZcfXuLt/RL2ywXUsxbhiwX8qYVZjIhBwu3SyNaEJP8U0K2EawL0ow7uqkZcOCy3HY5frHD2nVtcfb6dGOXipMe4LyCOmvJXL6CvKTd0Gw9hmQ7qto7fPipC7nUK7SAjOz5OSbGC7Gm4GKG/LmCfjhAHjVgFiE5S4rjxkAOlt+WVwnDugbVF9WcVuucOYkx+sZVD/VmB4fsdw7WWrFIRuQ/TRIhhDpQJJnVANgTEdh0QCnZ6Sk+pqPCCEteDpE/2JMDcsj5E9WSUyxsJV0dEA7gmAAqIMqK4UVCtQPfhCH3FSQ92JRL0+RJwJ6nHchAIdURxLTG+N6L+UYnhLMCveF3FdB7FKICLAeaTGsMTnvviRk3ManGbvIerOQ03GNbyFNfsFx3OWEWSWTBfR0jHgZBryDraEwdzq8ky12TwZEoIhaAyAbVH83EJ6YDjSw99lGn9rBxpnzMttrwR6B5nxjGivJITYywdMDzyKC7ZsRkNEFRiyBIjWNxTDipTmmv+XXgec3lNf3LzmkBWJSZ/3MSJNazfUnaIpGQodgLdkwBfByw/1VPwVw4Ry32eUYDgPrHW/XnE4it2dC6/YMCTtPQaV2/5npZ3cQoBy8CZzBPXlT3EGbgjEii7isBPpYArn8K7ckiVT+qCzIRPtSjnwOIVgVH7WKDYE3wLj4nt1B2BY5YAl7dzqBArix6EYWkeg10QrIwnnNhYfhXQnUrYFfezfRpR3qbk3ZLPdYuZFV98xTFCf5pBL7B4w9Awu2C3ZX1JSWtOss2AWg3JA6d5zupLMot2QRaxfSJQ3BPAkyHF7OtXWVmQ5K6Bxw9glqY6VrSMSwYzDWtKS+k/TcygJRgvjmRLh61I6yDr7EsCX6YJU/K5fO1ZWZIqWaLihEAOYbKNQLmLaM8lyl2YgGhUAu25JJAfyWgzHGmWtYYH42STpMPSMTyqeeMwrhXPnWWw1fKVx7AhIM4McJT0uxZ7vg9qJOiwNSW1IoU+ZeB7fKxR3tMrW9569GcK5X1IE0oS5Z3H8bEmYB95nrozifqa9TGZdTVJ0moXlMkWhzBNmpiDh68k7EKivPXwdapaSWoBORJgSh8nJnbcavpmC4nyzqI/M9DHAN17tI8K1FcWqiNTN24NyusB3eOK9STnJczewTUK5VUPSMEqkcyW2jDLWoWA6h1cY6A6B+ED3KpIEzJMzC3vLGWsht5dfT/AbUroo03saGJEC7Ke41mF8qqDrw3k6JL0lWFE0SjAMfxIthZ+VZIp1RJBS0pjuxExsbI5mEi4QID5MKQnyVFDVZDtzP5NoyH3R1aQ9AN/WgdYh9hUgJIQ+5bgEwSb0WjWjwBkEp2fJa7ZP5oCdkRVEnx6z37M/LocMpRVDD4AIYX3aM3/A/Tv5roRm7ow8+uBmTFN7KPQ+uczkd9gMr/14T3ffR5f/g//2S99vT/+u//lv/Jjwz9n701iLMvyNK/fGe70JpvdI9xjysiZrElUS7WGHQKEEItuNbChlyzYwxIQYgWsgCWwoFcgUIHUNDNdUHRXVVdkRWZkRGQM7uGDzWZvvOM5h8X/nPsssrKrKXUVnVnUlUzmbnbffXd8dr7zTfySA8vFdx+H7/yHf4PVtozmVajKngBs1yXKBCaTlq6ztHclk9MdfWcF0HRa/JDRw1cdNDS7HHz8Y5Q7htpiSod3Sn6uAmwy8fF1BmUCfiv1FWdP77m+nqNtwDVmv53JIL69ABReGLheQemFbSs8eipAZ2gsamMIUwe9FjCSwEO+Xx/AbDX+pBcWEiDzqJ1B9YrZ+8J+KoXs8zqTgCIvbJ6vxHsWikCIoT6qF6b0oVdP7wQEmEbjKg8+sjJ5IGReXjMdYJl0hMIMhVwCkexaZif9kwZ1UewlXAr8RDSXqhcPi+pj/+VOvF9uJr/XW0PIk6xUwpLGgvlOidcwAgzdSeKv8jIYdBMfB4LCRKpBjZLmFFwUsjB6QYNFknmjN8xVMqgfJuJdE+lOgCCvM7u9vHiUkhHlgINIa5P0rLgTD9xQhSib3fulXCmsr6kjo+pFWpvksYktTIOz1CWp+8TqRVmpIc7Ax/MxCONntzLQt9vogeuIxxIZQh3205ZReqq/boMAYvF8rDlJkf+6jx5Hr75WNp6usx4EtIVMwqWS/HSYJAaM0a+W7lGRAUfpsZWfJ7DiokQ5mDDKdZPUWA9yne12f11S0q5IGYUxTSmPKnrz5IZgvDZ6iOchSnfVg7916T1dGZ8nL9sxMSQqhbuke8BFSeN4DYufOT9J7sg+lCjJWlMdQz8P+2Csbr/N9D4Pi98fblcksYzBRcLoMHYaJvClgvxeD9FrNtkDJJBtuDJKDB+EViUpp2nk/+n+SYskrzJ2BCamJYXm7KsQ1MiMp+oPCWeK20ty1AcDcthfw8S+jbK+B88hMEqtk58sLcM0Hk+8T3S/P6bEqO2l1LHzsXuwfofI26z451I6eJrwED+nhCAlJkjkyPIMDmVkJN1+e8MksYVyEAnEpn1JzFYC2lLD8PXzkfoex0qPJBEf7zMV00MZw77knt4D6JQynCYGTJOug7B7yu/P70NG8eH1T/eJjpJcn47f7dlCW0fwZdUoI0+9kyLBJgZNhfhsJGZv/1ylXs6g2YMgohw03/sCffIHxmdDeUZpdZLCuuzrcllvpdYkhRHtQ8XS8YYoFY6S2z4IWC0fyP20wnRJeho9rFFmO543GP2JIvMOX3u2lQsMlf7az9P1FiAlO5Y8pollfXgfJLlo0Eq8jIWO+/GAAYpyY+X2nkRf6DGgJ+1LsHpMgU2eRTX4WPUhwMQbkcfq3jNMhHEU1tfIfR+Td027f/8kR04/S+m9KgSRsuYG00jSrWmdPF/x53JPO7zds9UC1oSB3J+TMG4TED+j1V+TBafXj17Hh+dHger9H9uO7HAQmWw6J70AcdU7fJmPCbcp0VZFWbFyTkDcwwRYpfby0CSTTTLSxFw+DPJ5sCSvZLrm4/IwdTaxhD+7/w/XhT2QfLDOiBsSy9j1pHTYr0lhYWQ9R0nsz25v3OwDcPsn4RIf/kIAy7f+vT97YPnZX/3FAJa/1FJYgCezFaeTLbf1hNw4eq/J9J7iP5rUtIWhKXq09mgdpHeyAO8VWgfyo5rMONRUfued5o3jFVermfQndhalPUOTYU9qtA6gDEXZ0+uAd4pdm6MzGdUV8xY3GJT2uMFALuDHFI7gwWMx5UDI3AhAvVfo3OEnYEqHGxRKe2GjIP5OofL4kM4DDEp+H0DZGA1eeNpORtW6GNDa0zdGwJD1hAJh/LwSUJn+6ETAGkyA0qEzj496jNALS4R1UaUTH34bgXLlJATGhOj3kh5BN/WEzGN0lBrG8JXxr50JBO3BxtCeLBDygNf7DxVfyR8jl4M7GOSY4366aVwvAVodCLnsl4odmMrtE0b9JIJQLccbMj2GnoQ8oKMPU6o3BFT2R8LcBaUIRcBVEvwiwEZFJiSMDFBi1ZyJoCqGz7hCGCcVwBcyCA1Bje+l+lgAjoBXV4aYkBj3PQJgbx8MLPQDUJbssMmH5+U1vgj00avYzyKIjgA02LBPH43bTx7M5IHzuchobR39pEmOF9N6XRnAqzGAREXAmnyUPgtjemzyw/kYTuLKMHrJhliTEVTYe/SylJAsnk7TqAh844AgJjAHEwen8X4eU3rjJEIa5IXYxZc62FwR0FoAvByvijLlEINJ1Oi1TAm76dybTo0yK4jsVfKQxgF3V0lljYoTEkn6nBJTk8/WT+R6JSltCu9I7FjqaUy+ULuRN1YBcIwJ1UkWGjS4CET6aQSvCeyo/b/TedJOWJ0EBNKSgLGL19PWsj0TAYnq9kAsdR+m7rwEsIRFUyMYfsiWPqx3CBpMvA8AdDwXe4/h3rsqD8keMAajxgmNoON19owdjimwJ6Vy4vd+Unkz2S89RDlpvJdTcrCOPtgRbEavstRIqeiPfSCXHfb3sDNyXInxSceTzoE3CowAnGEqzNxQ7gfT8iypCPASA6zGz5/kAUy1Pui9JzRVXCVfbrp3lY9VIuk+jPd28qxCBNSRQXzohfZWjZNb6fR5G73waRIlekgJ+/tw7JTso58weiCH8o8D0gS2Ht6LaYJhHMQH2X8/DrTT87n3ZPoIXqV7ML1eJlaSvNhlcd8t2Dbdyyp+nsRnKfr+0vOuXcBZxrCdNGmRJnZ8TEhWLoyTKOlvRQrTSf9OSxr86wTo4gSEH6tm9PgexImNNAmVwEYC3GliIsTwF+X24DKdp70nMwKcuN/ja7Xag17P1yS4CVSnChvi8ZkBeb9ABFdxbBNTacfezU6Y3XRtpM81hu0oOQfp98kvLdUjXdGXAAAgAElEQVRh8kyETI+VK3Ly1HgOgxIwiRMQ9jVA7OOxdn4Ei8FoQrkHh8mPiZf+ZOL5Tdde+bAHbbG+ZKxEGf2IfH3xSA+l9xLq00YmUu3re4IxAqoSKFNqDzJT5cfDOhLn9vuKnPev/X/0x6aTGFfL7N6fCQIGB//HgV4Iwj4mUPhzwKBS+7HVfp/N14Es8fPrZ4Hiz7CWf6rl/0/psb+kyy81sDzIxMiTa8c//eYnvG4PeLNY8uPVmxyXO55Mlvzw5gm3qym/+dZX7Iacs3LDcb7ltpvSeoNRgS9WJxwWNbfNhM4Z3j+44Y8u3yTPBr5/dsHb1R1fbE+Y2I7PlqcYFVg8aihNz7PlMWfTDbf1hPdPbpjZli9Wx4Sg6AbDyXRHO1iM9izrkt4Z8sMtISiUCmTGY7VnmnW8Wi2ojnqOqx0hKK53E8rDDc5rTidbPnn9iKen9zx7LXLfR2crpnnHqilZlA27PmPb5nzn5IplV/Hs6gilYPF4w6xsmWUdy7ZkXRcsHjcYFdAq0AyWedGy7XJy41jWJavzOcdv3dMNloOqYfCadV2wu5qCDszOtuw2BcZ6lA68c3rHzXbCalPx+HjFQdFwW0+4uDjAbSzzd5fMy5beGda7kjwbKLKB66sFB0dbdk1OCHD2ZMOrL04pT2pCgCIfqJsMPzewzDh8umJ5N6WYdrS1oC5tPUXRMwwG5zTTScvRpOZuJ6meISjKbODmZkZe9Qy9BRVw9zlv/vo5l8sZ7VL6TLX1lFVPczHFvr3FAn1nsW8MMhl5X3L2vWtuVxOOv7nj4tUhp2+suL5YQK/Rpz1+lVFeWtz3Nvhdzvx4S9Nk+M6CV8Iy516YcBWg00web6m3BSpzDMsCVbqxRiSbdgx3JWjxdLqpJMz2dwXqcc+wsajKMZ039L1haC2+tgL6A1KbUzhmH+fsfq3GbzMoHHqZ0b/RkZUDPJswHEk/m8odYWsZjsOYyNoXmlAKk66nPeE+J3tvR3sxwa40w5OOfNLRv5riZ5Loile4mWewHr0z+AG6swF0IJv2DDcl5YWh+16Nq41MeGwtyiv0cQuvS9xBj2qMeFqPxANktlq8xgFUDMQiQHZvcFXAnzjUZICoFNCNws0dxXlG/X4Hg6J6kREK6GaecNijbjPCaUeoDXZpGfKARYKudu8OkrR7neEqYUT7Rz35RUZ3OqAGTfXC0DyWACSfBboTR/naSshRJsxweyKTQAnAqkHK6lPwlIvhSCqB56AIE4dZmtE3m60Uu7ccIQuiOjBhPP781lC/IRMiibXUrWKwsT7JKXQrHtoE0pVX9JUMvLsTh90YJi8V6/c9xa2meTyQ3xlhkb0iv4+DrCCM31DJgMHWagQvwyQGPmUBtRRfsO731S0JNKQk3ZTQ2jwSGWp3IMyyyIDluO1OgEv9OJBtFd6EMcXVbhXllWL7NEiw1aFMBjRnnvJaj6BTOwlC8jYwfSXy1xGsbyWMZPFFYP2ODJZFkhylt0r8mLoV6ahyIg1Ox6a7vZzWFdL7qlyq/YkhV9PY4aoV9SORrA4TSUvdnkFxJ9vwVuSnzalMegQLdsPIuPhYq5KvpS82scDNcWD6KlW5yHu1c8iXEqiTPKzNqUhfTfOgr7aH7Ztq7IdVEWDaWrpxxcfLKA2tzxjZ/PI2UJ8pYacbSdhdvyOe0ZTWCnFiZKGilDawPZW0Vj3svaLShRlZMy/PSXXtaY/0KFVN25qce0lajkDOFbIvykO90HLvNCFOsEgSbXMk5zSxhInl7CfqQaAQtMeK6lK6TF0m20ns6FApqtvA+qlhcuXppjKATz7VfBVVHSiaeUzGnch9mG3EQ2q62OF6JAPxodDkW0mH1QMUK8/ukaG6EYnq7GVPc2JH4GvrMDKr3u6DrhIIChqKlaObCcrJtj5KptUYcpWYd5HTarKdj6m58izY2tNXeuwNTbLZoGV7pk2TNSK9rR+JF1TtpIc02zj6ucFuBZy0R+ItlARk8YdmW0d9llPe9NSnGeXtEP2iClNLLYmL9Tb5cpDtrh3diSW/Fxms3Qlz2c8lZVYPAV/ZkcUVdlbqdIrrjv6wAB++5mk1rZNezUkmk4FGM1RmTKo1dezftGqcUTGbjv6oxDQDymiRj/YOX2ToToBmyMT+YLadsKq9EwVCVew7YbeSSkvn8fNSWM4EvIx4OUNmxKdZWFTTEyaFvFcVWVCtUXWLn1ZyPXYNKB29myJJCFWxT2vNM0mJBZG5wh60pqWXLlEyKzLYuF6oU+SyAGGVZ5BkrMPwdeCXvJVK7VnIFCqUlnSsic1knP/Y71da7+d4NH8ZlzQu/Yu66H/4Kr+4y0074Yd/9B6fXJ1R6AGrPBftgqt6yo9fP6b3hmnWMSk7fnp/yo9fvUHtMp7tjrlpJ3y+POWT+zMWRcOHL56QGccb0zVD0Hz/7ILvn12wG3L+8O4tcu14uT1k3RRcr6dcbmds+gKjPRPbcX0/Y9vnfHx7xvnLI3ZdxjsH9/TO0DnDqim4v5uiVOD2csFmVzDNe9re8vr6gNfrOUeTmvvVhNt6wifP3uD2YsGuzTHa8+Pnb3Iwr7lczcAr/oXvfUBpB263E7ZNjo836fGk5g8+f4eb7YThusI7TQCulzNerRb8lbOv+MbJLU2XMc06lnVJ21u2Xc7l1YL7umR1NUNNBrZ1wbToqHtLaQempVAU2a1lczMh3OcoHegvKwav2TU5+kvxn3702RNul1PUxqJrgwK6wdL2lsPZjtX1FKM9YVBs65w8H+jvRXunIivZrgu63lKWPa41UDmaLkPdZeSZAIuwsfibgt2qpK8zXGt49+iOwWtWN1O2mxLnNDc3M3TmaTcFrhbfqNkazu/mLKYNeMjOc8IqJ7OOkHna+5K+tZgXJd4rjJGAodWuZOgst6sJBMWmLtBri10awjIXFtQGhs6iV5auswQv0uskXdb3VkCbArsy7FYloRcNW35jYGPRGytMtTNgAmYjqb9mY+hXOfmdwWbDGLC0Xcn5842AOrPTmK0RptgEuoOAbxODrvCFF1Cp4sDBKbIrizIBuzEwKLJJh2o1xYVB1Qa7NPh1BgraXYbqFQc/BTaWEBT5rUblnmypydYKs9bYO4tuY+XHzqDXlqG1qJRY6xTKBkKvyZYi53brDLNT6LUlu9OYnXgss3tDtox/fGqDqZPkSEuXrAYyj7GRye8U2UYGJtUVY8WK7kTa6xcD5jyXPtWl0DxuIh7N2VcyQMtuDWYykK8kBTlfKcqv8nh8IvdO8uJU4YCXDtf8Ps6mO2FY1aDIlyqmFIN71FHciCzbtIp8qZm81COLxiD7n99qAXVGmNRUDWIajYrXGiXsphqEfTJRaiz1I5rJK0W2lfOoW9kvV4rMtrhT7JMaibH+kAKTghEpc7YWQCHeVPFNPkziHY/VxxCkVoBzcRse1KPsU6KzdQzwsQKOXRErbdayL7oTIGKbQHEvElLdCjjPtpCtNamXGETi6ItAcSeS6yQVNYkpBvKVHLvdyUA/GPEiljd7Jh4v589V0Q/Zy5fphDFO/cSyvUA/k3/rThjD1M+bbWD62pNtY4iUC9hanrfiLoz7lPa/vI69yl72T5J45Tokls/WgXwdxtel5NvJawFqxb28l2kFVJk2hVJFuWgtdRr5KjJqVlHdeop76datrjx2F7DbKLePUmiZ8BCgB3JeytuwTySOUtZUw1Ld+NG72JzJz7OtnJ/ydiBfBfKNJ9+Kj89uA9WNk3Mepa2jFDlKhvFyHNkmjEmvKkq47U5AUXnnR4YZL2CYKJJJyczFymPrWHVSyH1qa1Gr2Fb2BaBcevQgcth84ynvXQSywhAW927PrqZbQkG+8ZhWjkfuvzBKvX1kSe1WQFC2k+tQ3g7YRs6JHuJrh8S2ynVPagYVwNSyXrZxsW4kYGuPbeV8Jk+nCpCve1EmND5eL7lHdA/ZxmHbKEutHdnGyWdaI9u3W4fdOWG6oxRax+sTrEwGDJXGW0W+HBj9cF4kvb7Qe7Y0BPKVI1/34jte9zHMyAtYbRzZOnruPGPSrN06dOvIVwPKy/+VD+N+KRfI1gN6kDoQ0wyyzfUgadK9VLEEq0dQqXuP8pK+q3qHXtVyz0WQaWtHth32DFwENKbuUYPsl1SMCDhUg5fU3BgmxOBRbY+KCbuqd4TcChCM0l7lI8hKMlcXUE0v68SUWuU9yjlU0+1fu2thcKhdi6rlixBQzf7fqZpkTIRNi/djoA/Oi4y17eSr66Fp5QsE7CX2Mp0DHx7Icz2h6wkJpLrotVQRWjyoHSFVjiRvZZTUBuekjsTv//21r4frPUyR/cvlF3b5pQaWpR1497vnPFpseNEcUbuMw2zH9w4vOTvcMLUtSgWKbCA3jqen97zYHPKoWNO4jCezJWfVllVbUk1aplnHR+eP6Zzl1eaAH12+wVmx4TeOXnDfVZwv5zRNhopM331TUWU91/UMBTSDJTOe48crymygHjKOyy3TvGNRtpTTjhAUj9+8lz7L7YRuMBwsdkyLjt5rirInN47JQc3RozWZdSx3FYeHW1bbUmT3medHyzc5LreUuTzQdZ+x2pUs65L5QY3zGhY9edFT7wqyzLFdlZw3c1ZtyeA1t/UEoz1Ge3xQVLOWEBR22qN0YFq1+KBouox1m7PcSBN7/7jDVI5gA3k+ECbCcvadZZh5vNeUBy1DZ7AbjT/o2e0K2sHQDwLM6DVtb1G5yJMBzLxn0xSE3OMfSIQBbDmgdhbvRWraD4bQGuZP1ujjjmreErxC6cBPr07ZtjnowGJeM/SG2UGN77X4WweNG8wYBHH1+gBMYHizIxSO9f0EBoUqZbQ3zDzGeNomR/eK07n0nioFqtF4LwDFF+I9TT2ENh/2NTWAsW5kHcNxL7LeTuPLgDIBei0gOg72gwn4xgqb12vcQmS5vhQtVVAy62WWlrCzhF4zDGYEmm4qM9Cq19BLaI3aGeg02aRHt5q+sQzxugH0hw5fW5E1Kwl30q1IelWvJOzFRs+fkQHw+l0BK92qYJgJePZZoDv04iU7kFla1UsFTsgCJneMtQg6jAFNw0S2TZTp+sLjqsAwjQyeBjcRljdJz5SXHtF+jviEa8Nwn4MTT+1QSf9oewhmF6t3DHSH4ktO1T9ywwnDGTJJBzWteG3dNpN962JYzzQI+xXrZ1xO9JumgZ+iuNO0J16uZQRf2kUvZZJybq2kn8JYj9PP5VyJdDHW3RSp8oY44FajnxP2MsfEVmSbJFFjlP02J2FMDA1aBro2MouSXqv2vslajR4+keWpuH8xOKYV0JU8o9rFGpbInqnoU+2nAjj6mRJP7QB2HathvCTSqoHY1SrHYZdGvHVR9j1M5PX9VACx7D9fAwvi14tSu1oxxPRZPewlvSlARzpQE7Mn4LM7iCm0KskLiaA6nt94/SRRdS8NTYN30woAG727QY7JZ9JhmCST3kb5Z9yWHvbrm1YYPe2EPYO9LFqYVkZPqsgQ07WR/fGFJLB2c9l+GvhLxZMaJcvynEXJaAumDjRHwhJlW2iO47MVz29KCE4gvZurfQVLqrR7IOVOEtp2oSVAR0FxK4DQx3Ce3SPLUMlx6lgz43PYvmlIcsvUZZh8zPpnvKLit2QPWuIzLCybvBbk/O/li7Kf/USkntkmkNVhZPFMtw/9eVjfgoL6yEQ5t4AJPcAw0Q88vOkcCQPZLjT1iaSyJglsks3arfgO03uBgOJRHpwknPEzTlKeBdCmLk4UscYnVZYgvsUoPXVllM+GgLd6L68fBLAleazPJIjH1tIx6WIPZpLrulJYN1dqSdXt9vu2vw4xMKiIEtsg2w0GqTxhv64v1F76b2Ufk1zWFQaf61iVJKwmyLZ8bhgqExN6I3uf7c/fMI3nOjP4wkQLgh5fT0B8oEZSdJPvEh9iwmoe72GRkbrSoAbxggYj7y/bMgSrBSzGcCBfWnxu8FVGyLRIcR/IlDEq/g2MrJxS4+g7GDMCQBUCocrFo6mUMHTRrzlKWxOLqDVhWhKmFX5WCpAsi/Hf8lxGFjKFAT1kJGMwUChzYTPLQljJqpQvYEx+hRjk44WRTPuiNMoaYWxhDyhTcmsM7BkrRhI7GX+mlEIZgzKGVG/yD/v6CyGFDcQ/1n/GX78gyy81sGyGjK/+8AnPP37MB9dP+J0ffoe/9ez7/C8/+S6vP37Ef/s7v8lXf/tdtv/zIy7vZ1z/7ae8vlvw2z/6VT76/Al/78fv88Gnb/P8/JjuRwd88dvv060KPvjkHQDajw743Zfv8t988qt88nvvUm8LwrMp5ncOuHh9yPrvPOKrD97k+etj3HXB60/OuLw8gN8+4fYnJ3z64Vv84efv8MXLU15eH5L9zoLN3YSL58dcf3ZM98mCelOg/rtjLm8W3CyndB8e8PrHj5j/13NWnx5x9dURu03B/bNDJmXH7tWMouz48v94l2fLI67u5rTPZ9x8eIb/4ACtPdlvH7K6mvHob+W4Dw8YVjnb+4rQGp4tj/nqizP6Hx7S9Jb7+yl3rw64+fSE9ou5ALmvKoqPK24vFxyVNW2bsd6W+GdTVK84+19z/E3O7HPL5npK9WVObh3FRxXVuWF7OaW5rgidYfYcHv9PAsa3m5Lh0zmLScP0mWX390+Y//2S7mKC+4NDeF2wXlZUX2U4p8lfZXTbnPWF0AHluXxITb+U7+XLjPrjQ4oPK/rP56iNIXtR0Oxy7l8tsJc5q1XFcFPi/u8j7KuC4z+w5FeG4qNK/lieTwRoXWboKxmpTw9qyktL2Frmf7fCrjX1usDtLIvP4OJ+Tv5phXsxIVtphs4w+3HO5JWmfJmhG8Xj/0sxnzZMnxvc5zPMFxXDxQTVaGZfKfLnOfbWolvN5JVm+kcls59anixWHHysKG40xx9oFj/KOHq0xqxl25PXmukzg1kbqgtFtyyYPdPSg+rkw8XcWY4+EPno5FyR32hUpzn+yFG9lnPX3xfScbm1KBWYvDQUV5bJV5Z80TL9SqMmA0oHHv9uoLpUTF9ojj8wqE6T32vCoKnONW//jzW+8mTzlpMPAtjA0ceB4kZYtvmnlslrxen7t5iDHrvS+IuS6kLz3n91jVJw8EFOdp1RXmuqSwGts6+EicuXmvJSk20EnBS3Cj3tUZ2iuhAAbpeW6UsBv6Fy5Lcmsi2RybQiNfQnPfrpjupSUk1lG5rqQsWAHKkaya8M8+demK9GRtmLzwRgFbfw5v/pmL4KFFdixpqcSy3H9KWkjpavo3x0EOnd6YcD2UaRrRWLz4V5OPoxqEXH4aeO+bNAtlQc/ziw+DwyWjd67Kk1O8XsZaC6CmQrYcSztYCxw7fvUQ6mLyWwKoseTLtB9vFeztvkXKSO/Rsd8y9hciH/n5wrqnOp5TG1ktL3txqqy4BdGWYvhd0obhWnH3aUN0ESeJfyHrqXWg+7C0xeiZ+4vA0jqCtvA7OXfvTj9O+22AaKdzYc/cTRnMLkdaD4/pL588CwcJS3AvaHSWD+lefok4HJpRe2s2b0Aete1qmuPd0iMDkXAO4z+fnspcdnMDkPTF8G6fNUwsplWyhvhZ2eP/PYHZT3nvImcPhTT3cgLF5KA7V14OjjgWBhcilsTHEfmJ47pucen4s8M18FhjLQf6fm4IuB+YuBxVcDh585qmvP/OUwMpbSaSky48WzAbuDbuGZvxjG47PbQHktxzi99BT3nsn1wPR1LxMtVUwrbWH2ykkS7EJYPZEdwuy1o7oNVNfha0m2PoN8Iywq0UOaGD7l4OgnO5GczmUd28DxTxq6eaA9CZGRlMmGfBWorh2zlx35Shi0s99fjr7stG7qRrVN4ODjtYTruMDiy57DT1vsDop7zzBh9A8XyyD1LbeBfh7Z9lzSXPN1kPUj06l8YPmDgepGrn1WBznG6IHNV4Hpq57qdhCp77HIUL2B6magXDqGqWLyqubwx2uqa0Gy5VLqR+5+w5GvPet3oLjryXYCgJojAcpDqZm+ajn4ssG0gWznsbWwncX9ICmxtSPbDJT3wq4O0uSAy6BYepFvxg7R5XehuOuYvqjxmZIJrz5gGk95K+xitpPOT9N6YRwRJnKI+5OtOvQQyJcyEV09X0ePMBQXO7k+64HyuiO/78mXwtjZ2mN3Dp9rJq+E0evmimw3oFsJNTKtI18OEZwOuFImhqYfXwmb13thPWtPvnZRLj1Ieu75knZuyJatpPOuO/KbWhjVQdJhJQFaUl31EMhudwLoB09x06BcoFvYeE9ospstuhH5qmncCD5DZABdJRUwZtuRrTrs9RrdDOh1PX5GJd+v2XSgIbvZkl9tcYXBrhoB4zOpGrG3W/E3+oCue3Tn0N2A3rT4KsNPMvRyi17tUI2wgz7TmJfX6G2Lajt0N8Dg8LkZPZuqEwms2rUxEEkLa9n1woRuawGkXf81wJaAq2o73KwgSVYxWhJhjRFG0nmRtWrZ95BZCQMiguvYWwlIuqvRso0UsNPLveSXq1EGG4YBjBEWsx8IbSt+0GEYpbYhrue7Xr43Lb5pCb38zLftz/+Kvwtt+6fGCr+IS8pP+rP8+kVZfqlTYc/+iZPwW//pX2fdFfTO8MZ0xW7IeW92ywc3T5hkPfOs4Siv+d8++zb/1Lc+4fcv3mLwmqYWvVdZ9hTW8S++8wH/5We/yTePb8jNwEdXjzmZ7gC421X4oPBeMS073prf82J9iPOKu7sZh4fCYOXWcXF5wKOzFeu6QClo6pyi7NhdTXn63jWrpmDzYsHsrRUn0x1fPj/jV7/1gg+/eEqZGEPraJuMp6f3fPnsDDsZ8E5hrCcExVtnd5zfLzDGs72rsJXIDbTxOKdZzGq2dcGvP33JHzx/mzwfaLY5rDLMSctk0gorpyQV1w2GatKxWxewzJi8JdP026s4CjNBpIBFT7PNefL4npfPT3jvG5dcb6Y0dY6/LMmebKmKntW6wq8z7EHHyeGGXz99yf/w4Q+g1WSHLcZ6Dmc7Li4OmSwattcTJic7dldTkQo+2tG38ociKwbauxK9Nag3GlxrqOYt9aqkWjTU64LD4y33NzPwMD2u2V5OZWBUOXTucPc50zfEE/or777ii9tj6l3BZNqwfjVHTRy27BmaDHOex9AheS6mj+R1+qLAHQ7MTrfsvlzg5wOqEbbLrgz63a10dzaWw2O5H+6/OKJ6a02zy/GNRW8MnLXMZg1V3nO7nNJvculYXVYihdVBwpvWGaoaCK2hfJXRPBo4fLpiW+cMTUZwiulRTf/RguN/8pKLz04JJmB2MYzIBkkhDiIZJcD8nRWbTYl6XYos8aBHX2cop7Dvb2iSFLkVtjtsrEjw5j2uMVIfsujRywxOWrgq8IuByVFN/WqG7hTlN9bUz+eYxzVu0HBT4CuHng6o16UkDisBaL7wzN9Z0f7wkO7ICwN+3NA3dqwgsWvDcNqLl9LKAJxBUV5pmjOPOm4pPqkYJtLPifXY20zOQQx/GYONHjXkH03wFoZv7/A3BcWNoXkk3tLy3DKUQapWBgWLHps7wvOJdDqeOsxOEpLzk4butiS7ixUh7zao8xJfecLEUX6Z0x/IrK07GlDWoy9zeLOVazsII5NfG4ZZrCPZGvxEakmESYRspWieOFSrxAM6yCSI3UhCbyDWeGTQfbtm+oeVMJP5PuRJDZJw2c8CofBUL6zIoqM01TSK4aSnfJFLHcPjgemXlvoNT/Va0y9E/tme+AcBMAJqk0+yX4QxzXfyStOc7fsXU4pxeSkDugRqpl8Flt+NSbQVVFdSN5JtFO2RZ/JamKft2x5fBMxGky8V7XGguJWqknwpwT7DJPorj2VfVRCZ7epbnvJS0x8IcE/Jxd1CfJwpobc5EWlv96Z0z7oSihtFP5P1JKAn4Cae6pWhPfFRiqwiIyhA3xUCBpNEdP1uHKSqeJ4nEjZ18LFi+xZMXsHuCcy+gu1TmYQwNay/5aheSY9jYm3VINfQ7mR/XCnVLCM7F5lXb/dhTCoC8H4WKG+ihzATqWpzIv5O8SmGOPEh7GE/l/0eJsLEltfi0RymxH5OOaZuLlLm4k78mq6U+2P62rN7rOkOxM8pfb2QRQlvexh9s6W8f3EP7bFMVJS38l7FnYDJYCNTXAvDlK/l3NZn8m/dx3TZyDr2M8X0XIBZXynaI2Emxz7TLPapRsbLtFFarIStdTlj12ZzokYptOkE1M5eelbvajnGR1Itkm33qbOJnRRpsPhCs228PnNht4eYwpxUDKYNdHNFHvtWh0pScduF1I3UJ8IiF/ex/iNTtHMt7GOfZMMiSdW9gNl+oimWkkraLiJzZ6UipDnUTG4cpvas386Yvxhojs1YUWJaqT8pl46+0mPQVb5ymM6zfTMjX3th4Cth46fnUhcyFIps5+nmmnztGSp5z3wp45ShMnRzQ75y1GeWxbOG7RuF9GkWwiZWVx3tUSY1Jb2ATFcZ8V3megwbMrWjX1iK2x7de/qZxVUa0wRslKS6TD538usd/XEl0lSrx/TZYBVD7Ne0tUN3Dl8YzLYfgRaJwXTCzJpdh6syTDOM8lrlPMFodOcYZrn4LCO7FrQamUzdOVTvwWpU72Q7u46gtQDMh4v3hCJDtb3IaOsOPy3R20Z8kk3L2BWZZ9BLlUrIrLCUvXhDMeLBDEYL8HyARFTzoH5kTIv1wlomiatSUhkCIosdpbvyvmG5gqLYh/2kmhJjBFQ6J3UiD3yU6fjG5WHqbfr/z1u8xz8AvX/a5RchFbZ4/2l4+u/+63/m2/3ir/9b/9iPDX7JgWXx/tPwvf/oXxM2rC6kky/34mNTSC9ep1CPG9xOfGcjBR3TVtXWoI46/CYjO2rwLyfkt9JvZ7+9ZvhkLjK0XAYEVA61jcEoUS4YSoe9yRgWUjwvEkSRB469ekUApyhuNe2x+DZ8EbAbzTD1mK2EF/RzP9ZiFLcmpoMyDiRdEfvfchcO7ScAACAASURBVJGUdUeSpGZ2Cv/NGvWsihUS0J4N4rEz0d/TR/lFkPc2uxj/Hgcfqe5hOJAYz+zWkG0U9WNPcadpHg3YlcHNpEuvO3FULw3hN1eEP1rQHXnyO+kCRMHwqEOtLdlS40rxtPVzGTBu33YU13Kd2mMvfrXoiernMgiSqHliwALs3vRU53oMUNk98Rx/qKjPVEzFFF+VraWg3u4U+UoxTOIA+E4GijoOFNsY/FFdKdojGbB1Cxkg9gcyg+9KYaxsLfszTGUf+jljOqRtojfsQLZhd7Kvs+daBp0G2hP5f1BScD/7StEeg+qFcQC5LtOXsPxeIFtJGX0/k8Fjd7D3eqUevJQCW78hTFO2kYFaUNAfBOafw/33ZbBfnWvqN3z0rMUBbCPhMSoyS91CBni6l8AO8SFCdwDtqTwv1WvNMIXySu6Z7kAG5vm9DI7qx4rqIrB5W6SJKVUzFdDbRiSm+UqOo34k56JbMFZsuBxmLwPbJwpTy/ZdIb+3jfy8uJMBt8tlkKwc1I9g9jzQHidpoiRt9rO9X1CulwyClRMmpD5T2FiL4SNLYLeB5kyeD7tjn7q6QySgJso/N4wJqN2hsKIkQBU79toD2QfT7s9zqvuYnAc27yjsRradkklTb98wUeTLEMGQACbTyP0yVDJgLZaBdqG+BqJdJoP8bBN9WTambToZRNenmuI+jGExs1cSVNIeKeZfOZpjTXugmFzGe6GVAawrFHYbWDzr2D7JRomnK+Q5A+k09Fa8at1c0S3E+5dqL0wrA+zqxtNP1Vj1Ud471m9ZJpee3al06LlCjfUoQyXsWp+SWyMT1s8UQwXTc09zpJmeuzEkZIiJwLaJzNQ69gAaYYbaA015J4Nfl6kxpGUoFJMbR7sw0ku5i2xbFiVpgRFkp3qUlEKrByTdM56vdiGMQL4JEpQyjb7gmBDbV9Jz2M2UMJWZojk05FsBB2n/kyxTKmjkHqluoly/1JImq+U6lTeO9tBgm9RzGWgPpOOxuhrwuaY9kLoJb8QTaJsITIYwnjeXKyaXQ/xcNsKQZkp8kY2nPZQBeVbLJFY305GpDuPrtRP2rFuYCFp8DI9R1MeabCe+wpTGmq2lDzElzHqjsDsvCamdSAyTJNO0+2qPbCVhMboXD10X9zdf+7FLMXlapdZDfI4ohSsV+f1Ad2glkOiyZvmd6VilYluZEDOtsH4+15jGsX1aYnfC6mnnRcppRO6pBgEq/dxgai+yzAD5fYcvRNYp9VCBbN3THudUr7a4WY7LND6T+ybbiG/RTewIbHyuGSaG/L7HFRobAVc/tdg6aYURcGMU7WFGednSLwSs+VxjNz0Yhcs0djswzDN0L52SdudQzjNMLCbWkXijya+3+DLDF5ZhKpUfACp2RupmkGTewoJS6DaF3iSfpUJ3DldadCugTPmAqyzZXUN/WIrHsRkIRcwjaAZU3TEcTzF1L4E5234EZz43mGUTU5QtvrJj5YluhrHiIxhFMAa9E8arfzSX6pLCSN+lD/sgHe9RTY9fVOhdJ4E8m3ovK42ALeQSIKiSJ1FrQpkJM2m0hNvULWExFU9k1wsgbDv5nroqlSJYI4xiqvDQWuSrMEpZE8gECGUOq83Y/6jKAtqO4DxqUsp2huHr/khrCNsaZeXchi6+X/B/XMZaFNFPGffhIegDAZ2ZHaWyYZAQo9DFzkovLCvRK6mMke8/s53Umyk32Z+AR9I+Bv8P7rv8f7H8wgDLf+fPAVj+y38JLP+Rl6PvPQr/7H/2z2NU4FGx5mV9SKYdy65Eq8DEdlSm5/P1Kb918iV/5/KbPJku+e7sguf1MX3QGBX45P6M5a7i1x6/4nc/+wbvvnlDrveRzN9eXHHZzrjczbnZTlAqcDLdse1ynsxWnG/n3G8qThdbplnHXVMxyXqawfLGdM12yCnMwMVmTjsYFmWL1Z6X14c8Pb1HqcBB3vByfcA07yQIp8uo25x3ju8kHTYbuN9WTIqe27spv/rOK54vD5kVHeumYF62rJuCbjA8OVxxu6vY1uKt3G0LFvOaadFho6fycj2jynt2bU6R9WTGs2kKiqyn7TN264LHj5asdiVaB8pswBrH+RcnkHumxzW71zPscYMbNO+8ccv5/YL+yxnz79xRZAMX54fQaCg9+bTjvdNbXq/nbO4noAPfeHLN518+YnosZse+N0yrlrtXB5hFR/CKLB+oil6kuL1G24DvNbbs6bc585Mtm2XFdNFQ70SuOj/bSCpvZ3GDoSg7hsGgVKBdFaChWjS4n8zJf7CU160zyDy2GnCDJnQaZQPKevw6Y/p4S9cZ/MsJT37lgpvNBGM861dzDp+uuH+1kEFL6bFXmYREvN/grwtm7y5pGvkDNFxX0V8ifkO0MITl0w31smRyWNN9tmBYOBZvrFndTjGFw+0s9tbKIH7mCZWjfJ7jvreFZxP6sx4VE3r9zoINmFvpF02doOXLjPaRk3RXgEFLOmzmyT+paN4c5I+/9Zi7jJAF2UcF2dLQH4osCR2w9xb3uENf5VJQfuyxJw3uvMJXHrMRf40vPLrV2I3CdIrmTJJdWfSwzChuDP13apQKDPc52UomU/yBMKM+l5Alog/XZ9I16RYO1WrsWtMfyPEUV1b8kFMvtSWtjkEVCld6qteG3Xs9emso7jT9PDDMnAQGeUY2Kb8zDBORtLpKfu5jH6grg0yeTAUodWcO1SkmL42kkF5pvIX6nZ7s1gqbFPs1gxGw7fIQ90nSWlOdiLeM9QSpasVNPdVLI6xhDONpzh4wh330v3oorzXNqY8dqDIxMEzjRIoNEYRHr10ux5RktXYnKarKw+y5ZvdEGL76iRO590TWNQ2jly55ZF0lx2B3QIDmLJDfq3GCKHlBfQJgsc6jO/Yil14Ke/bQA2kaSS/VTljb4l5AWnMiTGQ/l8mK7iC+dy3AM9syTlI0p4HpCzVKKlNHq3KynoBExo5IFWD20rN9Uwv4jF2oqefRtMJANcfiRdSdAPM0sZH8e/g42dMo8iWxEzAmom5lorA9VGOaar6UtNvqMu6rh+lLT32m9/UmYZ9wmrYjAVFqrB4aJsIkBrOf7ACZ2KluhV1zeWTKvMiAm2PpRrQ7aE4FMOuBryWBDpWKE4OMNR6J7UshN82ZBDslZnDzlqK6ikBOxUkNH6KkUSZchlLu99mrgd2ZHftzE8PochlslreefqbG9GCZaFJMLxzbR4bU85k6TW0t51fFoJ98G9nAtaSuBvvgmnZynN1cJttAgHk3VWO4DcgkiTfyvTnU5BtZp1x6umlMfN0KIC/vHe2BGTsnIZ7HUpFvBaDaRj5X27mhWDn6qaa4dzRHhqAV04uefmownac5MpS3sg5K0U2FVSzvBIDrXt7XRHZvKLVcm0ZSXW0bsDsvPZidAMeULuszmL4WptHGyQHlAv1M2M5+Kp5Ju3MME0N7oKOM240TdSm8qJsZ8o0E/fRTi905YQ9rT3cY/3blwmbarWyvuOtoTnPy1UC3sGRrR9DQz63Ia4u9l153XkB0I1Ujygf6qaW463ClERnsuheQFmtnglGyH4VmmBrKqxYfAavd9PSLXCZG1h0MnuGgEClrOzDMC5Fpd16kq/UwVrGAgHZXZSKB3fWEyIwGBbqJDGMEyLoZ9lUoIUharJH19abBlzm66eR7N+BLO9aHJLCu6m4PavNsz+gpJaC1H/ZAt+0IRf61oJ4wrWS9xHL2g4DftJ3UnZmWEIFplo2gFCBsYzhF7K1U8fcpoCe0HcraEQiGrhNQmeSwMSn2jy0JiP68nz34HkL4R5LD/iWw/PNf/tyBpVLKAL8HvAwh/HNKqW8AfxM4AX4f+FdDCJ1SqgD+c+A3gRvgr4YQvvyTtv3mD47Cv/Rf/DO8Xd7yRX3God3x4eoJ78+umeiOF80hC9titeMny8cA9N5wUm7ZDTlWea7rKY8ma96Z3vF8e8Qsa7nYLZhmLV/cnXA4qcf30yrQDpZtm3M4qWkGOwLI3Dgq21MPGfO85XI746BoMNqz63MuVzMeH6xZNQVtn/F4seYgr/lqdcTJZMu6KzAq0HvNLO/YdDmFcSgVcF6TGceulw+NTVPw9uE95+s5zmsBiF029nYu65LMOuouo20tee4w2tP1Fq093zy94SevHqN0IMvcmHh6UDW8vj5gMa85nW05X82x2jP4OGse+0HrNqNrMqbzBqUC3mv63mCMpyo66jbHOc03zm642U1pB0PTZASvKauOk+mOu10lAUNAvctZzGuaTo4vy+SDsO8ts6pluSnJMkeRDbS9ZXdfUR00aB2odznGyH5pI0FAZd6za3IJsglKPt9MQGsJFsoyh9aeYTAMvYl+eC9hQ8Ugn9OdQZnAdNJSNxn9JkcXDmUCrjVk5UC/yskOWlxv0CaIPzVAUfXUK4lRLGcd7S5DW3n/oTfYPEp0ekMIiqwYGDojia1OkR3Ih+a06ri/mqHLmNJn4z7mjqGVsB7VaULhwCv0ZMB3ibKSpFVlZADjG0M26+R1jQzGMAG1NTAfJNW18JGFl9frwuG3IpUNhRM1wKAx5YDbZCNjr3OHb42sX8sAws56htbAoFGNJpRxBNIlEwuiGhhEVRDKOArOPGpnpWM1C9Ar1HQgdGbsbVU7I/vqxRNJQFjCQwGraAiZF+mxBj9xqNqgzxrCeZTjVk62k8fajigLVa3Cz+L53GkJkMnDCJRDLoPlkHlRRMRKF195GWTFZF/l1NiNGtI+tJpULmh2muHQYe+lLsUdDRQvMrpjAeW+EJBbXhnaIwF87nCQ67IzsepBAGOS1ZrInAtIlePRwx54BCMhQLqLaoBClBIAePGqmk6NYHeYe+xaj5LibC1VIMEKuE+BTjpKYu0ulasrXBXQrby/q0Sym2SdqUsz9ZASFMPMo1sVw1Fit2AZxvqHFCqUZKBBJ/lf7J61Elg0TMLIBNqNgAgJv5H9S9tUQeTY/Twm1MZBclJ0pJCYfi4AGwQEgagCdAzwkV8I6/8QRKQ0Xknxjfc78Zi8MMAhdTzG5NnUP5u8pXYbOw8jaAo6guoFY2CM6cZbCm8ZAWDymdpG1h0qxh7RlO4KUZEwZww6SgoRSR2Nxxp7aFPwSwKnKUE3KEaJ8Qhm+33gULbdX8PENKd7ylWMvsukwkgAU5Jdw75PMQjgBR4kqwaRHzthUrWL68drlZJjtdtL48fjNAKuTCMsqMsi+C0U2W7fffnwmg5TFf1/6fWM4TkJmAa9nyBKkwJjd2nYn6eUFuuKFDS1By3Kh3gOhWkNRo1hRCowym6lZ1PtWeL4bJk2jBJVFeKEgIkKjFZYf9ukvkqRErtCxWspDFo698XS0c1FBmr6GA4Uz43pQgzAivsWQ3lUkIoVW+97OSXBWLY7drDG8yX3TFSA6RRGJJ9rKfAoeSVToFPq7EQxBgT5LALRyOyilXTFashWAlR12nYSCmm+JmHWTryjqo/d4BHEqmF/HZTzUh8y+BgmxL5HU6Xj8Xir0TEwKF0/qXnR6NR9OfgHIF1kqMEYVJKVxpCgYEVKmwJ5JGk2gsPBMfZeDlFumkDjQxZQCwgdZachyDYSS5rSXkGAZOrFTL8DRrlsksIaI37MFPQTYsjPQ4YxbSctPyuB/Qf97ucsfzGA5Vvhyb/9Zw8sv/xX/s1/7McG/9/0WP4bwEfAIv7/3wf+gxDC31RK/SfA3wD+4/j9LoTwLaXUX4vr/dU/acO3myn//Sc/wLWG4nnBMJEP9w8P3gUDxZWhn3n00xq+mODebuC64HknfWyukpS6Cx7z4Xtb/MuJsBK5zOjrQdHdH8uHd5mSKcEfDGyujsiXimXsnnOlyFr1AM+PPKZRrDcPpJ3XiufvzMjvNXj44mxGttToXnFzdiSy0CAMw/1OWI1giP1qwnBIyl+gfTzw7O8djv1s1ydRXvqmx58f0R3JwLD+Zkt2keO3ChdlY5OXgY++uyBbyaDLIYOiwcK1htkdbN4ouV8sWHxiGWxMgRygtRAy0L+yIv+yYPNUk59nDNPA4hPN7mngvgoUdxoT4MtsRncifrXyQmNaaI8rrjeHtEd7L9X0haY+qGL8P9z/Ws/kWYafBjbtnLyVAdX6XWFCzKnHfLSgOQsUa0X9ZKB6JWzVoMB+DNmZwsTCetMwerfqx56hlzCT4cAzfRnB7aNAsYsS0VZ8P7qH+qCSEItX0C1ydm87Hv1dzfqdgmoL3Txj8Sqw/I58zpbXin5RwtnA4qOcoSqoAAJMLgLrd0UOrJzCxtqE+u2e4twKcwW0x5byWrE9CsxuFOWN9NUN04AJ0C8c02eW+rHn6CPF6n1NcaPINhbbyOx7/YYAjsRmTV/A3a9ryldWWJSwZ2ryteb+O2A6Sf8rb4T1yTYyQGmPlMxg36rI8kh63vRVYP2ewn1/Q/VhxeQ8cP1XPIcfau6/bzC9sGhBQbCGyatAdyD3cXUdqE8tm28OnPyhpjk20sV3KgP/w48Dd99XTF4rdm8a8tUeIPUzOPxUsXukaY8D82dyfu+/JwFKdhfQbi8N7Oea8iawDCVHP1a4XLP8rZ7qdU51JdJTl8PBF472QNEdZNht4O4HwnrOPlVMLj3Xv6FYfKjYvAtJczr7Sgak27clVMjHFM7F54FuoWVwnUNzBtMX4jkjaIIO5NeGg5/C6n2F3WYUNwq7MwwVHP1kP7u/6YQRGboM3SuqC2FH2kORS1c30lFXP1bMvwgEK16x5KerrmRg15xofAGHH3tW72nsTkf/mgx2Hv/ewPptS3Osmb0M3H1P8/R/H7j7Tka+jJUaSqpRiqXIJl0u/rp+pqWH8krOVXsg10vkwYr2CI5+4sfBrs9h/rzj6jdyTAoQugnszjSTS8fd9wzHP4o+xbf1CFzmzz27My3ewLliei7vVZ8qDj/zLL+hOfrUsX2kmV44Vu8a8eoVsWLBi0x181RT3AW6WnHwuWP1nsE0wiBOzuV9q9vA7pGWAXWU76og+9DN1FizIemue4ZPOZhcOYKF5XtWpNg1lDexo1HB4ecDy/csB18M3H/LcvRJz+aplWqWtePuW5bJxb7wvp+rkZkSFlSA29EnA2oIDFMZANfHEshULIWhLJYCIrK1PDv5RpiufqLZPDFMLjymjd2MV3709Q2xaxEl53j6omH7tKSfiofx8NMeXyi2jyUka/ZqoFsY2gNhymcvO+6/nROUsKsHX4qPLl+LTPn+W1YCl+aS4Hv6w4712xnewvGnPf3MYHcyIG8PNFktnY35xkWLgmb3SDO9kFoJEysdXGnop5rypo+fpRl9pchqYRyLpcMbRX0mQ5/qWt4rX8kgvp9qXKEo7kRKnLyO3sp3FWDxZUt7nJHfD6zfzplc9eguWmu8fH6DgBUVk2Oryw4VAs1JLn2Mc0M/0VSXPcPMUFx37J6UTF/s6I5yhkpja083txx+2tCc5bhctmPXLd1RSb8QVm+ohNmUJFOp/Ph/2HvzWNuy/L7rs9ba45nufN/8qqq7q7ur7VY7HmLHtuwQYkcKmYgiEgQEkFAUgSIUM+U/AhJ/AxICKVIiLP5BDJESKSDAENtEOHG70+52z901vHr1hjufcY9r4I/f2vu+amI7kt3G7vhIT3Xr3nP2sPY+56zv+k5SsWEoX+zwWcLm9ZLyosdOY69kSCjPGnTnqB5OmDyraY8K+mlksGthECcve+zEkG49pvNkVzV6XVN9/Ajdyrj3swTdB/KzHc3dKT7TpBuLcgnZssenGt17knUj3sYipZ8lZDct9b2S6btrqscL8suG7lAknOWLHe3phGRnMZuGYAx2LydZtbhFhq4tbpKS7HrsNCW9rFDO4WcFrpBsgOSmJuSRwesknMftTcXHGAJ2v8RsW3yRYmcZaEX+fE3IpG9TbxoG6WuIKa2qd/giQa9r3MEUc7MTZlAj8tc0GRlJZT06BAnJKVLxUG5rwiRHrytCmqCsw08K9LYi5JmwiUNqbNsJyJyWqF0N+3PUtiaUOco6wa9tJ+xl18vvm45YAA6tFxCZJiKz3ewE/MXni9TWEqz7EAgcFwmKYmQgx9cAKs9u60Oi5DVUtbCUcNszObCMfU8IQf4+vOZVMPlPAJK/l5WU/9SP7+JT/I4ylkqph8DPAv8Z8DPAnwQugLshBKuU+kPAXw8h/DGl1P8Wf/4lpVQCvAROwm9wgJM374c//rN/ipe7BT4oHs6XfOnsHlli2VY5h4uKMu2Zph3vXR+O7GOiPau6IDGeLLFcraekqeMTx+d87itvQBpIy54Hx0ueXe1xtLfjzmTLF771iHTS8+jkhg+u9snzniKyaNtNQTntxvfLyXzH1W7C9mrC5ECSVbX29LsMvCJbtMwmDevNhIO9HZcXC5Lckhc9bZNiEqncCF7hrBYmKoCZWMJ5waNPveTJkxPMtB/Hw3eG0Gn272xYXk+5c2fFtsnZnU+l623REZyWQJt1BrkDqzETK8dWp2PATpZbqmcz0Egh/V1ZIdJnOe5QSusBQual2L4x6FlP8m5Bd9eid4Zw2AmrVktCqTppOdjbsfzyESgo3lyxPZuhSktaWLpVLquPOwE4JML4uD1L9iKlO4l+jolFXWb4gx5q8amQeeg0utH40qNnPb4W+aibetReh9/Jl0yyNNiTHnOd4nM/VoLkl0a6BYvYOWYV4XENLwrczKGsAAKSAImH1qBaTcg9uop9egGypaJ6ZMmu5TzC6zV2nUnnoQq4PUd2ntDveWG0EvAzK9diK7UgEBmIiZfAmKl0Q9pZwOfi4c0ujVR61AIgh5XSpFJ0h47yeUK374U12Ig8srkrATTKgZ17YTadsEe+lOMxtazwyuq1oj10JJVGRbaj2/dkNyJZRQEe0p2wRS4XAD+kHCa1YDBTiS+yPZKFFqlBEXmkqRXtqSW9MeQ3avSq9ouA6uUYQOShhFtWKmnk2PJLQ7/wURI5PDdK5rLY19bKOblMtu3TuD3NWO2SX0toycCAFVeKdl+8uS4TX/LAmtkyjFUbyom/tTmWBSSfQrcvizv5tbAy/TSgnSJo8UtmK0V96pk91Wxf86RrFeWjjD2QkoYovk1bMtYF2ImM8bCwlDSvsGdxJX+omni1Y9MVwh6JhFJYP+0E6ExeCuiX+ydOkA9u2SSQoJehggQYmUGfxe2XskilfGSy9C1zNyxiyPYlBEX8s9JNWd1RlJcClssLkZvKwhrYqYS+hESuwxAKlG4iK6Nk8Sup4/kOtRbpIO2UxQOQY9BdoF8o0rWExEzOA/1EtmlaKbkXH3Ggui9AqbgOsbaCsWMRJK1yqPsYeiAHsCle1ijrzAWIFtcBV4iUM93K84UFk21kyxDZt/he2xMvpTcidRyYJh/7LIf+wgHogryuuPHsTkWOaXoZg+qOLGpKYq+8tj6K/s91HMs4RkM3pi1jAml7ew3tBKZnTiSXRs51YOIGienw3KAZ6zykxuHDzJR4YwXo9rPYe/oKozYwVsFEn20RazFKTfqKB9WnxKAY8ZFqG8bAmrQSf2q+Fm+ny4i+0yGd1mMLTdJ4upmOftpAdWzitY6BNpFhM61IMoewHJfKPiCy6rWnOknI15KE3E81xbXDliqGDwm4JgjoNK10LJpG6j4kUMijW0+3l5DufAyP0iQ7H72CjCyYaUSqqe2tRzYYeR8Ii8Yr1SMynj6/ZQx1K6+3EwGzwch9YGqHzzW6FcWEnRp058cuTpdpsrWln5n4fnQCwHcWnxups7EB0zi6/XQcr2wlHt/Bz5luJUl2VApERlP5ECWoTryQk0RqaRJFuhYvpM+k9sM0lsG/aaepMJ5GmD7TuiiN1beBOl7Gxk1TfCIyV93GFFYXRlDkYoiPLxPMupXvxTxFdRZcIOSGkBrMusHnopZRvSPEmg2z2hESI+AuNcJOgshWy0z8n9bd1oG4yGJm6a38leHz1gtAtE62CePPagjpeWW6HIyW30d56fiarmdMin01rAdumUgQADh4GdOUMdV1CPEZGEytbkN+QKSvaXLLVg77g+jDjNKO34SZ/PbHq1Dgu4Kx/E+/A4zlv/bPBmP5XwD/ITCP/38ELEMIg/P2A+BB/PkB8BQggs5VfP7lqxtUSv0l4C8BmIMDPvfe41FCeL6YYb4yY/2Jmum04fx8D3WTyoS50uz2S/QqgeOWsM4EaAVFqBL6RvO5mwkkgfRlSnmRcTaZoiaBswcpL/tDVKOxqefdr98jqICrJmwOLel5Sr5V9EVB+GiFe1ny/G6C/uaEzEBlAuYqBaswqXjA/LtTNq+Du8q5flGQtIpkl9N8T0V4XuDvtbhtgq5iEFGUZrlOk20VVz93nzKF+pG8YSdPEup7DtNqVv0eptJcXh9jGkUaPVVOpSQbI114i0BoNSHzhLMcm8ZS7UpjfYE5N3DiSDYxROE8G7/c1VVKula0x550mUhQx05hZ4b+wJNeiF/G95p02qFeTnF5IJzlXPV6ZAardEHaKlgabJGRrzTtiSXZCbjRFik832b0C0mztNOAazTlmabxqVQxNAo7DwxhO9mVwTaafC1/673G9Tn5WsvEGuisrI4nWx0nA7Kq35yKh840MrmqlxkqDZh1EgumFT5KBpsTJ0CmMiQ7AUTyBRyYvS3Jm8VSUeUFKruV76SXCa4Iwlh3UZ70IhPmKKZLphtYfdJRXJjxuutW2CJXSG2FdqCtjqEoEjwUjAC16XtJDPpRpL0amZTiTAChbhXlS0N7dAuibGHo9yIQqwXwZktwuSZdi2wwu5GV8HQN3YEa++1AgGhxIcy0dNFFOV/cN1qAULoSX5orYgp6K+mo6U7AUnEhni479xQvzPjeTzdqTFMsz2OY0kRTXArr6hOpPBj8UkMvoJ0F0t0AuOL75UyYWOWgPQyjdE53UqvRz0TumG7E12iaIfJ+kCTeBioNoMC0jH7C7EaN6ZbJTq5JSBDpLnJfL97WTC48m4+K9yxbgSb6uwAAIABJREFUi9xL94wTap8LgPKZwmyJtSkyIU9q2a+PAMy0YaxgqO8osJJ+qrxi8iLQHEtq6QBcywuZZPezKD+zAlKH0CPTQLaS1+U3cmxBC0OZrzz1sR67BHUf6JS8xk6JYybjWt2TsKYi+vx8FvtICxkP3QfSrUz2J2eSaNktDIwhOINnSt6T6RaGHtCkCrhSkV/H0KPoqatjkmUaw36G/s9kJ6CyuJC6i7SScdO97L+8cShnpBs1kdTZbCVANVvLf/upsLKDnE95oLoNkipWjk2ZYAskbdQLw5zUAk5Mr9hN1Lj4UVxFQOoY2bHyWpI707WAq8mNsKTllYQskYZbJq329BMzBhzpNo5NBJ4gdRTpWkC4BOVIz2S6le330VOYrcMoEdVWvJmDvFXAniRfeqNivUUYk02TWio9glYCdJMPp5h6I6yvNzHQpxLvabsQH9/kwlEfGFIbyDeBbqpJK8/kRcvuYUGIx14f6hF0Fkvx5NWHZvwMEhAZRrCnXKC8FvDWFIbJubCrLlWkO5EyaiPHndQC1NKdHeWb9ZEwl9oF2rmmWEK2srT7CaaLQX1ewOgQzJSvJQV1dy+lvBTvetKA8l7A1sbSHqVka0c3H1YENMoGkkHumwkIc6X0KRaXPd2eJKDaqXQNu1xeq3sfpcZBgnxSCRcCaA9SAao2YGKwTz3JUBap44gdkknl0S6ImjGCS+K1VyGQrXtcKkBYWVmkMLUVqadWkZm8BbPisZTrkq76GJAjAUEoUL0n2/WgFcnO0x7lEhrUe0Iu56yDj2mr0u0ZIuhWPuC1GpliUUFIyqpprCwGNP5D1X6msajGYo8kGVa1wvYrHWT8rBegp5SA1lRLOFGiMVUvMlUH9A5fplJR0li81gIaE42uhDGUlOZXJKO9FQnr4I9MDKruogzVE3wi56mV9Fgqhd7UI1AMWQpNJ+E+Sskasvf42QTlYwps20ngDkDb3aa9Dl2SzgsAHCSyfS9BP/Pph6SxIziNXszQNHJOkXFUSgnIs/Fc2lZA5CDd1TpKX/2HAWv8OQzPGx6/WWDP8BgY1e+GR4Dwu6h38rf78R0DlkqpPwGchxA+p5T6w79d2w0h/A3gbwDsffJO+KMf/xpX7ZSvnt+h7xLcWzvyzOK8Jskt5mHH/YMVT84POZxX3JiZ+IkST6hipcVey6OP3fDseo/2usQ9atg+gsmspflgjr5KcVNPfrfC9ob83pbdssSl4lGzc497ICCV8xJ92tIvczh14mFqjJRyH/RgFarT8LjGvZgQSo86aunXGf1RQHUG8gBXGWQBP7fQa2H8Ys+ee6Nhu0tIrxOSpaS0Nnc87PeEq4yQeagV6l5Dt8rECwfoyuCzgD10AnJ3WhjMTuESYW98EWDW00wsai00SD+TlNL+UUc30RzeX3FzPeP4eMPVN47IH22x35zjJhLgwnaCO+pRW4OeB/xHKsLLUmS9NqOfBZrHPcllip15gg6YnaGfewlSeXOLfWcm6bKxQkP3ivQHb/BfOMAuPM2x+MWypaZ+rad8mtLP5DzTtYIYxiKBHgo79fQLT//RlmA1rBNYWMzLDG2hX3jsVGHnjmSXsPtoj2o06UrqILKlgNf2xDF5auj2AuULM7JH1UMnvr9Vgi8D3ZEjuzDsHnmSjUK6DhX1HeiPLaqVaxp0oD/tcVcpeMjWivrTNdVKrqOdCqAMBpr7jmQV5YszkWZPnml2Dz3ZUpPfSKhIfXJbS9DvS8pwfqXx37/Bvjsj2Ql4qe4LGwcSuOLTQHYtssPdY0e61KOvpXpsUb1CBblf+pmAzt0jj75fY746pbhS7B47Jh8YugMJd+j3wdSyOGHqGIjRMXqp2gcde1/MaI8EAPVzj5mJrDQ/Ey9heyTScuUiKMtE8t4fxlX/mTA8/d4QLBJBay5AxGXCvvULz+SZyNWvf9CSnSUUV0pk4dGaajphVbOVnJubyoJGthYWMr9W1HfCbfpnLwChXzB6HX0m41qfChsYFtDe7cnPolRICUDsO0U/12RXEExg+0iNvr5spWI3ZaA+jYnQp+KPLC+kNsPFnsMBPFX3RMZczdXYVZjfyM/1qSQMo2HyQkUwIeec1MJgzp4Kw9buKfKlAMRszVjj4FNoD4QJ8KnGFYp+KlUU1T0Bla6M1RMHKqZjyud2dS8AOoZwBFInAHH7ULr/TCtM2vp1xeRMU58GFu8wypnbA2Fby7NAdVeCgpqpsH5BQ3VXMX0u+x0Y2mwb2DySc4EILBthBHf3FEmjsAUU14zhMLsHiZx7CeV5vIciG9ztqXG8qzs61nxEP2Nkr7t9sBNDsRS2qj4WObQkkAojnexg9kIqKyZnws6Wl576SNMtBIyu3jAj06mtSHfjN+sYelOfSLepy2VRRRj6IdREJKv5UhYG2v1EJOJWAGIwSmowtgLITSPMZrsXGaGdMKvdnibdBIqlw2Wadl+PEuyk9rT7WljzyHTVh3KNyyvP7o5m9RFDcSVJxM2hYnrm8UaYUm0F0PZTxeFXLcuPpdH3J/dbvhaJ981bJeWVsInKy6IGQLMvycHBQHktk1XTC/upfLStpAqlBQT7CCT7maHZF3AqqcPiy9NWPkdsYURqeyyLjsUyjGOrLZjGs3mYMX3Zs32QMjl3KB9uWdQoS94+zCgvpcYjrQRs9FOR3e5el86dTHsB2RtHdSelvOhj4qt8ZnZzQ3nRU5+kdHuGbOOwU/kcGTyMQavY0Sgg2pYC1NvDlOKiI905+plB7Rz1aSYSZyDdCgDrZ4biUvbbz4yA5DYQEkmbrU8zkjoC8M5LMuusBA3dfib7tYF03dPvZfg0xXSefpGQri3toQTkZKuebNXhciPy2pWnvleSL3va/ZTyRU2/n+OmCeXLin4vH69j0IqQ3CbKKp8QElk8Uz7gc0OyaQmpodvLSLcWn0j4jU81uhUQ2J1MSW9EudYdT0g2HRqwsxSX5WQ3rXgnXcBsW2EeNbgyRcd+yWAUZiMdlVgv/1yQSpBEi+S17QlaE+Yi3QlGgKfqhFlU25owLcBoQpag1xV+VgjYbWPITQjCMiZGkmUnsadJK9nXpEBvdrdz49kEta3AB8KsFL9nCCLPVQq1qSJoTgTQJgmqMFDVIlUdAGMEempaSqptmhJcZB4RKWxc9xSwmaaEXSUgE1BWPJvBOVSaCMjsJSlWOSfs56uP38hv+f95XmRffwupsL//+M4/vpOM5Y8Bf0op9ceBAvFY/pfAvlIqiazlQ+BZfP4z4BHwQZTC7iEhPr/uIwR4VNzwc9/4JNk3SqaXgILdj+/wz0oOPnnNzdcO8ftr9JOS4z94if9fj9n7M8+5+Pv3Kc8D+s9eUv3iCbuf3NG9nPDG37Vcf6qMUrQS3nIkj3aENqF9OeHo85rmT3bSJbcbIsWhuCro/twN5f+ScfmnPdnzhHQjk45uX3H45cDLn9TsfynB1IH5X7ih/Tv3qO4k3P0zz3j3nceYVpH+wA2b7YLJM0P3fVvciwmz9zXJLjA9czT7hvawpDkOnH5OVm/txEiK49cKTn61p/0r19S/dofsrR3uF8oxkv/4Cy27eylXn06481nP9VvCzCU1NH+4ons65e4vBc7/Rc/R3ytksuXFB9hPYacy7v4jx8t/taD8WsG6KLj7JceLn5hy8B40h4a9/7tEhcDlp1ORZNYz/r0/+Xf57/72n6C6o2kPZLJ8+WNgDy33/3eNLQzrNwQ0n/yq5bmesf91aA81mzctR18URmG1PODur/bs7iZxwmdIqkC3l7B4R+Lq61OpUZAVS5lUTl86dn9xhf57B6xOwTuFWlhYpex/Xe4lO9XsHgYmTxOOvmypXgjjd/M9cPqPFNp6No81k+eG6l6IVSZw+vmYaPhDO7JfXDB77tg8NMKA7gLrNzT9NOALz+FXHNtVyuUPw/Eb12x++YTTz1me/csOewh7X8ik/+8sJ6kUxZXh6CsdNx8T8LsNmnu/ZHn/zzsOfy7HZYrlJwKzJ3qcIB7/agVGcfYDJcV1IL+RyoHFkw77xZLL71WcfNFSHxl0r7j72Zr2IOXZT2rpI9wLHP8/jqATFk88xXXP0z+SkqwNi3due9iG8IvpU816kTE/C0wuPNs3PflS0y9kslueRzZpqrjzD9e8/NEFx19s2N3LyFaK5hEcfbll80Oe4klJSBQnnxd2QXeB5sgQtHjqlIP9b3Vcfibj9JdXXPzAAlfA0Zd6tA2sX0tpjuDOZ1tWH82YPbfUR4lM7Hthd2YvLPlly+YNqeU5/EpHtzB0c83Rr22x05TVGzntgeLkc3D2Y4rTz/ekG8uLHys5+XzL+Q/m3P1jT3n284+YvAzMXvS8/CHxa+690/HyR3JsCfkVI1NynibM3pfPp24uE36fBva/1bF6I2P1Juy/E7j55xuSb5XM35dJ9eJJi2lzvIGbn2owX5+ge1g8sfRTkSIu3hP/2/otz/1/ENjdSdAujJ7U2bOO9jDl4vs0xUvFvb9/xcUfPGT5loDM/EZqG+ZPapSf4HI4+FrDs58smb50NIcJ2SowfdmzeZSy/3Yj79nXJDZ1eiYdc/OnjuZAc/DVmt3Dgj6mie69a3nxowl7b/ckjRtZBl31VHf2ePR3XvL0T99l/rSln+ScfG6DLRZMX/YktePqUwWTc095YUk3PWlVjkBq+lL67myhOPjcJec/ccLhF5bsXp+TX3WUl8LW1CcymZm/I2Xmk4uMds/Ql4q99zqe/lTG6Wf9WEp/9b2KB//XmtUn5tKvl2mUFU8fATaPEsprTzvXTM8t5ZMNN5/ZF3bxpUygCYHuh+YQBJDkS48rNNk6MH1ao1zB/Ftrth9dMH1/R/8Dc7J1YP5eTTebUNx4imuZQA3Hn60c69eSMRBn790Gs+3IDwp29zPSbWD/nYb6RH6ePe+YfyChIipIima6sySbjuUnZmRbT5UYjn5tSzAaN0m4fisn33jym57145y0DpRnDfnSUB9nXP94z91fkvAS0xmmT7bYWYZygWmiqe5mlC8bgi7Yvq45+YKleLGl//590rUlXXeUFyn9PGH2zJKdbfFlyurPlbz2PynSrRVQdBnHcD/DZ4r50xaz69FVj9srSJqMybtr3CLH7ITJcdOMAjDblv6wxOWGbNlS3y0o365RAfpFhk9SksqTbq3II4HmJCdbec5/IGPxrmfxvuXlj6QUy0B50aEbR7KKpfTMKd+5ZvP4joDdy4Y0TzBVR+Hh5tMLps87spuGdm/O5HmNTw1Zosie3pA8PABi+umyJ73c4tM9smWLTw3pLga1uEBysSb/QPPsX7jD4u2dWBSWFenxDFcIwEpWHabqwXv6ownZ2Zb+cEJ6tSP1geRoGtUFgexixwc/fcjiK9eE1FCf7FO8e4k7XtAcTCluRDEw+DlLGLsbfZGi6x7T5iTbDl8muFSTX1SoXYMrDyneucAdzEnWiBS17qkeTMie3RDKPCY1KZJNC0aRPl+hujlmVaN8oNvPUb0ju6zweSLXFkSm2llMatCbGnu6iF5Sh3Ja/I5FRv+wpPxgI0xglqBWPe3DPYovf0CqlIAxpTDTDLNpcHslycaSRsbSbCR8RnU9fjFBbTtUngkjua5wR3M4u8SUJaHIUCEVhtB5/N4UvWvAe2EVtUbtasJsIoAu1pIo52mPJmSXO6kPcQ69roTRdA4mpbCc1kkliFLCaHYdzKfQ9ZL8utnGuXBATQrCzUq2P6S4ag3bmjCfELY7VJGLTFVrQtPCfEI4F1Fg+DYp7ZD6irWg9IfBXAzoCdbSffp10s9+HX3nhHB9g68bSYHtLereKeHp87FiJIQYPvTKJF6fHgPgXpxh7t358AR/eLwaLgT4px/8RtDg98bj9z2Wv8WdCGP578dU2P8R+J9fCe/5Ygjhv1ZK/TvAp0MIfzmG9/zZEMK/9Bttd/+Tp+GH/pt/hV2foVVg16VsdgU/+Ogp//jZQx4dLbmuSuo2Q2tJ7Tye78bnlUVP06Y8Pr7hfDOj7RLuHaylkkN5XqwXOK/xXpEmjlms9NhtCkzqOFxUXFzNyQrL0XzH1WaKsxptPO0mJ5t1pKmjrjLSzNIuC+4+uma5ndCsZCk/mVjcdY7a7zCJo18WmEWHt5q87OmaRPwoXliv4BU6cwSrZTGq7OnrFJ14fGcoFi3NTUF5WFNfTNCznsmslUqN1mAKy3TSYp2ma1Oc1ZLsarUkpyYet0spD2rqywmkHlNaXJWQTnuy3LI7n5LutdjrArXXScVFrwmNwcwsPjJL6joTxjUg1RaJpI7m0472ukQVjtCLpGZMCfUKeo2e9igtclpUIPQanTuRLrexfqLXqNKKv7MyYzInhfgth5L70BjMKsEdd5jM41ojSaaNAa+YHlfsLieoXktXaUxFRYOudPQDITUdCjnGWH+haiPpqr14FJVTMOvl3EsPKqBrg59b8XQWHt1o3J7FrI2kdDYaP4kftg5YWNQyFYlvGcZkQV+KrM7sxHMZ0kHqJcxdupIaDADdRNaqlHRR5YUdVV6N/lF3t0MtU/GNapFBj3UmOpCsDXbuyG4M/UxkdSENJGvpEh2SKXUfk0TbWKsRpaIuEwZ8TK3cE18rXuGmEuqUVJp+4aQqpBAvl+4UzT1H8cLQz+X8fSKsnu5FDmynMtZDQM3g5/KFpJYmG01/4MguRQbkM25luzGBz04lvTWpiXUNUhHhikBSCyvT7YuEcKjqaI+djPMkkK2kqsOVUsXR7Xu0Fba13RdmzJa3VSBDf+LgNQwa+oXUTLgMugNRBkBUCWzVOM66FxbeZ4MHLiazpkPiYmQ794S9HrxUA7MZTGRYrTCVIhMOZGth//q5eE6VVaPHzjRyHFm8li4XT2ayE+ayuBQvootJs+V5oL4bPYeJvKfTrTC02kn6qGmF1cxWRI+Z/Dfdyd8BQhp7R4O8bvCamiZ6TYfQqdWtnPl2CV0Y38kLkf3aIsp+L4XFG7yrwoqJL3L3QPy/bpjzdrKvwVuq4rh/u/cVIvvuB1nvK995sRt47FOdqDG9dPCGDj2NupO/S+eisMZDmqsko96mVppWAoCSSl473Ee6E2mpNyJ/7/Yg2zAm8HZ7wkj3M6niSCpGdnBYKNJ2YG5vu09NK1LroSbEFVFqfSILRv0iyqQj0B9YYVuocTzs0L9pbwOUhnHVTvyiLjLbxY0fKy2G8TdtTM5tb8fYdCLZ7adqZI77qfhaB4+eWDdk3CSJk1FmPrB2Q1+nj4oF5eSYbK4obzw2V6S1eAFtLmFawkjGqhLHmJ5qS9k/ihi8E72VE41Lh47RKNedKKbnlnZhxnF49RoP1SA+SlKHcRoe2oWxdzXdWOqTVNjU1o/BPyhGz2W2kjAe5YnMqoQgoYWBdZm+3Yce7j3xNadrO0pabanj55GE+XSLZOwNtaX+UFdoMIqkdlEyHOJ95Uffpk81aWXp5ilJJZJmkebG3IT4GPo8Te0kgdeGGBjkMa0HLSxzspNFgn4/x1R2BGM+N+hWthuMEolwEJZTDRLi1uJTI78LQXo5ewGabppF5tNI/UcdAdZwPUIQSWwWFSmRIQ2pkQ7MaS7g0UMoU3TVEaLcdkx+HT47jJIQIC2dmyqE277IQUbrParpxFs5+BdDIBQ5qpJo4jCR4B3ybEyBVV0vr/FRluq8yGKb9sNhOQNjWeS3Uty+v+2fHDyS0Q8arAPvbo+jaQW8Dv2Wr3o4v10G+8rjnxqL+EDou9/8eb/O43eFx/KNh+Hef/JXftu3++Rf/2v/v58bDLzy7+zjPwJ+Rin1LcRD+Tfj7/8mcBR//zPAX/vNNhRQnG9n3J+tOCwqjA4c7e3Yz2r2ZzVvLi7Y7AqUCmSJpch6rrYT3jy85ON3L8gSx8G8oupTNk8X7M9qnNe88/yYJ1eHbDcFWnseHKxYn814ebnHdjnBb1KMCVyvpqS5RanAza6kuSmYTlq6OiWdduzNar73zguODrbMJy0kgZvNhCLrmRzUqNRLT+O9LZ95/AHTsmNxd8PevCI0huam4O7Jitms4bV7V8z2K7JpR3CKct5wfLJhPm1IC4tJHemkl3Pda9mb1qADygSqXU5Ryhvx+x8/ZVflWGsoJy13TlZkhYXWwCYli1UY/Ttzjh8umR7UpKnj4HRDXvTszqYULxOpyVhqykmH+qDkow8vUJ3GLzPKaUuSWXzpWHxJEtPuPL7GTCzlt3KclTAfkzlU6lG5fAnq1JNOO6mUMAG/zAgBZns1s6OK/GsleEVykzA/2mE2Ur1h1gl+T45bD3UWmUf1iiR1JLNewm2covxCCa3Gr1PyZxnlYU31bEayTGBqmZzs5Fg6AcL5dZRezC3Kao4+m5AdNAIuQYDjgYxtdmUIhSP9IKd8qZme7kivEpnc1Yb09a30JK6lMiIkoJtY1L3W6EbBwpK/m5NdaybPBMC6qUgys0sjPksrryufJajSomuRSdtpwC+sgPk4kSrONeULCVK68w+V+Dof1Lj7LWwT/MLCfs/0iUFZkRGHNPpZ9y2YQPlSMf1AfJa6GdJZA24iPYSLdyBkAXu/Jb+U8SpfKpHmrjT5TZTC5m4MdFETh+4Vi2+CXvR0h048Qxk09xz6oBVf8FSkvFITECeRXkKN3MyLT3Du6A8de98CX0gYk88CZidAW0fJcXmm6O719Pue6XPxg/qJhP4op/ALS3/oKC5iyMklLL6p6RcBe2BF8rfRJDvF4m0t/tNSUlizDex9Q40VEb4Ise4iMP1A5IJoSdItL6IUbSfnoXsBOEGJeiBbyXPtLNDPBqAO0xeBxbuB9tCPoUyugO7YRaArk1aXBSYvBKTbKaPMdwhU6Y6ceEmXEjo09EZ2hzJ26U5Ad1JBd6/H1ET/sKI4J6Y3q3GyT5DfZZsgctCFdAkmlXggpQQefB5oTm99T+tP9fK8GD4UUhmT9sCTbgPdfmD7mhcJZydj5DOYvS8Aqp8JIHEFLN4bQqoQAG5g+7qTsVbyOx0rIkIiUs2he9CVMsb5MrD3rh/BQnMamD9zpNtAspOAG90HXCnAN1uFMcxn+sKPILK4DLSHsHvkmL5wTM8ck3MvoAOigkSkuvVJoFgG2qPA0Zd7vBHJ/eTMM1RcJHUEk3Ug2wYm54HyyjM593R7sq1+IX5i6UqV/YhXVo6vuBJ/p/K34TxyzwlQzNfi8wSRHWZbuYbF0hEMtPtEaa2nWDoI4p0tzz1pJUCTIKE+ykuKbVoFyms3gsmkCbRHin6upBcygdkHHbMXvagfzqSfcOigtGX0fe48piPW2si2xTcbpayZnLtPYz2Kk05OF/2wyt2GHA2/N52E9qQ7PwYcEaSKI9t6kla8luWVLIzm1z3KB8rLjm4WJbVTRXHVy3FlIsFuDhXpNnoCvah4tJXj9MlQowGz5/1YkWE6CRpKKlEp2FKTbh26D8y/vqI+iiFEO0lVlj5STXnZYwuFy3XcX6B8fyPhRhMJtbGFop0bkm2PcpBfd7gMirOa+jhh9VqKqS3dniFdtuRXDaaR48ovawFxCwFxeKkQqY9vU2iDVuTXrfTSzjTpusd0AvaylSTCJrX0VvYzjc/lszW/qCRQqBWPaVJJYi1KkWw7XKFJrxsBhInCTqSew6eaZBVDGCtLP0uExe4D9d2ckBqy60Z8n4kipHGxtEgi+PbYaYKdyXmjFMlNhV5VmF0bfZQCgLsD6bH0WawciR5IgP6wkITYXGSmvkwlCKhIhAF3QWSz6RCyI32SI6gsU3TVile8tSKTjf5JP8sFpBnZbsgzgtYCIkOU3Bot4UKHC8KkwB/MUb3FH8zwh/KzitUhquslLGhaCoPp3K2XMgTCYgbDv/kU9mbyr5f9YbRIXvMMipxgrbCXr4BKlWXigQRUlhF6K7UgEUgqY2S/r/RUhth9GdwrdSQ+/Mb/nPvu8Vl+Fz9+RxjL79Rj/om74TP/1V+kcwalAp01WGdou4Qy76nblNePr3nnTKj22bShd4auS8gyK+Aq79hWubyPjae+kQJ7M7NkeU/XJQSnUCYQvKIoO0mCblJM4vBOGMquylDGk+aWLLNsV6V4+bxCl/LlZBJPv8qFqQuIaTz142K7axLSiXQN6tTjrSZ0+kOrdzpzhOtc2DOvUKUjVAY1E4DrqwSVeznmRHoHiQyimVrcJvYPemQWasTHqFNHcMIOAvKzB9XGUITEj4ye8mrsGVRdZPl0gNSTXGTYAxvTRoVlCTMBrqpw6NTDyxw3HdLRZIU0qDAuc+ha33ZbdQKuggGfe3Sr8RNHep3Q7wnrpXqFjz2IutP40o3bMithBYfkVwLoVvoJlROgZPechIhUkiirbDzuNES2Us55ZLsc8vogr3FTJwAxMnfpVlbIXRl9PoUEPAiLqHATT3oj1Q9BBwjx+H0ETlbhC5HPmq3GLRx6Z8hWkko7JJQqK9c1qSQp1U8kpGqYxKfrCKwig5RUwsDpGObjylhjo2IPnZFzNtWwSi5/a08s+aURxqUTP9kQbkSQa+TzgC3D6FcU5kJhC2HudCfJtUExrogPTJxPw7jtIV0WYpdf7C3UvRrZlwEImEaYOpcLuJMgp1tWb0gjVWFgkCKb26oRaEm5egSf6pa1GLY/3IfBML4Ph6L2V9+XA7MzgCSfCluZX8fAHB9fo+PbrrllWofzTGqpeRk67ob9JpVMqO1MkmaHjsVXk1blPXp7fEOC7NAzOHQsDs9XTlgskGPJl5LIOpxLSGIyqZfrrawcB4wfEbJPxW2C6Ezkv0N4zzCOQwfh2N9nhak0LSMz9mqabboTwONi96J0dApb5iPzOryXfcrYSzj097nIQgUVz8Xcdi/akpHtk/RWSWL12SsXMzCmSvqU2MUo72Vthb0bGIahq3LYt4QSyd+TOozeVPEVC2hv90SSmNSD35Gxw1Cudxj7PocuP9MJmJZFr7hP9+H9D0mnY19eDKORtNtAfaoxjSwYDNc5FdVAAAAgAElEQVTFp1KXU9x4+pkct+kj0zswwa+MCyoyvSth6cZOysgwvtovmNTxcyeGDIXbOeXIznUL2bePY21axhL6Qb4+nN/QFxiUGplRn8r/uwyynYDVoG67ElUQmXeIHY5Dwqzyct5Je/u9IP2Q3PaVxs83ueklddblEnwy9FTaXI2hOQNjq4JsO6093VSPtS2v9jkOvYyvJqACMdhIApFcLt5J094GKBEYey9lEUHYxgHoDWylMNFDN2MQENcFVAjo1tPPJXhIPi+G8VG3nsbk9mcYrq8wkK7Q43tEBUQmHsHY8Fy5TkAM2HG5vmVkO4+dGEztxwTcYNT4fNPKAoM3cWxjkqo3wlwGo0g2nVSXRD/kt48jRKbT+lfuBUZwqLzIjH0mIT2SrmvH7wC8FwZSK3QbP3BcnKMMKavDc60f5Z1BS4AQSbzZh7+FQMikimToqwyRAVRDEiwIi6hfpWsjABz+PrB/St3+TilhMXsrIT4gIDRWgYySUmPk3JvutqPy27f96sM5gvcCKl+Rr46psENqrDbCWA7HO4b4iHQ1DEB2AJfAb6li5LuFsfzr3wHG8t/43cFY/k70WH7HHl2X8OJqj3LSMs075nnH5XZKu8uwfUKSWl6sFwwl3DdXM3TiURra+H5a3kwhKPRVir3TjvJE12mKeU9zNiVMLFig17QK/GVOmFl8pigmHdW6QC9T/MTRdobWFahYak8vMdfZpKdvEszGkB1XdE9mwjjkfiw6142m7+IKm5YkUjd3EppipbjdNwlJq3D7DtYpwSqRUuqE2d0N2/UCs0ykSgEgCYTSg1O4KoHcSQF97kAFVGMIKuB7JeXuMythRaXIVoMKkASp1TDS6+cLYQOHig0389ApaKRP0eVGCuR7Yaxsp0S62WlcgKxW4A1uLgEx3bETSeJaSwVGiGmcVsJcJOxEiuyVh2SZiKyqktL64kLT7auRGekKheoEyAFy3I180CfVkKwqk/R+HkiXZtyH6tU4YdOtyCFBAnf0zpBd6BGcuTKuONdaUj+NyCV1K2ErulMReBnskYXGgId0qUU2mgugSTYKO9PjpCS/UtSnoPYdodLorSGp4qSvk5oGO5HKivY4jOmgeqVxkwB9lE0bSXG1kzCCjGytcYVMUrOllmL6Ts45aRXdvhyTcpHF2g/kF9KvN0wyki0jc6h6mYi6OG5ZTG5VViZQCUqAWzdIFGOXI5Kw6/JbEKR7kenqVuEmAvyVlTJ5FRjZTt0rAY+D17MkptXG8J1MwFq2jtuLUtpsGY8t3Mpi+3mI95oAImWFNdvdF6mnxPzLeY9grR2OQ9iisRQ8rhCZRiYd+VLAop1AWkFoBYRJeq0Au6G+RPcxbGc5yH7jvbwvAKW4hvZAkW4C7eEwEYhAMUp76W+B1lgfEWS7yW6YiMrkzqcS9OMK2U5SiUTWlVC8EDlmvpTzS6pbkGnaECeDwsIkTaDbF9ZLW5m8SVJuBGaRLc3WIlsdJmMmzguUh2R7W/QetAAD7QIu3CbkDkybslEuudBk20BzJGORbQb2SjEkIJNIn2O7p4WZzNRYpaJbKC+EtexjCJBPZDu2jAB2CJKJ16O4kb/prYAVW9wmu44gP4IMYcoga8I4BkOqqYvBR+nOYUstktKpgMABPCZCymCLQW4JqajcxroTkW8rXCJ1GEXs6yTej0HHhOdOEmSzZWQyIwAegnwIYexo9AaKG0e7Z0b5tHLCqA2ps/kyMoARtPYTNYL9pJFxdpnsR+9EVjoAv+H6pFUMVttCeSn7M+tAN9fYTJFvQgTKgXTtR5mlT4bZPORLy/Z+SraLwMEFSBTllcWWWkKOLKRbT3tgyNYelyv6qWwr615JDVUCeEjlOkhFjQAZl0qKrU8U2VqumQoiF052HlfGZPFKPLSm82RLj880SQzOG8F2fH5SOQGpCnTrsNNYDRME+JjWS3J2TH0VpjN+x1m5ptrJQl1W23g9A8oaqfEo5L8+MfF960WKauV40p1jqH8xnR/BYVKJ3NSnGnyQc6idMIsH4qNVDpKdxU4TlPVkS3kzu8KQX9a4aTbWiRDTXFEiVxXQHMivWgncqZzITzsBbS7VqN5LvUcjgEt5Yf9UaiS8ZyIJqsRrrqxHWy+/b53IVrMkhg1JsFJIhMFV1t9WijCAQwGVuuoimES2qWSMcEHOw7moVPLjNUHLvaO6WFUSAdMQ0oMdVgiDVIcMktY0GQEnvf2wvNQBWhOyFGWHZFctgT9FJsE7A0h0ktKsnAcria3E7kzpqXS32y1NfJ4dAeqHgOOrybAwsojBWtnutwPQoWLEdZAksRtTSWiPMbegcngMgPQV1vLV/bzqoZQ/eL6rkmD/GXr8nmYsD986CX/pv/8JZqblS5v7NC7hL9//eX5p9yZ/+93PcG+xpveG//xj/wN/4Vf+Lf78m/+Yq37K81qW6r/04h5/8ZO/zD+4+ijOa/7ona/y3379R5gWHZ8+esHG5rw1f8lNP+GL1w+YpB2TpKMwlkXa8PnLB8yyju87+ICvb+5wvpvxkb0rvvDyPv/2W7/Iz777I5xOt2y7nMYmnH/zmOnjNYuy4d/9yP/J3/rgx7muJ/zo3Xd52Sx4++aYH77zhNYn/Pzbb/JnPvkFfFB8aXmfb704YTZrsE7z6GDJ17/xgM+89YTn2wXTrCPRnnuTFV+6uIdSAaPDuN2vXtwhMeJP3Csb3v/SPXSv+Imf+DV2LmPb52gVeLFZkBjHT9x9m8t2xs9/801Oj9eEoDidbnk4WfJ/fPOT/MBr7/PO8ojjyY5Z2nKQVfzCex/j9eNr3vsHj5n+gSu2VcGfevPX+IUXH+Ofu/9Nfu6Dj/NgsebLT+8BsLeoCEGxPJ/zw596m7Nqzvl6Rt8l7M1rtnVOs834yKMLWpvw4qunmHsVeW750Qfv8itnj/jE4QW/8vQRRdFzZ77l5XrO9mwGieetjz7HB8U3n5/ie80f+vg7/Mr7jzk52JBqz9OzA44Ot+yajO7tBeZ1McF//M4F37o4pm1S8qKnXhccHG9YbyaYxNEtc04eLqnajKbOONjbsd4VTIqOXZWTf37K9nXL93zqKW9fHNNclRzcX9E7w+6DOQ8/fs5P3/sqf+sf/xg/9amv8HNf/yR/5OPf4Cs3d7hczTje23L1y3do71jxZmae198844OLA0JQ5EWHc5quTjEvcnhck2aWJHE4p2mqjLDMOHrjhsuXC/7Ax5/w+bcfy2LCxPLDH3+XL5/fpdrl+HUKuYdWs7i/4dH+kmVT8uz5Icl5yuMffMa7z47ZP9hRNRn+7Rk+C8w/tmT1dG9kzQFU7pjMW7xXdG3KfFazVzY8eXosbLZXlO9lNHddXHQQ6W/yaEe7zpkc1FRn0reaHDf4DyZwvyFc5OJLrTQhC4Tc8/GPvOAbb9+D1PORRxe8894pOne3HuOrkvJpQv3QMr2zo3oxk1X43INVkHrSaU+/zMcOS7MzUToaJ+1xwef0I1ecf/NYOlUPWul/NYH0KmH+PVfcvHtAmDr0KqG4kPAje7dDX6Vkr21FAeEUr33knCdPj1kc7dhVOW6ZUZzU9O9POfnUBWffOoa5Rd2khMMOlpmk+p70qNbIostMfKV+4jCrBH+nJThFcpaJAmDRE6oE1Unipos+1PxM+k6L1zZUF1PSazOy1bYMJKc16Ren2Gmgnwe5rlHNUDxLaU4tambhRgKX+nng8NcUV58R/2u61rSHnuROhfrWlO7IUZwl9DOPm8e0zplFPynweRiZxnQTWSQN/UmPuUnIbsS/2z3oMFepMPNOUZxp2mNh9bO1oj2Icr0rPXpi+5ksJuy94zn7YSJglkWIbBmlkfNAcSUpztm1FvbtWtGchMiyRZb+bs/kvVS6WYu4KDGNDE1UAwwqiHStaE6FdTYNpGtoTrldrIo+4KQS6XG3J8BWx4WA5tSTX2qyjXyvDSxwtwj0+569rxk2r8ek4fuOZKtHNjhdK+bve1ZvavqZJ9mJqiGLvmC0VDBNnim2j+X35Zl0lvYLaF5rKZ7kUT0g/xbvOa6+18hi0EL8qoPsOqlg+9izeFvTz0R63O0Jy+cjE5juZJFiYF3RsSd1cVuTMywolRdhDLOq7soYH/9qYHc3skKRmc02sjCRrWUBoT2AyQsZf5fJAofyjAB96EW9/j7P9D3zCuspiwDNiac801L1MlGxpgO2DyRdOK081R2DLeK9ur1doOn24gLgtTDdyg6yc1n8yzaBzWO5D22puPxhy51fFH9fcySLGMWNZ3dXuhtlUUYWLdIIxNNKPocGWXBSSYLy3ruStpsv5RrOnjv6UkcfqEhfdS8y5fxapNM+gd19LeOlGDtSTSesmY6As59IUm67J3Utw+LF5pFm8Z6XHs+4wGcLLTVE17KYoYLIy3UXaPc13VyxeM9KN2bl2d5PmFw6upkmjwC/OZAgK9PJwkpahVe8qLJ408005ZUVIFsKuygqE7lmrhBA3M00+cpjGgHpwFg5Mj4iYz0AaJdrfCb1Kf3MkOwcpvc0RxnFhQRH+UzALEZhKoudphH4O/ABN0lir6YkwhICybrBLorI4jp8Kp7RkGp0bQUIDvPuCHp1bWPFyi2j6jMjbKkH5X2U3sbkVe/lddUtc9cfTUkvtqAV9mCC2bYoF3DzXNQLq0YAdmrQq2rszlR9BJpaC8AcvJFFBlqjqka6OyNIDXkmQHmzkyTaridUDSoxhOH1bSsgWhtCVTHUlQRrUdkrqbDf7sP8NtB5+7wPA19fVb8ZPPh1H79rGMv/+DvAWP6bvzsYy9/TwLK4/yg8+Kt/FdPC5LnIN/qpfGll0WuTbcRXoXuRthRXYTTZ9xPp2rKlsArlhWzXpzB94alOhl4m+b2L0fe2FMlbuhYJnhQQy0pseenZ3ZOJXXEVJZzpID2ToAOCfEkU17LKvL0vXxTaQnUqX7KDr2ZYBR/kSUkjXzbFlR+DBwBZOZ4rZs8869c15ZnUF0yfSRm57sJYVq9clOtEqY9IimJ4xDZQnQhrmq98lI/Jl1u3kC/F3X1hYiRmXiYEh1911Md6lAMGLZ6cbB24+ema+S+WY5JhWgXqYy0ddEOxeZRYZetAfSIpp8rJuWbrQLFyVMeGpB4mAjIeJjJPSSvj2k8Vkwt57iD1yjYyeUm3geZQ9gvy5Tv7QFZ9h2tkOsg3jm6qJV7eCfuSNFLG3S0Uk4vYJReZlHTr2D5IRonasNIuX+oKF6WRs5eOdm7kfpsK+5RWckzpEPoQy85NB/lKvogHxq2fxXqA2NNWH8k91O4rips4Ocml0NxlMkHu5vJcbeW4BlmfyAMV2dZjmsD1JxNmz/0YXtEuNNlGaDifSLT/5MLRD9KuOObdTFOfKObvO7SD7V1DceNlnOswdiuGRJgskNcpL8e6e6A5+lJPc2DiPS5fqpOXPdXdVN4Hc5FRDXLdfqZYPOlp9xP6UknVQJC+OZfLBMe0IvvqJ3qsIKmPNXvv9qBgdycRFmfj47go8rVMNvqJGhm1zSPN3rtOEmoPDGnt6Us99lXmm0BSySRxciF+IW8Uk0tLUIp2T74oh/vIp3I/uEyR1tJNt7snkf0+kf47qXsQ5l75QLeQioL6OMGlMLkU/1U/k5X3bGWxU0N9ZCiv3Mh4DJ16SS0T4Dpek7QSNmgIIhnYpnxp6adGKhaue6rTjOLG4jKpuOnLuL3mVbmZIr/uaY5TtA3Uh5rFkx4XGaZm35DWcg/kSyf1GZ2wmcV5TfVgcvt5ftnRHGfjRK+4jNfqXkpQcq8mtRxzUg8eRGGGfKZIto5uPyG/sdiJsCHSf6nwmUK3YSy3ly5NNQaPtPtS5dDNZKxDokh2DhUCdiIAoJ+JsiHbSH2D7iU4JUTJ3hB8kq5tHBsdv5Nil9/O0cWwlmxlo5rB0e1n5NctrkjG8+n2EmGtKkmk7WdGOiS1GmWlKsi+slWHnaUoG7DTRFJ3lUgIvdGxRkJTXIlPcGBumtNcvt+c3JfZSsJPfGawpRk/35KdxWw7uqNC5IepIr9sI3Mk1zJdd7giGf1++XVLcyqmzaGKws5TTCXeLztJRvbMFUbkkoVGd16SUXORDQ4yRhCgoLvoW4sVE4P3T/fiayPR1HdKJu+tCUWCm6bYMiGpLTpKLAngJgmmkgoKN00xrUNXHd2ppKdmVzV2ltEvUnTnoyohoFuHqTqau1OKlzuaezOyZYuyEu6ie5H9hdSMiaVummLWnXjmYgDMAC7sNMU0Dt30koC6bglFgp1l0rvoAqru8ZNUAEFrRzasPyhJdr3ILaNfTnVW2C6lxLdXtYQyw00ydCf3pXIBO8tIlvKB7BY5ydWOkCaE/BVPoPUiM57n8loPupF+RHc0l2CbSYaOaapq14zVFsEoASWdxU8ydCWvC1oJO5cYAQhlil5V+L0JetviJ7mAqLitECsqxrqN3n5YBjrUaQy/955QCrPK0Bup4rGsNoTFbPQq+vlUfI3JbVeyVIYYAVsD26fl9cEMNTZB/paYyNJpCcFxXgDYtoIh+GZIMx2OeZhv2yh/7XvxLradsJDWorQWgPbKQyklnZOJkefFMJ1g7RjCo/JcOieVkoqQgT0Vn5ckyGYpgxdyDNwZmMdvZwyHAJ7IGN7WjcSkLefiuWlC296yrrEfE61GMIlzUZr7T8Abg/R3eM6rj1+n4/K7Qgr7+8Dyd+dj+vF74fg/+KucPFzyycMz3l4ds5c3PLk+wDnN3f0NT947QU8sRwdbLp4ecPhgSd1mvH50TesSnpwf8sadK97+0gPCzPKR186p+pSz8z1QUE5blILdqkCZgFIBdZ6jH1QioTUeaw3BK3ydUB7UNGdTmPUkueVkf8vLy/+XvTfplSxL0IS+M93Jpje6P5/CIyMycq7KKqqobolBSAgh1iwRvWBo6A29AiRaQC6QaKkFYtFs+AEs2NI7hJAQEhJdSKWurKzKIWYPH56/yaY7nYnFd861F1lZRSs7S5kBZZLL3d8zu8O518zOd75phfOTLa7+9Az62R4hCNhtCVF6SBUhP6uAd1vYbUFGpSZjYN4Y4F2uzCgd4J2E9xLRJ0mMpj/Tj9Ra6dKhLB32LxaQJyPEiwpu5dGc7zEOZpIMX/7ZOZ585w2utrPJQxqcBEaJxcMddp8vIUcB9ZT7Xs563NzNmGy7L1B9XMJ+swNeVvAzD6iI2XmLfWJnzGKEdxJhbyA7ieqthPs+GUF7WaN42MJ+MZtCTmLlITcaoQ6YPdij/WIOLFgHEosAuWAarf/pHO7JAPWmRHzaUZJsIoqbQ28iVISsHMKogEECOqI86hE+nMOeOehbPS0yFBuB4RsdglUwl+yRtOcOokwrc61Gca3gywh/zD5R2UqElYNca+DBgDCkVee3Br6OCGWAXiuYrUD3HmtphAPcIgJnA+Jtgeq1Sp5E8Jw/UXBzMJCl8dA3Gm7l6QWVEbHxMFcGwUTIgamj3aMAs5UYzj3qFwrdEw85CISzETGQxcpeOLcIEKsR5uMKiALDQ3ZSqlbCrTwgI+Y/NRhOKQ/un1iYaw3XRMSlRfVpCQSgf+SYsiqTh3DuYe4UFp8C668D4XxE9bMSw0lAfcnFCbuMk4R1XHH7UTP4p7hRqK4ENt8bUX9SYDxmGJAIwP59i/KlIRu1pzQ2+6N0KzCceKhOonktsH3PQ+8pwx6OI+ypYziSitCthLA8jtVPJDYfMPF28THZiOGUSa56L9A9swxZ2iUZ4x0n8N3DCD/3WPxMo31Mdql5w8WK4YQM1uwl+y2FYw1Nf85rBXByXl+mjsU+pa2mhNTtex5HP5JwM8Fu1DeU6/Zn2b9E+bCrmerpKy62jEdMsI0qws05Rs1LgfYRmUi9B7J5u39A32xOPu3P2JNpdkDUOeRFYPdOgNkKFGuey+qnwO13GD40HHN/1TUXqnwK2smBKaqj7647E7CrCLNOnX8D5bTN64jN+0BOgDZbgf48YPkhw1qqm4jdO5Q02yXHoT/jYk/9lsBUePZhFsnHa7ZA94B/646/m38GbL8GVG95DtVbyq3LW3pFg8oLh+xVLO8idu8ILD4hU1NfBXTnksfzNDEyaUFRd1zs6h4I1G+TXzjJVl3FxUXdRtg5e15FBGYv4pS+OhxLgqyWzFh9GafFre1z/p++Q2D1cUD7QE6yfN3Rd2c6LjyZ1Em5eX5YGDA7+iddzUXA5pL3aAZwxYbM0HCSFrV2ebEzTJJWX/IcchJufRXQH0sMJwJ6x0XB5tLh+juGwUsd35fjkmMnIu/3/SOygfMvRty9X8AuxLR/OYKLMHM5LVLYmcDicwa47C7Y+5gTanWHabHM1SBwvWVoUPsgecp18gD2h0XZvFB4v9MyeyiLbYAaI9pzLlb6UkyLkLoPGJYKumPdy7jUB7m+i2jPFJorj/aci2jsm5TT51NeAOECABcvxgUBeXXj0R8rFPuA4s6hP2Viambc7FxyIbP18LWE2XjsnhYo1x5679GfGdiGi2lyzIwp38eIXOSMArBLDdVzgSZKYPaix3jMfsfuvEB9OaI/SxU2d6x3KTYeZmu50Fpy0SovMOjOw5eKcvOSixR50Ua3DNgZTgzqtwyn8aVEsR4xropJ/jocay6OCIHydsRwUqB602H3vMH88w7DaQmztqyEWRjovUt9uRzb4m6cFhPGI4PijscqR4J31qkE6PUAX5uJQYxaQo4ew0mJ8nZkGI8QU82M8BGqI2jKCbBcuNCTJ1OOfgJAsncE5pueoLkjUI2lgRgsK1l6e/BiGgXRWVaNlFyRjFpC7HvEWUUQXRYExaU5LBTk542OXZfbjpUlw8i/E5CdpLWDPXgsEwsZC0MgHQJiWRBIZ7AW4yE59j4Izr/P9SJaI44jROqfnDyWmW3se4imJkDNIDTLW1N/JZAktcn3+SXckeWy92Sv8RcASiHF4efZ0/lLPH4jgOW7T+Oj//I/+pVv99N/5z/9tZ8bgF9LKuyv7OGsgigDbtYz/PDtIxjJipA/ePIZpIyotEWxGoAIXF0tJlAJAD95+RCv7paQMuDNdo7y8R7wAlJElIohM3FQ0CrgfLGDSmCjrC38nP/2nsNX1yOUCoCI6PcFmkc7CBnhncLoNMrKYteXwMUAYzy8VYAXiE7C9wr21MH2mpLBKjDgxgmE5x28k7CdwbAvEKJg/cbOoKgsZs0ApT3iwKqQ4BWG3gASkDLALTx9kAJwg8L2zRztUEA8GPDyeoVxMFAqQKr8QSMwjpp1GE3g8a9L3G4aKBUwdgboFYYzj+DJGhanPfSdRrsrIfYKogiwvUbYE6iZrUD3yMMOGnbQEMcjYqScL2gATrCGI1V17N82KdAnAjJCNA5hZzAOmuAiLW6FUUEEAXPcw5epRiOF0ggVAUd/XrEaMGzKKRDGV4HVCQuP7plF6DSECgg6wp54XpdOQb2kTDJosq+iVYAO9FAWnj5TGaGvDeBYbSEc6D392h5uzv2F5x0rIJ7sWcky8wyf+eYeIaXhjkeR5+CB8qjnZGiUCGWAaiXUHb8swtkICGA447b7C4eoIvrzANkzFCiOCmKnISLglgGqF9BbibgpoLrkLd0qxMbDHRFgwgns3/HQe4H+IRNlfU2wglQHIzxgbhXsioXkfsZjh+REGyIidqwlCVVILD5L4aMiqKyfUusXygjUHuOZx3jE+y4qAIFyRjvjRNDNyErlaxcKHpPeMQwoqphkhYLhVxVBdA63AZB6ZlNC4xmlg3Jgx+ZwEqc6lGAiRC/pvy0jhsd2CkaBpL83p2YGA6y/wSTOqCIEDlUZvoqwM3B1f+S5Dmc+FauzpmM8Ase4TKFEKqdgMrwnB+1kb924iph/TmWAr+iZZUVEqnlI3//t43gIwpEEdK4GhCXIGZPdXKaJ6HCcEkUXBDOsGEggakeQobpDl+14TAlbUJQMj0e4V0OSpJ0loLfZ28w/rAlhrywraXi85TXldLrH5L/MacY55EQNBB3tI4Jq6eiTzF5Gs8/nlDzTc97jUTO5NvtfxyNuL8v27PxQUxEMU02DAdoL+uX67GPFIfQnCoKyYkPG2c0EghGwM5EYUExqilwRFJM/M/s27fKQeipikliWQHGX7p8U0rR5zu8WNdCf7Cpe++xnDErAlWK6j4TDlDYbCsp/bZMkpwLoT8UUZhU0jx9IoUilmCpB1EBlQGYreSHzOBDs9cdqUuLk5NYM5gCCd9URTG+fFocwquz91Ac/LZUIXKzYXxh2n9bcRhS8p0wbMCw5ZnLktkXABNYYhoNpNpO9tNJR7ukqvscyGM+f6QzH4d9qOPS+dicKQQHlrUN/opOXVnxJITQs6JeXlqCQKgSOVVBku6MU02t9SYAJpGug6J+MUiS1kJje90EDw7FmquuM9RAiAG6mUnJ0XmQT8CkAKCgx+ZrJACcwna4plEjvXXpJg04gyfI10sZJbuorBV/I9F5MwFExGCh7Pn0hoVoHX8kpmCcHUGXWuj9nqqqrydzr7udYuCQDNbuAkJhwiJTCqoBcdwJQjRZKyonl4NL7ViIaifG4TJ/LEkFLuEWJWMiJHYxKwNcaxdZOoUB5//crQ2LaN0Jk6iqoiJiClUoFX2uCvRARmoKsrpaIBdPfQ2UOgA8ElRNLWRYIlZ4YbZQFWfayIAgWAkhBRTwWzd/n45NkPGNdTkA2pU5Oz+FBp+cpeWADhSAgLQzZVSUnJjf/fmJWpfhygFCMZFGtRRzuMYXpOaIwB/by/vUVAjCGYFKmZFkhDqAyJ70qdTgHIQEhIZT6c3+mn//cfr6qj3vBvL+yP78pj680Y1k+fxaf/P2/AwAIXrDrcK2B84GhOlcNiksN925PqdkuzWhSkoKomKg6u9ij+2SB0IQUFiKgWjGlfAKcMEYFYGUhbgxj+TcS9pjJneWNwHCSJIezZKpfMyimP2dAypTgIAiQQtp2/ULBV2QAdu9bVK8MhlOPxelvDW8AACAASURBVM8Udl8jw1JeKYwnKQrf8kt2/jmw/iAimoj6pUL7zEG1EuW1pEdnFVBdSfQPwpRkaTZkBGLjodZ8M0srYI88yrc8Dl/SsxRMmqytAvROwh6RobIrD70l6zb7XGL32z3KjyuMRwQCahDwRUSYebLF/2uFm+/RK+XOLcxrdm7lbr7MrDD5lSCruOMHXg4jsXOOmW4FRIrLd/OA5qWETSmUdnUY9/5BgN4L1G8E9o8ZtKB6Ml+qJavilh7mNkm+LFBdCbRPYpqARhQbif6hQ3GtoHpeR4iI6pJytu7Co7hjcI9dRrhZgOp43CZNrs2WoTDtc4v6MzMxb75i2En2JZU3BLF2kXoY9xLxWzuoH86nPjeIOO1LtwTRDI1g6IpIdQo5vdFsgd27ATgfUP64Rvd8RPNhMcl+JwAT+dqchhg0wUyx5j7688BE2QcOix8b9KcMtsmTYLMRKQmRlRXNS0nPGggy1MhqhvYxg3iiJoPVnYkpPAfA1KdY3vC6d4/IOLkm+dp2ZPz6U/rkQgEwfZMTyO5MoFgDwykOnrGR29FbweTWlA46LhmuVGw5GZeWk3w7yxJDepjGY9atyJHeKt0SUJY3rGAo7zjWQQGb71rMPjSHROMUgORJFCefHY/Bl7w+IqYqh2XA7HN6/kKRehzTo1iT2VIjx7i5jAnMYApVEiGBhFpAdWThdIsJZJd3BAXjgjUquefQl7wP7vcx1m/IroaCTGv7UKC6IoM6rggAm0tKiMcVx8ZXOCTpJha0uqM0tj/j2JodjyEHx+iOY2xSDQcnpcDsNdm6DKrsgh2KrjkEa6mekm0GKPG88rXI9gWRrKI+ScDVmJg4JZKHi4Axp9DaGV9frg8ppq5CAgWUKXsj4BrK4W3z5aTR4ZhMZpasM5AIKf0TqG7C1EHpmsOY2xmZtt0zgaOfprAyxWNmwjDHKBQ8tt1jOdkQ1BhR7AOGBWstWEtCGX37QKHY8HvLJRCFCFRrnsdwRMa2PyEwKO9iCrdBklPzs7B56+ErgWFxkOWbVGkyJDBZrpNPb5YSQyOl7uUmwFWUzweD6R6Okqmp3SlZ2Sqxpjl9VFrKcwEyqdUtZfjlnUd3qmG6gP5IJtn7IXXUVWKS+AedElY9pmPKQNSXBOa6J7DK/q37UuthSWbRNgLjXGDxBdnGcu1hdh52odAdK1R3ZJh0y5qPcS5R3h1AlExdib6QX0qDDZoewOFYo7yjjL66dmRWE7hylUSx9fw8S3Jh1bNKw5cSxcZiXBroPWX8vpIwWwfpI8alRrFxcFX6jkuyZdUH7B8b1G8dzM5hOC4YlAOCV5Uk19JHyoBjZKhOBFRPRtTNCBLU4CFchPCBUm4fuLClBKTlcboUxCMHT7AqCX4ZCuTgFsUk0c7+QtU7+FpD2AA1sGYkM4e+Mdy2FFDtCN8UUJ2F7Czcqpo+S9RwYLWiEhADvZJZkuorDTV4DKcVzNYCIULtB4KfIjGA95Jc5egmBhRSwjdm8i3CR4b7SAk/K6C2/QG0xYhQGshtS0ltZSAGxxTX1EUZjZ48j5w4aIhtS7DmA8FkAofwgTLVPHdPEuAvpcBmvyRwOI57/kn2WHpKao2ZwF6W3AKg7DZGSlljZEDPvUdsO4iqInvo/cR4HqpDsofMI8Z46MG8f2zAP10a7M+xqV95Key7T+PFf/GrZyw/+3d/MxjLrzSwXHzzIj7/B38bAsDZfI/9WKBQHvvR4G49w4PTDV5/fgLIiKPzHfZtiWfnt5AiYjuUuNk0qCoL5xS66xrHjzbY7GoUhcPQGZS1RVVYLKsBt20N6xS6TQUhI0xtWU+yqbA42WN7NUOxGKG1R/tmBnU0whiPuhzhg0TXG7irGuJoxGzeY3fXIEagmo/or2tUpx0K47Dd1DClw6wecPtyheK4h1IB3ku4USFGMp3VYph+7r1ACBJ+bygD7TSWD3bYvF5Azi2ik2RBAZTViHZdwzQjTld77PqSYTCDgZR8jt0VTIHVEdVph2FfoF70aC9ngIooXxmEb+wRXtYI5yNiq3H+7Ba36xnC6wr6SYvxtprqPKIC4smIorIYbmpARAgroU972DVllqgYIgMBVmycsLIETgCVh1AR+mVJmeOevYzYGH5RiQi5sAhWQez5JYq5QxwkUITEuqWk2JGTwVAFVK81hvcGRCdSki4QawbS6I2Cn7EWxJeAP7GU+d5ouMcDQ1Iqj9iTyRR7DbMVGM886s81fBUxPrIoXhuMpwTibslaEmHp3fQV5ZJRAn5O6WuYO5QvDYIG6kuB7QdMiYs6orrUGE78xM6arUD/fIS6TenBTqYvQC4KCC9Q3kjYWaQk+jON/kEgOzwwmdYtabgt32jYZQLdxx7FlcJ4yn2Vlwq+IYAMhgsWkGTr9JZy1N3zgFBwAcQuAqq3ZH7sKiKCx59rVIQV8CsHc6NRvxLYfMfC3OppUcLOIvzKofnEYDgNMGvJTsuS8mU7j/BVnJi0oDl5rt9IdA/DxLwBiWndEsA2rwU233IQg0TzipLb4Th9oTqCZHrKUgVLvjZVhJsHVJeK12zMq/0E4H4e0LwgywEQUHYPAhdHZPYip9CPlKJr5xwP10TUl2KqvshF8W5O6elwHFCsRbqWAASw/RoXcMwuMy58rt7zPKVNXacjptTSKDKDQ2Cu+gRUS55HZphz3YpdRBR3lOdOf6/FlFY6HnH7eQyqq5TomtjpYiMOBff9AUCLAOQqiu4BJa++wrQgoFOvpvSJJZQMackVC3bObWZwmutkzJaMrGkxgcz2gmPL8J2Y2F6CXN0yXVf1SL2GyY99F2GXqS6oTL2aR8mzH/k6NzvcW/RcprTY7BksxVTl0ryO00KPLw51CK5JCb/HBHfd2UG66ksCel/z/ab3capsmTzyIS+myInd9VV6XcV95eoVep0TU5lYToALFjkQR7qI/pjvCaaLYpKGuoZMsByQ+h8j2of0qufAn+FEoFjHyX/dnUs0bxj40p3IafEqy3DVwO3X1wydsTOCd91Tepu9/dIeVCqqP7DZiJju31zlw4qN5OE/lVNtS7E9pPQGzfsmn6cvxMR4+yJdw3S9hCdbysoOAvzsI67uDsAWArB18hUi13xgqjLx5uB7z52e+T7I9SWZNSTwTdUlisnB3YkmsJ7Tt8yQnZAWZwnmpY+YpLDJ12u26TWeclfXqHSdFZo3FsMxg2fqtxZ2qaB6VoawKktBDSH1R3LbcgzwtUpy6UMFCmug6Pk2e1adBCNQXY0YjospadbOVKo7idCdh11omK3DcGxQ3Yywcw3d0i/vZgzTISOvpnoQAClFl88NRh7qSUzygLcWQcupw1J40jnDWYHyil5XqnBC+hwPE0gLpT6AFy0nFlymsJ6oUp9moaDXlMLeH4MpIXaSwTKxVXY2VZE4xNR9mdnO7ItFCAS091hLSLC+JNd43OuDFKNlkA5AVtF5xOQXFclbGusSou0Tw1lMstg/9/hSGmv6d5ao5n0mYBlHgroJhOb01pDGKKXIxvvbydvIvtBf9Mjg8S97/H8JWP7nfwXA8t/7a2D5z/y4+O5J/Lf/x38VV8McNkp8uD7Deb3H1pY4r3f4k8sLnM33eNhsIRHx0foUSgb0VkPJCCUDYhRYlAOsV7hpaxjl0Q0FjPYotEc3GrS7ErpwOFm2uNk0OFm2GKzGZlujrCy8l9Da48Fyh88vTzCb9RhGjbEzU/9lUVnM6gEhSKzXDR6cbbAfCjjHDs4QBJzjh0KwEogCqvAIQUDKCO8kytpi7DVmix7b6xkAQJgAkcx0SgeEKFBVFu2mQr3oMQxkaSfQuC8gk6xXyMiuTCehKwchA+y2xPHDDba7Gm6gHklVDr7VKJcDAMA5BT8oNMse7dsZnrx7hS8+PUVxNBD8BgFdOgSvUDcDht7AtgYigdtoJeYnLXZv5pBz6vXDzkAvR7hNATm39EhGEBQq1p3IkwHhugSWFkKmL4IuAbydhpg5IAoCSgCi9qxk0RFQEbAC5qSH6w0X9XRAvCsQq8DfpwkFBglz3MNuSui5hdsbqI2CbwLOnt3h+pNjbjMAIibm0ARAp0mQiIAX7AANQKwCkzo7fmj6YweRqkeiSccWADGyvuXo/Rvc3c0Qo4C4LhAa+ljlThOUzegvDAVBWmi4SiwHOZ1DLCLEIIAHA+K6gBgFzJM9xjcNhKcs0dUHkJRluVFFqI5SaL1RBBkPLVTt4HsN0SnKlzvWpYQkCxe9hNpLuKWH3imERz09tjMLXJXQO05oxxP+Xg6U2/qFh77VCAUZXF8x2VI4sHLE0pMZSlbGhIKT7FwLc/jiB8YnFvrKHCaGjn+7RYBM46U7epLsitJrvUvSzJjkdEmKqfcC/VlALMO0UJAlnFFhSvvUrUieJt6PxZ2cah7ISrK/0824iJClgFluOrH0FVlVnyoScp9priGxi7QgYAXKm/Q5oSKk54Q/FOneDRzXYMh6y5G/i5r3d9RkkaNO419F+loTq+Or1P0oE2u/TkqM+zU8uSPUxFTdkTyfYwI9XkyT/Kz4IJspJoly1PcAbZZdC8DOyEaPS4JN16Tr0RFUqoES5uKWjGpOGLUzHmtmhTOgz0D94JFLMst0HkyMzcwgn8teQv7bF+nYRwLm3FF7P2UUgh5TN+MYAphqTdyMLDgrWJDSQxOATqA+L0iMxzwW3abKE3NPXrxjT6gcDmMiPMco3zM8UR6PT7JtvTs8N59f9nyOK3pxfYVJvit88gIPODDhQ0z34aG30uzT56/Cl4DfeMRJ++SR9JgSV4djgs/Mwup9RPeACxy6JfAqthHSpqqRlDSL5NHO4XW5viUHkflSTOdlawG7ZEiedIeqkHys03s8ya7zokuUBIJZImpnnNgre0j1LTYM/DPtAezunkqU1/Hws6QiCVqg2IeD7DIe9pMTWIEEcD0BuStFCjY7bGPq67yvLgk8l3HBhNXcwelNqvsZw9T1mRc7XEUwev9v04b0nhdTT2z2hE71KOm504JDrulQ96W7fI3qGTzF8KgDCwyk653CtISPGFcaZuenDk5fSqjOTzUpuetSuDh1dOZqmFAcWFs1eLKpAQSYWsAbOZ238JTTCsdaEl8zIEv4ANeQSfUNwaSIgMqprZ7hRaHUBJI6AdisLA1xCmIS9gDURJa5pq5J4djRCSHIaqbfi84e6kmyBFaJKTBJ3O+rzEFBmZkUgr7K/MjjbPQhiEeRbRTWUf4qxSEwKMbkI82M4gFYT9sDfo7pDNM2p2vrD9uL1gIyeSn7/h47mcBn9lr+ZXjjFwHeX/CIMTIs6Jd8/DWw/Kt/fKV7LG83M/yjj76L9qqB2mhIB9zuOOH96JyR9y/1Cp8u6VWD5IRROEwx9r6KuMFhsmStQHHHL+92GVHeCSw6frnfzeYQOuJaz1DeCFQSiLpG2fKL541aoQAwqhp6ALRg9LybAcJX2FzMYTYCpQb2f1zRk6EBMQLFCBh5mPDYRVrpT6usxQYYjipUAeiXFZaXlLz5WiSPDmVlOfF14YHxqISJfG0w/DKqE+PgyxRskMrFfVVA9cDMAu3FCcwgMNsgpQ8WjPYuCx5XFdFcC/QPCixfCbzZPMTJh8BwbKDMYaIGANvvKBz/oeGXQBfRn6fI/UWBozW9JGrkJBBCQ/U0+ZvUu1ddRbiGHqPxuEH9OqJ7UGH+ghK92auI7sxw9brQlJXeplX7yqC444p7fgQznyYK7eOA4x8L+EIxOn+XV70j2kdzLF5z20XM5ySxvjvF6U/SZF2QAaDET3FiuOa2fYkp3j0oOU1qEYH+rIDZYAIxriHjkidgd7tTLF8djll3lK3lLr5gNKrriHElofcRdq4pu0zeI07Q2d23f1QDApi/iHhb1jj5YfJu9UAujUYEhiN+KaiRASDdQ43mNScS3ZsCdl4kOV+c1NxmB7QXBq5mwEp1G7G/0AxEuavTvagxexlSPyFg5wyrqK8od2wfGqw+DFN6bU7BzenEh95Fng9DIsQ0tjk0JGhg2xZYfRQSC5EYgUJgOFKYvQrYPWNnoBqA/pShFvMvAvoTAr76ism/ZKoC5CgRpcTRh/zC648EynVEfyyge7Is9XVIHjYG7JS3vEbVHSV0TMUF7Fxy0uyAYcVrW2wjzD5g95Sz8/kLj/5EIgp2JkYVkcvU23OJYDTcDFh96A/F8WlSmj1jOcikeyChujiB1CwDtTOB+sahO2Zaa3+c/FUpIdc2cpq8tilFd/OuxuyNR1BggvEusrvUcIKpxgBX8h6tr0JiZxI7mvyEiEBzRZkfZcECxSZgOGKKrukYeNKeKcwuLbZPNeZfUILpCzExM+Wa8slyEzDOBJorj2El0Z9IzL/w6I8lFp9bjCuNYkN5YZ50B8O0XfYsUqbZH0ksPx0xrhjQYhuJcntI1u2Pubii7IF90gPfF7bhhDYzXfGGTKs3As0lJ3jdmZ7SvZu3frondRfRnSnUVz7dGx67xxqzVx66ZbotP3Ppo6tuHLpzzQl2+n7SLbD4fISv5CS3dTUn1Wbn0Z6T6cqVDzl5ub4c4SuF7kyhuvUYF3wvFNuQwrEUlI0TCJ+9HlM6sYGr6U9u3lgELeAaxZTYO0dv4aDgC4H5ixF2Sd+ZbgNM69CvC5R3Fr5UGFYK1bWFGg18IbD6eIBrFHwpUb/pp9TYoAXsQjMMxwgUtyPcTCMqATtXaF72iIZhLGrwsHMN1yiUt/bQ05j6EMeVofxyDHAzDV8xCTeHzaiBwChKYPmRxfZ5heomJQsXZOmUlWheHSa10lWoL0em1SrOMeyM18nsmBjMVNmD/9BsRoRSYzgxqK5G2IVBcdNjOK9QXg9AiLBHJYRjnUZ51SMUagohUvsBblmh2Gim7c40zM4ilAp2RhaQoEhBb4e0rxLVVQ9fa+jtCLsqobc87vG0htmMGE64CkMpOTssi5sewShuazcy/dUFuFWFqCRUZ+Fr+grVboBb1Sk9NyUjF7lPU0JveoKsACDOUFzuEeYF5G6EX5Z8/bKivLazEMc11G6EbAdEoxEahuK4ZQV918MvSuh1B1Ubpti2A+Ksgqw0ZGfJ9PnAoJuUBCvnNWQ7AFJA2gpyP0D1yeMYItQdaziilBDDCJXAldIKYt+RcUsgTxrNKo57oE9k2epoCSx9AKxjoq1nx2ZMr5ukr1phSpTtBnoWnWdgjpQHMHcvPRae6cNQ6gD0CsP9hsDnWkf2MIO5gRUq0JpeSSn4fHVvH8AB/GX/Zg7lyT7I+wCyLIDRTn2VwmhKX4WYmFWEkJjNvwBU/nxQz/9feiuj+H9/zlf08ZVmLKuvP44P/97fhao8fvvZC3y+OcZJ3eLF3QrDYLCcd2j7AsOmxOOnN3j5yRnOnt6hGw2acoTzEpttg6K06F/NEGcezaqDlBF9V8D1GqYZEYOkBDUC0QvItwXCmUUcJYrVwETYKOA3BWYP9thfkk1szvfwXmJYV1ic77B9uYBaWUgZ2KHXMChFvi2AiwHBJYZq5iFGSTar9BCSrCeigDQe8aZE82yLcdCUwG4NRO0QB4ViNcC+bhAXDvWHBfoPBsTE+qnKoaos9neUwtp9llEAYk9QllktBAH9sIUbNUzJcKE4KohOYvZCoXtA36g9dTC3GuaDDbpXc7JWp5ZM0iBRv9SoLyNuft9BzSz8poCcWZiPaoxHgSvhRw5irxGLgPnFDsOfrWDPHOROQdok10QKjzm3MJcG9tEI/ZbHT2YgAIFMnHx3j/G6gt4q+DogFhHFW0o7i1t+KPoCaF4JbL/uIIJAcc2fjyce4mhEaDVEr6A6TgrHBw5iTKzcipLWqCO9p8ces481fE1mqbhlmfj+uUdxLSkxDIA9dShfaRR3IgWIAMNxwOJjehJdfWBLgk6A8phsqtpLFOsk0bSUcJbXArv3HarXGnae/L0pvMbccJIjnMB4wsRccyshLUNY5CCgO4H+wkH2Ekc/EmgfMZhk+y6lrG5GSZcagPJWoL2gxNHVMW2H43n004C7bzCxcfVT4O5bEYtPyJ6OR/QcZw+yWZPdCgW9lc0r4Pa3uAjkS3b1RQnsnzvMPuN5lTdM2cxeQLMHtu/7lCoL7J/QZzz/hH4+uwiTHBSBrKKbRyw+YuiOCMDqJ0D3kKyk2UiCzfPAQKgtfYO6pVTv9lsC41HA0Y8kds+B5mXy0Clg/5Sy48VHwP4Jx0/1EfsnZOh0n1ifjkxQfUX2pz9h+ubmfWD5Ia+7nfFn0nNbCGQay1t2EjaveY8MxwIheTSDBsZjem+P/yzg7usEUkwxxeQNlY4LHaEA2gtBf2cAqz+Sr2n/lP7k+jJi/UG6lt8G5p8yiVa3yau5oKSU7BP/rfdkj+xcoD9hCqsI9AEScEZsnwmoxNDplt7X2WdZnsrxMTugu4g4+WHE7hnri+ZfxOlzavuOQHmXWMyWi0vlmq/fP+JC0/Y5a5baCybhBn2Qq2YGs7rmsaqBwT2rj8KUtrl7pGD2HKfyhkwGw6WAYk+Zpdlzku/qxG4meSaAyW8pPK+ZHrgY0x+LyYt493XFxbFHHO/dUybu0v8rsPyEIDlLmc3+EC4TFT2EUQK7x2pajJQjfa2+oKey2DHpVg2YFpCYqsp/z94EDCs5eTW7UwlXE/QWW4Ja0wZ0J6xZqt9GmC6ivHO4/aCAbuPkteyPyDaalozj5h2N5WduqqnpjykftTUnU/OXDsMRK6F0HzDOJGavLVytsHui6BtPMtdim5QuSjDEJ6W91lcO7QM99WVGzfuq2KeqsBEot2FanAW4cGC6iPLGIhQS/YmCGiJclQB5G1BsHLZPS1S3BPk5QCdXlfQnCrPXFvsLg/kXI/pTWhemReB9mEKRghEob5ksy6qtAF9I6NZDDQHjSkOOEXYu0bwe0T0soNsAOQYEI1HeDtg+b+jr3Fp0D8vJl19fjbAzpr+6RqVFppBCbATM3qE7M5AemH28gz2uoHrHdNi1xXBWIEqB+lWP7qJCsXYwW9bGAIDqLMajMkmjA3ylYBcawQjMvugn/6ROoHY41qjfjKxYqTVU5+Aag+zvtAvaHXTnJyBbfbFD984C1dse46qYqlv6hw3K6x6+0nAz1trovUOoFNTe0t+4Zfqr3o1wq3JKetVvtwjLmoA3yVPl6GGXBVRLsA8lpk5JaVk1Ax/gTmdkNEcHPysO4DAklk9KiMHCz0vo2xZ+VRPEuoBQaMhdh7CsIduRVSkpFEh4T89mRSCGwiDXhGC0iFVJ8JurTxIABcCfKwmx6xCbij7LSWqrD17Ntj94MTsufsTSQPQj4BziYka2ct8dwCnAY8k1KuIe2LGW3khjEIeRgFcIxC6xE5oMaRwGpsI6d5CqJk9lHNOxJgnsfQ/nzyfAfin19S96ZND5z4BbflMYy0d/7+/+yrf76d/+T37t5wYA6gc/+MGv+xh+6cff/4f/4AdP/83vUgIKiW+dXOKyW+Cd1S2EBjpr8G998If42XCG9baBqh1GqzF+PkcnFYbBILQa7zy6wYOHa1zfLvD7736GVdXj9y4+x2A0xqAQfjqHdwpRA//8B5/gSlVQxuN7z19iM1YYB4NvP3mDHQzauxrP332L9aahfFFFuCF1wy0HPD5Z4+blCnKQqM9bbtcphCiwPGnRBxZjVq81ive2sD3ltAICDx+usd9XmD/cYXfToFkMGPYF9K1B83gPu65g5haoAppFj34ZEXuF5rSF3ZZQpYcxHuOmRIgCYmug9uyoYwmzxAgFKMCcdbCdAQYF3xqY+QhdecQyYqgFmmc7uHnAk8e32L6ZYxwN5CAgn7WIW9J55k5huHDw3+7gvYS4KSEHgdNna2x8ieJOovrmBuO+4D6PBvS3NcLKwcxGeKfw4JtX2HUV4AWqd3YIUkCdUVIcDHDxjbfY3s2gW640+6XH00e3GP9kxWCl0wEhCpRvNM5/9xJrZVA+bjFKCfX1PcJ1Cf2wwziL8IsAfacRIIEk25UjgaPeaIQmUMI3kKkobhXsUYCASKv3lIYeffsWa5QQg4B4twVuC4QiAnMPN2e33HAcYY89ZbJWobwTcAsg3AuFEs9boA4ITkL1EnYZYS8siiuF+L0dhvMAeZdK5ANTYZmoCzSvJPrHAaoTqK4kxgsHv/LASLBsTzz+jX/l/8bPPnpCgPx7a8QXNbqLABgm5EYJ4MEApwXKawk3p4R09oVE+40B5kZDejJ343FEmAeIUWE8T4mI81Q3ooDhkUPzzhZ9X0B66tBCGbmS3VKKmSV+4zHZSLtKPsOzLDFmbYsIAm6Z/KlGwJ5Q1mp2Au1zBwSBYiMxPBshW67o2qOA9n2H6qWGGjnZ7h9Qbmv2BNFRA9IRIJffv4O9qbH+RoTZEpAjCvRPPISXyUdHxYPuBboHQHUr0J9F2CVB5fidDqNU8N/eI2xLKAfsn2WWmwmhTCkVCKXA/mv02bqGybQiMkW1e8gaEF8L7J8R5LMSAEBEkoQK3P6Bhd4oqgWSZHFc0bfia/q4uoe8JrMvgPU3I6KU0B3QPxRpcUJg/4T+TbMn+BOeYHU4jxhOmCY7ftBDbAx8zUl9DqNxMzFdu/GIxxUNgZZvWIVi5xGhFChvKVPsz5NvUfJzqHvsoTp+3uYE1+6BwLgQ030QDVi1kAATAMSCiwntBce4XPN5GYzrFrArLkzsnwGQBJ7tIyBKiXElcPU3PerXMsmd6SkVgQsBoRLYP+HP+lOmtEIK2JXA7hlg9pRhjkuCREQxMYn9mWQAzolA90Ci2AG+TvJiQ59jPjc7FxiXEr7JAXKC41dy+1EzbbR7ICegaHbszbWNwHAksfsaz0k69vXun6WwKA+0j8Wh31MD41KiP2Vth5tx3ILhvmwjk/+Xk/FhJaAsF3/sgmnMdkY5a/tYYjjmtspNnBJDXS2wfY+LFuUuQjlgXDLhMxiB9lwx1ElxjFwtoZKij4oOejwZxsOFtWIX0J0piq187QAAIABJREFU7N7hIolKHcLFLmD/SLHX1wMCAv0JP6/tgvUm3amEjBKuTjU/C0nwn5jJYGRKuBXs1kzH68s0Zkbg8g8UZq9iqsPgz8YlwTXrLWJiQSN8LTGsFKQHhqVC90BiXKX0SwgIUKpq55qKjQD0p5qujNMCug/YPDcwHe95X3KBxtX8rlZjRH+qIQPIIJcphGhBZld3AaHSaB8auLlhkFJJQC1d8lNaXq/xuJiULO2TGuXGYTgpEEqFkJhx6cEk1MQaCwj0pwbljYOvFdrHJUwbMK648NudG4SSY19ddvAzg+G0YNjUWUXJcq2hXISbG4SKYHk8LaHGCF8RNCN1uApwIcwtiulY7NIk73FEWFRUJUsGCLllAUgBvR3hFgXlriZ1UqZuylBqCCnhliX/X2no7QC17RFLpqyHQpE5dpS0uuMa+rZDmJeAkpCDQ5iV9FMaRdAo2QMKSanq9ptHMH2Sw7qAqBTiok7soOC+ujExlBqQ6Q7xAXFRE8AuG8ieCa9iR4ZVdAP9lP0I0fWIsxqoiukYRGJgURZkQQX4YaMUYjdQquoDGVRrAWshypJPKwsCwmEAnINISa/sx5SU80J8KW1WRF6vaB1kUTAkKKXCCiXTn7SNyEU6acwBdP5Fj5/vA/0lHh/hR69+8IMf/A+/9AZ+BY//6r/7b3+w+Jf/5q98u+v/+X/5tZ8b8BVnLBffvIjf/4d/C1fbGf6ldz7E9TDDabnH//H5e5hVZCQB4PbtAn/jWx/hH3/8HB88voQNCu/Mb/En1xfoRoPd1QyQEe89v8THP3qE4qLFcFuhPO4hBPCvfe3P8KDY4n/66HfR7iv4rcHjd6/w6ifnECcjfuvZS/yTP/oavvc7n+BtO8PrT07x7W+9wJ/+5AkQBRYXW3gvYUfNYJzC4/hkh11bwRiPVdPhbl8jRoHSOPSjwb/+3p/iH/1vv4/ya1uUxuHubgZsDFbP1tj9+BjH373C1c9OUT3ZoSosNtsGeFUSPOgAOAnRSchBwi8dmrMW3794if/zx+9jdtTBOYm6tPja8TU+vj3Fo+UGb3Zz7LsSUkaMn86hn+2xnPW4ulqgng/oXs8hW7Jx46lnsM+rBvJswOPTNXqnsf6jM9gnI6KVOLlYY9+VMP94gd37Dr/7nY/xar/E649PoY9GFKVF+OMV+mcjHj+5wdUfPsTjv/ESn/7pBeLcQxqPsDco3irEr7dwNxWZu1uB8bdauFaj+qzAcBpQPt7Djhp+UFClRzPryeb+0Qr9U4tvf/AF/uxHz+iFDAIwAfrKoPnmHbZ3DeqflBhXEfFZh8Wsx+bjI0RDn6IaBdyjAWVj4T6aY/mda9y8WWJ+2mK/rlE0I8a3DaIgGDj6ISdT+9/tgLclnn3vNT799Jy9pW9KhCqgfKswvDcAa8MJ12/fYfN6AciI4lIjvNdBaQ/7YoZ4OiL2CvULg2IN7N4J8CcO1WcF+gsHFAGiU1g9XWP70yP6EEPythQBcmkhX1Tw84CoWZ0BAEd/KnD7LwwQKuLof69w83tukmcU1wr2yYj5Pymx/a2BlTAamL2Q2H2T6bLdYzKh8w8V2icB4lGP4o8bdE899EaieSmw+cAj6ojmczLi7WN6UWMRoDZkpO05g5FEp6agHRyPKH9WYTymD0g6AV8HlFesu8kBRdFEqJ1CqAP0RkLv6b0rv3+L7kdpLDyrSY5+LHDzLw6IvcLqhwahYJCNvbAoXxiMK060zEZO/ZnSAu0ThkhUb9TEIA0PyEaPRwGqpyfUrQLqF4pg7p0O4U2F8z8Etu+QtXazALOTiTkiIJ1/GnHzW9zX6Z9EvP0dymxddfBsFWsmAXO1X6B7b4DYGKiB4TOhoKR9+WECYRcRdsGeU9WJKQV48nlpej7rNxL9KX2W1aWk17Flsu76Wx71KwW7ZEqzGgTKK94b0mLqOfRFSgbeEjDWl6wGMRsxSb19QUtBZqAQuXhgj5jKXKaEX9fQ8zv7IqI/ExhOI2Riy3NQye5ZOpeQA49SUJAiM17c0T/KoBgeV64c8QXQP2AY0vIjStZ3T8mAhlQLcvRTj827Cu3FQUYck4WiWFOWPJwIStlBQD1/GdCdSfjk+ZQu4va7AfNP1eSTVB39V+U6oj+S2D+NmH+OKSV3XPEeEIFy+vqG23Q1j33xGXsTRYjQfZKAvw3YP5aTb3XzLYfjP0rhKh07DnO4UPPG4fYbZI9yHydTR3lNq1syygUbgSafX3lLxjkUPIfqjmygiLQMZC9t+1CgeUPG0pUC19+POPmhmAKQsn93WBGYzL8IaM8kRIyor9nNSTluRHdOQEa/sEB1k7oi52JaaACYTCwCmWA7JzjPMuPhKHXGDhHz1w77hxrlJmBYykkCnetnlp879McKrkoBP0XyU84JFLOk3XQR2ycEZEcfjrj9RkEmPL2u2EcMC0qlEYHqhvJmnSpcXMl022ElUd16hELAlXLyiEpL6S+fH+BKierWYX9hyNoKyqhziFNOv70vz84hQO2ZwtFHPdoHBHemDVPdiXARuydMYs1hTP2xgmk5zqoPGFdqen6x9QRojUIUAv0Jw3+iFihuRmzeq7H4bMD+SYny1kF4gsHm8y22X1+yUqYnO2sbgeatm+5ZOYbEdA5oH6XXR4Jm3TJ5V/VkYaVNfyeJdHlLGbHqPTs3Gy5WVG97+EbDG4moJfTOwjcadqZQv+4JQiOg9w5RCtozhoDiag8/J7D0lYYvJS7/uQIX/9cA6QLMVYtQG/iaPh9z08KeNECMTK4tFNR+nDySUYgvh/cUOrGOJdNmEzso1y39kUjeVcN6DrhA6XFOXA1hCu0JixrCeoR5Rb9mqaBudnDnS17Xq83UdZm9kGHZQG5aTP2W1lFyu2h4LGPyQaYkWtENk+8xDgPEkp1acZsiyzN7mABidI6sZWYyY4RQEtG6L/s187/z/zNzei8t9n6K7JcSa5O38y8MAPqnePxGMJbP/4oYy//gN4Ox/EoDy/k3LuLv/Pd/C7uhQIwCLkjUhcW2raBUgHOUsEoZURiH3a5CGBR7CJ3E/KgjqARg5iPsmt2Fi6cbbN7MGRpQO4RWM6G0DIBNHwaNQ7wtgKVDdGQzRMkQGdSeiyqd4mt6CbWy8G3qqpSRXYsbBo3EmQcGBrdAAPBgn96xgzABcX+wwgorEOvAn0dAbAwneVVADhSAZFehtGQ+Qn1vFcgEiL3iNsaUmjaKKRiEoSSCMlydkg08E1WhALWXlJc2HrCS6bFFgOy5Cpa7x+LDAfG2gNnQkxMKMlcicmILTylmBKYuQah796KgdHM88VB7xdTTgavqahAYzx3ZxSL5TlKXYVRANJyghiIilOnY7j3y5JY9e5FdkSmcI9YeslUIJcdH7zlpcDMemxoE3Iz1KwwVYYpp1GTXAEyphQR4ACLHNSfSqj4F0/gkbe3TZbNkdYIC/ILpeNUbjeHUE1CdBMxeSHbVmTgFdshBpAAYINe3iEggo7cyBZqIFMzDe2xcZf8eIHyqlukpGZUpsZF1GQLjMsLsyciKwH3o7lDFI+0hWEGOiVV0BIMAQVT9hoX107VSBE6qE9O9kbeV+x7VkIBQAkb3xy+Ye/K+WTovyWMTIYEURylwMOk4fQJiiudm52ml1Iqp6iR3T9LXTPAyLhIjIg8AJNeZsFqDk+YcHJMDcHLiqmvSvZGtK/dCX3xJMBJSD97Uv5lAoE6ppYzpp681g1ufqlOC5u8zyAz6UGui+tRfmfaZjz0zPDk5Nr93WVtyeK2rMQG1UKT01mMeY74OZnd4fvYe531l2WUOUIkSU6F70MlHHgjQROT1kx5ThUm+36ZOxTQOEAyyycE12Tc99RQqsKJD877IYyUcX5N/HyV7E82OAMHsKBcGkNIvU4dgI6YakykcKtBPmYNC7CKF3vAtz3RQQ1CC9NnEf9w7B8vjjnkSlcY613NkWTzvX46LnRE8Zc9svt9Y08Bt2IbnkmXR7Kg8BBfl+0B3vMZqTPuJmGpBckgPeyAPn5spKy5JO2NKTb0XlJMkt3w/H+7lfL6+oow5+4S5DUxVMNP32PR5fQBmWVYpAn2vrhSHQBudP9fFwY+YmU95GPecKCx9ToI9BAIJTzmgK+n75XmKdA9zm0z3PRxbzH1+EUCMye/OahDbMMQnJ8NKGzEsFUwXJg9oBpTSs49SukiJuiJA8UXyfxoxhQvpjt/3wTB4J59frvOYJJghLVDl74UUcuNLqqhyvQtlxIcE3Px5robACpIUrONqAbNjxUvUYhob3TOIR/jIcQIBbCiSZFOJNDcg2yccQ3Xk4OErfTjONKZB3QOSluE+unUIpWI4EHietHuQOWZt2KGeJD/y9WGFSU5gIxAXqe/xvieUSbJh6paUvU3nw9TWHKATJTs14SMVIoWGsJ6MdTse2DVgSn+N97opp+AeAFMViCSwFDb5E2M8hOvcD9DJf5IfE0ry7wzavOeYJiAXpYAYkp9SSob/5P1n+e3PY4FcfZK2N9WQpI7P6NyX/Z5AktQmT2UGkeHe8QP4pepG8rYBbv+vgeUvfPw1sPwVPMrnz+Lz/+bfx7AtkdM8xZhKzpcOeq3hlh5qOcJv0wdKqpLISX92ztV4d2ahrjlbjZLgRJ/1iJ81UL04pDI6ASwt1KsSuhcYl2FKqYz60PuoeoK24u6wGu9WHmovoVp+IMYUfOpWHvVLJk8Kf5gMs9tSTN6z7iGljXmiqsYEeuqUQllFFLcHX45dEizqjhN5X3CyzKoGgSi5HcaDHyZedskPeb3lcbJQ+vCa8YyeODtngie9hpTbySFJZQTTHseTgMVPFXsWPVkE1QkMZx7ljZrAimrFFElvl+m4VNpemjj2ZweAYzYEQXrHiZXZpclpBlv+MOn1ZZqMp8lxTAB2XEU0LymhDalkPIO18YhgqL049Cf6EhhOAuafkUmQyfOUJ7Wu5vHZGRmh5UcS/ckBgGRQMBylBMviMOHKBefFFujPON4Mvki1EMVhkmYXBA45ZXCqgugPwGQ4wpQoGVRE/VZg/W2H5nMNs8UEJAiO6TPLSZi6YypidcNr058eUj7z78d0DG7G61VeM4mxP2M4lms4llExwIogNfncKjIC41KgexDJtC0FVAf4GmBsPr18usXkKYoSk5ctXyc7J7CjHEwczk0ReCCFBvkq+Z/WSAEbfJ7ZHoCabuM0CWcZPfdTX/L90J8JmE1McjSOtdlzMmgXSLUXTFc0Wx5PdZ0YH8PxdI2Ygq3YnRkniWdmXHzF0Kooed/4kqmG44oT8fptnO5lOfIYmYwqUN1we8MpUF6n98A+T2R4H+kEGLPEU/f0S5rkH9R9nIKpmrce7UMFvecYuFpMdQp5Qp8noeORQHEXp3oHX6QJa0fQUW44mfAmVWrcMDwo969mtkv6QygTQJCUAaPueA+pkedQbLgv15DZGo4kqlRhYdoAV6dJkMCU/ptfW9+w2qK8Y3ehGshO6SGmQKE4+SKnHkgwWCpoTKDGl5g6Q/OihdkHSB/RHSu4mZiCuXTHYxKeHrzqLmCcSxTbgN1jhflLTghziFKUBFvlxmNYKbiK+7ezzNqx3zEX3QMERwz5Eii2hz7HYclra1omTw4ryfTW1A8qHb9HeA0ImIIC6hs/AZBhyQl8fU2WKEs8M8OWQXK5YehNPrfccVlscjUEj3VYsuOy2LLaAjFOqbVqOCS+HsAB7wnXyJQiGyYQmQNnhMsrOOmz1WPanxo4HmRzxdQrKUdWbfiKk3A1BgyrJBdNgE/3DLcq1wRR0gb4msynGtJEOUT4OjHLe4KxKJH6IDGloEZFya/uGP6lxsDeyTVzF+xcJ5ALmA0/7FgB4icw5moF3XuC0QRW7Eyj2DK4SLjMYAaMRwZm5+CNhO48xpWB2djJe2g2I4bTaqr1yD2Y1eWAqCXHNUSoLgEwQ3ZQpDTTKAT0boRd0sTMdNUMbCnfVq07JLvWGmpvyfK1Fm5RMggogUw5OLglA4ZyHYevDWTv4JcF9HqAnxWQLdk64QJE7xBrQzA6sG4jgz7hAuAjYm0gOn7R+bTPLHFVg+e+AMrI9z3CrGJ4UKkhBkc2LwMzcwCZHHx3AHs5zVWl595n6e5Vg0wVIcnfKHKq632vY9pO1IrgMAXkIEQyhPmRAnsQAo8BOPg587/LgttLwToxB/P8ReE94j67mJlFxddkdjJGxNH+OYbxz/VY5p7Ne4+/FH/8JX7Lr3zdyPNn8dF/9lcALP/D//jXfm7AVxxY1l9/HJ/+138H75zd4sPPH0CoAGU8lIrwXkCpiOFNg1gEzM5a7G9qBrzMHIITKBuL/q6CrBzEZYlwNkLcFgRKG/qupE2Mmo7QG3YR6rWCO3L8wMo1EpEsX471x8pCvSzhjjyES4xVy8lC7hjMq6x6L+BmBEvjqWeFBQC9lvBNSuE8HaA/qWCPKPkDADUKDCmYRfYC4cxCXRbwFWVwwQA+MZnCC4QyoLhS9O/lugZBIBslWFvhAb1j8EuOZ2e5OMFxqAOqVxp2EaAG1jW0zxzUXiFqShNlLxDqiCgiUJEhVXsJP+ex5/26OYFd95AMoK8zA8brq3oBX/DLNf9O71ISY8Vx1q1ItQacnIbEHPYPPHQruSgwZ0BLZjs47pT9jUd8vrQETP0ZJ+1qIAA3OzFF3vuKPY8hsR/jKkkbgWlFOFc0MHk0TqvEXCUXqQaCskHdCuRIf1/HSXJnF5GLBnkc0vmx3oLgy5eYJk253y0fZ0hVDrkbMIPN8oagNIPsmA49poAeTtyRSt0Jcl2qIhhXMQEnMXXB+ZLbmL2kPNDOeW55jM0OaB+nRZAeX+qQy8AmA6NgMAUFqZHAVaZjVCNgG0rq7ILHPjGDqVJCDTzW+jKie0gQoHpg6v6LBE/9iZjSkNVI8JoBaGZN8kJL85rPzwBKJkDCWPp0DzeHsUMCL6pLP0/7YZpt9v0dqgNc6taz8wNQ5iQTUxhIVAfAmVfnoxITG5pZrdy/Z5cCekc2JwNSO0+LDrlnseJ9l1OGM6CwM4YWiRCnxSGX+gWDAso1FwOE4z1YXROUAIfXuFpMzJAaIkwLtA+46JBBstkdALxuM2N3GJPyLsInFhWCgN/OxVQfksOG2DGZGB57+F7I4+YLMYFkyMP7IINtV7I6YlymBQwNVOuDTDI/gibzJTxBiG0okcwP3ROs2Hka1zb1JqaFKtMF2FqmonmCyWEpJ8at2BPAThUemhJH28gv12JIoLrzGOdctDQdx9+ljk+ZPGbSEYyXmzCxa2qM0/2Vqy4yqDL7cGD8BOsgguZih9kHJuum/efwmem4EqAJhmMtApOdeU8cFgLus7xRAOXaw87Zlfj/sPcmsZaleX7Q7xvOeOc3xouIFxmZWVmZ5ayuqi4b2nK3ACPMFskLIyFv8A4bDEgGGyRgA2KNkNgAArFASEhI7IzaCIGxjWj3VNVdQ+cc03vx5jud6ZtY/L5z7suq6q5sV5boavlIoYh47w7nnHvuvd/v/5uCioygDTGxNQas3a/PkDtW1GsCStW4IUyG31G7ihGeWzlcl/wcZXVG/7nZd2TqyjFZPEo/k42LkvG4UBZi990Rz3F/HGaimCyrxcAsDiA3kQM49AkfQ9UEIjwGg2Y/ZXWGFFAN2S5KK3leZMs6jD6Ju2cU2SnJ61RvDGs6bIBsHbp5ysoO0zOuEtLS2y8CQTIZUBlfN1Z2DOBYIrJxDnacQG8MmVkhENKeEQsDoITAEKijKr4Rg5LRs6oYCFTqWLcBnhfLgYDsHLq9HOltC1dqiM6z3iPTkF1kM/u+ydrCF5qgUUp4Ha/jxsLnmkxo66LnD7CTDDLKSSFE9EfGQQR4rffnVZgYrpMlZAxbQ/9jIiE7MnJBSsjODnUiwjgEpQAlIBpD8Bk7M4e0V+sG8CmcZ3Jw9CZS7urYR6nV53/vd0nCwjp2UDZMxxVNh5CnlLLe67X8sa3fh89VhNwDkcYMwPDHqj7u3SeEsAORw4fiPdzQA87YVxl+UldmCLvk2Psdlz9p+wIBPr/4jOVpOPkPfg7A8t/4p8DyZ96yp4/Dg//w36bUMRCI2Cnlf/2iue9a83lAciMZ+qB2iw9dceFECaAY7qPrXXk2gNgNt2MgezatD8nwKRePyTrKhiyGibNqWTgeBCWErBPZPV9fESGjPK2ffN+Xf/mE7IodkQHqF9U23/2+lza6lIvw3WKhZx2Z1lg9ILMmO4KKXqJ3vwNN1RjSBttFQH4Te9U6Jn2md0ybzG4ptetlcQAXOM1BGKpdikuGc/TnsvexyG7H+slut8DuJXq9ZIkJl7vUyH5RGkRktuLCuT8XuHdJ95K/vpKlXZAR6UvGfZQy9kAaMgKVePy91BHg/7vJjnlSNeVRNr8njQoYzlO/wDDj2NVWRylVvltI9fULZkwmSW8Dlu+G6I/jax707ryla/qRyMaG6HPrWd2dbI+SxoBuHhNO6930X7VhYHb7x+6DV3rWrmfxVMProZuKIWm09231TBuvzTDI6oTH4PMyE14D6TpAtWF4HGl4fj7/nHExuwkD89P3CIpA5qYPKunlij2oUx39Wz1wM2NWgdhMDFLP3i9HMLdjkfqkUBWHoASzPL7eYyUcBrD1o1LDfnMpkFR8HWxB0KI6grd2xs+THkC6NILVvu4n2zFRPTPsEvq2guSxBUkf2cAWOnAx5XgNFDes17Cxb1B1iBUiDHvpmTjdBnSjHjiEYRDA8vnYYycx+LV6+Z204R5Lz+spCLIlZrSrw1HtTtam2jCwgSEGjfQVHcP7NLLnyuzklEHw+YLiYEc30evWhSHZ0yVxfy2frwdU6doPjJvNeR32nxd90f0gixykmNzhZOMGFq4H6z1TaQqJ/M7F88t96SWi0oSBHUUIw3H2IKsHN/q+vLIHZoLH0Y3I3iVbhoT1skjKAH2U1XoCsigHpLRXxO5H+s96KaXPBGSU4XJwshsmUvIaJZOxWsVrPl83Vkg3rAygZFlAb/0Azm1JhlQ4svey4zGohpLFnTSU+8QeTAIlEcEF1RKs3+jrLXqmUFj65ZKthct48lXjGFYDDK8tgQmBqDTxOaQY2MT++u39gT2QHMCw3oU/uYxMorQeLlNRYuoHECRdGKSdsvOQrYUbJTAj1qKoltaFwVsXWVndOA6DJEOBpPMQse6kB3fS7ACdMJ7sWUPZp7BhAFZeywGM+VQN51l2BKRBip3P0u4Y1KB2AMyN0x1Qi+eoB5794/VSUGE8QqYgawNfMDm0/7zrAVlQURJaW0BT4igbgq++WoRyU/4bWu5sE63lficKsrEDeCNQswSuWhIYAj8OnuLt6UEE01HzhJJTpeJziSgr5eveS1IhBHyR7LyPJnZstnZIWkUICGUGUbVkCe+xgaLpeF61AozdgcCmQxgVZB7jdSDcPVbzHggMiYbo2cPh956saP9vYAdE+xTZ+9JX7xG8J1uo9U6WqvWOzewrSXovY1zzD0E5PTDstx7YSRHv5/G5upF+cw4DoxlZyeHnwE66e/93P/oc/fP86M/+qO2fAss/dPuTAix/oXssk9SSCZs6TA432NyVKCYNKjVCcqdgT1qImxRu5HF4eotrs48wp55QZxaJ9mifjck0Zh76VsMedXh0covz6xmSDwq49zYMs7kugNQDgp5GM3dMYNQBOGzhOwXRKNgDh/yzFPUjC6gAOaIXU+23yL5Ton6/hlABbptArdSwwPETpmvkz1MmVEqWwoeRo0dz3qF7nTMl8k0DUSlgbBEMQ3oggWQpUR9YyFqhPQoYPVMMSwHgSwfoAJNbpN8tsX1iIScGvtaQlYKMDF3vMUzvJJo3WpTTBt1dge4RgFZCbRSyG4H1LzcYfTfH6qsO+WuF7ddb6LMUqhaoHzrIlhURx7/BDr71W/xwL14oVO830C8yLlJ6P14XZWVv1hDPC9iFpddxZiAqjSACFt9VuP2mQ36u4d7dQv7BCNsnHrqSaACCnDQMDLGqBewkJqUCqB4FZNcKLn4plmcBN/+MgbpNkF8LtPNA5nBiIe+YBFe+ZK/h9rFnfUQtcPsti/xVgvXbFslKwWsyxDICE58SEK6/YlG+oMTZtQLrN+mzLM8IJOyIA4r8guCnLQI2pwHjZxLbU4/yTKDZE6je7jD+gxTtIqDdstKiPg6QjkEpestBRe8tDKnH6BOF9ZsB5SuB6iGvp9ELMl3VCRfL+ZXA9hGPa/xMoF0A9SE9pMU590F2AtkNnxOBwwQAaPcofc5uWHpenfB+048EKzQ+DgOY2zwOaBsR5byUdtupQ/lco7gKuP62w/hjjfqY8mNbsEZl8bsK9ZEYwF/vCwuSATXcN7KiwgOL7wdc/opHcku9bPWIrG92J9DsB0w+Y4WEGQfMf0hgtHo7Brw4YHVMX6eq+TrLjguo9VtkuEfPJeqTgOKcoBIC2J4Q7I5eslPWTDn4afcETC1gRmTlkg1BtbSsMKmOJbLbgNVbAtNPyLhVRxLlhYfNBaoHZGG2IrLaDtj7IfsrN4/lwDIHSYZbWAFpJdZPOFQrrgi8NicqXmdk4aUBVm8DRbzm0k0/qGBNSLoGhOM1c/DbAddfF5h+wnAWgLUc7UxQ1hgrKbwSA5Bq9phamS4xeBd771xzKAZmX9dAdRwwfsEAmaQKWB5LssQHAqNXAXdfjZURMUAmvwlYvq2QXwdUE3a4bh8JjJ+TvVu9ITH91GH5tqJPdw5kd7xeywsCbDPi0CVbUjab3Xlcv68wfhaiKkCiPpAorj3u3laYf8TXw5R8HBEkg2ZUTACNQ69uvFsw6xa4+Rr7L8cvffRDBmxOFPI7Aqn1qUJ54bF9wL/XpxLjFxwaXP2Sxvwjj+1xD8oAUyoOCcaKQ43Ibl5/LUFx2Q8kAkavyTA2jyUmLyzWT/Qw8CmuPexEoD7kuStuPOq5QnHjUR3owfdJkC93QTcLCfdAIbvzSNdHaq7QAAAgAElEQVQe6dLi5r0MqqH3Mb912DxSyJYExcna4vr9HOWFQ7py2DxMACgkFZlZ1QLllUMzVygvDIQLqI4TFFcWLpWoHitIC6xPFaRhYFCQQDdRfP0coNuA4tJgfZpBBJ6jds7BS37rsHw7/Zw6AFEJ4FImxwbBDsvqOGHYDoDqWKO4cpA2YPNQY/pJFxkuXr/dXoL8xmD5dj4EAE2edbBjhXaeDj7HbOWGwWF9kA4y1SCY0No+UMhvLWRr0RyUSLYOzZ7G6KzF9nEB1XpozWssWRus3mLdCAdiEjYTyJYe6V2Hdj9FurKwpQI8kN22CBDYPipQvG7R7nMCN/pkheZkjOy6weaNEYqLDttjSl/zyxbbRwXKsyb68iKzmMe+0OMR9NrAThPYMQN8iosWAYDLFGTnYOcZuplGdmvgU1aEZBdbuEkOM9ZIb1tUT0oka0pkixdrVE+mSO86dIsU+WVg/cjrCgEC9ekE+XkFn2q4ec5+01db2FkGvWzRPBghf7GGmyRQNfickzGSuxaqs/DjNFaAKKCgvLbdy1C82sLsj+ByhfS2gS9TAk/rIKBg90ZkJXMNn5EpVlpSZjulcVg0FvbBDPp6Cz9mF2YA4MsUcqt3ibBxs9Mc+mJF4J5kBF5ZClE18NOSbGeiIbc1wnQUFxF+ALwACF4Db9d7IoVSTIKtWzKYqy3CfEJ2dLUFkgRhXPDfIQBFzr8bTi5FmvCx5G4g9rmtaSlhzTJWhfSAuIsLnSz2dTpHsO08uzfThIE90Q8aQoDQ+sfrRvqtB6hfsL/yp1aS/KJsf0oO4ydtX4ixFEL8KoDfCSFshRB/FcC3AfwXIYTPft47+Edti/cOw1/7n/4iPlgfIdcG/+LeD3DWzfF3X30Nb8+u8aS4wfdWJ5DCQ4qA62aESdriMNtAS4fvXD/E1/fO8Q9fPEXXJljMtvil/TN8sDzErx19hF9/+R5Op7d4WKzQeo0PVwe4XI/RNgmeHN2gMgm+sf8KL6s5Xi5n+HMPnuO2K/DdVw/xF974BMYrfPfiBEVq8NbsGmfVFIfFBhfVBOs2xa88eIaPVgc4X0+wui3xxqNrtE5hv6iQSosPrg9hjMLXT87wmx88xcHRCu8sLvGd1w/xL73xQ7yo5jjbTjFOOtzUJW7uRvjVtz/C3//OuxgdVuhajaLoUGYdVlUO02l86/QFLqoJLlZjvHVwjVRadF5DioDv/v4T6HmH+bQCwAHSo+kK55sJUuWwbjKsLsb41a9/gP/nH72H9/7sZ9h0GV5dzzAeNShSg9fXMySpRZm3OJms8ai8w9/7B99E+ZQxin/+4Wf4vZsHcF4iVQ4vX+whn7X4pZNX+I3vvYVk2sJZhfG4wep6BJU7KO0wKlrsjypUJsHl7QQhCBws1jh/ucD8cIO3967ww6sjhCBQZh0aw5nJdp1jNGmQaYerVzPsnSxxczGFUAEHhys0RqOqMijlobWD/f4UD3/lFS7XY2yvSvzK+x/hH3/6Bv7M4zOcbyaouwTTosHNegTnBFPBMwPvJbbLHGKrkR5VeO/4At8/P0Z3l+Hg0RLLTY4ss9hcjPDgyQ2qNsWirHFTFWjqFLNJjasXcxye3mK1zdFdlhg9XGNzNoaqJZInW6SJhZIeQgQs1yXCRY7s8YZVLiLgrcNrnK2m2NYp8txgkre4WY+QJBabl1N89Wsv8Go1RV1lyHKD6uUYcr+FtxLHh0u0RmOzzSGVR7vMUSzY6eq9QFOl0KmF7TTeOz3Hx5f78DF51qwzInUJCOWR5BbdJsV4r0L94YxBRDJAGAmMLMppg+rFGJgZlJMW27sCQgZABEgd4FYxdW+p4J/UkCLAthr5uEXXJnCNgtAeoVMoFjXqy5LDhDGrRhCAbNqyj3Vq2OMKQBUOUnpW1RiJtOzQ3hQ4eHyHq/MpkssEZs9C5A6zeYVtlcEsM8iRYY9tpfk8hYNQAUlh0N3mgAfUvEOSOLSvRjh59wKvXuwBQSB/kaB50kHlDsED8/kWUgDXny4QMg+Rsqd2Pt/i7sM9ZKcbNK9HDMTKLbyVSM5TQABm2k/uMSTdCisQUg94QT/1WsLMHcTI4uBgjcvnC5SHW1SXIwZtFQ5yoyCOW8hnOYdkpy2ECPC1hqgVQuE4FCgs1IsctvRQtYRPyKS3xxZqQ2m7rOJiQweEsUNykcClfRCUH0LC1FZG9gIwM0e/dukgNgrJUkJ9bY36uoBaMT3YZyGGTUnYqcPkA431OxayYqCWywE3dkivFLoDB1XJOIizGP0wg5mEgXXfhZoAdmYhRxbiPKNdQHNAY8uAPpTKpwH2SQNcZcN3jR87JFd6YD2l4fknW89uWJ8EFOcK3ZxAR1iB9thCrxQl8nek3ofuSxDsN48M0guNbo91M9mVhJmx87WX3fehR2a6k9D3bH3P3gfNYwgaGL0Q2D7mcK1XZ/TDp3QlYHMOoXTNXto+3CnZkO1u9+jrN5Mob48+ewQxBFf1oVnNicPomRrUCmbEpN4hHCoFXBqQX3OY0U1Za6O3PUsemeMNvc+9X143O4UAQAZfdTt23RZA8ZoDHV2RjW/3MciakzWi/z7EOhMMagnV8bmC2ilWhOV10s7BPtqWLFR9BBSXHG65FMMgafuQAzczRvTPUhETBFUtsuMgK11GOXyvfkoFsjuP6oGkmicHPZ+SPbG9YiRbBmwfcFBUvuZgRAQ+Ty91toWEGVEuLzwTqLO7nQc72XD/XC5QXLC3dHzu0Mzk4HPenHBItHoTGL3CIHvuA3fGZ7x9tiJrHiL77VKgvIqy7jaGX9VULZiSj91OYtLxlhJRW8jBtzpI1SMgd2nPdGOQYwsP6CjRbScKxbWFGashhCnZetT7CuWlY3/qGxqTZxY2qiSEJXPfDxK6uR7UMKphX2hQggz52lKBFNntIDkk01uGASWVhVexciXaCvTWwGsJnykqeW4b2EkWZc8WdpQgWbYw0wyycwwW0hIhlVCVBaJHtZfqysrATVPI1g2+UShBJjd6RYX1A5PrxtnQr0lWeMf+y8ZANAZ+Vg5MMjoDtz+Gut4AUrL7clKSldTRA9mH9BhLwLrepdYCZFoRAsR6C5QFUNU7BrFnS30A2haQnweSQsl7TOnOg8kPQzX8+8eSYO9vfypSYU/Dyd/5OTCWf/3LZSyFEL8G4J0Qwn8nhDgEMA4hfPJT7/cFgeV3AHwTwDcA/PcA/hsAfyWE8M//THv9M27FVx6Gh//p38DJ/hI32xJZYnA83qDUHX7zg6c4PF7i8nyGp08u8enzQ5SzGkfTDa42I6Ta4uZ8BpE6qMTDv87x5OtnOB3f4v/+g6+wO1IGeCuxWGxwdzeCVAEuLlKFCihHLTa3JSZ7WygRcHc2RbZXI3wwhn3SYD7fQsmAq+sJF363CXzhh4Wq0p6L84sM4rhFuMgoE1tYnD66xsvLOXCVIcwNVOpglylGx1vUn03gZ5ZT0cxhb77F1Ys5uxetgLAyLn499vc3uLkbwTcKcqPhUw8RBMSsg6815kdrbKsM6vsj2HFAeNTANXEiFpnM8KCFrzRGBxWqszFC6iFqdiu6ow5YaYSRQ1Ia2Jsc40crbF5MIRcdXKOQvUxgv1LDrVJkrzVZwcc13DKFqiTc2EHfargHHeR1MqSU+olFcpUge3eJ9gczpEuB7VMu/IXyCF6g+EGObhZgDw2ECgi1gtoouCmZW2F2CaSfS4rtWMVgx6z+6BYMRjLHBrACaqPox22Bbs8hiYFPds+g/ChF9YaFXiq4sUd6LSG8QHNskZ9r9gGeboFPR1AVF2fZDRdn7cLDTxzSCz18Odq5BZIAlTmE1xkQvZbZtcD6HQbu1CcOspHwxy3EdYrsRnIxLABzYCBahdGnigFDBx44aJH9sED92CC51jALLlp94ZDMWogPRsivBDZveugNOwW3px75a4luQeZw85aDMIKps0lcGANoj+PPG0rBXemRLCX9tk8Nyk+TGCbFxahPAPuoxeS3cjKDX3VI7iSk4zmTjcTohRyqK8zUI3/NazAoIL8E2j0uZHv5cl81ITuyi8LG/stAKXt+KbmgzSPIGfE1aA7oDfZv1tA/LJHdkIXtU2SDAvSGzNzlv9JAflJwsR7lzbJjaNT0I1ZVqAaonhrkZwn0FqiPAkavmICbXYsh0bN+5JBdqgEMsF8y7lvJ6o8u9nb6JEAaMSTKCieGxXCyDtg+7CXaYvApAkB2E9Du0zPay8p1zWvMjPG5YKN2L6B8yWvSjO8F0CCeZxfl2oY/7xb3go4sMPvE4faraidBB3ZBNlEu3deStIudd7f3nUoDQNI7GxSQX0fPayngM4KI/CbKyNvo+40BVtLwOHoQcj/lua+3UG1AfUAPdXEhhroO+vz4nVcfsceyD77pj605oB9ZxWTOdsEwpeqE5xzA4MFM1wSRQ6jRvWqXnl3rU1j76/Z+IqpP6P3tU477JORenu3TKH+Ot2336FdFAFM1uzBIxfsFv25Ya1LvRankvZCx3m9pone2m1JxEKIv22vKjhHIJJsx2eSddSSek7nA6Nxj80giWTGwanTuhuPim5fnxeYC+a2HMgHVgeS+HVD2q2t6cMsLJqaaUgyBQrrZWQXS6FPuQYjNRfQs+yEEqQ89yu5ilUcu0cT+UiBemw2Prxvz8V2sZOlZS4BghbeXg2ez91+7lIBO+Oh1TinVTjcEKC4RQxKyLQTGLw22JwmKKzd4VUevWlQP0gFY8TWh/1M3Ht1EIdn6KDXf+VpdKpCuXJSeqxi8trufGSlkdxY+YTBQkALJ2hFkNQGqcfCZxNXXExz+Tsf9zSmzFp4gTFcGzUGOdG0gOo/mOIPeOMqNu4B02cGOkyh5xgCwVE0ZdLKxQ+2HTwWSFd8ofbJr0DIqGuwgg/bRSwtPGbct6L3VlYEtk0E+7DNF6X/nYScJ9NrATBPoraUU2Xr4MqGsNVHwKc+BXneQrUX9aIxkaxkipATMOIGu4jqqs2iOS+SvK/o7jadkNgT4kj2esnWUE8cQIbmu4A4mtAN0lj7NCPr62hLROnY6AvRNNoZy3br7PFjTcfDm/U7q6jz9l2W+82r2iappQhaz6QaZbihziKohS9jLcRGBYL8JwT7LECjF3dYIP+qHjMmuoijInMbHEVk/waIns++mDIayX3QGfXfl55Jf74f5/AQv6OC9VIr3/UO24BxElM3+qQCWf/vnACz/xpcHLIUQ/wmAPwfg3RDCV4UQDwH8zyGEX/2p9/2CwPK3QgjfFkL8xwBehhD+2/5nP/Pe/wxb+c7D8O3/6q/CB4GXZwvIxCPcpcjPFPw31zCvRhAWyN9aw1oJ7yTwWQk79oATCCNenHKlUb65wnadI/8gj6b8ADdxSC/VENySvrFBfVECuYe+TBjg8sAgfcWFdPvQAEYOYCm91EjWAu0iQABQb2/gPhrDjjxUxeoMnwLhUQPxMueiKaagtvt+eIy+Mw7RD2KnHmorOVEeUb45einZH7cUqB9ZzL+ncfetDqJSSG9VLFwHspvoI9VcQPZdfd2C027VCNSPLYQRBBKBRenwvI1PAHvYofgogysC8kuB1Tc6zH4nxfqpR3YrobdcrHcLh5AGpJcK0gnkF+zHm39P4vZXOiRnKdIV026zK8r70iU9m7oPe4kLSTPG8HzVQ4KEdhFw8LsByzdlDA7hl395FrB5wn/nl/zSqg/jwsrFiP0RQV55JqOPlFPqfnFVPfTIriWqJxbCCeRnXC3Vjy2mP9SDL0439DKaiUB95JHdSJgpF3v5lYBLdn7S8jUXdKuvWeSv9JDy2s53wTjpkh7Y/JryzfJcwMbFt0u5AG8OgPwKQ9Lo+pQeTjPZBeV0M07ZN48pV5t+BFz+BYviRQJhORnvKzEQuGDtOwfLc4F2TjmlroGrbzFMKFlSllo9CBg/I5vSHAZ0xwajD1ICkDnTSLdP+LmSrOJCGFzwdxMmqBavOd3fPvF48A+Bm69J5Ne7pE/VANuHYrgOvKYk0kwEbMHzIDzTbEdnXCCuvuKx/7uxvH0kMH1usXxTo5sRnK7e8Rh/SnlldcJzld3xeku29Le6LIbZeGDzhO+H8XN65rYnEumKUk14gsPigr7O6pjApLggGMpumeI6Oveo9wU2p2SKEOhZLi4JmNJlwPU3eB3sfRdYvcUp/fh5GNiV3h949y4ZpfJ1L63j9ZwvyRr0nlCv6SG1BQOI+lTSIHjuxy88AcM2oDpi1Y40AfltgCnEAA43jwX2v2dx8zWN/DJAt2RQ0hVllNUhve3pOqBdSF4zF+wy7CaUEnczMQD2yTOHZk8OSbjZKqbAxoCh4tqj3peYf9Dh5r0M5aVj8fxYUNZ5qsnMBHopbRm7CzMmF4+fe9SHEnvf77B8M8HowsMUIiZwYqj+aPYIpsorj/Ujhf3vd6iO9MDauVQMacmbxxLZDZ9TmughNTvgBQCuELGuhN9LPXhJtnz86iGlx/mNH6wPuuG5z295LpJtQLOQ0FXA6NzC5QQStuhZI4vNQz3Iwc2YycPlFcGcGfE8u4Sy5GzpsHpDo7giy5RsPFZPNJJtwOjcoJsq9inesU9SOnD/BMN9+uoOl5OxSlYO7Z6GzQg25x92lOWOCDDSDY+tWbC7s7imVDQooLjyyO4Mmv0EesuORFMSDG6PJKQFRucO9QHlsqOXDdoFF7Eu673d9HPqLUFOH+wzelGj3aMcVjUezZ6OXmQHEYBuqpBsYxKt5vsg2VpURylEAPJrLprZl0iAHKRAsrK4eyflZ1EXE3xvHLYPNGYft0xJdQHLpynypUd+xcexhYLP2E9ZXnSoD1OCpK0bgmv01sInEs2+ZkquEtC1QztPkN2a+P0s4HIOQpKNZVhPTsCmageXSbicXlszVijOGrhCo5trJCs7eP6TFdmydk+juOxgC41k1aF+kCNZWaiG4UDSeDR7CaQL0T8LIAQkK4OQyqETMruqIRoLc1jClgr5eYX2sESyNlDbDt1+QRYv4boJLkS/qGZth6RkvNvLkV1W6BY5io+v0by5D701cIWGsEyFNdMUqnFIbmv4XMPnCRACzDRFfraBK1NIw4AfVRPc+ETB5wp6YwZAIyylz3LbwE+L4ec+1ZAdgWKI9STJdbXzQAoOTmVj4SYZ9MsbhDyFHxf0dcZNWL9jFBumtg5ezh4cZinlrpuKALDpELKEIUF9wI6UDOjJKZcVPsRuSfopB7DoPMFkDw6tQ1ByF/4D0LsZAtB2TIIF8GMBPgDBYZry/71vM26hM7GHknJXkfMDbmAcTQdoTXBZ1dGTKSmHVYrS1zRBiNJbSElw+KNezT+khgTAj7OVAH3DfxpSYf/kA8vfAfDLAH4rhPDL8WffCSF846fe9wsCy/8TwN8F8K8D+OcAXAD43RDCL/0sO/6zbuU7J+Hw7/y7OH50CyU9Tid30MLjw7sDaOlhvURrNJbLEuNpjc1nM8jDBlJ65LlBph3qLoFzEs1tDllYJJnFo70lrjYjNN+dY/yta6zWNNjYTQLIAH2VoHjvDptVgcVig9W6hF2lkGMDIQNcrZFNWlijMZtucXc3wny+xe3tGDpxMJsUybhDUXRYrwqI65Q+za2GyDxk4oGzDL4IkPMOUnokqcWkaHFxMUNoFEQrMXt6h84q1JsMSW5hXhcoHm3QPJsAhy3U85xplPv8spIJJW7BSsjLFP6opQQRgG/0zv90owEJqKcbjIoWNxdTTPa3WJ9NgMRDrjV84ZHcKvgnBMX2sANqBWEkOy6dQL5fw/9wDDPzwMxQbtcplPMazbMJ/MRCtIqMYgDkOE7Y7lKIRYdwnUHstwhOQCUeeF5APt0ifDxC+u4K1YsxDt6+we13D+BOWoRaAyqgWNRotilCrSHySB9sEsAD408UtqdcBEECfmYglwl7K0sLbBKkhxXcJ2PYhYVacTLsJxaiVsjPFbo/UwOvcvYvFg7JpIN7XSC/kGgOPPzYITtL0L3ZQFyn7AQNwOzJEndXY6SvEnQnBuwXE0iudPQOAv6NBtnvFagfOoTCQV8lSN9Zof14yotekE2yBwbpeYLugcH4Bymqhx7iqEVedNheliieJ2i+0kJFGaXdJ9PazQLcxAMTw85VCSD1UJmDfJbDjjzEXgf9WQ7zqEMIwPj3s11lzojJwHqlEB7X0B+UKC6Bu69b6GmH5Hslmq+0SJ+lA/tny+jZeaeCWadQKwWfB6itRPkqAu0zzcfWMTV34Xk9BgHEROWgA3zuyeC+10BcpcgvI/jvJIrXEvWJI+Odeug1vVp6K9Du+wjEBNzEYf5djeYAaE47pK8TeMXan767FG9WGP2D0eCvFR5IbyXafY/8ggvxdhGGWhMRYopxxiFU3wcL0LNZvAaao/gSWg4x2j0CeUo7A0ICJMtYtzELw+JeWg6EGDzD58W9wa+ZOei1QrIkmyo8hnTePqmYoIk/695skH2cQxiCoGafksj6seEgy3F/en9ueSbQzcG02Zae1j7hN1mJoaoovQPsGGiijPJ+r63qeLw+Ye2RaoFu32H2fQ2fcEBhpg77v6Vw917A/PsCN99kmrNqONzqpvRQjz+jF7gPXvEJpYmrtwneu1iXpGr6VzenAmZKj/HmMYd8xWsxhFCtnwLpksMC3UQZoACao4DxMwLNdsHBkC3pm82vQqysQOyc5HnQW97Gjji4G39KH6stRGQdeV7qo4Dx81j3UlH6OTrjgMKO6OPtZhwAuZz3TbZkIhk2wzTXu3dU7Fzleeh7LG3B18PlfAyb7wBxfch90NueJaWHVTUEcLpGrO4JsZKLVTuqBbLrgGwdsH4s0e4FzH9I5reXkaar+BxH9MSOzh02JwrNAVUA1ZGMDDVZw9EFq0wu/qzE4W8zCbc+2NX0JBsONHrpJSW4ZGrTTUA73bH57VwMFSUuj7U8DgPbyMoTDg3KCybm9kMEaSLbXXG4AgFMnhv4VKKZ7xbCqgtYn9IfbcYCiw8MNica7UKguNq9L4prpvf2DGfPEOvG4/arGvMPuYC3OZ+rOpIYv3RDBUzPmkobsDrVKC89sjt6Qk0hkC/9wFb6ZNexqlrWtvQg2ZRkhvMrg26uoaud/LNdMMVUN5SoJls3VLDsUpol8mvD+hPP23VjifKCww/VerKVhUQ7UyiuLEOzCknp6Uwj2bohkMmnDBjKbgyagxT5ZYvqJOMwYT+Dav0QyKS3Dj7l8bHHmHUw6R1ZU9ntAof6VF7hA/KLGnaSxhAlsoHNgn5eGYOTACA/28DOc0pSQ4CsOtgF13r98Q/hbLFnMyiGOUEK+FRCbQ2/21J6TeH9LiTIergioQe4sRCtgR9lEMaheTBC8fENw4HqjrUmLUOShI1hP5HhE62Bz1PIjlJW4fzOW1lEn2Oid3Ui3jNo6F59CHs51Q5st2QX0Rkgz3aAU8kd6NxWEFlGwHcfdN5jC0OIktckZQiQJ0ANbUcZ6/2uS+/p++w9mrG3c5DCfgGfZYhs6o/5Qf8Y258IYPnkNJz87X/nS3/cz/7Nv/VlAsv/N4Twz94jFkcA/tGXCSwfAPjXAPxGCOHvCyGeAPgXQgj/w8+89z/DVr5zEn7tv/5X8WrFRXddZXC3GfReg7zosHk5xezxEnujCh9/dgTYuNJJPESjEHIHdBJiZHF0sMLtuoRpNbBKIGsJ+bhCeFYi6AC3sEjKDs4o7C22uDqbQdQSIYbFqK2E3bOQGwU/NwPISlZcJGcXCupbSzQfTRmyMnPQ0w7uMkf2cAvz6Rg+j4BHAXLRwm0T6GtKR9OlQPs4dm0+qiGeFUg2DCKxC4v0UsM87qAuUnZ3biT8UQcsEyRL+ilYAcEFpFor6Fid4ePxia1CcaZQPTWQlYJqY3CIBXwGAkAvgIcN0u8XaN+roT/JId7dQP32BPUjBzHvIF/Rd2b3LB6c3uD15QzJs4y+mAnZWlf6HZDdUNaZX0m4NMDssRalPTFIbjSrIRoumlUl0B47hgCdWhSvNOpTA32nKXMdOySXGqpj92jxWqKbBnRHFvqGvaYiRJnfooP6lODbZ7F2pJc79v6hJx1gJYrnGt2eh3xUIfvHY7R7DH1pDzzySy6y/EkD+TJH+s4K1SrH+HsZQQC42FC1QLcIsGMHVUvAE0SYfQt9G7/sLgW277fIPs3QPm0h7hKMP5XYPiKoCkmAMALJikxBshKo32uQvMgoC92SIe3mIT4fWePihYb89hLth1NWtNQYFuYQBDJBEdhAkDHXW0pkq5MAV/jIpgsCoZRVLM2hhz/qkHyWobgQWL5vMfkDjc03WmCrodcSei0QErKG3SwCgRdk/tJfvYb7P/ZRPSAYqU4dRCtw+JvAxZ9nANVQJaMC+lqQ8TOBbgo0hx4Hvy3Q7FPKW5zLIRimfC3QzhgQNXohsH1IhhECWP9yg+RZhvxSoN0DzMxj+qGEzRk6o7cMX4IAJh9opMuA5bvA+NP+fLD/dvQCCFJg80Z8nWOacX4VF6yWCclmGkFSJWAnAbagV27v9wKuvylg5xbT30/Q7gFBU6bqCiYZbx8LtHueEnQvMP0Bg2FcjqFGxSeIEjOyun3/qi2Y3JxsyO6aacD0Y2Bzys+CbkalgjQCk2dk82zBRfnqLeDoNz1uv6pQXhAc1Yfc73RJEIlAdrOdUYJdnNM31i76jl1+VvsEmH3kYUtKGF0KTJ473LyvkN3wNukyoHogUFwELN8B5n/Ac1k92HV/Tj8J2DziIr464j53U57/vd8jy1q8ZnpttvJYvaGgalbQJGsCWi6QmRZbnQCHv+2xfItdnfUxn9/lBDebx32qMs+3S8mcc4HM947N+bi9LDBIgdFrB5cIbE+oJGHIFa+tIAmsNqcCs498BBUed19leFF56XDzrkZ+FcORUp6bdiGGTt5+2/u+G/orbS6GEJ/yioAu2VIyzKoWJviOzskEr08ViguP9ROyhumSQMyOxNBx6zLWCTqA3pUAACAASURBVJUXFtWRhhmR9Z59QvlkdUC54fjcwWYC9YEEBDD72GJ7rMjan5EJdYlAtmYQT3VAX9/mlKz3/COH6oDTkulzi2aukN+SmevGrM5SHZOllQmo9yj7n7zYMSyy82gXGi4RKG5s9AOKmDQcBt+crhyqowQuYzAQEOdXDR/Xa2D02uHuHc2Qrji8KC881o8V9n7QwRUSsg1YPdUYnfshfdYnEt1UDnU76ZodqdmdhS0UbMnuyj69efLCoNnTyG8sNicJxmcGtpRopyom/QrMPmlRH6UQLiBdO+iNQbdI0c4omW1nCvmNixJYpiInG4t2kSC/7OAyifowQXbnYEuJ/MZg8yjF+GUH2XmsnuaYftpg+zAbknR1zccdnXXwWqCbaejaI7tpIazH5umIKeUrSlr5/jWoj8ge9/JZqoT4vsjPq6EWpDopUJ7V/PtlhfYwR3bdot3LEKRA8bpG9ahAsnZIli0QApqjEunKoN1LUZxVaI4LZJcN7CSFqizUtoWdFbClgq4c9LpleE/fgbpq4KY5vYYAezA7VqrYXFHR9boClIDPNNSyYUemC/C5hqwN4AJC/Led5lBVRxkryGyGHszp3nsuIasOfpRxCLVuyFTW3a6O5F7XpagahDzF0GkpBMGjsYBWEJuavw/0UvY/h3UIkxJiWwPO0zdpLMSmQphNACUh6pYS1j711kdJq7U/nsgaPEGlEAhtB0gx9FQO3ZVNQ7YTBKFDaqxzlMR2ZgjtGZjKnjX9Sdt9qeyPynN/9KZN80f+/o/a/qQAy4f//pcPLD/9t75UYPm3ALwD4C8B+M8B/DUA/2MI4b/8qff9acBSCKEA/L0Qwl/8Evb1S92y0xjZKwh69FINVSH95HAIjBgHFBcSZhIGz5RLd+EFfSCCrsiwqC56JTyGKpG+KkBXBB7pkjJO1dCH1hfduyL6ojqgD40wk4D8SgwSVBay9xMlTpkZ5U4GQ8UFanMYkN5FRkPwPvpetUffHdiHOPS+Jmkx9Ki1exiCH9JbSqZ62Whfo2Bz0GfYCKRLMZTKs74kno/oCczuuFjOL8iiVA/pdUs2PF4RayK6WcDsA8q2zHi3f+zQ498sQ8eQ/GkmAXqzqxzo5bq970t2nMDXR2Ko4kjWfZw7n7+L5F5fE+Jj6EMQfb3CruDexYVvH/TQ7HNB3gOuZLOb6EuDQWbaJy2qFkNIhYzTcYLIADMV8XohE6Hr6Fmr+Z6zOT1u+XWUL8bEymTDlFVd8Ry1+2QwpKNE1Bac0PayvtknHpuHXByaEfex3xcEDAEgxaVHN2GQgc9isIYGILlYZn/ljmnqJWgi+oVYl8PJra7ISqiaC/h+UZvfBlZrdCH2PsaKkC33tU9TtaWA3pIR6WZiAGTFtYdwwOaxjPUcYagvUe3O62VKAgNpgG7O8I6+Psfmuwl+X8thC4Hiisff+8HaiaSkcEMg0uyLoeoEAMbnDqaUaBZc/OqGjEe63u1H76HMVgHNnKyOdJQ39v5GaZgU2cwVdLvzidJLJ2KYBdCNBIpbT6Yj4WvYh3ToOgweOp/EbtX4XvOax53dBdT7EumGC3AR6FUc6j0EGZl2uqs0AeJ5v3HYHlG+lt9QxlheO1QHimmakZXUTRg8ZEHu/HvSAsnWMV00ykj7a7Cvzej7DPu+xT7dVMbFpxlJJNu+ViMMXY89gNM1f1Zck9lJNx7tVEE3nPDbTKC8tNic6OGzOr/lbXUTBnZIdYjdlwHtjMAoXXvKrZcO3SReAIK3CzLWtzQED91Usb6nrxkKPRsmBwaoWbBXsE867X+fbAmsNg81Ji8sugnftww+8ZBtQLtQSCoPHwEjEP2eEkjWDi6PC1kb0Cx09OCRwe2rP/jepa/tczUY/aH5HQPE/kN6w8yYbBd7GAWSTQR3E55nacJQgYIQOyYF+Djxd6qyqI8z6MojqSxsrgapqU9YKZIsDcwsgXBAdlmhOS6hGi4mu5mGrhxlrBXrR/TWkmmKxyAckN406BZZ/IyPSclNgK4tzJj9lrqKDJLAIPskix4gnYdL5CBR9YlAujLxPLHOhNe5GN4nycagm2fQtWVNR+sQEgkX+x37Gpg+fCUkCkGz0kM4VoXYklUj0jj6/LYWIZWs94g1KkOdivHsdzQE8q7QrAyqd9URwlICKUIY/IVBCsjOwUwzqNZBNib2M3J/e7AVFHsafSKhVi1Crof9Fq2BG2esQ7GsHvGaktjkroErEkAKqA2Bn50VUJsOsmrhy4zpqkLwsUKAG6WxxsRxP4oEctPAzkuobYugFD2GLgy/C1myq6npLEIaexddGGSjwjn4nABHNl2U21LG2vsWQ8oPajJ1UXIZ9w0ARN1RUlrcY+9CQEgTyl5trATpaztkZPbShCxi/H3QBE+i9wre90Vat5OwasXbSLkDh32AjrG8XQ/C+v837fB8kDLe11ByGjsm76/nhRCfZwgBAr2m2QXsADumMXgM9SH3t77qpP998Du28X41iY+dlz07eQ8kDvvV3/4+M9k/5xdMhQXwi++x/AUAlgAghPhLAP5lUF/3v4UQfv2L3O+n1o2EEJwQwgshZiGE5c+4n1/qlpUdZqdL+CCglcONmEFNDLptgnTa4mC2wdmHh5idLjFPLJanBbq7HIcP73BzN4bvJMJDA79JYRL6GcweH7vLHcoPU1SnDFaRW4XH75/h0w+O0Z14oJPo9gBZS3T7DrKWCAsDcZNw8dIC3d6u6sLPLaojT3lmYYFlAggJMw6UjnogvdLoFi6WnNPo74461KUGAqV49QOGj9i5pdxyz6ObCYTMY+/RHW4/WSAkAWqjhu40WzAtUVQK9dcMcJsO7AccoBomL44/0ti8aWEWwOhTjfa0g75MMH//Grff30d40GD8WwVW71qkVwrrt1k1YiceoXRwlwnslOfi9Ovn+OzjI9x+XWL/d4D1VzyyKwU7DrClRzO1UBcpVBOTDtdkw7p9Hn+y5IdNt+fQLfj/bt9BLyVcztshKCQbYPUWwzb6MBsAMAuPcKVY9ZGH6L2UuHvfQ3SCQFkA4bSG+qSANPRsBtlLHAmmmwP6V82Uj9P7LtNrBTNnrUdzQICjag4E6uOA0UveV3ascegeGJQfcxpQzVi5oVrKJutD7nZfG+JPGsqYYwKfcCKG1wDZNWV2wgGqE2j3A14fcaGkKn6ZNocewgqUr1nLYUuP0XOF5dtkE2TH5002At0kDlVmUe5YAX0wiq6Bdh9A4LDFjR29vdcSYsH0RzkWMBMJl/P261Mutq2Pfa+SzydclHjOyfT4hOCyHx64wkMageoR603MCMgNUB0QDMgWCLGqQG8JpKqTyJzlUQKoCA5dvhsk+ZgEaScBXkt0C9725l2CKIa10GsZBNAuyPoJDyyf0nfXzgFIDnVUw9t0U/o8TXwtggS2j4HiXMBnAt2E7Fovz1s/1bHPk5JLMyEQXD8Fph8K1Ee9Z5UVRLtzxNTS9amkFPE2DH7YvjtWWD6ecPzblry+XRwecDAQX894fM0egUj5mlUXPlHopvTzpmuBdk8AkvUOLqaS0rMZGedzevjaOYcvXNRLdFPKMvs0T9UwIEaE3f5QakdP0mbBwcj6VCHZELBwAEEQ24xZAp9so6+xBLZaUeY35h/hFUZnHrYEbr+aIL/2lCw+UTAjfg4mFYFluuUAINkCLsoOq2MGvUgD5DcWzZ4cAH95Qa9TM1PAFEgq1o00e3z9ujnBcrql17u4CnA1hxMcECS7Gg8JtFP6GqUFNif0aquO56hKJYpLDmlUJ9CN5QBc21kP6gXWj1lREgQHJt3kfkomBxs+1lWYQnwO/AcRA3tWAWEiYQp+ZqVbj25EcLI9Uiiv+ooPAm9p2UmJnAOwfr9dpjA6p+xTN9H/OlbRGymxeUhfVnHrsXmoYtcp+xnNSCFdOVSnI6iGlSMq9qaaUkM6oN7XKC4t2oMULuX7WYQ4LMkLBMFwoL4T2CsqCbqxigMygezOoHqQonwNNAsNZQJ05WEyNQygqkPNoY/T8KlgFcqlRTchY5XdWtixIjjNBVQj0OxpSKuhGo+gBYQNaBcJko2DGemdRDXwuvYK0DVBs094/duRGs6ZtAG6cjCFGuSg2XUHM9bIX9dojjkF1ZXjYHFr0B4UyF5XMPMMqrLwZQJbcj/9LGWH50hD3KsFAXYDLQ6JErJysTOUPaABWgmoyqDbKxigYzy6OX2PZp5zyDJLUGw72EnG53Ee0BJ2lEA1KtYhJdCNg0skpJXw0xR6YxCUhNclunmKFKB/tHWwoyT6IvNBitrNU6RX1QAEm8Mcui453GgovxXWwyX55/o1WRcTh0NKIoxTqMhYdns50suarGQE/kEI+EJDbbuhUzO4OI2RgGgdkFFCDOvgJhlko4C6o4eyamCOZ0heLz8fmqMV/Y49WJQSfjZinYiSOxCmJJCnBHKRIRQAWUbvCaTjz0OiIbQCNhWQZ/x/z3JqBTQdH2/CChPhPKW6synQdgSkSTJ0X4YQIKQcwnyEEPd6KA2g43PHYwjG8Fz0ya3OQVhLKWz/XIkmcwkQYA64dScv7z2XwViI++fsD2EtQwifk+P+wm7/5Gren/v2I4TiFwKTn7v/F5TC/q+gifPXAWz7n4cQ/uYf9wm/zK34ysPw6D/76xAy4HC2wbLO0TYJzDrl4qxwCBUvVDkyCDcZ5GEDt0qB6OuDFSiOKnSdhlumg8/Pdwoyc+ynXGuE0iGfN2g2KdCyMwo6QNQKWHQIbT/5khBW0Kc0ZRdjUGTbQhKYsqqAIANCSp8ZfQAERbKWBDXxdnZK0KlqBgLJWjHaP/XDY8s2xsEnAUGzj9OVXKgLQz+eaBRCQkmSWmq4ESWhwogYTR5YAaACZCPhJuzpdLlHyD1EI4d+yOyaKap6S2Bg9uhrE0YweTUep11Y5C8TMkwzdiLaCasGZCVjbHwY2FppxOAX8/F8+MxDb1ScPBKE5FcS9YmF3igk690U3hZM9uz2uT86ykURFyKQGBI3veJzs68rQK8pUzRTz8cT4LlxDNVxZRgWNUAM+Ul2j61qyj1dHuhdqylTJiNMWW56F0MxBGWqrmDwiGoxpIOqhsdjpgwPsiP6GtNbObBfvNB6qRrBq8/I9PYhTcIRkHlF756wGFhY1XAx7DMypcKx9mDweWYBfQM2mRoxpFYKE9l1y7979tmM+djpLRlWVYshqbL3q3RzsvxD7H+sDrBjvibC8lwgxGFD7xPE7ph7htiW3BdpoyQ0ViC4Ij6f4GOq6Jlz2Y617mP4h7TUyJ53s8j+pzHIqSM7zUUsWe1uyoApSt128sh0xdsmGwyKgKB43xAZXlv2UkzeB+B+ZXcYqghUzQWyLSNwTONxR/YzaAw1Gr0iwidgMuUynhe/Y0p75s6nGNQbLkMEcdEXWEdWPspXi4sQQTt9ZHpLdk52gIoBNn3RfFBieF/ILqa4xtdVV7t+RURAIy2GdNShCgS8bTfjbbspZai23IUOSReQbMnISkvGM9kC7UwMPkEzIgPag+ceTLlsF67Th5rI+H7gtUQ225Rid+7uqzrifgrH47fZvfeD423716Af5rnI/PddlwjxuCN50E2ZlNqNY+hOyvum64B6T5KhjmmywkVQKcVwztk1SbDcv291G2KAz64btAfT7VQMaa99yJMIGO6f1DvVxAD+VQ+MsEtpVQLpxqPeY+hTz0734JmKC9Za6KgwMCVZ9t7niEAmuZvuPF8MRQKSjUOzR290rxxQ3W6NQsAQr6+adRu9BLkbSSR1GBJWEXiOhA+w2e52fZ2FNJFl7l/jqNRgai7rKlT08VGVwvdmnxZL3yv3of98kobPoWvPsCAb4FI5fEdJEwZ2V28dXCEHJrkHerJlUI/wvbqBLK9wvG+QDPDxmSTLG2KCraH3T5p4kQcMctRkbWALFWs+JCAI4nwiIWwY3s9DlYWLtRVaQBj2QQ7Hk6shgMgngv9OmcKarLoIbARE5yGth5mkrN6YJExbjQm0LlNQtYXP1cDI9onNQcTPGE/AK2wYmNX+8dU2rtWSXdqzMA4+1UMSK4N14hdQZNWYQOsZKNQ5Hm/rILxnIFBnB6bTZ+yzFm3Mf8gSyIqJusI4dlfWMSgogq6gJRnVWN0RMg3R0sc5MJI949l7F++lpQ7gMIIs0ZldOM8gY42MoHWAMUASJWz9el4IylETzX33kQ0Vgs9/bxuksPcZSSF350srBOsGlrsHogNIvs9eBj+Ays/JWiPLG+6FHg3bH6eXMrKafyoYy3/v58BY/s0vVQr7vwP4y/8khOJP4L1/4va/APiPAPxfAH7z3p//XzcpAnwQSBKHZZ3De4FvP3mOZNoinbfIyw7ZXg1hBMaTBpgZeCsBGSASj2TaQowsmm06PKZvFb7y8BJio6ETh8XBmv14IqC5zQEjgcRDzTuI3NGnuU4gEg/IgFB4hIMObmEAHZA+3iJkHuqkQsg8wnELXzpAAcmdgis9fEHvXMhdBFqBQSkLC6QEg+GIOfTCARgzIlscNQPAUMc1F2meQDLoEKssHIQOCFmU6JQWkAHIPIKI4QYZfaI+8wgjR7ntSsHuG4Q0AFbwbwGEcYwPP2ogOwGzcMheawTNLzg7ddz/uUPxjB927ZOWnsuxQ3KrAIfYARcIblL698y+hbACeiPgJ47HE29n546LFxnQLjxZZAPUb7WwRUA38wQWKSCsQHIjYSce7kELVxBk2Am75rzuE3LjF75DBDMewglkN5IVG7lHfk0ZslcAJEGmWTgIS2BIYEM2E55dg64IAxNnnzZDr19z5NDNfey9C3EAseuO80mUAldc4NQP47luCZBMPEZdxdTiPC4+dfRTzqLXLu/BC32L+UVM+qwEVE1Q2c0DulmUGUruK88vWTIXr4me7VUN/9hJlLhO6DPsgXZQAd3cw4wjQKt5Tl0e0B542ElcyJZ8PbuFo4+1BeB5TD3QdhnQzXm9srIEQ5dd9aCX1GAI3fBpiFJpAu0gOcgxI3YGCkfw2gNrsss9AIsLmXgOfErQXB+GgYV18WeU2obYY8j7t3thCNXpq0t6CbIZBbRzPqauKPvtayqkJYhwRYCuApJVQPWAAM6WiFUpGOS2tsQArHvwzX5AxMRTpgW7jMy27i0ogT83I57DoPh7aeK+BCC786gPeY0EwTRe2ZEl5P24yDMTApFuCmxOCfxcBL5ATNaNKb5mxMcRIcp3VWSqVc9g7xKJee7JNNtCDEDsfn1Gs0dmSgTuv8uZuGxHBON9BYmMwM2MKc22OUHY9mS3Ly7jQr5dUN5uRgSoLhNDArE0PD7h+NrTw4qBzXTZrh+yf00p548hPVlcqLvoC+w4QNs+FPG6YnhNDygo22QwTFARwGc7H6cZ8XFdGqsv4rF0E/qL7YheQq/4GtVHcbCjgWZB+beMATtmItDsyeH1TzdMz21nUe5vwk56WxP8ujQOZWLlh0+ixaCNwC0XQ++gy8Xw/z4gx8VrlWykgBnRYyhjthppGYbD2Ly3XYgI5BgGIzyY8it3VTDdSKAbSbRTOcjS2ylRXL+v1X5kLxUfrxtJ2FwOQwYInlebxWNwrBiRluypqh2ExQAgg/z/2HuzGNuyPL3rt9ba45ljvnGHzLyZWVnz0O6mjWcLJARCxuYBCdkC84AsEEJgN288AY/wApYQD0ayJQubNwQyoy1hWnQ3arurq7qqq7KycrjzvTGcOOOe11o8/NfecbNcTWU32ZLd8pZCEXHinH3WWXufE/tb3wTlgQTz9GxuXz3iTQ/0laTmTkQG2o5Cz2A4blIHwiCZ7jscq4WhmYegnJHG1Fbk/SNNlxtsomlHkgiLlzRbmwr72TN+7Tiiy01YgAhAvLV0k7D47SHeNHRZAIutJMuqLjBVnXhFXZD3trNkuK0+jGXhTjMwfxglMtzO0c4SScY1GjuKqA9TtHW0M2FFVevosjAOF0JllEJ3ThJhrShLXCYgyGYRNtYibU4MNg/pstbjUrlNtxaXGtpJjMtidKj9cLHBpZFIg03omowNNjXYSTKASl22Q1UIgdnEKLpp6IjsHN4Y7FQaA4aEWiXg0mstlSR5LKC26VC2vz29le+C/NyDxkjGQxILwHQCQv0okzFERu6TxLfSWISp7EGnn47wswl+ksvYR5n4K7W69TxqFeS4/ap4YEitFd/mKEeNc9R4hBplqMlYvmuNMlr20wPHfh9Kifw2ANjheYwRVvMnvZLGCIP5k4mww/jUP3r7T9t+mlT3n26/X9sO+C2l1H+rlPqv+q/P8sDPxFgCKKVy4A3v/fv/Pwb6uW4HXzr1X/2r/xb3Jyu+MLngph3xtFgwTyqe7+ecjTb8cHlGZzVvH1zz4fKYP3L3E3ZdQqwcWnlelDMO0z2/9ugt3j69JjGWy2JMFnW01mC94uFsSeMMz3ZztPJc3ky5e7Rm3ySMk4bUdNxUOdYpzqdbfvjsDmdH6/56izTq0Mrz6OKQo8WOWDvWZcbxZE9rDc+vFjy8c8XVbkwcWaxTGO25Xk44Pd4wilvqLiLSDqU8z67nHM33FHXCvfmaXZOy3I8w2nE82fNsOed4tuf5xYJs1HAwKajaiMNRifMK6zSxscTaopVnVeWUbYRWME4aXq2nWKuYTypMeE6tJE1xuRtRvRiT3imodglx3qIUvHF8w7LIubmZcHy0ZV8lTPOaV6/mqH3E3XcvGccNmyalqBM2V2MOTrfs9hlJ2qG1oyxS3jhd8ujikCTpQkWMYTop0dqxvJoynlfsl7k8tkhxTmOLiHxRobWnLBJm05LIOJY3IgGZTCt2uwwTWdqbDIwnmrTYm5TJvQ1GedY3Y3yjGR0VjNKW1WaE0g7bGlwVMT3ZUVUx7TpleiY/T0Y1u31GljdsX07BeHTWwaX0kcZv7qmfj4nOCozxRJFlv87xpRF2OLfo2OKXKX7aobQnm9RUL8fCLmtQiSVKO2xn4DLFjS04BcajSk10UmFf5nBc460mn1ZUZYLbR2BFGqvHLSa2+EdjusOOZFbTbBNUZcju7NHa031vRn0iV8jxoqZbZviRaOdUGRYDph1qJ52lam/QRzXuJiV/ZijebEkWNfbpCDu1mHUkiyQhEVc3StKJawPao0cdbhuTXBnir27YX47ASwgWgDqv6DYJRA5VGNkPoJrw90PpVtWlwU07cEqY+tzBtJP/47VBdUqqhTJZ6Ki/XOKdIvk4w+ae7qhFFUbmdN7inbxeXSrinR5AsBpZzMuEbmGJV0aY2xi6icOnjuxZTHUm3aa6g+a8JbqK6VNdTSUBPFGh0O2tZ9qG6iFgqMMBYd11pXGLjuRZLImxoXS+OrP42Il6QSHnUWEYPdcUd8Un3e9PwNkt062syMRlseJWlmwqRXWvxWwM2aVm/6Z0slZ3OpKlkQWPvRp84T0D4yIB9fFOfOZ25GkXEr6lvEiYu0nwo5+IxBzk+dpDy+hRJL7yUjpG06Ww9PFGgqF0JzU3Oshxy3NLemVwIU23ncs8Jiu5f/5KU506khstv78UX316owK7HBaVrtXgIa8Pxf/eZeJ3ro4FIDUzeV09425qmcfqNPjl41tm2GsBTvE+jPNMFuJGz9Xg/e2ZYq8V+wee8RNJ3U1uRDafLUUm3SykR7ZXAUjaJ6JcCayZ3CYBRv3cqLBA07PffXhSspXHulB91DPGpr5l7HUrTDGEfcSfZp3bmdw/3njiAnb3BBTqWpjWeiGMeVSIZ3Z3TzpCs5VUyNgs9KHOZL7SlSyC6Vb2W55IIJPXkpRrKlnIMBVDpUzPHPY1T8lGwLkkH98yzfFePM69ikGqSgj5B5IoG1WyyOGj27mR2g9ZCOhGISW3Ds8R5hMljO/4paU4M4NX26a3Kpa4YGB6u1QN8l4b30qyo0rAe5fLvm0inZztWAdmWRgh3XqKE012I/LpZqrD8fOY8J42jaTp9ox636HZe5KVh/yyG/osm5kw4iJplhqV3rOsGwmmcpH4U5u5ISrd0DdpU5FPj64ktEeYTRmHJK8KM9dODMnW3vZdNl4YaiQZN9pbmoVUoOzvpORXLfVBRLy1ci4uIum6zPTgUTalpMSa2tHMItJVi02F2fWqB9YOXduhmuX1hNt2GpEtmyCz9kNtC0C079Cdo5vEt4xqZnrhDrpxQ51HX3Giazt4Y/vgJh3YXdXKB7lqxZs6sJowAF3dhBTeqsFliYDR2AysMUFlpcsWn0SofYVPE3RRCbDU6taX2vdIJlLJMrCSoeMS7yWBtmpRbTcwoK/7Pj/FQsJtLUnvsez33deHvF5ZEsfCIvYey7D11SRKKWErQwfm65v/SdbzZ23e/YFgLO/9R58/Y/nxf/C5MpZ/8afd7r3/Gz/zsZ9RCvtngP8CSLz3D5VS3wL+U+/9v/K7HeznuX3pG6n/S//9n+T7u3ss4oJNl/Fzk8d8WJ0SK8v/c/0W56MNzivOszWPikM6Z3i0OuCdwys+Xh3iveKN+Yo/cvgRf/ujn+cvf/HvUriUX1m/w3m24eP9ERfFlKqLeGu+5OdmT/j25gF3sg3fX53zYj3j/mLFHz76hMflIZ9sD4m0o7WGOmjzT0Z7VlVObCzfOnjKi2pO4wyJtrwoZtwbr/lkc8ifvf8dfrg75+l+wUFW8Mn6kMO84O3JNb/y4i3++fs/4lcvHvLvPvz7/Cff/pd5+/Sa62JMGnXM0ooXmxmLUcnL1Yyfv/eExhm+/+oOv3jvMdtWvBWdMzzZzHkwW7NuMlLTUduIo2zPuskp25hJUvPjlyfMpwVfPLwU4FyPaJzhONuxrMe8NV7y5fFzfrC/y7ev7zGKWzSeP3XyAf/Ts6+xLTPO5xu+MLukdYZYW763POfJkyP+hW98n+t6zG98+Ca/8O4nfPf5Xc4PNjx6ccTR4Y6vHb+gthHn2ZpJVPM3f+sXeeNsydloy8v9jElSc12OmCU1N1XON4+fc1lNqGzEYQ1DiAAAIABJREFUKGr47pP72HXMe198TtnFXKwnnB9sOEgLHo6vuWwm/PIPv8Cf/cZ3+PXLN3hnfsVH62Penl/xm6/u8dbBDZtavEGPnhxzfLZhvcs4WeyYpxUfXR7RtRHjccUfu/cxf/fD9/jS+QWjqOFVMeXeeE3RxTxaH3A+3fL9j++iI8edkzWjuGVVyiLEyXhPbSOqLuLV40Oyo5LpqGKzz3jz6IaX2ykHo5JNldI5zVsHNyySgl979BZ3DrYcZgXf+fED/s1f+FX+h4+/weG44NnVgjRr2a9z/tWvf5u/8+FXOZrtibXjT519wP/85KsUtTDJPYC2jWFxuMN7RdXEnMx2OK/46uELnuwPeHxzwDSvuLieYSJLd5nzha8840fv32V6d8soaalbWUG1XpFGlnHScFPkaOU5nuz56uIFv3l9n+v9CK08m8sJ8aThF998xK9++BC3jxmd7Lm/WHOS7/iHzx4MCxcnix2rfc6d+ZblfsQ8r3j8ozOyO3tOZzsePTpBZx3eK/7Qw8f85uMHoDwHs4KmM1in6TpNFDms1SwmBVnU8cmHZwAc3F1jtGe5GqO0LKI4p/nGg6cUXYL1mo9fHTEa1ey3GSdHW9b7nK41OKtwreGLb73gRz+4z3tffkrRJjz58ITR2Z62NXzt7gs+uD6hrmLePF3y5HrBe2eX/OBXH9IdS/etyoW6UTcJ/qDBd5p43NBd5bz3lae82MzYrnMWB3tuLqco4wUERw6cwneK0UFJ+WKCPqwxRl7rfFqwWo3xTpGNG8ZZw9XTBXrc4vaxqC5azTffe8x3f/MhLBree/CKp6uFLFBcpfhxR7SMmX35mvVmjNZOFqjWiYx7H6HmDaNxzf7ZVNjGkcXknczZThZfissxyiqSk4J6nWE2hrtfe8Uobnj//XuS1h07fKdZHO/YfLTAzTp0YhlNavabDHYx0VFJd5UTHVXYV7kwisetpBCvNd3ciYLCKcZne/bXUiGgt4bkwZ728VgA98TKQtBVijqtUcrz9tkVH/z2PVnIcKIqaU/bQNeByizqJoHjGreNidfB33ivgueZKDHysFgQWFd9p6JbJZhCo+5W5KOa8ocL7J2G6EVCe9SRXAjAdlNLtIwwD3fUyxwdrBUuAGKOatxeaq/MpMUtU7JXJigxBCh1M0u0MrdM+tsbyn2KeZEOLKvX0J50xFfi81RvFLTrdOgcbuYOTmr8MsGUeqjE6SYOU97KR7uRqCVMJWBXFgNg9ExTngWrQRQk/pohGK28Z4k2IbjonYrsu7kA/yxUrRwE+beR+2YvjCxAhJA908Du3ZbsWUyyFZDYj6k8lcTueCfAuZ1LKm8fiBVvJHDO1JJu7WMYP1GUp0F+Hph0m8nzxbugDBiJ53/01NAcyOKHVwyJyTb0mHa5yObjffCLTuR35UQq7mJZPIj2iuxaXqPqYPvQM30k+9y+LYssqhN5/fYdx+ipFq8xwnr6SID+7g1FdsUgve/tGT249Sr4rit5/uxa2PM+zK5eSH+qS4TtzkLv8P5cMXnq2N8T//HopRtY3OJMixfdy9ybVhY9ykNJQR4/kyC23T1DdeI5+i0BePl1R3UYic+5EEl4fiWMeVSL1LrLYXwhYWJRGdj7WHz3kxd2YLvjvScuJGFXgtw8+zuG7EZ8ytnKDqy3KaViRXl5X3cjOYfjbYfuPPu7iYSYXXZSxbKQ6pR2LMA1XbW04+iWoQ7EWbJuQydpTH7VCFBshEFVzrN9kMo4t90APJVHUmzLFpcYumkiYUiTBNUKSNSNhc6hvKebp0SragggsqkRGXAkLGcPQAFhEkNvZ7/1wV26qFF1gx/nQX6rbuW0vUy3l7F6f1sTohSqam6BX/Qa6xiAprduYB1VkuCLQryZ1g2gUsWR1JLU9WsP9yLlhdvn+4ntH8EofxB6LP8JAJYASqkEeC/8+r73vv3/uv/wuM8ILP8h8M8B/+drRZnf895/7fc43s9lO/3Kkf+X/vqfZd1k3B+tyE3LL44/4rvlAz7cn/CHZo95Vi94Uh7w7ccP+Lk3njCOGvZdwpujJb+xfMC98YrvvLrHwagkNR0/f/iY82TNX/vgj7Ld5Pzb3/q/+d72Lqsmp3WGZzdzZqOK+9MVWnkOkoIf3NzhW0dP+V9//GX+zBe+x9/58KvcWWw5zne8N7ng/3j2JfK45emLQyaLgmlWczbasm9Tbqqcuo04GJU8fn7E/GDPF48v+GB5zJ+++2Ne1TO+f3mHNxc3fHB1zIODFR88P8XtI/7EN97nh8szEmOJjeWTp8c8uLtEK09tDa8u5zy8e8WrzZQkkhWeuo0oNhlx3mI7w2RSkScty/WYO4cbFlnJBxcnOKeYjSuqNuJ4sudiM8EYR13H4TPH0zYR755f8KOnZ5yfrrhaT+haw8nhlkg73pje8GsfPiR6mnL0cxd01rBcjclHDU1j0NpTb1NO7qxZria8e34hF+UfnzC9s6WuY9o64vx0xeXNlHad8vUvP+bpeo7Rnm2R4pzCO83p4YZ1kVMWCfdOVmyrFOcVo7Th8mbKeFRTlgnOK7R2tJsUjCcetbjnOZxXTMcVi1HJPKn4zo/eIJ1XNEXC6emaoonZXkxQjUYtGvJRw/4mZ3JYkEQdN9dTTGJxFxkuc5hZE9RdClsaYQ8VjI8LmsZgOyPAQHmU8SRJR1PHuE6hbhKSe3uqmwzVat587yWP3r9Dfr6juBqJPNoq6DSkFrWJpe/TKaazkt1OrnBMJB/S3UWOOmgY/VZO+S3R2dpGY5YxduyEad0b3Fx6WE3scK8y4nt7lILqOpfnDFLrxemW3fsHpO9u2F+Mpf8T8J1GlQY1a3C7GGIJuSK3YBXp85h24qWztTEiNy8VvLWnuwrxvIGBA8TbPO7QmwgUuLGVi+mbVJ6jiCASqTYueHJq6VHVWyNyasCszeALdbmHsYRn+VSCnED8vXZqUa2WOpe1plk4YQG/WuEbjdpH4m2upUKlPQx+D+Ml7OrwlpGTcB1Fe9ISXce4uPeheamaAezIDd7r9NJQnbcDC4nuNcbyzRTSnRntFc3CYzMnoVtTO0hR82cR5VsN8asYl8gDk7UwBvWJVBBJL6HHjRxmr8kupCpHOkI98UpARHXqMIWAiYHZTIU1rY9kXspTkffqVnzVppBjUB85knVYcVfyfDr4uHWrhtfUTRy6klCibiTy5cljxfZtqWaxuciaTSUVN+1MZNTZpbCQ0svpifeSEL17IKyf1yLHNrWiC+Dg9QTseuFDuJH4k72+TQefPIHt22CK3lcXWC2FAM0dlHckqbs/ztGewHBy69UzDExJfhlSshcCRuS1M1wQmkr+nl1J16nI4wUUmYpbVrkJDGTw+Mrc3p4jNpPzSzeKdCl+Y5sGD20pNTRwKxGNdyEQ7CYAo1TYmWQljF28E/aqOgrs5V7k281UwJJuxVtsM0JitICLeO8pTm9Z+CGltpA0b1OFwKIp5FciYW9m4vusjtTgLe4l08lW2M12FBJhPbQjAXQiOZcwomahRBbdQR1Yz97jLN5GYXeVDSoKG7ydlciO0TJPceGpDjXZMnRdhnCw3r/sjCJbOeqZYva44+YL8eBfFXYLtJV9xNvbDAHxb8qxkmRd9amE6GQvidrKCShzkbBH7ViTrSzNOAQp2eB13bsgDRZQZmMBeM1EDQqF3hOtW+kDxSOJz5dSPeLM7Viy6/D/IoCuqLAhoCjYIRovgVKegcHsAazuQopzEypWJnpIVM6uGwkzShT11GBa6em0qSK7aqmOYqJCEorjvaQyKwfZsqE8ToYEZ1M72qkhvRGQFxXCekY78ZsCQ7Jwz2B2IzOAOhM8n10mgUzaerrMEO86TNHQzrPXvJniL9WNwyXBh4q8X20srKnUFhmiohveg7rpJC0X0J2TsB/Ax8IQqk5AmxslwmpqDb0EOaTXqtYOSbh4P4QI6W2FT6VvvK888UpJam0PEF8Hi72v8ieSW9FqSLlVdSOA7idxQM96vub9vJXDvsY4OofvLKoP7Ok6SZx9Hfj1vtnXWcafxlr+rO21RNk/EIzlL/3lz32/H/+Hv/R5MpZ/GvgbwCfIf8AHwF/03v9fP/OxnxFY/pr3/p9VSn37NWD53c9SlPn7uZ1/9cD/pb/9J3l/d8ayFiZkEtfc1CPySOSj78yu+N7ynG2VksYdRju2VUqkHZvtiDfPrrkpcpTyVE1MGnfsipSvnL/iqhyzrxPWqxFR2pFnLWWZMJ2UFFXKm0dLnq3nVGXC22dXbJqU5WbMG8c3fPjshNGkRmvHdjkmHjXkWUtVxzTbhNFBSV3HJEmHtRqtPUp5uk6TZy2b5ZjF0Y7V9YQo63ABmESxpS1jfKvJD0qqfYJvNSfnay4vZijtmc5LrNOoYIAr9imu1SJD9KAnLa4xctFvwsp3p/GlkUCiyKEjj90H83ji8DZIBA9ltcluY0bHBW0T0ZYxyji81ZjUYhvNeF5RPJsQnZQAIkFVBIOHR8dyQSxyfEe7Szi5u+Lqw0MJC0qCZ7U2IofcC8jAKQlVKiQpNz8qaZuIrpTfVWVITgvap2P8YYvvu0tDoJI+aEjSlnKdiSRzLf8E3KwjGTe0r3K5EArSS6IgSS0kQMinwoSQOOKLeEjEpdGYrcHOO7BKmJxNNAQfmVKHUBwkqAlQrYQh9cFNeAEQ7qTBV3Jlqis9ACJdKlwWwEu/IJk41DYcJ8/ALPTAAqcYP9EU9wUodHOH2Wpc4vHHDeZ5Chrc3QouUkkULrX4FJ3C5pJs3F/QuUwAiR07zE6jrLApLnMkSyM+qiQAiVhem9kH6WIZgEqhBj+nrhm8f9LNGLynQUbZ+0X7gBMf2A+bCUjRNbQLCYYCbmt+/G1dTr+5RJ5bWyXdrblchPcSNhdBvJe/NYd2CI2S+iJPH1/vDdjA1PQXFF4zyC37+3bB26sbBtmfCuPKriTcySUh+ElDN3YSprO/TS3WjQQZmUoFL7Q8n25v/Xe9t6w5sKRLI3JPJ3NhahlbN/Ghh1TGn6ylMkh1ISE17S/AufWWBgZGhXH3fX7SJSpAOdqr8JpkPvr993VIquM2VEmHCqAw5r5SqAdJ/YV5O/YkGzX4MLWFaCe1SfFOQEfv9zOtfI9KYU5sfusf7S+ue49uP3fK9ZU9kKwEXHnFEARlShlPX9nUy35tFvbdCaCLSgbmx9T9+Stjzi8DWzW9lUj2nkZvgrwzU2F/PoCvW3arP+/7io7eY4xn8JMKKBBA24dR9edoN+5DwcLDTD8nInkVoCD31VZ80y4R0OQSQrWKeFXR/Zz6cA6owYPZy621lcCiPpirDy2SUJ7XwE15G1zVh3/1dUIuEqloX6GTbiQoqe9C7cOC+gTo1wOTejmyVLfIfvqtD4Lp66G6XKS4NhGQ1gctyftTHtdMldwnFUmpcv39ZWw2kdt7cNiHKplaHv+p1+OCJLbxtwsEqbBnfS2Qi9QQdKbDceo/400tz9MHNg1sVHjPgBwTZf3gTY33jm6kB1Bmk5DWnUm1jg7MmM3Cwlrt6UKQUX9c+/3rtp8HN5ynt0mr8liXCkD3g9dTqm5MLaFDzUyAoO483ciEz4IACD1DmJIN3tdo39GNJLXXR5+WSCqHVMX0oT7W0eURppJrCq8UpuqweTTMj1SxSEJutO+GihndOnTd4dIgB1UKl+ihLqaXsvreSxiAma7aW1AYEmh13Q2dljgg1AWp1g639xJY+VnYRZQa9oP3Q+IsRuO1HoDjADZB/Ip9oFB/PLJEuipBui6tkwqVIJtVVSNpsVrfBgD1EtrXt9dBX6gx4fUk1v62HtAGcCn3c/i6EW/ma/JW3/wOAPZ3et6fsnnvhwRa/xrr+bvd/imw/GxbIBT/fG9/VEq9B/wt7/3P/6zHflYn7PeVUn8eMEqpLyil/irwK7/nEX9O27rJ+PWbN9m0GbsmZRQ1dE7TOc3T9Zw06vjfP/gSL67nVE3McjPCecXRuADgcLHDOk3dRmx++4jT2Y6bp3OUglfFhOevFlRNzDv3L8mylrqJ0MaxfLLAe3gVpIoe+NGjO1xcz5hPSj55dcTd0xV1FTPLanRsMcazfTaj2aScnK9FqlZGNE1Es0qJIkt5neOswQOjeUndRujE0lURrja4XSxeu3UMxpOnDfmkJhp1XF1PRRIXUnDLfcL+8QxrNa4KZvnMEs0bkX7FjvSoZHKyF/+kln9kUd7hK4NtJKQomdewjVABzLjLDHeZoTJLcTmmqyNMJh84JrWYjzOwirqKJZDo0ZiTxU4A7ExWsZJnMVyl5KMGnotMLp2LBNRPO/Te4EuDfpWiGk2UWtSiIXmeQK3hKsWMxf9XVzH2OsVkFlUbdK2obzKRcVW9EQxhzxwkP8wp1xnKeOJXCW5s0ZVCbQUgm+NaAlpaMVdEVzG0iuiowsee0ScxZt5AK4wKHlQk4EtbSC4iJh9FqGWMzx2jj2MJPPLCuLk0VNA4CQcyhSZZGmENa42dOEY/yIivI0aPI0wp0imz08RbLWDOKZJrQ/ZCVi9nH+pB8tQupGqEEIAiKb3yffJY9tPNLW7kyN7P6GZWAqRqg+4UPnF4LbUqOIi2Gje2ZFeaZKmJV0aYraUZgEX+UhNtDC6GZKNAe+KNJtppspcRupXQo3gtYCEqBSThIL+U0CY7ciL72irSG0U3sXQT8WjGGzUAt2QlvkcIF7WxLAK42BOvtbAFrbplCYwk3uoW0isdwnA8k8dIynEIIsmuQjBM5tGdQrWa5EY6XU1NSNYVj2FUKEbP9AA6dCcSP1OHNFkrc56sROoXlbIf3SrincxDshZQZypFs3DS2dkKMycXzXJB2s6FHUtXMH6qyC+klkdAu9yvnw/dqKFzNlkJYOxZs37udSvsYDuG/JVI6MRLCH04ULyXi/++wkc5YQB7cBMVMH4q4zUhxCi9ViRbgjfOE+17QOkDWCbU3DCAq/09R7qSdFcVAHt26YMPVRg+l8gCg2n8AAqySx8WAW7/F+QXbqhegQD+M8iWMu5k7Ydj1eUCxNKlMEMQgodKWHzghv5RZ7jtQ3UC+Ewt58v4uTBzvVSxZzaza092Lcm03Vhkl/mFJ1164p1n+sSRXUkNjSTzElhkYd3Slbzmwx+0Q5puvAsBTzsvHsDKk66cVIaE5FibQnMA0d4P4Cze+sGPl107THW7vx609p206dbJ63GeZCMLQ7qT+Yk3PoTvhLqf0gcAosiv3VBnorwnu5FxFWdyDLOV7Fe3nnTdz0tgDh1kN47sRtQQ8U5AngtexPJI4xKYPrNEJSG9OoRVGam8eX3xwCbCZPYAyiZKvIYhOVU3YXEqSPZ091qwkhY/ZR8WlV850rUl3bjQjyySyvSmGzySysH4RTsECfUALi5EMppsHTaW4CITZKlxIcdk9nEpwL4PZFGSMAtyrmfXcvyzq1ZknVetAPBYwo+GFFcvybXxzhJVjmRjSbaWLldDQm1fx2IaR355CyJsqsS/WEtQULR3xBuRhkoQVgDZM01USLpqO5LwqmhvB++ksO7CrCbrlnjX4SJFei0X//GmCWmyWoKJOk/+osQlmvRSVj5MLcwgHqKyoz6KiXfdAJSVlw5Rua8VQNkKeI1Wlfzvm0ToxhHtmk8DSufDgonBNG7oMo12TUi2NZh1RbTuV2HA7BtJus0jXBz8mlUrLKR1EtITfJQulZ5Rl4dAn86hrABTXTTia7QeXbYCSJ0balOwXjo0vUdVASjGBh9HuDQWuWtgH/WmgM4KeAzprj5LII7wqTy3T2K5TalboNaDuygAuyHd1QmojCP50iEwKInla3jsawE8fWAPSN0IDIzlsE/n8G03sJDe2luZa7+v17+Hr+H+ofrkJ7+GnszfTYrsP66b/336+ny3+PVMHe/9j4D4szzwZ/ZYhu3fB/5joAb+FvC/Af/Z73KQn/vmnOa3P7mLSSz3jld8/+U5TRWR5i1da3ihvTBxlYEJOCceMus0TRexKwz7OJHwkrnlk0cnHD5YsVqPefVqgW81tYKPXhzjrWK+KNjucjCepkho64jtPsNbhck6/MuMclzTlRHPHh1hJh0vr+cih7TCmpF7rq6mAg5jjx55XK0pywSswlaG7X5yy0gFwKdiJ0rEVkv/YKpZXszEh1VEqMiTTBqaImZzNSYadTgL1S4NK48KnVi8V7jcobWnKWPqrfxdlQbdKKKDCldm6HWEO69olhl60eCdEt9SBPmTmErFjF4YigegXiak7+yoLnPc3KEiT7eV889UitXfvwPvNsKaKmgPHbpS7Jc5SQXbl1OitaGbWsy8xewVzcjjUocuNF0qQTSqUxB7qMGuY6KdQa0kTr31ibxOp1CZxVVRqHLRuIkleRFLX2LihWkcSXdmu1B0py3Jixh9ndIcOEiFlXNdSG3cGlqXghfmx64SVKtxE/GedYkhfymdmd1ULhYnjzTb9zzt3BNdxtiJQ5VawMMuyMScyC3jnSZaRtjcC/DLGSo++gTIPiDB1ArThPvI+ogwT0ER6XRg9baaZC0MWjMX6aSPQoR9p4j2mm7ar37C/DsJ7QTSZUxx7hj69eaeeCngsJ3KnPnXktl85ImXivoAbO5IVgaz16HLUNiGqBBWs5t40itNcxDO6SBPpFMkSwHMURH8UiEdmCCxU07hUvBa+imLc7lYTJcK3RhcJBf7+/tqSN7tA01MBc0Cxs881cltlU+yElmpqSS4xabCqppKwHpfiVIfyZjSrTBi8UYuEHUrCal25Jk8EpmfbiG7kG7MqARr1ZCo28wFbEV7SSTNLzy7t2D6cajFQcCgiyS4RDfAOMxzkPMBUi+kZK4kJVSovulHmvKOJwq9kkONAnIs491r7EMhYMRUkF+EJEsjfxu/8Gzfkg8gUwpgFG+RMD2SSKvIL+SCvpkJsT+k7jZqCJLJLnt2T2SVuBBisxfGoMsFwKQ3kvzaA6VehokTH5iykF35IWlWgKCnym/nQXk5B3b3FaOXns1DkQXibue9G8m48mtHdailXqMVBlGYGpnj/CL0hWphpnw4X+PCsz/XAwPY++ii0mM9Q6Jt769MN+I5NC1UCwE6yd5TtyFcJSTT9j2hugU3EzniwEYlQRLrCTUXwoxFpYSkZFcCbJKNMILKe7JlmPNI6l/64+eULJLEex/knMJqKSsARD6XBNx0qSKqgcDUxjsBnMnKsT8TX1t1oEXCupDnjwtP4iRFM9m5gY3swV5UyoJStrLUM7lgN60j3bjwnvLsz0LISynnWv/418NuTCPHvFckWIkQIN5DurWUh2Y4XhCqWHaeeq5J14547/AREnrjw1yF+pBo76iODFElAEo5jTMCor0RUJlddxRnsTCCQSoqPk+Z8x4kojzx1tKNdVAgeJLW0xwkZFct7cyEc0UYQRcHqfTYYBNF4oUZdJEiXXU0s2gAlfHWYkOtig9ptsoDVryHpnTUh9IHml3UtHPpoOwmBrN32FzLQsK6wUcpUWmJdg3aJrhYE+1aulFEduWGhFgw8r4E0mVLNzIkNzV2FNPlkiSrnBcJaWAUu0lMvO2GoBwfUkDlvHOBoWzpxobsQkBiHGnwnqi0IfX6tk9VWUl81ZUlbgWkxUWHMzowfjL3ugkS084Ff2lMtGlRuVSIuES6PXuW2gfGMNq3Ij8FTNHisl6eGv5v1xafRwKGshizbwQcdQrVvCYfdSGER2tUKfdRNoyxcwIYIzOks+KCVJYw7jowiqH3cgi+cU5kqn3oTttJGmtR4NNEwGrdCBDLElRRyc+5vEnUZn+7v9fDeuBToM07N0hlfd2gUpEa+KoG78LtNXQd3kcCKq39VPKrD8FBA1sZQKe8aX+CofTuU9cW/W3DpvRP9WD+E7v944+P/4FS6q8BfzP8/heAf/BZHviZU2GHB0hx5th7v/ldPfD3Ycveued/4b/5C7xczujqCN9qVGLRVwn+tMbfJPhMLkBVZvFlhGoUuta4LEgakX/YfXpln0rZV2f0XYntWDxIXoEfWaKbCBeLAV83Uj3Rpxv2rAEIoyP+IWgf1OiLNEi71FCT0F+I9TK4vrohvTK49LYHrZem9AEK2bUKfXmhg1L716LAFc2sX5kWyaUL7E03Em+UKfoIfnBp6BJsbysleone65K1buKHsIFu7sieG6ozS3ZhRNqXeZKlphtJEmTz9YL4B6OhqL2Z3aY5mgA4TK2GSP94B+2sZzc8eajK6HKo7lgmHxnqIwljaCciZWsOPi2pzC7lArKdyEVk7/GRXq8gYVTi/cqWtzULUSX30S3UBwwSQZtKOqSLGHx3LmaQ/fWdjvWhzL+pRP4YBe+U8kFeF06tdiyP7SWSfWJhD4K271rSSzMUmqc3Aoz63/suQG8gXQbfUs3g7+m7DeONeJZc6skuFeuvdkw/iERKVAvQSdbgUpGPeSMX1DaTBM/xMxlvN1ICyEo1sES9/K+Z90BA2JL6UJGsRbaogrQwvZE51a2MJyqCV2suwGz8tPdHifdMdeFC7zVpXd8b6WIZa3pDkJuK7LAdKapTmH4iwL4d33rEulzAaTOT+Y2K4OWqBdDUByrUz4i3rO+jK4+lm3T8rK8yYZAHuoiB1TGV+Ne6TObh9X5G0/hQT8HQedgnWCZbH3o1hdXIljInXgnQcbGcv8pLyma/5Vf+tjPRy377fkoVehvbKYFBDV2HTt7z3UgRb/3gF2snimQlxzzZyO02VWRLAV494EtX8lkiQCgwNI0fKl96L1qyllTQvjOy774DOQYgF9+vMzx9YqhpPMWZZnThKI81+aULckc1dEqmG8/2DU12KSAj2clc1IeK8QtHtdDkSxfSWoVF0a0ACtMK82cTNaSB2lQxe9JJXUXPlu0EHGgryZkQQl2C1zKqblMogcEj18tyvYZ8Kf936rmmmcocZDeeZiyf9aMry/a+IVvKeEzjKY+lO3F0aWkmGtMSKjDfjpTEAAAgAElEQVQEoNYzPby3XXjuyQthp3oJIwRAu3KUx4Z040K/oySH6s4PEs6eXZNE0cBSdVLpISyenDdx6TGVSCvbUJORboLHLJOU0qgSz1o9M3RZGFcmVSGm7kNWDMnG4mJFPTdkN5JG2kzEq9hOpHoj3lm6XBPtLd3YUM8EpEoHrxtqO+q5JlvaAejI4wzNTJPe2FspZ/Ak+kgNTNcA3LZO3oOpnM821cIClo7dvZh8acN5qolKR3VoGL1sBwBYnMaMX7WoLngOPUMSqrKS2Kq6W4moN4p429HOImHb9tLNGO866gMBYNr2ctGg1OjTTzthKHVlsXkk/sGdSEajQhJTnVEBKHfYWBPtO3yiaccR8b6T75s21HQI+9ZNYkzR0Y0jAfpVkLwqYRtdLpUfprLC2HWOdp4OSak2F6bQlC3tPBV2rha/n80MppKxRZt6AF3VcUZ2UdDNUumyjKRaxY6E64hWFe1hjm4cumjBKGweD4ms0b7FZhFm14h/0Xn0vqZbhKCuRsDLIB0NUlOXx+hdAxp8GqOaDh9qSQDMthLmT2thJCORu/rUoLeB0YyjgXVUdXcrWe1BVBwJExgZSW2tG3ndfWdlZFBljU9iSWjVSiSr/e9Gozo7yGFRCunDlL/3vw8eyn4LslScl8d5L/fr79N7IJNYxvfatb//HcDl4J10FvRPVIV03e3z9GDRmFv/Ywj+wVqRsfaA8zXJ608L5vmZ2x8Ej+WDB/7eX/l9kML+lc9VCpsC/x7wx8NNvwz81977n6lD/qwey/8O+HcAC/w6MAP+S+/9f/57HfTnsZ195dD/i3/9z/HB6kTCdPDkpuXDzTEAd8YbIuVIdccH6xP+8Mkn/MrFQ07yPZG2/ParO9w7WPNiPeOP3f+Iy2rC890cAKU8Xzl4xXU95uObQ6ZZTR63bOuU+9MVj9aHpFFH2UZksZzkR3lB2cVc7cZ0TnNvvuaDZ6fM5wWrmzHfeOsZP74+xmhHnrRkUcfVbswfufcJv/b8TR4sVrzaTUgiS1En3JluuSpG7MuUto4wsUVrz9Fsj/OK1W5EU0eYyDLOG6omRinPJK+p2ohZVuOB54+PyA4qmipiNKnZrcS0ko4bmirGRA6l5c3eljFff/sZP748pi5jlAZnFVFsiZOOcpvx8P4lHz054fB4S9NFwrYCUWRpLkckJwVNkaATWV06nO9Z73KaXZ+DD4cnmzBeKDYZ+bSiuB6hao05qul2MfGsHhYMousYe9bgW827b7/kw9++S3qnoH45YvHmis0uB69wNsiBjUfnnShCrlKmb64pq5jFtGS5HmPLiHjU0hYxqjLExyXWaniRYRdhad8qTh7csNrm2M7g9hFm0uEvU9RJjbtJJGim0aA90bSl28UQeWaHe7ZPZuKxTC1+F6FrSXicf+mazQ+OsHdquElI7+6xnRYp7nWMnXWYnREZ7NslXZ++uY5JzvfUmxRVGkmZdPCVP/oR33n/DRmz8hJgMw4re7VG9R7To1rY+W2M3pvgM/GYSouXctHgV5ICadMgd0u8BOU0BrOKJOwn+Pd0o7Bjx+Rsx/7xTPo5D610oJ620AbJVqvxqSV9Hsvqcy7+TG8genOH/9GE5m6L2ht86licbal+41AWXoIUU3UKO3FEG007D/5HQD0oiL43plkIa+dyCcKxwY/Y+y/jraa616J3RgJwDi3R1ghQnImPdvREFi2ivSRB7h6KTFq3EvTiMke0FYbB3mlQyziAUDUAPN3J4kVyowMIFfBlH1RwlYo/VoNPHdEyIioU7cyJr7KUyoz0RgfAKQE1LvWfkryaSqoFohCuE+0U7VQWqdKlSLRtH5SjJHDHJR68LD6NXil2DwJrHMApiAy5G/nhPdqNhWFu5158bcGrmNwIKK8PbkF0N/F0E0tyY0huQhdnL9Ft1VAhIv4oAZ7JVvpC+4qV8TNhenUn8zZ6qagXssAUb9UQRGOzsLA1vpXN2kykqs0cRi9CdUZYeEg2t/5LE8Btz3B1E8/8A9jfC7UgmsEHm14Lc9z3lnaZ/C0qZBGpX3hziYwrKiSQJqoIkmxPeXZ7jJQXgFofeEYv1BDgY8OChPRChgWYw1vvq2kYGNPyTObRZjK/+StFuhYA3IfoKA+TZ479uVzEu1gWktqJ7GPy1IfFOjlOPSDtPZkukWPqjTxf/kqRLaUTtsvVsJjTL4Z6rYaFz56BzpYSJlOeKZKNLDhUhwFk7gOjX9x6/+K9LPZ4LWOvZ4qoup1zF/egUOajPpB9qdDNCZDsZIFp8Ak6eXxUBvlz8PdGlac4DhUW4RJJWwHcxYkEXJnmNoAo3vnbRUIH+dJSHhiSfd/fKUDbK+gyTVR7dvc02bX45WwqCaYuhnqmmT3ppDbECIC3iSLZSZJpL8cdwL3pe0Y1Xa7Il9KpWR0Y4kJAdu/fVJ4hwEcWID1RJSFAzUQPLPHQzxmkrsIGO6qFkUWTXunR+0UzOb4iRxfPZHUYYZrgi0ylR3TyoqWeSzhPf070IT+mduGcUgKiDyWwpzo0TJ7VFGcp+WVDNzK3Ka5RkPM6j6kszUFCvBNwHJVWFnciHeqGGnyi8UrRTuTvfW2ICv2LygqDKKyyFnBtHc7o8NmmSVbBo9h7BBWIDFpLb6fzso9OFlZcbIR1fY19U9ZL5Yjn1l8ZOjmjbS2LakWDnWZSORIbVNlCr5gIwMsbg2o73ChFbwth/YyWgJ+ixmexMKGRuQV2wxic9F9aO4zN5yk0rfyu1CAJV3V7y16+7nPUWhJbjfgyewmtD3UjKo6FyYyMeCi1EQD6enWIDwyl1uKP7AIw7rdPAVrHp8KC+tt+YvsD4bF88MDf/8ufP7D86Jc+V2A5BirvvQ2/GyD13hc/67Gf1WP5lcBQ/jngfwEeAv/G73G8n9vmUfzyh+8CoJGagMf7A+6MNzxfzqi6mB+vjvn69Cmt09Qu4kuLCwAaF5HGHfOk5O2jaxoXcZCU3BlvsU5zNtqxtwmRtux2GRerCasyZ55WfLw6xDpF3UXsioyjvODN6Q2f3BwQa8tmm+O9IjGW2Uy8kuxifvjqlFHasLka01rNcb6jqSPeX50yyWrKLmZXZKTGsrkeo5RneTljlDWMpxXeaWbjigfTFdfrMV88vUBpz52DLWUdczTdk6cN+yrh7YMl96cr/ujpx+AU1TbF2yDlKCJM4hjnNXHakY8EwOkfjVH7COcVXzl7idIQJx1x2nFysKXcp4znJc+Wc6JXCV84vOKLxxfY2hDHliTp0Ac1B9MCXxlcYzg52HJ1NaVrJSxIFYaTszU311OqMuEbZ885Ot6SJa2sko4sk3HF+LigLSSlY3xYyoXLNiJbVHxhdgnzlqPpHh97TsZ7sqxFa4eJHKSSQurKCLuT0J/dLkMpScWdjitMZmnXKfGoJbnWeKfgRSbeQgUqlavduo0EVBZCS5jIoloBsKpVmMwO4KnbxujCYG4i8qTFh7RVpTzplcEfNtixZZbV4tVLOoi8KES8yKntRJ43vdKkS0mNHT0WPWxUKI5me0wunlMfiV9HKwfGE20MZmtg3oL2mJsINbKoRuHHHZNJRZK2kvDay4YPmhBqo/FeZLo2cwJqxxY/6jCxJbqUYmxdSWDP+P42rMor9k+noHwIcNHiUS0MqtPEVxFmp4OHNKQVWmH4o0LRvBjjI9ArSVzFKdY3Y1Qn0f4qJL7qFsxWLiJUK2O0uaNdp9iRx53WJDcapi3t1GFqSW31qSSv2kwkeL1EMV6ZIdgGkLlMhJ1uJ552Aj7umS8EVO5E7mwaSD9JUVYJGFt0uFjAgm4lYMk0Ur3QTcTjt1jsA/hUpFcGszZ0M0s3kovHwQyJAKm+m68biXrAHrZkl0beI0Yu6m0ibJ1LA5udW9JleD1pvw8JctK1JIW6JIS9bBXZlYBZEIDZjSQ0p7nfSLBR6NvUdbiotuIJ7YN4BNDJWKKdAtNfiIo82EfiL7W5H2ouXCIX+TaTC+3pI5Fm6lbRTkU2XJ66ART46Na/KgsNr/0DCKyKiwQw2ozBVytVGAI+RXrrhyAbCenx2JGApvJEFBNeS1CSC/7cdsqgYDC17LedeJpF2M/wmvygDtH2lkHuxgKoo70E4vTBLb3M3ffsY+KHsJ5k/emQmJ6xb2fh+ed2AKh9p6gwyDKmHsDszzU6MPwuCsFDVsBVdaSGapJ2JqCpT77tOzDR8nwukflzkbC7ycaTbOS+7VjRjuW1oUI6bJg7m0C2duSX8vhmqoYOT5FLy9NEpRzTvlvTGwbAI/7ncDHbs6mhrsNUAgptLsxrVAmo1F2Yj8AY2lzGqBwU51I50oyF6e63Hgj1ADbei3qp97S6SNQIvY+znhq6kaIJbHa6ccH7qMN72VMdynnUs+D9Vp2I/NbUMkYJoArhPJohxEcYYFECtCMBPv24ulyCf/q5iUtJr3Um1Io0DhfdSptdLF7oLteDNDnZueGzz6Zyv+pIAmLasSbeO1wiz9Mz3S4wvt6oIf0Y5DVs3xK5rmklOdYmativqSTxFc0Q3iNjldqQZiYTZBMtsvJd8C4GeW8PzHTjBjmsTfQAEqPK0o2jXkFKvOuETe1kEa+dhJqOUUwXGFGXmvB/22HKDrQiCsypDyyc7hy67IY0V5tHAgKdv5XFWocdib8RraWvcgCGagBsLhEpqEtCCKAxUi3Sy0NTAW8+0ngjtR/Ke9worIApJT+33W2KbEiNxQXwHKSqynn8KJNxxBE+T8V3GSSpPolFdjukrP7E9+GNoWWfPdvaA/QkQcXBT5omAioDiJQ6kujToBJ43UP5k5tvu+F+/c/Dl3W3X10nX+3vna38p9vvavt7QP7a7znwdz/LAz8rsIyVUjECLP/Hz9pl8vu9Oa/41htPeHt+BcAb+VI6F6sRf+jBUzqvmacVv7F5g1cfHjMxNZG2fLg8YlXl7IuUysZc7CcAfLA+4V+78w+oO8M7k0teFVMuiin/zMNHfOPecxZ5ybrO+NN3f4xSniTqGOc1mWlZJCVpZKlthNYe5xQaTxp3HIxLfGb50tkFNqSalnXCrk05PdhKBYbybKoU2xmeX8/55rtPOM23KO3xXnE4Kjk93DDPKr5/cYev3X3Bqs4ZjWremN7gnOJqM2ZXZNxdbPjx9TGd0/zyy3c4uLtGJ5bxvKT4eMbZ21fYbYz3iruHawCU8bz9Jx6RnhX88NkdLoopdhsTRRalPLsqxdeGh4dLZuOK7qTluy/u8qPrE2g1WjvuzjbcPVqz2o1ILg1fees5B1nJeFaRZi0Yz+TBhl2Z8q9/89fJRw3ffn6fq4sZnTWk8woUjNOGJOp4981X+LDv/Esr1EGD7TS//PRt3jy/pnMaXWgeXR+gtePO4Yb7Jzdkk4bkoArGL4Z/gM0+YbvOaa0hzRruvnWFfZUTf33N4XwP5xXFGx2Tg0IAavC1Hi72qMxC4njn9EpknZHDZY7ppITIkZ4VjE8KfOSZfeGGcSJym9nDFXHa0Rw58IrR2Z6jbI/++hpjPMlJwcG0YDoOIQazFpU6irdbbAbtJqG8b8nGDTzcU7URB/M9fpPw8FvPsFPLbz29RzJp6E4a7Mjh9xGHx1vG767JpxVubMEpOqsxxpGM5O2rLLgywo0s/kGJUp74vQ3qoKGbW/H31obpuMK/UUptQwf6sCE2FrdoGZ3v+IWf+zHMW7yG2RduRFZ90MCkpZs54o3i5HRDs/C0Jx0+dfjcUj9oMCcV7cwOn0R60uKdoj52xAcVdmqxRy3NsaU7aaX3UXn8xOLHluyopDmy+CJi/sdf4XcRLoLqfiNVIp0k27ZHHbOzHW5i6e7XLL55RX3eyYV76kB76qNwvtyt6HLPN7/0GDdyUpGSOuxpg6kV9f0Wm3mSh9tBHpm+saM495TnHT6zVMfCniYrYYPXW0FE44drqjca7NwSL+qQ4Ouoz8JYjhuaOwKOXerE9/huSf5RCt/cwKzFjh3dSMCIHTtcHIBJakWaDDBpsbkTFjeC5tDRjUXy3s485d2O4o6jut/Snsr8ukQunGcHhbAvx+2t1zfx2FQk+u3UUZx7mq8Usu8DJ8d83Mm4Uk87F9BcvNEN7GJ9KCC3mcvCRjPz7O5LGm47c9QHAkKiQlGfWpq5XEgX547yzNPOnSyClLK/5rST+pAjR3XiaRae6txSnirKM5Grdoct3UjCcpq5p7zjKE97Ob9i87UGr6E8t9jc0x52lO/UVKcWm0L5pYryxFOeeqr77S0bXCt27zXC1E4dysLuDU879uzvu+DZBZyMr506moU8dzuThYv6QC60XCQy7HYKuwcC3KsTN6Ty2iSk+J6EFN+NLAJ0E1k8uPmyHCObQHHeAzIo7knoTh9G1E4ltAkEoNULWWCoD2SubMKt7SAAo/74V0fCVO4eiJ+6mcvt9aEA2S4sNnglMnivYf3QDEywyDtFglwvpMKkOlTy87HI+SsRGrE/M7RTJawzwk66RJ5TB0m6SxSrL0J+JRfINhYJeT1XQfod5t+H1F8TPJA+AOqRVJxs3lZs35APn5svSADZ6j1NvdBDWisamoWA1uooSLdLTzdWtCNYv2VoR7fs7+ahYfI0LEgFcNnLlcfPpa5lf25EVm5FemwTRXUosuBmrCiPNcWppl4I8+diRXWgAsiTearninqq2N010ncY/KddKjJqFNRzg4sU1SKEGWUS/lPP5PX1+/Fakd4Iy6ysMKK7c5EhR6WjOJXX2MuY25GW+hMj4z54X/yh7UhLuBD8v+y9WaxtW37e9Rtj9nO1e+327HPOPbd3leMqx3ZMORCjCCmEiIQHhBR4hxfABEVAyANBClIQIuIFkFAeAEuAACHxEimQPGDhkMTYplzlsm/dW7c77e73Xu3sxxg8/Mec+1S5Uo1VFRyLKW2dvddZc812rTW+8XWyv2ONCxXR9r7KpJkGAygGAYJOgck0JtEUxxE21vJaYwmJK49jv56my2W7NpL9rvbF66o6S72IaOYh7TTExppmEkno0CyWPsdA0Y49kE2FtTSZyJLLgxgXCajscpEF2zzCpKGwn9bJa/rXamcJNgnRrUiDAbpZhksiVu+NcIGmOcgFVHnmU1mHGce4LBJvp1IS5OPBp9MixbajVMBt00kqbBRKaqyS+9GO+shuBVoNYBgtXlSsvQ/g6RnLSBhPEPayO5gIuOwZxF522y99UE4PCKNQ2FHn7n+MQWWZgErlWc3+ZwChEgykAo2Ko29/faVQUSjPgeF3Fejf80MQoML75/5jv7gfw8+Pdkmdc9thd+X3/Hs8f1h+UCnsLwF/Cfg68M8DbwD/nXPuF39fu/sjWuZfOHInf+3fYJQ2HI623JY5l9dTwrgjigyHkx2rMqWoJGhnNKkYpzW7OiYKDEWVoJQAN2sV1mjm0wJjFcvlyF8sPyscCYhY3owJ0g7basazkmKX4hzSa7jMpCLkckJ4G6LeKFDa0exiqeOoA1RiCGNDe5dAZggSIzJDn/5KK+EzOrJSCaIdWMVoVrI9GxPMWswmGp5j6oAw7TCdJowMXRsQJR1KOerbjGAizye2st83wrQEx+UgA3WJyDlVLmDCbUVKSigDu/nhluXZFF1o3KIlOI+xqUgDQVis8EFBsxX5q7OKMDZ0N6lI716EFG+1YKQ+w+YGVQa4zBBNGtpSklfRSEhR6affA0e4DDFHDc4ooouI9sR3H9QButKSPBt4NipwcjzTFr2MRD45MhA5CccZWVxsUa0GC+lFQPlmg96EA/shSaP3lR7hOsBkFhc5wk0gLGSraKdGfK2RnIPwTryLNnFkF5pqX6o2JADGDw58b55yIidspzLwCxoZKCsD0VpTLyR8phtJKmq4lnqQaKsltdQIE4UTiWNYiIQxudN0mcPkMnAPt9rXPSjqhR3+H15LV0yklzG7kgGmMFIO1TJE+ve1BN3IDcmlkvZ5X3zeh/30f/fSuGYu7KEy955dKVoXuWi8ht3jvjNR6j6cErDRJ+IKY3QPcMKthO7oTjzJ9UKkrOmFHiScPdMTVAz1Jb3Htk8x7X2xyrNMkhbqWVUnbM7opWJ3KtexD0Pq+/F6f3K9J8+TuoY+5ZMhsr/34OqG+8Ai5RkxK5JQ1fUMJQMboIy8dlj44JwbQIsE1YYivezPvfXgo52K7LGXb/Y1G+J1FS9lL+nr6z368JheYoqS/UjuRFoa1HJcIMfWB5X07JQNGVJljWfm+kFrvHLDoLrL8ddMttXLE62v3+iy+0CbwB9D63sWxc+rBuamHUkoTT0T6V965agOpX6mr45oZnKM0Q7xjO5pwsp5H6QP5qqdT82U9eKVJJrGazf4Q4d7JxOvaT1X3vPrBkZKAn7U8J6RvkRZLyzlfmwnUk5vAwF2fRhNcievGW/cwK7FGzf4nvtKjr6MvveCB00vgfVMkhZ5qOvrMAr5DNMdQzWJC+Q4op2EzrS5Giox6qkaGKHQV6447ffVp6AGtX/uzAdNRfJYO1LDOtFOznWyEoap99H2nvI+UKdLhGmUGg5hp7rUB1kpf88p78eeyj43YzUcf7y1tCPZZ+vPe7Lq/ZwaEymiUgbAwjqK1LRnBHsPY1ha6plUJSVrJ39PRdLaV7QoJ/vWJ8g6rTyDew+YAA9woJ7L3/HW+ntHD1UioWcY5VzKNRKpL4OvPCwk8TTamWE/e28peGa3snR5IP+OtL9G997YvsNTAtCM+HK1oss18VKSW7txQFjYQWocVoYuE7DkAgEHNtaEhRk8ovSvrWXCwCnpi9RdH3Ll7lNekRRXG2i6USCBOj45NWilMsRpRbgVH2W/2EDYSGEs/RislgCd+6oTeTxc1dg0xIUiV+3l4DbUON9BqWvpl1TW+f5IkaYGVSey1s4f72sSUWCoG+nDe3Tjpa3GoetWkmDBh+7IvzYV9lRZK2E+CEMJ8tggOX0t5KcHcC6OxJPZA7zX/ZG9XLVPWbVWPJdaw3frsvyObbgolOd7KewgV+2638NWOudQfcBPn+jqWVnXiSRYhYHUjLQNRPF9HYmRfXA+zIfeW+lTYb+vr/L1wJ7vsjjrRHL7+1z+wEhh/+0fgxT23/neUlilVAr8n0CChLf+L865//Af8tz/C/gl59z/4//+Y8B/7pz7499vP35QYPn6hh3CLwTOuf/g+678Y1ymP3Hs/uwv/ws82+wRBQZjNc/OFszmBe8srvno5pA/fvo5gXL8rd/4Mn/8y9/iG1cnlEVCGBmaKuQnHl3w4YtjfubN53zj7AF7k4Lzl3v8a//Er/I/fvqzWKtpapGHpmlLU4c8Przj6cU+Wlt+8vScz+4WfPHwgt8+f8A0r3g4XvHVzx8znlS0XUBVxEymJe/tX/FiM+fyakqUdnRNwHRa8s7imk9uDzBO0XUBTR1hqoCfe/9zfuv5I0mdjQxp2lIWCd06htgy39+y3aVDrk9Xi9z08NGSqg1Z5CWvbqeYsxybG3Teoc+kYiK+CWgn8s3jRp2kwu7XmHVMOG1I0pbdKpUakk2EXtTYu4T4sODB3pqnLyUp9/jBkovPF6hRh9uFqKGoXqSUBA69DXGhI9xo2rlBT1q++Oic3/nwEaPDgt1tBp0mP9zRfDxFvVGIlDhpKS5HIt00Cp112E5zeLTm6mwGWryaq/UI02iCWIC20o7oZUxz0AmIBJGfZJbkLKQ+Fdnt/GDL6vkMXSnsYcN4VrL7VDy2fW8j+7X4Us/GhNsA97DC3saoeUPwIqU9bKXnswzAKeKrgPRLS8oqwj0bwaMSe54KyMx8p+V+jblKpZfTKAHw2qE2Icp4EJpLIq5ySDdmqwkK6Y8kksEKjUY3ApJd5MTvaRThKqSbGvoC9nAZDF2DNjdoD9xtZohuQwllOmoIbiPsQQPL+F4i6T/jbSwVIs1CPJY2l3J7gMP3r7l4NSe6jGi9x9KkFhc7omVAdqHYvGWIl5pu5IbeRQKHapQAT8trfYEOXSm6uSHYBJI8uxbvoI0lmbY5bQmW4QBgnQKXyHpBKbUeveSy9yIG726xH409KFPUB4ag0MOAGysAVjfCYLlQJkX6GovqSMC5STzD86Am/jwVIG5lsqAbOV8d4BNnS0WXCbDIzxXbJwaXWPLPI5qZI70RJi3cKXr5q8kc6aX3KXop4uSpYvNEJjz6nst25AgrCdwymaOdyOQCVryDfR1Kfi4hSX2/Yy+7rPcE9Atjdr/9bmSJNnK+kxvxb+rWh3758+MU1AeW9FIPZfY29j497+er9yzxUkuY0773WDoPPmMBmD2IbqbCrKVXjupA3QMqJ+xceqUoHoh0dfLCsHwvoF44Rs8lxEsAgk+c3Qq7F22UJMGOGPyA9Z6TvtFYvJfliSM7U5QPHMm1TPYEpfdD1iLHza6luF7eED6EyEJ1ICBv98gw+VRuXglTg2Tp72XPAvYD8XglQVy9ZzUs5Vwkd15yGgmIaEd8W8hVULqBJZx+Lv5JmXBxw/u0mamh01N1MgGRXt9PLlQHMokReS+ibj2TmApLKhUxbgiTCiu5LvFKroVIY+Xa55dyTvrEWqnxYfCoJnfCZJpEwHgPEnuQXB76NNxA/JrVvkw+2UjW/fZeUzX0Oqa33jM4EUCdLN0g6exl0Lq9nxTo04J162jHfcWLGmpnwkImPKKtgNewFNCcrNzAwEalJOS6QCY1hP3sQ3IsXaqHZFEJ+RLpbH51P5nhlCIqJUgJ5DWbsSK9E9mqDdUQ4tTLYbURABVvLbqRSpBe5ttPPJlYyessBRyaSBHv7AAo+4Taak8CkGwgEx26tQMY7D2N7cR/ADtHWDmqPQlY0q38Lh7LftJOPi/6oCUTSxVLdRBJYm1hacbiA43vGtppJN2WqbCSyngAPw/JLhtM6qWikUwOBqUEN0VbCQzqstB3ZgpgDap78GESTVRIWJELFCbVsl5tJJioNhK+tLuXULpQD1JS3VlMGmLSgGjdiOx1FBFuW0nMHfkAACAASURBVFTdDuE+urX++0kAabOXEN9W4Bwmiwi3jbCC1tLOUqJ1LRUn6xqXvsayeWCptzWEgQBREMDrAWmfSqusHUCfTWP0psClCfQdmD4kiD7Ux18/+X4NUE07gOWh67KQfXZ5itqW9/v1OhbwPkYVRQIMwwDq5v7/XuuUVErh6kYYz9eTXntpLOC6DhWGAir7ZNfv8E/2r/Xdlu/EKX8oPJZ/4ccALP/d7wssFRK+uvUq1L8L/AXn3D/4Ls/9Y8D/BLzyDz0A/rxz7je/3378oJzy9rXfU+DPAB/8gOv+2JZHyR1/+fRvsbQJ/+nzP4NF8V//if+W/2P7RVZdhlaOeVRyEG34l77y68zCkixouarHpEHL882cx6M7Dt7dclbM+PLpKz64OmbvaMPTasEiL3lresOvfO0LTI631HVEFHdkYctPPjqjMQEfXh4RRR0/N3vKq+2MuyLjhZsTxobH8yWd1WzGCcZqVk1G0wV8+c2X7CUFv/LVL3L88JKvv3jIlx+95OV2RhcGLMYF1+sRX395ylvHN4zChlAbPrw+4p2TK95975oPVsdkYcunbcjjvSWrOqXpAlrjo9q15Z3pNV85/Jy/O38b6xQ3yzHtrOMXv/whzzYLrjYjlII8abhdjbBNwMGjJU0X8LMnL/haeEpnNZPjJVfLMW7W8JMn51ineevRFfvpjlWdMfmJM+ouxDjFcb5h1WTM4pKvfvyEJ4+ueXE9Z5zX7HYpqtVkecPntwumx1vaLkAnBhdZ3ju45utnY947uqGxAbsmpkhTTh/ccXU3YX++ZbnNiQLD3tGGug1ZbXL+6BvPOdtNATi/mmE3Ee3cMj2WntKu01KrYhRHXzkH4MXFHnUb4nKDa0KSvKUqY8KHBaYLyHP54NpejCllGgX3sCJJG4ogIooNzcyQTGuai5zjd264WY4JD1s2L6c8eveSq8AxG5VcVCF6HeJaRXa6pdwmpA+3VNuEeFrz/vEVn90uaPKQILDsjQtevVxg55Yw7dAXKaO3VkzSmqYLicOOsokYJw1nNwKEuzqAMiC5DmjfrljMdtzdTEhGDcdvb3h2vuD4cMXF5Qw17nAOHh8ueR4vCK9ilHbohwXBxyOaBy3J45KmDomTjt11zt7JmvU2w1YhTx5ds64Sls/m6EXNapeRzSu6q4h33zvjkxeH6MiSZQ27NIMvFARFQj0J0JGVMeNtLB7G1MmEhGeK9aJhf77l6vkeutCYiSGa1TSLYFAP8KBFv8pxRzV2G8lkhgFGHXbsMGUoXpZHJd0uwp522DIkANpHNWwj/tQ/8zX+1td+CldJBUw9FpBVTwwuNyQvYt78xad8utjHdAHh5ylu3NFVEfG7a+rPJrg6EBlubCF0dOMANWtQGrrLBHVc4YBuFTM63mHeBnc1kgHfz24IHBSPQtwypp3C/tt3lE1EFhjW4RwXeinttCP/szcsXy2g0TQj8e7a3GJqPQD0sFTU+yIrdrFFpx3qImH9noHMoO8i7LQjuo6EWe3Z5tAR/8Id1VcXtBOLOq7pupTwnS2NGQ/hM/FSUe870p9eUWwT1GVC8VZLfBVKz+V+Q/gyEalWAMooqoct9YHG5YZ2Iqx+tNbYUKSE9aGcv2AjzEX5UxVuFeOUI1r5ECajWP10A50mLEJe/mlLeKfpFi2FiYRdPmrJPo+p9y261R7UwO6JkZTqDNIb8YjaAKrTDvdTJd1lTnXomf5HDjvpiK4iorWi2pdO05uf74gvQvGy1sLgtRMrSgGlSG4DqkM33J7tfkdYRJhEwDcOTG5JL0P407fUH+0x+xYsf9KRXmjKE4uJNSaDZmbJLjTFk47sRUi98HLQn2qJX0WYFC6/4ojWMtCKNmrwakqgkngWswsBas1MmHdJVL2XtZbHEmoVL4W5b0cCaPrBqUkddicTMlUoSc89m97lsPPgpx0Lo5ude1Z6JOvGSwGPOFi/a5l9FEiQkmewy2M3qANuvwSLbzgq3/0ZlsLuJneONlWsf75i+hup+AbzQKS1nWN36plOC/HSCaHSCkB0yoPEPeUDnBT1gteS1OVaOSWA08SvJVAHXh7a+0VTmeQJSwkdipduAHbbhwHTp5Y2U5K4vLsHyef/pGL0wicN76DNtByXr2sR76r27KNMDAjQFeAb1FAcKcYv9ZBOnKwsTqmhhiraSVpv0+nBY7ydBkQ7qebZ+0gShpuxwobiaVTOkd0IM2wSAenxxjPLCYDCbYTR7jJNfez3p7lXFZT7wt7W0+g+5CcL7r2VSuS/LtA04xQbKtI7Q9BYikMNDhItIUX2QewnYyzNVP4vbQQo13sh0VaO0cSKSMmEQb0IfSeyhPy045BwZ6hmIeltSzsOUbHG+BTjoJb6k9eBbDvSpDctzsl91eWaoAro5uJVtIFPzM4i0FDtxUTrTjybmiGZ2aYR3aivMNG+CqrFZBEmk/Tdej8l3EoPpo0CSbPNYgGtkUaXHWaUoFsjCbmbWpJjkwibx+iyxWYhymWeOdZo00IWS5hPE/iUWgFtLgnQVYdNowGk6qKGzmD2JhIG1BkJ9OllrUrdhxA5B7knLV5LfEUpSae1XgartYBav55KPHhs7t1yr/dTKqUG9vY7gbD6PdUnrwUivQY4f9gmiz+Ii3L3qp5/lIuTk9fjucj//MP25C3gZxCF6r8IfOV7PPfblh+6bgToY2j/d+fcn/yhV/4RLqP3H7g/98t/jnWb8vntAudgf1xwtR5zON1yu8v5px59ylk547e/+Zhg3GHWEapVTN5YU35zjnlYYSsJhWlejMjfXGOMJgoN200qia6fz2HeEr5K6CZWpKMa9DpEHfoUMQvuLkZ1ivHbK9bXI5JpTduE2CJEZx3OaOK8ob7JiBcVxmhMEaIiC6sItWgYjSu26wzXaUnyjO/fXGHS0W5jYe7KEJUYgrNk6H/kVYp7UJF8kFE+aYlnNe2VeG9Vp7CpJbkMhHUyPk1xZIURbDXpfkl9keO08wllIvUMNoEkpdaavUcrCRV6OWd0tKPYJIQvE7qplcL34xL7MiPaaOp9Q3xSUF9nxLcB7cwSFJrgrS31MiU5i3Bf2KJ/d0y9b3CxI74JWPzsJRefHuAiKymxfh2tHc3TMTbywSKZJd6vaKsQ1pEkaI47aDT585DqSLxPNhFvkotFMusSK3UxsSNeCcNTnXSkh3L8qlHYeUf8MqKdinwV5fsOR7JOvW9IzwPKd2uiixjz0Kd+anCpYXy0o/nGjPZJzezXUspjR/2oRa+EZXOJwym/X5EjexXSTtwgLUzeXtPUEe4sHVJJlQF3XGPLkOg6FG9Xp0hutIRwbEWeVh36TrY3t5iPxyLFmlji4wL7yVjqXsYOm7hhYNocGrnesSW+DIfQl7DQZBeK3SMvTY2FmSvebEnOI/HsHcmgF7xctlaY3HoJsPcRTQ2qlW5N6z/KbATqYUny9VwYwJShmicslVznTlEdd4SrAOsBjgwUlJfLOenlW0u9Dj5UJrmVihTtGbZuIt64/FzTTJ0PqhF5WrSVvs6gkoFdM3c0+xZVC/PZzBz5K035wJJci1S5rxOKlyLzqx43TH4nZvumgQCylwHlQ4Oq/fWZSLVNM/H7uxJGM72RQJGeWe0HvGEhstZwy8AmtRMBHsmtJtowDML7Spt64chfCTsJIlstPXMnTKGXMnuWpk+dRUlwUXamqI4czb5h/HFIOxHmrd5zxEsZLFdHwjibkST0Jrf3A3L8ucQJKycSXdl+dWDJLjWhTz/tJda7U1/lE98zhcriAy7kWlaHcr2irSJ/pQYPYTuF7Fx8U9UBpDcM91Z5LGFKPRPdh/4oJyCkXsjxdCOpNNo9lGMs3pBKo3YE+bmwhPHqvm9S+jmF1WvHEhrkPBsXlF5+C+QXItUuTtSQ6Ko7YQDbqWX2kQcia0e9rxi/EHDSzIRVW/6k9JKGpQzme0lyuBMWs/SsbriDdGnF1+ag3pf7PdrKMabXcgy9eiFaS7BUM5Fe2PxCWLWemQxqqc7pQ3CqhaS69nUkfadkemvpErEvtCM1AEa8DD67sVRzNVTivN776QIoDzXRxrOIu/ue0F4GHa+FNcPB5g3N6JVPTa0c8U6AeHGgiXa+s9HLafvngB/4K+kyTVbiExxdGOppDzbuwXJ6Z9k+CEhvLbuHmtknUuPSJfdJxiaW+zEqha1NlnYIqOm7KJONYXccDEE7Ir1VXsVgacci0e2De0YXRoKTgGouDKELJMV1dN6yPY0YnXfc/kTE6MISb4wPtWGQ51Z7WiS8hcWkimjbs7q+Ysdvt1cH9JJZ3Vi0cazeShi/bO8rWFYGFyqquciDRxeGZqKJN1aCgaJ7uWgfqOMCRbxpqefRUONSz3y9TCKBPM6zvMr0qbSWeh4Nct+gEpYSIFpL/UlQSc+kai31IiG5rWmnMfGyHnoy20lEuOtQ1tF4ZtRpqUpxocZFGhtook0zdFdaX1HSp8W2s1QUBcuaIcF1VQqIqltcHNLME+K7Chf1oC0kuCtwWYwZxQSbyleCGKlGqTrMOCYomnvQ5/s5MQ5d1LgkxAUB2teeqLJPtbpnCF2aSA1JGqHXhYDBtpPakqIapKouiQUEdkbCeno5bA8atwWkiciJewmtUlDcs5YDHrBu8DXKBQlxG8EjKn4tCKiXv/ZAspcRW+clsxYV+NRYgCi6B4zDtr679PU7sUnPkMI//oxl+ujHw1h+8u99/1RYn+76m8C7wH/pnPtL/5Dnfd0592Wl1J8A/iPgrwN/xTn3le+3H79fYLkH/Lpz7t0feuUf4TL7wrH7yn/1r/D25IYXxZzOai62Yx5O14xDCer5jZdv0DYh//TbH/Orn7/N6WJNHjWkQcssrvjm3RFaOc4+OeT47Wt2dYwCZlnFi7MFAPsHG5ouIIk66jZk83xKsF8znxZM04pPnx8K4xM69mY7luscB0SRwVpFW0YcHK65W43Yn2+JA8PLyzlx2jHOau6+uYAHtXgns44o7QgCS3mdE81qujZgb2/L7fmMfFFQ3OYEoxZ7nRCfFJJ2upPRZTauKW5zVGQJz2N4c0dbxASJQQcGa0XfMxpVrO9yMB68OuTxWcluneJ2IaOTHbtlRjatKC9zwnkjXtRGJLcEjmAdoE9LdGCpNwn5vKTcpBwfLzk/n7O3v6X++/vUXyrQWlI8g3EnVR2pgVYT7VW0dykklnRSU91kHghadBEM4D1JG4p1KvtciUQSGHyow6IdhA5VBmBALRrUWYo5aFAb+RZ3mUUVAfqwwqxjlPd4Ou0Ipq3UllSSHDe8vvHbCJz8HjhUpWHaoq9izMwQrAMZEDcCfMOd9n40kaHGV+G3pQSa3JJcBdTHBu3lp/phiXue040lFVW33g/mJZ26k9oInMKMDelFKLLGSio5XGQJlzKj2+5JRKWqxY8a3wY0e1K1Ea0lJMdFbpBGyyDfyyGXAlh7Oaw+rjC3CeFWo2upycBBvBIGqh88ushPaynoZoboTio++rTTXqZlEpF1Nocin+0rO+K1ojySfWnHwtrV+9anawoAa8eybryR8I/e95ddKsoTkYLptk96dIRbLYzDvgz4o7UARoX3CnlvXu+P632xYeWDTBKp8uj7K4Naov0l8VTAWvGGIbrTBKWi2RMg1UwkabWXf+bnMnDvgVM79RMGCvHsBvc+1/48jZ4rqkORbvbdreXxfQVHn4Iar2H3UOSnkpzJ0O2qOkivZdvC/DiSpSK5FW+i+FrvWaR6T+Sn5bFMrGgfBtJMRUoaVAioyO+3n9xKD2k7YkhglX0S8DaAO38/mUSA0bAoH3gSMfjDTCLVFSBhKyhZp/drdhlDCivcyyH75/RVJn11xSDvXDJ4LKt9qdOwofR3Fkd6SCbt/YlDZUQI7VTOm6Styj71vbDRjqE39f5ecT5RVEBjVEgXZ3+svW+zrzJxSkJpxMvpGa/NfW1FdSCv03eWWt+R2vs+cd5P2XsVQ/FbOn8cwpC5wesbViL11J2wZiYROWr/mn2liLJSFeICxJvpE2hlG94vGwoAk1oJPwgM8B7M+7FGX/NhYgFfIN7LZiTMW1DLvaUspEsv1/Xe3f5zJll7ADVSwz70Xam91zVZOsJa6jr6c4HDV29IyI1uHau3AqKt89LUez9m3x0M/phLOb/JytCOPRPk+1JFJqqGkCATK5K1QdeOZhp4b6sd/LJ9JYiN1MADaCMVHiD7GlQWk4lXPmgszTQgqMQrndxJ92fv9e1rQkyixMeZ3EtndefQtcWkwSAXlnvL98n2lSWdlyG/9p7su0N1K0AwaCyBrzEBBrDYzEJ064g2ne8DlaTX3nspabLim9SdB95FS3mSEq86nBI21SmRsw6+Tr9uWJihi9QkevAIhrsOk4fo2gznVRmHTYQtdFqYOJvId4xu78FWD5CDopEAHC3MblCJtFRqn7w/0jiCXS1yV639vinp9ixbCDWqbmkPxoR3pfzdGvFeJ4FIX33nJSAAtekElEWhbK+XuHr2zmktnss4GvyTqqxxYSBS1747E8RD2Utf40h+94/3wT6Df7PvuOxB3WsAdAB8oZ/xcA7XtKgkHmS8gDwWBlI/EkfisVRKjqf3WPY+Tyuprj3z+F17M3vP53db+nAgZ/9QeCzTR4/do3/rL/7IX/eTv/QXnwLXrz30N5xzf+O7PVcpNQf+V8RH+Y3v8v9fdc79jFLqPwZ+2zn3P/SPfb/9+EE9lr/NPQUaAIfAX3XO/Rffd+Uf45I8eeRO/9q/jq1CoquQ9qBDFwE2k5suPYvoMifsR+DoJr7kfX3v7+hGPtVuYkmuA+pDQ7ARZqUbe8aje00ic2TAQXSnQUsISDfCa+ORwddMvpTjlcws1gvH9FO4/ZIjO9cy+xlBO5PBIchsd5fjmUKoFyK1Gj8TSZGN5cM5aBT1viW+lS67oIVmIoM9p2X9LpMBWvlQWLW+zF4G7Y7iVAbCr8/k94OceAU2EYlTeu0HGUuZIc+uHav3pFogLKE8EQbHBTKALE7cIEnLL2S2u/jJijce3HLzd0699EYGd+1Y2BMbyXaqfRmMpreynvQBygBQd9IPp31uTzMVJqUvvO9fS7fiLcvOZZDWjmUd3dz7nOqFADPdSEri9BPpsJN4eYYeORmYCBvVTBzj5zLoqvcci9+5LzOv9xT5uciOXNjvizBu0ZqhF7FnP6oDH/4zlmvW3x/ZuQ+GaXxAh28K6nKYPjXcve89XJ7VTG4Um/c79r4WsH3MwM5k15bOR+z3gRlhIYmM5bGAhaAB3ci+J3cCylbviXyuTxzs/UfKCvukWxi/EHZDGdg9ckw/84mNPrAmqKE8km3UC7k2QY0wSA/Ey9azb/HaUR6Kp23vA98/52D7hoRYTD+3rN/SJDfC2Dl1H0bTpTB9aqnnmnomvzdjRbWviAqR2w1VDUr54BxH8UAxfiHvo+JUkZ07okLSC7tcWCYbSWplWIj/zsQw/1AGvcWxJr+497e5QBggNCy/4Nj/mmLzREDj5LmjOLoPhXl9/yXMQ85Zfi4MiQ0gvZFzEjSSEooWdmrzWA8gqssdo5dybfpjDBoBB8v3NPOPJNCkOBb5YjvxXr21SCSbqQTTgNwb2yeQXimaOcw/srgArv+o4ug3HJtHmtnnhnJfmCGnYf1EC9MzknAf3dx3ARYniuxKmKZyX9i2ZuYljRPF6ExqGWwk+z06M6zfCDGpvO+VEVAa7eS9ll3LxEV5oClOHcmtkvN1quX9NFWMzwzNSNNMBRTWs3sQnF8bdkdSK1MvBAwKIypAymQSoJLeijwvqGH9Lkw/ZgijqucCWuK1vF+chuzSp6t6htbE8rkbbb0/LoDsxgxJn31KaXZjWT+Razn7zLA9CZi86igXAZNnDcVxRJcp8mvD5lFIcucHcD5oyCRenreV7ZtYMX1uJATHB8HUk2AIbul9iiZR/rtDQC1KgnPquSK/lKAaG/v7ZCeAoe+DxEkdRrI0NNOANpP3aXpnxJvt/XX5ZUczFYarSxXzT2qK45gu9R5B75HUtcOkmnKhye4kxGb1lmbvW2Y4T8lKWLmwFFAgwTqKdGmGigRlYPMoZPyqEzYVAV5dFlAchcw+qeirJspFSHbdiffOs2Odl0hKbY8iv2wwsaaeS6r06KylywMv65SKkPyiZf1mxPzjhvIwIrnrKI4jsqsO3VrpOqwtzTy8D8QxvoamsZIeOhXw1WWaLtPkFw3VIiJZdlSLkHgjA/xmJkmkTkF+VrN7lJIsDUFtCHYtJo+o9iPijaGeBSR33X2YTWVQjaUbhcSrBhtpmnksoMo4gtLQjkOSuxrVGopHI/JXJd04GoCgMo56LyK97rslNdGuI7qRL6by0YR41Uri6jgkrAzRTUF9NKKZh2TnNe00GrYZblvCZYmLRLLZLFLi24rqMCN7vqE9zNG1GcBftKxo9jOCUs7t4FvcNNhQE64r2kVOuK4wubCCGIeZpeJVBHTtg3O8n1KvC8zeaAjaAcDI56wZS6prsGtQTSfy06q7B3dthx0ngy9TdVaA4XKLy1N5/rrEZTH6eoU52hvCfXTRCBCNAgnzUUoYR+dQTSuMZN3gwgDVmfswnqaV3+MI2g43ymC5Ro3y+/+Po/tQnh4U9j2S/Xb6Dsq6ETlqmgxhQX2yq7ypFO51+anW8ncfwONDe/pk2eExwJXCnCpfSzIwmX5xxqC8lHbYHt/OkH5PYAm8Hurzna//wyx/yIHlD3VsSqm/AhTOub/+Xf7vbwIvgT8F/CxQAv+3c+6nv+/r/oDA8slrf3bAhXPu//MymfkXjty//N//aWojXZChtpyOVqzblHFU89vnDyhuckYHBQfjHa3VrIqMJGrZFinvHV9xtpmQRR1XqzF7k4K7TU6zi5nvb6V/EjiebVhXCUWVCMO4GnG4WHP+ag+dGEbjirYNOZxuOb+b0JYRJydLru4mmCLk3bcuKNoIYzVKOb6wd8nv3JywXOeSGltE6MgSJx37kx0vXy2YLnasb0YcHK9RynG3GjGdFKw3OQ8Pltzscuo6ZJzXJFHH+Wf7zB+u6aym+miGOW6Y7e1Yvppy/ORWPJa3Y+bzHXFouF2POJxtccByl2GtJo46OqNJog5jNWUZk2YNedxyuxoRhJY0bnk4W/FsOScKDJtdyttHN3z49ITRrGKSVdwsx0RxxzSvqNuQMLAYq9hsM+Kko9zFBJGhXaYEkxZ7F5McF7TPRrgATr5wycXNDGuUVHXsYrJpxeO9JR9965RoVtNWIZO9gs35hGhWEycdxvd08o0J1UkHsSUaNbTrhHDcYm4SXC4sqeoUp+9ecbvNqXbxvSdwncImwqWG0aJkt8xQgZVBUeCwnSKILN0qJpw1KG0xZznRwx1KQXWbMjosUMpRbBPitKPaxhwcbri5ngDgjCKflxijaaqI+XzH3fkUrGLvdEXThexuckHRgSO4jQgeFczGJZsipW0D7F2CSw2qCIiPC+plOoQbYYR9VdoRxIb9+ZbLz/Z57wsv+fjV4dBnGj2PaY46VKdh3Ir8WgGdIprWWKuxdwnJVcD8KxecP1uAU4TThm4dSyrvoiHNG6qXYylGf2tD8XI8MPo3T/dQ0wZnNGHa0pYRKnCwjmDScnS45uLVnGAZYsZW5NhFJNcg92FEoxbTifeGdUR4WKK+NaI5NETzCvWJT3B+b0ezjaHWBDsfWJRbCUmqA1RqcK2G0EoA1PM9gp0WmbdVJGchzdziQoeuNMkbW9omlBm1sxS7L4nI5kRSinVscMsYrCI8Lmh3MaO9kt1NTnQVEm08expbgp3GZg4XWlRmCEKLuU4Id1pYZQDliG5C2j2DroWlsIl0jJrc71ftQx1ahVm0wrpXGhc4oo2kB/dy2m52T7dEy4B20YF2ZJ/HlI/Ew9eOHd2BBFBFzxKaQ1lHFxo7Now/itg9Fva42TeSDB058pcBxalQrUEtDLpNJQk5ugtkwJ46zFSuoWoU6UVAO5UAItXJes3cQuCI7jQuEhY7vRSPYXwVyGTNniFZlNS3mUj596yEOcXiMWzmFju+VwOMnym2j9wAMHuprnL3TLoyEuAk6cTiOcSz3P1rRmuZQGgnzicMK7qxMMIAXXYfBNSNfZ/nTiaprK+fC3cyAaeMIn+pKE4FQPehTdFGvJzptUwE9V7J8oEhexUMbKRJZELMRjLZ4UJhtuOVGiYkcDLxYCNY/A7sToWdbv1Ep+pkwiteCrNcL3xolIXyzZbsWUR2KexVdeBZxI3vn1SKdipezLCUbSkr5yaoZKJRNzIJJEm4UBwLW1/vO6bfEpluH9BTHViyczmPzdwxeiGsVHHqmH1L2MFmJsC9HcvkZh8EVM9E7lo8UMy+JRJZkYAK+O9ZbmWk0sT5ntN47YbJtvJY9jm7spSHmnAn8mbAe1QlTMkkajimfjIo2ggzbFI5D4Mk1nmJ9likwcZLhUGe0wf+9CFH7URCmUR2LBPIex931NNAfJp+Qjsq3JB0qjr5u0/aDUtHPdWMLmWCocvk3JjIM+A+BbmZKNI7mcDrmWMbStBPuZCAIUlI7sOF5Lw6pQYGMyztkAxcz+S7omf0k5VMSrlA+RoS7//0r5fddKJSifXAoPZ9rX1wkDL37G3QypvVxIpk2WEjqYDJz0WyWy4CsltDuDPsTmPGLxqR7y7CAWxnl+3AYIJs0wUKE/dpT+K9NLFnsyNFelnTTaKB9QxqK9+lncN4BrRfTBYQ3whYbmYxYWEGtk85UK2hPkgHAK5bg942dPOUoDYCOAGbhgJU4xDVdJg8RjedANumFYmrT5N1SklwT2uwiXg/Bay5IUCqZ35V3bOXPhyoqEVSmyWoyktTjZEwIGtFftuzjEqhOp/CuyuFvXy9dgSgrCCJcUUloM9Y/zx7nxAL0jvppbMq0LjX/JdyAuy3hQF9z6Xvx/zDACx/6ccALP/97w0slVKHQOucIfPQzgAAIABJREFUWyqlMuBvA/+Jc+5vfpfn5sA/h7CV31JKPQC+5Jz7299vP35fUtg/KMvovQfu3f/sX6WsI94/vkLjiIOOy2LC5XrMw70Vb4zuaJ3m7/3dP8I/+ye/ynU9YlVnlF3Ei/M93nl0xW2RkUYdx/mGXZtwXeSM4pZxXDOPS752fsrhZMflekxTR7x/esHL1Yy39m753bNj9mc7Ho5XfHB5zHsH13xwfozWjif7t5ytp2jl2Hw8Z/TOis2LKW7UMd0ryJOGq9spv/jOxzzb7nG9HVHVEV88uWDXxbQm4GozQmsBKbYNODhas9pknCzWJGHHs+s90qRlu01Js4ZimfETb51Rm5BJXDOPC371a19Aj1psHTA/3JJEHRfnc4K0I0larNWYTmOdwixj0sOS070Vn//2KXbWEdyFcFJjCh+jnRpYRey/fceDyZqPLg5pqoijgzWrf3DE8Z94xYuvn2AWHUFieHx0y+efHYkEaVFRX+TERwXvHl3z8eUBTRnx4HjJqxcLAUVGEe9XOKtQn+R0TyrijzMZQE46Dh+suHq+x+hoR/lsQnBcsj/fcrseDcCCzKACJ2Cp0hw+uePq+R7JoqSpfET4dYzNxKeo5g1cJSJhHcmgsssd2gewmG2EqjVMOtQyIn+yZvdqQrBXY24lGMhNW2g18VU4hMRgYPR4Q/npVGRluSE/KKieTbDzFr2KmL615Hiy4dPLfcyrHN16lmSpKN+vJeH2QYva3YNIVQYyON4pki8t2VyMwSlU3hG8SlBPCoJvjqjerFG7kKDQdFNDdlBQLlP0OhxkqLqQAJvusCW8jOhmIk3VHbQzK7JbBel5KMmsjyvsdSJS1blFzRrsNkKXGjtvCW4ieFBj72KpSonATDv2fz1k/baoBAgcwVbTzby3c9QRnceiHDgusB+OafcEkCkrg/p24lP5PIhop5bouCT9e2OKhw4bCiMVbjU2cp69lzRS3Smq05boLpRexbE8DjIYdxrSKy+5LRTJndSMmLEhPQsxuaPLJS0V5LyoTsn9knoQ0infxyaAQTc+OAVoDjvCpaT16lJYX7Pfkn+UUD40BDvxK5Yn4k0Nd0rk07FXS/hrYENAi7TXKYaAHxf47fpKlHYq/mLdKcKNugcBjSLcwvbtjnAbeCAqyozkRg/SWV1D8XZL8irCaYc2yqdQCuOe3jD0DqpOBlLFmy3pqwhdMwSf9N5IkwrIsqH41bpUEkq71+qX0xvpMwS5buFODYqM3tPa1+QEtTCFkY8h6JNP24mAAm2ETTc+tdQkXuLssXav1EDD6IVj+1gRbRhk6u1ErmG9EIDaAz+T3Msth0RiLT/JUo5PGdmfoBLZqm4YUm6jnfheJ09l/3Qnkk9RSggQMLGAqvTGV7UYhuTdzduW2bc07ViOPz8Xxtik9/5Ep2HyzLA9DUQhUAgAKQ9ke9mlACwbCFM/JOBmAr7yS8vyvYBwC/W+93HeuKFXUqpo3KDo6H2nXS4gVRtfS2IELCVLJ0m043u/aTuWfe2l3MmteE1tAOOX0vWY3Yh3VPlrKRUwcp4Gb6ZXEulW/Js9Yx20AoabqYS9AL4eRKSmzcTbDrxnUneQ3rSc/ULC9HNLujQs34kGyWi0lc+WLhNViPEgtK8eCWsBf20uCa3lgR7Atw1hdGloc009U4zOLWFh2D6MBhlyVDi0EUDYp8MGjQC4dCldnb0HU3fyu43UAN6kGkXul3hrh77LaCOS3XakSdbiPTQ+8AgYvKnR1lAtAuKNpctEQtun0ppIEbQO3TjilTC5zSwkqK1IyRcBXarIrwy6kbTZnhUPK5HG2kQTlJZqPxy2Ge4s9TxgdNbQzEN046tSnCMoRQJsQ1EVRKuGZi++rx+xDF5KtCK+qzF56IONFGg1sMjhth3YPNVZbHKfWekCYZVdoLxkvBsqRUwWojyQdKEeQKULlchufRAPimEd1RrxWI4TAchlOzCgJg0Jl+LRVK2RLktfRzKwqK9JR/HBOC6LBRR2RiSwfRdk24nk1TOPqqyxo0xqTooKF0fyuGczXZ5C3Qx1LT14/LYQndflsOY1gNjLYJ0TdtI6VBwJSNTq/nnfyTwaIyDQ91W65v5a3DOsv9dj+T2BpnW4tvm9j/+Ayx8UYPn43/zRA8uP//L3BZZfBn4ZUZ5q4H92zv3VH/V+6B/1C/6jXKxTbHcpUWSIdUccdMTa8PJ6TtdpNnXCy2LGr37wPtm7K/63X/tpii5m20rq1xsPbrnejuhMwNVvHXNRTHy6asg4rrkuRnz11SOOplsOsq34C1vNN58+QCvH8/WM2biiqGPu6pw0btm0Ce11Rhq3LKuMk4n4M0/+yCWbl1PUrCEdN7RdwOX1lCyv+ZXf+iLLUhK42jrkd1+dcL6e8PSTI5xTjJIGFTiivKHpAtzznEVasKpSpiMxMy3mO8pdQpS3vFzNeHU744Nfe4u//9nbTE42BKHl+HSJ1pZQW6aLHdNxSRJ1wlQ2AWYXkR6WdJ+OeXqxT/Cg4OTBHfk7K9KsQSXCUEQvEtRew90H+3x4fkT3dCwe0NUIkznOlxN4UMsH62cpnz89JJo0HD5aYq3CpQbz6Zhtk9Bc5QSR5W6bo1PD2+9cEBSaMDQ4p+ieVBzub2jeLWVgNmq5/WCf6cmG3VWO22uwL3KqJqJrxfv55nsXRFmLazQ0mni/4ur5HmhH92wkbFiriVcaPW6ZP1niVjHKKLq9jvCwpJ1a7LSD5xnRhzn5foEbd0QvY8IHBdubHCYt1irSk50wAS9ignVAUCjipwlq1hAW8hYzE0PwQAxU9usz4jstKZ2ppf71BR89O6GrQ9yiJbkTZqI8NbgyoHujgkYTlJr2xYjoOsSNO5Hb/pE1zdfn4gGNLUo7ukVHW0SSXHkZoxpFN+uY/U5IdTaCwGEnUggfTFrcQSOD+FtJtgumrXgQ96VyIyg0+TMJO8KCqQPpCdw3hCtN/rUM1SjUcUX6NMEetCTfyEgvAgnZ0ZJiun5XJOcEDjfqcKFj8nFIeFCiNiHtzJC9DLAfjTFvl0R3Uq+iWvFJOiWhMdFdSHPiZz0/HrH+ciOpm9cCttqJpT3sPMsjA+f60DD7RkS719EcGDn/raI+7og2mnCnKN5tJIxo6qTa4kqRPwtpZ5Z20ZG/CuhyCRCa/W5AfKsxqcxox6vXBqorRfmwo14IOAkqxeRDkYWlZyHppZYAnvOY8pEcc7SWMJ/sQgugOTI+zEgAYlArxp9rZh9qlBEPpsyky7aamWX8uWfXAkd+Jvd3cq1oZ9IbGpbS87h7bIlvAqKNIr3QpJcByY2meNwNYTfNnmP0cUR91JHeKNqxFTn0816q7wHFTphJHEx/NyKofDdoJ2AouRVgNX7mqPcd3UhknLqF9fudZ3/ktdZv+XTMqZOKlIUVj+1GkV45kluo59KxaCOR39f7ApLilfNJo7B7KAxNcWIJGgFd0RZfcC+MXnbpJes72LwJ2aWwUbPPZJCT3KrBc9qHHykDo5fOB4N46Xwlj/X+wsinaZaHApKzS8f0qWX00g0y++mnsH1DwFQ7FoDdTgVINBNFO4HZp8KC9X7ToBKp7fwD8cRNP7PMv2Wp9xS7hxL6o1vZfrR1bE8DkpUTKfxOBmfpjSOohK3rcunNRAtIHJ0bH2YjqZ7JrTBYex/K+di8ed8hOvvU0I7VUA2ia5FYqw5GF5ZkaSm9b3f+sbDx20daGM1U0Y0U41fyuvmFZf6JkRqVzwzTp5ZqTw0SaBMDyleQIOCwnXiw4T2eJhYAWR5oqc5ohcVrxwKcwhJhUQvp22ymcuyJl7F3qVR2rN+IGZ050lvD7ihk/Mow+7Rl/MqQLg3RzpJdW7angdRxLDTjVw3KCaDtMt+1GwnIStZyTaWTVGMDkW7XM00zDRi/ksF4dmckYOiuk3AkJ3Jg3TlGZx31RCai0jtDl2rKReil1QZlIb/yskoL2bW5708tLOVBSJdpxi8a+f7cGLSB7Fo+P9tckV02VHsBUeFIblvS20482yvphIzXhuSuRTlHeRyzO4mIV90wqZJdd4xftpSLgLAyxCuRJ4/OG8LSp4UaqPZDzxgq8lcVNlFMP9lRLyKy8wobSwBQuDO0k4Bo3aEbYYabhTCCykmwDwiYUw7CTUuzl0gliVYk57uhYsT5vkoba4oHKWYU0Y5DSWutO69esER3JenLNS7QdJNYGEStsEmAiwRU9ompqnNYDw5F5SDyZRf4MByl0HUn/knnsEmE3tVEdyVoJKgnCSQUSGtU2fh+UCspsHGIi0P5N08EhObCKgrIrLwXU0Ik8fUkzjOJTusBVPbdmD14vJeS+w7MtvM9qT2te38MgDCQYcjQowmoMESFEhikAg11DbqPWbYCHo0RUPl6/2XPVPZ9lj7cxxn/45z823X3gNS/zus/7jUp7f+//HCLc+7rzrmfcc592Tn3Uz8OUAn/mDOWyRuP3S/8N3+epx+cEK310AVnQxlg5WcyWOtGjuRaC0uxUQSlSHaiLdRzwIlcKF7KF9r0qaVaaLaPHONnMoPWZTI7i4Pq2JK/EK8kIAOxGT5xUGby02uGtLlqrtHGsX4LJk9llT5aXII0xNvYR7mHhXiYoPeLyUxrdXCfiJhdOMLaSal1oIb0wGjn2D6SAc36bQkzCWoGeVx2LT4kZXyghZf52FhmAaOdY/22IrmTbYe17wKzDL6e8kT65IoHjslnsPyi4/A31RAXn13I7LWNYfuGSJ6yS0e8c2xPNaNzy91PaCZPJZQBfNn3WGZz129q0iv5e3wmaYnaOLaPNJNnls1jzeS5pTjWTJ8aNo8C4vX9jHp6I7K2cl+TX4lnqFqIn6g4FG+sbmH5Ux0PfkUPAxNJ9pNzVJzoIT3SxDB+IdKozVtw9JuW4kDkR+VCEgCrhZIo+pX3QoUw/8RST8QfpTsZCLQjzfaRHtIZnYLySJIXdefIrywXP6/Jz5QvtPe+2CPtr5kMPtNb7++8tGxPtaQolk66yiJFcaBJl47V2xJaM35puf5pxeiF3PeRLwFPl/JluTv2Eij/f5tHAfmlJb0Vv9f2DUju1NC3F5Yijdo9CChOHZNPYf5xw+XPJcw/Ntx+ISBo5N4anckgObs27B6I5230qmH3IGb1ruL4NzqKw4D0Tmb6u1Sx+KBm80Ys93Xqu+g6YUF2p3LdTaQojjR7H7eYWHP7xYC9Dw3aOGwgs/7NWFMtNKMLw/pJQH5hibeWu3cFRB/9ZsXy3QRlYHzW0Y60lNYvLddfko7P/d81BLVl+zBk/LJj8zgcfGvT5+L72h0FMjBaiVQx2bjB11bPtJeiyWCl2hN5XnrtmH1ecflHM3QL2a14RV0AkxedDJIrRzPRchyH8h6dPjU+pVKOMVp3Q3x+suzoRgHbE+mdS1aWqLC0owATyQx9ftlRHIXCfOyJfzIsHdlVS70XymDQweqdiIPfKli+n5FfGmysqOZyz4SFdOP1lQfVXkAzlfOWXZshkbJcSLiIiRWji25gj5ppQHbVsDtJBJAVjvS6YfMkYfSq5e79mP0PKtpRSLUn7LmNIL/s2J5GpHeGak+8ifUiZHccsPdhzfZRzOyTimYekVzXlCep995JGmd6ZzCpos21Z1s0B7+1Zfskl3qOsSbxyZvpTUt5GPu+SUsz1t6nKIyQDYWpaj1L1HshAfJXFU4rigcJxZFci9GZJICaWJNetxQnEaNXDdVBRP6qYv12xuisJVo3bN4ayQDdQTMJSG9bdiexJIJa2e9o59j7ZiGpl6GmHQd0qSbaWdLris2bOflFQzOVa9pM5Vpk5yXtJGb3ICK7EaYqLJ10Ayqo/DULWmHQ8vOGoGxpFildrqlmAbNPxcPYTCJfFdGiOkdxkmAjxeTTHeVpRpdqsquW6LaiOs2Jly1Yx+5xRnrdsnsQ4QLF9GlFtYjlM/D5lnYvJVzX2CSk3k+Il8KUJdclNgnpxhHVImT6/7L3ZrH2bPl912etVeOez3zO//yHO/a93e1ud9vt2E4swIJYlggoGCyEiJAigQRBct6B5AGB8oDEExIPvCGIhE0ewFKCIhsriWPaeGi323379nDH/3zGPde41uLht6r2ud0eAo5Fd+SS/jr/s8/eVbWratde3/WdvrPGZhG6db3MsDpIGDzdCuuURehachFcZtCVRVct5fEAbxTpdYXLxJeqK0s7iLG5Jr0sWb06lLAd330Ht9TTiOGHa2weY4qGxVsT8it5fy6XrsPOq2gq14fHmELkkHYQk1xuaA4HVLOY7LKiHcWkL9YUD8bkTzf42FDtpxLsEynyJytcFkmXYtWilwVuOqA8zskuS5pJQjwXr6GLtfgSyxY7jIlvtrg8pjwdkD/d0MwyonVNM0lJLjYo7ynPx2RPljT7g97jaEJthVmUuEEsvY7LSgJorKU9mggrWFtsajCbBr3aYg/HwtpVVkJuAOWEJYwul5JWCtSnY5JnS5qjEfHLJe3RmOh6Q3M0kh7M57e0x1N01aI2Mnnu9kbobU07zcSvmUSoosYNUnTdotZb3HiInWaYRdkH2XCX0ZuN0StJDGtPZpj5FjfO8EbjI010sZS00zhCL8Q/qepGfs5XApCyBFXW+FEuPZCREX9kWcv/46hn4ZTzUJT4yUj2oapF2rpcQ5ZCWcnPqpbE1m2BiuT1vglALCSzqiTB17WksnbexqaRfsi2RY2G+PVGtjMaioy2aVB5LqFBm0KAYBTt1l1VwgB6v/NH3gGQGL3zVuo7YBNkX/NM/l+Uwkp2CbBai68yinqQ2PdY/n9d7vgv/4VgLP/TPwPG8j/7f+ex/LNafqCB5fBTZ/7ef/03OJquibQjixpuigGJsSzLlFf2bmmdZlFlXC1GJElLbCyNNSRRS2wcq0LMMGnc0ljD3qDg3mjBO5cnfTiadZqmMTRFTDKosa1BKc94VJBEluvFkDRt2SwzBuOK6aDg5c2EvcmWg8GGbz87ZjCs2CxypnsbFrdDlHHiTdsmItm0CmU8tozIxhVVETOaFGxWGd4psmFNVYgEwtWGbFKRxC2ruWi60mFNuUqJ8wbnFVp5mmVCPKmFCQsprjq1+FZLdckiBeXFe1ZEkFpwCrU1MBaqx7catHSEKeNwTTczhSS6zkrck4F4qVoFsQctDIZZRNjcyWOh1kC1WuSjV+J3HOwVbG9z8TtkVuS2QWLWpbCqQYsvDaoIKa6VgUq+AEena2EPay2pmplFZ8H75oFpgy8MqjSogwq3DL4E7YluI6lb8UgXoRcvmI8CG1QrEQyApHVuNC6TzkA3sOiNJOz51AljF/bXpQ4ij1mJ1FCXYvCyU4sqhXm0mUgevRY5qx1IvUf3mMslOlBVGlPq0BEnXi1dKtmPRkKd4rXUkehGPGDeeHQtQDHaBCnZoSW5MX2Ko029VMlsDGjfS/W6QCVdy2SCyCt135Fnh5bsRUR14EDL32wu++KNyOBc5InXGmcCy5DLT1OKJFV8YB5TqT7FUzXyvGgrx9AmMtEiM7y7iREb0lmrfUnBNWXw6QGmUPiY0NEIduD759hMQKkpVF910YxEamoKOedei5zSxYjf84Uh2goDB6Br1UtoXSLyUZf6/hz6SJhVXYX9iHbHNV4Lc9dJLaOt1GgAPSNWj3f+NZfQ162oUHNh6iCF5U5yajguePpgINXKpNEufZcQICJPbYc7314SJLs220lMu3RN3UgYU3ojk0SdjFGCsUKNiduln2bXnu1JqDHKZN+yS5nE03XYblBMdeEyuiHIh+VvupFJuuwmJL+GyQkXEkGBPjjJlL6XVHZyUUm3FU9dVxHRyUPlGiSAO/k9WckETZfG2h3bqAzHNGynDaE1PoSrSYE9fQKoD+E9Xd+gbnf7nSw9PpL+yHgNuJ1U2GthMV2i+rTPLsVXJKoC7mRySiR7XaKrvA8VJnqEEZUEV8Rrtw1pyVGoCVoKw9cM5W/xdnfuXKhM6So1OqZfJj6D76324R6xu6ZMLUF0co8Mr7M7r2gXIFQeihe0S+btji3IpEK5t5todUl4/yEQL6o8dTjP7SAcTy37UxzIvck0vp9oUTbIbQeKZCO+Qt34vmdSAoVk4qcNAU7S4SnnorumxZ+o+qTWeqxl4i5SNLkmKh1RIYmwLgqfsdb3YX9dcBFKOizvXvdy7CTMKt4GOWshfY3Kyt9UGySLml1yrBIJs7a+n3SKNzJxYmrfJ7zq2oe+Q4KfWFhAXfu+/sPFut+XLuhM2SCBVYAS9tqUDt16mpEJ56Tz7IrUVFtJae1ClHQt9+N2ZDCFC37K8P494D0m1JQoJ3+LStv/7rWS99jukmuj0srnxnSBSOFviQBpbySlVdeBIbsDhFysPxHYI3Jaja5aXBpJAmzrZIIm1JH0KdNVI12QsQmVKO1OMhq+n1AKVbZSJeIc3pg+3VbXu1AlqToz6KLBR53PU1JgvVGoSipEaFoBgnfWo5yTx8Nreh9k0+7Ce+7WixjTs5CqYxutMKKqDV7QDqi1u2TZbvHeS0el9zsgGepPuuOqVEh+9Ts2UxhF9z0gFYAOzH7X8kfJXf+45c+B5R++/Dmw/OewpK/c96/+t/8RxXVOchWh2lBpgIR2jN83VLPdl7zNQsDC9u5NGspjGaTGaxn05i8U9V64IW92/pNmIgyS155oI9KeLsa+OnJkFyLFa8cSCCFG9xBDH2RbyVxYVN2o3pAfhYj6bt+TRQh2WIsnxqaEyG153z6SsINqT/UD16gU9jW9lQFrtFHiqaq6m4v4o4bPhC0VNiVE8Bf0HXXxKtQRDIXx7QayyspAavma9Oi5WHxm+QvN9twy/NhgUyjPLPkTYaWaiex381pB9k5OVISBVCvsZ7KE8kC233eKWZHCRdsQs7+ml9yUB7JPzYh+EJXOQ8dcLQMtr0Tm1g5V3+8nybDikan2AkNaQrUnMrbiRB6LtrKdZCGvbwa79NnsWgZ/2zNJhS335Iu8OJTE1/JAzkU7kHXrhj48xJSeqKRPHO0GPt0A3CYhjTeR97M+VyQrGD21LF4TcFNPAsO+H+SWK9ieeabfoR/MxhsfAAN9p1znZROJqKyjY9JsqnovVD1RRCWYQkB1PZFjaioZkJUHkoyrPD2DmF8Ju9zmIiWMtnIc0rlneyqDaJfswEK0keORBIleO9zJ6bpqiS5AI14JWxyvZPvdwN1m0tM4euJpc4JXxyNpzKJGiLc+JC/K9WtzWc/mXDF47vu+vHTuJe1ztovq7wak46fCuoJcT8LYh/CQ0A9Y7cl6vRbWefKhJMaaWkJB2lzRjLpBXvjsahkwlfuKwcuQhDrenQfdSu9evO1+lwTiLnymHYivjjDo7/oRlfdUM40J3X8CsnbXRedl63xf/b0pMPXKCVveDDTlgSSyFkea0VNRDEiNgYAMU/oeMCgn4KzNRRUQr+Vat4lsd3ui+8F8upDz3Hm3olIG/Zszxf67VgrRpyJT3B5phi+dpGePVA+Gsrmn2NekCycs8FyAV8cId0CgY6zvJqJ29xFT0zPD9UQx+chSjUWBUhwpBi9dn47sFVRTxeDKsTk2JOvgZQsAMJt76qHqQYVsQ2omugCV7aFcE+lSjm+bieeunBrSlTCh+aWkqkaVDIC3B4YkSFidIcjV6I+7N3L+82vXM1vaSl1H58/rwFIz2A36460n2kpfYNUBpTgcowCqqrHuQaxuhc2OtpZ6GmFjAVwdmOm8htltS5trqolc6OMnFZvTRNZphdEs92Pi0GdYHERkc2G2y33N9L2m99SlNw0uCb2L3lMcRmQ30odoCofNNKZ0lPsR+WUjAElLCAwKtscxg4tGvvNzTZtpkkWLzaX3MAqpqFIL5WlGmvS2DZ8vCZExpaMdmj5ops0F+LlYES+tAJrKsr6XMH5SgYNmHAUAJ8mvqnW938w0AhTrSUy8bmkmkaxrZbGpJlk0VPsJybLBK0W1F/cTDvmLsq/kUNYLS5hF1LOI7FJSW021qxcxRQtaCXtZtsJmT2PiRYPNjOxL7dCtC39LeibYxboHbjYVkNWOhJVObmvMtgHnaA4G6NrhEk1X4aG3DS6PsFlEvKr710vvYyNMZutweUw7SYlWNdVhTnpV0ExT4puSdpqia0s0L2j3BuhyB0ZsHmM2jUhTa6nx0FUjzKiXz6VLIgFjnVS1DeE7ZSOy0EEqvsbW4qYD9LbGJxE+NpI2O99K2I33AvScE4axq/yAHtT52EivpNGy3m2Fz1L0eisJrk0rQTdVLayjEc8kSYxabnow5vNU+ijTROSenbeyqlFJLGCztbvtg/we9qNfjJbHYfeaUG3yiVTY/rEAhFsr4DD0YXaLiqJPAscO7HWpsEqDs7KvwSup4vCaptkBXRDGMkn6oJ5+CWE8/fruJL8CeOdRAcj6u2DzB71u5PyBf/hnACy//Z9/fwDL6E9+yvfxoj1xZClzS/PAwipGTWpcJV9uy88GBi61RM9S1GsbquuMClBekVxrqQ+JHG2qaSegRw2ly8IXtxeQtpWePju2qFrjRy0QY4/Eq1QfyEVeHocZeUVIuAvde6kMjpuZRTdR6OCTYI167GnG9JUgLhFg2AV3dLPhugLlurRCxepVcIkwWC4Rxoh7Jf79LMzGe9qJw5UKnDAy8UpRHEvHXrSSigHlA6MQGKd6JiEoKNieChNTvFaTPU56tqE8kcTCdiiSYylEFwmgLgQUo9ixRI8zmqmA6XomoHf7qKW9Mv0MdQf047X4x3TbzTYKyDWl1DA0Y1mvj+R9J6tdFH87lJCPzQMVugtVv25vBFR0+ybAxlMcS9l8edTNDMpAvx0IEC+OBf13klivoJxpqpmcD5DBaTPesTkoqA4lSKWZ+J4xy192g1nZz44da6ZOwEWpUE48c+pxxPKRoZ55koWwMC6VgJF2QAgHUawfQL3nSK81yqvQJ6aoJ7K9euZDYiIUp5Z2oQW0p7K8Lc2jAAAgAElEQVT9rlrDxVDHkHgVZtshChJpm4BLYXNf2DWbEaohdPABSp1EOvfYXGFLOY5tLq/TFXAHzJT7u5AR6XEMISKtpFV67Xsmq/MtSf2M6pmMaiqya+k7VAHwdpUnCrwKKZCy/4zUDkAXAujLfXmvdScND8mQupXBdXXoGDzTNEHOTPeeDGzDZIRJ5PW6he2J7lmf4kDTpQm2uVwfnUwaJe9pcyYSaJcIC4HeMWtdKXsRytzjlaeZCHAuD+RnV6OCQqSnVfjcJaECSUG07lggAYQuCsBqLNdKV5UUr+V8VqHmox4JE9YMVO9zU06YMZxIfdfnGlMI26QD8yzHInQ8pnJcuxqJZhjYQy+dhC5SIZQFAYuhq3GbBa9aJGC+8/Z51TGFATBqRZPL5zUq5LH1fcX+Nxwqpgf2yVJ8j9VUQG6T77yLEgijejYrvRV2DCBd+n4CrhkIMwNyL+nuK3WYHBHmfff/NtXBtqD6sBpldeiapE+ibDMBwO1AKlNY7RIrleuOXQCsJkxCrT1NupugqkOfotdaQok2d18beiqD5aDNQbcC4LyW68brHTvnzK5XUlcChL3x2FzYC2805UwJe7cSiXtUyv53rKbXUvMRF47toVgFTB6qOzACVktHM5RKmu5z53VnY4nudC3qYJOQMBqbiW+uHZpQCaWl59GF+iqjSOfCxrlYAll06/t1GqCeRr1s2WvxZLpU03XLtpkW1s0TqkkCm5xoTO0CuBfgFZWe8iAh2sgYoANaUidiSJbyfBcYO2HcBGQqrWhzE6w2AmKxnnYiMnUfIUBcS1ppM457ea1MRjlsbvr16ka+F5tpIqAvVtwd4ikPaEUziNCpVIC4VF7v0gibhn0ZRpjSiow31pjSBrAmrKDL0x54dsDbRwqXReE1DpcYXGp6htIbjQvppD4cIx/JcfaRRrdO0k675+bBI9ixZh0YCWE9LpNzSBSYyrUwWD41OwASG3wWCcDUWkJ08hjTOkg05UlO9mwHVpQVttHrcO+2nk6y5PMEb0zwR8aoqglAMhFg66DrlXSjgbyXVvXg1+2lO9a0tRBHn6gX8Xkqr4fAEhqUD95IrSEJIK1jIo1c83S+xS4Mpwv9ueuT7KpH4li8kWUlz9H6k+mqzu/AYhecoxW0Yd2R4RNLeB6xVON0oFQphTcC9oljAZz8EYtW/TGWKpM723AeFase0KpeneO+G3/+YC4/uJzen7j8QDOWX/rhzN/77/5DAH7+9LeZ2wEP4mv+6fpTfPX2nJ87+wqvxJf86uqzPN7u8crgmi8MP+KmHfFxdcB5estFM+FXnr/Fv3n+NX7l4m3+/fPf5Meyj/jFxZf4tRef4mfP3uGvTL7Kl4vX+L31Q/7x49f5hU//Gs/rGQ7Fs3JGqlv++uE/4Rfnf4GpKfjHV29gvebfPvtdjqIVv3T5Jc6zOb9x8Sp/8fgDHhd7fGp0AcBv3zzkR/YeM28HDE3FP3r+Bl88eoJRnh8dfcj/9OTHeTC65ccmH/E/fvDj/OTpBzzZzviR2WM+n3/M3734CSLl+JHJx/zDi0+jledvP/pl/t78S3xtfo8f3nvKB5sDFlVOHjXcH8zZ2ITDZM03VyfMEumlcl5zUw2YJCUfLvZZlymfOX7BvXzBcbzi29tjvjU/4sXVlNlUKkseTW55Y3jJ//zln+Tf+/Ev887yjEFUk5uGcVzy1ZtzPnxySDaqefXwmsNszdPNjJ8+/hZ/74Mv8Gh2yyiuuKkGDKKabZvwCw9+lf/qO/86f/nsXa6bIfMm5yRd8fc/+AzWav76p7/M/3nxFp/fe8qvPH6Lzx0/49XBNVub8Hu393ltfMUkKvm/Ll5FK8+n917wvJjycHDLNxYn/Ny9r/DLLz7Ps+WE8p0Zj378CSf5it+/OGOWlzy9nPFXP/1V/unL1xinFfMi5/pqzJsPXvJ8NeZHT59wWw0orXxJNs6wl265rQZ8+PwA7xQ68kzGW5LIopTnwXjOb73zGucPr0mjlg+eHXKwv2a1TTmYbHj2fI9PPXwJwDQtuNiO2TYxl8+nzI7WKOVZbzJ++vVv8a3FMZerEV84fcrTzRSAF/MJs9EWrTyJsRxkG67LIQ9Gt/z6t98gzRupwvnKKfnbczwwTGuu5yOO91ZcLYdUi4yje3Oub0a8ce+Sbz895ocePUPjuSqGPP34AGLH3sGabZlwOluRmpaniymxsbRO93Uve6Mtz1/O8K1GrSN+6AsfsqpTXv6jc9ofWpNnDcfjNd/58AQazec+/THffHHMv/rat/gnT15ju045P57z+IMjhscb2lbTNhEPjm+YJiW//9VXiI4LhnnNap0zHJYiSX9/j9c/+4z3nx1yerTAOk3dGm4vx2STiurZEGY1+ahinFe8fD4D7RlMyvD97ajriGFesdpkDPOazdf32P/CJVnU8tGTQ4azgu0yE1n4JsZMapGdL0RSr0eNSNcHNcXlAGJHPAw1K5Fjb2/N4t0D3EmFLw2H9xas/+9DqkMLszBIcAqTWM4OFjy9nHF6uOAg3/K1dx4S3xrcoxJba2g0ZtiijYRqtVeZgMmDkuY2FRlxoVHHFWlWUzwb4WMPkUNnFmMc5t0h1bElO9lQVzH+MmXy6pzFYgCLmORki/vOiGZmyY4K9O+OKd6qRD69jSBxmFvxouLBzVoIMmyVWfRl0nezuvMSLlNc5qRCZdaVpnYz6B69NajjEjtPJNl5GeMTh6q11H3stWBVSAtWtAcN2Ueyjc2jlvQiojpuSS8N1XnD+fkNL/7guO+ubYdSdeIjjzuq8dsIM6mxcwnvihca8/ZKzl3i0Mso1Lwoxm/OWX4wE9b1sEJ/lH1Cvh5vFDbxfU+nciKpjhYBFL1R4C4yTCGTZy75Lim6Aze0mJXpZc3xUqjqZuTJLzTbB9KzvPeu9IZuz0R2Hy3upCePBIgOXii2p76XpnaBRPVxS/Yklt7fwa4DefNAJkI7VU+bywSY/uKC8r0JyVz1bHQz9f37SG/CpFsAzaYUtYpqRd6uG0UzlO00E5lYjLbSe5wsZJ0uqD0EiEBxLnU4/WSdl5/rT1cM3017RllUDkH1MxRlULKSMB3ChGx1IOdIaiyCmiQkF2/OpXdXt0GZciOA9uaLluPfMGyPZV/ThaMOeQYuFnVGcaxIr2U9XYWI8vQqhq5uRLX0DPKOARW/erWnGD11bI816cKzOQtdqVdhEiN8NFysehWAbmR72dyzfKSlX3g/dDWGPmrd0mcrOCOTEfW4SyEWdr8Zql5S3S1dwFTH0upG1DnpUtQANqWX9Oc3ToKFgnS4v2Y7pj0KaikvHuVqIrUm1UyTXzucoZ+YiEqZ8EmXTqpMvLDrbSZSWBtDHhQJ8Vr81dnc0gwN0TaA/XCcsouKek+sMPW4q1KRiYxkJYC7zTXx2oqfc9Vg80hUM5npfeAuCrJbD9GmpThOiEqR8OraES8b6r2EeBMCcJQiXlbYYRzYWtdXgKjaiZe3cSKnVcIoeyPAWbdOfKnpDlh1tSHtICbaNuiyxSXiB3aJwWzq3nfrY5HYuiwCpaTKRKkQ8hNYQ61RZSXMZwDtd//OdzOJIP7SwFB24UUEKW3PtEZGWFml8JstapCH4B1h7H1ZCVtprYDVEN6zu+gCeA8+zO+W5X7P4hyuLP/45/wxy/cNY/k3/gwYy//i+4Ox/IEGloM37/njv/UL4r9rlfSj5ZY4b6hvM1SliU622MagXqb4kwouU+ldU8K+lGeWaK3Rr63RXxtTvlHirSYe1KjvDLGvlvA8JV5r8aUpaKe299vFCy2DlaMKd5WKX+tBifkwox16XCqsZt8td68i+SilOrQoJwOZ+kDqHdr9hugmxg4cow8M6x8uMc/T3gNjamEWy1MZXLRTS/40QlnYvl4zeC+h3hP/mstkxjVeSDKni0SG2ydV1rvIfZvJwKkdin+smQoroWW8GmaK6Quz20H4ogbsfsPka4msdyn9Z6ZSgQ2QKP3iQcvwgyiEYMDysw16bchfaDZvNOQfxtjc04w9w8ea4mRX8G2HjvTCyMBhIDURLpLtxyuF/8KK8iYjfxz3X3AuFbau2oN6JhJlm4mEuD5sUZUkYXZVCMoKa1uetgw+CrOq3SRa4lFOkd4Ehm0k++UiSG/kdTYRP1/+UuSzXWVFtFEU55bRB+IHvesxs7kwdbqW45ldB9lmYANdSu/76vxcnVw6mcu5cokkkDZTT3odBi2NDLTiJWweerILkRaW+0HWrCU0ptwXZnT6DXm/q9cd2UvdV2XIcZR97eTgpvKsHsHkA1g9EjY7WYo8t3ueMHL0ktyokP32kQwwpUsvPKZ3dRHdwE58Y550vpPBdhLeNpPPQLKU/b87uBQfo9QxDJ9o4pX03HXJne1AWPXZN3cVF20efFXVbjCmW4nCLw/k/Hbevs4L2knHi2NPfqF27zWRwWC5L4+ZQtj1rrbC1CG59FgRbYI8NZGB0OBC6gtWDxXxJgxMg38u2nrWDxXZlQz4Ok9hVPjA2Mjgx6by2c6uVO9vM5V4/tLb0HMIfaJqfulYvK5Jb+klqVGQ0Vb7kgqqrKR4JiuPjbvAMTk+nRzd1MGDOJDt1GN5fj2W/r9k7kmXjtVDgymlz68dyL4kS8/mNDBSY0UyF59geagk8CmW92JTFcCDl8FmvvPmmjLIXAmf/TB4bwfCVLb5zm+qre8DsFb35Z48fOGEXVbCLld7qh/U6xrirUOHUJ96IqxQduNYPjKk8+DHi+W52ULWFRWSWNrmiiZXZAtHHdj9ru9w9Nz2Pj2bqnA+A5MbK8oDCYKSwCfYHhvpQoxlH5SnBwbxRqSLnUzdGZEOd7JdqS+RY6qcAIWuO1G5EAIXwpY6z2RU+b7WomMiTSMS484e0X1eTB22MRKQkS5tkKJryj05Hp0fNd7KIL8eywA5WTlspqgmIUAn+GflPLneYwcdg4QEVOUmSN0DMCk64GBpcx08xa7vP4y3Xra/drRDYfdVK0ysVxJAZTORrTdDkfiOntZsTxKy2xZdO8rDmPSmxeaa9b2I4Qsb6k5U38GYzFuRJE+jcE+ytHlXSeFoRqZndXW4bjt5s24cNjOhPgKiIvgOrTCuIF7JzkKjK5GpRuuGehZSU10nYXa0mch6k0WNzSLaoazDlLbfH68QaWlpcanpGV5dye+iaogwle19jZKKqkMIUEJUWKq9hOyioJmIedlUjnYYkV6X1NNE/MKlRdcWl5j+PdTTmGRe443GbCX0pR3GmK10UNI63CAWGW1QHumika7H4E2UfQphOY3DDiVYqbteXBaJHHld4QYJelsLIwqo1qEXG9xo0IM61Vh8EqFXpYCgJAbrsNNc2ObKiqcxgMJuUVUjzGPd7Ng963bJrkrh8xi92H5CwuqzWKS6VphMtS2D1DbqAaCqG/FS1g1+mIuUNomlS1Lr8Np452M0RmS4Ssn+KCXS2kEmoLEDdnGEX4Uv4S7oCOQ5SSzJryEZ1q/Xu3WDhAmVlXgwq0q2d0cWe9fz6a2VYJ8/JNFVhWPhAxD9I5c7VSY/8B7L8wf+4X/yZwAs/9afA8s/9XLvszP/N3/pJ3ha7THQNalu+eb6hFcG11g08ybnuhpynMkHYj/esGwzUt3y3vqQvaTgMF2zajPeuT3lM3sv+IObM55dTznaW/Ez997l7z/+LAeDDZ+dPqfxhspFtM4Qactvv3zAyWjNfrrlg+U+r05ueG9xwPV8xF969X2+fn3K9e2Iv/T6e7y3OOSnTt7n11++xkG+5XI7ZJJUxMZStDGNNdTW8On9l/zui/ucTxc8ns/QyvNo75Z3n53wE698wLweECnLos55Pp/wcP+Ww2zD71+c8ZP3PuQrl/fxXuE8fP7oOd+4OeG16TUfrfbIopabzYAfOXlCYWP2ky0bm/B8O+HBcM6vf/wap7MleSQBQDfFAO8V5+MFqyalaiNqaxjEDc9uJrx9dsG7z054cHRL1Ua8Obvk9y/PSEJVyL3RglWT4bziJF9xUw14e/qS//3rn+dT91+ilec7Lw9J05bTyYr3nh6xt7cmiSzbKqFpDVFksVYzSBtGacXji32mkw2NNdyfLvjGN+8zO1uSJQ1pCG1qrCFPGqxT3N6OODxY0VpNUSUcTdbcbnMmecleVvDxfMb96YKbYsBym1FXMXvTDa3VWK84G6/44OKA/emGcVpxs80p6xjvFa8dXvN4PqMoY2bjQl7jNJttytHeik2V4JzmfLogjVq+8fSU0bDkfLrgYjMKjGhGbByN1dRtxMFwy4v5mPpywOh8ibWaYp1ydjLn5c2ENG14tH/Ld14cMRqWbMuERwe3PJlP0dpTlTFtbTg+WnK7GqCUJ4ocZSG66sGgwijPcpnjWs3h0Yrb5QC8wpaG4V5B22qcUzRFjI4c/8qb3+YrF+dUTURVxQwHFXUTUa4TZvu7MCpvNbP9NcvVgCRtUMpTlQm21ijjGYwq6tpgW4MrDTqz7M823MyH5IOaskiIk5amjrDbiMOzBbfLAXFsqauIOGmpbnLMuMEuYvS4IRvUwiCuI/zA9pLqZFRTL1IZQOUtSnvsMpGamk3GdLJhuRrQlhFR1mJbjS8MetjiaoPaGLKzDW1jaK8zfOyIpxXtdQ6jBt9qdGpx6xgix/7JktubEb6ISPZK6tsMYkfyPKZ5UKFjRxRZnFPY1pBmDcWNhFbFoxoTOcrLnOyowDlFPU9RjcbnFr2IcAPH/vmc29uRSNsLA4mDVqFakfGpWY1+nkkg1ahBhfAk7xTqJsFHHjWtMU8zmv1Wwqf2a5FZFkZk/kEGD8K+xu/l1K+UUtETSZiUjx261PjYS+iVBlVrDh7dcvXxTI55IXIyv1fjK1m3LhV2JBNeGKmg8dGdwbUDH3viW0Ozb4nnMgBt91r02uCGMtOlSx0mXyRQqz1qoFGyb6kjvo566Wgn6zdl8ECPLKoROXi0VrRjkarXE4epJQTLlKrv23SRx4WJty4MSVmI19Jzqxx96FNz2JI9k5qfbhLFxTIRqWpFeiVhXm0Ilep85M1YpOzliSXaSLp5O/B9Em5nr5AS98C2hXCoZKHFWrFVPQOpQueoi6V/VFdyiOupgM/0Wpi08lDqR+qZky7UYEVwEdT7DpysM1mqMKEYekk3ingpk2DN2Pd9uiDHvBnL5FcHEF0invAuVKmbpNSthEApJ92q7UgmCjppvovoC+5d0rFpMrkTb2RCK1nIddMOILuRiZ0ueKqbHOvWF69koqXrPG1Df6hNZRIUQujUncAwl+wmDifviQdXKnJ8P+HRefjl/YSgpVqY00623nlNXQz4nUy8C8rpXpvedtJ21W8/XgZgHgX/fhv2KRdPcxcohRJrQ3bjewVBH8AUEuNdJOxluRf6PqtuAvCTEzYiS1bkNzLppYIvOt7K711SvLbSwykTD3Ku6rH6RCerNwLctaVfT5uJBNlr1bOZMkkj2wD6iZd440JAkqQ6iyza0+TCQNYTTToX9rEZmTBZ0zG4vr+3mFoet6n4ZbuMgWpq0Nb3kwIu1j2IEa+2+FF9YAC7EDObaeJVCxqc0WjrgvTVYvOQRmwUKsh9lfOoxkp6cEjdVZXtQ39cEmGKRlhJLwAX5GcnI1Z3ex+VEhayG8N7v/vdOflOqBupG+lDhpqdR7K1u27MuzigY1nvdFECfZ1JH/5TNwIqncNX1fd6MrslgM3vCfS5y1r+s+KQ4NH0VfXP9vw/ZPlzYPlnv/xAA8vBm2f+p/6Hf5dtk3C1GqK1p9jKjFk3QB0MK5xTpHHLcjXAVgalPb5VqMjL9WwVybimrQ1+meAzi4od2njcTYI3HmKPzmTQqROLKyNUrWWQWUToYYO3MjhNDwqqmxy0l6RU40FDNGxoN7Hc5MYNdpGA9phRiy2MJLZ6hbpK5GaVOtSghbXMrnnj0YV4XvLzNdX7E1zm0JWSxNGt3JDcqEWvI9zAoioZ4InPUAYdXks3oTBcwvTiFXYs5e0+8fjISZKq2zGVdiil8D63MphcB6Y4Dh5PugGDoh04TCEBR+51kYB1iw6DPB+YVbMyeONxA/m/HVuihcGlni6pFA/tQUN8FdMOHLrSvdRM1apnofuU0TsSn64g3Y4cqlFh4BjSPAuRjulmJ+UxhaIdy3HV9U4upGtFO3SYKsiS1sKO+Ehmw00lf+/Y3I519XrH/JpK9Yyv8uCMsFO66QY0wkbaoUMXMgCuD4RVjzZhsGHEg9oFznTyJ6B/zJvuy0HR1ah0RffiYVSUh503MwRHBdlZJ3HT1Y7Z9kZ+twNh4LoETZvJwDR/ocV3hhzD7txJ0mLwd2WhS7Em+I66lFcJB4rXqpdN9mmnCpHklXK8Oh+UgIUwAEplEC39bTLwk/NFH1ojITmeZKl6NrXbz2QuvtEubKlnQDf0nYt3U0c7lqpboiIM+oJvtGP4YZeyapOQhklI3rTyXrqE2rtJoHcHoKaU96cbqKb0n0fd7j6XfQqrDoNCJQPlbh3aBlAQBuSqBQlokv2LtvTl5MLsB29msrumOpkf0HdUdmFDeGHy03lIpOxSRgnMSqgSMnX3eZTBngQCqT6Z1BTCXMZ3Q2vMzg8pEr3d412xuW5DEFUZXheYTlPugF8XCqVr+azXMxnwt0NCWmnwhiaq962aMhwDt1MMRIWwdE0IlJLjE0rdy13KKOwUHzbbXQcqgApT+p3UL9pde20AC+1Ajsnd60033Web/vh0kke5voQ51rXsYz0JrG2XNBuul7jYAY+uyD7ZCAiQkDuRerpo9zrd7H7HQzZ3ARx1AUmqv1ZQYV8C8Eo6aWisSDbCQtZDCahqBvIT2LHm3b0zXD9tpog3rmfnO7ApT9ht11TC4je5hBJ14EW5cKzsTgHSXT8Sjre7lkztd/e2EPyULl34DAS22HrpWVxbmpFIJ5WT+12XmtrmAn5043eMa7v7jum8txLGE15/h8XqQI98JzjqWRTSgMXPKKBR7tndfbJnOc3u/vzdnxMXOhmlN1F+N4G57D4r2oak1k2Lj7V4L1sJ6tG1+x7wJRejMNEuBOuAMKEq1K9I1yNo69BFix3IY+0w6r2epmixmYTvqE6OeqdD0qUiJb3Lbtos2kk5lbC+LtkBJV3bXTqqkzTYDrAJMFN4Ix7Pu/uO9T1gk3RdLUFCHaunlKyrdZLq2vc+ipcTpXopqzdKGMvOi6nZJcB2ATxtl2br+vX3jJ8KnZUEcGcDIOzSYGH3mrvj+S6M5+556rbVP0eLjJVuNXcY2K7GJIT9+A4Idl7K8HeUlmTZLmQneDhV59Ps3hN8ksn8rsV7v3vNH7f8C8JYPvqP//kDy2/97e8PYKn//96BP82iFFxuhjy/nnI2W7Jdpfxrn3oXE1maOuL4YEkWtzzav2VTpJweLNg7XHF8vODBg2tU5BjvbUknFVqHma1Rw/7xkkdn12hjefuHHjM6W5NMZIbEzCNMbMn3Cnx4TX645Y17l6TDmnivpFpkkDjGx2sevXEBieP8wTVtGZHvFQIky4j8SYQetoxHBef3b8iHNX4dwVElg/9Rw3S2JT9dk51I9xKHFT53lNuEN370Y6KjAn9Skc1KouMCl1v2jldEZ+KdHD9Y4hOH3q/IDgtG95cMTze4w4b4/gb1aIOdtbhcWAhzWggLAeijEn1SYu5vSV5ZQ2bxsUevIkxq8Ynn6O0rdK04eOtaQGKpaM8q9GElX/alYvjlAd5IcbspFfZeBeeFyHUB7kknGtrjT0uUVez90BUucbhxiz8vhZVwEoCEV0zfvha2w4LLHM3MYkcOO/C0U4tLPXbgsCcV9cMamzt0KVUdzVSem11q7Csl8UoRrzRe+zAIFR+WHTgBQlaAZnXUEq81/vUNdmgpH9W4VOS13tCzPeIpUdTnNfWepdmT/alPWspjS7wUcNnmUlyva0XzWkkz9pTnjcyAl0oAT2BlJHVQysjjtaI8bUnnUB1YykNHveeoDiz1nqU6tjSjLmnV00wcLpZ9azNJCC0eNMQrRXViKY8t2aVUZJhCgpVcIn608kgKxrvZc90oilMn/+5ZbObJrjTlsaMZy/ks77VkV5KyLAyGPJ7dKJqJE7n2F1e0A8/wiWL9hsh3instzUiSl5vPbEUyO9glG7vY00wEGDQTx/ZcvuCakcMbGDyXSpbtfUd55Nieeap9kVduX2nIrhXFiaM89GRXHt9VMwwFXG3uO8oDeb6PoB5LKnC159meO2F39pz4qNYhVXno2d5zfYE7elfd4WJZRwcehTkQcOeMPLc8dqQ3nuzSU0/pE3SbsWx3+aajnkiCcbLqUpA91cxTnHjqCaxedRTHfgfypvJ4Mw7b2Be2KtrK4LY8komQaEvfu1keKNYPZd2mhO09GZCvH7pQdQPVTI6bi0OacSbgtB3uenhdLPvqIyhOZaBvc5Hv1uNwbIbCeMzfDgNIL8dle08A4fxtYXqKE0V5SF+bEW+EMbKpJAPXU/l7PVZ9IrENQC3eeFavCFBrByKrNqUEWzWTHXiLCli+DtWBDObTW5HqNkPF8nXoQqfqsQreLCiOpOu0Bz/O9wE9LgQdbU8VxaFMhmTh2rCZCnJMOS4SAibX9vqBfFZMLdLndCFpt93EVDeZYAPQjddeJMdTkc02Y8X6vhZAmMDmXPpry33dV5t0k2fre1rSixuoJroHuOWBpOt2kwKuA6uZBFN1kxam9iwfGVb3NduTEGCzL/67rvJifa6JgyS4nKkeZG4PJSAqWzhh1xYugMadD7GeKOqQBFzuyf60mcLGimqqqSa6B2bVRLE5kXVuTjVtqsjmlvU9kXVWE00zUDRDTbEn0lybqn6Cpgs+6tJt66HIoqup+AvjQs6dKcWTqFvP+lzCSlYPoh6UNQPN9tiwPTTCbAW2a3sYYRONTTTtQLpUXaSIto70pmF7JGxZcSizoMWhoTg0PZDXrWdzLyG9bXGJolp9r48AACAASURBVB1qin2DzZQwtUPpy7SZ7Lf4IDXbkxiXaOqJoZ5FRJtWEm9bR7UXEc8rysOYOqT4NpMIm2oJQ0p0SFg1EoIUSVCQrh2bs4TtSYKpQl9wAM0u0RQniUzMDaRnVDmPHURERZAuDww2NaxeHaKsp9qTkKHiNEM1jvIwk6qP2oUwI7n+m1Ekz6kdzThGVy2bhyMBkBqUFYDajmLaYdwnybrEUO9n2EGMSwzFmfR02lFCM8toJik+CiC1dZjOH5lF+NRQ7+c0syx4ETV2mEpXaBZjh4l4IpMIl8f4LMKOU2EeB4kATY0wjSFIR0KECCBTmD6XJfhI43OZxfPDDD/KRaKaxbhRLt2YmSSqEhlIYkmdVQo3GeCTGDcegta46VD+H+SqbjwUiaox+CQW9tJo+RdqWfxoAIMcBjnqzr/e8xhFeOdQUdT/wxgBgMYIQxmF92ktKkk+CSq7wTp8D6j03vf/ut//yMV5PpEi++fL9+3yA81YDt888z/63/81BnHNbZkTa8cgrrkpBmRRi/WK5y9nDMYVWksylVaSJAvQtAatHd4rbp9OOXp4y+1iSLtMMJOaOLZYq5iOSrZVTLHKpP9xnRCPapxXorKwcgPU2hPFlrqMGI7LXn6oFDRVJF2V2hOlLcZ4osiyWQizmWQN1TplMC1oGkMcW4oXI/SsJopbAauRIx9XFKuUbFRTLlN0KiEczSrBDFtcq/CNRkVeAkZaTZRabG3wrTCz3eSWUp5hXrFYDdDKS99lo1GpxVsF4X2hPThFOq6oXwxgJt2Q+UFBcZuj8xa3jYgnIhX0uUUZh44deIVrVT/FHKUt7WWGTzx62OC2IluLJxXNrUSRmmEjYRqNBiMJuRjR9ehVqJUZW1RmYRlJj2Rmd5K7SuEOGvzWoGuNyxwqb2HRSULCBeTBxy7I+wLDZgI4TC1+E6HHDX6RyHsqDXSMbUitVeEY+SiwwcqTXUSUpy045P9njawXiG5i2omV5wYG2uXCpKpG4QZhZrcU5lg1AobbicVsREpH5EmuREbY5gK8bCqsuI9Cv2UqjLHNHcrvui+FvZRuzexSU57IwCgqJKwDHfyzThCyrhXNvjDZbmgxQZYZrfSONQgz9Xf7KtuhAwPRsjP+y7Gyme9lZj4S4KucTA50LLUpJdW3Yz+6Y9v3Vk4kZMTHQe7XMdpjRzzXwXvo+15Lm3l0qXpGWbWqn+HXIWAkWgtz2+YCXIHeswv0U3Adm9szw+HY2SSkOC534SnJfCe7q/eEXdCtvPdoE86ZEvDsI096bWT73SR0YDGlp5Tep9q9ru9B7BjykBLqQgdol8rsYtleMxK2J15KFVE6l+NW7UlIi1cCProaIhuurY7ZUoFVU1ZYvvJQ9Uxax+Z2LJ8phUlrhvTnVDyb9MfXZr6XUNp0FzbSMXAdI4gWz3A72IGsbpG04MBeBxCWrEKoS7QD9H21Uim+73j9SUUByPPitSQbNyNhvIVVkueZYpfInN4GmV+0Y6JsKvsJd4Jo2K0/KhCWOniXdR2Om6ZnbVW7u0Y6pYFu6atlOua879is6MNgekkk4l+tJyrULcl10OZyrONNNzmg+p5QU/veK9r5WvsexyZMECTCLnZsWteX6CJhZvruzpBCjKdPKjalnPsu/Rh31yd859rt+i/ZnetOMtpJNbtjBbJ+Fwv4NI2cW21l3TaRazoqd/7Vrju131Yt9582VUTV7v124TVdDVKnSiAA63grYTPSlSoA0FS+T2f+BHPZ7Dym3bUbbx3lzBBVPrCFgTEN3tb+2leSMCz+1vCeA+gEObfd81RgXzvPKypcG6nc26Kt6//epQR/Yr/C850JYVTbkAqshTHuvJ+dz9dmmmgjCbySEqvJbmqpMBlLh6WockI/ZSSg2pQueFIFFPeJubo7P0EeHFJufdivzifdJf5GW9szp3erfjp/qHwXWWF4ofeH+igkLVsPdzomJa1XGE9TSseljwTA4/2OzQTQgYXsgm0C49wlz6rqTm2Hczt2s7HydxtS6BuLChUjPbsciXdTNXbHMBotHs6QSOv7pNeoD8mRVNXg++x8lF2ybrcvTStptNYJ82gDWO0GhXCHXQw/29CR+V1/93WDiowwmR3g68N5/I6BtPaTTOr3bOcPZy//yOXPGcs/cvl+YSx/oOtGJknJXz3/PSoX81vzR/y10y/zzfIM6zW/evEWn5++5F8++Q6fyl/w33z9L/Nffu6X+Y3VG1Qu5rcvHzBMa96aXfC7L+/zN/+lf8hptODvfONn+Zm3v8rTYsZZtmDeDHh79JzKxbyzOqP1mp+YfcCH5QGxsvzO9UPemr3keTHlJFuxalO+dX3Ev/PK7/Hb84cs6pzP7T3j1x6/yTQvWZUpbx5c8urwmq8vzpjnJf/Boy/zvzz9Egf3Nowj+VYdRhV7b2y5rkc8KybczIZcrYfsD7ccn7xgHFfkpuEPbs54c3rJ82LCKK549+qYzx8/5+lmys+evsNvzR9xVYxwXnG1GrI/2koy6dNDfvqtb3FdDTmfLBlFFa3XfPv6iJPxih/ee8r/+vUv8trZFS9XI87GK662A9QZ/FtvfJXfuXnIXrZleL/mw/U+j69m/IWHH/Ob7hVmky3bKuaHT5/xZD3j5+5/hb/74Y+RRS1p1PK+PeTkYMmPHX3Ml1++wiBu+ItH7/NL3/gRlHbc21+yGGZo7Xg0veWjxR6312NOTubiOR3f8u71MQ+ncx4vpxwNNxznK749P2K+HlCuE+6d3mKUx3nFs5czfv7zv8v/8dGn+dLpY55sZlysR7ROk0Qtt7cjTo8WeK9Yl6n4FMuIw4dzbuZDDl65ZVMmRAeW9TJnMikw2lHWMed7Cz54ecCXHn3M7z+/R/l8SHnW8G/86O/xKx++xdmnlhzla755dcz8xZgf/6lvcJIu+frijPPBgl//6DX+yht/wONij6+/PCWJLMsPZrhJy+xwzWab0jaGLG9487OXbJqESDluy5yijmnXKa/eu+Tx7YzJoOTlyyneKV55eMmmTthWCWWRYFcxo5M1bx5c8gdP7xEnLeVRgjaetoj4zBces2pSYm359tNjtPGcH855fLHHZFTStAb7zTHNec3h0ZKb+Qi7jYhHNW+cXvLux6eg4N7xnBfXUw5na8ZpxfsfH3N6dotWnhfvHIvceVLjrcJVhlceXfLheyecv3LFi5sJo0HF3qDgo4+OyPcKilXKZG/L8nKEOmpRkeV8tuLp10/QpyU/8+a7/IPf+xzptCQB3KHCPhvAUUVdGdJJRbNKcYlmcG9F9WKMyixfeO1jvvLOq5AGidGxQ12kuMTjZjX7extUSNl9fjWFy5TJa3PmzycMjjakcUtkHJcXE1qvGEwLqg/GVG+VjMYlyxdjeFDTbmJGhxteny5478URnz5/zovNmMvrMZ99+Jx3fvsVjt6+4uJqQjVpSQY1edowfzFGFwaXOdLPrdg8HxNNa+qblOR4S/18iDqocI0mukiwI0d+uqZ4OsJnMlmi9ureX9lOHNP7C1brnOpZhj8t2YwTfOIYHW+oyhjz7pDyyDF4sKL+eEx8b4P71oj6pBX/ZZjcMaOW+nFGe1JBaTBrkbKbs4LmZY5qpHLI5Q41lAmd+LjAvzfEDgTwSxqpZvvpCnUT4w9qoqcp7ciRXRr0FxcUz0eoSuEmLWXk8Y0meRH1QVwA0Vr17KlqhclObw2b16WWaftaS3wd0Q4d0VpT7QtQX3+pwDzJRIp9rSleq1GFkUmXSpM+WNO+M6bec/jEES0iuTYyCRPb3hPJu3LilSzvSyVTfV8k3TaT97l90KJLmdyKVjIx5I0nf24oH0qwWHFuGTw20od84Pr01+TKUB21jD6MWL/aktwYmvsV5nnaS2KTuSIqJKjMJV4mHYDh0xCC1tKHetlM9s0F31s9laA31U2ShUmH8WPH4lRRo0S+vlWktwLg66moJUbvR30vsDfyuupA5PLKEzpSPc3UkdxqksWuTkQ5CdLCQ34h4Vz5S8/iTdnH6Tc11d5OOu4ikSOaMkzKbKX/WBJY78ocA6BtJDzKa+kdNpXGxSFEq4T2ALYPLOmFEf+oCh28A7k+TCnhVc1EUY/pO5O7WrB0DvNPKQbPEDCjRP7tp13qqsbmBMAOi0859r8qg/tmJP7Dck9RzxTxUkCOqVQvXa6nwqjGa5m8GT5zbE8MxYln9i7UIUCu63btFBGmDlU5gbmU3lcZtC8fadKbkGgburfTpQ+qCgncajM5RzYROfr2yBAVnnqmJFzOQLpyRFvH+jyizRSDS0VxoInXnvzGsrqfUk0FCOsmgLsw+RCVPlRZ6d7DXE0iJh81rO/Fu5CrdFf102aKwVWLU4p6In2d3aRgeRDLRIWSUCYQ0OtN1MuWmUbSb6pBVw6XmyDJ3fW8qtbTDoQljgobKmDiHWsc5L82VJu0uSG9FQCtPNQTkfWKlDghuS5p9rMe1KpagF89iUNPKT2AVK1IfKNFJXUtdduDVbIYvW2whznRvMBOh2AUfpztbAl1ix8k8jOATqCX3eqqwaUxRFo6O42Wns6ygSwRcNfJaQOb2Hkr/TCBpkUVIQE2PM9HErSlIrOTvK42Io31XtjLtpX5+yiCbSEBPVrhrQvJr3fAZLe9uzLc7x7sf5cf808DLL9vlh9cTu9PXH6gGcu9t4/83/nfPsPvrB7xrcUxn5294Fkx4aoY8TNn3+AX3/8iP3X+Ad9cHLOpE7KopXWaFzcTkqRlf7SltoZtlfDGwRXfvDjmZLrCecVhvub10RX/4MPPUJYxg0GFAl7fv5JB8maC9YrWGm4XQ37k0cd88+oYox2fOrhkWWd8fLtHbCzjrKJsI3748BnX1YBvXx9RFAlRZDmdrbhcDWlbw739BYsiY7XOmYy3zPKSi9WILG7ZVjH3ZwsuN0Pe2L/ia8/vMcorFqucOJEPfLFJSbIG2xryQcUwralbg3OabZmwN95ycTnh9fuXfHS5J5NhtSHNG+o6IoosbRPhWsXh4YrLJzPiaUUzz0j3C+qLAfffvGA/2/JsPeH+eM6iznn/8RHUGjNu+KHz53z1/fuY1OKuU37+p36TX3v2Jtc3I7yXGgYAazVtbXh4esOHHx+RTSqU8hTzjPP7N2yqhG2ZCNO7jcgOCt46vuDrT8945eSa9792Tnxvw/F0zc1mgLUSOBPHlu0qJfkwg0+vZFutwd6mjM+XLG8HDCYlxSrj7PSW5+8dMb2/wHmFVp7FR1N87DGTBhsYypMHt1xcTtCxYzrecvvRHn5gUauI7HxNscyIspbZZItSnv18y/V2yNXFhHv3bnj2dB+1McQnBVnasLwcMT5as1lnaO1xVkkITCJfmvGgxhhPljTMb4boRYzfr8kGNVFkWS9yhpOSPGm4+nAfPauxy5h4r6SZZ3LD8hDvlzTLVEBB7Dh9eMOLjw5kBjZxDPcLto/H+Mzy4JUrHj89IBtX1GXEYFSxfj6SQdfK0B42qLURdrjSDO6vqb81oTluODmdc3E5wReRBNzUhjhrd8e+NJjMir/ZOHxtMEuDO6oZjCqadya0g8C47tfCxlSGfF8Y8XhU08xTyCz5uKLcJDv2edLAbYJuYfD6gu17037w2kuttZiy0v2C+nKA116O1TqBRqLtSRzxVYzNBDw8ePWSVZmyWuf4i5TkVlOeN+IB3m+hEjadVhEvDbPPXXHzjQP0/S1NEZM+Tqge1FAY0kP57OhS4Y7CF+IqFk/URtMOnRzXWgvbPLREt1JzES+1sN8K1KCVMKaPM2FSBr5nqKK1pjprSF9E1DOHz2V9PpcnmBtRBtixxSwlpdLFHv//sPdmvbJlCXrQt6Y9xHTiTHe+mVlZmZVdVd3VjY3b4PYgLBoJnrBkCQn8ZAy88B8AI3j2O0K8wC9AQi35gZahMeq2e3R11pjDzZv33jPHuKc18fCtteNk0pPAJWWWOqSjM0TEjrWHiLO+9U1lYIjO6YDqxxWf+6hH8aMa3TOL+lOT5LoS/bkft3f+aI2bH5+OHi9pWQVSvtFJXhwRlhai0VBbKgLKG5mYMk4wc7rxcEZPNQCy+FbALQKKGwk5CPSnAWFpgV6hvGCdSHGtYI8DqjcKbhZhTxyqzw3sIjBp+zigvpAMrAnA8MiieG1Gb6ibEfDkGo7ujGxueN7B/LAmMOzI7OrmHru88Ji8YCeim6b7Kk7Qi7sEcmumMYtANjgzjOWKMvIogPlHEtv3PaafKtgZUF0TuLQPmWo8LA8sb679cHWk9PxKws1i6saVGBYEjDL1/IYiQjXsKy5vJbqzgGIj4Ys4ArGoOf76NdnZ4ThAtQLFJqfqErzrnTiw2BNwW3cS9RX9sK4ig1rdUBbtS4LJox8oNE/ISGd2kceAwKl9mFK050D3wGH5fc3e1AVB5n0g0i/Focok+3R7Sp3LW8qcQ0F5s50x0Xn2IjFYijLv8o7ASbfctq8IFs2O0u36MjGfqTbDbLktsycAtnOGMbWPImafUs6t9weWkX7hDKgZoBM0xtcUiRW2NVlul/php6/iyNi2DwSqG87HhiOGLvkKqK8imodMK84+Y19Qlq0GglWzO0jKTYOR9VQDFxb2D+UowZ698qm+hEE57WkCkAmUZs/r9jmrTKKkYsLsyQyaJqA90WNycL+gp9XsAnZPNPqlwPwzDzuh797sYkqJ91BDwHCk0c8l6hvWhsxeDeiX5Dhyr2ux8dg+Myi3EWbnEVLqcHXH7Zp9wLBQqG4dhjmTdmXattl7Xj9tSr1VScrrAuxEj6As+z5FjOiXhv2qDQPgfCWhujAG8CAziUisqA2wC436sy3CpMCwLKF3NqX1OvjaQFqPYVnCbC09olqO21K9JwgMgJ+XUK1FNIoAsdCAC5C9RZSSVSKDg58WUNsOYVKQ6bQesTSHypDMTt4DYaKziBVzPUQ/8DGFIQsa6F0dw3u+BCxHP2ROcs11I/nxKbwnti1ZTaUAqYCQH+8RrWMabLpfKMmAn7x9gPeFCGE0Yga5f9ot3nvO/8fbV4ax/C9+Bozlf/WXjOX/75sUES/6U/xkc45fO/8Iv796hhAFWmvwcXOGRdXDBoVSOVwNU8xLsoHfenwJLQP+6KfP8N13P8dK1Widwd94/jF+vD7Hq3/1EPY7EhNtqSpwEvOqx3eOL3DRzbHQZB7rgsmXv/D0DX7nR9/Au29d4s16jp0t8eLuGKWxuPv0GNP3rnBzO8Nn9RI/ef0AQkSUpcXxtEVjDZ4sN3i1WiBGgaO6g5IR+67A4DT6XiMEgeZmAne0RdMV+N0Xz1HXA25X0xFUxigwnXfoOwM/KPhSQoqIbjBoricwiwEXnx8DOmDdVZz8TgeY0qE0DkfTFjfrKYSIePZohc9enmJ63qBtCoiJQ11a9DriajPDXVOj//AI5q94vLk6wmzZQoiIrjP441ePIHWA3xkcv73Cb3z6bbRtgcWixWbDnod2VeHR0zvs+wKfvDinvEgF7K8mkHuFy9UM7rqGOunZlacDursKf7B7BrEx+ASc0AYvcfE7j1D+4gr7qwmlufsCRW3RP1LAroTcaIQiYPlsjdWrBYQVKE48WhnR9Oy6Wn2+AKoAMxkQ68DJ8psSmAY8fOsWIWk+pQy4vVyw561ywJVBe1fzH9GgcLtigNSuLeGchNhqvHp9DASBOHewbyZwAVi+u8LmoyXiyQC3M5g/2mLnJTswNxpOBcQ3JTZFRPl0D39VoF602L1cQJ70mB212L1YoLUC8lGP7z1/id/76VsIXqG4VvRHnlkmmKbQnQCJy5sFE0t3BUSn0LycUQo8SLz84QOgiBg+n0L2Avu6YD/eVsGdWggVAKmgNhp+7rG/mUAWEbAClx+fIk4cyguNwQvojcTwQBK0aQI2u5BQe5WCZALMRsLGAg0AZQDxqIP8tAaMx/PzO7z+35+hlRFyqyAuJ9Am5WB9uADeps9WOCAGgfJG0p/6gyVgYkrbjJi80GgfUY4lOwFczhDPPPRWwfcT6IFBRqGOiJEpoVEAxa3Gxc0jDKceaiehLAEGosDshcTeGfgnPcRtAbPhpOv2B6dQvYD6/gzujJ4x84b3u1u+rj91kHcFZdcLC2wMJaqLCNEq1BcSzVsO5lYzxVQISp8jUNwqDCl0zM4DzFaiWEkETbYjmAgxSJgdZ9N9EaF2Ek5FVG80dEN5qbUCkwt6FH0dYSf0TVc/qVLqpgBeVAxfkvSbQkaEImL6qUL7kMe7+/455CMGLw3LA8DNFTrSCujXJYYjTnAjElvlKd11kzh+lRdkA8s7+iGDidBbMiztg4hiJYF1mWpoALPTCBqwR/Rd+grQK826m5qpoNU1J7WlI0sTL8yY9llsBIqVSiEuEfVlhJuyUqgJFXQD2AWBT/Ynlit+BlTXmgBuRrY0poCk4u7A1MlBjIFa5Upg/5TnS1qgumA9VVRA9VqN4VsAF0NmL+iT9KVAdU3wVnYMPhKOIE2n+hy9J6iSA//enZC1rVYC9U1A0zHgrLqSKO9YvwMAi08CulMGg4kAuHnE9HOZqo+Qaq14LoflQTasWqC+lKzJKViHolqRgrJ4LYRSoLxTKHYR5YcRm3fI/kWdpMoKye+agnCCgN4zmCafg1xjQvk8E2WZ7koGlownk2nL25iAQpaiCsw/jaPsOWhg+jnB22QbxgRT3RIs6pY6b9OkUJ7ZoaplWCQGryJwY4ck61PKdUCunSKYizB7yo11Q5CmLCWzOaQnp7TWl5SkNucS9a3D/gFBmt4L1DcerpJwFeXF0zc8DvOX6fkh1b4YiWrFY9ofH8Bt9mWqPiYZK38u15GgbEqrwDCTKPZMeFV9RLHzyA30+X06/ywwBdYwvEZ3EWbPwCLdH+S6kwsHN2EYYLUK0ClordzwGImQzxtDj6qrAVEUED6mcKXsofYACBJFZE9msXYEOi0QtIFueHzMxvK4pn0tVgOiZtorwJoXOZCphI8MB9rbMWRHDim5NYEc1RFQqc5TEhs0Q44ExsAg2XuG8UjKvM3aws+Z0qZbdwCHLoyyWuEC9KqFW9ITqe9axNIALiAqBRE8ZOdG6avo7CGMCJTRwgMIAbJ3EG0PoRQlrUZDtAOEFAR9OfwnS01z6E+bZK5a8W/dMAYaIQQyqllG6+8F7dwPBBKC94d4CPuJEbAWwhhEkZPMvihrFVWJmOpORPJ04kuVI0IpQKWxmC9BkvtMZQK+MUYC2a/zLR6sEj+Pt681Y1m/9yR+8E/+IbrBwGiPCOCt5QpaBPzBh2/j2x+8xI9ePcT5yQYXPzrH5PkW3kuczPcYnMb1yyX0IjFog8JbT25wuZkBALpXU8SaoTtX6xmcVWMNQVQRYuKxPNlhdTdlrH/kJFeZALcpAB2wPN8RjFwtILYpqjmFzUDHxCxRwy/nFmFvoNcK5fsbNLsSZW3R7QqgV4CI0GsNd2ZhLg3ssUOx7FmpkGRhcqO5bRMgTEBsNJaPNyz1NhHlpUL39gAM8pD+qtIEcqMggoCvA1kJAZgrM/7z8HUgk7c/pJNGAbhnLHrPUrnJZxrNNweolYY/Yjqt7FlIHgXGNFRfR4SpR/nKoH86cOwLh+KNQbERaJ4E6C19ce0zh+q1poToQRglBFFHzD9SaM8j3IIprnpHBijU9NWF4tA1qjr+Q8jevvpCwi4iJp8nyVTH1NMIoFiTTYmS3j29lUlOw226RUD1WtFXGAHp6LMTnuXiw3FM6YypHPxWjB2NbkowIAf2FrYPA8LZAHldoLhjl6RqBKavItYfsKsy+xezr1F1AuWafYq+zBM/Tsql4+OnLwV2z5iEmhmY7pxVCJmJyAxSsaF8sboSsAuMq+WhYIn5sGRPqbTA7psO9ecads6gF9nn4BFOstghh3HiJSL3e/Ka/qruFGMfZ17ph0g9mNXBZ2V2KW009SOONQFTjAnHAINj7PTgX8oVA0jbNvuDL43l5ny93IuZPVu+wKFDr02JsB5jNUF1kwJxIsaU1Ch4LdcXhwTNKIBhyYk4wG0PR9zmfWlif5oSdhUnz75KE/uW12iW/rGwnmADIDMkQvIzJhkcIsb04lyLESUwueD1kWVz0iGF02BMz1XdoTqBwIjsjZvw+JUrntfqmpN/4Xne7ewQYAOQmRgWSV4WkJJmye7kscqBMjRWY6QQnZosgG5iCj4SqVaBTFR1y2L07J+LkpK3PNHO6bRRcEy6jSg3WQ6ZfIfpMyMzb9KSWSpXcayzEJ4AYZizDqFIUsHciZn9jnZ66JJkF2scPYnDPAXDJJ8kwSUBC4Nv0rFxZK+Cwdj7afZIXsZUMi+S9y8lpUbFY2D22TfKv+cgG5u8ptKT0RsW/N+iu+R/TNvLFRTFNqBfEsiqIabtCxRbnlQRKOuUqQtTWnZe5k5OMljpPafpa6Q0lF6/6s6jOdcj6FE2oltKFLsIxDhWS7iSzHEU9DkiklVU6TrMXZwmdahKxzGUGzJvw1SiWnmOMZMlKYwnaJ5TX/CcZgAZU42ELyS9h5WA2QV0xwQ2Zudh50x8DZr35Z7LqA7XHo8ZazHoNQXMxqE/MXAVuzC7E41i4zEcaUgXUd1QRulLOXZuqiFAdSHJMCP03o+gzE40ivWA7ryE3nu4qSLojhGyD3BTBdUlgJR+z+mjsqe/T+9d8isKdKcakzcDw3pK+iTz9SUHBuHIIaC4adA+m8Gs7dhpKWKEN3JMjs1eSWkp99R7B+ED7MyME+fiao/+wRTKkjFUfYBMDJ7eWwbuJE+iHDzDbKSA2lu4JQ3J5q6Drw1CpaC3A0Kh2Ps50VCbAWFi+FyBFMhj4afFYZsClJgKQDUEmKFmCJPoPWTTIyzq0e8qrB+rO6Jm+qze9ojJryish9y3CEdTRC0hWzsyh1HL8fgLHyHaHmFWcXEhV3UoOYJQOE/Q58OBRczhOrn6I3kio9HpupOUpubnWgeUBVnEzErGSNlqiajZ4AAAIABJREFUTpwVXCAQ3T0ZaQgEfvkWAyAkvyt1+Lv3ECa1E9z3VApJf2f2awKIw3BgPzNbmYJ9vnzLPsz7P/+pmOTnqcfyyfP4zs+Asfzhf/3VYCy/1sDy+BfO47/zP/59bIcKLkrU2qKQDrfdFEZ5rNoahXa4vF1guWgwOIUIYBg0lAootMfgFLyXvO6dxNGc7JsPAne3MxwtG3SDgXcSpnDp/SEhRIRSAX2vIQRQFA7OKXqXnYI2qbPOS0gZYEyqEklhPm1TQCm+EW1jIAsPqSK84/K+TFUoSpOdKUqHYaCsNXoBpRk6FAEEKyFNgN8UlCL2GpARceCHoJkNCJ5jK0qHvjEQKhIoqywtkICMMJWDTfdLk3wLVo4BPpARsaHk0e4Nu/z2BnJmEXoFMx1gWwOREnNjrwAvICYOMckOVe0ZMrTXo/wTVkJUDP2JqwI4YgosREQcFEThoUw49A1OHMLejHJEFIEMmYjpb2nMJgC9ghgEYslZv/AJYKb9E2UA9noE2RCA6NiJJzr5ha49YQXi1HOBIbFJ9GlwEhcLLkWprWJnX+khdlxUiJrblw29c8IKhhi1cgxZiUWA2rFyRe3ShGvmILea9TNOQDjBBYQyQK5ZxB0VECsPBO6falJwT/JPZYkhkJiDIu3nIMZqEtXJ0QPGICMgCgb9QCQAaeIYLJJvuqFML1QBeqPg5wFqJ0ejRNB8TizSeIJIZdcRxR3linovEMrk94rs/tNretLGrsDkK5MDFyoQUrVFcdg3gvtDHxsnYAR/XDVn1YxushfpAEh9ddhODhO5nxKsOo4RgcA+FAcZWw7PCCVBeZbh5ZoZ3aSFCkE2L8oktTw6yBNzDU2UcawlkAmIjvukDwsMY8efScEvPh3DFLKTFzx4TgGV9jmH5oyhKTHVzYBjy4sQukngOoXK5NoPX3M/VJ+AZeA2csjMeEzy2yYrqxJ4ux8yk6/7MUylO4BqXx0e5yYcc17oEhHjey+HvbiaNSvD0eH85aqJLKX0ZVosuBcC4+91HeZwInGvXiTX2+RKhvHaFwdAm6te8rWQ71fdYQwA4Ou06ILDNZ33hYFLSOfh3uvlW3pMHiOPy2E/c2XRuPh0L8Rl3C93CBZSycMoQhyBWAa9/ExhLYqvxLgoIGJMoUyH+h/p4xjKkxcTMujmvieWKBfWp4CcHKTjKvoMc0BOvk7yeQ4qsY/pGOZFLF8c3h/5es+BWzn0Z6xqcYfrJe+nCEzeHQOS7gUHZeCcF4pyQFFQh47TXHGSQ2PuV3aUiR0NRqSAIaQFkPiFVNpDLUi6Fk3qdxSH6yvvt+7j2Ck5Xtt5X+MXPxPyPuaxisCxyxQao4YAVxEg5kAevWcNiQhxTGQVIdfMiC9Wp6SOyFDkCpbkb0zvz/vXEADIIYw/j8FjAmO3JKtQGOzjC5lCp8IYUhSMpGdRCejOw1UqJfKK8dxHCUAKqNaPIFpEMqJffBy3qXpWkZChJFjLv+dwnCwLlYNHSB5L6RLwizEF8gT6IzPTmFlDH1m5koAqAILJDFpz7ckhWwcj25jeK2MnZQKGX2DwQiTQTOE/6AcCSoAAM4f23Je3AgSe91lJ9Sf3WI4gOAPc/Nr5fiEIMjNgvB/6k7sv89+8v7dvXwSYMYcN5QCiP+v28yKFffI8vvOf/wyA5X/z1QCW8s9/yFf31nuNT9cn+Hx9hFVbIUSBZ5MVFmWH3VDg8XyDXVfCbQr0TqHrDSrjcDxvMK0GJsUKsFj+YgJTONyuKJkdnIapGNAhRMRs2qHdlxgGhe6mhtYes6rHfNoheIndqsZy1kDrgKoeUBgHYzyKwiEGifP5HlJG2EGjawuEXsENCtNJz3L3KBCCQBwkitpCG4foWaYuVcB+VUMIoCwtxCUDXcrKoiy52lQUDrKRcFaRQRQR5tpAVh4x8DSHRkPKgLce36KeDICIkDoSBAJAx45P0SnEXsI39FvGTlHOOUjK9zYKdldAbjhupMmW0AH4aErVRemY0tpLmDtuV6V+UL829OOddFBrDVl4IADToxZhZ1C/osEeawN5VZANbTXczhBUbxXTZCMgBgm9TY+XEWqjx6talD4l04K9l51CcaMwdnp2CsJJxMS2Ci8gWv4XL2/or5QDJ0wQAMoA1Un2jRYJYDQSonZMHe04oyquNMGrDhCNph/TM+hDWAmzkYCmDEz0EuWNhD/KM6nD9Z1leBgSIN9LlNdqTDVVN4YA1ZPZFbUHKk/ZJzi2XMQuvED9Wo69o1HH8TjEIiJWlACbDe+XvYTaEhwWa0nAJROgmrsRZAoPVNeJrbOcEMTao1gLAqZIMFSs+NrCijEhF0GgvCNgUp2A3ouDNFBTGgkJFCvuB0AAz2RMbkM3YgR9iIDZckGBSbLclmr5XLOT8CUZ3epajPuPyO0AZLJzj6nwQHknoJJk0Ozz5DBNbjPQTAwkGQKmxIaCY8zHoFgjMaEEjmYnRpDCbWHcr9xRyn3i+TM73q9aTmBDmSfuHEOWQ0ZwHFED0fB1gwGEJegq1hiBpm6Q2FAeVwIkMvj5Pe0mBOQQTFsVHiMo1XuMyabCs/syT8wz25qL3cvbtFiR5ySe2zY7HPoZc+VTOKS8ikDGevJGpAnyYfKcewnt7ABEhiN6EvWe2y3vmMgZEziJJgHGPcFJdR1HcK0GoFhFmF0ceyVzcEsGn5mh0+3hnBVrPkd1Saaaqmby+IsNt2ea9JiWv2cgaHZxPIY+petW13yMSmxjsYnQbZL9rSnLLNcB0gLNY753s1ogJ7xmBk23EWYTx3RXETIo499Un9jH1NdqUrOVtBHFnq+Za1SAe9tP9SgZVKqe2/IV5aSqz77I+IUFgJA6XTPw0y3BqwipM1KI8fjk99uohMgAJgEvSqczOOV5dokJzsDWNLz2GNiS2NXUcQqkc6X4OBH5GKTjqNtABnjIzHhKf97z/0ref/ZksroHAFQbRqBlmnA413v+zGMVx8f4xMQS5EaOzwNm61mDs/JJOkw2NoNplcCeTPJReS/xVQ0ESEELyD6OIFLaOI4vKIHy1t5bjOB+hfQadiJRruwIJKMg6y0HMr9coBEwm4GMcaqbYeqvh6tlCtwR46JIDsQBgOKmJajdMelUhAjTuCSPTamw+tAHGRXDcISPY38sgTaBCcGpg+w9fC2ZcCv4vBz0k1No6SskeJatZU1JkrqybkUiFBJRC/hKQ1o/vu7IbN4DQlEJIIFLMTheS4WC6Bzg4yhzFS4c0miBUfpKuTSBHAExZak5kAcuAUgfACkhup5hPkodFgSMYihQiCMYFIkJjVIkGW8Ci84Dgx27PPMXZPq6B97iYJOHUh4qQqxNzOU9MPgnSF1HYPylhNr7NSN/oVsGlX8e+Py63OLP4OsrcvtaM5ZHv/Aw/if/y6/j92+e4s3tAufH2zHZ82jS4vXlEr/yzmf48c05dncTTJctmm1JoCQBWAHUHugVjp+ssdnVCE4i7jTE1OH4ZEdPXadgzlrYXkPIiEdna7y+XHIQa4M4c1BFQLBk0/TRQOYxCMirAuFhD6wMHn/rCq8+O+XzIljT4SRM5eAua4jTHtELxF5h/mCH3WpCJq3yZOMUP7DNYiBr+roCZISfJdlMqs3QGwU395g+3GN/W5MN9JyAm7UEvr3FcDmBGFhvISpPNlEHiI0hUzdxkBflOIsLdYC5kxieWpw92OD2RyeYvrvG9s0copdcbZ17HD/aYPViSZYyAqfv32C1mSB+NoGfBIKsVh1YsSKg/qRA+5aFWlHu6p71EDcF4ukAeVFyclcA8WkH+aKCfWhR/7RA+9xBeAF90sFuWU8SFb1lxVqifeKgt0x69DP2dOrHDYa7Cmov4U8szOsC7ihwUqsiigsC05BYCj8NiCrC3FH2Go8tyo9KDCcpJOTMo7xSKWAlAiZCGg9lPMRPpgjvtsBnNSeqkd4tiEhQZgk2cdoDV2VKeZQY3umgX5ewxw7wgqmQ7zpARhRHPYa7iuByYWE+KyG/tYP/6QxBAZPXAv1xhJ9E+CkZxJgmdfKDHfrXE5g1AaCdZ8AHtG9biFZBtQSp/QNH2XPDkAl3xP20c8p47cJj+kKjfRgQTwbo1yXKW4Hd+xbV5wbdI8cxejEyw3qjRiYjy2ur99fAby3ZN9mxVzNOHCY/KjGkTsVhSY/acOy5PzqivpSws4jhOGD6QsFXQPuUPs8soZVDCopJNSDdAwaPiACIf2MN9+GC9SEV+zEnr2QKLklF648HiJ2G3knUFwK7dwLq1+zs9NMAs1IEnR5ovzGgeG0wPHBQO8XFgiq9dwzlz9IBCIltjgCiwPwjifYR0zmLO4bOSCtQ3tC7Wd1E7N4hExvmHqKVMGuJ6obhHZl5yoEr5QrolxirRUJi/6QlyxsMMH3JgJOok/8xyb2X35fojwXaZx5HHyqsP/A4/V2J7TsC9QW335+l5NGYFxyQ5GdA98hh+qmG2XIMEATqxZYy6/rNgRWzc2D+IuDyr4HnJALlDbB/RhDZnwDVJT8qm6fsitV3GvNPgN3bvM67M0qQg2Hf5vKHAs1jgcnrCDtlEMrmXYHqGmgfUtKdJdfdWWLLphFHPwa275Ch3b3jMPuYQTLlLdCfcAxjXUbB8JbmcQq1ScCTIUBMEvUGmL/gpHPzDQbtyJ7JqvtnPL/LHwDbtwVmn0XsnwocfRSwe8I00cVHAde/IrD4KT2jiAS2+Xxm1tfXHMt9r057zgn84hOP7XMFs+exiDqDWHocXU2PYXUd0Z2ThVM9gbidJuAik39wB0wuAoa5wHDE+6avCZB2z/h+mrxhL6WvGOhz/COP/UOF4YgLEsWGr1nfBNhaonnEQJ3ujK+9+DigP+L+Ty4CXC0wuSQw2T6T0M1BHlzsDtuYvQwjM1euA4YFexerW586TdnNmWtZMsPcHfNayLJi07DOw5cExdM3Hv1SHpjuJE/eP5U4+WOH7kSh2AU05xLFlkAwLyS0ZxJmT59lkUJ1CGTpe9R9RHvKCfbslUN3rFCuA/YPFCbJY9kdU37dH0sc/6DH7bdLlKuI+trBNA7daTHKk4eZJHj1GH/WbYCdKpQrgoP9I4Niyy7O+ppeS7Pz7Ml8bDC5dOiP2B0pPQF1e6JRbj1kHzEsFEwTUKwsRIjYPa8IuocIN6G3sVhb7J9WcKVAfePQnmoUu5B6ZQPM3o0grz/WmLzusX27wvyTFu3DCuWdxXCkEZVA/aZD+6iC3nuYrUVU7MmsrwbYqYbZO9iZhllb9GcFqsse0gb0p1ViKJGCePg5KHyA7BzcvITsXUpHlfQXCmBYFlzYuGoQCk3J7c5S5tunILpJwRqSQkHthzFMJyoFPzWUyhYasuF9wgeElOwKJRBKDbXrKZ1thlEKG0oD2fSIlYHYd2QdfYAYLBACwtEUoukRpxXDeIzmfUoSHBpNKWwCb2MgjxAQTYdYl/x5sIBj4I/IXkop+dz7zGTGBFme69yY9MoPQ4X7wT4xRqDtyDgqdWAd832Z1UxS2v8X8/mlXss/9/bzIoX9z34GjOU//mowll9rYDl9/3H8tf/hP8JtO8HN7YzSTh2gS0cWckXgBYDSSID9gKmzMOp4kDBWAXBklKQjW+CrJOEz9OtFCcoRM0sX7knu7kmkwoQyR72XY5pg7oiTVsDNCFZkz9TDLFnUrcBw5iAblVYkyWrlDsLsVZQ9HxsVC++TwuOwP0lqEsoIs5WwU8obpRMJKBzYHjuL40qiPXOQW4JuldIOc4eebgWGJX2MwRxWvosV0yLNiuyWXTCYxU0TK+YFIBnGAeRVasFagJQSmdlCNwt83TqivJWjx1G1Am6WxrznRCuY5HFK8jNp07gEDl7J9I9euiTva7iyHgyP4XBCH6duyDC5SYQ7caheGso3DQM63JQs1tgvF8h8yIHnFZIT/9zPJoIY5W6hINNh50nKp7mNLIXMCZpmc/Ap9sdMfNQNmTg3IfOVUwxdndMHD12EWeKXZXmZXcjySmkPUsUsvQvmEIyRn3f/NaLh5NqXqR4APLbCcUKWmSIWoycmxKT7shdRI3lNvyhRk44eR67oH2RtecVftxiB0xj6YpG6JtOEMLESXIUnW1VdJwYtsWwQ7LbTmUFL58VXfF6xiaOsL3vgyBZGuAnPT9RkpYYjAZ32O7/H1IDxMyB3XsrhsN++5Gv74rAPvk7jk4DZcGKv7o1vlOTdk3jl7WfWB7h37btDiia9ejxXWYKoW/rt3CT5PJNcM/dk8hrlc3zB6oHMApmGx0d1aaU+h9msOUnNXric5phlhVnuyvd3khsOZA8pfzwkWqqery3tvWOgD/t4X37HPsokF5SH4wV88ZjZOcNlfMnfs5e33FBOmLcfTJK7ynQddxg/r9Vw+Bn8OIa0qTNSIE1Y+Rjp4sgk5XGM0sQE3vP+ccfSdTbEcTIPkLnSKYQmM1sHOSBQ7CJceUgEzSmhwZBJyp8huuN4+D1ddwXPkfQ8X94ImDaO76/sjR3fU46TzbFnUfJ9Mky5XXmPmBi7BuXhvZ79mWRrA4aZHI9hlt7mW95W3k4eC2IGfgGuph8yFIJyyRDH3+/Lk7PkMwfZ8Nq592KRrKZ0B1aP7y+R/Jb0HfqK0sf7kmT6GmOShvKxuXsRSV4btRhZP9nz/0IeC+XIAaGUI9OYU0t9SXYvXzfBCMg+1VjozNKKMc3UlzJJW5FkqRJyCOP3qEUKoMmS3OQDDVn66hCNhC8VVO/paZzxA1iECGmT53OUX+djmTopVZb0RvoWTQKm1o8AksxZ/iy+dy0pVm1kGXHeLllHeiRFiAilZqhNzJ8jIvmqCVh86oQUPrN+BHPCJ1ZTJBlroSBs7uJMn4+lPkiIbUAsFYN0wNcSvSf4Swxc3v+oKWPNKbH573D0QEIConf4QqdklnmGQMCWwVcez70eyeyxFO5w0Y6/3wvTianb8gsdlhkw5sqS/DpKfgF83vc8jp7J8cUCRu/k/b/ntNcv91DmJFjvASHpscxvR6XYY/kn+Cu/fPsLY5GfA2BZP3ke3/lH//qB5Q/+268GsPxaS2FDFPBRwgd6DqtFj2eP7uAGDecU5XSLHpPjlgzc1CIUYfTbyY4MV6wDymUHFAyu8TMPexQw/eYaYeGgN1y5DDNPhnNuudpfBsSUnhhmnvcDoxTPLjxkKr6XDpDPGkrxegERk7dNgCxeTN6n5JlTHf13whIoqkaQ8ROAfWAxLCKGI65SIvmeggH/iZUEXf7YoXvgDr4sGWGnhwRHXx3CFxABc6WhG4JXl0ro5UCQFFT+MI4IRw5yIPNiZ9zX3E0nkkdNNQTEsQyQJ8Mooxx9I1MPX0a4WYCbMHVS+HuAOZfQJ19ZeZsBONCfeYbizAjOKPs6TMCjRgLAafJseYzcJGJYBsrNdPIEthyTnfOfdfXSUKaXwn7cjCBaWr5Gfxo4cYwEjFEniZylzzAqbqt97iAC4KYhyejEGAQzHAVKGM1h0hnKdD5ymMwgIPsEKjdpH4t4SNlM4FEEYPd2GENbxjCNIn3dmzTuvmVHn1/2wAVD+WZ/yoTODMZ8HaF39NjZWRpbL8aS8GGRwN80onvsEqDn8fAVQW8oCJBDAQzzg/xv9OCBTKlq07WbgOhwFKG6A9jIkjkCeD4GgcfSVwkspfePm2CcyGTJX54UtudxlNraeZLzGYEcdKMT8xc05XzdWcRwHHme0wQzmC+ChQyW++ODfNJXZIV8xQnzsODxjZp1CsMi1UKYw1cOLfIlEniLEI6SxzxZH5YR7YM4Ts6zhFS3/L09S5ULU7I0bsbr3yXGKvsFM/CUPYOQ5JACc3puz9V8z4+eQWCcfNspx9gfSfhSjJ4+X7LWgSwKmR9fCtg5ZYLZR4mQFzHI5ohAoMh6AI7d1axbMG0cF3NCSiKlXDh9ft3zBdo5QU+/TJPdlDiZF1LclJNSV4oxzCiDCYI37mtm5Oh5i6OsNRhgmAk0D+mH9jXDYAjAgX6RVBsVATyTNgPsBGjPeD58kY8dQ2t8mfaj5Pm1df58JHtG8HD4HAsa2D+SZBtnAt3pIYgHOIT0ZE+pN/yyEwFXc79Hia7ndeZK7sew4DkYr2sjMMwl2tP0eZ48kf2C15IaklRR8vzlxY1+IdGcy1FqysUHpCoPAd2FUc4pPfejX3Ky6wuONYcSqfYAPFwl03Um02dXCuRJckfKzMMIGlUXR1DuU7VGDr9iiFre1xxEJ1CsWF3Bz1aB7lgxYEmL1I8IDFPJyoyFhLRkIL3JC8wigScBX8oxgdQXfCzHJhkGU8qUtprAV7jnX4yAnSpup1aQg2fqagKz0oUUriXT/qSQKkdwyo0gAT+eJzehXDKUcty/7CGMksCqP614ngp6De1Mj97HYDhmOXhI68dFq6i5r3aZmEAt4KYGsZDwE41QKIRSEdglsCdChK+5wmLnhoB2So+iL9UIHt28OCzqDA6hpI3F17zgQ2Xoa5QCyN5IRekqAPZM+pCAYIDsBvja0L+YQnkyOPJTg1DymOfwHRESEA1grUfyWHLH40HWCl4/yNJcS4AZs6Q0BfXEyiDWBUavZKrtiGX6m9EQLvk+7aHPcgzgkUmumoJ5DnLX9F3KA0hNslXEOKbBQkqymzqtbuVwnj9JWhoDhBAQKgFMgABSKj5P5W0qAtEEKnku5Ch5zWAx3pfB5vvTV5bE5hCfcUx/2teXkmf/8vbVu32t60ZCEDDS49cef4z/89W7+PXnP8TrboH/8Okf4IfNQ7xd3eJ/+o2/i/d+9RPMHvWYqgEX3Rzvza/wzz7/ZqojkSiUR6kcqocOL9ZL/IN3fwe/u3kLD8sNfqd+G//ev/kh/u/bb+CP/+gt/OL3PsP3P3uMx9+4xsRYzE2HP3z5FHU9oG1KHD1e43vnrzHXHV42SxwVHX5w9wCDU9hsJ/irv/pjXLUzPJ2u8ZPVGW5WM7xzfodPPz/D0fEe3z1/g4flBv/bR9/FLz18g8tmjn1fQMqA9XaCo3mD752/xm998g38vQ/+EL/x6bfx/ukV/vDlUxzNW1xfLrA83eHxYoNNX+HRdIPf/+wZ5tMO69UED843uF1PUdUDrNWoCovearTbEpNFB2sVkLoj4/t7qNKh35WYzju4bYXJrMc/eO938L9+/kswyuO9xTX+xZvn2C4qhE2B995/jc9ulghOwfcKv/zuS1w0M9wZB2s1xOcViu+ugFWNx3/9NVyQmBU9pIj44cuH6HSJX/u3/hj/4vO34L3Edx+/xg+vHqDZlYhO4sHfuUCIAm9uFzAq4G/9zZ+g9Qa/9+YZ+s7g5GiP2z84J6iZefzKdz7Gv/z+uzh5usLt9RxybfBX/u0f4Xc/fQvLWYvqbYvtbz6EeXuPv/32T/BPP/w2YqsBE9CfcmKzXO5xdzMDooBcaey/x9SLzgvU8x7bdYWnT28BAJ9/dIYn717jW8sr/Kb4BRTLHt1dBTEIPHjvBoNT6D46xi//+g/w49sz3F4uMFm2aN/MEGsP+ywCO404dcBVAXdqcfyhwdVfD3jyzSsMXqFQHq9+eg44gfjNBt98cIuffv8J/aACCKek9v7G+x/h//qXH8CdecjKY1IPaLsZhsceDx+v0P3GA6x/0UJtFP7W3/w+/vmLd+CDQLcvYKYD9jc19EmHo386wc3ftnC2gPzGHuGPZ9C/skLz4yP4KuCd9y7wqX+E4gcK9oMGgxcIe4OT79zh6sUxpt/e4GTa4PVvPoOdRbhzC9EpqOOe135xluTpEX/3ex/i5X6JH508Hvs3yzcG/bNUYF9TFl5daaw+cHj4zi02v/UA/XngotHbqcsTwPzRFs2mBu4K5KXp5jFgjzyElRjOHPpGIZQBkMDR9zV2b3Pie/6dK7TrKexVDfvcQtwUOP/gGrd/eA577KDmFuJFDbdkyBKedHCf13j4SxcYnMbm904xexnx9D/+GP/qkycQtwUXVgIB8LBMCa2VxPCtFsWPa8z+2jV2v32GYRmgTgTcJAAKUMc9wpsK808kmicR3RmBsbCUaqtWcmHomxs0f8ROUBG5uHP6777Ciw8fYf6xxO6dwGTgMyC81WL+z2uYHbD6lQEnDzfo/tkZ2ocB1ZXE7nkCqw+A8o7gp/lux0qjf3UE6QT6U0/VRVowKNbA5hctZj806M4i3JlNgVqAnFrMfrfmQsWCYL15IjA8GiC3XNCycxanbz6gjP/2wwUBqYuYfvcO++8fswJCEQACBJvdWUSxIoDbPwsQXmL3VkTTUDGiO/ZQCgfMPuPjogGsE5h+HnH9KyLJxhkiYucC7XOLyccG1Q3DVprHybf5qyvsf3uJ/jRQRu4PbHdUrOdYfByxek+Oi2jCU1I7LEVSIgB3C4X2qcP0Yw3hgeahwOQNJ2FXv0xPcfa/RoWx17J5y2H2kUZ7TsbQJ79YZtYzo217Avy7X4oIE4fiUqd9lyNznQHp0ccen/8HHke/V0ANwP4pqJBIi0h2TvVNMEw7BoD2jJNS6VLn41SOwA0Ado8V+mMCnu5EQHcAArB7pqkKUMBeCxz/yGP9DQU5UIbdPAsYPtaQDuiXCvV1xPInAy7/agmdUnPp9RPj4ll1RRDbPNJjunJQIi3UCPRLqiKq64jmjB8F3Ql7IXPvZBRAfcOFAy5mSUwvQhobMLkMuPh7Axb/R4WgC7QPBYIqML2w8KXE7okelQbSUQbrJgLVjYSdCpiGwJgJw2Qam3PJzsZtRFAaw0Jg/jnQnvI509cBxc5j/7TCMBNj8nCxDaNSYPO2GZUQxT5AtQG7x3r0Ivst60W8EQhaE2gXGvWtR/OI8s/uWKKYKyhLNtwXAvUtpcyqxxgAVN153H53gsWn3OdguFBT3bGP0s6m8EagPxKoVmQVh7nhgoeNUL2GGgLaM43Ziw7Nk4rvnccTXj91PS4K9ifTtHgW4epyZPTNhsE86/d5P/srYwLjclQ/xKVBubLojyYhCv6oAAAgAElEQVSHvspQIkoBP9HwlYTsA8JRgeqq5eKTkQjHVVJzCKi+gNoNGB7MCLSnEpNXHULJRFo5eMh1g+6tZWKjI0G3lqw2MQTceu/gphpm1ZNZlRKxpsxCaAk/LSBbhvqEE7YS5HRZYT3CpIC63TMFVkj45YR+0ABgltK+AgAlIGxBQKsEZbPzelxEkJsGiBHheA6x7yiNLcwBOAIHz2iMo8xW9MOB4YwRqCt+tw7QCrHrDxLYGCAmNcODvOe2hSBzOQwQRXFIjeUknnUjAKL3B3B5z4OabwdGU33960aAg8ri5/D2tZbCnn/nNL73T/4Rns5YcD8EDYmIZ5MVfv/mKb59fIHf/On7+OXnL7GzJWamx10/wXm9Q+cMXu0WY7flm/UcUkY8WmxxsZ3hZNLi05dnePrkFouywyc3Jxh6AyFYaD+ddwhBom0KzGYdSuOwa0v2MSYApmTAvOoxeIW79RRSBZSlRduUqCc9hkHDGI9h0Hjr7A4X2xkEgEk54OLqCHGQOHm4QYwCs3LA51dLTGcdhkGjKBy6zqCqaOwOQULJgBAFpIjQymPflgzXAaBUgB00TOGwmHS4uDjCo0cr+CCxbSp4JyFkRAx8vN0bmKmFKSgrjlFAyYB9U8K3DNyJXkKogNArnD3a4G49hR8kJosOfW84xn2B6CSqRY8QBIZ1iZPHa2x3NUzh0O5LfuiriOgkVOFRlBZ9bxAGBVl4qtuiQLgtIE8G+LXB7PEOu6sp1MwytRbcRrCkUMpZTym0CTC1hbcKoWVJPABARsi1hnrSjEAEyfdZn7Ro1/kDW9CLW5BqkXuFuCRw06WD3RcMHwIQdgai9hArA9UIxHda+OsSceYBL6AmDn6QEI2GWAwIVjGYRwdARYithn7Qwt5VEF6gvFawc/r5ICPkTo0y61iwf3JMrAUYwDLxZLzvDAGTAKXfOqB6UaB/4PmYFAgEGYEiQF8UcAuGDMUqQDQEenHmoG7N2EFHto6BPygC5EajvJFonzqgCAw/CgwOiiqOITe+iggLB/QSZqNgTxzkXqG8luiecMxRUjYmBrL1xQ33P+oIkRJ/ZfLzugXl5Hqd0nedoG/zKPLYTBywNinsh+x6eaPQPaYp0Nwyst/NUhefJCNM2Tt9Y2ZLWbabsBrH3OqRuRWRTLivyH7rRoyMsS8BPwsoLxXchLUq97seizVZrWAi7NJj9pHGcHxIgjQb1tWEXCM0ZFkkwUb7iBvSe/mFhFbVMsgny8cBjBL4kPyU0pIFzxUlOYiHk7mQKmUoVTc7if44YPZCon3AfdRNkjnOmaCbU3uFI+PpakrIVcuxqu4gvx6WXP0H2CXZLyOqawHPelv4IkINrOcpNhxnlDwPI0A7jdA78QXmOOiI+ooAIjNxqge684Dymp8NZpeYbmCsnOmXBM3NE25TRPov7RxjBUmWVmf1hIiJrW/ym46Tet3E1CdIBnhYJIn7VqC+orQ1M4+6oRzabCO6UzF6Oc2Om+yP+TMrRZCCjihbN3sye3nfhyMxVvvcT5Idw4Aq+nSzjNqXfI5qE6Of2HX6TVM3bGLhVeoqdHVKQrUMGooqgTHNceqG+2G2vK7IjNL/JwcCU5Xko0z+jaPUff7Soz+S6I4JCkwT0Z3IkW0UPjIRNqs78qS/i1BJUj2qEwxZb+nY7ajbfM0f5OVZLaFbbj9LikXIEnvKhKMCqlvKb8l+Y5QUDwuBch3RnohRLRFSijMTYSO8OUjCs7pklI5GHiez9eiXamTOq5XHMOXnm+7DqCppHiiUm8AgGkmwXOwCVJ9qbvYB3YmC7iLKlYObKLLIKVhHhAizC7BTAlpXSph9gJ3JsXaGVSNJ6pqqYHQTYGesGdFtCgIyrFcxzb0AnRBHeXK5cmR4vyQDVl2qRWkp0VV9wDBX7NucSVS3bpQCi1RlolvPczRNgXorC2/49/ZBiXJlx3TYnPwqAgOY7ktxyUJyfMXajiFADHRKPY6gBDXUmr/Lg/Q4alafhIIsnbSJVXRhTHkVNgBJGhsVJaQMzQGiIdMLgP2VRhH4uRRUNTj+Ld0nB4exuqSgFJjSXHeQtI7sn+DvShHIpYTX3HMZCwO5bwEhECYVRNeP7CeAkR3lB+M9yS5A8Og9/ZUxku0EEBv2aAljEPsBQquDnDbLZe0B+I0YI4URfeGWvZr4ErD8M24xRsS+/3Mf96fdvhJS2MfP4zf+03/9UtgP/7uvhhT2aw0sp996HP/+//zv43WzgBQRPkicVnv80eVjvH18hx9fnOOts7uxduRmM8V80uNuPcXRYg8pgKZP3TxRQKUE2ElhsWkqvHd+jU/vjuFSHUmMwKS0KLVDBHB5u8DpcodNU0HKCCEi+l6jqiyM8mi6EkJESBlxPt/hdj8Zx971BjECs0mP3b5iQuyig/cS7abC7LjB7maCh09W2PcFdtdTvPuNC3x6cYrptEOf5b7JkFFVFl1nIEWE3RSQE4fomXg6f7BD1xkczToMTmG7rlFNB9hBwzsJbThj8k7BFA79tiRYEoA2Hlp7hMDEWdcrTBYdTmcNXt8cQarAUCMRxzqV4CRU6eF7xVCj6xL6YQNEwbqQIDjuvYY4Ggj4Iti/qSKTX0uPkMZmOw1VMG23azmr8o2GLFnR4jYF1NwywTYInqtGoTpv0e/5+NhqVqK0mr7PCMwfb7G9maKYpYoUHSBkhG81e0C9gNimVFcdCaKTjxdbgzjxBNhpn6TxCJ2GmfcIQcJvCtaN7DUwT7M+GRH3KTV2lgJ/tgaylfBTD1QBcqUhH3VYLhpcvzoi0L2cQEw8q1Z6iXBkIfY6pZ8KRB0gao5FtIrgqvAcZ+UBx/RabMwovwbAflUdD0A1hRixbiSy87STTJ8NPG7Ci4OcroiIdapfqQJyJQ28YL+qT/UuToz9qAiAXmt2o1aB4U+dSGxEknwnmbjoJRN0B0GgbNI2B5FqOwieVCfhE+jOnrrsZxMBCFUEPHgOND3H4zaS1/B+N2WYhDF8KLN/uZIDSCFLxyFJa5O/N41Fpu3m/lP7wKJ4ZZhIOg2j3FivCQz1nkyzm0RUlwwHyn2vuQ+NE9QkF2/IwPmKQUoHTxxGlmkMdImHiW0GdfcrK3wdx8RdBAJt4ZJ03xLU2nmEbg+Ab1im2puI1GEakT2beifG+hc3paQzS2xFYCLuWHPiU7VKktxGw+1l/7AaCFiDIVhl0iQ94tITLGdmb1jwWOQgo3G/krw6+5TtnN9zd2t3ElFfCrJcyf+cE3hzv6qveNyzd1b1SUY84TGXjj7eLLN2M94vPYH1sMBYO5K9yzn1Nne/Zl+zrwh2cwfm6OVMnlxvDuc2L2RkqXQOsckJrDnJNlfGZDmy2R2AWj4+udJC9RjlxsEwSVgO7OEEKJ0224O3VSbQm497HndegNANQUeWNGafafZl+4r7ZnY838PiHsBJ1+kwJ5DzpYBq4yj7t3MudOiWYLDYRbgKY9fl/XoW1qIcjqkvD55HV1NCnnty82sPC4Hqlu9rX+EL1Sjjcb/nXc8JyARrmVFO4M6TDVKWcuL7VTnZU5q7ZqM6eIBzYnROi71/HoMSMG1Av5Ao14HyXnU4xmNVTZIHZ6BtGoJEXwjo/gBY8013yYOb2EjKctN1GGK6znhC83FWXfK+lnJM3dVNwLBQo89VDocU1VBwESP7cnnQkRJ/BXQb4CYE9XrvEYxEKAX0ziOUEsIlb2pHnyvfewFuolLdiDx4McFxByO/8Hv2DTPVmIBKdR7RyAT4fAKAafy9QygPIr9QkJmktNeMoUDSHryQblZA9h5ycPz8TeATjtLcWJrx9/G+8QWyT1Qg5zcAOPhE78tLv1wXkv2a+TVCHAFk1IpsJUAAmKS1X3g+QL+mkgepbfZJZoax6yDKErHrDs/NE2XvgXAv9TXVjYyey3uvk38fOyz/ApUjX/e6kb8Ell/h2/Rbj+Ojf/xfoqwszuc7XG5mmNc9Ll4toSYOR/MGt9dzoFM4frrG3asjmCWZxroeoGXAejWBMgHhTYUwCZBTS7lXbzAkNkrKCNfq1NmoOQGfOdZ0xARmVDj8vFOAAjCzMJXDsC0wPWnRvJxBHA8QEvA7DdkqhLmDvjGUjQFkfCIAz/qF+E4L1+rE6AkmvDoBfcYkVKgIsVfAwiF2CnJqEe8KRBNRvWaJfXjQI3qCPVM49BcTiOWAYMmeAYktioBaDhAvSSG4UwtZelT1gOZqSuDXKpg7Cf9uB/lplRgLAfG8gb+ogQCEOkBMHcRNgfKGnpjmmwPgJNROwh876GszdhGGOkC2ks9bDsBlyb/1BDSQEaIIwEYjlgHFpcbwbGCi6zQHDPGfMyLgFx4Igq814/OLKw07D6wHSQEVZi/ItJkAc8XAHj8NBEQBEE6iuFaQDmjfoWSvuJXonlroOz3KIP0soLhJDFgNFCsBZYH9W0xSDSYmryZDjMoVkyNDSTasfkN5Wigj7JwSu+GE2wxFhH1oUb3IZc/86k85cR6WAdOXEs0jsg72xEMMAiZJoPROJD9hxPwTCTujFHGcVCr+XF/KMdLfzg7H1NeR5zDQ+ygHQA38pzUsmYxbXQPtI/o/M7s1eZ38ZxOQTemB/ozA02zJiOm9xOxlxN13mYhqZ5zkCw9s33OY/1SPTNn9MBTdAO1DeoDLW4HtNz2KW4VizYmvnabJA8jICU/gMXshsH9Ohm/6ShxkmTuCme5BJCBInujqhiv3zSOep9ln3LfyRowSxe4MKRWTHkfd8fx05zyGIYWpmC0DdAg20iS5j1h9J+D8t3n+23OmdAYj0J2nzxbN14uKUj4A6M7pL869jm7G67W+zJJNJpBCAsKRSYuKIAGJcTN7gge9AwNAJLB/RgBtdmTQjn4acfcLTGAdFmk/LwgyfMVzmn2vubqjO+VxNTvep1pOAM0+YvuO4HPAsTSPI2afYUy37E7FmGo7ecNQIwaExTFgp30ooHc5ATeieShQbPnz7qnA5CJi/5h/G44I0oICynUCR2nBodgShEgLtA8EZp+TVZxcejQPFMyOx61YxzHUinLDiPaEqZ8ZiKqe33NwEhldvkcmbyJ0FwAh0J4ImIZSzuahRH0V0J5LVLfcj+qawUBuAiw+9WjPUoJxwCh3NC1DcJi2KbB/KJHrblRPT65L3tb6OiTPYgJ5CYANRwQw5Zr7XG4iVB/QLxQ7Lh330xuBcuPRnCmGIaXalHLrsX2qoQbWrIgIdEsCOzUQLOwfShSbiOrWo3moGdbUYwRv1ZpjK1ecPHcnCsWeLN3+kYLqMI7FpJoVVxEMiZiqW/qAbplXjxKgbcl6tsc8dsU+0itb8r3nDVmaasXJ7DCX4+KLLwXqGw85ROwfa0wuPWWUAISPsHON8s5i+1aJcu3RHylMLix8LWFrOXZb6i6MzFxm0YaFSioIDzsny1beWLQPChRbj2Gh6PE80lB9hG4YHqQbj+ZhgeqOE2k7Uylgy8PsHOxEs9ZjwnmD2TEtlaA1oF/SJ1ldD3BTDeEifC1h1hb2yCAYgeqyR3deorxl3YbwEW6qofcOoVIIKVwoGAk7p1ezuhoQFYGk2VL22Z0UqG4HvkapYDY9fG3IRNqA/thAtYHnb9VjOC5RXTRons9QXbTwUwPVOnpqJxpmMyAUCr5ihUZ53TJwxwe4qWHFSamg9hY+hQ5J61PoDn2Nh75GwC4MzIaMJRUnlGoKFyA61oO4RQXpAiWohUpstBxDfaAlRGIYxeAQSwPZDEBk1Qd8RKyYDJtBViw0pac+MKHVefocU9iP6IcxgTXWBUSXQJ+m51P0AyWpKeFVdMMIHEfGUalRmsrJG18jasXHh0AA6Ty+UP+RbzmZVcrx55i+C2MQrR3ZxBHUCYkxtMcUlKfG8CcCyvy8ETzev/1ZQPJP8VP+XADLf/gzAJb//VcDWH6tPZa1sjg73mLXlbBewTmFXzp9DS0DVk2NwWk8eXyH1z89R2811NxCKUpZC+3QDQblxEKpgOZEAb3E+Sm3JyX7HRfzFts902WVCoiVZ0vGxEJrj3ZfopoOOJ3vcbmiPt7GAnCCQErxjeG9RJx6erm1h9cKYeoByZATVSbWsJdQCwu/NbAlgK0hcyTSKlQZIGYe4VWN4lkD22lAKhSTAb2tICQQFwS9/RmlkUXl4L1EcJIeysTYwQvEykOoyPEGsrbDkQOCIJAe1OjjQQQB6b5CDIBdkmGDFVBBIhSUdMqpO/yzriLkBzvITgOVR2xKCBXhTi3lmnUgW9bIMUXW7AWGY49gUtJHAD14muBxOCUD52asL3FRI0w92a3MtAUBv/AQReACWgFEE+GfsMpEDgL9Cb0XCPSzxf+HvTdpki1LzIS+M93JxxjfmC+HysqsqpZabQgzzDDrFcaCPcaOBnYsMAN2bFhDA2tWLPkdLLRC6kbdNJJaXcrMysyXL98QLyJ8vNOZWHznXI+sLknQaplVycrNnr2IcPfr5w7ufr7zTXWAPCoE5SFbgv5QSvrZOtaW2AUn/KGKgA4IZZJaaiAWgJ959I2A2UjE2qN/HKEOKskgCbZsSrjNCaiuAcazMK3chgKQiYnxFVlD1yQWsaKU0pdxkpMen/HLOpQ4XSchAdU5H99fM4gkp7sy2ZfMkoiY0oF9lY5TCcqYFg52TKuqgkAsDGllPySp6JzsVCwoHfN1wLiiN8kuwySjxcoCew0ryEqKIJKkLkB33LavcgqvnIJxGG4Up3DBDIpzGAoCWTJfCowrvt/MXsIuQyq5J1AcLjCFAAWd2Kck00UaE4Jk8mMvCKBy4ElxAhLBELTk7sWo6dmSPrFDBZk2X1IaGnqm+4YiqYUM75OjgOok7Iw75mYRoxWndFTO82AXnJQPZ3we95/gLiqygkISzPqKclKXQnaKbQrdmZN1y5P7KAm4xlWqfllxWwyc4ukel1ypyR2Gbh4xdjlBmV8e+Zp1FcN7XJ0e2+CBFI/jDSnRmuMh6B8XSRa646p8DorKzKCrKNH0ZWK6DBAXeQFETEDFNkDuMIzmxFbnvkVX85pxc/C90SMBR17z/Rm3dXySw1swVXwMyc+ZJaa+oldOOiQZalqokalXMcki83FwtZySPccl30i5UB4CP5BNhhROdXyskmw2TkyU6iOTSiXQK0mm8UE4VZQMAPJlujYSMxcVtzGlFxuGDFnH8aoRsLWaEnT5e1o8mkue/5ieC8CVMrG5YmICcyCPKwFlT2E2w1pNxz4zjUjhOb6gby2DchHlKXwss4cCKcX3FCyEiNTVKCdWWaQaEUSgX538rb4AQZ+LsDmJNjFjiKfFKogEvGsJlRZZQyFgZ5rBRel962a8PsY5L/zuSk/pwTl4KGgBUaRwKwHKaBOL6usUQmQYzBO0wLDWDLgq5SktV6gUMJSY4rQPuZvT1RJB83M5WDEl49qFToxn+o7Pn9sLQ5bWcnxRy2nbvuI00JfydC0pPianwuZMTzlGhBJwjYI+OviVhogch3QR3kgIzW1BlFOSa4iYAtmiFHDzAlEJuFWZ3pcF+yqDmtKHcydjMHJiBaMSUD6kfZSgxNOkAKMkfU09nsHIqTNS2MAEYCUAJRB9nBhCASCWBIW8NiRkCCnEh9JaESP8jGFCKgQEo6Bc+m4TYvIEZmksg37UScZqNELNv8nUyQmRpt+FwcMwH2jFfsl8v1b0TRpNf2Z+fPJOIkQC1Mi0XUhJx0++PwfwxJj8lPKUIAsktvYBW5pYUJFrQoT4oUQ1+SmFVoj2AfCbtvOAgczb8x5Ca/7/q8Dlw9tfE87zd6bH8u/w7TeesfyP//f/CP/y7hHe3y8QI3BxdkCMAt1o0H2zwPzjLZQM2O6aSa5a1SP272eo1z1m1Yjbuzliq3H+bIPxDy5R7CKO/8EB9vsZWabLHh9c3ePrP3nKL5iVg9ABsVfsgOwUqvMeUga0uwpSB0o0N6lbcWkpOVw6FN+UGB45VBcdhtcNZZZHjeLJEeO7Bpg7xIFfjKL0EHcF02EHJrS6RyPqr0p0zxz9ZY8HRCfpmRNA/VKj+vdusf3iDMUHR9hRA6/LSeY3+06i/g/fMeDGsN9R+CRnUxFilBDLkSueL+kzZLBGnGo63NpDbxTcykMdFNSLI+JXM9gLh+YXBt3jwC7Min7DMHcoXxUYno84v95h/INLVmp80GP+xzX2n3pWqlyxt3KSIyZf3LgMmL2S2P/IoflOY1xyldwtA8q3CsOlp+cvAMWtYvrmykM4gfq14oT40mH+pcG4iinUgYySPpxYHzWc5I5mJzBcBMRrdkxO1SBpn/3KwdzqKXE1qJMcLfvWouJk3X7SQ76qJhYw159kxmk4DxDnA8LRUBZqecwRKRWkPy2geqsQyojhyrNj8kgwPntJFss19AvqjiBj/o3A9if8kJ99o6BbYPcpA1zMQST/E99LxS5N6tNxyZO46j2w/1GgRHArUeyA7vdbNP+kwXAWp7RKbhM4fGpRvtUYLunXrN4oxFRrUt5FHJ+nSXWu5kj+J6bQElyoBFy7RxHlPdM0s48qJ7qq7iQfLW8J6tonEbOXIkk8kXrgyADJ4QQMzA44fBSgW4HqXZoQJDmmaxKb3SawooHqFinIhkxgf0FgzmoWyi85WT+xi/01x6IGTla6S4Jw9mZi6tpcfB2x+Wliw+4FhnPK86r33D85UvrUPk71P6l2ptidJIi6A05l7UwszfJHc+DYXSNg53y82UcM55x4SguMZxHVO3Fi8hMLtf2MnZeu4mdH85YMGwDMv/M4POckWx+5/cwQsisyTkmauQ7HHOIkHR2X9OQBJzCWq02ad+xMdBWrT7orgXJDxvdhgIuvBfSR1+CwFpi9JgNX3bEKJddxuJqLBLpLnrQu1V0IYFgBy28CuovkbWt5XsclezC7RzxOzRteS+UmwjasYcmyx2DI5k11Hfla9RH9maTc07OyBsAkz4yCoGxYUerZnwlU9wwk6dd8HwOY/JJ50u9S2rDwp4TfqBjCwj7HiOXX7D7UHa+LcuNx/2MDc4yob9kZ+bASJm8H4DUwrCWq+zDVnug+Vdp0AYenCqtvHMa5TAmsvEZZvo7JU8uFH56P2WuL/kIniTrPbZRZbsoFvu5KotzEyWtoUgVOdyExe+P5Wgm4+TKx2XsGxyAiyTblVL3DOi5eBwBgDp4ADASbmaEcluyWLHYedq7gUkhOlEC/JhMZFNC89xjnlJ2W9yPswuD4mIxgsc/MJo9Fd05GN8tH67dkDH0pUG4s7IwBMiFJVs2Rzw+aXaDlnYOvJYalQn3r2NF4ZlBsXQKGYvJBBiWgRkpCXS1h9vQAZi+jHAOGc7KFoRDQR49xpVFsHZCkoKZ1cLWGr8iyqiGg2Ixon1aobi1co5I31TG5VSQpaOfhK4Vibyf5qGot3KqEbTSKzQi97dC9WKG87TGcM0BH+gg1eAQloXoHX2sIf6pMYbpqgGs01BCgege7KJIMFAzOGRxTXhMrLHtL4CWAUOoJ1Jr7HqExENZDjqxCESkVNqfWqmNSjNkERHsHaAlfG6iOLKfsLKKU8IsSetMizMofhOwwDTalpYZAEDlYiLZHWM3IBveWfZXWk8kcHZnG5F8UbQ8oRWZzGOnTTCE6Uy3LYBGXMz42hMR62lRj8hDondhHSMnHOQ9YMq1iYE/m1DcJYPJZAhBSkrXMoNBQMhuH9IZy9HyKqkI8ttPLhraFKAqCUaMRjy1TX4sCsPb0OZNeU+ikmnOOP/91mCQEhIfy2/+ft18bxvK/+FtgLP+H3zKWf+ObGxX+6NULtNsa6JnQdzOsKA1dW8gA7Lc1QdLrEi4xPN3QQKmIcTvHCK6wujOH+6/OIZ4GdNcC8bsZQsMSdG9r/OJdzdVUAXrxbgtK3Ur6A8duRtlO6qnEUULlXsm+4KpxYnjMvYLbzaEdEAZ668IXc2gBxGPBCf9ZgNwyHS+OrOFwTYS6NRhXAeYuGdp/UcHO6cnyTQQkcPjTcxS9gI0zBo34DHI4wXv35QV0pE/MJ2YMSJK2QSBsq8k/JTzgcgejTJP2Jw7yPb/MpQPwxQzmIICo0V8HlHcSvhIQXmG89Gi+KhAV0HxR4PjqArIkkCj/vMbxWUD1mqvz2FeQI9A98ShvFZyXkCNYNl+AAS7pNSlxlEkaKSHvTwXw1Z1Aea/pZ1K5+5KrwsIJCMmeTLsMMHsxeRd8QXmp6gheijuJcKgR1WlCHyqJ2XcCdmtSWAkn9uOaLF95LwiaZwGzb1gQXv6relrZ56wTUwUKJMGJvK0ZmDECxw98WoVmJ2ixBYRTk2+sfKdZ2F4TyByfM7GyvCUj5esk8XsUYbZyWuXe/K5D862ermPdnsDwmHonzZ4TSdek+31E873EcJYqbEqg+ucN9/VWYDwDxicjqj8vUN5FDHca5Z1AFIqeOnPyY5EhTIB1w+qN9pnH8ucKvufqPutQBKpbTOA3y0nZ/0l5b7FNUjl/SpwEMNUeREmmQziCSjUQyJUbTJ2dqkvdj4mRKvYE/Znt6q8pM/ZlStsMSIEkXOQBkrxUAH3D68zOeUxn3xIkB0NZZNRAdSMe1HskH1rBvwN8jeY1t6E7srdqiBjXYupXlI7dotLFFJgTp1CSKFJlis4dhwnARx6H8i7JX3tg9KnDtOAiiq+AxUsWvrsZAVux4aS/u5QwB4aqZN/asJapI47HSo08tzklNDOXmXmVjq+bgZduWQfSPpJQA5L879SvOJwJzL4nUNYd/WdyBModQaAaMaWbZq9j9kVm0KWGiPaaYEX1nJDqNoUoJaBrZzyGIgFzOyerY44cX7E5HUfpyJKWG8pQkY63Q5zY9ZBYv3JLFYEaCLqnbksFeAVU9xHHJxLz71l/VKRwFV/yOhRBJIaK505ZAlr2ESZQWwuUO0pjsww2qyCGczMBQlcJhrukYBsRKBcdlgLFnq9r53wd3ROI5xCb/DdzDHCVxLCUE7gjuOYknQ2NfgIAACAASURBVP69mOo0gPrGo19T5aEsCKIMa1ZixNShqTvWuMxfs3LLNiK9j+nbizrJiw2/f9gTmiTh51zoKrch/Z01HEzTpS9P9THV1ETWZVhKPUWFaT+UTQsVZfLlRV4D40KiOMQpkTTXgajeU26KJMkdGaSTvW12LiepbWYNfa0IfiyllfQiCpiDT0FJHv2FQXnnACiMKzJQykaozsMuNcceIor7EeNZCZFAYdQCMdUy6Y4fhNIGxCCghtQpKUDJrFQTIFc2QLiIcW2S9SNAOAU9eggfMVyUKO8cPYiB1w18hNmNsMsCuuXjMHI70nvYZQG7MDAHR5ax0YjFDPpgEQUoRU37r1qHsCwIECsN1TtEaRC0QLEZYVclVM9+zVgqHo+jZcBOiPANnxuNgmwthPXwq+Qn9AGq8wgpKEcdk1w1gzkQfMrWQo5+YkPN+xYRinLWzkIKQf+jIHsqXKBfUiQJ7egYppdkt6K3U5elHCxCaSCtm4J6APD6GywBYwrFEUN6nkgs9Wj5GACxKih3lawdiVUBZHAHpDAecWIo/QP2c7RAVXK8KcQHQkB0A18ryVOn24OE2BjCxDpGHwCXVrpyTQkACEmP5YOeygwqo3OYfKDeE1Q+ALv/mrT2l36ebr9lKH/jbr/RjOXi88fxg3/8X+J81k6VFf/w4gv84f1H+PLuEj86f48//YNPMT6xWFwcMQwGZ4sWx6FAZRwezfd4uVnj88t3+L/+6MdY//gOtXGYFwO+encBuy/x6Sdv4ILEy3fnAABtHMbOwFQO58sjbu6W09+d1WhmPdovVoiPBywXHa7nB3z9/pyy2X0FU1sY43F830DPLaQKEH8+x+zfeY/NdgbclIgXI/2bNyXKDw6QMmIcFYrCYxw0qnrE8VBhPu8RATin0B9KxFGiWA0Y9wV04xBfV1DP2ymJ1g0KP/3oNX7++hpXSUJslMd21yAcDCAjZpct2pcLBpxUHrOzDjECQ18gBiDuKPNdf3yPzbdrhtIIQOmAGAX8UaM6S3UctwRUeqcw/+k9Nm8WEE5CrEaINyX8PKC64PbHtw2aZwd0xwLxroS8HOB3BrKXUI87hCig/6LB+HEP+a7A7NMtdt8vgCpAHBTE2YjQa8AJpno+6uE3BaqrDmOvgZuSDOiNpudxyc5S01jYI2fl8qgQ5o7BPaOCmlmor2qMl57yWie4ULCy0N+XsI9GqDsDP/dQe7KJWDjIm4KR9T86ovyjOQ6fJC/mkwFxUJh9aXD8fCCLrQCz4co1JCA/PMK+q6EPEvNvge3nEWFlIe/NFNrRfC8xnEf4kpLB8r1E/8hDDgLNp1vsb+Yo3miGxaTKiljQmwoA/uMO/mBQvtEYrjxQMSVWbzWqG4HDzwagUwwMKD3Kb8vkEw1TEikCEM8t9OsC9VuBwwumt+qDhL1wmH9B5lofyKrKUWD8tON18m0Dt/LQW4X5S4HN7w8ovi9SoAg9p4efjShfFhgvPfSOUmDVJ2YdmJj22TcKx+ce0USoo0y1CzGNRSGkcKNQ5lRYfnHNk3+THhqG04zXDnpmEd5VTDVNLFFUwPjIYv7zAscPPPReYvklcHjBxR6AzF73hEmuxUZiuAyo3kouktwp1G/jVONA1jWxsp8OmP9ZSXB6GQk+GzLUDFGJqN4quFnE8hepv/A8wq7pF4agF1q/NyjvEuMZCXKjIjvbPaLEtbwn+9h/NKL+qsD8JXs/fcXFkOHao/5OQw1Adx2x/Aq4/z2P9b/Q6K/IZpf3BL5TUE3Dc0sfKRNW3Syifk1m05ec8DZvInY/5uINADRvJI7PAs7/H4H9h0DzVuDwAftq2xcezUuF4TxC9fSOhrQw1l/x9SYGT5Apnr0E9h9T9uyqnBIbKRuveRxEpJSvfSJQv6U0evF1xN3vAPOXqX+yS2D7ELH9LOLi/xY4PCOIVD0XJ4C0QJFCn3QbYZdk1stN8rQKeopXPyeotLMkzyuA+l3A/oWcroP5dxHHpwLVDaW64zLi/E8jDs/l5K1UY5z6P+mfI1DurlPX7VHA7NNY5ilwqs++VTK1xZaLKP0FFzXMnos7i28ChlUCbkmCmkNWqk3EOBPoLyn9LXYR9fuA979L/1n1Pk6A2+xODLSdA/VNRHPjsf9AIwoC5v6c12V9QxC8/Mbi/vMCrgZWvwiUZy8JDqdOy3gKI8pdpdVdnNhWpsVy4c+0BMb00MdpsSWok++12JIVtTN2SebtcgEkJ4oyLVW3DJFhgmySBy8F5t97HJ4ozN54evDPucjpai48mC6iXxGEm5bH3xcC5hgwLiTq955KhEudklq5SDCsJOq7QGA7l6hvHe4/LbD6xkK4iO5Kw1UC1X1AzhTQfUjjk6jfWdiFmhY+ogBsIzF/NRKcjhHjUmL+bY/dxzWkj6jfWXTXBtWtm9RCAKB7D9tohIIhOVEKbluRHfZlUlUkqXB3oTF/NbKjs5KobgYMlwVrQfqAcaVhDmSFi63DcKbRvOqx+6Tm+Ob0i4oQ4So1sai5r7N6N8AuDcq7AYcXDep3IxlMH+HmZgrr0fsRodZMjB14bKSPBO73bpLW5hoSs+nTe1Mh1Jog2gf4WsNseth1BdU6yN4lH6iFXVXQrWW1SGcJLAHIdoSfl5Cd4/dnAopy9ASKkj2m0SjIbYtYFgzSSXLXWFPmKjv6IEJVQLYDYqkhDz3CvILcd2Qf2x7hYgnRDohNCdGNiFWqMRkchHUI8wqitwSkpYHYt2RHM5M5jIiLxKaGU8ovAKDtyFJKgdgPEFUq3k0+zWgthNaIbQfR1BOTGZ3DlPb6gEHNjGQcT6wl78ipRPJXymBjiBBSnCSwMfz1rOZfcft1YSw/+c//7TOWf/Y//nowlr/RwHL+2eN4+d//11itWwgRMToNJQNGq9FvS5w/2uHu1RooAtYXB7R9AdtrLFYdrFMpxdXA7w0lq41nqMzzDupLAgq1tGhmPQ4vlynxMRVkz9KqyygQZx71Lwr0jzjB1RsFv+CHljwfEKxkNyIA2REo+dc1VJ/YTQBYOoiNoe/o3DOBEwCcYPLjsx7yZcWAmZVndUWnIPrkmZQRceaBQUIO8uRPmQWoIxMz1ZFBMzlwRi1H+L3h8/OtDFAbDX0UBB3py0um5M5QRZQ3TMQsb+m/8WWEPaP0tLiXGK75c/WaH7bdMwc5SgYV3TCVFOCEsNiKVHrPCSNX+gQg4uRLs7MIv/A4/2cKhw84HjeLKLYEHLkUPKfyiQAmcu4k3CqQaeqTbyACZivg5pw4b3+MqWrA1XFi1XK6otlxkjFcMAmzuhXorikntdcWzZcF7DKmagWGqlQ33I/jcw/zEBRJMjtktuiFE6kAflyRFR2uPervKTvKniuZ0xM9UOxP7ImrOWbdiuk1QjqWCIB0AvU7sjLjWUTzisxU9lmqTkxjLzb0x+VJKuW4QHfFceZ6AjvnBMxXZEyLe/bvHZ+y6iIH1pgjxzuc8VybA9BfxKkyY7hgX2KxiTi8ICAAEjMYk8xzSMxi8tpJl0JnUil9lDwe3SOyd+WGf8t1EVlKKC23WewYkqJsksZGegynxM7cOpPCTszhxGLZBWXBdnbqFiy2lMfmMQqfmTOCCzPJNgGRisZzMiMlbzyOzes4+QPHJVBmNtZxX6objisqoLpPoMRhqobIyavSkq1GYPAQwOMxrtP57E7MWpa92plAeR9Pjxu5z5y88rzqlsepuiHDFApKe3XL7fiSrxc1pYcxVVOwtJ1sm+rpwRSe25c2hd0cmJ4p3QNG1BOE2BlZouousD4hAZjsNSu3ifHsGUQzLEUqeE9hSSvuW2ZJIRPwmpFtc5VA855BNLlGo7oPaK8ZPGNnPG6hyGDrJEf1BaYKCnPMSZmY0jLzOFWPSRLpchJpQJJrkg2ev/bYP9VobjyOj5miWd9yfGTr49RbqEbKeJVloE17xb7IYsdgLdPS59yfC8zepICcbSAwVZSAulpM1R7jXEJZBgzlAJqgAD0kUNaxyoIJpHEaz7CSia0jqBiWrKDQPa/z7lyiufFJDSEnYDysJVRPYMI0VnrfxqWekk+HFaszbMPjVd15tFca9a3HuJQTw1zdjjg+TpNdgamexRwYhiMtvWy6Iyure548WzP8SHgyrOXGwc4JxmRKG+VnFusucnWK8AwCsnMFOdIvCZBZzWwoQLbUNWpSPkgbU3Ipk1L52Uu2alwXlLGWEsX9CLsyEC6ivO1hVyXkGDCuyADmgB5fS+ijhxoC7FzDHB18waoOOXiEUk3VHawRCVCdxXBRodiOGM5LmL1LAFChvCVgM3uL7A0OhUpMHtlBc9dhvGymKg3dUVbqG42gWKcRCgXdWkQlJ3+jbzSKmw5+XsDNNKQNUJ2DHP2UmBoqBdlz3HLwkC6gv6xQ3vYnL6gSUIcRodAM0pH0M2ZghwD4eQG9SxLJVDfCz02ep1CoCQRGIbgdLZnQOjrAeYRlDXlkwphb1QwDciFth8x39lCKY484rwEXIEaLWFMuIB6kwk6prC55FR1lyigMopQQIfAxRnMbOXwnRqAsTqxmqhD5gS8SSOyoJGvZ9RM7ia7n8/PPAOJixtfoBwhjyCJKOY31YfUHkACicwzv8QGiSA0Kx2P6Ui8AO7KOJI/ZewJHpU7bS1La8MsVIQkoCiVPabAPGNNfeUvPiXb8qx/3V9x+bYDlf/a3ACz/8W+B5d/4Vn74QXz+X/23CGWEagnAdMsJjUvNHuaAaaKai7x9lSftSeKpMXl3snQlGE5ac0CISnI+RE5ufXGSl+ZSauEYfFFsYjKog0ERqSeu2HBF+vCJw9k/V2m1nZ1cdiamWPk8gc9JiCKQzSjvxRRZr/qTNFB4oLyP6C/5ptQtX0+OEW4mTo9L7RbmwAmxHE/HJKrThLzYYuprA9JEb8fHdNcRi2+4apwnTTaFa+iWK+Bmn05Qei3XkA0odjwuvsyT+ZgmBKnzbqD3ixPx03kOhuCheUNJjk7HyxxZej4Vc6fJRbEjGCh2Jy9ZseM4dBtPXrKWCY+QKZSgEFP6Y05ILe8prx3WXPHPPW3SMvxDWk5ifSkmn1Gx52q8awgcq01IPXb8YsrXj3QEM7kLLV9TusNU/nx8pKbJW0x+LtVHSjgXPIf1fcDxkeTxn2cAyzFVtwH9BRdDVJ8kcOskF+s4uVdj5PGpU6dhwNRlFzSB7RQmIzFF3uebyQEj4uTFyrcc1Q9g8szx/cj9KPcR7aVEsef+MFkyoruUk9+N5yOBkCR1zeeQ8jmyEsUhwDYS3nBi7GrxgzEzgCMf4xRWlAJUcqm57vm44sht6e4kvcwywqDILmR/Xe7bY89bApJJHtivKafTKXVTDSmlsqIMt19LlMlrGFS6DszperYNARQEUOxSpL9Jr5MCX7wRk5RzXPAYm2MKUjKYJsx64NjzYpEaY9pvOfXdiZgqFTxBSn5vyRyOksaV6xNi6uYzew9fczt5P/LEXY2clLNKIAGtDDhSV2J1TzBAqXs6HknSrIcIW4vp2ioOnmEmidXRQ2avBMzRY5zzs5V1EDwv1b1Df6YTSOP55+ci/X6mJbOh2wA3k/Su5cCbFOYjB8oiXQIUeXXfF2ICiw+DXnJ/oC9PdoRcz+BqdgoGnQJVjJhkjOOcZfVyjNMiHMDjLYcAX6vEEkW4Rp2e3568dQCgjwng6LTokcNLItgzmJik3PGnelY5PKzrUV1IgUFiqmrQnYdd6ATcMNUxAJkVDvAVpZ/6YDGeFUDA5AM81d7EqVhexEhZ5srwGrABw5mm9zD9TnVB/o5LE+4c4gLA1UxbNZsBdl2m7+Q4AdZQsKYidx3yGpOQLsAbCTdTKDcP2JS0v8gBNErAHCxCCnqJWkLaANkzlTQKka5zSQmq5tj00RKg2TCxQZkd4hhDAjv4QQ0G70wAQtBXGGomuyJGqCPZMTzYn9OXJifpWX4pe4doJIQNCI05+RFVeowA1HFELMhmibR/vJ4k5MC6jGgkhA8cs+R3pxg8gV6ZQmWSxSNXZUQjoY4j3LKCakcCKusJ6nLlhJRTjYZwAX5RQh7GFEgjKGHtHUR48Lo2AaMkBZ1krjnMJqet5uOh5CktNYE6ZEDzEFTln5WEOHaIVQkxjIhNRYlq2j68n87ndL4ebEP4MIHCSaIKcBwxpnGfjsF0H8DHJq/mxPw9AGoM5VGnbcdIRjB3Qma5aVEgjuMJtHl/eo54ENgTw4kx/OXx/vJjHtaOeM/+SaUIRLOsNgHVyUeZpbG/gpX8NwrjyX2Z/wa33wLLv/3bb7THUhgPd2WxvjhAiIhuKFA3PXwQGPYNyspiGBXcqKF0gOs1TG1RVvzycE6hOxaQJqDbG5TnHZxT0NrDWY12UDC1hTYe7ajhW8Z4i4NGXDhWjAhAqgh/0EAQUEuL484AlYcuPcrKot1WkCagvymZBKsj7v8BAJ3TGCTkzCIMCrAScpH6Fp0ATP6ACBjPNVB7diuWAWph4VsNUQTk7IXoJESrEGvPGpOrkR2DZQCCYJdhFExZfXCTvUSYcyVw3GnWUMwdMCigCGitYJ9gq7D/SMKXAaoTGC88zFbBXrFXUQ0C3XMHMUjEIrKIPkaM5x7HVOth1571IlXa/9pDHOhVjXOyxr4mQxjKkL5EBI5PJX2v9xLjuYPsJSWQLRlZSuQE9p8FRBmhdwqhCIhFhNpL+MajuFdMkxUMMtp/5lDcqMQIRkTDyZIcyMD1l5wMRx3QXwoUG4HhKiSmkSxr+ChC70HpCyKGls91M+7D4UNOLO3awWzVVF4fipjCTwTGc3YwUo6VEkQ1fZZT2XxK1BSebJc+CozriEPPZEU50sMbTJyknN0V2V9WS3ARxRecr5bvWWvBACCJ4Syxow0XXOyCLEMuqB/XZBw5+PS4g0jJqvSl0rsamfKJlIArwPTZNPm3C8AuAsxBYNywwkMNTHbtB+q38qIPKwQ45nHJYAsuYBBMlHcC3SNO1suNYtXDMnUx2jSpGul9K2+Bcc1JWnUn0oJBWnwSAuOSHkvVAS2SF2kr0T4lwC7vBNxMMOhophLTnBahtmkCqPMCk5gm0PYgOGEPgE7dihDAfpHZ6pOv1ewTIztLwN6k+x2lga6mtHMCbwn8AynddcmEWt0x4TEvqFGOhVQbw2OiOyQGRkx9ftkjOZwJlPcS7RMy0uOKx2UClSm0KqeMqp4g0dViWnjJtR7tNes5XJP83MlvPC6AIl0n/YVM5ypiXEtUt5FprBIobyMgCXjdjEnCGfxmP7E+chFN2tTPl7yzmY22jSGbnybbXAzhgmF3mcKsLGAKgf5MQjo5+RnVmBaC0iKFnXOBKvciCo8psGcwlAWPC07Wyw2ZPD8T08IFWbkkiZ4J1LdkLqtbhu+4Jv3tTP6gl1CNaTEuJqCv+PycbBqUhEkZGrYBir1keu1I1jhohi3lMZtjhH2sUe7YgegvFNne46mnMayZThoKLhzqVGjfnyVvbFqcoPeSdSgiKLSXlHR2l7w4XSl+wLaWG95XHAm2bCNR7MlM92dcaMohPeU2oF9S6pjTUcO1QX3nMC4YtsP9B4oj/XvduZo8rg/Beb6pgQAuM3u24QJIf1FMi0R5EYCJvJRyhkLCzsjyBiVQ3ju0j5l+Ki1fSw0BdqmmsCPXVBPzqQZ23wpHKaZd6gm8FzsypxBAsXWpsmNEf1Um6bNmn2NKgC03iqm1PQN01BggbYAvFV+r83AzLmZpJeAbDTXQH2oOYpK5lncWdqGhag05eMSUNitixLg00EcuWISSCfFRCxT3rAIBTgtMvpIwh+RfzImrIUKOAfa8gZ1ryAU9kMWWPk2zywyqJaPYWqAWcDMDHVjh4SsFOQZILZnG2ln4SnNBoNLQhxEhJcjKwTGAR0v4FVc0RQJIrjEwu4EgUD5YBDiOkwcyzAt6J1MQT5wzTBCx4uKKNRPYlYcB/qymdLV3XGRLiwTC+gnUT+Ay90I6f0qClfJUE5JrSKzj36U4AWeA4NZoCOvIbGamMiXECiknIC2cT2mwKSE2Rr4mgHhkuM4Pgn4yyBYPGFMjEJ2fGMUJtOYAHsExCqUIwpVCrhvhJiLBe77FmHyVcrofAIT4K4DlX1Y38qv//Jtzi5hIrL+Lt99wxvJ5fPLf/TfJeE7/Vf1GoftJD/V9ifCsh3hbwp87FK8M7AcjxMYgmgjZMYAjR6CH6wH6dQl7aSH3GmHp0HxZoL9MiZiHLKtIrN6GqaHFlv17vmFq57gOCAacYKfuwmIjp8nBcBkw+1ZiXAEQDGkYV/RTmQNXi+2CATjtRxbla5MqAChl7C8jir1I6ZUxTWg4YcueLFZTpLTKEqjeialUmVJB9gW6GlNcvW7JnilLMCBT0bzqMXmBfJVTLSmXFAEYziLqt9yWGhL4SPUOlPcB9VtMsiBfURJJppTl5+aYPFQ6sWgrMtBT2Is6JYW6GaY3ZJ6MB0M20y5ODG6WSTLgBVNNgzkkMJDL1eMpxISl0ydwoHumRgKUJwaVJJopOKTYkKHrL8RUYG52ycN1c5LTTvUD4ITQV5yoVrdIE8nT2KQHuiuyxs07StkoN+QxlmkSm49vZsR1jyTXopfI7OIEWkSgdyymFMtcym1nvEZ86ph72MtHKWg8JVKm7Wama1yK1J2X6jB6ygxzEXdmPIDkTZtxQil8qgCY55CNyJX9JLHLcqMoCA5yGEm+vrMXqj8/XfNqJAs7pWjq0/E5FZ5zsqi7DHS5DTXwWoyZmZVJRZDkmqxRYBCQGuPEkg4rMuY5mKh5H3B8pFDs+Z50NVAcmApa3Qf05xL1+zDVDigL9hoeT8d9Kno/5s6+zGqTyY0iXU9JOZGDVqTFgwJzTCxavubYSYhUiE72TzpeG8uU7pmVBOU2QNkTi+lLoL4NGGdy+vxrbuhPEwEYk9ICEai2Ht2Zgkn7IQIwe2MxLtXE6gl/knVmPxdTUBVMYuz6lURxPHWt5seqIWBYqRRek7xAQkwMJAQBTJTsPnTJ11YcA1QOpTmEqfLBVQQt7ZWCaeMkg3TJO5blm/PvB/QXxQQygsoLPPydvYJh8gVWG89qilLCVQLz1xbDSsE2EuUuwOwd7EJPDK7uyS4jAvV7i/3zYgJsWVoqQjpu6Xd+/sX03eR/ICUlIy4nbxkVMRJFSgsd1knGmnxz0obJu+krCVcKVHcOwkeMKwJDvh9YID+cm4kN9oVA827EcGYmVjQH4UAA40Kh3Lh0npI6wBJUBSOZMmpkqskAio2Drwiu5MhxTd2MSNe4C0wkTWwrmWaG3xRbB+kjK7JSwAqZO+6b6gPBXYwIOiXIHh3c3EwBQHnsZmvhZnqSmNpVwRAafWKsdWsxnJVQQ2AFR+uhBg8300BiZPmZlBeeJPSRC7iqpd/PNUyW1YcRblEQGAGUeAIIlYEcHJNOtZiYT+EC3LyA2fQYL+pJYhrMDxeNRYxQ+wFuWbEDcl5A9S5dU3JiokVMDGPBYB1hPXxjoA4D2UgAbmaYyDr6KRE1SsHOx8S2ynZENApuWcLctohKkelM7/eoJT2IWk7SWmEDmc3DgFCalArLqo5QGUQlmcQ6r+hjnKXgnkIzXEci9VFauPMZ9LabjvfECiYGMCeyAnzO5BksC3ofc0+kTyyh0Q/YWXdiGHM1iBAQ+5Ydk6NFmDdJturZIVmY9P4SBLlJghqT11GEyL7J4dR9OY05M5SZVX3YR5lqSU5j0SdGMktR64pBP3l7WvH3fDzwACj+MhaQ6lcjuB/0X6bXHQa+z6z7AYuaWU0+NjAESIoTgxoiRGJpo3M/DAb6y24h/uZLYR//LTGW/9OvB2P5Gw0sZz9+Ev/h//afAADu+xo+SAxWoylH/M75G/wff/EZfvr8Da6qA/7p6w8wK0dU2uG+rdGUI6SIuKqP+PLuAn1X4Hefv8KfvXmM4Vjgkw9obPrudo0YBD6+vsXr/QJGecQoUBcW3393js8/eY3eGdy1NUKQaMoRg9WYVwNuUq+lMR6lcdAqoFAex6HArBxRagcpIrZ9hc2uQVE6fHrxHn/x/hI/vnyPP3n1BGVpsagHvPvqAtef3GLXVnBWwY0KUkdARHz25B2+urlAUTj0vYG9rzB7dMS8GrA91ui3JUQRcHF+wObPLhCfd0AUkMojBInlnB/Ch2MFNyqsz444azp89c01TEOGN0aBri1QlA7Pzzf44i+e4OqDexy6EjEKDMcC19dbvH29ZiDOoUC56vGTR+/wJ//kY4RLi9mqQwgC7V2Dat1jOBaYr1vsb+Zozlu0tw3gBKqrDnbUMIXD8LYh2yoi1lcHbN4t8OGHN3j5Z49x/fkN7nYzLGc9Lpojbo4z3N/NEY8a0BH1eQf/rxZwL3qEkZMVoZiWJwSwOjtit68RrESz7GGtgntfA0sLXTjYQ4GrJ1vsjhXGjt7Qy4s9br45g1qPQBR4dLHFm5sVTOngnUJ4VyHUHj/97BX+/F+84NhT4IyeWbi9weLRAd3P16h+vEXXlpCKH7z2UABOklXeFHjy+Tu8frtG7DSqiw7+izmqn26gZcB22yAkBl02jpUzMrICp5dYvNhhv6uxWrUIUeDw1Qof/f3v8d3tGuO+gGgVg3bOWV0SqohYe0jjEfaGixWrEeJNheLjPbpdxfuOBnJmEbcFZCfhL0cUjYW9qemxfXbE8KbB7Pkew6BhdyVkqyCHVG+yYveouTGwlw7zyyO6L1acdKjUFTl3qL8u4P7eAe5djVilftGlox/YBGBQZMUbD3kguxivB8jvKx6TgRUyUQLQEXonYc8dhCNzKM5G7kMv4NcOolco3iv4Mk4Eh/50j/7VHHIUMHsB+3kH+U0Fe+Gg9gryeQvx5Qx2GSAvBvhtQsGVh3lTwF46VC8NhisPsR4R7wvEOsAsBtgtA3vmXxgcfzawsuiNxnAeEJcW1TclfJXY517ArtMX/NxCvS7pUbzyMPcKZidgV5H7IDElywAAIABJREFUlQC9fWQhjgryYoD8toZI4Nk1Ec1rieNHDsV7xeqWqx7+pkLzvUR/Fejh3imEC4v1Hxa4/wcO5VtNBlsCkBH1a4XhLE6ePBGB8tMd4j9dwc3IGNtlQKhSnZGMmH+tJ4vCcEkvcffYQw5UHpS3MjExwOEFWe9gGGTUfC/R/qxH9UWFLGkeHjnMfqExriLcswHF1xVCGbH8Eth/RKZ9WiQTXEwarxyq1wauiajfCOw/c7j8Q4X7nyWJd8FxuBkrWLonPAdmR996SMxluQH6cySAntJWHRcK2sccr265H34eoPcSZsdFgP6Kfmp6uFl9M/9aontCBcDsVcThA7KbPtkohEfqD6SveVxzUWP2ivdHKdA+jRNLPn/JYzD7XmA446LV/hMPs5E4+/OI4UyifcKwJDvjMS42EuU9F5raJ2HqeF18LaE7Vt5QRkt/um0ExvWpZieUDB5CZFBUPlaUSnMhK3uMj8+pAGif0qtf7Ak6hjNg/RenmhOf+lHNntd1/Z6fEeOK4158y0WTUADVXUR3kXpoE7uc7Qe652KPHJF6WLlQ17xOTPfIa9kbLtrOXwXsPpInT3fNc+4aYPZ98txJstCm5T4XRyb09udMf15+FSbPcf2ekui8X1zgFZi9DhhnlNYfnissv/Ho1ykFPXA/V7/wGBYEu80NQ3NyV2d9G7D7SGH2fYCrBbprgdmrMMnITUdrweZHCsuvPYa1xOKlQ3utU5JwwLjk+T08VRNDLwI9z8WWEnpfUC5e3tOb2V3wu2f2xqK9ptqouvM4PtaI+oHvWHEBDGClimskVQkLSfn7QqK8dxjXGvoYMJwxvViNAd25RnEIqN8NCEbi+KTA7M2I9lGB8p7sbrFPYUAbB9U79Ncl5BARSgF9YDKw9JRZZ+CcQ35CQd+raxRcRcVF/foItywpe94R6KvWJYaTEmC7KqGPXFCo3vesX9ESej9M8uJgVAKMAno7wJ5VkL2Hak+SYNmlRYoEwkJlKEdO0l/ZjicAKQRCU0z+z0my3A6IVQnZ9j8I6gnzikB7e0A4WxIUdwmMSTmxnlNS7C97LEMgYFTyVIeSmNOcOBvbHqIqEds2yXMTywpM24x98ol5DxhDSeyDBFkAUxKsUOpfu49j+SWMEsOvTo/9/3j7dQGWP/pH//aB5Z/+z78Fln/jW/nJs/h7/+s/QjsatG2JqrJodxVU4RG8QIwCZW1hR40YmFwqZIQbuSoSDgbmrIc9Uhtoagt7X0LvFOqfbHDcVwhHA6iI5rzF0BcoqxHtpiYb4AVQBEgTEKzkZLd2wMHAXHZM9zymCW2nEAtKI6OJlMGKtI0U1COsQPGoxbAr2UuZJnHwgnLPOgBVgLpNUpUAuLWjxDWIaUzRpNcpA8N9FLchLweIlxV8w3RPu07hPAF8TGTqaaw9hIqQNwWla5EhQNMEvwgM4ykC5aglJzRRRRR3agq7cM8HxIOepKxRR6AMKL43sGcB0QToew1fs/fS3Gm4BY9LTgKFjFAda0fCJx3wsp46IO25Z+2KTEyjidB7ykH92kHuNYp7SRZ3zjFSZppWkSvWybj6lKTpZhGhCtAbghW79tB7hZjSRVWXZK41k0R9EVNCpkj+VE4Yc1po/jlKstihiLDPRpS/KDGuAiWgGhN7LUdOiMc1J5m+irBLSkWrd5SuupVnX6d8IEnM7E4CcG7G8bpURaOPZKm6xwF6L6BbTgR9GVPoDDs23TygeqemxEU5At1TD32QlMGGFNqzFakvDAjq1M1p9vw/FJQTR81z5ZqYujlPjLCvY/JxCYyriGJHmetwzuf4KiYlAllPgBP54SxO+zlcBpitPHkpsywxSePy+4RpsSdGOz8+bzPXVQSDxIDx93GNqSLFNfRXIyafdmJE83sEoBd5WJ8AXn78cIapFkTaE9tVbIH9J5xcmz2lnWSDME2sTfImD+eJWU2dnNKdxho0pqoTfUyps2l/mrdk63KFDMOvMB1Xu4iYfUcpbGasXZNY6yMDi1jrw5+rO4bSUC0gHkyCqU7wFVn9KE6sNeWy6XxEjtXsTteum/F3NfJAukZMPY+uJnsexcmrnXsMMzsrPdL7O05srExSTV8xqXNYySnx1RuywMOa3t3c2zr5zecM/hnOki87cvtq5LYzUEJikbMvmf5nSlGlpdIghw1Jy7qUYc000moTYFPHZrHje6a+DROIYMhTSjV9qHp4qMA4csI+LFlVAvB6UDZOFS0i0i+7f0ZG3bSZVT+duyIdk+w9p/f65PHNLCmQVCjvw5SSGlSW9mOqxcle5JiABftE5XR+8nsz+58z8ys8xz5JvOODa6ISk486hx9xG8lXnuSpOXyK/cIC5c7DVRJ6yIBLTPJx0waym8DkVZUuQg0B41xN6gCAIC1/ZjCsLUC6iO6SXtDsZQYAX0pUtxZ2rmBnEuYYJuWJ6h+AHM8QINWRdaNMPBKADXFiWUXEFCKESMASJTspKRFWkDbAzRR0R9ZVHz18xWM+LpjEmisz7FLD7NLvCWj5BLSiEpO0d/J9Rkz+VF+S5YxKTJ53CFabMHwrSYxtoKez4KIuPzcp7w1lvgDAcddqqnVBJHuKAOS0Upm3EyJiISFGzhNCoci4xgjpCPx8bZD9t7JLSbdJCZPZthzAE0o9BRXJgcx6lII/+wi/LOnvtSfJafaIRsO5B4Qg4EuJr5nJhY8Q/UCGMUbExMZlL2k0CqJ/yDiqiT3NvZbwgUxmCtmJSk7ezWg0a0Py8wtDEJgqTPhi4ocMY/Ztpm0jJJD2K3yOIgPILIPNftiHnZcAWc2HfszUiymU5PnyfgKWcRwpmc23h6mxf21/5en+vwuM5Y/+078FYPm//HoAS/nXP+TX9yZkxNuXZ9jfzXC5PsAoj3/3069hCq5mXF7sYa2CeFkh9BrupoKUEdFL6II1C94pNOsO5p2Bu6nw9JP3OPvd91hWA+KmwOMPbzG/PKLdVfB7g+71HOqevka9sBAyIliJ2aonqAQAJ2Df1Sgft3j2+98DKqJ8eoTZKKiLASgCfZV7DVk71N9piMZBXQyw380mIIlANkrOLcLVCNF4ftiKCPGsw+Knd5CNgwgC5rKD2UiYxy1B5tKh+doAIqJ+ckBsPPxBY/GzO2DpYB+PELWHnFuIxkOMEsIKVI+PqL4tIO4NwtUI9fEB6pMDsLIQPZNdF19ohJlH851GPLeov1Nonh0ge1ZRuDNHJuxNicWXmiBoxuOtbg3EZwforYRs6YOJRYTaabgnI+TZiPqVAj5qCXZXFuFJT1D5bY34vEf1XqL86RbFrYK9tvQFLh2/uOqI5Wf3EAODQIZPe4yXHsU9ay8mSaUHmm/ZbYmVhRzJbkQTgZLpvvbco3mpoVsB3wT4OjB453nPRNU1Uw/92gEJVA7nEcN5YC3Di4GptQMBaf9ihF0FLP+4hJ1HmL1kVcIrAXNgwI59MVDCnLyeqhfAeoTqGWCkW4HFzzWlzQ5wH/VkCzTg5gH9Y4fucYC0ZIzK9xL1O4HuqUN5x25ON4toP7IYzgJCETFeelZuHATKW4Xuw5Ey2nVA95Me1VuF+i0nqZAEDP1lQPuBg/DA+uc87vbKQh+B4YMRq59z0qsPAq6O8FVE95Me3VMP10S0HzqEArj6ZwHdxyOq9/R0+oJdjuPvtKjfsZtVdwTKdhFZyXHLxOJxHbH4SqJ/7DCcR5QbMi3dE49hHdE/9vAN2Yv+OqB+x4qW8TygvOeX1HiWJqcK2P3OiP4qYjiP6B6nio5bAt3jjyxmryLaJ5StL78KqG4IzMdlRLEl8BqXqXdxSUl6f8lOx/N/GcAwK6B+F2EOrGrY//sdHv2fEc3riOOzCDnwMYcXAe3jiGHNmpJQkl2ZvWSgkV2yN9XXwPbvWXSPU9LqMmI4p7xbWmD2OuLwAcO/zIELBccXAdV79l+qHlh9wUTY3U94ToYz7vP8u4jd5x71O6aBuhpYfBswrAU2P2HVRfZYigBc/zElfsN5qjFJVSNqiKhvIg4vIsY1cHweYXbA9mc+vWe5z9vPA+xC4O7vB5hDxHARMZwxQCv7It08ycJniVW5oBS3eRfQPuUx2HwOIPJ1XENma1jKBBxSGm0bcXxCUHn/M77XQiFSiBllzrsfkXkJhYBtBHzyjzKYC4lJjGjehQlQAAz66i8FDh8SSC5eOdR3BLiHpyot/BBIDRdM3+0esZZk+5HC/qNcScMJfkjha8oS3EgbMX/l0dwEHJ4LvP89if6CQLBMEvjtxwy/Gs742XB4qrB4RTbg/jOJw3PKcgHKuHOIkp3Tx1vsyXStvxjRn0lsPuf5DBpYfOtw/5nC8TFlkDnALQc+6Y4gXvcRs7cOrhLY/ih9Hq8EDk9lAogCzVsL3Qf0a7Jm1b3H8bHEsKDX0jYC41ymRQ+BwxNFf+cYJ1nvuOT52T9TsI2AnQn0FzLJr5kyGzTQnSt05xLDSqDceJRbj8NjhWEp4A3QXuUaioj7Tw1mr3rMX42YvbFo3tr0/g3YvaB/cvuJmby13YVCd0lvp6skmrcDxqVm2NSOFSOuohS7P+NjzNFhXCqoLpBB7AOOjzXGpWJ6rKTX8viEsmk5Bgxrhf0HBr6Q0K2HXSheI4VICa/jFC7lKzKL/bnG/KtDClyiV3T27RF2oWHnEvpg0Z9rmL2F2VsUmxFqCEyJRbJYOErN28cFjk8MlKWk2C4UdO+hjw7HJ1RshIKpsPARrtHwlQKrSgh2ukdk74a1gblr0V2XKLYjXK2gdwNU72AbnToqJfrrEocXDZB6MfV9B7sqUqekh+os3NzAzQzcgsmwcqTHMdQadlHAVxr9VQW3KDBc1hiumomdK26O0Dd7yH0PN6ccNmoJd1FDduzYBAC3rGDPavq9z2oGCSkJ2VmEpkBoCsQELrPMNKwahFkKWOoHgtGR/ZZTPYgUiAl8hkWF2DDZNRqF2JSIVYkwq/nhUlAWnKWzcTmjVHc1JwhsKv4N4DaqpKLJAC6DtyldVkI0NcS8gZg9+H9WT5JWURQMA8o9mXn+nYOGnCOzqR4E+yTZa8zJs0oB1k6sZLSO/3yY/iED0L/sXwynf7+9/VrffqPDexpjsXx0wI8vblAph95rzPSIz65ucNc3+Gh5iz/xTzD82OKT9RavtitcLQ6QZxE+SJSXDtuhwqrs8fXHCo3xMDLg2XyLV4cVmmcHzIsRSkQIERGCpFzSKpytjig0Za1ZGgsARnn0jYUQEbNqxKIYMDvrsKh7vH1qsGgG2MKhLke0fcnKk8+B9apFPxrgBYOI+qaAiAKrZUvm1Tjc76kjc5XH2bKF8xL1bMRoPOZNj/6nHkoFiOuIurTYfCKhGodCe4iLFjEC83LEoXSo6hHeS4TAkAqnA6SMUCrg8NQCRcBs2UPJACn4d8wGjIPB4ROF5dUB+34JVXi0HzqcaYfuaoC3Cs2qQwgSfVngWBjI6x5GBhjj0XqBRWmxuXAwyxFu0FCFhzcaRW3RVCM2H2k0hYNbWjTzATECdiSbWRiH7onDUkSMlw7NWYfWScjGISh++Q1WQ50NcL1GUTrEwqFHBdE42JkkQ6wDhmigXxwRvIRdptXEhvJkv3aQhUf/iAAflUf0At0jAW08xrVneJLSULWHS91hfu0ghiQTVBG+Dgg1ABOgagcPoH0i4RceoRGIOuD4HAhlJKMdBI7PI/yZg28kRBTQpYNdFogmYtSc4LqVg+wU08SvI/yM7LOcWwQnMUaDqCOGi4DOAGLm0D7WGM8CQk15bqw8m2ZkhJuRIRYBCRxQxigl2V5fJebVJBZRR8AQ7B2eC7iZh24cuicGCAL7j8hg+wIMYAIQrQQMwZ2oPNxM4O5zBcCjvyZwR1RwcyB4gfZJhGtOsi4IbmuwKVG5CegvEmNdUUYVC76Wn7NX01cRwwUQi4DuWiOUDHbqrxiU4puQJvDcb1+R7feKDMFwRqkfvED7VCAWAeOKk7YoMNW1jOsTUxtasr9qFLBNgLQK+xep/xMC/YVIvkoBvzM4PFf0O5dh8rHm4B4ISjn7y5ToKpFeJ06MoLASQRMARRMRIhk/CKB9lM5VTNUxiSHvrkUKj0rXUwUIT3AvBx7vnDI9nLOeRw0M0PIVz4VwBNJREvgenir0F4mBrJGYh5PsUHp2XCKxm/og4eZJFjjLHZSAHGRKUSabPqyTF14LBMWwoWAIMH0V4ZxAd5G6D+ciydsSC1Yl6eMMEIH7FQt6foNJxymmEJzkm/QFIKrcpcjvmshsNoSZSGw1wR+MQHfOa4WdiicvPqKY+iEfglFXY6rM8Yb77stTErWI/JurACW4n1Gd2Gk3o3dUxMRaHnhd5OOSvYyuSQtOFf8fFnJKcxaBXlbXCAwLmVhjjp3jS75EU0zBX0ywjugvFBn+NE4GbzFE1c4oZ/Ylw7aiUhPoHFcpMdjzcaEAuitmCGDaz8TWJnVBTtt22aOY2NGcCk6WM3ImI9M5R1JWlCc2k0wNXycKoD9T0MlfLX1KQo7AOJcwkqC6vy4nmTeTfAUABlHZhoyenfN7IzPS+dafF9N+MylZIOQuU5OY/pLeYztPx79WUyq0SwnLCCeftSjUlBIfFeAaPl9WDCySLk5MoC9EUmIkBnVZJFaYUs3+qk7XlMC4Yi0F60Ae1CGVikFBUk4KCxGyt/s0dQxapjoV/IDNdDOdGD7+PbPefA9SsuwXFfdlRh+ibwqmGguQTUznW41MqQ1aTM8JpSYzm1g5X0iY1sHPClaeFCf/aGaqATKkBOOKHtdVBdnpKVk3SjH1UoZap+8XTTbSk2kkayz5Ogl4RgEILRGj4JuhVEAg0BZGE4hJIJYFk4YrQ9bSBb7Rwwm4xcxQCsHnpdeKQjAAR/2/7L1JrORbfuf1OcN/iDnizjlnvnmocpWr7LYtD9AYFqbVahaMzQLUSC0hudmCxIpd75BZMEjdajVCApagFniBG1u0Gw9ll8uvXMMb6r2XmS8zb94x5v90zmHxO/9/3Fe4bIO6pCrLIYVu3Hsj/nNEnO/5TlqI9CD+TGFEoxS1fdyCwJgge5PtxEYPZpss297jMmUw2wYMKaknibcugbh9bZLufJNtWi5mtx/AzfRb2f4dQJVVBuAv4K+EP5Vd/XG8/VmZRT/utx9rxnJbJyyu+nzjyV0eL2c8Xsw4K4YUznK9zXmymnF1NqKf1Txfjrg7vWZdpWSmITGObZPQS2rON32qeUYvrVlXKV9/fI9NmdI0Gh8Uw7QUwFgk1KX4CRLjOX05YZAJJV/UFq09k15BcZVzb3aND/DBy0P6WcU4LbFZwzAvMcZT1gnDnjyeTtfiwbTi3wxBMRgVhMJQNZZBVnF+NeJ4umQ8KEizhsvrAUoFJv0tg17F5csxxTalKBKqIsEaj84bkrShqBKc0xSblDenL9HGM8pLDkdrHh1ciAc0r3FXGVVl6e1voTToCCqvL4aM+wW3pwvCs5zQd5SVJQwct/bnoAO1M9jUkT5L0DpQrFLSfo0pFMm3+xxOV5RlQjKoWSx6mKGMTEKtO09btUpZbzJUqbHGEyrN+nTAdp2JfDkgx98GysqiBw2bSxn59YclyUC+2ctPRniv0Imj3iY0RUIyLQiNJnmZkJxZ1NLic3ln14WcUzyEjSXNxXPnN5awVxH2akzmIALF2WgD1tMfFCJVJoKosegwVSVpuVlekVwb7LhCZwL6UVAf1uhBjZlFv4SFEBNyDw4X2C2kzxOSuYFGiXcz9+DA9zxNX2azk7kmvMxwJxVoGN9a8u795+AFTKhGekeZ1ITK4JOA7zvGJ0tsvyF9abHDGhWTZ0Mi8tneuJDjkXp8bWgGvgsisksNByWqVqDleDTDQMg9SnvqsQMXB+0DqcjRlcJuNNQaTMCPHKGJg9hhQGcOn4oUWICeRxkBhSH3XUWI7zt0paO0WEBueeiw10aCuDSQ+U66ZtZaJNxG9sVlAXVUyvHQwvCqWkfGFqg05qAku9SYQvxjIlEWUOLyIOcjFyl0ceTF1xoHtM0gEIg+UgN2qUgWWiTGPQnF2txvqGaB8tB3YUObW0F6TYt43IYx/CgRf17bw+kykRA2g4AbeJqxl/3LfOdR1XXLXgc2dxuaYaCeOpEtJ/FaSzzlvpeO1sgwVns+Du6j7HDkJG3XiYcwKCj3XAcos/M4+EslVRhgfTuCgVz8fwIKRcrdSp6bgaeZeOpRoD6oaeto6lEQcJxJMFrr3Wt6wsyaUlJk22Pj08gqx4TkarILIzPb2D+ZhSi/ixMcCWyPPNsj36WtulSk5620tAX11RiKoyYy9xI4Fowwv/VI2N62bqoeyrGoBztZddOXVGifwOZIUUdvna6EifRWQqJcLnLcpieAWRhgmSS4OfDQtbz/tkeKVootNU8C1NsQsDbBtXcmctamL6BfgtUUm2NJNXYp1GMB7NVYGNZqHKXElfgat4eqSywGkS/7RAKSJGROtqEaS3CatwI+i305F3VfdfJauxZ2t+lDuQ+rOwJwqpGwjdVY9qEaiYTZG9k2dGTEw24SoBoLyNsc63gORE5stlIv1fRj1VeQ66KVr6JkX+02ThqNZaBf92NdTynvsbon6ozNocZl0slZ7AnjuN3XsRbJS79tX3eyW+VhfVv2pw2GWt3RsRpo119KgHTpKKdG1jcQGXM50aKgUQJwfaJo+pr1rSj1VrtJgrqnJXTLKMqJ6STh5cwStKLY010VUjFTrE+ymOzaBuUp5g+NbLuWEK5yaqnHhmoirGlbM1MNDXXcnqDk/Dd9TdMX37EAXB2vPY3PNMXMROCtqUamswv4WI3kEi2vyeV73VtFsWck1CoysD6VChj57FPUQ0nldZmhikwtWlHupZ2kuNiLILlvohRbxQ5bSRRXTaCc2hiWFWW/WknP5iSLYU/gM0M5S3C59HS6nqTPthUy9SShGWeEVIPWuH5KPU47wNkCQTeIvd0afJZIemxuY8KsQ1VNrHtRnXy2GeeECFbV9zGNqg3G0bLe6miIH+TUe30whvpoSH0w7KSzMjaJtTFZGkHzDZlsVeMnA/x0iJ+Ndo8ng50k14ed1DaxAiiVIlS1gMRWSusjs2itAE0dZbc360mM6ba9vQfnhdFVEZz+WfdYb/JXtx/t2481sLTa8ze+8E1+4ZWPqJzhlekF77845P337nFrtOSTj4557eEp47xgkFV89OKQEBTfO9vno6eHXG9zzpZDGmcYHa24XvTZVglZXvNodkE5z3lyNuPjs30S4xkPt7x+5yVv3j3l4nqISTyXiwF50uCcZvPBlIt1n/HRisob5ss+dWUpa8v7H96iXma8+M4RzmlCgHvjK9aLHK3g4nrIepXz03c/pakNdydz8V8CyyLDrS3zbc6mTHCNpj8oub4eMN/0qBojgRr9inqVYqznaj7gldvnlNuEpjYUi4x37z/n//j6u6Rpw/PTKZ9+ts/H5/uURUKe1jx68znVdcb2vM/th+csX4xYriUs42rZ56PPDnFDz4O75ygFSb/i6fM90knJ6uWANG2obtWUpSV5npIkjnriSL56xbPH+7jC8ouPPiJcp7iNZTracufeBf5FzmsPT6HSDAcFD956weJiwPhgTe9wQ3CK4DSMaoz10CjKVYYxnlcenXJw91oYYxUwvQafB+4cXouX/CLBpI4kcVBomlsVzcQTBo7Qc1RFwhv3T5neXoANkHicU9i9AjQ8un2OPk+w1kFhGN5acfp4D0ygLBNhIZcJZlSDV5h+I4ygDmxOB2TnimaR4itDvU0wmUPZgK8N3itO7l8yfnSNXhpUozl7MhPm4o0V6pU1IfEYK3GoIfNMby0Y3V+g14byVo1yMPlaBiaweDbivQ/vohcWvVeS3lnDuOburUv0ylAdCuhbfjaWSYQDR1Ma1EUKjzYEG0hfW1CcDgRMJR4qzezhVQcg6oMGvxY2NH+SCqB7dY3aGAnqGctxGLxzhb212dUiTB35cwtOoRJPepoI+3avwK8t6fFGPGxbja4U+pMe/VfnqEqT31lJ1U2t8ZOasFejt1rCe6ywdz4NFLcbKDTJtUEXMgDTG02zX2MvEvyDAv0kJ6wtxYNKKm086K0mmCDb96RHceLE5zrxlMeN+GPPDKN3L2RQNBSpcRg24rstFMWxw00a3NDjpg3JUvy69cwL+3erIWQBO6mkmqeB1ZsV2ZnBPlqx9+Uz7EYG/c3Qi3RbRXCcIecqAlNTKtJzGXAlc03+Wez+KyWdOFlHz9naMP3iOcmlwVRw5yvPqV7bYjYaP2nYfGlL0LD/pZcEJfvYet7ylxZ/f8vgscFulEwQ1IpqFNjeaboEWkBqbrJA/sXrzq+rvDBB1aEjWSjWDx2mgPRSQpJ8Fhh8kLJ+o4ppzyINb/2jBAizmt6Zovez5xT7AdcTKbjLxA9t14rsGgnlmYpHt+mJJ3F7FMjONfUoxM7N6FddarIrzfKhgPHVK47sWgB3PQpcv+s7/2x+alnf9Z0/2KXIOZqLvFbYQGiGsHwowFzXgWok4TX9Z5p6GGXRuTCBQYtcu61qya4Um/sN+UX0Va9FCr65HbraGZ/C9iR03tVmIICuOITemWL6PUeyEo9qPRJwlC5DlzTdAtHlKyLdTxdBKnOG4t+tplG2HMOfqomiOJDjs3zoqYcCVIOBzYmimir2vttgN/K64iAw+CxQHNAF/KRL2NwWIGlK8XX6RKpr8rPIilpYPYDlQ/FWVyNFOYvgOoPsMpCsBHATWjlwILtuO0hBxw7e7Fq8rXYlSdPDJ7L96Tx0PaIoWaaLjPX2UJhfn8SwGScTKPVQEqmVFwl1NVbCmmt53eCFpCTnF9JrWo1l+enSM3guqoV6oGIPrgCxJMp8XaoiiDQUM/HZKic+VgnIkdTmJpd9vn7NMP3Isbol7GSyDvTPvATSBLqE5Lbbd30ktSP5lafuaZKtJ7+Syh+XQDExZNeOxX3L+FMXu08zAAAgAElEQVRH79yxuiWVM+VEsZ2Zjglf3rMxMVr2oRposrmjd+EoJppyFD2ZkVlONj52yxrSlUhus6tG0qhXddebqnzA5YZ04VjdliCecmbJ5o6mr6mmlmTpWJ8k6Cpg1450IT25ybJhfSshP6/xaZtmrUivCtLrBrtxuMywvpVgC5lQM4UAkd5phV0U5Jc16XzXuViPE8q9hLpvJXW8FMtR70WBdlLhYhclbpBIeq3VJPNaPKm1ly7UsiG9KCS0x4VOPmuXpXhIxzkhEZms+EJ912OpQpBkW6XQZU1yFWccXUAvt8KU1rIPQSkBV5GlTV+u0JuS5HKDqmrs1Zbk5RJV1R1wVHWDahwqdnCq2JEZrCHkKXq+Rl8t0RcLeXy5RF+tdgmyTUMY9iWBtm4ITSM1JFkqoNBaupoRpQVoAhjT+SmlS/MGyLxxU0bvKk3i4w6Y3rgrI5Jaldj/1zJ+LG/hh3D/Ebn9WIf35K/eCSf/xa8ynUqK6ZOzGd5r3FKSK4ejgsX5ALW2JCcb/KcD6XXUgd6wJE9rri5GJL0a/a0hxYNK/I2DmvAyF5YogB7VhBiOE7ZWBnsTCc0xgwa3sqhGE3QAHcQ76MAf1DLRVGnyvYL6kyE+86i9Cr9KpDuy70gurQT8FJowcOiFJdhA/zOD+plrVucDutLhSpOdGqo3tpinOfVM2LUwalA6ELzCnsnAv3eqZYb4USHbParRxtGc9rFHW+p5ht7I3ILve7CeZFCjP+iLj+ukAq+YHS65/mQaEzgt/Wea1RsVgw/TyGQEwkmBepl1Mkk3E8/j+JspuoLrnyqFEVrJoLL/ROSiKGjGDjs3uL5neH/B5oMpwQgb4ftx9Dqq4TohpIHhR5bVKw3ZmYQrZBciFVQeTKUoXykItSY5SwhAE0FNm3yoa7quxc3dBjJPcppIMMy9Spg1AK/oP7bYDSzearBzQ++lYvl6Q/7CUpw0HSuWXhjsSgJxBk/ET7V8JOsjwPZ+jWo06YUmu5QZ7paBGX1PZuPrkbBOZqvY3nXoQryZ4bU19r0hppaBk8/ktbqWYJ3D39NcvSOStPLQSZVOrbqwnGoqA9m991oWIvZnbna+0um3hbVQjQxOpcolHtNC/Fvre478hcjg7FY8bOmVDBaLfUl4HH+oWd8N9J+rKNeC1lu4fFU8n1JdIkE/vfPA9Vsiv0TD+EMJDzn92cDgiaEaC5MH3KiniMmac8XgWeDiq570UpOfxW7KvRCDZ1RXIeJTmL7vuXpLwOjwU5H+bW5JXY5qJJ0zvZZzb9fQfykDzfkbwpoMHys2J4H8TNF/KQOxcibbNngWWN8WNsQUgfXdCH7i73Yr0tLsSlg0n8q+LF4L7H9DmKV6qDqmqThow3sCw8cyCE+X8n7Z7gtgsW1H41BYrtl3ApfvCvDpvxAw08pDfSL7ZMpAcaiEQQqybUHLwPbqLekmHHwWWD5QzN73nH9R0W/rhIoYLjOJxzVWNblcalsGp571saacKRl0R6a1HiryS8/ikbwXko30Tq7vQv+ZAFkClPuK9FpSOntnntVdea+m14FkKwBxdUfTP/UU+/Jz8UiTrCC7FnZo8rFnHv9WjSG/EGCRXctxlUoEARIQAcBISapmrhg9qVg8SknW8vx0KaE2bQBOsvJcv5IwOHWxEkZASNDSxZnOBQBWI2GZxp+6GLgCmyNDupQ6l4u3LXvfdSzuGfGHHmnyKzlnPlGMnjZsDuTzzUT2V0CVoxobkpWU2C8e7gZZyUp+aifgJL+WgJ1kI9Uryu8Ce7J5wJZegMbSk13WbI5Tipki2cRQm0RqRzbHCcVMkc0FINqNVOvYMpCuRKpeTDTZUv6HguVd2a/sWkDD5lDvArgcjB6XbE5S7NbTOy1Y3+2RXUn9xvKeFUA5UF1XZrpyND1NNRCgk648+UVDuSc+xrb2JtkE0rlje2CpRorRZ430q/ZFxlj35fX5lQxwq5HBlgIU64Fi/FhqOLYHluzaYTdSeSFeYE16WbG636N/WrE9FB9mNU2pRoZk7buwJruVqpS27qScyAC7d16zPUgwVSA/K9ke5+SXFeVUqlKagTB9upLjml6XbO70sWsnFVEjYdvShZPU0r7FxO5K1QRMIdUpykmYzfYgjesqpPuxdPhcntv0Da6n6b0o2J7k9F4Usr0uUO/lJJcFzTRD+YBZ1zTDFJdLGNHgyYZmJP69ZC7dkJu7fXovCsy6wg0zzLLADTJh+zY1m9s9kkUj/sxFRTPJSJ/NWb+xT//xAjfM0JXUZdSTnPRyK57HWGmSXm67WhOfGEysHbHzgmaSCzAralTpCLnFZ1ZYwRBZ7XFKeingT9cOqcAReai53oD3+ElffIVVE8N5dOysjGExEeT53KKLBjfKMPMtOE9IpSbF9xL0shBAFEFcWz0S+jldZUhVQ5aiFuudFzJNBMy1ATfWoDYFoZ+j5ivCZIi6uIY8E59l46AoIc8EqKoIxCJo7R47J9UjdUMoqy6Yp0tvDcIItgCvuzmHyjJCWaJslPzWNzTfPgioTFIBlCH6JduE2ZvhPFVNlxr7p8lZgxdg+qfd2v9Ff+WPfSrs8b3w2r//Lz6855v/5Y9GeM+PNfQ/6i355Te/y9uD5zgUq0MpxF01Gd9dHvPV2WMu7gz55tUtvjB7ztdG95jmWx4ML/FB0zMV5bHls82Un3jjj/gnn77LOC/51Uf/J//w6S8A8Pr4jFWT8nwz4St7T/jDy3s8Pp/xiw8+Zlln/Et7H/Ab529xf3DJ//3iEf/mg69zXg95Xkx4upry14/f59efvc2/c/8P+K/dL/HOrZc8HFxwXg75dDljsc35qZ94AsDT9ZRNnfDqO+f8sz98m7/9t3+DjzaHfCs/5tH4EoBvvLjNo3cv+dYnt/l7f/N/47evX+XrT+/ySw8/4puXJxgVePOdl2Sm4Tc/fY2TyZLaGQ56a5qg2cvW/P53vsDem0uGRxecb/rkScN800NrzzuHp/yRuUO9Tvn5Nz+iCZoPLg/4qa98yNl2yPOrMathxt/40nv874uf5F/5+ff4rY9f5dHhJe+vboMJDA42fOH4Oe+9uEWySvjif/RNHq9mjJKSP/nsFrf3FnxmDjh5eEHjDF84eM7vfXafr95+wnkx5NsHfR7cueDZxYS/dv8xzzdjKme4/vYJP/k3v8WH9w+YGcfT/oz/+Ku/xX/79V/CJI6mstRbw8+8+glf++QB2Zsbfvn++/zuywecjca8efuU7/zJPXq3VjTLnPpI8ZOvfcrXP7pPvef46S98xJPllMvFgOoyZ3p7wWIzJbu/4iCvuOgPCWc5X333e3xz7xaJV4Sx5u1bL/nON+/hraaZNbg3K8KTPre/eMq2Tjg/GzGYFPSziuaeZv3eHtWtmqRX06xSVg8tLvWY29JuPvinAzavO/wg4AK8dnjF9/b7qOMSf5GiGoW5vaF5PODW62ecrY+pjysGe2ten17z0fk+xeMR+dvXuN+b0X+u4Rev6P3miLOfC2ADhydzQlBUjaGsLJsHge3LPuPbS3oqsPxwir6zIc0a9G9NqaaQvzC4n1hRnPUIiWd0vML93ozFazD44gX3ewWnl3c4/Mop579/TD0O6Eph7q/ZLDKGB2tW2RAzqXjnzgve++gukHL4zktePN5DZZ65TmkmjoevvOST/Ajda6jbWpCBl5CrFxn20YrtRY/yQHPw6JLVSUb4v8YM/rVTNpdj1NOc4nUZyeYf5JSPSl6cyGyynlUMfi/h+c8b7n/lMz79o9voWnHr7Zc8ezHDPk9591/9gE//0etSHXB/i1+KF9LdKtFNzuqRgPzw6lomaJwlXcLiVS81F6NAtec4uH/N+Ysx+7+bdCmq9UACiCTtVHH1roDb4qfWhK8PaHpQ7ksSr7tfcHFsOfhdi0tFdrh66GWCqtaE1JNMS+rCcjq1mIOC9VmOXUtXbjWNiY95wI8bsqcp9otz6t+fiG+yiOm6M0Vxu0YVhuxKURw5qs80zaOCsuyxvStf5OULi2pge1fqSnQtbKVdK9KlYvlA/K7BaLZ3a/qfJnFyQyTMLg9UMRnbp55kKf2L1SxQH1Sc/FPL5c9V2N9J2dxz4CDvm5iACev7juFnMiGga8XmniM7NYyeeooDRbpQbO47Qqz4mX2gOP1pAbtt56q30H8u4HzvO47LrwTqocWuQTfijXOJYvUwsPeedH9evaVJrwxo8cNuT2Qiox6CdpISS4DtsYDj9QORDefnOvaWBqnIGArTXBx6ti81mzuBZKNY3/Mcf63g2S/0Wb9ekawSlg+RvuG+TB4MnkHdk5qHumfIrz2LV4XpRUkacX4Z8EYzf8cx+2MdJzhkMiePbN38zUA614w+VqzuKdRjzeJBLn5CH32UM5E2oizFnmJ7FEROmEHPK4pDkSvvvSfP25xI1cXoCfTOHZtboJwGZVkfGYoj6L0IbE9E6qrrlPUtzfhT8flevGuYvi+BPeU+5OfCfiZr2P+Thsu3EtJF6LywxaGh/6Li5VcknCwYYYKHjyWcZ3sobPbigcVbOqa2OIiTbSMBkbb0bGeGJnoml3ct2dxTjmVyZHMik6e2DCzuGfa+E5i/opm/kpOspc6l2DNsjkT6aNcwe7+k3EtockXdjynLvdjjGxKW9w3Tjxqavo3HOWN1R7P37cD8YUKyCmRLGUBnL9cs71kGL2RAvt3XbA8U0w+BkLC6k5BfSThQ76JBO/EhFkeW3llDMdWYWiZfl/czJh9u2B4k9M5rNkdJZK0TtjNDOreYdU1x0heWbZKxOUpRPpCsLfVAszmUOpBsmNL0DdVIMywd5UFK3VeYaYruW6qpJU801cRit57VvR4uVTS5gOrR1Zb17RTVjKRyZJixuieAXYXA9kj6MpthQjmRWhHXT9ie5PSfbalmKTY31EORla7u5axva46/toWhMHsu04B0f7pMak6aUUo1tigHvc/WVAc5+csNbedicSzWmuyypDjMyc5LfGZwuZXE2b7FLmuaUUJ6sSUYTbM3gBB2faTzEqzB9xJJ4J3mpGdr/CCnGeckF2uq4yHJ5VbAazoBq9HzjQQAxY7O6qCPbjxmLf5GXcZgoYNZNwb2ue0SbvWmIqS2e4wPhDxBVY2kxWoN/VyAZFVLomyaSHBQ6w1vAWkIcLVADQcCmBMry4AOYALCntY1KkvF99k0KBoCoNqUWtj5MSEG+nzeYyk782cIKH8Q4Pyr24/k7YfKWCqlpsA/AL6AELV/B/gu8D8DD4FPgH87hHCllFLArwH/OrAB/sMQwh/+WcvPHt0NJ//5f9L1+OnEiwe50YRKo3JJJnVZgGkFc2HyMIG2ykOtDZhYbzHyIj81AbvSkekQP1fbtRZ0IL2WZM2uJFuJHEw7kXTpUkWZXZTgQFdm34ZymELCM1pmqcmlP6qNjA9GBgz1SMJEfNrWW8gMtsuF2XK5MHtteXewsn2uJ14ZJ00q4jPx4jtK55p6ELptkzJqYfvqYayfiANG3dClPkqBNLuahkp1vqy26qGNkSf6lXQtcq/iQAZerSSp9ZOpIFI25doZdWE6TCXHTmQ24r+TGoTYS9duepwQ8zFUpvUlNX05Pmhw8dgFJexIuy/KiYwru1KdjECCSISp0rW8nujp0rUMwn0S9yPuY3v8vJXX2o1sn7dyvCT2XlgqXclxausN2kqFtvNLBTlnpiQmQwo72Pq/2mW0kkV/4zNe6g0gRHaq7ftzedzvhtjVJj9NGasknLBfLtstu2Wj2n2D+P/2eo+9iKqJ9Q091SXVttUM7XXgMnl+y5q5GOwiB5zOn9SeF3ej/y7YuK64LaohXnORyYnX5+e2s7djNttzqpywc/V453UL0b/VBW/oeM6KnbFe1yJ1k2v2xvlyoWMDpRpEEkiF4RXGOlmIjM9u5XcQps6ndNI1CXlp3wfyWvEQqu76SFbCuiXr0HkSJSVSagfaW7rYsXLi25Hj0u5He6zFE6qwawkpKSciafOpLDNdBcqxpHo2ffGuhXidhbi8lkVrvYlt2nIbuqKr3XutGsX+Qiv72Fa9dB7CGKqiG1m+TyVh1+XxM6TZvTfaipf2+U17bd8IF2m3x9vdMtvPJlsKm2cLkRAmG+keTDZSUZFsPE2uu+NmCwFjQUn6qYvJsTpWYrR1Kt21TPt+3/nuJMF0dw1LjYewvC2jWA1ELtn6zWzhxSN3I+TC1AHdtIE1LTOpY8/g7rMwKGHu2m1tP0db+XKTRZATQ1ZMJceiPbfeynEKWnXVPe37yBa+C9/RTeheg4pVG6H9rJDl6lpYwpYpbiszOhl1W1cBJMtGPHaJwlS+O4Ym9iC2YTOtcsEWPoIHoiSz7SikC6AxZXytBbP1EjITv/eUv7G9PQEesg9yXHZ1NjfeH2WsnonP0ZXHZzpep6HzUIrcM1aFuBg809aFALrywm4ZhU+0SDDr9gTuriNdOvEhgjCT8bkg/wtWx3Oz62cMcRvaig/53NyN83yiJcAmhgqZ0svfKtdtq3IiWfWRCW3XIQ9Cdw5UAF25rstRl072K/Y4dsmitQBDFQKqljRXnxh01eD6KboWeaeKzFkwGl01HWOIv7FOH6QipGzwqRX20QWwwpC1+yA+Rt9Vu4S4j50P8sZNla0PUHUACI8s88YYWTUSfhNiFyQ2fgA0rmMoQ5rspKg3uiFV3cgxbNNVI5unGifVIc535/Fzt5ZZbBNW6yjjNTe27cax5k8b03eBPaGTqoYbbGK3jDaEB+L+uB1T2LKoN9nC6HvEGAGVSkX2My673Wbvf3CdSCuPvcFI/qBbaLfN///3Wf4oMJb9HxJj+d6PCGP5w54G+DXg10MIbwFfAr4N/GfAb4QQXgd+I/4O8CvA6/H+d4H/5s9buNKBr779MXmvIjRKZN4bS6g0ZljDMqE5rAg9R1hbwtBhNhqzsOilwV4kHTioD6QyQjnxLRGkI1LVCrvWInEtpRuyvCvR4yEJ4AUYagf1WEJQmln0Xm0U2bkGr0jmGj+TMBtTxi+tJs6eTjy2UDFgQuRuvicJiu2y07nuZl6bnjBByUpqKuxWYbcCvJLY42g2URYZPwMErEF6JWEipkB6Awt5nQRgeJKV9DS6PJCsZd/yM0WyUJ3PqPON3W5EOrnnyC8kFMDngWQVj0kF7pUt6y9vsSvZXtcP0b8mo+kWlAcr4R395wLmZJsEZOaXcryKQ086l7TIdN6CSbpaDgFlsm/JMgL3CrJrFcGpMDguk8Hr+lHD8EkMQxhJcEiwAoS1iwEgNkjlxVq2odz3JKs4YIxdh8lSglnqiQzwqnGgmsh21EMBSsFIx6HyMdBkJB+QEhQSOkAu/YDiTWpL0HXsP2z7QVtQVU0DdiP/03FQny5ltj/EEIi2izJZQXHsZXC9jWEeE8guZRC/viMBPT4VkFkPpZ4inQtQKw/C5yYLgtkN/pcPZSCXLIP0JzZQHEntRTUhJiPK3+uh/J6fy/5vT6T6wqURkKWyL62PqgXLrQcMJEglu5Jro9wTQGUK8aHpJkip9wZMlE8KUJfaDrsWoFYcyLakS9kvl4sfy67lGLgUygksXpHl905l25JNoB5KiEqxpzBb8XQVB/H9a+gCT7qfY/lf0BKgUuwL8+MTOWbVWNZlSqmHqCYC8EwZ/WRWnrt8pFg9iAB1GyKYFh9aspL6DFPJYNnlO2DW+tOaGCpiYvVZM1AsXhEQ5VORqwIUe5r82lPsS/9jNZX3silj4IuRgJR6qHYTHOlOFpysBVh4K6yTdkHCayr5XwtKs7nITNuJq3b/80tPORMgbQthqaqJrNNuJKAFJLE1XYZYfyNgsR62QAnyK9kfHZNlm54MpFWAciqfp5sjQ+/KSUqshdVdI2mVmQDC7Z6EvPgEqqGwh9LJKPvvUiVhJj0Vw2vk93TtyeYel8H6RLM9EEDQ9BSbQ+k1FKltrN24cJQTWU62cGwPdAcwvBUQWY5jBcdAsb6lWd0yJFvfTZwoL8EzbXJqNRQg4zL5vcmkViO/cvFzKMr0928E0EQgUvc1LmUn7dyIH3m7Z1jela7EFiTVAy3gxEgAzfpYS3F9X7E+kn5CU0cP4UrOSTmRocfmILJ9GlZ3UgHaKwEZyVqqS6qR7o5vspb16iaw3bOYUmovCJBey5ddMTPYtZOeyxsTK66nI6gVaWw1MgKuUtX9vR5qirjczaGlmFmanpIAGgflTPoiXU9A4+qOBKLoSkCBbkIXPCOTHfIYwJQ+Ts46mr5hc5RKMnSuMaWjnNoOqNcxPKc4zDCFkyCbnngAzVbkusVRJhLPkSSaequox1ZksTElVYAUFAcpPtXUY4spHdVU/IJ23VBNbrCD+xlN31CPU8pZFpk/Qz1KCFb6HU3R0AwsPpNU1XJflGJ2WeJzg+sJSPKJlvCbVEJs7LLEbBp02VAc9tC1o9rvYTYVTT9BOU8zSHADCblpxlnHFKpa/gfQjFKR2/YSdO3wmRWJaiVAs57mAtCcl0qQ6KU01xvpwgRh+WJdRkgMzTTHTfI4sFS7jsq2czJ2V7agVHkvVSGxN7INxAl5gnJOQJzRIm31HozGD3P5vf1bmsiESJrswGUTtysEWXZVy+MI3micvF7fAJCt/LX9u1ISstPKXa2R/7UdmWkiXkWt5R6PT1cfAh2jqFr/pNK72g9rQbdSXQtZJpLZ6IHswHQIqFQkvuKfNCildj/bu7VyN7p73AX9fN9dJfYvj8fyL/HthwYslVIT4JeAfwgQQqhCCNfA3wL+cXzaPwb+jfj4bwH/fZDb7wBTpdStP289qzpj1JNEUN9oVKUl6RIIfZmuvvfgHJKAzWO4xl6NHzmavZqQeBjV6F4jADSNoQn9AI1807pMBrnNSJJBiWmbqla4cUM98TGdL9YWGBncNUNJPwxJoNp3UBqaQaAZS5efz6VHMCRBGMtSEdJYx1AKMKpmsmzXCwL4elKjIN4mqAeBaixf1tXtmnocwMeKhAgsvBHWT8CJLKvtBXRZiIBCPqSagVRtNCNHPZTXrB/I9ha3YmjHuIoR8bJNybTo4v2DkmNXHjiqWzW+1vT/uCfJoYa4TxK2opwi6Bi+EoFQPYLqqImslQzAi702MdJTTdrzEz2AQfZV+ViHYWLi4WhXsl2NdoxLe2xcJsC22BcgBeBNoJp4ir1Aueekm3Hm8Ums2+gFfE+WVU92ANVlcSDf8yRrAcZNPCdtkX257zvW0uWyv/UoVnn0hcVq2ajmqCYYSQpt4jnwifzuMzlHTQ/MRhIxpXA+xIQ+kcSVUzmv5b5n+1op2zpq8HFbmzyeq2EEvXPp1GzTQ30ioLBlpFwe2N5u5Joby7VSjSJbbUPHsraMt8skFTNEdjFZy3EQ0CHHoU0VrUcCLuqhimwesU9QjpNLBTS7TF6jaxnUN7H2QpjCHXtS7Mly2roFlxNrC+hqKuphiKEq6nOgtb2GAZqRSEWrcVzeUJgmCR6R60A7ARm6gvLAg96xrhC3Oyde87tJBp/Qhaa4XN6T9SgCgF67DAFUsGPJdQSFkgIqg+Kg5HzmF7EYnt05lC7JyGCG3XKqGPKSLKW3UnlwiawvnQuIaYFpO6nR9HbMc3btYxJjW7MRz9lARWY7At9aZLYtkHapMOYujyAs1nlUox271uRSadKCNp9EOWFkau2mDW9RIi0eybUQdLu/dExNW+mRrEL3bReM7Iup22OpOh+ujX5EYVbbOooQwR1RBaK6dXhLB7DbVFeXxaqOTCS17fmoe+LlNUWgyXXHqjW5ou5J9Ueylv/5mKbaJs82ERyqthrD0zGEbaWCSyX50iUI0Eihq3tIdxUtEI9VEhUn6e49UPciUM7bbZP9DEaOozDCIi0MRlFHz2PQu/qTNnW1ZRclyTRuR646BlcqN+TcKi8dncKyxvM6kBqNjoFO5O+EeA5aJkoJM9nWWeSXLqaH6q56ZVd1oSITTJc2HbTUe7g8dp1a1bGUkvYrEwt1X/4f4oQOyLXRpqEKAJF9lX2L77lUQIBLtYDOGyx0x1AFAaF263YTBXFCLlgBse1nU5vyaQrfsaNBxwmf7tpRuJ4mJDJxJGy+HK+mZ2TbIiBpk1pDPE5yHKKXLbLG3b6lwkwGJfUoPtVdXYbPrLCOkXEMWqGdADtJpLX4RAvL6EL0Onp5XWTU28/ekLRv1iDA0nv5GYj9lImsMzGRoY3Pi/2MIZGqj2C1sJAm7i8CKvHgBtIJ2Va0tOqPtkZElY3sm2f3+hbMNU7WkVipDHFOWNXa4Saxc1IpaByqcfh+umM7W8ayuQlubQcKVeOkQqStDIkpsKHtiozMZEgTwqYQlrQo5f9ltXvcPs+abj2kyW4/tCY0jlBWUJbys6qhLKEsO8D5/TelY8BO3ewkrTekrS1w/Fzn5Q2Glu9nStv3wl9UPfmXpG7kL3N4zw+TsXwEnAH/SCn1daXUP1BKDYDjEMLz+JwXwHF8fAd4cuP1T+PfPndTSv1dpdTXlFJfaxZr3v/WXc6/dUDek1Ce+2+cEgB3kZEPK45PrnnxByeoQuMuM/SwhlqTTQrUxmCGDTZ1JB/1UGvD7dfOUA5+9qe+i2oUh++e4ccN40fX0HOo1EvIjg30HyxQtebuGy9JH65gWuFOZOTnR04Cco5q/KRmdHeBnRvy+0tQItM0a406Khl+z9BMHby9BCAk8iXic6kScLNGQM3Aobxi8MTgbpcU9yrM/TXKKdKfvJI00KOC/ELh7hX0nlpcz+P2a6lx2Gvwr2ylV1CDu1vQ3C6xbywJiUhuh+9ckpwlmI2mOaqoXt9CAPvOguTawMMN/W/0qGee6TcS6kcF/X8+xHxpHiWBwmBmZ4bsWcLoD2UW0KeB7SulMJP3CuxaUY+9sKbrKCMeBoojR+/ThOK1ku39mnLfU099TJc0XZXB9thjt4rtkSeZq8hECkjN/tpl9HNJ9UO959GlgInemer6/9mRg3AAACAASURBVOy1xeWB4ssb7Fa+XO1WUR04xh9IqXbvqSG91FT7jvqwYfShYfvlDd5AcexoBoH1KzXZlWLyXfmCak4qJn9iWL3SUL22FUYy96xer6lHgeFjTTqXNFFvYfiJxLZv7zWsHzaM30shwMnvyKTA+p4kVQrI8aTXiuLESbXCz11RHDmaHqzeqqgmAkDzc9mfwRNN/4OM4tBz8NsJ9dBTjQPb247yuGH1as3yDbm+kpUMjFf3vXgAh4Hlaw3ZJYw+0sy+YcgvZLCzfthExlRx+595yr3A5U8ERh8rrr/YcPy7gYOvQ+8MUDB/3aP/+iWbk0Bx6Ln6uYrNbc/ee5r5Ow35uaRd9p8LW3751Ub8ZQNJpEwWUBx4TBHIz2H5Vk01E4ZxcyJs5OHXPddve8oDef9sjwL1GIZPA6sHntm3A5vbgfXtwMEfyTFavO6E9bVw9W6gOJQJg+JQnjN8HGgGcPllT/+ZJGgOPgv0TxXZFWwPpHc0Pw+MvqdpegLwiiNhNuuhgLc7vyUqB7uB4RNJsBw8DawewP43A0d/ICC8/yIwe9+zviOv3dwObA8kfGb0aaB/Kvu6PZRzVU7h8osShFPOpHezHquOFVQBticSDJVfBabfcywfKsafCJsWDPRPA3YduH5LgMH6tmJzS9F/Frj4ogQObW7J4LD30pMuAotHUsWQnwdW92Wg3XspEz3bA9XVUOQXgdFjz/hTz+qeTAis7wlQXzxSpAt5jW7g8l0BTBc/AbPvNmxOBDRm14FyBoMXns2xgPimJ+zQ6o5s3+hJYHlPM3juuX5DvtbOv2iwW7qJC+UCl+/K98j4U8fmSMJ5rl6X963dCmARGbji+jXTyb19qjrFSAv4XAR44mtUHYusfGD+qmH+SDN87tj7dsXoie8mQdK1MLOjp471LU26ChT7uktevfiiiuBTxaoKovdRmOrhM8foidznrxiKtg6jB/0zT7oObPYNw2eu6/cs94SpVg6u3rCsb2mmH1Vs9zS9c0/vwnfAvHfpGbx0mCoweOkoJ5qzL4mXTdeB6YcV80eaYir1GNnCCyu7r+m/dORzx2bfMP60ZPCiwRvF6pb4ZMuxZnXbMP6kwluYvV8xelKxui3SyeHTisU9KwmmM0051jF8Ss53NdRs9w2mFjZ7dcewOTTMH1mW9xI2++KjXt221L14HY40dU+uie2BoZhJiFJ+4SgmRthkHyhHmiZKi8uZYfLRhsHTgtGTit65I1l7Rp9suHg3w6eK+Sspkw83uEx8lpdvpWwOragCtj4y9UY+2w+ixLcIlHsJwcL4/aXUXRjxP+bnNcv7OfVY0ljrgSa/bFjez4Qh3XrqsWV1L8PlGrsV5jPEeg3lAunSYUpPOUtIFo7rV3OqsWX04ZJqbMmuKuqxZfjxiuIgpThMGX5vwfYoI1k1DD9ekl1UKBfIX2xwuSZ7OhfA4wPVLOXi3QF23dA73Up35EI8B9vjHLupCUqY06CBG9LcapqyPckp9zOylxvKgwyzrin3MvLnK7bHPbKXa+x1STXLsIsS3XiK4z6b+yMAXG5J5gXVJMWsKsxW1lfu5xS3hjSjlORqKwBvkEjXZW5xvYTNazN8z1LcHlEdDbBL2e42STV9ckUzyiJQ9NRHI/E8Jga9LqkOBzSzPmhNczgShjSmv/p+ipv18QMJ8wmpFaC/P8BPBujrNUSAqddbQj+T4J0I9EI/EwlwnuLHffFojvuEoYDU0M/l+VpHT2QEpMf7hCyBg5kA0tkYjvbipIjC74+Fua0b8VPmaQcqUQqVZ6jZBLU3Q03H8nh/hppNJSBHKVSeo9IElSQoawlR1qr6Pdnmouhksm2oTggBlVhJg23rUSLr6Kv6c3dJmW0IXeJss2NHv+8e6uav6kZ+DG4/NI+lUuqngN8Bfj6E8LtKqV8DFsDfCyFMbzzvKoQwU0r9E+DvhxD+Wfz7bwD/aQjhaz9oHeM3j8O/9T/8Cn9wepej4YonV1O0DmxWGZPJhsWqx/HegtoZNqV0PPpao3TApo76vAejmuAUvVHJ9qrH4e1rzj6bMjxcU9eGpra4hQRQ5Idb6tqggGadyExWEtCJE6VAYSRRVMeZ5MwTCtMxmMmowjVamNWlJaQBrIegUBsjQHSRglPC5tkgy9KgVwYOSvwqId0rqEtL2FhUqcEGKYbXAWUC+izFDRz0HPo6kfoLE2jNGaowmI3G9T2qVpJ+awL2SoBWGDWyfVlcZuoJpUh6SbxUalxmhEEj++sV6IBZGNy0QS9EsuP6nt7hhuqToQyQItvBpMbmNeHTgZzIIBLa7EwLQJxJiEi7/3olPY56VMNZRkhFluyzQHphqCcyqxgM0unXCGj3iZTUtzLYoAWsq0ahCw13toQXUrasK0mgtUtN0/eENC576lFVW6wur7Ur3flLXN72DwbcxGGvLOO3L7h6PBPf7sCTXRiqqe88mtWBIz03wuIq2ebkSkI90FLNoSol56i3W34wMsjUtYBju5FKDD9u0CuLLkQS3fTkPLdSbTduSC4s2dtzig8mXVpn0w+kC0kzbr29LRjx0SuHEia9GQR0lBvrKk4KzDX1UFh3u9KYqgXtoUs9RUvtQ3Es4L714aVXAlyrw4b8WYJLhen0qTC/veemK3MPdtcVaUqFj32NLQOeLNpZdZmcAMDLcfJJQDvZLp8FlLTSxMRbRXYpYTooSOct4yTMVD0OXSm9+PRE+uoyduxMlCI3A3mP27XsVzoXNs5uoJzF6ocoUxfWHHqn0reXLKUrsu1fJPrxXBbBTh6Zqr5sT3a98x/LSaOTUiexAqIe7I6R3dCVvLtMZMgiLY3nt2j9ZMJ4eyPvm2BEVl2NRL7qkyh/jp2Qra+1lep6K/tiKmEEW8WEKWQfRJ4s8mW7iczpTB63PtcdywrZdWR0ozxaNbvtwEcp97XsdzWBND4/XUQ5YlRryHsm7psV1rz1LFczCZVpQWI1Esl6sMJeSj9ilKG7nWRX2MEd033z/xC9tJEtq8ay3abcMdbJUtZpt9FnuZJk5XQuQLsayd98ZBVNGbrOwa6TVAuoFcl+ZJ1TAQB2SycLrgfR5xsln3Yrz2/fX/VQSdJzZIJdLqyw9ADG4xmZ7mJfYQuRmbdsp25C571vesJS9M8lZbft9mxvNsrTm56AjVaiPXjhqAc7r2IwUVZqYsjVRmTT6VqkiU0uNSC9Cx/91KFjZm0hADTZhuhzDzGIiF2foiWyfLJfrXwa4jJK8Ssna2GYTJT9uuSGB64laxo676uuZdtMJf5lFQJNFj2mAXzcLxc91NncS2epD5QjQ++ywWWauq86f7At2s/+yJLW4XN+zXpgRA4MElITpcKt/1fsEwq79QKco39X/NyBcmKk8mNP0mRtZEJdrskua+nKnNoohxbA4rK4fgWm8LhUYyrxvLbrMNEDG4z0R9qti97VQNMTIBy0whaOJjfd/10mtSLVJCG7EvDnE03TM2RXJeVeht06mczZSAiSdgG7rKj2crkeSo9d13irO9mojnJau67xqcFlRoCwkWVrF8Qj2nhCYjCrUpJlnYSRoZWoA3xkMaP8Vr5rjTxWiqA1aGFGfT+V7kqjaHMHlHO75bSvjX5nVTY7n+PNn07ktKqoCL1MmEVjxMt5g8FUZSWsaJbu2NX2a6IN8AF5vTVdYmzwHhXlqy0mUErtpK03l9X2YzYifw3Oi++yZS1jt2Vo19fWjLTdmt+POW4mx/5F8IgPhLr685/3A24/Kh7L1/+9f/Eeyz/+tR8Nj+UPE1ieAL8TQngYf/9FxE/5GvAvhxCeR6nrb4YQ3lRK/Xfx8f8Yn//d9nk/aB3912+Ft3/t73A8lOTTSbblq5PHPK8m/PbzV3h1di5JptdjvFfcmi1YFBnjvMRqjw+KopFvk2dP9vmVL7/Hr3/rHY4OF5w+nzLaX3N/es26TtEq8L2PjqU2pDbMxhumvS2Xmx7LVQ8fFMEr9qZrruYDZpM1vaRmWyektuFq1cc5TXWVc/LggqJKWK5zrHW4xnAwW/LyYsx4tCWxDqM9L57PePPhc773cp/RoODqakjWqykuc8bHK4oyka7I2mCto64sTWXI+rUA2KDI85rNKsMmjnqbcO/OBasyZVum1LVBawG2rtGcHM5ZFhmr6z46dQKgCwu1wgwbtPbUyxS8Yni8Yv1khN6v8E6hrccYT/hogH1jyXaRo3QgeZoy+clzlpscYzzriz56adCNov/GNctFD6WCdE5qLx2f87RLdVSVgnEjAP2mVgYEDA9q/Iuc5O6a8qInnYRZwMxKmsKiU4ffWuygxtUarlN0A27k6D1J2D6sMFcWN3aoWqNnJf4iEz/tSYl+mgubbAJqq8nPDNtXys9VkiinCCqGKI0aRt/I2NwKNHs1VBqSALWCnoNaY+a2q1FpAbAfiBdGlZr8paHci7LKqUx8qLXBbkSumiwU9ThgVyqGNKkomQqEvQp1lYr3c6Gpx1FunXl6TxKKA08Y11BKn2C150BDemZwfQGB5YH4LbuQiEokfMFGgDP0ZGeG8tCRLHQX4lTuOzluBgGR8XQ1BzXmWtKAbARmzcSRXJkYgBMIfUd6amNFiUyUmLWMom8GBakA9dRjYriWrmVgV0092aURiWoHQgXgtGFKyVpk1aYS0Kka8akmCwnBcn2RdmdnMSyjFGBSjwU06hp8Bnalun7O4sBLz+OFwvVkX9JrRbnvyS405Z4wo0GLtDe7jgXlRsBkspZ1Z+dKKlSWrbQ0oJvoJ76WZbcATPzJ8lg10AZMpUtFuSfgObtWHYjpQpEyWX8LdLMrAVP1KL6tlIDtZLHztG7uOPKXpjuerTQ6P6cLYfKWeEzleNcj2WbtIlBMBCA3PXB9qWsRX7OATh1lpXYjoFOCxCRYy2yV+GePA72XuwmEFjDXAzmvsNuOdttdJkA22HgdbNvwJNXJq9uJAblmd0FULpNjks2FMU5WUlFSzlQHoFWUB+pGQo6yKwFx4tWTayRZhi4kS/x0dAFgdisAP7sMVFNF72WgnKr4niN6AdXnnteC0VYS23qDm0Ec7EWgNHrq2O6JH1NCdKIcNo37mApbW02ENRZQKEzt+ElDMTNd2JBMJMT30Cp0TLeuBPRIhYkchzTWhNQDqeSphqqbMOrSUdvtyeT4J1tPMREgWMz0Dsx+X/iSBAHtwGDv0rPdk0F8k4s3N9yQ5FUjkci2YN9UO7msN3Kt2BjuU/fEF1uOJLQqv2zYHCUC5lS7bi1djY2sL9l6dCm+TJ9Ej29Pka4kAGq7rwX4xrCnJhc2lBBIVj76PLWcTx/kPI+0sNOFeFrtxlGPDE2mpbsyhi+1HZatHLiVRSerhnKWfA54yr7LsTGll4qVjacZiFQ4v2yoxobeaSWe00THqpOK7XFO0NA7LammkhDrMo3d+hhAFtcRJb/pokHVnnqUdOvWrVS7DTmqxRcr4UqeemRF1quI4Tzy3Py8oDiSHu10XuMykd6m15LUatY15X5OshBNezOw2HUjMvDM0PTET2rWNT6zNH1Ddr4VT2YMFKqnGXZZY5YlIbcxuCjE5SUSkGR2+6kr10llW5+nBBK5CCrFh4oLhMx0sttOG9hiqJvS3LLuAFcwBuU9N8N92r+p5QY/GaI3xeeksy1QU8sNYdTfPc5T2c+tqOjCqA91g1ptuuqRLnwIBLzeTGt1XhJftRZQ2LKkG0mvV0ki8lkbtyUynDQNwXmRy/ogDKQxUNd8fwVJe/v/JIMN/se+bqR/fC+8/u/+EIDlf/WjASx/aC7YEMILpdQTpdSbIYTvAr8MfCve/wPg78ef/0t8yf8K/KpS6n8CfgaY/1mgUtahOH85ZlMm9LOa58sRF8WAF/MR23VG1RgW5wNwitHxik8+PqK3t6WsE7KkZlNkFKuUbFChCs2v/8m75MOS1AjVvvpszMdO472mmGeozFPMM8zcclFrLvUAXxpsv8EtUpRTXCnwFynnW4POHL1BxfnZCJs5eNyD/YaXF2P8KgHrcdpiLi2nlSFUmvn1JLKoGnue8L4+IQS4ro0Uu5cGnGJxNiQZVmyWGaHR1NYTao3OHdWTAX5Wk5ymrA8seEXthGI7vR5RnwlTq0ygLhMBR1vDs9UBelijLhMCiTCHOqD6DldpXG1RtSY9N6zUgOxaUyYim/WHFW6eYrJAuU2g1OAUulLM/+iA6rhGpR57ZWkmDfrKsnw8RpI4A6rUeCVsaTLX1FMJWvJ5IFRiwOg9Tdg+qEjOEupDASuu0minKC976EICMrwKNBuLXllJ7Es9POmhMmE6lQfm0k/ZDtDtlZUvxyqHNA7cr1JMqdDnwlwS4qBmI12mzf/D3ps8y7bl912f1ewuu9Pe/t1333tVr0pVkspCCMtgMwUizMARBBAw84Aggr+iJhBBGIIBBgxDM2PCgBH/ggkjLFBjqVTN6+69p892d6th8Ftr53nlkuUQKlyl0I64cc85mbn7zFzf9e1OpccyVJHywaCcnOdxDsVG4Zaa+p3FzVOCKGAO0mMZ9iLrDTZS3WnGtF+hSszXQQbRfiyFDW6PgU9ZQmt6NZXGD6uUEnyf4vETc1lsJazJBfnCKB80biySj0ehEkC27XFgSpDApnEVcQtP+WAliGguYAe0BCP1Ct2rSZ5JhOpWWOf6VhgJNwd1U0hScp0nBuR660HRXMH624HmJwW+FrAsTKxUNYwpwCpUJEARqd4bxlVIr5cyedNqyjV0l4lF7lN3Y5sG8lWq1ki+peYdU1oummkbdpv8a6OAtqIXb6WvI/WNpr8QGWsuXo9GS2BUYu7KhwQoduJtsm1iuTpQCSSbTgbr9SgS7tlXwqiWayVsmRZmKKfihlJeM3sfJGQme9MS8+wT+2r3TN1/MjhNzFSaECi3ck2kekFAvvIxpU4nhqs7Ap75V5FxJUXw3bkAhPo6+4lTSq6G7KU0KUnU12rqprQHoGZKxrW7o68OldKadQqjWkmYVH8K1X3ENkcwU60FbPenKnn4ks9tLq8v15HDc9l/V0O1lvXZAxCguToybFGLXHlYicx2/1JAm8gtZXxV3csx65EprCtqBASeyHtT93I/2hYgJnY2TkxgDiWyfZRwt/kRSA8rNQFjM0xvC0kuXijp6jzTE2AvNnIvlbswsXLVJrB7YR6lMUfKO/GI7l4aqocwsczBimTVFwo3FwZJhZzAnUDcoIgm0p4Le4NOycM6+1yFhasfAtvXVtQ0kNhKeV8oL12nvpRJo8Vbz+HSTKygL+V6lLvA6DXVxk/nvXrw2C5yeJISZ7Vcq2L/iH1LzLr2wuBpD3ioegG05TZOScTlNk6gJs9HiiJDALAeowQPDfL+DUZRbQPaRbpzS/3g0UOYQn9IgULduVRu9CtDc+umCbhgSInfkuo7v4oTyxi1AGrbhikoqXpwqGin5FkzeElDD2D3XtJi02PFXvbL1woVZV3F1h29nTaxaQqKvXR+6iEmNhSK9ci4LNBDQHuR0cprwLQe3Ygn1XQe4yKq0NOkiR4E6NjW4xP7anov94s+MswStCOJrSqC3Y342iSVS0hsqnizzWHEn9fo3UgoCsq1xzUGu5cQxWgLolEUG4dvDL42EvzjxdMZGwGBtvNoFwhWHycBkhzX+lEmN5O0VcXkNXzk/bPbEd2lGo3OERfJHO8COiXFKh8ltVYn4NiJBFYNDm2Tx9QFYTATKFQxQkrIndhM58UjGSJRW1Q7iKzVeQnb8YGpliMnvCIMp+zsI4bSe9QYv+alzL5NqU6RZN9ozRF8jk48qEpNoUNTDUneZnzEKGYAqDU8lp9mxjImNrXvpwAfeV2653149Lckn/1Z6bB/nlTYv1p+oZefd93IbyB1IyXwQ+DvInM3/wvwIfATpG7kLtWN/H3g30HqRv7uP08GC1B98ip+9Pf+U4bOEoOCTSFJqyqxNzZSXRmGM7lZi42GIAzENIuZaj+KjbBBdn+cFS/XMoOuAiK9OhWJW07DzAXyOlVj+EbemAIAZP25niFa0sAB+otUPN8e60jcTAbj4xKRcKa01u5StulrmSWf4vWjDGzGZZLypJ8hS3uYQg8IMjg2LSJdaWFcMdUzZAlcHiyNqyNLonsZ2Fb3MlCbZpBHAQ3zLyO3v+WZ/cRO8kA9ir9NO5EGNlfHYJIsic01C6ZHAnTao2zN16SCdzVJ2Hwj10NCDVJtRGZJZqnGoj4et5ul1NYk/wuFDCYlVOJ43MMJrH4kyYzSMyfnOQ/cs9xQZo/TuUyBI/Yg1zXPoOfI/+ohTkEZ1YPM8MtAUvyhuTMtFPm+SSxNLwOc9TcF6BQ7WW8+ZypI2qmbyX1khmOJe3+msXthD0JKQRWmR8BEsZV3nk7VAraL5PTQqGXfXCNfiMVeBmZyjCJr7c5VOodyTmbXQbxJLk61J8WeaQCaUzAFXETqtdQ6ZBYid+ahhMVwjcj6XKOo1gKgJARITTIhe5B0yf5EBhH53o6JzPaForkLEkAyFybGF+Kxq+9yYAhpYJuAwyQ3O4bTmEGOyZdqClzKFRc5pTQYOYdZ5mf6SPtEU91LwES/UsyuA/2Jpr4XAGB6GZSPM03RSim7GQTgBKModykVN9dnaPlX7kTCZnoZOLk6g7HjZ7cvFLNrT5sG8drHienJrKJrErAdRVYo+y9MTLmT90DUivnbgf2LckryzINHM6SqiiEyuxrYPy+n/XCNgISxkV7D4hBSlYeZZH15kG87AT++EkapSGXyUR0/s3K4S7CyvXLjGZYmgQmp2cjHkWtDojmel7we0wkzZA+B9sKy/Lxn/6KcmCrTy4DX9HKOm/cD/Xkhg+CUhiqBYJpi71Eu0p/bxIRFxrkgnmLnJY20l/eVfGik74AUCCPvSZ+SPI0klZaKau0liXSIAuQM6X/xe2b5pHhcew7Pq3QeJRjFdJ4hJYrmUJ9y46a+vgwOQqknCXBed07Vtm2AEHFzg6v15NXztZ5YlmgV5WZkWBYTC5f3K1hFuR1F0th5uvNS3qNOJKDKpVqOvASm2o7jZ2GUMJggn3W5bsS2XlJBF4UkspYyaVPddOKhS+sNKfDFDJJIKgmsIol0s0LGBUmWqWJSIQwePQbaZw3KR4qDwxcaMwpYycDOdlKpYXcinZSqiwR4RrkOevDEQqPGgG9sYvZk30NpZH86f6zomFl5bm0lLEeDGiSARg+eUBrMfsCt6ql6w83lHOjBT0E+6nEdxugJZUootVpSU61Gu4BuhbkDeZ12AV8ZYc98xLTCKPnKyHk5jISZJI0qJ4AkV3/o3uFOa8x+lIRVrQm1xWx7VNcTFw2qH4m5osJq/Cz1MbZOwJLW4ALutMZuewHIRUpt9VGAV51maXyUuo6mPN5DSbKqO+lvFHZukOekwJ+oNcpL0E4+hgyiQl0Ko6hBHXqoSgFm1vwzbGHUGr1vJ1YSpY7ALYiHMae7htUMfbsRsJh6HOO8QXW9rL9KSakpuZYhMXmPpaQxTvLXaZ+GEaz4FlVxZITjOEqSqtZJkqon5lDOvT2udxR2NDoPwTOlu8bwNTAr5zwcw4JyGqzSTFUfo5u8lRgt+wdHNjEcgXJM4UZyAzzaxk9XnPyLLH9ZGMv/8OfAWP63vxiM5c8VWP68l9W3n8VP/uv/hO9evuftYYVWkZ9cn7GY9ZRWbv52KHCJdWyqgRgV231N0wzUhePm3Yrl5Z55NbA51Dw/2RKi4rN358wWvchIS0/wmgisFi27Q4UbLPVsoDuU1LOB9qEWb2TjiYkF0pXHGPkGdYN4Is1yxPcG1RrseYcbDHFIb06vZBZYgZ2PuFZYNDQ8e/7A1fUKXQhzpo1nPJTgFHYx4m8r1OmAsQF3XYs/sQ7Qa+zJgDtYYVbvK/TpIK9/qL92Ps9ertnta9xdDctUqXKw2JMBf1/BYmR50rL9bEWcexg0utOTh1MVgbOLLXfvVyL3vBywlYMfz3HPB+JB/KaqCujSw9sav/SSvquFjYtlQFUettI5qpx4TeuLlqGzhINFt4awFH+n3Rh8EyRs6MSjGievbTx6Iz7WqCPF2jBeSnCTGhUY5P+IyFCDorgzjC8Gka/aeDy+KqA7kWjFmaf6qhCJaSMe1XwOYu1RvUF34v/UncKvPPbO4ufy3FiILFX1Wli8RUC3emIhtRePZV5UL49FLfLd8lYYCt3D+Hqg/kGFryNulsJ3tNTegDCSRIhVBC/7Xr4rUvqlYlyEyctbbIUdjMlflyttAGKqdfGLQPFgMAfxXAJTJYzpE2BPbKubxVSJEig2WhIqSWxdSnfNHYUE0F5eI/2HkhbsZikp2QirXWxTP2svzIr0sylcOg638ix+aDk8D5hBZKTFVnygpk+go5e/h+TXBCbZNVH2Y1gF6muRBBdbjavjFKASCnle9SCJmWZICcRJ7hcR9lc5SQ4utgqfGPB8XuxetjucZsZO/JD53EQt7/kMUKRXViYBQCYhHvdfZqCe+0ftXl6fQ6ryYg8y+ZQ9eVl6ln2LOYky946qNPh3TWZcE8PWJzCSqllybyWPtjfOI+UmSW3nsr5c5wRp/ZGphsW2TInO5ZajNy+myZXqCILyevJxZD9v3uecZlvsYPcmMHurJc01MaW5T9O2MlGn0uSHsMEkb6lcs+5cUeyZvHk5WTj7ZHNPra+O10uudZocSAAsJq+g6Zgkij4BfZmMS0AyyU9zn2qekNSjyF2LxMg9lri6WaqnSccoYS7J0zrGyU+Y5ZLZq52vfZb9Ss1Q6n5Mk0LSqcoxBbdIz5vJhFa+H/I1CcVxcu/xfeRLEsBkmoCU9x6TLFd5eZ8NS3WcUHPH90A+t0f2U7yQPvl9fSG/6zGmpFc5Fl8l3+VS5LLZ35kngJq7mNLDVfKTPuqbTBM8uX/V1YpqI5Mwtk8+1DyBi4D9fmWEWa5SH6lR06SOAHEYFjKZ87gTN1e75CUqUScUe5lUKfZhkhwD2taw8wAAIABJREFUk4c3s7D5MywnxkaTeyHTCtNniptpTBdFHqtIrKdIVEMhrGW0evJRKp/S10ud9jVOkmPtZXKl2MqFCqV4Ku1eJLw5Pbc4OMaZnSpXopWKFV/qr3lGpaNSthOsHGuecDC9yEplHx2x0FOViN2PE9CW7s0g66+EMZ3SYvWx5zeW+tjvGVM35hhE4jor0L2b0lQld0DAuACtKJMIEWE3BycJtEqlzlKDOQyQGfMiyWJTT6byAm6jThMD6WdAAGhZCLDLdSGZ7cwdliB1JfuWaI0E+0w3Tt5mAtvDKMAty17z+nUCrePI42XyWJalgNjMaObHE6hT1hIfv3byg8q+xHTOp25L73nc6zmB9/y87Nf8s5a/ApZ/6vJXwPIvYLn4zpP42//jf0RtRmrj+GJ3SmE8J2XL2/2K2jpOypa7bs5hLOhHy9msBaAyjrt2htEBpSKbQ82sGlnVHV/dn/Dm4o732yVaB0LQ4gNUkUNX4UZDWY1UhcMHzaLu8UGzbSusCYzOEIJiOevxQbFvK8rS0R4q6mZgGAzGRIrCEYJmGAxNPeLTz/NmYLNpKKr0YZ2AsTYerSPDocTW8obWOuKcYT7rabuCqpLtlNVIu5VkXDcalA4Eb6jqQaS9uxJdBJSOaBVxo0HbgNZh+tzyXhNGTVE73GiYzXvatsQ/lGRfl16OxKAoasewLyVp97zHD0b8jfcVsQzo2mFsYDwUmDp98AQIo8HUDt9ZlA0CjLcFFBGlI9EpdO1FOlx5mcwbtAQKBRkgTEDQRFRr0Gc9RIU/WAHrlYfOQBHk9yJ9cB6MBBRFJlkXCtnvwUBQEnqUpLhEJeCwCrL9Ln1Ql0FArI3TRABOTetWPgGhCKaV0KRibXCzMAHqMPfgJXTIL4TmNjt5LkUEp6bXqKCm/k+/9AK+tchBQ532wYtk1q28gGIbsXuNWwaiSs/Nkl8tQUVuHqdBn+lEZhuNAEwJGXq07jZV7nhh3l0GpUm+rJOky80DppWwo1CHSRmQw0aKrWK4COhOpcGNhC3pMfWIJhloHvCTZKrjeUAftMhmT8X3ZNpUqdCkbRqR7iovwMt2CldHFEhVy0wApsiOpcJFJ3Bl98fCeD9LnbLbrBxIoUBO1A6+RDpaK5Ht5m7SPJA2fQ6QUlPICaRBdXMEmr4UIG0G8VGKd1a2k315Uo8g103CeeS+jFZkxMOpePp0mjQhCvPuZnlfkjzZpa7StO1iK+oI7RTl/dHrmHtnx7mAYFFopP1I4NandH2TJKoC+NPfMmgNRyloXnJPKTAx2SCArb5NSoVUNWI7uYZZghtKJqDzuFs1VwxNzH7a12IjEtMMlnNwTFYMZHVDruzJ+/UYDOcQnXGhpu1lBtAMR7Ceq06yDFX6WdO9szsG5BRbkfYWu+M6i4Mw2Nnfq+KRwVWOr4XATJMH9tH+Bhnsu1pk3yD7LL7eR0xBTKE2deqpzN7nBMaE6WQK9ZL+0zi9P9xMTWqFHMZjxvi1yYncj5z92VnNorzIg12VGMlO6m3kOss5Mf3xGtlOgMckZ01gt9glhcYjL2auLcmKC0kqj5NPLgPBok3gPgXnZFCaWf4cxJNrMPI6cq9mnighZvl5nPbNJIY6KvGP+lJPvmoJkopHpr/Wk5S22HtcradrNnkzExCM6Rr6InssxZdoOpH0ap8muKwASuWlIsX24gXNPj9XS7hO1FJ1U6395D3V/ZFZ1n3ALQzF1k8MMAgwzX2kvpJJDbG0qNRvKkyofI8Eos2hbipJguU5ObDHzQx6CAkAHkGzGUOSYev0eAK06f0l1+0ogSWC7v0ELGPxCDiCSH+TBJwgfZTygEh7Y6ovUSFK/ycCJh8vKtWaACgX8PMS3Tt0OxILSejNDDAaCe6xmUEUuSxOvvenrkcXxNuZ1pu9j9GKWkBlL+QwCgua+ygLi+qGI5B8DDC1/hqzmh/HaNShO4LTGI9dmXBkS+EY3JPlufk1zomsNQPGyWebgPDjGpKQwoCyrPZnJLr+eWpGfunDe56+jt/6OQDLf/L3/wpY/n9eTr79LP6d//lv88X+lKvtIn/e4IOmMJ71esbJyQGlIqM3xKjoukKe4zT1bMA5jesKimYkeI2/rWDp5H1kItErimYUxtErcJr6tGPoE3s2c4RW/CbFfGDcVqjKE/vHVIGwjM2qo72aUV22jIMl7C1oMDOH743MLDoNOqIbR3Bafk/ME5mtrB2hN/JYWpRPbF+n4WSE+1JSWx8/xykBMSsnYCnPSjtFbAJmPhJuK2LjUTbC1kqyKKDbI3OnnvUC2nREbyw86VHvKvzKY7ZGwmLmHkYl6a6rEbWV8JY485i1wWe2zEgfaDQCDPXeEGYCGmIps61Z3hyqgBp1Aj8JFBXpOVGBBxXE9+ebMM3Q6kHh5xL44pceszOEQl5v96mT60QCeiZwNw/TdlUOpBkU/tRh7y2hkFlm5RMrc+qFhXQCyoanbvJt+iYkRiozIsdgCoKS+pcotSuhlAAZgiLWnuLeSmJqtjsMAojMQVhRu5NBzrjM8s0kiz4J07GHMlLeG9wiCPv5SHJoEis5LsPEDtt9Yld6AWv9kzCBvqgFyBR7AWkA7lRSbu1e0T0LFGstoUAR7EEGVeVGJYmysJKmU8QCxkWg2GlhSF0aSB5SD2CTOjFrCUbKabbCmqSk13R/2oOie+Gp30m8v0veTOksFDY3B+pELVUx5qApH1IVRQIVUUPMqaOnEsxjOmET+/M4+VojTNdaeWEniRLsk18vkjfZF2GHhL2MNknwbUr8LeWeMJ34UHNqbtRMntDcd2n3amJP87XOYEonOTYwsU2ZTcqJqMon0DU/MmoyABQp8ziH3L06zkV+Pi4EOKHkuDK4c408L3sefSnP00O6TqlPtlyn2ozheJ7zOXd1kuumCQOf61ouVAqwSQE9B9leuZZ91yNTAEtOqS230J9IqqtvxCfp6mN/qbx/5Gc3k3UNK9neuFBkr2gGF1mCPbG0Wv7X7hHQSe+lnLqaU49tK/fDuFCTxNoeOAbyHMT/Zvdx6q/MwT31nci07SENmBH2Kg+sM3smxyz7btsUwJP8rSZ5Dotd/JrkHKSD1M2UsI4JcNsujQMSc4Y+Mqs5IMunPlNfKur7OH2O+EquT7CZgVYUB5moASbA6JPfD4QdnBJx93G6n3MoU7YXiOz5yA5mX2WWqReHR+Avpa0GIyB3AqLxCNIzux+snEfbxel9FpWswzUCVnL3pvJy/syQJqK6I5DpTiV0SN4Hkraagd9jZs8MX2fMQFjLHC5k+gQghzCF8UQDw1xTrQVUDQtDcQjiY6zScfapTzf5KfP6Mwto0nXtTw3lVjyS9hAY5+LbJMapyiRa6fM06fgysA2FQvfi4zStTyqGHOiVGOG9n1jN6VgfTwQkqXOWw2dQ6SuRGY8zS7FzjCvxaebE2ahEPgwIYExJsqYTKXK+Dib5JEMGcRqUi2myIb2mF1mwSmAolAIip45HDWY/yjV7JP/Nx6I7AY9onSTHybuZeoLVmLot9REgT5LcIqvSongxxxQKlJ+XA3syS2m0/JyBXWb6HvkvVfJTToDykcxXZMtagoGyzzJvIz8HZBveH4Hjz0h//enHJszgPSjptHzMXMZhnECnMpo4PGY1HzGWP/37n9Jv+dNLjJHY9/9Cz/1Zyy8MsPwPfg7A8r/7xQCW+s9+yi/u0jvL716/4N1myW5XE6Pib7z8MR+cPrBez7g43zGvBqrCsbubcbnYU1Ujy0XLYtXinObiRCIFx+uGp+cbopGwmHI2olRkcdqiFNhCpJ+q13QPtbB0q56idOAVs9MW1xWUq564t5SrnmLZY+cpGbTXVIVDpaCZMBj5tArgNwJs9cZiHwxm7iYW0DxYdCueErO2mJkjPpTgNHY1SDCNFrZIHaRGZLboQUXK5cDzN7eYvUg6iwctrNFsEIDohL2Lpby2+KOZ1HAAqEh9ZajfWpovrDAgRliLMGqKGwu9Jp4PhFHL40Vg/oUm1kGYw718mOrUkakd4BTNlczE6dMBuxbWUPdyjMVOUb+zAvq2mpN/aggLz/KHGhWUDFJXjmIr26nfWcxBCxj2CnNQlA/qCDi9SjUXSoJ49gY1CsCu34mss7qXx8sry/KPLKZTNJ9bqmtDLAOhDhO4UJ10ToaZp1hrqYgYoP5KPKbFVgsD5UUOWWwUce6pr/QUTBMSIDMHqbvQvaZ8scelBFeCYvaV5vwfizdi9UON8opiJ0FBelCYQVFsNMVOAJgemSSi4vmVXs2oob4ymE6Cddws4hthoWZvE1AMsi+hiJT3sq3qXgJ8QsnEJpb3ArL8qUONAu5klllNDGM0yXfcSy1LsVYTw3f+B0GON4XF6D5NGiAD/iJ1moYiUm6FNTStYvZWej+l7zQBK0eSzcLsrYTe6FYCbLSH5kr+RoTqThgQnaSv5QZJnE0/205kneVWBs52JzUWepDjqm8T2NvI+qs7NSXNhjIB270kwxZ7qb2wnfiETQf1tfjZTA863Re2FaBpW7l+JoUnicw57e8o52j1o0B9wwQoZUJDthGNgLzqXga9xe4oWxVvdJzCg+ob8UrXd5FQiF+3uieFzxxlqPYgYCvLSrNMNYOyYivdouU2bcvJfpJAY5Y4ayfnKiooHyLFPmIPkn5p20h9G1l8GaeKjeYmiLzWZCAmj5VbYRyzf7DYxik51vSRciO+4GInLEG5jcy/DJJa2so2TSvrsompKnYChJqrxPCmBNfqIU7y1MyuZqAGArKDEcCWPZFZJlqto/go+0ixk21FI9dJmFIJuCn2kflVkDTQmPYbCSAq1wJI67tIuY0Uu8R6DnGqnrFtpDjEKRlU/jFJRJWPzK795MnVHppbmejJAS/KSa9pNFDfy76YDqptZPHOY1o5Bkn+ZGIri106pwl8qJhAeJD9sN1ROltu0ufcEKnWnmqTgluGNAnTxlR5kb7QE1AtdwE9RgnPGSWkSDthGbXPHug41YlIEi8UWz/JKYtdEKYzhdpMrG+IidUM4t3dBcq97MDysw5fiV+3StvOrGkGgNpBsQ8Uu0D14ASYKmEbkTk3qrUM1F2Vtx0ptl6krVZRPjjKBzdNRghDHCnXLlXqBIqto9h4OeYxUOwctgvYvccmoKW9PFY+OEwXsIcg/ZlpQkCyF/x0zHpMgUULTX07pkklhd3JOoNVk6dYhUj54GSSYOuFZRyCeER9Ov9bn8CXyFO1C5SbUeo8Co2biwy1vD2k4xtwtZHnjgKW7cEnICnANktvtQ8ToEQpzH4UGaoisYtaWMWI+FODMH96kH3Sj2SwkkLrU0WYJLgqH8RTqgSkmnYU0BfT++puLz/34iENjT36NAE1enSXPIVGJQ8s6G4QljL5UVXXSyqrz9fEC2MZgvRPjinUJwE3lUFY/hvCWCofpiqRDCizf/NrQDEvqXf0sZ8xloWwlo8SXicf5mNv5fSCBCR9IIYgHktrpYbk8eOTzDWxl4mVVEqlv4VpG18jsvLfQzjKYlPdyc/858PXA4H+avmFXcz3v//9f9n78Ode/qv/4b/8/r/xH79m70s8iifLPf/4Bx9xcbLn8mTHF3/0jOfPH3g+3/Ll1TnffHHFQzej7Ura+xlF4/BRGKt/69d/nz+4eo6LmuK6wFwMfOvFFdebJcOhYLHsGB5q5q+2fOeDd/TKsL9aQBl58eKe+/sF85OO9rbh9MWWphrZ72rsT2qef+ea7bbh9HxPrzV25ojAb37rJ7zfL/mVT97y/stz6hd7zl+v6fqC2aJnNe/Y7RvK5wfCtuT1d98zovnGh1fcdY1IoZxmcXEgvG2oP9oSV47hiwXlywO//vItP3z3lDj3hKAJRr6An7xcs4sFeu6Zn7TY2cj5sy3+cqTvC5588MCvvXzLj/0Kd+aIUWPf7PHOEE48n75+z931ir/11/+A634OOhJ2Jc3zPeHDHkwkbgqaN1vOn2/YHyqRiS08v/3dH/KTck592jM81OinPfaLiubba8IXM8yoKH7jnt5bwjzQv3aYe0v7xrF4vqM1lvl5i3uoOP/knm1peP7JLYexoDzrUecD41CiBw1Pe5ERz4O8Nlpmr/b0ShOLyLPvXbN5mNM/8TTPDnA+0pYWPwu4k4BbBn79Vz5nHUtGNOFy5OLlhv2uwZyOjMtIOB+Jo+Hkr90ylJFhHqm/ueVvffNP+NHbpyy+d0c/FOg3LZ2y8Lrj+at7XB2ZvdpzOFV849O3XF2dAgpzKsx5sIo3//ZPeO9mRGfRn+4wzzs+eHPL89d3vDc19lWL35QUv7qhi5b4qhN5rlboVwfst/d0Y4H6oMN5y+x794wlhNYyPhsZXo94I97R8LonDgbzzT19sIQ3Pd0qfYF80EHjGWbClhYnvQRidQb/omfxZE8bSqJWPP+1K/ZXcy6+d0Pz/MBuM2N4Ksk3298YGBuFfzpiNobulWf18Zr+riY0UeSf39yhngyodxXLf/OKTdegvrujrTTu+QidYTz12L1muAwsv3NPfzNjXIJ71TPMZPDmazi8kkj7/oWX8vhvrRnWNf0ZvP7Nr7g7zAhW0X7oGF447NriFpH+eZD011cH+hrGxtBcKdzf3OB2JcN3WhYfbWhrhb0tGBeR6nsPHM6gWyr0d3bwrmL3JuJm4L63ZywVwxPP8hsPxIuRw4lCvepQtyXjJx3x2Qg7i/+gI5w5OBS4hbA+618JDB8PnH28ZjvTOC2+2/1HjhgVsVD4maJ97aSHdRnpXnjcmaP71kjfiGby8JFn+HCEztJfRIbTSPdmhNctw1mgvLHsPgx0H44ELPVv3dH8o5r73xop7g3dk4ivof1kpF9K+u9wJuFUfg5nv33FZjfH9Ir+XNjK7tOBEAyHjx12a+jPFP0FHN6IL3r97Uh/IV2fw0rRvgzM3ik234ioqPGNYvvdge4Th38xEtuC/SthsPcfBqp7zfZj6L/Twb4Qv6tTbD8RQLb/QFjD3aceV2raZzIQ3H86UjwYNp9KHczmE+gvI+2/0qE2Bd1TYWp2n3iGVQpYKqB/ErCdpn2qGM4gVIrDC/Ht9WcSSDScCnPkS9nH7qVnPI3oTnN4AbvvDuhtwe5DSa+9/Zsj8x8b7n5d2BM9wvDvPtBvZriFor9QtM+F5eueig9u92HEzcH0OgEWcHNF+xSGc9CDZvchgKI7V+xfaA4vhR1obiLtM83mU2GTb//VKPdqLUnN/anm4btyzP2FsOvaQftEsX8Nw6kkGA8niu1H0J2L53jzDU33hFRhEVl/Kj69YSWIsb0UlUx3odl/ECFo1r8mfkVQdJeK/lKk5v2ZhECFQrF9o+lP5dwKS6roTzTrb6UJnY0EjGEUuw/M1P2JlmNpnyV1DsJkuVqxf6XpLqUzsX2iKTeSBNtdaHYvNaZXHF5o+jNF+wJ2H4oqYv2pQjmpEhlWhsNzWbcvtTB+M832jTBE2sH2jUkgQ2OcJN6Oc0t3btl8QzN/F9i9tBivuP+WpdxJKu/2TcG4NFz/a1BsDYdnlu7UgBJGbpzLfkej2Xwsfrlhabj5DZ1k+XK+zSg/r79h8IWmuzDMrj3336pQQWSwt9+rKPaK/QtD+8QwzhVRKe6/Lbr2cWE4PDO42mCcTCZuPirwjQalaS8MvpLAp82bmt1r+S5oLzUYzXBeEQstyoQEdHavSswI62+UzN6PrL9Zo6Ni/9SCEeZ/+2EFWjymflGwfV1R7AOHFxXaQ39ZYrrI3a/WzG48oTYcXtb4maE/KyjXI7HQ+NriG0N5tcMvK9xJJSFGWoNRtM9quqcVoTbYFCTWvVwkAJUnPj2hsoR5IUFM1nD4cEF11xMry3jeiPXjrAGl8MsSULiLGUob3FlDaAoUCn9SCeA9DERrCUv5PVZWmMx5TawKAZPOE5YNRHBPV5hNSziZS3DQoiYWBWHVoNAin21KAYBGi1TWB6kdyR2XSkl4UE6OVQrKUl5blShjhOG0Fgq5r1QIsFqI9zMEKAqUlteqwoqSok9eTK1QVSXbR02MMiGg8vqVYgoYeuzf1BqFANKf+U+n9aGYAoT+HMsP+f233//+9/+nP/cK/gKW//zv/Tffv/zuv06+w/6i/r37P/73f+nHBvxyS2GffPci/p1/+LdpfUFIBoBn1QaP5qpbMgTDb558zj/8w7/O6aLF6kAEtl1FCJpl01HogFaR/VDSFCOF8Xy0uOOP10/46uaUj5/f8Ha9wnvNrO75xtktP7i7BOCj0zt+cHeJUZG2F63VvOmJUbHezHhyvsXowLarOJ+1XG/nLJue8+bAH/zxKygCL1/c89WPpeajbgbafSVdlbuSb334nt1YcrNeUFcj+0MlN5AOzJsBrQOHrqJvCy4vtjxsZrx+cs8P/+QZ2IjqDOasZzHvGL2hPZRcnO0ojOfd9QlhlOqSIvkwjQ18+PSOH372lPlpS1OOrLcNLy/WfHFziu8si7MD+y+XxLlD3xeYlwfGTUV91tFtKpYXe3abhrOzHfe3S6r5wPDlHC56TOFxXcHy9MDm3RJ0BKfQy5HghCX2zhCvKrjsCYMRefG+xDSO2UzkD7tNg7YBvy3QMwc3FWEuPZQEuPj4nrt7qYKpT3q6h1pqTp4MExM9HJJmcG9Rq0HOxaCpL1q62wZ0xC5G1I8bxguHchp70RI/m+OeDKhNQVwI6PWLJCM1Mqi1lx3+XQOXPa+f3fOTHzxFzR3sCliM6KtKvJGDglORLYunMYCBaEXyWz/f024r6IxIfE8H9Ge1ePheDigdaRY9h6u5+C5twGzstG7lFKEOUAbUwVC/3NN/vpDP5aXj/OmG+x+eC+M98+itpdiI5zFWkpaHiqitxXQadyLsvPLyAR+LgG4Nix9pdh+JXNbuFcMHA82fVAynQcKJFiNcV9iXB+KfzBmfjejU5Vm/s/QXHrMXH6hKvll36ihubfJYJR9l8n1qJ0yhcsKGjqtIqIMEG1lwT0bMbYFfOfTBTH7DLGnLASZRQ5gFins9Fc5L2nEEJanS5Z3BLUSu3VwrDi8Ddqsmeee4EildsVN0l9I9V66FGdZv9lLto6D+StKpqzs9BdxEA91zR3VlKHaK3acj9tZKgNMiUj5IVQzJX2c69bXglewpHFdy/1V3iv0nI9W7AjcXqXmxSf5XDbP3Ejg0LiPNe0X7PCaGJAUaJd+rPYgkV/o4U5fkALmPrbsQxrPcHENxpCNSZKrHWpocGCRA0CWmPPdHuhn0l57mrcE1yTtowS0Dix8ZSXOuoLyH9nmcGHQ3kwCnYicgJ1fKuBlTmm5md4dVkurao2TX9FJpooIwtt2FsLlufrwHzCPmLadI65EkEU2SYS+/2wMcXkTKe5nQyMFGOUjJHqT/UgVJyz75AfSnx4Ad2x2lyrkvVCel13Ai7HlOnJaUWEmejkoqctqncq3KjXgmc/px+SDg2DciYy32TJ5SXwqj7qvkPbQCGLoLWP44TkFVto30JwKamysJENKjSHVnV0c5bE5Rzsxu9k4enmmRkZ8JC276YwBOf6JpbiRdur4LE2PZneuvdTDmfcsdmjl1Pfs/gYmRLLfCpI5zlVhReZ7yTJJxX4v8Nqcu52TxqCSRWju5Ps1tqvgo0kRBAra+yn5CYW7z30MpLHB3qphfedZvLPN3IUmEJWzJJEJqbLLcNf2vkhe6Vonxl2tX7sLkjxVPpppShsutTGjkKh85PsXupWb1mZ/kvkViMocTm7yiSfHQShJyuXZ0F5b6zuFrPSUY557KHBbkK+ndzLJf5SPdmaFaB4q9YzixU5JvkcKLJvmzAntI5zOl+ioXKR8GhvNSpLhaEoDHnDqc/J5HOa4kG5f3g0jUTwpMJ/sxLkRKG7U6+iMrw+TH9MJojquCYu/S9TRTz2X2uQ4npch1E7tvUpKuGr0E+gxHFlV3I6G0cm1SNQm52/YwgpUgI9V7Ym1RncNdNBRXO2JViHw2Jc5m+WxUCuX9JMOfpLI8ksOn1021IZlB1EqSeLU61pcokfXiU3LtrIZ+EN9mTn0F+Tkd1ySZtclb6cOR2YRjymxK9o3DyNdqQnyQ31N9SMyS3okB/TNA4Z+WEPt4nb/s4T1PX8dv//t/8VLY/+u//8WQwv7ceiz//1g6V3DbzwlRcdMuWJQ9D0ND7y2jN5zXe/7Xz/4aF6s93WjZ9yWllRty2XTs+xKrA005cvcwx5jA+cmeP3x4Sjdaqnrg3UY6PNxo2LqG/7t7SbcrmZ90/N7bF9TVyG5fc7o6cPcwpzOBvispSsfdes6rywcOe4lzPNzOiOeK9b6hPusYOsv1wwJsYDbv2W9q6rmAPILiR1cXaBPQOrJ9mGFKjzYB7wwPtwuW53u8VygVWe8aQtB8cXuKWcq3lw/yWEje0qJ03K/n+FFjS0/VjGgd6bqC6DTOKz6/OQWvOOwqWlWiTORqsyCMmnI+0B4qWI0UpSNWnhhkxrbflygb2F7L8bR9SRw0XV9Tv9rjRkPw0tW5uZnLINVGzImbPpO0jrgA6llHGDVqZ3EmoktPWTq2V4sprVWtBjCRsnZ0TSHgqAzgFXcPc0JnxSfX2eRh1SgFw30tYT4+GThsIO4LKAJmNTCOBlV7bD0yHkpMGVGD+FbHhxpjIvQikcUpCdexgYjIpGMV8G8bwiygouLL//MFahnEcztz0Bnxk2qRqcS9xTgIKdwlzJ2Arrmn21XY6xJ36mTd7ysBVF5RvCvxdeTQilRZPIMC1lQCf2HlUHtLeW3pnzncHy9RRSQ0cg7vvjwV365C0nId9K9Eum0e5KMhpO5JqfaQDjXxxILykoDbvpAvgtCk4+wFjMWcvHtVySB/W6KWgeJagI8+aIIVGXdIPlKVZLluJR7SaKBfBAHhcAxSCWryCIaTEb0uxF+oIipJsFVnJsAfdQr3SdLV8kHTX8g1zGFF40kgkEKDGqiu7dH7WEaGJWSPZ/b+9hkjAAAgAElEQVQPZt+ir6SPVHxuIi/2n88ko8HJsRQ7nVJUxSvpG6a04f4sYtYG02fpoYDnmMKFin0COKNIZ3MgTkT2SQVJmS3uJUSqutWSjnpUtrN/mXysRgCZadUUiqMRMGl2agJnMoAG6yVMKGomOXcOEQrJV+iatD8KYgEqeRIlvEVNIToh1alI/Ulk/pkRP++QwOxagUoJzFa24eaK+RdSxTQuJG3WV0w1MMNKukD9ubw3gKn6qdyIj7PcHANlxln2OyYvpJKE1gxEMjDzpfgw9SB1TbmeY1gen5fPf3kvEw3FnglkiAc4gcWUlLr4TI4DBbP3yVeZJZT3x2TaItXQlGumgbl2YPYyyM4hYcGIhFxFUpVGGm/OSSBQwK0uBOAGm6TYPklIk2zZHgQg1dfHOiWVngOk9OPs3xSw25+ITNzNFd4yTTyEFE7kZtLHOSzUlBDsKzkfOdXWVwKM2kuNPYjkfEr8TUnF1V6SX7NX1ZcK60T+PC6YpPG+Fh/pYz+vL5mCjcZFGqhHpjAe8VuSqrfiNPWfZdBjo4+hUEbkuKFUk4xZOwF7kCYcGkW1FjA6fyeyUu0h+qOP09XiaRV/nwCKyYM4SrCRHqQSJRRKwFV6rfJHkCqTAmkflEL7SNEHTn8QppomE+QYlJPnqRiJUVJjXSP1T+NSJKNuZibZ7DiXgJ8hPVau3eTdlM8nqfEp9/L94RpDMIrmZsA1dqqwMZ3clygIKdFWZNNBXlMZolZEK0qTodSYLhxrZ1yWM4MZJeAnWk0otQBNq7CDgMvs1xRPdQog8hISpCNQSK1QTneV6peUBjuIZNh0UhljDimMJ9UgYbX0WiqmsB2R46bk2BDE0xgiKqpjbUoClVkKa/YjoS6FlR2TZNR7KbnWJNZPi/QUOVY1Skqv3nWEWS2gMAfi5GAdkM/NDAazJzKDTqXEi9n2RzksMCW5PpbCTiE9KYlWKaZakfx4jMQQUmLsEfAR4yPQl2TDSsnnZDom9XhdHIHmFAb0uI4kLyHKNtJx/qVYfnk5vT9z+aUGlsNg+cPrZzin8T8SJsa0wm4UO8Vnq0DzlWFvoP1wpP6iQO2gexoZU1CG3St2s8j8WgYrbTdnnMmXX7mVQQQKml0aUBZwege+rjAzee/VAwzjjFkBtmvQTRpEWbjp5lQljEXDYg8qFuzeeGZfGU4eJLhhtYl0F6fMPURTUwVY3UXcvKR9FqluFPGUVF6eZrdLUL9bUjQwX6cZ15Q8t7iNSfYa8XXBOJ9TBfHHtE8js7tjP5/dwdLnL3uItqLZQVTFsd7AwarLX9xSYn54Ebn8f8RHcnguA8fyIdKfS9rhcFKxCLD8LLD+ZEntmaL23UwGJP2JlRj4VBY+rCrmvQw+yo08t7usUnBKzWIjA6XZu8Dug1kqKy9Z3URUkJlWCcywjEvxT7VPimnfxkUlvZKlwTePghnSG9w3pZS030MoKmoPsysJ3tk/rzFDZPHWc/+ppblOBev3ErceSkW5jriZSfUEmsOzGruP2FZzeGGEvejEu3R4VlBs45ToWD9IcfY4L4SxmxWc/mDk8BTapwXVbQJEuyj7+BDoV5rhxHLxewPbDwqqbaRf6XTvRrrzkvou0J1Cc2VZvBXf0P6ZSQNndSy1b0iDwIJqIyyFbSVVcHbt6U8VepTXFTsZTIVCwjOyVygaTbnz7J4bqk0K0rBqqhy4+/WC5Y/Ee9WfSAl7cxtE/oR4z2wXqe9Hbr9bMn8bGFbCvLqZVAWYMTLM9ZQoWm09D59UlGvxXOUuRgmtkdl9ifHPnxpy79cPnsMTTSgVpz8YGRea7kRjxuzZUiw/Gzk8tQLotLAcYyP3erUVH82wlC5KX6TZ5xSoIUmRIvtbfi4epXGmqe6dpC5GGWjtnhtmNyK/6pdKfGVOzo/pYwIrgXLt8JXGzUQimMNEolYMi2NgS/tEU2wjy8973NwwzuRLOIfXlNtAd2qY3YySGJl8aPbg6S4sxEh9J+xDfTuw/qikufOpX1TCNqJKdQn7MLEk2kWadx3bj0S2ZTvxeg0rPYG3YS6yKjNE2Y9zI92QiUkeFpJsGQzUd2PaHyhaj6s05384MqwsxdbRn9nUpVmBgup+ZPFWBsb9Sv43Q2D2bqC7LI/szMbhZhI8ohyUDwN6aJi/G9L9IimetvW4uRF2pBD5YrF1AkZKPdU0SNWKhJIMK5N8bYFQKsaZZvF5j+kcw1mFcnHqNuwuCpqbgfZJSfN+YDgtpvXvXpY0N/KzBD1FhqUkcTZfbmlfL9F9wPSB7ol03AQLs7c9blFgOk/7pKC+dYxLMx1z+SCF9eNCBnZ2L2mfpg+oELH7kfZZg68Us3e9eN5aCUnpnlbYvQwA7cEzLsUHX932hEpkkihFsZVthEJSLk3ncYtiSs/VCURVtz2+sRT3HYRA/3SO3UuaZPusorob6c8Lio0TL+EgHjx8xNcGuxdfXH/ZSGerVrjGUOwcysvncNQK23l05/GNxfRewEypKdejdFFW0v/p5sI8lfcDbm5FeXCzT8ySvHfcsqT67J7u43Ppkuz9VNHh66SuOLgEXD3jqpI+SBcY03UxncM3UmWhB894VmNaRygMdtvjZ4WAGqvRhwF8YHyyENA5ONwydZiue3Q34Jc1ZttRLWpUO6L3Lf5cnh+tnrZlth1hUaP3PX5RoRNw8osSu+nw8xJ7vYWykPvOGPSupT6ZSeKpUmA1xbKCGCmutoR5jZ8XFHcHiFH25X5PAVJnMUhwTJhJX6Rb1diHDhUCat/iL1eYux3m2Qlm8yipdHSEkxnmZkOsS7kGRqHXB5F0tr2AucISjUENI2ZXg1Hoh71IPTPrNowURfJHZhYwV4YoNT1H7WTdejGDGFHdIEDs0BETmMuATO1b+Vv2OqYk1thUwvL5IOvLvZJlASESux69nEM/HPfxcVgPHNm6EKbQnbhv0VVJ7Ae099APROemLss4jsefQxB5awaWVSnPTymukk7riUE8kaoqBQh+zfuYwN1PeTRzWM/EBCh9DNFROjGYCsbxuD1jplTYMIwiZ/3pJQHF4N0/Azqn5TEj+pdkUX+JgeUvtRT21a+exn/wv73mx8MTfnf3AZV2/NP1M76xumFpO/7J/St6b/nWyRXv2xUfzB74ndtXHPqSZd3zjZMbfrS54LLZUWrPw9BgdeAn92f0g+U3X33BH98/YVV3bLqam69O+OST95xVB97tV2y6ihgVdeHoneHlasOPb86Z1QOLauB6swDg5dmatw8rLpd7xqCJUZhEoyJ3uxlniwN32zll4ThpOt4/LLk82bFpa5wz1OVIiIpF3bNpa77z5D1f7k64WS9QKrKad1x/dcrick+MiqoYeXiY8+sffsXvffmCp+cbtl2F9xrvNUXhpWLEa0rrOXQls3rgpOn4/OqMJ+fS5blrKwrrabsC7wx1M/DxxR3vd0uMDjyd7/j9L5/z8bNbfvCD51y+WnPeHPjR9TnjoaRe9vyN1z/GR8U/+vwNQ1ewWLW0h4rV8sDd1UrkmPcLlquW3a4m7Apef3zNV7cnAASniV7x8sU9D/uGw31DfdITI5SlY7+rMdZP+3exOPDFHz2FpUPZVEkCzBY97b4UOW5v0UXgex98ye/83sfoxUjVjAyDZTbr2V4tOH+xZrOdobTQMN7JdTs723H3xSlo8RoCuJuGszf3FCawbSuGvpDju1liyoCxHqWkFgbAWuk37buSMGqaZcfhei6sae2xhSMEhR8NobOoTmMuemzh6dYVqgxwX3Ly0QPbPz4lXI7EXnx0gHxilVIlY4owve7lB3d89cNLSb9VCFvZSrKeXoxH6fF9TX0pkmCz1/jzkWo+4EaLbw2qNxKW1Br06UBROrrrRqSpz1rclzOKD/YEr3G9hZ2waHoxYgvPcFdjDhq/8hTLnvDlTGSAZYDaw6gpbi3+dUfYFKi5wxRBEpsBc9Gjf9QwngTsRYe7Euly83JH99lSGCArUlY9aMyzFv+uIV4MqLskgb7sCb2hfFcwXDooIuXbAjeL8KQn3lZwOlDPB/rPFuhB4U79dD7oDZSB4r0Ur4fXHcErqcIpAvq6FPZlTJUwo4RIhUr2iyKgekNxp8WDqsA+WNxCQqj0oI6Mr474uTDjAHojYVp+IfJg08l1H08C1bWR1FrFVH8TSqma8U0kNJ7yyuKWIicNrzq4rqaqGIUkHFfvLcOZp3lnOHzgKO9FZhwKYQVVzHLkzAQj7L2G8lYznMik3Xjm0a0EQtmDhEWl5p6vJwNvRB7XvXCs/qll+7Gnfm+IhUhf/TxIONNB0T/1VFcZHCmGk4hbBso7g6+E9ewvAvWNpruQROBxKcxKfS1Mbu4T9c1RwuqamJKWJSRKjZL2K2BRUnx9Had9dY0kDwtLF1ONiZrYLjNAfyqTR9EcWU2RESvcHIoNjCsJfOqeiFRYj3B4KQnGRJED60Fk17nOo3/iKe4N9a1MkGUJ7bhg6hXNKba+YeqpFNm2TDR0lyKhdXOpuqnuZd/dPIVIjUgNSwqLcgsBsMHKxF+umwmlPE/Oify9eoDuUoKeopbJpGF5rGkZl3LMvk4TiWs5ToD6WljQkEC1m6upyqbYxamCqH2maN7HqfdVJLeKcQGLz2Nad5KD7o4JtNEIeynVLCmYaGBKBg5GJK3tpZpqZEIhzK+vobk59ncenilm73PNR5IEpwlZ0yfGNKkETHf8PZSybyZVnJQ7OT/lNia5bbIENIr6PjOQ6fodZAIrs7wqHFN3QSZks6y82MvE2pjqYGS/IuNMUxwktGn/3FCmAKkhJQzL94gw+uNcS+KtVRQpHGhYpO+NMU16HCTMx80ly8EmNjEYYVK1SyxswgfdmaHaSD9nfefoT6XjMk925QlBkIm1mJJ8TRdwcyMhQ1pY2HFuhFGNEV8ZqdAZI0XyWIZCQ2IjfS2fG7lqRLmAr03qkhQprh4C40L2Jz9H3tuPGEDAV4biviMm1lVYQak0iSlVVqXOSl8da0x8bSXF1icGOUleozkmyyovIUPChCawpUEdeuKsEtY0SWijSR2Zj9Jos2w2s45Ra1Q/iOfSmgl4Z8Cck3NViF8HmY/WwaMqE/nCGUUW6/0xFdYYAdM5xMcYka6OTl6XU2AfJ8D+rECff87ylyUV9lf+vb94Kezv/INfDCnsLzWwrD76IL78L/4zjAmEL5skAQQ9Kvqnjvq9pXvuKG8MwxNH9a6YmItQSP+cFJTLQEPSFdUkdXNNpLnKlQUykMpfHno89mrpUb60hxMZKPRnMuCw6Ut5nMv6QimDllwiDXytBw3EE5S/tHV/7FTrz2XQkaPzTU9iyNRU4OxmKaUxyBf6cCr/j4skC6qhvpKBjHipRGria4nEn9IpFVP8e30jzy82cnzZd5QL4Zc/iWzfiGQrGjg8jyw+V3QXchz9uaSUFjumHqzsh8rVATm1MnfSubmkTo4L8YTZVs5/+9Jz+nuacaGmL8zsmSu2Ed+oSfo0DbZSHUGOtg9lkkml/c/R9pIYJ9up75InKvlychfa44j6HEVvhpjYi0f1A2kQIQySsLPj8lhuPqyONQA+yahy151tI4fncp3788jsS5E95S8fPZCK1eX8VPe5kkMYmOIQ6c6yDEj2U5gt8Q3VN3HqFuxPVUoIBUK6R62iuQkcnmlZdzonw0pNLOHsnQxsYvIyhUI8X74+XsvuUqF78VY1NzFJ2B6dTy8SRV8r6lthJvN7SWRPkf5UBk5uJo+V6yj3ciB5pOTvLhXVDyeKxVeeYaGPSZOkwVIXE+smLFt3KkxaTrR0lQwibR/pl0li5kU2qR3C4i5T7YE9Vj5kOdw4Pw5UMzPjKoUZ5cs+10DYPrOe6Xp3qeoivS+y/9L0GahIgmV3IqEa2kXqexkU+EJ8WXqUdY3pPNk+Tj7MLCfMA9WooV4HfKGmovicsgnCUKOgWodp4Lt/piXxdIy051o8jbk7juP6c8G99jJw1aO8ZkySu+Ig28kl8tXGM85lUJMHo/3SMLsa6U8txSEkYKCmYvjiEKZ1y+enDIbGmaZ+8NM9MSw01Ub6+VTInraUmhqF9S4OwnCXO2GMM1CSGp6A6QL9mT3WLySFg/gcj94xUUrEaR2mj9LnlxjKzGpn+SWAaYPUPCRJYrGXCggUlA+O/ryYBvOTJ1crqruR9omwE2aI2IOA+SwXFHY+effGiN17hhOL6QK+1tjD0SOWmWY30yndNia2KqXgRqmYkO9UYeaCVYRS9kOeq6bkVekBlPNi955xmVLBh5i+fxTV/Zj6Ew1mCPSnFttKCqqbCUOjQkyDc0ALOLGdP6aJ5hqPeDwv0aqp69BXBnNw+JmdvG9RqQkguHkSaoU4MYzDaSms7eG4nXFl02dzViLI/WX2I+NpRXnf0z5rqO6H6ftDH0b8qhSGT4HuhMVxjTkCFR9wi0Kkqp2XlNIQcYtySlQlSZtDZShvDgwXMwEjiQ2NOsktoySqBqtRPkgFB1K/ESqL7t3EFupR6jZMOxLqArMWUOQXJWYvCam+ElloruDQLkzHNo13ulESTTl+b+ICuh/xqxrdOXCBsCjlZxCwNBzBlLCXB/5f9t4kxtI1PRN6vumfzhhzREbkzcx76w7lKpdx27RlusUCxAaJwY1gxQIJiQ2bBu/Y9hYhEGwwk2DDCqmRegGyBAJkuW23y5bpmm7dMTMjMiIzhjP+4zeweL7/P3ndVWW3uqz2bXw2OZzpn8453/M+kx9nEC23VW0a+CLhNdB0QwqrqDuCI6MAFxCMglyX8JOCXsNA64GIyaQhTwaZ6VeASmcRspRJrT5wH4RASKjUgIuprFoB1hF0GQ1hHcGYEBBNO9R7hDTZJaD21R+JIeP5p4FUvPVyWTiHkJidrNX5ge38ynPeXp8bTU9ln8KapQibLcGbcxBpOgAukXJKE6yFiJ7I0Has/+hf01pAKqBrf7KvMU3xlZCcPoXW6H/08c6RpdQaoaq+ss8AIrAkC/qVbQB2r9WzpD/rFpnLr73H8ugvCFj+1385gOXXWgorVMDB3gZvbmbA2ENkDqFSyA4riMqgfocXof7mFu6LCU5//QrPf3wyhKzACdhD1nzguEFbaXR7EuZBopt5BBPQ7SNO3OmVsmNPtmajETLHgJxKoHy/AzqB5kAiJB5dy4Q90Ymh3qE9tOgm/aRMDAEXdr+DWmqGrZiAdj/QGxWDXkQAPVt5QHvgITrWapSnDEPwpvdvUd9fX7RQSw23Z4FGwiw4XTRLgfIsDCXmchoXniHAa7Ia3gD1eQdhJZI7ieaAP+D1Mafa0gL1I4vkVqM7b1GfKMhWQnZMiZQOqGJ6ZNBAe2SRPzdo9jgdrw/JINRnjsciBpO4lAv4/Jol7904hli0GPrQ9JZJfvWxQ3Gp0BwEmKVAc+hZU6EJmpIlF0ftjD84yUpg+SHfvxt7Fs43Au2BQ/FSsRz+uGd2KO3tgUNz4BGSgOyK8rluygCWbhIYdKNZN1Kd0fDWh7Ns3nXIXyruy7T37Ai0U/YjCsfAF70FqhNuk1kBTeD+pw8MXnEZhxj1AQcEZi3ItlT0DqpGYPWeR/FKsvB+xfdn6TuDbWQjKOuOx5XMERcEzR4BUTeNrMQowBuJ+qgHs5Q9D96aWmDxIeWpZkvmoTl06K4ViuuA1btAfsPzHCT9X9URF5vJgttlM7ImdhT7N1smV+oKaM84vMluJQvbCwIoDmgoL+1GXNB0E4H6KGD6KUEl31MN/qw+DIWMCO+nN4fnO0hgdAmmJ7b0uNX7XNz7BKj3A9KFGPx41bFAdgeUj8iAJQtJD5wVaGf8TKkYxJI+cGHpMqDZj/7BB4FuiljNwmOS3gPbczIy0pK1ko1AfrOTGm8f0bPUxk5IO1JD9QeAIYik3hfI3wRUB5IsVMpj3g+KggAgAWn5Oeo9qvBkrYpXZNdcKuCVxPZCYPSSQxbVAttHXPC6/vwVHOSYTc8CSdZslGKogPBGkrGL28BAEBkZKcEi+xiikt2RAZJOY/VEYnzJ1ydjxPCU/A1TRsmksE/RZoJ+QitR70uMr1xkBgU2Z4pdjyOCH13R8vD2oEdaiXqPIKkH2t1YIVnyuLrjvhuQz0uW/H4bwHuy6wtkn6jgZ7sJWD6jPNYbXl8EosDoGlhfKBQ3PAZeC1RHEqoKcMagPJHI3/QDPg6aXAqsL1Ik674TMiC753edVzEZNjJT2b3H6h2NZK3i4CWGRu3JYShnc1aktHGw0de3eAO4AzF4YXUdYEqBLudnozpkN20vwzYVq0ZcIngucoHxtaBUf0IJe7rysKmEO0k55JoKJFFSX88lslyi3qO0ePrcojrVkaniazZBxWFPGJi7ZibjkIXDISWAbqzYsykJcptZBLeO3kZT8ru3HfN4mFKj75UUVqA6VgMo79NR6QsERNCwqcDoRsKlEs1+inYi4ZMUydKy2/Ew2Q1yALg04QCr43ePzaMcXgnUxxrZnYTLUpi1RX1gkC5V9CbLYRiAUPC4n2bQpYfedLAjDZdLyIZrBdkRbHcjGStnDAcRD4CbpagODbK7DnasYFYKLpcQYw1dOjRzA6MlXMb9AiKYlgLJskM7M5T+2sDKD4DS7shE2oKg2SwFumkCMTIcFhkJMTKQrYdLmVYbhEBIJNqJgQgBzV6K7LpEN0/hc01mzwYoLRESJh7rrY5JrDs20I345df7KfW2Q4AZHicr+xUWTlhKRH1hIFKFYBRBdqwhCUrCpwp6SWAWphlZwzZWh/Q9lalGSPRbvkemwoYIpIYuy2430JBlzWRXIQiipYQsawJTgGmsbRfv7xDyAujsADqD0TtmMQQ+vgd5+/OBTQwhAEWkyyNAFVpRjipFlKZK+iKtY+qrtUDGHJGvALsevEqz66GMjLjQmu/VdoPMF85BKMpkRZYS4DsHoRTBrZIDAO7TX/mF6QH1ljw2/v2nE17qzwaff3X7p377WjOW2Xvn4Vv/5b+HRFu8vp/COwFfa6hRhySxqBYZ4AXgBY4eP+D2s30+UQLBeEAxOVVMW/haQ24VfNYDNwl32kDcJwP7ERTgx+yW9GkYQkwAgoRuTNDUHjgIK5DcE6i5PMAnHrKW8Dk7G2UMI+nLuBHZiW7uoNeKwEGA5fAeQ8qjqgXsyMOs2aHIxMW4gM65oWbJBUDPQHZz7hOi/Axg2IhsuHgItDIx3TRK9kQAugOL9BWDVlQV92tsoS9TBn1UDGUozzx0xW2rnzQw1wmn3vE5QQf2VcpdgEHQTL20RQwFqQRcSmkdjf89OPRRUsdtM0vJ8BsQ6Pf3JwumLLo83tf1i2bAFQHprWSJPRgO41PK2lTsaJQdgalPgGRBeZ2MnXyqiRIpEwY5o2wIALNbsqAAgXc35QK0l9x1U4/sjUR1HM9ZlMm5jEEftiAzrZodgEZkmtp9x67NWAbP47KTXNUHu5RNsqp8us352n3SqbAEcQj0DQMEd90sIL0XQxqobMm2J0uBZj9gdCnQzOOiLAeveQcUl7sAEmAnESMI5vv1oKedE9DLjvUf2Ru5KymP7HGf4DkwQpEZ9QkGls0nZM05CMEQ4iMcwa2uuE+6pi/abLgQN1uqAIQjOz2EoDT0yAaBgcEdpINxyJEsAsozglkRf8dtEYNgsq8mhwbF9+/L5Ps0wv5c6JILb1OGr4SI9AwWFRIxLCYGgqiGMsBkSeDmjYhdkzGpUmFgNXk//262YWA/bS6Gx/TpmNJGhjzj4CK9J6DkdkX2OTL/LiOw6dk2l311X1ysdqDEcpcaanMOXHqmuE++7AOa+qAbl/LY9oE53pDNf5tJ5nGPKoWEj+m9yl1BYOqMiMwrBgZYNWE3ZIjHKllxe3pfZ7LxKGPdhM14rFQbdhLImJA5pG8m/LwM127cTgQCl6AJ1nqGtAclXkfVSGRe++3rJY5ex23udu9B5jRKHOPnYPTaojzU6EZMVqWfmI9JNj3DCzRzieK13wUISUEGVxF89NswBC7Jns1FZO93YVoiMMlT+F3voksIYPtQmyDF4BlSrR9YZrP10TvL1022Pn7meqVFPL6ZGFjdZq6QLh3aMdlss2FQDHsbY0JoR1azmygmqOrorW4JYM2WPlfVENTINnYlSlaCqCYM12TPTvbhNEHuGNhB7qujpDN2cHZjyVTWQg4gDKCUsldzkOWM6amHJiapxmFEyn5Il8no84+qgYAY6rM75/y8CJiNRTuNIM/FcxIAvSWoVV1kcyI767IIxiPLa0earK4j6JOtAxSHSOx39AhGDn/KliE6svWQjYXLNHyqIBumlkIAPlX0iGq584zFhb9PFZlTozhA7nzsnPRMWu0f1z/Xk82Epz+074zsfX6Ue/ZeRD8wmt4oCOcBKSA3Lew8g6q6QU7qMz08v5eZAvG7VwgCxNoOjxUNjwtciINvyk5DqiAaRxAYGdSBGY0M5cCWdpZ1H1UDP8khGktQZTRBXazmQO/T7D2RUhI0xrAd+Jjm2oNJG32MbYcwyiHqdvfcPtEV2L3+W+djYEJt3Na2g9AKwTq87Zvs/y60ZqJrDzh79vLt9+sBaP88pRCqt/yy/bZE3+U/MWMJ4J+JVNijx+Gjv/UXwFj+1l8xlv/Et+AE7u7H7HLcK7F5KJBdGsjWoHyvhbk1TKibOCz+30OkNcGD8GQ4uJgWwDZDthBkjmqJk98F7n6R4Cd9vSuVTx4kbG0gHYFDu8egCYgIMOKPUM9w2lFAcSlRHQPJg2YiZamGFMrm3QbZJym6KaP0m70AvVHQpUB9ZpFdaRSv6BkpXilszwmEkqXC9v0We39gIBx//NJ7oD6kv2r8kuyFS8lwmbUaYtW7SWQGToHsjgDBh4CQAnojkTywK80sJPTGIBhg9EJGSS4nh6oWmH4KLL4ZPUEHrDpoJwH5J0wAzW8DNhdASDwmPzID0Ns88ShekrGQLZC0EXScBUy+5JfR9jygeBVQH+ziBTYAACAASURBVAgcfcL0wTJIZLf0oDRzLra3F8De98l2ybdkq31KomxZ/l0dSdSHTI4UPZCWZFXS5W5B7lIRQS7Q7AOjL8EOv9eMP89eR8B1RcA1/RSojhn2AwFURzze6YNAfRCw//0Q5YYBqiEDN37hsX4isfcjj/JYIivJRCZLPnb8MmD5PsHs9McK9QEw/dJj9UyieBWweUymsLjxPH8jDPJQ1XCRlywpW85fB7QzATvm/1UnAb7mMShuAtRzHmPtKSWuDzkoKF6T2fEG2P+hw/odLlBcR0YMIcBsOT32MU4+WQXoa0pmx1cOi/cVhAVmnxDEeAOMXgFBhYEpqQ4lytPIOM5FDGMiUJ987rB4T0HXfO0ueq72f2ixfqyGdMr1Y4HJC34Ow57A6BWPizM735HXAukiwGwJJnsWXMWy+s05weP4kgFK6YLfL15Tkm22lGpWRwJ7P2bojy0IgLIHgqt2Fj+Ht1ww20ywpL4LsCkZsd5rxCRDvsf42mLxHhnJwz+x2DxSMNvAfR4LJAsWxNf7Mhapc2GqqzAs7E3pmfwYgPTBop0qNDE0p/cbjS8dmjn9T8Ubi+0pqwe6Cbd98sLBbHdyWrPl+Rlf8fzrNd+n2hcobj3yNy3Wj1MUbxyqfV4f7YTgK7/3qGcS2dKj2pOsgOgltzFopZ73YJOMF2WIHttTjfFVNyRUmi2Zr2TFhez4mlJnUxKMjF47NFOJ7khgdENglb10cIagphurwW6w+AbBlohyV7N2qI40jv5oi25s0Oyb4belPJSYf9bh/iOD0TXPARMfJXRkyboRmbse9Ob3fhj69DLVel+h3iMbr9qAyfMW20eGAN8BMqa8SuvRTiQmL1sgAKsnKcavOkppxxLjqwbrizQGL1ksRwZeC4xecQHtcjVUUQgHnPzOEsuPJkiXDragBLY65Dmffxz3d08ju7PYPjJIFwza8amE1zHR0nMwkazDAIJ0zUFgdm9hVh2agxQhFdCx9sHmEcQ1AbricR9dd/TnRXAKT2CW3VkGFd0xlCsoAasFipsOLpFIl26QSZuNG4CLWXboJgYuk8hv2jjwUDDLFosPC0xetNDrFs1RRq/dio8he+VhNhbdWA+DR4DyYyoMJIQNyK82WH40Q3bXoT4wMKUfAoSy1xXSO4aVBZkgKIYRycaim5Ex8olEsurgFQFceh8HMAJIl9ye8lGO0fMtulkKs2pRH2XILzcIWsJOUgLT2g3SVdlYmHWfLKoHRq4+TJC9YaBNOzfIbiru1FuLe5srJHcV3DhhsNBYQ5cdROVgD3Mkyw7dNOE+9uDyLUDXzvm7nr5YwM2LKO81SO5KsnVaQm3jl1oEfQAgNy1Epgl0c4NukiC9XAKS4UR2mjEIaFZAxHAivW4QOklwW0eAWDXscVxW2H5jD8XnK4YKaQkkCuphyzCf5C0GessprmzaQb4pGsvORs19hBDwRlFCnCioZQXRtPDjIrKHCWWxRkAsy50s1nvIGPDTH2dhHbCtyeLFdNZgNOTdCmE2JkhcbRHGBURZk2nsmb884983JX2LWkG0niDNMLAI1iLUDQN3nIdoOoLENAGabne+e7CnFEJdD8E+UNzu4KP3U0mEtt0tqIUgeIzHL9gaQwJrYvhZEWKoHgkuBgUBABzTXXumNEl24K8HlQAgxU9nI/80qPzTgT1vBwf91e0v9e1rzVimjx+H9/7d/xjVmYef2njhA+mkgf9sjG5K9i+oAFfErsHcw0waVkdMW7hlAuiAyQ8Myl+poD7L4DJg8qXA4p9roe8M7MxBrdk1V59amAcFO/UDywgJ5FdMzGwOHfRGwo7j/QBCyn5B2VGSVz52UBsJe9RB3xnorUBzxOd5BUDGH+839Ca5nHLBbkI/Tc9a9tH6/cRfn5cQPxyTIdWUdbZHDmahuJkd0F60EKUGxh1CrSCshKxYT6G3ZEBVzfj/bhK9cWkYpJOj5wSZvXfLp2HwFtoiYO8HAne/7JDcszohvwm4/2XGdEN76DuD7A0XYotvsUMwMAgVuhLs+rsjMO+74KpjgsL8zmP5nhwkh92IFR2qJjOmK4IGrxlukd4qspAx/KKXcuqScsrRVcDmiRiCN7I3BNoujUELGffRbBnCIUIvY6RfsJt55NdkJvvXb2fsqdueUbI6+wS4/06AWbOTUG+j9zYybYPct+jlmwHZnRiAxeo9snwuDRi/IACTkcHUJVAfRZltPCblKcG+yzjskJYgs50F7P/DWHL+DmWc6R2weeqRPEjWBhR83/qAgMYWlG66LGD+Q6A6jOmqAoMkUlVMzq0P5NDt1zMduozezj3WPrQzsn224POTFUFWsx/Z2G1kbgsyjuUp90FXO59lz7QBEUTH4+RSYPKCQ4Se3TXbHYuoa4K19TsRpAQCgHZKYNwzE80BOx6FJVPT10psz+gThiDo6uWn1WHvRcKwqOdnVsBEMGQLMbC4ug7QJWW1fQekanfhItKStXd5lChPCG7bGYG3dGGQDA7l04LP7722QVHmGSSGEJT0IX4ZiT5ghV7aek7JYH7rB2/u+Nphc6ag2jD0F1bH3J90xdel742Av++xbOZiYJ6zO26rzShlHN04lMeS+xjPh1e7lOt0EXbSWIEoX+VnQ7WIfkUCX9WEwVvaTngtDJJOQRZsfOmwviBIR/zuG6Rx8TfPK7J83YhBJkHuJJCmDGgmEunaY3OmkD1EFs5gCHxRXc+0IQal+CHR12Y87tlDfN3IQqcrekTbqcDo2mF9rlC84bHvtzVIMqk2lwNDaHMmBjdTxdCVkve3Y75Wdu9ZhfFWqKJqA+q5QrL1MBuPdqrQFQL5LRknn1AqqtqA7LZDO48pvBuHdqq+4s8OUgyBPsnawSe999EPycOD37iQsW6jQ4hMYl+xUe9x+5OFhR1F2Wkboi+dDJnNOCj0KgL0il5Ms3Go99TABKcLi/XjBOmKrGh+a1lpET3pNnY7mg0Tfs2Gnl4IwKws02sjG6xqj+y2RnWSI1nTB+pSgoN2pqFaD7N2g89QdR7OkBXqqy7MqkN9nCJ70w6ptEGTtZQd6zWogPFk+VJFgBdZTr3p4CK7pssOtjBQlUU7TyLryO8kW+xeBwFQlYXLdfwNIKtkczVUZwQhIF1ktyuLbmx4PeZkErPrckibNQ8VfG7gDdlGV1Cemryp0O1n9K7mmttjPUJCgOZybndys4UfJ2Q2K8vux8ZBOD/0QMqKQTt2nEA1DLORHX2dQQB608LOUqjSxvoiRV9nBOlBCMimQ0g0XG6g1g2Ec0yl3bYQ2wp+NoIrEqgyAicX4CYp9LKCT01kRj19liFA1JZMpNFk9mIyrawtQbL3wxBCdBa+SCE3FUKRAi5Q2polTIRtOwJOrQZGsfesvs309QAYHcHuV9jGeBueBwDLDcQoB+qGMlbvCfgiUxmqGiLP+H5lRUBpbeyRdMD+nCm32y1EnpOpBCiXBXaJsm/dQtMS6Aa/822uN3E7BV8bZDfhenAZGdkofR0ksd1PYRn/McBiGNhX97Mf+DNufxkYy9HR4/DRb/z8Gcvv/jc/m7EUQjwG8D8BOAFXD78VQvgvft7b8bVmLJOiQ/1LJfKswzcObnFTjmGdQmMV3PsrnI5LvLqbwb/JMH20xup2hPOLe2ybBNlsCwB40AU7Eu0EWBlMf+ke1kvMfqnC5nYOMW5xMi1RtgabRUFf5khC7TfwMXFUaQfxyKEtE6QpEzKReCSTFmnaoWkMxo+WuL+cI/3mBrJKYI0BvIB+tkH3fIT8bAPvJboNuyNDrVBJQBw0SFILrR18baC0R71KkU13qVhtzdPYPmQovr1Ec1sg3avhPxlzAf3uBl2rkY0aJAD0vkfnFFxcEHSthm0lvPEYjWtsP5/BhgB5WsNk9Ftaq+C2Butf7pB8maJ9VgMrg5A76FuDw++8xvXzfdz9WoBMHey0Q7dIuCB+rWHfLylP9gLh/RLL5xOEzKE95OIbIsDOBYqjLcqigBp3KGsNOAGROODDCjc3Y4i8pRT3wxL+KofbbyGXBj7zsPsEDHrcISwTNPsemHVQiYO7zOHmFvpOozqNgReJQvNOA5NZ2OsczREBsJk1aJYpIAPMnWZK4qMavtJIXmvU7zcIpYbILTaFhpq1wCV1rG7isNAKyQNgn9a4G6cIOqA5cFAHDerbFPkrhXaPX5D1qUfxXKHZY58jjhqUJsPqQwe9pD+rurCQtcT6mQAEWev62EJVEm5mkX+RoD7mtofcwe4JJLca6w8opW2nAW6vQ32QoJ0HBBPQHDqyg2OH5riB+H6ObhLQ7APtowbNTYJgPNzIY/S5xuYx0Ox5SoejRNtnHuZBQXiJ9bsefmKRf56gfuQw/lShPohS7RBgx0B70qFq1DBUUQ8a8x8C9fsNzJcptk88zFJC1QKbf6GE+UGB9XtMYoXg8RGOQVHVOZNW/WuJ8rFloqmnz7ebMSEPATBLeqDXxx7THypsn1lsRcD0Bwb1GPTnbjhYaY4tRCsHGW9+Q3CwftezX/RTMyRpSkuAVB3zeGav6WuUHUFkc0jw7hIOaoobgjQW2MfFuga2zxz2/4hetHYGmC0XuuU5B2E+5aIVguqKIAWq0wCbc1gRdEB7aKEfNLI7Mup9sFMP8tppQDunhzdIJo5mtwLNTA4BY8t3JeoTh+RBwY4VylOP6acS28cOsuM1ZAuC0KCpMEjvuRC0Y/pH0yWHEu00BmWlYvB41nOJzTtxYJNSgr195jD7nkKzR6C7fhKQ3wDNYUC4EqhO6IXP32AopN9ecPBCMCtQ7wfIKPPfvAMUV3wdlyqUZxz8+ITA2qVUDQBM8qSqQ2L1LjC6VAia8uf6gP2e1XFA9oZSy805j5VqAsrDGFamZZRSkjnenml2RRp6onm9SqRrJms2MwJn1XJIEwQ/I9JKlGcCxTXVGe0sYPJcoRszACvM42BCK4L5CNwggPKsT5qVrN9ZBnQF0E0F0jsOJFwqGVZVU0p8+x0NvWXyaR9OVR+awTupGnqOg2Bw1OqRRjtlqJnsANVKPHxAgGc2lPdXR1QF9Omizb4AhEF+57A9VUPiqc0IZgtNf2Z+T5a5OpRIl5Tobi4UzJrXsK4CzJahQc1E07soWbfkEgObcd+dEWQWNwG6JNtuyhD7OTW8EmhHcpC5y45ezHqPQ0wRJJbPJsjvPLpxinpPYPqcUzTVEsBVhwRk2zOFyUuL9YXG/JMO7UShmUmEMx09sSnSBRlemwmkLmB7kkL4gGTjUR8YqNYgXVjYnJLHZqqQ3UsGllUBdkRgb7TA5lxT/dAE1PsKzgikaw4x2qlifsOMbLQRGAAzk2M1pAPGL2o0hwbaSHRjhfymQXdEH6Y8yFHvaWR3Hbp9/o7ZXNHHmQno0qM9yAApsHmSwxmB8WVLhWOhB9BbnhgENY7SagEt6AdHoVixs2+gawV/lCK9a9EcJsheNyhPE4xe1qgPE6T3Hew4QVfEHmUt0U0UvErZkTlOoCqL6mKEZNEHSGXsw1QCdpzALDV8ZhC0QJfm6D247cSQaR2RvTSbCFYbCyiBkBh0hwWBcAhwmYbIDaAY2ORTbpOsO7jcAP3nv7Fws1GUDzu4vYKSWe/hRykgBOSmpsTX6Nidqcl4GjWwqLAOvkhjCBF/g0KqIMuWya8A/5Ts0BzqRNoOyBLeHz2WIspqw2wMsa0A16fISojZFBACIqaMo1covC1vBYCmpV8yMaxI6QFvBJhCRVmttQSfUrBOJLLDPWklEnqFesja91YON/8WeP4zQKaQYgcuv+63fzq7YQH8Zgjhu0KICYA/FEL8dgjh+z/PN/laM5bzj47D3/m738Zvv/kFfP6wj0nWoOo06tbgaLLF5e0cAcD+bIuqNZjlNR62OeqKF7o2DsY4OCehYyVEVaZwtyny8w2kDNguc0AEJHkHKQOkDDiZrvHla/o1fcdFXz4i0Ks27FXTxsG2ClgkkEc13CrBB+9f4ccvTmDyDl1lABkgtYdvFYIXkImDbxXQSuyfL3B/OwFaCZE7hK2G2avhHWs1ylUGbDUwYq0GXqfwRfRSTixCrTA5XaMsU1Z2NArCeIiFAQ4a+JqmH6EDQidZZeAERK0g91pI5RCej2CnZFd9Fj2lex32DtZYrkYoRvVQzfHw+R70MSU43SKDaCQw63B8vMSb+wl8paEfNMTjEv4yh35ni+aex1ZuFXzBiaBeKfjzGr7UkIWFuszgEi7ek5MS9sUI/qCDeDAI+y3M8xTteUdvivGAF8g+TVEfO4TMQ64VhBNwUwu5VQhJ3I8k4NF7b3Dzxyewxx2EDMBWQ7QCPvWADpClgh85QHuItQYUcPj0Hos/OYQtIps79kwdPu2gcge3NhC5hU4c/BU7/VzhoVcqemUDxH4D3ymgjmzXtIPfGIiGckZ1UsG+zjH5QmLzmME+ds/y/OQWQgWIVxn8SYPk8wzNieV5DwQfqt75fMPYsvLk8wzigw3aGwZB6LUcvumFBbrHDbA29Nh2fD99p8nqHTiYeQP/smBS6h4BWHqnUD9uUcwr1M8nyG8kth/SY9udtkAroTaUTnczD1VK2BnPQ/ZGojn0EGc19A8L2BEBc18Lkd5LNAceshVwGY1gIsRAokOL9JWGT4Bu36L43MAboPuohPo8h94Kvl7FP21BP2/7tIa8TbiPz7ZoHzKMvtCoTrgwyl4TmBL4SDQfVQhWIv0shbRAdeqQv1IEspWEGztkrygRqs87iFYOvuv0nqXkwoOS+SjB91kcIOgAfaejCoILifyVQnPg4dOA/JUaPKLdKFZuzHkNJLcK0u6Ydl3y3LczstU+BZp9/nibtQQ8oi81hgjFEK1kwXCn3rM8eiHRjRlYlb2RKJ9YzP9EY3tB363LqJ4QLoJcFXa1IZrHILnhPnnDBXz5mMMNl/A1ei8rgIHF7sYMSVINg79GlwLrZwSFwpNFZoUMkN4SPLNSJGD0goFY9anD+AuF+jAgv2Yva/pAkNgnQCMy0yJ6UmXDAcD8+wLVSTyGMTir9952Y35mzZLVKEHyfh8rMwafaAyUUU1Mtb4mO9rsAfUxPerFjYhAiHaE+gjIbwKqE4HRi4DNE8r4kzW9vek90Pchm00/mODnxBa8nsYvMDBzfXq5tMDoymP1VCJZ73ys3Zj7lL8maGv2gfSO6eH05kYWeCoGf7NwZJ91DdR79LEDQHEd4FJaSgCy6t7QgmHHAfMfMYhJuPietx7NTAwVHNUJmfBmHoH0F2TX4YHJpUO9J9lFC2Bzzse6VCBbeHTRN7w9E5h9RiZZBKoIqn3aLPI3XKD2fto+5VxFtryeM2So99wmaw44XcKQsD4ZW5f8jHUjpnvXhwKjKz9I2ssTifzWD77TIGO4VBl7dWOYkin5GetyhjrZjBLv4tajHXO/NqcKxRtKfrfHHCK0E4HJJaXdQfY9yEzcrecS+Z1HdShRvKZE3CUEnX3oU7IiwN2eSIyuyewmGz94ZHXpUR5p5PcWzWzHUgtPT2x+x/7cZiqRrtinCwDlqWHqcM3u5GzhhhTlnsnut0G6AFWHIVXXa0EZ9m2H+tBgdNmgOkkGVl12AemCEmRdebLLOftXkwU7dnVFr67ZWLRzg2TVQVYW9XE+eCh1FRk5R/sGJeMEwgixD7Vit6nLGc6X3TRAZKtVZXdoCICsLBng6Om0hWEPK8Bk3YpyVAZ6MV3XJwp6VcNnZvCAukkKvapjIJClL7Pp4LMEsmwQMkOfZttRZRHZzFCkBKGRxRR1O6TzcgNjCq0Pu3qQEIY+TLTdkGbbs6qibneS3B5Uvh3goySBa//6wCC5DU0LkRhKXpuGUlUlKYcNfpDKDixmL8n9R4DlTwaTP0syG7r2J9/357j9pWEs/82/AMbyv/3H81gKIf5XAP9VCOG3f57b8bUGlum75+GD//zfx+p+BPWgAS8QdCAwEIBZcFHbzj38yEHfa/qI+pCd2AsGRNIsLpi84UTdpWC3VvTu2VH88UmZfKjqGKBhuVjpxlzYNQeBgSx1TEZMAZbehyHNkWmEfL++J6zvv9LRdye7fsLPH2c7Yn1IdRK9je1ukdbf+mAMFYNYVL1Lf0yWXKSqjtuqy/gFEcMbzJaPa+YByVrs6iH6ihbFx9kxX9frMKRtFq+YLNkH2tgYTtPsB3QTj9FLFdPyKG+D5J/tlIErsoupp5udfLFPjO0rPPo6h+H++P/diF4syna5WOvGYuhF6wM8VB/ssuVirZuGoV6mD0qQHSWc9YGIibtRihflr+1elJg2GCS2suPjpeNxFT4Gl1RcvPoon+vPlXBAfUBvpjfcZgamxAV6lGPqmiyRbHltDPJjvfNR9t10MsoFAcSFFh8XZAyQ2bC3bvzyrcTCZBf0sj2nt1B2IS6wxVdSeb3hddlX3fTH0+U7GasuWZWSrKO0VzEsqA+C8QmGPrhkxUVlN+YitQ/l6ZkNXWGQOfZhKX2Ijh0B2W0YQpP6c+MTXrPpw64vzaUMc0kiA2LWDGapjrmIzd/EEB9NnynlyFwYNgeUPhbXlF62EzEs8HvPsK7ISJZnlGT33weqiXUysYaD/qrdOe2l2C6NgCCe5/5G5ovnxxYCzZwSbpdy/6TD0InXTz57SarN8ZUFkVe7epMQQUp/3Lox4vVFqXmXC3RjgeINgUD2ENCO4nZnbwFDsbsOkvVOkhrE7nuEnwsJvWU4ULIOsZqD03FTBpRHBI+64ndxMyfz2YcDBcm6mD7MJl0GlCcS2Z3/ymOa+Ve3FcCwzUA8vh3lqvRS8/tddkze7UOC2glZtCFMygPtiMevlzSna49mKodz3B/rIZTKCGQPjsmrGZk51YJe2T1Jb3kMtUlXfpCJVnsxnXjl0E76JPIwpML24UfthJJ04eh7JagU0BVlqZTs70JuuoL7q2t6ylTLupluREDQRcltsmVHYXmsmPQro180ApBuTI8usOsWDFIMAAYSaCcq+ggpp+2rd8zaMT209fCG7GG6pAzZ5gL5nRuCjMzWwSW7gKFmrpCs3CApZshXQL2vkS74A860WLLSzVyhuOmGAJ0+gKZf9MvOw8f0U2FDrH2J4CNwH3TlB5ls/51qttx2vSVYk61HeWowuuoGWXrQYggS4nd9gCmjJ02IKP31cLmCN2Kw7/QApwdDNqNkV7gAXTvYfFedIjoPKAFnGGjkUgW9JQCUjqFGwnrYkYbZdPCafsVeNi9rB1doyMZBtg7dNIGu3FAnozrP4BqADNzI8Duk8xCdhyo7dHsZw336EKD4WDtOmMAaa0W8lgzH6SWwnsyZz1mF0lei9B7Svi6Ff9dMobXcX58oyMYOLGNINFNbY+0KU1vNwLiJlvaoYBSE9wznyc0QAhS0hKgtmUdFuaratqyKUQxAQheB3FuhQdzAeP1U9LYO4To9oPMMsoFWO6mr84P3cagZ6R/n2S8pmg4heyuoBxiAJCni+B4+7PyV/e3t8B4h+HznBrlsqGpKXbUeJLJCfXUB+RPxQA8ErSV4jPUlsHYHSt+SuQ7eyrhvbwftCPGWx/Lt0J7+3/LtH66fjk3+WQCW3/w3/qOf++v+4X/3m18CuH3rv34rhPBbP+mxQoinAP5vAN8OIax+ntvxtZbCGu1wON5i+8lsYMDyS436/RbwAupa49/6jf8H//M//FXkP8jx6//an+Dldo6PPzuDyxTya4npv3iD65f7+Hd+9Q/wdz/+Dppa4+zvGfzN/+T38N37x3j+9y/g0oAPfuU5Huoci02BD4/f4HvPz9AGASwNivMN/u33/gh/8PAEzxdznBcVnn98AtcIzL8vcPc3Ophrg4u/doUvrw4oJZ3F4I3zDX714gW+e3WBi70lXt7PUTYawQkUkwbVp1P4kwb6MkW3b2E/6CA/y9GNAmQCdO9QlgkBCCuQ3irM/vprtFbhWwdv8Ht/8g0AgOgE6guPyQ8Mjv7Vl7j8nQuUZx5hQlRs8g7VNoFcauRP1xDSw/5wzuTYDti8Z5FdazRPG4QAhJsEF79yhUWZw69zrE8lQishS0Wf30Zg85RSVACoGolgAs6/eYPrPzrFs3/+BV48zNF9MYYrBLrDDullApcA9QkHAd96/yV+9PtPByD35G99ho//r2cwv7iE//05yg8aZJ+n6B63sAsDdVIhPC/Q7DHMqHy3w/jHnJJVv1RBPs+AZ1tYAN0qxdNnr/H6/zhH9c0a4j5BkGSKmn1g+lnA4kPAnTUwqYX57hjVCb28QRmo9zawn41hJ2QGhSMLahYS3bsV1PMMNgee/Y3n+Oz1AWyrMfluhvoo4ORXr1H/7hlW70c/7nkDNauw3WRQzzP4NKDZF9gWXFialUD9rQrm0xzt3EMe1ZB5i3qbwnyW4Tv/0sf4o9/5ANkHS/g/mCNZAuunDMVZfhiQ3jGJFRLIfuMGd79/wvPzzQbJZcKFzYYDC4BgIQhKDnUZUB8C9qTF+AcpXAps3rUwS4X8tcDmqcPoYo3lKkPyZYrmtIN/adAcO8hGoDkE0juJ9QcMowoKqI/8kMwLAJtkx8T4NKD+axXkZzmCCoPvN7sV2F6QjWr2uci3RUDyjRXy/22K7XlkJbIAaQnc+G8C+O050Bxb6GWs6kh5XMtTMiZBA0n0ywFAaximZUceqlPQG2B7ETB+zuHMu//K5/jep+coPkkiU0igl6xiCq3n0Gn8MuDho364EeAvaoSHBLIRCFJifBmweJ/bqOqAh2+zQiZ7TR9yeSqxPffIbwS6f3mJ7X2OdSeRX+qhqzW7JcDffIPXe1DA9il/zIsvNYprMi3dmEyZjWmw6/ct5KSD+TSHqqO09ICDsW4sUf9Chf3/M8PiQ/q9g+Y+zn9MMLR+Fj3jZwKiA6pzi8knGtIGlCcCwVB+27NLXouBEbMjgdmn7C91CaBrXmvtjMBy8QEw/5jSURsZW1WxckR4oD5g8NPokj706iwgiMjCR9C7PZPD1W9P+wAAIABJREFU9QwQiL35tYCjvx/rVixlzie/Bzz8AmW05SNKyLbn7DCuTnidmDVZUHqv2aXrMtYFbZ46jF4qqIrnIVkErB5rVqKMBTbvcPDV3ZEVac+ByReSAVKtwPJdib2PPcpTEb3OCg/f8dj/rozsCQca1aFAdeox+xjYPKYUdvqpjN2qwP2HmtfxMmB85bF8xsoUlxKAr58oJCsgvyUbFRSwOaUcd/QqYPUOq1nGVxYP75uhJiVZB+icC756X6I+BGafEnQtPgRmPwbqPb3zYfaA1wW8/hWJ/e/xvG9PJIooa272KBNtZ2QNyyOFh18I2PsewWN1oJA9OHSFRHkshwCk/I6g22uB7aMIuCVQ70lMLi39ngGojgw9pjMJUwWECX20LpHQpUB5pGAL+lybmUR+7+CMRHVISbHOWP/iMmD6hR+6Z1ePNbIHyoqbmYJNBeoDdq5uTjWSCMi7QmB8ZVEeawSpYcqAZO2wPdGYXHZYXxim/F5TKjy64Tbld1GeHb3gPhGYvLRQdcDmUYJs4aC3QLNvUB5JTJ5b2JFEO6XUdntM1hNCIH/donyUMxTrRCO/d6jnCuOrFvWBgezI/HnDkKD6kADSZmRVZReQ3XcxIMggWTnkV1vUpyM0c4XRZYOgBLYXOVTjkd/UqA/5YdOVRztRrFZJJFxiMPlkheZkhCCAzSOD2ecN7j9KcfTHW7RzAwQ+1ycCxZcW9XEKVTNsyaWKLOd9h25EOamqPFQlUT7KkL9uh37QdqKQ37YI42QA0gCGECQkGrImiBVSwKca7Yy9oqkkwOz7PH2qoLcdwXLn4NOEns+yg52n0BsCO68lZK0QjIRaVrBHU0jr4QoNfVdBtB2qZ3tIX1ccqq8D3LyAWtfwRQK5rslUejKMQUrA+J0X0zn4aQFRNpHh3DGdsG7HNsq3WMGeqQyB4NXoQX0pHBnUr6TSAoP/fACuUT4bvIfIsh0YDgFQEdRqDV8v6O90HqLvuXwr5TVYC3V8BL9Ysnu0v7313vjToFbhJ9+8R+h+yn1/dbv984BmIcQYwP8C4G//vEEl8DVnLOcfHYen/+l/gKpJkKctWqvRfW8K/14FuzGYHG/QNAZtaVBMa1SvxhhfrNB1Gl2nMCoarF6PAeORjlo0qxSPLu5xdbmPycEWm2UOfZXCfLDCo/kKV4spyocc6bRBc5/DzGvY2xxqv4GQHkoF1IsMQ6hP4iEfDOWU8Zbt12gvRwgyQOy1CE4ilJRryoMG4TpDSAPy0w3K2wJ6qWEPOgjtESod5UkCGFvkH6co328GT6ZaariJAxIPIQPUdQI7c4DxEJUa/JDiSYmupMcTAKACYAWlmFbEL2MBN/IIYwuxMgiph14q2H0LtYyfeAHIhq/R7TvISiK52KK5Gg2MkZ9ZpqpKwE65LaboEJ4XcGcNt3ul4LMAtZEQT0rY24zy4rMNypsRROBk16wk2gO+D7eTTJBPAvRaottzEHn8EXnNmhgdOzPJSAaoin2fIWcHKQJQnG/QfDwdpJb1kxbmxsTuOPZ7upwSyTBi3UzQQNBhOA5qwxoU2Qp0Zy3S5ym8ib2AGWWgwZANNGvJGpkY6JIsBcF05pFda9SPW6gHjeKVZBdnYBcqAmAnjlU4EwfRSISc/h+9oL9HlZSdpnfxPaL80WumALfTMOynHcV90gHJg0LQ9LS5PACeTK3wYlgwky0kyNs8c9ArObA6faiUS/j6Zs2Fe9+j2c4DZc92F2TT9+QFGSte9igZTNZiqD3pmXVvMNyf3VL+GGSAdFzgekMWv5sG6I1AdeFgHiRUQxbEFgyAKs8CggFEF7c3BmM1Rw56LaErgWbfx/TlyNAJAlG9FQNr1OwH5G94XdpxfEzC6yFZCFRnHvmrWK0S2d5utKsTqU54HFS7C10avwCDk5ZAfchjKht2fTJwhJ9Nr3nNJcsdOyk7gilWIfCy7IOS+veXLeWk7Swy0aMImrZkErsxGebtBWWyffWK3u5UEy4jW7x6j52zrATi6yWrXeBSN+Z2Fa+47azH2b2mbDEwcj4G1DTzKFuMVSBMOo6hPlMmTfcBQbpkomw3iUz6W9LNdg5MvqSH0ie8v0+DBviaZsPO0t6PqhoCqGbWV6pwG7yJTLZmkJXe8mvHK+zqOFzsE04QGTG+l9c8XyomqfbdmekqYP1Yxv5PEetbGHJV3MTk4jjkFzG0qHjj0Uw5+OiZ6l4F4GL9Sl/hIS33TVd95Q6Z4fEVgVp1IjB+4YdaEwAEp7FOxhYC+Ws/AAy+Fpn5nkVspwSE1ZFEccMBgWoYtmMzfiZ1HQbGvp0y/EoEnrdsEVnZ+DvRFXEQc2VR7xGw6YbvZcodQOwKMdTBMFSIzw+Sx0F4MspeATaTSLZ99QkDkbqcUlCAMs12LJlIK7hP2QMZYpdSvuuNYEdlQRazG0sEyaqQZo8qHF37+ByehxCvdxGYjsvALDnUjwRJf/Ps8xo21+jG3AldxmoUG9COZTynHkGIIZSIXms/hO90I4nsrkOzb6BahjqxtoNBPu3c8Hi/rtAcZlCNHxKX+4RpSAzMqaq5z8IF+JRpwsnrLarH01jHwhAiaQPDdYxkX2Xss6SKgwm6NlcM8CokzMrBJ+yN9KZnkMnMCh+QLFt0YwNdWrK9ShCsWQ+v+R56E6WhRsJHpk3VdhdQ1LCKo++xhCRz3NfH2ImBqiKQUruuS5eTLRWONSgDs9t5AkzrCURjFUl/DQvvY7psDMPK9BD2E5SIFgRWp8jFFmGUkS11nnUlWlLN0ocG9axfXxXSgzPnv8J4Cud3clQpCC6NJuMZWc6gFcOIej9mHxhUt0Ni7QDqurcYxreZ2bdDhKxlQE9/68ElsOu0BHbsphQM6+nluf1+9QE/w+uEgTUdqk3+HLfQNH/2g37K7S8FY3n4F8RY/ve/+WfumxDCAPh7AP73EMJ/9nPfCHzNgWX65HE4+zv/IYIV0PecsKpGxHoQBoHoLReZyR1L7ZMlJ5pehxg7L2BzdhqadZ+gGKeoZ7vFHAIXkLbgojtZ7RaZvVyyX7zUB1wUq3on3ex7CvvOPm+4sEsWsWS85AKkOo4l8FGuazZ9iiOGCXIQQLpA9AwIdAWQLoHNBRe7IWKu8pSS3r7HTnj6ZboR/TV9Yf1QlB64H+0sLqx8lLnNxFD8Tl+Nx+ilxPbCY/qZxPpJ4NT6gBP10QsyB7YA6lOyGL08sD7gPrVzSgzbqYiJjVwc6DJg/RSYfNkXWu9+6PuFZjOPRfBHAsV1wPaCHpx2wseNX0ZvleeiqhuJ2E8ZZYxxUVaeBhTXXNh1vdwtj3LHfW5ndUywVdxwkVSeBRz+MRdUvWcrWfO82RxDf2BQXNz2/9fOYrF6zkUyuwR5Lts55bsuYwDGw0dA8Wo3/R9d0e/TS/dcysV9fRhDSA54fsyGSZX1XMLl/Hd9wEVeuqCHK4/7q1oM3X7SIpa08/3SBd/PbChJrQ8obQ6SzBp9H1EuOufiPr9h9cbyPYn5jx0W31CU0kZPEysyovQ3AbJFQDMV2F4A+99joqeMUkhpgckLi4cPNLLbtxJhmzAkueZvPLpCcBF855nc+khi/NLH2gV+R3jNIJH0ntUa6SJ8JYxldO2weofbmj34YXFttkyQlS2lp6YK2J5KjK88NmdyAD6jaw9nWNbuk51EO33wQ11DEAQN6ZIAg14sHvfJpcXymUYQlOV2Yy4WijdkSaSLC/6UEksEIFv4CKZZ/p0uY0LoiF6toMjg9AsknocwMJXFa496LpFs+JnpJeDZwg91I7oJ2JwpzD/psLnQMSyFC+JkHZA9OLRTSih1xc9N7y3LHlidka482pEc2OPi1kcpLGspxpctlk8TsiOWaaPVvkb24LA9UxhfWXgtUM85yPIJkN1Hn9iWICFdOQbjTAXG1w7VniL7lEqYDRmiZOMZ7JFScmkLyXRXy3NRvGFFizM8X33ib/bgUO+rKDslCPQ6Amyz++z0bFbPniGwjgNCYHuqhyTbZBMG2azZeDRziezeoZkrZPcO5bFGfmfJ7p0apCsOxGxO2W073rGT7ThWnNxaqNajG3HY1RUEy+mDRbOn2QE5igs3wc+iqlju3nvr3pavImAATH2qb3bvkCxaVotIRDBjCWAPGGiTPnTwSqDZp/cuWVo0+1ykqyYgWXYDKyQ7j3bObevDaEY3lhJlJZiqOtJMNVUC3VhBbwlIVMNFqjcMhsnu+oqL+EdCEJLc14OEM2gCGpsr6JIgoJtqhCgL7j15vR3CjhSy1w2aQ3opXCyG16WHyyVGX27gxglE61GeZ0gfLFRlhy5PVpn0dWasdujl3y5V0OuOaauZRLLo0I01kocG9UmO/OUWvjBwWbzmjUR6U8JNEwIV56E2DUJq0M4SqNoNXsEgBVzO15edQzdNWQeSaLR7KfS6g88UQd9II7mrIKxH9WiM7HUJN07gNb83VGW5DwFACEx5bT30ogI6i+54Eiu93NDFKzctfGEoVe0oi9XLBlAR1PT9kIIhO3pRwx7k0IsabpxCbVu4UQKEAL2oYGc5VNkOXZG+SCCrDnaeQy3r6HV0CImGLFvAOoQ8gc8Neyn7Co4oERV1Cz/KIeoGUApulkNtW/hMEwwKAX23oWw01ZCbGsHoXf8kQKmq0fQoZgl9jkZz/yJzGFJDUBjBl+gswrhgbUndUDZbNTs/Yv+n0V8JyQltS+nqUBciKWc1Ue4rWRcitCbwy1KEuiFYKzKCvRD4fCkQypog7q2AntB13NbgMVSL9AE6JjKSvZS1//8eYA4pumEHGCN4hBCD7BchRJAZWdEeFP8kSez/T3osR4ePwy/86z9/YPkP/oefDSwFD/7/COA+hPC3f+4bEG9faynspKjw7WeXWNQ5jj9cI5EOP7g9wdl0hZfLGSbvN2itxi/tvcE/ePEYv/7Oc5TW4L4ewSiHL24OcLK/QtkaTLMGme4wNg3+8PvP4A8qaBk4NJIBRjkYAHtFBeclXrzew+PjBzy/3sdHj6/xo8sT5EWL/VGJ+y0TDqT0WF1PkOzVaL8YQb+7wfY+pnalHn/zwx/jdz9/Fyf7K9zcT6G0Q7NJ8Y0nNzDS4bYc4e7jA+hHJdr7DCJ3mO1tUdYJKhHQ1gZCBMznW9zfTKELi/KxQAhAlre4mGzReYlXPz5CSDxEI7F9BwiZh0gcGgFkRQulPKQI2DyfAgAuPnyNRZVh82KKTeEAK6AnHdxdb1rkomr63gLL/RF01uF+nmLvZIXy1RTdX9+iuxxh9t4Dnk3W+FHzDn2vMmB0VGJ1W0CPO1S/aNE2BiEAxwcrXH95ALVlUNDDtwBMLZIvU3QzD70WyO4FHn7ZQo0smqsM2XtLbP54hvDtNR5qA51YdKsU1RNA5ha+0siuDJo9j+Rsi+r5COJRhQDANQpSe6wmCfRWotvvYO75Jbn5TgvxYNDOBM5/9Qpv1mM8XI0Rcod8VuPOT+CeVghvUviphdiy0gaZhygVzEJCfLDBIid15C5qqBcZqhPAJx7F0xXa78/QfauBeEigTios35fwjUJybeDOGnSbDNICqw86yM5g82sVxMsM/ryGSSwerkbwiUN9JHH67RtcfnEI0QkkD2Qq7VELc5OgO2aITjtXcKcNtu94+PsUyR0ZynbPQW8UgvRwcwv1YLB9RDna5qmH3kokHyxRfznhvuQC7ZGFSD3knYHXHIA8/JKjDP3UwRuF+sxClmQA+3TT0Ys+zMQhu1FwRYAdBdx/i6BcV0BzZCEKC68zlOcezZ4YvJ22ALJbsoHbC6arunmH9vMEzaEH4NHOYvhQFgZGJ2h6geqDgO05Bh+18GDP4EUHUUvYF0znJSMmUb7fQN0buDRW7KRheB038dBrCZdKJsNqprgmS3q6t+dkiosrSnnbJzXKBwYH9cYtN3EISmP1iy1EpZCsJHtTbwxBWwaIwMEXACAEuJHHMpC9Z2iOgN7sBjkuk2j2BMpvtFB3RD/pfWRuPX3XtlBopxzCVecOshJI73iNCAc0ewSC9ZFDUAbNPn13+W1AfURGq51otDNuljcchpXfbJB+lqLZJ0hr9hS6McG9CGAwRxyONfOAZpZGdpXez/Seybr1oYbNwdTPUT9IY6BNO1Vk92qFdgKkD5rhMx5wuUZ5GtBdUoZa7Wum1G4YnNWNgepID37FAXQ7hfIoJmRPAGk57KvWPNcQlALbPH799Z7tt/zHZku5vvA8D+2EKo3NUw6QvALSBcNguomAqjkoqvd5bdX7EvUB4BIdz5OAzRS6iYhe+wA74ucnvYt+9pmACBr1XFL2eUC5KCtTEjQzAeEVdBkAySRaXQHZvRoY0vrAkElfcRgR9G4f9JYBQ82tRLLSA5itDwTaseEANHprqz2mKfe+RLNHxm17JpHdhkG+6hIM4UnNVMZrCehGhqqHJiDIBOWhRH7HOhUIoH1XI30gqNZ1QDPnYK8dZ2QIHYdlzYzDjepgxGtuLGJlkI4+bU2/saFcXnYKSax50bVHeUSQ7XWG6kDGACeyhTmAzbmCV2MOJVKB6kCiOpDI73QcApB1L49lHHhxiZUuyRZ3Y4HxS8Hzsa8hHLB6RyMfKyYpm3GsSxIDK+6yUfTKRuZ4bOLnQEDXGs1UwmzZz1rvSeR3aqhF8ck4Mv4SSSoHH2szlcj+P/berFe27EAT+ta0pxjPcO+5Y6bTdro8V5XqgZbqGQk1zTNCTH8BoX7q34FAvDUI9S9A8I5EQzdN43KXy21nOjPvfM98Ytjjmnj41t5xrl2Ni24XOMEhXZ174kTseUesb31TKVOwlUD/oMLuqUZxx4mbYqPgCskQHsf9UkNE1Vj0zxYYVhrCU97bHWtO0JxniFpgmLNSJiqgPzJQHQGJTD2nZuc5sVBp9GuNQkm4SkHOdWI1w6GKpeF6ZO9hVxnkkMPNFDIpYJc6MawS2U2SsFY8D1JLYJ4RyCZALkLJapMhJwgv2GdJ6Wtiz0/mZEkzhVBlcDNWvgAg8F0UZBpdYDrsaoWgJYN7APo8jTr4Q0OA3LZwqd9UbxQ9oUZPDCcA9nHGyFTVzBBoVgViIFspeoswLyCFIJAd/ZRFxt8Hi6gU3xtjktJqgtosAdOqYKJqAsFRCjKY4+N+Iux9iapWBIijHzN5SKNSB8+kExAqKdlG4On9xFyKzLByxPuDB3Za7z1prPotPZdS4IMU2T88/u8+/hzAfwzgXwghfpKe+wcxxv/hd7mSrzVjWTx7Hj/99/9zzvKP6kxLOZedcTbZlfx9WFI6FiWmyHGfJ4lYCuIYJWPtGf1e+W0KgUlyPZZccz2uYHCKT8E8pk7l7qlUvT86SP5GueAYrGGX9IqNsraRjRzW9PVEwUGQbg5diq4iWyRthF2KqeB9jIUfQzymYCCb2MYli+zt/JDUOIag2OU9mZgC+hMgu6X0iMETDKup3o6SMjJUs7dky7IN2SSzI3uo29QzmPNLCOB2tA8OvYxmz4HVffZQ9Ymtu2H4S3FDydPI7Jqasrb8LqbSaDKKRZLGZVtuh27JbkrH4AqV+g+lJZvAL+I4JRMuX1rcfidjGM42TtfQxERrMmsA0DyQE+vQnpC16U5E6tmLEzNnZxx8SQfsH8uJOTM191f1ZPbGYnUGcCTGLg1Y5+8c2mM9XTs+576NzJBuuR7dHo61T8E1IRMpgY8Dotl7j24tp5CRcRmUhpGxYtcbnwe439k+TsyfsvTbmDqwRy51zI0dftmeg5H+SGD5wqE+U8g3ZE+HefKSpWMuksS2X/E45ZuAfsXtsxUZt2lgdhWmxMuJKcq5D2M4RpSYEj+ra3qI1MD9GjsoTZIaisAgG58B1RVTDLffMKlDEVNS4shCVZfs6xvmjJXP6jClI7KigD2B0lEaZZowDSwnGeTAY0c2nAXxquP1OZ5bU6eBl4vwqUR+mMvpmpWW12Bx4+FLOd0bY/AYfYdx6l6UngmZInCd3lBWKAKZk2FOWSGAqfdQOg4kRkmgbiOKG4/2RMO0YWImReSgvDuSyLcxSVVTmItkHUkwXI7PuF7dBEpDVQovcQxaaY8V8h3ZV+kjbEmAJF2EHOLEFuouUNa45f6P5123Af1KId94iLF6oaacMJgU5HQXphCbkIl0LMLUU+pzmdJcYzo3lAryM54suuoOZfEylaofJJj0p7kZ0yzZIepQP86Y4Llz/Dz2ZBXHMRRZyDDdr8NSIb9z07abrUPIOeBVbUhy4sMMv+rJwo7dkwD3QXWUHLpSIttSwik8tzuKwwCWYT8pJMdGMn1aQNrAeg0g+RE97IL+yfF51Ti4OQewuiGT6IskEa0pL+zXBmbnk5xQTgE6wUiMYTVBCZi9pfKm0JMSYlgYmNphWGrIgeE3IckGg07qBRug9gPcIkdUgvdNBIQL0DW7H0d2U0Sy/uMxG9lnOXj4Uk9so680hIvQDQNvmKTMvkYACZSkgJoYJ3mmtPT20QsXJ5ZP9g6+yrhfgkoP4QJCqaH2AxCAUDFMxlcG5raFWxcQNjAQJ1Id5WesvBA2XYcJAKjaMo20GQhkBncANoUh67csKOWse/h5DtlYMn2R0k2ATKNf5qzdCIDsBoQim9hFxMjKi4zMHgAG4yABqYwM7SGVNH2mDA6hTD2R1iNUGWWj6V/MEgARgiE8mSbLGCL8UQW5T4OttB1icJP0dHqktFMoRc9hO0D0A+IYanPveMVM06OYZ8nn2CJmhuArSUpjmRN0pUAe7qPja1I6qwgRMTcQ+5YSVCk+SG8VbT9VeCBEvqZuWN0xhuoYzZ9j2A/wYRjPfY+i5D7GECDuyVPJIOTTcmLdQMxIasS6oXRVKcSOybUiy8hQjiwp8KG8FfgQ1DmX2Ef5gUQ1JtmryAwZUuBDyeu4bQCBZJK7jp2Xh9f82u/A36xKJB5kuP86j98bxvLf+1tgLP/hb5fC/j/x+FozlqLwOP67b3CxneM7p1e4amc4LhvctBUu7ub49tkVfvHmDD/4+BV+8uoZfvD8NTpvsO0LDF6htxoP53tshxx1n+F01uCuKfHN5RZfXp0gZhZ//OA9hqDw1eYYRnmcX6/w6GSDZjC4u52hmA2Ylz3qLoOUEbPM4vL9CkcPdvBBYpEPqHt2DG1fL/Hk25cYvILzEvu6gFQBs7KHkhFDXWDzsYEyAXlhMa9aFNrhfLtAvy1gUx1JuehQZRbrssPFbo5h0BgaAyEjTMHOyxgFtp3BelVj33C2LAaBHz59i5988RFWRzV+eHqB7VDgxc0RMu0Re4NBAF4FWKsw1BnmRw1238gxm3VonUJ7XaL+o4jlcY1Nm+FkvcfdvoJ3EqIcsPtsDfmshu00q1KshGwl6h/3ECIiKxy6bQ6hIiAPEqRiNmBXZxAyorYK1apF+34OOAH5gJrRZmMAEyELh+Wyxc22ZC3JzHECzwvWqnQS+rSDbQ2rXbYZ5g9r7K8rIAqITgJLi+tGA2ZgTcldBqhIX+dtCVU5+L0BkudUzAZELyAEUC07bG7otR3OK8TS06faK4hBsCJj7WBmFuKLCvh2DTcohNoAOkDmPD/RSVaO6AiRe2BjUDypcfN2xgHVVsJVEe4hGa2oA2RDX6jZCYgg0D12rHY56hGdhMr9gUltFe7+1APwUBuN8GBA7BSgOcOYXWoMjy0wSMhGIswd90NGoOUXo94r+IcW2GmolgNje+IgdwphxvoL2UnEVPWy+0SSHR8EYgaoHRCKiOxGcvLiiWXty5qssBwkxEcNbKehbgyalsdv+LTF9iaHiHGSagtPb6g9iklqlOpUbjV8FXDjBGLB/ZYDEE1EyAK3T1HWbtf8MtvfaPQPHWQXEUpWxvgyIuSso5EDcPt9xZqbu+TB3Ei4OQMjEFk1oxKbWr0VqD8C8kv6cYMZvaEiJelyG+wyILtR0ySPnUfoWkENIjEerNaJkv5ld+SQXSVvddApECkc6j4kEmAU0A17LSmRFYhgFYld8vUQnOwCgGgEdE1PbNTcPp9FqIH7PqwB1ek08UX/cMiA2Us9+S133wS94VZA9twnSErl8yuZWEpAd3qq4/B5hIgCquF+5DdqkiD6IkL1rGmxizglUutac/LvI4n+OCV5R8DUvEY3il7WYR1hdipJEDlxhiBTJQsn56QFgiIzp2sAApMUeExn7te0UwBgL2lUk+ydHkZM3lmm4GoMCzJxnLTTGI6QkprVtO8MWuIkmqsA4elJCwrJ65vB7DmpWFyRzeM2ceLFFWQqETlJKTytDNImr20Eyks9+VCj1BiTtTkRQptDdkfLhrTsgVQp2ZrM3D3vcwoCipKTVMA4gZnBF9w/1RNIuZLvFYGy8uaRQLbVk8xU2qR2mI6Tga4jpMs4OaQ5KSYCoFrWQoxp0rrj9T/6NUdPrhzoxY8prTwo3mumyThJ1HGf8jteuzZN3ukmTinDwQC6zVLSLNLEIENfxklNOztI64E0AXbL1GTdkpkeLS/BCBQ3nIjJt6wCiZJpxkhMLSKQbzTaUw1vwEmeCMiHWQqySoyO5+SMnVEq3J7oKWFaOsAVJXQXoTszpfWSheekTVYnZi4DdFel+yJiWJJNHuYMNJK2IBvax4nJH1NwEYHmgcLsvUPIBfolvaYm+UFFODCpPuN7hI8YVgrlpYUr1TSBA6QJoYbL9SXl6lGmiaiSkygisMdT1xl8IQ+s5I6yYWm5PF17nn8BhJw+T2nL6TiMybSuUFOyrnAVQs5l4nTG75cQJzAcx9TqlKqr+iSVloDaDvAzwwmJykDPC/hcJXm3h1tk0Nse8XRBz6f1kK0luF7NeC/2HjFXEI7puIewnINvMuSaPkpwO4RLktqmY0/lCKZjRFhUBNpaAqsZE22FABYzIAFsERdp/Bcmv6Zoe4z+ylgmJdp9sJYkvFMSLTBtn5gT8MauhzxaI/bDQUo7gtFxjJ7niH1PADvWjkyP30yklfr/2mMZUx3Kv0kq7B8ef/vhowLmAAAgAElEQVSPrzVjWX77Sfyz/+o/wq7LcTqv8aDY4/O7U/ggEKPAUdXi28tL/OzmMd7fLPHoeItcO9RDhhAF2sFM3ZWnRzs8X9zh7X6F3mksiw5n5Q4/efcUZT7g49UtNkMJ6xWezDf4xdVDzPKBEtI+Q6Y96j7DUdXiel/haNbCR4G6z6BERIgCSgbsmxy2ziB0gFAR61WNZdHjcjeHEBF9ryEEcLRoEKLAvs2htUffG2SZg5YBm1crVE/2cE5CqYDcOLggsd+UiFbi6OEO3WDQNRmqeY9mxw+O+apFvSuwWjXY7ir4XkHICF1Y2H0GWAm1GuAbDTMfEIKE7zRgBarTBm2dAxuDOPMQKgB7g/nTLYafrhE+bdjbuTeIlSOoLDxiz3AePO65Ph0Q9xpi7nB0tMd2V0Ebj6HXBMbGY3g1Ax72UNrDdQYq83CbDDABqvQIThAgZYFgLnV3QjDEaP6wxv5yBqgI0SWZqozI1x38F3P4xz107mDvCgYbmYDYK4KlMgBZgL4y8LP0bbO07PqUkWFMVQByD7HTiGWALBzERQ4/Y4+osAIx55dVLD105eDuOEsqCo9oJeSeUlRIQNUS/lEPISPE+5ydi+sAv6TnCQIES2nABx0hj3rEiwKhCFALi3CdIc49RKPYX6kj9FYi5CyQd2VEcSXRPnXpi1MQmJkA6AhzrSkPdQJu7WHuFBNWU4hT8V6xgzMSDEES2MlBMEjICkQToe8U3NKjeK8RNaWuCAKhDCjfKrgywq7IwBDA8YvELT1kR8mmPfKcOR8I4MxWJqAQ4asAVcvp9QCmapX8ViKoFJhUMmzH5xFmL9Efe+Q3CsOa50h1AGTyMm/JyIecgUpmJ7mvnuFPISdw40CUagZ6rgkgo6RUEgBC6jQclqw9Kt8p2NVhAgXgQB+Sfu1x8iDbiOk1dhETeEnHJiWiQqRzOUsDUQuIwNof4emHEqP6Ip33KGMClQQ9UTJ4SLX8mxivBWACIT4nwPxw3Zhqb8ZaldEbPawism3yH3aYJKMhZ5CPLw7ALahDCFBUVEjwPCRZak85qvAENsIdam6k57J1k3znSQkiPSZ1yOhDH4NrpnApMXqdWdk0emFHkDyqFnzOgTjkQd2S3aVTo3l+ogZUiwl8jJ72D6qhRj+9TwFEiqmyYy2QXYjp3pbD4eeYYFudB3THcgJwuiUgtXP2UHIgflAYjHVVo2rElQzDkQ6snwgEViIAo9/e7CmvRcSUyBsMFTcAEjMeUxUDl/tBcNG9c+Rm3JZsQyAWFUH6sBTIN6xZYcop94/VT2Sv1XA4d8BBUaS6VImUFBqj6gGC2yDd4f9jCNKoUHDlwQ87TlyMXZ9jLQyQwOVYZeRjql051NzkWyoORjXDMOdn0uibnL2z2H2UTQqdoMWUtDplF+CwDTJVkwUFmDZMDKw3AqYN6FNFi7JpX+pD52QcvWkKqf5jrO9J4N0jsaOc0Mq2fgoGUn1M3a2sV5EpT0BEoLhi+I/wVApEfWDkR7Y+KjF5uoNmdU3QYlouq1SQvKVJlZH6RcfqGwAH5n8I6bMngegYYWcapnYT0By9m2L83PQx+Ww92e+M6a1mO8AXOgF8AkWRWGSy4gRGdq5h9g5Ri+k1IwstEysdcjVtU5SULMNTzaH2A2KuyGanIKNg0ndQYtqE9QSPBX2XEILPtQPCrCBT2pF5nR73XjeCzXHfx/2ewnbGtxg1MenT89ZNUljR2w/CfBAToyolRNORMR1rQkZ29d62IMYDkzr+PkpjnfuQ2fSHdUz/FxJjl+VvhALdf8Q4hfl8UEUyPv46BvP/Ix7LH/y93z1j+b/9N78fjOXXGlg++cE6/of/6N/Gq+YIn90+wNl8hx+v3uByWOAvrp7go+Ut3tVLnJQNrtsKhXZ4fb1GjEBZWGhF5qhuczw7ucMXrx7gyeNbfGt1hX/8xbfgW4U//c4L/PT1U7JXAETuIU2AAGAyh3aXA07iG9+4wMv3x9CZR/ASRTmg3hX46NENXnz1AHACak0/oz0vcfbpFa5uF/C3OWLhkS969JsCCEyOtS9n+ORP3uDzlw+BXkEvB7gtDSqPPrrBxb98gONPb3D1cg0AyE46+FcVwqnFbNVifzXD7KSBlAG7C3r9RJ8+BD3Bj18k4KIihJWIuYfcasRjixgE9JWBWzuonUIwvE7MWQvvJdm7VKzuG43svcHw0EFkAbFVBGy1hjzt8Xc++RL/yz/5LkIRoHcKbuUgBkqiikc12psSiMDR4y32f3WM8LyDbzREK+lFKkJiRQTcPEAcDQgNyfbF2R5Nk8PvTNqHALVRnMGuCLriZY5Q+en8RSch9hpx5mAuDRmyTzoIEYHrfPKDypZelVh4yFpB1wLDmYO51qloXmJY8QvSzw4SIBEEWUwreeyqgPKNQnvG5SIC4miAOM9ZjbE+JN2GPECe9kAUKP+PEr7gwF14wK091I5fAqYW6B5bFG9MAj70DroZByTZRqB95iCsQPlWoT8N0LU4SKV7gf4kIL9hSq74dI/h3Qxx4aCvDAvie0ypqb6MUI2AWwWovYSvyIIJK+BOLeRWE8gVYOpuTk+vHNIASxOQ2aOA7FqSjfBjRYeYBmB2HqaOWPbIRviM2zB1yQJT2mvQACQQdEQ8GSAvMwJRS+AX0+A/v6GfsHyjCCKzOIFK3fC46Ya+yGFFabO0BKb5jZxqTHwepx5HgADF7ATaMwZa9WsyCtmWHq7hiH7SUebuygRCWkyAazxn4yA3v2F/pi/ZD9udpH00lFDrJgVeRQI/4fj7cMSwLrsgqAYINOyMx2Csd9E1wURIxzW/YwqtcJTbd6esMLELDoZZsYEpQTIKMq9mK1MCMDtSdU2vJEICHGMNWzcyZAfbgfAHoNI+OJTYM/CIyxzZqLGH1c54TQ5r9s+OzOXI4ARDhtTsCCBnb2IKFCIDq1ugvAioHzNoyZURixf0GtJrhsTu8PwEk7bZEdQNCfDqBHjGczACbLJBlM9mG/aQ1k8EyosD0ByBUHV+sEvoLk69sM0jMU0ABAMUNwS7475LhyRp5bVv9nEa1Js6ASqDqW+Wn/cjU8Tj3R2LCdzL4cDSqi7CNJRCIxx6QEN+2J+xM1Y6Svmr88DALctz4QskSSztC7ohqM32EbY89OuyqoWhWCOrNyYCV1dMwB07SqMUibE8MMYiHOwNUbAjdJSh7z6SWLwKky/RlVx/dyRR3CaglgKjJsCZfo7BWlFymWNqrAisaBntBMNcIN+NHaEEs7YSyHaBnaTH7KHUXZwmlFTqFB3Phxp4bhETM98H9AuFbB/IlIXxOhCUTifwOia62kqm74KQ2F72hVLezXtRNyFJ0xMwjJw8y3YWrtTT8sd1jQ/hkUKh1ARwx89o2RPU0aLAPlQRgfJymOTuSNckIpBtDkzjJCHXBwZTDvckynoEwmNwVVLOdD51jiqohsFYcWS3IpimayNU5yZwOEqQQ87kWb23k9w5akqLIQS7K42CT2BQ+AjpKB2XA4OuopYTKBYuHIDkcGDzYqYnme6YCiv7xFiKw3sBAnFoCdEmaXJ63xhGJLpEkd9PcB2BXmYA6xggVOaHupFRVnsvsGhKk1WSMuMYJ/nvBPh+LaX18P/09xFUjmDuvhw3BRRFawGpCPr6YXqvSJ2b8T5b+dd1goaAGOMU7PPXPe6H/XzteyxPn8cf/Lt/C8Dyv/39AJZfayns1hb4n8+/idZqxChwvl/gJ/EZNj2ln//i7RNIGXG9nSFGAXtRQh4PyHKL3V0FmXlEL5CXFl989ghybnG9nSFEAd8pQEX85ZsnBJUmQGUBwQv4RiNf9mjvCoKyzOOrrx5C5B72qoRYDrBWIXqB882CYMsAvjbwKgKVx+2ughCRssG9xtApoPRQM4dunwGzgF/9/Akwc1ALC/GyhHjSAzuD92+OIE4GXF/POaOW84b1J5YM7BdLYO7RvJsTxCwtohfAIBErj/lxjd35HLJyrDuJHPijI9iT1wb+yMGXAbJ0EHOLmKStw20BUTlgYxCKBKZUnFgU1JpySgBYD/CtwlfbY4QyQM4tXKrokIOAP7Zor0tKI08H3L1aIx47oKP0TDiB8KRD3BuESGYm5gH6dYFwzA+5elcQZJoI0QogSPi1AyzDHcJdhrhI8shOwJ+kD/Y8AFbCnjj2RQ2SYDiQSYMkA4cA5O8NhiOP4aGH3KsJbNlF4GsS46P3ZOui4vJGFgkeaJ45iMCBibmTGEoNrDy8P3yQSkcpjt9mEE6g/lEHbAyBquGEQMi5/X0moPYK3Sc95J3hsdIRUUWIwAAZAEAQaB97yF6iPyXLJwcBe0KQ66ok0XIKuhGIvSFz6YHhhIE0MTFZIYuAB3yZmNm07fmbDCEjGxGKwG1wApCp7mROMGqPAmQnEkMg4GYBvgpwXk1yxJBHeCcSAxMn4AAJuAXPh7QHNmBcTnYn4fscvoxwc9ao6IaMaJQRdhWRX1GuGTT32+xFCsghGPRFujZTXQQs74ux6sUXB8mUcLg38ANCGZjAi8N2+yrC7CQrVyzTouUoCc0wyWF1I9A+8kAUyG+ZohpV6uNcEMRJSxAiEjsmIthlazmA85qS3XGAPrI+7YNUPdAcJh1EAkTjttfPeL6VY2+j8EwuHo/RCPjG+pP2wT0/skTa3sS28RJnArBOEwGCg+coKBccQaarxNQBOtbSyFRRowaCa5UmCKaBao8JcAaTJK8Lng/pE1PdilTFwm23i8O2No/klOaa7cTEfI4BQ1FzO+ycADIqYEhJuSMwHmWRI3A2O3q+KQVMPZ0lr3OzA4GD5/umYKQTMfl0GTiFezJRAlnhR1bvAMbHJHDdRoQZB/GshImTX3/atnSdjIBvrG1SPZ/3hgC6O07AQnHiBJHMu5MJYDYRPifoCgt698lY08tsZwLVRYCd3Rvoj8A7EES2BQNw1MB1m3t9j8wfiJNMuztKfvPE2gR1j9FM0uWoDjkDMklGR7lstiFDHCUmVs/lYyqzSMBYwJnDvZjt4nQ/R3W4t5UlIFb3kstHL7orCFBdIWBLgWzPdbXHEsXGo1upyf/OShJKRvuVnAD69BmiASvHehQgGDUxUeNrvKGfPsqU4O3oR/YJ8FI9MCbxBrhMTVUbIwPmjUSxs2jOcphdSls1gom3CSyqgWxkd0zprRoO7KPwlJ2OYBUAsq1PsncNs/cY5mrySUMCbeqjDObAPqreU6LaeAb3uAjpI/y9EakrFT3NQJLHapidhc95kftcwjQO3kioIVWczA1U4+ArDWkDwWPExCyOLKRwTLols0kaXXX08kYtEQDIwcFXJo1XPMLMkM2UArAB0UiybFoSRA2sGhGD470GIMxywAVEIyA7BuyIGAEtmWQLAPZQYTIylxNIjBHwgUm0ozdyfB5g+M7oIVUZGcoYk49UQrQ9w3vGlFohDqDSJ6/imB4L8HM2AT6RZQcWUusPXjM9L8TBOxm5TCEEvZ0xHhjLEVSqw3X9r2Qxgd9kLv/w+Fo95G9/ye/vI0LgzbsjPF9u8GS5xePFFuushQCwawr80aMLdLscIQh879E5sLI4WtV4uNzj46dX+OTsGtW8x9lqBzFIPDjZIcscTsqGwMRJzKoe+ZoePykDVqsGcqfhrMb6wR7LowZZaSH3CuImQ/loDwDQOqBc9Hi03kLogGrZQd9qCBVQrHoMnUFR0t+nGwHMmC7qtxmyygJZoFRNAH6Q0N/eUbpYeHridKCfcjmwxzIIZJVFXlkOxg29YvqkAwK9gbHyyGYD6i9WmD1oIBVHHipn8qtwAvpJw0HWXrFnMwoo7SEzT0BpAvSbHFFFlC8MWVAnEY8HIIEsteQHZvQS+srg/C/OICoHk9GThwCYjeR2dQp40CPWGphx+erGQLYKqhfARQ5ROcQ8YPGFhCg9B08SEFYiNBpqr4DExkrLD/TZWQ05SMTEEKqGslB9nkFfGshGYvG5BgS4vq2GeZUjVAFxTu/GCCJ8mRg4K6EGykyjivArD9kLhCLA3CiojqC0fG2Q3ShEGZFf0/umakpfZUewYy419I2GbCSK95pprolRMrdMls2+4gQJAxUEsmsJs5MwW4n8ViK/4jEsLuiBUwMQigi38JxlvlEwu4Pns3pNVtEtPZm2LSWlIYsQr8rEavD1+Y1k92ZOcFlekFE1Owld8zi4hYepBda/DEmCGDF7paBqheqVQvWa61eNhNkL6A23efSU6Vpi8StN6WVLRrp8q5BtOMAd60oWX1IKm91yv81WwM0J7svLNPhfBx7rWsAXYZKWRslyetUKVO/Y2SkisPxMsl4oI5BSLUGmLwCz5zqqdwKLL9TE2I7VPcWVwPJXQPVeQPXc1tXPeGzNXmD+UgKSktooWB1TXgqYWmD2Cpi9EigvUk9mCgabvVLI7iT0PoV0Od4nwyqieidRnguc/Cxi+QVSSiMI+sZJYJvAVEXmML/hcrINz6fqgdkriflLCekE8lvuX3FFf+jsNScOdM0KnyiBxVdcv0yhSflthGq5P9VbieIKKK45oC0vGOrFzkuChPIyYvaKgKC4jodBewoQCxrpOLAGR/Vch24EZm8TYyOA5efA7HVEcUFQOntNQDl7y8mMbAPM3/A+r96IyTfoKqRuRSC/4X6oltubXxPsz995CA+svgjIbyOOfp6ef8Of2Yb9jdkmojqPyG9H1jeyumYgsNFNRLbh72afZL+G3rriNqC8IMsxexOx+oKD7PI6wC4YQjb2dOa3ZB1XXzqYfQoFayOK64h8E1FeRxQ3/Ln+IrEcjuBAOh6//I73TnkVIByDnMpLbp/qx8kTboe0EfM3AdVFQHXJQbhuIsqLgGzDcK0xjEq3EYtXAYvXjtUlDSt6imsGOpma2zZ/x9Cr+TuPxSuH/I6ApLxJAWERqC49dBtR3gTM35Cdmr13qK58Yssi1Q+a4WCmjsgToB0nT9a/skmGyf23czGth2CN4DKrI3QXJ5ZJOi6zuOU2VJdhknDmu4DZOwfdRph9QHVukW8D8q2H7gKqCwfdRZS3ZCbzjYdpUodoYlGLDYOJpKMMVLdhArrSRWS7gPLKYfbeMhBuFyAdUF3wu9s0AdV5j3zjMX/VAmCqbHXB71VXiFTx4ujFrwN0G6C6iOJ6oH8wAqb2lGy6iOptC9VyucNKo7i2gGSK9Ox1i2AE8juH8n0HXXtkG4flr+p0HDyyjU2TFXHq9tSNh+oDsruBx1YAurao3vdQXUC2tdCNR35rGRRVSIJXF6G3PRnW25aBV9uBTPdmQH7dwRUK+W0P3bjpOzC/ahki1Tro/YD8tkcUAvlVS1lrrpDd9tB3Hcxdn5jdCNV7qMbBLjNKcBOTqfesI1Gdg7ltIWtS/LJ30NuOtSmp6zJqCTEEyN5BthbRSKhNSzZSSsh9R9DW0ZcoLAGe3LaQg4PsLETTQ/b8KQZHACgEZM1jIEYvo/MEk5mZQJyIkb2UzlPKeh/kpYdo+4ldFD1DjAh4LUTdImpFOeyQAnyc47qkPLCcAIQkYIa1fI2U/H96TewHxMESOFoL9D2BZ2I0o/cH9nEEjcYgekpYY6omidbxn+M/eP/B//9V/z7owfy6PuJhgvh3+e/35fG1Zix9Akw/f3+G09Ued3WJ50d3eH+9gt9rfC5OUSx62K/m+Gn/FOo8h1/XeH+3wMmyxsufPYZ61OCrmwdYvJS4OFoiOol/+dka+mkLkzn0VkOpACEj3HWB/csKWSPQZxl2IsLfZdDrgT6+KNDuc6j3OeqFgVxYXKkZwt6g2RuUO4EocvSVQVQRu3oBFD7NxEZ6F1sF78m2ZbcKrpOIZUB/O4eMgJ97gpbjgKHOIJoULHEaEL+sMJwNKHYSfk25o93kBH+CoRlDAIQC2q8WDHjREUoFeAFAAMM+g+4FfXEuRywC+iQ71RuFkFGe17cp1ruX0LcC+F6L/rrE4guJ7VLBXBqojrUOdhEhLzPIjweoFK4xskPle4nO50AVoM7JfOU3LCOPAsg2El1FABYMoF/nBOLQCcykwf1RAFqJ7FaiPY6oL2YQAsgvNIZTevb0Lsk+Hd+TbegtHP1qPgP0nYIvx5oMsoQiAG4pIDuB8p3A9rsc0Orrw+1jdiJ5ICWKlArsCznJu4o7gU5KzF5LDAtg9RlQP00hKwLoTwNUSxlbfpt6Ox1YJQEOUqt3QHsmODiOZEoaJDbEkUGsXqsEkAgSVM+B7vZbjOt3FZlVsxEp9l8hKGD2mmwVg0EILnwhACiYvUh9jinJtIxJmqxhdpSBqZZeQb0HTClgdpiShsv3DB3JE6OW3wrKUGuCj/o5j5HZEwg0jyNUTyDYnaYk3etUSVKTTZF9koN6MkyQZMkoqZJQbWKEHFOho+JAfv+RhGrYD2n2BEiTRFOzMoJhJkwwJnsgKKVsAASGgQAMGZFDYmCHiPIqefsyUL4aCD6zDVks1SafYIYp0Vd37CJFBO+X7OBT0x3PK99H+Z0Iaf8HsjUisYRjKnS2SQEoKalYOABbMjnSpnOV0qSHJQcSpmbicPtQch/TF5RPqcAyeYxYZUEpYcjIKGVbslZIMsLF64Ctlim5l9dpeU7ZoTgC8qvEWCaPl+ri5DEMqS9QdUiAKsk9Gw7gu7UChKRUVCQ/Y5uYMc/X02fHpOp+LabjUNzS5xcVATfTYxVsKWES20cQyoF4UClEpsAk5RSevrg2qom1lC4i33pEwYoFCIHWpPux577obly+QLYPsJUkULz16I4pcdQ1/9aeyATCBUzN+zjbxUm+KQJ/HyW1uo6TpNWVAmUT4CSBBxl9phmbnUf7QDOcKXkui1sPW2kCj9c99s8LqAETCDN7doQGLYCS54GfRZTSKkuWSXe8BmTDBN/RV2p2DGZRfUSWrindUP4qbUwsIF8XNHsddZveP3BZPnnzGOTEz4wxFTvKkSVE8ozyesvvHGxl2J8r2Sk61nNISzZTTrJSpguPqbtR8HNRdwcWRbfs4fQ5E5f7NXs17VxBdQGylFCWYF26yBTfUsI0YKpuYvDoo2Xy8JhQm986Pm9ZqSEtv8/VfkAUOaKSKe03BdhsHYIxHJiGSAav0gBESv5leq8vCOBiJvm8ZApwFNx3XVuIoOEzfr/o2jOwp2PfoCsVQqaSL9NP6wpGcn8GgtdRvqp6vl/0HrFigq/wAcICqrVwy3xi4uWvpYPKIQCB+y98SH2rPi2D15jsHYJRUJ1Lnl92ZQrHfRaBab5IbKCwAcKkbXDj55ki6xcipA8Q1kMBBzmrdZAJEMI6CKsRKsO0XB8BI6cgHQLFDmKes5/TB4iWnZbjz3FimkE4iS3sCKDF4A6MXYwQvZskr8I6RCUhRhlsCPRC+sDgvTFYJ0QCQ60OEth7IUCUwfoDM5lA5gRKx47JlPL6G12Wiu+foOtYJzKMMtRRZpvWM/68nwib9m+sHrkvdY0SyVeZnvubJMICEDL8dYGyf3j8Hj2+1h7L5R+dxb/zX/8HuOtKtIPB4+UWv3jxCEJGnJzscfVmhT//0Wf4i/Mn2J/PUT2o0dyWQBBQGw35vIbdZxC1xt/78/8d/+Mvf8Bk0bsMcebxR994h1/+/Bli4TE7amGUh/UKZ8sdXl0ewe4zZMsetjWIQWB+1GD/fg45tyiqAc1VRTbtUQ+/Mfje917j5794BgAQvUT5bIf2zRyffP8dXv6zp3APLWWVQcCsO8yrHpvPjxBKskarjzbY3M4wXzfYXc4BHaCuMvi1g6wV1KOGgTS5h9ga6Ict1C9m6L/ZQdxkCDmDT/yRI1M4SotURH7awg4a6mUBe+pQHrfo3s0muWMoKGeUViD/7gbt5ytkn+zQvZ8BAsjPFbqPB5jZAHw5g5slryCAJycbvP7sIWQnkH+yQ/dygfknG2zfU/tlrjXwSQP9lzO4RYR7xMTQUHmYVQ/x2Qz2KGD+bIv2l2smbu4FhmcDxE5jKu2SkUE7rYbeUJYqnIDeSdjHwyFQqNUEYY/3sD9fYjjllztUhLql7ySmIBtxRr+n3Gqmdc4d8ldZYoYkhmNPaeszftia8wyLH1wDAOp/foryT26webGihPdWwy8CTj6+xdXFkoB2p+k9vSNoN3uB/swBQeDJJ1d49/kD6D3ZTHnWwd/mEMsBYcfet/K1RvedDvI8hz+x0BeGMskTh2zVI7yYJfkoYJ8PiK1Cca7hCsqUzJZenO75AHNuYB9xGX7OZNfiUqL59gC5IwAPcz95TPVOwK4CnnznEhc/OaOHckEvZ/98oDd4Q6+RnwWYOwZfhAzIbgXsKmL2vVv4/+kY+08I/mMZIFpJX+L3W8hXBeyxg77TiJLBPGYrYHb0EvZPLcovM4gANB9bFO8MdA3UzwKqdwS7ZDgE3KcNyn9ewc0B+aMNulcLJsUuOOgb/Zb9UYRbeUBGZJca0gpUbyNufxhQvVXsNnzaAu8KFBdMJG3+rIH6sqR88Shg+ZlC/SRi9kZg+2c9sNf0pj7iOTAb+pZnryX2zwNCGTB7odE88YgmYvaCqZjV+4jtNwkcZz++wd3LNYpzBbM9hNwQgAD1c4/lZwrDapTsJYbGE4jHJAEuzyXsgsDDzRIwBycX+iOg+3jA8i8z7D7xePDPJK5/zPfQe0r2VvZMTs3uDv6s7mFAeS45WfCYy7SrgPKdhF1wX1zJ621Yssaoe8DJgv6YQT+uIhO5/RZQXBCU7L6BFKwksHjB16iOktzqnN7H/jhi/lKge0Ava78G1p8HbL4pU8ptTMA9bfs8YvVL4Pb7wNHPgeYsyVFXvAcZ5gPUT+nnDFliiucR5TkngEaf8LA8eFrHY1FeEtDWTwS6BwHZlmx0MNyW2RuB7jRi9Tmw/UTg+K88br7PgdviRcTNDyMWX8lJxioHBhyNlVhM341Y/gpoH7KqKUr2ShLYkbHNNocgH5/zWslv6ZdszjiBtPk0TYx5QO8JLLtjMUmWzT6yKlrrRGgAACAASURBVKugB1REYPWrAFuSJYwKmL+hd3Vcz/rzgP0TiWEFzF8ynVUNkbVTMzHVQLVngutM1VpRA7M3YaptEpEMnZ3Tr4mYqpBKgf5IoHofpnopU7OHUvVAvmNvJL2Iad9agjnKtxODmKS5oxdymAu4mcDsfUDzQE4S5CiBxWuPzTcVFq8CujWTYYc5/aHZngAIYJ1Pcechh4jmoZ6AvGkIlF1Bj7kvBRavXKrIAvaPFWbnfkqC5bkXWH1pMSwV5by7gGzDGpn2WB1CfhpOBuwfK5TXEab2kzc1KqA+UyivWd+0eNFh+0mJ8tpBtx53386xeGnRPDST3FXZiPqhwvpXQ6o4Ym3R/J2FcBHNmYEayIDbuUS288juLPbPi+R59RgWagLFIgK6dpMktT/WKM8HNI9zrH52i+1318hvHfojqoiKK4vmLEO29cjuBkAKtA9z5LcW/ZFB9aZB86zi72uD/NZCuAC7Mvy+8xGq9YcQHAFkr+8wPF1zH+9Vt/hcTnVB2d3AFGwtoGpLMGT9BNYItiXUngmxqrZMva3MQbbbWvhlzt7LeQ616VgBU5mpRmYM6pGDY59lQ5+n3HcIVUG5bJKrxjJjTcqsOIDXJEPFKDUV4lBhEiMlsyFC7BvE5YygdlsDWjEptk8SWkfA9wGou48JlAT6gYxjS/ZclCUBaT9AZIb+yLo59G8qyZ8JbE4M5AhMx8fIZv5GYM9vR40hVaj86zx+LzyWJ8/jD//u795j+U//uz94LP+NH9YrfGd5gf+1/Qa21zO0bYbj0x3aPsO+zWFWPf7x59/Ew9MtmrmFVoG1DVmAP7II1yXkckDsFf77v/oRAed5Brfy0FcGv1RniCZAbjWalgE4MQ94FwTsLoOw9ByqnQJExN7PYG4VrAlofQHRS2Q3EgNyCBPx8vaIQC0AcWnRXM6gBoEvXjxE5gBzYWCPHcQg4N9VuJMVoCjjlL3A5uUKwgrsZYnyhaGvzQC404gqwp1XMK1gclor4E8l3KmHepdTprJTiAbQbwzTDBU4qyaAwVcpfVPAmgD7+QIm+cgQBULLgaTqgVXZoTFLCBGh9hLubIAaFORWw7UKeSfgFoDfZjDrDq9enkLvyQLaz5bIOoHdixWyncBw7Bmc8qLCcBSQ3UrYU4HiRiJsJeIFB/blG4VdMUPRCHSPPPSFht1p6IYVEOZOMiGzzlFccvAZpUqBFwK2VciuFexKQllKH5t5jswDy58b7D4JBH81ByPDOiDbSPSygPSUNYoI9DGxWTuGgkQpWenQaPoOHbD/6Qnit2pkLdD+5BgZ+DpTC1grcY0jMshBwGwk+ixAdwKiTtJGL5DdKty8fwSjKeV0FRDaEtIAweUobiVckaoX7gzMXiC/yaZ7Q3UG7lZPtRe6ERisRHatIHvAJB+jdIDcAf03A0IWkb0zKC4E9h8TVEoHBjMdkSlHo6H3DNeRPQAp8ebLU8zuKOm7+y4lmsM+9evFJD17o6Bb+siyDQNDRBDY1wVW2wizkQwcekg2ubiOiL8oycQFTVl05OC/P44oL3hd+tzA1Imx2mgyRA0wey0ZGiPENBj3L0uyT7dA/eUCygoUl6MPE5i/iuiPgPJcoB/I/Np5xOyVoBRwL6FrQM0B/6acmDKzj8DbAnovUmIoB6TFNb161c9zgouNQGhy+Iosnd5LBth4AXOrCGbuOPAiu8ztmr0lg13/xTGMYErp2JnIOo60vB2XJ0u+npU0BzYzGDH18wZN5tVVEi6xcvmGA3T5eQazi6jeMn0ju2Oip7Rkbs2GbKrZkSnONmQH81uyv1Ew/AcARJSTP04OgI6UbMuedQy+oK/L7ATKSzLUAO+xfBPhisTCxvE8pjRLx4Aks4sABKQXyHYxgQIyiCIQ4NHLmFj3Db2pdqCaYvU5GVs1JOAiBfLr1JN7G1Be8ZoNmUhJogyd8fkB6FA+Pkp9CZjKm4AogdYq5Dc8pwziAdTASoooySaVFzz+xSUZ03wTsPhSkfUVKUQrp2Q4ZJiChsyO3YzLL+nzG1NqpY2orgiMTDMm/x4Sa3XHrADpRAJySSlxR+a6O+axjIJpr2bPbWpPFOavqcYwTYB0XD4DiyLKq4BhTu+d8JFS3pQuPH9LoEeGl9892TZOSbrFLQGnnfO6kE5Cd2ECBvk2wOUisakEg1EAuh9VFgG69SgMe2flEGEQ4HL2ztIjCMjUuzr6tMsbx4AakdjWIKAsPXumFlPirrIR/VKiOiew80Yj3/gEIgMZzuRTNYK9q1JHXuOZQJb8jN4IlDcOQad+XJU8po1HtpPINqkfMhhke8/Qni5AVhLGk0VWLVm7fCeY2jp2dApg/g5TvUwwQHE5wOcKeqnSNihAkR2XA6XS2Za9qMWtZ22MZxrs/D1ZXJuqR8qa0lfVWuS5RMjJkuZ3qR5p8Mg2fpKgVo2fwnxEYlhV6yBdqirxZNLdqoDuAv+2UInVdSgvyW6OLGJ+J6E6h/wWvE9vLeQQkN+xS1R2lGsGIyEHn7zlCSgqCWQEf7IhGHTzbGI9eW14qG2PMM+APk6hPlMHZ2+hW8vO0N4CSkB2A2JOVlN2SQqqBPSmI3u87wkCtYSqh3s+R0/2sR8glCAjmYCXsARi4j4LCUD0dpLLisEe/jYyhEoSMPrAWpKRMbzXWzkB0fEnB9KUrqagnpFsIruo+Td7r29nAoThsH6ZQK4UTI1NyxfAJF8VYxjQ/ZTZ+6DybwAov85E2P+fHl9rYJlrh3968TF8EPjBt94AAH728+eAAJ5/cgkfBf705A2+2J/gYnOCbF2jerqHUR5VPuDydoG8sGglsFrWyLTH3ayCfzvD9/6tLyER8dMXTxFzj0+fXeCrq2N4p1BkFq0scPTRLWIUrBC5KVAdN9APAvy+ACLw4NvXOM+PkK17hF5jVbWIj4B2n6P8vID7wR6zxwO0CrhyKyzP9rDvFyif7nEyb3C9r9C+nQOPBjgrITOPPLdo6xzxj3fob+jBK09aModzB/OsQ/diAfu8B5yAedDCuQohC8A6Qm0VVj+8wdXlgiE5EZBLizx36FcGsWHdCT5qMewMqy22GqHykK2C+7jD24s1Fi8k9icZwspD3RjU3x7I+kWgexQhWwYB/cmzN/jiH34Hm08ZQpJ/Y4dmW8AUDuJxAG4Lfph91EH9qoRdRsBK9McBYW2Rv8pglwHhIatd+mMySVEDsQgQO3Yo2hUQZ/wgtaeAujUM1ukEuocOwgkMDxhe48sAv4jQ5zl8Aew/Dgz42RP8tI8YQBNVxOLjDbreYGgNK0cEZ2jdA8u6kbnD0EtWXzzo4SITaA1YG2Gf95CXGRCYaKt6JsaacwM3Dwxm6SW6jwaIWiG/ZI2KrjXsj/ewdwXKC43meWKbT9g72T1i9+KwjsDKok/Ms95SvuQXHnqjDj2GTkHNLOxCTaEnMZG9vozQb3K4JU1tzeNUK5Bj6vNTtcJw6qHvFLoHgSmvrZwSaduHAUElRlLRQzv2QprUYTnG/7dnAW7GpFV/naM9ZZ9id8qJkqgjuhPKll1FaS4EQZ7qmZraPCZg8GWgbMsACPTy7Z8zAEanrk8Ak6SxOz0E3ogkuw15ZMjRmZjqMJiiGjB/oRK7wMGsm3FQalceZkc2tn4KyEGgfeZRvFcQDujXPK6zVwL9MRlTu4qIIPuW30g0jwLUQHld1ED1LuLu+0zf7dfcxmEhpoRbV5F164/BJNiUaukLTlbpFqkfkRJKETFNHBXXAuVVRHci+HpB9ilKHg9IoHnICQxIoDtl0q1qJdw8wjZJ9poSQIdlko72wO5jAsn+hJ7ioLmNPo/IdmkCI4X1CM9tDDmwfyqn2gtfCHTHArYigyn7Q3ei2fPcDcsUmJJSZl3JRNdhdfCWDsuIbk35YHdEVmusF+Hrktx24La3ZwROo1eVHZoEWrYS2HyHTGgwQHZHEMvKitRHWRIY6UZMybRMFpVT5cnYIdmvU0qqo/9vWALSslKkPlPoT8iaCy9Zc7IFK0A0UF4GtA8I0ss7grtRWr1/KpHfxWldUQrYSk6hQGPCa3ciqUizZMzyO8pq908Jun0GJnyGNMGQwLKdCfhMQXqgT8tsjxV0F9GcsXe0W8mURpsCk1K3YVQEZe0JJx2CFkmSyuPUnXASyeepg3If0R2pBOIlfM7Jlf0TieqCcnDTBNiKLLI3iY1dSaheol8JrL+waB4k//xwYJFtxq5GgODbzgVqowlaK1ZXBJVAe03WMN9Qajl6H/s12a18E9CvNatNCgGIlJKqyFiKpFqEJugdwevo3XeFZHrwnUN/rNnhaNnfGJOU2WcSw0xC9WqqMAlGwC6zCXDauYI3gO45oVGfqWkiAAD6owwiRpiW4TwMaErhN4UiKB0iXKn43pbqDeQS7ZFElBrzL/dons0QMgllA/qTgoA/AtlmQH+SIwpu17BUtBnsPJoTsprZxsGXEq5U9CoGVp7EHe0icSdS4qtOUugIX2h0p4ZJs7lG1AKuYthOd2JQngf0a4PisocrFbKOXsGQ0cepeg9hPXyVpZ7KiFBl/L6bmcmPxioVyr2FlwiVgXABvjQEowCkUfC5ggKA3iIUadgsBJepKL0NhYZwgb7LVGcCJKCYG0QtIduUqpUescopr805KTyuE5lBTGmpyAwwWKbFtj0ZQqMJFq0DjGbiq/NMfHV+CtaBVgR4ziNWBYRLgFVJ/m2wgNYpwC75ORPAi85Pyf8QgkwlQC9ljICQDPq5z3YmMCtnFWLTEggKwSCge3/nCj6UzAqlfrt/8m8ol/19fwjg98oT+bt+fK2lsPk3n8Y//i//E1zdzZncKoDTxxtcvVpTFplS8IQTBCE9A0hUJ+AWgeAHlD1OwSqOnX8QEfnjBv07MnlTjH7ymAkPRh8FMjT9EwvRKOiaDJqwh5l33Qj0p+nmSb/7MiK/YmLk+GUTMrJLUQGqZYqjCOlvqSZCxDRg20iWqzecvR/W7JKLgvIz1fCLN2qG+ExdUI5fmsOaVQtjtLrqMXnz5DCmBCYvnUXymgF2wQ+dUAYyqYntkZYesZF1dHMOMocVE2OzxGaYmpIuu4wwmySLWwWU7yXaxwHFpYRNlRIiYpLiAkjLoYyN55ED1KDI6kWBKTY+2xA0RJ1kXqlEfBx0mzolUm7JvriSQFA3AmZPYCAdl9OdJN/Xva680eM4+tRYEg4EzTTPYCLlmnP6AykZ4oB+7NyTFhjLtaPg8c82HAAHQ29af0Sf5FjDMPa72TnIQjeURJo9rxP6V/k6Wx18Z8MqsVepvy8opGOTvJDHXJZueK7tjAPbsSS8ORPT76o/9An6nKBLNzwedi6m64nXCyYWy9QpdKbG1Nc3DtjHrryouN1qYJl8f5TO8z2mBjhUKUw9gv6e97BPXW754Zof3z/WQoxpmtkuTr18Y4H6ONs/xeunFNWxi26cQBmPtdnTt2VqMl0yefVGaSI9sXHq/qN3le9jP19Eto/YfCJRXsRpWwEmkw4LMkDcmcOkwJiSGSiYYGfgHQe6uo3Jn8hz6TPWN0hLwGTqlMJoBEwb0TwgEEDk59p4HOyc6wOQ6ji4vb4kc0c2h+cs23Hdyo73whhgcjhuY48jB3RI1QxklRAJYsbSczKWArqP6JfJ4+lZ2TAlpbpDbYMaUvVGStnUXZwGwOO5VH2cugPHcJ+xE3GswlBDTOeaaZ5jCucIckzD8ybtYXljJUW/OPgPxzTOyScYI1zO42PqgH5JoMMaBgKmYSGnz5eQQJnPkrdWk4GXPk7VF7w243Q9jp8lY2pr0GI6NjqBBlZkiEkaampKJEcW0FaUctqK4Acg2yQCgRzPH9clPJcZNIGNzyWvuzpMIHN8z9h/yGv2nnQ4AXXgsBwAk8TVZ5I+zFTNwfuYfxvPpfRxSs21cwVTc8PHKpYxRXq8r4IRMDsyf77gJOjYvShtmJ7j9yW7FfsjDbMnQ2MXCtkdpZ1q4HeiXSjoJg3KBT74WzDcL35WEUDq2sMXCrr1077Tk5rqPBy9mJCAK/UktYyC+xtM8jKGdL0qAV3b1Ok4VoscEmHphySjnF3UcOuCQC9Gfnf4gzyUy+R58AWBhxwCfMG6D3YtSrKDvZ8koyHXqSPSE3Cl+g43M/SQDm56DUJENGraJ9XRRwkByNYBYypwes1YASJ8mMAcr6EAP8/ZF5n+JjuLmBvAjeDOIxqFaOTEbI6dkWJgr+SY2BqVINjzMd1bESE3XH4K3BEjU2dTVHeShE7y0uRpnJjCsSokpboK5xmkEyOQGaa3qsTmAUx71ZTditEjGSOlpVJM6a88QIkxHKWx43piPPwNIIiUTLCdklp/jTX8jX5KpeiDvM9YyjESPK3rfqekkIijB9P7yUd6v6vy1x9TquyvS3I/eNGHbObXvcdyfvI8/vDf+c9+58v9J//o7/+/vm/A1xxYLr7zKP7ov/hPkWuHVdaitjku6xmaLkOZW0gZcFo1eL9bHBjK3MIOGkJGGOMxDApaB9Z07HNkpUWZWxjtUXcZhp4j4Lyw8F7COwmpApzVCF7A5I6fL62BKRxsa6YaE2kCwsBRn87SF10UcL3iTNA4VpQRwSrIzPP1TlCyu3BQhYPvFWuKMg+3MxB5wMOHG5y/ZYclrIRcsHtS6gDfaMBSGooA+g6thK4cQhCIQSCmHkkkFk4U4wgCwNYQkFYO4v9k7016ZNvSLKG1m9NZ6+bN7e9rIjJeZERkZGYVKokpFAgxYsKgJAQSkxqVYMSUCXNGSIgBA0aF6H4AI0CqopCKQQKVWRmREfG6e69fv95Ye7rdMVh77+MviSATiBLxnsok1/XrZnZ6O/atb61vLRkQrETw/FKHAESrEBoHcYqWfDIAS4vQKkAHiFEyviQQSIeYgYmKJkhCe/h9wagWG51bVQCcgFwY+EEBVgIyMKqlV4xU8YLZkEcJfzVCPJTwc0aApMzBUHmUm56xKIMEUgTHyOYCXDTMSZEYTgCbEeKuJMhuXC6uRcsbaHFg/mIQZOncnE0JX3nIXjJKpaPbZhCTwZJZRDfZuG4xsrEhe867QQC+DtA7Gdm3mGfXCdgLS4nXVjNLdEe5c2o8MFogIFQBxQNnCVMQPEBA7hqPYidzFEzKgkS02PczDzFEsOt5rnwZAZCfGgoIkcUrWewyozEgFGw8qI6NElcGVA8Sw4WHPooov4xFuxMYLxxkP2VUJqDrmpCLvuIgIQwwnnvIYSp2M5gLyMwU2QiBceWhRsFmShUyC6sGMjHCicxopTzIZCw1rjk/6Ev+XRoR5zID5m9otPT4PXYeGyKxgZEAcjKkUpFddVWg7PwYAVQbXxulwSzG2cxobsjUAbFYt4BdgMVeZB1lBMf5nhHBRxDT33OzCwnAReZVT6A6rVf1bEZJS0m36qYojAQEqy1lkTSlIshl1t/UhJEp3P4RYwpEsB9iwyk1GWLDIBnl+HJ63taTVJNsWmxKJGOf2IAwKwL3oCkBNguRozzsjPsxrpGbMSk3M2f0pWu5ICvqi9jQ2U/FOaMc+L7E7LmabLbuQ2xgTY2QlC2ZJb8mNjI8otSZq7ZzEWNRuN3lnoBYuAi0o2uoq2LwvJyus2TWQllsvIaAHGuiBh4PgMd92AgU+5DnJHMjS8XmQE/zHFciOq8SwDILMjBqJTYYbMNmSHEMNDjrpkaNGtlUSY0Y3XFbXcmGQ9r+YZ3ALeAKoN56mDlPStF6jPF33ZPdLNopNzOtA5jAobQBygSMc5lBa3pO9zyGlLemvMtkoiRga4JpEcgusiGBaErE68A2zKRMICuI1HwIGJZk5dLyXTllNuph+nvO4BTMwEzGUOMijhgc6JSajKyKo4eJuZn58xxBezL+cTlL08ft5HyjnUnIIeQYoXSvNHMZZaYedq4omZUiMnQhN8cp4WbxzmtQxs/PBGZdreALrr/Y0+THlQTP6fMgB5+bcOm7yJUybxfjS+jM6op47iLDGGQ81jLKWWNUSPoc6M7F9QjGf5SKADBmWorEEibwEhsujAMB5yoj0KNc99H/jYMcHeyyghxsbsR7LSGjYQ8ErylhCPYTaBLGxUiRKGXNgI/s6+OcSl/r6Cr7KE4m7kMGplFSGgoaOQljo8R0YvOE8wSmiY3UamoG5PgRmV+bgW46LolpTBLW9LcEOh8/UkRIYjKTNPdx1Ih/BFLNN0FfSO99tIz8d0xgMyQp7V/1+A7kWH7XgaX8q1/yu/sIADqjsSwG/Nn7Z1iWPayXKAqHU1diu5vjqjnih5c3uLlbEUeNGqYrMO4r9F0JpQK62xn6tkTwAs4qbO8WuJydYK3CxdkRAcA4Kvh/uoB732A17+GNhJAB475CXRvAC5iHCjACflC4vDygqgx0bSBUgB00/NuGdWGvsFx1+MNP3iC0GvNlzwgRGbA8P6FYjgRd2jPTUAWEGKchRolyNkKIgM2TA/MXVUBZWQgRoLUDRgm1HlFcdsDKQDyUlLPexPiKXYHmi2ker1iNCIYzIcWXFeAEls8ONBkpLfS7EtViILAbJMSaRjih8dBPW8zeKIh7MsaiI+isr3VmNH/0+1+jvFXc1nji9F5Bzi0++sF7qFZi+eQIqAB/LFAsRn4x9hJhlIxUOWgayJQebuUQRgVfevyb/8I/RvHyhPCyJ8AFMD7U+fcgAkRjgRUNjhBAZrqT0Ed+4Yn7En5tERaWIDU67cpRQFwMZE3nFtWtykx0KCg79UtKc/3aYvaOz5W3Cu7TDmoQ0FsFKKC418DKwDcObumjfMsiyAD7YozZegLQNA/RD5pgUgaIIYLYUWD1x3cwTw1Z9pKAWUZGWhqCJXhmWsqLIXbduV06RmrAk3EW6xGhIADTr09wC49wOZJlPxIIujpAgHNu1b2A2xi4MsAt+aVfRBfSACDogPGMgNCsPcYLh/ouMlCS8mg391D9FBJO11M2QIIAXBnQfWQweythzhzB1xOTmXE7Jwj1JQu4ce3hZx52SSbErD2Ko0D5IHksgFiUCAxXDsUBGSAiPJJp1mSvZXIcnXlmA8boDH3ibG9iel0dMJ6FHH1jVhH0F3T4lUYgAOifOqgeGDYsNIdLj+GcYKP7bCDLv+L7XRMZDU8gmhoIrgnon3ioDixS5yz8zXxiN4MOsMsks4wOuQJoXzoefw2cPrEYVwHjxsM1lKYjAMfvWc4kx2ZQ+8pBt0D7nMtRMc8xFe7dU0+muo4NgajkOHzqM5BIMl3bANXWE2gM/OwXR4JGFZctbED7yud8SWZ70k0aALondD7trwhk7QxxZpAMrY9FvTQEXc0Nz01iWoUlsCDAD1BdQPuMMuu0Pl8IuIYS3sS083qbmit2Fp10Bf+emwNzRNdjsoJFG1AcaGwybASqA+dlXbzl9udRxhrZ9f6CQFL3AcWR23h6gZz7p4aAw/ci05eyD1Vy7Q2Yv/MxEJ6AaFhH1jauT1pKgIcN7wNqIEDpzymLDkpgXAqMa4Fq61HtA4Y1We/uiq/Rp8Suc78Jiqlm6C+i+cwrsuv8XFP6K22IDYWQ42OkBboLMps+XlNcRsDpOb8jeA3zeFJazPUGQTdbX4gMZkRgfEhi6VPEhoqArTsna2xmlOqPS843moYNPa9EdB7mPCQEXYR1x2uLMt/I/PZk+rsLfnekdbhHoFQZynltLWHmAsMy5UMGDGsVQSvnJ3ldUYqZnGLNTGYjme6CLJorBEFi5yPzOZlmCUfZLCQwruhU60uB4SzmQw4e/YWGmUnYRmawyAPN9wgX0F3SmdcsyYrZmUR/UaC7LOBLGR3YI0hdKAxrlZUUh1caciAw8IWEayS6ywKqsyiONvvr+Rg3kvIgh00BOXqYOVlMs9CcyxTgPloP6TzK7cBlREZRGn4eXK0wrrmM8KgJYBZkbQFQRlsTeAnrM3iVx5HX40imFSFw5jI2mIRx0NsW8jRAtpxVhRR0nbUeXkuIweQ8T+E9fENZas62bDlr6UudgWIoNVnbIc5kSsllJGCY2McEKhXZVEg5zV0CEbgyliSD5AhQg4rrjw6zCYQCmHItlZycXBPgSwAwsqOQEnAO7tkGwXsE76c4kqKILKOb3Gd9ep+YjHsSO+o9wWQC5EJEV9jwTaY0stm/8ec78vgux418qxnLs99/Ev7dv/8v4U1/hu3Q4KuHM7T7GtV8xMvzHT5/d4FnVzs4L2GcxGg1nJMotIPzEn1foKoMxqHAbDbAeYnTtoH+UEB8fMLZssPtX1wAAELt8Mn3bnAaS2wPDeyoaV6jPUIQ8E4QADqB9brFw/WKLqNbjfBkgB8VNlcHPLxdY/7khHZfQyjeHJV20/IKB9sXbGZZGSuLAHQK9VWHsddQ2sM8VBBOIJSez1uJ+qJDf9ewyq8dqvmIYV+RvWwcghUQJ00WrXYIg4QYJB1QGwecCGSWzw84bmd8vlfA2pCNLAIB19wi3JeoX5zQv5tj/uqA4/tFlB8T/IaFhYzHBocCWBpgWyDUPjrYdrD3NULlyBoGgj0EQDwdgLc13NJRvhzZGTzvEa5rhHOD6lcV+o9H1F+W6F8wR9SvLNSW8w/CiQgOCDCCYOEsnIBfW7q8bkaI+5Js59wjaA9hJcQgIpsngTNWp+K+JENYe1TXGsMTh/ID5w7VUcFXHqHxkEd2Lf3SonpbZLmtWZM9BADz8YD6ZzX6J47rjtsjArMau+8PEK0GHBBmDmqr4VYOxa2GmwXgYoB8WyMxjWbtIAcaPBVHSrnNxqHYqVj8B6hWwl4YiBONcPSJrxNABnrjiixk9UHCxqKewMfDL3guZB/lcYFgs7/0cEsHeaJJSZI4d58YiE4CCqg+KNiawfVyBMw6oP5A99Xx3KF5pzGce+Zqzj3cwmP2hUb3ko67QQMiMq36OLlsBgH4ii6fqudy9WmS4vL5CBpPAsMFJeJq5D7pLrKnJTv9iW1MDFT72kHvZYweAdrnHqtfSJxeEnDSfIcA7ow3kQAAIABJREFUYDz3KO9lZtyKE8FkuaXk2px7FA+8vl3s7+iWhiztC89syVsBsyJYrG9ELqR9jBQZznm/qO7kxFxGEOrLCXABPBaqFXCzgOZaZCm7L4D5u4DuMs6oRQOo+pZGOMePGHPhNbd/9Qvg8ClQbgkqhk1AfS+gOhrlBEVwAEEGtLkWWXbIsYHIrmue88QcmgUB4HAuMhAuIgBTI6Xo9S0ZSh+ZsHHF95DRAMYzSsfNnPsxf8OZxOLA45EiT+yM14edc1uLI58vjgRFyy85v5jYueLIdZR7mhJBJMY3MoYRsPmS225WQHU/McN2RtOgIAlY2xcB5Y6xMwjAcC5yXqXqA3wpsoycLr88Lo/jbmSUxCKyUa7iMZ+/IcAjQBFwDXKeZlrPuOYc4/Elr7fZB4KafsN97J6K2HCJTF88n2k9s/eUW9qZyA6z83ee18iKM8JJmj4uY9brnYdtCFjLfcgmP7pl9Ex3KVEeKLGXFlh+6TCsOG9abUNmu9iciNFPgecUoHurLwlwpUOOpbH11LRK11piFQHk13rNc1dvPfq1zMA3gfPi5NE+VVDdVB+VJ0qQq72P7KxHd67JCJoQz5HMQLOI5ktAkn9zvdWeMuNhLbF4Y2HnErr1OD3TWH0xwjYKw1pSFmsCyn0EoY+YSldRxql6Os9WO7J540Ki2rvcfEBsTJyeaszfW5i5RHHyGFcK5c5G0CVRbS3GFSWvKYPTNhLVlut2FRnVcmfgS0aOAJRbJ8a9OFqM6wK+ENCti3Jukc18EhBWvYNZF9AnBztTKA58X7kzGM8KZn+epr/JyPYN5xXKhxHjeYnyYYSd6ezES0MjzlgGSTaWjOeUXyoGB99oxocAkZ108LXKLG5111MWKwRUZBaDACDFZM6TgGWtGTUCEGx2BtAyP48Q4GYl9P0JfllTWusoyZXtSOmtsXkOMpQaoo0fpMcGNyDA9PNmcnONs5fi1NEldjAIdUlTHx8oAwY4e1lMVipBPsrLBDiXaRldIpSM7GI8XnpycQ3eE0w+ks4GOxkOZTmsENPvcTkhZVAWxQRgf4Npz18Lj3xHGMuf/mu/fcbyH/393w3G8lsNLOc/eB7+9f/i38Db4xpvb84QRgnVOLhRQpUeflsSeMkA1Vi4bQk5ynijCHzOxS/rKMXUMQZADQLmjAV7kKmKI1MF7ckwjjIXchA0Eim2CmZDlqD8oMhuVCG/V3Y0JYEKUCd2Ml3D+U4AcAvGeviSToXu0b8ismh25cmeRfv7VMh5RQbALunkOFw6yiTTazSLZ2H5u0hzSnEbfB1QbCmrlIOAwDRflSSHqeDwM0ag1O8Vuu8NqL+o4OoAs3aobjRcjDqwS4di00P+bB7BHcGKWRFIuCpEmQpy95nuj2TffHy9LwmWmq+47PzFmUYKoqTTNizizDLkeUAEvl8NAramhBKIzq8PivO1hkYmdhY455mAVcfrIxUrrua6Vcf50CRtNHPuR2KcVMcCK82s+mLK5fNliF+wgseyooQyMUJmRSao3Ip8zORIQJJmIKWjmU31IKIsFnmGL3X3haNkURqCAjPnNqQZPduwMCdDEzKzWRzpvKpPlEm6Bugv6ZIbRMyFFPF4VKlIj9dgEWWW67TflJCm+S8z5zoBvm9cMb7BzmPx7qb9cE28NmPhPM108j2umuSTwnJOtNxNwBBpG+uJzSr3yOwTBOWVdh6/vN0EDNKMYfeMUlXVR9ml5ecssUnSUII4bESWeQqHDL7zXFfAJEWN8lk5kp0aLlIHGnneV7fxHFoW4uOK+wtwH0VIMsok8WNBnOZB03xdklC6SuSZwsxAyni+RsrsOL8dQ+V3EeQNERB001wq5cVcri/F5NQaUuHIv8Mjz70mthCI56KeZkzTPYWfIzqm9ueSIMRx9tU2EUQYZNfSLO8NnGUtd5Ok2sxFduRM+8nQ+PQZRJ6lTTLlPFMb5bdqJFOU5gnTMtK8Y3JVTrOZ6ZpLs4zCEUjYaPyUmhBBAEVHkFK0nBnVfYjB91zvGCW+6d6eZOW6Y2ZkiLOIxYnMqK1lBgXpoUy8duYiSzTTDKYIyLOqQDz+jq+3VZyxE/Fz4HkeXATSrhSot4zOEH6aMQ1yYmXVEGDmZN5EiMY3K5rQSBdNmEYCPjKGPs+SPnaDBXgMdesniXC87wYZcykTaBx5/s1coto5uGoSZBHcTJ9JWxNMqjjzmYBakCLOgxN8kWGdzqcIATJGbFQ7h/ZKo76PYy6RdcxOqBFQ5TzjVLB7NguAKL/tPKShVDWb16RtF3SMtbMoi4xsrPABw4bgLP3fq0csZgJxMVbDNiozompwsI2GPvHkm5WGjjOp6XX5Mzk62LmO5zHkv7loYJPAuxo8wValJkY0Hs8kVZV9nLMLAXZGNtPXzMRMs5qhjKCunUCdHF2cqY3zmFEKm2fypIjXgge8h5tFt1cXOGMZZatpzjLlVub5Tp0ko49e78HX1xqij/OeaRY1SjpDMUlxE8sXSs3lJ0bwcfxGYiKB6e/J5TXNXKZHkon6MElU/zIYy7OSUQ5cFQSSAMFkdmyNTGbX83et+J4IDBOw/MszjiJKaTNGSIBRR6DqHHLuZXpOCuZiJrAYtzFY+2vnKx8/QgiTLPb/Dpd8F4Dl+T8jYPlf/m4Ay2+1FFaIgFI6jE5BiABReLhRQqiA5aID5habp3tsrg7wtxWefXqHFEQOL1A8aKDiB+CTT27I3r3u+KW7cUDlEWRAda8YITBzkAuD2aYj4CxCNpfxlyNNghQgGofiXsEuWAhJK1DsJTYfP8Cd8QNYPDB+w1UBfhblF5osCEBZIz5uoTtB0BXXFRQoYZSUzuljssEH3NzRsEdGm/rzAWbjM7uheoHmRmRwp0aRi37VMe8xSCCUXHay0vcli63qTkI966Ced6ivNbA0NBg6aBZjATh/tUUKnFcdsxhNV0AaQaZjoKGNfNnmQkv1IoIfwSy7MwuvCPJ0R9MO4QFRsxM7vhy5XZrzfnbOgshVAXbBTnrzXtBNNPD9+iS4zDOH4dLxOJ6PqG/5nK0DZXt3kXWquF1uFtA/cxk4jZcOiy/xDYOaYh9B0hMLaYHix3t0r/m7+sk+B76nWadx47PbaohsF2e9KIWzz8csyQyaMRiuBsZzh3ETYBeUaI1PLYIE2h/32cBmdh1ioYTsWAnwueaP71nkes6XATyvxYH7niSmCZCWO8TGCMjoCuTron3poQbui//sBNswf9AsQwaDbhZgViws+ysPVxDYueqRWdRHfbS75/b0TwKGixDnpKJUMhluJGnfIHLxbheBDI2NplqxyBdhAroJtPRPHGfcbMDpE8tzEIEeQXfIjIhwwP6n4yPGgdtdHLldKR+z3HEZ/VMfAU6AXdCd1sf5tXFFgOoLglZbE3gHjalIjqCku0pAmEWM7ujQGiTQvXLon1DKpTsyW2Qsuf/DRYBuuV3tU0ps7XwyC5pdc99mtx5qiMcvusgOG8Z7CAv0VyHmM1ISmLMTQ5SIav7bX6Q8R96X+nOyeeUhTOuuIvt5FtnVdHw90Nw5DOciGz7pLmS56XgGzN+77LTZX4XskJtmJ4cN710AI0HUyJnQovWRXWR8SQJ4PjqIApHdqwXNlQxZPQTKOc2Cc3g6NhOU4TaYhUB/STBmG4HhnNLy0ws6iPoS6K4EukuJautRHjyGjUD7QqC7EjBzOvGaFbexfZZmyoBq69g8qHntdk8EkllUYifHtcDxFTP2hjVfo0bkWUXdE6iz8cTfXUHQ3Z9J2FpkUBniPUKakLeLjq28JoLiMeguyVZJE+LxlRg2IkZ/eHSXMt4LeV7NXKC7kJhdj3QpfSmzwY4rgOLIc2OaJE8VOL0kYBzOBPqNyE2RFKFhK4H+XGWDL2kIrton0bzGESCq0UezqAB9ctAtQZqteSzJZvK9BJeUeyajHltJ2EpgWEnG8FRkXIMSMA2loO2FQtEyLgSRxU2A0tYCciQgNXOyer7gsn0psrGQ6h36M4VxSWdXX/E6HqIk1VUS3YXGuKLcVJ0MzFzxdS6g2A1Z5ukaSldtTcfVYaNhFyo2fSQVRqOHbWi0Y2cSrpDxODpI42DmKoJFRSnsiu8fznVk/mg2lMCfag1cFecilaATayWhdx3GM432eYUgBU4va5hlAdcoeCWg2hGqHeG1xLjWkMahvyigtx0dXx3lpS7KVsezIn5HUarqZhrSes5WBmA8q7iflQKEgOgMfKVh51G1ZBybvRFMyu2RRkQRDJp1BTgydHZeYDivckXsk2HRoYeIxjNuUbF5U0ZgJSUZPyEQSk3w5gKCUvDLGkFK+FkFOA8/q+DnVX6dGA1dX0NAmNcIhWa+5COgKQYDGEvpqqOrK6K7K6T8pqQVIHsYwWTQKoPYoBVdZKUk4FTxuaqc5i6BXw/8EiOZjH6EgNCa0SRKQShJpvORmY7QGiKypEIIgnohJmmt/KthR5LJ/trHX2cO81vwSAqJ3+bP78rj281YfvY8/OF/8u/g6ewAGxR+sLjBr04XKJXDh26Bq+aI69MKtTY4jBXWVQ8fBErlMFgNG9ttzktcb5f4g+fv8Oe3T1Bqi/NZh0YbdLZAow1OpsQvf/YMi+dHOCcxr9kxqbRFOxbYHxs0zYhV0+PmfoUfvbzGfTdDoRyMUxisxv12jsWih/MSSnpUhYWxClJ6eM9tqUsDJQK2bYNxVKhrg2U94NBXcX0O3VhgPeswKwxuj3No5bE71lCKM5Y2RqIMRqMqLE5tBYgAKQMuVicYp3BoayxnDJkdjIaxCj7OmIYgoAsLrT2qwuD+donzywPavsLY6/i8gw8C5+sTPrw5Q72Jy3o7R/XihH5XQRSeBkZWQhcOZWVhjIKzCu6k8ez1Pe73c86Wznrc3S+QJFC6cBi3FUTjoAvHedZdBb0wsK1GtRpgRo31qsX2foFm2WMYCpofyQBdWti+gFAeQgJSelijgIeSDYLaIuxKYGUo1Z1bhFFCVLH77PilnJjsYm5gRwVxV0K/bDG2BXTlYAcFVTl4KxFaDRQexYcC5omBqi38bYXqeYv+UGF+1qE7lfCdhig9TYxaDXiRWXB9NkL+smEUx4UBBsWGQk2zpFAm9jxAP2jGnrQKQQWgpjlS6OOXgfaQRw3fOM7stnTcw9JCaM/9X9CyNc3wAgBKzyiVMkmIJA2MVsx583XqRgaog0KxlxieOITGQd1rRp1sNdzMZ/MmObCh4ev4t5JmRsVOYryM29fHDmkRAMHZUrLZIro2R7Y/VcaKJjsicHnFrc7S31ASAAsryMTGhpI5cxBWoLpXsE3gzGcbVQmBEmjZCbgzi/qrMs89Zhl1SXZZJuOTOjZgTjQwkgMdpxFjTthQEJkpSYyXGgSGCwcV429SQwkCEGaSy/pyMh0q9gQNw4XPRizSCLjaQ7eSnlMGk2FLyZlVCKDcMd5FWM5WqpaNraDYVJCOjQ194syrWUb2akZ3Y19ShpzmCn1BJQVZSb7PF4hxHWwIZeMZwQaG15P0VjqBYeMxfyPhFRsE3GE+B895WTkKlDuRAb5dTNmISU1hlgHVlsDtsXmOLziv7DUy45SAZWIbVQcMF4w4oTMocryMKye2HSDbOa65jckcyFXRtKiJf4ssdnc1NZPqe84spjxJKiuQ3ZVVNB4qt7w+7DzKdaPxUpJXqwFZNkw1xuQ0HeS07xB8fTJvUt00V5kMk9JyXB2ZVJ2upwmgqiFEdlbkBoTqydj4gkxsfcfXBs1tSIZEdkbHad3G+dI4j5kY6uIQJcBHApf+QqI4sIHGWb7JxIgz0CIzfomtlSMNkaTh9sADkNO+ypF/V32YDJwE6KxrwzeuTbJ7kT2OLs4+RnGYKM/+xjmLLHwVTYiS0ZAraYSUzHHMYnL1Tg1er4HmwcFWMrK0ZG7rHZ2CpQkouhDBKiWzyUjIxUgXGgEhG0ol8A9wX2wts6NxkEDRMdNTR0ZVOpoICUc22cwpr1dxGUkO6wvuR3kgY2wb3qeKIyNaihPnXV1B0FEcHEJs4ujW51gUV0dWMEQn6YHvT6+RIyNI1OAZo7JUPG8Cmf1NzKYap1zStC+IQFqOdNL1hZwMuwDOfmpKdF0dTfn2I+yigGoJilRnYJdVBp6uorlQMuN5bAyUQHYGuiaaCblAs5+UV2kn9l24kOcxQyEhB0tnWueiLNZRSquisVKqzV36Hh4nUBZZ0NDEbmsIEL3h/0OA6KJk9rE5T6HJkIbwDafZUBZ5GY8fmf0MgSxnZi7dxIYCBJVCAiH+LbGewU/mPgBy1MhjcyXgG46xfyUeSfeA7wBj+Yf/6m+fsfyf/6vfDcbyWw0sq09fhT/6T/9t7Nsa3UNDUOAFZqsep12Np093eP/5OTavdjh1FUHTTYXq9RHdtoasHfxJQx0V3MZCtArVjcL4WQf5toYvA/zSQtUOblAorgvOtK04Tzf7VYHupQMcUN8odK8sZl9ojOsA/6pHuKt485s5zjQeNYv/g0aoHYp7zS+BDV0/1QiMVxZqr1A+SPRPPLA2wK5AqDwZjKOCXTgIJ8iWxtOXpJy+DtB7+WiwH1M8xwiMGwe5MpBvatgLAxgJfVBwNQ1HUvFc7BkdIhxdTN3CI5Qe1bsCw3MD2SoWMgNZz5RnSPfIKTuyfFBZKpqKuqBYlIYktXIiF5wAYC4tijuN+oPA8VMHdZKw57z5l+81dCvQPXcot3Q8lQOLz8QkFScywXJkzIscRZYBV/csmlNB5Wowe68E+it2OFUs8tXAG2B1D7QvyIbNvxLY/YgSY1cDs2uB4TzAnHlU71XueLo6fRmxyE3nmKyNwPh7HcS7Orq8ErQUJ+D0MZedTDqa98zpkwPn5qo7gf6KsStqnExQ0hyeKwOkowlN99xh/pXKMtNiD0CSMTPrKL+VyCY3uhUYzjm/6GquQ3gW3d1TRJlyBBEamVGW41T8pogPGV0/U0HV3ATsv8dit7oX6C8jMAEmSXcERfM3Au0zSpd9lJKrQeSolSStdHWUdAqgvsfEuhZ8T3Gk7BuS0t7ZGwGzmKRids7jXt0jO2u6mtdwf+mx/rnIWYXpWkkskp1P8T46OpEycoczd8IAVZz/Uz3ybFwqZAHOGQbF42vnyOYmakA2ywEiKwqaodR3jyJYYm5hknwnsGAWQP2BxSYjhCL4qSe2z1UEk1VcnvA89j5G0KTCOy0jyQh1y6iRcTkZDCUpcrWLM3Mji107E1mqzvUSTKgYct9fCDQfyIQdPpL5OlJDjJJQEzBNTKidcb7TFyLPUI5rgeqejLCPcSi25r/Fiftq5nx9eWTOpQgB1ZbOooeXsdBsKS21NY1sdBugW6A6OOxf68xWS0t2lXEX3Fc1BBxfKKghYHbr0W9klsVPc3zTOU33Zt4PkQ1vvEKWICf5LARifmNAc+vQnycHSL42GyaN0/Ey8wn8JgOcovMwTVSlxM9NmktM7rZFGzAsOR/Ie2AEkXF2MW27MpTrNnd+et4D1Y7zg0k2m+YSlZncdH3B9VCa7KFbh+PLEkl+yflFArryxBlDMyfzmWfq+sQ6yhzt4irJTMc+5KiTJIF1pUD9YDGuVI40kW6SvgKIkTwe/UbRzGdrMZzpHEETBFDuHYYzneWxuvM0BYpglZ/fANU7+ELCLBjjw+uejKqJ7qzSBvTnGvW9hZ1JqM5nZ1RpuQyzLGDmcjL7QbyXH/l9OGwKNDcDxlWB4mg5C9vQPCXtF4SAPhqCqIEMG5sblK0mQ51yO2LclNllV46e2YvGcxaxUuiuCpQ7h3JvmPsZgafuHJfnQn6tsDESJYEBJSAHylrlYGFXFfR+4KyjYZSJtB5i9HAznYGf7Lk8fRzhSwXVMkvSF/y+5dygj8cmNV6RjXp8qeBqjeIwkmEtyI7KziJUKs9TCuPgZ2UGXGQkPSNSSv0Nt1d4IDQF5GlAii3xDSNJgpSQ/TjJdAtFo54Y3wElp/gS5yYZanJ9LSL7mdxXB8M5ytFM2+UcWUjBmckso81ZlHEfHjnEisEQkApBgGntJGv9NTLbMBoIrThfqTUlrsA0i2ktIMlahmGcJL/O8XnnIMoCIbKwv1EO+yiSRCTW9Tc8El4Jw/AbX/NXPX5ngOW/8s8AWP7XvxvA8lsthS0Ki90/eoJuX0M28eL1Au3XC2BQuNsuoDqJ0/9+DtNr+DsasHT7GrASr588QPQMlBdHBTiB/jljHuzaotxKiFahbsb8hTeec34STqB75iL7QjdMiID+KQf7/agAFeBXlknJPjInIkVQSJgrQ2DVS0AGjBcOsLzpd68typ2goU7hcfZ8T7ZIBwgjGYvhye6EgiyNj7OcbhYYfdALVA/cvqACxisLNA64rWjk0kY2TDHH061tBpVyFAiNQzgfYc8cGZvo6lq/LeBrT+bGxk7+hSOwu7AY1x7qFIu1EwGbW3oEHQgMnvXZHdCsPR0sVUB1L+A/7iEGznkev++grzqyVAGQJ0UWwyEyUoKMEaLrpg5wz8Y46+kfAVkAHrBzz/m5keY1zQ1lt+1Pes40PigUWwm39NC9iFLJaA4RJaLDBggyzr2uLNoXdLyUrcyMhCs5AwiwWNedQHGvowssi0X1VZ3dPJsbEcPQJ/YrqID5VwLdVcC45o3W6ygTPQr0TzzaZ5RLpkJxXHuMTyyGKxdzDkV2PDXLgOaWskZfcVmUIgdAgteJBMF6w2tgOA/oLwPMEqjvCGDho5w1AWc75V22ry0lkyDjMn8TMLtmAXj43iQFtg2PJ0AQk0Ds+MIwuqSMklvP15GFC5Q6z6d5N18SlAUdYBZkS8wyZLBlFtHYxPMnzXvZBT8nq19ym8ySYKy/DDF/kZLo4gian5zxuCcGSHc8XmU0GUnxGQlUpmOQoiYIpsMjEOyjjJuf0+bWo9zymBFocT39lY+uowKnV2S+GN0yxXrYOh7Tevp7AmhJ8ufqEGNiIkCOoJI5idy+oIFxyetv9xOL4Sya9ZzHY6ymedbkwFqcQp5nTY6YjuKI7A5aHGncUj942DmZrPFMZHku2RWayqTcQFdGF9LFxDLaxQSeWcwSNA3nEZwaguDFW4fhTOTIjfZZdDWNQKR9yntYc8fZ0dNTxXPWc57Sa7qn6hPlsa4Guo2KEQxky2wjsqxUGrImtqGple4BVwiMKwEbXVRp2OIxnFEyKzybBPmzoAkQ072KbKvI852M5uG1ZBu+rr53lCsvJ0Y1zYUm0Lr6YszmN+NS5JnIIUbIlBFEl0cy4ADlqnrguTRxbvb0XMLWqYkQUO8cn/NAfyazMQ3vQxFExmurPHDsYFxyu808gV2fWUjXsDAvTh5FzIE0Mx57Wwm6lvLrFd1FdHNNBaaMUtkzRclrBHvCI4PdYSWzYU+QMfakp7tq2kfpKLWGAKqd59+jDLTaOeiTzwAW4GfLNARk1UN0PQ3IuaHCA0XL2AoR82KT+ysjYgSKPeXCwkZXVxGluY2EbRS6J1WWSpuFive1qPzQZMPoVkuAlZ1QZZxbPRj4IgJy66dZ3ZLsZNAgIB4px4Uk+C0fBhR7E6OdmPUZCkppyz0ZyXFVQLeGsuMuymgbRbBZqhwLkpxbi4cOtlFwM41xTQlmkAJuVnK+1ZGFlB3Xx/gYAkY309w3Hbe7UBjPKqjjCEQgq49k6IIS0Pt+YhZLgmc1OAznFeSRXT0705ADjW6yKU/OkIz/T/s9KyFPPeyy4tynUvAzAkO3qiE8WcmcfWkcJa9KwtdlBJbfZBCFc5TDZlfWQBlsWeD/EstRl3F75DSzmMBnYhujkQ8KzZ+2Q3aH7Xo+5z1C1yO0HbLra3p9oclKFprL1Iqg0hjKXr2PjKRH6HtgGCDqGjAjMy7jfoVxnMyClCKofBwrImV2gQ3Oxx+XDX5CBKXTc9/8+a45w35XH99qxvLpj8/D3/rP/i38jbOvsLMN1rrDf//29wEAf+vqS9wOCzTKwASJ/dhg9ApnZYevj2dotMFVc8Sf3z/ByyXdRA6mwmA1Xi23+MPlG3w9bPA/fvl9zCqDs6aDEh6LYsBtt8BhKLGsRvggIEXArqtRaHacdscGdWVQFRbPFge8O6zQFAaXzRE2KOyHGl/9H88Qzg1+/MlbfPGwQV1YLKoBX7y9wGLdQQDohwKLWY9SOxTSo7eUoT5dHHDbztGNBUaj8ep8i9vjHN8/v8Vtt8DXHzYoK4NuW2Nx0WLV9Dj2FX765B3+wT/5Af7GDz/H2+MaSnp82C4gJWWym0ULAeDNry7xvR9c4/Y4x6wacTU74ec3VxgHjR+9usYXDxs4J2GNgh00wihx9vSA46nGk/M93t+vMJsNOHxY4Cc/+Bq/+B8+hf2M1baQHi/O97Be4v44w/mixb6vYIxGv62xvDqiUA5tX0Frh3XT4/pujT949Ra33Ry7rsZx2+Ds/IT9vsHZ2Qnb7Ry+0yjXA4QImDcDdocZ/KggC4+r8z1u/+QJVj+5w8PDAlVjMKsH7PZz1A2jWwBgHDVMr1HUFutFh+1+BrOrgMpBVQ5NM+L4MCP4lwHPXj7gfj+HEAHWKhSFQ3+osNi0aE8VlHa4WJ9w/fU5isUIqTykDOjuGqAIULVFWVr0b+dkpo8FJa2a1NUfffYl/uQXr7m+ADAeg7/L8wFladE9RBdgTalr0AFybuCtRDUfMV7PEOYOoleon54wfr6gedDcUQ47Kqj4euwLlE9ajF0BoQLU1zXMFVlt2UuIJz234U0N/7yHP8Us0kEi1A5wAvKk4DcGxfsSdu4xf3XA6csVpAGWP9hi+2FBp+GlYSF8XZGB9wJyM8LtCxpovYzLDwCiTFW2En5lUb0pMTyjhLe4V2h+vMX+egmogObzAv1nPS4ujnjYz+girMDcVRMLk8pDdJLH8mqAuK4gLLND5UFxnvlshD8WqG407NxPJldnFnKnUe4JUGwdyOarAMgAeVKZ/RHPe+Drhmz1ViPlTArPmJac9ccIAAAgAElEQVT2IwuUHs0vo+Q2sd0NAavs6WyMQHmqrwOEERmgA5hmS11sEihAxtlizgrHL3Uv0LwnU2xnAfUtGwjJoAWIDHBk2XwVUN1SEZBMvlKWZpCAeN0ifDXLua31tSazPHKW2TU8LuUHBWnJHNvlpEoQliqHJO+FBMzS50xU1U4ZpsVJ5BlUOw/ZyReI7PTSQyfH5XMHvVO5yRMkoEaBckvQnfJThSera+cC7Qs6+ooIjKWlQdK4BEJBxcK4YhPEzgjKaVA1HS8IGlHpDtEVOLqTCppdpVlT6dh4GJdxlnaMktRogpWAdH3H16Qc2CS3LQ7x/ESZrLTA/vct5p9rynnB9+8/s2jeaqieDQBXT+qW4ZzyZjkmFULcBx9jVAIBMhBZUEeQXZwY/TJ/Q2CsRjY0imMEeDG+JWVY6naS69pooMVmikB959FfSEb6xBxW6dg8OL4UqO9DZmPLPWNPigObQ8ktlQ7Lk2Oua9gIqrZ0dGW0TYxhmSHO8TJTs77nbGlivl1NxUu5jXFGmqyxMowyGVa8vpJkOCkXqn2Kt4hg8kjZc7UL2SgpSKDfMJtTmpBnipdfO253/ElmRtIG6CGC0YZNkcRmQ7CRkVnaUqC5czg91TRAiqBWRuMm24hHYF1mUJ8erhRZSuuioVDREiidnkjMbzzKLRui45nOM7qqD0jmYq7hrGsCg+NCYH5tCYZ9iE0Ezqg2NwbDhm6uNFsSeZ9VNJgSEaAPa4nZe8NlrjXqW96oaLQE9BuF5o7GM0FLqGgOlHIxvU7O4ZIS2EpRGTB6SBdg62hUBDaMiu0Asypjo0TQXKiK7rdjzNlMhKihw6wYPSAFZGvInio5SWJHSylsoQg4o7GQakeCZA9AkxUVgTLZUEio48DnhZjMhjwQKspls2EQAAyPJKERAKYIkswgJqAa2UBhbAThjCNJLGMGm4+Nd4TgcqyL7rWPQG+KLzEG2cQneC5Pa4JOF9nOFDvyeB0pGzS50hY6s5u/9hF8ltw+nun8f/r4XWEs/+hv//YZy3/43/xuMJbfamBZ/96L8Ow//PcAGRCMBKygrPXMolyMGI8lxFFTQveqxbitkGazYATOX2/x8OUGajPA9hrFdQH7YkSwAtVqgPgnS5hFYJREGYDKAaPMcRrl1yXGZ5ZM3sIgWAm51zlPUB9FLnzCzAJWQpSOM3BBQIycPyvvVQ6x93OH6h0BhuoFuk9HAgsVoGYW4aaGXxvom5ImQxcWcALlrSLjmaSxvUR1zziE408GoFMoNgPcuwZ+5nkMQLZUn2ScCwMgyb4FTRkwJCALh/BQAsmFVRDY4G1Nl9rIqMlWwj8ZIO5K+EWUJX+hMVx4+CdjjuzA0gIHjWLHKIlix+50kAHjlSOQkIHOuyvDdS/jjWRXQJ8EzJWF2sUifmCxbc48Qu2hdgopv9FXMULjvcK48dBHAeEF7MJDvuhgHirMnp7Qf7GEtGSE/cZAvy/hlnToLfaCmYV6ymIrdxL965HZmoGsmRwkiqPA8P0ezZ/VaF9bFDs627nnA/dfRrOgdZTTqID6RjNOQwDdRwb6XnPUNHCurv9xB/Wmhp0xssRXBBFBB5QPMucymkVAWBDEFHuJ/rlF8aBg1g7FXqG+YfFk1iFHbMDH2S8PsoZrGiDJQbKYnU8zbckUR47Ix8rVEfS5KFntozQvUA4qTQI4gPv9E5yT0J/XnMkSdL7tn3rU7yW6Fw7VnUK5BY6fcBZTOMAt+DvnNAOqO5mNd3RLM6j6jpJVswjQvchOzDrKolMepO5EZmRTrEJ28C3J3roSqO9ihMkcOR8yORELK1Btgf6CsuBkGGRnIYPH6p7XJAFEwPwtGTRfAuUD2dHZNaMommuBYROltFFSTJkvJcAJIIUiyuAOBDrJhTixuoyGoStyKvq5PwR8KU4DIkB4ylSrhzhPWPBc61ZkR+fqjvtmaxb9w3lA844FvJ1RSp2MvYJCjvQQLs6gRTAo3CQzZjEf5zaLlDnJbesvyPTTTZnbpGJuZZLd2oYmRONKILmzAojzoARkyShI+AgG58gOvKeXXFf1QBZwOIvGQT1ypAUANLce+48lmlsW70lyTgdtgtJhIyidbRip8fCTgNXPyVQmJ9rZuyi7j2yrqwggxrVAfcv3mgXlzum57kpkFUC5jzEklvvY3HocXkuUe35+x5XILr26C3m+djgTaD74bHg0XAjKo2tEdiUwVD7O0bmKMR/Cc5kJtNb38bNTIctL63tPk55Lgs00i/l4hlAawKwEVl84HJ9RIplk3oyUCdmFWVqez2obnYnj3GF3KVEco7KhDeg3Es0dDYNm7ymPldE12TRkx5LJFkBATTddrk8NAcNKQg8R7DRxvveRUVO1i8ZCNTC/ZmSKcFGG/kCJagJc3TllrtWe8llguuZn7y3U6HF8WaK5pQSX86IBx5cKugVmN4amOAoZWALIbKu0yCxstSfDOi45g0knaD5HkyMRZeB+coQGQViad3xsfORjrmdxYuZlEALFwcLVirLWGUFYfTvC1YpusdGYiOc7QHUeriFA4nOUCCdWV7eUx9qZRHMzcgbSethG05m2VtC9y060eabREXAmR17hA/qLAvWtic6vAeOmhBo89GGEXZZRtiuirNXCNRq6NfA6Ar0Q6EorQHnt6KFaMm3JoVY4HxstEinrUp1GgsHDgFBrMqAj5ymHiwrN2xPkoYef1wSJj+pp0VsynIWEbA1S1EjKs1S7U44FmZxdabIjjCOYkyJHiUwzl4/YzLKA6Aca8aTokih1Fcn4J4I4YR0wjAjzhrLYNH+ZWNK80NSBGgkCo6yVTwUymECOIAnD+A3QB6UQ+iFLdh9HlADIM5X/r5jHEMHstx1Ybl6HP/rb//5vfbn/8L/9D/5/3zfgWw4sq1evw+v/+O+i+JMF7JzRC/wCRO6Mu4pfts21wPFjj9lbDtoHCZz9hcfu+xLNTcDu94AmFt7LLz2GtUT7gnNaab7KLAPqW4HjRx6zN1H6dAo4vUoFLFBtWTwVRxYgZ3/hsPuU0SDd04D5G5Hnp2QsuI+fOKx+rmAW3GYVC4U0JyMNv+QOryV0NFpI4douOt3ZOQsMaVgw+OjcWD0gO+7Vt3ydbvnFlORq6QtKtyyi2pcBy1+yG6y7OIMT3StT9lv1EGCW7IKnrndxmAwRggS6Sxbg9QNnjqotCx3dAQ8/AhZfithNjyc0TAYP0lI+lb7MEZALrDHtywxYfm2x/0gzN/CMxUmScw0rCd1zX48vJRZfe4wrytGqbcDuM2D5S3adTUPHR91NLpsIXKarkaMMcoE7i4Vb7FSn+S5hWRyZucDsvcfphUQVu+/rzy2GteS2vOE1VpwCTs8pBZQuYHbjcXzOL+/l1wa77xXQLZeX5Ibjmsc/MVz3f9Nj87/JGB4erejXMrMHAOff0r43tx4pKLxo2cF2pYizXbxebMMZtzqGhh9fSczfRuv6Os7CRWOLFDWx/qXB4bWOhafMMr5kRCItJYfjQqCKRhXjWmD5leP1sfNonygcPw54+r/wS54GEyzejs8VVl9aHJ9rqJHFU/dU4OznLrMMqQMfBOfAkiNkyhR0VZzVi7EazT2vTWkYvO4qgXJnASmw/X6B4oTsMDqsJZZfW7RXGqeXArN3ZBZyZl4gU+MK5LgR3bOA7c8ETT0U5aBmyfzEauvQPtEoTh4QgGli4RudTe1Mot9IzK8dZ/iik3J55GyiV2QYxiUZGj0wBqK7kHlecX5t0J9TOqVbumf2Gw1fUCaou4B6R2bi9LzA8usB7ZMSw1ri7BcDDq8qVHuXi71yZ6PUlIWkjdEI/Yaf1+aWrpFkLySKzqNfK1R7n9mScaVQHB2OzzXqLQv4xZsR/XlBmeFMYvFVj/Z5lUGiNJw7czUbSsNKobm1cI3M2zWcF5h/3cEu6JiYZH+U5/J+IM2UL2gWEsvPWxw+naPcuwieApf9wUyywehIqmM0RH8ec+ICpZjz6xFmoWFr5v2lWUCvKeUE6PzqKhqblFsLX8kMAnTrYOcKKh2fM83XFCLLKW1Dt9H6weL4vIAeAmbvyVh4JTBsOMNd7iz0yaJ/WkHF+IqgBOxcQY4BzXWL4aLOs3XjWnHbagE5sLkibMjSzPqOkkP+X2FcaxQHh+Jo4RoFFYGI7snqUM7PGbzjRzVm7w3K9ye0n6xQHC3k6GBWZZyBI1ukWguzKrL5StAC6mTgGw0fXS+l81AtZafDVQ3bSMzeDflcqpOBPaugjwZmWZJlKmSe0StuW9izGqq36J800K2DHFye5/Mxg9DOFMrdSOlkBDpycLERRzMXr3lfClrCLDXq922USUZms4yzhXH/5Ohzca1vj3BnM84LDg7jpkL5MMDOCxTbHq4pMG5K3vduO0BKyHaEuZxBHUfIztBhNMovx7MK5W6MJmOUmErrIXtL5ssFuEWJ4r6Fr3kfcLWG6m3MfPQQgyNrVijYZQXVW4iRP6FQsOsGet9T5qklpZ4AIEQ2eRG9gb1YMOrDekZ3aJlZN2EcfFVAHjuEeR0NbzTkoYVfziCHGDGiJeS+I+CxjgBJSfhFA9mPXF78fAXFmA25PQGaoM3PSqhdl+tE0Y+cUYxAS0R5prtc8X2FpkOrEhAnUvVhXkOceoS6gjy2fP+xBZoaQSvOOzpP2WcIeQYyaM5vciGBTq/HluvQCqIfv8kAjgZoaoJCKSYw94gNFFpnQBe8h5g1QAKT/TdnDXM9n3Ipk7GOjC6rWpNhzHOdNr8+u60mwOdipEtiI9PjEcgEEOcvJ+ZU5AiTCDLjOoRSBJV/eV2/7vfH///LAPS7wFj+c2D5u/uY/+B5ePYf/T18/9kHvNuv0JQGSnosihHvDktczFt8dXMOvKtQfnJECAKzesDz5QE3pwWU9Hg4cFjJfjXHZ3/zS3y1PUN7rCgNNBL1psePnl7j8+05Ht6toBYW7qiBIuD1yzt89cUl6k2P4WaGoAKayxbdfQNROyxXHarCwnmB+w8rsp1GUGZ3KFBfdTCjhr+toJ+2MG0JWTg08xEfbR7ws//1I7gzC9U4PLvY4e2fP8HZpw84/uk5Lv7oBtfXZygag6Y2OGzj0JUMCCcNuTQI9yXC0gJGZgawOO/x+nKLL27O4Y4FyvWA4AXMroJejfBOALcV5zJnFuJEoyEAgBNQe53ld/ogYM7oaOk/6uGNRBgl5pctTu/nXKcXzO5cW4jCI/QKYpSQvYB63WJ8qMnczh2lnI2Dvi3gqwDfkDF0FwboFdRJws096neaM63JAMMLhGc9XU49IJzA7JM9ul+sKPusOdMqB5HZN3PmsPqZxv4PRoiOHcT6g8KwITtd3SoMFy4zuL6mkVEyNYIEgvYoP2i6bPYEJ+O5w+KXNHzonnlUdxL9Cwu9V5zb/aBpVtSEzKYFBdi1g+xYxDJfkb/3zy2aNxr9E4f6g2K+po9zcYrnobyP2aNxbs7NuK3Ntcx5nr4MaN5LjGcBZuWgjwrNtUD70sPNPdZ/qtG+CKhvBIZNgLQiR2oURwLO/orn2tVJZhVQ7mQGjt0Tj+JAp0HdgmxZRdCV9jdlgSbjn+a9wOlVZCfDJMkLms0aswhTpMwYMwVNjPvQwOwtYzXMkmxbcWTsRpBAcZjiJeBp8HN6FZ0+4za6Evn4JJdPNYCxNzOC4tOrAF8E1De06c/5hTHiw1U0WUqNId0C3XOP5a8khg1nTYcNGdVqm1hGRmgIS8Y0yQBNzJoc12RcAbKnZgFUdwS57Qs2l/RpmuMVlkH27XNmjybnzyz9jNLQNFMaFPdNnxjJkaJkyq34BuszXHDWlzOsZAVV/8hgp5yySRlLwBlZNXLZuqdJT3HkMQBis2zgvHJxSkYsBCSu4XqLI5k71ZO1o6QvOpDGz31iD2lmwqZYitqpbwNOL9j0KbOEMkk1RW6QJTMiFc1qykNA+1ROjqoDYv5pyFmydsb3pTzg1HhM+ZcAm2Cu5mxtMsixDUFruQ8Y4myiWXBWd9iwWeSikY6OUTIpg5VMOWIeZZRCemCI7CKZZZFNccoDP/fjkiwcYrMLiE2ekSzYuJSo9qnRJDNTqfuQpZl9NHcSFrkJMKwkI0biHK1ZsLFZHTifOC4FmvuQ5yi9YpbmsOa9pDyw+VTtafbjtcizkd2l5DUSG4pqDNnldFhJKIOcBYrABmJixvUQciaomYvM7D7OGZWGDRgTzXCS1DXNljLr0WcTG1eKvL0Aj3W5tegvC1RbspFBRhfUmcwsW7dRqHc+u6KOc4nySLMingeZGxl8v4NreCyqBwvbKOjWYVxrqD6xafyc+JINsJSRmVxw1UBH1eQQm1xwi5Nl9AmA4Uyj3NkYE8J1mYXKmZg0y5IodyNOrxrM3vbw1TQ3aRvJBsvovwGyh02B+nbMBkKJ/ZSW5j/DeZXZ8uJgCKw7i3FTotwZ2LmGPlD66mqCezfTsI3mumK2qjoZuHnBGJazCvpgoDoD1xQ8v51lQ6KiFFcdR/hSYzwvUd/0sUlEV9YEsEUfjXwKRZBtHNySN8GU/ekqBRXnLyEE5LGH3cwgjSe4TnEjhYqZn/wRg50Ak7EEjprxOfAgcI1RItm5VasINiOoS8ylkhDdgNBU3yyGQ+DfZ3X+HYUm4G0TWG4Ifrseoq4I2P4yU/mYXZQyMo8TKAWA0HUErlU5mfYoBQwDIGQ26Um5lME57ocxj1b1G3DHX4fF/I4Ayz/+l3/7wPIf/Hf/HFj+f35Un7wK/+J//nfw1c05/EOJIAPmX2h0Tz3NZVoJ8aKHVB7meoZQeBRbFWVblNWRaRMYftgBNxXqDzJangeYK4vmiyIGyCcpCYAXPdQvGsjoZgqwsDy9JCN6/Nhni/z5G6B9TjlS9/0B9a+qOPMRGbBOoH9psPynBcYVGdZyT7meXQZUH3hjKVqgfcrnhgvPTMoUEh9ZKTcLORJA9cDxY4/6g8zOgulLtb+gDM/OKO9yNQtIOQg074HD9zzUIFDdiVxIJ1MOBOD0PYPlzwp0TwJm7+hSOv8K6K9ELODJ0PUXlFWWDwr1HbexfUGTnmFDOZxZsph1DU1CpAVOLwIWXzE+AGBBSwMYj/lXEuOKmYmnl0C5435UD3GGKjJjqTAsd7E4ugwxNxSAiEHaK4+zP5UwcxrVSBOlXT2lis2NwOkFj3kqMPuLgOUXMbA9hshLC7qY9gLVAwvm/pnD5T+WOD2fGM3mhoX38bVHdU+QogZgPOMx0C3Z0t1niDJDusBKEwvuwHPRPQ2Yf839TrNXaQ6Lkjrm6SWXRV8A83cB9z/1WHyh8hxRKmaFB/bfj3EWBbD8gkV5tQ2odgH7j2UuoAl6yVYXR+YGDhceq78QqLcBu08pH9x/iphPyrknABn4QvAa6S4kdj9xePY/CTz8UKK+jSYtHbB46/DwmWJUw0bkOS45RsnaA4uM/oLbCTCLb/4156LGhUB54LEwC77m8JHA6ld87faHlNE2H6Kz6FJg83ML20i0TyTMHBmULr8MKFqPhx8ozN9xn82Cx3t2TQZj+xmbDuWezHB9TyZ69blH+1RiXBNg2hmv6+LE6/XsLzwefkAX4PM/8zi+ZKE4vyZTrDsW/0ECh08pLV58EVAdPIalRNEFFEcPs5A4PZNYvPUYVnQ1ZXwDGeogCSxcKbD82uH0TOaoCgBYfE329PhcI2jKHY+vFM7/dMTtT0vM39E0q7sgoEhOpAyN5z1oPBOo7gJmdw7DSmXjoeLIY7L6Kna6BZno+bXB/pMC1S6gO5dYvHPoLiQWby12nxaYv6e0//RURWdNKhS6c81oiEaguXdorxTMUuDs5xa7TzVWX1r0G4XV5wOOL0vogcsXbgJMxxcKm5+TYV//YsT+0xLV/8nem/xKmt3ZYedO3xDTize/HCsrq7IGVnEyRZGt7iZF2d1oWLIML2xDm9bWhgEvvfTCBvwX2CvvDO8N2QYMwwYMyDKkBjU11WQ1i1WsqszK+U0xfsOdvDj3fpHFZg9Ad0MkwQAeXuZ78SK+KSLu+Z1pEbE5pdzSlQKzhz0W9wqUK6ogguZrub506CeK7GMK21EtGXJfiHQ8e4gALF4vYKd8jyivAnTLVNr6ikmxk6cOmzON2acdLt8tUV9E1C96XD8oUV3SMwfw9d0nUJ29rgCw93GPfq5ZrVEIXhPJU9fuS0wfOYSSYS7tARUS46c9gpFY3zSozz2Wd8kaZ3llP9lVjfQzgfEzj2Lp0M80uj2Ffipw9IMGwUj0c7K09TkTVH1BBnfv0x7dXKM5lJg9dEnGLFG/7BGURHegoZuA9Q2NchlRXlpsbhRQPdNYQyGhN57ezQmTOzNjrbcO3UEJO5GoX1gGGzkys/28oK9v6+HG7CyULg4JqKr1sDMDXxHAmZVPoJRKHwSgPVCoL/wgb5UuScMXTGatXnZoT0rUz1qs79RQNsIsPVRHNqY94aLfrJmAaseaia0+wo13MlA3kqjP+yS5tGhu1KifNgiFotSz8ejnGtOPVuhORvCFRHXeQqZaDDfWUH2Aq9QuJfbQQK89GdhaQyaJ5/b2GOUFZa3FRYvN6xNMfrri++btCUafr9GejREM2VXpIrp9jdGTFqzA0RARMNcthAuwBzWCkdAbBzvR0K2HeblBd2PKlNhVBz8pyHynVFK17lk30ll0N2YozjewByMUTxbob+6heLFGdzaFtAF62aI/HEGve6jrLaAk7OEYetnCHoxQfnqO/s4hAWWlyaY2PcKshpuW0Ot+COHJXZbqYgW/P4VMrGSY1RC9g5+U3L7WQW66wdsor1aIkxFE0yGMKkpOASCxkGE+IcOZpKSZGR18iQkYim2LOEkTM0H5qbBux3IaDeE8Gc3VBkPfZIxJDkuJbRxVDOEpDJlOrRDXG4iiQOx7iOkEcb3h40/GgBSImy1EXXN7Msvp/BckqnHbDCAQwAAERVVh8Ezm5wQw9GY6gmRRFAiLJUQKEhoqRnIqbNt94fH/xO1nU2HzLaXT/ryU2NC2P/+x/gK3XwPLv/6b/re9AX+Zm1ARf+PwIZZtiQWA8bjFUk0wOtrC9hr6xONktkaIAo+uSxzfvsZL7ANlgFcR8twgnPTwY4MHN17iE32A7lTB9xL7h2uMCotn4xmM8RBWcZ0eBEZ1j/VRATm1TIV2Ast9hXK/xbocob65hj1VsNcVolGwcwfhBVTp0T9IVSY3GU7iK4mz25c4vzhheEjlsD3UEJ2EPmrQVBXURiIasnWdZqAHgYYagFJm8tpTevdEAPbvX2LZHpIZ8UCxlOiOPNRWop/Sayd8As1lYIVIlMDNFv3GwJcKesMaADuLEK9t4J+O8PaDJ/hxvAXRC6weAFFEmJUegkKa48R6VJGM6dkW248mkFbA7nkACuqNNdrPx4gm1RLoCF9JFNcCkAQTuRvRPjdQjQAOOzRtBXdkodoC3YlDf8jFAKV3O3+PiNwvO6YksL9h0adkANGw93H/9gKb5QEQI+yBB0yAeFRwsZhK3d3NHvZAoThXA8hevsEJvxul6H4Z4SYR/ZmF8AX6dxsoEbE9GzEpdaXgT3uoriSbc3OLtqyBABSXCvbIwk0U9JJMid6wM9IfWJhViX6PXrvmpoO5VoBMiZU1J4zbexbVE4N+b1cin+XIdkr/oR0zHXj9msckXTfdAatZ+hlgT3tEZRButmg3Nc93FGgP+Dy+BAoI9Hv0wlBeKhBKoH5jie5yjuZEoDvl+fV1gD2x0BcGqmd9SHlFptBOIuxEcrhw1GBzg1J21xKwtwcRqicD3B5k5jR7bjiUyIioOQsQXoKVIgyd0BsCvOaE7FkoKOvuTjzsy1TYLiIZ2EYOrOX2JPmdKiCUBLibNy1UZ6C39EFvz7i4b29biE4iaAXditTfCHp2KqCFACSwuktPrJuQtdjeiHAHDupDg1AQANsZXzeL+zyu0jOJU/YM3HBjgcnn7J/zRU4rlbBTgd6KNKgRWL3hIbwauhTbQzH4z+yUzHB5wVJ6O+bgRzf0821uSLRzOYTT9FbC1cD6loGvgW5Pwo2Zvjz9jJL15lgO/s/uMKJ6SYlv30m0h2J4LbqRQD8H2pUcfJ7BpKCPGeWz3SFgtmRQ+iltAf0mMSwHAC4px98es6Kpn5EZ6icMJPEF0M3JXPUTifZAQNqC6ayOMu88XHMjkRJG+fG3PTX0vyoG8uhGoDsQKJca/ZzVG90+exbbAwERFLq91F2YJM8+pQHbKSth2iPDoJaUNBo0BxxuRDavXPIzrNtTsCOBbt8MfY3bM243wo6JiyMMIDMEbod0gN4aTD9rsb5LP4GdsrBeN7xO2q2irF2TucIIWN0pE4gENjfoR28OKbfOCb85IId/K9DNDX3EFa/z5rQYElfJMuqhYkT1QL9HUGmnAs0RBwPdVAIogAg0BxLlIiXaxojtqUF7REuACHlxqYYuS18nmXvj4UYa/UzCGwE7S3URHWX6diyHYYmrJVylhoGW9ApmLdFPFdp9MrlZ4hzLJN0vUmprqlDxFVBdUdK+OTUo1gH9XoHNqYJZmXQMaHXJfkMR+Nzsqw1oDhUtKznzxIjhXNqJJsAV/L0fGYQiMbhp5m/nFV/7lYCdGMhawxdMaUWqDvG1IqvnKYft90ueQ0VJbtACdmpo1wgVfcP71VD/4fZKblfBD45QpCqVWTEkwwYlICcF60AKSU9jStr1pYIal/ClhG48uuNRYl572Kmh5NYx4VVvNF+HezVCIRELQ+A/ryF9QHdodlUhtQbkGKJLEtzSICoBe3OfctbaoN8zHC6MOG1Z3yow/TxCAZRwK4FQashxDWhJPyR43WVQKUKEmxguiFMHZRxVrBkRAn6vgkp9lBACKAvKZ6uSyinrEesSoTaUIBsFaAnRWoTDGUTvWEfiI+XFqmDa7JZSYaSaEkxGBJmFQSw15KoBXgVkWiHWJYTzCHtjyMRyCikJUMuS25/ZSoByVCxUdkwAACAASURBVCkoRS1MYh/VjrEsDO+XPJIi8n1RaPYgRRH4+Il1zdJcVp8oQGuCSp3AsHUM4knPDWMgrAWMQex7yHLHtr5KbOXU2AFIZpD5Ktj8JSbC/sTtV2hXfvb2S81Yjh7cjLf+u/8cMQic7K+w7Q3eOnyJTxcHuLieYD7bYt2UaC/5RlLMO8zGLTZtAa09vJcQAuh7gsBiv4XWHvNxg6fP5xAq4s0bL7HoKiw2NdplSR/g2CJGAVM4NMsKquJkzK8NRodbNE8mwMyinnSY1h0Wmxrd1iA2GjABsvAIawPoCFk7hK3G+GiLrjXwFyXEfg+lAlyvoAuPEAWiT9OkIIAoYGrL9NMlJbBCAOGqYNpmFJAmIFwU0MctbKuBIAAvMDtZwzqFdl1CmqTjD0CwCtJ4CBWBJxX8hABV7Vn4lYGcWISlYbrkE4Xtgw7mWQF/u4X6vKIU1gnoxyX83Rbxkm/yeiOZqlkHiNIjbvmmpdYS4aTn/yUlSDARQgWUn1To7reIW00J6T5TNYpnBv2ZhboyCMc9sNSsMGkkcKOFvy6gGgJw6AD0EigD4AVEKxGLCLXim1QYBVRPFNozglfRqUHyGlUkGJwEmGvJUJyaVTDluUJ32w5dpLJRCFWAXvJx3dSj/lwjFEB3q4e+NHAzD32d9wMon2nYCTsUIckU27nnts88xh8RJOotZbe+DtBpu0UE1JbSUIDgvbyU6Oesc4ljSokRs1yTzKcvI0ZPJNqjSKmsE1AdQWOoAlM9CwLY9oZDcaEYelNGqI1kZ2N6f+8OWVzvRwHmSqI6ZwiN2/MoLlgJUz9RQ8qlLxgWE0yqwem5v+ZKYfIZcPVVD3OdKgUcF/3tqUNxruArSnFZZRAH2Sc7KAGzpAw4VJQEh4IqhCwpFgHQawE3jaheEuBCUBbsK8p5VZtqCQ75elCtgN6KQfJKT2RA/SyB4XWqh0nDGYCKBZfWK9JTyhqzPK9JEt05f6+3BLJRA91+wOShTLUQGNhtX8ch/VX2Isk9uU/dPCbAwe7aHLpTLCkdFoFAN4fo5IqHPHDp5lQL5DAi2ZMRt+OI8lokr1bqLz2jKsGOeSzMivvhKx6HPMAwKwJ4acn2FwsxpI3KnnLX9pBgG+D92+OI0RMy78KzqsWseOyqc4H2EEkKm9hqR694XqDnUChp6e9ujgXKBbevvAa6PbLKlAPy3znV02xSeX3zSkVMpGTWV5R1tkcpHTUBdZkIEFeRhfcl90e1O6ZSWfqOQ0kwU12wG9OXYuiLVB3ZRQ6nGJqT1QPtAR9n9IygNCgMfZ3SJt/7NiZ5KIFqDjcSkbJU9rlS3munAsUiDom3wYjh9VFe8b5mQ393c0RWt7oMsKMkI95EtPtiYJ5zh+f2VA4SZBGQfPFx8BF3+wLVJQdwQe9CnuyY10y5YN1Idc0eyOyLz0E50mI4zzmkSXXpNZJl3W53PvM5lhZDx2bOKsi9m7mLUwTKaHNwj3T8fZSANwLFJqSkUkBvc7hNBowczFTXAe2c7Hq3p3ZVHpoprLqN9JkLPjZAH6xumZBarNkn2U/Zw+mNQLlIfutthFkznMhsAppDjfrCIRQEd65KYUT9Ti4/hPYl5jpLi3P3ZXnt6I1de7iRRLFwaA/5OVGf92gPC3rL0/umr+QQ8DTIdAuJkNJjixWBXxSU9KqePm+zTl5sKaBa+oaLhUvXgBxUNDkgqHrRoj2qYFYWdmZglhZRyySjdXC1HrIEmOxKia4bm6ErVNqAYOTQAaq29GqGIkmXLYGinWgU1z18rclqr3tKYTcWIoFJt1dCbWz6LC+gGvo7ZUupq6908u4qyNSnKftXJKs54VWJwdssLKW1HAIkltCowa8quiR9dR6xMkOCLITg71v2T4qmIyOamDzRW4TJiJ5RrSDWW8QxmVGx2jIYSCkyrT4MUthBahvjF8N7fhYLdD1/pjUZzlelsAABX5akZgltiGlBmTywOfhHiOTZlLvwnp+9/UXDfH5VpLDf+2tgLP+XXwzGUv75d/nFvdW6x3/79X+E33/vD+CjgAsS3//kNVwuxviH7/8zXHy6j9sH13jv7c9RHza4d3SJq+UIzXWF7baEc1wpe6fwjfd+yvqGZxOc/4tTqOclTg6X+Ojf3MbFvzlGuy5w/7UX+PKDz/HWjRcYjTq0z8YQOmA2aSAe1TDTHs3nU+zfuwIEYD+cof0/TuCdBFYGX333M8iFhjIeZt7h1p0LhF7h+NY1ms+mEJ/WuP3WC8RFgeP9FaKTsNcl/MYgrA1mswZFbXH7xiXiJ2O03z8EXpbQxsP8uMbtBy8QGw35soB4XOH22y+Az0b0WHoBNbEI/2QfzYsRICP8yiA4gbAxuHnzErNZg3BV4rWvPcHkdA2UAfF5CTFyiNepS2ns8O2//wPIa4P/7D/4P2F+WuO7f+cHUJ9W2D9Y4/63HxI071k8eO8x/sG//49ZpaEi5PMSZ69dQK0lvvatj2AeseaheK4hRw7FUwPzqMTb3/sY5mGZGKGI6rGB0BE3v/UE5eMC5t4a1U9KHN6/wvRjzVqHpxXkzELe2qI4VwSKM4vqYQE4geJsC71QkHc2kLdZffLv/Uffh+wE1EIjjhz8gYU+blhLcadBlBE3vvkU09cW7BotA278xhNMf1gAKsJcaYSxR/VUo3qwwOzdCy78323w5nc/Qf1ZAX/Ss+Py3ob7YSV+43f/CGHmMH3rCuG4xzu/+UkCCUDxQmP0nZewZxbv/96PEU866KVCvNPAHVvYI4vj33zK3s8HS4yeSHRvtAhjj1gEjH5SQDhg8s4V3N0W02+co3xnAb0V+N1/8M/gb3bQGybJZn/f6DON7tjDHjuEL60x/lTD32vhRwGznyjc/RuPMfvt5zDfuEJ74tmbKiOqZwr+fov9v/uEyaefaJh3l5h+pPD1v/cj4L0VmvsdZXtil5gaTMThP2da7pd+/wPMfqxh52RL3JsN+jcaHP5LhXe/81MuAr9+he7Uwe5xIm/vdhg9lUxffm+NYiEw+0hi8jfPUZ2zu1U3AsUiJTq+09D/9b1zqI7DjsPfeYLmtqU38p0t2hses48k6udySP69+e8+wvbLDXwVMf+RgPzNK4w/F+jea7D5zQ3a9xrUzwWKa4HR7z7H9nWL9jQgfGuB6iVl8sVKYHu/R/V7L9DesvDvbuC/vUT/5S3su1sc/qHA5Peeofw755AeEN9coPrGJepngtLsVeqTPPE4/buP0H9rBd0IpuAmn+rBH5G13J6RkVQNZffNTY/V+x0ThCUZ6vZrW+x9BDQ3PBnYN3tUf/sc/Txg/wPKspsbHgiA/Y0V7vzfFup7F4Ov9frrPZrbDuMnQL8f0ZwEREF59v2//zH6PSbi2info9fvdli+32PxWy3K67Twb4Dl+z0mD4HrLzu0RxGbOwHFgqD36AcBi/ctqguCweuv91i9HrD+7obVHDXBwvJdi2IR0RxHLL7bolgBq3sB088i1rcjDn7s0c8jfEGwu7nNPtJ+HnH5dQ+zirj+qsXRH1pszyK2NyOuvubhavpqx48j2tTl2h5F9FPWbURF2W9zFtHtAZdfCWhOgev3HbanApu7AdXLiOnDgOUbQPefXOP8W45JtzVw+XVe6+u7EeMnARff8Ax5e8BjdvDHDud/k8cVkhJvOxFYvM3tWr0OPP+Ox9X7EdVlQHkdE9iLuHwP2NykR3N9lxLw7RmVAFfvBbgaOPpBB7OKePkbHiIAL75r8fIbyVcbybJffiVi+Qbw7G/xnO1/6NDtC6zvCjz7TsD0kYdZR2xuCyzeAuqLAF8JXHxZ4OpdgtbFfYnrt8mWlwsOBcpFgN5GXLxPduzx7wS8+LaHsmTNL74qMH4ehkFIFMD2lExoe8jH0VuqXBZvEuCNzj3KRcT8o54Wh5HA+EUYukuzd1u6lK7qIla3Ja7fkonBTn5kw+N0/mWFcuFx9UDi5Vc1nv6GxJPvUlp8/hWB+jKgn0iMzj2ef9NwQNFF1Bce5SpgcV9hdYtS8OaI4VtBC+guYHuk6Ec/1HjyWwrFipLocuHx5Dsa00dczC9eN1jfUnjy2xr1OeXS2yMO32aftpAeWL6mEZTA1Vtc8NuxxLNvG/QTCekjVnfopVRdxMPfKRAFGEi29HjytyroTcDoaYvH36lh1h6LewXOv1zi6kGB5lDj+TfZMbk9Urh8u2TI03mPyaMG2xON9U2DqAW2Jwp2ojB+tEVzpLG+qREKgebEIBiB5qRAKCSKhWVP5sLi6kEJs7A4/8oE5UWLxZs1VBtw8X6N9e0CunG4fqNCMAKjz5YornpszwrorcfV22PojcXmZomgBdZ3KqithVm06OcazY0azUkJ2XpKccEwpfEfPUUsJPTWQq86tKc1RAQW70xx/o05Fu/MoK9bBKPQH1QwVy3sxEBtegI5F+jvHBmoVYvNvQlk7xAKDT/S9FlqCV+bofPSTwoI69HemsEejBAKje7GFFEyKEpYjzAqAevg92qIph9ClMSmgbxc0Z9pHfrbBwSGZcGOzL0x5HpLULlpEOYToLf8996ETOaCct5wMB2kvHFvgqgVYlkQ7OVuy65nauymQWxaPk9ZMOwnBfTEvocYjyivdQ6irimZTZUhKAxQlmRJYySTqRRiT8ZSaA1ZlsOXKAoIpfhVGAij/+SXkvxK//4Cg/lLehNI70t/xV+/KLdfasayeuNWfP+//4e4XIzhFwWZLy+AIkAUAeKygL6xxf50i5c/PkJxewP76QSqw5Bymjvgbr7/HE//8IyskoyQS40w8pBbheJKIhgyIW7fodxv4R6NGUyTJqJ+FCA7AbOU6B80UI+qQcaUS++79xuE8xLVC4aohILMQX9GAGRf7XkLgD3hz/OHox0n6eKbDczHNbvmLgX6aUQ09GeWlxLdQaB/8X4L8bQi+5Kmv8WSiyyzSgxoYhx8xelnTr2VTqA658Q8Sgz70p4E1K+tYH80g7/fAE/YE7j3Y4HNLaA/9hg9ZAWHCID7xgohCJT/YgLZU2rm6wi7z5Cb9Vs9xj8pyFoEeicvvmVRPSrgSy6c9YbPv70ZUJ5LbO9bTD8waM4iZbDTgNEjBVczPGf6U0XG4Zj7rFOgi+oEumMPYQX0SsDfbzH9pzVcTUklgsDoKSfVzQm9le1BBCQw/YR+o807HU7+nwLrW/RE2gn9gosHMVVhKNi3t/CNxt6/LuhzPed0Py9i+hkTQPWGrGF74lE945ul2QLLL1mopeL1tKa0bntKP2qewustpYx7HwHX75KNzJUKbixgR/TcImYGDui/vYL8wRTlFdmlfo91CMIDF990KF5oiCBQXSSf6+ciefPIIpXX3IbJZ8DqPjB+RAZQvL2G+ldTTD6PuPgat2X9OiXXsgOKFa9RgqG0wLuI6A4FNu+1OPm/CmxPyNq1x2Tf9v844PI9gdFToDlJlQqJzUMEJo+5UO32gemn2bvK10N5TdYkqh0zqDcEMSf/Hxd2V+9F6LXE5BGZNF8D+x8wNbg55t+2R/QaT3/K87J4Exg9BdojMfQpTh4StF29FzF5SIl5KCMOfhixvpmlnHxtTT8BmtPd605a7sf2jJLd8pLsmfAM4un2BOoL+i55bAKiBvZ/yLRmJv9GlNdkY67fAQ5/QDbJ1SmcqGLwDCKllG7M1OvmmFJON2LwTtDA7FP6FZuziP0PIpb3JA7+2GN5T1Heuf9FX6+IZJHMkpUhdkKGce8TJlgzcTp1Qs6B2aeB0twRWbP5Rx1efqViQFbPKo3NDUpOt2cC+x8yZXT1WpLodsDeT/nYo+dkKKePPPqJRHMqsPexx/WbCns/DWgP6CW9vq9RXwRsziSKFdnDKATWd9ht6Wvu9/WbCsWCYGr2Md83RucBzaEcgpD6PXq09z7m8SMLmoJrHJlHSliB/Y/6tOg3rCrpd5Uidgwc/sjh+g2N+ccO6xsK+x/2uHivhFkzoffyHYXRszikKIuY2LIUlNTvkak7/JGHHTEJGZGSVukoZ+0nlBk3h7wOVc/HywE/61uKfZIHlDmPnkYm+O6neXOkfHf0IqB+YdEeGmzO6Lfe/9ClgBzed/zMoTlUaA8ZAHT8rzusbxdoDwXGTwJ0S69wfU7f5Oq2RnXNa6rfS17kY0oS5x9bdHON8pKpvO1B9joCxcpDdgG+UticKdQXDMdhzUaAG2tsTzRmn7QIBRNe+wl9p66WqM4tQinRT1SS7VL2Wy4I9ru5Qj8TmDymx9LVTGBmHyp9rvOfNFi+XmPypMfVgwLTx36oy9Abhu24SgzpzjltV3UBzbFBsfTo9xS8EagvmcxqVg7rWwUmT3q4WqGfKYYVTRXmP15j/doI0gPlpYVqHdzIoDvQQ4iW3nIQ4St2Nuqth51qlOctIAXWd2qYNetBqpc9ugOD+nkL2Tos3plh9tEa29v0cZqNh2oDmiOD8eMWbqyHBOHqvIVoLLavz1AsLbyRcGMC3vHH19je26OX9nmH7qiA7CITkkNE+WwDESP8uEBzs8bo8y2Wb0yw98E1mttTVM+36I6pwKqerrG9O0VxbQdvZn9Yw1y26A8rlC953+Kqpady1dF7Oq/hRhp666CWHaWvFb226vk13K0DqEWDWOjBixmNQr9XAFKgfLEFQmCy7MV65/+rC0QlhqRdse3g98fQ5yvEqvgCs4gQEUblADpl2yOMCgw1IiFANASr8H5IoIXREOvmi2xijGQaWwbziOUGcW/CQB4h6JvUGnCOftD1ln8zGTHsqLdAYZhWvFxDGEP/Zm933s22/WJwT05j1TqF7wQmzKZaEVFRmhOTxxIA4mpNEJl7Jn3gfqYk2OgcRJbK/hzPJIA/24eJL8pmY9f9qff7826/CIzldH47fu1v/9Uzlv/kH/1X/9b3DfglB5Y33tuP/+H/9PcgEfHp5gDrvsTVtsb+qMG8bPBsM8VyW+Gdk+c4q1f4ly9vo9IORlFeem9yie8/u4NSe7RWoy4sXj7fw3jeoDQWi+UYvlE4vrHA5WKMqrLwXsJ7gXHdY3E1hjQBpnCQMmBad7hcjGEbA2k8glVQlYPWHmXhsLwYo5x2kDJif7LFi8sZitLC9hrTSYPL8ykOjla4upqwE3OhMXl9AesUpIzYXIwwP1lh2xbo1wVgJfSsR1FaNJsSk1mDtjWoKovV1YggWwCq8ihKi7rscflyhsn+FlJEdL2Gsxq+VdC1Q0wexPGow2ZTYTxusdlUeOfWM/zww9s4vX2F54/26WG1CrcOF/jsgzNUNzfoO4MYgHrco9kU+PLdJ/jJ+RG6tsD4+zXcby/QrCrEXkKNHXwvIdYa5dkWUgY0qwqQEafHCygR8fjhIRAEqsMGIQg4qzAad9isKsRGoZh36BclYAIm8wbr5xOoqYVvFNBL6HkP12iglygPG/SNQewVZOUQgxi6ROujLdrnY8SCsq1y3kLKiL1xQ/nztgAWBnHiKK01kb2iEZAjh7AxUFPL98uWzz35RGPzmkesPOrPCrRvpDfBjQZoQYRsWXgfRw5oFNMVrxXssUW916K5rjA/XmPxcA9iv0dcFBCdoIx05rkdIMMprUB35KFXSQp6aIG1RlQRcmIhVYQ/Z4eralO3Z/ZuJV/s5GOF9esesQrQFxp+Qt+tWijgRodwWeDGg5d48ukRhBWoXih6GnXkgCWl1EZNSWp3kiYkOkBsNCYPJZrjJJdNflWAwVnNDT/0xMprA9WxIzNqPp7c7xGuC+iVhBsFiEAJKCIDc9zrLfCiRKgDRg81uoMIP3P0KW8lcG+D8PkIqqN3VfaU5UpLP6YbEczmGiL2twlsb3mYhUxMqRiYzKi43aHAEGAje4H2psXoU4N+Pw2NFnKYIravdVCXZgD6UacgI0mA6urUWZlkf0yypFxabwS6+x3Kn5aUA45j8lYTbAFANKxUak7jK8/BfRSe/lMRKF8uLhS7L3uGda3vpc7dkj+rzoHVPR4TEYGgkpS5SPcrGETUz8mSVi/lMBTLnapmxaGbtAwd09v0+JZDDlcRgOeuX2AnfTQbejmFI+urG4H9Hwdcv0G/apYlZqmwr1IvqcbQmelGIDCacthQLDh8MGv6kH2d5LuGILm84lBj8og1SarndZBly4gAJMHh5iZZZL1lR2aWynb7BH1Zuge5A+5RcrurixQWNRdDTUZ8ZXinG55T6ZPs95VwJL1lndDqDqXY0hKoSktPsE/XY35tsfM0DhLBXF0SFAFVN2dYGY8ZWUD6V7nNqkvVVUk67EYMgiLYTj2YPqcp76qpdMPH9iUHJHbMMKtuTrksJa3pb9Oxy8FBmTXN+xBlSmm+DljfVEOdE9NisQuW8ymsbZVkx0lWrlpKbF2VPNo+XT+RCcxmi9QPyX2ffk4J6vaEg4hcVSM8ty8H+Zgt/b1BJxmtECiXDPdBxAD2o/xiaFmUIlUlpeMrBYFXxO6+WqC6sAMoFiEDYHzh8TIjW15YbG+Ug1ycB45+0XLpETSTXIuVT3JVJvvmyqb8niM9/dhmy0GYTFLbbk9i/MTCVxJuJKG3KWNgLGFWPtWb0TMt+wjV0++qul14kXQR3Z5Cdel2abgpgdbVkkOP1HspIiuBhq7VVLVTXvXwJX+fE3PtRKNYpLoTHyBcQCg1/y5EBCUZGOTi4P/MwMmNNfSG3Z3SBV5bPiBoeldVY2H3SoYZdQ6yS72UqaLF1/R2ShsgNx29kjGiPxqjeLEBtIRcbuH3x4AQEK1jiu6o2KXICkplszw2agnZWMRCQzQ9YklQzNcC31jktqXX1CjIdTtsTw4BQuC/Ebk/UacXk5S7AKIMIhN4hfdfrDQRAhjVZDHzfQe5L2Wo0XkCzpz+6ljjApt+n3+e5bAhJBYzJY/lSpWfrSP5824hItr+z7/fn3L7NbD867/9UgPL8rU78ey/+S+gigC/1hC1R7QSr905x8NnB5AqovigRvdOg3hVAFOH2CrMzlZYXo4Z4lJ7yMJDf1Zh9tULXP/RIcxKwE2Sd2QSEUcesALFlYKdB5QvFLoTD9ExEAURXPSOU7R5EciadgpqLeGOyMAFFSG9gKsicLeBtxLyvICwya+2klBvr9CcjyAbCb0R6A89ZCshey4w2zOH+Y80Fg8C6ucSzS2PqCKKC8XKiS6FRZQRoQxQazXsT3El0O8xPKR6YmAnAX6U0nNd7mkDupsWcq2htmTUguFC1s8dyicG/YFH/UShPQn0AZ5amHODYCLKc8lQlK2AfHuN/vMxwUxBACIiF+UIQHvqMX6kCAQqFq5LK9DP6WdjyTZ7LqsXAv2cC1MAgAD6vYDJZ3JYTItAFlAEYHPPoXqqh2Ly9ozPFRQX5uykzAycQHPCfbdTPpZZC9hxRH/iYC4I3kTcVVTkUJy8EPIVty3oiOlDBhgJz4Xk8g0uZsyKAKw7jOwjLCgb3NwhuzcsjgMXulliWCwFujk7ULc3WNVRvZSDNyv75KRNnjPPdNz6ZcTiLYKT4jqV0J9kb6L4AvNRXZC5MysMNQf9XmLE5wlYKPrtlm8G7H0o0B7yWOd6i5wuPPkMWN/hgqy8SnUIgRLIXM4NcFtzz2VzwhTebp9S2elnuxCoV/1T1WXE+g5rKAAu3EdPk6/I0HuVvZ26SQX0U6C45uKsPaCXzY14PKRNjJNnNURzkrpMU79qPyOICCaxb0uCiVxfIN2ua9ZOKAHsZwz92PuYbFB1FXD5Jfo29WbnCXMjYPQkYnWP16Ju0vHYcN8y8NDb5PtLnYe5kkM4cCCRqiVkz/8Lz5RbgNs9es4F3PaEgMmsmVhbXhEwLO8LjJ7tFs45sbef89yINDQolrxGilXE/octFq9XlAG2BA8ZiOQ+2mLFBezqjkwVEzwfOvnbchVC9iVOngb0Y54LmxQMGXCZdToXbUzBYOn1vt0xt8U6JcVe0b/nSrELbkmdrjnhlT4wHkezjejHvH6qa0pnV3ck5h8zHdZX3PYMLvoZAWqUYpeOm6oQREi9omOZOit53OyY+523JxiBYs3Ki/weonpg9NKhn0q4it47VxIADH+ffIW+4M9FAKYP2ffZHOTajoj9P97i6p0RlI0pJTsFHXmgPnewEwmX+j+lY5Ky6gJDa0p+hpgNfZfVFSXo3Uxi/NxhdUujvgoIA/hL/tIERIRnPQiA4bVfLuiP6ycS9QUrOrj9MYFCevcgCLiki8nnyN9LS7ZN9gQIPh0XehFZryE83weyv82X3D+zYbquWSdPnqP/MWiBYkVPIF9nfkiLLa8t7EQzPKuWvJYrAjXVhgQ4I5Nxpwq6CTBLC7tnUjcoF/bVJdN0q8se3b4haDy3rOc4YMBTKARM6lA1S4tQJj+9CwysUQRJvqCXMQqGd6k2JPZSD/3R0gX4vI8JdEVFf6NZ9uj3isR+SxTLVEESInTjE+PJGgxpI4rzDbZ3Z9CNh/CRVRshIhSKnsWphlkyldfXVCkVlw0QAvqjMdNdVx1rQABAskZEr3pEJRELCdmxYzQKAbW1CK8AKb1o4GcVPZ6LFm5WETyGiFBq6Kst7OGYdSFNTiwV8KOCclJBhlH2Dn5cQHbs0kQCYbHSfA0bBbVoGbbTO3obhYDoeoRJDSgCK9F5iBAQk78RRvN9pNCQq5ZJr1VBIJY8kuh5fMI0yVx7Sx9kZu2sG8J6hnqSpiPj2HbIKbJD92X+LhlkhMUaoioRmwaiqihb9Z4hPlKSeXTJ45g7N5uW4C4EAsq2xc/rsRTjEfspQyR7qQi4h5/ljsnEmCImoJjYSiCBxVc6LmOMu+TXn2Euv5BM+2fhkl8RYPn17/7VA8v/93/9NbD8S9/m75zE//h//j380yf3MK9b3Jtd4PufvwZjHKZVh8vVGO+ePsPGlvjw0zN88+1P8JOLY2yaAogC3knMZg2un85w743neHw+x960wdUHh1B3N3j77AV+8uIYUgaUxuH6coLJfIuvnDzFH3x6D/WoQ9saw8JNuAAAIABJREFUTMctFqsRisJByoi2Nbh5uMDFeoTu0ymq11fYXNe4efMST5/tU3WwNHjry4/w0dMTvHXzOT744Daq4yYNhiLGVQ8lA55/dgAz7xA+H6F+c4EYBaxVHAxtOPkRKiJaCVk7vqddM8Tn9ukVHr+Yk6HbUuaIiYWQEaHVgBOQUwttPMoysZxWotjrELyEu6aXUG4VcNwhbDTUUqO8v0T3yRTV/RXapkC4KAAFiFmPsNWAihAbjfLmhszfH5ygvUnDzL03nuPTn5yiPtmi3RSInYLcKKgbW4THI4LsCT+448RBLgxlq6MIP/WAjBBFgHlUwr3W0vs5tYhB8Dh0CkjgU84sOzlVRHG6Rf90DH3SwL2oITxw9qUXePajE4Q6cCjhBSXQEw+1VBBO4PDLL8lCpwAkWXqElQESwwmXV6cBQgdgaSAPeuzNNlj+6BDhVovQaAgToJ8VsPsO4+MtNi/GDERqFcykh3tRIxoCOEQAAcDMARsNvZCwxw6yJtsqdYC/LgATUTzT0F9aonk6gTzo4K8LmIWCPbVcrF6RfSwuFHwV4fcd5DVT+cLYQy0YI+8OLcRGIY495DKZ9BNICakXEy696dceCAJ6oeBHgdvtBcyVHIC4mwbEkUfxxJB5fCVkwp32qD8q0R0HhD0L87RgAE0j4CYBcr9H+cMazSmlk1EQePWHDAfydYTaJsbw7hb6h2PKq+cB5jItaEsCrZBYT9VyiCB70hXhuAdWBsWFQnfMienokYab8P0waMCP2dspLGA2ApvXOKzoD7jdYmlQXEtEGZmcu++gXxr4mpL03Au5veXJkmoA856L70t2rlbnEtu7DqIXGD1WaG6wSojMIFlZX3FgZU8s9LlBlBFmRXk+kMJ9Yu4VJdhpzgLDpTYKqhMDKyo9UFxzWODLmPo7BdSWsmtXk10EgO19i4N/rrF8HRg9owxTtxiCpXw6xr7idWLfbmB+XKNYpiReQzm2WRKcZUm7rziIGT0W2NxmWnB7lGuHIupngsE5aWDR3Ejn2zAwyFdJDjoDqnOgPeEworimVLd+HmFnghL4/dQPOsUgR8/bXJ1Tojt+vGPY3Ii+WJf6I/t9DoGEE8PwpDrncCHbCKJOgx23k2vrxIQ1pxzIAEB5LVI6LAc0/Zz70R5zG1Z3KRsvLyOu34mYfCaHUJj8uN1hRP2cSc4isMIod0iKENEdcDsnnwesXpPDdiACm9scmI0fR9gJWUPV8vjp1EVbXRDINSe7iqVimbtGWVEkAu8X9I55La+YQpuZw+nDgO0J5dZmHWE2HHbohsxxSNVd/V7qPN0SSIoAVFcBzZHE+FmgT/FQDkBX+Px3At08yXe3HExUi4D1DYXqKr03C1afqCRlziFKIpChDDqBc8eBQ0j+zKg4bOjmEqqNqX+U27W+JSlR7vicUQBuTLlwFEiJstwf3Uamx24ilE2BS6UYek3thFUum1OFcsnhxhAqVHMI1M0F6pcBuovox5T0FgsHO9W73taCQ4oogeVdhdGLkMKmdl2kzaFElSTz9TmBPSJQLD26uYJpApp9xedPXaau5OOaTUB7QPBeXTr4QqI5pgy3WJOllS5Crym/9SVDirzJg8cUipbSZKUne1leOfRzjfoZZbPCc4AVlEB1adEeGAL2leNQYk/zb/Y0iqWDG5G1XN0pMP9wi1Ao2CmDeXJAENUEPDc5dIesKEF4FCL1ZqYwqmUP4QLctIB0AfCRNTIxfqGaRHiysrInKxsKNTwP+0YlZO8RjIJetgi1YW/nuhsksVGIwU+ZgajYpGoSFyBCYIVJVewYwyRtHdi+MoH2GCG2HeKoHP4NKSG6VypQ0v34/G4AvrEud1JYkPkFsGMrvd/VlAD8f4w7iWyMQ4clYuB3JXfMpiWTKQqzA52v4o4/RRqbbz8Po/wqSGF/DSx/QW/la7fj2X/9XybJZ4TcKMQiQq9Y4o4IhH0LOAm5UilNjh+8bhJRPZcsIgcXkcWlQn9I+Zu720I8q8iwLXaVCzIr/NZMYMyLDSBP0Pnz6lygO6TUKwdfBJMm/2nxxxj6mDxGfA4uwrhA7PfIgkUFCEu5m/D8kAxlYvjS4CZXUkQVB5leSEBF9tgtULJkaEkJWE5WzIsX9o3FgaFjjDqZ0GiY8AkA9YsU0JGkTJQE7diB3J/oRjwWOSEyGAaEREUWMnsrZRo4kqVMpeMln196Lty6wzh0LWYmUvWU1Zk1hqqEQX6WGKX83L7kQjAYPj47TPm8OcWxWPD5gwHMkgtP3aS+zzJ71vgcZp3kZ3G3yNRbJmnKjoxQewRUF7vET7Pm4o+JjDsZYK6Z0Ft6x7KMa1gIl9zX/KGfUwB9nWRyTZJ9NVw05XMZCh5D1TPYpH4uUSzjsCBSLRekvuJiLydL5ph/NxLD8+epuFlzIa43SNcZtzGnV5I1AP2vNRehdiSG/XMjMtZRZxkimSAkqVeUKW3S7xbuMcnIpI+JHaTsitdUKhkX6TES6zEwnZ6Lr1DQD4gkZw2JXVMdr0eVfJyuSvLE9MEvAhc7MsvwAtM5VR8he+5D7ohl3H9iDWfZ75m6NBdkw7JfK2gM8kdfpmtaUbrnzY4N8yVQLsiE2hHZMmVfvW64YM1sp6sxMFdRUjLHxV7yinVMK2znPCe5gzFoLibze1n+GlIZ2zhI/HwphmqKYMRwXbJigfcDEosFpm+KAFQLJo4iMpmTLBoXv6aJQ4CK2aSi+zqdI5sHHbnwPqVjdmkQYFItRbpuc1pm7n0UgccoKqZlBiOG91KymUzz7GZyOE7lkiEt5dLDjjitF56eMVvvQF+WE6o+ARcfUS65WLJjOTB7AM9tUAxyyddfTi3txxLFOgxVF/n8AVmFwOtcRJ4nEcnGcT93So58PJFqOPK/eT64mM4Ma5QibUccfIhkT8kmtnOFYpOljUw0zX777CtUHWt+KIfk78yGH5TNoSboSa+nYulSKBETTmP6G2lj6lBM5y7u1A2squDnpHD0Kro6sXohpgJ7Hk8RAdnTgylCHGouoqb8Um/ISmbJaD4H+T1cJqZT5FRLwXRTZQN8mcHDjjH0pUwsK5NJdePgDYNZpA2QnYev8ram6zQBrOwLjQJASlKVnUdUEiGxeVACog8IlRqkmhDpvpb3jTLJUDu/u1/a96gF9NqSZYwR0ezSXtU21YhodjjaWQlpueiXjpJQESJk5+BHBaT1iRVOiaYqXQsuEDQ1FqHQgKS8MxQKqnXwlebrxiXgFSI9iZ6Mr2wcYqkgtz2iSjUd6e9l+4oXUEuIziOMDEGi2p3bocaj0Ay+6SwQ8AXgJlxgHci6A7RiomtL0BOL5P/z7CUemMMqAarC8HWTQZJL+5+2TTg/gKWY+hzhPT2IKQ2VCasKouWiLdecRCUHYCd6S4ZyqARJoCvEITkWWu28k6v1AOhEVQ1+SVHXuzedvMYPKQG26wfZ6+Cx/DmMJUxqI3wVlP7sLb9OnPuTbOMrstbo/U4W+6c8xl/49iuQCjud345f/85fA7D8334NLP/St8lbZ/G3/sf/FCPd48OLY5TaY7UtoVRAVVj0TmN9OcL0cIMYBbrWwDaGzBKA6azBalGjqC26dQkz6uG6tIpdaYh5j7AxELVjVQgANbUIViK6xF61GqKXiCKy2sJKslkAAW8EYLI5C2TcVETsU/R2LyGsSB+aAmHiIFqFOPIQa4VYpr8tPcRGI5rA+5oA9bxgtYYAZCPg5y5VaygC3j3Pmo2xpz8wAEieODFywLUBFMhEKYbPCE9wKhx9bLKVCKMAvVBwJz3QKabEdmq3DTc6qKcl3NTDHLRwz0epwkNifHuFzeMpFyu5CiRgqLtw4zDsf/bNxZmFflEM5zmYiFhGxMpDXRpEE3c+uZ7MjuoIjPzYozxX7GG09JnlhUN+LL2W9D9NA9SGzI9wBFn2IECvJOyRhVqkPief33ABd8CQmwxQpQdkR6YtVBHwgOwkco1IXrBk/1r2arlRgFmwLyyqiGLB2gwAsGc95JWh7Dexdb4kK6ZaAWEB6SgZLi/l8Piq4VAie8JyXUmoA/2bSZLtK25L1GmbwOCnDLizZ9OsmLwaZaQkvIzQS8lOUbnbrwgyegis38iS6yG8Csnzp3iOpRWDD64/8Bg9VrBjysSBnW/Q1xyM+OQJ1U2a0FeU6HqTjmvyJranAaMncgDxGXz380BmUSMdOzJ6ei1QrDgIAsjm+ILHQnW7Y15cy90AJnC7VCcGea60ZLDsNHDf0naadfobpGslVSi8CjTysCVqXoOZoZUZ9FiCxexTBDCEj0WJBIzTY/ndAEBEAmTVp4FLnybpKcjIjXeDi5AeK0t+81Aj38fVBL8M+cIgaxaRj62axBDb3X0zaxcMBmYzvwZFBqB99t3l4xjRHAnU5wTReptAUi0G4C08WcjyivePgvLxHGoTilT/kTx/lO5m+SlQn5Ph88VuQCNtTN2LTEreVSIkYAuCnCyfzSBIeAwewzyAyeAnD8rcWAzprjIdX7Pha8OOBVk0w4FJlgmrLrF77c5XJzwHNqqjLJbXAQcb0mcJdfIKCoFyGZI8eVcFIpNvNA8Msrw2D1pUl85xOpe54iTLU10lhmtZdRgqT2KqRBGBjB8k/9/NJMwmDvf3BQcEInDwUKxTZUolMDoPQzKraWICzjFd02TOMpjO+25HEqYJr3TcxqGLsbry6KdyeD0BGAAsXxtiGOZl2e3Q4amS/LjesZ15AAABVFeeUuU+ojlQqBZf9IXxfPK77ghE85CLgVcBdrwDxjw3kf7FNCgIhUjVN/Q02rGEbncAPwrA12TZslfxVS+tbgLciB5IX8ph8BJMGmYoQLVMyLVTRUCfn7eLw3YN3kkXd9vfeLixHq4lJE8kkK4Zn1+XSZKcQHSW6hKYK/obSzKYUYoBzLKiJbGJiRXkweIAJBQK0pNNzD8XPgwMYdTJi5iA8KvVH/nxglGIhYTowwCoWJGSWDYlILrMIsgBEA+1ITFSEpslry4MktlYGkpzlUidlYr/zl5JJb5QMcIfRgI5t7uWMojNfknEOAw8CLQVAW3envz/9Dcih/8ouXs8H/i8mTnMjOQrjOXgscxhO8l/OWxXBqQ5vCf/fQLSse12fsncS/lqaM+rXZXAzwebP+/2CgD9VZDC/ju//VcPLP/x//6LASzln3+XX9yblgFbW+BffXYHRhFU2k7DOUVQeT5GvdfCOYVmyxVeMe4RrcRo0mH90z3EXsEYj2rawW4KPLj9Arp0+N43f4gYBSana4iLghUWlcfBfA2hAtRCYz7fACJC7PVAxReKHFvcv/uCQE5Gyk2vDcRGofqswPxgQ7lmEDi9cU0JZsEPhurOCggC6iDFQk881FoBAdDPCy4+ewmoiOKzkiDNC4LKqUf1qIAcOahGQN/dACoCexb6hUlJqArCSlRPNGIvGUjSC8TaE7CaANky0CbqCGiCCXPFTkdVesixhdQBopNQhYc7toit4gdKK2GvK8QioHpiUFwqbJYV1FqS8RSAXkpgzyIe9BAW0EctQaOJiEVEeSEhtvSM+Bsd3Jxv9GolUTwzCHVAmDkukooI8doGJ++8HECl7Fl4r9eC/ZJrQRAlAb/nEMepVkNHAlyfWKG5Q3/Lku0uI+BYyVHfXyLeagnKARR7HZm2WSDrqQA7C0lySdAfTjvgrIPqBYorVkEU12QCiqWA3SOY8hUBRSgSs7RhGI26NAOICibCjvmhXlxKuBsdgWMCnEGzf3GXfkrZY3kt0B9QElY9I0AuZ93AsotAUCctgAjYAw/pmOqrOqC4YsBOd+wR396gvJKsTfEpFXQUobd8PrLEfNPPQCuU9M1GjV1wiwOkJzPPfsGYEpeTx3BN4NkeB+iWnlRh6dPMQLFY8Ht7yIWscFzI2lmEXglsbnEBUz+P7ABN2yAdAaNZc/tmP5GQjkCCCyQCY6Z7CtQv4rCIzSmoeR/ZZ8dBTAacxYLguXq565y0U7LzAEFnMNxuNyV4MEkayqk/uy6l489zT5/wwPJNXlf1eYTd43O+eqzJEgOr18OgimgPeZ6b4wjNtymmR6d3/Polqzfy/uUQFZXUDf2cQCAPTwhs0nHecAEubYRqAEgqJlzF4x4TMMvAXjgm4wYDditOKGPN3Z12lABoQRDoyyTt9KnL0QDzjzzBoI1D52Q3Fwlc7RYcZsW/eRUYDQx4JJhSzU5pkqWVAO+3ej0kH2cajCSQnoNJZM9zZydIQw6mmoaCjC4CYGeUMWb1RGawg2H4Tnv0il9SYQiqyeEzmfXOYEfma8HF4ZjSY05QPX5qeY2WOwWCHQmyypIDAjcSyfubXq81hrRxXxAQmi3llHay80Wv7okhsCpKpCAb/k636TWfwLirBdrj3b4NSoA0+FA2AfEUUCNTt2TUQD8hG59DakSgxDUqbp8rBVwthsftp/Sm9mMucqUn0AzMZCOo3YZhaCLdLh2XbF86f4WAHcnU+RqHAKl2T8KOObzKnZFkRslodntkrPPz5us9JOa2nfP587nKiqB+IgeFhSsFRk/a9Pw7hjGzsBAY9if3pJLlE7ATBdVGmLWDnarBd9ruEyS6Wib/roTsyShHhUEhwMEIWUS99vTgTtXgc41KoD00CIWEnTCllgA4dUama1avLfTWo9vnZ4yvFfqZhq+Sd3PD0J+YACoCoFqXBrMSdkZvphspsoU+Dgm0oZAwVy1iYoAzUA2FHO47AEcXBtbeTlNwTecHySlcgLCeIDWF2qiNhXQBcpt6K0dkdUWfwmSsh+wsfxaRzo8EXCALqsQA6oT3ENsOwkeCzHyTkl7OQiNU/BrAXGIfY13we2Yps9RUK4LCzGgWhiCvLoeAHDgmZ8P5IbgHztPvmTyRwrodqCzM7m+1AhTDemKM/B4CEChzHcCjEEx09Z7P7zxlsPlnGZQKufu/FPRSDrLZBDIB3lcpfuWfAX9mIuzAqMo/4z6/vv3C3H6pGcv6zZvxK//D76OzGtu2QHddYXqyxurpFGLsEJ2EujDwew5ndy7x7Ok+PXRlAMrA8J6CDBpMhFxoyE4MstjuzEKUAfLCQPYC7lYHcZkAnhVpUZsnSEgphxJ236N8rtHvM5QlAwR1c4v4cEw/21qjOpdwdYSbBYheQDoG9IQUCBQMJbpmkaS9gQuH/iwlXi752Dm0JZT0IkWd2QNum+p2nXeujtj7icT6dmJpNiLVApBJcuNIv9KCYUR8bDI0iPhCTUv9TA7BOLIlU5PDZwZ55IYLw1BE6E2qgCgi7CyieiGHVMfiWsDO4sBshVfY08wOEdDwWFKuxbAZOwXPYZ5MpwVQMGlh06RJnCI7OADNEf01xYILKV/HQSrdHSWWK0m7kKWlJia5aJL/JfmkL9NisIyJ7eDxsnv0ROUwkszmQOwe01c7diczRVGyJL055uMoywWW2STp74b3t+PdeVd9OjeJJZOe/3djspn9fkT1Qgzbazb8rpud/JMLWKDfB6qXu9RLO6U8m4t7DImoedGpevq0CJZ3kui8QM3n5VVWLCZAmKXSsscQkiPCbh+F30mOdfJJ7eRyGNiAUCRWJDFbr8qgg07HucGQJprPX2ZkhpL7tMDsE0CB5LnKICO/NoslfWWZIdt5+OLQnZf9bwAXy+WCC1w74fm1UwG9iYl14zkMyeOVGYHMWGWmSreUU3Kxy9+5UQrFSYCEEtsdW8/wkp0UNgf1dPPEKiXwbUfs6etmArp9BVwmdi+/tmICxEF9UXrKJMowBOdkX50veZwyi035J69f6dK2Innc0nWiux1rZMepNgUYJJd8viQTnoq0H2lBnvYn+9l8AeSAHL7essySz50X2tmTluW+qnuFtbQYfmZHBDki8jnLZQrsSUyoTEymtBGbM4ViSZ+dq3hNiSybE2J4ziy7HqTSandMiw09a1Fh+J6P2atJsFkanFmm7FvMr8fMWjPNmAtCltxjSMJtDiVGLxlWI/wO4GX2cEjl9bwOzSbsGLYspfaU3fYTDh2LVUA/kxwEJHCZ94Xvizs5LfAKi6h28mPZM+hGuN15yGyrtLtwm/wY+XHooyNDR3CCxGiHIZBpkMemIULQO3n3q4+pEpuXGWs3klBNSNJiMSyyKcvloEFlMJak3NIR0PlSUWJvE0tZUl6bk1QhgGJpEZSE9Azlybf8PiR9QD8zKJaUfg5UQQCDtMCBRFTp+rIBoVQIRkJ1/gupqcKFBGgJ1DIL6MfsoaSVJrFmiT0KlYJeW6a1+v+fvTfplSxLzMO+M90pphdvypdDZdfYxe4mmwRnE4YFwRIEaGdAC2+8sRfe2Ibhja1/oZ139so7r7wwLA+QRdCAQIgU2RS7u7qqqyrnfJlviPGOZ/DiO+fGq+4WREpNsItgAA+ZGRlxh3NvxDvf+abAQJ4mVnJEmerIhCXWL9oW0meAQUuUzab3eaMY1DOQCUwsZHoEISATAHQhAs8I0AUOUlkXxmtC4CYh654gL9NQNetG4MLhe965w1jGsQhGjYBR3AnmCXlGEJdrspkApbixeiQoFbcXDmAvMo8h1YoASNUj9IeS3QuZIeuYGD8VpbFSEhwmkJWks+Fwrl9JcfVxG0odqkbSNoUgmEyPu7LUGMYjtGbVSGQ+kTyYKbxHST4X4j4GSyCZjvluImysL/nK464E998GHJP09uvOWC4ehV//D/+bn/t2f/9//+//2s8N+JozllIEfHt5iceLFRdBpgN2mxKidCiqHvCALzzyoxbdoBlaUzlKVq2AmPeUc075G9cfDXCVh95H1koC6k0GV3rYiUfoJfzMIlQsiXcTR/kjAAQRi+PJWnYXA9NWQXAoPGBbwy+uvUKYWbSnZBiSlE92QHth4fKA/pjl1Wp/WIUNitI80TPFlV7KQK9kS4ZOOMBV/FNctJBDDP/wiIBRYP8wwMfzGyZhDD7pl56T+8qPoJKyRaa10mcaYO/3CCZg/66FcMDR4xXMhrUMw9LF7ssAOw1ov9Mg/PKWK+dVQHvfkb1UAf08UHbryOwkENFekHVNUleXc2LT3PMHP2cn0M9jCMWE52InAd2ph26Afk4/m4ySxu6Y73UZgayrAoYjhqr0cx6rcGIMbwEQwyo8GUPD8evjsfUzHlMCgcMioD31sFUczxnHnLJQHltiCtozlrrbisCuX3rYiq8ZpkDzbo9+wS6+oGNgyjzATljS3h3T79sd8xrpHf2vAEaWxVUhdnl6DLPkfQxwZZRGDkAfS+xdBnSnBPvDDICkl9aWGINO+hOP9oQAKKWScqWZY+4iU7Z/5KEaSk274wCfHyazKVEVILMEAMMRF3Zs3KaLry9uAppz3hP9jIsHXhNA1fcPrE57Gr8LbMD+IZlOEdI4cOLZnKW0ZoyyuP1DjguBA9kvU/OYuiOyI7YCth85elS7gG7JSVq3ZBfs7hFX4INgEEtzHjBMgGEmoiQR/A7KgO6IjE59LpnEqxmQomsW2W/fi0zhnGyY1zEIJE7ou2OBfpmYxCj/nN1hcVpKPO9WE7jI8riMHrX6nkBzLuCNQH1foL4n0R0zfTcBAp8B9Rmlv7tHBLrDTIyhJu0J0ztVfzgOb2KYyrHAMCcAthMxSmeHiRjrUUZ/qzl4Pb3CCIDshN2L3TGPWw2s/7CliN2bAu0pJ/3ecKGgn/G5IHkMQabFjoBuFkN5SoFhFsMyBCsUbDy25C8dpgL1mYKtxNgHub+n0JxI2JzPDRXfkzpAbcHttws5LuikBRqyquyI7JY8Hx99xUHE1FobRqlqN5ejP7g9SuCGYUVDJdHF54YZWdJueeis5K8ggXYpMVSUabYLgoGhJNhOAFt1ZHqaU55XtxCjhxjgRJ3PRW+y4aJAP5UYSu5DDel3CtCcqnGBqZ9KdHMueO7u65E1pVQzLgpIgf2FgtcC3ZHkcVQSzYmKrKUfF3lGsBoI4oTFuHjZz+X4PRSEgNlTototFMzOxe8TOfo0fSbjc6zfaJdk1lwZfYqSz3cLAjxbSLKwS4V+JmMiMD2MACB7dm/6XIzgCQDluAXHTSQ2Ln43+wgO+5lBd6ThMwE7VVFGLCkHVUmaCbTHGaTzsIWCz+ToJxU+oFtq+jkV2TQ70XC5Gpk7WyqIgWC6OzbwWqJfZHAZk2zhCcC6I9ZmDHODfpnBTjSGuYGdZhiOCtiJhi25fxlZv2GRwc4MU3EXGaWeEUgHzfH0paYHNJ43wZ6kFLbSCEJgmPK4fKbgY5/ksMjZd1kZuIkBYrqrNwrwAcOUgMtHP6Wb0DYjm4Hs7sSMz1H+Kgn2oucvRLAIyURXNzFw0wx2xv1CMClWWAK2EI8PSkD4CJatQ6hy/tvcqQQxiu/JWAciYqVHKAyZRgBhUhBUJg9jkokqdQB7qToksYsqXtcENBPIS1JY78ftj2E7WhGoZQYoi/F1MJoy1zGdNYxsJIIHvAO8g9CaoNJ5vl5I/mQmBggFpsXq5MWMioA7/kehDhAjMZJCyZ/4UWQ3I8splIIQ4mf/xPf87eMX+/G1vkLWSfzhq8d4uj6C9wJFMSAM8Yu7jymexz2GXmP15AgQASJ3MFUPyADfaOhiAESAuja4uCBAOvphXIDXlDsKJyCOe/ohtScwVIFdghMLf9oTyA0C9oh9kCL3oycuTCyEFwidRPmWABSthDztCBYnnr2BGtCLHm7uIOcDwabF2DkHCbiTAeULBZ97MjQTDzd1CDIgW7b8pdvHBEwR4Moo0e0FxEBZ6HBs4XMPV3i4+x2GY4JlX3i4KPezpwOG2WFlaZh6+hAbdkBOvlBA7uAzYN/kcFVAmDhkbymbFV7AzyzwJoe1TCTtTxxmDzfINoDeSbiLfmQxh/PhwIQNrF5QPVmTEP2Xei9j2l1c0Y9AGoHVIy5nuE93HOCmDgJkvYIG/DQC9R6QsYcyf6OQJKWJkQIAnwPZSsJlAWFmEXRAP6ccMfkrhyMPlwV4TbAmLJDfsnMBZJeBAAAgAElEQVSsekXfZmLLhpnHcOTGUnk7dchvyA4DQNCsEEkLANlLg+kTifbco1/wHIeFR3EVPZSlH0OgXBl9WIKgzRvuo1+6KG8UI/Navpbojj36Bas01ECAZCsGVMnuMDkqrsXoC+xOPMwtJ4vCcqxVvC6qFcjf8rx9RqZWdWRIhWU4UmJgbMnaDwIOMjdqKwlcj/zIQtqKgTfdqYtdjxj9Yfkq+h8Lgi+fERR0R4Jy3Z7HmPyJXvMYZewdTJP/bM06n2Eu2I0oyZalcKYgee9UzxXGkJMsjN2MEIgr2oCd0mdodqxw8foAkLNtYKKm4UKALXEIhlpGZvSUK+T0PkYWRnB8+iOCArMhm5KtgO4Eo8d3lMvF1fbE4uqG19VWGDvqUi0NAKQ6FmHZGalaLsQAkb3NY9VPFsc5ehx1rHmZvBoO7Gcmoi+QMlkXvYD5hos5SXKZfIXCE4i2x4cgLpf6KSPrm204VjYy4CbWv9gCqF77KE1DXNwRX+mblJbnOZQsqb8b5tTPItsZFRApbIgsBxlOYQkyGfbD/UvHLsUk69QNxlA0byiFBciySgvWl6RgmCgdRQDyDYHSMOH2XCZgKzF64BhQxc+HafxXxiddu/LKM8ApMpbpfnE5GJwlgeK65zWXQJaqXypKSotrSv9SrY0tBfJtTM4cAmbPOQE3DcfcK4Ixl0UJtA1k/xJhMn5+YjpoDHnKN2RmUkdlSi61uYh+Ti5S5KtDz6TLxRgQlB4qVowknyn7LiNTGqWb3YIAZqgkihsXwYyI+0pME2tQ0iLGyK5G5lnv3Rh6VbyuKQ0WlOl6LUZAHiTIUoaA4saP8lfdWLgYfpRCkdqlIrBV/Hx6w/PM1gOTaneOr9dc4Ejsouo9a03idw9/UWCUZbPPEWPAnlcCsvfoZwQgLicI9bmKzChGf17yQ7pSwU65IukNpaaqcbCVjNJvx9qU5Lc0BHiuYvKqLcl6ShfgjITPFfS6gRwY5mMLRZ+jjxLXXGOY6uidjYxnRsaUIW08Bkp35cjkAqBXMTHRUrBXMnkyNb2MPtaa6N0QFSw6BuT4A3tpPUJBqazsbGSSyfDp/QDRDQhSQsbtBXPHEJ9An1L0PYJAMsT9hijtFL0d2U4MFqLrx9eOIT+pW/Ingn+QQoHEncUKKWMIlSOgjImuIQK8kBsgAdHMEFRGljK0HVKtCFI67F3GMAQIrSCMGYGkMAbCmMNrvONr4g8SYy0FxiAdFWWwifGUMjKsdxhQpeinlPIrElh4fwj3cY4/IfzsH+d/mvH8uj7CX8HPL8jjay2Fzd99FB7/l/8dhoc9VOZgMgutHbaXU8i9wvyDFfbfX/LL+XGD8KqAPxkQLH8ZCRVYWj+xKH6c06uWecwebLF9O4V5q2Ef8EtBXWajFy6/kRjmHm7iIRsJJrHKkZXL1pyg2pMBMncI1zlC5VA8M/SsqQAdvZO+CKheSNQPfZxcxvTKLScI7cMBcq8YoLNR4wTFPugQWnomzUqiP6Ef0y0HlE8yDJOA6jL2Hk7ZhSkCYB7v4X48JUApGF4Df5BR9ucWk88NvU8LjgcME3cTeJ68FNi+yy7D+j69YbsPB2RXOkouPdgFKTF5xuj/7bsESNlKojt1mH2u0JzHcu2SEmJXBtiZQ/VMY5gHmLWI14QS1eKtRPPIYfJEYffhgMnnBu2pp6S4OoTjdKcePvcoX2q09yhHLt4ScJpdnFRkwOxpwOqXOHFKPYL9/FDBAM+aBWmB7TdYO1C9Flh/7FC+UmjPPMo3Et2SYTJmC3THQL4C4IHNR+zZTGDCVgG6FSjeMGafDE1A9YqTBVuQ6Zs+YYdgfkNg0FwwlCbJRVV7CFGp7wccfUJ2STcELwB9ibaIibQVn59/xglrc072W1oBl5HN07uD3M9Oue9hwkCc6RM5etdSsAdluOzyXP5owO3HBs05j2X9IXD85wxg8VpgmHKyX19wnKRj5YLZCCw/s3j9u4phPBlQXAWYGrj5ZWDxKdmqlMIre0TPZsD+AYHc9LnHza8QxM5/DNT3KftOsmxdc7yGKTB9EbB7GIvQn1I+Wd8PmLzkvd+cixEESMvKBm9EZPoClt8Htu8Bsy+ByaVFN1fYfoO/IOdferTLJOtjZ6R0BELlG06uu4WEqQOloZG9uvpthwf/jJLy3UOJySt+TtYfyDHwpHpDsFNdceK7faTgSiBbM2G3nxG457dkJc2eHkpnMDJ+wvP10gH7+wKTV2FMmw2Sn4H1BxLFNf2IzbnA8Q8sLn9L4+TPPbaPmCZc3LKbk548MhQuJphOXlvsHmimHt+EuIgTE1/rgPX7Mk7igckrj803JKbPwwh863OmJTfnAssfOWzeVZA9MLmkbEu3Aev3NPJVwFABsxcOq/c1TB0wezrg5lsZqkuP7WM59rKWb6ksMLswprgmqastBCavLa5+1eDkzy26hURx67A/1yjWHvt7EpNLP4KeJHHtJ3JkCb0WKFYO7VJhTDLV0XdpgNlTh/KqxzDRaE41VB+QrR12jzQmrx02jzVmLyzW72rMnjv0U4l+LrD8pMf+vuFn4pbhLcXKkTmcSZRvB/hMYvW+Gb1nk9cMY7EFmdf5U4vmVCFfM922uCVoqM8kzC5g9rzH7kGGyauei1FzjX7Kc8vXCWQ51PcM2qVEee2h+oDydYubb1dQPTsxvRHoZ5S5Zluyabv7BtWVhVkPqB8U6KcidkESNM6edeiODCbPa7hKoznNkK8shAtYfZCjWHt0MwkdtwkQvHZzFSXlHtl2wPZRPiYk91NB8N96eg0dFw1Uz7Rbsp8SzgCT1+ySbE/NKIP3WqC4tdC7AZv3Sxx9so/eRNZLdMc5issa62/OkG15TabPW/QLg2FC9higrFfvBtipGZmx1LWpa4dhppBtLMxti/rRFPltj/Y0Q/V0j+bRBHrvYLb9yPLtH0+QrSzk4DHM2a2Zrx2y6wZ2nkOvOgzLgr2R6wb92QSpZmOYawgbUH65Qv9gDr0b0J4WKF/u0J1VcIXC5Is16sdzFG9qBtYoATfJoNcthiWlJHrXw85y9AsDCKB83SAoMo7ZdQtfGXTLDOWLPRfASwO9o+Q0KKa8dqcFzN4CHjBXO7QP5yie3KJ5/xjFZQ07zWBuGwQt0Z9UyC938KWBnWUIUiB/vYObZFDrBvZ4AlX38KWB2rTsuxQCqu7H2g03ydiNmfGesbOMCbmaYFZGGayse4imA0KAOz+CGOiX9LOCstlJDtEyuRYekPuW/7eu4WcVZNsTkJb5IbX1Tnqsn5SQ+4bS2dhPGaqCrzGalSKZoVS2zIGup1Q2VomIlr2foukQJiVE3R4mwUaP20HbEVgKwaoQpRCKDGLfMJCnLAhgYyosQJnrGKRzV0YL8DXOQRiDYN3IFIYIlJkw2xE8Zgaw7DkNd32XCV8IwYqQu2Az/MUBYvjJ5FjvfvYL/wKPXxQp7G/83s9fCvvP/49fDCns1xpYlh8+CH/3f/pH+MGzC3z08A2u6wk2+wJaOzgnoZTHg6MNXq7msD+cwz5uMZ832DcZlAroXlXILmoMvcY757d4u53gW+eX+OMvHiM4gW+9+wqvNnNs9wUmVQclPTa7Eo9OV3j6+hiLRY19kyHLHIxyWK8rFFWPIhtwezXD0ckO3aDR1hmwMghTB1gBtdZwM4ds2aKvDWbLGtubCbJJj/66QHleo2sNPn54iR989hCQ9H+q+w1TbWsFzAYUkx7NVcXQndxBXDLQR04GlJMebWsgRYC9Yh9KUARMIYtS4EECOrB/stWAo2806ADIALVRI2sV7nVQzwvYuYdey0Nv3jt79M8m8POYZispSy1eabQPBojSQdxkcbU5IFSOybk5vxjkrQEkEI4GBgo1ikzSxQChPeRlDl95qK2E2Qk077APU28U7Iy9hv2xiwxSgLASkwdbtJ8sYOcOsqNs2BdkHNVOwp4NPP/CwTzN0Z86BtPEVZ+gAbPml2h3zwK5g7w1DBkSAWajMCwtqicGzQOuOJuNRL8kgA+n7EjM3yq0DweojYJqxcgIpeRalwEp5RSSAUbVE41hwQl7cS2w/cAiu1awE4YbwQlAB6g9J+nDmYUsLMJtRkb1hkmu7aknaxgwdhgKDwxLB7NSZK5nnl7R6D9LARM2gj47IVgXFtHTRfDZHzEBV23jfVDx+Is3ZKZTbYmwACRBa3vuMfuc3sLuhOmpvCcxss7ZhqCwO3coX2jYmL6arXhNmrNAT7DgMcoOsDMuzPRHAaMH0PFYyaQwTTV50II6sEfp/IKOjNUgRk9xCjXR0bua2LBUczMsCF6ZuhkTW6MMOSV52gnHsV9wQSNIjgf9iQR01StKpelJo+yX/jeMgTq7x1xY6BfRM9zSa52Cg0YPqwATUA1luWbDa5ut+f/J65hkpMOMLHnxVo6hLKn/D2AAUHEdg1zcYUGBdS8Eg2ZH+W4KOGKqMNnzdFy65sKAsFwUGOZiHMuvpMrGpNH6gos0usZXr1sck5SGayexqsaQAUye3CD5fJAx4OfOOKWKITmwT3GouODQz2OK7PEhmdfsA3aPgeJt7HQsEPsqY92KOnxm5MDxHaYidj9G4BlDU5KvWTjeA8Jye2YXRjYsvW6sGImAyWugvAmozyT6Ob253ZL3avKXZtuAdinGrscQE1YBsnHSHhi65HVNjKutGGgUotRc1+HAjiW1qeQ1JuPM/zd19DTHeWb63MshjGmr+3OFyRuHdiHHOqrkYbYx7Td5X4Pi/QFEFnXtRz9tArbOMGSoXRKgmj2PJaWwBomxixHA2MeoY6qttAydoiSX8mDdhdGv2i1in2Ht0c8kilXymvK5+lSR1Y6eymzjsLuvkW/5OhVDp3z0jqYalFRFw88hmUWAPZL9TI31MynhNqldso0/pOMW4ivprWTagOaYxylTAmvvoaJn1eUE1CnkRkZpLO+vMO5PdR7DXEE1nn7TPsDnArr2UI1Dcy+D3vuxjiX5Le8yy0xH5g1jdkyjBYD8pkc/NzA7i/7IcEwiKxsEYHYDfZ8qdj/mCrJzGOYGunYxCddhmOoxkdlsBrhSQzUWPsp/2VMp4TIJs2Ewj52YkfmUPStMUk9j8jvKIfp7jeIcYT/A5xqyswe2ONbW+ChvHYOBIvuZuil5cTzcJIeMYFTd1vBVzs/94EbZLA8ikOkEKNVNqbBSHkJ7gPF50Q8Ik5KgtRsOnkvnKJ/9CelsqqcZ95Vek5jDrsdXakGSDzIBz7vM6R2sEEL4Khs59JTLRnAJSS9lYiHHc/I/XRUy1pTEXsxx+/+mhw9/IzyWfwssf0Ef5YcPwsf/5D8HANxezgERILcaIWM1hdgryk4DIKYW4jqDSiXsnYS/10G+yQ4TPhNQPVPojhk00y8ZnGMrJgEGFQNeBjHWKPiMbKXZxgqDOkocvYjBOIHeLnD7ZivRnTgCA2AMLEmTX5eH0UeSZGSyOwT6SCuiPFIw9ETz2Nmbl6R6kT074oR3TOy09DYiiJiQeQgc6RdxsrA/SDSZTMlfPGTjuL1hzmMEgMlLMnjTLxT7HxVf0514mC3DiVzlUbxRY/iN3scvqujLSWEoquPf73Zf2kmcgJYc62zFCTy7QeON4ON7BcZOx34RfU7+MEEKEvD5IYTGZxF0xF/k0hIM5NfsIFVtDBJpMQZepDL0lPKYahhSSqTqDpNI1Rwms66gjJHpifH0/Ve3k0JmuuMwMr/F2yhBja/L1nFMGnod9T766+ZivG+CxBiykyohsjVBS3HNDrPk9cpX4cAqOk6oixvWPpg9Jx/dUfQKldxmtsFYP5CkgmbD7do4WW5PxBhgIwdO9LujFKJx6LMcptxft+TkXQ5MDhUBYxVFCpDJb+M+KiC/CaNnK4GSPham+wxjOE0Q9LdVlzym4pr3eX3BiTmZrMOEO6gYuBMLxocJJ+26TsdzmCCy6oEJt+0yHtdtkjOSBUk9jwnwJi8kpZ30rg0z+tlSf2iSZ0LEseoC9vfkeO8lljHJI6U7yGHTNnUXxgCdFHyU/G7FjY9+OzGCDtUhShExJm/aksxee0yGS/V8LqWtpqTY9PlMCawJfAvP+z7JQVM4jI4MkW7I+iTGSA309s2eOzTHauwNHaKPNCgCiW5Bdj6dYwpmKm5iiE/N0J/yxo/BOzad+4rPDRV7RW3BeguEMLJaZKwZGNNFAMB/Y+w6TDJXl4lYlwBedxsDbRr66vuJHENgdHdI5U1M7t1x7ieUlBerQ1WGu6NIAyjVrc8lvy9isE7qCE3XR9yZt5o9r3XqciUgpJR+KNnZOUy4/xRCZcsDQEpAUThWXySmvbjhhF8EpolmGwIJF6W4qYM0gcgEtlR3+Nyqjix+tiPL6Er671Iv5lizYu74P1NHoxbjdzuQFpTid5shIHK5jNUvnpJLTzmoy+V4HfSeUs5+rsfz9IbgbJioUT6agpgAQHUOtlAwO4v6IkN+yyAV1TiydXd8jqmnMn3HpPMYfYcDZd2yi4CqJzufgnBsqWG2Q0xiFWP4Dc85LhzEbfB7nSemGvZH6nqg/7LSUIOHVxJ616Nf5shuOwQl4UpNUKbZn6kadzjegWmrcnBwpYYYPFQzwFXZIY019mCOvZQ9fX88b7K9PqcPUVoP0Vu4SQbZ2VES6XIFve3oc/yJgJ+xazL6Hl1poNcN3LwYw3mE5bjBBfjKcExid2UKEBJND1/lBwCY/Je5GqtFxODiwo8/sG4Ag4HauNKT6j0iyzj2WmpFtlIIiLpl/UfPHs4QE12TxFVYAkE4h1QLkgDiCOSSJ/JOKBSiZ3OU01r3VdbyLpsI8N9J1joM3EYCoo5sOn4WwwgQKPpwYBaTBHX0d8a6ESH49/gIbXfwi94N74nv/1keyb8UDvmbAiz/g//6577df/5P/4e/9nMD8PX2WHovULcZ6jaDmfaACjj55jXUsgOcgDzp2SmoAhaLGsLSoxYKD/GwgVQBPklCa4LA7jjAzh3rBGRAdzHAHQ8MfMm5cmi2TI71RYCfOPRHHm0MGmEhPTsl+7lHd9/CmwB7NkDVgl49QQ9jQEqxJBBLvXk+A5NMg0B/auGqOFnOyTrqOiYuLgO6E/7iqh/ZOyAmoF+QnbRTbtNV7CLsLizrDALQHfsY2BJXsC3Qnjm4nF2EPiMraScezb2A4cQSBO0FhlOL/FagWwKqlmjOQwQl9P0JJzDMPWZPgPxKoTvx6E4c9J4y4ZR4qTqCWoJBVk1IK9AdAxCs8hgmDPlRkYUSjgE49JkF6PbgJRMe6E4C7JRVEz5nKI0rGH6TrcTo5dJ7gf6IE0bdEFS6IlA+XIbY98jjsNMQGQYG4njDRQZbRllr4CQ/HZ+uo/euPzBk7Sm3oVuOdVrQMLsITieRhbkRGBYBk+ecANWPWAWioydK7yLIXIVxnGwRJa9LBgBl63gvRIDZngDVJcFMt+C+VE+w1C15DGNQT6xL6BdAe5xAV/TYNTxHrwVZqQBMXnAVvDkj6GlPBUwdQZbgdelnAsPssFhS3xexXoHALz3kwN7C+t6BKdQNx8uVYqwrqC/oTVNdYDBORdDcLQW6Je9dbwRcnCS3xwKLLxy6pUB/JDB9HqIkl5M11VGC2S9iamRBL1u2IQDefiN2280IcsyWvqN+QdYr2wZkW4z+xu6I2/GKbBArGCgPTems+a0nyNwxLdZWDK7JV5yU21IwCKcQmFx6TF94FLee91yUGduK4Sv9nAsuKYQmSAbfpO65BCp1HdBP46TRcb+y57g3ZwT++dqjW5IF2T2UlF/OGbSSKk5YgcH3es3zzNecOLQn/H5SQxiTWqWNoUgV0M0JTup7lPbbgtdrqLi4sHugkO3Zw9geUQ6Zbz1MzXHNN9H7qXjddccxS1LIbsHzqM/kWGmR/GjDREbJY/Qp7zyaYzGqFbKdx1hDUUqUVxZqiKFsEWz2EzmCLtZieKj+EKKTgnjaBbsHi7VHtvOU3Lee4TxLiXzl0B5J5CuP5pgAS3cB9ZlCfuvG45aOP7qllLK48ci2Ph67jD2WvFcSME6JssMkBqmoAzBqlwq2YB2GLXjdUn+iyyI7t3dxAY+AsD7TsLlAtnYobh2GqYKtJGTnka0duiOCsGznYGrPAJwujNtNlRz9lOOfryihLW4c8tXALsUuwOxd9AlirNCQNiBbDXHhRMJWrNcwtUU/kwzWiRUc9DAyZGeoZKy64P8jsCtyPMetg88kbBl91DEYR0efoRzou1Sth9lZLmyse9hSjWxgvqYPkfuPHsKYsKo6x95GHf2ORo4AXQQgW/VwBdNUXaUhew9XHnoYhQ0oXm7H/UnLdNn2OIMzErJ1Y5BQ0AJwAWbVQrVuBIv9IoPPFcymB1yArge4iYHZ8s9hkcGsW/ojtx3Mpoeqe8jBI3u7Z9iQBODp//S5QndaUnIamMaqdz3E4GCnBmrfUz5qJNS2JSgEIAd2X7pYvaE2HVxpIFd7uEpDtfQ0qs5Rvqol9KomkKwMgWlvEYyCagb43EBd7zimrYUvNFxF1Y7sLIT38KWBzw5eyOFiRlZQMZyH4IxMo9y3kLvm4KlM6b5VxuvhgdHXKPmnaDuEXDP99W5Ca3rEEJ5Q5oD3ZBhTB2TqpoxsZcgp9z14N/m+kMcQolgTkv4fUh4AaAKTXY/Rv9hHEFzkBJTDQO9lkR8kqeNx3tlWYiiFIKi0lnLZEAg0k+Q1AcmxGzOG+6RHYinvJs6GGJj0M7yTfG34i/38JSS0f/v463l8rYElINDXGfrWwL8sod9kWH3vFCdHO8ALuFqjuJQQrcLqehr72OIH6FkJ9UXBwBITMJxaqL2EnbtxJXTyTEE2CvqtYd9eEQvQe7KIeitQvDIoriRc6SlRiyX1xXNDRrCRyFYS1WcZ5XAKKC4puUhsDgKirO5OlPvA0Be1pa5feAIO2bGncPKM1SS6oZRPbxTMVkDX3H+/9Ehl9GYnIXuB/FYge6shB6C4Esg27GrUe/YtZiuB6ZeKclgJmI1AthbQe6ZE6rVmbUoHmCvGoSc2VPUEHvtHHrLnBBFeoLnHPsX8SqJ4o1C+ZVBNCmpxhixpmrCa6PWzFT1gbsKJXuoyDPoQMiI8KOmcRKlelMqpViC7kazgaATyKwGzY4ruMAXac49hEQF1moQFSjazG8lxWnNsVM/jWvzoUMthNmQMyui/NDvQH9tE4HkSoPdkuWxBUJatyUqLmAw7xC5DGQEcpaaRWVwA+TUladkWKF6rMZ5/mDEJ1RWUQ9mKgEB13M/sC4JaevmifG/g/+0fCdiSEjYTZYbZmuAuMeTFNcfWxxRc1gocGOUEyofpga1tzhmcoWtK2Mb03ci+yYELAXrP89UN+yQTe6viuNE/xgRT1ZO51HeqDmQfgaAjawoAkMD0eXx9y0UFYfleG3sKk9Ru+0hB17w2KfE2X9+RmQ30IXLxBGiPJZqzGEK0iwsh0d7iyhi40vDfw1TECguOga55bP0RF5u6hYiSvUBJYMZ7J9tw/yl1tZ/yfpaR3SxuPbolE0n7GZNcVQvIPrDzMIajkNmNwUk9AZgzPK5sS+CclAE+MlcpfAeg1NDsA7INQQ9C7ALc8V7icYYxfIfbEhimkZVNaaOeoNsWBJtJ2dEd8zjSgkoC5/30IIeU9tCN2C75GVMDx6ebk/nTDcbEUFuIUeXgNZnHtKjD8C/AFjKGRzFMCDjso58StAcpmMiqgPZIjWmo8VdMXBAgC+oMP0PNyaFaxGVy/N5mzQSQ7bgPSo/JkHotUJ8Q/MiBAM/UXOzRLUGX2XMRid2EB2msVzxfW6mYThvvvx4orgbWdcRkUyAm18Y08eS51I2H2fuRBe2ncgTH3VKR3RVAc8b+Q5sL9FOeW7b3I+PXHjEF1iugPWWiqG5iHUhUCpTXDEvrjtRY1ZEkoUkmKXxSJhh+L07ivnaOoK6h9BIBGKY6/p7xYxhMtzQwOx99q5aA9XaISckR8Heen21LNi0IgWxtRxmntCmQJ4UCEYz2E7KJdqrGOgyXSdgJj8OWrEJxmRwDeHwuILsYUDRleI2qyWjp2kGEEDsWmRRrZyaG4MS003kEFVKgO8mZxFpQOuoyycAem7YFgtHBjUE5Pid75I2E3g8IRo4JsrLne3zOvkSCwIES00URX89tuGkG+ECPYmsBR3+i2tH7Z3YWPtMQPX83Qwi4KoOwAcOyhJ/kfG5WkMGtDHyukK066G3HsZzlPJ5piZQkKyKbGZSAaoZDDUhiDQ1rUhjUo+HnJRnfgeysagjcXJUBjuBeBIYGBQGoXQ9fGIimh3AOvjD0V7pAcBhBnC8YiCNaguSgKH/18xJjBUlkFkU7UMqaZxjrPrw/BPBkBiHTMYn1wGiOzGACmeBnawy1CeHgpXQ8R2S8X5BA6WARuo69kl2HMK34Z9syBTaxqFWJMJuMAFSUJYGukhBZRhmrtQSjd35EZiDy7CB7jYBSFAVEUQAmA8oCIs/4nshEiixO0JJMF5GRjGmvP/kQd57/6cTYn/75Cij+Gj/G8K2f488vyuNrLYW9+PZx+LX/8T9D5zT+o7PP8LQ5RqkGvG5neLWf4+9efIpX7QK///mH0Mbh4/M3eDy5wXm2xZ+sHyGTDsdZjb3L8Lad4qzY4V++ege/9/AL/P6TD/GrD17gotigcQZn2Q6f7c/whz9+F/fO1vil5RuUasAn63PUg8E863BW7nAv3+CfPvkWhkHhg/MrfHP+BuuhxHbI8cO39xAC8A/f+z7+t09/BcNViW9/5ykyafHR7C3+9PYhPn9zgt985xkmusfeZtDSYdVXqHSPT29OMckG3OwrvH9yje8uXuDz/Sle1XPc7Cto5ZBph++evAQA/D+ffYzvPnqBPhpOXm1n+O7ZK2jhIYWHDxJ7m2HVl5CCKbIn+R5/8PkHWC72MMphanq82U0xyXu8eHaCbN5B/8kU2e/eYPVijm996zmerxf47oKiNe8AACAASURBVPkr3HQVfvhn7+D3fvMT9F7hX/6rD6G3Ehe/8RrNYPDO/BY2KPzZ5w8BJ/CPfv2P8L/+4W+hOtvjaNJgcArbugD+bIbHf+cpvrw6Rr/JMT/b4Z2jFT75F+8i/+YGbZNhPqtxezWDLizsJsM3P3qJN7spVldTPHh4gzc3c5RVh18+e41//fYCIQgMg4KzCrZTODreo/1Xx8B3tiiyAasXc6B0ODreY/VqDllLnP3SFW7/6AzhwxrTCb/ktz84xse/8yVebWe4P9vi+0/vo5p22F9VEL3E+fvXuP7eGXQt8NF//Dlebec4Lms8Xy0wDArGOPSfzhEeN/DX9F3AAb/0K8/wbHUEAOh+sODk9Ntr7G4rSglVYCLxZQ45CPjHDdQXJabfvcbNiyPoeY+P7r9BPWR48vwU2aSHe16N3s5w0aL4fon22w3ycoDWDm1rUBQDlAjYf38JO2HVzsWvv8bzyyVCrYHcIXuZYZhTfh10wPn717h8eozZxRa7F3PklwriuxtIGVA/nyK7qIEfTmFLwM0cpvd2qL+Yw88chPGQmcO3H77Gn/3gMRbf15j+w9d49XYBvzVMYB4Efut3foQ//OOPgPkAcZ2xyqeMXtpeAsZDVRau0ZAbDXHewXxSon04AJmHUAHBEiBlLw36M4f8jYL7sIFtNYqnGWwRIN7dY9jmEK3EvQ+v8PZmjvAmh59bmDeGvtQj9szm1xLNOwP0rYY9schfaXSPBujCovyjCvuHnHybnUT7uIdcax537nHyhxrXv+GQXZEdQQxu8pVH+UJDOKC5cMDcIjgBmTnIlwXku3v0twVEL6BaieqVwP43G/hBMgHbCmTHLfpNjvKpQfN+B1Fr5JcK/RHDpfbvDZANfbV6J9B+o0f2mpP5fukxeaqw+yB2wvUCxaVCe84FonDRAtc5ggmQtURxHWt+juhfVq2AnXAbDKMCmofRFxx9vfk1F6bqCyYam1sJ6QTCt7cIn0whLX20248cRKw3UlFOLwIw/0xi/5Bqg+aCC3hec6GtfmyhNwrFlcD+EaXj3anH5KnE7gOHyZcK7UmgMmBNBYc39JX2C4ZJ9UdxcawVmD0J2D/gwlr9jkV+peiR3XJhLkhaDiC4f5dzYcZWQPKnugxo3hmgNhrzzzH2KLqc42N2B89pvwgo39JLPf+RxuabFsIKzD5XTP2OXruU4JqUBjqC/fWHtFZkKwmfAbPPCeJdTvtCf0TpPxdj6F/1hpaLxadM5+WCScDuHY7B8Q8ddg8UggaOPrW4/aZmQNkqsd7A7p0wBp5RJs5xyFdkd9cfAovPIoM+YR1PcQ10S+67eilgp1xcKG4Cdo+SNy8uXgQuyknH12y/we2xhobnNrn02L4jI1Dj+OiWC1XDFOPiVepNBfg8j1GMiy0ypgCrlmqL8k2ULEcPZ6rzUB0XQvqFQHnFsaxecdGhPRZjv+70Bdnp+pw2huKans+gDzJ03QU4I9AdcQyHSqC89tjfpyqgvPIxWddjf19h8eMB9T3Nap85Jf1JKpwW3phIHMbFgHzjMVQSwwRY/qjH7lGG8sqiOdHINw71Ge0ps6cM3Zq+tDHEiYsrxa1Dt1BQkbm1lSJDbsGFgygfd0agfDtg+06GfOtHm4RuAppThfLKkcHuCRJtIZBvKKU2O7LfxXWP7aMc5ZWF3lvU9/PopVVjCJhqPbojhcmrHv1CR6tOgG4suqVhqNGtg9kSVNtCwRWsjgHIduuaDLPZO+jawWUSunVQmx5QAu1ZycXpzmHzOMfseQ+XcZFBtY4MswdcKVG8ruFKQ4BsJOxEQdcOqrExPZcrOLJ3ZFUrQ/mvEPAFZaZBSah9R5DaW/hJTpAcFwaSR1JYD7FrGBbUkbGVuwbDvfm4SKJva9ijCpAC+mbPmpMqZ0ptrDwR+3ZkTUXbA1rBz+OHJ0mwFWXmcrU9VKI0LVBFD0/yaDr3VfluCPRP9kOUwnpASIS+Z3BQPzDx9Sc9lom1Hey/tUokgdy/CVLY3/zdn78U9v/9P38xpLBfa2B58q3T8A/+5/8EvVd4uVug7g12+wJ/78NP8Lw+wvef38ff+eBTrIcCn1yd43y2g5EOx3mNP339AN5LPD6+xfPVAkp5/MbFc/x/X74H9YMpvvP3foTPb09weznH8t4G07zHqikQgkDbZDCZhVIe+22Bxxc3OMobvNzNAQDLosGnL84xmzfoB42z+Q7PPzmHudeg32U4Pt9gXnTY9xma3sAohwfzDXZ9jsr02HQFpAjoncKbNwvoYoD3Em6d4fSdFa6eHeHjj1/gR8/uIa8GVEWH7a5EWfbYvJni7OEKSkbDugh4+fkp5HxAuM0gW4n/9O//Af6XP/odqNJiMavR9gbNLse98zVePz/GvYe3mGQ9vnh+xkCdLJrdvYCpekyrDtsfHuPj3/4SX94cwzmJYVCQ0mOoM8yP99iuS4ReIV+0+PD8Cn/+2UOo0kFph36TQ00GXBxv8GY1hXtd4eKX3uDV5RGwNXjw4Vu8+PIU5WmN5k0FORsgXxUQ79RwlyWyh3v0zycQ91q4RuP9b7zBy9sFpPSoryvAC0AGiEFCDKx3mTzYon4yhzjrMJs22GxLBt54agVD6SGMZ2KwlRCdRJgPWJ7scHs9hag14AB50kN/VmL525d4+6/P4WYOopPAYkBo1Rj042cOcqvIJp83sG9KQAF6K2Eveqg3GbzmxHv+zVsoGXCzmiDcZCge7NE9nUI4AfnOHuZ7U+S/e43mj09gfu0WIQjUT+bwmec5HncItxl0LaG3lNHiUQPxpMRw0QOdgtpLuKmDWXQY6gz6ykBYwC481E5i8lxg/S0L4QTMmgm0duZQvtRo3u2BQUL0EsF4QAWIViFb028bjntgZZiU+50G2acl7Mc17NZANrHjrnLIX2kIH3tJBf2uPvdQjYRbWKhbA3dkkS9ayD+doXmvh742sDMH1Uj4PKB4TVm1asnWhsqheG7Yr7rwkDVDnrxm6q0tA9zUI3+jRqbcbATaj1uYZzlUTen2MPeYPFMYZpwElm8FNt/pYd6acfLZPRhQ/ThDe+5hHu5hn00gB+4jlA7ZW00GPAe8Dph9wXRfBKD/do3wpmBysae/188tpj/M0Nzn5yu7kWi+MSB7q3lcBeXsPgsYlo6guxMIeWD9CzACHFsc5NU+i6m7W4Kz+WcSrmT4kXTA5JnA5gOPkAfotYR7p4V6XqB8zR5KlzM0yX57D/O9yViBohoxgrHp84DVN+MqadTb9ice2ZViim+Jsd6ETDuBYaor6ecEGpsPPIFkIzB5GbB/yGqY5NUMmuArVRFVLxh2pBuy4OVrymuHacD8C2DzHn3fCHx9e0xglXzXuiYrvX2Px+3zgNkXwP6BiPtgoFN37HH0CVOvhQOKt+zWNNtoI1gcPMl2Qt94fhtiwAqZVpdTZu4NrQFy4HhQ7h79qU1Mkr6hGiFb81xvfs3j6PuUUCJwvFQfpb+xWsXlwOQ5f38LDzT3yKbqGlh+6nDzsWLFThE98kuO2/ypI+s5Ecg2lI/nKwKdxCLaSiBbB+zeIftevqWaQNqA5p7A5BnZ5+aeoCx8IOufwnnMnse3/QYweUGg1C3oc8w3HrcfKZRXlFgHQZn+7h2C3OItmf8RTE/5GUoS7GzP/sggefzJfpCAlO7IHFEGfACWIkS2OjL3zZlEtiLbnm0JEPsp2eDpa4duJinrjsFMyWoye34IdWlOmAZsmhCZRcoI9xcEvOUVQ3nyrYPNJepziZPvd2hPDHYPJeZPHIZKoLhlsnD1hsCPYVqUVM+fDWhONFwmUL21UK3H/oEZ5fWJXZaO/t5s7+N7FeXVtcP+gim9zQmTiV0mUL2xsJWk1P5lj/pehtQ3qnp6bptThXzjx8qebNXDTg36GaXUCBi9tdmGflVbMgCpPlNYfNlhmGgIF72uMUxomDAZd/cgw/RFj26pUV71aI8zqN6Psm2A46C3PeqHFRnpCOCED+jnBqpxyG5bArZcw2cM8NG1G2tTyDLGOVEzIGQawyyD6hzlvoK9n/lVC+EDvFFj+I6P3k5fKOhtD58pqE0HP40Mp1FQu34EfAAQjCRobAbKhI9KAumbGm6Wj0m2ctdHqTGoYmsHiMHCT0r6Mz3Body1oydzZDWNhrhZI8ynEJsdWcp+ILNYsitTxNCfoCRlrINlAFBmDqzpavMVP2l6XizmQN0w0CclyiJ6KAF2XPYDWc2mGa9XaFqI5PGMwBIhQGQGIUl07z7Sfu+e2896RFltCIFy3n/Hxy8EsJz/FQHL/+tvgeW/96P88EE4/8f/LWZnOxyVLY6LGpf1FPsug5Ye212Jf/DRD/D7L96HcxLzirOa220FaxWqqsPZdI9VU6BucwLAyyWyYsBvv/MEf/DJR6jmLfaXE7z74SWcl3g0XeEHV/fQW4UiG5Bph2nW48nbJZxVuH+6Rt0bZNrhZj3BsMswOanhvYBSHs5JaO2wW5e4f2+FXZtjvy8gRIDJLOZVi9tthX6foZh1KLIB+zqH7TQmiwa7VQmx1TD3GthBoZp02K9LzI5qdL2GEEB7XULNexIjXpC5AbBY7rG6mkKVFm5vCKR6CTOLPUtBoKo6bDclvnH/Gk9enpAVkUA+7TD0lPGEdSoklggXLcImgzzqETygnhWwjzpgZYDFAP0y56S4signHU6mNZ49PYWoFcxFTUXHLhvZJ6E95JscbuIgphahURBWQq8l8F4NOygUVY/25QQ4GlBUPZrrEsg8EAT0W4PwqEVe9KhvS4hOQcQEUr8YACtZ35KzasUXAcXFHu0uh3xLZswvBmQvM/SnlkDspoAI9ODKTsJXDrK0ECrANVzR09cGqsUYfKQagWHpIRY9/M5AFA7i1sBXnI1mVwr9mQUUA6eEB9zCQnTsCG3P6PWFAMxRiyExUh90TNWVgQDYceIJD4SZhZAB6nUOV3BCPcwCgg5QjYTZCjTv9pDbeB1PegQryaxpTqhThU4KfQqa4C1/QTYxSayzWwI1aYHyjcD2PYcw4eu6+xb6VkepMtCdMYl2WDqmBd+3vHeyQMA3CSMQzK+ZICusgJ3xfZTTCgzTMIY09SdMBPaGk3vIgPJSojlnh2p+I9DFnshsJdAvU/IrWZTiiixS88sNsh+X8IqsVlCA3hJwJKCVEibtlMdZvpGUeprofTYBxRUZI68YiuVy9nBmt2qUIrengYFdnRjvFTkwjGvyQo4T5KC4T11H8PRGjJ2N/ZzHqWoBbwKko1SXnZYEd8UrjqmwPObiLT//7M0jgFEd0NwjKwkBTJ8QsA1zpskypAZfSahMfYr8siBIARg2la0ZeJXfCPSzgGxDVlG33IZjMPWYMDvMAqbPBNqTmDY65RiZPVm52TOPzXsSes/jhQBkdwiwCjoyU7MoT5aI9Tr0vSZQa3YETqlGpj0LmDwXZNg2UQLegVI3E3tMV4cQJVvFeyL60lP/XlBRahqZSG9izRAO148dyDiksmaH8CeXsXKlPSGjpOs0xkyrFQ5RHhqlxRHopT/TAkIKxPEx2Cn5kBOwo4yWoGP/QCBfAXofRjauOyIAAjCmvS6+tFh9aOhPLaPEO6N/0ytKmkWg7NUbSmKli9VAa0qn6Y2OYULu4H9XPVlY3YUoPSWT2i1ir+0Qg89i1U46tlQdJAJBURBAsfaoT2X06QsUN0ykznYEmClROCXcFiuHoaSkPN2XpglojwSyGE41VARMXkXWNGeAjM0FiughdiYGBDU+JglTLivuhAm5jD2bLouBZdHHnHyzBG6sQUm9mtnGwU7kKDMXMSRKRYmxqaPMtpRj4qy0ESjH69MeKZiG94fZOrQnGuWVxTBVIzhUvUc/j/VhPqB81aA9L6NkNnpCYxiSKyT0nsE+rhAwm9htCabbulxA7x36I41s48ZQK6ZfC6jWEQg2BKe2UshXw+g3ZYhTSv8V0C2tSMNUUwbtA9TgIQZPqS8ie1xTMhs0Pbu6tnClht4PUVacMVCptjG8j8BSDp4S4s5BdI41Lc5D9J7dlZmGcJ6BQ/HvAFnHxDKmLtEgRezbdJANAZPPNdS6QSgMgpZjpYlwDB9K9SZBS8p8AfZsao4HvGcgT8YOTVHfAVBJXht7L0OesSMzhvyItkcoODdLfxc+EGw6R7BZZBCbPZnI1IuZwoDuPpKk1zqCxDvBQKGPktqiQKgbbvsuQ2nJPIqigN/uxu0RiPY/3UP571I78jWvG5nPH4Xf/J3/6ue+3X/2f//jv/ZzA77mwDJ//2H46J/8F2O1SGEsrq5mmC4atC1nnPamQFAB3/zoJX70xQUmxw3uzbd4dnUEBIEQBNwgkX9RQP7KGt3nc/jCs3JDgBP3iYXOLWynIZRHWGUIuQdkgMw5+a++V2L3EWssIAMByS5D+SQbGR+IAFkr4LSDHxQmRw3qTQH5NosBDAE46+B3Bmo2QH9Wsksy9zArBTvxnIxfauS/eovue0foHvALTbSKdSJOwKzYswkBBBOgVwpu4iEGAb8cgE4BxpOFcoIslPHIn+boTxxC5Qg2ssB6lI6TfASgeKNhC/YxesPk1PH8tEfxLEN7n35VdzxAbAzya4nu3EEddxAiQH0ygVcB/aMe1Y9yNA8cggnIrsgYlZcyesHCKBnMbyVDfAxTeeUADEuP6rlCc+Ep1Yt+Ua+B4XyA3GoUbyQDjjQ7NIeph9lJdEvPyfCpgxgk/aq1iAyJRxbrRvoPyMD1S14HvVWsMWklfO4JsMoA3YgxvTe/EWMqrdkB2w8dRCdgtvTHJXlad+xHz29KqOU/6Mf0huFFZifRH3lUzyUB59mA/LmBagSa+w66Tr4MBkEJy0qQ/JqJxv2ZQ34Z/TsqxEk5/WnNhYfeCggfg0I6HneSbrkijCBODmSTulN6ZlNK8uxzTuwo9Qsjm6g6MTJWZs+JXNBMKy6uCPyki4FaINhzORAME3SHCbiocUu2JSgyO90Sh1RSwzH2GUONEkMYFA51DQajL7O5RyYvvcbrg5SsuKYUTjjK5VzG62j2BC3BAMXVIdU3bTs9+qOA6oWAK2MK8JrjCQDD/JB2PEwPnlXZA+05Q3CKKwIe3QBj8nHJbs9uSeZomGJM/Rx9xZGRSQm6wnOMzJYBX9mKzI4teH+ZPfsu0376BUF08ryWcX+ciEbQ15FNa084Kdd1iICRLFmSLpod4meNE7B+EdnUAWOyNsKBRbMx+CgBGJentNboBfVprAhuhOUxI07OvebrmDYcRmBOsELfKIO9eH97xW3JAaN/0TRhrKpQsYIhVXgMk69KD1WPWDaPEXjKHlEOx3Ppp2K8funeSwm/agjoJ5RNljeeQUYRFPQzgekLPwYJecPQKxlVXwmMJXba7MOYmOo1vbmsoaGftD1S8XNAH+7+PrsUq7d+rNpI2zM1ZYVBcp/CxfspAr8UsqQik2Rqyub6qRylnXdTcYMksziU9FKO6cjxKy75XJtTieLG8zsnnostxFiFkXyQzsR9xJCdFEomXbwPYzfpUMkxwTel8WYbsoGTVz26pRkXDYI4nHNQAnlM402JpgC3380p/8y2ByYtySJ99OIKR59ntzTxe4WAMHWbJvAlBzKbwgYMcwXZk1k1MeRItUyzDXdqahhYRoliCo1ymYRuyNp1RwbZxiIIUOa5c18ZOzl42FJBtX7sPG1PDMyWDGN6uEzC7C09w3f2JXsPW/H9wvM40ucpHVvy0arBR0WGhKrt6EnlOAu4QiK/agHJBFvVOdgJuzHHBF4b4ApWjogApFoLXyjo3UCpaGvhItCEEtCrFiHTcLmC6tz4f6q1EINj+E9K001+TuvhC41hniG7aQ81IDFlNhgFO81G76bo7SFlFkCQkqAwVo2IcJCRCufovZSAaHqESUGQ6D2ZQ+cJ6qwjg1jmo5cy3PERin6g9DQxgN4f3jsm1ip6MNPzwOHvsc8SAEKZk8EcLH4yyTWd91ceSo0M411v41fYwrvviYE9qWaE/x0ohVVq9FwG928AhbGm5Gc+D/D/YnjPT8pp/zKPvwWWf/UP/dd9AP8+DyGAR4s1Xm9n2Gwr1C8pId1tDDAfgLVB+Vqh/XaDpzdLoJdov5jhCz1DyMhsBBUgrGBi6adzCADTL1g5wh46CbeVMJscw7sD5JqgMn/FDzoTUGMf20qPvh/1/SlEFqAaoHiSsftvzuRW+aJgcNelgU7GWytg1gJuW7DmQtFDIByjQ4WnnEGtGOoj/2iJ8hoQPqPXYIh1IA0n/6pR6M488isVf7nRD+W3OfJbBsyoNsb4SzIc+QqQvULz2KN6IWErThS7k4DqqWYNRgu09xxUr+A1UN0C7QcBxRcM8xlm7GLsjwLMZYZh4bD4HFhDofhBheYcQCAoPP4XGZpTIL9WKC8ZQjN9ItBcBJSXIvZVAcPsEHpTvpQYZkB+Daheobhid1dxBXQn3PbkRUC7pazH7ICUUaVrBu/w7xK79y3KZxrZlhPyoLl4Vr6WqB86TL9U6K9yuJJdiUHTrzb9jJMH1chY9RDlePNAD9spmajyDScWx38iv7qan3Pcq5cypssC2UvBwwzA7jca4MsC8y+A9oSyItnxuKtXAsMu43kXwOwLNfYSZmuMfy8vJfJVwOYDQLYS+S0B7uzHKnaHcTKQrcRYC5G6AM0uYPdIoLjmfeUuNfaPyNYVO4HJU0qHOMEX2D0maJs99di8LzF9FrB971B7MHvKXwwup99q8lLEyaREfT9g/mN6jHTD1FW1YjIpQhrfyBjFYJgkD1RdGJkoOQDbdymntKVgeFJHv1Z7LEZfW35DANOeUAJ79EnA9rFEto3SOxHGaozuiH9m64Bszfswv2WQRojM2+RFiHUzBAu6DTA1r0O2jayOBqwVmD7l5CnVPpSXsZeyY2KubkLskaWErp/Se8jQGckaFMsew8Qc0n8XK1BGtiSG2ewD8jUBl8uA8pr+ruKWk6HUzWj+f/bepFe2LL0OW7s7XTQ3bvf6zJdZWR2rimSxiqRkCzQhaGLD1swe2Ib/gDSR4IFh2SMP/BMM2IAn/hEekbLNIk2ZlClWshpWZf/6d7voT7sbD9beJ+5LlUQapMBKogJ4eLeJOHHinBNx9/pWtwPyFQdRKZ00CAlbCUxeOeweKh53yefKVwGm8azSMEB5FRNdo8y1uEw9deEN5k43GGtYwhYolp5+MMVaDN0QDOqGTM70hWOdiCH7w2qKyPo43re89hgmTKKdRrllvuZQcPrKwRkRwTdrXqo1gZ3qGaLUTwTytcMwUWOSqnAB1WWArj3MXtBbeKtHMx13l9FX2M8Epi8dgVyUXk5fWgwTslH7+wS4JnYoFqvE1hFcjqyUUqwpscAwZRVItqXPrHpt0ZxrmD3PXaoEKZYuyhEFu2EFgaSwPPbVpcMwkUwAXvkYksbPpX4qka+Zws0hGKtY3JlEcc1gKGlFDOJhrQgEE15lH/sVA1De8GuzdajvGogQUF44vobo+8uXA+o7GbK9h+o92mPNRNWphC0kyhsLETg4mCwH2FLFv3l8vskr+uqyLeWNtpT01F3aGGKUgplip+EQYqASw25Ky8oRs3fIV3wPe0MgZSsGKnktUF302D3IkW1jsm2PMfRJdXyP6L2D7BykDdi8k2PyykK1jkzx1hIUqTCG+kjnEQQrUIQP0Fv2NrJ+yEP2Hrq2qO8XKC4aaC3RL3I2UHQeavBjt2LQAmLw0IFAS7b8W5xAnNlT+ioHh2FqoGuCiDDR0LsedppBOg+zpRdRuIB+kaG4ajHMMvoJbYDaWQQtoDY9mbpKQfaOfkEf4EozSjjl4KAASj7nOfSugxwoHzWbGGQD3s9sBWA93Dyjj1AzaMhnCrIexuCg7KpmeFA9MNAnMYmDQ5gYgtIYLOQLjSAl5KYB5iVTZmNSrXCsLZEdgUgwCupmD2iF4aSCXrUwINMYJgLmqub9tITc1DDdMHoV5bYh8zlIgqQiZ6VJ3zKYph/Ibt85gto6BvyE6JMcHMGf9+ySvMU0AiDYkxFoJtYyMXvWjuFCYh8f1/VkNWuC9OADxLRC2NVA8BCT6tCJmUJ16vYQptP37JgcyPBShpoG2/FDzmhAkokMnT38Pv2saSmRTfvoA8Gnt/xT5BxgTHwePl7cTolNz/WzwOTt2+3fSwH4L3jmaLr95YnaL9ztC81YFu89DPf/h3+MrOCHRpENrB4xDl1rEILA8dEe2zpHt81hJj28U4fgC+1HeffiZIfl6zlm5zsY5bCrCzjLlYZrGK8njEfwArqw8F5ACMBtDZB5FLMO7ToHVKBvb5CUP2qP4ASwMwiFg8wcQ0ombOEOg6QMtFVQhYPrFJnNpQHOOwgZn9/HyWgvIU8Yz6m0w3BVIogAORsQvEBwEjJz/HqQgBOjjxAA5MTC7wz0WsGeWMgdZaEpmEX0TAr0lYNoFAG4EwiZ53Y8KAvNHUStIM86yCclbBmngXnyMUgIy+fN392ifj3h1H6n4I7JJgYTgNkAeZVRgllLhp2YQI9kLxFKB9EqehNnHsF4ej63mvt10gNbM8pUg2I1id4qShXnFmKvOX2fsqMKg4Bs+TqDAHxF6SU8oPdyrDEhyxAlh1YcmIpAD51wAm7qIRuCAdVy0m3nDmpHBtnPHOReQbaC9TS5h6pvfTBGZkLXXBSG+JwuZ1WKuJVk7Eoy1map4MoQ04Al7NRD13IsKEdMcrUVvV+pVzMFWKg+prZOHcxaxX249RojYyICGcvsJi4MLOtMdPQlkh1mnY20Ai760yDB8xGPaZCAL8g083WGA9MXIuN2i/mzE4/sRsEVhx5WykuZKGy2aWIc4HMeOxcZ3/4ojAAnGDAhNqYt2ynZu1QVk63ImmYrPkfqkrWTEFNu+TrJbsVeNkkmNmhuOwEO9tmygiVIykltxeNHv2AM6ZgFVhtpIIX4ZJsI7KMvUjgOG1JXbL84/Dw9b1p0RzbZnwAAIABJREFU+1t9t6mLFYjsoMTouUvSyQTEEXgMZB+lihnBe9qOauNrCxhrdFxM8A2K15ZqxcgchziiTHLj1Keqa+5TH6tmUqesjMMwW2EEdCll2BWH349y08hwpq5XaTECaekO76PRZ2r4s9T9acsooY2MZXcS9z0er3Qe02uExFilonqyLYmZVR1GFlZ4nkdvEP1+scReYJQF8vEEfy4mFWfbBJAxgkFp6W306RxEYkAOBx8kOyfJ6qRew7E/NL6vU9jMyC6pyCJvqABJA5Akr07MfwK42SaMKcWJ1U3Pka4//jHCG52lQwTswpEF7qd8r6WuZgCx6urQscr+TjGyxl4fzmdKrE2pySpeDzr2zaYeUd0e/I+qZ++p6sN4rQ0Tsr6JJU3y1PEYyEM6si0kTEOmVdpwODeOstjUQ9kt5OjL9Dp1Nh48nIfP2oB+xuoVEfc3saHjPt763hsmDwOspkl9n/1Mxlos1uQEIcYqGNVT+ptYT9xep0fG1OUynl/6ItPxTccxneck9U6srfABtjpIWbmffgzFcZmMfycok3WFJAMfFQbpvSBcGBN4AYwMpysUsvUAlx+Od5JuByluvccJzik5jim+JiULhzE9Vjgf2VoxMpCUmVIG67Xk/W/dRhbSh5HNhIyBOTF0RwxkN0XqzHQBqe8yyMM2gxKQsQ5FJAZPKTKYhomwoo+pxNHXGDTZ0BFkxcCeBKiCUgcgmvorBzuC2dQV+gZQlfR4MpDH4/P+SdgIJIXgtt6oBUnAMlaLpAqTyBKGeF8hJYJ1nManTssESkf28nPsZJTYvvmjJG/9S2KRvy2M5W/8O2As//kvGMu/8i14AT8otIOCaBRaSWatfuQQKgux1Vi+KOBmHDUP6xzZlYY7dTA3CqqLnjETsBzm0Ncau24OtZNwZUBxIemZy5kkiMwBVsItc+itBDxgeoF+4ZG9n6F/x6O4kGjuUmaJoFG9DFh/NaB6KVHfF1CdgcgCvDMImUf+WqO7B1SfGPYoPuxhXmWwM4/8g5LBDxmQrUVMWRRwdQm7sJi9X6K5Q+Yqe6bRnnmYWkBaBsP0Zw7zn2i0J5Q/6p2AK/X4h0dfawgAeqfQ37HIX2voWqB+yyF/aeCKALXSDBHZSLiKx6R54DD7cYb6XkD1/1ZozgPya0k/n6eXz+wE9g/5AVJfV8iuebzb+w76Om57L6GuC+QrYPtewPxDieYe+zbLC4l+EWABlM/JBMheQnZRIuLJEObvF6gfeuRLTtNtGWD2AnZCb5n5NBsrRsqnhqmemyjp0GR4By1QXKpRrtfPAJ8z5GX/9Q7TH5Me6o65oNEbfiDbih7BfAns3vbsHgUgrYJqCKKGKdm9/tijekH2uTlPHXhcdPULD7OJgD6+I4trgf2jWIGyBvpjQLUKw8zDTikBZqhJXITs+Uc4X3IR7nIw1VMF9Ccek88U9t9qYZ7kTLfUAC41nAHyNXsepeXxKy/pfROeSZr1l3tUH2WsAlixo7L3AvkySvUsGcGgCNTK5wK7dwJEy9dhC8A3gnUjA+9nduyNbN4ecPQDg/o+wVh/FJBfKVQv+XuzJ+gB+HrmL8nula9D7KQMmH526DT1WZTUxoV/Cj7RNZ832/Bn+4c819ULyl1VF9nOCnB7Mp7dCUFdfs1F5O4RMH1KmWk/peTZbDkR379NPxylztzfbANgzWPdHYfYaZkkmTFp9IZhKM0Dh6MfKTR3ua9k4mNdi41yac3zU71igqXXInYHUq43zA9sZntCINvc8SiuJfSez+MzYP6Bx/6BhNwR+PcLdsRWF7GmxBPk9XNg9im7LKvXBBvdMVm6+acB/TTWrDQEVP0CyJascEmAtj0TQAreeR0BVibgc6C89Lj4TWDxI4HuhB657kigek3voa5ZI7N9HOt0OrKvCQwFHdnajNLV4jo+b4uxJ7c74sLc7KMX0BJ8lBcEZMMEKG8CdvclWfycEtj2TKB65dHGFEwgjF5O1XHxa3ZhBJVyoFQ0yUbZaRnQnEl0xyL6RwMAghpEpjnbBWzfkiivAraPJY4+IiO3fSSRbQ4Aqrjh/po9j5GIi/5sEzB71qE5z2JFiohJmWQfE0OeanrkAJQ3ZNvW72hMXjns73O4VF4z1bifiLF+qD/iccg3iZFlT+z8iQUCUN9hhU+xcmidGtNTTe3hjIpVKh5qS8CXb2Kyc5AwDa9b4QFTewynCrolkyx8uAV2+H5SQzj0dGpWoUxe2RFo0GcrUd54qIZSVQCQQ/Qwbh29kz6gP9JRCkvQoxtPqXXwaE7lKBdujyREwc/GYulgS4niagCEQbZxaM5YGaM6dl0OlYatkirGA5JsfLaxsIWC8GJkjF0hkd8M6Bc6+iENJs9bDFON9lSPQ4tsPcAVGZnExkHvLVxFnzxVQgpm52Fqi/YkQ74cIAeP/sjA7MnquSyD2fP1Z5sBF9+pcPZnLVRrsXurxORFh35hRhCJEOBKheIq+uo0vZR6P0TfdcHqkdrGag9KUIPmsRbWx9Ajh9WXSyw+qOOiDZDWw04MVD0g6AJ62cDdm0B29HJKALJjF6eOFSesNNHQ2w7DcQHVxH7Mjumr2bJGEAJuno9SVb3mZCYohZAryOUOomIVCrfH3k27IOsmeg+16egfLg3kvoOfFhCDh+yY0irbgeBwIFAU1nMV7QJlq1GyKpzj74QAfLxvZFABRGAoGaYzRECoFLsuvY/+yQHoenocraM/sm5ZbdIPCBXDeoQQBHJ5RsBoLZlCTwZ0BKnp63SzHtCK0tYQRiZyDO8xJgLHgf5Ia3E75Cf4AHgHkWWUzIow+i0TWBRKjZ5MSPkG8OUfNc+eTCB6M/+CKhFP9c5fBVj+vNw+P+D423T7wjOWd/+bfwIAkNOBQTZPZ/y+Y9JkcdxieDIZF+xQAaKPbwzDUBPhoidvo+FLD2gPMxkwrHOInqmiak8Gzs5dlK1K9PcHiL1GyDzUTtFnNHP849EyCEV07JDkcwmyS1sJ1VN+q2oJN/HQO4nhbEDxJMNw5KFqgf6eRfmZieEmcZIL0Hf4qEPYa5i1gt4zAKS4ooTTxOh52SP2y0XWSwDdqYPZyJGpCHGxqjpAWIaWyJ5eQVtyKlc9p7dP1+Iw4Y9T7PacqZP1w8hgRaZAdYcpaPIhuSIgGMBlBH12GsaJpCsD9I6Mo97HgvuMv5c9WcPiWqB+EMNFosSuueOh95SByYGAsD+iB5Cx4AGyix2Bxx5ZStMEF6BeY/QDAnE/I1uU2ARvuO+qEZFlCVA9OwylZShLkpQGjdHrFDTPVTBkW1LQSPIvecMgixSL7zMeL7M7BIKkBbKwGNkOs2OSpOwjKzSySNw+ZZhMwaxeUF7qCow9lInp6k7pwwyKxzqxm4lpGmZASkSdf8okSRun/90xf54v6Q/sTsk2VK/SopyMhW5jmIrka9J1rFqIxyBbHX4PxK7QyAzpPYNMVAQKielSPaP+zSYOGKZ8LYufCDRnYgz4SYty3fD/5DcjwOQCfZjF450d2D5vCBKnT3j9DVNeb9ma+8q+Si4QXRZ7C9cBwzSeE02QwWqDyAbWkdmI57NYerTHcvydsAc2z+wD2lMZhykBw5yJo7rhNhJYSp8HiZFJniqA18H4visPi1nhKemdPmdVQ7YOI9M1TMiKpeNv9gH1HYnqgtUHqscYxJLOsfAhMlWU8rpMjP7cIMmSqT4CzxnB7hjOkiMmW0YwE0NO+pnA/KnF7h6HPSYmctKHSPYneWgTS6abEJMqA/Z3OQhI13pi1dLnloq+X/YzBrTHrHpwhvuourhGiv2axcof2EUApvGoTwkoCN54zc6eOfrxIrsFcDvZ1sNrSk8Tu+Yi+2Vz+ha9Fsh2lAYDPI/SMigmMZ4p+MfUYfT6tYs0aKP0O12TtpDj49LAKnmOh4pAqbyyZJg0z4PP2Ldp6gAVgRb9ptynfOPiZ2TAMKVfUw58nFd8TWbPcBhbSug6MluNx/4+6yxsKePnE++jeg/hwPM6UH7cLaIvb4hsVMDI3A0TSYBobjF59ta10XMbtuLXwoex7iX5BoMUhwqKMtZRBMDHHtgk+83WNno4RfQCMt20n2sy2UNAcdGhPc9vMbthZOcYbsSqivQ/P19kTD5lKE2qxsiXHdrzgomnYw8jMMS+S723GKaaPZWNg9n06E4L6IZ+Q9n76M0UkHH7rmBwjFm16M5K6MaNQTY+4zBW7wb0R1n83Ijn2PoxnCYxeMkvCR/GfswEQnyu+H0KpEme0IBDUI+Rb8hpfa4pW+7seD9VD2NfpRwiwC1ih2lrx5Ab0dH36DM1sr7BKEpeXaDkMgQydzplEBzYWG8U9PWOdRx1R9mn93BHJdS2i8eRYJBPLkZmUwyO0ty4D2OnZJ6RpUwMYWLnFD2QochGiW3IDG73PJLJ5BuUx9zRdxnBoOiHA7uYgndiJ2ZSJyR5LAAG9mh1+DoEQGuC2fR9fN43klhvsYnBe8C5N4Af3zTxPlqzA/PWLQzxPglE3n4e5/Az8cZtpvIvGeTzhWcsZ4/Cb/z6P/5r3+4//z//2d/4awO+4MCy/PKD8Jv/83+BWdbhz1/ewdfvX+DPX9zFb737Ef58dQdKBFzvKrx1vMInl6d49/wa1/UEp9UeH746R1n20NKj6QyqoseXT67wxx+8A5072L3BvUc3eH15hMVij3nR4enFMaRkeisAdI1BUfVwTqK/rKCOO3jHQKDFYo/dvoDtNJNlLyasz9jmkNpDaY+8GFDvckxnLTYXU8ALqHkPpTzmkxbtoLF/MQNmA5NdnUB+1MJaBa0d7KDheonHD6/x5Kd3oU9bDLsMIncIVmJ6XEOKgM3VBHACes0p7fxrN1jdTCG0h1J+DDAKvYJoJMLMsmbj1ZxyVBkAHUGgCpie1uh/cIThccc6Eh0grw3kgwa2ix9CrYKcDJjOWuy2BfzeAMajmHewn0xhjy0Wd7ZYvZ5BZB7lrEX9esJwnKNY4SEoN/UzC9Ey3dXnHqJywEZDnnXw1znUKY+7t/JQrTGhhFcMTHPF3AJrgxAlyKqycFsDYQWyG4nurR6i1lAR3AaSi1B3Gwy1gdjpUearNwp2YQErIKxgXcbdHqFn4qxwwPk3L3H5o3O4iYea93CbDGqr4OYWajZAioChNpAbDXm3hZAe9rpk+JMKmHyq0H13D/eqxPwDidWvpA/sgNmdHeoPFvAlJbziTgs8LyEd00LtJMA/aA/pvZ2A7AWrR1pFObUFsiU7/1QjMBxxYBJMgF6rUeoqPNDfsRBdrG7JA4LxkLUaU2QxHaCf58g2As03G5hPCnRnDigd1BWTdlOHIgAOTtoY2nOng/m4HIcpw90BolWYfKaw+5KFWakIKuk/TD7UfEm2cji2KJ8Z2DJgOHEoXmkESXCk4t88VwQOgk48zIZDiOadHnKjoXcchgQTFQpRvgoA9k4PdZXB7Mi27h87FK9UDHLi/mQrCZcHDKcWasOaADtl2BGBnoAtAuyJhVprgo7YFzrc7TH9YY7dNzugUTBrxdffKOQXahwMeR2DcI45gEIAZbdVGNlir8k8FlcMNmGibwTTHZAlaW4ZUFwK1PeZRNsdUyEACUw/46LfTgCzAfZvecw/kBzmbDmYas+YJGu2lAkDkRGyDArSu4OflYCLwyQEYPoE7O1rOXzJNkB7yv/7oxToxOHE9jHThlPqKn2sTMj1BmMQT7YFmnNeH+WlwO5tj6MPyNCZbUB9nyz1UCGC36hIMGT3KeckO55kvgCHS8UF0J4RTKsBZHePAoobMcp3U5BQ8gKLgNHHKC2weysqBzSPadq+2ZEdLq7p9y2uycDLgcFRzR2BbMUUWNVTeps6Mk3N71XHoKX2OIJPdTsoiQMJ1RIIpjCYIMgou4yhRNkmDp4yngfdUKLrSg48hinvk20DujkXscOMjC/AIUWQZFSHVIenBVTD6hX69MkYp+GQy6Ict0n1LBxuDFPuY3Ed3pDkmyZg90COqa+6CxhKXgf5+sBgAwfQnO04DEiy1xR0JS2Bt8vjYGFHqa3LmLrazTloq2KPZD8jUAN4PGzBKpIEwLtZTGgdAnTL4YTL6dd0GcZ6HbMn6HOGoUbOCHRziXzjY8Ist51tPYZJDDwaAppjheqSQLqfSeRrB9V6DDPFMKX4GZfA7DDhZ7vqKI8tLwf4XKKb07/rNQcaQ0kvtbAB3ULR7zpTYxKuabiP0pJ9DoJeZ9V6mO2A7jSj93nLlFjVeei9wzDTo/S1n0pke88BQy5hdnaUzvqYAGtLhWzVo7lbwOwdJbFxaJKGjWZLpmyYmdHnqlqH9iwj0xxZWADwmRyltEkuno6P2vWwi5xsuA0ISY5r6EEWPkA2lpLVONAQsTfRl5rMZe8OPZM5E17tvIDqbkk+rUfIyWamwB97VEJaTxY0Nwz9MYrVJzHwJ9WNwOgo+3UEkAlIpmRWKQkQfTiwm203MpkI4VAt0nQI8wlGXyYwAt0EiP81mWz62mggBIS6hcizw+9v+zQ1AXlou4MX0jnQwxXZyz7uc5LF3mZNP/+8n/dgvnG3g2T2i95j+Qtg+XN8O/76eTj6Z/81posa59M9lnWJedHhs8/OMT3bw1oFZyWGVY7Z/S2anyxgzwaowiEE4Pxki8ubGbRx6Fc51GyAtxJZMWA+aXH1yQlC4SB0QDlrsZg0ePHZKRAE9LxHVXXoB412m0MXFmfHW7y+PBorMjAbILWHHySmRw36XsMOGlJ6OCvhGw0YD7k2CMc9snJAt4xm6CCgVwricY2h1VA535T+KqcXsnIIVkAWjimy836svoDkOdWvCSzs2QBh+IEsRIDfmdELqjdMjA2F4+OCgLrRZGfPB5gJ38DDNme3o2ZRejjroZ/nY5egngxw1zlC4QEnIKcD/M6geKXR3rH0gG4yiE4AdzqE6xwhj95G4AAapxbiKqM3sbQcdLUK6OVh//YSbmFhLg2G84Fy3QgkgwpA7iD2GnACYWaBXo7+Tb2Vo3cIANxb/MsvXhZwE4fsRqE/cwSPZwOyl4aL7SMHWUsChWPWaXgd4BcW6CTMikBrmHn4wqN6qtHcZ/dgSnvtT7jdfCkwTCjbDDogv6Lk2pWRien4OrK1RPPAAoWDeZVxUTZhnUTzbg99ZeAqymjthOXzet7DrjPkl4rgqiOY8BqYPJdoT+jdDCqMVRMA2WrhCFia++yNtAuC8PlPNJo7YfTiyZ6L5f6EibK6ZkVIkAwNas88spU8MNYTPl8CY2bD1GKzpTS0Pw6QHTBESTAQk4AvJfpjhouIgdJn4YH8WqK9Q7Y/2yRfJWWzbQyw6k7IxqbkYlclxQBDiPIbhtfsH5HxDhKwU3Zqmg37H6dPJFxBMCW8QLYUYx1H8ismr6HquVAd5gfvZhU7FoOktLW5I97wGaqWgVvzjyk5bE8J+nxOBsybxDLy3JSXZKyGKc+pasUoAQ4SyFciBmwJ5NeRcYuMOz2AfN7uhEMD2WMM8HEZAXVKNxUWqF4L1Pdjkm3Fc59qLOCjlDsmCEtH9rq+K8ZFPP1wBLbCU4o7egpjJ6PZI6oScCssDCguIys/HCTNTI7kAr8/ogfMZxEggsy12bFjMu2n2REoIwY8DXO+5nzJDkUAIwBIoVauONS75DcB/eKQkEvmRRxqSnyUw+q0gOXxTCBw+oxMNqXFIoZsMagshYaZHcFdfkOQGRTB5W1WPAHVlHqb/IL7e3IEkrcBnW6i1HeK0WdZLAkou2PKeHVDgJrtyLymEKB87WFzMTKQ/ZyfE8ISeJo6jM97uxcyeX1TmqxqEUvuJetArg5Ms3SUVpfXMTTqNIYuBbLQwkU2uuPQQzeUkqf3HZ/7wPAGCSTPp/B8PUzOZSWGVwL5hveXLkSpMgFOqgBxhkE9rIM5MMM6Mpr9XCFbOzRnmoykobRX9gH1uR5TaxPQdEYg27kIMg8hVO2xguoDihsGHKk2oD1WfJ6USBs/l/OlZTWI5+u1BQOhTONHZtZlZECDFMjWFv2Rho7VJMOU0taRzYzJvapz6OdkDLMNA4xkz+fXjR39jcnHO0w0VE8Q55OPGHxfZqsedqIxzPTYQemVgKkpDQ5aIFsN6E4MhxxaHECmYwJsftMhGBnPiYwJzX70UsIHJtjGp7UVAWnyVfqcMmwAY9ARwCGH7D1czgGl3jNEJkTwCwAq1X+ArCcZbvZfUrmgoBoyqYldu50wS0+lJluaQFNkI11pIHvLTkwhACUAFwhKty1CZgjyAIK4MiO4lOLAhib201G6KpqOtSKDpX8ySXA/Dwyjz3JMjE3ANEqBx1sCc+5NqWzwMd1V61seSlaQAIBQ8iCPlYr3Cf4NBnL0WN5ib39mlyWfkID033aLjOYXn7F8GH7ju/8OgOX/9d/9jb824AsOLPN3HoXv/i//JbT0ePLqBEIGvH33Bk8vThiC4zgJUrPh8L0AJa+LgWDFCehFD7s3KI8bdM+mkC3Bw8mDNZafHJM1m1tgkCP7xB1w3EYQBDW5g9xpFtdXDmJloPecXLojMlx6wz8S9shDxoWhHPhBxxLvFPEdZZeTmFLZxiL2jOBI13xsf8SFd5AB7shB3+jx/u0DC1lL+IILZsod5SFgI8qk2AMI1p0AlAPLgOK5gd6TiZCDQPduC3GdwU8cQ34cGZvwjS3k+zO0dxyDVaL0yt3rEbwgK9eLMQSkvBDYP3bIr8jK1I8HlM8MvapLgiSz5SIiBaZkG4HuOyxsH44C9Fagvecw/0CNC+e0yEdgjYbsmXoaFF8fgPjHkq+zu8NeRV0z+ba4FGjO6dEcJmEMkbAVpbuu4II+v+ZEP9UlJKlmex4BlePX1QsyYCJuo3rBcJf6oUP5QsHnGDsIgyTDVL3kYp4STp5zszv0CqaY/LTwzdZciAsXg1VikIbL6UO8+TUHsySY3T8iU2g2keUy3KbLuEB3BX2DTBhmmEuSBjf3WFczeU4ANHkO2AlBDn1AGGXPZkvwIgeBfh6QL8XY5ZhkgypeC81dj8WPBdozMUpWhxlBVLc4SDyDIqBwhUBzh+CNLAf3U/XA5j2P0+8L1Hf5+uQQYlrrAejQp0cfJDsuyQrpJsSpPC+TBLBS72FaPPsYusOwEgKXtNAd5jwHqWbC54gF7GIEO0Fxn7M1H19dMLm3uLglt52xWoTeRS78EYD6HhnFfMlFrc8IbvKNj9JbMsyqI/vhMzJLQcZkZY9xQWirA0smPGXHlFMLFNdcXDMohscQngCoucvHVRce/UyOXYN2IkbPGqsiotTcRCCzPST4qp4MYrGM9RZzPra68tjfUShvPOo7EuWVHxN0ExBRXZRKe7wRLDTMBMpLsmbFioEw+TaG0ET5cZI7JxBi9pTx5usAZ6I8Nfr7honA5MKhPo0+vSjBtgWvEbMneEkVKrbg6yWjmEAgy+JtSQDDBFbcqmxI+xW7KA0TaslsRXZ/Et87PaXlaf8pSQ+orjxU69EtVOxcJCuXrz2aU0UGrGI6az8lWNKth80lAVSO6GUNY9BMeWnRLfQYODR5OcAVEi6XbzBRquW1nUJt6NEW8dzH9N8kX4/XpKnZvThUB8lwAlzJU5hAEQTG0JnyklUhZkdWi+eKLJ0tyXLpvYPPJGwlofcJdFFinJjiVGOSqmiyjYUtFWwlUV4yeVb2AcVli+XXpzC1H0OEEtundw7DXKN81aJ+UBBcrS0rNjqyc4dgrgH7h8X4e+EpM7aVIntaMOjG7CyGuYZZU/LoSoKYoATBkBJk1mLATZAC3UKjetkyUXVg6i3lq+m9GCBjgE17kqG86DBMCbjoK6UnkvUYh55ISlY9VD2gP87HxF3hA1Rt4bUEpEB3nEHX3IbeDWNYjnCeqbGgfBUAaztaPlb2Dq4ykJEBDIrZD3rTwVVcW+lth2AUQV1rITcN/Lxkx6aK9R5RRgwQ4Ol1+0ZFiJtmTIe9FeITlIRas3vRns9ZESLYVSpcgOgtQWI7QNQd/FEF2Vom1N7sCIyMJkOoFEKuCW7r6FNUihLUJFVNDGLyDiZWb7BkGLsDOA4FGUeREmJTGE9iK0MgEI2PCUYDyzW37RywmAPrHX2PR/ODFDVJXVNtSQJ2zjEVNiW6xjqPEAKEEBBVSZYweikT6BRa8+dNC+Q5f5eSZY1GaNoDQI2S2OSfDElaewt3JPAphPjXOy5v3xLw/NsQ3vMLYPnze5t85X747f/1P8OXZ5fwQeDY1FhGPY4NCqu+xA8u7+E37j2Fh8DLeo4vz67wrF5gO+Q4L3e4bKb49vEz/HhzDwBwktd4WK5w0c2ghcfH21PcrzZ4tlug0HxDXu4neHdxgx9f3MXXzi+wH3JMTIeLegYpAqQIUNIjVxbzrMWqK3GS1/jh5T08Pl7ipqnwaLbCs+0CE9NjN2TIlEMzGJxVezxZMf2j6wx++cELtM7gyWqBR0dr7PocvVN4e75E6ww6q6Gkx1U9waPZCuu+xNVugpNJjet9hRAEvnZ2gWXH43Je7vA67ufMdOi9wlU9wWZfoCp6TPMer5YzvHt+g85pWC/xaLbC63qG47zG080x7s820MLhw5sznE/32LQFhAhwXsA6hVnRwQWBbtBY3UyhiwFvna2wagp4L6GVRztoeC8wK9ltWXcZtHKQAijNABcEbrYTPD69weAVWquxa3MclS2udxW+cnaFT5YnuDfb4uV2BqMcQhDo4/HwQcBahaNJg8FJbLYVTGZhBwWpPPIoZ54VHdZNcbim8h6rXYn7xxv0TuFySc9ullsqRwYFYxz6zmA6aVG32UFO7ASMcZAysK9TBKxuJjDlgPPFDk1v4LzEbltgMmvhnERuLHZ1DtsaQAacnuxw9eII2VGH7z56in/xwZdQTDt0jYHJLeyg4Qeyt6FTUJMBrtFMFO4lqjm3y0GjwzCo8TO+yAZsNiXycoDRDpubCbIJpdwpKVkWDkXZo20yCBkgANhaA5Yurv2gAAAgAElEQVRDlZDSkFsFoflHwBQW/TaDriyTinsN3ylUiwb1dQWROyAITBc19tsCIQgIEZmeTqGYd2h3sdhZBYRaIztu4Z2E7RRU7uB7hdBL6OkAuzcQWUyd6xTUxPKYAJCZgxQBzkqEIEapti4GDHV8Du2ZmCwDRDyOYpDAlMnKjJAVh4RnFcYJjC4s7M5A32jYOwMlTCpArg38hPUCCKB02Ar4heXz1ZqMfxknuDHlOTtuYZ9XCHmAPmkhPqowHHumLrcKoXLAIOjhHiKzUgb4gl5w9JL7pwPkNg61goBs6JNG9JnCCYR5HKaZAPQS5iYGXs39OGyBB1n7cwu91hxG7QSGWQCOBsgrAx+Tgrs7DqKntNhnAdkyBnipANXKOLAiW56kir4kQ6U3sQ+24SJcdgKuokTY5VGKHqXwwh3kpN5wf1xF7zQERtn2MGPiss8imxA99D6GdCXW00f/p9kyCXuY0T/uU7KzJEvmDcOzCHDow3Z5wJgWrem1ZphJDPZquL/9EY9DfiMPzxvnkQiUCGebGMgWkmRZjL5w1VAqno6Nzw9Ds1H6GUAmvhZjgnKIKbopYRogsBum9Ki7lL58KxRHDgc/qXQHCWIQlADLnkOQlBJstpEJlQffJsDHsoOXgxuvgeKG0uMkGR79cPKwbQBjKrGd8HFmx+dkAEw8ZlHOnIK5QpTFpwRkV2JM8h2VAeLAoqZE4TFpNKYXp6CplJasBioDVMPrIMjIbE7oGbalGJN08yWHPz47hFgJF8OiNMbuXuF5bESgh9jEHtjUb5r6TtOgLyUOpxqslHSdenfT6xmZ15hArLsAhAAbvbMpPVo3YWRBneHAKFUUSZtqj+KBEWngHcauU1P7UeKrWw4hhpLJvqzKAWwpkW8cq1xicrNu/C1ftBiBd7Zx6I4Usq2L0l+M0ufxehqinD+CMdmHCADpyU3yYno9cVDfuIOHGZ6spbA8l6PkNPZqpvog2fvDda/4WOEDxOAZlORDPE7i4D+VgiFCKcxGHK65z/dvys6R0ZRyTLOVdY+Qxzdk8qoaBdnbA1sZDh2mY9ps8n2m1wO8AZARwiGp9jbjeUtKKgay0iKmy+Lz9wXe2Ifb+5i2kVjM0PUjKOXv3EH2mpjdxFj+DD9nAokJeP7M2+cSY7/4UtiH4Te/89cPLH/3934BLP/Kt/zdR+HBP/mnCCbg7/7aT/HDy3v4zr1n+MMn78BaBaU83j2/xsf/4m3Yhx2yTwuob61Rr0s8fHCDFz89R36/hrMS1R9OsP66xcMvXeHiT+/iP/+Pfg//2x/+Pdx7fI26yyBFwOrlHFAB1ccGzdf5SZ9XA8q8x3ZXwmQWQ69h1xm9cHd38F5ivypxcr7B8uMT5A/2aJcFF64moDqpEf7kCPZbe8ymDW5eHEEUDuplDp+HWDkSEC4KLN67wfJmiupHBdS/t0QfwdnwqsKdr1xh/Ud3oH91BfunCwxfbTD9f0p0J0D7dg90EnI64OioxnZXwr8uEE56mMJCyoBunwF7jdnDDeqfLBh48ZUaeTGgbTLMZzVuni+AANz9fYnlNwSKC4HNtzvM389x/x9+hg//5C2UryT27zjorcRwzFRaYYH1t3vMTvfYfXaE6q0t6qczBOMx/VijfuThS4IPUVnkHxUYvtqwZiUu7mXuEG4yQAKzDxTaf3+H8MkEw9xh+rHG7j2mFJqVwm//g/fxu//yWxADvYWuJBM6TAPmH0usf8lBtgKqJvvgH7bIPirR3rGABBb3N6jfPx4XeHIQPIZe4ORfauAfXmP52TFkQzZ48mgL+68WYzl8c9dj8SOB69/kB6basqh8+u4a3fePYbZcDNUPAnwWcPxDsmzDUYB71GLxewVuvuMw+VSjWwTo93boXleYfaQYDNTzOWYfSeT/yQU2//cdtOce6ryFu8qhWjnKMcsLLhL271qc/ZHC+itgbYsM0DsFO3WQswHV98sYWhPQ37VYfN9g9S0e0/K5Rr4kkygcKzNsCbi5xfQjg7t/3OLT/ziDO7G49zsar//DHue/k2OYRE9dAzR3A+z9HvpVRlbyyEFvFe7+kcfFf9pC/2gyJtL2M8B/c4fqe1PsHwRMngv0c7Kg5QUDZtZ/v4F4UmLyVGD9DQe1lTj5IXD56/w8kzGgS7XA7Alw8yse935f4NV/EBBEwNv/O3DzDY3dOxbTj1lHs/mahV4rmD2ltGd/IpDtPS6+KzE87HHyvQy7x8DskwMrXD8guz19StYpRDZh/XVez0c/YUDP0ScWy6/osRczyRlf/92Ae3/Afd2+zTAdsw+4/hUR+2wDsjXDdoYJwc/uMddR+ZLXb/OIXtTqOT2KkGQy0/WY/GSzJ2TMrn5VYPFTYPcW2TATOyVf/bZD8dIgWwPb9xzufU/g5d/3OP6+wu4xMHkiMHnNgJr2nH6++acBy6+TaZ++cBgqidXXgOKSMuB8GeICM+Dq14DqpcT+oUf5WqL+Zovj389hS8p/d+9anP6Jws23PR78H8DFd7jgPPrwEPy0eVfg+Cce28cHZYRugeqVx/KXBE5+FHD5bYHpM4Hdo4Cz7wfs3pLIr7mQXn/DYfahwuyZQ33GRU5zT2DyNMR00oD2hCxaPyfjS8kz02KTDHTy0qE5kwiCEksG5ggm9B4JtOc8x/f/wLJgfqJQn6sx4Gj3WOD8Ty2uf0lj9sRjf1+ivOT7qrkrsPjAo5uLsXbFNLETtCPjGCTllcuvKgxzqiFcAUxecLG2vy8xfcGE02wbRg+fz+hDBICTnwxYv6ORbwLytUN9rtEtmMpLtQZ7NHf3Na5/3eP4fQlh2dF59SsGsgdmTykXXL2nkC8Dr4GJxP6exNmfdbATBVtKrN6TmD1lNysEcPqDFpt3C1QXFno34PXfqbD40MLsHC6+kzP8KwZlyQGj9JRy3QMr3M1iRVApRmA0ee2x/JpCcc3XXSx5XUpLyW2QwPFPO9hKYfM201ezHRndKrLkzYnE2ft1TJDNxooNXTu8+HsF7vyrAet3De7+wRrX356PQwNhua3ieoB0TJ+VXUB3rBh2djNg9yhDsXQwWxdTWB02jzNMX1js72qUNxxQBQ1Un+1x+RtzzJ5ZmM2A+n6O9kSiuPGoLnrYUo/HJUlcGfAjYDYW17+cY/rMYfbxDt15Bd1Qnlpctdg9ruCMwNEHO+weT5AvLcymgzcK/SKDbrh/xUWD7rSA6jyaOxnWX5J48L0aPpPo5xrFZQcIgc27BY4+rOFKDVsplM92aO9NyPi6QF9oBJXlyz22X55h/mfX2HzrFPMf3aB+d4HyxR5BCDSPJihfNfCZQneSRan/gCAF9LZHd1Yiv27H+pDuJOc1ogSqT1fw0wJ2YsbAHZdL9phu7ejjzC72cPMcet0yfdU69I9PmQ7bDHCVYShR7BENRrIHc9fBTXOoXQc7y8muZvQaCst/PtOQvcVwVkFtesi2h2g6uLM55LqGO55ALfeUvtYd/KyKybP0ZgYhEAoye+pmh1AVEPuGbOktVi/kBqJu6bPsB7KYt8J+xs5M6xCmFSAFRNNRGpuSYlPIj/MH7yaAsNvTW5mYVUMQHNrYg5RkqV3P+wFkQ5WKDKWM/ZUHf6XQGr6uf+Z6PvgA8Rd1Wsb7wf8bAOhf4vZzAyx/7R/9tW/3d7/33/+NvzbgCw4si/cehl/9n/4rhvRsJ/BeoN9lMNWAouzR1DlsrSEyDyEDfJdSsiS9jYWD32vo2UBWxokxpEY07CFUjRw9aTAeQgeEOibxDZIezEFyGhnDT0LmIRtGuKfJe4h+KWZpB4hWQg6c0puVGrv6hpmHamWcgLOrUNUSbsqE2TRlhmTFxDDlxFs1AnZGL4hq5NixF3SAL+LkzgRkVwxmcRWTbCHJCsADduEAFUb/YDDxuaOPzxUB/nSAvDbsPRwEzFaivWehtjT++/st5PMCrvJ8fROP7JLPicCFsnBRfhd7AkNcCA/zEJNdA8xGjlN8lwfYBdkRvZOxT+3QwZeORT8Po38yyeS8QfROMdQmsQ5BAq4i4EwTYgigP7fIrjTshIyIrskqqP5NpiOl1rJTMEqRizjxF4jJtuxrNFsyPrphum3y+pkNf67ag8RPWsqbk/fU7A+hG2Nibg7oHdDc9zBrSa9UlSo1oscrvq2TvFkOZDWKa8rfklfrNqsBkEnRDb2E+TV/6TXPDeXaYmQZgkTs2uR5SH69JO8VnkBV7ylb7U7INiWfnSvC+DpVf9hn1ZGtSCwMgNhXyd+n4JR0rJPvsz8KqF6y9kIOGCXfdsJt3p6Gp04++BjMEg5Mh4sMhssOU/R0nSXGSNdc4CZfWRB8HspKD6xSSkBN7IiJfraUniw89/92J2Zim5KkNvn/kmdS14fjm1gbnmOMKbc+O7A6SQoqhwNLwYANjN2PwvL18RxHX58L8XWJ8fgNs3geEnMWpdnScmFrKzGyAqohK5CYJPCjkOxBZLB8xvOr93HfooSPibgYPXX9nPLgdA6EZ5J16l+8XUuRwn3MLoxMHEDmwmzDGAYkB4yMlGojoxg/s1MybArJYTckDp5SG9+v8fwLy/RdeMS+PoyVQreDaCA4UBgmYjz/Y/q04rV4+/Podj/iMBFjx6Q3ZPLG9FpPtop+zNsMGBfbQfL3QYJeL5HeayF+RjIAKF2bug7xfRk7A8dUXTEG4JiGTNXtAKN0PQbJ86KGdPCj5HcIYx9wSrlN73/ho49Sp4AeJsWmpGOvxJgwrPoQ7QEhskd8vsTu6NixmLyHMibvinCrH/HWGtbFZNts72MlCcbKE685NFAxWZmdlPzalm/KU8dE2sjMqdZTEl2pMQGXEmM5dkqmhFteR2SofCYPn4f9YVuulJBdTJfNokrDhluf42GUwbpcjqAuVbIIF0ZJsxo8/YzRH6y3A+vIkD5f+XsZuyGDEKNnMQE2nzG9VXaOKbOIz+c8XKlHKe7hQKd+0Td7JhFu/Xzc13jurCeT19l4X/BnmYJwfkycFdaPfZPSejJiqScyAb6Cyb4Yv/bRJ2mBuB+3Oyp5zcv42cXH8eCAIDCG9gjPPktIjGmyIoXjKMmf9QOBmxCHZNXbYTW32cJbnZTCecp/Y1hPul/QMYQnPU7FfkuA0tnkmbzt+wRuMYt+BHzh82wkAKHk4XcpkAc4yHmtxeitBJCkqmNYz+1qEecoif2LUmH/TbfPpcX+bZDC/gJY/pzeZl+9F+7/j/8I9xZbWC9hlMOTH9zH/EsrFNkA5yUy5VCaAc+XR1DKY5L3UNLjcjXFW2crvN7MkJsBpbFY1SUWVYOzco/eK/z0xV2EAJQVH+M831i5GbDdF1AqMJ3VKvSdRrASp2dbbOscWnu0rcGk6tC0Bt4puFrj/MEK1zdT6Mzh0ekKL1dzHE0aXC1nyPIBubEIUUYKAEIEpkwPCt5JuF5hctSg3uVQmpLPPB/QbAtMFzX6XiMEgUnZYXk9hco8009Li6NFjd2+oDKuU0y/bTSE9jCFhTEO+1cTiMpROrjLUB03aLYFVOaYHGslhPH8X3no3GLY5PwjrT3gBWWZew14gfykoTRya6DmA4OMFjV2z+Yw5w0ena7w8cd3UZ3WaF5Ooc8aIAg4S4ncGN7jqS8x0x7eS7i9AbTHl966xKevTimFDAKqsHA7w2Tcmgl1cjLAd4pJsJ2iV3Y6wC9zqOMOWT6grTMm13pwuACQVc78OGSAAETpEFquFPVsgPcCwQuElnJK0ccU2z1TZPVKwZ5YwAk+1gpgkBBeABOLMEgm74oA2cbwgsrz67MOuMyZ/uoYWmTPmBoatIeI+yEs01rhARwNwIZpt76IgUbGQ+4Ut1vL6Dm9NewAABNllQAZr1qOQEO1ZM/6hUfIPYpXGv384O0NSfJVOg4l8gCfe8ocJUZ/bVBkS5NcMUhKJl3lx0WerAmUXQHYuYPayViyTtmSPRmglhE5pJsAfBGg1wcZpDc8j7IVgOeAoF/QA5tSWSECujsOaqvgc49sqUZJo5t4VE8V2nPeN6gQk1bj0EPG4xQHFdJRbqi6Q9CLyznMYSovxqGG6pOPlcMFOwlweUD1Qo4VMynIyZuYjJwHFK8lulMOeYKOEtMAnneB0b+qazEOVuq3LPJLFV8/rxGfUwXgszAej2zJc0WpXRyueXEAOJJAn15ZDnHMVoz7KC2HRkkGJpwYZbRjeX1L0D7e4n6nYwXE6py4HymgiHUY8ecZHyejr9VrjFU6SeaY6kRuS3Bvg0jZcwiQxSqYdBxlBHXJCxo0wd4oa1QH8J/++cgGJ9Ape+6bLTECrmxNUGMnDOVJVTRjLU58vcIzXbVbiMOxCocBgrR4A/im7Y8DS1Au6DWHK3p/C9yawzAphfDYMkoM7aE2ZQTncdiRALFuCWjTgIIDGDGmawrLaiARvaghglmz4/uxn6fhUxwapEyPzw0QRsAXj4twQL5JYUHhIIEFkG1j1U28fhJwHd8P8TyOsuAklfQYZbEiSjlT2E+SxSaPJ7eRPlOiF1aLg2fTBlYqTeUIgFMNTnntokf0cOy9TgA2jAMUW8RBaxzoJKmrbmJCbKru8DgAsIQVoi+3XbAD9La/VTdMnE1yUR8lnqyLkdGzH+I1JUbWl0FQQD+hl5bX+uEYyD7AFel80C9qS/GG3FZFVj1EnMprKYzX1+h7Tv7cIQH+VGEjY2VNrF6JAJmydDWGD5mdGwdNQVGaOtaTxF7NlPrK9weDfKQ7VL3Ag4+NHtBRzupD9NbKUf76hl9TMuAHUkDWPXxhIIY3AZCsO/hpTiAegefYZWn9GOYjolQ0jF5MjDJXIIHa9GEWZadJuprA5u1019s/B8bvxecYxBEA/tvAnfeHZNrP4wXv6dNMz+k83vBCfr5+5GdIXf9/YZDEpn7RpbDTh+HvfPuvH1j+zh/8fABL+Rff5ef35oLAbz3+GABwtZngYjOFLz1Wr2ZwXuLyxQJSBKzbYgRcFxdHePHiBMOqwCc/uo+mznBzOYeWHtZKPP/kDN//4WN8+PoMs2mDsMrQ/2SO3a6AlB5CBMyLDkNj0N4U2K1KtOscvtaQmcPNhyfIMov9TQm3ydD8eIGhMcDLHHcfLnH55BjhJsfQanz86R00qwLbpoC7LNA2GX2Jz+dobko0NyXqTcFtPa0IYhqF/YY+Nf+iBF4UqC8nCLXC9tUM9skE/SrH6tMFqnmLcJkTpKwyrC6nkB+VcI2GvswQXhYwrw3EdYZ+m6F+OUV2rRC8gPcS5sKg/2gO9SqDu8ohL3KIWiEvB4hawRQW9rIEZIC+0UBHACg/K/h9AEIQUJpeKvVpgTBIND9ZIFQO/U2BT16cwVxp1DcVZCMhfzqBDwLidQ6/M3Abg+yVQXahMT/bA59O4LYGxTMD9BJP//gh/M5Av84gdgr+soBeaugXOWQnkV0pZB+WDFW6yoEor3WbDPq0gf5phfpyAt9qyEbCbBSySw1ZK2RLvh5WkiioWiJ0EtVnGmqnkP1ZBVzlyD8uCHzjQkbuNELpUD1hWq1e8f75xznMpYGcDpCNANYGaq0IyDZMlc1v5NidGtYZ5XlLNW5b7BXUcQe1VUACOTkDndReYvJ+geqZIrjZSUp+t4rbLfkhX1xI5JcKwgqULzSqpxroJVTDf9UT1mborcDkGUFMf+JgdoI1GiDQK1/FSff9GmYrUH1qEDTBCowfF+E6esDMho/XO4HZR5Is+L0Oix9IwAruU5TaVi9FPB8ydipysV08zRBUQHElkS0JkosLiclnKiZuSlQvJcpXEtUThexGsnLBAmFiuY0rie6uRRBA8UIjW7GXdvKcUltdC5ilRHvHw5cBumZVhvDA7CN1AH6CYKm4EeiPHcrXlCCLAOQ3PDbVs9iTu/BjoE9/5NGeckGXbQ7Mn7RkeIc5g6SKK4Gjnwpka1bi1G9ZuCmLx/MbgXwlUFwKzD5jIJYtgcnz2BWZEXimFGI5kEUXDph/wG3qWmCYU1kgAjD/xI8sf/VSwE48ps8CXBGQrfmzFG6kazEG8WRrEVNdA/KVwOQ5gWC2oiJBRNBt9gH5EsivCSamz9OiFkBgzciYiJoD1UuCP0gy8kGxl1X1DEmyJYEYO2IDJi9iavANlQTzTzzMnpUk3oALtchkJlanPQtYfORGBnOYcz99RllpAn2pl1fXrMNILGYC3qyywFiPMnnpMX3mx8qR/ihWjBwJdCd8rRCU8LoMOPrYR2ArMH3m4UrE7lKM6b0E94jHE+gWlDBn2wCz5YLcTrhQnzwPoweQjGyI/boBk1cOug0xKTiGZNnUKRpGQAXwGGfbgHzlx+PSHVMSnS/9CHaLZYDe87HDVGD6nGmp7Sm3XV14qC6guvSorvwYfDRMBLoTgeImSr0zAkmzDyivPPJNQHNCplbXQL72KK89sh07VilP9ihWHkefDMjXlMhWVw7ZPsA0BEomptkWSx+BV5JRE1Tma/489ezmW49+LjBMCKZczhAgWwoUS4KZ6oI+Na8EipXj9b3yt9QEh/5YU3vkawdpA8qlgzNAcyKhG0+57rVFu2CdiKk9uplAP5XoJwLFjR2VB9nWI1/akUnOVw7NqWaYUn/oyjU7N9aDZGsbgSeZT904hhqtLLIVpaHFTT8qGjhQIautdw5BMRE3KIH8ZoDZDrfAP0Gr2XtUzykpYUiahwgBqg0x3MzDrAfoxsJserhCwmwHeC1QvG5YR7Lp4TPKVrMV5Zaq98jWPcymh50oyJYpu3rTQbVkSgFAdg5m1cIVTKFNNSGi99B7C9k5ZK93VC8tW+h1x/PTWIYHFQrDPIPqHNlGH6A2HZnd1rLOpbEEkgB/FwBZDxCDgy8MZD0ASkSPJFlMd1yxUkRKMqIgKGUHJiB6BjNhsJEB7SHaDqKzkHULWbcQ/QCx2TMoKLGQic3cNxDWQdQthw51y6/j/UQ/RODoKYG9nQYbtxPajv/6nv83DULT8HHWcluDRWhbhK7j984hWMvkWGOY9BpCZGQJKuEDhImThSF2fir15j8pIZTiPyEOX6fvP/9PyZFJ/cXt5/f2hWYsp1+9F07/238KDBJnj1borcIvnb/GD17fR70pcHS8xy/feYHv/enXIbyAPmlhMou2yXA0r7F8dgRz0mJY5/TyATi/v8bl6yM8fHCDl1dHuHe2xqvLIxRVj/pyQl9d5QATcHZng6unC6ijHo/OVnh2eYzZtMH62RGC9pje2WMYFLptDshApip3kJeMO/WFhzzt4NYZ8tMGSnl0H8+BBy3cXkNPB7hljlA6Bphc5RCnHdSzAsOphZl1DCTpJOtU/nwB96BDsBKm6uGfV+yWvCkoQbUCoXCQuSNwywNCGb9/VSDogHDaQ75mpUkKI9FrDXs2QF8a2BOL6mOD/ihOj79cY/p7FTa/1QAvCsghdhFuNQFGDOCwkwC7YJefm1vIWsEfWaibCL62CvaUHYbVE4X9VziRMlcGtkrdeRLdqaP0VAfojYSde+iNxHDiIFsJ1QqId/cIn0zYkxirV1IqrrCsNZGdxOS5QH0vjIoor7gY7mKia3vC12hnadpJX+n+HUfpWc2AEntsMfnIIChWU6hOIL/h/ULhIGqF8qVCc48VGfmKzxgEgYZqmRQbVIDZKOTXZDdSnUWSG2YbMkXla4HN1yyKC43+yP9/7L3Zr21bft/1Gc1sVrv7059z2/KtW64q24otF4FYESEIRYHgBzA8EIGETMIDiD8gBMQLDzwhISCISERCRAacFwSSnUBQrNiOm7Krv9e3O/eeZp/dr3Z2o+HhN+Zc515X2RGpInbhJR2dvfdaa/ZrrvEd304kzleadk9krFGJ/w5IICRy83nYfweu32YI2ehDPtzUM3vPyjpdSrR9rnET6Kbycw84fCGSU5I0t7zQZGtYvSJ+0dlH4qE7+KbMgNfHCDAYyQDLVsLadYee4syw93tw9UUoL2UQp1vEazoK7H9Ts72rKG5ke5tDQAn4W74RBACuxOPoS/HibW8rYd5MmuUOitGLyOYBTD4Rf6JpxFPnxuKFK67ED1jfkp7L4kr8atlaBt6rh+J/3XsH1o8U4+eSlJptItu7clzmH8g2dnPxhW3vJslwFPADUimim11IRjSK6rbUjegusr0j19PoIrJ6pAYw0cse7VaAweo10E6RL0Sq2e7HdCwSK5KqH/p118cS2lJeRrItLN5QzD+MbO9Id+PoUlJY6xPxs/bpvsW1JNbuvQvNofQq9lLQ+ih1Li4i1bFUvdiN7Fdz0BejC2DymVxnq4d6qDwRUKeYPRbA0x+fbAnVbTj8lvgOoxHw5cYysVDdkuPbJ192U/FBlovA8pH4/JoDRb6Qc1BexiGURbtUYdFKL+P6gWb8PHVd3sQhMKWbyGBZ9lk8cz4TBlN3DEyfqeMuSMWLD7I6FGDf7KtBjju6FI9fO1eJ/duxcdWJYv7Ys75vmJwGtidaEp0vBPz07FrvofO5GnoNQ6bY3krVDE3yiG6EmfK5yGrbmWJ0sUtFjQq2t8VvuP9+x/qeZXzuUR7W9wy6g/ImpO5LkWxuT/QAtE0rLM7iVUu2EZCfrwLrewa7jYyuZODf7Gmmzxz1gay4730MVhjoyamjOraMzwU0rR5Y8nXv6TTYVFXSjSTRtu9WHJjBKLLV6sAMrBhBgGK2jVSHmmwbPyVB7mXFpoVsI99tzZ75lFw3GkVx49jesoxPO0kbrRwEcBNLNFAdWYqlsITzjxrW93NMB91IYZtIvvREraRT0kfsJtDumSHFVnfiFy0vWpqDXJKiJ5rZ45rFGyNsFSgvXfIEplTZpUd34vN0pRyDfOlo55Z84WgOMuzWY9qAKw2hEODYHGboLjJ6sqG+O8ZW0js5+XDF9tEMXyimH22o7o6xG0+2bHDTnKgV2aKmujth9HyDH+cCwAot739aS5Js6nz0452vkxgJuUng0aE7jxtnyX4gn+ls0eFLI32bU0t5tqU5GeRwibsAACAASURBVFOebgilJWQGk1jAdp4JeF11uLGluKjoDkrsqsWXdkiZ7RlLXTmU97gDkUeoVkBuu1+QLVpikhDbRUPIDWZVD0yYO5pIzYgLuL0C3cjPfUUIMDDI5mpDmI+GMJ1oNbp24mfM7I51NAq92BCmY1G5rCriuBRJepLcRq3FC5mnmbskoUVrYQq1QjUdsQdVSqGcJ4xLeV9mUYv10FmplhtJaM3sADrjdAzeo+p2l1xr9C6x9jN1JXFTyXqzTABmkUOIxOSRVHm+81saLWyl90OqLCTJ6stgNj3/qUfPlmr1XRnNzz5+KDyW0/vxp3/sr37fl/t3/+Ff+6e+b/DHHFiefOEo/jP/7b9J5TKeLecA5NZhdKS0jtOrOaNRy2ZbUBTdICn1XuMbw+xgy3ZTYqxnOq5ZbUp8Cv3pVjlDIqQXSWQ5kw9RljnqOsOYSPNiTMwC2V6Dd+nG08s3myR9GDmi15hS0itNFsTTCbL8Tu/+z2QmKp+2dFUm6ZUmMj/csHoyh6nD5B79wYguSRPREVUZmHfEymCmTtYfGcJvVFouNxlxFDCTTjox815vD4y87KuJ4DTYgFpmxEKkmVFH4jS9RkdmRxtWpzOpc1lkYCNvvnHKe49vM9mv2D6bMrm/Yn0zgkoYMhTEPIBXsszDjrg1mErjJ2E43sorYh5Q9c5rog5b4k0Osw51JYZ+TBTwtrHycxHILizBiqQ0ZmkZCunK3Eu1MRv5gotZhJQ6qbzCzT3ZjYTtuD1HdmnpDr34RMcRP/ey/EzAlcj0FD6PxCISUwVNtjDiSZ2I1NJUcnPtDgJmJZ7Z7EYPHZOmkZChbKFxr9QC9DORXuaXBjcR9gxAV0o8tynoqL7rsDdGfG21sC6mUqlr0kvapFe4PWFGVaosiNlLPlUENOZLhRtHugORofZJlO2RRzkJMuqDcZSTVMr+YSrxb2YpoVM3CjcNcr3dWPGCJl+a7mtWjDBpKAHRIv1EkkbTQL5nAHpGjwDluRbfp5bJgL5uxm7VAHp7iWPIQDcit+tTO+tjCZCRICHxfZkW2ccUmFLdCkn6pwbw2e9zsAxSUPFJySBWeizF46w7qbHpfaK+EPmoCkgo01Zha2HzdJcSR9P2Zut0jJPcsn7YUjzNUD7JipNUb5DrJUbbjQWc+5KdzzjV9ph0DABUJ8CjZ0LaeWT8QgJ53DSSX6uhZ7KbCOvnRrv6I92Ib9ZNxO+LZqge6c+tHM9dPUgvUbVbSQrN1tDOIWZQXvTbkVJHZ8Lm9gzp9l4Q9rkRb3EvGybK8txEWEz4tHS0OUie1FyeN60M6n2Z/K8VQ8KmT+HQumPwn5p6VynUTXeTMm4siackmakMEFPIUPWybC95mkPfZSlVM6aNbO4pRmeyPf36IMlUx2qYHPisn7b3U4MsR96rBhliMBJuo7zcn7pZYqwrYXLdWO3uXds4gGlJKI2DfLZPRvWloryWShvtpCpGtlOWbZo41AL1+9ofL4Co1MB2qsHLveuobA4U009SANJIJivcSKW00Z00FBhSbW0jPZEmSSh7v6XuRHabr9LkjRJQa+udhDSrIs2sV0KIXNQVaVlq51UNRpgv7YRJ1k6kt+WVbKsK0OxJaFXffdm/VwVZr2njkIhKFGCdryQpFeR68lliTksln8kkA+4VH/laXm+rdL9bicS2mwjANCnhtZcY94muKCQgaGQIefKfajVIUpVL96yJxrRRZKKjBLq2YahXaWcmrUfkoNmqoznKsRuPL80gpzVtoN2zmCagm4AfmUH2qXwUj2mIdGOLH2nKS6mQMZVINoduSC0VKyHXqU9yl/7qc42tvVSepOX2fk/lQppISLLaBNzwUWpNkny1/2yGXNJa5ftCD3LYaPTAxKougToXhq7KPqVWRXZy1l5q2u9v6yTBtU9r7TyxMLtKlk62R/n0fO+/1Fr+3jpU3RKtEbCZwOrwul4Ka80uiOczclbVJiCaqktU0zL0WYYg4LVnFPvHdwN9qauyxwrKpvf0qa5dX18SduAw1ZYMPsuXE2df9pXyGSnsHyTJfcln+cfeY/knwPKP7mP05r345f/qL7PclnStHXx5YZWhZx1hnXF4/4bFcoLNHO02F4+bSwCurxyoNdntiva6lCCbCMXtrYQBXZbYtcGPE4jzSgDXNlUBRCXAZSZ+OTrpxfTn5QDIVKcE5IwdNAaz2YElPxYwpTfC4NlzKb23a013nEJxkl+p/1IOGcRCwnxMLV/27ZGX5QaFn/nBIxdKCc8JhQy2dScBMtlCJI52m6oE2A1c/Dh9wW7VbqCYsyupj0gHZwoH2iVw7rYzZilQZ1+kdjaF0ITkDfMj8aP1vZKmVjtgkwJ9YOcrspVim9Iv3SxQnpphO6ORgXjIIiqk6P21ACQJsBDvj5vs9rPvg7PrtF25fNnk1xI24nPxwrXzOAQNgYCX/EaOWXElgGbweOXClPlSBlb5SlEfy7aIPDEF08xlGfJ6+YLypTC75SVsHkRUJ+sMafDbh8D0Xqy+6zNqAQL98X0ZdPRMpwSmSB+pqWRbooHmIFJeqEG+B7ugmW4iA/s+MMZnu2ARUzPsoy92ITt2K6xOH5GvW5Eo5kvxFfWhMv0+BCsDIbsWIAE9Syvr7gNiepYMdumg3XTnfevrGnpp6kBB96AyyRW7mQzqdfKA9Qxtfy0IA5VA6igdixRs0/tISZ6wdr5bfx/s1PdERiODl26aznvRA+gdmB48cjAMRiQcY8doBiODbz9SKQBEmKji+iW5Yvrf57I/fXVB1Luahr4+wW53nysQ1tG0wjRqlxj0xC4KON2FoaiQegpL8VK1e0rel0KJ+roD04lMtJ2lz/MiDgEzPbgR5oOhO9SNklfsRuSQ/XXQe8X6MC6ZxEmDaJXuSSFdZ1MBP+1UDcd2CGwJO3Db75t485IcdCXgTruYOhllfbaONPPkHUO2se8J7cN3JCxLAFi+kdcrD8VKmL12ogepczACWqS+QTyLvbdMe1m3fenc97JUn6lB5tiHLfWfh75P09Zx8JjqBCj7aghXpmCeBJ76ABkJ8xE2u99nlwBKVqXgpgTw+v5NV6phvcO1lK5jl+ooUIhcFKgODeVCgFhUUCyFKfSlvLln+eTY66E+Q7y3It18OaxLJhI8bmKGbsk+UEj5mFjdVLXADshG20/ASPdm/5nuj4X0Rfphu6JRKQgppo7NFHrjonSqTg3Z2tNNzCD59IUsA63opsLG9ee3/24NhRrAnsg1RSYq17caAmyAAWiZSvo5Xw4rCpkWD2EC1STAFFKvo0kS0agELLlplvyFWnoxE5ALmR7CdtqDHN2EIbgoGiWMXYj40gogDVHCbYz4avv1BatErjrL5Tw0XsJ7Wo92gW6ayXoBFSN+ZDGbDjfNseuWUFhhKAtLH87jxxazdYPfMRRWZKe5wa6k81L58KmwH5Qa2M4+qXW4x4ZAKDPMphWQ+RI47QOEhAkMEAVU9rUdIbdDIE9UCuW9BPL0IOnl/sUe5L3cWZlZAY1GoeoO5bz0VvasYQrmUYn1Qylinsnf0nMxs/I+o1EuSWLrBqwV8Nf/H6MkuPYs4suBQDYF/LzcI9k0nwZ+w34YAXM9o9mDvpiWqc2OOeyc7K/Skgir1A5Yvlw10oPR74U7/jGDfH4ogOWXfwDA8lf/BFj+Ez9mb92Jf/5v/ixaRT5Z7TPNW0JUvLV3xkfrQ843Uy7O5lItcrbPwcEaoyOX11Me3rrmfDXhZLahdpaz94/Yf+WGg3HFB+/dgVzCTMYnG5SCps4oyg7nNF1jMTYwnUjlyHI1ZjypWb2Ycv+VS569f4I+aIheUYw6qlUBXlHMG5pNzmSvZnM9Ynq4RalI9e4+2esrmk/EI5rt1xztbTi7mBO9ls/8xsK8oxy3NFVG8c6I5jjAcUM5EpkuEdTYEbcWPe0ITjPdr9gsRsP9wlzkZGtF/WoDTqOT7y6sJeDH5n7oQQwXSSI8dbC0sNcRXRpNRASYRyhPKurLEXrSEZfCKBIU5bRhOmrY/oMTqi9WhE0K09kawl6Hvs6Ixy3GBngyovzcgvWLKXZhJACmDIyeWOo7cvMqLgz1g1bY10Um21NZYR9vNcSo0Oe5gMSZjJTMaYE7dJiFIYwiqpMgGjcJjO+v2byYkF8YukeNBOnYiM4CvtVQyw31+OENlx8doFoFJw36uaTeAqj9ViYyak08alGXOaZRdLc61NpISExMDEgObuYFnFR6kOTGXCowdKOH3jFT7QYj7qRDrS3FtSTw6lpj14r2Tkf+XKbwu31hO9vbTvytSpbh3qyIL4ohUMWPArEM5C+sTCpMo0iilTBzfhok8CYyDBj9zA/1OPZSZNHlk5zmSKS90YoM1q6SNHklsmM/CahOUVyIL7GbB6nQSJMWbtwnAb7k05sKONedJOeSpIA9eFROlhOzSHZj0u+SqOwe1Yy/PqI+Sp1/SvbLl7L8vluwZyF9GSkuhfnMljIJgmJgxjaPPNlSC8DPEcY+SU3HzxSbhykMaCyTJO2eyKDtOk2oOJWAq4Qc9WxjcxTIVtLXmC8U2weebKGHkJ5gI/lC5Md+xCBxdSWSmLySfdGtBOb06zE11Lckzbe6I1LhfCFy1/o4MYBWwD4RJs+ELepTUX0pYNlUScqbwmh6cC4S2Yj2itGpSE6HcI7kjS0u1cA2qpD6Do1MHoSMIUQmFHLOlBMpb8/+5jcJOKaxTA9Ii0Vk+bpMVtmKgaEuLtXAjKkokuvyUqpCyisBy72st0ugszmQ/ZR+TAFvUe2AeTSQrcSvaGqRDNvtzufYzcXr2RyqYeIrW8vvdg31iXgou6kAwT7xN1/uwmfa+S69tbiR66oPkDGNvK7ZU0PoULYSMC8J15F2X64lu02+ysTE9mxmto5s7ivKi0ixlM9BdaKTGkD2uZvK+rJ1HCa4em+dG8ukgC8V47OQJgqUvDZLbGYC/tHKPmabOPgk+4oQu03X1VrOqysFxPpCDUE+biTnu6/U+VSabZow6gOIyutAfaiHsJjiZtcd2Xv+fMr26gNzsq1c7/WBItsyhMF0I0WxEsmxK9XADrpSk68DzZ7IaX0Gzb6muBHfZbGQepZuYqTCJaX2Dmx4lj5TbSSrAj7rmV4B0TI5EWinhmLhWbxqh5oTFSBby33IjQTsqAD5wuFLQx/YJMdSMbpwtHsSYtNNzY7RTF2SthIg3U41owtHyAX86y4O3zUvfwaUl2CePrHWlxq78YRM0l/z64b2oCBkcg7KK5Hj2q0fALH2aaImgeps0eJmWZoE3q3DVkFCeGaZJMu6iBul9Fz30iSwFzY1ZCmFNgHkPixpSM+tZSzjxlZ8pKVJ5zpKuNRIwL7yAd0FqZHppNpDN55QGuyqHdJt+6RYXxh06zGLijAtxAvp+4AqhU6BPXrTSLXJpKCvBDFr8TWG0qIr8SHGIgHGPmE2gcvh740TANhIuqrqHHGc5BTJKxmLXBhIGABbzDNhNGOU550fAGmfGqu6TwOyuFwNDGQvYX0ZEwzPOYcqix2DCcSqQpUlsa4F1LadyG/rBpXAaN9L2YPNzz5i8sLKfqhPg9fv8fhhkcJ+5Ut/5fu+3F/+tf/4D9w3pdTfBP4icBZj/OL3fQP69fxxBpbzt27H+//5X6Xe5kxnNaO8I0RFjIrFusTagDFyga6vx8wON2xW5XBzVhqUiugPR7T3WuxZjr/TMJnXrK/H0GjMXsvJwYoX757s0sKOGqkuabSUjSuR9Jgriz/qUBsrEtAXU0kkTQmmGEl4jbWBLFDMGrqnE+av33BzligbrzDzFn+T7xJJbRSPZqKf7jy64vJ3bxEe1sSzglAEKANqk9JLe0mrAlUZdK0II0nI9AeO7CwTiWIRJD210ygTYWPRWwE8AuI69NgJIMwi+sYyPtXEryzYPpuSnVS0m5zRBzn1LY9NCZ6hCMSZk4CgNBiPqb4kuzaER7VIZ4HZvRWr05l0d76Q4w+gLsV7EkuPWVqiiUwfa5ZfblFb+YI1a82X/vR7fP3JfdzWYq8y/CjswOMsyLErPfm4o3sxIk48qpJQl6ihe9hQvlvSHgQOPn/FzTePcDPxa0oHphz7aCLZQiSt9T0naaWpLiOMRU6jGgnLiRbCXge1IbsWD5CpFO2xJ+aB/MwKwzxv4bl8YZhKYRpFdd9RnBnpiZx4CVNKA7cu1ZBka0V7KFJT3SjcXEBl1JK6SoDi0uA/vyH72gTtoD6Wrrt2X9jZbi9QpBoY5aE9dpilxR90mBtLttICtBAmujwXlmn7Wsf8mxnrV4WJLs9lYO+m4oPtAalpRA5bXOy8nypIcI2APlmuzyNuuvORhqnU95iNHuR/IRPg4ZLsNr+WyQ03Ewleuy8AvbhStAcCEqt7Drs2gzcVGMrT28NAcamHhNGQx2FwgobyQrH8EUlTLS8U27sxSSUlLAkE1JSXwraZWtjgXj7qxsLS6k4ktzb1peY3UuNi1yrJ63YD4JeZ374ovTkQwNpNhAUsrhMwPWCXpKrBlclbayLFtfSXSv8qNMfys1w/Ed3Ie+xGzlW2geokgVOf2NcEnvNFSk9NktTNvcjsQ/HNDlUwkaGuxec75stU4ok1qU+zl60Gu2NUVRCZb3kO1S0BI+VlZHNP5NT9sXXjSL6ScyjBJPL/y3Uvk+fiyfS5PO+SXFN3O5Aj/jkBJj1YmT31XHzZiEw2sd0949/NYP8dkUpu7qmhzmX0ItVtrKPITJ2AsL4uqJdB+lImk8qLHQCTipWdmsC0CGCZiU91e0tkozHVbmQrGfRXJ4rxC2Gt80XENpKU2k902I2ATF8Ksxq1AL12piivYmK1Zb+aAzWk0yrfgw8BmuMXCdyFnYy6m6gh3bWvUekBYV9LIqx1AuZ2Jx0dFBQkz+dUpJh2Kyzy9LnIOtupsKvbEz1UkeQr+VyOLhzre5mAwalKgFf2t9nTlDcpZCmB4vLGU++ZoftzqDCpdx7PfB2G5FqZHIhD6qsvFN1IMXvqqA/lHlIsAtWRGe5JKsjxHF0LcOyX2c40xUIkkb7Q1Ac7CWvPRndjzeRZSzezEphTCkjMtrtzUixSoFQbqI+zoUZExeSn9RFbedb3cuYf1WzvFtitgEEVoZ3pAaD27Ku8N2AqOV7tnhUQaNM9J7G+Pdu/uZMxPnM0e4bJaUOznw2sbzRqAINSQ2SGn91IS99qlRjgBKTdxCSQqsjWjm6WfJdapUAhPWyvqXy6Bj2hMPhcky87OQY+4maZyC0T05itOtr9nGzlBiZWt36QxPYgMFs2dHsF0Siym4aYGezVBnc4QdciHe3Z0T5519ROpK3jfEiGjSb5KY3aeSy1RvUewZ75DIEwKdBV9ynWs++5BAYGFOchz0QK+5JvsweOqklBOn39ifOE+Ri1bYjjQnycc7nx6cVGGM6qEZCqlNSTbGvxSnovbKZSwpzCp1hLFSJsqx1b2UtuYxS2METpqeyZwz6wJ4FLOjeAShnXvvRzv8y0zv51nwW1n3r0DPAPUSrsPyVg+TPAGvhbfwIsv8dj8rm78eSv/YegI1nhyHPH+maEtgGby8Wa507A5Cojjh1Kp9m5rUW1CnPc4Gqp3IhbK8mZUaFMIHTJM7myAnBmHaE16NzjayuBOp0WCWzyHaKjSFtvMsLMCePXiL/QHtXi3dQRe5nh9iU9DBVRjaSD0uqBLbXznaRWtYo49bt+zbUeQGzUUQBuBEKS0YwcLMQfqZzIcYdezly8e5KSqETiG0C3wj6ZTaqAiIAWUAiQXRncVHo2RUIbCGUkv5SUTVWbXVUHSLjORDx2qhJGBkSeO0TkRzBrqXtQvvfwibyz7/HsaziypawHEODhFW7PoZyW1NQkY46HHdknOW4irKepNGYrDFjMJNBHt+KDU166NokQs0h5anGpVqGXBQ6Sr5c6P/uqjv455RXZRliw7iBJmJ3C7UtIjQoKV0qnqKnUTsKpdhKv4bil9M7mMEl5k2xL/IBetr1Lx7KQEaqpJdDI1GoI/Ql5ZPzUiC/tdhxYux7IhFIAqe4UzS1PcW5wZUw+uAQEDwQImm0CSrfF69aceEmibRLrdCjybRUTELhWu+5SG4fOyn47e7aNuOu17Ks+IMlX0zd773nr9uSaLRKA6wd4cvwFUFS35RhINYUc62wtAMnnSX6tduvoZrtt6atVtBfPoQAGtevkHDGcj74Cw+cCjoKVUKF2LufKbiTkyDQi7e4DgV6Wr/b1GdmKxFCw85O2PbMj3kxTK2LyigJDNUQ/oO8ZQ5vO0yA3LuQ92ieZrGXwG/bgy7Q7UDg8n3orTfsSY5SYKFfKvmUr8YSF5JvsPZnQM4oyIVLcCJPVB6n0nZCmTT2RaYzgc/Fy1odIYFOapOjBUy/L7ztKfbkDP9lKmJnep6iT7UclGb9ImnesZQ+OerlsSBJvN0k1Hf3g+aWuxX5gL3UNwpz2D+36AbqAzR6IwY6l6WXWUmEir5d0zyRht+J/tNskAfYM0tD+WndjSUjtjxdAfSwBVf0+oYTtLG5k2cOEaAKF/TXW14m4kp1SIoHrftKj32eiHNN2T4Cd7Be75Odqd46lzD4O10GwMkmQbRDZohJJrivUsI7e62uSV1Ou8R0jtb2jGZ8KyO+PWc9O5hthXPvj3Psvh/5L2zNpadLEKrJtkufq9PtnOjl9IUmxPYvsM5EH++Rl7bs4bd2fC5W8qanWw+8kx/21oZ2A1/57o68U8YUEMfXXmPgud8fG9tUdJr0+14lxjEOIUw8MXSn3L7uRqg0BQjLmCbms21Z9V2fyImowdcCNzKcAbLNnKK8lEdWNDKYWma/I0KUKpE+ZdSOTPJpGEnSXXhjnvvcyCkjWnawnWpHyutJgKz/I7fvzLgoCI2FAGoIR0CmeSJG9tnu5gM1echqjAL02DNcQSg1+Sd3sqkh23s9ehqzRrduF4yQQY9YNYVKgOgkI6tnJvsqjl8vGTKO3HTEz6Lod2MeYmQEkymfGoJtuB0ITSJQqjrADcN4nL2QCVD6ItNX5HdBsuySv1YMMmBh3ALFf7svrAD5VMaKVLPMzvkcBfAFlzY4R/cxrlFLEvl7kM/7K/vhF7/msb3MAnC8zlf3v/c/f6/HDBCy/+O9935f7y7/+1//QfVNKvQr8bz9IYGn/8Jf8v38opf4j4N9FhghfB/4d4C7wt4Ej4LeAfyvG2CqlCuBvAX8KuAR+Lsb40R+0/JHtODhasT+q+dH951Q+55U3LvmdxQPOtjPe2j/j6XaPx+GAn3jzA752dpejyZYYFcejNVpFPrw5ojiUDks/6vjS3We8PTvl7z1/i8vVhB+795RlW3KxnRCjwnnNaj3i/r0rGmdRKpIZz7ou0CoyLlpOL/Y4/NwV1niORls+udnn4f4N7zy7zZtvnHK1HbGelUxzh1aR1c2Y0cGaMu+4uZlQjlu61qJ0ZP76Fc4bZmXDpsnZ1jnWevK7nuPpRpa1Lbl7sOT59ZxHx9c8Pj8gzz3bRYYeO6azmgiUmeN4vOE7H99hMq+ZFC3LbYnWEec0zhmO9jacn83BK07uLpjkLbWTy2R5UmKCpns8wf7YgvbxnPxkS+sm3H54zcW1sK5Z7qRn+MBICI1X5He2jMuW67MZt+/d8OLZvkhaFdijiq6xKBM52l/z4vEh0ztrNosR072KrjNMRg1XT/aZ3RGG8/jtC148PWBytGXzYoI5aPFbiy49t4+WXI/HxCpjkgKXNmcTsv2abpODV4RRhz3Lyd5YYYKiuS4hi9SvNIznNdWmQCmp/CBCeXcjlTXPS8o3VyKhPaxptxnltKU+HxEzTZg7mRyICvP6GrcuaO6LN3d6smG7KYjPSvzECwi3EftJLiB9FNCzjviklO5Lp8BIX6Y/L8FGVKexG0V3vxU58H4Hiwx36MEG9Ej6W3lWog5amqrEjwL6qKV9XOKLSBgHVCmdmt1ehzYRLgqagyDgeb+luygktbd0dDc5urHUB8IAtweBmAfCqw3+Rcn0saa+Gwn315hvTfF3GuKqRDlFtxcGSay61eAqC05h9lvCVcH4iWHzVoM9y3FHHfZKukT9mxXq8Yhu32PWZpgQiHlEBZEus8zIbrTImCuDepzR7Xs6s2P4XaXxpaQGj55Y6ruSvjt/x1KdRLoTh1kaVKfwdxvarUU3wtqPnlp8DtUDD7OO8p2S6tWO8mk2+F6r2zIwyW809XEcvMjbezIx4luZLIlWgK5IGYX9NS3U9zuy72SELEl6FwLoq3teAqxMHNjb4kqAYnMYBSQnn7Pbd6hW6leaI/FIyeSETILUt4Rdzhby+vquo3xuB4AfrCx3+1pHfi6f9eYwcvAtxdWPecafWNoDYW3L8xR8Mw2YVoPuPdGKfCm1Gu1BGDy/+UJRHyU27m4cZKF2q9i84pg8tkOVSLsfUEHT7QXyhaa6J5M3xaUMiFWAbZog8aMkI74Xh4qXzX1JTN7el65PN5EJjt7vC+KfdlM5lt1EAOz6kaThdjPI0j7YShKDy7OdNLQHtfXxrjsxJpauTXLOfvJsc1++9coLRXktcstmRvKZQnVbUZ7D9q6AwvUr8lpXQnUC8/cj1S2FCmoIrjG1sHTdVICr8rB5KOurbsk25SvxbrYHUc7HvsJuXpIDj4VxNrWww+1c7aS449SvWe96LvNVpD5UdHPIr2X92TZy86bGtDA6j4nNFvl4toroAMvXNKOzKIBzLhUrIkuW63tyGnFjGF3IgHj1QDN9Kum16/uabB1xY5Hu5gupU+kmSt7vBXznm0h9oCWMiCS5XYmstd2TFGFbf3qyyo2EIdJepJrVsWx3M1cpNEt8wu1MUSwlmVY8rrtk29UDIzLgUmpV6gM9TBQqLxLd3ufqCyVe/WmSuW4C9ZEm20iq6+ZOJvUi+5rJqWdzx5Bt4rDeYuFZPcjI1yK1rQ6tBKRtI/ky0I01+SrQTQV4wCVDYQAAIABJREFUZeuAG2uafUu+9tQHBtNGJs8btpOcfOlpZ0YqSI5Fnjo679jcyShv/NDd6EuR6bqxTt2kwji2U/EJjs+cyG1zhQ+GkCmavYxi6dMkgyFfOrqUDqu7SH2QJ79uJL/pqE9yRqc1m/sl4zMJ8imuBSg2BxnFVYcfGdxYE5WivGrp5hn5TUs3yxILarEbRzvPxGLSBLLKpVRa8ynvqj+UwKHQp8KuO6kJqT26dajG090uBQj7QBhZ/DgTOa0VAOcKI12ZkwK7bvGTXDyyIeLnhTCs4z4hTS48Pykwq0aYyrEhdp4wzdHbdgCesczRtSOMssHjGbNEbNROnt82EuQTAmiTAK0Vv2ehRTqb5wIqk4wWa6DtUD4Qy0LYyL7LUiUAWnz3VFjV91amDsshtKcP6zGSCEwbZLu8l78Zdp5LL69TShHbdvgZXpqw/25S2GE7fr8kNv7j+DD///s4Vkr95ku//40Y49/4/3ojfmCMpVLqPvArwBdijJVS6heA/x34C8Avxhj/tlLqvwF+N8b4Xyul/n3gyzHGv6KU+jeAn40x/twftI6jt4/j0X/2H5BlnnvzJQflFk1k2ZW4oHm62CNGxauHV3zzw3v86z/+W/zKi9e52UgMdQgareXC/Uuvf51fOXuDe9MF716esK1zusbyo4+e8/VvPWJye8PPPHyfJ9t9buoRL25mWOvZXo94+41nvPv8FlnmOZptePrRMZNbGwn/+XCGfrCFD8dMfvSazbbANZbYaX76C+/zW48fkRcd26sxeIVdGtxhx70HV3zl5CP+zq/9FLH0ZOOObpthR45bh0uenx6IF3CV8ebnnvPB795H3Ra/5eZsAllg/1jA6ov3joljj17awdsWph476VA64DvproxBoXTkcw/OUCryzvv35ObjFXriiFc5k4cr2taS/8YU99Mr6psSXTrUi4I7XzxjWRfUVU78ZMyDn3jGyHY8/qVX2T506EaT3d/QrAreeu0577xzXwbxZUBPO+J1LoPjWxLwQB7IzjIJpdCR/FrTHgROvnDO6UdH3H/tguYXblP/ywvWl2OySYf3mniVCwsLZNcWArh7Lawt5Z0NXWtxlUUXnniVi+zzliM/taigaB60qLXcRH/yT/0erbd889dfx520vHL/ksu/e4/NIwlLcnNPeWpp9wPjV5dsPp6z9+oNe6Oap1+9S/bGiup6hL20ZBtF9WrLT3/+A/7R73wOe1jTLXMevnrBs4t9fG0onuS0r9VMvlFSH0fcUcf4vZztmy3KRIpJS9cZ4mlJmHrGH2b8cz/7VX7pt78EJlI+zWj3wyDFnd1dsbqckJ1l/IV/8Tf4vz75HMvTGeOPLfkCFm+Jj7BnMGMeyC4tbi4/2yvLv/Lnf51f/K0/hVka6c384opmWWAWFnW3lk7Vi4LJU8367YbRBwXd29tBnhyVsN6jFxLEUb3akZ9a3CyS3dtgvjoTdhPg82u6xjL+2oj6J7boD+Wz6nNhie1G7fxgFrJX15R/f8b2TqR72GKf5QMrqryiPQiEPJAtDeFhjf64FKnigwZ9Vgx+Y7WxlC8M3USYyvn7cPKXH/Per70iAU8Wmlue0RPD9qET5r+MTB4b6qOIO3TYazuk0e5/W7F+VcKhumnk+MfOOD3dl4nktR2CwObftlRf2dCtc/a+nuH+7ILN2YTRJ3L9aQ/bh+nzMO+ItSE/s0OSrnQRyoC5nUfmH6YqkJ+6YfN0RtRRKm6utNTOfCkw/46h3Rdmrr4VRVWQRebfsegWFj/qya403UHg1q8qzr4SmXwi/q3tQ4+upYd0e0+YEFOlQfmXb/C/uS+M776AZDcV2XW0kb1v2dQLCM2xZ+87hs2jHStVXqiBCa5OItMnIrdsDgUAVw8ckw/twJJt73umjyXBuT7u/ZaRvffkGIzPAmc/BdOP9S5QqpMU3j6Qy5dw8O3A6qFIo+vjQLYWViZfKTaPfOp8FfmvH0u1j6TgxgRABchGLeFhoYjMf0+u0cufkH03ayMAODGD4xeRm7ci04811a3I7CNYvAmTZwIat7eFaW/3klz2Urye7UGgvNB0Uzlu8w8YQpfaPTUEfO195Fm8mgamiSWuD0W+PT6VqpdmH4prYfndNDD90AwhTi5VwGwfOubvWCangepQgkwkuElA4fXnNaOzHtAmRnMZRSUwgtVrMHkilSftTLav3VMpHEx6Lqt7joOvGwlE0nDwrmNzy2ArqA/VsP12uwuw66bCMh+8K1JNkXHufJREkZ5ub+kk6474Qry62SawuWMG6azINoXpXN+TYza6CDRzkVC6kTD0h9+OXPy44tZv9LUnsi8hg/lH0g0ZDVTHfYBOktfWkdFlSFUlmtGFJOxWx5qD9zoWr2WMX3g2dw3jM5Gp+lwN/aLTp562B46bkOpQMgnSagQcz554XKlYP9DMHod0z1SMzzp8qdkeG0bXgS4xsPWBYvIikK0cm3s55bWAUFsJyJ0+c6zvWiannmiRHswqMjltIEB1O5ce4EvH6mGeenBrVq8Uko5byQTI6EICjEztMV2gG0tHcn1gGJ13VCcZ+99ecvP2nPLa01fc2I2nnRvyVZBQoGmGGwlADbkiX4i3s7jpqA9zRmcNhEh7kA8hTqb24o/00ndprre0d+dpgAquNOSLlm6eJw+7ZvK0EoZ2Iuxk1ApTdcJ8KpGm+tJKemxiQgf5qAuYdUPMLfXtMcVVQ7Aas2lojydoH7A3NTG34sfMLbhAmObCaiq1+3uMQ8hPGJfoTYW7vYd9sRC5q4+8XGeiugQCP8OAqs5JCBAI46mUgD6AXhqb3jMEIMEOZPpAbFuU1iKhBWE4X0p3jc5D14octl9PjIP8NbZy/H4fc5rW8/uksH9IMuwPhcdycj9+5Ud/AIzlb/zRYCz1H/6Sf6KHBUZKKQuMgefAPw/8L+n5/wH4V9PPfyn9Tnr+z6nv5vZ96dEFw5955QNeObhm0ZT85uNH/OrXPsfFf/cK2y4n/to+uXVsuhydBX7ht3+SF5d7tO/P2Z5NqFeF9Ew2GV9b3Ofp+T6/8Y9+hM3vHhKj4nP3z/j2r77G9H3LZlHye8sTzrcT7kyWtKtcei1Lz7ffu0/0ivq65NmLfQ7uLdhcjmkfT0FFukWBH0WsCbjLkhgU2YXl17/1Bq4xOGdQtcbMOtzco1eWq9WEX/z1n5TAGB3hozGzow1ukYvKY2kJFwVmZXjvvTuYWpHljvbdOapT6JWlzDtOT/dlkL22vP7lp5TnSQrolbBgH07gomA0baAx2OcF7753l8eXh4w/yMSjuTFwVjB9bNgsS/amFZtHnkeH16ha8/D2NaZRnF7NWS9HxE/G6Aae/PY97o8XVHcC+ZUhv9bcPViSnWW8f3oioHIqnsKwzhg9Ez9LeVQNMjhTq6Fsnh9fYmrFxfWMvW9bnr3YZ3tbsb4ZoRqD6wzRacoXBrO0YIXNiAb0eY6uNPWLCfFZSXaWMZ3WjJ4b4utbkQI7kbdl5xn6uCFbK7768UO+9tF9CX3ZWh7NrqSLcCOeRjXyQ31HVQmgdUHz8bMjsrWie39G+TTDJvmrPc/4jQ9fER/l2QgUPHl+iHpSotZWZtkzz+aRF5nv2tLNo/h5Fxn1dYlfyeysWcmg+u//8o+n4yVMVnGpMXnAbDWr8ynTwy3FtWLjClafzCleiH9k/UqkuDLCAB10Ipvu9CB/zs8s5ZXi7/zDnxIvrpUBnns8Ra8s0480bplz52gh+12CWlt8GSXYSolENF8qsqUAm+YooDeGyVPIFpos8+L3GiGy2G/NsI9LbJJUmq2wnroD3QhwcPuebKXIl4rmbCwes60iRhid952KErhi1wq7kkF9OWrF3xkgK52cUy+fBbPRFNep/zPCzdvwncd36W51ZGvYey8QlQyYTaWHJGTdwOiFks/TKGC2wjL3QTLFNbgjx9l3TjCXKVVkKmy0WQmb4C5LMJHiOhK+ugc2MLqIg1dx9EQ8xdpE8j25Lvu0WbuFbCnslEhZZfs3T2ZDzYxpVKreUKhOvGwxeQi7IwdBMXpimTwPAhajHIf81pZ8nSTqSRqbX0oS9eaBhO+YSiVJHxyMK0yTGK8Extt7LeNPDPbGYDeynbYWWXuxFJm16gRUShVGSn4di5+wuBG2T3eQ3RiyTe/vRKTVVZKJPtpIKrFJ4SG5eOvGz/SQQh0NbO+GJE8WsNH9yJbyUmbglYeYC0AtLxXFZRwYeu0EaBYXmtF5JL+Ra067NHkwEy9tcS2AE6T/cvTMMDrZyvZv5Di5BAp1p4RF3A8D4BTJaHreRbLNp+XCg50g+SXzVRDWtJVroZe7+lxY8fGZ9ISOzlJIlxYWbnQu+1Rcy/nObvSQgIyWgKl2L1KcG/JlHKpBlm8KQDt8p6WbJJByImmvIVfUtwLL10XO2O4pJp+oIUE2GihWnuI6DGnP9V0JM+u7XnUDxWUngU1Lz/g8UB/J34MRxtDWkXwZBzCQr8XKkK88tk5S40oATXETGJ0FOU461bpYlTyGvSS591mma0BBvgysXkv3oEbOtWnFo6wCFKuAbSLtHOaP/SDj7I+hCjB/7MjWsr2mDuRLn67R5PkzQIDReSDbhJSG7Xd9q3UUdjlJO20VRApbaklMLsWL6fMk7YykhF9hPG0dhuuouiWhPW6smD3eppCogN04kfmuHNs7is1dkyYJNM2hwrQSbtNLo826RTdOmEv9UkjRJqA76UMNmZy7yXMB2yjwI6n2yFYSLtPsKUwTqA8EhIgfN8gkQRsprhpWD036DujIbho519cty4cW1YlH1i4a2UYrADCm6g83FiZQv5zWamV5dtNh191Qf0KMrO9aYdCSPzNqha67T403dd1hth2m8ejGEQqD2bQigdVKPJ2ZEf/mWpJXQ2FQjcOkLlTx8mrwAb3cojqHWTXoxVbAZNVIbUrjhDH0PqUHh8HbqepOanBWFXpbo7aNdFNqLexkDxb74XOS96rVFnWzEglvld7TpH9VLb/XDVQ1NC2xknDKgWGsavmbSr7VNq0zeGLX+z/V4N0ERCoLEt5jtCyrl+t+l+oROdDfY9ifUmfVyx7NP6YPRQqV+j7/+6Py+IFJYWOMT5VS/wXwMVABv4RIX29ijH001RPgfvr5PvBJeq9TSi0QuezFy8tVSv088PMAxa05+9kWrQLvPr/FeNzQZZ7tv+YY6cD2CzWhzWg6ibAu5w31VYnu/YiNxumMfNzy0dUhRAj7He3IwHXJ82KOm3tqZdCLjPfev0N5ULPcykAQAmEp3XLBZ6Aj+qxgmUlQT1QCVkC+aJabMoE8Q3enFV9mZWi9QjtFuMkxBw2hK2hOx2L3SKCw2/d012N0o7lYTtJgLw1uN0akdOcjdCkDAFUrVlUpPZRjDy7y/rfvoR8E8WNsZXDc9wZWz6ZQBvGZREW9LLAHkvTZe7nWr3qy0nF5NSXOHM9XMnh9fj2XL7WbnP37S9bPCxnYZZFvXN3hz33l6/yfv/ol7AaefPUeGHCVJV8r2olUd+itprrrKS4N9TrHLg2h1tKH6IQFKDJH5RRuk7G5F2GdUZ8E1NqSLTVhVYhfEhnAxZWlPhYZYJ9EGU0kjAAFnTNgwZ+OUHmk3RPmZvKJYTPOIQf90Qjm4iv0o8iv/ObbjMbJi9dCq7JUYxAlcEnD+lRiKkMWcTOP3UrvpXj5IGwt9R0nEsfK4vY9+ULhkh/MnY3EV7ixA7MS0pdiwCRZpKZ5q8I8H9HNoTg36M4OPjrzoYRURW3wj/cJE/h733lLPJWO9GUfKC/MUF1gFla8iguFm0pfpc8EBLSFGTyko3NFu5dYhKXh2XsnlBdGrjsnfsyb53PxBafBbDcLLD4n7MPoqU4hI5H1iyn7JqVu1gzVNyGD+Ez2QSfPZH4j3spuX9icbiIBTm6UVD0bS5cki71fzlTCXgYD1dmE6RZUUGyfjMnWWnxfH+TCRhUysOzDbfTTHPeoFvnpgRbfaEkKxlFEJfthWlCdYvKxsJdmoyXcZSWS2PJpRujZ1mWJL1N9g0u1GyuNutEs35D02Px5Nnj4tBMgNv7YUFcF7ThQ1vL3wc+Y+iO7vYBbSMn9+KlJ/rGUZqpE7mrXinaeKmc02Gsrky9a2KHiWuFLYZ3c12Zsj4V5BDk2bhKxa42OyWfmhfnspnD2a3exBnTy6ZkKik/ywU/mxmqoXVFOBhjZUlhJENlilwB7ttBkS5lwKG4UzZGEMulOvLrRwPhUglFCprDfnGIaqc/pJjtvoKnT95ERtmz8TJ5QXs7l6HfGLF9NFUppwqj3V2sP83fskKTaTRQkj15MHtZ+X/LVzts6+0hja2Gl8iW0vzUnjwJiqmOFXSnsNlCea1QMzN435OvA/jtqqGPqfY2mipSNojoR79zoRTpWW5kgaad6YCmLG0nqNTXUB5rihiS9lvqXvXeSxLWU5FK7lfdNnsn5t5sk4Wxh9r7e1V00MfkvJUANxH9nWph9JLJRV4h8dfo4MYQbT7NNVQ3JptX7GVWU45ZtIgdf05IubAL5Sr4zt3cL8k1MyZ8wfRJTAE6qVckUWSUpoYNX1qUE0k1ge8swOZN0VxXEr2jrSLEI4u0s5RhNn0hqq2l7VnQXRqRCZO9dgIhJ8mVXKGYfCZBa3TfMHzvyG53ST+U92SZQJv9lOzeMLj3dWNPNDHYbUt2LfOcWN+Ammr43dPJMFADZylPaBNi2Wvojk/eyTyG1PpCtNcEq9j/wIi9dB46/IYE6ugn4XMKBYoT544BpAuWVopvnjM89tpJgnGwr6z36RifBO8nXu/eB+OWChfG5gFo3LzCVY3TpJaBoZMiqkPIKJC3XNrLNbixy2r7rF0C3ntwFpqcCKGfPHH5apL5PT74yQ9XJ4bc7dBfwI+mDHF10oBWH324wjSdbOvwow259ug8aiiuRyeY3EsRjli1hnPWDUhl89+xiJcPQbNWx96GMCVVEvJWVQ3Ueu6iIxoBOaa4uYDYtBHm/ah3kFlN14sOcFZhFBQFUjGTXFbHMJOhHI2xlKymwIStRTUs0Fh0iqvHEIk9sY2L7spS2agy66uTYZAq1qYZeSwX0tR6q7eirSwbWcag0SYAvJL9m50TC2rSSCAvio0xeR5XnO7+m1jsJbF/hkiVZrrWoLMlgrYWmITa79FllLdEn8Pmy/xNA611lS/o5xu/htfxMgM+fPP7oPn6QUtgD4H8Ffg64Af5nhIn8T2KMb6bXPAT+jxjjF5VS3wD+pRjjk/Tc+8BPxxgvvusKkLqRL/yX/zYHZcXTxR51k+E6g808McJ03HB9KqDnldfOefzBLbCBbNoSvOGV25d88OFt8fotrfi3WoU+bgjnZfrSihLws5TQHVQCchMPnRqCdnppXO+Li3lA1Zo4lYTWcr+meTGWEKCxI1wVUjkxkgCVbj9I8f1YmKao2dVWbAz2pKZbFFJrYSSExi6NMJy1hO7EkR+YEJ2CUoKJtHecgFiTbp6Nlt7NrUG3GqIA2Ggiauwxp7kENdztoNOosUNd5uLNa7RI3+7XZB+MMI14uHq5YXerw1xZ9P0KdzaiuNIEG2mP5UZplwY/CpJaWuxmsk0tM/mhFIbErmRQ0O2lm9heS7zO0Z0SP9uJJ782tPshfbknqVopCZduLJUO3V6QtMwzi5uIZE1AlSQWNseSQpotk3ds36FXvTcDRi/k+FT3PKbSFJdSD5FfGdojkcm1+8KC6CYF03QyaKtuBWzyxzW3pFtUO/GpVbfiANSKKzXUIrQHIUXrywDetMKG5NfiGes9bhKKIQO++YewfpBkZvOUvrvUhEwY23ZffFAH35Kag+ZQlt+nzYpULEl9MiCxJ31HYXEFzT5DPUh5ITPYfYBLtpZajG4qv4eUzBksA+izG9g8EkYvW8tkgrDRsL0fsSuFS3UVeaqWyFYKP+59gBJMI5+LFAzjhelYvinyxN4D1+7J/UFm5Xe9mqMXIgGUOoldfUeevIfdJIWopHHI+LkEcWzvyjkd2KNOrlcZPMp7TJsK4aOkqTZHkN/I3+xWZH/dNA1ck3QRYHtHMXkmYS3dTBikkO0qNARAiTfONJFsC82+AOy+usLnMugsriObe4p8AeMXQYCclnPeexSHEJ6GoQIDJed8e0fAS7aS14xfBJavCPjug4vy5Uu1Fn0vZwKvo/PA6pF8XgbmaC11GL3sEeS1phF/X3mZ7gFKGNVsneS1jWz3y32WvbyuB0eji8D6gXz288Xu+NbHAhz6mo6+LqWX2bqRbF/Usuz1fS2JqyORQFZHmmIpgKkHO91EgKPp5Fz0fZx9cJLpIvWhxlQx9cyKZHvyIgygpxvvfHbbE02xEOamvBYfY/+7SBgD3VQNaoi++3Ry2rF6lFEsBKBUxxpfCOCWbRWGqJ3LMqSXM4GxFFjT7MnEgq3Sca3TPcfKun0ptR4+V5SXIrvse3S1EzBWHcg9MtvI/bmdSjWHbSQYptmTz+Po3FEfmUE26DO5xvK1LD/bBEzlqY8yso2wr/WeETZwJufT1nHove3Db7JtQLeRZt8MVSTBCIA0neyXzxX5KkhvYqbI1oFuZvAZFMtd/2HU0E00rlRMnzlJhR1rRued9B0mXZcvNeVly/Z2Qbb2uJHGbsOQwqpTYqryYGrp2tQuir/wUEBecS31IADlRUt9kpPfOAGeG08o9PB+Xxps5Wj2M0wtSbNuaohKpKC6C7T7GcVlQzcTL2C27mj38iGh1Sdwnd80dNOdLzG/aejm+ZDQ2s4zRs+30lcZhGnTtVRwqBBRXcCNLW5q0I2ksKIUbmLJVh269TRHJdlKAJZUcgRh7FIgUDfPsJWwu2bZ4vYKsqst1cMZ5YsKN80FdAF+ZMkWjYTtKIUvNPl1LWmqtSOMbBq3aEwtnspoFWaT6jaM+L+V77uutTCMtZPPCEhYT2aGkB6UGvyRqumIKcU19oAn/a+chPJEJQnHqhXAFItM/rcavaoZ+il7n2PnBDiC/K0HaHUrADKEXWAPJIZOyfNFJjUjmRVm0ehPS0u13rGALwf3mN166GWm6XXRuZ1v8jPyVJQihiAMo1Iid+0fPbiLLy1Xm10NSftSsM5L2xjbTpJke8zx3cJ7/nEePwQ9lnuTe/Erb//89325v/Rb/+kflgr7PwF/FjgGXgB/Pcb433+/t+MHGd7zLwAfxhjPAZRSvwj8s8C+Usom1vIB8DS9/inwEHiSpLN7SIjP93yMTMfVcsLp6T5vv/qc09WMDQWH8w3TvGVkO65fzBl9kqFejxzcWzAvG5Z1QZsCaeyk42Bvw3mzT3Zp+fGfeZdFM+IrP/oh/+Mv/xnMg4rXb13yzuYedtJhM09bihdPTwLxrETfqrn7xoLz5ZRmkxMbTXZl8Q9qisLRNhZjAtlJxZfuPeMbz+/Sehn8zN+6Yf3NQ6b3l1gduH66B7GvsNCgI8XdLfcOFnywuYW+1eKWOdPba6p5zmzcUFeiU+muS+xeS368pesM4fcmuFlk/9aKTZUzGbXUbcaDwxtWTcG6LgDY3IwEEAMP7l7x4pM72Epx/OAKk8KJPin2iWcjjt64Yvm7R8z3N6yKkubEM/3I8rm/+D5f/cZr/PRbH/Dr774GUaUqAPFKbT9f0WwzWBi+8OWP+eDiiPB4KlUPGrmpF4FX3jjj49ND7v3IFS5oWffZIV988Izf3T7C3G6Iyyl/+iff4Tf/7tvEqcOfOJplDoVnPK9RCtwnM3wZiRPHaNYQT+f4Q0fILOpIAn3cIqc8qfipB4/5B7/zefSsQ13nlK+s2C5GmNLhliNhp8YedVKxHY84eOWa1eqIB58742l3m8lrCzarEndVgIqU9zaY/3vO6I0l23VBqC2YyPj+8v9h701iZU3TM6Hnm/4phhNnPvfevHlzzqqyXd1GNsaLBtTqFhI7kEAg9YpuVi1YsQPUYlzCjh2tllgjthiBRAskaEy7TZVdLmdlVWXeIe9wpjgx/cM3sXi+749z0y53WzhxleWQUnlunDjxD/FHxPt8z4TNqmYP5oMeeFXCfLiG/8dzDAsCp/kHSyxfzCECOwjtLMIfWQzRwB85qJ7smrsY8PDiFtfrCe7kFKEK6BsPdacRz3pgXQMfb+G/NyW4LgJ2F4a+oMOAMHMQO4VYBajGYYgVmtcC6/c9+0F3GuZWQXy4xdoqVD+omQIJAjDx3RXcmwZuAZg/1Ng98inFl4sL9T8yGOZkmtVOYP2pxcMn11h3JTZfHkCftegHhYMfFzD/yg02/9cJwrsttpMSeKrw7q89x6v/5R0MJw76TsMtHGwnIZzA/CcSmw9dko8a4EGLdmow/bHB+tsWiMD0dIsYBdqfzJiCedSjuqrRvj9AFh7y92u4WRwXNyAA/35L1rkKaD4vcPsrAfPPFIZzh9NHS2z+j1NWm1iB2U/prxrmMQEYepfzAoGvAIBVIaEg6O9OIxAFps8idg+Ao9+PEP/qNWb/5wHu3jepX5LAYPeeZZDSTmH4dg+8rFC/IhvUnkbYeQqxKYH4uIVfFegPaT5ta6A7TQtLOsIeechWYPqlRH/IkJyDHwls3wGqa4Kmu295xJnD5Icldg8ihgcW5a3B7onD4e8qxCkXVNRnCr4GNp8OKF6ZVB1C/7PZ0G82LOIoqRRf8b5+IbB5n5VE9tBh/gOD9ScW/ksD17AKZ/PtAfUXBdonA6qnBVOcBdC8SizmFtg8CahfS9gJMMwlhkVMUkVg/WHAwQ8Fdt+hR3nz6QCxZf9rdYkkcSbzWf/I4/ZTsvDDr23gf28K4YDdAwVfRYSnEsvvOhz9DpUC3XFi9TqRgDz/r3cE1mYFrN8LaF5K9Asu7vgiQlqJYkM2vD0h0zl9JulxPGQIkRoEuiN66oKhDFX1Au05FwmC3qfuSq9TSi3DddbvAa7hOQlapL5IeqeLlcTqfcp6u2Pg4HOy1bd/xaN+pjF5mRaQnIA7Azt+AAAgAElEQVSdsKfSV8DmA4+jfyIxHAgIr7H6kAtgk+dM2NTPI1Yfkf12U4nz3/ZozwXiNeAGlSp3BBnQQ4XuSKYkWrKsrgGO/yBid6pwuPTYPKLHbZgrzJ5abL9DD7FrMMq4dct0Xl8SRC5+FLH8UKNYEzRyUY51OACP02wjdMcOR0SCR8+vPExeebSnBqv3JCXzWx5TtZToZxKX//KAj/5BQD9RKJcW7UmRnqfA6j2JxecRN99ROPudAFuzViR3c9qpwPHvE6yvHmssPh/QHktUtwS2m4dUiZS3lKcOBxq7M4nqVmJ3xr7MoBnSM3ktsHmgUKwp+d2dqsSoG9RvBvRzCWmLsQolGAk7lSkFOmB7oWG2EdUrh+5JA90xWVdvJNpjnZj6gN2JQnmtYA8MzMrBTjSGiwLTZx3c1CBOydpvLhSZ/C8jiusWdqYpHRXA5oFG1UjYRqC69ZB9gJsy8Gf2rEe/UCjuLO4+rDF7KtAfGUA06BYKZmOwOy8w+3KH4aCAa2iL0a3H9kGB619WePQP+TmtdgNWH9SYvLSQLsA3GkFzH4aLBtVVh2AYLOPnmj7J3qM9MaiuBexMIWiB6tpCZyCqCYCGwxJm2cMf1PClhOrZmVl9tUb37hTCUT6ceznNykJbD7eoCcBdgJ8Y+GkBfdvCHjVMKr5jhHSYVRAd+zHFziFMKqCi0kl2Dm5eUVI8MI3Xz0oo5xFLg6gUAW5BAAsf4A9qqO2A0BRQb5bwR3NACajLO8TCsIZkuQEAxElNmWw/AFUJ1CWJ5OzBdJ6ANwFQsesIFusKuFtBzLjaGTeUyohJw59lkqYWhpLaugIGS++kZ+hPlBKiMBBKIfb9Hw8iY3i7iuT+7X5ibPimHXx/cW8xxn/7/4/tfJOM5W8A+PsAfh2Uwv4DAP83gH8RwH9/L7znezHG/0YI8XcB/Mq98J5/Pcb4b/5J25h/eh7/w//hV/E/vvoldE5DAHh1fYAPLy7x104/x3/7j/4a/sZ3f4AvNkfYWQMleKyldli2NUIEJoXFzbZB1xn82pOn+P6rB+i+nKF4vMX5wRpfPjuBqh0mTQ8lA44nO7ggselLXL2e4713L/FqSUO4tQoniw2UDOisxmZXwVmFjx++wWdPL/DJu6/w+atT/HPvPsPvPn+E909voGTAs+UC1iq8d3KD3ms8vzzEX//oD/EPv/gIdtA4mO9w95NDfPQrz3G1a1Abh91gsFxOUNaWgUEvD6GKAN8rwAmUiw7zSQejPL766gimYW+AXZb4je9+jh9cnqPrDLQOKI2DlAF36wZhUJgudqiMw+WrAxRTrj4FL+EvKxQPtjic7fD6xyd48ukr3Gwb9IPG0Bo0sx67VQVdWdhVCTVxeHiyxLOfnmJ2sQYALOoOzz87QywDJic7eC8xvJhg8t4dNi/miI2DqRxCkNDGwb6YsJ7EBJTzHv2qxOxki/73F5j8yg1uX85x+s4Sh1WLu77C5c0M8bZgCM1Bj/h0AnfoICuHGAQDjzpqzepZj+HLKar316gLi01borusoRYDH+MFLt65wbqt0LUFTOFwPNvi5R+cASc9IIC6GbC9bFAedpjUPXpr0HUGj09v8cXTU4itgjga2AWqIxAEGetlAXEwQAhgMumw3VbwG/aFmtoCXzY4+OVr3K1r2NsSDz+4wlc/PcHswRrzqseLL06SHBso5j2GuxKi9MDaQPYCxZMNQ4p2GvWiQ/dqgocfXeLFl8cw8wH2rmQFTRkgd4qJqzoCZQCspB+kpMz48IMb3F7PEHvJ6pqDAWFtUL3UGD5psTjY4ub5AnqlIN7dwV1WOPvoGuu2xG5ZMyVXk80XQUDMB1Q/qLF74lCf7NC9miDqCL1UcKcW5awH/mAG890lts9mDLpREaIIwMpAHvXAixrRRITaw1xrhDIinAxQr0tKk7MKaBogDtP2PhyAICA6ieP3b3H1Zg59aeAOPFAEND8q0J0HIKWqHv3aG7z66TH0mpUt9oMO5R/WaN+10Nca7pDdnyIA1Sd37L4dJGACzKWBTcFO3XsDpPEILqURlh5N02PzxQGZ8o84dBTfb7B7wg6I4oqJtMUdw2Hi3KKcDIhRwD+dcLgsI/3UlkCHHYmUHoZHHcLWwCw6+FcNzJ2EayL8wmH+ewar71iIgcoFWXj4VYHZZxrdKYNc1FZi/kvXCL91guUvOzRP6fUNml49tSXzTwaUJ/v0O5e4+9/PyTKnOpzuCb3biMD0pxrDjN4wiIjiTmL49g5+a6CvNXQr0J94TL9Q2LznUb1Wqb4nwNxIDI8smh8VDAbaMPRl8qVGv4hQH2ygfneG3TsO059wKFeWgTWqF7BzeiCHE4fyNY+leiPRftrh9H8ucfsdspr9uUP9lPJ2aYH2QYBfOFTPisR8MAAHEeyEVVxYYIUIFwu373pMnnLQ7w8j7OMe6nWJ6g1TadtH3Md+QbXC7mHA5LnE5gkB5cGPgNX7BB52RtmuHJAk2WEMbnJNxMFnEt0xWfJggPaCEsnp04jlt6gusFMqF9afOIhe4uH/xmTT7iRi+hS4+zjLeqk+6E654KIGgQigfkOQ1p6K1EkLLD4jQ9sf0ks8+wKwM3ao+iZi/pnk+TFUPJR3ZCClpXd2827A9JnE6lMHuZM4+V2mB9splQKuIUiOAti+I0ZVgNlwISg/x+Hvk9W2U4HZC0/QZshUC0/gnBnwDCqFA3xNJUL1RrBL9SoxfGWuGokJlPM+13C/1u8Bi8/IePYzifZcwKyoJCDwIyCXPZNbXU0frdkFgsjEgtuJwOaxwPHveezOJOrriPU7ErNnHv1csh/Vsgt2/pOA9lSiXDKRV+8Clh9qmB09pKv3FSYvWI3SHwrUl0yQdZUY/brrJxKzpwHdQmD+zGF7pim1Tey/6iO25woikvn2BcF58zoxshOBYhtRpzCgzSMN3UYU64B+rmB2AaqP6BcS/QEDioDkFe253yZ1WEobsXlgMHnt0C8UJi97rN8pUWwCukPKwlXH/alvAspbC1crrB9rLH48YHthMHveo01s77DQaF50iEqgvShh1mSLi6VLVSwRqifTGEqCyvt9pK5R8BU/rybPW6a/puoVCDBF2IYx2CZXlrhGoXrTIhoFO9UobnuEQkH1DA0SIcLXGsXVDvawJsBcdXCLCvq2Tb7YiFAaCO8RKgO13BFsSkBuesB7xEkF0VmEphyrSiAB0TvEyiDUhjLZziJWBJ2id4CSEOsdYsOubGgF9MO+ZiazodVe8ir815jLEAhEtR49qmP4j/d7r2VOgc3AUNBrGZ0bZb1Ca0AKSmLz8wsxso9CKYLRfxpz+ReFsfzWv/tn/rz/0+/8p3/uxwZ8g8ASAIQQ/wkohXUA/glYPfIIrBs5Svf9rRhjL4SoAPx3AH4VwA2AfyvG+JM/6fknnzyIH/9XfxtaBVy+OqAkdaMY6z+3ECuGprg6IjYe+pordL5KXX4HHupOIScbuinZlWJFpgGgl87OOci4OnKFuWMghp3yS5xeyizVift+whStz54yMfZB2oOA8o1KckwxyrNEVghohk/orRj7zbqTVOpuuA9mJaG79EVYMCClP04ynDZ5CgsOQsKllMzAwYeeN67cR819hcToAwyaaYeuiaNHNEsH7cG+fH1YBEyfSqzf475BAN1HParPS7gpPWXtBz2qL8qxly8HVMjh3pd9GlBFimi384jmFb1SqiP7Y6cxDRaUmEmL5OlLIQxZetlENC/FWCKP5DOTLvnnkHxnQ5I+3rsfoHTRbIEhdQIC3OfcLegmlDjmgSUYjF16vtpLDCkbAxCRqg74t3K4N+ykY/MF/VE+lWOv3w9ovmJc//YhC877xdv7YtYcOool5Wl2IsZC8e6YcjPZcxh1dXr9GtYDMBhif8wxpazaaQqq2NKrxX46HkN/hHFAMysOfvmazxJLvUv7sYroj9L+CoyF71Hw9TabmKSjKdBoFxH1vQ7CnhLFYc4BOoe65L7H3FuYgyGylNPO6VOyDdIXJ59vmCNJHbMklMdntpR29gci+SpjGtgFiuRXy4X3uTi9uKNsUfX0o+VOxeGAjFbuuWSYSBxTMrMclVJSjH5Odopy+6rj76KgpM/V3I7qgc1jykYBjJ8J2QuYw0KkS6/LwNczKD6OfY3J51jzNQ0pPCX3Z8oB47Blp8ljmboHXc33i0w1DCq/l9Nrwc7RdEzJ15jlpj51wQqf9jGFjCB9PnqTpMk7erlypYOtc7cgJZR2wu1yeOX++YJ9iEGRaSuXcUzK9AXP3TBL9Q+OPsPmip43Jqkmae7Awb+fc5j0qSuRslfJwXSgnFQEJICJez8nyZ1M0riwl+96k3pQcyelIPsWdPJB3iUpcsQo2c1yW9Vz8PKGf+tqslbdQVqgSGmmuWdTpLL7/Lt8nm0tUK7COGRHxce4iueGIVsidSRSxlquGIgjPc9H9rKKGNmjmCSevsjyZl6brubAarYew1TtGbJtQChFCjnhZ53qub/DVGLy2iIYSh6zzBRA6hGVUG0Y9zt3ENIT6Ue5Z64Vob8NY6+oTHLIYLjfwgVELQlyBob82DnlumazBz/5OOkLTduU3KYc6PEcDjTq1x18pUevp6sZ4CIivY7SR/hSjYO0WVu4iSbDvHXwRjIxdaJh1mlgloA3OV02IBQSsvcELl2SHBqJoHl9Uu4rk98fUK2Da1jFASngK4bnRCMTsxYp89QSdkafYg4VEj7ysZr9iFEK9jR2nu/bwcPXCcBE8DEJlIRCjcxd1AyQkR39jrlvUUTAJ1mqr80IZmTvEIrcM8mgGpmYO0pWuegXag3Zu31HZE5G7RygBB/rI6WuQgBKMEVVsypFtuybHLskheC+SgnR2vH887njKFuNRkG0w15qmjsvs7w0f59WZv+4EFLVR8HKDusoZ3V+3z+ZvY+5X1JKgrHcXZkDb1I4D0+Q2N+f/5/8kgD4c05j1Yr/HyzlqGUBOEdf5f0uyvEA0nMKSemqexsIUl4bgeDfksDm8J6Y60ys3VeShMDt3PdO/mm8kl+rHfmFB5bNw/gvfPoNAMvf/fkAlt9oj2WM8e8B+Htfu/snAP75P+axHYB/40/z/BIRf+uD38b/ev0JNgcl+s4whXlucXa0xmu/gG0EinnPxNDHDq5VQGIOhAkIpwykiUEAXsL2CtIWsEcBzYMNhh/OEUxE+46DnlkoGejdvJwBg4Q/8BC9RKw8hI5QbwqERx38RkNuFcpbifYdC2Elpu+syMrpgO6dADHQL9nLCLlVBH9HA8SrEuHEAm8K+KMI6AjZCQyftIh3BbflGBrUnfs0LCvYIwdzrdE+dFA7CfV4B/+sgZ9zpToaskK+YUhNVIol9UHwMU6yQuPIwvYKei3hUwhR9qOFuUOsHXyvUE4G7IYpLr7zBq+/fw4/9ZjMO/TTAlFHtO9bzA93aCuH/qZC+UahO3cobhS6T3vo5yVCEVMQSsIIHvDnA7aygJ95mNvUBVVHVJ+uYL+3wHDsUdxIDMcBzVMFN+XQYWcRsQwwGw3fRPQnyWtYkmlhDUUauFXEg++8weU/PsdwzNdQ2sy+KNgDfpkHk/ysa8kFi4se4g9rejkHxv+XlxLdhUeceIgt5aVwTNp0MwJxOwsM3DmKsEfsHVQtB1w/8XANF0TMRgAnPXYogSgxHFJKORxkgzsQGo/iUmM4c0A07Bh0ZFDkQDDUntFj2l04eoO/MugecqARnufCTbi4ANAnq1oBvaMXrH3kUL3WHExnEe7DDvF1Sd/qlOevuBXYPvGIU4fieYHqRmD1icP0pxqbD1zylAqEW/bqqY7A398xgbU9jbBPeky+V8HOeZ7chINudSmwfRygNzzvEAmY9wLdA4/pTxVrABYBkxfsWts99EAKN4pp0SFLM6MUGI7osZVWYPeug1orzL4UaM/5BVeDUjs7Td7KhwFh4dD8qICRTEKdPhXYPSJ48EXE5DnDN7bvpK7OCCAycMZXBJu7c3ogEdk36KsIXwfUrxSKFfsXczhRVPSuTl7Qqxo0ayGioowzyojZTxVZyVR0n32P3TGlkyJwsSUDXb1DSgol0CxvgfaMCz9Ztqo6Mh3dsUheWoHN44jDHwCbd8la2Wliv1ZcmGDyKhcbQvLcqpYsj50CvmV3pVnzPJS3YEk7+HfVdUR3SuBULpmC2Z4JyBfA6mNg/mMuIvVHZFtVL1C/oaw2e1Kbl+xq3D0MOPgsSSA3BB3FHRcJhrkYAfZwoEdArjru3+EPgduPNMHnQqC6AnqzX1zwpRglr5Tn8bW0My522DkX83y9r16pL+l/9gWwfZzO7xsyOFHTEzrMBXQH7M4FZs8Ctg8lqqsIbwW2j9JCQsTYW9mdCOw8F0LpBxaYPYujdzEYAZfSRLEEuiMG6vQLgkX2eUb2apYCuwuB+g1fc5kW7ShJFegXClHTR1suKetsT+To250+D7CNQL/gd2l1zZqM7phhWoef0f8pbVqAUvR2FusIUQhsH0hU12RBKcfVo6eWQUkyLTZG7M4V6isBVwmUdwEu+RnX70gcfJG84ZJ+z36hMEwFpi+5gDDOCol9ckJAOol+TnlutRQYJhK6J9M/TAxcA0xee6ze1TDrPWAu7wJ8KdBcAn6hobuA7YWEdGVSC3DxqFtImDZCdwzQiTIvqIgEnimrdZVAtSTALZce68ca8y+4yLZ+RzN0SwLTr1w6H5rAOEa4SqE91aivHdpjjeaNg01JrTqBcDtJgW+lxPZMY/KKjynWAXZCAK/6gPZEY/oVga2dSqg+QgQNX0o0r3sMM4NhJlHdenY/FgrdaQFp94sXeushB4/hkB7CDFSDEsDCQKYE2qglgpLoTgpUl0B/ZFBeD0mCy/tVH1De9OiPS+hWQ68HhELBzioUywH9UYliTVCt1xH2sIBuPaSPsEcVFyK6AL3LYTOUUcoEZMmmRbgFg4jcxKSkYIHqMqZFB7UP7skLGtYjNiUBtwtkHq9bwAChKSA3A2LNQJtYsBIkVBr6rkWo2c8sB8fHrtpx3+KkJuirSoiup8/SKwbx5HRVrYDCEAje92fe8z7GqmAQTowEsgm4xsLQb2odRFXSr2lTQms/JN+j2Ke/JgAopASU4rpXrg8RcvRlxs5DaC7IoO8JIpVMYUKcLzMLGWOEKMweNAP8ovrTTPv3SbD/D8DyL2/f/O0bZSy/6Vv5waN48R//+5S3tWTMoopQrYSfekr8Di2w0iiW7EfUS/pq7JSsnE/Apn88oHhhMJwTnIUiIpgIcyf3IScpfdFPKBUbe/U0WOTeSugN00MZt5+izC2Lzn3JAVOvyVAAZAldE2HWBJkAhxM757BZ3nDAHBJrGlKirRr4hcrQmgSIIoeD/piBNv1hQP2GYEbvMmNCNgFAGo73bKFrIiV1IbEbDbc5HBCABEOQETWP21cEbmbL5wkG6E896q8oR1IDpWChZAWG3gK5bLx912L2QzOGghDUcdjrjzig+iqm4Wb/c3nNYVj2HOJzuEVmPzNTWtyRqcqDPg+YbCgiYFYEotV1SmpVBK9mwyHLNRwileXPxV16TMWB2E7EGPCT2U9Gr1OOpjcC1aXA5gklX67aM0x5H6XlPnnD7WTPVg7ncQ33T7UJIB3w9cv75yYshO9OOOBGtU+GzIEoamDVwO6CFQvSIg3eSImbcexqY8cft+ELjMXsrk6s3cAFhvqKA2GxSq/lGRlWkfxTxTJ5DNOCRLFiOmpOtLzPLtjEokIQMGR5kt6l66FP6a+p7sI13OccyR8lX2M5JGYtBYxExet9DN6o+VqVqdYgM5blbRz9cyKSjfVlGrAXfN3z+4whNfvfM3GTAMJOBSYvA9oTngezSYzjQADB93JEf5ykq0NiXkRivA0ZtJjDYBKbo3qynuUNX8PijoxWDlgSMYUZ1QStsy/INGY2ea8SIFMUDNk+gOcPYC1Gf8B9FQ7YPhKYPudnl+6AYUoG1U7EGFQjBzLjxZp/42qyhwCPfXe2Pw+8Dvj8Od0zaMC0ZBTz+1P1e9ZumAk0l6wrcA0BZt7XnPTJahKkVE4yREFzOzlspz8g409JG59DJmYv9z/yGtsfIzvwMMoEiw0TXm2T2MBuL6vUOz6GbCzvlwNQbJm+bSf8O5k+p/J+NFeUUhZbht0Ua4/+QBE8rXgd6R1Gxg+R51gEMp624bWSqzykTaqDmvvvi3S+O7KuwfDcqWGflsqFBz5WRKTwq5gYZp6jzIbqLl0zNVkt3Qb2lU4kq0ZyCX1igYu1h2skbCPHgJ9cdcJrT6Wk0whXkp22DV8z3YWU4hvgC5mCxPJ+pZ9jRH+oUaz9GAwkXJLWScBsHVytx0oSgjwJvfP0BKbvdJ/CjeiNlFBdHN+DwkfYqUK5dARIIHg0a/ozyzseo+oDZGIVdevhKsVrTInxWvMVgVyUZJeGuWLQUEtwF7XAMNcwGw9fkoUMBdnF4mbAcFQAkZ5Dmeo1fMkuTjXkEJX9bCQtHyNcTHOGHNnmfc8juxiHRYFiOZBVTHVFwsfUJ+kQi1QtIgSTUH1AqFn5QbZPQAyBwG1ewBv+nT0oYO4GslQxQg4ewnq4RUUAvB4wLEoU1x38xED1fG18rWFuOoTGMMwnRnomQR9iNInpk3JMf4YLECEglJrhOW3uapT8XSRDG2ozehyjURB96qgsNNnMzQBIICoF2SXWMTGHoTRj4I/Y9WQCMzuXAnogJYR1CPMGoh0QSw3hAv2L5l5QTv67SEZU9GR1gXtgMYfsFIYMpybYjEaPjK7o9swotOJ2QnjbNykZZIR+eNvDKCWZS+fJCOYeypgSZfW9IKEYeB6APVOb9zGxofdDeLK/Mm+PdSNf80/mHsyEQcYey591u8dwRjv87Mf9U24/L4zlb37yd/7Mn/e3/p//7M/92IAR3vyC3qLAX/+lH6I52iEWTGLVK4V41qM8TB08ih4r10TIiYWfBNgJwVGUDLboTx1U6WEP2eOnN4Jg9WgYB8jh2CMUEe7Uon64QTSJ6StBGd1KJdlrhD/v4RoOZvUlGYDiTkA8Ygx11GRaupMAO43w01SKXALDUSCrdeARSvqRXB1RvxEYFoGPOXNkm7Z8rlAQyI7Mn+fgFk8G+o+S7DGk2oXujM8Tk4wwFBHdeUhVCgSH/SGZCeGAOoVn5Fj00HgILxDPe0AC62/ZlITI1wAiDaQHESfffYN4NIxSue1jyllELzHM+RiABedZFmtnHCJ8RY+Q3hEw2yN+WfgyoroGhoMANaQwmgLwTRzTTH3J/VEdBz47i2l4p3+oP4oov31H2ecioj8JEIEAK6rUldcSNA2LNAzNCIijFKyySOmkZC8iuoeOniibKlQ8WYVhzuNSQz6+QO/VlACuOwsj2IcA2gufVoF5fUkb0Z3ymu1OIlbf8mPSbCj4fBnIiZQOmisLutOI21/isNt/0I9dlDKxefnWnUR0x3FcROzOYmK7uPCxexjQL/icOWREDhHDPMIfePiKIKI9Z7Lj+iOH3UWEm3Hw1TsOja7hgNm85jG37w+JDWE9RygJ/qubgGGO8X3hav5O74D1E/AaCtzP8oayvO3jMEoukQbnKJnQatYR/VEY5YzdeUB7mq6zktfa9EXg4sghEnMYcPdtB91GeqoWBDLDnOfLptdVDmQAlx+TrWlPKUntFwRUriFgHxYESN1RxPYdAtr6OmL9AbB5Qrljv2ACr3BIPYa8htszgfahx/Kv2hEgRZXBYySAS9ePiMDqQ4Ln3TkHRb1LCwgTPn73QMBN2WW6ecTrrL4OgOTCg+4iNo/Zk9gf8noxm4jdAwKpfsHXk6Ev/BxafchBPYN04YHtQzKdwwEBUU739BXBB1lQsrvKclFEdwH9MRJYJGjvTsjGiQCs35UpcZgskp0KLD9JizpnYlxMaF67BNCB3RkDavgZTfYsSqA/Emiu3Phe6E7yIgWBWXvG8BRXcx9dzc9YBD5mmAnYGRf/XJWkxDMCLxEi+gPJ1+6MUl5Xk0ENisBceGB3Tsnl7kKmeplAEB+4iJH3e5gJtCcC2wuJ7TsCuzOBYhWSNJl9n3ZK4FdsWHQflRh7QNtjgfZQobhjPcX2nCeqO5YYpgKuFOjnEpsL+tm6hUS3IIOlU6LsMCfTqds4LozsThUXXFLi7OaRhBwChqlEd0Qpit6RJUMAohBoj/h6bx4qbB8S7Lg6JeemLknh2P9oG5kYPj5vlICbKOzO+HwyLbQQNArszlKYi+S28udB/g6TLsJVEu2JGjs/gxEjuNyeK8ghYP2Ohm0ENg8MNo80QimwveBwrLsA3Tq0RxK+yAM5ry07U+gPmFLK1FmBYsnvLtZ5sEO0PeL2tw/L8XwhMnRnd2bgC4H2WI9S5SwFVtsB3kgmzfqI9tQkm4FEd6zTQkRkl6Pjdbi7MAhGjuB3d1FQltw72KkaE2bbEwM70xgWBt2JQSwkXKXQL1j5wa7KgGFm4BoNXyp0RwV8oyFixDA38EkKHCUwLAq4WjG1tnOIhYZaDegXfMMxBdfDTfnv4aAYQfBwYOAbTVlr52DnBtASdlFC+IjhoIDwAXZWMAyno8TVzguESjPBdUjyWCEgdl1ahAgQveUCg1JwsxJ2XsBN+PwIlBnDBwLOUo+g8n7yqp9XBKVGjxLeWBnEukRM92f5aphUlOgCTJtNEtsRRKZEWHFf6mod4APvs24PXoWgrNZl8CYTS7qvVSF7mICc0dyX/O+y2PslY9wDyRgob00AUkhKYdM/uB2VGEul+DitGc7jXHpMel6ZpMWjFDbuJcT5lhJjRWI27/8shPij/6m0Pz+r5/Ivbz83t19oxnL+6Xn8+L/+OxicwmZTIbQaovRQOiAEAaUC7M5AtJR8ypkFLkvE4wFxy7hqMXHAnQFS15JwYux3NGvK56KOrAZpPAMvwMeFAwd9ZUYWMZaR6ao6QiXpJABEQxbVnVjoNwb+YoC8Mim1kbIH7KIAACAASURBVL8PBR8TVWRtSSD7qteU9tkzi+IrA1/HcfuZoRWODKhvAvRKwi489FqNMlMEIFRkX4OJY0dbBOglSN7GUIbkBQWK29Qhabkd2bNjz9fcnkxFyHor0B/x74Qnm6u3klLRTiAUEcXjLfznUz4mEhjaAw+9UWR2kzQVSKC43DOWuYg+lPy93gj4Mnk/IxJY2TOtvtqzqWp4m5nOPtmoyPgi7I8fCRgTTIlUcJ+Z1H0YSZb+BLM/t9JTPikt9ykYLiS4BrBzVoUAGP0vQRMEmzuR5HcxHUvyrqYaj/kXAcuPJYoVWbsMHLNktj/keeJ52w/z9+sJMsDMrFtm3TPbm/1do9fVJ2YyefEyc9kfkj1lUiOH6AziMnOb6zhGqWGfmMu4Z4NHlj0xPlGl7ZRIcrK93y/vbwYjjOFHArEE+gAXQzJzT9lZ+n/yvgrPY8sDNtlOglTp9n83Hk8aQvO+IzOWEiiWEcOCMszqcn9+hwOguowM3Uj+w2JFttGskw9OEIxzX/geQromIMiI2xnPR+4jBdLxCLKuMi1OCJevBYweR6aG7v/91i1vO12HvuBnRgapZD5Tx2HN4/SlGL2a2TdoJwwEyd5IAKncfM+Mmk0cwzDoy437Y7H7a1Q6AiE1xL1KInXMZk8zPxu4mFHexlF2mz2J+ToRnuxXZirvh3PkDkBf7N8Tuk8sbZcWkgp6O/M1yMUQjB7D/L6Vae4bPbJpoTAz1Hs/axxDQnzKzshKA91iZFVVh1GGCXAhQsR7XuSkZMi1GJmdBfI5vOc1TSysTn5HaRO7LgjWIRJYHeXIZNTyNZHPVb4e8gDtysRmB27D1YlZ7Alm6aHeX3OuEmM3YndAmWn2nFJFkRhkyT7PYSb/CCMe0+/lyLjxGDI7mbclPOhND0Du3cznKPtN2aXJ57p/XYyf50gLtJrbd3WqQekj+gM1MppR8T3EzkskRlqiWHsEReZO2ji+L1y196Le94fqHRneUMjx85fVGfR8Zn9oMKwVya+9r/K++PFzydWcD4QL6fM/wlcMkMm1Gvl142d1HFNc9c4jFDKxqQ7CMtwmpuske9R5/iXUzhEcSCAoCdU6+EZznz2vr2Dk6IENRqbZwafnI8iG5Ode9mRK+zX/XLq+pSXoyhU6/N6Ko084358rSuI9wCHu+/d8JMtpA72Yjox49mgKl5lXSlyzXzJKSZ9lYj3DtIDckSEUjt7IqBh4FPPz5l7G5HuMWpLdTDJR+DC+r6LRENkTeX8OF+ItietbXkrvR0A2/t39ipH7VR5SJFaS3sf7VSNv+Tmde2v7fywmuH8+s99R0V8pjEG0dnxMdG4v1VVq74f8GkOZn1ckJjcOw77+5Gfc8t/Gvv8TH/cn3X5uGMuP//af+fP+1vf+8z/3YwO+YY/lN30bnEbvFDZ3NczzMg2thr2CHYf+ItDf4ssIXCum5N1UyZMWoa7UKIkya4HyDhhmSFUPgGrVyPCojjH39KoA4bZI0h2RegU5uHTHqd8vyWTJOgFuW7DjbiggXCqFH5BWpzGunlLudD9MBRhWBcMrXjM6vrrkh6drJPQuDXKCx6J6jeoS6E6yzwgQt5T+ISaZpAZkCreJgv41O5Wj9FY4oFjJke2QjqEloU3DUgmU1/vQmfnnTNob5or9fXORAnu4yjt5mUC0BOrXgPD7JDaA++Um9B65hkOknaZBqaWcUPUZJOy/wPuFQP2GMjb2donRV8TQFcG+vSJLfCgFZL8fZaKuYfiHCNyf5nUaUId9sAywD6gISYIY1F6WmIF1MAmYBnrCyiXlVW7CY8pyTfrhIq+fltH8UfMYihX31RuB6dMIX/OxHHg4zOmOw1LzmmzaMOUQw7ASDk55W5mZrq89hplEFLxe2zOB6TMyEbsLerpCIVDeBnRHTCHMQxi7OZlQWdyl1MFGjDLIfP7lhEDKTjNTkYrLXwK7Cw7r1Q0TEvP1UGwY1pJ9iNJR6uoq7mcGQcWKQ0V/KFFfBxaMV2K/OACBYsNrVXj+Lgcb1dcB23OFyRsOgcOUf6c6QN7s5YCUv/G1nz0P2DxUKO7Y3TdM5ehpZK8lj7m682iPOE2qqzgCWzXE1KUXoS/pSTNtRD+TqG7CCLr6OQeFYIDyjoNxXiTIUjqAMkQIBt1ksA3sh2SbPofKZUS5dOgXmr2Cmv60fi5Tfx/I9E04YFZ3YZROF4rvLbMN6A/SCnsC60xXJIM1TIFyHeHNPv1St3EMfvGGYGdIni5fJuYoMiiHFQABw1wlgBbTZ7pEdePH91yU7EjULT10IuxltcXaY5ip1DcM6DZAt2TpXCnQvHYEUpVMckiged2jXxjK3iyv4xwsNYKd65ACUQggshQ1pw0LH1lsn/4NcNt+RzCju5AWBrlIhzWHY9UFVkE4slLlbfrsS0BauAA7pwfaNgRbqg9QhYArWVxf3Dn0h3rcf2njmPJoNg79odnLRl1EseY5D4WE3jnuX5I7Zi9a3r7q6IvzFT/35RDG8BxpU8CWEjC7HHKTJI9TDd1lcEVfp7QRw0Jj+pJDpeo8kzYz6FApaTMCxV3yfgcBvfVjiB3lsH6skpA+vgUYhGMoTmbrc/AMEzqHMUgmFJLbSsAiKJkYygC1s2PHYVSC9RWGrB7lnwzPyeE9akvJn50aFMseUdaUiwqwU3HH30cloSoF1RKMhYJsKxJgE17CrAb4htei3lrKX41kMI2S6fojYAmF4j4lRYLsLEJlUAyBdRNFkor2DrIpoHYDmbHBMUVUCYRSU446OPhJAbXpEUsDveb7T25Zs5GBodwN8LOSoDV1N8IF+gRrw0AcANKFMfRGGgWZZKKh0nvQlo4DPjLApi6AECBzhYZRkK1FqHQCnT6luBrIzDwKQeAWI5k7o8eEVAAQUkIuN/CnB5Ct3ctCpUQsNeS6Has6RAZjWZ46WHoKMwBLHY5ZeooYoRIIFC4F/ihJuWyMBNZajX5J0XP/sN1BVCUZx8ze3QOJos8+0HvgOgPI/Pj790kJBMuOSa3HZNbMUsYkSc39k3xjJCBr07YSIxj7gZJZ5/ZA9D6AzP/OgD2nueZ/+8DU2syuplschr1fMwHYEUzm47q/nXu//2eVwv7l7ef79gvNWJYfvBMv/qN/D6JTkIcDwrKAuZGwi4BYe5hLw+j8VsLXAbKTZB8HMa7GZW8hUhdb9mGKSJaw+rIYZYPDQRjZGYKBFDaSnicq+iNdzf6sKMlcdSd8TC6YN3f80rCzwMRWRY+laxKwuGMEuhxAJmLCgT6H1kQFCLcP0Mir5L6MiGbvSQyKz+Wm6U0rMHp9fMk4+5xwmGW8+aZbyjn1ToysjK8oI63eyFFWarZJPqo5bA8H9Hdm3xc9r9wHaQlQgkll5G/2cfqjXDczaiEl167FeA6yPDTvT2Y5GJqx9/UREN5nHAkg7BTIyaZRYiyKz/UFOfBFuiSjzcm4ljK2KOiPLK+T/E/t9zWzlpnhyx5K12D0ImYmxRdcOKCki/uWPaJy4H5mpuM+w1Os997PvL3yRnAhY8N9zKA3J/1m9jCDOLPF2Is3zJM8F/sYf66kJ8ax5XO4av+eo0czgUrP/VHpOgV4Tsw6jnH5oz+1oJ8yH7fZ7sGn7O+xchX3r7jj65qZq8xw5pTWKDGu2o/sQ+B+Z49Ufm6mIzP11adk2Mx2ZyYqpN7qvHhRXzFMBdgvwASNlNwqkFlDZcnyjezdkAbg9JkhXV4s2vs+IfgaT58H9Idy7B7M11KW32dfp6vFeDz5ODLgH8+F4GuXQWnu1PNlui95Le8ntLqai2f1mzgufuXH588Ula5HBi9x/82Or51KiwkMxBLjceeFJ9Xzd4j33itxz/yJ5EmMWqB5Q+DtiwRUK4FyFTFMCUhzsmrenur3DKQvuKil2ojqjgsI1S0XTELaflZYIDGv2YOag3G4UMQkarNFYgATs2yR/GlcQMmJsvR2czEnB+Lc93Hm63WYksGL6Vq1k5TGCoz2AhGQPIYEu8OUTF9eIDG7AFenhFqL8byoIY6LMSolmWYJeN5+9nDvfdc8XlcmyeyBHL8XpON+6DYtZO04COf3Z/acZgA9zBQXUBKj6CoJZeMofeXxpeTeQoyLClRURLRHCsWWADYzmK6WI1OZP1PGMvpM5ERgmNHfyQVFgt9g7rGiEW95De8n4mZ2KxTJM7rj4kdWFNgJfYXFZv/8ZuXgJgp6S1CN9HmpWwLGzDT6SkJv/diBqDoPOzMjM0c7AcN/VEt5vhoCXE3wSGuFHhdL6IOVKFYWvlL7z9p0XMGIPUPuI5QNI+gLJWW9vBh43KM3MoO9BEyEj/skWJPDcQaEkl2V0geIIRA4a8HkWCUhB54P4QLcrOCiautGz6CvFMxdz237kB5XQnVuZCxDoXC/6kL0fkx/Zcc1wZ1sHe9vHRcUXWDdxmag5FVLhEJDdo6Jr0mVlKXQ2T8pMpCLETHLWCUTbEXvWO0x3AuIyeBMMAQHWo0eTdFbhuVYRzB5vw8y+yit2wNAKSlvzWBKq5HxfQtg3mcWs8R0sHtvZ2G4rXxzjr7IGPmzMUxmzbf0u9j3EGYvmx0xwB8H3ILn3zm3B6qZgdR6n1J777Z/vjCC4eg95a73QWa+jWE+El9Pf/1Zt1/8VNgH8Tc/+gYYy+//F3/uxwb8gjOWohMoXhqoQUD/pB5DPPxSwU4Ups8jomAnlKsVQxzALzjdshJBb+iJUR0/8JuXagyRGG4KJi5qDpKTF5QB5doIBp8kdiL5oJpX9M71C2D6nMmRUZC1CKmEl6vAgpUhbeTqZipWzuEovhSor3KIA1m2XafQvGI6YHXFAWTP6u1lZUGlLzkjUF2zA0u4/cDNwJm4j8HvCQSCEalDDAAimlcZGHB44hcggEiJX3XN4+qOmQTp6ojqSkIO9Ce6hoNqdwxMv8wv2n6oozcssSeNGGVqOeSout6zijlNsVzyCzCzvHZCpi0P2L7ikNoeUdKsU4UDIlMpx1qGWrAKYskEzGIZx8G6WOfaBwY5FGsyMKEA4g2Pf/Iyoj0lG6n6VNGwBNyEw3d9xcCP6Vdk41SfQJ5OpfRTDpHFHQDJnreggX4u+PoXqZA9MVp5GM7XqRwYklPeBVQ3/MLK5eHVDYFBrmjIDGeWy3pDYFxdA+WK++dLnls7BcQ2XyM8V1GQ0c3spxoiytXbrGCUHMpFJGgFuL/FHYFClEB7KuELYPaMvh/VcfDMMrkRKKbhuU/VFhmkjAsCBqhuQ2I0+RqbbaBnKQ3bOZRDuvQ8lgOuSGw1h+o4sl057ISJwfwCPfzM4vYjg+lLDxEjuoXaS93Sd6Oyke/TI763dc8h27SJ1S3JbBbrgGEuR8mt2ebHJL+skOyo6wNcKfcslk8+u2Yf9lRswjioMyWSr5OtOdTz/PE16We8T/XsHZQ9X6Mc7KLb3DOXFz14YGZLcFPfOOheQfgI096r1UgALTNJ3pCtk47HDfD5hmkKd0kVDq6WKRQr7mWOel9towageW3RHRsIH9G8oVRSDfvt9QcynQO+N+1UwlreJ3zE9KVPCzEBcmApvTfsJsxgkPsXIZ0ieEBi8KxAeetSOIoclQFqiCOgyox4VIDsI5rXHLaHmYLZ+FGOGAxrN8w2AcIuwM4Vv69aMoVMQFUMw3EMeCFI5zWm+gCzJahSQ0SxchgONMpVko0OYQxokTZCawK3ESQByFbqzGzmsBiA13OxDvu/33mYieK5SAypHFhT4RoF6Xm9FHdkKqsretWKO4bjFM6PNR5xrlB/1cHODIrWwc40VMsaC35nEXwFldKUEWHWFroVaZGIoMU3muDDkOVzE0OWLGhWf/R8TGZFfa1GaWfUEtGSHVWtHestQpk8XTvWdJgVw2tClWpHVpY1HK0fr2W9c+OCgbKssnCNgl5bgrQQIY2E7rhP6nYHcTSBiBHlbc/3jgBCqVFfDdw/KSAiq0RMqkIRMf8/yV+HANV5IETKUUGZqJsV0Btumymckuelp9cQQrCaRAgymj5CANB3HdxBRcY7gbvM5noloDY9/KyCsjyHKjou/g17EJPZSNHzPpUYRbPskFNX1dAzSKeTEL2HDIFASAioJBmVPkJ4z/3OktIsT5UEmCJGMpYAfxfCKGcV7UAvoHWABYQUo5xV9G7PkKkE8DrHug3rxkCcDD5lTwmn8AGyE3vQJgRE25OpKwuymF5BoOD9zqck1ECQuOsw1oQAlLy2ffJCRsSSybG5H1I4vwduSu0BnE3BQpmdDIE/O8fk1r6HKMu32c17clORwNdYK9IP3I6SZC/vs6F/3M8ZaOZgnq8xm7G/x0y6xMgGdkzm+wGMQT4jnPw6gL0fGpR/Hv847AHnnwJ4/tzfIvbn+S/g7RcaWFbzHr/0L32O17sZZkWPzhkoGVBri6np8XuvH+BvPPlDfLY6w49enuH0ZIntUGB5O4HSAaZwcDKibwv8+pMv8dtfPsFvvPcFamXx/esHCFbDRYHtroTWHo+Pl+idxlmzxsvtHDfrCQrjIEXE+0dX+N6Lh0DpcFB32CynWP5VwBiPYVAIxuPiYI0QBZ5dHqIoHCbVgG1X4GS2xVdXC9RNDwlguSsxn+0QtMd21UCpiJ1VmE463LUFZpMOV1czHB5vsLydwFQOReEwDBrOKkymHUKQkDLgzaaC0lxunjQ91tsK0QsIFRFuSsTaQ5aei2VOYPvLPLfZpxqsRNFYBM83fARwMNsB1mAx28AHiTdfHWGtIoSM+ODBFX784hRCRMQgcH52h/X1HHcfAKFlEMH0ZIvKONx8foSzTy/x6uUhNxqAYjbAe4mqsjDKY7eusUkF6+89ucSLqwWOFxvcvjmANAFKBYTCwVoF22tUzYDlbQ250hAXLWIQCD0j+k1tUdUDus7Arkoszte4XdUoa4vltoA0AX5VADoAqYYFVsIc9HBXFWIZMDvbYPNsjlgFiCFCH3WwqwKiDNCVhb0r8ejJNS7vprCXNSaP1tg+nwFBIFYeZs7ji14gBgFVBPheAYMETAAGCdE4mMpByoh+ZxAHBTFI6CPSunZdwMwGzKYt3vzwGOrdLYZlCZjI5xGRX8omQKiAYBVEq3D47i1uLueQK41QBMCw5kZYwM/SAOUFYsnqHIiI2CZ6WID+4SpAXxm4uYdeKbgTi8XxBsuXc/595RE7BTmxCFtW4kQdoKYOeMV6mc3HAXKrEKYO5aKDfT4hu3ngARkhSg95WcAfD5B3BmHmuM+Nh55Z+NsScicRqoBoAsxSMZiiDGTIdwxg8NMA4VlWHqoIcTBAPavgmojYOIhWQm8lQ6GigOwk/dVFYkYuHMJSYvVphFmq0bvsJ1QahNpDLzUgBNzCwdxo2BMHWInqpUZ/zDAhnPYIWwNzI2AfDBA7jVg6mCszMokcOAG9VVQmXKvUmSthD5JOPgKxDKiem8SqJLa/5z67SUjp0gJuFvh6ycCqnBRaFEp29brsCQbTr2MdUD0zcE2EOx9QPC+SrNagP4wobyW2n/aQdwbRRMguqQ90hEl+7lxfozoJO49M3B4AERV8GVHeqFF9EKVA80pg/ZGHGBg8lj24179cwNcR5TVlsv1RhJ94yE6ivJbwTcS2UxgWAeWVobT8JELv1Lh4ExMrv3sU6C0vI/RGwk25j8OcqpJhEVBeGtgDsq320KN8UwAijh74zJj6KilKUkJ2XlC0KaDKbFgFEFRMiz3AMFdwDZUflNVyYaK6VuhOIibPqWipXyl0J2RTymWqg9kI6FaNHcdUzUhUVzL5WSOKO0UW3/H6YZcyPchZGdGe0W/OjmMgd4jaBijv+Pz5GIqlGZUGXEziYk1xR8tDVi8Ua0OZ8kyzX3hNub+r+NzVlUJ/KHDzrQblLRCloZphQ0DXnQiozozhdNPnZNBdbVDdpM7YFRfLXCNGpYDwBT27BuiOBKrUTZ27ggEu7pmNGrta+0OR9r8alTG+wj45WgFmU8AXPDdRiLETVngzJjRLr2GnrOXpD2gNgAT0kUJOOs7+bbMFgMn4/PcXifJCVk7xBrg40x1J7mfBYxCJlJm88didSJgdF0XYaZoXlgqyxX1mhbm4lRehhOeia3coUWy4AFRsqlH1o7cBu3M9VqkwCbqE8EB7JDF96dAdMX1Wd1ywUi0rSqqlT8dBMGI2AdsLMq31bYCtxbhwV6wD9JaLCr4SsDXrS/oDherWwU4U1JBk+7VEfWnRHWmy35u9FzT3mWb2WdqQJMw1F5GmepTy6s4zBTdS1p0lxTJ3c0pKnwHATvToaRWewUdmZUcGF3GKUKr9dgcPX2vodTFKqfUmXYRNgVBxAcPXGrLzEHG6T331TNDNIDBUBnLHxQBxn7FM/aSh0Kw6MQqyt/ClIRsLALlrU4o9KMyy3MyUGk2mdduSPcw9mvKel/NrvkcOgfItkBmHJAVObOfY7dr3ZCPT44WeEIzGgGgdpFKUxxaUBUX/9jbzv4XQ3Of7wDMDzfv//4sCLv8C336hpbDNxw/i+d/9DxCPLMrJgOPZFq+uDyBEhHcSi8UWqx8dws89ZqcbrF/NqN9vOMAeHm5w8/IAcALCSqhOQH+wgRARw6Ahn9bw73QoKwv7+Yyy1vOA8kqiPwkIZVrp7SXC1AGevi25VggTD1EGNPMO29cToAwQa43m8Rrb6wZipxDLALXmMOSbiFAFmFu1/4I6TMP73ENOLXBFH2ksA8qjFv22AFrF8J4oUL2RaD8YIO80YsHuxSgZqAMv2LH2qIP8qoJvwuhNgAdrQ9ZiDJLRO4H2XYvioMdwUwEyQu6Sj6ynnLVYSrSPHBbf11h9HKDXIslROfDqHWXB5VKgP0qMmQP0d1ZwP5gzNKPhAMUEVwGXh/aph7nioC53CqEKOPodhZtfdzCXmmFGLw2GBxb62jCspxVwhw7CSpRXXIkcjuh30rlL8ir1hQZ+cXdnHnonWfEyZfru9okHvEBxx+MpbpnIqbdkfocjj/qlRn9ELy+DlwSE5yrdcOJRveTgm6XEahAYPmqhnleYPBXYPIkjQ1su+XeuArrHFvUXhpLbGw4K0XBw6s5YY1LeCLQX7JHsjxkOJCIraULJ62j+mcb2UcD0qcTuIYf88oYVBv0hQ5hUK96SR+odA4Gi5mPthInFky/I9veH/L3eCuyeOMhWon5FL+bugYCdRNSXlObOf5L8r5dM1rSzOEoX8zblQDl2exYxeSawfUyZulmzsmXyXKI7Jqvm6jyAkVXfXRAomTWBhwgC9auI1cf7QKOoWe2TZdzlrUhJvHyd3SSFsKQqob2sNMLOIqZP6V/ePuRrPHku0J0CesPUYwQmzqouSXC/Jj82mxRyFDmYs2oAY2G7tMD2scD8J6xvyXUzxYrbtPOIYpk1+8DsKZnz3P2YlRVZMl7dcIjsF2KsZ4mKwKK+5CA8zPmaF3dUOzSvU2rqhUB3EjD9UjIUKQKzpxHLT4CDH2P8u/pNQHcs6fO+wtjtKB0w/cpj+aFiIE5SHjQvI0IKeGlPCfjtDKiuqJKormOSIgv0h/Rtt+fA4kcBm3ckzAajjE317L00aw7MuqNX2awjik3E7lzCrHid7gNw9hJaEWNi85hK3J5IzJ96XP0VhdlPU69kS0Cht1SG1JcRZhfRHstR2aGTrDXL40dVQBruAb4GGcBNXvmUbMr7yiXPoU7e1OomYHch0zXCc10u+btyGRk01WFkeKNMPtg2YPmBTsnX7KcMqfrGVQLVbcCQwqRsIzB7QSCwek+ivOE5C5r1Jut3FMplYrYPKUNmPQjTYzl0R+ie3tbVY/o86XWN6FMnZ+5Q7Of0ejeXDuvHmn7xVwRI0hFouEqivnLwlUR7JNFcke20tRytCqrb14+IAAyTPXvsSznKh7OyIwcnZb80K1jkHwkpMlsOs+2xhtlxAag9kZi89Ey1fqhx8v0WvmSSKl9Tpt8yMZhVNrNnjrUqtRirfXRLdjoUAraRqK8dhrka1R7DTKK8CzAbss+qDeykvKQv2uwCVB/SNe+xfVBCtwHFyqE74qKSWXuUNz22jxuYlYOv1ciK+5I1JcJHJrRKYPLTFbbvz8c6E73z3HYfmQib9kN6hgMJS1DElFrKWn1NJjtKkMlOMmPZe9h5AV9ImG2qZpH0x7qKDJxZWwxzA9VTQdB81aI/rlAs+9Gv2p3VqC47Xj+HJYo7BuXYmWHX583Az5rBozurUb9u4TPzHOLIQutlj1BpekEL1o9AYmSmQ6nRnRSYfLFhmqxlDQoAuIMKatMjVIYMsCXDKqyHmxbp3HgC1M7uPaSGoUdy1yNM3/alhkJBbQegHxCbkum1TQmxaZnQOliEgwZyueW/AeyDdfwowx09mdkX6j09nUlCKzYt4pTx8mLTkpk0mpUkMfLvQhi9kZCJVU0e0vGWAV/HgBwh5Vu/jznsJ0tk71WSROsgjCaj6vceVwiBONj0u5yE+zOwxz8DcPyFl8LWD+JvfvDv/Jk/72/94L/8cz824BccWE4+eRD/5t//1/Dl3SGsV1AiYrOtUFYW27sKh8cbzKse06LHHzy7wHTWYb2qIWQkW2k137OvSrz/qy/w4xenQBA4P1/i8genOPjkBiFItL1Bf1dBmEAW7mKJ188PoSaO6oOBjJNaK4SzngOFDghOop702F1OuMOSoE109HCGqYPYahx+cIObFwuCsTsJ93AAthoffusrfP7ZA6gd01xjESE7ifrdNXbPp4hTD3VtAMHKB32r4euQqlAAFAFmOiB+MUn1KkCYeFQvzejnFBFkNyKgV/RO+qmH2qjRq6d39HciAPUrhd0TC3hB0Pv/svdmv5Jk+X3f55wTJ5bcbt6t6lbX0l09Pd0zw+EMOZRpSYYASzJkwNA/YMCG/Wz43a8DwwIkWa82YP8DfpZfKBqyAJs2ZRmkZI1mOEuv1bXX3XOL7Sx++J2IrOYCSvIYnCEYwMXdMmM5EZkZ3/Pd7vRUDwyY8gAAIABJREFUn+WEPPmyOmFDoo7YjaY98pRvhNHwuTAmQ4hONJEw86g2MUVlRNeJgUl9dpAkZ510WK6/7ok2UD2zUnUx95iNHhNnha2Q5FpfBWGieghVhFmPvsgleTYTAGXXeiyKD9lwQyL7VlxouoOInwf0TgswJqkgk1fT1AKas1pqT/JrTbeUJOHiwryV0JrA42TvtZKUU4VP5wIlvwMjkMpqhSslkbU+CxSXEqwxpKfGbC9f1E6NCbbiX9uX0w9+v0GKGTPxDBeXeqws0b2EXOUrPfpXh/TDbEtKuRWA1i5lrPIbAYamke1LOmtMycryXO2EOZEER+jmkuqrO/GsZhu5ZvLbYYxS8fyMMSF279kdvID7hE/U3hObNeIpG7oGo0nyYU8Kh9ozWs2xALfxOm8EIAC0x5HqlfoKG+ImjGm3rpTHD4Fdkua77wgVdkn6VoeU4+JKjt8Xwmh0c5EtN8dyU5rtGGXbQ6LukIIraZUylqbdg+DRu5g8hLpn7/d8i7UYGJIhFEiYQGGw7EY8dK6SyRZhNkRWPlSamI70+pYZ+24hIVduosZryhfJL9wLwHFTAWGjHzKdy5DGp1iJpD1fRzb3xWuq3P55w7keQZzZpwdLsFHyXqZ0Y+1IIV9yvBJQJCC7PVCpNkZk/f1EGLGsiWNX4/C6ApLvVapQysuQpOsiLy9uw+i97BbJawoUq0BzOFSGyPthu9RkW/m/L0iWiCTjT9e36eWa1cMEh4N8G2kTKCQBkpCJ3HzwKUOyMaRjFK+3gNtuIdJMuxGgpb34Mof30/E8veWPHTooVST1jSbfbmK9RMqsUiDV4A3ch0u5UsZeQpCSP7dSlLceVwi4E5Ztf00OXs1o1Jj8rL1I1ctrn1KGhSHLtzLu+cqzuyOMWXNgsLuA3YrfU/soYWmJLXvbtzwwe1G/5Qn2AoaHGpCh41P3kXapx6RgnTya/Uxky0MaLFEAp92GMcXWrh3dMnvrdanG0CEJaZI+0SGddfAAmjbQHgrYy9c9IdPJhqNE5jsRWWzIkwc1xK94Pk03gEogyjYkEXbwOkr6q7d6TEAVGX0gZFKbMkx6Df7QbJtu4JWiX2RpktaP0l+5rkWGmW2dyIcbn7yfCYReNzR3J9ityId154X9cxLGNAA1SWwdpPYSzKR7AWhDQqyfWtmH6xp3XMn6kkxaxUh229IdV9irJvkpUyDPsISYPKRGfJxlNoLH4ViyTbdP1B36Lyc5elUTS7sHcr0nlLk8pu6IpRVmMY1t1Bq92REn5dhjKV2ZDbGwY6LsCL66nqGfEiBW+Qh4B4Cpejf2Wo5gb/Bhvi1jffvvsAeLdpDeijx1BK1/DA6IXiT+KqXQxrf9nDDKalFaftYCfmMvnZUxRpTREiiUgHH0QVJgwx8Fj4Nkd/j5T1xC/OXvsfxzDiz1n/6QX9zFe82P/vcPuHq2pGksq8+XAirfTMmf5bigefVP3+HTf/KYsLaszmdkuSM0BueMSBDfFJx9+w2fvTjBflnw7v0LTidbvvfvfsz2XxxzNN3RXlQQILYac5Vx83un6I3BNwb7WQmd5v57F5Lgmnvmv19ifzaBoDg7WAugtIHDf55RHDbEQti9xb/KiXlA/c/HTO9s0YctdqPg1jL50vDZD+5TntSoew2PvvEa1QngcU4TbeS7X3uKul8LO3KRyQ3WosfeGMyyY/pJTl9b3GmPP3QUjzYsz9Y0jzrc0pF/sKI/7dG1ojg39Gc9+eM11fOMofvLvrfBTSUMKdto+I1bjn4/45vffEZ5qTi5sxIf4nduiEbqNdxRL/7RHRz+SFNcSeciCqrXmun7txTXEn6kakP1wlC+0eidFFf3B4GTX30j/YxFxM0C6psb1h/IB/nsY4v6tVvsRpGfG8pz8aH6IuInAbuSupPpU0P1YE3MYP4zQ9xmLD5W2LWA18UnBjeN7B73ZBvF/AudAnrkWNuTwNk/C0yemvGGPb9WRCtyun4hzJ/dKuxKUb0w+BKyrebRbwHf2GCatL0G2iPpQl3+VCRx1bkAweqVkh7SVtE87MQHF+CD/+kW3QpQ9jlj9crseRzBjs8jy48TY2Yi3bGnOwhMn0sqre6F6ewO0+RBLb/3MwF3u4eO5r2WyUthrI9/oGgfN7iJyNDq+453fqcGBDQd/yDSz4QZHQDr4Y8F/DRnjuXHnvbEs/w4cPxDT7YTYLC7F7n9bidsMDD5Sxf088jDf9yy+6Bj8iLJEzcwfR7hr10zfxLHG/1uuQ+TAbnOmtNAvhqOXZ5X3w3U7/W4UvpA3USAzfZRwK5g825g/aFj/mVg9qVMChTXkXwFu7NIfSYdo/mNYvGlXG/b+5Gb7/TMvwy0x/KBePxjT34r+7e9H1l+0pHtBCzY5D8tLxTNmaM8V9z/nTqVksPZP2tRHmbPIjffDJz+S8fxDx26ixz+1DF9GdjeF7/r5mEcqy3O/u9WmL4ZbB6IF7o7gNWvdjQnMjbbh3GcdJCQHAGuvpR+U7uOrD6S48pvJfl2ch7YnSm29yO+kr7E24+EZbz9dk91HmmOUspwSkqu7/sxNMlNhIU7/oMeN5M+y26Z2NlO9r28jmzvKzYPFKuvQ3kTef1XIpNzz/WHmoPPPOt3Rfa4+kA8kuvHsHmgmFx6mhPF7EXP7kz6LNtDNTKtbiIhM6v3FMVNYPVYgMvldyTFtL6jmL0M5JvI9kzjc8X0VWB3T4Dz5a/tr2US8NUerr+5D0rqJyLTLa8D9bEemcV8LYFB/UKxOzVjguzN1zUXvwbLjzsOf7pjcu4xrTxn9ryT9OgadvdE1rd+X9i94iZw+dfb5DVPaeOpDkW7SHklUsj5lx2LJx3be5KY2x7Jscyet9idnJjpq0B9R0Dm5p5m+tpJP+kDOQ/5rQC3g09rmbDoJHdgkFsGqzj4dEfUsHpXp+RpOP7BmqtfMaze0+xOBGDVR+LJLG495aUTJtFFjv5gw+5EOiclmVixO9Ep1Vhx/HsXwiLnivLSM/+yoTkU0Hf1rYzmUHP7viXfBJoDOXfXH1mqC5/CesTH2y4Nt1+TwB1fKlaPZXalm2naA423is0DTX0sE4TlZU950dNPFe1SVEK7kyylF0eaQ0N17pg+2TB9uqW4aOkWhuLGcf1hhukiN1+zFFct05cd/VRz8V1DfWyo7+QUl9LtHHJFthMgNkzytEuR7pcvN/RzQ8gU9Yl0G7YHInOuT+SNL9s6bj6wIs1cO7bv5KwfZHsAiXxWuIkmWM3kyQp720k3ZReoT3PqU0v5yRtJJt50rN7Lmfz0De1hRruUXsnmKKO8aMjWPeXrHbr1VF/e0s8tGFFila92NMeG7b0cu+qwVzWuNCkBPbC9X1Ke16g+pP3txPu9LCmuWrq5JWrF9kFFtunZneXozrN+PCVbtzRHOdlNjb3c4iqDvW1QPlDfK1k/nqJckBTgqx2rby3JLmvy8630ehYaXxj8NKd4disVIUDIDb4SeezuXkU0GrcocFM7sprm5RXmYoXetrhpJv5UDf1RiWp7+pOJfD+d4RYlqvd0p1NiYUfgGascNy+EmStsktB6/OGcMC3EA5pSZQXsavTtJp3APbDzB1PiVPoxVesIZS5g2HnUNqXaai3rcJ6wnEGMhIMphEBYTMa/0fVsv3VXfnbiw1Rt6sQcui6LnDifytfBnDibjF9jumyWJSmrRWVGGMUYJO02RuJ2yz5NN8r+pcqRUIvnduyptBmhrgldT2hb+ep6QtcTu+4rP8feyffh5+H3/w+g8hdpUSmd+Of59Yuy/FIzlrMPz+Jf+R/+Y1qfURhHlfW82s757slzupDxup5zvp2RZw4FWOMpjOOo2PG6nlMYR2l6nq2XAByWNa3PaL1Bq8jLj0+59/Vzvn30ki5kfLE+4mZX4YLmZLbl2fkhD06v6bzhejOhzHsOJzVfvDrmo/uv+fjlHZaLHcuq5qauuDif8+79S1ZNwaJsUSlX/fXtnKroWG9Lvv3OS356foezgzVX2wkHVUPjMnqvaXtLnjlurmaoLJAXjrPlisI4nlwe0e4s00VDaR3Oa7a7AtcZ8qrHWs+iaji/mVGVPdtdwd2jFbnx1L0l04HOG3pnUCpyezvh4GCHDxofNLOyxQfN5dUMkwVM5ulqSzntqFcls8MdSkV2uwJrPSEoTg82vLmZEZ5OqT645XBSE6LizfWcEDR50VOvC5SJ2MLhOsNyueXqYk4x7QheY3NHCBprHU0jekPXG6bzhq4TSYj3msWspveGvje4PuNoueFmNaHfWrKJYzZtWG8q/NqCidhZR19bJgc1IWia20IkRpOe2bTh+vUCM3GjJ1MB2gS6m4LDeytuVxOODzdc307Ji562tTJR0RrUxKGucszZDndRMXlnk4LaTOoh1mgTcL0ZvZa2dPRby9n9a978+JRYROb3V6yfHIh8+qYafZhZ5fCdgU2GPuyIrwvCYc9k0dC1FrfLUI0hTjym8ASniK34LDnsxP/qNerKwmlLlnvCF1P8LFCdbeRYbnKKk5r2qsIuG9ybijjxqJ0hu1MTv5zi77bE1ogEHJidbdhcJHY+Aiaidka8qiCezUoArsoD0SmyK4t6uKNf5/KcoDA7jT90qCwQayPeUadkPVkAp+V7UOLJ3BkwoGthB9/uRI1ZlIkdBdlNhjvtpIrtPCdUKeggD6heE9PrUfVyA616uaHy03QH52WdQ+eou9OhtnINZitNf+owN5l4Pyee7Fx8ZUP9iC+FyZa+WtBNYq6OJFXJXmZjynS0IfldA+Y2E+9oJ9v21b6aBpDpQc+YLG1qYcGVE59pdisf6n4msnAixDxithq70jRnDt2IR9WuDDE9VtdyEz6w4L4KZIm5z68VzZ1BAiVfCmHwm7uO/NKI3DxIerGKe/Y1Gtn3UIlCwa413UEgGpi8lBvmbhHFB9oqhqTsfiE+0vK1kTTpIOdhYGG7g0D1RgvbOhk6FkW2r1yShM4E9NskkXaT+JVO4OJKknKHTltAOnwzkeqL1FgmhXxK5DY9iaEVhYB2Is8W5lmNrPvQLRuVeELf9mpGE1Ogjfwtv5VJA+Xk+hm8uNkuJUynIK92CeWlyH8HLyfsGfGh11SY4HQ8qQt1WAYlwbD+odf2bWY85LLf5ZUwvoPUWHeMnsxh0kH3A2AWpYN+S7U2yKwHxUZWp4CtA7U/zpROO6R6241cXP1cJMrCxspjhkTlkKVqlyR1HWS0A5gbfKgDG+5z9mmx4zEKi+0q2ceRJU+drUMadjSMwXrFTRoPH8cE8kFpk6/TvvrEOBup4HHFEConzxu6S0d/sAZbJ1m1GiTdMgbeCuge5NBvn0dI7H4vjO+gHOAt2fbQJzn4H00nTKyrNDYxwkO4YFbH8XdIioBUazXU9Nitx1WarJZ6Igkmk8CtIQFYwohIvmA9Jh8Hq0TGW4qMtzsw5CtPSEy23YTx8UM9jAB1YVN1t2cqx/egiICdwYeZznHMJNE1v+noljm6D2Rb2XZIEl4Zv8SYaRn7YHUKfnLj36Pd8zHKxfT+O4xRkHUMCbhDhUwKPlK9J1RWmNYmJdv2fmQv48AuVxaz6fZBRanqRZJuozCnPo4dm2M/Z3q+6v3Iko5s5tu1IKn2ZGBAifvaov37QtiDUqWEnYR9qNEQ2JOYSABC8ny+FcTzh3ss/8TQmj+GxfzDy+DB/PPAWP7V9/7zn/t6/9FP/u6f+bHBLzmwLB49jGf/9X9JVjqqScv6fCb+uyMv4SF10qKbiCk98bwglAFVOWKvoRdZoV70ZE9KurMetTXk1wY3C4TjnuxFjpsFYpIbKqcwWy2+ukbRHwgTN/s0Y/OBw16J7DPbiW+wuNA0d0OK4d9XhLhZIBYBvRHPn13Ji7B/0JK9LAhZZPZEs/5aICqoXmmaU0kSrF5p3G+sKX9nzvZ+TDeriv5RCytL9UICI1DCmuzu7c+xOxBJq/KKbKvHN+T+0GNv5AbUTQOmkYoWe6vp7vXYN1b8pj+pJAhjJ/LRyXPF6iOPriU0prjUdEu5ufczkZDOnmp2ZzLz72cSTDLUBuS3iu5AAkTKc017JB/Qdi2MhJsP0llhqco34uFysxT68ERuBKOW8ArlkHEykfK1obiG7Tty/HYjPrLiSuSqgyeSKDeg+S1sH+xrRuxGGMTFD3OaU/mQthuFr+J4M5ZfqxSysl/X2B2aGJv6LOIeNZQ/riiuI9sHMHkhabmDZmColhmqX4gQv72Gn8wgClM9eZE8nEuRiuo+Up/JNTUEdQxhI+2h+By3D8SDV74RWa/dqpFF0l4CPAAOPgtsHmpMneTRw81VK2OS34ivcvaZSLyKK/FOQkrqDVCfyr7YjfxvqISpLkQqvLszVDeIpNJNpapGQIFIbE3DWO/QLVKFSoq11K1I/96uDEHJl6njmBDbHipJDF3JTWVzLFLB4nYI3SAVosPQRyo36qmrLyMlDUc2D/QoUVUhbSvJJ5WXc+ArhV1FVl+Dw58IKzVIFqsLSTxuj/a9qC4VwQ+SVRAQMH0hN66Dj0ukwnDwqac+0WOxvXb77sVsl3xkSbLZzUUOPIxrsHKDqLzsFyAM5R1h5wdv6fxLz+a+dPYuP3GsHmVjJUZ7KF2wNrF3po/Mn7bcPpYQkIFdy5r4VtdjSgM2sDuVdNU/vEjVkWL62tMsBcz7QnpOu5kezw3A5EI6WIcE2UFW6kqVqm3k5lWkjip52VLStBFpansgY6NTYm515djck+ASkLEqrj3dwoyS1+ZIY7dyXFkdMG1kdycbE2L7KnUQB+lXLS8DppeuUp3qaLzd35wPgSXdXDF/2tMcZ5IanWS1iy+cVMyUkiTbzXVi6mUs7TawO02BOpuY0pUj/VSPr5MhzdkVCluH8aa+n2qCUUxf9+maE4a1myrmz3p8pekrGeMhSdiVInt1VQo7SR2bQwJvNPJaG2Sg0ocp/aRDh6rsizCWxTqMXsCoFbs7GQdfNPSTjFCIrLibS1r6IFHOdpIHEKxCt7JNXwjA0H0gFFqkrG/tb7CKfqqxm0Bx2dId5ZhaAElMtRwhHwKNvCTWNrKdYUKhS4nKrtSpI1W25wtDtnW4iTCPIO9X+VVHf5ALsMmEqVSpB3ToBzXtPvF1GBvtZMInZops3eFLCSSSZGIzAptQmNGnPQCnAbBEq1F9GKWo+L3ncJCsmtpJmEwn7OLktXjogtHpuHSqDIq4aUZxUeMrO8pQYwqI8YX0c7ppRlb7Mbk3GE1+VYsf8aCUzsze0x1PpGolRqLV2KtaejABs2roj6fo3qNaT0wBOSHb+xsH2enQy+nnJXrXS72IThNLdT/2YYYiE1AUQHdO3iuHipGhWiXJW4mR/qDE3jboXSdsW+qvBAiLSratNXpTS9WHHyYk7Qjo1LaWhFmtRcK6bYQhbFowRtjNuh0fQwgC+nyQFNlcPJ3EiGo6YiX9Wsp5aFritBKvpNEpGCcS51O4WaMyQ3QeVRYSsOM9qhTWM3adhO0M1SUwVpCMQTl9x1f6KgfZ9nSy7758O601DMFBbp+464XFBIi92x+j1nuvJfypUth/reUvgOWfuPwFsPw5LPOPzuL3/vv/hFc3C7LM07YZ06pjsy3JrKe5LcgmDn9ZEIuAygM694TrgmhTMuWyJys8bmsxK4N5sKNb50wOa3bXFcoGuJU3wcHjGA874i6DIkha5cwJA+MVapMJaNsouhMvrEdQkEXwsg7lFEPEdZw67OsEXisvwTuZpEHGlEzqp35kX5SXfkwSE6M34nuIRcSsNe7YoTphX5SXbcfSo3qNSt67UA7MjBrDSiTlMkAWMbcp+bLTYBJw3YqP0d3rsM9y+oV8COs2Fd1PnQQSTT30GtVqbArlGTyFMogQSgHI9lokJ34A6SeO/DxDOUV36MEgntOZMD261eKlahXuyJG/yaRbM3WUSuG9BACZrbBCQ8JncSn/r+87slszBrrk15r21GN2WsY1quTdGgJW9r6jCCldUhFTWMvQMfk2W2NvjbBFk4jqZX1DoIwvhHkaWJR+Id5GYRXUyJTYTWKnStmXYAVs+4mMVVTQnvXY62zfnVq/5RUrpHM024nE1pfDjKoa/VntqaN8k1jfUkJ5lGOclBj+rnxiCJIUN9g9gEAh/s+7kWy9H5ehl3MYL4D2RPy/5SsBMMFEFp/D6rEEGLnUQxq1vDbym8RwhD0AEr+TsEO6l8TXfrFnk+JbQH0ItRkCZsYi7YEZSf7IkFIb+3li+8rI5KWiuSP7OFzHdq3GPk27FZ+l6aQSx64TW+H26/MFI1NnGnkcb10zWQP1aeQgSZmbU5V6KuW5WS3bCIUcnxl8ozqxWIn16Wd7RkuAb5LBemGgfC4ANb9OrMFUKny6uQQYFVep4xXZpi/26xn6OoeuTKn4SZ7WVkD68NoOiZHzFckzJtsqrsUH2C/knAab2MsqsUhVYsIO1HgcdivrGDpqTfL3Dem5OrFJpk6/t3KMeznyfpJGu2HiIZ3/JFHsp4wVNnYroMSXso6slnCjgTmz60i/kJqnaCScRzzScl3p1Gc6VEAVN2nyIVXeDN5Z6QkVoPv2xIZJ0mXxXwow0V0cx3YYf528rjHbn/N+JkzbwJQFK5LioZt28CRndWLgcgFpptn3U5oueR7tnjHMUk/pkBY6+GztVsKMska2OXSqDnLi4e++SOOROm/lmo+jXBhku65U4zaLtaddiLxyeIzUy6T3rz6O57CfSGXPsH2f7/21UYvndei11F7qeLIm+TS3Anq7qbwp5FupPdFOnt9PRII9LD5N0AQrlWPSTyvnK1/L59lQYyaTLCFNKIhf0aVtD/5aAcIheadVqgka6pvS5EeG+B57kZqSwKaM1aDKSOc8U2Olj4pxrLExXZB1pUmcwRuZbX3ytQ4TgEkpMTJ9OlV+qPG52c6nHs2UReCi9Id2YfSpx0yn/k0/jmfIZZJE+ZjYNz0el6Twpmuh8SNYHNhI3Yn/MmTiM5VjjuN+mTb5M0Eem+n0PpWOZ2APey++zVQ3ozuPLzNMLQE8Y5flwPwNjOMAdltJZR1+ljdLnXosHbHIRrA6JLgOtSlRa5ET+zj6aZVPjKLWAlbbTnopE6s4pLai1N53mZk9e5hqUb7Sdfl2D+YQzAPyt8FL6TxD7QlGCzgc/p+WAROMoT1DjcjbYDAk/6RzI9Aen991exA7VKH0/biffwRz/EkhPn94eSvU55c+vKe8F//qu//Zz329/+hnf+/P/Njgl9xjGaLiLx1/yeOTS/reQFTcXM5QOjCfNKg84FY59k6NmfXkk07qJABMhNMWnKYoe6rDGs5avDNkF5b2i7nUcFzlqKDAyHOqR2uywsnfUsgLThO3GXglkqs7Df1hQLWa4lzSX1WnUEedgDkTiVYAm76xuIn0JpkrS5h6kdktfEpLFEA1sGvRRKl8uM6onqT4ZgPFa4M78Oi1EYnbjUEtByNeArIIsFC9EmDdpg++engzVgIqD90oK7HXEkpkdsLMxQjh/Vqkh0ctplHM3r2lfCqhOMuTDdmNQXeK7tRRPVqPTE/5RhOLSLZK6W1bAUSmTYXZa4PuFf37NfZWo9L+5TcaU2vC1I/VAeVTSz8P4s1sNeW5Tqymp3phUigB2JWk03aH4te0V2YETQfvX4MaQoviPuG1EeBnOoWfBPxU0l9VhLDsyVeSsGs3cpNQXkoYUPOoQ/Uatwh0p17SWhFA1p6EBIoU7ZF4HAdw6RaSHutLYQb9w0bSeRHZY3mRJHinwjZ3ByFJ1iQhtjsImFbYGbuRsSbIuLWnnvqBEynj+80YtGJaKF9nwgIOEfxFFC/ejaJfBkytqF5pSdQ9crSnbgwcUkNqbgGrb/aYnfRv9vNAeQH13SAAdbgxzsDsxMsbjSSKaq+4+Pd6yis1BtP4IrHXb4TpJQi4FTmmsJDtcaC8EKDVHwgItGuoH/TYjRxb1sh5URGaU58khBFTC3BoTwL9VH73pUgkp89IfZdyg90tA32SaIpEMmLXsp7mOEk1ezmWfhFpj8KYuKtd6k1NN6DN6V5O5kth4l0Ji8/g5huweTeFAKXeznwlN1z5mjEZtltG+oNIvt4HEhElebW4kXNfXkiq7FCR4SZyPeQ3CVRWMHklrG7WyP71c3mtV+eJ2V8LGOyW6fsBIxtZn0oa6+JTYeaHdNuoBQS2R5I4i2Jk+upTuU6qVzJe2Vb2afo8jPJQojC2wUooUHsgjKfUWLBPza1jArBpXbUwlfVZpLgU3+kA/qqLkK53Aa19YnNDAlB2I/s9fZXk2QkcCSOqKK4HeSqp2zBK2ispjCrKDWyw8niUBB5lW2GS85Wwa7u7inaZQmuisFsykSGs9MC0u4mwpHa797OOYNSltNep9O/uzlIP7yayeOIZOmr7uQCP+TM/hh71U2H43ERAbHkj+9YvBOC5iaI+0dg6yTmT9LOfKrq5+Fenb/wI9OoTTbEK2G1ME1+S3GrrOD5PACVJJgrlbQIFQUBRtxAw1S7FA2t3svL6yFDcegFXyfeZNdK1KUnAcT/uedruLmC6SHnlx1qW6as+1XzEEYiZToB0cRvoppp+oinWgXwTRnAzVIHMXjjqI0NfaZoDQzfXNIeG5lBTXEv/ZnXuxomsrJb90El+6nOdgBxjom3Uauz4VCHSLuSzKBqFqQPN0lBeOrKdp10KcxwV2LUbk14BqtcNykW6mQAuYbXTtnORfA7dptlWAEq7FJ9hTMCmPZJZlmwnDD1aEaymnxn6eTaCyaG6I9jU67rppYZDCwM7SD2HXlIVIrqT11OwOk0USnLsGBbUemImHZu+lICgQcLZTzKpCBkqQazG7HoBqrlMOLi5Hc8XIFLaPrFiWuFmlpDCeXQnPawqRFTjJAgoyVpDbpLnssRPMrrDQtbzNrDOtEzcl5kA9l2Sp4Z0zQSIRSZMpTVSaVJKB++QFqt6J1LWBFoFZAZn1GmNAAAgAElEQVTI5H8jOBz6MpUiGunmVE2q+HBe2M3EVkoAUbYHcErJ70nyGgdWMEZiZoRd9YG3+zXHAJ8BhKYOTZW+0FrY1q6X4J4kw1VaC0va99Kvad4CkVrt92no33x70Xpc/347CfwPQUt/0pfS/JGey79YfiGXX2rGcvL1e/Gdv/NfsJzXOK+p8p7bncQtax3YrkuWyy1tL2+iu1VJVvZoHelqy/HxhsuLOVnh8E6PqbEmC1STlt2uwK9y1MSR5Q7vDEfLLVc3YlZROmKMdCnWlxWqkg/fatZijGw/eo3OPVoHXCcywph6GRd3N6xvJsQ2yWnS82OTfGURketamRHV1hOcRq0sLIU1VROX7u4iJgu4OoPE7H3FgxWAMqBqw+Tehu2bKWhhREH+r6ZOALICPesJu/2bgiqCPH5tiRMnfXYKxqRbr0Rm3GlhU51CHbfE65xoIipKn+HQtah6Ray8bN8pyN/yzmnQN8LcxplH1QZdK/GIAboWNjfmEWxAr7N9H2AWiYUwr/pG2GM/9QJMi4jZaPxU+g7tjcEtRJJMALMSCZKfBvHrWYhzh7m0Ui0TBZDbW+kYVMnfFXJJQR2YwmExdbpxS/st7I9KjJkS756CbJ2CBqyAbJ/WEYpIzAMmdb/ZW+k0VF7hZl78cIkBG6aIfJFqXYJMJgxjojs1+rwGBjnYOCYnDsm1kcSGOalnybYCMstXBjeLKdVQ2POQalBMo76STjvIPN/2BOpUUZMlRnBYR9QC/vu5zNDrVgDnIGm26z0LO1zLZic1K3ajZALjYOgJTOOXR6JFmGX91hi0apzhHxY3l17HIYF0SFbtF8LgDp64mO2Z42ynRrZ2YEvcRB4Pe7YyGpH3Dqyj7uTcZrUwfdK9KLJrFeR6GXxdUe/Z1+F8abeX7wIjizRKgtN5HJ5jOgGNOqXWDsztEO7jqzgmIlevFL7iK8mcIPswJPIO6a4DAMt2jEmtA/M1sqle1mU3e09g1Ht2YmCLh/M6dP+Jp1NCc7qlTBh0CwHabpJk0n4/NiSWTLnEzJWJyU2/hxwGlntIAx7Y9HyVklAVox/LF4xBXdkuJoYxpt67tMk+jv18Q5opirGGBAYmUtYt/jq5ObdbuYZclX7OhFUZWMpsK68DEBAYs2H76bz06dxEGbN9QMwAANL5Gc5Fkp5HIwyfBPPsr7EhTdi0jCyc6SSlFp1YxyRjHq5105J8fnKMQz2Wq9K+JTXO0MM4sLh2G8VbNoCVBLxIYNDu4shEDuyrinG8duSaF9A6vu5Ktb/WdjKegxx6YJAHn+GQdOuLxGwWwo6FTCZBhv0aUkmH9Uazf10P4E5kxvLZ1x6IxNXuhJGMiakTcCHHua8yYqykGaTC0suZPJPpXGm3Tw8mputll/x7Kj2ml8cMqbbKM06aDsyuLwW0Dl7EqMWr6CZapP9apMty7Qt4HZ9biOrHrgQIukpLXUmZvNeZwnQBQkzrZ59a2wepH9k5XGkkZbY0I6hWMRLM/j5FJY9fMAlAD4C0Sz2VqdoEHwm5BAP5wowSYN35MU1Vu4Avs3QNRpFOZnpMoJUHJfVNpkdfpu78yFSiFKpzxMLyRzyBcWBezQjS9E4qPaLWAjrzLLGKe+ZUxUgohdkcKkyG9UVjGLopRyns0EkJjAE/A6g0bwHRFOSzB3iy/7GwEtaTtoHNiJlBrVOtSZD1xb7/KmuYmEFVFFIRYrRUi6QU2ZEtVHoviwX5P4jcNQHG6MNXGdFBHjv8DHtJ7hAA9Cctg28zhj8HjOVZ/KuP/n9gLD/++3/mxwaQ/ekP+cVdTosNxwdbjIr89Xc+5pvVC7qY8b9cfIvnmwO+c+cljc+4bKacVhu+LA/5a2ef8rtvHvPw/g0Pq2t+232Db5y8Idee3/34fX7zgy/4YHqOVZ4fb85YdSWrtmRqZeaodpay6vjo9A2/evCCT7envK7nHNx9xbuTK35wc59p1vF8fcDkeM007zjfTPmNs2e82B7QBcO2y7m8mjEpOvITz8X5nOXRlt4bms/nvPvdFzQuI0TFq6dH2ElPUfY8Prriqp6Q3Q88fXPId771hNpZPnl5ymzW4IPm7tEKazxaRT774g7zky0PDm55tZ6TmcCmLvBe89e++xN+fHlG6wzWeKZ5z7opmNztOL+eM5/VbIuCadVStzkxKpSKhLKnbzOOPrii6eXyeWex4sVqwbTouFpNeP/OJY2zPHl+DHmgOGyYVi1NZ3mwvOViN+HubMPz2wNWr2fYwxZU5OxwzdMXR1Abpu/f4r2mqXMWxxuazhJeTTh5/4r1rkTrQNda7p/c8GY1o9kU3Du7HvH0q/MD7n3rindmt3xxe8TV7ZTCetRZxO9yudladuA0J0cbVtsSe7pj+2bKRx8+52dP75IVjiwL3Hlwwa637NqcD44v+OHze9w5XHN5M6Moe5SKrC+nTA9r8sxRWMe6LmmbVCQcFGFtwQbyRUu7KsjnHb7N0FlA3VbEr23JdKQqejbbUoKNNgUmi4TSE7cZhx9dsW1ympdTTt+9Zr0rcZ/O6O912KqnKnseH17xBy/O6HeWozsrrj8/JJaB8tGO+C8OqB93qCxgbOD9u5d8/vqY0Gvu3rnl1dMjslnPZNpwezNBXefc/c1XPHt5RD1NATNVL+EuzpAXjmZdQKc5e/eSV0+OyRYdfZOBU/jkH85OG9pVjp44sB7vDNFp2GQUF4b6Gw3lJyX1ex30Gj3riUERtxn9QyeycwX2oKWvLflLS/FoQ31VySQGoHtLf69D7TKiDZipw2ce93IiDNSBw0573GUpILPWhGUPjUE9buRxpy3BK1hbsqOG9qrEbHQC0SJjUgcd/cYyvbul+Xw+TiSoXuHut3BjyXaSSpxtFPVpZPntS3a/e0J7GLHvbdhdldirDF9EYfWfSf1Pe9ZDENWDdOKmG81sHzBjbzW+AH/Yj124aDA3Gf64l0kbp1CdPH76TLP+moREVM8zXBXp76TH9Zp+HqleadYfeAkxakUBEPLUHzqVlOX+IBBtILvJZMJj2ZO9zAVYloFsq8cQGt2liYZKJkHK15p8Fbn+1UB+aUaZu9lqJi8Um/ekG3j9gSdb6VQyLwoJEHlqH2H7rsNszAhcB5+zSrLgkIGfBCb3N/Q/WpCv96B+8Ef3s4hbevILQ7uE6hzqu1J1s7snx4qKlOeKzSMBR+1duWGy14axQiJ1sso+yPOlegS2jx3FGyMe2cm+8sdPIi6lSWcbAQuDpBWVulr1HtzpTlG9kU7T8hLChLFzVTuRSfsS2ocdsx/nMumxFQ92v9hXE2kvoHz6XJjq7UFiYrcKuxXpcFYzTlAU15HNI2FT7Rbc0b4KxBcpzfhDNQYFhSKx11rWZZr95MP0hfiUfQnVG2FdlVdjOJCbimx58zgw/0yP/uJREo9MMvWz1B1LCkRK8s7Ne1C9TAzpPPmGE1C2m0iohKkWyam8nnwu3Z0+V9Sn6RisrNdV6bxo8VkPgLafSU9syCSFeEhw9YMcfq5oe5UmGKC6DLTL5Cu9lRoaooDf+iR5La8lDbdZ6rG2yE1Fwpw1yT88E+A5e+FoluKJDUb8td2xgNtmacjaSDdVo69X9yLpFfCtx0kAAO00tg40Ryb1tia1zyTJcL2MUzfT5NuAmwpjO9TPCHiGfBVoDrPR+ytVLQJ4tJNz0S4L7E5SYlGgSoUr9Tg5E5LfdfSNJ3ZZJPJqDD4zbRw9vUToDibCQrcBP83Idp6QCxA2TWITtSJM9xOTdt3j5/lYq6L7xIoaAbBuZiEl3LqJSRLktF6txFc6yeS5raefW+yqRzdOAnm0MI3R7CushrAk8XwmgGvFK2t2vYDTBEAjiRm1MjsYSotOjxkTP11IExQKva6Jk1JkuI0wmaMHdACSNttLa20m7OnBfAR7qukk3RX2jGYCnDiHyq1IdJ0XcBnjmBcVfdh3WQ7MZIgSSmkzvtJjOSwp7Ic/JKv9115iIP6bPP4XefklJvX+tOWXmlfOleO/+uC3eW9xxV+ZfcI/uf4Gf/+f/4fcKTd86/A1VnseTG6osp551vJgfsNVN+UbyzdkKvDbX36D79x5yc6JpDSveqam47eefpN/dvUemkjtLGfTFcfllr988jmLouFvvvszupDxj19+xBerI37t8Bn/4slDnuyOuNpVbF2ODwLEvvjhOzw4uGXdF7xaz7k7WfPewRX3Tm/51uFr3j244j/41k/Y1gJa/va//3tsu5z7s1ucN1RHNfdPbgRg2YZdJ2DlaLnlR8/vSSJu2ZMZz6ToWBQNT88POS63fPj4FbtdwaotOZzU5MbzN979mOaiovGWInMcTWruzdf8zbOf0nvD3cmav/XBT7h6ecCv3X/O0aTmYFrz0Z033D+8ZVJ2hI1l11oOJzV9b7jYTfibj34GwAd3L7htS17dzHnn3jXzuxvai4rKOh4sb7HGczSp6YNhuy3RE8d00lIUjqdfnvArj19AGQhBczzbsTzYUreWR0fXnLx/RYyK3DqUgsx6njw7oSoEWDV9Rt1n+KCxhaPuM37/80e0fcbZ0YoQFA8Pb5jMWopJz3KxY76o+drhBd2qIEY4fXjNy9WC2UFN8IZZ1fLyeoHzmruLNT99c4fvPXrKpin44OycadExyXsmy5p6l9P2GZe3U3brgghMJi2hM5w8vAETcc5wdHdFtyooJx1V1eHebehvSrFABI3bWtomJyscNnfMD2qKk5rLyxmFdTz48A2busCYwL1ff8XDd644WuworONffvKQvsnQuZcJARspDxu0jjQfNphKGO4YFZsu5+hgi7GB15+coPKAUpHVqiIGxcH711ysp0wPamzVi/1jm+MuKpSC5qYkKx35ueHyZsbkdItb2zElNlSBOHf0tUVVnnLS4doMrSOx16hlR3OvZ3m4pX6/FT+zDdh8mBGFGNQ4C2+ygLnK6O711BcTMBE76aDT9CeO5e8VFHd2qDwQrnO0joSpFylz5YlfTjBHraT2OgW1QTlFlnnUqQRZxE0mXuinE2Gt32nwB4448eIjfl2gnGK3KYSpaRRx6sQ7rSIY6A8d4aCnf9Ti54GLVwvpXF04musSoiL/aEV5IeCuPQy4mcjbVZKtxzzIvp2mPrZSfu9OHX7hUK1BeYWuDSRWXmXyGN0qwsIRbWT3GztUrzFbTXMq5e/mNiNbS2qwvtuwfewwO43ZGGIpbD4HPSGLZGuNe9RIgq1LvXQ3Gvs8ZyibJzGRbimS68FPaG+lFqk7iFx9V8CZL8UCEI14gLe/uRvBWdQRX6bwrJTe2hzL4+s7gfJ1lnznic1Te2ZSWOXI8kea3fMZplNs7wd8GYVNboTldgtP9SzDbhSmUezOktd5mgLD8ogKKfAssfr2xoiEHpFE64FtV6IQkMRfkt8b8guDq4QRthtFf5DY+KFr1zBK3aORLteQC3ue30g90cB8NidDeq18dUsBY81x8kdmkdlPcpFXV3H0n+pe7etgKvnbICcurhTVGzUygYP/uj2M9HOpizFtStYt92yhpJvKerJaiZQ4E+m0S2y3XQsg052w1c2JgO1sJ88b2Da72Ssbsjqy+FjTT9N1VO3Z7agGNYUcw+D3daUEYhUXIiP2yR/rJgKMfZWkwxNhgbWL5OvEQHvxiA5geagw2t1VYyposBJa5StFv1CjEmDoGo1aCRuuJMxrCKeKSiTK64cmMZHQLvbBSr5QlNcB3cvfBzZZArfU2Hm6vadpD4ZeR9icZeKBNXsGOmuFcXcTYVC120uh+4nG9AIWvU0qmRjJGrkoNvcM3opEupvqEcQByWOqx7qibq5pF3uWNyQffbAy4ZAnkC7yXj16he02oIJsSx4r+1Lc7Fku3ccx5XbwyMr4D9kEgWwb8EViVJ3IfU06jqjlGHylZRtWZL/BymNNG0aW1M1SVkaSGPtcWExXadzE7L2bRmTBMVO0h3nypPaE3Ajo7QWEZlvpbgxVhpvlUm0ykdqYASz5MhOv5MDIRSBEdOtGBhIl110sjMhwCytMqVbi4xzCk4wSYGiMhBgdTAmTVHEyKYhFTswzYpULsANhOm0mXZqDRBYYU2DLPD3PEqtCvheWWOZQlQJUh0TYwaeZZZBlqCJHZRmqLEXWaq08riqTJFdLUE9alBEZLIPMNn0pm8mXUvuf/7ivPEflOTq3/MXyi738UkthZx+exb/zD7/Nb59/i1w71n3Jp89PibuMD7/+gs/Pj/hbX/spbcj4X3/6Ed97/CWvtgtu6xJrPNfncz587xUv13MeLW+YZB2//8Ujpv+84p2//YRvHLzmH/7+r6PywLfee0HrM3a95RvLN/xvn31AVXVjDcemLriz2HC5nfDu4TWfXRwD4H42x3x9Q7Mq+OjxS3762T104YlRYQuHMYHvvfOM/+vzxxRlR555tnXOcl7TO8Pt9VRuGF8V6Ps1eeHYXlWc3rvl4pNjOOzIrKe/LihPa5pLqYfoa8t333/GJ5cnbC8nMtNeetRljr7b4Haii1O5x9jAZNKyupxCr3nnvQusDjz58gS9SX6Lyo+S1fsPrjj/vbs8+MvP+ezLO5zcWXHx/AC76OhvU5qZVzDreXB2zdPPT0cW5Zvfe8LH/+d7TL59Tdtn1JcVmMj8ZMvmy4WwPQ8a1OuCcNphXhYSztMrysdrup8tsF9fEX5wQPyVNfpfzWm+1kr1xlErdSIaVK2JVaB8asUL9+6W/tVEKjMKeaM+PNyw+cExbhoJlUcltke3CnfgydaGe999xcV6KuyYAjvrCM8r9P2a8GyCn3vp3ywi6qCDi4Iw8cxOt/Q/WNI+6ESmnAeyT0uigoPvXbD6/RPaOx7daMzdGu9lRjv7oqQ7c2RXGeF+Q1hbpk8y6l+p4VLWXSwb3NMpqpd6jeLhhvrNBKx4T9EQvrlB/8GM5t0Otc4o32j49RV9lxGfVxJQZJB9a+SG089lXMoXGd1S0nPtRnoSlYmwzSQgaukxG01xpdk9dBw9uOH6syMmLzTbx47Jlxm7r7coHWFlya8MvohMXyi6hXgkq9eSAJw93sAP57TvN9inhdycLx3Tj3O2H/Tkb2RGXCWZanGt2D30FBeGqCPug5rq/5nQT6G965g8yfaBKL3UPoQsYjdSq2GvDHYj/Z3Kq5FB061m/kVKGFYCFLpj8bROn0kS8errnvknhuY00j/oiLVh8lRYwGjlxr56JanI2U58t5NXiu2DSH84JCdDWDjMVYafBA5+krH+d2rCxrL4acb6A49yiskLuZt35V7G2ny9IW4yspUhv1WjpHUIvmlOIrMv5UazviMz27pT5CthnwaQsPhCOh99GemPHXpnmD7V2E1kdyaAxG5h9e2O439qufr1QPXckNVQ3xE2b/oycPUtWfeQQFu/1zP53JKvRL6qHXRzYT77RWT6LAWp9JHtfZi8VrTLfbXG5GVk+0AxfxK5/QBmX4qUrz1iTD4uLwd5osh8y3N5bnMaWHyi2d2LHP5EElpNK8ExpPPZJ9lzloKUpi8i68dw7//wXP5Khl1Dv5BAoOZOZPEpXH1b0qrtljGcJr+N6WZW9sVNRVKskyvBNMKwFLeRzUPN7p5IystLAQ71HUV1LoFG+W1k8y7MnsD2oWL6TADQ7fua2TNJNAaR7danIlsuz+Vcm06x/GlKp50KkEJDtoH5C8fVNzLKc/FuZjvZTrZVHP/I0RwZuplINjcPYZJYP0k7luRiU0N7DNNnkayV/W0P5Fye/EC2uz3TI8spXk4BPtUbAQO7u4pZen6zVBQrYeLaQ/Gwrt6H4lIxOQ+s3pUE3MUTTzfT2CRF9nYvYR4ChbqFAIrBR+sKxfSN4/ZdS3UlCa79VNNXe3YSGMF2eyDPn74KoyzY7iLbuxrTwux5z8V3c4xU+KJ7OZ/dQlKMo0pjd086jqsrP4KjaKBZaqpLSYRWianUToDX7FmHmxrW9zMOvuhTP6Zne5Yxe9bRHWT0U50erzj4vCcaRTc32G3AbhzNsaWfSjpvO9eUN+KzXT3KmL72IxtoN55gFZt7hsVTRzts617G5I1H94HmOKO8dGzu2zH8aADKxUqOtZ8KCJu86ghW0x1kBKvIV55+KtdAdd7RLWxi9SNuqslX8pnSzwzFVU93kJGvHPVpzuRly/pRweGPVmwez8lXjnaZiV9166hPcvK1J7+WSb/6rCK/6dk8KJh/2dAtLHbn6BaW/LpL4TwZ/cxgNx7T+FTLIvugNx39iSTU6s7Tz6V2pF8I+FMuUlw2o99T1CIaU/e4WY69riFG/KxA106CfFwQwJnSeYcAnWgTUA0BfbsjHM4IucGsG6I16HVDWFToXZc6L6P8fbXby2CVQu0aYm5RXU+clJIsW4hfcvj7KIXNrSTGwshcqrolzsQeNoYFabWXyLZvBe1o9dUgnsSAxqb9qldyqGRpGlRRyPE2bfJBKtmH3I51JLHrJam2KIhtO4YBfWU7f9ry1uNjjLKef8vlF0YK++A//bmv9x99+g/+zI8NwHz/+9//s96Hf+vl7/13/+33Z//Rb/Jss6T1VuSaWYA8EpRi1xS8aBa8qWf4qHmznlN3FucMu22J0pFNLzLPq+2Ui90UFNRzhSoDP3z5DspGQmO46Up2vaV1lpfbBW0tz9vWBXWbE7ym7i3NLmfjcto6BwW9ESldrDM6q+j7DFt4fJsJg9Ubvnx1QnSavHKsr6YEr2ldhvMaX1uxIUUNladdF6gs4pXCKQVBE3qZTnU7C1mQmhUFb9ZzvDfi4dRAJ9JEVQRik94ovCY4TbuT/cUrNm3BtrfEm5w4l5tukQ8amDnW25IYDDc+JzpNGwyxyQhBSyiSV5Kea2DnLarwxLRPF12FM4qmzsVz6jWq0bRejErKK4JWY8IlXowp0US6XkBu31q5oXNZqhkRDVfsZJpbdSnsyMvNn+4VzmVj4BKdgc5QNzl6Z/ALMYgpxLMXKmEtUHBbV7htjmo1KqRewRuL80bAQw5ZrQlZlPG1EfJIt84haJTTqMaQnVu5sc9guy6TP1FuBL034pvdZagoYFF5Rfkkx01EfuW1APM4CfitRTdpW4BrLLrTkgacPiNca4WJ6LV0IqJorYKrQoB3ENZlSL11hw6zNZKOa0mJuuAn6Rx4TX6VZti1GrUOUSvqTSkAuZEZ7GCR12Kdke3SDHYRcTNZp/LC+oQc+i6T2ptOgptCLteZCkr8xex9YDGT8zJuP0JwkoobrWiPQi4eTV+kmWAjx6q97FtWi/dDvJKDJyfNJCOpu37wrmXyP/G9yv9VUPgJqI3BNFr8oFEYNruR9Uqvovw9ZBAtmJ1GRTlusxG6LdtoQqHQtxkmXV942Z4Kwk5kKfU2WKQzMwpbJMckr5FBOqiiGtkzDGNS7kD56oHBmsr4Ka9QXiePbDonVgCY7hUxatnfel/GjpIxru/KOI+dhBHp/+zV6O9TUZJapTNy/5ruZ0r8t8gY93NhtEIhqcJuomQ9yaMYjMgtVXjLN+rkb6ZllE1KCJjCl3sQEfMhQEXeB3Rg9NkOHjsJaxJZaj8XdlMoQzUGtw2yW5Eagq/UPnE2eYmzJAH1pfzcT3SS9qkxXMsnmaX0gg7e0/S/XG7MYiaAWpipQXgmY6+9TBKYbqg5SfLSNDba7499XF8OaBlT08l6+ql4M03/1vXhSYEpcjzRDGFWjBLMaAdGVa4ZeaoaE2eHc2a3qc4DNT5fQtxUqvoRH6V2aqyJ0j5V4wTSusWX1s9EHi1S3SHZNO1/J/Jh7REZoiaBuzQ+MSWfsh9LnTzo8hpX9NNh/+WcByvrUkGuS5N6SCXxWaXPIyVBOSbtg1JpvFM6rLg7ML1MDGVdSrBTgNH4QsBYlnotlUr7qWVfXSGv7/H9JQFNUqDNILc0TupQTDek8IJxoFM9zZBCrcMw/vIc45LkNMprwCScob2A4KwRn6N2cmyhUGStsOQh17KtXvZnCMoDuSZC6r7UqTYkGmEYFXt/sHgGZSxEAZD8miR/aoSoxasZrSZmGp8YQx3l+BSMibQKAbOhNGS1T6/duH8NKYUiEnMzBikphYT5dGGU/2qfmDyj0S6ge0+wRiphUkCPTsympM2mIB9rUH1g8GsPADPkmYz74PEMcq+ikG1L6mryeSreYjbj6JXESJjR6KFMaa267dN2Bv9i3IO04fvANPrkbR3SZAegNpyHgT39YxJb1fC4wSs5eD3Tfux9lGk9Cvk9CLAevJeEsE+IfXt76Sv+ofX/sfs0pEW/7e38N1w+4w9efv/73/8f/61X8HNY/u5/8w++/3Dx3Z/7ej+5/t0/82ODX3IpbNNY/unv/AqvXy+53Ey42lV8cHSBtY7zp4ecLDesL6fc/uRYAnt2ljwXGaXOAjEo2k3BYtLQbnOaXc7hfMff+M6P2bUW/7riaLmhOqoJURGCZrcu2DxdoFSkry1+laN1QOlAc1OiTGT3eirBN1FxeLbCNxlm3rP7eEkMiv6iQumIelYRGoO+zdCFZ3tborYGtckI5yX9VYmZ9igdyU5q4qsSWk32UrZ5eLyhmLUM/rMhOVZPHQQlbF9j0FOHyj2qchzev8V3GgLYRYcqPThN/tpKCM7UkT8p6FcFHMi247LHzHvCRMZs+qOSuOw5+N0SZQP2ZxPyo0b27dxCKcm2qjYc/ZbMlsXSE6uAeV7KB+r1oMmHWEltCzqi7++onliyZSfVLXca9EkLGooXljgX6eXxh5eYtUkJsZJIa68zSZY96ojHnRTRzxx++v+y92a9lmWJmdC3pj2d8c4xZGRGZmVWucrubty2BFiYFxBSizck6G7gB/BG8wa/ASGmJwQS/4EHqHekFsZ4qLJryIrMyMiYb9zpjHtcAw/fWvtEll0yRoXKaXGk0L1xzzl7Pvusb32TR3bLOhHZ08MWCof5X2QYLgYgY4l8+UqxQmEvkd8o+MJj9nOD6ZcESPBA9aMSw/yuxgMAACAASURBVLGFuZMYlh75tYJXQPlWYfaEwKx4nuHozzTcaU85XekZTV94+MKjekkQZbYSwmLshmQIjMfJnygmygbKDoMki6h6geqLDNMvDVmlrYSf0pemazF2adop+1OLK7KKk+ca3gRUv8ihGoJnVzFJ1i4c7MyjfGmQ30hUb+mry284WLEzJtYufqZQvhVkZnoxBtFkK4mjH0uYrUD9ycCO1aXD8Y8klj8TyG8I+nzpGd5Tc3DVfY9UwPGfS3QXjqAsDygvJcor9odWb7hP2UrA7AXKdxIuZ4k902OB+S/IEA7zgOXPBOzSoT317AKs+CVU3Aj0S4/Jc97uggw4+2Ng8jyllVIa2DxwYxCM2QnMv5TcnyqgeWQx/wJoLzxmXwHnf8pe0X4eMEwDjv5SYPaMfajVq1SvwOTY4lpg+pzMzOSlwOIJ+2XzO6YVT16SsRpmAbPn3J5+wQF2v6BUc/4VcPqjgMUvxOhhBLE0hkVAtwxjTYmwQHnJ1N7qMgahxAFmtmVib7ZmIM70a2D6goPn/QdMnJ28Jng8/fOA/Yce86/8KOUrrwKyDSA7BuuYPd+bbYCLP3aQA8GkjEEzy58TZFVvGawyTBCv64D2lGm2xfUhUKe4DuiOAiZvAoYpWcrqbYDZAEefewyT6H3TwOILj35BNu/8Tz3shKxnv2AHY/JIAnzP9FWgpHIAZi84sDr63GPzKf+Wr8hSCsf97BfA5E1A9ZosYrYlW0mPHX8vbgPmzzzMjp4+1QZMX3o0pwL9Api99Lj3f3Q4/txxMOzYGes1sHjqMEwEJq/JTC6+8shXZDSPf2aRStrzNYN9yiuPySsyZyc/GXDyE4t+IUYQp3pg8tahvPHYP5BYfB29tVc8brOXHsWtx+4DgqDlUwuXCRw9sZi+dmNgTnnlsXxqMbn0mD8jE5Y6QWcvPI5/1qFf0r+abQNmrxzBdAMc/WLA6Y87dEuJyaXD9JVjX26sbnEFAejRkwFqCDj7sxonfzlAWmD+3GL+9YDmXEbvHydNpm8csk2IzCPlrqYJWDwdKNuMqbf7C4lsy3Ta5AOk1DEF6DAkx2UC+dZh8VUP3QWUt2Gs79FtwOJpD28Ejn/aYPayx/zrDvNnPVQfcPRFC1sI2JwdpounHe/bObdBBLKXkzc9634a9pqmoKJ842BLgWzrcPIXW3QzheLWYSgFlp/vWL0igOqdRb52WH6+Y+jTziPbOlbdzPmdkK8shAUWT1tIG1DcWCz/YoX8lieyuGZ6a9DA8k/eQdqA2Vd7DBOF4l1HJnIqMX22gysEpi8azL7YoXpVI9sMmP1iDdl56MZh/mQLvXcjm1q9blG86yBcQL4aoBqmyxZXLco3LaQLUA07ROXgWVWSRxVGpVB9tYHPJKZfbjBMNYp3LVwmUT7fIrtt4Y1E9WwD3TgME41holG+reG1QPXkFj6TyK5rAEB2uQccuzfNpkd2uYPZ9gSkAiPD1p2WEDEEKBgJ2TEsUW9a5F9do/ziCiH2kuoVmcWQ6ktcgJ2YsUbEZwrqZseEVwCyGcbUV7FrIHoL2VrodYNQaMjVHvJux8mhbU2wmwCp9WQt6w5+UsRxUUDINUKVk500GmLXUOKaKkIAJss6gl5Rt0hJrqLpKJstcv7eRKtHboCazCuGlK4lyXCmn6lrU8Y6EaW+EZYT+p6VIkohdH1kJmUExNFLGpnT4BhUBCEIMIeBjOP7/5yP4DQc/jn31/8Lnsv8tj+iJPrX/u/vyONbLYUtPn0Q/p3/+d/D2/0Ml2+WQC+ZGJoFyKMebm2gdwruvIdQHmGTQW1jGEfOyg8R2Qph4wxn7BQMBmNxOgKZjn7pY4dYgFmRBfSa8iL2/YEzlnF2nQmGsXdSstJAdUwDVXuCAR/TX9n9xAEzB45hTGBkqECIgzWB7sSheMsQC1vGmdGGX7hMpAzINgLdMX1L2VrCliFGfdMbM8wwpicCZITGnrrZoXsQMnpb4ox9H1NJ02x5fiuw/8ihfMUaie7Ex8E4ABHQHzNRFIGDUFdgrFExGzF6jbINgzVSt2N+GyPM20OXHwvhGeqhGs78q57HQNdkw7zhQLVf8LylDrmUlJiWBwD9EWVlQGR5ED1VO4H6nkf1Vh56A3sOlrujWP8R/UEuS/6Y944bJ7WhmngNpDTWlAqo0/nmteJNrJeIyaS7Dxj4MXvBAWdKCE0skMs4AG9PGb7BkBAOsBkLf9jfYQZ4Hdg1eR4w+zoxQvFDFD/+qceQ1Rpcdn7HgZkrKFlUbQoV4bFWLd/HsnqwjuEoJZ5iXEe+CiNbYOMAVLWxAmLK88WBaUxCdJwpH2YM7hCBxzfVN/QzyidZORC/2PuA9kygekvfkYqx/0ELdAugehfQLchMCM9lewWU1zGZU6aeOvq3zJ774jVBiQgMlMk23G6zZciI6kLso4wMVpdAVRjZpCxK6HQToGvK+HzG4yU80B3zHJstT0bQDPBw0S8nQqxt8DzP1dvDvSFdM2Nqpkn3gChtNJFRweF5EwFoSg32WoxSvW8kUVYES/2c5xTAuB/pNayaYIpo8q+lbtAUapKSNLNdGLvtEtOTOgxVZEP6Kbeln4nxevbR3yY8g0mGKqbnCv7fGQa0lNce/YQVHP2MoSnexM+PoGS3uvbop5FJTKmabRhBRzqOqo+DyfLwWUnpr6naw2ViPAamjv2JgqxHSny1pYSNAUNZrF9h+qVAPxXj3+jXFMjXDl5HNjcyhy4TMA0rMnSU93rN455tffxcHaSs/M7i/qRrQHc81vS08XrpJ6wN8ZrryHYeLpeozySKWz8ec916yMiMJUYu2yQJaUpUjtf7RMIZgWLl0M1ZYQIAZmPRHenxWmTCKOWqXgPVFSs1bMFeyBEYOqo8dH1IMvWRKfVGwGwsU0htAm8EeGbvR18k0jHP6LsTMcWUXk/2OdqKJz4tX/gwSipTTYgI/I7XezeCjfbUwGxd7J0UY7cje0J5DKU9pAUHLaEbThrYSkF2BBWqdbCVgq5Zy+EqPSaKytjVmDodVWMRMslUVfCaTL2PKV0VLsDn8bWxgoOewPi51wKydYAUY1orU4AV5ODpZTQSuh6YsCq4HbzXcjuCkYAL4+dbDtxu2TlKOwVG5o/pq2Fk6VxpoFpLNnCInZKthSsN5MD3ey0he1aE+FwTQAFjqA20BGwCWHKsHElS1TF1VUUGUAiyoD3rTUKmmfyqZXxO0PvoAkFiO+CX6zNGFjH93bpx2ekRCsP3AqwTAb6R0BqMprRVSdaISIYDja9PUtFf7p6UkmBQSfxydyS9j+oghTX60HmZJKrp/yEcEl7HtNb3cEDqixQRLPqA9zskx8qPBCjT8nyI0tdUc+e4LyEQDCY29lc9/hZ9lt/6VNj8XviDh//xr325P/zqv/qN7xvwLQeWJ98/C3/4P/77yJTDk9tTZNphkbfQ0mPVlti0OfpeY1Z18AHYbCtUVYd+0NDawSiH3moyjgD2+wKL+R7fObrBn7/4ALZXWCxruCBQ74txvUIELOc1rq9mKKY9nJUoyx7bVQWhPYKVOD3b4OrNgl7I6xmycsDwcoL8wx36zuDB6QrNYHBzM4XOLYKXcIPEYllj32QY9hlk7pAXvFF0jUFeDmg2BY7PNtj89ATZpxu0dQaTk4Vt1zlk4SBlGLdzaDWqeYu2yaCUh+00Qq2AzCOb9ZAyYOg1hAgwmUWzjvsZq0jKRQvvBbyndMQOCkIA1aTlMREBfpAIjWZFSR+TE7VHOe3QvJnig8/e4c3NAlk+oL6cQEwtQsNv/clZjabOELyAVAxAESLAewlXaxzfW+PudgqpA/xdBn3SYthmmJzyfY/O7/DyegkpPYZtTqBkBVB4yMzBdwqqcPBRVimN56RarSEKB2k83M5ATSwW8z3Wm8k3vif8TewKzeIkhAfC1EGuNfzCcsLCSZhygL0uRg2AmPesmrk2BMU6QJRxfVsDUVrozMUEVQm5VxCDgFta6DsNezpAFxbieQk83sO9ruArj9QLKXzsUAX7KLtTzr6GwgMD5Yv54y3q6wrmTmNYOtarTChZSTJa0UtABuizBu51BZx18LWGrBV/T/UzNc+rnzjInQIkGbkkkwYAv7BAK1kdEwSgAsROUUo4CFa2qECm2YQoxQTc1ENtKdMOJsDEiZCQB8AyHEd0PD5+blk/08YBbSvg7vUIrWLS68JCblg/4yYesntPlOEpDx2OeCzo1Yty4MDXq0bClRxkqa2Eq+LgsCejmmKH8xuF7sxBNpQY2olHfqswTD1c5WFWCsNxHIg1cqx78Rn3W+85iSQCJ4JSumx2q8b6EYAAZpjTUyqcGKtbggxwFStvILkfSWIZBJCteXz6Bf19PgfcvY4VRU5Ar8mI6x3Z0bRfuhWwMcxFNxzAizjR1i89sjuJfuEP4TACMGtOYhG0CwwTTmyxF5UTZeyNjBNwEZQmkBYUxkCQ6i19uGmCDOCkUvFOYpjH7doRmKeJMTthKmt3FMZ6mv6I7HiaBJFpQiTKvFn9Q3DWngSkKqAg4/scJ0GKG4bluOxwGaVaHBFltao/TJjpOoJ2RCm1i5Ne+gD+U/2K3iPKQiOAa4B+dpjM7Bc8jtJyQitbi7ESR6QAI80JLLJziMCPk5j5nRgnC22U54o4GSlT0mrD9w8zxMAdTtbY6jChmK4NFyfwEohsj5mUastDEBBL68nquhwo37FqBIj7JeKEbfR4m5oTQMOUkympqiMqkQ8BUYHsJetDCPxS2AwnojgpkapNVBuiPDQC8z0nyAiY+Lo0YZAAbKpPAvi+fBMwlIeEz5Tgm3pLdcNJpeT5HStS4vbKGLxjS742VfI0p5L1JvGYpWMcxv3lc5RkcmIlTRSljlXdpn7Ow0RFkn5yXw6THD6llEb7gIlhONxYvJfMirHGBeJQ+2JLgeLOxW0kwA5CjLUtqR4lCE5c2DIy+luylN6w4iR5Pl1BgO9KCdkF+DxOIMbJnTSxyGTXeK8IBK6pw9Rl793XxSHsKr3GxcoT9prKOJHLPktnYipsAuOWYDF1Z/o81qJYPwIh2Q7wuY4S6SjdlAIqgm2vJeQQ60qifDfVlvicFSNi8IBKVg4Bte2YCgsQICsx1pXwwgwRvHoC524gGE6AOQHFlLyq1QHoJmDpPX9P1SQhENim1wEH2eoIIuPf08RFem70b0a1mbUHVjOBSOfIsFrLoB6A4HUYMPZlvv8IAd/wW37jqV+NTb71Hsv8IvzBg//o177cHz77r3/j+wZ8y+tGWqtx3UyxLBp8dnyNq2aKy+0MubHIlBsTOwttYZSDkgGzvEOpB+z6HMu8wXUzwTxv8fMX9/Dxg2s8f3eMP91WUMrjsw/e4YvXZzg+2qNY7mCdhJKBMlkvAC8wLTvMixZPn15ATweYzMJaCSU9zLTH9fUMR8c7LMsWX7UGQgQs5nu8vl4iL3rMFg36QcPBQyjBvstOI5916K5LDDKMKZkhCGSTHnc3M5z/gytcXi4wO6ohAOy2Bf7RZy/w1d0xNu+mELmDUAGmIOg0mYUxDpOqgzzxKIyFkR71YNBo+lPrTYHl6Q67fYGi7DEvW1ytpljOGuzbDJl22LkCIQhs384gJwP8LoNe9JBFi+AFfFAo5h1CAJp9jqADXnx1BrPoUGQDZo9vcfnqCAgCat5jf1PBTHtAAfamwOThFvt1idBJyEbh9s0CZt5Ba49mIjHsWduxXxeQ2uPrr88AAA5AGgGIqUVeDmjvCohWYnKyx+YdKTlnBeROU157lUPer4HpALfOsHp3TLBYK4QZKy/UaYfwtoA+b4AgYK8L9olOBc4uYoDSfIC9JagMlYW8M/A7A1GxQFuedDhd7nD5ZgnXGPp7bjIMCwvIAFEr+IrAECrAZwHFswzF7+2wOjJQLyv4pYW+NQiKTAhkgN4q2NMB3ZljV2gn6SFVAf60x/BkDjH1GJYOohewM8/E0yxQdlwriPOWns1fTOBmAbjJiY0DIF8WkJKARN4S7Iqa3k7VsFLDFx7la0UfYEmfoLqhbtJnEbB4oLiUqD/ykHsFnxFAogfkQF+um7kR8MDzVPqCAF7embGLU70zsYsxxMEICCpbiWwl4fcG/YmDhBz7P1MKafVGoX5A2bOu2Z3pswCnCKJkz8AdvWP8vuwZmw9BWa2qJbyhAkD24LaCqZr5tUJ34pGtos9QAeZW0b+oCMJTbyRDYMLoG5W9gGwj04TIZBeAagS6hz0mT7KxVsOsY2Kq5Wg/dS7CC2TbCDyH2O2XMeDFTgPyWwk5EPHIgRMkwhGUQQDlO4H2jEmtxXWU4lqQxahiP+CWcsj8Rh56IRWZ+myd6kGicsNTXWB2lAsX14e0WBW9yanXVNcEOUFHsFPz/HMfAbHjADf1nUoHBEuWvFvymHlzAC4IQPVaoLkIUfrNYzpUGI+xdGSMg6RnUEWgmLowfUzqHGaxVsQeAAC9uhyEJ5UE/GFQnm3I+Kf1Jv9jkOznNHugXwjKXi/EqEKgvJGsIysnxAg0zZbHgAN5/q5rAjgAY3KqcDz/cmBgT4g9iapFlIEyaCfI2DFa8FyUl0ym5fVBNQmP0wHMiPoAgL0h25/ADZDYTQKUYc7ndRtGNUxIqZiI6gNBQFld+RH0en0IG7IVAUO58ujnAtk2dueCqbPZzqNbUNnSHpHpdgUDeYYy+p0dqz+cOTDaACdBEsAVkc3NYrBQtvPA7sDk9/MYaLVjCFE/lShvGcwDIPZqsk9yqFilkfyJ/VRG9logeALy8trDNBxM9xPJYxRTVIEIHGsPnwtYeZDXIoBy0o4qp7Te/JZssG49+qlCVtMXqGIli/CBr41ex4N3mWA027rRD6wbT++oEMhX/F23HvlqQHsSL5AAmL2DtBFkeUS/KcGbtPRgukxCNZw5SrJuW5KVlRHEqS7KOB2Pt+wDA3lmhsAwpsZ6I6EGTqy6XEK3Di5XkRkNUDZ1ejqCzAB2oHYOwh1CdphKLCA6P27TMGMIkNr0cJMMsnNkTycmTkZagkMfIKyFcIlBPShx9KaFnRdRlorYqykg1zVETG6F9cAACJU+swSForOsDLEeAmSw3weVYrAIOqNPsh1GFtRXRawhISuIwRJchjCCSjFYJh1HQIkQGPiTGEzvGbSTypETu5seyScZxHuVIrxOg3UQGgScqXYEIKiMoDQkZtUYwDk+F2tL0iO9Rih5eP2veoS/4flv0+NbTOr9TY9vNWM5+e798Af/wz/D7yxf44+uHmNZNLjczVCaAd9ZXAMAtkOOn727QL0u8fDBLV59fQIxSOQXNfpOQ8iA2aSFEAG7Oscn5zd48uocJreYlB3ubqfjLH/QAWeP7nB1uQCcwOxsh92mhNQe3koEKyH2CuffucHlyyPAeKjcwfcKYZCYn+2w2xbIvizRPkz6KgAmAB0DZ8r7O9TrEvrKwC4cWZ9eYvpwg+3thO9pJeRsgN8aDnbzgDC1ECpAag+8LjjIuddCPS1hH7cIK34piF7AFx5ikMjWLGd3FWsKQhbIKG0U3MxBTSzEiwLuXg95YxAU8PD7l3j59ghYZRAnHcTbnExKwXTR7/9rX+Gn/9djhrB0Aovv3aLtDZq3U2Q3HAQMH3WQ2sN8XqF91EPUDDMJpUN2qRE+2yM8nxBEFQGiF8gf7RCCgHsyhZ15MjUC8CXZIbPhYNieDpC5g3xZMAQhMk7ZSqI7daM0GhsNc9EATyZAAIaF55eODsjfafTHBDqyE7BLC3ND5i6/40y8LWPa5TQWhRfxhicBTCywNlCNgPl0C/Enc9QfMYVU7SVcZA3VLt58TUD1SqK5FyA7YDj2TOEVAeZNBrNlZYHL6Z1kAA5gf6uGfFrCTgI760BJ7e4jD9WIyEzS53b7DwhGdCMgO/ow+yOH6oXmsmPv4nDkUb1Q6I4DsjuChXRs+xMek+VPJNbfDcjv6Hn0GUGQK1mXkGaPIYH2zGH2pcL2Ew952kE9K5CtCDZ8LKMv3jGNlVUUiHKzyEDtyaCpPjJescPPx0AWXQPNfQ/Z8RqcPZNozsI3JNN2EqKvNKB4J8Z19HMAgmBvmDIYKFvH2oWFx+S5Yg1EGRiaEidJ0+x/EICpCQr6RcD8KbD5hMEq2QpoTznAzO4EbJT8AodKhPoez1Hq/RumPGbFNbct7Z8c6Hu7+y0COtXGwbyPbNxAsLP7jsXsCx09opTsJgZLWi4rMVHZluCnnwemr+YC3QmQrYDmXsDsK25Td8RjrhuCq/acHlevuaziJsrxPVBdBtQXlAnvHhPcZRt6BhMzmZg+AKP0WHUHqW8CatmK53yYkAVpTyjhVx2ltn2sP0iMi24oRfU5rw1XRJ/knNszzHhRlu9YocH0Ux6n5oKVGElObbYRxO8JNGavLOpTBd0G1PeYniscwaHqDsDSG7J62Yqg730QaisCKzshU2n2YayOSDJg3XhsP+DgzRU8Bsnu4HIxdlGaJmD7gYx+0QNjmK89mlMJXfPclpd8P32JAtU7z0Te+wp6T5Dk1aGWxNR+rMDINx6yZ3BOPyFoSPLjMVgtUN6cOhIT85fYPpvH93WUoUvL684WIq6bjOUwEbCFGBlgn9FX6iIzq97rZAQIns3eo5+pkRVObJwIgDMC1ZVFc6q4/JJgsZsrZHuP+lSiuvYRqBHQDZWkNHnHdRV3TEztF5ry4Vi7kW3p365PFKZvLeXBwMgs5isHO2EicbtUMa01YPWpweQNGSYf5ef51sW6HA9Xxq7E1o+TcUniKjzQnBlUlz36pYbsA1TLrsVhIlHcWUqLtw6yd+iPMmSrAf3SjEyftIF1HZER1TtKMW2l4Ao5AjpbJN052ThvJPTewuUKriTTCCGgd/QUDpVml2QhYTaHv5ntgPpBgfKqH1lOHigCPq/pcfQ5lTDSecCF6IsUUK0d2S1KkSnLDYL3NNlYuHkWJ8jIEMIDcnAY5hmym2aUsLqCnym97dAfl1CNJesaJzpUM4xy2CS/hRBkBl0ANMOD7DSD3g8Ee85R3toMYBBa7OHMNERr2UfZ2vh9wf5KuIBQGibBRqZSxHXBewQV/zZEBlDKg+xVK8p4dzXTYAcLP68gNzVGma9WEG0PBjxFljKxmgDQtEARE2dDOEhPf1lWms5Vep+U3L4ocRXGjM+HxHaGKN2VChh6wEf5q5QQRYGw3x+Wn5Yb15M8k0Kpg2T2b3iEof8bX/OrHn9nGMv7/+Gvfbk//Pq/+Y3vG/AtB5b5xx+EB//iP0MwAcsPV1g/PULQgYP30iMYsjpBBbjSY/pUw8YC5/qBi/I0ATt30CuFxROg/idbtK8J4KZfK+w+cTB3EtlGoLnwjKuvDx1WLuMspCsZZT8mBDYcoLqCFQTNPcrI2gcOk2dq9OsEEQM14mBYtxwYzZ973Px2LCqvBVxGxqN6E31/78l3yncB+w84o9/c98ivYvDLmgOP7ScYB9h26nDyJwrtaWQDkl8o5zY3FwQH1ZuAux8gJvrFSooIBM7+PODmdwQWX4AeQMH3zZ4B9f0D+HJFjL5/6bH+WI6F4pt/2CN7bXD804D2RKJbHqoG+oXH5JVEe5pKxznw75cB1RuBYQ5Ur1m2vviZgiuAyVs/+qy6I6D5cED11Iwz68OUpd7NmUB+G2fCJeVyZst1LJ5w9rw9Jbj0miyOsBwYDVMxys7a84Dp1xj9hN0RMH0RPYEeWH/m8fH/MuDd7+VwBpi9CNg9JNDJbwSKuxDDMLhNyy8c1BCw/UCjn5FJ2D32OP4xZ93vfhBw8mNG9E/eeJidx+axBjzrAJZPPLYfyigX4zlafkGP0zCNg21NX6Yckq+T4SL5HUHM9BXvA/2Mf08z+t5wgI7ATrpsHZDtAuoLiX4RMHkFLJ/0ePePcwzzgLM/97j5bYXFFxwgDROB8sajXUhsHxO4mT1Bj2qB6SuH299SmLwmCKiuPPI7h3f/2GD2nAEeqj14iYMEqncetz9QKK4DJm89Vp8qZOuA6tpj/VixdH3Lz5fZkXnbfyBw/FOH/QWvmfM/a3Hz/QI+J3MlLbD9KBbRR0nc8ecddg8y2JLX+dmPBtz8wGD+teNgdKrI+CigunbYfKhR3HmoPmB/ocgAtCFWKHRojw1Uz261YaJQXrZ4+69PcO9f7rH9uEQ/E5i95Jf8+iMz+qGKlY/gI6BdEOB0MbTF5bwu5MDPQTc79NFlOx8rC3i/KW8oTds9kDj6fMD6YwOzD5hcWsjB4/W/kWPyKmBy6bB5pHHxf27w5g/nmD3ntSQ8MH3VY5hrtEs5SvOSl23+ZYPV9yoC2YED/vLaspOuD1h9ZlDcejQnEtU7lsYvng1wuYTLBHYPJZZfWtz8QOPeH3XYfphBDgHZ3sMrgfJdj5vfKTB9Q4bF7D26hWJgy95h/bHB8U8avPv9CuW1hy0FJm8cmlOFfONhizjZIoHJm2H0x939lsHpjzv0cw3VevRzhWzjUF9o5BuP8k2L/QfFKO/b3Tco1m7swaOE2KM50TxPlmBvmAgsn/SR8VBozjKo3iO/GXD7/QKzlxb1mcb8eYftoxyqDyivBtx9N8PySY/9/QzltUW/UKjesoaiuO7RLwwg6PVbf5xDdzwP2caN4GDzkcHiaY9hpqF6lteXlx18rnD7vRyTdw7ZymKYaZi9G72BzQl9kOU1z0tx1aI9L7D5UGP+3AIeyO863H5/At0F5HcO2WbA7lGBfMWKBzl4tGcZsrWFaiz6RYbdQ4NsyyAbBGD6izu0jxYwmx5BSeweFZi86WCua6x/sISpI/tnAbO1UC0L6ZlhILnszqFfGgKR3qM51cg2Hqr36JYEPNnajiCYnYc8Z8WbHQBg/8kceucQtER7pDB92cEbif0Dg/nTBmrfw5dm9AIKF7D5ZILiZkB3pFG97ce6CluyXkM3FmrXQ3SOAMN6NA8mkDbAbHo0FwX03qF4tUH3YA5VW3QnOYrrFu1pLzKLDQAAIABJREFUgeK6RRCUZGYv71B/9xTFmxpyW6P59BTeCJQv9xDOYTiuoNcdfGUAHwiSjCI7JoHmwYTrenKJ/pMzmOsadllCDI7hNAJQHT2Oes1ANWFjqExvx0G+n5J1648LDBOF6ZM1IIHhpEL2ag0/KTAcFzC3LWTdwVc55L5FKLPRH+lzTX+mUVD7DnZRIntxg+7jU+RPr2DvH0G/XSHkGfysgNx3CErBzXMEI5F9fQN3OodoLdwsh77dj/7B/nwKvesJ5pSAaHokf6EvsjGUJ9WCpBCe5MtMMtHhwTH0ze4gK43Mm2j72PGoIdd7hEkJ9APCpIBc7Q7AbrDwswlE14/SWmSGtR+7GuHiBHK1RZiUEDtWi4i2R5hVrBbJM4iO2x4y9kKKuw3CfAqxbwg4k08yejH9bg85nYyy1LErEjisJwQyhVoBXX9gBhMDmIDme52boR9iT2V+qBMB3kt5/ebrIaIs9v0kWOAQ/pOWO3wTyIrxvQOEUviVDyHH7f3Weyyzi/AH9/75r325P3zx3/7G9w34lgPLk++fhkf/5X+Cs8ke//bFz/CjzSPMTYvbvgIA3Cs2uO0rXNZzKOlRqAFSBGz7AifFHk/uzvC943fovcKL7RLdoPFvPvwS26HA1ub4/Pocv3/vBZ7tjnG7r1DlPY4iKypEwHFZ4/VmjuOqweV6hmnZQUkP5yXW2xLTSYsPFmvcNBXaQWNwCo+P7vDzN+fQ2kPKgGXV4Go9xelih8EpOC9glMfV7QxZbrGYNMiUQ2s1jHJY7UvkxmJedNh2GQanEILAyaTGXV3ifLbDly/PcHq6xbYu0O4yLI/3cJFGmuQ9nJeouwxV3mOwClXeQ4mA19dLHC93uFtP4O5ynD6+BQAclzV2fY5tmyMA2N5VULlD8AJF2aN5O8X971zhek25aZZZWKvQbnI8/vAKz764wORiD6Mc6jbD9+9d4vN353BO0FtqJbRx8F7CO0G/Z/SNChFgrURRDNi+maE4bTD0Glk+oK0z5OXAbsYAaEN/6cVii9c3Czgr6T9XXLbW7A/V2qHvKP8NAcgLelcRAJk5mNyi2+WcAS24n0IElJMe+3cTVGd7tHWGyazF9q5CNW/R7OnvDE5ifrxH/YslxIf70Ts7WzSwTqJrM7itoYzYSvZDeoHQKkB7fPL4HZ4+uQc5G7CY17h7M4esLLyNX3BbzaqChQWcwNH9DdbrivtlHKXCKrCaIn1n5J6pvr1CiDUsUAFm0mPYZZCFg99RYgwB6MLCbjmIyxYdhusSaidhTwZgoMRUPGwglYd9UyG/kuh/0KCatNg9nwPLAepNDjujL8eso7dzPiBYSQ+sAGDFgUHea4ZcqXDwyG40faReADKGbXX0K7oFB4xqo+CLGFJ1o9CfWYjSIdQayDz9mLWEOx6Qv8zQ3Ru479cGXrMzU9SKHtpYnwFHX2f2xtDXJwNC7pFdariKkywpPTYowM4c8iuNYeah+uiJmjgg95Bbnge9j1JQyUkenxPw9vcHTH6RoT+KYUMqMqjHlH0FE6B2VDOkdNdhSmompcP6ykHu+YWcvJyJaU5BWbqOk2glQbTeCbRnTCmefK1hJwHdhWWlUFxv+VqhucdU4KD4niS97Y+oGhCeUlkVZa9B0ouZjmVxI8ZQsX5OSWZ/6lBcavot7eHYtPcsineaP19rSmsF5b+p2qM7CqMPTg5UkQhHlrE956RYd8x97449srWMsr84ORblyLoGumNOrLRnB4kyfao8zq6grFQ1GJlGSIzAGeCEjak58ZFkoUGAntYsIF+RGUUgE+dyThylZbiSkyDDjNeTrSLLvRZjQJMInOgIOv5MpFL0SY6VEjay1xnTd7M1xjTfxHJ7DXQnPD664aSSbuN9QnAfknRTWk6K9HNO6OW3h3V3S04+yR5j0JWuyfqKQIbUbKhAsAWZ2nG/HdlYO6E8tV3GuhdFJj9NlIzS6Z4SV9nz/2RPY1iP5jrpjSMjrvpwkM7G64Y+XrLfDB/i5KEr6C9UPRnocZsrgeImjMcl1VHYSjBYah3QLwXKdyFOoGHstmSNSjxH0SdpU1VNn0KUyBAnKWU/40Sb13xNqovJ9pyIydeBYVKFhCvI6EobA6PidctalDBWlSRfqm4C8pVFc6ZHKXVieNXABOUhsqyJYbYFGebkEwV4jQ0TGVNt2RXqCslgpVLGmhUPG9Nfs51DP4v3JRvGsCxBxTyZ4ZsB3bFBedWjPTHIby1cSV8m5ef0/HrNJN0gEMPcKDlm6JAYzy3ltH5kZoOiJBcxwEsO9D0GCejajaywjymxdqIpp5ViDGMKcZmuoPRWdpyIED7EuiIygJTKenijYtBTIFsaCPgBAlrZDHBVNnoyZTswrTV5KaOHMghBkNzbEQjLXcueywR6E8gE/qqEVQq+rx0O0tpUFeL8N1//PkuZPI8pqCiFAr0fYvT+TwDBewghCPgSSE2g1b8XEgTEoJ/3JK0yyWbdN4HlL3svI3MKAL5t8f/28f8Dy//vH99qYJl/8kG495//p5CTAco4ZJnD4+Nb/OTnj/j8cYNuVSC70vAfNwivC7iTAerGjAEbduYRMo/qmUF/FGBPBsjMITiJ4Dk4yt5QUtKfOMheQtX8QrPHA9RKk+GIX4R6z+f0TqB9zEjH7FWG/pR1H9gY6J2E2fCLtHloMf+5Rn0/wM4dFj/XaE85aEyM5hDDMmyU9okAZN/bYH9bwlwZqE6g/aBHHtcz/VKhO6GUsX7o4XMPs1EQAwd1qia7JWMYjTdMd83vAlY/8Fj8XKF+wOvCFfShqYZVFLtHHrNnEt0RZZfNWUB5KbD5wQBzq8cUvwBg9jXoIRro5ZI2+saOKWcDGKZipwFmLdDcdyjfKGQbYPuYjCW/8DlAyDZkXbOVwP6Rw+SlItO6PwzMpAX2H1uULzRnVGc8ZosvA7aPYk0D4hdxyy+pfkkZKBN4OVjzmgPR6m3AMBcx/ZQMpyvJxELw9d0RB6i6Dth8B7H6A+iOOXA2e4FsFdDcE9A7+ov6JV/TL4DiKsCVHEi6AqMkLI/F98mDZXaIYSFkPF3BFN78Jp1PRPkgxgCJ4oYDnd1HPNfFLdAtuO3DnANM4YD23KN6JVFeB2w/TLJDYP+IxfbwHFz1Cw7w6vtMz1UtcPTE4ub7GtIysfT2t4F7f+TRzyT293kObQE0Dx3yd4psaDwPsxce+3scOKVACeEC9g+Ak58E7B7KsXw9BbYUNwHrT8nQL75yuPpX2IVWXPF4pnMse57j4i5g9wHBxzDlfuWrMLKzxTXlSs1Z7PkLiPJieqyCBOp7ZDw3jxXy24DylgxYfU+OSbfVJVk43fA4NRdktl1OJraLg90xOAPAze95PPrfOHhafcKy9XzNbdENxmTI4paDZXigX/KcZWtKSZNMNEmkXR6BhGP9xPoTgsXTvxzQLRS2jySOP7c8P/ckjj8fsD/XaE/EWAKvWtY/3PxAY/GVx/5CorxOsjyJzace02cS+YpAwGwCihUTVwmyDmmw+YaD19VnCmbD7SyvPJpzierSo1skySzrd+r7ARd/TP+cy4HZc4v2OKZgF2JUe9BvxfVU1x77C0WFwgSYP/PoZwKmxig5lQP3LcmJ+zllruvHGpO3lEb2cxHPAfe5uqY8VFom30rHJNaUZiocZaPtUjKJWAu0x7yX+oz+PhG3O1VFFCtPpqinvLK6cmiOFUzD693sAqavWWafgmaYksnvhGzrxwCTdimx+j5YYXPnoVpeX+tPFOZfezjDao5+KlHcOtZy3FeQQ8DsOdlQ3XqYrUO3ZJUEAlDeWIKGPf/enEpUV/QoLp+2uPusgG7IJjtDhkR4MqftkUS2C5i8Yon9MFNojyjFFZ4gZvKmRz/XyDYWzVkGrykjze56rD+rxrRbb1jtIUJAvuZxlzbAbB2GmRoltAloqtjfmDyPIoa02ImCLRja0k8kll80cLlCc07krfoQZaMECO2Rwux5x8G/EJQySsArifqegW6ZtpvtXARwsa+xI1gsLzs0FznMhmxre5ZDtR66ZX2G8AFmbykbrS3aYzK8/UKTZUViZnt0JzlU46BrCzs1GCYKZmtHsDsmigZKXLuTglJXIzFEoJRd7TGcVASehUL51R36Bwuy0pc1hiX9emrTA1rCG14HwnnIzsLOCwJSJWBLhfy2g+wdutMSZkvg4goNs+3RH+UwG87+2ImJvY7xntwSlMmOvkW9btCdT5DdtAy8GSgz9VVGINgzKTYYCbVnSmuSkvrCQO17iMFhOCqhImAzb9eAEBjOZ1D7Ab4yXF+mKJNddSMTK9uBYK3IgMHCHU14r9h3CJk+hOpoSm0hAdnasR4EWjGspjAM+3GRrcsM4Dz8NIese8C6g3x124zr+4bcFCDz5zzEdo8wrSglTQxmBJIieiND3UJMSq4zpbO+n7yaQKcPMcFVHQJ54jK/kVT4vu8xxPdoDaE1QttS5jq+wJNB9I7M5phQLA+psCmYx5ixZkQodWAs4zL+ihT2Vz3SOsMvgeK/5ePvDLC8+Ge/9uX+8OV/9xvfN+BbDiwX37sIf/g//Qf4i59+CLXoIUXAx+c3uGtLrDYVfBD49N4VXm/m2D9b4Hd/7wv8yRcfAZ2C6Fj3MdzrgVbh3/39H+F//Ze/C3Ovxv2jDTZtjrY3eHi0xtMfPYSvPFmmXuG7j9/iy7dn7IPsmLD68MEt7vYlUzjnPcNcZgOZk/gwiw5SeXR3BUTuIW8M3MRjdm+L3as5qvs71C+nCIbsy7/6b/0E//tPvgv0ZCxC5iFrherxBs3TOT76R6/x7McPIK2Au+ggbjIEHaDPWgyNgcpjYtld/PtKwS4cZMdSdFvF1ExJk39KHRVV/KDfGvjKQ28U7Nwhu1IYPuyY0GoFpPE4O9ri+kfnDGoB4CeOCZ+KNSkIgHvUQlyyQ1F4oLuwULMB8usSw5GDbGjed0cW1ZcZmgcO8riD2xpAB5hLfvkP5wPEXkOetUxW7SQZnflAX6mO6auDwOSlxO4jj/zBHvLPZmjux4TO43jTe5vDnwwonuSwk4Bh6aCWPcSLcmQj7CTKcE8d8ksNO4mJn2tKiMw2+vcGQbanjsEtjxqOey8LzD9ZYfN0yfCNTqB5RK9lKFxMLxX0s+7kGFYCkGUqrpiEOc7SPuA55o7Q15ldK8jv7eCeTOEqei2DoHzZfaeBelpyOyugO7eQDbstu2MPP3EoXhuC439Iv6YrefxcSZbLa67b6+hfPCfQGpYOxTtNduqsQ7jMMf1aoj1jd6Ps4sSIZDAPBKDWXJjw6YuMDF1+IzlRUwbYhUfQHnqtka3IZHkTeI1HhivV/9gJfZ/Vc806HS/Q3RtQfW0g+4PH0Wzor+zPLKZPDMFHShjdi9GrqDou2+ecUBmWDuZWQTcxvbYII+vnSg+94+eI/siA6XOB+gEZj/kXwOY7ZEp4zXBCqnlokd2qUSKfmLB8HbB9jHHCR3YYAdD6d3vkLzLYqWeAUSOQbTBW4SQWqXncY/JFxlCimBKq94IsW85rGcAYwNId89jmtwL7DzwWn0c5+UnsCs3IpqVtbO5FFjQHumPHCbItmSqfR+Bl+Z5+mSTXgR2hF1ymbqKsOTJs3TGPaxaZvbRP3gRMXxLoARgTX/MV/Za64WRFtiEYcJGlcbEGJ/lYm3OBbEtmzmwTQ0h/a6oCUi3DZnTNjs/yjRwTQFOoj604ASV7gekLTiCpmsDV5Tj4Nc94PMgwYaysITMSmU3N/szVZxKTl5TQF7ce/VSMHZn9HMhXZI8S8wSQlS0veV8dpqyOSXVICcgHLcbJheSFJRvI64tVGvSHqo5JnP38UAMEgZGpFC5eLzVZvfaU65Z9DBKKHlJpKc8OiizP9K1DfaLQXJD5U230jW5Zl+LyaBlpOWmUrwPqc07sFDcew4TnGCKC/T4cqr9a7md9IZFtGEyTfNfSHTo3izsC9mEqRk+rqfleLp/MnMsFpm8dvAL6KW0kw1Qgv/OwFUNsgmJvJfsuA5pTTrSkh24C6nOJ8iaMbHRQGNlJNQRka4fuiFL2fhqlmTaMKai2ZGiOdGGs3QHICA9V6t+kLN7lgoytPEyaiMBjkABDkqpLG8Y6lVTtk3olVUevprCpUknGz4THMFMobgbU57E3eU95eXFrYUsF1Tr0S84iSxdQXPXYP8jpH105+CxePzFFNlsPENZjWGQxVZc+2cmLGs39ktUuAvA5JzRczkkE4QNspcbJTb23o+9SNRau0vG+FmLKa4DsY2VJzsAe2TrodYP+fDpO1nkjoXc9ZDvAzQpWrfQuVtMw8EevWnoshYAv30uFjZ81vW7ov5wXfG8A5K6Fn+QQvYWbFaxGAeALDabBtpQiKwXhPYazCcx1TS+q9/Rp1h1BYOyzlNtmZCrt6RR61QDWjYA2SMlQHiW53C6Oc4we2Uaxj8sARu9lSIAQOIDWBDhTyE/yQsYkVmEM//Z+YE/dEEwnAPue3DXEahMkNnIY8Nc9Qghjkuyven78/dueCptdhD84/6e/9uX+8NV//xvfN+BbDizLTx+E8//iX6A8anA8rVH3Bh8uVvjF1RmGXuP8eIPLmwVcpzA9qrG7rSAzB2UcymJAP2gMg2JVxW1B6VrucP90jVcvTiA6iaOP7rDdlRh2cUDvBNROwp9GaUMAhAzAKkOYWEr5GtYxyMnAipBVAVVZ+KsCWDBtlHfx+OWyV/Bzy/8HAIYSuiBBELWjvDEF/ARNMBUCmD7aC4jjHn5noOc9wpviwMhWlPEliWF1vkf79YzrU4HAGOCAf5DAbABWGWVjF9GsrT18E2/eNYNyhmMPc0vgoxrBxNBGQe0l7JKhQ3KrkK0Z8DIcURqobgzc8QBzyQF+MEzsTPuVALCrPMxWMlQn8HX5pUZ/4qA3EsOphb7V9Mdu1KHbUgFuaQErWD8RwbKq5Vj1oLpYybASqB9Z6J061B5kMeZchShhlBxkHTHgqLiSqD8iO2vnHnotYaceuiZwsFMO4OVAhk5veI69CXCTQ9CQnYQxBENvBYIhCOuXHvm1GhNL9Z5prtkt6zhUzUFT8go3F36sYuCXMj/P2UqOsj9vAoYp00wJnOj9VQ1ZwNR16mMFCTsqCeRcwfoPs6OELiWxCk8/bH4jUb0N2D8Q6E48Ji8kmguCCa8xAsZsI9CeUlYJUPJnthL5HbB7RBAbdEzxHLiu4jrKKG1kIZMscOAgX7UMyanvcVBUvhVoT8MhIRSICZkc/JdXQHvMY15ehhFE6S1ZrH4RopyQ11HxDvQpnTPwpXolGF5zI2LMP9CeRNYqelXlECcBqMbnBETsGnUlPbrvh/+koBxXxMF9TTDTnmG8JrM1X5+tec10y0NQSkqsFV4guyNzj0DfdpqQ6GcccKsOsS+RzyeZYJKbNWdhZNfbE/aorj4DyiuBYULJpIkdkUFilNulwW0eWdV+Htl1H0FfwfqH7ugg0VMtmf5sHaWOseLC7KLn9yVDdXgth1E+1y2jD93E3tSTqAKIYDExqAlcZitK2UKSi7oQUze5Xfmdx/6hRH4TYo8kQU3qIM3vQpQTE0ABsYbBhbHKxOwO6awAj6srCZzMNowBN2PtRh/7TTeBlR13BKTFDQGf1/QRdxFUJzWM2ROgDKWAaSiF3D1QnGyJDKiuI1gqCIhsFStXNJlLl4kIKgOybRgDa4KkNJFSzJg4KjB2i9pKMPCnDVADPcSqI0vtjBi7RXVHyeP+QqK8DWMy6lAd0mQR6EkeJvTJShvQnGgG40igPWYAEdlogrbEeCWGUg4B+dqhOSWwEYESUN0ysGiIgUaqPwCu9LnzmmE+KZwoATX2HB+AZ3F3SGJlWA63tzlR4zVbrOj3TVJMht74UcqagKMrRAz78pFxdFCdR7+kr3eYSOR3ZCxVF2B27PXUe4vmXg5dk311pcRQSmQ7D7O1sBN1kGt2Pl7fapS6+rgN2W2PfplBN27srUyBQdlmQHdkYDb0xyIcOjD7RTaCOVsq2AmBWn4zIOgE7ggCXaEIHn2gpDSG/nB4QXAoHD2dqrFjyE57XiG/6+BKzd5NIWBnBtldB1foMXRI73r4UkPth7Hv0ms5ekSDEpCdI6No5EGuqmT8TufzXvM7WXVMbhUD3wPv4RYl01kHB1dlEI6VJQzX4aSD6Mm2ql0HXxLYJV/q+9UgSXoaDEEq+oEeyn4gIIw+V9EOBICD5c8UiKNTFYkDpKAXUyuCP2AEhsLFbesHhvYADACSkn8fLFnNLALCBPpSRUgCj4m9TJjAuYM3M3k7AYQEDCNjyZuU4u/OfTPZ9X25bPR//rWY4/9phyWAvxc9ln/PgeW3um5EyoCLh3cotEWhB0xMj+tmAiECqqqDkR5F2aNXCpm20KWFNg5FNqDKe7hswKYu2NFYOUyXNXbrEjfbCdRkQH48oG5zFGUPqTz6OoOeWgyZgVQeWWFRZAO2+wKucEwjVQGITGGWDxgGBV1ZmMyiO+4QvIAuHGyjIUuLEAQ/UzJAJD+fJpsEGfh9KgNUaUd5Chp6DJttDlFaBKmA6MHzXsBPOAK3M09QmT6zucfQa/iZg8g8QsdENsQewlDSo+hLBxdvvsEKBJviCwOgJIYFwaAruG12HoGrxNiPCCfgpw49KA2F8YATcHOCPlcE+MpD9AKh8PT+AYD2cBMxJs0iACHjzak/cgjG0xfTEjgJK8YON1+k7qvABG6Fcd+DAkLuYa517CELUT7LiglkoLdMRS+foydtWFD+HEoH1KzVgGNvoYj9fEFzvf0yyrd0DNFpxNjFxW3kPg6zWIouQVZPc3uCokeO6agEW3YSAP9e52ER4CIDkXxsLLnnMQiKx8SWcUAVtyX19vk8xv/792TBKgLfyLwgBjbBc3sAslZeAcomhiyBUA6KQ6zsSAFPKWDK6xCj5AnSE/MnLXs4hykHgC4/9BsmObAt6VMzvYBTARICiINsrwAVXyPA7SbrJMbjKzxZ08S62gKjJ8+VBDI853FCtRXjxALDspLPJ3qzyni9GIxVBqmHLoVVyTg5mwKv0jkA4uDW8HyzH47rbE8iII1A0hUYwQJ8/D36vhI7OfYWIvoq3eF6l+6wzz4CQNanxH1VBF2pUgMg+JCDGK9JaQ9JrbaM948ItoTHAdjiIL31Gf/JWFMSJOhjjtJEDtAwvj8N/IMgq6pbrlt1PAZAvGZ1TJUMcVLDBkAf0n29SawWE0HhCaBVXF56+BwQLSKjT3AyTMQI9pNnymvKOnXD56XjNgYRYqVIGCc7vKFMfrzPyLivkemDiDJYiTEhlpUHGPsuXRb3U3GbEMTBr6m4HhXBISXGEWA5MnlBCaRCeltwhxMbl+4PuiPrFeI6hMPYm2jz5A0M4/sReMxdBCjJr+3y6E9r+Bnzmv42RH+mMxgTXL3itSUcJz5YGyGgBozewH4qke19BABcXkpEDQKQcTIpMfBjP6SK1RtIx53LZcm9h7QHkCx8oFfN8dgKQfbSl2JMuU3AM+2HHIChSvJmQORkY70WI0OaqjZ4/ONnXwakpGJewIdrMF1bqmewjivi93nghEHQkaELAT6Th05FGw7f4fG1aXlBCo7iAhC0hFOxC9K+N1APgKt0vBdFT6YW47q9kvF6FnBSxs+nGNNUheX2eCMPE7CGYBqIywKPB0G4G8Hu+x7G9PkAAK9lnNDSkIMn2AuAN2oENi7nBD0nEiJw84ArzXvsvIR30TMYS6R93O7UbykGBxhFy58PkNaP8mGvJYRInx9N1tAFJrAOblwGoseR12bsvDSK4DFVdYQ4BlICSKmvNvB97wXqkIkEWUT33nlKoM46psBGBhMhMFE2yV3Tvik5MoyUx4YD6EzXfGJX3/dQJr9l6p0cr5NvgruQZLS/7HUcX/fe30MEw4KJc2OHZfJFysNn9a8Fl++HAP1tQOa39hH+yvH++/SQf/NL/u4+rJW4fHGE17dzPLs+xtvtDC4ISBmwuZ7g+dtj1Lsc4skEt28WkF+W8F6gtwq32wk2dYHmjqOm7HkG7yWwytC/mkAAqDcF+lajbTJo7SFuDYa7HOWXGXyrYQeF7a4EgkB2rRE2GZyVsFuDvBjQtRmGVQG7N+haA/WigLzO4K5zoJeoph1CL1G8NhAbA+w00EuY3MLMO5h3BnhZAoOEv8sRGgUEgfKVRvjLOZlRAcALYKcppxTA9KkGTIDeSohOMsBkpyBXGsM24zVda4hWkqUtUtIA4GsN0UuGpqwN5E6j/CoDnICoNWQrUFwpFG818jsB1UhAB4gtw1eyOwlklGiK3CG/paxM3xgUyxYi9zB3GsUVL73yrQIGAdlKbv9eo3qp6Hm0AnonIfcK5lZD1RKyVigvJbI7Nd7QdS0YYNILmI1gYIrxgAzcjoGBL9k7DbOJvXWNwOwZYt9bLEZvuD61l6heKIKqOLAStYJqJWs8JIBA5hOCQCVbC3YF9lyeCJQ0FjdkT7I7CdmKyMgx6CSFciQJJjyZRt2QGU2SNLPlQK18S6CZyuTzW67D7OIAKhBk6D1rRUY/7podjdkmDk7joLt8GwfhhgM/3QgWsg8C5aUYQZPZgaEYLSs/+iOH/I7rUY3A7IWnJA+IhfCRvWi4buFTB98BPIgBkB19j0Ew5CW/JcjUzaEzUbVM8tWxjzDVbAAYX2vWYgy34T9uW/JYmh1gNvRVpl5D2ZEt03tuqxwIuqWldLK4ZkCHtIivjxLAuE7h+B7VHoBiSj3W+3geIsNVvQ1kG3VkDSXZvSQbtRNex2Yfu/JqDvzNNjKY0f+l92Gs60iSYDnwp+opV6Rskf9PD+EOlRfZJoxS1GxLRm+UakaPstkhTmJwe82O57W443aqLqC84nbmqzACTID7ldJ4i5vAMBfHwbCuI8hNwCFKOKUN0PsQr8uA6jIgX4dxgC8t3+tNDBRx3AbhowQ0HIC0iTUf2Zbe4jIGsIx51NduAAAgAElEQVRsXsXBbr6mPJPXZGTiejKa+cbHCpcoS5zxZ7bzBO1Zur4Dz1cTfbvrgGJFCSIQgdHAdas+INtw2fSucRuTnxog+Mu29KNymw/nWHg+X6wcTO1h9uxCpOTTR98yt0tFj6jqgeTJTMdFDjw3BK5hDJFJID/bBZg9X5tYN9WTpUzLKlbuvUmBgwRa2v+bvfeItW3b07t+Y4wZV9x57xNuvu++WOVb9VyuJMDlJHCDZLCAFlhCogF0oEufpkWDICO6FkZCAgSyMZSNyraebJdfvC/VDSefs/NecaYRaPzHnOvc57ILi0JV13hJR+fsfdaaaY011/jGlwKmkzRi+XxKaIxp+/2EoRqnD41xeWSzZ8IcJpUfWMp8FQafZH8NkzouNkYPpYC8fmLPsIjRM7j9+DdtkLAcLden79zMVj7KLaG86IYxmG482dJhKh/BhUzIBQTJvrULcTFCdm9qeW9dpjGtH6SoygXStR/OmUAEvwETmUMQ8NSPB7O1JLWLQTtyHzetH8ZG0NGDuRT2SLeRaUx1DLSRsdwDW+k39fEz4zFbO4BV00SmEki2sSLDyvOro4T0tsYWBtN6slXcXwx2STYWZQOmE7lwsrGYTRdVAgqzlYRcU1tM5QX0+kCylhuU2bbCLIZ+QcZHRlO6JFXn5b2sRf4q1R5yrsr6oS9SvQaedKz56MGsigyasLqWYMTjqW/XskgUAliPbjoJ2Uk0uumGz3D/elT/3R8imPPCUPZeQrfbv2peY9Re75WsG2EPNcIc9r2Sfadk26eCBWEaozez9zCjtTwnhM9XifS7Sszud97v2EmlBjA9AMU+aEfrHfDrAbDaAb3Qg9X+ea+DZPg8A9r/6R/9z133OZnr5ySvr7/GOQGp/5g/4f8XwPOL/fhCS2Hvf30v/Ed/5df47vIhHkVpOn559ik/3N6nNB3Pqz1K0/G3n76DMZ6jyYZ1k5MnltQ4jss1R/mGv/HoS8zHFfcnC35yecJvvPkx/9u3PoSp5S/84t/mop3y9y7f5HY1omsTvvzgnEnasGwLbqoRV1dTDg/XvDG7ZZI2fPvVQ5QKJNrzK/ce82h9wLrNKZOOdZfxlb0LfnJ3wqIqGOctB+WWTDuuqjGLquBkuubeaElpOnJt+d7NfR5O7vjo8ozEOGZFw9PLfX7l7Ud8dHmG9RqlAtuP58y/fMM7eze83Mx4+Wqfw6MVq23ObFxTtSm/9uAz/sYnH2ASj3OKouhItMfowF5Z8fx2jjGeaptTjhrKrOPmbkKSxqTUVpPeGsqv3gHwxt4dP3p2RlG2ZImj6RLSxPHewRXf/ulbvPXmFcs6Z1sLON8uC8ppzfZ8jJpY3jy74XI1JgTF/mRL5wzL3z5i/5cuOL+cExqDGVnKUYO1hiRxFKllsSrptinHZwsuL2b8+pc/4VufvTP4u90q5fiNW07Gaz769AF7h2t8UKzXBWGRDfJbpp1MSFqDLi3jSc3qxRQ0jI43hO/M8T+3otlkwgpbxS98/TO+/dO3MKXcDNPM0mxTQm3QI4u6yMErxl+6wzpNYjybT+aM3ltQVxn+RQn3avxVjvJKanEOauxNwfj+iu3jGX5qhaW+zdl7447bF3PSeYO9KUQKfNpJeiriXXR7lvfffcVdVXJ1PkPFlNBQepLrROTJWthPdCDfq2luC/TGEI5a8W4eNfhNSnm0xVpZpGhvC/ILQ3O/QzWGkHhU4dAXOfrhFntToDqFn1hO799x/YNjeFhJqmxkwk1p4aU8T7+zob0thiCrdKFpH7SDJFu1slhRfLBg83xKGFuUDpKaGxSzsxXWaapXE0IqE7n0JsHejzrPVYJuBMy6whMmTiTafaptZPaThezP5wE/lUmWSgKh0ahOYw4a3GURr6EjuU3IbxTbBw5TyzZcHghTi75LhDVEvKdmo3ETJ9daCeM5emKoT8Qj6QdGnKHzUxJCA27iIRF2X1kNTpje8FaFPy+kr7MMoKOf2Ev/q14klK80m3csyirMViqXypeG6oEjZJ7kVmSDbuzRtSJbaNqZhDDV9+Lr1hp7YEmvJIypORZ/bHZpsBNZOKofdqhWDwC/uDADYwOwve9J1mpg5AR0qzgZDkPKbHNsyS8T+V2taPekjzRdx37dXBiA8qVme9+TruR19l6LucgGBiu/Fvay2QtMnilsGYOzvISF+RTat2uKnxb4PGBHIb4PcSFkLa93RawR2ouvOZDqpnZPjll38pzeP56uhPksrqS/M4vhUNKLK4sYpt6xNKaRcCndyn63bziKl0ZktzORWG9+rmb0w2JgtAfQWQvD2c52IWj5nfRV+rgw1I1FPl4fBfJbNUilu4lUNblC2NFegm8q+Xd+J8ffnDpGTw3FpYSLEWD2xHH3vhkWAdJ1oDmQ/3MFUhM0letlWgbJd69c0G08h7hI0OwxMOtJvetWTdbCeFUnivFz2f8guw+RFW926bt9z6huoZupnc80wOaBeHZTaRSJTHEY/LrFjRyzCnLuSeXZnsZOzAi8fcKQ+jqoE4yEeCVbWaBp9mTxa3ThqPf1wFTmCwGfLhNvbLaMfaK5HEdQinzpWL4lHZnpRtQUysk+s+VO0pxWgWYmXl6ILHAP+nMGOXa+CIP0tt4TkJ5Wu/7NXhmQNGGQ6vbdrCpAM5eAsa7sA5CEudMWtsea8kaCoTb3DNkyDFJ45QLNXFPeOGyhaWaK0ZWcf7pyLN5L2ftYQsNEBiwsXTPrtymA1LSe+iAh3fh4b1UkW8/6XkJ5I4w2Cuo9w+xxQ7OfDp2fpvY0+0m81wSypcUbken25wHCEKcbh41AvrhoBskuevf8YISlbOcJ2VI8l6ZxQ+9m0ApbGPJbqe9RNtDOU1kAW3WoENB1DB3KJG3WrBr8KBvAr08N2u4Acl8P44sUN0qHvky5ccROzX6u7uPvNjXdgz05rtSQLCrcJJdFl2UtElulCHlknusO1bSEyUjqVWLPpJ/HfvQecPaLKKtqqFDhde9lElnltoMk2YX19A/nBLS+Vi+i+soRYz7nsXzdV/n/CIvEtNkvvBQ2PQm/dvRv/r5v96+++i//wM8NvuDAcv8rJ+Hf/cu/wffuHvDze88B+GR9DECiHWfFii5oFl3JJ7dHfHj8nMYb9tKKV/WMH16c8ZXjc26bEY1N+Bfv/5Dfunyfx9f7fOPeSw7zDT+4ucdxueEw37BxGcu2wAfF9XbMe/tXPF7uk2rPusn44PCS2qa03nBTjTgabXh6t8eD+YKPXx3z6+98ym+/eijyV68IQXFvb0mZdNzWJfO85tVqyv3Zkq/PX/LXn36ZzhnmZc35zQylAm8c3/Lo5SG//v4nfOvx28wnNWXacbGYiKRXQZo65qOKIrHUNuH8ek45aqiqjKLo+OX7j/no5oxtmzItGowK+KB4cbGHNoGDvTVVm1JXmRBpraGc1vKz1Zye3vHq8SH3376ic4bOGpQKGB24uR0zmdZstjm+NfzGV37Cb/7oy+jEozScHi548XKf45Mlb85u+QefvYlSgbfvXfPpoxOwGjWyUueyv+Hq6R5qZAVgTjrcJiWdNtjrkux4S/dsTPHWCq1lHLdtgnMa32lM6rFL8caevnnDxdWMJLMURYdRgdWmEAmuguCk8kNZLeCpkYnC6MEarT3bTSGy6tZg24T5fMvdxRSVO9lGUOwdrllvCpzVaBNwG2FOZ/tbVncjAS2tJjvdCkgHqDXFccXhdMPlYkK7ySimDc2LMey1JJkj+WiM/caabpOJ3FoHXGtgmcC842tvveSjz+5LWNPLQlJ8Dxv0pyXtWYfOHeEm5/SDS25XI9oqJWwSdB09qCagtgZ91BAucnwf3lQLYPAjD5kX3+rGwHEjtSHLhDByJCOLrRPMTYI/adGXmciSjazOq07BzFL+JJfk5X071IQUpxvql2P0YQPPS9xRSzFpaV6N5PhKL/7YRKTT2bWhPZM0ZjfymHlL/v0RzZHHjT1mZUAH3NySxHAsAugeDO5ZkttEKk5WwoLbfQGW+XlCNxPPpc/A36/xnSG5SqNMK5Dfato9j3m4pTsvY6hWQDUi/TYLqUjxY0d6k5CsxXvq9zuoDaoT6XdfuZJdGUmM9or83FC/2aJqg9nqweMWlAQAhUzGZci9PCey58JiikdVgpoCzZkVgLwWiVm6itUkaZ+q7MVHO/HorVzn4pWAx15y2t4TL7QrYsfuWlGfeopzjWn71GNFN/Oxa1eSm/HgRiHKXtUgD0y2Akh8xsCWtAeyPZFSi182i1Uh+ZXe9fXmAp5M7AdOaoUt5LkSxuQpXwkALS40xY34RO0ohlClIsuWiagaQIpPA+MXAhhM3UtkoTqT7TWHIqvsq1ZcLmCoPwefRTAVg4Rctvt+6hOtXc5Q8eIigMiWEi7Vg2UQr6xpFOkyhg9Fv2ifANzsK7p5DFBKBWCVF1FqrQW0mRj6NH7hWbynhz7abCne1aDFD+xTAWFB7cKZepl0L6fu/bhJJdckaAGN7VQSu/O7MAAtCXwRr64dyTVKNoHNA+mGFXmhGkKR7ETApCvkGpYXAm6TrTCJPlHiXS2JSaQC5nrVoctFglxcCwgMRpGuxDdt6jBUq/T1Kb2/tQ9raqcSQpVswhC0ozupHAEJENrc16SRVU5iEFI7k4oU0wS6kR6ul+nked0ojvFY85JU8RiqQDsWMFtei7+07UObppp0G/2uKwGyfeiPCj07H7ClJqk8SR2o92TQJLUkUaebCF6jLFlHuWyfVNzLl3v/sEsFZGq789I2Mz2E7fTjV/lAspWQHdMKCLelFqa5DZjKY8d6WMQIEVwLEwi+ZxOR8J9+Oz5VpGuHHUldyTA2nCgT+goTkW3L+bczQ7oWhld+xyD5TSoJ62n3Yspv5USOnYrnFcBnO2a4lyX310nqbhTpxoKT8B4QMCq1LnoIH/Ix5Ed3IvH1uZE0Wy8sbzDC9oVEE1JNsmgIqcHlhmTVCENrPW6So7fdcL2DigAy0YP8VTknYT5VK4Cus+KVBAgi2R2AJyI9HeSzSn2uixOtd+xiXzni/Y6B/Nn6kf5nGxnFstixr0DopOcSraXnEuh9liQxNChNCJGFVUZ/3n/ZP35Watvv/x/18P6LXzeSnoRfO/w3ft+3+1fP/6s/8HODLziwHH9wL5z8h/8pe+9J32IICus1RgXWEQisb0aojeFLX3/O7/zoAemx6NPS1PFgvuCnn95jfLhl82oMpaOcNhRZh/Wa+sd7dCcdo3lF1wmg0CZgPi2w79T4ThI4pNMRgtNSVfKqkIlkUPKzV4ynNdvHM7IHG0KAZpWjEk/wivRlhn3QCCujkUln7iXg5q0KbzUnR0vOn+5DGjA3wlCpTUJ2byNsmlPglfQOnpeE0lM8TbGjgH9YE7xiNGlo6hS7Sgc/JyApua2CeUeSOexFKbLKkxrvFNoEbGMopw3VsqD8LKP5oKL4UUlz6IWZeVDh7jJIgzAuVmNGluxHYoKr36/RSSBc5Jh7W8KjsSTJaggmyMRfB0bvLdh8Oo8eU0gnrXQzaqDVkDuyFxnu7Rpe5bixp3iZ0ByIyb4PANLjDvWqwOeeUHjSm4Ru35LeJigbw1zOFdUvbglOYZ4X6FYmunq/xV/lhMKRXSWYraI+iRKttaI9i+DktCFsZDWwuEiG8BzlYPRCcfd1i9kIOHMjTyg9ZmkoLqWLzBtJUJ080bRz6KYenwaKC0N97CWMKJWuxexFKpUzE6mRqR46zEqDhuxW0e7FAI7Y7zj6NJWAIC0sGsD4SUI3DtixMGZJBc2RXOe97yXUJ0CA5sBLQNNcVoSLczmHbh5Q3U5y2xzJ88bPYHtP0bzTMP4opz4OFBcSstLsxVAcB92+pXiZYouAnTuShWF0rli97xg9MVT3JJFTOVi/bxk9TujmkjArwUsyuctvFdWZXJ/iSrG9J3LE/FrSR92oDxkSSawwAZ7JE832fsBlgclTjakDy3dle8FAfSJJp7pVpBtJFbWFYvWegNPJE836TallSSITtT2T+2d5rmj35T0dnSvWbwbKVzL5ze9kUlcfCIvSsye6he19YZN8KmCiZ0g2D/tJlshyXSbVLLZU1IeKdi6gxCe7MTf7BG6/RpQpQ+8v7f2uppbJ4va+1PtoJyFGzYFsp4o9kKYV8Lb/I7j5OZg8UjQHO0mt3GxFXmhLNbBJ41ee1Zsal+3kwqMLz/ZE0js393c+y6Ri6Eb0qbBn7R5kdwI2yqvA6i0VWTIBgfldBCpX8pz8Vp4zeingoDpV7H3sufmaJl3JORSXAiDya5Ei98E+5ZXUmZgmsH5DMftMUk0nL6VmJNkKkCuv5P1wqaSIaicgNFv0E1u5HD6TSbS2MhbXD+XeOnneSzAV1aGcT74K3L2vmT6W4KDiMrC9pyiuRI67uS81R+1UDUmo2sUxvnBsjwzFnUgxb7+UMDoXUGI68YXaPHZCLoRV023sZbwVKejmTJOtRG7cM0zNTMceVQaQ0o2kr7DZ03QjAXqja4epA7dfSiVUhxicFN/jbC2ga/lWQrYMFLeOzT2DywX49fLqfOlpx5riTvx46/sJ0+cWnyhWDw3ZKtDsCdOWL4Rl68bSlejTKLW+dWyPk0HKbQs5R9MJIxY0ZJF9BJEHN3MtLJ1HwnMm4mHsxlIbND536DawPUkYn9voqRXgYUtNunLUhyYCQc3owmJLTTvRA7hUXupQ+o5KFEPAT7aM466VOpbt/QJTx9qZi45mX1jM3leZVI7qOEM5kRU3+4n4SRupiHGFJl12NIeZsJ/LDjtO8KkmXVvqo1Sk6y+3NEelVJ8cpmQLCf55vRqlfFlDlKP6RGOqjvp0RLYQibDPDK7Q2FJTXrVImrYkswajqU4yiusOs+3oZhmmE3lou5dRXFRU90t0lPLm1zXtfk5+XbN9OKJ8UcnPVxUhNbR7OcWrDXaeY0vpgsxvGuwkJblrqO+NKF9s6PYKkk0nAC8zmG0nKa+THJ8ZkfhG5qzZzyhfbvB5IvUly3bwYyaLCpTCTXORoTrxBbrcCLjftBIYZP0Q3qPrDjfO0bVFt3YAjD5Ldj93DjcrMFcrSZsdFajOSnhP3Uni67qW3zctYZSjqpYQwaPq5axpIuE9RYZabQWwpRKqo7rIBLbd5wN6jCYUOWpTCQgtc/GGbitUlu28j6/3Ur7+szGEpkVlqQTvJHKtQgR1Kk0JTStAMs+h2zGXwcVey7bdpcFGuevnMMfrfZexSuT3lLr+01A38s+A5R/ex/TLZ+Gf+0t/nkxbzrdTvnn0lP/1t77J8Zev+JXTR/zo7ox5XnHbjPj0xRFffeMVB/mGp+t9LlcTnNMY49Hac2+64sVyxoP5glHSclOP+fff+i3+s2/9K2Rlx3TUoFTgg/1L3h5d878//wpFYrndlmgVWD+aEw5aCQcqW6w1pImjzDqKxPL4+SF/5us/5NuXD7FO0/2tQ7o/uuLDB8+5rsd8/PiUB/dveHk5Z29vw4fHL7hsJnz/J29w/OCOm7sJf+ztR/zo6pR/572/x1/+9I+Sp5ZXr/b4U1/7Mb/50w+4d7zgajmmrVJOTxbcLMc8PLzj2fUeXZ2wt79h+dN9/uQ//12+e/WARHuulmNGRUsIamDb1E3G/N1b7p7sMX6wQqtAYhz13znC/PIt60VJ8jxHv7/GWoP5rIT3N0xGNeOs42YzYnM9IrtI+E/+9f+Jv/jRn6BtExTgrHhI3z66oUw6PIof/OAtvvGNx/zgu2+Jr/LBFq0DXZuglLBz771xwaffe0D51oq9UcWLT44pz9YY4wlB0dQp41FDYjyNNbTf26N7V26Axwcr7v7+MenPLaiqjA/feIb1hvPthFfPDnj3nXMS5Vl3GRffORUQGFkhksD4/or1Xcm7b1zy9GoPHo2xZy3ltMYYYTOnk4pR3nK1mPDw8I55VvHtn7zNV99/zk+en8JVTnJvy5tHt3x2fsh41DAtGu62JXWViZz2QqJEQ+EoHme89yc+4ycvTpn8rRH1b6ywnWFvJpqoxbpgXLZs64w/8uA5v/3oTVmkeJxjp4GD9284naz45DffwWdStXH88I6bHx3iJh5VWuZ7W+6uJuAVf/znfszf/M5XSecNqIB3BvNZgX27RumAv8rRjUa/sSH5aMIf/5f/AX/9N38Be9KiU4/SAfNJSfr1JdtlIezvyHJ0uOLq40NJQ66lmgcv8l3OGvlOuc4lRCPznL19zd16RH1ZcvTmHVfP9oQ9BsLUMppXNJ/ORFZ4Gn1CTkX5aWQ0jyzlYUWSONZPZwMwTY5rumUGQUnH7U0piyAKVOLJHhXYiceVngfvXvFHj5/wP3/rm+QX0i/ZfmOLelLS7Tmya/HgBhNoThx/5he/z//5f32IelCRFx3uO3Pe+Y1H/Ph7b/LhL3zCt3/4DnqrCXkYApaKsw3towm+DLKI8SqF9zcY42kfTdCteGrLhyu21yMO7i2omgz7U5Fq+1SqblwmTKr6cIH7wZxu5lGnNeazkvt/7AWPnh4z+jgjvwncflMWRWSiGEjPtnSXJWHkGP8koz7xpG9usJ9OsIcdsx9krN5z5NdG6mAOO/CKP/9Lf4+/8nd/SWqbTiqSH0wof/mK9XcPpVf3xHH/S5dcr8Z0jyb4TOpVfMaQNpxs1SAt7cNwdCthWuq0If+oHIBze2LBQ/EipZsIyGruWcrHKeoXFmyvRhQvUua/fMHNd44HuWh1z5EuNT4Vv2+7FyLzGBg/jbUiHpZf7Sgfp/g/ssJ/PKG4jmD7nh8CrIIOhNOGMh6XHcvkXwJKRKKc3Sqab1SU35bFuXYuizh2JExjc+jxpWf244TlNzomv5NiSxi9DKzekWszfqYHtjCkDEC8OZAKnPGjhHYuC0QHP1Ds/1vPuPhf3hh6TIOG8KsLmt+Zka6EQVM9Uxw9uCgB3pOnsPggELJAeifJ1q7cBXmlGyUMp4kS12+uUN+dkt8ySI61Fb92dSqLhPatmvnfKUQOGtnabiqAsriUjtee8bVjkVvvf0/TTWK4UJTPClMktRvNn12g/6+9CAhlzrJ4XxZRRi9lm33Sb1IJW67jtWinapCCmlaY7GwRqI+FXVdB5Mw+UTSH8h6MXkSWrxZgWp0qJk+kZmbyROTS6UrYvWwVYpiS7LM6EhDes3g9A9svFAWzC8aaP7JU+yKH3t4T6a0t5PoIWwvzTzzNntSNTF56eS8yYb+Dgm4q52PavtM2UNx4XKaGlNvqUBQ4ysLo2lHPDcVCOjrrfc343LJ4Ox182srD5Z9uePg/iny+mZmB4dRtYHssixsA6/uG8tqTLRx3X0pZftnx4P+QfW7vKWafycLK6MKSn2/ZvDOhmco+N6cJ06ct1XFKUnuamaG4cxDg4hcTjr7vyBYCkG2umD2pqY4zylcNz//4iLO/29JNDPmtpNFuTyVBV3fSGWrLBFco0qUjWbcCbruAbryk2K46bJnQTbTIXp2E+7TTlHRtMbXFlQm6cbu/RyKfbecpo8dLuqMRtjBki1a6PG8r6rMx+U1DfVxQnG+xsxxT2UE6q+sO1Qg76YsEXXU732Ivj0WY+JAl6FUtNSMXS/x8JEm2iw0hS3F7I8zddtel2VlCmYFS6JsVfn8i49w5VNNJ5UjbQWJQ28j8+SDMpvsZ0JYmQyrt8OjZzaaFNCXUDaGuUUUhoM8YAZXb6vNSV+dQWfa7S2dBmM9/XI/la48vfN1IehJ+7eDP/b5v969e/Nd/4OcGX3BgOfrSvfC1/+Lfo0gsL67n/PzD5/zk8gTnNHlmKbMO5zWbOiNJHKurMbOjDU2b4N0ut8g2CffPbjm/mXF6sOTh9I7vvnhAvcxJRx1J6piUDZcXM7JRR5o6mjpFqUBRttR1SpJ4qruCdNxxMN/gvGa1zSnzjruXM0lvrRLSvRr/fET+9or3jq75/u88pJg3NC9HhKklH7cSFKQCVZXhW0M+bmURqklQJuCWmXQeTizBKvJZg9aB9rMp/lgm+klqaRaF6PEzj75LyR5u8D+e0B468b85hSqdpNCuU1RpSV7kwnLsWVABlXpYpWQnW5qbEl1pijdXbG9G8vqRJfusoD0QH1ff4alahTqtCU7hG0Nyk0olifHko47m5YjkuCbNLNXzCeqgJTjpxnS1fJkVs4ZJ2XD1fI6ymmSpGX/tlsWTOWHswCrefueCp5f7sp9VCqVjNKvl+ACzNOJ3AznnVsKFAGF5TRz/EaDoyqBOavxNjllr3FlL+jTDTgQMuP0OvUwGf5sv47UMkF6kklZ4rxFG56XIXUMq3sZwr8ZvEtmvlveFAGqdSKJqIR62HqyowpG8yqRP8U66D3vvpdsmqE2COmwwT0X+6tOAbjQ+9yivmL1zx/KTPQEfpXSW+kw8dd1EfHqmks9B+u6K+sU4pggr0ltDt++EyQ4qdi4KQE1vEukU3RrShaabxW0XIlu1c0dyF6VaMdnWTh3Fq4Ru7vG5BEv5NODmjuw8ASUyQu0kUKebBvp0yL7H0icBP7cUTzO6iUd3vbQyDKDE7lmyiwQ7iWEwlYQSyWRZukGbA49pZKLXxXoYOxGQlqwV7f6u41OkgrH6ZhZltQPQiGFCkRk0jZyDnTuSlZGkWwfZUuOyGDbTRQnsAtZveDmeQ49PoTzXkoQMg8Q0v9FU9x3prSbdKNpZGPphe+9aiKy/pIQSU1/lvOoT8Sb2KcumEfY22cj7PnqhpJs0Ed+gqWF7z1Nc6SE8pwciIJNdOwp0c6nF6X1tdiTjqq9RGXyFnUxm272dhBN20kM7EtZy86YnXWpGL6VmJF1DdSxewfowDEDP5wyBT71sVfkIXibxPcjleLJlH1YhoDXZSsVJz9z2nZlArEEQAKTjMfssSiermNQb2KWHhhjcZHZ/v14B099Tmn1hrAG6aGXSTkBdfq3i+yHn1Hv8lI8S0WR37Yf3ogqD/FKSjgVQiTQ2etstGQ0AACAASURBVK7b2FNZi/w0qSW5Od0IKGv21HDMffiTnch5mlZAWu+PbGcxLClKOnupZrIVcGXa6AVcB7qZwlSvyVCN/D4YqI4V4xdhAEPpRlh7E+e19bGAKtj5QNM1O3ll+tp1iL7CpNr5FnUH9f7Oi9h3myZVZH5jmnQv8/1conEj27Zl76+UcVrcerqRSCH7ceLynSRaOShvHdW+9Db3wUI9oEsrj43Jrzqm9gajaObyvmcrCfhppxpvFGnl6Uo9+AmLa0c3NZjGU++ZoepFt2EYYz5VknAbGfVdKFOIfkFh29ON1KWI6sBT7xuylQQjtXNDUsm5JnUYzteWkhBbXAmT2gN6kAArgHQp/sRubGK6riKppcolJGqoNUm2Hm9UvJ+rXWdllKeaxkvoUeOlZqV22MLs6lG6nVxSUpkDrjTCLnoGZjloMLUbpKxDCm0qgUrKxdRdo0kX9ZAwG1KN2Vp8ogmZHsCfG2U7kJdpVOslvbex8T4QYlJz+Bwg9HkiwUOALxIJBYKh3sQXKbrupHOyc4OcNcS0XEmnFV9j6IN1NPJc5wlZit5UQzJsSMwAAiWxVmpLlHU79nNTCdNZZOKx7CWwfWBQ74GEHXsZK1NIDGEjchWVZzuprXME74c+S6kl0Z9nJ/3PgNXXH7+bDPZ3ebyOU5RS/3RIYf8ZsPzD+dj/ykn4j/+HXwXgb158iWnW8Hw5Qyu4N12y6TKeXe/xp9/9Cd+7uc8obTlfTZmXNcs652SyprZRk689ZdLx4+dnaONIEs/JbM3L2xlH87WkxLYpzmumRUPVJSxWIw7mG6o2paoy9mdbFuuCLHPcmy95dH5ICIrpRD6QmypD64AxnrP5ipd3M9om5a3Ta17czqlXOe++ecGz6z2UEsZuNq2wXrNZlKRlF3tp5cNYr3PSsqPIO5ZXYx48FMZTmYCrE95984LH54eDNzB4TYDI8kFRdDinaZqU6bim6RLqTUYxbjHG09SpJOWqgLda2KfrKb/y3md8/+IeqXHcXk94cO+WF58cU5xu6NoE2xjSsmMyami6hBCU/L5OOD5dcPXZAclRhfcabyV4KCss9SoHqzh744ZXzw4AxE/ZJpjEk+Ud28sxpB5lAmnZ0VUpSW7p6gTds08vCwmWaQ2hSlAjy2RWsboayxeyU6jcsX+wlo7SRY4eS7+oTj1ukZEfVrRVSlpY2mWOLiy+M9JZqiRMRo8tfitAMTuosa0RcKthfLJhczkSn+JBw/HhivPHB2ACxb7cFLvO4BojQHhRQqtlAcIraDV7ZysWj/bABIIOpPsN3W2O2WtFbl16dC1f/m4k544OEqwy6wiVwUw7tPb4ZyP8SUvoNDQiodVbTchkgYH9dnh+uCjwY4dqxaNn5wJyVeLxqxRVOvSVgGhfesYnG6onU5kwnm3xj8eEB7Jo0D6X2bTZCmvoCwHoyoIvAtnJlu7FmHDQwl0qXZYbIxPuB62A570Wv0rREQT7uRXvXylAU60S8UmdNahX+TABNRGsh4MOFikh33kww0mDrxL02kgqbqdIV9K56vMwgEYfg42yhYTWJDcJdk++RM20g+dlrMcIuH2LuUlxM4tZJkMFR0ikw1S10Y9ZWvRFjptZ8lcpzRstOnNwmYtseuJJbyQwJaQSNKO8SKXNVkvQYwfd3GMqLb7KiVQoJBUiz68R+bZDAKeF9n4LMYW6OfDkt8KMtUcOvdWDP7NPNa7uWSaPEqoTP4Tg9I/JE8X2gYCMdCULAT2DmmwjE+QEUGongKy4UtJ7u1WxK1UAYH4nHZrZUg19o+s3JAlYWZFmlxeabioVLpJkLNfANNAcyaQ9Xckixvi5YnsmwLqv1BG/nfR0SgCMgNVuItesnTGkzIJUAuV3inYqgNlnMWgmAkwd60H6h9RfyPZNJQBY2VjrEv2MvTw5aIY6Gd2J1zJbqGFxQls5HtPyuTobSTeN20xir2UNxY1ncxY/G/E9yu8EQCZbIru922+2FC9kfSDb6H2h2SoCskwJwG4FkGcxQbc5iONwDKNzkUFLIFPcZqyA6Rc3moModU5EttwDVp8I+5ls5LXpRuYg/faK6xD7dXcMbFJJt2S6DXK8SkBMtgoD+BQwpHYsYi59nr2stF8Q6DtFTQv5nScokZr36cI+UZTXnupID2NHd7JP04YdKLeyj57hSxoB2X2FSb/QFBI5xx749N4+W8Q+3Cgr70pFceeHyhfTBeq5JtsI2OtG4jns025tqQbva9/X2Y6lvsUnSqTBN+Ln7MYia/apIt1GoFoFkkZYwqQOuJQhYbVnhnvgmlZhqEoxlaebGLqxprhxwzXuE3CBz4FYH2tH5IOy83DKeBU/ZTeW1FlbSEKsqWKYTQwBkooUSbztpka8lx4IATs2ZAsbq73UAIjTdUySDXE/G5GnDvU8pl+YjOmx1pNsI6vow5Dy2gNPXcWQn0SjW4cvE3S1k8CGWJkSjEK3TsZbH8qTJWjr0dsWN80FgLogUleQOpFYaRL6qpLOfQ5cqihvxXmRucb99qAtZMkQ8tNvqwehaMSnGYLIahtJlh08mD1QhJhaq3bAs25EYtunw6YiTQ7bGlXk8u8m3jiDF6Da14b0Hsv+PEPYgddhELz2f6+xtv/Ixz8VHsvj8Gt7/x8Ay6v/5g/83OALDixHX7oX/tR/9+d4tZmyrnOOp2uqLuX6bkKSOgm72b9j0RScv9pjsr+lrjLsNhEmDmSi3WreeO+Sl9dzbJXw7lsXfPrxGd/8+qd858kbuGWKnnb4xqBSz3RWsVqWJKkDFei2mfglo8+xmDXCdo5bXGfwVSKSyr2KzY3Uk+iNER+hh3y/HibX5lUuE9cksH+04u7pHkHF9MeRg1aT7dd4pzGJwz6e4KZOujNrM3QlEoCJJRu1NNclqtXREySsG1ZqNZRVuJGXfr5SJr+6VYTTBvO8wI68pKey8xemkxZ7WRCmMsHWd6n0B+55ggoStpIFko3G3tuFpKDAThzJ2uBKAUJh5AToRFBUPEtpjp2kZjoJ2ghaJu7S/xZgakmfZ7T7LoaneHRl6Hu5UFBcaqpTTygk0RMFdibJqMpqCVspPPml9HvV9zrxK76Sn7uZePtU2CV9KqewM8focUJzKAyLHckKvR3HlE4rxzt/+47qOwc0D1uSq5SkkvRGKbNn6JBTPkr/grw3+a2iOpFr42dW0j7PNe1cZHDpcnee/aS+PfAkG5lw98xet+8Jqad8looXJg809yzpZWRHM5HSZXcyAajebzBXWQy3UNiJj+wpA7OXRiYx2Qiblq60+CVPW/InOcUVbB6GnRwvDcNELtmqgSVs5wIagpJk0NnvJLRTuR7dRLpbszvZVy831a10eYaUWNnSy8CEwfNpwM49o6dGJnZ9R6fesTLVmfgsfQLtvtz3dMsQMJOuZRJlx4H2SD7b5ZN0AAXNYWStS3mtaZR4Tj00h8I+djNhWstLxfqNQH4jcsxkrckWck6uDORXmuYgUF4o7DiyXkqSYQlSDQMCBOojAUHVqZyrqSJbU/THL+dZH0oqaDfdMWc9uyMTRQFG6VrASQ9gsoUwdeMXwiK5XCb39YEaAEq2ir2iuTDE3VTY374qRwXoRhJyki1j0Mx4B6h8IvvpQz16z119uOuhTdeRxWrlOMsrAcuuEFDrMhhH2WN/XroRkOJTAS+b+4ryXH43euVZP9Tkd0HkiVsJmHGZeEbL8/i6haSR6gbafchud8fTHOxYYPHxwfRJBB1qd7/xmcg8mwMBKeMXUo2xfrj7DPV+TFfC5Jln/UAzOhd2Mb/12JESZu+5bCddhgEoqsiEtXORzXZTuSeXV57VG5ryIt6Hyh04sKVIJF2hYk+k9Bsm9Q5oppvoR0wYQm9e78YNRknC6FaARXWsSDaBfBnZxVTCevqkVRf9neWlSDG7iaSkuky2Y1phEruRfIa7idwT8oUfJKBF9DZnay/bL9TgEXWpolg4upEeQF3SxICXIEDEG6k8cblMmG2h0U62WdyJt9MnuxRjCV0RVrEbyWJLvvI0U3ldDyp9Igzx5KVIOMtrh8vl+aYLkS0Tv6by8ne29rFGI557Fr9TRgpvFPlK2D3lA+1UklN92veYBrpSvJf1oagn8kXP9KmhK9Nlcq1knwKK0nVfiyIn2c4TdCsLCunaDaE3KgS6SYKpHN0sGUJxdBdo9iQsR3deOi4zSVXtZaBBC9Bsp0bqb5YtzWEuHszLliYG6WQLG1lLN3Ra2pEhW3S0eyn5ZU17UJAuW7pZJimsW0u7n6FbT7qWhNFunpGsO9woIb/YUp+OBCz2FSsh4HMTVSUip0XL94xuHWbT0h6PSTbiGZVeTJG+uiKJQWgOvHRG6s4N4M+NM8y2kz7b3KC3LSGVv32RCePYgzwXCHkEiVHG6ke5qJT8a4Ax+hl9kaHrVn5u2p3PMfZMhiJHb2tCngroS8yuvsQ6Qp6JPzMxUkESgjCWSgkz2Yf99MC0D/XRaldzArLfn5Wivgb0+uAdFcOAQg9A+xRY2LGVfQos/MPAcUi3jXPw+LNS6ncP93n9Ef2YX/xU2H8GLP/QPor374ev/MW/gPOaSdGwqnO22xxXJ4zmFXWVsTffsK1z2jrBt8JWEBQ6dfg6QRfCEIUqwUw7XGXQq4Sw3wkTtsgFWI0tKvr58EoAqQ5DAA+VQY1F7krmoZMPpOo0oXBD2inLVFISNTsZpge8kj7I3Am7EgNt/MwKCFxr3CyySBuNeWeN+3SyA4UTJ+meffKkgpB6keRNRMLXH4fc3OIqnRNGJeR++F0/YVKxziKkAeLzdF8zEIGDWesdWxaIEkdAx7S+wqE6LceUBELmMWsz9M71skJXCPviSkFdQ19kvD4hiSE2L1OZKAeRSJqtVEv4bCcRszOHbmSf/QQ8ROAKIkf0SawNqBR27kkWOsrphHXxmbAjfdG9ywEV2TZeW03vUxT7yYeJ0rHIyvhMALErZDtmuwvgsZPXPnuRheknsd1U5Ky6FeBvtpr8Rg3sQj9ZdWV/fLIf5aLfKyZmdpPX2LfosdKdTObsKAysiKnUwNSZOjI9QY6nm8o1CYlcb4hgxkQ5YivjQYCqXKv+uvWSxH5xQHc7UOhT4gKEot0PsV80sjNhJ1Pr99GzIT2b0V+rHjz2HXA+ZZA+9p2fwjB8nn3oKxN64NkzGsRjK64E+PQpoMOCgNuxUP216NmrZCv77msRRucChPqFhIGxcrtj6LeXxpTMYftxf+laJoTtVPbZh8YM/47jcCiRV7vjGtix1z7XupPz6SZy/dOVorwUMGiL2DXabyeCKZEox+31t614TV2xG8Y6jieQ/xvOVUepc7cbv8P74HcMUC811DHNUoUds6BiL9/Qx/bax6dPG+0lh0Cc+MpkHoRFJAJGFyWkzb6iuJFxH/TueFwpwLed71JGe8+dSyX4qH+PtZXJfT8pH65FtxvHPUNpWnkdyHHYQu5zPSupYw9lL7ntgeWwQBBlnL1H8fVx1DN3vdTTlhJe00szm5kwZGn00fkkevVyAajdSACXacPuGiPXtAdx/bjsuy4hJpH29yS/+4yZTl6XVsLG9emoPdMUtADM/nc2l6TSvhezH7P9NZbPeZSlvj5BVQqXQrYRKanpekmtMIPp+vPJpS52KQoTFD8Tdje2pF4iRPmpHz5LydYPEtD+XtuPrT60p/9333UpgW67a9knLg+fkW43iINWQydnnxQMoKz0U74OnpULw7gZKoucfGakx1IYIxf7J/vfh0RjGjeAyd47p5wkn+rYd9n3VvafMZHbq8G3qOxuH/35aBuloH1oUZwneLNbhAmJEnBYJnI8r9WBKB/Ps5eeRkauT1t1uVR02HFKumzxyecXd/pE1v49DFoNEtrXWcoeAPbMoE9N7MSUvsgdU+lxo0x+D+KJ7M/NBfl3HIfDNnv2MspXVS8FjeO0l7p+Liinr/Iweidl7fsovZefjZa/Y7iS7MsI89g/XvdIGj0AU/ownD4dtmceQbbTV4P8zKNPce2B5PCS10HdIIn1n/v/z1WIeA+vAemfffwTYZB4nOG1oKB/0scfCmCZHIdf3fvXft+3+9eu/9If+LkBJH/QB/D/5lEkln/73b/Px9sTfuvJe7x9eMOHbz3jsp3ysppR2ZSnl/v82Q8+4vu39/lw/xnfunybB5MF3395n7OzGzZtxtlkRaYth/mGv/Hpl9i7t+Dy5Zxf/NJTHi/3cV4zL2o+fnxKPmn42tkrWp/w0U8eMj2saNqED969pLIpnzw/5vBgzeFow11dclBuqWzK8+s5755c8/HLE0bjmofzBT/65D7JyPLw+Ja7rcw2XFCsgdOjJZlxLKqCUd5yvRiT6ECWyYf6w9PnnB/OeHY358vHF3z/2QNOD5a8+OkxX/3wMT98co9ffu8Rt/WIdZdxvRzTbjOChX/pl77Ht16+RZ5aRmnH06s9ulUu8skImH/hvSd8+wfvkB7Ucs8z4iHNjiu+evqKR3cHVE3G/J2KdZ2zeTUeWN8kcThrKPOO7V3Jn/zwB/zm73wgTFRmcXcTOG44PlpycTGHTUJxuhGf6E9nHPz8JUoFzs/3KKf1IKOllmoGksDx/Ts6a1g8m5Mdb8kST9skEiLzbISbW+klLDvsbUEya3E3OclRhVIwLVo6Z9hcjkBB+uW1dEzWBlaJdBseO2Gpr1LcxGHmLSZxdM/H+MKjSgebhPROM/q5WxZ3I0zq6VrDO2++4qfPTlEq8ODkjovFhGaVo6cN9qbk7W++5JNnx4ROc3C6pOlS6jqVasCg4CrHjx2+UIyeJDRfrzAf1GxeTZmdrVg9m0mPY+ahsLjzUlbOG0W35zh974rltqC9LVGZh7sUfdjiKkke1pmDqxy77zBrQ/q1JfXTqdRMnHWoJKBuU3SjsCcdJneopwX22DN5e0FnDfWLMclxzYdvPuW3//aXyW4V03/hnFfPDlCZwyWBcJnTvrsR2fN1TijdMCFFwenZHVcfHeNLjxshibw3gp7DyKE2RnyfaRBmOwCJMInbtyzkjuRCnq/e2mCbBH2V4cq4qFJIMA5BSXfmXYZpFO9+8yk/fXw2dFC6MgyS0nDQwipl/9cvuXu1j75KMZWiPXboSmPOKuxVAVNL2Bp0rcnfXNN9OmX7fgc6kJxnKGD1DnSHvYdFvI4hlbETrEbfJVLnArLwoUCXFnUu/tzsTrP5+QZuM9KzLfb5CD/ypLcGO/OYtRYwFxmo4lKxvS9VIjZKZ+3MYVYGHytLiueiCiheGrqZsI/NL1YkH41xZaB66Bk9MWy+3FJ+mslCRx6oD2R8NX1oUhJQlSa7k5RjlwfCyFE+iYnECgl3uksJaaB8YahOxWtpOmFL7UiYflNLgu/2LUvxIqG+7xg9ScSTduDxhUdXmvxaPKnJRgCZqcV36ktP+cKwfb9j8sMMOxEw103Ev9juxUWPMgwBLcqDHUkH3OZtS3prhoUa1C6NuPc+N8eOkAbGjyRdGS2Kgd6PGbTaLXKEXW+lHYX4Xiqq+zKGxs8U1YkiW0iysMiE5aOfbEUx0fdrqqiI0J1IfZUF0wrLni3U4C3VXVwEsAysb3Us17o6ZvCpFtcS8jIsZuSwfEdT3Ciyu8DyXYVpFbaURR/TiFey2VfDcZTn0rnY+1zzW5FNukzOubyUBF4BJorJs4AdK/lTCmNuYvVs0Ir8TvyoyinGL8UTahpRP1Qn0R9aB4LSqBBItrB6Rzy5w+IFEgDU7Gv2Pna4VLE9FnYs3cRFlwgimz0t4LeNi0jXwnpWJxplBTj7FOoDI4sIBZjW4FMYnUe5apAOzWQr8th+QWy3KCDARbeB4taDUnSlQsfuR4Kw18WVSF3tSAmDHCtPTC3BPKMLTxuvXXEtvsxubGinu3qSdBOGcwtT2a8tFeW1ozqKaZ7K4FORc6SVpNr2izlJvQP6EkIU0NYMvZnaClubbg3JxtMcyYpH0NCVitGVG+o/mn0zVJsk9c7vmWw9uvWs3ipJtx5tJWE2v7FDeI8tNOlWvJCu0FHtIiyuLWLAToD6pIiVNA47MiQbhyuEmc3vOrqxsK8hyo515zG1o9nPMLV4GUU1IayzdgFvMtK1le3VTiSvjcONMnymd32YuZbuS6VIFhV2VuBGyQBik1WLG6eo1uPKRLyckwzVenRtsbOUZN0OYNftj4RhdQHddNj9XDycsULEj3JJkFUKPy6GBXe92OIPZ0P67M9WhgwVJIl4JdWmFrazZyz7ypFRXBmMctthUSmm6VJ9PihH9bUjnZXt+UBYrSDNIHhU0PK7ECRRtm4gTQUkZ9kOSEaQqXoP6e/ltwwBjDwvdP/4p34hHr9X+u0X+PGFZiwnH5yFv/Df/wafro94vpxxeztBm8DD41uu1mNGectyU/DG4R2dMxyVa35ydcI4b/FBDf2LISiqJkXrQLXK0YsUddwwGtesrseY0pIksT/QafK8Y70opeD9pCIE8M7gWk1fP5LlHdXFiGRpsHuO4rCibZKhoiQbt7Q3BWSecl7TVCnqVQH3a9yqT7hgkHoCoAM6d/hNShrDUey9+O28Tpg8XLJ5NCfst7BMSY5qbGcIbfzgmoC+S2WSNu0IVgvb2mhhOytDmFqR7N6WmMLBy3xgSouzDdVtST6vaZYy8U2uUqkzuc0IuUdvzcCA9hNkXwRCEumXzJO9THFv17hFKkmjpzU8KbEHdseqRO+bqbR0EVqRAqeXCXYi8lw/kZ/dSLoHu1iwbhpFe2LRa0N+o6nP3MDAqsgk97UcqtYkGwlN0Y3C7ltUrWXfG40/6lA36dD55guPbkX+FEaO7JXsP2jwuSdZGpKtoj6zmK1suz1ypEcV/FT8hu29DnObCpDwMol1YzmnZKlF8rowgz+um8r/ZXdS4bF+x5FfmiGV0ucyGU6XMfQmMh3ZnaJ6w6K3Oha8y8Q6AGns5atOPUklDLHPRKqb3Wp8LlUNroD1l1th8VNJjuy7AF0usk6fBbIbswtxWahYEwJ2GiguBPzYsTBkulNDSI3LRBY7sBy1GlgdW0aJmo2prmthitqphPmATCi7mUyAXSHP6YM9ZPsM++qf00ty6xPpSlSR+XGFjL2kih61uXgC05VcB59K6f3rvYPZUibT4rOMISyv+cOUF2no+g255n1likgPhZmoDyXldOgnjIEwKkTWaSHbqo93bKV01u1AhMt3DG/PdPYl5sDAyCUVURr9eSCS34q8Mmhh0obePyvhMn3/n09g9CrWUYwZntd3BiofGTPLkNLZjSHdxv2VApz62olkK8oGn8ZJViEMYn0gE3Y7VuBhdOmp9zR2AvmNHKuO17xnUfvrPQSzFJCuBPDkC5Gv9qx1thRg6TLZtwQqSbBHM5NeQFvu2Dk7UsN72k3kGLsxBCWAcPbIs3xbk2xl7PT1H8O5Rc/a6NLRTjTNTDO+cGyPpBPUJyJvHL9yVAdSB/L6te0Zy35cBi3HnFSBbB2kyiTKWPOl/NyPoXQjE/NupGIoThi6Ifv3LFvtWMxmT+pWRHK5+z+XClBoZ4psGQNeCmEc26kawI1PZNs9O206qQZZnyWYTsZtFkN92qmcc71n6D2MfcBO7/nLF04k9DMJl2knUtdhmj5kRklATRWGfkbpU5Rrny8lSTXoney6Z5eTWiSz7VgATTA7T6YtVGTQhQnM1n4AMqYW71y9L4nRSR27EVNhKruxJr+1bE/TQdIrdSRhqNwwjac+StFW6lSUfS0IhsiyeZGNZwuLLQ0+Sor7cB3TBvKblm6SDN9R6dpSneZkd1akurFXUjc+AlgzjHlliQE5GlP7GFYm3z3ZqsMnGpfL//UATO6lFjuOgFVH1s4FTOcx6xY7y+VeuhHpqk/0cE6mcajOSx1I4/C5EX9uBFpJ5SRsp5Yak17imiwbfB4lnpEh86lIf4MCjMIsW+w8x2w7fBkDdAJSoZKZgck0jcPlkbHMDaaysZpE6kLMqiGUqYyHdYufZOhtlKkSwVpk5VTToeoOPysHj6RqLWiNXm53EtaeUYzBOr7I0NtmYI3RGtpOpK0906gVal0RZmNU1QiT2QOwWC2iqkZ8lD17GEN1VM+EKrVjPBMj8tifZTf7x+tgx9qBZQzeDyxk/wivh/30EtoQht5K2LGe9F7R12pL/iEf5e8BLPuE2RDCFz8VNjkOvzr7V3/ft/vXbv/bP/Bzgy84sMzfeRje+M//A5zTjMYN87LmxSfHMLYEpxjtVVhrCF5hLwvMUUM5aqirDGc1vhavZWg16bRF6SBBOC+n6GknSoSbjDB2mNyRpJauEUmtXiakDzZ4p7GthLqE24zJm0vWdyNCq9EjK4xXZcR7l0hIiLIyQe6OJbofJ6ufZqvRjaI7tKjSMZlV1D/cEwCBSBz9cUv505zqg9h7GXVpZr1LQfNlvJl6hd5v4GUhctlUgjm6uchndRcllEkgWYpUqJtGuc1ICt576ZUdhQg+ZB/5jcgcuwOPWYmPyBfiy+t9eHYS8Jln+nFCfRR7Fu835J8WtHOPn0rCZ3Pg5Tw0pAs9yEWDkdRK6SmUePftfU8wkMWi+tELTTsT750t4/ntiyfSVEpknTFYQzxFshLvRgFdK9KNGoI9sqX4hbpJ2AGv+5J2m9zKF7edio9v+8BRvBL2z44k4bE5EqmoHYlPMFnJjbjdFz9mX8RdH0uJu4oTfG8C3Z68J+NnmupYwEwfJpFuJEVSd8KOuJj4qWMBu4rgtJexgryPo5eK+kiuSS8HtROZUBWXAohcKa/XDTtfonotkVFHtudWJHNJTA1NNnqQjorEVYCsj1JdZRk8JSamS45fyHu6fJehBkGeHMfUXZTHpv12ZUJd3Oz6EnvJanMQAV8u5e/Nvsyqiithqfoky25C9E/FIA0txyPBMjs5rbI7uR1RtupyyK9FomXHKnZjynXy8U8S9zE6F5Ylv5Hr287l/HqQ5CP4/L/ZclkEdQAAIABJREFUe5Mey7LEPOw7053eFC+GjByqsrqqurpJNwTCEiBSC3ttwIa99VKAN1oa8B9oayV4YwHaGTAM2IAB+x+IS+9ogBZFqskmWUNX1pCRGeOb7nQmL75z7ousZpMC3UKzDF6gUBkR7935vXu+800A96O8Z91BeUtfqtkxNXP/HtmZLLWLitejusssDo6AJ8kcH3cd8gIC/TlBbLHJ6YzHInhfHgGsCIAcMHnfdEvZ57ggYDJ7rjIogjE7p+8SILDUXe6GjNOgPKQ0SjXQz2ebFESTZOM5PZLplWmXs0T50XXPcm/pmaDZnbNrsXqIBEkyn08O2qM4BpqUm8RyJIAjHX1tIvB8qP4oW9VdnAblUQLjUkxBOe0lQaRM8lSVwmeyXDkk8EPJZQpvUcdt5qoLgK9t3oYpGVRa3gu6P96buuNxmzaBh5IAoHnj0D7RrCa5oeQzewuHlZxCcWQKTJGOfZXmkNNKCZgJpuLUV+kLAVcLVA+edRsLCZP6KUXyHIpASd/kv0znnfJasl45wCaDogzEulOF5q3DcKJQbj3GuZpkkxAC4ywH2PB9U5BPJaaqD93zGE0bJ8l5TnzN0lDpUpqrJpDOgIwTRQRFrua1kDamiST6HPVAqajqObGhek4w+FJCdQG+4gHr1mNYaxRbD90RYErLzsvq3hOsIcm4jWCyZ+T2aOfg/piDgy8kfCVRbBzsgt2VwtF/CBwl3Vn+qlqHUCkEIydZaAYSwST5KwBfSegupZ8m6acvmLJKACZgF5TBTimqNiAoMrvl/YhxZaCGMFVrlG9bDE8aJgzvLWLaxygFVJ9ktQOBdQ6tIUDV0L2fwJ5wAW6Wejp9IOgTAnJwCEYhlASa0zMhgUqmwAZEo9LkgIB56BEqAlbpwhEQSZ532dN7GY0iGxjCJEVFOH7e83tk79hpGWLyZ8b0Wj7HQmWg7w6IStE/mfosY6ERpYTsR4bntANiWfA1Azsm2U9ZMYBntEdAVRgG3cQIlAVgHcRojzLY/Pq/ypdYGIb5ZND4WN6al+yBTMvku+zH47nqhyNjCLwD9iYPZeqaFCax1Hlbzh2Df0KkHzRGwNp3mckEKvO/fwlzPLp2fy2LF4/A83vvsdQX8Z8s/qtf+3r/9cP//Bs/NmAi1b+niwCerHd4cfGA/X2D680cUUUUzYhyMaB9O8O4KfFkvYM8GyFkwO52Bu8l4rYAdCA4MwHaeBSFQ/sXJxBWQH5LeYDZSmCU8C3TTYUEg3/OB4TP5rCtQXQS6psKsQ44fLFCtBLNWQv1VYXqqwKx8ilQJPn2igj50R6y9JAHhcXTHWRHUKk+2UMe+PQ8fL6ipKxkEb1feGCv0b7voK4KFLcKwkmgYGpjvBzoTQRgNgrV5QF4XSFUkV7MNCMtHB9wbuGTB1LAPrHsYdsI4GyAbBx8kQY3DesI/Ec9Z95fHjCcB7gPe+ithPnhDsWDQJw7esQcwVX9cgc5Sgy/u4ddU0YWHcvLQ+MhBoXxhJ2RsYgwG3p8uhcOcmS65O4TJsIFE7H/wEMlkOTqiLi2BIIngXUOadAz+0IjlJF9fp7Ax15aiAD0TzwZRhXhTjz6C58AY0R/HlMojcC4ChjOAuWYNoHAlB5qZ2Qz3SxiTJK1ww88e/5mBMH+Rc/kyuRL6p87SM/tm6ct+uce7UeW4PxlqicZBIZ18j0aQP/jewalzHjd7DLAriLsixGujhhXrGNw8zB9HrLfxFcR/VnEeOYxnvDLerj08HVIs/+YAB3ZxAhXR/SXnp2NK6C75GDSvjegfR4omTw91g2YXaqeOOX1Gs4iDh9bCAfs/6MR+48c7ILrCYZga1iTUdYHDqDbT0YmTp4FDCvg8CKgP+cgvXtpEQrg/ie8NsEwZGb/oUd1Q3A0rkOqCSAbmesisoQrSk5G2DkZSoDXY/dhQFARckyTDqnPMPsHowDGZcTmJx7DmgzN7iPWgrSXPA/dcz9J+h5+lNietcDuozQZMyNIscuI9pMB/VnE4UVA+9Jh9yEwrBOD8N6A7scD9u+TUd6/ZJF9FJi8qP0Za0H68zTJZIH2qUBIdQuhYA9eUJTMicBreniPANnVAje/63H3OzyG7jKBwzmw/SSgfcbBeX8u0F5Sgjmcsg+vveT5aZ8KdE8jth+yGsE1AofnnODozwV2HwbYWZpcSOd+977A/gNg8yPeZ/0Z97G/4LlunwmMS4HDC17P4VRg9sZj9wMCHjcT2PyQXXm7l1xvf8L3ZPA+rrhv9PGSDWufHhkSu2DoTCgJNnyRJKIjmTYA2L2UaC8l2qecYBtOxTQJMS7FFEzTn3LbOe2VLBXPb/s09WL6NEkjKPk8vBBTCFC/ltj+gAPWYAgkD89EYnO5b8U+oL2QGFYCwwl9r3e/bejV1QKbHyi0lwK79wmWmree4UYlj6u9UBiWkiyrAIaVQHcmMaw5EWNaJqHe/0jBNgLtU4Ht+xr9mgyYKwUOTyXaS4X2qZgYx34t0uslyq3ntdZAf8pz7WqB/lRinFNmeP+Jhl0ItJcaeojozvSUirp/T8GVAv0ZpbzCE/jvn6sJIOeEVjsjq8uJgXTOtUB/QuVI9iWaPZ8V/ZoBLlEAw1KiO9cElYnJdJXE9qVGv5LQPUF6XnwhMC6ZvtqtJXYvNfbPFPbPFfoznbyiArv3SyAA25dmYiqZTMrqDVtTsutKicOlnkZavhTozg18LdGdKgxrQwAMoL000L1H1FyHaxQOzwuGJj2rcHhapG5Qldajp3USYAgGFoUIVysMp4agSwC790vYRmNYG+jWY/+imADz4WlK0tZA+6yEnUmMK43uaQlfSGx+a0lg3nAyUe8pNW2fGPiSYNeXTEqNRsLONezSEEwEYLhskl+a4Fx1qdZDSbKeQqC/pAKqvywxrtnB6EvJqpGRDKevyDiGgoDRVxqhVLDzVBmiJYJR8LWGPa3IbiaPdqgMRO+SN1UAKcU1aonxpMR4VsNXGnJw8LVhXYiPCA3D74QnMEXab8QIv6rptdQSoeGMaljWECEkDyrBXJzVCLWhp1NJhJMZYplmYKU8Jq4aPdWGxFzpgcy0S8TCkJlUnFSY1qEIRGNZ8P0g8IylAYaR/24qgtwc/qMSwH4MKuWRhQUAMasnMCmMQczMZWEg0naE1nxN7q8sDERVHgN5ctCPc9PxyLKc/hNFAWE0/9MaQslf/Z/WPNbvpsr+/fJ3bvleM5bLH1/Gn/yrf4qL+oCrwwJCRNxuZvBWYbnsIETEvi1RVRaXiz1+8eYMQkSUlYW1CmfLA242c3jP2gtdONj7CtV5B2McdvcNYCXMckBZOgyDhhuPttTYK5jlQBlsp2BmFrbXgJOQtUMYFYRiebwuPIKXiFGw/mPQfG0UEIpf/gC4D5sS0BFyq1G8d8DQGggVKaO1EqIIKGcj+usaqD2EiohWst+w4/6JXiGHAsW5A2RENU/vqTjAVpWDPxgGDZUBzUmH9rZh4FARABWhK4cYBfzOQFQecZAwK8pv7aak1Hd1lLAKm2b5lo779KDhl45BRY8Ta70AqgBdOfibEnHmIVqFOPOolz2Gr+YEn7VHbPVUwyEOGrKT8KcWYq8hzwem9vaszogidTkuHdQDAWbUcZp5lC0fjqGKEKNAOLGAldw3E6Z9FCYgdgpipK+rf99CbUgfhTJJcVMlB8BJA/2gpiqInBKr95Jy0TlfK9v0RS5YvQFPoBUFwUKWbebXq1bCnVqojSZjrAHfBOj9MSSpvKHvTHWUA4vEgIfGw9xreu32cvI0TXLLBCiFZfKqCGDVxb16JxAnB8NEzSqO8oayNVfHKUAoVzsIexxU58GdiGRSh3N/rPWo48R+iACGNiEx1v2xyD4UEaGOkB2DjATDWjlBkliS4dyjuFMpeARAFBjXHmYjIb2YJh3yvRcKghw5srOyvJFT6icDmMjolPdkn8ZVPPYaJgYfElMIlBxTeJTEJOsNhizk4+Ak4TGxtDkBV3omy0ZD5jCfb4ZlpPXsxXQeRaB8dZKM56TRMrGnC0q6iy3X4WZkYynVjlNYVjDcH7MnMCsfkmy4xsRe6xYTSxyK4/1AWSSOYUSJ4VUjWUw1cps5eEb3+f7BVIkS5aPrYI+hRZkVAxLbnM6d2WHab9UfpbSu4TrNITGIqbLDzbju/Huzf9TfKHn9zD5O4Chvt9gRbB5DklLaakPGmhUMiQEvxXT/CsegH3MgY+WSV40sd5KZFmKS/o5LMqGuEdPnQDoyShM4nSS+x05Gs0/vSef+2AtK4JTDwzKrmq9L7obM5zZ7MLPqYApSAtlC6eNRdZA/OvJ4z2XGNjOlWR6ez6OrubJyG1gB4h99Zv3x3lED+x77lZoYNF+lCpg5WUs1JD9sYqOpKIjQHRnizFrma2FaeiBzaJEaKZ/VPYGlLxKLLI7nOWjK3at7vjd3o/oi3c+BgLt68LCzNLCX7GzM1R66j5jCpSK3m6s2cuAOk3vzcbG7UYRc5RHxWMUhHNn1rKIIWrwj45U2wtUSZufg5oqqBJdkwObon8z3VQ78yexploVmFYEc+Xc5BPhaHcNukjzV1fzeR8C7QUHp/uM9GJiyaiibZTqxhu4c+tOCbPoYWNdRqHQPk4UVPk5eRkphPcO7hJjkqwgp+RWYXid9gBgDJa09PYKhVBA2HJnJFLojcrDRo+CnKBnoI5NPcQoUivHok01+ReGoOEAAoBNb5yP9jFoiCnFMiA0BYnCIlXm3FiStM/+cvZDckCTI1Oq4jzkYx/upKiSmOhDhuR38VazjdGz+uO4sWX3MEuZk2O+mtuZ/M2Rjem8MZIgnmSt/OTGXj8N7ACDmbSa28ruS2uNuH//21+KSEL//4T3qPP6T+X/5a1/vv97+L7/xYwO+5+E9g9W4283wcKgxDAZSRNh9geXFHtttzT7HTsPuyskriSjY3Xgo8LozBCspKXbeDLjvDPqbGmMvIQqCD3kSYS1BhRARYUhfsCpOIBBRwD6UMOsedlsygbanvwGlh31TIxZMUT2cSYg9K0ggAFTU98VWw/YKMBHL0wN2/RL9QwUkn6W61/ALD/N1gWGpoQaJ0HggCIi95uTZjUF40SMEAdFJBoUcFGIR0duKSbNFAKyE7wug9oCltLdTFdm5QQCOCbrxLgHVMgJWMzTifga39NBbxUTYVEcisqw3Bb/EmiwpO5QoG4UHinsFNw8IjUe4qiAkgIOC6iW8APzrBaQGQi0g36REhH3BZNIkj8M9Qbl8VUEldkYMAmqkj2qsWaWibWJnZwwymfxMWRZcaIiBwSHSawxnHnonp249s2dgRfFG0xfXCsqoDT2pOjHEcnwE2LxAc8Vi90lutFGpOiMNqqKgnNQTPLl5RBBAeU9gLq7JfrhZRPVVAV9mHyEgPY9NjQAgWN/RJn/cVk6Ap/jGwC4jyms+yPSe1RUQrP+YBusKmH0tYZeAvJaw8wg9iilIxJeANIAcxJQEq1tuO/cVSisQkFJfO0wDUNfwZb6MqK5UGpAjSTBFShWNEHv6QPOAVQ2g57QTELeY/IJRpIHngWAsCmD+Cw07T2x7WsxWks1tgCLJgzOLqHruPwePOTiA+6hGgWIvpsE4Imt2RABkOl8IRyCU62XqN2IqtS+3wHCWfItzMfkgMyjKx2AOrIPozxnikn10OclWHdjdh0iPYlTctsgT1cMRSElLP2GUAmaPSeal26Nk1s4JOMcVX5+lx7rlcaqeA3rhkCTaPEbdAf6RnLi8i1AD0J8ek0Lp50uy7ATKdEuw5CsBnQfnTS5yFwiW3w3SAfr+UdG5B9yMFSdR0Fdo50xp9WUKdYmpOgMJ8CU/YwZmzbcRdkmJKkfaqWvRCKiOEl2XEoF1qtLQPcFK9ubxO41AxOwfScSUAFyE6vMAjNelvubkQrmNaM0RWDExFpN3T3dHwC48g2jsLHkiQ4Rr0jURR2BT7LieYMQvJQIz+RZTwrHuIpRNHkGTttFhAidMVhWs9aj5s7IEcsID9S0nOr0Rx7TUBJhyVyMnCHjeyl3EOOPfsv8PQqLYBYxzifLhCD5sk+SoCdDpIU5S1smfG/iaMgEo0zKxttxyPbaWlF36kGTEx3oONfkXI3JqLtdBIGcbiebGwTZJ3trzda5hX6Q3yUu58+hPNXSXU3IB41NwSxfY95n8lznsBkjn6OCP8tmekmw1xCmltkppsQAm8KeGHHKDKQQKSjBMplIEco+ktmTzRPJ6pnvCR6jOk9GTEbpzQFSUjh7cdM7GhWEiqxSpp9S9E0gTCtZ3lHcDxnWZAB2g+gDpw+SX1JaJrUHRD+nmXK/wATJJUukDdkAAygeLoI4eUjkeU1ZVT8+jTvspfDz6JCP9oQAIHltLH6MnyFMHm96TEmW1hDpYAlcAaqBizDcGKofcJDAWNIGp7LhOekUBBEpv82RLlAKyt5SnJsAtDwOZSi0niaoIAaKnNzEqAagUrDNaxKaEaIdJ9j11TSZJK2I8AsQQjoAyEuSKfuQ59R6Q+pgSm2+mkDyUIb0/pCCdx/2QQhxDd3xgz2aS3b4DUB/5HWMIlLcm9nJKZvUBj+WplLymifacApv3P4FT8Z06k+8CyF8pk33nReGXf/d9XL7HpN7ftHyvgSUAzOsBD9uGvsfS4ccff4tPry4QNwXUecfkaElmSABwg4JrNcNqbumfFJ1CWDrcv14CXuDkxRabL1eIpUfzrEO7K4GHgl7EjUS4cGT5Kg/RJwN77Qk0HyrIjmlmce4gdIC8KRjrf8dUOXlvEFYO+tqQkQHovQQlrOOZx+HTFeKCX+hikJA7hsIAmOSx8qKH+KoBAuDOWMXh64hgFeROQXgBXzvuTxAoHiSGHwyoPi8JZmYRcscvT2kB7AuEEogpnVLvJEGF4yA/mIjiTsI3abY3ALGIKG7VxLJU1wL9RYSvyHyV9wKd1BPboYb0IOkERErzdKcW5trAnjoU1xrlPQvOm1caruGDt3jgzHR/EaAcH4h6Bw560gDYbCX0gR2F5oF9hm4eUF4rRJHkVAuCMLPJQEaiuqEXMegIs6U3c0iMmbSYwkBUKyaWxOwlVI8EWjlLXN6yk3D2dcT24zTIO6Rutmsed5Z9cYY4+foUPY7FQwKHB0o+heP7xiXZMztnkAsj1TFJm8vEkgGYZuHZt0jwVm7IJEESUOqex+ArsJhd0W+YqynMPs2yK0xVEvqQ2COTfJp7MQ2S45jAhE/nJyVxuhoT01c+sJdvWLMCRgRWi5QbMkm+JDjJiwjs8+ue5JRCritLXXkPJVauTpMaaVFjTs3kPmdvYVSA2WYgzHVlUKW7iPHkyKy4GhM7mnv/ig33Xyam7zG7YvY5aIjsVnknmHIZACRGLoP0qI4BNv2pQPHAwWt/ShA6dTRaXm9fY/IEktEUU8KlGtKxgWBM9ceAnKiB2etAv+T8yBLJMe2v4zkt7ymXzEE1ZHgEjKUkVHcEjMLFdF2PhfLCA/UD5drmwPNoGw6UsxxW9dn/xg5EghyGwJS3DMnx1dEz50t6Qqe+y32c6jDUyNcjJG9i4HGUm4DuTELGCJlqZsw2+dP6OAU66TuPcSH5eQBB1LDktpl+yckFMiUEJK5MDF1BEJp9oGZP4FfsCLb0wFoNXxDki8TclZtUK5JAjvQRs6vAIvcugeKBElgG5wDVxmNYqqkeZJwxHMc2SVLYRfYuJpCVJe2m434PiyPrlj+3/C6LE5uo0udI95GJnKWe2DE1xKncPhhOKOkhMqFTpRcBmF1ZuJnixEdiWKWNU2UJfctiYm0zkMuBOBmE6YgJwKpERpiWYCpoSq+jFFNno6sTmEpj1/rGw86P4T4iYgrzyZLyYcWeSDujIqPY+hQaxf3RrYcs5OSlVIkBFB4wO4v+ooDeediFhm0k9EAAbmcS5hBSHVKE7j2BeSUBy30wQ4BKEwmIcQoNCkpAH0Lqe3UYzgzkGFHe2aNKoVTQe4v+opz8osJF6EMGM3HyVIpHA3FfKuiWQTgTQBKs4Mj9jz6QLQz6WIXB9QE5kGc4MShvR0gfyD4eKKl0c3Os89CJYUzVIKFQ0B1vMGE9xnWVkmZjCslJgTybAZiZNImZQ18YNCN7O/kqhaUcNi4L/j4CcAEigbMwK6ZAH+ECf1ekLkkBCB+geno5pfVHBZP1DOVJlSGhKageSl7PqT5ICQb0LKq0/gDRDZCPex5T1UgGb8JlEEiJqRgcZa8JSIosPc2/k3IK1RGDpScyg8oc/iMEoiGohPOUpD5iGd/xWkp5BC85xVXJI6j0/l3Q+d0eyxxOJAS9pd/1cWaAJyQQjqByAqWpz3I6P2ndQh0DiHIYz69cHnlEJ5CZt/P3y9/Z5XsPLAenoLSHVAHLWY/Pri5wsd7hrRdYzXvcbFhMW5926A8F6sWAZydbfHu/glgMTIJtC5Sf1Vj8oxvcfXqKw5+tYT48wL1ucAgNXry8RfnC4fMvLmHPAtRGIZyzVsA8beGsgnxTov5oi/1DjWgFyqcthtYAGwM8ZdCOAzgAPx1RFg6jCRAiwnxdwS35QfMfdUCnoc86hDcNYkWfXwwKdhERTy3qPy8h//EW/Z+vEN/vEbwA9pqMSZ3SXRU3Jnr2TOZ00dgrjD/qEA4aovYIMiL61DF5zYoLUXrETsEqykuDo1QSAvCHNHC6T8lxNwrjRZLCmoB4V8LNPUQQiB906HRNn9Kph16NgAqIn8344HqvQ/MHDfaFhr2wkHsNuw4QnrNuwzom+WWA84qSxsSO+UD/YPOVhpszVTWUEb6WsCsPtbSIb0ump9YRbu0grzVZNwnYUw/cKkbt/9BDtwSUdhEwPrMovzbwBdB+PKL8usBw4QAJFLcK7sLCDfxyU61EKFK/ZSRwswuR5E0cGLhzi/1cwjwoSCswngSogQFFqhWABNyKA4DQBIS3nBQQ4ljJMJ5EVDeSPtALB3OjUTwItB+PqH9RwM6ZzqpSKq5dBJiNhFtGDBf0gTZXAt2TgPGUwL5sBdpnIYW8SKa8VpSt6jb3VwLjGaWmDIxhuJTZcBLDV0D9lsAwGmCsYxps8bhUz+ePL4FwykHe2DDV1S7ZzzichsT8iakCYDglqHQz1lMMa87mVzcC/Wny06qUBNuKqY8zypT0mYBqUJh6BKvbiN0PeI1irgRIrKTLhe0z0AvdpXAVw+eYncVp/3yZGMH0fLNzAqBxGVG/FcnvGDD7WqLYpBqGSLbY7LJHGNPkQPucEtViQ+8jmVYOvHNwj5sdqx3GFRJgwiPmg39zTQLNc75m/4IyX4J5Hqtuge6SAF8OQH9O9nJckKmTjkmu2bPcnfMz01wRvEobUW4i2gsJIekhzNLX7PUrNmShXC2OoLqmHNTNMEl6o0zyUAPYxF5UtwS2dkbg3J9JmJSSGhJoc0kmm0FSBguZVVVjYjnbOIW2DCuB9omCGnIyKtCvxTsTD5RcJqnqbUR3JuEasqZkZwVcQ8+tq/mzHwEIoL2QBJG1mIDP1DdbIAXrEPjtXirMvg30UGbf3olCdUsQPM4ZVJSPTyTprbIEl5RAH3ttIclQsnoCqO8DunWSeA9IvmoxJeJGyQklX9D3WW4p6xwXErZhqI7uj1LOHIIUBSs95Mj52sNTM01C5TTWcaFgZyn5VZJRFElua+skM45kRLsLjSqBfVeS/czyXnorI4QXiGkCDpHnRoQIm+oyomQtiO4C2nOF4kBJ8LBIALMlEK1vGCI0BYCd5EFvmqCqNJSNGBcmMfgML/KlwJA6LUPJn5s3I4YTA1cfPbasAZFwtZk+k64++tZczfsRkf7BmO53X9PT6WYaMgFyO6f3kIAPkFZTrltI5D7IkNioqCXkEBAq+jJzYnAA4E8Mcg0JQ38kQiHRLcqJcfelQiiTP9gFhEIiFALSFbBzheqGqbNklSOGMwOzY9gTQHBcbC2G0wLFg03rlJCW20KtE/Ofql92Dvrgjj21RkLYADs3UDYAPsKuCpTuka8SGqgB+Ag/Lwk2Nc+hW5Rp4iLSJzkSaEYjEYSA6umbhBIMJKopTZUuIJQadlWiuGl5L4QAGEUmVUr4OZNlhQtkHF1ALFKq66KeZK1i8IhKHVm6LHf1moE8dUFgmbyUBIhMjxX9iFglT6UjEA3zipMQw0iQWZpjS5d1TI4VOR1a4HHXJR6B3SklVogUOpRkr496Mt9ZMhAMYZK/RufJimdPp3MEgy4CUlESGxLozGBTSspalUK0lmxlBpQ5gGjyYSZfqPfTv6d9eGffxDvv+74v8W+qV/keL99rj2X13vvx+X/338IvPMQgGbSy1wykGSUgwZoLHVEmWWCoPdlLK6EGls8jAPogYX60hfvTJSCzb4QDTN9E+LWjFHXlILc040eZdepAeaMQZcR4EqDbY31FlGBNw5YPdd0eu8jcnNJBgOsyO4nhzPNBWjAghn9M3pXA91Rv6VGLkv423wQUdwp2GVC9JZCi7ImD5PKWM5bjSUB9JdE+DygeCIjywCR7Y8gOiSlhNZepu4ZBJ7rl34o02EdiudgJF1OnHQHOcBoQVcTsKzWVsUeZgEgB9JdpPxSloOWNZOqsTomfiRV5XCORE2oJ4DgIJ7NGsCEHgWiO79Et5Y8++cvMHuie8LVmywF8+SAmxsWmQa/ZYyoyn4q/CzJIQXPgrnoCGJGYKs6C8z39OVm77NGTo5gSRX0RUd1SwpqTdoMm2FIDUDxw//Yv+f7FFwzlKDaUWAbFgKDMTsk0gJWOLF4wBOX1W8GqCI8pTdSnahLpuP85GVXndeXAuARSih3BrU7rVV2Skya/5uNC+Oz5UinlMjNaGWDX1/y89GfHwXJet2vIJkIyRbS6jdi/BJqrIxuTn6x2kbYvCU7KB0whOroloMiSypzQmfczM4FkRuixy38rtjEBTLKoImbJL5kIOycIyumn41Ic5ahJxhhlksQW/Fzkdbo6gbskATQtK0jWfxZx+zvYZc45AAAgAElEQVQC9Ruu25eU71a3PFfDmt2CWSbKICtxPDZHKWX2ZAI8D8MJ99HOBZorplxm35kayM5Wt/w87t9jd+B3fXO5386X2WeF5FOMmL0J6E7l9F1ZbCPaS8k+w+SdjKkmY0rIvQ9wJZm9YSnpCUyMX7+WqG9TsNRcTN6wDIZFfCQL3RN4RkkGzM5EkjbzeOvbMEkdH3sNRSToMwcyi+WOvjnThhRcg4nxGU4kZleeSaqerKJ0BEYAUBxCGuiLKZm13JB5ku7oQWWqL7/7laVUM3vyZJLHukpMPsIMzvK9Ps4lGboU8OIaMqDZF6rGgJAGXN2ZhE73YbkhSJw6BDsmtJouoth5uIrPBF8cJb8ZYDVvLboLw1TYGY8LAtO1G2eSKbKJTaS3mcycSkyzK0UK+FGJZRMw+0BW8RDga5lSdhNz2dFvSPAV0vdGlkkGuMRWZw+iq8kys8ols/lZakxZJUEdoLoweQoBfn6C4j1W3ntIT6+iHOOUqlredGhfNFN6LNULWWrOY9eth+w97KpI31FH9s41MiUFOwxrg/LewhcEWmoIkDbAzjXM3k0+R19KmJbSVz6TI3whUWwtXE1Jqz44nt+Z5qRM5+CNTM8+DzdPoNaGqdoDPk4JrUFJZF+j2dmJzZRjONZvGDG9NkuM5egniSolngKuUig2I0KZWUV6J/3M0B+qUipu56bqj2jkVKsiB0dWMcmIRYwIWr7DugKA7Bzk6OAX5cRayZEAEYJ/5zgkTOxdBnw5qTYmP6TwkVUgANnQCQBGyPx7LembHC3CrILwBI1ydO9IWqEJJO26hn7oIYbxyB7qvN5AYDmrE2gU78o7Uy1IDt0RgcxiBpoTUHzsjYyRLGcIiIcOoiwQhxFiViN2Pdc5nyF3VSIzn9M2sx8yEPTlhNXcW5n/XVf8f/CAVL+873YEtE7rC4g+/DIDGcI7LOjjypDja/I4+jvr/+6SQGvM+/S3XP6ueCx/r/7Pf+3r/f3D//obPzbgew4sy49exE/+5X+Di/kBX745Q1Fa9PsSsvCIQTDsZlBA6VEvBnTXDbI3UgzsZexfz4AFTSvRShRXBv6jDvGqOtYprBJwLQJgArsg5x7FGzJsCKB/ToCS0SogVgHmVsOXEWHmARmh75L0tSErKK8LiEA/XKgCzIOCfWJRfmtYD1JEuB/0iA8FYs2yeHYRBgbrbDTZSJdqKOaBsllLqaadBxQbie49C+Ekog4QIzsaJyA+SPY1mojQBIiR3j+3CPRnjjxueVAIMw+1V/CzADmzwHUJfRCw6wC95SDTLj2qtxr9MwtIoPncoHsakg+V100OAtW1xLiKcLMweSflKOAuRqjSQ76qoVqB/sMBcsPuTQBQO8V+xhcOZkuvZg58yZMBUTNYxmw5yIfkrLcvE2OUZLvSMWlVdXICG/bCwdxqgnMdUX9Npm5ccefLO4HuaUB5JydZrS8j0xb3abCb/HD9eZiksO7JCPOGXsniXmJcBwapJBlylsF2TwOqa0nQ/rxH8fMa/SWrTYazgPmXEuMqAwwOdudfkoECeAxukbokF3EqOqfsVhwB2SxOYRwApm7H8YRg3804OVHciynoJZejs2/yWOVBiSABe7ERacCaKg2SR7W8FTi8DChvZQKtnHjJYNQ19O3pA711h+cMCjIbif6pQ3HP6+4rTDLiHDrSPWNwkXSUFD/uggTYc5mPTw2YuhtFYJVK801iHZp0T8wJeJpvKCH1DY8rs6HSURps59xOsUnbTOEkubYi17eI9NzOAUIy152sE5NngFBSblpdEzhnSSBTLzH1L6oh+eLcMSgnA8piwwAZ4RPrtxRTXUmUwHBO8Kj3lEMDZO76M064uAqTHNYlaapNvrtQZEn8sWYkM6Nml6TCW26fnwVMaZ3jIoGokcc2VZjk7tKUzhoVzyvAY8pS5ZBAmj4kv2ZiIeubgP5MIncz5gAVXwqMaTtqTHJKRTCfpdmUK3P/uovUy5jumaiP50z3fN+wyr2QiQFNn6OQQrDGBe9lNUaYA4FyPr/Kxgn8+ZTiOy7FJKHNoTrFPmJY8nW652t9CvwZ1gz7qW89di80qnv614cTgeqOMuByk7ovE0AkUOb5dBXXKV3q2EwTIjkJOcv982RF/jmfT2+STL87fm9kBi57kaMiKFRDxP4ZVTPVPcF7Bn6ZTcv3hhri5E8UHhhWEq4SKLcRw4LdlLqnBHdYqWnCVHdJvlunxG7FtNdiT08jg1f4OckVJTmwJ4P0HAzEyT4JWws0N26qCtFdmCpJeG5EAnvJfz5ESnlrCVcxYTb7VzOgBJJXOaRjtQHtE4Ny61mRYgR0y+NQ/REYCc/zwtRVgmB9eLf6JO9TcT9iPCmgO59kqBLByEmCzO+gmPyhfJ9rFNTAehMm1guUNwPcTE/3TA4VsguN5psWw3mFUAiYnYccPOzSTEyoHFkloneW16hUDL6yfmIibaOhOw/hAtTgYRcFdGvhmiSplZh8mVEImN0I32jARyB5KUOpIfv0fxvgKz0F6kQjobcDIAR8bSCtTwysQkw+0PyM5P3GzkoAkL2jbDZ3V/ZM+I2GLKBwCdRICTk6JsoWmiBViKm/ki9K4DUF68RZBdFbgqwiVXZoyd9JcfRSCnEEV7lnUklKWNP+Rq0mABvLgpUjj5fvBu88XjJDFuIR8D1mKB+vI0t0cx2KEJMUVihJYJg9lZmlzPUk1jEF9rF387tS2Ef79FeCzL9uieH7XzeizuLvVf8BgGX7v/3Gjw34nkthjfb4T198jqt+gatqgffXD9jPS5xUHT59e45y7rB9M8eLZ/e4ul3hg4/fYjeQoru7WUCpgObFHnVh0VuND0/v8LP9D1AWDnEjUf+jW9zfzlHORszqAYM1cE7ig5dvsB1LXIVTLC/3GEaNsWPtCBKL+eMPrvAXxSWikzg532N0CmPjEO9KnFzusN3V8PMAlB5FY+GdggsCunEYVxrPf/wG3745QTxoyNUIqSLMcsDYa+BgyLq+PGBRWUQA+2+WKE87hE/nWPyDW9y9XuGjj95g21eogsDDw4whQFFANA6XH2zwsG+wmnW4vlsg9ApCR+gTC/e2RvX0gO6hItt6r+Hf6yEjIBYWwko8vdjg2+EU4zri5PSAh7cLLC/28LsKfRWAXmJ2cYD/ZoWzj++wPVQYdwUW5wfg/1pj99sWZjEg3lVQlx3Djd7rgV4jXlWw5xauU5APBrgYYLSHu6nh5x59LTB7esChrqCvC4xnHig9MEoIK1F/o+CeDxgXlA3BSkAHqI0mI+MwSW9jFeB08pCNgoB/1FBSwC0c3EzCrgKi4XV1C8qGew3EEwu3MQSXS8dZzac9xJ83GD7uETsFJxXCwgGjhByB+rc22C5mWD/Z4f6bFfdPRYxPI7IXWH1doHgQcLZCf87QoXEVERYOux8qFE9a2G9niOsR+nWJ/cvcTUpWWLUS3W/1kNdFkvhROjh/JbD70CNWAfCCQU5LB7nRsGvP+hov0F+mB4Bjl+F4zhnh+V8ajIsI++MO+KYGJCXICIDeKc6k/+SA8HWT0jwjsBMYnziMawE5sHM01Jy0UHs1MYtkaAhC3YxAy584RKkhe7Lgw5qfe3vmUL02sHPKn4VNFRcxwp6n5NMIiJh6WiPQP3XQD2rqAw0Fga0agfZFhLDA+N4IsdOQg4Q5COx+5Dngv+Pgq3sSYHYS3ROP6rVC9wMLuVes9/Bk88eTRxMGIyXkYc6/Dadh+n7IQVZRKAzPLYq3GuPaUyL3QQf1aQ3pBbpLrq94EBjX7BPtLwLMXmJchaQq4PYffkwALBzBUveUQN6m7lK79DBbhe65h+okFp8Du5c8191lRHXNup/hFKhugd1HgNlg6iYNJg3Mkzw5T34BlNVuPwIAfpaYKizQvgjQe6ogKMENEF5O0kokOakaeA8MZ0B1E3F4xvfnBNhhzWNVQ5KflmCCZOCEQK6isHOR5NxIzLGY/MqhyJ21lKW2T1OiqALa56nLVdJ7GLI01pPVLx5EUicQJGfwZg6UJmdw6CsBOyNDTfBG6Wxmvykx5nF0SSLqGgLqXotpcsOXmO4Tn8DncCLQn2uYXUR/SkbMzgR8Qe++iJgmAuSYAqYCGPZSiMmLLB1l5pwgIaOu+ghfM0zKG4Hx7MicZ4/zsKLMXx9SgnQJDGt+NjIw94YTjDkx15WUxVf3rNAZ1o9AWgpSM/s49YEy+ImgMiqCwPZConyQE6jzJZmow1OVpLJpAsaTtRWBAUreCAxLpPTWOO0TwPOUfdF2fpQ6759pqAEotx77Z3pSqoxLgeYqYJyx59OXvHbjzNBPKgDbSDK4CihLAdvIKSCqOORjlClsSWBYKlYltTLJ8OU0kUT5MF9nDpR59meKE6ASgFDoTiXKbcS4rGEOAf15AnpDhJtlABpgZ7kui/aEYaVQ3nt054b9r4mVDJWCr9XEYuueLLKdCXTPa4yzFKAUgf5JOQVBicA+zigF3CLVr+jUZdsrqDHAVSpJ6wXcyiB0nCQYVwWEjxhPzDFZeCSwdnMD19An6gtJkK0EjBZASoQNRgIpbMhVClFWUyqtnRN8qt5TMjszU/Ks6unBDAWfQUoJQBUTmxoMa1RURwATyiSxNYoVJAkMqZGhQKGsksxWo7zaw62bo/d1dIil5vhgsAizkqE9ScY6pcYmaWpoSgjv+UjMIM9oxNHy/yEgViVk2yPMqwm85QRaptS6d8Fi6sTMnZexNJTFWodYl9PLJrY4ezK1PvohVZzAogiBzWaCY0p6ScVxe0q9m/AqBOtCHi8prEjk/ZQSEX8NE/n/J49lxL8fiP6eLuqnP/3pb3of/tbLP/+X/+NPL/+L38G/+eIl7LbE7e0S/RcLXO0XqFcDxj9eo/jggIdNA/N5hfvCoLuaY7hpgFFB/3mNVmvYL2cYa+DNzQliEaD/ssF4EtDvU0rq6xIHX8AFCW81ggZuPzuDCALjQwUXJcybgpHeo4SoPW6uTqDuDRafaWxFgeLnNUZNdnDYVECrcfLBA4bbBqIMKP60gV0FBHAArpYjul2F4togdhrVLwr0QkPsNVQrIUeJ4ssS7rpGZwuG0bwqAQUMdw2EE7j3JYYvF4ifNRCtguwU6tcK6kGju5pDvyqxv2+AViMW9GYWn1dwzy3spkT9ykC3EtWdgPcacqchHgyiU3A/W8A9cdDXBbq2RPVao/UFxF7D3CssP5XYrhTqj7do/+0p1Dcl9E6hVRp6r+AuLdQvauiDhHpdwldA+aqAuVPQvWDi7DyivlIoXhvI6wJuFVG/1kxK/aMawwpY/VyifxIh9xoy1XOYViBaBQSJkz9RKDaK0rd7CbsOkI7rdxcWi58VlHZ2EsWDhLqnzNnsBapvFexPOgSr0LzSDMRZO8w/NQgGKL/VCBXlvxH0qOjXBXwTEYTA+t9qSsDuNIo7hfJBwN3UiC8G4A9XHPDsJPwiwFwbFLcai7+UE+skooAMmGL/q9cabhXhhUD1lUEIEs1rie6Zh9nRbzJ/JVFfA2OloA+UTEsrsPiFxOY/HiBbhdkrjfpKQjqB+lsygcLLyWe2+IL3VzBAfSXgPhog7guGHh0EQqdR3QnMvmFCY6gjqjeK/ktvMP9KoP3YQh0URBRovmLf6snPkzRpkDj9Ewk7l/DzgJM/k4iC9xkkmdbzfwPYWqF+w5RbsxdAFCh2AlFI1G8F+0bXEed/JMmcL4D6WqL5FqhvAEQBNXKds1cS/fOAxRe8DgxfEVh8Sdmy7gVO/kShuqNHdfkl2TDhBOZfpYCbU4GTv0TyKUUIK7H8jPs2nhA8lQ8MTWpec2A++xaAIOAsNpLgJAioXqK6ljj7M4/9c9aiXP4BwY68M1i8ovqhuUqSwMQyD5cei88VyvuIYiNRbIDFVzFJTyWaNxHmQBl1sZGwq4jlF/TNmj3P5enPeAx2xX2EiqjfSJx85hEF+w+rW4LNp3/gsf2Ix606BjPVbyVm3/D+NIejhLd76bH+dxLVHY+52FNaWOwom1/+gtJmJgELrP/SY/+BQLElWJh/RVB68jm9cIsvI0wH2AWlwlEJLL7kYLtM7Ojq8wDdAv0FsPwFAeXqC6YULb/yGE4kwVFgOFpUQPOG3l56biNe/n6HcWHIwp9EVDdANAKrLwIO75NFr2+oeij2EeU9Jo9jTsWVKRCKQUECJ194NG8C2sukbOgJEIotMJwJLF4FjEuB5SuPwzOJ079w2L+UmH8TsPqFRXepUN2RYZUOaK4Tu5kSk4czwe187lHfBjJ8KzkloC6+8RjWErM39BdW9xGH5wKmA85/1kNa9k02NxHdOQe4DG5KFS4lq1P6M4HmOqK+84iSssn+XODkU8tQK00Fw+yth6tl8pcDy1eevZ8XAssvPapbh/FEY/bGs9JlLtHcsD/V1wKLrwmAdB+x+NoBUmDxteXkUEEJLgS3U24C7ILbWv/5CD0AZoho3jAJVKVkXD1SBVHsI6oNB6zVvUf14OFKOV0300aUGw+TgJQvgOrOY/9C8zoLTjDMrjz6c4n5tw4iCpQbj/6UgKl5a2EOAXrk9XeVxOzKQnoB5YDqxtIDWEvUNxZRS/RnEmc/7zGuNOZfD+jODWZvXPL5SkhPZn31+QA3Y5BTfW1Rv+3haw0BgerWon2i0/Uh2C72DF5Cku0WB4/9ewazK4dxoTD7ZsDuByXm31pUbzuMJwXqqwGH5xUneUYCgeFEob6xUCPQnWnMXltUbzomy54Y6C6iuh3hGvr1yusW7TOuwxyYvioDZbdqpLzY1wbldYfuRYX6TYfxrEDzxQZ2XaO6GQDB51h1M6A/L1HcW5iHDrp1sKsCxcOI4axA9foAvyihDxa+0Sjue5iHAaE2cI2GHjyK+x5qDFCdhRwDzNstIBV0ayE7S/Fa6xAqDV9r+FKiuGkhO4doFIrbA/ysgGotZO8guxHCRQgpIA8jkIKDpIuQLkA9dFA20hNZaMie2RNy1yE2JRNiHTtk1bYlSHMeQkiIfmSVSNtTcisERD9C7FsIISC6AXHeQG72QFlADiOfpe0A+AC579mW1Y8Q3UCwJgWwO0Bk8DbyHoRMNSW5QqQf+DfrEPuRLOVoyUxay0RYKckSpqqRifUM/K6NXZcUORGQijbPiInZjM6nvsu/Akw9ChgSQvzKOpLJLxrx/0kK+zn+9PVPf/rT/+lvvYJfw/Iv/vv/4afv6U9+7ev9zP3xb/zYgO+5FPa9n6ziP/s//hPc2hm2rsZpcZj+FqLAm2GBpR4QIPAw1iiVw5tugRAFCuVRKQspIkIUmOkRr/ZrXNY7vN/c44vDGa4OS3yyusbGVui9wcFy6n5dtmi0xRfbU6yrDrddg99av8V1PwcA7McSJ2WHh6HGy8Ud9rZE7w0qZbEqetigsLcl3rRzLIsBRnloEVAoh81QY28LzM0IISJOyg57W2I/ljiv9/hyc4rfPrvCz+8u8WS2R+cMWmtgZMD1do7TxQGrkttwQaKzBufNAdYrGOVhvcJ9X0OKiJOqw+A1OmsgRMS8GLHpK/ggYFTA6BSeLnb4ZrPCohogk5avdxo/ObvC/3P1Hj4+vcFf3DyBUR6lcTgMBQrt0I8GPvV2/u7LX+CzzTnPTV+iLixGp+CDxPn8gK9uTlBXFoX2ECLCeYkn8z12YwklIm52M6znLbrRQMmIEIFZYXHf1vjx+Vv82dtLPFnusekqaBWw2deIEVjMetTG4f5QQwjgcrnDzX6GrjfQOkCpgHHUeHl+j7u2Rm0cdj1n73Q6V7vbGcxshJQxJWYHGOWxP1RYrw5QMuBuO8OsHjA6DaM8pAzY7WucnezRjwaD1XBOYjXvoWTA9e0CT8632BxqxCjgrOKsn4g4XR1w/dUa9XkLISLabYUPXtzi6+s1itIybOpQwJQOxni0+xLRS8jCo65HzMqRDHSncfp0g/u7OXTBeFMhANsZ6MrCGA+tAg5tCQGgaQbsNjVWJy20Cti1JUIQkDJi7OhliXsNtbLH54AXECqiqkdYqzBvBuz2NepmQN8VCEEieIGishACaKoB/WjQdwWUCtPnFFHAd2TMY6qraE46dHtei2o2Yhw1BACpAoKXcL2GkBFCBcQgED1nMZtVh25XIXoBWXrK4R0Z63I2YtiXUKWnxaTTkJVDOBhAB4giIHpBNrf0CJ2GnlumSI8SUBFy5rjOXgElO2pjp6CWFn5nIGcWwSrACfaiApAmAG9L9qw2nonSEux4XXgyxyXl7bJVlH2nAjYxCsSFo6RfREBFHg/wiP2MZOUFAE0trugk+1tF+rsA92dQfE1WVwQy13E9AlvDkK4qQLQpalaQpYMAwsxTEl8FqOUIv6GUf+pBdUmGmVjkqCNi5WFuzNR5mpl/1TEsCjrSy36QCGVi7PvE9GafomGIl2wp44cA1QUhMXAjJs+56hjoFKrACbqerDVltgJuESB7OcmS834Hk7sm6XUXTsDNw7Q+RJEYl2MX6CQ3dSJN4GDyR2aZqUznQbjEqC5CksuLSYbOa8j1uCZO8tSYw6/E8e9RJpAc6cuOmpMGURx90MHESQIt07WJmhLoqJFkqzElcaYws2VOP80sKn2cWdqeK3ByTROAiRnObG3uDIxJ4uzqiGLHypNcFZTPWw7deuwdzp2sj73ZuUsyByypniFQmf2VHpADz0++X+2cDLBKsnhfcVKEUktMvalyxOQRzv2gWQKcE2qlTfLzFJgUDIBHkwmImPzEU4qr53tyr+s73Z1I0nCLdypyRIyJqWYicjCUoucMhFCk5OKYzkvyasskhZ+CuYZ3uzpDSsXN51GnGhYm7KYEbUu5su6zB5me2CjEsQvTPuoCBhCUmGpYgibwj5KBTAAgPZOBbZ2k6inBlucw2YaSNFoPya9sGE6VPxe8L46VN/kekwm8ky2Nkxf32O2bZNBJ+pz7P2mdoFSY30FH7yW3JyaWMRi+ZvKqjiFVGTEdmz73mLo2MYUMRSGgOjcFM+X6Fr0bKZ9VyesZwXRbxd/lnszco4lAf+XUJZp7MFONiXCU54YURJTlu6HSyZqV9iHJdKfF+aNM9nG4T14e4QFh3fE1j9Nlw6P3JnAZH/tHY8A73ZvAL/suHy/5b/8eYTZ5PXEY/sbX/qrl74QUVp7F3yv+s1/7en9/+N9/48cGfM+BZf3D5/En/+qf4m4zg/i6ZhAPAHPRYTwUEJ2CyIXgVQCKAH1j4GcB5Y2CXQQyD6nsXh4U6zTqVHpexslrk316UQB+5SCshHmQ7DVsOHAYLh2aLzWGUw6EqmsGP7TPA8p7if65hb7X9BqtA7dVALkGwlURbklJYjSYUgWnAV7ycg5nHtVbymLcjIMI4Y4DGDejZG84ZVjPuE7eqgMHEeOaPj67iDj2DaYZ+Q3ZBek5SAP4gM4F26GMDMpJ4SHVG4nuaUD9hpI7e+rJrNVx8jCuP7xH+4fnadBFBs3OOZssnJj8RAAHDMNpTIPLNDh5oAeuf+Yw/0JjXKTQozS+Dialq4Lno3nNig7f8Pj0gQmsZsdU1DyYcsuA+ltFqWAEv5CtmPr9crBMFPQnjcujbFMNmAZMcuQ1Nlt6VaNJg7jEMmVvWnUj0D3h9bInHuU1K2T6M+4bt8EHdLERU5pn0BwI5QAiN4+T7LL5WmI4o4fRlwwPyv4+XwC5XoIJhekcpw4/u+C58iUm2STAwUZ/FlHeiel94zqieks2Loc1qTHJACWlkWQzGBpkF+lDGjCFC+ValtnXSUa2ICCYfQ2MJyktteHgsdimDkzQ36p6MdWM+BqYfRUxnJJxq9/yO2xYc2CYA3PygC4npbK2hf/ev4yobij9czXXU96zkHzM4UCa0s35lxK6PXoRxyWPRbeUd+qOQTzVTQruMUDxwHOf00Dz4DkvvqI3Mfef5loXXyJ1EabB6UApoV1w8K939KDKkQO+XA2BCLTPBKpbrienruoMTMSjYKrDcX/o+eM1rm7JBvYXwOLLiO5CYPY6oH0iYQ4pdKbm8bjqWHWSz3ceEJrdcZDORFj+XD7E6fiA5DVcpGRDYApEWr4K2L6UUy+knbO6pdikkKCnXFeUZNl8yUTU8oEdkObA4yl2cUohHVcCZe7KFMd7i/tBmaca48TW5uMDjomfvuIxqhxoFBhoJJJ8c5xTJsteUZ6v3G0aJb2Qh+cSqgNmbzz6tUT1wOCg+oY+yXw+uwsJs48TGLEzDrBVT1lmDpMpdvGd50NOjM3SUOFToFRJ5i5LeKMkwDFtCtpKn5XiEKY0VURMnZO6YyelL9Jn7jbCp/RWdhXmShmer9mVx+EyhdB4MoX9WqHYEUTYhrUh3GfJAKVTiWJL72T2YorAZFYCswjTklVUY0B7rlHfeuRaGzny89ueKzQ3rL7wJeWpwjO0yBcyPVdYrWEOyd84JBAmj95RSnIVTMcO0WIXJkDhKlaMtE8U5t+6KVSLIFROoEaEBHIseyqDEUBgb2YGTEGRWXQNg44QI1yj6OctJcqNn3yWZh+mkKDhVMPs2SWJkNJhLX2TauCxmr1D1AI2eSejBMzOwS40dOsnGWrxQC/deFIQTCWQpQ8OvtHpGUhWVO8thrOKfyspRzWtg2odhrMK0h4B0ePJE3PfE9wrCbukJ9TVCrp1ZBlT6BKrzkb2UCoxMYhRElDJkT/7Sk+MpRjDBKxiIVkXEiKEDUBKj1W7AX5RTqAs14P4KcWXNSUx9XQKFyawQwktOzVzNYmfFVAPLaAkpaXWI9QGctMyMAcgyw+CtGg0JaRSQuQakiRvFW3PehEAufZDDJYhQNnjWBiIYeR6RjvJUidg/Oj3U0JsBqNaQRw6Ar2mpszVPZK6ftdjCZDRfAwaUz1J7rOMyVMppEQ8HI8Z1tFPGSOE1ojO/fWJr+kc/Y2psI+W0Pe/8m9/0/L3wPI//PL99lgqj//6gz/E//nqH+JORMwqi8O2wmreYX1+h04Rr1QAACAASURBVE//+D3MP9zQg7it8eR8i8NpgcOmxmgFZ8Ibj3Ix4CdPX+Pz+zM8nMwRnUB90uPJ4oCvPr8gaCsDLl484Mlsj0I6/LtvniPsG4zvOcwuD+h7A+EkuhcC9bM92rsGbR2gNwr6skXczPD85S2+Deco1j0waIwLQOqIorTo3jZYv7fBoSsxokL9pEUIAuNNTUB8Z+Df7xEGhafP73FVnAHgzL2+6OG/buBPLWAlhBXo5w71eYtuXSCOEggC9klE9aqAft6ibVJnQunJdgR6L7u5gVoP+IcfvML//UefIGaWxAuYBwXzyRbPFwd8+ekT/IOfvMKffPoequWATjSIC4cPXtziS3kBtWMf5/Jij/OmxV88dWRfqgBhJfRZB/9qBvFyD/vVDOFihH5dMEH1RYf+roI+7SFlxPB1A18HnL//gNuwhj7rMLytoc4HxP+3vXcPsjU7y/t+73ff19597z73uWokgSRkGwkndmHkSGCrgD9IFYaKSYzjxBUTTDlJQagC56+QOGUHVzlOKOzIcVzYDiE2oRxjIC6oIkYICdB1ZjSa0cy5n773vn3XtfLHu76ve0ZSSfKcOUdnZj1VXd370nuvb+1v773e9Tzv89zssfHOPe7entD2K86DBEQjPIKlW0j0NZ5ktDOlLCOaOuTC2in3ptvUuyW9Ua4s2zKi6UdUEzVcavfFosOIer2mt7ok/OiYxUXjiryG3u0IdgvM5YZymjBYW7I2WHDzM9uYlRqZRa7vSSg3GrL1JfW9PsXFisLCZHPG8Z0R4SwkKIVqpSGoIordCmpRx99LJXYR0ttasDFYcu9zm0yeOGR+us7gHUecvjQhOdIFWLEKxeMFUVZRTVNkEZIeBFz99pd57tmLxCeh9suhsragUoOqAjA9S7IfUq2qE140F5L3HrE46lOYhHrgTAxiy/jZiPxqwfb2CUe/v0lyJDTbJUWZUD+WK5OYhyoPXymxTYBdhkwfDxjcCFheaEi2FyyLEfVIx1Ot1RBYws8lzJ4pSW7FVBNDBYTzgGgp5JdKwjyhnFh9nSTCxJZytyQ4VkmT9hSe9Rb2bkWYp+dUnxlgA5i884DDF9aAgHxdo1qCWnsMlxdU0smHD5gfDclnGVFfmD9RkV2PyXdropOQ5U5D73bIchvk8Tm5HegGwKQmPYyZXbW6iJw0hKOK5sStxAXiSUF+mBEf64aMSlWFamxohobsVkRQaeF6+kwNsSEeltSA/dxQ5bfjtrjTwqHYrpEmQiwsL9aEM5Up926EmEQ3JuoVw+j5kGJNNwSqNdOxYP27ActNKB7PkSZjebVicCtkuWXhnlCsWd2MAn0dktYUS3sty2sF2XMZYoR8Xce2eLzUjNqNCvucSshbBqx/B47e1UAA/ZcjwlIo1i1TGzC/2pCcBhRrQr5hMJk62Q5fCSjWNfg73zSMXgoox7C4UjP+XMRy92yTRQycPgHxidt02tXNmvQIptfUmXn+ZMnq78Ust9XoqnlmhvmDoWZYOgdpk1h6d3VsUgv929qH2vSU4av7tts1aHvgBre06J5f0A3H6DjEJDrW8mqDmIDZFQieDzh6uyX+mDC7ok7V0QLmlyy9O2oeFS20z0/7RY0ydOjm1NofhM6kRjcFqqEWcq1jdDzXjRKTwjSE5FRluPlaQL4Jclvdl6XRrNxiNWS5oxsYNtKNh9HLkAYB+Wqg0Ti1dDmu84taPExeUElvuaKbJb1D7cesRur2HC+0rzYshToVZpf1fFvs6DHt/I7G2iw3hOHN806zAfm69qRKA9mxbkAtNiNmlyFysU8aZ3HWg9k64lb9oJuHYiXQuA80ZqYaBtoeMRQGd5RJrgZnmcD5mvaulsuQcgX6d4V8Qxhe12xY0LaFPA+7PksbCqdXQ6KFdbEuyh6mU0NkDdNLEf09Qz4JKFaFtedqjdOZNcx2Q8bXLVU/ZLmp7qnzi7DzUWF2IXQ5uSp7n11KnCO4ZbkWkcyUdSzHwuCuypLLYUBfdL5Or+rzFqOAfiDkayHDm8bNRUg0b8g3YjVMWmgf6GIzYHQjoJgErk81IjuosKGw3AhJUmXXqr5QroQaZ7URUoxjxtdrirE6Ctf9gGhhCKpEi7VAWG7F9O7B7GLC5Pma08cSVp/PyTe0DzKoDbMLKb099XSoBxHFJKJ/uyBf6zvDokidi/sRSV1pT2o/ou6FBL2I+LTExmHHGFI31P2YMK8JrKWcpMSzinKS0GQBYi3Z3UI3KuKA6DjHpjEmCjSaJA4cS+dMiSapOs7GIU0aEh8tNe5kGVOvZOecdA3NKNWsz4WOKTycY8MQiULqlR5R3WBGmWZqAlgtVluTnmA6x4x7BCcWM8gIZmhBuiywWaJS2tDtXDonWBujt8cRbQYmvTaYOtT/qxvIX81Mds6tvVSltdZqbEjj2M5IXWbb+BDrDHgEcLIu5LzDq5PcShTpYwTBmWkQQOXyTs/lW35Zwqt1rX0T9CZaUHfbNykeacYyvXrJ7vz1HyHs1ZiDFFxIuo0NMqjhNCY9CCmuFRBYNcIpnOxMIBhU2JMEG6uDa7XWkN4Lu1iLthjIN40GycdQDwzpgbKdYa7sUcty6g69k0oFEE2lkxg1mS5cAOqRIZwF55wetfDp3QtY7OrjItC7rU6arTlHUCrTBzB6WXeO8zV1z0yOA/IrJdn1hMblGTY9NSWxkWMbE40qqYZnMrPWLbTJtJ9HZR1nC6Z4KiyeLOl/PmG525AehOTXCuLbCSbVx5tfqxl8MaIa6464GCg2G+2TzCyrnxVOnmyz9JxzZ4pzibRuF1U659FqbOjfVFfBVoYWlMpABoWONXGup638BYvKC3MhPdCFWVjIqxjQoKKLw1CjC2U7+neEfENZUKmUlW0X+wivYsuqoUatgLJZyXHgMuY00zHb19uqkUaJ9O9pD1O5aonmOp42fqNlFFu2oJV3lavKAM+eUAa8lSCJOZOemTZGwx17Kw8rVs8kUkGtDGIb8bLcsvTvSMcapcfqdhk0uF10XcAmp7oojRa6sGxdVotNZVnjmTqQtkYp5VgX3qBOp727uiBqnX7jKY6tcuYpkZqGmFgo1ujiMvQgXewIauQSz3ROsBp9Ug1xZiFnr2XddyzrhmX0koszkTNGLV+zDK8ra9VKolqGPDmBaqxjCHNlrKKFnlfLLX29WilZ3XMMYKDSuXLFxVw4a/1844wxbue3dTtt2VIT6RyHOZ3srhrTyR6TU2XtlInTz5z0SB8nWtruuMqRdLEe8VxZ1+RU3UODykWHOLOayDl5al4iDO4Y5rsBg1uGsLTMLoRd3EWTCsNbDbOLKoWNZ45lrHX+wqUu+EfXK6aXYqqRmqBUw/a94uIt5tbJNXXzIj7VcTbZ2XEHNd1xqWyUjlmbXdTokibVxfrKixUnj6k5Seto2jJ1dU861826L4xuNCzXg+4+UU7HSEnjnG5rSE9Nx1y1hkC9fY0qaSV5UW4oxsqiJFNDsRJ0DrhtfqPm8enY47maxrSup03q3m+lOqrGC0MTi0ZhtKd8owWRSfT/I5cvWA6cvJkzqWO8MJ1BTVDpYwYu21LsWRxKdqwMY5TrZ32Uux5NEZK5MkpLZwYTFfo5kB4b59AbdNEhJhKyk4agtOSrYSfXTKamez+35j7RwrBcDzuWtUmhd2iUTT5qKMehy7NU06HW9RggmalRVr4a0tuvaTJlmZQpM84xNlSGsR+44kN/rOjmRljqXNR9jfRoEi0WgsJ20sTWkMkk2ruZ7VeIhXw9JiwM8UyZPkRZRX19lB2u+gHR0pKc1B0bWo5DssO6iwUpxzFiLPUgJFq0TqKOOY6k+35sZbMqtzXOhCokPSioR7HKNEOhyULCoulY/VbOWffDMymmsc6ExpkTucW3DdWdVjc0IrL9knIlJjmpNDOzMt1j2kBlnsbJPqU5+241kWisSBJ2RVXTi2l6IfFx0TF8bR4lIsqaLtS9Fejkml2mZABSapFpA10vteY3YixBoUY50hiColYJKWjMSD8+ey7rmMbyrDiySYSJA4Ky6eSa0mgEiRQVGCAUmkFyFkXi3ofBsRZw4h7TJi5zUoSgqJSNSxONIEmirmdS8gKCAKlqmvURwfFcC6yywiaxMo9tH2M/U6ayff87eaqNlT0UY8/cX8HJX/WxbRIjixySWK/L20JUi0nqRpnFLD1jJbMzcx7yMwnpq5xd257FIDz7uzXbqWtlf12OpW3lsEVx5hzrzIW0L7M+G1P7XOfiUiQ8++zT21rznnNFY+s0+xXwqLvCjoN1+/7oQ/f9cX+t+oWHfmzwiBeWa2/ftM/87F9gnOa8Z3KDjx9eYbt/yqxK2V8OuTQ85rnDTWaLjA88/jy/ffMxPnjlWb4w2+CZ0V0+cXSZflQyrTJeeH6X7auHfPP6bf7NzWv86avP8Zs3nmSll/PUyh5HZY9ZlfLivXWaOmRn44Sl65+7MDllEJUcFz0uDE742G8/Q3B1zsbKjMYERIHh0uiYT9/dZXM0I68j8jKmakKe3rjHJ29c5ANPPsenD3fJq4iqCVkfLHj52R0uPLXHldERt+Yr9OOSKDA8d2cLEUscN3zLzg1uL1a4Ox0yn2Xsbpxw53DMkzt7PPfiLvGwZOB677b6UwKx3J6PmeUpu+NTosBwnPfYOxkSBJbN8YzjZcbs5pjRxVNW+0sARCxRYHjpzgZyI2P9m/e4+/IakwunHN8b8V3v+RS/v3+Ruy9s8Njbb3NvOiSLa/ZvTBi8FHHtu15ifzHg3r0V+mOVMYSB0R47YHbcozcqWBsuuHl9nbWdEwZJxfUb62xsn1LUIYtFShw35LOk6zs0tcBxwtZT+yRhw407q4SxIeuVLBcpTREyXF2wcP164c2Maq3Wfro85OK1fU7zlNlpD9sI/XHOzsqUL95eJwgtzUGK7TeM1+fMZhnmNObC4/vsHQ/pZRXGCsOsYO8zm+ryGxuyVxJMZOm/64jFp1bhqTnlIma0umB6WzWi8bGygpI1yFFMuLOkqQM216fce3EdmZQENzLq7ZKNzSmn84zwk0OWVyvtwxs0hKchdqvAzGLS9SV1HbK2MufweIg5SghyNRsK0oYkrWk+P6S+ULKxecre9VWi05D+08dEYUP96xucvlOl2sNnjpg9u4q9vMSUIelLKfl2DbElOoqoNypkFhJsFJj9lOxuSPmOJWuTGUef2qDeqMhuJMoirjbKuK4YbGqQrMEWobJvhxn96yGTb7/D7We3MKk64DYDw2T3lMWnV6mvatxOUAjNqNHNI7EMt+YslwncymjWaqhFx7bSIP0aWwVIHmLFEi4DzGZJ9nzG1p+8xaKKOf34hm4UjTVix4YWWS1VhvVS5gx5tLiotiuycYF5dki1ZoiPA6KZUKwbmnFDOskJf39EsaGbQPGJkL9zSXg9U+npabsppZsdoC6n2b2A5dWKeD/qNo/CUjcc8h2V24ebOdHzfZrMSZFTWDxZko4Kyj3VcwaTkmYaM/lUxPE31V2/YjQPCApXcIwMQSFES2F5pULywDHc0knbq8sFnMT0bodU754RfnZIfq0kuRlTbjRE05DkSLo8zmINECg2GuITNQbCQH6xoncj7vJum0ylwbNruhC0kcbI2CcWpJ8YkG849vBSw+B6SLGmmzD5VoNNLIOXIrJ9LZrnV5UlLld04WtD/f4a3AiYPtkweClkua0bT/nTOf3PZhQTy+iLWhgvdzUCaPwFZ5RUQzHRcVqB0SuG02uaBzm7ooxo3dNNjGihhfbpk5bRF/X6Lq6nVGl3u7Ewu2YwPcPaJ5RRbI2xEpdBOX0chl+Ek6etGp09qZJrk+pGyPhFdfYVo0VbPLVdAYtBN9EKy3xXug2geqjS9nbT0ER0GxNhoVLxNu/Uim7U5GswuKny/MUF3ThITpzM2m2g1D3deIsWQv+2xmAcvU3U6Xeu/YynTxtWnlVprYlgdkXo37JkR4bZxbDrnyxHri/V9ScO7mpG52JHmVQbwnIzIMytGgE59jWZWvI13fTJN3SjsWUxm0w3q6RRmX60tCwuCOmhHmvvnqEa6MZHOVKpc1i5DYBGc3O1kBSyfb2tWNFzo7dvXF6nFlhWVHK+9rmaw7dHrD6vWaety3E1EuJTbX9I5lrctxmfTaYbEto/aQgazSltZdTpictkzfVzx8RCb7/h6G0Rky/USKOFd9u/Gy+sc1wNXP+r0DtQeepyLWBwV1lQBEY36q44nV4MmbiNGiuw+nzJ9HJCf68mKA0mDaj6AdlBzXIzJp7rHCSnNcUkYrEZML6uxXSYmy6vcrYTqbGRsZQrEYMbS5bbGTaA9KhmuRXrhmgN6XFNvhYxennJ8dN9Vl5YMruS0dvXIqYehqQHFeVKjHHxLm3vZ3pccXo1Y/yiGg+FuRYtxm0IJEcl9TDWDQI019REwunVmNXnCzUbsnTRKyqvVebRRtLJgFsXWrEQ5g3RrNR4E6s5ovFpSTWMieY1NlGGOZwV1KOUoG57MtUlOb4z7WStpp+obHlRqlPsosAMVaLb9JOu37LrnZwVmH5KMM+xWUwwyzH9TAvNXoIsS3WWXRTYQaZs4qJQ9nGQIfNcj2dtSHC67CJKbBio0ZDLzHQLPZXQArS5nFGoBfVAe2lkOu+YTX1TV0iWYkt3/8Z0LrFdkRmcKyars6Iavkxh+dWKyjdBjuVY1uz7ow/e98f9tfqfPPRjg0e8sFx/+4Zd+ekfAyt84Kln+cS9y6z1Fmz3T3n+aIuVNGdeJZRNyN71VZ586ja3jscMspIkbLhzOCbLKkQs64MF906HfNPObU6KHl+4s0mzDHn7E7dYVAl70wGboznX766S9UvG/Zy9oxHNIuLpx+7w/HMXePptt3hpbw1zfYCNLL2rU+YHfTZ2TziZ9pDAkiQ1cdgwX6asjeecLjKSSM1gev2CPI+xJiCK1cgmCCxNE1AWEYNhzuy4z2iyYDbN6A1KijwmCA1NEzAeLhlnBa88u026uyA/6BHOAux2wXi85PjmmLc9c5MX725QzRI2LxxT1SHTecbKaEEcGuZFwvpgwSvPbjO4PCWLa7aHU146WAOgLGLqaUywDEkvzSiWMaYOyIZqNrScpmTDkmIZc2XnkJevb0AZsHnliMOTAc1JwvbVQ0Qse5/dpJnUKt/NGqI7CdW2bmNLoLmiQVZjlhHxuKCpQkweIbEh7lXUVUj0xYxySwvFIDZEca1mM1ZlmPFKQXWSMt6ZcnpnRH9zznKWYhcRZI0zX4kgMkT9GuPMY8w8JlnNKY9TNWw5b/SSGMgDgnHV3TccVWpAM49gUmLrAIkNUdxQLWLiQUl1nEFgyVZz8uOMeFRQHWdE45K6CAkPY9gp1BxmGnX5qlFaU+cxtgrAQHSi0s+Wsdl82z57hyNdtFvBLkOkCGBSqQzaYXP3hIOjIaYMoQxI9kLKLY0SSe5ElJsurzXVAiDbWJIfZVrMpQ0UoRqouEiTZrvAGiF0+aw2sgS5Fhj1uFEDl16D5CE0QrS1pDpKO9VANA1p+pr5KosQO6jpTXKW05TgKO4MOYzLW7WDhmg/dhEXej2xJb4bq6nIPKDYrLUXOjHER67Xt7XoBzXQcYYt8YmTSe+WWoguVJ4TLYVorgVYdOiK1V5NcC+FQOe9GWgRHJ8EFBuNM5HRBZBJbGfOEpZCsWq0D68SmtWK6CCm2S6JbiU0mSoV6p4uoPNNZcHLVS0E67E+djgPaHpGnX4zZdfPsx42VjZf2RhnZFMqAy9GOtOVNnu1zQ1t+4ul1oW3idzcjusuT7fYaIgWAU1iyfaDTk568nRDNDszwmnz+lp2LSyFamCVnah18R8Urj91rmoOAn0tkpOAOrOdGU411p7fYk2LPhMr8xYupeupa3qWaKa3JVOhmKjKoC04qpEqHuKZdP3qJlb2UKxjfsfak1msufiZQhnl2v1f26/ZpHQ5qCah61MMc1yB2zJTdMWmSZRxr/t6/IFTJRSTM6OdtsddGmXjW/VCNXLFbvDqAqpJXMblawxxopyuN7jNlT3f04vQKUXSQ53LckU69r5VPkQLLeLqnnTKCnGmNW1fvUm077l3T588XCpTHM/UVCVcag5stNTiPZpDa2jUMu6AKxaUSSXQ86ftGbai/dyteU6TnGfElcU2IQxvG8qBY+qcxDoslCVPD7RAjpbKfLdqgyjXAg6r/bvRUrNHi7Ey1eVYC9jOWyDSy22PYmsmpIY+Kk3t+u4TtwnhFAliXO6sW0ufqU/0mIKKjhms+5r7aUI6Sa/mhEo3B1Hh3p+hEDSaj2piXA80r2K4257O1ninjZ1R4x7H8DZnxj1RrkVqey607G6TKtPfKibqXtAx8tgzoyBw74tQz53k1FANA0wI/T0tePV8aShXImd0pKyzifQYbaBsMtapDCLpej5tJISFso91L+x6OZssJDk51yNaW5eVadxYpGNvbaBmOtJoH2t8WmHjQPMzoYsXqUdxxxRLbQnzGhMFXV8mgZONhgFBUasbcdFAbZQN7cWEs5J6JSXM6zMTnqKBUJyja6XZl6Eged3lYLb9l63pjo1DqI1KUutGi8hlqZLYvDzH8p3JWV9lzNNKWONIGc6q1r7QJNaezJaBdMZCNMo6E4gyoC1ilcC+KtvSGJXNtsVhXSuLWRSddLW7zRgkDF/NNp7ro2wluK9FW6NotImO1Vbll9zva8U3SmH5vuDfu++P++vmn37VYxOR7wR+FgiBn7fW/sz9Hkfw1e/yjYvahPT7BePRgtJEfOjS5xglOfeWI7YGM962cpdeXHFwOGS0M+X2yZi3b93l0uiY2gR869WXubJ6xNXVI165tU4cNQyiks3ejA8/8ylGG3MCsWz0ZnzrxVdYy+b0hwUisJot2ZjM+La3f4EwMPS31ZH28a0DmlHDd/zxTzHMCq5e3WOc5bz3ynXqKuTq6hH9pKKflaykOZdXj5kvUuKkpp9UDPsF/X7BZLhgbbign5Y8sblPmlU8vb7H6voUY4WtjVOe3rjHU7v3uLJxRBgaTk77XBoeY4cN/axAioD46pxve+IlAMYXplzsn7AyWvL047e5tnLIe7Zv8vTuPTYHcw5PBjy1vscrd9a48PQeG8M57926zt3ZiAuTU+o6JEkrXeSvlSz3+wxGOSxD1kZzhr0CjKgz7cqCg7nGnEQrJVlU857LN3j3O16mrEPu3Fxl5533oAgY7UwZTRaET8y4fOGQ5GbMhZ0jwkHF+tqMyfaU916+gckjdi4dYo3w1PYevX5J/5uPwAgXdo5IsxIReM9j11lZndPbWJD1StYuHjObZqRr6jRqq4DhzgyMkPYr/sg7XyQeloRRQ9Yr2Vyfap9faJCeOotevbrHzuYJwTLg2972BWRY8y1Xr5NkNReu7WNqQYoQO6zpjwoktLzr6k3qImJt65TJaEm2vuTClQPyacp4c0Yvqxhuz2iOUoLIwG7Ok7v3SF9MwcLW1gnRzZTJSB1CokGFZA3NdoGZVJhBw9Yze9y9vooElicv7PGOK7cJBjXBeslovIQyILuRIIlh8VubGMfgbl4+orxYsbp7yuaFY/rvOiJYBvQuTbn6+D2CZUCxjBltzYhXCqK9RN1LRw2jS6fUGxVhrJqu7F4AKxWPP3UHMbDxzfdI9kJsYgiPYnUnjSyPb++zceWY6MKC/vYcubIguxdy7do9bGKQyLI86NF/PmX3Hfcww5pwZ0mwDLHDhiBtkCsLdRtdK5HUqFvrY3Oa1VozPCclMimVGb2yhJVKnTMvLAgLQVZKxrtTonlAuVUz+qaDrmAfXj1B1guqrYr8qZz+S+qS2hqAIRa7VWAj6F/X4r7YbpTpBJrM0AwbbGKox4b6Qkm+U0NkGbwS0KxXyDzChpbk5RRpYOXJIxYXG4qtmuWlBtM3VGND7+qUetIQTUo15bKoy+lSMD1DPW5oRg31Ss3o6SPNVF2vqC8VOg+F9urWQ8vgvfvUY0O+02ASS/P2me6691AJWik0PYNcXujifNQ4ibZQbDVqCNazWGdmlm82HL+70jxUcPmbGqkBynyaFKrBWc5p685aXqywGyXV2DK6ekJyFJCcaLHcf+YYE6vhV7YfsLiir2k90PYCMdo7jOh9orlQrhrqkZP+XluCwPxaTbFhKbZrkmPt0SwnajpWbNYsr1bOOROyfZhfaYimyoK1svp4LuRPFix2LNVY5fkmtZSr2meqsv4z6bjUai4WuGig5dWKYqvGRJDtaYGpMRJalCy3jRrxrFjqnpp/tQXlcscwuKWbAK1c/7zxVlBBb8+SnOh4Ftcq5heNRmucWuIZzC8ZZWPXXYGU6rjqHkyvwezqmXGUFm9OIhproVAN3LGdWKohzC8bygkuQkdfj+WWFqj5plAPHANcaR7m/ALMd7UfsUlhfsF28v/jd2uV1fScMVJlWey6YnShzHTT06K/Gug4y7H+b7GqvahtK8DpVe1zXG4L8wvC6ROWxbZQZ9D0tMezXNFolmqgvZ2nV4Pudbah3p6vqcS5mAi9PUsxEbJjQ3ZsCQuVorfGR6dP6CZJvmldG4voGLaExa7mlqanhig3TK+obLkc63sjnhuV6uMMj2I1wlpsK2M5u6A9n1HuHjvV4wKVoC83AhZbQde7WQ20UG6dYU2om0vlUIuXcqQ9n9LAYkNlyjaEZKamXMUoIJkb5rsh5VDNg0yk/ah1Jk5OrdeVo4DZhZDlmo5HJd3intsy3w46N+PWqdUGuHzZkKqvfZ/LzZjkpKYcBGR7hZon5dp/ChBPGzWKSoVqEFCNQhbbkZNQu948gWoUQqCv7WI3ZXo1042EeY2JA8qVmGoYUY0imiwg30goJzGHz6Q6hpbZTEPCvCY+yqnGMcV6SriomV1I3XmqRkE20cdsehHL7ZRq5CS9FgiF5U6PepIqY2gsNg277EupGo0fcS60phfp36FAo/cVa7FxQLEzxGQR9aSHfZVhKQAAG5ZJREFU6SdQG8xQ4+xMXx/f9hKCec7yygpm3CO/vIIZZJQXxuRXJphhT5nGKMQOe9h+inWSWrM+1qLSSXDr7Qn17irN5or+7K7RbE/09Rv2taCMo7NCtZfpTxgivUwjSQJXiIIWnuL6KM2rI0S+nDusPcd+2sZ8yU8bb2JbJvSrMJoeXxkiEgJ/B/gu4B3AnxORd9z353mUGcvh0zv2ff/LD7A/G3DqGMH3XL7B5w82WeYxaVqzXCY0eUQQNzx+YZ+X760hgaUuQyS0DPoF80WKbYSVlQWzRUq1jJFFRLhadFEKq6MFR9M+pgl4fHuflw/WKOZJFzWwNplxfNqnyiONK8gawtBQnaQkqznV3R7vevcX+dTHH4PNgiSryfd7iBHSrQV1FdKUIRIazDxm9cIJx8cDAF34CkhkwAjWiMYsLCOCQUWUNJQz1b8nt2OCt80o7vbp7c7Ilwm2EX2MWhfQ2eaS/KAHoSXs1xqT0EYOzGL90h3WhHcSZZQiZR3CWUAzaojHJXUVIvsJpq/sljRCuFbQHKb6v/cSmp2C9AsZH/ru3+X//uw3w16KjTXzMH8qJ85qmjokuJ45ExYXt9BXQ49sklPc6WNDS3Yv6sLn63GjcrBeQ3xXWR87qbB5CKEluxGTX6yQUiMXkoOQ6mKpY+qrWQmhRbKGYD/BbJYkWUVxlBGeRhBo32y+o6xbNHdxAs5xN5rqIsS6qIT0IKTYaHsINC6hXq276AjJGphFRLOAuqfPbVND4Mx66g3XvB4Z5DCh7Rfd/jdw9/04GajRCIvIwqAmeTmlvFoQ3k0IH5vRvDTUPsCBGm+UOzXRUaRFhnP3TQ+F2WO1MmCzgMEtYXbFEM8C8q2aaOqyKAe6CIynumve9Cz1esX4kwmLXYuNz6IIyolh44lDTj++QbQQ5k+XhAcxNtbnbd1rg4sLeHHgHJTpmoyiK3PMi8OOeTGpdS6zmgtqIp13adQhuGXd+rfUDdfElv5t3RVfXNad0MHLUcdQVEPXl9vXRXo0V+ajvlgQv5J2zGK5akmOhGpF+5LHL8L++7RPOCjPGLL0QIugVlKI6AK3XG+YfDri5G0NyZEWgNmBSvKi+VkR14a555cq0jvRmcuvc2wWo69h7Hq6Q8fwSS3kOzXRLHTMmWMnY+v6oXWuooV0PeI2OGOc2j7roFAmaX5RXX/rYcuAaK9x3VP2bnBDyLdUUtrGKxjnHtrKEPN1J+21OOdV7fcevhR0rqvFmuvrTdpecmUI25iGYlXnMJ4qO5OvK7tYD7T/VRqYX7bUw4b+9UgdjXtnURLiXJmbzLm7rmkhZ2K3wE00asLEygyen5NyrK/p8BWVpvZv6UJde4DpIjCqoY4v33T9ynt6stY9XB+oEE9VuqkSO+f+KefkqnnL1urrkO0rK6eSO3W0Vaa0jSXRMXQ9wa43tVjXfmaTKPPZu4tzqtT5bnrKrmWHytbMd9RptZjoeKSG9ERZsWooXb9z3Ve2NjsylENdrDcp7rNd+4XrnhZRYp2kN6ZjKnGSXdDir3/XUPWEaqw99SsvV0wvRl3/Zd3XIrTtF46WltqxiP17hqqvPZgqCXXnnQuvb2LtC12uB/TvmY5dHd7SXMkmhcFtdaFtxxgtrBrcZFq0VX1X6C0sxVjoHap5z2IjcNmXrsCaqFFRkzhm2Wqvbcua28AZDbVxG6l0ShITtlLms77ZJtY4j2qge/rlWEimLuIiOutrLSb6/o7n2pdbZ663d2ZIjmuWWwlVX91rW3ZQI2UaTBqoQ60rOPV/1VEWUfltvNDjCxrnrWAs0aLBJEHH8IWFvlgmko5FzPYKqlFMPQhJjmuqscpFw8Iqg2j1uBHNyg0LZRuze0uafkyTau9sUBq3uRWSHpVdb2or1w0LZejqLHRRHxrpkUy1R7IeRESzStnBNCTZW9IME0wSEJRnPahiIJqVnatsULlCz2ikiNSmk382vYj4RHtGpWgw/VjZyDgkqE3nPtvmQOprFhAuK+phogynQHiypNocEhaNMpmzQh1j+wnBVOWsUhtMEhHkOrYuTsRYdXctyrMCte29bOWoLSNZ1dgs1f7ONFFn2V6qxzNfam9mICp9DZx5D2iupHNwldcayJyvB8pKi7+W/QwCOpdYQJIYWxRIHGPz4lU9krYs1YwnDM+iQZxhjy3LM/YROifYL3GFhS9hNLu/H/G4kbGs2ffJB+774361YxORbwP+urX2Q+7yTwBYa//b+zmOR7qw7D+1a7/3H36YL5ysU9YhJ9M+Ta76nWxYauFXBSTjgs0VzTksy4i61L6DOKkppinJsOTC2glffGUTiQ0b61P2X1hnePWE6VEfmUZII6SXZ4ShYbW/5PorG8rchRacdDA8jmhWa3rjnLKMsPcybKALcWKj7qpWCI4jzIr2hWGFbN0VetD13JnUqDxtXGNr0aIwVslgvLXEfkEX6e39pBZMvyGchZiNEruMNGtuHmumXc8w3JgzO+iT3oopLlTIMlT5nFFXR5yMrVlpCGZhJ+0KC9HCL7SaYzeuWd2ccvzKRHPdxhXspfplmrqcuNq57gb2TL5XaIRLUAn1qCE6CbFXljQnCfFR2B0PgUVK3VUOKiEonPxN0MIwsSrxdZ9BNrLEx2qgZKOzCBTafLxYi5PGFYVWoOkZ7LBRxguQIugKpeQopNxo9DFdBl641DlqVhrtiStdken62aRR5obQEixDoqkWBtmhSvR0AacOiBoBIm7hq4vC5FjHGc/VLMqkWhDbyFKNDOlhSJNoEVGu2M6zKJ5pcdCaO7VSxsDl5pVjSzNoSPe0MM/2gm4uq7EaRbUh7yo/coVXX4sbNZXQjQXNymvllrpgr8bKKIULt3B150vd18LSpCpPbKWA+Zo+VnKsLJhJ0YJupMemi2r9n6Znu9dejHu+EdRDQ3wSdItUabS/rM0KVMmcLvzFOOOsQiWTbc9hNFfTnKA4k3xJu4ittf+sHLmYlbaQcOxJK4Fseu2ctbI/2+UAtkZKxhXYbY9eKwksVyz9uzovyx29TzyH5FjjNKIlnbyvSVTCGJ/qmNNDLTQ6Aydzdgzt72qot5lEiyIxWghGyzN5pXEmtUHlzJ4qPcbRdcN8R81v2miaLpPP6hiLNWF4QyNJWtMdE50ZSklNFzOj2YoaZ6KxHk6SuLBd/9p586CgUmMklcXiYnc0aqOYSFecimMH4cxsKsyVMWuL9NaYpzVMCaqzxatYVxTObJdRWPf0crTUyI/AxW3Y4Ex2CrwqX6/NzGsLjbBw2X+NdXE3Z1LKdhw2pHMflXNywuRUzYWyQ2WUQBf28dx2r5cu/PVYgxp6hw3zLe1jDEvrZJXSvTfaz/F6oGMDx1a6DQigy8rEqow0O7Bd4RLPrBr8jILuPR4vtDgyIV0xdJZ5iCtotIgFZdUWmwHR8mwOw8qqjDXSYtLEahzTJIGLS3ESWHsmXw0LNdEBuvd/VLg8SNd3Wo6CzhCqhZgzA5/WkClaGpXJuhiZ1lk3nhvKUUi+Kgzu6YPUqXSFcyspDZxplxpKafF39oSQHNcUq5EaQQGtSY5Y28lkm8QVfXPtA9TvBks1CNznSBtxoqY9TS/oxlr3AqL5WY9jmzsZ5g3VKCKe1k66qusdjBZsNpKu/zCojB5PFnSFextZYgPtVyzWU2eSpMcVn7qIj8a6TSXrWD+NTAkLQ1AbgmWNSSOVnhp339oVq0lI0BhMGHRjt5F0USrRvKJcSYjmNQTOxGhZY+MAE2qcCkbff+3cmjQkXFQ0/djJZFsJpmZCNsOEcF45s6CzXkrTSlBdPmNrFqTfB0F3OZyXGjnSaJEojVXToMa9vm0mpohe74yDbKrS1daZ1YqcGfNEIZ18tW4dUwP92xhlCts+RVcA2yhUthHUObZuIAqRvNR+SWP18UX0vm00iXN37YyDWofWVuLaoi0267r7n1fLV+2rWcPX/m/TnBWBbWFqTFeYvqrf0jnK6uHJuf/7MnXJ+cL1EZfCisi/BDbegIfOgPNZLD9nrf25c8/7fcB3Wmv/orv8HwDvs9b+lfs5iEe6sFx5Zts++Tf/IttDNaUZRgWzOiWShv3lkDhsWM/mLOqEZ2/u8EevvUwvrGiscHuxwqKKWUlzlrV+Y3/bxkv8ysvvJA4b4tCw1ltwlPeIA8Ol4TF7+ZCijggDQyBWf7CsZ3M+cfMSq6MFq9mSw2WfLNI34sG8z8Zwzt50iLWQLxJWVhakcc3V8REvn67Sjyv2ZgNCsYyygjSquTQ45nOH20yXKWlck5cxIpbJYMmd/RX6g4JeUtEYYa2/5MbhhEtrx9w6HmNMwKifd+PcOx4ShpamEZo6ZLIyp2pC4rAhiRoaExCIJYtq9mcDhlnBzmDKp2/uEojSEWlaa0xHEzDoF5zeGTHamepz9XIOTwdcWj/mped22X1ij9t3VtndOeLuwQqDQU5Vh2RJRT+puPnyOv2NBauDJXcOVgjChtEg5+hwqEH3rnEsSDSkPh6WVCcpwaBCQku/XzDdG7J98Yi711fJ1nKypKJqQpbzBFMHbGxOmS1T8lkCVhitzZnuD8hWCsoiIowa6oMejCpsEdJfW7A47GvxLxaZRgTrJePRgqKKyBfKCEdJQ3mYMdyZsZinhFFDtUh0rEUIVUCwDFh76pD9WytaNE4KZbJPYhjXXNo95PpLm/Q35yyOegzWlvTTkqOTAfWJRqWE0xC5qKuw+jQhXimoDzPijSVJUjM77CPzUJnBlRJTB2ds9jJk/eoR+zcmBMMKM1cWndgy3JirqdE8IjyJMFuF9phaCEYVZqZ9dUEuNJMaKULijSXlNEGKEAykOwvK6wOVRva0T5W5BjPLpETupZj1ClsEyhr3Gqi1p9MkqPnPMuw2QyQPtM+y1I0a6TVEtxLqCyVy7FbTFmxikUozZyV39x3UhHfTjokOl+5LyzlEm54hyIWg1s2RYOEMa1ZqosMIG1maUYOUAeEi0NiRQhfH5Vatr4N7rHq9IjyKlA1LdIMgOgmpJ8pOt4Vtk2kB1xaVTWa7XmJiVR3IUs2Pwpcy6ms5cjclPg3Id2vto50qU1qNz9yJm74ei7rhSlcwR3P3HD2dn7DUMdgQfSzHyrah7kHl+ianuuFjBg3xQeQ2PuwZc7pe0ftiQr5p1D10HuiGQXmWyRqUuuEiNZSbDcmeW6RZXKau7Xr94qkbc37W69mktmMS46m4HkBY7Dr5a4MzIlJJa3oQdBLUlo1vGeRooc+XHkjX89j0nNuzY22bTP+n7msPYzmxDF4RljvOzRt0rjPNzy1WLU3fkroNmfOZkV3frtsACEq3keFcl8NSN0K6DQ2jstnWCbllGcuJsuUtqxrmUE6cI7KLG2mzJquRGh21jGw8OytQyzGdW2x6pMV/UDlWt9bolKAQRte1j1D7B13f5uKMHY1nluW2dD2o6bH2UVYjd34njk02ep2JnOtzpn2PYeHmw7m/6mM7xi4V5yzbFrs6Jy0rq+eJMypqGSxpN7LcAtS4Inl5du60xXc1UAawSemckKOl7fKg26K3yVrHWUvb5xrPLYstNdvJDjWSpUm08A0q9/zuf9qNhWrQMmzOldixp21fpLJyQu9A3YBb12MxkE/aDQMt4KueOvZWfR2DCZXR7B1oUVesKLNqIo1sUUmrGv8kMy2uq74+ZrvBFDkmNV8J6B2pURLQGQi1DGOUGy3Icy2a2w2QJhHihek2Y7KDCpMGlKNQDaBm+n9iNOu0e70icf2sRl15XZZoNQhelUVa9QOSqT5GPNPCOaiUqdP8UTXOCfOGYj0hWhqaNCCe1jRZ2L0Hk+MS48x2gM7pFatFP8KrHWvzuru/dQWhNJZ4VnWMZutWi0GL99qxlFlIkDeOtVZ5aFDUumwJhSCvqYeJutXWBslrlbIGosVpGqlZUNhu/ljtzYyCrpfSRgE0VlnKLD7L1zRoFmb7t3OdlaI8K9iM65EUZSttGzXiGMvuOvc3zbki0c2ZLnbCM5fZJD4r9s7nXrYS1rYIBV4bC2JLHZv2Yn4Z050vI4/9cjhfdD7qheXDgi8svwaIyBR47mGP4y2GDWD/YQ/iLQY/5w8efs4fPPycP3j4OX/w8HP+4OHn/MHjG3XOr1prNx/2IB4GHpQUNrqfD/YQ8NxbdefhYUFEfs/P+YOFn/MHDz/nDx5+zh88/Jw/ePg5f/Dwc/7g4ef8GxIfA54SkceAm8D3Az9wv5/kUS8sPTw8PDw8PDw8PDw8PL4CrLW1iPwV4FfRuJG/b639zP1+Hl9Yenh4eHh4eHh4eHh4vIlhrf0XwL94I5/jkc6xBH7uq9/F4z7Dz/mDh5/zBw8/5w8efs4fPPycP3j4OX/w8HP+4OHn/C2KR9q8x8PDw8PDw8PDw8PDw+Ph41FnLD08PDw8PDw8PDw8PDweMnxh6eHh4eHh4eHh4eHh4fG68MgWliLynSLynIi8ICI//rDH82aBiFwWkX8tIp8Vkc+IyI+669dE5NdE5PPu96q7XkTkb7vX4ZMi8t6HewSPJkQkFJHfF5FfcZcfE5GPunn9JyKSuOtTd/kFd/u1hznuRxUiMhGRXxSRZ0XkcyLybf4cf2MhIj/mPlM+LSK/ICKZP8/vL0Tk74vIPRH59Lnrvu7zWkR+yN3/8yLyQw/jWB4VfIU5/xvus+WTIvJ/icjk3G0/4eb8ORH50Lnr/Zrma8SXm/Nzt/01EbEisuEu+/P8PuArzbmI/Ig71z8jIv/9uev9ef4WxSNZWIpICPwd4LuAdwB/TkTe8XBH9aZBDfw1a+07gPcD/5mb2x8HfsNa+xTwG+4y6GvwlPv5S8DfffBDflPgR4HPnbv83wF/y1r7JHAE/LC7/oeBI3f933L38/j68bPAv7TWPgO8G517f46/QRCRi8B/DvxRa+03oVbn348/z+83PgJ852uu+7rOaxFZA34aeB/wrcBPt8Wox5fFR/jSOf814Juste8Cngd+AsB9l34/8E73P/+T21T0a5qvDx/hS+ccEbkMfBB45dzV/jy/P/gIr5lzEflTwPcA77bWvhP4H9z1/jx/C+ORLCzRD4EXrLUvWmtL4B+jJ7fH64S19ra19hPu7ym64L6Izu8/cHf7B8D3ur+/B/jfrOJ3gImI7D7gYT/SEJFLwJ8Fft5dFuA7gF90d3ntfLevwy8CH3D39/gaISIrwJ8E/h6Atba01h7jz/E3GhHQE5EI6AO38ef5fYW19reAw9dc/fWe1x8Cfs1ae2itPUKLpC9ZxHsovtycW2v/lbW2dhd/B7jk/v4e4B9bawtr7UvAC+h6xq9pvg58hfMcdBPqvwLOu1L68/w+4CvM+V8GfsZaW7j73HPX+/P8LYxHtbC8CFw/d/mGu87jPsLJz74F+Ciwba297W66A2y7v/1r8frxP6JfhsZdXgeOzy1Mzs9pN9/u9hN3f4+vHY8Be8D/Kio//nkRGeDP8TcM1tqb6G72K2hBeQJ8HH+ePwh8vee1P9/vL/4C8P+4v/2cv0EQke8Bblpr//A1N/k5f+PwNPAnXLvCb4rIH3PX+zl/C+NRLSw93mCIyBD4P4G/aq09PX+b1Ywan1NzHyAiHwbuWWs//rDH8hZCBLwX+LvW2m8B5pzJAwF/jt9vOInZ96BF/QVggGcHHjj8ef1gISI/ibaX/KOHPZY3M0SkD/zXwE897LG8xRABa2jb1H8J/FOvLPF4VAvLm8Dlc5cvues87gNEJEaLyn9krf0ld/XdVv7nfreSB/9avD78O8B3i8gXUVnId6D9fxMnGYRXz2k33+72FeDgQQ74TYAbwA1r7Ufd5V9EC01/jr9x+NPAS9baPWttBfwSeu778/yNx9d7Xvvz/T5ARP5D4MPAD9qzwHA/528MnkA3rf7QfZdeAj4hIjv4OX8jcQP4JScz/l1UdbWBn/O3NB7VwvJjwFOijoIJ2iT8yw95TG8KuN2mvwd8zlr7N8/d9MtA65r2Q8A/P3f9n3fOa+8HTs7Jrjy+Cqy1P2GtvWStvYaex/+vtfYHgX8NfJ+722vnu30dvs/d3zMQXwestXeA6yLyNnfVB4DP4s/xNxKvAO8Xkb77jGnn3J/nbzy+3vP6V4EPisiqY5o/6K7z+BohIt+Jtjd8t7V2ce6mXwa+X9T1+DHUUOZ38Wua1wVr7aestVvW2mvuu/QG8F73We/P8zcO/wz4UwAi8jSQAPv48/ytDWvtI/kD/BnUbe0LwE8+7PG8WX6AfxeVSn0S+AP382fQ/qbfAD4P/Dqw5u4vqMvXF4BPoa6PD/04HsUf4NuBX3F/P45+EL8A/B9A6q7P3OUX3O2PP+xxP4o/wHuA33Pn+T8DVv05/obP+X8DPAt8GviHQOrP8/s+x7+A9rBW6OL6h/9tzmu0L/AF9/MfPezj+kb++Qpz/gLaS9Z+h/7P5+7/k27OnwO+69z1fk3zOub8Nbd/Edhwf/vz/A2ac7SQ/N/dZ/ongO84d39/nr9Ff8S90B4eHh4eHh4eHh4eHh4e/1Z4VKWwHh4eHh4eHh4eHh4eHt8g8IWlh4eHh4eHh4eHh4eHx+uCLyw9PDw8PDw8PDw8PDw8Xhd8Yenh4eHh4eHh4eHh4eHxuuALSw8PDw8PDw8PDw8PD4/XBV9Yenh4eHh8w0JE/j/3+5qI/MDDHo+Hh4eHh4fHl4cvLD08PDw8vmFhrf3j7s9rgC8sPTw8PDw8vkHhC0sPDw8Pj29YiMjM/fkzwJ8QkT8QkR8TkVBE/oaIfExEPiki/4m7/7eLyG+KyD8XkRdF5GdE5AdF5HdF5FMi8oS7378vIp8WkT8Ukd96WMfn4eHh4eHxZkH0sAfg4eHh4eHxNeDHgf/CWvthABH5S8CJtfaPiUgK/LaI/Ct333cDbwcOgReBn7fWfquI/CjwI8BfBX4K+JC19qaITB70wXh4eHh4eLzZ4BlLDw8PD49HER8E/ryI/AHwUWAdeMrd9jFr7W1rbQF8AWgLzk+hklqA3wY+IiL/MRA+sFF7eHh4eHi8SeEZSw8PDw+PRxEC/Ii19ldfdaXItwPFuavMucsG971nrf1PReR9wJ8FPi4if8Rae/CGj9rDw8PDw+NNCs9Yenh4eHg8CpgCo3OXfxX4yyISA4jI0yIy+FofTESesNZ+1Fr7U8AecPm+jtbDw8PDw+MtBs9Yenh4eHg8Cvgk0IjIHwIfAX4WlbV+QkQELQ6/9+t4vL8hIk+hzOdvAH94X0fr4eHh4eHxFoNYax/2GDw8PDw8PDw8PDw8PDweYXgprIeHh4eHh4eHh4eHh8frgi8sPTw8PDw8PDw8PDw8PF4XfGHp4eHh4eHh4eHh4eHh8brgC0sPDw8PDw8PDw8PDw+P1wVfWHp4eHh4eHh4eHh4eHi8LvjC0sPDw8PDw8PDw8PDw+N1wReWHh4eHh4eHh4eHh4eHq8L/z8JPtB5VvjafAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "evaluate_embeddings(ratings)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can observe that some movies tend to be widely recommended or not recommended, whilst some other have more variance in their predicted score" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Neural Network (non-linear) Matrix Factorization\n", + "\n", + "We don't have to limit ourselves to the weights of the linear embedding layer for our user or item embeddings. We can have a more complex pipeline combining fully connected layers and non-linear activations." + ] + }, + { + "cell_type": "code", + "execution_count": 15, "metadata": { "collapsed": false, "scrolled": false }, - "outputs": [], + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "plot\n", + "\n", + "\n", + "user\n", + "\n", + "user\n", + "\n", + "\n", + "MLP_MF_emb_user_fwd\n", + "\n", + "MLP_MF_emb_user_fwd\n", + "\n", + "\n", + "MLP_MF_emb_user_fwd->user\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_relu0\n", + "\n", + "MLP_MF_relu0\n", + "\n", + "\n", + "MLP_MF_relu0->MLP_MF_emb_user_fwd\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_dense_user_fwd\n", + "\n", + "FullyConnected\n", + "64\n", + "\n", + "\n", + "MLP_MF_dense_user_fwd->MLP_MF_relu0\n", + "\n", + "\n", + "\n", + "\n", + "item\n", + "\n", + "item\n", + "\n", + "\n", + "MLP_MF_emb_item_fwd\n", + "\n", + "MLP_MF_emb_item_fwd\n", + "\n", + "\n", + "MLP_MF_emb_item_fwd->item\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_relu1\n", + "\n", + "MLP_MF_relu1\n", + "\n", + "\n", + "MLP_MF_relu1->MLP_MF_emb_item_fwd\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_dense_item_fwd\n", + "\n", + "FullyConnected\n", + "64\n", + "\n", + "\n", + "MLP_MF_dense_item_fwd->MLP_MF_relu1\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF__mul0\n", + "\n", + "MLP_MF__mul0\n", + "\n", + "\n", + "MLP_MF__mul0->MLP_MF_dense_user_fwd\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF__mul0->MLP_MF_dense_item_fwd\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_sum0\n", + "\n", + "MLP_MF_sum0\n", + "\n", + "\n", + "MLP_MF_sum0->MLP_MF__mul0\n", + "\n", + "\n", + "\n", + "\n", + "MLP_MF_flatten0\n", + "\n", + "MLP_MF_flatten0\n", + "\n", + "\n", + "MLP_MF_flatten0->MLP_MF_sum0\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "def get_one_layer_mlp(hidden, k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " score = mx.symbol.Variable('score')\n", - " # user latent features\n", - " user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)\n", - " user = mx.symbol.Activation(data = user, act_type='relu')\n", - " user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)\n", - " # item latent features\n", - " item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " item = mx.symbol.Activation(data = item, act_type='relu')\n", - " item = mx.symbol.FullyConnected(data = item, num_hidden = hidden)\n", - " # predict by the inner product\n", - " pred = user * item\n", - " pred = mx.symbol.sum(data = pred, axis = 1)\n", - " pred = mx.symbol.Flatten(data = pred)\n", - " # loss layer\n", - " pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)\n", - " return pred\n", + "class MLPMatrixFactorization(gluon.HybridBlock):\n", + " \n", + " def __init__(self, k, hidden, max_user=max_user, max_item=max_item):\n", + " super(MLPMatrixFactorization, self).__init__(prefix='MLP_MF_')\n", + " \n", + " # user feature lookup\n", + " with self.name_scope():\n", + " self.user_embedding = gluon.nn.Embedding(input_dim=max_user, output_dim = k, prefix='emb_user_') \n", + " self.user_mlp = gluon.nn.Dense(hidden, prefix='dense_user_')\n", + "\n", + " # item feature lookup\n", + " self.item_embedding = gluon.nn.Embedding(input_dim=max_item, output_dim = k, prefix='emb_item_') \n", + " self.item_mlp = gluon.nn.Dense(hidden, prefix='dense_item_')\n", + " \n", + " def hybrid_forward(self, F, user, item):\n", + " user_embeddings = self.user_embedding(user)\n", + " user_embeddings_relu = user_embeddings.relu()\n", + " user_transformed = self.user_mlp(user_embeddings_relu)\n", + " \n", + " items_embeddings = self.item_embedding(item)\n", + " items_embeddings_relu = items_embeddings.relu()\n", + " items_transformed = self.item_mlp(items_embeddings_relu)\n", + " \n", + " # predict by the inner product, which is elementwise product and then sum\n", + " pred = (user_transformed * items_transformed).sum(axis=1)\n", + " \n", + " return pred.flatten()\n", "\n", - "net2 = get_one_layer_mlp(64, 64)\n", - "mx.viz.plot_network(net2)" + "net2 = MLPMatrixFactorization(64, 64)\n", + "net2.initialize(mx.init.Xavier(), ctx=ctx)\n", + "mx.viz.plot_network(net2(mx.sym.var('user'), mx.sym.var('item')), node_attrs={\"fixedsize\":\"false\"})" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--------------------------------------------------------------------------------\n", + " Layer (type) Output Shape Param #\n", + "================================================================================\n", + " Input (128,), (128,) 0\n", + " Embedding-1 (128, 64) 60416\n", + " Dense-2 (128, 64) 4160\n", + " Embedding-3 (128, 64) 107712\n", + " Dense-4 (128, 64) 4160\n", + "MLPMatrixFactorization-5 (128, 1) 0\n", + "================================================================================\n", + "Parameters in forward computation graph, duplicate included\n", + " Total params: 176448\n", + " Trainable params: 176448\n", + " Non-trainable params: 0\n", + "Shared params in forward computation graph: 0\n", + "Unique parameters in model: 176448\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "net2.summary(user.as_in_context(ctx[0]), item.as_in_context(ctx[0]))" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 17, "metadata": { "collapsed": false, "scrolled": false }, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Training RMSE 1.3127, Test RMSE 0.6534\n", + "Epoch [1], Training RMSE 0.6074, Test RMSE 0.6405\n", + "Epoch [2], Training RMSE 0.5929, Test RMSE 0.6255\n", + "Epoch [3], Training RMSE 0.5789, Test RMSE 0.6122\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[3126]: Change learning rate to 2.00000e-03\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [4], Training RMSE 0.5650, Test RMSE 0.6006\n", + "Epoch [5], Training RMSE 0.5560, Test RMSE 0.5965\n", + "Epoch [6], Training RMSE 0.5532, Test RMSE 0.5929\n", + "Epoch [7], Training RMSE 0.5504, Test RMSE 0.5914\n", + "Epoch [8], Training RMSE 0.5476, Test RMSE 0.5893\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[6251]: Change learning rate to 4.00000e-04\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [9], Training RMSE 0.5448, Test RMSE 0.5856\n", + "Epoch [10], Training RMSE 0.5431, Test RMSE 0.5855\n", + "Epoch [11], Training RMSE 0.5425, Test RMSE 0.5856\n", + "Epoch [12], Training RMSE 0.5420, Test RMSE 0.5846\n", + "Epoch [13], Training RMSE 0.5414, Test RMSE 0.5833\n", + "Epoch [14], Training RMSE 0.5409, Test RMSE 0.5822\n" + ] + } + ], "source": [ - "results2 = train(net2, train_test_data, num_epoch=15, learning_rate=0.02, ctx=ctx)" + "losses_2 = train(net2, train_data, test_data, epochs=15, ctx=ctx)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Visualizing results\n", - "Now let's draw a single chart that compares the learning curves of the three different models.\n", - "We'll use the bokeh library since it gives nice interactive charting." + "We can try training with the Adam optimizer instead" ] }, { "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, + "execution_count": 18, + "metadata": {}, "outputs": [], "source": [ - "import bokeh\n", - "import bokeh.io\n", - "import bokeh.plotting\n", - "bokeh.io.output_notebook()\n", - "import pandas as pd\n", - "\n", - "def viz_lines(fig, results, legend, color):\n", - " df = pd.DataFrame(results._data['eval'])\n", - " fig.line(df.elapsed,df.RMSE, color=color, legend=legend, line_width=2)\n", - " df = pd.DataFrame(results._data['train'])\n", - " fig.line(df.elapsed,df.RMSE, color=color, line_dash='dotted', alpha=0.1)\n", - "\n", - "fig = bokeh.plotting.Figure(x_axis_type='datetime', x_axis_label='Training time', y_axis_label='RMSE')\n", - "viz_lines(fig, results1, \"Linear MF\", \"orange\")\n", - "viz_lines(fig, results2, \"MLP\", \"blue\")\n", + "net2 = MLPMatrixFactorization(64, 64)\n", + "net2.initialize(mx.init.Xavier(), ctx=ctx)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Training RMSE 0.6292, Test RMSE 0.4896\n", + "Epoch [1], Training RMSE 0.4623, Test RMSE 0.4818\n", + "Epoch [2], Training RMSE 0.4539, Test RMSE 0.4811\n", + "Epoch [3], Training RMSE 0.4486, Test RMSE 0.5017\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[3126]: Change learning rate to 2.00000e-03\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [4], Training RMSE 0.4462, Test RMSE 0.4950\n", + "Epoch [5], Training RMSE 0.4144, Test RMSE 0.4506\n", + "Epoch [6], Training RMSE 0.4054, Test RMSE 0.4489\n", + "Epoch [7], Training RMSE 0.4026, Test RMSE 0.4497\n", + "Epoch [8], Training RMSE 0.4013, Test RMSE 0.4503\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[6251]: Change learning rate to 4.00000e-04\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [9], Training RMSE 0.3997, Test RMSE 0.4504\n", + "Epoch [10], Training RMSE 0.3912, Test RMSE 0.4476\n", + "Epoch [11], Training RMSE 0.3898, Test RMSE 0.4466\n", + "Epoch [12], Training RMSE 0.3892, Test RMSE 0.4470\n", + "Epoch [13], Training RMSE 0.3889, Test RMSE 0.4472\n", + "Epoch [14], Training RMSE 0.3885, Test RMSE 0.4458\n" + ] + } + ], + "source": [ + "losses_2_adam = train(net2, train_data, test_data, epochs=15, optimizer='adam', learning_rate=0.01, ctx=ctx)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deep Neural Network (Residual Network / ResNet)\n", + "Borrowing ideas from [Deep Residual Learning for Image Recognition (He, et al.)](https://arxiv.org/abs/1512.03385) to build a complex deep network that is aggressively regularized, thanks to the dropout layers, to avoid over-fitting, but still achieves good performance. " + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "plot\n", + "\n", + "\n", + "user\n", + "\n", + "user\n", + "\n", + "\n", + "ResNet_MF_emb_user_fwd\n", + "\n", + "ResNet_MF_emb_user_fwd\n", + "\n", + "\n", + "ResNet_MF_emb_user_fwd->user\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block1_d1_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_u_block1_d1_fwd->ResNet_MF_emb_user_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block1_d1_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "ResNet_MF_u_block1_d1_relu_fwd->ResNet_MF_u_block1_d1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block1_dropout_fwd\n", + "\n", + "ResNet_MF_u_block1_dropout_fwd\n", + "\n", + "\n", + "ResNet_MF_u_block1_dropout_fwd->ResNet_MF_u_block1_d1_relu_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block1_d2_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_u_block1_d2_fwd->ResNet_MF_u_block1_dropout_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus0\n", + "\n", + "ResNet_MF__plus0\n", + "\n", + "\n", + "ResNet_MF__plus0->ResNet_MF_emb_user_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus0->ResNet_MF_u_block1_d2_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_relu0\n", + "\n", + "ResNet_MF_relu0\n", + "\n", + "\n", + "ResNet_MF_relu0->ResNet_MF__plus0\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_dropout0_fwd\n", + "\n", + "ResNet_MF_dropout0_fwd\n", + "\n", + "\n", + "ResNet_MF_dropout0_fwd->ResNet_MF_relu0\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block2_d1_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_u_block2_d1_fwd->ResNet_MF_dropout0_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block2_d1_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "ResNet_MF_u_block2_d1_relu_fwd->ResNet_MF_u_block2_d1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block2_dropout_fwd\n", + "\n", + "ResNet_MF_u_block2_dropout_fwd\n", + "\n", + "\n", + "ResNet_MF_u_block2_dropout_fwd->ResNet_MF_u_block2_d1_relu_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_u_block2_d2_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_u_block2_d2_fwd->ResNet_MF_u_block2_dropout_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus1\n", + "\n", + "ResNet_MF__plus1\n", + "\n", + "\n", + "ResNet_MF__plus1->ResNet_MF_dropout0_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus1->ResNet_MF_u_block2_d2_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_relu1\n", + "\n", + "ResNet_MF_relu1\n", + "\n", + "\n", + "ResNet_MF_relu1->ResNet_MF__plus1\n", + "\n", + "\n", + "\n", + "\n", + "item\n", + "\n", + "item\n", + "\n", + "\n", + "ResNet_MF_emb_item_fwd\n", + "\n", + "ResNet_MF_emb_item_fwd\n", + "\n", + "\n", + "ResNet_MF_emb_item_fwd->item\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block1_d1_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_i_block1_d1_fwd->ResNet_MF_emb_item_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block1_d1_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "ResNet_MF_i_block1_d1_relu_fwd->ResNet_MF_i_block1_d1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block1_dropout_fwd\n", + "\n", + "ResNet_MF_i_block1_dropout_fwd\n", + "\n", + "\n", + "ResNet_MF_i_block1_dropout_fwd->ResNet_MF_i_block1_d1_relu_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block1_d2_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_i_block1_d2_fwd->ResNet_MF_i_block1_dropout_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus2\n", + "\n", + "ResNet_MF__plus2\n", + "\n", + "\n", + "ResNet_MF__plus2->ResNet_MF_emb_item_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus2->ResNet_MF_i_block1_d2_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_relu2\n", + "\n", + "ResNet_MF_relu2\n", + "\n", + "\n", + "ResNet_MF_relu2->ResNet_MF__plus2\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_dropout1_fwd\n", + "\n", + "ResNet_MF_dropout1_fwd\n", + "\n", + "\n", + "ResNet_MF_dropout1_fwd->ResNet_MF_relu2\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block2_d1_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_i_block2_d1_fwd->ResNet_MF_dropout1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block2_d1_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "ResNet_MF_i_block2_d1_relu_fwd->ResNet_MF_i_block2_d1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block2_dropout_fwd\n", + "\n", + "ResNet_MF_i_block2_dropout_fwd\n", + "\n", + "\n", + "ResNet_MF_i_block2_dropout_fwd->ResNet_MF_i_block2_d1_relu_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_i_block2_d2_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "ResNet_MF_i_block2_d2_fwd->ResNet_MF_i_block2_dropout_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus3\n", + "\n", + "ResNet_MF__plus3\n", + "\n", + "\n", + "ResNet_MF__plus3->ResNet_MF_dropout1_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__plus3->ResNet_MF_i_block2_d2_fwd\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_relu3\n", + "\n", + "ResNet_MF_relu3\n", + "\n", + "\n", + "ResNet_MF_relu3->ResNet_MF__plus3\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__mul0\n", + "\n", + "ResNet_MF__mul0\n", + "\n", + "\n", + "ResNet_MF__mul0->ResNet_MF_relu1\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF__mul0->ResNet_MF_relu3\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_sum0\n", + "\n", + "ResNet_MF_sum0\n", + "\n", + "\n", + "ResNet_MF_sum0->ResNet_MF__mul0\n", + "\n", + "\n", + "\n", + "\n", + "ResNet_MF_flatten0\n", + "\n", + "ResNet_MF_flatten0\n", + "\n", + "\n", + "ResNet_MF_flatten0->ResNet_MF_sum0\n", + "\n", + "\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def get_residual_block(prefix='res_block_', hidden=64):\n", + " block = gluon.nn.HybridSequential(prefix=prefix)\n", + " with block.name_scope():\n", + " block.add(\n", + " gluon.nn.Dense(hidden, activation='relu', prefix='d1_'),\n", + " gluon.nn.Dropout(0.5, prefix='dropout_'),\n", + " gluon.nn.Dense(hidden, prefix='d2_')\n", + " )\n", + " return block\n", + " \n", + "class ResNetMatrixFactorization(gluon.HybridBlock):\n", + " \n", + " def __init__(self, k, hidden, max_user=max_user, max_item=max_item):\n", + " super(ResNetMatrixFactorization, self).__init__(prefix='ResNet_MF_')\n", + " \n", + " # user feature lookup\n", + " with self.name_scope():\n", + " self.user_embedding = gluon.nn.Embedding(input_dim=max_user, output_dim = k, prefix='emb_user_')\n", + " self.user_block1 = get_residual_block('u_block1_', hidden)\n", + " self.user_dropout = gluon.nn.Dropout(0.5)\n", + " self.user_block2 = get_residual_block('u_block2_', hidden) \n", + " \n", + " # item feature lookup\n", + " self.item_embedding = gluon.nn.Embedding(input_dim=max_item, output_dim = k, prefix='emb_item_')\n", + " self.item_block1 = get_residual_block('i_block1_', hidden)\n", + " self.item_dropout = gluon.nn.Dropout(0.5)\n", + " self.item_block2 = get_residual_block('i_block2_', hidden) \n", + " \n", + " \n", + " def hybrid_forward(self, F, user, item):\n", + " user_embeddings = self.user_embedding(user)\n", + " user_block1 = self.user_block1(user_embeddings)\n", + " user1 = (user_embeddings + user_block1).relu()\n", + " \n", + " user2 = self.user_dropout(user1)\n", + " user_block2 = self.user_block2(user2)\n", + " user_transformed = (user2 + user_block2).relu()\n", + " \n", + " item_embeddings = self.item_embedding(item)\n", + " item_block1 = self.item_block1(item_embeddings)\n", + " item1 = (item_embeddings + item_block1).relu()\n", + " \n", + " item2 = self.item_dropout(item1)\n", + " item_block2 = self.item_block2(item2)\n", + " item_transformed = (item2 + item_block2).relu()\n", + " \n", + " # predict by the inner product, which is elementwise product and then sum\n", + " pred = (user_transformed * item_transformed).sum(axis=1)\n", + " \n", + " return pred.flatten()\n", "\n", - "bokeh.io.show(fig)" + "net3 = ResNetMatrixFactorization(128, 128)\n", + "net3.initialize(mx.init.Xavier(), ctx=ctx)\n", + "mx.viz.plot_network(net3(mx.sym.var('user'), mx.sym.var('item')), node_attrs={\"fixedsize\":\"false\"})" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--------------------------------------------------------------------------------\n", + " Layer (type) Output Shape Param #\n", + "================================================================================\n", + " Input (128,), (128,) 0\n", + " Embedding-1 (128, 128) 120832\n", + " Activation-2 0\n", + " Activation-3 (128, 128) 0\n", + " Dense-4 (128, 128) 16512\n", + " Dropout-5 (128, 128) 0\n", + " Dense-6 (128, 128) 16512\n", + " Dropout-7 (128, 128) 0\n", + " Activation-8 0\n", + " Activation-9 (128, 128) 0\n", + " Dense-10 (128, 128) 16512\n", + " Dropout-11 (128, 128) 0\n", + " Dense-12 (128, 128) 16512\n", + " Embedding-13 (128, 128) 215424\n", + " Activation-14 0\n", + " Activation-15 (128, 128) 0\n", + " Dense-16 (128, 128) 16512\n", + " Dropout-17 (128, 128) 0\n", + " Dense-18 (128, 128) 16512\n", + " Dropout-19 (128, 128) 0\n", + " Activation-20 0\n", + " Activation-21 (128, 128) 0\n", + " Dense-22 (128, 128) 16512\n", + " Dropout-23 (128, 128) 0\n", + " Dense-24 (128, 128) 16512\n", + "ResNetMatrixFactorization-25 (128, 1) 0\n", + "================================================================================\n", + "Parameters in forward computation graph, duplicate included\n", + " Total params: 468352\n", + " Trainable params: 468352\n", + " Non-trainable params: 0\n", + "Shared params in forward computation graph: 0\n", + "Unique parameters in model: 468352\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "net3.summary(user.as_in_context(ctx[0]), item.as_in_context(ctx[0]))" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [0], Training RMSE 0.7046, Test RMSE 0.6775\n", + "Epoch [1], Training RMSE 0.4861, Test RMSE 0.5299\n", + "Epoch [2], Training RMSE 0.4662, Test RMSE 0.4835\n", + "Epoch [3], Training RMSE 0.4567, Test RMSE 0.4819\n", + "Epoch [4], Training RMSE 0.4505, Test RMSE 0.4653\n", + "Epoch [5], Training RMSE 0.4491, Test RMSE 0.4582\n", + "Epoch [6], Training RMSE 0.4427, Test RMSE 0.4555\n", + "Epoch [7], Training RMSE 0.4405, Test RMSE 0.4524\n", + "Epoch [8], Training RMSE 0.4360, Test RMSE 0.4507\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:root:Update[6251]: Change learning rate to 2.00000e-04\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch [9], Training RMSE 0.4328, Test RMSE 0.4504\n", + "Epoch [10], Training RMSE 0.4172, Test RMSE 0.4442\n", + "Epoch [11], Training RMSE 0.4141, Test RMSE 0.4426\n", + "Epoch [12], Training RMSE 0.4114, Test RMSE 0.4416\n", + "Epoch [13], Training RMSE 0.4106, Test RMSE 0.4413\n", + "Epoch [14], Training RMSE 0.4095, Test RMSE 0.4390\n" + ] + } + ], + "source": [ + "losses_3 = train(net3, train_data, test_data, epochs=15, optimizer='adam', learning_rate=0.001, ctx=ctx, num_epoch_lr=10)" ] }, { "cell_type": "markdown", - "metadata": { - "collapsed": true - }, + "metadata": {}, "source": [ - "## Acknowledgement\n", + "### Visualizing embeddings" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Contrary to the linear model where we can use directly the embedding weights, here we compute each combination of user / items and store predicted rating." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 26.6 s, sys: 5.26 s, total: 31.9 s\n", + "Wall time: 26 s\n" + ] + } + ], + "source": [ + "%%time\n", "\n", - "This tutorial is based on examples from [xlvector/github](https://github.com/xlvector/)." + "users = []\n", + "items = []\n", + "for i in range(max_user):\n", + " for j in range(max_item):\n", + " users.append(i+1)\n", + " items.append(j+1)\n", + "dataset = gluon.data.ArrayDataset(np.array(users).astype('float32'), np.array(items).astype('float32'))\n", + "dataloader = gluon.data.DataLoader(dataset, batch_size=batch_size, shuffle=False)\n", + "ratings = np.zeros((max_user+1, max_item+1))\n", + "for users, items in dataloader:\n", + " users = users.as_in_context(ctx[0])\n", + " items = items.as_in_context(ctx[0])\n", + " scores = net3(users, items).asnumpy()\n", + " ratings[users.asnumpy().astype('int32'), items.asnumpy().astype('int32')] = scores.reshape(-1)" ] }, { "cell_type": "code", - "execution_count": 10, - "metadata": { - "collapsed": false - }, - "outputs": [], + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Top 5 movies:\n", + "Schindler's List (1993), average rating 4.43\n", + "Casablanca (1942), average rating 4.39\n", + "Wrong Trousers, The (1993), average rating 4.39\n", + "Shawshank Redemption, The (1994), average rating 4.36\n", + "Close Shave, A (1995), average rating 4.34\n", + "\n", + "Worst 5 movies:\n", + "Scream of Stone (Schrei aus Stein) (1991), average rating 0.00\n", + "Mortal Kombat: Annihilation (1997), average rating 2.25\n", + "Home Alone 3 (1997), average rating 2.25\n", + "Crow: City of Angels, The (1996), average rating 2.25\n", + "Children of the Corn: The Gathering (1996), average rating 2.25\n", + "\n", + "5 most controversial movies:\n", + "First Knight (1995), average rating 2.95\n", + "Half Baked (1998), average rating 2.85\n", + "Just Cause (1995), average rating 2.96\n", + "Murder at 1600 (1997), average rating 3.07\n", + "Twister (1996), average rating 3.08\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA5YAAAH0CAYAAABGqLIGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXm0JclZH/j7IiIz773vvdq6utVSa0MIBAYZAwPGHpiRZ8ASMos3DFh4jG0GNDMeDwaMDcZjwWHRMD4YMAZsz9gsMvsYzrAdgQ8IMIsRjM0iEEaIbnWr96qut9x7MzOWb/74IiLj5rv31avugq4S+Tun6uXNjIz4YsnM+HZiZkyYMGHChAkTJkyYMGHChAnPFur5JmDChAkTJkyYMGHChAkTJtzdmBjLCRMmTJgwYcKECRMmTJjwnDAxlhMmTJgwYcKECRMmTJgw4TlhYiwnTJgwYcKECRMmTJgwYcJzwsRYTpgwYcKECRMmTJgwYcKE54SJsZwwYcKECRMmTJgwYcKECc8JE2M5YcKEPzQQ0Y8T0V9/vunYBiJ6GxF9zh9g/R9HRL/zB1X/swER/QUiepiITojow7dcPyGiVzwftN0NIKI3EdFbdlx7DRE98odN092GO/G5mDBhwoQJzw4TYzlhwh0EImIieuXo3M7N652MbXQz8ycy87f/AbT18jh25nbX/Wwxnktm/jlmftXzSdMW/BMAf5uZ95n5P40vxvPvBgAi+jYi+so/dAr/AEFEDxLRxz/fdPxRwl3yXEyYMGHChGeBibGcMOGPIO4kBuxuxPvQ+L0MwDuebyL+sPE+NH93FP6ojev7Sn/fV/oxYcKE5x8TYzlhwl0EIrpKRD9CRDeI6DoR/RwRqXjtRUT0/xDRU0T0+0T0d4r73kREP0BEbyGiIwCfTUQfTUS/QkRHRPQEEX3dGe1+EhH959juLxDRHy+u/X0iei8RHRPR7xDRf09ErwPwpQA+PZpT/losm81NieiziejnieifxnrfTUR/Op5/mIieLM1miejPEdF/ivQ+TERvKkj82fj3RmzvT8V7/iYR/TYRPUNEbyWilxX1fQIRvZOIDonomwDQGf3fNX6/GGl/jIi+iYjqWD7R82uRnk8fm0ZGbdkXEdGvRxq+l4hmxfUvjvU+SkSfU2p6iOj1RPRbcczfS0RftINuRURfRkQPxfH8DiK6SEQNEZ0A0JHG39txPxPRK4nocwG8AcAXx/78cLx+szX3/XHMjonoN4joA4noSyItDxPRny3Kf3ZcA8exrjfsoGnnuiWiTyGid8Q5eRsRffBovP8+Ef06gCURfTeAlwL44dinL47lPiau8RtE9GtE9Jqijvcjop+JNP4kgKvbaBzR+6VE9HRs/w3x3EdF2nVR7i9SfE621PFtRPTNJKbkJyTPzf1E9PVxbb+TClPmm8zLznUbrzMRvZGIfjeW+edEtPXZoLv3uXh/IvopIroW5+bfEtGl4vpLiOjfxfG7RvJ+GL+zrgF4E+14xmL5WRyba3E83k5ELyjqOs9637AU2DJep96/8bwion9ARL8X2/8+IroSryULj79FRO8B8FNn0TphwoQJ5wYzT/+mf9O/O+QfAAbwytG5NwF4Szz+GgDfCqCK/z4OwhApAL8K4H8HUAN4BYB3A3htUYcF8Odj2TmAXwTw1+L1fQAfs4OmDwfwJIA/CWFE/jqABwE0AF4F4GEAL4plXw7g/cd0F3W9DcDnxOPPBuAA/I1Y71cCeA+Afx7r/rMAjgHsx/KvAfDqSP8fB/AEgD9ftMsATNHWpwJ4F4APBmAAfBmAX4jXrsa6/3Icx78bafmcHWOwbfw+EsDHxLpfDuC3AXz+rrmM9D9S/H4QwC8DeBGAK/H+N8ZrrwPwOIAPAbAA8JayPgCPAfi4eHwZwEfsoPtvxjF4RZzjfwfgO89ab7vWI4BvA/CVxbXzrLkWwGvjGH0HgN8H8A/jmP+PAH4/lt0DcATgVfH3CwF8yA6atq5bAB8IYAngE2L9Xxz7Xhfj/Z8BvATAvDj38UXdDwC4BuD1sX+fEH/fW7T9dZD1+d9A1tBbdtD5GsiaSuX/20hf6uNvAfjEovwPAvjCHXV9G4CnIWtuBuCn4lj+DxienZ8+57ycZ93+CIBLEMb7KQCvex97Ll4Z57YBcC9EMPX18ZoG8GsA/ilkXc4AfOzonfW/xv7NccYzBuDzAPxwpFXHsbmAW1vv34bN5y6PF85+//5vAH4JwItjP/8FgO8evS+/I9Iy30Xrzb5Z07/p3/Rv+lf+mzSWEybcXbCQTcjLmNmy+CcxgI+CbH6/gpl7Fr+4fwXgM4p7f5GZf4iZAzOvY12vJKKrzHzCzL+0o83PBfAvmPk/MrNn8ZHsIJtHD9m0/DEiqpj5QWbeqv3agd9n5n/DzB7A90I2/V/BzB0z/wSAHrIJBDO/jZl/I9L/6wC+G7JZ34U3AvgaZv5tZnYAvhrAnyDRWr4ewDuY+QeY2QL4esiG9SxsjB8z/yoz/xIzO2Z+ELJxO4uebfhGZn6Uma9DNnV/Ip7/KwD+DTO/g5lXkA18CQsZ8wvM/Awz/3876n8DgK9j5ncz8wmALwHwGXR7TN/Os+Z+jpnfGsf/+yGb+DfHMf8eAC8vNEUBwIcS0ZyZH2PmXSa6u9btpwP4UWb+yVj/P4FsmP90ce83MvPDcf1vw2cB+DFm/rE4zz8J4FcAvJ6IXhr7/I/i+vxZyJzdDKn8zwD4UcjcAsC3x/YQNUmvBfBdZ9Tzg3HNtRAmtGXm7yienaSxPHNezrlu38zMN5j5PQB+GsO63Ia77rlg5nfFddIx81MQ5j/R+NEQpvbvMfOSmVtm/g/F7Y8y8z+L/Vvj7GfMArgHwvj6ODZHsZ7zrvezcNb7940A/iEzP8LMXRyrvzx69t8U+5i+B7tonTBhwoRzYWIsJ0y4s+Ah2pYSFeSjDwD/J0Q6/hPRjOofxPMvA/CiaMJ0g4huQExRS1Omh0f1/i2Ilued0ezpk3bQ9DIAXziq+yUQKfm7AHw+ZNPyJBF9DxG96Bb6+0RxvAYAZh6f2wcAIvqTRPTT0TztELJxOssU8WUAvqGg+TpEu/sAZOOYxyMy5+PxGWPjOolZ548Q0ePRDPCrb0LPNpTM7Aqxr2P6ttD2lyDM8UMkppl/akf9LwLwUPH7IYim5XaYuJ1nzY3n8unICKXfgGiklxDG8I0AHiOiHyWiD9rR7q51u9FXZg6QcXuguPdmc/wyAJ826tPHQoQ5LwLwTKQ14aFtlRTYVj49H28B8MlEtAdhmH6OmR87o67xWG59TnCTeTnnut21LrfhrnsuiOgF8V313kjjWwoaXwLgoSgM2YZxm2c9Y98J4K0AvofEdPdrIwN4K+t9J27y/n0ZgB8s1sBvQ74vu74JW2m9VZomTJjwRxsTYzlhwp2F90DMlEq8H+LGhZmPmfkLmfkVAD4FwBdEn5qHIdq/S8W/A2Z+fVEPl5Uy8+8y82cCuA/A/wHgB+Imd4yHAXzVqO4FM393rOe7mPljIRsZjnWdau824LsA/L8AXsLMFyEmwcn3a1tbDwP4vBHdc2b+BYjJ3EtSQSKi8vcOjNv4FgDvBPABzHwBsnnf6ad5i3gMYsKWsEEbM7+dmT8VMnc/BOD7dtTzKGReEl4KMeV7YnvxMzHu/3nW3PkrF83mJ0CYuHdCtGzbyu1atxt9Leb0vWf0YVufvnPUpz1mfjNkTi6PnpGX3qRb28o/GvvxXohp7V8E8NcgG/vbgZvNy+1et3fjc/HVELpfHWn8rILGhwG89Ayt/ri/O58xFquSL2fmPwbRnH8SxHz53OsdYj69KH7fv0HM7vfvwxBT63IdzOK6O9WXs2idMGHChPNiYiwnTLiz8L0AvoyIXhyDL3w8gE8G8ANADqLzyrhpPoRIoAPEJ+k4BnKYE5Emog8loo/a1RARfRYR3Rs1Ozfi6bCl6L8C8MaoMSQi2iMJpHNARK8iov+OiBqIP926qOMJiKnj7XrPHAC4zswtEX00gL9aXHsqtlvmXPxWAF9CRB8S+3uRiD4tXvtRAB9CEjDFAPg7GG3YzknPEYCTqG34n0bXnxjRcyv4PgB/g4g+mIgWAP5RukBENRG9gYgusph8HmH7vAFiLvx3SYLO7EM21N97hjbmLIz7c8trbheiBulTIxPWATjBjj6dsW6/D8CfIwkeVQH4wljXL9xCn5IW8bWxPzOSYCkvZuaHIGaxXx7n4GMhz+bNkMp/HGSz/v3Fte+A+IK+GuKbdztws3m52bp9rrgbnosDyBo7JKIHAPy94tovQxjYN8d33YyI/uszaNr5jBHRnyGiV5MEaTqCWJ6EW1nvEL/g1xPRFSK6H6KhTH0+6/37rQC+imLAMiK6l4g+dVcndtF6Rr8nTJgw4RQmxnLChDsLXwHZCP8HAM8A+FoAb2Dm34zXPwDAv4dsRH4RwDcz809H88JPgvgi/T4k0Mf/BeDiGW29DsA7SKKDfgOAz+AtvmfM/CuQQCvfFGl6FySIBSD+PW+O7T0O0RR8SbyWNtDXiGiXD+Ct4H8G8BVEdAwJTJK1EdHf6qsA/Hw0/foYZv5BiPT+e0jM3X4TwCfG8k8D+LRI+zXIuP78LdLzRRDm9hjCfH/v6PqbAHx7pOev4BbAzD8O4Bsh/m3vggThAGQTCoiG68HYrzdC/Ly24V9DNGE/C1kXLSTwyLPB/w3x5bpBRD/0LNfcLigAXwDR/lyH+LvtYni2rltm/h2I5umfRVo+GcAnM3N/RrtfAxHk3CCiL2LmhyFBn74UIqx4GMJ0pG/lX4UEsboO4B9DGMOz8DjkmXkUwL+FBKF5Z3H9BxFNFuMafs44x7zcbN0+V9wNz8WXA/gIiHDuR1Ew9XH8Phni2/0eAI9AzFZ34axn7H6IUPAIYor6M7Hsraz374QEE3oQwE9gczzPev9+A8TC4yfiO/OXIGt3F3bROmHChAnnBolr0YQJEyZMuFNBkjbjNwE0z1LbOOEOBUmql89j5n//fNNyt2F6LiZMmDDhzsKksZwwYcKEOxBE9BdI8k1ehmhef3jaPL9vgYj+EsTP7aeeb1ruFkzPxYQJEybcubijGEsieh1Jgt930RDtcsKECRP+KOLzIPlDfw/iS3u7feEmPI8gordBAt38L9FfdML5MD0XEyZMmHCH4o4xhY0O4/8FkrT4EQBvB/CZzPxbzythEyZMmDBhwoQJEyZMmDDhTNxJGsuPBvAuliTDPSR59s4IZhMmTJgwYcKECRMmTJgw4c7AncRYPoDNZL2PYDOx9YQJEyZMmDBhwoQJEyZMuAOxKwHwHQsi+lwAnwsA9Vx/5NX324chjwCCYw0AqMhDIaANNYgYgQmGAnrWqMmDAfTBQFOAIoZjhZo8LGtoCghM8KzgWEERY6F6tKGCpgAiIDCBwLCswQw02sMFBSKGj3X1rEEADHl4VmAM2ZcDCAoMQwFrX4FIjj0TGPLPB0KtPVTMX+xYQRNDUQDzkGvasQKDEAKh1i6e0zAU0Hrpo1EBvdeoVAARwwW5BwAa5XI/XVColLj6BKZ8LkD6CwBGBfigclblVEYrzuPCsbyKY1+pAMcqX9fE8EzQFOCChlYy5kYF2KBRKY/AlI9t0HmubNBglr56VhvtMQjMkL7FATcUgGLOKI6ZIgYY0ErqXdkKYEJTOZn/IP1PfSzhgshjNDE4Xk9z4uO4pTkkSBvMJOsjqNhfBUUy1gQhl0HwXoEZ0DpeI0CB8/oKQUHrIP2MbYGQfwOA0QEhEAITdJ5PBe8USDF0vF6uSYp9THRSpIdZaJDxBUKkgwEgtqdUQAgD3d4raM2ZJiKWe5iEZsUIgXKbabyAYRx5VLdSDO/l3nRPYAIRAMi4SEekJiLAe6FVqQCktcEEDgSVx1fqAct1pTiPSwiJBmkzhKH91K/yfHBCgzKDu1x5T7qPg6xP0uX5OM7FWmJPUCYgOAXSvDHu7EmIjP1HoM36Iu2kw3BMQztsFWACSDE4KMlWl19QBGh513GgWGb4C+I47kOfZNCEDihIfSr+zouMh/oZgOa8hvI1lvrz31wWQxs+3qPjsUr309CPVIclKafi/aG4Nw1XegA9AYY3j128P5VLbecXIA/3gwb6Eg2JJjXqK0VaUh9L5HuL+aDiWjkOxXkKAKd+pvbyYpY+k5N7ONczkMRU1BP/5r6mbut4PrZF8uKS+9VQliNpZT25fBjVVfaLi3I0/E71UlnGp7EahmSjrRDb8EUdiOefy84nYBDJc1EvS3tpDaZxzP3DafqAYtx9MXZc1E9FOWw+Kir1DcVYj+so703PR/nyL56DjTb47LK57+l5xzAuZd9PrS01qgubdG+0gaFsetWluvK4j/tQ3lq+VrD92saF8u8YGzfvwLhOFP2/TShfjxv9Ks7nNVPck8YG2DIGGw3sOH+7MSbmrDm4jTStrj/yNDPfe3tqm7ANdxJj+V4ALyl+vzie2wAz/0sA/xIALn7QC/jV3/wZAAAbdGQIVN5MWy+MZkgbtVgHAXmjLnUOm/7ERPigEOINlRZmqtz0SZm0YRtWvGyCQ97ojttgpsy0JSYv0ZS/JXEDnvoxpr08V27EVWIiEpMbN9ZEmxvcUGxkNje9sQ0ajhMtiXYVmaTUrjA7tLlpjX9TnUolpmKoP2/GwyajsbFh98IIbdQXIj0qnKJVCtDGHKVy5VeOi7kkJRvoi1bWijBtp5mucgw41pc2+hvf3OKLvnVDHwBSu8cqMx1pXiiOl4/tJQaCsTFWaa6kz3yqHmYCOwXSkZlgGja0aaObOzHu1Ob4pi9YZjJSe6mviQkp5jv1DzxiVIr+5zLFfJXMjDAto91S6v9ok53PKd7cZAORcSrrKf6eesCKNsPoeMwM+bgzMqfX3AbNIf7dRkPZvqeCeeLT11LdhNNjk+Y2M3e8+ZLxRfvjXU+5cU5tJObmZpurciyLzfxG3wJATOCS3vL+on6K67gsSzyco0CnrwFgYjl2aQ3EutMrVRfH5XqJzxZ5AhthxFhvjuupDXu5cyQuaB426PkYOLXhVm40BWrEGIzpzGOzScfGpj3OVd5sJibLFeui3MDxsEbIUx7bzDyQ/EvnKQAhMYZpaaVXQ9F+YjrS9UQTgMyElfeWY5U3y2PGMv0MA13pJPl4uIWxLMc7VHjWIB9pL+cg0a5H8zZ6J23Q56VDA2N5+n1BgQrmdPS+AeL6HK5TfA/uZCz9MCZ57EZMYUbJhMY+lMzwxnyN1+j43lh/HrvyPQHsZIiHDgxDMH5dZSY4nxj9Hp3bWT+N/o7Kbb2vvLegc3zPtk/BzroSveNzZ9G6rc9n4Nx0PMvyN8WtlgdAzzUeTLz9177lCx96bhVNuBnuJMby7QA+gIjeD8JQfgYkyfJOKIhWrFYOtfJYuhpKe8yNhSGPZ7qFaMG8xsxYHPcN9qsefdBgiMZJR23ezDi0zqDWHj4o9ACcNTA64KDpcNjOUGkPTQwbRHvYOQMfCAdNjy4ysZ0l7NUWy74CM2FeW1ivC2aHRRtpPLQKOG4bEACjPUJQ8LFcCITaiIZRK0ZnDbQK0KPNWGtNZu5MYqiZUGmPrjOoKo9Ke6xsDWNs1MTozAw2VQ8XaXdewWjR4IlGUdpNIEraLgXvS02UgtY+M7LprZg0Y5XxcF5lZtsYD+d0/ivaOZV/V5WD9wrOaTTGwfYGSgcoFdB3Mq6VCXBOnWJKS4ZY6OU4nshMtk3HAIwJUIph1xU4EOoDB+8J7HVmSrUJGwyw62W8lJGvnIpaIYIIGyjulIIVxpiqkL+YzApEQTaokVkYNJ4kWi9PoNkQPV9pj+CMfHidAum4O0kMUvkh5KjlAhB6DVXJrpUdgF6BK4AaLxqHvKHEoO2KjFQez4JOMMCeQFqYx6wt0pHxCATUHnAKaALgZb5ZCfMJpoFJ8gSOuwQyPjOmSJrHzAAw4BSoCeBeARUiPQy2BDLpqcLAGKSvoFOAiX0jFi2fo4EJI6mLPW1qwdISdnE3mjZCJcMVIOcJEpfSANQrGaM6XiehHTVvbFTIiYaQS61ZYv4SI4i4aawCqFXgmje0XmQpMyBQ8pubgfa0SWXNkRba0BDqtYLfC3EuIsMRN6hqTfD7UXDjCVxFBqv4W4J8ZKQMC11axiSVz0yJHjbD5IXevBFO8+GFkUuMHRzJJriO608zqI9zVcdxqDA8A3F9kSIgAGZN8LXQBiYoK5t1Pw8D3XFc9Jrg96QPqif4BUO1hDAbGEjdpXUa9+SGwZoyg8Va7s1jzwTlAV/zwDhEDarqpa+pfAJrqTdt1pWlzHxsMEg2TcDwO9RyPUTmgSD3KAv4GcOshGlkQ5nxQkgbfmFSVA+wkbECkzC+WubI13Je9wQf54+83B/qgWER+gmhGsYsVNInZWXc/YyhIqPLKtYVZKyUo8ywhGK9pfKhYuhWysg/hrJCK1gYR+UA3wC6A4KJtCn5bffwrKF66avQg/xsUwBCA1AcL9VHpjjOGwDoXuaKdRIoSB9kPaSydIrhkjFKBzLprACzlj4KAy5zw3qTkUlCgERvMLHtSBv5YcxKBiVpQ7MmNa6j9M4RAVFkVOPnKmmCc98R15YZxiSNHcLQP3KxjUKLXWo2swAjrtnMHPPQx3RPyfQmpHpLDfoGMzsWQoTN+2WcGKwoMzhMp4+zpnbUThr/U8KTMSM9YiwTbSPZ1YYgKQlfNmgfMd/pu7C1b1vKjRn5U4xlqm/LWJfYJtwYn78ZDXLD9ja2yFruCrz2z+zxtev+ttf7q7/evZWZX3fbK75F3DGMJTM7IvrbAN4K2Zr8a2Z+x1n3XDBrvPbe34JljVWoUUWz044NbNBoLjp0wWRzSkWMLpislXSFhjKdd0FnE8W5tghMOHIzVBdCNmFVtGnKuPY1GuVgWaEiKZfgi6cpsEKlPDwTXHzTNNrBM6EPBnNtI50BVTSTTaa6CpzrTUyfZ0IVaTDKY+0rBFZolNsom851wSAw5TYBoPUV6mh6qojRpzcghOExNJjGAmJGamgYi/J3GlMTTVvTPanuWvmN+l3QMMrnv30wqJXLc1Brj9ZVeZwda9TKwaiA1p0WOQeQmEXHuQmFOXPZh5L+1su81/d4GOVxYhsYFcR0OmxqvBNMNJ/tvayppCVPWm1FnJl8F9SgMS+0nql8wKDBBoBKeTH59XpjTGvlN8a8NM8t70/X0/wljbgiRqMdbNCwftP8uNT0J1pLzf+GBrjQ3hoVcv0JY+1/Ok71VtrDxnErx7YUZqTfYpIuptQ2mmm7aIadTKx9XGeaeOOeVA+P2gCQn53AgzZeRUGDK7TZqf++sD4oNfXlcRqvNHdle35kvVCOy1nlKu3RWoPa+Cz8SVYUYwGTtDvMhY5aaR8IOgoKnFdF3QG909kKQdFm3c4PtJRWGKU1Rpq/ZOodinVd/k3jlOZBF8+GHq2DNGdpH6Fp07KDizWb1n05j7mO+LuKgsJyDaQ5Kq1JADG7FuHYcJxMuweT7mHMswVK2Gy7RKpb76CvrKe8Vlo0ZCuKM86neigKtqigKZmfi1AoCTni9fJeRGuWOPiJJI4CNQBRg8ainE9yjFQubPYhzWXqdZrXUMz5lv27rNtyR1sIDBHb4yi3Skla8vxEa5C88Q2nx11pRnDFRN4iBosQ2rASIGKwV1kbvFEOQle2IEkCu6JvY+ubDUsP4PTOPAnI8kRhYCi2bc6HCcApC4Z0XJpdj7Gtzm1l0riUdY/NvbfVm47zPamrUQgZjwFsaoPH94/pLPvMp+nYMA/eRV+qhxMd2LyBeXPcbsYUbaHjFF0F01T+3kkj4eZzdEYdOyl6FnVt1Huzcuep5zxr7y7Ctesev/zWl972evULf/fqba/0WeCOYSwBgJl/DMCPnbe8Y42WDR5uryCwwqVqBcsaN+wcx3aG+2bHWPsKc21xaOe4Ui3xVL8PALi3PsGRm8OxwlxbPNnu43K9xrFrYChgz3SYa4u1r/HY+iLuaZbovEEfdN7kX6lX8Ex4qt3HwvQIIByYDk+0B7i3OQEAPNPPMdMubx6XroZRHitXY+VqPLA4BACsfSXMsHIAFJRyOHZNZEwU9qsOfdBYuXpg9kDYN11kZIQpcqyxbzqsXI0XL27ghp3jyM7wgtkxWm/Q+goX0GYm47CbY24s+qCxX3U46meiEYx+o/tVlzdyAHDUzzDTDrN4Tx9Mbm+mLVw0SQaEaZ1pi2e6BebGImgLxxpLW2Ov6rG0NS7Wa6xdhUornPQN9usOR90Mc2NR6xUO+xnumS1x3M/QB417Zg6GAo5tk/ugKcCzgqYAaGS/VEUBy9ifRrvs53lQd+icaHqP+wbWa3zIPY/DM+Hx5QU0xmGv6nPdJ32TmShFjEuzNRTE3xMAOmfQGAcfFPbrDmtXwQbG5WaFHgbHtkEVmeq9qsdRN8vlEvOXNrwHVYtaezzlZJ36oNB5jfnMwnmDvarH9XYBrQIa7dB5A+s1jAqooiZz2dcwKuBC04rvKICZcbjSrHCjn2NlK2gVUCuf11FgQogMpw8KNm7kK+1hoyadAOxVvcyXCjLeCjjpaywqi0p5HPcNLs/WeKadD4wWUxY41MqjjePVOWHq17aC0cI8NUaeFa0CiAltrHtlK+xXPWyohT6voCrO1gBVZYV2r+GDggdwZb7CytZoo9a9Nh6N9pgZixvtPJu7VzpkWnuvoSOzMosMUGsNtGLMtMeyr6AIqI1D7zUCA7PKoQ+Eq4s1FDHee3hRGBMQZpXDuq8yM6kVY6/pYCjg+nqRGUyjxYKhi1YSALCoLE7aBlcWa1xfzeEiQ07EuDhbo3MiKHJBYV5ZdHaeGb951cvz2jaojT3FfF1dLPHo0QV4JjRahDh9ZLYuz9Z4/PgAzGJZsXQGF5oeJ22DmfFo+4FRC0yoFUMrj3VfYV5btNZgVjmctA2aysHFcfKROW4qh0Z7HK5nqIwXl4K4xhrj0TsNoz16Z1BVDjPjxLIjMoSzWlR1J22DRdNj1dVRKSvCDmbKjON9+yc47GZYdTUUMfaaHkYFXFsuUBu3UfbSvMW1Ezl/Ydbh6ZM9XJq3uLGeZR/i/VkHQJhjTYxlX4mgJs6L9Rp7TS/ww8uEAAAgAElEQVQ+4k76O6tlHivthWlignPSD+s1Lszbjflp+wq909mFYNH08NEFIAllAhP2mj7PgfMK+02P47aBUgGdrfL7wAWFeW1xvJrh3gsnsEGh7SsYHaKFBWdBQ2cN9mY91n2FvaYHEaPRHl18rk7WDWa1xYVZh2dWczSVQ6U9KhVw3NWojTDyrTXZ4mUe6Wz7CloHLGoLIsbhUu4HxG2lqSwUAauuQm1EAFVpj7aXd1gIhKqSPvW9xt68hwti2ZKuzWuLwNLWrLZYrhvszTu0/VDnxcUaTx/un2eLsRWzxmLdik9+VbvMAGoVsF7XqGon1jaNhY+WOj5qq+czeRat1agqjxDErz54hflc1o1zg9DDGLEEAgAX11PpKrK/12LdSSwJZzWamYW1esP1RSm5BgB149B3BqaSejkQTBxT2xsRksT7tA4IXolFDgPBa5jKiX9+UCAl1j5iZeTBLFZezNKOj64B2gR4Jz7+pvawncn9SIIAbTzAMhaJ4Q6eMkOqtJgIJLcQtmqwsskWMQDHPg2uJJF5Lyxlsh9odMXI/upnuR4EEhcHVwhStzHtidEvrVs0i6VMKlMwzVtVbWNGfMxhlu4NpVAhtb2DgQaQzdy3YhtzSoMwSUiJQhLi7JKQab5VBvJmDPD4+vsQY8kAQl4E73u4Y/JYPhvc88FX+ZO+/VOw9hV6X2jCWG9IwEvNjQ0aCoOp31jjVt5nKGTtz/hauUkrfTPH2hG35VquA9iQ2JfPaLpeaqN2tV9qW0qMfTrHKDczpcQ8MLIGQ9GmRH18rfxb0jWWwKtCc1GWKbVbic7ExJTn0viUGozEAKSx2vYqDbwpuR9rFkLBQCXNXTLxTdfGfoCn+zdcK/1Jx76l5XiUwWxK+lLQnqQh4NGaGWuMxrSle5N25ZSkf8vv8fltGsqyrXGgmfJ36W+70U4R9IVHH8VS45J9YMe+sRg0ALu0hmM/WFUcA8P6oGLdYHRt63dsLDIeaVJSf7f5MJ/yAcYWTcRZYMjma/QNypqZsmjYHKt0brxe0/Xkh4vRM57Ng8cagW0f+3JjVG5qtm1+cuO8/d5U5WidpXOpHGmxN9sIKFTWXa4ZHUSLVFwnwun+AcOGr9z8lYFJUhmMypcotTXjjWBJX/nCGr+byw1beX859ozTAWQSreVcoThOpvMMbN3kjWlMdI03utswnueSzvE4l+bk2x68U1UXm+dxn9JxuWnftuHfGE+I+fqzxZkb4hGjUK5FhmzqmU6ZNp4L28pt23xv2a9uecxOa8TG923r503WwjYtWx6Sre8P7FwDG62M77lJ+6ew7f5d/d64b3tfT7VzRj+2ayzPaHMHtr02b4rnWO6mWtLbiHP3KbV91nN4Bn77zV/wq8z8X936nbcPH/lhM/6Pb33xba+3euHvPe99A+4wjeWtIjFlD8xuYB1qXO8XAID762PMVY/3tpdQK4cT2+Ce+RJPrA/w4r0bsEHj6XYPc2OxMD2udwu8eHGIa90eLlQtWm/wTLfAYTdDrT3e/+LTeGx1IWoeQ4y66vHk6gCKGC/aP8SNbh6ji9a4b3GMp9YiEb06X2Lp6o2NrA8Ki6rHwvR45PgSiBiLSqTXNig4r9E5jSuLNaoYGfbE1pgZh0a7gckD4aQXDU5nDa7uLxGYsOxrzCuLx29cwGLW4aDpcW25wP6sQ6M9jtomM5z3HZxgZasscT6IUuokLT5qGzhnsvnaftNjbSuR4BPD6IBlW2PRWPRONF5JszKrLTprcGmxxrKv4aJf5qK2WPUVFk2Po/UM+7MOvdM4qC2Oo4ZmZSscr2a4tL/C4XKOedOjNh6Hyzmc1bh4sM6aDRc1dj6o7MuZmKu9eQdFQOekP0SMVdugqmQcD+YtKhXwyGNXZL7uPUJrDbquQl2LdDb1rY6+osu1OIk0jdTRVE5MNJnQtRVMJe0sj2cgxZjN+yyh7roKs5lF11aYzXt4rzak0qujGdgRqn2R8mvNMMZjva6hVEB7PMfsoIP3BO90bst7ygznPLbXtwbNXLQ7tjdwhzVo7jHb79D3ZohiqhkqajeC12IaqQO8GyTfifm1nYaqJFIpxyizZmZhWwN2CtXCoj+uYfasSJZZNv/aRB/itYGZO7jWQNcBHABTewQvziLeigaSvQYRYBoHu6pQ7fXoT2qoxoOURHr1Vku0VACuM8JomJCjstrDBmg8TCN9805JNFSroPYsSEWTR6fgrQYYUI0Xk0HF8K28HvXCgQMQegPVeIDlmCrxgfa9hqo8/NMNwAR1tcvBlnyroecizecQJedrLZLj/cGPNknSyQRpnwAsDehCj3C9Bu87KDNEeOUblfgNEiS6a6uBfZeZsbAy4sO05xDWRuqupG4A0NcquCtWzq0M4KJPIwP6WMNdsVK3VaCZB44NeOGBVgF1KJgiAL2WzWwdQK0RX8uewAsP6kz2n0xBjag1IEsICw+ySnwRk4+lVUCiswoyVpbAi4Ac2fSwBgjguQetDHg+OCFl31MjfszV0xXcHiPMAigQ9Er8YN1FD+qi4KiSa+ZIwV7xUK2CXhPsRY/qhoY7kP6yZpijqDWK/mF+xtl/jwIQapY2AIRGfK50S3B74kvJBLBhsJFyoWaYZZzT5FNYSz0kShyYtfg+MuReCuKbaZaU/d2CBsyK4BYcff3E5zDVq1vAHjCaawqsEf0jkf3KlBN/yFBFv9RG/DHB0Xcz+t/ZPYZpCWYF2H1pg5z44/noXwiSY+VkfPSK4jnxndQdABZ6dBtpNIDqZBrdPPl5Sn1+ztmfTPUUyzCq4zguBgha6jKtLEs3E19KtweYFeDrwa+wOgG6K3jWMGvALWRuUl8o+qm6vUi7ljFnE+UE0Z+xWspchSr6WypE/1WpV9YHMrOc/FvByAGHkulm0EB9KLQQi++kWUu5HJ0W0a8y+jXqNs5NP7Sr+/gKqgffVCDOu4m+vGmNtch+ranfoYrjAKkbAMxK/HgBgBwjVBTXFsPN43k/9EV3DFZAqGjD3zIxUeSQfRiDJoQ60sUABYZKYQc0xA8ySroTsxIMiY+klr8UaU9zQ0HKaFs8z4xBokXIvrMZibEphDpjv01Ano2QdtqcBAxyvBH4KFUb1zoQ13z0JS0Z4exTqYd60lopowuP68w+rLyDkUuCxsDZbzT7km5hnrPP6Za6tsnLNq6NGPwzBZg0zNeu+nfiNjPAtwcMP5YWvw/hrmYsL+kVPvPqL+FGWCAUmsWWK7ShwkfsP4RjP8sar9kli2VoYFljcVnMTnwUnx66BT7ioEUbvc8r8piRbMqvu328el8C1Pr4FNhgcHBPC8sax36GxcUONhhUyuHEz/Axl9bwIKx8A00h+38ms03LGpY1PurSQ0JzTGeSzDhnyuLYzwCI/6f4cA4+f6lPyZ+yIYdVqKPvmfiULl7QSztBo3mB+Fh20XRV+qBz3R4KGgFdMPkYQPYVTOiCgUbIpptdMFgoaWfs09dFv9FjN8Nc9XH8VJHKRKNRFutQw5DPtHTBoKKQr9X3id9lAKG+z0Ej4MQ32b80YPCrSz6WJvqNrn0d6Rn8YisKsHG99MHAM+Gjrj4ED4VjK2OezGwVBfTBiOlrTPky17Iu1r7K55NvraHBbNjEr115f6M8uqDRRJ/YMfZMBxfnpfQTTf63c22xdHWuP/uRguOcDPQa5dH6CiamatkzHfpgcrum+KL10Qc5rb8+CLNe+nSmtD3pdzpO5xWJz+pMu8F3VXm0fnjNJD/b0ry690Nf05il6703qLVD702eU2Dwt+3jF7tWLpqyDm3NXmbRezHXVnFtpLrbOAapvjrupvrCBzuVT22UFgzjtR6YsHi5vFNWrs7nkolpgiLGTFsEVnmMyjrKZ3theixtg/lLLVpvNrSOlfa5XBICdd5kX+wqjmMy52am7KMLAM0rxRS5DxpNNNUf1m1AV4xj8vFNbgBpzss5TWsmPTfJRzm5DZQ+12ntlOsgWU/oGGwt+xxHwVrydy59YpOvsB9ZapTzOv9ACxdULpMsE5L5ePLJpXitdQZVNCtf2QqzaPIMiECw1v5UOynFUCqT3pnJj10nC4uC1tIftdZ+Y4+V/LLTnIz9VEu/7tJqJtVXXk+pk5Kpa/NBLgsV03gCyPU4n1JHiSVKQhYsORE8acWoy+jihGxWy9HyA5ANRulvThjSLymmDYud/CzE+Uhjogtas19qoBxITu4RgQpSaqfof0heSWCgYrPq4/lnC1YMeOkLJ+uWZC0R22MmhCINU4ItrFgCDXyLlI9jsWHxMNwcxppCJvgYIC/9tkU07k0rgDjPKmxERS/bSWmXcsTZdF+yqEmB6cbjwRj8RZNf5NiigpGFjKWlxFmmjrxLMzrW6G9cw6ZKb6xx3NbelvE6E2dpT3dpeRmnzUjP28b4/BnM19b7Nujdou4c0TJomfPAnEHPzbm2czGA59VAnneOzsL3PPcqnisYQLgzOd7bgruasVyGBj978kEAABvNX7swdEmC0cjvtOFKaUnS9ewrFBlGv2XhpmAfAHLQnXR/uZEfB7Yp6wU2GcLydzoen0u/y0BBiSGT4yGojSGf83iWgX7KenZhl7ltudndhm1mr+dBGp/ybzIZ3mW2W26UxrSON0dj89Jt/R3370G65+Z0b6mrFOKlOretoXT/eG7HtJZBXcr8l+MAKGMz4VvBWfMViqEvN4uluXOJdH48B7fSZnl9bNJaIgkPtvX3vONQbuB3fyu3p3EpUZoyl79v1oftdYzP3/x7vWuN72rzrD6cLrtJw3no2dbWzfo/Tkezq/3Ni+df68kMenzHFsH7qU30We2dOS7jDdI5NkvnWjdndaTcGO/axZ1jnk/hrEAuqf3znk+0lXSMf5fnxtqMbfXubP82bD5vBbs0L2MT6lMamILx28ZsnGct3YyxuNkYlXNQrp20JsePw456TzW7pV1K9W1p7nQFp5mdjXIb6357Wzsxft8GnOJFz6TtZuex+zHcvP9863Rg9M5TJ87JoO1u61bK3/Tazep9lvVNuLNxVzOWN+wcb3v8A7DsarigMKscfCD0zsA5haaWgCqV9uisQVO5HAigrlw2m9Q6oO81TJHCoqq8mFcyYb1qokO+xLJOTt9V5UEEdG2V8yFKXQZ1HQNDWEmnMd54Bq/gvcJs3oOZ4JzK9wPyt2urvDnSMaVFCGrjY6C00GCMmFmCSdJTeI35QkwevdOoGwvbi9ZDaZ/9/2xnsomd1mIWWEoWtQkbUk5ntZhOFvksU3u5j4nxcwrKBPhOQ5kA0pzNAZVhBKugap/TcgSnxMyy1yAdYGoP1xno2gtdnqAbD60D7LrKZoOlBI9UjNyYaOh1zP0Y/bIY0HWQqIAESWERCM2lFsyE/qQGaY7mnrGMVcNGQQGqljH3vcr+WqlvqvbgaGKqawnOwE4iBeaE970WE8SyXgKgGLqWtCq2NYM/oidQLSk5dB3gW50DFnBK8xH7DmJwp8VksIopOgBAM6q5hbcaIV0v/PBO5T0sUnKUUl0yRXTJWJatkpQqisG9gmo8QmuGfI5+qE9o0nItzZ0dIinm3IopNYYlMb3sY8oRq6K4j4Y6mMRsklJbkdZZnAs7jAE0i99dGwVEHPuZ2k3pMRhASnOQxrAM3pCCPBTHtHDybJ9Uw85Gs/SBMOysanlXcFskvUttlpv5xosp6MJL2cInjeZisguOfa4Y6FRBezQ77TS4CkO5SL7aswjLKqb1CEXQCoAaL+axcT6oV+AmSD9M7Gu5KU5rwUU6UhCMVD5FFE0pUKrYXp/mXeaMmIb0KAqS2sSwjHuv8macU996Ba6DmM/G5z8FpkhpTLBv5XlI82qEVlrrbPqbU57MPGgVz9dBjhdezIzjM8DJDDgFwLBqCG6RaK6KdR/bpE5tthcg6Vg8ydhy3Iyn94kb1gGndEWM3FY5DjlYSBOATiOnn4mpiCRljJhL874TupzMNSc/UpJ52kgrU8f645wjQPpRM5Dqi88UFA/PcYjpZIjF1LmJ82WVtJnmL81BnC8x7SYxo05joxiqV0N6hVTeUjZvRkxTwoZzKhzqCVwzVKukXB/zknoCzwLUSWnTeGsITYBqoxl1MkdHXLMdDebKFYsvJ1POX5lozms7IOYFLdIFlfkso8lk6rMcJEKAMA9QXRwfD3AdU/wUvsLpmqwlSaeTxoICiZkmRbPmIrhLTr9hpC3ylE208xZEARTzghILjcSDqXVZTzK3zul10vPKgxm+pJmJc1iGDI5pRJJJJEmMQ6k/vVvjmJTMaz4Mw725/JgBCwOtZU7PnBM1pawZMz5lPcnENZ0fjf+mOStvTSOSjlPnWMVcpyPGd4PZ5CH3bNbQbxHi5L7RtrawkTN2GACcxmhfcDNw2c5A8tmM7JiG8Xzd5XhfDt5zVzOWV+olXv+id+AF1SFarvCe7h5oBLywPsRFvcRvrF6Cfd3hGbfAC+tDPNxewUtm17EKNR5tL+FStUKjHJ7q9/GK+dN4b3cJl6sVTlyDG3aBp7p97JsOH3bhEbxrdR8umDUa5aIZpsdj7UU4VvjA/Sdxwy4QQLjRz/GKvafx7qVE/X1gfgPX+71sJulYYe0rXKlXuGRW+PXDB6CIcU+zxNpXWDlhko+6GV7+gkdzypNn+jn2qw77ph9yXbLCtW4BxxpH3Qzv/8DTCKDsK/pfnrkXD1w8xNXZEg8eXcH9e0eolce1dg+dl6iorzi4hmPXoFYeT7d7uDpbQlHIaUiudwusXZVN2u6dneDIzrC0Enl0biyurRe4PFtjaetsClcpn035XrA4wo1+gaWtUSmPC3WLZ7oF7pkt8cTqABfrFmtX4d75CR5bXcALF0c4sjM8dnwBL3vxdbz35CIuNi0WpsdjywtYdTVedf+TOO6bjfZs0OicgYtRQ7ViXF0sYVQQekmif15fLzCvLHxQuG9xjJl2ePvDLwUz8OpXPoKlrfHMao4Lsw42KFyerbNG1QeFx48PAACX75XzKVKqVgGH6xkWtQUDuHa0h8YEXFys0cdIqEdtg0vzFsdtg4vzFjaawlmv0XuNo+UMtje4eu9xjli6V/d48ngftXE4Ppnj6guPYb1CZys0lUUdI2lK9E3CPXsrtM7gaDXD5ftWACSC5tH1PVSLHvddPcKyq2FjtFStA2ojTFHvDCrtURsRxqRorimC5HLdoGksrDXwTiIG7s07rNoG3insXVzh6HiOS/cdo7MmByRqKoveGbTrGvuXVliuazSN+LAezDt01ogPacz/mgQte/MOJ8sZ9u9pcXS8wOxCKxFtjceyrXOE0OW6AQeJ0piiqh7fWKDa67G/6MT/1RrxLe019u5dQakgZp/WwFoNDgrNTCI5KhXQtWJyPL+8QgiU/WOZgb6rUNUOSjH6zqBuHJZP7oEBHLzwOPvMdl2FvUUnmvEYAbI7bsCe0FxZ52BH3sv6ShElAaA/arD/whOcPLmH+nKbo0gyE9rrM6Bi8SmdM/xaY3Z1nX2Lu2UNMGF+/wnWywakpG4fI076J+Yw97UwlUe3rsCtFsY9EOh6DXVfK+amrUFzuUV7Y4bqcge7qqD3LZQKWchmWwP2Cmbfwp1UoLkDtxr6ooVfa1Atuz1difGPP6mATkEdWIReA0r8YkkBvtVR2KShLjj4ZQXqtDCIgUQwcSyMO13swScVsOdEWMA0MKC1R3AK1WMN3EEAz70wICstm+UrFmi1bLAWktdVX68Q7u2BVkNfr+Cv9tBP1/CXXBayqMMY0dILQ+AXYWAQIlOplrHeWs7rIz34dFJkfmqGWmmEJsBcMxsbXT9jYcYSX3dD/FaZImMQFGAY1TNCCxMQKkb1tIY9kA19mDFUp3IeyOq6gb3sMXu4RjCcc0iGvOkX5sDPGPU1JTkvr4uDoO6S3x7DXmBU1xTMiYG9IEyC6hH9TcXvjVX0i3RyrnpGhBRuLr5wZimMen+Jo48i5byUyg2+l6GW+t2Cs1+aOpFvn58B80eN0GVY/P8OFfRaNqtuAeg1hN7HDPxcfDhDBdSHGu19z36Hap40sAeyHvShyr6C5KU93Srxd1wNfpHJ17J6VIGN+DOatfjesZLr1YkcJ6MrYuS8lADgI/OWGQwDNO9WsPtyLhjx4fQNNn3pYo5RcPQPnYuvZKikveQn6uaReYqpWHQ/+E8mv1CzouxvqTyEua3FdxMk88IE8X+NfU75VUMNmCXBxRyikv9UBCi6BVhR9PGkDd9TxHEQv2ZGMOJjSW4Yd+U4+n7SwAAC2dcyVLThUyxCmsRAi99lqAjKxhyvnjcY2+SLGsodc3xe09pMYy0M80BDeraEnqE+ME4HB0NiOuMPxQMzmxjBoh8pj2XQHH0sOdMA8EZgJslfi7xeNxDr3/DfVMXvMY3R9zJdo20q6jEjWeQATeMGbPK/UvdppjMJlrZabey4d5sifsIfDu7qqLCvfPWCv/aHXoUbfoE2VFioDpYNWq5w4me4qFdouUJFHqtQY6F6nES/xYXqsQo1PKvsF5nOKTA0Bexr8aF8xu7FNCCb/o4H8fqhm2OhJT3FQvc4dHPsR2/2xIQOOfTExzL5O95TLRFAaOObZ6Zsbv/Yz7KpZmoz5dtMZpAL1cNDYaF6POMWuWyq+8Q36ILB5WqFQzeHCxpz3aMiH1OzLHLf5rrHkZtFOsWks1EulwUkZ2elfPY/c6wxV734QqrBF1KB81gtvTB1pd9XFX0M59qii/5zS9fEFC8VGuUw1xZH8ZwLGpYV9k2PWjlc7xfZV7L8m8yTE5aukTGJDKj006KLx8nX7kVzSfvyZCsBmRIdickuTVj3YrSD5KtYmkAPeTilDsdKfN9ITKVr5TLTnnzsShPZPdNLOhUnURCSn+F+1WVfy2PbZP/FPujM9CafyBPXoFYOMy2BqwDxx7ynWWLpGhzZGWbabvQp5QtN9CTz6tL0GoCkmYn+aUlQkvwfxceywr7pMg2AmI8nv8yZtjk1TTIhb73J6VySD1vyle2DxsL0WLkaC9Oj9VUOlJRS4gBArWVN9sHkcxfqFq2v0LnIQMd0Jyk1T/I7rJTP/nalAKExQv/aiS+tViFHlS7zkCahxsV6DQB4plvkcayUz/MPiMnu3Mgznnxl03uBmbLfYVoLN7o5LjQtlrbOEYsB4KDusu9goiGlr0nBwQITWledingMAJebFZ7pFtmiQ/LryrgdVCL4ST6PrTPYj8KTxjhYrzciUBslApvOGcxjELJKe3RRSJFMvtM9jZGUQUkYU15LeU5NzF1aa4869g0Qc+iZcbFvJvtA5hyZxMK8xr7eM19haWtJpRLbVsQ47hrUkbaUu3S/7nHYzlBpj0VlcdjOcHHW4rhrch/mlc3zSCTpacr8tdbrTF8SAFZxDMu8qD5QTs0xM0UQJwA2qBgITSxAGiN5h8e+kWl9MiRdR0rLk3KaasV5PdTGYdXVWZiV/FOTFU3ybXVeoakceqcxi6lACIOLSNtXqIzHzDgs+yqn6iFAUvXEOvsowLBeozHRBzcGd6vj71U/pERJOVIVYSMoW6IrRekux7COKZ6S0EbrIeVSWkNdygMbBWSeJQXQsh2evVuF0QE25oDVOmTrHUUcfVAlOraJAcuSMAgAqsrF9CMStI0j7WUqFe8HH0hV+EyW0cDlN2Uhn6wJwJgQU5wM9xIh0oBslaU153M6Wmt4T9kyCUC2iMlpOcKQeiRF7pb0H0VKkiCqRW18juCtYtqSNEfeDxYGZURtBjb8OEOyyEAqGxnBQJv+nsSFBo2GqOPpcrQCyOlGssYych+JYUllktXImEtJ9yeUW+dUrqAj05CYx7G97Xlwluau/JssD8aavWL8ksWDaCuL8RnTsq1uKq4V9d2Kyf+tmKjflBncpVU9Bx0Pfv4XPe+RUz/8w2r+mR+//7bXe/GBh5/3vgF3ucbyabuPb3nPa9DEzcXKRg2DkZx2R91sY4Oy7Ov8ES8/ir0z2G86nHQNGiMRPnun0TvRolyatzjpahgd8oeaiHN+vL2mR5+0DM5gr+mxiia3s8rBus30J4EJtXHQinGylvxsKdl1SsbtPWE2s2J1RSwfZR1ynraElOTcOYXZLEYAtSZHEtU6oKo82rZC01gQIWpn5CU3ixoaADkflUQZVdmstwzrX9Uu590CccyRZWAq0YaoaLILYmjN8E6hbsSMON2TPm7GBNg+5tRyCqbycFbnHFu2M6hnYsKrdQDFfFzslJh1Rs1SGYwgxMAAHE3HTCPzHYq0L97FHFjxA6h1wNtvvB/AEo01MImZrmFwALSJvp2xLd/rbFKbPopJ8xSsErNbQMxBo/knGJvmvlbMhJPZMTh+MDstH7bGZ5NeZTibv6KPUTqZwI4Gk9b0QWUx1WUvEVApRkRlq8T0rAqDiWgKolCagiZzyGxmi8FULpmmZtO4+KGMpqpgkuMuRg5NGwPNw8fVSdRPWDW0GSN45o87Y6jbsJRN9Zoo6lQYaAEGc1fF2bwymx8mk0FHg7lfU5g0pr4kWtOmx0rkY1QstPvYXqIvmQpH00+1jMG19v0ghU5mqukjHOkiFrO4jY83UvvyfKtWIcw91Fo0W2UydLWO0VQBQCObBmbae5I10QQxoSORZqeP7iPHGn4/iIS7jyZw6ZloSa4VpoyqVQh1ELPEirNPWCoj88iDKaATU0SyFCNicjYvI0tRi5LMHZFNS5OJYDKNpJ6gnEQpJZY+qFbeI0mjJRFL45xHU69kFnb9RLQbIc6B6iUwjJ+JRg1Ajqx4rSX4hdCsrGjdbqwJoeFsPqc7ynMpmoKoPWBkUzplkbUWYt4n2hqK05PoS5qZGNcsQ6KcDlqCsr5yn6v7Ya/GGjjuSPoVhvtlzKQNPwPak6hxMryhjSBPCEY0GutONIirLq61GE2VVYzy2hGWNkaBLbR1oSo0NtF8MlQMF+sJlZTtYjRPP2e4InLtEFkXsG6gLUn5vU8AACAASURBVBQxznw0wwwVg1saIpQqgC0QonYnVECI4277qHGLG+S2A8IenjV8D3CMfsoW8IlXCHKeLQANcIy8SnHOATkHBSgNIEY/JQVoAtDLbxXXo4prKjFQlOR7iV9RAK0BE7WRaZ7JbPIwVEQlVRbQMdpuGjeJuDpoMNO6y1qrQguXNNJJu5Xazc+SGa1NQtawpWdjQ3uX+uKkPBfrPrUjDFHRnsKmRjZqMpkKprFkrAintW+MrHUD4nGMJpujyqbxS3xlHI+Ebaacpaltlm/Hb2guXzBFp86l48LMd1fU2I0oqTT8LrV1+Z50Ln6Hb2qCOur7qfJpzauC/i04Syu5E2PmGFt+78BWzekID968mgnPEXc1YznXFh966VGRTAaDpWugKGCuLTQCbtjFhqbo2M6wZ4RxSFq0FHxnT/dYLkTrliI2Jo3KhXqdtVZllMzWVQgg7JsuR0rsg8kaFgCimfFmI6BPoseogKO5aAhL6Tmz5Necm0GrVGqJysA8SRvTB53Ld96gUh5Hc5G+N9phZWs0xkGTRHxM0QMXVZ+1MCmqI4AcwbZ11YZGLWmJ0v1JKp00TmWUx6R9SFqMdD1pNxrj0DqDWvusOREJt2hFVrbCorJonWi0jApY2SonA7cxl2V6D5VSfR8IipCl+inwC0OYfxOTldcxuub1KD2+OG/hmXKiemaCiVLwpIVKfrpVvCdJ1ylKrI0OCCztKOLcllac05akv2UeUJH0S5LqpnI5IqRWjG5PNC7W6XwtzUuZ55Oj0CJE4UjSDngmtG0FY0JOj5KiCRJBTBsxaA4I2AgulKM9epUZ6SQVN8bDxsTYVeXR9wZVFA6ke9M9IqlPUvM4vlGynwQaQofQlsqmepO/cjYfjVJzF32DVUxFAoi5qtIhawaSwCYEhboetERJo5Ck90nK75OvrEnSeMrXk5Q+lVM6wM7F9Hc2t4OWwat8f2rDOzEFrCqfv9pJOJIZepLUK3XtYecmJ2JP9bqFyT6yKe9krg8D7aYS/2fEdZKEUq4x0HPR3nmvcooVAHC9hp7JDp699NkvpB/BKSjNGzsD9kkzwVmwwl5t+ECnZOiACF9CIKjKx77K9exfmDQlsb6QTGCjcMbHZO9kAryVdtIrMaRULbGOvtHiD2jivDklG6uZmMpmzQSA0Gk57wneKqDxCHM9+O8CCL3a3LkloUwhpEljn/2GQyGMSQw2xXIqMu+pLCDXCyFA9g8tN2fEQiMh+3e66Jso1zEIK5LvZx3gm8gF5zQFwoDnza+OzF4UVgADgwECQh3grIrmjclPT/qY/OSAYePPpmAeo+8hOQBMCE0QU8i4kaaCKVQbjOWwu01mmmzE1BeKB0bHST8JkXF2gzltFlZEk0Y3P8cudQeUFcZbxobyxjoxvNk31NHA+KclkYQEZVqIOEfKFpt/IDNNZR0Uz6f7/Ez6mMcwmc6WzEEUeKT2UzqNzCzGV2Ew2DDNTEKalPYmrSMQR4aHCjPMKDyIgj5V+JUnxjJo8eNMfrJg2mBiNxjAtN4LxjKPxxbGcrB7HDFC4/VYMnKbD5T8zxTLEMaMTWl6Gsk/fVz4tuZx5IIhHS+7EcO0jREbM6MbZqs7GMutdQCDye625c+jY9r8W772TvV7S/mzcFON5JiuczKWt6IVfb4xRYW9Q+Fi2o6Hl5cRmHCxarH2DR5bX8TS1rh3foKVq7FvOjyxPsCleo33nFyGVgEX6hY3+jlcUFiYHo8vr+JC0+Ipuw9DAXtVh0v1Cq2v8O5DuWaD3mCeLtZrKADvOb6cTaTmxuKR40u4Ml8hMOHx5QXMjc00BxA0BZz0e2idwf17xwCAY9ugiuaERntcNj2eWB3kCKFzYzdMugB5zvarHloF3FO3eHJ1AM+ERWVx1M1w/94RTmyDk77BlfkKR90MqyDM2l7VwwWFp1d7wtjF+66vJRdoFZm9eWUx1y6H+08mZIn5s15jr+rFvzKeS8zOsq+xqCyuLRdoKodGe9igcNzV2KstbqxnOGh6nHQNjPbin9j0ODzeR208Ls3XuLZc4OK8xdpWWPUVLs5bzI3FEydSposmZilqrFGiVU4mU4frmZhNGo8+Mit7tTCrihjPrObwXuEVV68BAN5z4xIq7bHf9Nl8bdVtmk0dzLrYvwqKgFVXw8QARoumR2sNFAH3XzhG5wyWfZWZyf1Zh5O2waLpseokLYVl2QxpFXB1f4mZsbi2EpG6D4R1X+Fg3qJ3GpcXaxyuZ9AqYG48rFeZsddK+n3SNtAq4NLeGkfrKLgwDi9+wSEOuxmO2wa1caj0YFaWNO5NZbOvpiKGMQ7Oa3m3E2N/3gnzXAeYuEbWfYX9eQejPZZdjXsvnshczsQcPCTTOx1wMO+w7GocLDp0TkMTo+2rbCJV/f/svUuPLcmWJvTZyx97R5w4JzPvo7qquEgNg2rUVY2E+Ac8BBItJvyQZsSMETMYNWMahBgitdRQA34DkxZILSRA/eC+MvM8ImLv7Q97MFhrmS332HFO5MlblTdL16RzYm93c3u5uW/7bH3rWwyEBXRPS8DtYcZ5Dri7uWBaPRILX4VA/oilAGNPfn9L9BWcfvXmAdMS6kaA9wljn9CHiPvzUMGm95kBfGZfTwLBhxvyzbzMHawtGPsV0xJgbUHfLZUOR3FKPX7+kw9wpuA3729hGaQN44JpCgzeCfTe3ZCf6P15qPQua5vYl2dK3OvbC94/jPjpV/f4cBqRkqnl/vSr+8ZWYFrg43kg2lk2ePWKaLmnS4eRfUzX1dXn4o++/ICv72+QosV4WOqGkTEFt8OMtw9H5GzQjZH8Yl9dME0Bh1cz5jk01cdiEA4rrC2YZ4/jccKyenQhYpoDumNsLAwGXOPdjOASTpe++o1SHsAPuW5epEjiZp1POE8dbTxEh+PtCRIzduB2CTB3Y67g3diMn/z8He4vA5aFfMplntw/kP9tbZcpuP3qAR8eDggh4Wac8f7+gFdfPeDhcawiauMbMi96DvUxLYEFzTKzNxz6Ya3jbUxB30VcLh18SE82N2K0OI5L3RgyAGYWXJO5IX6/cq2IevXDWimiKVkchhmnSw9rC7E8fKobH/2w4nzq8fqPPiAmh4V9mmVDRzaH4uowjAvm2VfGTB8i5tVT+KZLh66LuBln3J8GeE/0U+8yznPgTTWDefZwjtrRDzRmstHT91Tu6dzDSxsTP4cuY7p05Gu8OjifscwiZEabPDAFcfYIh5XGMjmUDFif0TFDZZ4C+mHF5dyhGxcsc6jMmONxwocPB3xu8kPEfKZ3iuVYtpW9cwnkS5wMbE8snlIM0ToBZtoYZI5BTLROi5wM3EDIMkUC/wYg9gv/bslGSKWKRoPuZkacOsAQGyb0ETk62mCx8m7hTQgQgycunjaJErFcHDM60sqxiZkJYXym755M7TlaOI7zKywmY1FF+FAMEjNX/BBrny0zl8pq4YaEdBGUi4pWLIvTSWzkUqBYMahMFGNATB3elKFySmN6iHibAlzUOUZAmuYqYKXIBophH0XT2CCSCraCbXJMqhGKKW8IiYhObV8yleFBINVsgZiuypQmjsYbBrq+jUiP9Cejtl0LPj2hwoo4k65Tvcs3tF/e4CTBsG0bK0NENq/25e3zVzB/Jb++5hkQubfCXvu+78J3Aq9/zakASH8Alr+fKReDU+yr4I1YzS4xYIoe59hV/6hLDBjYn8/kUj8LEJqiR+88gQkGJp2jOICX1aP3HnOkmIeBgeXZEjAQqyEA8i2MvvoECYCpVE225M3JEd2Ww5/M0SM7QyI5xcAbt/FlsqZgzbaCFilvtpkCBJvm7xNsrn5Hc6J202eqM7Av3pIdZqbzpkwWvcgLGPGT2ftASX6ArGBkYbRYeBwXttxZLmu2RClucdIcC9UQCJgZcOVCMdRicrRgBypwXbnd8rmzie+b+O0YBpYG8ATQcmpliPVLLJqdT1h54RajQ0qtfyKcsuZUY7ZpvxlrS12ArwzyY7LVWpgy9UH8rVIxiBwHb00OXU41T5QFtFgBeZxTsYgMjlIhmjPldzUGn7UZLhfE5CqQKoXiz62rAwL7cvH9FKtmyhYxOqZYg+9Ji/MnVG+hQtPcsFUUJrnmk0bjbhCjQ2Z/p5QsEls2s7W132ShbG1KjsGELeTbU5rFkdpEf1Mif7R1dUidqaAyJwtrbQUkzlnY0gRyIPdDAQdj7KZ/AFkUyQJKCzwqj+613DdazBtEZ+tnuW9Sd1ILDjpOY+ccKU8bACUX2rkvtCMuFl1+m5E/lvKvogWpqz5k5L9U6rsvZZp3MTqyUnO9Jdsm8JUcUuI4lqpd0i/xAcu2WaMNn0Np/l9EM6e+lKxCEBUZb24/X1N4bNu1W/9Asd5K+ZmfoZwLj0HiGIAc4zLTgqldq9vD85PXmlKmgW2bcLw4ysXQuovbJGNZY8yx31kuzWpe6xSKnlrRNGu3Qc7YuDwUZRXJqs8l22rlLcpvTocakTqeLD2KQUGLm7itS9rWLNNiKZe+WQO2XGPj36Y3K6UeekfQb6Fltge4XtmgEEaCY+ZAzafSPpyM42uoLPDnxm6o10s+NT56tUjA2G3O6XYJc0HeKQIARdzrc1NtkwASLl/el3Se3hG59sfUNpdikbhdqYD71Hwik4BKBqu1tRUsFWkIvMuYTbt/UqcRACpjaNp5Yg9kmOKYBSCmNbupw1imvAuDgO9vtoava2Mq990YoKCVT8VmGMuWQNUWqQM8B4sBUipNwV2DQMt59Q9XBW5o7iQCFOWcBpYCuGo5dA0pJBdWSC71OW/jDAJZwkzQ94OBUEFpbYUCb6aB3lJa3ytwFeCoASC4PNDvRC2rtPwFpf01gEHrX3V54N8ZQIHR6tqCp0nal0sD/abVVdtb+15QYLYgXafdS0UAagWF10yraug3h3dg8QkWVt+xy/OH9NefftTiPV/92VflP/xHf7+Kg+wDjEtcx71/47U4lvLjq2Mr6msk7eNQXkvPxZi7ds0+LuO1eHz7DaBrd+y5+I7XAiE/F1vuSb3yw3nlGh1Aex/7cH9MjktMxH35+vvnxMW82nb+q99t1959ut6XtEHnvd6HtpB6aRzAl/RXt+ulY6TbotvzXdLH6vne5V8r+8kvytP27Mf/2XiMUuSV42Z3/kVN3/yyXalP3W/5/tFYh5+s73Mb+rI5Vet4Sdl6RbDfCv6Oz+u1+JUvil35ibnR8u8+X3t5PkFS2PZR6pPF7cfq2I/dS+6RvuZ3NbYfm1cfewECV+dB9aHVYFm+f/SFqs5dy/OxB/NT6Zm5UIdt34/n6rf4/PRcf3Zp/6owBVfHuX7Pn7j/V+b/c5abJ+3b/xA+J8iyv27X1CfT61P3eNPYp8efvW9X6tlbpdq568Dk2fQd799H871gzn4ny9lf0XL82bF7Qd3PzingBb8bL2rex+v6WPqM8fo//+t/8IML3Py9v+jK//aXP/mdl/vTP/7lD9434EdusfQ242f9A1lxiqtAEiBVyZMoa4JCfczZVQVRUbr0NiFmV/+KT6ZYj8SvccmN4ikgZkke1uRqKQWAXCw6R8qfFqRQmsv2V0zAq7cJ59htytXAV/tUilrlHnhKflK3ZX86tijNrLbpDKlZekO0nVWpfAab6uJc8ug2yljoMdeKkMDzi3tR9NOqlNp6a9h6pJVRxa8wsdUlsPiS476v7N8pPqmbdqBZBgBUy+n+3aeBt1BnL2tALqg+iXWjgduj+7YmW63G0k+x+OmxEL9JZ8vGl1JA9v549QPkdkkfAFQrqJwTS47s0Ov2ik+n0INlToiFVKjcekwc15PYUiHjWX0sZXzZB1MUCrX/pVb9c26bp449W31F7AiQXfztBo8ADLEMSrniX0kUQLMBcdU/lI9JTFqxBGhLmezcS516zui2GnWf9/XtNxByUn6NIuaUmxVW5yuFlRP3C0X1HJVMiooiiqXP5Y21k+cpKzOC75MBmpgWsFmFpmgrrbCwZVB+1SWmrZQrz6VR80EnPXZZCX1pS0etn8tHQY39qs/Ld/GnFMueWIXq9cCmnnZy0zTklWLi1knNFkmJ3UsF0XXiFwpRnXR5Qw3UdW/ElupJHkOtPClgQlslAEU1A8U61I03peUvO3CrAejeeiLWiIxGDdRAwpUWh1VR3TZtVwCncNurNYavKwXksykxR4EtNY/FiaRuofWJEqURgTWnqHzS/f1YST55TuquBEg1h8e58D1pdfHwOxaSUq6x5Cv3+St4k82G8qh/YHQ9mzAScivEX9HgqY8cDfbGj48uUn3er7zZ2tbGxDRrkDyXQD1WrUYyPzcd24FgObYDoRuQKf2SZ0g9UjXJHJRxUvd3D1jlvkF/V3mNGucKNq8A7n0/Nns3ul11gK78fUnaj+HuuqubC58qT9rwTJl/nem5/a4XpRde8+I6Pvce7er6Q/qrTz9qYLlmh7fLEb+6vELKFnf9BbkYfDsdcVo6/PT4iEsMOIYF76cRX42P+O2Zwkl8MZzxsPYs077g2/MRr4YJp6WDsxm33VzDFfzL96/x+nDBIlRMR+xo8W/89ftbjB3Fm7zpFvyLD6/x5kA+TvfTUCXuAV74swjNvHr80at7pGxxv/ToXUJwCRYUkuDryxEL02zHsBIVdgmV4gOQv6A1BTfdjN883iIXOnZaAn5x9w7v5gPeTiN+dvOIt5cDlujwapirT9X/9+EOQ4hI2eDQrfhm6lHQAOSxX6qkPwDyl/SpSuBLeaKoOythnMsSMHYrvnk4YujWKmO/RI+xW3GeO9yOE05zV33tDgPFRPQ+4c3hgm8ej7g7XPAw9Viiw+vjBXf9hF/ev6p9kNAozhQEnyp1CwA+nEeScvct7MHNMOO8BHib8e5yQE4Wf/vnX8Oagv/3my/hfaI87Dv5cD7URbO1Ba8OEwDgNJPfl6jw5mxw6FdMq4cB8OZwwZwcHqcewSVcVo9Dv9T4i6epU1RY8tG6GyeMYcVvH27qfFkWj1fHCXN0eD1O+PbxAGsLhhAxR4fIisFCW3s4DQgh4TgseDgPKAXouohffPEOby8HfDiNCCHWMAAxWcwr0ZW9y0SNZTAn38UXa+wXzGtA5yNRipPF6dLjMCzwXcbDecDrmzPePx7QcbiClC35DvqEoVtxnnqM/VL9FadLt1EdBsTH0uAyBdwcZjyee9weJ1zmrioMd/2KZXFAIX8z7yJisnQMwBevzjjNHZVvCnxI6ELE2K1493CotEQfUg0bMM+hAt+hJ9/oy4V8mPo+Yp4omFzXpVp36CKWOeCnX9wj2Ix/9fWb+oyGLlI8TAbn1mXc3lwQXMa7h+bnJQJG6+oqQLy7veDt+yO+fPOI9w8jkqKUf/n6EZeFhLVidLjpF9w/jhUEHg7k33p6HDY+f/Jc/OnP3uGX394hRYeuj+hYvdragi+OZ/z67SuUQuEMpnOHw82My7kn/7spbDYAWjzPgMNxxrJ49IcV53OP8bBUYaXENPPxsJBP6GmAH5rKdOGxjNGRaNPq0I/03ng8DTBMcT/cTvR+OXc43k6YLl0F8o791lIiX60//em3eHceeY5ljAPFfX37/ohupPsbee6//vIRb98dEQ4rXh0nfPvuBl9+eY9394cKho93M80lvr+Xuas+lkKtP0jc1NmzP27E6Uz+pOL/mZNBN0Ssq8Pd7bnSiAHysVwXX0MwjOOy9VM15Fd55Huc+d6+Ok748DjC+4R56uADCT7F1WE8zHj8MOKrP/qAmCwuc1c3ayyHAJE4r4fDTH61x4me+RBxWT1icjidBgzjgteHC759OCKEiCFEBJdwfxmqcNh5ovKXOWDkdk6XDs4nHEf6/uHhgK6L/A4loSpnM87nHv1Az3YIiZ9fEmbyLGK1Th6H25ko8skiRwvfRRzHBcYUPJ57jMOKx8cBNzcTLuK3GR2+eHXCb7959cJVxtM0HBZcHmnT2nUJljdhvM84P/ZwHC92PCxYFoqPW/2Lb2ZSPF88un5t1P5oMd7MyNlgXXylp4oImsxTAJUunKPF69cnPJ4GwJCoVX9YsC6+zn8D0HPDglfduGKeAjyPRUkGviMl+GUKvJFCGzHO56rWnrNFjga+j5XKbRzRjdPiEfpIc3siF6DusJB/Kb/b4uqRZodwWLCeOwaYhdXZAd/xZujimk/n6qpl1bjCPpbkS4rZVbVvOQeA1M4lbAjQANpqmo+k0DxlE0aORUMiWwLW9f55RlM/l6TBrIAd2VApqg2uNNVyvXmjr9NJkJbeaNmfz2a7OSUbR3JcXy+gPtOGEFzb5NkWa6p/Z7FK9VvGR9oL1PNIqnx1/iow1lV+DDirza2Wx7Sxle/783rs9GbDHt3/HqQCIP2I2aKfSj9qKuxP/s6X5T/6R3+/xkwUgRnP8fjOHNeNLJG5xuYT1VeJ/bckh4EFaja+aKz+2ruIOXk4sw03srJ/ZGdTpc7GbGt8wVIofuLe6lfY0iSWMgB1oRLZbyqztU4rrwpw0NYpsTwltq4C7V0iwjYSUsWzAmpiq58xpQr36H7pz0msGUD1pRFrmKRcAGdLVWIVGqxY5iRem+SV4wJatOVOjpdi2MKWazwzY0r1M5QFzJ4eqtVRZVylH9SmsrHKObVABICBNwh0zD8d77EA9YdeLFGb8WKrIABlYSs8flDWvG2cMXkMJQaZ923zwNqCGC20yIVYCKUumlfcJ7YWSl6ALGeRfUa9z9UPTdq1p243X692nnzC2ku7WjMZFJJlKVfL4ZM8ZWvxFJVSqyxt4nNWfw95EW1d2oSMEUtbsya2H265TsLKSD0FoDZlEpTQ9xTSVz2eomgqi5ZrFktTWMmURCwAVCELugZVDVWuK+y/KAsosgqY5n8kY5EoNE1aKDSN/vHOPB+oQmpDDV9jCgrPUSMWy90Pc1kthcExqj0imhHlnLI8JqX2asvmB7zWxfOgDqpe+PC9oZtlms9SvdHyAJitL1TatV0vyuS8thrqcwW0oNOLRPGFlRA3QGufLCxZ5ZRC3XD5Zle+Ulx9kvS5oixc+2vYwmii2dzbIu2Vqb6z+EmqAiDSh4Snaq9yTsQ9dFgeGU9ZCPNiW6yRVbFVLU7luOGg7xu1zn2oCF4US6D44sBWRf7OIUnqVMot3yZ0ghpjXb4olOp7o+syCS30jbTNgEOjfP66R4u7iPVVjLpZ15PVmlluhbRfr4u5CLObN5K/Lh32Vm+DpgIr35WCby1PjaEovV5TL92ExEDLczWEhiS+QOZadhQWaDM3pX6jwgsBjRqs26Defxp81NAk8r606vryFDvscdPGYinXfSLty9jci+fyS3+g6tDHPpWugayyazuu4yqZd1fr02P6TN+vWvP0O+J3Ud5zZejvz6Q9ZvxcK+o//Yc/PBX2L/6iK3/5v371Oy/3j//kVz9434AfucUSQA3STsHd2y9QLE29VeiqQlUDgMHFCga9JbqpppkaUxD4V4oEH9oslsWsBBU3pjQ/cj5Xw27oOkqrowZQZ7An1jQBh/u69HFZ5GvAY5SVTsrvfKzPrHep0h0FpOmg7XJcUv2sqJaaAqfzCk1JVEaNAhRefoDrmMl3Hhe+xkJEIVBppLXvLlXAKtZQANUqtGkXjw1ME7jQYhMArb32KUgwcFPo99s2L1u7f4NJP9HehbV8tlzUsbnymdrdwKmAJwBsOZN2t+uaMETrU/vcQGVtoqM6dDkSgJtEIbDxcWvAUs2vHc3WmNLobKYF7raWYn+JwI8xuZ430AIcRoULyarORjktJtd2UD2ANYkAdClb2mO9/5RHJ2MIPIq4hNB5UQyKMU2wAjxf+ZqN/6ZrYyN0zApkeXPCmNKk/DkkhjWluTBJHDugUcb4W+1LBZnbX0tjadVtfd7QMVEMbHj6njCkSsN18NhU5cWybUjYlmlMC/IuYhECdKnPmdub6zGZK3JM2iFtJKCi2q1BkIQs2a8QBGTKcSPgvM1PyWOMLHjVM7CnOJayBZ4CqCyegiuPBiLlr0cT+9B5tQCGtFfO78BmBYr6uGl5i1wvZemxwm4xq+fnHtTuRDk2dE8+VzrhrmK7kJP5YdDEQq6AZulLcdx2XkwXLrMAbXxkKijgg6LeVVZdC3Vc7qX6i2t5du0zheJw1vN+2596qVNt+oxU1EZF3ZiX6WxbmzWIrG1UAEr3BcCWsl4r0xeX7TH+DdjQfOU+68dKjWENRfKcj6Vul5qKewzQ2ifPRnuWisFGZKFObwOiv1/p0ieBSnk6LATkzHNDc62I2uSrJ1RbjZqXH6Ve7o5de5Q/mV6a7xPpGvC6CkA3zz2u9u2T4O0F9+sl7f1O6WP34YXps8DoH9J3Tj9qYNnbiD87/gq/WV8R/cstmHLAJQVcUocvuhMuKSCYhMfU45Wf8H4l+tkrf8Fj6hGzw+hWvF9HvA4XvF9H9DaidxGjXTDngG+WI279jDk7ZAaxAMXRdKbgm/mIwa3IxWJ0Kz6sA+4C0SUfYo/Oxk27nSm4pIAlObzpiDI7Z/LXdKbAm4zeRrxdDsggK9zgVizZV/VVAHw8wtuE3ia8W0YABJqn5PGmu+CSQv38YR2wJIqzKVbY+3WosTsHv+IcO1alJcB+8MsGcE8pwJuEjmmoMTsMfsUUAzoXq98pACzZw5uMSwwILm1ihA4uMk15xjl26FzCJYYaViXYhJsw42EZMPoVSybF22OY4W3Gh3mscS91uBHxQ5WNgIeVKEuyCQBQ6JaZaYVTDIjZ4k9u3wMAfn0iiu0+j+ONAQA4BhoTUf6VNuh6LAqOYcaSSSFYxlNiisqYaT9TYwpuuxneJHzgeykqsMew1Nimj0tP1mabKOQO1y99P68BzpQ6lgCFj/lyOOFx7fGwUMgYbSXXFnHxb3Xqu/R9DGu1hEv9EotU1JVvwoLHtUNQ1uKV+zio2KUr06ujso53HDNUNmIWjoN6WQOO3YLzGqrFvGMlR0s0OQAAIABJREFU2lxaPFLxwQWA237GHH1VbfYuIVgKj3BaOki80+ByHQt5vqRMAHUeBJcqNV1il4pf7sI0ZWsKvj0dKsDqfMK0hLoZYA2NoYTjqe8E7q9Y8GWsH6Yet8OM0xKqhR4AbvqFFG+5/8ElnJmaXQrFQTWm4Dx36EOsDAC5/m6c8OEy8DimTSzWY7fUMD3eZUwrUdfn1dfQE5IKGitgTQ5DIDqydxS6hdgKzXcYAHqmq1+WsPEHlrKE3i5hVJwtlV4ubAVrgGn16DjEkYyNqFPKdwpV5GtM2c6nOi7a/xhApal7R7FeT1OH40BhgcRHd+jWuukiqtTCVhDGgoz3yuMZHNFMJW6rKPyGQDRgKVM2cFZWehbfY6HxA83PV2LdyjsiZYs+rJhXmmtCj5drgk+Y5oCbw1TD/+hwI1otWcL4SOgfmRsFwDwHhJDQhxZ6x1umka/tPRmr6rZF11E560pUa9nEk7i60kbHcWGFDl5VuNem/Kopx13HoWxYcdhxzFqpy/uMZfE1rItQ3Puewr98bhJqJylC57papbZyGBduT93MZkaEYzp0Tg7WEbNBVI+9nMtNzdgo5scTobjM4UNWj2Iaa4HUlLFBgRL6Q8KMGAbA5MdNqC3HbTxb40qlvBZmGRjxTRbwbErzTS7EpAB4k037UifTYtuuakeAaZvat7mimowtSpP+ZPOU7iqIQdgQewQh5ehx2W0ISSiOTaiQej2qpVeLWW0srkaVrcG8UewBfd2+PZIEPAlwrhuQu/MaZEm+SoXd9Zv7avSGwj7pMvXf5zYhytNh/mh6SV6NiD+S9ln2+5m/76mg/I0ON/KjpsJ+9WdflX//v/tPYU2uoAsALTJRcI704xGLRWdJUEcvxvUPqvgHykJaL9hHT6FJBLAIRTbyInb0a6W7SlmXlWi4Eg8SaKIx4mfp2NcSAILNdeElVNXeN0AqdFVntuI9EkJjTRaD8mkzptBixhamwjp49sNb2V/H8EJLgI0WexGrp8Q3lAWr0FazWkwJVXUv4iMLF+8yUm5iMs6WDc3VC81x932Jjv343IbSW3gxL5RW/R7UAi1As4hW6yVQRWw0Jfk80eJxYH+0VPM0yqtYFluYCba28oIFwIYKK/RVff1TKuxWkTZFWxclco1zmWPikWiLiL7UWGKm0VZRDJxvsfwcL7RyNhzbr1R/0MKLeU0v3fNt9pbqSj8GmgS9bYsZawtyosVK4TJknsgCqor3yKLCbMdXKJh6bCWuoSyCDI+XtLUuYsRCB6KmGvHj4Tx6gSRUVpQWZ+4JFRZo1/MiS8oSi54czzM9K7ZP9UdUL7qqX2K01NaguHLKQlevjRQfLs+u0lbpBGhxJuszUyhOWmhWz5K4Dk/01Y1VDyBfoY7GoEi8OOl7NEDgVUPmheVqiT4a7VNro9BLLdqCTyiqepEnViyhWDpVBsCLGLWwkHKENit9FQEaV65QYbGpyyyWArKrxaopBiVoKizPw8jHi6FFps8wq20B3UGLT7kHKCBKpFpwFltaHrlfQskUSqtYhGUhu2q1FlTxm7qAFWO8LFzlhadj6pld/D2hO/J11J8CsxhoKmEd/qzqZeqo7Ifq2HfZ83Gm90pwesNU27ra48VtsXS9kbEqbQyLL0/vgVB2RWRGXSfzQ46Z1VT6JgAYTbO1aPTctF2v2gjkz8eVVJ5YPNVjWftb2rmNlR6AiWjjry2aBpXGu1n4qzLKbpoAgF0Nsi8V+GhxIGmT4XZR23X8w8IUShrTLFZ6edQ4T41jyHPsSfv0XFOUW31filXtU/56dQ5qXz0NXjhfBTHSF7M99jnpKgBRi4kn5z8GyvT1+zr0+O8A0FUgJG34SJmfskw+16+a77m8ejG178fH0ne8By8CfzrPlTZ9bjv+6X/7w1Nh//wvQvlf/gqosP/an/z6B+8b8CO3WK7FYUoevzqJeM+EWCwe5h7nucNXNyec14DbfsYvz6/w5nDBr+7Jaf+LIwl7LNHh0K34V+/vavB2bzOO/YJX/YRLDPh/vv4St4e57k6LyuktB4D/F2/fVKGSQ7/g67evcXckS+S3jwf0IVYfRID8ET+wUMLPXj8gFYMPl6FaDrzNuO1nfPN4rFadsVsxR4eFRR0kjf0CZwteDTO+frhBzoZ32QP+5PUHvJtGPFx6/OT2hHfnEWt0OA4LurAiZYu398capLwPEQ/nngCXyzV4d/CpxlW8Pw3oulh9HJdIgjTnuavWAwEql0uPvot4d08iDYEFItbVYRhWnB5HjOOCx/NAQg+LR9+vOJ0GWJdwdzPhw8OIGxFtWR1uby4YuxVfv79B16W6Cy473t6nuoNeisHjuWfrQKqxBsdhxcKCHZfTgBwtfvHH3wIACa+4hOO4VKGky7ln0ENg6HiYKxgFGoAr2XBwcdrJ/+LVGfPqcZkD7547jAOJmgzjUgVFcqK2iLDLGCK+fThS2clgnUk8ZV0d3tyd8P7+AGML+mEl8QpWDXWuwNqEy6mH8xmHw4zTaQBAAit//OVbvD+PeHgcEboI3y9cB4lJGDTBBgH6zieKR8ggfhhJkKILqYL+6dKhHxZ4l3E697i7O+P+cUTHQjwxWhojlzHczFXUZZ4CjEuIs+ddcRLgAIDQ8+bC5DEeF5wf+41QS04kJJFW8mXuhrUGhheQePf6jMscsFwCYKhvYSAxmPuHkXbsI/kxhj6xAI2vvoLDkcZnOpO4UBgi1plFjligo2RTg45/9bN7OJvxm9/cVQDqhxXrJRCItUR/vfniBGcKHh7H+hxbVmdOq62CFjdfnvDw9ojXP3nE/f1IfpX87L/+ySPm1ZMlkoPan+6H6gc53JLIzOV+QBjJbzgvjnw1Afz05+/x229eIa8Ofow1IL2xBW9uz/j621uyWI4r1lOH/m7Ccu4QXs1YZ9+UQ7OBO5A4WZw9+tsZ6+JJtOgc4PvYYl+uFsYC3e0M7zMup44sKMWQpSMb2DEi8wZCmh3Cqxldl3B+7GFsQV4t+lfM8jh16O+WJjpiCwVaT6aqwf7sT9/i3cOB2mxo/nqXcf/uANfzpsviAFtwd3fG+29v4IYVt8cJ798ecffzB3z4cKjWlP6W3vlidbuc+7rhYW1GXByG40KbfYsHTME4Ljg9DGyRorlL474iRYvj7dSsTACWOSDOJGJSisFwWOpmVlod+fMmg8PNXC1c8m68fxxpI2rycIEZA9FiOCw4vx/x5hf39ZmVZ926jBASlsVjnTyGmxnz1OFwnGD4N+08d0jZ4vzQIxxWvL4549v3NwhdRB9IyOv+PKAPkWJJX7rajp6fo/kSYF3BzZHYPB8+HBB6sqzHlcRsvE84PQz0rK30Xl3OXZtrfM/SxWN4NSOurlraXJ+qUJC8Y08fRhzvLricexKsWR3evH7Eb397d3VN8ZLUHxdMDz1QADekSqv3IeFyP8ANkebpgd7PMvcB4PBqQowOcfbohhVZxHtWi/GWxHvi6uvmmVe/W3HxdTMLoI2z11+c6D1iCvLiMNzMWOaAHNsmmHMZcaalXndYsF7ouUzRoUQLP6xkKb4E2kDijSPXZeTVbp6pMJLgkGymSdm+Z1bE5IECdDcs3lOoD3FxyLODP0TEU+CXXqkWUN/z+29u4j1ltW3jIeQax7KsljbGZMPHlWbxjGYr0CMATfqV1AaeiNzwMRMtbSZFS5sZ+w2vkJsID7ABwHXzRQvnyMalL3SdBozVf/WKddSgAvXqn70H3BrQ640j2VyqYLxtZplsaNNQ+4zrVHZl1w0sPBHvqXVnsx1nXc6+7GsmRqjjCiHXzQyVvcjYqqw/2lRI1PpvavpRWyz/zb87lv/mH/8beEgjElpQ8LV4JFg4ZKyFgM6cQw39sRaH3q71mEPBVDyCSVhLo7vK9zWTcI9OUsaaffWjTMUimISZy7IoWHfOHM5kpGKRYWp+AJu4mULjJHpsuz8bSyU/6Y7fQNa0uhxy7X8CU+VMwpyZxqesklJmgq393bSXy9+Pp26P9hvVx2UcY3E1Nqj2VaUwMKmeX3dOM8FkzNkj2FRDpEhIFR0ypY6bhEpRdOUl+02dur0Aap09lxt3bUjF1PLkrw7N4UwL36Lrz4WEmwBUazdZqlMNdZOvvN0l3EtkOq1QoffzQH/XoWj23yU0jDWlhq4Ruuf+fkk+fc2188/FW6V2O3Q2ImZX768cB7CZB/uYsVKu3C/d932d+/mmab3ad1iH8JE81pQN5XXf1/3xa/FdtWVePgemzq7J1T7tY+LKsSo2tcun+6XDBAkjovpwq7BAclyHAZJ8exaCJG9z3bTSFmm5divOdf0eXEvSDqG0CqVZ1y9zT8IIPVdeQQuDI6JcmZ85+fxce+TeaLqrMFTEn10Lfkm79GcZOz0WWixsXxe1E9jH8t3H8N3Pm/34l7JlCmgfbJ32bdBzcj8m10IVberU5aIxPOS7nNdMlSfUTNWG6npYrvf7uXPXklYg1lZ3+bwXGbt2TStrW8ZnJ70Y1oevCMrVc9ga4TZGGMmvy5bydRl7DPKSvnzkXlzLd72M/b2+fi9qXt1u+aw7va9rb5GSa6+1aQ9Snm3zy46Zj9V17fiT7y9DOC+20n3MOvcd5+2zdX6He/3Z5bzg/Iux4ec+r1eu+7//i//8B7fq/d0/D+Uf/xVYLP/2n/7BYvm9030a8Zfv/gIAAYRLCtVHMReDSwo1rzUFS3aVCrsk1wR2CvlT6cVmLk0VtrOxAhS9EBQVWikTwJPF1H4xDLQXvGVQsl9obhZgapEN0GJT8kl7Iiu3iiVVFtEz+xWJ/5peZGnxIK0AqxfY2r9Onk9ZCOrNMwFYQteV69sCuVxdGOp8+79Sr7RB+hF5MaxFfKDKkvGVRZ2zT2N/inotnad7tUZSmxX/KL0Q1PEnhQYMYLPwk3r3Y6gXE7K40+FR9m2v6rUiTIMt2BPLLH2+vsirAjO786IKKxZv8deqFFalEquTXrhsF488j22pSrli6W6xG82mDTqO5Z4KLPl1nfs4lltaclNpzVnHpwT7MbEVSwm/ZFZItS4/WWjJ9Zs2lOZzpGMmVl8goC6kMs9NoirzqWybUJB+DxRTrerXQKqACesSWaTt9j5v5j4v3PbjuO+PXrjmaGCZIlp9puTazOd4DCSepLGoMe1qrElDx0TcSCjCVU32yrwUldqqWLsDBJvr2VJnFB2yUpRt2dwTPcYyJtoCQ4NPfTVeKdjKuch032LIiuHZQqGosNXiodVn92mvNpuhaMHqnsn3tJtLWixoU56qy6jjtV5s6cX7hb0pW9ryfqFvsLG+VL8wTcd0yvIhFMxrip5KFbZSeeWY0FlFgXffH/EVk/7qbioLiUnKqmGoXRsqKNdf6ZdF+ocm8vMZSaiwhcvi6kHU6G09zc+OL06qjzXeKd/vyO/j3aMtqajs8t1qaimw9fWrQA+NGq6mgpwy/KGOG3+vzSjb4/q70Fw3asDY9n1Dw8zY0kJVG01pZe2xjZF3cW1nqd+/M+iQaXXtuv0zsWvDM/sJT8r+Pmm3l/A8Vfba3+9Qx9X0mf36XY7HiwD49yj/D+mvPv2ogSUAjG7BiUV4bvyCNTvEYjEnj97FuoCfk68COLkY3IQZEwNPAZWdS5giWTE7F3FgC885diw8Y5A5ZMmUAm7CjFwMC9pQXglb0ikLmOzQa7AVGTiNfq3frVJ2lToEaATb4jDqFGx6YhkTAZnXA8XeTMXiGBbMyTN4Wur1IkwDAN7HClLFZ1L7eQJkjRElWwE7I1sn+ivAUPLtLWpBrmERoGAzVvXXWxJUmZPDIazkz1oMjh3FDp1YUEW/U2Uya2AnPqgCugFg8LmKyYgYy+vDBdYUPMw9vEvofdn4vDoBogYYePyE9ttCk5Qaf1HmVSqmiqKUYjCwEEZvc/V1BciCZA3QeY75tvoKZEsxOPYLhZRhIRBjChzHH9UAyRpU/1XLFGYBdnfjhCW5ekyXX4qhRZgAQL53e6uO+L8aU6q/bUwOfeB5nBzGw4olNss2gDqWWpxlv+Gi752eK0KvGzqaBwLsvfiaFoMuNKucjGk3LDRvVHiYai1LIgjSgLS1mUPEtO8GqDRwDb7lswaG/ciUPyVuY22LQwfeQBJRlDU2q/t+M4CePaLSDTcTVhYLoby673SN90QLl+TFz5b7I/69Uk9/jJgXotLaUDZCLt4TLZJEagqFrelow8B1pVLzah8DgXkJb1NFV6KFc6VZlWTsXK4bBnv/YAl5QoDVwHSpblYABNTdwHRIpoBrtW8ROxHqduC4gVnmvBKIsSovQBTdZSb6u3UZcfXwxwUxurrSs6zUWzc05Bk2ze9XNoWyukb8m6kPW2AsftB14yHbFpS+YOOvrBU7NyFzuF55NnIS/2sC/sYRjVjiBepNpRo2h+m+1rGftG8WftkMEYql+FLDoPZLKMwlo24YFFVO9fVVlEb5LD6+xpRG+eZxlXA2BJC4z8kAIhDDmwXFgmIwcl3GkVCM8bQxooViio5H+F2TK9XPtyrogsawUi4FzIkAjdwqAePiG8zzHhltA0NvGGiUoUE4j0fyarOgoNE9NXrUoF3TQfcoSTaY9psOO4BYP9ey0aiS4mssfVN5hNZpdu+PTZ/y7vO+jdwXEqHh52kPwjTY0CBRgK3ZnquCNtq/e49uZMMnm+tZ9Njsvgug3rRJPd9Xx7Q8+bhJ+0t1lfvjVzeQrqVnQNpHgd5LgN0L8nwnMAk83Xn40SWD9HJ77Y8u/aiB5WhX/OvDN1iLq7TWc+orvXQtW0qaUEtzMUiwiJlUXOW4ACA5r1PMDt6mDc10LQ6X1KG3D5UqqemdUi/RXpulCmgUrzm1W+BtqjRKALDm3OpnaqRQKaWda7G13MDgVoNpq+J1aTqn1OPHbXk6WVOeUDIFQIvFtlE885PradxI6XMvXCTfNXVRzmursY4/SudpjG/D/IRqqdsp7d9bfKUO3UeLgonvw0+Pj1fz6HY9F+NSNgE+Rv3KMJVWrS3Pco+lnGGcNu3TY31gEHeNxrn/vgHZ2eIQVphOtUfl2Vutr5WpabIyz6xZFPBfkdSGxP53LBWDg9lSGYGnv3X6d0aP/SgL+F3brrEBlujR+XVjyZN7JEqcck7Av+2e9n3snqfD6s8C8m/GeXPMdts5IED1Zpyvxn6Vz6WYqrIa2J+7tk2xDBwD78OwbNqVBIwziJY+y30c+7XeEwOgOHpKUrbou9jGNdB4izppt4sBWLitIWznvYBfSRo8lwJ0XbwSi/UKhbCgKosCqYJ7P6wc83UbakaXJRb+FtbH1Lolj8DxlCxCFysQDl1ksJw2ZdJ5MBDOlf1QI7poEFpIJMt5pWbCFtSSyb9RYrfqcbJ+Ow8NsAG0dZzrgrsJXwGg6+X9w1ZMG8in8prFWZ5Aw4BR/Fmh5qOAyJINbS7wArkoIJN3PmjGlqoUKpsFFQxaZTUGgMh+pnK8GK5Ur7LbGJao6qptcRWAluRonKVN3OYS3XVL80tTVMpHGiQVHpPIx65NSw1o9xbn9cpCU8p8btE/WfY9Aw1VtChGgfV9fr7/NBD16OYP0q6+fd0aqOh21uIK7Lqd03LaRrPFis+BM3Wcbr+5Pgb8bHwS/KhyzZU+NUBm6t+n/Tab/K1tH6n3uWMaAX5iKj7bvz26/QjY0vsI8v1F7X3muFx/DYN/sryXnn+mzu+cvsej/of0eelHDSzfrgf8z7/8t3EbZizZ4X4moZJjWDD6FV9fjnCG/KmO3YL3lwG3LFjy4TKwqA7J3r85XPD+MmAU8YElYJoDvE/4+d0D3p4OCCyTv0QHZwtOU4dSDL64OeO8kPVzXj1eHy94fyJhjttxxmUJ1WdFFjt9INGDbz8cYQyqdLrIp6doMR6Xak1cWK5e/LgAWnAsq0dKpPh5ONLic1k8Qkg43w9wXULXr7icenTDihASCaAwte14nLCwRPoyB/TDWq0bImaiFywi3Z55IW4sO+93SSmp0ttG6IvDsGJZXA1w73zGupBYw3wJCD3J2/f9inkONf967tDfzFjOHWyX4H3CfOqAaBFuZ6TV0e660OayIVlzWbgYIIy8eNZqrbOrcuj9uJJAz69vAAD+i4ksHLMj0YJs4LtUF3cFQDrTY2NZcMD6zAHrS1PwLAbl7EhYoM9EJfQZZXYwQ6K/faIdUKYElmxgzp5UJI+p7h7bLiGfPJU1OZRjpP6xUqfxhUQLeMfZHkgABZODOURqy2LhPnjkMQM3kXbstXolWxXqTroFLZBEkZN/mUhlk4QSzGoAB5Q+AYslatqQYU8O+ZjaQsly+cnAzBZlyPS34+1jUei0pYojSND40mVaPI0J5uxQukL5XNkIOJjF0sLSl0oldA8OuS8oPfctUpvtYpBu+JgrdDzS2JWu1B14O1kyNvD1ZjF0Xj4Hqses9Nm/d2S0+CLW+Uf9ZVBRAJMM7MXCJCDdKlVYtiKU0JQX3aNFvEvw7x3STd5QB/2941iCFEfTztwnXqzYCy3M002Cndi60pW62O3eOZzeJBRfYCcLK/3JBv5scH5Nljm7GKRDhn90SGOGu1iknsdY5udM8yT3Ge5skTu6Lg8FdiYrRbGoSppuIspiGgvsSrRLia9nVhGhAHIA3MXARsqLTOfchcYnDfQ5d3S9yTRvqC5SLO3fWsRDQeoBkwA3k7Vjuc2wfM+zp2vDo8Fyl2FXGoPlVUH3wWC94dWo4fYUsAUGSH1BCVS2yUAJgDtT+3KgMfKzQTxQX2GordkBfjLIoSCcTMVQKEDuS20TAHSTqUqYmdVliyvwZ95MMUDxgL8AceR2dVwfaNzdBKy3Bf07Kit32FBEbaRxKAFwFyD1VB4A2JXOwQLxQGW5C322kf6ZRNfYSGWmjmiaqaP8ptB5k+k7CrDeAG6lz8VRPSZRH+yKqkybes5jOT+o3HCicSyexsCugJ8Y7/WAm6mN/sz9jVRmeCyY3zy/CP9U8hdgPfJcnvk+FervegTcQuPlLzxu4L8G8CdqQw50bXGoz4Y/FxRjkEMDXTZSjGCArhHKqNzXcF8QDzSncwD8pSAHs6GnmkTHqL0FqTOwa0FxlM8tvBHVGRRP96DW7QxMLDz3DNxcNqqsNgEpGCrDUBk0Ri2+r0nU/xwM3FSQBlPbJdZCu/D85mdJ+ifARZ4vmSs50NwQi+PmGtV3eb9Kv+o5vg6mHcuuPQc27cbatPGQpEFVbSfvwhRr6DPXQ30x9XyzZu6OyU+t3Z7fGJiZTmxTafmszBU8tcqqa3T/r6Vqjd1bVNsSoOaTevU4bdI1h2B1bg92n7Rhh5k/CoavbEj8PqcCtWH3NzD9qIElAKaieqzZ1Tm1ZAdEtk6wdUxi2c3JNfEVtVM8RwJQEkIEQI2rdV4JGEamPZHVQfy7CuboN2VNigq3sO8egM0ufMqWKY146mNlmP5U8MRSItROnYwBjN1S3cgvKZPqaLaVNlXzGMM+i7bS6bTfWta7nWwBKrpu9ZSL+lztCx+X90pKtr5BBFiDwStMG5fI1MB1pV8UwzQ3oTfR54JSaLe/vjT1cPBYyLm62y/jb1pgelFMzJbDDECNt9rRzsU0apfa/axjoe+vTAPD4EdAmRyvNKnSfvS0dcbnpv7BynYlc7kGLfSBwVbdzQL0q9soh1Whjj/ngcGJ9MGqsjbloF0vfeG5XsMrFAFxUhZ4p5zA0aYsVQ+Ffmh/qSDpfBs7rSFVxPrioH7RVJuBRklTfcmhtPALpdVfWUmmjUXNp9rdxprr12MjgcZV/tyrvsg5Xa6hdpZAgdyLmgNG/uM80n4ABJz0vZJjMka2IHc0/gYGxRQUL6uC1rdiCgzPLQKH3E9H5cFSe9OACs5LXRiXBpZ2Yy0AFwbUN8d/rfQTm/mWA2AsAxyZDzzXIfl53EsAPZ/yLNgGJOpnp9oBDsBuqfdxLMgBVVkxUxbqs9waS+XGkReAmUBrcQVplPL5+g5PFveF6zL8Ofelts8ASAImuYzC1yW+h2komwWWjFkdh6E958UBhU2jaZD7Sv2NIJANBrhZhfwAaAzTgDruokhpClA8bwA4KjCHxq7Jod1DAssEhHNPbc28CC8eSFn1Icscpk6XAGpbncNo7xJLC3uT6biALQG7up8o1KY4tHxQ80E2C2heA5HHzQQuw5jvFW4kMgDap03bZbz4GZVQIfHYjsvcgeH5zD9kFZAVICfTwAP7b9YFuQFw0/pC5ZkGrgrnz6aCtOIIuAr4pGdc3WfTrpO6LfuyFkfXS3tNoftdGDTKfaf+tnbb1O5HsUY9Q6a+/2xoz/QGtED60Nql54aMuwZ0Gyugek43YKkApsjzY2AKAficCDxmAYVyveFxuMagVr9FNaSQ4f4BW6pwXT+1r9WXWbWtmUQ5U2nnBXjuAaR8122WtJkzql/P9UWXdRXU7X/nrpS3iff5pD0GzybV13bBc3W0e6i//yH9sOlHDSxv/Yx/72f/DI+pV+qv5Eeo1SkBUljtbcQlBfazEhVJorieYl/Pe5sQTK5UMDkm9FE53rsIi4KH2FeVRm9TjZdpTa7+lwA2tNxYCMQOX3JIhtIohhalig1pMZ5cTO2P0Fm9JX9Hb3KN2yl02cGTT5r0USt1Sh8WRQ8WkSLtGzmw/6a0XUSPtOJmZ1OlakqScwCp5TrT/EfX7KrSa+8i5uQ3CphLpjqCTZiTr/TbXAyCpXAilxiqWuZGHXSn3rvU8Wp0Ri22tCYa45uffAtvMx6XHkAT9ZE8mr4rVuN9fFKpR4717LMq9FIRdtK+pKQqy+JEQI1vKhshNH4GgeuX64H2TteiSQZEczSmbKjX1hR0jsKD6PNyx8Q3UQs2iZ+t7qOIPWmlUf09JrcJO6N9bskvM9e4pzJ+WijJ7ShqKRsER/6o8hfAE8qksyJSpXwsfap+rOKDZg1qPi3C9FRN0sC7vBFrkjir1jSfRP0+6HyCMQXQ12D6AAAgAElEQVTT6uvvoL4H4O9Sn1BUddK/qaLcKuOpaa57dc9PqcIWYOO/KWUKFVX3P7hUfY+tzYjRwfvGSNCbTuKHSZ+b76n4UOpzMrZSn44Ve028qAk/YeOP6VzzH6WYsE9Xe7K+oDBDtm6YVVEsFkQC2iaa9wng8EXOZZTVUYgQ8V0tpvpo6nq0iJQwEIwq19gCk9qK2fBmlxGRGkeruDqfRdxIvnN/sdtULOJPyOeyopZuNtQAwGaU5BD53aXVS2Xcqx8lH8/SXtvousIO0e9TI9dLmUBtx0ZcSfZGZCMm7c7JYjtv73/dKJMVrkFjOMi48LhD3QdU0Skl8iQbluvTZ++lybjcaLjqfVVjwko9ldKrboSIKwnY0Ofku160XwFJdSyApz6Tz5mB5GdRRKRqOdiCg2toZG+q2pe/LwPqu87DfaxxTE2L1VrUs/Hkmtoe9f05f8hiPi3Ks0smFwZ4PLeuZdLXXxnf/bBc+/6kvJek58Z2d73B0y5ehW4fKeO5MXoWhOryXpJekPdZIPv8Fbv78RHA+nuW/uBj+XuarMkY7FpDdwBAMIl9Dx2FzzD0A9LbdQMoqyqqybAmw4VSfRipnKx8Ett1udhN/QBw9EtVorW8YBeAI9bRva+fLwbZWvQ2EtgrW5XQpyEy2C/Spt3x0saChYC8ycgMOEWwxJtchVNEWCjDVDETAT20v8vjY1uIBup782mUsiSvVUC85betHtUfbzOPNf213G5r6JjLubZZ8muADaAKFjnkDYCRdmqQu5fzt6bUnS0tFiObBXKNFtbRyak5pJP2QZV81hZEHmPHgNxwP7WfpgYMADaUZ88g71q/SjGwZesnWdV4bYbd3KNSy/WqPFmE1kW36q/d+c5aUzbng/SpDk6ieSz1gH+vuH7DeaTPGnRq/8hWXwOtjsdM+4ZqYEXXNrBlTEFwW4VmOQ5YOI4duZ0fDWAR+G7POZWf1GcCnuIXKffIq3Y6pa6pBZYAEkLSdcnnOpS2oBQCwwIkt4CrLcyMKZuXeS1P91vumxp748rmvSO/6AJGDZi5of4aZYXdt92aQvRYUwCXVXvb2NZxsAKy6VzrH+pfu3v2JDyWZT6qBovbfG2DwJgCa2VcCIg51TY5R6C1PYskPKR9QLc+jlJeUyrmxTLXofNulGu5b+3zdg4SANqOleSt34vZ1iHH6++NHowGVFu7W/lPFIb1X6CCyoJWX+EyIUBSjfd2fJ6uEJsIlQKV6u/TNmGbYdc+AE3sB2pFrwCpMajHN4rOn5OKubp+3YTbkHr2wGgP6vZ/96+ia2BSn8/qXOFKPta1DdDdIZYn5Zunx+sxPAUh19qPls8Ug6L4f0ZUkQtw3afxmfIYPBaIKuyVcbmWnr4mmlV3fy9wvdyPAcT9uSs/K59MHwWD+vOV8szu70frVo/5Bsh9rM0f6cOLHqdrGx7PlPWdH8/v8Tj/EKngD8Dy9zblYvE2HnGwCxKAb9cjrCm4cTPeuDN+Ob9Gb1c8ph6v/IS3yxGvwxkJFm+XA0a3orcRp9jjNkx4WIcq5vPNOmKKAZ2L+NPxHX413eHol2pFA4B3CzlafNmfcIo9WShjwJvujK+nG1hTcBcmXFLYWCSX5HDwK0a34l+eXsPbjIMnAZQle8RssWaHN/25Ao3zOsCbhM5tgeX9MhDVN3m86Uns5z4OOPgFvz69whhWHP2CX08HvOoneJPxsAzVkveT8RHn2IEUUQcc/bKxtH6Y+401cmQrqKjJepNxWjvcdHM9Jgv53kVMMeB1f6l0ZTl+iQHHsOD9POLAirXHsOBh6XHbzZijxzfzEXf9hHfTiENYEWzC1+cjlujwxeGCOXoEtsKJBSwVs1GkHcP6JPTKaenQuYQC4CaQH+s/f/sGAPDzuwfM0WOOHp0nJeDBx82i/tvpQGPRESAOrF5rTcFlCVUZ9jx3cDZj7NZqcZtXjz5EzKvH0K3NosTtPs8BMTrcHKYKTHof8fY8kqV27qroiyj0OksWuJQJvh/7BUt0mNeAQ9+USh8fB3R9xM044yG6jdpp4PAtssEg1jkZx2rhXj1CaP7AxpAozLx6pGTR9yumS4dxXKrqqZQfs8WyePT9Wv2Axd9YlGIXvkbKDiGS322/YpoDQkgVFK1sSQNQVUydyxU0nE8DfIjoOg4hk2z1D+6HpYKSGF0NQB9CE5xZOSB515PITIy2nhfFU2vJXzh0Ee8eBpRscHx9qdaquDp0fROEydlgmUgNpxubmI6oljpWVQVAweqPC96/7dEdVrbO0bn51ME4onUbS8HRw7hWsBEXsnB2w0r9YCuc1LPed/C3K5xPWBePvLgaCL1MDu6WNuLi6hD6WIOqx4X8k4UaL5bHkgxcyEiLqxYhN0TkhYRSDINYAEizA6KFHSNZfrgPxqCqjpbEYjOc1/Tkc2xcQb5QAHk7JOSLh+nTxscaACwHXseHQD69HX9nX1xz5LoLyEe5gPybb1aUxcFMFriNwKNHGRNbjwpwcYBhq0sB0b7Z5xjZkM/z7MgCw9Ypsxry0xXBG/ERnslH2Fxcs8TVMkv1gTaLaVR4FXbFzAo++gJ7MchjYSpsaX7DrsDOFvmQ4O49UadDo8KimOpbBl9gJ6LUim+uZd9EotOT36ydmt+o+L+JX2UBUDryFc5dgZt4jnfkO2tX8gmMR/LFBaiNZjXsi1ng1qYgmrvWZ/EbzR3514o/LQzIf3qloUwd+VymgX0dlY+l+M9+bnJzoyHbaGg+McjLfaun+lAaVAqln2jMycfSILNrQXGovsPZlYbvo2lUe/EnZ5picUB4MERd5zF0E/lJZqc2tFKj1bvZVB9o7ZsKoFLGq28m123Yb7b2Sfnqmcw+iaoMGM5neJ6yf5/4lSam7opvpPjOFoONb7H25ZS6xF9RaNOmqHIMaCyv+Vi61g7tY7lpo0MNJSN+pgAa9TRy+eYpIKufhW5rUOeF+G9KWbVvpdW9KUNAO7btlVTrlv5I+7QPqbleph63ZwGn2jAwpVSq8JN8APmSlnId6O4vMWU7bnrjYV/3ftNC+ntl7K+ma2X8If21JVN+xJzkn/6dL8t/9j/+B3iIwya8SMwOU/IYXKxUSR20W2iiUwzIIAqoAKmJQ3Z4mzG4FbkYPCxDtfRomqiAvEsMFOIjW/Q+4rR21fKl41TqlJgKewwLWVhVPlnIS3gQbalbWTRHUmdTVRqdk9/QGQWwpWwr5RTApi9T9NViEVzCxBQ4eSYDW1+Ezik+qtqKKOBur54qz/bCAEjKFepmUvdEqIGew3E4mzn8i6/0y5QtOh8RbMZ5DZsYlwJmxSoo7ViiZ2uSWLqIIikqoOI3e8uqm49zVy1dQtWU8B01HIPLXLZ7srvuXa5Kn57bLeE9crbwLm/KA5oqpzUUisKasglZEbnfcr2E8pAy9PXGlHovnLIqO1MqAFyi31hPBYhr+mzOtobb2FssCxqlU+6LlBezrVRKuT6xCqgxaPdMh0ZQu/hi/ZEYjylRKI0YHQKHTxFqqwCtSnNVIUAAojbmTGBSQKS0YeHxbbEvr1MuAWyuF8Arn6XNOZuqNDpNoVp2JFyHjhfqPT1TEh5ErGJCqxQBrK6LmGePvo8MnNv7o+taGBNRRl0VZdOxGFOS0A3AxlLT9xHzFKifPkH8qo0pCCFhnsiJTHy3vU81vEdRY1zQLIZN5ZRVRVP7Lu8bubfOEWjV1s4CbN8hmeJ4yn2RvllHq8is5pHML7l38j10ESnyJogtdQ7E9Wn4D+cT0upgLCh+6OrgAv2t94nHta6FitncFwG/AAjkGlRl1E19nI/CjeStoUAEyGQOsb89eHNC6tGxRlEoX2IRsZJsBfREEyXFVtfz2KWdpd6gip4ZT1RPI+FGhP5cgLw6GJ9JNC22/hmDWmal83JfSMwMLdwIW4Xz4tpYCVXVlLrZUOmkorJaTPPvlTijYoETaqxji7EAsmhIHCxapoHyRsL3DjciNwdthWu4PgvU8CG8AVGtNTpOag03wueFCaAta5ouu1d5LWh9q/nL03ySt+BpPFXt/5fRrIhAjWUqvuAiGiXWx5onqeO5bWZomqsphoTDHLZqubLQkDYIrdeghSXhhYQMoSnYCE9tTHW5AY9ahQZrO7Ci780GeO6HsGALWKVd+6Tr2LfryqlN+3f1PVvutbQHZc+072rfrpXxkuP4BLjblPFcpdfT9yEUfApU/l//5T/430sp/873qOF7p3/rz7vyP/2Tn/3Oy/17v/hXP3jfgB+5xbIzET/v7vFlOGHOHgmW6a2phhORsCF6cSyhQNDhSRiQGuKCrXSxWNyFiX0Tn4bUWIvFlz3NYCnrdXfe5NEhPvZpX54Oi3ET5hpC41peOXaNkil/B7fWxdoxzCoPteWuu2zadxt2YQ2uvIWutQPhaZ6PUVQ1QN8DUkmlUJzPvd9VhsEb1+JrCtVW/5UkoTmetD00+q0erzcHkkK8Rs/ch7QYuYw9/UvXE2yGDbtwFeHpWOhUQFbH/THJ2+1ii+7TiPXqcQLmCX1ooRYAwJhW3rVwD9f6JXm9i7vv9Mt76JdNWdsy0+bvtfpq+I1An3ufkEujj+q2br777XfYjG73livFYNy1T1LYMQL0MfGjlM/S11ou/x2HdXOsC0/zCHD8WCrFYBhW5GzRdXHT1lJMDefhPR3v++29AFDB8T6lZKslVUClFzDKYTf29bh+3Yxx2c0FI5Z9lxmYR74HlF97tZUC+HA9TIjj81a1fR/yA6bAMaXf7dwDahiNAg6vUVqsSOBp7MgKbmwF5PJZC58BSphMCjNP6bF1oaoEO3TMyQqmGAzmPRAw2NKN1S00YtlUx6vlWGIrqntAY4UNuJQ6yjUAIsCO/5ayxTmG/RaTCrlRZEB4zKX8emwTgoSVu6UNArwg7eE2iR+e8k0lNRuFLlRfANC5ZOr8qK+Z2W2ulf59dlpb36mO3ThKvWvrW82ix0JUs83uu0omNnAGNEBX0+S2gCJeuad6oa3BmtStX0MVbDVgZxRwtbvyxZPX7PpiVD6jxyeqOvDMbbj221O2YUWKIev4pi8KVH3s9m5K3+Uzu1fSs+V81+OcvlOUG5nHz/wWa8ud/H2+LHwUcD53/Dv3/zvk+c6P4PcBm39If63pRw0spxzwfzz8LdyGCTE7vFsoxMddN6GzEb+5vIK3CefY4VU34d10wF1/QcwWH5YRvYvobMLD2uPL4YR38wFHvyAWi9Pa4bR0CC7hj28+4NvpWIVjRGzmYemRi8FPDiecVqKTPi4dvhzPeDdxuBGmiIrFlKyTFE+wdxG/ebyBNahCLyJOskSPu3FSVNiAziX0TM8EaAFxXsnqMK0ebw4XEhtaPcYQ8e3DEUO34tAvePd4wHFY4F3Cee4qcHpzuJDV0mZMS6NOCs3yPHeValoAHLoVS3TVcur5uqFbqyVM2jeEiDk6HPsF80oUXwOgDxGXJWDsVpznDn2gWHtjt+I0d5XKebr0uDlMOF16dF1EcAmnS48UHW5vLphXzwIrploEhdKY2UJxGFp/xLIyM6UyZwIYwWX89rd3gCm4vbsgZYtl9iTcUQy6EDfAd7p0BBiYYinCJsYULHOoi9blEmBsQegjcrI1zIoPFF5Fx/ETS9t6CSjJIBx4EW8zW5DIkhonj3BYULKlIOauVHGNUgxKBvpxRYwWcfHoBionrg75MQB9QhhXpOjqAsvYUoOwl8QB0Hk8S+YFsoDgxVWLhghV+D4hLY7okGNEegxwx0gWDV7kCr0xLw5uSEgzlYMCuJDrYjAzPVEWlK7LSJOjck+h0h4lzp5YbdJiq7XF8O55PgUgZLiBweBqyZIRLcwYAYNqeam0yC5XYYgyExQyA/H7yuIqJbMsHFbGFArCHjLwgaUN3yytjMXBDrHG5SvJADOHZpGQMgAt+GxpwdwNYM4OuF2JznlMNO5Cb3v0lQIIVygMzCG1BcRM/cGY2sI6lLqwdO890utYw7bUMB/ZwJ4t0l1sC9U+U6iXIRHNM+RqUZIFvsmmhYYJhcKx9Blm5Th7FnVVZWaqLx8yhwcp1fJjVsvhRgyKzzDctjzmajGxF94sHDPsxSL3uVp2hL5X2DrVvXNIA1E4kQE7UaiX+CrBLhRORixH/mSx8nF3MYivMvy9RTyyucFSCBjI45CJklk8IOFHigcsUz8Lhxuxi2mhVQyq+qrjcCOew5NUY6QHU1Xpu525TwDR9ApQbKOYEkWWyosD0c1yKHCVCgvYmain3XsLWKKbauocjTeFOXGTorAWojmKUm0cC9xs4CYKAWMi0WgrFZZpk1lTYS/UztQX2GRgee9SqLCmNDplDUPDVE0TmXbK89oyWMt9gT+x2qmj8bCLgeP9uNQDdgHSSOFGdNv8BVjunllUvCC5iRSEAaqj3qfIIVgWGi83Nwqk7GX7C6qiqV1RaaXFtlApomxrCioNFcAmDAlAx8MjFBWW6+R7VfF4RFWxdRPRde3KbTOKxtrtKKZM6a3t9DSPYNS8yVS2XVoZxbR+6vqLp/qlvSaj7lrYlcqVUB+F6xAQahPYGsmqrRw+RJ5Dm0SJ2TSLMVApnMWZZp3lcjQVlp4TAxsLsuMQH5sdlTYeVC42FtQK7HZUWABVFZcaIs8vtoAfrSyxjlJm1LAlFTQry6qUQ+FG0Gi86n2i21jv28cAo8F12ukVcKrVh/dpT5+t6rtCrd1vuujx2I2LbtNmrF6afs9AacHfbB/LHzUV9qs/+6r8x//9f4I5+U0Aea1+KtZJEVbRKqFafEYrlUpesYasqmyhpWoRnEsMCC5V8CFATH8GsNnh1zRSAE8C0ztWhNTCI0Iv1cmgiXOI1VWeSVEezcVUBdBrip5a1CQxQGniI3ljKRFAKm2S8vbWPKGdWkPKnqL2mQuq72DKplJDBRhqyqmzpdJGZffds//cqmioe3VDnQTM6iQUUcNjUIpBH1ZYQ6FihIosFNO9RVHTgPdpL/qjw8NIWzXdUT9+WohEaKLimydUSa3KKVaZveWu0ljNlmYq1FABwRsaoho/TSXUSeqXvJqGqBU2pY1CTRTQLHRTnUfK1WOg7yMFuC+VplnHrCjVRWAjjiLnrVhtlJWjtokptbquaxwqovgxmLD0y6bnnP4N3N87ABWg61/Ba5bq+juqtp6tKc8qn24stfJX3WtNu6yCK4oGJTFm60LCtr7LPUOhTQXZwKjXl+27bFOXKHWa7fdNfra6ifon3TOprzQaqLTNlK11TT7ulEHlnmlhG+MzW8X4vLQ1qXGR2+/ISmdsASx/dpk2OXZ9rWNfsKUWllbHpk26T7ofhevVx/eiL5oOuV98yeei8sl4bMw7pdEvATy70JIx1X3S5cji3FzJo/u4V+002Fob99/37dVpP2ZAawf31xSzDcnDc1VTOTXtstIsPyMV0/wX9Y9LPX5tcSx/9T3Sc0bfP52uldEqbPNfvn9XC9e1dtR7igaUgKd9utJOfQuffJbrr1nKdQPV16t1lY/ij4+DiCvnjB7f5/r3zPUvATet/7v5/pH0Wda85wZln+8jdV61fH6qLS8ZN3wGEHxBmZ+T/tl/9cNTYf/sz/vyP/yTP/qdl/vv/uKf/+B9A37kFsvORnzVPaK3EWtxeIi0FXZ0M3ob8fVyg2AyTqnD0S24jz1e+RlrsXhYB/QuwpuMSwp4FSbcV/EehykFnGOHzkb8fHzAN/MRg1vhTKm+ihLe428dP+CSKNblkjxuDjMeVwpbcXOYcY60XViBb3EY3IrOJnwzkQDQ0MUagqQUEuN5PVyqCuuUArzN6GzcgE2pZ4oBrweicc7Ro/cRX5+POIQVo1/xfhqrUM2cPFkkbcbrwyOmJDE+PUa/VkAcXMIlhmaxLAaDX7FmslgCQOgWTNHjyOELtN+j+HUOfq2+nnL8EgNGv+Jx7XDbz9UPVI4v2eG0dPjieMZp6eBtRu8jHuYea3R4fbiQ76byIyxA9RcUP8ebfuKNgwYw5+gxdlSniPv88v0rGAN8cTxjzRRj9MCWShH6kffceSbr9NitdXNA/DHn1Vfa3nkOsLY88Y8c+oglOvTdugEp/z9779IjybasCX3r5e4RmVm1q85jX92+LdHi1ULAoHWFkEBITBBDJJgzZ8hPYMIIGNMIiQFCYtQCJghmSMxA4tHSRbo0Otz32ffsR1VmRLj7ehgDM1u+3MMjM6v27nvOPpwlVWWE+3rYerjHsmWffZaLwTgH5Gwx9LH6QHY+4zR1cJYwR49DPzc+p3k5EAArNQ/HESlzHw6NBfp87hFCxqGfEbNb+Qhay+dn7P+YV76WCgUmADH6qpQkCdnQdwlz9MjZVN+9fohIyYlFt1QldJ4duo79ARco58LOmlbkPVT9C7uO6/UhV5/IlNgHDmA/SFVCq2V67GBdRicQ0ZwtcrIcEqVnaKmGxciifDhfqlKSIsfGDWKZThp+giAWaPH/SxbeF0ynDkTA4WFayHuSQycQUi2XogOKgR9i3WAoM68LbN2GIaTJoztEzOcOvk+wDXlPvITqq2YsW2P9EDmMhC1IswcICAMT7hgLhnZKe/FjVwl6crbIk6uENzR6uPvEZC3RwncZafKwXWaSn1BglCkXAjctQrYzS6zZaDl/tDWubI2lO9tKyFP97sRq2SpzxheU2YGSEUuxhIoYmbzHdAU0ieVYFdjGjw/FgL7tmDgnyHH36NhiccyL756GgHj0TOoTLcxsgWMCHgMwKHkPQBfHG1vdHHuqvnsKqzXjM+Q9AFbkPY6Ak+N3kyq6nhaLMMAwQ21/j7zHEODYUpoPha2PgRjCSFzGzg5lKHAfHVufgliFd8h7zMTl7SS/M0lIcoS8x8wGbjLIA1sslbynBrc3i8VVSWpAYsUsqIQ9tbzIaBKT+uRerLs75D2aXy2qagHme5YtX8SWMxvZcqlWPCPKqLsYxPvP37G6WeLAythU61gRS6lYGe2MauFSq6ObTI3vqpbA1uJM7RwDNYZkXReNgkaWrewcP1es+bMVK25DwpNNJf5xM1vJq2Xf0kKIFGgV8N4ktghrf7RPLYEN1EovrzKNY2nnpc9qWSPP10tD3tNa9ciYlR/jFXmP/C0ap7MZ970ywKLMqCVWLV98EcvhTEPes1g11/Ne/TrrhWWs6rhtFXG5VmMOkxTcKu9tta3e2cqriZbrK4tlg0Boy1cZzWbcbijtV4cZOwpetQRbI/XvVHbrkGQj07bfewoumc3Y7tX/TPpkRf136XulH7ViOZWAP378OX42PGEqDr8a71HI4H1/xkMY8f8+va/Q1Xf9GV+dH/B+OKGQxV9f7nAfZgSX8d14wO/ff8BX5we86UfM2eFx7vE49ggSGkEZVp0wvwaX8c2Z8TDljcE3lyO8LXicOnx5/4Rfne9AZPD+cMYpdpWgBgDm5PDQzziGGX/+4W1VUubEMQZ1A35+CNWiyZDRhEEgs6qknmdW/ObZY3wQhXcOGELCNx/u0PcRxz7iw+MBd8cJwWc8XXqkxDHk0huL8xzgLOE8BdwNonxGj85nPI1LXgA49DPm5BEl3pv3GeOlQz9EURZKVQqCz5ijx9u7C8spSknnEy5Th+Mw4fE04O7AUNm7YcbjucebuxFj9Dg9DYgPDk+nAaFL6EPC49MBJbG1UFlCmaGziOLAygDHLwOmO2ZXjdFVf7Np7OADw1DvjxP3/VfM9GptwTx7xMmz0kIG/RCrBa4Ug/HUAQYIPUMcvchgLWEW5aeQQX4MQCD4IaFkVlrS5Jldc/K4iGJhha2zlskG6YGVMuvYP2w+syJBZ4/44Dn4uDB5GicQ0sLWuXRvkaJHvjjMRz7UyNHCfNNhPBSkNw5J2DYBAI5gBfarpB1W/LE0PpxRC+vFAZ0QRiQ+MZ8OGTQ6lvsuAR8Dyz/zJhye6y/JAqNDOibQxWPuM5AN7JBrvLkyyy+w+DHNh8x9PibQk0fsGepqXKmw3GXDbxAdsaIBAB8Ccl+Qjol/fGcHJAM7WYwPwlYqShAib8JjV6plxwgDaDqyIojZIvUM30W0TMRhAcwWsStwXweYAlyA5VR+shiPbgk9EC3sheGY8WHZoZlkQY6QQql+Uu7JYXpn4b4OiG8sKzFSr//W80ZQYJV2NFIfWIk789jPDxbm4tgdLJRavvuVw1wMsieYi4WbbN34d48Gk1he7GgR7xPsR498dAyTPZRmY0QMbS0GuS9wstF1o0W+s3BnyzA8S8i6uT0zvDXf26o0kRflYDLMKJoMitRno0G6E7htIPhHCxgg3xX4J4t8WJQ7F01VnJANhq8s0j3nMZmVCpMN5i8s/LRsqEFA98Fi+gngL5aZQ98ZdN9axIclmLx/MrLBZYUiH2gJvi6KVDgJ82fH19zI8le4n+P+ugsHqw9PMgaymco9LbBHAP5kKpROFTZyLAu/tPi6Pxmko611VJhpILizQXxjMPyKlYXcYwOFZdlLx/WUniGkgCgDgTeR8Z6hu/4MpDtWSG0EQ1YP/BcGyL2pipY/AyAgD6yE+pG/xwfDUFICiufPNgPpwOyuxXP9eTB14+/mpa7wCIGVmgoD9Rcexzww7DLeL3BRhaiGR8L4k+ctRs8l7buOjRUGUZO5PTdxO+6yKFrF6/oByBkelxGVbbV4IDwRyBqUYCoTqE18DQBKZxZFRZTR7gMhHXWeDfyZkDvDbJ0VosvXAMCPhNwb2JlA3oCMgZvlwKdfIKYA4GbiOYgsA3nAX0gUO4aL2gzkzsBN/PzmjteyP9MVFLZ0Bu5CyAftB9WDCDvLsxGWNaRjxuOwKJDFC/w2LeNe4bMbpVOVD5J+tYyvqqho/iJKtMJxW4WOjMyF21k3otgZAkwmqdPwZzwPhdVxbFNVGMH3bV4OVtr7z0FhqwK8VSz1XbVRvK/60ijhV6RFAPT0dSF2aipSKBW2SvJyfdv2rqV0Z5yXwpvvTZ2vqus3IO1ylcJ9GzoAACAASURBVPyWpB81FPbd3/05/ev/+b+NMYfFaqUWAKyhigp5bFNL2rKFk+6lLXFJC3t8joylTbdISm7l3ZKTALiCgG5hZgxDXZP67EFqt1JevWNutLcn3x4kddvXllSD8yzX9/p9q669dlt5n0u3lvueLM8R2bR92f1bzP6b8zUvE0NXAcxrNbieN71ei+9ca+9RK08r3wuphce2c/hJr5AbJ6BLG/trau85eFU7+lnT3onnM/JwPv1Fvz1/e2PzXFqN294vYdvm9tqqIrqGYj63AG6ldixeKrt7nLz5Rf/UH81b7e+tl5fmq21/i8f7Pj/mr1kvzz10O+UMmRUxSxs0/lk5XrOOX5LzNfe3bdTvt2XcWwqvso48l27lbTbOry7/2jY/Je2MR7vcPhkG+Jp7tzbXO5e3U2leaucFeV6c2225rVDPtH2t7Jhn7r3Q7qvqf135772GPzF9Fhz2h6j3hxjTT6n7hfS9lcRN+X/4n/xmQGH/i//u93/wev/lf+IXv/a+AT9yi+XRzfh7b/8UH/IBqVj0liFvH9MBU/G48xPm4uFNxlQ8Di7iMTJc9kHiS85y/WMc0NuEqXhYU9DbjDs/IRWHr6Z7DI493FNx8DajkMXBRVhT8PV0h6OfMRePBz/hQ+R4kABwSh06myvTLMBxHE+JrZjvugsiWZGzwJrC4TxMwWNimOucHQaXUGAwplADtgPA0WssRSYhakOp/Gx4wmPqMWeH+8Dw3EIWg4+w8rR9Mx1r3Rp2pfUjPfqFnbSQqZBcCw4Qn8jVcp1NSORq3XNxHHsxhQo7TmQrzFahtxquRUPGTBLypcYW9RFT8szQ213gbcHX4x1fz762B6xZPa0hPM48hp3NSGSFHXeu5abMpEK/f/8BAPDV+YFjT4pcAK4gvoPAhS8pVD9TVXy0HQC4CzPm7KqFu/ZboMqXGKovp4Z8OYYZRz/j2+lYDwgycSxNDSHzcRo4NInU2caaNIZwiTxHhxBxjkHWR8GbfsQpdjjHgGDLKjTLLNBm/Y3UOls/XWs4HImuDx2TKTv0jiG5Y/K472Y8zR2CQFI1tqgxhF5CyHQuI8p4zslXBdJLPE0vsl1iwCFEXGLAXTdXsqpCWIWE8W6B7ybpy30/YUy+xsYM4p/bu4zTzHislBmirDFLNQxNLgaDsOde5lAhyVqX+gir32/KFm+GCcYQvjsf6nrwEruUyxRhE06whnCaF7pYa/iQqTRr6dhFfBx7vBkmnKZudWD1MEzMel0sCjFR1tPY1/nspY3zHND7jFxMjRWq7McfLkMND9P5XInD7rqID5cBJM/TFJloa4wc2zVmt/IbDjL3MTt0PjHU2GeM0df4qPocAUAfIoIrOE1dJSfTg0EvvuXO8pj2IcG7UuegSOgdADUm7Jz84lOpB2vix/v2eMFlDnWNdT7BWcLTpa+QdQ0hc+hivT6EhNPY4W6YcZ5CPcAbBP6u7MRz4vitCs/O2SJIvUnqdbYgCmwcYNSDsvBqHFd9v6obQm78OpU8TPtVx6Epp/WMM8PvFcXB7VmEkDCOAffHqYZAUj/nFlqeMzMQp+Rq/bruCRxKJ4SMoYu4TF2NGavrRP3hK6Q9LQzDinLRWLTjpVtYeAlVDkaXLLJpWJpCpvqO5+QQhPxM/aWtywgCjVf0zDyFGnJGoeTDEHE+Dfjc5ENCnOWZdgtmUuHz6hPtQqoIGoWw+5BBhcMoOZ9XpGuKkLkK70RLvFhtB2Df526ISBo6Seoo2XHoFyXLssIYTEyUxmPBPtNUTCVAK9msfI8rJF181WsoGglLU8PZCGwdBCFfMxwTtw2rI2RvJhQO9aJd1LAyEjpGCeF48Zi1Zm7A/7KpZGd8T+sBajzZVhsxWBAk7YFJNePJV/HHNUUOe7ba+IoUaFOfKtpWDoaadjQky1qeptz2MK1VVPcOeFQO/av92/oSA1eHotVKe+OM8mabN9KWUKdp6rru7eHOtv5nD7puHIRvG/u+h4d/Q4nw203e86NWLDuT8Afd1xjSG0RyONoZGQb3bsKHfMBbd+HrbsK38Q734ntZyOCdP+PJ9piKx72f0NmEezfhKfdwKDi4iKOdMZFHJIsvwgVT8Y1iafDGj5UYpxOlVpXb992J/fMsK6waAkXDoRwcK0Vf9h8xFY+pBHib4VDgbUEwGcHeMXFQKCK3xcUt/praHivCqSpYBxdxcQFf9h/R2yMuOeB9d0IvSs/BxeqraQ1x3aJYnkSZ1fAjBxcrqZE1BafUI9jMSmRxiGRx72c8pa7Wr3JMxbPCa3sMLnGsTzJVmT+lDg9+wqPrMbiIp9jj6CPOiRWjd91ZFMwJk/cYs8fbbsRB4ot2NiP5hXyIlV1WeIMpiGThDR8CdC5Vwp2j+HB6U/AxDihk8NP+BAuSeKYZ92HC2bEv5Tl1HGpGXgT3fuLrMhd6KAAARz9zfptZ7uxxdvx9Lh5HP0s/Z/RSv0KbrSG8CSNCEz5hLqz4vulG9t8NU83b2VTvq7LvbcajHeBswdHPcCJX5zK+HB7x6Ht8a47ofaprIJHDLLEv9UChjU3ZEhUNLmEuDp3N9Tk4xV58cwvOjhmYLWjFYKxMykc/4xR79D5hks3+JHDzQqbGf+2EDCu4jDuBrD+ECb3EplWfXD0sOMgBSy4Wc2HF511/xugDzpGVyN4lOIlPq8qbxp7VGLJ6qKEHEIVMVZB7lzC5JRasKvXaly/6C8+zxHpV5f/sQv0J8bbgvpvkEOlQ51kJsGJ2lRzqTT8iE8PpAawOIN4fzpizQ5aDmkH6r8p1G2bnEKL4bbul/HCqBwa9y+hlXvV5U1RD5zJOlvDQT3C2wyFEbldjkBL7IHtbMCZf7w8+wdmuQvcJqPFVB5/QO/ZxDdLXGm9VDiucLZilvuByJRzLwqhdiAnA7roZY8r14EL9nWO2cJbwbrjAGcLo+CBk8AlelB9VDvUw4e0wIouCdt/NKAS8GUbpJyvR9/2M3KyJyS9xd43hWLy9TyCgKrPHEPE4dei8xuNl3+U+JMRs8dDPq2dszk4UUZbz2MVKAqbjngtfB1Dbuu8Z1q+uDK2Ce5C8bw8ju05sDsu8LbXdYz9X1m6ja927SmZ26GINtdT5XOMaBxnjQgZT9PWwQQ9oJses4Xrooc+DKspeyp8dH8RpXOPJ6aGCqTGE1X+8FAkrJrFWta3RefTifz50sT5XqVg+lElt8JtPS31IFXGgSrKRtXuxnZCksaxFXAxUKRxkLjUurx4K5GzRd7xu9KBDnwddFymtY1iXYnAcZkxygKB1xFSqUg7wYUv2rMCGsPi3Z1HI9YAlJSds6lyfcxyDV5V8jbnKZGwQsjlWWvWAoPqlC+s6kdTjDUp2rJRbvyJpI7CCzj7oSx+1TWDxz4YRdvFkK/u3+nADWJReHSdRVkhY4lskkRKWaRskylolKWuVoAKG22sYHGCtwOi0WAlJpIoepE4Nq3MLkdEmbU/bUCV3q4CpQqnKlJJurWKZrs3Eq75tEt2S6VnF0lzVRTvlaVvPS/Vf1bEZg2fN39/XxPk3kQzyTvjB35b0o4bC/t4/957+nf/y30QiV9lcAYbBxrKwwbYxCBMtisges2eqrLHKnLpMvioWFhzfUpWJtFkgt2Co2/tbeCqAK/inxmZclW/kaL+v+2FX1rPn2n0JWriF/O6ltk5gzVDbXtcNaxs79Dn5WtZZlUE3RK1styCr29PfLXvtc5+VzVYtZEs9C4R5D665B03e+74nb6vw7M3V1fiQuZr9vd8vANVatf19ehW0tK2fbkNTPxWWvMdsu61/CzXeg0U/B5e+1YdWhj252u/Awoi7y1NAC3Pvc/1vLWvqt7uF0W7hxXtw4xXsVjcf25NbYHVtb0xusSm3a6yOFZ6H2T37S3JLtmdOmG/Nw9V47J2Ev3AKvgfzv7pe1szDi/ybtlX+TV+q/FrXzjxqf9r82zk1zbp6lXmg7X8r894E7m0i9zax9b7ZXVsVkq2b2luLZSvPa+p97l1yq5/bNdHmayxPn5taCLN+f5V8n5PntVu0ds3sDJte+xzL1Cr/jXxXr9vn6n9pz9Fa9Dblt8vzqs29dEN2s1P/S/Xt/qzcWr830jM/Ta+S4fvmfbb9V47Dq+t7ZR0/RHqNHH/0H/76obB/918c6D/7b//gB6/3X/s7/+jX3jfgR26xPOUO/8c3fwuTnKArc6meCitcrQ1lkeTUXD+rkqPkOu3GQjfiChvahmHQDW17kmgMaogAAAKTWocbaeFaXlgoWwichmXQmIzAOsxEW5eeSlpb6ilshRXKqWQbsqENDQEwpEjDKGw3UUQcR7HdECnr5JJpvTnchmLQcAt14yZlrhTCzUau3Xy3Y99Cztq8K7/NVulRCErzo2Js2x8DFAPfM/NnThKKQ08uNY8mg1V4hdopqV9JbrgdWmTYbnIM6qliuwG9rttUAp3KipnsasNa+619llNMDmauoTKY8IaKWZgzW+VkO5cqY/sdQBu/sI57hUWhxsGsfdbx0fo07EFLrb9zwryi5bdYQiVsT3LbTX3dVMo1zd+GBjC01NfuTrT9NlyAba5p37dKRjunAimrAdS17tX6oSVcwxYe1SoU2PR7Gx5BmSPrvGCdp4WTNQyjq/LtKXo7ls29bLDArtq+bjfq2MnXfgfWsmr+W4qgAUwxKGqNqAyQBkWsFy10TctUH0V9voVJU1lcSeozieeZpG0CKjMpDFDkc/G0CvbOjJiNQlHMWvYCQElLylJGSUqM9lUtDCT5aRkKo8+UJkcLlKxd923okPY5MQrnw5LXcj80tuatcCCGsMABG0tQfXdmITISeF9tw6B+r7LKGFRGTH2fSixOI4Qrq3eiwUI+0tSxDLjWJayxm3HZMpEaXf8NVJCJTl7e/N9Kdc0B6zE2VMdnV4kDVnH/VjEAaRmXK8vO9vlpb7XwTABkzGotabHVWU57Qwvq97222/vmOk9LImPaOdbP7ZponuWrzmz6aNp3easx0ALpbC5dj12b6jvl+lV4pWjfqmNH9vU8m/Wcb2XbyvM3kF5U9rf3XlPHK8p877yfIsOPMBGAUn8If/vSj1qxHFzEP/32rzEVhyKQuK2FTpPGttTrW6tlG9eyTa2FKAvMc6uAtiZtC0KBWTG3tnEP2/ra9Jzl67lye33dk327b2vLvUQ69JxFcFuOyFxZGLfWrT1yFrUA6mFAaxW8VbZNr7EEPkcspNfzZg1sf1drXqzHtDxjDdqT7zlyI1WouRx2rRxLvev7n0Ko1JI/PSf7S+k5Ip/n5H/Osruu47oPe5a9deXNr/mNzcmnpNeOBwFoEK71mrap0K8rGW9W2CgAbdpce2k8mFAK15uK7UZju7kyWMenxMvvg1elZx+mZVyWTXBzrW13I399VnVNKtxt+y7p9tfHyurdcYXU7bQln6mV+9kN2I15lHZXqT0Iec1mt71Hqz+rOmo/9uTSr/X3YrPD37lOYfPOkIOV1dVbz77B4suGZU73yIt217RA45ZDsbU2pXA8ZRuu9yW/MhR/VmoVjDo2Ut9mTPYeb9L/TPO5vbfN/Ir5N/J41mX2XJur9bJ5B7Vjv1mvt5SUNpRG/WywCs2x6qe7rdus6tVCV0uRrhT21ccXptZs+k/N9RdfyZstYj2g3oh/lV6z3P4xKGlt3z63zr3Lr67vpffiK+v87DH9Xfq1pB+1YgkwaU0hg0kgrl7e+AWLMsbKHfsRlObIrhA7kSupjPp1abKGxO/Oi/JVFiUMaOCua4ukBcdqJDIoRqCsZqkPZmGtVd89hdfaxhqoyqkR5c+sNpK8udM+uqZ97aezBVb63/rTtEmVaWvo6oCyjoMtDFu98eR7LM+4jp8za4tkK+vVxtQQHBol1+rcLlBYYK0E58bC3Fo9t9fWLNgkfQWKxmESeUKNpZiv5NtTYtvrxhDsnmLv8iqwfTVgqOK9M54GAnE167m+1a5+X/pdmvulWurVAq+xKZfy7TigsWKuLZp7yuceZFPJRV5SZqXmq/48B01cz+3air+FTC7lGyXD0Kp/1/Wv5V+NL7Z9v4bFVnKWvDZZ1OfW8A7Q6NhtrZ+btPIJ2jw3ijJQ+dSqfwXrBaQ9Qmlu1kMnlXGbV9etjtmqP3KrmpRknKoCet2v7TrRa7vKsFpbRSAeq83WTcbPmh0FlwCj4ltZAO0atFuLOjXXxZqvn7dW6p1N+KvYrknL0lJWxrSFutYDgG1qX8x1t7hzf7vh3iqnLRJAFa6rdvZ22PJZLfqG1gpw22b7mcximdU6zXKrrm0tYGnd/1vK+HY8tj9cbZ9v/f0eybRKkykVGrv1Obta29r2nhx749imW2VIFTce35sHdLuKE20+0qaN5T69Yuzas5pbiuhGV1w9C3t17SmPu5Dya5Gv07ZtLO+zehjxXD83116lYOF6D3Ervba+T13HVP/7jLZ3ru8sj0+vd6/O7/lc/pjS78h7fkNTKhZ/cX6LziXM2VcWTyUSeYwDLAhzcehdwpgCBh+RyeISQ41RGYVwYUp+IX8oFnNisoGHfsLjHCohhyp8s5B3HEPEJIyHSt6gLJudy5UVs1WsnGWSnm+FGdK7Ukkd1A+O5eONXhIyitYaaAwhJpYhF4s+RFjDcTK9yxjnAOe4HY1LqYQK+qLT+JlO2BiVTKEl0GhTcFlibS5wVYX+5goZ5rx6rfNLmfa6cwzfdcLWp9+DkFzE6FZxKo1htr+SLfoh1niZK/gtCQOf/FgpS6DKBqDGnCQy9f7Hj2xmGo5zjYdpBc6sfdINoxIUKOGAwoyXdgqo6W9LoqCKXikW1uXVj6pCcVGYVU/vWVuQIwe5LxJ4niGyssGp5QEQmGCBmOBASRVKMSgjB5O3QSCxDYSwheAahbWSfNfNIGG12a55fWHWPwKsBLW3Xa4w3LpRFxiu8aXGrQSWtgHUjX2F+EqcSdtljlvpqLIRlmQrcQNlw1A+R4uiMzuGEQpjYSV2yAamW6JEk64XkvZUHGElNIHnGdnAeLnfbrCTxLScOHakOeRFacxNeWkDUU9OynoDrocqalmMlmOGTpbztrDSLbNiMSyDptS0obA/tygDZbSgoXDZZBfYKgAzG74H6acjIBqG+iq0sP1N1HXUwJxNZqjiKgi57jCyYWuV39wH1myHWk/GAuG04NiXkGvRVEsZgDXklQCrcTQVEisB7WkoC1Ojjqv02yQDEw1oINjRoHRLf7VtjQVHHhVqbAs4SH0USJzCAyVGpMJh2ZqzBKnXOjVpbNIaSy5p55q9t0ENbK8KWoW6Kky09k9gvR3BjbxuNWB8jWMp+3pyBJtMjemn8pPjNjXGn52B3KHGFtT4gtpH8jyH1NSj14TfDrnjuni+UOML1piDGquv+QnSGItFYlqSRY0xahJgBcJdvPSj43iexRNsZnivnQ3HPv3MZKPh2KcqD5l6ZlCCwmF5HcA2CrSU5XHmcSCDmkfjnF7BgPXd21gGuTLATsv4kOU6dExW595NG3V8ZS3qmNYYkNq2/gS18R4XXjnOQ8s8qQyA9M029UjdNvHcteU5j6kytBDa1iKLsrTXttnGZdRnZKuUruDJy6u/ttPe20KUtd5VfMjtGJimDBa5Vb46Fk2b2zGoSZVG7NTb3m/yVSv6S4cVG/9iQwQyBmb3gGn9d+9g4qa94RMVzmfsFs/K9Enp/0dK6687/agVy/Pc4f/8E44FQ40/Yk1bi0A9YaW1L5VsWBZ/JKBCsgD8sj1xbZPm29bXLvq9B6B9IWzytf5By8tlsXpsoUO1DgLOzYt4r+6Lfi7L9bNZ14FbdQDrp//GKWOb9PZl59rui1zS1MiT5Ec2NrIbAi5uuH7Z7rQ/2uZ+0y/9fdQ9t/pRja5f9r9Nnr3D+yz1lc0cF4A3G5pdFT850VYjVGk32zKOrp2Hpjkr11xzr8q0+U6yFNu8lha3PLQ/nGjquFHfNlXRpP7tcnYEkAlX8rVldn8ct/dbuUyoLmWa323zbOqq/jSbZ60u2+0zuO3nZhPQLveWtn21iSGAnFtf22uHAJgdZsrts2edbBg2efdkNq62t/Td7T8bxQDGNhuX7bO97KD4OW0f2u0mZGeA5Hv19du2TxDlz1xf3za1t0aA9Wbq1vu0XQPN5oidqLdjApC1zedlnpdxuO7L1Tt+u+638mN9XWVchm/9W7IN1L4/NvogtjvNRkC99+Lm6lrQWxtAf97p7/adIvdXz4589uf1vVvWrqs+Y6n36l1S70tjZ+3zMi6GADfd/t16KfHaMMvnleyiZG5fjHURPVPv9ePzctobg9ekW++lnWvbubtZ3yvLP6uM3OpDfXfzhytl6LX9xgvjuveufE7mF8rW6y/81ryUnrUEvyY9K8N1hZ86Rs+lz5J3296nrO0fQSL67WaF/VErlt4XfPHFCYPEMpuEvKb3Gd5lPF4GtvaI1WycA0JIMADGia15zhXE6HDoIy5TqDTZKTkksZAcjxOmieODaX3WEuaJrY3DYcY8O2h8rX6YMY0d1GKWE1sh63NNBs5nOFcwnjrAsKUHZFDkxJWShTsstOxZ6LVtA4EDhOAnm5ofAIrEqcpnDxMKjC8oo4PtM4wF8myrIuyHxG0atoY5seSoNaxEu4Ls2S7XmFSwYoma2RJGLVGCASBxsNoyZMDyJAsbMsrEZUuW7zPLWaIFJgdzSKDJAV7IZ0bHlqlDBsmJ/+qkTk/pdYz6zPupaBfNKoqFiQxMl2EsoXzDYVboTWRttj1kCLQmGpnkhaCWEiU7MWDLDmtWMDPLR6EsBDfJgkKBiXZ1XQ8uzGTZwnEQa5YSZUyWrQmzRRn4nkliFZIx0EMJ6rheMxtQxzKaZOBOFqUjlANbZdBYNJgIw4iVgaq1qLVkAWBLjli+bALIAaXj/pgMUE+wF4MyLKQnrYx2Nig9LfWQ4fFQIpa4bOwBgDqCmQzXO4qlwGLJq0QuUfruUKGU7mJRAlVfMJMMTOa/eZAdrRwomcxfS1iUITvzr3jpWE6b+L5+5lN4sRQ5wJ+4v+l+eUZtRC2vp+5uZlnzQMuGSRVVh3rw4yaDdCT4E8vbnua7i6mWGrJs+ck91R9hO/H7Jg9soQHUmsLlw5NBPALk+b7JC/GJH4F4BNQKVjqCmyDWH7ZMLaYGwETR8TzfV2tM7gAbqVpLqjUjgtd4D5hMqJY4g2rdMIUtGzYudam1wkpo3dLxZwpSnrC2rhDgH3ntlMD3rI79gWDUQiTkOf5skI983Y1s1dKxB7hON5qqDIHYEkmOxGrHljFVWornfEbWgFooyZL0g9eNG+X1ZXQNoj4bMAQ7mao8k0MNi2Cm5bkkx2NROpaLPKqlU+/lgRCeeG0Xv1aaVZEuHjzXgS2C/NxgsVj2gJn5Xhp4bqrFUqyyZKR9sWLqfJHMgUliKB8Wi2W1nBHYF1TXQZb51Wx5GXc3olrCqsVSouyUIM9eL2uksYK6EUj3+Oxksj4DsqbysvbyAECsdS6iWhlJngE/LeOs1kWd17qulfyJsLKS6TqtyQL+Qsj98u7T56+1uuk8AICbCSUYXhtqxUwkY2aW56hp22Twb6iV59kAZA1MIVlrBjaSWGJ5rbqJ2Cqu7TvJJ+1zh8B1mMVaW5p3nI6N9qFaJi2/a/SdXZ/7TRltQ8duxTSrzzCW565ayG3TFhbF0m4slnuHIe1B1mJRJRRnal2gZX52FcVGCdw7zKpWxkI8DyR/My3zUsuLAi51ktX3F63bom0bzbiYRe7dAzxcl699bbu1re9W2pPphmL5WYcxvyFpL5rDb0v6USuWRz/jD7/8U1xyQJIYbErek4qFffiu+hAWsrCmIMmb3huOKwhgBW9NjR+i/o3ZwbxZ+y4UmBojMEpMxJYcyH/B99rwJnqvJQry70qVV5PGKdRre2QZrb9hG6KiTa2P6a3QFW3ID4MlTEibp32mXyIJ0kSbdvfGoA0lwvKbFXkPQ3KXawCT+zB82VaCny3RzzYciI7R3meVs3//BICDrluzJmja9s01LL8KW9YcLRHSlhRJ8z1HxrRHpMQ+dOVqHbTkO/odwAqm3PpO6lhv73Mbazm0LfU7bOvXtA0V8hw50sJiXCpL8F7bt0J/bP3Ytv6723vcFlX5lvo5r90y8t6QG1hc/oxZPi8CLr+WCun1G588swmNA1k/duMn3MouGWCLBb6Uv2h+2634dcmBVbsGq2jNNQL44EDfFfqLLAp9262k/ZJyIKAIlLmYnbWCZW70PsnfvM0IOYMxWJMJtRuVZh7JUEUG6LXSjFXRMd0OgLb1B7RAvqV+Yxr/SVr6mhp/yiifk/pjypikdgHIGK79Ixufyb2d556sezuwV2yqnrX0b/MbAMVgav1Xd+7vpnb39trNZZVjMwZaX/v9VlvbdLWgl3Yqi+0mb0sQtLSBCpn+nLRiHr710LV/t3O5I389RGzz76znjSD7Y9u+73Y38tfjxJ93Gtmb780XVnS0jRs7+/beM5v/K4v0bj24+c571mp4q809COleecmzb23dDPJK29l8x2cqQLVM/cGSa2Z9vR0V2nxfCfUKBe8T7726X9v1/7v0W5V+1IolkcFcPDPCgplY10ywtt4DgEKuKlqpMcPotUXB4Xvtpmhrtub8rioCqtDWINfVIeJaSWjrjfk638aNge/tMNZulRpg/ZzGhpgnNuU12PZzKct47Mmi/bgiz7lRT/uX5W4cl4utwcdVkSQyKNBNm0FqwiikvPiXZim3RaeljXL8UiKwQqljdyu2Z/VrbeasrQNYSIX0862UbmxsCvZZWl8X0FuVJZ2TRRYNwn0ld6NYtdfa+rb1axsAGoXIPrsONOXspL39eJCvTmRA2t8bJ3+5XMu7W9UL17XkHgpytWHYhUkuG716UJFcHfNbBFGaDwByWc89AWtHJb3Y7LZuo8REiWrnq1HaVnW11YtFntr7O23c0mu2m+DXnxUZZAAAIABJREFUTn2p60xlXM/31dy3ChwAavwTVbC612oVTkAw783nVUEp3j5Dm0187XsbxmU1r7cUJrOS70WFYq/8DZmu0jZsTdvOrgKxrR+yuTbVj/XZGI7tGFRFy9wcinrvlZv8Npm0/761dB0G4hU/CzfTHlHb1cHIXh9emMtP2pTXtrbr32KrbK5W4K6i+Yq2bn2XZDdZdhW+TxiTXSveph5zKy/w7ATfrPuqjhfKXeV74fl+xZh/ksK5PbD4nkrapyqZr15Ln9unV6Qfk4WyTQQg/y7cyG9mmorHnzy9w2nukItF5xOTvmTLJDohVdIbJbRRpSC4jCTkO94WzMmh8xmzbPqCy9W6MkVfY0IuDIxM3mIAzNFXNleG1nJ+gBUC28SC1PIaX7LvIyubQkSjG6gg5DtbIhy1NmlSeK61hHn2q7xdl5CzRUoWXZcRo1uVKcUyxNYuLJu5iYVZiAl1lH3TGKpxL2tsyWLgfFnqaUIFKFFNTo7jTwqpjVrgcrZwPqNkjgNaclsX4HxGio7zFCttZVhLiLOvMgCLgtTGAgUYQkxkYJWUhQDrluD0JXEf+jvGIc1jgDF0lafdMFRCHCUcaJgArRDTAAuJTt2UF5ajJHPNOmn4l9J6Zh5OUTFUUk4Id6wvDBMGlvGmpbwxhBJdJdWpeR3BB4lrGhlWXZVCqYMIS3gGZfg06zY0RqVRGDQxDNv4Utu2IbMMriz1KxTYEecXmDQPJJbxteB2VJHQuJiJCXcoiW+cksoUsGxKXEOmKngMz7YryG+FbyuBjo6dtFshzYTFiVM3rG08SZ03Wq6bPvM8Xxo+fUeNkiL98/I+UAIeQiWAaSHXphPod58Zvt2G4ejLYonTsZgbfFWQQWgh4G35IaNcPM+1jItC2E0ooFFwciq/EvdsY2q246QEQhqnNG/mSDcBQsCE1Kx/rav1u9TvsmbqWlEIeiuXbjhVNFXyh8LwdH1GNY7lzNDyVd6wQM7hiPN0XL7CzLVtlbNtG2ALWVuvtGmSrRY1jX3JEFoD8mVtLcxYWdQ0FudKiWtlISz1yNwYmQP16ybtz5DXz4jOv2F5lQBoFfOyIX0y0QAONeYnCYx+1V+S9k0zRzpfpoGmz3aBAMizT04g7g3XQSUhIgjUUaD0Ip9R9xGHutZNZlInM0t/8hLztHQFdvz8TR15WoiTlEgGPKxW4a9CXKSQSoUzloAFZqrQVrlfSYraA6pmWZi8uVaEnCgt30kgoqsdv0GdP4aQLu4MRtYhGXBsT3V9oHW/AJVxEzfWcn2VRLohVzLSNyWSMjJHLQFQbUPaNM1yUD4FzYdGjvZ5p7a7+m8DLbn2Y277JWubrvPU8sSojZWle0+p0fd5807YvaZTs2lLq9w7BNkDP9R8Ol97KJCdcdjL07a928Ym7YVa+b7pUw8+drP9kAL9Ln1W+lErlkc34++9/1P8cnpAIYs3YUQsDh/igKfY48vDIy454OAivp2OeNef8e10BAC86894jD3m4nH0M74dj3joRjzFHp3LOPoZD37CJQf8+ekt3vVnzMUjZlchhV/0FxQy+OvLPe7CjFQs1zUd8ZPhhEIGH+YDepdWsSw7m3FJAZcU8Ht3H5GKxTl16FxGZ9mvsrMZ30xHxMLK78FHpGIxprCyZh18hLcFg4v4eryreS8p4G/dfcDHOOBx7vGT4YTHecCUPe67CZ1NSOTw1+c7DKKQH3zEKXYgAMEWxGJxH+YaqgQATrFD7xN6lxCLQyoWd2Hm63JNrbiXGND7hMeJxzQ4DuUxJo9DiLjEgId+wtPcIdiCSww4hIjTHBBcwfvDGd9cjnjoJ5xjQMwOb4cRvUv46nSPwSfEYmsYDWeoMv1qCJnHqQeRqYcOhUytzxrCeeqQisXfef8NvMn4xXfv0fmMQ4gYk4czhPMcKhzXGMKbYar9A1APJXKxSzlb8Kabqgydy5izwzFEPE4d7jrupzVAEiZZZwkP/YRgM74bmaW2kMEUPd4eRszZ4aGf8M35AGcJg08Yk0fKFt6xQupswePYI7iMuy7icWTf0T4k/N7dIz7MA747HxB8Rucyn5wVKwcqVPuZMh/EBFcQs63wVWYR9nxPwtA8jT2O/QxnCKepw9vDiA+XAX1IwljM9TlbcOgizlOHoYuYoocxhCmGCvUNjpmLg+Pn5TIH3A8TnsYebw4jzlPHLMvZog+pMhwfuggn7McKIX93vOA0B4xzqIdF3hX0PuHjyOuiFAPvCrwwMI8ChS5SJwCcpw7GEDqfMEV+/rwtlelYD6R+cn+GswW//PBQD4n0gEjDpVhLuB8mOEP47nyoz7KXsVSWZCKDh8OIbx+P+MmbE747HeocAMC7+zNidkjZ8nuni/h4Hirz8P1hQiHgdOnRdRz6SA+5AODnb57w1cd7pOTQ9xG99MEawsMw4eunIzQMz2UMuDvMuEwBhz6uDryUWdnZgmn2OA4z5uTR+YTL1KGTNdBazPsuofMZp7GDl0OjLORr3i+M0TGybMFlXGQOUnI4DjMKGYxTwHFguUrhAzsdO2V+/umbE05TV+fg0M+wBvjwNFRG6CRr/83diO8+HhG6hLthxnePB3zxcMHH01CZpg8HPoDyjsuOc6hzpvINfQQBiLK++5BwHrvqv1+KQckWoUtIyeH+OPLhorhhTNEjyeEQAAx9rO/flGyFpw89r0+SPtwNM05jJ4eMbtW/vo84n3u8e3NGLvxOWQ4MeZ3OySFGh2GImGePvkvMXu5KXWfjpUPXJzwcRnw4HeA9v9eDKzhNXf2dm+TgL84eg4zZNHk4R+gC/8Y9nQaELtVxC4EPc8cxCBu4g/cZ0xR4b04GXpnNJ49Dw+BdCq/DXsZkHAP6PuFy6XA8zJgmX9fW/XHEh493n7rdqKnvI6ZLYHlCFhZtfranS4ALmbkW+lQPNvUQcjjEyuHQhYycmQmeskEv97KsR4APJ7OULblxvgQfut3dTxgvXWVn7/qEFHk8dP1YQ8w2DsB3GWl2fOhZLEpZDkpTdAtbtxxy5mT5sLRwH1wocqAMVvot1+184d+S2YEICEPifhEfWJdsUKKDGxLSxYtSSPUg0wpzth56grAcIgJ80CNlKNnlwEwOBSv8PC+HmKuUbMOFQMshkx56GlrKNQfFdbz1kG17qLb9rOcVLTeBsmU3yrDJpkKqn4VqE5aDH1oOgionQTHrwyM5NGjLa53Q8am+HRvZqcnX/m0OJFeJzErZ3Q5X+/1qvNrx3VZtdq7tybaXd6/sb2T69ZH3GGN+AeARDI5MRPSHP3gb9L3waL/e9Lf/+bf07/3X/wqCyRhLwEQemSycKTjaGb+K93WT2NuEp9zjaGdEcrjkDt5mOBRcSocHP+KUehzcjKl4TMVjLh4WhPfdCd/FI3qbqp+mKnwFBndSpv3+MfGG/uAiUnErR91MBsEUeJvxIR5qvin76pM5Z4ejjwiWw5XMxcObUr+rz+gs7bJSyz+qsyh3T7FHZxM6l1lxtazgztlx3wzh6Oe6EU/FYZA6UrHwtmBMYeU7OriIIvJoXM65OHQ2Yy6ubrjVT1TLjDlU30JvCubi6vXOZo4jahPGHDC4iEQOp9jh4Hlcgs0cNiX5qhzr9dZnkcggka3XepdqSBmNpXlJoW6AVOn/+nJEIYM33VTHEODNTO/Sav7OolB2jRKrYz4lX+uOMg5BxsSLsq5KpjPLD7+O75g8cjEYAh9DG/Cm7zR3rKREX++pj2r9zRDFefC8mZ+zQycb4FQsTlOHzicMgZVs3aw6CTGjPqPOluoDW+OSiuLXWvYVctuHhDk5lGIRfMYUfUULALhSWDufqzLNdS+QaoVC63dFG/QhYYy+hsNxdgnBA7ByrrLrpmyKAc4WdD6jEFvQNY6nKrCqiCms3NslDmgS63rnRTGTOWz9llfhh6IHEXAcZuRi+ZBaEBGtT7MiC1Th03UGoCrYAIfW6buEcQroZAOu46JkYlomJVfrM4YqOiGEBWmh4XEAYBo79MPMhGPZImezhOOJvm74c7bwntEO3hcJ/bNYuzVPKabeV1/DFsnAcmr9vHacz1dIA1UQK1og8zhWf/IGVeF8FjREqRtLUn9WITmLo4cNgrAAb8YpW/g+reK5EoA8OfhelL9o4fqMPLlKaAZDjACgxq/SNf7YxPIVGQO1ummInWqVtOtwOdUaq5tHtejqpi3ZxvrBuzYu1+zOLNgiGEpjKV7aQ7IwXWYLOLDcby3lFktoGUfVUl8tXABbpJMVi2CpVntDTXgZlae2bVbXTBILY7eUr2RhhFUYmpUFGFjVb2o4D1osm7rx91StpUZD0sim2s5MLvbZKTHhmI6NmszMVvZkFpIWtaQK+qWG7lCrm0G1PK7CjTRithZOHQM7LeF2mFzHVAtkLdsoWlXxECI9mCUsDTnakP6YhTxGZdQwJVjkW+VpLZeaT9YWGawstQZNP5uQJ9vQJdWSSajkOi3BjqFFESPb9H1Xu1nmaqucrCx+zf1tH68SXZdpFZxb4UZuWue0PjTtviK9ZHG9NRw3+4IXZMQ6725dL8hbs77GSvmKR/Y1Fs9/+B//+//rPw5l6lPSP/UvHOk/+m/+mR+83n/rn/zfX+ybKJZ/SES/+sEFkPSjtlg6ZLzzJ0wlABZwVBDJIZiMTBbv/BkFBpEcMlm89RdksgjICP7CCgc5vHcnTMXjIDR4wfAprPUjovhl3vkJTn7VgsnwhtsBgKmEWhZgoqB7P8PL/bB5M7RKyvvuXP0PDy4uvohhyZ+KwyFcUOTt5A1QiDf/27oBoLesCHU2cexNEA6OFcZYHLqQIEE9UIitXTygqdbhxWJ0H6aVn2KBgQVhaPIOeg2y4fbrp7qtRxXATsq/cSN/l3xHLxYBZLzrmc9elVlrGM55FyYZhwIr/VdFWz9vk29i/HVdrv3QufjZ4bSal0D5po9m3/Rd79+5XBXVbd9bAiPN37tFqQBQN6cHH6ts7Zi/Gy4o4Jipmlr5t/JYQzhI3kIGPYC7br6Zt/1e6xcr78pCLnUeQlwpor1b8E1VxrCqTurne7Zbj21LdrQd90OIsmbWc9I1bXb++hBUFfAtedRWJmuWfG37W1/XWyQ7vZQ9dLGZ57JL5AQwa/XWh3ab1ArLiu21p7PGuK396OKKzKrzzTrcecv3IS0yBFRlnm/GVT/18AAASBgdV/60jXx9WK+j7TwDkPfPy+nWQTR1Tfluvy6VjRXk7T2srun30jVjcpA6tvV3S937JFXgBfGa1K9lveWjfIvc6opEqdiqUK92kob4niFmYN7M7VUfBlQL4e6OVBRrXX/qYlBlazbG9TuwbOA3752VHISViwM2Va3q2pKKqcKvirfO62ac8p5F61NSMy61Fj0HaMaAZByoGY/KsL4ddzl0urLIvJDK0OyijRCtbd8tt7SNVR5gV2nSe9rehgxJLW5XfrZ78u/1izZ/d9uXeVbFrV1n8ncJQ4Il761KaVkve2RHN/twY26eGdVXz+Nr87+ofH1Km8/ku1Ykb/fyWZk+pf8/RN+/rwy/Sz9I+lErlhkWT3nAh3xALA5BuMs/pAOm4nHvJpxLh2AyIjkc7Yyn3KOQxcHNiIUtd51NeIwD7vyES+4QbIYFobdsbfwmHXHvOb9a1ubiq7L2lDr0dlEsPsYBb8KIqXhccqjKXZsiWaTicOfnankEWFkCgGAznlLH5ENk4ImJiJQkSJUKVa56m/EoVlKFgaoldS4Ob8OID3EAAHSW64/F4Sn28DJuas0ElCnXoROLXyUlyh7WlJVF0tuCMfsrBSUVB2+zWCWXDZqOefvXG65DrXneFhz9jHPqan9U8fS24OM8iHW0WcJk6/hZU1DI4pJYu2nZZ1ur6pTZ4vuT4cRrZz7AmVLl0DytcqjKo/rrpqZdJhUSyJ/mKw4WhEgWweZqaW2tompN7V1Cb0uVu5BBzI4VTjK1PM9jFqipreUBYEoewWV0krcQQ0zvwow5O5xjV2Gz2q/cWG/UUqlz3H4P4qesn1Ox1TKqFluFOWsbWSyFRsZeLbla5yzW5CTW3JaZORaL3mXEYiv0F2DrbHC5yu3Vyip9KWRw182I2XH7sgbUgjwJhFatvr6x0ipxVRDFVee5hYTXfhWD4ApyMTiKQvZx7GENIxOcZRjhWjlPdZ62rLDKiAyw4nyaOtz1My7RrzbgqsSqPIPPuAjcMwu8V2GVfEiEFZT2vp/xNLFvuheIsG50O59wmUP1hY7ZVaisWpu1XQLqOLXWaO/Yb9027LV6aBZchrOEKJC/ll25hbVnebcYQ0jZsS6xmRddj5rad5UxhGMfkbKtiAUtq/BrYCFGU3i2c7we1PI+xeUdo3712pbCodXqqtBqaup1tiCKnzk/73zPuSKW3uXdaCDW94YxmOHCpr4r9K/68VdIt4+V5IvXSqkWZRcih9U6zCuf/i37s1qoFY68ZZdWy7VClW0DA1WotdYPoFquAfZ3hyF48ceOs6/WbIVMAmylVv9xYwklL2ut/sYUC9cnhmgS+4RbW2DFKq/+/SU7WMdwayv9cD4jzp+/9XGuVGipaaypCjk1Fit/dR4HSNsLlFTzqbKtIceoOSuuedAopU2yErrLGBkv17Agt9lVCVffdoF8Vjm1/kYz22U5bpmUjbBJy/UKRQVgPK2U6Ao5VTjpVj6td2tFp8VkV3Wbvf61aUeRqJBRnY9WOVboq1okNyG2rqxzK+VzI8TeQUxzCNJefjZtT9W21/byG6zHZXcc8Py47Sr/t086fjBF9yWlEvtD8mNOebt2fpj0U2PM/9J8//tE9Pc3eQjA/2B4M/Cf7tz/3ulHrVhG8vh/Lj/DT7onZFj88vIGAPC+O+FdOONPLu/R2YRfpXu87074v08/w8/7J0Sy+LPHn+M+sD/bV+M9fv/wAX81vsHbMOJjHPjfNKB3Cf/s21/iz85f4OgjrCn4erpDZzP+4vQWBQZ/cPcdvhbfzVPs8eXxI/7448/gTMEX/QXn1FX4nMI578KE+zDhj779Et6WCu2Mmf0Wp+Txs7sneNn8PM49epdw9DNaptvHuUcuFqc54Mt7Dplxjh2OYcYfffgSx37G237EH3/9M3xxvKCzGR/nHnNycJbw87snPE49vCn4OPd42488tqKo//XlDrP4DBYyeDuMOMdQr3Uu43Hs8TBMuMQAL5t9IlP9Dd8fzvg43df7g0/4ar7HQz/hLy8PuOti9R/8q6cj3g4jHscevzi/x0/vT/j2fEAfEo4h4s8+vEWMHj95ODHsVEiYtN0p+urTZS3hzYH7o/IaQ3gae4GDGrwZJgSX8b/9yd8GEfB7P/2AOXmcpyAQSoNjP6829X9+fguAN6Lqh6awz3EO6ENCJoPzuYe1BcPAG77gM8YpYOgj5ugxdLFunhVqeTn3yLPF8c1Y4YvBZfzyuwc4VzBdAo73E5MyRQcfMrxnfyT137o/Toh5YP+iI1t3pylg/tDDHhKO9xPm2dfNn1qCAdQYrc4xrLFkC+sWsqZ58gsJkED5+iFiGgP7CR0jxsce/f3Em1xRTtRXKM0e3RAxjwG+S6Bi0fWxtqswSYVQdn3CdA7oDhHzuYPvE4zIN88LSVacfSU3cmKB/+uPX8AOCaFP4udkkaMFzQ7+LsLYAue4zZyYHMd1uc51mgROOnD5PDu+D/YFsoGfwhwtXCj45k+/gCkG/qcXVgwApMkhHCKgylM2+ObEhwb2Lta2dCxtWMifylOAfzvju794A3sfhYCKN4PffgxCggOGJ44O9iEum/GLBwrg7hPyxTFsLmTeVBrCh6/fgd5H2FD4frSVCMeeLcq7yBvK0cHdJzw+Bo4pe/EcG7bx06HJMdFGX5i4yBPDKQ8ZGJ2QJoFJgQjA6BgmeMgLgY+TDVg0C4yxK8DFcczXQ+afQ08wJyZHomOGOTnQUBZopxJtCZHN9LVHuiNQX9gadmEYZ36basxUhVpOHxzSuwQzWbizRfoiYf7OIz3kutF0Tws8FASUQchokkAhA8GdLO+LO45v6SaDdFc4bqUQ1FAguDPHWfUnhroSeKOWe6lTNqNZYpaSgZCnAOSAfJaNiWWylHLmuKemAKWnGrOzBAJdDNKbgvire5Aj5B5LnEJi0pkSOK85G75/EQvizPdggXJHyCNgzgZ0T6BogAhQBjCAWXgNgF7glj0BKufARDNGYoHijcToFBntxPFk6Y4qxJPXCVU4oZ0FxjgQ3KPh+JSeKizUTdyfNHA8znRHHJ+0W4h1whOA959vxnAXA3Pk9epEZl4T3B7LvsTZBFDjXvoTXyuB5SOn8wq4C+fR+KcghscqAEfPUFV3IQt0H4F0FN3NcQzaElDjQRpiuXLHZf0FyP0yFmQlXilxDM4KB8WSR+NtFukTWiiqxC9VwFbupJ/nRV6O5cv5/AVIB27PFFTIrJ2l72G5ViGvBJgaL5VQvOE4pUnqyYDNEkfTmjVpkmjlxcvaauK7GjFla+xHchILUv5WCylk7TVxSbkCbQML+ZAS9SnUGQY2L2OxzV/jcGuVetBQoeSosVtb8qI25qbm19jKdfxa/VYhxLoubsFrpZ0t2dHWWmwKx9JsodPbZDaQCo2puYqtqePYtL2r4N6S6Yb8z1n+/9GNrv+WpF+9Aub7rxLRnxtjfg7gfzTG/F9E9D/9kEL8qBVLXTQf0gFJrD8FBqfcYyq++iYWMjilHtYQTrkTghn1g2Oo63fxIPm6ahUcfETnMr6Z75DIYcyLlW4uTEZiahm2ShlDGHOAE6ueWgAXKx6fLs7F4ykCvU+LnxaoKj++sY4AqKetY17jC1XxvBMLxuKrZnHoohCs2AplKzAItsAGJuqPwoxbDFsJYlnkjYWtBJ1P9RlNQpbTQu2UpMWL1Urj9AFsnVALm94nseSw1YnnQcN4OLGQGAC9QiA9k4MksVI5OSmtBAeNX5qSauiJf2u9APhkWP3rnGWLWCoWw2FmxaxabfLK2lLI8G9AYylorVD6OQjM0YFJHoxhK4zx3PdOCDFU+XFNHdYQQpfgvPq7LVZIL2y4oWe4npKF6Ck/s/ey0qFroOuWefM+I9/HWo6JMhY2YFctVqVaIZQRuPWn8yFX2ZXtWK+T52t+SLVOzkdiHbJAx/d8l+AcgWyGMVgsH1LGtHX3Ul+Xq5KrfdY8zmdQaZVgwB7SxupS4AJQHOev4+4WH1O1eBgDuI5/vdjyQLAd6n0EVIuXDSLvUdeFWFYMwXVszSIQky06gA6JFW5HIPBzaCzvAq2RwOIA6Ji43kOuvnzVenTMi9+U+BVZSwvzaM+KmDEE24tcjuqOojxktpAYYvZZtxBpFEdyD8AgUch7loH6vNzT1LHjlXGFYZSWQI5JOGjIlXTDWO5r6Q3IG1Y0G1IO3gyJhcSK9WUozOopSqmxBDrIvPjCn4VlF47LwagMhPS2sD+cMKtWw0Eoy8ZLfT/fAPAEQkHyBHhipTJQlS/fS5nWl9AS4Pm5gyWke1GSRX+ijv0RsxelV+s6Fs4veStUtPpYym+GBJ1XX0KeTHAweixyxMCMp0bYQSkY2XQSomd21PmLImVptfmqcSAdED3fL51ZNsuyxqgjkDcoPSvAJgvjpzCGKtEHb2AJxQNFfQA9kAsrMCBWfktHdWOce1GQAsH00teChZ0WfE/rUuVKZTMdkA+SzwP5yG2XsGyoyXD7+Xv4WJZu6VMJMn6yJkoATMeKbh5Q50vXWgkqM8tKhj8TuP/AolQYwmJFa67XZAAKhsdEx3Awi6+kugaXRSHS+8qSC4Mazmu3nAGs+GOSJeRB1+JaCUqNjyQM56vGz0apyT0qM64ql2TwvI8l+D6XMXX8KmssYRkk/Ujtl+01Hd+mjNTLzwH/vVJcnrP2aZ0N62pV3mgzp42CdpU2ypReaw2hVbFq10b7fauwNfXeVASvxuZ5xXIR5lq+2s/dTrYvsJ3bz6VX5F8prr/BiWB+beFGiOjP5e9Xxph/AOBfAvA7xVLTWDz+anzAh/mAVGz1W7ukgDH5CoU7hIinucMxRJxmZlA7hoivkq+EHL+Y3+PQRYzRIzi2xPU+4XH2+MW373DXz0jZVZgZkalK4V9+fFMJQjqf8Zcf3+DY8xHeeXqLrvGpqtYQgWepRW2MHs4u8ERnCL86H5FEMWXWUVOJdjSpr1XvE/7y4xsQ2IdrSg7vjxecY8B3lwEPw4QPl6EqZwzfs/jm6ViZGYPL+PbEv8xKoNH5XFkBAeA7YUBsrWydT8ycahYYWyFTIXHTzIx8FT6YLUJI+FaYBzVcizIDfj0zxOo4zPjqwz36LuE0dhyeRRglv/54V1n+NHRKq3A5Ua6/fTxWy5/CybouVZbJ72a+//4LtvZ+/d09nC8IIWEc2eL4dBpWb9helLvThXcCatVTZsKpIWcBgIsQrSjM7GnqEULG6dJVuJFCj/qeWX7PY1fXS04OXR8xz8w4ebl0jdXO1fKsnAGPTx2sLQhdwunMMjpX8PbhjCkGXM59Ddui5C0a3sRYQkpgCJmSoojCbcA+ayk6WFfq+E9TgPcZzhHmKaAfIsYxVMUyJ4tYfFVA1dKYIsMTL9PiIFmVRs+Ww2kMCF3CNIbKeJgELmYdIc1c0IUMYwtKtkgSCqAfIlJ0mC7CymoB6zK6QJgnniPK7BNsFXIZOdwQEarymWYliymIE/fDOEIW66r1BWn2ODyMMAY4P/ZLf3xBvGiMAVb0wsAK73juFl8wC4AMUmTLZSFTrbT93Yx59PwukPdId+SDEBIoZjhEbkcgbl6stEnIZ6gwOY0yLt795IzzU48yO9guC0uk+FM+jLg89VC2xjx6uCFVIpsS7eLfRwY28K5PCW9Ksqv8VHiNl8jyu54PbdLsahgdFIEHajgbCTFiQ4YdCHl2NUSPOwp51eTgjgl5tjWkj1E4oXwfvjxhngKT7hiCO/CzG88BphMoqVhxwxcT4imIH1y6AAAgAElEQVTA9Bm+T4hPHcK7icdVNlb2PtbnxMjaJiUoshzGxop8JTKJD4+hg/FNe9nAHBMoWbiHaYEUGkKZHRPzCDzQ3IkVnRZZUQzsQ15gksnCdhk0i4V4toAqT9nAvMmgs4f7PWYy13w61hrSh6LlvLODHVhehZKiGLZYv8kwXQJdAocZchxKKs++hnUqstZotijvpN+zZXbPjjWCcgpL6J1suC5LKKNnoqFkax08oGYVRofe8fhVJlBfQHKIUkaP0mfQxaEccm0bxcAfEvLHDp+bci+WeIAPLNR6bwmY3AL37PMim55o9Jk/R8sW+QKBhhr+DixkR3JYUg8xksVK2yGDdEwLIVOROlLTHrAOedSVhVG1DZ1kwGF19IBByykRFJmFsInMomQYySOHCpWASYmZtP5s2PrcFZhJ5tNiOUwRRV1D2BgyzFlJGxkNOBxPMhVFQIKIUCjtlcJFqKRBRsmHmoOcRbldlO7WB1eVttbaWW9IqoqVPrNlUdy17poR2LC+Xk3rosgRFuIjau43Vst6AOOa6215s+Qx8n1P7VKdtG51mu9X8Fq9uVVibyjGTdevhu9Zy+Nz31s5Vp34zVYqNe1xgfzjTsaYOwCWiB7l878B4D/4odv5USuW3pTqc6dQ00IGd2FCIgdvcmUvPQjbaWj8CXuXqoXqECIsqBKPOMOKU2czfnJ3XrWhSX3Q3h0vK8WxP14qdLQlNWlT8YsfF5GBk3AJaqW0hsQKeU1O0T42RRQ5a6gqs85Qtdxp+A0lF2n98Ywh3B+mVX+CX6CAalkEUMt4tZSJ4qbfAVZw2wWlMrhBywCFMsgbOEsNw6fEQPQaamKxCB7Eb811pfrDFUK9rqf2hUrtQ5UXHNpgO3bOEIxYcFX5iYktt/3A9RoAQfLYIa7qqNZJgY8iLL5ObEXMG1+oxbqo1kJjlvIteUcW/6rWUqjKshdorsJW9dqWBCg0JCRVRrBfGRGYIt/Qql31FVICDmsX0hP1Tak+WeoLVXjjrGOoFr/FP2vJ76xakk21jqp1Un2wql8PLWOiyp0eDBhhwYXlX1onCpmW48MFlk+hvrV+7gxyXhRYNOPM/W7GQcfHUf1Ns01dfJ3vWV+q35b1tPq1tKFpX+Tiwwj9xUXNbxXuBvZZM5592YwFWvxSir7uAPRwgJUqAJ4WvzOFn5pGdoMaogBSpsZkJWaMXVhWWVmryl6RMavjKe899cXSerL6xzVrU+a7ZFNDJ1Cz4TK60dPNmRxq5HaTbDgObPvZGN0s8ngQoW4Mx1O3+HqJom1ko1WVtHrP1+tpDIABX2s2MWV2K1nU34vIVN+xMrllswcgZ8d54nojod/TRSZdN0+yQa5WusmtzQGyM8zn9c93SmbJR2CYqtZx4X7EU1gUgyoH7+F5o2gktAqQT+LPbPyyiSQDujjE0QHFoEwORS2pxSA3pguKIse56V/kAwEWCkByVRSaGUKMAlDiNnkTvfS9sv4WAzp5tHE9kRzKbPldJf03xTCLbTO2abasmHxuyl78/7D4CwJ1zgWiIKZfFVzaa+YE0dUxNQTQ3MyfpmY8rzbgBNDULToQATTZ5y1So12vMyyfl/pNFXllsQIALPGVd5UJlV8gzm0lhgCazCoO5qquVT2tJIC+cYzeWykajRZEqyG7lv9KCWnaIoi1a/nbpkXhu147y2vE3BzHVY9eq/t8Dx3ppsKmYtyqe28+NnWu+7RT/w+cdtv6Xfqc9CWAf2B4c+cB/FdE9N//0I38qBXL9/6Ef/fn/zP+Kn2BsQQ8uAvG8v+x9y49tixZmtBnL3/sR8Q55z6yMquyKlutpkqiWt1CDBBDBL+AYUuICRJjxAghkJghhkwYMaNntBDiFyAYAIIhNCWgaJLKm3nvuefEY+/tD3sxWLaWm3vsiBM3b1Zl3iJNCsXe7ub2ctvutmx96/scxtzgFDu8syfM2cIg4TF1eGMuuIsUC3nUIx5Thyk5HMyI9/6IW3vBJbZwOsCpiDfmgpg1/nL6Arf2Ap8skQCZCT5Z7AwZZe/9EZ0m44NlTm7tAKci7mMPp6KQ9zBr7SU18Nngc/eIlDXG5EQmJUKhUwEfwp4gvdA4mpEkVdJCJONURFtIcY5mxK88xZh22uMSG/xh+xH3YYdTbPFl84D3/oiQNHrjRXblF9MteuNhkESSJWWNVnv4bNAbL+RHBgkPoUerPVodhDG31QFTsitSnJQVpuTQao9TbKUOn40QK51iixs74hxbWBWL3IvHUOC+n7cnvJ8OeOMuGBJBmG/sgN54/Gq6EQIiakuB6JbxMEiI0Hj0nTDkMhHS3k44h1aIlkLW+NPDr6CR8X8Pn8GqVIicqB3nQorErLNvCkMvn/dZw6kkUjN8/F1DbMOPoRMCIu4fE0ZplYX8R6uEN25AqwM+zHuJoz2HBm8b0lE92hEf5r0QNoXSDmYABoBH36LREUc34r5AvDvj8ePuAXe+x8d5h854IZxKUNJmJlNKWQkhUcIir8GyOFZHtDpiSgaX0Aib75wsbt0gdfCYsxzL3s549K0QM2mVhaAJgLAFMzHRJTQ4uAkn3+Jte8E5NJijRQL1qf7Msi8s7/NFe8I5Njh5gsE3OqDREa0JuJ87hDJ/WYYHWMipQiY5HADSzs741WeW0OF6f9w/wOqIf3Z6JxtRfA3NH9pMOdgJVkfczbuFUKr0f4xW2nLjRnwzHvDj3T0+THshoAGAz7szxkLMxG19mDvZ8Dk2hIS4n3shfpqihSn1/Hj3gK8uN8SSrUm3d4wOViXcNAO+Ho6IWWNvZ9zPHW6bEefQiEZuvcHGkj5DcLhtBpx9S5q4oZHNu5gW4rG9m9HogPu5F7mgWOYJa+G6ovXbW4/OeNzPPaxKIjUEAKe5xaGZMAQnG2Z8PX//8e4Bd3MvBFy99bAq4sO4FxkhHtfbZsT7YY/eeRzdiK8vR3zen3E39fJc47r5Hg3BCYSfyat2jmKyec731uNh6iSPjwYxE8PzFCxuSlw7jymXyc/RQzPJ+IieZdLSFq5r72ac5hZGJ4zBwmlCiVBM/4z7qcOX+xPmaMpvOEnbnY4YgxPt4rNvsHczQeoLkVnOCvdTh30z4+gm3E19CU0gBMsQnDyTuf6Ld8JGzbrAzCx9P3ZC+sUkXUplDN6hKYRdrG/MKB8O6Zi8xa4hbWcm0Wosxe8rlXEuCKWHscWhnanMwnJ9bCd8e97h101943EaW2QQ87SEK2jSEG5dgA8Gu3bGWEi3+L7tWpIi8kVCidvOoSsA6SIDZZNMM7kVBL3E9YWkcduPuMxOvnelbkYO8fOCCa66ItvExGupII6oXiuoBQCCVGKGeCamqkNudLkvjY2iw5uzQlfuDbc3Jo0QNFxBrHDisAVraQOMSKXoXGRPbhmLmtyJY/L5HFsemSWyeIO51BMLwRbrbwJYoP+xbJrK5ph+suEoxEgrHctqwyOvyxTmaZWlTMmjICijp67IvC6bjbyttcwkRzXxkMZTo3BrSHL+bT3YXFv/ryHA2/K2Fvxr0mpT4IW6X7rmNQby73DKwG8FCptz/r8A/IO/7np+0IblN/6I//T//VdlITgVxkj20p0L+yXDXZktE4C8+AAI4+RWgzCUBcfOeUyV7mAsi+y51NdaImvhF0hjg5yryWw4pQyBvY7lpWALsyTH81FsYfGYKYjuXy0pACz6fbF6QYgG5exWDIeuxPmRqDxd37pAcZ9YdBGB5XdMbccCDzRJhI8BiLdEPErspVIL0yBDbfl8fU19LRHGGGEl5BdRLUDuvUFOJC6+1bwjNsDFOwEs3irOC0CIYfiFplTG/zT8CQCgaYMImDN1vy4eNu6bkN6U88KaB6zYB1OB47GGnlZ5xfbIDIoSNwcgFRIZ8XKVl3csWnY5lnN5LVK9eGUgJC8pKvFS5aiQZoKnEfwQFWvf4iUShtLywOaXLacVo2A5x0yDOZMXL3sN5QgKJayDXF7xrBHMrYxfxUjK8Clum+S1icplUewyFsJoGMtOfIH3AiCPkKZYvFWeqARSx/NGPESmetmzviDDv1J1nl/kgHz+3+Y/ojLaykMZ1XI9747X5crAlkYzOyJKPpfwf/gfU97qJfq/e728aHmBwfVwvVwHQ+j0cv1fTH9EhDYKT1galSfIGnuhBErH0LhqPkieMvZ0Hov2Yb0o4f8JEgco5695LxSVA47fyzTOqkCdhdyFiX+weLdYPPzn448pBo2184qnKjVpgacx8cZcyGYK1C63GV9NX1KMX2k/1y2eEUN1iUdGQeLFpNxUCFn4dqvSvgLn+4Vfb/9TnFo1BkyIU00TKEDzOCgaexWUjNMKDlfGMbmMD9MXNA1Nta5l75umdn0diABFB74WKwKZbyOdY5IZJnHJVR9ZqzBr4L6ARlKpk8lIkluPiZRTCFzE2VU5iU8FzpcMcPYQmCC3U0W6D8lmPEQaj8FT3DDDEx9m9b1iLC88zgAmifNb6p1K/OJYdCypf5R/rnQsx42Opa91LAGZO3Lvq+N83cd5uQ9ZAyEsYxIq44KvvRSCl8T3VAFTFd+Yq2eML3WHKuaRYynB79vy2TOctNgtviKQyQWeaRSQI2DNUgdPe1UeN6p6RtUhxDJHeZ7yTyavx6m2bWovl5Dd1NdhnfeTeRI+aTutPGtVH1f3rozZpzyCV8vkMl4yAF8y3MQgfK79mxjFuo5NO19s47bea21/Jj31kj/9/mx6RZ7/5xXF/D59v/SDNixDIjmJb857hKTxph8Rs8L9fY9pdHhzc8HoLQ7dhPtzj7eHC97fFebYwwUfLz1C1Ogaj68/3GC3mzCMDsYQRPSmMKD+/FdvsT+O8N4iBk1EJVnh0E9QKuOXH24Ecti3M3714QbHPekz3j3s0LReWDXZOHs8O0Rv8Nm7E3JWOA0tnAtwJsIqgua+f9wjBHqKdv2McWwJFleNQdfTrvKxH/Ht/R65ENGczh3+8PM73F16nC8tvnj7iA+PewRvsNtNErN5f7+DcVHYOU8nkiRhg63rZ9m5VAq4nDtYF+AcUeDPs8G+n3EZGzRNgPdGoI7TZNE0Aae7HUwT4cr56A2aLuDy2KLbzxguDbTOGCYD1wWMjy20i7i5GXB/v8P+OGIcGkSvsb8ZsW9nfPPtEbaJmCe7MtSMTfS/GMCXU0uGWhMp9i4D3W7GPDkonTDcd0BQ+MM/+RYA8FdfvYV2Cf1uxjQ6aJMwXxqg2tHsDxOUIjghAKRJi7HW9h7zTN6RN28umIOVmMbgDTGoDg5NFyj2T2fkQPFO2mQcbwbs2hnf3h3A2oJ+tOhvRvjZ4nhzwf39DspkNB0J16dAsEqm258eW+gmot9PGB7pfpom4qc/fY9vzzucHzrYJgqkVlhRQTBZiRszCbYhI5+NuHY/w88Wpg1oWw/vLaYzxQEak3A5tbj57IzHhx6u97QLHYwYh/1hwHAm1lg/kdxAHI3Q5JuW4saco/9+cGgPE6bHFv3tiGlwZMR6BdNHxFlL3JTSCTEYiqcDcPzihHFo4C+0eaNcgm0C2jbgfN8T1DMoqCbBtOQtCZMFDBnFzS0x885nuv92Fwm2qDNMX2INk6I4v8Hi7U/uYU3CN794Q0YYALP3iCcnMULKJezeEVT+8W4nb0ttCWaaJkNENUlh/+6M8/sdjn/wiNPdTgxlALj58SOmmTwM0Rv0+xnnD72woTZvyZs137cwRy9loyyIf/TTD/jlL98AXkPvPVwbEEqs87vbM77++hY5KpiDR3x0cG9Hijk8zhRzGMubPinovaeY04uFu50QZgvXBsynBrYPSJE2IRj66fYergm4PHQSk5mDBhKg24jkDbSLSKOFvpkpJvqxpd/KbODejjQ3Tg3cu5nuTzF8VVOYb71GNgk/+rsf8P7+IPGn7W6GsxEP7/dQJYYwTfT7O/7RBQ9fH6BvA26OA+6+PuL4hyc8fNxJnFzzJXkXXYGTX84d0qyhS6xhmgzaw4ycgXl00Caj62ec7zsiUcqK5o3XNDdmg/2by2rjbRqb5TeRge4wI8YST8ubYkGjvRnB0hreG9wcL7h/2EObCH9pYDoi74peoz9MuHy7w7u/d485GFwuLayjZ5KxkWLDJ4swOOxuRoyXBv1xJJRBO+M8NQhJ43TXozvMeHc84+sPN2g6j77xaGzAx9NuYcQu74np3GB3Q2M2nFsYm3A4UOjIxw8HIeYK3qJpPZyNeHzo0fYefrZwTcB4aiQO0/TUp3B26N8O8LMlpuegYfqA436EUhkPjzvs9yMe73Y4vLngcu7QtBRz/e7tI/7qq7e/9rqjP04Y7spztY/QHHbiIsa7HqYPSLNBfxwxTa7EF9N7fP9mQAga8+jQ7WaEoOWZxUzgfnJC1OWaQM/frJb4bl2MOq/x+ReP+Piwo/s8G7p3I8UUq0KMpU0s0G56d42nFk3vEbxFCgquIy/vfGkoFrVA+U0bkQoTdkq0MdnsZ2EE14UEzQ9OYr794ICk0N1MxPAN4hqYJ4c4GDTHGfNDYSkyWeD7rgsUHz8tcdfZl/hZBagmygZlmg0wLe1UthB5JUXPEd4EAxYSrFkvm2y80ckEVhxz6hURdXm1MG5zSiDW61kvxsvWg1iXyR7F8szPU7GGuW7e1LpmBPJGIaoNutqS4/N1nGwCtZ37Xm0019corwDeDNsk3nhRqdoU1JC416vwaWHBrbRMS59Y0kWO1xEh283E54zl2miuNh2eGMp1ua8wXn/bKUP9dcmN/E4klX8gga7X0t/7+33+j/+rP0PMGj4bzNkgQUMjIUEjZo0ILTBOdj2nrBGhngTPakVQTc4DADFrgY9xikXug7UkGeaaoCQ/axnWnzmP1Ics9XEdvsCnuM3Urqci4wC50p8Tf09ZSRnb6yO0wFKdijIu9RjV5aSsBQZajw2P2XYc6zGpj8WsBDK6jKVaeYLruvhzKGMi7Snlb3Uz67rqxPdp+7mum5lrW0MLI9bg5L7VbeL66nHn+1C3ZVtnXfe1+1YzBzPUbnvuuftd56nHZaUXmSysWmIuQ5l7NJZJxjtt5ms9v5+r79r8fKl9L81bPr9t//aa+vu2foZfAVh50XOZPxxDvap7M28ZAmw3c/a5PjH0i1EPL41DrV35XNoK2Nf5fdF6fWlsuF1Xnx3V3GcNye26AVje0Zz/2ob7tbo4Pvta3XV/tn3b9pnL35ZdX7+CrG3qq5EeACRe0+iMtBl+1j7M1f2pURx0vZZx4XHYjvF2jHJp/7Ob+vW1V+bFNoZaJGp48cnrtoLKqBEmfJ7PsX5lPcbXyqfjkM3QJfY6F9KoBckhGofb+SwuxzKejIbh36Veu4AYtaHrPvLYVd9lbK+4NVgXUha+mmJ6GZEBhScojO+camOgrr6auwytfOJBEpRIZSjw3K/u56uXZVwHl1N/37Rv1XaGYvJnvr7+YW8MBaqP69qUvenLq2GRXP61B0uVVFZrA0Rtrnv2x3VlvPF0qj4xWKpjanvupXtzpd9XbvUL13/P87juIPzOZW3OXW33q8bj+VOv8kCuynr9nHqp7L/4D/+d//kVkhx/renv/P1D/o/+yZ//xsv9N/65/+G33jfgB+6xnLPFL/0txuwwJY53o8UyG3g+G4lPAfDEiOIX15TsKpaQjS/OA6wXxEEkS9ZGplVRpEeAp4txTrURVct8UD1JDJjtYvGagfRcXdsdkWtGEieO87u2QH/JWNp+5nau6133g+OVtuUv/2lctuNcj81r07ZdWuUnBh63AwAefLfKe21h/Jwx81yft/meM2iuGTxb44bzXSt/295r7a8NoOfyXFvsvzZtx+27jNWnjCGt8ou7fNu13tb4uWZ4AYsRdO341vh5aSzEcH3FPXjOuHzOiLhW77T5vjWU6j5sPz9XF39OzxjlL43BcxsBz93LJ/MOEGbp7Xpc1nZX+rNtz7ZfqEIRruevvxlskw9Pj12r79oYsZyPr+D6r0mvHfNtCk+53qpC2RBTT49f6ccqS/kfKy66VI/LlQX8k74m8/z3apEaF0uJ6gSXd+25r64YT9VvmGGeadOf9LrxfC49dx9zwSBfbevqGnX9c/6E7fDiPLgyFs/WX9Ur9+5T5W/KumpsfKINqDxYV9uEp+euGbBS1pX812t9xmB94dqX6npVncv550b11bPwuxhhVd5n633WEP/ux180EH9DxuAn2/Ha9H2v/w2m9FuSG/mbSD9owzJmjffhKB5LJpMBlgV0hH5inABEuAJAjDqtEgaQcZryQjQRsxKSEleRmfA5LoMNRar7qSG0alfW4nGb0xKQvzLgshKvVe292aZrniw+Rh4qIoWpP3P/tMovGsIaWcbhJUPhU160mlH3mjcNeLqQ4vvLbagNQvboPef12y7sr41tfR5YyA0kxnS7OC2pjoG95lWs28Tz49oO4rWdxIzFm1V7SRSASylrFQf7jKHAniLOy8d91NBqOb+9/jkjs17Ic3xvbcQYnSU+mD9vPUJ8zbX7vK2v/lzHoXJ86rW21WzKfIxjh+sxYO+cuTJvtvckcxu4/9X51XqmtENYaM16Dj2ZP2VRq/XTe7f1SLGXqWYJrsvga1LSBYK+eKjq8rafWfqmLq/2fHEfrsUxq83E3XrGrhlE2zHYlnNtE0O8cmIQcQGLBywnLZ4wPlenGLVIgwDlvlXzqr6n4vEDGR66sNoSIy8ZTHn9kwfUUw9TTpvfJZOC1HNeLfmeGDlqY6ReM4IUFk8T15Mh5CFbN4m0ocRqrwKZ2Bjg4zUhCIAV4YcuxxnqVxsS9U4Of65gfSsvU1YLbFA6UI7Xccb8ndM1T1uduM3cTv5fPXRVXGIkf61Ux1dXBpbKBTLI37few2vtA657AznVzdw+fLgv1fiopFYMycBmKpR4xzp2sy5v9VPb/ldYYmirfLk6vvLMrjJU96w6LtPwWt/rMagMtLzNmq8c3N7e6sInr53tHFTrY0sbr/TpGePzyTg+165r6dqL6Era/oRfNvBeV+Y1T+XVsq/95n+NdHW8nkvf0yh8leH6N5RyxgrJ+Lct/aANy0tq8H9evsA34wE+Gty2A0IiJtCLd3jbDcIseJpb3LQj7kbSaXzTDTj7hnQurcfHscexnXDxpMe4dzP2bsIYHX51OuCmmzBHAx8NGhORAeyKRMn7yw6dI/bDnfO4Gzq86SnG8jS1aGwQbccMCFueDwafH84AgNPcwOkkxENOR3wceyH+6Z3HHA3mYFaLlr6w0vXW48OwQ8p0bPAWPzqc8GHa4Tw7vNsNeD+1mIPBrvFwhVTobujQlnjL1gVcpgYZEIbXXePls9EJp7FFY6Mw7Pmopb7GRiE8Aoi5r3WkQdnYCGej6Hd2LmCYnTDnGZUxeYuu8RimBtZG3PQj7i89Dt2EYXYIUePYT+hswFf3N2hsEPZENha0TjB6gcmexpZ0/mwU46NvPKZAsarTbJGixk+/+AgA+MXHW1gb0bmAKRBh0zi7lVbmvqO41vuBvJtMLpQz0DUec6BYmGM3YQ4Wo7fCnsf9a13A5O3KYNI649BNaEzExwvN05SIaW/XzQhJY9/OuD/30JpkWXw0QoCkixTHMDpYm9A1Hg9naqNzEZ8fzjhNLe5OPZyLYhiHoi0KLFC3pUz6zoZF13iMlZZpTBqni0PbehiV8XhpcNxNeLy0EnccKxa/xgVMsy3xuNR/P1uC1aVFSoUNm2lo0HUe49Cg72dMkxP9Ro4HBVAkVoApkJQHABz2IyZvcb60ZGDaCGsTGhtwupBOY8pkRNXkUGwotEUL8lKkOVyJFVIqw9hEsahZSazazfECozM+3B3IICjjTnHAAMr87DtP5GKXVt52HIMdA+mHAkDfzzidOtwcB5wvrRhJAHDcj/DRyPxomoDh0opB0naFifPSwBVCKiqbrn97e8bH+z1SVLBNLCRZFEd9sx9x97ArfQsYLw26EnPctAF+toshlYkgS+kMPxvRGuWxso7JsIyQWbkmwLmEcXRCgBUTEYoZGxHL/PKeYjWtC5jGEuccFZouIGfATxZNFzBPVow0bWjcORbxze0Zl7ElKRiV0ZZ49/NjR3MtLyRbu/2E80MH4xL6/YTzY4f9ccTl3ArxU9N7mZ9KAfNk5b7osrnQdF7upVIZrqe4ao7NTyUWzLYBMVD8Yw2/5bhBvtdN78XAZtbMnIBm7+UZkSLF1g9DIazzBqb8llIkTdTp3ODw7oIYtfzmmKTMmCyx1k2Jo26LzBI/t1PSmC4OdhfQdx7nSwtbflNGJ4yTk/h6P9NvO3oN15Eb1U+kc9kUqajh1MI4+t3FoGEcaevOI40VPzPibMQ4ZlKzNBm440yxqUEhR9LxdM1Sl20i5otDs/OkmWroWdbvZpzv+qtritck2wWKtc6AapLcJ60zwmChm6Ll2sYlvrjMH9vTMyXOdJ7OlfnQB5q7hZhLqSyx5+DjWDYPclRwhxl+LDI5QUN3AdlrIkljYjOdJb5Zd7HEcVMsco4KqtyDNBuACYcUxS7moOh/Bsm2tJGYUcvGgiplq6JbmT31U/eBiOgA0aSFV1BdJC1UYNmoKOOYMyjekQ3lUMUz6pJXZWoHk3YBZNzUsY1XCMM4v8QE1nWXYwiK4jY5TrE2xAqB2EqmZmWQqlV/6hhLIeqq2iMxlBlC/iUHigGuisGcq40tzrOKhyyGea4Iqtho5zbyZoKqSJueJN54KZsPq+9V8+rurjZ6rhnz9ThtjPCXYNNbm/1VxvPv0+9M+kHHWP7kn3+T/9E//tcA0OKYYbCL3AU9wOpYPY6bC1mvYv4o3nGJwWTYqUgubL4DwJxM8QIunkeGLM4lBoqv5cSeVJYemKNZ2le0N7cexRpuy/kArGCRXHZdH9fDTLCq1MneOaYL51RrXHLyyTwLe2UvZxDvbF7BNrkd/L1uTw2H5TZv4XN1nBN7olaxcptx3UJqX/LwsiG69FMXgz4JVMpbShwAACAASURBVJTZf58rYxvbVrc5F48ha3nGZ/q7LTcmJfFgfI9qT+FC/879rq9fH6u9i1oRjb0uxk0dX1V/v5a2XsvtOLD+ar0hwV58o9Ze0Csbwqs8195N7B2uIZJ1W9g4r72bAFZwTmHeLd6y2ltIecs91YuXrWYx3qb6HMei1WU8Fz94zRP+nBe3bgcb2ny8lgVYyoN4AmOkjYDnvMOJPXk8LtVuPHs/6yQeL14QVUzI12GQVzzS5XwdS7fN+wRKWl++GZdPrjayIp1LUzEFpzLnTdV+zl60OpGxyAtsvY8b7UtZBNc7+GndJonpu7JSehKfWD5Le/n85rqr8XTM2Fs2MVaeQvayFVZS8QpyeznVHsYr3lk5Lx7LTfuutXN7v1JVVn0tL4SrRe11GAHlUVGRgVAtYMVIKNdmzYL3WO5RVEJi9WulZzyWANYe0mve7bh4FMW7iPI9bu7ntq46yb3DepzqeXjFUFB5MTJWZZVrVgZEPT/K/Kzb/MTIyEsZihlv67bW8zBX56qynoXzpCVfru6vqsup69nelytjtHpsbObq1UfKS8euGVbYjONz6TVT8RV5XmN8PTn/yp/BJw2631Q5XNYz4/l96ub0v/4nv/0Yy5/9+TH/+//kH/7Gy/23/vS/+633DfiBeywB0rL8+fgOCQpv7AU+G3yY97j3HX7S3+McWhzdiPfTAT9qH/DVeAsA+HF3jzu/wxAdjm7EV8MtvmhPuPedaAB+5s44xRZ/8fAlfrx7wDk0uESHzgTMyeCzlryNf3V5g4ObkLJCbzy+Gm7wo/4RAPDttMfBTrKIClmLLt/HcYefHb9Fyhr3vsOu6ENaHdEbj18Mt5gLqcyNGzFGKzp6AMFWD24i7Tk34heXW4SsceNGnEKLP7v9Fd5PB9zNPf7uzXv8cjhijA6fdWfSI0wWX11usHcz5uLxvZ960mUsemS3zYBGR9Fk+zjtRFsuJI05WXzRnPDgO9H2Y4KYs2+xdxPeDwccmgl7O2NOpujdjbifO7xtL6K1x7p09xPpm/1kf49fnG/x5e4R93MPHw0+70/Y2Rk/P73FznrRvPPJoNMRjYmwKorR92EkxuC9o7oNIJp8RiXcTx3mYPAPv/gFAOCf3n2JzgYc3YRzaOB0xP3UrWK/frQ7AQDui27gGCx5sTNpzp19A6sTPuvOGIPD/dyhNQFjcOI1PzQTHqaODP1iiJlyzc7O+OpC8zQkjcE7fNFfMEWLz7ozfnm+gdFJNAV9NMQmrBOMSvj2skdrA971F3xz2QMg7/ofHz7i22mPby577JyH08XozVSHVosXlDXkxCta+n5oJozBoTVBzj3MLQ7NLGP1RX/GN8MeXZG/8dHIRgvf35syTlplnGeH1pBxyhp3rSUEwOPU4m034MOww492ZzzO7cpTzvp6+2YQ/T7eOPlyf8JpbvE4EXsv69zt3Yz3lz1SVqKNx5p/57kRQ5n19x7GFkZndDbgPDtoRe2bgkVMCrvG4zI7/PHtHbRK+MuPny1eRxfwOLayMWJNwtuO2vrN+SBGCyMIWBYoZ4XPdmf84uEGf3R7j2/OB8zBCAz6D46Pot/ok8bOeXw478SzftsTE+eHS499S7qKU2F9BYC/8+YD/vLuHeZArNmdDaIz+Hl/xs8fbpESedhPY4vbfsTj2OLYTbjMTrz/KSv0jYfRCZepwbGbMAWL3nk8ji127VzkjZSgBNgrfz90JHcUjcgmNcU7Zk0SRu/OBtxdeuiyKcZokIexxU03CWOpUlnQF4xk+JO3H/Fh2OE8NdA64djOMDrh68cDOtZELO368njCV/c36BuPN/2AXz4c8eXxhPcnmispKbzdD8iA6C8+Ti0mb2FNhNEZo7e46SZkAMNM3u1943E/dGgK9HgOpLO5a2dM3uLd/oKYlpj68+wwBytIjGM3yRjyHGAWdPr9KszB4k0/4MOlh9EZw0y6jbyhdOgmfDzt8NO3d5iixXluYA0hTIxOaEzEGCyG2eGmH+Ve0rNjxhgcQtK4KwiSd/0Fvzod0LuA1gb57bdFH/E8NbAmYZgdjqWd54meizel3G/PhPRRitAqfUHHPIwtIUYKCuYyOYFZs47lMDnc7Ef4YAS10bggbX4cWxy6CXfnHjc76g/Prc8PZ/zVx9sX1xUvpX034/HSEjKgIDX4t/1w7tA2pCW572ZMwSAVREjOhAYIUQs6J0QjGo/H3YSUIYgXBZIr4w0yRrjwpkMIGp/dnPEwdAVtYbErKJmFpAnQOol25K6bMUwNGhcQCpKkLV7ecXJIUUOXTSXniHWcGeBD0GgZ/VC8ycZQ2W1LXvppdFAgtEUobLbORszeCiv6OBaWbqAQQAFNYZ4N3ggpVIxaNrK0zgR5V7QplmYjMlJKZ0E+pKQXsiZggfd7LbJYvBkkElvlWPa1FFbGdnNIOzp3dUOtbPKogrxBrtpgEpI3kpcZjleGNlAZxvXmjloYbrlelnViVlhV5asN93rTivPUm0/bVG9W1Js7sSpja0i/sCH9bPnXDO3tua2xWG841flfU1/9/ffpbyT9oD2Wt3/6o/wv/mf/CL0lmOhpJhrrvaPFw8NEC/85mrIobGTheiovGqUy5mCwbzzOs0NTFnWTt+Th0RlvdwNO5UVJ3j+KVZuKGPC+nUXU2EeDvvG4TASf61zAHMxqTqekyRAwCaeB2uwc0YqzRmSKGm3nl3ixCn5Yx4PNnl4YMWg0Lb2kOe9waWBsRNNEjKODc5EWZ0ULEiC5klDgd/wSqVkE59kscUxZCUQp8YtLZ4KuWYYulbgnlaENyT80rUcI5gnsypYXl7URKWoYmwRGF4JGmC2azsNPdoHbTZYgQL1HitTulZczFq9QGXDbREBlpAqiG70meYdM8hpaZ4x3HZABdzMRbK+8ZJAh8Dp+3sWpeIwdnRfhZAWheUcGcpExYJgQv7x0s0gq1J4RuSYqoEghKAUom5BGQw/WWROcKGPRDGTvUSlDtwXmNBN0CQC9LAdDcJ4uIgcl+aGxvLz4pcIvtwrSg4yFmp3jrBSAJhGde6bPajTIbVyo0DUWT0dQpCEZ9AJfqvUMhT69/Hel7CZVFPOlPG6LygtsiiFTANSoKZaKNUGLpIUKCrkrx3SW4wBWOpaqQLky3+egkVl7kr0gannJ67Ohr8cou/wqaLme49H0RG1N3eI2kFip6uWpR420i9Bng9Sn1UtSX/Si72cylFdUXpmoatZ0C7sENdN9ymahhDePGvGYyKPjVYGKUXf0qBAPpb5AmpZ61MhNJq1Ht1mYhBIrZcv5ooGYGipbtAbL/VZeka5im1Z09TIOBbKWbYaeS96m/KZMhhk1Ob6aDDPROVk/Ff3DbMgrY08KqQGSy9S3IjkUu1z0FEt9GTCjQthlaK+gPRD6DDsoxDaLJ8ZMxaNRdPWSXepSGUgGMHNpA4czBiyajWo5pwP9N9PGcWcXDxMAaA/xxGSz/EZ1aQvPGz0ppLaMk6V6AbpGF+1GdyKoY7JrOB1D6ZKlPiaXl74W7UMoGjc9K+gZSC2dU4n+koNobnJ/k8tLOx21Xfvys+mqNpbxQAZSU44X7192dDzrMhag8s1Yxkkv5zRrZlr6HBu6H6wjmhVgR8Af8GsnPVMbeWyAZe7FFgI3NDPkN8rhVGZCGf/SF73MBzNV9xjU51pDNC+2iZRpL1Qn/TaojGSXsQNKv23VdrfMvazWY8ZzGSi/4UrPNJn1XESGaLTy8QISo77zczguZZuZ7gl78vie8PVcP3sl5XedOH9GNorawkRSCdAxl3FUG69kLmOn5B2jCrSnrofml6Jy9FKfJFX9DjZp5Snk95gYXqWe6p5yec8ZUbWTnz3Oq3TFCMuaxqPW4qzr4TJX+rbbtn+ijmt5n/WQXju26avcm2L9q0/ZIp8yPj+RuJ3/43/x7/7WvXp/8ufH/O/9l//Cb7zcf/vP/tvfet+AH7jH8q0741//8f8CgNheT7HEk6kIrRIusa2Idggq22mPMblniWSYNXZKVqCzBzPhIXToDb3VmDxmiA4pa7Tay3V8TWBorI6F1OWpRInVEUOk3buaRCgkI7DbLfkPy2Fw4np81mjLk5YZXtn72uqAx9Cik/YvT8feePisV/XTmCzSEwz55TYxLJjHrYb9Ak8hsTXpDsF7E+ZkVp7QkDQaEzGXGNaUFcbosLO0W25L3+ZkkbIqHtNlrGuGXr6nAFZjSG1L5FUt7eX7cP+uR8oKRzcJlJlhu8JgCyIS8pHuj9MVRWJJNexzihZWpydyI1uZGrmXXHZWaG2Qeo1KmCJ5MKZgxZt3jc00Z4XWBCnLVVDcMVB72hKbuhqnDRERQ5A5bT2XGQuREM2dEutYvIaNiStJC1TX1DG7KSvxctR5uA0M4+a5wf3hPtV5eO7VMb7WkNeVxysmLeRCpngp6Vgh3qkkMliawpm0Gu8tZDkmJR62VDxRuSrDmpo8aCGKqvvCcbY1DDQkjdZGjN7CmbiCZE5+0bpTZSwaG5Y4vaoOHtsacjzNFrsmCNPuEies4L1B33op15lInrKiDas3c4WhucYkidXNWZX8VF8NPQ6hxFOaBbr8nGQGe0d4w483zpQib0kIBq7Ex9ZQY/7uZ/LSuuKFiWWTryleGIZFc0ykc+yRoQ2vGIzoFCpF13O8HwCJrcxZkb1X/274eFQUk7u5huMl57husyqxa7w69HFN+sMLqxCWeqCAHBW0TVIGMpYxi7RRNvBml4KQ/aw8LCrDF4/NzHFh1eaSbKJxPBpvcGW1bDjxJgJft4LfqmXzyKVq86ls8ACLJ6b2wMgPBUueuv8A9aGUJ5sTJi/xeLzp4hVy8x1XpVUSbUFgMR7KplGuNskE2pqrjRNfVvYGAMNFVdUuGadSV03Gs11M8zX8Olcllk8vC36GpYpRWj6L6lRluGS9GCY1lLfeEFFb71Wme1yPO/VTPfHEMSxZ8lTxkNUtpHZXfSxdW4y0eqMzbw26ZZNpPVZZDLCt0bjUXW22cN9QjWMdu8lpayhuvnM78tOp+rJBtjWgtsbUtbmQ1uXWm1XP1vvKn8GzhucnynnZCN0Opnpdm76nYfm7krbSfn+b0g/asPxmPuI//8t/WRgp67i759gta+0vFpfexo/VMVh1PFN97Ussh6vP6ZknSA2jKN+Z6ZB/gbla2MvCaVMMG3FPfmelv5nrWm2DcaFXGAevPdheOl+NwTamahWXpZY+oF7Q1OVs28mf61ieJ2+gpYFbpsnnNsBWLI788iiLqhwXggS5d5UHVPq+HY/6XN7ke+6hfG1u8EKDF1Dbsbh2H7epvm91W3gcr7EpvnTdtu9P+qOexpq99IZ7bT9ekz41R+vv147x9c/lvbJYuJaEERLAJS3ukKwq4Wg+xovMa/ehemlmk3Gq467q35t+Kkh9uTYXt9AoThqY0yZvNZY+9+sX+JZd88kArNt+NdVzuZ7/n5oHKsNn9WQ68aXhhQVa1igMfFU7AczVNZHza8CnVuZDKGMX6/iuzVogZ9os4DbVn3khKkZc1baE8qzKFP8sPxdewFbp6vKjMh63z1BVjSl/VmXRLPqNm0Uzp+1CsE6rxfqnnnPcl/zMdFCAyuaqE+LJbHju/l6b13VbqsX2k7ofrzXqdelFr821Dj1pn7r+W5Hv134P6smr4mndm/jfa2lzwz9p6KwqW3Vi04iX2s51FaKZzXPr6ntn8/mJt+xTbX22DZtmb+8P8OR5/SS9pu7tRH72vfGJ+fSaer5DelW5zz6/f8P1/Jplf++6fp/+2tMP2rDsTMDPbj/gaCcM0eFu7qFVxo0b0ZqAry43aEzEFCz2bsLHaYfbZkSCwt1IcX3ORJzmFu+6Mz5OO+zdjClaDN7hUqCxX+5PuBt7tDaI98jqhMeJ4r0+21EsHQBho/04Ulv2zUyxWHnxkPoCze2txy8fjzA6rWKN2INws7uIJ2nwFs4k8eYB9FscSrxTiBo3O4plGWeH1gV8fNyhaQL6xuPx0qFvZ4l74RiMm91YSF2AscS5MNzXmYTz1AhDaM4KraO4EF9gwswG2Ba4K3splIJ4OvYdxRKFEpfhTJQYk8vYoms8fDRoncc4O+xaYm0dLi0O+xHnoUHTRDgThR1zfxjgvZX6ajbTnJQY210/S/wJE5LMBVqbM9C3HlonfPjmBoDC4e0FIZgFopsUmiaK0ZozMA6Eg3INeYhsgQEDgJ8tjCU8nh8tlM6wDd0zrROiN7BNFCgwC40z66MfHBAUzJ7wScRAGjEPDtpmxMHA7imehYlJ2JPExCRN5xGDQZiMMDLGoJEeHdAm2IMnRlP2lOgsBCdCdqLJM8GbA6os0IlRkGJNBALaBKQC4dW7gHSx0HtPcSUZUDaKpyNPBqoPBNPl/jNUVEGYD9mToduINFpiGbxYghUXjw7H2SiVqf7iNeHYm3yyQJMIGpyJeRFB0V9PkG+wBEOkxZ5q0rKhMBlahPYFesyQXIBguDZTPq+RXYR6cLSQfeuXMmYNMBy57NCr0RDEbxcri4T6D5slpkUNBvkQoB4c8j4sXiAA6mQBk8lINSAIKpenMtREY5/3kSDECoBdPET23iLcRDo2E8siDD1UzEXTOVWgvG2EuhjkLkFNBIldkb8E8hTltsCgXYKe9QLD1cWYZoN60tAeiLtEng2TyYuhsGJvzDZT3lkh9eTFzCbDXAoUtk8wF43UZYF0kncs07VJwd1pxC4XKK2CGWmBHg4VRLh43uxJI9zQcTMohGOCfdSIO4bZZZhz2Xgq0NfYoLAx0kZBchlmpHKTpX4YX6C3xQOYTSZ46kR9NBe1bG5kgiqS54v6YsbFK8VQ2GwAe2ZvFpAtYAaF2FOdyaGC+hK8NewzmjtNUMwmL56sBOigkGxGchD4rxvKXKugsGGfYQYFMwGhJ5gkw2EZwkrlU/2pITgxAKSW2qZnGjuGHZOnD1CeyokdQTYZzptamjdZAWamsmKbYR+pb9lQnXomqCWwQGBDT9DXWME/7QDMN9fXFK9JZqI2QlGd7JlSsYyJxxqWimVDwhb47goKW/7MiDJvFoOnJsFJbqkLIGhq80h18hiaUa0hrRlrKOxE0HBuo8CLM0Gba4/eEyisXaC8vGnFEOgaogwFmAFrL6ld4MuxLcfTYvibOSMrRfc8Lf3j3wTDpAUK69bHeJxoLMuzPC+7GtkqqEjXqphX48h9ToagsPyfr+UNJR4zTrVhKjY9bwqVjTiVueylLL6WDiz9XRm6ZbNJzl8zstNSb9YQ2HTtWd1ucqUKtk8ZsEpSVnpaR72RpBLdr8UzuxSUlboKa31y/OqmytP0nYzvaxs2v4MpQz1xfv1tSj9ow/JoRvwr7/4pfLbw2WBMDkYlxKzhdMDP+m9XDKY+GzgVSfPyxsCgxExm+nzZN2h1ED1CnwkO6VSEP1AeqxOmAscEIEyuxAxLw2lVRLgxy+cNMJ8hghoZf3z4eFWDcguFZXjrVo+TYancxwQl5Q+3DlYlOB0xvHFoOJgCBCllGCinUGCpXC57Q2ttS66nZoINSaPR8ckxhjzWOpZ8npk+Q+WV5TY0hu7B/Nai0aFocNIYhzcME44rFt0aWlrDdJlJdwuTreG6APDFnoiY2HD3cSFK2bK3bjUvV+eSgVFkhM/JCBRW7heUnOfvNdNuKLBMhklyHxgyXOtYXmPFzVnJtVOw8jllBf+FWRH01H1jb7PfwJq392wVz4pir5T7mAGBxRr2lmO9GZ3K3Kw3Wq5t7F5j3uV78hxTbw2D1CpjfmdhdFohEmpI6/Z4PR40FgyPXfItupkEha1ZV/07gmg2RdqG4MQEu62hsVxfYyNqrU9qF+l/1qzC4a0psN1FG9R/ZgTaymNUz8eapbfW+BT4+ucGB7vMjVT3LWrsSjw5w9frehh+u50jKzbXvKA8GOFRw1UZOrtlzl2gsFwOnmh0LtI4T6G5W0hx+MzA2QUqy6iTroLpMsohBo3WpiJno9DahPi5htULvDpVUN9c2rDcQJrbjDQRKZUydrmeYyWfAlahDTwWuppXNbRW8gDwlcdbKfKsKn6cqjXCxidAmYzxc9pkqL2dHN/Nx6dCfhKkr0sdyqRCyoIF+XFlgcSID6UzZmGnZYuo1MfxzDJ+5Y89kar6fyXPzLBMHhAutyYyYVht7XGvCVF+nVTDc+sFd1YrGOtVpEbdn+0DMm2+r4yJtUYm5xsE+lo2Zio5jSeeQa6jXnirql69Obf9X8q/1q2twfSEdZY3Tq6NnWIDOj9vMNTyHPXvIK/LWTe4bnvVmWrKrNtY5owgpNZNeBEK+6S+5TOP0aoZL6VrVlTdjmrMt/9flZ7Ld+X4s2VKh/i6aq6h+v60xJfLfa49PwBj8fdpST9ow5Lnmc+GjEOVxLg8xU4W9BRfSS/3KVlEaBgk+Ew6cFYnXFIDpyjmkReXrgQiTMmKAcoLcjZq2PDRmRbLrQ4YIhmoWiWRPNkKsvvEC/tUyUOwHAc9lUMySHmJtwQgxmstlQKw4VzYRcvCfW9mMTR74xEzxXo6Te1OWa/iLWtDko1Hq0qMoBjStDCxKokh2ei4xF9ye6rFaC0tUn9/+p/aMkcLrRI64zFHuxpjrRKtEbIu7aJYUBTD0ZZtO5aVISNziSfkmEnuz5xMidkMq/jLOv5tK43iKq8xlyvnSkytUqSFWhvUOS9GJS9063EGKIY2VxoiCQohaxljXbWrlnWp+yXxuCUWk8f90Exi8G4NNDaE6vhNVeYSf9YgY5oNs9qYtXpZvHdF43Qrr0Ljk2RT4RpsXZW28u8lZSXXNKXv9bs7Vn0X6HdJvfMUD1nKNzrJA69ezKuqvjqxNu1K3qIkU2LZaoOuE2//QhTlTFrFjFqztDUmDaNqg7XMCUV1NdYTU27jK+NwaRuAlWEWK0ODGUgpxjNKXt402HdRWBttNacACLMqb1Lw/a7/c0pVmamUVRtE1q43QZQiBkdl82rTie+rLURZzIiqAGgbZSMjZwVX9d2asJo/xgQxXpXK6HpyYZHBuY7FXMId6FrbxWLYQhAQpl0M0JyV6CTWaRtOoWwAS+AAGbYyKuv6lImrGM91mQoKuSAWklxT5zN2I0uCTZ4N0UjOgKkIo1bGoRxUJXZSkTe7XhVX/avr2UrrrHbi65XvJm1DF7bjQ+UB+en+HSVX9XdrgEphABo8XaQ+Fw7wmlT6wzGq9ao+l42UZ8d2a4HV7eSyriyoc52vvmzTt+zWRT4x3D+1QH/xXCHuupL9Ofvq6oFn6lDVc+DpNZuLtmEk9fP8OxhOtefydfmfGnmvSt/JkHp+NL/zrP2uBhzX88kb+t3qeP04fbcevrrc38H0+xjL39F053f4r3/1D4R2nyn7fdICN2XvCRN/MG17Uxa/7AEiGvdCcqCykH7EpHGZHRkT1QKRFjX0xpuCkYWp1glzsEKkwZp+nNgIiJmosVvnJZ/C4gUzOgtVOe9gM+V9/cJieKct9TIpRiwyCqFQnduywOWFjHioigQBL7TYG8DeBhYC5/qkPYVgo76u9iAwXTgRbOhSJ5fB1yhYmwSWy1TnMRgonWAtXct5ciJWWqXyIvC9eZFzvySGtiyeJXYVWKCnAFIkIhFegE4T0aVrpg1HZfiUhQGPuXgvsLyvmRUXILH3lOg+r7w+ZYxE7JwXWGXRrTURlHCdOUGYabUu46NQLZKX6wGIOLzWmcTfQZ4M6wJS1IhBk2ejWuTw4o0hr6Lft1mIKE0LPbk+EzmJLuysKWhom6QNPEAcT6ttEuZcrkO0AbPCNSp4oYC3ieC73GcmEgEKlHchIgFA+VOBwKq8kKIoLLTxeSEtgQJpGZZ7wkQo9fVC2sK/ibzEEeumeAALhBbcRiYvAU0UzcL1vlr5194ctmNtIshwk0R4XMptCnw4F7iyKXn4fhVoc2bmXGChjQdIKH00VB9Devn34eIKWixELTXFff1Cr8tn8hKNRWic8/Ji3uSFrKXuNwBhD1ZYPE5cVplLK9IYaY+Se8jjAlUYkL1eaPlZ+HzWS1ww183sxjpTPq+JYMZXCwARZa/6XG885KrPvPjV1dhzXu4b5+e+cXu245uq/lf3eOUd47Hg8rnZpS7lNXKblrbV5ZW2CbkKtw9YvH25sAfbcv9EM7KMaRUPzBqTDGnmY1BYEbykisiFSVmYIVklhaQ3ovTsvA3FyMlL26DzUjZ7KWXuQjyh2WZiZv41U7aFSRlYxOtB46gLlFsnJYL14lUr1yJjGeesBEIp41QZvfU6+xo7aHbV+GSsILDLhcu1TNxTQ15ryOrK+OV5xm2QPlX52Egvz2tuu5Al8TO9Yh6W9qqljhWUtPo5Sb/qcQD3YXOwjPO1fQzxNuYqPyeOha7Ib55cn6sxk/FfG7OrNtbf1XKfn+xR1uNbp7z5rj5xrH4uqOrY9pJrXtdrdavN/+fyvSJdNVK3bfhUXX/LUgZWyMO/bekHbVi+a874N//wv8c34QZTctjpSdhhL6nBrRlwSQ12esZ97HFrBrz3B2iVcWsvmJLDmBx2esa3fo8bO+JSeMR3esbRjBiTwy+mN7ixAwDgksgbGZLGjaV4zY9+JxDag53wzXzEO0fQylNsBUILLIblEB2G2OBH7QNSVuIVZS/pzsx47w/CeNrqgClZTMmuDNVGBxgk9Mbjg9/LMdKXfMQptDjFFl80J3zwe/hksLeTlHfve8l/tCMeAzHrskdyb+YV/PYcGzQ6yPUpa/Rmxjm0wjDLaU7kbbz3PXrjhVF2iA57O2OIDjduxIPv4HTEOTTY2xmPvkVnAt64Ae/nPW7dgHNoEbLGrRvQ6oCvpyNpcRa2VPKcRDjFhjaN48d5J/1hb2p93ckTc/DP9qQn+ouB4nI5DwCMcf0zuW1GaGQ8BgoWmaMV+C5rnALA2+aCkAweQyue2Z2dcSr9u4QGVkeB+tI1A7TKuPd0H1gr9MaNmJPBzS9HegAAIABJREFUscQKk0c3YIxWvOe2eBEfZtLHPNgJD6Wcznh81p4xRIf34wE7O4unj+tgaPScrMCbtUoI2YhxvbMz5mhh9aIVOkaHzngZzzfNgLu5l+tT1hjLeO/sjEtoyBtd5vwQHJyOsjlEzMEEdeb46Eff4bYZcAkNxRRnTQzJibzSrQnCNszH3rYXjNHh7EtMbNE57YzH/dQLU3GjI9rihRmCE2hybylwiPU2jUqCNGhNwBRtYbWlz1/0J2iV8cvzjWxwtCbgUq7nDaKjm6BVwn2JCa8ZlGsI9tFNeD/s8eXuER+n3coL/KYdxNses0ZvPe7GXlhud442Sk5zK55b1nYEgC93j/j6coRPGp0NaE0gVmhkHJsR74cDeYtNxMU7HJoZg3fYuxlDcOJBBCCe5Dka7FyJlzYBZ9+gLRt4teeY63uYOvGIMpTaFVi1Vln0OZ2OIiUVs8KuSEZdvMPOeYzBisedIdj8/YvdGWff4OIdjCIUAQDcFw1Z/o0plXFoZtyPpKG7dzM+jj1uuxGPUytMwDfdtIKbD97J9byBuePxZkmnInXF1/hoEJNC54idmfVS2es3RSMbnXx9LN5xngO1zirXtXMeF08uK9anJEZYg9YGnKcGnx/O8NGQZmkl1aTKOPioi0SWFWZq3pBNWQnvwL6Z8Ti1cCaKBu8YLJxOoqsJkIYmsyTPhc23Lx7n89TAFc96KMy/SmWMsysboQQNZymvVI37HCx27SyapSzhJTqXhWdgmB36xovWaEoa+3bG3aXHr5uYC4Dj6/l9bE3CZXKyidvYIKEesWxwdgV9wFqtGbRZm9JyboVwKOPJZdBxyPddN2Eq9zwl0oyMpTzeeGT2ZIBkzWpugpSUoApCKDJiZd7xBjVD1mNURdNyKZ82QTWcK79jTxv3beulvVwOsS6TxBhQDETepC+szyxJhrLxvoJhl3GOiWTFZONSQTzHOS2bmDyGABatyrJxWG9CCYEhG8h8fW0RpbUsGBVaTYra0OZz/Jk3Ya55rK8ZWWqzkVRDh3ngtoYY53uOrI3zVmPz6vSCsUesxesMRNKUV9+flFO34TXG5P/PDM8fevpBG5Yf/Q7/+Kt/CY++lcVUTBpn32AuL2yWXBi8Q+88zjMt8vpqQdKYiEt5EfELvTFRFkUfLj12mxcCC6NnAKexFQhYY6msXUOLn2F2Ip7NUDWt6EXqo8HtbkBMWl66plDRW03EObyAa4u4My9ieBHCItjWRJynRhY9Phrc9iMGb0UE/VLKc6U9KQPD1MiLg8l22AvJLyperAKAD0ZeprzQaYp8RR2DBZCsgLUR0+RgbRTIF7+IWFiZvaaso8neyL71uIyNEAPFqNG2hXDp0hVyHS3eVV28dMAS+zRNrnhxl5ijpgnwnsiH/GyRksJXb26QMvB46qF0XuUJ5UXI8DrWC2XRafa05kyi0vwS71qaM94baaMtWp2s2anYy6toq7VtaTE3Tq7USV5V1wTESERC80TsDsaQ15d1Q9lL6WcS0nZNoDZmBWMjdh3prU6TgzFJxorLAABVPK4iA8Hfy29ukWBIC2FS0THVOsN7g6YJmCcnsN3MCxGdYW1E8Ba6LPIUQF7V8tJQhha5vLiIQcO4KKRH7LnOqUgrFN1SwyQ+hcQIAJouIHiDWDx52hSPsEnwky1towUDj0XdFsMLrnL/2dvKZbE3WNuENBv8ck8G5XhqF8+gzYizXi1mHM+fccGt8SIpBy27/64LmC8O3+wO8KMVmB0AfNV78sYDyEXOgsmiclIwhTAojpY8qXnxZCMrfH04YDw3yFFDu7jSYm3agOniZIzTbGC6iDiZosGqVwsjVTRmk6fzJHtB/dau8jLz86+huRImQ/ew3E9kJQLjqpAqKZegbRLtWCQF3Zb+zEY0YcXrYIqma+nLh8MewRukmWRCTEPwznixUJYXnZTXdAHx7KBcgmkiwtnhw94jDla8L3cdIVF08YwlX0iwWMIjKPIm871UGdoVHVr2IkcFRMqXg8K3XVwtmFLQK2+ucvRbEK1aBSABHwopFffBNDROSufFU61oUatcQh4NHo8dclQ0FxQWci6TpF7VkJdXlzkkxFuJyLfgEkwbEUcLZdJC/OWXuZu9pgWuV3hoCuqleINlfAaz6NJGBZR5lCez9pDPFaKBvahB4dz2lIc96KYqmz39o8GpjZX+rsJDF5DOFWb0O6azS0SIBSwedYD6Pi31nFlOJSuJYxzaBNHzZc97IfUamuIyq2NH2UgAFm+0eBCBsWtpfMr3wZVxY88UX1KuHV1akARJQSWFqcwTFfTi2QUwFxKtmfVno4J3uZDSFONBk/fYsyc20L24NM0qdlMViZrQJChuL49ZBoItMaFx0+bS98SeYUXHmYiJy5cw3gRU4CQpizUedSWXInZeQRasNHRrZ5Iq3nObYWrveW3obca7NhhZZmV775bygatGYP2xzqOWn4MqU0g8quU4KhtXmpMhZDxXU228Vf9rZuXaLq498Ve9pNtjW6Pw2jXPpGse4VU7cP37725SiN8d2PyDST9ow1KrjEPZ/SevT0JQGQc1FQKfiEZH2fkzKgENwFqK2mXZ2T60ZNA5gaKWGDKVcewm2Ulv7Tp+UKuMXTvLTrlRGapdYrZ27bzEbJrFY2m0hrM0taxOUCV2oY5La0uMF3s7lMqwm4ATgc6qjM4tOnYcb0b6bbSLy3FX7DlRWaF1QaCrRDwCyUMGZiokJXFVZx1zR+QkhQylIpdRtrSvaOLJuDWkfYgm0H8Xyu5lyd8EgZs2DZ1jQ4jH1bli8CpqqzFLuzmpkq+OYyLYaJZdWteQbh0TVTDUlnXylKJjAFZji9ImpTLBtRTHIpLxBSwQWs6nlJaYKWZ7BQCtF528XF7Y6/ivWMopC80Kyqx1BjZ6mtpEKYvHUSnyTPI95bKVKvFkSEu8k070klcFvszfeVwrwhKOmePydDVv6vI0aCe6PifkPRUidEXMo4itlvJkuafMCqtVRtIUi1a3h2PLohh+WcoDSvxaRXIiO9/ACuIrcWPFaFBVO/i4fLap1EfG0fJizgLT5Q6kuBDMiEGtyv2oro2RIMPiRajeQwzxljKTXo0VG6HKpGWRImWUzRJFxzLDtDKNJRvS3D821hieXMOuc7WAEkIYgODSTxZM5ZqkkLjPNbS4zpyVjIPAnctisCabEW9DuZ9CJlMWecEbuR6ZmJT5Zj4J2yrGXE6qbEbkhaW4tJ8NMom7rLwdYmvXxoRSiyHOtyKv87HRu4LJ8hgA5G3ZHEOGsC5z/REAYoG1Z5ABoUvbArUhTgX+zJ6GTN6fHJUYOAwdTmzM66pNSQFB01xPCjmYRVew2pQRGHBSwFw2pnK5JquqrKpP0VCbahKeuIyjQCsB8pTMBR6ei6EDII+8AQHkZMhImMr/wvSZolvDa79jyllXGpVKiHIY+stxkjlrgobWC+u8tE/6yX/MdyCWxPLviQeKx7M2xpIiXZ2MRf8SEMOIxljLOTEMNI8pVgYRG4n8n41L+cwPxQRid+YywHOb6y/5E81lMbLq/vn1/S3N3hguy9xng2pVTq6ybhP3i+fS9hwIqgvu6yoLHWMG4/V119PKEFLqKYxZyuAbgbX1lq/24kn5L/1/Wheu9O3T/XnRUPsuRtx3yPu9jcPve/3v0/dKP2jDUoFgim+bAVOy+DDtoFXGu/aCvZ3x8/MbtDZgChZHN+LjtMPb9oKQjciN9I3HaW7xeX/Cx2mHm2aCjwYn3+Du0sPZiJ8cHvBh3KF3XuRGnMp4nFrkrPDF/oTTTHDH89zgs90ZHwaCYN52IwbvXpQb0Sqja/xKbmSeLd4cBhJGBwSC1Ng1WcWlyI3M3uLN4ULHpgZ94/HNxyPazqNzAR8e9tj3U4HrNBLX+PZwwTA7GJ0xzE4MYV88s8PsKvF0iBQIi5Rbk3AeSG7Ez+SZ9LNbyYrsuxmjtyRrojMaGzBMDfp2xnlo0be+QKY8zmND8iTB4HTuSG7k0sI1AY2NOJ07pKiwP46YZgvnYonvJO9ZKAuolGhh2Pdz8aRZMWiGSwtbBNJ3HUmwvP8l8c8fPz/De4tpbCgmMSm0bVgJt19OBMtzbSAIVvFSKpUxXJpFbuTioGwiuZGkoU3ENLRwbUCYLWyzlhvJSWN8aAGvoQ/k8WYv33RuoGxCuljYg6f8Je6Q+57LgrHZzQjBYD5buJ7KCcFg/tUOuUuwe4/gNS02QYaDSHQELV4I8gQpMeSQAV88OiLdYTJJgpQFq+oi/H0LtQsSF6gslZ+9RR4NyXeMRqQ9TBuRCrSJYVIsfq67AH9qofuA+aED2ijyKH62IjcSRwfxWjBhzKlFbhIUy314gxgUxZrtgxhecbS0sAEozo69QkWmQxVSF5EOyeWzS7TjzjIk37S0ePrMLwbHWdP1sghUyAMt7tK+8lTxQtxlEZTXZ4N865G+7ZAOcYmpy0D6tqGYLAXaZZ+pT9L2kWBkeR+RRW4kL16Pr1rkNyQ3oiaL7BVgMxmEF03nFPUt9xHqrkXuIzA6itOrDGVcNHIEcpOhBgc0GZgVcp+gRrfENXIM2ahJRmKXSDCeY+MU6D7YMgZNhhostFfIu0SLQpuhL4XAbJegLxapzUs8WygL6hLr6X7eIPYZsUsk/TGRoRKOCbpIVyRH19qTgr+l4+ZCn929QziUcVaAvZQFdCz3oYFIg6gSJ2dGakO2lMdMRVojKLEtsskwE0lDOJENofFJDUiqpMwhM1b2CIfvKoiMR9YkX8JyI0hUBssxsBSJPya0/8wiWyA2GTpSjKNK5AFKrsiNXEhuhPuqPUjCQpPciB0UzAiEHZ1Tkf5iC/EksdxJakmeBABSS8fMRO3yB4j0CMtvaJbsmCHxgLGDeFp0kRNJbSa5la3cyETnY1vkRnaAvZD8iPY0Fu4ETG/xayczALGne2DmxSBTqZIb0UWawxb7r+xP2AEix8LSHdnQfbUX+j2J3AgAFSCbJixdwh6kbIDmISP0SsbQjDT2dTyginQMKNIrLUTOJatyDzMdZ/kZlSEyM9wfllBheRSWC0muui9NGaORpD0AQIeM5FRpX0bsFiOUDXQ9ZyqXZUEUbYrk4rHVIcs4ZwMkp0jmJpPUSB1DuoqFzDx2JCFSx5ZyvCEbYiytInGqG68iy69sE3sLkYtnVFH+1fjbJS9KfrouV0ZnXoxm3lgu5dS2pkiApLqcMs5FKkWkQKqiobDIjTxj6KpM16qUxVjPunzf9BlY+v2sIbu66IVzm7KveiKvxYduNi9+KCnj9zGWv7PpD9w9/oOf/je4S52wg/psELPGmB26zzzGQpPms8VeTzinFj4b7PQk5xoVcRd32OkJj7GHUQlORXTKw2eDh9SjU+SV9NlAI8Fni057kTlxKiJCo1Mz7uMeR0MxmWMilllTPaUitDDZ7n4yC0NthIZTUfJy+7hvKesn7nNTfk1OBYzZEeuriohQi7RKNtjpGVNy8Nmg1R4GGWO28MmKRMvOTDhFislj8KMr7LFSf2FabbXHVN5WTkWRePHlCUpstNSWKVP8nikMs76SXzHlnjkVcYkNnI7wycDpKMc4npPj3DSysO0Caw8y99mUJ+clNjL+zAbMDL8ApJzbnw7QyHgI3Upug/M4tTDBtkW2hds0JYtWB+kH5+Ny6rY6FaU8boMw24LiP0PSInWTskLIBr2e5VgdD8nnWVaF43eNWuIUmUG4NzOmZHEOLVoTVveZY2OZUZjbRPIhRu6/01HkbACsmIZjVgjJSMwhIQDofi5jEhGSkf88V7gspxa2WpL80Wh1xJQM2tJfn/WqTk6cbxlrilvkWNm6z7VkELeT+8J1M0NzEAbnhTW5ZkWWmMzCvDzGBWa3ylf6xQzAYVNXLecCENvyJTjsrJd4Sk7MElxfU0uD8NzjeFiuR643EXM0SIVBmROz77IuL7ez0UFieWv5pDo+tP7MY8L5t9I4VkWJsa3ZgbdswdzPWpaI87wkZcReMWZwZmQK549VSAHPP6cjaRQX9thYmKf5PtUyQHXiced7sO0Dx0TWIQUAlhi4zfOmjqWs58M1CZ4VG22pZzsXOP1/7L1dqHVNth70jKqac6619n5/+uuv06dPTkJAQ9QgUSMqeOHfjRFyG2/URMTc6JW5SPBeSBCFQFQM5ELRJIoJRMQbCah4oZAchKAhaPTEpE9y+nR//f7svdaac1bV8GKMUTXm3Gu/3/u93X36fE0XbNZec9Ws/zVXPfWM8QxGV2Xel2XJvtv7cEC2Z7Nx29/n2+DD91j//JzY+wPTk/bLvUKi+VfLY+O2V+L148FMiLaeIM80ey8FMNKXMEIfSmRj4sbIiKDo+tF2yEzdApK65Uwj0rV//r0PbeT7vh+rWV0p7H2xul3yisKrG7Nbyfs0+ve3GDRfxibUELZrpv3vx8SXfbO+XRv3hT5TRjvQ2zT0xnty/1s9+zpu3ePz/yjpORD0oTYAnRl/tg28fb3V9y9tEz9zfd+WLyvnK9T9sWX/iOUCAP78V8z/8/SV09caWH4vv8B/9L1/rgm7mDiMCZqMobN79v5ahrZh8Ru+WcUMTMBiiKVtkh7XUTeefVNgoh2VZeNq5SVjNGORDWxJm42D/RBZvMJjWtvGyTYkgXrsQh/SwEQu9qa4gPxYzyqYYHEFjyqk4WNKFiZMbnNkIg7F5QH6D8UeZC27WIJ2n21E/MbDhEjMb7UztgFDrFhVsMEr8iaNkyZmuUVFF1Q8oFITeLDrPsSCicT5jY+FVPCbEH+fmYfeTQuIGI/zKDFG3Q+2Kfa2zUMz/eyAyFqRYkHWkDSj+sX6zVqkbYxPv0myjS/p2FkdtVITekihbmJQMrDZZAFoSrkx1GbiSySiE8aKe2VgH9/PTDK72e/2vbGjwZmBGmMcQm1+tVnZaWvPrfubMq+bQx/z0Mr2IhIm6OD9gAG0/vi+GJst9aP54AbHjHaVZD92XfTCRCvMZLiZ11rdrP6ALPUBwDIPrf0UuKkHi+mrmGEzi/9o86OzcXZzGVR0YxgKShYBHBupmJx/nbLeTUkYaOx8KWEDTOx+73/rfWnNVNvGJxBvxt6+FzZ21kfSsdvnM5P85iOo+a3cJrrhNrQtVEMFKBiQ6Js/UjXNWrZ+tt602MYmqdhIUz0238i8BSYyphVlDY0Rr6v4iDZTVnKm2bZezOSvIwcxP0bf5IoycPc/tP6S+bmlutkcmc9pN4/mmxvqZpatZVKsTWGZnQiLmTFzDuIDy9Sd0iwRN4sHUV8OrR9NDZlJFI4jtz49UVQOvRwbj+YXaYyDPTts/dMun/qstnEtfe6bP6P6ZDaTY6au5gt0P8IcJJ9TrG2Kv5+avPqu3wFbW3VNNyEYaztc33cga6Pi+1xibNYaVQKn2uqg2hV9nxNQMbXdBk4YjR3bmO0Ct8Vg/P9tzbu221z4NUty3dq3MYW123TMbEj0UbltT+s3miXNE9NOA347ZLIx/7Ws5OrQNn6Q+Xpa7AdQ9I127e/7kmR1MT1978dp044vq+NWO2/Ue7MOX7lv2I0277PebM+HxvpWu79K/q9B+rmP5W/SZGzL59MDlprwbj0ghYJfGIV9+mI5NZXL+zDjnEd8Nj2icsDDOjVlzGse8K3jAx7WCS/HK3INoiaZR0Sq7bNTWDYntY+rmER+8/CIaxkwkZjdfnZ4xHtV43w9XZ6oihYOuBsWJKp4M4s63RQzMgvYXJmwlIhX07VtcpcqIkQnjY1oaS4iQDSvA16OszA0quj39iqqi2MseJgn3I0LBhLlQgOAL6YZc04YhxVLiTiq4mILw1LiRk3yNKwtnIv4ilZc1tSUBJOGaGESAaRcA15My+ae45BVXGnFZR1wGEQk6X4S5cl7zX9dBhyaol/FNBRcF1FhvJsWbWN/0rACSa9YeRhyM+0VX9HaFAKZRYAphYofvBdF3fvj3JQGowLlSQPZ2+m5qfAZe+FD0ZiiIQN4f54Q1JTVQPa8JgwGmNXHMgbJz0x4OE8A93h5RKI2uKgp7/UyYpwymMWk2oCShTUBRPmvlIDrPGDUcnKO+OHDPdJQME4ZOccGuIh62JqcQ/cdrT0Qur1flyDAY+lAZxgzllkYwGEoOD8ekIbcfPW60FDAsgSkJMJNZl5tAFSAEAHcy45O/Gm+js1/1ESVgrZ1VSGlEGrbUJ8fJoRUdZxFZGit4k+WRjH1JAJqCa2vMXYF5LxI+2MSU+W8drC5ZmpgNqvI0/nNCDAw3K390EHHq7E4NeByGdBEaLTftSjAM59KAtbLgHTIuL6bEA/S3qYc+n6C+UgiMPg8IB5KA2jrPAAsZsZZffjICVjldyfQKTfRHS6pbYjXJYCOGdD5CEMVs+6holzl1RIRN789ShXrdQACUEpSk2lqFI8BlXqNMs9j7cHr21z0DTlFRr2Kvx8GBV+RgUddV2NFOSf5TEFY29jrJiS/GcAj941uJtl0H4q0DWib4vJ2AB+LAKaVwIeK+lZNfy2fCo+YH6OE3sB2472qMJSJs2iYD9tQN59EDU1BJtRlG7HEzSxZJtOB0ij1EAG02I5TAcNC0lcmESbS8BEUJTwGTxXhB4OYzqqZbvN/KxAzwsQIs5jL0iobfQtPAQLKQUyFw0qoo4AEMxvkhGY+yKkLnpjJMQ8yr2aiW6ceeoKD1EOspsnNPxQ9rAih50+MsETZ2yruDwWgVesfIObWAxDWKP3RfsQrIZ8+fYca1m7y6c0woWahpECLVnTfX9P6mSWUipj+yoSbaWk00+zI3RTWCclYCI+2rw9AvETUUcctMuJCarbcTaktzIe0XT5v4jeEbtY97O5TIR8y807rE+FJuBI14mkmn0HzWf1VTZbDqmOk35dmLmrrxosV0Rb8NZPQZiJLTZDGi+fIPVtAb+atvt1PQJb11Zt3ujzts13aALB9mdzb7NuzOaneLUV/3uBNalvy9+mrCBP1/u0xXyPPQx/Hm2nfppttJDWZ9eU/U+D+YOAjAOItQHvjrOBp+V+jJPuBn5vC/qZMK0f8+vUe98OMaxnwbjlIyIO44pRWfDGfkELFJYtM/pvrEa8PF+Qa8G4+YEq5SdlXEN5cjzgNC+aScM2pgYQxFry5HlsoBAMJ765Ta8vDKkzX4zKgnghvLgIY76cZl3XYmLsxk9QdC744H0EQoZ5SCVll3XOODdx5CXYf7NyuF5cfAK5rwpQK3r4/YpwypmHFw/mAJUekWDCvA7KpAt4RZgMtCuRMtdbk071JyjIqC2oqtrFinhMWBSueqUhJGKN1FOBrbJAxkesQm+qrxd28LkMDdtfLKKaFlxFpKEipyDX1v1uWtFGGNRbIAqCDGGWSfvZYmoxlHlqMycNhRSQW30agsW6mdFpLQJ7WTR3LVaX0laGyeJtEAkZMMTJfEigy4lgam5PXiDyqj6VKtIdYG6uVLwnIhHKnLFeQMTbFz3pNnbVz/pCsgAlMKAdCLRFljq3csgbgYcA6yvqqS+yMSmBkY1lKQI4VFICaLUZiZwN5CciJm7IlAsuYLwGohHoI4HNCPVLz4URgFGVBeAmoUwDPEUXVLk1JlAKrkAman2GZAvgawUdCPSeUMbR4lHWJ3Td0ESGTkroYD58jSmLUScGA+oVSDliPoYEtXkOLk1hV9VD8/dRUdSpyQr4GVIsfmWXTbH6BdWCE97IuVh93cA1yvzErlUCzjpU/uTcxFecHSdeADCC8TwLEXGy48Bhlsxig4CQIC28b8Fn8W0uFqmrq5lLbld5G0U1R/0wylcoKxHNQuS/ZyNepgi4BPAYFL/0HkUnVJCvAQ5B6k4CcetS+Bmunruk5IKxAOYlPpLWNScAMJ1VmHBjhKiCvHgkW6y+epT+1EMKVUEdq4O6Jj+W7gDJB+lAlPwHITnDEQFA8k4z3QggzITOQHlSoRsFbOpvDm4Eg9Ycs5ucmvpNA95ULq3z/Nn1VIFFT90G0PtREDYSB1GczdgBmm+B4RbvGQfxHy4HV70180ORzEn/DEjC8J3AwUIjmUxd0PfNAiFdZz/Gq61D9zgT8BcQrISxAOQhINHBVFcxxAHiQ+aqjADkBkgKUzK9SfDSpgRPKWs5EoBXNx7JOsuY5CHACiz9gOt/wsVzRPo8LkA/qVzh0/7d0wdOd91dIcRa/T0ABlIIHqlJf87FcOtBqPrpXmQ/zKUWgDizVP7SmHvvSgxlOtAGWNYq/aDlAx5AQZ6mTYxeMoSL3MkkddejAkkPvQx3dfdx9Clve53wsrS+QskFbv2DKACf5LM7i7wr0+5k6EK0JTYuuRjQQEhT8UhXfTQPHkhHqVyhj64GLAR6O1JnVKtcbAGvgTPwTaxSfwg2go/49kHL7EtoAIQdw7ZBFgL2W44HmVwGWltfKcz6WnwIs7dqTtGvTTday5eOngHqX2vjvBJOeBaI32iA33K7jOVb35+mnm77WwPLx4YC/9r/+/SgTi+O/O/2sg/5g26mnOrb/PXVsj1f5cQXJD9LfO3bH96BiBGEmXCPw1++/Kc7p+oMOfVCYuMH/cy+bCRNu+FsnqZsYeDe5U11b/AV4N4rgQnqUB+GiJ6rQh22swMP0Qh3YGWElXHUT2RL3H/ZQgPcHqYNWOTSfzvKgPg/AMAPrIPWEGRiKPLjfHO9BmZqz/nnUh1QGrvpj7R8Jl1HymYVsCcCoP0BRNypRH2xZH2LnsQsEAMCcpIw5AcMCZBV6uAyS76L5DwuQjydMs5SfI3C4SJnr/RHDsq0vMJDMylfrKgcZ0+Qetgc9NU0VWCfRC3n1hbRvfjViyMCU+6bA1gR0Cu/mft3W26jzO6koArj/uNYRbRN3XGTTc1AhCUB/vLX8eJG85Ti0HwxOwOHaNx/5NMiPsm34Ut/cgYFymGSOVqBO8iseViA9MuoYkU+DzKs7JWZjQtyJcMh9rVsKbsNnfaoDnADEgPTIyKeoQeMLAAAgAElEQVTUrpnwB7GsvXIY2gbHTsrt+xFWNPYBkPLitb/WUfLVuN24hdx/aK296cwoI6EOQ2s7VSCsjHyMaCfvZXvibj+mcWFtQ+obLW1zsPVB/f/hvTANy8up/TiGBah6v22kwiKblnJwIQ+0/7axkg0wY72PGB4Y+Tj0IOKQueRom1JCXCWPrfs46+HGMbZ+1NQ3m+N7xnIvzFScDWTIxjVdgeU+AUHGqkwR6cIoEyHOMqb+9JlyBzNxZhHKWGXc4sJg2zwrOIqL1JcPCSFzmwcmQsjcAYWWF7IcMBgAifpdqCPpmiAn8oFWF1XpZ54IdQjyXNXneD6FttYsr4y3XI+z/D88MvJBASER0tW+J6xrl2RcG7gipKsIYHRgKWNHqt7JQTf5K1Cj5PcP2WJlVnnixIVb/Z55iQv3Z0QgpLkiT8Im1CRjKc8XKSMfCeP70oDNxuywcAMkca4CAhYGmNuGlQMhH+TzuMi4mKhKA7OFdR0rmExAXKo8G0bZsNt6XI+ybsGylq2sMqqAi14rI0meAMnDAhTTtbrvgIxzXPtaD5mRDwHpWlEGav1Il4rlxQ0Vlo9McZb59M8IUtCTp6BrmBCXiuoPBIiQLlWemYkQVm5AqEbIurHvia6xBv65f38aCxeAdK6oE7XPw8IqykNtjQr4I30eiZBOWHu+sDKIGWUM/RnD6N/FrOs2yhg3YRhto/UFkDVgY2SA2OrnBISZ5SAI9jzk/r0ndPYf2IDEBvRYnycq8rMtR8b4FsjYAzHpI7d6qPKT1z1jZvN6K/nvkfTDt4/bOvigOWsrDO158MRcV8trbTXARs9ch42FXNsIG7nrT16deM9t8LtdB/t0Czju6/nS5A4CNsrUuPH+a5i8a93PWvpaA8vXLx7x+/75v4LKASsH5BqbcISIqnRxFABYamy+kF5wwUQnLIi798sEIMIVWpaJjgSqTaTjlJb2v/lvLi1/91v0rKXVvZa48Qk0MQlrU/M1a+IqdSNiYNct0LvdC2ATbN0LT3hxCBO38Nd66AzeiE0UFmGP4vrj+2cpEG/8DlvfIM8hu8eCoFuoFtI6vHqu+ZZaf1nbeVJT1tviD7RpixcW8PNgjDAAzKvEfrx3/pwWezQZm2fmjbXXK+a1PUaptaPW0EPLuPE0v9Wspra+XzbGRdlimW9u8xuIkd34WMxQ+y22R7VtmbzwBwO45Nh8Ib2YBoAWcqayhiNh6hEO3DivzvfL7g+BReWexTz0UsTHblVfO+ljv4d03n3IFBPoyG78rF1WzuJ8Q5mp5YVrp40nAFxbWA+4PGg+nl4wYi+WAaD5qtr1rgzcTYjN/48IOGtcSTMrBtDiBFo9ck3yWezSnU1R201wlfioj2oJ4HclG/9FvebDBpkqcjv4dmuMAdQcENKunZrZfBet3NbH0MOY7B1xzB+SnZ9Sew+51v1IdZ5ceJLWUOsiO389Rg+LQtz8CClwC3zeO+LKYzTfxo04iPkR7vyKWxB0hsRzHKqw2al2YRDXv3aatU/7S+Y/6K8TGiO58f9jNNPglnwYD/+orXA7Wlee6/+mrshqCXBj3bmNa/u/AuaHJ58xOFa5XsS/r/nlsbbZ8pKBXu6hOaxtdvjnVI7bvYCMadHX6vKR9VnHoez6W6mZccoGWk2Nc+iAhQAqEfVH2PlQJQea/BiS1mPAKWyYJwAgtSCx9tlnTP2zPveETdgQ/32xHDkJ+GtjHVp5rShGB4x+k07oYwr3HnDzsp0a2q8TxnaTb10o2zYYM9jAnd7fuurbcwuotB8jfTb6qd/5C2/A2v754q/dBEQ7MLcpeNf3G+mDLOC+PR+BrZ6rYw9MbwLU59rg23Gj7Gfvf668D9T7lRnEr5D/k9nJ//kT7/t5+uj0tQaW5zLir7/9Bby5HFEq4X5aUJjwcJ2wrAn3xxlLjjiOKx7nEfeHGe8vYsPy4njFdRmwlohpWPF4mTCNufmyHcYVxyEj14Av3t7hcFyaX5ptHM1s9O3DocdFTAWXy4jjUfS3L9cB41i6KSGjiYfkHPHqxRlV/QlTKk28ZRoy3j4eJSA8A5OampoZKACAuJU9poyH8wHMwDgWLEvEN1894s3lgOt1wIu7K87Xe5QcME65AarHx4MIgVRCGkoT9DDwMWpMyaI/emZ+ar5mOQdMU8Y8pxZ2w8JymJDLch0QU22mpaUEpEF8v8ZDlrARxChrQBoL8ixhJI6nBZfHEdNxxbok1EKYjiumIeOLN3dNKMSerESMoPEuzSR30cDyYegbyXHKWNcoJnFzBArhs2+/AwB8/9dfIgwVwyh5AnHztbON5HgQux9pN1AzNYGNYcrIClIPxwXLmrAusqZKlv6tc0JSc9i2UVbzzumwYkgFj+pryQzUJWI4Lagl4nBc8P79QUxkU0HJEqePIjcRlXyV8RsPK5azMpZDxauXj3i8TFjOB4ShIthBQw0Sw4+BkBhV49pRZIQo723TmkYRkqHITVRmvSakqSDEiutjwnRacXmYEIYKIKLm0MpLY0a+DohjaUIpdY49YLf67xmwKdcJ6ZCRLxOG04rloia8LcC8tnsqbS5Y1+p0P8u6uWrMxsgIY2nhW8DUgseTmv+2APOVEA9CY5ZLAggIY8E6a1mp101DBS8Bx9fiE33+walthi1Ie9tYR8ZwXJEi4/p+ar+OTcTGArkzMNwtWN5MGF/PWB7GDQCZXl2Rs4wF54A0ZawPI8w/MZ4ywIT8kEAH2eXx0su+++yCxx8eNayHBLyvOh/HuxnnN0eZj6mAzwPC/YryMCAcs4yH35yN4tdaLxHhmJvoTTknud/WjwkfHTMosnxu30sdSwxV/o8MXgl0KFLW4wAEBmdCOMm81HNCOGXUa+q+lRaUXgHl3ednXM5jWwPxUOSw4t2oZgaQcSFgeDFjfSMhbez/4fUV6/tJfSoJ4W6VtWD+tXPqc0YM5IBwzIB+bxEY8ZSl/QbWzST7UMBrQHqxynDa4dIcxYxCH/XhLjczdzbfw0KIL9YGvGsOGI4r1vMoIj7XKKFrtE3xxYrybsDwjSu4BuQ5IqQqIkaBm0gRz1HGdI6IR4khPIziL11rQHlICC8yxkPG9WEEDRVpKAiBsVxTE1YqVzVTv0bQC3le1msCIiPp9yq/H0AauoYVyIdUUR4G0IsCXoJ8f86OXj3oGM4B9HJtMTdRCBgLwkF+D/M5IRwL6vsB9NmCeo2y1krAdD9j/uJ4a0vxcemYwY/qRzvWDqaitlXXMJ1ycxGwmJVkczlHmf9K8h2sJH7N9gzQwwITUgLQQyIZkKuE+GqRtaXzTMcsbgEGugnyvbDQUgcJ9YSxtnrNRxmzmGs3qygLfaRm5ZQD+FA0tqh+3yL3cEuVgLX7MDeBoyR5KMt1uvT4qGxgeXB12LPTYkoCLbSSWEiQmORbfOLYy6FC6pfKGyBl/TLfXbZx0XuaqataU7GZ4VqqkHBKLpblhn1UYNwsfwxAkwL+TFuAbbFH94DPl4d+OOLBrmdfrV4q1Ezy2Xx0GR3d6+9Csz7Y1Wn9sbo5bN8/B4zbocOtz7G99hyu9bc+MyTSb3LXb2b6+iQ54/kad+BL0tcaWL5MV/wz3/q/8MP1tLl+qeKbZ2EqLGSCyckbq2kpEOMxjzjGtYWGmEJubM/7V1NTnLX8PhzB9fXQFGSN+RxN4t8tnhZygWoLH+BDCXgF1gDGci8PYGNIpQxlcqgic8Qhru3e88tRyy/ts3wXWpiA+jK00AFW1vVlagyiie14ZjWF2vrmP7N7PGu5ZwEbswlq4T+MkbUwFJFEidfPhammDqFgfpmehDOIoeL16XJT1t+HOyDiFjbBMzaD9tPaWJhwGlYEiLCQiRIZg+fHA+hhRHyYB0tDLE21dUoiSrQPp2As7bpT+BWlYdkYvTjMbYVa3FNTlX1xvG7WYVNcNVZRmd5IjHwvYW9iEF/hw5BR9JqNi42DZ0J9mfv33vcXgLCv+n45RVHDPcZNmAETP4qhYr3rqsCBsFH2NTGmVva9KgXrq+Wttb9nPQjxZTEThlRQjoSs3yNrYyBgGvv3G+iMLd93ZrmNzUnm1yv+mpIyo4doaSJP33rsqrAA6gva9MkOj4Yht/rtB9Uz20MsWEaJXTuOedOmFCqqCjNZG9apl2dMcTmEjTqvMXdjysA3LqI4rAqzrPHlYqg4vb5om4FyXMVf+rDKIZPGRu11afl31A6hQmDUw7pR7m3qqybi0wSqOkPdxkNZUhGSAspgvuUd1JUpS/lTH5s9mw4A02FtdTU15Fe8GRdpFwOvr13M6vVVDgxfzJ2Fsvt1bCis4MnRBewYagWSFBh0tz5RoSUSs8DgGW4G6MCA+vAxlNm2sy01I2zX1cY/ROm7gTbZiMtBE0+iPxgU4BExogpHRT1cIsjhE+sBVTC/YBJxKvNZD6cMChDBqqErD9cCkJvCMCoteShtA2jxZJsC9aG0scJQ9ZCNWj7Sazj0uW/M86HKxjaygIikPtpZd+6q6otRVIMRICCOGMt1EDD1iYkrAVPtbF37wIE0O3gyAOMPjqCfm6KwgiA7qGpl8e7afgce2B3ySN/M13xD57jnKy8OpNp3xESsCOLba8kOsrzolLWf0QEmINfbV0DzGSCxfgLN39vKaAqx6jsrYIV6Hdb13PtgCKMxlRVo9lEGYvbKs+zK0ClqvuyODaUs80SFsUdfpO5NT8r1eYprQ7+6zUvY9G2fCLsyN6z47kNXTjTrgbzru5Vq7arbIlqO59p0o8++D5u2fqBf/f6vBqg+mZn81Pt+QxL93BT2N2vKHPD99R5LTQ2cZDdZj3na5H+v7/fx4uz/Hy5HLSM24OljsgHYXLdyDGhktzk/5/Fmm32MMct3K6aUB0cGxsKNb8oldxNcK7+y1P1uPjSQ581DHzFuNjez7gF87Lr9Pfu4Zfv4aHH3uV0Hbh9mnbGta1/v3vx1H7PM2mR59p/Zex/qpcVWW8bNfQDwxZqaWauZwhoI2cdT2/Rx1zagh5p4nHs9HjDs++rX4Rm3D+Oua9qc5u1P/D7mGWphIJ6LYXYrPe2f/wwbc1BA3l/msbHWzcxwV97FlXXLnNV/dpm34VQszWsvz4MmG+fzddyERbE2b8uXVxOW2vfdt+/Kw7NjAQDX69DMV31dz8aE2/epHUtr/1iYiPk6gELd/CBfdN3ciiVH+l0kd22fLlXnCGKF4Ntg4Sl8m9Zl55Tn+7EZsz7n+7m3cVi86d2TArBZ0Cu2axDYzouAhef7uZyH3pcP9A9MYuYdawutQYGxOvPSZg7r2xhujMX+y+lP/PfXgU0s1o1pp5ZV9utF29wOZCyvmTnuHxLUP6tF2S3/+b5+Nyab/vhsyox4MNDYJ8tr7Wjmqfq5ey8YSIWrdGw5qjKprh8OEhqDXFuIAItWT26sWugNbT8FBjnFY5Ca5iY/SV8x+VAXthatuNDHrTE9/oHu1/5uXJvPux9nt3ZuLgNj6n3bbs2vreFK6CFHnpbn5731ab+O9sDGPxI03xOWy742vLvuyiHGVhXWpQ7+0JEX9/v2/ezluk7u1vVNwNIGn56044PA61af/PvnPt+lL/lJeP7+j90EfGS9X1bf7lH00fd99OcfqPMrpx9hXH6ePi39xIAlEf02AP85gG9DpvZPM/OfJKLPAPxXAH4HgF8B8AeY+YdERAD+JIB/CcAZwB9i5l/+UB2FQwvrMdfUmD9jCy3MhwV7X2rasG9AB0yedZN4j51pOqZV4lE61g1AK2eKWYPIi8/kFHOLQenZMUu2UUou9qQPkM4QkDG4eJHeJxToIMl8GY3ls2uRGJc1YYhVY0lKiA1ro/VhTD3W5y0/s6JxHi0ZC2WbGos/GdVPzgMw81EcorA7BtZ8zMpSaQPe7HplYcKMJTK2KZcIhjA2nmUzf9D2G6NtTjtgbfNLeppvPpjXRYDuNAgz6GNDevaJgWYW3MNk9FiUxtYwSz5jP8w/r5Sw8YWT+9FYFjN9jrEzIyFwU50tOTQ1WQ/smv8co/kBdl9CAV45a4gYDZ/hgc/eV9WPofWRmVCLACeuLii5mo6KX1yPY9hiMrpfIfPhrH7uzJeOxK8QcPsHV14tcetH6MBTZeobR21zLdTiIFqZzWcvbud06/enbTETKWMbKpqp34Y9Uf/CuurcjX1sJcakjadeM7O0DVPl69drJYCSmCgGFd+wz2oOnYhQgNXiGgLSdkDjPJpNpSt7ieCxtI0/+w2qmiXKGIjyMKv5s/cZ7W1H90/c+ANyZzRs0woog0FPY90ZuNiAEm2f97MzBsXiCQYVzLi1gViDABp7BPt72dqpr5mamimKmtpZPETontPHVAS2jJUl72/GCsLUnHBzj4K9ZvKn+Z+Y8bkxNZDFBpBaJm2b9kvM4ciNGTRER9iYAfYvgO4Qgyj1Nt9IYOMXZ4q9TezJXH8ZKhwiZXr1WtKDSw7UxqOV1cCUqZGqKZ8DZSZm0wALQ0VltP+2Hiv1uqK2MWm+YGXI9Zo8OvpqycoFd/VWY6NqJGeeuMUpdi/sUeX8ChkOPDkQ4vH9kxiSBBEHs76Qlb8DjTo3rf5AW4Bo9QSdQ1vb1g870yJbi/15TgzUyAie1YRbs1aOmr368Clt7LCt4xYY8+NspqwebPYm3QaNIhjTXz8GbGyAtR+HG/kawLK5sfuwu/Zlad+u3XPtOUC3WU+36vNj+kzfnwWIt+751PI+lL4k/+a78Cnl/yZKDDwhYn6W0k+SscwA/ggz/zIRvQDwV4nofwDwhwD8ZWb+40T0xwD8MQB/FMDvA/A79e+fBPCf6OuzaakRf/Pd5y1EyKMyUcdhxRQzvricxJwsJxyHFe+vE06T+D4+XCekWBADY80RLw4z3l8nTEPGWgKWnFqsvc/uz3h3nZBCbaDIwAgz4dXdBedlkLAVy4D744zHq7CCp8Pc4h5aqlXM9EYNCWK+kqWItD3X0PwJDRisa0RU/0GgA4B1Fb/PkiOmw+ryMubzIP6CQ8F8GTBM4i+5LEnDOwDTYWmhQywYuwGkltexQWnIzU9S4glW5CUiDlU28rF/+0MsqCU2Hx0zp4qxtnAecq/kS4P4W46T5M/XhOG0IF8HhFFBsfpTpVNGyQEh8kY4hBWY2QY1HsRPqBpIIPWjUzO1NIo53fyDI8DA8noRn8A1aOBy9c/kDgDrRb42NMrOgcysiQCeY5egtVALGq+PEne/IXtlD1zkHloJ66nL0FJSn6kA0BKwHov8WKsvGjQotvny0EH92tYAmElbJsT3EfnAcv9KjSFAwDaIeYAKaFA/ndcfEFoDsgYcp0KoAeJntKrQxFhBl4hyLN0kx8ovBFoJZaygNaBYnD/ziwto99jGhIcqoTQOEvKijLqLsLzaNlLTq+aLAyCcA+rIYDN70zbTSihHtQfSftom1YcbITUbq2r2Rhq7Dyyb1arhPkjDjaR3EahAfl36hmkJKBYHsaKFvAAT6rH2H1MF1jX1OHLhElBeFKR3EeVUO5MDID0KYOIg4xtWQj3V1nYLeVGOFWFWQQ8HbsZ3AeuLKKFBZkJ0vlXxQlhfyDjHlVAnCbNRDhVxDqqmbR2Uui3cRJo1dqHeFxZq7TS13jjL5r4czOfJKUhmCzcCVZHUskcDTBq+goAyihp3Hbj5BFGRTbWBk+FdQDlwUx4WNWRCvqs9JqKKn6QzYb2vCJlkDO4Zw4PGPFQwEK9aj/k2qrq3qXlbmwG0OJZhlTAgFoaixy0EagLShTab6jqIYqdtZOOV2ma6AeKAFg5ErkvbygFtvIL65NUo45RPjPGd1FXHXr6MG1ThmTXcCFpYkLCiKTuXg9RjoTxMQZ2KqF0HVRAtOt51EJVhQMJMUFFlXgD5RC1MhVehriO1MBuhaHgKA6lZ2lRGIJ2hoUao3W8hO3q4EUK6aH/Vdy6dgeUlPjlJuBH9jq4d4FAB8hFNOTvOfc1zlLFOF6hqvYwvbD1EUQQHoSn2wualgXpsQJSEG2FpC/c6a8IGhNq6ZHLhRlYbO+zCjeBLwo10lWdTY62DqNGC0FVhr9z6HrIoHXOipqhr42UKpnGVw6Ga6GbokFC490VVdamwfhexibW5UT5VYGTK6zfjWKrPqqlR95AkvQwOpuLsFsIeBHMHdx74hsIN/D+J13kLOHKfc3+Y0pntrYKt5TfVWlNm9sqwt1Rhbyqz7kD0BjTfyLcB1DfShg2/cf1m8uOyf/2S9HUGmz9LifhjZH9/HBUR/SUAf0r//llm/rtE9B0A/yMz/y4i+k/1/z+n+f+G5XuuzBe/6xf4H/uP/xVcc2qqp6UGrCUgF/H1MnVPY8LmLKIt5iPV/N7KLgajMmSlBqwqQiNsEjVGKkYxkDXBH8DYpdiCzBsgtLRRDa0SUB5wfif6zYixYl1TYznMV2lv7mXA0wSB7Fop1AV/CjVhne6jJPfnNTYzOwrc4h42s7pQ4Z5PErzdmZeKf4/GYdyZmzTfQWMIyQSMHGsVugAPF2piMRQEmBYdSx/AngJru7tKZWOQHGMl7bWnMfpmX5kXAE0UIx1WEKHFi/Tsi+W1zX9QM6pWtnvwNVaHuIHsWqiZ7Vm5TdHSH8NZ2aT+QTbExkYxRFzHiTt4/xm7Jr493MQ5ZD7F76kJ6Rio0vY/Ucg09s+/B5qpYFMGBRqb1Riu5l/ET8wHpd87dmuv7gjXNgPkWV/NX4dpy4jF3pdmDteEYbQeBaQUuPs6MTpw9m3xjJpXtbT6rG5GA80YVT30Gnt/ImvcHdc/M8NbqffX1qc3s0sshwMK3De/zqOCSGOlnEDHZjyKK8/fPxVgjv1eEwSx9pkvVuAu4NHKou2PvB8bi5UZdvltrK3MNtbYfuY3EvtDDgigYz24aeaYFd2csm2+9P1YZVzskRUlDxmTqXkByCHG0hlOyiTX3FrxBxcA2mGBbLK4iW0A6IcjJhri+6qHINIf3m4uK4kqqg2bsY9wc2RtYTRmkqNTYLUNMlxda5BDIO5tbmySth1mJqoMp8wX2hhbnFHfJ2NYTRilf2e4s7U6Hkxoa5NWN1a1C66Y+Eoz27Sx494nYR37prsxbm7sDWxWC+VljHCSA49PTS0EBzuhG6CNgR1qeJBm34E2Z8oCdyaONp+1ufeb+sb+y3eKWA4CSJV+29qrtPlZkfG1tncz2D2IobLbv9szwZh3y2tt9s1xeQzkeoaeqrzfh7vYz6tf3xvA4sfDDnH8s9PKYTdefvw8Smp193GUzK6c/fLwnzGepGeKbu2j7ZbpNru4//+rpFttv9XOW2b5X1bWV23Lvs6Puf9HrOs5JvdW+j//xL/zV5n5H/+0mn486Tu/+xv8B//cv/BjL/dP/J6/8FPvG/Ab5GNJRL8DwD8K4H8D8G0HFv8exFQWAH4rgL/tbvs7eu1ZYDmFjG+f3uMXDu9wKQN+7fISKRR8Y7zg9XDG33j/bZzSgnfLAZ8fHvDdx9f4zukdMgf82vkF7oYFp7TgB9c7/Pb7H+JXH1/hG4czrnnAF9cT3l8nHMcVv+fb38V3H1/jmFYkqrjkAUMs+P75DpUJv+vz7+GH8wkBjLfzAb/0+Rt89+EVAOBbnz3i/TIJcNUVv9aAl+OMu2HG3/zic8RQcT8tWErEkiNyDZjnAZ+/esCo5qvvrgdMKeM4rBsK/d31gFxEVfY733iHyoT31wkvDjP+7g9e4XBc8NmLK77/9h4v766YUsbby0EUTwPjt/+WL/C4jIih4t3lgFfHK1KouOaEMRa8uRwaUGYmvHr1gPM8YlH11yEWPJwn3J1mzKsoxhqQPAwZl2XAZ3dnvL9OTVTmOK54f5nw8nTFm4cTXh0vuK4Jr45XvDkf8dndGY/LgLfvT/jWZ+/xxbsTTndXHIaMN+9OWK8J3/jsAefrhHHIYoqr4HNdRSnT2vDy5aWxyymJiMX5PGE6Ligl4PX9BWMs+Nt/63OAgc9+8S2uy4D5OmI8ZJRCuD/NyCq4U5jw+F7Mrw+nBbWKcu6qSrDzVZhhZmB9N4HGiuG4ouSIYVqxXAZMpxXzZcB0tzTxlFLkoGF9Nwm7+HoBFIAPY8blvSi5lvcDhlezKDQuAXGsjRk29dbjyyvWJSFfEsZ7oQfWOYF/7QA+FQyv5q5mCYAiI6pCpil6CqscwEXCUlAQgFnPojLKS5DAooER7zLKWZjkcL+C344ILxfURQ4yKKnqZCbwJSHcrVLOJMBvuJe5oMCiiMnURB7iXUZ5SIj3GeXdADoWqTNV5GsSBU4oi1wBDAyaVIX5hyNwqIh3Mh91jlLuHECvJMgmRRb1TgVuje2NFXROsrE9ZWWTVcmRIaBM1VB5DqCpIvz6KGzKL8zNFBXXCLrLHbzngPA2CcPzqvQfVNs8T6UpmsY3CfXzBeHXR5RXBTSUVm78wQAeZIPPiREeIuqr3A4/6DHJxvNlBp2jbNoNZAEY/86I5VsFmCroHBFmYWOpENJ7ks+IQdcAvisI7xLqXUF4jMK0OuaMrgFUAuqhIr6PwlReA8p9RXyIwgKHDlTCY0RYCPllQbiG1gcQEK4BdawCOkZGOAfERRhGKCOe3ohKb74vSG+3bC4tAm5YN9yH747ILxj5KCxoOgsDunyjIp41HNUoQGD63oD584L0EJEeCPPnBYfvDlheV5h55/iDoBtbAipQTsoy577RT49ysFBG2eTECyHfs4ubKWMR3wXUARjeGzCV/OXAjd0DSaxjUanszCtHIL23Qw6rl5DvZBzKgRs7W0b5bHnNOP5tiV1aJjSWUmK7CgtYJ0Z6CCgHYHgEwGpuOUg9y0tGegwYHoH1XoBbWAVoFWPrQmczyyQMIVg+pyysHRhYXgmDZsxmWKS89W7LrOVT73Oce1njW8lTY/8sqS5ZPsr/ywtgfC/vrczxHePy+acDy+FRygWEfQ8q2BKyXBcWXfLVQedcGcPxnbBtZVL2UtteEzA8AExyr1FhZikAACAASURBVIGqsPbYiVWZWwOGnIDpC2C9F2uROgLpURjfaqAW6LFASeaiHHq8WWMhqQL5SA2Ek9ZdE2ksWmEj01mYyBqFWRSmmlrc3DJpPY87xnKUdqUzI59I+8FtjUfVx6pDX/fGqMo4dEBZk9a5crMUMKa8xi1gb+zuwMpYc+ufgFNuYLemfhAR8hbYWtzYjQl1A8CdOfRsa9A4lsLY6i07MO9BZ7MeUADezY+3688rwlpfa+yMavAstz0i9JqfX8mwK1vbRuVpXM8n+bSfpHNpaQPu3MHAzeu7sfRl33p9Usf+EOBHBMG/UYlBP9OmsD9xxpKI7gH8TwD+PWb+i0T0hplfu89/yMzfIKL/DsAfZ+b/Ra//ZQB/lJn/yq68PwzgDwPA4dsvfu8/9Wf/zeY3uCoDOSijthSLY0nqZxgxKHs4O5aRmTCm0q5ZrMTiANKSOzC0lNX/cEwSloS0LlPOBNDVJt0i8sqGOXeW0RhJmxJjOk15z5jJjVmmqZuq7x2A5lu3LqmF38irmJ5KWdSeOGkojS318f2sDVVZ2ubvp/6CXDtzWUtEiKX5x/W5EobS++QRsPWbU1+7vX8emFByQExdZdLayIUQh9pMa/3Th+3Bbea4Lh6fNIqbxD6AHqJDAU08CIhorCILi+rH/ImPHKGxlI2FBDbx9prfnvqoNV81Y97sBywHAQcuxiBFZdiUlSI1zW3+aP54lNFj9KnPVWMNlyDlGqvkT3Jv+O/cfG8slBcxsfIqOsuW6rZ9hJ6n+cZpmXa/ZwTtq2aMX6rd9FfnceO/15hp7qffFrLBm/mya6fda9ftfkuenbQ1tLftsXYQmq+csEKuH561A4RVqgJ+NuycH2eCgKuB26tPBqAAdN+l1MtrfmzOfJap++eFOYiJL/X2NAYiC6izMWvsVIQwYfvY8qztj+jmycZe+rAHNgSF1DzPzTu5smwMlDVsJs7GohtTpu0y0OXZBxvuOKtcvwsDgEoNTDa2hNWceJTNashiYhtWMe2VgUEzn91sED0j7HwTWeedimu/3azj3hg5n0K/V9aLGzu/PjyriQ6+Wp9qv8c+C4u1eTvWfjNL1cZeb7flTAJmqKh57+g36WIa3sY/9FcVIe9zUBzbVnrZ7bqFTTDW0rGC3pQwZDMhls18KN7HEo3VDBY+Qtd4WAXwfmpq8wn97vhHkmNHScGO9Q9w14KbP83TxsmeX7zbTLvrgNRhQN4+3/hwuk285Wmgw234/Zje2vxvjGrcmvIs5sbEE7s1a6y4jgknVxb3/9s4ebDg1ueGmQ7ba18GYJ6Ez8D2nvadco/uPZP5Y/Ox3L/313h3/cY1M2v1pqyb9x8ov+Xb13OrL/v7b+Tz779S+ir5nxubj0i32vXLf+aP/NRZvV/43Z/xv/Znf/yM5b//j/w3P/W+AT9hxpKIBgB/AcB/ycx/US//GhF9x5nCfk+vfxfAb3O3/5Je2yRm/tMA/jQAvP4HfgvfDQsuWdQa74cFS41ivloDjsPamMLChLtxbWDzblqaqI6J20yDhIcAMaZhaQzVvNpnPewCM+FO/TXnNWGIIqgyBm6xMysL8CM1q93IzGuYi/vjDGZCLkHj/kkaYsVl6b6Zowtd4f0sxyR+lXToojzTIEJER42RmWvAUftba8Do6sklYBiqMm91Z+5LGIZ103Zh2IAQ1maeGqasoDRv1DkNqOYcEIc+BlZuKQHDsG6Eb4ahoBQBoxZv8jAuzXR41HiF6xo34jdWV1978n92pqD23B2OpamAlixiQKeXEmZgvg4IqSCMThDHNuY6LkFDF5j5L1cVOGFCnJZmThzTqiCcWj4zGw4uNmhrM4CooRxK7rt3ZkIYVnk9ybiR9rEdWhC3a0XXUtD/zWw13Uu/a+lj5+vYvkc3d3WHHbYu/BPbQDLpnMfTKrEuA2Bgum0gmikwJDYgoZvE2rwR+hoqJPHo9JWrtpXRRX+03DZX1tYTN79buSCHORSgYQnQwUpw99tmxq558ZudKSy7dtB9bT68tntpJs9ul2j+vbBg6VYX97UCAHRXwWsE3ZV+4GDTdNIyqqplEqvJrVTVDibWACRuBxFsn79YgVV8QvmwPUyiVCVsgfUzB/CxdHEYLxpjByME2UCeeGcKi74hcGbLbO11wiuwftj6qgQ+8tbMl9GBhpnC2uaeHCDTtVVeVYn9aGPXBIq6iIgBsGqHIoHFf3YNqC+rgGXubW/ttHZ7cGyAWse7HXTsRE82Bw7ejNvK9JvDW6bMjO19vJub/QY3MCgH5G84E2qfyI1bwLZ9rt0tRqB9F+wZYe22aSzUTYOdiWubI0I/IHgm3+bQpfUDbe7l8Mr1P1T3Ofq4mw+6lmlmwZ+c7PCArS+2JqDCRwaAZPw2zJe6R3hBJGJsDk88eNs8aj2o0/Fgx0y2g4X6/Ma/MV0OCLV22Lpw/XmigLtfi1aRXjff49ZPHYd23cBvaxi28+pNYdHHbQ/MnoA8NyY3QY5fS/ul78bjJqiyIgjtkK7dyzfy7K8FfnLfR4Gkfd/be3s+uN9VFtC4T/6R09chPS3zY9rxTNmfev+PMz3p59cg1ZvKbz8b6SepCksA/gyAv87M/6H76L8F8AcB/HF9/Uvu+r9NRH8eItrz9kP+lYCowr6bD6JSyoTLOnSlSmI8zBZeBE1x1DbUlzq0fKUOiKFizkN7zl3X1Ji8FGsDbT5cw7zK8Ik/pIb9qBJ77fGqgemVYdwyltTY0nfLoatLunx7X0gDQm1jr2nRuJuiPqlsbE0gAq6XUTfSjEsZEdRfcqmpsYch1uZDtyhrKAXKN3Rd0pZEMDBj86z9aeDzxpOWiJFdmXKPKBTkdRvGIK/m60lYrsIW5iU2sLHOErcrJG7AROrpY+vfkxOesZRntwHSDeH5rYSaCUMB57T5QaXdmPM19bLtR1XXSp5jm7P1vN14EQG5ig9nbgoN2Gzq89lUFlyDiVFKfLpxtYmx/tkt+rwqhTabz7KOW7Ef2+D6zeF+Dncb0OoXg/3vGcwgYLDVgV0bqxsv26Q49oTbJlLrDRpWwV4du8XuPfsfbm1/2W98bd3a5tnyuvXsUzsQ9mvM/ndlsLX9UX2iHbvIvi5rRkltg9uutba7schJQOFjbL6BLbOtB7tHmeDma3g2pRwGrgY+3f3vEmjgfi+j+1Jnks8sbwBwCZ0FI1eW5WkbN7dDa+DDbVQhefzGWz7brT/3ejNgN9wmNTwtpg3n2ySbXPtc+8oJ2/XJ2LC+9j+9j1vGzHw9rbn++WLPt8YQ9zpto+1vfuJD6Mb7YxhLi8fXliljo8a6YSz1M3obtt9H/10Oeqn2+w30NGZNmcDnfBw3zyDGFkjY+Yo+ezZKueSuOwXgDQtLvU1Srh7stXEO27VhbHsJneEibfvOAuCrJCph4xsqFWp/47Ye9vMGbBnF2v8XUNrH2ZeJXRn+x9cEiZ6wam5hNJBpbQrbuvdso1XU/QCp12eHU5aTzV9zv+Zpm0e/o6GIueYTVnQ3b5v+Am2d+L74/m0e4Lvf+1vXbjJyt74TvogboPRJurGsbIx8vR8Esn7Ob5W5b+ut68+lJ+PwTOYbZfnDio8Ccp/4FftKwPDTv8Y/Tz/B9JNkLP9pAP8qgL9GRP+7Xvt3IYDyvyaifwPA3wLwB/Sz/x4SauT/hoQb+de/rILKhLUGnOcRhamFoChVVEvF30+AXlYBnnURdjOp6at9fp2HJnpjISIMFJ7PE6KWBU4KNDSgNoBlHlpYhRAYyzI2M9ZlCQ1EAmjfxLyKomwasrJIoYFAQMreBKYmloCqDnhamAi7LyvQFbBMGEZRcDUz2FpDY5cMIK5z6mahBJSll2FmoJ6xzGpG2gEvhAGyU2Z3am3mpLXExhYxC/NCsaJmCbLNq/zScBG11rrI+zBUlEXYNy4aPiNVUATKEjYmpRtBGUIDg1UFjfxT0VRciSDiNkyIJ2Ehy1VBbHR5TMjEfryVDeKiOzG3MfTCQDSIOaiFcmivFsLATObc05kSA6FuxWVKELBSARqLsEnGhFTgiTmhmYCm2gNam6+gqcUG3gLUHRN3EzQSnBmrq9+AZNC6TWzGAyq7RxVlNyzCJmyC/lJ48Rmr0+71INKzP1aX/frp+HczWc2XuDNgxbXTM5LWhltjA2xNh3Uc+KiB3a9RwJ3ma6aRBoLHKmX5YOH6I95EZxjgSYRj+CDj6SX8+eCYJ4aou66hb/5042xiIo1N0E1ZfZFBc2ymrh408rHIZwxld6j5LPLAG/M/MIAofaOiqsCVwKF28Rtro03RWOUxmDWfMWU25tXWmfSrBm2DsjF1cn2bIPn0O9Q22fqdqC9LV/0ldCXfOfQDAMt7quLzmfr/5UUBzaGTcmOFsWyAzm0DzwIqa2Ok9XqEtF9/B7qirIxVPdquWtdBtpAVUkc9sNuAUwMD7MF/JV0D7nPHRJlgTbnj7ffRPzsYTpWXmt+rZ5zCQqgn3yddO9rudhinQj6UIf6ZgFMXlXJFMVjvbeaSFUHnuvmAZltr1L7nVAh1qB3YVgLH2tRXwwox211F7KkBKoYq3346W1BTbYq7H2Qs2/OVWjvN/H0vtONN4zcHE/53q+4awujqvzqGG8EgSw6s1b3yKeDAIDZgZQN+oOXbI0vzMZ7mkXHpjzbL08O/uLZRzw/giQmtZ1p9v0PpYK2JcO2+Iz55v8R9/zy49Yzuk+QPW1xbtqc7u+tWj587W++0zbc9rUfr05O2+Pt2+W6Bv1t93Rb26eknxVreAt1fJzbyyxLzLn7xz1j6iQFL9ZV8buSeGBezOHv+W1+ljjFkfOv4iMOLN1hKxDkLS3iIGYEqHtdJlFlrxBAKrnnAIa0oHHBZBw0fImzkMa24ZGEuWQHrkkWQ5Rdfv8N5FTbT4l0C3YfzblxwzQmBuIU2uSiDaXEim88lif9ligWRGGc1d7WYjlXNXRnA6TAjKghcNCTIEOvG6Ter6WquAePp2pjZECou84hxXDDEIuI1cW39NZPVSU1RAfEZHaJsjksNT/ICPY6lXTM/0Ri3ITkAAeelBIxJYnmar2QIcj3FimVNGO5zC29irybEM97NTWjIlG9rCbh7dRUT27j112zxGXWMhjvRn/dxKeU+bm0kYpwfhN0+vb6IkI6KydgBgo9bmFcxn40H+YU0/1gALU4lmES5NlWkqW4Uck1sp8Vk1PzMpII5hHhQ8RSgmbcSAXUNiHeiAmJhVrpJLAAG4knnKAfEadX+E+o1goYqAjuFOstHaH6kzayV0MxOjdUFK5g2wMwAJYigTZZDi3DKqEtEOOVuQqoHH8wkCrKj5KdJyjGf0BbwnrtJKE0MzgHhIOXSQcFbUF9ZM4HNoQHMZsI6aziNg4jksJktFmrlNMCvG2cLASMTrUBtUsEeVX6ViVZAStzB71UPMe66iA5yAE/m+KU7sVXNWjWMSfsMAMfaNx2LmqBeAzDWLSM0hw6IFXjzofRfXwXO7MK+8FgbMKZzFHBqZqbOBLJ9Bsg8n4rEg5wE2PNQt092M3MbKyiHpk7KGmansXoGxrQ+1sMSDtz9NiuAAQ2wWniY5o+qYEH6w02ldGOKR1DlWIiY0AA5TGAgLLJO6lR7HEj9LD4GlKMA4vgYUA+M+BBQJ5ZhJ62bgWCbfxMLcSaALdxIlLEhFb/pcRfR/VaD1OuWQAf6OtfhGtrYsZ4JcQDiue+2OTDCJYh/orJ+Tfk0MMKj9CO9CwoW0dEAdxVRjox4Vobvqt8JxwrXQQSQwqLhQyo1P0vrr4w/GhAZNNyIXQtqWlwn7j6RBC2HmoBL2+RrmA0QxLeY5drwXgGEsWEzSdmAhLzJ4jMb3oeuwEuQth8/fYeazqExnhIyQ+ehoglgcWCEtX9HDfyEBx1/XdvGvnEAwoOBVblHACfhCfhyYCu+kfEydtjCiGxAh7G+QGM4WxgTHXcrb2Nau2c3gx4OuO9+Z49dG8nlo16OhY+p7qy3tbH0OvZMbme54dapv9bNhzd+wi5tQKUDYz3DjTy7JbJnLG8BHc8Ye1/OJ7E79+DWtUMy3rjm3+/BJQEWusV8MDftsms2nh/JVH4yq9rq3a4Xf/3ZtAfruPH+Q/f+PP3U02+IKuxPKp3iin/iG7+CH6x3KAgYqGDliEsZsNSEXzq9wVwTppBxKSOOccH7LIqeL9IVj3lCBWEMGT9cTviluze4qHzXMa54MVxxKQN+MN/hl+7eYK0RmQMSVWQOuIsLAjF+MJ9wOGVUEI5xxdvlgPuXAmiuZUAi8XczQBiIcS0JuUZ84/UZhWmTz+r/4XJs94xRwNlSEwIYFYQARgoFKVSMIePdctS8Gdcy4JvTIy5lwDmPeD2ecc4jrmXA/aBgiwlv5yMmFR86xBXXMrQQLLkGnNKieUMD60MsSPrUXGrEKS24lgFjkHqTE09KQVR0p5jbGMwl4ZjW9nrJAyLV9v68ikrti2HG+3WC+dHmGvBinJGo4O1ybPPgX6OFfdG6LlmY4UHHD0CrOxDjsg4oTPid3/x1AMD3L/eIoWKKGWuNLY/56UZiHJOAtbnI1yfXgKSA28YSAO6GBWuJmEtq4W2GWHBZRVV41YMJKzcQ4zTImnpYptaHa064GxeUGnBMKx7WEZG4lenvJ2I8LiOGWDDGgmuWNg6h4uV0xSUPeFzGpjZsIXXs9Iy0PcwieGV+xrYOp1iwqkKu1X/NCYeUEUPFZR1wNy54XMa2DooerARijLFgKbG9AsBagsTsZjRxrRQqGBDf55RxWRPuxrX1J+uBxargdYg9PFCphEASz9aUlgE5vImhYggVj4uZrod2nfQAx9oyJvGbtkMdC1tkZdmhjv3/4jAjEOOH52ObuzGVFuIIkIOZacibQyW5zqgMZFVOZiacpgXvLge8PF6bVYaVc5pkPYjQmNRzXrop/5Rkp3d1/t/ZhLIAvNS4vbYmh1Ra346jxPy1eZjXhMO4YsmprVtvGh5DbQJqJmSW1G/dfMNZx1rmqiDo59YedmvORNNqlbFNGotY5kv6CqD5vi85tgMQO+Sxw6a7w4I1R51DqdtUos2qxHydD+OK83VEShVjyrjMIw7j2ixbAGBU/2rSZ8xaYj9M0rIshJSVm1LB6sTi7ADM1KCnMbcxsIM6E2sDgEEF1tgd6DGj1WPXUypNDM4O6HybljnheFxQmZDzNryWWOuETWgqK9/yMQPrkhBTxTBkLEtqB34h1NZHZhLLFoKoStt4uDBIgFjHNAE0tWIhgvgVx9rM5c1n3dwqxDdeDqhMXM5cR+DnNNaNtU4TiRsK8vXTtz4xSbmAHMjZLjmQWBlZbOUQ+4GjHeLFJM+1FnLJDiSqxEu2A0NbZP6gdmPur2MWxyJjoYdyIYkbyZNNuR0oWSgoPYhpoay0Td5v1kTfLMQUNO+TcFTFlWHiVhbqiS20lpbtrWh8d/Z+yWbF4Ok3y19p685gn1veGyarzey8scG8+Ux++NAtN/aENrvPNkCQtnU31O/QmBt/39wvTXuQ6d/vwVdrn40T37iHtybQHwlYN+4Mvp9MH9ePfX++LM9HgMhbuNPf/nVIP1eF/U2aPv8HP+ff/5/9fqwcUDkg67GQn7DKhFwj6m657fMA+sPglE292M7GjxBowM6X68v0QHL/mU/7Om4lu9cA362ybl0rypIBnYG0vF7h1ve/chcoauU4tvW5e271R+6VOKL7flaWzbSBAPODtVe714+9le/f++u+Dbfue27MDDTuD+D8s9Vf93U8l/brZX/9uTbJ9e2GYrO52NX7XBkmELXPaxsz2+T7tjy3Bj1b6x8XPqbrvi378f9QW78s+fHwY7HN5H7gn/kR9O24NXf7ufnQd/LZ3z+779nOUO9T2LbraXu2/d58tt88PbNQLUzMvsGbMbANpPXX2oZPmLOPOWXez4ufi/08Kmjw/b81/5t14cv42M0ZYbsh3U+w/b/fSG4a4a77+fBl2f/P1bu/197f2gl+WRuea+ettO+nv7YpX663jfq+vx/a/N6q60NteO7e/VwCrZ+bff0N80e7j/fA4SukW+XdTLfW6Q3csXn/oY31jesbs09i3DQD9UvCj8mtdPPH7nb9T1bWs3ODvubr7n/XRuD2Mvfs4X5Jk303nuvO7qfAX9v078ba/SBL9xXm/Ln2fDQo+9h8XyE9W/cz17+0rR/Txq/Yj68EWr9C+f/Hf/DTj2P5W/6hb/K//F/8iz/2cv/U7/2zP/W+AV9zxvIQV/x9p19HBWHliLXGxlxWlmv2amwmIOybAcNAFZljA4nG3BgYNVZwqamxdAbM5hpROWBSnfCVAwaqDWwak7S3pc7KhAWqG6Yyc2jtsFP/RAIkM8uJqwezz4FVa5+ULcyUtL8oeI4IVFE5IIXyDFAVhtLEgSwZuNyD5n0oFgOcHgxbnwzA26sHyz5/M8etEZGM/RJT4fTML6MfG2uTZ3gBtLG0PAGMS1amOq3IHFrdHnxae419uwUQfb2rhqDxde1f9/NmbKExdnb/qnU+d0CwP9QA8KQPS46Igdv91g9Lt8D4c4cOe8bKmENj9VLsIle+PDtosFc7YOj1YTOmPm8uW39l31YPlK191j8vuGXsWbS1YSbUrp+W7Hsb3Ri3sXUHETZPRZmoacib9Wz3WMnGWJkZtu+vnwNjnXKOjRVq/VBGy4P7PfjaA/+NSJT6nJuisqkqe/bL2mqm6/b5LUVhY4waQ7Or2+qXMrf5zaTez6m9VmU9LMyRsSjMPYxRY078JtPmKcfmU27zAe7iab6+WggxyfeFi/iXV2VjGi40ZdPaTcSfHAI5xmnDCvlh82DehSgCOqhv4+tMyj3oarFS9R4uPYzSE9XiosrVphy8A+9988+dtdorzBJ3f2I1/2amDhD85txMeQP3cuyasTd7ETLz5fbhh5i6/zTcvVZuGwLN40O9mB+jmWPbHFVq4UI+KZkaMaA+d32cLLyP+RzvmS/2IUY8yIKbT9fWDSLy1yH3UXYgmXgrDmTPUl+/+bLuDlHIwPb+YIE6CGX3P3SqNmJs1ka73psl9wen/gs08/E2FtC23Zoaa6Orm2DfeW717AGkH8a25G/4qlqfbocboQ+C9ufKegJc4fpmz4xbhwW7fjypx9+3f30uuTZs2vGhvrj7PoV/+lHB6P7jrww0P7Ken6cff/paA8tzGfHLb38bDjFjqd7HckUKFW/mo5hylYQpZlzygGNaUZmauWVSE8yTmluOoaBCFGYXNUt7NV1bfh+jclaTsBfTjDmnBp68j+Uh5RZf05IBhxQq3s8jAqGZgFXdEOYacBhyAwEGCoa4BYJmBpZrwGmUvtnG/vE6IsWKMRVclgFjyoiBMefY2KzjuIp6JoC1xGZCZ5v5OW/N3oZYNjE+A8mGf+/rCHR/zDHlZjIWY22b8CGKgNKQSgOUS46YBjGnW5aEccxY16RiSmJuVQphmnLb7PZNMVq4FNtoD0Npm3TLYxt1QMzUAjEeHsRE+nhaUKuYctlGOqlJpG1ATVQpJjMV6xtkMYUSP72yRlBAi/EZnA+pve435HkRRdowFtlzkAkzyQa5rgFplPHyQky2YWYGYpL6fGzTUkLzsYyDxSLtYIZUtdL8NkEuTAg5UJC7CZeJM4Uk7WIG4lhR5ogwlr5RIjTzsJoDQqob/0jz7wT65sraFgYRcwpjER/LKP59FMT30kzpTISJArfNDc+iphpG8bGsFrKjEmgs/YfWxoIhwky63r1ZFxRs2OcmxORNwfgqgjfXU+5ma4X6/bapNlOw0ftY6qttqgHJNxXx3TTBH/suzg4gBKkHLozQpg4T93Bl0yVgOZqPJW18uWgh+Qyu3CWo6BFtFWotjwECE20pIpryRBwJUHGabdxNqdiBGgUle39MBDQfyzw4H0tru4a5MBATHwPqyCjqe4gsPoGr+lICCgaYEK7UroeVUA6McBFfQAMA5uMXFBDVoYMYYqAG7sIuKiRkvn7ejwxOkCiaT6ZufmsSESLbSIk4i65pE4YJQFz6nHFQn0INWcHJxccM8n+dGOks81GdeqkwkGjCOiGTBFrP+tnOxzJkFfExH8kqIMHiUoLQfSyj+P0BcD6W0tkyMUKROsxPj9j8UdHmcONjqflr6jFKbUwpo4+9ChDVUXxemyBRAOIM5LtP322GReOdQtts31f1sZTxY8SFmm9j87GcSX1sHSjU9ocFfd0AXfAmuOvsgZr0per42Fh/rI9lGzcX+9OzmRsfS60veCBrZQfv46jPbZfPRJg4EEIGaqLeP33cbuKPuvp8PU0AKfQ+GMt7yy/T6mhlOQDW2GH0a15l2g48OsNIPQboPu3LBTbAfmNa6+vE7tq+PNe2DVPs4lg2X0rCxseyFUXU7pFxIFB1hX0JuPWvNxnbDwDajwaW+zI/lP9jgfQz6ZPB6U8glaec/89M+loDy0NY8Q+//FX8MJ+Qa8R0XNXHckTmgM9fiI+hMY5jyLiUAZUDfuvpLTKHJuzzmEd8fnhoDN14yphCwVwjHtYJn03nDZNYmTAGeRqe84BvTP3aQ57wndM7VBCuWerfs1RLScgc8It3b1FBzR8RQGPRriU1djVR2Zj7Nh8uZUvHWPCwql+U+hyeXknsymsZcHi5YqlJfSlzY2rn3P3/ppQFIIMQSdiRIZbG1AFo7KFnFe0+z45Z3iEUzCUJU6lPgqVGjKFgqbH5MgJbwaAYKqaXGVf1z1yqxuiMArYN6Ft9zcfK/pSlNMDvWVEDx0A/HPjOL74DALxfJtAujzFXNn+jChztfSQZ4stoYzWpf5kxlwxhv8xHsezaDogPIxE3/0M5KIhSFhMm9Zsk4laGZ1BNQIqIN36MMVQ55Ch9zp8zd66OWfQHKQCQdFysPmbCWkLzcVxyxGHIuK5pc7+VZ+ymN4POG/atIhAc6yh+dmuOzQ/QmFTzRaxMSLEgEFqsWQCYhoy1BGQ3ZAe5FAAAIABJREFUBuYraaGCbE666XZns5ObZ9L7W+xbN39Rvx/Hb0ms0YsqTwPio2jrxNjUQds6qz8c67pidIaQ9UDmugw4fC7+jV5Ey0S3TOgrEjc/QmbxAQWAVZnJvX+e+A4OjTn1Ju1Wr7GOprDdhbX6GDPTxpfP5/OHOUBnlf0hkfXVyvN+f61tgZvvoIhpSX3GutphEjnmsq2B76zqO9jrJgKWJTZRLhsXUQ1PCLFqCCnxM1zX2AB9TP0wyA6s/Lz4sE92WBSj+gTaWGh/KVRw7Yc/jfVUH1F7kraQUIDEzLXvSqztPmJhW020aW/NQKECOSJqzGEqWzN5RcXSTxUTg6nYhs4ccwngwGCNx0uAlE1ozKufz2oHDEBnexVk1dJjifoDrapjdYuZhT0bSkBNFd5HkQJv/AVN4CuYOFjofohluYUQPi5RlEMtGxvvS8dunn2cXe/jaOJkfj2AsY3Fa3V51n/vY8ly6GVtaYc7ZZeP3L1eCdiYas8yen8672/I+rlnmX3Ze6ZyH26qurrcgeWmjcDGLLwxmja22i4yH8tb9+9AWB8r6ddzirEbM91926xcHY9nmb19O+yZQLjBkn4AVDjK0rOXW9DK7dXqk0OnXePaQ0Xb5N/7dvvqXfOfDMUeXO4zfAC47cdtA7736Va5nwgmn5T78/QTT19rYLlyxK9cvokxZOQa8b35HgAwhoIpZPzq5SVSEHPTU1rwa5cXTYzm+9e7Zmp6LQn3w4zvXV40kZy1ygY8EOPz4wO+mE9PzA6vaj55P854v4qozuM64vXhgl99fAUAOKQV83LYmLqtChSGUPD/vvsMgbgBkqybxVyEgTThDQMjk4ID21j/IJ/a5v44iECOgdT/7/oaUyoYU8Z3r69wHFe9565t5l8eZsxLRCTGF5cTxqRhNxRAX85pA3zGVJBLbMxqDLKhHXXTb0ImtoEvahqYS2ymjEOseLMeMaSCL9YTxlQagDBRlrIO+LXrC9wfZ3z/4e7/Z+9Nem1ZsjShzzpvdnO627wmmoyMILMUWVUCBAk5YcAAgSikqhohKDEoqX4AEgMGMEGM4AfAAApVhQoY8RMYMQCknNAoUxGRFX289+679557zu68sY7BsmVu7mefe997EcnLG0qXjs7e7ubWu29btr7vW6i0gxARr+wazkms2wE2tTPDA6PIIhN8rqlou9wVsNQxq9iK7MX95589QwRwselo/K3O4Wqq5FFlLqa1FK9QpwUTG+YAKc7qtPgcBwMhA4yZFtvWKlSVxzjSopUX77xYt6NG8AKmnhRnlQq437fkuRwVTO0SdI+8oNP99FxUlYdzMgtwkCCIhD1VkJVHVbusbMtHhgYGCSFTnuWCPS2SSi8xL46U9vBOIQYBbTzeDBo6eZQBzMLbeCuhTIB3Mi3SRfKwpoVokNPiUsQslKGNhxsVpI6TgEeC/gkkYZC0OONzd50mDy0LZiQva7QSikODCII7ZqGJpCYKAQRLC22VVGGjS6Fv0mf2nvLnw2kNRAG5sfkHLDgJxUqzaV7GFOYjK9OmugmBFOaG+uAwkLru6eUaonWzeKrH0zp5DiP94LqkdJt+reOQjNnG59A1Qk+CGt1xg7j2VJ6VZJBwWJJBIq5T3ZyAqAL6XpH3lD2X7D0VcVKVNZGu8+LWhCm0jcC0+LQpfZUMIfYwlh7LACrHSgrhwN5dFSGGFM7FRNhRkuGyhFkmw8TvVwhNnOJTJoMiNh52AdF0vURoSS1W9gJ+HeBPEqGhcYYAPIepSAvMYCIp2oZUto4QKU4ulxls8u45VjGl9ooxeVs7mTwt1KfRxCnGowDiIGZeq7zYZE+njHR9FPBVTF5PILJaqwTiKBDbAH9oyGOmqa8ZnsixK6OOwEjtAKvKOkAk21rU5GkVg4BoYo5pKb1ArGL2VHGMS6EnT1wwdE6O5N11bZwpygqbjOo6zjxrmY0hkrcT7LFE9l5B0P3SUt6+TuFVmgjdUb/IFHpHdQJ2+9VXmGqgfMkbDbCIifCAb2IO8UMe1VRfViHvaW4Ek/pFchsiVC8mxdP0vshhYyKy5zYbGArQJ1LnnXks2aNXeh5TH0qbxqbwWJZ9es5jKdNrJUpAcf7p0RcBk3c75QExtS3no1I6ixx+hu8HkGN4ckghqmxhfCSPJc9nbkPOg5+VAjVOnTTVS5Ye2KVxnAxHrms2IIs04hGP5QOvHufJbQsRUbGVuSjzjLFE+YmcbuZhxHTfbG5IAenjo+FS8j5C8li+06CL8/sepOc5KBftPpNmqsQ8/9JoPlfXssxsnJ8z+N/DI2JOX/pdO95r8Z7tH34Y/+X/5j9C73T2mrB3hENnsAfAeZXVBUtvT6nEx7vsQtBintMwFJMX/2xkTbEqdd7B5pAYzFHihTgfJXcoBDFTECzjUnI+007+3HjgJ1KmHXghYqFUR7vJVe3gHIXO0IaMDYYLzmChadEvChirSHWVys84RKyux09/SF6mciedD4aoPuApFeeXynmZ15TqWBogwVMcSynYA4CJK8IvWy6eNwHY4CgW5QzLBEA78EHAtMkAHTQgYoZ7MuSyPES5A89jkfojK+gBkDqF8ChifFL8ToKBcpzMkp8lNI1nSJL8DJ/knW6hA4KluZIXogXPDCJmHlVWAEwGF8NTQ4LVZnhdMjbyjjJ/LsrgH+6sZJiMTUSR43IKAUQnpl30Mq+0q027/WIyPgTmXCl+Vvg7h/JwyVDxi7qWO+TAfDdcB/rOcUbzoj5OUNFYnC/rkowYREwegNIbwLv5/NkLoE7PCocdWd7D85Khm7Y8n8os5hLYA8UGWnlwjE6keqg4xeYs+yPHFiz6BSAjMRm4kEVfJoMNKSxH5rKp4n/pSeD2l/XgNnB6LjsW6UuO3LlFnsDExyp4eiU/TiSuW46rKTD3aghQqBcn5xBZQTE/Z1wvgGJujhQuBSrS54piieaFFHPzeN3nxRzqFjAZsZxvEVok91NENkhzKBVuN+eJqczM4+PzcaoLxwGNCRoMWYy7mOokx2QkB0z8v7wgj1NMzgR5jgytzp6aFPpFxRzflAwjKocNKhFEjuGZ42HyeAlQ/gAZ1jweeayRy8gCQQxJj5hBOoOZGx1RFX2f4nFKmyC63P+R+kn2X31Rx/ExqW/itDhmmHba1CiNO+Q5Fqf6FotyhhKzMUmNnC+wc//x5UjQW64LbShQn5f557EV01zM4xQLY8wv7JyFx5KNuuWzPIuZWkBaZ+/HYozEtJ+Wn6PslSvL4+ek6AtO88B4WsBXy9co3SOmeVauEWKhbFp6Z5fTI7X9QYzQeVbnv5dwYzFd53zPIiLfdR2YD1Ysvovi3CLtWePsXNli8f83PB41ZN9W9l9SmX/2X3394j3P/uhp/Ps/+Du/9Xz/uz/+wdfeNuA991iuzYi/df0pnlQH9MHg1bCBFBE31RErOeJn3RO0ymJnG1yZDq/GNZ7Xe7io8Hm/QaMchQexDZ7XB9yOK6z1iCEo7MYW92ODVlt8d/Man3QXWGmLWjocfQWJiLuxhQ8S31zf4d4SR28/Nvh4fY9PTxdUl/qEk6PwJxMMVmFjBqz0iF/sbyBExNoQbHXwOodw+NbVXYbb7m2NWjms9AiXoKNSBNynOvRO4+nqCAA4jDU21YBf3l3h8qLDthrw8rjG1VUPIz32Yw3ryaD8cL3H3tYw0uMw1rioKegYw1h3Q4PRq8ztvKgGdM5kiGWjHXZ9jU1NsTwZQipExMpYnKzBddPhMNaZs7oyFoexwqYacde12NQDeqdxUQ24HxpcNR16Z/Dm1OJmfcLdqcWqHlEpj9vjCs5LfHhzQmdN5m9yuaNTiasqskeWQ4boBOvd9zUa4+CDwEUzoFYOf/HZMwDANz+6Re80TkOV4YYc2oE9k7tT4mPWY/J6+hwy4zSYHA7hcGygjUdb21zHfjRo6zH/5w0P5sl2pxreSbQbCgmjVEBjHHbHBkoF9F2F7fUJ3k8eSRZXYW/tdjVgcAp9V6G96GjOWY3+voZqPbY3R1iribeZDMRyI6SEH3ovoNQkHDMOBqYijydtGATUW4thMIheoNlY9IcazcUwxftUIYdCcKNCvXYYBw29Ik9l3di8ocOwQ95oqWqLoTOoLyz6YwXd2uzF5dAHAIVBYIl/5r4O+xqy9qgvbebGeicRrYS5GCAE8uaFdyl0yNrlzRPeZNAbCsrmRjV9tgpKh7yhoysHe9cgRqC66QkmKJKHuXEZ8ua9RDgRj1Zt7ZxbKghqydBKf9AwlwPsfQ25sbS5xd75XZU9lqKOiIOk/BKk0HcKCAL6coTrNCADVBUoFAEAcVshXo+QJsB3mgzXZIjLnUa8srRoHhTUxsIfNMTGIXYKYuUmw0pEhFERl7T1iJ3OnEyx8mRkcz2ToRI7TUYbx9hUmLyvVk5czpb4pWKQFEszpGD3pwSLXXmITpE3k6GTyQMaU4xU/crArwJ9jwKyl+RZuvDZsI+JA6t2Gv7KQYwScqfgLx30nYbb+rzJIo/JE+wnz1g0IRtl0QSKSymQYxqqo4RfB+JnpsV9NBR3MpoIfZespbSoDVWcOKUC0HuVxWJi5tNFmF3qB0EGlbqT8Csq09cpnmYkA0sfJdw2oH6hEVVEqIrFeUTivlHZei/ha0Dfp9+ZUWQuqV9RXuok4NbkoZRWZG+dtGQs+po5joC+T5SChjyRqidD3W4j1Aggymz4Cg+4VYQaZYpFKbN3MApAndKeRQ0094K8bIo4qeogIOm1iVADMnEp61eSvo/URnMEhuuvvnLVqe3UNzIbDcJTeWqQ5FHtkb2pMXl8zZH4q8HMPa7kfRSTBzZVr+QqBj0Zopyn2Qm4VTIqFaB7HitkI0t4kb2EahDwFY0ZcyppDKhPy/iQMnmN2QsaNKB75JiUbOBl7yuAUFE79QnZS8qbAMEAugNcO20IsDdUjcgc3gecz0h55PI0MtcXkfIQPvEN5bztc49l4g4nni5Znyg2XzDzlM9hpgLKxhyDkxo2lcNllt5dWcwL9rpznUqu9AOYbLFRlXmfhcFYKvtmQ1/iIX/2jKE7j/95ZnLz5laYG+flvMt1EMDbPJbL/M/mx/1YlH3WuBR4EKfzbCzOvwSj+K+PL3+83x7Lv/Fh/Ff/23+A3umZAqbzKkMrS0VJhm2WSpHMqWKVR1646cTHIpjpFPMLwIwPBkzeRr62VFhkj2XpmSo9lhP/KebrDPVbco+Ww8V5M8yyzJ/jny25UXwPQyRnapLsBeHdPBlnnkj24uX0EdnrtvzPsErmC4nE3Vkqo5Z8rbIsbjOPFXOuICKCV1PaBXai3JBjb2jpGeSxAZAX2toQrsaNmtqcoJ+lymPp8aS+mM5nJUte4AIzQZycLufLnj9MniKBXHawcnoxxmkXnD2duaH5BzLmc8wv4ryZ7yM1eWpnKpRFGu6n5TjP4pYV/ZI9nqxGKQr1yVRG7vfU1sw7Krd0yx/W/EPJKwIx95qV3rKl1xDcH6m+LMST+ytOkMvSu5f6voQv0WonjWUJr2QvW/6lxuTtYw9P6V0svG25v1n8xi3Oc1/wuLKBxf/LcWbhm7IODCvlugNJPKfoSz5KLyj3C5et4vxaKPp+uTW/HCtOV3qTy7RYpBeLa+WigBdYosgrLaJyX5VzoOhHVuaESf1fzhNBokCRn7mFZzF7ztwk+pI9lgvhotJbynWaeSwFJq9bOUcDyKCOyB6/mceyfM/zMwHM+mbpyczeW+67sm8VsheSPZxnPRx5DNM9AKKY4v+xt4u89Jiem/LZAfJ4iVQvHhNgai/1depHnlbs5eQFd5FvXsADE7wzPe95kb2AX87+s8eyEBT6KgcJ70zvhNlPED9vcWE0JM9Y0AmGGKZ0fC2q6Xs+itW5OPM8ZVEiMfX1jJuIeR9mwyL/dhR9tjBwcruKbMpXUD5fvBLytcWjLYqySo9lLqP0iuZnGPPCi3azB5LzmHXXGYtqsUSYlf2o16/sQ0zn3maQnfvOY3M2/fL5K8r5IsfSO3uunQ/K/Q2O2fi+La/H+hDzMS7z/FLHuX77gsef/dd/NTyWf/cH/95vPd9//Mf/9GtvG/CeeywFIom/eIUgSKxkChpOHiJAZZ5irV3mBjbGpaDbgQxQq1Fpnw1Po/3Et0xcvnLjKIqYFVS7ZNRGIHulTBL+GJNQSimOIUWEExIxqsxp5GDhrAKrVUA3mmwYaRVofRTmxpfWpC2l1RREm71Dq3pEn0RKpkDfEsa4/MxP6qRiZpwS9DapphaGmLUqe7RI+ELm8krhDQDwPhm8o4RK+YQgAIbmMncuGbfeUzqG51a1w9CbnEaC7tPao+9EhvyWb1OCJGOu4ApkuC5ARiQH646RuIFNOya4rUrcPp/z9pFEURApb20SD7UQFIGgBUO+D4CpXFKYVXn8lQrwTkEq4iUKgWmBmzyHSgUMMBmOHDzVOUSRA51nNdggEeMEgxWgkCxCkVfPWSpbqIimsRhHBRcnSHGMJP7CHrPSEBQyZsM6iyMl6HKpRhsEcn7OArpycKOeGeBZwTYJmWRlWMF8yvRMJ3i54F16S7zGYCUpwyZoL4dwYCNb6kCL+CDSojZCVR7BCwQQaYnFPaSMhKjlRR0rzYoi9EMUGfIcxnS/iohCTp8TLJcNaV37xKE202JNBUTI6UdQRqiKNkf8MG1/Z1hxXpgSt9N3CqrxlDYHvwZU4/O4MFQ69PxyihBV8g4OpAQcI8jwTEadWVnYk8lcyAzNlhHKBDhoWkRp8oaKiriYBHMW08ogIkF7I0FOK0+cUx0QR1UESscEZ00exsh8TB4zzov7wAnAROrDNAYzA36USfVWTi9m7sfUV2rlSKglbdQIk8rukI3vPPcbj3jSgA7U3pOGWJGXNre34vmZ7nViUr4Vqe6muCYA6DC1lY063vjwAqIOs82b6OR8M6IK0z3Z2BJARb853LeiSuUITH3In00ABgmxTorFzH1l456hziU31qTngjeKIih/HWmc2RudeM1xVJNRyu/Xc/2R8o2DzHWMbGjJSJzfEnKfNgFmMGjLvFsBVqWNKmaYLasXM5w5qxkHAdQB4VS6nr7cEXUk73P6nJ9tgcybzXVNEGJaeACxov8ZIpzqjlBcYwOcN3yS8SPKDUWA7qkTVFsA8CA4NG+k5HcOXct1Z5RAwAR9Tn2WVVGBvFHBBq/wYvKks2UhirbEZHBzOzldmlvSAcEUqskC2fiISc2YvW1AMkD5uSsgxyIge+foRDEGIameFkYdEKd2lf1SGIox5Zs3A4q9QTaMH3jnCkMwe/9LY6eow8ywLNI/apQvjd3iyJsIi82X2aZBWb+yrRx658zxqJFaGPGzykbx4HSZT/n9bH1w5vxjx1sM93Pt/Ovj6z3ea8PySXXEP/j4/8QLewkPga3sYaPG3jc4+BrX5oghGNTS4t6tcKlPeGPXAIBL3eEUKvTBoJEWb+wK1+aEe9dCIWCjB2xUDxs0ft7f4MqQqIuHhEKAh8SlJpjhy3GLVo7wkNioAS/HDZ5VB4QocOdWaBknAsCD4l4eXY3OG3zU3MNGldUmtQwwwsMIj9d2jRAlbJRolYULCkPCVMj0pmqVhUJALV1O36oRna/wcXOHe9fi6GrcVEfc2RYhSqz1gFpSSI8XwwVq6RAgsNU97mwLYIrd1yoLLTxceosfXAUjAtZ6QOcrhCiw1gP2rkElHULBbu+8yVDjRtnchqOvsNUD7m2DS9Nj72pU0uHkKqz0iIOtUSmPp9URn/VbXFcdjo6Ufi9Nj1o6fD5ssNYjqf4mFVwtAkzCn3D8z3vbIESRQ9IAwNYM6LyBRMTB1XBB4rub15Ai4GfHJ2iURaMsem8gRcTJVVmVFwAuqg5KROxtncSSdI4HyuVoGbBWIwIE9ramUCpeYaUtDramdroaWvgMk5aIuK5PqKTD64HmqQsKvde4qrsMob4bV5CIaLTF6FWOsSqTWu9ubKBlwMYMONgaIQqs9IjnzQE72+B2WOWQPBxrlOtQSZfVg1nJ18UpXM7GDDk+Kt9/clXO72BrXFQ97oY2C2WNQWelXx7flR4pHxlwtBWM9LBB5XBADAE/ugprPeLoKlzXJ+ySENaY0g6enodaudzHHOv0uj6h9wZHW83SNMriTRLjYogyzxsbVG7z2owIUeBoK0gRc8giEtui+vogs2rxs/YAAPjseJGN7lbbHCOVN47WZoQWHrf9OqMsWBGaFXsB4LLq8apb42l7xN3Q5tivAHDTnNB7EtZykeqwG2jcXZDYVIQJPIw16rR5xWJkAPB8tcerboPBK6yMzc+QFBFb0+NVt0FIm3HHscK6Iqj7KoVS4t/uEEVWMu6dxqYaMTiNWjucrMHKWAxOp76l56fRjtSzbQXD6tJpHtbKY/AqK+022lHbxjqfW1c0Lp01aI3NiBUpYo7/WkL978cGnTVZGVmLgDd9m9Wdx5R2Ww+4PbVojcPajLjtVrhqO9z3NOdCBLY1lc0bh1w2KzLb1J8sxCYECbMxFJ9RMqWo2aYeZtz03ukcXgoAWmNz32Wl6iCyWBuHuVpXI45jBSFI9VgntV8Wdtv3NW7WJ6IMpOeRn0sjA3qns2hcZzXatIGWQ0xFkSgCDq2xOAw1tPIwMkCl+1kdu7caUlA4rDYJqA1JBbhJ+e77ehbeSivK5zRUWdBNyZg3R2PR76NTaCqiKvhIqBytAuqUd2816hRmq60sBpc2h4PEphlwd2zxVY/akGoyMG3sAiTi1g0VjCHhskrTxjSjbWIUaCo7qy/HBvZeUnvS/ECRZ4a/sxhaeoZDkFg1Q66L9xJV5eBSiDBG+pToJ5M2aTk8WBmz1jlFEP70/lEqZipEjMhaDYy2ylSClAZA1nkw1STexmlCSH2SxlOISXdBJ+VjpiFwn2X0D+/NiSQiZ6dwVUTnQEJJiQlZAUyGa5jQQnQPsgWUETQl77gwLIFUSd6IKQ2XcoON8+Tz/FnioVLvWwzHB8gZNgQLQ/aBR5DTLTzhU57FptaybWWdzub9SHpgQky862CDvMy/vDbL9My5x+r2nh4x4kF8+9+l472Gwn78N6/iv//P/m1cmxNsVPhsuIASEVf6hI0a8NPuaeZEXpoOr4YNntaHrCC71iNaZXE3tnje7PFq2GCtB4xBY29r7McGtXb47uYVPuku0SiLWnoySkTA3bgCAHzU3uM+GWQHW+Ojdodfny6hZcBVdcLJVTNvZe8NNmbAWo342eEmL+Z4AT4Ghd5pPFsdczgRNraIYzm9GQ62ho8Sh7HCs9WRlGldhVZb/Hp3gW09YlMN+Py4wXXTwSjiUvLi7flqT4aTDJmbWSrLlmkB4KLuM8dSgEJq7PoG62rMyrXcVo7nedV06JwhDqYMqJXDwVbYVgPu+hbbekBnDanrjjUu6x6D17g9rohj2TVYJ4Xcu67BYA2ebQ85Pxtk/s8waF5obpthtuiRIuKQFkc+SFw2PZQI+OnLJ4gR+Phmh8ErHIcq8zDXaTHAYk67LnEs04KJw3pIEXEcKlSaVFsPpxpaB1pMpB/V02Cwqm1e8ISIrJ4bgsSxqxC8wmpNXFeVvOeHrs4cy826z6JSWvtsTPDiZdMOGKxG1xtsVmRgDFbjtGuga4/Nusdg9WyhwiJSHMKB45OGkGKPph/7cdDZgxwSRLxuLIbeIASJuhnRnyo0q3Hm/aYFjYIdNepmxDCYBANHhoMzx5IVbwGgbiz63qBpLLpTBVO5zLEcBp3rPQ4mK8yyqFZ/qCErj6axOTapdwrBStTrMS2MQuKSyqxqy3BuN6ZFUlLhZQ87gOx1ljLCWQ2lPca7GogCzZMOPi0kvFWoEseSlHQlXOJY6o2dkI2ZY+kz5N4fDerLHsObBmpLHEteGNldldVZReJC6o2dxMh68jialc1cUWnCFPLhTQVxNZIHeVTZE4UI4jBeWoJkjwq6cXBHA9k6hF5B1lMMUCFAHkEvIGtP/NEqcSzbwqslI4V9iILOeQE0PntRReJYRivJS5rif8ZOkceq9eTd0ZE8iEDidCqgDtPCqfASIgioW4OwCiTiEwREL0koZesmuG/yyKi9gr9ywCihThL+0kHda/hNcvdIZH5nVsGs4wSXDeQxkidJXK/k3ZGDSBzL5MXT5IkSA/W5Ok5CQogUIzFWIXuoZM8iN0jeI/ICqZPkW4hj2Qn4NhIPLYm6iEheItVJuK1HdatIodNM3igRROaBRUP5hIpUSoHEsUx1ditSO1UnAbeiOJQPOZYRPnE4gyGuoYjE4RMhKaNGwG0iQVIjEJN6rPCpDSkeo3R0H8MoGcLqK+YrRkSd1tUWuc4hcUzdOkIfSWmWeYX6BAxXX33doztqOwSNbeasecCvkmAQq7xmjmVM3MPEozTEAWUeYVTEP6SxRF6IM2cPQOYs8qI6KsDsE2cxcQQ5ruVM3TVMHD81At6QumtgjmUaA18BZzmWieeZY22WHEtfcCxFUqhFastCaCkY4mi6ZvL8sYeS72fuZBSYxePMMVWTZ45jnYqIGVczw6EL+DSASQk3XROcV/Iych9JhzkPkw8BSBsRWN0VU91EjBS/M4KUWQVm6qusnMvlAIXNtziXPaKiqFspdoS5t3LG1SxVb8X5PDl+7czzWh5xqlP2Wp7jgfIc/P+VY3kmj0fq/zYD9E9/8J987XDRp99/Gv/OP/27v/V8f/Cv/w9fe9uA99yw/PCPbuJ/8D/+W+hChcHrvHtro0SIElKE7GVyUebYhiEK8mhBZC8A/ycPjsyxJwFkD03pjXNRZg8de6iYM8jeHgDZCCxDlUzliAy3LeOOcXre7QSQF/YBIreD40Jy3EnPMS6Tx8VInz0anLcLEpX0COlpLtvN/2ecQTHFIlyqvvLBO98P4L5F3mUflEYaG4SiOF++b8pzIX3mPJb1Lsvgo9z95aNsB7e30gQPHpJqcJnnlBaQYtpFXnKRPCfUAAAgAElEQVRCz/ULx4SUYrrfh3n/lmPMMRw5ZAsbGbz7zHzhMgZdqVRc1leIKQalFDFzjv1iTnI/5PuL/i+/87lyw7HkJ5dzn701yzpxv87GgDmbxVHGNyw5zdwfJZeX2orMb+byGOI98WzjLB3nz1zpsi/LPi37c8mzns6LHH6God0c75DL56OM+5j7lRcCRWdzDMll7EjOY+ImI/Oq+TvzqEu+d5m31mGmEj3nNscMFRcCOazNctc/99ciVE2GdBbfy7rP0p/p49wuJCdA8SxiMS6C3xd8nVcduQ/ncTIZLj6LPcmOBcmK1JQu+jncmuqO2cFhZOb9kerHvHkxfc71iJj4ysyH5QUle0VEka70jOS6FH1X5Mf9NPOeyKk9y3zK1dqMK7+EzBXenQfeHyyeycc4sWmxPPtetH32Oe+6nHm/RmTY88wbssyj5OEWi3ksF8pf5ijrdG71nNPN5+JZ79S5a+deuMt0fO4cx7icI5G4xiL1U5QFL/fci/6xl/1ymRgfScN5F1oDlP48z/GtbZ+1Y3FvOJfRvJzy3nN5TnU+35SzQ/VYP7zt4Oe4eBzPGkWc3yMG0VuPL3jfo+Vyxb70Pe+o15dM99ayftM6pOOH/+XXz7H8XTcs32sobKtG/I3VZ7hSR9io8cJS7Mineoe1HPGT8RmM8Dj5Gpf6hFd2ixt9hI0Kr+wGKzVmmOwHZodXboON6hGixK1b4862qKXDd9uX+HS8wlb1MMJj75sMVXVR4fea17h35LHcuRYf1vf4bLiEFAHX5oSTr2Cjgkq/AEMwWOsBW9XjR8fnMCJkj+XgNWyU6L3Bh80OJm1X7VyLVpGHlRdZIQrc2RUCBI6uwofNDlJE3NsWW93jJ4enuDA9bqoTPukucF11qJXD3dhmWOg3V3c4OlKFZfVcqqOGkT6nZUN5qwlG2idDvlEWd+MKF6sevdeopM/pW2VxdBWe1kfsbJOvMzz2uurweljhsurReYOtHvBmbHFddei8we2wwtPmgFf9hjy8KRZp5wy+s74vIJU6ew0ZxunTRsJ1c4JERO91bsP90BLkEgJX1Qm19Ph/Xn+EEAX+4Poleq+xt6QI7ILExgwzA/Z1TzDVbTUk2KbLUNLd0GCVIJR3CW63aQfYoMhTm7zCJ1thUw3ZEGMI1Ju+xWA1btYnAASNbrXFbbeCUR73XYOnmyNcgrMxtG8M5OlzQeKmPeFoKxzHClcNwTNPtsLL3QabdsCT9QGdnZR9lQwZzsh8YqM8BqezYa/SAv04Vqi1w+gVrFPQKuAyQfCcl7hedXhzanG96tBZQ3zVlP/gFU5DhetVh8NQoa1ILXddjdnD3CfY5OiI67quLPZ9jW0z4K5rsEkeaCMDTtagSkq/h6EiqFflM1zuzWGFprLY1CN5eL3CYDWsU9i0Q4ZOcltCFGiTd1mIiNNAsMJN05PystXYtjTmQ+JkM+ywrSxe320QATy9Okyxbq3GOqkHMwTy2NUIXmK76bLBwIZ4neLBAsDxVOPq4oQ392ts1j1MggcCwP1+lbnOWgcMvcF202URs2NXI0Zgu+7RDRWEIJggb3Ttb9fYXJ9glEc3VEnlljai+n2NzfWJ2jZQvodjg6Yd0XcV6sZOQmMAhlEjJJ5y31XQxsNahbq2GAcDmSCOmvPvDYKTaFYjKRPLCM3zz6oM16uMxzgYuFGham023oeOoH9V4zB0BqZ2mfPtnaI4rMkYt7cNxNpBVwRV950GvEB1OWTYHqvtur2BuRzgrUI4VlmRV21tNkjdKbnH2NisPXlik8dbGo9w1OSRNGkjZJQQK4cwKNp51wFSR/hOkQf23swFfKpAHNUUjigc9MSX5PiWKgIH6ocoI33vVPLsihRzVCUOXYDoNcTWQryqEHVErAr+XACFXzGB+JCdBKoI0RGvUlri0EWFpMSroVKcT2EFeSxDih2ZPJaBVWHrkD2roY6AF9lj6bdJKTcihwYRTsCvQvKSJg8Sq8LKCDmkvJoAfZcUbjXIIz4K8iDG5D0957GU5DUcr766Zal6AdeSpUCqsMmwD4Bv2TNKHkv2FLLokzoJQE5ePvYEzjyWBpNXjcWSYvLSAdlIjhIwe/LwstdXdRS2ZBbH0pEHGjGpwtbkdWYvtExj4Js4805xO7jfok5e2KXHsqK+B+hzFIDuZPayC0d1C4bu9016dyRPeRSTJ/qdHstIZZPHUmTPV+lVLQ0svs79MVPLZU9hagt7KufKsWnQxeTNXB6lV2/pBQXwdlXYpZHJZTLPlfNZGtmlh5Lb53DWY0l9neZMGaMT83xz3qLIo2jb7Cg3NpYbEjmv+ckHiq7nNjZQnDtTtymvR4zQr2KYfg1HTA6u39XjvfZY/t7f2sb//H/5l/CL8QlsUHhqDrBR4fNxizu7wrfbWxx8nXiPW3xc3+GT4QoA8HF9h9d2DRsVNmrAr/orfFDvcTuuoaXHE3PEtTni4Bv8+f5DfNzeo/MGnTfY6BE2KDyr9gCAn52eYGsIunihe/yyu8a32jcIUeDluMFGjzPvohYee9dgb2t8b/MKAHA7rtGqESZx1y5Vh5/3N+i8SfmSQbd3deLTkZFzVXXQwuNC9/hFdw0XFG6qE3auxh+uP8fn4xa34xrfbm/xYrjA0Vd4Wh3RqhFD0PjF8QYbM2AMCtfVCS/7DQBkT+p11aGWLnM734zEnWO+ZOcNbqoT7myLVlkMYYLN7m2NrRnwotviouqxVsSJPLgaV9UJd+MKz+oDXg9rrPSI22GFm/qE1/0arbb45uoOPzve4INmj3vboPcGH7U7rPWAnxyeklHpDCrlMHqNSjk0ykKJmDmWL/ptNg7HoBGiwE19zNDil90GPkj8K09+ASki/u8330CrLa7rE+7HBpXyeNOvZlyX5+0eSkS8HtbQwuNo6yQMJXBh+syd/LDdo/MGb4YVKkUc0pv6iFf9BldVh9thBV3wnLQM+KDZo1UjfnWieeqiwtFW+KAlyPKz5pBh1tz+zhnU2mXe4+enLRpt8bQ54kW3BQCs9YjvbV7i5bjBr49XuKw64lAKgj2fXJU4lh6919kQ5o0CGxSUCLiqOxxsnXioxC18M6xwWXWopMfrfo2P1/f41fEKaz1CioDeGwyeNiouqj6nPyTu6W5oMo9rZcbM9QxR4n5s8KQ54lW3wYfrHd70K7goMTiNTTXk8DwXdZ95ur0jg/Yb63vcjS12Q4Iua4taO2w0zUkfJGyQWBmLWhFfbT/W2eN6WdEzfdsTH3NTDdgNDaSIWJkRJ1tlPuPJVvjOxWsYEfBnbz7I3MFNgnsrSRgBozyetQdoEfDp6SK/yyrp4SKFDaoSF+/D9Q4/u7/Bdy5v8dnxguDcaXPkW9u7DLEfvMZl1eHT40VGAlw3tEH06rTGtiY49Mma/Gz+weVL/Pj+GcZk2K/NiM4ZKBHw0WqHf37/FD4KbBLX8Nn6gDd9i4tqwMFW2esdosC6GhNsvsJ10+FkK6zMiPuhwcaMedODNw0u6x61drjtVqi1y+MQokCTNi144+Cy6dFqi9fdCirN1auGDPI3fYub9jQLn9RolzddlAz4g6uX+LzbYj/WEABu2hO08PjV/gorQ6vZPqEUPlrv8PP7a2zrETfNEb/cXeNbF2/w6fEii8I9WR2zEBwA7Mea+JPp+e2dxlXTwUeJzhIf96Lu8eq0RpPeETaFRbpoiALwfH2Y0xs4zzQPrpoONtDGB1MQeAMJIFRG5wyetge86ja0sTBWaDTN6cErXFQDPj9s8L2bV+i9wWGsYZTP7x7mDx/HCjftCfdDg+umo3mfuNouSLw+rXHR9HjWHvDJ4RKttmi1RaNtfm+PXmE/1jAyYD9UuFnRXNwP9Gxd1D2kiHhx2KI1NlMVNtUIJQJuu1Xm9HI4K4D4d02iHxyHCk/WJ9pMTNzQprK4rHvEKHA/NNjWA14fV3iyPmE/1Gi0gw0SH6z2+OmbJ19ghXH+uGx73J5axCiwrsf8TFbK4/VxhVVl8/gyB9f5xPtekVZDZzXWlc30jdGpfK1EzVTKZ37tUHATAcB5hQ+3e9x29PvUjQaXbY/OGjhPauqMsOkSD3PTDDj0NdpURxeIUwsAp9HkeN4Acigv3uxyQaKtbP6sZYCSEYNTaAzRBbrR0IZWO2BMXM/aOAxWY7Aa62bEoatzOxh5URuC8Q8pZniMJHqXxQu1z2gT51QWEQSQqRExEgoksEp52VeFuCAjdUQhQCiAKa52EXebD6ZZeC+ztVYunRnNkUXYAjLyQcqQhf4m1fUpX7H07rJBVyARynMZ6VAqzHs5bUapErmAqa1B5PjSpap72Z6zCv9L8Z7y8xKFsGxHea5UFU9Ijgf35A5dpM3c0aJdX9Eu+8U/+k+/dq/ek+8/i//uP/nteyz/2Z/846+9bcB7blh+9Dev4z/8n//N7LXjwwaVoaouKAQIWignA8klMRwWcJEiYgwqL6JZPERLn8VT2NtVwl2rFNtx9DqL6VBeGk3yLHL5ugDKhygzJDeni9NWmESElh59Mio5X4bPljBNrpdEzF7IEpLI9/BimeCU0/ZTCRUVImbhE375yuKpFyKSuMkCQlkKf5RHFh1IXpTyXAlj5eu8yGHvVXmOvXFswHGac1DYsv4uyAcoEV58KxHzjzYvMk9pMVimLyGxQsQZHLc8X5YfI8GcWZSEPVPcPoK78o/N1G/sbeOFcikMUvZXmRdDbKcFB4XWof5j2Cuyp5G9XqqA2PklVK8Y8yU8dIJII6ssKxkR0o+0ViF73ZZhfaQMcMWYzvo3jdNyfmVF3dTecqMzLMYkRPFgnFmpmMUi+Pkox46vlbuIpUhG2ZZzEOiQxFgA8rZOeUwLkDx/0oKIwwcRHDfmz3kuyZhilZIgR3nw4qu8n2OBlm3icD2heM75fhL0QFJSXkJ75Qxiy3DcHEpo0U/n0k0w3Kn9nH4JEea2zEIfxTmkl06SGnSM4iw0lxc9rABNnFWRYbe84Az+DAyXVa2Lz7O+i3NBFCDNl8ViawbPFQX0lhedvNjjxdsCcjprT0SG385+qovzfK3si1m4IO4nLyBTzM649EAUZc8guouXZ/S8MJ3CJeV+ZIXZiGmxyvUQBXSW61WGPYrz+j5YVPLBH0sRklKldAm7LcVMuG7nxFS+zMF14rzKurIAzPJH55yHZ3ZdPBRpWR7nPBxc9tn0eLh4F5iXzZ8FMqeXobPlfypqDqslHmDMYXUATLBceaZMLuuB96sYo/K98lg/hMeb/U4vFt9UzrvlPcuxeSwvfDHbJg/D2/J9RznvOsop+JXzfOT62/N8dw+89f4vWIevnO7M8aP/4uuHwj75/rP47/yTv/dbz/d/+pP//mtvG/CeQ2FdlPjJ8Sm+0d5hCBqfdJeQIuJ5fcAH5oQ/232IlbbY2QbPavL0fLTawcDjF4drrM2IjR7wstvg25s3+PXpEjc1qS1+errAfddgVVl8//oFfrJ/Ql4Z5bEbG1TS45f7LUIU+P2LW9wOJOSzH2t8a3uHn++uszjOfmzn4j1O47LucVH1+OGr5xAi4rLtMXqFPoUE6QaD55eHDNncDzVaY7E248wAu+1WGJ1CPxp8cEke1MNQY12N+PWrK7SrAVdtj5e7DS7XVP+7roFNO4MfXe6wH8h7d981uFmfSAk1wQzvTm0uL0SBi7ZHbzWGpLJYaY/dscFm1aMfTVbCEyJiXY849DWebo7Y9Q16q2GUR1tZ7LqGdn4PBKPtRoPrVYdXh3VKX2O3X+HJ1QFv7rZYNSOayuKXr67grcLN9RHdaPKuKu++W0u7nBw25GJ7ghRAV4SAOZ5qVDWFArnenFApj5/85AMAwNOP79Fbjb6rUNUW3kus2wHOqxTSRWK3I9hzsxrzjqxNRiJBAZMK530DYQKq1QjvSPhl6Aya1YihN2jaMYvl0A6thN3XgBVQV6SoKmVAXTscdw0Fs98bVFcD7cyOCrIisZrgJYIno2e97XHsNMZjhWZL3qqx18CrGmHtUV/1GHtDu5cAhAqQhrZmOBQI77Jy6AheFPuOxFvCqEj8REXotYXrNGAl1NbC3xuoS5tCZADCUP7BScSThtxYhJOm8A5ewDQuG5C2T6+kFA5BrRz8roK6GOF3FUTrcigV2+spdMeJII6oqCwAiLcVYhug17Qb7gdFsMRRZHEaoUigJo6SxF1al+NyZpGYtQOCoLAbDWG34iAJspiEc0TtMb7cEPzvGz31rYiIHbWXeXPRSsi9JtjclZt2cZnTWIcclkHdaYSnI+zLNfyVo3ASKcSJem0I3qZIDEaeJMKVy/E+xSGFyLi0wFHT4rAJQJL6rz7XGJ87asNRQw4JLhcEzE5gfObJgOgl4tZB3hmEjYc8KoTWTwtHGSGSGE9sPeRBIdYRshPwmwB5kiRWo4CYOKjyqCCtgNt6yJGEa6KhHWwxUHgICidB7VKjgNsEKqMK0PeKhGS2Hnqn4NdhEqEZWRyFxG+qTxXCRYRf0Xd9FCQQ88RDdtTnBBMUqF9LuOd0Xh8E3FMP81LBXwUSLFER6l6S/ZLCKmAVKTahF1k0Rx2oTcHQ0KpOwG4j1IAEKUywwKNAqABzT+PNsDbXUp04VqI5JkiliAgVQeCCIhgk3xdMRLUXGLcR0gu4JqbA9wSNNAeB4TqifSEQNeAbkECOJpibtAQv9E2EPgj4BjBHGmM1JFEWCYwXBC01B8Bu6T5pCSZo15Q2qkmoxzeAJjQ+fAsIR+I5IgLDFYm5MMxTDdQ2zicYytutkEVb1EDp3Qqo7whiGPR0TZ/S2mBF0NJxC1RJ4IbzrO4iuudf3bA0B2C8ACCoDBFAAjKOznM55kj/o0AWGKp2EVEJ+BpQXTqvqA1mT8+ArwqIp03CMABCEjHKwek10LyOsBuai6EC9DEiVGIm0iJchK/TRnQX4RsBNUQETc+KGmKad2ImiiPHiGAE1BjTHBPQp5gFhxgK62sB1afN1obmpDnEDP+UlurmawF9jHBrqq/wMUMuGWI7g8ImESLqB2SoadCUHws/SQ9IRxDLc8JFEJN4D8c0FVHkNNxXQZOQkU91yHkkcR5pqc/ykYzEmXiPiwmSKyB9zPVjHwGPSUjRkx7ATON0HkCGvJaGdAnX5bqzyBKfz/aemMqNcuqHB9BWTs4Gaoh556sUIopFfvTuoTTvNOiL/jrHOZ2VfcZIpjo9zONh/SeobYbcLo4fnW/6Xx+/xeO9Niw7Z/DT3Q0+PV1g9CrBMAR+WV2hToYSe0g+1Rc4DQYvj8SP64aKYgaKiNEpvOlbHPsKn+qLrLhpR42DDOisRjdUeCG3sx3scaDu60eDwRJ8xY4a+75G35E82t2pJQXNUiwiCLwxK2jtcTw0EAKZd8US28FJfOYvsrAGQzlY+pt3u52jBaQfJX5t6W3lncK9auFOGgcn0XcVbGdgR5J7dzYZDDLiF5YUMQUAbyW6rqIFDIsaWTUjyvddRbEB0/2sHHk3KgQn0RWB1E+6QbASQ2/gncy73TsVEVJZvtcYB4NgJY6nGmFUUxmdxsuwRegVbK/pvsSRusUawUmcFrAQpIU374q+cQTtZb4SAMRRkgETBYbeUDsSd+f17YaMglHCGVLvHDuTICfp/p4MjlNSDe1VJGNAAhglvEoxAAeJaAUG2wBewOkAWIkuxdU79TR2I+/WBgExSOIZ7Wj+eBFhdQUMEl5SnuN9TWm9gO8VfCGDLoLAwUqKSWYlek8wUDgB5UgVc7hrIKyclNmkgmdRjwB4SeVSTDCBUEBxhBUIiVfEMbGcryCsAIKAjwayl/DC0DkAUFRHEQRxp6KBHCQpgEbAJgPUS+R7OOi3t5LyQ0VGijMIMiKktD5xl4SlusZR5h9lOZD33SUjQDiqs3ACYWfoh1bSAl6mRXx0ZlrsjFSH6E1aNAiKscefB0kcJUvlsrER76s8/6QViL5K4wvI1AciEncu/3jyD/UoIVkN1FI9pROIB+KTcXpa+AhAJGVPKxCTwRrlpI7phSZemkhxI/OCS0AeFWKvoAbiOk1cMUFKpYLq4KOBHAQARRw2r/KCDykNIhA85RVSzDocJbU/cbQgp3YJD6gTzXUIgTiADGsnKP5lAKKNqTxKiwDEUVHeAlAHRYvLo8xGGatMhlFl/pawgD5N44YIqKNMyq4ic7VEcV4E+ixi6ksBRCEmgzIt8FnJlOMR8mJbBOYR0Tk1pDkjgOjTnLPToixiWuypUeRFNE1KXmwLoloGEFSPn1+eZ2kuikhcOlauZQ+T6pE9ellRs1xkBmSuHC/uBRI/LCEAee5m5VKRDCNeEHLIT+4ji2lx66Y0mTdHQzBx0wqOmMjzi+cs8oKU+YnUPuQ4jaFY0WSDIZXHoCBfi0cX1l/kiMzB4+8CFO6i5DWGef1ioN/soEXuj9xXnJ6NljTFqG9Fzqcsjz6QAZ+fxUj5l0ZFFADUlC9fD5xvOscL8LJfYrovqKnOZLjx95j5hFmxltOZOR+RPwczpSODmd7fwfC5NIbpcxQCAnFS15XTOHK/hPRs8vXc7iLNg/8RExKAeZmiKFuV48d9Oh+LbARBFM51kfskpAVDlJNBOuN1sgFVjm9pqHIZYjKYOL8oRf7P6aOYzpdjwfmWfc7P4PKYjDruQJGMX3E2Ha0L5l7fd3pOyzrlDM/X53zd3lYG1xtIeF/gryAqM2KOjvpdO95rw/JJdcR/+O0/RR81KN4jxbBjwZtToMV59oYkAR0pYuYMlp5Evu4hM4QVAFo5YghmpjIbIDIUkiG2BK1UM06iFCGfX5YnRcgcSiMCbPHWCknVlQ8uyyQ12wnKqSBFgEsQVf7MUFqOj8iCOiz0wn1SxoDk/9w+rm/5AJTqtksYKtebD4YSc4xHvs79eO6/iyrDhik+osswW24Hq+kuH8zyewkhXUJmy885fuAHUzzEUvwHQIYJZpXXBBfmvmIIcdnPDNctYcdcr5KvyX3K361XiACMDLNyGRpbKg4v4b9lfqUQD6dl3plJsGieV1LEHCaGNwPLo3wtL8c8xglmzfH92INcQnb5Xm4//39M0ZbzVwV01geZeWdlX/JnPpaQ4DJ/n9RVOUZgVlvmBXrR+BIyzJDfEh7LarIMtWWo+Tbx70rV2RI+y+k2RVu4X0tIqvUKW2MxpviTJUR4qbobFnMtFue4L8v0zimsUww+X8BeAeIptSm2Hdffe5nhqwxPncrm/Kff8HOKx5w/x8Gr5KT2uzz4flbONQVsm8swgvLSspw/9F+lulgW80n35/HXU9/zzBmTAm+MwJjG17KyLei5YAXrUmVWFu0E0ru6nNNp46s8LwDY9N35aeGeIa6FeyArz4qpj4UAxlKtVsQZ/HXGyeJrKmK0pFj8AB7KdROgTbgSYszwTgCCQ944mUO60KCKuRJoCQct4ZEMW43IYV5yfbk9zKXiPighoqV7g/stt2Fxne8tobVIdVDT+HzpoxzfvABPf6WRUIiocBsz3DRttFEWcXZtBl8tX8jlef7uBKDCvK/LPsE8j0mwaeqnWT2KMRRBPGgq/eSVVgTYusr50b1YpEGCzaa2l3UrynqIcY1TtmV/FGMuyqQPb33Yd4uhX7bvQQgNfvbCfFqW905lLuYfSsN0UdY5A4l3FB60ozjBOz7n/mORtmjj1K1n4L+5nMfPvbXdZdpy7v0Gx29srP4VP8q18u/a8V4blrVw+KPmV3jpLuAhYQQJfhxDjT4a/J56hWOoUQmHY6ixlgPuPEFWt7JHHw0ZcMLh1m1wqU/ZGG2Ew0oO6KPBK7vFpb6FjSobQEMwuE4Ks2/cGis5wkeJlRrwxq1xqUiw4BQqSESo4k0rRcDJ17BR4TJhd4ZgcjopAozwuHerZERKNNLCRoU+mGw4hyhgpIcRPqvbhihQS5fzPvkaQ9DYqAGnUOXPSgTYqHDvSPnWhYl3Sn3pc17lcQoVjKAyPWRO03mTy6U2kvFuhMfR16iky3kOwaCWFkMwaNWIzlfQKT4o18GIgLUecGdbbNSALo1LK0ncZeeabGRLRNgoYUTIRizX7+hqqqNys40AFxV8FBQ7NCh81NxDiohP+wvU0qe20IPP9ckbDcqSaI+rUzsVTBpfNqYBYK0HhChT3NOYub0HVyXF3HpmVAMk0iRFwN6Rp9FH4umu9Zh5w3xfLT2GoLLxrpLxfW8bNMqhVRZ7V2fO7pXp0PkKe0fjoYo39eB5I4TqyYazlh5j4QZg3jKX56NA7w0alYRQkrpvKTLlosyhZ5jHzP9ZPIjms0SVRHQ4pM7oFVba4mBrbMyA3uvMSSaBH5Hv400T7n9Oz/WvpMt1ODl69nkzpVKJH5n40mXIoUnYyOW8tPBwUeV+GoPCTU3P8u2wysY3iyFlzrKIWGmCOXO+JSeY48eGKLAxA+6HFpdJMMkXxiQrFXMbKulwtHU25mrtckzbOrXNJng4AFxUPXZjAxsUWm1nmwUrPeJ+ILi3kgGdM1ibMQswLTfJeAOM82Lo/OA1PXdh4pRzep3y5c0Zl561kq/tgsyiSjw/+VkGgN4Z1ElAquQwlxzsTTWQGrAnREmtksiIIyi/TBswALAyYxa1YQVnFjWiDSWBlbGzDSnrVZ7bUhBnuyl44ADxpjnmLnO+6d0d4KPI6fN7M6kx85jUaRON68DPHSs5+2JTpxT30TIggjYOtAwkeFUPiFFMBjLma27rVRZu4zbqot6j09DKo05tYlEqfo51ahO33XmK30t9JSEFoBMlYbAamkPvBAGjwiwdc8eZr808a6T21UklmXnXHPOXYwLLtCFnlJ/xuo32WczmqxxGeYxJiIrrw5tCY+oTfqYjpg2mGAGtQt70KTnby2t8cBoAOeRQyQ02hYr0kqvNRxkmibnV/J5Y8phLZBVvJE2bafPQRWWYmXy+2LjhDScK40ObaTKVn+sGdi5NHPdsMxZhd2b2QsTE5y3aiHyPeGCVnCVHuvUAACAASURBVBOiyRth5YbD8qEoypwb62J+rWxQmR5A5uQu83vbUeb/wNj+isdyo+LcdbAhPOfZLtPMNnPK7+8q/wvX9e3tfaeR+9fH13q814blJ8Ml/rMf/v0cMsCll5dWAVp5nAYyRiLox5Ff+lLErD4G0A49K5fpFKDce5ljua3bEf1osugEv5hdgp5WFcnjU7B0BVM5Cg4fKeA6i2SULzKZQgWM47SgL39kYhBQOuTdcoamTrHLkM/HIKb0aZdfygg3KEgdMo9MGroevMiwVJWClkNEhMSvE0A+F72YiUlIHREDsvCCkNN9nCfvEPOut9RU5ux8UjELdrpX6JA5fTEIxEFBNo5gbSnIeubC1T7DUx/ET+MfCxEhmG9XQlkZtgoQf1BEhCTdL1aOoLCJq4YICglQvrzGdHMKKA8Vp91zx/eJKVg778zLSDv9nF7lN/m0i+8SpFPH6Y2pInEOBQgqWKUdav5x5T7Pkz3Vm0MIgNLKXqZQA8RXo13iOO0A8/1iUafyN85NQdWRAk5HnfhgMX0eE0+OeYUyZm+FcALRxCmfOMnpz3btGTKkE8zTUL7QMe2sT3XherHHgKFAspeIJk5p0hwUHohVLHbRxdyrwm32DPubyuDP5MWJk6dFReLsAQhtmH6kfXFP+iGWCTYcasb/oZCDn54fOUiEJkD2kniAhTdB9TKHqGAeTijWyizPX3L1Sqgey/5HmWC1foJQSVuEBEj5qoG4erM+QBozB1q8aQo7EPUUyoIhzRBxgii7xEc0U1vzHGPPV5hCHTD/SUTkcwAQdIR0IgUfp/k6cX9oXFVP9Z/BLyOIq8hzLC2K1Ji4h57gl74h+CgHfc+wTH404xSGIwuKSEoTBQrYaeo7rpuY+i0qgr6WR9BxBpXj9uY+TPOcYavcXmmpHER+Rqf2yRRy4pjCXWQoXGo7e2qiojFkfiNf43qHivpQWuoX5muJOEFPARA0MtD/IeUTDF3nvUpfTZ+jBPrEJct8uASLZZgltYP7CAgcriP1p/eAS2EiOFSJr4A40nef2uFHIDT4yod1xbOWyvNIdTVUjygBWGTYJr9j45jqqwA4GkeR5kpkyC/bXZH6MBt6S/iiAEQPyKqYe0XYjRmUOsE9pQVE6huumyhhxDwfijnB8PIo07POvxc859WUB+cpLSYoKcOR07MRDPK7kH/iZmWEaUxzU8N0D0NVy3PECUSGxE5GUlovyem3TDA8ZWkUianNZd2m36apTeUxM3JKL/Xi/pkxlo3OIt2yTstzi/Ie/A9T/z1mGGeb+xFj7Fzej9XjbP2LfN55LNtavI++qNfzHIfyXcfPvvQdfwlH/N0ON/JeG5Yf1Hv8x9/7X9FHgzGSJ8ODdzcT1A+0q8//2dtohIcv3hI2KhjhyStZPJXsOfFRzryO/J1370vPQxmzkvM+p5h6Dg7J/z3kLA/2DnLaZV5vg6QyxJTqLVI4Dp+9duy5mtpK+fDOOMNslwd7e84pspZemBBF9iwuz5X9bKPMEGAfp/Hi9pSfy/w5P/JWzvvJFZDWnHbx1uV2L+u87NPl8bY0b4Op8nUXVa4bjxHXpQxPU35+rB7Lsh7Oh4dz57G0b8u7rP+kDjsp1S7hj3zEYrzPta1s37n2nhsb9hQ9Vu/H1GUf66dzsNxzSrnL/JbntQzv7M/l8bYxWNj2+dwyv8fG9lx9H4MgP3Ys+3NZ9tvmwvL/Fz3YY1X+/6L15TSP1ftcX5X9zJ9DFKgKDxF7f8Qj8+5d7V16esrPfL9+x7icVW/G3KNUwnP5swT14bJM7gNuE7//XV4YixyaQYh0vlh1xsLD9q6xzkVymqL+EfN8sfy+yG9cDsHi+6P1iHigxPtljrNz71x259r/tmK/ykJzaUQsJzHnu5yvcTK0zhoz5f2Pnfsq9T1XP2Da5OTvZ8bsbJ3KNp5NwPnFeTe8w2A6+3h/gTF+pyftSxhNj5X92O1vHY1H+v2xMnKeXyLtFzME351myu+LzK9HYL1/fXytx3ttWJ5Chf+3+yYOvsYYdIaqMcSRYXtsYGgRMnSwVi7HBmO4H0PaGIrHhsrJmczpKxdolfQIEOidyRDMSvkcWxHALBQJMBlIDKNb6ZGMjGygkkeVYqKZBxCx5eIzw5XEFJ6E69coizHoHBqFeXAcvzBEioGnRJi4fFwPUJ+VPE8AOdwI3xMgoETIHEM2FgJEhlMtDQDuQx9k5uSJxXcppnAjzNcLcYofNyTv8zLsyHLBxzC3coFd1skmvtu6IojtYaxmnEMAM1gaQPxHPs958qGKxRJD38r7VTKISt5kaQToAurFeZMX3s/4lgAytGsZksQHkcONOK8QIvEEeQ7YxD0sy2Y+YRlCpAwpwke50OdFG5cnRIR1KsOzyvt5Ma5kzDHWQtHGKf9kaHM/JDXeUpWXuaC88cH9znMo939Kz3xEkfKXRR/yIpzL9ouxLDeAyv6aoGQit6XiuWl1XryTITLlKQQe5YWWEDOGxpX9WRqDWoU8N/j+EmLG8DbvH3JBAcBoD5vixEkZZ2FBlApwKQadlCErFzMSgutZtonbSeFGZP5eps+Gjoz5emnkMJSP4XeUX5y1LUZAKW4bXfd+3r8lvM1UjhSUQ1k2qR+zscRplSZhM+Zkeqcodl0K9RKLeVkKsTF8T/B7SJLL5VwYE84nG1ZRQCo/W0SFKGYIiwzzw1TXEsLI50mYLQk1LfqWkSUqKfOWfFWOacfQQyFBYjPs6QcIcRJJmIlF20j5uLjfpy8RiIWyikjIoBLlAgDBJ2lMgclTDYCUuWI2iLLhWloGiTOaxy8tnJchVoKThDgpeJZCBYThTKT7L3qoSMgTYEIt8MLdT+XMQmjE4t6FYZ6vy6ltZ60EXkZweVEQz5WRMRHkJfeL/AHMlYSKOi7LLQ3Vsl1cfiEUNJvQ7OVjvmbJtxWYoVdEWd+i3pxmqjPOW2lRnPfK5XuK7yl/wfPj3J7F0gA/t5nDdS8hrY8YNGLZPwIofASPGolldz5qUpXtWFqYyz5dHuHR5k31fqzML3P+XXkux/QLGJDltP1dOCIedxT8LhzvtWFJC/WAz/oLuCDxzdUdAODT7hK33Qp/cPUSe1vjuurwotviO5tb/OJwDSUDvrt5jc/dBgdb42lzwK/2V/jm9g4vuw1q5fC83eOjZoc72+JHb57h2xdvcLA1emewMmQMfrTaAQB+ub/CtqKwDit9xIvTBr9/cQsA+Kxf4arpoEXypoF2pI9jhf1Q428/+RQBAi+6LVZ6xEqPUCLixhzxw/0HOFrCYT1pjuitxn6siUuXFjFP2wO0DHhaH/Hj3TNYr3DTHPFmWOFvX32Cn59u8OK0xR9efI4f7Z7jZCt8uN5hqwd03uAvjk+xrYd03wlvegr83GiH3mk8Xx+w0iN6R0brfqyxMSM2ZsAYNI62wrPmhJf9Bmsz4OSqbBx1tsXajHh9vMBFM2BTDeicQWcNblq654PNHocUzHvf17hZn3DfNWgri3/h6hV+ePscT9sDbvs1OktBwJ81B/xfr76BbYrLyAaBkgGttqikR5VEeH61v4IPIgejjlHgpj3hrm9hitAr/9rznwMA/rdffw+resxlmhT0mj0VWgV8tKZxf3HaQoAMiVVlKWj56ojbbgUtA37/4jX2tsZnxwuszYjdWOOmPeHz4waXKWg6K/5W2kPJgG9u7/C0OuLP7z9AiALWK+z7OvVTjW9vb/Hju2cwkgKN74YGfdrQqLWDkR6f7C6gpMMHqwN+ubtEjAIrM+KPn/4cPz0+wU/vbrCuRqwNGdNHW+E4VhAi5nEfU504YD3DyJ+ujrjraVzXZkDvDV4cNniyOmKlR3xyuMR3L1/jL+6e4rLpAVBs0G40qI3Ds9URnx22eLo64k3fQomIXV9nHtS6Jv7h2owIEHh1WOOj7R6/vLvCty7u8eK0wWA1rFdYNQNOQ4UQBS7bHkZ5nKzJEPjvXb/Gy25D4wegrSy29YDr+oSf3j2BD9S/tXF5Y+Gua6BkhA8Cz9ZHxCjw2X4LJQOFxzlSMPKLZsCuJ97jZdvjvmvwLz7/BFoE/O+//k4ObbNtBrw+rMh4ExGV9vjWxRtUyuPHt09zLNFa03w9jSZzx757/Rp//vkH+P6zF/jx7VP0o8nw/e8/e4G7oYWPEidr8KQ94WdvrvOmyUdbCj30q7tLXK8p8PpxqFBp4hj+yUc/x//x6e9hsBpX6w6XaS4Z5fGHl5/jT198C84rXLY9Xu3X+Phqh892WzzdHEnpmjcFgsDFqoeRAXddg2ebI/ZDjeumw2f7LW5SEHubVLsB4OnmiMuqxy93l1hXFqNXGCxtCK5roh3UxuVQRZtqwCe7C2gV0FuNj7Z7BAh8ttvi44sdPj9sMq9zVY9wqSyjPP6Nb/x/7L1JrC1beib0rS6a3Z1zz23yvedMO03ZZUOVpepAqgLKBWIEJWaFGIDEBJiboikxQ2LGACFREgZmCBWIQYkBglnJjJDKEr3KOJ1OZ/O625xzdhfN6hj861+xIs4+99738j2nX8pLuvfsHbFitRGx17++//++P8Dv3X8Lr84rKBnx0WaPRlv8P59/gF1L9+exr6FkwK8//Rz/52cf4XrV4Re3t/i/Pv8Qv/HiE/yj1y9oUy5I/OKTW/ggsa16SBHx6WmH41ChNg5GBpxGg29tjvBB4n5ooGXAi9UBf3T/BNt6hAsSg9PorcbT9RmHocKfefIaLsjssfCqW+PQ1zkW8cPtAYOnDcLTWFHMptX4zu4eADAGheNY47u71/jB/ilq7fD6tMKuGfLz99Fmj++9foa/+gs/wMHV+Oy8RastuhSnujED7scWb85tfs6+s72DFgEvmgNeDRv0XuP7t0/xwfaAP7v7HP/Hm1/A0+aEK9NjrQf88HSDXdWh9wafnnZotcWr8yq389PTFq2x+M7mFkpE/N+vPsRNe4YQEcexxrP2hEZb/OD+BjftGcexxqYa0vNH7O1XbQ8lA14d1/jlmzc4jDVOY4XBajxZdfj2htYAPzw8wUebe/z+m2f41ZtX+Ph4heumw8lW+I0nH+N3fvJnvvS649tX9/jB7RPEKPBk1eXNzrUe8Ye3N3iy6nAaK3y43WM/NHBBorMaMdIzfXYV3nQrvFgf0TmT34/fvX4DFxXu+jaH7GxTnDgA3PUtSs3ezmr8hecf4w/2z2gMhxrf2d3iTb/OcmFSRNTa4a6jaz/YHPDpcZvHd/QKNy1JjL08bbKElxAR23rEaaywSeuD82jwfH3C4DV6p1Epj1Zb3PYtbtozfJR4fVohRoFfuLrHyVaIUWBX97jrW9x3DT7YHvDJYQuANsUrTZvGN+0ZLkrcdQ1aQ3wI56HKm6aNcdCK4vrPo8G5r1BVNO619qhSnHI/mvwu4E3EGAW6oYIxLsusSRkoHjhIWKegVMAwaNS1wzBoGONnJGXeS6ybMSkPIL37Js8M3nQzxuVQKqUozKbSDl36TeJNNedUvo7jjPNmbgq3ipHer5rDdRZxslr7XI5zEnVNIVhahxyPy5t0fI2zGlL5vBnI9dLfKR5WJj1i3mCTuY3I/QgpllmANs+mTa+0Acqb7MsN+DjpLHP+0h4WRRmXvnMZEPFR2/qbkH6eXWFF/BNIxfu+6Tt/fhf/7b/317FVPc6hwhu3hkLAM3OEER4/Hp/ACI+jr7FRA/auwU73sFHh4BpU0qGWDgfX4MacsHcN6kQIsndtJjn5sNnj1bDBWg8wwmNIARZ7R0jpB80+s7sebIOb6ow720KKgJ0ecEpBOiWTa6ssaunwcbfLhB6MLoYocHYVnjUnag8ETq5CJT1aZfMNGSBwsERO03uDZ80RIcpMdPLJmX7cd6bH636NtRlQKY+zq3I9L9oDzqnso6ux0cMMbTzaeoY4cjuZcKVSHidbZdKOEoFc6REnW+NJc8bZVRgc6Vg2yuIwNthWPe7HFmtNRkQlXT4+Bo27vsV102E/NFiZEZX0uBtaDF7haXue6WvKNK7WExLMY72phkz+wa7Mx5E0QV2Q2FYDtPD4wd0NAODF5ggbVG6rDzJriTJxyXFMBE9pQVEiWaexygQfx4GMNf6hNDKgswatseidzvn4+hAFDn0N5yV2qz4jlJXyyfgKOA0Vdm0PH2Q2/jJSF4nddFuP6J1GNxpsmoGMX6dxf2xQ1w6bZsDoFFyBfPKPGzOpMhrs+YcpGUCsRerSj6eUEW1l0VtNTKPNiGNXY9UME7IqIrQK8EFgsBpNZbPx4IJEY1z+cRsTOkSIJ1AbyruqLU59BZMMcCUjBkdkIUJEDNakH2EiholR4NRVMMajqUjH0nqS1nFOom1s7jv3JQQJk9oiBMU/CwFUxiECGEeNuqLPNsVjCxFhrYYxDqdDgxgEtlddvh+sVfka/qEf+goxAHVr87uM0TtGBoWIGPoKq3WP06FB3drEWErldqeKJCcASBXgRoVmNWZSkGEgAb2mHUkWSUSYFO8NAMO+RrUdobUnuR9PCBAiaZVW2zHFkWvUzYi+qyh2fNTQxudFgRDUxxgontwOGjLFTuvawVuVWU6ZcdWNGtEL6NonBIwWOSJpp/KiRWkPbxXCqKBaR/HaKXYcAHTt4QYFVYW86AhWZn3SGATifYXYeIq1jkAcSXNTbS2CpcWbTHHm8aQht5Y0WgdJn/cGYuUzshY7NSE9EYCJEztqSAhSr2iFk44LKxEbT8gSAOjUvl4BJkCcKT8jItEEipNmtt1xiqfNW/eKNEYpAyg2MsXkIlAdSPIxUCSnFFcecq8pns9EwCPHw3LsbDQxxfRSbK+IoBjnFPcZWjouBwHfRorPTTGzvo5ZUiVUdC5UVB6Q4n0DSEIGgFuFLFUTVSrHA6FJcdUpRjfHIosivraKJImT2gVJ7STtTpLhoJjZCNUnfcYUI6fPAnb35fVG5JhikCNy+wWofJ/bTvVn+Ys0VapHltSQFikmHJNEkIg5nhgRFBcuyI2TJV4y0CMBfaK2ZB3GQVCMbil94qcYXzlSrK1M4xsFZmNaMqJS/HIacwkEFaEGkcebJHaoXSwrw7GnqscshjNqbh/F6VKFmKRlONaWdRnFNG4iUhmEiCJrweZjfD7lnyFbxVjNUK841R0FshZkqXW5tFg4fpXTEnQWcRrzHOsYp7LL9jBYxXXnY+9CO/EQdCY0FbO45Fn5RZnL2NtleoAMPoaA8vcSwX5Q1vzgA13JS1DtW9Klfj9IDyzUh1n+9//i3/ndGONfeXttX2968usv4t/4r//WV17u3/9n/u7PvG/AN9yw/ODP3cR/47/957B3LTGQpti8MWh0iZ2SGTtLxtEAgVZZ2gWOEo2y2I8tIXNpd7BRDrVyGLzG0dVolE3GmJy5vUpE7G2T2Q1Z5oMZKHtvZhIe7Kg3BpV3vkNqM7NwsrsmG4AAshvjMn6Jd0u1IIMxRmIHHLzG1pCBZj2xNfZez1ggXZDonMkuvUoQkyMbNOzuW8apsKuwkeSaye6y7CJbxi+5IFFJMjxN2j1ltkQ2QGvtMCSGPTa+Bq9gZECtHTprUGtHRkFCUo30OI71xZjOkqXPB4mhMG5yfGmSEVAionfU36crQqdu025xncab+wFMLryNdtnFFZhkL2K6J8ZkWDeaGDFtMsyZnXFMBjgzC3ISImZjc2JKFXCJXMonl+bB6WTsTcYfu6YyMZVMu8GDTSymKmBdjVnvlY1HrmPpCssGSukay+VwXqOofucljPa5bibCkskAdF5ml2V2wdQqZHdZVzBUMtrNbrSDU6i1z3+dl9kIZwbKmD6zZi2/0Tg/b3Tw/cFGaenKqop7lu/3kvmzdDXm54OfRZbjWDcjGbR9lQ0vdivl71LGbAyX85/j27K7KfL8NRVJjrDBCQC1cdklmGVFrJvce9noHUcNrUPa/Z6ub+sR3VDlHXHugxBY7LBT+40hgjKtA5ybVldcF7dd65B36zl/uXsOkPEsZYS16oGrLO+ss3FJ/UCuk9sLIJfvvcyujiU7JURE21iMVs/qFoI0iGUy1DmvMWRkSxVgjMfQG1S1hR113lXn63kcnZOIhSRJDBLaeBrv5PrJu/5lHnaBpc0MP4939POQBzbImfCBnxVGr6lMMbnyinkcaOR71CpUjSOkoWAGBSbEwXtJjJ5ekYsuMHMv9k5CqpjZPQUAxfOf+hiDzBsGwQtIdr9NBGe8QcJlUfuRNxj4eOSFP0t0JPdXLkuaJH0SRHb3lYloLaaNkmAl5XMyu6ZKE+D7L++sJY1HsIqMBjVZeqzpzJsaQgdqO29EABA6uUn7hStvQNr8mJPlZXdtAJmYTkzfRe1JW5fP8ybHEsbhMpk4rtA+ziRyBcEedRRz99cgCldeTHkKortZPSUhWsDk7lzKqpRGAIrPEXPXWIEcg5nJ1kqitdIou2hwJISLZU3K4WFD7F2ssGxMXzJs2HBbGHLAZPRNFRZtXFqnwFRB+bE4Vh5/29+HY/BI38rzF9Jj2d8bGuR7f9HGXP77lLNsd2mUfkET5nv/4c/esLz+9RfxN/+rf+UrL/d//Gf/859534BvuCusDQr/6PAtPK3PGILCJ90OAHBddbgyHX50eoJKenTO4Kru8P3DUzxtTnBB4ePjFWrt0GqLz89bvFgd8Ml5h201YHAan523OPY1Ku3xS1dv8PHxCq2xRNhjSUrhx4cWPgj8wm6PNz25yO37Gt/aHPGj/RMIEXHV9LizJhshMi0o19WIRlv83qsX0DKgqSycVxgdIW7jqPFke8507qfkblWp6Q0VosDnxw3FSqb8SHlXlcUPPnuKurFY1SP+8HCDzWqAVgGf7bd5Ufd0e8J910CrgPNQEcIlIsahglEBL/sNnJN5MbBpB/RWwyaDpTIOXVehacfsZsJxXZX2GJ3Ctk1ug2nh2lYWr48rrGqLz+62WLcD+tFg0w64Pa6waQechgrnU4PtpsPnt1tUtYNRHp+92SFYie1Vh2HUacGrUmyXgLOqcAEB2tWQUSUpA4QA+t7AGKJDX7c0Jr/3Bx8BEdi9OGK0GmOvoStinq0bm91CQhD4/LSDEJHOR6CqPGxa0NlB58WSOxlAB5jG5QWbGzR07eBGDV05xCAzNXsIEi+PBvACcktolpC04L19vSEk6KyhtpZiu0YFWfncLmbwrdcjnNXwvYJZUTluVHh1R+hNRmt4QSIBWXkgYmIGTotCZunNMYCDgqjSYiYt1uTKIXSaWFRXDvGoITaOGHwjLXaY+Tf2CqJ1hNZUtJBSGbmihRk93LQIla3D4WQg1xaHgwFanxGp0CtajAlQeWlhJRKj7vHeINYBckUoZBypzWIUiBsuJ1CdPBZVmBYYA8WQofFUtpWpzaDPqW6MEqg9hh9sgSAQno95EY1BQrR+WkA6AXlWhG7s/FQX5zdTzFR/VAhPLMY3G/itTygWNXO810m4nlhvZS8Rtj4vuERHYx83DmOnCPEqUDD3Zgt74wEdMHYqo1IiCIxnAfvEE5IwCsSVhztohDbAdpJYbzmJCJuYmmMdYM/ExCtHAb8KsL0kplsBQskAuLOEdHSekSkoWjjKka5HYj/2ZwlhBfwm0BjpiHikefHrgHiS8E2YYrwSuhLTWI2vNnCriNDQ99gTKuauQmZVzWylBwF/HRAHCXGmz/5uhbAJE/J0TG5bIS1w65jQP5HRm0AetogmrXl7AawisYAC9KurIkQvIAyAY4qH5IVTFYGEsEEAskti6AKQCbWCBORZ5Oc3qAh5FpCrpBFYAcIis6zqTiBuI9QbASkJ0ROJ1RmJqTVUgDAR6iwQ6gjVUflqpDGKAhBrOq47wK0BkRhipQdcM6FwzBgbaiApb8HXVI8aqA92C8gRmVFWWWqLT+UwcuSbos9DKqsBzJHylIyjqqe8vqF67Bowp1TmSPNjDhH9zaPL5Xcm3QGOVMsIIU3GhnA0Jtx21dPYA8jMtvqU2lxNqB4jmOY0fZ5QQ2SUjNlUeWEeFVDda7gVGWBR01gzC/IMMdN0jeppTuQ41U1zEOEbYlhmWgVpaWwz06wCVB8RlcjIl/ARwQioke7PYOjdpbuIwAyxiUU3GAHVUz3AdH0UdD0E4I2A9PSeCEpM96dPLMf8jBkB4WJGKzNimfqUUbkwjZ30Rf94XAqjNHJfmeWZEd+IzPbM81kaR6XBVJY5IcYRQU99BrBAUIt7MU71AciI5qy+EBGlmP5G+itdRFA0frF8n6RyIWhMqbx40SDLbSqM7DzXiz5zP0oUfZaWxxbtWdrTD9pQXE9tisiB4/nEF7QqAXzvC1/xp+mLpm+0Yfktc4//4Dv/E/po0AeTtBk1JALOsYa5cbBRz9hfPQRs1GhE0oWMBhJ0XSMtTqFGI8ZZPYfQonlm0/XE1mqjSjqXFWxUuTwpYtZpBJI+ZXqbKITieg0PgeYDO2OyLfOyziaApMsoZuVxUoiQSZeSU4gSq18a0UcN0ur06JOfCmtlhijzuLGGpo0qM96WTLglO26AmLHq8nXE7KryNSVzbkkCxN/5L2tzMpsu94O1KJkdl8eJ3JH1oyy00xgk7bdCl5M/c11cZv2Rg0LAOVDMHmtyKoQ85jNktGhnOW+EBKtZ/qUQLsXZ+hlTLydyxZY5sJvZcVs1Jl3QkPoeMgrL5Zesufx9DDoTUXHZJ1/NyKkAZPZcJrriOef2MtpbEjRx/2yUiWk4ZA8BPsZsv5y4/exdwO1dspjy/cnkWqydCSBrSS7ZZfN4p7Fj8q7cn9RGKcKD4yUzMCctQnYLlwViWRJAzdtM5bDXAx8LRT+lCJk8i1Hpsgy+d9gzYgwaza9ajH7+quYYYpo7lYnLuB9lTDePVUkW0CiL3pM+Y+lRweOWzyWEnZluL7ENl4RTJSPukiG3nOP5uNDf0lOCvy+ZZtmzgonDlv1aziPraLI2Jl87hqlP3C+dPDaUDPReSvGHpYZm6REBzAmYyjzM8db0aQAAIABJREFUSMt5+R5avq9CJA+TWIwFI9GXTJ8lELVk/bVFPVx3SZbGc5VBsOK9uUTjLzHfsh4k3xd8Lbvj87Xsls/yX3TtROAVo4AOc01GxYRTYd7z5VgCU1hJbvdivGTKqwq9SD4f0/EvnUSE5OtFnCHDSJJbMQgEOSHCjJi7Qi4sCmSkPkYBJGS6ZJ0tUeWL7NSJGIrX1xldXnxmrwApy/KmfPn7EkoqvpfkYQ9YiXnjEVTGA8bfOKHKGZFdTBrpTGIyaKK4jEbFOXnY/Pql4bG8trwgfY+L9jxmJC3z/zTpXfbQY+hhfExjMj5yfFFWjA9fLG8bI+Ct/X0n2vi+aOQ7Et2Klxhg32Multf89+/Rpj+G9PMcY/mNNixPocb/dv6VbFSdfQ0AMNJBIeKcnPlZSuTgG2xVjz6YbJiwkVJLlxbuZHTYqPJC7dqccfT1zMAJUWS5jp3uMaTF7hB0drlVYi6qDtDCh+uTIuDgSFCLF8G8iA9RpjxpwZkW/qUxIJOLLy/Aa+Vy+UZ6HGyNSpLY9/3YYKVtXlSPaTtxa4aZ4Dkv3nkRzPGUvCCppM/stLxwZBbc5cKTF9+V9BgzA++0qC4ZdJm5tncGjSa3496bvABmN+Gzo4D+dSIP4sU/Gxncthz0n9h5eWEJzEXoK+mhpcerbgMA2NU9Macmo7dcOHMaPEnbsLFcLt4Gr3M9zFy7HAf+zuWWi2HrFUav0Jop/k4XLKaDV2gSAQuXwT/qeTGdXHV5jAFaAHeWiBmq5IYcMS0sy0VaaaD7tCDjPnKdLIIuRczuv+wePjiNKrkBA8iou48CzitU2mWhdWDOpJsNiNQXdhmutEvxnSEbGxxjCyDHi+b6REQ/moyc8/iweLpWIedj5lxeBPOiiRfCfMwVRAvMbMsC7lqFTOywqi2YSZfrKr8TEYxAXcRzToypE1suEQtRjCnH0uZ7MDHP8rEQBCrt8/xxrGqlE8tymgMuYRgMqspN7umFC661CnXtZuQSY4op5XxTjCUTTYiZ2y+7rLL7Lv1Lc5xYWsuY0ZK5lseA3XO5LB4rdovVyfWTmU55HLiuGIGxN1A6ZPIJdps1FbmEIk5SGt4paOPJFdUL6MrDjal8NmjSuPLCVybWWl5QZ3ZUgNhVI+ZumxHZ5TMklDo6iXL7XqiQrwWQy6NCkcc9ODlt4IuYvQu4jMgswTIiWglZe/IsKGJQs/HASLcK5I3AzKcRmDG2mkDHvZjHlkYkV9PUoKV2b3mMYz+Tnm7uV+maWbat0LnN+VWc2FBT/0mjViT0Lj7UzE1arXJMsahfMs20XENCnCKmepN0hij0kvO82eSKmbWP00JfYNKbLcZxFi9Xjm/qM+nGpvMCcx1LNhxT3CAABI4hZCRUYK5BWbwkGH0rUTaOoWW3UkL64nxeANJjLtGt1OesO8vHi7q4D5OpNLWH82Zbt4jty3qW+V6Yyp257HKeAoksE/V1uu8eGDHFXMwvnLczlv3m62SRt2zjJUN22Zf3ibFM8/TT6li+rY5Zu5b9uHD9W9MXNTgfM/i/aL1/QlJcbGL/vKVvtGHZCItfqz9GgExalmqGHhFaSQilEgFj1A+MQ85nhMvoFSFXda7HCI8brWZIIaFZER4CHNXl01OnEnLHyAOAGbpXfu6NmSF85S48o4jlcU4TojXX1+RjRnjsDbESGuHxop60NRlZdUFmZO4S6seo3KVd9nKsPOa7v9koSEY6I49Lw/MSmhOqaf5cVNDJ8KfyuP1yhorx7nWJ/F1C8ZaIHp9Xgtj3AGT0xpV50rgzcsb1lue5TYwmllqbUwxqmPq5QDF53Eo9zbJe3pwoDdKL18c5SlSiEaPX0NIvxl3M0KhL0jDlXJXjyp95XFyUtEGS5u1iHxf3d4mu8fmyXpbZ0SLANXPU55KO5QyBaabx4OQSmlIiTy6U98M0XiyhU+7Sl8bdctfeNTJvViz7W37n+rhdy7kq26plgK3HzHKY31mFUc1GtlGTYfkuXdGxUVk2h/o6IUAc61y2oUTclm0tfyDfthYo1ySXPAwupbIfy2OMDC77Wb5nXDNmKRyA5syH+QbCVC6hZhxjrFW654r7p5TIudR+UYz9Y+86/s6bNmHRB1lstHCbsfjOxvcc0ZrQp0vyNVJGuLRhwOeXCFQZn8koW5lvYoOcS8ksV6Fc5wxF4rp4TgsmybJvHKeZ2ygnq2PZzhkzZG4LIJNhLgUQQ/oepvawZM2XSbm8wniJi+dCCEzu8MWCvByLLAlTbA7xtTQWi+dHFHWm757rSGV6PxmzOWtpwFwyZkpLaGaUifn1AAI/CmU+3hTgMsprF8kvDywNwEcNFTFvf9mfsr3lM7kwwsu4xQdtyOMS89+L+d5l4Cz6XQ7Ho6+7L2pkPZIWt9rD68Xi+xeo5+JYlJV+gbIulnfx+gvW8zvSN8Ww/HlP32jDcogGvzd8hCt1wjnUeOWIxvpKdVjJAZ/Y6xlS+fm4xVNzgo0Ke9egVZYMMNfgqTnh3rdopEUfDA6uQZeQsl9uX+G1XWdGVjak3oxrhCjwrXqf0dGTq/G8OuDlSG25Nmd0vsrugEoQqtUqi7Ue8KPzEwDA1vQZfRyDwug1ntYn1Alx21tirG2VzYtvG1S6hlhknzdHAMDB1VirET85X2FjBmzNgNfDGjvTJxbcGqPXCBD4sL3HydUZ4WS5k8HrjHSWhsKu6nB21czF8mhr7Ko+IYs+0+Y32uLsKjytTzi5Kp9faYu7scXO9NjbhphmvcZKjzi7ChszwEWJl90Gz9sjMdrqEZVyuB1W6KzBB+t9Jkli5NJFmd3WePF+VfWTS2gy1g62Rp0Q1o0ZUEmP3799DiEivr29w9lV2Q0uQGTWWoAWgfuBUOYlW6xExNFOrLCHgWQMVsZmwqLjWGFTjThbg1Vipi1R4H1fw3mFbZIK0DKgVg77JDNzGipctYSqsh5mqZcZo8BV3WPwGoehypIDvdO4P7VoKotdM6B3GtbzAj3CKNoW4TKVmFw1hYgwifBosDoT77hEgNRUFoMlRuNVZXHs68w8yyigTgy7g9VoK5vlR5yna1jrs0+xu9YrCCDnbSuLc5KQYOIhbosUMUtZlAy3+1ODqnJoTEKtE5GPtQrrlsbFqJDbTt99Xkgz8VHJKssyImNiDRbJqNMy4HhqEILA9dXpAerI7m4uSAyDRggSbTu53HNMMDPuChHRdxU26x77Y4u2HTNSLACcTk2mjVcqYBw1VqshI33jqBE81TGMaWPGTIyz532D1a7P13KccgwCttdotxybrFBVE5HNOBiYyj1ghQ1ewlQukd+Q9qOpHBG6CNIuZITSjhrBC5iaYo9lOk/ELURtHzyR4NhRE8NsY3M88thR8KKpHWyKhWYXO++JFVYm48TtK4jGQ6Z46TAqIAB65SiGOBK5TAwC4aShtxbeUiyw2lr4g4FcOUJ2BOC7BLcwKUsVZiQtQgXExForqnTcSojGIzLCxojhoCYWWWBa+KU44YwoDmpaMSXmXshIMcB8HTO/prhlmDCRsahIzLIrB7E3ZNRUCcFLJCvCJxSOyzH0V0TMWGFj4yFGCdlLuBQjKxwSe2vMSFg0idE1scsCQKyoLpnKdWu6nlEvaQlxdG3IsbLCE9ss91OkuNhYRcgzxe8GBTIwnYBK530VoRJ7K7HCJsZaCZgOsLsvvwKVPTHiAoAcxQxN83ViXJURyoopVi7FGate5HhFaZHRxShTvCboXEbjOA4WFO8ogLxgn7HCxhSnOqS54jjBVA4jlnJMY8GssBITK6yJs/hEjjck3UkqU450D2dNx2LuogDVLTCxxwKZFTaqmFlpAUwkPCKNhcAUG5oerxw77Yq+SMxZYcPimgvGDverjBdcGkdRUvxlUJfzCDeN47zwqYwl06yIyHGh1OepPZeQzrKPuV5GZC/Ul/+mZyX3U0zXc72z8eVH6h3G86xdy3w8ruU4XciT02JeSobYB4yxswK4svLiy3U+2t4/gelnqWMphFAA/iGAn8QY/+ZXXv43mRX2oz93Hf/Nv/ebeDluESCwTS6pJ1fj5Co8q484OTIijq7CTg+4sy1CFLiuOhxdBYpfs3g9rHBV9TjYGloGMvzUiM4bvBw22Jl+5s7pgswo1+thjUaR62IlPfa2wXV1Bkt/VMrN2i1FRO8MxqDwLBmDZ1flOB+Kh/N4PayykVYph9HrzMrKqZLk2lhJh7txhRBFdh+9qU9kJHmDnelxdCQdstJjNmZuhxVq5TJz7MmRgcxI6VqP5D6b3qidMzDSz4yhRpEBycy1OfYuKBr7keaA46f4eOcM1mZE58iA75yhNtgKtXbYmgF3Q5v1L61X2NU9KumyxuQydondUxkN2g9NjmNiJGFlxuyyerYGPkh8e0v6Z5+dt8RIq1we697p7MopRcyapWdbJfdXQn9CFGiNxeCo7JUZyQXVGWLtTC6urDHWp3wcmyQAbKoBWgbc9W3u1+A0dk2PwWlsqgGHsYYAcp84ror1TQ9DhUp7rIzFaayyi+qz9oSDrXEYalTKz2Lj2JhRCcHjWLHyOzPicn85noo1zbQMOFuDbT3gMNTZWGUXWGbKLSVX2G2T0Ql2+WT32N5qtMbhPBpskowKu7OaxPoaI7mVKhmIPTXtnrMBXTLjauVhZMAxsZ76IOlYMkbLtrCrKl/PbLcAMrttSG0encKTVQchYtbNBMgVtbfE4ivSs7+qRzIOB+bdR44DY2MdAFb1iPtzi+tVh0NPskJs0G2TnAsjcJX2OCV5mxgFGkPG37nQrmSXWAC4bnvcdQ18Yhxm7U4hItaVxV3XgBmm+9GgrUcMyZWa6wXot56Nezai2QDn/D4QZstSJ5X22XVYyUAIZooTIxIumdloa+NyXna7ZUOfGYh5AwMoGEeTAf1kc8Z5qNK8knyNFMCxq2dstgCwbkYczjVJ1BiHw7nGuh3QDVVG5uqkncdosfUKzqlMDOa9zPI07BZcaZ81SJmdNwSRDf22Hmexms7LVGZiB65cHp8STWQdPyYrq41L7t+0IcA6et7LzHK7WfeZyVmIyXWYSdfYDZo2FFx+lzLSP/S0sVAbYg7WiQ2an2OePyaHc1bljQg+ZgyN+5DclAFk12ghIuyooXSA94QsWlvEIqdx906hqh28p3dBTNfrVDbL4thRpw0OmTcs6sbifJo8kr5oMpWDHRJ7eiI2AwAhI9yoiNE2Tqy/xPRK1zIhHDMn84ZTDIAyHogiI7HAHOmOpdswgBgEqoZYi4UAuW+neypy/GraEAnJzVaaQARtKrliR5BskQARugVyDQeQ3akFtzMx/MYosquykDETvsVIZG8AaCMnu4RHIm7zAsIEIlHjfrBhVbDlZqOhZIVluR2k4+xGzOWwm3DAZeSSDabSLZVTMgZ5o4Vdpssk4nTuMurL7cQDZDWKyQh/4Ep6CWLkdnLbuS1LlFBg7jLOTL9vQY3LjYoH6ZLBWtZ7Ib23ATcbqwuW3zvQyeWh8vtyqt/Vpu//uz97VtirX/9W/Gu//a9+5eX+z7/5n71X34QQvwXgrwDYfR2G5TcasQyQ2LsGNkoivUCNAIEhxSp2vkKAwNFVcEFh72r0yaDovEGftCfZxa/zhlz6EvpkkyEZIpE9ZEKLEJK2JN3BLkiM0NmNMUSBo6UfrjG5CHIsIICE6pEEwzm1jV3m+uQaeBYxt4/JJ9goAzBzrxuDRiVJezHH6gWFs6twdhVsUNijIY1HELp5xuRCx32Am1wFg6RyOmdm7lZsZPggc10+EMrqo5wRnQxO57g8QEPFkA0hgGIdYyRj14pEPOE1IpCNMxskjmOd3fWOYw0lTZYR4fpYPsSKSZsRAGzqT8Tkcna21UwqxAeB+7EFM/aG5FLIBpst4vc8kKVOSkmQMsZxzG6KVTbakGIQI5C1NtnYIu3IuTsml82Gw9maLKXBMhXTXJDcCIDsIjamvYyxkKC4G1rYtKiMUWAs5pWNMTYwYhTwUuRrSzdD5yVGMZFwkJwIuREONs2bU7nMUBimPogc0+gS+QfPRYxT3GZpaHFfT6NJ107jnd0Lk/Hmw+TifhqrjFJyGdpLjEXMagg0J66YSzEbl2IRnvoqRCJVSot9NiIOQ0Uxl4W8CMcvcqLjVUZEJ+Rvao8XU8ys9xKnoYItjCcAOKa/IRkVLshcb4wCiZg0xSNqiiH0EkJQW46ygi1kOErXRTIM6JoYBZyT6IWBcyqXUxqWpSHIdXrF2qAmG2Wla6X1ZDD4dN+HIFAaXVJKeC8QIxlkbFzEQH1juQoyIkVeZfBYs5zGvmvgnKI8mAwpZ+kYLcgTipv6yK6ezikSXE+I74MNeEGuiOS2ScZATO0v20D1y5w3AsnYSEQ9sZrNbUwbJz6tnjgWNJOj8PXFHMRA480xoCwpwuMdvIR3EqeungxUFOvfZMDEIDCk+46vZzSYyhEIkcbJO5XlTbjvOd6b40fTHNJ4kNHiHHkjeEdt4oZ4SchscNQXRMA7ZGMlRiCo9NkLjFGnuFVqd1TTsxYdzQMZS5riTZPcSAhyYp/+EmkMIjNqz4iGUixrSEZOUCK3j40Ny+31AlHJyY040LyzfEp2PWW5ETaagJmBNcZpfBAErJPZuGJ5Dj4HpOfDCXg2xCImIyrVzddFSWVFOcXaBmbPTjdOxOJ4vven9kY2gCIItV8aWUAhmYIJpSwMy9LQE1HQD3H2+8ZkCGX0bGFUJYtjJjeSnq+SAVU4ard4wD4KCFnMARYGDI8HGzZF/zLpzEXD8hFLKDVSREzoZWlJzQwvMSO3uUxyk9JjcaJFH5Yvuseyz/rxPmmW91KpeWK+GsTxqyjj60pxHkLyx5mEEN8G8C8B+I8B/NbXUcc32rCshcWvtp/hldsmltYRPkqcTYWjr/HMHHHwDYzw6LzBTve4dYTqPTFnHF0ND4mVHPFy3OLanHHyNSQiWjVipUbYoPDj/ho31TnH15lkJK7VgBAFXqoNNnqEjwIbNeDluMFNlaQ/kpspgFlsWeeJQOhb9R4eMucrGTzfjOsc68fkPhzDByCjrQCw0x1ejpt8bAgKz6sjjr7GydW4Nh32roYLCms9ZvfNW9Vm9HGrBxxcPYu7a5TN/aX+VNCCED2bjOa1GnHyFarEOsppDCojuI2yuR52/2S317MzRAKUyHrOjsq6qvrswnt2Bi4q7EwPIz1uh1VGcUsWzSUhztHWuR8uxYuSXqnJ2p82KPzi5hYAIbWV8mjUxMTJLJ8cm7dJSPU5obtlfBu782oZsNHk0ntMKDijxcfkcnyyNe3QJ6IgKSI2ZoAWAXtL7rYukHvvVUWI+cYMuB9aQkISaQ7H2XLfGSFutcUhbXDUyuGmPuPsDG7VKmuZ0n00bVgwUs2xdTnGN70EW036ryaNNWuh1spBy4CTrXBV9bgfm6yB6uOkJdlqm/NznYPT+X5jV1NijZUYnM7XMHLNxiMjtiGSnqsWIevDAkTE1DmDPhnijKqyDipvVhgZss5qZ03uFxMona3JCDEbpPzZR4FaeQxe4WlLz/wruc4GY608ztaQu2facFkZC4mIU0X3TwS5yPMmDG+KbMyI277FTXvG/dBkoxsAtvWQdW25/+x67YPEKrX9NBL6zxsaHJd4057xRq9gvURjXHanBgg1v+vbPMadNlhVhMTX2qUNId5oIAQSoE2M1rh873B+Lpddr7m+01jlvlI/Jm1U3nCpU97O8iabQFtR39hFurc6I3mss8rfn67POAx1rrtNbtGnhJoCE0nTqrI4KCJ7ao3FvWqwbQacEilSBLCux+wqrdLm2ehU1kflNk+bEIRYnocKOrlZ85xV2sF5hU0z5M0SHkebNl1iGq+82ZPuAR9FdvHmzSdGLEv3bIA2QWrtcR4Mrtdddkkv4zS57S7IjErXyXug1NztRkKh2+Tyri+4pvsgMLppg6lO7eR3QKUdpADOg8lzwKECvOGi031Q6rOWGqaErFoywDPBk88EVtYpGO0xjBpV5WaSVKvaYn+i9+uXSVXlMAyk36HNRKolZcTQG+ik+VpVPhu6vPlR1UQaRZqvPm+kxDCdC4WxyghwLgOTPRW8RNOOGHnjI8wRS16d8+YJAGqbndzNIyY91AnpBCAm5mYpOJZW5A0/fgYYEVea0FafjFxlfG63lAHBKwRPCK9nI1JMmxYyIZa8+YAIQjkZlCwQy+hF0uLE1MdkKMcLRit1TmS23hnSGZGOYdLjZJ3PMkVM5/KxC4bBJWiNDetLaWlsPjA88dBwlpgMcEZbw+TWPndlTZtPIjHFhqK8S21ZtqM8fin/JcN4eawwkpfZLh1/tA0XjN4/TV8q/acA/j0A26+rgm+0YXn0NX7n9s8SKhimHcgAkZg/nz9wlWSkjWUwgDmBB78wS3dT6xV+JJ/MUClgIo94UEcU+D4mggZeNALz5yNEgR+IGwCTmyUnkY5dIqcAHpJHXCKN+AP1LLutPUbiUe64SkELxTI/jQVm+ZcEJsu28fGyXUuyjyUZUDk2jBoxIU9+p6b8PF9lG5fEKtyGZZ+X+fhH/OVxTWNeuKSVxBdlOTPSisUYMLlFyYLJ35dtDfmemxYLJTsmnwOAT8UOTMCxRJmWafrRLwkhIv5IPckLmSVpx7K/nEpx9JKU423nPhZXc8KO5Q/tI/cKt3NJKlKSeDDxx7Lbl34Hacyme4CJPnKeOFHgTxcv2grMSTb4muXYR+Clovf0A9KO5Q4wb/qXHEzLBUU6FoPAG7VBDPNxennBPaqk8ec+5UUT5xVTW0u2UVHIEJDrWtp+T+5Vd6Wb1YV7ho/f8nkRJ7c6fr6L+6YkL7l4vwHz+6p4T90xihMEbgvk5dI43r3ZTItNYO4ytxiXW4EJkZAAvMBerWeLycOlBecSSeD3S3l8OQ+8QIzAXq7n9x2fz3UW38vjb10czttxSPN3fp0EGJdzeeG5PJT94XMROAng9kLVs/u9WNh2l1z5gMyeyvmWn/PfYu5zVHIErGynuiJJ33acNQB9WniPi3Hqynn+EskWhoK7sFrmwBfHiFiR5VwMty3X37Foe7Gotvy9/FvkO8tm1hefxn85dVzPmD77ojxflCeKzCH1JxTzX37Ot3HxauBfz5lHZmqySh/UW8ZeF/nfZtBcetTEcpweuXaZpusuPVCP5H1Lee9b7xeq523lvuW35ou04611fsH2f+Hyv4Z6v1R9f0wp4mtDLJ8JIf5h8f23Y4y/zV+EEH8TwOcxxt8VQvyNr6MBwDfcsFypEf/U9R9iCIY0KQudRoDIbUr2V2AyaiYNQpFZS4kEZtJ/Yxe9OjFdAnM9uMkwfcjyuWQAzCyc6Vp2ZX1MkzAzg17QJiy/c95Sy4+lSUqJjUtafOX3ss7y88zdcNa/OctqOQYck8ljeXF88FCDbtmGpcHO53l3eznOM3bK0hAs+sjnlkYox1wyusyMoCWrZ9nWpWG/HCNuJyMyNC4xozHLNpSuxhHIrJycXIFUPCYNwm1g9G+Zl41xk0hgyrT8TeLvy02AUt8ux7UllIzHu9Szm1gSJ/fOMq50mfImQ5w2eDjvcowZQZ5cY5G18ICHunlUflk28rGyzrJ83qVfng+LcSjdcPneBJDbXhqFZTzrst/L55GR0UvPKo8rt4fjFcu5eYz90zmVYwx5k6TcCGEUY9oEmYzASxrVy02Vt21msQuxXMYxFfnL65d1lgbqchNnudHCZED52oT8ydy/ot+e4vDYRTXLnizmSSzGN7cTmJ2bbUSUUhFpgyOyAX6BPfRBf9O5mTbhcqxTPBwVQNfkc0EQYRBLmywXo9y2wkqYNiomayXrEHoxEQxxXFtCmqicdCkwN9YjJrfIOjx0lwQQZUgumSnuToapreWGFY9bYUw+MJS5P2UKAMxPseqMAsTMgnnZEfPVbKT/IqY2Ck/GMhWRjLZ0/kEcXi4DOc+D5DGt4kT6Lmb7AMmdFdkSzGN9aQNEFHOY+pk3AESKMSz7yGWnKeJ5F34+VVFwH4Fopn5lN9Q891T/cnZKt1iIYl8kYpb5seGaDeui8Fj8t9xrmV38kMT4QVmXDd0LY/2WxPtQl8qdteeLGJVvu6/edexS2cVcPFbvhT2Xd9f3oJD3yPNI3V+P3fbVpa/JsHz1jhjLfxrAvyyE+BcBNAB2Qoj/Jsb4r32VjfhGG5a34wr/3Q//ciYY4Xiy0jUHoAlk8pRstCQ9NmBaoPIxJkNgAgh2pVkuJDmmqGRbdI6IEjgeSeuQ4odKw4YWVVLGFHgfZ3Es9I6j4H8A0469mNAyLgdpsR6LxSAvfnwK0heCAvWFogVNKH7AlAo5P1PIA8Vvy+LHLreTUS8BCspXcbbQoXYnoyJRs8+uSWgGa6/lMoqyopOQxuf4GCEo/gaBSACyvlqx+z4TeY4gUoA0nhmBSsyRMQIi0c7HxMwoap8XTszAmPPyoo63ePlHtBR9LpEQZoFkXTYZL7vblC9n1o5TxTkZp7ICtwtTmXlxmMpg5kiuK7VLOGJRfODSAzz8RVwumvhNPVuMYhoH/i4x6byVPz78sSRIKPXpOOUFR1rMSFwmVSjHLP3oi5DO871qBaAwXRORmQijLvItYnkyUpLGKKrJjaj8PG3Pp34zw2VV/GJ6AGU4V2LgRMSkhcf95kUyd9EJYu8cxay9iMismXnKuE+pnFkdF9BjOQoMVbFYLJ9xVzI3pj47QYyCifyh/E3MLINLZsKUnxeV0+Ka8mf2SxTnykWwBISjDtK4p74W8yKTRmF+vBflyVEgSKR7TWTNPq8nJIxj4qSfjksPBD0xY2a7y09DiUhtZbtLprmQ5SsnzXlmT+RbLeWLAij2Ieg6WVyLYgyLfkGksUmfo5jaymXMtAE9EA0gBxooZg3l+02kvkAUc8iul8Wc83MoHRDMRErC881lcn+jLJ4jtmlTf4Oe2pif4YgHbYtqunXnmorTZ74FB6B8AAAgAElEQVR3siZj6n+ew4KFU3gg7UF/qSQdlZvHBsX9oKe2i2Tk8RzkY9zf4l6Ngu6t5VpzpmN5YR06m3Mug++fhWFUns/MoTymfC8XLJ+zMRfTffTAu2IxL3lcxKJ+fj8Uz322q/O9KuY2GN+f4UJZxbFL4zSz8UXRn/I9gSnvu/LMfqvK38FZObHoB3ekfHcXv19luY+l8vm80K9Z/0o32OW9svwtfqTOS2W/s43vc/596/kK6/ip6/s5TDHGvwPg7wBAQiz/9ldtVALfcMPyw/oe//6v/C+QCLBR4xSY0ZTuIhsVxqhho0IjLA6hwUqOGeGspcUQDCGGiOijhgKhhEMw6ANpTG5Uj7OvYdIvGaNwQzDwUaKRNtfH321Uk4ZjgdYpQdqQjIKdQwWFkGPKQhQZZTXC5+OkmxlQS5dQzsS8WmhStsrOdCdProaWHmamBekzWgsg61iyZqUR0ypHiogh6IzgAlOsJ2s5lhqOpb4jy5X4KFCn2EvOR2NFnztvsqQKgBxv6KNIpESTnIhM+andPpMi0ZykRWJCmhkh5bjVEiEt6+D23Ftij31SdZkMipMudQnTHJH+4qT/KEXIWo48Nxz/yQiilgGjV2gS42yprziRRGmMXuU4Tp4HLovHBJh0PrldZXtDFDlmlc+fXIVauaw1Weo3ahkuIrSMlLKcS4ki8wZCrV0mjsqsvspldLNE23js+W+IxFjL/ed5XOq+1sph8KRHmxHCOGm4xihyTC9vbgxeJzkWim3j/jDizfIzPGdlrCwnzrtEoJfIrUqMuD5IbOvhATKb+5XIq2IUaPR03y89CphtuNEOp7FCo90MsRz9RNTDbLgcE0rPEdVRKZ9JqMp10HmosKrH3KayndYpNCmO0QeZY3kZhV4izUTGI2FUgPUyu9SX38vwglKGhhOPJZNA8V8mmtIqIERy1+cNRK08nFfQys/c+MtYsG6oYIzL7yWXXME5LpRd7QHkuLwIQnSN9hithjHFPBVkWMxiOyG1yBuOJSLrPTEYs3s7s5/yZqVzcoZQPtDd9JPYvCw2uTieDQCR4uS4vQk95o0xZoZlIiK5QNGz+7wM8OneCmwQ8sZekoxxieFT6kgu2hybp0Pe5GPXaqkiMZJGOh+DyG7W0vi8IcfsopyvRE2lnp5x3iQRKiAkQieKv0ubezwmvHmmA2AloOOEnDoJ1A8UFd8/OUnlAhl9zZtuXI9E8nVN1/DmFrev3BBlAyqzvhZWRJ6goozCwhBWzjao8uZbYWCUm3h5k44RQlF8lrw7AuRNuGTwT23E3GhJG53CJVIcZpRNG6F500vSOZE2grgf2bBko5rjBlHUyYYjj7OIc5mLKB4YlpeM6uXmRGmolPZYtsuWRuMFxPIBQhmLE4s2LOucVbwoc7Zx98DAXVjA6S8TE126fS4iqbMyL3xftO2txtn7GJ4X+nopzYz7d5X5DU0Rlz0Bf17SN9qwfGW3+O0f/3U8a47ovcHrfg0pIq6qDlemxw+ON1nW4qrq8elpi5v2DCkiPjtu0RqLVlvcDw2+vb3DJ6dd1gC86xqc+xrGOPzKzSt8fLxCpTzR7zsNJSLuOiIA+PbVPd50KygZsO8afLTb45PDFgLAzfqM41DnWEYipSDChpWx+NHra0gZ0dZjpq73XsJZhd22g1Yk63Dsa9TGotKTzl6MAseeWP6GvsLV7gQA6IYKbT3i9s0GpnFYNSP2+xbtekSlHY7nmvTlZMTN1QndSOQiXW+wXfeQAhgcLbJPHeelt+p6NZDun6VtZGM8+nOFqnFwVkFpn7Xr6sZiGDSuNj2OXQ3vacFTVQ59b9A2FqdTg3Y1YBg0NqsBx3ON7brHYA3O+wbrqw6nQwNTO1SVw2nfIFqJ9U2HvjeEDqd6QxDwlhat7PLFenzjYKA06f4NZwNlyG1wtRpglMftD58AEWg+OMGOGm5QUDVRptetzfTw3kuMhwoQgGodEAXRzyfk2fUaMlHex32FqAPkyhFKqgNCp6HWFv6soVYuI81EmU/XwAnEHS3shYpEm3+s6Af8pIArS65igwRMhDSeFmVJt85sRzirEM8acpMMy15BvzHwqwBxPSL0alrkiAhU6dfWSapHB2Lqc4IWS+zK10vKa+W0SGg90CtatKw85F4jbB0t6HixpQO5z50V4tpDdCq7wonGTygyI8cuLZBaD3HSwMYB9wax9bRA0SmvSS5pg5y0+NKiT91r+DZQ+6KgPE5ADgJ+5+k6HYBRQqS2UpvoV012itaLK0/tHCRiQz5fYiStP8ikEVgHmJf0On35wmY3PdFLxLWfFqBOQB8UxVhd+YzMCicIFWL9QQHoewX31MG81rA7nxat9B4xtwohacNFHaHOEq+ufV7UqhMh+37nIM8kgBfqkNGj+nOF++ce0ZAeoBwF9ScImIPA/XMaHzkI+HWA3iv4dYA6SRrTtACNMkL2NPahDVBHiVBHqEHArQLUWU4aiMn9UJ0lpAPcmnTtoiIELcoIOUiEKkJaUDknCWkprwiENJkjLaDcOsIcSceP0S85ikJHEWheSgzrCN8ScqY7Qi33TwLUkDajEvJm7gWOT+m4OQicngZUbyS6XUwoSYQ5yKxPJwPgmwhb0cJYeMBX1D5I0lIUAdCdgN2SrmIUwGiov/os4CvqA930NKa+AbyJebFsTiKjUtzWoCOqNA4QhALVZ8Cu06PWROiB7rlggKYDxquI9ecCQQG+niMc0gKhon/1mc7rM51TI5URJWA3dF6fAbul66QFhAN8O2kz+iYdqwFzTq+WhhAz1dH4jTtADchtVAONoVvT52CoPJd4dqJMeSLgWqDaE3IYNZKGI6C7tIHQCuguwm4F9JHao0bKZ44Rw9Mvv/TRJ+o7AKh+Mnqko/rkQG3S54hg0j2m01weI4ISCBWg+oioaG6jAvSJ7rNg2FilMnnPkMvieQsKqO8i7JrQsaCpz8EAUYlsoEkXESq6VnURvhFQI+l/RiWghpjuO0EIr+d7gspUltrA5dOziowQ+0rQPALwdXp3nTFpZzrAGyBUAvoMuFXqR3pmIGhuoqA5z6izngwM4ZD1KoOm8ZMW2eiULo2dEjM0k42TkHQv2YtCRECEiCjEQ3Q755mMwyhpHIr95rmhxM9R+qkp9TKFp3HM7Yl4qFOJuXE6O79AuUUIiEpA+IAo09wrQLr0PaSdJxR7EOkYj8NkeE95y+80NgsjrzCQsy5maeAv0tIwfGAsl2OHovxLRqiY5mtexxe0Lr/BxuhXnWKM/wDAP/g6yv5GG5Y35oR/69u/g5/YJwhRYqs6jFHj1q1xDhX+0u6PcA4VVnLEK7vFs+cHfG53kIi4eX7CvW/ho8RW9fhkvMJfuPpxZpHdqh5X6oxzqPHD4QZ/6fpHOPgmI4k2KjzR9Iv56bjD5nqAB5X16XCFv/bs+wCA13aNVlkY4WcxcZ036EKF33z++5CIuHWrjFAa4VFLi0+Hq4xebtSAIWgMCx+etR6gELBRAz4ervOxk6vxne++wb1rsXctnn/3gFu3QucNrk0HlX51Pht2maFzpzvsXQtgihtd6yG33UPiYBu0ymZUzEWFa33O/XRB5djLk6vRKos34wrr52NCR4kBd2d67G2Dm+qEO9uiVRb3tsXVBx3ubYtKOjz/pSM+6a/w7DtHHFyDwWvcfPuMWlp8NuyovhQLyQgoo4uMRL4ZV4SkKWIzBYCt6dF5AyUi7scGLir883/192GEx/93fIFGuTyGRnocbJ1RYS0DrkwPKQIOtoEUEb3XGX3c6gGnpEl6U50wBI2DbaClxxg0tnrAvW3yX0YjdWKFfVafoIXHm3Gdxldi9Bq7qsuMvrdjCy1DRnNHr1CpiVH4blyhkqQDejsQWcdKj/iwucedXZHuqraZxZfQTULBK+nRe53RUEarGKVfaYuzM2iUy/fN0da5vLtxhef1ES+HDRpl87gRCy8x6u5tQ8y8zkBLP2PNbZSDTMy8IRKr767qsR8b3NRn7G2TUUdmHA1RZISXx0OKmFlwT8yMqx208FhpizdpXFhTlRFxZvR1QWJX9SRzkth7m7RJxYy8S5T2F3/jFhIRPzje5Oe8Vg6npHcKAFoEXNUdtAh43a9nqCMjuozCX9UdPj9v8cE/scfrfp1RTQB43h4xegUXVUa4eWMtRIGrigRH7oYWKzMiJASX0fNf+Mt3+MnpGtYrrMyYNWgB4KY+45PzDiGKrEN73XQ4jjVWSXeWY2pDJO1WJQLOtspMvK22OI50X7AnBjPqcn2HsUYlPXkRJBkZRr9NGt+1GVFrh/3QgNlOd3VP98ZYY1MNWT5IgNh6YySpIAHg29s77McmyRQFrM0IKSJedys02iXPApK/eNae8Nl5g0Y7bKsBL89rPF+d8KZbIYLQ26uaxlVJ0u482WoWA2uDxLYaCC11tGFXK4ejJZ1fHyR8Qr9bYzF6lfMDtPF4tlVGiEMU2FRjloHiOfJRYFcNk7yUV9hUA/ZDQwzbSVuW7nFiCb7rGnxrc3ygrasTK/Lg6NlZVyPO1uTy2VsgRIHDUGNlbGYONoo0jQmxrzK6f7Yms+Yykt6neWWkft83qNNnl5Bx1hbm+1HLgPM4/eaxZmo3GmyagVixA0kY1cbN2JBbY3Hoa6zqEaPT0Gn8r5oenx82+LJp1Qw49DUh6drluG4lA1zSEB6dQl2PGZ1ntLqqRzivMDoFndh+WY7JVJa8NoqwG9a6BJBlWlgaJkSBqu1hhwoQEd4rVJUlGZgyNlhMklnSWDinIdL4hiCBxMZrnc4hPUJM2qalNmxIrLuMijPyzkg/yxS1tc33LIcZOUdMvsNgMqLP4SXG+Bx+xOE+k5QPIeA5ntzLnA8ARNKQZdkdjl0GJvskh97whp9AtpqYiTbyXyYdK1JG3zlsAJhZStk+4zjjsg1JxzMnkb4/As3NyMoiLiLVD6BJzsfhMVE8MMzoJSYeMt7mTqQyyzL4+zJxeeFh+y+W+9OmbGx+BZbh3//pi/gq0pLw8ucpifhVTNTPKP3Sn9/Gv/0//JOwUeHs65mbqY1qRraydDMtU3ZTg4RCgC8UZBVCdpelvDIbVZx4Ee4KF9dcNsSMKKUkvSmPPZYetDX1qSS+KV0Xl6Qsy3pnBDsFK+6MBGfhDrmsj7+7ot1lGZdIRi6RAj3Wx2X+su08djzWpYvrpTaXBEaXiIHmY/SwreU5Njje1hcAM0OhPAYgu3BeIgQq63qMVOlSeozE57E5uUTmwm2LF+p8rJ+PtYXrWx57Vyo2RN957FKb3tbGx8iSgHm/l6kkHfoidVwqrzzHbp3LFIppZvfOx+pil9G3tf/SdWW/LqV3EVO9tfzis7hwrGzHY6zO7zM3j7X9ImNv2aYL5y4xWr9tHst6+POy/Pe5F8pY+WVZJVHSnH14sbh8S3qfuQYuAwRlXTM23/dq41T/JabutzcalxGMZbsu9H8WX79cfL/HeP1Uadnu8til+XtXOe+7LHtsbC4d+6L9v/jgzr8u2Xwf5H1bXy7cM/Pz83YvUbN83YVr32rrXBqz4rpLQ3jRVfSR9NZR/iLL7XeM36Vb/J1lva0Nb6nnYd7He/ne7fmi6Qtc87a6f+8/+q3ffQfBzdeetr/2QfyLf/df/8rL/V//hf/kZ9434BuOWJ5Dhe93z/Gj7glckHhan+GixMt+g+NY46P1fdZKfNWv8eFqj49PVxAi4kV7wN24wuA0ruoOHx+v8Kw94X6kHd8n9RlPqjNOrsb37p7hxfqIzpmsq+eixNPmBCkifnS4xsrYhBR0+OS0w4frPUKUeNWtsTbjzNDQMuA41jhbg+9evUGIAvdjm+PftAxY6xE/TogC7VoPGLzGaaxmsXm7uocUEddVhx8eniBEgV3dYz80+LXrz/Fy2OBNt8Ivbm/xk9MTjF7hSdPlOLuf7HdYV5Y0OM2I+yGhskm77LrtKB4x7bje9i1WxqLRFjaQ3tp102E/NGiSxiG37zDUWFcj3pxWWNUjmoRGdFZj1wy47xo8XZ9x17XkotvX2DYD9n0Nozw+3B7w8X6H5+sT9mONwWo8X5+wrXp8st9lTT2O/TLKZ61CjsV7eVoTOlDZrJV33fZZQ48F1P/iRz9GgMD/+/JbqLXHddvhMBCStu/rzNqrRMTNmtyp7zrSkxysTvppwK4ZcBoJoXqxPmJwGrd9i0r5Wb839YhDX0MWqIGSEc/XR6z0iJ8cr2jXNkh0o8GzzQm903i2OuHj/Q5aBayMxdkaWKegVSBNORFxdyYk4WZ9xutTQiwri3/s6jU+67Z4dVqhrWxGDgavMtEV77YzWlBpD+slXBrnTT2iswa1dqiT3t1dR3p/lfK4P63wwfaAz4+bHKc3OoUxoRWbesR9yn8eDZSMOA8mx4TxNU3SAjx0Da5XHW5PLZ5vT7jvEmLpJZrKYnS0Q75uRkAGErRPu/MfXu1x3zc4doQ4VtqjqSxWxmbEwjmFyjhUmlCX01Bl43HTEAp66GpyV68sDl0NIYDaUN3c5m6o8N2nbyBFxPc+f5aJtJrK4nhu8u6+1h43qw5GeXy63+ZFPyNt/Wigk7v78+0RH7+5wref3uGz/RbOTbv0H17vCRWSAoPV2DYDXu43GWW4WncQIuLNfo11O8AHidHqhD4Av/rsFb73+hmsVVg1I5rKYrAaSkZ8a3PAD97cTH3rKlxvO+xPDbarAae+yjI2MQrUtSVkqa+wW/forcaqsrg/tVg1A1x6h9kUE7ddDWiNxZvTirQc03zGKFAZl2MFh8Fgs+rRGIfb4wpKBVirsFsTanh/bHG16ZKbPY2NMQ4hEDoiZcSvvniJT49bnPoKUkbs2h5GBnxyu0NTp/vTkhv7i90RH7/ZoW0sbtZnfPzmCh/d3OOz+23WF7zZnRGiyDGv911Duouanr3BalytCVE9D/SO2TYD3hxXM31LaxU2qwH9aPBid6QxSAb9oa8xjhoqxaLtVn1GvlgL0nuJJxvymLHp+X22OeHVcQ2tPM59jcpQG0ensFv1eH23wS+/eI3OGRz6Gkb7/O6otcN5NOiGCtebM459jSerLv/GHMcaNki83q+xW/d4sT7iJ/dX+XmqlcObboXWWFivsO/rfE9cb7r8HGkV8GRF9+andzu09UihCk6jrSwq5XF7arGqSZ+0MQ6HM3kchCDQNEnD9FzjydUJg9WwTsE5iaax2DUDpIi4Pbe4anu82q/xdHfCvmvQGIfRKXy42+MPXz79cosOALt1j/tDS/d+YzN6bbTH3X6FurGwVmG7GpLG6nTvX287OC/RDVV+Nvh+uN528EFgsCajhrX2cAnt6hNyyyzV1ip8dLPHqyN5Poyjxm7doxtNflcw8jgMdO1m1ePU1Whret69l2hq8iw5dRWCV1S+iKgqj3FUqCpCE+2o0a4G+IQYKhWgVMhhLSEK9B3xXGzWfZI6Q9ZXHQeD1brH6Ui+zRyzGwONI9chGSF1iiSZRIpLLhBSPyiKvRWJCDG911gvEwkNzaikJcbk4CSRGIo4kQo6iiGOVkIYCgERKiLHvwJAEJCVRxgnX9hSaovlhIRKTMaMeooIqSP8mOKHOR6YEcyAiWwP1F4hMElBMVkfW2sRc5JDZkb2RGgYXcHWXBrdHLuaQl0uWl4lGWKJWLJmKMryUj5GM0tGaE4PNloWxii34X02PC5tKHyN+0R/mn669I1GLG/+8efxr/+Xfyu767FgfaMcKuWykDy7u51thUZbxChwSm47TNqxMuPMlafPwtcRT5oOR1tBiThjl2Xh9XU1YkyL2dFprKsRp5Ha0hqbiTQAZNITrTyMDDj0/z97b9JryZJuCS3r3H03p4kTEffeyLzZvJ6iCkgQNWVGFWKCGDGrQkj8AWpS4xqUEAgk+hk1Ysi/gJIKxEPwyKrKIsn3sr19RJxzduPu1jH4vs/c3I+fiLg38+nmfUqTjs7e7ubmZua297b1NWvRpleEpQO3I2BILL4hahidi8i2lMHbQiyyqTbysrmwlvNCR4fWBWidMHhXaPq3rS+yDGOwaGyAaCzKsdoq3VgivaGQGbKGe28KM66QSwAom9zGxkKaoRSK6HVNjhGjhjXT8ZQVht6h7TzGkTZuWmcMA4mhd53n+7EkAP8nBl6UEBrXEHiqQ2dEIDpnBecitE443JEm2mbPP5zBlB8rayfCmCwb5KxgbGQCj0lbkq6jRIswGiiTC/OuVpOYdGTBbgDlRyNnheg1ctQwQi6hMrROiN5AaSCOfC5TCI+w/uasitahbWhznbyGcexpjwrpZIEmwbSxsOvKPQo7rsg3aFB75QcSQEb5YS46h8CU4yk/vr2F7sJExsGbAmH6LT/eLIotpB6q+sEtIUuc66mbiDQYKEs/ikL2UfoddPmRLhqOvQFsJgbhDGqbf1xVx/PLzMSFMMJUP6gs5K0aWgeFtCOriXFX5XJcnSz94O3DFMoUFOWBAnRdAuWnJoVcE4gIUYaIXAPAqIEucT5qnIg1ADpm+LmoTIy/vC7qvks+7DQ2HvbRIG0jHQt6Iv3IoDzHbSwbFljOI3WJyULSPBdGwsMMk4kImYfLU66sQtkIqEDzlznnU9h/ATxgDVZeUf5mk8g7wjmtAJCbVPpV8o943QhBiTlQzmZ2PLaR+po2mZ5NNS+610jbBAQFMyhEzhGNXSp915yXWRhlLeWczZhxPYNuS3mSwrIr+WTCvqk9kA1gBmYC5zFkmyd2UQC6IoGpmZS13Ad0X+2Z7ZTbKMzAJkN7hdhm2BO1lWqmVaDknyXDuYgW0CwaqXltZI2S26ZHak9FOo9E+Z86Ut5nZhKb7LidrDg/lPudQXmvAYUJVgUiYUnNRPKiIl+Xac6kfnKA6TmflklnVFAl7076GdtM9Th3L2vqT9h+9X2PGWjsAKADb/YzPa/UTgzKesTEoCycPQM/f8P5fHxO+kXPGFObVTcL514FFkyPwnBb1tQKu2uq8h0LW3PFAqwy56vW+Yn8VVezwuqAsoYLm6xBYSiWvEjJmSztcB5pvUZlzqCmuSjM03wPKSUvME/zVxh509RnGdPMa8xzVb6vFvmXxeunp3mR/Mgy9zxPVTBbdbK6p1wj30ecp5m1eF35t4lZY2fsscv2pN4ibGXVGb/o+2OhP0s23WV5zNG/CLKrLliMuz72hvJGb+Zj5R3u8S5e3P/jH/29r92rt//jD/IP/vu/8xtv93/9t//zr31swDfcY6mQ8aQ549W4wZhsyYnqo8O9b7FzBPh2bsQQLC7bHkdPgO+6O2OIFiHpGagU0HjZDGi35BG7H9uSHyPepZQVLlqyjh7GBk7yYdoBfbDYt+Tt6IOFY4um5EVYnTBEgz5Y3OxOJc+nsxlakQW30REH3xQgum9HeM4tk6JVxrYdC+A9e1c8LT4afOvJbcmbef/qHmdmrbxmSzQAnEYHZyNiouvEMi5Abd8Ns1CqMRh0NsK05CXMfL8QTTlWg+HWxkIEZByF7sWksetGDMHgYttj8BatI+/Xph0xeAetE64vT+i9xdX+TFbppHG172FNxHGg5xWzKmC3sbFYeiXMsPeWLMtNKB5jIUpSICbIGC3ef34LALg7dXAuYttRXpPCJE1DeSUJl3vaAYhnLCUFa+m8XAcAV2yd9mwFFq3BEAkYCyskgXhil2z2lLNzZut0zgSKu4seKWk0F2T9rXNgBFjXQN/aALuNJeelaRO2T47kPfQWuvUzuZ06t6bWGazfA4DdTXk3NdC225Hkc7xBc9GTx2XLoJYBv2YLtfcGdheLcSIGZm3MCrqlHUoxAgQNt4vw3qDbjoVxMycN145IDETNxpdnIaCuuRgQAhkJAEB3ofRhHGwZu2pCyfeJovOXFcyew669KXk+cj9tElKkgGazJSNHd9VDqYzzqS2/aFonJrMCAUAA1nFO0zh9/Qo5lhh0MoDmOpBx5ZLmM6eJHbS56tnLB5o3k4hAigG8vaA1GkYDs/UQCSW5vnt2Qn9ukKKC2YYyHjHGDH1T1kAMGvZiQPQG5oKefx3uqC0FdKeoYOxk6BG5o2JUYyu8sYnYRz0ZXlAZRbRID2kgBQV9Sd+dCJo3hQrmioi3ktfQV4kNGKBNsljOGajb9zyCN4iBGYot7RhT79iwAZZyAvTzgHh2NB6bgMFCPQ3ASIaknAE8XbCJMmFW5s9ejhNjaGaQpW1CHs0ke8NrNDH7aWoqnQlFBpMcJ91QuFTJGdFuKScFiMGDx5BdQvI0TzmwpqXiftiENBjED8LEzFoZByTHLEeNzIYizQaRiWEWSKOBsgnaJfLeMFus0pm8QpwfV/Qy2ZMCgI+RsUipjDSYSg5JFS8RGZ3YeKXzRDLGxgta2Aq+SZPXRIxKzJBaPDcDk42Jpyaxoehcs7B8yeIyGX2AufdHgYw4YhyyeTK6yPNlI9fMSJX4vEho1cBA/suarksGj62gkMelpOTaWuoqT/MGgJ6VGLjARgxhuM1kTCmGDQE/ajIGFcNOBhOB8f3ZW1aMTb7qn5CA1TJOcjpW0iOVzJSKagLGfCzzeIqBZzkFC5As01WDd5GrEnbVB8iSvXQzsFKDSmD+vOQ+Ok8EPFInZZScwfoaOV+Gmx/iJwFOVdXaSLQy/Kn9R0+ujEVu9yYQ+GUA4peouwYM3xpe+7vyW1O+0R7LZ3/tWf5b/+jfLyQnKasZeUxg6Y1a6kHy5NYExH2qdC65jlJ5NbcOQMm9qiUIAMy8mmv5hlJE8L4uddtG04atrrPMtatJNKQfMjbPXk7yyupCMiA5XtTXqb21/JdaUH4uWj4Jx9di8XVby9ypKLIHapJhiEk/yOdagvB67nPV9lqRvkmpBczXxinnRUZAiApqL+1SEF68k2t9qNuWekJGQCCtFpJfXju/ZuojiodZrn8sD22ZryV1qW2ex2UIypu+vRdrYrYGgEImUb/PoHW61se13676DuU490Mt7mCt1jwAACAASURBVPloX+t+igeYNxG1OL1sxFVleS5fgXy8fl93sCZ1KJqsdf8YdBaReqyv1RIGVZMoLDcmwFzbVbx+9bmy+XxIOFHLOOS1tcYb72lOVJk3eoDVvL7NDFxfMyOdwPomWTbSAgIf2+hIOxKaJaUOv1qeW36HiaC7eCxkrMsNsrRbj+Gx3Y30qx7Tss9yTu6hV66R9VkBBhKinzf34PzK8eKJqdh6y/n63FIuou4jMH9msoYqj1XZXCcsJB9Wxri26ZbPX8ZcxxTV8cq7VKQqVuqo5VjyNCcFIFT/S65exFxD9kuW0h4wtcnjKvqdAmLKRXItindawAGq5YZp6PNSAYfq0Nw7Jc95eXHVYHluafH1L/Neg5q178FUvV48Vxk3lmPnz1BWFXjL1Tlg5nFc/Wmv142a33PWx7Xr1uYDb7jPCriS+suvl98E6HonsPSuddZ+XN/w/k2evTfd9zfa5y9Z3trnt5Q/+6++/hzL/R9/kP+1/+7v/sbb/cd/6z/72scGfMM9llYnvOhuMSRisQSIsIW0Fwmg1KyhhsEjAZZYGDGBiUSlJukQgpWa9XOpRyisnULoo1Vm3cVJX7AmlqlBr/S3vj+Acm0NjJegVt7XbKXCkCn6eZKDWJeYdcmZNDrBp6kfQiyzLDVxT5lXvmfKrOcINQOFUqfW8it9EK/MAnzGBXivWQsBlHvWr9eImJZAdfl9K/mS9Xy2W/IinTm/pa4zAzZAyZddI+gQsFzXk3HJmMQgMY1bzZgFZY7qdmv2UC9eOi3EMhMRTB01I+Quch+pL/dbGg3q8hj5SK1/VxsZauOBNZNmX/0c6vmp18nSuKIwkeTk6nnJ/zVSEq1yec4yt0azZ6v28rOxJvO4pO/LsYjhJC/uVxsEliQmEtocKwZAMQpM7/NM37A+/nCuc/EOpzQ3mEiu5Lxv83FiMfbaWCLebhrn3PhiTEII07kYVQn3Lt6r+nktAXZlRFky2cucPDaXs8K7B115igEGy3k6j0fWBIAiQ5R5HhQbVhJHIwAVptIJKYp3OpV8s3oOVbXOlnM+A/gLQ4eA+zUjxgMGSsmPUvN63FQ1D9XcSb0S5rZi+EisPZkxhZ4vP+e5MkZUXqICyIUFkgFmASfcfgnnlfEujTHyWgwxlWFIaZrbnKbnOntANUiQ3LNc9U0MCMAcHNcGjAyqE+Zr+EsVMT5IW/VOV/LRKpA/81guDRo1wKoZRYF1g8ZjfZGyhn6AaT6WgD/Pj9f6ltScevD6UZIeuQ8wNzJIHZmvZVjlcuxyv9pItfiwrmlKyvAf1F/Oc6lI72fgdDk/s8Yf9v0NGP6Nx94KqL4kaHqb7e+d2n1k/G9s87F1udK/L1X+sut/7eV3Opa/tSUkjWNs8flAVPx7NyAkyrU8hgYXrkfIBp3xuBu7IluQs8JF0+MUmgLA7scWG+txDg5WJ7QmYGtHjNEWAh6hxZc8zNFSwvrt0BVq99ZQW0IhL7mcAvwAAsQ+GgxMpCNSACJEPyqSj7hjIh2h4ffRFLFzAIVEQsDW3dAhA2hYa/O5JTmIk3e4avtCy99ZAoNDtLgfiChHhND7YMumXOZGAJyE7DomiRlFIsBEDNE8AAAhGjQ24Dy6Qo5C4cSUdzl4Imw4M4nLGIgynkhEUiG4oRBdixA1tu0IxxT0Qiyj1SSsbnTiUFiao0NPxA812CHSFwOtKFQ2Ro0X13cAgONA5EgUmkv1x2AL0FEqY8u5rEJ4IwA6A4VsgeoRMJHxxKTROpmPgDHYsjkVILBrR87f5VzbrApRTYgG2zaW/uSciri9ACbDBDZaZ7TOF8KHxkZctERaNHhbyH6UyoVcpX7uAkJqoAwArhqv5PsO3sLZyIL19Gx7JgUBCGwKKNEqlzxaCTH23pRNt5UwwkJQYaFcwDhabFpfQn8p/DgWcOZYOzQmVY41Hc3xOFL7QopjdMKZySxSItAkoFBIL1JScKy76DkU1tpUPNvWxpmXOwRdiIf6c1N+Sa1NReNUQJI1TKoixwEoRlgx6LL5dtZj6B26xuM4NiVvGAAaF/i5EPh0LmEcSG82J42mnQg+miYQlb/o0QLYbTzujx1HG8SZF7x1HuOwYWAZkaKBtR4xGn5fcV9nxfqw1I+mCQiB5jQGA+MiIufjpkRAwtpIwDY4KAHIicOKGfBq/lwYrhuCKcecDciZQpStCwhBlw2OzI+A1rYJGEaLwGmS1tFYzyfKoQbo3lCsyesNlEF57bopLDgnBddKuD3dK3iUcWmVkZSaGRjknuNgy3ViJNBqyvOugXUMpoQNAyg523U4c04EmgGUaIS29Rj6hr5TIkr+MSJgmwh/dmjaESmpaT4zybRIKHQMmp5ZMCU/XeY/ZyAMFsYluIY+k1plCm1WmULGGfRLmHMaDUwTZ+ctf1Z9b6E5Z1q881plhNFQqG3Q0DYVwpScUYhX0kg6w0lCkROgbILh74/I+eWhtzBdoLBpS3IRrg0Yjw2+aqE8ctbbFY1fECBPvaEc8kh54TlwuLpEorSx5Jprx6+TAqKic1nNohGUmcKgZ5IVAJAAsw2IAxuPo6IwX8kZZ8ComJgGAFQT56HGSVF+uAKy18gRUwSA5NIzGM5RAS6T8aIG8qJ1nPl1VkAbp5BcTX1DVBTuK2HEChNqqXPXxdAh4F3akJJRdH8BTDncYmCpw3kFCEt+pISgFjCdp8gBziUuob11yaqEEa8CLcGptRefpwKadYplzFX9VRBcAViVMXns5ZrHwJ94oiV0eWGXUHka42pZIlP+X0KDMcPjPE7pwEofF3PzwHO6PP+GssT5yynDI+9/V76e8o0OhX3/X77J/8H/9Lc5FNYUnUIBFUMyxUsJACFr8q5lVWQo1kJlJaRW5DQEYGo15UpSe5O3T7ySYzJodMRYhdUC6yGemsFZvaEHULx/Tk+B+TkrJCiYKsZEjhW22eINpXbEM2h1whBsYa+bAQXWs5LjAsrkdWQvqABjAaECIAHSSXM6FS05AMXTtOZttTrBJ10AqAJKe/X3pYD4Zb/EK7bmbROAJKX2lEqp80AlNFdAohVGusqrV4cQ0/XzTWwtC1H3SXI0xUtZhwAHHn/tLZC1If2SUnv1cjWfy9+kXM2vePYE8ObMm0mdqnDfaZ6W4ZrLXMt6/LWcgni1JFQXQAHKtRepPJ+KaGkCVfnBsyyeyKqubKprT2I9fw9Df6X+5JETcifJqaQ+PfTeyHHJX1xrv/a2KYUCuCx7yR54bGSNMIh6NCyZX6ekodkg8kAaJ03eHiXrolpnYkQRD1upJ9fzpl2unclG5AqU8A6i1oGTvMi6lOslbLJ47qqbl8rzeSllxUuQs5qFu9Yhx3UobFk7i8GWvL6KrRGo3tfPUUKPMyZyJvlfJp7vUe8sa89U2eCquRdG+rq225N8x3oe6nWxFk4MTBt3xZvl2tNXz720YzJt/Nd2XmsbXFkLefJalXy7Oq+uDrktG+w8ebjq8FQZTz1vcu0yzFL+rxCv0I+GmsYOzPLfssI87LNyjqoIJPfV9z20OUfZdNcb50LkJH2txwJMIZ8K8xDj6tyDTbpcsHymAobK2piDAEyHZ/d4QPAia1cvANPyf3XuwXJZfo1V7c/CmZcXYl6/npdp7FPdeizyXryOy3GX83iImR6AmXq9La6d1XlTeeT8bP7ftY0vCbzeVt6pDyvH31p35Tv7bW0+Vt46N79m+1J++F/8NoTCvsh/47/5D3/j7f6Tf+c//drHBnzDPZaNjnjP3cNnQ3+WPC+OQ0mHRMPz2cCpiIHp00TvUYBRzApOJfgs+YoEpiT806qEYCmktgaWPmukrNHqMNOQlLZq/cU1XUCtUhGOl7DdpXZhDRYlPLcOO5XwVwCzcF2AxOI1MqyOCNYUYCzHExSsiqWfazqc9b0f0zOscyCX2ol1+OpajqlcuwyjTZnE4p2ORfdRQnVD0mhNKMC6LsvQ4TWtyvq8hPEK+ZIIdi/DdeuyzAtdtr98XdetPbpmMZeZ25aQ2dl1VX+WGplLjco6VLeuKyzHJcy0Gtua1udyPNQPNQPSWqGE89L9MpahtnUer5yLqb43SjhvDdCBOegUgLkM8ZPXy2dVA3upU4dG1/m+y7mswbw8p7W9t7wGptBWa9IE4CpvdOnDSh7tMiSXrpVQ1Ek6oMxjnOc1C8Avz22W34sHJRgzI2Cqw1LJizvNmdap9EHqLQF4bWCoDQ31M6nbr0H6chxSxJggYZKlTqK+ScgmVK5A8RygR2YJnj0rHpN46+px6Cock0Jw1QSy+Xj9X+n63iigWvovYyp9leMCvgUo198nCrN1MMv5rIDKLM8WE1gv4bW1wSARO3PWFSivd9uyoJUAYBQQNwuTZ0KXbBioJoXMzJdLoJhzBtgYQffkYzKemgF5CcgfA8nSl2UIKD/dnBgIy7pnb1muPgcqqV8rxzKnPOWycqMSHppLiPAix7VcW88PZufrNcTDqYf2EFgCUAYLYMnvKySxBJa09jB5AStgWW67vDf3s86dLPUeHYvCA0RTI9dcGQJm6wYPS7X0q9ty3Tzr4xo4fABG18ao5v8fAssV5Looa9uCXLX51lX3lgrvBL7qNVT3AXjj/C7vMzMy1HXeBibfoazO05sOvOM9vxI4/V35jZdvNLAck8GrsMXLcQefNTbGIySDc3Q4hQZP2hPO0aHRAafQ4MINuB1JQ2nvBvTRYYwWnaVQ2a0dcQoUCtkZj854jMnii36HC9dj5NxN8Wpu7YiUFT477+E4FLbRAfe+w4XrkbLGMTRoOfRtxgobLMZk8KQ9IUCX+xLITGhMxOthU3IeW0OhsEtPqNMUZtvogC/6Xak7RIsn7Qmn0OB27HDV9Lj3HXw06CzphYWk8XLYFg291gScg5uNsbUBRqXSD/F8NjoiZPIedtYXVt0aKIq25GvWcRTPna/CbrfOow8WRpF8S2cDzt7Bmoh9M+J1v8HGeYzREJtsM6LRU5t+AUydgFien/uhRc4KjQ0FtG2dx4nDSXtvESJpmwHAZ8cdnKH5L7Iy3s7Az4Y1S3tuQ4iRMs/XyNdtHGln9jxnIRq0PL7GBhxHV8J4ySNLbTsTcT+0iKBNXeBQWJrr6ToKPTUFrIlX95415DaNx2mgkC9rEm62Z5y8w6FvS2gyQOBxlLBOnTFGXTyrdZ4mQJpkI4e5ipddWH21SjiPDvtuwKFvSVeTrxcPreMQ6KYKGT57W7xtYoSwJiFlCkPuOHx4245FXiclxRI9HAprSDamDuvddwMGbzF4WzydVidYQ/2UkF9j0uR957w6z2HcmedmGRas+XUdknuxIQbl14dtARXWEjMvQGBB64yu8dAKOA2ugBIBWeJVBihk+3husduQ3mEQdlkA225A4siBlDQaO7EFU3hlgALQD66ECUuYb84KTy5OuDuRhqtzkZmh6XlsGo/7UwcJTR5Hg7al0MemCfDeFI+o1FGKwiWbJnBIbMQ4WjgXC/CMDISco7U3sIyQhDYDVSgs53Y6F4sOnwDNhpmDx8Giaak/AjCJlVaV99dXR9I29YZ1Likssz83sBxKmhjY7bYepxNJNDVNwPnUYnfRoz83ZLgA0LKOogB97w2x6VZ5hG4T+FnqEoI9Dg6moeM50TOzbUCKGu2un0UByPyKkc9xKLNcC/5ecNtQXseoaM4HYgYWKSbqk6Zw3N5i++TMobBzQ4HMd4oSXkzPUsYqOp5+tLAuwnF4rzbEZi0h7WLIkFBbCa2leeaQcg799YMtckninVeKQpwpr5ijZpjRV3JEASB5nr+kSzgshcJySDzLOYXRwDaxMBTL+hl+zVBYCT8VLUWAwHwciDUXkVh1k7CkyvqehcKmIumUk4Jhlt9cebBrr3/i9SQlJwXTBUQOc81RQzURCPphPqsw67pUGIMzszsrST/wek6GVXvss5q0EpOackjVog6HfKomFYOCMhmZJYZmjLpiKJF7ZUw5qhlTdAAwGT1oQKsyRnSODQcLhFHCY6sQ4RImK8eELbeeg6pdGFBI7QrALICdDTY1ngaH15am1FR/1ZCwhqDV4rzMfyEAw4z0as3Dq2RO63lfjrF0sBr/GwD1OwO52miyAnzXgP3sPotDa+/rZn7bS8ZDo/1fpfKNBpbn6PDD2xc4jC3nDhJ4OHmHEDU+a3YlnFI0Gs+8+SKxZIvIG9TeW7QsDi9ixxKGeehbvHSbsmmt8+mUykUIG6DNPmlGkjC95LOtefJCMLjrKAdQ5C/EiyKbXwlTs6yFGBd5FtZGKL7vWUAE53/ddR3GYOCDwW3TFUF3uSYDGAZXQiOXpB1St86/8vyjL4QiKWk4F+BZfL32VAjxiB8tDLeTkkKKGsaSHMFdE4pMQmRhdNmUyDnLGpkpUn1jEobezTwWNdGHeFRyVgjekDGWc88A4N7FIitBIsyqfEmdjh2UykT6wRvOFKscHwXcSU5ZkOPTF7HhXCVg2hBGkRXhORatS5GhKOylCjg0rDU6iDgZ6VUeHeX1Whs5Z28KEy3hfGrKUVIm42AjIgtzK0MSJuNoSv5T7aWUfJYiGs0bE+mfbHplc6bY85mzQgwaR8ted29w7h3CSJtGADM9TG1oo6gti1WrPNswCaOqEJMkr3FmGYW+cUVagjwwmT0mKNIItOmhxobekX6nJ9E+pUmvU+uMOJpiVRedTQCT9ygrnFmOQq6XPst8ipanbNJGBj/+6Ir1XxnS4Sy/djqjbygHs+RGAbNwTPEmDW2DeDYI3lLdKkTMj7Y8lxwVbWJ7U35xh4blagYD79IU3snXx6DhTw5ICqNNE/usAvqmQTiTJudoM/KoEVqLPBp452iTKD+KGfCcJ4eg6byElHqNUXKn5P4ARsf3G3XxepVNpLzXtGH1NtMcjqZsYn3DZqNRwzeWJB54R+HFE8dzdRsNiZPzMxhdIi/auZK64HkJo0U+WUSbMDYO+WRx8Jr0UHm854aNBLJeZMMsX8tRwfPcS9jpaBMwGIQirUHrLhoLRIUg60DmNKiZNy44O72vyHkC90XmNjSJ5pQ3yVE2x0khuAQMGqdIoAOh2kQrTGGySSE6A3g9jVVASALgNUZraU0N/Pw0pjw+AUJBI/LzSs7O5iOIruugkaqQ66gtXe81UuVRVUEX71hk7VblFfxAsZtKdDRtRrJ8L6+QWH/VO0P94I13Pxio01eXG4nOQI30PJKtNukKUCPfNypEN2mJiqcvDobeB4XIeYkqEctubPh99TlNFbAs4IQ/4yopBK+gJH8yKqRGkx5ovcNW07WiRQuTWZqDQaEC6YvWwEvPGXVVArIlDd4CWDh/MMtYGFjmRk/91aRvKiHIepzGUb5G+ONdZERkzmpUJONJaqYpKf2QOZkRFckUpAkvFUAi8yjvEzhsBvSFXO/7p+U4QzQPnLCan/cMWKoiN7L0HL81DOYxYLkEYBkzoDlrox5rWoyrLrWHWU3v35xjWbVfzdVqKO8aElwef6TM5nPxfomBf+e1/PrLNxpYWp1w2fTFeyYb5a0bMUSL1oSJ/dRRWGVnAxPOBAzsqXMmojGOdCwtAVEJQ02gDXXL3jEBfwJkS8gqM1CSpmRGw200NhYvTP0ZDEkjNeQ9o7BPqi/tk7doYhAVkFuHPCquB4C1Mql9oxOijdg1I6wxGI3BxgX28Cg42bwrYj8Vb5HRGdHOmTFtFS6nVIY3k0dF+mENeYKMzkzIInmKlAtp+HwJu8sUohkchckaw7Iq/KysTdA6wbEFu7GB5os9MxJaKQC/no/pu5zO+GqsUs/ohGjYGs4ejpYt3YH1HcU7pVRGCHk2B+LpiGZi3ixzyOAamEIiA1v0JcQwaNag5E2VhOABZBTQKiOxZ0XCFMWzI/deA5YlpJABNoU68nzoTJ5p3vMakydtTWDmMalz7qTfy/eTxqWC0pPXQkhQ5J4yPvnGN+yh0CYhaepf1FPepJDL1GGSxkZEZQjss4EgYxF2XXlnZE1YFxG1KiBV2D4l3JEuUAWQQ2Ukk8vvtda0G5HrtU6TTInKgJlCOpPJTGIDxDaWH06tM20OgWI8oHYy0Dz8pcwG07UmIjUK2kTkBjOJE20istLIoH4olYkwg9e4tuKNxsQEaqbrlc5ELhI16QrqjKw5zFWnoj0oAErbjIRIYJ43T2X+dObrM9XTigwQKnM+JhtfDA1M7pcyiu7hWhhk1kzoYujeAJA16ytm3gfaRLhVQmIlBLSEyFIfMljnVbwzLk/gUNaHzrTx5vBZeZ1d5g/IZPiAkr7ryXMD0CZSwoh55yttlfBT2QDaxPVlt5un64rOHRsx6mvpIc9zd2Vzbaqdsnh0UqbXTDSjgGmdyYZdZfLIVO0U9l0Za92WyjNQCcXHpZ+i71jv9LTcE9P7ajx0Pab/HHJaiFQyiidp+pzkaizcluI1qvhaheoZ8e/nkpzlyxT+7JcxLI9r0OdS5QkkqeozU/rDw9IZJSibAd3UJuavVX28AgHA5GVbAoz6OUs9VbWx6EeuriteQdDnUcaZUT3XZR+52SUaKCBSL+ovgMG8rekeZSkp+jGa+pln16t6nqruQGEeslv1NQOTN3jl+lmpr6vqPeqAWltqa3XfdM+1NpbH3rSkv+py/zJetfzI67fVXSlLIFkfX3v9pmO/lSVXa/GvYPlGA8utGfHX9h9jawYMyeE2bJCywhN3glMRH41XcCriHBtc2jO+8DtcuzOGZHHvO2yMh9URx9Di2p1wFzq0OiBkg0NocPAtGhPxgye/wCfDBfZ2hEbGOTpolXEMDcZk8MH1PY6RGFfHZHHTnPD5sINWGRd2wDm6mbdyTMRUuzEevzxdw6qEznqEpEu47RAtvnPxuuRYnkKDzvpCPiTlENpS/4PdXam7tSM+Pl7iqu2x3w/4ot/hZnOCVRGn0BRCo+9dvqIwXBVx9C12bmCPqoHVEQffwidTQPbGUliqaH46HXHyDbZbAvNWpQLyJST3whED7xAtGk2e4HNw2LmxsPH6ZLCxHkdPIcvn4HA3tnh/f8Dt0KGzAa0JuB3I8/rtq1v0gY0B0ZRcQh/NTJLi2f5IEjRxIi+6H1p0DFb3zQCrE3766gkA4NtPbuEjMek2JiIDMwNCygp3fQutgLYbkEGgXkKAJYw3Z1UYZne7M0I0sIY84pebHr236LbUrpAW+WhwHh2G0WK3GQCgGCnu+xa28TgPDS53PYWXRoOm9SXHMbHh4cn+hDFY9N7iycUJAHnO748d2tbjat9jDBRabMwE4oVwqDYSSC6ogNAxGDSO2Wg5ZHHbcYhq0thtRpz6BrvNCB/JK+hcLmHS42ix2YwYvUXXeaSksd/2JezNM5kVgXNq+zw47Hc9Tn2Dtg3FICIhuQDggyHg7nwxVBw5rHHT9QCAEClKIASDjkMaxYAQAoVjUp/YiDJaQGVstgOFAXqDthuprTDlKIZg0LQR50OLnBT21+cSuhiCxpZDI0s44WCRk0K79eXHJUUN6ATbToaJsXfY7AacDy2ajS/hgQAwHBsWmifAHL1Gs/GTAWgkj2O79SUiwDWheOHH2xbuYoRuA4K3Rfg+RYV4bGH3voQhNtsR49nBtgFhtMQCqicXgHj9bRMRBgtlE+KoibVzNMUjLGA3DORBMpuIxF4sYgdlJlAOITQthRzGwUBvAoX+tRHxTJ5Z3RI7p2oSjIQV8vh0QyF78YsWuUvEhpkU8pG8f2oXJqZMS3XzbQN14cnrd2yg9h75tgE2kQCZAjJ7chUbW+ASe1nZa2oz3QP0GkkBwQBtIq8pMAHAnr2mR7b2yNe64/PC3XY2FEmg8hQmqAAcaawKQLYZ+mwRN6nkEKqRvfsmQx800jZCf9rQxt5V7JfsAco2Uzv3DrnJ0LfkqdWBgFzWGWmToQ4KZlCIHXmrVAR0BGLL7JeKPFMqAckB5pZ+a1JDx/RIICtuM5Rn75fhthK1oz31TUWF1OYyZuVVacveEolONgTgtFfQI3+eGkCPQOwAMwDJUh+zAczJwl989V2dGamPAKCFgRWAinQ/HQg86ZEBMCYwZQb6zCRH85C1gGfA9ASWCilSxkQUxM+4hBSCrnEHeg5klMowg0KymGlFqkheVOq7Iq+hJyNWVtRfZJrTbFC8aypoZMPjUXQ/PQKF5CehPGM9gp87L/FxWs8qAckA2dLx1ADikRRPog58vZAfyZypeV2VuS3up+LPYskHNtU11VyV5y/ERdW5yZJIbWYDIE39kDoqTM9z6TEroLf2lsq6SNN1xZu5ErYqbUHOy33TvK5cP5NcUSje3pr8aOkhpfl9RGKlTAYmg4V87iods6wUvVf8WnKsV8q7AMQHZWVeHrxftPNN9FQu+UH+KpVvNCvs/o8/yP/Kf/t3AUykJzXRgLB31p6YmpiDDLBUpyZLqBkx6T24nfn9J8INzEJAl3VqZss1Rkm5x7KNZXuPkVzUifJzrbS1bz7MNNzqdoXMYsbOuZjz8vmWOZX71J/wtaIearktdd0efM6WXzCPfLlM8/DQTFk8IqsXLNoUj0k1n1Nd9fi35KwP1bjkOS5DdqXUz6WYc6vG6lvVoTMaj8/D2jyWNlbGVPelHufy/Np96lL9CM003N50fWX5LW0s214+n8d+Pdbmdda/R+b+beVN81nXqf+/a32pu/bMMyAkJKs5Q3muLweg5AwV4pC3WcDr+V6+f9Nnce3cY/XWrnvsea99vpffYZi/XXusclnZ6NUeEhnrkpF0efyR754vtXl57LO8nJO0MoDlul8pD/pSzXE9bXKubDqX7a/1W+Z22fV6HtY+r3J8+X2Alfrv8N3+po/tm9hD55vilTbmGSVfriz6+djPhKq/b1cW62Nr99F7rS2DtTlYey6Ley6dyXLd6ldsPadroZbVmniw7upm1CPP7B3KAyC0uPbRz+XKeniTZ+3BvOCR695232Uf3rJ3Wa3/SB/re791/bzpuKZ10gAAIABJREFUHm8ob12Lv2b76/f8Chd9hUv+r//h733tzKm7P3qR/6X/+j/6jbf7p//uP/zaxwZ8wz2WTke82N2hj27GTOqTKZ4XOe6ZOMVznqR4KoSoRuQvhAzGqIk9s5bGqEGqYe1AIWuRkNUxWFgm8wnsTZNSh7LGpNC6UMJGlxIYlNMJPjZnzCxhmWaSGAic2yfAmAg51EzioTDScuhhqECmyGAA0/egqb5hhBkTeMiUKRqP9bWJQ1tlbBOwrqUqJi+MhIoK4+WaQLx4iZa5pjXIn45hLrtQ6kyvRZ+tZUIQ0VRcsnPWobSih5gqw4WQVtQGAsldlfFM7KbrQvPApOMo4yt5gNy+4RzFmRFh8Wsu8hQCaHNWhWRDclyJmXLa9CyNGWV/uDAy6Oq9eK0eMB6LIUWeA99DQk6LkUXOV/MguaKyjiSfU1g6xTBDRgMUlsWlWDwAaMP1F3WgJqbKwkqqMOW7ls7M+zdj9pTX1aZB8jyTn9YmSXTMySokd1WIPep71eBfmQwEzWQgar4pNdVuOfN9WFqDHhTdK0f1IDwUoL7mmuSj2lEpnTmXdS4xkqtnUUrGJNGR1QMjyqy+jLXk7s7bKfNQA5KaIKSMjasLYYXkYqlcgA9kDbs05YTy/ENhyjEsc46S1zl7LQQm0sdlCGW9BqSOPP6l96Lqm8wDkpq3mRd16npLVFCHTEo7aXGtFJnD4vVaB6urhoB6B7tkE63ryDxmFGOISpMmYC07AqDkEhYSFWBuFKnXQX2/em1In2Q+VHVMYcrXrcdlUEhmvlLRExlLrsevUOQ/CLxyKH0GxIJaPInLNVOvm3qJ1YBkbRMt6x8TYJ552qTIV33t0Vr77qnLYr2tAcclXn4gmVLfX8Y4RZNPQ5J1g2pqHgMNGQ8kZJbjXLvmsaLSfBBvsks+AJzvUuo1vDz1hjl9p7L8vNb/H6v/WDe/wjVfprx9TCu/B3+p9/v6Cn0l/BrfQb/l5RsNLJ+4E/7OB/8YPxpeIGWNF+4VfLb4yF/j0/ECf7L9GLdhixt7wC/GG3zYvMTPhqcAgO93n+Oj8RpDsnjijvjz83N8r/sCH41XaHXAM3ePb7nXeB23+NO77+H3tp/jPnY4R0fyIlnh2+1rRCj8+PQert0ZIWm819zjx6f38IfbT+GzwcfDFXZ2QKc9IsuJtDrgC7/Da7/BDy5+gQiFT8dL7A3Vcyriypzwz88vMCSLlDWeNgccQouXfoeNoTC+mBWeuiNaHXBjj/jR6QMkKNy4I177Lf7V/c/x0XiNT8ZL/PXdL/GT83PchQ2+1b3GVo/ok8OPDu/jpiH23JvmiM/HPWJW2DAj7o07Ym8H9InCeT8ZLnDtztibAT4bnGOD5809PhsvcGnPOCUiEHIq4uVIbLq/Ol/hyvXY2QFjsrjzHW6aIz4dLvCdzSt81F/h0vX4tN/jve6Aj88X2FqPP9p9in92+ADf277Ep8MFztHhe9uXeObu8X/efhc7O+IcHTbGY0gGG+PR6gCnprzWn5+eIGRdQpIB4EV3i9d+C60yPukvMASLv/3+PwUA/C8v/wBb6/F+e4eXfodGB3xyvkRjQjFWfGtzC4OET4ZLOB1x71t0JiBkjQ+6O7wctzAq4/ubL3CILT4ZLrAxHve+w7P2gI/7S7zXHvDpsIdVCX20zOwb8f3tF7gwPf7F8X0kkOTKne/w/d0XuA8dXrS3+MnpGRodcGkH3IWWwqRNQKspXPhnxyfYuwEfdHf42fEJrZ/2iB9c/Bx/0T/Fnx+f4qrpsTG+hHbfhxZWpRK+LKHXG+Nxrgw3z9ojXo8b7OyIjRkRssHH5ws8bU/YmBG/Ol/hD/af4/+9f44nzRkAcIwN+uDQmID32gN+db7E0/aEW9/BqoTX46Zov142PTQydnZEyBqf9Xt8d/cKf3G4we/vv8DH/QVCMuijxWXTF6PSs+4ApxJufYeen/OfXHyCT4ZLfN5TWPreDdjZEVfujJ8dn2BMFiFpbO2IzgRYHfF63MKqiJANnrcHJCh8cr6gfG7X4+VA6+bS9bjzHULSuG7PeNVv8YObX8CpiH/yxfeLBu1l0+OzMz1npSis+bvbV2i1x4/u3y8h5p2hUPijb4vkze/vP8efvfoW/vr1R/jx/fMSag4Af3L1CY6hhc/EKP1ed8D/d/esGHK+tbsFAPzF/Q2edkekrHDvO7SGvrv+5s1P8b+//B7OweFpd8SV68sa+P72C/zft99GSBqXTY+Pj5f47sUrfHK+wPPugJfDtoTSp6xw1fSwOuJ23OD9zT3ufIfr5oRPTpd40p0wRoMxWZwDPZfnmwMu7IBfnq5IIzhZDMEiQdFnOji0NuB+bGd1rU44B4cX2zukrPDR6RLf3t3is/O+sGXv3IiQNM7BweqEv3nzU/zsfFOYu19s7qBVwr+4fQ9XDYVI33sa9x9cfo5/+uoDXLdnfLh9jR++/gB/fPUp/uL+KcZEIdof7l8jZeonAHwxbHH0LeXp64Cjpz6nrHDnO2iV8bQ94hfH68IU3keLk2/wbHPA0bf47v4VGTCzhlMJnw173A38rKDw/uYefaQUCWEPH6LFh7vXxAqcNQ6+xYfb1/jZ6QkaHfBq2OLCDdAq4ehbvL+9w09un+HffP4zHEOLL4YtOhPKd8+FHXDrO9yOG7zY3uLzfo8XG0qtuGmOuAsdxmTxs8MTPN8c8L3tS/zo7n1cNmfs7YidGfDL83VhT/+s36MzHq+GLd7f3NNc9Tu0NuD97h5aJfz47jmumjOsTjj4FtfNGa0J+NXxClftuaR0fHbe0+8dpy5olfGq3+A7F69x8C3OwWEIFpdtX+71yfkCz7oDfnZ/gw/3r/Fy2BYm+N/ff44//fw7X3nf8cHuDr88XNHab3s0bETe2hG/uL/GZduX53sKDWLSOHr6Xfzw4jX64HA7dnjSnmjtR4uzd/jw4jVC0rgfiUBOvrPk++x+JKI/+W07e4c/uf4Uvzhew+iEw9jive097scOQ7TEFYFczqWs8N72Hp+f97hsewzRlnnTKuPleYuBSQ4BYO9GHH2DnRsxJoM+WDzpzhiiLaSITkccxhaXbY+UFW6HDjkrPNseMfCalZSXw9Di6faIL07EXi/fhzErXDYDQtY4jk1JUxmYXBEgNnIx9vfB4jy6wsRvTSqvx2A5RYLz/Hnv3ntbjP6SEiPGdWF/98HA2Vj+19v+lBU6F9D7acsshuE1Y3LNQ+BMxOBdyccHJrmopRwVQOkvYlgWY7vcA5jzHMh9ifgwzQzxcn3NxxC8gWbj/LIsDd/lfc16zZeJUVsIEUtqRF1qpIw5C6rCQ6P1DBcv0XVWM2P0akSZ3POBtWPej9+Vv/zyjQaWDhFPzQF/1H6CPjlcmxNexy0+bF7imb3HjT1gpwc4FfHd9gvs9IAXzS20Svz6NYbksNUDsAFuzBFORWiVihZmpzx+b/s5rswZV4aAk1MRPhu02sMg4TvdK2z1iAiFrR7xYfcKV+YEny10m+F0gAHpVvpsYJBhVMKl7eFURKsSnrkDjUlFtNqjzw5P7In1MTW2ZuBzAU7HiYpeRRjebD5v7kmTUUVcuxOG5Oi1pdd7OxD4UiQNYVTCi+4WrQ7Y2QEXhjZaAn59Ntga2kB1mkAIWrqn0xFIgLbUr40Z6YfQDOX53DRHaJXxXnso51sdoJGxMR7PGprvm4bA8XsdsNEjnncH7M2ABIVnzREGCRe2x86McCriFGkTstEEblodMCSS/qDnR3NjkHDdnBF5PFbH4oHdGI8EhaftEbFRuI8dEhSe8OZG6mhkPO8OZV60yjBsftzZASlrXLme5GB4s73nTecpNUgM0o3KZTNa/xedUcfP8BwdrS1uT2sCMSFTnuspNaVf9PwTbSCRS4j33g1odEBIBlvL8ggq46Pxip6ppespT5Z+ZDvuo7SztSOsniR0LLv7POfepqyK0aMx9P4YWjQ64j506HhTLNfDooBYqS/6rVs7MgHWtK5DpvzOLRsPtnaEzwTsRQNW+p00ba4HPt+BCLFe+y1tCAytd+n/MbTl2qBMpbOqqa8AGhWKDm6j6TshZI2OjToJCo0O0IqiI1ob8HKc5H4EMApRmOZNkUbGy3Fb6mg2vcuPrmwGAeDT/gKtCbgPXclZlrX92m/Qc+52ygqvx00hHQOAY2hKX1JWCJnIy+T8L85PiFxMR4RkcB9ajNEiqIRf9NflMzxGi9YGnIKDYSMIgAJwoehZIdHzPXjawB48bYL74BBYe7bOF5d+n0Izk08KHKMoRhwxHEh0hVZ8D9Bn8c535bVWGSPn6Mrn9SenZzj4ltqNwCf9BSx/t8tYUlYI0Pj4fEGfweDwq/MlAOCj8xXGCkSLYUFAo8hGJdAGdYgWL4ctRIdXq4xP0gV8NLjNGwBT2obIP/3ydFXatzrh6BuOHCHDx2fYl3mR6I+YNH51vCqbszEZhHyDo2+gVUNAnechJo2PTleIWeHH989p086AUvLTb9WGAAPXPXtH/UfGR+dL+ESRPifv8Em+wOthgz64AsqVojz2z/SevhuCBbBBSBq/SjS+IRpo33AfKdd9YMmmMRocxrYAB5kDqzcYgi155CNLlgze4uf315TrnXSRNBIANUaDu6HDaXT483SDMVi81PRcCOR8dbmRn6drnIYGGRSRJJt0oxNOQ1MA0cm7AowCM4v/JD6FSBgdhhYpo8gGyblYeQ6N3hc94CJ1xPcLSeOH6QOcR1e++4+jQ4jmwUZd0oROo6O1eO7KsdtzN63LpHBSDGRNixg1bnVXwNJxaB4QxsWocXvuSnROzsBpdAUcCQAKgTgEvDez63NWuLPT9RL9k6Iu4OYAQKJ0UtQlkgWYQIp8hiTlBnwNgCLhUhOglYiUqEpkSS/RARUbL1VSOAk7shxanH8QRcTvKUKmirBaePffGCJen5f3NTlTqacwrPV9CbjqKIo1b+/yuHj9f91Se+nXxvSWUlI8/sqAQ/U7uZHf1vIq7vA/fvpvlTDUIdGPfKMJ+MjmSrQqxWtSW36tihiTRWc8TqEpDLP1j+9Vc8bBt2XzIpsiARLCPgvQxqg1ASdPEiROx+KpBFCkRqwmYCc/psIAK6GogTUL5Usw8qZKSFCkeP4RiUkXJloJ3T2ODo2NcDrhODp0rG0X+MdYqYzOBviiSTfpCGagaEvWoauiBynjNToVEpVpQzdZClMGOkchyDEpaDURphidigaksNX6qOFYn3AIpkjAWBOhFTAEg8i6jnWIbz1PdZiBWF/rOmKhBFDa/d9+/j0AwLYbicyFLX/CGiz5uQARxQAoIbE1Oy1ZIvnZsK6bWBBrjT55X1sHRV9OdOfKhsUkjKOl0OCgWQdOPRCyF2ZX6ZfIt0hdP5IkzEzrDwBUnjG4imQLhRKr2QagDocsYdUsoZK5r8EbWBeL7Ipi9lfR79MmlnBdOl+FEUsIMM+n1gkpEiOs6NstQ36hSBJGhOHLxssbaJOhTYSE7IpmnLZp9nteh7hKyZHDry2x+0pIqJwr4bcJUBr4s/HbQCZtOflBzwnM+jm9lzBT3cQpRE2+H0wu4bjJG5g24p/136K6VYfTaEoopITB6iZOVmaWRSmyLhImyPf7Yf8hVEdtZtYBLO17BdWmsiFSlmQsVNHAyzPLcNHDE7062YxICK9sZGRDJ4QnEpYp54EpxFH+B0V16rqeY+BspvMSSlpvgniT9efDCyI9qXXyMoggZxG2p7xGbqjPKipihR01MbpyPZF2KM/NVPPKm74pTJKfT1DEQpuq41wvaybZqa3vTJQj41W+ejY16UcdzqmJCCc3eQp5FdIPTX1ITcJn5+eFiKU27st8Z8PtmInUpcgNqEwEPEFBe1UIeoQ8JdtqjMIGbIhUBwDJcCTqiwLJa0wSGpM0RxZiIQ45zUw8kxXNmcrU1v2oCpGMXF9kHbj/yQGHUIVNKuDWA6nFVy6HAGTeOZ2nryMiqbF0LKuJLKb0HUDvqQ/ZAH3k17z+h8DPxKCsr1l+6CJUNmsgDBMxDzTgPc1bIYcBJskQgLRKDcl/yLzFOD2zmhQm8TNIAsp0RvDqAalONkAUoilWPApMxgMAkdeW1kAKgLbT2Mpa5q83I8Q54D1EdZ8agGQzP7bmlFo6tGbrnedx5uCqxq4yZvd78CweKbMc0qpvy36t9XPphJsdWwJQhXm+a/2+BqiorgHmOeR1v98EMuUZVP1+U59n93tTWQLYt1zzpXIw10Ayl5++eyt/qeUbTG/z1vKNBpZGJWyMx7U7wWeDl+OuvG91wC/7azQ6lBDET3GBm+ZUAKGED74aN3ivO+DluMXGeMSscDdu4BN5jp62R/b+kBenjw5WJbwcSKvyWUchLwBZ4591B3wOCt+5bPpitZ/6TeF3W+sxRFtCuIRtNTHA3LqxAL2jb7CxHo0OSFlDq4SUNe7ZQ9BnVcKEDmOLznocRwejMrZuJGBpydspITUAcNn2FDamKMxs68ib5hN5OKQvkicq4TECehyDw8ZEjBweU4PWMRrs3YhBWwzRwKhMHhDvsHUe90OLrfMYosG+GXE/tLhoBwzBIkSNfTvgddygc4HYV2OHpIBd49EHW1hh5T9Am/TAG+zGkpdyrHJdY9JobETKwEU7wpmI23vyJuzbEX2wSNkVQLlpPEmncA6qgGrHAKl1AT5qaAWck57Cckbqj9EJSrFkTJzE7J2JiEoXo0LUpLtZA0atE1rW9jQmwXtTpEdy1iXnVETmU6KwocBMpzPWVE/1RStULOMk4p4Y3BFYl/xW6sMklxLYixoZoGU9aXemqGGaRMLnOiMnCachEB6j5lAd2qwoQ/d0Lk5MrBmgfEBaQ9YmDJXYOVQuUil+JNCnVMaYGKyJnEgmfUeYzGNTpK2XWZy8CSjC8EEjKcrFFL1TpXL5zGrDDKOg5yKvRb4kJtYlHS0By10qQFWYd8WKnaBp458UVMu/fwrIEVP+r2KLe1DQm4QYLNCiGDoy6BwsUJKREoHe8iPPuX+qzUCge2tmfQVAYGYToS23X4ERNWqAQScygd1Y51eKPElmgwMDEMpL1ATaooJqKm1DNYH2nDCJpYtupxgz4pQERhqghtgpXeQ8wsSAsLre5EoPEzMpB33WSJtUgJryCioppCYAYqATFs5BIbe0e9K9QmxA/221e2KNQBV52lsGVAKKVC4ah3A8fyNrFgZqRxmFDAJwsMQUmtW0801NLqytUMIeCgYkU96i6VUBJtkSq2hUDHA0CuNqsjSO7BTMmUBjqlhhVVQFIKRmYg41PY1JsyYkNF1rRgVzUghbAk/CDJtaMCtsRmzUxBg68PS1VM8MMlcTeM1GQY/U3whmD3UK2gMRvJbr+krB9gSSsgGgFZSnOQCA2FH/8y7DHolZVgUCovYEDL+G3Ig9K4QtXa9HVQCOigC2pNOYLN9fpEYN5f/aEzPZMpNq1vxnAA4Yomt486/jtJlPbg54iBUWiK0q780AJKMKOFVZAC/VMSMQnSrMtVCqzGls1IwVluqoApCzVhMrrHxM+RmbgdZnbGSOMGOdTZbq2R4I3dQvAUOFVdZiYoWtGV4jJhZZw3MRJqOIzFPWeJB3CnC7zPg6sclO10i7OqAw0z5khc3IIjOzBvQy5mypTIqhI90fqMa2YIWdgd48GSSgFgBc6gqDbV3vnVlh5+B6NUczTfdezdvleqsgvAJ0SxBdqrzp47cGCh8BiW8FuL8rX0v5RgPLjR7xr+9/CqMSxmzxomngs8GFphDTK8tSC8lhb3o8cSfsTQ+fLN5vG3SawtpedA4XpseNo9DMBIW+cziGFlZHvOfuce3OJTx2SBZORbzX3Rd5kxN/ow7J4tL2uHaUX7bh3KmIiTgnZVXCWjfGw+kIqyKF8/E30JAMLu1QAKTP5IVs9Fz64sz3PUeHS0f5aefOoTUBl01fZE2u2zN2HI46bCjnSSPjuqGcCQDwWaPl8EcKy4o4R4eR+yQhrEOyJCnCYZN9tCXHUCMXuRGRHtkZCmcMWZfwR5Fc6TtXvMSNDnjSOs43Mzh0LfZ2wEVDoZ2Njrhse4zR4Lo9Y4x1GKPiP41QeYj3LJ8yVl7KU9fQPTIZCzQy0nO6/ml3LDl8DXuHt3Ys7aWscW5pB9CyDEnDEjUAMHAOR8oKp64peShC9iT6qvJfwt8k3O/YUQjYRTuUOTc64ZIlM/qtxd5R/mFkECtzLn1sNK3hc0eSLgAwJoPjZkBjIjbOFxKrEt5bgW7JQUlZFU+5lCEatDwvYnBoTEQfLGLS2DqPY9dg14wU+qZIDsHynPTBFoNDrcEqXnzxngt4b0xEv7Gl3cYGtmaTZ95xuwL4bRUOeuoaOBuLbEzKJNES2ONdj0tkTkQqBgDGYIpxgnJxdNGAFS+9VpPsyZH1Oy+3/RTBEMmIIWs0Z4V+a5GSwrb1hZBLPOqiNQsAfeew60Ycm8D9nb77Tq0rOTyK79M1fnpOGwpna52H3xryBugpquDoWmw2I0UNbMwsN8fvLLrNSI6ExIaQ1lH+0YYMG/VeIXDImrURfmOLZ14MGJJDJMaScUPjb5rIHvxJNkU85GJc8RvDOq5iaMkYN/RZcy7Cbw2srTRyFx72oXEwLsFUXvycFLo2FE+NGCLC1qDrAmLQCDuDtvMYWwfXUNSDUkDYclisXGvzjB1bqYy4JQ+x1mSVTkHDsNwJgOJVjzsNrYGw06X9nMnLrCsvdxynHCdhuVY6I+wm7QOlM/xoSFYlMyiPGikDymSEvYZuI0ZraeNt0xQ2l1QB6Mom+NEALBlDHVCFREm1Cd5r+MsK2GfMSHoodCGX/zEw2xIbLnxUFNrG3mEZQyFCqj3RSU3eamDySJuMtNFFrkPIiZQQNdkMH+geYa8m8K8APyjE3SNsKu9QwsieYYDalI0/j0kFmgs96kkHkoFE2KnJMxwmw4B4nGlRocxj7SUTL/PMY7llQ0Sm65SnZyqSGcAEmpABHQj06jB5NjV7vsl4UF3HRh8hJCp9VJiBnGQIPAH0GgB85bFUEUUixPs5cJbvEcX1BfiJF3fpsZSx1LIhqiasUnnhlWTQzaRJWaF41Ve9g2nedl2UfA6WpQaZJQqIGlwCxSUQWgVEVXvSp3cpy/bexUv64L4rbT0Aj2rl9dcMLL+J5XfkPb+lJWeFTnt8Fi4KoASAl2GPU2pwZU/wDAJfhR22esQrT3lQV0w045NBqwM+Gq9wZc+4Cx2sTtjqEc829zjFFr8YnnCeouUcyYRTbrA3A4xK+Gy8wI7JbC5tj8/GPZ66IwDgLmwov5BjPSg3M+AcG9yFDZ4394ggQNlyzqBWGVs94gu/K4Blw2CL8sOmbxrJHbxpjnjtN4hQuHA9ztHhj/ef4i50OMcG39m8wjG2GKLFpR3Q6sPMyytA9i6QN9OpVIhxJGcQAA6hKWDYJ4OQNW6aI46hxY7ztQDa+A/JYKcD7kOLznjsNAGic3SFeEdIQ4jcpsXWjjgwGc6LzS1ejVs8bw8FmD5vD9Aq49Z32FpfwKr8F9IUKfe+pbBiBr4pKzzrDhT+jFB0QP/k6lPErApRz94NRAhiM/rgCsAHiHxEq1S81GOyJW9r3w2FSOTb21sMyRaCjDEa3LQnHHyLy4a0PQUUNgwQ39/ewahcPNFC9vF8c8AYDb69G/B63GKDjMYEyovLGhvlS97h/UjEH0+7I+5Gyp954gZ8/+JlyQG76PoqrFsX75xViXLGGPDK3ApovemI1InyC3MJK79q+jIn723vcTd2uGynnF3PuUg33Ql9tLjpQsldE6KJlBW2biwh5wDlu910J9z7Ft++uMU5OCQQkNzw8weAvY4FoEtEwvPNEX20xXDS6EhAmMPkBTgbnXDB3n6JIEhZ4WYTSh9Es1XaumiHYgy4aAf4aPCtPRHDfH7eo2WvdduFklMGELh7vjuUyIKaYbruT86qaLj+3tOXuBu6IpEEAO/v7wlM87OxKpX7pKxwsyXD1mFscNmRkWLkiAEA+M71a7w8bxGzwtWmL+RJAHDRDHh53iKDgP3ZO1xtejIK7MgoULNbSwj+EA263alEEPTBouXnGvOUI3azPVM+mnfFMCCM21YnRDZ2+GjwZHuGMxEn78qxZ3sC0Cfv0O3n/RFjheS+fffmFc7BYeRoBkkvOAwtGjYMSd2LdsDrc4fGRnQ24K5vSaN4bEqe2/ZqylmWfECJ0KD+aXRu3i7NoYWRcEQOCZdIh42jdSZ7pJGjNYRcpHVhZqiQ1IHW0WdQUhNaG0o+oq80XmNSaGzEaXS4ft7DJ136JpsbayJCNBiDwabxGLxFy/2Sz1XKqpCgbFzA2duS2mB04nOppBIAlHYh6Qg+koFD2j2PRLCk1JQeIlq2wsIuerVibLBspPHeomsouihGjZTIkCERM2J0k3H4aEraxKbxOJy/eiyscwHjaIsxRVIftKaUBdGblSgNAIXVvGkCUlKFbEWOk6ElcNSJGBry7BlJWyWfMCm0bShM5jGqWZtShIAFAIxNiBwaL1EUmteJ4nks7O4SvaETFBsh15jAkTTA6QYll9DGQuwCbiezhzyGahwCHNmQkmtvW82ELXUBIHE0ihhbFAqCmTF110XarZixC1iRNIA6DH95fcYMdNOxRxDT8rgYb5bt1R5COVb3C285PwNw69JUD3IS3zXHctmHB+NEMaa8S5kBwLXnswYm184/1p/fld+q8o3Wsfz+37jIf/9//jc4FNRhSG4GunwipkGfTSHcAYCYdTkGYPaa2PkMfJ42K3sz4JSawgabsioeSACF0IWu0bA6zsBGypoIJvgTIa+1SjjHhsmCEpOhqFKnkHsAFIKocnkvICdlNSNJEZAgRClCmHOMDRodyn2kUA4o/YDLproGUXX7uvpEj2nyFtbSLrUHUTa5QtoxljHk2TWlHoMv8RRK7usYbQFNKatyvE5+pjZN8ZZKH+yKuU9AFi58AAAgAElEQVTqTAQyRBISGawkqOINJY9knF0jOa5yHzkOTOQaAEqOrsybUhkxaxiVEPkZ1etC5kiARgEjKpUQ6fq+sx93fi8bquUzyFkVD5/kuMqzq3N85Tp5X9+z9nDGau5FekbWUa6O1c+nrrOsK6Ap8/3rtuu1olbmXDbXKasCUGvN2loqpgYgS33bem7lWUrdZaJ9kSnBZHmMieZfJISkTPegvCGpJ8yFdallhUgiiXKMJXRbipCBSP/qz149VgEMy7HV9eVZlvmvQt/l97wAXhlT1WeZGwnJrqV16nmaNsS6MCTW59fmVzbJIiUkecQiPSQh1PW5es5D0CWUu85DlrDiusSgp/xj7l+Ketp48+Z9JpVS3XvqPPeZN+H161JULiHXJby5mtsHWsJ54YqoNrCSW1zn/T6mmZCCnm3o1zwDQnAiYyySNJg8pllkcKo69e5R5qTMVT1PWRUwMbuuzo+VvksfZTz1AlzkyCJjntO73CjX4OLXCIV9sMktG3C1TpwCzMdYXzv7IPH/Kj/ywcKo+8C5s5I/SeGQ1P6M6ITv80DKZfn8l4BjDdxU674eX700ATzMNVSV9205d3Ic6x634i1bzqd81utnIONYPt6Vca1ConpOFm086h17Eyh6h2X2Tl63dwBwMiVvam92/k33fdexP1L/rcdXyjvNw1dsuy7//B/+J1+71uPmD7+V//C//I9/4+3+P//eP/jaxwZ8wz2WEap4In02uA2b4u3bmgGfxwt02uMQKKTyld/i0hIt9n3oGGhFHGOLG0eU6sIwOiSLe99hw6Gkx9AiGQphHRKFmb32G2iVcNOccIgth7Ia3DQnZg1MuLQDjrEpG2mA8hdFGuPWEzvbleuLdytlhT463LSnAvz6SCGjrQ7MFEvtieeF6h+RssYpOHQm4LN+j431NPZ+i4uGPBOn0BSQd92ci0dN6N3B5EWNjjiEdraxF/pzAVfi/dlYX8JApTQ6PpCFAFCIkvZuIO8a04tv7cgskSxLMnRAC9wNHTrr0ZmA25GYFK879nTpKadTvDeSa5nylHcqDIdKZZx8UwDIRTNAA/j4nhgh39sfELPG2bsJJFv//7P3Js2yJFma0KejDe53eENMmVlZI1VF1wZEWgRhzaYXiCACLGDBmgU7WLYIsGLR/QuaFSv+DIIAIjTdVFd1ZWVmZGREvHjvDu5upqYTi6NHTc2v34gXkVmVGUWqyHvX3UxNJ1Mz10/POd+3AYSHxVLspfFkUVCr9W0OGqaAwkdnoeRKkNQVy8tofHUJ5bFl0HWYO6LVL1YmIoBKeHQWRiWcFoOr3iEmWV0zGSgSuAJ2FvBJYl4Mxo5cGpeo8HDs0dmAfe8we73ZGW+tG0oS0GQdVbZKtBYLH1UFIJ0JcJ7mrdUBk7PY9QuWsMaYspXFFUvDEnTtN8ewAmSRYYBMbqgBzlM/jrOF1ZHiKAW5o6pi5XCenkkp0xqXPHUwJqIr9ykkivH0XqHvfXUP5eM5CxhN80KIDO/J+mM0udKGoKrFhGNemaZe64jTsaeF836uYMt7haGjTZAMsjo4Z4As0A9LBTpMmtRSxTtngNHheOjRD0u1hADAdLLVFVaqhOA1un6pIGxxBhnAMCxwTm/iXIUA5scO/ZUja1GQiF4RSVEWCLOG3S30TAUFYwMWZ6BNQPAaSseNbmvwRNykdYQvJFMpCigTEYMi3U6gArSwKOQoobtA7WHCJUEkTLLMCymJFCp5CdWRRUTKEmubBZSNiIuCtHFdC3JcYgFA8dEAHcWS5iyQF4rFVWMgvVGRi7UESCeNtKfjeVaQe490MBBDWK0orgB6dr8zaQVIDJYcW214h0ACrdunJJfT7BTlmeV2Ua/zSnwkAbBLqsire6jMwCKrpQIyQziJ1BWwZxKRJHF9i0QeIuSjJhfIQibEVg5EEGmQJsKipDNk6Yf05EqaJZC6BLEIKCeRenL7FIkATbK5ujFmUwhiTIZ0BXyWOmss5ZjIDbOJb0MSSF2iOos7Z7a5WpPYXTSbDDlJcr+VQJYZ0ouVKMjQ99hlqEJwQy6qFMsY9t8dWEonkPoCtANo/Gj6ItriZqryGhsLVDdh5USNqZSe/magxkeyO2h1FW2AWCrEOK0rrD5RH0UuZTqKj91YsyKQm3jfZLltuYJTkWnMWhdaGahMjpvNqo33JRRD8Zt5M+5tP2v9mvIpR+0FAMluxILGEWUcnouxrK6wEsh6G9va4txLMYHVxbY5x2Pb7tNUV9i8LSOLdTzA9+AclHP7yrPc1sMuwm2M5QYINuWJjC3pU3o+78ZlNuKpxbVuCJS+no/NBQB5EZw+AzQ5VvQiKHwO3J9vbLT5v2bDSyTiH1jb8C2e4V9hH+nvKv2OFfa3NAmQ9AVH/EiRq/SGS6bKQlDsFrmgAmSh1CJWK+VQgCmfMyIiCYlBeWgZKaayuFhGyGo9G5SvboJkoSEX0iVpkrMoE0eLBC0aC1N5aVSrpCyWK5FIpiGT9a7KiohV8oAtNQxSbSHYAciKKQXFvGkZMWhf5RF2xtFCHgK2yC+0lkIqK9SxlbX8AAlV66L+xNoHKUj7sJWU4L5xW9prSF5jPa4KMy8kalu4P32RyuhUaNrI+lTkmsjyEwAgc95Y8bi8c8tiV0iM2AIXoDA28WlKEDjpivsssPZLIlc3R7ZGJqxj2KlYwfVgAlnkShuUTOh1qHGJnE+ITFIfBWSlLNDrNZZWiFwZfbtCRqTK9aL0SxRAxfWlLDbaXxnA0Pmqz8VMvWy547YouY4fuwrWZwuocYgMFKVAw5oLsqyl0kcmLCh9F0ICCCVOMVS3OT7PeQECmKQBlpBzqABWMZgSGVav+dlKKwuIBQBrad6oCoJo7KTMa35Bs5UtrG2MJdgVkI/pta/QZE0VItNnmWA7T7GBMkHWMtb7y7Fv6PzGKshzledULD+e1tL9MzYQm2+5zwBgbKjgSZXxZCucEBnaNM+JiWAts3pNHyrxE407KljM3eriJwTFRmoTSpwjX4fVIlXAsVIJqYBOIUWJfUS5L43l2NBCW2laMTELsRCZDE6CwBPHVQqZK7uxEBnKEEhXBfBzP3ImciQiVaK+xI4IpditNJV8UsUaGynZathRPhgg5kJK1UVI094n+svWPKkTLXrKGLSswmy9ygKQJiJLXrECUmWknCBUQjpzvROKgCLHKWbDZZX5U1Z+vDajB7fEeVoClkJlZJlWsAtAmITUl1WqLmCYCZkaEqScCdhW45QWK/utSciQSCIh21QW+AJIubLCMuttZeMtPxZZFzdKyeAwERlKyZJ0KcdkpAIWRW5iN4HVOqfo95YWy9S2JFdQls0KhAFJbTEoK2BZYyS/S0oCSDaVtgFgMpRSb1R0vyrhkljnAg0okbkwuQ0v+HkScH9FxsZ1syXkobGgecqssFkWUN8yFQNr/GuZi1nn0qYylRRt/CS9vS4rAoeilJfLPamWRGQC/nJtcwVebb4mLhIiVyKb3DAzi0pytAXTtYzIY7wC4nXcxQqWynzgOEpOq5toA84uAJ9nYyxF04dzMNmA2g1Y5b41QPFrLalo8ovLx74WWDJIb50jztp4DiwvAsgLbXoWWIrt96+75rzc9ypjU177ksTTQeI6nwOnv0t/b+l7DSxjlnjj9xgVAcNDpLiJUZK24UMYKgHNtZ7xdtkVkWqyWB5B2nAPhTX2MZCFkrTubLXkqe6IB0/WTI4dBFBj7HSJRyTLoa7i7+xGyuc4sXvloHzVQ2OB9KW4w/qkNu5qZLEMWHg3vzyFrBvnosaL7lStl73yuJsHDFpj0B73S4+dWWBlwCmQYL2SJKXC5Dwu6qqxyDGLx2CruyQADI0UCxP0TMHAa1XdY9kyOWiPk7dAh6r3Rtp0FPc2aNIdi1puZFpGQ/0/LBbX1uHgLfpMun8Hb7EU7TOO5Wq13WIBi+ekM63EyFysZRnEcqtkwsNcYkuLNY4YeskKOBhfXQFjkjguZB3jWCpTSGekyJi9rmQtR2fJzbPESCmZGoudgtVyE1OXssBU9L9icQFk8efD3NXrgdWdUskErVIlUElZYOwWhKjIKlkAsw8Kp9mS1cr6Yo2kOSlE3rg1E/hKGz206gYcJbRSNNaF8CVEiSWQBTOYgNmZeg+YhKXGP3qFaAMWrytYMGoltuK4RRaQDjpi8RTH5hYNrWnxL4AqCQOQBTHnlRVWiIx5stAm1vg9ZqVNUSL3a9855oliplbtNL9oWghZct0MQSHqtX0MaNiSOZ/K+0CvVscQJLJdXTBZ9gV5BWQ1BkpkxOZa7+heL7PZuH7yMVkWc1GmGre0aTvPfV8sx6phqz1pLI21M0VRpU6SU1iEAWvUpSTgnUYyRfJFNwLbIiN6JrSRiF4W2RGJZCOSX6VOqji4l8g8XyOz61JZ7M6ZokBKiayHQSJ3q9RNWsjMkzuBtChijC2LiBxXK2ROApgUSSwYWu2RxbIQ5ZYxSwWs5VkhyMJk6yWCzIBT5Cosiiunk+uCJgNJyzWGKpcF9lIskExgE4itGaEASwnEki8rCeEkWvfFrNNqmRQAGrmRLMuqtVgo2dKZFX3PbTtYRqJY+rIA5EnS4p/lSNjCUdxDsyrWzQTIYkklNtUC1BIgFwnhaSOpWixTkR8peSsLpyHrGDKQjKLul++Rr88FUBTrX4oZIhCwERF078pcq9Y1m6EmIsfJSoCtXtVyFgspTZbFYsnspgLSYWX3/A5JLQXFAtXiWpqHlFDcU4nRtgIKSeOvXLkfgcanAk9BFkygaVsueWS5pw3pDZUJqElAWJ4bxaKozwBEWsuUXhADdXhqsRRme1218MWykSHL+Lbgrcw1UQiAWBqmtdYyG2vWNBeELZskGQ1REPd9C85aSyCDvVzYiVvyng0YE+IJoMhSrIB1fXXRObFe/3UWy2oRPAOW7edvtFi2wLCpe63oDIi2x8RZm88AeP3exlFyai2W7Rx6BoC19W0A7dn5FrheTE1fnxz/pnQG7C9dI/KvYMX8DafvUVO/dfpeA0sjIv5i/BR/NX8EAPjz4TPM2eCNv8LnyzV+v/8Kj7HHJ/YeXyxX+PP9Z/jZ/BIA8I/2v8A7v4NLGh/tHvDT6SV+2N/hy2WPQS34Uf8OL/UR93HAvz58jE/6e0zRFpBKbop/OH6FmCU+nW9xayYkCLw0R/x0eoE/3X8BnxXeuD0+7h+gkGpcZicD7vyIx9Dh37v9OQDgzo/opCeBd+mxVzP+dnpd4yF/b3iHQ+wwRVOtlTEL/MkVlX2tZ/x0eokEgR+bd7j3A/6DFz/BF8sVvlz2+Pdvf4afzy/gksKPd++wVw4uafzt6SVu7AEuarzeH/CFu0LKAteKSFp+MN5Di4iQVRV3H5THTruiG6pxYybc+wE7tcAlXWM0H0OH39u9w2fTDV71R+w0kfccg8WVdni3DPjh7T3ufY9eebx1O/z46i3ezHu86E74i9vP8JPDK/zx9RvcLQNOweIPrr7CtZ7x14fXGLUn9tZCPMLyMWzNZDKeBIFxWCoBxavuhMfQQSLjq3mHKRj8x7//LwEA/+fdj/CqP+LGzDhGItdh8M9A7M9uv4AUCV+5HZGw+I7IfqLGi1simpEi4wcf3mOKBl+5XXULfmEnfOV2uLYT3rpdJcthTdWPh0cMcsGn822N5zz6Dn988wanYPFx/4ifnl5Ai4i9cbRJUDYdmGzn8+kKw3DCB/0Bn51I6P3azviT3Zf4crnCz463uLYzWdKLfM5cNj/6ct851rVXvroxs+v0wZOczaB8JQO67aYqwfPD8R6fnm6wN/SczNHAFWmYWzvhzbzDbUflaJGqyHrIEjtDLL2sN3u/DHjVH/HVvMNHwyPeuhE+KfiosLcOrriO39iJgH009diPP/453rkR94XAaNC+yPws+Ox0U8FvX44DwKPvq8szb9R8Ne+gZcJOL7hfaC4M2mMKFJe7tw5Hb/EnP/oJpEj4V+8+rtbInVlwNw81tljJhI+GR0iR8dnpum52sEv3FExlG/5kfMBPHl7iD37wc3x2usYcdLVA//jqbXFpp3t1bWf88nhdNz9e9idIkfD56Qo3HW2msbYuAPzpH3+Bv7z/EHPQuLYOuzKXtEz4ZHjAXz28rn17Nw/4YDzi7TTitp82REIxC1xbus9Hb/FyOOHobe333roaHzwHepfddMRW/WbaVwImZinudKgbRidvcNPNGPWCN9OeNHOjwquh3JdpxKvhhAfX142dvujssrTRn/3pF/h8vsKDo/v2sj9CioxPDzcYDa3k56AhAPxgf4+f3L/EVefwqj/ipw8v8OPrd/j0sM6VD3cHAKt3x6PvKwlR2+Z2vG/sjC+nXfVC4I2rq85hDhofjY9rbLzIeFy66oofs8CLfqpj6MqmmE+yjgNv9H0wHPDltIeWCYfFom9Ic667GZ8f9vh3/tEbzMFsnjktEjpNJFMnb/Ci3OOXwwkSGXvjcPAdlqTwdhpx1Tl80B/wi+MNRrOgVxSm8NW8q/fzwfUwKuLRdXjRTxAi4971MDLhtp8gkfHL4xUG4ysJEm/yvZuHTbjAw9zTBk6U1bPk6Cxe7k60YVmYngfr69g/Lh32ZsGb04jX44k2Jwu50ce7B/zN3cvvvO64HWa8PY51E48JqDod8PY4YuwWzF7jqiNm7JwFlrLx9bKQWx0Xg6tugU80931QuB0nYo0uz4kQuTJwZwBz2VRkjwwfFD7cH/BuHog8aSGSrZM3leSJvUHmshm66xacFoPOhLoZyAzZh7nbbNZxfDeRRwksQWPXLfCF5Ie9Q2avMRZPm2mhTbCrYa595lAJ5zV2/VKJk4TI1cuCNmkB502N5Q5B1Y00qdYNQ+912eBiLxXaIEPdPBR1E4sxB2kaFx1pQUio6keXTUzeLGSt6Tb2O2VRWa5raowFTDokZfEkaOKSFes8Z6ySSxwLfAF4UVwyl7tqJ1dALFG1k9t8UufNBiGAJzHnuWoaX0A17AnBbW+/M4DlNojyuWXB/bp0Bkyf5P8mS+M5Aj9H2bXcjIuWzN+lv9f0vSbvef3vvs7/5H/5T6qVj/9xqgQ0WM+xVQTAho2Rv7cpoS2LPvOPcXusBR2XSD3aMjjvOflISwhz6TzHaHGJuSmPy2YylPP+sDVPXnqZYH3pfJ3P97YsgVb6IJ0VywQk53l4bFqCkpbk4+vqfK4vbdsvjVM7jhf79Q195R9QABvGPE4bko2zetqXcHvsCdnHWb3Plc95zufXpfG7dIzrbtu25n/6o3FOwPJceU/buC33Un108HxL9D2OAc/+eJwTwTxH531pE/Wbftc27eDPX9OW59JKaLJtxLlr6TcX1Hy+0OB2blw6/uxO8vm59/15+KYf/rNxaufitn3r568di/Mxf+4+vHf7uZyzay71q73m68bwuXzvM9menYQX2vGrpG9qQ0uQ0+QVeWWf3JDE4Cz/c3VeKPO9XNjedz7g6Wvk7yR91/ZeSBfbef4svu+8eOZ5+FZj8uwi//lLnsFK73f9+W/S19V7oW3v16ez38n37duF4+89hr+OPN+23l+xjveq7z3L/Nbt/Tvu3//zz387yHv+4J//17/2cv/1f/o//sb7BnzPLZYc59gV1k4XdY1bBFYWUra2VAZSKaqMR2uNYyBZASoTiJRdQ65TZSY62dJ/A+villk/19ifAhKx6jxK5Er60pWddiEo1o4ZLhlA8bnW1ZTrY1B5zhbKu/Yru2h+ltGTY3bOf8PiZjFYYpzkdtEu8soiWsstf88ZRjlmkMEhu3Oeg2kAxe0yVUIZGgfKp1Wq4DXlXOWlRN5SrUuZyHulAR2pGT8GjUuh6a9MmcyWyrFYWUBrYvtsWTQ39/9sDHl+cHzXFiDjAmhbWTB5x5Wv551bPscL7pVEZc1/HvdG16Hstq67xOfMpXSdqOUCa/uojefg55yRE1AqI0YBpXJl7OQ20dhT+9OGaXL1OeJ+lYixknf9W8GIWMljANRdZyov1/7z7nQtv31mW+DTbOLwvT1nik1nn+uzIkqMUnENYxfVXMquGwF8b4qLYo3Ha+YERIYoQXs50m51CpLKbBaL7djSuJWdar4/xbWs7mrzerOMa4qyxgcio7Bzrp9liW/M5R7nKEkbseTbzF2OAxOoMWG0O8+DUBL3N4mnq9s67mcbC1msMYPct5Yop5EduLj5wHqI3JZWKzHzTSn/gijHBd1gnSupTU08Vbke1ik8X/ReAtZnzKkoz/xlptAm33NWgZaEo7gscizduZsen6suixy/KFaXxuriGASNF5MWc/sF1ti2KGj1UNxgyUWx6TJbPIoOYntsJWcp31uww+U0boucD01bSX8RNY6Rx4M1DbMq9aq8dYEUgAiiksx8l8TuksDaZh6n3NRX9R9rRjRjgeoKW2PyeLybubDB2k/mCKq7Kl9X3UixShNtfpSKpWv7AkN1R675m+s2hDNtG7nvXB63ofSlPoqFJKqdh229nIfqaJ7hJlW31PZ5zU0ZTR83r5bmGXjyXJy9fp7saX6HKfIsUBVn37/h+nYILpZ5/s45/36e2jnwTDu+dmMjX877jZsU73PuPcemrfs7bRh990f+d+lbpu81sHRJ4yu3wy+P1xuXofu5x8lZvN4fMQeNq87h7WnAy3HCV8cRAPBqd8Kj67AEIm65Ow3Y9w6nhVxNr3qH627GFAx+8uYlrnczfCAXPFsAxlVh7nzzuENX4u32vcO7w4gX+xNiknicuko6wlY7rRKmxcB7hY9vSY/u6GzJRyQdg/b44rCv+n99cV1ZvK6C4jkDY+ehZMLOLvjicV+F16fF4Ee3d7ibBzxOHT6+ecTb44glKFwNDlZFLFHhy4cRXUf6Wb31OE5dAVHk9rEbXI3hEyLjNHcwJqDTESGSG8++dzjMpAvn4ypDMjmL3nq8O+5gOw9bXGtCUOg7j2my2I0Oj6cOSiU4Z9D3HvNkIVXEy+sT3t7vcLWfcJwtgle4uZqwsw6/eHsNa2OVE6ixfIUllcf78dgjJ0mxduVHfTcsmBcDKRPmaUAOEn/4oy8BAH/7xUsolbAbXMmTMc+G3q1lIb3fzRAi4zh1mxi7lASG3sMVZsyb3YQlKJxm6t+yaIz9guOpQz8smE4dhCSmTyLgybjZT+h1wJtH0luNUSIsCrurGd5r3OwmvH0YIWWGtaSnFgO5BaniLnQ6dFA60dgeeyAL2M7jR6/u8O404OEwwNoAU8hpYpQ1nlDrhBAkUiJ2Tv6eE83DrvdYFg1jSDMuJIlpsuh7T7Ggxx631yfcPYywHbmthaAQg4JUEX3vMZ06KsdpCJlrvGBOgLYROQPW0ni62WDcOZwOHXZXM+bJIiWJFAVMFxA8xdvZPkDKBL+QmxQA3N4ccZo7uIniBZVOMDagtx4PjyOBzyigTII2tIGwOFNBE7O2zicLIYkwx02GyGRKvCEytdk7jdevHmFUxGe/fEEkK6Bzy4muoX/A/mqGVhH3j2N9l8miAxcW0k7NSeD6esLd2x1evDrg/mFE9qqCqBcvDnC+xOMGiX6/4HA/FCAqMOzp3XR66GEGj5zW+Edk4IefvMNnb26QvITpA2zVw8t4dX3EL9/cICca4+VoMVzPmI8W3eixzLrq1aUsoLsIIROC0+h2C7xXsDZgPlnYwZMbGECxkQDsuMDagONjD2UiyY9EiqFUNiIGCWkiolOwo4e1AadDBymB5CWG6xk5C7ijxXAzYz7ZEpuZoQzF7caF5DB+8IM7vD2MdN9UxjAs0DLh/t0OuisbRSUO9OaDE959tYcZPfa7GXdv97j94BH392OJCQXGF6QPyizK02SpvYqIlaJX6Edi510WigXvh6X2FUBlujWDR/QK19cTxYaXzcN5spt71Y9LjXVN5fcgRYGx3GOO2725mnD3MELpiGU2NKdlRvAK487h8G7E69+7h/MaszMUHx0klMqwJsAtGstssLuaMU0WVzvSoR27BdNC7vCHhwH9uODF/oQ393tYG9CbAKsD7o5DkcUROJ06aBPhJlPnIr/X9yOxEd/d7WC6ACFQ54zREYdDX98z1gbMR0tgKQqojp5TPxl0NxP8ohEDxewqGzGOxAJ+OPYYR4fH+wG7mwmnUwdrA0KQeHVzxOdf3HzndUe3WzA9kHu9Ku8dZl0+PfTQHc3hYeeInTkLYiAGMF7PCEHCOw07eHrXRpoPw9Vc540Abbhpnap+ZVyIFpbdFJOXuH15wMOBXO3DojDsqc7YbEbJ8mwCQL9bMJ8sTBdoHgYB05O2qz9Z5LIRIwQgS4y0MrSxyXM2JVk3u6SKCE7D9AE5CfhZA1mg2zuk4gprbEBYNOKsYHYL/KEEhUpUyRrdU7nRqZXB2cu6qSJMok0KkZG8ojjmsvkFlemaVHQ0eTMJWEEL5w9y3QxSmQAxsy/7srHE19dNgZKniytDM7BFfwzAmKQpNxteOm+ZnXkDDljjnNvEyCljJV46P8/M0LzRF0tsNh/PTfkMxEv8LxSqt0ELGtuNJtogKazR7LbbpCpn02yOPBmLtvxzoN5uDrTXXPrLY30u5XMJWX4TwP4tSRnbjf1/aOl77Qr7w7+4zf/V//ofYVQL5mTw1u+gRcQLc4IREb90N9Ay4hg6XOsJd37ElZkRksKdH9DJQAQ6ocNLe8KjJ2F5FzWO0eIUSPvxB8M93rj9htgGAB59h5AUPugPG/KeF3bCu2UAALywEx4DxRS0rri9CuhUqDFwo15qnA3HzLzoThWkcVuYTZWtKadCrjMHU+OHKG/El9MOo/HY6QVvph2uuxlakpA6W9NeD4caWzUFs+mjFLnmZevIzixwUVfAa2XEwVuMxleWUc7fqYA5mArQXdCVkZVIehY8uL7GyjHZD3+/n3vc9HONF7Iy4t71WILCq90JkzeVvIfrZcIddtfdd0X4vsQmCZFxXGxlBR2NhxIJP7+7RYsz7UQAACAASURBVM7Ah9cH+KgwFRKemCRG46vlNSaJgyO5kd6SjEUr6n4qGwQpC5ychZQJfYlRMWVDYewWnJytMhuqsZIeZ4sYJcZ+KdbhDF3Ie7RMmJzBbnBElFPInVj4my3tu46kPubFYNfT/XRB4XTsYWzA2Dv4qGq8SMuS2upasoSJlCtrKktrhKBqLEpvPebFIEZRNwb6YYEvMUGyEAzFQt7TdbRo1DohZ1qk8++ID6qCdSEIPDtn6kaEsbQQVTJVAiAhMpwzVJZO1do7TxZKR3RdkRsJROQSg0I/LLXvfBxZQJuVfZQXeJav96qeZ81DIXI9Pj12QBYYb6ZqFQ9e1esreU9ZfJnBrxZaZsfVqW6ABKfRjR7zwcIMHkqtlmp3tMTmKIiohgEix8d4RwRBdvDwC8mNSLWKlocHC329FPIeWtgykMmzgroiOZYYJLQNCLN5Iu8BgLwPysJe2lgXhmlRUH0kgMfkPYUVNDoFRAHZB9JVlLnGEbE+Yo4UMxQXItIRfSyWy4Q0l3nVB6RZQ9i40Vdk8h4kATxo5C4RW2oWtMBLgNgF5EXR4oQtZycFXHlkp4jAZhcgjhp5KGYamYFZ1QUfgJX5tFjKUMhvAKzsnp4YZ1u5ETDJj8oQs1oXnBnElsntByAWURefuVkwy1kWAo1MjNssg5GoDBGKdU9nyFkijRHqQRO5jmkWkLGQ5GgAOkPORZJiltSsYhXLEkhDgnQkGRL7QqYTyaKUzGqFSyZXCRI1Uz9YjkR6alccM4Rn8p71c+xykRtZy+AXBJPzxC5Dn1i6o1jDPBHmUBmAXIDYk7xIMiDyHgWok4C/+u7rHuWAWORGmLxHZAAJSN06XnJBZV5lq6KeabGeDDaSH1mSRAdQ2FmLJU7ERrKktbKD+mIOdB/YWkpERVgtp6AymDlWLaJKsTArrQxUZrJ5S95T5FlkKH1Q2z5xn7NeiYeSLvdpwWpVLeQ1WdPxotZWCXlqGwTlay2k1fW65BW5lNW0U0ZsyGlaK+tqicdTxtfWYpax9vWM6ZfziFZupEkXrZwFyHE/W2uwyFjZZc+A0BOLpWjaK7bXV8t5+S6LlXhDfsTFpPUZ5nG8mLieMys/1bsS5YiUSx2CCHNaMMnlfE36VtbGC+P0q5T7f/zP/91v3F20/5Mf5t//Z79+V9i//M/+h99434DvucVSiYQbPcEU6RAm3ehEgJEB13oqMgkJg/KImIm0Rugqt2EEyXh0MgBmhkIqRCjE2ipB7rZXZoYRCZ30cMlUl9uQZSGMIcKaThFYTZaeMi0jrgrZDyefJToZYWTEy+4EgCQ1mKAhQWCJCqMmrUtfCBaYlIYTgxkAmFWoZCmyAL45aox6Qa8CXg5EzELnQmWCHfVS2V0lMgFcrG6rWkQsSdd6euWp32p151UywcpYj/G1DIT7Mo4sD6KLPw0TplgZSSJFxHqdFBmpExiK5Ahfe4MZTmt0KlCZqpVbEVXLktvQqVD7V10HbZFpgajnr4a53gdb6mIXXduMOZrPXHcrkSKwSlKw2y8TMJgiEWKLi7Mt+RjEcRkxCYzWI4NclZVMkENZpIpcZUxiwxrM3wGg16vMxlAIStgd2eqAwRSpDx2RMqpuJdC4ADdtar8zC21UEaFsJLC2ZEoSXbGC9iZUIE/101+vFFmVxerObNQ6fpo3Tgpg5LysJcn9WiVMKGZXCXLx1s0xcs1N6Eo/q/6mibWd7P7MzLE8ngAqgQPfT2a3BYCgZSXkkWX84khAkaw2lE8VVl+en6yRmbOAtau8TyoECUomkl0AIARgTUAcBLoubNxPU5SQRfZDiIwgJawNT+JMWWqE25KLj2zaiSpZImVGkGT1TkkgqlzbJqSEtUVeR6dqRWkTgboIpVa9yqRy0btc/SNVBUW5gPhArKqikRuRRLKRlSBCDqGQdSJpklJXEKhAGYL6xXUkLau1JyeJsBMQJlVQm4o7rzLURi4ToM13bRKSzEhaQXWRNqi6WO9HBGgRXxaRQufVRbjsqle3v3I8a2oD5xEqEbutLHnOd/xVrrIiOYNkShqZDV7FJgajACAzopakX5kEoEkKhNuUBCC6hLgvDLrsRsxEH2zpUBlRUX1RNVYJtnKYjKgEkpbIXSJrRiyWFZOrBaNKj6iMyPIiBRglT4A1dQlCM/oAYBlIlrYrAJHkR9jlMuu1rJjlKj0hAGhUvcZkCMgkmwFJYCsVVs8sQLIr3zFlJUiuJGPD1CoSg+cCeGVjbSrgMDDQVLnIV+QKfKo8SWNxEjGvmwpnrvOQGR7cBmL0zXoFsitLKNUlyiZDVgVElnFPDZBrWUNFyZMMKlDh8lvAl9UKFll7skpzoAE1qsmbV2AKAeQgSrvzFlzx9IgriyyPm0iFITifgVGggpBallrr2oDO9r7KAozZ3fscpNjzmfA0ifXWNW6+Tbu43ksg7BJ4asDkk+ubcjLfxxYUtmWeAexNvRfaX+vbgDrR5BO1XkA870Z7oS9Pzouz78+kto73csF9TzD6u/TrT99rYDkng0/dLd64PRIErrSDSwqnQNbGWztVxs1TsNhrh79cPoQUGddmLqyKCr3yePQ9dnrBVGQ4euUxao8lKfz1w2tiN4wKCaICWJIukfgb9wq99ohZYtAef+Ne4bojoPK4dLAybghXlExwUcMFjdfDEQkCX877qp/I8iOfT1fwkba6ek0yKC6sIA8gpkshMnoV8NPHl0hZVMZKtka+mfa47Sd8cbpCSMS8yaDpL999iF4H+CTR64DPT3sA60KdQQovir8MOxhJLIKhMNrtzII3hQWQXWFJMkXCyITjco2+aCrGTKyIvQ748rjHzi54U5jgnNcYLLnxapVw1Tl8+nCNq27Bo+vgo8RVt8CoiM8PV+R6O6sKZtmNmBf8PkncTX11P2atxcEEHBcLKTK+WPaISeKHN/cAgM8fr6AV9e/kDaTIuPPDJi507BYIAHdTDykoFpTjKDsTMM1dtZb6qPDoLJTMlY3v7jTQX0e/VDGLqom471b5EykIZPqoKhvfvlvwOJMLrtURp8VUwMcWzjeHHZRMGLsFdxO5bBkV8fH1I07e4KFYPwm0UR3OF6kMFSurKsWyrt+FyOh0hA/kJssW16OzsDqgMwFHZ6trdLWCRgXnZbXeTouB1QEuUAzwg7M1VlQpikZmoHucyZ36OFuMnYcLCksg2RBjQpUJ6QqQ9VHVONX94OC8xqG4LDPI0zbhOJexjxJapzp+S1DVzZxlYyZnCYDqWOVedBkHdhtfvMaLqxOEyHj7sKtWU60jTrOtz7+UGbvBQQrUNvDxnFHdqHMW2I8zHo89rnYzTnO3YTu82k9VDiZGid2w1HpyFuiKG/I8G5jigrksql7/4esHvHscEbwiK/YY6sbE/vqEu8NAbmwmws2muCYqWBvhvVpJdYqVl9gaVbFU01/nNGwXVle+Iu/RdR5aJczOrBbgKJCTJLe5QKA5BnKP1DqSO7qg2NBhJBflxZlaH1t8WcKG2R9ff/SAkzPFzRcYRtpIOp46mAKeY7GOjy+POB17aBvRX804njpcvTpiOnX13T1c0eYdxy4vi64xptw+u3PVwg2RYS2NoVRkoc9FUsfuFqQo0ZUyOS6Y3SOFIEu+2S9lTqP2M2UBW6+j8eu6AOfIOh3LMyoksV2aa3JL3X98IKu5VzUemrVKuV7bzfCLrq7szJSZksTiNLQhLwDyCEhV4oe8EMi12Xt6L5PFOxbLfoljL/PFTaZqmaZIrvJSklQOSetIcrVk6ZwMyGJpjIuEunVE1hbJBVLoVN2Ns1eQOiI5DdUHZK9qmabziIcO3zVJGxEc/S4LnShmEbQxkp2q7p0wZKXOpY0AaJMiCwJSOld3/Fw2MFiCh8sDy+YA1RrPKSey+rOLbI4CsAkprNI8VBCqtI40Ecmr6hWALKgPosj48AYDUGOYOb4ZQdIGScbqgikzcqCNE+TivpoB0cXqLi9UovoDtW91C8Xqtlk2flDkapDFauEHmjhqAFEU8L7dWKkuqBeAFeuritLuzJsy5RyVizVe9Nw9NWON221BCyOcBuxvjjM45r6U+9dqarZA9ByUsjWyPcZWye3mAaoEyxNJkaYPl6yZnCqobMqoFtAnmZv+v0d6bwvl1wHC9vg5OH+urEvX/jak8q7/h5q+18DypT7gP3/5v+EujvBZQ4mEJSukLLFkBSsiYnlD+KwgkTBnWsz1YsGcbV1An5LFKBe4ZCDLk9QLD58VTrcdTLGy+awgRYLPCldyhs8ap2TRScprRMQpdhiVgwKVK0WCQkaEgCqzey5Wz1EutVwANa8UCafYVdIfIyJ8VogN061EhiptNSJSXeVYzBJ9aZPPCp0IcFkjZrmx7p6ShRGx6DFGzGXrkeVR2rwAxbUa0VhNSx7uu298RSqJ0Nk1Lml0MlRrn0saCqmW4ZKBlmSFnqLFUGRMuJ9SZEzXFlqu1kpO58y3bJltSZrYusxtAYBbQ5bju/0IIyMkMnwmTUfW7OTUFakBX8oIadXIbMvuZEACaXKu7UtYkt6wC7dpULSYm6KphFI+KQyKaPytDFVyhstgMiGWoXFRl/FLtX9SkNU+FbkXLWMF+nSvZM23JF3GicYqNO2nMV/zsvt2S5jFkiWyzHW2gANkKQ6F2IrndiXVOrMY8jmWk+FrW4t0tXQyc29e6+pVwJJUbb+WsVqv2+P8vJObuqr90pLdg9d856RfKa+eC2z9fzWcalu47ev8yNViv+z1xsLfjgWNVcBpZzHqBfPebPKxFb31Llhie59SdQFvrdEMcEe94Kab60ZXS25mZcRN2RgTIsMnVeR7JJRIGzKyatEu74j2L1/H+VtmbS63JRzj8WlDBrQkT4qwl7UPnSrPVZElYe1Tbm9LFNapgOtebsjPAMAN84ZUDKDNF9eT63ynA6beodMBrndkuckCnVrfY6psZLD0CkCbRIbHm8uVCW5QT1i7+fkzzZwnrwuxIfVizd104TiAek6JjNCtmr4tUZqSCWFQ6PTq7dAmzlc9KPrVRb8d09hTiIFWEZ1eN0ylyLC6eEIkidiJer+qF0Ln6+ZXykBnwmbDlUnCgg3VQ0KKjNj7mqfOl5E+53KcSM1WHvdkaQMhWvKmaL0DtEwQv4orrEoIdg0j2IyhXWP+ebOoJUpTZXOB2cb5HG+qXSJVq6zjZxYz9qKQlaCtEKPppwvWbAFk8gJIhkUjtyhG6XSxbv6bu9UFfrP+NyUPQHF+ZVxyAZtCZALRhTgoN/WsxHdUWNYNmV0LXBpUlXVZlF9EZfkigKghB5fK45MazwOQUt/Tn+xa8JkV8uz7WZwk13nJgMefV6tgIdq6kDdvLmj6eaELBGZwGZA1w8Zju/lb0hOA+J6hdO8bcfetAOjv0m9t+l4Dy1Pq8H9Nv4/H2G+kQXxWdVHDi3oChLkuPJldNUI+ATXtQjNlUYBRWbiWemIWMCLVhT/XZSRZeDpVwEezKG0X5CGpDYMtL7a5Xl7gt6llaAUIDEhRrHTIGxAB0MKF3Ws5djRBVA22lGVd9PIimBfu5wteTrz4vCSX0lo2t+1c8/H3S7IqIasNAGSNRSsDUpYVTFfwc+Et1IIZZt09z7epsyywPldXAIAlqto/7mu7IF/vz1a2BsBmkQSgWoX5e2vtbtmB23HmjYK6gdAsxtv8bT/Ox7eyCzdtZ0DFmoHqDMA9tzBv28B9OpeuOc/bjt2lNp7ff16Yt795fL5dwPBC/JLMjxTkhnnOeNwuyOtCueQ7X0TxOLQLKh7P5wAQl8sszikLhAbg8Xi1qQV656mV6dGKyK6UHBHT9plSMj9pPxNsUZu47Zflj5Qk2aAt66+o50Jc29ber3Mwduk+PPd884wRJV8LyNpz7QKHx50BBPcVeLo4b8+t7Mbb8/ydXY8pb+lHse4x03HLYFzbU8FWGZtmDDnPpTZQudv62DLO4KQCiLydhy0L9HPH2/K4XW19zKarSlxzW9emXe2CvfSjLZctphB5/Qys34sL8rrgXctBc4/ypXNn+WoR51aKs8lSQYhAIbZpxqll3OXLZK4WxO+UBBqm3rw5ziRS7Ti21n1mrH7iy3d+rpbZgKe2/+d/z9Olc1k0hC/5yTjX69p+PkE7F+q7cO/SRdD3tI76kctt8KG41D48Y4271P5Lcy9fOP3c/Dqr8/yWfStw83V9ee8yvkXeZ645f3yevUaUZwrN3Hsm/Z2AwW+R91uN4W9T+r62+z3S9xpYxixxHwfch6ECQwKCBj5LDI3lxGcJIxKmSBY5trywFegYugoOGawNitxPj8FuYiBX6wrt/BH4ibWsU7AY84KQVLW0ACsoZVARksRYYghZdoSBppYRp0BblG0cH1tU6LjcWGE4v5YUFznqBSFJ0vkscZshKwS5gpY5mgqy2boCoOljqmAPWIEYxyi213EbGVhUi2TUNTa0XYz7pCrg5TFhYK5lQqcCpmCQlICLNFU7FaClwBTME7DFlpN2ccfXtQtcvlcAqjA7uy7PoVhsm7hJBgDVCtIIj7fHGVxwfgOajz5u3XXZXdgn+eRFz2W7sD6aIUnYAlpMYfMFtkCLgRVbrtgt2Je2KJEhC8HSEhVEktXCyeCLx4n73IJTTiqLeo7rjyV+lMeEGYfZYpKba4yKWIKGbsaJiZY43rNth48SRiX4KGF1rKCNJWhapmVuK2ukcn4GXEqufVpaoWvgCRBrwRmDLGZHBrAhTNKK3H25zW3ZSor6zPAPuVYEHrmsFrC1YEuIXOYYuZhv4yfjE0DlG0CbS4wnbyIwMFtBc6huw6rEsa51r5JGQhDY4r5noLq2cmJgxGWlJJELmOLvQqwAjfMz+RNbbKrFpbkHWeYKzKgOKpPbhbP21OsKYGIgyXnqfCx1c7vZpZmPr2Aw1mupzKdAkvrVAj7UcssIPQWyDZA4B4CpxGLSAptc+GpZZ8e5DTkVa1SUFeSBLZrlc8qCPPVKfr4WDVDMKN6A+WxTMTf5eRxxZqQBAH5XcH4Kk137jfUismCJi59b4JRr3+m7EI38DYPKCuSwybs53rxs89lGzbdJLQAUm0ehoI/cgHlu2xMw1rSPL70EWp4McElMFsVutwx2W1dMnN0/BlHngEE0ZZ6ndtzaMp+grPdIl4DbRaD6zOf22HlZ7ecNiF0B5Tm2O29+O32+MV1s14XNkWfG6FuBsfOp+tyYibP8z9Tx9YDywrV/36Dy/yfpd66wv6XJZ4Vfuhvc+QGhAElfwNwcTQVWUuT6ncFXrzyWqFdg6Tt0OmApi36OcwxJ4n4ZMGj63Lp7MfHL49KhU6ECppO36LWvDKwcY8kuYQyiYpKYTWHtjLq6iEmRn7C3nrt8te5bWlCM2GHp6jEfFYKVlcHVaV1j5UzjznXyplp2TAE7LcuqlmnDvDp5U8cHWF2/fAEqrfWJr2EgwSkUyZalsMS6cn4JCrbEsWmV0OuA42LQG1oExyQwlFi6R2er29V5/CqPYcoCS4mDYxACoMqeSEEgoHXBvJ96YkEtFh3O0y6yFqOKZeqpxZKvA8jVKyZZgU1KtEBfgoJWK0hqF3GLobqmxWysfV4Hig/SEbPXNQ6xBSMtsGGiGQYkqhDULEHBlZjW9vrzxfK5JevcCitFrtdTf1IFbMHQvWXXtJRXC5GSqeiTMuBDtRKxaxewBQShxDOmFOr9bEEMsMa8MUgAgMDjz/FFciXaofJErbO1gp23ha/nWLP2MwEJXZlcAZKg4PgmVVhX+RdYFpKVyjp7Zn1qAUgsMW2pSIFUvUgAyUqwSx270fllfZ3HErsYvKqucimSRQlZQAwZzmnkRPGMLaDjuMqcRY11jCYUyRgCL+06geUWKF41VmKhUOZhjSss+SswLLGN/AxwWSy/wxZJLkuU+6v0KtuhdESKqi7kq0teGUchco0dhMg1TtUvut6jTd5FQagMpUj6BQDColYLX2pIwATFBjIQohhL0gDltrZzoOqbJhStUJINYfdBXtelVGLyyjyQBQhy+XXO1DatwCUG2VjO1vNSFwmWUk8tv4AMjqfLUZKrYhTIah0XBuvJy9I/ieTlSlQk85ZZmN+NSUCUTY5cGIDB87GMNYplsd4PBo3818u1rbI0OTZ9Z/IhlSkmMa/fsy9xgazNSegX8E+9Bd43ZZakyNiyhAoBBIEs+VxG1U1lvMH/RUHkRAz4CuHSExDa3NuNlZQXpVKs8Xt8rL6LzoAOt6O14p4BS5acoLKxxhvytbLtD5VDshR0TDCwLhqnPC6i3KdWS3WDcFpw2wDZ2vzGCsxltXGUFdM1Y13LzajhjheBWmrKFXgehMqt5utF8FV3ibgeBv1n5eFpMZduycXdm/OLN6D6uXat4wB8A8h+ruzz9BxwbcbvIvA835R4D4B10WL8u/Rbm77XwNLKgB90d7g1J/isqkurzxQ/xXGPbeyUMxqpMLm2bqyTnVcLGAiYMCPrqH2NcWOiFZ9UjbXba1ctoFSWqe6m12a+4IpJrpwpE1Mr0MZx5WoxJQkSeui4bc/FFGqRKusrWwx7FapltFcBc6S+W8WusAI7o6tFsrVM8nlio03V7XZnyAW1dTllltk2bo76JJ+4ybbf279sbVIywVtyhe10oH8qVFBvioXWqLixTp67wjEoi1ZugBvHkkYraxtzFhgLwAcI/LQxkAxeKysqM4OeucLyta1llGPBVitY2lgwuU3cZgb9tgHiDN7ZssWsss+5wvoyngz4Afp9GIyHVwTeldi6M7cMw+dtamO7ePOALaQA0Jk1HstoslhavSWsasFoO56tu+7GXVWslk7ehNAqEjtsFtW6yZ+NWt1L2w2EmCSiYSv5yiYbtKrWTtqI4H5jYz3NWSBqBtXrpkELuM8Zcdt7IUtd7XHWQOR72JbXzlUiRSFWW7Y68pjzRk27CcBAGEAlg/J63RRp30JGRaAvFsCyUVRj/2QCiPOJ7pOOxARciFouxYHxxgmf5/a0bp6ty6kQGamQhrTnyMqz9o1BKPeNQTgAJM2AeK3zfCOE5WhSkWZZY+LCxmJJeYk1V8q8qbdto27iw6RMiHLrNp2UqOy3Gwttqbv2NROQkorHAxuLZVbrClfW2DtAyvV4BWJyHVcaxIwstxtuQiagy2s8nhBPVoAZErm8W5Js3geixHhlAWFpPIRMgFnnNE2qtK5Hq7UU1aqXZWEo5vlt43ou5U0+AqvlHWiZ8hPrAyCJXXcFOgUcMABhwMv6h2ad/YLj/75jErLILjT3oI6hkJv7UwERDyUDX4mtiSznxtrYVtZ8b+ti66jAGr9XwM8TZCTyCmyY8OYc0ZSxpRi8vB4XbR5e2TfAFNhI4GSUPoi8+vQDK9GOLMC87WIuRDwZ5BFQ5hsyu0xzO8p8FuVBOVtX5dLGS92vOS+NbTPPs2iGtsU+PD7tPXgPQFSrOtvHaG97dem+cP6iefESID//y1/P8j6JsbwENJ9DvDgb8nMkfKGsJ+U+KeTC9wuHuJxvHPJvcU9+0+l7rPT4jel7DSyNiPiBfYc5GcyZfuU4jpHDzH2JO2RLYYLYEOAAqPGJFYyUuEv+fq2fgsPz+EIpcgW30dD1nJhAiIHvpZikS2mnls33dOEtw0CXPwOti6TcgkLN7qlbUpJ2fM7H5enYbetY6962dTtWW7KX51JL6sJ1j3rZAFY+HvTlcs7H89LYXsrD95fB+Qr6nra9dUV+0tem7fWYFlvQbbZtuxTLytZwYEvoAqBKsLRxnpdAIVDxQQUxSqQa/wtgs/i81I9Lc+5rU5E3eZ/53bb3PNb0vB9Gpg1oP99Q4L48cS9p5Gjaspkw5dLYX2oXp/Pv7W8qH933q7yQAJAbwNnWp3kRf6Eersvq4ureAMnzGFA+bhFxntiSfOkeKBmeHFvPNXGXmst9Wn4LxKHi+hnYeEZcSlmmy/fsQtIyXbi369+vm2NKpaqVysnaLbDgsq1dZV2YTVcWAqIm99oufV4O/SU33PU4f36urxuvC5Vr39b7vdb9dWPGMZStLNLahvWYqGQizaZcszEh21UPgzagWrxx9tzkLDZrW7Z6icaiJ86aLBpCk5ovN+6ljHnR1lMbuJ5rQQeDOH4oGay1i90kNnV/65RRAOWFBXED7oGyUVLb2LT/CfpBA/LPij5/wdTjDWCsZTafa3sL0C737Xlr29l1Tzp31uZn8zV9+bp+VCBSQCXPw/M6LrX3vH8XmvqNqckj8oW/52XkX09dz557j3Kaqf6r1fcNt/li379Dee83Nu+3xnivsi6W/x2v+136zul7DSyRiV31Ss0wKeJNIAKWG3XCKB0+8y/QS4/7OOCFPuKNv8KNPiFliTd+j1EtMCLiMfZ4bR5xH0bSucwa92GocZd/OHyJX7obDMpX9lUAmKJFgsAH9hGnSOyqx9jhI/uAX7gbGJGw0w5IIO3L8ss6RYO9chiUx6fzLQACkS5puFRYbZPC6+5YGSSPoYOWEXvlyCJbLIyHMECJjGOweN0dAQCHYHGtHf7t8QWu7YwbM+Hz6RovuhOsDLj3Q42B+7A/YCoEOfd+wJV2G2KZY7CYo64xfju9wEWNh8IsylIu12bGwXewKtRFXq8CTsHgVXfCFA0OnmQueuVxChZXxuF+6atldacX3C89buyMKRq8m0d8MBzw1o1Vj/OtGzEFg4/GRxx8h175xhKqELKscZUAcGOnEne4ape+cyMxS2aBazvDyoC/fPchhMj4/et3OAWLo7c1LpQsx7LGrt67cWPlbJk/D0tXpWGOi4WSCXuzYCnxpIelw946nLzF3rrVShup7cfFYgkKNwMxdhoV0amAd/MAU1yAb4cZMUm4qNAVbcyYZLHqClxbV8u66oiS3wWNd8cBvfW47kkGJRS3Z6tDBVouqgriOI9uLFqT15vYRSkTdtZj8gYhSozdgoepx/UwY/YazCDJGxHzYjB2S5WUyVlgPqzXkwAAIABJREFUZxecgiZWy9ImtgwO1uMwd7jqHd6dBgzWV8upCxq2xKTOXiNEhc6EavV8OPWwOmLsluoWHaKCDwr7ochGyAQXdI2LJPdlcoGei7TIYD1yFnBBoSsgi92ZAVR20sfDAAC4vT7V9i9B1+tTJgvhcbJAFtiNrs5dH0kewjQu0tNscLOfcf84YBgWaLUCnMfDUF1CtU5YnMZudHWzZ3Lkyjr2C+bFQMpUdEPp+uNDj901bZj5oOAXkpFIScDPGmORsghBorMB02yqnIUxsVrH2KU3JVFlNZSOJBXSeSzOFL3N1eq4OIMUBWwfqtwFWzFDkFUGQ2tyIWXJCnL5zVhcYYi2Ad5pKEMamikJijEEqianv+8g+ghlCFhGR5IKZrdUF1pm4wxHA3PlEL1CmjT03iMcDdQYKrAKk6bVXXGFFF3aSjeohMRSFKxduUiIIVa5CCEzhMpIRZoiT+V9xSDI5FWyQmTkWaHKH6gCJlQGXIPaVIZwErmP5Appi7tnBqAztWEMwJ0lF1HWu2QsU+QvoBPErJBNglgkkASkBzFxqozc03F5koj7SJ8DIKJA7BPkIkhn0GYg0l85lY3conspHY1V3EWIommZdYZYBGlBDvSZ5R1SV9CTALUJQLYJ6ihruyAA4QW0ow7FLkM6gTgm6JNEtBnSU5lqEvA3391iqSaJOBSvAy+AoqcoEhD7DOFpLJUrepZiterpSVS9ROlB1maJ2q6qJVnqEh7V3TYrgDFqLt/No0AsGsdZAspR2VVyAoCIdEwAkE4gdRlyoeuzKH0ASNuy0TkUgfJITxbFrAE1cxuLjmQCksmQy1oGBKgvqilHUz41C8S+AMlyPQSoPYLyCdZCVavlUITSrlyOc74y7q2OZbU2tthTUX7+C6zjw/mzLHnK2LUyG1kAMqz34glAFmtbsqDv3F6RaPyBbTsZp9d6uIzSltq2dAaquL7GTZnbjqYPbZnVIt30/xLg2rSp3QjI2Owr1LGT67lNugTm2g2C8/PNOZHJI+DS3yep3RR5rt7fwpSBf9AxliJ/j+2xV3/2cf4P/8V/gcelq9qIFEem4QJp/3Hc3rwY9NbjVLQDx26B87osrCNOjjQGOTbMqFhj+x5PPawJNVaMFyK8qD3NXV00GbXWBQDzYmCKxaK6UJXFHOnP0QKO62ULoNERJ2drTBdr9p0TZ/CuulERk7PImY55r3A1OtL9W3RdYKZECzaO2aSFYCrHY43TEpLiXnRZRHL8lfequooBQAgKxsSyIEwbQo4YaffdLxpSbRd/Ssf1Wq8qGQcvSqXMsJ2HcwbWBgSvkJKE7TxpA566TTxWdYUrC1TeHfeLBjMvZgDIJMzO8V3B00Lz6vaEDOD42JcYq1QXqawrxnWYjuJpY6C4rar1VeK/OO7P2oCURNVvo5g16p9SaY2HYuZOQYtlpRJc0TjMGchBQlkCFcYGLLOGkA2BTGEirCLzi4JQJCrPcWJSZwyjg3MawenaR3Yb5Nit6pZT4pu4ffwDI8siWkhAKuprWhSkJctaWBR0FxCchiwuTymKSiyhdEL0kuLLAi1sqz5bJk01ZGovx3QpmxCdguoDxXWlspjTqcZySUPucxwnBgC6D0hRkD4byEIiivtkcGXVUhbpslieUpDVLVEaaksqOnFCr/FeHAOGvB43Ow8hM5aHbrWo6IS8lBjLsuBQXdH1m9cNkEoIEkVdLKg+Ih4LwJl0BRoAoMZAfc0UjydNQpr0Oo5dsXBOanUl9LK2y14tWB4t1VeATC4gT/ce/kDgFzoBTkEMgTT6bKpadVQBAFNWFl6Sfl2Qa785biyVvgmQO6VKSLMmgJSxxpJx/Joqf7lts6K2RwFRgEZ2EqJLyAUAQeRSnih1ZZjrBcFpyiMAYSOBuoOhtgE1Jk3uQj0uu4h0NJA7j3RiVXhA9M2qVIDK5Xsm8grqAKDE5gobqf1sJePnoegcir5oG2bQ3POyth9ZAF1cx4djI1M5Xt5pNC5xHacgt/V1EZgU5JVHToLujSrtlbmOLYIsoLSMs8hr/GUSwEzzSdpY71+rg1hdPRdJYxLEk/knyvc8lTaI0kadVsCs09oHv1qyGKAJL5B7Gj9kArZZ57WuRZJr7qwIbHtZ+yuGABwMvmvKNkHMBeA2LraQBdwbAsQVvJf2AUDuEsUyBopl5WdDREHn2nsssLK4oimDraFRII8RcGVuB4FsU9V55M2ILNe4xmxW4FvjHksfCNgXciiRyWoeUWJBqV3ZpDXGUa5lZ1PKCwSyeRMBQI2rpPZlCMfopAFYpoDNIKr7sIhnAEYU0BMByfn4HAMcjretqBrrWMlyXyroKvVwjHRCBdYMDtebjq2OJdbicymK29eC/9rHuMVVnP+SJbqWx19k8xln152DzDYm9sI1DD43xvJLoPCs70/SBSD9TaDuvJ4sLtT93MmK/n91MPZv/vv/9n/POf/jX7mgXyF1f/TD/KP/6b/5tZf7b//Lf/ob7xvwPbdY3ugJ/+Sjf4mfzK+RssCH9hE+K3y5XOFuGfB74zscQ4eddnjj9vioe8Bn7gYA8El3j7swwkWNnXb4xXSDj/tHvF3ImnVjJrzQJxxih3/18DE+GR7gCikQkfoovO4OAICfHl/i2k5IWeLWTPj56RY/Gu+QIPDFvMeVcRu3Ui0SHkOHg+/wR/s3AIC3y44sojLCyoBRLvjp9LKytF5pR1a/0EGL1YXupsSGXusJP5teICSJGzvjfunx51ef48vlCl+5EX+4+wqfzreYg8Gr7ohBebik8bPjLfbGYUkat/aEt26HlFf9vVs7VUZcALhbBvTKF2IkIkp6YSd85UbsjcMcTY3BPIQOo17w5bTHziwY9YIlKhx9h5tuwjs34lV/xJ0b0KtQrZV3bkCvPX443uOnxxf4oD/gwfdwUeOj4RHXZsZfP7zGoH1lo10KGym7smpB1pkv5z1CktiZhVgwIfCqP+Jh6SFFxoPrsUSFf/zBzwAA//e7T9Brj9tuwsPSQ8uEB9dvYgM/GA6QIuPOkYVqDqZq693YCY8L5f9keMAUDd66EVZFuKBx0034at7hxlL/dbGYmeIW+dHwiJ12+NnxBQCKGTx4i092Dzj4Dh8Oj/j0eAstEnbG4RQsXNSwMtb4zy9PO/Q64FV/xBcnsuLvrcMf7b/Cm2WHXxxusLeushUvSW1Yd5nwia2lLura9xs74xgsBu3RK485GtzNA277CVpEvJ13+Hj3gF8er6tF1yeFyRsYFev9ve5mPC5dIWKivzELjIbuH1uU712PV8MJX552+Gg84M4NCMU6OxiPhVl9rSNyqWDqsd+7usM7N+LREalVrwN67bE3Dr88XiMDdUOq0wESGUdvwVIYV5Y2fe7mAUomjMbj0XUQImM0HnOgcdnZBcfF4o9uvoIUGf/v2w+r1XRnF9xNPWQB/UZFvBrIc+Cz43X9PbaFIGsucyFlgY/GA356f4s/uH2Lz47XRXqE3iM/urrDKVjELOGCxnU34/PDVZ2nt8MEiYwvjztc965Ym021ev7piy/wb+4+gPMa+27BaBbMgUidPhkf8Fd3r2vf3p0GvN4fcTf1uO4djoutFtmcBUbrK6HWy3HCYbHY2wV304B956rWI1uFr3uHQXu8KfOUNwNTBvqyuWd1wMlZ3Awzeu3xdhrJuuw1Xu1OSFng3WnAi3HCw9xX63lfCJucp426P3v5JT6frvAw0zN520/QMuEXD9cYePOvWNY/uXrA3757gX3v8Go44Wd3t/jhzT0+e7yq8b6v90ekLKqr+oPrqxVfiYzJa9wOcx1vITKurcOb04jBUF9DVFiCwlXvMHmNj/YHig0u7ucPS4e5WPRTBm6HuRlDXbwJJF6OEwBylV+iqs+JEhmnxdT+LUHjup/xxeMef/zyK8xR43HpKnuzKQzcR29xWgxejBMeXYcX/QQhMvbG4eA7xCzx5rDDVe/w4fiIz47X6HXAoD2sDHjnRgzFW+PRdURk5yxuSzsf5g5WrxqpXxz25IEAknkajYdREW9PA3aWnq9eByJUK94M3KeTs3i5O2GJCs7reu6mp7LvpgHX/YwvDzu82p1wcETOF5LEh7sDfvLuxbddbtR03TvcnQbkDPTW1/huqyLeHkcM1sMFhevewQWNmERlbH4xTvBJ4uQs9sVzxAfyDHkxTohZYFpMjfu2OlQCv2khMLwyZit8cvOAr44jeQ4EjethxrQYhEKmx4Rys9fIWeBqmHGYO2qjLzwPRU/05Oz/x96bx+yWpPdBv1rO8i7f+y137Xt7enq62z0ej+3xzNhjbGcSEyUREGMECThyiAgCLGQp/JEEmSBQEEnEIiRAQkoUyxIQRZgQGSWAiKIEr5Ez8diDx56tp9377dt3+dZ3O0st/FH11KlT73m/5U63p3uYku793nNO1VNPLadOPfUsv3BADDgf9dYH2tOGo1UCo6KB0l10fOkP0sdFC22cRYq1DDvjygVIswxFplC1EnWdYTKqsVy79ZisGKxlKPPWBStrpQsAZhiU6gKWCWFDkLC2FVCtO5wl3+VgdaB4DxKHDgnDYa52AaQYEMy66Z5uOXjWHXyGxLpDYx1HE7fdYTPB2nB/sELBqCgytVbc+zp7YTg+IOJ96SrQA/pm2yQr8y7YFdVrfdCwEPAqiiJNbYBlXRCrIYmOhLcQXTi6JnkuFW5JiE+fU574HtEhwTVWGW/wgr6wSgK2TZ5/O70v0wdaY/n0d+/aP/d3fginahzgRlzgHo7aZJjIOmA7UjCfhXaLWsEVlBUBj3KpcydsxbAWTKM2EktVeHB5FmBLCM6EAOcLoXq0yD/S+W66hToGpCezzR1ZhyBCcT7JDJYqDz5uFEwm9beT3rwz4xpLghvxeXPuNqgOtN6BxTdaopRtMHclE1FlRRAUAAT/ujzxw4rB5o1loVyjJSTXPTD5xrhIt2RKSzS3Be8hYYYgEkrR9qL1kmkoZx3wO33gyKSUEvVRgBuJVrwYX7Ixji7BjZCwE+NNxlifNDeADqM09pGLoUxGsg0mrpQvDVYUlweAUraBL8B9uJThKPzHPfMCahwIJy7PmIcb8UIMwZYQfIuyXniJ+s6Na+eraWyHH8mjOQugFwGYUuxr2xqOQmjUngfAf388jdg3l4IApXAsFBjIAj1T4VzoEMGW7vcCzvj+iiMma29aS31D9BvfL+QjSAJXHEiI7rWaB8EwDhxEOJD0mza9tAF0fHX4ksQDbUTbKKpwGpUWcBu7tRcQGiV6eJpxECM3bl30YuN5d3WIEGyKytNmjzaWmdAh+A5jLrhQ3UpYP170nsbjFpvxxIGx4nw0z8NeILLYIEEhjURMwX/oHkUWpnExhgV/yRjOJPZtdP3o+jSXzjIjjtAKOEuLNHiP8NFnOTcuym4re5GAyWw2tJvbEBmYxja2aOkF+dkCrULWFLHfbEyT+IqxNONIwmFOhijJLAr0Y3u/tRLIchXyp6kHgxJFaSWLFYruy/wGP45SnF4b4yPQRnPb+o0qT+a6ha83yUcb2xRuBICHVyHYj64exhHu0YabrEkI45IJF8H2SZPDwaSISd06SDAogfcI+iPMT4qaG/iJBQjTyxv6NbxAiZrJdG3rCQTp2JJGC+i0Wj31mf8bRzylOuI6081+ep9+x21HVOYqWicaQ3hNdfS9YZb1o8IOlUufbetDy/rawW2at4FnW7VtcX5fZWxW28830CfRdfwolsPOrTvlYwtfg8+xVZG6XSN5SV4uzfMVaF45L4CX/5O/8E3X6hXP3bV3/9q7r7F89Se/rbH8xpMFBCxO2jFay3GnOIWxDEdqgkfVFM9PH2GtM8xkhcftFHdHNeaVC2eyNz7G47rAWmc4yFc4rCe4Vc5x3Djt2UTWuJnPMdclXptfw9OTEyx1jko52BIDhoN8BWUE7jW72MvXMGCY5ivcX8+wO3GntI/qqY8Ma0JAnYIrLHSBZVvgzugUygg81hNMuAoC7UxWeEVd99oXjv1ihUYJ78fYBSS5US7AYbGXrXBcj51GrlhiUU/wsZ138NZ6HwtV4MPjI6fR9ALxRDRoLcfLZzewm6+DUHTWlAFKpdbOhHYiGqx1Bsk1Hqxm2MmroLFcKIHdrMIjPQ1wLhQBdqU4yqzFqhojF2vkQqHSmaMrV5g3E9waz3FYTcKJ+XW5xEk7QiEV7o5PcVyPcSNb4MiMsdYSN0YL7GVrfOX0Fka8xVo5/zENJzzlXIW/xnLcW+5CG45x1kB7wfxWucJJ7eo4WxWoW4lPX3sDnFn8+sOPIOcaB8UKJ83Iwb6s854gvFd4bVA1BYd1PqqyRWMF9ssVjiunXbk7PsG8LV37pMPk3C9WeLSeYjdf46iauAilXkiWXlu0Iyu8snBa+NYI1FoELevd8SleWxxAMoOd3Pm1VsppA522W+Od5Q7yrMWN0QJvzfdg4QTWT+zfw5urfbwx38c0r0PU4UpnQVNH2KGxxrI1wkHCcIODcol5W2Iknda6UhkeryfYK9coRYt3ljM8PT3BW4s9TL3Gb62yoJUlLeqBH4NMaJz6gwQYjrEP/jPJHG+H6zFujJZ4Z7mD25MzHFaToJGc5A1WrTshjzWWlRcaP7xzjMNqguNqBM4sSqkwzWvMsgpvzPeDxnKctSiEO7WfN0UQnq+NnGbs8cqN005e42g9Ri4Mpr5u0uqdVSVe3HuEnCt8/sGHgvC4U9Q4XI5DMJxcaNyZnkIyg9fn++HjTQcHyyZHLlXQuH798Aae2z3Eq2cHAYYHAF7Ye4yztoQyHJXKcFAu8ebZfjhsuDFx1hTvzHew631yl02G3Aukn7rxFr7w+C6qJsOsrDErKizbHJIZPD05wZePb0FpgVlZ4fFigls7czxaTnAwXuO0KoPG0hiOSdGgkAon6xI3JkssmgKzosLj1QS7pdO2tYYHn9Xr4xUmWY37yxkKb5LfeDihWFO1qHPsj9fYyWq8vZg5c/8mw+2dOYxleLiY4uZ0gaP1GEpzCO4iSWvLULcSglv8wK038Mr8WtA6XxutMJYNXjq80dPkCm7w7O4Rvvb4JnbKGk9PT/DVw5t48dojvHJyDdowGMPx1OwMxjKMZQPJDR6udrBqM+TC+5u3Ga6PnVZz3hQQzOK6fw8neQNrGSolUbcSe+M11m2GZ2bHTgvvHbhOqhGWTYbM+8TenCwCVBTBQzVa4M70DICzOFi1OZ6enuCN+b4fixEmeQPGLNZthpuTBV4/3scnbt3DSuVhza21RCEUxrLBaTPCvC5wezrH4XqMpyZn4MxiP1/huBmj0RL35ru4Nl7iw9MjvHR6E7v+WzCRDe6vZ85qRWU4qsaBj1vTOQDgcD1GITRujufgzOKVk2vYLatgKTArKuRc48FqilleY+n90B8tJ+DMYcxOC3dge7IucWd2hkplWLUZWq8Bvu6tSd5ZznBzPMfrp/v40OwUj9cTzIoKqzbH87PH+MLDu0+87bg5XeD+mbN42ClrlP59LYTCg/kOJkWDdStxY7LEqs3Reg06ANyZnWGtsqARrrVE7bWut3fmwUKFDuVGsg3R4JeNW6NJAK9biRcOHuPthbPCIu33aV2GKNoUHfyscpYW18crHK3HmJUVlk2OVnPMyhqcWRwtx2HdBxB84ce5g1pbe212rZ2vuhQahdA4q4qwxpyuS1gAN6ZLVMrN2XHWYtnkWPr3+Wg5BoDgcmQMx07p/MMXXqttrNO0ExRVLrWLks2AWglU6xxZ7g54pTDRIaAIh0Mxjm3TOE1o2woIHyGaDmLokEm1AtK75gipe8GmjGEoihZNI6ODL/fMwQB5rGTvVkS4soy5yNS19wsPh1kR3ixFUCaJkXPrIJ3osE1EJu+ehonchuhQUHgcXuE1ltZ2Ya9CBHnSxg5IeXTYYUgbSgc7uguoFdrsD5/o0IfFBwmUEo1lDzvWH9Q6mqwvuA/9DVpUonkV6frb6fc7faA1lk99fN/+5N/+owFeJI5aCqB3L9WUxBFRgc2IqzGMAwlK2yKMkranX55HebqIospyZ6bp64sjb6b0t0WLHEq9SJV08u01btS+NOor1T/U/pTuUPTMy/AVp/OclYciYm7r80Bv4F6w0Ij6bojvjeielmAnOniN81JKP+Xz4vK9aOzhOoazGKKXRgO9KKpm/Hxbvm3RWElbufVgmLQE2HLIm/Aa30t5Se/FWqChSKhp+VRjddG9y6SUh/PybKtvuEynsTqPL8pD2rdt9VL589rZ03wA2z/M6fG4vzc0ljEvPc0jzrdS2vbeps9J4xX4uYTGI+6HYJLGbG+TE0zSbFp2E7qITM02mBvqJyTPBnjaxnOaztVU0fVFm7GhjjZbxp3qift5QKuZBgMJmqiNfP3588Rpm4Yl1sJFY8sC1qO7piAzPRrkh/Zu8GSTv7G53pBWLe7/9B20w1AZ0ePNa+8z1+MN2DQjZP3fwTcOfXbouldXr8+TTP439Xvwj+ytNa5s0NylvF2QWJxvoH2D03lIyNlS51b5JNEaXqqeof5PWUmWskvJR9vqTt+Nc+pNebh0PU9K6wnoXVlWfAJ+v/xfvz98LO/+1fdAY/mn/3+isWSMCQCfB3DPWvtjjLGPAPh5ANcA/CaAP2OtbRhjBYD/GcCnARwC+Alr7Wvn0S5Zi2eKIxzIBSqT4UG7C8EMrss5St7i9fo6Mq6w0CX25RL3mz3cys5QmQzHPgKsYAanaoSb+RmO20mIunqsxjhtR5iIBs+NHuGN+gC70mmqFrpAxjQOW+fPcis/w8rk7lTP5LiRz/FOPYNgFgfZEmeq7AmareUhKuxrq2vgzGDmzXaV5WiNwEpluFEukHkbiqXOkXMVsDMpzZXzPay0xI1yAcEsztoy+OlNsxq7WYUH1Q728hUKrnHalmi8ieid8SnO2jKY0u5mziTUmdKqcFItowipjZFovM9ILnSIztoYiZyrgBnqosLmuFYssdQ5Gm+Wl3NXZpavcdY4f0plOMaywVkzCvkPqwlulAscVhNMshqlUHhUTdFqgZvjOVbK9UmMwUkms+R7uZs7zXGls4DJOfftNZZhr1hDMoOvHN4EAHxk7wiVzkJ0V2sZJlkNZUXA6zyrS38S607PCSOUosISHMjReoxcKudH5s1Y122GSdZg2eZOKwcW8DiV5ThZl2i1wMF4HUxDC6FwUo1cX3u/M+21PHkUFVZ7QXy3qFCpLESQBYBKSRzNJxgVDXZHFSolQ/RR0qRxrwmhKLDanwBnQofv1bqhIFfu1Fpwg3HeYlnnrk+KBvN1iZ1RFXzqBHf06dR7XDRY1TlG3q+mkCp8E8l0V3lTs9LT3ilrnK1LlJkKJ/GV920DnM+VhddaSze2FHRr7LUcjRJolISKosJKodEo4fxvDUeRtUHAqJoMjLmosHSCTgG7Gq+BJd+mMlM4PhsDlmF/dwltOhqTsglms9pwrNc5rGUoig6WhEwlKagWA7BaFZiOVzidjzEaNcgEmb8yLH3wKsacqWTbSIzGddePjYTRHNNJheW6cLQzFQJLLU9LjGcVhDBoGtcnzoQTLirsrAJjQNMIlKMG61WBvFBoahkCepHA1LbOFyrLFeraWxAogbxQPthXxycQR4Vtw4k7ne5r7+PlTuRdu4xikIWG9afyzToDGJAVCm0lIXMdgh8Z3dGzhqGdF8BIg+cuQA5FhZXTxgXPAoIPk5lLiJ0WumWwtYCYtlArCT5W/oQd0Gt/IOmjwKLQwbfJUkApH0wFFNm15UCpXVAeHxE2BC+SxgXEiTf+mXFBjGjX2VCYSrhAKoa5oDmVcCaCDP6au4A2FBiIzD2lAasF7FiBz6WL8Jn7QCYUkETDR4VFCEDDKxekibcMRlqAA2bkIsGKNYceG7DWB1lRDKa0YMpNQOPpm9yAV9wHdLGAZhA1Ayygp648jKub++AxujS+TheJ0xT+EIADjKKPFgZyIWCkdZE6uSvPa3rurvXIRVs1uQ9aIwC5+gajwlYMykdi5a0XZi1CVFiKPst9VFgKcgMGiJXDKLUhKizzUWMBuYaLaioR6MURPimya9AAcSBbOF4o8qisGExme9E6mQZM1vGuCwvRMBjhxjSO6GoFgsAe2uED5VjKyxzfFI3VZDENN20p+i0YwJTLY6UNUWkB12/MvZbgrW9jhhC11AoEQYmrrj2W+3wKIfor93SozlTopP4godrV3z1jBjDC0aG/sTkumJ+LfX1EqINobESFBUJU3sAP5R8SLKnOJCpsLDSGciaiE0e0jQ40guzv79H4bhwaxG1JDxXSw6S4HEe/n3q0+jc3IrvGjy8piHe0Linofzt9U9J7rrFkjP15AN8PYOYFy78D4BestT/PGPsbAH7bWvvXGWM/DeB7rbX/PmPsTwH4V621P3Ee7Q9998z+9P/6BwA42BHCosyYA6avTAYBE+6TH6bzleyAw41lyJhGbbLgb0haTcl1zzeSkvHmKQB6wW2CRjLKn2oDSXNK+JmELakiXz7ykezq29QqEfwFXZNPJdVHkCIEt5F738BYE0gbfsojozqBTaxLybRre6JxJZ/NVBtLAqqxPECYxO0h89L0WhnC3Ox8EskfD0AIcLJN+0g8xf6UqbaLhDrOLFZtDsas95XlG3yl/ojWC31AH9sw9tGL/f+CFtn7BpK/W6oNa327Bet8kWKtuLYs+EcOHVgCzi/OWNbzfbRA8F1NtbKxpjZtT6otJP5jf0RqD/klNkoEMz5KcZ1DOIzx2IX5ZjtfRvpLWl7HS3cd+xpSIt9GCnhD2mBtePBBjOsd8vGjuuJyNuk7GsvWCzXk00jjRWMQa//j+ROPXfx9NfFc4f1VRHlBNOaTR/OM/A7JB5F4pbxkMkX3tA9+E/vrkdaMMYTnxvMyNGYxfWNibWs3ZvTM8bv5nK5DhFSLnj/iUPlY80vlQ7u84ErmZtY4Gw0yF3MVu52LofH3eTiVKYNxAAAgAElEQVTr/E47/jp/S1g4E7FE5ZNqrq1B5/sX36diiWaQgn7EzwfflaRc7BcZAnsAQdPEhIFpXYTinn+fdbwz7nat1gczsZpeLhYEW4oQ2zNNs8xvhjuaPT9D8t2jCMBEV9ouAirQbVCF7Qf6SAOJUB4VCdzEZ7wxp7JJ0BASqJ84bVt443qpHho36haKlMptT7tqmQ3PtvrADWzqmRf+Qt2a9fuExof8Sw0LdfeEDpBQ1s3lWIAJ9W+BxujRAAJkCIAwf0LE1QgKJQiApPHsT+mu6TSu1PeRsBML2+f2XVpnIrAMalQpPwlyaf50GqXzv/9zo+wwjXPaET8f+rslxTzEfGzQ3ZaSZ+f2wWXoPUG6sjaTUlLuK//l+0Fj+bS981fefY3la//mf/xNbxvwHmssGWNPA/jjAP4agD/PGGMA/jCAn/RZ/icA/xmAvw7gX/G/AeDvAvgfGGPMXiD5VibDR4qHqGyOr61uo+AKt/JTHMgFfmPxHHakw4e8mZ/hpeVtvDh5B7XJ8OrqOvayFWaywlvVPp6b3sPLq1u4li2x0AUe1Dt4uN7BXrHGZ3Zfw5cWd3CQL1FwhTNVouAKr68OoAzHp/bexNu1w6M8XI/xPbtv40tnT0Fyg+fGj/GgnqH1K65gFpXOcLNYYC9b4XOHzyLjGtfLBZTlWLUuyue8LvDC3mMUXDnfxvUMs3yNmawDLQA4a0ZojMBZXeI79h4BAB6sd3BrNMdLhzewP17jqfEZ3jjbx53pKUqh8KDaQaUkGICPH9zHYT0BYVbeGs2D9nIiG7y13Av+cS5S5RyL1mntMqExki0eLqe4Pl4GbV2sLZzXBW5MF3hUTbFWTmu4X67wYLWDW+M53prv4eZ4jrkqcWs0x73lLp7dOcJhPcFbp7t4du8IDxb7OBitUGY17s93ULcZnrt2iJPKRRdd+6istZZYe783F0HT4OZ0gVxonNUlCuEwDh/4SJmtFrgzPcVENvjVt18AAHzX0/cxbyROViPslHXwtau1xEi2qJXE48UEAEIUv2nWuLZxg6PVCGMfxOXxyRRZprE7WaNuJcZFg5PlCPuTNU7XJfbHa4dzmLkgP40WODqdwDQCu/sOk1QK46IbzifIpcbZfIRr+wu0SgRYm0zqEFnQGIabswXWdYHTRYlru47OfF1i+XACsdPi+v4cyzr3/iIOG7D0+JDkx5dLjVo5fxUpnf+YNhyrVYGibNE0wm3aucVsusZiWUIrjulOhbPjMXb2V6jrDNY6nMXSRyGsVjkmOxVWyxJF2UBrjp1xHaJ6rioXXKttBRiA6aTC2XyE3dkKJycTlOMGQhjkUmG5LpDnCoJZLJalE2wzFSB4Fo8nEJMWs+naRQptJNpawlQSo/110BBWTeZgaQxQjNqgPayXOcAtxtMaxjDUVYai9MGVatmDhinKBuv7U8AA02fOoLy2tlrlmM7WMMZBBWnF0Z4WgAGy/boT2BUH4w5uRrUCjBu0JyXGN5ZYvzOFPKiQ5SoEdmkfjxycgbAOL3Elke9XMJqDC+MgTywwOlhjPS9AMCKGtNT3SjR3HM16UTjIAg9VIE8k2qcqMGGhVxLFrEZ9NIKYNdDzDHyiAnwPZxbtKnOaqImCPsuAUoOtJNisgVl6WA/uYFasBcwiA6s5sNs67Z5vA+MWZi0dbEYjwEcO/oOvOcyuCpAaOHEaS8xa4DSDnWinKTTMQVV4SAurGYq3crS7Bmbi8B3FgoMphuZmC7b2Gr/MuvE4lGhvt2ArAXnG0d5qkb+TobnmcQ+EhTjOwKzTaDADqImFyY2DUzAOc1GeOtxJXTrIBdKQcY8BaHLrcA7nAqY0yE487JDfeOuRhSmtg+pgFtmcO00J85pAr53KT71w4DU42RlDO3PP1chhBsK6MtmCob6uMX5TwmSuDqYA6zU/onaaL1NYZGccamSRzZ1wIWpA507b0ewZZGcc2RJodp0GSrRO49ROELAZdek0PGpikZ26Ha2aOG2eXAGwQH1gIdcMME7TJSpXpt1xvJvM0526Nlnh8BGZBdqJRXHstJomR8CBzNxyh3YCZCug2QXyU0CNunYUxxbr21skmEukbO7abpnTfhLOIG+Bes9pHk0GZAtXH1inscrP3G9dAHIFh+/o/+Vn8POj01LxCMdSF35z7YU4kwGjRxbNjLt5UzjedOnKxDS0C2AOuQDUuOPR9al7psZe++i1g6J216JB0BLKpYWVLGj1mLLQJYOo/OHbiPk+sqHNvAV0wWEKIFtYtFMvcGkEDbdcO22pzhm4cn1rJQuaRtFYDyXi+kyVzGuLLbhyPLsyTqsYsB9t11dcea2vf3eDdtEHNKK2Bm0oaRytoylrC51FwmKkFQ4aS227d1m7yrmyMBkLgiuzFkawnqbREfX0jIUVdADhysfCZozHSdimVjDw1nY4lQw9GBAS6I1k4NoGvjeS54k0wL2Dg5gH33cx5ueg4Br5AWwI01ujVPWvHQ8s9G1M60Ih/P2cPki8XjG9pxpLxtjfBfBfANgB8BcB/FkA/9Ra+4J//iEA/7e19rsZY78L4F+w1r7ln/0egB+01j7eRn/0wh374n/777iIieS07E/faYPlwkSb4JCtlNuwkuMzbY5oM0kn3pybcOqrWhHw+1zD3J8A6u1xGAGAccAo1mH4tQ7TDYB7aW0Xdc9qBpGbQV8gxi2M4sGngUJI2+iE31oE/D3GXV3WIpwqi0LDKB5w7qzyz3n0gkbYdoxbhylIbTRwm774dNvjnoX33rIO0y9eNSzrfFkIl45Oj70pl8OpM93Js2Ydjh1DZ84l3cYPxj1nwsD6DeRgCOv4RFOx7sTWj0HvpJxw0Mbuy2QrjzmY5omTsB2/QH9FJQBz+Hos+thzMXbckA+T34SjZR1dPw5uJ2MCPl7o52RxJcyu0Mfw4+Dx6Xq4e3EdlM/Cn2jb4WsaV18+aACoT4Td0ETQSbkDcO5M8MJJPeWlLqU5SXkJK41wxCw6XoAO347mHeCw0TxGHNEOPLRUETpTNV9f8M8imoS3KTo8uLhu0hRYj6/I6uhdIN5D+5xQEZv1xe0O2iHrMON4wx0mXMP77m9eaIHfaDpTQt69fn79Icw6GgNKpjRgNQ8bjjBvGXpmkL2+V5HpJCXrNzd+HK20QZtF+YPPFY2LdO9ooDekaWLdxskKZ8ZIWgOTOZq9jSK93gQi7+vSIwPW8m6zxRHMJoN2hTbqhXXCn0Bntle6v8H8kCAtSYPX9v3imN/4urmEYOZIAhflYcaZl5IpoZ8aoVzsi0dtinl188P27hvZ1cNizZVxeXnNXH+QCSJpkKLNNVeeLxpL/87QGPHGm1rGbfLvD287QYDqp7kD+PHm3dwMY8BsWB8s70wwg3bNv3MM3WaZzAsDCL3fAId5rzozTiNpLvp+yr3Q/YTJCjiBCOibXnJ3P5gnktkhEM19BFPEoLnz/0iA3DBlTMeehAXrBFfyjiFzzo2NPuvKBoEkEhZCvbpfjvor8Onndixk2Og9ZRZBSApjg64umlO87QSNMMej7wzdC+sCurqonWn7ehrM6H2kNKiJDGtu10fx9qVX3kZ9Fif69kZ89u77dvbKpXyck4Z4OS/fkHY08JDOiy28pLTO1RResh1XzTukYb2Ql0umL/037xON5X/+Hmgs/8y3uMaSMfZjAB5aa3+TMfaj7yLdnwLwUwCQ3ZiF6F7uGQC462Am5mei8L5Y5O/jIl8ZtxFl1j23DDxyhKfIWoIEQ9F/ay3gzLT8aTkJYCRUWgsPHu1+Oz8dv2iGUPEI12FzBXiwcm8+EpnTBEwjdGsYAAdCz5zAB+v6wigeftvecxJGbI9eTwADnD+P74fQvzxqm2+XNcmKFvqI6NhoV9DRCP1JbRfRfaATwOJF2wDWdsJwx2/yJbbUHmI27qxIkKH2NPR19XmD0Ig+D3Rvm11JnE9FPNH92LxsKCmOzVU1qk/FE5TGsV8vCUkb+FMN7+ffqKP7F0y4og8iswzW2PC316nGEWGWwWqXd/ODxRzgNhCEk43AGjZ6zvq/e+DUSZt7wg7da5N7tHEh/5mob1JhKdyzUR6Vtrf/u1dfJLD3PuSWhU3pNlMpCy+IeR85EirpOQAnvCK617KwqQL6vLJ4LvtEgiOADlQ8zm/7ZZlim30S8sMvBlEfaN+GqPHhNYl5U0mgj9hC2W9U0zp5dChAvlkdPSrr8og17811Rv9Z1tvsMQvwqtvMEl983eePNUS+D1PQ26vRxtuvkSD/MN3lD+2wgIjv9+a151mj1/4wbaM5ywCwtivOejn998QCYsX7T5L3AAAE0aUDPvTfAxHDfwRi/TFyfdPNhV5qovK9sqxrQHiX+nWF75vt90t4h5r4nfJj2PTrJ1/OJ07n7PYvNDlsBgoln7BzN+DJePEGvXnDo/4B+usKXV8oKCRray/VybOBvIz4wObahpol+aO9QjLvY3obPG5JW9t2XruSOfRuCFLb+v9KfF0mXbHcpQSzK7bx3aL3DZdJUurj+b5KFhcGePwgp/fSFPZHAPw4Y+xfAlACmAH47wHsMcaktVYBeBrAPZ//HoAPAXiLMSYB7MIF8ekla+3fBPA3AeDOx/fsv/zh38VCFcFn0vndcdRGIuM64EMqIxyUgs4C3iThRxZcBTgNCrLDmQl4lY2RG76EnDkfSc4sVioLvoqSm14AFAqwEvs1ks+csiJAPqT+aFQ25if2bSSfP8eDr4v8NQOOpfNtbIwI+Irkt0hpCA8SQPBRin3yYh9H8g+LfQZ7vni+rm4MbOjDmMf4eYpjKZkJZrXGshAAh8Oi1nLQVy/2SzS2A6aO2xD7WrYe53CncNADBKtBvnPkywh06z+B2Qc/xOgZ+TcCXTRhwn0cik4cLy6MudDwgPO1JB6pv8n3UiVjtIFjqQQEd76PKhqvUqoQKl4kfnLxuMf19vzmrPN5jLEcXVkW6iP/SsJ+pOdEi3AdyWcy9lsD+n6CnDlfQilMz8+S2kzzLp6PHa/MYRhqHvyjCTCcMQulRXcwxGxvPlOie9Tfsa91kMNt5zNKYfobJXvvT9q3MY5jPPbkuxnTo3fBWNbzR+RJewn/sOtHPzYBP7GfVwoDpbnz4xREC8E/kWhxbqEUh5Sm52dJiTASiZfYD1MnfqD0nSdLDxOtGb2YDlEfdHlZVAe1jUGIDmMx9q2kwy7im55TXxjvRwigA2H3WI8Ox9LDD0gdQv8DCBiOgVfDNt4VZ3li+76aNvEFpzbSWNquffHfQJfmXLweReue8fSMp21IqKVx5RZGe+xHDG9qYr/R2C+UDkxT4HdLByoUaCTSCvUA2nsHXbaDMtgWITVRl/RosY4WCziWrFuEUxpknREv0txuCrtXSeTLCPQtYcg6J/YJDe2O6o7vUVmLzreU7rGIZ0R/KcWHsnGbz9uwps8TQbV3PfR8UJDsfEnpQNFydwDp2LLdQSND/1BuqI64fQnbjHhM+YxonStEDwqW8WHEFoFpqO3nCbjJ460jclXZ54qC6Nb2n0dj6zhvyz9winAZXs5L34hMGMp+6wpu7/f0ngmW1tq/BOAvAYDXWP5Fa+2fZoz9bwD+JFxk2H8LwN/zRf6+v/51//z/uci/8vBsB3/7H/5BmNzZ2dMpvRUWVroIaGEtE95nQALgAK8Bm/m1XrlobuTXQdHYRO3MZ9qJhfAng8F8yPt7AN53pPH3vK+K9P4turDexKRbsZjuzInkqosKRyYe8H9N4b+ZZGbkzcJcI90frnxUPu3zR3lF5dptpTvZDD4VdOrInC8O9/3mfCGAYNbGXbl4wTU5gv+BZejMfyKb/xDJzvsPUBni2YrO34G3kf+D96mh/KIBVOn9HoQrJyvXP2rc+Z+EQAEWPb8IwLfHRjwBEK2fB749YMD62JVpZi5vMLOz6EWqA/wpMToTtthEhrcIUf1E5fgzedc3vOnaRyZz8QdTVPDRBRE2KzR3LXd9Qc8oSl1v7ljXZq79eOY0T4BmaWFy5vquQc9UKZgQ+jlO5mZkXhXa3nb1ad+nNH7MuPrM2vk0kTbJCgRzKu3bb8iXxXZmU84Mrqtbw/t21a5NtvJz2L8v8LxYBkB1NGic9dq1l1Y54/NwZYHSbewYA6ABQ+PrTdXAAUvaBe/bZBUAzzNTcBp97v17BMCW1p3W73TaLN4C1o8BjRH3/kKijDZO5LPiTTstA3hlwScMcmGhRt6awtPNltaNPWf+HbRgRWeWKWqXkXtfJKBvnpYtHG0jXd7grwM3B9XErUtcWYicQVYWumAQtYXOo40y8+uBtTDSPTfS+UmZnIE31o8RC+ZyvHX16ZKBKQtnbuk3d9r5FzHt/JKIN5O7+qxw98CcP5aorZsTnPl3wvo54CIQ5nMLVXTruvBjqkadCSH5ZMnK+wkq1wfkp6dK107L/Prjx4Dmbuy3ZaV/V1m3ftC62jPHZX6t875bbh1gYZzIZBKse/fpXaJ5Lhrrv0ksXOucde8URdfkgGgtVMmQL5z/msmY8+XykRqdaan/bvgxJr6Ytq5/mfNtE7V19AoWfE2Ztp1/HGdhrtEYAghjwFt3rUoO4X9b/x7BEp1ubde555U7+swCOnP80ftOawdXvi4/B2nOkm+Z5QyiNmgnkeXHFZNoXNsB9y2BtcHMUJWedz8etHZS/wnPs5EMovW+hH6+ytqE9yR836nvaZ5GOyLLGbKlhi5414ete0di/z/33nh6LXp9Qe849Vlsnsu18wWkMaU2hffLdP6+4V2S6LWT5o/be7i+14W3pLKdUBzz0EWFtd13XXf5LWdhnXTf/Ei4Zt2774gQrf58D8IpEA4egh+fNwvv9TXrj0Wa6Bvu6gDInItZAJHPZKrJTO9RfnDWk9difLJ4jbSCBfpcRfcTNum7YAXr6A2kDeH6IkF26EAg0OrftNGJ3BNpE2NeriBcp+mlJyv27qdvRHh+n6f3HG5kIP0MgJ9njP1VAF8A8HP+/s8B+FuMsZcBHAH4UxcR2pmu8Yc++zsAEKA6lHHRVEmTSJq8nCtUOutp8WJ8ypxrVFr2tGO1dkFrJrLBUuUhgihp52oP2UGRRAEXebMQCo3/osR1pKfQkhvUHkSYtBOAm29O29dpC2MtWJyUPxVvtQhAzaQ5XbcSmTBek+QAk0njRfwQoHisMYyjd1K0T0qkgaF1Lo7OSVolWrNIu0Ual5a0AQNaKxdR1IT7xnrweqGhtEDutYhKCwdrIdXG6X6sXSRtSOH7K9UWS9+vhdcyVU0GaxnKvHWwHaaLYJt5jQEi2rGmZigaptOGdhE2Ldy3pvVaH4r0CYRvEABg3QpYwyF9ABrSsNUexLlW7hlF72TMBt/gMEbCOJB4r2ly84Rh2UgIYSCkQRP7h3oaaRRU0iT1gKK1A3QmbQwDetecG6y1ABcOHiLQ81+sNMqu8znuO7eEqKvMOg2Yp7f2UVeJnvGwHHEUVMZt+Na5gDidqbzxmixrWOebHLUTQN+fmA4pvMm2NclvTn3i6OnaSbkUpMYBZzstketPAJZM1Bl41p1WdBqxbpNkWgGea5hGuLzRx9S0IkBskG80y7q1gcwkAxSG9bxThMZauEA4DF2UT+o4xcByt7OymjufZsWD73bPfD60y/uOKxY2Zj0NDEO3W/Em+sF3mqG/QWFRecXdb/JXZuhMNGN/ZaozOT1nDYcVpotISn5bVDe6ulnLYX0AI6qTNdwFSaJE2ndPJ2x+I97jyJ9u4xv5DDIb2ku+0OQ/2u2yu7JARy/WYMW+h64ST08aBO1Z3D7NYDMN7k2oraB3zv11Pmi286kVpvNt1M71wB18ON9e3nIPKeL6hBnyryX61h9SeT6Z7QJ9aDf+oTzxaLwpfTRP4a/dT/LHtn2fTjqwNazre+H6wwk9JDAxf5As3IHjEyauWRAYmQYC0m/kg2g9Pz0PDV8WgD+M9MIDzYdWhIOHsLGP/S1JUIsEEdFkDgoGnXBOB749TWfk+xr823nXb6687c0bOgzv+Oz84ePD0DgATRBgleheRX8Q4g6aRcgTC3dhHsTaZcTCVr++nvD3hBv0ngZtQGjZ0LBFY+EIDNRt+896PKd1IhHi0vLpvYjvRKkfXuXEFn6jfT2t71BKhLatWty0ji30UuH8UlrLqwiOTzL2v/gEZd6TNHxI8a2Qfl8ES2vtLwH4Jf/7FQCfGchTAfjXr0K3MQJvLPcdBh0Jb/Df0UToANAT7kgwig9uUlM0So8xCeUDbAI236v0XbgMwHqcbygoVhqaPg2eFdNdYPNruYpo90zF/PWS5RumaFfnf5Ov4YxseKU5h/babsKEWAAV8j7pC6o+j76j7a7bpnsltrVnsK0DHyZKiRuYp73ZtwGQHYAmDLpQoOs71QgMpt4Rp7tWUf9aC1jFoeqoSDTHtl1v1pMsG9HXR8MJVxpbeIR3o7ORNdo5g6fD3wF6cbnBj7zbyJu0rGXQTeKretWU1OXa7X8rmTwbLmvazeV3Yy+jJGAYzIa/qBdS4r/pnAFg6q7tPdoWsGvZu44FJFtF5Sx3m5KmyzqUNu63OH+MSEA8Z/zJnA5q4CHRHxq+aFPNLOv5dcIiBO1Ky7CKb+SJ/Vm7vCR0bV/DWMRY8FWN71kAjHUbQkq6ny/drPbopBvZdJ5EeZjhXYCS0B+Jj6thXvMS9Q+zYZPM2u6+s0KJhF4T0SIf27D5Z72NNhD5yqZ80tiQ3y6VpfZb1vlaaqo86gd0m2xCz6J6GYOzHNl4Ka+QbFQ+3fRTfWmbtgggvU0+234PwEbgGGbhsCij52EapuamJipk3d+UJ2axuVjZKPCYZX0ePX8segfDpydeXqMARzZ+32306sSvWNxm6guadkPti78jlxVI0H9lWfpjYHlgA8vAubIB21wWetd24N4QvYFt0+DfWGBP+yFeTthmPSydb8m+c/ATme55tmUZ4P1S6Qk2dZcSXL+d3vP0zdBYvmtpJFp8fO8+GiPRGgEVrWZtjL9oWc83MfZJ6+FNBqGz82tUVoBHMzxGkyN/yzQN4QEOPY/TNiGO/DrTutM2bKtriJ8h3MfUX28IN3MbbZ68zcFf0/a2R+nedZDH81Kq8U3vDfGWpqG2X4beFsuRDf5iOhfRHLof4yJSG7ZhTALn76u3fVviexu+XFcYhw2hODqwGC6HDe3qVZMdmLeXzU98Dt0/n8bl+Lps/52X77IWQr3DjaEDmkj7O9TmlIe0H20yx0I+bL67W+9vOziKVS5b0jbMx+HM5zMyhJ26dZzSRaq3EWabO5fLtMFHAU/HjPzSggy1IXywS/XVIP9X2WBdlHeo7nRTuUVwupBu2r4LDho36r+orktP1qsldpXxuMr9q9C5xPwY0m71aLKIeCxc2IFKBwTRrbTTOnr1blkXsF0A38rDUH3npYHnoc5LjtW5n5yhcbhsWy9K70H+S4/ZN0LvijTei/S+FDjfjzy9S+kDLVjuijU+u/MSSt6gtRKHagrBDMa8Rs40DtUUAFDZDCVrMTcldniF1grMTYmMaWRMozYZxrzGyhTImIYGw0oXWJkcGdM4kEvMdYmSt+DMoPVHQyvtggYdyCVaK6AtR20lpqLCqRpDMIOpqLDQJQAEAbG1AhnTKHmLh80OAGAqa1QmC4JvbTLM5BoZ146ukSi4QsZ0qB9wJsCtFVBGYCJrCJiQ91Gzg4IrTGSNk3aEkWiRMY21zgMW5l62Rm2kL5dh5J2QiMelLnrBZkaihTICreUwPvjOWmeYCIcXSkGGAKDgCrWRmMg6CP+cGRRcYakKTGSNpSpQCIXWCIxEE+7XRmKpCsyyCmetww0thMKJx+3cz9eojUDhgxJRYCA6YCCBbCRaSGZQGxECNq21w9ME3OFExjXeWO4DAG6UC2jLsFI5cq5hwIL5NPdlFq3TDJdiU42yUjlK2cJ4GpxZlKKNAjtJ5EKh0TLko2Qsw6It0BqBncypFTkzkNxgpXJIZrBUOXayCsqKQDMOigQAY9mg0RKVlhhLN56NcdijI9kGrNHWuEMTMssmM2EKLkNm1fF1awQEM9De7Jwzi5FssVYOP7SULVZtjnHWBPNwDhtMqGvf7lrL0L9ZpD6gAyEKbFMIhUplgW4hVeC5NSKMI5mlU9AnA4ZFkyPjBiPZhv6plYS2DKVU4MyGAFEUjKnw5uQA0HjTdDIxbyJz89YH6GKsMyE/qwoYw7E3XgfBpTUcuTe3tr4/162EtSzgnQII5uhx8Kd1K7FTNDirCozyFllkVr+sc0hv4ktBk8bejFswi7XXho4yFdpBAcYYs5ivS0zLGoIbtFoEM3VrGepWYlI2zo1ACYzyFqs6R5Ep1K1EJnXvMKn1QYAyqT0OqjPrL3zwJAAhYJC1zMFDGY5cqnCIQmbhSgtIb/5OdLTmKDIVAoU1XiOcSxVwV4XnndpH/bhaFcjyFlI68/G2FTCGYVS23sy/M2mv1jlG4xpKCahWoChb1FWGLFdBwG28tQCZeQupw2GJtc7sXbUeK5QCBRnuggD5AxXOnem58ubMDq6qW0M4Nw4qy88DrTrnutDvzEJ700mqy7QcPHNqKy50h1nKHE5qVig0q8yZiwsbAvFYDwHFhAETFqYRYNLAkGWEQjA3FoV22n5veh3MqA1z8FBkwkowM8IETTqTvi4PpcRK7cpbdAF1NAt4qgGSiczWGZyptWWu7pojwB5R4BzlzWml6Wg1vDPLZgBrGOwoVeldISkGm/s1Szt+YOFMRol3ZsEU72Ii0Dh5s2ebdSbQAQKpcdfBHNQCBN3jJpuvn+RAbiDWziQZsIDwbRO28xsEOjgo4j2YwsLz6ZgjGKRgCuvzBN/gBDYHQIg30aMBp40OZwXeF98KZ75sCH7GdAJXbEobw6p0fpqRDEz3KZ+NeGToCUWxBjVAp0Tlwth4v/r4bydw+vGkPotSLNdvaJojk+JYm9iTMxMBLqXXo0TwWdkAACAASURBVEt500OduD6GTSiWuM54fK8iVCZpsN2p7HwOjfOF8wEetvH0LSycfZDTe4pj+V6nF75nbP/yL3xvEOo0eE/oApyAVJsMmV+5NBhaI2HAUPA2PKNytBkmTSEJWK11G+o4VSZDydsQAXVIC6rBIWBC1FpKcX4SBI1lrh4kkVijNgkY6MhuxFjmhWEOZQSk36Q74VQiYwacmfA7Tqn2lYTNOPplGpWWIuyGcsxpbWmDT79JS0wReIOvaKSBjXmII3eGvosEWqrbhMi21M5+X6Ta1os0oSRwhCi7iUZwm+Y21SjS75QHGpP4PmnB4z410bwjWiQgDWmn43GjKMVpO9I2qEQTmrZliH7ar0Pa9PO07OdpiKitsUaehOS0/tBeZjfaPMRrWi5O27TU52nttqV0rtA3Me3DlL9Umw9sfpvj+c+Z7e0ldDR2NC5pdOchbW7sO03CGPkuU3nn76wHLRgussaI64j7MI0u7Gj0YlL0nsW0qDxdD2mqU61kEMr8X+nbGkf1TfueIgTbiB5FuE3rJiEx1c5vcykYsmQIbU6iBsf9FbtJxBF200i5cbmYL+KHfL+NxxuOfapDZFVG0V77GiUTzVXyhybcaGPdbj5EkSXf42hXHLCX/b04mmzPxSO+H32q4nzxxr2DwfJtsf6eZZ3/MfkFJ1jMAVv6SZLt90/cf72UQikxG0WTRd8HlgSDlE5MI9V8Met8kMl/OBJmAn1GNGz3exuGMtGkNg6l2Gw7Gouenya665CPwfnIDsFLxWOb9ktcj2XbBTm7pYxvUozZyuIyyVjGaWP5TE3W03ptX/5hcZ60zZcQ3jbqGOBvQwAdqm+Az4t42aR5/rdwq6D4LrX1XN6uSPdrf+V9gGP5kaftU3/5z73rdF//t/+jb3rbgA+4xvLeeg//1Vf+GIzh0JoPfmyD/19yP3y4iVi6wESzNpgxpQseug/iNrO8reZX9EHlW57Ryxi/QUMvd/IBDzxZdGHiTfeh7bV1iGbahIGFejANnS7Z6COW1peuikMLH/0mGvThYQMfxrT+be1J6yB+4jDwMX/fSEpppItsfB02D9Gzi3jfWm0UJMQimG2FYCL+efy3x88Qr9vqT/kfKjf00U/7Ia0DA3mH+KE5tG1MLfpjEG+wenUOvA9+/IaWhfQVCnlYF+Bio/2+bYHWNj+h+F60KdqYJ2nfpHtlhi5whk2mYxxwgyV0WP9ZDJK+saFLeO8FHhnaUKVtPW8+p2sF/eSbHT+0vAAIgWB6z9MxomfUz+fN09TzwQ7kG9h4b9uAMuIp6oOhPeHG8NJYDOTbuPC/DY0jvABPRHlEc2BNImzEeA4Jf49HDG9s3If48PktzbG0scTTwLzovYfGBz5Kvmkbz2MhOf37pIkEJGySOXezT+2j6/PmyVC64HtAAhSN0yCJi8YnfWfT+i/g8bz18cpCBra0I13L0rxDbTi3X6O1Je2fgbovzWdEfuP5FfrgKnycly4UxC4YnycV5N6T/FF6X+NVbkvpvuRbLH2gBcvr5QJ/9oXPYWVypzUE6zSOlkGD907a4xN20gKSNnKb5ohSrFEjDRqADa3TthP9bdqYlG7IH9GM+Yp/Uzu3+nKC9fJdJqXaobTfUm1czGPMe5e/w9/kyeqR+o9exa+T/sok6uu2vENal/SQwYBBMNMT8oc0rNu0enEfXcTTkKZpm59eX9uzvT1xSjWU2/rlvLTtO8PQmW5ehd5gHQNtHbpPdQxpVy+rOUt/X+QXGWvTzvOxTelcNljXZTSk22hd1HZgu1x3Hl/b2nGR1u0i/8XzgkSdty8Y8j00kTb2Iv/JoXqtZQHP8qI2bOUzlWDje4F5t1Pd1ndsS7mhPWh8ryfrJjt4CuSU7ry3BVjbGogs4tVGWqbQjhSHsVf+YiFuyBf7UvvDlFa6SF2WxpV2yReUH3oHt5G/hIBzNV6w/fBmSNAaqD/Gmxy6f67g+yTtDHk6xrcJkOnvQeFvaP6dk84V8tL3Y4jAEwieW+s+L53Xtwybc/+Cei8Wmge+P086vpc4fLh0ujD/Fd6/b6ffl/SBFizXOsfvLO5iqfK+qaH3s5PMBJM52mST7xaZi8UCX2xiByAIGbWWASw9roeEmtbDIMRmZpzZ8JvqS4UCYx3MSAzKHgPdtx7EHRheP4xlG6ajcdtyoaEtC2ZvsfBC9TSqaxuAHpj7VYOjxNAjjj+3MY8hReK/xnBIYaAjbXK8meXcBF8r433gRAQ7QqZ88bikKTY5pKexHxv1SZG1oT8AZzqnLUs2qY6/zc1nDOhuwsaXzOpiQHnSqjOGABcS9zOZu8Wad6LryiMyw+vM5Vg0b6i+GMyeMQe7YQzrAdenY9jdG95QxxtRek5mdvRbCBMsCOI6rEUAtO8JLTHG68BmM46QvE3rvqGRB8K9FPZk0IKBzB392FnLwj2CNQk8JX3FPB3ufSmN7nzi6FmX2W4ePCRt6fra9uBr4jEJczAqbyMtbLCE0KwPYO8T9+NAm5O4b53JJFWEni9e8MmL23+OwN4TjGnecBvobnRqb0PMAG474YboxJroILxtCpCAh1vxEDOMdWXDNbr5x7iDV4l/99prESwbemXTjTuZXVI7UtPDuK30NxbALBDDbQRTx7gcmTTa6JrgV9JdJP02rINZSYU+FvEYbVot+rxZ0iaSTyRL2k7JDNCPf6fjT4n4GHrhon4mCJdOk+qF6tg/kSFAvDDLOggXDjA1FOLzcslpQsn5L7rPvCY89tWLeQc2+NvQhqfjnPZBj5FoDtD1FkE+nQoxXToH5ulcpnpZ8jsVvoiGRdBMxpYLoc54rvYKR+s/QZsgKjPQlm3yFBvik2GrFjzUk5bbVsF55TEsvA0KdJc8REhfmwtTOl6B0EB/ntMfFwuhl3h2Af34+krpqvnfp+mDqGi9bPpAC5aVlnjp5AYYXKAN5xPUaRlIYCHBSkUbRMIRjE/n45NwY1i4Htoouzz9Dfy2jeC2k3HGAE1h6ClfLKxws7G5j31jAPdx7mlAmA2bY2N42MgSf4yEqkh4G9RKhAoijL9oA9Rrm0XPv2bonuMV3SY2xVwa+NvbyDKEDSbhEPY2rFQ2PaXnm29vvKmkr9SyHblL6ey5zhV2UnPZeF7EmyLa5MUbyNjvZYC3AIMQYwVSYAoq7wNjhHanm9IYsy828aU2pRsY4o3KD20s4k3B0Jch3aDy5B4AWDYIvTKo3Ug/TvHfbSk26wM6rLa4bbQJpX6hPk37AUnepFzAJES3qTU6d/NWdh0bNrzRtSGzVtGva6O92uXR2reLRUOmonlHZeIQ/5GvVfybCBjFOj4t+lp6DdjMdu8TCRG09iTroLXe1DTkQzcWxr/v6Zyym+9Vnyh6JsBxXjKO6AXkSPqR4CgchmEM1u7XOjEAt6AZjHSg7w4TkjAdu51s8B2Lxsv6uR6mceoyEQsZVJzyxfnRPUtNHEOV0fzYMKWO5n+6vyTeuI7oR30WXj1vIh1jE/b49sFVGM0fS2PUN00lQaNnck3Ch4nGgMYybHw3TVxttE7GwgrXHb4i4OkmPoYOzxK9uc80YDI8cWIaIRhLMFH3bFiBsMZvBFFBvy96QWpY1N/pu4KB+/6aKXQBZYgGw8ZnKSzFQ1iH8VyOyw18noaExbgtQUjV6C3/NOeCab6n32lFKXPCG/FhN2nFguyFaeB92vqNQfI8afu5aUufPbFweA7dreXSOXPONbMWlrFNc9IhWjGZaB7E11fi7YJ0kcB+YX3fTt+09IEWLK/nC/zM8/8A/2TxIpTh+MTkDaxMgS+v7uCN5QF+6OAVPG6neKY4whcXT+MHZq/iN84+goxrfHr6Gl6qbmOuSny4PMTnjp/FJ3bv4WuLW9jL13hh9BDPFQ/xUM3wC/c/iU/tv4kTNcZRM8b1fInWcnxsch8A8CuH34HbozkA4MPlIX796Dn80MEr0Jbji/O7uF2eheA8xrrIqveqPTxY7+BfvPklaDC8vLqFvWyFA7lExjTuZMf41fmLOPUCz93yBCftGG+vZyiFClrPZ8eHKLjCh4vH+NWTF1EbiWfHh7hX7eEnrv8z/NbqWXx9dRN/ZP/L+Nz8ORw1E3zn9B3czM5wrCb4J0fP41Y5x1LleG7yGF+d34IyArN8jbNmhI/OHmBXrLHwiNIvL2/gerHEneIEx+0YZ2qEFyfv4OXVTdwuznDcjkNb71e7uDs6wReP7+LO5BTX8wVO2jEeVVM8MznCK4vr+NjsHby6vIbdrMJbyz08Mz3Gy2fXsV+s8AcOfg+/dPgiPrn3Jl5dXcNJM8Yn997E88UD/P1H34eb5QJHzRgj0WKtM+zIGjO5RsEVpqJGbSV+++RpNEbgernAymu2v3PnAd6q9jASLV4+u4Flk+M/fOEfAgB+9q3PYq9Y46PTB3hrve/yzK8jFxq1kiikwif33kTGNL48fwqSazyupjgoVmiMwAuTR3hjfYCCK/zg7it43O7gq8tb2MvWeFjv4IXJI3x1fgvPTx/j6/MbKH2k27FskHONz+y9ijvZMf7xyXfBWI61zvBgvYMfvPYaHtQzfHrnNfzS8UcxEi1uFnM8rqc4bkaYZjWmssGIN/jiyV3s5mt8z+xt/ObJMzCW4e74BP/GtX+Gz6+ew68evoC74xNcy5YAEOa1e6eWeNxMsFI5prLGTlZh3paotITkBi9OH+KN9T5u5Atcy5Y41SN85ew2PrrzAFNR40vzp/DD+7+HXzt6AU+PT6CMwJkqcNKMMcsqPDs+DO1/fXWAkWjx1nIPE9mg0hJPjc8AAAfZEhocL53dxPftvYXfPHoGP3z9FXxlcRsrlWOtMtwazXHSjKAMx4enRxiJFo/rKU6aETiz+NFrL+Hl9U383tl1MGZxUKxwvVjgQ+URfv3oOSgrsFYZrpVLHOQrcFi8vZ6F6L3fsfMQxjJ85ew2cq7x9PgkzIVb5RyPqikaI3B3fIp7q138idu/hQmv8bNvfhaFf0efGp3h66c3kAmNjGuMZYMf2HsdY97gl4++I7zHu1mF2kg8Wk+xk1dQRuBHrr2M/+vt78GfuPsF/OPH34nTpkThIxH/8Vu/i/vNLlYmx1EzxndOHuBXHr+AUihUWuJ79t4GAHz+8Bk8t3OI2kg8riYYywbGcvx7d38ZP/f2Z3Fcj/Gh6TGeLk9CFOl/fvcr+F8efAYrleNDk2N87eQWPn3tDXzp9Cm8OHuI15bXUHvNvrIcT43PMJE1Xl8c4OO793G/2sWHxsf40ulTeG76GHNVYq0zHFdjMGbxnbsPcDOb4/Mnz+BascJaZzhrSyjDcb1c4KwZYZavcX+1i4/OHuJGPsdvn95FKRSO6jE+sX8PxjL8zskdfGL/Hr52dgtrlSETGgfFEo2ROK7GKKTCT9/9Rfzi/GP4vcV1lELhu6b3sStX+D/e+V7cHZ8CAB7VU0im8ceufxn/+/1P4pnpMX5g9ip+4f4n8a899QX8g0cfd1GttcA/d/1VaMtxPZtDwOIrq6fwsJpiN6tQCIWH1RTft/sWVjoPa8xHxw/wT08+gtvlGZQVOGlGOGlGeGHnER7WO/jR/a9hZYoQKO7l9U28tdrzY8XwvbN7mOvSjWE9xUi0mKsCn9l9DRougvjDZgc/vPMyfuXso9iTK7y0uInb5dytT/UUn5i9iX/04GP4qQ/9Cg71FF9e3cG+XOFYjbEr17iZneGN+hpeXx3gM7uv4SvLp/DpndeQMY3vKN7BK81NLE2BXz56Ed87u4c/OP0q/t7xp/BseYg72TFuyDN8bvU87mQnWJoCv7u8i/1sha8vbuL7d18HZwa/M38ae9kKPzB9FQDwfx5+As+OD1HyFu/Uu/jw6DF2xRqfO30Oz48f4Z1mhtv5Gb44vwvJDFYqwzOTY2RM42vzW/jsta/jYTPD42aKeVvgw+MjfGr6OgDgNxYfwfdN3sA/Ovou/JGDL+P/XT6DZ4ojPG6n+PHdL+C/u/9Hn3jf8Znd1/BrR8/DWI5np4eYihqCGezLJX7t6AV8ZHKI+9Uuvn/3Ndyr98O4GTB8dv/rOFVjvLq+jo9N7uNUj3DYTHHUjPEj+y+jNhnu1XuQ3CBjGreyM5xqtwd4s9qHYNZHKWc4aUf4k9c/j1+dfxQZ07hf7+IHZ6/g1foGjpoJRqJBxjRmssJr62sAgE/tvIEvLp7GC+OHuN/sYt6WeH78CIIZfHV5G8f1GGPZQHKNp8ozPK6neKo8xUIXeFjt4Lt33saxGmOpC0xEjV25xqur63h+/AgGDF9d3IaxDJ/ZexUPmxk0OJ7Oj/FGfYDXVwf4vtlb+K3TDwFwUdVnWYXWCLwwfoi5LnGv2sNBtkRtMhw1Y1RagjOL/XyNkWhQcIUH9Q4erGbYLdbgzIbvfmsFTtoRFm0R9kgFV2gtx1E9wTSrcdaUvn0GI9GiNQKLtkAuFI7rMfaLFY7rMSayQe7XWs4sKpXh1ugMh/UkuAApK0KQPfo+zrIKjRGodObqYQY7WYUH61mIMC+5waItQjmKLk9Wc7lQWCmH011riZ2schG6jfT91rrvc1b7aPEay7bAfrnCWVNilldotAiR47mfM42ROKlGmGRNsFKLLem04ciERq2lixyvBXKhsWxzZFz38kqvqKn9+GRcd9Zj6IL6AQhuRsQLWZfFAR9jXobceIi3mNfY0m5b8MXB9LNXeNnfy/QtLPx+oKPC3viua/bH/scfx0TWWOscp20JgpjIucJxMw6LQilbLNoCUw/jcNaUyLkGZwaNkZhmtV+QWjRGolbSvVBC48ZogdN6hEx0EVc5s1i27uXfzSusVQbm65oVFc7qEow5KIbWiGCCC7gXrRAKmdA4XLsNfelD75Ppaqs5doomvJwEbxC/XABQ+Q1e5aEJ6F4pFY5XIxSZQikV5lWBcdGAAaiVhPFait1RhUpJCGaD4BRHrayVDJpfxizKTDkIAH9PcIuqle5+BFnAmEUmjIMuKJpAhzGL3MMS0N8iU2g1Ry41qibDuGjQKIlVnWFnVGNRFSiyFoJbrJsMWnNMygaNEr3IlgQZYS0L5pZFpoJZsWAO1qBus2AaXGYKghscnU5gLcNsZwWlBZpWBogCgkUgyIymlbAWyDI3H+LFrG0FpHQmp20jwbhBljlTXiEMmkYgyzSUEv4+Cyaq1jI0jYTVDFmhfJ8DUmrUlYMJ0EogyxWsBYw31SN/MePNBfNCQWuOthXIc0dHtQLtOgPPNYqyhVIOIgK+77iHrnA0TQehYHjQnFsL6FZASKehNz7aopAO2sAaQEgD1QjI3LWZolkybmANh1YcMtNQrQD32mEhddAQE0+kZRPSQCsOkWmoWoJnznOYcQPjTdAd394k2PcHAKhKgkkDmekwJ6zisJpBjLpNgzEcxsMY8MwEDb1pORhDuGc1A5c2/GZeq2wVd/AMS+nas9MGGlbxHk1rGGwtnCZvpMNJrtVOi86E6cx0awExUdALCVZqV59/b+1KBhgGcAAtc/R8P9rW9SMvVYCNYLKjzZYSdqKcRUDLe9ElWc1hpx5KR3GwXMNWooNuyGz/+Fhxp6XJbAftoJmDn2i502QydFr4Jsofm1YyOE1sKO/oMe1hHLxGjjXcaYEz66ArZKdRDGZ2XsssFgKmsEE7yxoGZhgM9ZXXqgEAX3OYsQZTHKxhMCPj7pUUrtT1DdBpqmxmveato0WwEWSqyVsGU9jOgoE7LSdvOKy04FWkrYHTelnZ9Q1vWNDShCBK3ILXpIZ1GlneuHqYAUzmtK2kReMNgx4ZyAV3xg6SVJjefNN09fKawWYWvGFe6wsY6dqmc3dfNIAunIaMNLwms0EjZrw1uJGAR7CCke4ea512UY8smJ9mlrt6mAF07jVeXhNmJLq+pfwSEBU8jIVvinLtBACTO2gLXVqIymmiubdgEBVDO33yfY9oGHTh55NybWEWgAF06eq1wvUTaTbJckJULPAcoDtoPtQ0bxA2nTzSSFrqB2KdA2INmNzPRQHwxvdHFIiKad+H8HXKSNMZaUqt7GsfmYo0vsC5cCNcuTGgsRANIiuBjjZvvbbYduVDO3154runmTTROycivuJnSRmgW6aM6LSljKBEEo0laekDD9EyF7T9QxbUkTZuSKPLjIUVrMtLa842LV6ssSRLhCgFLaOxsNxpGy1nYNr/NXZY48nQPad7SdowwtrCY2qZsFWbGPfx0P1tZS7SvCZ8XCX91s/9hW965NTi2aft7f/0P3jX6b7x7/7MN71twAdcsJx99Jb9wb/xk1i0efApNNZh1TVKYFw0wUcvFmQAoMhUhKemUbUSmcdNc0KRRi4dntqqztxvw4O/m7XOP5Ixi3WdQ/iNueQmCEvGMrRKQAizERRDaxfJdlw2TqBQApzb4LcphcG6yYI/HQkfpiegdsIN1Wutx1JTHJNRjUZJtK1AkTvBzxgGKU2HWVdlENJ4/zeDljDYmNsEy0z3eFfK+ecJj7OmtcNoI7898h8E4AQC6WjyRFgRshOuHJ6bExS40NDK4bvluULTSGSZhvaCUJYrCGFQVZlrQ+wT502BY/9B5YVAzp2/Iwkyxps2q1YAhmE8q8CYxWpRgHGHT6e1M5023peITHOFjHzpgJ4JsBsnt/sj7DoSgIL/oeLgwmHLdeDp7mMgM4eL1zYyjLHVDCJz4y8zDdU4/LquPx1fZKpsGgFw64QywpAT1gmUrYBqBJiwnR+epwGgh2sHbrtr2uhIGwRKMssm/DzGLHTLIXMd6gj0qUxm/j/23jXmliw9D3rWrar25buc0+dMT0+PexLb47ExTkwEgcgIiEUQEiIogIQESIhYgT+QIBKCEBFB/kVEkECAgJAoMkSKFCEifoEUEBAicQmY4NhYjolnpscz3X1On8t32bsu6/Ly413vqlX17e/0Oe0eexq8pE/f3lWr1q1W1X6f9/K8JU8eybqGmXJU5edIQFTKwCxNBqaN7MKZx8dt5PnZ/IxFXeZiuoAU9JxHzxCgidd+FIkPZZ7rsegMSMmzr6Uyic/ntkjc2A2BgoLZBsZGB1d+8ZSl0r+4ResmQmkgDuJPh/kXXfL3AdBtRDpa6G1AGs0c0wZAb0LOAZhdw11EGuy8ji7HjE6aAR4pBm3ZbdPuPMLRcpuW5rVUxLkKD47Xxmaw2EYGiS7NQFKKADuv5hyEhrh+HdMn1zjitRzN0m2bMOcb1MT9OM6tSFP2d5XchACPq0klLyLf9Ko9AHrv+XnIOQPhEu/p3s5AV/IOtgnUG14Pl0CDgdoErit7RfrOzxtNenYPV7lfl7iu3C9Zw4Vbej6eFK+tuLYDfJ9ql/qsgCnzkr0i/cgcmgSMZuG6zONWgE1QowFtA7cR1OyyLKBf8kjaSjFQ73XKa24JquH1gSH+UzQrEWQfq5zLMbdT+rSJn6fBzPkV83MoSpJa8JYcibWbufIK1PC9LsDYEo8dgPIaZBMrIZo055RMANoEdVimJXuTQg0VBQMMza66OiseLAEx15N1ExfslhUkKjB4L+67KStPZD4yZV2xdteussT3ldpU8mGqoJCalNejQgcKswt4zp9J+R6r1ZrWrqol36XkJI0oCokC5DRVbtEKOgP/JHOXdoIqLsgC/uswkeJanAGvynt9BjDzXJDUwtVW8nGWcdXATJYsvxZUdY5qMJgBdu1ev7zpWIBuabM+X4AtVm3UbuV1/eq65WBRAKii1Vhq0FU9vzX790k3enlFpaX7+HJMal6rKpTlvqqnwPkry/r8K66p3Yfr758GSK7LL/3090C6kd/yZfriH/tDn3m77/+BP/obPjfgc+4Ku7MTvnr+FN/XPcdN7PDBcAGtCF/qXuLC9Pjrt+9iZyZchxaPm1t88/gQv3XHbmHvHx5i70Zcuh7f6c/x1bOn+NbxAR61t7gJHZ6NOzzvt9hvevzku38Dv3zzGBfNUJLUa0X4qD9DIoWfeOfr+HA4h1aEp/0eX7t4gl++fgwNwpd3L9l9ojL3j8HiQXfEhevxc8++BKcTztsBPhocfINInAD9vQcv0OgIqyOeDTvs3FTco6R83O8RksZhbPCDjz8GADzvt7jsevzKx2/hfDvg4eUR33p5ibcvbrCxHh8fdyWR+VfffYqraQOnI67GDm9tjtCK0AeHjfV4mus2lkH7w80Rt1NbXOE2zuPFcYMHuwOOk8Om8ZiChVKEs3bEYWrwhd0tnvdbTJlw57wZcTV2uOx6PLnd48GDHkfv8KDr8fSwxztn17geO3x0dYavfOE5ntzscbk/Yus8Pro+wzRafOmtK9yODTaNny2m0WDwFilphMDWpi++dQWrE27GBo2NMIrw4rBBtx2Qksaj/QEb6/HzX38XlBS+8uWPcZga3PYtdpseIWlcbobiFjJFg+fXWygFnJ8dQaSwbSf0k4PRhOtDh+1mAJHC4WoD3URsdwO8t2hcQN83uLg44vbY4uJiQEganQsIUfN9vNqAJo3uwQCAgeq2nfDiagfXBIy3LXaXPVskJ4umDbAZpMfIioe3Ht2gnxz6Y4P9RQ8FoB8chg93oF3A2YMjhr5BFJCkgXbLCo7gDUwTYLNCIAYD24ai2Jh6B9sGhMlm8JjQ7iaMvQMFg3Y/YnzZobkYESaOezY2wW5YOeCPDs1+wnR0sC0Dt27fIwS26I4DewHEie9ft58w3LTozkYMVy3sNhQr8NA3aDbsgTD1DikyCLUZ+PuXHdQmoD3jOn6ySKNBPGjYC/YGMDbBT5aTvie2ZIo1Mt46QBPczjPLdG+LpTMOFqZjpUsYLOw2ID3pWNnzzrEogGJv0ZyPoJTzpXoNumq4jwfTbGEMLJSbXUDKQJaeN7CPRsSnHdSDCWaTiqIJH7csiGZBnV420JdTUTKkWwaG5nJCvHEs3EhCehDUNzfA2x5mE/j8wQANC976owbxsWdBfzTQew962QD7ANxa0CZCuVlyod6y0m8rIgAAIABJREFUgLOJUDeWhd2DBe0D1MGw0KlQQJm6NdDeIp5FBhc6AwJNs0XU6wIA9KgQ96lYOfVLfvekswj90iJtUwakLNiX9pJC88sb+D0h7iNUUDDPLVQC/MMIfTCM15oEFTXcU4vpER+3N/zZvd/CX/I6kyG4JxycJ5a6sMnWsQAgKaQ2wT1zIA3EloVr2wP+bLZIJUcgR7DXBrEhNB9aHkdOjxE7YstjFp7drSkWp+RQBHx3Y4twnxzgrhX8GVsNY8cWM7Ea2qPF9CBh+0stksnnw2zJUx5ILSE2gLtla6K9zURmA1snSQH+gmBvFdwtMF2whUwHtkSFDX+GAkLHn8OW4G4ZNYQN17M9AAKmS4LpM/iw3JaKQNgBZuRxSxsSa2kGBg9hR2iuFJLla1O2YGbvfvgdYI8G/hxw1wZxk9t0QPOS0L+9luZfv7hbYDrneZoexWKmAuDPeRzJAfaQrXNqthg2NxrJ8nrangFVMnlc1xqk2QIpAElPKJbK2C6FebJA+0zBnzF4TS7fu1YtrG46ALHja+yBEDcKeuTryQBmYNQVN4r3VwaHZsr3YAKg85yOhGRmpYSOhNgomJEHFTsGLs0NlThW7RViA8RWwR0Ifsf7UsXZymZ7KnMX62Wys4JB+/mZS47bMpMoFjiWWtZ5EZudAUlyud3K0ikWSLFUJre06C7WWgNmIh6TFPlYgTkd2ZpImj8Def2dgDQeM5nK4riiHF+A+2yJrIGWjrxn5L8iIBkF46kcL5bbVZvJgttbWUFlIgXAJRQ0Xlt5Z+BMQJ7nSYtlNlgtAOKKha++Zg0aT35fx5SfAJmfBfD89Sqfp7G+aflcWywf/cgj+od+5vcikir+53V6jJD4B61OOj8zxy6Txa/TUNSpOtbpJoRFdn1dnZT8FHV9zRhbl5qhdO0jvmaQXIxz1YZRmYW0moeMoWaqrb9LKSk3ajXRqo36WH3NHWtsqscv4zvdVt3efW2uWUtlne72u1Qzyvt+vaZrJSGdmOerUn6siZ7qdTqdqgF3GCtfZy1OlU9K67Du+3Qbp645fbzuc53GQa471f6r2rvT76sYJEvFE8dOjO2T1gbIyt4Ta/QmytdXjrWM7e76fKrX7Smt9koz/YlNfMK+eu14j09SG5N65ZrwWPj/a63Nqf4+aS6l3iuOr18Ar2zn1CavPst9qCwNC0bQU9edunf3zbVW17/u3F9V6rG+ag8V3cGSgApAlUfyFWtzat+uj6/X475r1m2X76v3NM2Hv2uC26nxr4+vz+Vyysr1qvr3j+ET3pmKinXq5Dq8yctudX6Npxb1PmldXgcMnNqXNNc9OacT+6Cud+ea+z5/0rhOlTd4F7/2nvwuiuX3juEVfd67hz7F/vmk8kbP7adYp1/4d743LJbv/OufvcXym//cb1osf83FqIT3Ns/xNw+PAQBf7K7RR4dn4w5X0wbv7V7gJrS4cAM+Gs7w3u4F3j88gFaE7z97ho9HJil52B7x7cMF3t7eFNKHy+aIx80trsMGv/jibXxxd40hOvTBobXs5vqouwUAfOP6LZy3AxIpnDcDvnN7gS/vXyJB4Wm/x8ZNi0DlznhcTRscvcMPXT6FT0zo0BlfLJTndsQ3Dg/RB8cB6m7AMTS4nVo4MwdSP2iPJbj9/dsHmJLBg/aIq2mDH738AB8O53g27PD9Z8/wrcMlhuDwaHOLrZ0wRIdv3Vxi30zw0eC8HfC837LHV7bOPeh6dMZjiDyO5/0WOzdh50YM0WGMFm91B7wYt9hYj7FKX3I7tdg3I54e9jjvRj4fLY7eFavlo+0BL4YNGhNxPbS43Ax4ftiidQHv7q/w/vUDfHF/g5fDBkOw+NL+GudNj19++Rhb5zEEC6cTfHaFbg0Hx1uVkKDw5LBHTBob5+GzK/PDzRHXYwerE676Dj4a/K53v4FECv/nk3exaTwedD1upxZGJ7zsO2iFwjj89p6Jml4MmxKHKnGYl12Pm6mFUYQv7a9wDA2e97ynjt7hoh3wctjgrB1xNXQwOmEKBs4kGJ3wzu4aWzvh/ZuHbCUjhePY4O2zGxx9gy/urvH+9QNYnbBzEw6+wRRNJofhNj6+3aF1AY+2B3x0uweRwr6d8LXLJ/hoOMO3r8+xazw2NqdYSQa9Z/VyZwOGYOGjhjOp7ANRmjzoetz6Bp0NaE2ATwbPDls82PZoTcDTwx7vnl3h2zcX2LfZmhgN+ondyS/yHjvvBtyMvL6HseH0Lkljm+OEOxtApHDVd3i0P+DJzR7vnF/jeb9FiBo+Guy7EcPkkEjhrBthdcIQLEZvkUjhKw9e4Hm/xVXPsdetC9g6j30z4oObM7ZsZ4uxxBbfjg2MZuXIecfjf3HcwOiEbeNxPfC93bYTjiN7F+zaiT0GHn6MxkT8/EfvlGdg2064PnYcZ6rYxf7x7gCrE75zfV5AsXgE9BOT0BApvHN+jW88e4jvf/QM37664Jji7IL+lQcv2LshaUzR4Kwd8eH1WSFVeLBlYounNzucbUbEpEvbiRR+9PGH+MWP38boLfabsXgXGJ3w7v4Kv/zsMULS2LUTrg4bPD6/xfPDFhfbHjdDWyyyKSnsugnWRNz0HR7ujjhMDruGPRnONwOmHF8tqXwutj22zuPJzR6tC4hJwUeDlDQ2jccYDFobcRwdzrcDts7j6e2uuPu/tT8ikcKLwwZv7Y942XeZtRtonUdKuoQW/OgXPsR3bi9w1fOzJvv0m88fYNvy/h88e1h8+eIK33j2ELtuwtv7G3z92UN85eEL/OrVBcfhJoUvnN8yAVt+dl4MmxJGIfHmD7fs6XDMXgyXmx4f3ezROTYFjd4iRI2zzYjj5PCl82vOw5yF4+uhyx4QCQTg4bbHmFnPR2+hdYKPBo/3bJ6LSePoHb64v8EHN+ewJuJ2aLFpPHufTA4Pd0d8+PIcX/vCEwyRSTzk2bY6YeM8bqcGh7HB4/0BL44bPN4doBThvBlwPXXwyeDJzR4XmwFfPnuJb1w9xK6ZsHMTOuPx5HiGrZv496zfoLEB1/n5BYCXxw2cjXi0PUArwvsvL7FrpzLG826EMxFPbvY460b03mLjAl4eNyV8ZNNO0Aq4Obb4wsUthmAxeIsQDDbthEfbIwDg4+MWD7c9Prg6x9vnN3jZd9i4gCkavHf+Ar/45O1PLXe8tT/i6fUeRMC2m9BkD4nGRDy93mPbjRi9w+W2R589aKbASu7HZweM0eB2aHGRPWF8MJiCKfv6ODFfg1ZAa0PhZzhO/I6WFGNTMPi+By/x5HYPpQij5/t8Ozbw0UArgtEJ1iQcx6Y8e9d9h132solJY9+NUIpwfewKRwAAdI3H6C26xhfegbPtAB+YM8KWtl15x9z2LYiAB2fHEprTNR7HscEwOpzvBlzdsvlUayp8BLtuQkwK/dgUbgPvTQlrkBARpQjeG/jRwlg2oxkzh52EYApPgPRRPHGE3V8TK5xNylwFpoRxaJc4tMNSQTgqv+dcE0poTWG2zxUk1ENlt2FKKKEgxiQOuSEwQ74ipKALy72EfUhRGnO6p5TDQzKA47RWHI4i/yUERLuEFDIPQMrKRAHkOaSFPMfAF4Z/oKDuwsIfdYn1X6RgqpTF5bO4bdcs8LUiQRC+KI3KJLEEpWtlV51OScZ4KrXO57Ws1+P/Y+VzbbH8gR/b0b/1F38YN6nDRHZhlTqmFq32SKThySBCwYD4M2k4HeCThVEJWiXcxg6tCvBkCjtfpz08GQzJcb18vVEJkTQ67TEkV+p7Mui0xzE2cGoGfxEaBksWLp+DCpyKiJk9QixxBgkRemF9XV+XSMPqWOo6FTFmq23Jw5nPhWRgdY4/TRZtDoTgtdEwiuCTKQxqdTGKioUWYJYyq+eAgUQaTkf4ZOB0LLk0AZR+p2QLsJ7Zu1I5Xohv8likjVZH9NHB5f7kuFGEMc5z1blt+cGNpOAysJR6UtbW4ilbtTvjYRThEJoy1rpOfV2ZS14T+ZGt/ydSaDLZU03cVFtP5/+63J/GxHL9nO90Ps+MaSa3JcoKU6zqsq7MuhaLJV8rQmfYpXOI7s4c6/ytsu/WLGuSu3U5tnl/ylxFYJXrT81f1gk4bXWUekSqkCYJ4VIUt3LMeWeFha72NnCac8TK/avr+WQWYzrlhSDKiTr37ZoVrx67MLbKnpO8q2trodEJWjGhlJTaY0D6aW1A71mRFTMIliJEYvU163XWiuCTXjwXqrp+imaRC1f6XZ8L1drXz+cpK7Hkf60Z/tKJ+kZR2SNS0onrZK2EmKveazWhVp1mituSec75e2XtY9KIScFoaSffU5MQcuy10QQfTDlW1rViIpQ213mL1/l1pc+1t4nIUVYvn0WZa8lNW52jqt/6+VwbhU55XKSKWTFV96L2DJF49DpWvq4jpGjrOnz9fG0qz/Ys3Ncps+a2UD4LT8GpHLjSVp2Wq+Sureaqs+JFvCCEfKy28ClNSHH5O/cmpU7ZVec1hqICUu6k5anWloBlHtVK+Ee1TtzXypJfWRopMbgQgHPKs2RxnUIBM/dZzNfeBIt8xnXcJt3zuSbPKmm5sLTkS8x1XrM7bUmpH4v76lRzK5/vmFLnuZ60sEuc8RrknLj+VP+r23yv08EbWajXIGxVd71s9/V9b5v1RfeUO338Gi229/d0or3XqfMKK/iryt/8174HyHu+8l2yWP7zr7ZYKqU6AH8ZQAs2LP4XRPTHP+txfK4tls/9Dv/5B7+rUD/3wUGDikXv4JsiwLmcLkIEMmFCFYFMiHtEaJmigY8aRlOxjBnNIqsITqKBF2sVgEwGFMo5EchqoSpEDZstVEPWQkouzNqttrEBRhMSobhe2vzDKYJTiIbdX3N9YAY6ov0TTb/NMYZsHZhZUyW/p4yrZoWNWVMvP55CTiOxXlqzJlm0gXVeTDnnXCjXiFAiZD/1NUw6JFYZZlgVBlWtmdGVtZgaTeuLlUKKuJjWwobJ2tf6x1nYVIkAm0lfhiNrdJsuZMKdWSARYUV+tGMWSnQtsID7TFV+VCH9EWFCNJ8ifMlxccskgOPriNlJpWidMsERLVhGa9IeyR0qMY1EirWXmbiFEpg0x2QCHSGVyKUm8hEiGxBmIaQs3iyg1UnoU+BxK5tAk2ZyDyG3qdsTbWj+L4Q9pR9hRBUyoUyYo2wCeQ0lmuRKSwxwu7yYNAtCmaFUyR6IuhBlFGIS0cjKckuOUKAQbhTylMxKyuuAWesqx6c8hjbNP3xCZAPMQo8kZxdWV1JzfRHIFBXyGiFMWWhscxwm3wCayVZkHeuxCyFR5Z6pRj3HaMZqbgQm42iqsRmaiX8kT2UtwdQCZG6nkH7IOqH6n8BuehXzaX2uzoOpomIiFMmrqVDIXMhWRCTSRZ4/KXbT1L1CaqjcNxV4rSnHk9bj0qNC6niuOnDMoR6Z7VTqFKIUYai0S8FR2E0B3GHFrHMsCukJdG6zTIAKU2i5V5KzVC3wEceVQfpakqiUHJHE942JXQjTqOaxVQJhMSwYyrk/UQhHhNREYth0BJQHyMm9zHOvGT1zPkfSAFWsokjVdzf3QTrna81rippIxVZjjXnIBoDPoEfWJfJ+UYQSO5oyu23NCqu9Ato3lEarooMqMcYcrzezwqpMWlP60ViQpWiv8jrPsYQcJ8vxffK53Pvq+anzP3JjKLGScr6wtq6BUr0Xc5yh9F2zvhbiHMLMolrnxqyfmQrkrvNYyjUyh2TmvhassrSsv8iJKe8FwjJmUi/r1blE62ekzH313JTrVrhYztVrULexHns9h5PgsbqH6znXeO4UPpdyp90FwK36knfv6r4s6gLze+XE9l/ksqz7OwEa1xj/JKBdH3sN8CltfSJIvq+Pz01Ry03w61dGAD9JRLdKKQfgryil/msi+l8+y04+18CyNQE/sH9arHJ9dtcUC1cfXbYyzlY5sa6JVhxgi1SjI1s/shWE4zA5X1FnwsJqVcdwJihYJWlIdPkesmWx/lxfK1aSobUnLTsxWwJrSxAwa7cFPIrVI0HB5DdmzPMYu5FdYVQqLrwyBhlHmXdlKZA6MqZaAypAWdKf1HXXFora0kQ0x7WaPG6x+txntZA5CpBXQEnHIgqCdRGAL2OoU8TIuvtMIiTtKADHnJakyW44dbymzLlYoJNY0rJwUVlsxBIiygAAixQs8l8UBeuYQLFWiDukXB/kPtNsNZSytoTY3F8ktbBWCUAXFzvKCg+9spjUZR0zOo9//ixKBW5rZr+VOuLCJCk/ZmuHrF/d37LfteJC1kwpFOWGrGdJbVIpAGR8AEodUWLU9+RU3B+PHzAZGBZlAM2f6+MxsJLHubhQcki9YvWJbF8+te61hSJFU9iJdZXyReay/t0vig5U+z0zJ9f3EABip4tSRawv8oueIisnVB6z3O/CYKxXv+Z5ngzSVRmQ1mBriggfSuY139taOCruZdXEhAFY0uEAKEoHXRiVq2cwA0tR1sSOSZFkzGKpUjZVgJjnHTtm06WkEKOCsoTYqdl1rBr7gglWZfkmL0skFEDH2iI1s9WWG53vpQKDdrU8t1AiVDHr9dqkNZlF7T5WC5ByzhBSq2blSF2knsox8iKoAguwDg3ExACObMVqWvepUIiISM/soKXfDMZgaTXG6ngFTBbjrcA5KsAt66QyYQqDJ04zIwCeNymgArGy4VMWBo75M6GYxxnQoqR7UJEq4pL8L3BFEqVJpTDQ4S5jpxJFW32c5utUN6c0gZrnugZXM9MqZsXNChTUQG5x/+dm5jalHlXtAUsAuwJFdToPabAA5zVgXcvcNK8FFFVtqyUIPAWm1u2ugNVJgHkKWM6v10X5JGD5pmUNPOsxLsp6ne4BgcuGcHLu88mqrXVfr2r2def6BmvyRm2+Jmj9niq/AeMldlG9zV9d/vvMR/K5BpY3vsVf/uAHixAfRVAWYXrljlO72KzTdoi1TATH2p3GGEIU60sWCJRCSVlRUjKo2fWmCHPV59IfsjuXopKyYm0FAqH44wNYCFCLXZCtUUSV9Uo0flkgEB988dUXS1I5JwJ2JTTWxxZvORFcKlWVjO0+spK1i4+qBMLix0+48x1ZuCvCqqLZErMWjmj9X+rR3ZdO7dIj7YjVQISd2lJXW5GA5RioPn+in3rNauFvRQd/R4hcCJpV+7XV7J4f4KUkUDUTWNC7o7WsBQq653vdPsAuRSshkq0Ws8Ze3I5q4o8ibFbruz5fr4cIptKuzJ0UVYIG7lifeC3z9/V8k4KXfQGcnnc5rhBK2oaq/RpYyJrkfTHKHuKBLn8hK0EoVHu4CBOVVKEi4LNgHMzdc6TmLpCAeEIAD2JFkrWRbZ8ZQWXtamFGZ9BAec4LTb4IV5WgpiQE6JQwW1tcpP3KknZnj9K67ZwSoQKzWoRtTdAr+nxVvZcUzWkJFoI1sLCulb7FskAo+fkWwqSq1rJ6du48cmsht7IGnxIE63tZrlP3nF8dL2OQ7+X5WAnMeW61lWxRqv1f0i4I8Knf9cIQWaehoNN9lvtQgyPMa1jyf9ZjoNVa0XzvqFr/2qJaL/56ry/2YLVu6dcg+dQWqDsApB77Gvgjj1nVpDrqzrosNW11G6o6psp+nPeOuheUVLfwrjWovGvvznNtAautiXfu+/pBqNuXPbf+/Sq/J6fbLeeqeqTm34jF+s/LcPdYXRZtLT+fWpfS7KvaXF+z/h1Zr0u1H19HrJfclfL5TrdKzcdfRy64t597TnzCGF8bBL5me/de84br9pvlblFKGQD/B4AfBPAfEtH/+ln38VqvV6XUTwD4a0R0UEr90wB+B4B/j4i++VkP6E1KZwJ+4MHHsCphSgZTtMViaHXCrW8XlrQx2tmaGRysTjAqYYwWG+vLMQDw0cBnq8DOTRijXcR+AewuCzDRiFgOI6kSY5ZIoc0xc/X7RCxJViccvVtY8QAUV1chdJHxGJ3uML+GHNMRoi4kAuLCO3hbAuwlvYjRXFdA96bx5bO4wgqzq1ZYxBclUrDZkhaq2C1xJZZj4kYrLrXOxAL8FdgCWFtcdZ6TXGvyGvtg4GxEyG6zdf7PpgnF6re2PNWWNckhmiqrjbjhikJBgdNxgBTaziMlVdVRxTVXwHPIRAxiCauVFpw2I1u1xGW2so6lqKGzNU/qSbtiZaOkSq5MUSaEwOuUkoY2LGkWhtiilOA9Yar8mdIOJYUwGWhD+XwlgWK2qhYZQNaz2reU2ymKgix4mGyBBIEJBbyGyZYfEFuuFtawinygWPTyr7pYoyS3p9YoFrTkNeffzEBCFA6s5OG+VAUYk2eXZ22puA6DkIkO8vpqIFUuhUpXMWfZhVRn1+J7gWVWFqScG1N3YTmPhSssuyiD5lyT/GxV/WelS/Iaqomch9Gmsu4AkOq8iOLOaitJVlLJiNszgAXBwqg5/Ye4t9aKlKDmfI0pt5tJHxBUzls4d7XIQSnWIRlPUQphlj6kv9dwhYW4bMtcZb1lPpKvsWzQ6lxSSFN2D61dYYHsMoq5TQBqUsVFlt2BiY+5ue/itipjyHkMF0oOATwCMuq55jp3XGHVLNiSqeaj5jEDmPPLyfWY26ytQfWYBHSSI+iB980dV1gBSxqcS9CgpJ0QCxwpFAtgsQYmLK1V4gqr57lIOwIkBZwkSwtXWJXdXJOkhMjtVQ4/C7fbNUhWhJJKgmwGkJJ+Qdw/s7tobPCpi6R5KOPBDHCKy7PK6yev2bzeZcxivSvPBorL8xJQzWhnDYKgAC37O58XN+aFEqoGULIWcp/y3gDyOtfX1dZEGaPUVdWca/fVe1xhxX1VUn5wZcx7vN4HK2C5XueFYisfWyi8MJ8r15xw/V6UU6BlDSxXiqiTZdXvou11X+trToGl+r1W5qbu1lW4wwK8WD85JoqzU6BsvWZq+f9k3VNjPtHefeWTAPMry+cdWH53xv9IKfW/V9//FBH9qUW3RBHAjyulLgH8RaXU30pEP3+qMaXU3w3gq0T0Z5VSjwHsiejrnzSI19Xb/UcAfrtS6rcD+MMA/jSA/wzA3/ua139Xys6M+LsufwW3scOQHJyKSFAYkkNIGu2OyXi0okJa00eHRBqt9ojQmJJFk487lTAmC60YwDU5WObGd2hNQCQmUtEqIZHGxkwYk0UfHTbGL9pqdYRWCYfQFrIcACVdiSed2/CZlEcXohwhpJGxhlxPCEdqN0irYwF4fczxmnnHbozHmGy5foz8ucljS6RxrMC0VlTAsgDoZuVKKq6rtYvufa6Z4kocyNxxF250wBBd+b8mv9F5/QvBDxjMdsbD6oTrqSsuynX/Mi4hYxnDXYVA3YcQrZw9GGF1xNW0KS7FAtziSpXrMpFOcW2sfiWEMAZAAds+GViVEEgv/gtxUJ1CpDVhMS52vZ1JNxody3UyD3E3lvnXjICirDA64awZcfBNOVa7CcuYa1KetYuy9FmnewFQ3JqVmuOZp2gWMcziziwMvvJf5T0kQFbcjp0oVCplRWsifHYVljWRcdfxz0Jo01l+Zus10GomjqnJgdZpeMQdOZEqyhtTuSTXJDDWJMSksGv4WRZWR65HRTkjTI82x4BLHLacE1dqubaxEaO3aF3AFMzCJZ3ZVGc3daWYbGZ9PlQxzPX1XeMxTA6RVLl3UoS4BmCFw5RZj0M0sCaWuO5SP++FUyRWwvYLzL+jRlFhNl3LgSrvNVmP2vVfFESy12vyHlHQ1AomYYkNFauxtCesscDsUm1Ngq/cxUWx5au1lxyp4kIuLuC1J4ywWq7j0NcpiGrFVV24zWp9Taqum5Vn9VikHYn/XrtqCzOla8JibFLEtVxcv2NWAMnekn5iYMWYUkCMcx/rORbFCqnC0Elp9vDhm8pJ+FSWXsselLkiv49qU3xWUFBUSMLAmZVG2swSc1HOZMsqJUBpXnfSVHL4fppChooyau0WTlEXS3xRUlUKDyX3UjxX8kQX7KBrRLBSus2d5Zh2URzR7K6+KIqWypjaK4QwWxDFk0DK2rOmuFqvxlcfPwU45HPtrbMuatUuqvFJG5i/Lzxb1iDtPtB0DwgqgOw+4LiexxpwVt9PgbDFsVeM485Fr1FOAcCTQLC0+6o+7+njTeq/xvzqx5w/vwbC+ixA2OcdiL5e+fh1iYmI6KVS6r8H8A8CuAMslVJ/HMDfDuBrAP4s2G32zwH4iU9q+3WBZSAiUkr9IwD+AyL6M0qpn3rNa79rZUgOv3D7Ls5tjz41uPYdAmlcuB6tDvj28bII8J3xeDltce44LcjLaYOtnYpl87LpcetbNJnZ8WrcoA9M+f7u7grfPl6gM2HBunkztQCAB+0RHw97AMwIedH0+Pa0AcDWzimaBZPllAxaE9CZgG9cPwQATocRTbFuTsHivBsKqJI0Hq3JggE4plLG4KPBrmHq9iFYdDbg6e0O28ajswHXQ4tt4+FMRO8dfP6xfrjtMQaez9E7bJ0vAEHSN4gFFGDrrBAbacUCdj85bNuJhbXqLSQkRrtmKu0IhbpQyffeorERUzDl+67xmKLBYWyw70YcxgatjSU1RQgGF7ue0y9oKnGNMakSEyqMg5vGF6Fb6wSjCcexYVBCCtvGw+iEbz55CAXg8vyIEDVGb+FyCogmp76QPobJgUjB5fQBtfA8TjbH2CkMg4PWCW0bEIKBtbHQuXtv0DTSLoOplBTG0SEFjXbD6Qx0tjj3g4MxBD9ZNK0vQp51sQjcIhh2nUcIBn6yaDsPIULyhwami2hajxA0KOkiIBdhMQuOIiiKhVLiImNex5Q0U51rgmsC/GQBAoyLCIOD2zC5kgiMOhP0BG9gm4Do2XpKQBHEgTn+UKyRxkZEb2CbCD9YaMfgUOWYRq3ZjzMGU4RLOeYPDZRLsE2OLY4GKShQ0LAbPqZNQgymCIrazTTrMVsgTcumhZoMKQUWBJVmC612Cc+v2QxizqciGCSvYZo4e0VFhdTqCMbQAAAgAElEQVRbjhvc+RnsSbyoTcVam44WZh8QbxzUNhQLLwCkW5cth8T/Jw29DUWwpYnXXneBLZ4AW0hzP1c3FjgLc92g2cKYFNSgQWeBLeleQ7URt0cL1UVQtnQWbz1FoImp9OESMJqZMKhJTDxUj5MUH0vg82LRFKuuWETFyjfpmUyIuJ7K94XaCDUakEuzYCvum9naNVxbpE3iOqSgRs1kIrtYWUO5a3PQiGcRymvoQSHuEqZbjbhLs/Vl1AvLWGqy5S63RTZbBRUyOZGC9mBSIAEA2cqpJw0yBNNn4J3nkJq8Vhlv6mmO6SMjFhhC6nWxOkET4qgQOypJ5FWcrTVpUojbBHWt50TplXuzuP4mR8DIbtJq5DmJ2zRpIHZ83gwKtCFQALvFRjAhjpA8Sf8O0EMWInM0j87tpi0x0REBMGxJVEkBHTE5kFglWxSBVeonBx6DJraAaUB5wGRyotQweU7YEOyR11QFjke0PeDPPr2UqfM6A4D2swVWRV4fHRj0mkHNxDqSeqLne0kO0BPPkRT/NwPPkQwKgKnBTrIrEKMBewBim++zqdrMREnAfG95vLx2YjmtyaaSRbHsAtnSK5ZVndc/kwPV+yZZPg5w2wBgxvkasZImC5iJrcVqNT895XYr131ZG+lHrONUj5Oqc1iOjQ/kf3aeR00EBFSALFvpyxhWdXQgkDkBs/Izpmhec7YgUln/ZJZWOXFdXbiwVu3JPSeloBItrc51WIJ4BujV/NbDzG3W8z9VCjBduY/fqS/f12Emq7YWl6yB71opsJ5/fX0e0xpv3wugPw/lN2Ds2eroM6jcAPg9AP7EPdV/H4C/DcDPAgARfUcpdfZa/bxOuhGl1P8I4L8B8M8C+HsAPAHwfxHRj71OJ9+t8vBHHtNP/pl/DEN0C+KbRGphJRKryakUAUIHL0BKrCg1oY7sc7G6SB/iliqgQixj4rYKYGFNqi0Ga2tJbQmRtuSYfK/ZYBfU/Rn0xUoTKEAoEVauokuK9FAx4Qpdfl3EMlVIP2hOWSCurrX7a+k/j6G2FNaa77ULKzBr1oXtVUhDBNQAuEOEst6+dXjKmmSmJoQR8hgBZC4DtBDMwnpTWz/WdPmnxlDTtK9p9uX83PdybkrNbce4pKOvY1jv5JJal1plWX6NOGeYgEU5Vi5Jpyn+1/T19X2r69aus3LPF0rmPKaTv1XVGq/HdVIdi9NrMbsE52O1yy640+JGK+6b+XhZz+oHVSwc97n9LNYocf4wALPrKeGuBUHJuHCXcbcsVq5qmAW4sOZWa6AqK1ZZ0EU/eY/Wrsv1GgvRzmrtxL23PlfintMrLCL5XpX46Px9sQ/rdVzf03vWWMZaAChhaWE5VV/6A2bSnNW9XhP3CGhduNbK51St86nY7nX/i41fjVH+06rvdaxpXaeuty4nnvFXWjruE+TWdeS/vue4yn+1ACvfy9BWe3P1+U6M9LreqTHdNxdZr7x3l+M98f6oCYU+RVm4PmN1q9YC+Xr89X6tP8t6n9qrqzYW06lBFJaP3HLQJ/pcX0CnYzRP9Xun7fv2/Il6p8Z3Z8q0+r8eyydZF+85duqRWU94AeJO/BTdCwhe556vxvFG5VVr+6prXqP+K8fzacb6um1/Bn296Vr+33/iX/6NTzfy3vfRO//qv/SZt/vNf+GPfFK6kd8G4GcAZHUc/gIR/fQ9df83IvqdSqmfJaLfoZTaAfifiei3fdI4Xtdi+U8A+CcB/BQRfaiUeg/Av/2a137XymN7g9//xb+CCAVPFofUwiDl3JAKx8TWvIksOjVhoAaNCpjIYkwOWqWSfxJAyUeZSOc2+bvkv3RZlRfz22PM6jk5Lu1IfkGfbMmFGUmX/JfiDmtUwjE1pQ2xaiaoklfy1DWlH8x5NcUV2KhU2G9vQ8txpEjFJdgglXyXERpbPZU8mnVJpBa5MaVolfvMOTIBZsdt8zyljhyTHJ71OfnuVESfmuz+m13eyBQXZHEjFjdmdg9uFq7B69yZkuNSwLLE1PrslivjlSKuxM+nLQDgzI753ppFzKcGweqIkExZE6djcbmVMUjuTgAlX+R9rLfSXq2sCKQxRYutnco953XR2Rrt0Fl/0l1V2hCmX1GmSH/HnKNTxsduxjO7rtw3yZm5vu+SU1PcqGVMwnwcki6uuo2OCxdhWafadXrttlyvT82CLO0O0Zb5SN11/dpNe4y2pB+S+jHpBYNynS+zPg6gPI8127K4Otf3Rq47en6WN9Yv5lPnDE1QxTNh4/zJ+Za8rtFgYz1ufYPOhoWSRrwM5vumC5OwKIjEnbd2VRZX36N36Ozsoi8uxwT2lmhzf5HYLXkIFk12RRYXZxm7MDXLvZJ+xJVYxiTj9ZFdS23FfCznxWW1nse67pSVP84k+KjhTFpcJ+0RKQyeU0zVTM2RFFobiyJO6k7BonUeKfGz09iA0buyf8STo1YoSYw2AUWRVisaxXvC5rjmmrlYYsRDtrzXLq214qZWdklsuFJAkDhahRLfLcnta0WipGlyjj0mAHZ3LWzGFcgSN1h2dc3Mw5VSyNjIFv40p0uiSjEm6yJKNVGgzWPnOGtuS+aMwigsCpqFgkcs3apShuRYbuS5izJEFDAlpZFNS+UMwAqDpkJjb1hIrPvArCCS92WtWKnIxor0K6l/RBEknxXY2pvbWChbpNTHRTkTZtfbEpNcA2ygclWtPtdKLQFLdWyvXFcrEBTYOl3tTZUqYjWgkGwtgLuMTYHjj9cxlpB2cS+qU5WyDsDCslkscGugeQrcVcqduStaWAFf6fJ6H5itr1mBy1NTWus7TrZ3ah7r/tb/q7Hep5ta9HePFuHkmN5kjHW/p08tr3sdsPwaAPLeNf3NAgAgop8DWyFfp/wFpdR/AuBSKfUHAPx+AP/p61z4icAyMwj9eSL63dXg3gfHWP6Glu9Ml/h33//78XLYwEeNjQsgAMexwRQMtq1HiBpd43E7tNi1Ew5jA6UI28ajn1xxdTwOLRoXSoxN6zw2jt0+X95sinshCwn8Rmssp6i4PXYl3sWahGF0aFsPBWAYXbGGiZAhDLQxaJzte6TErpcm55DUiuBsxG3fFotb04RyTf3UNA233diAQ67fNCxAXOx79JPDOFrstyOOQ1OscxLXczx0MJbTI1ib4CfLvwPZ3bFpwiJ+Zpo4H+acCHvub5GfkTC7f44WxiYYmxADk+9YFxEmA9cGBM9xkOLaGXK+xW4zYTg2aLqA4A1SVGg3ntfmtqtyZ85CjM5CmViz/GhZuLIzmYz0qXVCGC0oKlw+Ygbmq5dbKMPunSFwDFiUnIH5R9m1fN/9ZNkCGeb4GONY8FKa0OZcm36yvDZRsavoxOsRM7mMuC8qTWjaAOcCjocOADj/ZNCwLRPytJ3HcGwW7qmSz1FYhONgoUyCbQPC4FjQcwn7/YB+cAiDY5dSSReT9OwKajl+SGJ+SgqJ/LY2ju+htqmksgijgWk5bjCMFm7j4XtXcnGmmMdoiOc9GWghpVEE8rpo6yXnpBDupMnAdAFxsLBdQJxMTkEBTg0RNF/XpGI5pCxwu92E6A2T6igWRpVNMDbCH5tZ0MrEOOzWasqvk2n5uUhDvt5xjk4oFIEVJMcNmvMRWhOGF12Jn1IugfL14vJpu7x/jq78OBdLn5DkALCbgHDj4M4nritWOwDubET0DHIoKpgmIR5sERp1dvVNRwvVprntLJx2lwOeXrXFZVU3sZAjNZ3Hx9fn7Bbb8PjVNoB6w+6wk5mtU8RCutIEGjXUJoK85vZ6y9fn+yWuoKqLDAx6W9xvi1uqo9kd1mugjZwntc9zCxoqz416C7UJoNHMQnp2PxUBu3s44HDd8ZgVQbfc9811w+CCUNJ9mL3H4ckOcAlmE3B7vYc9n3C42RYBTG0CxOqsFBAHM1s2NQCvoLpMrpXXW7cR/WHDY0Neh6g432lQ0NvAAEWA2GRynlIq61XAi6QmSWDX53ycgoLZBvQvG147cUnOgEVvA/zzDvZyQkoK/sDvWMpj1y4hTQZ+0mVN9SaAVCYDiwYxKoTrBqqLsG2APzgol8q7JAyuuHIH2fNe83oACCPvbd1FBsbXDbtPC6hyCcoQ4o0D2shrYAm41WXOyEzGatSgXQCC5ndLVCCXeG8p3i+qi6CD5T05GN5bUcHuPMLLXwN7T5eAa1Yqsxt2Pq4JuLU8p6i4ns9AV4DXNvC5Sc+u4CmPP69Tuff5fVEs5l6YcfK/qIEzDzqyj6yaNGgTeQ96zSBP5XEFBUUK1EbuOz9nKvG6yZrWeXeFqEmIrpRXoJbHLICSTOK+xNXc83NMXX6uKd+zoHh8bYIe9Awi83s/dan0UZjZo6rcYmkGp4HdyynnxkwWxTNBRbXMNylrJS6uJV/mTGAleWZlruwyvQLnlF2RA+Z3QQ0SEyD5aQtozetPmttWcisVzUB6BQAX7eV+Fyy6wNINtiJYKgRV+SdsAY5zaEAhUqomdgejCjiT75ViYQ0EX0WadApgr+d56ni5Zg1cTwHZNSC9B2h/L5bvdfBLRH9SKfV7AFyD4yz/DSL6S69z7eu6wv53AP5RIrr6NY30My7nX3ub/o7/+J8qVr3eOxCArfMwKuFq7GCyNr8xEYfJLcCny3F7o7fYNh7HyRXSi9HbwkZ6vhlwHJsSoxcEZAWLlBR23VTILnw02GTQKoBPyC5mDS5r861JuO3ZqmpzvjrR4Kao0bShCP/rHIRSvLeL+lyX8/6NQwNtIpyLGIcGrglFey2uj20GzKLxdpktMyV9py4ABn45Pk+IGII3hWm0zpunTUKKGq4Jhc1VZ/AXA18TvM2CSwaV3sC6yELMZBfxeEonhImBoNt4pEyqUtaD5tQrImjaZp6PPMnJ6wJcTAbJ41UHEGD3zApbtNwZlKLqI055fmIdEJfJ3LaATBoNA0aJ2zMMopRL5T8IM8lDUhzDlhSQhVOlCcokBkcaHHMmgmYWAop7Yv4hUU0mdMjxccjgQx0NyBFfH9T8w68xx7iJYCG/grVWHZiBT8Ks0XaJjxN/VqNhAUYAqaFZS54FyIWmXQQJETAFxAMzI2mTmMlUUmRomscimvuU5yECw6hZKKoF+pQFiDbNP5ZxXrsSB6gIKgtyRWiKCmTzL2kWJKCoWCZ0z/F78SzOMZNZ4BUtOQjQo14IUwDmXIGytgrQg0baJOij5rqVY4E+6pld0RCUV3N7KvcB7kNNuW07WxbMrUbcJ7Y2eFXizxSB4wv3eW8HhdQk6FEvWVLrH/AsNJHN53NsXGq47ZIkXsC257VMbZrz7lWWjpKyxnJyexVz3CHx+pS5NQQzKiRHJdZImDbFomFvFVLDsYOKOCYOxLFwhSHW8DkzKIQtx+ZpD47P6xViKzeJ4x35fqEIm9KXIjALaY43o4yHVMgxj6s4MIlzM3NILp/PbZYYS49y78txzXFpgKwtx/6llvcPOcxsrJrnHVuCO6gcY1nlmUyqCJzJcWygrC0wC+VQOYZwzHGjLa+3ipjjOkOen5VjXJ+fIx57icdrV2OU+MkGM6NqngtI5pGvdRyTKEy2LLwDRs4LK2zD8X4lplTx97DDpy564nZJzeNRWciODQqwMTneUeYn1woYkvtKerkPamBQx5VJWwXQaMAeOcaSnw1uI4mTm2zbDJzK2CXGUq/W3VVgBfM86pQ92mMGS2nut47ThJpjLEmhsOhSfjYkDrOO9dMhr4tZPic1gCn19VxPETJgyiBOqdNxf3kvFYBF82+OxDGSyWzHeo5rlEJqXrNSTgGtNdjM87zDhCtjPAWAahAlCpW6W5rb/9QxlvXxU22v/98zhnreawvputyxJJ4ChK8odUzqydjUTyq5+l/9c692F/31KO1730df+qOfvSvsN/7Fz2Zu2aD439YGxTcpr+sKewvgryul/hKAgxwkoj/4aTr9rIpW7KJ42faYsnsZwMyal02Po28yUYzDvhnRe4ed41/jwTPBjcsshw83R4S0x76Z0HuHEHWxNj7o+hI36XTCpBjQjN7BmISLbsDLflPSaOzbEUO2wm1cQI8lsKFskdw6j+tDB61ntsKQGQhj0NCbNLuwRY7VlJQiUgSwyXmjE2Js0NiIPihok1kFE1shm0wgwwnPUYhpiitVtpr6mNOCBLNkacypLEICQBwjKgx8dZFUH5QUk+CkHHOnONYvRQaxwbMF1Ccmw4lBo3EBEyx8ZOtqmCyUSvx5cEBO6ZEig1dJIE+JmXZByBYyFFKa5FVJMp8mA6V4bE3DyoVp0iDFqTgoGCQCz4lmchl5j8WYLVgZUGozWymRMlMfstbYJqDhtZI+S85QRVzPECDU8zEL+e2szlOitVZZUN/QDL4MZtCWgaK27Kq6yAnqNQMVnXJ9PQNL0OyelL8Wa0YCn5MXv3xPaqnZTcjab2TNNpbAVfaFz3VCBiikAJ0qd61sBRQNv0PR7i76E0ZF+S/9y7kMApIGSAuAY5CiJ8XCmFyf11yAS6krCcubPPeI8saUdBRA7tckmIHHXLzABHTK9bmunnhdU4s5Fk2EAcPHCCyQpy3BTLwfqJIyzTQTqhC4zdTlaxXNmvUOc9qJMjfADgpxx32qoKC9AhHvIzsoxC3fN9Hwa68QHYOuKOBdljDwvSbDQDC1XI+FZ1WsDSXlglcspNlsyUkE2HyPc56/onkPioXhQjajZgE8Axkl+zPx/kJOm6ASYI8KQVHW5jPpDAMOmoVZMDA2PZO9qAiYIwMxc1RFUCeFco9Lmo2OFVsq8ncNwGaClpRTlwjJjPb8DJNhwVhAsekz8M+CV2wzEM7CnBnUDFAsPzNM+pPvq0YhpQlgkBjVDAyT5XOpYbIXskBsxLLDc9chrzEp2CGnxOp5PAJGGIjweXsEJjBQ0J7BQ+hmsBQ6ASwKNrcTOl432/NcJs0ARGWAbiZeR48MBh23F7MwLmCt7OEjzyXZ+ZzpkQEwA0+/nwlutOf67oaQ3Fryfv3iDsUQCTvkfVcpNQTIyvhIzf/dLQOY1ABmyJ8z6HIHyqBfFWCpAyEZFqLLmAW8GJ7LTBwF2J4QnVoQ0OhIiI3Kzz0htgp6IiTLIMyM3GBs8zObQaL2XEf7/J6xCqbn/TvnM53BOykB3AruSPzcqBlMxkbB9oSw4fnUpDsFcDsslEPyu6PDDC6TZQBfH5tTptASPMnPg50Brq5If8R6ye3yOyEZKnWADGh0Xofqd7IGNwWMVha8Mrf8bNUA7VRu1gVIE0WSmgF0sfBlQCnkStIej31WBNTgsiiOqvmfKvOYsgBES6B9x6VXc517rY6Lxpfn1mD0zhhWwJ7vF80N1H2o5eHPRfkeHisRRaVUUkpdfBqD4utaLP+Zezr/mTft8LMsj37kEf0Df/b3AUCOTTMlBkwrwpRj4eoYrRLnkww0cm7EHP8ohD8JqsRjCUX9OuasxGXRnJYAmMl6amKgNSGOxCZpRfASjwOUOL263SLP52vW36XtmDRsjgWSOKopmAooMrOrXCNpDZxJ1dg5d2Ud4yTWWRlXTQgkReJ4xGJZ54yU/2viIKrWa00etE5lUufFDJlsp7FxQRhE1Vjq2J51DFSd0xJAsQhPU46bLBZbVdxpa1dbyTUp85D4JjlWrLakEAJbvNfxRkuX4dW+SjnOqEpBUFP7i6W3HuM6jYHET9XtyLiFUZVJaU6oGOs3+vo8zSkrKJ9nq3VaMMyWfSAuQyf2QPmPO783i77Xayb0/irPqR6rxGqVtQy6uLhKe2LR1mKhVaeJfLhuHoOef8ROEfoIgE/ZwqnFQpnHKGQ98r2OE1tPfJFHM7K1PHkDZdJybOLWCswEO9WeqcdeFD/VL3mxmCPPp465yjkoF8Q9NZGNtFXfK7mRApBFoVALTdK+WLsroHuvK5WMTe6hgHVRIEgcm8xt3Z5Xy9gxqbPuW/oRJY5Yv+v2gaIsEYWAsH3WlogShyZCWLbCrudWLIZipZcl0NV4MSsGFvMC5nyZsraxWtP12kcFWFZMFeWLeCNQFW+WiW2oIriphWWxJhcrmFiNaBaWeQ71euTzme20KI3sDJ6L4JhBcx3rVlhSsRTMtSh+ZK2TWuRk5DHSnNsxybpny9qnLHW+0NrqpmTeVT/lFarma4sSoXIxpGpu9bOwsBytXCJZKYSZKbhqcyGY1y/Zymonfdc5JBeWI5rnIedLrKP8ZldWb66Tj5d6c5+1la00ceK+LkBHHs8CKCrcYTe9A0JWc7hz7J5yCjzV67AGQuufyoW18NT6nyqvEsFl/qfGVTdR76cTv2V33nWnujp1XMa+Pvdp23tVua+vV5Q37gPAz/373xvkPV/6V74LFss/+NlZY5VS/xU4HvONDYqv9Xolop/J1LTvEdEvfdqBftZlZ0b8nRdfR6s9xuTwPOzQaY9We3TK4yN/gU57XIUNtmbCTexwYXp4MngRttjqqeS43JoJx9iUXJcRGje+w8Z4PG5u8MzvMoFMzsuXc2BGUji3QyF06WODS3fEdeAYub0ZcZvNI0ZRyUXZ6oCNmfDReIaQDC6bnvslzv84JYMHTV/IaercmHWRXJfH4PCgYVWwkN18NJxhayfs7ISnwx6Xub0b35Yck2+1RxxCg9YE3PgWZ24sY7A6FsZdAddCDDPlPIuNCTiGBls7YYp2QfzSmYAhWmytxxBtAcOcxsRhaydc+w5bOxV35SE47N2IY3A4hgbnzYCrcYON9bA64ta3GKPFg/aIQKwcEBIZIXPxFTnPzrKFesokOValkkYmkSopZ751cwmtCG9tjkik0AdXlAytCZlshy3SfXCISWNj/R3SEbGSA8Dt1MAoKqlknIkYgy2EKq0NCxKbkDQOU4MQNfYtj1srKilitCK2vrcjYtIlH6QoLxIxK+vW+dLWrsnzjwY3QwtnIs7aCWPO7Sd9CBsxgALiORepLjkiCUzsIrkMRTnROU4rI/HMx7HBtp0QIlt+jaaSc3L0Fl3jMeY0MzFptDaUfiTnpCg0Ghsx5HQ2t0OL1oXCVDwFA5fBlOR5FHd1ACWtTJvTwvioEaIp45R0OXKcgAUBjhC1dC4gEco9JJpJiOS+ORNxe+xABJzthgIOfTTFKwBgRcrk2YW+a/2Mx3KfQhgFAOPosNuMOPQt2tYv0tpI+hmAlSMhGHStL+1ME8cWb7sJY/aeMCYVwpjjbYvNbirXsjt7Ym+EyaDbToUgpmkChsGVWGpr48JDIQRdvBn8ZIuru2s4NlplshrJx+jz2FzDHgoSLwyg5E8sxDJVTLYoeYLnd491nOZG3NmZGGZ20aekOA6wSbCO30vJsyVfXOkBFAVHHGxJk5NGwzGuvYXpYpFg0mQgeRNBgMpxgeW7zjGSCgz0ieMfi3s6kN3bWRHBsakSg5ulR5sWOQ2pjrmTGC6d08SUB5bYBJhj4ErsqoBMiZ28dSz8W5qBaMrzycdJUsSIK7ikENEAtTmGdmD3bBWym30CqCFOO6IENKriag3i80iVm/Ymze7IGWCrBKQmH89uv+SogJPiWeAIGHIcoUG20qviqizWztQRMMyWMHbTVEiblW/fG5Ticow8nsrlMbU5Rk+zx0RRPNQWb5WtYz4D3hzvp0dVwHsB+wIUUSsx8sOnCSZyShMiPi9tLlxaKbvHYrbaiqsxKcykSyYrDmQvpxVY1WDCsgrYSboXFXhs5GSNZgWJpF8hm71IiqcHCsBlaz5bC9nzhBbAUsdqTGaupyi3EecxQmFmBq8VHWm5LkUZJD9uOq+3AO8TwHANLO/gmqqeAOc7rrBS7wSQuqPTFUBbj2GtHMj1qK6vqnOYj9WW7Lq/O3Nbje0kLlbLea4Lred4Cuiu2pN/C0WAfD/h4vumuPLTANH/H5f/Mv+9cXldi+U/DOBPAmiI6LcqpX4cwE8T0e/9NJ1+VuXyh79Af9+f/sdx4zv4aLB1E2IW/Kec13HMDIdH77BvJtyMLZP3OI8h2MJm2HuLzgUM3sJoQmNiAQQvjhtsGg+fYyBFcG1sgAJwyPGXkvx8mBy6hoMvjiO7pdbpQpzhuE4fDS62DAalXxGaWxNxPbTFetnYiBB1YV2U50tcY1sbcDO0RTj20eByM6D3DmMw2LVTSYguLIkAcNu3xdXT6oSpJEanYhmsrZBTMDAmwWTgIf1NwS6siwCDA2sSxsnCZrKXlHSJ5ZRcjl6IdKpcj8YQusajHx3ahkmVYtToWo/GRtwc25I+o7aAsYVwBkLj6Aox0ZoISSmOD01J48EFK2Subtk12bm4iD0FUN52QpjkhTwlWwY5tyWPXynCpmWAJ8RGMepFLsuQiWJKDKsitC27Zx8HJpdggiReLyZKChgHB3EpLulD8tyVQiYLSnBNwDRy3LExCbvNiNFbjCO7cAs75Z0UJDSzrSpNJbekAko8rNICVFS5b1ozoGlbX/oAWGgVQGOd3N9ULKhMSMU3TCyJvJ5ADKbE3gqhEok1tiJkMjneNVUWQSF9ikHABoMwYxOmkfNuikWxXguxVBvLPkUh309taG4ruwpLDG7yGt1uglKE/mYm79FmtmQir6dxEcYQxt7NVv1sCa1je20T4HuHdjdhGlwhaQKAZuMR87OakuJ45cEVS5Ru2NshjGYmUcpABgC63YTh0IAiE+3MKXiApvUYDhxEpizH95ouIk4apuG51ilFtIssFPp8Puf7jLlvuUeUc+dKf3EyZa61FVfmSYHHpgznFFUKTFTUZVKl0UC3TDokltQC5rJVszsbMY2W565QiGb80RWiKMrxvraNCEcLZQmm4c92GxAGW4RQ3fJcxQKcvOZ5CZgNmutQtpgrlDVUOT5X4qMl9lpIouQhS14viJp0M5P3FMtxVNxPfr6knbJOQoAElP2QjhbmzPPzmGPBC/FXTm0jeUvJ69K+PKuUFGg0UE2EcQlhsFA2cfy7opmMTMBwjoNWMs6JY861xL33NsdM83yUJYdSMOIAACAASURBVAZZQjwkwHeq3r8S7x40E/xk8huxNqtm7ks1kQmeushg2WZg20XOA/tpi8R7A3P8OFAAfLEGC4lPVhCAwMA/Yc4bS2q24AuZVHXvF+Q96xQpBI7Fl/WRPjOp2Aw6aFYy1ORYYpmXGPSQXdNluSXWuSLGESKf0r7CXAcV8G/SPG4JN4gc/qCmCkHk974oD0r/eR0K0CoeJnxcS9xvXlqJ11cypwpwAZjjD08R+8TlPMRivwZCc+7M+cQdS3FN3pPfEdI2qksX1kU193GvtbMGoCdAabFoC+hcg71cV0iGTpY1CJT/p6yw0t4aCFdrUn9fXCvX48Tx+8opYLoeUzWuV5X/54/94e8Ji+W7f+Szt1h+/Q99tvGjSqkGwA/lr79ERP51rntdh5B/E8DvBPA/AAAR/TWl1Pe/4Rg/8/KWO+CnvvQ/4VvTWxjI4bG9wZAcnscdPvZneK99hmNqcGF6PPHneMe9xNfHx3Aq4p3mJV6EHa7CBo/cLT6YLvDI3Rar5wN7wGN7g+dhj188voMvty9wG1scU1NSdDxyNzAgfGN4C3szwpPBI3eLbw5v4SvdM3gy+NjvcWH7kppA0mxchw7XYYMf2n0IAPjYn2GrJzgV0WmPC3PE3xi+iDFZGCTs7YghORxCC60SAhlYFXFuBxiVcGF6vD8+REgGbzW3eOG3+Fu238EH/hLPpj2+tv0Q748PcYgtLu0RezNiJIuvH9/CuR3hSePS9XgynkGD0JoAnwwu3bGk/HAq4ul0hp0d0VYpQaS/jfFlvABwHTY4tz0+GC5w5gbszYg+Nbj2HR42Bzyfdni7vcZzv8PGeDyftnjUHPBk3KMzHu9tXuAbx7fwTneFl36LQ2jw7uYltmbC+/1DbIzHbWiwMb5YWMWqLGlkngxnSFDYmQljstAq4WFzxI3v0JqA7xwvMCWD3/34l5BI46++/Aq2dip1tCJcebY+C9j60uYKiRSeTztolTBEh854TMniUXPAdWhhFOHd7iXGZPHBcIGdHXHjOzxsjvh42uHC9bjyGwCzy69VCe9tnqPVAd/o3+L9kgxufYvH3S3GZPBOd41vHh+i0QF7O+HltEECp7SQ9CdiqX7UHvCd4wW0Sti7ET929m18e3yA9w8PcNEM2GSmiz46HHIqEq2ouJU3JmJjPPpstdaKcOGGYuGWlC/Pxi0eND1aE/DxuMOXNlf41eMlztxY2h+Cg9URX+hu8WF/hgdNj5vQwqqE5+MWnQkIpLHNFmaxND8bd3i7u8EH/Tne273Ax+MOU7QYosV5M+AYGiRSuMz9H0KDY2BQ/lt2z/HSb/DxsINWhM54bK3Hpevxq8dLJCiMwWLnRnSG982170r6lMftLRIUPurPYDU/gy+nDbQibO2EY2hYOdT2uJ46/PDFR2h1wM8+/z64bLk/bwY8y/0bldCYiLe7Gzgd8f7hQbEMd3kP98EVS/a725f4lZtH+Or5U7x/eIA+uBJX/tXzp3jpNyWtzJkb8Z3DBVtgk8HbmxsAwAfHczxoj5iSxcE3xfr+Y5ffwS9cvYM+ODxoj7hoBlxNHaxO+L7NC/zi9RcRk8ZF2+PJ8Qxf3F3jab/HW90BVyP3K8/EWTPCqoiraYMH7RFDdDh3A54Oe1w0PaZki6VfK8Jb3QGd8XjS8z6dEivZAmns3MRrYAIOvsFFM+C86fHB8QJOR/TB4e3tDRIpPO33eHt7g2fDrngCbCwrc4bslfDjD34V3x4u8WLYwuiER90tnEr/L3tvGmrbuqYHPV8zmjnnavZe++x7untOXRKryqTKaDS2CHYRAgYUIQopMOIvIRq1QkAJpRDFthQCwQb9I8QEG/JDEFEiQgQRUqVFRau07q2qc2/VPf3Ze6+1ZjOar/HH+77feMdYc6699r7n1j23uB/sveYczdePMb/3e5/3efAbt09KvL3U6/3Nc3zz+iketQe8vbrGr11/Db/n/At8sL0qnuK3NjcAgNoGOJPxRb8p/VozGuRJu0PKBtuxgTUZj+oDPjmcY1MNSNmgjx6HscLVao9DqPD1zQuMidpvkfFiWGE/1gVF8HS1RRcqJBCSQsI2ZIyH5NDFqjwntY24Hlqc1/T87cYaT9odvru9xE9ffYRdaPBiWKG2AUPyqG3A2o+4GVvcDg3eXN/iebfGm2tq64XvcRMaDMnj490Frtod3lrd4ju7x1j7AZdVh8pGfNqdMfrE4Ytug8YH3PQtrlretBtWaFzAG+0WzmR8cHuFTUWooUOocF51qF3EJ/tzPG722I0NGh/woluVsJGzijZvXnQrvLW5xSFU6KPHmCzW1YinLT2zMlc/3F7inbNrPO/XWPkRffB4/+w5fuX5m6+/7ljt8enujOZ+08Mb4kFoXcDHu3OcN/1sfMfoMDKi5M31FkNyuO7bwt3QR48+Ory53iJki91Yl1CZxoWC4tiOdZH6AYAueHzj8hk+2Z8XCaEnqz12Y40hOlQ8f5xN2I9kSF82HW6HBuuK+mJMFhc8T677toTPACib7y0jSvrgcdF26IMvMlyVi9gNNc4ZRbMbaqQMvLGmtqdssKkGbMcah6HCRdvj+kC/p4JIGaPFRUv3H8YJTTQGV9Ab3qYS1tOPHj1zM+hzORv0vAEtm5pie4y8IS7SPsaglCPM8oLGCHytlvxJibkfxmnJnGVj0kybcoLKEMSFIDVkE3qpU110hlXSEkA52cJ0r2V7ZuEhBkjRlk1fyxu+cr9sWudkqVyXZzZaaQ/31zJMRYegzPC4edKI1gSGRxP3lSSRCNLHZyEyOjSD79dhIiehuA8wLH+UHpaMMf8gSPPyA1DPvmeM+RM557/20nsf6LH8P3LOf48x5v/KOf9BPvbLDxHK/H6m93/6Iv+p/+bvRQTp6y01IUXjcR/rmd6fkPwAKLqMkopOXPJwfI9nPUdnSLMvsa6kvFwBzLQUU7ZznT6Til6h5K8/i2HQJ4oRrWwsGo2SYp40BCVpaK0uTz4X3T8FF70vLeum4aVJveBDssWwFe1D0TaU+FYx9LQuYsoCvUzls9YDnPqCFvbH6iXjp+NeZ32p/up2HGunJGtI+9KbVH5sshobqV/xOuO4hqRFnmk4am3F+XwgmKVR36VOArMUyKUu976267HS80DGS+51NpU5m/J8Dh/bUFy+vyWGWMcNa/1CgQXLNc7O45N1jKW0TY4vx0i3Q7zwus8Ffqzv199TRokZ1tfI8aksNYbquOgc6rhi+azbM9Nc5P5Ytld/X87nU0nGRsNu9bl8ZA7IYkD6Rv9Y6yRe5mUbtCSSTsuY4PmiS45Niy0AauGT75zX8dY6LRcfepF2qjwNy13+nBGpVip1z4xacD7enXO8KMuZFpKOIb3WTe9hYbzW/aDnM8DPjFqcZek7Pr6MZxZvbfGgiKeDr9HeTN3WFBfxynHydk+xtTz3k5ngtzp/ebClXgYl9rJInIiXw0wQYwRbPK7g2MjC8gxMUD2JU81miv+U9rIHcVYPHecKTHUqjVZesKL7KOek7pi8dFb9lTELhjxqr5vSYgwlaS8Ke2HveM44nja7KbZW4nFLrO3SW8T3Z5sptleKtsxcreNqBbasF9g6npn7QutGzuIj1YtfyNBmcZsR8/HgcoVwp3gxA2Z5FfiseAwlX5mGug7aeOE+NvIMZMz7SC5TP1Qz55Z+1BbHlhBU/Wguz1Md58Ot753uuTs3jnnw5LI7ZUq5y3KW9TKn/56qJxWOmafxTjpVl2VfzF9hL8/ne0gn++lYesA1v/rvfjViLN/90//Kl57vb/7LX5431hjziwD+uIQ/GmN+AiQ9+Xe87N6Heiz/H2PMHwfgjDE/DuBPAfjfX7fCX1YK2eKT8QIOiWIiQwuLjJWjnegdxzb20WPlRtyEBmd+oB3C5IuHp0+evW2uxDiF5Iphdu577GJdvEI9kwB10SMkh7Oqh4jbk/chYB9oh1A8MceMjNpGbEODkCx5LLKbGXLkReEYsuSLuL0Wqh+SL3GFK08eKFmI3vTklatcxG6sUdtIcX4cHym7iQMv2LUxK0bJIcwhQxTzObXH21S8BdrAkbbGZNF68mzJYre2sRhgh1CVez3H2Mn3/ViVXVOJJewCeTc29VCE35eGlV50106M38mwHvi+DJR8n+3WAIDzlmJMJZZQIMIZKHOjG32BPesxBYDA7MGy42pMLrBjMj5MEXZ3vNjVxmzPsYJalN3ZXPISEfdjZEgaNp0yxUPWHO+ptVIl3jFjMnz0BokBimEovyvSxqjIicourVybScd1GD2qKsxInMr9DAfW+qOVi2VjJpWFO8032WWufMSgdqkFtjrtAKuyOF+BIGtW38S7tkLSJMaJiNfLjrPkL3XQBtfS2JEd8mEg6r+mHUo7tJEmdYg8xp5jP+U6KmsyxkJgUfvBw1eh1BUguLOxtPMspFGiR2sMCkzWV4GMK20UmIzQVfBNKDvZMpYAEAfSTS1kVMx6rHVjl7v5Ev+YIrFNJyYeEkiqkEwBIOgzwzP1eTGyZPdbSINSsIUQyVgUaLFoiWpioyVpUe4ckdZ4ul+0R2Nt58ZLNsgi5RPJcEoN6R/Gyk5xaCLTA9AiRsnbzBb3wGTMJIqpm2Ie8xTDaDNBF/XqU85LF2vtYinLUDkZKMaJCRZJ4LZs5BWCK4Eh9rYYNjP5hTQxLhfyHmmHGFIGlEdgxmRvmQ2V2jWLDRSj0an+EEilxFVK7GSejB3wvaUOScEvDSaj1DGk0mIykITdGZj0CCsmLFKEQCTNc2rl/fJEzNY8Hoq8CJl1EEWmQ2IoZUz53plhqepvZTzVfJqt6Y+Q99gRxBoLFMkV2Ekj0fD1mScwMaOaKc7QasIjM1vAz+QrJH9NGKXmvCZNAsRInlsfwvI8YxIvZYmRreCvZrK+xAiVx6TECmJuoBYDeLlvsDTAjhhKcl8hoTpxzfKYaiLKA6kN3yVhkbr+QYblfcbeAw3LUveHGJYLo/1kHTG/9mhe96Tl3sxLDUd9/r5yT5Tzo/TKqdKcOjnnXzPGPCiG4KGG5b8I4M8C6AH8ZQD/E4B/81Vr+WWnMTt82p+TURg9rkeCcm0cQZueD6sZUcw2NOgiwTP2oUbLUMAheQyeIIetG5EywbZ2I5F/YAVc96uy2BeimduBDFeB1FFeDptqwG4k72njAiIbnGIACXFLZSOu+xYGwMERZCaDjLohOJw3QzE6BNqiSVZSNuWeIXiMNb3VBb7yfL9CUwWqa9dg3QxwJhOUhRevkrcBSjyqwI6kruK5AcCxnq7ET5LsikdThRmDLIBiQI3JEtRNEbIIecuBtUNDtGiqUOJTh+AoJrS1OAxViQvtRl/0PiU/7SErbL288K8cxYiKwQfuK+nXtiKx+t2+KeQvY3QYlXFU+zArQ+AwYrBor5zAaFIyvPhPHB9pCySmYi1QLzqYJpf4z2HwSMGiWY1lse5cQt/7QlxCcZdmFutIfQIgG8SGdEPH0dECGmT8DLua4rCakWJLZUcVKB4cgeYU6A3/qhQDLLrC+Cqsq74iwyMninUMgyPq+jgJrxU5mGgQPMU9WscGIBtEABuW2RTSCvIaWQTvEAaHsaK5aix7k7iuYeTYYDcZPaEnEfhYTX2QI2mUJhYjt1bFqWYUfVOSqGFjsyYvFrG05vJZWF1zNIg+I27JsOwxGYA5WMSatUQzSpwaEpBW03OVGRoVWaIHAHLvkNcGaeeRWjPpo2ZDouii4clGSmpNWRjknpEMDce7AUW8HgDMzmPMoNi6wZLxIgv/waKYvMEi1WSgpTpRORIXB0zxfInkTyDEL9Eg1mmKOWPyF3oALZCAGMyc4VUMB/EycbtMNEjCtKti7nJlgZEMnJnkDibDyW0dUpPpGgBmMDDJIK3iZKRwvezeImXABAvTGzKO9xZpZabFD8fWlYV0lWeEJyS5Yqc6ZDIwUpO5rdQPoseZfYY9KHKeDNL503qmvZlIScQGtZOep5RFxCi0ME8VG1pMHGIHg7hK8LeWpSMmcp3CpmpRdCczS8uQIUhGBwwQGyrHDSRPYRKdN8lwmZyP43p4kqKAQZHRsAO/p1a56FjCGsonksyL5TJNBMlsyKJZ+t0b0rFk6QsYul/qnGppMxHmkNwEGauuMxi/B4elU3JF0naTQXO6RZHYscrwFcPCMUFPqpjcx1J/FfIegNh25RFT8W13CGAsyaskljOicZ76RAwdzYKryXuK55C5qYqOpcQcCuOsMhpFe1VSmV9hMvhKOQz/IBZdo3QsTWmDSGPo+7WHlF4wUx2RMemkKtkQfc+SuEgfKzIp2nA0U75F5ihhZpxmg5m8xywpY+yYoSkambOxWxqEx/KT6xZz9ZieY7ZmpsF51ONpMJ3XdZnlvTAkT9TxlQ3VY8dPpWNlPtCQ/KExIvX8++qmXzDG/BcA/iJ//xkAv/CQGx8EhZ3dQMKZm5zzzSvd+H1Ib/3UVf6Z/+oP4xDr4oGUWJUxuRIbWGIEbWQWVWI3BVBihYbo0LqAgaGlEvMmRugSzgkQpNNbirGzyEgwqG1AFyvUNiBlWyCocl6SMG6u/TC7TqC2wsgqBql45sTrV8hq+J+wnQKTB2nlRwzJUV+oWA1t/PXslQUmKCe1bTLYvHrD6v4BJimXQUF9i6QLG3PiWZR6C9RTDOw+kAc4ZoPKpiLBUrtYYv3EgK5dhOeYkUrBPAEUr6LUHwDGZAtBk7RaPI9ihMZE2qMAsBvqwlwb1UaA9kqKDIyUrT2mIjUjfZSBAhvVEjHeRQzBwZo5NFMIoWSjAiBIpsSQ6Ps0IRS1mf6Kwe1dQhCvm83FQB7YgyXXa/kY7f3U38uct0nNPbo3RAfvIqwhcqfaUx3FkBfPp2MDU4z8kCz9birvq12Mn4aDVi6WPl9CauX6jKktcr0Q8gipk3iwl3BGAIVkSDYZTOlP1nXlzzrmxRjyOAr7rLCw5myKx3WCg+YigSNjI3nomJ3CssobHKJvK+eqKkxe6jRdS30DjhEixlZhg9WwUYkXKpI87J00hjy8Ir9jbS6bIOKZjaresvkhfSCbKiKR41xW8Fd+/lye9R/1u5n1g8QfWSaZEqIr5ElCp8QT8UaTxBJBzd+qDjNyKrlXILL6WsfxVdZkOJ9mMVeymrY2lY0WqcNsrcjzRuYSeE7PdIwXsVJ3YceKHImvK+u8NJdBKnOe8ynkUwkwQsKSaAMhBWHXndAGZU6YPCfFYs+xtDllWrWnQBsc4p0GuJyFt1k80SKZA2AiOZINGSZNokbThoIxuWxYTVBh7juBtAKF/KiQIbH3VSR3Cix4tDBeyeVkIqQqjLqvk1ymjRhgqo98Fi90NtPmgLQPIPhvntorns6yaaK/87iUlX5Sx2Xghd1XvmvvudxnVPmyAaPdb9oa0pNZ8pTPmvhFX6eOF5KapZyP8ooXKRKVCsxXE8UcM/65LdqzODN0pe1LQ+QeY+WOEaWhzuV+M/WdLmvZjhO3HpOSmZV5Xz3vSUv7VP+9W7n7Tp4u815j7aHmw0Pb8+D8zHzevkL65r/+FYDCvvdefvdnvw9Q2J/9UqGwDYA/CeDv50P/G4D/OOfcv+zeB3ksjTF/CcA/DyJ1/usALowxfz7n/B+8XpW/nHQztPirv/WTGKNTMUDkmaAf7AlWJoHTRH+f+IeRFyTMVqqfZ1kcIE+MlnrxQx84pqzA8DD9wDIsKMVJMy8vXnQ5q3OLF4+x9ANbdmplEb3cORKIjUDEDEPeop08FIk07krMjDyQ/MM/W8zpHyTxVqm4B4HSFWgYe7hm90uSH8+l1pyCYM1os8VLsYwTkWvk5S7wMR3To38kzaIOyx9jHcPD+X7e8NYpQ5V0TE75gZN+0TE9Oi0XE061V87rmJ9kcCcJ+53+8dX1P3WfTqcWACXeCbjz4zJbHasy9XdgPl76nO4TfQ3nbfKk7SfaZxNUSK8M+BYxxJKCxC1ijPQDW86pthRPkn62DOU9W9xIW6T+nO/sfqPatew3vv7A7IpmnIyI2aIOoOPieYi4m/TzaQFEg57HTS+ieq2fyM+NCROUbeDnx0RDn7OZ/Wj3DBUEr4VnC1iHSSKCd/EDwxNH5Q2Q8qOdFk+jWuyZpKaxmkNBawDeM79M4rwV1M5koDjCI50vUD2o1wDn0SnPHUDXy72yJyXnRk/tTjYjch+MFXvVOONk87SOMVO9pHxkup8upv5Mi8Wq4bHINsMmkmIoGfC12l7RcWdWTausLpJ8LMfOWbXwNjyOLpJHEpmul9dR5t8ty3M9u+l+asY0fxw/39lSWZNnafpeysRUtoxdlspL36nXkZxjYt/pFaXnhxrfom8p5ckzyuNbtCyVt0rmtD327D0wLSGhSw/ezAMmdS/vJHVMLziyOqffnbOCMXsHGL6nyJ1kzLx15TYzvWOL5uQd1xqK1316B89fC3fapJ7tZVuPtpN/J2bPjJnqfcfzdWTcdT/oPCSdNCzvSfp9NVlmiwGQY3l+3zKdNCyX1z6wbnfWLtB1XORzos3LMXwdw3LZhqNtWuZxagyWdXhoX9ytWanfj9L3JXkAfz7n/B8BEKdi89AbH5J+f875xhjzMwD+RwD/KoBfBPADNSxrF/DeoxfMqGhxzYyNGz+gdgFfdBt4kzCwx+52aHBWDUgw2A51gZYO0eG87rEdiIUuJov9WFFMmk14utnhuieYrbcJfaRYzMNIemyXqw4de5j60eOsGbDtaxiTsaoCBtbDkxQTxcHVLuL5fgVjCLJJun+0gx6CxfrsAMe7uP1IMX+Vm5PLSEzeODpsLnquQ4XaB2x3LXxLOn67Q422HQkKy54KY4B125Nn0UyQVsMeHWdIKzAw5BAgT0eIkxfIOZJuEM+AZioTaY1VQ9BWOS+QUK2PF6NFVQX0fYW2GRGiRd/VaFcD+q6CryJBQrsKKRo05/2sPPkrHgTZ2a8bjjsNtnhWSGuPDMm6ofZun1GMZXvVIUZTYspSNkXqQ8oYe9HRi7w5MEEMw+hohz4bxI4kBiSWTMgzbJWQgoXz5NG0ZvISxoMjOvXNWEg4nEsYO0+wxd7Bno1ABu3sO5LBSBKflAx8E5GiQRpIJgLgTYedR64T7CpMEg0AeQoUrNPYKR5Pdv7LYmlwk1aeiK5X5BlANDAbpvhfR4JIArTYc4nKGyywYop82b2vUll4SwxcqVsdidp/HYHOIVeR6mtz8UYAAAaLnDMZz0J9v/fIPsE07MUIhkhHokFes2wEx/Eh8AqxneqCwDp5AsMMls7LZ8/9MtJns/PkWLgIkwEaLLVBftSjgRmo7/MqqoUlt9dPEE3TW+RNgN16pFUssEUAsHtXYpVynWAGi7SOlK/NBbKZNhFmsMgmlRg4APA3DuGC+3KwHDNH4+F2FuGc+2ckzT9zcPR3MKQNqBdzAj9sJ3inGQ1Sk2AHO9VTpGR6S4Zdm4v+HEEG2ditAEQgNQQrtYNB4n7PDrAdG3lthu1YF48NUiElEc3E6sYiNkCqM+0J9WQghU2ieoMggCYD7mAQNgl2NHB7g7DJ8DuCbAKkNegOhmwXWUB7THDSjAkCaYDkqD42EIS0kLVYkI7hQPBAtzfF2ALovlxNBCuuU5BJtRD3eyFgoX7xO4vYUjuzJ3ihSZSfPwBhBVTXtkBe9SaQjQRxzh7U5praDoBi/9hpG1cZbm9ge4PYkpFgIhgKy5BGi6IbmVQ+qaY22YEfjTXBbcVAMiPXt8EMhplqmrfZMjwXVP9qx5qNDP10I0EtTQZiTZ9jQ1BcgsLydR0wnuG1E8GA6bMdUZ5hk+i4HalObkDRjxTYqevAUOGpr8q5nvpYQ2EL2Q2PKTCVly1Q7Qh+K8+GtHUGB1VQWDcY7psJXm3D9BwkgYHmaQx0Pe2IYkyLEZw9t5nrng3gu6ntNnKdGKrLHIsLHUtMz4xsFDmU3x2aYyjGaaqoflIHgX9ma+aGLKa8ipdTNqnkniWc1NHfpSGlJU7uGHxSTwXJ1f0/u4/bcNIA1nkb3IXCKkjvBI3HbBPl6MaxUecfakSeqGPZ4JB2nzLuThmy9xiDR6G+/Pel6YfIyPwhMIj/FwB/GMCWv68A/M8A/r6X3fhQw7LioM1/AsBfyDmPZrmj8wNIZ77H3/X4A1hk7BPJDoicR4TFj62fzYhkdrHByg7oU4Ux25kYeWUSxkKhbwv1u8Bb05m9w+aoWVyBCbYnhEACzZVEkNZYGGKtSXjnjMh3vEmFHVbIfjSjqyTNlAlgRkqzJM4ZLhnOZhLSo4lRVqC2Q/KwyIXJVb4LZFc+6zKknpq8Z8lUeozEB5igsfpcPJ/YRHW7hH3VmVQ+W2SkS1P6C8AMXiz11fXQbJpSr2WfAcC42Rf20SVrrFwnMGSB9kpawkVlXARKK5qfM/ZXTO9bOZ5xHBZqMLEOCzxZ+mjJujrLL8/ZQEN0R/saQIHwCpxXs77qFJMpMFzNmho5RlTgt84ef2tKbK+um0BqJR1jBi1EMkcYV7OaM3KPjJPh/ijEMQtCIt0PGShjJd91zy6ZajXbrDUZ4THDjhXVfVqMgcAQgQnKuGyPfBa4Y3w0EeZoYqEZI2ieM6cK/HTJzFr66olBZXLp1yXioLLTOUFr6E2jZSpxqUfG9+611DZvT7MDa9huzoBTn4v3zhDxkLeprIaEel9gruGK+s6rccsAvGzmQDnb+f2dQfPU2Yz4xBQkiskGSXs1JKlFYM5AzGpucl5Gw1llDPnvWEiEjnWWWlfqFZ/BLPYdJmOMk07p0iMVuA7dlZ02VFRf6nvGRIvzUWJCBWHBC8kxYw6nXC6WpS16Ubo8r2GtUMcMl6Pzl+ukvlzP4ch7+ujCfbHQLrGlr5tOoTF0u4E5gsKoexfX3UFL6LacSnxdp9hdAd5Y0QiE+ce5V07XQ+qojaVlPZSxpI+TUaPGDpjDSZWLtaBV+PjMgNH1WaZ8t2pGf8mLC0/1nW5DLOzPVAAAIABJREFUmZ+ZP8//3nl9qXYcg18+yFB42TUvOb94rB+eXta/i7LvGJivUMeHpPvrbo7/Pfa8v1b+P0oPTG3OWYxK5Jy3xpj1Q258qGH5nwL4TQC/DOCvGWN+DMD1K1fzy04ZWNsBHw2X6JPHmSN9xRfjGjehwdN6i0OqsbIDng0bPKr2eDZuELPBm80tbkKLIXls3IBP+zM8Zu1CbyMufI8Lf0CfPH5z9wSPa/o8JIeNH1jj8QAA+PBwgbOqR0gOq6rDJ4cLPG23sCZhF5iN1cYiGbJyI25Dgy62eH/9HGO22IWGNBiRceECVnbAR91lMTIv665o9EkcpjUJa2aCfVQd8HF3Xthku+jxjbMv8GxYYzs2eHN1g+fDGkP0OKuonxIMPtxdYlP1GJIvOn2Sx8CMt3UVC0Pui36Fs6rH2rPsSHK4avbYjg3W1TCLedyHGms/4Hm/xqYacNkc0MWqaAfeji0eN3tc9yt4JkM6r3vcDg1qG/G03eKTwzmetDtsxwZ99HjS7rDxA3579wgrPxKrrJ1YZWvW1xRD93m/RsoGjQuQJaRopFUu4qZvMSaLn776GCkbfPP6KRofcF51RQ9xP9aAsM8CeGO1gzUJt6xvKaRHIVmc133RtXvS7tBFj9uxLdpzm2rATd9iXQ1Fq64P1LfOJjxu9qhtxBfdBgBr1AVPmmfJ4bzq8KzbwNmExgXScGMmXTG2r/sWrQ84q3u86FbIIE2y9zfP8Vl/hmeHNRofuE9IK1NiOrWsh3j0x+jKponE7dY2FgN0O9Z43BLR1HXf4p2zG3x+2ExEUNEVL/8ZIwNErxAgsqnaJWYQpjqJluNuqHHZdLjuW1yuSaMtZoMQHVY1MRqnbLCuxtKXQTTjzq6xH2vshrqgAmrWOnzerZjJluJWG0csqKKjFpPFpibXyk3XwlnSoDyMFRFQcfxvTAariuKE33v0At4kfOf6UTGsGx+wY/QC1SHhvOnhTMKzw7oYqrJBQMgEGsfH7QGfbs/w7uU1vthvSuysMRlPL3foo6eYWWZJFvRDzgZnDdX9+tDict2X2FqRS3n34gbfvblAiA7rdYfGxTJGj9oDPro9R84UN7rtGlydddj1NdbNgMNQlY2HDKDxEd5F7Psa522Pw+ixqkK5XgjAAmNY182AxkXcdE2JHQ7RFkZjIQbrg8NGXWtMRogO522PDOC2a3De9tj3ddksEiZnmcPvXl7jebdCN9C4nTEh2he7NRpmGO4DbUBcbfb47PYMTTXiou3x+XaDJ5s9nu9XZaPmcn0oBqi3CbuhRh+mGPIxUp1jshhZM29VBdx0zUT+Fokhe8UkZZerrhi8AHAYKgzBl02UTTMwmgUFXRKSxcWqA4AythdtX/ppYKZuXafr3QpvXt5iiA6HoSpzQRiu+9FjCB5nbY/9UOG8JQRM6wO6QBugN/sW62bEo9UBn283hRzOcV/UHNdM+Sf0o8eaUSNybN2QduWL3arEJY/Roa0CnE3Ydg2aaixtOPDzk5JBXXMcc19hs+rpNyhQKExVRazqqay2CtgeGqzbHv1YlZjry1WHz282960s7k2rZsS+q5ETIV5knJzJ2Hd1IWcTpE7OpoTLSJ2HwaOpCR2VEiGUNqthxiYOEMu2RjHpTZ0YDc43XemfcXRYtSPpP5Z3BYX1UIw20DQBfe9RVYQSStGi4j4dR4ekZLCIlI2QNSlRG2oVs2xsLuFFVR0KcgqZmLEDY94pztwiBgdfh4L4KbG5mRiyCXHkyoZMjqboGQqKBnw8ja7E09I5lNjemTYkG1KZN11KrC3fh2wIyWJBqBWJWZWNFEkJFKsb1EFt6Ij3X0I89MaJxOQW4z1PpGV600GSbIZIGXpjBbgbRqSv07G7uo5iKeowmWXSG0H672IjRferbJ5ko0JRdJb6uN4EMHi5kXtsA+93k8H41W/Lzhjzt+ec/08AMMb8IQCHh9z4UB3Lf0N9lf1Al3P+udeo7JeW3v6px/mP/cU/gsd+jz55fDacAwAeVXucuR7fOVyhcQHbUONRdcDn/Rmu6j0SDD7vN6htLEbe15pbfNqf47I64BArbMcGtyPJdby/eY5PuzO0LqCxAYdIC+IbNiyetlvs2AghI+4WnxxItPiq2eFmIEFxIfQZksPaD1j7Ab+1JZH086pHFz3GRAvlw1jhjfWuGEq3Y4vKRib7meRGbse2SHM8XZMQtRg2H9+e46wZcFb3+Hy/wWVLQtbboSEmWJPxZLUn+RSQSPVZ3RdjQBteQoqzqYYi6Gx5gb0bahJcZuZa8a7JguSy6bAf63K+8QHbocZF3eO6b7GpBzY2B9wMDR63B2yHBrddg0frA266BrWPaD0t0PqxwtVmzwZJLB5TkglxSGnyLl/y4kva60zGrq9RMQvqRdvBm4Tf/OwKAPDWY1p47foabRXIcKjHmcfxlgWe6TjQVgHd6OFsxmEgGHLOBrtDA+cSmioQMZGLhfVW/qZkefFAC85DVyFFh9WaFnXWEIvu9tDA+4juUGO9JnkbIhdJZcFEbLEW67ZHiA5dX2HdkoExBIf9TQvfRJxtOvSjRwiErXM+FYZbghenQjpT5EjYABh6D88LJ1kENO2IofdIyaJpB3T7Gu16KIsM51NZXITRo25GDIMvZdZ1KJ6wkdldY6SY57oZ0Xc1/yVItMCph97D+QRrE/q+Kv3lPC3yu20DW0fUvOiJgSDcabSo1yNEmiSMrnjtfBUnSY7eAwao25EWVqODY5kSgUobvt/5iPG2ARLQPO4Ko24MFlUjRDsguZO9B5KBOxvLD39hwfWR2HRNRtxVqM57jDcN3CbAulh0FMNtRYsgA4qfHhzcZiyLqnjgjYL1yO3IsFUqmon5RQ3zaCDW3N4xnJd36vcOuBwpzrt3cKuAuKsIQn3wME3kRRnVOQ0OyICtI9LBE3RYpDsGVxZpRkhcOoJ7m1UkCLWGYgdTyFaMT8RuGwxDlHlx1zEKpCF4NOo0BQuKFqAnuLZ74ZFWmXQLE2B6S7DN88DBpSikKXbrkC4DMFjYg0U6j3A3DnGTpsXowZVYJYJt5iKPQXBFZnkFChzXDgZxPUFv4UDxuz31ud1LMCTlm+pMUFjxbHcK8iqyJS7D7W3xVmVPMN24ygU+KuVln+E6i3AWUT93yA6FNVbiN20guGKqM/yBoLCuYxbXkRh/swXimo4LVNgGEEwxEPOtCZRnqrh/OB9khrgmlO9hk5lVFcTyORIsM6zyjL00NdQvAsU0maCufk+eR4HC2oFZV0HQY9crOHOTma0V8HtgePT6qzrPbQeoTIlNNJHgvW6geFfXmwKBTPyseobvppqhr1z37DLcgeou0GzJs0Ck2Q1QvFYOqG6B0LJj2U9QWG10SP9KfVPF0FWGtzI5PmKNGQusDRNkV2Cqus42UhmpIthvNvQZWEBhA8NsuX6FUTdNHlQ7cBl+arPArw33g7DAZoc7rLCaVXYWZ6r6SkNIj8FWC1su94H2ptLcU+yuwFHPrzCuFvbVPEGBJS/D+QEgiKfAdYw6L8+7MbBxPleXUNhSPx1PrJ19Ks/CunvMIyv9pY3JYwYe1Bw8Fhur85rdND93xKF9N64X0zXmJXVa1k/nq9Mv/JdfHsHN66bmvffy1/+lL5+85zf+zJdK3vOHAPzXAD7kQ28D+Kdzzr/40nsfaFj+afW1BfBHAfxqzvmfe/XqfnnpnZ96lP/Zv/wPo0+kS5myRYIphg1AsE8N+Tt2nv6ach4gWKX+rvPQkMbZvdnMIKEAGPY6QVCtSTNmWSljyRor3wUKuyxTl60ZWpf1W0Iw9fkllPBY+5fHdbnH4K4aZqr7RkMI9WfRrlzC+vR1p8o8dk6XeSzJOykeyX/5rtLlnSq35JuP989D67eEQd5tHwqLqz72KkmgpEuoqS5zCWEUOOR9Sc+hY7DcY8cl6Q3b+5pz6nfkGGxWf15CLeWe5Xw7du+pdOqVeaoux44dg4qeOqZhofflr+u1JNJafj/27J9q9xKWukzC4Pog/NE97Z+XeXosXmWeFO+FXqksVzKlzEUhi4XQsToeG6+7F6qKLfM0mMMGjzVoSf50ZNxlDE4SqZWLT6zilnWT+p1aHGoCsmMP8X0LwMz1WEJh9fFln5VF/pFxXOaxLF+8NMpQOAo7fmh6wNwAcNSDcy8k8WWPz7F7lnPn2GpaP+NHFu0n++MBxsJJI2CZv66/nisvq5vO49i7Z9mGO/MFx+f1qTKPzdk8v1Y+H033/G4cLe+h6SHXP8Dg0vV4pXJeYsS9NL1qex+YXrkfAfzqv/PVYIX9ITAs/xhIWvJ9AP8kgL8bwM+JB/O+9CAobM75P1wU+PNc4A803Ywt/tePf7xIVHRMplN7lqRg+JPIIQi0BkCBnBVdyWpEzzC4DNIjFOjJuh1I4kHFOQEolPxNMxaoSwhESiPnBHKif+RJHoCYafuOJUJ8LCy1OZOOn69jWbhozUJNFR+DK7ARX7M3hb1OoatgxFvUezjOT/T8jM3wNZVrTC6wF7AxIYy3WcVoifZginRM2GiJkEYWNupcMHAV1bkQw2gSm8HRX6all+MCd7F1RBqIBMfYTN6RaGDbMC2kGC6DZIoeYfkxqFnmJNqykBEvCQBYn2AsEG9pHMw6FIF0eCJyMULeAgCZdQEzyMMDTHUAyGMi8TsDxTMVchqb6XyViX1WC3+D68y6fbmW7WZM2n2GRNBzzfWRRYWQuhTCm0TlRENlA0CwsJ0lLcIqT+ywGSiMvdw+6pjFopF/EU0U7TswuyBm37Mn4XLx1pjMXhH5QWaSmCKULeWXh4OvExZDJoLRQud3GGiByRtkUWJ4bGeLzmDJM4G9K6lcb6IpDK3ZT9AdMzIRTJXoWFTno/Qb6LMD7MEA2SCtVSBSMNP93N92oM+pPrL4U6zCtrdI60Raim2azRPb0RzOhtsQyHsm42UHNvTrSeh+Rt6zJ89Qtix4r+BOpP/HfcZMoranvpQdfR32IsQa2YMJaTJrTwphDY1/YQcNpE8YmcwFJk/n9G48C6pbYTNNSh8QKFqL4g2SZ8IAxRNX7cmTJqQgDACZyHQwzSHXsw5hJA9KbAHbsxfGSL9OYyDajYV9VerMHqC88NgUTwzPG9ESFG1DmQLZkQeyzINxWokWJlQ1xhJTZ0fDrK/8jEUaoOzoXGyYSMfO8xHvjZAKuYF1H7mt4gnJBkyoxIRENfcn3588JuIVrn9W+YgXU+6JDYpnSMhHTEYhZpF5IB68bKbxS37ykmk9RjtOZdmRPIN2xJy8pycio9dNNgAMWiKvXUYhkUk1ComKHYG7OpYohENFE9JO9RKvkvzeFO3EPM2n8ltkH6BjCRQCJwBzHUt+jkVLNC9Jf+I0LjJvS531vPEqD87TDup5j5y3kPeIxDrfP6vDHR1LdV1G0YSckxPlad5ZzIle5I8jz6D2EOqfPD0Hj3n15D13FEaqf8MX3kfpf82MLPnJdaXu8pjn6fpyjypj9o5c5HNq36XsH/F99xqWuu6nDGG+Ttf5VdKDiHhO1On+fNV4vo4B/zuUTP7q1k2ln8s5/7fGmEcA/iEAPw/gPwEZmPemh8ZYLtMawNdf894vLRkzPc8AOK5pIuqQv84mOJtLHAxAuP9C4CI6jhxTFlnMfkYwcoSMRJN0yI6+GJ9yv+SrPRWi42ZMLrpqxoA02ZJhMgBTYiQAFNF6qpO0jaRSDIAEq8hQ6D7DxquxBIMjYy/D5DwRUphMRBcAjJviGCzPfGOnOug+NRalHsQammEc1RMJJX/rUYxMaWcpC4BxqZw36rhokpEhmkqMhXGJ+oZjKmTBZ3hxWjxwbHyXWAs7zQu4XHbFJc4DzORqDBkumQ3Zknd5q+dJSF7nzedyJqbWLIanyVO8hskkNSF/RQZCG8fFWJiMsVKmych5qruUXWgq9ctXjFszfU9NKmXMdoCNui6rui5f5oYJSCx1fFbtL98tL4rlmPw6c16Z251V+2ceCq5IIcNQ+dE9qm5QC+4K8x94YDIqZcGuCEj0AkFfAwvqY/1i4WMGZvZZ2i+fU8UMdkrDzXjwDzDPmTwt/ovsiOr3bDKEGC3VGdkQLFJYU2kYyNAu/W9oMaUJSwr0yqKwvdJ5uidVKEa6PEeyGE2St8q39CUbKrN+ln5hCKQ8h5kCJkpfa3mMYtzIZo9aIImhIPcno6413B7QsZTzjD1S9wlyRmpoUS1tTSwZkC3KxpC0JVXUNtpP4c+VKeVRv5oy7TLyZDRwntkq55jky/XPeo6ZaQrHRvWLLKituh9Tf8t9GUBu8gI2x89KlnchjQ61m/opNXmCxOlnmzdpsgUiVJ8CFLfGz13yXI5jeKwzxaiaGQVWzxWez07IW6S/M6yl2NJs53Bi48wEUXR5cm4xbCPbjAhTnuds6F1eIIYCMa0wQSTddG0xbl4jZYsCLU3KE1qMYKmHmeojfRkxHS/XqDboa5EX9VTHpR7Ur9P5rPqg3JanvIvRWd5L9NlklY/cJ0ZfUnVU80Lqki1g5F5lSM+c46o/7jCkguvDbdKbXNpwMTIJjLpOLsqYPQt3PIdmuhd5Lr00a4eCW965Zun5Nkc+HzlnEjHOUl/I++4etlP1O1bOL+uC+bgeMyyla+Ra+X7SI3wk3WtYSjvvuX++ftDfzd1r7ynzwUbiQ675UXpoEmGgfwzAf55z/h+MMf/WQ258qI7l38A0ZA7AUwB/7lVr+WWnykZ8/fwFulAhMORVi59vKtoq1XBWLVJ/F8I3qs84CYO0hhn9gBmsdnYdxxHcB6lLmeL3jjGiArgD35vfO8/rqNj9Ot+5dwktW0LO7oPOHkt6La+rdKwN5d2b78LwcOR4geC1qt6tOq6ueRk88RjkUV9TPwpH6zC/jtuxKFufO3X/MdZLyeulm3fLLbh75tTJLB5wz/I3UY+rWfx9eWYnfuFO1m/6/KA+OVFhEYeHPqU2iPTxqew5DPkORPTEfbJuL583wMwm5T5Yfof+XjKjcV32kgFIGmVZ51n9cScd7UM1f+KCuyQv/i5PZu7jl60fTv3VN+q1xbE1FfLEsinfT6ZTLyBO6b5rl5P51OR+xeftXonEUw/RQx6wBz6AM5ZS/i4pnB8n2bg3b31crlsSepycnNoywXz1m5cF6jrnE4vJrP6ceukfP/xlpZcu0l9WjwfOp4cspMf7+n5Zlwe+xJfv/6N5vcLx1+qjxbm7ho2ZnZuNxTHDb5nvkXfGSePpnnot0/G2HvkFyfr4y+bDq71/Xslb99J3yevd99Bn8LW8dr9bjMfXWMf9DqfvGmP+MwD/KIB/zxjT4LjP/k56qMfyj6rPAcAnOefwanX88lNIFjdDi5u+RUgWq4q0//ZjhTE4tPWImCyRpowebRWwF33JmljnQrSofCykKyJRIGx3MRs8Z/Y6MVgte24qF2ENsO+r4r30zIQnbHea5RGgZ8IZgt8SE1yPkG1hMRQpCe8SDkNVpAkIAmsL25skz8yGjY/YdTVyNqiqgBAczlY9Mf2NHuu2RzdUSMmirqahO3QVPLO+CWELAAgLn2corSZXcW46Jkxx4+jhXFJsdBmBCU6EqEXOp2jhq4hxIP3LMDoiHIkW3keEkRgRm3ZEdyDilnH0pSzvIw77psCH9ZvUMlOdpGEgYS0hPslgeHKg+2JwyNHg/PKADGC3bUmeYHGNGCzGAr4KMAalr3Qcl/MRkY/XDZHzjCOxQyb2hAfuQ9ECFeIWY4BK2ndgsa9skIKBb4glta4i+q5iRr6EFF2pFwxJiYTBwThqgzDwWZ+wWhGhztBVsEx6A95MyCJubkkTUzQ0xZsqMhLWp0k/0ybkZBFHC1clGJsQBo+qIeY/K54lxdTnfCpanykQvDuNtpRjfSLjjL1jabRwdUToHaqW5goyiOXPT2Q01ieQ1IQhyGQGqjYgBoscWGKEvdACDddjly21HULfn00hm0kDuXdsRVBtgOqXoynX5WBRbwYYk9HfNBNE12dkgUQDBF9rIgyA0Pnyy1o2KISRMBu4lkhz3GZE7PzEVgjArwITBAGIFraORArEiynDZaTOk85oBhsDVFB9NmDY1gUubX0iuLghePy4rygfrr9pI3LvYGom3CkuEJCWqWVd0Sayvihpm5qK8wUKE6KpYyEGgsBzE/U5XJ60LYMB6gTrI11rKA/TkAZtHixMzRqqYuiIt5jLqs4HhN5PJEFVIij+zk9efb7WrgLStgJ8gm0j0raCPRuR9h7iMjOtMJuwHAtD8wvCIBoYhrHnYSozd27ykPIcRUWQdcmzzMXREhRf3vN1mmDuwiaZMOmzJlPCA1LnkW0iYiJBOUTqR3QO9nyk53Fk16q8e1wuOq9oSGfWNIREsT4jBQ4z6B3Nl4aJmhwjShyNp/Gs6Ttw/jyGAKhONk/907lpDBIKGRVE+5S9lRJSoGVCTDDIbSx9Wc5JWYOl/u0t6e5Kf0QDsw7A7eu7LFOdiHgpM0Rbfn8sacISwRK3R+D3Au1vEsPxLTKjZEoYgQ5xAM9ntZFohHjL8OZA4k2nnvsnEuwe0UxhCozuKHB4CSlg8iaCK/M1A/ejlOtQYP7yTOUqlZACQSmYYArZVNGGbdJUX8dw+0AhEqaf2ld0GIU4TN6/wIxARzyeMldIWzNP59iTSNqYyqqS96Ho+0bpl6lvZ57ZJcRU8hCPfJwfBneNYWNV9kqKTWumvHHk+qNeUKjvxyx8fZ+uu5u+z/LQ/XAKzqvLWv49lqfkpzZZgLvf9bHjBxc3nzC2lqf099fybP4oPST9UwD+CICfzzm/MMa8DeDPPOTGh8ZYfvt7qNz3La3ciJ+4+BS3Y4s+eazciJAtDrHCEEkqQ7QZhYl1O5IrUSQ5RNOxixVqGxAysbLWNqB1ASFbXNcrrPx4R9NRGFt3Y1MMwtpG7EKNlR9hkbELNZxJM1IfiesM2WJTEcur1oW0JqO2AbdDWzQsK0sMmgPjVcSL2LjA10fctg3XPZa8+0ASKSs/FnmCliVKYrbYtxUqoUtnuQYxfkXXUeqesikajiKPIIa76Ctqr+uYLCqbChus1FmOj8kWmQPpk9pF9MGj9gGVTdjXI1ofMCaLmGyRlbhlwz+kuZ6fsNfKGHWB4m491xHA7L6e43LfONuV9jqbS5sAYIzzN3HLhvkQPMWmsrZjymC5BCIjknoH1u8co0Xtida/9hFDmGOPjMlYcd5tFYr3OSZLzLKJ4oe7KlBb7aQfWaDVhuQTnMmofETf0CNe+4hVNaILHn0dCjxcPN8iBSFQcPlN8Y7as/wu98dkCoOwMRlj69D4iL6hOhhGB4ieY6VinWXOi4ENYEbdT31PeY8txUGPcaLv9z6W2GbnUtmwkSTXa9p72bzpa1/mqcDXJf5YvH3OEdZhVPfr8uSzEBytmqHMRcskKt4TG64ka/O06VRPr9+CGBAIt8nwPqJ3CW0zYqjiTI+yqWmjIfH4O5cwMKycNpdoBTRUEZ4NZF3fdTvMNo+snbQxmyrgwFB+2QjxVUSo7dQe9SvvJP9AG0Yx2sICLJtWFDduyvXWJozew3I5oqloXSrx5LIBJdfCkG6l85E2dmpXNnJkflpHhk1i72/TjrAu8ebQxBg88MaMbKwY0KbOYGgO+ipiMLQ5NAisM6OUXWLffSpanzIPnI/IGUg1MRsbm5D8BMWXDQHruP+rabWaM5AqWzZ6ZK6J5qZA/GV+ygorJQtfBbJHTUaqzSTnwBqkkTfMsqcyDM9/2ZBKlUWKieQlfILlkA0KmeBn1ZKhaV0EVrQZU8ImuF8yGPZqKLbdcpx38gTFt7JhA3AYhdqcAxsrPI7GZiRhDE682cMbYRL7npnDwLCRCxD01zCE1tYRSW2SGZuR2nv9yvcm4xND200x3gHqiyybG4k3ZXisZe6bitqVXZyg6XJe4uHV2PMgUfsVI2lm+IBxDJU3zFjq81E4v0CT4dMEb5c4eunfRoVkAGxcmmkVz6EdM6itfLcEn8/W8O55VjDNTKEKljcKGlUvznsKq4AKg5gkeJZe2WRVm8zU1pwEYroYs5TZkM13+ragh5RhOes76QK7OJ71eKg+42dVG7d5yb6mDbTjttRpY9Ms/nEdi8EohurSwFrcc7S8Y4alvmdhWBYAgfTRPc24A08p1ne+e7Wug/xZfl/mf+L7rMivktH5VarLkZRz3gP4K+r7RwA+esi9D2KF/aqmd37qUf5n/tI/gjG7wgzrTC46j2NhfKWFrWZftciz88LWSkylEQMHKQk7qhh43qQ70FptNIr0hTZAJ4ZWMmKFKZZgu9NCWNcvZTMrS/JZsqtq9ldgDt/VLLHa8DvGCgrMX5Q6SX2sybP6LZlglwywSwitGGrLcqVsbSBGVfdTjKrHmFdnefJft6iH3gQUI3o8Ut5J+O/i+BJSLXUao4XjhZc2uMV40+2l/iEDNTD5kmaBfQgD7DFGVB3fK3NA1zeqeXqKOXWqw93+0JsQ+tix3yf9m5RU/oIAOFauvuY+Vl59r+6Lu9dwHZa/8+r4Etas89d10p+1UXqK7VSu0+Rby/jlOUx9Xh99rTYyj8HOBemwbGfJmxfYUp/ZVu+xbd+ycuAM9flsTvaVbtvy3H3su8s5c6cuUocjn2cQ9cUicZb58rDu0yUx2KmKLc9J/Zb1scfrKueB02NdGHfzkb+SH4A5e+yinfJ52R/luCne8Dvi9noQNKmXXsAemzOljEU9T/TDg9OpyZFxB+abTS7eu5LEE/qa6Y52n85K93s6cvNyYa7vS3f7ZdllUx1QUMZHx/NYecA0/qrsWRlH+nU2rMdYaGVOqItNujuuUl9NLCRlnTJKSh2OtUXl8SCDYTlvjtwzy+fE3LxT1vI5O1XuqfNH8l4+6kfzOzbvTqVT8+5I2SfvfcX00jF5xXy/J6NQ3fsr//4PnhW2/fp7+b1/4We/9Hy/9a/94NsGvD5+ETrYAAAgAElEQVR5z1ci3Ywt/uqHP1k8Q91A8JamCmh9wPP9qhhYbRWw62o0FQka7w5N0QDshgpnq76cDwxfHAcPazMuzvdFk1B29q3N6JjRdbPu0fO14+CxXvfY72lbrmlHjKOb0fHnRDvx3kfsbluGCEbS0mPYYI4Wvg2w7IUQeKhjqCBAC7QYHEEXg0W1Yk9ksLAuI2wrmCbCVQlh72HbWGCAArmr2kByEoZgXcIcK7vwcbTzXfKaGFxzpGOweYLIRasWYxNs0jcEK5V7jKPjtkpIPTO/BgvXRMTeTWUcHOyatfPqRPcdCBJoNoGgdgJJlIWbsJ3KwnoVaMdQMcFisAWWZlmTL31G2pR4NCAHy7p+BEEzTZxYXwHSzgNoh5l3gQt0TSBXAMyB4G9ZWGEdQ6XqRKywFf+ayg9zMjC9JdZBYRY1AHyC6RyxO3YWacVkS8KSyqLOoqdW4Fa9RW7lWgt3a5HajLSKMAIfBKadZGBifeWFUfkulwrUK5oCq0lNghksMYi2CW5vWbfP0o+jI+IXkw1sZ5BaZo6tMi3yqjxBlRhKJZClVCdqc5vgDpaYVAWCNU7EKnawBa4k/e92dH0W9tVAbJZmJF1BDeXSUDFZPNmenwtmW7WjofLlM5MKCbSsuiHPyfgolb61A7VX5qTW8YuraTUlC0TNrOkOBuEsobq1xODqaY6bTHp4UchImPUzSH4GsB154cYV6+kZECkK511dW4yXiZgaezMxQAJwe4PxgjxJZqT6uwO13Q5MPiSGm6E2GjAZC2v3FdZQzYzpcukTG4h11USU3fZsUFgwbQKSZwZSZjQ1oHOuo2tTQ59TTccJUsh9wo9jdW0QWxA7buK2ZmDckFYi9R8A7tPxnI77A32ubkmzUBbd7jCR1ZhMZQspDZG3sB6hwaQd2NNYG9bYzC4XNtjkAX+YL+pTJWNF4+Y6TOQnosfnKF9J2U0agaQfiRk7rRtIY7G+Zh1FZmiVtbPoWGZP5UXWWQRoDIX0JbZ03PVAWNM50RiMDYrmYaxRmGIlH2FMtQO/KjcoupTZ8VxJzMbLZUq+ssjWrK/+MNVLWFjdQH0YG27ziq4TBttsAX/IGC5ew6Dl5Dogrqg+rkcxbEyi8uww9aOQaAlhjetoriePaf4xoY3riMxFrjUZMGEifpH5VLw2Fqi2GbE1ag5kJG8m4y0DJtExAHADEVLZMSM5mo820Hsj1YZ0K9mZawJ5QW3IXEcD11N9CrtoAmJl4QZ6J8SKN+C7PLWDdRyTp/tjMxHSyPtI+mLGLCwewowJFptR5q9mJC73KO1LbUyU58ZM1850LDlfzQpr1G5u5n6a6ViWk9P7puhYmnkb7zD6lvfnxFKr4aVLncvZPtIRz6p+d2qyJV3H8l5M877R6c6GxTEjVl23hMRKXR5m6OvdDTP/fl/dThncs7zxYGP+R+nLTz/UhuU7zQv8uZ/879HlCjFbdLnCmB0sJjhdl/iY8nhFTJ5KALAmoU8VKhMx8hsgwqA1AWN2iDBwPDsjz9YxeVQMhRWPpxyTvACCmx5L4omM2cIpAL6+/pi+pZzXXrWKfwnG5O714EXYO55Gan8uf5fHlmXpvJbJIc2OHyMAEi+tVW2eeYy5rUsd0ZhNgUfGRbucyeUYeR/n2qFU/lw/VLcpwcD+nvnbR/oeAMZsUUl+qn7Hx8fM2nQs3dExVe0Rj7buM12Xpdbp8tix9um8ZIyPeYIfqrt5n+d82U5vI3v2784XPSYPaZfu62W7Tn3W9TuFJDiVTmmw6r8vJf454ZV72TxatvWYpuvUj6e9ucf6RqdTWp73pYeO/33335f0mmB5/CFl6rakTAiBpef9vvekzkNQBNO1KLB3YP75WB46r+Xn5XXHPNDSBp3uey4rQ95/j/l6KmeDWm0YOhlzPid9bvm4xdSuqOoj7Y3cjpDnWrGjgnIXFAcw8yKLN93ajIX2O1ISGPnxeak9uuOsgVNfl0PcnoOgQzBfK79uWuZzat1a4K9GH+MP+sYTC+As3lAxKI49n8c8iMsVt1qNa83TYx7k2fp+WZbJc5iulK9ZynWZs++42/na+MtH6rNMxzpa5sPL7tH9faeu99Qby+7Msz/l850xnC6YGYV32rAcq5edf/nMfbBn757+vJPnqevufQBOF011XOZ6zxjeV84Dyyzpv3vANb8T6XexwftDbVjuc4Nv9m/h17unSNnga/UtYrb4sL/EF/0Gv/fsc9yGFue+w+f9Gd5ur/Gdw2M4k/Fu+wLPxg22ocZVvcd3do/x1uoWn/cb1DbizeYGT6odruMKf+PFO3hv8xy3Y4shObSO4pvebq8RYfEb2zdwUXUAgKt6hw92T/D++jkSDD4+nOOy7gjWykaCNwnXY4ubocVPXX6EMTs8GzZYuRGNDfA24rHf49d3T9FFGqLLqsMhVtiGBha5LMqfNDtYk3FV7/Dr2zcQksOTZofnwwp/6+V38WF/ic+6M/z4+Wf4rf1jbEODN9tbrNyAPnl8e3uFtR8wJI+rZofPuzOkbEoM6hvtDis3ok8UM/fJ4RwXdYeVGxGzwXZs8LTd4ot+g/OqxyFWxTC4GVtcVB0+2l/gsu6w9gO66LEbG1w2B3zRbfD2+gafdxus/YBn3QZX7Q6fH86w8iPeP3uOD7ZXeHt9jRfDGruxxrubF7jwPb558xTndYft2JSYydpG1C7Am4TGBYzJ4aP9BVImhuAxOiQYPG23uBlb1Dbg0/05hujwD7z9LaRs8Nc//zGsqwFXzR43YwtvEp5160mGxia8u7kGAHzeEbVmFyq0fkTMFk/aHV70K3ib8P7mObahxqeHc6z9gN3Y4HG7x2eHMzxu9vii28DbhJFjFisb8e76Go+aAz7YPSkxrbuxxlubG+xDjXfX1/hgewVnEs6qHtuxQR89GkcxpxYZn+zPsK5GPF1t8fHuAtZknNU9ft/Fx/iou8SH+0tcNB1aNyJlgyF5HAJ53ysb0UdP2q4+oHGhfHc24XGzx+3YYuOHMp6f7s/xxmqL2kV8sj/He2fP8Vvbx9hUA0K06KPHYazQ+IAn7Q6f7s/xqDngeljRs9C3qF3EmCzO6gEWmWKas8GLboWvrW/x0e4C752/wBfdBn3wCMliUw/ogkfKBo/bA6zJ6KPHfqxgALx//hzP+jWeHdYl5nVTDbioOvz29hG1PTqsqxGtJ3bmm76lmM7ocLXaI2WDLw5rOO7D5x2hIM6qAbuxRkgWl02H677F77v6BN4k/NJn78KzXu5ZPeDZfgVrKH61dhFvbW7gTcJ3bh9PsdI+ICaL/Vih5nu/fv4C33z2Bn7y6jN8+/Zxiem1JuMbl8+wD1T+IVR40u7wnZvHpe5vrOm98NHtOR6tOsRksRtq1J7eXX/gyYf45S/eQT96XK46XDQddmMNbyPeP3uOX3n+JiL38bPdGm+e3+KL/QZXqz2u+7ZA1WMyOGsGVC7ipmvxZL3Ddmhw0XT4fL/B4/aAPtJ49YHeZVerPVZ+xGeHDaFNosMQHSITsA0ca70bKjxadTiveny8O4e3CYfR4+lmBwD4bLfB080OL7oVhuDgbMaqovfSYajgXcIffPrb+Pb2Ci86eibfWO1Qu4Bff/4E5w2xhu+GGs4mvH/xHN969gbOmgHvnF3jm8/ewO99/AW+c/MYMVnEZPDW+W15P3ob8UW3oX7lWPQu+DJvtkND7+Z2j49359jUA2Ky6KNDN1R4tD7gMFZ45+waKVuETJseL7oVdkOFiuMF31jvSh8OTC4XksXXNlsAwBAd+ujx5voWH+8uULmI667Fph6Qs0EfHZ6s9vitF4/wt731XexDhef9ujzbtY1Y+RG3Y4PbvsHXNls871Z4utrBmoTH9QHPhxVCdvjo9gJXqz3e2Vzjg9srnNc91n7Ayo34hN9zITk869ZofMCLwwpPuZ7PuxUaF/HGagtrMj64vsJ505PxN1a4aDrUNuLj3TkueT6uqhEvDqsy19bMon59aPH2xQ32Y40ueIzBYdMM+Nr6FilbfH6gufrdmwu8fX6LF90KZ3WP/VjjGxdf4P/+7O3XXnc83ezw8e05MoDzti8cBY0L+HR7hk0z4DB6vLHeYzfWiNmUWP43z28xJoebrsWj1QF98BiTRT96fO1si5gttkMNx8/5yo+FV2HbNxxbTxslQ3D4xqNn+GR/DgOUMrdjjSH4EvZQ2YRtT2Rwj9cHvDi0OG/o3TlGi3OODb8+tOU+azLaekQ/eqxqilWXd8UQHcZoUblU9MLPmoHWA12DDODJZl/4DVbViN1QY9/XeLQ+4MV+VWKEPf+unrc9vcuGCt4xn0TwZROi8rG0Zwgefe/hOV7aOUKepUQkiKLjDaBsUgyDK/HhThAtioBQyAa9j4V0cBlaUNeR0GdLw9tMut8SM56iLWgz5xLGkVypEkcs2uF6k2vS/6b4cilXYqkzv3MNEwEKeZ4gzCz3geHjkmeJoc5EhmfcXEJO0rTpgCLnJoR4RVpNDHlL10mowEOY8MsmiGwizDYYTmyQ8F9NkKjr+qP01Uw/1DGWZz/xVv4Df+FPYD9UiImIUYRMJDLbqxgDIVp4lwoRh5yTHdJxnF4qlpkjJSax76pCSpHz9LISRtZx8OXBcy5hHHwhZAgjEUwA853oFC0xtDYjk5FMO73GgNlUXXlBaFIKnawlNkxpgxxLyaJpRoTAfVFFhNGVuDuprzCICsFDYiY3eXFYl2cvqhTNpDWJ6cUncFr9QkiRWD7T4FiLcnoZCdOgrZgd1GSCxyq2UFclxGGCyiLR9dYmqreOgeJdLcMvLHnpxEFextNLqbCJGjCjpEF1TurU476i9jHTJ2ymv5jKsDy2qRyf2ix5i44oMkp7CpNpsEROERh7KLATztu4jNi7KW8mqsjJUF8wQ2Zpv+zoyYuZ4bjCzEmTAnBtQAoWmdk4hSCDcIUo1xUosRAdpKmNxjN7pOXyMwgGzKysebQEix4WjI+Sn8t0vZPd7zxnwBTHprB7BkOQYYEOM5Nhie2RHyUmBhEWRmRD7JbRUP4AXS91GBQBjcko+phRPV9e9afUSc7ruCL53BCxSz646UfTZmqDfDcgCLUB1UGStF/Hgy1ZLXU8VZPmsG+fCnsmuaz4ORjm7JPF+7GKyAc3sWlK22yme5n1Ei7DjJbYIIXJMprJQ5BR4MgmGjrPY22CnWDUCRNTpejI8jwiuBgvGKyCRUeGOluug+FzUl6gz8JuSVCviTETBsirCIx2gqHyPDH9VLfCFFknmI6h3j7T53ZiAAUwwaplYRQM3S9DydBugu9N7TXj/BokZtiMDAlXyUQzMeNmlZ8ew6TqklH6StqJYJhYBkCkPGxnkdaEPTbBzPoalvoTkdhJzWim/AtUn6HwFcH7Sx/ys1PGO3FeFrN8zEhlSXttr+cH1TfbPIPbg6HqAAq0FmDobp0LvFzghpJ3YT7tmYlUtTe1CW53HEn0kJQqgmgDmOt9WjouUMoC68wo8EuCZJsCDzUZFFKQGZ6cMXtOdTyiCdNx6bPUpln4ALG9AjpmNhtMDK3CCusy9xvpkVKfqvll5s8iMvVzZqbb4oUz6jnNuBtSANAYRoaKe4LtU4U5j4yiq1pgnBlzOKv+CU6YsZtmg1m871EdzMTXaI+xma7PBtN7Z+kF5oykz7TXUC95oE7p/skGsOp3hcqayp8ZWPqzPmbUMX2fvk7at8xD3XOKmGje1hNlHEuvYD482JOq63CsTt9j+n//7R98HGL77nv5/T/55cdYfvPP/uDbBvyQeyxrF/HO2TW6WBXY21ILcgmd0/C1Y8QkiWFBArtM2SCvupNzOmUDrKfvxzQyl9e/DIon9VvV8/NLGBaVN53Lzd12NVWYDFreoZ+V1U7tPgYvexnMLav6HmsHAIA2nO8n61BG9wzO1IwTEYcqoqrDjADl2P5IzmbGQHeqXNksAAB3oeCrRYv07n3HylxC3oQZNC/G8VS+Ok+7kqCRu/3l1unOsVk91Dww9Zz90FYR2aejY7H83Vp+LqlefG9UGTUHzfmA5UbDdM2y/vFOuAXAfSHjtyxzWWH9XSdP9btDLrMg7zgGZy19VB3p71M/cJnlDHSqj/16A2iVkXLqGWMNS7gjLJZi2IPnzpFrsmKtlPoBoA2FJs2bofukmdqcfWSjjP7C5/l9xQCSG/J0PTAZZYvrUavx0HXmembV71kkRrAYw6zGkhcfs83SQAvsXOs+ICNylo8cl3YnNojSXcMPXD9ZzBVGS4CkGZb5AsUYBsAL4jwdXy722NBbdkw2ecpfL5L5HhPNZFj7eZ1MEgOLMzaToV/kIwDaSFBGYblYtymB4seV0SfXGJbgkU2abEFjADEYDGyvvisjxQQDUXIt+SropRhs1F6+RsbcUH/YzkwOEdkkFXkLLstt7XzR+orJjlOddLxaqbso0gzKyuAGu25qu8SeyqMv8Zp36nbPe87t5m0xnfoi7dd1j7S+sWqzyw1mXq4YaHK/MopKX5bL6X+rxxg0DlP9VVl6ky2ruo2qDfJ39lIw5Vh5VSZ1zcuMnyN9d9TQKZnLpNMnVdtPvv+XZX0PE22RXskw0/X5XvJ+1eMPzfdUesXrX6tPvirph7nuL0k/1Ibluevwjz/9JXyrexN98vhG+zm2scUn4wU+7c/xN28+xrOwwZXf4bv9I7zXPsO3D28AAP6m9Sf4aHiE29DizfoGv75/A2+3N/i8P8PKjXhSb/G16gbXYY1fuvk6fmz9DLvY4BArnPsOffJ4r32OmC3+v+2buKr3GLPFk2qHD/ZP8I31FwCA7xyucFXvVMyXxcoNeDZscD22+FsuPkSCwYfdI1z4A1ZuRGUiLv0e39q/iV0kKM3TeoubsMLzYYXWTQvXq3qHxga8UW3xazvqhzeaLZ4Na/ydF9/Gd/orfD6c4feffYhf272FXajx7uoFzlyPbWzwzduv4UmzQ588HlUHfNydI2WDtR+xDxXeWt1iZQccUg2LjE/7M2z8gHPfIWSHQ6z+f/bepNe2JEsT+qzbzenuva/1PprKJCsbSlklVEUNGDCA+gdMYFSiKSZMGDCiExIjQOI/IITgBxTTYlAiEVKBsiozoyLD8YjwiHD319zmNLuxjsGyZdv2vue+99wjMiM9FSZd3XN2Y/3ex5atb30fntV7fDVssdU99q7JMizX4wpX1Qk/PV7hsuoSpFZn+OyX/RYfrW7wVb9Boxxe9ms8aY54kaCwf2PzAj88PMMnq2u8HNc4uQrfXb/CY3PEP7v5GLuqw8lVOXaukp7gxMpBCw8XFT4/XcIFiV3Vo09wz+ftHV4lyPOX3RbWK/zb7/8pJCL+yYvfxtoMeNoccGsJqvmyX6NSHmOSRfnu5jUA4ItuCykiTq5CJT0CBJ43e7waVtAy4OP2Gl2oMrya2/2Lbpdhx5X06L0mKJ3w+GhFY/Pnx6eQImL0Cnvb4JP1NfauxsftNX54eIZKOlyYHre2yVI5lEfAL0471Mrh4/U1fnx4hBgFLuoOf2v3M3wxXOCHd09xUXfY6gE2SrigMsS6Ug69NxicRqtt/s6xiY/qI+7GFhszoFUWnTd40W3wfHWHVln89HiF729e4dPDY1xWBE892BqdIyjs05ra/6Q54PWwRiUdrocVKukxBoWLiiDlGzMgRIEX/QYfrW/wk8MVPtlc49VAUNjB6wzfDFHgcXNEJT3ubJNhvb9z8RW+6jd42W0gRMTajNiYARemw2f7xwggdMPKjKikh5Ye+7HJsOer5gSJiF+cdtAyYGt6XA8rSBGxNQOOCYp6UfW4HRv8q1c/hxQR//eLT2CSkXdR9XjRrckekwTR/mh9g1o5/OD2WWZ+brWFCxKHsabPUeL721f4k+vn+L2rL/Hp/jF6pzPs7ncuv8TekhzRyVW4rDr8ZD9BYT/YEFz7p/tLXDUdQhS4GxrUmp7Nv/f4M/zRq+/iZA0etyfsTI+Dq6GFx/fWr/D/XH8EHyR2dY+vjht8tL3BF8cdnq32uB5WWdbFR4IhaxnwulvhvfUdbscWj5sjvjju8Kg5ovcGNih0lsblSXvEruoyXNomKKdL5Z1shUZb3A4Nnq6O2JkePzteQIqI3mm8v75DiAJfnrZ4f32Hl90Gg1cZruwjQYqViPh7Tz/Dj0+P8KLbQMuA91e3qKXHn1w/x0XdZ8iqUR6/tXuJP371Pi6bDh+vb3Lf//D2KVyQ8FHg4+1Nej+OOTTgZCvUysEoj8NY4731XX5ulQh41u7xk/0jbCqa0wzXftwSVPK729cY/PRTfDO2uB2aPNbPVnuMXpOUljPQImDwGh8kSP4YFE6uys97oxxe9ytsKoKZHm2F56s9Pr15jH/t2U9xcBVeD2s0yuLkKjTKYmMG3I0tXvcrfLC5TWEKlP+l6XB09Nv3k/0Vnq32+N76FX6wf46NGbDVA1o14ot+h7UeYYPCi26DWrs8J6SI+b3+vN0DAH50+wS7msJEjq7CRdWhkh6/OO1wVZ9wtHWGTKv0Lryo6f3wulvh4901Tq7CyVb53NOWYLcvug0eN0f8eH+F72yv8aLfYGt6nFyF729f4Z+9/PAbrzveX9/h8/0lYhTYNT2M9AQdVRaf7y9x0fTorMGT9oijq9L8puflw7QRfjc0uGy6DGPmee0ihT4wA/sMCjvSLhtDYQev8NuXL/Cz4yW9Z8cKz1cH3I4NxiTxpWSAFgEHfk+2J1z3LbbVgN4ZDKnfpIi47tusuy0Fwco7a7Cu6BntrMGj9kShEUFCiYhaOxzGCruK9Ljv+gYhElx48ASFXZkRh7HGYajweH3CywOFkEgZEspMYlsPGa5fayJQHJ3KkPvauCwV1VuNbqhg0sZVpX0Bn1VZE1yKmDcHxwRvLWGuzGbuEhnjOOqkya0yUSOnEARq4zKkmeGvAGZM31qTLBRDVYUgvXO+TwjS2XaOGMQZMVcmKScpK0KEsWxXMuJlzPdxud6TbBRLPTGEeLlh7x1pamf23nKDPLUrJjgtinaWcN0yv3vx4UU7lrZ8SYAoBGYbLkto6z2HQUSG5+ZMf3U2+2/Srzh9q6GwH/3BRfxH/8u/kQlzTp7cGrV0MMLj2q1ghMcQNGrpcONW2GpaTNy5BkqQXmTnDXa6R+erTJ7SeYMhaEgRslEnRYARIR8/OnrRP6qO+fMQNHamx51tIEXIi29gInRxQVFMnHR4Nazzj9IYdPa6uqDQaAstAqQI6D0tKIgMZYLx9F7TgssbrDTFp/Ve0+JiWKFRDpVyuBsbrPRIcUDJwwvQAp4Je8agUUmXCV6W1wKk3RmixBiml/fgdP4xKAmAGuWyfujJVVnzkg2mRrms+cnyKlz3AIF9WmQPXkMLiqM4Wvqh3tU9bFBQIkxEG1HCBZkJL6SIM81OCXpB985QTEacdEC/Om4Qo6D4KAgMKdYkRpGNBCZU4RgYbgtrcIrULv7B4jwq5TP5EGt4jl6hSfFunPge5yWaiuotBcXInKyhH9XRYFWT9inDvFXqc24zxyuOTqExjlCXXuE0VDDKo6lsjhkj+ZOYdVg5D5Z84R8ONrYYUu4Daa8qkXQPR4MI0t/sRoPGOLj048jSKSEixW56DI70UAHk/FhqRRRziK/lfHWKs1EyYnQqxzI6r3LdORanGyooFVBp8tr7pIXKiwQAuZ2MZOB5QXlSHSrt8wJEK/Ly5YUDJhmXrqsAEdE2Nv+Y+nQPgHwfx+lUhTeZ5UMYZg8A46jQNBZ9V6Gq3czLPA5z+L2zClXtZtB+LoMXMQyRFyJi6AyqxuX4H44JihHwTsFULi+AtJ7g/QztLxcBPnmpsl5kKkcnjUnWQuQFlHPEkq20zzFCTBzC9eD/3hObNLFhp+c8ed2UDpkBm+H63D+SdXaPBrJxEIpcHcFNjNsz2H8EsWIzXNxKqMbD9wTF50VMGBNmL0HQhV5AymUk+LnADCrOrNnczgyxF5FgzuUiScXpXuA+VDzS/bAye01zPsxUvYRtW0nIgk7R9QxvZTj5EqqeoMoiCoLlMjI/MVoLKxHrkGG7gj277FFkVluGARfHKF8QjJM9XZIhzRNEGBIZxstJJPg3wVxTvzB7qEeGWQZNnsOYYKv8HZI8ib59M+rjTWkG72WvKZBZgUXy1EqLSfOR0fhjggOrdK+Imf2U65494FHMYZ0cHpBdsoAcRIbQRhkhHZXNl1BnILOSSocZQyrE5GElht2YYekzFlWe3g4z6ClDkBmmm5lgE8sr5x9Vus4is9tmb6+Y98WM1VRM5WTWc4kZLDVDoRf3UKOK/l/AYQu7ZmpHway69Cjdk3bh9IBHtLyv5K5bAlTued2K/PI4LjytJRBIRMz67bwnlvLMbXugrkvIaVnGvevwlnYvjt3L5x3yeLBu5Ri/JS3r8cf/468fLtp8+HH85D/+C4DC/ue//rYB33KP5d41+Kevv4+jrWaLQiaDKBf0/N2mxSVLlLCx01sNo0jEXgoSgleCgFd/PHyQF7/AtFtTpR2zbnw/L8yXC14WeAeQd3ikDHkh3BiHEJFF53nnx6iAweoZ26kvdog48UJbJOMmFHqVlXa5HBak5/MAGS2jU3mRyDtvJQxQqfkPMEutTDtWmO2clTtNvPjmuFXWVOS4TA6WZ/0/3nXjhXBVOYyjTruAEt6T8LtSAV/dbKBUzAvyHPgupt0zIeIsaJ4XaryzB9BCNgJYrQiX9OmXTyBkgFIxx71yIH3uc5ME0P0UIJ/L5LjHdF0IFMjPO4G8WFZqqkOOXwQJsEsZcTzVma2RFu0+y9Tsj00ev7xrKIq41xQHK2XArScDQ8iIurY4dhXu9m0eQyBBsnlepV8S1v9bQn7LsebF+50XWUz9eGigVMDpUNNiHrRTycH3UkYcU/1ynCvD3SKSAUCLfQA4pDjcg20hTchyPIhitqAXSXQ9x5xGAVV5DEHg6BIWW9KCXciI/lhNO4V6HqUAACAASURBVKUpllTIiBMvqsOU58kmfUrF8cBFP6X7o5dQjYOIAnev19OOcDqXk4iQhnayj7cmzdOy36cVg6w89q8qyNrjeNPOfohV7REdtXcMFMt8PLVTP6ZYQnuoUryqmBZWEVCtQ7+vyfjQFHPtBnoXSBMw7GmjTKgIm+Jm3UlD6AjX6TkRA8NtnaRYYK8hVMRw0nNIJ+9Yp/JsX81io2YGkUz/FQnd215PBl2CeYZOAzoi9NPinuPKvNe08F45MgaTBEhu695M0N80/0Tt4e+ov4QJ8HsD0TqEk56eD5MeDI5fsxJxkNOC1otk3GE6riNip/M4xLKtQUyyQ5ycyDHt3F+5/LFYSeoCLu9FisVN7RxlNj6ydFKngIbqlmMwebHMkkhWZvkiVAkqzfUMgvLXEbFxwEixt6iTHEsRMws2KlPMJvdzVDHnKziWMs2PaCIgA7WfjdQKU9xowBQXO4pJVimIbPz6irwxwkmEmoxavyYDluMb3cpD9L9EjGVL8koAGct500FwPDLV3a9iNoZ47ruNz8Z6rNiopzH0a+6nNCaIU3+m/qPG03wQEbBXfoqxDAKumfokO4jEVA/XxCwhlY30ZjKSRUSOuWRDi59h4QHbzA3CLNHRYh4v2k4GDF8jvECsS/g08nsytjHFQk7WR+47sGHGx1HE28a5MTkNB/gOUX4ojcPintytNWaGdFnNswbn0sgRi+N8rFxGsSFbvEKLrpgbYA9aimwQLy3MRaZlsdzW+6fut6X8fy5uk39nztX5obQ8/zWMw69rTP4m/XrTt9qwrKXDZdXhDy8/x8HX+KLfAQCe1Xs8Nkf8i/37aJTNEMQfHx7h4zVBmX5yvMKzqssQzL/9+Gf4yekKl9UJvTd41a9xfWqxqiz+/gef4dP9Y2zMQCxoroIWxIAJAH/nvc/xZYJFXvctfufRV/js7hGkiHhv/Qq3Qzvz8HXW4HK7x870+LNXz6BkxJPNEYPTGL2C8xK3xxbPL/YwykOJgJu+xboaM2SOYzmZofHYV3j/8g4AcNc32DU9fvriCuvVgKtVhy9vt7jcnNBoh9fHFWwy9j6+usF+rGFkwG3X4NH2lOtYp2u5vAjgyfaI42iyh64xDnfHBut2wGBN1gEFgN2qx2mo8N7FHnd9k5gbAzb1iNuuwePNCa8OKzzeHTNj3OvjCu9f3mE/1LjZt3h6ecCruzVWzYi2snhxs0HfVXj2+A53pwarxmJ0GlVil7RWw3tJBrCIuNh25L0ayUspRMT+2KCuyYv16JL6+NNPnwMAnn90jdNo0J1q1I2F9xK79Qk+kGfPeYm7fQshgKYdESMyREapgNOxganIG9ZdtxCVR72ycJa8QENn0KxGDH2Fph0TsZLLdR7uasBK6IsREQStqSqHw10LpQP61w2qywHBS7hRQVceKnmGQjLgNrsOw2AwHiu0O4KODb1B/5Mtwtqjueox9hrB0uMvVIBK8WwhGYmqIu9TsDITJoUosh6qHxS8pcWyXjkyNpyA3lrY6xrqwsL3mn58DOmvBi/g7iqorYU/GMjWIViJamUzk58bqE4hxXDplYPbG+ithburIFbkfVLawfUGsiIBIJf0TWFCMtwi/MsaceWhV47Wb6NC6BR5Oi7ttIEwKsQh6bWuHJEvyYiwN2RDrC0RZx0NxU9GJO3WQGRTnYZsHfCzluK8Puwno/mkobY2k1ZFJ4EXNS0er1JeAOBSzFrts8cLX9WQTweIL2rgykGkxX+MAH7e0MJURjJMrg1wZZNRBuDOkHF8OQIHQ16QOuR4N/WjFv6Zg6gDcNSkL5piCvWtgH/uKJ+jgtg5iNcV4sZDXGvElZ+MVRkhToqMhtZDvjIITYTsBPwmQJ7SQluCjJAIyGsDOQq4C08ELkzoIgBxUok8hjxi8lZDjQJuG/JiXL8go9FtPfS1hl+H5MEQ5E0SoDycQPNZC7uLcCvygumThnAC42MPuae+YIOj+txgeO4h9wpmLzA89ah/3mC8DDmW0rwwOX5PBMCtI3mpXCInqSLMXiPKSFqOEdAngXEXoQZkzcJQRegjeZvMnZgW6RHwTbo3GRjmoGkRLWLWnwwKMHtmsiFPmdmT9qbwAr6NUyyjjtBHjeFRwOpHlJdvAFF4r6Ql7cdQJ+3OFjAH0CbEANIKVcC4izAHAXMAxh0gR7pXuqRrOdJ1rFHqG8AcyI3lWjpmjpTvcAXonj6z3qV0gN2kMg3l7VZ0DWk9pj2MFVDd0H3BTOf0ibrErUi/ctwB1R2VzXnWNxKn5w+trt+eqj3lGwXVn40paYHxgo4FA+gj/QfSf0FjHRVpq+oT6SKSxiNQ7clADxUZVDQukx6kr9iAnPJsXgnYDV0fDGCOpBOZ41EjIB0dA6hM3wjIMWlbStK+RARcK2b6kFnzcoxJO1KkOk9GlvA0Nzhe1DUizdmYiZaUJR1RXwuYI2DXVF/pJ++u7ijPUGwkBD2VI10yTgO1kzVy2ZiVLqZ7xMwrx/9ZxzRozPQvgekZYG3RoAHpMelLRiBKAWWnNi29lFGS3iRrSbIXVHD/l2OX5jJ7X1k3tPROZh1LSXmW5QlP80Z40hQVEQgqUv2UgPSF9mnei4iAmDz39z2QdJ7/ixAnT27p5UyJbddJv/O+Ib7sp6Xtew83+8B3EWPu31xd1gldpncwPH/w9kv+UtJbDfFvcfpWQ2Hf+/1H8d/5n/4BAMAFBbvQyhvTW8AGBSM9bJiYFVyCRvJnjvHiWEiCFiYonJpgnmWa9AYZRihTXg5j0DlOotSnK3XlJCLGoLLXlGGcpAemMkyzJCXSaYHPyQfSfmRYZ4gia2MOCSbLMTnsOXXFbnglPVyhjakXkEiXKLy5XPYKc+yDFBO8cwnr5NhHo3y+nu/JxnHyKHN55XeG43qGZ6a6szYd57XU6luWVXo0uc/4WqZW70ZaATDssfQMq3t9PkFImVCJYyDKckovNB9nKCJ7rkvvcIwie6V5EwJAhoYy5JXHgt/ZD2oUYr7RuKRR57pwGeWxsk4z0pvcjukcezGZsIg9sucIeeaxIfHe+Rgx6/tl3zGNfFkX/sx553iSBKtlCCZ72LmeUz44G+9S6u2V97G3WjLENTJEU+XNgDKPUpePjP8UmyMXv9YLz3oIk2dbijj7JQppY6fsi7Jvls+EKPoSALyTUOzlBWZ07jEge5tR1Ju98bP4mHTN5LkV0262RPZ8z8Y4pD5jGCama6Z6FNlz2fysLD3WomjbYkc+w1JVQPYIRxBjMreFr3UT+zK8mBiQVaHvV3jEqYMXv58CE9tx9oiICXLKn4EJfloSmnCe5UKr1A4sr8serMJtwOUw5I+PJdbVe2XlFWLKm73CyVMh+BwwkQoFTF4vPh9AcFke71T/DHEFCm9mKltx/2DyjJTXIf1XZ65Z9nPqz3LhLjwyU242tAQyI+s3TQzt5O7LnrLCYMgQ0MW4ZdipRGINpvpHgamfyvEsUzmH+L2d2sb3ZSbi6ZL5M7GAg4IhlFzHct5xO2LxtXxey8/l2GFq+8xQKgyVyWgo6iWKvBZNn+CbyWCSycOZPLcPeuKW9c3P5CL/wuA5a/jgzPcz977L9b9sOlfeWz2G58btTL5n7zt3zzfN76G6vSGfc/m+k4f0gfTP/4dfP1y0+fDj+J1/9KuHwv7L/+LX3zbgW+6x9JFiEV+PKwSITBpzsDV6b7Cr+gyBvenbrPsXosDGDDh5It9olMWrfp3j+ZSguLiNHrJO36YaYL2CixJGevgosdbEssq6iwECRnq87DbYVLSFdzc22evokwHHRh9r5QEgIxBx0pBK8YcuwXU5iP9kJ6MpRNKIkuAg+jobf52tM3lA50zW8LIJBqzTm/p2aLIxSeQTVTKaQtaVMykwXnGsn4j5Hs5vcDoTh7BR1Dsipbnra1R6ghbz8eNo0BqHPsUinkaT4/WUjFhXI/ZDhXVl0afYw1Izr9Iu624xE69RHlL5/J5ig1EX0NPGTPcdhwohSLx/Qd7er/YbaOVRaQ+XCBeoPiEbpBz/OFjSFeS4wBgpdm9McWXbZsgaZkqS4cntq1LsIBuLNC8i1vUIIwP2fV0QD5CWmA8Cm2bAaSByB61IRoeNcTYIB2sgZchxiTy2j3YdTtbkWMW6ssQmnIx4vs7HiXyADXs2SGpDmma18blNo9Oojctlb1d98hCHbBQzhLqt7cyzTXBlnY0kgl7HbDxbp7JHuK1HjE5nmDXDqGMUMCmWlGMoAWCz6jE6DWsJ4kmEDBSr2yVdNzb8pIwwhuDZeSxrep+Mo87EDAzTNpVDCCQ/ZAwd321PkAK4PTTZQK0qj3GkeUR1iKiageZ7X2XDhjcZyCgn47JtBnRdhfVqwDBOmm5CANtNl6DtNFbGOPS9yQZQw3UfiJCCiBsmY/hyd8LdsUFIUkRswIo0v49dDdKaIzh5044UZ9k4OCdnUE1liKHZWYWqJe+z1h521FDVRCTB0PGqpfjNcdBZ9y3yxoTxs/hKralu42AgpEfwClVDz5+1ClXjsoySECFruzE8fHtxQj8aOEvjqg2hFoa+gkwkaMFT3GSzs+hOFXTlYSqHvqvQbkcMg8kGt0lIB97AcFbNNmNCkNCJidt7meHtzuo8xqRzJxJUXsJULm8mcD+SdhyNpTY+9w+3KwaR7hP5eeDQASlijj3lOa60hxs06kuLGGmTiTcdpCB4eI5nNYSA4L7i8IQQBNyooXSAqRzGwWTpKpbZ4s0SirWNCFZmNIRnGakUo2t7DanJIosM3xcRblQkR5XQE8FyyAGSBl9M8bCe4mSDoPsNIQgAwFuCZYdeQzIKIG0m6NrBH8/RTL9bEpUnyaaISToIAEREGBRtTjgJmeJqY0TeBJCJoTo6kp5CQpkgACJBhmPqJ7phMiZjNjxTeQFQrZ+kqVIe0ckJTs55pLhkqpuYIOdBTKzXSXorG+t8TYZCg2DSvMEiMG2M8HFmHWZoNZfPZekwyTeVhiDHj5abKLyxgbSpUR5n2PXiXGY4XhqZfKxsHy8SyphSOf2fjXnEBB/mVBo2/LmE9qcyopg2FKZ6zg3ZXMZib4HzKQ2pmQHM7SqMsyxRU9YvnSs3HpbpnHGdyyjymH1+BwOzrEKZHtoLmJ089/836a98+lYblo/1Af/e43+Km7DCGBUq4TFGhQCJY6ixlR36aLLn0QiPY6hgo0YjSUibP9/4FdZywDHUMMKjEg4ybcPd+jVqafOPsI0KCiF7lG59i63s0UeDlRxx61tcKGJi7KOBQoSHyP8B8m7aqHCljwhR4BTqXEcpAhph8doTe5oNVEcbVTZOOTWSFlkrOeC12wAgw5XzslHBRoVaWgzBwEeJRlpIEWCDxilUWMmR2iQC+kBkPezhXakx5yMRcfA1jPQwgoxrGxUaadEHk68ryy3Jk6jd6Qc29WMjLbFRSiJFWskx18lIj1vX5vrZqLBRAxpp8dJuUMvJk8xjY4SHxzTeB5+M7eJ4LR2GtGU9Bg0bFJ5We0gRcXfREAEMAmzaCh6ChmGvNATWCffDZE9j0NkzTeRGybBRA06hyudtlKilQ+crtGrEEPSMiMlIj0o6KAQc/KST4oJCq+zsfr6eN1fImCciny4RPTFRFABo6bFRA7pQofMGEjHfk8XZ01s7QGRjlevNMb5GBNgoYRKRlAuK+kcSI23nTSas4mMuStigMvlV5w1q5TKCwBV9wORQZVtaRRs+zCpM46bQKIs+EWNVCU/EqAMAuR48R6hPIoz0OLoqe9UzGiDVtZxPjCrgMsbcn2FGmMRMygBwt22gBZFsVcphLBg/Zdq0AohQidOUn8wIiEY5HFyNlR7RO4MAkedZo2xGMnB7uS+kiHkeEmPw1Dfc3o0ZcNjUaWPN5XnAc7jfTXnx5hx7y7ndecyK9jBhl2aCKuXgosxIApE2pTjfpfQSQ/zzM1HEqjNBl0ntOYdE4f7h/Dh0wKZ5YaTPzKzM5MzzQ4oIuyGUgZEe/ZbmKTO28qbdcp5wfZkIionMeCNKFG0q0Rr8uS5IvHiTp1x0lX1Urq9YEovP8dgA8/UeP8suyDxWHLtfrtUYqVFu1DFShK/hDTktA9xq2lATIsK3E6qHCLGQkR0lq6VK3mff0jkqG5nga4lYYPbNcq771F4mantoHRracfY8c79V1UIS6GskISLCatK0LlNYTVwDjFQoURgl+qFEW5QIFj7GZS3l0UqEhVIBwUxt4eNn5aTSffk/G348j5oFQqW0dFL/87dyfs7W/A9YLBkpIEHSQWeMhHtSZMV5UZzL/bmoH587d/90rLie708GUlzeVpbJ/5fSQ+9o6JyVLEvz9lxW8Uze+Zg4V1fM+vRstc5lCtzrw6VH8KHbzlZ8kWZe1a9RpbfV8W3pm3oz/1LTt6GO3zB9qw3LW7/CPzn+TfzZ8T3YoPBhe4POG3w1bHHdr/Dd7SscXY1H1RGfny7x0eoGPzk+ghQBn6yv8XLY4GBrPK6P+Px4ieeruywB8bQ54Hl9hxu7wj+/fh/vre9wclWWYQgQeFofIEXEp/vH2Bqi237aHPDZ/hG+u30NGxS+7La4qLrZgqxRDjdji7uhwe8/+gVClHjRb7DSY2aLvTQdfnD3PLO+XlYdDq7G3UAeUF5AXdUnaBHwpD7gz/dPs0zCdb/CHz76HD/rLvGi2+B3L7/AZ4fH2Nsaz9s9tqbH0dX47I5o8AevcVWf8LpfkydUW3TO4Nlqj0Y59Glx9bLbYFf3WOmRmEZdhWftHl91W+yqPklvkHGwtzW2ZsAXxy229UBl2hpHW+FRe8KrboXnq332+F73LR0/rVFrh+9fvMSPbp/gw0R/f7IGH29vcGF6/ODmGVZmJKr/tJCrpEetSWqkSpDdnx0u4IPEuhoxOGrDs9U+SwF8ddjAeoV/8+MfIkSJP/ryO1hXIx41R9yOJDdy3bcz6OyHScqBZSyOY4WVoQXsk/aIm6GFkgHf377CrW0yzf5hrPGoOeFFt8ZV0+FVt4KRgWQklIeRAR9tbrAzPT7dkyyO9Qr7ocKHuzscbYUP1zf49PYJjPLYVgP2Y52p5SkPjy/2W7SVxfPVHp/vLwEA23rAH159jtfjCp/dPsK2HtBq8nZ3zuBoE6Ny8j6zMdEai8ERvbwUEVdNh8NYo9EkUdB7g5enNZ6sjmiUxS+OO3y8vcGP765wUfeIcZJXqJTHk/aIL08bXDUd7pK3/K6vMznWOnmDV2ZEjAKvuxWeb/b4xX6Lj3e3eNmtMXoF6xTW9YhT8sherTqSLbAVuhQ7+jeuXuVYaQBoK4uNGXFRd/js9hGYhn5Vj1gZCykibvtJbuTJ6ggfJF4c19AqYFsPuD612Zt+HCs4L3HR9rjra/zeky9RS4fPrq+yQbStR7w6rjJbbaVJe1fLgM/3l3mhzERjp4JG/5PdNf6/14/wu0+/xM/2F9nDDQC//fgFDraGjxKD07hsOvzs9iJ7sZ9tDhAi4me3F7hadfBR4JCQAwDw/tM7/Oj6CXqrsWt7XDUdbvqWJGE2B/z49hF5yOuR4p4v7vDVfoMnmyOuT202lEIkLzrHaD/ZHHEcK1zUPV4c17hoCTVivcKQxuXJ5oi1GfHiuEZrJmmBGAVW9YguIReOQ4XLtseu7vHytIJRAd1o8GxLkhJf7jd4f7vHqxQHzgzFIQr0VkPLgN967wU+3T/BXRrXJ6sjNmbAD14+w6ahDaLTUEGIiN969BJ/+uI5LtoeT7cH/MndDv/K4xd4efMIzhPM+YPdHWIU2JoBWnp8cdyhsxqVpuf3OBqS6ZEet4nJ+Wl7xE/vLrCu6B0xeoVuNHi8PmFwmuRJEiM4AFz3bR6rEIHnmwM6ZyCjyOzQo1P4cEcoi9ErnGyFD9a3+PxwCZXGYlOTUTVag8frAz57/Qh/5/2f4uQqvOw22Wg2ymOtR+xtjZuuwZPVEa+6VX7PPa6PuB5X6L3B57cXeNT2+GRzjR/cPMNl02Fneqz1gJ8erzLL+FenLWrtcNM1eLqm38mXp3XWngaAH10/wa4hqYuTNbio+yzZ8ag94TDWeZ4IERE8PfNSRFyfWjzbHHCyFTprMDqFbTPgvTX1yS+OOzxtj/jJ7SU+ubjBy26NTZLY+J3LL/F//eI733jd8Xy7x8/vdohRYNf22VivlcPP73bYNQNOo8GzzQGHsYaPAn16T314cYvBa7w+tXiyOmWpkcFqvL+7gw8SB1tBgDabVmbMGxt3fQMhJgbvfjT4nSdf4af7SygRcRwNnm8OuBuaPCepXoQaEiLiyeqEl6cVds2A3mmMTmHXDLlPS7kRfhZXad52o8Gj9SmhhxQq7WDS+/ui7YnzoWsQATzfHtBZAx8FdtWA26HBoa/xZHPEi/06k78xGmrXDLBB4tDXqI3LSBhGUdTG5Q2K3mr0vYEx5PlWMszuKUM+MuonkQBaq6CTV5tJAVlexI4apnKEtEgkepy8l2hqi2GclswlgiR4kujQOiQUySQTorXHMJi82SAEee9LQj9gMvhprhebKUm2JGtcyzhjzwYjJRLXgkqSJyiQRsyanVm0z1he5QYGEwOWYRB00XzDI/hEmijvG89L/fG53Eic7UyUZZ/9X4RD8D3n3J3nNlR+k/7y07c6xnL12+/H7//3/wF2zYDRKxwTtKytLNbViC9vtzDGwSYY3f7QZvbP46nOELChN7jcnXC7b9G2I5yXGAYDf9IQJuDy6ojDsYFM2kbOEVxtSHCazeUJfZIacL3B+qLD8Y6YO5v1iHHUOZ4nRiB6guNUlcfx1QqQkWA9LpF7JPY9ubUQkh4WNyhIE2YsrTEKIh7xAhgl1C5Ja/SKIDc3FZFYNB7YG2DtIHRAOOpM8qF2I0GNBBB7BbkiWFtwBE0KTIqSIDmZZdEm9kwdITpFu5AusQ8ynMUQ7EWtHXyvMmugMAFxkBCtRzxq+j9KInPpdC5DHBSwc8BRUztMAPYawgqES2IlhA5ULgt3O1lAaiLiJmFQ+FqA6lsRVEesHaSKkD+l8bJPLWAlsQ5WlCeaMMXzRAF5UIAAAjMsmpAhQLKXCFWAiAJ6LxFMhG+JSCSaCNlJhFWg/3x/pm4XUAdJBBbbhJVREdEEyKNCVEQE4nZEZCJHIluIhvJnQhG/8RBWQnYSPrVf9hLVtYRfRdgdkaaU9PahoM+PCjSuVkxU8eldrToieZFOZPIP3waoXhJZRxuhDwJuQzT/SIQmUROMSHUCfhWheoGQ2DBDPTEVMuGIdETC4tsAfZJwqwB9IKbHKIGoY6bah4hUfgCCnsguzJ2EryN8S22To4B0gBoE7CZkIhUiH6Ft4FBNQvOqp3nsWqK2l6OAb6bPoaK6qFHAVxH1KwkRgf5pyMQrcqD2IjEZCiegO+pXu435B5LZF4Oe4FbmIDBcBdSvJeyW+pBhW9WtQNRp7DSgOyKIYWiUPtF1dhNyO5j4BQDqlwLDYyKe0ScBaWk8RBDQB2B4HHP93SoRuqzoWt8mYgekKTrQ3PUNEdL4mvrOrSJ0JxIJR8wyA6oTRMqS5giPZxTUl0yGEyoqTziaVyLQOX1I47Kmz76NmZRCWjH1YwCalwJuNRHH6I76YLiM9/qlugP6R3TcHIlcpr4Bxi31aZQToQ0zU7oWOZ5PxETacqJ+YRkI1QN2TSQmUSLPO9VTrJ85YpJKQCLR0dNY6S59j5iT9xzT74Ci7/oE+Jbq52tAjfQ5mERkswXal0Ta4uskHZGcw9LSdaEi0hkml0GkfIKha+1GQJ8A3UXYtYC0RC4iHN3DchS+FhN5z5GeP9cISA+onvK1O5E/ByOghgjhAbemz8FQ/q4V+T0pU5t8Q2Q3QQFRCwRFJDSauMrgGpqXbi2gj5Haa6mvqkNE/+ibs8LqUyTCHFBbRAQQiITFblLddRo3Ju9Jc98caaxCRePPRDh5HshprBHnY8R5Za+UAurbCLtiNlcal2CoP3j+SEfkNRB03teCiHkS0Y0aY5ozc3IXJrJRlshTggZ0n96/iUxFBMAbyo/mbnp3nSaiG+kAb2iMdTeNJz1D9J5R4/TM8ByivhGpH2K+PmhBc8PFnE8m71EL8p449T//lmXymrT25bYETXkyMU62vSKNAZH3nDFs0nkRkd/7JYRUOhoTvpbJaPJ994hxIqLg32YiOQKQCWvKdmQSICUy0RNLplBm6Z/nuUXkPg/GRTIcmJmJC09oWS9R9B2fu5fKQ0tbb3l56cEuva/L/+euLw8vvK3n0v/xj/+zX3scYvPBx/G7/9GvPsbyB//Vb2Isf+m0NQP+wSd/hpfjBoPXuKw6DF7j6CucXIVPPnqNMRCE7nZscPGMBM4B4PF7R9zaBmPQWOkRN0OL71y8JsZXGdAomwXkf3a8xPevXmWheIaF7T7o4YLE9bBCfUG7sI2yuB1bXD35CiEK7G1zj4Snkh6dMxi8xt98+mWGrDGEjWMsX/brDOFqtcXoFXpnZmQyrba5vtcD7dwzzPD59/ZZMP5xc8zstGszZjjXdd+i0Y4gXMqRdAsAkzysrbEZbiZBYttG+RmMbG1GHG2VoYIMt2N9y+NYoVI+e4JGr7LHZ/vhgMNYZc/dylgcxwpaeeyqAdd9i4uPSXB69AoXH/eolcu73yzUzAQ/LAqdYcp9A5KGcdnL0hqK2VQi4jTSmH7/7/8cAPDTuwsYFdAa6m8BoLMaUiAzw66rEULELPhuvczwLvaeAuQldIFEnzn+tNEOx6Tz2Bf5EuwM2DUkGH47kKHrkzdnmzZPtvWAm66BkqRXySzCvMssRMRxoDm8qiyOyRujVcCzNe1m75OHsNbElsplsD6k8zJrM7IHjHcOa0MSNizHw162trJQMuDQ17hadbjtmux580HAJdhjU9l8/ZDizjgmtNQM1Sn2bnAqt2PXEsswx4AyEzDFttqsg8n9f9H26KxGP5q8s11pj0o73HXUvyEIaBUyydLoExHBRAAAIABJREFUdH5WK+0gBcXYcvwuxdUij2eMVFfrFJ79QfKk3W5ze2rjUvk0H5UKWNcjeZVObd5hVZIkGFiaBwA2zYCbQ4uL3z1h3zV59x4ArjYn9El0m+CUHrGrs1ZlW1P8d+yrHGPpkjh2jAJXf/uIV4dVYiX2uS8FgG0z4PVhlWNXQ29QNRYxCYiPVk+70JHi95QKiGnHX3hJ3rbBwFQu6aEix6+ayuU4V6Wm2NIYSZsyukT+5BRUca0QpP2ok06o7Q10Y+FHlj8BZPJCMInQ+u8ecOhruJF0P7Vx0CqgPzQQJslAJdKlat2j27cQlYOuLLpDC73p0J8a2hAMElXamMwwVKeS6HiKuUz1QxTwlo4r7eF6kze2vJfkaagoXrVekTg8r4W81TkeMUZAJn1S7ifqd0ClWFMe26oZ0HU1bX5ahaBDrpOoHcZTjWZ7QggyzzNmY+Y57KwCGktxxTXlL5TPslWkqWphmhH9sYHWpBMrRMQwGmhF7/5x1FlPta4thECOqWUY6ulUQ6d4S+8o1lQIYrDWxmeCKTvqDKdU6Z3iRo26tfBe5PFWOqAqYotN5TD0FepmxDiY7MlpG4vDoXnj2uJNyVQOY2/I6DYhx5VKGTH2hhiwQ4qhTXJW0dP4cZ05hjWmGOngBT1jcYrNBUAxyGGKy4WY4KwxCLTrAUNPm9rBSZjaEZt3ik9lUq2QNlx1xVq0xNIdA8k5CYA2qQvvlNAhx74iAsFJqCpkhmsI5BhYWaW2WKJEVY3LbZY6ZG1YWXuEnpmPkDasKbY6x54ygRnHZQJJ0ghkMXgxaa0i5ZGJocQ8vlQUx0WcYkP5vnyODbVkuJUxnUh5qEhs3oXRIqJAFEnORaT74nSc4yxzjCUKo1NgIlNCka8Q0+eAxA7I58X8vjIfJabjsfjDdK0IgozDc5YXeyPjtJFcEkItL+UNlYcDJef9Vxp9S6fi8ty96pWG/EPfgYcN0TL943eo72/SL5W+1R7LT/5gF/+T//Xv5/g7jonjZIOCh7zH6BqigIeEQshxd3NGUzrGsWT5eJrJDEP1UeRrOUapjGuj43GWfxknxfFrwLRQWebJi/eQ4pS09Lkcvm7JCMp5cxzQuXSuT87FOy3juMrry0TxQirHh5X9WLLmLusoRYBL8Ztl/wLI9efYLi53Wfey3nx9Wf+H6p5ZTQWx8Kqi7vGB8Vr2bRnTVZa3ZMgt6/OmxOWyEV7m8S73L8exjCH0Udz/rVzkuXwnL9/T5+qxZObl+KzluYfihspYo4facu74rN5FPueOsfFmPc+Nc303HWdmc477mto6xYGVzLmZfCdMOrEPJTYi+Jl/CL6zZEouUykBxO1b9tWb3uzM3JvLKsZsyY47y7MYw/IYw7imY9MCmFPJDAxM8WZvKuPcnOGYu3eBPXl/vy1l++JsDk11jEHMoGb5mtRP5XxfppKFl68rnBv3Y9OWqy0xj2t8KL7oXt8VmS/7HhEQEmScoOhPOcW/kdESs9FAC9PF2KaFc8mom+8v4XJlvxTtog7i70W9k4Fxb8EcMS28kY7zvcsYQb4fD+RRMos+ML/fKT20qC3b/dC8XL5Y+TMwbxvbRWygLO/hSwt23pmBI+IUl1i+wM8tukvjo7w+4F49hRdnwxSXzZ554Ti/Ms9F2aV25NklC8NOF0YJe3bv9eUbhnc2Mg89Q7n/z5x7U/7L8XngkXgwveV99m55vCkDnB/7t5XzpjzfUqd3qvM75vWrSH/23/76vXrNBx/H7/6HfwEey//619824FvusTz4Gv/nzffggoSLarbQzCQGhYEA4J6htFz0nlswlPIcnCImr95y4V3ef66snEexOD2XyrqVXsqHFnfn80Cqz7QgLu89t4h702LtTQbUQ2n5OxwWRsC5fngXQ+Hc+fn3+3W5h/tP+SwXvbyAfZvxc7a9/KO0XNzdu+5+u99UzrnF+tsrs1h88bFz24Zn6ras52ye4OHNyod+g8+sjc6sFBbnztR1SUSxPAa8Q18tFyNlxd9hEfHQs/TG8s4tSN9hMfTmfN8+J/OlZ/rmnAEYC6HysyvJdyqM78H9tr1Lm99lIZev/Zr1PZf3uUXX2Qk7Xe9L4+ZrjMMbH543Xbd8mM4df1tesbCF7l1zplJl24o5kz+fK+ehej30rJXnl4bamTrd82w81J9/UQvVNyzIz71yH7z/TfNryjXd8+7z62zZ5+b8m/rtgbo9WIu3PS/n8nqXPnrX+97QP280lN7VuHrL9ffOvaEvvpax9bYyf4n01nr8MvV/1zp/g7Z97f77q5j+OrThgfStNixjJAPuaGti69Qjeq/RO4PBK+yqAWNQqKTPsMw+EbisEimJjwKV8jhZg0ZP0he18qgTo+FdX+cAdvZIEHMqwQ8PQwWddv8r5XEcKqxrErjnYHhVMNgpGeC8wugUBb1HkWU6lCQT2CiPw1AXTHpzSCFABlpbETtkaywOA0mFGE2QvatVl4kB2kR04oNEW9lsIBz6Omto1tqjTwQb7HliiCEbqC5BBXUKukdMcD9LJBCs3cgQPaM8+tFAqUm6wnsJo32Wkhi9hJQElTOGKPOVooD8U1+jqUjiwnuC+FXa4+5EMCyCB05GhZQhe3dCFBm2x5C7EAQq45JECGAtSSc8viQY4+u7FZSKMNrDAfmaiW0PqGuWO0hskVmrEKgql+UzauNobBPkk2GH46hhDBEJ8DzmoP26clAyoB9Nrm/wEnVDMg6VcegHglayRMSk30h52JEgZ8Z4jEMaTxmxbgcM1hAkTUaoNIZcDstLeD9tHdMCMmZPjU7SGkrFDLt0lmBdJAuhUdUuw964fwIzsxqSoWCoG0TM0gQoNmFYhsI7RTIfVqOqLUH80rzj8glC6TOJAkBtqWoL5yS8U7kPpApZvgICpNkoY9aAZOkJRILdxZikE9LcojonAoWkmScVERy0qwFCAKdDPXlsVYRnLcUErzIVyW0MhTwIw88CSw1EkrYYO4N6ZTH2evIkARlSx9IaSnvYXmcPlK4YMkix2TFJMhA8TmC1GXA61AR10wSFDIHGo2ksToea2qYj/Cihao8wKsjKE6wu8CYUCEonMEHiUjneSkidvJIRiOwtNgFSkfwFw+xm2pReQqiAYKk8Kac+jF5A1QliPSiSM7Ey6WXGKb/kaWy3A8ZRU1y4iFS2iLCdIbkHIMs4mNbCHisIE2iedoaO9RrMHilT2ezx9FZO8EGBLDNR1kGaAN9rKo/bmuQxopfQDUHS2QsY7ATlBZBhhpwnj7Gq/axvVeXhUzujk9QXIiJ6qlM4aZjtSNBLjpFPWpqsQcr1577neR8C9THLaSgT4EcFoQJJgACZGCQGIFh6pqOVEKnP4igpTjLl608a0ImYJM1DIZNkh54kSOLIuhIgWQuA8mW5kULKIkt2jCQ3EnsF0XjEUREUOQiCYx44YPEbpNoDLPGh4hySOahJpqNKsfkRk3e19vTZSorNj0jXiEn2o5QVKTVdWcqj2EARjUccJLIH1wTiMiiNLJb7ACY+AB77iAlmmuL0Y24PZlBN4QWiTrH/S91JRe9MkdobTcjPDJfFPANinOY2wzZjkiuhGP/kbfWY2vEmuZGin0ThfedzIiLHGLLMR1l2lv5gCG0pScIpIscvntvwyEOSfsdERO5H1iw9u+lwxrCPAoVu59T/9zYjZpswi/FaGuViaivkfft3Vpfl/zfAXbmeb9vzyJ7lMt8ynSv73Pk3fefGvG2D4zfpLzx9q6GwH/z+Zfx3/+d/CwokVeEhMQad9SzHoLMcA8tCaJH08ZKEAqcSlsqwU4Y7VtJlyYgylTIEDIfkGEpmUWUIZymLASDTzbNEAMsTAJP8AkMtgQmSyWnpgT1XFmtyllBPLUPWzGTdTYYu+ihncFAAWd5kCQctUwk9XSbuD5YcKGn2l55PNmbZcLde5Vg2nWLgrJ/kHs55S7NXN30vjcySqp7fO3y+dxSv1hqSoSnrsYTU+sTuy5DHclxZYoA/M9Q2S9WkeMzlfbkvU3wjM0LKZCyyVmYIMsUxoYDyTt7oGMWD50dHTJklbJPrcc4TzBsh5biWfcfXMlw7LsZw+ePF93C7Zwx0i2u4fF7UZh3RAma69HCXBjqAHJNYwh6Zqp6PZXa7BTyTNxLKYw95zFnz07kUc6nndVx6531a5ClVHkdRVppLXmQNQd44yONQxCKV8gacyrqX7IX5/hRbN8VryXl7FCu50yKK25ivW3jAs5xD8fzlceY5UORfyjEw9JPvm61puO5F29joYmNIFICSpRc2pkU0G+vZgE1GXpmCk5A6ST4EMnIDG2hF2dnDG9mgnK8Y2aDM14VkIBXezZlXuIxh4gVimWcJkSw7pyAUy/Fjy9iyKCbDQkUyOjgfFGXGIv9S868cFGBaeBfGBC/oY2kEyeK+MD/Gi38io5ralo+rmGLBEsSzGHvBGxqy0BQs+46PcV0kZuRzQDKQlrIRXyeVcXElvLbsn6J+Mwc6k6KUC9/CYMrfua6l/VK2M30nAyvmctg4etCbW9QtCiqg7NN7HmPM7Nip/pxKg6aoY76HP4vpcyyHjdvJBt+5fuHXTXE8ty8W54H7hlvRFPFAu+YXLupQHJsZpVyVZR5l3c/1//L8GQPonJP+rR7oMu9fIpWvk3tlvaGO38i7+1D6Gvd8E8/lv/jvfv1w0faDj+N3//1fPRT2z/6bX3/bgG+5x1KKiEf6iD6YvIgp9RKNIj1FpQZ4SLRJ+81FhRrzWEkA0PDp3jn0NUSBWg/3jIBWJc9VisVkg8tHgY3087hJOY+L5MQac2VZs+/vgJe6B0tVdIyp35epSfUurz/Xbk6lxt3b6vBQO2rhZsdqNa93/iynfFpN41WrSaer1fO6vxVO/A51pvGd5k19xuCbtUn5d4cBL65piifu7P1nzkcATdK6W25O3INXF5+Xv2WtIc/IQ/fz93PG0DkoLIB7398Wj7uMuXwIkj2Pr3PpmDs7zufiIWMUaIybxUeeS2+DOZ/Xg1v2Q6L1N+5sX9wbZ3M/9vpsIn4QGO3v/dZWyeGS1y73+ux+m8q2lvGh9N090DYk49Xn/3zsofQ2mPpD4/im/GbJzD+/qS6iPn/+bPmVnxn4IQgos2DdqM6svO7lxX00n9MP1nGxWbGcE2fWnvMV9LnP5aVxMrRFMx27f+GZAhcvkTc9LyK1pZQVeBNbh3ioTwr7dGnoxKKdsbj+wVQu3tP/eLZDv0YSuB/fmcsrnpszdTtb7LvW5Vx/PTDm5+/HfYMHeGfj5E3jNW1k4N6cOXv92eOL+bQ07h64/21G0P0y5heIZX+cva+4/g2X3bv+ob54xzGfGfXvcs8Dxuq7lPuwcfiGd9e71uld0td8Jr+JUfmb9JeTvtWG5e3Y4n//4vfQWQPrJUxipGNmyFr77OlhWOboCKaokweECU4GS/BL3iHP0M0gMaZzpdciRpHJL6ydmByZCc8YWqSUmkol0QdDGKu0mCEPwuQ54nz4x5sXOnNCBiKYYLgSQyv5WmOIIdE7mSGMvMvOXhjnVPY0yAStBKZ3k5Rh5vUIgRjrhKTtO74v+PvxiByjyBBSkTxTMUE3J+a6lLefGOSETFBPhldFgRgIWihkgLeKtJNKggGBidEudREz35UaSAzXYuhaDAJmRSyabtCAKK4BsgeCX7DShJw3fZgWT0KHvLCSeoIgZiiKjBPUz81hvAAmONg4QTJjgs0hUP4hQXN5dzzfzws+9tIogsQBVC7DGGOCoJbaVFznzMjHsEtud7qUPS+lpyamXXIhEwQvMXtCFp6Z1N9UJ5HkYVIZ3I/AHFYWBZ1jSZkEZZtBuPhehnMF5HEiqKGYe2lkpD9bwOvYQ8R14fmU4Fn5fvZGlA8I3+8JlgYAsZ/GblZH/iVkb4kt2p0nrJhfN0qC1Fkx/4GvwtTWkOpqi40h9rJxX5djCkxwvogJZsaLEp3K5Tp7QcdKqBgnbj8w1eMcAyOf57rJol85H84reYSYmAQy5mtFSMdQnE9shxCYWBn5e5JBIg9NKldGiFFmDw2zHkYd6LiKgErX1HQsG1Y63mvTjDBlUT+uUwndo7EAbegFkPesWIgKL8g2TeVkaCD3obh/nwiCJH1ccV0xLkFHyFHCJ4kjUc5zpDoyo6aOgE9txTQmIgoIS+VERXJEkJj1Y163J28qwSeL8eL5BZCkE/cVezwFSGqIvZaSpI1yyrBMZHkcqhuSjMt8bkgr5lITqd9k/47G2JkUNSAc6N3H7500d4XluiNLQYjinZQhn2GqD3uWy3PUiIVBsYjnFREkj2RTX/PcC3PYJY8DRJpLCe6aPcRFn85skgW0kvt46eFmiYrSs8htL59pvj8zpJavEW5aAbvMfVP0BR+feQ/53R6Lei6GVxS/0dPBqR+jKMoun51cQdyXBinWYjlrMT++9LqWe0A5j3NT8YE88veIeVnlu7PMb3HP0rM7q8tD6W3n3zF9LUOwbN+59v51SH+d2rJI32rDslYOv7V7iaOrMmyVWUNdVKikywytLkpoEdD7FCOYiHd8AbMrd5ZZsiJEgc6ZDOcEJm+DEkkSwesMi9Qi5O9SRAxOZwkF9gapwqjlcwxLZOgk31vCXJfexwgiEGJY5phgogxNrBTBXG2QqJM0B3tR2Js6ejUZyjJkqCm3vewHrmfJeplhj8mgPAcFZZkOhkmGCChJx42amDqpDrGIK6XPWgX4MME8AcAm6YSSzbL0krD3ynGfpDoCmN3nUoxaW5EntG80xOJ6v/iVKj3Rs3d72qTg61n8uSSSWsJ+OXHdGfLL9WL4JsMGS7bYabMCk2Eq4gwCyjIFUpJcho8iSw2U870kLeJyidTo/PeyrxkmKcTENsp1YPhuGQPLmyi84RAKQ4U3PHIMW7q27APeYOHv5fX8HcBMrJraVubLq3YykktG1BJaGs8Z3bFg0wQAQTGoOnm3bKUm+CfPtWKcOfbUJ/mNMpXQUakCfEP9ybIFeQ6mja5ypVLGhzJkmPuP+ybfrz18RXkIOcG1BZdbq5xvjnULi3YDabMkfQy4d13+L5A3aoSc8l0umsrNIt6AYrkE9obJZDzw+ZLVNMNxU36qiC8VAmmsgVBPrKZsCEsVEes0b9NnqQNiJSeI7UJcnKG1QFr3RGTobFkmf0a6ppxDQpWr1XmevAGU+7wYwxk8OEzxqfeM+ghARtLXTbFvGYYbyzkspgVqumcqgDyEsRJ5cyZWfO80/jPZB85TTuOVN6sARD0Z2zEbKmleLObBPYxepHxnzwCwKCvCG74u1Sfd59U3NyzvwXvLlPQiZ/0IpEV9MvxKw4znVRT3pSqWxc5+h2Iy4CJE1kiM0yRcGix5Aj9wHbfpjOF1z212xrM4M5i4Xgyx5SxKxt97jZuaVeZ7dvFdGoMPGWvLvEPR5nlmUze9i/GyOPcuxlJufzkmX8eYeweDatkf39T7+OZ733DsHev4jdNfN8My/pL98Vc8fasNy53u8K/vfoSt6hCixAu3g40Kz80t1nLAD4fnaITDKVTYqB5f2R2emTvYqPCL8RJb1WOjery0W7xf3eCl3WKlBtigce1WuHMNaunw2+1X+Hy8woXqYKTDwROe6NquEKLEd9qXuHUrhChw8DXeq+7wxbiDFBFPzAG3riVILshI63yFtR5woTr88PQMEhFb02MMGoPXCBDY2xofXNxmaO+NXWGtB6zkCFsw4N7Y1XR9ewspIm5ti63u8ef7p3hUn3BpOnx+usTj+ohWWbweVzk+9Dur1zj4GkYE3LkaO006bUPQqKXDjW3Re53jSLd6QOcNxkB1aJTDnW2wMz16r3PsoxQRazVi72o8qY+4s03OZ2t6vBrWuKo6XI8ttobyvDAdXg1rPGsOOLgKX3VbvL+6w5fdFhszYK1HfNlt0TmDjx9d425s0SR9T9Z0HIOmDYMoIRHxuDlCioiTMxRfKCKu+xXWZoCLCo/rIyrp8P++/BAhCvytZ7/AyRncji3WZoQLEhsz5DkXosDLbgMhIrYVHdfCZ8mU27HB2ox03WmNSjlsqgE2KNTK4TDW2FQDTrbCphpmUirWK1z3LQar8Xh9AkBQ6Vo7vO5WMMrjtmvwbHuAS3qgjXbQImAMKm9WPGpP6JzBfqhx2XQAgJOt8OJug7Ye8fzqkHVBATKiG+2yEWyUJy++07BBwsiQY107a1Brh9GrbNxv6xH7oYLzCo/WJ1yfWjzdHXLcqlYetfIYvMJpqHC17nAaDZEbBYltPWTyKibXYl3NdWVx2zW4aHvcdA3W9QjWKz1ZgyrBkpm4Squkqwfg1X6NtraZSMt6hcFqjFbjcttR/LTymeAqAlhVNm+InJJ24qYhncFuNHkDYrAalSZt1lPSJX19u0YMAs8e3+UNjcEpKj+KjKY4nGrEILHddNnYc6nMEmVxODZ4dHHE69s1tuseRvu86XK7XxESQgZoHTD0BrvtKW/0HLsaMQKXF8esAVmbpFUpIvav19hcnbJGpLVEkhSCwHissL7sIETEMGhsNgP2hxarTU86hiub43SliOgHgxgE6pVF11UwlYOzGk07wo4awtBmCcee9r1BcBJ1S2RM3AYhCKWha4orNcZjGAzCqFCtRgRPxnt/qgAQgdHQGZjazRAYSgRCUQQB96oF1g66Jii57zTgBcxugE8ID1V7OndXwVz2cKNGOBiYiwH2tobcWDJwAbijmTz6AFB7MiRDQkbogNgp8uRUyWPeKYiVQ2SPskrogY76BgczeVMDgGQAsjEZT3oy8tgjrCJwpGclykge1rsKsU0sJVWYvM46QBw0sHWQLysSrq/DRJTiJ2KVaALEUSHWAfIkyZFv0zkJhJWHPGqoXsCtA4QVkFZABMDX5F2MAggN5R/qAHVSQBQILR2TPW1guG2AGOneoAE5CggP+FWAHCQRvTggNGnFKwExUJ18G6BvNaKKiJr6QI4CKnkifRNTHSP0SVDdrECUgD4KjFfnrJt3S6qjfAFADYK8cSBPnFtFyIG8j6oXiGmFFRQAEaFPkvrRAHIEGdCKhkyfyLgMejJEpJtiTIPB3PMlAXMn4Fo21KluwcSZN1R6AV9FiAioPvXFmDzPMvV7pPGLybuLCEgrydNtiXQpqAjVS/JGsucxADG1JQogVGnKdanNAKSn8Q0GUD3gm8nQ5JhNaek/e5fJ009/IgLCpf+B8i3rKTgfTH3JnkVevAf2Vqdz2dAtjGr2RDNJT2moRkl1nCnaiWI82GNb1J3rxO3nvMrrl8ZSNszlVEb23C7Ky4REkdonXdH2xT4UH+N+eNAIL+skABEiohQQy5gS3muQAoI3KBZpaTjdM3xLo3RZlzNGZJnfgwbwX2Nj7duUvtXkPZ/8wS7+p//b38XeN1nDkmPqbKCFKht0+XwkDUsjfD5G5D8GWnq4oDJJDX8viX5YU84GhVo62ChR6kpKEWYEQg/FLYYoZoQ8S3IePsbeLy7zXDIpTnNI3lhOOTay8JRN+pExE+pwfZYkO5THPKav1JBcxgKV8ZLsYWV9S64P90nZX1mPE3OtSO5LneJXl9It52IEy3PAnFjoXGwhe//Yc8wL7zIxqdGb4tfKxHOQ7ynjPEsNTK5n2afLtnL9S7KiZTvOxURyPcvPpZf+XBzkclzfJQ5wOQb8fCzHhedx8Zs11XdRh/J3phxnJWI+do5IZ+kFZvKlkoSovG7ZztLjfq4/yrLO9TN78rkOTL601MGceYrjdHz52y0Fsrc/y1qk9FAMbOmVj2f6KeedPKklCVV5ryuQC+WY5jlX1qUYL/HA/2Xf8fFzsaH3YveKYxxKULZt6bUvf9JKrzblRf/JMz3vFyljJjWia2TuJ86X8yvvPSd5M40NJhKfRXszeVX2KCN78JekTFP7p+Nv1A0942mKQSa2ZTzouThXv6XEUDl+557jXOa5ehVjQIzM6Vycf36obmVhObyB2yPi/QXpcgUaxX0499dNYvIgi0U297yrizpniP/sJs4XD7Z9dl3+fqZtyzLfVkY5iGcX6m/I8031LMvJD/uZ8oGZh1Ys++tcvcrxPlOvswbHMs8zx97ZUFm+n77OEvqBPnhjuW+4/mvP4q/bN/gGBtxb2vi1+gv4enPvHcr4l//lr5/gpn3/4/i9f/irJ+/5078CGp3At9xjefIV/uj2e5AioPcGY/L2aUEel5Or8uK8kh6dMzCKdqcHr7NBZoNCq21mUfWRYjTZi7KpBnTO3GM0tV5lOGvpeai1w5A8LybBUWeLU5ChqGRAZ01e7AIT4yh7etigZY+GSVDLiWVUpfsEKj3BaqWIGJLMhVEBo1PQKuRFNXtFKu3zQpgXseWCwvm5YaxkRIiYQVDZ61YuyoFpUcr9w+dLKCof88kYLaHJPnnPrCfPmOD+CSSpwuXMF3nzOFSOgy0XrgwFjREZWtsl6YeqSp47J7NEyVJg3aW4RZUkGsrY3El6RGQ5DaXmTKMMFeX8y8WxdxRXq/REGiJlzHmVi8MYJETB7FnGvAIEg9TaZzinT1IFzO7Jc1FwH4qYZUG4rPJ8Nly4vxPEsZQjkGoeO1sytUb8/+y9S68lS5Ym9NnT3fc+J05E3FdmZWWLLlUJ0dUSiAkTBggxoGAAEo9JM2gx6AlCjBAtJkgIoZ4gVCOE1BKIhxDiDzDhNUIMQEigpotWo6Iqu25m3nsj4jz2dnd7MlhrmZvvsyPuoyo78yZlUsTZ293cHsvNfa9l31rfwt6tslecuZ/miioxmexGKO3KMUhbnRtmraq5OgKgWFRTntWRWFVaD91xoLGHyvwkbhYVW2zpxXqqmeJJa6RYPD1s7wNhF23fKyj+tILQKsh9wxY3K/0nTpnAqRN2MXGMIjRlNStK0SCKPLv6NjbSyp9FKZPYTVX3Ma4AxbN6GRu3n/QWLyoxmVLEhU535wVV6xV4eY669BC7WF6WV8+s2VLm4AMvAAAgAElEQVQxyLUK+7jafjy9IsrfVdRbrKCMVa69VBCTojQJRbXYQEHr2g6+pI9ossHehVHiKYFdjGPvGkjxYlsye0h9UZR13eorCBfQXuadHFRF67e6TnZyr/UW61hX1dAuQUZURUNvGtKiAeZz2tau6hCt1he2FA+ScoLH02SQFQ2H3UCbDHmsWsYvsYEtDhDtu0y5Ma2KnJXIEy32js7TOBsKJfNU9L38KbKNtJhCln2/Hnb9ZGxxdyLCdCFneS108r6aLkKO98cUKIbUbLJWgnB2z4PicRG6x3Gr72GF7W51q/teJl6Zex9DKR70aZtbQ+BEJlKnB4171+I+TrDrp8la5NfFYPZ7KH25dlxdylb6UNihmpd1BCFsx68Yzxd7Ofu67+nzGjJ3dbxS+uvUe46/r1yef0/dfm/k6+peHePXlW9T/5vMq42joipGUf+8/NLK99qwzFXjnBxSNc2tDwDm6gg1YxdBSb0BAPNKvyhiAIlRJS54UlRX5+0yteO9QinnQ/A7BCisQ4vRW5ks6LKEqlAq4AwZUal7k0hsWkiS53AzPsLFjj6Ngwy+Ndq2ox+LaTv7K+dVjMnskBsAmMP2C0txnXtDsn8fKgAxd6hDfY6siQEsKJFSFUtwLaYvFQ0IosNGRGLjQurFLvYscA7BUrZ8jVrXloeS3h/q2b2RkthQu0RFJAZwDXTfvSc3QCFAknhBpYCU1E7mffwfGU9bDOUuxrHLnUnzpWvFMJW/vZaimeipdIRBDa2qimL2uvhFMUr69ZCzbpsgksMRoLg6iq/aYjTlF6xUBVU5ZlTQoO7e7wwp6UtctPgXSBmWgellTfNqBE6dUSlFDMb2O1zRGYNkPEodOdYUjI7oQnC0ZmQ2g1Da5bVrN8RBYrsUox21dEgNbz60WEDVKdmiQBTdlHgxFFuMnKpkJPSbM6puBmVndO3i/eSYEHF0bpHtnOSJkyL5IkV+bEQ2BIXn1grnFmwGINDGrGzu4ke7Os142Obf6oghY7u/BRdxevyXyXFae/25ZpDw2rTcWU8IZDeDfHf9hVIJBdSRtfvWvsyp60v+eB6TrmSMZoU6dMYvQHn8OlltmjggT8azVBb8CNYLWTSD1/T1tzGJQQUDXFWqJJZTrpTxo3bKGNexZCDkQ7dhsO+xoUXFEVlQ8Vsl1a376rp26eTFw9sPCs/Iifo57q5F3dabv1jfXLUI8U+VcXb9dI/dTl7yEyfjGnFdnt+m7OazfZclKp+flWsGbf3AuV29ywUO5OFCTr5e73e7yfw4Xch9V0c+1915OlyfPWZX+/qQjN9z/KoBJX+vyPl629dHd9VQkvbas/LhcX0bQ+v5uL6+yrc2zr7tGL5r/1/Txzce95+xvK4X1ju+A8r59738Gd67X7XyvTYsP3GP+Nd+9N/jD8MnyNB4aU4oVeNdPuBNusFv+q/wWCY4lXGfJ3xiH/GzeAcA+Mzd4zFPWKrFQQd8Hl7iM/eAN+kIpzLu7Bm3esZjmfCHy8f41D8gFotz8Rh1RK4ar+0JsRr8LL7ArVkAgNu6w4+GtyhV48t4g4MJu/yPTmU85hFPecBfGN4AAB7ziFFHaFXgVIZTGV/GW6zsnH8wAUtxHA+5pbs4mIBBJRzMip8HijG9MWtr+z5PuE8Tfujv8WW8wbl4vLJnHMyKcx7wRbjFoCNiNXjlzvgq3KBANeKjl+7cUFyJIR10wsEEnLNHqQo3dsVTGjCZ2MarVcUpDTjaFV+uNzjalV2HDebscGNWPKQJr/0J7+KEycQWGyrfP/FP+Hy5w2t/wikPWLPFa3/CwQT8ZH6Fo12xFgunSosJHTTFwInr6JfhhtK/2IDIcaEv3RlPeYBRFe/ChFQ1fvf2cwDA3zl9ikEn3LoFc3YwquIhjs2NFwBe+zMMCt7FAwBgzo5S2UDhaAJO2UOj4rPhAefi8RjHlkdV5vfCrngXJzidmwuzVgWfDY/QquJNOAIA1mKwZIcfjA84pQEv3Rk/W1/A6YzJRMzZYc0WVmc4zln61XrAaBJe+zO+DEeUqnC0AT8a3+Ehjfjp8gJHG2B5TqnqtvEymIQ1W6RKKL+MT9x3JcZ20Km5ij+mAbd25fjeEZ8OT/j5eoOjJabdNVuEQrlIJe721q44ZQ+rCp7i0FyiD1ZiKIlo6CkOuPML3q4HfDSc8JiGRs51sAFLIm3sYAMGkzBnhyU5FCj8cLrHYxz5/lWMJsKz3L5aKT46Vd2OA2Avh20sAPCUBmhUjDbinCiIaDQRS3YtBvecPH50eAejKv6fx49a3OxoIp7i0J4JrzNeDRQ/+9V6bO8FrykGcMmuXfvRcMJPTi/xm8d3+GK5Qcymua1+PD4hFYNUNUK2uHErvlyOzd35hV9oLSxH3LoFpepGXAYAPz6+xR+fXiEWg6MNGG3Ekhy0KvhoOOPvne+Qi8ZkIx7jgDs/4zGOuHULTnHY5dGdbKTY8eRw52eck8fBBjyGEZONzb1eXPmPNsCbhPt1ah4kuVA6o8GkNs81W9y6tdVVisIBbjnm+T6MuPMLkbcx0u40ocWyofij4zs8hAmnRM/kLcvli/mmpTGS5++j8YSfnl7g4AJe+AU/P9/i4+kJXy3HtlH2cpw5bICuPcWhrW2jCpbk8GJYkIpu7R5twLt1wmASClTzhjm6gFAM7vyyc5efk8McXfP0OLjQNkBlXrlo3HCMt3jg3PkZ92GCVQVzci2FUswGo424X0f88PiAVA154PS/STojFoqhfjEsOEePgwu8ZlOLXX8MAw4u4s7P+Go5YjAJVhd4nXCKA5zJSEXjHD2MLliSxY2n5+gcHZwuGHm9vF0mDCbTfc0Gk4utnmdPlcEmnIJv8/QcsjBHixcjxWaLe7+3CQdHKaqWZOFNxtM64MW44BxdI7C7GxZ8cbrBdy1HH/CwDKhVYfSxI/OjWO/BJYRkMLmExB5I4vlzMwTkqhCShbepeebkonbnZHNUPJ7oPu43fUvRuJsWnIJvG8ejjwjJNDdugFzqU6ZxTD5ijbYR4knOZIA2cSW+vA+dsLxBnVjGpejmHi/3znOMfkgWtQKjj82d3vK9TMlgcAlLcG1+vedQxcaiX9jDRjbyxPNINk5zothsaaN3cW+baRey+lBeXaiOpOwiNy6A5lnTE82hbpuVQkzV8uXKJiYfK0nv+mxEVr13gRQxcsWT5YJE65qrtdRrXjXdceqXPzMx1tXSG9j93/eRLfEYv1G5NOAvL/sa4/47uWT/ipfvtJHwPSnf6xjLz/7S6/ov/Rf/NNZiWyzenB2czjgl32IGJa7NqrIphjYSAQrHQ/bssVrVFmNZqsIpDhg4j6DED4pykappSk1LXdIxwfbnLhlBRZGSdoEt3ktcd/t4PTHu+noSlylKl5wrlZU0NqZEcZBYSoBiAddkW31ncnPh7Ul4etdY+ZG7FutoLtxhhTVVfoSkVEjMKBHDxEIxrCGb5vpqdIFjplur6UcnMkmPZYVFfvgu48161lr5ke5ddHvG18g/XgdWEE7B7RhsxR2478PbzbW39D8iwC4ezgmDbda7eLmUdfthB/ZMu+Ku3KPV4iYs5wXRpbGZXYyg1Jd7mJiRVKvayHKWaHdsxOKKfLEpfDX05trvgpw3akvrE7txVaDFqZluXbWYUUaiezfiPo5O1pX8lbQ//Q99z3Ir54mMRnfMuFtqng1t3o7LWKT0yPQlQ24fe9diLPl+py6NjMTtNZSf+yck/Pkv/G4tmYKUiFRH0hPJOMXFuWe8zXlzLzZGxr7JtH/VO5cpnRG7UxsjqHJtbaFyCqJkYGwmpJyVl0uviZ2bNis3VJ9ZSPv6EtPZPbs9Ii7KXwV2btlSr18bWmKPLxB1QcVl3OLCLW7gpXdrLpvympOG0rxGONWRpBWqldIhNWRX1ssFoiKxgoJ+C+LeK50QRbB2LLeCjEibIq+G8u8ZeVXn8nip4DaEV6H1VZKmVEkVzxDw1k5RDSmX6/t+K6+BPl1SQ347N2ZJQYTcocq98g08ZwXuXaV3iq3aoMC+LxmnKK3MNtzchHuX6t7TwNTN/fi7FHbvpc/9jZKxc3+9O3Kr041FXYzrWvxlXy5VNZlzvfL9yku7uWoXimlUvA4FSVdVPXdDLRdtXvtR6Ppo8+2Npd4IUdL/XrG+iiZ+gF+pEdRc+7G6IsP3Ipb9XD5g3Fy1hz6kOveVrxhnl7bh18rife1/E/W9q3vNNvv6vq6vyW80xg+N5+vqfZc+v0H5g3/3lx+HOP3wx/W3/uqffYzl3/obv/y5Ad9zxNKogslEfOofsRTXWFxf2BnDlPCz8AJWZczZYzIBpzTg0/ERqRi8i8T6aXXGnB1euhnv4oQbG7Bmi1P2eIoDvE74nRdf4E04YDJkfKxswJ2Th0XGJ9MT5kzIyZItPjqecB+IOfb1sGLJbhdjmYrB6BdMJuJn8y0A4MatCMU2l17ZjbcdknKwy47EhfpziNlgSQ6vx1NDJo424WfnGxxcxNEFvJkPeDEszS1YDMqPpycs2bXjr6czj5F2Kk/R71KxHFxArrrFdg424RwdDi42g1qMgInRrpd+bX0aXTCYhHP0uBsWZkkNiNngxbDgYR3x8eFE9yB43A4rnoLHYDIOLuIpeJyXAa8OM9ZkMdrUxkqoCI1NjLijD+RWnA08v5Xm6GjntSq8PswwquAn714CAD65fULIBnPHWnozhJ0xdFp5c8JHMmA6A3cJDo53gJ+WAVoXjC41Y3KNtu0aD47G4K2gNgqnxSMlg+O0NsV5MAWn1cPqgsfziMO4ohSNwPIWI0fic49DQEgGIVkMjALEbPDl21s4n3AYV4Rkm3FlTGnIUUtHw3NqqURA7/8QLSwzl0oqC+8SQrRYi8IwRDydRmIEZTdkrUuLR11WB+8T1mBbOpCBd/bFkAKAyLHH3icsq8MwEOOoHxIZDCYjcn5ZpSpCoJ1ya0szMs+nEcbmFjebs0bOCiV7+CFCqdr6FEPKui0+cl04Tpqvj8HCeUaBgoVht8gUDazLON2PQFWYXizN4IvBwg+xGUKlKKzzABQFN8VmTMiOuu7idcM8ENPq4wg7xmb8AcD5fqKcp4pSY5RgYMfYduvXmd5HbkjEzKoAbXIz0M5fDrC3EdoUpGgQZ73lXV0MzC2tm7Q4WJ+wnjzskJFmC+3zTtFKkfJhalcQzo5y0UYDM2Sk2ZEhomtDAdLZU9zdmFA5RYoYSTUZMloy5UONwQKJc4RyvGpkVlg9JsTZEeusGM+Sh5eNsfLGo/rSXCtz0BQPdkwoC/uocmxgPXvgJqEsGiVo1ENCvbfkTstjzydLChobNNVVNnAAcc1VzMZaLcdQR3Kprb0xYjgmV1eUVVzhcdEma5grxdNWhRbPCQ2oRbc4ORhALQp1rJAclyrxGGVMU4b60pOLra27GDrFzJXKUo7H4iv0wpuIzJRZdUUZifVUrwqYqA+Vifkye/pbAcqvWIQJlDeQ+JgONK4s18sY48ZOKgyuqtB1bc5xy99oZrXlrlQgxtnAz5MDdCQGUrPwd47vM2eFePvdNVSzEussyUa1+EIUoAxoMZ07FlEKq4ZdmPnVkRyqrhynywyzYHdfNgIk1pTWkxiq/N0A7kkhD/K9wqyUt7NqdomtwrpbeeyK+pZ7qugzWKY7RtWkUIVtVFF9HWid9jGOUkfkDmz1AOyYXHXY6vT5KDXHZAprKYAWl0nPGxqbaemYXxWf62M0r8VIthymfWzmhaHc5sr5ZVVvoCs0xtjLsjMOe0NX5Fi668S4+7assH1/3Tyknap5fXdzv7afsWPFvVZkTL3h3oxSil+k9ir3sf++a+eZoK7MVap/yJi+ZkS/bwPgQxsIv2rl+zDG71i+14blQxzxP33+202xF5THaCbGCW5HDCMuIBuxDe1gi4vGyopqU0KZMOXvjh8jcJxij5II4uB9bihFThrOkzIHANZlVuC7gTMSoDUpxADaTnpDXLKC9cR5LfF+WhQwYKeQSp62P/F37ZhSFWk10LY0xfOnTPufk247uX/i7porSEmkXCqg7fAXJhqRF7A2lWLYmLBEddfJMVGIBN3Qlnf95bipRExiC0rcFFo5/rkouEHjiyGjBEO76LqiBhr7u/FIbeiOsKSi28HmF56jt6jstANA7ZKVi3JenujX7vw40Dy6xPJKYvLkbdVo/PmHXkhEFLrrVEvq/iiKp6lA0niwhchQBP0QmVW6XhWFt461BQU6H3VTnhY/svamtvMtDhN4kvi2rHCSOK6ioBeN1XksbmRSlG0X+1kwrbyhZceai0oKixCL8I/PYjclNrgRatV49MOOaKMRPCSFwErvyonfZ4m3U9jIIfjHcWUCleBGqKgQu6TtPVGF9J+ETAWAXjWiqQhdgnZUkaEnBUl1c6kKqR8LoxqLxIhlhSzoS1ZIIrdM15mZ47XPpv1oqKwwW9H8+D5E6iv6LahKVeJoSZ1BoVeNdXTQi0byjvrjdu2iWwyeKBXJuQ0sYWUxekdyApBEaVOAmxXy2SBrqms5obyqRAiSz+z9UIBkPexKaQwkeX0DWtSmVFYDWE4Qb5NCcRY2bSQhoixZVsarMxuhjaAm/F6SZO9WDAljWSGrcEkUcAOXVEvloFiJBhtgqig2KnRTFjWnKMgnAy0pClg50wEoZ8/GD5CfPCvCm+HHnvabgmj2imhPwlINp+vI1EZLRaBkrbNBEVSTJV1XN+IaRfdD3j1NIdV8vG7z1ZHuESrfS17TVdN9y2cFe1aAUlsC+05xFyNNR8XpP3jaosAq1Qw2lclgVHlTVE23vqrhe2xZZhUoTrXUEKoCWa6voLQFfD+sVVsqhczXicLI9YtVLUVFVarJU+5pMWzsLoCJ2KWbMCsbhN+x6ATkWTXZNEOMDeneWGpGIf81geZaDa8THrsYolR3G9uO4MeonTJaNRnNOmwylD6rlvc72PBQbf0Wuxkh/fNbrNop7Lq7B82YFFIeqcdrsU/3AWBH3iOGpRBCVbtdu3tfqW59A+1ZacZj19+lgbgzdLo5tON6e1Za/U6vERmplu+yM07k9+B52O+z0tZ/bxDymNtndPK7ZiD14+oM1NZHZzxd/dujzhdt7ozufjyX87jSNrVRL+ZRP9jGbgjf1OC7Jpf3GpLbff9zwp5fnfK9NiydzviNm3sAQCh2h7gAQBy2FCOX7qh9WgIpZdyeRqHUV9hcCqXUrg6AnVtfc0ma+IdH9akFtvYbC+yor9STOt3Y+JjRe/IeccWUNmQMAFCP2/HLFBPi+ti7hV5LsXD5qF59D8qc3+cyoTYXQnEpLB0K+r5rm8F46I4dVEPZegr/y/H0fT9v9/n55DK7KYoM+/FjZ7zWw17e19oF8CzW473pAHbXkHPzLh2I6lwB+dz7ZN2f73+fAKDcqC3dhYyh6+N6g2o7z5seWu3l03/XCqhHQOvt2G6kLMdnf/sJdIVYW9mds6i9XLq265Vnqx5ZmerlJc+LGDKX/XYLvF7eo/77hVygKvKBn+XOPa5czK9WevZQsbG0Pp82ACDfkPtkPmzjlZKO/b3DpjjItb1yduVzOXKbCsgXikveEfrQdelwqWV0pZNF6eQMdd2brXQK4k7u3XVyvFzeF4WNzIo3Dd+rwVSFdIPnMUXSzrOBdWOS8V1zDZSvldMkXLwUnxFHXD6IAHpW2HTNra9XPHsh7h7oi2N9vNalPPlcPmyGfi8n+bpzVSwXbWBTTBsK06Efl4+EjO0SQdkxt/a37n1K97X3g8xJOuQOVHneRrxYUw0x+q7lUr79325NX9O5Vfecqn4u0sb75osrxxX2TKXYGzatP1zI/nKtdWjfrv1LBb9fE61e917oOpQ1JIfe9/nZGr/2s1Y7WV2TT33fhfKMXh7j+ld/8p63c7XvS9lc1P3GCNw1herrrnlPuWoMXjZzefwD8/jaOeBrxvRN5nXR57cqz+779TXwK1nauv31LN9rwzJki//3/lWLW4td7jXDCGVvFEiOMolvktijUlSLOdoQS9Xc04wt5CoH7OKVCsevESpJ5yX9RGaXLM0xRn3cECq5B0JtrJ2C7gFoMTZakBa1sWpePn3iXlXZdUypLZ6nRImHYYTQFHqfZtVikIQ5s7UhdSSmh9HWphxLnBCzlori38Z/aUgVZufsgurbcVN4XHX7zmOoRQGSbkEQRlUJkiog1zZB7PqXY4fcoYIo7YG9gijxN8DWrrjFSbu9kikolhSJjZPjDaXt2q7yuTt/+VdkJWNnhFAVhdynSFC1taVSd07mdKGYZKH9ZxRKzqlAqReEzn+3joTZVBBMuUaUZxmKKAwVTUaF0SFUoJgKFTWKK1sfGltMTwaxXMrfCnbbUpuy3b90uW6x1G6bjygjMv8uJUNTXBg9a4oXIziqYGPkVNtxGut2r5Ww1QpyzSha+9yhbFXXhjz1bJp6dw8otYIk9q62+ymUKp2yLWkRTOAE9f2SidsaFiOgIQEKG2om6SG4belHB2b9VDKerXGdOMm6tGvAbnG8bnTdKWuCjLW0FoI46s1IgRhgItcqcr2ikPbXZ0EsN2VH3C0lkbm4iap+3cjtDGhJ0+m+UT1BlkR+qGiJzFXdZK8jyVCaFUSsv189wtEnRt8MPh5rp3T2RoEgRv253TrIm5h6I7elNVGVUR9Fz0nt5ttkxojz2q/p7SY2xFJvqLS03xsr1W7ur9XKved/nSHeENdLNKvu3SZ3spJ7I2tWb7LrxylyFxdNQf3QuUs2F8Eucbw8GzoB+U+bbkSQuW53oq1TlsGl0Qfw2hX5dMbU+1C7XbL7fiOGr+vdbQXRbc/DBWLWUFTTjU1h935o607+qb3ML9c2rRvV1VGbXLr131BGSTci87tc472h0n+v3bgu5Ve3dpocLpT2q+jbZel/Uyp2CFhVCjp3vyW9zoH+mitjkLq4qI8rxy7H1b9bulfl1T3PS/fVnQDQ1tl7539tXjKM99X/mus+VL7unn2wj2/Z15+Xv7/le21Y3rgV/+SP/g4+X+5QqsKtWxpL5VMc8PFI8YNeZzzFAS/8jIcwoUDhhVsQimkMhm/XA+78zAyVFQcbcOsoNvBn8y1u3YpQDGIxLf+lsEZ+tRzhNbHbbW0t0KrgPkyYbGxjlnjAOVFs5KcHivk8JU8Me6pAq4LRJHy1HBujouTZFGIbQWCEYe9gQ2MvPLiAc/T4wfEBD2HEOXp8PD3h7XpAzAY3foXXGaEY3K8jPMdFTjbiKXom+yFSnRsfGnEQsDH7DSYhcazlwRE7pzCKCkFLyAajTXhYRgw2NVKXVDQmF3EKHkcfMEdHDILR4ugjTsHBmYK7YcH9OuLoA87RIReNow+YbMSb+QBvMkI2O+RzYFIaIcN5WEaUil2+zoOLWDje87wSo+SPX76D1QU/ub+DNYViRpmAZmYWu5QNrMl4MVL84zm6RrTjbUbMGkdPbWtVcTsQa+EpeFiTkZjl8BwcJh9bu4LeGl0xuYjJxpbihlLWGNwyA+LRB9zPI4yu8DYhJIvEGybiAi7xmAcf8biQG6a3GZ/dPOLdMuFxGeBtbnGmkteU5EbkQ7JO5bsg3sJ4aE1p8l+ixcHHFmN6HAIelwGD45ygWbf2Bpda/OoabdsAkg0bIRVyTE4zB4fDEHBaPG6nFXOgeOVSFKwpLeert6ndI4kJvhlXrNE2FkJrCqzJ8DbjiVkdSyGSHyFHCsm2mNXJR5QKvp4YGsXdnu63aTlVYzJ4fUPxyV8+HBvy7WzGyjGO4kp/GAKMLnicx/ZeEEIayrFKn2+mFQ+nES9vZjzOQ0t/AwB3xxmBY1hjNhhcxHkZ2ibUYQy0Rhe/y80q43p9c8abpwNS0hiGtIsTvh1XvH06NAKkdXEYp4CVY11jtC0coFYiAhK3/nGIiMm0eTuXG3mSbLZ5T++CefWNAVLqCFGRMQUxGnifua5rpELDQPd6WRyGIVF8LW/UNNIidpN//eKE0+oR+B54l2BNxuPTBOuYYI3DGI7TisenCcZlHMYVD48HHG9mPJ0HSIokP9E7X9L5rNE2uUoqIee3dmW+y+xhOH5XNi0db0gOYyTdmd/pMRqUtFmifowtHroRNBUFP9LvCslPY/QR88opm4LleGGgZg07RpxPA+5enClsJNgW97yLNY4GbowIwWDk9uW9XYrGslCM9OgjzouHsQVGl/YcS5shUK7gFC0GXospWGhTYDlOeT0PMJZ+N0vWMLZAm4K4OJJN0hwDTGpKLdjimoOBnyJKpjjwkhWMLRTPDCCsDs4nrLPDMEWEQLHhORkcDivOj9uz922L9RlxsUBV0D5vRHqmIMwO2pUWypIzh6rwhpmMOSdN60HIxbKCGxMkl7FsIvc5h4VESpTqWhTGY8DKOZhLUrBDolzIpUsLpoESyWK0PiMF08ZYMxM6qYq8mm2DVNUW3iJ5eGvS0J5TEYnLOm9gCylU4XRdZsg03kob5CUp1Kihh4zCset9/llptyYOp6mqhWtQg9vGFLKiMJN+c1g2BGVs/aatXCNkUi1+kftpm3Hb5lnbDJRS0UJe2iZod+220bTfIJPxtdytCnTtJUFVX8TAlX4v3WhlA0d39Yqi0BzZkLk0HsVA7WJ2dzLoPj7722/KXxrBl4byBwzna8b+tePtmss2fs2MyW+N0H6PyvfasJRytCu7wupG6f/CU/oPrzO0KvBMLe+ZhVXKaCKsKjja0OjMNWojsxHmVq0KrCZXPKvKjh5+srGRx5SqG419qRpO5517qSgkTmdYVZCKQQGxtpaqkKChoXBO9IPvVG5GnVWloZhSpO1QbFOOJX/kOfnG1ioGszBrLjwWpynmFLrwOKiN5iJaSGlqjKqK2Uh5TBTDSp/lWF9XaOBlnMLwKkaEsH46XVCY9MaZ0tIN9EyzYIX7HDf6ecPEMpnH0ufTlPkakVNHmy7sk9YUaDb0NRt3WhFjrTCsGtOcQ0AAACAASURBVI4l1bxdG7LZrSFpSyvs86lG1xhaRWmU9jZ0nX9oVEWpQCwahedHKQ6YQZcJieboeO5kBJGrJSvTRQHQbWUKQZPUf7tMZDhxX3L9bnyFmGJV63/vyi3GTC4KM6eWUdjYdXNVjVk4djk0lapMpW9Q+VwF98VGl1yvumuFRl9rUuJF7krtf2N6V3WAlO0lkPwbrXxVSKwg05ylrkYHGjWPhJXnKnX7PmI27frMa0WMePKSoOtSZwxSu9tGRc8WW6vu+iIj68wkUafV78asFHAOrh0rRWGNDn1O157SXww6qQsAD8vQPgvjrBgZD/PYDL2NTInIkcSo7F2hU5LNHSJ3kmtLoeubESqERtE0wqRShP0YAPdX2dBEpbyyOeuW17UWTWy2lfKxhmDpHOj6XLd7rbLCu6cDzY1lPWcHrWkdpbiFSgDAefE052hwqrTxcJoHlLyNcV06qIs3hVAVCj9ItSisHYMt2GjKbADRHGjxRtBcz+ctLKDd/4K2yOfZNy1sc6Xn4zxvMkh023woeWMMRgXm84CaFR6eJqpfgNgpg0qT4VaLwlw8alE4iYHT7g8ZqWtWCKtDyQqxN15yF9Yg75WiMKehfVbKkFGmgJp0M3hQFVLYZFiC2cIPunmI4YICrMVvhkEFUtTIfB2KQlosUICZOQWyIuj5YTWb18l3KCGaxiqbkyKCWFF8s0KOtCbCajbtmce48jlUbAYWnwvBbN+5tLYvjtN3hTmYzghRiGIcSvUeQqtAXGhMed7azEvXb2cYFEVtV7WdL6vejaMCUFWhdB4vqEDmeqoqZLXFP9fVbCh3ZzjUmd/3/RzL3uCRoir2HkjoPl8aVN01z4ykVp+e056ttu9vd/172n+vgSXHLq65tCXfW3o5XRp1z8qVVi/77ZYDfX/O0PtBI+7aeK60L33s232PUL9BuYbSfqfyq2TM/SqN5c+4fK8NS8M5H388vkWsBl+FG84F+IBRR/zJ+rKxwr72Z9zHCT+e3mIuHk/J485FDDrhlD0+Hp7wmEZMJiIygvgQR4wm4rdvv8CX6w0mMzcWWQB4igNSNfjR4R4zM78u2eKHhwe8XQ/QquDT6bHlABMlPxWNm2HFYBI+P79oiGPPCkt5yRaMlvJynZOHt7F9B9Dy3kl+McmPd+Lcf1/MR4w2NVbYG7/i6AMe40DKuqp4PZ5a7ryZc7Bp1JabbU6uGV1KVbwYloacArSb3bPC9miqoKwfTWecokfkdCKjjXtWWEdMvK/GmdDVwwkxG7yZD3g5zXhcB0yOcg3eryPm4PDxzQkr5yqLjIzmohG7vGDCCguQISBopuRJy1Xh5TRDo+In74j46NMXT4jZ4Nyxwkp+NSliQIyejk+N1ZSQMLnucR5gTMFgc0P5zpznTFhhAWBwuY35/jQhZ71nhXWpoZDv5gHHiVlhGQVsrLBVYY0at9OCkAxOi8dx5FySyeDd/RF+SLiZVizRdu7adWOFFXZSNgQTp2sx/H1ePe388/XCCjuvHjlrDEPEu4cDpsOKJdBzYnijIBeNJ0aZzqtraFZjhdWlGZRidHmfcDoPGKeAp9MIP0RoRmaFoVapinUlw8qYykyxwMPjBGMzxjGS8ZhMU779kKB1aShTyRvZVmFjd10dFAA/0PUx6MYam6JhtAUIq4GxBY/vDqhF4XA3N6MvrK6xwpZChlJaCWHzh7gZ7R0rrNyX9TRiOAY8vZvgpo0Vtlbg8c0RypTm6p5XAzfF9usbmBXWT7EhGtrk5r7/+GaCeRGgdcG6OCLIYrf4OluYF4EM/EXDDgnLkyeW18U8Z4VlxV37jHi2ULY0dCLO+hkrbJxdY4UtaTsPBZRlIwJTtqCcDKEE08YKm55GQAF6TEhPA9SQNxSHEQ9JmZG/HFCHsrm4R01owTEhJzJewa765TwAtwk1aOR1QL1JxCo7cfsKqGLMifugr3tWWCEYAxqLZ2FW2EbQpUHXREWfe3bXqlBd2QjAAKhVN3SnNvd7NNbWqgBilyVW2JoVMeF2Lsd60aiHAvUzg6orqt+QGVUUkEDagKtQs6Lzq25kO9XQ/POhQC2KGExHdlXOVEd5NBfI4kj+1RHLLEAu4iorGCYFSodKDLHgMbKLbR4ru7EzAY/f3HuFsGjPCkuy0z0rrAf0qpCnSgROHo2t1Z4V4ovvrtXpldhmAXItFUNJZYU81uZuqlfVGGvFXVXGXBy5zlddmwuqWRTJzm6Mrj0BUHFivPHaM4B9UigD9V8sscISoVRtxFhEgETrSQeWTeR2NaCENMhxKANfJ32rhObKqsN2nbgeV4ud631VRJAk49YZe1ZYfoTk+ubS27vi8rqW70LaJK7RwvKKirZGxUX2a1lh+dndufgWtbkIm64/bqO5Xu/QPmyGbOdqv7W5HfumrLDAhb2l8NwQ79FKcfmWefUo5qUBrfbjulaE+bUnMbrqjvueWN5df88b3x2na3gz6UNG44cM3Wt1/7z80sv3Oo/lR//Qx/X3/tN/Dg+B3AYlYXso5A4nyrJmxMcZyg1p1aboSn5LQcfE1VQSXteqcIoeg6XE3T3BjbQfsoERBEwXrJ0Rs2bTUMCe7EcMoIOkg5A8f13bS7LN1U/cS/fEPYSwVr5uYaTI6ILEbpNrNkicwJhQFgVrtsCQkAy0IkTLmdKQJ9l5FhZdGbvkdHTshpiybm6g0k4bH6eqiCwfcfGTdCSS81ASKUsuzcAGqiRWljoVaO6ZazJX3e1FfjL+xGiS7gxeq0tDEgVJOIwrtALO7HIn8hZkqS+W8xWKy6XEv5Jhs+U3NIzOigEmeRrF1W9DejZUzVpS2lOXHJrcNUtz24wXDMX99UpVdqcU97bNeBQ3VkE1JIF2nw/xss3+O4DmuijXi4uj5HDMWVM6kpZrE80YEnKknNUudcYupx5f01DGTO5w4ionaUFqVVCdm5hiI5ESZDPCz66Gcj+EVZmYkRmF4uPyqya5FSm/YGmx1CSbsmuroacsQ0pLgpbe4zKpdu/eRgzSm7tj01EklhpkCG45JA2jWLzObW5ykPVNLNbUkDKUZ1FyMdLcto6sz0jRtPjoPqm2NhmZUUFtyI1NmJ37vI4iQDEapV4teJa7sb/PytRdLLi49dF93JJ872K+04Zs9cckLrtfP9KXUuxixy5/UJ0BGzuWaFn7rpCBrTlPYyQjunRIkrKl3YMWg9673sn4ahdTruvm4gdsBiYbv8p2wZ7c5g6R6YmbOvRO2Y38Sfqtsgb6OHLpK2ooXxqit3uBqrq5EQrLtanteRREubFe60qon7gzqrq5GVZsLNmZDXd0x+Sa2F0vSjKwi3Ojc9ucd2hJt2bb9b2MVTcPmS+wGfXftYhrI7AxavPYJMa4GRsFm8sk0DYbmqtk3c63+O3uN765XMqc+1KpvRbLXjuDsq8r8gXwzI2yqp3MdohWDxNVJnjS3Weus81F7Q2rrq1mzPRxmuiaF8Os67KhicDe6pB+1PN2mqF3eXvrJor+kOo/fIMYxd6wv9b+bn1ednat/rXSKTQ7Y66XZy8n7I/v++ku7u/bZZcfGs+HDL5vUL4RyvhtzZA/hdnyB//eLz/X4/SDH9ff/lf+7PNY/p//wS9/bsD3HLFcs8UfPb7G/TwiV4WRY7rm4BCjwWEMLf5oCY7jQigOaRpCy+VnbebYkdxiQ7xPLf/gw9OEYYjkpsXKIkCxMwrA+TxAs2LtHMUk+YEMxpWRGQA7EhxBTpabBbUqhGCbMUCGW2Y3rAslOZvdk+ocITbOZpzPAxlfrNjnG3IHDMHicFixLK7F9oiBsJwpTqaCE7IHIRMiRdS4vDMCUiT59Ep3T3zUkqKDFGdjCtJqod2GrEk8TA4GdkhI0bRYDeMLciClz08RYXZwI9WpScNNEc5lnB+Hpuw244wV5N5QSAshWZLSBNiUaqUrykqIiP2EkKjz/QRlSqujGQ1qipeusAPJXGTVFOCqYDzFBilF8SwSTyPKpfYZeTVtngC2eBUF2CHB+YT5NLRzNSmkKbcYmsi5AVvsiqRSkZiX2QK2wAwZeWZ3VVugbgjZyzOhSpoV0xbbIkpwUU0hU4YQEFHMtM+klBsitSpFoSwGesiEnC0WZUxIs6VULwBdz+0ZX5AXg+wzubWp2lKpoAJwJEdRtutqoIaMuhjkQ6L7VRQpAa42lzTlKU6oJt0U2HoEudSt1H42tc07n/nVV5jsyPDcw8YSowZiiamLIZIiV7a2LPddeRxRQd1WKF0odY0Yxq5u6WlUpZ3/KRHb67l7/QrqlXQjhaoHoDw5qFvOn9gZ4OoFPS+otD7qAJST28Y+8jtnNsiekEikTtvzGeVsgUwImbK1JbXXY6FzVSG7AqwaecrAym0FvVPGqmNDI2o6nxQZMIvZIWeSVL6OtFbqbEjRlvUGVq5FOU8KdaC1UJftGawjSDtaNeqgaDyiFFo2Jhid0GNEXDzVURUYCj2rs6Fx90aQLVCzIXlMBVg04HKbryoKdaL7SIy+FDdG19O9RVaoA2tvgQ0nX2iNC9pYmFhLZCNrvQIKbLwJYVkFMJTNCJC+igKswKZUX7lKRGeG1yajjPTsZUIgD4nGGPUmK0MbA7WCrrMFyBrKE1JrLG9aFDbw5TmOGzqsbUFebEe8pp4bjEw41d4Lq+YxsmWheB3Oht4DSQOmQEV5T/JaU5SXsx4KIcdZbaRR8h6YLeAL1MmijgUIMi8FPSXUVWCzb1+qz4BsNmgev+JnOBrKKVoUqs1A0qhlQwEFHW8GfFHbeQ5zqV2qDpiKyru1ijcJm0HKSH6dDRm1UQGuoiYQ8ZImw1Tyf9LYSyNxg6DRTHCnIt3/ynMioiY6rwroPeMrXSOEPKYCAYTcV3rXqQrK28pGdeX1qKJCGQr0zO97RXOplZ4FVRQghGugOSiee7GbnFVCI2YD0HKsom7pb/o8kCIribGU2McNDSV3UA1CVnsCJCmqEiKsOXUTNdAvCrSxCLLa2tC1I9ri+qUbntq30aOOmwHfne+MccWv9vZd6lfpqrb/FWpD7ds4+nY747n95Y2AZ8ao9HHpWvyhcmm4931flmuG+Yfq/3n5lSrfa8NyshF/+dXn+PnhhtxL3YpUDJ7SgFP0+GR6wjl5jCYyec+Ctwvlrng1nvEUB8RsMNmId+uEF8OCxzDAaoq5vHErzsnhZ/4Wd8OCObmGhALA0QVoVHwxHik2U1UcXcBX8wGvxhmpaDyFYedKWaqCVeRiGrLBp8cnFEZFvc4wHPN4sAFfzDfN5XRyEanoRnQjaMloEwaT4E3GVyPN7eAiztHhs8MjHuOIc3R4Nc54CgPWbHBwEYNJiMXg3TjBsxvpaBPO0TVUUxBVQXK1qs2NVOI512xw4wOegsdgMiK72mlVMUcHb1Nz/xSkcI0Wk484rx7HIWCOFs6URtRyXj2MLnh1mPH2POFmIIKflDVuxxWDTfjKHmBNQcp699foAqNqIyM5rRRn6m1uBDXHITS33aeFSFF+/PIdSlX4E0arJx8R2A10EfIIRvGOQ4BRFWd29RRimsyEL4HRuhfjipAN5uBgDbl59uQ1S7Ske3G7htueXMSX9tgM8ZgNDkNAyho3Q8C7cYThzYSYDMVtMhKmFXAayV31OAQ8juSz5UzGpzdPeAgDnqYB3iYY/oEVRuXe5VVQSPku8XdjI+/JcIbmfF5p08boinlyeDEtuD/TupK4yZQMrM0YbMZ5pPoi3xi315Btcbb0dwkO0xBwXobmwlsKjc+53GIAvUvNM0EQ5rvjjDNvrAjC6m2GsxlPw568x5jSYjuFUGlkQqLz4om8x6XWlmze9IzSr27PsLrg54x2AttGkyA/WlccxhVGVzy6jUBENnpS0jCchuQ4rXjQB7y6O+HBjS3OEABe3s5Yo22I+OASntzQ0CUhXpn90FxxZTMFAD65e8IXIETY+dRIc5QC7g4zvtQ3jZhnXR2mKfCGWUJY7RYPCCHvKYjBYhhju9frQAQqpehGSgIAw0jvlGVwe/KeohmdZdKWZOBchnMJi/e0aZI1Bp7bsjiMY0RY7UZCY/MWLwrg9YszHl0mFJk3DK0ueDS1rTUZ1+3NjHtFG4bTEPGo6dgTK7q1KExTaB4NALCsrj0r4i0w8KZi3+fs6ZmsleI9S1bwTLQyHdaGhCtFJEg9mj2wK3YjduH3wjjG5lGRs8ZhDDg530hzLG+0iov6bAa8uJ3pHby6xl6udWVPCNPIdsJKhE0AmrdLzhqL9fBDwjQEPJkBzuVGinV2AyzLX+aeooFnsqWwOnLt5/t3NkTeQ+7opj2Hq/OwLrVjcRXyHgXLRmkeDIZDaPGrNSsYVzDwWg+2EnmPqZgONB9ZW4fDisf8TbXh58WPCcGwm78vzbtCm4JoHYwrKJk8GArHYlchcpoiaqFn0TnyeBJEXc41ki61J+/J4n0iqHzWGI4BwbJ8koYbEslDXMIB8jJg49z6jBxMG2PNCsazTFfTDEYA0C5vOap581EPeWOV15XaDoTso6LFxpppI+8xlvovUZP7u2E6bNmMqRt5T5ENmEJj21yR6maMJoXS5aKGrttmHHu00MPUWSGCpotx099+PlaSaqzHl+Q9YiDXft10hp4YgpVzM2+GJRv27zEsr6Kjqj+v9p4HwHPUWeG5p0Hl+peI5RWjuXV7afh1m4dXxyiGJa63t6su4+mv78s1o3Z3Xn0Ynr2Y57eJ3/yllV9jI/l77Qr7w999Vf+F//z38Kl/xFIcPl/uoFXFJ/4Rd3bG3z79AEcT8JAGfOKf8MfzK/zGdI9QLH463+JgI166GT9dbvEXj1/hj+dXeO3PmLPD2/WAN8sBBxfwu3ef4+8+fYIXfsagMx7TAK8Tfj7fIleN37r9El+tR2hVcb9O+As3b/FHT69gdMFn0yO+Wo8A9kbl3TDjzi34v95+BqsLxS4mS/GLReO8evzwxQNGQz/Cb9cDji7gYMOOROQ+TMw86vCjFw8AgHcLGcl/+OY1bqcFr8YZf+/+Dh/fnDCYhLfLhDVaWFPwm7fv8BhHOI5fvBuWRlw0mIR3y9SMMAB4yXGQEgvqTca7ecTLacEcXTNItaL4xlPw+PT4hHfLhCVZOJNx4wLu1xEvxxlfnI54OS04R4e7YcGb+YCPDyeco8cXj0f84O4RXzwdcTuuGG3CF09HrKvDZy8fcQoOk0sI7C4bs2kEL6IwvTzMsLrgFDwxhwJ4d54w+YhcFD46njGYhL/1Rz8EAPzmZ2+xJIunecDoI3IhY1aMt1wV3jwcycDkOEhxMQWA88LXVYXT4whjC8YpkMLpEs6Lx3FacV48DmNgcqOMXIjd8/w0oKwG00sinzKGjNx3DwdYl7E8DTjezchZIwYLP8SN0ZHd8m5vZoRkMZ8HHI90P8+LR3gzQh0Sbu5mQi4ZgTembAyZ0cAyoh2jaUp+Y3tcbENzSySlYzwELLNHiRrjTcDyMGC4XZGY8MXYDGvJLTfOjpSh2RGiWxVGZr3UuiCsjkAdRn39mLCePMabFfP9CHeIUKwIr6vjGEsgLBS3qF1pSv76doSaMsZDaEQyeTXAamBfBChdYG0hAphIsnNjakZKmkkBcoeAWjTSapi5USGzMqV1QVocIdVfDUBR8D88kSswgLQajqUEMUBmRShkVlAvw5ZiiNk+e8S7PDq4VyvSlyPUy8AIMT375Y1v6A1shVoM1F1oCl95YsX3RUBmJFOPuSm45mce+bMA7QryyVIqF0tosb03SJ+SAVUXA3Mbke8d1DGhni0w5i1XpyJUFIXjIE+WEL+gUQ+ZEEB2nRQ3THWyUEmhHPOGbNlCitKqN4RvyFBnC7UqlBuKsYQr0I+W4q+OCfrJokx5Q99E4WR0avipQ7wtKAdKf6PPGjoD8VWi2EXFiE0B/FuD8EmCmg3ck0L4KMN/aRBflaYg2nv26OD4sDRVQnQSuecVX2FPlIu2DIT0mEUh3nD8m64Uh+YqzEmj+Ar3wClZWFHMI1CG0pgj7ZNqKU+qxC6aCvfEa8cQ0uKeFOJtbXGKJpBiXhyNKbzKmD43KI7O60gpSFSmlDZ5qCie+stThX0i1zkdgOKon3hbYU8K7gSEF3ROJ4qRS4ctVi6PdDxNFe6RxpkO1KedAVQgvKI4SZIbxS7qDMQjKC6S072kY23oiVloTulQ4e8VigWqJTTJztw2gDQBdqYxukcaj1mpTX9fMX/63ZVO90TtAjROWQsqA/GW+zGAO7PclCBugH+gz3mg8Um8YLHUblUcD8quo306keKxGQygeQ9vKuINp+RxgD1RvZayBXQfMjvA2DPLZqE61dAcUIE8URsSL2kC1TEcV1kcYM8V1agtn2muyIOCWcjwywOtWfdUUYT8NdLx4gF7qi0Hr0obOmiWyn0oKE7CW23XT6pNziQ/BRNoXejMY2Y5t2tEVmqTRx8/KSlXJD6xTy8kyGeTtd7k0UqHNEqffToZ3eRfkR3Pmce0i/ks2BVVKqpWDamk8VLsY39e54piFFSlvybSX51ru7YqOi99FSuo7nW9v9llnZHZy7PZbxJj2bkwPyuXxy4Mx53bdd9397c1pXjMl0nF3zf+D5g1//N/82/+0t1Fp89+XH/7r/wCXGH/w18NV9hfqGGplHoJ4G8C+MugJfWvAvgDAP81gH8AwB8C+JdrrW+VUgrA7wP4ZwCcAfzVWuv/9qH2P/lLH9V//j/7ZxEKGWQAUEBxdRJ/KEaYIHyhECumRm2pPCT1Rs8qCqBdLzGBudtVzlVDg0hEYjHQqChQrV3Lb4tUN9KevhCDrGpssv0YexZZQQbE3bOvK6ynck2fx1PiFzPLQhA8AO2zxHpKvFgfNyblcuwiC6kvcYZSJB50J6uOuKif4+W1QriTCzGs9vGHjSGVFVqps8lTxrs/dik/YEupIciUIJpKEfOoVls8qMwZ2N6JIvN6cS/eJzs53jYAuzldlv4aqdPHiPbpSXqZX34HtpjALb6x7OIpezZWeQ3I+1ra7L9Tm7V97/O1Sl0xEN83xn78PeOqFHUhkj5Osc8hSzLZXLRlXPv5V3bX3uIWt3GqZ31dxsvuGDGx/y3bxZ3xr5k2Eju5j/mr3bMgsZsAGlPpfr7boASZ6lELWUNK1Wdtl6raeHS3djbm2a1tLTHbbIhSBb5Wl8aEKvPvYx/RrZf+nvW/91d//0V2ei/X/pyMs++PmEH35wBsuXar2gRz0Z4SJKOrI7GRLUWBvCMkXlORklyzaiRA7b7r/f3qYynlextfv1Yu6vQxgqpbu61up4C91yWsjzHsj+PiuvbiUYRmyLF+KnKtICT9ePsisYqXiICqO1ftXZ+XcY8K++8ik8vPl+/Ha3O6nG9//rJekxue5fH9NkXyrJLSvZeRuoxdvTa+iudzwZXv6I5f3q/Lc8D+3uHCEBDR6i63LC5EfmHgPEOO3jeu7vOHjIXnMrlYPxfHVC+7vr96XcRS5YNuk1fW/K6f91137fp+jN+kXArlA+VbtQtcX+fvq/dt+/wW8vjWbf8p2vyu5W//+7984+vX3bD8RbvC/j6A/7bW+i8qpTyAA4B/G8B/V2v9G0qpvw7grwP4twD8HoDf4X//GID/iP++tziV8Yl/wq1ZcC4eb+MBTmXc2BVOZfw83MKpjLl4eJ1wSgNe2BlrcXhIAyYTYRRR0N+5GY9xxGASoU1pwDk5HHzAp8Mj3oQjjnYFwMYpKh7TgFIVXvm5McUu2eLOLXgbJniTMZnYGGNbapBsMJqEyUT8bLkFQDk5l+SaYbxmi4/GUzM8l+xgVcZoEgpUa09SiqzZ4gfHh41BVme8WymH5mQj7sOIF8MCrzNOyTfioY8nyvU5mohTHFrOTTGOT8k3V89SFY6O3EjFEHc64xw9JhsRitmMdHbRnZPDrV8bGjuYBMNss0cX8LCOmGxEqpxHMwy48ZQ/9HEdcDfOeIq+5c58CANSNnh9mBvRUm/4J2ZHzUXD6T0rbHNfSxaDJfe2G0c5Bf/ePbHCfnxDjLSErlJqmqEjOypV4cQusBOzwgqSqQDM0cKzy9s5EKvo5CMikxyt0eLQscL2hnMuurG1ToPkywO5ma2UC3ONFochttySpss1KRsPh3FFSAYhWRwGziGXNU7nAY5z0Em+R+nfMconbUpOSPmdErbfwHGzwrKq1EYKlJLG4BOW1WEcIrdPBp5sZpB7Y0JkRldJOyOIsJAticE6+IQ12PbXMHqqQG6rw0AodGB3WmNKi4FeZt9YYQEyeoUZ1g+xGdvi0lorubWKTGIkan4/MCkPu68B9Nnazf3RuoTlTGkappu1GXEpGspTyMaNuAmiAm7Y7r+4uQpJk1IVYXGEBp893JCa2ygAhMU1A0aLy+G4scwKGu2HxGRCFdalFrMdTw7uGKEcbQ5I3CyKQjx7uOMmM+vIHdG6zGy4m4ylTi0cN83tlKi3eFwmONHsUloCkQYJsU5jjFWVXO8MEQCJC19Jilzw2AgTMh3tM8pqoFzZDEjZ6ODce+XBEdOpY9KawGv+kBp6K3IsT247HigesT5ZYCxNM6rzPkUEbGUXNN2MvRoMfRaXtMQorBitmuphNVTnRGk9qhgF0qYo0UkTYgps7n+qQkXTFMlqKvSqUXzZ4tq6GDcVFOpYoN9ZcoVjlLYRzDBDbLUVeqX4U8WIp2bXwKqAMhbooIlxdSS0UycFMFqrE6GyxYEQIVuhA/neFU/IieZ280QMqg3B6VBfJXkKC1/HurhmNK24CjNrZoWl9aWSgo4kj2KpbvGV2FktGU5VEaKYjt9de9WBkOaqABN1M3SwGzsamyqAhiDpFRybiF28W1WMDCpscXrALr6tWuyNNwOYM8lHZKijbvJomytZNdIgHTWK2+5TVYBml9Pi6g5B00mhMFsvFOhzlI0FHgsjiL3cSUbb3AUprAZQEaiSsad08+Q+WtygQnPZFHSxNzAFeZRjPVLYdPa9wAAAIABJREFUl2t7Fa3Pgr2tJ32LDC4MTkE4L8sz4xmdfPiYXEd9q6vjaaXbdLhq8F8zIvU2vn5el2PcCKWez2M313JdZrt6Ms/++7U6bRD7Y4p30xoS+3Ug1zc1ni+7/QUbqt+l9Htgv47lF4ZYKqXuAPzvAH6rdp0opf4AwD9Ra/1cKfVDAP9jrfUfVEr9x/z5v7qs974+fvC7r+tf+S//KczZI1YNpwpi1UjFYC0Wg067406VZgAKg6wYUJJeI3AsoeTDFEPN8udSdbvWmwSrCp7SAM2r3OqMJTt4/gUMxcLrhFJ1y30pqKbEhZaqGpIq56WdSxRTYuGkSKyjVQVLtjvEcLCpIa7iKkqyrQ1hi2wgApS+RZBfOS/suC1XZceK2ep08hMjtK8nRp3UF+NFGF8FEcyV8mjGouHYuOnr5KowWk6uneyuHykyLilrh+I2lJcNQQAN0T16TijPMawyDqmzu17Iibq+5T7JWCVesWJjj21suJ1BTMc39NWaDKMqVo4/lJyRlhVla3Jj8u2R3B6pTYwaGl1bXKlSlV1uNeeFpPMbKqtavR7p6r8D2CG5jbmVUTWtsIt57ZEykY/n1Ct9flLJKwpsvxeyrvv11N/v3iCXsbQNbR6vxA7LfdK6tPhfMWD750HuaUMWhc35AnGUz5dMusKgvMbt3vWortSVcci9EfRVyLBExqbFGSYmDttusnPpWW7LfT+EnlLM5oZUC4LofWo5KWVMjTXZZoRg23gl7rOU5/ORjQNh/xVZ9PF7YlTLr4AxtcUGbmtoL0tB7noUWFBMLQnb+XPpmKxFE5K5eCFd61hnxRC/RHINx81K2heJ8ZM4XsiakLWqCHXe6ZL9uune3ZLDUuoICoyuvhRBkvs1U7n//udajHs5p3Vp8XnCACyflSGj3Upc3yUyr0i2JNParunnBVAcmhKiK3bVbQhtVo0ESFDXWlRrRxBK+f6MKVcx+VjaNjkg6LIUxQg2o8nNa4CN5J6RlwjTFBECCTpamT03XrEQvmkxdcuD2SOWChsbb6/8NssQOwKnLXZO7c9dInn99X2p2FzA5XtH/LKvq7bxdrJoffTj6OfTGTl9eotdne64kBS1tDhdHWGPbay16qId6aP/LOMSpJvl05PGVF6/rd3LtkU22E+7dX3ZzzVLqjt31VC57PObGIffoPT2U2/DXd4a9MO+HMe1fj9goL3XmLx2/df1d9Hudy6XN+07FOn///53fvmo3uEXhFj+H/8/QCz/IoAvAPwnSql/GMD/CuDfAPBZZyz+FMBn/PlHAP64u/4nfOy9huWkA/6R4x/hi3SLWA3uDKGRj3nEUx7wqX/AOQ9wOuEpj7gxC95Gind85U4456EZoG/TAXd2JiNRVRxMwJ2ZcS4ef7K+xGt3wlos1mIxGSLSeeXOWIrDm3jEpAMyNG7MinfxgJfujFIVHtKEQcedYaJVxSkNmLPDD8d7Ml6Lh1MZThEJkFMZb9MBgbcAJx2wFoeVvztNKM9kIgwKJhPxJh4Ri8HRrpizww+GB9ynCac04LU/4TGNWLPFrVsw6IS1WLwJBwxsNE8m4iENKFU3o/zGBliVUdiwPGUPjYqjXZGqwZotjnbFiRHgObumpM/ZYdAJj2nAaCKcKihQfE3AYxpwNIHqmYTHOODWrbgPI0aT8Nqf8fP1BnduwVoMluzwyp8x6ISfLi8wmohQLKzaXHOdzs0wT8XgPo7NLVoUvaMNzYB+SgNS0fjt2y+gVcUfPn0EbxJGExklLjgnYhAUV+ePhhMKFE58PGQLb8iIP9iIhdt+5WekqvEYh7ZpcbBEJHWwoW1I9K7Tr4YzrCp4yyl0ZGPj9XAmNNwv+Go9NvQ6sFysKrA6tzhfZzJe+hlfLse2SfKbh3d4SAN+Pt/iYEMz3mRjAwCsyi3vqje5uXiLQUeEVuQBIK7iMh9vMh7CiJd+xpv1gNEkaFWwZNeM6hu34iGMONiwy58qmxeDoY0D2Yw5JY87P+M+TLjzM87JIxaDVHTLkwpsRFqhGMRimizPyeMUPbeZMVi6t18t9B6I2WCwCRPnh5V0RAUKRxuQqsYpemhVGwIPoJFfiSu9eAwAwE8eX8IZQtgGk9r1silx61ZYnfHlfNPcyr3OSFW3TZhaFV6PJ/zsfIvPDo94ux5aWiOAYp1p7dF8Jxvxbpma8X3jybvifh1bSqM1bZtXPzg+4KenF4hM0EXvE93k9tPTC0LrbcIpcM7Z6HFwEXN0yHVz/5bNniVZ3HjyaBiYCEzyy9ImFrUv/T2GAY7nn3nzwTHBl1EVIRtMTDT2GAYyxotuXgin4HH0Aefo2iaF5RQ/Qjb2Gzf3eLsesPDcb1yAUhVv5gM8eyLIZuLr6YwvTjcYbMKNX/HV+YiPDie8mQ/NsDp6ul48ONZEuYeFNCwxCRqwbWqNLEPxEslFI2XdCMJejEsLe1CKSM8kfRNAxG2lqjYvgAy9yW2/K3Ifz5HWZ0gGrkuLNLmI+3nEp7dPjQRO1opStW2krdE2srTBEiGWeKPEbHBaPQ5DaHMyujTiunN07X7O0VJaKCZqU6piYRIzmc8j5/wF0NJWGU25fh2TrVlTGnlaKRpeZBstjmNo5GWFN0+lvTk4TD7iaRlwGAJW9iQRArg3JyK6+y5lcBHz6mnziknDANrMmXnsuWgMLrYUX7IZI3H7Ml8ZeykK0xCaJ4lsergunGcLO0DbhLqZVszBNc8J2XASrwepL2mnZEOpJ80Sz4sQ7JYiR6FtDsnGVE5bHl/ZRJINIiJtIgb0WhX8ENt4ew4AIToTy6VtJLL3Ss5d2EzRzQW+3+QowmLOhnmfLqkW3tS4sEPEM6KFRXSGaiNFE+IbIfoBNiurgtip+zCH3lDqDXfZHJCNA2ajfebuLW1cWoyqbm7lslkgn/vNg2vkPeWivd4ov2bQXyuXlusHmF/bRoKstbr/3h+7imBeO96P4dqYfl3Kr9NcLsov0rC0AP5RAP96rfV/UUr9PsjttZVaa1WXQUZfU5RSfw3AXwOA8bNb/M0/+sfx0XjCOXncryMUSMm89Qv+h6ffaT+YBxfx9jzhbqL0Hm+ZwMWbjKfV45PjCV+eDzj62Mhz5oVShfzg7hFvzxOsKZRfkllMH+cBpWh8dHsit0dVMa8er/4/9t4kVpctSw/6dhPN35zuNu/e+5qbWS+7yjJYNsIuoWKABUhYCGHLEjAAISNmMEAWYsCIERITJCwzSYQESEgIAUNEJzGwkSirSNJVlZWVldiVzWtve5q/iWY3DNZeO1bEif/cc+97WZm3Krd0dP6I2LH32k1ErPZb6x3OtwvEqHC8bLDrilGsk3MGi6pDaT3+7vnXABVRJTQ35zRi0PC9xvKozdaNtrMoCo/SDpaKEBXajqwOrrNYHRHgS9talKXH9qKGqTwhAm4qVEsCemmaIicmX68bdL2lROktoSxqHbKrYrMvKaVHSthd1R363hA6oQK08XAtAbpwzryQtMXGhIx62KZUJ0oRWh8njm92JcqEJFlVPZp9SSh+nUG/K1EftWg2JWzlYaxHu6mATqM8a+A6k3IbDnkio9NDjFIEufSpCJ9ShwCAayx0+kBWC5qTP/z+B0AE7P09fG8QGgtdO0SvYes+o0xGAD4Bo6iakPBMkVwAdURobHa7UxtLLmW1J815QWkT1ILSZ6h6SPrO1gK1tQTLvk4IAzq5/G0KSguw14hrTy97ThmQgEr446XXPbkLNhpYENBKbAy+f/Eh/CIgrh2lQmBNt0JO85E/rpyygDXrHA/W6iF3WkptECsCa4EH4iJAXxmi36kEux/zB1u1GnHhoRpNEPUBKZ1C+gB2BKiikstVrAONeRGgNwZhkSxGNhLQi6WPpmo1EDD0BcBcWoQ6ULqBiAR5r6E7BX+U3JtTTjvlaO5iAk2JmpLKQ5H7HyLI1TD9Vp2mRPYa5GZYRnzy8hFUALq7Pn8I9V4jLNlPC1BOwewo8bw7CrmecuQeFcuQEQQ/2Wr0px5PXz6AOwpDioCo8NmFJkAXHXMydn8UMhOg96mPYw+z06ntwbrx9PkjdHcCJbDfq+T2RlP70UbRNZ3cK5cBz680/DLA7DRCHYfYMkV9q0DgL8+Ta57uyFWSAGswuEdGAmBRbgBzke6MulPZVS+UEaahpO5uSRaJYCNe7mgMbkm/fYWUvy+5bSrkNCab5+/ALSN8TVaDFwkspjsJ5JIJkIthAC6vFLqzANMqvNwqdKcRu/N76I+TxVADmwRow8ndfY0MgKMC0W53KgGREM3seqk72tPB0prt91R/txtzbaGie9lis9ur7DIZ2KXTROx3zMTRGDZ7AmBBAEKV3BYDEAtg3xDwzpNnZwg2gbQktz9yd6VzoQD6HYG9tAlk5zIBmkRNwDquAXY7+q0doFwCiKkFeE9F7YcC2DW07q4GnAd6BthZA7GjPqIFfIcEiASEBJTiU7vM3/cMsF4C7RbkYmlpbXwHtPQJRKzpt1oD/QZABbie2txtItSdN7eAhB2gV+m92YCEnwgED5gVgA7QFgh7+h8VoDj1yzZCawVbAqqNMFpBJzdR7CKMAlShsgulchGWjaNFeoekRy8YAJdrFAsCaDFWwTYJKEZYLrWPKAp6LmwTYUoCviGgmwEEx1YEhqQ974kBFIaBX0xD36TsLprW2JCuB76kcdr9AoEzsoi9ZRpaTx6H8kjP/Xh/A2lt1VBPgvcwyBBfI8AalVKPSCtkkhmL5C4r3FwlAI0KNJ/aIc+BBLiJSkG7mIGUpiWD5HjRJoP3+MFFmOlh19iRNVOJ6yq1o5BBgPJcMK3smhtprrTDdVdY/sSncwwadEioydbKGaCckZE7zx1fmz5P6rAgKedh5jGcs5jmeZrW5zqvIXT+5HbVfu7lC1lwf8nLz1Ow/AjARzHG307H/wNIsPxcKfVIuMI+Sdc/BvCBuP/9dG5UYozfAfAdALj37XvxmydP8LJbotQev3b8Al0wJGR2C7y3vsiuqJu+wodnz3HV1QhQ+Mbdp9j0FXzUOK33OG8WuL/aovWEXHp/uUVteuxciZfpGlsUThekYb633AIAXjYksALA/dUWF22ND87OEaPCpi9xf73N9LN76b4vyFL28GmOkWQLklGkBb5sa3LZwxD713kDk7SyISqcLfcwKf7woiXr3N3VDo2z+PDuc1x1FRpn8cHZOTZdBR8V7q232VJymTS6PmjcXe+wTxrvZYoLPLnTwCbhXCG5oC6bbDlxSRveOJtdTFnj2CQt8a4tcXq8y+64vTM4XjZoe4uzd3bYdwWMjuicwd2zDZreYr10WJ9d4aqpcOfRFp2zcF7j7vEWhQ642Nco1j67dZIL8OCmyuPbtSVpRJdtdslcnG6SO2bEvqN0AV/51mcAgGebFeyiQ3XX5bQhnBaDBePVKaWIaVmTHhXsirT/9dk2uapGrB4Rgi+Nj65Xd0kxUZ653C4LxkpFLB/0OQ2KTvPZ9xbLky1cSjuySblYeT7ZfVKlce/bEnbVoi4cdm1y/T4JOPkqIffu2iKnZgFwzV2UNd4muXZ6oQEvrc9WNWtIS83xogx+tHq3w7YpYRikJuis+S6tQ9MV2SKhFMUrstKF7+H/XWdR3U9xlvcpr2ym7ySMYiNlvCQALB9RrtquS0qQhCZbGI9dk6zQQcGYIX8s0YLsLso5ZgEM2nYM2vwYKK1A31ms321gdMT55TLncrX3Qo5xBABtAsrSUbqaZsill11hU07SGIH6gx5hV2H1qMF2XyIKS8bq/TZZQ5AtAc2+hEpAP5xHNzYl7F16X3CqCkSF5Yd7+M0CIWjY+y6j9modyaqyrcn1uvDoWoviQY/YDXGW+ZsYFbShlBHRGZjCU15YGxA6C50sETEogPdYQfGiMaWkYOsLACgdoELKhesMYD1M4eFbUtypoKFLl+IlLVTpqB5bKfTgJqsUUHxzD9cWiL0lC0npiNZdCYxAlAC96BC3JaINUJVD2JZQX+8Q9iVZQqKCqcnapjVxRqEnSwznz/VeQxVDGhOlIlRByje2/FDKEQUUlOLFpNhY5uH63mTFH208n2NjY+K4oleIpc/xuSEqlGVKhwNyHTaWx6dgCg/XFLC/3uRY45EXTVICeqfhS0qDEkraN7w3QlBwrYUvPWLl0OxLaEN5jjndjE6WMNprFA9rEp2c7sam3MvdvoC2MY9HG8oDy3mP2d3Vt8IVOc1t6AzMwuVUHTFQvlOJbm1sgGsNbCWUnp6Uic1VhTctlB84hZUUIe1hlfMiK0sgUBxDjKhymgpTUxqO6DQpN5NCEV7BLDxiQELEBthtOLsVOz1mrANglg6+Te+sFIscnB6eByC7BDO9sddD3tqghryinJOV953Ic4oA+l0mJSArHBVIwciKyZ7Gq2o/pAth12Gf6uW8vhikiIKVfwJUixXEwNiCGFSOYwVIaSXdZ5UYey4JfXaUbiRrimksnGYkgxuNJBsM1kxBymDVi4NAKc5HBRzMY6nE/dPCejshhN7kxkrKprGy6JpgFpFzds5KNVwppvlL/2k+1DBmUZX6F/M9J0zGG67NnZ/cNh3yFy7/05fRyK/KTeXnjQr7dwH82zHGHyql/iMAq3TpuQDvuRNj/A+UUv8igH8XhAr7mwD+dozxL9/U/nt/7jT+W//dP4NCeTTJTXTvSxzbPSrt8LQ7gtXk2rcybQbgCVFj60tYFVBoPwLvWZg+tVNkgJp75Rbn/QKVJrcXl570q56Y/5OiybGbLmoc2RYXPeWoY7dLdqNk4B2ryH31absGANTJvdYllVjrLFZFm921GldkV0eJHMpumhyvCSC7Rp63C9S2R53cTFcFubXuXJnd3o6KZtRnZV12z9QqYu+KTC+AQZhJj7hRAY0jV9YpAq5RIbvpMSCRjNkstMc+3cvHDPDTB4NdX2BddNi7AoXxBCbUl9kdTroMsQDEgjgzTLUd8hvSnowEzJMYfx7v0+0KMSqcLvfwQWf3tjnwHha+K+sQgVE8pgQJ4nycVeGyKyi7qDGYD58HAOcNWkeC4qLsMwouu4dRfKXNqVJCIFcxjtNkppX7a51BLVzNdk2JovCoCxLQOM6Uc2AyDRIQiF0MOWaQY16dEBaLRFeMFEPZdJSnkl24ON40RIXemSxUcowYx6XK2Ed+LXH+T87ZyQKgUXE01yxwah2yUNokjwNryD2SBVzvWRClTti1i+MamcFn9zFrQxbg5G/uh3+3TYkYgcWyy3GFLPRJZF0pqEokW6XiCPmWcwo2+xIFC0SpXpvAexRIGAveoEgCFwsn3EfOfyfAf/p9gWLRQ+tIQoTwSvCdRlGTOSB4Q7GHHQmNPq2BZEw4Vx95KOgcz2cs5cpjFFjeY95pup5y4nGcngKyt0NOm5KYZF0M68I5+XQR4DsNXYSsDGCk2Ozmti0obQmD+/SECMLeCNRQutYY6KVD6M0A3rOz5FkAkACUhRwMDDHHrUUQ85qS0WcEVqepHjOlmqw+YICaZKXP3BN7DPAU90KgYKZb9pPuVb0iUB5ug93pVCQAoMpDbS2gCaSHGMYh7x6lhUnt2EheA1GRJSRZj2IVoHpFluUqMcyBGNZQCotNshjDDiBAnCpFJ6tjqAeGO+pxyhbyGiCrTx6Ton7IwplAeTQGkBcPstKmvpQjGnVD88KeCKZVcMs353t0nwBzQFazHHcXkcF7okkWahZ+0noy+A1bx7JlSg8AOFEsq3RDlAAwETRusx9oiQo5XYa0+LDQAYDAcywysBDPG9EoBCNgAIMRVjiifwD5YS8CndYx2JjnSApRBN4Ts0cC0TXEK3K6kGgwEv7yOMJgaYsGGayG2+f5Z8u+FMzy3MXJfRNBLVsz0/FU9roGiqOut5EteQrZEpmFPa7Pz28WSidtRNHPHC1SqOS9ocYW2bn7o8LYmovrY7zmcjo9ntz3RulGpm1MpMU5i+WIlkPtv6qk+r/7n//i4xCXDz6I3/jXvvwYy9/92zePTSn1AYD/BhSCGAF8J8b4n33ZdPy8Bcu/AEo3UgL4RwD+JuiR+u8BPAZZpf+VGOOLlG7k7wD4F0DpRv5mjPF3bmr/7rfvxb/6X/3LOSZsSMWhs/DHAgaDf7CAQfTFLGBNU0ZI4Y0FGJkigwFxJBgNM8cMPgMgp/XgktN0REIRPQRAAyDHHclz3Aafk3RPU2tMwSm4DjPjsi0uEgyG/09TgnDsD3uKsBAif2uFUUqPaeoQCZDCAoHzZrQG1vh8jufDTH7flOqD45Km49MqDu9NMZdG7A9Jt0xXIedX/pbpKab1pqkZ2Po3pNuQ8TADEIqkb7gWr4HJcN9yvWWaDu53mm5kmr7jUAqT6bpJuuYAbab9AhjF7lx75Yw0pfQ/p2CYuXaob0l7RBJkxDOkuJ4a0nRIUuS3Wn7DJO8wume65/RgkSI6kIWkYQ2Rv5hxMkd8j6wbAzIgiiRqlAolpjoSIELM+1xKEwZoGaUbEfdKmm+clOnEjJiSCXeQOeZ4bV7G8Ixi/hSy4HYw7YsQoKbt5RQimdZUR1o1+LYMbILByjOxUowYwggcZG4UBsFuCpgy7W86/1znpjmHOM/XRKqJWevGCDDmwHWeo1nODsM6yb7mypQxnf5mhvrQ3gGgYhJ8ea0OtTV5Nq7RMX2wNciC9aZlCpAj+5XrPOGcyfokpQbk81xtlkHnMnftJiFgsm8mr9FXCwOT9RjdLImX7+2ZsYysW/K5ObSWE/LnaJtDPmVyDglD19pJda4LMDPjlO3fQNetXTJvU+dQf69q9zZjf0WfBwU7vKL9V7T7xuWL3Duh4Q/+kz/TguUjAI9ijN9VSh2BsG/+WozxD75MOn6u6UZijN8DMDfIf3ambgTw77xO+yxwLEwPrwfrXWn6BLpCQDKNLzLoyML2CFDoPKGYWpBgt7IOjbco0ofXJwub0QHHVZNBOwrbw0edATvIItVlARWgFBYMKrIo+hEC5RTwok8pHVhIihgESqMiKhZe0/GcEAWQpalKbfokPHIsKP8urc/Wu4yamMA1OL2ETda5GBVsOucFA1GYkMA4kAVMRgAlBFM6Ry76MVvQeoFOysKnNYGE0CRckgWPXIWldazntUoKAgZxkPkxpSAZk7BP9CbXqTRedkll2tj9eNeUgxtgVHDJMsNWLFl8so4x4IAUKIZ8i0Df2xHi5uD2OgZFYCtZjCoDGzAaJoDs4kkolOTixe6DLIhKd0KiV2V3MFpPoG2LZNGL8F5l9zoA2Z1LChpjoVglV02T3S35G8b5Fvl331kY64c6SZhDVKM8l3le9MAl8D1wpDrXiqxm2pC1jF2GeU15Z8p8nYwQ65Kr5cgFMZBbmk7uX0qMM0YkpEmV2hyEMD7OORhnmOK+SSlPypDnLafPSAwP9Z8sbjZkITjPshCWg1fQNsA3KaWGivkD6zuThW+lKS2HskPMJru+aRtGrnU8z74xUCWp+keucxFkSStDFtoYRVOZJGzpYc8xQmcWbJIFcgCTEAIcM9Xs/sdWOSXmUuaX1GkcAeSWx4wqC3omufOxZRBq6I/Hui/GFsCg6T+7+ak4MJjOUNxsIBAQ2AjVGrI2IbXZsZBAY45sfRQM7TVAC69TugeVySQLI6URUVMXR80CSKrux/fl4jFwgApkKSuGuRtZQryi8eyTQoVd/gRKJ1n/knXT0H8VkYWlqJHjq5VLVs1A11XEgAQaVY555TQiuc+IbHWkGNSBRmmN5PQnCANIS1RiLti9UCG79qmgBusbx81ZEXvGU+MpRu9NC6fPAGiO85IEpLQmaqA1Xczy+CStBtNK6ydoj0ObcxZLbtN0KlsJaQ5TX8Ltc+QeKddcDelGeA6zZQ/IdaRrqPJTd1yVUn/wAKn/bI2F6F+LuUvPy2ic6f5rgjHPs3jGRpay4fNyfa3EXE3PzSlXhrkarNC5zi1i/GaVBFHcN9PnnDUw0zsn/MaZ+29ob3QPXz8gqL1SqDxQDgqNo4/bK2ic0vqKPqnfW1T6ZS2/ANITcOqn6feVUuoHIJDUt0ew/HkXHzUaX+RcjmyZ9FFn9012z2Rkxl1CiCSXyCEf5KYbYtoAEvTYgvXcLfM+lxYyo8g6uQ1lZgiNDhktDyCkxJDOS+tT44bUIADQTwRPrSJab9AC2SWyiypbCbPrHL+8VcS2LbNVK0YS3HpPMXDWBLS9zYIQF47lYxrbJJzzeI1oDwB23kCBBLQuQe0bFUepN/L6JOGha+yoTxYqOjcIGRTLaEbpBrQOaLoCWlOcI8f9cSyf1jHH6bEVcGo15DjI61ZG6nPfE7BSmeKJ9m2R++Y8jc6NHxMWbjgej10mqW3AJUZKukqmq+m6mdw/3GsMvfVDGIQd72xGtNMmwHuV64egRl8CpSi+CCCBg90goSKKwlOcFMc0iq9gEIJ/DANCJe+1/D1UMcf65W8lC5WKrKmc5zJfj4oshIqE7JDaZmE/Ci+CYRwp52G6L3iTLazSGsrPAwmUvL/Ss2RJYB3mINVL8VtAerfzM6SRXTMzo6HiYAlUyLF2PE9yzkyZYtpEPI1ScThO51iojVJgyHM5HGgbhnyPQWUrK1TM4FMqDUIXfrDMJQEZEUOsHisNmP9buCRQputprqAiUBBt2eLqdY714v+DCy+yYIM4pHZQRRDCIy9M+m+iSAGBTDNfGyxo6ZjdKnldkivdSOBk2icpG+LSDW6qKrWXBS3m7BOzX/shpqwgwTOfg+hbDR4PIxfYVCdaQR8w5IyUzFJUiKz0sXFybaCJrofRfbmwEMntlRMzjgQaSfMUl8OYeRRRcjgRWTiN5QznE9L1KeM3oV9FhZCEklAlJU4SUKIOg5Aj39dRtssPdhyEDqaNL1e81jxfYbyfFICgEGreP7QXvRExb29QsmCG68KMZLBHlrpUQj2hT655jWE8c+TNCCB+EUdT9UpLZFZODe9+r4Y9MRbQGLAlivYFX6HNAAAgAElEQVTjzS6KiX6/kH2OxzVH3+yQJ3Mnx32tz+lcHhBi5udGifHdICRx23OE33TPTLmxj2lft2n/hjHP9XvjON9wTLL9G8vrjEu0O0/zmz/Hf0rLPaWU9PT8TsKluVaUUl8F8BcB/Pbc9S9S3mrB0gWNl+0SrbPZtZTP+2TVkvDsHOMGkKAGDDn/ZG5Buk5xZiGmuDlhEeN+ZG5JFpxk7FeEtCrFTAf/D0GhsH5UD6Jt58cCi7RScRucr44FIeleypawgdkf5ycEBiGH7hsEGv6thXUCQD7HRd43dZekcwbBa2gzWM5i0Dn/HP9XworFQomsE0NSAqTxsuVLujSycEMTRm+iIJhxfjNJq28GT0i0MdptFkSFAMGd8PiDEAKyBUe4+HH+uZwzjt0a0//gZ3IxaqI1Cz7JMpKTuKf4Na7P11mIAUAChha55kDXQxJURlDtsg1Z5Ftcck/SrS99GLKVUwHRU1Jthnfn+3N8nB6uDfMiBS+mC9kVNtdN+yC7Ckp3PNFXZrTSeCEsB9na5sWYlbg/TMYa1VibLmQRZmg5sX1IcYAQiobIVjzuB0gIqHFUb85FMyZk3mjp/2h5skAlaHBC+GIrW4pV47qZ9ypUBtrI4Bf8ECWk3DzOkKxHfhjrqLB1jduKQFRq2Ce5oaF+zGONuMad50FiWJvJMzj6nRlNNTkGgAQaItdapTVRok0A0CrH4VGCe40Y4mCNQVpPqb0KyC6bQPrNzxa/V/QQ38eWTtq/9DtbRPk5DoLeCLIIxQmt6f4cIxmRrGAH5inFdMYyXGt/ZMkIGPa5nrQBDIJ3Gnu+V4EsqLzV0/yqgCzgZkseW5t4b0JsAyXiK7n9GbfTHDeY5lLxnsuWOeQ2olb5Pz0DKlsH36ikfYKJIjNbKdMzoEbCznWrrnw2VMSAtDw1rwnBZ1TScz5a80kMHd2I6+8wSZsEZhHbW24hJU6o6UX5W/Ytn7lkcZzG+I1olGPkZ2Suqpy/me/UVLi/1q7sU/SjeHzx+hJwW7Pt8LF6xfGBsUxpfKVwK9uf+3+LMtf+aDrn2rlF268llL8Gvdeep7e9xFus8ZuVZ7dx81VKrQH8jwD+vRjj5ZdNxFstWK5ti9+8+2NsfDWyJLbBwgWTQWC0CnAJiKdNGNhkbRxUuvJ+AsgZ3n6Ut8/nuE2ub3WARkSTQGmGGM90TcUMujPXT4gapXHp95i5oryHA+gPgAz8I+kEAA2il3MRsuW21D7nIJzGckqhWFpbWbCW55hugKzEGvGaQMTnmV62FMvf3A7Pz1zSe7YSazWgnk4FfnbnlYLsFGSGhayMdiqESdkeuxlzrr/GEaqinONprKtK69OLNuQ1+T2lORtbuflYWnm5L3bNlWvlg4ZNeRHl/KhJG8Oa6gz6Q4ivhJjLeSdlzHAWsMVacpwtzVUcuULL2NuB9uE8o/NyfC2XcRymcINV5G6d5y8rOQb31Wk8rpyvQzHIEeQGHcIAUqREH55dbsVaSY8FuX5sYVVirYf41oGuwiS0S2HhljTLcwBG5+didaXLtPfjNqwJo7hqak/OVcznMuKsUEpZ67MiagAN4n4pJ56kQSqawkSwZEWMvC49D2j9r9fnscl44usx2MM4uJ3p2OYUPMytDW7jatR3zl2Hoa7SMSu5yO18UHwBiW9PbtvshixdwnMd8a5TGPY8+P0gOck4eJBIpaNA+R8UeWKt+Xye1qjyGlyPQUVWZJlkmc7rwcx0qsfoptIlXirLWCkkFWNUCTmeV7rQB9EOK+qy4kTEdOZrfI/icAqMFU95vBgUP8wIK4wVRKzUkcIUqI73M23etihgFI8qz7Ngyf1PlXZSWSWs3Fn4x4RWWeYUNYzYyuduin2Vc8bHTLekY05gweScnA6pWJHKFN7PanC3vhafmhsR/Wd6x4JifmwCkK3b4xuEkmFKoxjDdMx8ihUfE3l1VOYEylcVMWfTKT1Y5gTgV9W/hYD5SsFv7t4b7rm1cHRIefhF2wXeXmHzF0S3UqoACZX/bYzx54KR+1YLlltf4XdePAYA9CmBMxcG0eHCCbeZoeZUD8zgM1CMZKyZ8WMUSwAjxoeBZaQ7pxR0IkgAkcx/FPQAGCFnSkZzFC+mIqYWzSgYiRy/mT/gyC6aMomxMYO1bwr4AgyM2ijOa4aJnIK9SEZQtivpmrYjGadD/6VlU4KuMPM2BbbJjJnoRwsap2sgrz/JyJnCysgf4clbjq2ADKoyAiIRX42MeHmA+VKCyWKisjujuKYUuSPmL55knrge3z+xHvJvROQ2smtkHIQYblueuwYkwww708FjVYKZYWZuhEqJYQ7nGL3pB3G0QBhbUeQ98lgyenyOLZVTa2QU57hMmP1cj/uS980xhrJ92faUi+A55L5k/3ItmUnNcYrjdR7FLr5ByTFski6+FibXgOtrIOmcoBqy1Y2tafm/tOTF68wnTcnkvqmrKUR/c/RM1sy7wQKZr/MzJOtGMe6gcq7IENgKS2swcpVOdE1fEX46rUw/dzlh3sKE0Y5pPLmKtJID2bIyBWsa3lnUn3jt0Pg0cr5UNeG+uU2l4ojBzow695EsTorfE6IPudWj3AtxfC4fS9keQ3/5PL+3xJTneZ4+/zzGyV6k+L8xgUrmFXyDkuPwgLH1Tawz95OnmNcuWTqneyamZ+ZaX3LuphKTiuO0GwrXc/1N32Fz71JeY7ZQR3GvGsY4K2zNvQPksSRbibmLw7VrdM69d3F9vvI5cf2QQDmZtnGfc+OJc+sj6JvQNtff7YWuW9aZvDdep9yKrhvanl134OA83Lbdm8obxU++YV9/VkoCSf0vAfwgxvif/rz6easFyzO7w7/08HfRR4M+Guw85aaqNQHs9JHSYkyRX/toRhbCwYI4nOtTdH6IGpXu83Gh/OiaViH/b8OQy5HqhlGaERm/yWA8ZA0buCY3sfxJ7feI1nS9F/dOEVP3Ke0IWzM5Zch07NO+p2WaRoTniq240noqLUlTq+RgDY6j81N025wH0xcotcvzw3PdBYMyWYGnFl05PtnfaDyp79F6J6UEg/1ISymnHeF7yMqpYVQYWzYn1t4gvvCSRrbiWrE/eI4Z6EkJuvk892EE7YfAnACMkItjVLkNGe/L57gviWI8tQhOnxH+1kmrrlFx9hs4PZbrjMl5WaSll0GXps8EgFklA1tm5XgPWTine2+Kssx9TF3aqS4yCjIwuNmz1Xdq3WUF0tRSNUVvZTAttvBKGiSCshHPmVRsyfmbzqlM1QJM1iYMKXAy3xjHaMqY1OfxHEKAlkWCXHEZu8/LuoMrPxfvh/CCOSunLENd5PYBZEup7Mt7ymnKFk6dLJjaDPOaLadhUHZNkY5lvtG8tpothQpSSabm1gfANUXZ5HlRYq6yMi4gp4zJc85tRCSL7Lgvds3PlngR1pCVgYKL5+sUdyusnxOONQM/CaVbRnpmpciU25XSAlv8pnWnCiyeDL7GggYLT4zqK0Fhghria9+kcHtcpq66+aGR0g+GsebfgnY+njsn256U7LqdrkvgKOk+LZU67Jp9TShgoXAiWF5D75Ulrd0o5lTNt59Tl0wVcvH6Vpg0n+dyEFKH/SKFy2tlKgSK8SlZB7g+dj4dDwjtk3oH2xLldgLezCRM2pCPjPx/sByi64DAeq3NVwlsr7h+axpHtMzzNW9abi3o/wmVXxA9vwXg3wDwe0qp76Vz/2GM8X/+Mjt5qwXLbSjxvavH2f1VClxTt1UAyR02wEUDq3zO3ziuo68xvIeYXWDsGjp1Bw2TB2NOmJ1j2OeYwSnjdEiYmyvT2FDg5u/XlF4pLE/L1IVS/j4kwMq2p2ObYw6He8aumJJxD5MBTNuYtntTX3Pnp8LHTelNDn2D5/qd6+sQU35oXm4qIwuwaE+eO0THofk5ZK2e63toa/78TW3fVOdQGTHk048l81m3faHfcr4P0XPTXj7Ylrwfr/6eX2PKb9PHHMOjbpiXA8zH65ab9tqrbz50XnArB/bgnEdEF+fH3E/bmb4oJS2veoHedt7m5uI11xTArAXvEFN9iIaY+r1pK/CPqPR1IWRSL4qDPG2yL1FGrpM3jf3QXM/QMVs/qozu+4XKTXtPCFujqocY/Lm1mas7qUfLpWYY9+n41GGGXdI+I9CM2p77wCk1oV2Nrx8oIypuIRSpV/Vxi/fTtU/LjND2OjS9sv3XuHfS0o33qJn/ry2ovMb6vLLt28z9bdp5zTa/lHv+DJUY49/Dly2xz5S3WrBsvcWPN3cyeA8Xn+LIpGVn0PRzrF7MMWHSDZVBgKQ23yUwmSjaAQYLIcf7AchWpWmMGBfJiIegcsqKqRspu69i0mecjEcKCn7irsta+SHOamDs8lz5sYvtNC5KJpGXcVRzx9PCuQsziqh0GU3npaurFNqUGEd2s4Ww8ght//S7K+dgGs+Ur0kaI3JaDp+g/68JQvJjrsV5YMS05ZyDEIJMGF8PfqzJlyXHI0m3TTGwDMgzJ73yLRMX2KxV5iTx7FYrr09KdollRpPHJGN01ECrpHt2bEyDGvbFtX4l0yItFHMxS1MGcspQRwxupHKOZBwTXnGerRzSSjIXv8QWFgbUkVahOYZxglw6YvrluKZuxXHSBlIb0r1N9guIecPAHUY1zs8o9wKPW16LYg3eRODhcoAHl9fGFh/MWjeuxaLNMBM57oufxzQFzF+zqy7XBTCy5GTAHSPAeyJSyoyJADRlRqf0KWDW6iPHOp2L6QtNXpu7T54n4sZCjxjPiIZR2wOIDFu25DUAYzdQuRXUdRfa0XzIvc1bS7pzi/HooOZlHTnm6W/ueHpdxiXmBgHMoJjfuihpoTskmBwQDKb0zc3ToTIzDLnMwgt3fM/0Azk9N6o//+4SnwI6nvl94zpF3NyvHJfcw4euHxrXtM9DbQPze3X6TN1EnyyH7p37Tn8JZc4COD2+Vm477695323H9oUsdF/y/P1SlD+NY0rlrRYslQIK7aFtRJeAeHpvUFs3AMKkunMucAYQ7nIpJ+FksbUClBpyO1K/ceTyxaAdUxfGAOT0CtnKxf2rCK2RBSiwkCTpyy5YAdIKc8i6J93FYpIEKI9izELiyGUrDoItj0upsRvX8JuvI9elPmO+JteFx69UBPQwf9KtLCpKHaITeq1O7etEJMXwxOF/JjzlPsTAw6k0p6O4ISF4A+ndz3PHdRi5lefAzADwTL5v+ducv+wYMXHTlCdqRCRGoBizhedz7uMYx/dFzCguZHycoI3jO69t8pm+lGAAgUkaF8mwqjiKReU5mc7ByE0uu27FkRA+azHjtsU9maM6VJh2Xhs+N60jyxwjz+fkfE3jJ+X5MKmTn/cpBzBHL9eT/avELc6MQwqlk/01KpJe8e6hDXyApjC5Boj41Bvm/VXlplvVzO853v/aw4nZNcvWsKkCYXpdNi0EHRYc1ERJIK1Q16xqUyYaM7+n9eX1OSZ07pkYxXOLvcbHcWat+B7hUnit8BZMSozR2AXjrAJGqKuSePkqzMWL33K8N5RrbfCzFCY0yfpTSzMTdE3JhZv34qupm/Q7XyU/ZrLLm9bW4/Ylr9X49LVXz03337YfUaSAOXq1XLvv1RN8fY1x8Js3W39Ko1zXG8Y3R//rMPm3ogOJlsn6qNuuz6G2ZVuHjqeb7hZz8bp75YvS/mXW/0IC6y+wvK1036a81YLlnWKLv/HwuynG0sJDoY8GtXI57hKgeMlCeTShQKE82mhzjFwfDIok+DShgEG4Focp4yr5NyHMpoTo6Z42WIphQ4AHxRQWyuffAEbxghQ7Z3L8ZcExkOKVwbF5fdQwKl6L12S0W4oB9LmPEUpsamMu3lPGgXIfAEYxoFzCjHvxNFaNi4whtCqM3JS1iqNYUlmkG7OM0+S+pqiyHB/I53lOebxW+ZHrqhwDrwNA8ZwAUJs+0W/G9QStc/GNUxq0IrCoGFVGsg1R5bjH3hsUCaVVFs7BWiRFhwSYkvMq13O6tiYpImQsqos69ynXS67fnKvvFEH2kOuzRO3tvbkWw8pxjjK+dS6ukemYrrlcb0m/rDPlFzkm04h7YhqT9Ejgtnitpm7afE7e5ye0KBVzKqPSDms3rUftUF+l9Te62UsvCzMB1WLU3RgpXy3X4cJ9WBNmEXpd2od8LF3sQ1KWSRqm45m65Udcjx0fMdSTdYpi/ZUalEbTWE727DBCIeVT7to5D4qp50TfG1g77Hke6zRmNUYF73VWtDmnURQefW9SbtlEX1CYunjL+VGKvR6k0D6gyMp7mHaf9k1uZ7I3o3hHqJn9jzTHeS4wgJtFcU2iC4+AyaLKXhVTILKYBVbk+3KKJmkJB4a2xO+RlT/3NVZIDZ4fauhjMsfDXIg60nMj0TsLIpZTgLBwDfIueNPC7cqxs4ApzYaHPAiAsQJg7viWZRpjOYsoKwSMHGN5QAgfWfHlOA4JbFJ44f2kEl1CYIkzwx4x1nHSFo8nXct15TWxb+b6OVimguhEQJ5VAswdT8orBc6byi2EwBv7AEbP823LrWl+hXB6Y3lDJcZN/b22YP66tPyqfKHyVguWl26B//3Fb8CqABc1Nn0FrSJq00OriMuuTmk7DErjsXcFFraHCxp7V4yY7so4tN5mYah1Fn0gQWtdtmhckQUEZnDZBXdZ9PAJnMZHhdo67PoCWlGKh96bsUEgKhSprW1XQilKrcEuvADl16zLPjOlDLbB4CjMgHFqA+c1qsKluhqFCdi2JawOKKxH01tUliyrvTOZ8VuU/ahPa0JiIClthKwLIIOJSKZdCkkji2tiMIuEuMvvThY+WMBiRtKK8z5o9M6gsB69MzCG5qvrLUJQqKs+Cz1T92LpVsxMJbv4KkWpFNhV2VoPBWC7I+CnxaJDCAou9ckuxTK3Z9+nlDWJ4ZQpJbwfQElcT8jDxvoRE2msH+X2lPGOrjeIQcEWiRlWEcZE9D3Nk3capiC4wewynIBBQlRAVLCFQ/CakIDz+DV8a6BtgLEpVQUzaWpgWEdWaiC/wfmbF5wm5NyAnODe2EDMdAR0EeBbA1OGIR2BYFijV9A2IjiVXXMZcARK5O9MtGkbEXoNXQSEPgGGKKKZXYoBpLyUxEAyMmboDFmgORF96h9eQZVp7hODmueCGUYFyvcJ0P0RBFgif5uhXaUjYmOIyVm6/AGLTkMVYTgOCuhTbFopOM80lyOmutNQtUfcW6DyY6tuawaVe8p3iTIMX9w+CSNFoHyZKo5c7dTeINZ+cHuV+fd6TddUumYj0GkCPHF6DHyiMOTYNHHoi12DveAAWD7yamDup0i60t02CQfKK8RiGJtK6xJtoFyTnBc0qrH7KQC90+jLmPMHKkf7JFYhW+Q496ZuNbpFgHIKqlfo6wDdaLgyZoZbd5rICDSX0Yp5TYKF6mktGU1WOyCUMTPbvBbKKTgdoTlnKG8Fg4xOC0XXM6/I6LIa0N0wt0EDulcIBSWw9wbg3IGB+6oC7I7oj+KZQ1TQyQIZTerPROi0rsojI4aGgq5rB4QC2XqJAERLv6Oi+grpf8oZSfk403EEQinmUSfHlgiE1A5b5IJFfiZzfQPoHmme0zWPPJehIMTUUEaYViHYIZeoaRXc8s05TN1TuwCNRSWhSkXAlxHapfH0g+sx6zNpzWielVOZ/qgjTIr7zPk55dzz2ouNEjVgmrQOaU5Un+4XQkrO+Qmas2CRaeR5435HiK0JaCej2/JaKsHYy7UD7V2A2ud62qc9rVO/rLMVj22ur4c9JPvhdednLFvL0zneL3zfSBiFaFdY2WU/47ynom1Rcv18Yr5Oplv2MzPmuTamArI8Nwwm3cehAvKY410PKReyEgCzRcVIMbshjqXs6TgndM4KedNztxCcua1bC45vq6B4wxr8aShvtWDZeoOPrk6x7y2cN6gKBx8Uem/gHB2zhaRzBqX1aHsaclU49N7kNBykmR6QAq31lCokKjx5eYSidCnvG7KGvChIUHt+vqbcXirCmICuLVBWPWJU6DubmXsASVtOAkIIGlXdAQCcS6lQUjtaRzy/WGVtNQsjUwsHC07GBGy2Nb13k6CzWHTYtwWutjXK0mG/q8iykIRHANhc1Zmx1zoJCEBGC9QmZsEFSDGZWrj7Bg1tPBpfZrff/O7yCtpEbHpNDL7QdmsbsE0CA2uexwJEhC08mk0FU3q0npAITemhTcDVxQLKjLXrHFcorQjttiQhU6S/0MUg9Oy7GggK5XFL8/FySeMrAvrGDgKG+BqYKikkGp2EIZ0FAl0EuFRfF7Sf/K4EkgCjbYBrKigbEHfF+OWd7rGFR9/YLEj0XkEVycpXBPS7khhskwSxYEmgSm20uxrQEbr06DclraeNKGoH7zT6bUGJtRm9NKjrcYRMkzwGoIoA35BQw3PqtgUJTzrC7yx05eH3BjAREantZOFQNsDvLGADYq8HwYTnga1DaU/6VgFFgN9aqDIgtGagxwyCKGz6EPY6p2FQtUd0GrEphjZNhCpCEgLpeYSO9KdANKX9hIIEjriztP4mDr81EHsSJGEDYmugFo5QM7eFeEAD4tYOH3oVgTIMgigXFhikBbtM9y480NK7J1uOap+EaZCgVkSgMcM8FmkS9mYQ4Do1MKfHPbCzA3NiQ1ojAEc9sLd5jtXeIJYhCZdxEIwBqJiYZ52EzyIOwmivRsKjYsa5CEM7bEli4ZQFXQ0ScE0kIbdLsc9OEy0x/S4C4BRU0JlhR8AQK3mnR2x1Tj0Ta9qnqjEkFMahbjh2UDs6H2sPtTcIRw6qNdk9NCySJZfn0akhHlFRW2FJnDgJkgF+EaE6jVAlZUYSrEMSbv2aOXshTDuVnwW/GhQTSrzv/PGQz0J5BXfkqR9NQktk4ScohEWAajX6e71QJCAJW2EkxLslCde+SMoFtvoFBdUpxGWg/dVpWq/0bICfZ4D615HW8CiNu6c+oyXuWjW0N6Mioc8Zeq5UpxCTIJgF9TTmmCyNqldwR0k5EIb0MLFI69ApxCJCNxr9iU/zCSAAXRmgd2OPlNcp7iRAN0mAXWKw3GpAtwpulYSJItCcRmTrZTzlvasQk5JKBQXlgf4s5P04KBJS21FlATDPcQDa+wG6S99snxQYTo2FsyTQAYNiIFiqr5JCICoSOjlOFirmVC3RJCEpqOGZCUPbNBY6z8oIUqSkMRsSSJUnxYdm4CQ1jCUrfrwaYnj98NqMoi4CCatcbyRMpuvD+3Zoi1Fp87PL93D99C6UipRcWHBPyg8uUyGIBfPMLrDgKh5xWjt1swTFz3XEYJGemn0FT8L18rtInOdb8z6UcynGlyclAmM3gevkTYX3WwmCryNoToXjAwLur8ovZ3mrBcvTco9//fFv4/e376OPGh8unqEJBX66v4On7RrfPv4M5/0C98sNfry7i2+snuCHmwfQKuLX15/hZ80Z9r7A/XKDH149wIfrZ/hkf4LaOLy3OMej8hwv3Qp/7+nX8K2TJzjvFti5EsflHi4YfLB8CQD4vfN3ca/ewAWDR/UFvn/xCH/u5FP00eCn2zs4q3ajVB+VcXjSrHHRLfCX7vwEHhof709xXDRY6A6VdrhXbPC9q/excyQY3K22uOprnHeLUaqNB/UVCu3xoLzE968eofMWDxaXeNqs8Vfu/hF+tH8Hn+xP8BdOPsIfbh7isq/x/vIcp3aHXSjx/fNHOKt2aHyBB/UVfrY9RYwKC9tj7wq8u7rAynTYe7LAfrw7wXHZ4KTYowsWV32F95fn+GR/grNyh6u+zi65L9oV7tcb/MOLe7i32GBdtNi5ElddjfuLDT7dHePx+iU+3R1jaTs826/xcHWJjzcnWBcdvn36GX7v5bv4+vFTfLo/wbYv8fXjp3hYXeL/fvZrOKn22PQVSu3RBYOl7bC0HQoVUBmHPhj8eHMHvTc4rprsmvru6gLP2xVK7fDp9hhNb/E3vvIPAAD/y6e/gXXZ4r3lBZ42a9S2x2fb42xlNTrg68fPoFXAz7ZnsDrgsq2xLDr4oPHuiu6zKuAbR0+w9RV+uj3D0na47Gq8t7zAT7dneLC4wuf7IxgV0LgClXEwOuCbx09wZnf4BxfvAQC6YHHeLPCN06d42S7xtfVT/P75u6isw2m5w3m3xLYvsbA9atOj1B5/fHkHx2WLx+uX+NHFfQDAnXqLf/rOP8T/t38HPzh/gLNqh3VBwjSviVIR66LFpq/QeovKOCxth50r0QcDq0Ie37pocWRbtMHgZ5szPFpeYmF6/HR7hm+ffIYfXDzEWbXL7W+6CrXt8Wh5iY+2p3iwuMKzZgWrA57vl6itQ+cNzuo9AOC4aOCixmfbYzw+eokfX97Bt06f4OPdCRpXoE11N10FHxXuL7aobY/zdoFdT8/Mn7/zCT7ZH+PT7TG0ilgXHU6qPe5VW/zw4h34oNF6g+OyxapooVXE80STCxqPlpcIUPjo6hRWB9ypt3iyO4LVAcdVg8u2Rh807i52eLFf4p96549RaYf/45NvwSZ3xjuLHT7bHOU0L7V1+PDoOSrt8P3zh9kddFV06L3BZVdlD4hvnX6O7z75AL/54Cf4/ZePsOuL7KL6F+9+jPN+gcZbbPoK7y0v8AcvH6AyHq03+ODoHFpF/KPzu3hntcltV+n+v/Lgj/B/fv5NbLsS91cb3Kl26d3i8BvHn+H/evohOm9wd7HDx5fH+NrZc/zs6hSPVpd4sjvK3gk+aJzVe1TW4eluhQ+OzvGyXeJuvcVHV6d4Z3mFnSvho8ZVW0GpiEerS5yUDf748i7WZYvW2+whclI12HQVVkWHl80C764vcVbt8MeXd2F0wK4v8JXjlwhR4aeXZ/jqyQt8sjlBl57NddGhCwb7voDVAf/8oz/EDzYP8dn2GIXxeLx6ico4/L/P3sNJ1QAArroKRkX8+Tuf4O8/eYx7yy2+fvQUf//JV/CX3/kJvvf8fWI+mD0AACAASURBVPhICsuvnz5DgMJJsYdRER/tTnHeLLAsOpTG47Kt8f76HC5qPG9WKLTHe8sL/OjiPo6rBiEqbPsS+77Aw9UVztsFfuPsM/TB5PCAz5sjvGwWKA255b6/PkfjLVwwuOorFNqjcQW+dvwMANAGi/NugW8dfY4fXj1AbXo82R3huGpgVcBVX+Hx+iW+//wh/rl3f4hzt8Qnu5P8bNe2x0mxx/N2hWf7Nb52/AwfbU/xteNn0Ii4X17heb/C3pf44fk7eH99jn/s6BP8zvljnJV73Cm3OLF7/NH2HdwvN9iHMr/znuyO8GvHzwEAH21PsbIdvrp+DoOA7774APcXG1gVcN4t8GBxhYXp8aPL+3iwuMJ5t8Bx0eCjzSk4L/VZvYdWEU+2a3zz7AkuuwWu+gqNs7i72OFx+ib/eHsHj1cv8QcvH+Jbp5/j490pTss9dq7AP3H6M/xvn377jfmOrx4/x4/O7yNGhTuLHZaWlMNHRYs/Or+Pu4sdLtsaj49e4LxbovMGu75EiArfOn2CrS/xdL/Go+Uldq7AzpXY9SW+cfI0r6VWEVZ5rIs2h2m8bJfQiNk1f9eX+Ev3foIfXj2AVQEXXY2vHr3A83aFbV+i0KQcL7XDy3aZ99Knu2PcrbfY9BX2rsD9xQZaRXy2PcauL1AaCh85Ts8ifzs3XYWHq0s0vsA+fbP4u/3O8gouaDzZHSEC+MrRS2xchRAVTss9njUrnDcLvH90jp9eno3ehz5o3Fts8rduYXu4qLHri5xbfFn0MDqg0B5XXYWrpkJduOwVxvc0zqJzFkaH7FEVAezaElXhsO8KlMlzi1NxNV0BawL2XYFF2aPpLQrjc2gBe3kd1S22XSHCEYZwDfZiY+MGp4rSCqisw7alb5LWAVoBnTOjtGvA4IWmdcgeXt7rTC/3Z2UoTfL46r1BVfTonEVpHULyoONQA/Yea9sC1vpRqicuOcVSYOBHAqEkDy+qzx4zbFxxjsISbkqdlUMmwjg9kvSSkvmzOUxAyrYDTSrfw6EHc/m2b0Q5/2Upv+z0fYGi4i/97B8uJ7/+IP7Wd/5V7F0xQoHtvYELGrV1aL1BoUN2a+1EvkJ2PbXGo3MGRseci64wIecx5JcRxy2xGyXHPbW9zTFk/FKg+CnkB38uJi1EhbpwObedBJsxmtrl1ck5Lyc50tg9V+uAztl8jl1pnTc5ro9zJGodsvKHXxpsiZXIsiFQzJEEK+Lr/KKRsTv8UuL22BrMfbAlNUbk89JKzHnkvB+sxs5RjJT3GiGQlVjriLa1MEbkWktFi48BxU0pjGKcoiLXzdRH8DQn9aKDUhHNvsyAQjJnHTAozbJ7qUDtzf2bkK2oxlIbUb4M9fDCDt6MtY6pX63J9ZXXiC2/ETRv3tF9OR5PqAsV2EWUXvYypqosHVxvRjFqPK7IL3IVM1IuW38lKi5bibNVGMk91tKe8k7TuBMNNG5k4CBlYnanZatxRqlNdBLIE62XdJ3VNuXhYw2+CXQvMHJ3zZbpkqx6IX38OK5LabKOM22DpVvQktpHGp9SyFbnTOco3kuTJVtF+MaOx+MGSw5bpYGBBlr7NFciHkyXHqE10JVH6MzoQ6TLBLqV9peSFuCIDOAUnYKyae+LmDRd+cH6y94EzDQUyTKcLJboNVSZ2rdkhRn5arGVOVnW8xicFmgiGO7h/txAz7XYMxUzbbktbofj49gi6of9OnX1UgtPdI8s25FciZlu9uwtw+BizO6/FVmLR2NVcaA7WczyOaZZxaFP6SKMNA8Rg7WWXYt53MmtPPfBViuuI+eBzwc1uCrz9WzuSXPea8QqWTnZRZnbY8+EgAGFmOdHeC1k12N2v07PRXab5jbZldWrwcroExJvapct0gDGVl+X0HgTcq9E5ZXu1LFMFss4zHtuj9voyaotLZaxiFDtOLb9tYqN2YrKNFOnyJZaxDR/gdaM3SJjQedUUIOVLs3tYA1Mbad3Fh9koJ78zAAxWSjzMVt6o6irMLhFJ0sle0hklF+VzgsU3ZFraJpjQkVG3l+R15jPh4QszCEFop1rKMtMX66vButh6kNu4TzHEdctjxgqXnNZhXitCGumdOcFxFgP3S/caIcFwkxFNTqmftW1aofKlK5rk8Dzo8Q5+bxPz4tz8hV0kA7ZlmxjltgbrnHXt6jzJ2mR/MP/+G/9PzHGf/Ln39Phsrr/Qfz1v/63vvR2v/tf/OLHBrzlFstH5Tn+/Q/+V5yHJXzUKJRDHy22oUQfLWrdZzCJK19jqVvsQgUPjZVu0QTSBGoVsQslatVjFypoFVAoj1r16KPBVahRK5fBgQwiPBSWukOIGle+RpF8TQwimlBgacga1KY+CMRHZWCdPlgEKFS6z/cYFeCjRq37EfgQjW0AEJKAN4Xy0IgotMPG1zQeEPjQkWnQR0P06A5ttOiDQaUdTHpDXvk6t13rHjtf5jmh8ZCWsGfU3QRcxPeEqFBphzbYDFTE4EWsjdt7Ak2SQDx8fmH6fL0NBaz2Od/owvTYugpVmo8QNSrdw+qAK1dn4BqZm1SCKxkEbHw1Aj0CMMph2npah3slaW1fdGRlkGBGPHZ2Q14kgB8Gb2LgJBcMrPYZIKjSjuJ1vc1CIMfyEtiTGQGeAECpHYyK2PvBnbIPBgvTZ+CeNjCQFMUD92m++NzWl7Aq5HVh+o5ti60vKZZY+yFdThiAqjIIEeaBlkrtMmAU99kFmy3yLmqU2qHxxQhkiNvneGirAjoBPDXMY1KUpP44PtoFjdo4dInWYa7pmSqNyxpgPre0HVzU6LxNe85nYKXGFSN6eW/y8xmizpZ3BsEqtcu/eQ8NgFEaxyVZwDZ9ldduCkTE7WgVsyViNO9xALVZ2g6bvsLSdmh8MXKDX1p697gE9sSx5AApMCrrspKNNfMyJnlVtNj2FQJUBi/jYnVA62y+1nqLQvsMeMYAU7m+WHt+V0zr8/qGqFBqD6NDfg7kNV4H+Z9B1gCyElSGnisZp82FlSWs3V8WHfpg8jFbfFs39M3WhsJ4tM5Sn2kOKuvQukHBVyaFpFQW9ikdFZBkOKYhUliA1QGtN6P4eAkKVUzAerxYK6T2+D4uMc0V98nt9BzrLWhkpSspPN1ofqbATEwXK1zlMUCWFmsCCuOzZYjb4T5JYTuk8bImXFOesuKU75fps1gZC5BybLS+6f8U9IsVpmayplLhnMGuUmjMmxZWpHJYy5DyK44UtdJqwjH4HK8vY9mlspWPmVZWYvMczaG6y7RkEqCJaaI2qY1CKG5Z0Qq2/IQxOBu3yUBQcv6nehyjhvReEYPSERjCAEOKRQ8zgiWn6RqNb4rmmy8MAFDXfDBZaTNp/6DgwvWT8D0S2jC5jxVHb1KmQqE8rybXRf2cNin9JpJjPi9BmK7lfz0kNHJ5FS1DwweHdWuhcXp8W8FV9DNd6luVPwFB9VflenmrBcsn/TH+66e/hU+2J3BRZ5fOy7bGtivwYL3Bri+xLDpctjXuLHZ4sl1DqYi7ix22fUkuBNbh5W6Bk0WDTVvBGp/d43auxCeXxzhdNGichUuuCQBwUjUIUHhytcaiJGFjUfR4vl3i7mqHGBUumwqLss/PKzMA+77Avivw3skFAhQumhq1ddkCeVLu8cnmBH36SCwKchVs+zEztqy6DDD0ZLMmprHqsOsKPD45x4tmiaumwjvrDV7sl+iSS0dlKcb0xXaJMqVnWZY9tm0JTqHSe4Nl1aG2LjMs27ZEZT3K5MLS9hZHdYtNcjVxfkA43HcF6sJhs69QFo7iWh2NYVV32DYl1osWu7YkRq6zWFQ9tvsSZelxZ7XDs6sVjpcNtm0J5wxOVnusyg6fXx6htC4zN8xo2OTCwhbpy12d4mFdZgZWdYfWEaO33VXwXuPDR+RW9pOnZyhLj2XVYd+RO92uKUcM2PGKBIhdcm/pe4OiIICeRdWh7QsoFXG23JMLUVMlRsygLnvsmgrLusV2X2UrMVt0T1d71NbhydWa1jiBBR2tGnTO4mTR4MVmCa1jHj/dHyh2FsBuV8EWHutFi4tNDa0jisLjvZMLnDcLvLxaoqr6zJi6oNEnoCGmhy3RzKgwjYuqQ9MVKKxP662x3VdY1h20DtjuK5wd7aiPkoQb7zWc07A2pPGXqEqHrrfQOqBti8y8FCUxvoX1iACafYnVssVmW2O9arBvi5SblVA7XQJSquo+M3YcJ3x6vMOuLdE0RbY0F4VHXThcbGogeQAY67N7UNsOsa2LukcEsN+RC2dZObQNra1NfYeoUJYOXWdx/+yKXMqenWTLeVE6tHuKpVVpjY9WDYyOOL9a5HcZW9+9M9CGrEpH6z3Oz1e4e2eD88slvNPZCnx2skXryLW77w0WVY+rzSLvU7bAb69qVIueAKn6waXp0d0LfPr8BN5plHWfx2BMwNlqj89fHCNGhbLq0exKLNct9rsS9aJD2xaDFT4oFJWD1hFda7FYduhai7JyaPYlyqqn/ZT2FADUiw7WBOx2VQa2CkEjBpViyWkOfG9Q1g5F4bDbVRm8arEi18P9tkS97NA2ZQZyMtYjBg2frOIP717g5XaR99hi0cEaj/PzFUxBgEiuN1CK5vT5izWKyuF41eDFyxXunG3x8mJFYFVRYXXU5Pej1gGbXU1rpilu1nuNuu4RI9C1BZQOWC06XG0WKEoH72msodcoFz2co2dbKhWatoBLwFOICotlm99drjfk9eANluk9RM+XwfGqweW2zmtRlC6j3S6XLTYXC7xz/xKdM9juqyyUaB1hrUffW/SdxWLZotmXWC5bGB2wLEkQc0Fju6lRLzqcrva43Naoqh6V9aiswya97ztnsG8KGEN0LFctYlRomgLGBKwWLXwAttsaRenoue0NysqhMB7bXYWq6vN7tdmVZB0LCrZ0UAro9wWWxw26ziB4g+AUbOWwWtDeaJuCMAa2JdZHDdqmyDgJJycbXFycvj7Dkcpi1WJ/VSNGoKhdFsyKwqPZlrAlzWtV9wTGFhVCAghbHrUIQaPvbH42vNcITmO5pnWm/UhWfWPIkwcA+s5mzwuAPClWixZX7QJKB/jewC49YlAZC4HeLXQNAKzt4XuLonLwziIGwJYJ1K+3BDaWBFxTEOaCLehdHDqDYkE0x4SMq3WA7yxM5YAA+C69j1cdjR0gzIDO0v3LDv2+TFJphLbk4WOK5L7ZGfK+CISpwN4EqgjZsyR0egATS9KttilvuFM5nj8XBbKu20gW/VEcfxws7ww4FhK3NonFpLhwsRGk2S+Br42QktmabSLRy3U1xgIqS+j5OGaBNwo64iBtEo6AjkOO7ORpwOezwDzRAKheAQZZWJWCZBZWw8RjQMas5qEn4dbP0D8VHPncVKidXp8K8EKKZCyEEYL0a0uZv2TlLSf/pvJWC5ZWBbxTXWHvC7ig8e7iIqX8OAEA3Ks32JgKZyXFbT2or9AmQeTR4hLP9RKNL3Ba7tF7g7v1ljTX2uNevcW9aoOrvsaL/RL3FhtsU+wZx9PdX2wAALu+wFFJH8+zeofOm3wtRIV12Y786K0KuDIVjA65nlEBdYpZsNpjnawU+2RZOSkb7F2BjSnJApesGqf1HlZ5nJV79MGg94bO6RqPVy+ylebxmmJP9q7AvcUGS9ujCwaNs1iVXb6PGdPSeHTe4M5ih9r02VIDAOuyw8L2JOgWFvcXGzzVaxyVLVpns7bzylQ4Kls4r3FUt1gWHVpvse3KHEt3Z7HLcRJXqsLpco8QFRZlj4erS+z7Au+sNnihl9j3Fu+sNjgtd7hqK6zKDo2zWVNfGZ9jFeV8O09CM1t07i23uGhrFAmcqe8tvrp+Aa0Cnm1WqAqHe8stzs0iWzjYxdkaj3tL2ifP9RI6CdAUW6FxZ7HLbb+7vsDOkdBcGo99X+C03uO5Ak4XNP7ChOyGbXTAo9Uljoo2a+n7oLFtS9xbkiLk3fUFumSpWRcdNn2JzllY41EZsgQ9ieRifX+1ye7bR3WLD4+e4yNzis4ZLMseqyIxYd5i16d0K9ahcTYj8nLsY+81dKJ7U5RYFj0Wtkfryf36zmqHyjh8DuDRipjXo4rbN1kYPav3eKYi7iz3ON8vYI3HRscMlLUsSUCsLTHFLwHcW2/ho8KDoyu8sEv0jvb5um6x7wpEAMc1PWO7vsgAXQ/XV3hhljgHKTrqwmFVdjiuGrgEhOWDRlX0WKR4nStdZZft0yWt0XOQRn5ZdbhKbuTrmhQiLmis6xYbU+G99QVK7XG+XWTrw6rqcJH612mcD9YblJrQjvlZIVd7drunsT9Yb7BrKjxcX6HtbUZHjlHh0dFlVow1zuKs3ucYYB807q232XX9eNnAB41dS7FEAMWJXexrdL3F0bLBUdVi2xGK9OOjl6TISWMLgdb3eQTOVntcCqtICBrrRQtrPK5MjTurHbZFgaOqw3MVcbJoCD07qLwuZ6s9VkWHz0Co1L2nVDgh6BzjVFqHfVviZLXHuuzwOZLlr7e4t94iRIXnEbh/tMUL4W6/KHs4TwovYwI+PHmGn+o7eGEWsDrgbtqnTVdgVXeIUWHfFdA64NHRJTb7CkfLBg/XV9i1BR4dXaJ1RJv3NK8xKizTs1OYgF1XoLL07O26AvfXW1JqFRWsCTir93lvh6jyWp6s9ti2Jd47uaDxJy+Bl8UiK9sigPvrLRpnR/FgnTN4eHSFAEods+8LvLu+wEfqlGI9iyo/S01X4N56i763eHz8EjtX4nmxzM96oQOWRZdj1x4cbfDCLvBgTV4cd6ptjhX8OCrcXe3w+OgFQlQ4LlsclQ2WtsPH9hRHRYsuGDwrVqitw/m+xoMj+sY9t0tUhcP95ZbSSnlSclodsO1KHNcNKkOKpZNFg21XYlV2eJ6sgt5rrOoOWkVc6AXuH23QOIumK+h9s2jzu/kze4Q7ix0+jgqPji/x3K6wrlo0zuKrxy9wsa9v5C1uKvfWW3yerHPrRZutwLV1+CwCq7pD01vcX29x1VZp79P79dHJJVpncdVUOFk06ING01u0fYEHx1cIUWHTVjlGcFn0aBPyu4zTA4DOWTw+OcfH6dvBe++qrdD2Nsf4WePzvXdXO7ywSxzVLfa9RZ/WwCjad5Seh9pn5ap8pk5Xe3TOJsu1R2k9NmksPqqsyL13RHs2RoVF0WPbFdg1FU7XO7wUAi+/z9Z1m95RZVZaMwI8QEK7SZ4ETVegawvYhIJvbciW794ZOKdHiO0SRJER4ZUa8oS73kCbANcZmMITcrsJWXDhkJmq7tG1NiuY2YNAYbDGmhQKw6CGHNLTtfSd0ioCKuYwmMihQ2lvKSBb+gEKa+EwkwiAQRZzOE1gAEEO8aHwIw6RkTGRIagMjEh0pE6TMJdpSaE3YKu7RF9PY9YMxJis36OQBy4T66u0QEsFCdM5jY+cphlSM4LloRjLX5VfbHmrYyzX33wY//G/82+iSaiw7LrjEhNgzeACIwORAeTYPpkbjR8+vp5ROVMuM85hxhue3WBIixzzOdKWJw1akA8l3cduJyG5snA9flBkO9I1hd1MZGHrJWvL5YuH007kl1B62chYTo6341izjLIpXFRyfKgacoflmDBxH7uzcMkvA74nafNioFQT0VH6hpyPLGka+WWlbMgaVIr7UzndROT4ralLDJPP88UxceKFIxFiGVlT1yluMsVYcQwdv1hHmj2O5/NZlThc1wMt/z9779JrybKkCX3m7hGx1t658zxuVXW3WiW1kEAqATMYITFC0BISDUj8D5rmZyAhMeoJ0GIAP4EhUwrUoqSGMV1SIaq67j03T+7ca8XD3Y2BmblbxF575+Mc+p4sXZcy91qx/GH+iAj/3Mw+s3aa2Y6dpNrfQ70glS3wnp3UTj9V9p1v2q0HudVrbI6AyDPUHu7C+wZZG5pvZx50bMOFQdi15/2sIu/8+XYnp9ZvR2t/cw593Vaf81nayWqfbSztmuWvThYvp58795JryU6fm/8fdnP17HPUujdfB7pvmeU9yuDrOIw15SDskYX2Zk/JOQJpH7zv0tFfrfluWZGhgjbRCHC0U3S0dUPK/MqBd75RdpK9kzOg+cK1k+5wyG9yAt0Pzq+D41gSGkuj1dXqsHAMVT8XPLv/bU3xwI2FVcZUZW2MtGjjyIMwbHJgIMrpPg98k5XUr8/jMjQWx8awaWPh8kBDEBD38eh+UbTz5WqhQYCdD1ov1/3a2jh59kdty8KegNGYcNumktBCh3DsZUxgm0PKaCyipNoKVmZYX6f5qdka8LI3n75Nn/fadrvefCxN/htxCtWfsP3GaOFSSPt3XIPGmllTD/HyJcnYRQF0rZbKFoyol7F7xjX5/Xp3n+n4m6UbdfQG0RhbGbb2sPfPA/q9ZPPs31cmB6C+n+jPiIO/YVsbfHj02tzpZ6DLZeNiax5u7HYWlt7ns9V7mHfb65gvsI2N75d/1/ghqC6Pa5t8/lumsG6sj2aitxRmR0Xarp1jff8Sku8vvTA2H5PpVcWgf26/UsfN3z6ljJPBr5efoqz8v/6r370f4v0f/jH/yT/4hz97vf/0v/1Hv/O+AV+5xvLb4Yr/6O/+MzyWU/c/VJOijePeL1F9pnKN2Dg0/zTzNzOfray+PKa53GrEUiMGfUqa311hwqQ+XtcyNBZS80ky/yzvj2Y+UYlq82PzbLGWLP+q/nFA903zLGKW15I3p8o1YFTfKh9YvjGP6YnbWtMzX0Xv55X8DgfiRxVu3NXe5+WWLMmArct7DHxv/jBbjY39zq55H6nK4ldo/lv2N2BPkmTyHsfJ/PdMnlJD00Bc87Abn2N4lz4/3XfO+8t4n0Hr2zHIvP21GJ7ev9FOXq2/gPhcDVqX9ylr9WP/rjCfzaDrtOr3Fh/U/K9cGT7Mi+/rS99tTM0fzNjprB0Arb9Wt/f5Km78LPlxtLr9OjB/qspA8C8aN7+mpTUtoBFyGdmWrbXKfe6srmovYqbGCmjlj2vLzz0ztRiwW977/Pm5NKItADtfMj9OlqKbcx8HloHuX+fyFz+O2pb5+fFBViPzskMne6YBPZ6spVqlb95Hs/e/E2Yd13cn8qKWV/qK9rs/oLLfPDuhJwlr86J+artTezfexzi2ZkYKdG2PHahZ28Zs6InGjFys+7D12LV+bHzbXmZ/KLgjGTvsNo/jV/UQrdXrTurZzQF530w2/7X9HPm8VQ9frR6fjFQM3H0/A/p9tWN2DLw7F7J8pj3xmhI5eDTZAX8ouYtD6x5i1fwzbT21gwnqmhE7nNODgXbgS9w0O6YRaoeEdugXGNkf6nxmsoNh+bLXmhTbwTf5aA927JoHRsAzoCl1u98UYFubO9IldyCxOxiEq8fLcARJ1s7Rf/C4m/+U9JLsrv1mYsn7v9Ik7eu6BSB0zl+S9SboOIJHd92fae7y3urTrbZfatOK043fPwUYfQZ4Ok7ri7P2OcDP6v6CMp+T54tA4ueuy9+nf6npqwaWlzriT3/79/BmWJBrxLtVfJbeDItQrl8fMCohwzlt+HE94WFcUJmE1nrYMISCD+uE708XvFvOjezhug14WkYMqYhJ3fUOQyxt00XEeFrFvOT78wWXbUTQa9+fL/jtLLI8qPmNJ2PIJeI8bJjUl85oqrcS2gZyUxM187k0Ztqk5lGWrqsw4uYS8HBe2rVpyPjxwxnjmHEaMt4/nXB3Et+meUstbufbu1mpr9HMXoRAITQKbtPmMgOncWtxQi1u57IM6hOTkNRnCgBSFHbau2nDkjsb6ZjEFM372xmL7XUZcZ7ENPd6mXB3t+A6DxiGIvLMA0qOuH8zY1NzN/MxFHM12ZTZBv3kfO/MBGaZB/HFYsJp2pBiwZ//xR8AAO6/vaKU0MaHGRjHstvsLhoXMQ0FzGKKU3RD1+KWMmG7DqBYkcbSTGPKpuY2+hdMjYGWa0C+JqAQwp2Y+ZD6P63zIBvHOSLeZTFDUY2vyWVsrsNJ/EnLEsUHCOJHhscBPBWku4yyha5xpc5UamYvxt7a4k/qBqb5wJTQNJVhVNbSQgjnjPo0gO6yaJUZEsNUtc68RInBuETRoDIQx9o2mY0p1WKBTgV1TginjHpJLQYkRW6yAACvsimGtgVAYkAODNK4o40dNBNwFjMlihV1i3KNIcygptmz+HCnoppekt9ZPw+6K9kCMFTQ+0FOxr/b+sZzDVLeNiSFhJGyEPg+9xekMWpaHEgAdA3ghwx6P4DvivoEQcb0KTUNKUfW+IKlz5fGyORzjwHZNLgA0o8J+W0BUgUtsfsgMRCfAvI3wnBLW5CYjpcIniQWIo/cdwOErt1MwrZpmr96qhK7UrWOphmhJSBkoJxrjzfp2TyTxmFU9s6wAfXMjfUyXoLszU8V8RpQJxZtFOvcEjd2zPouoJwkLxVCWEQLkt+obNY2E9ITIb+V63GWOIn8GFDuuGlM+KJEPToPnICaWLV9og3DIvVW1UiGDagnZe8kGQuOQFi0r1ddbwpQeEDTxoEYcSbRzAIylgwpP6PNgXwn8EnXiLF/KmAJK1DvGem3ARzR4gyadjJkAg0MTkC8EjAx0lXXoQa15wCUs8gTVqCcRAtq8RDrIJ+ZgDqixVWMs9xbdZI5Cave4neMuFIbR9qknnISeS1uYJ0UrAbRNBIDZWKkJxK5VLawAXGVPpcRiBuQT0BchA01aD/SFVgf8MUpLkBR9+ig7RGL7PkkcnDs7QJoGr00y+c6aNkg3zkAUX8zTXYbewPmqbfFJNeHDzJeNoZxlro9gykVuWay10HHIvRxA8uceU2jtd3yap+snGm4W19o307TYmaVKenYTb0f9jwKm5ZPXevbYkmyXBNmWwZHF4dT6wiFtQx16wtIfgDg6MC81uNjfYrGmxAKo0YCVd4zwFK/D47J4+8dc2/t11o5mz+nbT0Crx2eJ+Bwtu8Aeq+HA1pszx2rLbk2yc0v97Zu4h5T5AAAIABJREFU1X1s4yUZdppQN1Y3waS7TgdLSSZ6du1Z2rXBN9t4dkjwS00vHZr8DUlfNbCcc8L//dtfYUil+QAAAmjGVPDh2v2lUipYloRBCRsMXBBJiITfns5YloSkISJKjihqj35ZRiHtUGBip5/bIv6Ej6dTc7gvOeDxMmFbRJZ3w7kRU/jQDSFKaIn1Mshp+SCb62Y6WQjrLIHVbcNtG2oA7clRs+bPAfNJnOLNjp6vEesw4ilJQHgDOgYCEBjzZexEHDngMkz9lJh0M+4cqa/DtAMVZvq3DqNs6ryppJowXsdJQIZunp+Ufn8eJvAasAwVyAHXQUIazOMo+deA93MC1oAtVXkprAFUCI9L78POxNQ29PogfBo1OL03zcyErCff6zDKBvoxgRh4qucWQNxw12bB4zWRbtRXnYutmX7KRnyzYNGL0OlvKYEqoUQxq9uG/nd3vMhAmIO8rBXU1MAoQTbjHCW4dFlD20RzYGRPMc/Aek2ygdsImwIMyoT4FFBXQl4DKBOCAlEzIRO1pZiOsb0Y7QWiIoYsAaxDRaN85yHKxrQCdQ6IS0DdSIJls238ZH7ipnk2Qk0RocrGvL0c1UTNXo48RMSFtF4Cx9jkC7mb2cXNtyWypquAkzqEJjtVGYu6hLbJD6UHH+cU26m2yVLnoOZ1+jujm5iSXo8R8aqghQa3MSSwlrcXfNDNdN3649fMJpsJIIC4EHIZkC6EssVu5gYBJaZmlSDlQN36RDXZFxJAY8DPtLdXAhDBMUjeilZfnAmg2OahzoSwypjZptlkYdJNn4GdDagxIhQxK7dNqW1CZB5kbdEWu3mkbczcxsA2vVR73zgoeIAccoQVqCt1S9gCgKgBpjhrsPhVwG+wMeIAyrao/ZgI6JUg7gHpQs3kEyTA1OSU0BUKICu1DWRogFXnPMs66KajHSQhEMLSnwFMBqrMDFCAWAMXZvqoYFGEcWNVHFDLJrfKUAlxtfWyD+1g65KT1pPlXpU1rH0hqSNkAQg2lwYwyJ4JuiZs3oLVo2MUGgGK1AUAvOm6MJCmbQpoNXNjamuNMgl4ib3/lOHaku8JQFiAkPrzKqwC8L40hdWt/a1vrG3DHrKsibii9y/0svbMCJu7N3Rds1833GUGANbx6YIAYWPIxkJ+D2sv0wCizgvQ7yfKUp5VFmLVHB4AaTP7bWuM2xqWPuthQeZm+gwCwsq9zwoAOZOUt5tdy4P6ONkhRXsm2Po0l0c13afS+2RrT8aZdwDG/u5Mq92z2OZG7ik9ICq8A19tuAtQG2DdA5hnAMvdV1S7xcIx/4vaTPc8OII6Ym5AjFvwR5GbA7UxlfL2PtV50+fDiyDOA8pPlLF9v5FeUno/B36voCwPdj+Sdu+PXzjA/CXL9lPTVw0szTxn3tLO3GndEtYtCeGXatbWVbq6bZ211MchXGah81/XTh1vWpxZNVTszMNQBRwCwtbGugkAyXe7EcoW22dPM14LCfuhspkZBbf4EBKQsGNeBMnfWxTcFBgYXEw/gvienOQtz4WAppFS3yIzMWtxz7SNHLo/mD0RnPlX03J5vyCLp2Y+ZmauZNoI1Wi0O0lBGGcFhNp3icPHYGWWazHqzJ+sSN3sfCRN7vbd2taN5c68x7rhzdk2eXvxSZjvdrHnLB3i9jU/K/+7yuJ9sOqpvxEMAPHg/h4fmFbm+BBlgAe5zmctb9ot+92z4FX5ndWfytrPb11YiQjUVPemR37uAHj/odaO1slRNtVtbQ/cfIvqud7uY2RU3TTVUfthbVg9On6++3Xi3V/fp5bH9dVSudtf4Kj1j25962bK/MRE66nlT97h5yCrzaNtavnQni0JmyPdJDFE6/NsXF1MQjvdzlpfdvU2Bef5sIEasTPzKhPv5qaVt/v+bW+v2tzZuI3cbtWqITn7HOzElmuhXzDtiUZ4gX9ktvpTL8yHtX7YD6J4e2eILLmT6aKe3eaF+9gBAAKwfbMfa1t3gMtrfdUxKVHHD8D2sF9X9a5v2naC6gfSNvy+68ZtLu3ovJRTfzza48wnG9Pjc6G8xj/DAE7us72Ppr0gx5h9xL1ee1ocN2gFwPbmlba9DC+lGxvFFzfrLhX7AXoPvGAStyt3t/9t/akbzjs02V+bg3yH/SIAgPtX8n+BTNsben3j7Reiv/YsfcI4Wj4+fL9Z/nZ9dr3X+0K+G2vjY3k+FXh8SlsfBSWfMldt7I9j9pnyPc91+GvyvDQXz6/RC/fNs/RTx2DX5hcs8J9ynz5r/+er6/fp9fRVA8sxFfydt++bzx3QfdO8P58l74N09JWyvMfrR18uf93X1X17+udb9ZuPmvd59OnV+5i7L99Lvl4v9e8Zy5btd6l/Ptb3Oe0f23iJqevW759S32t17gsddibH5I7QPtafW+lWky+901/zA3j1GfvJm4EX0g2B2PxuDCc125gbbd1q09d53Im+9P2l48pX0jH+26ssb7fa8Z9f6ttPHd9jdsMeXoZXC3/GS53wurwv9fHG4cSzay8u3GNbnzeHP2v6mHy3xuiVMj4WnP/+ar23rn8s3+fUuctz2NH64/cb6bj0b1XxoizAfuw+Y6P+6ub7Y/186T75hHUmRY4L+4W2PmdufoZ0U/rjOB9/u3lI8Rltfu483/oNr+R5rY2PvWs/R75XfqMb0y0/6PePrFMT71Wzz1fSy/1/vZyV/az78VPSJz7rXpLnS2T4eeT/dID92e1/SvqlActfmjw/Y/qqgeUUMv7emx+wlLQLZu+JcWYXnD4QYy1Jf1MWUA2Inh35D9DJQwBgLXFHPuN/zzU0Ahm7bsQyABqpzDFVfTJEUsrrA8mMDwp+BKQmhw+ybYQkHlxaXgtc7ft2DCTuiXQsMXqw79am/mYjYUQxx2e0gfRbpDb2+Uja4olajuQttwhjLK8nDPHJk7R4cpYYjCilA0ojdjHil9eIWo5g1OczenMvj++Pn09PrOLH1PrVsIOr6+b+4MahwvGgg4gbQctLhwY2D55U5UiGInk74DtSg3fCFiuPpqn39fnxPZK+vNa3o6z+s7VvbRpxjLXvg5mXGyQnL1GXH9s4ktfcGqtdHYeYXC8drhw3Z0Y6cnPzduParXE8frb79EiXf0y7/agHNy/JcmNOntXp1gQAcEUndjmsl1a1y9v6c4va/tD/Vt7G8JDoY5szwt7E/kZfP5ZePPhqAuD2rtPX7x+stw6M3LXdZxzyWn+IX3iIvNKn42GN1edlPMp6rPcLDpn28rl2XwKR6HmMIOal9Mkam1tNkfrzHoGhl4PxnIxHZfuoZvHW3Pj77yOyv6qV+xiQ/cg47+u+gfheBGHPnx2fBVbs+fCxfC9c++hBiP39CNB8Dk4/bR19MjD6XLDxGfm/GBz+FMD5CXX81PR7TeQvL33VwDJRwUOa8ZAE4C11ACIQSAhvnvKE+7Q0xte1Jnw3XrDW1AARAGVznZVBswOttQqg/NUk8fqODJlrjUDEDnTmGvB2nFvMRIsH6ZOB2RQq5iImuPdhaYysVs/9sO4YTI8MrXbdwOA5ba2sgSoQENOGax5wl7YOtvQRfT+su/wnM0XUTVrhvQbY2Gn3/XmdTdTGx7OHHsEwMyEOG0oNGEaRaSsRp5R3TLhblfiD95rnlPJOO318xoyxtN9Mrq12Zt9B671uAyTmVgYDe7lC3QHpokyvSeOXtbHWclGZa0sljSOW+3hWwhD6X5+YJeZmrgFTKk0rHkMHvMYS+hrYiYFRWUiiTEZmwpoTQqitvI0XAY0tEwAqSZ8bSyl1ht7KpPVTY/eM1BlyxUe5h/oBBODZ+7ra+Cijp41fO5A4MI+GIHljECIoY9A0FkoyVmUdnxhrA5RG6uS13pZvUOKlRnzEvTx2sgDRmDQdcDEA7UMV5RzANWAY+5gzS6BxP8cW8DyEZ7uUHbCqJSCmirxFpEGcy/xvAhL6Zs0IqQgdTEVHpuUBf8mx+Zhb6KNgfdOYaCAGK+mUD1kUfLvWT7uvdf52jLDtcKnPASCuBMYy2ubIgCN3hlSu1NwOCNzcBkLkRvpEBO032ndm8cOkWDv7qLkLaDgoQMEnCxlWGEojxgpDQd1iC3EE1vLkwGbYz9kuPJHtZiuUeKmDkV0+c0ew/D4UEKETOx2BjGc2DSw+7s1c2wEQ4k7OZEROwdXH6LtuD0DNL6yE7rtmc6MkS0KEQt33zQ6LrP4ADYFC7ZqFRPE+vxaY3QiaWmgcRiNmspA2Vj/t+guRxREkhSp+6T18Calv9t4k+nNTKKGb4ZceEgOMXTukhDR+zpovt/qPsvu9hTDxc9wOltT/0R8kkfgMN1kIPayNY4u1UCvSxt6Psn3WubKQLExw4V6cjA4/kvX5OWbchS1p5taBOoGOVcL7/Efw176zk9nGzD2G+jrC/qDL5AqMYP3yQBLuM+3/PgeRuJ1eAaQm+4tWQb7eW3V8BOge27o1fk0OvACuX5D7xbG6lT5Dxk9OX3CLfk0g82uS9XPTVw0sCwc85hO+TRcsSHifJwxUcZ8W3IUVP9Q7DCThQB5CweM24buxShDq9Q6ntGEMGZc84FfThqd8wn1acS0DPmwTnjYJGP6vPPwav10kfwALoATwuJ5QQfjD0wfMRfwwn/KIP4wf8LidEMBI4drAo9e2ndOGQAU/XO+QQsUpBWxVAr8XJmwl4hsXMPppGzGGgukApC7bKIGzNUg6EeOyjTinDX/5+IC7acWbccWP1xPeTCumlPFhHbHmiBgY358vDTQ/rSMepqX1MYWKD+uIrIHXAeA8bFhLxJoTYhBw/LSMuJtWrFmCMmdlzbUA3N+cZsw5tQDuUyy4bAPuxxWPy4j7Ueo8Dxsu24AHZd798XrCN+cZj/OEacgYY8HjPGHbEr5/eNJA6gVrju2vseraBvfNSVgmrL+BGE8ahJmZMEwrYqh49/4ODOAPv3vEmiNmZdbNNeB+WnchIz5cxdFs0gDNQ+rB7ud1QEqyQ7hcJsRYMU1bi6u6rAnTmNtf0xRbaJPrdQQXQj5vTYs2DRs+XE7CwHsdcL5fUSsh54CUKpIGfjaAdHcSVt1lHnC+k/4vy4D1w4gwFpzvV2xbbD7GFGoLip2zgKSUKnIOqEVishoAy5uw3pYSUHNAiIxhzNjWhFoI03nD8jRivFslZiwTQiyIkVEKIa8Jw5RbMGoDYqZpzOoDXXXTPYwZ2yJl1nlATBUUKmJkDeYtY72tSQBIqg2wbk/S32HKDWjWLYBzwHAnhzAhSlgJLgos1eeZiJGXKO7Ok8xT3QLiIKzMdQsIgxw4lBwQUkV5HMS3+bseAqNuAWnKYNVAcyHUawIqIdw3FpMGWMLQ47rWDwP47Yr8fkS93zoQI0hbgQWwBAEM4T63TUy9JoCBeJ9RZvHzNtAEAHg3In+7CrvuEoEtoChJFV0j8jfCSsJrVKbfhHoq4DmiTtXt6CDEOJVQRyEJQxIyrjoVQOPCIjDIDq3mCCqEfCodDBlw8QRgqQJLBG2k/q4Q1l8lpCqnApoj6lj7RlrZhK2O9C6h3FXUQXa3tARhyXwojQnYWGHjh4DylkBbQLwSygMhPkaU+9qAYrjEvumshDrVxkALFlAUr8Jay4PIEFZCPVcBOCR94MgIcwAPjPgUdpvqOnL3uWUhAWs+8Uk36lHYcQ241CQsruXEOhfciFCMgba8qRiMFVYZXFv8QGWIrYOwuDY2Vwh4qUnmp5yFQCxdCPmelTBHiInKiYVQi6AMsNJOWKRfZRR5rN58z42RVIifSFlhuZFEUdFyMCAly64OQLoIqKpJxiBsaKy/5cTS5ztGvBDqiM4KewHWg+/t56R4peb3HFdScC2y5nsZH446jsYKG2X9RGUArqOMi7GtwlhhCTfIewyIYwfGOAHDI1BOtvaU9TVKmQ66gJokT1w7Y25V4iMjTaoDGntqbxudpTV28qHXWGGZhHnXfJiNFbYOQpr0jBWWXPkjK6yuz0b4xejrN/c69mUUwMIBqmessA7wko1RJ14ykqiWCKAs5Dg3LQgMjBpJDimzrLZjY70Dx3h+zYCo9+dvwN8D9dDnh1hlzu6ggPbt2LWq5F87wHjsjztM+KmssLcMXPz1m+kWUH8NuOMFwPxLT1+bvJ+RiL/EofYXkv7gT/6A//4/+QctPqRpTQzArcogYbEjLQaj1/L5+InexNTHObSypjH0Grujv6RpuCwOnGl+buW3794M9Xh4FVVeu9eOJqXmN2p9sDoBNEBIxA1YmTxbCe27N5U9xrv05rU7jQ3vzUstBUIzMTVNm2nQfOox20IzH7XkzThNFq9Bs3E5jqnXWLIbI2D/rDqaltr4SZw604D1OW55nbaqx9M7mnViZxZ6rONorvjMp9e0BO5yCLWFatnHycMur5ltmlzHvAZKbrX7zITwRp7do8Ln9eaGquWicIMVhOlZ/4/pllx+fulw3b6zynTLz5hMI2pvn5eOdU1Ou3yYi1vmwP4zV2kzxK4ZBAMUsPt+1DbvO2ydkfYpcgsBsxsnp9F58e+hvl35ElpYltbWsW9urDyrNdGhLrcOez8NJVEDZb0A7TZKUviwQ/Bz4jZKbdPj++brOfb72L4n+TrudmzjaRslC1twq22vCmgPlhuyf0q6NWe38vh+tTYP+V5bA4Sm2WnVkjPrfKl+P8Z+XL3Wsx76foyvCLTfvRZp9xy5sQSOw7nLc2Oze/y9M37uN9MHo5vPSkdlsNXZtM23xsTeIao1O/alfX9pPQM3N9tH8qVb9z1ZG4ybt8yL6873jQ7XDulFjaXvk90/L/TvOOe7x6NbN8+Svw8+IqeX9VkdH7n2avs/Md1cUy+l4/2NG99v1A/cGNdjvZ9y7afU90r6bGD4heP+f/7X/8U/ZeZ/68tK/zzp/g/+mP/1//Af/uz1/u//wz/6nfcN+Mo1lkSMgSqeamrmnLkG5BqROWAMGZnjDlSuJYqmqpkrCiAzv8itStDyAAkov9XYtHeZww7UJaqoRFhyauaFMVQsOTXNogd8zPsg8ZVply85UzMDpx5IVaYGJAnd9xMQk0/7zcw878e1mY6OqZvzmaYRWgfQwVsunTXXA24zfzOgGcmD8j1QimpyZ/6MBi6lHcCDTgm8Tn18VAYLJC9mlWKKV5lagG8fHN1Apw82D3T/UT9+AHbjnNWsdUxZAbjcElHjhXowaoAgxfrsmdZl7QcUg2oBi5PVQLR8fw6uhqGbdhqQFLNbifk4DD1gu42Z9VnmoLcTQtfchsAYR/lN6q678reAnvlmeh9L01x6sF0riSZPwWxqGkgX2F3fQFH7HagfJng/Qm/qauOQNM6n/ZVC1M132fxIFdjYPWrxLCtJnlicz57rd6h4tjMCWpxNCyYfYu2fzXwSZnpMSKPkryU2INjNQ22hMGLoa/hZapvTHt/U4qB6UGdmr34/5f1Lm+mnu+YXdBpFi27XwtB3qHaQAa2nFkIcK7gCIaABaEtBwRXbQYbWyZVkDOv+EEUOHdDAcfuN5Te2XbCBIXIHF0xi0gnc9oHU7zZWlIpS7LsdL0HAejvs0XENLLFhgzzruATQVCX0UVsT+50wu3tRVBOkZrZ9LVHgbvbqN8EGYg34ejDub0Wfz2/Yg6vLvjfzRbfjszVVCDzVDgzdoVvL5x96dpjhgfPRlNFkJ+zDP7GrU+fb0E1jsK6GdqTvbPVU2nXL973FRiyE6uO6MnVzXRzG6fi3med+WarBmeS6apjqzoS3mS/bWgb2gPu4Ho6g6Pj91kb6OOfH+wEiRwO0DsEYkG1j6mXx8nnQcgSL2Ofphwb7unxb5A9nrIBfY4cDiDaW6N+JD8NxANLP0rEfx98OTXxS+Vfy7w5ECDcOQQ4I+sYh7GvppfPRTwKln/nbR8flU9Kn5vuUNl9r4yMA+xeT+Av7+JWkrxpYmmbRwORJfQzt2hhUe0dofnrFHVV64ppcAxD2GkRiQgA33za/mSqqmbD85bBJNO1lYULQPAZmgv5e9FrzZQOeaTNZ/9r9ciTFsTIGmoHnWtKifnkGZok74Y6BPm5yaJ2Qz9HVV/X3AAF4RowzBMZWqZWxgC0yP+rvVuE2coTKBkpL609R2fz9VpkQgN3GNBCwMQHanwDsfGah+VMsDSiaNYkBHBtLAyoGfI9kJ1BgIN/RNTn2++4vdt9DqA3se3DW/3atrYE4ItY9FB3qY1eP1c9NS9n7jXYthL2c1MbRwtI8f7K9RNZzlOMlkp2uEQ3wYO+o6brJ9urejN7nrrf5vL22SbrxHj5qvq1+0XbeVlnsNKo25ri9Wbn1Xmjza/fAzVaey9v2Va4sBbWhcmPdxsw2XyYf7aUkn59ULt9e4BZPYifLcU24MW7+m7vd9K7Z/fpu17jntWvxZZKfXreAflIw2zavDRDpXB1nx68L06br5pQUfHDpaw263kPovr4CBtG13cfxIZfPLhHa5p3s/iXbQWgBD7xsDqOCetXytyVPrk2Crt+9LE0bzgZozTxWyxDkNE+BpZf3OWDgPThRE+FWppoMLL/l0E2YbadPro+EHrMk6nqz34Hels7mLrzRUfPnxhyAgkhb3ABb7L7m26lyGthlV0fgRgb1RcnGtckBJys6QPZgxw6yzM/UgLB+BqPFEt77UXa/1WcPEy1Dfm0F+wG7A4m2imXzIf6UNp/+EKq1CzRAan61jN1ny7fDQfqi9Rpi1nEhbxEA7J9rZpLNGg+TuR/O0D5vO+Bw495eHdyfebcS3RhK0v/4eBhxGOsjeL6pefXvTPfaeZbY1o37uxOU25+bmNP9bVW66zdBi70vXhqbF15WH2Ww/4T0uSDqJxlS/pSyv08/S/qqgeVaI/7i6Vtct6ERsZh/Yi5BfOTUHHQrAUOsWLNqLFNBqdS0O5tq/3IVWJRiaYDrug4CzA6grmn9ctpdW3NEUm1JLqGZV3rtkmmO5qFrLD2wiKFi0QDqZt7oSUYsWd0hqK+Y5i0lYBk38dksAUn94kxbZg/YdY2I6rtl5UxGAyce1JkmyjQ2vtxRm2X+eSVHUOikKna9loAllebrV1W2qj6aUX0Hk+ap3MlZti3uzCNlL9Gf+o28owaYRosBiCZINECk/WFdBwCwzgkU0IhKADhtCQBCG5Mj4PRMpACwOobYZk6o2gszcWyaHZU7RAYFGTNrk4v481n9Jeta0U1lK29i6oYvREbNfYNdxoCSI2pWwpO2KXV9bBsEt+lrL3hGDr0fNvDerNJkrVvYbSCbBinWHku1uvG1TbaVsc1ODqAkWqOsPoCyydUx1E06RW5j3DRvQwGXIOOsdTZgYVoo3ex4DZ/1mTR+qi9vsVk9QCiqNaijmJ3XOfU3aYD48tm4BhZNC7HEbfVjbuNu908K4DWAJ5LYrn6zMoS+8aoqg7XDEJ84QOK0mr9eobax25jEH9J8GZ3mow7qK8loAKIMFciEEnlPGmMb5KD5dI6ahsxvJG2+zS9U/SubNtXqcuuBQ2h1t2tRx63oZ/Vv3Gn+9E9hSNlGGKOIYgvdFNOWaSFgDeL3luRzrZLX8nDybWHXNut64khuLhVouLEHC9Aw8hsLWu7XwI7V9BbZCYsfXyvGEP8vXZ9UOqkLVSWv2Wi3Znab6Lb5p+Yzyk0r18fJiHAQGZQdqc+hj2T3SQE4AcShtdnk2kgBINp9yEHkRHQyWoxnxp6EZuAGds3vtNWnvrq0aT6nTeWhit/qFyZOKiMACt20FSo7qY9fIyGyOWegDipvARCpgaHmBwnstVthn2cHMqpgRMq9HCebR+yATfPjM9KgRtpD3ReyUDcbhpZnNZFvwNJ9Jtkn2Ppq8w6A7ZACPY/ElVb/ReyXvLFDo/bH4V5j6Z6hjJ1/ocjRx+QmMPTrHT1/GzdS8H9LFaiDaMRGrZzPYs+AW0WDWweuD77cs+TqoEMZ0n6Q7w/3OfXP3GeArmo53+Yxz1Gm4+8ufTJg/NR8/z+mX5yG8Jcmz8+YvmpgeZ9W/Nvf/zkudcRaExIVVA5YauqmsEq0s3HAQBVXJdmZYkZhQq4RU8x4ykKOM5eEFCoSVQSqqBxwyQPGWJDNHFNBxxgKKghzHpD0aRmIMZcBY8jtc6LSQJAPb1KZcJdWrTfswqBYWStn5rzHuJemrfSkQlZmDKX5nnp2WgPEAHDNQ/MdNZNg355noiXinamwD5liYVXM5BjoWuLNkf8A+1Ah/q/lW0vEEGq7ZtpWYUuVcV3yPoyM/T1q0qzPftTsAMEYYksN+OY0KxmSjLmZE5P2w4+BMcluDVh2YGjlAGAwNlyn8fH9NN9Wz5ZrDLTWprHLDmpia+bKlr8cZAP0MCNwO1CRueW2hrcSZP9lmh/uoVfMH9b6FEi02t7M2b7LWuu/B5K2jczITFX3Wta9SbBp9W3cTOvszZlTLG0d1BraAY9nkyW37u1acky2VmfUNWIWCL2f3Mbb+npkM96Zc+rBgmeWNXPz5dwPmqKSA3XTYbRxMb/Zo19uG9vAzw6F2ho2U1juoN2HULEDp5xD87n15YchI9/F5ofrD4VCYORz7AcZejhmeU3L78fC+//6cre06p5Nd2cd0Ma1m+9GBSG9zW4u3Q6o3GGCaddY5ywNOnZa3syj7bALQDNvtoOsQNxInfxhWzO/9poW9Ytvez/N4w+UzBz6pobU+UG3OoGd9shYcq3/fhy7HOTmEMKO6zakIaCz/fI+JI6MO7qP8Aty2QFUsMOZai4S+pu7d9pBAe8PbYisP/27P3Qjl2/nr2t9N7BUsWPaZTvM8JtwM0G+4Stbh0PfPifttODo80rYmzxbPjd+BrxZP/tDlXaY9nzYd+NwU5aPJf+gNLntlexBHbsDDQVbPubrToPa6vbj3ss++6zt1lvaYgeSrK+3tW6a/ehbatW8NK2HtdQye5BsWu+mRbxRwWFyPmHke/lPWXI2N5+4PD0OvgVqn9X9sbZfaON2/k/r/ScBus8ZznoYAAAgAElEQVS5Hf8GADL/2PibmL5qYDmXhH9++RX+6vqArUR8f7pgrRHvlxOelhF/9OaDhNkYVvy4nPDd6YpfX+4RSNhQP6wT5pxwP654dznj4bTgaR0QA+N+XPHNOGMuCX/x7lt8e3fFklPbODMT3ioY+fWHe5zHDczi1/ibpzt8d3dFZWqMpgaszCz1ug5YtoS//c0jSg14WoWp1ADVFDP++ukNsm5q7qYV85awbINoHCH31/20IhDjbtjwm6e7xmJ6WUb88bfv8G4+43Ge8LcfHvHryx3WnPD2PGNQcPXDhzsMSfxNz+OGp3mUTbVuZu+mDWPKTSv7NI+Yhty0wWuOeDgteJwnnMetkQQBwHUdGqPpOGaMKYs2OUecxg2XecT9ecG8DoixYp4HnE8brvOAlCq+e3PBb97f4+39jOs6YF0Tvnu44Dxs+Ov3bzAOuYWgMM1pCrWBqsqEx8sJtUp4Cdsg3p2WNo7Xy4RaCA9/Z0Egxm/evUEaCu5PK67rgBgqrvMAC8tAgfH2fgYAPM0jADR21lqFkXVeB4TA+ObuijUnXBbp37YlnKcVl3nEeZL+ywa8b+7f3s04DxveXc4AZEO9rQkPb65YtoS3dzMelW12TAXLltrGP0bRmF0uE1IquDut+PB0AiDsqt99+yN+uN7h8emEcSwY1F8114B1lfk9ssG277rZO00b1i1hGDKGKEy81+uI03nFEAsu84hv3lzxdB0xTVnHJwpzaqw4TxueriPOpw2zjssyDwpcAtKQQSTAp9aAZR5wd7fgcpnw9s0V12UQrWslDGNuzLPTaUUIjHVNbZ6/e3vBsgUsOk8xFQxDwTRsuF6mNr5pKG3s1mUQP78acL5bwEyYr6McKIwZ6zIAxBiGgryJHOMkrLgP3z0ihor378/tBH4YM5br0DTKITAe3lwRA+PdZepAUMOClBybSebbN1c8PZ7w3XcfcLlMqOb/B+D+W3keMMv4nqYN16exacLv3ojs6zxgPG1S9xZFs8uEXz084S9/eIu6RqTThmnKCviBt+cr/uqHt2Ad4+06ID3M2JYB03lD3lIL+QEW1twQGHmNON2tWJcBcdqQt4jptCmwCyiqJT7drUipSL4kTLXm0xmHIqFQYkVeI8J5wzhmXJ5OAkZzwOluBQCsc8R5zNjWsREcxVQbgy8Fxh9994jfPp2xzDKH45gRifHjPCCNsjs1BuKH04p3P7xBOG24Py/47Q9v8PbNFT++v5PNeyWc3q5gFj/rQIyn64iaI4KGNMlbbOFm1kXm/XRe8fTh1Pyka4ngHJDOG3KNeHOvoa70wGWeB+ScGgnUScew6jgFBeXn89qfEVvEw/2Mx6cTYmSsc2rhZEqOON8t+PDujO9/9YRlS5iXobFJx1gxpIJlTdiWhLv7BfN1xBt9zp3GDcsmPAZPjydM5w3f3F/xw/v79lwfU8Hj9YRRw0PN1xExVaxzwvle5JyvI0IsuD+vIGL8+P4Owyj3u62VECouTyeM04ZtTRjGjPkic8fqbwwA25xwfliQs1hhcA6IY8H5bgFBGLnPdws+vD/j/mHGfB31eRHw/dsL/sVfv/3CXQdwul9xfZTnR5xKswCKsWL+MCGOpa3TbU1ycKdaxfObBaUEbEvCeMoohVCLWJGc7lcwy1wSsDsYATpjth001Ez45tsLHp9OotjVNtc1tWeFHWTlTSwQprsNy3XAMOVmvZImmYNtTuAtNPbmMBZhtR7k3Va3gOEkz4laxBUoBLlPjfk6L/I8Hu9Xvafl2Za3iLpEDHcbtqcBpiUntXCIYxGjiiUiKDN33ULXeg56oEMQa5M1yMECsWimVWNazTqhWb7IMwqb5rffCN1suoX9IbHuMGZqj5sqAWMVC5BbgMoOLrwVhLUTWdq3RNxNxW/5LNtnoGuD3TV/cGH1NAuFY31WRg9XWogewu7AwPK3cDP+bwufdEMGfzhxBEqHvjwjszr+7tMBJXuT7H0bh2vH779Pv5P01bPC/vv//X+CKWQsNeGSRRs5xoJEFR+2CYEq1po0rMiIU8yoIFzzgCGIJnErEWeN9WgAcCmpacXeTnML92Fau0QVV23vblixKAOthRJ52kYEYkwKprwPpcUiTKHicZmaFsxiJALAVgLuxx6OYFPtlvcjtOulBpRKOGn4C9OIXZYRYypIsbTwGUJQE5uG4KzmsjIOYi5sxD12zch0AAmtkUtomiRATIHH1M2OPcmNmSSvObY6jaDHQoRYiJIxFSw5YtI2li3hNG4ttEkMFWsW4HAat6ZR9Id8pkUyGYZU2pibmbEHo2OSOf3wdAIz4Xy3oNawA4umHTIQYCCsmyF3v8SupSHkLbbQGEftj9XvtZ0GLAzsAOgapE0IYUqOSEPemRSblgeqJUlDaZtQC8dRS0BeEsJQBcR4DZjOCdA1Pc3s0MkHYGfybAPftDosprwlhxa2wzQWprXhChd/EDp+nUzJ5s9eNr6+sgVQ5KbZsBiIgG4ogF1MxLJK/iZLlcMBLgFxKk020z4YC2s7nDaz8KRkMyX0z65tMwWuszwDwjl3whpXRsaXxASWCTSW9gJt2htnCspLBGmID/IhNQCwhfEA5KVeXH2AtEEs5Dk2NtFpN5YATAKGOJNsrkwDtBFw0h1DUQKeNby86TJTzsg9fEghQM1n26bE5DXTUjUF3f3mTUbNBLaim/OayS/wXB5G98fTsQrXCB64m19aiJGx9g2l3QZLAJ+K9HkL4KkizEHCq0Dy0UZNw0QMcOK9iRw5+Zx5oJhjorXn4zDSut+4iakp2tjQRl1TZ5uroGEtbB4Cutmn5usxH9FiN8aLxGDk2LU+ZOMW9Pom5rBB5SJbvkE0fZQJYYOGNFHNkjehBFpYA4sjSXaNAcqax0KemPxqKslpL5uVY+r5JTyJlGObm9rr5oQW5iJs+l3NJ+NCKOcv3/dQRgsjYiE3pFGAtU9MaOEfbF0wSXgSDqzhIvQ+J+lH2Gy+7CannUlrjaxmsT1fXEjGUUGBxLU87Mu5y2EhV2wsYGPKMkZ+LTdzU123HHR83Vq1cDZ+Xph0bjxwIM2XpR3pUJfRh/7YgRf0OohdXf3V1c1f0fvp++5NVNu4eNB13ERY3w51vKYl3dV7BEy3rh365uXYYaPDNWf0cNvc+ZXxs/Eh3/dDP3b5j+OCw+944bdW3yfcY7fG5oV0C0N+Sfqzf/y7Z05986s/5n/j7//nP3u9f/o//pe/874BX7nGcqsRPyx3uGwjNjX3ZCYsJWLZEs7j1thWl5xE0/V0DwA4Dxk/lhO2HDENGX/5+NCATgyiDTKzzH/+w/dNG+fvAwMlPzzdNUCaYsEPT3eYBtFg/ubDXTORAzrJTK4BpQTcn1ZUBj7kqZu1hoohVvz2cm6bc2MbPZqPDVp3igXvns5N25hzxJvzgmVLeJpHnMYNj9ep/W4sqb/+8U0zlUup4OlqrLBomj4iRlazvesyNO1aLgG1BgxDxmUWLZrFc2QmXGYBUU+XaRcL0QK/X68j0lCaRvBymRBTweUyyRgMBY8fzhgn0W7VIjEPUyp4ukwCNsz3SmXeE8gQLpdJ9qOhM3qmoWBbpdx8HcGV8ObtVfKrZiSmgnURXznTUtnmPw1dI2HzYDKkVLAp8LT4jNsWmw9sjBXrMiDEgqVpQmVRUWCkVDDGimUZ1CpHAFFMAt5Ea5YaWCs5dlNKBc7LdVDtTcE6i9aEIuPu7Yxti8hrbH6kxixassbni4yaBXSS+iRWNSUk4l1Q+0AdQIckGpy8RaQxI6+icbFA8LnKCboBxZBq8+HaVulPZWqB64MCgbJGhLGgrLFpArhQD8WR1Rw1iXMOq88sM5CmglpITtEh40uREYeMsiobspngRQNZ3XEnDMKEWlfZ2YakvqNaV80KEFNF3SKGN6KZ2Z6GtuGlVFE1jiQAIDDinayfYr6Y7EJ9mG9qDQh3GXWOEotyiTtfzfhma7E3uRDCqUg8SjuZnpQFeomd3db5vQ7fLtgugwDGoYLGqlq/inhfkTUOpoFKGqtoM8YqQNT7HSfd+WRlUTUwuul3O4QwX7lTERCzRNECqDYQDMD84QykjlXWxBq7j+tJn6drkM9b99/DYPXpPfVHs4yLzeuJpZ6rtg10gPntClyTXH+TgUsCf79KXgBgAt+XNv9M6idr5oKk8p2595cgfp1rACbdGZlv21nB7ZvcfR8B6XclAZcA+K50v9YGYghFZWl+hmPpBw45gM39oBLKQwXmgPK3ROOKts4EULbDgELAQwW2gPqN7PzFL1wXtK6X5odr/pZBYp7CtCY23oVkTq1fNkeAHG7YoUCTg7t2SQ9YyPyhFcgDAn7LW9npU6Hm0whjN96kXVoCsh0iaH+3sYIuOrhfkHhgOQyAtulAAK3UfCvNh5aYGonRNuphQ+4+ouZ/yKP2TceJibsZL9CZbG0zXwnrOYPW7i/Oox5YeNNfdxjDg8RTRYTIxNRYlmnTcnZgFrlrwhg7v9bGMqsHJH5eAO2LyauawpDlYCKsvR/mD2hj1gBvA4yaN3AHk4V6DFZgR6D0zJfRkj0P2OEXa0evNXIhr/Vrkw7nl+rmwKbeA1sPDu1f3QOjI2jdNWUg0X73Gki4tr0Mls9k59tlzI/0VYB2tKl9RQP4Org7duy1vEcZ8By0vgJif59+Wemr1lh+/yd/yP/ef/efAhBfOvGzrKgQNte5dD88ACgcEDXMgcWi9D6M5qMIyIbefAQNpBzjPHpfOvOJMxIh0zC2dwB3X0DztQPwLN5lZ4kFBtWImQbxmM/XZ6E9APF5M383It75xZlPnNce7pgwHWgVQLl/AVv7pi0EXj7cau812vu+ef/F6to81mHaPdOSmV+W+ZAdtWkGwu0z0P2QXoovafVuW9z5v3nG1+MYGYmSD41xJDcSnzfagVzTLBrANs2mL99iJ4bO3hgU3IF63E/Ju9cuNnljD5NhfmSwQwkHCP07jW5cO85rG5emzUT3i3LfGzmR64vJ4GM+0vHlrfPUGgcaI+iOGRTYA3L93NaDYR4jMmnto/l/+biQnTXWxt2BJrhr/nfbCLt8LayJ01Deekm3cTE2UJ/8XFYF2krGtPO1O/pVMbo81i5jr6X0Oxsz+yLeg0RCB3Z0uOaByLFtdr/74/VbOxgDV9H97n/z+ethrHWTtmtv12/0fLr5ls0dtzEB0E3WfPIbypc2mbbJt3kgN054/ltj/Ay8L6P5mJypmY01ubJtfE1u99lfJ/Sx4EM+y2vg0cb2hTky0LDb/NpHxYs7AOCm7rjBNk2P/Q647+FQ1l932pcdSYttuK2/h3vLylnepnFz8kC1bF+aGkGK64u0j92470CMbexVBkb/bEVvbtKPL9NbslDP84zg5zg37MbXydUL3GjbyeDn8nndHbQcSW5Y70Efx9PL91ENGHf5vQaylT324VYdxzaP/fxIehFEHTc+N8odXwMvpmOeW/UeX8wfaf+WPJ/c/qeUeaXcF+f7lDa/sP5/9t/87uNYvvnVH/O/+R/8/BrL//V/+r3G8ienp8sJ/9v/8a/KaVemxo7GgcWUZw7dWT6q6VCQl0pY9SSOAMqEOlXEOYipTwVQqZnalJOyvdkGQe29zXSlnLjVFzZCmRhxlbdHVbMe/5IhFhnMlEVO/tBNQ/REsI7cH6L6cuTDg99Mcaj0/KTmLnGRWF8cxQSnDqx9R+tDGfvLIqjpUNu8aV3+wWjmS34TELK0EzS2WDsV0xdYVTOk9iIyU5wo5j9VWQjrwCKD5o8rYdGxLZGbGRMqsJ4ZITtfAOpjZ2MM6AkxgFC6jb6ZA4GBbZAxGx5lYPMdtxNR23xkY/fTFFbth9491b2oo5kaMTBuMj41AUHrGzYZw6TfCTrO+uIMm9SlHFMwE6JhtbHWOeK++eLo1g4DddTxLl1GKsBplrJ17PMK6H7bmcg0Eyl/Ootez3FzaWZ1Ntdh7TLYGrEXn81vKN3ErZnNBTTGQNsgmcmbmbSZCZqNhTfnIpXd7pG4SP/tvMhkDBkok9voMnamXNbnoJboZex5qioZ7R6ztjkC8Sq/5Ts3XjpfNrY2x8Ii2+e4MWa6TXVYgHKWeutB3jijmZ2ZDHW8LXsw80AzrwSQLipn6PIYgLJ2mXSeBrnW5sDWJrAb+6pzxbGv06D3QFsDOiZUb6wRm3cdA5v71je9FleRrQ7yuabDmrTnKQPpwqgjtbUWNln0ZaJd28RAXBj5LNftc5oZZaT2HI5rf5BLn6mtf7DJx7p2Sde8yECFwcpuKeuXwYGkTvT55djLAiqz/lYjQUIyEOLG7X7loGMxUMsXCvd7ZZM+pwvrPWvkKFaxvZNI69F14Z8zJPdNXPUesueM3ftm2krOfDX2tViT1BdtbU7opqt679uzLzjmUrvnba2JhkueMxz63IQiY+r7X0YZ35q6NiyuMrdfmuweYJJxaPd2lfZM9rBxYwi2+zSuMv41kshKAJPsS+LWZTczwgbgIX315oUcdH0O/XvcWJ53Hshyf5e152/uz8WgYyr3kbbB+2cytzXb14KvO2SIBt/q3KrWJeNi90nYame/dUAxZL1njAXXjZmNrX8/cdRDBGZto6+jW4ciu8MJ9yy23/whiM/TEkHu3/D6urH5YSIhA0J/lnl5XgOE/uCi9989I23vYKb11r9yqNcnvbaPI3qQ5dAPJtr9vVkf9fXy2cm37+r8KdrKLwKjv08/e/qqgeV3b57wn/07f4qnPDXW10XfQkuNGKhi44BI4hc5hIKlJGVfFdNSYz5da9yxhQJo2s85DxhjbrEoLZnpqjG/Vg5IoTRWWADIHBHcHWHaVGN4PcWtMcK2esHNN9Tk6ayyzx9sxmI7q59n08I6VktjIa0gROpaOev3UWvr6/AMsLfksfI++Xib1r61aeymRmTUfDxraNpeAtpv9pch2mEiidnpNZ9+Tnwy5lbL67XIANQ/NeB+XFGZsDrG1abp1j6bltgTMVlf7XfvYxrd+PvyxjLatJwM2PsqqTm3Z3u1WKNWp2d1LR7w6sejphqQvMYsa4ytgdDilvJh/o7aa0te+2qfj76z5jPr58LmybPhNrZWp7U9zl97R+o68wy7xszqw/RUtx6i5icdS8tnbdpOvjO2ctMCs2mTiIGizJ+qIbP9uI8XygxwrKIks1Ax6NrmXRs2d6XPcfez7XmhoXrgQg1Zqu5+agy6jmW2Mac6zbjfG7CSMjVWXKepNbNtk0uIsTpr7LEu08oftfD+u8j6PP8z9k+/+9O/3adXnyleY+9M3Nv4cte8h1jVNFrnWmNF+nVizYfDejLrgF1YobCPmsnATXZT6HXoutkxnR42VF6jb+34EERNW+/bZdpp3f1mjA/rullFmA+xk+1Z8lrPG5vTXaghrzH035l2IYSeaaOpm8jvtMmWz7TMxw2ml8nLuTshQr9G7PK5zphJ9ZcmQjdJphvXj7K/pkI8/nbj/f5iurURf2kDfgNE3NKgwzO8Htt4qW1fb7uZgP0Dx+b4eV3SbQc2jzLbA3cneyu9XxfP5NyP803gcVxfr4CTZ7PzqUDmE/Mdb4VX6/tEsPWpMrzY5gtr8pNkfOn6cY19YvpZgOP//DPU8VOTv1f+BqavGlgGVNyFFUtIQAXOcUUKRUOKSBiRUCOmkHHFgLMelRYmfDtc8W47IzBjChmZA85xwxUDAhjnuOEcNyw1Yc4DTlEY/NYaWxiP+7QKSFT/zlxZw4dE3CVp68MWcEqdhMcA3FwS1prwZlhQOTSz3VFVNlPI+HE7YSWZImk/CIh1QO4uifrMQp8YWJ3LgL91esRjnnDJI94OMz7kCbkGnOLW6vhxPWNS0HyKG7BNqCAMQUJ8TCk3YBnAjeDIykt9+SYwN1BoxEcGKjIHTDHjmgdMUQC4hDVJKovEDb0fVjyuE+4HmeNcA+6HFae44dfXNxg0fIYHwQaa7dqchwb4ko7ZFDOWIn6QK2QD9v3pCQDwV5cHDKHilDbM2tc5p917+zxsbSwMEJvPqpE1GbMvM7Ux20rElDKWnDCl3EKbtFApAN6MC1KoeDef229ribgfV2wl4n5Y8X6dEIkb4DbAbMROMwnZ0SllPK1CIpVCxbenK562EU8YMcayOyAwQG0A3A4DDAibjGMsrb+2BuacWn1zFkblp3XclTcwPcbS+r8q+FydX+6oDMXGWDtvCech46p/1xJb/FkjkgK6v3PW34kYD9OKOacWDzao7/QQKj4soi4sNbSyt2QBgGVTxlz1IQY6MVXVfGuO+OY8I4aK3zzdtffmmApmLU+6Lk/jhkCMy2IqSzRz6+zW0t204vF6wjd3V3yYp53Z+MNpkblX0DQNGR/mzjJrRF7XdcCYcqvbyn9/d8UPl7MQPMWKacjNP/rNtOLd9YRaA8aUMa9DI9EyZmcPLmOsiDp205CbC8Gak4SIUSBnJuhjEibNZZP7goFG2mTswCEIi/Jp3FpeQIDypLF/zT9+zbEBQguLYt+/ub/isoxtnRiB2WWeGimX+a2fpxVP1wkpVUzDhqfrhPvzgss8tUOEadraPJpcYqovzKA5B0xj3vnDp1ixrKmRfdUaUAphHAtyDjhP/b0UiRvTs83VOOZ2CGGHB7WKLABaH6YhKyO1HAz0eLviKz9fRzy8uaLUoIdL3GS3sDg5hyaX+df7sD3LPGAYhRX8uowy96EixYLrMkrM4RrUr3zPkmsEZKN+F+bY0twGorKMbmtSn3K5ZiypYuYv92TJsfmwCymYsoIr6VneorDSLgnjlMUPXA8ZptOGy9OEL01pKMjqoy1+5HYAIMzI5ocedSwEaIv8acwwt4SoB33mF26/2UEGaO+2wN7HUr+P5019+oFaaBdepx1QUS9rMYZDEgI1VEJUn9e6hd3BiMUcJjucqep3XqihH/M1NyZXzuKLG8ayP1gpAZyNCCy2/jVcmLQNT95l/pIMuWYHB+YL7M3j7fDA+2VasgMGD/x9FrPOKi5WJXXALVVQ8zltyWTz9YbDda1nV+6lAwj/+/GwDYfvh7871tQbQLpVc6v/rx0aEECVb2JLunFI9uy7v3ZrXm5dx8tnLj8LsPyFJKofz/O1pq8aWL7fTvhf/upfw+MyotbQNhzLlrDliPO0NnCzbBL6Yl5lg2IbpVoJKVbMy4BxzO2FOKSCUTeuT5cJ45Qb5btpA4bBvyBllaQkpCyjhlpY14iUnJ+mvgBzjhLOQ0MaWLu2GUih4roMbTOWlLzFtB+NeVVf/ha2wVhB8xbx/9x/I2OxJpzOSkNeAtJQ2kZ2XYbGyhlVQ+LZMmMqO61DMW2XaVFqaO3Zi9ueAo0BdY0IiVs7rPHU8hblRZgDKAA1y4urrhEUK9JYsC0JacyNkj1NGSlVzJexE8C4h6CNn8mQNwkCby8uGzPbqNU1AhX48duzbDjfnxqTqGlUyrqPNxhH29yoJqN0NtEwVKF6J0YahX21KPU7l96/MFSp14gx9CWdxoIQK9ZZ7Zv0ZfubqbTNR57FttJCS3gtAgVGnSMosbRhxDWp4oe7OwnHMSdQqjvNim08GkuqnvTvvhMQhiL9i0LEY+Q2YSygAJQ54odzbm2YlsP89sJQUY1Qxgg+jEiEqVHI23g2wpglgqYiZdRUHalrHkg3R6ybJQB4d85CtmNU75Gl35FRr2bHjE5AQnDkOZ1lldcgayxVqYvQqesrhJxkI3y4Pwlz7/uxa1MGFsITQPoaAJqKzNPVPX4tfw6NTfTxVMCXhOv9iHpJO83M091JNmuMztx67WRAH5S8B3PEkxGalP5mvjxMKI9DH8dUG9nM+3OW35jwpMQz15P8fRrqnjqfsSPvuYy1+2+uQeq1DaLOy8UYbpfY+rpjc7VnSCHMw0nqWGK7djWW1jXgOtY9eY/fYBKwzIOQJ+m9ehlk/vmibaO3PZ8n4ClhSRWXqQJPCev9CL4qsRUT1kkPA2wItn5vmHzL6MYbEEIZI6phGQsqhE2JhpZp2m9GtyCbXJ2rRQlTmqy6MV3HsV+rhOtU2jhRDj1MQQWWsYKuEe/WKPfIM/IemT8qQm6DTFiGKs+UpPdqJdASsI0V1/EWeY+7j5WAhjIhj/Isoy2AA2O1eq8ROXUZt6j3xxKwKfHNFtAIYcBAsVAYa8ByGsQFQO/5nFjGlABaA7ZUQUvAPCXQFqStCmynAfT45VufVdmCAZGnsbgGIMwBZWAgE7apNvccY8bdTnIITpmwDdyIe6gQtim1vLYxL9G5YWTq4IoBqoRljqA1yKbbtSnrB/2Zo3KUsSKsAdWIhdQFReanm/ECEBejRt6jvw36bFTTTCZ1SRnkPo46V3XU9y6jMQOHAmEmXvrz0ExUa5KlQ45JtzEOA42gh0lcb7wvKQe056eNZwM29sg0E1cz67bf9NElwBLPGHOlAfnnTd5VfPnZ1+v8jrkBy+7iwU6eZ2arx8/210yRGmjTDpDriOWr+teDPtfX5hLkZTmC2Fug9Yj99HfPTPvR5PtEJtQreW18Dn34ffrlp68aWH4/XvAf/90/w4/5DhtHDCRau61GfCgT3sQFmxq3b6yayzKgcsB9WtpnALiWAQ/DjGsZRBtDBQMVFAT8sN7hTVol1hj3TdUUJHTJ++2Ec9yQOWDUNiz/qqa53hw2EGOpEblGfDtesdXYTHft9ylkvM9Ta28IBYUJWU1VzTQ1UUWgiikUPOURFYQxZKw14UH7mDngPq54KmPTMAaqqBzwYZuame8Yyk1zWtPQAsCcB6RQnmks55J2GstAjLWkXbgXCfsRm4xzGZp21b6PoWCtosk6xU1DxGzIHJFraBpakbs07Yhv14/1XNKOLKkytXKBWEyjmfCH5w+iwVWA6evcHKmTzYWFqbG5MK2uaXoB4Jw2rDViKQlDKM0c28LVrId6AzHOqt228DVmSmuaH9P0em2jaQPt2nUbWjgbC5lDxPhmnHHNA7tbDQAAACAASURBVJYiGs1Etc2rmSOb6bI3+/VmuaYlNW0kACyqmbJ6zsOG6yaa9dYHpp15s40v42BKrHXa+tpqwBQLlhIxqbYUQGN7NtIsa4vRTZTPg4TSWV3fTNO6qNbNNLMxyB1lWr0jwZaNhc15M8/WfJUJpyRr/PFh3Jk2r840NgbR2tq49etmNk2t7JQyLuuAu3HDklPTxALAachtnmztWAzZoodlQbVfwiiNpglklpi11zdDCxskh2gi55gyrm/0kCpWLFkO5ywkkGlqvew2J/Z7irWFEhLtZic7s1izxsDdSMv0NzOVLk5zvZXYTJ39NbNaqO4eYHST4vO4IZcg5XXuiRjL26GRYFneFCrWB43rGAvmh0G00Q8yLsacDfQtkWlivXmvEYv566aBNG2jEZDVSi2OsJnfMneZrE07YLSxBNBksbYs9rB994RnKVWsa8Q05Z2W1pK1bfJLfbWTbbG0k3NAjMJevd11rahoHbvMRlwmDOA95BGIW/ilcpZwTC/ls41tVdcIb/ZdK2GIndXaDtfanBY5ADWNvJk+M0QrnsfO1P65KQZGvVcCN4ulqGui3GmsWYYevMkmv7lHaN9ZzcvNNFoOcftvNifkNtOdhKt/D6mCT31t7Ey/d2OodbZQT9zyyaGk5vFtkMqvhwXM2H9ucqAdtBQ7TNEDJSmjeVjqKhbKyAMpR5DW0tFcW/tTgT14dGNiLLxe2wigaSGfARWry8tzC9RY/oNMRw3aLa0ak7qcvAQcb+ErB15f0ha+pNF7UbPHQOEbQh/a3AuH5/6VLdsrKO/GGN7C0cDt7n9KfZ+dfmmg9Jcmz8+YvmpgyRDA+BBnzHXARdkw7uKKN3HBj+WMgQqWmhqonIKAqKc8KSDbsNQB3w4XXOsoALFGLDXhsZ6QqOL78YKnPGEIBQPl5rd5LQMqCG+Huce2LGJy+yGPiMQN5BkQLPrwHKhiSgXv1jMAIAWR0/w+BURtCAo2r0VMdJMefVUODbTVGvFhm8T8lqE+nxX/7/UtxpAxxoK/mh9wl1akIPE9DQyd0tbA2SWPGNU01UDlWhIueWz+pmPIyDViVsaARBXv1jNSqJg3A4ahAVUDlZUDVgMBVJuMBhAlnwDbUUGUmek+bVMDMo/rCUtJuBvWBhiyticAZr/pHZwJLiAv66dN+rMpEAzE+PP33yEQ42FcUDhgXoe22TczYwOPl21s5rUydx0kLhpzk5nwfpUQMoMCvKRmtUMQX2ADQ5aWEvG0jthKwHnIsgnSzfIHNWl9P58wpdwAlAEiA2isAGfVumwsCxPeXb7DkIpsmnPajYn3gbXNW64BVYGppad1aODFgMqYCq7z1MxTn5YRQyq4qgmq+YcyU/vtaRkb4ImBjY0fq764m2ljYMzrgBQlLmuKpfmXWh2AmKvanBj78a8/3Iv/cazYdBNlIGRMufU9a1kDNrYJNNA6KDiy+LPMhOvawf28yRg9Xk5gBs7Thqx1zFvaAeytAI+Xk9Sr8UhlUz/s/FYB4PEyYRozfv3+vpklWvpwOblNvYCHYShYs9R3VTPblEoz5SUCWEHu+w9nMeskYN0SPtTuK5rzHcYxaz0DUqp4nOXvvAzNpNPGb9Eg8DFWXOeh+ZXGVFsoIaD7fc5FLB2iAiYDXwAwO1/EEBhzGZs1hYGca5G+xVhxLWMzQ21MyNRVBk8/nkFJQh0x0EK0SKzXvv4ZYgoYhx6cPY4Fl/cnxLG2zdj8NMKbFlpcVQMPLWQN0Ez1uCgAMMBEaPkoVlzX0253RaZB1Omey9TBi8s3W/xQ1Yw0TTGjWTOYHItqtbcfJzSNu/eDNHAQWbSwkbFYPxxwwFCxWdzTpHXYZt5CjZg82nbOqllVzVdWudk06dqHXLEL0yFmeCThWuyZbksvQMLeEJoGhiqhuhinNZO0sQapU80nWcn6vjTxFqReiPmpLCzVriXu7NWZOgmLrQWLSRoAdnFNJcyK+v1aGcZeI+SuW7mwiMaRGBpnUteggQsDpbYkczfpNI2mhejgtC9n5pXGWsyEThCjL53Wbt2344llbP1YeU9k47WFrfwNxNHinWp+TzbXyWhE+B2wOlxjXVPAC6Cwuv4cfn9RO+fRks/zAji89f2ldJzHF9Nnga7b/ZcGb9f1LO8REL/U9hHIf4KMr4Lj1+r/ffrFpK8aWP6LywP+8Z/9u83M0AKaN9v/Le5XqZlNqtnPzvnfzNy8aZa9eO2ldCQ2sPpS7fVV6mZyjOfEAYB7MXA3lwpoL+huesH9IX58YbtDJXIbA/8igNGs2+eIbu+vfeHoaO19nurqAPpDwZgldw/Tw1GZ/bUywcpQm5+dmYWPl2VjXJXFL7kXW1D2X0Zj59y/RPq42LUdG9vx4c/997jIj3+pJj27h7/Jb0PhXqL+xSovzr5GmtyuX9ZPciYp/oFLum7emzmSvfwtdEZFY1xsL9njJkI3cDsmOUZj7KwuoHVr2q8rt3E4vkDILf9b7Rs7nQ8l0EyPcHhxWx9dc7u5hBtbK0O9PvKyeVlt01PEAtGHkCCWv6tjmtzFGPNzofO8KFMssdbHXSabEyZlTGTCk8Xt0zqc5dduI2T17tJhLczKeL3as8CyZTmVt8cXcbe49bKvx42afh42QlZGZCq0e0wNBai2yayELTBiEWbmUAjVkcZYnsBA1XwcgKjjY9aYTPJcAZNc07VirlPWtpVrltFFwzla3/Uak8xr9GZrN9bAoGu+PXpsA5vksd36oOulDlCWbXnGjMb+a6ljzNaH3fOB3PPB2rT1+8IG8/9j7116ZVmyNKHPXu4eEXufc+/Nysx6dKl5CISQgBa0kBBSD5gyYcS/QEgIJP4C/wCJYfcYCYkJI8SAEfSUFkKIrmoqi6x83HvO2Tsi3N3MFoO1ltly3xH7PDKrKk+qTLr37HA3t5ebu9tna63vU9ZNOwdfvNfQy7cPjC3zFpvkpi7f+/9igWyf5QbqdvW7/hw2huS6ex/Wfm9aOw0Y1OeS34vh9rjZ9ylJPlPWpi9u239n5hvn85v3kT6nNZmH5TMTgyPf6rRjtGEfteDKzL/9ewam/dr2VpcBKrcsV8xw7tr48Li5F/cfmzliXUmdeXdvr+N+7CRcbJ+03zf7IgzI+rfX75f83eqgXg5MWfs6gNvfPDnPf9/SZ6Q2dn298Eqddsx27+bPAjovxnFbxmcDy30d+zbv234v7dr1WXXYYqjfX/39Wekzs2/rvtPmryx97e1/LX3VwHJKGf/aH/8VZnF3BNAsV3rP9kycaqWxGpB7dk1Nypq5Z6Dcs+6p6xZ2dWo7bnwP7m/wGIvRnqnT5tG010jkY2i7+5z/dl3qlnWvbNvHvd7ivbQlgrutj2ndq+wOvD2+b2cTEJdy/GYldqtzfO5We20/+NvTmSYVZN8sdfd1cHqtlmvqbWXv2kg6Rraf9mOk98zGYJgsDRDvm6b/3pgbmqqxrNxMNz6Kn50+8b5sLrlRl50HrFPpNtqeL+qxb+n9BsZu0bO57taGzX5xc2ssbi32LXHDrTbaMbDtfO3YvgzbLtu/OwvJjWD25lpsNks2fd6f228Y3bq3tzaWPvKu+Gja34Nb9+XeXNVj+3vy2j11kHgieb7JkHns2wPA6lS+mnQ879X9sf7YY7fu8cee2/192R8D2t+v3b79I/bqYtQeu5UcXmgu3qrn1fI347J75l9LztzTL0ivLmo/Nif31d56j9xJn9ziT6n73prgxmP8yXXs66LbE+FVELKbfDfr/8j8+uiC/d7H81PK+dLv4qeU/bH0Bdd8cj0fu58vS96V/crs/Mx2/0aA67d0f/7aU1sI/n6mrxpYKlj5k9M7XEvE9/MRAcC34xmPacY///AdBl9wyQk/Oj3hr54f8JPTE3L1+NX5hENacYgr3s0T/vjNe/z8/IA3w4ylBry/jvhwHpFSwb/83a/x/z094jTMCL42qQtlTvyTt+/wbp7gHOHpOuIP33zAzz88AGD2xedl2ADXXD1O44JjWvEXv36LENidMFePUvi/vAa8ebwgBnbtO88DUihIsWxi4Z6vQ2Phe/NwAQBc5gGHccEP358Qx4zDtOLpw4TpuGCIBc+XASUzQ993b5+ZtRLAdU44HWY4Ry0O7nwdhGGQt8MPhwXLGoXhryLGivmSMExMfBRjkbIrhqFgWQIejjMu89BIbKZpxfUy4HBY8Pw8CbFQwOmw4Pk84uF0xbxGnN9POL294vw0Io0Zacg4fxhBq8fhmyuWJSLGgpw5LoqqQxEJBSWGOTzOAIB1ifDiwrdeEsLALo2H44whZnz/Z98CAA5/9IR5TsjXiDBlUHEYD+smfmh5Ele8A7tTBu2zI+RrhBdmQno/gGKFP2ZmzksV9RIRjhnlEhAOBVTRWASpOtCHxC5Ub1Ze6CmJ0YcBiAR3DsCbld3cFg8MFT4yYZBa1dPjgrwE1HOCf+CYzXoNCL9OKKcK92ZBnTuhCRwBlnBESG5o9d29Tqfv1TMhTXFwq1iZDwW4BrbCHgr8h4T6mDupSiQmMMke7uJBpwJ3CaCxcjmHzOyDnpgQxFEnF5kq8ByAUwbeJWAq0j7JOwgC0vqViAZA+CGhHCu3r/J4udXBzw7lsTDoEGIYlxkg0FibBd5fAuO2Y2FLyeL5PMnfidgKLcfTzyNAwPqTtXkzuEuQ6wWAFIfwHOELsL4tDci4zOY7SrVdG99F5O8y0q8S1jelux4CSD8E1ACQ6tReHfLb0gBMeGbfs/Imwz8FJtoYamMnHH8RMP+ogAaCP3vW4RVijvTkMP9BARy72pVTRXwXUU4V4TmgHGoDbORZL9gVh3qoCE8BdWR93nysCBd2QyQPvjcAwtnDZyCfVFuYQFHKWhwogvU/R0J49vAr51Wt3vSBraLlSIjPDmWiZj1rWsWJ4Aow/tIjHwnlwNfHi2OL9Te1eSlUIVFJ7xzmH1XEq+cx+K5i+D5gfaTmDZI+eL7/YqnLBwKptm9hbcf0zG6GdeQ2hKvD+sDaxuR4HGoA4oX7E58UFPGYlolJTtQ6FJ9ds5pqW2vkvut1NbI2aTlIGaqlDDl3AZa3hMNfedSIpl+s1kHVHa2DlDPyvyA0XcsagHzi4/EMrI/sBaFao2VC0y0toxwbuW5HQJ7YOqt6r8sb1poFcflh5mvyif9WLc08ofVTdX7zAUgfwB4YCaJxzHUBQD44hCuQH4D4xG1TzdPhPXD9gy8HlvEMrA/8uowXNLdJV7g+1XyNZ3St48jtT0+s6VhHIFzZekeBz6cn0bhMrpXJWqdSxsDvEF141wiMPxDWI2sJ1sQ6pTWhWzEJ8IXPcXsJZVRtT7D25cwFlon1XpVoxov+Z1ilXdEhXkj0TqXfFU1PFuD7rWOkngJh5flWE49Xljlqrd1dm1bnPbXrHaFp36quaR2w0VlVTdAa3NZN1oyVJedxhGZVtXqpqnHNeXQTBluNUE1249ezbqXV/FQLuis8js0qT9joJVtLdbce9zr2etKusD6q6mo64mdTdXFdJeiOfbfm97lmNcVfJNndVybYzQaDbUM1/bQbd3Z4dsde7KPuN/taxu3Y9nHYWkm5nN9fcPY1p68aWCZf8JPDBxzCilFVwAEmpwGTlQwhY4wZU1jxzXRhSY0A0ME1KQ2AYxy/nS6IviIRg7dDYjKO6AreTtcWjzcGBi/+KAslX/F2vKLCYQwFwVV8c7hyW+L6wmqnBCcehG8fz9KX2mQeOI/HSWQJtI7kayNxIXIIaW3xfEsOeBhFeiQycUt+e8EYC8sNADiNC/fHEGtMMfdYwVAwpQwHIAmNfPCELJT0WvYQC/LgmxZijAVjLFgSl9nIMEJF8LFR/KvGYfAsvTCmDJyuLAERCrP6HlyTf3BvgcOwAg8c5+Z9BT1w/N1hXBrIrkNucZRFyC+U5GIaGFjFWBoxzBJLI9hIErMXvmNAPQgLbk6lxXXFUNv1xVhhUypdIkFi4NZYEIVRdg3UwLeSYqyxIKWCHEOTO7Aahqsw3aahx9RFoXL3oSIPTLNP5FBH1vELFpiSQxoy543UqP5LqOzpPRbElFECdWutIyZ9IDRtQOcAGtBi1pTsoY6+W8cFb4ZUUBIDrhgrciAkYVEGgRlkxYJfRyYSKakykzI5IZSpcL6iJm0HW/19IJRYecwCE3/AcXklMMOnA1CieCIIky0g+EzqIQA0uLb5ECYhYXGEauLfvLrCyyIH4DGjCtAkZB0AaPS8UeEAGphuf/0ReCE3lb7gGDRuj89RdSiJUAhwYyeCUZ3DxqwMIHuCP2Ss3wJuKmKx5XMr0P1MAyEfmcVWLbslChvwUFDFjdaFbvGdAdAps9dCrBwrJmCxHjyDYUcok4OLhOx5cyBHzwDXUNvrOCEQM28GQj04zj9IXocWZ5bF3RyRUA/YnCuj666lgZCTuI3L/IIHliSLi0RYBt4Igec1UVXhe7HOXx1AIzWG1HLkzReaCuqkefmfOnjUQwGNHuXAQHkOQFWyEceLY8AsAKOEE+jGgUcrl2QuuexAAyFbl1AHlINjMpORF526gKuJuqUVDJKaJ4PnxRR5vjfaLvJAPjFQ1Xbl3PuXjw51qrg438CtDe1w4sJMkZAPHLeXT64BJgrS5oHPh0cGa650gKEAm5wszqmDUUBAloBYV+V6XWg7ATSVQYPTMI7awRmA7socGMRQ4HwEvn5VVtJE8KtDGaU/A7uUUyCsJ4f88OUL0lXH2XHfFBi4wvUpOMknibF0HRzmoxOQxu3T2EPy/AzDGdABnjttfsbtgpwCz4EygK3rAVgfXAOxDVhV18ZwXRwDdon/JA9uB2RuG7dujseU+Ssu34vMqX1cpc8QYMfHwgPaZojOnxoEZKYt8CPZMNC+N0u2eSZ0jjkBZTYOkgFnb5dlZu1jtd1I4fMdfDkBcw18auiOAT/6jOxTN7C67hq9GX8zFgY43rQK78AWGeC5yaNAS92KPTrQvAPAyPV3631CHnnXGIvxa94JG9B5I93ycnjVEm7K/iRr/leMK38jy+zveHL0FSP+b/6Nn9A/+u/+08b8CQBZyFuUSVOBmhJvKKvj3lXVskEqe6SOzJIjoilLk1oOlY1Rr1VWRGVqjKETglTqLra1+maRVIumMkJqOdbtdu+OWsk11sXG6kfdDVcZGlUfzfZXAantt7oM2/pUZ27vPrlnX9Rk26iurSqsbtkFW52+C6/X3T3QY8pkqUyFwJaBUIGOJQHRtHcPBrARjFfhdZZgISxL4AW+cbvcC8I3Zj/9MOk5AX96XMuwQvAbt07D9Khj7nzXnoO5z7rB4IQY5Z67tB1HYPudUtBixeHbfZP2713l3K6ONv/0HHp/9Dp7P7pLMlr7N2Lxer3W2+a/nkP/SHs0ofj9s8CELZyvHdu5/m503XbujS/Gk9CB3K7tN5NYl+GAuvru4u36Ne2YjtWNsjZu4jpWSsRi7o8d73atKa9JyVi9N42bBppGXXshmC9+k8AhaatYsTdacDaZGNYXsXU2xnO3aNq4520WV7sViDNjpW3S/ijAtWWTKVvj4/W8Xlvcy7o1Nl4Wta3fVoPO24ZqXTqoukqm28f315k50vrobvTnVqI75+6NvS6MW9ArXi7WZLFNlkcA2Ixp08tz2LgN29/NPXjfjjbXpFzrHmzT/rh5v7b5X10D7taFV8dSyWkYMHTJBAYm1OQ3viRZUPWi/TaE4VPmvg4f4YVUhZ02NxehhGaBau8+27ZN3ttzcT819zGdL0DPfs643fH2nGF7z8lk3T+r+3HR3wrYTX2tTffmDr3s076Mfcfs3Lib7jwv99KmDe7OPbHl3vr7U5Jt1/7fO+36pLI+I30SWNTyNxd+el2fXMcnpP/jv/kv/ikR/cPfvKQvTw/f/in9g//oP/+tl/u//vf/1d9634Cv3GI5+IK/d/wBc424lLSR5Mjkm6SDlY5YRMZB2VWXGhFdQaYAD2ryC9FVRF+Qa2jyFxq/qVIIe7kHtYAptT0AFPII5q3SYjoFACfPwLDCtXyax8pc2Po1D4BmwYyuYi4RSrlvJSqUPVTlCaxwvYrd34oL1X9t/KjKRmibFLRqOTZlU69aRYHOZqqyFRYoWwbXFEoD9SoNwayghEXYV4sBUbadej/WojG1HdRbQKrgfRpWEDnMITbrrI7zHsQoE+mtuFxlTFWpBo3R3dx7My7A9nuirKdW0sECy6jzy4x/67OUpWDcybUM4MXaXVl6Qc9r/fs+3oup3VvfAWy0XdUyy8Lr/Rhfiw3oVGBpgbDdcLBls2RCl1OwefXvvqHBx1Q2YV+/lqebA037FNtNAG3DXirCbpJo+2plwXsiYI1hU4aNY9ZNC+cIORsAqvfCPEKWXbUosNRzob8rWj0y19W6DYA3ITybAOw9Vf3YNrfMSkg1ZnVcdT5VmUfVbrCZuWDfG/ou2awddvfKsje/WPgKIHfi3bDZnFHQbDdopHLeuOnFhlRQS2jHnFiQqrjlt3Ica6eqBq3+7UWeoQGA0G+QitLbzQm7ydI2FHRjoC2+ZdNNwLpuSDTringftP4qgDLj1MbBlNnI48z5jUh99ixcT6xbu4lzd1v5ic1mkZTbpDA8tQ0yyLMAR83qT0AH6Ar+gQ6cdQyV8M6ec8ALMrh9jKsDj2cDlvKObWy6rklg6MYKCakeiWs+rTdMT5+avLmfu+O4dVzbTthomW5fwOgbNPeA2w6cbJh49bdaoO2GkyNj0TPEfPt6dy6XN0HjrZjlTZ6uXbmflw3k240aua65VO5ike8CCZ0j+3Eio31pp9GGs4B219C2K3YM9mnfnj0YuwWWbhz7LBD2KQBsv2i7k/91YGkG7JOv+Ui7vjDvbwVI/ibX/k2k3/X2/QbpqwaWk1/x9w+/BACsNWKmCA9CcBWFPOYaRTMyNlkKlQiYa0Suocl56EJIU6bQQJNKbGiq6JpsuQZEXwRchaZlqXqNqhWoUhWa1AV3JY9K/qZOpWpUAmjyIsnVdo13teVXIK1/D77gLHqHUeJCVfYgm4WhylHo+ChoVgCtUh46RtyWrZ6nHRftp4JdK1FyK28m37Qn9Tr9fS0Rcbq0dvHGQEAhz67HBnC1coklR7S84DqIvgf2vSNcMuv2fXc4i35i30zQfqgGpY6P1q357BxTQK9W7bZZUALGmBuo3hM/5coW5kNa27EUStM8LORwGta2ARDMPdGk962QQ1KNQWkPQsVpXBiAWqCyK0d/W41M26eNpIvxBAi+ssvzuLzom5ZnNxvsZoK9TxYwluqQJtlEGF5a7jUPW9jNWBaOa/XO1s99GM34Wku5NxsdJPdZ3ant+W3dvt1754CH4/xis8RuKjSdzam3Yb/OdDJWaVyx5IBB5A30XC4dlCpYH4YeDqB6iGFcN7IamtY1YJx6vxQwA0DO9hwaoO7eBWUDSjpw79Zy1Rm0IHRP4BXs+tKcs8d0AyBJfCbrJYp3QuR6fOz9qtXBma9akfhw1zQCZdNqzN3CK9fXEhCHDBBL6cQho5aAECtUwqTKPCf1zDU6huqxoOPhY9cp1DbymJEAMQeXtsAVYCCp1zoogJNzMpe8M4AaaGBHwaAP1ECigjU/lMaUvvGgIACVV/ZqyVamdZ4sHQA60Wak7AzglHYwNXBrj2X45otltSixn80K3ia8HPfyt8fGyt6Am15bTLkKQI37L8Q9E+pOKvIkTuJ+vzgZ91RrkXTFd6DnwVZRxcS6cbb4Bo5fEIdJzOBdS+ceMDjAX3y3QjvAr75bbrliWP8Uv25jEFmipJfXQNiub23+ra3YLYAzbrFcj3nuBfCSB8Lit6BT27VzIW1/a5K9F20L/9M3HG4RQekw0G7s3H5sTbkNVN4CtfsKWv9elrMpfw9Ub/1+Jd0FV/fKfa08u0lws6471tsveFw+CRS+Bsi/pLzX6vgdSvYR/X1MXzWw/H454n/4f/8dEbHX+LquZbfk0BaBXha8utBXcKXzOYoFzbLFqqUiiUupWjzaolFj2lIWghteCKnFRuPv7EIUQLMweE9YjWXAulxS5Riu/YLLumhqG6BAR2PjZDeviuXU+doEoyGLP317BBPTZcFSczGlvnvO9dftMV0g+d1CBRBXRPeib3ZHnIpri5X9b9Z/Yz0yXSBR4bp9qs3SsAEa8sFqbq/iOmsXZ3a3U8utM1Pau4FXjLbPVlMOgBADobGUbtw77W57kb91oWDdCk1M2+ZD1GRqgLaD3xZH1PXjCH3h1j4Wcq2VzNFFhy66PG0XdPoQ3Pqg6wdo84HfLRZ1fPT33nUS2L5B7fVahI6XLUfr1QWjN2Nmr2k3BX3cNIvEVO3709zodgsXbn8fh7bLr7Fyxv2uuQE6dL030fsgmUPcTCMrQ/wbBdx2u7jVLN7I/4gOn1s5rrG5GQJMnKTlOhlXnRdAX3SrJuBuweoXh0XixFC20jQ+A6tIkegcclb/zs45yWOtEeqOt6hbnkNf7JLbkHBszgMby4LGPDkCiiW7MHF2Lsu5/YJJfvvFNZIjR2hyPjnRC3dGlzn+1VX+uybicY5ihXNoLpTtlvZXaWuzkp/AmcWtjfuS5037GncSDuT4XuljY9tp144b9zoZF43P28v9uMIxblFj2fYxXmrhE6CxIflQC7Lrx31hAqEW30bosaYAajDaikUJRsw9AOT6/o7Yl9MMllqWPGt6zKuOpZk7m7lR0IigNnIjK1BHfHFyGT3us/Z7D5L6BOTs5UbISSyibZ++ex0awNvcO/uKsMfRy9sTveyBVG+4aRttx03L3wMwa+0jh0ZO04q0edDbYvuuEic9htG96N8LAhvs6jFusS+A4j1L667vm4dnd94+C/Z53uS5535r0+790465O+dvtPWjUh6bl8Cd4+3ij/xuddzox6ttlGw3PvEvyvlI3Tev+dS8u/T7DNa+pvRVA8tjXPBv/+hnzZX1KQ+o5HGKM4IjvF+ZTk6tlH4MHwAAIABJREFUiE/riGNka89THjH43NxjH9LcLHzeEc55wFwiE/GMF5zzAA+CdxVLjRh8xvM6osLhMV2RKYgAesApzXhe+cs1xtysfg2QivVr8AXvlkkIgtiyaWNCleAHQHPpHQOXp2XN4to7l4CHgcl75hyRQsH764QxZqRQ8LwMOArZzzXHZhl6GJZmbVqrxxD4y6wWpaWEjRUr+dpAvFqy1hKa1cq6tqolaooZq7hgBl8RHFseB7HEDTGzm6tY2qIQGV2XhOO44LIkIQKqmNfYWHWzlKftU6sVUbcoTym3c2q5UuKiWn0jFnr3fACRw+PxilJd24So5DCm3K6vxORLRK5ZgC34V4slAVhFmF5Jerxj19sglhyNX7Xv0HlOoOowjLmV7R1hWdlFd10DxjG3jQ/va7MktfhgKX9dQ7NileKxzAk+FAxDaRsf6mIYjGuluouqGym7b3KZJftmlVFin5gKcvag6hFTwbpEpCGz+ybQ3D+J2IKkLLohFtTC1+icUWuUunXGVFo/liW2/jqH1haA8+vGiFpt1jkixNrGX+NLa/aN1Eg3AyrxysmH0vqlVr8YeTWlLqE6nqHFw3L+fGVW2HRcN5tPceO26lAWRiNhyu3LrGsHjdUGgLJ4xLEwQ/HIZE4NN16jbIowyKPsmWSIuE9FQG4YCzMGQ6xYskAr5wh3KKL367t1C0CZPZ9z1NmMlwCf2OLlxILXNnSKRyWw++jqxY3UwSVhK1Zrl7o3C4OxG7p1r52zG0yxoq6cF0r444ktaAAzDS+ez9nNHAN86RyY7VaIopg5GYAwEvPACOBdPGgqQHZwK//troE3CiAPqrAHt02HaOqW9rmVN57UesXkQztQLvkoENy8RQwUabvBsSpCRdt4IEfwupHhwOB/YZKgBs7MJpVbmbwnPHsGwyrx7Lf5KHQw7Rd+Jlyhdk5JcWhlIh9XmCzEVXRGT+1D1XK47Q2IZu5OEebb1g45XhN1MKjl6oJf8tdIjUlUCXJccaIly5sWXnRJ/WI2MTyAq0M+fvkK1AmDriMDakjb2sGb6gbzHJNx0TYHAwqlD34RAG83gIzesSX1afPxwoRBCuLVIsngXLJVxwRekPOxbwDB9XhTZW+2ZEQbvVAPkN2YUOKlYECxEAf52YBMIa+hIKA+yXjphoaCcJhydcPA9bFtZXls22nObxyptA7wpordtNoAR31XNODb+2YBlC+78k09eyvnBpxSv/+fAiz5YpNvF3JzCwRvtF9vAWCpaxODewdgbiyWd9uI9k563QJq3mvONdD8uuTMjTo/ERB/NbiS43D+tlvx15a+amB58Av+wcOfs4sePFYKWCkgyVtqFgq/lQJGv+JdPmLyK8c3wjXQNteI0WesRgn7Uvha7wjHsOApj4i+IqCiwDcAwWUz2Cvy9kiuYK6x/a1t0PIUCCVXcKlDc7kFYGIWA7PQgjaut0ncMm2+Sq65nALdbfbDYcIYMqKrLQZVAewqbsCjLy2utJBr5QPsrruSF5dh+TC5ikq+ufd6R+1vAJtr9Xd0DP6su651hVVXU/09SJsUfKobr7rCVvIYBGDv3VH3broaS2vBscbV6jhFX/HtdEElh2NcWv9aGbsxWWTMFVhad09tq7qNAmjg15v7al1ztYxSPfKDb/GxOoYab2vdlbVPtm3a/6HFD4eNa7OOp3V7bUDS0au/NekmgsbrAtj8jrLxkExcs41LtrG89t7Ze9TiiIHGxJt8xVxCY09uDL3oSZ8RJd6yGxlkytVNk/046FhrUldYjQ+2rr2bmGTJox4SY8qbcu39r+RaXG+KHVBrsh4RRUDpegqNgbnNs2N44WJ7i8jLsjTbtJx4Q0f7aTdEqhmfQt0lXsd8H0tdpJ97F+kWZyn59cyeIExjLrUtNlkSML2maNy0bH5oLCzQY2A1raeAEGhDAFYrMzrvXYTzGhBTaXlCIJRT37zQzYytuae7lbY263hLPvUmaXGu4lmi8a/qXgszTlu34ZdmAWfHBrpBopsHMibmOioOUfoDoBFbOV1UazyrbLQ4Z2J2Tbylk02lsnreGNDY34oePyjl63VZXEedJ1RCA/Tqytz6Zt1c1aNBy203WECwxMA2rw49V8FtUK8Q+68suFf1+vjSpAAV6ItqHWvr3WLnst47Sw61c4VVSy6ZMjpzqemnASdudUBAy+Ny/71pn4JVbbut246ZBQr7fgh4V9CnwGnPEMt9Mf1SbwjXgZu2rVt3b4zfrr/Noq2AWOeO2fehzR/3j23G0SZz7IX16x5g25X/4voNUjO/7b+763aHblviFIDtf39q3l16tY5bwBKv1NfK2w+Y/N538LeUbvbh79LfePqqgeVCEX+xfItzGRpwC6iYa8JKngGfkPLMNSL50ix8SsyjgEbPK2HOKNbMSg5/lr/DFNYWJ6hg6hBWtqyV1Bb4Shg0+NLA1D7OzIKTKeRNbKGeB9DycHs13nG7ZeYlhjD6iu/no7SBLbjHuODdMiFTaBZb7a/Gbv6ypM3iX62r9pgFAjZWE0DLo8ct2FPwtNaA6GpbYGuMo/3XgkwlSxpDxrvl0MY2VwadzhHeLdMLEiW9NzqGFa5ZY+1ik9s5tjZWcngYWO/yr86PvIFgyHUsQAIYSAHAWSwnt8CS5qvkUHLa5DtTaot0exxg2RnnCOc1tXe2WnNJzj8tA1vmAFzxMi7yeRkaoLqsqYGjSSzDz0J8ZEHSPh7UtsmCheAr8pra36V6zFKec4R5GRDFQm6JpC7E1jj9bce3GJdmjZHU36Uyu7FarXMJzTJt4yH1OiVLUoC35LBxgXeOEDzhuvR7om0HXsZbAsC8Dpv+6ljbONc5B4yxwLmKs5QNOXcVy7WCr+ArvK+4rv3129bGZORsQsF5HjDEjCXHDchPAiIbeHWEeTX1tranBmqKicsc04p5TY1syZLpDDFjlrZ5GasQKmbRi10MgZLeK+dYIkkJk7wnrBIeoOBBx1bJiyyzM5FvY76/B94TsujE1upFfgfIuYcdWFIla72fphU5hw0YDaG28jhmUzYEB9bi9Z6QxFKuxwAA5Fgih3/0GFADvKg6hCibWeIWH0MVECm71AJwlVSJ5YN0LNlqT2YzIMQisZoCMnTOSz16zieuB4EamCWdo0JilMbcYk33RFz63AQB3UE2HhQkKjAPqcJPLCfkInr8qXlPUhXipuoQDhqOwP1zozxreSvF4wZwOdnDqbyMQ4/1BBrIpOLgp9zCH6iyhbsxOVcHN7HF3I3U3emJ413Vov8lyQUmQtKx6SfkuFrAfe0bDmqZH+VYa4/rIG6ifg7Sd4sybsUhHkwsbHVNZ/cFmLGhCi9iXzXP7jqtp4FNB/K9/MYIvPsbEHC8j63VsnYu3BvguxlOtwOZWoeWYdq5AcQvQYvFcZrab7WaVmwsfx+7fgOo7hYuF2+sjg1xSrl099JXweMe1H0CALyHA2/CvNdA5Z2C7DR4AfL2v18BpZ/cls9Iv2ug83etPb/N9FUDS+8IR8/un9eamjXRuwVRLJfebCWr9aqQw+jzix34KBZCdoet7fwUViRXmYxCgFkSEBkdGvmOTaqraYl2AGz/hsMYciPt0Xo1j7VERVeZodBSHsoYWC1OPRbl7T2EAl97nzywAYsDlU2ddvefy+p5AbTdULX4VcdAFoHBnLr9Vbjm4qvuxQ2soor1TY4T/+vk9wA04JN86YDVk+hOUutnA1VQgNHBpiPWHbSkQgCY0Ef6mXeAcJTFlHcEL8DQWvAANItg3gGvIGO7B3rKTBC0XGXJNQQ+mt8y9jaLj6+tXOeoEfK087YuAAgFTtoCoIGfaEhzFJi2f4PRVJSxU0tfsH10hGKuU1CmvxGYyIrMcTNtGuCxLMGuxZNQA4AAZNHLfVZXaLbMdSkca/3Se86PfAeRHNQIKCOw18U+8XpGj0NmLrs8EzqDcAeeMfTnvMU7Sr1atrXoeV/7S9bZNvFY7i2J3lgHgqc2vnZeaZ1ahgKwoIs6Mx/I1z4/wu7d4SucM5sheg/RY8MtEFQA6j3XsmfkBZQ9t19nyY7U3VoBu5bbY6SpuV2zFZDac6rHmvXT6fVobdJ2bGLVWx9g2trr5rLRY7KlTP1b/yWzUtIY+QbKfG8XORjrqhls024CdSsjtrJGunJSsOZ21ylg0ThyJ/VZ+SNAFop6P82c2qf9/LOL4X5fbH5Zu0tsv+aD/LdZ5zopT1flmuS6fdntHLBdQPbXQz+0+041eSJTz976raCjbj/Vn59eWxDuV+i7NtgNghfn6u323q2fhP12U+b98m+267W+tLLo08q3QLC6l8cNSFJ3bpD8Tf3vbf3mbwssLHIxnAmgLj2zB4PuBXDd9RXoLrA3xsJi27v9N2W3tym9DrLuPYL7fJux2J+78dvdOnerb/u0q3M/jp9U96159TGg+bnXb859tFd/l/4G01cNLH/x/Ij/9n//R/CRiVxokcCRQPCxol5ifxkF6ux2njg2R10/iuOYm9l3sovi4IXBjA5FmCjQd9w8ODYGAB0Kx8t4iU8Zaz+XxGXDulFUtDgaf7HxOK6/2IpDHStfIzEyFEhIKMxLXog3XO75NUYmnD0oEWoEwsyizhQIfnaNcKRO1FyQ/Iom/KzxQH51m93eKiLYTohpWlyHxm00NyZqBAUsyIz2YmiEClHINZKQgyhZhsTqqMC1n12L/wgLj2WZSASce+ySjrP9ANRBFmYmVkXFoUG9v+mdhyNgfaxw2W2JMDQ+SrH1ImVHOW+0xJqwN3HfKBD3V8gzfO7xP9QNIe0FHmbuQx37QpUCEGb+1y8OZaJGVMGi5lKvfDg1zsevrpXjMhDPLI5dDgS3YkOc0dyTLImHiJ9bI7nPHK/jNVbNc7yTl9izmgjhym1U4W0KPW4oLA5loDZnQK7HZnm0eB8dz5oIYeZ5EGYW/yYvz84qsUOO29Xbzn0OV86vovE+67PC4uxwJDEpbkv6IR9SLwyILZ6qcHtAromgk+v3NF54vNaHTj7SnilZ8KDyMVdZ4F1TiysKaDv6YQHyAShnyWtir+qF55O6pfkieWQBVnSOjuhMjho7BABnIB91Tu3u81VE7R2PVR0AzPIcr9gIsGseLR8SV0YSh4ZdHBnAeVwFKHFfyfV+6DwgkvyrHBukPt/74xLgFvlX2qNxUO1ZfQLcYJ8JzkcHbIlUKhCuAE58PMyAOwLhAvixtzHM/TlzxGOh1yshkex1tmfKr3xvGvGJjIXG4IVr7z/Az3OLCQRajCK5Pgf4Xdj/5t+EMrjWrh6/yHXlAzB84Gexxu0C3BVCjU7eNYSaHPyi78s+18voEBaOmyyjgy/UxOtr6n2skWMva3IIM8lzxMf0uSqTg8uW5Eeei8HBZ/v90PcU4LK8FyOXq30nx9fYZ9av/JyHa/9uwfPv9QFfnMLM95OcQ1ipzwd5pnW8/GLe8b5fq+9sv5q56l/OGzufAX7e7YKbPJDOhDwqoRHPCQrYkkXRdi7WiBfvL0Dmsu/XaN3tXRyAsAqxTPtmEWrgsecy+F5xPrltEmNZA49XkfvpCHCV2r3TvrvNO0HWJoVaf8hzWa5qGegPj9v13YxVe1/tYwwVO3vus8ZjtutlB8cV+Y41SmU5h37M4Ns2jtr/F3Vid0xPUe9OH2dsx9PEorb6TAzpTZddZ667Ue+2DT0WssVE2nL3/bhT1s3YV3P8s5Ktw+HV9t9Ljgj/1xdU/deSvqD9X0v6qoHlMGT8K3/6C5zXxBIH4h62Fs+/vynNvW7JAUMsmHNolpZNXJiwpmrcTRB3SCKHeY0bDT0lItEd+HWNm134LC5jQHfX0rSJgaoO6SdlEzekO+rOcTl2J92Sqehute6qh1Cbyxaz0zqMYxY3MIcYO1Ot3ZXPa4AXZljnCc60Q+OBuswA4FTTT9lpSSj2RcPMma3CWj2ToeQAiBukkqh4cWFzocIJ1T2UEVbZP30FJA9VzwtucXfCymUq2yvpS0+sbO19V7o2W1uUKwutAygLqcsf8oqkzkyK4jx4N9hh64oFMGkIOdQiO/oaD0ToDLeOGvO+zVelf9Ub3TdloHWEEqgRqvAmBlsmirCR+qjnqL9crUXCASQEKk1vDTxXylC47jXweOiYGAtPixdrVgVsdtmdzhVpL4+xb0QyJG5sJPFPjSlXP+qBz220/ZQ9F+bGqUXCxEm5KNeY9nEDZONDx0PdzoQ5uEsQMCB13ujYaQyX1mtZfZvenrRTmWm1LO2XxCzNg1jarmYFGIj1+hpapR7fpaQs2m9tu96XSMDqMQ+V89pd2aF2lzJl/7XafCqDsdcK1DQWQJiQdVzagxMrn9N+ZsdtsWy/+4+8o94O62q3Z/rVMt2uPbtFnt3A24w7ocfymXixZgFpiy/5PVQeA110Sd26Eah5ATAD7yLHg5DrpAqXfWvfC5mKiqYLqHqBJOy8llXYZbetTzYXW37tmwNv2plFIln21LbYRWM2brFnkbrEhW5wmbxu9XgSV0mNlWsLTmc2QGVzo8lYmDHWucBkO/yebuzIuw2+dsyyWIuFt22A2rmg90bHSt9B9t63GD4mEtK+u0pCEEOtf23cFayom2PizcovTTXqppMCDTnh0EiH9PfGEgYwuY2Alg0AsvPGTDH7yL9YjJMDJTSwDeobrVtwQtu5aO8T0Oel0WTdtL+9G8GMrhtXVn0O3aaN1eZzts9989yY7RuAdHqcsBmLLUsqZH71YxYQ31qwt00UY93s4K+3/QVb8na4X1g9+0kzTvvf+i67kTaA0XzOOqjdlXuv3lv/3qrvhpvvvi33LL436/6CtHdg2Nf9RWn//Xgt/Y9fWMdvOX1xX7+C9FUDy1Nc8O9+9y9wKQNW8hsym5V803wEurvqXOILQhnVfFSiGI1ZVPfJa06IEjPZ3FSFQAYAznnA4DMqeURfcC2pEekwUYyJSTTurRxjuXYyHJlpln1W81lCGEvw0Vz7fG1ah+oOOoSCXJmoZwysnWi1HSs5zCW2spXYZx9jaZPVAt23Tfu1j7FU90sAG0IUG2tniWnWyo6tyjSrrrCEHoO4lIDgunuqvk/38ayrcTPV9qmLs7qcVnI4iq7hNceN66uy3t6KsVQ3TDtee1ZcG79YTZ3WjbNS+y43V8ushB7oJC7q/rkW3fzocYjcZrlHyhbr+FqdJylwnXreJjuf1HXxFqGKjWvscZC+uWzqBo+Nayxy7/T+2Puv42djEO39Uld1yxZsJYT2pDubtrpORqN5nGMHJa27b7TI/Lihadl1GvFio0c3mmp1zB4L1oHs49VjCbUNuhmkm0m2/ZZohzepPFIqbZNJ11Lsqrq9bxpDqX3SOvRv63YYY2UmX9qS3+iGmW6i8SaY27h+ap12LLjt23z2t82vG2d2k4wzuBbPpysNG7uo99dr3yuT36g2pY6DxhU6oDEPt3FS4islK6Luvullww/6/AsLcN3NlVaXtMfel5bHbNZ4JbJpg9zHAYTmpmvvgW5O2Xza/z4/LQLhcjRSYn+vvOONLCu/9OI8uc3GxsaVV6spneRHgaLTwdA2m75bF+K2gNf50ohstqtKu1EH1/vUBh1o7dzEMG7mMZdHLZ/8ls2dMnw5sIRs3Lb22AX93mpk5rZeu9kY2yDH7Th9NN0DM3eAj0oftU0XoG/IyN83LVG3ytwv5A345H7urrnXZ7wEVy8AnC1fnx3rAXarXZtjFpTS5jDXo+N+p6+2rR+r77Xrb/TjRfk3ANZHwd0nAMq77d3Vffua2/Pxk4HRp+T7TJD1+wzKfh/SVw0sc/X4F5dvcQoLVvL4YTnCgzDFFaew4OfXRww+45wHPKQZP7884jFdUcnjh/mAY1rgHeFXOeHb8YxfXB5wSgvmNeKSEy5rQgwFf/LwDr+8PCCFIgQ/A4Kr+PmZ/Wm+nS749fWE4CvOa8Lb8YpfXd5wDGha8T7HDYOlSomMIePP3n0L77osySqLoLUEvDlc22Jc2zJK3J8mJmcBlhzx5nAFAFzXiBQq3j0fMKSMw7DiZ5c3OI4sN3JZUgN9bw9XPM0jt31JOA7rBiCc52EDnA7DijWHxrjpHWFeI8aUm9VXFzUKCI/Divk6NlCqluMxFpzn1K49DCt+OB9wHPkeXOeE02HGZR4QY0EKBb+6Dsg54PF0FdmQugHbSuSh3woVoc+ywHaOMM8JMTJBxWFcEDzhz//yOwDAw9sLSvFYl9gYIochbwDHfGWSFCbdcM2aDQDrEoVsw2G9RrjAchxqpc05IMaCvEbElFvsmVqj8zUxg+NBiTMqQiAsYkktc0A6rKDqUQvHkOrimq3BwDBllOyRVybrAFjmo3xIwFiQDitKDt0S64AQmRxILde6sG51SN9ZToJF1BUAhLGgzOxXFcaCco4Ix8yWVYCt0rLorUuAHwvqwrIVIHRNUi9Mj4RmzQ1DRbkGhKmgXALcUMWiTFxGYN/nuogOaSA+BqA+J2CoTYaDsmdh9+zhDqxt4EUiQy2hTqzRcAQSq52bClsUREIDAFuFRWqCViEb+ZC4jG+Wbo1eApOMSEwaVQfMnhd3h9IXBGK1bVZZB7hLAB5WLH91ZHf70BfS7jl2LU4v1rVD6VY8ccXHJJZJR+Lmz/c8vw8obwpb/xbPlh0JAwgXj/JYuOzsgLHCXVh2wy1sxeu78dQtgomlMyiKS/vA1j4rsQEATuqrUxUXc4LqtrpsRObFyqZSGWqx8lcWgdeQAxpqW1Q1K5pKfbwLKCOBRv6tYQDlobT5qfqf7pn77RcPmh3qQ4X74FGPtQEId5E5rVaXRN0KJdY+twgIjuwa51dxX19lXnk+52Ws/MW18kHiapzIuER3N3wKHXA3q5u4U4ZZXLzFKueaK7q4VR8JwzuPGqi7rYp1Ut25KQJe3J7DzJYvt/Jx8hyCEGYex3Jkl3YOjWCXadVmVGtiHdg1HsTu/a4AfnHNZTwsci72slqYg7hh1pHvj4ZdOGKpknh2zUqp59Sd9JYrrJd+hAuwPuKLU7hyOAEACRXpQKhM3cVV/3V6TwHEK8QVuZ+Hw8YluhqSYGdcYfdyI+wK213gNWSiubQ2q3GvPyziFpxl3nhs3Ic3Mh7q0itW2BaS4Xu+5v4tMiTqJh+uaO6fGlaibuJV2qvXk+vX21AAMvq06ibtiMfHhn/40q/RPjVrp4xXk58J6DqzFgSLu2qTGykGhBJAXtyz7SaQBXLaTvVycmYcS+/LxjWUdscMoLZW742FV/skbd5Yy0u/L2SOA/2YlVK5lRqw3WyQUBtPuwGgbba/N+kjoPtWGXbv4QW4vrdp8wnpVXmTv41EYIvC72n6qoHlKcz4D7/5v3GliLVG1NOWcfSPpnctb5MagFjk3nSLm4Kmv3/8NVbaWqf4Wo8/Prxrf3tXkSkAj9iQwmgq5IAHbJg/9VqbvKv409MPm7r2ycpc7OvRuqxUia3Lf1t7nd9srYn7OpsFEraMPk72uj376f68tc5tx/Fl/TZPJQ//WG+OVSvjbSfi2Y/JrfLvtXVPsPPj09PN9u7HZ5/2lgEAzdKgbLcv+rC7j6/d/1v92jPQ3hrLfducI9CPX47Vfnz2fflY227m+dFHL7n7TXjte6HnbrXpZt//gNrfQL8vL0g97qTX+n7TovsJ/f7cbwmRg/tuy94JAPSjl/PuVvvuWZ7rj+/Pn3vP7816Pq87m3Z86v34+HlswgM2a4ifAvHGGCS8nOtEbvNBjADwE/73U9clG08SU3bc5HnZ3o01fFOgGSvqFkFbnx6znDTe9M2he8bmn/Z7ZteX1ihYpN5s2stt4H9XXdzKA3nrftqx1XJ6o9EWvNke0/7eWlnuxqO145Xzt+q7+ftz0725YBe8DQSYirRf+4WxM9d/rI5duuzH62Pvtn0b93XdegFvwMCNb6Cjl9bOW2W8Vva+3F1fNhbMFw0weW6du9VPU+cngSLQzfpfHe1PfTl+JN9nW+c+Jf8rwPJ2/vs9/Wj7fkvj8Mn13b7qSy76u/SF6asGlu/zhP/5V/+6yF1EXEvcuLE+56HJWKgb7OALKlyTSFBXx0Nc8bwOza1zrRynGXzFaVjYRRLYSGqwpqLDFDNW1bU02ovAVuOvUbGTazGcl0WlG1jjS/XnSK7duzsG42qmx4lEikMYTdXaqDIB0Vdcl4QUC8tFVN8scEPMrZ/qdmrdV9cSmssat1P0CDXm0rELnlrdbrkLMgNnv8Z7Y7mTeFSVPdDjpXjkHMQN0AsjJbCuAbV4DOO6cVVrFjV1mZN/Y1JKfjNmpbuRhcAxpOfnEc5BynVNHoBdA82Cntj6B8Do23W3PtWmA4CyBsB19zmnC3bP2nW33FGLitnHfs77irIGOA/UzLICbP0CnFh6rPtZEGtszb7JI9TqUGe2EqqFcBP7qG5k+tvhpQsaoVkW9XrnqdHvE3G76xLgh8JWO4AtajL2ZOJonRJWieUSQLtGraFatosSmymxnGrddCo/kDX4irr75BLY6iXlU3VssSsOroneU2e4JGzc6azeHhF6jKGe07Gv0qerWE0PpTM8VnO9jrdq/6Xe77Yw1J1ogC2BQ2XisFi37mU2nlLjGM2cgY5HqtsYUXXrnT1LE3jivGYB5jJbG7n9Uu7qt5qANhkSKB0Xjq2r5hz6iqC4TSxiOw+8WPhqvGHL67l9IHSyr0jGYiDPv7gc+itbBZWW2K18bR1ri8/TcXVLt6IyCRvBzUwm1sZGrIBtB1/j+qrEtWnsIdDjAYXIyMac2RhFtWRqUuuQHnK5j4+1dPjVtWkLJY5L3O+qhGJyjc/Gegg061dbbomlm2Mx2RK4sew4/k+Jurzpk1pALIlZIz7z23Y4QiN72oyJ5iM0a1SzvlhdRmO1alYujdlUUjmgxxKKJbQGaljFkpp9SVLrPmmd+kwRj4/G6vnVdWujEsdJXLXGf+q4KlEeOcCugW08495iCScVWIFmAAAgAElEQVQWwNh/N0ukxZvSHgAbi1yzwhvisg2eV4unWsCcuQ69LRtCGHm1NaIq9Pp5Hpr7Sb2NltjKvou0Pc3KJs9rq5N6OZrfAlhn2ri3EL4AwmasXlj1XB+PTdqVYdvRxvEWuLXl7KbiBlPvyn1Rtx0na9G7V8ce3NONYwBukvfs675V3q4fX5Q+47ovruN3IX3Nbf9I+qqBZfIVf3x4z0CxRKzkkSWeEACL3aNb2KweYU0vLVdjzBh8bvqWWWJrprDilPhtaC12Wp6CTQBN3D6La6ICMQAvdt5VW1AlQ7StFnzesirY3xorBqDFBqpr6JRYUsU7wmFYW3ziWkJ7nyjw1TL3MYoKcvXcLf1F64pqxeSt9EcrSwHesPtXjpfkW5yiir7nEnrfBo9KaPGCt6w21iqkcYc2jvGWbqJ/oAbmAdyMtdNk40T3FiEb81mHwjIJ6O9vHQ/CevO9X6tHKR5J3GT1vtahbOLW7Hzax73ZuDbbtjz4puOn8YFtnEy8ko0Xuxd7qSBeYwYV0HtPqGPeHNMNCBuPZnUHtS67KdFiv3zvs90QsGVofhsDR+RAA9Ni2lgxG5tn6+Y/dgt8K3Qvv/fxfTBjUsVN1m4Y7J8pHcN9vJ5de9i83leUwcRJSr469rFt42fL07E38X6b+zqqizNAVDYxfTYWDwTe0BAX5JZvN3nthoNOeI0t5IaYuShA3hJq3VywOUDJq7QfG0IqT6DR4VYsoB6qiTqRETnQJGWHl+8NGh0QzaZLrKDkO5B2EDdlaQsBljRqE9enjbCL4d1N3sQH2hWplLnXQITrdQE3LOrqBq15zfkqba0aW2jJmsj87dDIXfT5tQtqCuziWqopw16/vwEOaKbUTX4nJEz7drgeh2jHow1aP1YNUzeckgZRr6sqs6wZN8e/6TdZ+ahOUWsPaRO2AEbrhQFZBf0+NjB24/fu3t4zRroJrS0KdPU+tvbB/Jb75nRPazf1XtS5K+rFQn4PTLSfdZunAVsLcslcdw+oWGBlv8GGfGhTzj3AtvkIm/6ZR2vTn/312vYbYwSYT8cr1/8myRqlW9o/c/t7gY//vlkuIAeBDcK9Vf6uje3Sj/X9M8bmbht/S+X/baWvGhR/JH3VwPJSEv7Zu5/iwzyiVIfTsKKQw3kesOSAx8Pc4vCua8RpXPB0ZY7/h2nGdUnN0ne+jhhSxirC2dOwYpLYvx8+HHCYVuTCi/4YeWF+GBd4B7x/nhpxxxALztcBx2lBJYfrnJBS2SyAva9Y14i8Brx5PKNWj0XExJUYJ8WCp8vYPu4pqRVPiUj4ezZKDN0QM57OE4iAYWBx728fz3g/D1iWiNNhxvdLQs4e45gbscn5PDYikBgr1iXyOkgsgcOQoSy2zlGLIWRw4hv77LLEjQWSyCGvASEWrHNEiBUhVpTiUHJATIVjAIeMLKy6ee3HfSBMhwXXywnDuCKvASV7jIcVMRa8e39inUkBnbq4901rj48tV57iytYKcGxkFnbgsnCs4Td/8ATnCN//+gEuUMvjHSGrQLoQQQwSt7kuXDYVBx8JVFmIvGRmXZ2mFcsSkdfYQGGIBVniN/MqbZfFuPOEYcyIseB8Zo0DqkDNHmniOM9xWvD8YYITgFiKb9Y/55lII18jXKyIQ0aWeFAXK948XnCZE67PE3wUIhu5tzX3NjTLnsQytt+OEFJFyQw8QqgoOWCeA+JY4HzFckkYDiuuzwO8sF3W0i2VMRXkOcIPpTHfksoA1R7D6CN/xZc5IE4Zy1NEPGQss8R2itWRxNrmxwLngDy7ZvUcTgvyGlCvbDl2Yr0MsWA9D3JP0eSJ4MBxm9LnMGVQ9ShyvU8F+cLXMfNtZAvKUEGLx/C4wHvC9fupgQOXarteLZJxyvCh9jYAHBdKjsdCFtzpuGL9MCC9mbE+DxKHyfnTw4JafZsfYSwoz6l9hb3E6JYPCW4SC6qOM4DDN1dcfpi4zEGs2MIgO54WXN+P7X7U5wB3zKjnCDcV1GvsH+7K1ztPqJfAdUnMab1GuKG0+UMqJTMV+IFjcVtMqVoPlX3WE7d3LPBJpaMIyNwWAKBLYOvwNXRwppZN6cv07RXzOTUpKj8VjlV+P7A116Gx88aHFVmOh9OK8n5AeLOgPKVu6T1q7DOD8jIHvj6StNlxTC50XoPn+iX2tml/B7Ymh8cVVNE3KJbQ7xUB7tBjdNvYFAd/WuX9w++AcMgol8jn59D7l7lN9BQRv1nYe0Hnub47QuX45sUDjxmYA9yBy4/yTqvFAecInDLitGJ9HuBShY+V39/XxPeqOI5PFnZh9yD368oeBH7iZ7V8SNxGsZq7VLkd59jZfCUGuD2riVGGmwPqw8p51KqdKtwoG3DnyHPjmdtLVxmP7BFPK8q7/ux9dpoK3JmfAUq1bzwEgjtHPqYSZmJdb6zGB47XxireCMW19rdzaskEuvwZwNcYIOGKAx5X0EW+T8UBU+X5WE0ZnjoztsqqJer1JgHGi9+wCW8kxIAmpYbq+maDl7ho8b5wq8TIT7X3OXD9TYrtaoJI1bIdTR2yeQFhtwV4Q6OB+exEqoffsSo/BYCtwGaDpVk8xVLaYlYVfFM/5rNrFvkWF66J0OJ0+7GeYdNOQpNzU+t983SQjJbl2QJtku9E86YgbCzQgIDq4raSI9XcL78D2+h5nHom4GWy2Hvz7x7AmvN9M+RGgbsLLJC6uYnhXgO66P3aNODv0u9icvS7FtT6GekP/83v6D/5x/8xvolnzDXiF8sDkqt4TFc8hBn//PwjHMKK5zzgFBd8vxzwo/EZS434fj7iGBeMIeP7+YifHt7j++WIU1xwKQlP64inZcQYMv6lx1/hLy9v8RBneEe4FgYUH5YJFQ5/dHyH98sB3lV8WCf8eHrCr64nOEf4bjzjh+WwaXepHoe44hgX/PmHb+Ed4WGYsdaAOTMT7GWN+PHpmd16HeGDtGUKeROj936eJH/Cj09PqOTwtIx4GGb87P0bnMYFb4YZf/X0gG+OF0RfBYjz2+ynDx/wtIxIoeCyJpwSsx8sNWDwBU/rgMUwpZ6GBXOOjYF2jBnP84DTyMfVwqjW2GuOeDte8bQOjfBnihnPy4DHccYPlwmnYcVSAk7DgvfXCd8cLphzxPfnA749XvDDZcJxWDGEgh8uE9Y14rvHZyy7+ooAdEJnw3yY5J6tEcETUij4cB0xxoJSHR7GBWPM+H9+/iOAHH76o3e4rhHXJWGIpREWAWguwh/OEwDgMC4gckixtA2JyzxgSHkD2sdxZbfeWDAvEeOQMS8Rh5E3QqKRvrlcBtTVYzotzaI3xIznC5c1XxIOJ14c5sybHAowlQDodFiwloDrNeF4nAEA85ywvB/hDxnH04xliY1wyDvauMwGAedapjcgNq+8saCAFg4YpxXLnFCLwzCtWM4DhuPS2FG9EBCV4pCXiGFaebMhFVD1GKcV68obEnll93ItOw0ZyzVhPKyYzwlxKA1U6yYHAKxz5E0bGQ8AWD4M8GNBGnMndlo9aPWIx9w2IrIQGVF1CKlbofMc4BwQ9frVIw7CDr16+MRPoh5ffxgZEH53bUyidQmIE5MtEXgTop4jg7/T2i3AQvbjU20WufqcEN8syO8H+NMKb6xs5X3qMimygPcPa/vg1gsvfMMpd2A8lC4D8/0A+maFV+C7irttcfCXgPp2ZeAxB/hjRn1KDHCuoQFJfigAWnhx78bawAOy68RBnkFXcwm+Bl4ATaXLoWhfFt/dd4fKeVcHOhZeqEaCO3N/6FAaqVCTOdHFrLjhpl9H5GNtC2J/9UwQ84a1hwGwyy45xPcB+ZsMt3qEZ4/8NiP+EJHflLaI8WfWu1XN4TIJmC2AWuFYm5hQB14g+tmhnCqT+jheAFKg5qYbz75Zj0BAHYldWsU6Gs5eyD86URAF1qVVQ4KSv5SDWBRH0XQk0TS+OOTHivFXATVSJ9pR99SVF86UOG8dgHAx5D2JF7T5yO608eywPrD7phet0TJ1opcyEnwGygBEIe8pE5P3hAsP5vrI2rQAL8iVvCcfVbOW9XKL0fRl7U4+lp6caH5S0zgOV/nGCmnPemKSnzqg6YbGZ2D+9svXPfHskB/4ej93DVxXe9spCmmQas3Kv/HM41MTu7E2Hc7AOrhwaPrIgAAZAR4lbQEDBWB4z3q0IDQt3TpsF+cb8h7R4FS9yyqEPIBocxoXaiU7anqboZPyWN1R7Qsc1w0wQVLTzhS36Zr4eNHlUO0usEoKpJrXZMYMxMesy7USEDkBhmodpuC2rrkyBi/IewhNQ5NdYFnHlfVTWbNSr1VLtM4fC4DaeQWw1m1Y2usKjyO5bR/adRacGTAJbK/RtJHOkeeXAprUzV7fUq9p87DuAJpJ2lftx8Y1eJ/PgvN7j5Me34PAVx6/W+635FgD93PKuZf+t3/yX/5TIvqHn3/lby89vv179O/9B//Zb73c/+V/+q//1vsGfOUWy+QL/nj8AW/DBec6YPQZBR6P4YqjXzBPEaPPeIojHsKM5Au+S8+YRQJk9BmjX3EIK76JZwDAQ5gxh4RTWHCMC6aw4pt0QSWP0Wd4VzHXiOAIgy/wjvBNuuAQVuQacIwrvhueAfCC/U26ilQJk/4ALEFyCgsOYcVy4oX0Ma7I5CVu0+M6RPzB9NylTXzG4AvGkI0rqheZE4dzGvDNcAEATCFjiivWh9AAbIXDY7oi+ooxZMwlwoPbN4UV0Vc8hRHHyMBS5R2Cr809tZLDQ5pxjWnDCjvGjENcMZeI5EtzUR1CwRQDHtKMFEo7P4UVgy84xJWvFymUU5oRXcXjcMXZMxvtm5FXCoe4IgUe72uOeBxmrDFzfUKSk8mznEbtlDmPA381rcuxAwPiUj0eBqnzdG35h1CQAgPgXH0D23vSnWNaUclhjBlz5vjeUdyPAVnzh4JDym08h1jY5Vr+JbBLd66+SaCsOeDhwO0OvmIMpcXXek94PFybbEgMtYFqjbd9GBes4or8MHE5Yyz44AjDkHEaFywSW0tgkqkYVC7EI3hqsiYax6puxNe1NFdvja0dmwXbYUzsBnsYGdySAGe1gM+xYhpWLFF0YqVtQ2SglyOvRrJIZ4xpRQh8DQAMKTdpkxhLk8OZg7qhUjtGxJb+MeU2p9c1omSP6bDwhrsjlCELiGYX5Gadl7YMwv6bU+iSIilIfC57H8RYUB55NXCY1vYMrDFgHHJzA6/VYRHr+XRYGlDUGF0dR4Ax1nRYcK4O47Ruzl0r2NIEjvXNqWKcOlBdBPhN04rFs7VPtWwBYHnjMRy4zOxr24ghciiBkA4ry52kipQKFrDlKgdCiGUjK5KjMALHgtWz9bdmz5auWFscsLoIZ4mxjWNhjVePZj1Xt1/17MgSY6vMvt4TssZHjwXFM3OwrnRUM1Y9FFaxHHlhZq6R42D9Mbe4XK8eFuDjdfXIgQF1JrFAirti9aL9qa6wyYBaAhAIReYNosjVjAySaRArkmOQXQMD6lV1LHVhGam7iQLI0Vhr1E3UAzkKMBZLeB0kblbbMUqbPIEiswYvb9kiRGmnZyignQKhJgajdRBvgJVBHjxQp4qaPMoocapFWWGdAGluZtXyE7vfOgKfL66Bj3KoPDbSH5c7KK6DuLFW1qts7q8DDxIlArxvjLDkufwycXk18d9lJFCS2EeJaa0JyKcvB5aU0MCuGwyAIEjbGSiXwcRYqtNL7CCtsa06yP3jOdB46wSUqAsxKeCUzSMSIFUHaufrIPGfvrfJAss6Oh4btcp5oExyfxLtWGF5rqgFjzxYpsVa0MRS1uJ9ZS7bvmtcJ0VCSRLfSm4DmMrUweSGOdVYFRVUtxhkc6xryfZrgA54rIvyC9dax3+TB3xxHJ9c93qdfSPmBViy5SqAM4DRxgnfYoXdu6ZaK6YFqC+Ap435dOa3ndoGWG7Gdzc+Ldl6XL/uZj50YHsX4O3bcqPP+8Svcrepk4Gl6e++7I+k32e309/F9FUDy19cHvCP/9m/L6QvvDAEOfggbnFraCQkLnR5AueBsrA7n2p3+VQ3x6i4Jmzup9LISRqLn2cpARDYzUiIRWj17P619MVFW3S0FwPvbLNFQN4gdnEii5b/Uxcc4vqkrictqUsKyb9KYiHuUm4V6n5naPwDsbsWCc2/obW3ItXNtcy2XduixwB54XZhZvLCEAd0zSwbSwP0l6LvHy/N11xvKp/7i2SEuD2xVAEBPx9rO74Z27ZTyW34WarcHhsTpO5k6B/C8Mxz593xLSCEDEq4oILh7eMhbkW/3AmIA9of+aCvXM/3se/8ebEMNApyHUZ90a/8Av1Vemjja6nr/epwGR7lY8vl7+nVf5DyXXF4kjnhCu+uXwLwPJAQZRgh7R15T184bL8qrjhc1MpRgcUBZ120AZgjwS8O71IXNicBFiAHX4D3kTbjdDbxc43QQsbzKq5Js/x71oWY624/3K62Pmhz1a/AEthoZvO4AlzicbMA0C7m3UcVBFzMoiAr6YkY1TRf9kCauQHnaeplFOBsySoAhJXrW9PUJ4DkzzrcDogLcB0npBnIaVKPTQBAWnpfyQMpA3mY2jMmYeZY0qFZILJZFE1XhzomfsUVIJU+34fFoUwJ1QGhACUCaeEFaRKXMvt6CLLAowAMsghNhcFIEqubLp4BzsOWDkKw553wGUnZ5IFRXNuYvIfLnmQgaiSk1RCpyAJT5wfINamJ9kxmHoM6pG08GvF8KWPkZ2UF6hiZHCXF9iy61TwO0ue2WJMF5IZURuaKWhq1HLtQFcnjbTiTIWrR+wf0hRw5NHIdvY4lQ3zPZ+pzBagpNEmLVr5ZaCvIUekRdfvjBR2Pa00ePsv4DKER7rDshGsWqEbeE3o5Cox8uwehE8foc1iBmtyWvCf2tlqiGb/ygtOS97SxDE76rG6Trr3D/QoM7+wK9fOSywBF18aq3SdtewXLU6icCNDmPrcZHViaZ7iNU9iW2axPBnDq7zATKLo2hmpx24ABA2ysRY68k3GjNmb76yCWombZM3Gjrc/BwUsZNbhWjwU1NaBJdtQo1ijqdb0AXMDmO3SXvEfa2QCkeZ43yQIx8620dewtdHvXTW+kX+4lG4e5eU/43TFT7l1wZ3/bOm4Q69wj2mm/ta9eLH/79KKO3bh8Zhs/lj4O9G5UaNd5t+7xV5Z+n8HuVw0sj8OCf+uPf4ZKDkvtXVECHAC8Y2yegHtyEpZ45Z4sx61nyc7vPZnNvbSXprB1A3hR9wupgd31pTp4h85bYP7+FGmFe5IEtg23pAduHfvUZ/4W8cy+PR9r+15kfS81sB+3W8caIc2Pb/f/Y2lPitLb1n/v+xNeuV6vC3jZH03t+jttfe3b5xwhmHkFYCNN0PLtrrv1XaE75xu2t18mPSe/2zd3d/5FPcRehs4BZXePSf/nbs+3/ffz3py8e3w/Jh+Zk4Ww/QrfexD0+P6Lbcs241LwMtl5Y499alr3q4bPmfev1fNJD/+nV7UNLnqljfcWHfuX9q1jt9pjV4ef+U6427Z7dX2sbZ+4iLJi9/ob4GfslkzER9vzsQdm367PmEv7bPYW3/x0fmwc9JG7s3DflHtrTvym6bU5tEu3Fut3+61l3+s7aYE2D93Nz4dJum9fpLgxeDd+3Gwjtct5rt083du5L+ROv1+W40w9u/t5L73y7N2dZx9p26uA4HOe9S9IXLdW8LF/tR37efhyXr469+6lT3onfTzP55T3G5X/u5T2C6jfs/RVA8vJr/hXH36Jx3DFSgG/Wh7gXcWbeMXkV/xs/gaDz3jOI97EC369nvA2XVDJ4dfLCYewYvQZP6wHfDc840OecAgr1hrwIY845wGDz/jD6QN+vRxxCCuiL7iUhOAI75YJlTx+PD3hUhK7pOaBYyznIwDg7XDFcx42ciPsMsuusH9xfgsAeEgzlhqxiPvgXCK+GS8YxPSgbZlCRqYuX/J+YavHNSf8weGpteEYF/zl8xs8DAsOccUP1wOOacEQCp7XocVYfjc941oSoq94Xgec0gIPajGWH9Zxw/B6SgvmEpvrZwoF5zXhmFbMOTaWWeeoudw+DjMumd1nnSMc4orzOuCYFryfJ5zSgqWGdvxhmDGXiHfXCd9OF7ybJ0wxYwwZ7+YJ8xrx3emMa44YQmkyKepOqm6h3gGnYUHwleM/xYXyaR4xRHZPfBgWJF/wZ7/+FgDw07cfsJaAyxoxRO6LurwqE/DzPHD/hLlV63aOcJ4HHAZ2hXy+DuzGmTJKZdmVeY0YU27/EjlEGTMih+frgFI8jhO733pxn326joi+4jInnA4zavVYS2D3YHEzLQKST+OCJQfMa8Q0rPAOmHPA+XlCTBnHacZaQo+xFNdXoLP5Wlka72tjA1a3z5xDY2kdh4xljSjFYZpWXM4jpsPCcZjkJMaS27gsoZE9pVRQK8eo6vc4yzWl8HimVDDPEdO04noZEFNpMaDaFkBkaKpDjLUxqF7OA0IsGE2MZcnswjtOa4thZW8H3rqO0iaVtnHgOE8lo0rDuiGmcg4o4vZ5+cCES8e3PcYyr6FJ4wDgONM5gqpDOnQmiFo9WzNjQZV4y3xNTKTzYUQ6Lv8/e+8Xcsu27Af9aozR3XPO71trr73PPuf+OTckuYhgIoIxmgdFJCCKBPRFEgJBRAn4B8Hgi6Loiw+KIEJQiSgakFyMj74o8Q/4IIKCGAkxRBBzL8Hcc87ee63vm3N29xijfKiq0dU9e35rrZ19PHcdzoC1vjm7x58aNUb3rBpV9SvEuCB1TueuWZlDZJQpSH8qceVJYiy7Q26eG0GfYQKQFZjGUtm01C0M8CUhvhLTXM0Bqc+YLx3iUCSVTCf8tzUrOYjXR19QrkmAjeaAMAhAE6kXRFALfxmjxHIesoxLEPAignh+xAoukmKmzgGYgrijMgm4yyWJ9WQQ4B4aShOkDYDIUtngm05iOXv1HJmCuM09ZAHXAZqZlJ4j8CqDpwCaCPyYQe+SxHeSTvYqbUi9RbjnxYsDkLGnIHMxIA8DPfFeJwGgKSyxlk7W476qy6vUt9hM3ycCEK7iqsoBEtt5JdQDhC7zRKkARQKNhHqqSG+jIL52zjrDaOAm3Gk/PSOMYXUPJO6rYSLEkTRm0lxhoXGbQqu5wtZe+iMARV1hg1p981G8EJqlVxPd14GXVB1VXFythEkYVXuJBTW3SJD0ZRbl2kvcn8Va1h4SKxqBdCHMr769VBdG6RdYx/5RlThFS6kRRloAZvQ0sNFs7qNhceWNV+Vd4uaq6a3q1UB0TMcLQHomVMEjRI1LfKdHTW0WfwhNLd40yD4is5R20oe1s5jClsIkalwooVkAzaJsKWQsltBiJgGNbUxoVubaoQnWZpFtcZUuRtBbH5sFUfeKd+mkiiXNjXOfFYL0T1z6bd5FG7dVD+6za7HM2LdY6vPY5gOn+PLSt6fnJcvb6tCFlnk2fdC8r5wXVkvrs3WR9TR6/t5V5nkdz+ho9HT5PegtzzdjrjpfX9ueF773oMXz68WDlvfU+UX5qZdPGrzn8W/9Zf4D//6fwHnuMOeIocsNwCWXgEFj2zpVPjrN7WiAKJajMUaJkzKAEh+/lWvAqMiuBmZiqKcpSezbOHarvIjzLPkXmUU4TclivvTEjRQYRQVcQIRbE/6sn3HsmhXCwE94Yw010JUYK6ZRzglCrKhF4sgMfbYfBCBFXIWXVAvzHJvFijS/ohCpyJgqPBsdLf+iWVVraHwLobY0DKyKEhE3FNFA3JAsQxS4+OjivriIS7K5LMe0IMUacmnqiyBqjkncm7PjB6Gltmg5K+fQBFJmAhgLQApBBNdKGF6NgiJ77kCBV3UsFsusRKEXIbeq+yfXJSejoSICEpNmQC/yg0ZNWLY4NHHVlreo9R2Ikc2VmhVRVVH3YlcbqIwJtJbqQaRANJTVkCrqrO7akZEOM2qJKNco9Jr7q6VXABYUWIZzj6X2gqZuySdJUVMzzIboyKij5LCUNeTGn+b6nETpsFyWouXQIkibMB4V6EXRIm2MFYKo5VUEFh9KRhPgaVCwmpZXkJ0LuvFX6Frlp7SioC7NB9VAaciN7eigoyJSGjorrI2TRgK3HJo8ra+DaXHTZogyNCrS6mg+wLoOhvRq8+3q4vNrtBM0B6b2V137UwYrsiUCL2isxrdzWugyIJ05LPW84GbAOxqn1wSevKxnc9cHxP3eUF9NWrH+TEkzAbC578szREVzbLL2r/SICyw7AU++00l5V926BQZd44J2WbTtUAUMKDHQ6edjWVAsGYJ+aQIvCV3ebV7yc9ZlL+mYLSyBnVJqcYg2HxOGMrWYMTCWHJ600EqVhBY43qZlf1qIQWvTCRonP2Q05FEvhBoqc5a4SPmr/duaVxKFu9P9Mmsfwa2n7mNSl2iag8yPoPk60fqlMQARMh8LgQiQMRThsu0jmBKjfJ6XnKMGpGSxo3afO1GOa18FqEn3JvcV4ez9Rj6ucC/5TQFVGFzcK02Liz/b3mS0vc8aD2m5MC3/aTukqGiKHZPrG1jnR2VVZI8VwaHm2p4il0ey5QiFKPwSM4sWo2iANGHGai83Rau5euu6OEXHFBpz3w1KY+2WeFtTTqmIO70dDrTnCFieDY9qWhf9aKtokXdLVeXQDLV7ilU7RCmuvrXR+m0edxQzD2wErBUj0t/wXUcHx/9232IJ9jwyXONdBwB7/i3UyLpy3z19jUbe8G1b9pQy5fc9J4QV8NBLhTefd/j701YIjR9/+d/8Uz9zgJvXr3+N/+Af+ue/837/+7/wL//M5wZ84hbLIWb82uPXmGrEVFIDugGArH+9eysA4AGLwuOesG1uSl/KaT9fYnNjPV5XbqH3XGl9sRF8XshV3wDKMN2226EBEHfGeYg3tKOfFxoO0v6l3Jp77vdWvIutb+vTfey5tX5oaa7EJznhDsqb9uOC5c8NITkAACAASURBVH3U9/mD+nzf+Eu+RHGY7jr5BdjmhpRri3Lpr98rVt/n6gQAHO7TZutpBwZ7xZBQ79Ehuf1UOOiX+YTAqIH1sIBXNOzN654rLhxSPxHAg/ustMW0DXyxBgx0y97b7pl7rstbHm5/f+xwxLcmoCF+trHbdVdz7xfY/+APC30v8YZU2QgPc9uocnDgpRHH51hX19vE3DW2NDJ2MOJ4RKbvGI/izmKZgnlDKwNH9wwRVr8GdMwLD3r9m/iOpOM7tlyXEEvGzh5t10K5v8eA9Ro4QKbGK1NoIy/TM2XMuqgkdG8YwGblBFY5DXlY9i0POma/eRY9DxKDmZe+2gbBQjvpPtwI0SLgOmLdvFZr6dbPLB8t/tvoCdSUGwBNwZL5Cb081OWQYxP7Z9ZRey6ERiXIUiXwcr8d1jCWlBn2WWkmPShqljMV5snTwACBVICl5QDKKdYtHt4JxE3xZJk/K7pYS+vAYr1kwqKQWVz5GN4vCL9UTCm34pRDBLfssxvElBjHS/I8BMQyjXUTf9DVXlPudRIvninaB2/qt9gEEvRW+2v92lmJtbNDRm/Zs/7npf9GI9Biba2E0dE9L3Wjn6N//fl5bnz/id1y+bHLco029feK4Qp4urf1X7LmId/eW4/1wqba7ZM2f7Ue7fPYxmN9gYZbjfODxv/YrX+Pn9+5K+wH9vk3Vf8X5f+X8kkrloVDS+Uxl4gKkryUimowVfkVLqoseqVxUrdMU+yi3jehzbtTWkoLS7nhlbJSqblMSjtxecxq+TRlbCtE271JUzLEUFusJWvf5rrpYzC9wmqKF6CWQxW+mpKsyJoWixiC5S30ltBFYLU6Njdr5wV/QWzEEpuodczVcU8xMDRLr8RZXRvT92Xf7bO4JmreuEIwq+tenCYzVoKvTxrfaFILn1emShZtKcTS+NmUdK9ouLVoPwtOSWvrrOM0GrR9sy7u+XqwoFqCIXkxjT5arHstZ6bJyv5XwsgNqtDaWHrPUlo0l0cn8DUe8v73VhyYjykQzXLK0Nx4oeVlXBZChdtKzdLpgbC28VlbC2pzj3TruVo/VtrMckSQvIkGHKT8bZY7BbVaXfeLCiyCjwFKGC8I61Ntu26CYueUBUdTq+vzu23mvdprRehky+dnfAcasJjRzDYno8WPsZ0HlDemmFU/aSyWQuNNsyLe37er+dGiAK4UQavCKryaVRqujnPvMj4T00qRahadls/N3eNNnKHlBfX9M1r+vDa2rYtZXI2fDujL85V0P6wUQOXNyuIErFxO23VvdbB94+TM1VnInlWA1gL41nK0cl+ze0kVA5PUXb32fgpooGBN2Hd8MoC11TrBbeHtetqzgqX+ig87Am97JIw8R6tve4sSuSCNCr8t9YRZdfRvodVhwscWW0/jTdtuvF7nG9ROYLEkKv3s7t+4UdoXAGYN256BCWIrWp2XLFZC+yb/oaeTtmNjvX5bmuFeCUa7gdT4s5jte9MD9Lj9s537ah4vKBAvKi977/XNmDf1/b507fcsdzd9+PfdHbq2Z3MfpHxt6brpdEPzhs6bd90dul4c/4XywQrkh9b72LqNjlsAo9+x5b7t4JMvn7Ri+Vm64B/6/l9CAWGsHWaOKBwQ9a1WOKCCMFex5M0OhtMD/HhlrWCxdHrr39bymDkikbgtTlVSdxhIUABj5tDSTniLZOWgKTJEmZ2rpBtJqgznunbRWVlCN28L68PGA24toJbmxCy5/vMWMMiP4+ez5ZPQKQp8BWEqi6V0azHds95ur/mx/fXqjqg93Z7G1VhY978HPLRnSd5aq/esz3YfwA3YkaX78JZbWR/lORbX3MwBieouoJSNuwVF2u7FPf7eA3raFn/A4RXyqvNgvqXr3nh7z4f/TdsDd1rRsvm+95v3nt/oF/sHcDPP1u/OtXvlJQArG8P3+9I9OSRaW/+3ngAvzWNvXi/Nxe69b673Dq78vRfpw8tyz719+SFAXS8BfS11lkMGO4iSNsv3rcX3fl+3NG3BkvbAwvbmeuvBse5n38vDf9lqU04q3UrgO+O/j66tHGzyv5fl27pamx2NZUXy6ssNGZtOb8vuutzdWPbe+/D1edFK/qFlexC0/dzqbT6/74Xm1/jO+t70d7evnfttcT/s3bfb54fc24xrBz4rEKmX6L733Z7hDYtebLvDI/qQPbrDo7vK6IeUD6z3wSvzHStpd8WGj5jz3gHBbduP33sfrLg2t5Bvub9/Ub6T8kkrlj+eHvDnf/MP4JvLAZUJhy6jMjDOHeY54jAIiEoXSwMyOV97EEly+3FOLUZwGpPkadO4t77PAvBSA949H9D3GTkHcA0I6sLWdeJSeLn04kpJQEoF49g1V81pSg1sZBVjmSW+83iawAzMU0KIS3L3FCsu166BfqRUWhv/A5o0R1/XFVwukhk5at2HhyumOWGeEg7HCeO1E0VI8w4yE8ZLh6DugiFUlKyxkWopSl1Z4u5IQEFCkvx9YBL+6XghlhaDySzWtxAl0XxIVeIqK61iKWNfUBTgo84+Pq8iDRn52iENGSUH1BzQHTJiqhife4nXszgmhrgMqoXKXCPLFMXipWlfmCX/ncVm1lFyJ5y+J3lMz98cJcayd3WmuDptjYOse9E4SC7U8uZJ37JAaZD9Y/OpOSD2FWWMCF2R+EfASXSMNBSkrmA8d3KrEpADwiGDS0AaBEiluUeWIDwI3EBS+BqBpHPQ+DBKjMPjiGnsUC4RlFh4wjKGxZG22EelyebVoPp7oZtiRUgSJ8tTAPVV1vAaEY+ljSHW1tAsP6EvqGNscZPN0mc//hZDpXuSp7CAtByLtKkk/fVV2jKAXp4/dm554UFyEmLUoKfIbd7VYgjNMmUWQaOJAeqrCKBjXOIQJzVJWDwbk8Q3zgHxlQAC5W/6RTroqrbR74ERjhkgoD6n1fUWY2lK0bGAnxPC4yz0mnsggPAwo2aNu8wkPLq417m5b17DYkEtS/v0ekJ+2y+xeZ26SQZGPGaUt+rvnCpojAKAM4WlnrnJscYUWizcUMV10njV1SUm1ixgfQUlln1qFlWzmjorIWWNz0sVuLpnUF1U6RrFpdXiHImbi6LFIsY3E8o5LS6gXZW99dStYyEB0EMG3nUSj3gowLsOeDWDn5MIxAzhg9EJyBxtXYiXfQksY/ZV1sEs1AyJseur1D+Wdh2AzCc7S2zveOjjTwczE8lYdMyyBzappqC5POk5Ap/N8jxafKRZZDVmknJoPIXO1eLBUUniTocqwEvPneyPxIDFLZvVXOMtaQrNpdjAiqB5R+k5iWst8eo5pGuUtVFrP1kMIaO5+IYxoJ6KxDDqP+6q9r3sDbpE8EH3pAE6nTLobYdvW3ioCJqDiDt2gEpKeyepsAywqeUJBcDHInzUGFCw7FUUkr3FugZBX8DeKuwt27oX+DGDLrG9R/lQxErvU2ppjK/RbnGylEn60LjhMIYlBo8g7sYtFlSfqd49rwZCNZPEseq8wFjiX1n6QSbELDksw3WZR0tLY7ywfQloaiBqc7BTD8po6apkM7iUW0ans+KCqQHbNKsjYQE42sRYeh7AdVMtZrZtBLuJ1gfbK9afMbgYyxsr8fawgVx/Rr6zjq/W38VIN4+ETa7NpugF443rb3vI4ft0XiNbC72nwceWeuVv7/zrrrn3jgJ/c0azWQ+7tj3nMp59uDL6symfhFX1W5ZPGrznl37fF/zH//N/EAUBuUZcivxYDDEjgPFcRDgSi2VFrrFZBq+lE0sUGJnF+japZTOAcS0JmSMCGKc04VrSjTXLUpwc4oypyGfpK7d73jrlrWApVCQqOKsLpgEFARClDIRIFUnjIa2tJX+3fkoNKxdgUitoFwqepgFdLOhDwSV3GJLwZSypWSQOaV4sugpw5GktNaysF2ad89fmKtbZrfXMgyYVRS3tlP65BnQ6Z2uTQl31lWtofImKTDqViFJDQ3X11jNzR7Zk9IC4MW9TwBQ3ZhcLCLhBep1LQAySyqWLdWXRW9yXVQFwhwalEmJgVAayWnLtfggVucTWj79uVqViyrryKZAklp8UhdcOSmxP+D1pT7Lxz+oCaIiuIVQknY+MCXjQKHNR9pauFRhSCSt3aADN1dkDOdk1bzlixg3Yk7+3AonStbQctUsbbn0aYq19tj6M3nmOCIGbezcARaoFUlrWYO2mrcIYo6GzxrjM1+bu3cbtc86CgpF0D3nXbivMhKL7x9yu5YYqN2FxD61FkGc9Aq3dKzk2t2BSGuywSMZVa7mLp/ZpZYoe6lhdrlhcqPXgB5BDh+aOHuTQwQ5vGumq9FAA2ASsuhxOtT1gGCMqcJpLdpsE0L63w60i7tPBrRcXOwSpDc22PYOmoCroCY9yyGK8MvAn6svibm18NaAoVQSoqwKwZHGFxIuyyFgOQkwQZBW2TYE36chci015JKzrecAoo90OGuDuE6+EXX9QIIyFi53khRemXCZV9FQh8K6OzYU4oLkXN0XGlAinbBjIjykZBp7TXDEdem1zEXcALU2Z8G6+1k9yOZErLe7GnleRFzAgm2ZdBPimIKj7r9CGlgu4Dt9e7pE8lgvPyfYCL+A2TDpPsxCroGt5ej1qK+teWYHzQPoke0RM4dDr1qchtxqvvastHNsajxzqKYJuAZ1DVeRfG5Pc/jC6KGNRoHTeHml0yZe5VixYlUhDzG20Ka9CWeefbXzTj2QHKzrPxj9e+NR4iWVcKzeG4O3y21zd30UhWvPzpbKr8Hjl8c5Y9zvE2q14r/3e9W2xe3b/zpgfagneKn4fZO38QB5+G35/TPmL/97vAPCeV7/Gf/cf/Oe+837/u//hX/mZzw34xC2WiSo+786YOWKuEcc4NQUFAFIQAdzcVsea0KmiN4SCQLW5owaqeHB9DzFhVrfUY5wxaNoP73Z6YFFuhphbXKcJ+z0XBPDatdO9Beye3fcKpBcEt+5o2+/ZlGbnmmsKdCBGChUBkk7CPptiAohSnPXXoAtlmQes/7USaUqMv+aVNq8UA0AfSlMwgUUJ66LUMddlExoHp3BZypPMoSn8ptgOKTfFcwvQxI6+bdypp9cDGR16QRg4JFnnGGJzZbZ2vg05Xnhl2tyTGcAcuCnE9q4saVF0jTZ7h5oyyCwpSIwnMg9W5T+gi+s5kVOq/Zzss9UlyMFA0vbZBHTtX5SONXiRuW02xTIu62MKusXMAmrJDQEplqbcrBRTnXcJoSnYplgb/UJv1XWoiFo3x8W93Obc1pS1L/fMmJLp94gpXzHU1e/X1k0SgCJEr5VufwiwKLgWvywKigFA7bleMi9xylHRmU1ptTjipjwGRkri7eDr2txMYTWU6ZSWPeNTyViKkhAWxRIQ1GJRLOvKE4Ljghxtc7PUMjWsFWWbfxtLlXEwrTwdjGapp89ncEFLW4lM/3IIqz1Coba0C4HEQ8H2l+1zYJG3CkSB9YolMzUvjTY2E2oUJGU2r4rEqMQN3RjAGjRHFckVkjIBiBvlti6o0e06tJ4pyk5SawjPOgbHhc6Vu6dXSE3pctazpsiqEktpQa1saKP6AHCT4CFAOIHBtKbLrP5MSlObE8uBQgAajJK1C/oCwUJTUxQ9QBAgm1sVSVaFnSsvilZrA42dhijD1kUloYMhaLNFldRAQpvOtwagbgGZPqJQVEVV+dvO5plX4yC7/WLbw4Ev+ZhKELd7N4cFtLnuFIpKG8XSUmJ4xdLYTE6x9BZNU3DdPid283AHFBQWZbgpaO691BT9sOgFTbFV3u9Z1IqzSC4vuYWtYF6PFzbX3qfAbPixmoTb3yuU1W2fFft9+2r+9dD+w3rt7tG4d39Tb1dxte97VtB7/b6n7CqYvt9Nf3cV0nvlQ+q9b013yrdRNH9RvvvySSuWQ5jxe4ffxtflhJkjDjSjIGCsHc61xy91E661Q6SKmSOGMONcBlQQTmEShZQjOip4Vw44hQmjWhqHkNGRxBB+NUsOy20M5ilOKBzwVAapy4QhZDyVAceoaUQ2KLM23lgTKge8Pl4wc1SltzTBOKLiqQw6VsAQZswcV3GGwKI8S58dKgiJCioHHKPMZ6wJr9MVb/OhXbfynAekUJBrbHknpY/aYj993OZUExJVBP1V9O2ElkURHWtERxWX0mE4LAiUhQkdVYw1YQhZ5q75QY9xxqV0SFTxkEZ8NZ3wKo2SJxQkuUSp4l0eGo1ewTO6TAF5NlAeJ3iasgoIiFPlgF9/9QwA+GY+iEJOtSm019w1HgWqOOghg4FDeRoOMbfrpzQh19is3ZUJfSyr/oB1vOghzc1ibmufOcgBQI3oYxZrOxgpFEx1sT7LYUVt1vg+SF1AFPzHbsS1JDzNA/pY2pgV68MMsxZvLfTWj/HFyqR5RBNVXEvCIWZci+QNDcQtNysRo4+l5Tu1McaSmmU/Ul2NO7v9dUySK9XotTaVCV0wy+yy/4Yohw9LHHNtXgCXLHwxRTfZfnbxx1sQMO9VYPSZVXguEa+HKyoTnqZhZRG3nK/WbogZRIzz3N8cWvi9dExzyy17yd0KIfmhm1o+W1OUrzm1399eraFTiatYb2t/7GZc5g65BhxSXh1OHNOMp7lXvlZMJUq+WPUymN0hBoBVDlRfz/YFY+1F4L0P/CGI8cEODMwzweraupgVfsqppY0yoDXrz+exHUts4GvW9jqn5nHgQdpGvd6njMvUtZyzNt9OUY/tQCfr2P7wIOnhRVGBPMWKOcfVoUTVw6NcJE2W52ep8vyZ54GAw63jZK19+17DCjTOt2fdu1NOmmP3NubW+G7PiR3aAXAHRwKSF0Jte94O2PyYhakdypglHUDzOLCDmpzjykPAPlvKL/MQaOmoNnXsIEW8JaTudqy19wSat8Jk6Zy+RYmRW8iIhcU0b5gcBViuyj3Lj9s8IqK411taLlNq2kEL5F7rk7ilGLuJdWaS8JMalu92iLY9sLGDDzsMMGWQ0TwJ2iGIHoQ2zwPb2w4MbhX33MDwqCmhrDwA0A5e7ICFffoldwhlYywmTtwqFzaXPYVb57CvsG2UyNU93CqOd5TThnCMnftGy55S6y3zvu0LytqK7r15bOezc/1GEX1J6fTf36PY3m37Utnj/Z3yXgVxj2+fVNmeXPx8lU9asXybj/ivf/L7m/Cd3Q+jCMeL4A5gZeHaCojeldLamODbhdKsl1sAG3/NBCQvHG0BUgwExn60t3RY75WpCYTA8hz5evbM37g96g+Rp2HKEZ3GeppVTCxV9YY+/1mEjYXn5h7qi7fkAIuQvFj41m1iEMRaE1g8Sqy3BHnhaA3qQysrlC9yuEk3Qu/q4M3RFvRH9K/k7wPASlgx68Dizrmg1ZrQ468LL7wla42Sa26Hfr7bYkJSjOt2JmzZPW/lWs3f0VXK0k+thFpEmDHLVUPQpcUScgOCUZf7vr4XfprgwGpRsv3g56g/ds1V0yk5W1Aav14rq/CGZ6vfFidINflD17AJSoDSgBskWm8JXOYuA+zxxt+zz7+pFqSG2rslUr+v0IJfKIauy0Xj1fzcvZXL/norlLeOeZdIa5+DxBpioacVs7C5Oa9cWzd0s+/fYgDZjd0WeiPkendMx5+VUONdTO/NjXhpsxHEfqRxo8um0Dp7brhm8fNjOovrauxt2xVDNve2SJheIPJCJ9x9J9C1+NHNeA211Y1jljTLdWlWPUOgPVs8854wbtdNwLZ96uVXXQeLvSPPDzcv9ttzK1Q718r1i9ld90K+r+d4WHxOQgCV5Z3HJESbG2015GDdG8XceL9l4bJYXYvuORPa2XAE3HvBl+p44a185Oe2KVv9aSVPVyBs3JrDZv/4tWkIxVtnAV0z3/lq/fza3VGEyPoAVm7A7TeDsM75as8rljHu6VG2R9te3NB51zLombh53vzrs421N7/3lA9Sgj6gj+3cG03b9nv9+ed2+071bfy+2KHhpryn7i5972l7U+db8PvFcX9R7hYi+k8A/BEAf4OZ//af1jiftGKZOeCce3w1njCXiMd+RKkBz3OPy5zw5njFmBOGlHGeO7zuR/z28yOIGJ8drjjPXTtlf3cdcBomXKYOMTAe+gmDWp9+65vP8HgYMeXYToYrA68Gcb398eWAQU+yh5TxzeWA18crChOergOGLi8KoZ40P18OmEvEFw8CGvOs41rM36mb8fXlgKwn9UOXkUuQU2L3RNm4x27GN5cDWK9d54QfvHrCu3HAderw+jDieezFQtHldtr9zfOxKVNyqi3WJYtpG7rc5gsAz9ceKdbmqjmXiGM/4zJ1LZbSlMBxTkix4vnSoetKsyacrxHHYcbzZcBxmHEZe4TAmKaIvi+YRgEyenUa8e484HQQoKWcIx6OI05dxlfvTui6oqfeiyttjGLxikrfZexQKyGl2lz2hj5jzsLH69ijVsIPvvcWgRh/4yevEVNZ1ZnGrilTRIzjcQJDgJkAicWz9CfDMGOaEkJgnA4jcomYNLYx54i+z5imhL7PGK8CwmMKHxFwOkzoUsG7syS7rJVQSsThMKOUgMfTiKfzgBBYAJ1KaOOb2+Z47RBTQd8XjFexzMVU8IMv3uLpOuB8HhBTQdfJ6XmtAUVPkUOsqCW0E3SL+auq2BrAVdfnZgkwsKoYK66XHg+nEefzgNSV1rYWAXfq+4xJ689TAoWKMscW39fAq5LQlueEfpgxXjscjhPmKbWYwJhqs2Ykcz+t1K6dHkZMU8I8JlGAIyN1GSlVXK+dxhMGxFQb/0qO7ZerVyv7dBWQHQOpAoDYKZhWIaEjBzx+dgEBeKcAUML3ijympvxQYBweJsRYcX4emsIegii/dQ4t9vDwOOLybsDDZ1dcnvumYALA8dVV40UlZrPrZwF8UsEhHWYQAdO5Qxwk8Er6ln365ssnvH17bIBSqcvyLOn+fnp3EPCuviJfE7rThHzt0B1n5Ck2gZAZCH0R69AUkU6yT2MqyNcO8SjxplyhYFhAPGbEVDBdOwQ7JNE0QgbIRVFAs+JJ9tV8TfKs5IDuJN4g8zWhO83IY2rWEEqlgVFRYDx++Q6X84CioEzdkBEC4/puQNAcr3UWuo5fXHH++ohwLBgOMy5vDzi8umJ87pura/dKvD1IFf15TCvF3wC2qvE7MFJfMF86hK6CKxrgVhgKOAd0x3l1aFPmKABZ+uylQ25WqWoxeoWQDhnQAxcusl9tr/IUQfpMwEC/nnocfnBGKQF5ig2oi4K8N0oJqHNAGgRQLQ0ZROLaPc8CgjZfOsRDxuEw4/LcI6Yq/6I8UykVVO0fUQB94jGDAZRRxky99Ds99whdAQVZA9lHVcboq1zrKooCkFlsLADp9yGjZmrgYNQXpF5+P+drknmcE9Ipo4xR+syEw8OEy9eHFySLl0s4FAHTYggQlR5chMgoz0lAv7Kur8Z0W2xuPGVRdHNA6HSvKv3hmAGGgLrpYV4DqAMWsDNTDgohPc7IV32/KMhbneMKvIciS1tAQL4MPE1Bg0jdgrkBUemPfccSH2speHIABgEfav1HFqCmTjVVBVqiYxbLJENcsDNJvaECo2mfWLSEvi5jOECvFodqQFQk86aZJGUQoDHD2g9jnYLIlN6s9RXEp93Dck0AgVwKI6+cVWqgTLvWQFOMlQ5RsrVChACsNaVYU/bcUQJXhzF2sGKf4Q4GwjIuFaAaeE9cK+MrPui67Ro6N302WuyQgHb6q8tBzvsskk1s1TlvjeprJmz4Yt9/jpTJn5Fi/J8C+NMA/uxPc5BPGrzni7/t+/z3/0f/OL43nHEpHX50fUAgxhfDGcc44689v2mud6/6K350ecSb4QIA+Mn1hCFlDDHj7XjAD07v8KPLIx77EbkGvJsGPF0HHPsZv/zwDr99ecAhZXShYCwJfSj46npEqYRfefUOb0f5obrmhC9Pz/jJ5YRSA94cL02BJZK8mWNOTXH96+9eIRLj0M+ihGRx25umhM8eL80KeR57DF1Gr/GRgcTl6DJJ39Oc8NmDzM1cuL765gHDYcZpmPDN07EpLc/XvrkqffHquSnTl6nDaZgQVcHsYhVlVJU3ADj0M+YSGxBMjLUpFjnHFWjL0GVRak9XnKeujdmn3FB6n84HnA4TphxxGmY8XQZ89nDBeepwPg949XDF83lAP2R0seDp+YBaCI+vBPHWg8WUEpqLklmwDscJBuRiSss4iqJbK+F0mJBixY/++mcAA69+8CTKyCQowVwJgyp1Fq92PQvQT1IhJyVRcE3QjAo0kp87UFcR+9JcwvIU0R0y5jGhG/Iqp2atAeWpAyohPMxNEUmpYHzuBZ3xnBBfzeBKqJOgywY7lVe3q/40Ic8J5RqRjuLmmKcIvO3Ah4r0MAtarlk8gsSWASL8h1RF2CvUhPwG5jNGAT7JoSUKD8cs6LOFEE4Z9V0HeswiqACAxa6VoOiuWf4OItzEQ2mn+3VUacGQOg8FfE6CgvrcNUGOUhV0WU08z2OEIZySCSHvOnBfEY7CZ56DALTMBH4o8gNvQpeCujQBh7CguapihiksiJ+GeEoQxMm+InzVidHuy0n4QwDGIOiujCaQ0SWCKlAfyzKWCRpdbXMP54j62YzwVSd1TcADEL9JqB03IIswBdSH0n6ASROn80MRpEpiERS17+6riPlNEZTUS1yhVKbngPlzjbWdCHwqCE9JkB5HRfl0x8akAqmgZQZwxwgjoZwqwlWRQAMkbg5AuAjqaT1VGTewxMSRopkaqmbHCJeAMBPKQ20ANOGsStapIpwD6qE2qxZNsie5Y6AAw08i8pFRDhLTFq8CEDO/ri2Je+1EmEpPhPmNXE9nwvQZo3tLyA+sQiOQntU6pSwovYxluR25Y8SLrGVNQlMcCfnECAp8UqMIyvEq+RTTWYVnfRxrz6jWJ4RmQ8WsUQXA6Nopb9Oz0lplTkETunOSPuZHxvATQdOsPTf0TGKAZqD2QO0YhSoKKAAAIABJREFU6UwoAyNdRFAPWedCQH6Q+cUrkE9yj4r8LYOCswThCxWgDsoPSP9UgDjKPPMDEKaFxjDL3MoAiRWM1gea1TdoFEc5yHw5SD2Oci+Oui6DfM8nIJ31uwLddM/A+AbfuqSL9MsExAkNuIaK8kTHiVc0kB/LIpbOcq92QitHtDmks/RZu0UxILXK2j5tFjrImvRvGfkoe7Imoa12WFBSda9ykr7TVdbGaOQgc4DyzJBFAalTdV1sP8cRqxhOqjYXib2tndxLF0bV2OCQGbWTe/HKKAd9t5dFiQmT7u9EID3F5rjEO1KR55eq8LL2JOiwzNKPWa/j2pplllm258YBDa2spFXn6vZd8xhgNKAkjZS6CT22z2GTp9Tm2dpZXO3G4cQXj8JqtK3G81bn4OjLwvNQZC32FDN7f6ysvH7sbR7IOwrdynLMt3NobVeN1vfu6aJ7fLmxJm/H+Ail83/+jX/pZw5w8/rVD/nv+Tv/2e+83//2f/xX3zs3Ivo9AP6rX1gs75REFd8bzug07uuLQax/gSrGGvG6v7ZYs0CMN8OlxSS+OVxajNmbw0ViJrtJ6seCV/2IUze3fI6fafxUIMYxyYn554dLc7M9dZNYrBSt9KGXviJVnLq50Vy1jln1vjhdGkBLiQUPvbp7HpdYInHHra2eWT67WJY4msMoOS1JwEq6WBA/V+sdMcKrc4ttiqfFwkcAjv0MwoKgSgAOXQYBeDyMKytkDFVifjQuKBBjSKXFC0XnohqIkdT99tTPzdVYAFnEDff1w1VinxTU5nQQqWNIBfHhii4VPGqdQMDjg6zD4KyunNS1NRG4p8ZnBhoPOxePlNSqCSzALA/fOzeglzjMYoEjH7upoDARSK9Lc6P1fQJoYCsAUPq8AjohYvSDKHoplRsQFADIamWIGosjNDLCK+FLHXJTkOuwRnAFIJZnSyczLCimMVWUviBEcYVtFj778dy8sBdU2IUGZmpzEiuUvP1jZBRFLSViVLUAVl2LJY6KwEMWRbovTaH2oDIc17+iFJb+ONX2a0ME1MASpwQAXWmuuFZqYnXV1RP5Tp94XuKZQAx06xjTRosJQDomDy7eSA0e5K7X7yt6cKoC4Q+Ae2rWSOjY9VB0TZb19+tg7qd1KIiJUb83IcQ1cIsHHwnEGtPFy2nwoEpBZFHUlJcS2kGYe0m7QUHuN/AWAPODoKMCAI66Poaserh1hW3pP4hRe7HSlQcZj48L8nBzER9EmqPAzRPC5CDmsuGBekt4V8aDzA2RUQZSIBe5z0e4hQGugyr/US3Cj9QOIMoRrU8GMJ1EcS0HQnklbaYhLKlFiDHrnlhJlbYuWq08rCXOrJ+L8Vj/1pPUy6/W7FxZcgDk15t7Wlo7vTZ/5iw13p0UQFa30MtBGX1HQOPAyK+lzmxbzLsbB2B+7a4Ba3RTI92tF72m1jeYFuE6rpFRV0K3c+tr7pVOGOYAzI+2LvqvLn34ttOb5TsIuHqk2W9RprpPU0Ow9d+Nz7qe05udOdr8Tdl1a0PWnd9jbi7TZ1gBSm2rbb+Me4qCF879Pa9YOaWm1fXtqjZorrBo7tvSnpuyxYFXfTcGbAdzxYCSVi7bvLR70frDiwK2mu/2sxv+pu4HKi7W7qaPzTtiafD+fu/ObbXINij2tbV74930bfzc9PFtabxXPqb+3p59T/kZWQN/UbR80orlpXT4P7/+AR77EXONeDcK2M2xm3FMM350fhA0yRKbq+jjYURlwvPYo4sFXaw4Tx2+OF3w1fmI0yDWsyknjNcOIVZ8+foZb6+DKB1B4hVjqLiMPZiBz1+dJV0FgOvY4bPHS3NlPB1GjHO3xCqxWKb6LqNPBV+/PYFCRVK3vlpCc4kbTnPLgTmOCSnV5rZqZRwTuAaUOeDwMDXXza7PuLwbmpvbdO6RDjNiZExjElexWHF8mFpahnkSt0MibtbHaexQyyJM9kNGniOKupqZC5zlfbScZ6RACiUHDEdxDzW3q5gK8hSbi1g6zKglInViyesPMkY+J3QPM+ZrQuwld+Z87oBMSK9mlDm0XIohcnO3A1PLB5hO4oZVcxDBG0C5iisQKjWejP/vCWAgfm9EmYO6kolyF4eyikOz/IPmQkSpqoVKc7lp33SJYqWxfHWpAmMUK9w1NksYGdABE+gc1Zpjx8YsrlVnzU93DWJtqyQWw8hLTkWLiToUcCbJP3jUfmZCfJuQD1XaT6FZRNhcmgAs+clY3JJMqNcXO01hsdDo+HNf1WoFsVo9R8wPmk+NIdYjjVczixdNLieaWdIii/UPWGD5+wq6RuRjQXiOYp1SRaHlxVNLl1m0TGiM7yLKUMXCxpIiAYUQJsL8UJpCQmp9JYbMReOAgrpslUMVmWkKKMPymTsWdMdJrGzdNxFgYP6iNMtimAKytjeXrHgNoArkh7oIWGo9qmatI7Ucvi7ovonIj4KGaoJ4ehfa6TwnRhzVqqc/wlEtluWhIlw15kvXjQAMXwdMbyo4SV65kElOsxlqratyCj4R6rEiPgWUo1jZas+rPHFhEpe12jPSlVATi5VxYMRJrUqBxXrDQBrF2lCOvKxzFPesOFNLN1E7lrozUA5oqSii5sErB7GGlQEt9i8o76yP/iuxFtZe5havoljOr1gsMUBD1eyeCdNrRpyAeCbMrxndO7EC2ul8OouAa5agOnBzQaOqNF9ILU+CWinWW+EJ21yDWDJrJ5a3ZjnhxXLYrNNXagJ7TYviZpZAsyali/AJLBbJoDkFOQFhBLKzWJpF0RScoFaV2mk/vVoWWa1bSermo1i94hWYH+SeWS3LQeuGxTpZeqlvFjGxGgvZWduDZQ2axfLgrGVZ+rB52pqVXiyPNWFleTNraBnk8/wg9bzFMj0zxs/XIHgfU9JZaJfnG80CFgqQT4QwOeuhWnrN+pbOAMLCX3k2lK4zgwO154QYoCzXgLUl0/Zt945RjrLONQHpyqiJmoVOLHNyDaQWw4EQJ7UoBiDMrHuGZH/ZfGY5rA0zK40klsmw7BuqQOmkP1kXfT9duaUeoSy0146QLoxsB3Z1QcYNM7d9TIWxtT6KdRKLxdJZ5G3/Mi38bBZf+1nzbqJ7FkuWfpvF0h9SQOqEzAtC80ZZbJZ/o50Wy2twFstmqQvWbqlnxevWHMQCubq/sVja+ME9z8uBhmtD6/n7sW7+uoMdb5VcDhhEoF1Zxrdl6w25AjDglRK+N/+Pslh+aoXX++s7LF8S0f/ivv8ZZv4zP5WRXiiftGLZhYJfe/wab+cDAhi/+vgNpprwPPd4Ox7w5ekZY0nohiuuucOvvH4ruR1Dxa++fotr7sS6+DjhPHf4/HTBNSccuow3xyu6zwS05+31gDfHK+YaUGrAaRB/nEd1G30aBzzotc9PF7wbe3z/9RMqk7jhHq+NZkNwnHJCLgE//PJrVCZcFSmToBbKWPBuHBrozaPG62WNx7O+Hl6LpW2IBe/GHsyE16crxjnhV37XW5znDtepw5evn3GZJT7ts4dLs0g9jz0ejmKVfHW64qoIiH0S1M7PXj838J1AwJgjjsPUEBWnHHF8M7d4SkMlFMAgmdM4J5xOI5KmCik1NPfXhy9GjRudMc+pubgOhxlvXp9xmTo8PlwF+bEEPH7/ij4VvLsMiKcFBMdQAX2KEUDiPGsNiIe5Ad4Mj5cWqzpNCSUH/ODXfwwiltjNPmN4k1udrHF1djjQP8p6znq9VkI4aBqVzy4N7XD4wYxaA8Y5NYTC7nOJsew+k7glQ04kEiV9+J7EtJ7Hrrn0lkLoH0fUKpbay9hJrJq6/y7pLjTua0oIp7pCP0yp4uGHE65zkkOK1xU+vYXN0fo0F11z/zXrpMW1xlib2/M8JXS9WLDnOWL4QcY4pl3goK4rmKek8ZEqMCmfGYsF2drmOaL73kXiUr+4IM+x8SW8UkREJs3zyC0+FACG78/IOS55I4NYPu3AhAGwoVYqL0oJ7ffUDnFaXKWLsTQAJosLLSWg/+VnAVp6GtoPYEwFPEmMJ0j2T1Je1Wl5/dr8W55IAOGXMujSIf3gjHIVF2mz+qVfnts6VZ1/0ZhdrgT6Ulx9MSbQ5yJVsAPUSL97xPQ0yEHC5wzEAhiQ2DADl17GSxWYIvBFabFUFjslDwWJJTmoS3EvcWWIFTxFVMsLyWiAQ7WTQ6I6pSXGUutUi/tT9MhWd17i7GCxkVME+oI6634lCBqlzpUI4F8fUaaEqvFprNbvcu6EbqDFftJhRjn3qKmC+4x87kG/W2JLW0qGYUnHQ6TooDoviwGtfV3iRgkIqbb4QpsDF0LWOLc6lGZJl+dB3bb1HRuMh7Y/SPftQZIF2lillwM7ChKf2sCZisQv1ksC//q0iv+057y5vs8BYRCexl6eixAqaolyKHqNoL5KrPWYQLGCotTJbj1b/1MEKc9sTlEP5MolgtRqLzGydUU7F+1jNq0aS97QKeB6KC3naHOD7+S9Vu1g8Crj82yxe4R4zCjvOnzr0ssBIQDps1mJWa4nPUjs63I4ZweTgzsUdHlAUWlxuTeAJesTkD48WBPLfqFjlsNM3VfoqxzOeck8uLZdlec46mEeY8nTmmkdn6hKQ/MWqLjNx0ruUJBpyVHa12XOQeqYezvNrr25fOqzQVnzlzJJ7CO7PvRwB0UPHq1t4IXWSjeWYECUyps8n6YwqdJFZYmxbHy3oocf5JU8Z2Vt1mmjwymeLb2LFdqCXq21pbZ09i/s3PduvQQ51I1YK4Rwf51yCNrXx5oeaYodNn2sKtNC392ypnutfNKN8riepCdo8/dvtvyX30Efv3PLj37Wbr7AJ65Yjjnhr379JcY5NVRUgzovJeDrdHR52EJLtg5YUvfQkDNLDg24xNz7TPnKc8RbtSp5JE6z4pkwCkCElhzxLipgxg6SZXNnLYTnSy/1WCwJvm8THNtYTGukOUYTVgIpLDujJSp/Og+oNaDmgKcksYnSl7YnbgIXEeOJsMTdEYBKOMe6erBZURabW10lPMXDAlPuUCC9gEhmBasLKieXgIuBdejYY+raGOc4gHPAOdUmjF1T34A9VnnynOukPwpj+/G0Fz6Aa+TFAqlCyW/rvqhXsQze1HHlqgJbi4tzNFwNeICAcxoWwUfrXfUHfdQf261L0SVpvkIDCdCX96w/3JekgoH9qBvCZvuxZBFmAi9jQJpc+77FRs6+vgkNTGtUTHN/UyEGUDrsh9cEjkKYQ9+snHMSgWp+of2se0GYvuxpY/Vs7lKVkIMoVZfYLfNloISuWXVKcIKi3j8nVcaKoGQWm3PglrSdWGmxJXY/zMUJS0yMErp1X7y0p0p4Pqslewytvzlws0DqlsQ1iumBHOx+cxszAYWBMXWgmXC5pCWGT/u5nFNbN2LCqAnjre0clZaZUCzPnWv/dE4tNrJGRmnJ7Rk5dnKPIW0zIXdiSS8GgLEQ3qyNKECOsiY5yPfqnjtb75pYls3f1z3C9o4IarVMLM0yNYGsJvnZoqyfCy3yt+7JoOt4uQjdNnaJsnZBLakAWt3cpXY9xw5hIuRLBE2KYMxATXEtL1VanbyHipZn0wA6amAEJwiDpZ4JgqVLK4GJCq0ErgbGgUVwFAuLO5goQO4WC3BU64XNr6aEOBHyNayE763rXigApyTnDBYfqO8I4sWinFVIB8l8q9Jt+y/qPVlDPYzRgw2zZCWLr7V3qIKKJAVRsWs2J7PM2Hw5yaFSs8CERUGxPgS9Nq7iSWtKSNP6nf4xhaPwyfjmaZMY07hYpZzHvbSNizJj1h5Y+7iui6Vvf315p0Lm1lyWl7FXArhTqDhofaWt0Um6nl4p28YBVkgMpvGbTCFze9RoCe7wyd6VtuezyRhOLjJe7LhYtzrWrDr+WXE82fOmvbHE+WLPQoVaECExitti7VdE2eBWh5br+plv2n343tuby2o+L/y9KXd481Kfy1zcZ5vCnTGWDvfrbGm4q1y+VPZoeomu34nlE8a3eV/5pBXLN/0F/9jv+t/x4/kBM0c8xhFjTbjUHpfS4bPugkvpWs7EY5zwLouL6oPmeMwckKjiOfcth2IgxhAyjnHGWBO+mo44pQmVA6YaW/68hzihgvDNdGh54/qQ8TQPeOxGVA5LPj/d7ZYj8loSpprwuruucg9afsKOKr6ZFRmUJf9hrgFTTa0vQPJYBmIc4oy30xEVhD5k5Brxur/gWjpcS4fX3RVP84DMAac0tbQnltNwKhGnNOHs8j5aXkbLDWnztzyPlpLlEOeW39LyAFr+QssZ2IXSUoTMNbYULpajMFLFWBKGmHHNHYaYcUwzvpkOeOgmjCUh14BjmtGHIjx3ORUtFjIqrTbWeZb5+NjPIeaWPmbMCQzg+0fJY/mjywO6WNCH0uZqeQi3Mbaj5pq09DG5Bhw7ATcCJO621ICxiMWq1NAQig8pt7yDhZfUMg+duDObNd1S4xw7iVG1HIMEtFxy9o61ONDzLAi9fSy4zF0DjXpzuOCSOzxPfcsxafug6NyMnoayS9y+A2h5CiNx4/E1J8mLSYyrAlM9Tz36KJYYS4dj8ctWP+uhy5QjAgGVgU7jR229cokYUsZlTjh2ueXELFXyBVpOvl4tlmIRF1qPneSxnMziqHG9KVRc5rRYPkNt484lIJDkEhw6QTQdZ1k/y0coaXrEii+5FgtyCXil8cFfXw4tPjfpGsGt0aGX/XOdunZ4ZcjGlmIHEKCsp+uAx8OI8yh5Je3g6WGYWtog48V1XvJYGlq0oTXbPrX0Oq8OI95d5eCpi2WVLujYz3h3HVZI0UOXV3kjl5jeJb7YH+5Z3kT7bml6ADRU6XHuVrkdzfprh3GWq9D6AsT6bbkk5xzRKe9bnkB/2EeMx+OIcU5t3Sxn5GXsVzkPSdflfJXr4hnQ49DPuE5d83bo1dps8em5hEavpRPyKYsAsXxPU9K46SXvoiFV931ehTfYwWizTjtEa/vLTOhcnHStAV2XMetezTmsPAaGVDCNHR5Pox6ortM7Ge1FEZ/nOWJw8d6GQDxPCSkJuvM4doLCrd4SHqHbDnBrDW29So4t7ANA4wkAVEUStnpRD3IpcENq5rqkUqqFELsC1meAKwRxNhXlr6Q8KnNE7MSzwd7hQ58xXr69xTKmijyLghgSL4oQgKwhF1XpadZ6PSSMGs9cCgmgmd2v1O5Vp4SQO+BcHfoCYn3tBaXbEInJ+nQKGmhp20JVVHFnxpJaSFFiW5yks2gb+BipRbrFz2rf7bqhO5vngutHrLR6MGrFDictNtUOLf1hJ7Accuq8Vyl8CIu2wlgrHcbHuk7BA8hhXovd1IOxVWwyub704GJ7qNb6N2Vpe1Jj67VRkneVNtftS6iwuxY8q7cxPGzLrmLdBr7Xt9M2NxP+KKXwp1VXiyf9kyg/A72SiP4cgH8A4jL7mwD+dWb+j7/zcT5lVNgf/v43/E/8uT8MAJhZ0FTHmjCErJ87vRfQUcXckp+HplBWpqbsjXWdTLxyWClV/rP81Ze1tjGlyhQsn8TdhDpr55PATyWh12Tuvvj69t0UgW3JHFcKZwWtEtebAmZgRNaH5dMEFsHTj+dzRfrk7Va2ibYBEbpmp4BH0h9MEALUhYxDo2Xb93be27yVJsD6+x7UaEuL0elBbvx9ZsJcxeq8zS+6VdpM2fMvMM8Tf71s9ofRbkrTqp2bi7lLV/1rCdUDMeYS0MX9HJ7Wj086763l5qZsSpwV72lTefled14NvKEXWNxCjZ82R8/zD12HhY7bnKr+2rZecW3J3ZP5uZNxiLAbQ21rVXfohNaz+W1ztG4FclOEDDxp77mwYkL9Xh7Tli/U6FQvCxPAt7R5mvxfU0A8MJLnsc9xakpT41Glln7F3L9LoZu+Wn1VCkOw1DkbgRTLQb71vwVaWm7eHr/frMvm/URYP+crPpUlJ6f1b0rISmhlkljtVJfPirbswYpWOT91A93sbatjkg6bsLw0bcK29rnkRxWXUC/YNa8Vq6Nrwl55AJqFz+qaK655Noj3iF1b6F+5tHnh1Lvr2XzM08DcIk3ZsTl690dPl10zfmy/bwVXX+7d20qRrFYuRgMGMst3c68EgIKGUPxtyjoXI90K9+3F4hvpX5+6YaP83eSbZbqxIt3QYsoQY3GN9OMZXbrON3ksnRJj361NG9vHIjp3+l1XUFOyd54T68vHQK7qvFDI6uytue9rO2/fwZ4yZrc3fCa/T12d3bLtdzP+S1v7xX53+tod9310rAbDwrN747yvj71uX+rvu+jnO+jbyl/6t//Uzx4V9vGH/If+jn/mO+/3L/xP/9rPfG7AJ26xfC49/o9vfhWX3K0sVwAwl7gSKE1INwuPWWIqU/sMoAnxMSzIoVNOiBofuCcMWvyhCeIi3KsCtXGf9QKg5LSsrQ+zXDSad4RHm4tXeLzAXVTQsjn6uqzjbxWdPSHRhJit8LwV3JpA6YQjuy7zWYRc337bp+VR831tBWazaNipv69r37e0bXm+pcNOX1Nn8XSh5ahrqKibOVvOynYmc/OLpPVs7jvCstHVaNU12grI7PvHWomlnftkbXd+/IPbd+aO7OtwRcsnacUrOtvv3j3b3KtbH/dOTjc/8HaSf3PCvPO8eGH85gdwOxZTy9noXZV9vkHpX/e659d2LR0PV0rASjBaFIJrHlofK5qN1Kgxj97ty/jqFeTg4g3NBbj1ofzxwqgfx/NjT1AIjNm7eG9P/zcu8U1pCJtxbGzjk1egthKxXd/su9W9reBnr8AtmAQBxV/bPHvte4slc8oMsBbi3foWyw9BC4rrS4cEgK4b3X6+665lLLJ7tLnNAXvPDK9443ht1zduxQQ0qwzb/lDLkClZzWrjafR83hZvzZmWz0zq2k1YW5qMTkD8nokbnVTCEs/nYDubQtj+btwqjYdeQXH7AkBLO0PV/aVlqHB9eU1fLAQEOxTYPgqb+L62JVfTdO+8e3wCVu90+ereDdafAzwLjMa3m+L4R5r2hjzfXJ9+nv7a4tVO6y3OolQGz5DtM6ButyAF3bHb7hm4UXi2n93+39J6TxlsU/HrsK2y6dMMptv221fMxyg2u2i6rt+79L6vbOf1HovlizTaHOvOXD+wv49Skvd+lz607c9BoU/YqPe+8kkrlq/SiL/3e38VT+WAWd0WzXK5tQjNHNXaF9VCWZA12COAV9ZMAAhUm8KZte+V9U6tnACapbMwNbfbToMwhK66Ul4jcXPF3LOCSkqU0KypRqNZ/LZWPrvv3VD9/LdWx+QsZmaZqxyQQkG2hFtaAq2tClvLqNVp9G9otDE9XVsaFxrU+qQWJbtnbqbWDlhb/oBFAdkWs4wabcL/xQJlPFuvF6/o3lrMtntha4Xc8nlPeTfL3rYY8FBx1uO9Ndsemmy/G69szxHEdbao26v/LdvT+3zxc92zHK7GU17szRnAzWHBXv/bsbf72fNwa0HzSkBs/Df6lnplI3xFtcZ5K625u2731upAwNFhh0S57B8I+Wvbfvbma3y0v74E4tUeiu4ZAd6v23gXZ3Pr9L/3W68Es0rfo33ParpdI2+9NKvqVkn3BxeLVfj2mv9shyV7vPWutv7+vbENSMsfVi3up1gdXlkdX8xS6/uVL05Sc4rsrgS33V9hCY71fN9eX/EaO2cLnnfYKXvS9Obe3jot1lYsh0qe/s2atO9O2fME+xR62zkv/erBC7vDP2J4y++9oLG9dfuo4g9xtt1sXQ23e2BjuW7Xrd89LWOjVK2u+xjme/U+ZIwPKXuK3s64xLRKKWLXdvvZ62tbb/uu4Z02bfwd/m7bbsudce62udfHPYVuq0zeJ+/947xUl/DyHsB7FL979z6Wn6u275/tByvR76HlF+V3TvmpKpZE9C8C+Kch2+EvAvgnAfwKgN8A8D0A/yuAP8HMExENAP4sgL8LwI8B/FFm/r9f6n+qEb81fo7nPDQFwQRpUwazKhbZKW5eMM6q+JliBCzKkV33Cpv1Yd9zDWsFbyP431OqvEC4FRr9D7FXLLeKo6fJj2Wf91wJXxKShR/hRjna9u3Llg6LF/SuoXsujG2urs49y+KeYOIF7nt9Y9Ov9eFdRU2R6Cyty5xW1mZr42nZWqe3tN4KsfuKpVesvDIB3Fcs/X5pa4p9WcZ4a+MZKq9fMy+87fF56/q65zL6Eg+WOrcKhQnxe0qB5d/0Y5kwu+eWapZ+r0RslQprY26r3tVzO991/1iNv57XMgeLNSsbxdLm63OC3hNufVoiU2piZEHQdcJxCDu0rSyWTolRYdLPzYDKGPrseoGTGOxcm7fKPeuea0MZ7biVbVYyjuvfFJ0bZWuHF95S7NfL8+reQ3Bj7XWKx81+d9bllbXY8d6UmZVy5BlhVl1TdoB9K6+fo7nq2p+thZvieq7Kr62Lrnc9vlFkVRlaxcjdEOPm4flpe8PzmbCvRLWXDrVrbSi/1MRLPN+9fvf6c3Pinb2zd+1mmnR/PT6obOfuad2h3StVpnCZFdlbi1futbuE73z/YMXSaKVbvhIve3c71fVr4b4la7PGZOPs1NtLs7Ba3sbLFxS9j1SCtlt6t817FKibub+k5Nz7Ud70+VFl7+DnPYrb+xW/b3nvfX1/yL17e/XbjvmplV9YLD++ENEPAfwLAH4fM1+I6L8A8McA/CMA/l1m/g0i+g8B/FMA/gP9+xUz/y1E9McA/FsA/uhLY1xrh7/8zS9hrnEVf7eyJm0ULC8IbpWlfQvC/Xvvb7sI8HuK2fZ9TFjivfzn9429R8NWGN5TBu4J0ts+txaCbfsPfT62tGz7M2F6r88boX7vF+4D+LSnQK3u43ZN7PPNWNtfG3/tRRpeEIj3CNkl4KX+FxfMey7Au/vqnmDpBZBV/T0pAx+4DjrUjrL2csOP5/fL/eFl/n+MALNX3wQkz6etpLb9vCnzHRI+mNZt3Q29Zbfems9EUX6AAAAgAElEQVRF6St7dH7gPL5V+Zi5bdvh5batih2e7NzjTV/m6np3z34kveTbNHp26pliuzvm4oHwEufb6+rOGEt/eP8c7tTZPpp323pifV9713b7WM90d6x7/f60ys6zhc2lLWkf28du/ffM7a5C2NrYs/sCLXeuvdg3cHvzQ9bt3vju+/25vKfc2a+r+x/z/H4H67PX54s8bX3Tt5MRPqDex/L3g2n+wPG/03au/Fwppr+Dy0/bFTYBOBLRDOAE4K8D+MMA/rje/88A/BsQxfIf1c+AZJr500RE/BK6EIur4zHNyDXgPAty4rGbMcSMH19O6ELFXAOGWPA89XjoJ62b0KeCGCouU4fXhxHvxh6HTlAP5xwFrTIwPj9d8DT2SLE2AJVAaDkfHw+CPAhIPObjYcTTdQAAnIYJ45xWz4QhMXap4O3zAUSCHFh5QdMrOeJwnEQIIMasSHkexAYQZMRaqdUHJJdgShWXc4+YCrpOEAG7PksOvym23IGH49TQ/OY5outKS88SAmOaAqpzD01dWVALIZaPPMd2vbnMYbGK9IPlE5RchkGR/pLmNExdRi2KRqg5EUsJmK8J3SEjTxEhMkKsmMcEzgHdaWrogS1WEwAXtcwYIuMgmZVtvgCkvyQn90nne/3qADDQfTai5Ig6B0HQ4yX+0n5vylXWOvSSRy0mAfkAQdKgaBwTX6PE92g9ipKTLfRF8tp1FWAsVpVK4FHSAWCQY10KDEoVdZS+MAWQz98WebG0MIBKCIcMzkH6OihyZA7AcwT3DDoU1ByWOLqABZXPLC6WqsTisoSa2xxohAXpjwH0FXSNku/PLD2W662SxAP1Wj+xuM1ZPjfCksPNgEdSBaYg/Y5RcyYqjXOQ+8CSL47Q6KVLAHcs9DG11DI0B/ChoFlILA8ejA8qxE8SM2Y52ahozkamlvMMJLFlnCrCs0D5l9el7T/KtOR0YxknTAGoQD3W9mNJVcdy6WrCJaCeKuJzQDnWFTpgPIeWQBxBUo20/ggIo/CjHqvEm2nfFn+VngLyY5VrU1iBgMSRkB8qQJJ7rg4V4RzBA4NG5amPycxB4nI6RhhDS2xfe17SetjcAIRJ+FcGbikrpA4vaSEqwIkRRqlbB+ELJ0a8CK/KwIhXQu3ZJSg3ngh93TtC6YUWqkCYJZ1APjGCpmfhKMm644WQH+R6GIH8wEjPhHIQHjMB6UItDgkMcIeWVoIgn8Ok73nJ+IIwAWXAMlelj7LUT1eXO1DbWfoGBhAnbUOS+N4sVVHjBG0OcRSegiFpITSWjaOu6ZHRvSMgALXjVSyj5cHjJDwtvfwF6VpGob0cZJwwAeUgY1j6h2qpKADUTq91QBzX14KelJQBLfG9pKGQz6WXOpI3EKj98vohbcsdEK/CP/sX5iV+z/ZgGWT8miCJ6iOQLsD8iG9d4iQ0gpb5A0JrGWQcpmUO0H0BCC0cZH3CjPZs1Aikq62lDsRoSe9tTb0gwQFI54U/HGWvcVzaEC9rA8j92jnaaM13b0EOeVkXKF1htr2o/VsKEttrbpwlNQyB00JfW0/dNyAgzAzW/d3AgpwjFxWAqu7t6OqxXG8ARAFgoiV+zf5EAlWWdCKai3IFSsTQfJNY3iXepZuwvCO3xSuPm1jKxn9d0z2D+ksW4Pau2fR5c6ZKjvZNnO9qrm5frMbaHGj4OMumMO4pyXeU5r34wdW63OuD8F6a1uPcDPNpFAZW4F4/Z+Wnplgy828R0b8D4P8BcAHw30BcX79mZgvf/k0AP9TPPwTw17RtJqJvIO6yP7o3xkMa8fd9+X/hqQyYOSKioiAg14iZA37Pw0+aK6nFNFYmzBwwhNI+W8xj97qsYh87RTW9lA7p1dol1GIsK1NDmLW/15LQf1YQwJi0P1MIfZxc5YBff/PjVRykd6mdSlzFfFYOuzGMiSTVw6S/HnY/fVmwQsCFoMH2Gk/okWPN3XIqsY3/Egqn54WvWzZxWKS8NURaG9do7ELBWFLjiU/zkahiqrFdMz4aX7eW4D2XWLNgG0opsHY/tri04xdfozK11CLbGFA/lkeO3b4bt/Uq0w1iqVnR96zYnWuzdZm+x+PWt14zN1ofgxeIG92jglHtWe63LtOehi0ddYfGKceWBsRiGr3LscUtendi7ypsf+2+0WHpA2zOe27HVg9QN+4oBwJZDzrMzZiIkcs2lnjZ03uAWLZ2fr39Z3t+iLilF/H89GvUZJCdtfO8CMTIJSD9ctXUImhuoIZqu3IPd+69Pu3GnltxirWl6TAXXu9CnDVFyx7IlgcV87T7+/5vc/l2SLVG79ZDwcc02nff1rtJB6ab+gvNypdfY0QGSF1prW3cILHKYIxo1mWSOgjcgFoqE9j2oA3GIpmuPC+c9ChnNWvvAP+WYkCswG4t2dra/tuOp9IeE6+u+3EaOJfythLAFZjiQtu2mAdF0TjJokBeqxjXSsjKn63gt/KQaIIr4Qb9tT0AhJWkbfe2cau+L6uva9NcwE0Y3Urue8KqHZp922L9bOdDWFBfeTM3m4Mhw3yokO75fKOAON7a95W2sUf7C/e3ygOwT+dNl8te3QWF2va1J1Bv57mzD4x8gG/dsP0YG0JpdY136kOe4Y3C9kFWu49QbPw2fG+5twf26t1Txrbjv4/Wl8b46P42mu32+sfQ9QG0fGqFwLvK989L+Wm6wn4OsUL+XgBfA/jzAP7h76DfPwngTwLAL/8w4o+8/t/w2+UVCggHmjFzwsQR7+oRb+IznuuACMaVOzyEEW+L5IY8hBnX2qEi4EAzflwe8SpccK5Du38KI661w7t6xIFmFISmwF65wxfxCVfuca59u3+gGd+UEx7jFQBwrj06unU2G2uHAsIpTIiomPWoL1BFREVHBd+UEwCggNCRKIkzbwXiigNJrslv8qnRPnPEKUy4csJcE17FK861b9cDVYy1a/RJHtArnpQ/pvid9Pi9qII7suTRtDEKB3RUMHLCQFnAk1RxnDmio9LGsByT19o1/p+C5BMNxLiUDkPIrd0QMt7mA45xbqBMQ8iIVPGUBwwht7bCC1Y+LUrBU5H1tDlG1FW7S+lRQfh+/w4A8NV8av0YryVHZ2lK/jFOjV4DhDKQI2tXOeAhjaKsqsJfOWAIM57K0HKmGohKy2Opx/vPZWigUrlGPKQRmSP6kPGcB8kLqgBUHiAqgPFceiRa5lmZMMSMV+mKS+nwLh/Qh9zGBNCU+0DVAVzV9h3AKkdqCqXRbjlMZV9HPKYJT7nHYAcYeoAjvJt1nUuLf174I3T6A4ixpNbmIU0Ya1zRZwBXfcjoqOJSlvx0Vv+q1w5x1jWSehUav0oLArTR5MG57FBjdcDhUwopcM6rTtbu3Ty0+ViOWNufgbgd7EzOhd/H0xotpzTh7XTEYzfinLtVHPghLQ6yBnB1zd3qfmXCtXToNf2SX6fHbmx5bfuQVwrwIWY8ZZmDHeJYztg+5HaAtXfQYjlt+5B17NL47Ov3IeOc+5VC73lp9FhdW0PLBQvIAcn/x967/eiyZPlBv7hl5nerql37cs7p27Q9zYxtxvBiYAZLPPnJb1gCISFxByHzgASCRyzwgJEQbwgkS7aQRkKAwG9IHv8Fg4QNjIAZzzA9PX05t7137V31XfISNx5WrMjIrPxq1z7dPX1OMyGdU/vLjIxYEZGX9Yu11m+ZlA+VSbryRkZSYDZ6gIsyg/1ap1RUKXduTBuDUkTUyuFoK1TSwyiPNuXT7b3Omy21GvNYAvReZHll2lSolZuMuUqbZ+WGVrnZxfW5WK/ymoYoUOsxFVW5yWNS7lgmCDNFzlS+vvx37xXWxub6S5spPors5WOKjRjulzciTbqvVbFOnN828GZO2sCp0pxzn7zBY72apFPi467wLmEPIS6cColzt8YoJozqTMLlAzGzD06h0h6+AL1ahexh9EVKmZuW36G8sWW9mrzTOQY+Fvd++ZzzG5i9mMqcwdzuPM6dS5n3lX8vb7TQxkyMItUvNjniNG0Sx73zdeXGwrkwknIjK6S1kkU/+fogJ7HdpZSiuBczbg/lPJQbDPfnYiLXLPSAN2pY1nttlpsQ/HeOf0rgVvY9xwZL4K4E03OQPt+AAN69OfBQeddGwIPA84ENh/c9viTDe4zp/49g8+ep/DRdYf8SgD+MMb4EACHE3wHwFwFcCSF0slp+A8CPUv0fAfgmgB8KITSASxCJz6TEGP8mgL8JAB/8uev43938Km7sBr3XWSk9+gqdM9iaHkPKp9h5jUaRMhMgcGE6dF5nJejkKjTKZmWnURZN+uC/6db5484fcACokt/N7bAi5SAKNMrh6CqskuLXO52JU1xK5qxkgA0K1itc1N1E0alSfkYtAo6uyh+hWjnYoPIHnl+ktaLk2rV22A91VlSGoPC0OWI/NOi9xrbq0TpDiplyWck52CorEEoGdCkROVlxJRrtJlaywSsYGfL1IQoY5bOSV+YwZEUj59NMykdIykvnNBpNSe9VSoJeaZfTu6yMxXGosDYWvaexr42FUR63XZOV9pI9swRLUkQcBwMpRksbAFRJXiVJwfBR4Pn2CAB4c1pByYg6ySUFKSflB36VEtxz0nbnJbQKCJFIgHiNVsbCR4EuEQI5L1Fpn616nLSdFS4pAyrtUSuPfV+l+52UtMY4OC+xqixOfQUhYm6TlTImFuqspvlWAb0bLb0XTY+TNfm8YqtWEPfIpFjxKQmCAFJmXZpztkz21qA2Nve9qmzug9qn50ayQloAAgDJFTtmRYnvP4CUz0o79FajqSwGp0kRDVMlSamQLYesRK3qgdzaU9J4KWOWgZLej8oWW0PZ6heCyIndh3Q95ZRUua1SWXNOYt0MUDJif6pHy6IiF3NmGgWAytDz16Z1LJ9nnguu13YGm9WAtjcT6+SqJvd/lkFrj743+T6tKnovDAO5xccoJrkot+sOh1OTXLkDtPY5v2alPdreIAQBrQPsoFHVNrupW6smSgLnvPROZpd4nhOtQyYzYkWR3c+HQY0WS7ZmJsVTpJyYpnKksA8aTCpk0tjYdd87mRVSZkrl36t1D2s1XFqD7PrembzmIT1/pnLou4rukfTvuhnQ9yZb/qp6Ciydk9n1XyQrLd/DIc2nSnOY+4sC0dO8hzQeetZpPn0Kb8j3kPY59+dowScXfb4uBpp7a+mdla3Bad618bCdRrMZcigDEzEJEYEELKIXUMbnUAW+hylHK+AGTfdL0ZeQ9GzYQVMqpiDG58gJKEPW9ZBSOan0XLleQ6Z8kpxbVADwVkKm8AKpIoKVyAROKoEgqyArTyDDSzqnA6Sm94B3ElJF+F5B1R7BiUy8pCsHe2J/zPcv0niEIVn02c0/0r9Dr3JogzSeZIvI1jcOn4hOQugi7VAAhUaAxsOaNVud6TgDDyCZwiFXLsvCbUQvp0CGLbRR0HknxnCGIABN6w8rR1DG12UyKpIROh1j8CBSOzpZTlMoD0wY6036SmEQKORDqhPSPPGxUFhDC6uw8ILM+MUee+A1KGUrS7JwihJ0clcMgAMmrqQlDhIR01QucxDKvxfIqKIAJrYFgXuuptwHMLOWnjNxxqmsE9KwJVCX6vAYl8qSxTYmWe9XLsZZ/n6gnMPqi2V+cj7PPw/l59hiKR4KYfyxGhbinwLwtwH8EyBX2P8WwP8G4J8B8D8X5D2/HWP8r4UQ/w6APx9j/LcTec9fiTH+8w/18eLPPY1/5Tf+MiTIelSmDCktL7xrfHJmtkOe6kWVQSW7jJY77LxbPy/lLjv/LneOA8QktQWACcurkgG91/fa5MKKPYDJrnhp5eCdUwYCJQCa7KQnkMeunbyjWu6gcptlez5OXSVVIc+caXWeYuJc+okSqM53Z0s2VAZiHNNaMrXyzvT9NRl3M3l3mo9zKa9jBtjO6om75IRBU07X0CcX2HK3upS7rMe7wQAy8OF6cxZVdg9ksDKZl5mbI7sPlvk+AWTABYxukDSe0c1yidGV+1liXC3rl/lC5/lGy/QMpftlmaJhmn80Fn2jaLN0y4x5TM7JXO9d5E9AUuxTTG92t+QxZWvMbNd6tlUag8xgJV83m5fSXbcExyjWO8sIlgFZ8S+/l/xvAYyKdWL8LOvd281PivlkZz4pu0u5N7NiC9z7WOeUGaUwfN3sfuBjiBhjfeeKDg8sX4D7GsySMlTM1TxHLmJBUFWyo7LOyOvKsbqlCyIru3PLQ8SoGHJssU/jKOvx3Gdlt2h7zgArQEptyd6JWb05udPSfKGYs2JupnM1u25+ruzzjOKd5WUlfN4MK9cPKrCYxuDOrTX8ez4nSw/BkoKZ6ggvUrzfqJGLtI45TrdUviNfB8wcf96vzMdeypribUvwUJYy/m1xbPM1TuOii2drL8aY5Nx+XLhkLm+xxvkYMInhLv/mx39BtMUXGDBxd83gSWAKbOIMQE06u9/PwvDzjyVQNJFvLvNSmT+C8zbOyHWuPNrq9th6P0ZZBK3c9wOA7ewY3iXzI8f06DniNhe+Se9b/u//8t/7+zHGv/DFrv7JlMvN1+Kv/tl/6yfe7t/7+//xz3xswE83xvJ/FUL8TwD+AQAH4H8HWRr/FwD/vRDi19Oxv5Uu+VsAfkMI8f8CuAExyD5YrvUB/8LT38Jbv4aNZPFjl80uGhjh0AVyN+wC/bZRJ5fVAV0ki6ARHl002SWV3VEZAOxDgyZFp/v0lmK3RhtVdt3kv30wqBNDAbtTqvQUeAio5Cbqo8wupVzYpVIhJlfdEYhO2ivGqgS5znbBTM5lt8zkSsvn2S2VZeCiRMgxpiSrvOfGy/2XrqKlm6mHnIAnvobnk+ZufNOXrm9cj3OO0lxqGOFzuyzP3CWYNwnmfZS5SgGOjR3dWksXTQDZlZLjazmPKecWLV1O+VpObVOWMnfpPEfpPBUOx/kCyO6nQ9DZJZPzo44uxjLLECDuucKSm2+Y5Esl11naRGAXXo7b5U2Y0hWWxnB/vZZib3kcPCZ2E+V5540aHh+7HubrC1BebqbM3SPLfKbzWN9yc2ce4+pyfK7PY+AYZ74fxg0cNZkXvofmuVAna404aacr3HGXYml1bnd5s4qvqaTPnhbsNptJtGbj5TGV/QJjjtX5OrLVHkCOpZ1b9PP6FDHaSzlqy3Uw0me31PK6sn+Ot+b7gI/P76t5Pltg3MQq44BjFPeu5TZN8gAp444BsoTPN9VKd07ejGO3UG6Tn9N5/O25+QDGnKFlKWXn9SnbKDeqyvjw8v6YHy9dbZc2PnwQMOr+vTDZ+IxjfLMsRJ6zo3OM7jmmcV6TiHEDbr55WebTPYdTyz7nYyrTCc032fh3uclVunb+OHksy1hu3qibyziPAX8XC3YZO3zeFXZ+0ZkUM8W8lu1HIFup83HM5n1h8yKmNuepfqZzMm5m0YH7/84pceYbL6UQYRzD2dQ5wH1gXPY3R/Mozi0dX6p/rixthCyUMu6Ufz9apnmdcw/GUr13ga4HxvreIHLh+INA8b1A5OPX5L3A6Zep/BxbLH+qrLAxxr8G4K/NDn8XwD+5ULcD8M+9bx8bMeDTeIkQJa7UETZqfOouceO2+E79KTpQbOXeN/jQvMV3+w8gRcCH+oS3HjjFGhvZ47P+Et+oXuMz18DA40qf8Fzd4W1Y4/vHp/h28wqnUKEPBlvVoY8SV+oEHwX+sH+BS32CER7X6oDfsx/hA3MLAPjhcI1L1WYlNUQJIxxu3BYHX+Mb1WsESNy4LWpp0QgLIxyu1AnfHV7glGIEL/UJB9/gFCooEbIXCPf7VB3w3f4FbFS4VC1u/Qq/3HyCT90lXtkdvlN/hj8anmHvG1zrI3aqxTHU+F73DJe6xclXuNYtPgsX4FhGH8f2Q5TwEHhjN1irAWs5wEaFLhg8M3u8sjvsVDcBt7duhUvd4gfdNbamp3jKqHFwNZ6YE17ZLZ6qI97ENdZqwMthh+fVHi+HHTayx7fqG/x++wIvqj1u3AatN/iwvsVOdvjd9iNsVZ/jMvtAsXiNtJCIMNIhRIkfdNcIENjoPscbvqj2uHUrGOHxWX+BPmj82sUfwEeJ37r7RWx0j2fmgL1voBDw2m5SvAwpSV+r38IIj0+GSygEHEAxkyEKPDEn3LoVFAK+2dygDwafDJdYywEHX+OpOeLlsMPT6oBP+wsYESCDzjGT32xusJYDvts+B0AAeu8afNTc4uBrfL1+gz9sn0MLjwvd4eBrHF2NlRpghIeWAT9qr7BSFl+rb/H99hoAcGVO+OX1p/jcXuAPTs9xadocP2ujwjHdZxxDx67l/JvB61NzxJ1bYaUGrNWALhi86re4ro4wwuPlsMMvrF7jD0/PsDMdjPA4+hptivV7ao74rL/AdXXEnWtgRMDNsM6xbFdVC4mIWlLM8s2wxkfNHX5weoJvr1/j5bBDHxSGoHFpWrTewAWF6+qEWlrcuVXeHPjFzUu8HrZ4NWwgEbHRA1ZqwBNzwvfba9igMASFjR6wURR3/NauYESAjRLX1QkA8Hm3gxQRF6bD22EFKSI2us/5cy9Ni1u7wq/sPoZExD+4/WYGfpemxctuO4mv/PrqLRQCvt9eZxC7UhYuShxdlWMwv7W6we/uP8Cf2X2G752eovOje/F3Ni9x8DVcVGi9waVp8cPTVQbfHzYUM/zD0xWe1kcECOxtndv+ld3H+O3919E5gyf1CTvd4ehqSBHxp1av8Nv7r8MFiUvT4bNuh6+v3+Jlt8XT+oRb24xuzFHhqjpBiYjbocHz5oC9bXBVtfi82+JJRWsUokhhBgFP6xNWasCn3QXFwXqdgTPHfnKc5/P6gI3u8XF7meNInzcHAMBn3Q4fNHu8GVZ5I2adYipPrkIlHf7xyx/hh90TvBlW0DLgRb2HFBHfPTzDVlNMLMv1rc0b/N7dC1xVLT5qbvEP9x/gO7uX+N7haSY++2h1R/GYkuLab4YNyasttAg42BrPmwNtSKb5vK5O+Ph0ia3p4aJE5ww6r3Fdn3ByFb6xfgsbaaPDSI+bYY2DrTMof14f8ruL74HOG3zQ7PNG0NFX+LC5w8ftJSrpcWcbrDU9350zeFof8cPjFX7l6mO0ocLNsMZKWfReQ0uPlbLY2wZ3tslzyvN8oTscfQ0bFH50usR1fcLXV2/xR6drbNSAje5RS4fP+h0u0nv2db9GpTzuhgbPmgOUiHjdb1BJhxep3e8drnFRdZCIOLkKF1ULIwJe9ltsdY+Tq7DWA276dQL69KxKEfCmX+Oj9R06r9F5g95r7Eyf7/XX/SaP+Rubt3jdb3Kb39rc4HduP3xfVSOXp80Rn5125FJe9TkkplEOn5wusEthJ0/S+nKoS4giy3y0NXZVhyFoDF6h9xofrPfEWJ/CcTgkZ0heTUdX5ThigOLiv727wWftDgDQOoOnzREnV6H39E0BaMPkaMn190l9wpt+nWV0QWJjiDPgrm/Qp1AXAFhpuj8abWGDQu80Luouh/EYRRvUJ1thWxGfwGGo4aPA09Upe2PVyqF1BsehwlXT4qYlLgglA4wM8FFga4b0bGiY9A4bvMpEb1r5vOHTOY3eamjlc5iLTmRmg9NwfiR6482R3tFGEm8oAaMnEpO7cUiLS678JbQJUeQwHS7zDQwOlyi9d3j+rVeTzQ8OO5iT0nHIBG98lB5IpRcPhxuM7YgcdjDfcOFrOHuATDm7y3HQ33GDYPQqQvJCmZo8+XgOQSg2fLjMj8Vi4wMiLm9oLP1l2eTstyiuXyiL8bRflhKxTGL1c1J+aq6wfxzl2Z99Fv/Z3/jL+Ky9QIDAVvcIELgbmvxi5xjKvW1wWbV4kz5SO9Ph5CrYoLDSFm+7FS7qDoehhpIBGzNgrQcMXuOz0xYXVY/ea7jk6hqjwNrQh/tNt0KtPMUnSY/bvsFlTeQ9d0ONRrssM+9Q906jdxrPN6SEnGyVyRiUCGiUw023zjvdVXKvZRdUflgbTTFIK23xplshRkq30lqDF5sD9kON1hpcNS32Qw2f4iZ5J/9t2yRyAzreWkMAKrF3royb7Oy3KU0Ls5e6IHN/rNTyDnxnNVbGYd/VqLRHpcdULk2KFdw2PdrBQMmI3ik0xqEdDIzy2NYDbtsG26ZHZzWcV9g2PRrtcHNcw2ifd835BW0UxxrSfb3vKO6U1yxEkVPACBHRDQbeS3zzKbHCfna3g5Ih11Eyoh3MZBd6Uw8QIuLUV9kdmYkXGkPxgFJE7Joeg1doB5MJH2rt0VmdxyllyC6rIl1jZMBtSyRKPgpYq7Fd9Rgcjf+ubSBEzPGO5YeQY/eUCikek2LzjPa43pxw6GucegOtQo7LdF5NYixDFFkmZvfkD2ed0vFwDKfzEr3VqI2DVgGn3mDTDDh2VZbJB5k/hJV26AaD2rgUu4ocgwgAWo/xvgDQDxp15dAPGuuG1iQECe8Fxdcl1+EqxeJx/BjHEfbWYBioH6UoltAoj2ObCHaCgFKkEAgRKW4sfVTHOMUUJ61DbktrD+dIWdCa3HSvdi29D2432dWWU/1wHJuUEXVFMZbHth7dw5l0JMWhAcC6GXA4NthtWxxONbnlpvrbTZeUFVorYxzaUw0hA2KQqBuywHedgTEeMVLsHst1tWvxdr9C8ArauKyUCBGxaQbcHpoctzf0Gs2KYg2rlBIofzWigDYur2Nd2xz7yCmOOD6PrTxVSnvUdyaTfER2L88xlmEiW9+Z/JxXdYpx7g2q2mIYdHb3ZaWKlaLLiyNOXZ1jLE3lIGXE6VjnWL/gKbfHajXgdKihjEddO5wONdbbHu2pztaYqnFpvSiulFM3SRkAQWmUqtohxhQvK2KaQwOpKA9FCBLBU6xf8JQmqrSg2UEjeAGRWFlN5eh8upYVSY735LVtVgO6toKQAd5SiiaeM20chlOFzWUL7yXNR+FKTCmgJLyTMLWDszrP83ivC/Stga4cmsaiPdEcKjUqeiAAACAASURBVBVoPXuT456dpZhSZxVMktMNCkIix812pyrHRIYgoDTFv9qe4jhzrG6KIaS4xbRBOyiYlaW5dCKf47hQNygo4+E6A93YPB8hCNSNRbuv7ysUjyy69nCdJvdMEyBVzIqv7zTFYDqZYjsp7U9Mcca6oXdKsBLSBMSAHCOqGkdrzCmXgNw2AIo1RaFQewGztuP8+NSnlRkMCAFKlZOulTXFhwodKGYzUNylEBQfmmMcRYTQEdEJ+hsE4EVOncX1hEx1UkqnmNJOidrnMQsVKN2VExB1oDRcoD6yOzSnhHJidFsvU1qJOLpqe0qDxOmL8nmArLW+SN+TrHd8THiRXX5jGZcpQH2X7u8lsoygtEQsUz6e5jkkt+ySAZllkKltHjOQ6xfYqXAZnsZy5rQtU2w3NS6ndEHZ/bu0Hk/mAV8sxnJuNWWraJyO615ganFsblmcXzo9iXvz/5OKsfzuf/jv/8zdRS/XX4u/+sv/5k+83b/3f/wnP/OxAV9xYLn9pQ/jP/Zf/ctok7LJOz/WKyJRkGMaDWagc+llx2xqY/yWyko+K6GsVFur8gezdMdhMJBJC5I7i3cSKinIRCIw3XFixYAVNwDwfoxX47/OSXA8F5Mi5N2fNAes4Iik1ESM7nekeItMKMFsbDIpuSSfyjtBLBfJiBy3VbqrxaTw8PUxk6gsuPVxjFgiThCJxj6yC48XmaRBiIjoJX2EvCSa/3Qt1wErFoIIHSYxXfwdmX1QmDyh3O0SaS5pPER0oNbJ1bnTVJfjkZJcEHH8mGomWeCX//gW5rb5g55dgNIOH39khYrF9cgfTaECEZdYNZ6LyDFh+SMNjDIW19Mgkjz8kUzzIauk6Dia33txZ8W8LbofASPJQh6fGD/I/HHmvJayuJ7jrphIQo7zmeXl/nlsPBYeh46jshGLNri+APXD8powkkYAoyIj4kgyEYu+BCauWJPcnnz93EWn/OCZQP8eimdBTteA0H/qr1Agcyk/4ipSnXLcXJgso/w6lykUyhxyalSgsnuWiYAVdIzXjsei0rmYlDEnEE3MylyOseMpSHNf9kV5Gou+43gN90cxcnE8x8pbjtOkdliG/ExznlInEHWh7PH8xnEsMeVIFWluct92VDB55zialLtURerTSsQqUN2Y4rj0qHBGkfouCU84NhMonkNe60L5DUmWKMa1LO6BMr4xzylmCmlOHYKxnaTAlrFsXFdYgVgV9/RcIQ6CFM80p/m+KOaU1pTqC5fi+3j9XVLoo8g5O6k9vhfT6ya92so8j1leAcrvWSrJPHeFsi68QNCUfxRBZHKV3F5SoKVL9fidE0keOcwfvMeXqNL9iJmSLoo8rEmeCdkL3z+8/iUAiNO5oH/cBw98HHy76fHZoD6pXvk481+g+FTxseKVJiKmcaEcL1oS0pT/TrKIBGqQruc5LuXNx1OuyDwOlovnonj/5XlIQmc5l0ho+PtbPPuTEmd/i/4zkc4sFnd+fY7XnbdZFoHFPuZ1F8HUUpks3qyPpb7e0e7SJ+ydMj3Q5hzsPljm5x8h7+Ta96n/QPl//osvQYzl+mvx137p3/iJt/ub/+df/5mPDfgpu8L+tMt1dcK/9Au/hR8NT+CjxLU+ogsGt36FvWvwvNrj5Cus1YA3do0n5oSXww4SEU+rAw6uRh8MNrrH5/0O18nNz0iPjepxqVvsfYMftVe4ro7ow8giS+53R7Jy9ReUDiIo7EyHz7sdXjR7sgjaFVbKTuJsjAg4+gpHV+GbqzfwkNjbBlp6GBFQS5tciy4mMYBD0BQfVzxdK2VRK4eVHPBZT26sGz2g9QYfNbe4cw32tsGLZo+bYZ1dCNnd72W3zTFcl6bDrW3I7SOxdu5MP4mZux0aNMqRG1XQE1fAlbKTGM2TM1hri7fDKrPsspvaVvc4uBoXpsPB1qiUw8HW2Joet/0KtXZ4Xh/wSXuB6/qUUyNcVS1WyuKT9iKnNuC0G5V0Oc8lWyzfDquUBsHl9BAXVYfOGUgRcDus4IPEL199BiUi/uDuGYzyWOsBnTfQwmNvmxy3qkTAVU3umneWrIrsNhqiyNdJEXFVneCCwp1tck5OHt/G9DhasliVqRKeNUcY6fGyowzeLkj0Xmfr+4Xp8LrbQMmARll03sAGBSXIHUiLgLf9Ckb5bKEHyK3po9Udbm2Dl+0WK22zW6SL1IcEWUHZMl9JnxmMObZrpW1mAOZYwb2tU3sOt8Mqu1txaobSdWqjB9wODTZmQOsMsR/bKlu7V4asJDyfh6HGRd3hrm9w1bQ42grWK7IOa5djATeGrMi915nV9tnqiNYZHJIbWKU8auWyNwCnV6gVWTElItpE8BWSR0KMAgdbQYBYfk/WQIDSVvSOGIXX6fhHG3KT/OH+Km9qrY3Fvq9zfJ8SERd1By0DXrfrrDOwVwAzKEcAT5oWnx12+GC7x5tuBevHWN4XmwN6p3MqjbUZ8LZbZcv6piL3trftCptqIDfKxIQMAF/f3uJHh0s4r7AyFnXyJgCA6+aIT44X2Yvh0Fe4aHochwqbakDndJE2A2gMpa5prcGmGtB7hUa7zOjMTNCcO5T7Y2ZnH0XK0zm6m2nl0VuNdUV1eQ6tl9jVNJ5SLk4nwTGEzOT80W6Pu75Ba01mmpYi4s1phdoki3Sqe71u8eqwQW0cdnWPV4cNnm2PuDmRJ4gPEruG3GeZGby1BtZLKEnvnMEprKspI/jaWNwlrw0mJXNBYl1ZDE7houknsbLs6scxlJt6SDHBYuKhsanJYyaCNlM31YBDX2e2ayYuc15iXQ+4Pa3wwcUe1qt8L3AsJ+cw7q3OXiQ8Dr43fBTYdzXWlcWmGnDXNdCKvFeM8jgOVb532ctjcMQSLUVMninkSSEA3HV18tKh9aoTW/Kxr1AnbxStAtqBXNtjpNhOKcjTZLvq81x6L2G0x6pKaXaSV8ixq7BpBnRWZw+Py1WHV/vNA5rFw2VVDzh1ZMU2ZozPZm8RZliujctM8MyMvKoH+CBhnYJJ90NILLrrhs65tLldssnTRvOYW5at8rtVj24wEIIYiuuK2MMn+XAlbVLHKFAnVmf2UAhBoEqb29aq7C0AAEpFeE8uliGIzNLM18lk6XZ2ZBC2A72Pq9pmzxG2eHuniFV64J2GmGM+lfZ5nmQxvnHDmOoLkHU7WNqc5flg4jLegJ4QbgFpQzxtXmdLYtpo4I3jchNzvk8eMd0MTsfGfxcbnLzRw5tNMo4bmYy+yg3xEjiKVKfckChJlfh3KP6WG1pz6+LcHLoEmudlviNRtjmfEzY4FKjybHzpErBcOl7KsCTTn5QvfflKWyx/8c9v4l//O/8oPCS6YCZkMh4SfYr3IzIYIoVphEMXNcKCP4BPZB78b26PYwMlYia9yfUjxSMyiQoT3oQoJv8GiAynLEb4gkhlzGPHMpuCVISJbeaFZQlRZnISVlDm+Q5LApIS5JbRBGWdMo8iy2Gkz0ADwISwZU7aUxKyhChym3OSFyYEYeIZnk+O/ylJc3qvJzkM56Q5wJQUhfuby8vkIQyuOE8gxyUNacxzQhkAWcnjWI1J3wXpBt9jE5KcqCbkQeV5lt16cs8uiZuY6GTwCpUa17mcZz5W5q8r5eaUNgxiytQ1S/O4RNrD4HpO+sTPAgNozl9YrgOT+MwJgJaIcZaIe9jrIK9jMb65rAAyECtTzbA7dNnOUm7B+fzy7zlLcznnfXJNXVqfclx8/3A9YHkzmpmbhwTKy+/wUIBEZiUuSWBK1+Y50y+A7L7M15ZkIy4p6SWrM4Mnzg9YFgJdIp8fSWnG3zQPVJ+9RuRMNm6rdDsP/Kwpn5Vlju/Mrvez+eU5D1HAOvI2kYIYeNlLxWhPelc5bqdgEtj0XkJrD2t1dpPOyi44pgqT+Ki8FgUwWGIrZsIXdjdn0MGFvVbyWvqR1XfO0FzGPAUvk+cMJrFRfE4lBV8U98Kc/VmkeWMFHkDyuCAlOafZSd4mlOqEFFYxG2PpmcJuk6UXCbmCFvXTeLJHCVvmZCFrUSckZZ2BArtr0o1PAIHdOSeAgb0qvmhhAAKMyjor8YV1+R6jsMDopVAq7KKoD0zZlEvzUvYAmcnCtw8DEgYnpULPzcwBicDUml+WGTtwttCzzHOgw/8WoHQgXBhkydmcFMCHLYHZcjm7NFvMeVhi/H0PQJUXc5mZ6iafu/LlW1oti2nL9ZbAzRwcFoXlm7uz3ivn2i7neXZ+NqT77rFL3ZTno7h3/mwb534/APYekmOxzXeVR9R/zBz87n/6JbFY/iP/+k+83d/87V//mY8N+IpbLI1weK7vCAQqRYyvSfEFgC6QVYqZYW1UkAioCzZYYARxDNKYoZTAXCR22WigEDNAVYjoop6wps7BKquCChG+eEVloIQIrwVsGBltWRZWsidyzZ6sEgQzMCuPryVZLZhdleM3S6siszgugQgaw6h087m5Ysf/Lnfd53+lINDI7LDzOef18JCZXdYpYmNlqyQArBTtnNeJnKckReLCVl5T+K3MGS0ZWDOTqpFksWJwPmdGLfth4LoEpst1YFBeK5eZXcuUOEyuwPKEtA5OE+AlwOLzcSkinB5BfAmwy3XnYwzeuXRSZ6umK9osxzkHo+WGx5zhlBlUSyDNrLflJse0HTkB0jy3S8f5HFujyYIcJvWWmEqzfGpMt5PvgwTstJwyCJ8DgQwa749jyv5JFknaJCrXdSk9R8nUO2HkLO6d8rdRKlviuQwpHpv75jRH+T6ZPZdz2TnJPZeShdTrEfSy/JWaMveWZb4JIMU0Qfx8Xr1O4H7WDl83B9cxtc3HXAGaOaUPyzZnDuV8q5mJNEhwvtnyXgYAKwMBzijgFaUqEgAqXc4T9cFycpxyCc684g2l1OfCXJT1yvQ0zHZaSubVaH2ag24uHMbAhCFz5lIGzWV+1TK1zwRYFnH8FP7BfYzXjSBW5BANTsUDMBika4JM85FAb0gARhXhHVkOpA2HQobcrog5FleKOAnJEGnugi/6VykUQ6aUNEnbCUJCVV8cWHKIB4Ac2kE/UKT3WWA/BRBlIV8YNwxyyAUw1YoZKJbHSxDl5AhmBUb6Xgky8ZZgidsrwxEKmXI4QSzqCiCGmMFkLOY8y1IiHD4nC7lZDmAMmUBxjvsCyJU6TC1eJXiPPE5R/AamoCPLE2eumnEUt7xOzP6WU13WCQtjnb1D5oAm9yNmv8+Vc8By2sxiVZ6bSR9z4Mfn88Bm3f2xAsv73+xz7bwLLOYmy3X7Upf4iJvhq1u+0sByiBrfG55jI3sMUeET+wRGeDzTd2ikxQ/sNYzwOPgGl6rFK7fFC3OHg2+w9w0aaWGEx943eGb2eOM2me30jV3j4GvU0uFPr17ik+EKl4rIOU6hygyYAQJfq9/i5MndrmT+BIDn1R53rsky26hALJADtqrH99qnibGyh4sKNqjMDvmiPuS0JczEWQJZdqENELm+FAF712ClLL5/fIIL0+GqavHx6RJP6yO0JNdOtsh9bXWb3XCPyTUVQLYW7m0zpo4QETvd4+gqDEk5bpTDnW2w1T2GoCbK5UYP2NsaT+tj7lPLgI0acGsbXJoOb4YVtqbHEHQ+zkySr7oNXqz2eNVtsTU9GmXxebtD7zU+Wt/h5Co02mJIlimX5o6tcUJEXNW0Zp0z0JKU8dt+hZW22bXWSI/fufkAIQp8+/IGg9c4umri3gogu4vdDiuEKLAx5JJX5jk92oqsjVHgbUdMlNuqh00A8zDU2FY9TrbCOl1fWhBv2jUGp3C9OeU535gBN+0aRnncdTWu121mzauUz3PuU37Sy7pD7zX2fY0nTZvlenNcoaksrtctWmswJNdEZudTaXz8b+tVTrnAintrTSZhYre8TUXkTYNT2DU97roaF4m4KEayNtXKo/cK3WCwqQecEoGPDxKbasiEQCdrskuhEBEr43DsK+yaHrdtg1VloSRZrzunMwA6DSNJkVFERnV3atBUFqvk0meTq591CttVP3EBtE5l8iW2vrE733rmXhejQO+IiEmImI+/vVsjBIlnT/bZYthZjU1y3WTL3rElN7rNasjPMj8zbJkDgGNbEcnO3RqbdQ+tQs7jejg2mXRFqYCh19huugwQ20TatF13aPuKwKH2GZQd3q6wvWrJ0mo17KChjUcIAsPJYHPZkSvjoNHUFm1n0DQWXWdQ124CcKzV8F6grh26toI2fiTyGShmmcmPAKDvDIIXqBqXyYoYXDknMxkSEwB5J1E1NjMf9i15olSNw9Bp6Mpn4OXTfcOkQPZtA7F2NLYo4DsNBKDaDRSTHpGZHN3BwFz08FYhtBrmooe9raG2DuyG59r0yUxxnaIOmQglBgGpA0KrydJjCEihVxBrR8QmAhRHrSJCp6jOSU81RhOIDIWV7U6NSn+OYY1AJkGh46KViKsUV1x7Ah0BFHvZS/QbB/GmQlQxx55yzBzH0UIHiFYhVgGyTZtrA8UpRgUMKw/RS6iTxLCl+FPh6HrfRIqPlKBYTi+onY4IXVwTIJyASr+HnYccErmNphhQ6QTcmtrleLxYJwAkItUHYOsAdVCIOpKlTQJiENB92jisI/RAbemTRChihOujxHD1xYGlagX8Onk8WAHJ90IA3DpCDtSP7kWOG42KQI05UYxq0IAa0lxJOq9aAmOBb4dA42eLZNBprZIcUQDmIOBWBKCCAnQnEEyKNU1DFB6Ihm4b3Qn4epQxShqDiICvYo5tpbGBfqcY16gA2Ysx7jkIIADBRKgUs+orElC1qZ4ApAOCSfVauk8AsoRyLK4c6G8wgHA8Zxgtk25q/Ys6zU2a99LieS8WEuOx8tzESprOSU/zKEJhDQXJQbG/5YM6biiw1ZbjNKMo+injjIs+GbDmY2I6xtwvW4MxtlHKzP1zXPEkTpbnL4GzvL7xDFArZCr3DPJc5A0K5I0GETHdvCjbKsscJBfAKgox/l4C+2cA7HTjYGE8f1J+JuUr7Qp79WdexL/0t/8KPj5cZopr6xVu+waHrsbz3QGd09hWA25OKzxbn/D5YQspA55vjnjbrYhpsx5wc1zjct3irm0yI+l1c8LRVfjB6ytcbVtSwLxClVheL1cdYhT4/G6b3crWlcXNYY3r7QkRwNvjCuvaZkXMB4FKe5wGg743+Nr1LWIUuOtqaBXQaAcpIrZVjx/dXmZFcFMPaAcDmxQnpk/eNAOxmBqLl4cNvJfYNFT3F568wet2jX3b4KPLO7w6bEj5XxGzqg0Sr263mf2yNjazZbJyt1n1MIpc4aQADm2NyrgECgQGRzE5x77KjKiZFXYwaCqL/WGFqraJCVTDWkXKalthu+lw6ipoHdB1BqvVkNkGry+OeH27wdWuxbGrYAeNq4sTNtWAj28uiaEyEysRQRFZKHx2w9sfid2yZKfcbbrMnHo61ohO4pe+9SliFPiDz55Ba4/dusepN5Ayomtp04AV6YvdCVIA+1Od2TC1JqV8u+7R9hWkDHi6PaG1Boe2hjEOfW+wWfU4nGqsGhq/kOwSFyBlxNW2xa7u8fHbCwCA9xJ20NhuO1in8GR7wsu3WygV0VSWWG0TWRQTTh0PDZT22G063O4pxrKqHL51/QavTxu8vVujqm22xFivMCRmQZNAAc9nZoRMu7WrxqIfNIzxtN5O4XSqsV73MMpjf2xwfXHCze0ms1Y6J+GSS+K6GXA81ZllVAhigeSdfm0IJBCTqUDXVthsOhwODS4vTjic6sQEyeyVJHfdWGKUtBTLAwBPLo84dhUBkWRtqWqHVWXx5m4NjsnRxud++76IsVz3iFHgdKR1rmqLvjUQEtCGmDNjAEztYHuND5/fQomIH376JMf/mIrYOJHAlVQRu20LrQJubjf5W8zWHDdoSE1f9YvdCW9vtnj6bI83txtihUztPL0+oB0MgSersGos9ncrstB4gfU2pdK4a1CtiXXUDTqzTH7rgxv84PNrBCuha5fXQ6mAFxcH/PDVFbFO1g79scLmssPpUKNZD+g7MyGu0rWDVAFDZ7Da9Bh6Q8ybx4rAYyIQY1bLZjOgNg77w4piq1KMGa8/38+u16jWA+rK4XBoEqmVxHrXI0agPdRY73q0p4oIrWTMADIMCkIFfOvDG3x+t0XfEbher3tU2uP16y1M4/KcCxnx9MkBr15eQDcWV7sWr17t8OzZHq/fbDMT5uaSNmpMYgE/nBq49A4SKd5svaH7pu8MhAzYrAbc7Vcw6T3rnUKwEtV6gLMalxdHYjpOls1jW03WarPt4JyiNUxMs95LXOzabC0eBo0nuxNu7tYZfDPgdlZjt23x9maDr330Bp3VOLZ1fta1DqiNRdtX6DuD7bZD21bYbVtIAWxrimN1XuL2do31tsfTzQmfvt1hVVus6wFGBtycVqi1z5snWnt0bYXtljYrT+m9frHpIEXEqzc7VDU9t8OgUNcOlXa43a+xWg3oe42q8miPFSmxnu41IYDhZLC9ajEMmubTCZjGYbvuIAVwe2iw23R4+3aDy0t6b9Q1fS8+vNrj+59ef2G9Y7PrcLxdUYxl4zJRn9Yex7sGpnFwg8J622MYNIIntl0A2F20cF6iayusNz2cU3BWwTuJ7UVb3DcxvwfZVXroDcq4RO8kXjy7w83dmmIsB4XNrkPXGUorkdpQOmDoCNmstz1Oxxp1Y3O/9cpmlt5g6bmBAHTl4K2CrjyNwUrUawvvZCZL5LbrFcVU2l4DQWC162CtAiKxF/edgesV6s2A/kA6hpCRNmWCSNeP7ygA8IPMmyvMvgsREQaF2EsiIAMg9MjMG50cXZ9TH4gCcZAjuRWTTTHjLpPZWTkSz6k4dWMOgphu+yJ/NoMtdvEVkfpIgJuts0KH8bqSlI6toKVrsMCU8I1jJ9nyDIxu3fyX29GBxqLDSOrHCJPJuVxByjUv2TKLkaSp/M11gBGwlpbmJSBZXJNJ2diyyt4GM8tlFAWBW55/TIndZpbgxViSEggX5Y/+6n/wM3cXvVx9FH/tO//aT7zd3/y//rOf+diArziw3P3yh/Ev/Df/Ik7WZFcwHySsl3BepTQa5NLEwfK9pRcsWwb4Y87xW2zp0jLkGKBuMAk4yOzeA4wpEYZErQ5woDwBDQCZbbbMoyMEcgB8Xdsc01PG1ihJMnOMDoOnvKmTHkbuh5XqGEXOb1TXloLmPe3+cwB/6XqVaecjEqPtlCBAqjBJ3MzB/RwzFIPI1oEyATWnF5ApbQDT8ccUz8WMu7Kw0MTkysVU+0p7Utp47sNISe9SupBJbqQkd/nXe9oRZ+p+nqscR+doe7Fap9jK9FEv8ziFrESDdldnrLClK5dUEZF3fBXFdXGMEc1VYsllIgD+KIiY25YiZsCEiGwJiTFZRJh2nllx43i9EBHBpRivxLAbI31EdVJSmFF3EpuU7rP8IQ7JfUxM+2C5+foYRWK5TWy2TiYZ1fT69AEVKub62fWr3F1mbzCWzae5SpT2Ma3nhKwAmMY88U4xx3CVLLosd5rDSfyPKD7o3GYU9+K2clv88eZ4rtrTOrdq/KAxs2s5Pp3eB4Mc6/E9XO78mgAMkij6WcnibdlqpjyoSGy03E66RyeucsX1og6knEWR5yWzHpvi3JyZVs3mKGK0ppVrUsZTsYz8qWGljRW68nypCHDbHBfHSo5ODbECWbLUskLEv5sAWDG6FLJSOcgxpozvocQKG1UkC+BArLDMkMuWtbxegsaY2XYFRpbdsl2JMQVCTIpUoLZEEIi6NDEA91hh9Xifl0QYWZY49puZdlPsHStpkcfTEDVrychbyo6SFZYZcLO5A2RJ1LQmmQWWn9lyjPxuYzZnXi8ZR2beQRb3DkZLmC3uC4EcA5gtOWJkKkZiCEUka0wGFMwYPCRGY043EQQCW1G/YIk6QlpWqJHv2SjpOFuNgp5Zh2KyyEWQdWmJSRXI7Ll874yKOWZWI4FQB+ozzRPdU5i+K/jZBMaUGbxOxf1KDMfI63nPy7VU7jM4GI9nNmCM9zbPi/DM3IsJK2zGAGw5L/AJWw2jKMaMVK+w5LKsi3NUtFVaK3ObsZimMN4jcy9Nkj1OnstFf8sS6PB9IcZ+H7y2bKN4zSx5jN5zD+W5XTg+aXNhbt4l05IlcGLFnMn80yjv4w77rvI7f+NLEGP5cw4sv9KusI2y+Pb2JjOU3toGUkSsFDFUvhnW0CLkRMsHV2OtyS3tLrGbaulxchWuqhZvhxXlrgyUY7J1xGL3netXuBuIwVAiZpfO/UA5Eq/qFl1KBtw5g6u6xe1A7q/MflkWHyRq5VBrh5enDaSIxI4YJGwCWb3VuL485TgdZuIs3S6pP0q8PDiFZ9tjPlYrjzenFbbJOrnvalyuWwgAQ+HGuLsYYBMBRmsN6mQx9VHAyJAZF9kCuK7s5HolR1dAVxB8MEMjsyTyNUpGymWY8mFyTkPrJRrjcOorNJWFS/kfL7dtzmvJDIHOKVxeHOG8glZjnBrHPpUAfFfZzDrI8VDdQHkcI4Bae2jl8eoNuS4/eXKA8wqD1dkKyeyRDB7bntbTrMYE1DlhfLLMxUg7zFIGmNpnC+AwKKzWDsOgYdY+p7vheKahN/BeoEp5CKWMeedfqgA7aDSbYRLnxCCYyTya9ZDT5NSrPsklYVsDaQKa7UC7zmnelA4ZyHGbZUoc3kjgBMvkMinTBkCE2QxwlqyaZmXhOpNzzAGAECHnV/ROwqxo11sasmhyrBVEiguLgsC5AHTj4QYNs7VwvYZqHOm6MsCnuQYorQ9vhPCmgms1hAnQzZDjyIIjIKxWHiXLIBOQyCqMYT+WvtRq4ygWzEnIJv07AV4hqJ6oPcJRA1FAXQw5vipYCbVzGbjHIBB7Ral8Vj5/jDkuK+eXE0DsFOTGIRwMxMol5kO6z+NJp53sSCBykBBrNyp8aYd86oLpx7ivvQa2bgTZDBqDoHMbT9qQillP7gAAIABJREFUo3Zjp8i9cpAEavMGDuj6ADqe0nXAi2naGRlH8D/InMojg89kJZmARRUBKxM4CDS2KkL0isAQA0FO85LBM5LlAFA3GqGJKc0DIDpJ4GLli02C9GevEDYewkmIk0RYB6g7Bb8OOS9eBiRpCoKJgMKY8kQBsk0bZalPaQVCHSF7Oab2kKDfKkLf6qToxnRdASYFoA4yu8ZFNSq5+i7lLxSk2OtBwlcRMgoEndwLQa6X6ijh1wHmtQYkKE1FkElZllnxDiaS66iOkPsUApHSf0QB+IbOywHwFd0i0pEQwcTs2sigKmpA7tOzYAgYS3ZXbQrAoUDujj65alqR3R2DQQEyqY1gALWn+WOXSeEAOUjaH6rJbdY3EeotucJKR/VVp2G3X1xDVb0c3TkdyBUWJJtbJdlVRNWLnEKE40qrNyLNP7masjthVDHPC68xt5n3EtgVlt8ZCtAvFVIUDrmqDtQngQyRgeuY4oXuDQb+MW16iEgyTVwlOWVLkSZGWjECytR2UOMchLQWchCTDYegU1s2rSfG66Mo5oJdeJNsXIQvgGaaP7oXCjCJAugXYD6PcwYu5+lbIj/H7NZagjMG7gv7ESxT3iQoADcAiBDHONYSiPHfmRwUCyoWj4k4brhMwauA9HHZFRbj2KMUEGFE70uALcte9hHHunxdrrv0KJ0D3fFMnUcA01LWd4LMco6+rOWLe+N/6ctXGlhKRAJo0kKKgD4oKBGxUQNqabEXFDvYaJvSdaicEkPLgEo5IjKRlIagkgTcmPyCUypU0qNSHpV0EzKXWhHgqFJahQCRfzPbaKPshOzDBwmpImrt0CiLOsWI1crBijGtg1chp3sIUcBLInSptYMKctJeiAJRidynlyS7SrFyRnkCOOnfTIihJI1BCJXBJANLtuBq5SGETOyayCQafD3nBq0SwCM31ARYZIBWYkISwiDaplg+l9xXAcAkoo1GOwyC4uwq5dELAqFKBvSCrMNVItkwarQy+0CWBZfkY3BLTIfIbJaq6LPSlCqBXRE5XYFPBC9BkMWbx2tT0m4mFAGIbEIDGYwpJhoREUqN8WNKBmidyIwksZVKiZyKRgiy5Po49iElgR8GkGxNZfBEjJU0t0IAIRAlvwMgpciWdqUiLJDj2WIEwAmsixg3Gk9MVnaJIGj+qE8gBpJBxAipQMBLjflfZQIRTOxB7QcoFREEWbZlsr7LRMrB9PxCAGQ2jQggUMdjVirCZ7lSPzFklzECpOl8srCzdZJlCV5CKBoDW9Bp7iS94yOtJbNiZjAh6IPtZRz/nSy6UqTYIxkRJIDUdoDMbYxkJEhpNiPAhCJlEZgci4LbJUtvzjcLmotsfWPFqbDysOVIqgjvU25ctg7FsS/BFqTAimBqk3PvspVZYvzLffPHW47XRCGy1ZKs24WcbIVOc5iBZmkxloUcqZ/IbmVJxpzXMeWbnCQzL2VCApn8m8lIBEZrNDCORWK0mgkCvFlRZdBXPGsRRf1kfuFrCUCCLJO8HrmtNAYnxrUqlPUo49imwOy/iIhiHCx/KU+gY6ycZusMg9N0r/HxDFyK36WCXt6fuZ/CSyIKQM5jrLiujEmZLdrLmuF4LOvIpZWt/A8MgpJ8vCaFzGX3EzkwbedBq81jS3H9RMmdtRvnc5L/TUKUc54BihinKV9Xgoh52+cU6BlYWJbjjIIuFuo+VFi+AnhMbp34CHmxsI6PBQezNZ2MKS60e6bNudxngdBC/6UFT5T15695Bo7z9s7KFGd/pwBrydNDoDhXth8LWXP796/leSj/8rmJrGL2+x1jwUK/LOc9cLvQppj9PVcmcv9J+WMvX2lX2G/+ymX8q//DX5ykGilZRkvWP4+CWTOxWTrexuM6qX7JNgpgkm5jzsA5Zxot81UuFWbMnLNfLrGpLjFdlvVzuo7ZdfP2zl0//X2erZPHO2esXGpzPk9zmca27zPHzlM4zOswuON/cynTKJTynWMLPSc3p/WYz9FDY+KylDbhnGzz9h8jX/nteJdeNNcF+fe5e3Mux5xhs/wLTMfI7uNzttBzczeRc9ZuWZb642vGPgiH8KaBDyL/Hscy/b0kw1KfDxVOWTIf7xJb6rln8KFnc7HP2e+l7/l8vebHlurl9s7MMW+WlK78j5Z5JsdSOTfnj1mX0u1+rDuee19ZyjYnbZT1J9rQ0vXL74H3LQ9OdUa38QE5UhWxPE+P7fShZ/RB5fJRz9E7q0w1zIyAi34fq9AutftFy7l5fwjcvQsg/TjyPEaOM8fZFbr8u3jdI8Dr4+WY9vEoi9cDQOi+Jez8XL6zr6U5epds73v+nBw/Rns/kX7PvaPf4376InXeG/z9OJAlAv/w178crrD/9J/6V3/i7f7d3/kbP/OxAV9xi6WLEp8Ml3hmDrBR4bP+AlIEPK8O2OkOv3f8ACtlcedqXFcnvOq3eNHsAQAft5e4NB1q5fCq3+Brq1u86re4MB1cEHjdb3E7rFArhz+9fY1Pux0uTZfZU430+KzdIUSBX9je4NYSU+jB1vhodYdP2gtIEfG8OeB2aCZ5JIegcFF12Okev3/7HEJEXFYdhqDQOooXba3BB9t9tpLeDsRkutZDdrscIjGUWq9wsgbf2L1FiBK3Q4Nd1eOP3jzBrulx1bT4ZL/D9bpFrdwk2frXtsQKWyuHt+0GV01LqUGS6+3bbkW54hKgu6w7nGyFwROJUKMdbtsGF02PzlE6CwbYK2NxGCo8XZ1wGOrM4rk2A950K1zWHV4et7hoOnRO47LucNOu8WxNie1fH9d4vj3i5rjGph5Qa4dPb3ewTuHFxSH3x+kjXJAYnMpAR4iIqxURRXBdANh3dSYf4jn5vU9eAFHgG8/foHMah67GqrJwnpKil5sGbw5rIgKpLUKkdASDU5ACOPUmu87ujw209ljVFi5IVNqh7Sus6gFtT4m7OQm4D8QKezzVmaAEICtnpT3ujg2MISKLza7Lrq7GEAuq9Son2r7ctuisJuKbNbXTW432ZgW5dthuO/SDzjntpAqJLAc5PlipkONzVRFn2/cUb+ydyrGwdUPEL9ETCUN3qNFsezhLFniV2rdWwQ0K9cqi7wxM5RC8RLMaUixyzCRCxO5JbXenCs16QHuooWuXrcBDr3Ni7qE3iIHceplMo7utIRuPOrkV20GTy+wgYbZDTp/AJBYxCujKZfdf21LicbOymeTF1OQK6wYitYCI8FZBGQ/7poEIAtXzE0KyBrtBoVrZHI8cgoQ/kMus3Nr8LmP3V46DFQLwewNz1cG+aaAuLKTyOd2Cf1uRu6eMlKuvU9kFV8gIfyQaSL21xGQqI1QVxnQMryrEpwOkCfCtHgkrIiCPCvF6IMtJr6A2Fv6ugtxYhJMmJtRkBRaS2E0RBETjyUXXBKCXEGtPLrTJZVek3IHxpCkGbuMp/lEA0Ml6yqQcTkBUAbFTEL1E3LrsXisORJsZ1x7ipBDrMMaRJvdlJtEwLzXcNlCdKCBPktwtn9jRpTbV1bcK7toR6+lRwj1x0Dca7tLnbXx1ULRjnlxf/Spd70GxWSZCniSiiAgVxWTJTsBvA0RizowmJhZQctHUB0mWRlA3vo6JVZXuDX2S4AxRzOYaFaCPSRFMrq36KOHWFGMXGmL+FJGuUScBexlQf05Mqr5GjvMTIbkoVuQKq04CoR7bl0NyX5SA3QaoVkCfBNyW3D6FJXdJ31DdqJKbqyO3Vn1KG68Nucryb3tJ7p8ipH57itFzGzoedWpjldZWgupHwDURZk/ussxYqzoB3SXdYAXoFhh2EdVewDXEwho0YPZA9+yLa6fmKMiVVlCfJTvpKDugOmQXYo6f1Ec6FgygemS3VVpPJHfk0ULEzKwAyQ6MinhQQH0L2A3SHAL6lNaxYDYVHtldVre0TqpHdk9VPdK9PL2O153Za4Om67OrbRjbVvSZga/pJtbHccyCWWErks+tx/li12nVk2U7mOR6LYp+0jxQ7HFE1AK+YhdsakM6Wo+gxCTOmq18QYvxfvd0XGRvApHblTamunEaFynSM1KswWR/R46yZMttMf/Z/ZdjPFlGURzjPRRuD8X5AkCza3NmgY20bsqyW/J4fZYxeQyULsRUoRxjnDLOpr6XWHbZ6prvgyWG1/nmnpwyv57bnyktsmOb4ximjd6//lz50lkvv8JGvXeVr7zF8t/9H38Ve99MLJWU0mOaV43PD0HnfIWU648An40UZ2gDX+tRSwcXFXqvxzx/KTegixKVdHBB3bOK2aAozyJEBiRlH2XOQi19zm9YWkqViOj9iPvnuRJLKxuXMi2ICxKNcnBREqhJsZllXkkAOeUEwOk0dGoj5LyEPK/0d2rNLS2dc4snWw05JrW0ILJ1sBxXCRDLVBBVys9X5h/kOnMLbVlKy2a5616SNHEOz1oRgG8dsYJyXtHSisl9cG7IJWtxmSuwkpSPtJS1lDnnvSsstSqNzybgHotx83neFOBr52vK/fE8xXQ/qURGxTG15b0yL/wdYxdpLmJ2bm5NnVvtz1l7y+N+Nj+8XgC5JLPcpXU0y5nmcJInkXM4prjl0trCcbZLx+eW8OyGHlh25PhhtoSWFh1m2R0SAVZJUsVj4mtJzvNWXOo/wiXG4FCsc0h98fXlfcSFY49dsdZljkijPJxXee14bvlam94LAsveA2WZW0fL68rnrnwWl6yofG6sj+z2nQm+0jEmNWNL3DkPAXYFD7yGyR2XmZhjsea0mSJzn7ypwqyc5fVcKM/j1Bpakp3xNWG21qVltGT55jZZsxJiHNPkUx3FxN2cNxQmVtbS0pfmkOPGl9hJyhyR5ZhY0+O5yvku0/3EY5jcF1kuUL5JjOMbCcIwIVWjdpDjq/nTNreWco7IOfkYxIyQTGAksSnmg0nGvmjJpGkY1zAPm9ehXI8oRgWY7x9mAy2tsPz7IQtRaZ2LYiTL4rGWZFhZ4JjJe0qXcJID2UX6XsxX2Rf3N0dUS8fLD8KS/EuxZfO+gClRzryd+Rzx9VEsApZ8C4Sla5ABXLx/uyyPYf6cLSGkeed4R9vzfh5T5vK8ozwg5cP33VJbj+nzoTl8qM+Fvh41b++UR+D3/6MvicXy2//KT7zdv/u7//nPfGzAV9xiGWPKC4kRXHiM4Io/dH0Yh6kTJdmQjvk4vmR95ETf5Kp6dHUCOD7Xz4AIEa03qBNAzW0hgbIEKl1QUMqhD0Syg5RAnkuIEn0clWUXx364cH/jNeN4ITABFqULcOd1PjckwAyMie1LGbT0GVRSnyqBQg0tfHYbDlHk7bDsipuALjJYLqyIcQRNc4BWgsVyTDEKBCDnp+ydHglZCuWZwSHNvYSPuOf+msFK8eIPIW0mFOvQOpPBG8ki7wEulnmIKitY3D4r4qFQbHuv8/iYlMljBHMyKXsluCvjccMMIAKADTIr7eU71ifAWAI2W8gfoshWZgaLc/fieTnnxlfO+3xuOBdmbiMBnrxOZ9rlZ4fkIhCXgXWS27vx/mT3VloHwGOq4APIDNBAAVRnrMdje1OgGROgnNQpzsV0fQmYT4OZfLiliHDFXGRlu6izqBPk55TOu+L+53nqZmMLaY64DK58lsXkfgWAjmNauU76HSNgk/V4nIvR9X3Ztbyce3FvHuf3Siw2n86dy8+7G4m4WF7v8UCZ9u9c2Rfg3FgzBDUBNd7pBMASaZWIOS8mj8VjCqLmsyFEhPXT94Z3U221XHMBZFdjMGCag5XZ+Pl3BrysqLmZQMXYeB2GGZEcyxwLsBGWlOdcOQIxMU1ngDgDsVyPSwlqSuDEfZRKZsRocVoqCTdBYCSiKt7vsWxjgYCE5wLyXAfvLrGQ+Z67tChiYBc27B6MaYgLx/g4pmPg32KQU1fWQERdsZh/PjbpMyy0V/YxWf/ynJhWzsBKLmLOyTAY2M3X+x1FzOUoGjwLNB5qdw5g31XnAfAm+GZ8sC8x/X1ufd9R3g1GHwFyH9vevbn+Mdp6ZBvvXe+x/X4ZS8TDMTpf8fKVBpYBAnvfoA86WykZELECxBbIMm6RrSrANK6SS+vNpD63N/8LEEBdihUswdJgKW8TJ36n/kaFtFTqQ7Y63rcClkpdKcOSJaEEKMvxXCpbg0IUQLLK+sK6wspIH6e3Cc9Bf+b2YQsvA7el2Muy2AL05vEVMj8Y5zNrs7zuXXFs83ODV/eUvvI74ItxlPXOWXL4mPdqcQxL83IufvNcDNz7lDImsWy3VITfNddjuQ8eYhSwUA9c85Bs0zV08zmY9XpWn4hT69FE4gfG9VCs3zzec6meK+ZUFpajc7Ft74qDfJe88+tLC9i5ci7Wjkifpu2FsDyHS+W+9XFsd7ENBlisfJfawT0Qk0CrHKkOH4yRxOweWdJ0WQGb9TmPrxSC2Ynvy5/X7J7CKc7IJ/K1S4plLOpkXXQGWnI3vE4zEFqu72KcaGlBK4BujPL+xDH4uDeMBAwTkBnlmyvP4r5CntPjpONLynpM/ZbXnlHqp2wwxfVlO4XL4RQofbF36KSPpd8FKcvZeEXufnFMZ86dA9zFPSCEyCBelHXL+53fY2eAXx7GHFgt/Rt8G41tPgTSxCJoHa8obsn71qkz4FLMz5/rX8x+F30uHV/sq5jC84D2/gy8C3Dfb6O8eEGuB+ud+b49SvZHHsMjxvFTwEzvBSJ/fjHbV6J8pYFlFwx+7/ACg9fZ5bMs/IFn11Euc/AEzKyARWHFnwFoeZaBxruISfjvQy5lcyBVyseAoCwMDs6Rk8yVm8eQV7yLTGTezeL379HA5P51949NFaUHFad3yLBEQlP2O1fM30V4MbqEjXXeBT4e0y5VEve/srN+z8m01H92pcPy93UO1sTCvydrXX4ll/79wJgfWrd3lgcUhLOXFHNS/n50fwDuKdzvoVTkcmY9z3ZdKGOlW12ev9JKNJfhMX0sKdvzS84puEtNv9dHf3bPnAOWS/09Rp6Jsnim/fdcj3v9xUcO+dxDVhQGIJO8kgvn71+HyRpNYFY6PlG8y6l4l8I6Pz/TjDnn30NWozz1j3nRLPX9jmd9sd+ltmf3QJb9J1UeWp/HyDc/d+78Uh/F/6dTNgPb+cWBcdNgqY+FF/6DwOihd+IjFAZRrlF5YL7JNn9PPRIELpbF+/t+tS9kDX1H+cIWtsfcCwuv0y/c7hcFlO9q94u09Z7tfjVK/IJK0FejfKWBpfUKnx4v6N8pZopBmyisg+TuF3PuxXncmo/kvsmxaxEgko1Un/IsktuUSqkShIhEfiLKdAsxWyzYTWkex8MsixzH42auVvQX9MFOqSUA3IsR4noxijHeo4gvEXJ04xIpRkuKCI614TcQp3zg41yHX+zk4joqYJyOYeL6U7BGlqApp0dgYFMcz8pyIoAp421y3JSTlNePrSkCiJ7Gq6qQY3H4fIxiqkwAEDqkZ7ioU8SicG7AMCRSjirV5xieONbN8z6Prynma6LwcwJ4bkfEMb6l1ALKL7dLY9BFhzJSWyKOCccjxnFwe9wGE5lwbkJWJDinYJlUXhR9xOIYy8qF/5n9QQtFgN3OYuqvzEXIdUoZ50p9mbqijInia3mcPHdL269xNh5gTNguSGlHFODk87GcX7BsmFg4ODE7p7eYJxPntllJE5aeCyKKSTKUSmxMICFZZKK+z8LI4EIAY7J6K2nN07tJRAHB7pUCowylsZjdRTkvXLmGEZCDQKjS2jHxTDovHegc0nqolJA+JVfnOSjnLicXT7neRMQkoXgs1o2JbyZJ1Od4L9C8CzfmOeR6MrmzlgQW4/2FcU4iMgkMy4SQcvaZeI80QziRjwtP/5ZDuldYPlco6KD7ZX4blgng85hKC5Io6smiPsY2cj5BYOI+GIt3RyYKSW0Kh5zrMKZHpaybcyemuZtYbsI4j5kUhPMZlo+YFjl3I+XCpBN5vdJ9FGXM487PUTrGvynv5ThJ+V5I9w3/DSoW98VYR3K6FjGOoSRHySQnDok4ZLx/QjV97t6nMAkMDWIKenjdoqR+87qlPW8mpuF1j3yvirH+BGMV77TyVcFzK93YNj83VAGT9UUxR/NchxMsV1z3/7H3Zr2yJVma0Gfj3tvdz3jPjYgbQ2ZUZWah7ipRNDSCTiFe+AO8ISHBPwDxG3jop24JiRZPPIPU/QfqhSckECAVArqoLKoyKzMjMjLujRvnnsHd92ATD8uWbdt+/NwhKksZkUqTjo77HmxYZnu7fWv41iFQOVZ/meOalAYHazq/V8tzLef65/ckt1/9hlfvq4WCIrfDx44qL+q2xXIcOPxfP1Px4Fo8PHdY3gi8+XelbvOxcmwcB/fUOS0f+/9o9fX51/TlKEjltXi4ro7U81ag8cg8v3P5Jvd8W8rvgeW3s3zcvsK/+Df+R/xkeh8xSXxoXmGIBl/4Czx3Z/iT7jN85U9xqbb4zD3BD+wL/GR8BoWEHzVf4oU/wT42uNRb/GX/Ef6geYFfuwus5IT3zQ0+0Le4Dhv8r9sf4o/aL3EfW+xDgxM1wCWF79uXcEnhL4cPcaW3CBB4qu/xF/uP8MerXwEA/nZ8igu9g0IsJEKtcHjpT/DSbfDvrH8OAPjSn2ElJ6zliFY4nMgefzV+iPvQAgAu9Rb72ODar2FEKOQyV+YeChEfmlf41/0ncEnhfXOLl/4E/2T91/j59BSfT5f409Uv8VfDM9yHFt9rvsap7LGLDf7f/Ye4MHsM0eCZvcEvxyeISWClpnKsFQ5DoticF9MpNmrEmd5jjAb7aPGxvcbn0yUu9A7b0EKKCIWEl26DK7PF3/Tv4dLsyj2v3Arv2Xv8ajzHp+3X+PV0ho0a8Xw6xfv2Dl+MZzjVA37UPce/3n2EP+y+wovpFNvQ4A+7r3Cptvjz3ae40HtsQ4NGUgzrSk1YyQlGBBgRECDws/4pfFI413uMUSNA4qPmFV66ExgR8OvhDH0w+I+f/jkA4H+6+YfYqBEfNTd46TZopMcX41khWDIi4vvdSxgR8NlwCS0jblyHUz1gjBof2Dt87dZopMen7UuM0eCX4yVWasKt7/BR8wq/6K/wQXOLL8ZzWOlzrC6RB/2oe44T2eMn/YcAKD74xq3wg9VXuA0dvtd8jb/af4BGelzoPV75FXahQScnNNKjkR4/3V/hVI/4pL3Gz/orxCTx1N7j31r/Ap9PT/CXu2e4tDusKIM6xqhx5ztIEbFRI+58hzFqdGpCpxz6YDBGDSMiPmhu8cqtsNEjVpLWyBfjGT5o7tAIj8/HC/zR6kv85e4ZnpgdAGAfLXae5umD5hafDRf4oLnD19MGjXR4Pp6iUw5jVLi0eyhEdDn/66/HM3yvu8bf7p/gj9Yv8OVI89UHg0u7xy5YhCTw1G7RKYcbt8IuUyD+w80X+HI8w/ORlE+nZsCJHnBl7vGT7TMA5PZ+YgZs1AgpEq6nNXSOg75qtgCAXw+nUCLh0uzwfDyFkQEnesC9b+HyddfTCv/u6S8gRcT//OpHheDpqtni8/05tIzQIqKRHn+wovXzk90HJU55rSa4JHHvWho7BH6w+gp/fvMJ/vH5L/HX+/dw75qSq/ZPTz/HtV/DR4ltaPDUbvE3u6ew0mOKGh+1N5Ai4afbK3zQ3SMkgVvXwWZU9uOzn+J/uf0Btq7BB919GY+VHj/qnuN/u/0DTFHjyu7w+f4cn26+xhf9GT7sbvFiOCnx0j5KPGn2MDLg5bjGJ90rXLs1Ls0Ovx7OcNVs0QcLnyR2nublg/YOp3rAz/dPcGoGWl/Z6+REj+iDQaccXk0dPuzucKp7/LK/hBQRe2/x/dU1AiQ+35/j49UNvhxOMQUiCDvRI1yS2LoGVgX8++c/w8+HK7wYTqBlwLP2jmR//z7ODFGI3udY+n+w+RL/580neNpu8XH7Cv/X7Uf407Nf4S+3HxDjdNT40eYFXFJlvTwfT3HvG3TKQYuIW9fik+4VXFK4cR2MiLhqtvjF/hJnpscUNcagsfeWGMNdix+uv1qkw3o5bvBq6tAqIoD7oL1DH0yO+7doFL0zPuleASCegXvX4nvdNX7ZX6JRHi+GTRnfLlh80N7hb+6f4seXP8M2NPhqOkGnJvTBopEenZpw41al/8/HU3zU3UAh4sLs8Mqt4ZIq6+mH3Qv8xfZDnJt9eRf8ajwv78Hn4wkaGfD1uMLHqxv67RhP0CmHZ80tAOD/276HC9tDioidb3Bpd2ikx2f9BS7tHltvsdETvuxPAFDYyXm+/uWwwaeba/TB4N41GILBue3xUXdD743hDO83d/jp9ik+3XxdGN+33uKP1i/wv7/69JvvO1Y3+Pn2EjEJXDZ7NDmP9VpN+Nn2CS6bPbauwfvdHe5dC58k9nntf7qmPr+aOjxp9hijwhAM9t7ie+tX5TllToUTM2CKGiER4zyAEt6y9xb/4PRLfNZfQCJh6xs8627xalphCho65+TWIuLO0T7i/fYeX40bnNs99t4WuUkRcT2u0XsDk+/bmBFb12BjRvgosXMNnrQ7TLnPVnpYFXA3tTi3PSIEXg0rRAg8W91i7y1iEtiYEXdTh9upxfurezzf03xqGWEyR8WpHRCTwP3UotEePsrCkA+A8neLCC0jem9wPzZotV+cAygMZwqUz5wNDCmJwkg/ZLZ7kdtn7gElI/qJGN1HR3mza8+gECVWxmHw+oFnGZ0nxTWRqkmEKEpObKsCemfAabGIkyGHN0Viha+9ztgIwu1qFQpxHJ8PkfJrs7EkBAWrAzwTjsVlH5kAbvKKcl8fWddFt5vfQ/w9cM7pqr7iTXeExIqNA4dEbIdeVI8RrzFZ22H4D4/hdaEpb+Ut99+//vTvy9+9fKdZYb/3J6fpv/pX/x7GaBCSLLGA/FJ2SZXYSyPCA+bYRvpyjI9zYbIarqvO41g/li4T4TCj7GO5IJkUqCbu4bpCEgW4ADMjrJZhwbB5yMjKbTK5zyFz7ZTzbx50QHwxAAAgAElEQVQys9bssNTHtBj/IdvqMUZPHkcdC3qscMxrLRfOIXrIElv3vZZz3T67O+sjhDPH3JkfY4qtX2IAkf+QTGemxcPcnaUOLPMYHvvMbKdMlvMYI+yhSzT/kDITKkAvS2ZOdVHCPEK2U495QQKU//M6UmJmc62/AwsF+VFlYF3nsZc795X7X4/x2A9IXQ5/EOoYYFnV+5hLOeexPHQhfxjb+tC1/NjxmvUVoE1A/QN7WC8z1ho1q+uPxcfW88AyBR6u1dqTgr0ruPjKnZ/rrTcDsZL1YeENgapZRavx8Ln6OHtZ8P/D+ubPj4+b70vpuCzrungjUfeHmWG5jkP36rr9IkOvIFWs5pDMF1ItN0oAEIPMTKtAivQ5Bko9QxeLRf8giIjnQd/ztSLPx+JzLpWRZJEndOEFUdqs7qvHGivZCWTvjzQ3ADxkKS0WwqoDhx3LHgKJvSQq00Uh7Ali6eXAVhmuiz0M+H99nr0SqrQQC4+G2kOEv5dJyvcKkEcG91nkdg7ZT4unQzWObIX/xiUe6SNYBmn5/eA9I0J2dZbVZzGfY++KhSdDsbotjydBFv1iDROYXVz5PlTngDnuNFRyrGVWl8Px4RGLXJk7HGWYLRY9tvpV1rsHVsRHLGDF5brIorJUvs7ydlB9feyBK+zBM3F0rMfKwT2PWi8fK28CQnjL+o49zw8qwcNxv01bb9P265p9l8ft7xmS/OSffgtYYdsP0o8//s9+4/X+2U//2W99bMB33GL59bTGv/rs38Z+tCVPYIwSLlD+PWN8oYz3XsHogHHSkDJB6wDvVdm0uElD6YgQ6Eeec/mlJND3FsYEhEBvNI530po2LNNoyuZDqQTnKBcggJyfb7kpFII2MTEK2Ia0biH/iPPmi/L0mfl3Wkaigz94CUlF/jRK0xj42hglmnaCcwrBKxjr4T0l1ZJy9t/xk4ZUOX6romHnzYjUsTCUQtB5qVLZKKUgstzkvIHJXYxeQOqEMElIHSEkKI9eFOTi6iWkiYjZvSwFCWkColPkBmwDwqhK/r0UyAVWygg/asqlF5abn8ONVcx5EWvqd2lCuS85CUQBc0LWO7c3gEyQeq47MbNkfnFLS6kcFsfzBoZddwFAGnKJ5PGkKKjPXkKohJRdueofVWkDhATCqOa6c/4+7nscFd1Xb9oE/6Xi8ip0Hh8AqATVekQvkUY1u+gCDzdkvPkrrp5VGzrmTQlorEGS6yvnIZwkRBOoDVXXD9p06DjnTMz1lA1i7icSlu68Oo/JxtlVmDec3O+cuxL5GQUANIFciCt3O6j8V8tX5k2uSLPLcRJUZxLz/dxnros31dyPNhBA2Kt598KuwbwmBWguBYCpignn+mrCERMp12IbgElmN8Ms0zbOfeH1MVb1ZVdf4eRD990EyjG5V7N7L7sui0Ry7hXF+6kE4QSSjRBeFnfYhVtadrcWXiCZWNYOX8+bv+LGmPNvgjfFEbRhTpjdbdmtlnN1+sx+GShXJI2NPgs/9yepvPnObqJpnWXHbpOaZC8GWdyhi6tmEyEGCSggGfqc2ggxyrLmUrPcMQsv5828ALkv2yzjPL6kEuWwZJ1iyOs3579M7HacnzHhxbzxB2iMLLuSWmJuJ2XZJp1lKjC7o+d7kkk0nhX9Xgg/yxgS8+dAcyi8ALhfMgGRZCBGSfLXEWJSNL+SrhGjnNdXHjutnWq+ZMrzJyDHeW0y2EsyQU6UaxMZvAonyiPCBO/SAbGla9gFNinMa6OXSDZB7PN/fg6SRGoi1P03IxkDkF2kc/obhRnESzrO7rC1qy+7hkabIKOcXZOjKC7HnOtQVoC5dvmU/L4sHZGI7ZwfVQRyb5ce87MMLN2RTYIYRXFbRppd2+W4XMu1S3Jxd845XwuA4ueUwwVyX6Kt3Jyz27KIQNQJaqjGl3//kso4N87HCnAEwHleAbpGehT32tr1HbmO2n2Y6z3mfoyqzdr9GiIddT1m1+/F8VnU5f4ZTFM9PKcFdB4yFh+WVP0/dKOt7zu8rv79OHIPu88fju1B2+Lg+2Gp5L34/kipZfS6Y4u6DvvwhjZ+X7495TsNLAHSzLNVpDb7cyFNcJUaosqLRub4fE/WYM9adUo2P2vOucJU6iVN86FmHou+iEU/QC9ypBz7OMdO0s0CkZXjac4jl/K5uv36HurrXA+37ZzKtPkEZA/jJgWQrRRzXfVYRH47RtaaF2099WzOu0Zv/ZQBCPWXOp/iPAfF30Ok8oJIi7xSqVgCAJAGPtfB18cgCo17oZvP/RMQy/pKW0utf4rz9QzQQp3XLAmkIBcU/AuR1+kEKtmVPuX7EqdKqGUTKlnx/eUtT8C9sOfVU53vj57nMS0BYcpjrX5RU6jqCILAKoNE/s99T8t2ZstC1ZcEAl4c25rkfG8QZZ6Sk+XYYowx5fvr4wfCrUEm38cpIvzBeOtnpwaEXLLSYG5D0KZ+MfbcRsjnU/Wf1wRvAsIjMuPPk8w/lvSM83iKlSGBgKGr6i2FN88CKWYSF5/l6GTeBGYAJoA0yeUY+BreqHF8F8d/ZtBaiGIGNdfJz0PKcZZjdS7LggGUgJj7zY8zP0cxX8dzE2bAuLg+AxhElHjTci5V44wUz0d+YNyXfCy3IdxyHYg8n2UpDJLaY5etnHpBRAFUgAWg+RMhyyLm+xiUchlrFltR5LQwjYzcFq+bDLQquYnI70SUNVs2qeV9x/UckWH9DkGWYxBl4ygiinVSpPwuDYAYVI41Bf32cX1CzP2KcnH/It4yz5uYVN6Qi7JJp027KLGsHDfJ72Sy0AnA5c22z9/5OglACLrOs1zEHNeXgOIMEAHZz+dEXvOJ5zQAcBkMseylKO8nWc/pOxbh53aTr09we1kkLsdKp+W93F/hMJ9Pc+zwYrNelyPxjSLIss5EzACm6I3nuOkCfHLfyzwl6vdi3fE7X2a51cQ/RbHH/ROljgIGQXM7j0MUQCbcQ5BVOlg/t8Bi3g9BRg3oHoCkw+trmda/N5Ucy/hFNZ7D+8XB/UfLYcPH19nbWzOrKup18Zb9qRwfHpdN3da7HD9W/+vK2475kXbeqo3vSkmY98O/g+U7DSxPzYD/4P2fYaUm7IPFnac4gk45NNLj5bgpOSg75XDnWqz1iJgkbl0LKwOaHON2YoYS3+RyTMTeW2gZS9wCxyuwu+adayGRcNHsS/zQEAxOzYCbqaM4BT1iigpT1At3z1Y5WBnw9bAm33wV4KMkdzoIuKCwMlNxX+29gZIRrXLL1CjeLK6XgtxojQx4NXRotUejPO6nBivjYGTA3tmSR/G87UueyDFoNMoXt0MhEgZvFm6VnXaYoipER0pGDF6j1b7ENgAoLoEuKGzsiDHoco9RAaPX6IxD70y5rjMOe2ewMRPGoHE/Wpy2I3aTRas9lIzYjg18kDhpxxIbEZMoLsOc+D5EImyy2meXYRqvANA7DaPovpVxUDLi1zenECLhyXt7TEFhdBpGk2XSar9wu9yPFglAa2gnoGQsuRadV9A5xmF0ZB232sMHVRLPNzpg9PSf4y44Z+IwGXgvsVmNxf1VK4r/kDJimjS6xtFYszWex84kUY1x8EHBeYXWukJW1e8baOPRNa5Y9QFSxLB1nl0diztttkCXdCLZGk/kVnStMQHOkfXf2oBp1LCNLzkEZSahSomUHXy91tSmMb64OXKfWFHC9TWtwzgYqOwlIGWC97K4bDqXrfEqFuXR0FsoHWAtJYWPUSBkLwX2FJCSrO2cu1DlOWdiLQAwJiAlcqvU1Wd2sQxeQemAcWeBJGDP90W5Q54TYUG05Sfqq25dyZdY3GGrtBp+1LCrCePOQrd+YXV3g4aQKf8B0Unoxs99d0RGpW3I3hDIc5D7tdWQF47G7yWBWCZJGRTkBbG8RC+gTITPngNhIi+D2mUzZmWB0BFxUsUqL5uA6LKiRCQiykL2IgiCLNtZeQRW+GVrfqlvzNduQrb4J7KGA8UyLkwsO48CtlQm/dppJBsBkwoARwLS2hfrdrG2DArx3BFIdxLxzEP0CvEk7/gFMrCcwV60cU5SnwTJMOcWLCQlTiCuw6yoEYmsCtm6KCZR9udIQGxSttiLcn9tRaJNP8iSmucAmuphK2bUqVhpY7akhU2E2pHyI+nlzpwJaVJDFq1oE2QGyHIU2TKXEDuykKlRIDSJwGq2GkaTCmBjEqTQJKhxeUw6GmxoMzlTJCIg6QhYBEuf2XodbTXmDBxTS5avlK2tUVK9asoyNICcgNAAagJiJvGBBPRWwq2/+aZOTTQuAIVcikF5tCjkR3JCsWgy+FETfU4mf5b0B0lyBq8bBvy15bEiDALoPrPN85DovOpJblyHTCjyBeh8NAnKidlSzR5Der5PpLyespWPrac5LH+hbGCCJIh5vKrHvP7ZmqwANQhE2ipBBsyKr0qJUbjO8ppn2bKck6S6GIhXDlgk24pMqDiW5GP8/wHIyvXyc3CMuEqGeUyLUoE8BvkFqPL52urI34+Bw0Olgpjnm8uCgKl22KqtoHUdVR/reTtWjtbF/x8DufX3Us9bPF9v+Qi+E6h8awXAb7skPLSC/O6U7zSwBIAz3eOL8RwuKpybPVxSuJ5W2LoGH3Z32AWLtZpwPa1wafd4MW4AAFfNDltvC1HEr/bnuGx2+HpcwyqPMzPg49UN+mDwi+0lLps9hqCxTxZWegzR4GmzRYTAr/enWGl6457bPZ7vT/H+6g4+Kty5Fq1yaFXeqCWBlZyw9Q1upw4fr28QksDd1OUg+gQjAzrl8GV/QnkyIXDW9JiCxr1roUQsG/8TO0DLiLWa8Ov+FD5KXDR73E0t/tHVr/ByXONuavFH51/h63GN0WtcdVu0ysMnief7E3TawUWFi2aP26kDAGgRSaZND6s8pkBL5d416LTDRbNHTBJD0Hh/dY+bscOZHTAETWBYJOycxXnb42bo0BmHtZngo8TeWZw2A7ZTg8tuj/upwdpM2DqLjZlwNzVoVMAfnF/jxf4EH2zusXMWo9d4dnKHVjl8uTvFWTNgDBpKRoQo0alQiAe0iIgQuBk6hCTQag9mDX52QvVpGXE7tPBB4t989gW0iPibmytYFQhw53ycu8mWNpSM+OiMyCe2U5OBPIHimAQu2h69pzyol+fXGILBdmpg1IDRazxd73A7tjjvemwnCyUoblRpAogfn92iVQ4v9icEVIOicZ/dwQWF84seX/VrKJHQaI/Ra7gooUSClhFKRtz0LTbtiI2Z8GogBceJCvjj977Ezdjhul/hpB3RZMAcE5EbAICRES4rOJSMaFTAGFQBuevTqSgSWMlyPza42uyKPL938Qov92usNiSTkAQmT/O0PptwPzalHikS9qNFYx1CFGg6UmxYRSQIu8ng6ckWt32Lq6sdekeEDj5InHQDJk85Qs/XPZRImILClNmQPz6/xdZZ7EcLKMDogFZ7dNrhul+RATUorNsJSpL8e6cL0F9b6v9utBAiobMO+/zZ6lDabk/2GCaDT6+uoWXEL2/Oi4KlMb7cLwBoFXDajtAy4nrflfeYUbS+Rj/HU55dDfjqfoNPvvcCL7frogwCgE+evsLoNRlKvcbaTrjed0XJcHIxAgBu9x1WpxMpLZwudX/00S0+uzmn5+bElbWkZMTVaodf3Z7Rc2M8tn2DJ+db7EaL9cWE/WTmOOQo0Gw8lEwYnMbmyYhhMmitw260aI1HiKKEKADA6mJCqz1u+zaTXZDiI2VlFDN2j06jPXPojMfd0JCCKChsrmhs26HB5mrEbrRFAWIUkYH4ICFlwkffv8Xt2NIcANi0I4yM+Op+XQg6prz2rzY7vLjboDEeZ92AF3cbXH20w/VuVeJCz9Y9YhKw2XS2Gy1cVnApGTF5jVVD8p48KdJWzYTbfQeb30E+Sniv0DUTXFA464aiGAKAfjJwXhXvmVUzFcWTywqsGCU27UhKhKCKsu1+aCBlxDAZ2ByO4YKiPmw7PPvDO7gosR9tIf+QMsKoiMmTQm3dTugng3VDz0WrPQZPisFt36BrJpy2I272HYwOsCpAy4j70cJmxcx+MlAiYZgMVi3N1zAZKBWxbui38nZH97PcrPbQKmI32MLErvNYeK0ZQ3M2Dgab9QAfJZzTCIGUWqtc926wtAb7BierEfuR5OGjxFk34KvbzbtuNUpZtRN2eyLSMdaXedIyYrdvSthJ2zj47PXESraum+h9OGk01hPRSyAlXdM6Mgoym7tA+d0BSHnGxwFSvq03A7Y9zblzCrZ1mJxGDOydlZVnkwZEgm48xlFDmkAKtSBhrIcQwDRqhPzcAIA2AcGRMi1GUmRaS+FFHLMrJSmxjKW1zeE4TTsh5rFrHeBzSI5tPKYhbztZ2QVS3rHir8RWh1lRJDMrtgB5LUVHCiwApFzLihgOtSkohI2tXhaFF7JcFsookchDhcMdOAyES0IJ31mCrXwvK43q8JL8uYS98LUCcxjIIVjjfrGCrI4HZrBUM6TzsSjmMJU6dpkLX8vnX1fqPh1+rwFlVlrUccIP2qyO1R41BL6zfBam64M2j/3/ffnWl+80sFzLEf949bf4qX4PMUmcqR0iJK6bDW5Dh2fmBvexJSDXtjhTezy3ZzAi4FJvcRtWcElhJSdc2jNcmW1hXT3Te5ypHmMk4HmhCbQSO2ZAADGyxiRxZnps1IiYBM50j3PT4wN7B5cU7nyLRnoYORN6SCTc+RZ9tPh++zVcUti3lkBlZny90Duc6CeFpIMsqQo730BnJjUpErGgykCMspp+wE/1gDvf4tP2a1yaNe58iydmV1gJzzWxOIbMRrnWI3xSONEDrqc1AMDIUMB6TZBy41boFDGQAkAfLE51jzvboVMTxjgvqTFqNNLj2q7RKYe1HjFFXdj+7lyLq2ZbLMXX0wrntsf1tEKnHK4sAWC+Zowal3aHjRrRKlcYSxsZCoFRozx0DmiJSWKjR0SIwrrpo8K57dEHAy0iXtkOPir8cP0VgJk86NLucOM6NDLg1rULsphLu4dEwo0jJtUhGLTKYYq6sFpKEfHUbjFGg+tphUZ5jEFjrSec2hYnesR90xRCIgbD57bHWo3QIhZr+xQULmyPMWqcGwL6WhBz6i7YshHXglj21maNlZ5obg2xV7bK41l7ixM9YqUnrLQrazIkUVgLbZblFDSs8rAykMU9aGKNNTSHVvpCOLW1xBwIAGe2x5NmRxZ5xfMgMAQDLQIxBJoOKz1hygRQ+4ZAvo9kyZciwcqACGJCPDUDOu1w1W5xN3UlZy0rR5h5UGYLOx97v7vHvWuwtbQJtCpgpYnplgFIiBKN9rCZlXfvbZnrjaFn+s62hSXx1nRQMha5hCjRaYeh0Xi2uoNEwniiCwlUpx3ubVNItJSMuGz2kHmuuLDc2duAGScB4P3uHiorehhYvt/dYwi6yPbUDjAqFHKuEzNCigirAk4sjaP3phBjvd/ew50oTFGh066sXykSnjZbjLnutZlwoz3O2x73psGJHbGzdkFC1Wqas94bnNkBvTXotMOtbtFpV4ibxqyc2tiR1o8KaBRtrn0iZUarHcagYWVA702+NsBk8DJ6jfN2BnenzYDOuAV7pM8gVmdZ66wgUTJibSZoQQCjM6TsG7OC47LdwUWJVnucmBFuI3HVbcsaTgDOGmJaZU+SzrgC9rh/K5OBZV7fnXYQuW+85lxmmXRB4bQZFmRcvTGYgir7so0d4aIq9/I74zT3JUSJKSqc2BEqK/UG48vzx14jAHDe9JiigpERTX5+pEhQImIMGoPROLUjrArYWH53OBhpC0HU2k44MWNRBBgZYGXIiiiaT6tIwWd0wElDbe81kdGtzVTWDl/noisAVYqEVvtCVKazZ4APEp2lOetVwKYhkOaMhw8SjfFY5TmVIpX5XVsCyFYFuCixsSP2ncE3LSs7K4k760r4CCuTTFaWtCVNmSgsn+vcZy0jOps9R6KADwqblp7TSalCOmaygi0mAa9nRMAKhUZ7xJa8VidFdWoV4LOyAyCPmElFpNx38oLJXidpJhtj7w32UNEywutQiPJC5rHg/jA7rdf0fNaEW43xCIp8tLQkL5KgY/YCQgHHfL3OColQEW2xJwlA3BX18aAyWWEG2oWcS4niBUKC4uP0jCY9myE5ZKkoyVSCVAmxJsHikjJArI4v+C4y8FmkPcvfhQQiD5hvUbO3wFF/0vJ+nWVVvtdkXJHDhfJ3tcytXbwe8rUl/dghlqstkof/4+N9LOSfx0DfwbHHiELTMbR4DGyyvH5XyneYOPVN5TvPCvtf/st/ggCBMUe+hyQz0yq9hGMSCJg/AyhMsIdMsfRfPmCAZfbSmiHUJVl+SJgNFkBhSyXwt2y3Zm09rJMZUGvGyHDwEMUDP4xDplVmny3XH/S5ZmCtGVcPWWDrvtaMp7XF5Bjz6iE7at3vmjV2Zi2VR2XMm53DfnJha+Rc/8MxHGMNrftX3wvgjQymdT8fsIxW/S9jzN+PMeoe62ddjrHH1p/fRK+96PORearn8Zi86vuO9e8xLxhes4+x5B5bG28qxxhDD8d3OJbH+v26Ng7rPfb9GGvsu9T/WL2PFbZMPXYuVbJmNl4uzMr7JnkcU04/NoZDlte6L4fHDvt57J7HZFD2KmL+Xn+u63pbWdbXH/blMP8ut33IcPu6dg5Zat/4s8p1HWwk55jwx+X9aJX4Bkr9Y2viETk8aKzq21H5vOvW4sjG+nAzeVS+Rzb4Rz9z+U1YPt72/neVybvUW8vmbTbdjz3sh8cf6QvFYD/MvfvY9W88d3jsyFgWP9cPrsfDa96m7cP38SP1vqmet/4Ve5uxv6EcYtB3bu8drjvaxmvW1m+0T+947Vu1fVD+6r/+FrDCNu+nHz/7T3/j9f7ZL/6b3/rYgO+4xfL5/gT/3f/9H5JSJYoFlbqQqcT3FC1PZqMUAjMjpwDF8JhYzpd4Ga7Pxpm4o9bk8HkTZ4bDIMglgV0mCiviwYMpsuaISQSY0QsoRBslibvI547tMWN1PbMzZveEwkrIcRSK3A8KmYVMiyTlnJC9xBDl+hfj5sTaFZFAYYVjzRa/Aet7mAkRmF05akY51qYVGntklsmcnD2zdnJMCzPuFda8Sh4sXgCFRbBmiqsp2Vlmak+xU9HS/C9o0WuWNWCOSeGYNI53yLKo+8+skIfjLQydwDImxBPhA/PisLxEjokRkeKFAMyxL4Xhj/rJ8VN1Im9m0ksqxxuFeWmleow81+LgO489VYc4nqRiLuSk3VE93CNyP8p4ZXU+V7qIS8lykIHqk9V4Es+xmD+XOePXgMvX8zxybE0EEr/5RHU8t1f6kmXEyeQ5Vgh1W9WY6hixMt56DrKMRMhrWM/XPZA7iAAj2AQ15RipSo7SicI6yEIsDJtgYhReGwdjA6hOkwpj5OKxDfR8JX6XZFbHsm6rdpKoyDhk9UzyPIeDec7yIoZILGKS+BzLgO8vaz6hMDuWeclruo5xSpUcZbUGeGwAkMz8/CCPW3qKzRORZVDdz/2ryVqAsv6RqldfTbKCaiyPjLX0o5Jp/V4rYzps+qCdeq0dYjReu9LlY1V4Zjkv5/6kSs41ZkmaZCBDNX889mp9pXq+/PIYG+rL/VUfRcrvjjqOjOsSc3v8XihzzfJgmeT+Rz2/9/h9JT2QU91+o1LkzHNXAZyo5zUvfPUbUs0Ry/+QvVRW5+o6C4Y8+B2CAKRLSEosZMhrrPwupuoYP4/1fiICIlE9i/uOxOyJrE0j8p+U50LMx7PmTYa0WOcUSyroeB1viPmaIqdyYzWONP8/fObK71Ql51L4eC27A9k+kHPVZl0eu77I/rAfR9p5Uzn6m3nQj0cti/X/RxtYnhcpzXP52jaW5w/H9M4A712vxyNz9F0tCb8n7/m2lsZ4fPL0FZ52W2xdU+LJNmbCWdPjl3cXsCpgCgprO+Hldo2LVQ8hEr7erdBmd6H7ocGz0zs8vz/BSTti8Br70WIYDLQO+OTJDV7cb2B1gFYUW6Vkwt2uRYwCH17e4bZvIWXEdt/i/fN7vLjdQAjgbN0vYoCYFKRrJqysw5fXpxAAbOMQAsXepCgQR4XVeV+S2Q6jgTFhkSMPAHa9RYoSYVJYnw6FtMQ2DvvrFWTn0bYO/V0Lu56gdcDQ20KycXrWY3Ka4nJ6i9Vqoti+TK4y9BbRixzDALSrCW7S8BOl0FA6wvUGuvUIjtOHUEoSpQP8pNCtJ4yDyS4mFJcyDRpN5zBsG5jOwTuFtpsw7C1WmxHjYODuLZrzAeN9A9UGKB0w3TWAFzAXA/xgIG2g9nREyrEXxDZJbz1OI+KdglT0SxR2BrL1SEGiyTLZ/+IUSID6YA8/asRBQbSUlsR05M5H71eBcE/WcdFRXIm0gQhJZELca1JEREDeaySbkLpADJScxmEVkHoF0WVCkpyiJHkBudUQTiCcZmpBmSDbgHhrkXSE3CnEMyIeEU5S+gSTZsbUCMgThzgqiF5RugWRgFHBXCuEJtH9Y8V2KUFpIoAZ2OcUHwVIMFgeVE49ISAngaiB1AWIzL4Zuwh9p+BPA8SUiTmaWJQGai8R1hGyl4hNJODShhLfwoQkTFgSVwFyqxA3AepOIawiyUXn9Ak2x9lk9s5kUlHImBuF0CbELszXOAE1CfiTSMBeUioCZu2MbSxKGLmn3X1YUz/FKOh8os/J5vQII43Ffk2su+4ylBx1ak+kKQR6BOAF9I42eO40jz9vdpMEka94+tU0dxLuIkC8VHAnidJA5HVtryWiQSFiUSPVxwoMvSPFlj+NZRx1CgD9QsBdJCRLfZTTDKrMPZ2DBNQA+E2EvpMIqwS1J9KVGiyqge4LbYLeEYGJHEHX97RGIGcgrXsB6QT8KhFwkwz+Eq2pTAITbKJrR7qWwbjZEgGNX0eYrYTvUlGucH28wW9fCnhDfREBUHsimvErklkSBDIRAf0LqbcAACAASURBVHsn4Nd0XO+ozeaVgN+kouixd6JsaEUEfJtJazIrcrQJekcyjgZAAvQAuI7GBkF9S4pkkwygtyibZZFIftHOCj+9nwFjzIA4aTrO4C0qIoPxq7mOQrSSAaVfJ9g7mo/QZuCTQZx0RHQTNWBGGpcaqC41JkRDCoOpofN6n+A2JG/hAekTfCegJgIooaF+hhZQjurxHbWleloH8ZQYfhmQqZGAS1pRvdHMYwIymMzAOFpA7kkOJE86p3PdvhPQfcJ0Qv99KwoIs/cJ/dVb7vSPFD3Q2JMAlEcBZtID04koCgo9JASTQZ+m9aP3iZR7VkAN+bOi943eExiLZgYpsgKnwc5gjuoUsHcJbj3LUPd0P4NzYAbSQtD50GRiIU11qzHX3yRELSACPWvSZWDOzyjXr5DTahAYDk2CGmgNM6kRjTODTJ8QLD3Xuqd1wkogVprokcc+MxtTOyh9ETFlhZRAsDQuqiPN788MYFlODD6CIVAbFf3n9vn5IOWVgPQkAxlmhRwrMpSjc3zsgeIoASIkzCy7qcg/5nWwAOApLUA5l1ppyWC8bu+QvKdW6KasNF4oNFjxI1DGtrT2srJgWTcrEo71cQb4PJdpCeqPYaYD8LtQeB4ZV+XktiCReh2AroFyym4ub0Uk9PvyGy3faWBpZcCz1R2aHAvGcS8rPaFRHu+tt9AilBg4fRLRaUeulptUGFAb7bHSE642OzTKY2MF9sZhaInsotMOF6sejfLkcmbJldWqgASg1Q5iRW6UrSFykKvTHQCgMw5tjvsBUJLct5r6fHG6L6ywIVGsBQCMXuGkHUvy+sb4QqZSu1JyPIcPdH0CYLXPxBREZtFaBymJRELJSIQZGSht2hFTJlBojC+EFt4QiymTaXDy9dZ4eOPhOxqPlhGj9YVsga/jc76RWFlXmEoBwOqA0Xg0OVeoUQG+kWh0gJQkQy0jehWxaqeSd7TEaniJdTfBN764+LHbILN/sltW01B8U2jmhOxTlaO0MR5SRuyvRgLOrUMwAa5Zspamqo0p10MMoTl5uyW1vDchJ1kX8E0ggJ2TriuV4BpiRfV2Zhhlts4QBHzO2Wk6V9RyWkdMWSkR2gDTeCARmYJWpOZPMTs9JwFjPYIJCI2CsTmOsglwykA0AcZ6xFYihexyJxNUtp5GJmWo3TBFKu/x2AYoTXEoIceiaBMRGpK9agK80VBtIPIF0Pg4xiO0kvKStqQMQAIB/jzW1OaNWM4hqnRCMBGqCRRXk/N8QiTERkJl61mYaPxC5VywCXAyASZC2Zz0vhOIXpKipA05RgekkIikHhe5T7RJys9sQwqA2EpiIE2gz5rYUWNDxydB4FGuHSkMAHpXNKHE3aQo4BpibEUbZpZW/iFXqRBWTDZBdB7TEwBtJLKKROkwRqlQcm+qhOAk0IYiR2cyEmkCgiFrPFQqRHSD0IgbD6gEb+SC1CG0EnFDConYSsBGOEWyjI0kpUMxpwE+ywkqYWoESvJ3ExFWBKZod0iXTW3Oo2gTjZvHITIbcCZPgkpwjVzkhoRMxJqa6Fi0BI5551QnoEcCBqEQm4RoSSHgu2xVXkWEheeBQDQSYRMQnIDvJMI6ADIrM/JYx7xB5Fx+0cygFonqCg2t3aSpD94Rc6f0KVuRaRyhY5A3p4VAApJJ2ZpHMvHd7LFRLFwSCJ3gW5AU4DcZQGQALqvxyUkgdNmKxMC70vyL7BWQdO6XAfwaGdzMTKOhAcIKVF+bsvWSlFqxyrfJDLBRU/+BDJYD4E5IVqFLBVgSsygpokLzCCuswGz9NEBo5n6xx4AseRRRxuw7YiKlOqn9vwsrrF/R2AHAZ4UUew3FJnvYKLouEQF0Wfs+Pw/EWisKI2ySgNvwfGEGRVU3Sbkg5mMS8K0kltU0g2uWR23xZi8X5WalC2/oawsybejzhpzZbfP/JDKwl/O6Ye8PmX+uGECrUVT1ixn8TzMrbLEwC5obBpMLyz6DqYDZK6dSHFEfxNLjRCB3Zl7jZKUV83/M7dCNDMpEkcGhRUwEAllHwU0BSxnQVMCSQSt9noHlUWsh97ko7gjULaylB+APmEHhgz5V9Rbl1SIF18EY05G6joG4wzZeU1+5RRwcr8H5Yd2HbT4KJI+3+50ov8OA9zsNLKfMunpzT8yfq0zi8MX2FKPTOGlHjF6j0R5f+Q3WdsJXOyKn2TQT7oYWIQpYHfD87gSddYU5sTG+EAj8zVdX6JoJ16FDzGyZCRQIr2TEZ6/OoXNwfGs8PtueY91OiAl4tesKOx9AYFDJiPu+hfcS5xsiorgbmszqSfFSp+2Iu6Ep6ResDuijwU1mjOPS5JQXnXV4tesKWLrdWVxu9tiNFre7DptuRD8ZSn9QBdt/dbMpAErrgF1PRCdSJmKqy+CNW7zddtAZAALE9Gd0QD9amEyiwPFQu2xl/fpmA6UDtCZygP2+gW0cbvYNmtZhu28gBLDfN9A6oN83kCqgbR3utx3absI4GvRBou0mrNoR99sOSkcCN6T8IxY9RakzGKz1+wYJdDwGsjTaxsE5Yl28u+uQosDFJZF03NyuCexan1n4gN22XQTKNy2Rc3hHv9ZuIotvTALWejouEtYnA0KQdF5FYs8rqTYCMeiJuV4hElabEVZ7bHeUOidFiWGvYRqPFAXWpwP6nhgujfUIXhWyAikSICPGvYHUCbadGfiUjrh4dov9aDDsLaXtsDGDZYGYQX8NcoVKxUob8w+NaYkVUNsApehePynohtgR3aixOu8x9BbaZPKeKIsVu1lPcKOGbh2CV0RsMLHfrIDUlFJD53QgflJk0R4MzHpCcIoAa2DrOO1AVBPIqzXMLvHtxQDnFMJI8yhUhGo9Wb4HQ1biKCB1pBQaAIKTZa7NmlK1hNw/1XnqKzLYzG1xWo3u6R5SJuxedWBGQNkGSpmRwZVQCfp8gFIJ4z6btAAIm7LVWhYAqc8HuJ1Fc9Vj2ltiJMzPln0yEOsigOgl9GaC29uyM1AnZCYKew2ZLevF1R9A97177G5bsnx3AUqTQgMCaJ9O2N+1QCKQHAcNdeoQBkXW8EnNP/JRQHSUCiVOCvLUU6oRE+n6zhc5c+iAPKW14gdNQJ7PJ1AKEWZwdBI4cZA6IvSaZOolxEUmfxkUpSHh/KwiAm2izxmgmh/sMfYGmBRZly8CkZTcN0C2ZCO7DcsnPcJdg9QGyKceYWshPtkj7Q0492k68zOjJIA0KSR2Exagz5cTKR8ceXUkE5F6hZjXNrLlOp4GJCcRn/r8rJOiJowKcHImCrkKc05JXgMREO8FlDy5QUI2HiGnoUmTQsyKj+QF0AXEe4P0wUCkLaMiJUZWDklNoSPJSaSW1rlqaecudKDUO0kg7DRiVm65vYE0+X2rIvyoi2IkZo+WNEmENmRZSbKCtx4CgN8aCJsVY54UNVLTmhU2lGNpyM9PFBR2AgCThH9Gaw0c2mEi9RlA6BU9lzsDsfJIEyl/kpcw6wnupsU3LaIljxMA1J8MgISKSHtNx4KAaOLMBso5RTvylCkeLLxWIym7UgKlxOH3hU5ziI9js9j87MmNQ+yzlsBLqsNlmfA+QaU5J6+NwCRnBtEk6DNA9/scopEVUQgihwcRcJ7DgsSsLPKClE0JEDlHb2rCHCqU2xdOIDWxSpMDsnwmUkIhijlEJBHIq0E090t4QXVxKIGc60EGoDXJDoAC9heu+ZKVUXn+ONUIg+4DYJY4hQ/XWYOjHAZEgDAtXMvJmjgrR2prJh07RLCYwW+qCIMYmVWhRIt8sXWoQqLrZ8BN1yxCBQ5LGQtm12H+zv06BNI8pjdUWb4cjvPY8Qc3Vt9/l7DY7zCw/E6T93z/T07Sf/Evf4xGOozR4D5QKo5GOiikwvDK7KT7aLGSxFw6ZgbE+jwfi0lUDLARaz1i5xticsyEP1Ik9BmodMoV0h6XZM6NafP1MZMJ0VOiRJUvEgl3nmj0OXUDQAQ+PqrMcBpzfyRMTjMCoJB2jFEhJmJVtJJzNioYGXDvCKxa6bF1DdpsoZ2iKlbIlZ7gM9vkFBWsDIv662sBFIY4ZgklhkIFq3y5jol5+Bjn6KyZ5Jh1lNlCuW4+PgWNIWis9ETXyAgtAvae2AnXmZ2U5Ta3KxbkO432C0KgmERJDwCgjPdmpNQPp02msI+qEB9xrkweO6cTYYZNJq4QWfYsK2abrFMocNtcHxfutwuqMFZyncw2yalN2oqZj+tgKzanVglRljY5bQnn77TMiJrvIVbIVJgv6/d5yHPGx3xWrISckuQwB6vVHoPThRGx1J/HzPk8l6yFsayNQq0fJKRAcT23OmB0GlrFQlbDdVDKCJXrSkX+/WSKhR5A7nNmgq2s0Jz3lO/nwukitIqIiQBy/Zkt4PyZ0yKsGkqdwn3U2X2dGQhdoFyarBRiOQvMpDvcfmN8Ud7U5zhHKq8/Vhjx/LmsgLI6FNZSVhClJDCOuiiNfJBlDJwawdpQlA5KRTinKW2AV+SeX22G5nyolFtUyvm+On0B/3dOkfJAz+MReU75es6nWqdE4GOsuFGKvBconyj9TnPIAdfhRk2KA56rkNs2Yc55mccSnKK8n4Gs+SqnW5A1I2XeoHO/2b2eLclCkHIDADFLMugzcU5bJuhcykA+8aafS7bwl42or81PKPKK1X3MISBMLPk+EcXML+AluewPGaDLygTCG2SJBRdBif/n2HmBzCcgK8CBYh1a1MnnalDDx9hiYtIMPniTHEEgpY7Xr0FCrOqv8nvyOOqcjCVGn62fOd2BcOTS/k2LcFRvcW9n+SVQexkccR5RGl91r6DxknUdBaCVHJ2zo8hik38YFwtBbvjcl+IOWax2+foqTvUYeBKZ/4BCHqr7SqztDJo4drvIIqIAGgAl/rq+jvtbx8svxsefMePV2jDL7ZT2spW3xMYnLAFTDUAOgNACEHGpOsqgtHBMVPcvXEzLvcuvD8ZRPVvHrn8Alur+Vsdq74JS/7taLIEHLqaPlQeWwCPjfGCBPFYWk/iwnjfec6TNQ1m8Vamu/Yt//i0g77Hvpx+/95/8xuv9s1/9t7/1sQHfcYvlEA1+NZ7j2q3hoqK0GVGhDwZ7b7ExY0kfsXUNTm2PrWsoDYUZMQRdQNHd1C5AjJW+pDP4q7v3yIW22sz7KEvuytupQ6N8ocC/n4iWHwB2zhaXWwYPWhK1uwsKl+0OPqmSQ46BlxYB1+OKWG6TgJFkYeRUClwaRZvTVnl85TbUh5wK4aLZ42bsMAaNMzvgeX8CFyi9AG/mf707hcmbLqMCemfIwieI4dOqUFIhAMDgNUzO2Vana7geVjAqYPRzrjymix/y2EyWW0gCjQolH+KYXWg5l9mQAdDaTvj8/pxo+XMKgZWhVBGf358XQMNAR2SAxDIEUPIVGhWL3Pg+ISjPWkiU21KIhM9vz6Az+JpKvxTqsrKUb4zz34UooGQq7XBfuN/3o4VRES5IAkhDU4ASF95Yt9lSvh3JchxTzkNnHVyQ6IzHzb4jwKkieqcXAEvJhLue0mN01uFuoHq0jDjrBvTO4LZvoTKNfw1iACzWKQPCmm3U6rAAbC5I3E4trKZchtuhQWcd7oemxANz6gqmz98ODaz2GCdDloswk2ypnJtMy4iQgD7Xfbdv0WaPAnZ5JqBjiyxormRxhV63E3yQuN2RvFS+xuqA3UD+WDHKbOGmdseK8EvnNcPXah0w5s9KRQSni6XfTwYnqwEpCdzt2jkXnA7Y7tuSf06IRO7XJmI/NAWwkKUYGCddrMZdM2HXN1h3lBtyCKZc3zWu5D5kWQyjKUDZWnLb3/UWxgSEDGb5/rPNgPt9gxgktAnQOhRQd7IesOub4ubdZw+DcdTkxu3nNABIgqzfImEaNWzj4b0k9+1pdiePUcDnjbM25Io+jhpK0bhDIP86IRO8JwDpJg1tAoxxmCayOruJcuYB5ClgsocAx0DL7HbuHKXrOD/fYZhMAaNtRzHkfW/LWosZGK82I/p9A6UDbOfR9xTvPfS2uIXb1i+eV+9VlhvtcmOU5ZqQwb3uHKZJQxna/cZEFlplKOVDsxkr5lzqT4yzddqspiLvFGdg26zntB0pCpjNWLwggleQmj0SJHQ3YhoMVpd7xCjhnYKQcfZ2UBExUE5F3TrEoKA3Pq/hWFI/+ElBtR7GeoyjgZQRSqWSR7EA/EDyjl5Cr12Wh4SQgOL8oYMpwDwFQQBeRvhJl5h5odLs0ZBEyV0YvYRaZ6+BbPEWKpW16J2C0hF+VFCngeL/NbmCaxsw7b95uhF5QtZ4Ek4kMJ6o/2FQZAGMmRCQczFm8C3OArkqOwVhwmytjwLiLCugfH4HiTmEoFZAJN5ZRwBXI8XTC5C1vCHOgUW+QjEDeCYpLJbQKGgMAqRQqN1DdVoqEAK5OxdlQgbEtcUS2UIbMs8A+cXmeoJAMJEsr6j6xlZTtuwWJIoZENSALmawW9xFZ0BcgP6h+awmLeJz/D+3Sda+OFtNa+RSjaMcPgSEB0qO2W19Bt6l3RrcHQKuGvxWCoW6iDSDRAaV7P68AF0HwJpj+h/1UQXeHjQCS3m+rqQH2H7Z5sG5B1UeHqy/fxPw+lsvCb+3WH5Ly4d/fJ7+8//hPyrWxTr1hk+qWGQ4DUfZNOcVGbJlrU7bcSx9wbF0EsdSKtQWO+rH4ykPjh2v+19/f9v2lyk+Xt+Hx1JdAHiUPl6IVNK4ADiatuSx8b1t2o/DdB+PlcfSWBwb07H26nMMpIHZcgTM76fX3f+6sR5N1VHJ4PCex+o8lnLksLwpLQbnRaMUFPP54pFz0AaXd03ncFje5v63TQtS1gbXjTf/hhxLi1Eff0yeh9e9LuUJv0IZEB6mqajrYoCZ0vG23zWNxrFxcp+EOLwGC7fxuu+vS5HxNuk2XtfOu/zEHMvBdtiHui+P9etw3/G6zczyxnzXwe7qWAqOo2N7i7Ee6/ujKVuObajetvAO8x3m7cH9j16Mxx/A152r+/W2bR3W+Vj97/C8/L2Uw53xMQDymvLaNB7HNtGv68frjj86b9W8PPY+f5f1/ppzS4vhY/dX7+7H1tlr6nh0NbxhDI9axN7UhyPlraxqb7luv6mF7ptc90Z5f4P23qn/b1nnNyk/+affBovle+nHT/8eLJZf/Ivf+tiA77jF0iWF5+MpAAJfO2+LxU6KhCEn5PY5z6RPClpQfskpuywyIGQrZHFjzFYWKYjkp3afZODDeQrZJZK/c53A7DrKhe9l6+To9cIKyiUkUSyDKYnSN+4DJ8p2+R5OSs33KpHQO7IssSul5mTUOWlzTRrEboj1GOtrARR3OrJUigJWfCaR4WMMVqRAiWHle9jVkD/X9xx+ZzfCmLXqrH13QZUE99xfAOC8fQnzBo1jSWtwWOcH1cVSpUt8KrlsVi58B29ETnatVFy4EAJkeWFiIJ81z0zeIwQWLn50vyjtxEiEPEz0U1uzOCY0BFGsQFznIcip29c6lrq9V8XCUCwgFSji6xYJp6vvbBlhLTqvIakiQrZ4ChmLFTCGnKi6AlNFXkeSWFOHlsBbyogYFKQK9F/Gcn1dR0yi7AY48TWzEPP8pFx/iijxo7xuCsCq3O44tkkobm9OnM1xsfQZFCvq8tw1oYyDz5U6IyjGCijWF+5DaT93IeZYxTgpIgqq3CM5lRIJKbs6ZjKp2k2SLD9VbBbvGycF0TBrR7aa8I4qp18q8b8yIYUc9xgoBrXefbFFpnbvLG6Q2dqymOdsoSmpmOrzfD0fZ9dJTnEiMLto1umJuI7abTOB4snYxQ+Y00blmK5SJ0CulWyRCfmzE3NS8ep+we+ZKvXKos9VvRz/tDguaC2wC2Sqrie24nmuapfKOnasxHvl+oTPcWcJyxREgvqdTIQcyRJap0Aq5CeCZMpxdlw/p1BKAiX9E7dV0o2UMeY1wSlDFBbpRgRbm4BC8MNWmToNT0nbxPXmcdbpOqRbkhqJIOb6mOlTp0LaMxPFEAnRNy0Ld04eP+Z2uZ1jqXakRyGTEVW6lCToHH9GHtYyNVZuK833yama87dyhWW34FmORe4q0Y8235ctdzK77HLaoRInmOq5q/pYj5Pbl3SupH7J8qrdbjkesU49UuoIy/YOXWEX5XVKjVouh8D0QCnwAAg9Zp2r3mGHqYEOrY6lv49hSW7zSF8Ocf83doU9JjMsj9Xt1XP9mHX1sLwdoH5zPY9e/0g5ZIX91jLCJoCIK343y3caWGoR8XH3Cl9PG7gkcdIOkCLh3rUYo8Z77RZj0NAyYOcbrPUWO0+/KJ1y6IPJoI6YY60JBXCe2x6N9IgQeDWusGnHB4nlGVBufYNWuXJs6xus9ISYRIkhrK1qWkYMwcBHiScbYo8dvIHO8X5SJFgZsHVNcUG10sMnVdwzGeBxu63y2PqGYhqzK+zHmxvsvcUUFVbrCXsG3ioUgH3vmuLGyy66wGyNNM1QQC/F5UmoKtYzREkusEHDyFDAKQNoIRIGb7IrbICLqrTlAsWRjkGDE7wbFQqItzLXW8XwrcwEqwJuhg5GheKmySCzdoOVSNg5S/vI3KeYBDrjMGY3VgbmH5zcQ4qEr/sVlCCmYI4RdAeW6FbTTmnK52vQamSEixICFN9J8ajklsfKAhdlcQWuixAJTWbi3U62gP0QJS7X+8Im3Dty42I31UOAza6xrFDga08aIrMacrxmrUTwVRwe1TnHK/J3KYhxeMpKCh7L5BXaHC84OI3O+BLPyTLzGWSyC7DVFDvJygIu7JLKCgjnFYwOcF6hta4oAHjM/JnjFX0geSUA62bC5BXFGObnSitS1vTT7ApX4odVLHGVMYkSG8nrgNcmMLuKA5hd462DEAn3fVsUISorZ3h+ASLaYllx4fpqRYXVFF+5vpjQT2ZhzVo15OabEsXKGhUwTKYAeI4rnXJMJEBKDwajm3bEdmhKLGStWGuMx360RUnhnIY1Hs6rEmfJhd1l2S3UGorvlTLC5+t5fjkWU+us4Mp9IyZnjtOclTUhkGKE3Cx1ifk0mRTKVWRYRcmRrcbs9nuyItKyWsmjZMQwmgO5JDTWox8MlEqwhj63rcM40lpJkRiXUyJlEI+Z22LFkTGhKImESMUtuLZopyihNF1nbVhYCznlFO/qdK4PmBU93Bc6JgvR2jjqzJytIPP4UiR3ZTdpdKuxUjKl0vc6nlWbgOBlId/iOU9JwE0aSgcYE8qYpEwPXZs9udoGr6BMZhrPc6gyY/k0EtGQyM+blEw2pAoZmhQUR4s85qLk8YIIfpIo7qYip7gCgBQUhAqIk4Y0ATErClMSEMYj9t/cFRZZ2QOQckhIUoQygZVQMZOCkettSqIoMZKl2N4UqvNZsVPYq1k5g6WiiRVSdUyvbsjNl8/LTHpUK8HI1Tg/XyaQG252NUbMTNhcP7uiCixJYpIgIKdZkQQCzzLRXDAxVVZosRswNZqKK2xR1nBhBQormWoX3oX7aKVYqkl++BiXQ9BUgGkVO1nAWCb84es5nvUYgEyY83GXY/NFM9inD4vv3OeEpdKA630MrNbnDwHjIchLR/p+BLiJx8Bx3dbh/3jkhmqcLIo3gsl3wXjvAGIfb+jhxY8N/bdSvq2g9zdQvtPAch8s/o/r78NlghkAZaPJVjsXZdm0qcqSOLsF0gaVN9a1JY43mD7IYk2r76Vr5k1zyj+ONTEJA63aBbJ26dPypGwOefPLbfAGGZifqWPuqXyeY/t4s/4reZYVI3P/6vQWvFGrXbti9RJZbtboWG095O+1xa5272NrHLcx1zuTa/B/vl6pVJhe2UJXb8hoQ4cHbLCH7oh1Hw5VfXPcypym41qtSebMulv9CpQcllnOh26ptUW6fqfL6rr6GMVPHagED+azllntuljPWS3Pw+90bf6cACGBl2IDtthBLOV2uNCOuXECWFjN+L5yraBN0U2VMqNYU3kjwhauxY/iwQaiXuKVVeW2/hXL15Z2DpjzUgK2qqP+lb6grINC3FKtjUPXxtkiObdB8nw4bwCw5Y18TcbCm6rq0i33tf7B5o/1D3++dy+7ZUwOgJ1q5zp4U1FtxPZ1GzxnVX93qputdwcMiou6BPVhqK2P9eVlU0h1DYekMIebH66fx8oby8MXHai/o0gPxjZWm7XxyDzUZbxpl2MXc18PN2BjXmdOAEPun1PNPE8JcKI5agmov095I8ipT0aBHNuU1xkrW/LG1h15pkQla8eMlwcydbX1MooH19V7YC9oPLsbe7zfVf9Drt8fijXPUxQZGxxYZpGAQkfF74s04wiR7w35i4iihOEB9Ozxo8V9D1VfF3vrVC2JNGOf+lhA9fjx8s1zeLjk36UkAcj6eaweb5kbEfOppYtrnvPFeVQ/B5gfx6Ob9cNnSszGfPpuls/Tg2IW4Yr179rxwQKHz8k3uWYBQN7GUFO/hx/I4CFIe1RWb6j7eP14PYh5zRgfAMfHyt9h/b1LeWM/gDf25Xgd1fvo71D327X1m23j9+Xvv3yngaUWAZ9urvHz7SUA4P3unghp9qfYDQ0+Ob3Fzluc2R5f7k7x8ckNPrs/hwDw6ek1nvcn6J3BZbfH57dneLre49XQodUel+0eT9stdt7i/3nxDE/Xd9g6i8lrdMYhJoGrbgctA356fVUIXZ50e3x2c44PT+8Qk8CX9yc4bcdidUtJoFEet2OL3Wjxw8uXmILGy36Nzjh02kGLgAvb469vn2IKZOE7awfsncF+tJAyFjfTp+sdhEi4anf46e0T+KBw3vW4HVr8o6ef47PdBV7u1/jh+Uv88v4CvdO4Wu0Lq+ovXl1g3UwIUWLTjLjZU+qWRv//7L1Jry1Llib0WefNbk53+/devHgZLyIjSSqLpIoCMpGYMEEqIZWUIyZMQYyQgBH8AIaMEEhIMGICDEBCoiSkQiABlSmCysqsiKyM5kXE6257mt15Zw2DZcvcmn78rQAAIABJREFU3M8+9937IiojbipMOjp7u5tbs8zd91q21vo+h84qXCwaVHrAEGN/LvcLLMoeC0NgRs1gcH+xx6tmgVXR4zAY4vcMAtuuwLrs8XyzwqrusDADDoNBG6lgbg417q/2uDrUKI3Frilxsmhxs69hjMUHpzf49PoMj0+2uG5qNL3B+6c3OK8O+P6LR1hED5ZWHjaiYlbawihHaLoQeLpZw3uBqiAeTR8E7q/22EQAnavtAnbQ+MNvfgIA+IeffRNGOzxc73Dd1NDK4WZfT8JXH52Qd/PlbpnQQEtDCLgXiwabtoSSAe+vb9BYgxf7JepiwK4tcb5ocHmocVa3uDrUhIQbvQdGOTxeb3FRHvCj6/tpU+PQFnh8tsW+L/D++gafXF1AK4dlMWDfG/SWvLpFRA59tVmiKCwerPZ4erMGAKzrDr9370v8ZHsPzzbrtIYA0DmFfUcUJqWx6AadwpvrKOMhhqGe1S12XYFFMWBpeuyHAlf7GufLhmh9dsu0bquKAKy6QScv5dmiwcvdEqd1i21bQkmPfVMm8Ji6pDEtSvL43+xr3F/v8WKzwuOzDa4PNXqrYK3CourRDRreC5wsW/JE9SaBIn107xIvD0vc7Gj9jLFYlj1Oig6f35wmr1JRWOJOVQ6bQ5XQTM+WDd3zW1rnZdUnYJ667NH2Bt5LLKoO+6bE7zx6jkJZ/KNPP4CKXgCmxkHcoNHa48kpce/+9PIiGe/sfW8jRU8IwHtnG/zs+QW+/fgFfnZ5nsKhAeDbD1/iuqVnte0NLpYHfPbqLHmX7p/uIAA8uzzByZoAWw5tAa3p2fxb732G/+/L99F3ButVg1XZY98bKBnwrbNX+PNnT2CtxHrR4XqzwP3zLS43S5yvD9gcquQBDF6grntoRSBFFyd77LsC66rD5XaJ02WDbtA0zpY8qufrA5ZFj6c36/RcDtHzV5cD2t6Qx7ApcLpucFK1+PL6BEp59L3G/VOiBnp5s8L90x2udouERluX9F5i79nf+eBn+PHNfVzvayjl8SByFf/4+X0sawLNaXsCofnmxRV+9PQBVssW751s8MNnD/CdRy/wyauLBF7z+HxL62oIOOfVYYlDZ1BoB608Dp3Bg/Ue1kts2xJaOVwsGnxxfYJFvLeb3qDvFc7WDQ5dgQ/Pr1IkBwBcNxUObZlC6B+e7NBaTffHoJMX/P3TG3gQyvC+L/DN00t8cn2Bylhc72ssSvKgN73B45Mtfvb8Ar//jc+wG0pcNgtU2uIwGFTaYlV0uGprbJsKT043eLlb4htn1wCAe+UeV90CrdP49OoMD9Z7fHzyEt+/eoSTosO6aLE2HX6+O8fadOi9wov9CrUZcLlfpPfl8+0KdTHg8XILKTx++OoBTuoWSgTse4OzukWtB3x6fYbzRYNtV2BZDHi1W8TNRIlV3UGIgJtdjfcubtAMBofeYBg01osWj1dbSAR8vj3Fk/UGP3l1Dx+eX+HFfoVV2aG1Gr9z/hx//PmHX1vveHiyw5dXJwhBYLVoU7TGwgz47PIM60WbZH7TVrBOpnfSB2f0m3DTVLhYHtA7hSa+s947I51h01YJjG1V9Cnq5CaCso1AYwa/+/ApfnpzASkC9l2B9042uGrrCIDnUpTJpqGNqIfrHV7sljir2/TbcVpTVNKr/QJdp6EjANKyokiJZdljcBJtb3C+bNBZnZC6S21x01Q4XxBtGlOePTrdohkoImxZ9Ni0JfZNiQcne7y4WQGg6AET02RO6hbOywTsxs+li5tyRWEjjzW949umSB57bUb6tG7QsFamVBLeFO87AzVDteZUFmspvN/2hApth1gnQ932keqsbU22eTlunvJmsY5RCJwOIkSAMQ5dSxEnInrlnaOoJnZGTHKtxRjFExxFG/AmOEAAZayLOEdRJi5FGigo7VKqSkKvjpEgrif+6JSekVnDnLbBqSLs9fZRPvnGa0qHsZIQjmeb0LzRTAcweu25v3hNcpTkxnzcNEwbogAYuOvYhvCkvG5D4Net/DX2WL7T4D0f/I3T8O/8d/8GPigu0QaDn7b3UUqLR2aDR+YGf7L7LaxUh+thgcflDX58eICPFy8wBIUf7x/gzDQ4Mwf89HAPv7f+HD9r7+FBscXOlviyPcWzZo216fAHFz/BX+wfY61b1GrA9VCjlBafHc5gg8LfOPkCz7oTSBHwol3hd0++xPc3T6ClwzcXl3jWnaSQVgnK/bxXHnBmDvje5Teghce9ao+DLdA6jcErXDcVPj5/hUoNULHdlelwappEbeIh8KJdoXMaN22F3z5/AQB40axwr9rjBy8f4bRu8XCxxY+v7uHJeouV6fB0f5IMwO+cvcBVt0ChLF40KzxabKGFx8EaVMriy8MJGRXxIX642GLbV9gPFKq5MMQNen9xwG4oUMRwQS09VkWHTVfhw/UVXjQr7IcCtRmwNi1eNis8XGzx+e4UDxZ7bPsSjxZbfLE7xXurG2z6Cp/fnOKb51f4fHOCk6rDuujw2c0p2t7go3uXuGprLGJYK4euNoOB8yKN4eFqByU9Nl1F1CPRIFyWPaHBLjcolMOffPJNAMB333+GXV/iuqmwLHtYp3BvscfgVaI2YWPtbNnAeYll0WPfF9DS4yoqdD4QJ6Y2DifLFp0lZNebfY2zFSnnJwv6Ma20Re8UBidxfbOEHyROzg8UwhsNyBebFfGMbiucn+9gnULbGZTlkDYBXER1vH+yx6E32O5qnJ9SqPWuKdG8XECuBlyc7bFvC0KQBKE0loVFAND3hPxZaItuMClPk8ObGSHUDgo2UjEsli0OhxLeSizXLXZXCyzPGvRdRE01LoVStvsCi3WHZl+gqIhCYll3ab0OEXXVRoTJxbLFflthtW6xvV6gWPTQ2sMoh0NbJPTTw6FMFBEcKnl4tYBaWiyWhNbatYQOGlqF8qyNhp5D1xnYnsIPi3qg3FAZ0O8LCBlQrSh8sG8MippoYIZWQ5cWSgX0rUZRWXRPF0AA6vd3sNHz3R8KLE7aZMQ6q+A2BsILyPMu/UD6yJ9pSptCBu11gfrhAe3TJdRFR4pD5JCzryrifpOgsLODgjnvxtzoXQE4geK8Rb8rABmgS5eoMOQXFdyTDrqwGPYF0MlE/6BvFNwTUuB9o2FOOgzXFXHm7QxEbRPfqJDEOwgnIBcWfmuA0kM0ClgPxOunyKUkyxi6uNeQnYQ/sZQDKQNgIrpmo4HSAUPk5NtryEbCn1rAEjcgbgwpWmsLsTEISzvmc6b2aC7lFwbDiYdfEAKn2ktIKzDcHyAismeIfILFK4X+0QDRKOitxPDAoniu0V+45ErS15o8io48L3YZEApy7wkHhDJAbyO/aBUAD+iDwHDqIVtSPn0REEyA2in40sPcRC0vhrK5OsBXY66j2cqU0+eLiF5pAswmKoQK8BowG4HhhPLnXB0gW/KMeROgdxL9PYf6c4WgAbsIkAMS0bzqBVwV4MoAs5FwiwC9Jc+Q6gBXAEECw2mA3gmYPdCfALIH1EB5i8OCPnsFuIry6exiHKddUI6ePpDsuosAdYj5fQWgWhrLsA5QjYA31K5dkoIZFKAb8jbbVUB5KeAMEAzlB+qG2qa+AN0A/SlQ3AC2AlQPeAMU1wHN47u93F9VzJbmDkHjlA7kNR2A/gxQTVyPA/UXBH2HoLF4Qzme+gAEna3fluq4SEeLEMfMa18ieSNFoHaqVwH9mnJLfQHoHdXjdRUBEDbmlApA7wFXk6y9oTVVHSB8gF2QzIWl62SUl+riGA2g9wFBiTFf1ga4UkC3lIPrKjIEzTbQnME5rQK+BMwuYFhGI8oh5VaqliITXCEgB6I2CVqknEs5hEiLQjJzJdWjnN0xtzQokpdkl7fn54bqeB1zWcPoOeW8WG/Ge0TY8bwIUU59gDOjJ37iMY3jvJUrGgBpA7wRyVsrQoBXo9c1zymlcZGMAdziwAQwIsBG6hh6NqLcVDYGkV0TqVK8FpAu3O01jnORbrTd8rnMc0lfl7M5t/vmgUmT0CrE73chowlBcsls06PezbmBeqTO//0//Me/coCbU/Mw/OHFH/3S2/1fn/+Xv/K5Ae+4Yfnx7y3Df/I//j4kPNpQYAhj7o+Cx8FTPmXrDRaqw85VKOUAHyQOvoARDkY4tN6gkkM6dnCk3DJf5Vq1OLgCRjpIhNTPEGiXuZSUS+dA+W4L1ePG1lDwKKVN9RLwDSRMzLvc2xIeAkvVYQgK1ivahfYKterT9QxAxHybAIVV9l4n3sul7ggBF4Qiu3cFJAh8qHEm8Vy6INDHNpaqj/ydHtarlOfpg4SRDo0zKYzYBYFSuoSUO4Ybq4Rky9fS+KjNWg0YMjRZIzwaZ1J+JeVF+sTxyfVbZ1CpAQdbTACZrFeUw3okgMcHEfNBJbR0CZwnz49lsCYGegKA676mndOipbxIr1Meqo59cxi1jevJObZaumTsMxeoh0j5sNwX8XeqlM/GYEs8bh9E8lwwvU2eyyqirDkn9i4kV/aOz+vuh4I8m3F8eft5DjCAZETzuPk857/mFCR5rqfJaGTmdZiehutzm3muJ5+f5zWabMMCQAoVz9c0YET3lSJMaG54XgPzbaoxn3lEjxbpegCJzkNLT+k/GbgVjwnZcc6DZG8rn2OgKV6nIdJJTHgsmS4nk01vFepiwKErEo8ln+ui8cnrb52acHO6uGtutEuh+gyKJQVw6AyqYkiRDzamDPDcCk0GOwNosex5rvluM4fsc950ylGVPslWZPeYZbqYmIMrstDBtGGdvS9zgCvm3QTI42GzHNL8OeDx9b1KdBgAIjjWmAeZ17VWRmqWkTuTcgrH+bJhnzwBajzHofrzcHofKA81AV5xLmH0CDirJrGCQhDtBn8/Fmovopci6WHRs6AisJcQkbOU0w2i58MODLJFyip7KHgsQoJArwTl5c1Dw4X2lMPnYm5eENNcukCGn1DjxkNwcjzmBTj+lTk3Uzg2f06gTBhBoEIGDsV9Mb8nC8WLMRaW29Ce6uXgSV4QPcbXLdxunOsk1JrnIfh/dp0IY1i8QEZrEU8zF2T2DsrDaEPuweHzVlDuX7xOZO3TwfjHIefH8gh5vHmMbMAkDYHHn/gds3o54E4CqcnisJMRJkajKPXBdTgmej5HNg6CmADIJMOCDe1Zk3PNYGJvvMYgSUabz44fa+SOcqvt3PrMvx+57pixNDn2uv7z48fkd8f3o7KYt3Hs+7z7N23nFyh39vEmJbv2B//ZrwEqrHkQ/vDsn4Fh+fK/+pXPDXjHQ2H3vsSf7T+Ag0yG0pxiJKcWYSOIiw0SEhEZ9gjdyJw+JC+5EcZGX27UcTlGFTIvc9TYuZJ/jMIkGQAzNFkuxygC5mPhz6+jCpn3Nx8Pf8+NCFb6c0AZ9xpZ3kVFcqzPse5ynOutFsdyF+3HXeW6rW/J7a6xzD8DU8U2P3YXLYI/MvgQBK5QHzUaj9X9KrqMKU1DMckvfh1FzKQd3P1jnX9HPLZFeWt8d83jLlqO183leDvTjc7XbH5iz7/zX9HXXeM+Rn/B/xlg6C7KDx5D25ujm7Z5H+zBbbrjYCP52No76hzrHyDwm7tK38WfBhGAUIwn8q362Tje5H7NS4fXK4B0YKZt8fc36IPl7+ztc3ZQRxU1l8nE2dF7fqxQTnb2E5o/ALNbyjN3XxCYwgTz/MT4HQBbC6+jbBHi9vvDumysse30bs3mcVS3tuPJcOTBDgIIOXDTMPvdYUMviGQwBr5eYDQK+fo+uz5HuOV6bJzM+Rsn5Y77gJF0eYzzhNFbCaRvWQZ5VIjCyWgYCtymD+E1Rrp2wm/Ixg0brDNr6ZZRx00mz5qYfKdrsxdLmMpVjHsX1PYxLxbnYefjz8ZEc7j9ef5KFZM54/YzwmPL5jhpI//sZ/b6sc8zY4rbOvZTOTHcgNE7d6wcm9dXlTd8X712bMDrjdp5vWPHj3w/Zswe7StZ5cfH8Np2XtP/25Q3/Fmhbuf9/IKP+y+9BCCE36DC/lqWxhn8+fV7KV9w01EOQqUHrEyHL/cniei+0habtsKqpLyvm6ZCaSyM9CnPkkMrBy/R9AZNRyTaT043uDzUMBE9sY9em23MW7hYHrDvDaQgpMfTuk0k9ssy5oHNjMfKWBTa4tVmCSEAE70NjHDorMJi2SYPTdsbaD3mEnBcPyuxw6CwWHQQIJL1orDYbWrowqIsLQ67EmU9JPQ+F/P6Vss25fh1nUZZ2rQbLmVA33Nd2gkvqwHDoFJsv1QeQ6ehCyZYH6kkeIe8XnToOhNDDD0hCnaGSNfbAkU5YOg1ympA1xpUdQ9rFbp9gXLZo2sMkaprj+5gEAaJYt2n/hhAyHtJNAsBiQ7C1JRnZO0Yzms7Teh5XqAoLZTy2D8nQ7U4bylcsVOpji5senMGL2Gb6DGOdA2c6yAE4DoVd/JBYX0yUEifE5Daw7casrLwnaLQwBw90EmERtFO9oLaFjJAGk/hhipQiOHSkuIW6RGYBoKpGdTSEvVFqyAWpC2GXkJuNEIZgNWA0KspQEsRd+CtoH4Uof0lgmze8e5lIs5m6HoUHuglfa8cxF7R+ONaQIUUqig6SXU6RaGcAXQ9eyhY2eR3bukhWkXXHBSFLcrYZi/JcyAAdBIixPFENAsZQw1Rxn6sJKqEXsAvx/BGWBo7AuKY4uZAT4paKClUUliRzidqBxEgBolgPPQNhUnaCzuSbveSrg8AIyuqRgJewK1cAnVhT0PQIc1d7RXciYW+0bArN/Eo6BuFoGPImAqQnRzbkwGykeQgWVIIZhA0N1ZOzbXEcOoRTIBsJWQvEv2Daih0E4Jk4GsPtZfwFYVX+jKMP9SxjvCAKwJUI+FNgOwFXB3pLSSFaDFthGwlhaXVAchC2CDidSaGrZkA1UoK5atDpI+I4ZAAXBWgWgFXjPQWREEREGIoYHEtYWsKPxVeQHakRA9rD9lTO0TXIGD2Av2Jh+oFyeDEw2wl7CIkhEd9EMi9J74I8NHhKDyF4emWxuA1HZedgFsEiIHkFRTNWfZA0BQqC2S6tQkUjsf3QUvPRsjvD0nrxN6boCi80VVxHEWkBQkUJqo6AbsKKK4ovNAXSOFxIgDC0li8iWGoJaKsxhBCCApnVY2A6gBbU8icsPTflfQZEnBx/BxKKUIM8XQUbghQuKocSI5B0WfhY9hqDKnlNtK9Zse29GGUZZAxJLeLMixJvramkFhfxDkqCgftT/G1i26pXWAcM4dH85yCpvXgIKqgaJ14zCnENN77QcVzgq6li8YQTwDpfuLiFVBsA2zFNC1jiGvySoLGx23KLoaFDtP7EKDjKZQScd0jRQgEfVZ9QJCjAStCoPDKIcR7l87pJqQQXulobN4IqJZCZ3lcFN4ooIYQQ4YpVDMIjP0EjOGbcS5eCwgX6N51mZGsMiMn349i+pssrDNRg8Q+JnQofmyTBjvKY15EGNvN26TrA4QfZcHj4udu4h1Ng8W4B5FRuaT+PK1B+h/ifzceH/epIuVG4DUUKRw39XXHXPI9vIk8xUx2yL5n7czLLePzdUbzEQM2H8udhuyx8g5HZb6r5Z0Ohb345x6Ef/O/+Xu47muC+9c9fJBorEFjDVZFh8ErCum0BrUecBiIcmNVdOgcgZKU2mLblajNkKghajMk/srLZoFl0RPlRSDKCBcEFhHEZ9uVKdyt0ha7rsSq7AhsIQKr5JuPSlBIZG9VSnpvLYEysCFZKotNX1KYWSDqARepHZiKAaDEfAEKGdx1JXxAonI4q1u0VqMdNJbFgNYSKEtpKCQ2ANhFgAnqwyUaBg67K42deAub3kArBxXRV50XCfAl57IEkBL8m94QoE4E2bFeJtqKRF8RQWyYjkIpj2XZY9eWBCBjFbyXqIshgRGwtzb3GOWhkgAZ5MDIOcky45C9PoK/MCDI5XYJpXyqI0RIVAfcRxXpIigcERN02zH0ECjNkHg32Vhn6gyt3YRCgbkiq2KAVh77tkhtWytRlpSPWBqLpisS+ACDijDsvxAhUQ4UhU1UCVo7LKse7aDRdbRJweGFHAYIjKi6zokEgOD9GPrHoYJS+lS374n2gTciypIoGhi8gEMLpfKJesEYl/IQOXQwBEHAA0EkkBk7KJjCpo0IpmLg8ELvYhhmpDBwTiSQg7IaYK1M9BjM4amUR9eaNHcpfaI9yMeiM1oLgRE0gdsK8dlkjs2q7infc18mD6RUjrxg/F0GmJgX2nVmEtqYaDJiSGFRDugag2rRo+vMhIagrIbEecrUFUOvY4gloOLYba+hI78l83qGACyWHQ77MlFRyIx3tCgtmkNBCp0KcIOELi1sDAv1brwfEABpKAaCKSoSsISVkCqn2KBrlPZERTGo5LxjWgapQhwnhT1KQ4AdttcJUVgXMRx/kFCG+uFwPhFpC5i7s1r26HtNABQijHJpzUizED1turQYGgOpPZRxGBoDUw+wrUlowKqijRoR18xZSX0xGIUj7lFgDCkVKqOhCALMGyqNQ3ASqnATZ6cbZOILDQFEQxHlk3giIz0F3cMkW20cbE/3WrAyo5Gguq5RMMshyW5Ex0aSe7BEWeEHmdpn6o/gBW2IFY7uizgnoWLUT8y5Dh5R3oi0IHEcA8kpyadVEHFzhnlShQzwnYp0FXGMyduLkVO0l2nDDuxt04HmLGgjTRhPm2ulp02yuLklSwe//wX21AsPdNFS4I0tXjze7GJqDfb4sSeu4E0uQRt0fM4L4lYFsg0/TBGd841AxGOloz5FbIf7DlldkfWfc7/mocex3xy9OBmnKbxVIHAIcHZ8Eq4836ALSH3RxlyAGMRoPPFrJD63YkI3gtvzYIPWjvVCJqfkec3aprFPDaYkX2SGXR76y/1xyY3AzIKaGzn5cFM1GSae6cBGJxtPuXKI7Dhm55Gdn12f+gvT68W8mSOyyed4tLzGRMjn/ZWG3tuYGrN5UGdv2cYd5Uf/6X/4Kw8XPdUPwh+c/L1fert//+q//pXPDXjHPZb7rsT/9ZcfE4m4Eym0RsQfGH/Q44tChwhQMXo4oOPLwwqgckCr6HzkXJK9pN3vpcNLfnkLfvGAPBpBEHhELwlIoxcIlcerNobvRLCFaU4DEEwAZMDN4Ry5p4JJnoWjHX8IEP/SQHkiE4Ls6GHinXquLwbypmz2EsEEeBOwaSR8GaJ3Y/Q2+SqkHyw5YPRGxB+ffS8mDzmDR9CLPcQdz0iWzR4sVn7i7mfaPY8yCApoBvqRaTsR26Sd1m6g/8IBXUeAEl1H7QcF9C21Q+ATkeyZc1WCuJW07yqSl7Qj71U/iLT76wvaHb38wQoIwHDiYa1AnyfGm+zHJgg43uGNTw+PAaCwOx/tCNcKeBUQTFxzCbiBdoYdA2cg7kjH27JpCQSEx8072kPc/fadgKvpR9Szt0cDwQGOd0MLwHug7wV5KAFYC+z3gnal6wDXj2HjvFYA4FhhkIB34y4vr78bBHz0WAb+sS4C/CCI77cIsI0A6oDAxO8qJIPG9wKiDAidgDBRGYmeMhGfRQHqGwIQhurKMiC0CwgDQAYoSfc8K5qsrChJHhoACAeqrwqSl4z3YLCAquK8ZTzO943OlIW4zrqM99YA6h/jznaQdB96DYg90ZuYdYgKGj1TsojzDLTOsie5mmp8l+XPBt+7sqthFoDc1zAV0q47AKimhmBvSARbEAzuIcgbAtA8ZXa/8vVyV6Nc0b3Dnin2jKgWKBd0DwiLBKyiiwjooTF6sAXVoedgBBuRjrxW7Hni80D08rjoRcp2voOIoBFRgfOa2mPwEd5Nzz0sDCwzejbiMxEfV7OtoCskD6DsqV7ylolRrqoFhiUdVx1gl4Del3D1OEbVksGdPA96lGsac8RkCuzJjONnzw972NhjqRpMvNE+a5PXktcmvZNkXONA11J7Ab6g58BHr2/yxPQE2mK2JgGj5O914QKCFuSV6ghsRPX0HLEXMkjAVgKqC1A9fZZ2BFBhryDkOH5vqD4AOEPeFNUjAb0wMAl7i+AxAq3wfVSMY5WW2iLvl0qyDJK8XnKIMmTgn4o8jPybEiSgW41+dUyzfrOiuhEMR8aQavZY24qOBUX1+DeCPc46jtnr6DWU4z2hOvrhyL1bPOZ87bkECZiDJA9g4OdPwisxAVXh+xIA1BA9jJYAZJJMA8k0xOc675s8iCJ6lUV8VskTRm2LcV00yVX1Ysy3jJ58Bpjxhq1JJO8at+uzd1z+uyP8GCXAv4m555GeFTHxos1lNc+fpL5FBlIT6DlIbYe0Dvm7iRueG5R8D0w9logeS5HayusfNcgyY3LiLWRZsIcyenupYaTv7I0MWR6GCHEN2bN6l0MpN+hm/2/ln/IaYXps0lZejhi+xwqPNR/j/Pud5Q3a/9FXt/Kb8guWd9qwfFBv8e/+7f8DAAH0HHwBBZ9Adg6ecmQ6r6Hg0XmDUg5wkGicgRGeAGaCwlJ1aFyRvg9eUR3psFQ9GmegpZsQozfOQImAWg3oHIlyCBK1GtA40kD58xzQplQWEgFbS7loDCLTR/AWHwQqZeM1AX1EJeXCgDGcG2q9RKVsAmXR0mPTk1evUgN2Q4lKDdDSo3VjLlalRpCY3msUMpJuRw9l76dhvIW0sGHkDdXSo4te2RwUh+fUOo2F7lM7DATTO4VKWTTWoNQ2XdtZjVoP6L1C53QC+CkVeU7Zy7wwfQI8SYAlQd7KN9XSQ4JyaKUIkKA2OP/TRICfq5bAe5iCIweRKbVNYD02SHRWUwRnBG9h0BIhwgS8hL2pDB7DQDFGegxewmSk9AAZeoNTcF4kCHvBMo65UzYCq+RAN4kXNd43DCrDHmH+3FmFKtKShHi9D0ge8ByUhz87L6DkyOk6WJW8sslLHD3V3hNFSW8VltrBeREBaeL8sjoMKAMAWo1w68wVyzm53N4qesV53ZQkEBcGZeHcMilGGPPBKRRYzt67AAAgAElEQVTSJ488g8x4L7HQI9ANy42AuMb7l4FllBjzjE0cc36OgW36iLLLCLuIfZk4xgBE7zCNlb2yeeGQbgCwlqhQhujBz3NB+16PgDbS07qYMWZqGOh+1JrDtGkdGLBmH8PlVaTRyfliOydTW86T93iIIDlDDJHPnzsbPadKBXTML+sFZFxniJA88rRW7On1UyJ3AJZ0vQSEY10EkJKksYk4Xp53F++TBH8f5cnjOwySPGoygtlEbyuD3FDluKbRAxqCgLcCyngcBgnJQC8iTHMlAfJG5IAjAiOPacYlOoHKF4jE8jFMmHMTU90w1uf24vkcrGfClyow9RyxNzLEcbgIltOrcWy5EubHTZpE/MheFvbkALQZa8VYn+eUe7BYpkGkvtOmLHu7MI41FVags02LWx6bTM7ME8oemLSBG0AbnQxsw2OLhrqwAr6YxRe+RSHPXUgGz2TseX+WrY1szDG3M0jeEEWqM57LjI25cZR/FxzSGsb2HUZAID6cbRrxeQ7TnALv3L4ueTt5jE5kXjmRDD2Ww9iPmBp40Xhmj2Ty2vHzynVwe7lFJt/kmRPjdwDjfZidm8jviFEzyYENIspFpk5uGXy5cyAvdxgzkzV8032Mu4yxu85n9fLwVWD6aN3pWXwDW23extHyy2qH2/oK4/AXGsP/8pbt/rMq73C06FeVd9qwbLzBn24+AEBAPL3T8BDQghFEyYDKUSVZMU1opNHoMNIlSgnmBmOlv9YDGmsiL1RIiJV8XsX8w6R8xrxONgoYsXL2niUjL4I/5KiZACmgOvaXI4DOgW0sh1l6ARPREXOUSg5bZR7BHDGSjYJcqWcjgNEjXaasAVPFFyAFjhEl5+AvyZBSjjiUAnmj+DjLho31/D9zOJLRItPcnSdKDc5JzUsKucuOM2LkBMTIj0AjfL6NoZG7GHLqM6N1bIOuHxVbVvZCqs+5qSEAPirVrLATF+YYNjtejzRuH5XubTQOWSHntpgfi0MgE8JkAP04Bgon9Z5CQiWP3VNYqdR+DOME0i8N31dzxXz+m5iHaob4Q8vzAZBCGEVUGlkJRuwrcWRFxUPkCqcIKTeW89qFIrTLvfYTPq28Ls0PI4BMbNdz6F1EqEQgOQQv0OQhbPE4jz/dT3GdUwgjK/Rh1IA4/BEypIiJvjCp3aRcI+uLUTI5rCzeWxNZgIygXlNOMYcMJjnGudGNSMr7wBQggq5FAHo9ynoSn9VLNAW77cWYSxs/uxi+CA/0KhBYSfRUT+bDhgiAITci2KDJkTF5/ByqpzLjhs/72fWxrmNDSI79udi+y5X3PIwtAKKX8DqMmCCeFEqbyT5NZRCwJhoMlj6LQcDpsX1W/nNDwssRLTPly7JsYp/JEx0VX8/1ZNYmlxgJkhRgNvRim6yEy1l4HeelCh8jOXI0z2gMqT4aBjPjjY0P8uDTeNkTnRtAnLsqbMzJjRE27DFLtyjfS1kYIEeViJje7HUuN4zt6NHoEj62Fd8PgkGBVDQc2ZDgtc28XSLKPQ+DhATEAPjibuCqryqJ5gEYvUnx8eR822Tkiem9IJLXWowgNuw1Y1s3Nwjy2zS7J7ie7CmCRAReu5hjnHke2WgC6J4Jcoyo4D4AkmlueHCd3BuYxoyx/Twvc4IKm60Le7xziowku3wM+Rxncphs3ojZ9a8xpvj4LQ9hmNYlr9j4/5ahePuVkV17u0/ug9dm3uex8lXzSG3MDa/8+/xHOz/G9e4Yxy0Z5de/Zpx3Got3jfF15dj87ihv5MX8TfkrL++0YXlP7/DvPfkH+IvuPQDA++YSAPBpfw/PhlN8u3qGratwpg74dLjAd8pn+En3EA4C3ype4Kk9xcEXuFB7/GX7GL9VvsCXwxlWqsWF2uN9c4U2GPyfm9/GN6pLHFyJIShUcsAQFD4sXgIA/uzwDdw3OwxB4aHZ4IfNI3ynfgYfBH7e3cO52UPFp2QICgvZ46Vd4WpY4F9afYIhaLy0axjhsFItKtHjnt7hTw/fxMFTTuh9s8PBF7ix9UjTAI+HxQZGONxTO/x58wGGoPDIbPByWOFfXv0En/b38GV/it+pv8An3UPsXIn3yyssZI8hKPzj3Qd4UOzQeY37Zocv+9NEXdI4g8fFhuhWovf38+4M5+aAlWoxeI2dK/Gw2ODlsMapPuDGLlDJAS5IbGyFc3PAJ4f7uCj2WKkOB19gb0vcK3Z42p3gG9UVnnanqFWPy36Ji2KPl/0KtRrw24un+LPtB/h48QLPhzUaZ/Ct+iXu6y3+ePtbONEtGlegjOtRSouV6qCEj3QuHj86PETvNU5Nk6hZ3itv8GpYopQWX7anOFiDP3r0PQDA/3b1u1jrFg+LLV4OKxjh8KxbjzQWCPiwvoQSHk+7U2jpcDPUWKoeQ5B4XG7wql9BS4ePqxfYugpfdGeoVY+NrfBeeYPP2nM8KLZ41p1AS4e9LVGrAVJ4fHvxHKeqwfcPdE9br3DZL/Cd1XNcDQt8VL3CD/ZPUEqLM3PAxlbY2gpL1UNLh1Ja/OxwgbXu8EF1hR8f7sMHifvlDv/q6sf4QfsePjncw5lpcKJbAOR530Q0ilr12NsSnSfal6Xu4ncFIzzul7s43h6nusHWVXjerXG/2GGhenzWnOPjxQv88PAQp6aBgsfOldjbkjhmyw0+a8/wpNrgZUdyuuoXyRt8v9zDBYFVjOX8vD3DN+orfNqc41uLl3jRr7G3BXqvcK/cYztUsEHiYbmDlg7boULjDGyQ+Jsnn+Npd4qn7Zpog3SHtWlx3+zwF7vHsNHDvdA9VrqHFg4vu1Xykt8vd5Ai4IvmFFp4XBQHPO9WkCLg1DSp71PT4mao8LdPf45SDvjfX34XRaR5OS8avGiJELxQFoV0+LC+xEL1+P72SXqWazXABombvkqRB99evsA/vnkf/+LZp/jh7iF2toSOGty/cPo5ruwCvdfY2QKPyi1+vLufNms+WBC5/U929/Go2sJDJL5aAPjD8x/j/7n+FnZDiQfVDqemwd6W6R78k+uPYL3EvfKAzw+n+Gh1ic8OZ3hS3+BFu4KNmrUPAmdFg1JavOqW+GBxjct+gfvlDp8dzvCg2tF6eIVdjM54VG1xZg74ZH8Pp6ZF5zUOlqI6VqZLHLrXfY0n9QZn5oCfHS6gpcduKPHR8hJDkPj8cIYPl1d42qzRe9ooPDEteq+wG0oU0uFfu/cjfNI8wPN2BS09HlcblNLin9w8wVnRAAA2Q4VCWny8eok/vXof96s9Plq8wveuvoG/df4pvr95nCIuvrV6leihAOBZt8amr7HQ9Pxd9wt8tHyFzut0X98vd/j5/gIrQ9ccrEHrDB7VW1x1C/zuyZfp/eqDwGW/xHVfp+iR9+oNGmcwBImDLWLEh8ZHq1fpHXE91Pho8Qo/2d9HrQa86FZY6Q5aOuyGEu/VG/zTzUP8K/d+isYZvIjv2MYZlNJhqTtc9os0/qftCb5RX0FLj7VqcWNrDEHhR9sHeFJv8O3Fc/zZ9n1cFHvUasBatfi0PceJpvV83q1pHO0KHy6vAABPmzUqZfGkuoGDxE9293FWHKBEwN4WODUtatXj5/sLerZtibXu8GVzkjZ0T8sGUgS8apf4cHmFxhnsbInOapyWDR5XG+qrPcHjaoO/3DzEt9cv8bRd49S02LsC3109wz989dHbqhupvL+4wU93F/BB4F61T9FGtRrw8/05zssDNn2F9xc3uBkq9F6jixvIH65ozJfdAo/qLRpn0EZMiA+W1xiCxKavE+I8R/tIBFz39NvPm9+d1fju6TN8djiDFAHbvsJ7yxtcdQu0TqNQFLmlpcO2r+Ah8Kje4mW7xFnZYDeUt2TaDOMG+tp02NsCS93DBon9UOC8PKD3Om3WF9Lipq9xXh7gA71jnJd4vNygdYbwLEyH667Gri/xcLHFs8M6bTAzNsVp0cIGiV1fEkUWBHFSR8ulihzUUgR0VmPbliiNTRFB3A7xQY8OBH6/toOGidEvKkZcGOUmuBWdVYkTmmQwrrnzAnUxJEopRr7nPlJ0UhYdw7RMRvkE4Mjjsl7S5koYkdnZROKNd86517w5HM+r2F+OMeGcRKFHai6+njeKOQLJWpnwGLgkHJ+0+S0jJsX4fY74ncAS4yb9HNU8R0PnY4wlwZufeZvHKCwnaNgchZF9f3P0nln5L77eZb/UEgLlK/01Le80eM9v/14d/vP/6WNcuwX6oFAIhz4oeEjsfYm1bNAGCo9tg8FSdtg6+oFcqRZDUGQoigGXdpUMKCk8KmFRygGdN7i0S5zqAzpv4IKEEj61AQBXdolS0MtwIXvcuDqdO7gSZg7rBaALGoNXODd7+CDRegMlKGyTDaMbV8PFrS4jiHuw9QYqwwXnsN9KDriJxgF7X091g4Mv0HqTuDjZYDDCYQgKG1slrsyFJE5LBwkFDweJUk6x+jmsmENemcfz4IvE58kvy85rGOHQ+AI6coY6yETV0nuNWvbovIEUHp3XKKVFEzlD17rFZb/EUnfJKFypDlIEbGx11NPJ3JPMq9m4InLquRSOTPOV8TwptE+qG/gg8KJfoZQuyZDmrBJVjZEOtYrgPdFDPngyaj0ECmknHKEeAo0zkf9UTpS5xpkUGs1cnivdQyJgE5VwHyRskMlwLaXDPiqXPA/i3PSJC3RvC2jhUasB+8jJWkiLk7hZsLVlUoaYJodDqm/xV2Kkp2HeT6bpYZm2TqOQDlq6CfdoHobM3v1K2VSf++ydTuNghYrvrz6GUh9sgSpye/ogYINKa81y5x94XmdSylSKXOA6hXTJyOEfaFbiWYk7FvnAspnLiX/gV6aDFh6X3SJFGnC4N/+IkgyG1G5OG8Th03ztynS46WucFqQEuih3IUICKuMxFBGgjAuHSDfWJC5Tvh4A1kWLbV+laA2eg4xt30ReVyMdccvqYRKazoXnL0Hh+qWy6R3J9V0cYx4ir6RHY82EU5Xlw/ceh6GbbG4hiKR4dlajjCBbPlOgQibHHKSNFVkAaAYz4bgFgKXpsRsKGOlRKovdUGBhBhwGk+ZaxutzPtM8EmPwEmVUcn30+BeR25UB0zgChXk+qxiuzkobc61y1EARFUUAt47zuFwcG4fM53RPzKXaDhrryLE6zOifUjRIBFbLeVr5/eqDQG8VjPLjnOJ9JgD0rNR6SSHTgsKeTQq9F0nRBpAA34CxHgAMGQo5c6kC0UEcnyMGgONImBDvGQ575+ea0wHyyJgyA6n7OsUoAsfjaJYU/SJCAp7zXkwA4ziig8PfWWlnpdr7MQrH+1Erz+uwocD9eS9QFDaF1nOfuUHBha/XmkDTOKqGjQm+PvhRYedIlDwiJin3mWLPkTE8DwCJtxWgKA+OxJFyRFCnk9GLLUbe1XkUCpeJIeKmIfR5xMKcWmoy3nkERRBjJEGGo3DLaMldnvmpufqce9pE9p/Dfo+5JF9Xjnkf8+Ov+5+PjyMTQgbOxKfmY7mrrSPtvXH46RvO941txVwub2nC/PQ/+I9+5QA3p+p++IPlv/VLb/fvb//bX/ncgHfcY3lpl/jvX/4dbG2ZFBwAONgi5hwOSem2QaKQFofITVYpiz7DjmYFjI2JUtmk7G26KuXZ5YpkGXf/D5F4ns9xniArPzmfY04074LAyvQp9DZXPo10OAw0Vs7L45DavHD+lophtT6MOXwLM2BwKn3Oz3NprR49oDFXMP/xYkRbVohs3A3kHTAO/eVdwvyHgJWb3ipoNeYCctguH+e8upw2REqfEGJpJ24kldfS49CbpGDlhUN5efzdoJNMk2IbcwQBUlBCEPhxfY92fdsSAiOybr57yHIxyqV8SgATeaVdxNgGk8+zDPPdxvH6cQev0DbR1nDbeeivUS6h0fIacD4az5sVBxVz74SgHcq6GNANenJsJH4fdyV5LbgOo8Lm32nXk9aKdzi5b2MoL5DzHxPaqUBEw1Up94/nz/XyvEEeFytDWrtEcM85fT4qEjoiYFIYc6SaiQilzo4hyfzXc65Z/LHn45bRLLM27aBSnRTyKzBRnJi6BgC6drw3GUk26RiCaHiECBN+RFbYmKoHALRx6HtNCLqDmjz7xrioyImECssKJjCGeDNVEK8p/3Izcm+IymiuUCkViOOSFUEroYyDi3mWvG5ccsVUqTGfOIWTx3rMYygV0RQ5O8ZOcj4ggzxxOLSMyMTpXokotgGAtxJS+5Q3yW0Hj5SfWFSEJMyosNye7RWFa/O4BMnbdhpC+YSGq4yHG2RSjBjRNI3biSkqbMyRTHPitbVZ6HK8ZwWjlJox1F4IwFsxhoQH0ByjQjs5znQ9sS9pPHxPMmVEVQ4Zl5rO3VQuURvla04h7JIUduNTOLsQEXgrKvmBc1aNh+tjaHrMzQtZGHRq3xI6KwBCZhUY0Wp7NSKSBkQZgsK8VZjmKqabLda3khBWPckz5XzGtmFloiTaMyBfDMXdRbTYr120T9yaEGEMUZU03xEw0I9h36wIc/g3562y0ROANkeJzeebe2ny4oGOkWfj9xTOnive0bARQVBYO8s1IIVNAxhzVOO5FDIrAxkfMSw559nk0G/LOadxrZwe63E7cBH4zYrxXca/OSzDDJU2AfaA+4offQ6kg8n1KWR2bsfkiK/cLxtbk5DfO64PuE0NMrNrQ/ZhYo/F/N7cDhJz421ecsNQYNrgXQbkvL38ByPN9Y7+jrXB//Nrbhmq8fARW/x2+29oTGN8Nc2/f10n5W/KX215pw1LGySetWssdA/rFa66BQCg1gMWuserdpl2vms94Kqt0+7wZbPAwhDHYTMYrMsO276k3WEvsWkrdJY8VfdXe9y0FYxyiSpEgLgwQxA4rVvse6KA6AaNVdnjxZ54EetiQBNpQKSg3EXvaXe2UA6f35wmg8LGnXeimFBY1l0ylLaWAELynEgfgI3NeCwrojvYRjCdV9craONgtMOrmyXKkoyym0EnQ2JZd2isITlFYBb2/EgRsGvKSd4g7ZAWIyCI8hgipUGiGXAyGR7WKlJiU59kXOwbAhvZHUoUBRkbZTngcKii0lvgZkOclttdDaUdlPLY7mp4J1AvOwyDmhg6rGTnoRVFOYDpM9h42+8rSOWAIGAKCykDnn5xDgSgOu3grMR2UEmB1XEnneW+7wjOk88z7QUbC1LTbq1vSXFiugCmHZCFS/+BbPPPC+zaBQFt1JxjScpwuy0p169XkBUZmUwpkIA6osKqSgdvJYKVUCUZIMFK7PdrhMITj6aVGagGIDjcJiqkDB6SK6gpt4+VE86TLDwd94AoPdq2JioABvnQI7gKBgEUHh3n7Pl4PVNMsMJmx7b7toKoHPq2ItTmqMwOvQI05ZgOvRyVudjE0ESuTJMpmxHROFRTHsu0o8y5dyJgYKCWRBEgbyuuUXmG8XDPCBXWn9hx9zueSwqkB2wca6gcbu2ks2wB2FYhLC3CroKv3STHKhwqUggFKW5ukJP2fMz3DKVPgDMhyyvsNzXc2lE+30D8ngz5H1pJ5xCNnMIjtCVC6RE6SVyi2Y9+iIi8wQSgJSTqMAj40idkbQgkwBP0EsGBEKkZ4CQiQDItRPCxficpL6sM6V4Kkb8RZQBagVCMyn1gcJEox7ApgRJAEeXaxYEvPCEJA5Q36AG0AljScdEJYBkg9gKiCmm+kq+PSprXcZw+Ig7rMPJjxlwyaQFXhjGXUpKCTpybIO7FXImKbXL+oewxooey7SZG9Fm6BwAVUbR5XMIh5f2pTsAuAsy2INRWE0bE0UC5eV6HyL9ISN2qjYZxQkCO3KGdIJTZivpI6LOxzyCQeDK9obxOBEyReUHcpJLvHQ2IIY4345wUFoRUzhtAcc1cEaAPOnFAQtD1knkuDct9RGdlY0S1wLDC1y6qj6iwQOLhBGhurhzRXieIyPFVojok1Fyux2ubowkD0fbJ1ijosS9uUx9IXny/8Vyn6KcYUWF7JMRcH+XG6+eL2XVxDaQdxyV7jOA7AWBeyYQMzXyZ/Thn5rHk632B0SjhPP2I0JxQYcXUcExclQFpzTkQjO7zEGXCDwgLaZQNGcl83cj3yIijEy7IECZGZBBj7nFqV9z+nHJFWT7xPTG5DjhuHCKrI2b1ciM2M7LSf85flZgYj/k4kkwzg/CuMRzr46jResyIxXEDMIhwHDjoNYbwtM0MBRfjmr9V+TUyTMNf41DYd9qwdF5i01XY9SWGyBcYgsBBkaHUx7wG5yle3zqZPEGDUxjiDj+hcdL1OQKktQpOerzcLZNHMfeSsffEBZFCQFxsh8/1Vt8ClAkxpOggCL0RAPHzxXocutF09IvBO/a9GGPZqZ2xvvcSh7ZIQC+DJN7HoSeOQu8Uug7oBVJYCkTAvimTF8bHGHxqPCqoaV502CePEZ3PeQWDF+SFAFFQ8W55fl4I8gCFQHW9E2i9pGsd/ec+vBVomwLeRWNbkKKLQMcRYj+xvwSqEscfAHTe0As1A+wJTsJHcJMUlhOv61udDCoX12GY8fYxqEvytEqZdkKDIyM2ydADro+/xDYQMEpQBDrS5WzOXD+2w8aAAHy8LngygDx729iQTj8+YpQ9g7CAeURE3LGN1/OuNv/gONaA6Ic2RDnz91TYAAvj/zDItGseegKmCf0oEwyxDb5miOeGuE4MyCEwDUcKs/YC6FoRxro5CBEbf/HeS/Nm/gmM64wcUZOP59dDjMAibGD67HwKbaI1xiBHKqAhjjW2EXINKVcSnIhhSGJch+hZSHMYaH2FlZMQpvQ6CUjjFDb2y7IGiKbIZ5/5sgR+ItL5dO+pzAjyAmKQJM4hehDmYDPsWeCoecfjEWl8ISr+SQagdkS8J1JuTjQOU3t8nPkAIy1Bal9g9LTEdREsoBAVahGSkQUWkRtlzueCHMf8OkUtlaiYp/lHWcw35lmpnXgKQvaq0pieAxJIDdjQyF9t3BX/esdbyJswUR7B3l8gGqpZHUHj5/l5E5KB4w09X66gnoQW4zOX+hFpY8OpCHoiAxDpJhLCqBjHycd8IZIsvck0V0PvbK8ChIh9ynjvgPr3mcbiI80MjYv69myExVccGdHxO78GKkwRbN+yeDOuF9NWASRr2kAhuTqM/fA94Qsk8CR+hbCcHBuIuUHF9xhw676CIOOejZYggFCN5yYevXitKzFStHC72fUTz2CsM2lfZu2F7LmZGdB5Pc/gPWLaXjJW8jFkm2e5wSEywB8eRzqmM+HMZcTty2ydkpGXV47vEsWUK9kzna/rvH28gfGVH/uqks/9TqNvpOOY/7+zP54D17vDwHqtZ/ANjLK38ix+hSF5u76YyeVNhfqb8ldd3mnDEmIM3+TEaE6ctoETrAnp1HkxQTTlsFkXcy+G6IXkXBYpAkykfLBZiOYxxNEcJVVHRE4KDxtRGvmp4TwDNgiNcSnUkK8BMKJyhvwcYhtx+gJj2Jy2yUBkJFBT2GR8quR1G8/T2MfQqpQXIcY3uMxRRwF64Uo/vreCGMOZ1DhHEQ10Kem/ACY5ERwiJxX9Z6RRkeV1yEiyLPW4DZhC0fglk8mEvapTedPHSd+pPfrzXkBFDyGTs4tEiDluJNCYidic55kMBQ5FzfM+VJ5bG1/mKvvvZ29gEYDSJ2V1Unjexo95KGqc42TOMexHJDRUQZ49Bitlz16SY7jjF9OPnjRunrnUsv6AOH9+6etsjtn00zW5HOa/Jiy/pAVkxzJ0WWDW/hyyLwChdNmxkCkIsx/WO37Ek6HI5+TYFHLvQfwcIvdoTs/ASJh5XyFb4xQomq1BOqbpWPJEZoXayNrWMznmO+Tz3XIAbsljpfNBjPJxZVZXBPJKlHQ+6HCrLUQ0yXQ+E2/IxsUGs4/3nggCnsPe0tyz5gP160UYDbMA4oVF9vk1SpArxsFQm+N9GvSsehnnKgBU9Nmxp5OL4fWK3/2R5ycL5cv3COYKlzfh6KM3V8qOKmrpecq+VzguCx6vBzyv7fx+zz8X0zFDjDLkcVgRMoM7RxPNFz+3DLKJ5m6dNPixnsjDIW95JcbF9GWYXDoRUj6/u/7/AiUf+VyMt0r+3L3u/LFxzY5NxBiIj/VOJf6r5vi6fu/o73g72Q16h7Kfn/4qb9mt8R3r+y3n/IsaTG89jje9/nXlzrri7v9vdH+LN1vX147haxiPb1K+xnP5VjL9tSkB+Doe13ekvNOGpUBIBqJkz1oQSY+iXB96elRUCl00xtjbpKPyL4FbXsXUT8xlyzkD85y9IMJU91Yjx1v+n2uwcclFSkCI0dgcHUQjelduTN51P8q5EyYIjEn6uOXtzJG8Eip6rqxnRhWDCGDmtYXIKA4CsrpjPuld+0qC25u94YIcDetbhhfiu3P+NuEmJsrWLME/Oz65JjNCpbpdXyBk8s8U5VsaIF7/Umd5I94Td+2aB2RG1O0253O6NWc1Km4Tjr0j1wdk9B3Hxn0MIEHgiCItbteZdJrPbfZ/Xo4pvHl7udE5qRNmyuP8/uExiOm88k2K+bh5nMfmcuf85gI/cp/MFeGj9TKj5di5udVyrIg7PrOMjng1bn0+tl6vk8EbGOpzEIljgBKc2yWOrc/cUMhLfszPrptfm5c8lygBehyZJ28o3VVma3HXEvL3iafjyHyOko8jGz9fM88Ry9eKvdazvZaJkp+3K+bnpxMQbFCzt/zIvKZGRP7jI6bznKyXmLTx2jKTy51Kv599P3bPvE3J5JrWMG/3yDq/sRL/BvOevNLyNX9N/ddtcEzyDifjub0Wd44/tXFkLLm85vdy/gzMy/yePFLextC7a+xvapj/UgypN2njSHkr4+kNIyzZf/Da8lX36xv08dbla8roa/f3qygBgH9XBvv25Z02LJUIWJsW+6GEj8iEAKF12ghYw5DQNiLj9THkb2EIZCWhxzkFJV0iZlfSp7DYzmqUpo9AKWNIJdcjvkgyDHVEjDMRxGKIADUjEWO3wY0AACAASURBVDsSP6SLYDT58TS3CHpzC955prQwpLUUQB/DUBWDC0XwGOcldOSSZKOY58AhvsnLGOVFnyPqHcbfTBe9t3yM0fh8dv3o3cUErprzIJmLkYFZ+Dx99wkSO8/XZFAWrUeQGD6fj39u7DGgSY6CR15lDvGlcNKqpsSfrovAOFnbfuY9ZFCXHHwmJ3VPEN+KAWjE5LyPeYsM7pF7RKUiDzEDziCQvDgnVGkPO0Sgp3g+vychAjyHbWceeikINMbHkGOB0TBmvkog0/XY6I/rjPw+TB5on3JaWfb5+qfrecNE0vecCxNA4qzMB5BzZTJ4iIxgPYjryEYzf058oOy515wnSoYJA5FAEDcmdU59srHNIdMEvBPvGQ7tZP5NHmcYxxqcTN5018tkrENk18djBIoCyn3MNdOA0RsNQCifAGr8LHRXmhF1EV6kuohrlQBSItgKz437kybmXjLYTKZ9igj0kuZmBYQe+UmTDFh+vBkT2wpxfVP9NM54TQyhzAFVklItMXoNvUCQntpgzkIGaQFG7s18PLyREj2conAUus51eK45DyiPTwcKtZWx3iBG7k5+QDIy+nTt3ICbgQIFSW1MHHS8ucFzzkuKkcz6TEaSGNcqyR3jd5edzy0KGSiE2eTyyeqxbAPGNZHZXLkPBneZb2bkwCg8R+AWOEwQIc03fRcYvZRyPM6bD4mPk3+EBAAHCu/lMbEcs42BFNKdc4gGTEO9v06RGO8nlh2yOeXyi2udjO24lgyKk84FPiduiTa1nW+SZO2J7PkKnKOY35Nx/UTAlE81PfZxDWbuVNrYyQ9M+56MJSkE2f2W3+ecg6wyntdJXxg3WLL+8kcm73KyCTSTyTHDJN3q8+uyunfxVE6un4zvDe8hka3zvLDs5t/f1ubIrzsmtGwob9TG6+rg7npf2f6k8lf09ZvyzpZ32rCUwhPEfwaV74NIiKwAkiHGBlUho8LEhh4EJAJKfRt+nWkHjHKzfsMEzj1xHMb/Jno2AcAql7yqORUHQ9ErERJYDlNmcFHFqHHnzyDD1QsREoS9wOh9BYAiUGgwobDKiCAa0rhYBlpNt7fmKKsijo/79GHkd+INFwYlokG7aDj7Uc5KTIyYdDzm47B8naI1kNGoZ5lyiDOHO7MRo+K5fN3yNQLGcOXJnAAiNRcBWo9Q9VIEhMIlbzJfO5cJhzLn3uSce4mPsZGZe5KFAEI0Hn1mWPKc+Hrug86PkPZCBGiDW/Kcjs/G/wEhjB5lmpeEyDSIOZx9WpswXbNkbMlRkREiIIhxnYFoPIow9YrmP96CDMWEqioCgsgQaee/4DJ+lmw8TmVGY55uKARJz5aUHhACXorR4cX19PS+SL/Fko3pEI1vjJ42llfqaFQuhcxoBwq+b6bX5xD7AmTA3b5v8ueTDMAxLHwmRw51jka1zOfEfRk/roXPZTYa3uNahfRPaOqPgJtE+s8G+q0ikMKhhQiTeikKQo7ygohDTFp1bqRka8Fj4zAUmXGmxeclHNP6UthyPM7Xp/XPhRk1WBHoF5GHosM4r7lcUz9hGtKeg4fk+Wjpcxwrh+Rm90QaL4XPTPtJbYZZfYxj5/GEI99ZJkkxPNJO6odf8Gx8ZsdyueYlN/6ykkKs+TOQjIIw75vPzYuY/Y/jSOHRdxoKo8E1OX7EsHmrwvINt+c8EUvmfU23Z2Z8i9wQF5gY0Hcq6GH6WeSbHhiNyjS8MGsvA/RJbczbxazOZHKzZzVMRZH6yWWcfU6yn9/eDMSDma0VxuOYHbvT28kNTO7p2zJIfc9eBa8L532jdbl95d1jO3btfDxvWo5sPNwqX9Xe157jmxcxW7+3Kr+kMfxalHDsZffXo7zThiUAtM5gqXsMXiUC4ZXuUCqLZ806GZ2VGhInnA2KKESURalsIv696etEOdJZTVxnyuHRYofLdoFSW+Kw9MTXtesLhCBwUR9wGAg4pxkMzqoG1y1xSq7LDs1gJu+43ilUmvp+vltBSo8qei4bS6A7vVVY1x0ZUgCaSPBbRPoPfjqbgVBhu0FjXXdgZNrSWLzaLFEWFqWxuNnXqMseUgUcepOoAE7qFu2giSB40KgLQtgYnIRRPtVlQ6cyFtbJRNNhlMOhNyjicebuYlqObjBYlD2hwsZ2mEesNAOarkBVEC1KZSya3qAuBvRWoWkNlnWPQ1vAGDLQd4cS3kssF13iDLOZVzPRUcQf6bIaEuIt02u0nYaOEP5VMUBKj6tXa4QArM8PsFYlpFvvBYwZEVq9FzgcCBKQKR/YyyoEgTGpyOllWwOhPJRxievLDgqmsLCDhjY2Q7QlD+/QGMAJqEU0DgV5GrvGQKgA12iYxUCbFFYSdUPcQGAQHFNaeCfRHzR0RevprILfGaDwMPUAZ2UCIYIMZJRkHkKeKxse7GH1Q+ZBix4jVRD1ALyALB38QUMubEIjFYqQbb0TCL2kOr0iGgIvCCU3gij56EkIVgICdK5TdM1OQxQujgewHSHwQgC+02DvGRsy7lAAxkNGZFxv5YgMW9s0tmBlAisSxo8GYURWFTFXMwwZdcIgEyJtOr7VpDie96NXb5Dpeh8VS9Eq8i4sXLZbFJVkpigQoHpLi7DVVDfz8oqdovzFaPiIQSLUbszv7CR5fBYW4RBf85FGAgDkRsGdOMqPHCTEIBISqWwE/JpQc4MVlPfbKITSQ3SSkHbZSBEgEB4vEEw8rwO1V4QEsBNkGPOQe0Kh9XVEZpWYerXYq6E90Mex1T55v0QTIwVqD9HIMWc1gPqL7QkvYK4lbB0IOdYDspOE3rnKUWHJ+NF7Cbum46oVsCsPvaPreezqIJFC+kIEutEgj5AHggZkG+cc0WIZFVZa8loGFQAJyE7AmwDdRG8m6+0a8Hr0Fskuzgkgj1QgeSruR2BEhS3pOm+oP4DOyR6wi4DiRiJIwBcRFTaiawpH6++PoMLKgUGQAFsTKqxqqT1pBYSluTOaa0Kd9TQ/bseVtK6qo3naZY4KS2i6whPyrBxEGpvjXEqBhArriwB14LUmechBJMRZX9C4XQWohgB3GIxJN8Bwgq9dZEftsmySMefpuByQEFATmm/8r1r67DUhtIaI3xYUnYPIAIECCImU8cLiY5zvQZg9YGs65hWg2whWFJ9PMmaR8olVRyBBPMYgaRwAHc9RYWWGzAvBcxJpPsJT297EuYgR2Eg3SIi4jArrdew/IuoKn/XFqLARWTiXGUBjSHNRSAjELHfhQrxGTGlBWFbRk5vOBdxGhVUCwoZY9ytQYfOSWcJp7GIcwy1U2IDp+/Muwxjx/9wQzubBnt67wMbmbR6TzaTtfBMkH8t8I4B/iuZG+7zkiu+x40fKLTAi4Nb3o+3/dTI63/HyThuW5/qAf//JP8D3mo8wBIXvVl+i9Qb/tH2CL7pT/NHj7+GlXeOJucJfNO/hn198jn9yeB8A8DcXn+LT4QI3doEnxTX+0fZD/Ov3foiftvdxqhs8Mdf4VvEcl26F//nl7+MP3vsJbmyNnStxolv4IPBR9RI+CPy/24/wqNxgCAoflq/wvc038Xcf/TkcBH50eIQHxRYqPs0uSCxUh6fdKZ51J/i33/9jtN7gs/4CF3qPlWr///beNNiS5Lrv+53Mqrp1t/der9Mz0wPMggEwAAVQgEhApmlToAhAEkPQB9qmxAjRFkNb2DQlS3aQZpiyP5k2FZbpCFkKiZIgiQqSEkRLsII2BEC0GCGKg4XEPhxiNszWe7/lvrvUkpn+kFl1696+3dM9vaFn8hfx4t3aMrNOZVVl1jn5TzKpOZEc8LnpYxzWPQyK+7N99us+54ot+rpCiaV2mtO9XXJVcTq7xOcPH2VuMh7KL/NKscMPPv4lvlE8yHPzE3xo+yk+d/goe/WAR/oXOJFM2DMDPrf3CCfzCaVNON3b5dnZCSqn2EoKpibjLf3LbOs5s6Bp/q35MY5mU46nE2amx0Gd82j/As/NT3Aym3C5HpKrCoXjXLnFiWzC1w/u51Q+YSedMTcpF8oRp/M9Xpgd4x2jc7w4P8owKTi72OKh/i7PT48xSgs+sP08/37vMd41OsPLxREOqpzfN36F+9NdPrP7Lo5mM6Z1j74umZuMYVKwncxJxTBQJZXTfHXyILVTHM183gAP9y9xrtwiUzUvHB5jUvX4H77n1wD4xbMf5Eg255H+RV4tdujrkuenx8iUobSaTBneMTpHKobn58dRYtktBxzJZpQ24eH+JV5eHKGnat4zeol90+eZ2Um2kgWXyyFv6V/m2dlx3tLf5YXZMXqqZlL3yHVFTxm+c/wix/Qhv3X4mA/Dtgnn5lu8d+dlLpUjvmP4Mk8ePEpfV9yXHXCu3OKgyhkmBX1dMVAl3zi4n61szjuH5/jSwWkATuUHfGTnq3xt/hC/c/AQp/IDtpM5AId1j8vVEIUPLZ9UOVOTMdRlu1zYhFQZHsz3uFCOOJZO2U7mTEzOM9MTvHVwmbFe8HvTk3zX1rd4cv8R7s/3ATioc/arPn1d8XD/Et+cnuThwSXOLLZJlOHMfJtBUlIazYn8EIBj6ZTKaZ6fHuPdW2f46v4DvG/nJZ6dnWBuUmZ1xn35hP0qp7aKB/oH9HXJ5XLIfpWjxPHBned5YXGMF6bHSMSwnS04mk55sLfHF/bfSu0UC5Oyk83YSecocZxd+HpR2oSH+rsAPHt4nEwbTvYmvDg9SqZrjvemXCyGlCbhgcE+Z+djPnri6+RS8stnvptB4ltrJ/NDnpsc81MFiWWQlLxrfIZtPeff7z3aetvH6YLSJlwqhoySAovw/u1v8dnz7+TDJ5/ic/sPc1DmZEFU6vuOP82ZcofaKg7qPqfzXX5n7yEyXbMwKe8Yn0OJ48u7D/LW0eU27UFSYp3wQye/wK9eeD97RZ+HhnucyCZcqob0VM0Hxs/xf198L6XRPNA/4OmDk7xn5xW+eXiSR0cXeXF6hNL6+W9Lo7l/cEBfV7w82+Hx0XkulCPu60149vA4bxnutvWn+fD36Ogix9NDvjZ5gO103l7P2mqO9GZMqh7jtODcfMyjo0v+GTK5n0wZDqqcJ7bOYp3w1MEp3r19hmcPjzOrMzJlONabMjcpB1VOpmp++NTn+OL0EZ6fHiPXFY8NLjLWC/6/S2/nvnwCwKViQKIsH9x5ns9ceCcPDvZ5z+hlPn3xCT50/Hf5zd3HKE1CaTXfufMyBsXx1B/7/PwEF4sh43D/XSyHvGN0joVNOV+MScTySP8iXzo4zfHeFOOESZVzUOU8PLzM+WLEB3eeZ2FTKqfRYnlpcYRz8y0/jZZTPDE+y37dp7aag7pHKpapyXjP+BUswsKmXK6GfOfoRb44eSsjXfDS/Agne/5e2qv6PD48z5OXH+Y/vf8LXK5HvLA4xlay4KDO6euK4+mEs8U2ryx2+I7xqzw7O8HvG72CEssD6S7nqh1mNuOL+2/hbcMLfPfoOf7N/hOcyg64L91nrOc8NX+Qo4m/b5+Zn2QYyvHE6CwAL8yPMU4WvGNwFo3lN3bfzoP9PVIxXCjHnM53GeiCbxw+sPKceWZ6gkQsszrjgcE+CscL06O8/8iL7FYDLpdDZnXKqf6EJwZnAHhqdj9vH5zlt/Ye5f3b3+KZ2UlO9Q7YrQb8x1tP8yvnv+t1tzueGJ/ld/YewjrhwcEeI1349kg64wu7b+X0YI+L5ZAnRme5UI4pbMJ+lWOd8L7tlzg0PV6cH+Wx4QUO6x571YBJ3eM9W69QOc3ZYotULIkyHElmHIae2LlijBZHEtoS+1XOHz72FF86fAuJGC4WI941PsOrxQ4Hlb+uqTJkqub8YgzA46PzPDs9wf35PnvVgGmd8dBgF43ludlxDsqcQVKixHGyd8jFcsjxbEphE3bLPm8dXObQ9JiblL6uGOmCM4ttHuzvUTnNy7Mdaqt519YZDuo+lVOc6h3w6mKHc4sxj4/O8/TkPgAyVTNIKmqneDDfo7Ap54sR46SgcoqDst/ONb6VLchUTU8Zdss+F+cjRlmBEscgKenrCuOEw6rnnwXaoHAk4Xm5WwzoJxXTKqOfVCF6rcY6xbTOSJXhoMjZ6i04LHv0k2pl/vHKao7lU3YXg3Zd7ZbDTJohRcO0pLaKwiT0dO0dHWnBbjHw85yLnwO8meLNhCFasIyq02IpzHL2gn7q21O180OZerpu0y9MQqIs8ypllBUs6pQ8qaisbqPMgHau8WmZ0UvqNkKrm287v7ZV7bzpibIUYbaEbmRY47ApjW6jyhra4VZh2YV1phNV2BzTLWNDM6RmfV7tRC+Hqplmnu3OOTTXYl0DZX0ffvHa9/edwAHuDTzGUtxGxY57g1PvPur+5C/+AAZFaZP2ZmoeJmX4xNeEmBYmWalk3XDZTBlqp1Zu4IZuZW1uttoqbPtgse1vi5CIoXYahWs7gE1YbZMOgMJRtmW2K2G7LoR9duke2/xuxpA2Ib0iDtM5jyakd/33pnNbDyftlgVYCYtt8tVB2KibDtDayXbOozmmm3e94dgm3+a6dT/gmavYZlN518vc0D2fphxVGHt7xUNog402hYw2/42VlVBhX1a3MrZ203HLsNdlHek+WNsHqlsNl95kt2a5u28Teg2r4UbrHxSbsbJXo+u9Xh9b2Y4htbIy1nQlTHXtfDexHjq7Lg7VXd9Nd7nNrZRvfd2mMq0f23hsX0vwqhtC3JRHNyGqnfPt/m5CoOXqJvD7N2MVQ0jiSllWwugczipErX5i92NPNw8c6taT9WvhNtSBbn3Z+PF503Xtxsg1y23ZVq/t+vZlWTZk2qm0q6G8nW3NoumMpexsF3VlHWzGhOJCumE8aTtelo7dO5/1rxBUW48LbLzIV7NN18PQnGv35uymvckOTVm6Cs7r93BTBiOraax/6W88JMLqmMmGJvR30zVZf6A065pq2XhpXsuL0UmnEXFqtrfqwdJRCu4e383Ldv53ytN4Z183Tbrdc2lYuW5XXm/ZdK275V/nWjceSy9Vu9yGGS+zXTm2vSayWkeaOrSh/N2qulKt1q9X44mDK0OaO8e77vW9WlrruM3F3XiPXIUrvXFXGnRln/U0rzOPK1i3/2uxfs3XjrniPLhyeWOabE7vWmzK63rTu2Z5umlcLzdxy3bL8vW//t980Tn3B15/ajfPlhx1H0w+fMvT/XT9K3f93OAe91hOqh6//srjjHre23G46CHi6GcVw6zk/MGIJIRK5lnFZJoz7JeIOCaHfdKsJkkMi0XKznjO/mGfYb+grBPKIqGap0hiOXb0kP3DHK29oExd+5DP+dR/SRxvzZkvUpRylIuU0dac6cTHyvSHJcUibcVQwDdgksyQZTWHlwegnJ/I3sgyxLBW6HHVirDURYJKbRtm2YxRqyvtwwZLhd4qfWRaodGZxe5muNygcoOdpMigRiUOM0takYd0qwxCMWAXGj2o/XsnNLLMIvGiD6FhpAe1D3tsxC+0g4X24XKVeHGIsL+kFlcpf0yhfRiiOCSzuEKj8ho79eVyhc/bzHW7vxwmsF3hDhPILJJZmKQ+NG6nwhXKj4MKk6rjpA3La15sblQvwxqbcLuF9uF8VlDDyivBvuhDl+uTlZ+vrwwTwVuB3CwbZA5k6m+bdhqI1LZz+/lQQZ9PcqCxqfMhf7XgUodaKGweQgZ7YdxcM/WIhWSiEQP1uIkRApda1FTjtEPPFPWW8SGLIZTOpT7ksBlvY4cGqRRqLpiRT0ctFNmeos4d9Y5BzVU7rsUp2jI3oYiNwIXUq40wvVA+jK7Gh/YpMH2LWvj0zMCSThTV2PrQNuPD6lwSwuDmQj206JnC9sLcernzOvwhPNBJCHkTn7aeKszQhyWang9VInGoELLXnJ8Yfx5NKFl6INS5TwPnw7hU7e1WjWwIqXJIKb6sNOF6hInUmzI4sKBLacPydCnt3H+qEGzP0bvs4xUXJ2xrW10I1cC14VeqFpKZv9bVeBnuKKEq2GwZMplOhPKIJdtV1COHSZfpZPuqzd9pSBZCNbatKEgy9S28amzbUMQmHBCgf1kojjofqjj3599Mf5FOhOKYn0JCFz5kMZ34kNB0LpjcLdtkYR9xPpQunXlbqEIwfR8G6evTMjQumfvwyXrkwx+dcu1cjU05VB3Sm/vwxnrYzMPoy+eUD8VMpqE8ITROVX6bTf11zC8K9cBfQzFCMvf2K7d9SKe3SwibnUBxxJc5mfnfvV2hGvuyebvKauhjb2nXpnzJzNdlG6ah0YWfFqI7CbxLfOijTSA9ZNmZZDVNJ5DOaCeztwltaF0y69y/YbkeLMulKp+/S3w4aLUF+UUf8md6y06JuBDumgabT6HOl+nr0t9TTnlbpFPQc0c1ElQFqvbXq84FVbsQaittOZKZv2dMLoiBZOGXy61lWKxLQJUOZaAaSMhTUJXD9P1zrQ3bdD7kNJ34fVxCCCV1bTip6Xm7VyMhPXStPZyG9NCxOLYppvH6SKb+3H1oq6+X4rw9q5GgC1+uZO7a8FWb+Hdseujrusl8+Zpr57S3q+vUGwBd+WsvLoSZstzmNPT2HdVAWhsmc4dJZSnigy9XE6KqFyHvEmzqnzO6BHGOuifLEFOWIdC68vXQJv7aObVMX4zDZP56AZgwR2k6d8tQ2Nqvb46v8/CubEJhBXQR6k0iKONCeLC0nRpVL5993n7SrluGwko7R2fbEQ226oZCt9fLujaMvLmnVL0Mx5WuR0l8frZbbbpflMJvFe7ZlXyMrw+wLFN3ns/u7DtX3d7pnIoJYbvGXwtxDqube0/aEF+fZmPrznUzbA4rZZlP11bNMZv264YWL+2yXG7CWtfXN9va5F4j9NVJ57yu0uHuluua+7zJEZGPAj+PVx74Befcz97qPO7pjuUwKfn+07/Ht2ZHqa3i1IkJc5NyqRiyt+jz3lOvMqszRmnBxcWQdx8/y5nZNgrHe0++ym7hQ0G2j805PxvzwKlz7Jc5ibIc6c04EkItf2/3BG8/dYHCJF4hNqmxTjh2nw/xeWmyw8nxIdYJ29mcc7Mxbzt6EesUF+dDRjt7K94uPz6zx6xKefc7n8M6YbcY0NM1mapJlGWcFLw4PUJRey/rKCuY1ymzKl3xWG31FiTKMkoKXp1uU1vVhnM8/tgFLhQjLs8HPPzoZc7MtljUCUcenJPritImnJls0U8rKqvYygp2w9jQJnTiyH3z9reIY2/Rp59W9EO4RVEn7ZjSQVqyqFNSbbBOmFUpg7Ti8nTA4GhJntQUtbfhqFewP8859sCM/UVOpg2TRY/t4wv25jn5ds3JRw559WCLk6cP2S9yiirh5OlDRmnBt/aP0k8rFnXSli/VhiyIJekwfcuF6RBjFf3MKwRbBzv9BbPKh6IczHOqSvO+7/1dLMI3LtxHLzHs9OdMih6JskwWvRVv4fFHpgDszfsrY1pro9juL5iWGVpZ7hscMq9Tdhd9Mm1WznvUKzks/Ljc2mivQqwcJ4aHDJKSVw63AaiNZl6mHH94yqJOOD6Y8urBFom2DNKKWZVSG0WiLak2aHFcng5ItWFnMOfS1IfuDLKKt+1c5MJ8xJnJmGGvpBdCcAqjKaokjIu1VEYt7Zl4leOmjKNeybxK6SU1vaSmMpq9ec5WXpBqw96sz8nRIecPRwx6Phy0rBOKWpMoy3Z/weXpgHFeMCtTtHLMirQdm9tLK5RAFu6xw0WPI4M5l6cDjo+mHCx6Xs251vR7JUWV+rGxnfHFZe0fa/dvHzApehzMckQcWWLIs4pRVnL2wIeGGaNIE3+eiTZMi8x/WHHCVn/hr/O0j1L+g9V0kSEC/axiUSVYKwx6FbMi5aH37pEoyzPnj7fCT3lWMZ33WvEkrS1HB/6eOnswbr18WVJjraKoEpLE4JxwcnzIK5e3Of2ePc7sb7VqyQAPHtlnVqWYMB57kFVcmgxbNeOdke8VXD4YMhossFaxKNO2XI8du8gzl45TVZpBXjLole1Y61OjCc9dOoa1Qr9XMpnmbG/N2J/2GfULZkXaUUQW8p4PG5sXGeOBH7M97JXsT/sM85Ky9lEZVeVbZaNBQT+tuHQ48PeNVRijsFaRpbVX0k4MRZEyGCzopzW74RpUlWY8mmOdcDDN2RrNOZjlrXp0Gsaql6Uff/3IiQucnY6ZzHso5djuL0i14czuFr2eH6tchrp/bHvCK5e2SfOK46Mpr1ze5tiRA87tj1tV6vH21F8vbVDi2J/n7TVLlGVRpgwHfqjEvPTPmHFecPlw0M6LXBlNVWn64SPmkdHUj1kPz/TDRY86lB8nDIbzViG8rBOUstS1Zjya4fDRDEWVcHw05fxkRKIN1aJHllXtPTEazrmwO+bEyUvM65TDImvvba0c/bRiWqZURcZgNONgnjMezNt3z2Hp77tLkyGDwYKTQ/9s7mcVg7Sip2suzQfkSY2xiv15jtaGxaLHYDj3H3PnOak2jAc+BP/c/pi0V6IEFlVC3ivJtPH1Ivf1MU0Ms1mvFTvr9SoEWMwz8q0ZdZVQ1Zq6VuR5xai5Z+c5g7xgcjBkuDVlsuiRpzVFlXDf9j7Pnj/+utsd26M584OBr/v9shXn66U1+wcD8ryiLBO2hguqKsEY1aqTHxnPqIxivugx6heY8KytKs2R8ax9Boj4UEKV1u29tijTFYG3qtKMjxwwmQ4QcW2eizKlrnU7B7VWlqL0z8Rhv2C26JEHHQNjFP2eryezRUZdJV6BHMgyQ1VpssxrAVRlQr/v1fGbj+uJtiyKlEFeYpywmHvdifFoTlknOOftUlQJRZEyHBRMp/6ju4hDh3m/87zy2hKlbiM+6lq3EQI6MUEjAepKY0rdqnArZVcUz50Rr/4d8nCADerYrlbtdFwSPkbbOsyfXWo/pr/SiLarAnRW0KnFVB33cDfyookqWYt4AC+SZkvdKqMjLMfgd/ZrozfELfUPLEiy3A60KuSird9PnNcNUDhS2AAAGOVJREFUSK0/vyDK1qptwzLSoPkIv8H73UY1tBEW+P0a9e5wzivpNR74TR7Zdl3Tu5dVF+jKMWs9wE5HemV53cu8KSrGyZXru/za5tV3mrsRCisiGvibwA8ALwOfF5FPOue+cUvzuZdDYY+886T7nr/7n7GVLiitZrfwjehxumCQVJyZbdHTNfM6ZZiW7C76jDM/HmK/yMkT34k7LDNODKZcmg8YpBW1VcyrlGmRkWrDyZHv2KTKx8c3YZOz0n9CPDacMav871mRsTOYc7Do4ZywlRcs6mQlDNRYoZ9VZNpwYTJqBW3q8JJpHqqjwQKtfCinn9LEN4C74XzTImsbUqOBf6kWlRfG2Z8MyLKaPKs4nObtS9C/eHxDbGu4oKj8S6esEvq9su3sJNowL7K20Qa+oVxbRVn6l1+SWMoiIc1qqkqTps30IZCmNWWZMMiDeE9IJ0kMZZnQy2rmi5Rer6auNXlWMVtkDPKSymjms4zBsGA+D+I9iWE+62FrYTAuKEtNEl4oTcPDGEUjQgPQ6/uXZlXp9sVUlYl/oRnV2mTv3BgE8p0Fxgim0v5l5YQs8y933/kRyrm/1knmr0WTFuKoy6QVwjGzxIvb9IwXxdEOU2iSnsFUCp026rJL8R07S1pPajONhk6M954rh1to1LD202iEl6RKvDAOQWwnHZSYSmODFxgnfgqMSYLLLXpYt2I7zUtEgghOk2YratOEBzbe9kIvp2AILxzpGb/eCpIb3EwjA+MFbiyQuuVLsFR+n4WGzPoXZ+MZVq4VzGm92z2LW2ikX+PmiRd0Ufj0GgEggkfaSrsdQKYal1mfhsN72a0EMRjj0wnTadBM1ZHa5Yu0KUvjWa7VUk22VsupK2rlRYL2E++lOVovp30oFeTh+JCGWni72GFnUsEmvK4zZYSaauxWjTpIsAPr824aWhMdBFW8N1gVCjsw7UtVFv4Z5QYGKby3xHWOT/c01Y4XBJJGICf1jYVkJlTbXrxHKsHl3mNug2faZqviPVL6RozLLGruPfaqFGzuvcFO4a9t09AqFKoSzCB48pWjmSqiERHCeDEgtQj7NuI9iUMF8R7b9/nZzAWvPyviPVgh21OY3GF7wetceE9a41EHWvGdZCZUWxZVKC9OM3YkE8EMgodWgZ6veSwzWm+8OLDJUlSn9ViW3sPWRghoXz5VeA9RMmvEe0JDN3OtZxKWnq1GQIUg2pEET7T37vh8Te4bhDb13pfWY7kQ6qEj2/NeC+/RlXaKi8ZT04gJmcz/h+DFCUIvZuDz0YX3aqq68RgFUZhQBRsPke15jzjgxX2sPx8I3tUgfuMSWhGgrnfRp0HrxVFlaEf3vEfVKX9ss60rRKNLL2yTzDuCNQkkUyi3ed3oBRj//XUp3oMva2MTL6ZEK5qz9NYT7O+3W+3rlW080MF2jcetEYMBWu9n12OZTpZCQk6HNJOOJ4krPZZ2TbynK3jU1C9x/no0XjwX6p4OgkRt38D6/FQV0mjymXuvms/fK8Db1Ht4TU9ar1gzxYcug/cwkWW0UfDS45YexMaDZlNZWdd6/TZ5LAWsltZrJ9a1ZXeyzKP1WGpfj9c7SdIV7+l2fDresW4ocitMZJbXrlvOTV616/JYron3tPdGI7bU8YB2022uofdYspnmmW6XAjobhXMaj2Wz7SrpbRjdsLL+WmW4mhf0Wrxm+C3w5C/91bseLrolR90H1A/c8nQ/Y//pNc9NRP4g8D865z4Sln8KwDn3P9/KctzTHsuHe3v88mOf5JwpqRBycVQOFk4zsRlH9YJFeBLMbMq2KrhscwyKY2rOwmmmLmMoJWfNFif0hInN0ThyqRioiplNOW/G7OgZ1ikWLiWVmsol7KgZBuFsvcOOnlK5hLGac7be4URyAMCeGZCJQeG9aJVLSKVmYvssbMqpZB+DULkEg5BLhcaRSs2eHfj1TpGrksolLGyK6siVNWlvqQWXzKhNY+FSTuoJBzZnZnvs6BkHNmdhU3b0jEwMC5dyqR6RKz/wfahKJjbHONWe41AVK/lNTJ9UanJVhbIJW3rBgcnJxFAGe2txVE6jsExs36cT3hjN8sKmjPWCicnJVcXE9Bmogn0zYKAKdvSMS2bEUBVMrQ873tHBE1OPGKiChUvRLO2ahtabFotxin3jPzb0VEUVyjZQJUWIK5rZDOMUb3vCe5+/VR5vxX9mNiMVw6HJV2ywrYPoTXijL2zKQBdUTpNL3eZzNDmkdAn7pk8qhsppxmrB5XrEWC/YN316qqKwKakYjFMcTQ69J8T00bhW1GM7mVHYlG0942I9JhXTHluFMbypqtG4VkBppBfs1/78c1VxPJkwsz0u1iMGqlwZG7wI9shV1Qo1peIFEKr2bUq47j6/RpBqYnIGqiRVNfv1gKPJtC0DeMGqIoi9NHYdqJLCJSgcM5uFa6gZ6LJN2zjFzGbeVnW/FbIweFXinqoxBEED5a99YZO2vEeSGTObcWh6aCw9Vbdl2K29mIJB+euty5VyNmn667w8vggthGacdeU0PeWv+fHUi6WcLbbb+tLXFdPa110lfmqckS7QYtmv+61dUzFBrCklUf73SBdcqoat6EgzVhtgJ51ROe29faggApW3Y7n7obV4aHr0ddUKQaWhXCeySSss0tdle25KHEeSGefKLYyTVihmO50zrXsMk4K5SZdjyp3Q1/46N2kVNqGnauYmo69Laqfb8e+NTXqq4qDukyqDcYJdE6ZIlKG0CUPt6+nc+DpZO8U4WVA7zWGdMUpK5ibFhLSb8/PnYjmRHTK3GXOTonBtWfeCmFRjFyWWnXTO+WJMX1eMkwUXixFHsxl7VR/rhNopttNFey21OA7rDOtUuzw3KUNdYpH2vPq6YlL3WrGQJq0mamScFK0tm7KXVrf1sFE9b86/GSc/TMpwfypqqxkmJdPa26kRGmuOyZThsOpxIj+kdqq1R4MSn2ZpNLmuKa3/D5Aq014/L4xSM0xKJlWPTJkgBGJZmLTVKmjmiy5t0gpZLUKeeeKvwUGZk4frUTtNFu63hUlJxLa6AYugjdAVOilMwjCIG/mpuhSpMuS6onaa0mgybfxH5aRkEUROaqsYJCX75fLeu1EaERiATJl2PHKiLNMqa23g67ZqpxlzTsiTChPKnCmDRXz5raIf7FJ24i2btIAQcbOqYzBMSwqThPH9qo3e6dJoGTggC9ubCBzrpJ0erbKq3dZoEzRTlTW6Ck00UvfjdnfqtdporIM8rdvyNjoMtVGtl7yhidho53zuaBDYjhdfwnkY57UCmo/U3XH+/pi1eZ0DjRhMd9x89xiRzvzSjeeww4pnsl3X+R3K2I7lZ9nvFNmgX3E1b9panv74zrLDz+u83oHqdsau0glzjQdzPesr0lr1+F3T9/Ra5+Gu8ptOOTal/1pex6vlcz0d0F+6viRvO3dnupEHgZc6yy8DH7jVmdzTHcvLpsc/OHiMymkqp7GuecHUFDbFdEa1dxvHTYOyK8jSfVir0CnSoSNkggtEY9vfDbVtRF9WBXyAVsRnHbN2s3RFgDat75a7y7ooje08MZoGx3p6TTmbxsIm1kWCmhf8pnyudny3fM3/TeJDmwSDuuk0Zd50Duv2aravLK/ZpLu+W57fdI9eUd7uuQCtGNG1ytwt+9XOb13AZt2+62VvOgvddNfz6nItu65fm037X4/AznpZbsQmVyvHa9Hd92rvjquJGW26168l6NSsu147XC+vldbm9P1/JVeu66Z7heDOdZZ/U/nWj9tU77rbVtJgc9vlWu2I6ynT+rpuGa92nmtDoF5XPq+V/rrAU0MTRXE1oahuuq+V/sayX6vxdY1G2SaxKrhKm7TJo/m9Mb31FVfmt8kGG+fidXQEq66d7np52nJ07dK1w/rv18smO2y8PjdwfPeY6/TOXPeN9DqeOVfNY8W7Ju1cpLKex6ayrZ/XtTod7fpQV65l3w3bXtMjtiGP7nGr98Dauk35rf1et3jbKrnua7Z5dfdWvKEq/Fr16jXyuyLTDXXquspzo2W+AW7mlr7TTNj91GfcJ15/PP7VyUXkC53lv+Oc+zu3IZ9rck93LAub8vTslJe9twmFSaicoqcMqTLtl+w6fP1tvuLWnS+GzdfARGz7lbi2itotv7D7qRCSVm0WfIeyObb9Eh2+PmbKtF9Im/QausqyShxFnWCR1kMDtJ2XRq7aK6SqVvW1ScdY1SrYrn/xTJRtx2em2jCv/NjH5qttE5qbBgnnJsR3XW3VWEUVOttAK0PdPKeaKQcSZTEbGp/GKvIkjKNyS5XS2ir/9dSqVvlVh6+maUirrBOypKasE5IwpqkymtooemG80vqXU6+qusw/DeMpuo3frnKrDhPbTwv/Bdp/aZUV9VMd5r9sjqvD+AfVsVXTeGrGIjrn64KIQ3c6NMZJO4YvUesfEoQ6zMOptV0qmgY7ijiMUe0YvPWvtV1pceOEOoxVa8pXln4sWKPc2nzF7Y7bWW8oNw3jpnHb2KVRgW3GDjbLXtzK59uVDG+UY5e/l8qxzfFK2TaMulFObcYMNv+bsqx/fbadL8kqfHU2tUZp2y4D2Ma+wYat7TYor9omv+Z4t1Qg9V+37cr4GlNrcKDTjtLBhpdxV0F3nZUPDtbXAxPmK11RCW5CdyU0xI2g9FJB2QblTqU7duo0rG2l/Bgl8Y14Py4p2MIIKvXnZkO6ttnezGsqq52eVsG2Ebmy4sOV1zox4PdtxyB1epwinQ5FZ7yPcyz3hXb8URNe7cct+XO74it/I/DV2DqMbZLEtfO+tq3ASnx4dhPmnVkfytyIVwmtSFc7Xkh17NoaP1zrJk8rbagusDymsVVXqdXhj+sub+pkiWNFtVVYhrbSObbZxeLneA1jxJp5Pp0K6qoOH5LcpKNcKwYmNoTqhnBmMbJUVrUsjw9htdL+9ucqzXyaYV2TbhN67W0S8rO086m2/cLOfSJhf6dcG/ZMY267mlcrPFJLu9yEW9ve62+FdgXNVoRL3Go+q9cn7G+a8rMMc5VQ/mZa247i7EpfoKu4Go6TEHYrwQYSRNBa29HpYzfHdpVym3PA26pbZZp9uiGjzZyWbcFcJ71OGaUT5d/Us6YutHXULR+NK9e1qW9duzX3qlvmQXtsx0iy1unqrG9CRq97HsfXUUWu6Pg1j6a1Ml/P8evrVo5f7yQ2z51m/frzotm3e62ukne7/8aO6GpH/5od3Ney3+3sOL5WR/ou45z76F3K+hXgoc7y6bDulnJPj7EUkQnw9N0ux5uM48DFu12INxnR5neeaPM7T7T5nSfa/M4S7X3niTa/83w72/ytzrkTd7sQdwMRSYDfA74f36H8PPCnnHNfv5X53NMeS+Dpuz0I982GiHwh2vzOEm1+54k2v/NEm995os3vLNHed55o8ztPtPm3J865WkT+K+BT+OlG/v6t7lTCvd+xjEQikUgkEolEIpHINXDO/Rq3edIV9dq7RCKRSCQSiUQikUgkcnXu9Y7lHVc7ikSb3wWize880eZ3nmjzO0+0+Z0l2vvOE21+54k2fxNzT4v3RCKRSCQSiUQikUjk7nOveywjkUgkEolEIpFIJHKXuWc7liLyURF5WkSeEZGfvNvleSMgIg+JyK+LyDdE5Osi8hNh/VER+bSIfDP8PxLWi4j8H+EafEVE3nd3z+DeRUS0iPyOiPyrsPyIiDwZbPsrIpKF9b2w/EzY/vDdLPe9iojsiMgnROR3ReQpEfmDsZ7fXkTkL4fnytdE5JdEJI/1/NYiIn9fRM6LyNc66264XovIj4b9vykiP3o3zuVe4So2/7nwbPmKiPxfIrLT2fZTweZPi8hHOutjm+Y62WTzzra/IiJORI6H5VjPbwFXs7mI/Hio618Xkf+1sz7W8zcp92THUkQ08DeBPwK8C/iTIvKuu1uqNwQ18Fecc+8CPgj8l8GuPwl81jn3OPDZsAze/o+Hvz8H/K07X+Q3DD8BPNVZ/l+Av+GcexuwC/xYWP9jwG5Y/zfCfpEb5+eB/9c5907gvXjbx3p+mxCRB4H/GvgDzrnvwEud/zCxnt9qPg6sT759Q/VaRI4Cfw34APDdwF9rOqORjXycK23+aeA7nHPvwc8b91MA4X36w8C7wzH/Z/ioGNs0N8bHudLmiMhDwIeBFzurYz2/NXycNZuLyB8CPga81zn3buCvh/Wxnr+JuSc7lviHwDPOueeccyXwy/jKHbkJnHNnnHO/HX5P8I3tB/G2/Ydht38I/Inw+2PAP3Ke3wJ2ROT+O1zsex4ROQ38MeAXwrIAHwI+EXZZt3lzLT4BfH/YP3KdiMg28B8Bfw/AOVc65/aI9fx2kwB98ZM0D4AzxHp+S3HO/QZweW31jdbrjwCfds5dds7t4jtJVzTiI55NNnfO/WvnXB0Wfws4HX5/DPhl51zhnHseeAbfnoltmhvgKvUc/Eeo/w7oiofEen4LuIrN/yLws865IuxzPqyP9fxNzL3asXwQeKmz/HJYF7lFhNCz3w88CdznnDsTNp0F7gu/43W4Nfzv+JehDcvHgL1Ow6Rr19bmYft+2D9y/TwCXAD+gfjw418QkSGxnt82nHOv4L9mv4jvUO4DXyTW8zvBjdbrWN9vLX8G+H/C72jz24SIfAx4xTn35bVN0ea3j7cD3xuGK/xbEfmusD7a/E3MvdqxjNxGRGQE/HPgLznnDrrbnJcRjlLCtwgR+UHgvHPui3e7LG8iEuB9wN9yzv1+YMoyPBCI9fxWE0LMPobv1D8ADInegTtOrNd3FhH5afwQk39yt8vyRkZEBsB/D/zM3S7Lm4wEOIofOvXfAv80RpZE7tWO5SvAQ53l02Fd5CYRkRTfqfwnzrlfDavPNaF/4X8T7hCvw83zPcAfF5EX8GEhH8KP/9sJIYOwatfW5mH7NnDpThb4DcDLwMvOuSfD8ifwHc1Yz28ffxh43jl3wTlXAb+Kr/uxnt9+brRex/p+CxCR/xz4QeBH3HJet2jz28Nj+I9WXw7v0tPAb4vIKaLNbycvA78awow/h4+6Ok60+Zuae7Vj+XngcfGKghl+kPAn73KZ7nnCl6a/BzzlnPvfOps+CTSKaT8K/MvO+j8dVNc+COx3Qq4i14Fz7qecc6edcw/j6/G/cc79CPDrwA+F3dZt3lyLHwr7Rw/EDeCcOwu8JCLvCKu+H/gGsZ7fTl4EPigig/CcaWwe6/nt50br9aeAD4vIkeBp/nBYF7lOROSj+OENf9w5N+ts+iTww+JVjx/BC8p8jtimuSmcc191zp10zj0c3qUvA+8Lz/pYz28f/wL4QwAi8nYgAy4S6/mbG+fcPfkH/FG82tqzwE/f7fK8Ef6A/xAfJvUV4Evh74/ixzZ9Fvgm8BngaNhf8ApfzwJfxSs+3vXzuFf/gO8D/lX4/Sj+QfwM8M+AXlifh+VnwvZH73a578U/4DuBL4S6/i+AI7Ge33ab/0/A7wJfA/4x0Iv1/Jbb+JfwY1grfOP6x15PvcaPC3wm/P0Xd/u8vp3/rmLzZ/BjyZr36N/u7P/TweZPA3+ksz62aW7C5mvbXwCOh9+xnt8mm+M7kr8Ynum/DXyos3+s52/SPwkXOhKJRCKRSCQSiUQikdfFvRoKG4lEIpFIJBKJRCKRbxNixzISiUQikUgkEolEIjdF7FhGIpFIJBKJRCKRSOSmiB3LSCQSiUQikUgkEoncFLFjGYlEIpFIJBKJRCKRmyJ2LCORSCTybYuI/Gb4/7CI/Km7XZ5IJBKJRCKbiR3LSCQSiXzb4pz7D8LPh4HYsYxEIpFI5NuU2LGMRCKRyLctInIYfv4s8L0i8iUR+csiokXk50Tk8yLyFRH582H/7xORfysi/1JEnhORnxWRHxGRz4nIV0XksbDffyIiXxORL4vIb9yt84tEIpFI5I1CcrcLEIlEIpHIdfCTwF91zv0ggIj8OWDfOfddItID/p2I/Ouw73uBJ4DLwHPALzjnvltEfgL4ceAvAT8DfMQ594qI7Nzpk4lEIpFI5I1G9FhGIpFI5F7kw8CfFpEvAU8Cx4DHw7bPO+fOOOcK4Fmg6XB+FR9SC/DvgI+LyJ8F9B0rdSQSiUQib1CixzISiUQi9yIC/Lhz7lMrK0W+Dyg6q2xn2RLee865vyAiHwD+GPBFEXm/c+7SbS91JBKJRCJvUKLHMhKJRCL3AhNg3Fn+FPAXRSQFEJG3i8jwehMTkcecc086534GuAA8dEtLG4lEIpHIm4zosYxEIpHIvcBXACMiXwY+Dvw8Pqz1t0VE8J3DP3ED6f2ciDyO93x+FvjyLS1tJBKJRCJvMsQ5d7fLEIlEIpFIJBKJRCKRe5gYChuJRCKRSCQSiUQikZsidiwjkUgkEolEIpFIJHJTxI5lJBKJRCKRSCQSiURuitixjEQikUgkEolEIpHITRE7lpFIJBKJRCKRSCQSuSlixzISiUQikUgkEolEIjdF7FhGIpFIJBKJRCKRSOSmiB3LSCQSiUQikUgkEoncFP8/rlYJ16dR+iIAAAAASUVORK5CYII=\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ - "# What if we let the linear model train for a longer time?\n", - "results1 = train(net1, train_test_data, num_epoch=30, learning_rate=0.02, ctx=ctx)" + "evaluate_embeddings(ratings)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Next steps\n", - "See [this notebook](demo1-MF2-fancy.ipynb) to try using fancier network structures and optimizers on this same problem." + "## Visualizing training\n", + "Now let's draw a single chart that compares the learning curves of the two different models." ] }, { "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, + "execution_count": 25, + "metadata": {}, "outputs": [], - "source": [] + "source": [ + "train_1, test_1 = list(zip(*losses_1))\n", + "train_1a, test_1a = list(zip(*losses_1_adam))\n", + "train_2, test_2 = list(zip(*losses_2))\n", + "train_2a, test_2a = list(zip(*losses_2_adam))\n", + "train_3a, test_3a = list(zip(*losses_3))" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[(1.2344593836784363, 0.713362567743678),\n", + " (0.648405224943161, 0.6597177934874395),\n", + " (0.5851686426639557, 0.6617723923579902),\n", + " (0.5194843295574189, 0.5935856424699164),\n", + " (0.45305402550697327, 0.5900366710629433),\n", + " (0.2977709783792496, 0.49029243049348237),\n", + " (0.27699643795490264, 0.4890738264390617),\n", + " (0.2709969203948975, 0.4920012930016609),\n", + " (0.2653717215538025, 0.4948571656539941),\n", + " (0.26125567021369933, 0.4921688241943432),\n", + " (0.23106717193126677, 0.48678314078385665),\n", + " (0.22838653769493103, 0.4875897347547446),\n", + " (0.22782497510910035, 0.4886196310732775),\n", + " (0.22744231758117675, 0.48978534644576394),\n", + " (0.227169718003273, 0.48986973580281445)]" + ] + }, + "execution_count": 26, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "losses_1_adam" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAABIkAAAR8CAYAAAAZ7cTyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XtYlVX+///nQhA8BYTON095+pSB7C0mKEYmZOOJooONYx9JrV8Hf2VO2RDamFnfnHR0yuiTOR1My/nszJpsTH95SBBrMgXCs4UHGtQycWInopPA+v2xt3swAdGEbfp6XBfXxX2ve631vm/vrive13ut21hrERERERERERGRi1uAvwMQERERERERERH/U5JIRERERERERESUJBIRERERERERESWJREREREREREQEJYlERERERERERAQliUREREREREREBCWJREREftGMMdYY819n2bevMebLcx1THebtaozJN8YcNsaMq6c5HjfGvHaur/UnY8w8Y8wzfpj3cmNMqTGmUT2M7Zd7EhERkeopSSQiItIAjDGFxpij3j+2T/z8TwPHcFJCyVq71lrbtSFj8HoMyLTWtrDWZvy00RiTZYy55+dMYK39o7W2TmOcybXnK2PMaGPMJ+dorEJjzA0njq21/7TWNrfWVpyL8UVEROT8FejvAERERC4iN1lrV/k7iPNAB+Dts+1sjAm01pafw3hEREREBFUSiYiI+JUxJtgYU2KMia5yrpW36uhX3uN7jTE7jTH/Msb83RjTpoaxTqrAqVpdYozJ9p7e6K1i+q0xJtEYs7fK9ZHeMUqMMVuNMSlV2uYZY14yxiz1LhP73BjTpZb7SvGOUeIdM9J7fjWQBPyPN44rf9JvKtC3Svv/eM9bY8yDxpgCoMB77gVjTJEx5gdjTK4xpm+VcaYYYxZ4f+/o7T/KGPNPY0yxMeYPZ3ltE2PMfGPM98aY7caYx6o+w2qew+lifMcY86b3mW41xsRWae9hjMnzti0EQmqYIxKYA/TxPrMS7/lgY8xM730cMMbMMcY08ba1NMZ86P33+ZcxZq0xJsAY8xZwObDEO9ZjVZ5JoLdvljHm/xpjPvXGtsIY07JKPCONMV8bYw4ZY574aWVSbWp6143H88aY77zPcrPx/jdjjBlijNnmjWWfMeb3Vca70XiWNpYYY/5hjHFWaUv3Xn/YGPOlMaZ/XWIUERG5kClJJCIi4kfW2n8DfwPuqHJ6GLDGWvudMeZ64FnvudbA15xFFY619jrvr929S4cWVm03xgQBS4AVwK+Ah4C/GmOqLkcbDjwFhAM7ganVzeVN/LiAh4FWwDI8SYfG1trrgbXAWG8cX/0kzj/8pH1sleZbgN5AlPd4AxADXAr8L7DIGFNtIsXrWqAr0B+YfCJxdYbXPgl0BDoDvwZSaxmjLjGm4Pn3DAP+DpxIijUGFgNvefsuAoZWN4G1djswBvjM+8zCvE3TgCu98/8X0BaY7G17FNiL59/n/wCPe4aydwL/xFP11txa+6ca7uu/gbvwvCuNgd97444CZgMj8Lyvod55T+s07/oA4Drv/YR6rznkbXsduN9a2wKIBlZ7x+sBzAXuByKAvwB/9ybPugJjgThvv4FAYV3iFBERuZApSSQiItJwFnsrGk783Os9/794EjAn/Lf3HHj+2J5rrc3zJpQm4qkY6XiOY4sHmgPTrLU/WmtXAx9ycvLqfWvteu9Sr7/iST5U57fAUmvtSmvtcWAm0AS45mfG+Ky19l/W2qMA1toF1tpD1tpya+2fgWA8iZ2aPGWtPWqt3QhsBLqfxbXDgD9aa7+31u4FTtlTqao6xPiJtXaZd7+ft6rMEw8EAbOstcette/iSTjViTHGAPcBj3if2WHgj/znPTuOJxHTwTv+Wmutrev4wBvW2q+8/xbv8J934XZgibX2E2vtj3iSUnUdt7Z3/TjQArgKMNba7dbab6rcS5Qx5hLvv0ue9/x9wF+stZ9bayustfOBf+N5thV4/i2ijDFB1tpCa+2uM7h/ERGRC5KSRCIiIg3nFmttWJWfV73nM4Gmxpje3j+IY4D3vW1t8FRUAGCtLcVTQVGn6owz0AYostZWVjn39U/m+bbK72V4kko1jVU15kqgiJ8fc1HVA2PM771LvtzeJVahQMvquwJ1j7+2a9v8JI6TYvqpOsT403lCvMu62gD7fpK4+Zq6awU0BXJPJCWBj7znAWbgqQZbYYzZbYyZcAZjVxd3tc/HWlvGfyp+TqfGd92btPwf4CXgO2PMK8aYS7yXDgWGAF8bY9YYY/p4z3cAHq2amAXaA22stTvxVLpN8Y73tqlhGaeIiMjFREkiERERP/NWkbyDp2rnDuBDb+UHwH48f+wCYIxphmfpzL5qhjqCJzFwwmVnEMZ+oL0xpur/G1xewzx1GatqzAbPH+d1HaumyhPfeePZ2+cxPJU94d4lVm7AnEW8Z+IboF2V4/Y1XfgzY/wGaOt9didcXsv1P31mxcBRoFuVpGSotbY5gLX2sLX2UWttZzxL3sZX2ZPnTCqKqovb93y8eyBF1LFvre+6tTbDWtsTz3LDK4E07/kN1tqb8Sx9W4znvyXwJKum/iQx29Ra6/L2+19r7bXeOS0w/SzvWURE5IKhJJGIiMj54X/xLNMawX+WmoFnb5+7jDExxphgPEuGPrfWFlYzRj5wmzGmqfF86v7/+Un7ATx76VTnczwVIY8ZY4KMMYnATZzdV8jeAZKNMf29ex09imeZzz/q2L+2OE9oAZQDB4FAY8xk4JLau5wT7wATjTHhxpi2ePa1qY8YP/P2Hef997gN6FXL9QeAdt69jE5Ub70KPG/+swF6W2PMQO/vNxpj/subhHLjWX5VWWWs0z3/mrwL3GSMucYbyxTqnrir8V03xsR5K+2C8CRDjwGVxpjGxpgRxphQ79LGH6rcx6vAGG8/Y4xpZoxJNsa0MMZ0NcZc753nGJ6EWuUpEYmIiFxklCQSERFpOCe+GHXi58SSMqy1n+P547cN8P9VOb8KeAJ4D0+VRhdO3r+oqueBH/H8kT8fz75BVU0B5nuX3gyr2uDdP+YmYDCeKpTZwEhr7Y4zvUlr7Zd4NnR+0TvWTXg2Qv6xjkO8ANxuPF8Qq2nPn+V4lk99hWeJ0jFOs/TrHHkaz4bPe4BVeJIi/z7XMXqf1W3AaOBfeBKIf6uly2pgK/CtMabYey4dz5KydcaYH7zxntgP6QrvcSmehNRsa22mt+1ZYJL3PfF9KayOcW/Fs+n523je11LgO2p+RlX71vauX4In6fM9nmd5CM+SOYA7gULvPY7Bk2jFWpsD3Itnmdr33mcx2tsnGM/G3sV4ls79Cs8eSCIiIhc1c2Z7FIqIiIjICcaY/xcYbq3t5+9YzkfGmOZACXCFtXaPv+MRERGR2qmSSERERKSOjDGtjTEJxpgA72fUH+U/m4wLYIy5ybvksRmeL9ttRp+XFxER+UVQkkhERESk7hoDfwEO41ni9QGepXnyHzfj2YR6P55lbcOtStdFRER+EbTcTEREREREREREVEkkIiIiIiIiIiJKEomIiIiIiIiICBDo7wCqatmype3YsaO/wxARERERERERuWDk5uYWW2tbne668ypJ1LFjR3JycvwdhoiIiIiIiIjIBcMY83VdrtNyMxERERERERERUZJIRERERERERESUJBIREREREREREc6zPYlERERERERELmTHjx9n7969HDt2zN+hyAUoJCSEdu3aERQUdFb9lSQSERERERERaSB79+6lRYsWdOzYEWOMv8ORC4i1lkOHDrF37146dep0VmNouZmIiIiIiIhIAzl27BgRERFKEMk5Z4whIiLiZ1WpKUkkIiIiIiIi0oCUIJL68nPfLSWJRERERERERC4SJSUlzJ49+6z7z5o1i7KysmrbEhMTycnJOeuxa5KVlcWNN954yvl77rmHbdu2nfP5LmZKEomIiIiIiIhcJOozSdTQXnvtNaKioupt/PLy8nob+3ylJJGIiIiIiIjIRWLChAns2rWLmJgY0tLSAJgxYwZxcXE4nU6efPJJAI4cOUJycjLdu3cnOjqahQsXkpGRwf79+0lKSiIpKanWeVwuFw6Hg+joaNLT0wGoqKhg9OjRREdH43A4eP755wHIyMggKioKp9PJ8OHD63wvVSuXmjdvzh/+8Ae6d+9OfHw8Bw4cAODgwYMMHTqUuLg44uLi+PTTTwFYv349ffr0oUePHlxzzTV8+eWXAMybN4+UlBSuv/56+vfvX+dYLhT6upmIiIiIiIiIHzxcUEB+aek5HTOmeXNmXXFFje3Tpk1jy5Yt5OfnA7BixQoKCgpYv3491lpSUlLIzs7m4MGDtGnThqVLlwLgdrsJDQ3lueeeIzMzk5YtW9Y4x/79+0lPTyc3N5fw8HAGDBjA4sWLad++Pfv27WPLli2Ap6rpREx79uwhODjYd+5MHTlyhPj4eKZOncpjjz3Gq6++yqRJk/jd737HI488wrXXXss///lPBg4cyPbt27nqqqtYu3YtgYGBrFq1iscff5z33nsPgLy8PDZt2sSll156VrH8kilJJCIiIiIiInKRWrFiBStWrKBHjx4AlJaWUlBQQN++fXn00UdJT0/nxhtvpG/fvnUec8OGDSQmJtKqVSsARowYQXZ2Nk888QS7d+/moYceIjk5mQEDBgDgdDoZMWIEt9xyC7fccstZ3Ufjxo19+xb17NmTlStXArBq1aqT9i364YcfKC0txe12M2rUKAoKCjDGcPz4cd81v/71ry/KBBEoSSQiIiIiIiLiF7VV/DQUay0TJ07k/vvvP6UtLy+PZcuWMWnSJPr378/kyZN/1lzh4eFs3LiR5cuXM2fOHN555x3mzp3L0qVLyc7OZsmSJUydOpXNmzcTGHhm6YqgoCDfl70aNWrk20+osrKSdevWERISctL1Y8eOJSkpiffff5/CwkISExN9bc2aNftZ9/lLpj2JRERERERERC4SLVq04PDhw77jgQMHMnfuXEq9y9727dvHd999x/79+2natCmpqamkpaWRl5dXbf/q9OrVizVr1lBcXExFRQUul4t+/fpRXFxMZWUlQ4cO5ZlnniEvL4/KykqKiopISkpi+vTpuN1uXyznwoABA3jxxRd9xyeW2bndbtq2bQt49iESD1USiYiIiIiIiFwkIiIiSEhIIDo6msGDBzNjxgy2b99Onz59AM8G0AsWLGDnzp2kpaUREBBAUFAQL7/8MgD33XcfgwYNok2bNmRmZlY7R+vWrZk2bRpJSUlYa0lOTubmm29m48aN3HXXXVRWVgLw7LPPUlFRQWpqKm63G2st48aNIyws7JQxP/74Y9q1a+c7XrRoUZ3uNyMjgwcffBCn00l5eTnXXXcdc+bM4bHHHmPUqFE888wzJCcnn9EzvJAZa62/Y/CJjY21J3YmFxEREREREbnQbN++ncjISH+HIRew6t4xY0yutTb2dH213ExERERERERERJQkEhERERERERERJYlERERERERERAQliUREREREREREBCWJREREREREREQEJYlERERERERERAQliUREREREREQuGiUlJcyePfus+g4ZMoSSkpI6Xz9lyhRmzpx5VnOdTvPmzU85N2fOHN588816me9ioSSRiIiIiIiIyEWitiRReXl5rX2XLVtGWFhYfYR1TowZM4aRI0fW2/jWWiorK+tt/POBkkQiIiIiIiIiF4kJEyawa9cuYmJiSEtLIysri759+5KSkkJUVBQAt9xyCz179qRbt2688sorvr4dO3akuLiYwsJCIiMjuffee+nWrRsDBgzg6NGjtc6bn59PfHw8TqeTW2+9le+//x6AjIwMoqKicDqdDB8+HIA1a9YQExNDTEwMPXr04PDhw3W6t6qVS4mJiaSnp9OrVy+uvPJK1q5dC0BFRQVpaWnExcXhdDr5y1/+AkBpaSn9+/fn6quvxuFw8MEHHwBQWFhI165dGTlyJNHR0RQVFdX1Uf8iBfo7ABEREREREZGLVeIXX5xybtivfsUDbdtSVlHBkE2bTmkffdlljG7dmuIff+T2rVtPasvq0aPW+aZNm8aWLVvIz8/3XJ+VRV5eHlu2bKFTp04AzJ07l0svvZSjR48SFxfH0KFDiYiIOGmcgoICXC4Xr776KsOGDeO9994jNTW1xnlHjhzJiy++SL9+/Zg8eTJPPfUUs2bNYtq0aezZs4fg4GDfUraZM2fy0ksvkZCQQGlpKSEhIbXeU03Ky8tZv349y5Yt46mnnmLVqlW8/vrrhIaGsmHDBv7973+TkJDAgAEDaN++Pe+//z6XXHIJxcXFxMfHk5KS4rvX+fPnEx8ff1Zx/JKokkhERERERETkItarVy9fggg81T3du3cnPj6eoqIiCgoKTunTqVMnYmJiAOjZsyeFhYU1ju92uykpKaFfv34AjBo1iuzsbACcTicjRoxgwYIFBAZ66lgSEhIYP348GRkZlJSU+M6fqdtuu+2U+FasWMGbb75JTEwMvXv35tChQxQUFGCt5fHHH8fpdHLDDTewb98+Dhw4AECHDh0uigQRqJJIRERERERExG9qq/xp2qhRre0tGzc+beVQXTRr1uw/8WRlsWrVKj777DOaNm1KYmIix44dO6VPcHCw7/dGjRqddrlZTZYuXUp2djZLlixh6tSpbN68mQkTJpCcnMyyZctISEhg+fLlXHXVVWc89okYGzVq5NtvyVrLiy++yMCBA0+6dt68eRw8eJDc3FyCgoLo2LGj776rPp8LnSqJRERERERERC4SLVq0qHWPH7fbTXh4OE2bNmXHjh2sW7fuZ88ZGhpKeHi4b1+gt956i379+lFZWUlRURFJSUlMnz4dt9tNaWkpu3btwuFwkJ6eTlxcHDt27PjZMZwwcOBAXn75ZY4fPw7AV199xZEjR3C73fzqV78iKCiIzMxMvv7663M25y+JKolERERERERELhIREREkJCQQHR3N4MGDSU5OPql90KBBzJkzh8jISLp27XrOllnNnz+fMWPGUFZWRufOnXnjjTeoqKggNTUVt9uNtZZx48YRFhbGE088QWZmJgEBAXTr1o3BgwefMl5ZWRnt2rXzHY8fP75Ocdxzzz0UFhZy9dVXY62lVatWLF68mBEjRnDTTTfhcDiIjY09q8qlC4Gx1vo7Bp/Y2Fibk5Pj7zBERERERERE6sX27duJjIz0dxhyAavuHTPG5FprY0/XV8vNRERERERERERESSIREREREREREVGSSEREREREREREUJJIRERERERERERQkkhERERERERERFCSSEREREREREREUJJIRERERERE5KJRUlLC7Nmzz7r/rFmzKCsrq7YtMTGRnJycsx67JllZWdx4442nnL/nnnvYtm3bOZ/vdBYvXowxhh07dtR4zejRo3n33XcbMKpzQ0kiERERERERkYtEfSaJGtprr71GVFRUvY1fXl5e7XmXy8W1116Ly+Wqt7n9RUkiERERERERkYvEhAkT2LVrFzExMaSlpQEwY8YM4uLicDqdPPnkkwAcOXKE5ORkunfvTnR0NAsXLiQjI4P9+/eTlJREUlJSrfO4XC4cDgfR0dGkp6cDUFFRwejRo4mOjsbhcPD8888DkJGRQVRUFE6nk+HDh9f5XqpWLjVv3pw//OEPdO/enfj4eA4cOADAwYMHGTp0KHFxccTFxfHpp58CsH79evr06UOPHj245ppr+PLLLwGYN28eKSkpXH/99fTv3/+UOUtLS/nkk094/fXXefvtt33nrbWMHTuWrl27csMNN/Ddd9/52p5++mni4uKIjo7mvvvuw1rri/+RRx4hNjaWyMhINmzYwG233cYVV1zBpEmT6vwczqVAv8wqIiIiIiIicpF7+GHIzz+3Y8bEwKxZNbdPmzaNLVu2kO+deMWKFRQUFLB+/XqstaSkpJCdnc3Bgwdp06YNS5cuBcDtdhMaGspzzz1HZmYmLVu2rHGO/fv3k56eTm5uLuHh4QwYMIDFixfTvn179u3bx5YtWwBPVdOJmPbs2UNwcLDv3Jk6cuQI8fHxTJ06lccee4xXX32VSZMm8bvf/Y5HHnmEa6+9ln/+858MHDiQ7du3c9VVV7F27VoCAwNZtWoVjz/+OO+99x4AeXl5bNq0iUsvvfSUeT744AMGDRrElVdeSUREBLm5ufTs2ZP333+fL7/8km3btnHgwAGioqK4++67ARg7diyTJ08G4M477+TDDz/kpptuAqBx48bk5OTwwgsvcPPNN5Obm8ull15Kly5deOSRR4iIiDir53G2VEkkIiIiIiIicpFasWIFK1asoEePHlx99dXs2LGDgoICHA4HK1euJD09nbVr1xIaGlrnMTds2EBiYiKtWrUiMDCQESNGkJ2dTefOndm9ezcPPfQQH330EZdccgkATqeTESNGsGDBAgIDz66WpXHjxr59i3r27ElhYSEAq1atYuzYscTExJCSksIPP/xAaWkpbreb3/zmN0RHR/PII4+wdetW31i//vWvq00QgadC6kS10/Dhw31LzrKzs7njjjto1KgRbdq04frrr/f1yczMpHfv3jgcDlavXn3SXCkpKQA4HA66detG69atCQ4OpnPnzhQVFZ3Vs/g5VEkkIiIiIiIi4ge1Vfw0FGstEydO5P777z+lLS8vj2XLljFp0iT69+/vq4Y5W+Hh4WzcuJHly5czZ84c3nnnHebOncvSpUvJzs5myZIlTJ06lc2bN59xsigoKAhjDACNGjXy7SdUWVnJunXrCAkJOen6sWPHkpSUxPvvv09hYSGJiYm+tmbNmlU7x7/+9S9Wr17N5s2bMcZQUVGBMYYZM2bUGNexY8d44IEHyMnJoX379kyZMoVjx4752oODgwEICAjw/X7iuKY9keqTKolERERERERELhItWrTg8OHDvuOBAwcyd+5cSktLAdi3bx/fffcd+/fvp2nTpqSmppKWlkZeXl61/avTq1cv1qxZQ3FxMRUVFbhcLvr160dxcTGVlZUMHTqUZ555hry8PCorKykqKiIpKYnp06fjdrt9sZwLAwYM4MUXX/Qdn1hm53a7adu2LeDZh6gu3n33Xe68806+/vprCgsLKSoqolOnTqxdu5brrruOhQsXUlFRwTfffENmZiaALyHUsmVLSktLz/svnqmSSEREREREROQiERERQUJCAtHR0QwePJgZM2awfft2+vTpA3g2gF6wYAE7d+4kLS2NgIAAgoKCePnllwG47777GDRoEG3atPElQn6qdevWTJs2jaSkJKy1JCcnc/PNN7Nx40buuusuKisrAXj22WepqKggNTUVt9uNtZZx48YRFhZ2ypgff/wx7dq18x0vWrSoTvebkZHBgw8+iNPppLy8nOuuu445c+bw2GOPMWrUKJ555hmSk5PrNJbL5fJtwn3C0KFDcblczJ49m9WrVxMVFcXll1/ue55hYWHce++9REdHc9lllxEXF1enufzFnNhV+3wQGxtrT+xMLiIiIiIiInKh2b59O5GRkf4OQy5g1b1jxphca23s6fpquVk9+LGyksrzKPkmIiIiIiIiInI6ShKdY389cIAm2dkU/fvf/g5FRERERERERKTOlCQ6x1o3bkwlsPvoUX+HIiIiIiIiIiJSZ0oSnWOdvZ/V213lk3YiIiIiIiIiIuc7JYnOsXbBwQQao0oiEREREREREflFUZLoHAsMCKBDcLAqiURERERERETkF0VJonrwULt23BgR4e8wRERERERERE5SUlLC7Nmzz6rvkCFDKCkpqfP1U6ZMYebMmWc11+k0b978lHNz5szhzTffrJf5alNcXExQUBBz5syp8Zp58+YxduzYBozq7ChJVA9+164dI/7P//F3GCIiIiIiIiInqS1JVF5eXmvfZcuWERYWVh9hnRNjxoxh5MiR9Ta+tZbKyspTzi9atIj4+HhcLle9zd1QlCSqB+WVlRQePcqP1bw8IiIiIiIiIv4yYcIEdu3aRUxMDGlpaWRlZdG3b19SUlKIiooC4JZbbqFnz55069aNV155xde3Y8eOFBcXU1hYSGRkJPfeey/dunVjwIABHD3Nvrz5+fnEx8fjdDq59dZb+f777wHIyMggKioKp9PJ8OHDAVizZg0xMTHExMTQo0cPDh8+XKd7q1q5lJiYSHp6Or169eLKK69k7dq1AFRUVJCWlkZcXBxOp5O//OUvAJSWltK/f3+uvvpqHA4HH3zwAQCFhYV07dqVkSNHEh0dTVFR0Snzulwu/vznP7Nv3z727t3rO//GG29w5ZVX0qtXLz799FPf+SVLltC7d2969OjBDTfcwIEDB3zxjxo1ir59+9KhQwf+9re/8dhjj+FwOBg0aBDHjx+v03P4OZQkqgcfHDpEp88/Z3tZmb9DERERERERkfNYYuKpPycKfcrKqm+fN8/TXlx8atvpTJs2jS5dupCfn8+MGTMAyMvL44UXXuCrr74CYO7cueTm5pKTk0NGRgaHDh06ZZyCggIefPBBtm7dSlhYGO+9916t844cOZLp06ezadMmHA4HTz31lC+eL774gk2bNvmWa82cOZOXXnqJ/Px81q5dS5MmTU5/Y9UoLy9n/fr1zJo1yzff66+/TmhoKBs2bGDDhg28+uqr7Nmzh5CQEN5//33y8vLIzMzk0UcfxVrru9cHHniArVu30qFDh5PmKCoq4ptvvqFXr14MGzaMhQsXAvDNN9/w5JNP8umnn/LJJ5+wbds2X59rr72WdevW8cUXXzB8+HD+9Kc/+dp27drF6tWr+fvf/05qaipJSUls3ryZJk2asHTp0rN6DmdCSaJ60DkkBIBd+sKZiIiIiIiInOd69epFp06dfMcZGRl0796d+Ph4ioqKKCgoOKVPp06diImJAaBnz54UFhbWOL7b7aakpIR+/foBMGrUKLKzswFwOp2MGDGCBQsWEBgYCEBCQgLjx48nIyODkpIS3/kzddttt50S34oVK3jzzTeJiYmhd+/eHDp0iIKCAqy1PP744zidTm644Qb27dvnq/Dp0KED8fHx1c6xcOFChg0bBsDw4cN9S84+//xzEhMTadWqFY0bN+a3v/2sU5IlAAAgAElEQVStr8/evXsZOHAgDoeDGTNmsHXrVl/b4MGDCQoKwuFwUFFRwaBBgwBwOBy1PuNz5eyetNSqszfLuVtJIhEREREREalFVlbNbU2b1t7esmXt7XXVrFmzKvFksWrVKj777DOaNm1KYmIix6r5endwcLDv90aNGp12uVlNli5dSnZ2NkuWLGHq1Kls3ryZCRMmkJyczLJly0hISGD58uVcddVVZzz2iRgbNWrk22/JWsuLL77IwIEDT7p23rx5HDx4kNzcXIKCgujYsaPvvqs+n59yuVx8++23/PWvfwVg//791SbVqnrooYcYP348KSkpZGVlMWXKlFNiDggIICgoCGOM7/h0e0adC6okqgehgYFcGhjI7mr+QxIRERERERHxlxYtWtS6x4/b7SY8PJymTZuyY8cO1q1b97PnDA0NJTw83Lcv0FtvvUW/fv2orKykqKiIpKQkpk+fjtvtprS0lF27duFwOEhPTycuLo4dO3b87BhOGDhwIC+//LJvf5+vvvqKI0eO4Ha7+dWvfkVQUBCZmZl8/fXXpx3rq6++orS0lH379lFYWEhhYSETJ07E5XLRu3dv1qxZw6FDhzh+/DiLFi3y9XO73bRt2xaA+fPnn7N7OxdUSVRPOjdpokoiEREREREROa9ERESQkJBAdHQ0gwcPJjk5+aT2QYMGMWfOHCIjI+natWuNy6zO1Pz58xkzZgxlZWV07tyZN954g4qKClJTU3G73VhrGTduHGFhYTzxxBNkZmYSEBBAt27dGDx48CnjlZWV0a5dO9/x+PHj6xTHPffcQ2FhIVdffTXWWlq1asXixYsZMWIEN910Ew6Hg9jY2DpVLrlcLm699daTzg0dOpTf/va3TJ48mSlTptCnTx/CwsJ8S/PAs0H1b37zG8LDw7n++uvZs2dPnWJvCObERkzng9jYWJuTk+PvMM6Jvx08SEhAAEMiIvwdioiIiIiIiJwntm/fTmRkpL/DkAtYde+YMSbXWht7ur6qJKont7Vq5e8QRERERERERETqTHsS1ZPD5eWsLSnhhwbYWEpERERERERE5OdSkqierD98mOvy88mrZUMwEREREREREZHzhZJE9aRzSAiAvnAmIiIiIiIiIr8IShLVk/bBwTQCfeFMRERERERERH4RlCSqJ4EBAXQICVElkYiIiIiIiIj8IihJVI86N2miSiIRERERERE5b5SUlDB79uyz7j9r1izKysqqbUtMTCQnJ+esx65JVlYWxhhee+0137n8/HyMMcycOROA0aNH8+67757Ur7CwkCZNmhATE0NUVBRjxoyhsrLynMd3IVGSqB79344defGKK/wdhoiIiIiIiAhQv0mi+hQdHc0777zjO3a5XHTv3v20/bp06UJ+fj6bNm1i27ZtLF68uD7D/MVTkqgexYeGEnfJJf4OQ0RERERERASACRMmsGvXLmJiYkhLSwNgxowZxMXF4XQ6efLJJwE4cuQIycnJdO/enejoaBYuXEhGRgb79+8nKSmJpKSkWudxuVw4HA6io6NJT08HoKKigtGjRxMdHY3D4eD5558HICMjg6ioKJxOJ8OHD692vA4dOnDs2DEOHDiAtZaPPvqIwYMH1/m+AwMDueaaa9i5c2ed+1yMAv0dwIXs0PHjrPjXv0gMC6N1cLC/wxEREREREZHzycMPQ37+uR0zJgZmzaqxedq0aWzZsoV877wrVqygoKCA9evXY60lJSWF7OxsDh48SJs2bVi6dCkAbreb0NBQnnvuOTIzM2nZsmWNc+zfv5/09HRyc3MJDw9nwIABLF68mPbt27Nv3z62bNkCeKqaTsS0Z88egoODfeeqc/vtt7No0SJ69OjB1VdfTfAZ/J1dVlbGxx9/zNNPP13nPhcjVRLVo6+PHeO/t29n3Q8/+DsUERERERERkVOsWLGCFStW+BIvO3bsoKCgAIfDwcqVK0lPT2ft2rWEhobWecwNGzaQmJhIq1atCAwMZMSIEWRnZ9O5c2d2797NQw89xEcffcQl3pU3TqeTESNGsGDBAgIDa65lGTZsGIsWLcLlcnHHHXfUKZYTVVMJCQkkJyefUfXRxUiVRPWoc0gIgL5wJiIiIiIiIqeqpeKnoVhrmThxIvfff/8pbXl5eSxbtoxJkybRv39/Jk+e/LPmCg8PZ+PGjSxfvpw5c+bwzjvvMHfuXJYuXUp2djZLlixh6tSpbN68udpk0WWXXUZQUBArV67khRde4B//+Mdp5zyxJ5HUjSqJ6lFYUBDhgYH6wpmIiIiIiIicF1q0aMHhw4d9xwMHDmTu3LmUlpYCsG/fPr777jv2799P06ZNSU1NJS0tjby8vGr7V6dXr16sWbOG4uJiKioqcLlc9OvXj+LiYiorKxk6dCjPPPMMeXl5VFZWUlRURFJSEtOnT8ftdvtiqc7TTz/N9OnTadSo0Tl4GvJT9VpJZIwJA14DogEL3G2t/aw+5zzfdA4JUSWRiIiIiIiInBciIiJISEggOjqawYMHM2PGDLZv306fPn0AaN68OQsWLGDnzp2kpaUREBBAUFAQL7/8MgD33XcfgwYNok2bNmRmZlY7R+vWrZk2bRpJSUlYa0lOTubmm29m48aN3HXXXb7P0D/77LNUVFSQmpqK2+3GWsu4ceMICwurMf5rrrmmxrb777+fhx9+GID27dvjcrnO6hldzIy1tv4GN2Y+sNZa+5oxpjHQ1Fpb4y5UsbGxNicnp97i8YdhW7eysbSUL3v39ncoIiIiIiIi4mfbt28nMjLS32HIBay6d8wYk2utjT1d33qrJDLGhALXAaMBrLU/Aj/W13znq2mdOxNojL/DEBERERERERGpVX3uSdQJOAi8YYz5whjzmjGmWT3Od17q3KQJl3s3sBYREREREREROV/VZ5IoELgaeNla2wM4Akz46UXGmPuMMTnGmJyDBw/WYzj+8d2PP/Knf/6TL8vK/B2KiIiIiIiIiEiN6jNJtBfYa6393Hv8Lp6k0Umsta9Ya2OttbGtWrWqx3D8o7SigvTdu/nU7fZ3KCIiIiIiIiIiNaq3JJG19lugyBjT1XuqP7CtvuY7X7UPDqYRsPvoUX+HIiIiIiIiIiJSo3rbuNrrIeCv3i+b7Qbuquf5zjtBAQFcHhLC7mPH/B2KiIiIiIiIiEiN6nO5GdbafO9SMqe19hZr7ff1Od/5qnNIiCqJRERERERExO9KSkqYPXv2WfUdMmQIJSUldb5+ypQpzJw586zmOh1jDKmpqb7j8vJyWrVqxY033gjAvHnzGDt27Cn9OnbsiMPhwOl0MmDAAL799tt6ie+Xql6TROLRpUkTClVJJCIiIiIiIn5WW5KovLy81r7Lli0jLCysPsI6Y82aNWPLli0c9RZkrFy5krZt29apb2ZmJps2bSI2NpY//vGP9RnmL46SRA3gT126UNSnj7/DEBERERERkYvchAkT2LVrFzExMaSlpZGVlUXfvn1JSUkhKioKgFtuuYWePXvSrVs3XnnlFV/fjh07UlxcTGFhIZGRkdx7771069aNAQMG+JI1NcnPzyc+Ph6n08mtt97K9997FhplZGQQFRWF0+lk+PDhAKxZs4aYmBhiYmLo0aMHhw8frnbMIUOGsHTpUgBcLhd33HHHGT2L6667jp07d55Rnwtdfe9JJEBooB6ziIiIiIiIVCMx8dRzw4bBAw9AWRkMGXJq++jRnp/iYrj99pPbsrJqnW7atGls2bKF/Px87+VZ5OXlsWXLFjp16gTA3LlzufTSSzl69ChxcXEMHTqUiIiIk8YpKCjA5XLx6quvMmzYMN57772Tln/91MiRI3nxxRfp168fkydP5qmnnmLWrFlMmzaNPXv2EBwc7FvKNnPmTF566SUSEhIoLS0lJCSk2jGHDx/O008/zY033simTZu4++67Wbt2ba33X9WHH36Iw+Go8/UXA1USNYCDP/7IQwUF/MPt9ncoIiIiIiIiIifp1auXL0EEnuqe7t27Ex8fT1FREQUFBaf06dSpEzExMQD07NmTwsLCGsd3u92UlJTQr18/AEaNGkV2djYATqeTESNGsGDBAgK9BRYJCQmMHz+ejIwMSkpKfOd/yul0UlhYiMvlYkh1ybQaJCUlERMTww8//MDEiRPr3O9ioBKXBhBoDP+zbx8dQ0K4JjTU3+GIiIiIiIjI+aK2yp+mTWtvb9nytJVDddGsWbMq4WSxatUqPvvsM5o2bUpiYiLHqtljNzg42Pd7o0aNTrvcrCZLly4lOzubJUuWMHXqVDZv3syECRNITk5m2bJlJCQksHz5cq666qpq+6ekpPD73/+erKwsDh06VKc5MzMzadmy5VnFe6FTkqgBhAcFERYYqC+ciYiIiIiIiF+1aNGixj1+wFP1Ex4eTtOmTdmxYwfr1q372XOGhoYSHh7O2rVr6du3L2+99Rb9+vWjsrKSoqIikpKSuPbaa3n77bcpLS3l0KFDOBwOHA4HGzZsYMeOHTUmie6++27CwsJwOBxknYOE2cVOSaIG0jkkhN36wpmIiIiIiIj4UUREBAkJCURHRzN48GCSk5NPah80aBBz5swhMjKSrl27Eh8ff07mnT9/PmPGjKGsrIzOnTvzxhtvUFFRQWpqKm63G2st48aNIywsjCeeeILMzEwCAgLo1q0bgwcPrnHcdu3aMW7cuGrb5s2bx+LFi33H5yLhdaEz1lp/x+ATGxtrc3Jy/B1GvfjN1q1sKi3ly969/R2KiIiIiIiI+Mn27duJjIz0dxhyAavuHTPG5FprY0/XVxtXN5D/atKEcms5n5JyIiIiIiIiIiInKEnUQP7YqRO74uMxxvg7FBERERERERGRUyhJ1ECUHBIRERERERGR85mSRA3k++PHuWXzZj4oLvZ3KCIiIiIiIiIip1CSqIE0b9SIDw8dIqeWTw2KiIiIiIiIiPiLkkQNJCgggMtDQth99Ki/QxEREREREREROYWSRA2oc0gIu48d83cYIiIiIiIicpEqKSlh9uzZZ91/1qxZlJWVVduWmJhITk7OWY9dk6ysLIwxvPbaa75z+fn5GGOYOXMmAKNHj+bdd989qV9hYSFNmjQhJiaGqKgoxowZQ2VlZbVzLF68GGMMO3bsqDGO6ua40ChJ1IA6N2miSiIRERERERHxm/pMEtWn6Oho3nnnHd+xy+Wie/fup+3XpUsX8vPz2bRpE9u2bWPx4sXVXudyubj22mtxuVznLOZfIiWJGlD35s3p3KQJP9aQuRQRERERERGpTxMmTGDXrl3ExMSQlpYGwIwZM4iLi8PpdPLkk08CcOTIEZKTk+nevTvR0dEsXLiQjIwM9u/fT1JSEklJSbXO43K5cDgcREdHk56eDkBFRQWjR48mOjoah8PB888/D0BGRgZRUVE4nU6GDx9e7XgdOnTg2LFjHDhwAGstH330EYMHD67zfQcGBnLNNdewc+fOU9pKS0v55JNPeP3113n77bd95621jB07lq5du3LDDTfw3Xff+dqefvpp4uLiiI6O5r777sNaC3iqqR555BFiY2OJjIxkw4YN3HbbbVxxxRVMmjSpzvH6S6C/A7iYPNi2LQ+2bevvMEREREREROQ8UPBwAaX5ped0zOYxzbli1hU1tk+bNo0tW7aQn58PwIoVKygoKGD9+vVYa0lJSSE7O5uDBw/Spk0bli5dCoDb7SY0NJTnnnuOzMxMWrZsWeMc+/fvJz09ndzcXMLDwxkwYACLFy+mffv27Nu3jy1btgCeqqYTMe3Zs4fg4GDfuercfvvtLFq0iB49enD11VcTHBxc5+dSVlbGxx9/zNNPP31K2wcffMCgQYO48soriYiIIDc3l549e/L+++/z5Zdfsm3bNg4cOEBUVBR33303AGPHjmXy5MkA3HnnnXz44YfcdNNNADRu3JicnBxeeOEFbr75ZnJzc7n00kvp0qULjzzyCBEREXWOu6GpkkhERERERETkIrVixQpWrFjhS7zs2LGDgoICHA4HK1euJD09nbVr1xIaGlrnMTds2EBiYiKtWrUiMDCQESNGkJ2dTefOndm9ezcPPfQQH330EZdccgkATqeTESNGsGDBAgIDa65lGTZsGIsWLcLlcnHHHXfUKZYTVVMJCQkkJydXW33kcrl8FUzDhw/3LTnLzs7mjjvuoFGjRrRp04brr7/e1yczM5PevXvjcDhYvXo1W7du9bWlpKQA4HA46NatG61btyY4OJjOnTtTVFRUp7j9RZVEDaisooK+X3zBva1bM0YVRSIiIiIiIhe12ip+Goq1lokTJ3L//fef0paXl8eyZcuYNGkS/fv391XOnK3w8HA2btzI8uXLmTNnDu+88w5z585l6dKlZGdns2TJEqZOncrmzZurTRZddtllBAUFsXLlSl544QX+8Y9/nHbOE3sS1eRf//oXq1evZvPmzRhjqKiowBjDjBkzauxz7NgxHnjgAXJycmjfvj1TpkzhWJWPVJ2ocAoICDip2ikgIIDy8vLTxuxPqiRqQE0CAth19ChbjhzxdygiIiIiIiJyEWrRogWHDx/2HQ8cOJC5c+dSWupZ9rZv3z6+++479u/fT9OmTUlNTSUtLY28vLxq+1enV69erFmzhuLiYioqKnC5XPTr14/i4mIqKysZOnQozzzzDHl5eVRWVlJUVERSUhLTp0/H7Xb7YqnO008/zfTp02nUqNE5eBrw7rvvcuedd/L1119TWFhIUVERnTp1Yu3atVx33XUsXLiQiooKvvnmGzIzMwF8CaGWLVtSWlp6QX3xTJVEDcgY4/nCWZUMo4iIiIiIiEhDiYiIICEhgejoaAYPHsyMGTPYvn07ffr0AaB58+YsWLCAnTt3kpaWRkBAAEFBQbz88ssA3HfffQwaNIg2bdr4kiY/1bp1a6ZNm0ZSUhLWWpKTk7n55pvZuHEjd911l+8z9M8++ywVFRWkpqbidrux1jJu3DjCwsJqjP+aa66pse3+++/n4YcfBqB9+/Z1+lKZy+Xybax9wtChQ3G5XMyePZvVq1cTFRXF5Zdf7ntGYWFh3HvvvURHR3PZZZcRFxd32nl+KcyJHbjPB7GxsTYnJ8ffYdSr27dsYcuRI+zo3dvfoYiIiIiIiEgD2759O5GRkf4OQy5g1b1jxphca23s6fpquVkD69KkCXuOHaPyPErOiYiIiIiIiIgoSdTA4i+5hJtbtuRIRYW/QxERERERERER8dGeRA3s1latuLVVK3+HISIiIiIiIiJyElUS+YmWm4mIiIiIiIjI+URJogZWYS2t//EPniws9HcoIiIiIiIiIiI+ShI1sEbGEBIQwO6jR/0dioiIiIiIiIiIj5JEftA5JITdx475OwwRERERERG5yJSUlDB79uyz6jtkyBBKSkrqfP2UKVOYOXPmWc11OsYYUlNTfcfl5eW0atWKG2+8EYB58+YxduzYU/p17NgRh8OB0+lkwIABfPvtt9WOX1xcTFBQEHPmzKkxhprm+CVTksgPOjdpokoiERERERERaXC1JYnKy8tr7bts2TLCwsLqI6wz1qxZM7Zs2cJR79/WK1eupG3btnXqm5mZyaZNm4iNjeWPf/xjtdcsWrSI+Ph4XC7XOYv5l0BJIj/oHBLCd8ePU3qa/wBFREREREREzqUJEyawa9cuYmJiSEtLIysri759+5KSkkJUVBQAt9xyCz179qRbt2688sorvr4dO3akuLiYwsJCIiMjuffee+nWrRsDBgzwJWtqkp+fT3x8PE6nk1tvvZXvv/8egIyMDKKionA6nQwfPhyANWvWEBMTQ0xMDD169ODw4cPVjjlkyBCWLl0KgMvl4o477jijZ3Hdddexc+fOattcLhd//vOf2bdvH3v37vWdf+ONN7jyyivp1asXn376qe/8kiVL6N27Nz169OCGG27gwIEDgKeaatSoUfTt25cOHTrwt7/9jcceewyHw8GgQYM4fvz4GcVc35Qk8oO+oaE80q4dP+oLZyIiIiIiIhe1LxK/OOVn3+x9AFSUVVTb/s28bwD4sfjHU9pOZ9q0aXTp0oX8/HxmzJgBQF5eHi+88AJfffUVAHPnziU3N5ecnBwyMjI4dOjQKeMUFBTw4IMPsnXrVsLCwnjvvfdqnXfkyJFMnz6dTZs24XA4eOqpp3zxfPHFF2zatMm3tGvmzJm89NJL5Ofns3btWpo0aVLtmMOHD+ftt9/m2LFjbNq0id69e5/2/qv68MMPcTgcp5wvKirim2++oVevXgwbNoyFCxcC8M033/Dkk0/y6aef8sknn7Bt2zZfn2uvvZZ169bxxRdfMHz4cP70pz/52nbt2sXq1av5+9//TmpqKklJSWzevJkmTZr4klznCyWJ/ODasDCe+6//4tKgIH+HIiIiIiIiIhe5Xr160alTJ99xRkYG3bt3Jz4+nqKiIgoKCk7p06lTJ2JiYgDo2bMnhbV8wdvtdlNSUkK/fv0AGDVqFNnZ2QA4nU5GjBjBggULCAwMBCAhIYHx48eTkZFBSUmJ7/xPOZ1OCgsLcblcDBkypM73m5SURExMDD/88AMTJ048pX3hwoUMGzYM8CSiTiw5+/zzz0lMTKRVq1Y0btyY3/72t74+e/fuZeDAgTgcDmbMmMHWrVt9bYMHDyYoKAiHw0FFRQWDBg0CwOFw1Prc/KH6Jy317mhFBccqKwlXokhEREREROSi1SOrR41tjZo2qrW9ccvGtbbXVbNmzXy/Z2VlsWrVKj777DOaNm1KYmIix6r58FJwcPB/4mzU6LTLzWqydOlSsrOzWbJkCVOnTmXz5s1MmDCB5ORkli1bRkJCAsuXL+eqq66qtn9KSgq///3vycrKqrbiqTqZmZm0bNmyxnaXy8W3337LX//6VwD2799fbaKsqoceeojx48eTkpJCVlYWU6ZM8bWdeFYBAQEEBQVhjPEdn24fqIamSiI/afPZZ0w+zzKGIiIiIiIicmFr0aJFjXv8gKfqJzw8nKZNm7Jjxw7WrVv3s+cMDQ0lPDyctWvXAvDWW2/Rr18/KisrKSoqIikpienTp+N2uyktLWXXrl04HA7S09OJi4tjx44dNY5999138+STT1a7bOxsfPXVV5SWlrJv3z4KCwspLCxk4sSJuFwuevfuzZo1azh06BDHjx9n0aJFvn5ut9u3cfb8+fPPSSz+oCSRn3QMCdEXzkRERERERKRBRUREkJCQQHR0NGlpaae0Dxo0iPLyciIjI5kwYQLx8fHnZN758+eTlpaG0+kkPz+fyZMnU1FRQWpqKg6Hgx49ejBu3DjCwsKYNWsW0dHROJ1OgoKCGDx4cI3jtmvXjnHjxlXbNm/ePNq1a+f7qboBdU1cLhe33nrrSeeGDh2Ky+WidevWTJkyhT59+pCQkEBkZKTvmilTpvCb3/yGnj171lqldL4z9jzaPDk2Ntbm5OT4O4wGMXTLFraVlbG9Vy9/hyIiIiIiIiINZPv27SclF0TOtereMWNMrrU29nR9VUnkJ12aNGHP0aNUnkdJOhERERERERG5eClJ5CedQ0L4t7Xs//e//R2KiIiIiIiIiIiSRP7SLyyM57t0oUmjRv4ORURERERERESEQH8HcLGKbNaMyCqfGRQRERERERER8SdVEvnRrqNH2aUvnImIiIiIiIjIeUBJIj/qn5/Pk3v2+DsMERERERERERElifypc5Mm7D52zN9hiIiIiIiIyEWipKSE2bNnn3X/WbNmUVZWVm1bYmIiOTk5Zz12TbKysggNDSUmJoarrrqK3//+92c9ljGGRx991Hc8c+b/z97dh9lZ1+eiv9e8ZNYzJASC4R0hw4waSMLwEoRGFMSC1S3V0stDC6W4u7e1rUXdLUU9aO3Z0mJrq8XDFrtP2btWDdVTta1QS90HCsX6AogQhTpkEsAgkGASksxM5u05fzDJBtGA5JmsNWt9Pv+YzMy61y8rubwu7uv7/L4fzgc+8IHnfP+vfvWre/yZN77xjTnttNP2+DPz589/3udsFCVRAx1bFBn2uBkAAAD7yGyWRLPpjDPOyN13351vfetb+dKXvpTbb7/9BeX09PTk85//fDZt2vS8X/NcJdGWLVty5513ZuvWrRkeHn5B52oWSqIG6qvX89jERHZMTTX6KAAAALSBd7/73Vm7dm0GBwdz2WWXJUn+5E/+JCtXrsyKFSvy+7//+0mSHTt25PWvf31OOOGELFu2LH/zN3+Tq6++Oo888kjOOuusnHXWWXt8n9WrV2f58uVZtmxZLr/88iTJ1NRULrnkkixbtizLly/PRz7ykSTJ1VdfneOOOy4rVqzIBRdcsMfcoigyODiYDRs27D7nf/yP/zGnnnpqTjzxxPzd3/1dkuQ73/lOTj311AwODmbFihUZGhpKknR1deWtb33r7vd+uo0bN+b888/PypUrs3Llytx+++1Zv359rr322nzkIx/J4OBgbrvttme97vOf/3ze8IY35IILLsj111+/++vr1q3L6aefnuXLl+eKK67Y/fXt27fn7LPPzkknnZTly5fvPvP69evzspe9LJdcckle8pKX5MILL8xXvvKVrFq1KgMDA/nGN76xx8+mCrabNVBfUSRJ1o2OZtkcGDsDAACgOu/88jtz96N3V5o5eOhgPvraj/7E71911VVZs2ZN7r77qfe96aabMjQ0lG984xspyzLnnXdebr311mzcuDGHH354brjhhiTJ1q1bs3DhwvzZn/1Zbr755rzoRS/6ie/xyCOP5PLLL8+dd96ZAw88MOecc06++MUv5qijjsqGDRuyZs2aJE9N4Ow607p169LT07P7az/J5s2bMzQ0lFe+8pVJkiuvvDKvfvWrc91112XLli059dRT85rXvCbXXntt3vGOd+TCCy/M+Ph4pp42nPFbv/VbWbFiRX7v937vGdnveMc78q53vSuveMUr8tBDD+Xcc8/Nfffdl7e97W2ZP7/KcnEAACAASURBVH/+T3zMbfXq1Xn/+9+fQw45JOeff37e+9737s77jd/4jVx88cW55pprdv98vV7PF77whey///7ZtGlTTjvttJx33nlJkgceeCCf+9znct1112XlypX5zGc+k3/913/N3//93+cP//AP88UvfnGPn8/eMknUQGcsXJjPHXdcjuzpafRRAAAAaEM33XRTbrrpppx44ok56aSTcv/992doaCjLly/PP//zP+fyyy/PbbfdloULFz7vzG9+85s588wzs3jx4nR1deXCCy/Mrbfemr6+vgwPD+e3f/u38+Uvfzn7779/kmTFihW58MIL86lPfSpdXT9+luW2227LCSeckCOOOCLnnntuDj300N3nv+qqqzI4OJgzzzwzY2Njeeihh3L66afnD//wD/OhD30oDz74YIqZIY0k2X///XPxxRfn6quvfsZ7fOUrX8nb3/72DA4O5rzzzsuTTz6Z7du37/HP+thjj2VoaCiveMUr8pKXvCTd3d27S7Dbb789v/RLv5Qk+ZVf+ZXdrynLMu9973uzYsWKvOY1r8mGDRvy2GOPJUmWLFmS5cuXp6OjI8cff3zOPvvs1Gq1LF++POvXr3/efwcvlEmiBjq8pye/ePDBjT4GAAAADbCniZ99pSzLvOc978mv//qvP+t7d911V2688cZcccUVOfvss/P+979/r97rwAMPzLe//e380z/9U6699tp89rOfzXXXXZcbbrght956a/7hH/4hV155Ze69995nlUVnnHFGvvSlL2XdunU57bTT8uY3vzmDg4MpyzJ/+7d/m5e+9KXP+PmlS5fm5S9/eW644Ya87nWvyyc+8Ym8+tWv3v39d77znTnppJPylre8ZffXpqen87WvfS31ev15/5k++9nPZvPmzVmyZEmS5Mknn8zq1atz5ZVXJnnqouwf9elPfzobN27MnXfeme7u7hxzzDEZm1lq1fO0IZKOjo7dv+/o6Mjk5OTzPtcLZZKowf51y5b829atjT4GAAAAbWDBggXZtm3b7t+fe+65ue6663ZPzGzYsCGPP/54HnnkkfT29uaiiy7KZZddlrvuuuvHvv7HOfXUU/Mv//Iv2bRpU6amprJ69eq86lWvyqZNmzI9PZ3zzz8/H/zgB3PXXXdleno6Dz/8cM4666x86EMfytatW/c4vbNkyZK8+93vzoc+9KHd5//Yxz6WsiyTJN/61reSJMPDw+nr68ull16an//5n88999zzjJxFixblzW9+c/7yL/9y99fOOeecfOxjH9v9+12P5O3pz7x69ep8+ctfzvr167N+/frceeedu+8lWrVq1e5ff/rTn979mq1bt+bggw9Od3d3br755jz44IN7/Dz3JSVRg/32Aw/kyib6BwEAAEDrOuigg7Jq1aosW7Ysl112Wc4555z88i//8u4Lln/xF38x27Zty7333rv74uc/+IM/2H3x8lvf+ta89rWv3ePF1YcddliuuuqqnHXWWTnhhBNy8skn5+d//uezYcOGnHnmmRkcHMxFF12UP/qjP8rU1FQuuuiiLF++PCeeeGIuvfTSHHDAAXv8M7ztbW/LrbfemvXr1+d973tfJiYmsmLFihx//PF53/vel+SpCZ9ly5ZlcHAwa9asycUXX/ysnN/5nd95xpazq6++OnfccUdWrFiR4447Ltdee22S5A1veEO+8IUvPOvi6vXr1+fBBx/MaaedtvtrS5YsycKFC/P1r389f/7nf55rrrkmy5cv333RdpJceOGFueOOO7J8+fJ88pOfzMte9rI9/nn3pdqutq0ZnHLKKeUdd9zR6GPsU+evWZP7Rkby3VNPbfRRAAAAmGX33Xdfli5d2uhj0MJ+3L+xWq12Z1mWpzzXa00SNVhfUWR4dDTTTVTWAQAAAO1HSdRgffV6dpZlfjA+3uijAAAAAG1MSdRgfTNr+IZHRxt8EgAAAKCddT33jzCbTtt//3ztpJOybL/9Gn0UAAAAoI0piRpsYVdXXr7//o0+BgAAANDmPG7WBL6wcWM+v3Fjo48BAAAAtDElURO4esOG/OnDDzf6GAAAALS4LVu25L/9t//2gl77ute9Llu2bHneP/+BD3wgH/7wh1/Qez2Xzs7ODA4OZtmyZXnDG97wU53r6c4888yccsr/3gx/xx135Mwzz9zja9avX5/PfOYze/yZj370o6nX69m6dese3/uOO+74qc4725RETaCvXs/w2FijjwEAAECL21NJNDk5ucfX3njjjTnggANm41g/taIocvfdd2fNmjVZtGhRrrnmmhec9fjjj+cf//Efn/fPP5+SaPXq1Vm5cmU+//nPv+BzNYKSqAkcWxR5dHw8I1NTjT4KAAAALezd73531q5dm8HBwVx22WW55ZZbcsYZZ+S8887LcccdlyR54xvfmJNPPjnHH398/uIv/mL3a4855phs2rQp69evz9KlS/Of//N/zvHHH59zzjkno8+xsfvuu+/OaaedlhUrVuRNb3pTNm/enCS5+uqrc9xxx2XFihW54IILkiT/8i//ksHBwQwODubEE0/Mtm3b9ph9+umnZ8OGDbt//yd/8idZuXJlVqxYkd///d9PkuzYsSOvf/3rc8IJJ2TZsmX5m7/5m90/f9lll+XKK698Vu7U1FQuu+yy3Vmf+MQndn+Gt912WwYHB/ORj3zkWa9bu3Zttm/fng9+8INZvXr17q+Pjo7mggsuyNKlS/OmN73pGZ/Zb/zGb+SUU07J8ccfv/vMyVOf+Xve854MDg7mlFNOyV133ZVzzz03xx57bK699to9fi4vhIurm0BfvZ4kWTc2luNtOQMAAGgbZ/7PM5/1tTcf/+b85srfzMjESF736dc96/uXDF6SSwYvyaaRTfnFz/7iM753yyW37PH9rrrqqqxZsyZ33333Uz9/yy256667smbNmixZsiRJct1112XRokUZHR3NypUrc/755+eggw56Rs7Q0FBWr16d//7f/3ve/OY352//9m9z0UUX/cT3vfjii/Oxj30sr3rVq/L+978/f/AHf5CPfvSjueqqq7Ju3br09PTsfmTswx/+cK655pqsWrUq27dvT33mv5l/nKmpqfyv//W/8mu/9mtJkptuuilDQ0P5xje+kbIsc9555+XWW2/Nxo0bc/jhh+eGG25Ikmc8Bnb66afnC1/4Qm6++eYsWLBg99f/8i//MgsXLsw3v/nN7Ny5M6tWrco555yTq666Kh/+8IfzpS996cee6frrr88FF1yQM844I//+7/+exx57LIccckg+/vGPp7e3N/fdd1/uueeenHTSSbtfc+WVV2bRokWZmprK2WefnXvuuScrVqxIkrz4xS/O3XffnXe961255JJLcvvtt2dsbCzLli3L2972tp/42bwQJomaQF9RJEnWPkfzCgAAAFU79dRTdxdEyVPTPSeccEJOO+20PPzwwxkaGnrWa5YsWZLBwcEkycknn5z169f/xPytW7dmy5YtedWrXpUk+dVf/dXceuutSZIVK1bkwgsvzKc+9al0dT01x7Jq1ar8l//yX3L11Vdny5Ytu7/+dKOjoxkcHMyhhx6axx57LD/7sz+b5KmS6KabbsqJJ56Yk046Kffff3+GhoayfPny/PM//3Muv/zy3HbbbVm4cOEz8q644op88IMffMbXbrrppnzyk5/M4OBgXv7yl+eJJ574sZ/Fj1q9enUuuOCCdHR05Pzzz8/nPve5JMmtt966u0hbsWLF7hIoST772c/mpJNOyoknnpjvfOc7+e53v7v7e+edd16SZPny5Xn5y1+eBQsWZPHixc8o1qpikqgJDM6fn0dOPz2HzpvX6KMAAACwD+1p8qe3u3eP339R74uec3Lo+djvaU+03HLLLfnKV76Sf/u3f0tvb2/OPPPMjP2YO3R7enp2/7qzs/M5Hzf7SW644Ybceuut+Yd/+IdceeWVuffee/Pud787r3/963PjjTdm1apV+ad/+qe87GUve8brdt1JNDIyknPPPTfXXHNNLr300pRlmfe85z359V//9We911133ZUbb7wxV1xxRc4+++y8//3v3/29V7/61bniiivyta99bffXyrLMxz72sZx77rnPyLnlllt+4p/n3nvvzdDQ0O7Sanx8PEuWLMnb3/72n/iadevW5cMf/nC++c1v5sADD8wll1zyjM9812fd0dHxjM+9o6PjOe+R+mmZJGoC8zo6clhPT2q1WqOPAgAAQAtbsGDBHu/42bp1aw488MD09vbm/vvvf0Zp8kItXLgwBx54YG677bYkyV//9V/nVa96Vaanp/Pwww/nrLPOyoc+9KFs3bo127dvz9q1a7N8+fJcfvnlWblyZe6///6fmN3b25urr746f/qnf5rJycmce+65ue6667J9+/YkyYYNG/L444/nkUceSW9vby666KJcdtllueuuu56VdcUVV+SP//iPd//+3HPPzcc//vFMTEwkSb73ve9lx44de/wMV69enQ984ANZv3591q9fn0ceeSSPPPJIHnzwwbzyla/cfeH1mjVrcs899yRJnnzyyey3335ZuHBhHnvssZ/qEu2qmSRqEtf94AfZNjWVdxx5ZKOPAgAAQIs66KCDsmrVqixbtiw/93M/l9e//vXP+P5rX/vaXHvttVm6dGle+tKX5rTTTqvkff/qr/4qb3vb2zIyMpK+vr78j//xPzI1NZWLLrooW7duTVmWufTSS3PAAQfkfe97X26++eZ0dHTk+OOPz8/93M/tMfvEE0/MihUrsnr16vzKr/xK7rvvvpx++ulJkvnz5+dTn/pUHnjggVx22WXp6OhId3d3Pv7xjz8r53Wve10WL168+/f/6T/9p6xfvz4nnXRSyrLM4sWL88UvfjErVqxIZ2dnTjjhhFxyySV517vetfs1119/fW688cZn5L7pTW/K9ddfn0svvTRvectbsnTp0ixdujQnn3xykuSEE07IiSeemJe97GU56qijsmrVqhf8Oe+tWlmWDXvzH3XKKaeUd9xxR6OP0RC/sGZN7h8ZyXdPPbXRRwEAAGCW3HfffVm6dGmjj0EL+3H/xmq12p1lWZ7yXK/1uFmT6KvXs25sLNNNVNoBAAAA7UNJ1CT6iiJj09N5dHy80UcBAAAA2pCSqEn01etJkuEXeCM8AAAAwN5QEjWJvqJIV62Wx2ZuTQcAAKA1NdPdwLSWvf23ZbtZk+gvioy98pXprNUafRQAAABmSb1ezxNPPJGDDjooNf/9R4XKsswTTzyR+syTSi+EkqhJdPg/BwAAgJZ35JFH5vvf/342btzY6KPQgur1eo488sgX/HolURP56MMP56GdO/Nn/f2NPgoAAACzoLu7O0uWLGn0MeDHcidRE7lnx45c//jjjT4GAAAA0IaURE3k2KLID8bHMzI11eijAAAAAG1GSdRE+mYul1o3NtbgkwAAAADtRknURPqKIkkyPDra4JMAAAAA7UZJ1ET66vUc3dOTndPTjT4KAAAA0GZsN2sii+fNy/rTT2/0MQAAAIA2ZJIIAAAAACVRs7nywQfzhnvvbfQxAAAAgDajJGoymycm8pXNm1OWZaOPAgAAALQRJVGT6SuKjE1P59Hx8UYfBQAAAGgjSqIm01evJ0mGx8YafBIAAACgnSiJmkxfUSRJhkdHG3wSAAAAoJ0oiZrM0fV6XrFwYeZ3djb6KAAAAEAb6Wr0AXimno6O3HbiiY0+BgAAANBmTBIBAAAAoCRqRu9fty4v/frXG30MAAAAoI0oiZpQT0dHvjc6mtGpqUYfBQAAAGgTSqIm1FevJ0nWjY01+CQAAABAu1ASNaG+okiSDI+ONvgkAAAAQLtQEjWhJTOTRMMmiQAAAIB9REnUhBZ3d+fiQw7JsTMTRQAAAACzravRB+DZarVa/mrp0kYfAwAAAGgjJoma2LbJyUYfAQAAAGgTSqIm9d7h4Rzy1a+mLMtGHwUAAABoA0qiJnVET09Gp6fz2Ph4o48CAAAAtAElUZPqs+EMAAAA2IeURE2qb2az2fDoaINPAgAAALQDJVGTOrqnJ7WYJAIAAAD2ja5GH4Afr97Zmf/rmGOyauHCRh8FAAAAaANKoiZ2xTHHNPoIAAAAQJvwuFkT2zE1le/s2NHoYwAAAABtQEnUxD7y8MNZ9s1vZnRqqtFHAQAAAFqckqiJ7dpwtt7l1QAAAMAsUxI1sb56PYkNZwAAAMDsUxI1sV2TRMOjow0+CQAAANDqlERNbHF3d/br6DBJBAAAAMy6rkYfgJ+sVqvlEy99aV46M1EEAAAAMFuURE3uwkMOafQRAAAAgDbgcbMm9+jOnbnxiSdSlmWjjwIAAAC0MCVRk/t/N27M6++9N4+Njzf6KAAAAEALUxI1ud0bzlxeDQAAAMwiJVGT66vXkyTDo6MNPgkAAADQypRETe6YXSWRSSIAAABgFimJmly9szNHzJtnkggAAACYVV2NPgDPbfVxx+WwefMafQwAAACghSmJ5oAzDjig0UcAAAAAWpzHzeaAtaOj+cQjj2RsaqrRRwEAAABalJJoDvjak0/mbd/7Xta5vBoAAACYJUqiOaDPhjMAAABglimJKvZvW7fmiuHhlGVZWWZfUSSJDWcAAADArFESVeyObdty5UMP5fGJicoyD+7uzn4dHR43AwAAAGaNkqhiAzNTP0MjI5Vl1mq19BWFSSIAAABg1nQ1+gCtpn+mJHpgdDSvqHB1/ReXLcuiLn9dAAAAwOzQOlTs6Ho9nXmqJKrSrnuJAAAAAGaDx80q1t3RkSVFkaGKS6Lv7tiRK4aH88MK7zoCAAAA2EVJNAv6i6LySaL1Y2O58qGH8u8V3nUEAAAAsIuSaBbsKonKsqwss69eT5IM23AGAAAAzAIl0SwYKIo8OTWVjRU+GnbMrpLIhjMAAABgFiiJZsHTN5xVpd7ZmSPmzTNJBAAAAMwKJdEsmI2SKHlqw9mGnTsrzQQAAABIkq5GH6AVHVOvpzOpfMPZjcuXZ7/OzkozAQAAABIl0ayY19GRo+v1yieJ5nf56wIAAABmh8fNZsmuDWdVumf79vzqffdlvcurAQAAgIopiWbJQFFkaGQkZVlWlrl9aiqffOyxfHdkpLJMAAAAgERJNGv6iyJbp6byxMREZZl99XqSZNgkEQAAAFAxJdEsmY0NZ4fMm5fejo4Mj41VlgkAAACQKIlmzUBvb5JqN5zVarX0FYVJIgAAAKBySqJZcky9no5UO0mUJMfNlE8AAAAAVbJTfZb0dHTkxfV65SXR3xx/fKV5AAAAAIlJolk1UBSVPm4GAAAAMFuURLOovygqnyS6Z/v2vPruu3PXtm2V5gIAAADtTUk0i/qLIpsnJ/PDiYnKMrtrtdy8ZUvuGxmpLBMAAABASTSLBooiSbUbzo6p15PEhjMAAACgUkqiWdQ/UxJV+chZ0dmZw+fNy/DYWGWZAAAAAEqiWbSkXk8t1ZZESdJXFCaJAAAAgEp1NfoAraze2ZkX9/RkqOL7g35m//3zyPh4pZkAAABAe1MSzbLZ2HD2oWOPrTQPAAAAwONms2w2SiIAAACAqimJZtlAb2+emJzM5omJyjK/s2NHBr7+9dz0wx9WlgkAAAC0NyXRLJuNDWeLurrywOhohkwoAQAAABVREs2y2SiJDp03L0VHhw1nAAAAQGWURLPs2Ho9taTSqZ9arZa+ej3DY2OVZQIAAADtbVa3m9VqtfVJtiWZSjJZluUps/l+zaje2Zkje3oqv7y6ryiy1iQRAAAAUJFZLYlmnFWW5aZ98D5NazY2nL120aLcPzJSaSYAAADQvvZFSdT2Booin99UbU/2m0ccUWkeAAAA0N5m+06iMslNtVrtzlqt9tZZfq+m1V8U2TQxkS0TE5XmTpdlJqenK80EAAAA2tNsl0SvKMvypCQ/l+S3arXaK3/0B2q12ltrtdodtVrtjo0bN87ycRpjYBY2nD0wMpL9brstn23RzwwAAADYt2a1JCrLcsPM/z6e5AtJTv0xP/MXZVmeUpblKYsXL57N4zRM/yyURIf39GRsejrDLq8GAAAAKjBrJVGtVtuvVqst2PXrJOckWTNb79fM+mahJOrt7Mxh8+ZleGysskwAAACgfc3mxdWHJPlCrVbb9T6fKcvyy7P4fk2rt7MzR/b0ZKjiqZ9ji8IkEQAAAFCJWSuJyrIcTnLCbOXPNf1FUekkUZL01eu5ecuWSjMBAACA9jSbk0Q8TX9R5O83bao0800velGOLYqUZZmZiS0AAACAF0RJtI8MFEUen5jIk5OT2b+rmo/9jYsX540tetk3AAAAsG/N6nYz/rfZ2HBWlmU2jo9n6+RkZZkAAABAe1IS7SOzURI9Oj6eg7/61Xz6sccqywQAAADak5JoHzl2piSqcsPZofPmpd7RYcMZAAAAsNeURPvIfp2dOXzevEoniWq1Wvrq9QyPjVWWCQAAALQnJdE+1F8UlZZESdJXFFlrkggAAADYS0qifWigKDI0MlJp5q5JorIsK80FAAAA2ks1u9h5XvqLIo9NTGTb5GQWdFXz0f8fBx+cFfPnZ6os01WrVZIJAAAAtB+TRPvQrg1nVT4e9jMLF+bXDjssXR3+KgEAAIAXTrOwDw309iapdsPZ5PR07tq2LQ+7vBoAAADYC0qifejYej1JKr28erwsc/Kdd+aTjz1WWSYAAADQfpRE+9D8rq4cOm9epSVRb2dnDp03L8M2nAEAAAB7QUm0jw0URaWPmyVPTSgNe9wMAAAA2AtKon2svygqnSRKkr6iMEkEAAAA7BUl0T7WXxT5wfh4dkxNVZbZV6/n4Z07Mz49XVkmAAAA0F66Gn2AdjNQFEmeurz6hPnzK8n85UMOyaqFC1OrJA0AAABoRyaJ9rH+p5VEVXlJb29+dtGidHf46wQAAABeGK3CPjYbJdFUWebvNm3Kt7dvrywTAAAAaC9Kon1sQVdXDunuztDISGWZtSQXfPe7+etHH60sEwAAAGgvSqIGqHrDWUetliX1eobHxirLBAAAANqLkqgBqi6Jkqc2nA1XnAkAAAC0DyVRAwz09mbD+HhGpqYqy+wrigyPjaUsy8oyAQAAgPahJGqAXZdXr61w8qevXs+2qak8MTFRWSYAAADQPpREDTAbG85+6ZBD8t2VK3NAV1dlmQAAAED70Cg0wK6SaKjCkuiQefNyyLx5leUBAAAA7cUkUQMs7OrK4u7uSieJyrLMNRs25Cs//GFlmQAAAED7UBI1SNUbzmq1Wv7r+vW5/vHHK8sEAAAA2oeSqEEGiqLSx82S5NiiyNqxsUozAQAAgPagJGqQ/qLI93fuzOjUVGWZfUWR4YqLJwAAAKA9KIkaZNfl1cMVTv701et5eOfOjE9PV5YJAAAAtAclUYMM7NpwNjJSWWZfUaRM8pBHzgAAAICfUlejD9Cujp0piaq8vPoXFy/OL7zoRVnQ5a8VAAAA+OloExrkwO7uHNTVVWlJtF9nZ2VZAAAAQHvxuFkDDfT2Vr7h7PfXrctfP/popZkAAABA61MSNVB/UVQ6SZQkn9u4MV/ctKnSTAAAAKD1KYkaqL8o8vDOnRmbmqoss69er3RjGgAAANAelEQNNDCzjazKUqevKDI8OpqyLCvLBAAAAFqfkqiB+mdhw1lfvZ4np6byw8nJyjIBAACA1qckaqBZKYmKIgs7O/ODnTsrywQAAABaX1ejD9DOFnV3Z1FXV6Ubzt5w0EHZcsYZleUBAAAA7cEkUYNVveGsVqtVlgUAAAC0DyVRg1VdEiXJbw8N5aoHH6w0EwAAAGhtSqIGGyiKPDQ2lp3T05Vl3rVtW27avLmyPAAAAKD1KYkarL8oMp1kXYXTREvq9QxXPJ0EAAAAtDYlUYMN9PYmSaWXV/cVRR7euTPjFU4nAQAAAK1NSdRg/UWRJJXeS9RXr2c6yUNjY5VlAgAAAK1NSdRgi7q6ckBXV6Ul0Ut7e7N8v/2ybWqqskwAAACgtXU1+gDtrlarZaAoKn3c7PSFC3PPypWV5QEAAACtzyRRE+gvikoniQAAAAB+WkqiJtBfFHlwbKzSi6Yvvu++vO3f/72yPAAAAKC1KYmawEBRZDrJugovmn5iYiLf2LatsjwAAACgtSmJmsCsbDgriqwdHU1ZlpVlAgAAAK1LSdQEZqUkqtfz5NRUNk9OVpYJAAAAtC4lURN4UXd3FnZ2ZmhkpLLMvpniadiF2AAAAMDzoCRqArVarfINZ0t7e/O6RYvSUatVlgkAAAC0rq5GH4Cn9BdF7qjwoumX9PbmhhUrKssDAAAAWptJoiYx0Nub9WNjmZierjTXxdUAAADA86EkahL9RZGpJOvHxirL/IU1a3LuPfdUlgcAAAC0LiVRk5iNDWe9HR0ZcnE1AAAA8DwoiZrEwExJVGWp01cUeWgWHmEDAAAAWo+SqEks7u7Ogs7OSieJ+ur1TCd5aOfOyjIBAACA1qQkahK1Wi39RVFtSTQznTTskTMAAADgOSiJmshAUVT6uNlLe3vz64cdlsXd3ZVlAgAAAK1JSdRE+osi6yu8Q+iQefNy7UtfmsEFCyrJAwAAAFqXkqiJ9BdFJsuy0juEpsoyT0xMVJYHAAAAtCYlURPZveFsZKSyzDeuWZOf/fa3K8sDAAAAWpOSqIn0z5REVV5efUy9nrWjoynLsrJMAAAAoPUoiZrIIfPmZX5nZ7Ubzur1PDk1lc2Tk5VlAgAAAK1HSdREarVa+ivecNY3M500XGEmAAAA0HqURE2mvygqnyRKkuGxscoyAQAAgNajJGoy/UWRdWNjmZyeriSvryjywSVLcvx++1WSBwAAALSmrkYfgGcaKIpMlGUe2rlz96Nie2O/zs78n0cfXcHJAAAAgFZmkqjJzMaGs8fHx7Nm+/bK8gAAAIDWoyRqMrNREr3rgQfyhjVrKssDAAAAWo+SqMkcNm9eejs6Kt9wBTMQGgAAIABJREFU9tDYWCYquucIAAAAaD1KoiZTq9VmZcPZdJKHd+6sLBMAAABoLUqiJlR5STTzCNtwhZkAAABAa1ESNaGBosjw6GimyrKSvL56PUkyPDZWSR4AAADQepRETai/KDJelnm4olLniJ6e/M+XvSyvOfDASvIAAACA1tPV6APwbE/fcHbMzK/3Rketll899NC9zgEAAABal0miJjTQ25sklW44u3/Hjnz5iScqywMAAABai5KoCR02b16Kjo5KL6/+vzdsyC/dd19leQAAAEBrURI1oY5aLcfOwoazLZOT2TwxUVkmAAAA0DqURE1qoCgqfdzMhjMAAABgT5RETaq/KLJ2dDRTZVlJXt/MBdjDFRZPAAAAQOtQEjWp/qLIeFlmw86dleQtMUkEAAAA7EFXow/AjzcwM/kzNDqaF88UPHtjQVdXbhkczNKZzWkAAAAAT2eSqEn1z5REVV5e/aoDDsjB8+ZVlgcAAAC0DiVRkzqipyf1jo5KS6JvPPlkrt2wobI8AAAAoHUoiZpUR62WY+v1DI2MVJb595s25e1DQ5mYnq4sEwAAAGgNSqIm1l8UlU4S9RVFppI8XNFl2AAAAEDrUBI1sf6iyNqxsUyXZSV5fbs2nFVYPAEAAACtQUnUxAZ6ezM2PZ0NFU3+9M1chj08NlZJHgAAANA6lERNrOoNZ0f09GRerZa1JokAAACAH9HV6APwkw3MlERDo6M568AD9zqvs1bLv596ag7v6dnrLAAAAKC1KIma2JE9Pemp1Sq9vPqYmeIJAAAA4Ok8btbEOmq19FW84ezmzZvzuw88UFkeAAAA0BqURE1uoCgyVGFJdNf27fnT738/mycmKssEAAAA5j4lUZPrL4qsHR3NdFlWktdXryex4QwAAAB4JiVRk+svioxOT+cH4+OV5PXN3Ek0bMMZAAAA8DRKoia3e8PZyEgleUtMEgEAAAA/hpKoyfXPlERVXV69f1dXFnd3Z2NFk0kAAABAa+hq9AHYs6Pq9cyr1SrdcPb900/PvA79IAAAAPC/aQqaXGetlr6KN5wpiAAAAIAfpS2YA/qLotJJon984on8wpo1mZyeriwTAAAAmNuURHPArpKoLMtK8n4wPp4vbNqUh3furCQPAAAAmPuURHPAQFFkZHo6P6josuk+G84AAACAH6EkmgOq3nDWN5M3XOEjbAAAAMDcpiSaA6ouiY7o6Ul3rZa1SiIAAABghpJoDnjxTKlT1YazzlotJy9YkI5arZI8AAAAYO7ravQBeG5dHR1ZUq9XuuHs3046qbIsAAAAYO4zSTRH7NpwBgAAADAblERzxEBRZGhkJGVZVpL395s25ZQ77siWiYlK8gAAAIC5TUk0R/QXRXZMT+ex8fFK8ibLMndu357hsbFK8gAAAIC5TUk0R1S94ayvXk+SDHuEDQAAAIiSaM4Y6O1Nkso2nPXNlE4miQAAAIBESTRnHN3Tk65arbJJov27uvKi7m6TRAAAAEASJdGc0dXRkWPq9Uo3nL1u0aIc0dNTWR4AAAAwd3U1+gA8fwNFUdnjZknyV0uXVpYFAAAAzG0mieaQ/qLIA6OjKcuy0UcBAAAAWoySaA7pL4psm5rKxomJSvL+ftOmHHL77VnvXiIAAABoe0qiOWRgZiNZVY+cLejszOMTEzacAQAAAEqiuaR/piSq6vLqvpk8G84AAAAAJdEccky9ns5UVxId2dOTrlota00SAQAAQNtTEs0h3R0dOaZez9DISCV5nbVajqnXTRIBAAAA6Wr0Afjp7NpwVpULDz44C7v8MwAAAIB2px2YY/qLIl978smUZZlarbbXeR9YsqSCUwEAAABzncfN5piB3t5snZrKpomJyjLHpqYyVZaV5QEAAABzj5Jojql6w9mXNm1K72235Z7t2yvJAwAAAOYmJdEcU3VJdERPT8okwzacAQAAQFtTEs0xS+r1dCQZqqgk6pspnWw4AwAAgPamJJpj5nV05Oh6vbJJooVdXTmoq8skEQAAALQ5JdEc1F8UlZVEyVPTRCaJAAAAoL11NfoA/PQGiiKfefzxlGWZWq2213m/cfjh2fsUAAAAYC5TEs1B/UWRLZOT+eHkZA7q7t7rvLccdlgFpwIAAADmMo+bzUFVbzibnJ7O2tHRjExNVZIHAAAAzD1KojloYKYkGhoZqSTv1q1b0//1r+drTz5ZSR4AAAAw9yiJ5qAlRZFaqpsk6qvXk8Tl1QAAANDGlERzUE9HR17c01NZSXRkT0+6arUMj41VkgcAAADMPUqiOWqgtzdDFZVEXR0dObqnxyQRAAAAtDEl0RzVXxSVTRIlSV9RmCQCAACANtbV6APwwgwURX44OZkfTkxkUXf3Xuf9zlFHZXx6uoKTAQAAAHORkmiO6p/ZcPbA6GhOraAkOnfRor3OAAAAAOYuj5vNUU8viaqwfXIyt2zenE3j45XkAQAAAHOLkmiO6qvXU0squ7x6aHQ0Z33727l169ZK8gAAAIC5RUk0R9U7O3NUT09lk0R9M5NJNpwBAABAe1ISzWFVbjhb2NWVRV1dWWvDGQAAALQlJdEcNlAUGRoZqSyvryhMEgEAAECbUhLNYf1FkScmJ7N5YqKSvL56PcMmiQAAAKAtdTX6ALxwuzacrR0dzSnd3Xud954XvzjjZbnXOQAAAMDcoySawwZ6e5M8tZnslP333+u8wQUL9joDAAAAmJs8bjaH9dXrSVLZ5dVbJiby148+6l4iAAAAaENKojms6OzMkT09lZVEmycnc/H99+fmLVsqyQMAAADmDiXRHDdQFBmqqCQ6qqcnXbWaSSIAAABoQ0qiOa6/KCqbJOrq6MjRPT02nAEAAEAbUhLNcf1FkY0TE9k6OVlJXl9RmCQCAACANjTrJVGtVuus1WrfqtVqX5rt92pHA0WRpLrLq/vqdZNEAAAA0Ib2xSTRO5Lctw/epy31V1wSve+YY/Ktk0+uJAsAAACYO2a1JKrVakcmeX2S/2c236edHVtxSXRET0+OrNcryQIAAADmjtmeJPpokt9LMj3L79O2ejs7c8S8eRkaGakkb8vERP7owQdz17ZtleQBAAAAc8OslUS1Wu0/JHm8LMs7n+Pn3lqr1e6o1Wp3bNy4cbaO09Kq3HCWJO9dty43b9lSWR4AAADQ/GZzkmhVkvNqtdr6JNcneXWtVvvUj/5QWZZ/UZblKWVZnrJ48eJZPE7rqrIkOqC7Owd2dWWtDWcAAADQVmatJCrL8j1lWR5ZluUxSS5I8v+VZXnRbL1fOxvo7c1jExN5cnKykry+ej3DSiIAAABoK/tiuxmzbNeGs6qmf/qKIsNjY5VkAQAAAHPDPimJyrK8pSzL/7Av3qsd9Ve84ayvXs+GnTszXZaV5AEAAADNzyRRC9hVEg1VVBK975hjsvUVr0hHrVZJHgAAAND8uhp9APbefp2dOWzevMomifbr7KwkBwAAAJg7TBK1iCo3nG2bnMxvfu97+acf/rCSPAAAAKD5KYlaxEBRVPa4WdHRkb945JHctmVLJXkAAABA81MStYj+osij4+PZPjm511ldHR05ul634QwAAADaiJKoRey6vHptRcVOX1FkuKLJJAAAAKD5KYlaxMCuDWcjI5Xk9ZkkAgAAgLaiJGoRx86URFVdXj1QFCk6OjI2NVVJHgAAANDclEQtYkFXVw7p7q6sJPrdF784D55+euqdnZXkAQAAAM1NSdRCBnp7K9twBgAAALQXJVEL6S+KyiaJdk5P5w333pu/evTRSvIAAACA5qYkaiH9RZFHxsezo4J7hHo6OnL71q35xpNPVnAyAAAAoNkpiVrIrg1nayuaJrLhDAAAANqHkqiF9Fe84ayvKDLsjiMAAABoC0qiFlJ5SVSvZ/3YWKbKspI8AAAAoHkpiVrI/l1dObi7u7INZyvmz8/KBQvy5ORkJXkAAABA8+pq9AGoVpUbzn75kEPyy4ccUkkWAAAA0NxMErWYKksiAAAAoH0oiVrMQFHk+zt3ZmRqaq+zyrLMKXfckT988MEKTgYAAAA0MyVRi9l1eXUVW8lqtVo2T05mzY4de50FAAAANDclUYupfMNZUVRSOAEAAADNTUnUYnaVRFVtOOur1zM8NlZJFgAAANC8lEQt5oDu7ryou7vSSaKNExPZNjlZSR4AAADQnJRELajKDWenLFiQCw4+OCPT05XkAQAAAM2pq9EHoHoDRZFbtmypJOvsAw/M2QceWEkWAAAA0LxMErWg/qLIwzt3ZnRqqrLMSZNEAAAA0NKURC1oYOby6nUVXTjd/7Wv5Z0PPFBJFgAAANCclEQtqOoNZwu7umw4AwAAgBanJGpBu0qiKjecDVeUBQAAADQnJVELOrC7Owd1dWVoZKSSvL56PevGxjJdlpXkAQAAAM1HSdSi+oui0kmi8bLMIzt3VpIHAAAANB8lUYuqsiQ6bf/987tHHZXOWq2SPAAAAKD5dDX6AMyOgd7efObxxzM2NZV6Z+deZZ0wf35OmD+/opMBAAAAzcgkUYvqL4qUSdZVtJVsx9RUNo6PV5IFAAAANB8lUYuqesPZ8m9+M+984IFKsgAAAIDmoyRqUQMzJdFQVZdX1+sZrmgqCQAAAGg+SqIWtai7Owd2dVW64Wy4oiwAAACg+SiJWliVG8766vU8PjGR7ZOTleQBAAAAzUVJ1MIGiqK6x81mHl+r6iJsAAAAoLkoiVpYf1HkobGx7Jye3uusl++/f/68vz+Lu7srOBkAAADQbJRELay/KDKdZH0F0z9H1+u59Mgjc2hPz94fDAAAAGg6SqIWtnvD2chIJXkPjIzkuzt2VJIFAAAANBclUQvrnymJqrq8+s3f/W5+d+3aSrIAAACA5qIkamEHdXdnYWdnpRvOhivKAgAAAJqLkqiF1Wq1DPT2VrrhbN3YWKbLspI8AAAAoHkoiVpcf1FUOkk0XpZ5ZOfOSvIAAACA5qEkanH9RZH1Y2MZn57e66xjZ+44Gq5gWxoAAADQXJRELW6gKDKdZH0Fxc7JCxbkC8cfn+P322/vDwYAAAA0FSVRi6tyw9mi7u68cfHiHNTdvddZAAAAQHNRErW4KkuiJPnq1q35/zZvriQLAAAAaB5djT4As2txd3f27+ysbMPZ+9aty8j0dP7twAMryQMAAACag0miFler1ardcFYUGa4oCwAAAGgeSqI2UGlJVK/n8YmJbJ+crCQPAAAAaA5KojYwUBRZNzqaienpvc7qm7njaF0F29IAAACA5qEkagP9RZGpJA9WUOz01etJkmElEQAAALQUJVEbqHLD2bL99ssdJ5+c17i4GgAAAFqKkqgNDPT2JkklG86Kzs6cvGBB9uvs3OssAAAAoHkoidrAwd3dmd/ZWdnl1f+waVM++eijlWQBAAAAzUFJ1AZqtVqlG87+6tFH80cPPVRJFgAAANAclERtYqAoKnncLHlqw9m60dFMl2UleQAAAEDjKYnaRH9RZN3YWCanp/c6q69ez86yzA/Gxys4GQAAANAMlERtor8oMlmWeWjnzr3O6pvZlra2oskkAAAAoPGURG1iYKbYqeKRs756PUmybmxsr7MAAACA5tDV6AOwb/TPlEQPjI7m3L3MWlIUefRnfiYHd3fv/cEAAACApqAkahOHzpuX/To6Ktlw1lmr5ZB58yo4FQAAANAsPG7WJmq1WvqLIkMjI5Xk/fWjj+a/rl9fSRYAAADQeEqiNtJfFJVMEiXJv2zZkms2bKgkCwAAAGg8JVEb6S+KDI+NZaos9zqrryjy2MREdkxNVXAyAAAAoNGURG1koLc3E2WZhyrYSrZ7w1lFk0kAAABAYymJ2sjTN5ztrWNnsoYrKJwAAACAxlMStZEqS6K+osi8Wi1PTEzsdRYAAADQeF2NPgD7zuHz5qXo6MhQBSXRoq6ujL7ylemo1So4GQAAANBoSqI2UqvVKttwVqvVoh4CAACA1uFxszZTVUmUJNdu2JC33H9/JVkAAABAYymJ2sxAUWTt6GimynKvs9aOjeX6xx/PdAVZAAAAQGMpidpMf1FkvCzz/Z079zqrr17P2PR0Hh0fr+BkAAAAQCMpidrMQMUbzpJkuKLH1wAAAIDGURK1mf6ZYmdoZGSvs/rq9SRPPXYGAAAAzG1KojZzeE9P6h0dlUwSHV2v59h6PaU7iQAAAGDO62r0Adi3Omq19BdFhiooieZ1dOSB006r4FQAAABAo5kkakP9RVHJJBEAAADQOpREbai/KLJ2dLSS1fUfefjhrLrrrgpOBQAAADSSkqgNDRRFdpZlvr9z515njUxP56tPPpmRqakKTgYAAAA0ipKoDe3acFbFI2e7Npyts+EMAAAA5jQlURuqsiQ6diZr2B1HAAAAMKcpidrQkT096anVKtlwtmuSaNgkEQAAAMxpSqI21FGr5diKNpwd1N2d1y5alMXd3RWcDAAAAGiUrkYfgMbor6gkqtVq+ccVKyo4EQAAANBIJona1MBMSTRdlpXklRXlAAAAAI2hJGpT/UWRsenpPLJz515n/fFDD+VFt99eWeEEAAAA7HtKojZV5Yaz+Z2d+eHkZB4dH9/rLAAAAKAxlERtaqC3N0mq3XBWQRYAAADQGEqiNnVkT0/m1WqVTBL1zUwlDY+N7XUWAAAA0BhKojbVWaulr6INZ0fX66nFJBEAAADMZUqiNjZQFJU8btbT0ZG3H3FEVsyfX8GpAAAAgEboavQBaJz+oshXNm9OWZap1Wp7lXX1wEBFpwIAAAAawSRRG+svioxOT+cHFWwlK8syP5yYqOBUAAAAQCMoidrYwMyF01U8cvbHDz+cg26/PSNTU3udBQAAAOx7SqI21j9TElVxefWLe3qSJOttOAMAAIA5SUnUxo7q6Ul3rVZJSdQ3UzitteEMAAAA5iQlURvr6uhIX72eoZGRvc7qq9eTJMMmiQAAAGBOUhK1uf6iqGSS6EXd3Znf2Zlhk0QAAAAwJ3U1+gA0Vn9R5JYtW1KWZWq12gvOqdVq+a/HHJPj9tuvwtMBAAAA+4qSqM0N9PZmx/R0Hh0fz2Ezl0+/UO886qiKTgUAAADsax43a3NVbjjbMTWVu7dty3RZ7nUWAAAAsG8pidpclSXR/3z00Zx45515bHx8r7MAAACAfUtJ1OaO/v/Zu/Pwuus67/+vz9m/2fPtvmUPZWmB0hZUXBmwGcdbx/13Ozi3My4343qhoMCoCC6ggDqO4jbe48yIzvhD5xYdbd1wAFnaUuhCW5omTbqkTZuc7MnJ2b73HwkIUmiS811OkufjunIBOd/zfr//KFwXr+vz/bzjcUWMUasLIVEjG84AAAAAAJi1CInmuUgopPpEwpWTRA2Tp5LYcAYAAAAAwOxDSAQ1WZYrIVFtIiEjThIBAAAAADAbERJBzZal1rExOQVeOB0PhbQyHuckEQAAAAAAs1Ak6AEQvCbL0nAup5OZjJbEYgXV+mpTk5bF4y5NBgAAAAAA/EJIhGdtOCs0JPrLRYvcGAkAAAAAAPiM182g5smQqHV0tOBa3em0ftbTo/F8vuBaAAAAAADAP4REUG0iobDkyuXVv+3r0+v27OFeIgAAAAAAZhlCIigaCqkukXAlJGpIJCRJbYREAAAAAADMKoREkCQ1l5So1Y2QaPLVtfZUquBaAAAAAADAP4REkDRxefXBsTE5jlNQnUXRqEpDIV43AwAAAABgliEkgqSJkGgwl1NPJlNQHWOMGiyLk0QAAAAAAMwykaAHQHF4esPZ2JgWxWIF1fru6tWqjvBHCwAAAACA2WRKJ4mMMR82xlSYCd81xuwwxrza6+Hgn6bJkMiNy6s3VlSoqaSk4DoAAAAAAMA/U33d7G8dxxmU9GpJ1ZLeIelWz6aC7+oSCYXkTkjUmUrpzmPH1Ffgq2sAAAAAAMA/Uw2JzORfXyPp3xzHeeIZv8McEAuFVJdIuLLhbO/IiN7f2qp9o6MuTAYAAAAAAPww1ZDoUWPMrzQREm0xxpRLyns3FoLw1IazQjVMvrrGhjMAAAAAAGaPqYZE75J0naSNjuOMSopK+hvPpkIgmixLraOjchynoDq18biMxIYzAAAAAABmkamGRC+W9KTjOP3GmCslfULSgHdjIQjNlqWBXE69Bd4llAiHtSIe5yQRAAAAAACzyFRDom9IGjXGXCDpo5LaJP2rZ1MhEG5uOGtIJDhJBAAAAADALDLVkCjrTLyD9HpJX3Mc5+uSyr0bC0Fonlxb70ZI9P1zztHP164tuA4AAAAAAPBHZIrPDRljrpf0DkkvM8aENHEvEeaQukRCIcmVDWerEonCBwIAAAAAAL6Z6kmit0kal/S3juOckLRS0m2eTYVAxEMh1SQSrpwkOjg6quvb23WYV84AAAAAAJgVphQSTQZDd0mqNMa8VlLKcRzuJJqDmi3LlZNEPZmMbj18WLtHRlyYCgAAAAAAeG1KIZEx5q2Stkp6i6S3SnrEGPNmLwdDMJosy52LqycvwWbDGQAAAAAAs8NU7yT6e0kbHcc5KUnGmEWSfiPpbq8GQzCaLEt92aySmYzs6MyvnVoUjao0FGLDGQAAAAAAs8RU7yQKPRUQTeqdxncxizRPngAq9JUzY4waLIuTRAAAAAAAzBJTPUm02RizRdIPJ//5bZJ+4c1ICFLTZEh0cGxMl1RUFFSrIZHQiXTajbEAAAAAAIDHphQSOY5zrTHmTZIunfzVtx3H+U/vxkJQ6hMJGcmVe4n+/dxzFQ9x4AwAAAAAgNlgqieJ5DjOjyX92MNZUAQS4bBq4nG1jo66UgsAAAAAAMwOL3jMwxgzZIwZPM3PkDFm0K8h4S+3Npy1jo7qyr17tXt42IWpAAAAAACAl14wJHIcp9xxnIrT/JQ7jlPYhTUoWm6FRHlJd508qZ2ERAAAAAAAFD0ujMFzNJeUqDebVV8mU1Cd2nhcRlJ7KuXOYAAAAAAAwDOERHiOZ244K0QiHNaKeFztLpxKAgAAAAAA3iIkwnO4FRJJUkMiwUkiAAAAAABmAUIiPEdjIiEjqdWFkGhNaalixhQ+FAAAAAAA8FQk6AFQfBLhsFbG466cJPr6WWe5MBEAAAAAAPAaJ4lwWm5tOAMAAAAAALMDIRFOq9myXHndrG1sTK947DHd29fnwlQAAAAAAMArhEQ4rSbLUk8mo/5MpqA6ZeGw7hsY0BMjIy5NBgAAAAAAvEBIhNN6asNZW4GbyRZHoyoJhdhwBgAAAABAkSMkwmk1T4ZEraOjBdUxxqjBstTO/UYAAAAAABQ1QiKcVsNkSOTG5dUNiQQniQAAAAAAKHKRoAdAcSoJh7UiFnMlJLq0slJRY1yYCgAAAAAAeIWQCM+ruaTElQ1nH6upcWEaAAAAAADgJV43w/NqsixXThIBAAAAAIDiR0iE59VkWTqZyWgwmy2ozpFUSg0PP6x/7+52aTIAAAAAAOA2QiI8r2aXLq9eFI3qUCrlyqtrAAAAAADAG4REeF5NLoVEiclLsNlwBgAAAABA8SIkwvNqdCkkkqQGy1I7J4kAAAAAAChahER4XqXhsJbHYq68JtaQSHCSCAAAAACAIhbxqrAxJiHpPknxyT53O45zo1f94A23NpxdXl2tRCgkx3FkjHFhMgAAAAAA4CYvTxKNS7rMcZwLJF0oqcUY8yIP+8EDboVEVy5dqm+uXk1ABAAAAABAkfIsJHImDE/+Y3Tyx/GqH7zRbFk6kU5rKJstuFbecTSez7swFQAAAAAAcJundxIZY8LGmMclnZT0a8dxHjnNM+81xmw3xmw/deqUl+NgBp7acNZW4GminnRa1n336TtdXW6MBQAAAAAAXOZpSOQ4Ts5xnAslrZR0sTFmzWme+bbjOBscx9mwaNEiL8fBDDS5tOFsQTSqsDFcXg0AAAAAQJHyZbuZ4zj9ku6V1OJHP7jnqZCo0A1nxpiJDWcu3G8EAAAAAADc51lIZIxZZIypmvx7S9IVkvZ71Q/eKItEtDQWc+Xy6gbL4iQRAAAAAABFysuTRMsk3WuM2SVpmybuJPq5h/3gEbc2nD11kshxuL8cAAAAAIBiE/GqsOM4uySt86o+/NNsWdqcTBZc57ULFmhhNKqs4yhqjAuTAQAAAAAAt3gWEmHuaLIsHU+nNZLLqTQcnnGdy21bl9u2i5MBAAAAAAC3+HJxNWa3py6vbivwlTPHcXRifFw96bQbYwEAAAAAABcREuGMml3acJbK57XsoYf0ja4uN8YCAAAAAAAuIiTCGTVOhkSFXl5thcNaHoux4QwAAAAAgCJESIQzqohEtDgadWfDmWWp3YU6AAAAAADAXYREmJJmy1Lr6GjBdRoSCU4SAQAAAABQhAiJMCVNluXaSaJj4+NK5XIuTAUAAAAAANwSCXoAzA5NlqV/6e7WaC6nknB4xnVev2CBauNxOS7OBgAAAAAACkdIhClpLimRJLWNjWltWdmM61xYXq4Ly8vdGgsAAAAAALiE180wJU0ubTjLO462DQ7qoAv3GwEAAAAAAPcQEmFK3AqJjKQrdu7UF44ccWEqAAAAAADgFkIiTEllJKJF0ahaCw2JjNGfVVdrSzIpx+FmIgAAAAAAigUhEabMrQ1nm2xbR8bHtY9XzgAAAAAAKBqERJiyZssq+CSRNBESSdLmZLLgWgAAAAAAwB2ERJiyJsvS0fFxjeVyBdWpTSR0TkmJft3X59JkAAAAAACgUJGgB8Ds8dTl1e2plM4rLS2o1t3nnaeaeNyNsQAAAAAAgAs4SYQpa54MiVpduEvo3NJSlUXIKAEAAAAAKBaERJiyxsmQyI3LqyXp9sOH9bWjR12pBQAAAAAACkNIhCmrjka1IBJxLST6bX+/vnbsmCu1AAAAAABAYQiJMC3NJSWubDiTpBbb1pNjY+pwqR4AAAAAAJg5QiJMS5NluXaSqMW2JUlb2HIGAAAAAEDgCIkwLU2WpSPj40rlcgWC7EnBAAAgAElEQVTXOsuyVJdIaHMy6cJkAAAAAACgEIREmJZmy5IjqT2VKriWMUZ/uXChIsYUPhgAAAAAACgIO8gxLU3P2HB2bmlpwfW+3NRUcA0AAAAAAFA4ThJhWp4ZErkpnc+7Wg8AAAAAAEwPIRGmxY5GZUcirm04k6SrnnxSL9mxw7V6AAAAAABg+giJMG1ubjiTpNpEQo8OD+vE+LhrNQEAAAAAwPQQEmHa3A6JWmxbkvSrvj7XagIAAAAAgOkhJMK0NVuWDqdSGnfpHqELysq0JBrV5mTSlXoAAAAAAGD6CIkwbU2WpbykQy6dJgoZo1fbtn6VTCrnOK7UBAAAAAAA0xMJegDMPs/ccHZ2aakrNd+1bJleVFGhrOMobIwrNQEAAAAAwNQREmHamktKJMnVDWevqKrSK6qqXKsHAAAAAACmh9fNMG12JKKqSMTVy6sl6cT4uH7e0+NqTQAAAAAAMDWERJg2Y4zrG84k6c6uLr1+zx71ZTKu1gUAAAAAAGdGSIQZabYsV183k6RNtq28pN/29blaFwAAAAAAnBkhEWakybLUmUopnc+7VvOS8nJVhsPanEy6VhMAAAAAAEwNIRFmpMmylJfUkUq5VjMSCukK29bmZFKO47hWFwAAAAAAnBkhEWak2bIkubvhTJI2VVfrWDqtAy7XBQAAAAAALywS9ACYnZomQyK3L69+y+LF2mTbWpVIuFoXAAAAAAC8MEIizMjCaFQV4bDrIVFlJKLKCH8sAQAAAADwG6+bYUaMMRMbzkZHXa/90MCA3rxnj0ZyOddrAwAAAACA0yMkwow1WZbrJ4kkaTiX0497evT7/n7XawMAAAAAgNMjJMKMNVmWOlIpZfJ5V+u+rLJSViikLcmkq3UBAAAAAMDzIyTCjDWXlCgnqSOVcrVuIhzWq6qqtJmQCAAAAAAA3xASYca82nAmSZtsW61jY2r3oDYAAAAAAHguQiLMmJchUYtta11ZmU6m067XBgAAAAAAz8WucczY4mhU5eGwWj0Iic4qKdGODRtcrwsAAAAAAE6Pk0SYMWOMZxvOnjKezyvr8sXYAAAAAADguQiJUBAvQ6KHBwZkP/CA7h8Y8KQ+AAAAAAD4I0IiFKTZsnQolfLktM95paVKO462sOUMAAAAAADPERKhIE2WpazjqHN83PXa5ZGIXlpZqc2ERAAAAAAAeI6QCAXxcsOZNLHlbOfIiI57EEIBAAAAAIA/IiRCQZonQ6LW0VFP6rfYtiTpV319ntQHAAAAAAATCIlQkCWxmEpDIc9OEp1fWqrP19frkvJyT+oDAAAAAIAJkaAHwOxmjPF0w5kxRtfX1npSGwAAAAAA/BEniVCw5pIStXoUEknSeD6vX/T2qs3DHgAAAAAAzHeERChYk2XpUCqlbD7vSf2hbFav3b1b3+/u9qQ+AAAAAAAgJIILmixLGcfREY82kC2MxbSxvFxbkklP6gMAAAAAAEIiuODpDWcevg7WYtt6ZHBQyUzGsx4AAAAAAMxnhEQoWNNkSOTV5dXSREiUl/Sbvj7PegAAAAAAMJ8REqFgy2IxlYRCnoZEG8vLVR2J6L7+fs96AAAAAAAwn0WCHgCznzFGTZbl6etmkVBIO9avV00i4VkPAAAAAADmM04SwRVNluXpSSJJqrMshYzxtAcAAAAAAPMVIRFc0WxZah8bU85xPOuRzef1dwcO6P8cP+5ZDwAAAAAA5itCIriiybKUdhwdSaU86xEJhfSHgQH9oLvbsx4AAAAAAMxXhERwhR8bziRpk23r/oEBDWeznvYBAAAAAGC+ISSCK5pLSiTJ08urJanFtpV2HP2eLWcAAAAAALiKkAiuWBaLyQqFPD9J9NLKSpWEQtqcTHraBwAAAACA+YaQCK4IGaNGHzacxUMhvX3JElVHo572AQAAAABgvokEPQDmjmbL0v7RUc/7fGf1as97AAAAAAAw33CSCK5psiy1jY0p5zie93IcRwNcXg0AAAAAgGsIieCaJstS2nF0bHzc814vf/xxXblvn+d9AAAAAACYLwiJ4Jpmy5Lk/YYzSbqgtFS/6+vTeD7veS8AAAAAAOYDQiK4pmkyJPL68mpJ2mTbGs3n9YeBAc97AQAAAAAwHxASwTUr4nElQiFfQqJXVVUpaow2J5Oe9wIAAAAAYD4gJIJrQsaoMZFQqw8bzsoiEb2sspKQCAAAAAAAl0SCHgBzS5Nl+XKSSJKuq6nRWD4vx3FkjPGlJwAAAAAAcxUhEVzVZFna0tenvOMo5HFwc4Vte1ofAAAAAID5hNfN4KrmkhKl8nkdGx/3pd+u4WH9+NQpX3oBAAAAADCXERLBVX5uOJOkrxw9qvc8+aSy+bwv/QAAAAAAmKsIieAqv0OiFttWXzarbUNDvvQDAAAAAGCuIiSCq1bF44obo1afQqLLq6sVkrSFLWcAAAAAABSEkAiuChmjBh83nNnRqC6uqNBmQiIAAAAAAApCSATXNfkYEkkTr5ztHhnRcDbrW08AAAAAAOYaQiK4rnkyJMo7ji/9PrxihU5deqnKIhFf+gEAAAAAMBcREsF1TZalsXxex9NpX/pVRaMqCYd96QUAAAAAwFxFSATX+b3hTJLuPnlSr9m1S45Pp5cAAAAAAJhrCInguubJkKh1dNS3nsO5nH6ZTGrXyIhvPQEAAAAAmEsIieC6VYmEosb4epJok21LElvOAAAAAACYIUIiuC5sjBoSCV9DomXxuC4oLSUkAgAAAABghgiJ4InmkhK1+hgSSROnif4wMKChbNbXvgAAAAAAzAWERPBEk2Xp4NiYrxdJv3bBAm2ybfVmMr71BAAAAABgrogEPQDmpibL0mg+rxPptJbF4770fFlVlV5WVeVLLwAAAAAA5hpOEsETT2848/mVM0nqTqd9PcEEAAAAAMBcQEgETzRNhkR+Xl4tST/s7tbSBx8MJJwCAAAAAGA2IySCJ2ricUWM8T0kuqSiQpK0hS1nAAAAAABMCyERPBEJhdSQSPh+oqfBstRsWdpMSAQAAAAAwLQQEsEzT20489sm29a9/f1K5XK+9wYAAAAAYLYiJIJnngqJ/L5EusW2NZbP64GBAV/7AgAAAAAwmxESwTPNlqXhXE7d6bSvfV9ZVaVvn3WWzi8r87UvAAAAAACzGSERPBPUhrPScFjvWb5ci2MxX/sCAAAAADCbERLBM0GFRJKUzGT0na4uHR8f9703AAAAAACzESERPFOXSChijO8bziSpa3xc7z1wQP/V2+t7bwAAAAAAZiNCIngmEgqpLpEI5CTReaWlWhGLaUtfn++9AQAAAACYjQiJ4KmnNpz5zRijTbatXyeTyubzvvcHAAAAAGC2ISSCp5otS61jY3Icx/feLbatgVxOjwwN+d4bAAAAAIDZhpAInmqyLA3lcjqVyfje+/LqaoUl7SAkAgAAAADgjCJBD4C5rfkZG878XklfHY3q5KWXyo5Gfe0LAAAAAMBsxEkieKppMiQKYsOZJAIiAAAAAACmiJAInqpLJBSWArm8WpJ60mm9bvdu/eTUqUD6AwAAAAAwWxASwVPRUEh1iURgIVF1NKoHBwb0056eQPoDAAAAADBbEBLBc02WpdbR0UB6h43Rq21bW5JJ5QPYsAYAAAAAwGxBSATPNVmWDo6NyQkopGmxbXVnMto5PBxIfwAAAAAAZgNCIniuuaREA7mcejKZQPq/urpakrQlmQykPwAAAAAAswEhETz31IazoO4lWhqP66+XLNHSWCyQ/gAAAAAAzAaRoAfA3PfMkOjFlZWBzPAv55wTSF8AAAAAAGYLThLBc/WJhEKSWgM6SfSUsVxOJ9PpQGcAAAAAAKBYERLBc7FQSLWJRGCvm0lS3nFU+/DDurGjI7AZAAAAAAAoZoRE8MVTG86CEjJGl1ZWanMyGdiWNQAAAAAAihkhEXzRbFlqHRsLNKDZVF2tjlRKBwJ+7Q0AAAAAgGJESARfNFmW+rNZJbPZwGbYZNuSpM3JZGAzAAAAAABQrAiJ4ItnbjgLSr1labVlERIBAAAAAHAakaAHwPzQPBkStY6O6pKKisDm+GpzsxZGo4H1BwAAAACgWBESwRf1liWjYE8SSdKrJ185AwAAAAAAz8brZvBFPBRSTTweeEgkSb/s7dXdJ08GPQYAAAAAAEWFk0TwTXNJiVqLICT6ytGjOjI+rjcvXhz0KAAAAAAAFA1OEsE3TZZVFCeJNtm29o2O6nAqFfQoAAAAAAAUDUIi+KbJspTMZpXMZAKdo2XyXqItbDkDAAAAAOBphETwzVMbzoI+TXROSYlWxePaTEgEAAAAAMDTCIngm6YiCYmMMdpk29o7OirHcQKdBQAAAACAYsHF1fBNQyIho+BDIkn6cmOjSsNhGWOCHgUAAAAAgKJASATfJMJhrYrHi2LDWVmEP/oAAAAAADwTr5vBV8Wy4UySbjt8WK/bvTvoMQAAAAAAKAqERPBVMYVE4/m8ftbbq5PpdNCjAAAAAAAQOEIi+KrZstSTyag/kwl6FLXYtiTp1319AU8CAAAAAEDwCIngq2LZcCZJF5WXa2E0qs3JZNCjAAAAAAAQOEIi+KqYQqKQMXp1dbW2JJPKO07Q4wAAAAAAEChCIviq0bJkJD02PBz0KJKkty5erDcuXKiRXC7oUQAAAAAACBQhEXxlhcN6/cKF+kZXV1FcGP36hQv1zdWrVR6JBD0KAAAAAACBIiSC725taNBoLqebOjqCHkWSlHccPTk6GvQYAAAAAAAEipAIvltdUqKrli/Xt7q6tH9kJOhx9NnOTp23dasGstmgRwEAAAAAIDCERAjEjXV1KgmH9fH29qBH0SurqpST9Lu+vqBHAQAAAAAgMIRECMSiWEw31NTont5e/T7gcObFFRUqD4e1OZkMdA4AAAAAAIJESITAfHjlSq2Kx3VNW1ugK+ijoZAur67W5mRSToBzAAAAAAAQJEIiBMYKh/W5+no9Ojysfz95MtBZWmxbh8fHucAaAAAAADBvERIhUH+1ZInWlZXp+vZ2pXK5wOZ4/cKF+q+1a1WbSAQ2AwAAAAAAQSIkQqBCxuj2xkYdHh/XV48dC2yOJbGYXrNggaxwOLAZAAAAAAAIEiERAndZdbVeu2CBPtfZqZ50OrA52sfG9OlDhzQa4IkmAAAAAACCQkiEovDFhgaN5HK6ubMzsBlax8Z0U2en7uvvD2wGAAAAAACCQkiEonBOaanevWyZvtHVpdaALo9+eWWlEqGQtvT1BdIfAAAAAIAgERKhaHy6rk6JUEjXtbcH0t8Kh/WKykptTiYD6Q8AAAAAQJAIiVA0lsbj+viqVfpJT48eCOiVrxbb1v7RUXWMjQXSHwAAAACAoBASoah8ZNUqLY/F9NG2NjmO43v/FtuWFQrpiYBeeQMAAAAAICiERCgqJeGwPldfr61DQ/rRqVO+919dUqLkpZfqLxYs8L03AAAAAABBIiRC0XnH0qU6v7RU17e3azyf97W3MUaJcNjXngAAAAAAFANCIhSdsDG6vbFRh1Ipff3YMd/77xsZ0frt23VfQPciAQAAAAAQBEIiFKUrbFsttq3PdHYqmcn42nt5PK5dIyNsOQMAAAAAzCuehUTGmFXGmHuNMXuNMU8YYz7sVS/MTbc1NGgwm9VnOzt97VsZiejFFRWERAAAAACAecXLk0RZSR91HOdcSS+S9H5jzLke9sMcs6asTH+7bJm+duyY2nxeSd9i23pseFjd6bSvfQEAAAAACIpnIZHjOMcdx9kx+fdDkvZJWuFVP8xNN9fVKWqMbmhv97Vvi21Lkn7FaSIAAAAAwDzhy51Expg6SeskPXKaz95rjNlujNl+KoCV5yhuy+JxXbtqlX506pQeGhjwre+FZWV6x5IlWhGP+9YTAAAAAIAgGcdxvG1gTJmk/5b0OcdxfvJCz27YsMHZvn27p/Ng9hnOZtW8dasaEgk9sG6djDFBjwQAAAAAwKxhjHnUcZwNZ3rO05NExpiopB9LuutMARHwfMoiEX2mrk4PDg7qJz09vvY+mkpxLxEAAAAAYF7wcruZkfRdSfscx/mSV30wP/zNsmVaU1qqj7e1KZ3P+9Izmcmo5uGH9Z2uLl/6AQAAAAAQJC9PEl0q6R2SLjPGPD758xoP+2EOCxujLzY0qC2V0jd9Cm3saFTry8u1mcurAQAAAADzgJfbzR5wHMc4jnO+4zgXTv78wqt+mPtabFuXV1frpo4O9WcyvvTcVF2thwcHfesHAAAAAEBQfNluBrjBGKPbGhrUl83q84cP+9KzxbaVk/Tb/n5f+gEAAAAAEBRCIswqF5aX638tXap/OHpUHWNjnve7pKJCFeEwr5wBAAAAAOY8QiLMOp+pq1PYGN1w6JDnvaKhkH6yZo1uqqvzvBcAAAAAAEEiJMKsszKR0EdWrtQPT57UtsFBz/v9WXW1lsfjnvcBAAAAACBIhESYlT5eU6PF0aiuaWuT4zie9nIcR3ceO6Z7eno87QMAAAAAQJAIiTArlUciuqmuTvcNDOie3l5Pexlj9PVjx/S1Y8c87QMAAAAAQJAIiTBrvXvZMp1dUqKPtbUpk8972qvFtnVff79GczlP+wAAAAAAEBRCIsxakVBItzU06MDYmL59/LinvVpsW+OOo//u7/e0DwAAAAAAQSEkwqz2FwsW6JVVVfp0R4cGslnP+rysslJWKKTNyaRnPQAAAAAACBIhEWY1Y4xub2xUTyajLxw+7FmfRDisy6qqdDyd9qwHAAAAAABBigQ9AFCo9eXlunLJEn356FFdtXy5ahIJT/r83zVrFAmRqwIAAAAA5ib+jxdzwufq6+U4jj5x6JBnPZ4KiBzH8awHAAAAAABBISTCnFCTSOjqVav0b93d2jE05Fmf9z75pN6+b59n9QEAAAAACAohEeaM62pqtDAa1TVtbZ6d9gkbo5/39iqdz3tSHwAAAACAoBASYc6ojER0Y22t7u3v1y882kLWYtsazuX04MCAJ/UBAAAAAAgKIRHmlP+9fLmaLUvXtrUp68Fpn8uqqhQxRps9CqEAAAAAAAgKIRHmlGgopC82NGjf6Ki+e+KE6/XLIxFdWlFBSAQAAAAAmHMIiTDnvH7hQr2sslKfOnRIQ9ms6/Xfu3y53rxokfJsOQMAAAAAzCGERJhzjDG6vbFRJzMZffHIEdfrv33JEn2irk4hY1yvDQAAAABAUAiJMCddXFGh/2/xYt1x5IiOjY+7Xn84m9WOoSHX6wIAAAAAEBRCIsxZn6+vV85x9MlDh1yv/cGDB3XFzp3K8coZAAAAAGCOICTCnFVvWfrQypX63okT2jk87GrtV1dXK5nN6lFOEwEAAAAA5ghCIsxpN9TUqDoS0TVtbXJcPPVzRXW1jMSWMwAAAADAnEFIhDmtOhrVJ2tr9Zu+Pm1xMdBZGItpQ3k5IREAAAAAYM4gJMKc974VK9SYSOja9nZX7xBqsW09MjiovkzGtZoAAAAAAASFkAhzXiwU0q0NDdozMqLvnTjhWt13L1umHRs2qCoSca0mAAAAAABBISTCvPCmRYv04ooKffLQIQ1ns67UrEkkdEFZmYwxrtQDAAAAACBIhESYF4wxuqOxUcfTad1x9KhrdR8eGNBHDh509VJsAAAAAACCQEiEeePFlZV6y6JF+uLhwzo+Pu5KzSdGR/Xlo0f1xMiIK/UAAAAAAAgKIRHmlVsaGpRxHN3Y0eFKvU3V1ZLEljMAAAAAwKxHSIR5pdGy9P4VK/Td48e1Z3i44HorEwmtKS0lJAIAAAAAzHqERJh3PlFbq4pIRB9rb3el3qbqat0/MKCRXM6VegAAAAAABIGQCPPOgmhUn6it1S+TSf3ahRNALbatpbGY2sfGXJgOAAAAAIBgEBJhXvrAihWqSyR0bVubcgVuJrusulodL3qR1paVuTQdAAAAAAD+IyTCvBQPhXRLfb12jozo+93dBdUKGSNjjJwCwyYAAAAAAIJESIR5622LF+vi8nL9fXu7Rgu8T+jXyaSWPfig2njlDAAAAAAwSxESYd4yxuj2xkYdS6f15aNHC6pVl0ioO5PRFracAQAAAABmKUIizGsvq6rSGxYu1K2HD6s7nZ5xnSbLUkMioc2ERAAAAACAWYqQCPPerQ0NSuXz+nRHx4xrGGPUYtv6XV+f0vm8e8MBAAAAAOATQiLMe2eVlOiq5cv1na4u7RsZmXGdFtvWSD6vPwwMuDgdAAAAAAD+ICQCJH2qtlal4bA+3t4+4xqvqqrSB1as0JJYzMXJAAAAAADwByERIGlRLKYbamv1s95e3dvXN6MaZZGI/rG5WeeWlro8HQAAAAAA3iMkAiZ9aMUK1cTjuqatTXnHmVGNnONo6+CgkpmMy9MBAAAAAOAtQiJgkhUO6/MNDdoxPKwfdHfPqMYTIyO6ZMcO3dPT4/J0AAAAAAB4i5AIeIb/uXixLior098fOqSxXG7a319bWqplsZg2J5MeTAcAAAAAgHcIiYBnCBmj2xsbdXh8XF89dmza3zfGaJNt61d9fcrN8JU1AAAAAACCQEgE/IlXVVfrfyxYoM93dupUOj3t77fYtvqyWW0bHPRgOgAAAAAAvEFIBJzGFxoaNJLL6ebOzml/9/LqahmJV84AAAAAALMKIRFwGueUluq9y5frm11dOjA6Oq3vLohG9Yd16/TxmhqPpgMAAAAAwH2ERMDzuLGuTolQSNe1t0/7uy+urJQVDnswFQAAAAAA3iAkAp7HklhM19XU6D97enR/f/+0vjuczeoT7e36Na+cAQAAAABmCUIi4AVcvXKlVsRi+mhbm/LT2FZmhcO6s6tLPzh50sPpAAAAAABwDyER8AJKwmF9rqFB24aG9KNpBD5hY3RFdbW2JJNyphEuAQAAAAAQFEIi4AyuXLJEF5aV6fpDhzSez0/5ey22rePptHaPjHg4HQAAAAAA7iAkAs4gbIxua2hQRyqlrx07NuXvbbJtSdJm7iUCAAAAAMwChETAFFxu2/pz29ZnOzvVm8lM6TvL43FdWlGh4VzO4+kAAAAAACgcIREwRV9saNBgNqvPdnZO+Tv3r1unm+vrPZwKAAAAAAB3EBIBU7SmrEzvWrZMXz92TAdHR6f0HWOMJCk7jbuMAAAAAAAIAiERMA031dUpZoyuP3RoSs87jqOX7NihDx886PFkAAAAAAAUhpAImIZl8biuranR3adO6aGBgTM+b4zRwmiUy6sBAAAAAEWPkAiYpmtWrdKyWEwfbWuT4zhnfL7FttWeSk35FTUAAAAAAIJASARMU2k4rM/U1+uhwUH9+NSpMz6/ybYlidNEAAAAAICiRkgEzMA7ly7VmtJSfby9XekzXErdaFlqsixCIgAAAABAUSMkAmYgbIxub2xUeyqlO48dO+PzN9TU6H8uWeLDZAAAAAAAzAwhETBDm2xbV1RX6+bOTvVlMi/47N8sW6a/IiQCAAAAABQxQiKgALc1Nqo/m9XnDx8+47MdY2NT2ogGAAAAAEAQCImAAlxQVqZ3Ll2qrx49qkNjYy/47LuefFLvPXDAp8kAAAAAAJgeQiKgQJ+pr1fYGN1w6NALPtdi29ozMqKjqZRPkwEAAAAAMHWERECBVsTjumbVKv37yZN6ZHDweZ9rsW1J0pa+Pr9GAwAAAABgygiJABdcu2qVFkejuqatTY7jnPaZNaWlWh6LaUsy6fN0AAAAAACcGSER4ILySEQ319frgYEB/bSn57TPGGPUYtv6TV+fcs8TJAEAAAAAEBRCIsAl71q6VOeUlOhj7e3K5POnfeZTdXXaf/HFChvj83QAAAAAALwwQiLAJZFQSLc1Nqp1bEzf6uo67TO1iYQWx2I+TwYAAAAAwJkREgEueo1t67KqKn26o0MD2expn/nxqVO6+uBBnycDAAAAAOCFERIBLjLG6PbGRiWzWd3S2XnaZ54YGdE/HD2qnnTa5+kAAAAAAHh+hESAy9aVl+vKJUv0laNHdTiVes7nLbYtR9Jv+vr8Hw4AAAAAgOdBSAR44LP19TLG6O8PHXrOZ+vLy7UgEtHmZDKAyQAAAAAAOD1CIsADNYmErl65Ut/v7tajQ0PP+ixsjK6wbW1OJpV3nIAmBAAAAADg2QiJAI9cV1OjRdGormlrk/MnYdBf2LbqLUu9mUxA0wEAAAAA8GyERIBHKiIRfbquTr/v79fPe3uf9dmVS5fqoYsu0qJYLKDpAAAAAAB4NkIiwEPvWbZMZ1mWPtbermw+/5zPx0/zOwAAAAAAgkBIBHgoGgrpi42N2j86qn86fvxZn32nq0sLHnhAg9lsQNMBAAAAAPBHhESAx163YIFeXlmpGzs6nhUINVuWRvJ53dvfH+B0AAAAAABMICQCPGaM0e2NjTqZyeiLhw8//fuXVFaqLBzW5mQywOkAAAAAAJhASAT4YGNFhd6+eLHuOHpUR1MpSVIsFNJlVVXanEw+Z/sZAAAAAAB+IyQCfPK5+nrlHUef7Oh4+ncttq2OVEqtY2PBDQYAAAAAgAiJAN/UWZY+vHKl/uXECT0+NCRJeu2CBbqlvl4V4XDA0wEAAAAA5jtCIsBHN9TUqDoS0TVtbXIcR6sSCV1XW6ul8XjQowEAAAAA5jlCIsBHVdGobqyr02/7+5++sHoom9V/njqlVC4X8HQAAAAAgPmMkAjw2VXLl6vJsnRtW5uy+bzuHxjQG594QvcPDAQ9GgAAAABgHiMkAnwWC4V0a0ODnhgd1fdOnNArqqoUN+bpk0UAAAAAAASBkAgIwBsXLtRLKir0yY4OOY6jl1dVERIBAAAAAAJFSAQEwBijOxobdSKd1u1HjqjFtrV3dFRHUqmgRwMAAAAAzFOEREBAXlRZqbcuWqTbjhzRurIySdK9/f0BTwUAAAAAmK8IiYAA3dLQoIzj6PsnTuiJjRv1jiVLgh4JAAAAADBPERIBAWqwLH1gxQp9r0+J3L4AACAASURBVLtbOceRMSbokQAAAAAA8xQhERCwT9TWqiIS0YcOHtQ79+3TtsHBoEcCAAAAAMxDkaAHAOY7OxrVJ2tr9dG2NoUkrUoktLGiIuixAAAAAADzDCeJgCLw/hUrVJ9IKBEKaXMyGfQ4AAAAAIB5iJAIKALxUEi3NjRoNJ/X9qEhnUyngx4JAAAAADDPEBIBReItixbpvJISSdLPe3sDngYAAAAAMN8QEgFFwhijO5ubJUn/RUgEAAAAAPAZIRFQRF5eXa03Llyon/X26qaODo3n80GPBAAAAACYJwiJgCLz7dWr9caFC/Xpjg6du3Wr7uvvD3okAAAAAMA8QEgEFJkF0ag+W1+vsnBYnamUXvH443r3/v1KZjJBjwYAAAAAmMMIiYAi1FRSoj+sWyc7GlVJKKR/PnFCZ2/dqru6u+U4TtDjAQAAAADmIEIioEidX1amByaDotJwWIuiUV25b59adu1S29hY0OMBAAAAAOYYQiIv5HJBT4A54qySEt1/4YWqSyT09eZm/WNTkx4aHNSabdt0a2enMlxsDQAAAABwCSGR244elTZulO65J+hJMEfUWZYe27BBr6yu1gdWrtRvLrhAf27buv7QIa1/9FE9PDAQ9IgAAAAAgDmAkMhtS5ZI6bT0gQ9Iw8NBT4M5ImyMJOn/P3lSL3vsMb1jyRL93zVr1JfN6iWPPab3HTiggWw24CkBAAAAALMZIZHbolHpW9+SjhyRbrop6Gkwx1xRXa315eV6yxNPaCib1d6NG/WhFSv0ra4unbN1q+4+eZKLrQEAAAAAM0JI5IVLL5Xe/W7py1+Wdu0KehrMIVXRqH51/vl6eVWV/nr/fv3w5El9pblZj1x0kZbGYnrL3r163Z49OpxKBT0qAAAAAGCWISTyyq23StXV0h13BD0J5piySET/tXat/ty29b8PHNDO4WFtqKjQ1osu0h2NjfpdX5/O3bpVXzpyRFkutgYAAAAATJEppldTNmzY4Gzfvj3oMdyza5d09tlSLBb0JJiD0vm8ftHbq79ctOhZv+8YG9P7W1v1i2RSF5WV6durV2t9eXlAUwIAAAAAgmaMedRxnA1neo6TRF46//yJgGhwUEomg54Gc0wsFHo6IHpwYECfaG+X4ziqsyz9fO1a/ejcc9WVTuviRx/V1QcPapiLrQEAAAAAL4CQyGvj49KFF0of/nDQk2AO+2lPjz53+LA+0NqqvOPIGKO3LF6sfRs36r3Ll+srR4/q3G3b9LOenqBHBQAAAAAUKUIir8Xj0pVXSt//vvTb3wY9DeaoWxsa9LFVq3RnV5f+dv/+p+8iqopG9Y2zztIf1q1TRTis1+3Zozfv2aOu8fGAJwYAAAAAFBvuJPLD2Ji0dq0UCk3cU5RIBD0R5iDHcfTZzk59qqNDb1m0SN8/5xzFQn/MgdP5vO44ckQ3d3YqaoxuaWjQVcuXK2xMgFMDAAAAALzGnUTFxLKkb3xDam2VvvCFoKfBHGWM0Sfr6nRHY6Ok5/7LHQuFdH1trXZv2KBLKir0gdZWXbpjh3YND/s/LAAAAACg6HCSyE9vf7s0NCTdc4/E6Q14yJm8l+j4+LjKw2GVRSLP+fyu7m5d3dam/mxWH125Up+qq1NJOBzQxAAAAAAAr0z1JBEhkQfGxyeWmj0nBxobm3jVjIAIPsjm87ro0UdVFg7rF2vXqioafc4zvZmMrm1r0z+fOKH6RELfOOssbbLtAKYFAAAAAHiF180C0t0tbdggffe7p/nQsiYCosOHpd//3u/RMM9EQiHdVFen7UNDetXOnTqVTj/nmQXRqP7P2Wfr3gsuUNQYtezapbfv3avu0zwLAAAAAJjbCIlctnChtHSp9MEPSrt3P89D73yn9La3SX19fo6GeegNixbpnjVrtH90VK94/PHn3Wr2yupq7dq4UTfW1urHp07pnK1b9U9dXcoX0UlDAAAAAIC3CIlcFg5PbLuvqpLe+lbptHcC33GH1NMjXX+97/Nh/mlZsECbzz9fR8bH9b4DB573uXgopE/X12vnhg1aW1qq9xw4oFc+/rj2jYz4OC0AAAAAICiERB5YskS66y7pySel97//NA+sWyd96EPSt74lPfSQ7/Nh/nlFVZXuveACffOss8747Nmlpbr3wgv1T6tXa8/IiC7Yvl03HjqkVC7nw6QAAAAAgKAQEnnkssukT31q4pWzoaHTPHDzzdLKldJVV0mZjO/zYf7ZUFGhpfG4Mvm83vPkk9p12mNuE0LG6F3Llmn/xRfrLYsW6ebOTl2wfbvu5RVJAAAAAJizCIk89MlPSg8+KJWXn+bD8nLpq1+VLrlkYh0a4JPj6bR+2durVz7+uLYNDr7gs4tjMd117rnacv75yjqOLtu5U3+zf796CTYBAAAAYM4hJPJQODyx8X5gQLrmGml09E8eeMMbpG9/WyorC2Q+zE81iYTuX7dOVZGI/mznTt3f33/G77zatrV740ZdV1Oj73d36+ytW/VvJ07I4WJrAAAAAJgzCIl88Oij0pe+NHEN0Wlt2yZdd52vM2F+q7cs3b9unVbE49q0a5d+nUye8Tsl4bBuaWjQjvXr1WRZ+uv9+3XFzp06+Jz0EwAAAAAwGxES+eCyy6QbbpC++92JC62f4/e/l77wBemnP/V7NMxjK+Jx/feFF2p9ebkqI5Epf29tWZkeWLdOX29u1rahIa3dvl2f7+xUOp/3cFoAAAAAgNdMMb0usmHDBmf79u1Bj+GJbHYiLNqxY+Jk0erVz/gwk5HWr5f6+6W9e3n9DL5yHEfGGEnSEyMjOq+0dMrf7Rof14daW/Xjnh6dV1Kib61erUsrK70aFQAAAAAwA8aYRx3H2XCm5zhJ5JNIRPrBDybuKHr/+//kw2hU+uY3pSNHpBtvDGQ+zF9PBUQ/OnlSa7dt0z91dU35u8vjcd29Zo3uWbNGg7mcXvrYY7rqySfVz8XWAAAAADDrEBL5aOXKiTfK/vVfT/PhS14ivec90j/8g7R7t++zAf9jwQK12Lbec+CAvnLkyPS+u3Ch9m7cqKtXrtR3jh/XOdu26UcnT3KxNQAAAADMIoREPrv0Umn5cimXk/bs+ZMPb71VuuWWP3kXDfCHFQ7rP9es0RsXLtTVbW36XGfntL5fFonoS01N2rp+vZbHYnrb3r167e7d6hgb82hiAAAAAICbCIkCct11E4eHDh58xi9tW7r2WikWkziBgQDEQyH9x7nn6solS/SJQ4e0fXBw2jXWl5frkYsu0pcaG/Xf/f06b9s23X74sLJcbA0AAAAARY2QKCAf/ODEPUVve5s0Pv4nH953n3T++VJ3dyCzYX6LhEL6l7PP1m8vuEAbKipmXOPqVau09+KLdVl1ta5tb9fGHTu0bQahEwAAAADAH4REAampkb73vYltZ9dc8ycfLlkiHTggfeQjQYwGKGSMLquuliT9rq9Pf3fggHIzON1Wk0jonjVrdPd556k7ndaLduzQh1tbNZTNuj0yAAAAAKBAhEQBet3rpKuvlr72NeknP3nGB6tXS9dfP7EO7Te/CWw+QJIeGRzUN7u69Fd79yozg1fGjDF606JF2nfxxbpq+XL947FjOnfbNv20p8eDaQEAAAAAM0VIFLBbb5Xe9CZp6dI/+eC666TmZul975NSqUBmAyTp+tpa3dbQoP84dUpveuIJpXK5GdWpjET09bPO0h/WrVNVJKK/3LNHb9izR0f58w0AAAAARYGQKGCxmHT33ROXWEvPuK86kZDuvFNqbZV++MPA5gMk6ZqaGt3Z3Kyf9fbqtbt3a2yGQZEkvbiyUjvWr9ct9fXanEzq3G3b9I9Hj87odTYAAAAAgHsIiYqE40zcTfSs+4kuv1y6/37pne8MaizgaX+3YoW+d/bZWpVIKB4q7D8d0VBI19XWas/GjXpRRYU+dPCgXrJjh3YOD7s0LQAAAABgugiJioQxE1vOvvQl6Z57nvHBS1868eHx4884ZgQE438tXap/PvtshYzR4VRKPel0QfUaLUtbzj9fd51zjg6lUlq/fbs+1tamkQJOKgEAAAAAZoaQqIjcfrt00UUTB4c6O5/xwa5dE/cT3XVXUKMBz5JzHP3F7t165eOP6/j4eEG1jDF6+5Il2n/xxXrn0qW67cgRrdm2Tb/s7XVpWgAAAADAVBASFZF4XPqP/5Cy/4+9+w5vqzz7OP492vLeI85wEidx9iAhkzATQijQAk0oLdCwRwuFFmiBshqgQKGEt1B22SUQoKwwwkpYCdl7byfx3ra2zvvHI1mSZQcncSI7uT/XpUuybp2jR7IsWz/fz3O8cMEF4PEECoMGqdNNN0FlZUzHKASAUdN4vKCAHU4nE1esYFc7LD6dZjbzXGEh84cNw6ppTF29mgvWrqX4EEMoIYQQQgghhBBtIyFRB1NQAM89BytWwNKlgSsNBnj6aRUQ/eUvMR2fEEEnp6Yyb+hQytxuTli+nM2Nje2y34kpKawcNYq78/N5t7yc/osX88zevfhluqUQQgghhBBCHFYSEnVA06bBli0wZkzYlUOHwg03wDPPwPffx2xsQoQbm5zMV8OG0ej3c+OWLe22X6vBwF35+awaNYqh8fFctWkTE5cvZ11DQ7vdhxBCCCGEEEKISBISdVB5eer89dehqChw5T33QPfu8PXXsRqWEFGGJybyzbBhvFhY2O777hcXx1fDhvFCv36sb2xk2JIl/HX7dpyysLUQQgghhBBCtDtN70BTOEaOHKkvWbIk1sPoMEpK1PSzYcPgq6/AZAJqayEpKdZDE6JFLr+fi9ev5/quXRmfnNyu+y5zu7lp61ZeLSmhwG7nhrw8pqan08tub9f7EUIIIYQQQoijjaZpS3VdH/lTt5NOog4sO1stRfTtt3DXXYErgwHR4sWwa1fMxiZES6o8HlbU1zN55Uo+b+dF1jMtFl7p35/PhgzBomn8fssWei9aROGiRdy0ZQufV1bi8vvb9T6FEEIIIYQQ4lginUSdwOWXwwsvwCefwOTJqG6ibt3gxBPhvfdA02I9RCGalLjdTFq5kk2Njbw1cCBnZWQclvvZ3NjI3MpKPq6o4Ovqaly6TrzBwKmpqUxNT+eMtDS622yH5b6FEEIIIYQQojNpayeRhESdQGMjjB6tpp9t2RJoJnrkEfjTn+Ddd+HnP4/1EIWIUOnxMGXVKpbX1/PGgAGcl5l5WO+vwefjq6oq5lZWMreigp0uFwCD4uOZmpbG1PR0xiUlYTZI86QQQgghhBDi2CMh0VFm/XpYswZ++cvAFR4PjBwJlZWwbh0kJsZ0fEI0V+v1ctH69dyTn8+wI/j61HWd9Y2NfBwIjBbU1ODVdZKMRiYFuoympKXRxWo9YmMSQgghhBBCiFiSkOgoVloKWVnAwoUwbhzceKPqLBKiA/uhpoax7byYdVvUer18EdZltNftBmB4QgJnBLqMRicmYpIuIyGEEEIIIcRRShauPkrNmwc9esCXXwJjxsANNwQSIyE6rjmlpYxbvpwHdu484vedZDLxi8xMnu3Xj6KxY1k5ciQP9OxJgtHIg7t2MWH5crK+/55frVvHK8XFlAZCJCGEEEIIIYQ41kgnUSdTXw+jRkF1NaxYoY6AJkRH5/H7+e2GDbxeWspt3bszs2dPtA6w4HqVx8O8qirmVlTwSWUlJR4PGjAyMbFpLaORiYkYOsBYhRBCCCGEEOJgyXSzo9jq1XD88TBhAnz6KRg0Hf73P7XC9a9/HevhCdEin65zzaZNPLtvHzfk5fHPgoIOERQF+XWd5fX1zK2oYG5lJYtqa9GBTLOZKWlpnJGWxulpaaSZzbEeqhBCCCGEEEIcEAmJjnLPPQdXXAEzZ8Ltt+kwaRIsWQIbNkBOTqyHJ0SLdF3npq1bmVVUxA8jRjA6KSnWQ2pVudvNZ2FdRhVeLwZgTFISU9PTmZqWxrCEhA4VdAkhhBBCCCFES2IeEmma9gLwM6BU1/VBbdlGQqK203W46CLo3RvuuQfYtAkGD4Zzz4X//jfWwxOiVbqus6y+nuM60RH5fLrO4trapsWvl9bXA5BrsTAlLY2paWlMSksj2WSK8UiFEEIIIYQQIlpHCIkmAvXAyxISHR66DhFNDHffrRKjTz+FyZNjNSwh2uzjigr+U1zMy4WF2IzGWA+nzYpdLj4NdBl9WllJjc+HSdMYH9ZlNDA+XrqMhBBCCCGEEB1CzEOiwCDygQ8lJDq8vv4aXnoJnn/CiWHYEPD7Yc0asNliPTQh9uupPXu4ZvNmJqWm8u6gQcR3oqAoyOv380Ogy+jjigpWNjQA0NVqbVr8+tSUFBKky0gIIYQQQggRI20NiWL+qUXTtCuBKwG6d+8e49F0TuvXw4svQmGhjVufeQb27gWrNdbDEuInXZ2Xh81g4LKNG5myahUfDh7c6aZsmQwGTkhJ4YSUFB7o1Ysip5NPKiuZW1nJ66WlPLNvHxZNY2JKClMDC2D3i4uTLiMhhBBCCCFEhyOdREcBXYfp0+Gdd2D+fBg/PqwgH0RFJ/BWaSkXrl/P0Ph4vho2jMROFhS1xu33821NDR8H1jJa19gIQE+branL6KSUFOI6YQeVEEIIIYQQovOQ6WbHmJoaGDEC3G5YsQLS5zwNH30E770nQZHoFD4KrO8zq6DgqO2y2eFwqMCospIvq6po9PuxGQycFOgympqeTm+7PdbDFEIIIYQQQhxlJCQ6Bi1dCuPGwZ13wu0ZT8PVV8PLL6vDoAnRiWxpbMRqMNDtKF5Xy+nzsaCmhrkVFcytrGSzwwFAX7u9afHriSkpWA2GGI9UCCGEEEII0dnFPCTSNO2/wElABlAC3KXr+vP720ZCokO3fDkMHQoG/Gre2datsGEDpKXFemhCtIlf1xm+ZAk1Xi+fDx1KQVxcrId0RGxubOTjyko+rqzkq6oqXLpOnMHAqampai2j9HR6HMWhmRBCCCGEEOLwiXlIdDAkJGo/u3ZB48JVFF44AmbMgGefjfWQhGizZXV1TF65EovBwOdDhzIgPj7WQzqiGn0+vqqubuoy2uF0AjAwLo6p6emckZbG+ORkLNJlJIQQQgghhGgDCYmOYboOI0dCRQVsOOtmbE88Aps2QUFBrIcmRJutbWhg0sqVuP1+Phs6lBGJibEeUkzous6GQJfR3IoKFtTU4NF1Eo1GJqWmNoVGXeSIhkIIIYQQQohWSEh0jFu0CCZMgPOn1PP6rSvRJoz/6Y2E6GC2NDZy6sqVDIyPZ+6QIbEeTodQ5/XyRVUVcwNT04pcLgCGJSRwRloaU9PSGJOUhEm6jIQQQgghhBABEhIJHn0U/vhHmDULrr8e1VqUnh7rYQlxQHY7nSQajaSYzei6ftQe+exg6LrOmoYG5ga6jL6rqcEHpJhMHJ+YSKLRSJzRSJzBgD1wHmc0YjcYmi43r8UZDKoeVrNomjzvQgghhBBCdGISEgl0Hc4+Gz79FDbd9h/yH7keVq+G/PxYD02IA+bw+Thv7Vquy8vjTAk7W1Tt8TAv0GW0pqGBRp+PRr8fR+C80efDdxD7NUCLAVJT4NSGWvNwqqWazWDAIGGUEEIIIYQQ7U5CIgGo5qG//Q1mXrWbhFH94aST4IMPQD6IiU6m0uNh8sqVrGxo4PX+/fllVlash9Qpefz+psCoeYDU6PfjaGOt0eeLvG2zmvsgf7cEg6RD6XxqrRYeThnlPVAIIYQQQhxDJCQSUTwPPor5z3+Et9+Gc8+N9XCEOGA1Xi8/W72a72tquKpLF67IzWX4MbqgdUfn0/WIkMmxn3Bqf7W2BFcHw6JpTYGRvVkYZW8WMB1K3S6BlBBCCCGE6AAkJBIRSkth8ilePq8ZSYZeDuvXg3y4Fp1Qg8/H9Zs381pJCSelpPDJ0KGAmo5mNxpjPDpxpOm6jjMsZNpfd1OLX4cFT45mtwuvO/z+g+6OkkBKCCGEEELEWltDItORGIyIvYwM6NLdxC8+f5qvjadgXLgQJk2K9bCEOGDxRiPPFxbyj969Kfd4AChyOum/eDHnZWRweW4u45OTZaHlY4SmadiNRuxGI+lm82G9r/DuqP0FSgdSr/B42B2oh9/2UAKp9gqh7AYDFoMBq6ZhNRiwGgxYwi5bAzWLrCUlhBBCCHHUkJDoGGEwwEsvwbBhoxlj282XY9KQPiLRmaWazaQGQgEd+HVWFq+XlvJSSQn97HYuz83l8txcUg5zcCCOHUZNI8FkIuEI3NfhCKTKPZ4Wb3uwgVQ4k6Y1hUnNg6WocCkQLFl/IoBqvk1L+24ttArWJCwWQgghhDgwMt3sGLNgAZx8MlwwXefVS+ahnXYqyBQdcZRo8Pl4q7SU5/btY2FtLbvGjqWL1UqJ202G2SxTcYRoQWuBlMPvx+X34/b7cek6ruDXYZddgZo77PKB3K6lfbcnSzA0OsDQqi3bWANH5GvrySShlRBCCCFiSNYkEq2aORN2v/wVT28+Bf71L7juulgPSYh2V+R00tVmA2DSypVsbGzk0pwcZuTm0iNwvRCiY9F1HU94uNRKmBQVQrVjUNXSvr3t8LeSAVoMj+wHGDa19WQ3GiO+lpBcCCGEOLZJSCRa5feDx61jPWsy/PijWsS6S5dYD0uIw2ZOaSnP7tvHvKoqACanpvLHbt2YlJYW45EJIToDfzBcaiF0cvr9+z05fL6fvM1P7sPv5+CO4xdi0rT2CZ8O4LZWgwGzpmEOdGtJUCWEEELEjixcLVplMIDVplH34JPYjh8M19+Eec4bsR6WEIfN+VlZnJ+VxQ6Hg/8UF/NCcTGrGxqYlJZGo8/HLqeTwvj4WA9TCNFBGTQNm9FILHsQvS0ER4caPoWfGn0+KgPrVrVUb49/KWqoKYDmQHgUvGwJC5LMzS63eN1+9tHS/tpjHzJdUAghxLFCOomOYQsWwBcn3ss93AWffAKnnx7rIQlxRPh0HY/fj81o5KXiYn67YQMTkpO5PDeX8zMziZd1uoQQoklwGuDBhE+ewPutO7CP8MvuYD3sctN54Lbhl39qH4fboQZNLQVXBlQIaUAdrTHi6zbWDJrW8tcx2PeB3G9b960FttPC9qe1UAtuL2GeEEK0TDqJxE+aOBHm334r8+/7Au9cF6dKRiSOEUZNwxgIgqakpfFgr148t28fv92wges3b+bC7Gz+2bs3NgmLhBACLRCCWAwGkmI9mFbogbWjWg2fDjKsOpTgqsbn+8l96LqOHzWl0Y86Wmfwsjg0UeERBxA0Nau1tI9goNXWIKs99tHSGJtrqQGgpQi1tVi1xdsewj5bvZ923uf+Gh86Quh5KEFuZ913e9+vhMBHjnQSHeN8Pph0ms6iHzUWL4YBA2I9IiFiQ9d1vqmp4bl9+9jqcPDt8OFomsaXVVUMT0gg1WyO9RCFEEIcQ3RdjwiNwi/7m9Wavg6//X4CqObbd8R9+wI1Pey50MP211LN3+x2etj+D8c+9LDH2Jb9t8c+ml/f0kfmFq9r4cN1ax+323ufB3Q/h7DPlq4Lf84O5vV8IK/1g913x/k03vGFB6tHqjvy/MxMbuvR44g/1sNBOolEmxiN8NrrGscN9fL12Y8z4IMzoH//WA9LiCNO0zQmpqQwMSUFv66jaRr1Xi9nrV6NH/UL4vLcXCYmJ8t/MYQQQhx2WtiHGSHE0at5ANieAdRPhVs+IkPMlraPVQB9KPfbfPtDeS6TTcdeZCKdRAKAJR+XMeJX/TAMGwJffQXyB4kQACyrq+O5fft4raSEWp+PPnY7/+7bl1NTU2M9NCGEEEIIIYRok7Z2EhmOxGBExzfyjEwMDz8I8+ez+76XYz0cITqMEYmJPNm3L/vGjeOlwkKyLRayAlPPVtbXM7eiAl8HCtuFEEIIIYQQ4mBJJ5EI8fvZ0f0EEvZsouqHDfQZkx7rEQnRoV25cSPP7ttHV6uVGTk5XJqTQ77dHuthCSGEEEIIIUQE6SQSB85gwPafp0immjVTb8HhiPWAhOjY/tWnD3MGDmRQfDwzd+6k16JFXLR+fayHJYQQQgghhBAHRUIiESFn0mA2XfUoj1TN4MYbYz0aITo2i8HAeZmZfDxkCDvGjOGu/HyGJSQA6qgsd27fzrqGhhiPUgghhBBCCCHaRqabiRbdcgs8/DC88QZMnx7r0QjR+ayoq+P4Zcvw6DrjkpK4PDeXaVlZxBuNsR6aEEIIIYQQ4hgj083EIbnvHi8f9bmB0QsejvVQhOiUhiUmUjR2LP/o3ZtKr5dLN24k9/vvWVNfH+uhCSGEEEIIIUSLJCQSLTLbTUwdtJv8/9wF27fTgRrOhOg0siwW/titG+tGjeLb4cOZkZND//h4AJ7cs4f/Kyqi0uOJ8SiFEEIIIYQQQpGQSLRu1ix0o5H1p/6OG66XlEiIg6VpGuOTk5nVpw9GTQNgbkUF12/ZQpfvv+c369bxdVUVHWn6rxBCCCGEEOLYIyGRaF23bmj33kv/7XPZ8693eOedWA9IiKPHh0OGsOy447g8N5cPKyo4eeVK/rBlS6yHJYQQQgghhDiGycLVYv+8XvwjR7FnQx1DrRtZusJIz56xHpQQRxeHz8fbZWUUxsUxMimJdQ0N/GXbNq7IzWVKWhomg+T5QgghhBBCiIPX1oWrTUdiMKITM5kwvPIylNvx/8LI9Onw7bdgscR6YEIcPexGI7/JyWn6ervTyaLaWt6vqKCLxcKMnBwuzc2ll90ew1EKIYQQQgghjnby72nx0wYPptvJBTz/nM7OVTUsXRrrAQlxdDszPZ3dY8fy7sCBDE9I4IFduxiyeDGNPl+shyaEEEIIIYQ4ikknkWiz8z67irMKl2M5fiFgjPVwhDiqmQ0Gfp6Zyc8zMylyOllaX0+cUf3cTV65kgFxcVyem8ughIQYj1QIIYQQQghxtJBOItF2p5yCZeUS+Pe/ef112LUr1gMS4tjQ1WbjnIwMQK1flGoy8eTevQxel41kTwAAIABJREFUsoSxy5bx/L591Hu9MR6lEEIIIYQQorOThatF2+k6TJmC//sfKNQ3kDGkC/Png9kc64EJcewpc7t5taSE5/btY11jIy/068eM3Fy8fj9GTUPTtFgPUQghhBBCCNFBtHXhaukkEm2nafDEExg8br4Y/Ad++AHuuCPWgxLi2JRpsXBjt26sGTWK74cPZ1pWFgBP7N3LkCVLeGDnTj6vrKTC44nxSIUQQgghhBCdhYRE4sAUFMAdd9Bt69fcfHEJDz0Ec+fGelBCHLs0TWNscjLxgfWKulut2A0Gbtu+nUmrVpHx3Xf0//FHgl2ja+rrKXI66UhdpEIIIYQQQoiOQaabiQPnckFDA864NMaMgaIi2LIFUlJiPTAhRFC5282K+nqW1ddT6/Uys1cvAMYuW8bC2loyzWaGJyQwPCGBiSkpTE1Pj/GIhRBCCCGEEIdLW6ebydHNxIGzWsFqxeb18v6ti1hkGi8BkRAdTIbFwmlpaZyWlhZx/WMFBSyurWV5IEB6tKiIDY2NTSHRtLVr6WKxMDwxkREJCRTGxWE2SNOpEEIIIYQQxwIJicTBu/deuv/973RfsQIYQHEx5OTEelBCiP0ZnZTE6KSkpq9dfj81gSOjufx+ilwuPqqooHHPHgCsmsZd+fn8pUcPfLrOkro6hsTHYw9MbxNCCCGEEEIcPWS6mTh4ZWVQWAgDB/LFnfP52Vka770HkyfHemBCiEPh03U2NTaqbqO6Ok5JTWVqejrrGhoYuHgxRqAwLq6p2+jnGRn0tNtjPWwhhBBCCCFEK2S6mTj8MjPhoYfg8suZsOVFeveewW9+AytXQm5urAcnhDhYRk2jf3w8/ePjuTA7u+n6rlYr7wwcyPL6epbX1/NlVRWvlpRQYLfT025nYU0NjxQVNa11NCIxkWyLJYaPRAghhBBCCHEgpJOonTmdYDSC2RzrkRwhfj+ceCKsX8/G9zYwYnIGo0fDvHnqeRBCHN1K3W4SjEbijEbeLy/npi1b2Op0NtVzLRbmDxtGn7g4djudeHWdfJsNTdNiOGohhBBCCCGOLdJJFCPPPAM33ghdu0KPHuqUnx953r27Wvv5qGAwwFNPwW9+Q7/UUp54IoMZM+Bvf4O774714IQQh1tWWKfQ2RkZnJ2RQY3Xy4r6epbX1bGsvp5ugTe8WUVFPFJURIrJFNFtdEFWFkYJjYQQQgghhIg56SRqZ4sWwbMfLaFmd1dKt+WwY4c6RLzfH3m73NyWA6TgeVzckR/7IdF1CHzImzEDunWDe++N8ZiEEB3K+oYGFtTUNIVHq+rrSTKZKBk3Dk3TuHfHDordbkYkJDA8MZFB8fFY5chqQgghhBBCHLK2dhJJSNTOdF1n0L8Hsb1qO9eMvIabx99MujWHPXtg50512rEj8nzXLvB4IveTkdF6gNSjByQnH/GH9tNqa2HWLPRbbkWzyjokQoj98waOppYfWPR6xoYNvFNWRq3PB4BJ0zg3I4PZAwcCsLq+nnybjUSTNMEKIYQQQghxICQkiqEtlVuYuWAmr656FYvR0hQW5SS0fHx4nw+Ki0OhUUtBUtgSHwCkpOy/Eyktramx58j5+GOYOhXuuw9uu42vv4YXXoAXX1Sz0oQQ4qf4dZ3tTifL6upYXl9PhtnMTd26oes6ad99R43XSx+7XU1XS0zktNRUjktMjPWwhRBCCCGE6NAkJOoAwsOil3/xMhcOvvCg9qPrUFoaGRo1D5Lq6yO3iY9vPUDKz4esrMMUIp1/Pnz0EaxZw7Nf9ubKK2HmTLj99sNwX0KIY4ZP1/m4oqLpyGrL6urY6XLx5+7deaBXLxp8Pi5ct64pPBqRkEBXq1UWyBZCCCGEEAIJiWKmxlmD1+8lPS696bqtlVvJT8nHaDDy2MLH2F2ze7+dRQdK16GqKrr7KPxyVVXkNjabWkC7tSApN/cgj05WVAT9+8OECegfzeXXv9GYPRu++gomTjzEByqEEGEqPR68uk6WxcKWxkbOXrOGDY2NBH+rpZtMPNOvH+dmZlLn9bLP7abAbscgwZEQQgghhDjGSEgUI7MWzuJP8/7Eab1OY9qAafy88Oek2lOb6jd9ehOzFs3CarT+5DS09lRb2/I0tuDlsrLI25vNavHp1jqR8vLUbVo0axb84Q8wezZ1Z0xjxAhobIQVKyAz8zA+SCHEMa/B52NVWLfRdXl5DE9M5J2yMs5bu5YEo5FhCQkMS0gg3WTimrw8si0W1jc0sLSuDrvRiN1gaDoNT0zEajBQ7/Xi0nXsBgM2g0GCJiGEEEII0alISBQj68rW8fLKl5m9djY7qndgNpiZ3Hsy0wdO5+x+Z5NsS2ZzxWbu++Y+Xln1ClajlVlTZnHFcVfEdNwNDWoB7dY6kfbujby9waCCohY7kfK89PzXTZhuvB4KCli+HMaMgb/+Fe6444g/NCGEYI/LxaeVlU3h0eqGBup8PtaNGkX/+Hj+uXs3N23dGrXdrjFj6GazMXPHDv66Y0fT9bZAiLRt9GhSzGYe272b10pLiQsGTIGw6eXCQkwGA++Vl7Osri6ilmA0cmF2NqCO/Fbt9UbU441G0ltN44UQQgghhGg7CYliTNd1luxdwuy1s3lz7Zvsrt2NxWjhjIIzmDZwGmf1PYvi+mJmfjOTy4dfzgk9TqCsoQyf7jsinUUHyuWC3btb70QqKgK/P3Kb3NxQcGSzwfHHh8Iku11Nkwue/P7Ir4/16+x26NsXCgtVR5cs/C1E+/PrOhqgaRo1Xi8lbjcOvx+Hz6fO/X5OSUnBZjSyuLaWH2pro+oP9+6N1WDg+X37mFNWFlFz+v1sPP54NE3jmk2beKpZ2p5oNFJ7wgkAXLB2LbObtXTmWizsHTcOgHPXrOHLqqqITqd+cXG8M2gQAHds28YWh4O4sHpvu51r8vIAeLusjHqfr6kWZzSSbTYzKCEBgL0uFyZNa6qb5E1HCCGEEOKoIiFRrDgcKlGx2cBqBU3Dr/tZVLSIN9e+yVvr3mJP3R5sJhtT+0xl+sDpnNnnTOIt8fxu7u94YfkLXDPyGm4ZfwvZCdmxfjRt5vHAnj2RAVLl+hLO//xq/mG9nbmlI/F4Yj3Kzsluh379VGAUfurbV9WEEJ2Druu4/H4aAyGS2++nZ+CHeE19PbtdrqaAyeHzYTYYuCRH/dPgub17Wd3QEFHPtlh4om9fAC5av55FzUKs4QkJfDtiBACDfvyRtY2NEeOZlJrKZ0OHAtBz4UJ2hB1G06RpTMvM5LUBAwAYs3QpDr8fW2C6nc1gYEpaGjd26wbA7zZtwqhpTTW7wcCopCROTU3Fr+u8VVYWsa3NYKCb1UoXqxW/rlPm8TRtZ9Y0WXBcCCGEEKKdSUgUK48+Cn/8Y+hri0WFRYHQyG+z8n2en9n59czpUk2x1UOcz8jP6nI4wZnDwoQq/hu/HStGrvUO52bTRLKt6aHQqS3nza8LhFVHXE2NWsS6Sxd83y9iX6mRyZOhpARuuglyctS6RgaDGl746Vi/rq4ONm2CDRsiTzt2qE4jULfr0SM6PCosPIxHrxNCdEqlbjf1wS6nwHmC0cjwxEQA3igpocLrbao1+v0MiIvjokBINWPDBqq9XpyB7Z1+P1PT07kzPx9d1+nyww9N17sCb1J/6NqVfxYU0OjzEf/NN1FjuqNHD/7WsyelbjfZ33/fdL2Gms53f8+e/KFbN3Y5nUxZtappPajg6Xd5eZyRns5up5MHd+2KCqF+lp5OYXw8ZW4386urVQhlNDbVC+x2kk0mXH4/DT4fNoMBq8GAUd48hRBCCHEUkpAoVpYtg/nzwelUHUX7Ofe5nHxjL+XNjBLm5FRSZvOR4NY4aY+JBqOf+V19/HYFPP9+O4zLYjmwYOlgwqiWzj/5BK66Ch55BG68kaXLNMaNA7dbHT0tPx+eeQZOOUWte7RqFfTpo8IPk6kdHvdRxuGAzZujw6ONG9Xi4EEpKS2HR7167WfBcSGEaAd+XccdmH9sMxrx6TqbGhtxBqbgOQPdUL1sNgrj42nw+XipuDii7vT7mZqWxkmpqexxufjDli1R9b907865mZksr6tj8qpVTQFXcObzGwMGMD0riy+rqjh15cqocX4waBA/y8jg/fJyzlmzpul6c6Ajau7gwUxISeHD8nJu2749okvKZjDwaEEBve12FlRX81ZZGdbAdtbA6YrcXFLNZtY2NLCqvj6iZjMYGJWYiMVgoNLjoS4YUmla021kcXQhhBBCtCcJiToZr9/L/B3zmb12Nm+vf5tKRyXx5nhO7zWZGYN+Q6Ypmbc2zOHm/peTrSW2KYSKOj+Ubdzu9nmgVis+sxW3wYZTs9Pot5PW1Y49xca+ajtL19txYMel2TEn2bCn2pl4up2UXDt1Hhtuo53ULnYM8XY118pmU+f2Vr6224+JtMnvV+tCNQ+PNmyAfftCtzOZoKAgOjzq108FS0II0dl5AyGSxWDAEjgy3XanMypkOj4piWyLhW0OBx9WVESFWL/Py6OX3c7XVVXM2rMnavvZAwbQNy6O5/bu5ZZt23D5/bj8fnyBcWwbPZqedjsP7NzJbdu3R42zdNw4Mi0Wbtu2jQd27YqqO044AZvRyB3btvFicXFTeGQ1GIg3GJqmEv5fURHf1NQ0hU9WTSPFZGJmr14AvFdeznaHIyKkSjGZmJqeDsC6hgYafL7Q/jWNeKORTIsFUNMkZfqfEEII0flJSBRDh/oHlcfn4cvtX/Lm2jd5Z8M7VDursZvsOL1OLEYL1466llvH33pk1yzy+1VQdDBhVFERPPwwDB4Mkyerdpjg7RyOppO33omjyoG3zoHe6EBzOjB6nCSaHGiHElIZjT8dJP1U2HSg25jNHWa+V02N6jTauDEyPNq8mYh1orKzW+4+6t5dFs4WQoi28gXWnrIFuoGqPB5K3G5cgeudgTDppJQUzAYDy+rqWFFf33S9KzBl7689emDQNP5bUsLnVVVN17v8fnTgg8GDAbht2zbeLS+P2HeSycT2MWMAOGf1at6vqIgYY0+bjW2B+mkrVvBFdXVEfXB8PKtGjQJg9NKlLKuvj+iUGpeczFsDBwJw4bp17HW5IkKskYmJ3Nq9OwB/27EjIoSyaBoD4uObQqq3y8rQoCmgshgM5Fmt9A6s17XN4cAS6LCyhN1GOq2EEEKIAyMhUQxtvHIj3hovuZfmknpaKprx4P+QcfvczNs6jzfXvcnb696mwdMAgFEzcuHgC3nhnBcwGTpBp8zHH8Po0ZCWdnDb+3xsWOHkxwVOdm9yULTZQfF2B9XFTuZ94MDkdvDUYw4Wfumge5aTbhkOuqQ6yE11MKJ/ZBjVFFLt7+vgAuQHy2A49LApOKWv+dS+n7rOaGzTEL1e2L49uvNo/XqoqgrdLvxIa80Xzo6LO/inSAghxOEXnIYXHkAB9Au8gS+urY0KsZKMRs7PygLg6b172RnoxAruo7fdzl969ADUelXbHI6IEGt8cjLP9usHQMHChex2uXCH/b15QVYW/w0sip6wYAENzQ6PekVuLs/064eu6xjmz496TMH1rhp8Prr/8ENTeBQMkq7r0oWr8/Ko9Hi4cN26iADLomlMz8picloa5W43j+/ZExFQWQ0GTkxOpjA+nmqPh29rapquD96ml81GitmMy++n2uuN2FbWtBJCCNFRSUgUQ9tu28beZ/birfBi7Wol57c55MzIwd7r0A5F5fQ6+WzrZzy77Fk+2fIJXr+XzLhMzi08l9MLTufsfmdjNLQtIIgZrxd8PhVutAO/P9Tl8uGH8OWXqkNm82bYuhXy8tRizwC/+pXqpunTJ3QaNAiOO24/O3e59h8ktSVsOtDbHCqjsW1hUivX6xYrDT4bJTU29lbaKCq3savEyvZ9NnaW2HBgwxk4pXex0rW3jR79bOQX2igYZKPvYCvZuYaO0kglhBCiA9B1HXdgvSoNSAhMBd8YWK/KFXbKtVoZGB+PX9d5taSk6Xp3IIQalZjIaWlpNPp83LJ1a9P1wdtMz8pielYWJW43Z69eHbGty+/nzvx8ruzShXUNDQxcvDhqrM/168dlubksqq1lzLJlUfXgeldfVFVxWrP1royoLq8z0tP5tLKSKzdujAixrAYDT/bpw7DEROZXV/OvPXuwBEOmwPkt3brR1WZjaV0d8yormwKo4O3Ozcgg0WRim8PBFoejqRY8L4yLwxSYaunW9aaaSY4cKIQQxzQJiWLM7/JT/kE5xS8UU/lpJV1v7ErBPwrQ/Tp+hx9j/KGFOQ6Pg7mb5/LWurd4d8O7uH1u4kxxTB80nRnDZjC++3gMWgebI+RwwIQJasrZAw8c9rvzeqGsDHJz1dcPPAALFqgAaccOlVVNmADBg+78+tfqyGHBAKlvX3U6omv16HoomHK5IqfwNZ/Kd7iuC5+DdpDcmPEYbPgsNnSrDWOcFVOCDUuSDYP9wIOrqOuCR+0LPwWPJBj+tcyTE0IIsR+6ruMNBkiB8ySjkQSTiXqvl/WNjVEh1HEJCXS12djtdPJBRUVUCHVJTg594+JYWlfH/xUVRdRcus5jBQUMjI/nvfJybgusZxUM0Ny6zrfDhzMgPp7/Kyri+i1bosa8ffRo8u127t+5k9tbWO+qbNw4MlpY70oDLJpG9YQJ2IxG7ty+nVdKSiI6oeIMBuYPHw7A40VFLKiujpjql2Iy8ffevQGYU1rKFocjIsRKM5s5LzMTgB9ra6nxeiMCsESjkT6BLrbKwN8bwW0lxBJCiMNLQqIOxLXHBQaw5lqp/LySteeuJeuCLHIuzSFpdNIh/0JcVbyK33/8exbsWtB0XXZ8NhcMuoBpA6cxpuuYjhMYzZgBr74KDz6o0pfMTDjrLFVbuVKFFOHTrRISDn6K2n54PGqqldMJQ4ao637xC1ixAnbuDB1m/le/gtdfV5evugq6do3sREpKavehxV6wg+qnwqSwr3WHk5oSJ2VFLir3OqkudlJX5qSh0om3wdXUe2TXnKTanSTbXCRZnMQbndhxYvE7MXjC9u/1ts9jMZn2HyS1Fi4dzttYLB1mvSohhBAdlz8sOAoPorpZrZgNBva4XOxwOlUtEEC5/X7OycjAYjDwQ00NP9bVRe7D7+f+Xr0waBqvFBczr6oqYluAjwJ/GN22bRvvBda7Cm6fbDKxafRooOX1rnrbbGwJrHd16ooVfNlsvauh8fGsCFvv6se6uqaaBpyYksJXw4YBcMqKFWx1ONAATdMwACelpPB8YSEAJy5fTqnHo+qB0xnp6TwcCLFOWL6cBp+vqWbQNM7JyOD2wFTJE5Yvb1pH1BC4zS+zsrguLw+X38/PVq+O2LemaVyYlcVFOTlUezxcvGFDxL414OKcHM7JyKA4cFRGLXC9IbD9b3NyODU1lV1OJ3fv2BGxrQZcmpvL6KQktjQ28s+ioqj6Zbm5DE5IYH1DAy8UF0fsO1jvbbezpr6etwLrfWnNtu9itbKyvp6PKyoi9h0cX7rZzPK6Or6pqYmoacDF2dkkmEwsq6tjaV1d5PbAhdnZWAJrra1vbIx6/qZlZqJpGsvr6tjhdEbs26xpnBFYq2xlfT37XK6IsdsMBk4I/Od2TX09lV5vxL7tBgMjEhMB2NDQQJ3PFzG+OKOxaZrtVocDRwv17jYbALucTry6HvH9tRsMTQv6l7jdTa+d8PElBjokawJ/x4Zvbw4EpQCuQEdl8++vhKTicGtrSNQJFrPp/Kx5oalV1lwrmedlUvJaCfue3UdcYRw5l+aQ97s8jPaD6y4akjOE+TPms6liE3d/fTdvrHmDBk8D/178b2YtmkW3pG78csAvmT5oOqO6jIrtG9DDD6s5YX/8o/q6f/9QSHTddfDdd5G3HzUKfvxRXR45Ui2aE75uz8SJ8Nxzqj5jhlpMJzxkGjkSLrtM1Z94QgUgdjtmm42+drs6JjzqP2bv3r0SLBZcBjs7S2xsKbKR0iUOsFBfD3PnqjW4w911F9x9tzr8/GOPhcKjggKVb3VKwfWU7G2fHqkBKYFTc3V1oUWzlzdbODt8PfKsLCgcodY76t/Hy4DeLvrlu+ia4cToaRZOORxq42C3VfDUlutauk1Dw/63a6+j+wUFA6TDGVAFO6mMxpbPj1RN/uARQoiDYtA0bEYjtlbqeVYrefuZvj82OZmxycmt1i/KyeGinJxW6/f36sX9gaPkteSdQYPwhAVMbl0n/J/P/+rThwqPJ6JLKj5s3cQ/devGXrdbhVSBetewxzMmKYnuVis6NJ0KwxZDHBAfT47Hgx/VEaYDuYEP8QB5FgsNfn9TTQfiwzqMbQYD/rCaP2z8fl2nwedTtbDbNAaCNB+w2+WKqPl1nYpAd5RL11leXx8au67jB6YE/vFZ6/Uyr6qqafvgY5iUmsropCRKPR5ml5ZGjg2YlJbG4IQEdjqdPLFnT8S+dWBSaqoKiRoauHfnzqjv2ZS0NLpYrfxYW8tfWuhCm5qWRrrZzFfV1fxx69ao+tnp6SSYTHxQUcHdwfUcwvwiMxOLwcDrJSU80vyPZuCXJ56Ihlrr7Onww+8CcQYDDRMnAvDQrl28XloaUc82mykePx6A27Zv54P9BJTXbN7M180CymEJCSwfqT4bT1+7lqX19RH1E5KTWRDoopu8ciUbmy0BMTUtrSlAHbFkCXub/W04PTOTNwIL+nf74QfqfL6I+uW5uU1rtdkWLKC58LXWkr75Jirgu71HD+7Mz6fE7ab3woVRAeS9+fn8vmtXtjkcjFm2LCrge7BXLy7KyWFNfT1nBgPQsH08UlDAORkZLK6t5aL16yPCU03TeLyggJNTU/mmuprrt2yJCiif7NOHkUlJzKus5K4WAtCn+valf3w8H1VU8Oju3VEB47P9+tHNZuN/ZWU8t29fVHj2XL9+pJvNzCktZXZZWcTYNNQ04TijkTdKSvi4sjLq+Xu2Xz8MmsbrJSUsqK6OCvgeKSgA4NXiYpbW1zc9viyLhVsCB2I4lkgnUYx467yUvVnGvhf24dzuZMyuMRhMBho2NGDvbcdgPvjOn00VmyiqLWJkl5G8ve5tHvj2AbZXbcere8lPyWfagGlMGziNEbkjYhMYud0qzHE4VMtOz57q+iVLoLQ0tD6P0wmpqXDuuar+2GOwe3fkGj4DBsBf/6rqZ5+t2oDCtz/jDHjpJVVPSVGH+go3Ywa88IK6bDKpOWjhfv97ePxxta+8PPxWGx6jDRc2GnU7dRdcTp9/XM26hbWsGXs5DuxN6/UY4+0MumUqJ905kYrttex88A0yu9nI7mHDkhwIsQYMUAsnOZ2wZ0/0otVH8XQpr1dN+wtfNDsYJpWXh25ns0UvnN2vHyQmhnKJlk7hucX+TprWhhxD19Xrti2BU3uGVz91XXt1XB1OmnZkQ6kDrR3Mi6Yj3v5w7Dv4w7G/U/B73KYfJCGEEEdSeIil6zrGwJQ+r9+PJxhQhYdoRiNGTcPh89HYLGDTgQyzGaOmUev1Uuv1RgRcOtDDZsOgaZS53VQF6uH76B8Xh6ZpFDmdVHi9EfetQVMn0JbGRsqDAWBgHyZNY0wg9FxVX6/qYdvbDaFOo4U1NVHbJ5tMnJyaCsAXVVVUejwRjz/TbOa0QIj3Xnk51V5vxNi7Wq1MDtRfKS6mweeL2H+B3c6UQCfUE3v2NB2JMriPwfHxTfX7d+6MqPlRoejpaWm4/H7+tmNHxPOu6zqnpqYyKS2NWq+Xe8LrgZDwnIwMTk1NpdTt5q4dO6K+d7/JzubElBR2OBzc3cL+r8nLY3xyMusaGiL2H3x+/ty9O8cnJbG4tpa/hY0/+Bz8vVcvhiYk8FVVFffv2hUVgD7Vty+F8fF8UF7Og7t2Rb12Zg8YQL7dzmslJTy6e3fUa3Pe0KFkWyw8uWcP/woEpOH1ZccdR6LJxP07d/LM3r1Rj2/X2LEYNI2bt27l5eLiiJrdaGT32LEAXL5hA2+WlUV8X4Pdj0cDmW7WiXiqPJhTzeg+nR96/IDu0cm+KJvcS3OJHxB/SPuet3UeU16bgtVo5cT8E3F73SzYtQCv30vv1N5MGziN6QOnMyR7yNHf4lhfHxkgOZ1qvliPHioEeP/9yJrDAUOHwimnqK9vvjmy5nTC9Olw0UVQWor/hBPx1DvxNzrRnA6Mbid7r7uPHo//ka+e2sjJ1xRGDWn37U/RbeZVVH2+lNRJLfy8vvYaXHgh/PADXHBBZIBkt8P998PYsbB8uQqzwms2G1xyiXp827apLq3m9WHDVMtTXR3U1h7UUdIOh/LyUGAUftq2TTWDtbeWcowDPcViO6Pmx6y7seLCoquT2e/Cgpu0BDdDBvkZPNBPYpxPPXG+ZuctXXes1Hw+9XPf/PZtOQW3FS1rS5jU1tDpcO+rve6zeaj2U1+3122O1DaHst/9vcm1Vuuo20gYKoQQohOTkKgT0n06FR9XUPyfYirer0D36iSOTqTnzJ6knXbw6/JsqtjEzAUzeW31a1iNVmYMm8HgrMG8s+Edvtz+JT7dR7/0fkwbqDqMBmUNasdHJQAaa71s+b6UnRud7NniYO82JyU7nfzpyV70mZjLfx4u56tb5hKnOclNddA1zUFumosxD51L2omDcS5bh3XWQ2iuZusE/f3vMGYMfPopXHFF9ALU334L48erbqrf/jZ6YMuXq6DoySfVdL9wZjOsXw+9e8PTT8Ojj0aGSHY7/Pe/qtvr3Xfhs8+iQ6gbb1T7WbJEJTzhNbtdTQcEqK5WXTHBWisBlcsFW7bApk1qil9Ln90P5LP+oWx3JO/rYLarqVHnmgYDB6qXwbhx6tS7t3zOOWS6HgqZ2vub15FuH3ycLZ3Cn4cjdbtY3Gdbb9f8+fqprw/XNodrv0I52NS/tVp7vRkfrfvpaDryz0LzQLv5dYfrNp3pPtsS/rf1toe6/ZG8r7Zsv7/n9HBed6Tv72Cuy8tTS6QcBSQk6uQJXPBuAAAgAElEQVTcpW61btHz+yh4pIC009Nw7nTi2OYg5cQUNIN2wPsMhkWbKjbxw2U/oGkaRTVFfLT5I2avnc38nfPx634GZA5g2oBpTB80ncKM6O4X0f527FCNPps3qwBk0yZ1edcuSE6G229XGU1BgZpq1bevOv/1r9UsuRZ5vaE/Quvrobg4ejHqMWNUJ9G6dWoAzTupbrlFhUDvv68CoeadVJ99prqx7r8fZs0K1YPztD0eNcDrrlNBVDirVd0WVMfTyy+HaiYTdOmipg+CmvY3f35kCNW9Ozz7rKrPmqVCqPBOq65d4Te/UfUvvlDPQfj2aWnqCQWorFT3aberc+3Af746mvp6tZzXd9/B99+rZrTgbMusLBUWBYOj445T3w4hhGjVwQZjBxpW7i/E7Izb/NT+2ut7c7TupyP+Pu6IY2oeZje/7nDdpjPdZ1uC/7be9lC3P9j7ErFx5ZXqH+ZHAQmJjhLB74+maWy7bRu7HtiFraeNnBk55FySg617a8sZts7j82A2milvLGfgkwO5eMjF3Dz+Zvy6n3fWv8PstbP5Zuc36OgMyR7StIZRn/Q+7f3wxH6E/230+efwyScqPNq4MZSH1NSo29xwg2rWCQZIffuqwDtmobffrwKg4AKTpaVQVhYZMnk8as0oUCHO+vWRIZXFolYGB3joIZVyhG+fkwP/+5+q//zn8PXXoYWtAUaMgKVL1eWRI0OXgyZOVMETqIWONm5Ulw0GFSSddRa88Ya67tRT1ZMd3kl1yilw002qfvPN6jy8Pny4ug2oTqvwms0GubnqMeh6aN9W62H749PvD2WB33+vzoNrUlos6ikK7zbKyjoswxBCCCGEEAfrcARaLV0+Etcd6fs72Ouys9WRiY4CEhIdhXyNPsrfLWffC/uo/rIaNEg/K51B/xvEwawntLduL7d+fiuvr34dm8nGtSOv5ebxN5MVn8Xeur3MWTeHN9e+yXe71RHHhucMZ/rA6fxy4C/pldqrvR+eOAAeD+zbp5ppQGUoc+eqECl4sIj+/VUoAPDnP6tcJRgg9eunGm0MhtiM/7Dy+1Unk8ejVrcGlYbU1kaGTElJMGGCqr/8cmSI5XSqJ+nyy1V9xozo+uTJqoMKVNdTdbW6Pvieeu216oh6Xq+actfczTerb1x1terWCrJaVWD017+qowCWlMCUKdFT+S67DM48U3WIPfxw9FTA005T4Vd5uTqioNmsThaLOh80iBJ/Jj9+UcfGz3aydJWZZWssNHjMeDCT3juVUeMtjB/rZ9w4GDDIcHS+XoQQQgghhDgGSEh0lHNsd1D8UjH+Rj+9H+oNwM6/7yRtchoJwxMOKDTaWL6Rmd/M5PXVr2M32dl2wzay4kNtBLtrdjNn3Rxmr53Noj2LABjVZVTTGkbdk7u374MTh6S2Vk1Va2hQzTKgsoQFC9QUpKAzz4QPP1SXH30UMjNDAVJKS8eyFz9N11U45XSqBC4hQYVWq1dHT/Xr1QsGD1aLKz3zTPSi6lOnwumnq5Co+XpTTifcequab7hmjVq83OmMPNrZiy+qaXzffRcKw8LNmQPnnafWs5oyJap897hPeXLzZCaWzWEOv8SLEZ9BBU0GixnPB58Qd+IoePttNR8yGEIFg6iXXlKP8cMP1REEm4dU998P6enw1Veqkyy8ZjbDVVepwGvZMtVlFl4zm1UIZjCoOZmVldH3n5enHkhw6qPZ3DGnCAghhBBCCHEESEh0jHHtcbGw90J0l0780HhyL80l+9fZmNNb6GBoxaaKTXyy5ROuH309AP9d/V9O7XVqRGC0o3oHb619izfXvcmSvep7NabrGKYPnM75A86na1LX9n1got3ouuoyCk5Zy8qCX/xC5QqJiaHlgUAFRjfdpDqQ/H61JFHfvmrBY1m7pgPzetV0O6dTdRPFxakQavt2FV4FT263Ws06M1N1In3zTWTN44GzzkLv2o1dn6yj8tk57Nvhpni3h9pyNyY8/NPwJ1KG5XNJ96/4efFTZCR7iDOH3ccLL6gj6736Kjz4YPT+ly1THVj33Qd33qleaOGqqlRaecstqlOqpcdqNMI118BTT0XWbDYVuIFal+q119Rlo1EFSLm5obl2l1wC8+apACm4hlfPnmr6YrC+cGHkIrMDBsCbb6r6xRerlr3w+rBh8O9/h7bftSuyPno0/O1vqj5jhgq5gvevaSrUu/FGVb/8cvX9DB/fxIlqO4Crrw4trBisn3yy+uF2u+G226IXyj35ZDWFsr5eJcThYwvWjz9edbn95z/R9YkTYdAgqKhQUymb18ePVwFhWZkKAcMfm8GgHn9urpqGunRpdH34cLVmWFmZerMKf2yapl678fHq/ouKouu9eqk3qurqyOc2WM/NVa+Fhgb18xF+35qmugw1Tb1Ofb7oevjilkIIIYQQnYSERMcgT5WH0jdKKX6hmLoldWhmjcFzBx/UkdFKG0rp+mhXzEYz1426jj+N+1NEWASwtXIrb659kzfXvcmK4hUATOg+gWkDpnH+gPPJTcxtl8clDj+3W+UIGzeGQqRTToFf/Qp27w5Nawt+fu7bF66/XjWguFzqs1xennxuOhZUV6vMJLi20aJF6rM2qCmM4QtiDx3a8ky7Fvl8kUFWSop6wVVUqFMwXAqexo9X261apQKf8Jquh0KU999XnVzhIZXdDvfeq+pPPAErV4YW3PX7VXj20EOqPnOm6tYK1vx+yM+HRx5R9RtvVD804fVBg1T4AnDRRWoB9vD62LGh+pQpqlss/P6nTIF//EPVhw+HurrI+vTpofHl5kYeiczvV8HR/ferb0xWVuR9+/2q8+uee1RAmNvC+/RDD6npkJs3qx/25v79b3Ufy5apVc+be/VV1eU2fz6cdFJ0/X//g3POUV1mZ50VXf/iC/UG9MYb6k2ouR9/hFGj1ML1V14ZXV+/Xk21fPRRNWWzuaIi9YZ1zz1w993R9ZoaFRT96U+h73O4YHB07bVqDOEhUkKCCr9AjW3OnMh6drZ6zYIKAOfNi6z37KmuA7j0UvVYwwOu/v3h9ddVfcYM2LAhOqB8/HFVv+wy9QYeXj/++NBab1dcocLY8ABs3Dj15g7qe+xyRYZsJ5ygXtOgDijQPIA78UQ4+2z1c3b33ZF1g0HVTzpJBZSPPx4dYE6cqBZIq66GV16JDiAnTFAhbUUFfPBBdH3sWPXzWVamwu/mAeSoUep7UFYGK1ZE14cOVe895eXqUJrNH19hoQrfKyvVf12a13v0UEF0ba16DM3rmZkqoAx2jTY/0lBCQiig9Hqj68EDKwT/dpdfukIIIQ5AW0Oi1o6LJDohc6qZvGvyyLsmj/rV9RS/VEzS6CQAil8ppnFDIzkzcogriPvJfWXFZ7H6mtXM/GYmj/zwCE8sfoJrR17Lnyf8mfS4dAB6p/XmLyf8hb+c8Bc2VWxSgdHaN7n+k+u54ZMbmNhjItMHTue8AedFBUyiY7FY1DSzfv2ia9nZ6nNK8KhrwSApGAwsXao+r8fFRa55dPHF6uBh4Qtwi84vJUVlGMEZal6v+swbviB2sMkmLk59Jg2GRmPHRi6/FMFoVCdbs8X409PVqTVDhqhTa84+W51ac911rdcA7rhj//V//nP/9Vde2X/9k0/2X1++fP/14CJkLYmPD/2gtiQ7W30Dw0MkXQ8dMrFXLxUiNK8nJKj6oEGqS6p5Pbjq+ciRsHZtZICl6+pDPKgXxsKF0fXBg1X9pJNUYBJe9/tDi0dOmgTvvBNd79JF1adMgYyM6HpwPu3UqaF6+G2Cr8HW6sE3tNNPVy/olp674OOzWiMfW/C5A/U4w5//8OcOoFs3FViFjz28Hh+v2kDD6+Fvtg0NKmAMr1dVherbtkUHlOGh4bffRgeUSUmhkOj116MDSLNZ/by5XCroDN8WVNfgSSep/d5+e/Rr8uGH1eumtDQUVoV76ikVEu3YEQqCw736qnp9rV2rptM29957anyLFrUcUH75peqkmzcPLrwwur54sRrfnDlqSmxzGzaoX4DPPqtCxub27FGvzwcfVCFlc7W16nv65z+HguRwwefx6qvVNOUgTVPbBQ9hecklMHt2ZMCUm6uCL4ALLlDvPeH13r3VL3uAc89Vb+jhIdqQIWrxQ1DP3erVkd11o0eHujanTFHfo/D6iSeqUB7Uz255eeT9n3666iwF1ekY3uUXvM9bb1WXTzgh8rGDCs+vu04FcMGDYoTXL7kEfvtbFTD+8pfR9auugmnTVIgc/toK1m+4Qc3T37wZfve76Pqtt6rXzurVoXGG1++8Uz1HP/4Y6iQNd9996jlesCAynA5u/49/qD+q5s0LdaqG3+bxx1X4/cEHarp38+2fflp1aM6ZA2+9FV3/z3/UP1BefRU++ii6/tpr6vLzz6sO0fC6zRY62uy//63e18OlpKij0II6X7EicvvsbHjgAfX1ww+HDiQS1KOHWqcx+DwFj3wbHFvfvqF/CNx9t3pfCzd4sAr1QXXXBn9Ogtsfd1zoe37zzaEDoATrY8eqnxmAP/wh+rk56ST1jw+XS+2/eX3SJPX6rq1V/3hq7swz1c9HeXnoH0Th2//85+q1s2dP9JGCQb1uhw5V/+194YXo7S+8UAXcmzaF/skQXr/kEvW+uWaN+p3avH7ZZep9a/ny0HtAeP2qq9TfaYsWqffQ5vXrrlPvT999p07N69dfr35Xfv21OgpP8/pNN6nLn3+ufr7C62Zz6O+4Tz6JfO1omvo9edll6usPP1TvS+H1lBT1Ty1Q/8DauzeynpkJ55/PsUhCoqNUwuAECv5R0PR1/fJ6imYVsev+XSSfkEzOpTlknp+JKaH1l0C/jH688otXuOOEO5j5zUyeXPIkN45VUyB0XY9Y96hvel/umHgHd0y8g3Vl63hz7ZvMXjuba+dey+8+/h0n55/MtIHTOLf/uWTEZRy+By7ancWi/vk6alTL9fx89TsrGCAtXar+Bpk0Sf0989Zb6v0/eOS14Pkpp0R+XhKdk8mkDiQ3YoRqLADVvPD996HT3/8eOtLzgAGR3UZ9+kiIGDOapoK51hiN+1+gzGJRQUZr4uPVN7w1qanqD9/W5OSoU2vy80OBU0sGDNj//e/vjQ3Um1TwCIUtOeccdWrNJZeoU2tuuKH1GrQcIoT717/2Xw8enbE1X3yx//qaNfuvV1S0XktMDK0HFhR+pJicnNBC/+EhU3A+c+/eqtuneQCZnKzqgwapkKt5Pfh6GTkyukNQ19V+Qb35fPttdH3YMFU/6ST4+OPogLEg8HfVaaepNLx5PRiyhQeI4fXg+M84I7IePAUf/5lnqg8nzY9GFB6Y5OZG1sLbNqdMCR09M3gKHsgBVAiTnR1Zz8wM1SdMiLx/vz/UUgzqDT8tLfK5KywM1QsL1WMNf3zBdeJAtZ3GxUXWg89N8PVjNEZOQw5/fBZL6DUV1NKRFcJfc+H7Cq7fF7598JeU368CquZ1jyd0u9ra6Hrw9e7xqA/7zevB4MHlivwg2rze2KjC99a2r6kJhX3htwnef0WFCiubbx98zCUloW7G8Hrw+dmzR3WJNq8Hbd8eCoGCdbs9VN+wQQVd4cJfWytXhoKE4PY9e4bqP/6ojmYbLviPA1AdgqtWRY5t7NhQSPTZZ6Hp5MH7qK0NhUTvv6+eg+bPbTAkmj1bdTo2f+zBkOjFFyPHDurxn3OO+t4Hw9vwemqqek9oaGj5fbtLFxUSVVXBY49Fb9+rl/pdWVwc6iION2iQCol27gwdUCV8+1GjQiFRS79XTjxR/S5dvTrUaRrujDPUGBcvbvkfZ+edp0Kib7+NDMmCLr5Y/UzPm9fy/V99tXrv+/DDlrt3g9/bt96KDMdB/Z0RDIlefhn++9/Iek5OKCR6+unQYqxBffqEQqLHHgsd9ThoxIhjNiSS6WbHENceF8WvFFP8QjGOzQ5ST09l6CdD27x9RWNFUxfRz17/GQMyB7Q4DS1I13XWlK5pCow2V27GqBk5tdepjOoyirzEPPKS8prOs+KzMGhy+KSjgcul/r4zmVRI8PzzoRCprEzdZts29XfBiy+qUCk8QOrbV/0+kvDg6NDQoP62CHYbff+9mokBqlEjPDQaOTK6mUgIIYQQQrST5p//g9NYmx8GHkIdfX7//7N33uFxVOfbvmdmd7W76pJlWXLvNjbGuHfTq4EQSughgIHQQgIJJQkGwg9DgBCSUAwhQL5QQgsdQq82BONu494lS1aXVrvS7s7M98fR7O5oJWODwTa8t665ZvY855wpO1rtPHrPe5JmaqruTAwSj3esZ2Qkh9E6ZmuqHgwqvbU1XbdtFcEKKkrQMUNT2zv/zAqF0ttrWjKEvbExvb2uJ6PV6+rS23s8O45m3weRnERCp9i2TcMnKtQyb0oe0aooiw9ZTPFZxRSfU0xGyY4zE7fEW5j58kyeWPoEfo+/05xF7fe5uHIx/172b55f+Tzratdh2qarjkf3UJpd6jaPsrvTI6eHy0zye+QJcl+mrk4ZRmPGKCPpwQdVBPqaNclcw5qmtjMyVOTz2rVuAyn1H6LCvodlqX82OsPT5s5V9wSo7xqjR7uNox0FkwiCIAiCIAiC8NWISSTsNM3Lm1n989U0fNQABhQeXUi387pReGwhuq/zyJ5V1au45aNbEmbRa2e8xvQ+03dqn6ZlUtlcSVljGWVNZWxt3JrYLmsqo6xRlTXH0vNpFAQKkuZRu2gkp7wgUOAaDifs/ViWSgWwerWKdnZGaZx3noo2Sv2oOuCA5JD2V15RptLgwSpa1iODaPdJqqpUhLljGn3+eTK6vl8/t2k0bNiOR0kJgiAIgiAIguBGTCJhlwmvDlPxaAUVj1UQLY8yfsN4An0CmBETI9D5E9mq6lXc/end3HXEXWT6Mvmi/At65vb8xsmqbdumsbUxYRqlmkepZtL25u3YuO9jv8fvikrqkd0jzUwqyS7BZ/i+0TEK3w0tLWqIuTNkzbbhuuuUNmJEMo+d16tSRsyYkRy2vWSJGrrWRVJh7VO0tqociakJsZ1clDk5MGFC0jgaP16iywRBEARBEARhR4hJJHxtrLhF0+dN5E5USQSXHr+UaEWUbud1o+tpXfHmdT6ntW3bDLtvGJsaNu3UMLTdQcyMsS20zR2NlBKV5JS3mq2udhoaXTO7uoa2pUYjOds5GTkSlbQXU1OjjKPUZfBglSzZttVQ4ro6lWPTmcHtuOPU5C2ghif7xCvc67FtlS8zdYja0qWqXNeVWZgabdS7t+S0EgRBEARBEAQHMYmE3cbWv21l24PbaF7ajO7X6fLjLnS/tDu5k3I7rL+yeiW3fHgLTy57cqdzFn3b2LZNbaTWZSB1NMStJpI+W0umN9NtHrUb4tYjpwfFmcUYuox/2RG2aWM2m5ihtqUpZftrLGigeTQ0Q1PrDhY8GnUNGs0tGqEWjaZmjcZm6NlbY/8DNUw0nnxaI5ilkZ2nkVugkVuo0bOPRkGR9pX9d7h8C/XREaOyAxoa1Iyrjmn06acqbyGo6LFJk5LG0ciRYgYKgiAIgiAIP1zEJBJ2K7Zt0/RFExWPVFD5eCW9ft2L3r/tjdVq0bqtlUCfQFqbVLPo8R8/zmnDT9sDR75rRGIRypvK04a4JUylpjLKm8qJW3FXO0Mz6JbVbYcJt7tndyfTl7mHzmzXsGLWbjFyUhcrYn31jtvQfBpGltH5EjRAAztuY5u2Wu/K0tbGjNo01Kq1FVNlBjYBn41HU68xv/p4vwsShpHBTplPRtDAV+ojozSjw7Un1/O9M55MU0UXpUYbbdyoNL8fxo1LmkYTJiQnxNC05OK8Fr572s9m3tH2vqLt6vbe2tc37ReSE+N0tN7bte9iP4aRnA3U2W6/dKbtSrmzP0EQBOGHi5hEe4jyOeVsvHkjmldD9+rqgc2rdbjeLfq33bee/o3CjJjYcRtPtoftz2xnxU9WkH9oPt3O60aXH3VJy1+0umY1/fP7Y+gGf/70z5Q3lXNI30PI9GaS5csiP5BPn7w+AFi2ha51nix7b8CyLaqaqzqNRnLKG1sb09rmZuS6zaOUaCRnu0uwy05fA9u2sVp3v6FjR3f+c0EP6Ds2dL7GsqOE6d8mTvLszEw1TG3ZMvjVL23WrLIp2wIGykT612M2xxxps3SRzb8ft+nfx6Zvb5u+vWyKu9holv31DKxvYHi1X8wmk+i2KK3lrZgN6U6XHtA7NZEyuqttX4kPT9a+nQm8vNxtGi1YoGZp3Vk6MpDab3/V671J29V+vktTRdg1OjIfOjMjvun2120Pyfe5s/XeoDnb3/f7sDMTancZUd+G2ZV6H32d9Tdpuzv6+C73/1VlX6fNNynb2/b3XW/vS8exO9jb+tmdff3sZ3DPPbunrz2NmER7iNo3a9n+9HbstqiE9msrZnVYvkM9lnzw+87R2KHBhA1ms0m8Po4ds8EAX5GPwMAAuj/diJpdMpsnc5/E0pJRJcOjw3mm8hk0TePEriey3rueoBUkYAfItDMZHR3NjQ03gga3Zd9GWA8TtIMEbVVnYHwgB8cOBmCedx5ezZtoGyRItpVNUAsmzif13KDdMB6N3VavmWYq9UoqqKBSr6SSSiq0Ciq0CiqppFKrpIoq17UA8Npe8shDszU0S11j19pKrrFQ9dDQbLVf5zU2ifLEa10Zf7qup23rRgdrQ63Ttj1t2x4dTdPUQudr57rsqM6urv0eP0FvkIAnQNAbVNvelO1Oyh3NZ/h2KZImHIY1a1TOo+nTobgYnnkGLrgAGlP8QL9fzcw1fLgyJJYvT+ZCyu14hOa3jtls0rqtlWi5Mo0S6zL3aytspbU1coxOo5Gcta/Eh+HfN4ZbhsMwf756jyKRzr9Efd0vXHta2937MIydNwu+DW1P7vvbME12twEj7H5s+7s1qkwzucTj7tdfVf512uyt+zD3kqjdXSHVWN+Z9ddps6ttv6rs67T5JmV72/6+6+196Th2B3tbP7urr4kT4ZRTvnk/ewNiEn0Pse22qIHdaUC1rb9JWzVsxyRWGaN1cytmk0nmyEwwIVYTQ/fp6tjb6lZ6K6nMrKTF00LYE8Yf9zNu6zgAnh32LJWZlUS8ESKeCBFvhL61fbng8wsA+PmPfk55Tjlhb5gWbwu2ZnPwmoP5w3//AMBRM48ilBFyXbejVxzN9W9dDzb86IIf4bW8BKIB/DE/gViAg1cdzAlLTyCuxbl/2v0uzR/1M7hyMP239yeux1lftJ5AayCh+aN+PLaHxO9R6q+TTVpZ+3qmZlKbWUtVThXV2dVqyammMdgIXtB8GngBD8qUa9vGixp65LxOWVQuHsDAvdZBeUW2ej++w3Xi/t1NfVq2RavZSiQWoTnWjGWnmxtfha7pO28yedqZTCn1/J4AraEgVeVBKrYEKNsY5NdXBinKC3DbLUH+dHsAbBUZVVyszKJXX4WsLFi7Vj0o9O2r/mO6J7FtG7PRdJtInaw7ijTzFHi+2kzq5kP37t2RgoIgCMJ3T2emWSrftsGys30IgiDsq4hJJOwxbMtG0zWsVotPij/BClsUHldIyXkl5B+Zj+7ZPQ+Jtm0TiUcwLZPsDDX/9WdbP6Mp2kRztJlQNERzrJkBBQM4pO8hWLbF5a9dTigWSuihaIhTh53KFeOvoKGlgZ539yQUDSWMDYCbDrqJG6bfwNbGrfS8u2facfzpiD/xy4m/ZH3deo578rjEMDpnuWj0RUzvM53ypnL+ufifZPmyEnUyfZmMLhlNcVYxkViE2kgtmb5Mgt4gXt3L9y1nzLeBbdvErBiRWIRwLEw4FiYST9nupNylxTuvl1refoa8ncWrZeCxg2jxIMSCDOijTKaNq4NUbA2gxYNkB4IUZAXoVhjksOnKiPLYQXKDQYLeHUdFOdteo/OZB3cXtm0Tr41/tZm0rTU9n5MG3iJvYjhbp2ZSkQ/NkHtfEARBEARBEHYXYhIJewWhZSGV7Pr/VRKriuEr8THwvoEU/aiI1vJWGj9tVMPRUnI4ZY3MwpPrIVYfI1oWTehOHU+hB92jq6gq21azQ+1GM8UxnxwjKTsjmy7BLoRjYd5a91bCfApFldl0eP/DmdBjApvqN3HVm1e5tFA0xOxDZ3PSfifx8eaPmfrI1LT9PXfqc/x46I/579r/ctTjR7k0r+7l9TNf59B+h/Lyqpf5+as/x2t48Rk+fIYPr+7lsR89xv7F+/Pftf/lz5/9OVHuM3x4DS+3HnIr3XO68+GmD3l51cvJtm39XDj6QnIyclhcsZgllUsS5U4fh/Q9BK/hZUvDFqrD1a62Xt1LaXYpmqYRM2NomoahGd9bc8u0TFriLbtkRu3IpKpuiFDbFCbUorRWK4ztaVvY9c9mQzNSIpz8eHRPYvEaXtdrj+7Bq3s7r6N13G6HbVLqGBgQAuqBOrBrbaxaC7vaxqq2oArMShO7xsYwDTyWB8My1IJBoEuAYNcggW4Bgt2CBEoCZJZmktE9I2EmeQvFSBUEQRAEQRCEnWFnTaJ9OyOpsNeTNTyLAXcNoN/sftS8WkPFIxVkdM8AoPF/jSw/aXlam5EfjiRvah41L9ew8pyVafroBaPJPjCb8gfLWXPJGgCXkTRmwRgC/QKUzyln8x2bk0m425YRr47AW+il4p8VbP/39qRB1aYPfnAwwYwgza80Y35s0uBtoMnbhObVGOkbSe9rewNQ934dkdURtPUaFd4KAt4ADwQfoMupXQBlkMVr42gNGo2fNzLCM4LqU6qx+9iEoiHqttfR3NLMgJwBxJviDMoZxAPHPEBzvJlwLEzMjBE1o/TOU/vrltWNowYcRdSMEjWjxCyl+z1+AFriLdSEaxLlUTNKzIwRiUcAWFyxmHs/v5eoGcW0kyEepw8/nZyMHF5Y+QI3fnBj2vWuv6aeXCOXez67h7vm3ZWmx38fx9AMLn/9cuZ8MQcNLWEk5erH+LEAACAASURBVGbksvVXWwG46r9X8dra11wGVnFmMS+c9gIAsz+azcKKhUkTTPdRml3KrINmAfDIwkfY0rglYU55dA/dsrrxk+E/AeCV1a9Q31KPR/dgaAYe3UPXzK5M7jUZgHlb5tFqtrr0gkAB/Qv6A7C+bj22bSeMDkNXhktORg4A4VgYj+5J5EL6tswJ5dvbRM0ocx4JM39RhNUbwqzfEqaqLsKYSWFuvEWZTLNuiaD5whR2C5NXFCG7IExWXhjNF0lE2cWsGHEr7lpipiprjbeq1zuok3idUufrDO8DQAOK2pahu9jWAraCvllPmkmWgRevej81ZVh5PV48Hg9erxevz4s3w4vP68NjpBtduqZyaumajobmep1Whr5r9VNe78vtUpfUPtr3tbvqdVRXjEBBEARBEITvDokkEvYY8YY4LRtbVL6jlATdWaOy8OZ5adnUQuNnjcmcSW11ik4twtfFR+P8Rmpfr02UO3X63NAHb6GX6per2f7U9jR92NPD8OR6KLu/jG0Pb0vuu63O+FXj0TN01v1mHWX3lSU0bGVGTY9OB+DLc7+k8rFK1zl58j1MqZ0CwLKTl1H9XLVLz+iVwcRNEwFYfORi6t6sc+nB/YKMW67yMy06dBFN/2tSU54bGpqhkT0umxGvjABgybFLlEnlTItuaOROzmXQfYMAWH7KcqKVUTVsx9Gn5tLnd32wbItl5y+jNdxKQA9gGAZNvibMcSZ5P84jakZZf9d6YnaMAzgAj+FhjWcN24duJ3BggNZYKxWvVBDX4pyRcQaaR+N9831WFq/E6G7QGm2lfnk9hm4wq2gWGPBww8N84f0CK9MiGo0Srg6T7cnmsWGPoRkaV6+5mk+aPyGmxYjGo0SjUXpm9uT9w95HMzSOfOdIPt7+set6jSoaxbxT5qHpGmOfGsvi6sUufXrP6bxz2jugw5AHh7C2bq1LnzFwBi+f8TIA3e7sRmWz+/08ffjpPHHSEwBk3ppJOBZOaIZmcNHoi7j32HuxbIuiO4pcBpRH9zBz1Eyum3odoWiIKf+YkjCfHP28kefx05E/pSZcw09f+KnLoPLoHs7a/yyOHng0FaEKbv7gZsy4gRnzkJdjYGgeljx1EhULxrKqbBuRwY+CbTBurMEZP/GgaTqRpUdx3gkDac3YysurXqZ9IvGjBhxFr9xebKrfxDsb3kkkBgfQUHpxVjEb6jYwd8vcRHsblRfq4D4Hk52RzZqaNSyqWIRpm4mcUaZtMr77eLyGlw11G1hbtxbLtrAsK6EPLxqOpmlsadxCeVO50m0L0zIxbZO+eX2xbIuKUAU1oRpi4RjxcJxoOEo8HCc/kk+0OUptay3NsWZi0RimaWJqJmgQiAWI63HCgTBxfxwrw8L22lheC9uw8Rk+LCzidhyzbWychYWNra5TWw4v0zZVZJemZje0sRP1nB+r7ce21XZqX5ZtueoLu07COGoz63R0V5lj5KUaes4acJeRNKPa96uhpfeN5tJc++vgB0j8rnSopaxdP1rbRANaJ3q7PnbmZ0f766yfne3/q/oEkhMndPB+ppVpGu1/Pb6qrfP71FG9DsvtHey7o7Y78eu60/vupMyjeRJGt/PZ7/wdMXQjEdWZ+rcj8XfGaNs2lO7VvRhGsp5T7kR9GoaRmLQCve28ddRrDde2Uyd126nfUdmu9iHmryAIwnePDDcThN2MbSoTyZnFKd4QxwyZLgMLGzKHZQLQvLyZaEXUlfRbz9ApPKYQgJpXa2jZ1OIysLwFXkovKgVg61+30rKhRQ2rM20wwd/HT69regGw/rr1qr0zBbppk3VAFn1v6gvAijNWEN0WTbS3TZu86Xn0v11FziyYuIBYXQxMEn0UnVzEgD8NAOCTrp9gtViu/ZdeUsrAewZitph8FPgo7Rr1uq4X/W7tR7Qqytyuc9P0vrf2pfd1vYlsiPBZv8/S9AF/HUCPy3oQWhJi/gHpnwUD/jGALmd3oeqTKhYevRDN1shuUfmoqrKr6DWnF7mH51LzYQ1fXvIl/pifHrU9APiy+5eU3l1K5uhMqj+oZsPsDeSGc9mvbD/Q4YOhH1B8UzEZ/TKo+aSG8ifK6V7fnbGbx6LpGk+NeYqiS4vQu+jUz6+n+r1qhlYNZdrGaViaxZ+n/pnCkwuxAzaNKxppXN7IxLKJHLHxCCLeCLOmziJ7ajamYRLaGCJSHmHGphnM2DyDGn8Nv5z8S3yDfZi2SUt1C9FQlHPXncsJZSewOXMz5084H3LAtE1irSqq5/rV1/Oj7T9iac5Szhl5Ttr1Ou6Zu+i+4gTCUz/jn4ecmaa/dPJLHDfsOF5Y+QIn/vvENP3Dcz9kau+p/GvJvzj7P2en6QsuXMCBJQdy/+f3c8lrl6Tpay5fw4CCAfzxkz9yzdvXpOkVV1VQnFXM79/9Pbd8dEua3nx9M0FvkCvfuJJ7PnPPPaprOuYNyti54KULeHjhwy4928hmTf81RMujXFx3Ma95X3PpXRq78MyfngHg2jOv5bOB7vuxV1UvHrv3MQCu+NkVLO29FM3WMCwD3dIZUjaEex5Vx3T12VezqWiTK8Jp2JZhXPOiOucbTr2B2uxadCsZBbXflv346Yc/xdIs/jTjT4R9YXRb6ZqtMaRsCEcsOQJLs3jsoMewNAvd0tFttfSv7M/IjSOxsHh11KuJct1SxkXPmp70qepDzIixsI/6XdHtNlPD1ihqLKIgVEDMiLGlcIvS0cEGwzbIbMnEH/cT1+LUB+sThpmmKZPQF/fhsTzE9biaRAAbW7MTZpphGejoxLU4cSOOranPR1tT9SC57SyWZiXWgOt1R3Wc/XVYvot129f7qjrt+3TO+6teQ+d1EtdkN79O3efX7lPbe74rCt8M53PI+bxJ/VxyyhKvHc1OL0utu6M+2pen1bUNPLYH3dbx4MGwDfc2yTyWjomaaj7q6MqESilLreNsJ+qkttfamZo7MkU1wFZ/exL97aBu+35d7ZzjSD3e9mZrW5ljRjv9oOEylJ3P9ITe9qua2E4xRhP92yTaONc1oduaSuVAO91ON4AhpX8b17GkGqxps++2O9YE7V53+Iz6FW12qo/O2Nmqu7seu3Ccu/JRvKsf2xqJv1OapiVf70JZwvzdW8qc2/gbluUfkk/J+SW7dj33UsQkEgThW8O2beyo7TKg7LiN7tfxZHmwTZuWLS1JA6pt8XX14evqw2wxCS0MJcwnRw8ODuLv5SfeEKfunTqXQWWbNjmTcggOCNJa0Ur1f6rVQ6elHj6xoODYAoIDgkQ2RKh6ripRbts2WND1tK4E+gVoXtFM1TNViXKnn9KLSvH38tP4eSNVz1S5+rctm17X9CKjJIO69+qoetatY0O/2f0SUWxp+7dh8IODMTINKh+vpPqFavf+bZthzw5D9+iU3VdGzSs1rv41j8aI11QU2abZm6j9b22irWVZkAtDXxxK3Iqz9ndrqXm3Bsq86NUaJq2sCsS5JnggTz8NvluX0jCvgdxILn6fH7OrSXxSnIH3DgRg6wNbiTfF6V7QnazCLFryWqgvqic4OKgeHi31cNs3vy9+j5+acA3lTeWJ2edAPZQO7TKUDE8GFaEKtjZudc1QBzCqZBRew8vmhs1sadiS0JzogMk9J2PoBmtq1rClcUta+8P7Hw7Asu3L0tp7dA9HDVA5vj4v+5yyprJEO8u2yIhlcGSPI7Ftm3c2v8PW0FbiZnJ4Xa4vlzMHnIlt2/xr9b/YHNqcGHZnWialgVIuHXIptm0ze8lstjRvUW3tOKZlMiRnCNcOvRbbtrn484vZ1rKNuBXHtE3iVpyJBRO5eejN2LbNkXOPpCZao7S29jO6zuD2wbdj2zYDPxxI2AwTt1X/Fhbnl5zPHf3vIGpG6TavW9rv6BXdruCGHjdQG6tl0OJBafp1xdfxy6JfsqV1C6PXjE7T/9DlD8zMm8mX0S85ePPBafrd3e7mjLwzmN8yn2M3Hpum/73n3zk+93jeD73PqRtPTdOf7Pckh+UcxqsNr3LuhnPTInb+M+g/jM8ez7O1z3LVxqtwHvaceq8Me4WhwaE8XvU4f9j8h8SDpPPQ9MaIN+jp78k/tv2Dv5b9NfFw4/T/5qg3yffmM6dsDo+WP5qIDnJ+3h73Nhl6Bn/b/Df+U/GfZOSRpuHRPLw2XpmO92y4h3eq38HQDAxNPdBme7J5eJQyLe/bcB/z6+cnNEM3KPQVcst+t4AGD214iNWh1UrXdAzNoNhfzOUDLgfgn5v+ybaWbYmhd4ZmUBoo5dRe6po+t+U56mP1CU3XdEoDpRza7VDQ4I3yN4iYEQw92X9JoIQDCw4EYG71XEzbTBybrukU+Yvom932z4b6Fa59a5pGji+HAl8BFhaVLZWJ66rr6j3I0DPI8GRgWSpKT9eT1w6S0T+puL6LpphpO6znlHViYHXWvn0Ey+4+HrRO6nbS3rRM9dkTV7/7MTNG3Iwny51hv2Yc0zQTrx09ZsWwLCsxJNjZtiyLmB1L1m1bO59BzmeN87nibFu2pXSSZaZtJj+fSG47n0euctrqo7Yt21JlJMucbUGAdiZVqokEaSZXh/VSjbEO9M6i/To9nh3UTzXXdqbNrpbv6jHttnLHVCXlH0porvKOtnX0ZHn7dbsy5x9RTv/Qtm2nbDttOuqzszLnWGzdZVa2L0/VE9vt+7U72JelMWHYBGb+YeYuvWd7K2ISCYIgCGo2smaLT942efpNH3fdBZGPa3njHy3UbYxxQN8Yhb4YGaUZ9Lu1H6CizBo/b3TNTpZ/ZD4HvHEAAPP6zCNeH8db6E0s+Yfn0/NXava/bQ9vQw/oeAu9eAo9eAu8eLt68WRJGrzdgTOkz9ANbNumsbUx+eDXtmT5sigIFBC34nxZ9aVLNy2Tnrk96ZXbi0gswnsb33M/eFpxRpWMYnCXwdRF6nj+y+cTBpwz3O6gPgcxpMsQKkIVPLP8GZxhdU6d4wYfx6DCQWyo28C/l/87Yd45dU7f/3QGFAxgRdUKnlr2lKtvy7a4eMzF9Mnrw/zy+Ty59Mm0/q+dci3dc7rz/sb309rbts1th91GUWYRL696mWdWPOPq27ZtHjzuQXIycnhi6RM89+Vzae2fO/U5vIaX+z+/nxdXvehqq2s6b579JqByqb2y5hVMy0wMo8zyZfHBuR8A8IvXf8Hra19PaJZtUZJVwqcXfArAKc+cwjvr31EP1W19DOkyhAUXLQBg0sOTmLd1nuv9n9BjAvPOV2X7378/y7Yvc+mH9TuMt85+C4A+f+7DpoZNLv3EISfy/E+eB6DLH7tQE6lx6ecccA6P/egxADJuySBqRl36pWMv5W/H/I2YGcN3iy/t/rx28rXMPmw2tZFaCv9YmCh3zKRbDrmFa6dcy+aGzex3735pOalmHzqbmaNnsqp6FYf885A0/fbDbufk/U5mUcUiznjujDT9j4f/kcP6Hcb/yv7HlW9cmZb/6vbDbmdc93F8svkTbvnolrQ8WLMPnc3QoqF8sPED7p9/f1oOrVsOvoWeuT15b8N7PLXsqbT93zD9BgqDhby74V3eWPtGmn7N5GvI9GXy3ob3mLd1Hn6Pn4AnQMAbIOAJcNJ+J+HRPWys30htpNalBb3BxGyu+zqpQ5Pbf/44hnrMjLnM/69aO/3uybqp/yj5ruqmHl9H26ntOtr+Ou33tX2m3X90Ur6D5+JdbbMv1bdx/w1s/zdzZ7Y7at/Z9nfZ1+7o98JRFzLnuDkdXr99DTGJBEEQhE655hq45x5obYX+/eHMM9UyqC3oxLZt4g1x4rVxYjUx9AydrBFZAGy+fTOtZa3EamLEamLEa+PkH5pPv9n9sG2bD30fYsfdf1tKLy5l0P2DsOIW80fMx1PQZh4VevEUeCg4uoCCwwqwYhYNHzW4DCYjaHzXl0cQ9gpSH6QdEwkg06eGNTuTFaSaVBlGBiXZKix+Tc0aWuItrj7y/HkMLFRRg59s/iQxmYGTD6w0u5QDuilD+LkVzyXaOXUGFw5mfI/xmJbJQwsecmmmZTK2+1im9Z5GOBbmrrl3Jb5sO8d4eP/DOajPQdRGarn1o1sTucic5eT9TuagPgdR1ljGje/fmMjp5egzR81kWu9prK5Zze/e/V1a+2smX8PkXpP5ovwLrnvnukS5cxx3HH4H47qP490N77r1tmv96I8eZWS3kTz/5fNc/871rr5tbF4/83WGdBnCwwse5nfvpe9/8cWL6ZXbizs+uYNZ789K06t+XUVhsJDr37me2R/PTnvPo7+L4jW8XPrqpdw3/z6X5tW9RH+vTLtLXr2E5758zmUiFWcV8/qZrwNw97y7WVy52KV3y+rG5eNVlNpb696iJlKTMJ8C3gD5/nyGdR0GQF2kDl3TCXgDeHWZSVIQhB8uHUWh7quISSQIgiDskIYGeP55ePxxePddOOooeK0tdU9dHeTn73qftm0Tq24zj2rixGrVdnBgkNzJuZjNJivPXZkod+r0urYXfX7fh5atLXza81NXn7pfp/+d/el+aXdatraw9sq1SYOpzUjKOyiPQL8AVqtFvD6Op8CD7tU7OUpBEIQ9ixM10xJvIRKPEImpWSn3K9oPUMNo19WuS2jhWBjTNrli/BUA/HPxP5m7Za6rbdAb5N8n/xuAi16+iDfWvZHQIrEI/fL7sfry1QAc9OhBfLDpA9cxHdjtwEQU29iHxjK/XH0n1zWdgCfAwX0P5uXT1WQPJz19EtuathHwtplMngDju4/nqklXAXD7x7cTNaMJgyrgDTC4cLBrxlGv4XWZWDkZOQkDVBAEQdj9iEkkCIIg7DTl5co0GjoUtmyBfv3g4INVdNGJJ0JOzre7f9uy0XQNs8Wk8dNGl8EUr41TeFwheVPzaF7RzPJTlicMJidiaei/hlJ8ZjH1H9azaPoiAIxsI2EkDbh7gGq/spntT2zHU+DBCBhoXg3Nq1FwZAG+rj5atrbQvKwZ3asnNM2rkblfJkbQIN4YJ14fT5TrPlVP98tU7YIg7N3ErTgeXQ37LW8qp76l3mUi+T1+pvaeCsBTy56irLHMZUL1yeuTMKlmvjSTjQ0bXe2n9Z7GAzMeAKDkrhIqQhWu/Z814iz+34n/D4DA/wVoibe49ItHX8z9M+7HtExyb8tNRDE5Q/UuGXsJ1065loaWBsY+NDZtKN/l4y5n5uiZVIQqOO7J41zDCHVN5xfjf8Epw05hY/1GLnjpAtcwRF3TuXL8lRze/3BWVa/i+nev77D9+B7jWb59OXfOuzM5a2Lbctm4yxjWdRiLKxbzyKJH0oYy/nzsz+mT14dFFYv4z5f/SRvKeMGoC+ia2ZVFFYt4d8O7Sb2tj7MPOJucjBwWVSxifvn8RJ4wp84pw07B7/GzpHIJK6tXJvOwteVbO37w8Ri6wfLty9nUsMmlG5rBof0OBWBV9Soqmytd+/YaXsaUqufK9XXrqW+pd+kZngwGFapQ5K2NW4nEIq5jy/BkUJqtJmapDlcTM2Ou8/fqXnL9uQCEoqHE0N7U4/MaXuD7FdUhCN81O2sSSYIIQRAEgdJStQD4fHDddSrC6Nxz4eKL4fjj4dZb1dC0bwNNV1/4DL9B/kGdhzBl7pfJuOXjgLakryGTWE0Mb4H68ujv52fg3wamRSo5Q9bCy8NsumVT2qwfB849EF9XH3Vv17HqZ6vS9jtmyRiy9s+i4rEK1l6xNk0fv348gb4BNt+xmY03bVQmk6/NSPLqjP5iNN4CL1v/tpXKxypdmubVGP7icHSvTsU/K6h7t85lUul+PTErYfUr1YRXhF0mlZFlUHxGMQCNnzUS3R51GVhGlkH2gSqPScvWFuxW273/DE3yRQnCDwDHIAIozS5NPLR3xGnDT9thXw8d/9AO9W1XqWT9jokUjoXxe/wJ/ZXTXyEcC7tMqKFdhgJqcoGLRl9EJB4hZsYSQw7756vPQUM3GFM6Jm0oX35A/e3Q0Oia2dU1jNDJ4+b07wzDdIYhWrZFJB4BIBKPsLpmddpQxLqWOgBqI7W8u+HdtP2fOuxUhjGMTQ2beHTRo2lDHU8ceiJ98vqwuGIxN394c9o1mzFoBl0zu/Lx5o+56s2r0vRjBh5DTkYOr695nevfvT5NP3rg0fg9fp5a9lSHQxlbftuCoRs8MP8B/vb531xa6lDG2R/P5rHFj7n0fH8+tdfUAvDrt37N818+79J75fZi05Uq/9l5L57HW+vfcunDioax7BKVP+34J4/fYa61iQ9PTMu1dni/wxO54Pre0zdhcjlG1IlDTuTpU55Wx3J3L+pa6lwm2OnDT+e+Y9XwzZK7Soia0WRya03j/APP57bDbiNqRul5t8qvmKpfPu5yrp96PXWROobfPzxN/82k33D5+MvZ2riVyf+YnKbPmj6Lc0eey+qa1Rz9+NFp+m2H3sZJ+53Ewm0LOe2509L0Px/5Z44ccCRzt8xl5ssz0/QHjn2Ayb0m8/b6txP3Tqr+yAmPMLLbSF5e9TI3fnBj4ro6dZ46+SkGFAzg6eVPc9e8u1xtNTSePfVZSrNL+X+L/x9zvpiT0Jw+Xj79ZXL9ufx9wd95fOnjaft//czX8Rk+7v3fvby46kVXe6/u5ZUzXgHUMNm3N7ztap+TkcMTJz0BqAhF595x2ncNdk3kC7rlw1tYXLk40V7TNHrm9OTOI+4EYNZ7s1hTu8Z1bgMKBnDjQeqa/Pad37K1aaur/bCiYVw96Wp+aMi3QkEQBMFFcTHcfDPcdBN8+qkyi557DjLbRgHMnQvxOEyZAvoeHNGlaRqebA+e7OSfMn8PP90v7d5pm6KTipgem068IY7VYmHHbKyYRUb3DAAKjy3kwLkHYsfshGbHbPy91cNN/qH5DP774ES5HbOxohbeQmVSZY/KpvSi0kS5U0fPUBfKyDLwFnkTmtlsYsfshEkWWRuh/r161/41Q0uYRFX/rqLyX5Wuc/IWeRMm0abZm6h50Z2I2N/Xz4T1EwBYee5K6t+pd+mZIzIZu3gsAAsmLaDpiyZlMHnUFLA5E3MY8aqaWW/h9IVE1kRAbzP2NMg7OI+hjw5N6NHKqPrypgMaFBxVwIA7Byj9oIVYzZaaMUpX72Hh8YX0vq43AIsOXaQMPK2tfx26nNiF7hd3x2q1WH7y8kS/jl50chHFpxUTb4iz+uerE/06x1h0ShGFxxQSrYqy4XcbEu2cPopOLSJvSh6t5a1svXtr8thS+s8emU3L5hYqHq1wnTsaFP24iOCgIJH1bbMqgmsK3aKTi/D38hNeHab29VrXFL0AXU/piq/YR/OXzdS/X582/W7RKUV4872EloVo+l+Ta1peTdMoOrkII9MgtDREaHEobSrgopOK0H06ocUhwqvCae27nNgFTddoWtREy8YW15TGmkej8GiVfLppURPR8qhrimAtQ0uYuqHFIWLVMVf/RtAgZ7wKQwwtDRGvj7vOz8gyErnOmlc0YzabaXpwUBCA8KowVqvl2r+RbeDvpX43I+siajbMVD3LwFeskmu3bG5J3FvOYmQaePPV725rRas67JTzN4IGRqaRyNHWXtczdHSfrkyEFivtvdcMDc1QU4on9u308QPAo3vIzsjuMNm2E7XSEV7Dy11H3tWpnuXLSjw0dkRxVjGvnvFqp3q//H58fN7Hneoju41k6c+XdqpP7T01YYh0xPGDj6f+2vpO9Z+O/Ck/HflTl4FlYydMvAtHX8g5B5zjzodl2xQECgC4bNxlnDXirDQTKt+vfhevnHBlQk9NxOtE4lw96WrOPuBsl57KNZOv4ewRZ7uS9zoGG6jk9OeMOMelpxqA1065lnMOOMeVNDjPn5fQfzP5N1SEKlznX5xVnNCvnng1VeEql94nr09Cv3LCldRF6lz7d4ZpApw78lyao82JtgDjuo9L6KcNO42YFQOSSZ4P7KZmfNQ1nR8P+XFaQmzHwPQaXo4ZcEya3jdfzQjp9/g5pO8haQm3u2er7yUBT4BJPSel6V2CXQCVb25Uyag03YmyyvRmsl/Rfml60BtM6P3y+6XpGYb6jhPwBhLmcGod597ze/yJ+yg18beuqe8wHt2D3+NPO3+H1Ps1df8OrWarihRLaZ9qXje2NrK9eburfXOsOaFvb97OxvqNrvbhWDihb27YzIqqFa6k5pFYJKGvrFnJwm0LXeeWGtH4xbYvWFWzytU+Zsb4ISLDzQRBEISvxLKShtCxx6rcRb16wemnqyFp+++/Z4/vh4IVt7CjttuEMm38PdselNdHiNXGVHlUmUy6TydvmvqCXvdOHa1lrS4Dy5PvodvZ3QAou7+Mlk0tCQ1bRWf1/KX6z+qGWRuIlkfVlytLDRPM2j+LnlcpffWlq4nVxMACbKXnTspN6Mt/shwzZKq2bX0UHFmQ0BcdvAjbtLGttodqC4pOLaLnL3tiRkwWTl6Y6NdZl15USo/LexCtirJw0kLXsWFDr+t60f3i7kQ2RFgwcUGiX0fvf2d/Ss4rIbQ4xIJJC1zHZls2Qx9LH8qYyvAXhtPlhC7UvFbD0mPTHywPePsA8g/NZ/u/t7PitBVp+qjPRpEzLofyv5ezeubqNH3sirFkDs1ky91bWPerdWn6hC0T8Pfws/HmjWyctTFNn1I/BU+uh3W/WceWO7ak6dPj09EMjdU/X035A+UuTQ/oTAtPA2DFWSvY/vh2l+7t6mVypfqv+dITllLzUjuDsr+fCWuVQbnokEXUv+d+cM4amcWYhSrq/YuxX9A0v8ml507J5cCP1MPb/4b+j/DKsEsvOLqAEa8pA3Nuj7lEy9wzsRWdWsSwf6tEzB/lfoTZ6J52vdv53Rjy9yEAvK+/nxZh2OPKHgy4ewBms8lHWR/Rnt6/603fP/QlWhllbre5aXq/2/vR6ze9CK8N87+B/0vTB943kO4/705oaYhF0xapCEFfMsKw/x396XJCF0KLQ6z++Wo0T3IIOHgH8gAAH3dJREFUrO7V6fXbXuROyCW0OMTWP291DZHVvTolF5UQHBCk+ctmal6qSdMLjy/EV+QjsjFC8+LmND3rwCyMoKEmKaiOpemefA+ariUetn4o5pcgCMK+iuQkEgRBEL4VQiF48UUVYfTmm2Cayix6ovN/7ArCPo8TDWJbKSYUqAd6Q1MGXqudjBpp+3qlB3V0j44VtZRBZqf899UGT55Ksm5GTBWtktIWWxkxuldX+bDq4q5jAcjomYHu0RNDLJ12jh4YEEAzNKKVURXp06595vBMNE2jZUuL0tvaOnk/skerKJDIugix6pjr/DSPRs44FSnUvLzZvX9Ay9DInaD+A960oIlYbbJ/bJU3LHeS0us/rleRRim6p8BD3lRlcNa+VYvZaLr27+vmSxigVS9UYYWtpMFog7+3n7zpSq/4VwV21E5efxuCg4OJ9mUPlLn2bds2WQdkkTc1DytmUXZvup4zIYe8KXmYzSZlfytzv/c25B2SR+7EXGK1Mcr+Wua+9jYUzigkZ1wOLVta2PLHLa4IRTtmU3ppKXlT8ggtCbHuqnXYcdtVZ8DdA8ibnkfdO3Ws/NlKV/SjHbMZ8foI8qbnUflEJV+e+WXaPT3q81HkjMmh/MFyVl+UblCOWzmO4OAgW+7awrqr0w3KiVsnktE9g403bWTjjRvTTKTx68bjyfGw+fbNVDxakaaP/HAkmqZRdm8ZdW/XuXQjy2DQ31SOm4rHKmj6oskVJebJ89BnVh8Atv1jG+Ev3VFyvq6+hPlc/lA5LRtbXJFcvu4+ul+sojvKHywnui3qijLz9/HT7SxlnpfPKSdWF3NF+AUGBCj6cVHi3rHCySg3gOCQIIVHFSbaWzHL1T5zWCZ50/OwbZttD21LHLdzfpn7Z5IzLgcrarH9ye0uHSDrgCyyRmRhhk2qX6xOvimOPjKLzCGZxBvj1L5Rm/beZY/JJtAvQKw2Rt27dcnmbUZf9rhs/D39RKuiNHzckNZ/zoQcMrpl0FrRqiIc2+m5k3LxFnppLWsltCiUrk/OxZProWVzC83Lm9P0vKl5GJkGkY0RIqsj6fq0PPQMncj6CC0bWtL03Gm56B6d8NowrVtb087fiYAMrwoTrXCbyxiQN0V9LjSvbCZW5Y4k0X16IkKyeXkzsTq3bgSMxOdmaFkozZxOjaAMLQmpCMoUPDkeMoep0O3Q4lAyStHR8zwEB6vIoaaFTeofOil4C70E+geUvqAp8bfKwVfkw9/bj23bhBa0vTcp/q63qxd/Dz+2aRNaEqI9vm4+MkoysGIW4RXhdL3Uh6/Ih9lidvjeZXTPwFvgxYyYRNZF0tpn9MjAm+fFbDZp2dSSrvfMwJPtId4Ud7+3bf37e/kTOSSj26Lp7XtnYPgN4g1xotvTdX9vP7pPJ1avUhZoXi0Rsfp9QEwiQRAE4Vtn+3Z4+mkoKoKf/ASamuDkk5PL15khTRAEQdh92KaN1WqlmUi+Yh96hk60Okrr5tY0PXdyLkamQfPKZkILQml6yfklGJkGde/XUf9ufZo+4O4B6Bkq11rNKzVu3bQZ+fZIADbevJGqZ6tcuu7XGb9qPAArL1hJ9XPVboOwxJfQl520TA3lhISBFxwUZOwSNYx20SGLqP+wPqFjQ/bobEZ/PhqA+aPmE1rofhjOOyiPke+p4/ts4GdE1rofZguPK2T/l1QI7SfdPiFW6TYKup7Rlf0eV0OgPsz8UJlIKZRcVMLgBwZj2zYf6O5Z5gB6XNWDAXcOIN4Y5+Pc9KFxfW7sQ59ZfWgta2Vej3lpev+7+tPzVz0JrwrzvyHpUWyDHhxE6cxSGuc3smDsgjR96BNDKT69mLr36lh8yOI0ffjLw+kyowvVL1Wz7IRlafrI90cqg/LxSr48K92gHP3FaLJHZVM+p5zVF3dgUK4eR3BgkM13bmb9r9en6RPLJ5JRksGGWRvYdHP60L8pTVPwZHlY+6u1ahhxOw6yDwJg1cxVbPv7NpdmZBtMbVQJ3FecvoLtT7kjKH2lPiaVTQJgyYwl1L7qNuECgwKJe3Ph9IU0fNjg0rNGZzFmvnpG/6b33tySuWkml+vey/pQDfFOoeTCEgbPkXtvZ++91Pfz+4CYRIIgCMJ3ztKlcMopsGqVSoB9zDFqONqMGeD//vwjRhAEQfiekGo+pUbZ6V41xtpqTYlQQ9XDUBMtAMSb4slhqm265k1OCBCtjqZFCOoBHU+OB9u2VbSD7T4OI1vly7ItW0VTpLS3bRtvvhdvoRcrZtGyviV5Hm34inxKb7U6jNbwlfjw5reL5kh5JMzoqaI54qE4LevS+w/0DeDJ9RCrjyX2n9o+MCiAJ9tDrCZGZH16/5nDMjEyDaKVURXl1a7/rAOyMAIGrWWtyWiSlPbZY7PRfTotm1pUvrF2eu7kXDRD6ziSyIb8g9V/sJpXNqdFm2iGlogwDC0LpRmAmk9LRDg2LWpKRGg6GAGD3MkqQrLx88ZEBGhCzzbInaj0hrkNmE3tIonyPYkIzfoP6jHDbt1b5CVnjNLr3qlTudpS8JX4EpNV1Lxeo3K1peDv5SdrRBa2bVPzSk3aMNtA/wCZwzKxYha1r6VHoQUGB8gckokZMTuMUsscnklwYJB4U5y6t9qi1FL2kTUqi0Df9Cg2h5xxOfh7+Yluj6o8fe3InZJLRmkGrWWtiSi31Hsn/5B8fF3VMNrGeY1p7QuOLMBb4CW8JuyOgmuj8LhCPDkempc307SgCU+Ohy4ndEmrt68iJpEgCIKwR7Bt+OILNRztqaegogKWLFF5i+rqICcHDOOr+xEEQRAEQRAEYfewsybRHpyXRhAEQfg+omkwZgzcfTds3QoffphMbP2LX6iE11ddpYykvej/FIIgCIIgCILwg0dMIkEQBOFbwzBg6tTk65NPhrFj4a9/VUbS0KFw33177vgEQRAEQRAEQUgiJpEgCILwnXH88fDCC2oI2pw5UFwMGzcqzTThwQehqmqPHqIgCIIgCIIg/GCRnESCIAjCHsWyQNfhk09gyhQVfXTEESrh9Y9+BJmZe/oIBUEQBEEQBGHfRnISCYIgCPsEettfosmTVYLrq6+GZcvgrLOga1c1Y5ogCIIgCIIgCN8+YhIJgiAIew377w+33aaGoH3wAVx4ocpbBKr8ssvg008l4bUgCIIgCIIgfBuISSQIgiDsdeg6TJumZkjzeFTZtm3w8MMwcSIMHAg33ACrVu3Z4xQEQRAEQRCE7xNiEgmCIAj7BPfcA5WV8Mgj0Lcv/N//wc03J/Xt2/fcsQmCIAiCIAjC9wExiQRBEIR9hpwcOPdceOst2LoV/vAHVb58OZSUwGGHKROpoWGPHqYgCIIgCIIg7JOISSQIgiDsk5SUQL9+ajs/H377W9iwAc47D4qL4ZRTYPPmPXuMgiAIgiAIgrAvISaRIAiCsM9TWqqGnq1dC/PmwcyZ8PnnkJen9LffVomwLWvPHqcgCIIgCIIg7M1o9l40RcyYMWPs+fPn7+nDEARBEL4H2DZomtqeNg0++gh69oTTT1fL4MEQCOzZYxQEQRAEQRCE7wJN076wbXvMV9YTk0gQBEH4vtPcDC++CI8/Dv/9L5gmnHMOPPaY0vv2Ba8XsrKSy4knqogk04RZs9xaVhYMHw5Dhih9wwbIzlblwWDSnBIEQRAEQRCEvYGdNYk838XBCIIgCMKeJDMTzjhDLVVV8Oqr0L270mxbJbwOhZJLVRXU1yu9uRluu02ZQanMmgU33qhmVRs4MFmuaWp/s2fDZZfBli1w5pnpJtMZZ8CECVBdDa+8kq737Qu5uckhcroMEBcEQRAEQRC+ZcQkEgRBEH5QFBWpGdIcNA0eeqjz+jk5EItBa6sykJqa1LqwUOnZ2fDPf7pNpqYmFWkEqq1hKONpw4ZknXHjlEm0ejX87Gfp+336aZV8+5134MgjlfHkGEjZ2XDffar955/D/fenm0ynngrdusG2bbBuXbqemSkRT4IgCIIgCIIbMYkEQRAE4SvQNPD71dKli1vLyoKzz+68bb9+8N57neujR7vNI8eEGjdO6X36wO9/79ZCITWsDZQJ9NZbSd2JeJo8WZlEL78MF12Uvt8vv1TD5R56CP74R7eBlJ0NDzwABQXw0kvw+uvg8Sizy1nffDNkZMCbbyqjKlX3eFQUFcAnn6iE4qm63w8zZih9yRJloDntDEPlijrgAKVv2QKRiHvfGRnK7AMIh9XaMJKLmF+CIAiCIAhfD8lJJAiCIAjfE2wbolFlGOXkqDxL5eWwYkW6yTRzppr97dVX4Ykn3FoopMyd/Hy4/Xb4058gHleLaap1XZ0yc37xC/jLX9zHYRiqDsB558Ejj7j1vDzVHlTE0zPPuPUePZQ5BHD00fDGG2596FB1TgBTpqhjTWX8ePj006S+YoXbZJo2TZ0zwFFHqWuUqh90kBpiCPCTn6hr0153TLCLL1bXxNEMA6ZOhZNPVkMFZ81yG1gej4oAmzZNRaf94x9uzTBg5EgViRYOKwMwVTMMGDRIXaNIBJYtSzfwSkrU+x+NquGM7fsPBNS28xVQTDVBEARB+P4jiasFQRAEQfjWsSy3eeSsnYir6mpobEyWO+bR/vur9cqVKq9TalufDw49VOkffABlZe7+c3KUuQTw5JPKUHI001QmiRM99cc/wtat7v6HDoWrr1b6pZeqaKxUfdIkZe6AMpFqatz7P+64pInUv78ya1L188+Hu+5SJo3fnzRjHK65RrWvqUmPTAO45Rb47W9h40aVm6o999wDV1yhDCLnOqby8MPKnJs3T51Le5yhjG++qYYytjexnn9e5el67TVlJrbXn3gCRo1SubRuuindhHroIXVdXn0V5sxRZbqeXP/lL9C1q9Kfe06Vpeq3364i2l5/Hd59160Zhoqs83jU8S9Y4Na93qSB9/77sGaNu63fr84dlJG4bZu7fWYmTJ+u9CVLVG4yR3N0ZyjpunXQ0uLWg0EoLVV6VZX6/Ujt3+dLRgHG4+o8BEEQBOG7QBJXC4IgCILwreM8+HZGly4dGyEOQ4aopTOcB/bOOP30Heu/+c2O9Xvv3bHePoqpPevWda75fMoksO2kgZVqDOTnK5PC0Ry9oEDpJSXwxRduzTSVAQPQq5caTpiqmaaKpAI11HHOnHQTb8SIpH7DDe62pqmilACKi1UkV/vjy8pSujPsL1WPxZKRSU1NysCzLKVZllpiMaVv3Ahvv52u/+EPSv/0U5V7q73+298q/aWX0t8/ny9pEj3yiMoXlkpBQdIkuuMOZYil0ru3Oi5QRuJbb7n14cNh6VK1fdZZyYg1h4kTYe5ctX3wwbB8uVs/4gg1wyLAgAHKIC0oSC6HH548v7/8RZ1PQYG6VwoKVML9bt0QBEEQhG8NiSQSBEEQBEEQ9jlSh0CmmkiOyVZXp2YnTDWZIGmybdqkIoVS23q9cOCBSl+0SEV7OZppuiON3n1XRcql6l26KGMNVMSWE4Xm6L16wUknKf0vf4HNm9Vx1taqZcoU+L//U3ogoCKVUpk5Ex58UPVXUqJmQHQMpPx8OPFEZYLFYirKztFSF6/323k/BEEQhL0bGW4mCIIgCIIgCPsoLS1uA6muTg1lGzNG5bP65S/T9QsvVNFzFRXKRGrP7Nlw7bUqwmvGDHeUkhNlNXYsNDSohPROeX6+GuYp+asEQRD2XWS4mSAIgiAIgiDso/j9yujpyOzJyFBD8TqjSxc1q6BjHjlG0oQJSrcsle+qtlblbaqtVVFPw4Ypk2j5cjX0LRXDUEnmTzwR5s9XQxVTTaSCAvjxj6FnT7XPykpVnp+/4yGpgiAIwt6FmESCIAiCIAiC8D3C41HD6pyhde3p3RteeCG93BmSN3y4ShqfajLV1SXzh0UiKp/SqlVKq69X5aNGKZPotddUziaHrCxlFr32mur7nXfgqafSI5mOOkrVDYXUsQSDKu+ZpkkUkyAIwneFmESCIAiCIAiCIKDrap2TA9OmdV5v6lQVTeRgmmqImpPUfMoUNQteahRTbW0yX9SmTWp2u9paNXTOYetW1cfdd6tIpfY0NkJ2tpoh8M9/ThpIzrqxUa1/9SuVtDxVz8pSUVOgZgd86SW3XlwMH3+s9MsuUyZZqt6nTzLR+SWXwMKFyVntNA0GD1Yz+4GaNXHNGrc+YoQa7uf0X17u1kePTibav+KK5Lk4+vjxKicVqPOLRt3m2aRJcNppavvKK5PXzNEPOghOOEENY3SSo6dyxBFqtsPGxmTy+NT2M2aoe6KqSs3e2F4/8UQYN07NRpmaUN7RTz0VDjgANmyAf/wjXT/zTHUNV61S+bTac+656j1YuhT+85/09hdcoKLuFixQMyO21y+6CAoL4bPPVD6x9lx2mbq3Pv4YPvkkXb/yShXB99577nvf4de/Vus331QzI6bu3+eDyy9Xr199VZ1jKpmZyRk5X3xRXaNU8vPhpz9V2889p65xKl27Jt/7p55S71HquXfvrt4fgMcfT5q6Dn36wLHHqu3HHoNw2K0PHKhmvAQ1e6Yz+YDD0KHJXG1z5pDGiBEqqX80qvpvz6hR6v5vbu74vR8/Xs3k2dAAzz6brk+erAzs6mr1e92e6dOVYV5R0fFkFIceqsztLVvS742cnOS1+yEhJpEgCIIgCIIgCF8bw0gaQKAilXr37rz+eeepBVRUkmMiFRersiOOUIm7IxE1O6CTHNwZtjZtmjJPnHKnjvNQPHaseiBN1VOHvA0dqh44nXa2rZKAO5SWqgfj1H0XFSX1YFAZCql6aqRTOKxm90vVa2qS+qZNaha91GPPz0/qn36qIrVS9YyMpP7888loq9T3wDEKHn1UrVNTzwYCyiSKxVTy8/YUFiqTqLk5OZQxtX337uq619Upg6693r+/MokqK+HOO9P1/fdXJtGmTXDrren62LHKJFq9Gm66Kf34DjooaRLNmpWuH3OMMok+/xx+97t0/eST1Tl+/DFcf326fu656j19++2O93/JJeo9eOUV+NOf0nXHJHr22aRZ6JCVlTSJ/vUvZeSkUlKSNIn+/ne1j1QGDUqaRH/9qzIwUxk1Kvne33GHMspSmT49aXTcdFPSLHWYMSNpEl17rTJTUjn99KRJdOWV6t5L5cILkybRxReTxq9+pUyi1lZVtz2zZimTqKEhaYSmcued6v6pqFBmYHvmzFEm0aZNcP756foTT6j7c+VK+NnP0vWXXlIm0aJF6j5IZdCgH6ZJJImrBUEQBEEQBEEQhL0W21ZGnG0nzaXUx1gn4ip1JsNU3etVujMjYnt8PqXHYqpOe/x+pbe2puu2nYyia2lJ6qn7z85W63C44/5zctQ6FErXdT2pNzam64aRNDmdGRtT9+3xQF6e2q6tdZ+/Y6A6elWV23wEZY45emWlu29QBqSz/23b0s8tGFS6ZXWsZ2er8zPNjvXcXFUnFks3sEAZrFlZyhjuSC8sVNFaLS0d6127qmMMh9X5peLxKAPp+4LMbiYIgiAIgiAIgiAIgiDstEmkfxcHIwiCIAiCIAiCIAiCIOzdiEkkCIIgCIIgCIIgCIIgiEkkCIIgCIIgCIIgCIIgiEkkCIIgCIIgCIIgCIIgICaRIAiCIAiCIAiCIAiCgJhEgiAIgiAIgiAIgiAIAmISCYIgCIIgCIIgCIIgCIhJJAiCIAiCIAiCIAiCICAmkSAIgiAIgiD8//buP9bXgq4D+PvdvanxY6CJVlwmqIy6OgFjzKRci2pYTKzZ0pTRj61/tNTcCqtVa62x5bK2WMrUoMm0RjKZI4Wo0doyIeKHgCajkksYt1Xkjxkin/44X9tNuQXn3HOf++W8XtvZ+T7Pee7zvL/bZ+f7nPf3+T4XAIiSCAAAAIAoiQAAAACIkggAAACAKIkAAAAAiJIIAAAAgCiJAAAAAIiSCAAAAIAoiQAAAACIkggAAACAKIkAAAAAiJIIAAAAgCiJAAAAAIiSCAAAAIAoiQAAAACIkggAAACAKIkAAAAAiJIIAAAAgCiJAAAAAIiSCAAAAIAoiQAAAACIkggAAACAKIkAAAAAiJIIAAAAgGxzSdT2vLafbHtP24u381gAAAAAbN62lURtdyW5NMnLk+xN8pq2e7freAAAAABs3nZeSXR2kntm5t6ZeTjJ+5NcsI3HAwAAAGCTtrMkOjHJfQcs71ut+1/a/nTbm9vevH///m2MAwAAAMDBLH7j6pm5bGbOmpmzTjjhhKXjAAAAAOxI21kS3Z/kpAOW96zWAQAAAHCE2c6S6KYkp7Y9pe1Tkrw6yTXbeDwAAAAANmn3du14Zh5p+4YkH0myK8l7ZubO7ToeAAAAAJu3bSVRkszMtUmu3c5jAAAAALB1i9+4GgAAAIDlKYkAAAAAUBIBAAAAoCQCAAAAIEoiAAAAAKIkAgAAACBJZ2bpDP+j7f4k/7R0jkPgmUn+dekQrDUzxFaZIbbKDLFVZoitMkNslRliq55MM/ScmTnh/9voiCqJniza3jwzZy2dg/VlhtgqM8RWmSG2ygyxVWaIrTJDbNVOnCEfNwMAAABASQQAAACAkmi7XLZ0ANaeGWKrzBBbZYbYKjPEVpkhtsoMsVU7bobckwgAAAAAVxIBAAAAoCQ65Nqe1/aTbe9pe/HSeVgvbU9q+xdt72p7Z9s3Lp2J9dN2V9u/a/uhpbOwntoe3/aqtp9oe3fb71g6E+ul7ZtXr2Mfb/u+tk9bOhNHtrbvaftg248fsO4Zba9v+6nV96cvmZEj20Fm6LdWr2W3t7267fFLZuTI9lgzdMDP3tJ22j5ziWyHk5LoEGq7K8mlSV6eZG+S17Tdu2wq1swjSd4yM3uTvCTJ680Qm/DGJHcvHYK19rtJPjwz35rk9JgnnoC2Jyb52SRnzcwLk+xK8uplU7EGLk9y3letuzjJDTNzapIbVstwMJfna2fo+iQvnJkXJfn7JG893KFYK5fna2cobU9K8v1JPn24Ay1BSXRonZ3knpm5d2YeTvL+JBcsnIk1MjMPzMwtq8efzcYfZicum4p10nZPkh9M8q6ls7Ce2h6X5GVJ3p0kM/PwzPzHsqlYQ7uTfEPb3UmOSvLPC+fhCDczf5nk375q9QVJrlg9viLJKw9rKNbKY83QzFw3M4+sFj+aZM9hD8baOMjvoSR5e5KfT7IjbuisJDq0Tkxy3wHL++IPfDap7clJzkzyN8smYc38TjZexB5dOghr65Qk+5P8wepji+9qe/TSoVgfM3N/krdl4x3XB5I8NDPXLZuKNfXsmXlg9fgzSZ69ZBjW3k8m+dOlQ7Be2l6Q5P6ZuW3pLIeLkgiOQG2PSfInSd40M/+5dB7WQ9vzkzw4M3+7dBbW2u4kL07y+zNzZpLPx0c8eAJW9425IBuF47ckObrt65ZNxbqbjf+SeUe8i8+h1/aXsnFbhyuXzsL6aHtUkl9M8itLZzmclESH1v1JTjpgec9qHTxubb8+GwXRlTPzgaXzsFbOSfKKtv+YjY+7fk/b9y4biTW0L8m+mfnKVYxXZaM0gsfre5P8w8zsn5kvJflAkpcunIn19C9tvzlJVt8fXDgPa6jtjyc5P8lrV2UjPF7Py8YbHretzq/3JLml7TctmmqbKYkOrZuSnNr2lLZPycZNGq9ZOBNrpG2zcR+Qu2fmt5fOw3qZmbfOzJ6ZOTkbv3/+fGa8e88TMjOfSXJf29NWq85NcteCkVg/n07ykrZHrV7Xzo2bn7M51yS5aPX4oiQfXDALa6jtedn4GP4rZuYLS+dhvczMHTPzrJk5eXV+vS/Ji1fnSk9aSqJDaHVTtDck+Ug2Tob+eGbuXDYVa+acJBdm4wqQW1dfP7B0KGDH+ZkkV7a9PckZSX5z4TyskdVVaFcluSXJHdk437xs0VAc8dq+L8lfJzmt7b62P5XkkiTf1/ZT2bhC7ZIlM3JkO8gM/V6SY5NcvzqvfseiITmiHWSGdpy64g4AAAAAVxIBAAAAoCQCAAAAQEkEAAAAQJREAAAAAERJBAAAAECURAAAh0zb7277oaVzAABshpIIAAAAACURALDztH1d24+1vbXtO9vuavu5tm9ve2fbG9qesNr2jLYfbXt726vbPn21/vlt/6ztbW1vafu81e6PaXtV20+0vbJtV9tf0vau1X7ettBTBwA4KCURALCjtP22JD+a5JyZOSPJl5O8NsnRSW6emRckuTHJr67+yR8m+YWZeVGSOw5Yf2WSS2fm9CQvTfLAav2ZSd6UZG+S5yY5p+03JvmhJC9Y7ec3tvdZAgA8cUoiAGCnOTfJtye5qe2tq+XnJnk0yR+ttnlvku9se1yS42fmxtX6K5K8rO2xSU6cmauTZGa+ODNfWG3zsZnZNzOPJrk1yclJHkryxSTvbvvDSb6yLQDAEUNJBADsNE1yxcycsfo6bWZ+7TG2m03u/78OePzlJLtn5pEkZye5Ksn5ST68yX0DAGwbJREAsNPckORVbZ+VJG2f0fY52TgvetVqmx9L8lcz81CSf2/7Xav1Fya5cWY+m2Rf21eu9vHUtkcd7IBtj0ly3Mxcm+TNSU7fjicGALAVu5cOAABwOM3MXW1/Ocl1bb8uyZeSvD7J55OcvfrZg9m4b1GSXJTkHasS6N4kP7Faf2GSd7b99dU+fuT/OOyxST7Y9mnZuJLp5w7x0wIA2LLObPZKagCAJ4+2n5uZY5bOAQCwFB83AwAAAMCVRAAAAAC4kggAAACAKIkAAAAAiJIIAAAAgCiJAAAAAIiSCAAAAIAoiQAAAABI8t9cJfvKNrGeZQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
      " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure(figsize=(20,20))\n", + "plt.xlabel('epochs')\n", + "plt.ylabel('loss')\n", + "plt.title('Evolution of training and testing losses')\n", + "x = range(15)\n", + "h1, = plt.plot(x, test_1, 'c', label='test loss Linear')\n", + "h2, = plt.plot(x, train_1, 'c--', label='train loss Linear')\n", + "h3, = plt.plot(x, test_1a, 'b', label='test loss Linear Adam')\n", + "h4, = plt.plot(x, train_1a, 'b--', label='train loss Linear Adam')\n", + "h5, = plt.plot(x, test_2, 'r', label='test loss MLP')\n", + "h6, = plt.plot(x, train_2, 'r--', label='train loss MLP')\n", + "h7, = plt.plot(x, test_2a, 'm', label='test loss MLP Adam')\n", + "h8, = plt.plot(x, train_2a, 'm--', label='train loss MLP Adam')\n", + "h9, = plt.plot(x, test_3a, 'g', label='test loss ResNet Adam')\n", + "h10, = plt.plot(x, train_3a, 'g--', label='train loss ResNet Adam')\n", + "l = plt.legend(handles=[h1, h2, h3, h4, h5, h6, h7, h8, h9, h10])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Acknowledgement\n", + "\n", + "This tutorial is inspired by some examples from [xlvector/github](https://github.com/xlvector/)." + ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { - "display_name": "Python [Root]", + "display_name": "Python 3", "language": "python", - "name": "Python [Root]" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.12" + "pygments_lexer": "ipython3", + "version": "3.6.4" } }, "nbformat": 4, - "nbformat_minor": 1 + "nbformat_minor": 2 } diff --git a/example/recommenders/demo1-MF2-fancy.ipynb b/example/recommenders/demo1-MF2-fancy.ipynb deleted file mode 100644 index d5ac68bd2..000000000 --- a/example/recommenders/demo1-MF2-fancy.ipynb +++ /dev/null @@ -1,295 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Matrix Factorization (MF) part 2: Getting Fancy\n", - "Demonstrates matrix factorization with MXNet on the [MovieLens 100k](http://grouplens.org/datasets/movielens/100k/) dataset. This is an extension of [part 1](demo1-MF.ipynb) where we try fancy optimizers and network structures.\n", - "\n", - "You need to have python package pandas and bokeh installed (pip install pandas bokeh)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import mxnet as mx\n", - "from movielens_data import get_data_iter, max_id\n", - "from matrix_fact import train" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# If MXNet is not compiled with GPU support (e.g. on OSX), set to [mx.cpu(0)]\n", - "# Can be changed to [mx.gpu(0), mx.gpu(1), ..., mx.gpu(N-1)] if there are N GPUs\n", - "ctx = [mx.gpu(0)]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "train_test_data = get_data_iter(batch_size=100)\n", - "max_user, max_item = max_id('./ml-100k/u.data')\n", - "(max_user, max_item)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Linear MF\n", - "Same as before, but this time with the [Adam optimizer](https://arxiv.org/abs/1412.6980) which will often converge much faster than SGD w/ momentum as we used before. You should see this model over-fitting quickly. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "def plain_net(k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " score = mx.symbol.Variable('score')\n", - " # user feature lookup\n", - " user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) \n", - " # item feature lookup\n", - " item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " # predict by the inner product, which is elementwise product and then sum\n", - " pred = user * item\n", - " pred = mx.symbol.sum_axis(data = pred, axis = 1)\n", - " pred = mx.symbol.Flatten(data = pred)\n", - " # loss layer\n", - " pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)\n", - " return pred\n", - "\n", - "net1 = plain_net(64)\n", - "mx.viz.plot_network(net1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "results1 = train(net1, train_test_data, num_epoch=25, learning_rate=0.001, optimizer='adam', ctx=ctx)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Neural Network (non-linear) MF\n", - "The non-linear model converges strangely with Adam." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "def get_one_layer_mlp(hidden, k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " score = mx.symbol.Variable('score')\n", - " # user latent features\n", - " user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)\n", - " user = mx.symbol.Activation(data = user, act_type='relu')\n", - " user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)\n", - " # item latent features\n", - " item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " item = mx.symbol.Activation(data = item, act_type='relu')\n", - " item = mx.symbol.FullyConnected(data = item, num_hidden = hidden)\n", - " # predict by the inner product\n", - " pred = user * item\n", - " pred = mx.symbol.sum_axis(data = pred, axis = 1)\n", - " pred = mx.symbol.Flatten(data = pred)\n", - " # loss layer\n", - " pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)\n", - " return pred\n", - "\n", - "net2 = get_one_layer_mlp(64, 64)\n", - "mx.viz.plot_network(net2)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "results2 = train(net2, train_test_data, num_epoch=20, learning_rate=0.001, optimizer='adam', ctx=ctx)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Deep Neural Network (Residual Network / ResNet)\n", - "Borrowing ideas from [Deep Residual Learning for Image Recognition (He, et al.)](https://arxiv.org/abs/1512.03385) to build a complex deep network that is aggressively regularized to avoid over-fitting, but still achieves good performance. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "def get_multi_layer_dropout_resnet(hidden, k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " score = mx.symbol.Variable('score')\n", - " # user latent features\n", - " user1 = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)\n", - " user = mx.symbol.FullyConnected(data = user1, num_hidden = hidden)\n", - " user = mx.symbol.Activation(data = user, act_type='relu')\n", - " user = mx.symbol.Dropout(data=user, p=0.5)\n", - " user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)\n", - " user2 = user + user1\n", - " user2 = mx.symbol.Dropout(data=user2, p=0.5)\n", - " user = mx.symbol.FullyConnected(data = user2, num_hidden = hidden)\n", - " user = mx.symbol.Activation(data = user, act_type='relu')\n", - " user = mx.symbol.Dropout(data=user, p=0.5)\n", - " user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)\n", - " user = user + user2\n", - " # item latent features\n", - " item1 = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " item = mx.symbol.FullyConnected(data = item1, num_hidden = hidden)\n", - " item = mx.symbol.Activation(data = item, act_type='relu')\n", - " item = mx.symbol.Dropout(data=item, p=0.5) \n", - " item = mx.symbol.FullyConnected(data=item, num_hidden = hidden)\n", - " item2 = item + item1\n", - " item2 = mx.symbol.Dropout(data=item2, p=0.5) \n", - " item = mx.symbol.FullyConnected(data = item2, num_hidden = hidden)\n", - " item = mx.symbol.Activation(data = item, act_type='relu')\n", - " item = mx.symbol.Dropout(data=item, p=0.5) \n", - " item = mx.symbol.FullyConnected(data=item, num_hidden = hidden)\n", - " item = item + item2\n", - " # predict by the inner product\n", - " pred = user * item\n", - " pred = mx.symbol.sum_axis(data = pred, axis = 1)\n", - " pred = mx.symbol.Flatten(data = pred)\n", - " # loss layer\n", - " pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)\n", - " return pred\n", - "net3 = get_multi_layer_dropout_resnet(64, 64)\n", - "mx.viz.plot_network(net3)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# Larger batch size makes GPU more efficient for this complex model\n", - "train_test_data2 = get_data_iter(batch_size=200) \n", - "results3 = train(net3, train_test_data2, num_epoch=25, learning_rate=0.001, optimizer='adam', ctx=ctx)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Visualizing results\n", - "Compare accuracy and training time across the models." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import bokeh\n", - "import bokeh.io\n", - "import bokeh.plotting\n", - "bokeh.io.output_notebook()\n", - "import pandas as pd\n", - "\n", - "def viz_lines(fig, results, legend, color):\n", - " df = pd.DataFrame(results._data['eval'])\n", - " fig.line(df.elapsed,df.RMSE, color=color, legend=legend, line_width=2)\n", - " df = pd.DataFrame(results._data['train'])\n", - " fig.line(df.elapsed,df.RMSE, color=color, line_dash='dotted', alpha=0.1)\n", - "\n", - "fig = bokeh.plotting.Figure(x_axis_type='datetime', x_axis_label='Training time', y_axis_label='RMSE')\n", - "viz_lines(fig, results1, \"Linear MF\", \"orange\")\n", - "viz_lines(fig, results2, \"MLP\", \"blue\")\n", - "viz_lines(fig, results3, \"ResNet\", \"red\")\n", - "\n", - "bokeh.io.show(fig)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python [Root]", - "language": "python", - "name": "Python [Root]" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.12" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/example/recommenders/demo2-binary.ipynb b/example/recommenders/demo2-binary.ipynb deleted file mode 100644 index 1a3fca1fa..000000000 --- a/example/recommenders/demo2-binary.ipynb +++ /dev/null @@ -1,137 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Binary Predictions with Negative Sampling\n", - "Example of a recommender system making binary predictions instead of predicting a rating.\n", - "Demonstrates use of `NegativeSamplingDataIter` to wrap an existing data iterator with `CosineLoss`.\n", - "See [BlackOut by Shihao Ji et al](https://arxiv.org/abs/1511.06909) for more on negative sampling.\n", - "\n", - "You need to have python package pandas and bokeh installed (pip install pandas bokeh)." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import mxnet as mx\n", - "from movielens_data import get_data_iter, max_id\n", - "from matrix_fact import train\n", - "import recotools" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "# If MXNet is not compiled with GPU support (e.g. on OSX), set to [mx.cpu(0)]\n", - "# Can be changed to [mx.gpu(0), mx.gpu(1), ..., mx.gpu(N-1)] if there are N GPUs\n", - "ctx = [mx.gpu(0)]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "pos_train_data, pos_test_data = get_data_iter(batch_size=100)\n", - "max_user, max_item = max_id('./ml-100k/u.data')\n", - "(max_user, max_item)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "train_data = recotools.NegativeSamplingDataIter(pos_train_data, sample_ratio=3, positive_label=0, negative_label=1)\n", - "test_data = recotools.NegativeSamplingDataIter(pos_test_data, sample_ratio=3, positive_label=0, negative_label=1)\n", - "train_test_data = (train_data, test_data)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "def plain_net(k):\n", - " # input\n", - " user = mx.symbol.Variable('user')\n", - " item = mx.symbol.Variable('item')\n", - " label = mx.symbol.Variable('score')\n", - " # user feature lookup\n", - " user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)\n", - " # item feature lookup\n", - " item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)\n", - " # loss layer\n", - " pred = recotools.CosineLoss(a=user, b=item, label=label)\n", - " return pred\n", - "\n", - "net1 = plain_net(64)\n", - "mx.viz.plot_network(net1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "results1 = train(net1, train_test_data, num_epoch=20, learning_rate=0.02, ctx=ctx)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python [Root]", - "language": "python", - "name": "Python [Root]" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.12" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/example/recommenders/demo2-dssm.ipynb b/example/recommenders/demo2-dssm.ipynb new file mode 100644 index 000000000..49450c56e --- /dev/null +++ b/example/recommenders/demo2-dssm.ipynb @@ -0,0 +1,2210 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Content-based recommender using Deep Structured Semantic Model\n", + "\n", + "An example of how to build a Deep Structured Semantic Model (DSSM) for incorporating complex content-based features into a recommender system. See [Learning Deep Structured Semantic Models for Web Search using Clickthrough Data](https://www.microsoft.com/en-us/research/publication/learning-deep-structured-semantic-models-for-web-search-using-clickthrough-data/). This example does not attempt to provide a datasource or train a model, but merely show how to structure a complex DSSM network." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import warnings\n", + "\n", + "import mxnet as mx\n", + "from mxnet import gluon, nd, autograd, sym\n", + "import numpy as np\n", + "from sklearn.random_projection import johnson_lindenstrauss_min_dim\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Define some constants\n", + "max_user = int(1e5)\n", + "title_vocab_size = int(3e4)\n", + "query_vocab_size = int(3e4)\n", + "num_samples = int(1e4)\n", + "hidden_units = 128\n", + "epsilon_proj = 0.25\n", + "\n", + "ctx = mx.gpu() if len(mx.test_utils.list_gpus()) > 0 else mx.cpu()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Bag of words random projection" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "A previous version of this example contained a bag of word random projection example, it is kept here for reference but not used in the next example.\n", + "Random Projection is a dimension reduction technique that guarantees the disruption of the pair-wise distance between your original data point within a certain bound.\n", + "What is even more interesting is that the dimension to project onto to guarantee that bound does not depend on the original number of dimension but solely on the total number of datapoints.\n", + "You can see more explanation [in this blog post](http://jasonpunyon.com/blog/2017/12/02/fun-with-random-numbers-random-projection/)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "To keep a distance disruption ~< 25.0% of our 10000 samples we need to randomly project to at least 1414 dimensions\n" + ] + } + ], + "source": [ + "proj_dim = johnson_lindenstrauss_min_dim(num_samples, epsilon_proj)\n", + "print(\"To keep a distance disruption ~< {}% of our {} samples we need to randomly project to at least {} dimensions\".format(epsilon_proj*100, num_samples, proj_dim))" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "class BagOfWordsRandomProjection(gluon.HybridBlock):\n", + " def __init__(self, vocab_size, output_dim, random_seed=54321, pad_index=0):\n", + " \"\"\"\n", + " :param int vocab_size: number of element in the vocabulary\n", + " :param int output_dim: projection dimension\n", + " :param int ramdon_seed: seed to use to guarantee same projection\n", + " :param int pad_index: index of the vocabulary used for padding sentences\n", + " \"\"\"\n", + " super(BagOfWordsRandomProjection, self).__init__()\n", + " self._vocab_size = vocab_size\n", + " self._output_dim = output_dim\n", + " proj = self._random_unit_vecs(vocab_size=vocab_size, output_dim=output_dim, random_seed=random_seed)\n", + " # we set the projection of the padding word to 0\n", + " proj[pad_index, :] = 0\n", + " self.proj = self.params.get_constant('proj', value=proj)\n", + "\n", + " def _random_unit_vecs(self, vocab_size, output_dim, random_seed):\n", + " rs = np.random.RandomState(seed=random_seed)\n", + " W = rs.normal(size=(vocab_size, output_dim))\n", + " Wlen = np.linalg.norm(W, axis=1)\n", + " W_unit = W / Wlen[:,None]\n", + " return W_unit\n", + "\n", + " def hybrid_forward(self, F, x, proj):\n", + " \"\"\"\n", + " :param nd or sym F:\n", + " :param nd.NDArray x: index of tokens\n", + " returns the sum of the projected embeddings of each token\n", + " \"\"\"\n", + " embedded = F.Embedding(x, proj, input_dim=self._vocab_size, output_dim=self._output_dim)\n", + " return embedded.sum(axis=1)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "bowrp = BagOfWordsRandomProjection(1000, 20)\n", + "bowrp.initialize()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\n", + "[[ 0.35554492 0.0736109 -0.1220893 0.11155054 -0.20963743 0.21141198\n", + " 0.12296599 0.12428369 -0.10999548 -0.16867855 -0.09068598 0.14154953\n", + " -0.24029303 0.11956739 0.02830955 -0.14226514 -0.45963028 -0.5456747\n", + " -0.5663947 -0.10585886]\n", + " [-0.31655627 -0.13582113 -0.13815539 0.42596683 0.25674546 0.5024462\n", + " -0.3122709 0.01826438 -0.0277671 -0.14526835 0.44378105 0.09626544\n", + " 0.24572927 0.36588538 0.17922089 -0.21583243 -0.30497772 0.19484927\n", + " -0.20705326 -0.13759173]]\n", + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "bowrp(mx.nd.array([[10, 50, 100], [5, 10, 0]]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "With padding:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\n", + "[[ 0.35554492 0.0736109 -0.1220893 0.11155054 -0.20963743 0.21141198\n", + " 0.12296599 0.12428369 -0.10999548 -0.16867855 -0.09068598 0.14154953\n", + " -0.24029303 0.11956739 0.02830955 -0.14226514 -0.45963028 -0.5456747\n", + " -0.5663947 -0.10585886]\n", + " [-0.31655627 -0.13582113 -0.13815539 0.42596683 0.25674546 0.5024462\n", + " -0.3122709 0.01826438 -0.0277671 -0.14526835 0.44378105 0.09626544\n", + " 0.24572927 0.36588538 0.17922089 -0.21583243 -0.30497772 0.19484927\n", + " -0.20705326 -0.13759173]]\n", + "" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "bowrp(mx.nd.array([[10, 50, 100, 0], [5, 10, 0, 0]]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Content-based recommender / ranking system using DSSM" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For example in the search result ranking problem:\n", + "You have users, that have performed text-based searches. They were presented with results, and selected one of them.\n", + "Results are composed of a title and an image.\n", + "\n", + "Your positive examples will be the clicked items in the search results, and the negative examples are sampled from the non-clicked examples.\n", + "\n", + "The network will jointly learn embeddings for users and query text making up the \"Query\", title and image making the \"Item\" and learn how similar they are.\n", + "\n", + "After training, you can index the embeddings for your items and do a knn search with your query embeddings using the cosine similarity to return ranked items" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "proj_dim = 128" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "class DSSMRecommenderNetwork(gluon.HybridBlock):\n", + " def __init__(self, query_vocab_size, proj_dim, max_user, title_vocab_size, hidden_units, random_seed=54321, p=0.5):\n", + " super(DSSMRecommenderNetwork, self).__init__()\n", + " with self.name_scope():\n", + " \n", + " # User/Query pipeline\n", + " self.user_embedding = gluon.nn.Embedding(max_user, proj_dim)\n", + " self.user_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\")\n", + " \n", + " # Instead of bag of words, we use learned embeddings + stacked biLSTM average\n", + " self.query_text_embedding = gluon.nn.Embedding(query_vocab_size, proj_dim)\n", + " self.query_lstm = gluon.rnn.LSTM(hidden_units, 2, bidirectional=True)\n", + " self.query_text_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\") \n", + " \n", + " self.query_dropout = gluon.nn.Dropout(p)\n", + " self.query_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\")\n", + "\n", + " # Item pipeline\n", + " # Instead of bag of words, we use learned embeddings + stacked biLSTM average\n", + " self.title_embedding = gluon.nn.Embedding(title_vocab_size, proj_dim)\n", + " self.title_lstm = gluon.rnn.LSTM(hidden_units, 2, bidirectional=True)\n", + " self.title_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\")\n", + " \n", + " # You could use vgg here for example\n", + " self.image_embedding = gluon.model_zoo.vision.resnet18_v2(pretrained=False).features \n", + " self.image_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\")\n", + " \n", + " self.item_dropout = gluon.nn.Dropout(p)\n", + " self.item_mlp = gluon.nn.Dense(hidden_units, activation=\"relu\")\n", + " \n", + " def hybrid_forward(self, F, user, query_text, title, image):\n", + " # Query\n", + " user = self.user_embedding(user)\n", + " user = self.user_mlp(user)\n", + "\n", + " query_text = self.query_text_embedding(query_text)\n", + " query_text = self.query_lstm(query_text.transpose((1,0,2)))\n", + " # average the states\n", + " query_text = query_text.mean(axis=0)\n", + " query_text = self.query_text_mlp(query_text)\n", + " \n", + " query = F.concat(user, query_text)\n", + " query = self.query_dropout(query)\n", + " query = self.query_mlp(query)\n", + " \n", + " # Item\n", + " title_text = self.title_embedding(title)\n", + " title_text = self.title_lstm(title_text.transpose((1,0,2)))\n", + " # average the states\n", + " title_text = title_text.mean(axis=0)\n", + " title_text = self.title_mlp(title_text)\n", + " \n", + " image = self.image_embedding(image)\n", + " image = self.image_mlp(image)\n", + " \n", + " item = F.concat(title_text, image)\n", + " item = self.item_dropout(item)\n", + " item = self.item_mlp(item)\n", + " \n", + " # Cosine Similarity\n", + " query = query.expand_dims(axis=2)\n", + " item = item.expand_dims(axis=2)\n", + " sim = F.batch_dot(query, item, transpose_a=True) / (query.norm(axis=1) * item.norm(axis=1) + 1e-9).expand_dims(axis=2)\n", + " \n", + " return sim.squeeze(axis=2)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "collapsed": false, + "scrolled": false + }, + "outputs": [], + "source": [ + "network = DSSMRecommenderNetwork(\n", + " query_vocab_size,\n", + " proj_dim,\n", + " max_user,\n", + " title_vocab_size,\n", + " hidden_units\n", + ")\n", + "\n", + "\n", + "network.initialize(mx.init.Xavier(), ctx)\n", + "\n", + "# Load pre-trained vgg16 weights\n", + "with network.name_scope():\n", + " network.image_embedding = gluon.model_zoo.vision.resnet18_v2(pretrained=True, ctx=ctx).features" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It is quite hard to visualize the network since it is relatively complex but you can see the two-pronged structure, and the resnet18 branch" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "image/svg+xml": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "plot\n", + "\n", + "\n", + "user\n", + "\n", + "user\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding0_fwd\n", + "\n", + "dssmrecommendernetwork0_embedding0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding0_fwd->user\n", + "\n", + "\n", + "1\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense0_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense0_fwd->dssmrecommendernetwork0_embedding0_fwd\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense0_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense0_relu_fwd->dssmrecommendernetwork0_dense0_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "query_text\n", + "\n", + "query_text\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding1_fwd\n", + "\n", + "dssmrecommendernetwork0_embedding1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding1_fwd->query_text\n", + "\n", + "\n", + "30\n", + "\n", + "\n", + "dssmrecommendernetwork0_transpose0\n", + "\n", + "dssmrecommendernetwork0_transpose0\n", + "\n", + "\n", + "dssmrecommendernetwork0_transpose0->dssmrecommendernetwork0_embedding1_fwd\n", + "\n", + "\n", + "30x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape0\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape1\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape1\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape2\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape2\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape3\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape3\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape4\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape4\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape5\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape5\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape6\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape6\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape7\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape7\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape8\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape8\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape9\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape9\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape10\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape10\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape11\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape11\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape12\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape12\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape13\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape13\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape14\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape14\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape15\n", + "\n", + "dssmrecommendernetwork0_lstm0_reshape15\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape0\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape1\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape2\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape3\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape4\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape5\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape6\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape7\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape8\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape9\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape10\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape11\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape12\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape13\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape14\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0__rnn_param_concat0->dssmrecommendernetwork0_lstm0_reshape15\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_0\n", + "\n", + "dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_1\n", + "\n", + "dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_1\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0->dssmrecommendernetwork0_transpose0\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0->dssmrecommendernetwork0_lstm0__rnn_param_concat0\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0->dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_0\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm0_rnn0->dssmrecommendernetwork0_lstm0_dssmrecommendernetwork0_lstm0_h0_1\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_mean0\n", + "\n", + "dssmrecommendernetwork0_mean0\n", + "\n", + "\n", + "dssmrecommendernetwork0_mean0->dssmrecommendernetwork0_lstm0_rnn0\n", + "\n", + "\n", + "1x256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense1_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense1_fwd->dssmrecommendernetwork0_mean0\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense1_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense1_relu_fwd->dssmrecommendernetwork0_dense1_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat0\n", + "\n", + "dssmrecommendernetwork0_concat0\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat0->dssmrecommendernetwork0_dense0_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat0->dssmrecommendernetwork0_dense1_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dropout0_fwd\n", + "\n", + "dssmrecommendernetwork0_dropout0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_dropout0_fwd->dssmrecommendernetwork0_concat0\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense2_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense2_fwd->dssmrecommendernetwork0_dropout0_fwd\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense2_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense2_relu_fwd->dssmrecommendernetwork0_dense2_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims0\n", + "\n", + "dssmrecommendernetwork0_expand_dims0\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims0->dssmrecommendernetwork0_dense2_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "title\n", + "\n", + "title\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding2_fwd\n", + "\n", + "dssmrecommendernetwork0_embedding2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_embedding2_fwd->title\n", + "\n", + "\n", + "30\n", + "\n", + "\n", + "dssmrecommendernetwork0_transpose1\n", + "\n", + "dssmrecommendernetwork0_transpose1\n", + "\n", + "\n", + "dssmrecommendernetwork0_transpose1->dssmrecommendernetwork0_embedding2_fwd\n", + "\n", + "\n", + "30x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape0\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape1\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape1\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape2\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape2\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape3\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape3\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape4\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape4\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape5\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape5\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape6\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape6\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape7\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape7\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape8\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape8\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape9\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape9\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape10\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape10\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape11\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape11\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape12\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape12\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape13\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape13\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape14\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape14\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape15\n", + "\n", + "dssmrecommendernetwork0_lstm1_reshape15\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape0\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape1\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape2\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape3\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape4\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape5\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape6\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape7\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape8\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape9\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape10\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape11\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape12\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape13\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape14\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1__rnn_param_concat0->dssmrecommendernetwork0_lstm1_reshape15\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_0\n", + "\n", + "dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_1\n", + "\n", + "dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_1\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0->dssmrecommendernetwork0_transpose1\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0->dssmrecommendernetwork0_lstm1__rnn_param_concat0\n", + "\n", + "\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0->dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_0\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_lstm1_rnn0->dssmrecommendernetwork0_lstm1_dssmrecommendernetwork0_lstm1_h0_1\n", + "\n", + "\n", + "1x128\n", + "\n", + "\n", + "dssmrecommendernetwork0_mean1\n", + "\n", + "dssmrecommendernetwork0_mean1\n", + "\n", + "\n", + "dssmrecommendernetwork0_mean1->dssmrecommendernetwork0_lstm1_rnn0\n", + "\n", + "\n", + "1x256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense3_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense3_fwd->dssmrecommendernetwork0_mean1\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense3_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense3_relu_fwd->dssmrecommendernetwork0_dense3_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "image\n", + "\n", + "image\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm0_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm0_fwd->image\n", + "\n", + "\n", + "3x224x224\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_conv0_fwd\n", + "\n", + "Convolution\n", + "7x7/2x2, 64\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_conv0_fwd->dssmrecommendernetwork0_resnetv21_batchnorm0_fwd\n", + "\n", + "\n", + "3x224x224\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm1_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm1_fwd->dssmrecommendernetwork0_resnetv21_conv0_fwd\n", + "\n", + "\n", + "64x112x112\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_relu0_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_relu0_fwd->dssmrecommendernetwork0_resnetv21_batchnorm1_fwd\n", + "\n", + "\n", + "64x112x112\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_pool0_fwd\n", + "\n", + "Pooling\n", + "max, 3x3/2x2\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_pool0_fwd->dssmrecommendernetwork0_resnetv21_relu0_fwd\n", + "\n", + "\n", + "64x112x112\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm0_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm0_fwd->dssmrecommendernetwork0_resnetv21_pool0_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation0\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation0->dssmrecommendernetwork0_resnetv21_stage1_batchnorm0_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv0_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 64\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv0_fwd->dssmrecommendernetwork0_resnetv21_stage1_activation0\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm1_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm1_fwd->dssmrecommendernetwork0_resnetv21_stage1_conv0_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation1\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation1->dssmrecommendernetwork0_resnetv21_stage1_batchnorm1_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv1_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 64\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv1_fwd->dssmrecommendernetwork0_resnetv21_stage1_activation1\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus0\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus0\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus0->dssmrecommendernetwork0_resnetv21_pool0_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus0->dssmrecommendernetwork0_resnetv21_stage1_conv1_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm2_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm2_fwd->dssmrecommendernetwork0_resnetv21_stage1__plus0\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation2\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation2->dssmrecommendernetwork0_resnetv21_stage1_batchnorm2_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv2_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 64\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv2_fwd->dssmrecommendernetwork0_resnetv21_stage1_activation2\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm3_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm3_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_batchnorm3_fwd->dssmrecommendernetwork0_resnetv21_stage1_conv2_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation3\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_activation3->dssmrecommendernetwork0_resnetv21_stage1_batchnorm3_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv3_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 64\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1_conv3_fwd->dssmrecommendernetwork0_resnetv21_stage1_activation3\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus1\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus1\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus1->dssmrecommendernetwork0_resnetv21_stage1__plus0\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage1__plus1->dssmrecommendernetwork0_resnetv21_stage1_conv3_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm0_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm0_fwd->dssmrecommendernetwork0_resnetv21_stage1__plus1\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation0\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation0->dssmrecommendernetwork0_resnetv21_stage2_batchnorm0_fwd\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv0_fwd\n", + "\n", + "Convolution\n", + "3x3/2x2, 128\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv0_fwd->dssmrecommendernetwork0_resnetv21_stage2_activation0\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm1_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm1_fwd->dssmrecommendernetwork0_resnetv21_stage2_conv0_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation1\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation1->dssmrecommendernetwork0_resnetv21_stage2_batchnorm1_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv1_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 128\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv1_fwd->dssmrecommendernetwork0_resnetv21_stage2_activation1\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv2_fwd\n", + "\n", + "Convolution\n", + "1x1/2x2, 128\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv2_fwd->dssmrecommendernetwork0_resnetv21_stage2_activation0\n", + "\n", + "\n", + "64x56x56\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus0\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus0\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus0->dssmrecommendernetwork0_resnetv21_stage2_conv1_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus0->dssmrecommendernetwork0_resnetv21_stage2_conv2_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm2_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm2_fwd->dssmrecommendernetwork0_resnetv21_stage2__plus0\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation2\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation2->dssmrecommendernetwork0_resnetv21_stage2_batchnorm2_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv3_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 128\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv3_fwd->dssmrecommendernetwork0_resnetv21_stage2_activation2\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm3_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm3_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_batchnorm3_fwd->dssmrecommendernetwork0_resnetv21_stage2_conv3_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation3\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_activation3->dssmrecommendernetwork0_resnetv21_stage2_batchnorm3_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv4_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 128\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2_conv4_fwd->dssmrecommendernetwork0_resnetv21_stage2_activation3\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus1\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus1\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus1->dssmrecommendernetwork0_resnetv21_stage2__plus0\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage2__plus1->dssmrecommendernetwork0_resnetv21_stage2_conv4_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm0_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm0_fwd->dssmrecommendernetwork0_resnetv21_stage2__plus1\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation0\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation0->dssmrecommendernetwork0_resnetv21_stage3_batchnorm0_fwd\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv0_fwd\n", + "\n", + "Convolution\n", + "3x3/2x2, 256\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv0_fwd->dssmrecommendernetwork0_resnetv21_stage3_activation0\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm1_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm1_fwd->dssmrecommendernetwork0_resnetv21_stage3_conv0_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation1\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation1->dssmrecommendernetwork0_resnetv21_stage3_batchnorm1_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv1_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 256\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv1_fwd->dssmrecommendernetwork0_resnetv21_stage3_activation1\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv2_fwd\n", + "\n", + "Convolution\n", + "1x1/2x2, 256\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv2_fwd->dssmrecommendernetwork0_resnetv21_stage3_activation0\n", + "\n", + "\n", + "128x28x28\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus0\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus0\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus0->dssmrecommendernetwork0_resnetv21_stage3_conv1_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus0->dssmrecommendernetwork0_resnetv21_stage3_conv2_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm2_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm2_fwd->dssmrecommendernetwork0_resnetv21_stage3__plus0\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation2\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation2->dssmrecommendernetwork0_resnetv21_stage3_batchnorm2_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv3_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 256\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv3_fwd->dssmrecommendernetwork0_resnetv21_stage3_activation2\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm3_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm3_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_batchnorm3_fwd->dssmrecommendernetwork0_resnetv21_stage3_conv3_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation3\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_activation3->dssmrecommendernetwork0_resnetv21_stage3_batchnorm3_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv4_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 256\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3_conv4_fwd->dssmrecommendernetwork0_resnetv21_stage3_activation3\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus1\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus1\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus1->dssmrecommendernetwork0_resnetv21_stage3__plus0\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage3__plus1->dssmrecommendernetwork0_resnetv21_stage3_conv4_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm0_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm0_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm0_fwd->dssmrecommendernetwork0_resnetv21_stage3__plus1\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation0\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation0->dssmrecommendernetwork0_resnetv21_stage4_batchnorm0_fwd\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv0_fwd\n", + "\n", + "Convolution\n", + "3x3/2x2, 512\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv0_fwd->dssmrecommendernetwork0_resnetv21_stage4_activation0\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm1_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm1_fwd->dssmrecommendernetwork0_resnetv21_stage4_conv0_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation1\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation1->dssmrecommendernetwork0_resnetv21_stage4_batchnorm1_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv1_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 512\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv1_fwd->dssmrecommendernetwork0_resnetv21_stage4_activation1\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv2_fwd\n", + "\n", + "Convolution\n", + "1x1/2x2, 512\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv2_fwd->dssmrecommendernetwork0_resnetv21_stage4_activation0\n", + "\n", + "\n", + "256x14x14\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus0\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus0\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus0->dssmrecommendernetwork0_resnetv21_stage4_conv1_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus0->dssmrecommendernetwork0_resnetv21_stage4_conv2_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm2_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm2_fwd->dssmrecommendernetwork0_resnetv21_stage4__plus0\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation2\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation2->dssmrecommendernetwork0_resnetv21_stage4_batchnorm2_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv3_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 512\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv3_fwd->dssmrecommendernetwork0_resnetv21_stage4_activation2\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm3_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm3_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_batchnorm3_fwd->dssmrecommendernetwork0_resnetv21_stage4_conv3_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation3\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_activation3->dssmrecommendernetwork0_resnetv21_stage4_batchnorm3_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv4_fwd\n", + "\n", + "Convolution\n", + "3x3/1x1, 512\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4_conv4_fwd->dssmrecommendernetwork0_resnetv21_stage4_activation3\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus1\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus1\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus1->dssmrecommendernetwork0_resnetv21_stage4__plus0\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_stage4__plus1->dssmrecommendernetwork0_resnetv21_stage4_conv4_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm2_fwd\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm2_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_batchnorm2_fwd->dssmrecommendernetwork0_resnetv21_stage4__plus1\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_relu1_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_relu1_fwd->dssmrecommendernetwork0_resnetv21_batchnorm2_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_pool1_fwd\n", + "\n", + "Pooling\n", + "avg, 1x1/1x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_pool1_fwd->dssmrecommendernetwork0_resnetv21_relu1_fwd\n", + "\n", + "\n", + "512x7x7\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_flatten0_flatten0\n", + "\n", + "dssmrecommendernetwork0_resnetv21_flatten0_flatten0\n", + "\n", + "\n", + "dssmrecommendernetwork0_resnetv21_flatten0_flatten0->dssmrecommendernetwork0_resnetv21_pool1_fwd\n", + "\n", + "\n", + "512x1x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense4_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense4_fwd->dssmrecommendernetwork0_resnetv21_flatten0_flatten0\n", + "\n", + "\n", + "512\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense4_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense4_relu_fwd->dssmrecommendernetwork0_dense4_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat1\n", + "\n", + "dssmrecommendernetwork0_concat1\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat1->dssmrecommendernetwork0_dense3_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_concat1->dssmrecommendernetwork0_dense4_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dropout1_fwd\n", + "\n", + "dssmrecommendernetwork0_dropout1_fwd\n", + "\n", + "\n", + "dssmrecommendernetwork0_dropout1_fwd->dssmrecommendernetwork0_concat1\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense5_fwd\n", + "\n", + "FullyConnected\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense5_fwd->dssmrecommendernetwork0_dropout1_fwd\n", + "\n", + "\n", + "256\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense5_relu_fwd\n", + "\n", + "Activation\n", + "relu\n", + "\n", + "\n", + "dssmrecommendernetwork0_dense5_relu_fwd->dssmrecommendernetwork0_dense5_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims1\n", + "\n", + "dssmrecommendernetwork0_expand_dims1\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims1->dssmrecommendernetwork0_dense5_relu_fwd\n", + "\n", + "\n", + "128\n", + "\n", + "\n", + "dssmrecommendernetwork0_batch_dot0\n", + "\n", + "dssmrecommendernetwork0_batch_dot0\n", + "\n", + "\n", + "dssmrecommendernetwork0_batch_dot0->dssmrecommendernetwork0_expand_dims0\n", + "\n", + "\n", + "128x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_batch_dot0->dssmrecommendernetwork0_expand_dims1\n", + "\n", + "\n", + "128x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_norm0\n", + "\n", + "dssmrecommendernetwork0_norm0\n", + "\n", + "\n", + "dssmrecommendernetwork0_norm0->dssmrecommendernetwork0_expand_dims0\n", + "\n", + "\n", + "128x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_norm1\n", + "\n", + "dssmrecommendernetwork0_norm1\n", + "\n", + "\n", + "dssmrecommendernetwork0_norm1->dssmrecommendernetwork0_expand_dims1\n", + "\n", + "\n", + "128x1\n", + "\n", + "\n", + "dssmrecommendernetwork0__mul0\n", + "\n", + "dssmrecommendernetwork0__mul0\n", + "\n", + "\n", + "dssmrecommendernetwork0__mul0->dssmrecommendernetwork0_norm0\n", + "\n", + "\n", + "1\n", + "\n", + "\n", + "dssmrecommendernetwork0__mul0->dssmrecommendernetwork0_norm1\n", + "\n", + "\n", + "1\n", + "\n", + "\n", + "dssmrecommendernetwork0__plusscalar0\n", + "\n", + "dssmrecommendernetwork0__plusscalar0\n", + "\n", + "\n", + "dssmrecommendernetwork0__plusscalar0->dssmrecommendernetwork0__mul0\n", + "\n", + "\n", + "1\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims2\n", + "\n", + "dssmrecommendernetwork0_expand_dims2\n", + "\n", + "\n", + "dssmrecommendernetwork0_expand_dims2->dssmrecommendernetwork0__plusscalar0\n", + "\n", + "\n", + "1\n", + "\n", + "\n", + "dssmrecommendernetwork0__div0\n", + "\n", + "dssmrecommendernetwork0__div0\n", + "\n", + "\n", + "dssmrecommendernetwork0__div0->dssmrecommendernetwork0_batch_dot0\n", + "\n", + "\n", + "1x1\n", + "\n", + "\n", + "dssmrecommendernetwork0__div0->dssmrecommendernetwork0_expand_dims2\n", + "\n", + "\n", + "1x1\n", + "\n", + "\n", + "dssmrecommendernetwork0_squeeze0\n", + "\n", + "dssmrecommendernetwork0_squeeze0\n", + "\n", + "\n", + "dssmrecommendernetwork0_squeeze0->dssmrecommendernetwork0__div0\n", + "\n", + "\n", + "1x1\n", + "\n", + "\n", + "\n" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "mx.viz.plot_network(network(\n", + " mx.sym.var('user'), mx.sym.var('query_text'), mx.sym.var('title'), mx.sym.var('image')),\n", + " shape={'user': (1,1), 'query_text': (1,30), 'title': (1,30), 'image': (1,3,224,224)},\n", + " node_attrs={\"fixedsize\":\"False\"})" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can print the summary of the network using dummy data. We can see it is already training on 32M parameters!" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "collapsed": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--------------------------------------------------------------------------------\n", + " Layer (type) Output Shape Param #\n", + "================================================================================\n", + " Input (2, 1), (2, 5), (2, 5), (2, 3, 224, 224) 0\n", + " Embedding-1 (2, 1, 128) 12800000\n", + " Activation-2 0\n", + " Activation-3 (2, 128) 0\n", + " Dense-4 (2, 128) 16512\n", + " Embedding-5 (2, 5, 128) 3840000\n", + " LSTM-6 (5, 2, 256) 659456\n", + " Activation-7 0\n", + " Activation-8 (2, 128) 0\n", + " Dense-9 (2, 128) 32896\n", + " Dropout-10 (2, 256) 0\n", + " Activation-11 0\n", + " Activation-12 (2, 128) 0\n", + " Dense-13 (2, 128) 32896\n", + " Embedding-14 (2, 5, 128) 3840000\n", + " LSTM-15 (5, 2, 256) 659456\n", + " Activation-16 0\n", + " Activation-17 (2, 128) 0\n", + " Dense-18 (2, 128) 32896\n", + " BatchNorm-19 (2, 3, 224, 224) 12\n", + " Conv2D-20 (2, 64, 112, 112) 9408\n", + " BatchNorm-21 (2, 64, 112, 112) 256\n", + " Activation-22 (2, 64, 112, 112) 0\n", + " MaxPool2D-23 (2, 64, 56, 56) 0\n", + " BatchNorm-24 (2, 64, 56, 56) 256\n", + " Conv2D-25 (2, 64, 56, 56) 36864\n", + " BatchNorm-26 (2, 64, 56, 56) 256\n", + " Conv2D-27 (2, 64, 56, 56) 36864\n", + " BasicBlockV2-28 (2, 64, 56, 56) 0\n", + " BatchNorm-29 (2, 64, 56, 56) 256\n", + " Conv2D-30 (2, 64, 56, 56) 36864\n", + " BatchNorm-31 (2, 64, 56, 56) 256\n", + " Conv2D-32 (2, 64, 56, 56) 36864\n", + " BasicBlockV2-33 (2, 64, 56, 56) 0\n", + " BatchNorm-34 (2, 64, 56, 56) 256\n", + " Conv2D-35 (2, 128, 28, 28) 8192\n", + " Conv2D-36 (2, 128, 28, 28) 73728\n", + " BatchNorm-37 (2, 128, 28, 28) 512\n", + " Conv2D-38 (2, 128, 28, 28) 147456\n", + " BasicBlockV2-39 (2, 128, 28, 28) 0\n", + " BatchNorm-40 (2, 128, 28, 28) 512\n", + " Conv2D-41 (2, 128, 28, 28) 147456\n", + " BatchNorm-42 (2, 128, 28, 28) 512\n", + " Conv2D-43 (2, 128, 28, 28) 147456\n", + " BasicBlockV2-44 (2, 128, 28, 28) 0\n", + " BatchNorm-45 (2, 128, 28, 28) 512\n", + " Conv2D-46 (2, 256, 14, 14) 32768\n", + " Conv2D-47 (2, 256, 14, 14) 294912\n", + " BatchNorm-48 (2, 256, 14, 14) 1024\n", + " Conv2D-49 (2, 256, 14, 14) 589824\n", + " BasicBlockV2-50 (2, 256, 14, 14) 0\n", + " BatchNorm-51 (2, 256, 14, 14) 1024\n", + " Conv2D-52 (2, 256, 14, 14) 589824\n", + " BatchNorm-53 (2, 256, 14, 14) 1024\n", + " Conv2D-54 (2, 256, 14, 14) 589824\n", + " BasicBlockV2-55 (2, 256, 14, 14) 0\n", + " BatchNorm-56 (2, 256, 14, 14) 1024\n", + " Conv2D-57 (2, 512, 7, 7) 131072\n", + " Conv2D-58 (2, 512, 7, 7) 1179648\n", + " BatchNorm-59 (2, 512, 7, 7) 2048\n", + " Conv2D-60 (2, 512, 7, 7) 2359296\n", + " BasicBlockV2-61 (2, 512, 7, 7) 0\n", + " BatchNorm-62 (2, 512, 7, 7) 2048\n", + " Conv2D-63 (2, 512, 7, 7) 2359296\n", + " BatchNorm-64 (2, 512, 7, 7) 2048\n", + " Conv2D-65 (2, 512, 7, 7) 2359296\n", + " BasicBlockV2-66 (2, 512, 7, 7) 0\n", + " BatchNorm-67 (2, 512, 7, 7) 2048\n", + " Activation-68 (2, 512, 7, 7) 0\n", + " GlobalAvgPool2D-69 (2, 512, 1, 1) 0\n", + " Flatten-70 (2, 512) 0\n", + " Activation-71 0\n", + " Activation-72 (2, 128) 0\n", + " Dense-73 (2, 128) 65664\n", + " Dropout-74 (2, 256) 0\n", + " Activation-75 0\n", + " Activation-76 (2, 128) 0\n", + " Dense-77 (2, 128) 32896\n", + "DSSMRecommenderNetwork-78 (2, 1) 0\n", + "================================================================================\n", + "Parameters in forward computation graph, duplicate included\n", + " Total params: 33195468\n", + " Trainable params: 33187520\n", + " Non-trainable params: 7948\n", + "Shared params in forward computation graph: 0\n", + "Unique parameters in model: 33195468\n", + "--------------------------------------------------------------------------------\n" + ] + } + ], + "source": [ + "user = mx.nd.array([[200], [100]], ctx)\n", + "query = mx.nd.array([[10, 20, 0, 0, 0], [40, 50, 0, 0, 0]], ctx) # Example of an encoded text\n", + "title = mx.nd.array([[10, 20, 0, 0, 0], [40, 50, 0, 0, 0]], ctx) # Example of an encoded text\n", + "image = mx.nd.random.uniform(shape=(2,3, 224,224), ctx=ctx) # Example of an encoded image\n", + "\n", + "\n", + "network.summary(user, query, title, image)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "\n", + "[[0.34404233]\n", + " [0.3254302 ]]\n", + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "network(user, query, title, image)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The output is the similarity, if we wanted to train it on real data, we would need to minimize the Cosine loss, 1 - cosine_similarity." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/example/recommenders/demo3-dssm.ipynb b/example/recommenders/demo3-dssm.ipynb deleted file mode 100644 index 55812f646..000000000 --- a/example/recommenders/demo3-dssm.ipynb +++ /dev/null @@ -1,113 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Content-based recommender using DSSM\n", - "An example of how to build a Deep Structured Semantic Model (DSSM) for incorporating complex content-based features into a recommender system. See [Learning Deep Structured Semantic Models for Web Search using Clickthrough Data](https://www.microsoft.com/en-us/research/publication/learning-deep-structured-semantic-models-for-web-search-using-clickthrough-data/). This example does not attempt to provide a datasource or train a model, but merely show how to structure a complex DSSM network." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "import mxnet as mx\n", - "import symbol_alexnet as alexnet\n", - "import recotools" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# Define some constants\n", - "max_user = int(1e6)\n", - "title_vocab = int(1e5)\n", - "ngram_dimensions = int(1e8)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false, - "scrolled": false - }, - "outputs": [], - "source": [ - "def dssm_recommender(k):\n", - " # input variables\n", - " title = mx.symbol.Variable('title_words')\n", - " image = mx.symbol.Variable('image')\n", - " queries = mx.symbol.Variable('query_ngrams')\n", - " user = mx.symbol.Variable('user_id')\n", - " label = mx.symbol.Variable('label')\n", - " \n", - " # Process content stack\n", - " image = alexnet.features(image, 256)\n", - " title = recotools.SparseBagOfWordProjection(data=title, vocab_size=title_vocab, \n", - " output_dim=k)\n", - " title = mx.symbol.FullyConnected(data=title, num_hidden=k)\n", - " content = mx.symbol.Concat(image, title)\n", - " content = mx.symbol.Dropout(content, p=0.5)\n", - " content = mx.symbol.FullyConnected(data=content, num_hidden=k)\n", - " \n", - " # Process user stack\n", - " user = mx.symbol.Embedding(data=user, input_dim=max_user, output_dim=k) \n", - " user = mx.symbol.FullyConnected(data=user, num_hidden=k)\n", - " queries = recotools.SparseBagOfWordProjection(data=queries, vocab_size=ngram_dimensions, \n", - " output_dim=k)\n", - " queries = mx.symbol.FullyConnected(data=queries, num_hidden=k)\n", - " user = mx.symbol.Concat(user,queries)\n", - " user = mx.symbol.Dropout(user, p=0.5)\n", - " user = mx.symbol.FullyConnected(data=user, num_hidden=k)\n", - " \n", - " # loss layer\n", - " pred = recotools.CosineLoss(a=user, b=content, label=label)\n", - " return pred\n", - "\n", - "net1 = dssm_recommender(256)\n", - "mx.viz.plot_network(net1)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python [Root]", - "language": "python", - "name": "Python [Root]" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.12" - } - }, - "nbformat": 4, - "nbformat_minor": 1 -} diff --git a/example/recommenders/matrix_fact.py b/example/recommenders/matrix_fact.py index 73f561a87..5f7d173f4 100644 --- a/example/recommenders/matrix_fact.py +++ b/example/recommenders/matrix_fact.py @@ -15,48 +15,63 @@ # specific language governing permissions and limitations # under the License. +import logging import math +import random + import mxnet as mx +from mxnet import gluon, autograd, nd import numpy as np -import mxnet.notebook.callback -import logging logging.basicConfig(level=logging.DEBUG) -def RMSE(label, pred): - ret = 0.0 - n = 0.0 - pred = pred.flatten() - for i in range(len(label)): - ret += (label[i] - pred[i]) * (label[i] - pred[i]) - n += 1.0 - return math.sqrt(ret / n) +def evaluate_network(network, data_iterator, ctx): + loss_acc = 0. + l2 = gluon.loss.L2Loss() + for i, (users, items, scores) in enumerate(data_iterator): + users_ = gluon.utils.split_and_load(users, ctx) + items_ = gluon.utils.split_and_load(items, ctx) + scores_ =gluon.utils.split_and_load(scores, ctx) + preds = [network(u, i) for u, i in zip(users_, items_)] + losses = [l2(p, s).asnumpy() for p, s in zip(preds, scores_)] + loss_acc += sum(losses).mean()/len(ctx) + return loss_acc/(i+1) +def train(network, train_data, test_data, epochs, learning_rate=0.01, optimizer='sgd', ctx=mx.gpu(0), num_epoch_lr=5, factor=0.2): -def train(network, data_pair, num_epoch, learning_rate, optimizer='sgd', opt_args=None, ctx=[mx.gpu(0)]): np.random.seed(123) # Fix random seed for consistent demos mx.random.seed(123) # Fix random seed for consistent demos - if not opt_args: - opt_args = {} - if optimizer=='sgd' and (not opt_args): - opt_args['momentum'] = 0.9 - - model = mx.model.FeedForward( - ctx = ctx, - symbol = network, - num_epoch = num_epoch, - optimizer = optimizer, - learning_rate = learning_rate, - wd = 1e-4, - **opt_args - ) - - train, test = (data_pair) - - lc = mxnet.notebook.callback.LiveLearningCurve('RMSE', 1) - model.fit(X = train, - eval_data = test, - eval_metric = RMSE, - **mxnet.notebook.callback.args_wrapper(lc) - ) - return lc + random.seed(123) # Fix random seed for consistent demos + + schedule = mx.lr_scheduler.FactorScheduler(step=len(train_data)*len(ctx)*num_epoch_lr, factor=factor) + + trainer = gluon.Trainer(network.collect_params(), optimizer, + {'learning_rate':learning_rate, 'wd':0.0001, 'lr_scheduler':schedule}) + #update_on_kvstore=False) + + l2 = gluon.loss.L2Loss() + + network.hybridize() + + losses_output = [] + for e in range(epochs): + loss_acc = 0. + for i, (users, items, scores) in enumerate(train_data): + + users_ = gluon.utils.split_and_load(users, ctx) + items_ = gluon.utils.split_and_load(items, ctx) + scores_ =gluon.utils.split_and_load(scores, ctx) + + with autograd.record(): + preds = [network(u, i) for u, i in zip(users_, items_)] + losses = [l2(p, s) for p, s in zip(preds, scores_)] + + [l.backward() for l in losses] + loss_acc += sum([l.asnumpy() for l in losses]).mean()/len(ctx) + trainer.update(users.shape[0]) + + test_loss = evaluate_network(network, test_data, ctx) + train_loss = loss_acc/(i+1) + print("Epoch [{}], Training RMSE {:.4f}, Test RMSE {:.4f}".format(e, train_loss, test_loss)) + losses_output.append((train_loss, test_loss)) + return losses_output \ No newline at end of file diff --git a/example/recommenders/movielens_data.py b/example/recommenders/movielens_data.py index 600fa5407..5f86ad17a 100644 --- a/example/recommenders/movielens_data.py +++ b/example/recommenders/movielens_data.py @@ -20,8 +20,9 @@ import os import mxnet as mx +from mxnet import gluon -def load_mldata_iter(filename, batch_size): +def load_mldataset(filename): """Not particularly fast code to parse the text file and load it into three NDArray's and product an NDArrayIter """ @@ -39,8 +40,7 @@ def load_mldata_iter(filename, batch_size): user = mx.nd.array(user) item = mx.nd.array(item) score = mx.nd.array(score) - return mx.io.NDArrayIter(data={'user':user,'item':item},label={'score':score}, - batch_size=batch_size, shuffle=True) + return gluon.data.ArrayDataset(user, item, score) def ensure_local_data(prefix): if not os.path.exists("%s.zip" % prefix): @@ -49,12 +49,12 @@ def ensure_local_data(prefix): os.system("unzip %s.zip" % prefix) -def get_data_iter(batch_size, prefix='ml-100k'): +def get_dataset(prefix='ml-100k'): """Returns a pair of NDArrayDataIter, one for train, one for test. """ ensure_local_data(prefix) - return (load_mldata_iter('./%s/u1.base' % prefix, batch_size), - load_mldata_iter('./%s/u1.test' % prefix, batch_size)) + return (load_mldataset('./%s/u1.base' % prefix), + load_mldataset('./%s/u1.test' % prefix)) def max_id(fname): mu = 0 diff --git a/example/recommenders/negativesample.py b/example/recommenders/negativesample.py deleted file mode 100644 index a78ecde93..000000000 --- a/example/recommenders/negativesample.py +++ /dev/null @@ -1,180 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""DataIter for negative sampling. -""" -import mxnet as mx -import numpy as np - -class NegativeSamplingDataIter(mx.io.DataIter): - """Wraps an existing DataIter to produce a new DataIter with negative samples. - Assumes that all the relevant inputs are in data, not labels. - Drops (replaces) any labels in the original DataIter. - - It only shuffles one of the input data columns, specified in the - constructor as shuffle_data_idx. So if the original input data - has three columns, ('item_ids', 'item_words', 'users') and you want - to keep the two "item_*" together, then set `shuffle_data_idx=2` - and `users` will be shuffled for the negative samples. - - Output batches will be larger than input batches by a factor - of (1+sample_ratio) - - Negative samples are always drawn from the same minibatch. - So they're effectively sampled according to the frequency at - which they appear in the training data. (Other reasonable - samling strategies are not implemented here.) - The shuffling is checked to ensure that a true positive sample - isn't returned as a negative sample. - """ - - def __init__(self, source_dataiter, sample_ratio=1, shuffle_data_idx=1, - positive_label=1, negative_label=0): - self._sourcedata = source_dataiter - source_dataiter.reset() - self.positive_label = positive_label # output shapes = input shapes - self.negative_label = negative_label - self.shuffle_data_idx = shuffle_data_idx - if sample_ratio == int(sample_ratio): - self.sample_ratio = int(sample_ratio) - else: - raise ValueError("sample_ratio must be an integer, not %s" % sample_ratio) - self._clear_queue() - - self.provide_data = source_dataiter.provide_data - self.provide_label = source_dataiter.provide_label - self.batch_size = source_dataiter.batch_size - - def _clear_queue(self): - self._sampled_queue = [] - - def _push_queue(self, data_list, labels): - """Takes a list of numpy arrays for data, - and a numpy array for labels. - Converts to minibatches and puts it on the queue. - """ - num_minibatches = 1+self.sample_ratio - total_size = len(labels) - slice_size = total_size / num_minibatches - def slicer(x, s): - idx = range(int(s*slice_size), int((s+1)*slice_size)) - return np.take(x,idx,0) - - for i in range(1+self.sample_ratio): - nddata = [mx.nd.array(slicer(x,i)) for x in data_list] - ndlabels = mx.nd.array(slicer(labels,i)) - batch = mx.io.DataBatch(nddata, [ndlabels], provide_data=self.provide_data, - provide_label=self.provide_label) - self._sampled_queue.append(batch) - - def next(self): - if not self._sampled_queue: - self._refill_queue() - batch = self._sampled_queue.pop() - return batch - - def reset(self): - self._sourcedata.reset() - self._clear_queue() - - def _shuffle_batch(self, data): - # Takes a list of NDArrays. Returns a shuffled version as numpy - a = data[self.shuffle_data_idx].asnumpy() - - # Come up with a shuffled index - batch_size = data[0].shape[0] - si = np.arange(batch_size) - np.random.shuffle(si) - matches = (si == np.arange(batch_size)) # everywhere it didn't shuffle - si -= matches # Shifts down by 1 when true, ensuring it differs - # Note shifting down by 1 works in python because -1 is a valid index. - #Q: Is this shifting introducing bias? - - # Shuffle the data with the shuffle index - shuf_a = np.take(a,si,0) # like a[si,:] but general for ndarray's - - # Return similar datastructure to what we got. Convert all to numpy - out = [d.asnumpy() for d in data] - out[self.shuffle_data_idx] = shuf_a - return out - - def _refill_queue(self): - """Fetch another batch from the source, and shuffle it to make - negative samples. - """ - original = self._sourcedata.next().data # List of NDArrays: one per input - batch_size = original[0].shape[0] - num_inputs = len(original) - - #Start with positive examples, copied straight - outdata = [[o.asnumpy()] for o in original] # list of lists of numpy arrays - outlabels = [np.ones(batch_size) * self.positive_label] # list of numpy arrays - # The inner list of both is the set of samples. We'll recombine later. - - # Construct negative samples. - for _ in range(self.sample_ratio): - shuffled = self._shuffle_batch(original) - for i,d in enumerate(shuffled): - outdata[i].append(d) - outlabels.append(np.ones(batch_size) * self.negative_label) - def stacker(x): - if len(x[0].shape)==1: - return np.hstack(x) - else: - return np.vstack(x) - outdata = [stacker(x) for x in outdata] # Single tall vectors - outlabels = stacker(outlabels) - - # Record-level shuffle so the negatives are mixed in. - def shuffler(x, idx): - return np.take(x,idx,0) - shuf_idx = np.arange(len(outlabels)) - np.random.shuffle(shuf_idx) - outdata = [shuffler(o,shuf_idx) for o in outdata] - outlabels = shuffler(outlabels,shuf_idx) - self._push_queue(outdata,outlabels) - - -if __name__ == "__main__": - print("Simple test of NegativeSamplingDataIter") - np.random.seed(123) - A = np.random.randint(-20,150,size=(100,5)) - B = np.random.randint(-2,15,size=(100,2)) - R = np.random.randint(1,5,size=(100,)) - batch_size=3 - oridi = mx.io.NDArrayIter(data={'a':A,'b':B},label=R, batch_size=batch_size) - oribatch = oridi.next() - oridi.reset() - for ratio in range(0,5): - nsdi = NegativeSamplingDataIter(oridi, sample_ratio=ratio) - - # Check sizes of output - bat = nsdi.next() - for i in range(len(bat.data)): - assert bat.data[i].shape[0] == batch_size - assert bat.data[i].shape[1] == oribatch.data[i].shape[1] - assert bat.label.shape[0] == batch_size - - # Check that we get more minibatches - oridi.reset() - ori_cnt = len(list(oridi)) - nsdi.reset() - ns_cnt = len(list(nsdi)) - assert ns_cnt == ori_cnt * (1+ratio) - - print("Tests done") - diff --git a/example/recommenders/randomproj.py b/example/recommenders/randomproj.py deleted file mode 100644 index 83ce3a1e7..000000000 --- a/example/recommenders/randomproj.py +++ /dev/null @@ -1,150 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Random projection layers in MXNet as custom python ops. -Currently slow and memory-inefficient, but functional. -""" -import os -import numpy as np -import mxnet as mx - - -# ref: http://mxnet.io/faq/new_op.html - -class RandomBagOfWordsProjection(mx.operator.CustomOp): - """Random projection layer for sparse bag-of-words (n-hot) inputs. - In the sparse input, only the indices are supplied, because all the - values are understood to be exactly 1.0. - - See also RandomProjection for values other than 1.0. - """ - - def __init__(self, vocab_size, output_dim, random_seed=54321): - # need_top_grad=True means this is not a loss layer - super(RandomBagOfWordsProjection, self).__init__() - self._vocab = vocab_size - self._proj_dim = output_dim - #NOTE: This naive implementation is slow and uses lots of memory. - # Should use something smarter to not instantiate this matrix. - rs = np.random.RandomState(seed=random_seed) - self.W = self.random_unit_vecs(self._vocab, self._proj_dim, rs) - - def random_unit_vecs(self, num_vecs, num_dims, rs): - W = rs.normal(size=(num_vecs, num_dims)) - Wlen = np.linalg.norm(W, axis=1) - W_unit = W / Wlen[:,None] - return W_unit - - def _get_mask(self, idx, in_data): - """Returns the mask by which to multiply the parts of the embedding layer. - In this version, we have no weights to apply. - """ - mask = idx >= 0 # bool False for -1 values that should be removed. shape=(b,mnz) - mask = np.expand_dims(mask,2) # shape = (b,mnz,1) - mask = np.repeat(mask, self._proj_dim, axis=2) # shape = (b,mnz,d) - return mask - - def forward(self, is_train, req, in_data, out_data, aux): - #Note: see this run in notebooks/howto-numpy-random-proj.ipynb - # Notation for shapes: b = batch_size, mnz = max_nonzero, d = proj_dim - idx = in_data[0].asnumpy().astype('int32') # shape=(b,mnz) - - wd = self.W[idx] # shape= (b,mnz,d) - mask = self._get_mask(idx, in_data) - wd = np.multiply(wd,mask) # shape=(b,mnz,d), but zero'd out non-masked - y = np.sum(wd,axis=1) # shape=(b,d) - mxy = mx.nd.array(y) #NOTE: this hangs if the environment variables aren't set correctly - # See https://github.com/dmlc/mxnet/issues/3813 - self.assign(out_data[0], req[0], mxy) - - -@mx.operator.register("SparseBOWProj") -class RandomBagOfWordsProjectionProp(mx.operator.CustomOpProp): - def __init__(self, vocab_size, output_dim): - # need_top_grad=True means this is not a loss layer - super(RandomBagOfWordsProjectionProp, self).__init__(need_top_grad=True) - self._kwargs = { - 'vocab_size': int(vocab_size), - 'output_dim': int(output_dim), - } - - def list_arguments(self): - return ['indexes'] - - def list_outputs(self): - return ['output'] - - def create_operator(self, ctx, shapes, dtypes, **kwargs): - return RandomBagOfWordsProjection(**self._kwargs) - - def infer_shape(self, in_shape): - batch_size = in_shape[0][0] - output_shape = (batch_size, self._kwargs['output_dim']) - return in_shape, [output_shape], [] - - -class SparseRandomProjection(RandomBagOfWordsProjection): - """Random projection of sparse input vector. - Takes an sparse input layer, effectively in coordinate (COO) format, - where the row number is implicit, because it's the minibatch record. - - See the simpler version RandomBagOfWordsProjection if all values are 1.0. - """ - - def _get_mask(self, idx, in_data): - """Returns the mask by which to multiply the parts of the embedding layer. - In this version, we apply the weights. - """ - val = in_data[1].asnumpy() # shape=(b,mnz) - mask = idx >= 0 # bool False for -1 values that should be removed. shape=(b,mnz) - mask = np.multiply(mask,val) # All (b,mnz) - mask = np.expand_dims(mask,2) # shape = (b,mnz,1) - mask = np.repeat(mask, self._proj_dim, axis=2) # shape = (b,mnz,d) - return mask - - -@mx.operator.register("SparseRandomProjection") -class SparseRandomProjectionProp(RandomBagOfWordsProjectionProp): - - def list_arguments(self): - return ['indexes', 'values'] - - def create_operator(self, ctx, shapes, dtypes, **kwargs): - return SparseRandomProjection(**self._kwargs) - - def infer_shape(self, in_shape): - # check that indexes and values are the same shape. - if in_shape[0] != in_shape[1]: - raise ValueError("Input shapes differ. indexes:%s. values:%s. must be same" - % (str(in_shape[0]),str(in_shape[1]))) - return super(SparseRandomProjectionProp,self).infer_shape(in_shape) - - -if __name__ == "__main__": - print("Simple test of proj layer") - data = mx.symbol.Variable('data') - vals = mx.symbol.Variable('vals') - net = mx.symbol.Custom(indexes=data, values=vals, name='rproj', - op_type='SparseRandomProjection', - vocab_size=999, output_dim=29) - d = mx.nd.zeros(shape=(3,100)) - v = mx.nd.ones(shape=(3,100)) - e = net.bind(ctx=mx.cpu(), args={'data':d, 'vals':v}) - e.forward() - print(e.outputs[0].asnumpy()) - print("Done with proj layer test") - diff --git a/example/recommenders/recotools.py b/example/recommenders/recotools.py deleted file mode 100644 index 70e069cd1..000000000 --- a/example/recommenders/recotools.py +++ /dev/null @@ -1,45 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -import mxnet as mx - -from negativesample import NegativeSamplingDataIter -import randomproj -import crossentropy - -def CosineLoss(a, b, label): - a = mx.symbol.L2Normalization(a) - b = mx.symbol.L2Normalization(b) - dot = a * b - dot = mx.symbol.sum_axis(dot, axis=1) - dot = mx.symbol.Flatten(dot) - cosine = 1 - dot - cosine = cosine / 2 - return mx.symbol.MAERegressionOutput(data=cosine, label=label) - -def SparseRandomProjection(indexes, values, input_dim, output_dim, ngram=1): - return mx.symbol.Custom(indexes=indexes, values=values, vocab_size=input_dim, - output_dim=output_dim, op_type='SparseRandomProjection') - -def SparseBagOfWordProjection(data, vocab_size, output_dim, ngram=1): - return mx.symbol.Custom(indexes=data, vocab_size=vocab_size, - output_dim=output_dim, op_type='SparseBOWProj') - -def CrossEntropyLoss(data, label): - return mx.symbol.Custom(data=data, label=label, - op_type='CrossEntropyLoss') - diff --git a/example/recommenders/symbol_alexnet.py b/example/recommenders/symbol_alexnet.py deleted file mode 100644 index e5d02f041..000000000 --- a/example/recommenders/symbol_alexnet.py +++ /dev/null @@ -1,62 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -""" -Reference: - -Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. -""" -import mxnet as mx - -def features(input_data, in_channels): - # stage 1 - conv1 = mx.symbol.Convolution( - data=input_data, kernel=(11, 11), stride=(4, 4), num_filter=96) - relu1 = mx.symbol.Activation(data=conv1, act_type="relu") - pool1 = mx.symbol.Pooling( - data=relu1, pool_type="max", kernel=(3, 3), stride=(2,2)) - lrn1 = mx.symbol.LRN(data=pool1, alpha=0.0001, beta=0.75, knorm=1, nsize=5) - # stage 2 - conv2 = mx.symbol.Convolution( - data=lrn1, kernel=(5, 5), pad=(2, 2), num_filter=256) - relu2 = mx.symbol.Activation(data=conv2, act_type="relu") - pool2 = mx.symbol.Pooling(data=relu2, kernel=(3, 3), stride=(2, 2), pool_type="max") - lrn2 = mx.symbol.LRN(data=pool2, alpha=0.0001, beta=0.75, knorm=1, nsize=5) - # stage 3 - conv3 = mx.symbol.Convolution( - data=lrn2, kernel=(3, 3), pad=(1, 1), num_filter=384) - relu3 = mx.symbol.Activation(data=conv3, act_type="relu") - conv4 = mx.symbol.Convolution( - data=relu3, kernel=(3, 3), pad=(1, 1), num_filter=384) - relu4 = mx.symbol.Activation(data=conv4, act_type="relu") - conv5 = mx.symbol.Convolution( - data=relu4, kernel=(3, 3), pad=(1, 1), num_filter=256) - relu5 = mx.symbol.Activation(data=conv5, act_type="relu") - pool3 = mx.symbol.Pooling(data=relu5, kernel=(3, 3), stride=(2, 2), pool_type="max") - # stage 4 - flatten = mx.symbol.Flatten(data=pool3) - fc1 = mx.symbol.FullyConnected(data=flatten, num_hidden=4096) - relu6 = mx.symbol.Activation(data=fc1, act_type="relu") - dropout1 = mx.symbol.Dropout(data=relu6, p=0.5) - # stage 5 - fc2 = mx.symbol.FullyConnected(data=dropout1, num_hidden=4096) - relu7 = mx.symbol.Activation(data=fc2, act_type="relu") - dropout2 = mx.symbol.Dropout(data=relu7, p=0.5) - # stage 6 - fc3 = mx.symbol.FullyConnected(data=dropout2, num_hidden=in_channels) - - return fc3 diff --git a/example/reinforcement-learning/ddpg/README.md b/example/reinforcement-learning/ddpg/README.md index 37f42a829..2e299dd5d 100644 --- a/example/reinforcement-learning/ddpg/README.md +++ b/example/reinforcement-learning/ddpg/README.md @@ -1,6 +1,8 @@ # mx-DDPG MXNet Implementation of DDPG +## /!\ This example depends on RLLAB which is deprecated /!\ + # Introduction This is the MXNet implementation of [DDPG](https://arxiv.org/abs/1509.02971). It is tested in the rllab cart pole environment against rllab's native implementation and achieves comparably similar results. You can substitute with this anywhere you use rllab's DDPG with minor modifications. diff --git a/example/reinforcement-learning/dqn/setup.sh b/example/reinforcement-learning/dqn/setup.sh index 3069fef62..012ff8fb1 100755 --- a/example/reinforcement-learning/dqn/setup.sh +++ b/example/reinforcement-learning/dqn/setup.sh @@ -26,11 +26,11 @@ pip install pygame # Install arcade learning environment if [[ "$OSTYPE" == "linux-gnu" ]]; then - sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake + sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake ninja-build elif [[ "$OSTYPE" == "darwin"* ]]; then brew install sdl sdl_image sdl_mixer sdl_ttf portmidi fi -git clone git@github.com:mgbellemare/Arcade-Learning-Environment.git || true +git clone https://github.com/mgbellemare/Arcade-Learning-Environment || true pushd . cd Arcade-Learning-Environment mkdir -p build @@ -43,6 +43,6 @@ popd cp Arcade-Learning-Environment/ale.cfg . # Copy roms -git clone git@github.com:npow/atari.git || true +git clone https://github.com/npow/atari || true cp -R atari/roms . diff --git a/example/restricted-boltzmann-machine/README.md b/example/restricted-boltzmann-machine/README.md index 129120ba9..a8769a51e 100644 --- a/example/restricted-boltzmann-machine/README.md +++ b/example/restricted-boltzmann-machine/README.md @@ -8,6 +8,58 @@ Here are some samples generated by the RBM with the default hyperparameters. The

      +Usage: + +``` +python binary_rbm_gluon.py --help +usage: binary_rbm_gluon.py [-h] [--num-hidden NUM_HIDDEN] [--k K] + [--batch-size BATCH_SIZE] [--num-epoch NUM_EPOCH] + [--learning-rate LEARNING_RATE] + [--momentum MOMENTUM] + [--ais-batch-size AIS_BATCH_SIZE] + [--ais-num-batch AIS_NUM_BATCH] + [--ais-intermediate-steps AIS_INTERMEDIATE_STEPS] + [--ais-burn-in-steps AIS_BURN_IN_STEPS] [--cuda] + [--no-cuda] [--device-id DEVICE_ID] + [--data-loader-num-worker DATA_LOADER_NUM_WORKER] + +Restricted Boltzmann machine learning MNIST + +optional arguments: + -h, --help show this help message and exit + --num-hidden NUM_HIDDEN + number of hidden units + --k K number of Gibbs sampling steps used in the PCD + algorithm + --batch-size BATCH_SIZE + batch size + --num-epoch NUM_EPOCH + number of epochs + --learning-rate LEARNING_RATE + learning rate for stochastic gradient descent + --momentum MOMENTUM momentum for the stochastic gradient descent + --ais-batch-size AIS_BATCH_SIZE + batch size for AIS to estimate the log-likelihood + --ais-num-batch AIS_NUM_BATCH + number of batches for AIS to estimate the log- + likelihood + --ais-intermediate-steps AIS_INTERMEDIATE_STEPS + number of intermediate distributions for AIS to + estimate the log-likelihood + --ais-burn-in-steps AIS_BURN_IN_STEPS + number of burn in steps for each intermediate + distributions of AIS to estimate the log-likelihood + --cuda train on GPU with CUDA + --no-cuda train on CPU + --device-id DEVICE_ID + GPU device id + --data-loader-num-worker DATA_LOADER_NUM_WORKER + number of multithreading workers for the data loader +``` +Default: +``` +Namespace(ais_batch_size=100, ais_burn_in_steps=10, ais_intermediate_steps=10, ais_num_batch=10, batch_size=80, cuda=True, data_loader_num_worker=4, device_id=0, k=30, learning_rate=0.1, momentum=0.3, num_epoch=130, num_hidden=500) +``` [1] G E Hinton & R R Salakhutdinov, Reducing the Dimensionality of Data with Neural Networks Science **313**, 5786 (2006)
      [2] R M Neal, Annealed importance sampling. Stat Comput **11** 2 (2001)
      [3] R Salakhutdinov & I Murray, On the quantitative analysis of deep belief networks. In Proc. ICML '08 **25** (2008) \ No newline at end of file diff --git a/example/rnn-time-major/bucket_io.py b/example/rnn-time-major/bucket_io.py deleted file mode 100644 index e689ff112..000000000 --- a/example/rnn-time-major/bucket_io.py +++ /dev/null @@ -1,264 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=C0111,too-many-arguments,too-many-instance-attributes,too-many-locals,redefined-outer-name,fixme -# pylint: disable=superfluous-parens, no-member, invalid-name - -from __future__ import print_function -import numpy as np -import mxnet as mx - -# The interface of a data iter that works for bucketing -# -# DataIter -# - default_bucket_key: the bucket key for the default symbol. -# -# DataBatch -# - provide_data: same as DataIter, but specific to this batch -# - provide_label: same as DataIter, but specific to this batch -# - bucket_key: the key for the bucket that should be used for this batch - -def default_read_content(path): - with open(path) as ins: - content = ins.read() - content = content.replace('\n', ' ').replace('. ', ' ') - return content - -def default_build_vocab(path): - content = default_read_content(path) - content = content.split(' ') - idx = 1 # 0 is left for zero-padding - the_vocab = {} - the_vocab[' '] = 0 # put a dummy element here so that len(vocab) is correct - for word in content: - if len(word) == 0: - continue - if not word in the_vocab: - the_vocab[word] = idx - idx += 1 - return the_vocab - -def default_text2id(sentence, the_vocab): - words = sentence.split(' ') - words = [the_vocab[w] for w in words if len(w) > 0] - return words - -def default_gen_buckets(sentences, batch_size, the_vocab): - len_dict = {} - max_len = -1 - for sentence in sentences: - words = default_text2id(sentence, the_vocab) - if len(words) == 0: - continue - if len(words) > max_len: - max_len = len(words) - if len(words) in len_dict: - len_dict[len(words)] += 1 - else: - len_dict[len(words)] = 1 - print(len_dict) - - tl = 0 - buckets = [] - for l, n in len_dict.items(): # TODO: There are better heuristic ways to do this - if n + tl >= batch_size: - buckets.append(l) - tl = 0 - else: - tl += n - if tl > 0: - buckets.append(max_len) - return buckets - -class SimpleBatch(object): - def __init__(self, data_names, data, data_layouts, label_names, label, label_layouts, bucket_key): - self.data = data - self.label = label - self.data_names = data_names - self.label_names = label_names - self.data_layouts = data_layouts - self.label_layouts = label_layouts - self.bucket_key = bucket_key - - self.pad = 0 - self.index = None # TODO: what is index? - - @property - def provide_data(self): - return [mx.io.DataDesc(n, x.shape, layout=l) for n, x, l in zip(self.data_names, self.data, self.data_layouts)] - - @property - def provide_label(self): - return [mx.io.DataDesc(n, x.shape, layout=l) for n, x, l in zip(self.label_names, self.label, self.label_layouts)] - -class DummyIter(mx.io.DataIter): - "A dummy iterator that always return the same batch, used for speed testing" - def __init__(self, real_iter): - super(DummyIter, self).__init__() - self.real_iter = real_iter - self.provide_data = real_iter.provide_data - self.provide_label = real_iter.provide_label - self.batch_size = real_iter.batch_size - - for batch in real_iter: - self.the_batch = batch - break - - def __iter__(self): - return self - - def next(self): - return self.the_batch - -class BucketSentenceIter(mx.io.DataIter): - def __init__(self, path, vocab, buckets, batch_size, - init_states, data_name='data', label_name='label', - seperate_char=' ', text2id=None, read_content=None, - time_major=True): - super(BucketSentenceIter, self).__init__() - - if text2id is None: - self.text2id = default_text2id - else: - self.text2id = text2id - if read_content is None: - self.read_content = default_read_content - else: - self.read_content = read_content - content = self.read_content(path) - sentences = content.split(seperate_char) - - if len(buckets) == 0: - buckets = default_gen_buckets(sentences, batch_size, vocab) - - self.vocab_size = len(vocab) - self.data_name = data_name - self.label_name = label_name - self.time_major = time_major - - buckets.sort() - self.buckets = buckets - self.data = [[] for _ in buckets] - - # pre-allocate with the largest bucket for better memory sharing - self.default_bucket_key = max(buckets) - - for sentence in sentences: - sentence = self.text2id(sentence, vocab) - if len(sentence) == 0: - continue - for i, bkt in enumerate(buckets): - if bkt >= len(sentence): - self.data[i].append(sentence) - break - # we just ignore the sentence it is longer than the maximum - # bucket size here - - # convert data into ndarrays for better speed during training - data = [np.zeros((len(x), buckets[i])) for i, x in enumerate(self.data)] - for i_bucket in range(len(self.buckets)): - for j in range(len(self.data[i_bucket])): - sentence = self.data[i_bucket][j] - data[i_bucket][j, :len(sentence)] = sentence - self.data = data - - # Get the size of each bucket, so that we could sample - # uniformly from the bucket - bucket_sizes = [len(x) for x in self.data] - - print("Summary of dataset ==================") - for bkt, size in zip(buckets, bucket_sizes): - print("bucket of len %3d : %d samples" % (bkt, size)) - - self.batch_size = batch_size - self.make_data_iter_plan() - - self.init_states = init_states - self.init_state_arrays = [mx.nd.zeros(x[1]) for x in init_states] - - if self.time_major: - self.provide_data = [mx.io.DataDesc('data', (self.default_bucket_key, batch_size), layout='TN')] + init_states - self.provide_label = [mx.io.DataDesc('softmax_label', (self.default_bucket_key, batch_size), layout='TN')] - else: - self.provide_data = [('data', (batch_size, self.default_bucket_key))] + init_states - self.provide_label = [('softmax_label', (self.batch_size, self.default_bucket_key))] - - def make_data_iter_plan(self): - "make a random data iteration plan" - # truncate each bucket into multiple of batch-size - bucket_n_batches = [] - for i in range(len(self.data)): - bucket_n_batches.append(len(self.data[i]) / self.batch_size) - self.data[i] = self.data[i][:int(bucket_n_batches[i]*self.batch_size)] - - bucket_plan = np.hstack([np.zeros(int(n), int)+i for i, n in enumerate(bucket_n_batches)]) - np.random.shuffle(bucket_plan) - - bucket_idx_all = [np.random.permutation(len(x)) for x in self.data] - - self.bucket_plan = bucket_plan - self.bucket_idx_all = bucket_idx_all - self.bucket_curr_idx = [0 for x in self.data] - - self.data_buffer = [] - self.label_buffer = [] - for i_bucket in range(len(self.data)): - if self.time_major: - data = np.zeros((self.buckets[i_bucket], self.batch_size)) - label = np.zeros((self.buckets[i_bucket], self.batch_size)) - else: - data = np.zeros((self.batch_size, self.buckets[i_bucket])) - label = np.zeros((self.batch_size, self.buckets[i_bucket])) - - self.data_buffer.append(data) - self.label_buffer.append(label) - - def __iter__(self): - for i_bucket in self.bucket_plan: - data = self.data_buffer[i_bucket] - i_idx = self.bucket_curr_idx[i_bucket] - idx = self.bucket_idx_all[i_bucket][i_idx:i_idx+self.batch_size] - self.bucket_curr_idx[i_bucket] += self.batch_size - - init_state_names = [x[0] for x in self.init_states] - - if self.time_major: - data[:] = self.data[i_bucket][idx].T - else: - data[:] = self.data[i_bucket][idx] - - label = self.label_buffer[i_bucket] - if self.time_major: - label[:-1, :] = data[1:, :] - label[-1, :] = 0 - else: - label[:, :-1] = data[:, 1:] - label[:, -1] = 0 - - data_all = [mx.nd.array(data)] + self.init_state_arrays - label_all = [mx.nd.array(label)] - data_names = ['data'] + init_state_names - label_names = ['softmax_label'] - - data_batch = SimpleBatch(data_names, data_all, [x.layout for x in self.provide_data], - label_names, label_all, [x.layout for x in self.provide_label], - self.buckets[i_bucket]) - yield data_batch - - - def reset(self): - self.bucket_curr_idx = [0 for x in self.data] diff --git a/example/rnn-time-major/get_sherlockholmes_data.sh b/example/rnn-time-major/get_sherlockholmes_data.sh deleted file mode 100755 index 43c8669e0..000000000 --- a/example/rnn-time-major/get_sherlockholmes_data.sh +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env bash - -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -echo -echo "NOTE: To continue, you need to review the licensing of the data sets used by this script" -echo "See https://www.gutenberg.org/wiki/Gutenberg:The_Project_Gutenberg_License for the licensing" -read -p "Please confirm you have reviewed the licensing [Y/n]:" -n 1 -r -echo - -if [ $REPLY != "Y" ] -then - echo "License was not reviewed, aborting script." - exit 1 -fi - -RNN_DIR=$(cd `dirname $0`; pwd) -DATA_DIR="${RNN_DIR}/data/" - -if [[ ! -d "${DATA_DIR}" ]]; then - echo "${DATA_DIR} doesn't exist, will create one"; - mkdir -p ${DATA_DIR} -fi - -wget -P ${DATA_DIR} https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/sherlockholmes/sherlockholmes.train.txt; -wget -P ${DATA_DIR} https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/sherlockholmes/sherlockholmes.valid.txt; -wget -P ${DATA_DIR} https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/sherlockholmes/sherlockholmes.test.txt; -wget -P ${DATA_DIR} https://raw.githubusercontent.com/dmlc/web-data/master/mxnet/tinyshakespeare/input.txt; diff --git a/example/rnn-time-major/readme.md b/example/rnn-time-major/readme.md deleted file mode 100644 index b30b8410b..000000000 --- a/example/rnn-time-major/readme.md +++ /dev/null @@ -1,24 +0,0 @@ -Time major data layout for RNN -============================== - -This example demonstrates an RNN implementation with Time-major layout. This implementation shows 1.5x-2x speedups compared to Batch-major RNN. - -As example of Batch-major RNN is available in MXNet [RNN Bucketing example](https://github.com/apache/incubator-mxnet/tree/master/example/rnn/bucketing) - -## Running the example -- Prerequisite: an instance with GPU compute resources is required to run MXNet RNN -- Make the shell script ```get_sherlockholmes_data.sh``` executable: - ```bash - chmod +x get_sherlockholmes_data.sh - ``` -- Run ```get_sherlockholmes_data.sh``` to download the sherlockholmes dataset, and follow the instructions to review the license: - ```bash - ./get_sherlockholmes_data.sh - ``` - The sherlockholmes data sets will be downloaded into ./data directory, and available for the example to train on. -- Run the example: - ```bash - python rnn_cell_demo.py - ``` - - If everything goes well, console will plot training speed and perplexity that you can compare to the batch major RNN. diff --git a/example/rnn-time-major/rnn_cell_demo.py b/example/rnn-time-major/rnn_cell_demo.py deleted file mode 100644 index 80b281b3b..000000000 --- a/example/rnn-time-major/rnn_cell_demo.py +++ /dev/null @@ -1,189 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""A simple demo of new RNN cell with Sherlock Holmes language model.""" - -################################################################################ -# Speed test (time major is 1.5~2 times faster than batch major). -# -# -- This script (time major) ----- -# 2016-10-10 18:43:21,890 Epoch[0] Batch [50] Speed: 1717.76 samples/sec Train-Perplexity=4311.345018 -# 2016-10-10 18:43:25,959 Epoch[0] Batch [100] Speed: 1573.17 samples/sec Train-Perplexity=844.092421 -# 2016-10-10 18:43:29,807 Epoch[0] Batch [150] Speed: 1663.17 samples/sec Train-Perplexity=498.080716 -# 2016-10-10 18:43:33,871 Epoch[0] Batch [200] Speed: 1574.84 samples/sec Train-Perplexity=455.051252 -# 2016-10-10 18:43:37,720 Epoch[0] Batch [250] Speed: 1662.87 samples/sec Train-Perplexity=410.500066 -# 2016-10-10 18:43:40,766 Epoch[0] Batch [300] Speed: 2100.81 samples/sec Train-Perplexity=274.317460 -# 2016-10-10 18:43:44,571 Epoch[0] Batch [350] Speed: 1682.45 samples/sec Train-Perplexity=350.132577 -# 2016-10-10 18:43:48,377 Epoch[0] Batch [400] Speed: 1681.41 samples/sec Train-Perplexity=320.674884 -# 2016-10-10 18:43:51,253 Epoch[0] Train-Perplexity=336.210212 -# 2016-10-10 18:43:51,253 Epoch[0] Time cost=33.529 -# 2016-10-10 18:43:53,373 Epoch[0] Validation-Perplexity=282.453883 -# -# -- ../rnn/rnn_cell_demo.py (batch major) ----- -# 2016-10-10 18:44:34,133 Epoch[0] Batch [50] Speed: 1004.50 samples/sec Train-Perplexity=4398.428571 -# 2016-10-10 18:44:39,874 Epoch[0] Batch [100] Speed: 1114.85 samples/sec Train-Perplexity=771.401960 -# 2016-10-10 18:44:45,528 Epoch[0] Batch [150] Speed: 1132.03 samples/sec Train-Perplexity=525.207444 -# 2016-10-10 18:44:51,564 Epoch[0] Batch [200] Speed: 1060.37 samples/sec Train-Perplexity=453.741140 -# 2016-10-10 18:44:57,865 Epoch[0] Batch [250] Speed: 1015.78 samples/sec Train-Perplexity=411.914237 -# 2016-10-10 18:45:04,032 Epoch[0] Batch [300] Speed: 1037.92 samples/sec Train-Perplexity=381.302188 -# 2016-10-10 18:45:10,153 Epoch[0] Batch [350] Speed: 1045.49 samples/sec Train-Perplexity=363.326871 -# 2016-10-10 18:45:16,062 Epoch[0] Batch [400] Speed: 1083.21 samples/sec Train-Perplexity=377.929014 -# 2016-10-10 18:45:19,993 Epoch[0] Train-Perplexity=294.675899 -# 2016-10-10 18:45:19,993 Epoch[0] Time cost=52.604 -# 2016-10-10 18:45:21,401 Epoch[0] Validation-Perplexity=294.345659 -################################################################################ - -import os -import numpy as np -import mxnet as mx - -from bucket_io import BucketSentenceIter, default_build_vocab - -data_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), 'data')) - - -def Perplexity(label, pred): - """ Calculates prediction perplexity - - Args: - label (mx.nd.array): labels array - pred (mx.nd.array): prediction array - - Returns: - float: calculated perplexity - - """ - - # collapse the time, batch dimension - label = label.reshape((-1,)) - pred = pred.reshape((-1, pred.shape[-1])) - - loss = 0. - for i in range(pred.shape[0]): - loss += -np.log(max(1e-10, pred[i][int(label[i])])) - return np.exp(loss / label.size) - - -if __name__ == '__main__': - batch_size = 128 - buckets = [10, 20, 30, 40, 50, 60] - num_hidden = 200 - num_embed = 200 - num_lstm_layer = 2 - - num_epoch = 2 - learning_rate = 0.01 - momentum = 0.0 - - # Update count per available GPUs - gpu_count = 1 - contexts = [mx.context.gpu(i) for i in range(gpu_count)] - - vocab = default_build_vocab(os.path.join(data_dir, 'sherlockholmes.train.txt')) - - init_h = [mx.io.DataDesc('LSTM_state', (num_lstm_layer, batch_size, num_hidden), layout='TNC')] - init_c = [mx.io.DataDesc('LSTM_state_cell', (num_lstm_layer, batch_size, num_hidden), layout='TNC')] - init_states = init_c + init_h - - data_train = BucketSentenceIter(os.path.join(data_dir, 'sherlockholmes.train.txt'), - vocab, buckets, batch_size, init_states, - time_major=True) - data_val = BucketSentenceIter(os.path.join(data_dir, 'sherlockholmes.valid.txt'), - vocab, buckets, batch_size, init_states, - time_major=True) - - def sym_gen(seq_len): - """ Generates the MXNet symbol for the RNN - - Args: - seq_len (int): input sequence length - - Returns: - tuple: tuple containing symbol, data_names, label_names - - """ - data = mx.sym.Variable('data') - label = mx.sym.Variable('softmax_label') - embed = mx.sym.Embedding(data=data, input_dim=len(vocab), - output_dim=num_embed, name='embed') - - # TODO(tofix) - # currently all the LSTM parameters are concatenated as - # a huge vector, and named '_parameters'. By default - # mxnet initializer does not know how to initilize this - # guy because its name does not ends with _weight or _bias - # or anything familiar. Here we just use a temp workaround - # to create a variable and name it as LSTM_bias to get - # this demo running. Note by default bias is initialized - # as zeros, so this is not a good scheme. But calling it - # LSTM_weight is not good, as this is 1D vector, while - # the initialization scheme of a weight parameter needs - # at least two dimensions. - rnn_params = mx.sym.Variable('LSTM_bias') - - # RNN cell takes input of shape (time, batch, feature) - rnn = mx.sym.RNN(data=embed, state_size=num_hidden, - num_layers=num_lstm_layer, mode='lstm', - name='LSTM', - # The following params can be omitted - # provided we do not need to apply the - # workarounds mentioned above - parameters=rnn_params) - - # the RNN cell output is of shape (time, batch, dim) - # if we need the states and cell states in the last time - # step (e.g. when building encoder-decoder models), we - # can set state_outputs=True, and the RNN cell will have - # extra outputs: rnn['LSTM_output'], rnn['LSTM_state'] - # and for LSTM, also rnn['LSTM_state_cell'] - - # now we collapse the time and batch dimension to do the - # final linear logistic regression prediction - hidden = mx.sym.Reshape(data=rnn, shape=(-1, num_hidden)) - - pred = mx.sym.FullyConnected(data=hidden, num_hidden=len(vocab), - name='pred') - - # reshape to be of compatible shape as labels - pred_tm = mx.sym.Reshape(data=pred, shape=(seq_len, -1, len(vocab))) - - sm = mx.sym.SoftmaxOutput(data=pred_tm, label=label, preserve_shape=True, - name='softmax') - - data_names = ['data', 'LSTM_state', 'LSTM_state_cell'] - label_names = ['softmax_label'] - - return sm, data_names, label_names - - if len(buckets) == 1: - mod = mx.mod.Module(*sym_gen(buckets[0]), context=contexts) - else: - mod = mx.mod.BucketingModule(sym_gen, - default_bucket_key=data_train.default_bucket_key, - context=contexts) - - import logging - head = '%(asctime)-15s %(message)s' - logging.basicConfig(level=logging.DEBUG, format=head) - - mod.fit(data_train, eval_data=data_val, num_epoch=num_epoch, - eval_metric=mx.metric.np(Perplexity), - batch_end_callback=mx.callback.Speedometer(batch_size, 50), - initializer=mx.init.Xavier(factor_type="in", magnitude=2.34), - optimizer='sgd', - optimizer_params={'learning_rate': learning_rate, - 'momentum': momentum, 'wd': 0.00001}) diff --git a/example/rnn/README.md b/example/rnn/README.md index a0846fa3d..1d1df6ed7 100644 --- a/example/rnn/README.md +++ b/example/rnn/README.md @@ -1,6 +1,11 @@ Recurrent Neural Network Examples =========== +For more current implementations of NLP and RNN models with MXNet, please visit [gluon-nlp](http://gluon-nlp.mxnet.io/index.html) + +------ + + This directory contains functions for creating recurrent neural networks models using high level mxnet.rnn interface. diff --git a/example/rnn/bucketing/README.md b/example/rnn/bucketing/README.md index 9bbeefd21..7b7883d79 100644 --- a/example/rnn/bucketing/README.md +++ b/example/rnn/bucketing/README.md @@ -2,6 +2,15 @@ RNN Example =========== This folder contains RNN examples using high level mxnet.rnn interface. +-------------- + +## Gluon Implementation + +You can check this improved [Gluon implementation](http://gluon-nlp.mxnet.io/model_zoo/language_model/index.html#word-language-model) in gluon-nlp, the largest LSTM model reaches a perplexity of 65.62. + +-------------- + + ## Data 1) Review the license for the Sherlock Holmes dataset and ensure that you agree to it. Then uncomment the lines in the 'get_sherlockholmes_data.sh' script that download the dataset. @@ -23,11 +32,11 @@ This folder contains RNN examples using high level mxnet.rnn interface. For Python2 (GPU support only): can take 50+ minutes on AWS-EC2-p2.16xlarge - $ python [cudnn_lstm_bucketing.py](cudnn_lstm_bucketing.py) --gpus 0,1,2,3 + $ python [cudnn_rnn_bucketing.py](cudnn_rnn_bucketing.py) --gpus 0,1,2,3 For Python3 (GPU support only): can take 50+ minutes on AWS-EC2-p2.16xlarge - $ python3 [cudnn_lstm_bucketing.py](cudnn_lstm_bucketing.py) --gpus 0,1,2,3 + $ python3 [cudnn_rnn_bucketing.py](cudnn_rnn_bucketing.py) --gpus 0,1,2,3 ### Performance Note: diff --git a/example/rnn/large_word_lm/data.py b/example/rnn/large_word_lm/data.py index b9cc3e8a8..0ca500628 100644 --- a/example/rnn/large_word_lm/data.py +++ b/example/rnn/large_word_lm/data.py @@ -174,7 +174,7 @@ def __init__(self, data_file, vocab, batch_size, bptt): self._iter = self._dataset.iterate_once(batch_size, bptt) def iter_next(self): - data = self._iter.next() + data = next(self._iter) if data is None: return False self._next_data = mx.nd.array(data[0], dtype=np.int32) diff --git a/example/rnn/large_word_lm/readme.md b/example/rnn/large_word_lm/readme.md deleted file mode 100644 index 465aaa1c4..000000000 --- a/example/rnn/large_word_lm/readme.md +++ /dev/null @@ -1,66 +0,0 @@ -# Large-Scale Language Model -This example implements the baseline model in -[Exploring the Limits of Language Modeling](https://arxiv.org/abs/1602.02410) on the -[Google 1-Billion Word](https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark) (GBW) dataset. - -This example reaches 48.0 test perplexity after 6 training epochs on a 1-layer, 2048-unit, 512-projection LSTM Language Model. -It reaches 44.2 test perplexity after 35 epochs of training. - -The main differences with the original implementation include: -* Synchronized gradient updates instead of asynchronized updates - -Each epoch for training (excluding time for evaluation on test set) takes around 80 minutes on a p3.8xlarge instance, which comes with 4 Volta V100 GPUs. - -# Setup dataset and build sampler -1. Download 1-Billion Word Dataset: [Link](http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz) -2. Download pre-processed vocabulary file which maps tokens into ids. -3. Build sampler with cython by running `make` in the current directory. If you do not have cython installed, run `pip install cython` - -# Run the Script -``` -usage: train.py [-h] [--data DATA] [--test TEST] [--vocab VOCAB] - [--emsize EMSIZE] [--nhid NHID] [--num-proj NUM_PROJ] - [--nlayers NLAYERS] [--epochs EPOCHS] - [--batch-size BATCH_SIZE] [--dropout DROPOUT] [--eps EPS] - [--bptt BPTT] [--k K] [--gpus GPUS] - [--log-interval LOG_INTERVAL] [--seed SEED] - [--checkpoint-dir CHECKPOINT_DIR] [--lr LR] [--clip CLIP] - [--rescale-embed RESCALE_EMBED] - -Language Model on GBW - -optional arguments: - -h, --help show this help message and exit - --data DATA location of the training data - --test TEST location of the test data - --vocab VOCAB location of the corpus vocabulary file - --emsize EMSIZE size of word embeddings - --nhid NHID number of hidden units per layer - --num-proj NUM_PROJ number of projection units per layer - --nlayers NLAYERS number of LSTM layers - --epochs EPOCHS number of epoch for training - --batch-size BATCH_SIZE - batch size per gpu - --dropout DROPOUT dropout applied to layers (0 = no dropout) - --eps EPS epsilon for adagrad - --bptt BPTT sequence length - --k K number of noise samples for estimation - --gpus GPUS list of gpus to run, e.g. 0 or 0,2,5. empty means - using gpu(0). - --log-interval LOG_INTERVAL - report interval - --seed SEED random seed - --checkpoint-dir CHECKPOINT_DIR - dir for checkpoint - --lr LR initial learning rate - --clip CLIP gradient clipping by global norm. - --rescale-embed RESCALE_EMBED - scale factor for the gradients of the embedding layer -``` - -To reproduce the result, run -``` -train.py --gpus=0,1,2,3 --clip=10 --lr=0.2 --dropout=0.1 --eps=1 --rescale-embed=256 ---test=/path/to/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050 ---data=/path/to/training-monolingual.tokenized.shuffled/* -``` diff --git a/example/rnn/word_lm/README.md b/example/rnn/word_lm/README.md index beed6fc8d..ab0a8d704 100644 --- a/example/rnn/word_lm/README.md +++ b/example/rnn/word_lm/README.md @@ -16,7 +16,7 @@ The Sherlock Holmes data is a copyright free copy of Sherlock Holmes from[(Proje Example runs and the results: ``` -python train.py --tied --nhid 650 --emsize 650 --dropout 0.5 # Test ppl of 75.4 +python train.py --tied --nhid 650 --emsize 650 --dropout 0.5 # Test ppl of 44.26 ``` ``` diff --git a/example/sparse/linear_classification/data.py b/example/sparse/linear_classification/data.py index 02984734f..bc5619a4b 100644 --- a/example/sparse/linear_classification/data.py +++ b/example/sparse/linear_classification/data.py @@ -24,10 +24,9 @@ def get_avazu_data(data_dir, data_name, url): os.mkdir(data_dir) os.chdir(data_dir) if (not os.path.exists(data_name)): - print("Dataset " + data_name + " not present. Downloading now ...") - import urllib + print("Dataset " + data_name + " not present. Downloading now ...") zippath = os.path.join(data_dir, data_name + ".bz2") - urllib.urlretrieve(url + data_name + ".bz2", zippath) + mx.test_utils.download(url + data_name + ".bz2", zippath) os.system("bzip2 -d %r" % data_name + ".bz2") print("Dataset " + data_name + " is now present.") os.chdir("..") diff --git a/example/speech_recognition/README.md b/example/speech_recognition/README.md index 74bad43a4..6f01911e1 100644 --- a/example/speech_recognition/README.md +++ b/example/speech_recognition/README.md @@ -28,7 +28,9 @@ With rich functionalities and convenience explained above, you can build your ow 
       pip install soundfile
      -- Warp CTC: Follow [this instruction](https://github.com/baidu-research/warp-ctc) to compile Baidu's Warp CTC. +- Warp CTC: Follow [this instruction](https://github.com/baidu-research/warp-ctc) to compile Baidu's Warp CTC. (Note: If you are using V100, make sure to use this [fix](https://github.com/baidu-research/warp-ctc/pull/118)) +- You need to compile MXNet with WarpCTC, follow the instructions [here](https://github.com/apache/incubator-mxnet/tree/master/example/ctc) +- You might need to set `LD_LIBRARY_PATH` to the right path if MXNet fails to find your `libwarpctc.so` - **We strongly recommend that you first test a model of small networks.** diff --git a/example/speech_recognition/label_util.py b/example/speech_recognition/label_util.py index dab1d1ef1..856373605 100644 --- a/example/speech_recognition/label_util.py +++ b/example/speech_recognition/label_util.py @@ -29,7 +29,7 @@ class LabelUtil: # dataPath def __init__(self): - self._log = LogUtil().getlogger() + self._log = LogUtil.getInstance().getlogger() self._log.debug("LabelUtil init") def load_unicode_set(self, unicodeFilePath): diff --git a/example/speech_recognition/log_util.py b/example/speech_recognition/log_util.py index e61407f5f..65c465811 100644 --- a/example/speech_recognition/log_util.py +++ b/example/speech_recognition/log_util.py @@ -17,48 +17,44 @@ import logging import logging.handlers +from singleton import Singleton +@Singleton +class LogUtil: -class SingletonType(type): - def __call__(cls, *args, **kwargs): - try: - return cls.__instance - except AttributeError: - cls.__instance = super(SingletonType, cls).__call__(*args, **kwargs) - return cls.__instance - - -class LogUtil(object): - __metaclass__ = SingletonType _logger = None _filename = None - def __init__(self, filename=None): - self._filename = filename - - # logger - self._logger = logging.getLogger('logger') - # remove default handler - self._logger.propagate = False - - stream_handler = logging.StreamHandler() - stream_formatter = logging.Formatter('[%(levelname)8s][%(asctime)s.%(msecs)03d] %(message)s', - datefmt='%Y/%m/%d %H:%M:%S') - stream_handler.setFormatter(stream_formatter) - - if self._filename is not None: - file_max_bytes = 10 * 1024 * 1024 - - file_handler = logging.handlers.RotatingFileHandler(filename='./log/' + self._filename, - maxBytes=file_max_bytes, - backupCount=10) - file_formatter = logging.Formatter('[%(levelname)8s][%(asctime)s.%(msecs)03d] %(message)s', - datefmt='%Y/%m/%d %H:%M:%S') - file_handler.setFormatter(file_formatter) - self._logger.addHandler(file_handler) - - self._logger.addHandler(stream_handler) - self._logger.setLevel(logging.DEBUG) - - def getlogger(self): + def getlogger(self, filename=None): + if self._logger is not None and filename is not None: + self._logger.warning('Filename %s ignored, logger is already instanciated with %s' % (filename, self._filename)) + if self._logger is None: + self._filename = filename + + # logger + self._logger = logging.getLogger('logger') + # remove default handler + self._logger.propagate = False + + stream_handler = logging.StreamHandler() + stream_formatter = logging.Formatter('[%(levelname)8s][%(asctime)s.%(msecs)03d] %(message)s', + datefmt='%Y/%m/%d %H:%M:%S') + stream_handler.setFormatter(stream_formatter) + + if self._filename is not None: + file_max_bytes = 10 * 1024 * 1024 + + file_handler = logging.handlers.RotatingFileHandler(filename='./log/' + self._filename, + maxBytes=file_max_bytes, + backupCount=10) + file_formatter = logging.Formatter('[%(levelname)8s][%(asctime)s.%(msecs)03d] %(message)s', + datefmt='%Y/%m/%d %H:%M:%S') + file_handler.setFormatter(file_formatter) + self._logger.addHandler(file_handler) + + self._logger.addHandler(stream_handler) + self._logger.setLevel(logging.DEBUG) + + return self._logger + diff --git a/example/speech_recognition/main.py b/example/speech_recognition/main.py index e45026343..b2ea42eca 100644 --- a/example/speech_recognition/main.py +++ b/example/speech_recognition/main.py @@ -38,6 +38,8 @@ os.environ['MXNET_ENGINE_TYPE'] = "ThreadedEnginePerDevice" os.environ['MXNET_ENABLE_GPU_P2P'] = "0" +logUtil = LogUtil.getInstance() + class WHCS: width = 0 height = 0 @@ -91,7 +93,7 @@ def load_data(args): max_duration = args.config.getfloat('data', 'max_duration') language = args.config.get('data', 'language') - log = LogUtil().getlogger() + log = logUtil.getlogger() labelUtil = LabelUtil.getInstance() if mode == "train" or mode == "load": data_json = args.config.get('data', 'train_json') @@ -276,7 +278,7 @@ def load_model(args, contexts, data_train): mx.random.seed(hash(datetime.now())) # set log file name log_filename = args.config.get('common', 'log_filename') - log = LogUtil(filename=log_filename).getlogger() + log = logUtil.getlogger(filename=log_filename) # set parameters from data section(common) mode = args.config.get('common', 'mode') diff --git a/example/speech_recognition/singleton.py b/example/speech_recognition/singleton.py index 01717e4df..fdb20c06b 100644 --- a/example/speech_recognition/singleton.py +++ b/example/speech_recognition/singleton.py @@ -18,23 +18,41 @@ import logging as log + class Singleton: + """ + A non-thread-safe helper class to ease implementing singletons. + This should be used as a decorator -- not a metaclass -- to the + class that should be a singleton. + + The decorated class can define one `__init__` function that + takes only the `self` argument. Also, the decorated class cannot be + inherited from. Other than that, there are no restrictions that apply + to the decorated class. + + To get the singleton instance, use the `instance` method. Trying + to use `__call__` will result in a `TypeError` being raised. + + """ + def __init__(self, decorated): - log.debug("Singleton Init %s" % decorated) self._decorated = decorated def getInstance(self): + """ + Returns the singleton instance. Upon its first call, it creates a + new instance of the decorated class and calls its `__init__` method. + On all subsequent calls, the already created instance is returned. + + """ try: return self._instance except AttributeError: self._instance = self._decorated() return self._instance - def __new__(cls, *args, **kwargs): - print("__new__") - cls._instance = super(Singleton, cls).__new__(cls, *args, **kwargs) - return cls._instance - def __call__(self): - raise TypeError("Singletons must be accessed through 'getInstance()'") + raise TypeError('Singletons must be accessed through `getInstance()`.') + def __instancecheck__(self, inst): + return isinstance(inst, self._decorated) \ No newline at end of file diff --git a/example/speech_recognition/stt_datagenerator.py b/example/speech_recognition/stt_datagenerator.py index 8fafa7909..e1f8f13b7 100644 --- a/example/speech_recognition/stt_datagenerator.py +++ b/example/speech_recognition/stt_datagenerator.py @@ -28,6 +28,8 @@ from stt_bi_graphemes_util import generate_bi_graphemes_label from multiprocessing import cpu_count, Process, Manager +logUtil = LogUtil.getInstance() + class DataGenerator(object): def __init__(self, save_dir, model_name, step=10, window=20, max_freq=8000, desc_file=None): """ @@ -86,7 +88,7 @@ def load_metadata_from_desc_file(self, desc_file, partition='train', max_duration (float): In seconds, the maximum duration of utterances to train or test on """ - logger = LogUtil().getlogger() + logger = logUtil.getlogger() logger.info('Reading description file: {} for partition: {}' .format(desc_file, partition)) audio_paths, durations, texts = [], [], [] @@ -245,7 +247,7 @@ def sample_normalize(self, k_samples=1000, overwrite=False): Params: k_samples (int): Use this number of samples for estimation """ - log = LogUtil().getlogger() + log = logUtil.getlogger() log.info("Calculating mean and std from samples") # if k_samples is negative then it goes through total dataset if k_samples < 0: diff --git a/example/speech_recognition/stt_io_iter.py b/example/speech_recognition/stt_io_iter.py index 6c9bacd1a..216dae8ac 100644 --- a/example/speech_recognition/stt_io_iter.py +++ b/example/speech_recognition/stt_io_iter.py @@ -86,7 +86,7 @@ def __init__(self, count, datagen, batch_size, num_label, init_states, seq_lengt audio_paths = audio_paths texts = texts - self.trainDataList = zip(durations, audio_paths, texts) + self.trainDataList = list(zip(durations, audio_paths, texts)) # to shuffle data if not sort_by_duration: random.shuffle(self.trainDataList) @@ -103,11 +103,11 @@ def __iter__(self): texts = [] for i in range(self.batch_size): try: - duration, audio_path, text = self.trainDataIter.next() + duration, audio_path, text = next(self.trainDataIter) except: random.shuffle(self.trainDataList) self.trainDataIter = iter(self.trainDataList) - duration, audio_path, text = self.trainDataIter.next() + duration, audio_path, text = next(self.trainDataIter) audio_paths.append(audio_path) texts.append(text) if self.is_first_epoch: diff --git a/example/speech_recognition/stt_metric.py b/example/speech_recognition/stt_metric.py index ec74fc063..26609627e 100644 --- a/example/speech_recognition/stt_metric.py +++ b/example/speech_recognition/stt_metric.py @@ -51,7 +51,7 @@ def __init__(self, batch_size, num_gpu, is_epoch_end=False, is_logging=True): def update(self, labels, preds): check_label_shapes(labels, preds) if self.is_logging: - log = LogUtil().getlogger() + log = LogUtil.getInstance().getlogger() labelUtil = LabelUtil.getInstance() self.batch_loss = 0. diff --git a/example/speech_recognition/stt_utils.py b/example/speech_recognition/stt_utils.py index 0539d59f3..cc0247223 100644 --- a/example/speech_recognition/stt_utils.py +++ b/example/speech_recognition/stt_utils.py @@ -15,16 +15,12 @@ # specific language governing permissions and limitations # under the License. -import logging import os import os.path import numpy as np -import soundfile from numpy.lib.stride_tricks import as_strided - - -logger = logging.getLogger(__name__) +import soundfile def calc_feat_dim(window, max_freq): diff --git a/example/speech_recognition/train.py b/example/speech_recognition/train.py index b1ae50b07..e585bfd05 100644 --- a/example/speech_recognition/train.py +++ b/example/speech_recognition/train.py @@ -51,7 +51,7 @@ def do_training(args, module, data_train, data_val, begin_epoch=0): from distutils.dir_util import mkpath from log_util import LogUtil - log = LogUtil().getlogger() + log = LogUtil.getInstance().getlogger() mkpath(os.path.dirname(get_checkpoint_path(args))) #seq_len = args.config.get('arch', 'max_t_count') diff --git a/example/ssd/README.md b/example/ssd/README.md index cc034689c..713a9ea33 100644 --- a/example/ssd/README.md +++ b/example/ssd/README.md @@ -4,6 +4,14 @@ SSD is an unified framework for object detection with a single network. You can use the code to train/evaluate/test for object detection task. +------------------- + +## Gluon Implementation + +You can find a Gluon implementation on [gluon-cv](https://gluon-cv.mxnet.io/build/examples_detection/train_ssd_voc.html). + +------------------- + ### Disclaimer This is a re-implementation of original SSD which is based on caffe. The official repository is available [here](https://github.com/weiliu89/caffe/tree/ssd). @@ -17,6 +25,7 @@ remarkable traits of MXNet. Due to the permission issue, this example is maintained in this [repository](https://github.com/zhreshold/mxnet-ssd) separately. You can use the link regarding specific per example [issues](https://github.com/zhreshold/mxnet-ssd/issues). ### What's new +* Support uint8 inference on CPU with MKL-DNN backend. Uint8 inference achieves 0.8364 mAP, which is a comparable accuracy to FP32 (0.8366 mAP). * Added live camera capture and detection display (run with --camera flag). Example: `./demo.py --camera --cpu --frame-resize 0.5` * Added multiple trained models. @@ -94,6 +103,7 @@ will open a window that will display the camera output together with the detecti with the detection threshold to get more or less detections. ### Train the model +* Note that we recommend to use gluon-cv to train the model, please refer to [gluon-cv ssd](https://gluon-cv.mxnet.io/build/examples_detection/train_ssd_voc.html). This example only covers training on Pascal VOC dataset. Other datasets should be easily supported by adding subclass derived from class `Imdb` in `dataset/imdb.py`. See example of `dataset/pascal_voc.py` for details. @@ -145,6 +155,43 @@ Make sure you have val.rec as validation dataset. It's the same one as used in t # cd /path/to/incubator-mxnet/example/ssd python evaluate.py --gpus 0,1 --batch-size 128 --epoch 0 ``` + +### Quantize model + +Follow the [Train instructions](https://github.com/apache/incubator-mxnet/tree/master/example/ssd#train-the-model) to train a FP32 `SSD-VGG16_reduced_300x300` model based on Pascal VOC dataset. You can also download our [SSD-VGG16 pre-trained model](http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/ssd_vgg16_reduced_300-dd479559.zip) and [packed binary data](http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/ssd-val-fc19a535.zip). Create `model` and `data` directories if they're not exist, extract the zip files, then rename the uncompressed files as follows (eg, rename `ssd-val-fc19a535.idx` to `val.idx`, `ssd-val-fc19a535.lst` to `val.lst`, `ssd-val-fc19a535.rec` to `val.rec`, `ssd_vgg16_reduced_300-dd479559.params` to `ssd_vgg16_reduced_300-0000.params`, `ssd_vgg16_reduced_300-symbol-dd479559.json` to `ssd_vgg16_reduced_300-symbol.json`.) + +``` +data/ +|---val.rec +|---val.lxt +|---val.idx +model/ +|---ssd_vgg16_reduced_300-0000.params +|---ssd_vgg16_reduced_300-symbol.json +``` + +Then, use the following command for quantization. By default, this script uses 5 batches (32 samples per batch) for naive calibration: + +``` +python quantization.py +``` + +After quantization, INT8 models will be saved in `model/` dictionary. Use the following command to launch inference. + +``` +# USE MKLDNN AS SUBGRAPH BACKEND +export MXNET_SUBGRAPH_BACKEND=MKLDNN + +# Launch FP32 Inference +python evaluate.py --cpu --num-batch 10 --batch-size 224 --deploy --prefix=./model/ssd_ + +# Launch INT8 Inference +python evaluate.py --cpu --num-batch 10 --batch-size 224 --deploy --prefix=./model/cqssd_ + +# Launch dummy data Inference +python benchmark_score.py --deploy --prefix=./model/ssd_ +python benchmark_score.py --deploy --prefix=./model/cqssd_ +``` ### Convert model to deploy mode This simply removes all loss layers, and attach a layer for merging results and non-maximum suppression. Useful when loading python symbol is not available. diff --git a/example/ssd/quantization.py b/example/ssd/quantization.py index 5cb74ba11..4e6e73996 100644 --- a/example/ssd/quantization.py +++ b/example/ssd/quantization.py @@ -119,12 +119,14 @@ def save_params(fname, arg_params, aux_params, logger=None): exclude_first_conv = args.exclude_first_conv excluded_sym_names = [] rgb_mean = '123,117,104' - calib_layer = lambda name: name.endswith('_output') for i in range(1,19): excluded_sym_names += ['flatten'+str(i)] excluded_sym_names += ['relu4_3_cls_pred_conv', 'relu7_cls_pred_conv', - 'relu4_3_loc_pred_conv'] + 'relu4_3_loc_pred_conv', + 'multibox_loc_pred', + 'concat0', + 'concat1'] if exclude_first_conv: excluded_sym_names += ['conv1_1'] @@ -139,7 +141,6 @@ def save_params(fname, arg_params, aux_params, logger=None): mean_args = {'mean_r': rgb_mean[0], 'mean_g': rgb_mean[1], 'mean_b': rgb_mean[2]} if calib_mode == 'none': - logger.info('Quantizing FP32 model %s' % args.model) qsym, qarg_params, aux_params = quantize_model(sym=sym, arg_params=arg_params, aux_params=aux_params, ctx=ctx, excluded_sym_names=excluded_sym_names, calib_mode=calib_mode, quantized_dtype=args.quantized_dtype, @@ -157,9 +158,9 @@ def save_params(fname, arg_params, aux_params, logger=None): ctx=ctx, excluded_sym_names=excluded_sym_names, calib_mode=calib_mode, calib_data=eval_iter, num_calib_examples=num_calib_batches * batch_size, - calib_layer=calib_layer, quantized_dtype=args.quantized_dtype, + calib_layer=None, quantized_dtype=args.quantized_dtype, label_names=(label_name,), - calib_quantize_op = True, + calib_quantize_op=True, logger=logger) sym_name = '%s-symbol.json' % ('./model/cqssd_vgg16_reduced_300') param_name = '%s-%04d.params' % ('./model/cqssd_vgg16_reduced_300', epoch) diff --git a/example/stochastic-depth/sd_cifar10.py b/example/stochastic-depth/sd_cifar10.py index 7eb320287..e995ea44f 100644 --- a/example/stochastic-depth/sd_cifar10.py +++ b/example/stochastic-depth/sd_cifar10.py @@ -78,9 +78,6 @@ import mxnet as mx import logging -sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) -from utils import get_data - import sd_module def residual_module(death_rate, n_channel, name_scope, context, stride=1, bn_momentum=0.9): @@ -199,13 +196,31 @@ def get_death_rate(i_res_block): batch_end_callbacks = [mx.callback.Speedometer(batch_size, 50)] epoch_end_callbacks = [mx.callback.do_checkpoint('sd-%d' % (n_residual_blocks * 6 + 2))] - -args = type('', (), {})() -args.batch_size = batch_size -args.data_dir = os.path.join(os.path.dirname(__file__), "data") +data_dir = os.path.join(os.path.dirname(__file__), "data", "cifar") kv = mx.kvstore.create(kv_store) -train, val = get_data.get_cifar10_iterator(args, kv) +mx.test_utils.get_cifar10() + +data_shape = (3, 28, 28) +train = mx.io.ImageRecordIter( + path_imgrec = os.path.join(data_dir, "train.rec"), + mean_img = os.path.join(data_dir, "mean.bin"), + data_shape = data_shape, + batch_size = batch_size, + rand_crop = True, + rand_mirror = True, + num_parts = kv.num_workers, + part_index = kv.rank) + +val = mx.io.ImageRecordIter( + path_imgrec = os.path.join(data_dir, "test.rec"), + mean_img = os.path.join(data_dir, "mean.bin"), + rand_crop = False, + rand_mirror = False, + data_shape = data_shape, + batch_size = batch_size, + num_parts = kv.num_workers, + part_index = kv.rank) logging.basicConfig(level=logging.DEBUG) mod_seq.fit(train, val, diff --git a/example/stochastic-depth/sd_mnist.py b/example/stochastic-depth/sd_mnist.py index 7eb93741f..6c95a23bf 100644 --- a/example/stochastic-depth/sd_mnist.py +++ b/example/stochastic-depth/sd_mnist.py @@ -25,9 +25,6 @@ import mxnet as mx import logging -sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) -from utils import get_data - import sd_module def get_conv( @@ -121,8 +118,7 @@ def get_conv( batch_size = 100 basedir = os.path.dirname(__file__) -get_data.get_mnist(os.path.join(basedir, "data")) - +mx.test_utils.get_mnist_ubyte() train = mx.io.MNISTIter( image=os.path.join(basedir, "data", "train-images-idx3-ubyte"), label=os.path.join(basedir, "data", "train-labels-idx1-ubyte"), diff --git a/example/svm_mnist/svm_mnist.py b/example/svm_mnist/svm_mnist.py index 679540198..3fc0362f6 100644 --- a/example/svm_mnist/svm_mnist.py +++ b/example/svm_mnist/svm_mnist.py @@ -20,16 +20,23 @@ ## Please read the README.md document for better reference ## ############################################################# from __future__ import print_function + +import logging +import random + import mxnet as mx import numpy as np from sklearn.datasets import fetch_mldata from sklearn.decomposition import PCA -# import matplotlib.pyplot as plt -import logging + logger = logging.getLogger() logger.setLevel(logging.DEBUG) +np.random.seed(1234) # set seed for deterministic ordering +mx.random.seed(1234) +random.seed(1234) + # Network declaration as symbols. The following pattern was based # on the article, but feel free to play with the number of nodes # and with the activation function @@ -41,60 +48,77 @@ fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10) # Here we add the ultimate layer based on L2-SVM objective -mlp = mx.symbol.SVMOutput(data=fc3, name='svm') +mlp_svm_l2 = mx.symbol.SVMOutput(data=fc3, name='svm_l2') + +# With L1-SVM objective +mlp_svm_l1 = mx.symbol.SVMOutput(data=fc3, name='svm_l1', use_linear=True) -# To use L1-SVM objective, comment the line above and uncomment the line below -# mlp = mx.symbol.SVMOutput(data=fc3, name='svm', use_linear=True) +# Compare with softmax cross entropy loss +mlp_softmax = mx.symbol.SoftmaxOutput(data=fc3, name='softmax') + +print("Preparing data...") +mnist_data = mx.test_utils.get_mnist() +X = np.concatenate([mnist_data['train_data'], mnist_data['test_data']]) +Y = np.concatenate([mnist_data['train_label'], mnist_data['test_label']]) +X = X.reshape((X.shape[0], -1)).astype(np.float32) * 255 # Now we fetch MNIST dataset, add some noise, as the article suggests, # permutate and assign the examples to be used on our network -mnist = fetch_mldata('MNIST original') -mnist_pca = PCA(n_components=70).fit_transform(mnist.data) +mnist_pca = PCA(n_components=70).fit_transform(X) noise = np.random.normal(size=mnist_pca.shape) mnist_pca += noise -np.random.seed(1234) # set seed for deterministic ordering p = np.random.permutation(mnist_pca.shape[0]) -X = mnist_pca[p] -Y = mnist.target[p] -X_show = mnist.data[p] +X = mnist_pca[p] / 255. +Y = Y[p] +X_show = X[p] # This is just to normalize the input and separate train set and test set -X = X.astype(np.float32)/255 X_train = X[:60000] X_test = X[60000:] X_show = X_show[60000:] Y_train = Y[:60000] Y_test = Y[60000:] - +print("Data prepared.") # Article's suggestion on batch size batch_size = 200 -train_iter = mx.io.NDArrayIter(X_train, Y_train, batch_size=batch_size, label_name='svm_label') -test_iter = mx.io.NDArrayIter(X_test, Y_test, batch_size=batch_size, label_name='svm_label') - -# Here we instatiate and fit the model for our data -# The article actually suggests using 400 epochs, -# But I reduced to 10, for convinience -mod = mx.mod.Module( - context = mx.cpu(0), # Run on CPU 0 - symbol = mlp, # Use the network we just defined - label_names = ['svm_label'], -) -mod.fit( - train_data=train_iter, - eval_data=test_iter, # Testing data set. MXNet computes scores on test set every epoch - batch_end_callback = mx.callback.Speedometer(batch_size, 200), # Logging module to print out progress - num_epoch = 10, # Train for 10 epochs - optimizer_params = { - 'learning_rate': 0.1, # Learning rate - 'momentum': 0.9, # Momentum for SGD with momentum - 'wd': 0.00001, # Weight decay for regularization - }, -) - -# Uncomment to view an example -# plt.imshow((X_show[0].reshape((28,28))*255).astype(np.uint8), cmap='Greys_r') -# plt.show() -# print 'Result:', model.predict(X_test[0:1])[0].argmax() - -# Now it prints how good did the network did for this configuration -print('Accuracy:', mod.score(test_iter, mx.metric.Accuracy())[0][1]*100, '%') + +ctx = mx.gpu() if len(mx.test_utils.list_gpus()) > 0 else mx.cpu() + +results = {} +for output in [mlp_svm_l2, mlp_svm_l1, mlp_softmax]: + + print("\nTesting with %s \n" % output.name) + + label = output.name + "_label" + + train_iter = mx.io.NDArrayIter(X_train, Y_train, batch_size=batch_size, label_name=label) + test_iter = mx.io.NDArrayIter(X_test, Y_test, batch_size=batch_size, label_name=label) + + # Here we instatiate and fit the model for our data + # The article actually suggests using 400 epochs, + # But I reduced to 10, for convenience + + mod = mx.mod.Module( + context = ctx, + symbol = output, # Use the network we just defined + label_names = [label], + ) + mod.fit( + train_data=train_iter, + eval_data=test_iter, # Testing data set. MXNet computes scores on test set every epoch + batch_end_callback = mx.callback.Speedometer(batch_size, 200), # Logging module to print out progress + num_epoch = 10, # Train for 10 epochs + optimizer_params = { + 'learning_rate': 0.1, # Learning rate + 'momentum': 0.9, # Momentum for SGD with momentum + 'wd': 0.00001, # Weight decay for regularization + }) + results[output.name] = mod.score(test_iter, mx.metric.Accuracy())[0][1]*100 + print('Accuracy for %s:'%output.name, mod.score(test_iter, mx.metric.Accuracy())[0][1]*100, '%\n') + +for key, value in results.items(): + print(key, value, "%s") + +#svm_l2 97.85 %s +#svm_l1 98.15 %s +#softmax 97.69 %s \ No newline at end of file diff --git a/example/svrg_module/README.md b/example/svrg_module/README.md index 63e7ba2f2..250995a57 100644 --- a/example/svrg_module/README.md +++ b/example/svrg_module/README.md @@ -1,7 +1,9 @@ ## SVRGModule Example + SVRGModule is an extension to the Module API that implements SVRG optimization, which stands for Stochastic Variance Reduced Gradient. SVRG is an optimization technique that complements SGD and has several key properties: + * Employs explicit variance reduction by using a different update rule compared to SGD. * Ability to use relatively large learning rate, which leads to faster convergence compared to SGD. * Guarantees for fast convergence for smooth and strongly convex functions. @@ -18,7 +20,9 @@ training script. ##### Dataset YearPredictionMSD: contains predictions of the release year of a song from audio features. It has over -400,000 samples with 90 features. Please uncomment data downloading script from data_reader.py to download the data. +400,000 samples with 90 features. It will be automatically downloaded on first execution and cached. + +YearPredictionMSD dataset: https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd #### Benchmarks: An initial set of benchmarks has been performed on YearPredictionDatasetMSD with linear regression model. A jupyter diff --git a/example/svrg_module/benchmarks/svrg_benchmark.ipynb b/example/svrg_module/benchmarks/svrg_benchmark.ipynb index db02938af..54ae81281 100644 --- a/example/svrg_module/benchmarks/svrg_benchmark.ipynb +++ b/example/svrg_module/benchmarks/svrg_benchmark.ipynb @@ -16,17 +16,28 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 1, "metadata": {}, "outputs": [], "source": [ - "import mxnet as mx\n", - "from sklearn.datasets import load_svmlight_file\n", - "import numpy as np\n", + "import os\n", "import json\n", + "import sys\n", "import tempfile\n", - "import os\n", - "from mxnet.contrib.svrg_optimization.svrg_module import SVRGModule\n" + "\n", + "import matplotlib.pyplot as plt\n", + "import matplotlib.patches as mpatches\n", + "import mxnet as mx\n", + "from mxnet.contrib.svrg_optimization.svrg_module import SVRGModule\n", + "import numpy as np\n", + "import pandas as pd\n", + "import seaborn as sns\n", + "from sklearn.datasets import load_svmlight_file\n", + "\n", + "sys.path.insert(0, \"../linear_regression\")\n", + "from data_reader import get_year_prediction_data\n", + "\n", + "%matplotlib inline" ] }, { @@ -39,47 +50,30 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Download data file\n", - "# from subprocess import call\n", - "# YearPredictionMSD dataset: https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd\n", - "# call(['wget', 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/YearPredictionMSD.bz2'])\n", - "# call(['bzip2', '-d', 'YearPredictionMSD.bz2'])" - ] - }, - { - "cell_type": "code", - "execution_count": 30, + "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ + "Extracting data...\n", "Reading data from disk...\n" ] } ], "source": [ - "feature_dim = 90\n", - "print(\"Reading data from disk...\")\n", - "train_features, train_labels = load_svmlight_file('YearPredictionMSD', n_features=feature_dim, dtype=np.float32)\n", - "train_features = train_features.todense()\n", - "\n", - "# normalize the data: subtract means and divide by standard deviations\n", - "label_mean = train_labels.mean()\n", - "label_std = np.sqrt(np.square(train_labels - label_mean).mean())\n", - "feature_means = train_features.mean(axis=0)\n", - "feature_stds = np.sqrt(np.square(train_features - feature_means).mean(axis=0))\n", - "\n", - "train_features = (train_features - feature_means) / feature_stds\n", - "train_labels = (train_labels - label_mean) / label_std\n", - "\n", + "feature_dim, train_features, train_labels = get_year_prediction_data()" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ "train_features = train_features[-5000:]\n", - "train_labels = train_labels[-5000:]" + "train_labels = train_labels[-5000:]" ] }, { @@ -91,7 +85,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 4, "metadata": {}, "outputs": [], "source": [ @@ -119,7 +113,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ @@ -162,7 +156,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 6, "metadata": {}, "outputs": [], "source": [ @@ -194,19 +188,6 @@ " " ] }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "import seaborn as sns\n", - "import matplotlib.pyplot as plt\n", - "import matplotlib.patches as mpatches\n", - "import pandas as pd" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -217,7 +198,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 7, "metadata": {}, "outputs": [], "source": [ @@ -227,7 +208,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -236,13 +217,13 @@ "Text(0.5,0,'Epochs')" ] }, - "execution_count": 32, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABKIAAALMCAYAAADXShqaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3X2cnXV9J/zP75xJZjITQElglcSUsMXyFAQLWAvS3nXFCBTs2qoUbPXWLbur1VK16G59qO3d7t1214eV2pat4l1F8bZ6rwqoi88oW+RJRcDyIEIIahLkKTCTzMzv/uOcTE4mM8lk5sxMJnm/X6/z4no61/WdE/76vL7f31VqrQEAAACA2daY7wIAAAAA2D8IogAAAACYE4IoAAAAAOaEIAoAAACAOSGIAgAAAGBOCKIAAAAAmBOCKABgwSilNEspj5dSVnXzWvZupZTXlFK+Ot91AAAzJ4gCAGZNOwja9hktpTzZsX/+nt6v1jpSa11aa72vm9fuqVLKn5VSLuv2faf47EYp5eJSyl3t3/NHpZT/q5SyeI6e/2/a/5aPj/ucPBfPBwAWtp75LgAA2HfVWpdu2y6l3JvkNbXWaya7vpTSU2sdnovaFrBLkvxakvOT3JjkqCSXJTk6yb/t5oN28e9xX6318G4+CwDYP+iIAgDmTbuz6IpSysdKKY8luaCU8txSyv8upTxcSnmwlPK+Usqi9vU9pZRaSjm8vf+R9vmrSymPlVKuK6Ws3tNr2+dfVEr5l1LKI6WU/15K+WYp5ZXT+JuOLaV8rV3/90opZ3WcO7uUcnv7+etKKRe1jx9aSrmq/Z2HSilfn+TeRyW5MMl5tdZ/rrUO11pvTfKbSc4upZxeSjmtlPJAKaXR8b3fKqXc1N5ulFL+Uynl7lLKxlLKx0spT22f+/n2b/aqUsp9Sb44jb//2naH1g3t3/LT2+7fPv/iUsr323/rl0spv9Bx7udKKf9fKWVDu7b37njr8u729+4ppZzRceLVpZR727/rPaWUl+9p3QDA3BBEAQDz7TeSXJ7koCRXJBlO8oYky5OcmmRtWuHLZH47yduSHJzkviR/uqfXllIOTfKJJG9uP/eHSU7Z0z+kPR73uSRXJjkkyUVJriil/Hz7kg8leXWt9YAkxyf5Wvv4m5Pc0/7O09o1TuTfJPlhrfWmzoO11nuTfDvJC5J8M8nWJL8y7u++vL39h0nOSnJ6kpVJNid537jnnJ5Wp9VZmZ7faX8OS1KSvDtJSilHJ/lIkt9P62+9JslnSymLSik9af1udyU5PMkz0vo32eaXk3wvybL2/f6hfc8Dk/y3JC9o/66nJvnuNOsGAGaZIAoAmG/X1lo/W2sdrbU+WWv9dke3zz1J/j47hirjfbLWekOtdWuSjyY5YRrXnp3kllrr/2yfe3eSjdP4W05NsjjJX9Vat7bHEK9Osq1DZ2uSY0opB9RaH+oIlLamFdqsqrVuqbV+bac7tyxP8uAk5x5MsrzWWpN8PMl5SVJKeUqSF7aPJa1Q7z/VWh+otQ4meWeSl3Z2UCV5R631iVrrk5M8a1W7M6nz09tx/sO11ttqrZuTvD3Jy0sppf07fKbW+uX27/xfkhyY5DlJntv++y6utW5u/7/wzY573l1r/WCtdSTJh5OsLKUsb5+rSY4rpfTVWh+std42Sd0AwDwTRAEA8+3+zp1SylGllCtLKT8upTya5F1pBRST+XHH9hNJlk524S6uPayzjnaYs24KtY93WFrrJ9WOYz9KsqK9/RtJzklyXynlq6WU57SP/5f2dV9qj8y9eZL7b0zy9EnOPT3bw7PLk7ykPdL4kiT/XGvd9vesSqsL6eFSysNpdRnVJId23GuHf5MJ3Fdrfcq4z9Ak3/9Rkt60utAOa+8nSWqto2n9zivS6oC6tx00TWT8v12SLK21PppW6PbaJD8upXyulPLM3dQPAMwTQRQAMN/quP2/S3Jrkp+vtR6YVkdNmeUaHkxrTC1JazGibA+P9sT6JM9of3+bVUkeSJJ2p9c5aYU+n0u7S6nW+mit9aL2AuAvTnJxKWWiLrAvJVldSnl258H2Olgnt8+n1vrd9t/0wuw4lpe0gp8XjAuR+mqtY0HPuCBtOp7Rsb0qyVCSh9L6fX6uo+5GWr/7A2mFVz9XSmnu6cNqrVfXWv9NWmHcXWn9PwQA7IUEUQDA3uaAJI8k2dxeU2hX60N1y+eSPLuU8uvttYrekNYaRrvSLKX0dXx6k3wrrTWu3the9+jXkpyZ5BOllCWllN8upRzYHkt7LMlIkrSf+6/bAdYj7eM7dQbVWm9P8j+SfKyUckoppVlKOS7JJ5NcXWv9asflH0trjarnts9v87dJ/ryUsqr97ENLKefswW81Fb/T7mwbSPInST7RDrc+keScUsqvtru13pzW7/DPSa5LsqldW3/79zp1dw8qpTy9/fv1J9mS1ppXk3VVAQDzTBAFAOxt3pjkd9MKKP4urQXMZ1Wt9SdJXpbWotebkvzrJDen1ckzmQuSPNnx+UF7PO3Xk5yb1pjc+5L8dq31X9rf+d0kP2qPHL46ySvax38hyZeTPJ7WYuPvrbVeO8lz/0NaayR9LK3Q5eok/yvJS8ddd3mSX0vyv2qtP+s4/t+SfD6tMcDH0grPTt7F3zmRVaWUx8d9Xtxx/h/TWpT8wSTNJH+QJLXW77d/gw8k2ZDWQvTntNfTGk5rra6j0+qOui+ttwHuTjOtQOvBtP7tfjnJ6/bw7wEA5kiZeec1AMC+pT0etj7Jb9ZavzHf9SwkpZRrk/yPWutl810LALD30REFAJCklLK2lHJQe8TubWmN2F0/z2UBAOxTBFEAAC2nJbknrZG6tUlePO5NcAAAzJDRPAAAAADmhI4oAAAAAOZEz3wXMNeWL19eDz/88PkuAwAAAGCfceONN26stR6yu+v2uyDq8MMPzw033DDfZQAAAADsM0opP5rKdUbzAAAAAJgTgigAAAAA5oQgCgAAAIA5sd+tEQUAAADsX7Zu3Zp169ZlcHBwvktZ8Pr6+rJy5cosWrRoWt8XRAEAAAD7tHXr1uWAAw7I4YcfnlLKfJezYNVas2nTpqxbty6rV6+e1j2M5gEAAAD7tMHBwSxbtkwINUOllCxbtmxGnWWCKAAAAGCfJ4Tqjpn+joIoAAAAAOaENaIAAACA/cqaa2/Nhq3DXbvfIYt68r3Tjuva/fZlOqIAAACA/Uo3Q6jZuF+nkZGRWbv3fBBEAQAAAMyyzZs356yzzsqznvWsHHfccfnwhz+cl770pWPnv/rVr+bXf/3XkyRLly7N29/+9jznOc/Jddddl6uuuipHHXVUTjvttLz+9a/P2WefPelz3vnOd+Z3f/d3c8YZZ+Twww/Ppz71qfzRH/1R1qxZk7Vr12br1q1Jkre85S055phjcvzxx+dNb3pTkmTDhg15yUtekpNPPjknn3xyvvnNb3b9dzCaBwAAADDLPv/5z+ewww7LlVdemSR55JFH8ra3vS2bN2/OwMBArrjiirzsZS9L0gqtjjvuuLzrXe/K4OBgjjzyyHz961/P6tWrc9555+32WXfffXe+8pWv5Lbbbstzn/vc/NM//VP+8i//Mr/xG7+RK6+8Mqeffno+/elP54477kgpJQ8//HCS5A1veEMuuuiinHbaabnvvvvywhe+MLfffntXfwcdUQAAAACzbM2aNbnmmmty8cUX5xvf+EYOOuigrF27Np/97GczPDycK6+8Mueee26SpNls5iUveUmS5I477sgRRxyR1atXJ8mUgqgXvehFWbRoUdasWZORkZGsXbt2rIZ77703Bx54YPr6+vKa17wmn/rUp9Lf358kueaaa/K6170uJ5xwQs4555w8+uijeeyxx7r6O+iIAgAAAJhlz3zmM3PjjTfmqquuylvf+tacccYZednLXpZLLrkkBx98cE4++eQccMABSZK+vr40m80kSa11j5/V29ubJGk0Glm0aFFKKWP7w8PD6enpyfXXX58vfelL+fjHP573v//9+fKXv5zR0dFcd911WbJkSZf+6p3piAIAAACYZevXr09/f38uuOCCvOlNb8pNN92UX/3VX81NN92USy+9dGwsb7yjjjoq99xzT+69994kyRVXXDHjWh5//PE88sgjOfPMM/Oe97wnt9xyS5LkjDPOyPvf//6x67Yd7yYdUQAAAMB+5ZBFPV19090hi3Yfr3zve9/Lm9/85rEupQ984ANpNps5++yzc9lll+XDH/7whN9bsmRJ/uZv/iZr167N8uXLc8opp8y43sceeyznnntuBgcHU2vNu9/97iTJ+973vrz2ta/N8ccfn+Hh4Zx++un527/92xk/r1OZTovXQnbSSSfVG264Yb7LAAAAAObI7bffnqOPPnq+y5i2xx9/PEuXLk2tNa997Wtz5JFH5qKLLpq3eib6PUspN9ZaT9rdd43mAQAAAOzFLr300pxwwgk59thj88gjj+TCCy+c75KmzWgeAAAAwF7soosu2qkD6kMf+lDe+9737nDs1FNPzSWXXDKXpe0xQRQAAADAAvOqV70qr3rVq+a7jD1mNA8AAACAOSGIAgAAAGBOCKIAAAAAmBOCKAAAAADmhMXKAQAAgP3KN659TrZs2di1+y1evDzPO+2fu3a/fZmOKAAAAGC/0s0QajbuN97hhx+ejRtn9xlzRRAFAAAAwJzYb4OokZGh3H//Zbn+2+fm2m+emptuOj8/+cnnUmud79IAAACAfczmzZtz1lln5VnPelaOO+64XHHFFbnqqqty1FFH5bTTTsvrX//6nH322UmSTZs25YwzzsiJJ56YCy+8cJdZxb333pujjjoqr3nNa3Lcccfl/PPPzzXXXJNTTz01Rx55ZK6//vokyde+9rWccMIJOeGEE3LiiSfmscceS5L81V/9VU4++eQcf/zxecc73jHrv8N+GUSNjAzmlu+8Mv9y55/mscduzdDQj/Ozh/93bv3+G/KDH7xNGAUAAAB01ec///kcdthh+c53vpNbb701a9euzYUXXpirr7461157bTZs2DB27Z/8yZ/ktNNOy80335xzzjkn99133y7vfdddd+UNb3hDvvvd7+aOO+7I5ZdfnmuvvTZ//dd/nT//8z9Pkvz1X/91Lrnkktxyyy35xje+kSVLluSLX/xi7rzzzlx//fW55ZZbcuONN+brX//6rP4O+2UQdd/9/5CHH75+wnMPrP9YNm366twWBAAAAOzT1qxZk2uuuSYXX3xxvvGNb+SHP/xhjjjiiKxevTpJct55541d+/Wvfz0XXHBBkuSss87KU5/61F3ee/Xq1VmzZk0ajUaOPfbYPP/5z08pJWvWrMm9996bJDn11FPzh3/4h3nf+96Xhx9+OD09PfniF7+YL37xiznxxBPz7Gc/O3fccUfuvPPO2fkB2vbLt+Y9uP6Tuzy//sFPZvny/2OOqgEAAAD2dc985jNz44035qqrrspb3/rWvOAFL9jl9aWUKd+7t7d3bLvRaIztNxqNDA8PJ0ne8pa35KyzzspVV12VX/qlX8o111yTWmve+ta35sILL5zGXzQ9+2VH1ODQg7s8PzS4fo4qAQAAAOba4sXL5/x+69evT39/fy644IK86U1vyre+9a3cc889Yx1LV1xxxdi1p59+ej760Y8mSa6++ur87Gc/m3GNd999d9asWZOLL744J510Uu6444688IUvzAc/+ME8/vjjSZIHHnggP/3pT2f8rF3ZLzui+nqfnicHJ5+v7O07bA6rAQAAAObS80775zl/5ve+9728+c1vTqPRyKJFi/KBD3wgDz74YNauXZvly5fnlFNOGbv2He94R84777w8+9nPzq/8yq9k1apVM37+e97znnzlK19Js9nMMccckxe96EXp7e3N7bffnuc+97lJkqVLl+YjH/lIDj300Bk/bzJlf1uY+6STTqqf/OT/mbvv+a+TXvOsZ/1Dli/71bkrCgAAAJg1t99+e44++uj5LmMnjz/+eJYuXZpaa1772tfmyCOPzEUXXTTfZe3WRL9nKeXGWutJu/vufjma94xnvDpPecpzJjy3YsX5WXbwr8xxRQAAAMD+5tJLL80JJ5yQY489No888sicrtU0X/bL0bxmszcnnvChPLD+E7n77r/MyMgTKaWZY499bw49ZO0eLQgGAAAAMB0XXXTRlDugNm3alOc///k7Hf/Sl76UZcuWdbu0WbNfBlFJ0mj05hkrX5GNG/5XHvrZN5M08q8OfdF8lwUAAADMglrrgm48WbZsWW655Zb5LiMzXeJpvxzN69Rs9idJat2a0dEt81wNAAAA0G19fX3ZtGnTjEOU/V2tNZs2bUpfX9+077HfdkRtsy2ISpKRkSfTaCyex2oAAACAblu5cmXWrVuXDRs2zHcpC15fX19Wrlw57e/v90FUo7lkbHtk5IksWnTQPFYDAAAAdNuiRYuyevXq+S6DGM1LT3NgbHtk5Il5rAQAAABg37bfB1HjO6IAAAAAmB37fRA1fo0oAAAAAGaHIGqHIGrzPFYCAAAAsG/br4OoWmvWDTXH9kdGdUQBAAAAzJb9Noj63w8/nuddf0f+4kcPjx17990/zI+eHJrHqgAAAAD2XftlEHXb40/m5d+5O3c9MZSh9I0df+DJR/Jvb74rP9s6PI/VAQAAAOyb9ssg6r0/+kkGR2uSZCi9Y8d7M5QHhrbmI+s3zVdpAAAAAPus/TKIumbTo2Pbgx0dUb0ZTJJ8qeM8AAAAAN2xXwZRI7WObXeO5vW1g6jhjvMAAAAAdMd+GUQ99ylLx7aHduiIGtrpPAAAAADdsV8GUa9ddWhKe3tw3BpRBzQbeeWK5fNTGAAAAMA+bL8Mok576gF579GrMtBs7LBY+VMaQ/no8UdkRd/ieawOAAAAYN+0XwZRSfLSpx2cW3752BzevzTD6UmSnHZQM6cYywMAAACYFfttEJUkB/Q0c/iS3rGuqNGRJ+e5IgAAAIB9134dRCVJf7ORwfaC5cMjT8xzNQAAAAD7rjkLokopa0spPyil3FVKecsE51eVUr5SSrm5lPLdUsqZ7ePnl1Ju6fiMllJOaJ/7avue284duqd1da4TJYgCAAAAmD09c/GQUkozySVJXpBkXZJvl1I+U2u9reOyP07yiVrrB0opxyS5KsnhtdaPJvlo+z5rkvzPWustHd87v9Z6w3Rr6282MtTuiBoRRAEAAADMmrnqiDolyV211ntqrVuSfDzJueOuqUkObG8flGT9BPc5L8nHullY52jeqCAKAAAAYNbMVRC1Isn9Hfvr2sc6vTPJBaWUdWl1Q/3+BPd5WXYOoj7UHst7WymlTPTwUsrvlVJuKKXcsGHDhh3OdY7m1dHB1Fqn+CcBAAAAsCfmKoiaKCAan/icl+SyWuvKJGcm+cdSylh9pZTnJHmi1nprx3fOr7WuSfK89ucVEz281vr3tdaTaq0nHXLIITuc6xzNS2pGRwf35O8CAAAAYIrmKohal+QZHfsrs/Po3auTfCJJaq3XJelLsrzj/Mszrhuq1vpA+7+PJbk8rRHAPTLQbI6N5iXJyMjmPb0FAAAAAFMwV0HUt5McWUpZXUpZnFao9Jlx19yX5PlJUko5Oq0gakN7v5Hkt9JaWyrtYz2llOXt7UVJzk5ya/ZQf8doXpKMjDy5p7cAAAAAYArm5K15tdbhUsrrknwhSTPJB2ut3y+lvCvJDbXWzyR5Y5JLSykXpTW298q6fcGm05Osq7Xe03Hb3iRfaIdQzSTXJLl0T2vrb3SO5nlzHgAAAMBsmZMgKklqrVeltQh557G3d2zfluTUSb771SS/NO7Y5iS/ONO6BjrempcIogAAAABmy1yN5u21dh7NE0QBAAAAzAZBVHP8aJ41ogAAAABmgyDKaB4AAADAnBBEGc0DAAAAmBP7fRA10GzuOJo3ajQPAAAAYDbs90HUkkbZMYga3jyP1QAAAADsu/b7IKqUktJcMravIwoAAABgduz3QVSSNDqDKGtEAQAAAMwKQVSSZrN/bFsQBQAAADA7BFFJehqCKAAAAIDZJohKsmiH0TxrRAEAAADMBkFUkoGeRRlMbxIdUQAAAACzRRCVpL/ZyFA7iBoWRAEAAADMCkFUkoFmI0PpS5IMDwuiAAAAAGaDICrbOqLaQZSOKAAAAIBZIYhKsqRjNG90VBAFAAAAMBsEUWmN5g22O6JGvTUPAAAAYFYIopL0N7aP5qVuyejo8PwWBAAAALAPEkQlGehpjo3mJcnoqK4oAAAAgG4TRGVcR1SSEQuWAwAAAHSdICqtt+YNCqIAAAAAZpUgKq3FyjtH8wRRAAAAAN0niEqrI2rH0TxrRAEAAAB0myAqrY4oo3kAAAAAs0sQlW0dUUbzAAAAAGaTICoTjeYJogAAAAC6TRCVZKDZHDeaZ40oAAAAgG4TRCXpb4wfzds8j9UAAAAA7JsEUUl6GiWjDR1RAAAAALNJENVWGv1j2yOj1ogCAAAA6DZBVFuz2RFEWawcAAAAoOsEUW3N5pKxbUEUAAAAQPcJotp6mgNj29aIAgAAAOg+QVRbX7M3I+2fQ0cUAAAAQPcJotr6e5oZSuvNeYIoAAAAgO4TRLUNNBsZSm+SZOuwIAoAAACg2wRRbf3NRgbbHVHDOqIAAAAAuk4Q1dbfaBjNAwAAAJhFgqi2ztG80VFvzQMAAADoNkFUW+doXh15IrXWea4IAAAAYN8iiGobaDbHOqKS0YyObpnXegAAAAD2NYKotv7m9jWikmR01DpRAAAAAN0kiGrrHM1LkuFhQRQAAABANwmi2joXK0+SER1RAAAAAF0liGrbaTRvxJvzAAAAALpJENW202jeyOZ5rAYAAABg3yOIausfN5qnIwoAAACguwRRbf2NHUfzRkasEQUAAADQTYKotoFmc4fRPEEUAAAAQHcJotrGj+YJogAAAAC6SxDV1tco2bpDR5Q1ogAAAAC6SRDVVkpJmv1j+zqiAAAAALpLENWh2Vgyti2IAgAAAOguQVSHZlMQBQAAADBbBFEdmj0DY9sjo9aIAgAAAOgmQVSHxdaIAgAAAJg1gqgOS5qLsiWLkiQjw4IoAAAAgG4SRHUY6GlkKH1JkmEdUQAAAABdJYjqsKTRyFB6kyRbBVEAAAAAXSWI6tDf3N4RZY0oAAAAgO4SRHUYaDYyOBZEeWseAAAAQDcJojq0OqJao3mjozqiAAAAALpJENVhoNkcG83L6GBqHZ3fggAAAAD2IYKoDv0do3mJ8TwAAACAbhJEdegczUssWA4AAADQTYKoDgMdb81LBFEAAAAA3SSI6tDfGDeaN2o0DwAAAKBb5iyIKqWsLaX8oJRyVynlLROcX1VK+Uop5eZSyndLKWe2jx9eSnmylHJL+/O3Hd/5xVLK99r3fF8ppcykxoGdRvM2z+R2AAAAAHSYkyCqlNJMckmSFyU5Jsl5pZRjxl32x0k+UWs9McnLk/xNx7m7a60ntD//vuP4B5L8XpIj25+1M6mzf6fRPB1RAAAAAN0yVx1RpyS5q9Z6T611S5KPJzl33DU1yYHt7YOSrN/VDUspT09yYK31ulprTfL/JHnxTIrsbzZ3GM0btUYUAAAAQNfMVRC1Isn9Hfvr2sc6vTPJBaWUdUmuSvL7HedWt0f2vlZKeV7HPdft5p5JklLK75VSbiil3LBhw4ZJixw/mjcsiAIAAADomrkKoiZau6mO2z8vyWW11pVJzkzyj6WURpIHk6xqj+z9YZLLSykHTvGerYO1/n2t9aRa60mHHHLIpEXuPJoniAIAAADolp45es66JM/o2F+ZnUfvXp32Gk+11utKKX1Jltdaf5pkqH38xlLK3Ume2b7nyt3cc4/0NxvjRvOsEQUAAADQLXPVEfXtJEeWUlaXUhantRj5Z8Zdc1+S5ydJKeXoJH1JNpRSDmkvdp5SyhFpLUp+T631wSSPlVJ+qf22vN9J8j9nUmSzlNTG9iDKaB4AAABA98xJR1StdbiU8rokX0jSTPLBWuv3SynvSnJDrfUzSd6Y5NJSykVpjdi9stZaSymnJ3lXKWU4yUiSf19rfah96/+Q5LIkS5Jc3f7MSKOxJBltbVusHAAAAKB75mo0L7XWq9JahLzz2Ns7tm9LcuoE3/unJP80yT1vSHJcN+tsNPuT4db2iNE8AAAAgK6Zq9G8BaPZHBjbHhnZPI+VAAAAAOxbBFHjLGouGdvWEQUAAADQPYKocRY3l2Q0JUkyMmqNKAAAAIBuEUSNM9DTzFB6kyTDw4IoAAAAgG4RRI3T32xkKH1Jkq3emgcAAADQNYKocfqbjQy2g6hhQRQAAABA1wiixhloNsZG80YEUQAAAABdI4gap7/ZHBvNGxVEAQAAAHSNIGqcgY41ourok/NcDQAAAMC+QxA1TmuNqNZoXupwRke3zG9BAAAAAPsIQdQ4/Y3tHVFJMjKiKwoAAACgGwRR4/Q3xwdR1okCAAAA6AZB1Dj9HW/NSwRRAAAAAN0iiBpnoHONqAiiAAAAALpFEDVOf7OZoSwZ27dGFAAAAEB3CKLG2Xk0b/M8VgMAAACw7xBEjTMwPoga1REFAAAA0A2CqHH6m40M7jCaZ40oAAAAgG4QRI2zU0eUNaIAAAAAukIQNc7iUrK1eGseAAAAQLcJosYppSSN/rF9QRQAAABAdwiiJtDsCKJGjeYBAAAAdIUgagLN5vYganhk8zxWAgAAALDvEERNoKdHRxQAAABAtwmiJrCoaY0oAAAAgG4TRE2gr2dxhtOTJBkWRAEAAAB0hSBqAv3NRgbTl0QQBQAAANAtgqgJ9DcbGUpvkmTrsCAKAAAAoBsEURMYaDbHgihrRAEAAAB0hyBqAv2NRgazJEkyMiqIAgAAAOgGQdQEBjpG80ZHnpznagAAAAD2DYKoCXSuEZXRJ1Nrnd+CAAAAAPYBgqgJdL41L6kZHR2c13oAAAAA9gWCqAm0RvP6xvYtWA4AAAAwc4KoCewwmhdBFAAAAEA3CKImMNBs6ogCAAAA6DJB1AR2XCMqGfHmPAAAAIAZE0RNYOfRvM3zWA0AAADAvkEQNYGdFyvXEQUAAAAwU4KoCfQ3xo/mWSMKAAAAYKZe304XAAAgAElEQVQEURMY8NY8AAAAgK4TRE1gyfjRvFGjeQAAAAAzJYiaQKOU1MaSsX0dUQAAAAAzJ4iaRKPRP7YtiAIAAACYOUHUJBpNHVEAAAAA3SSImkRPs7MjyhpRAAAAADMliJpET09nELV5HisBAAAA2DcIoiaxeIfRPB1RAAAAADMliJpEf09PBtObJBm2RhQAAADAjAmiJtHfaGSoHURtHTaaBwAAADBTgqhJDDSbGUpfEqN5AAAAAN0giJpEf7MxFkQZzQMAAACYOUHUJAaa20fzRgVRAAAAADMmiJpEf7ORwXZHVB01mgcAAAAwU4KoSSzpGM1L3ZLR0eH5LQgAAABggRNETaJzNC9JRnVFAQAAAMyIIGoSnYuVJ8mIdaIAAAAAZkQQNYmBZnNsjahEEAUAAAAwU4KoSfQ3dhzNGxkxmgcAAAAwE4KoSRjNAwAAAOguQdQkBpoNo3kAAAAAXSSImkR/02geAAAAQDcJoiYxsNNo3uZ5rAYAAABg4RNETaJ/p9E8HVEAAAAAMyGImsTiRiPDpWM0b9QaUQAAAAAzIYjahUajf2x7ZFgQBQAAADATgqhdKM2OIEpHFAAAAMCMzFkQVUpZW0r5QSnlrlLKWyY4v6qU8pVSys2llO+WUs5sH39BKeXGUsr32v/9tY7vfLV9z1van0O7WXNPc8nYtjWiAAAAAGamZy4eUkppJrkkyQuSrEvy7VLKZ2qtt3Vc9sdJPlFr/UAp5ZgkVyU5PMnGJL9ea11fSjkuyReSrOj43vm11htmo+5mZ0eUt+YBAAAAzMhcdUSdkuSuWus9tdYtST6e5Nxx19QkB7a3D0qyPklqrTfXWte3j38/SV8pHauIz6KeHYIoHVEAAAAAMzFXQdSKJPd37K/Ljl1NSfLOJBeUUtal1Q31+xPc5yVJbq61DnUc+1B7LO9tpZQy0cNLKb9XSrmhlHLDhg0bplx0X7M3I+2faGTEGlEAAAAAMzFXQdREAVEdt39ekstqrSuTnJnkH0spY/WVUo5N8n8nubDjO+fXWtckeV7784qJHl5r/fta60m11pMOOeSQKRfd39PMUPqSJFu9NQ8AAABgRuYqiFqX5Bkd+yvTHr3r8Ookn0iSWut1SfqSLE+SUsrKJJ9O8ju11ru3faHW+kD7v48luTytEcCuGWg2M7gtiNIRBQAAADAjcxVEfTvJkaWU1aWUxUlenuQz4665L8nzk6SUcnRaQdSGUspTklyZ5K211m9uu7iU0lNK2RZULUpydpJbu1l0f7ORobSWozKaBwAAADAzcxJE1VqHk7wurTfe3Z7W2/G+X0p5VynlnPZlb0zy70op30nysSSvrLXW9vd+Psnb2mtB3VJKOTRJb5IvlFK+m+SWJA8kubSbdQ80G2OjeYIoAAAAgJnpmasH1VqvSmsR8s5jb+/Yvi3JqRN878+S/Nkkt/3FbtY4Xn+zMTaaN+qteQAAAAAzMlejeQtSf2P7aF5Gn0yrQQsAAACA6RBE7cJAT0cQlZGMjm6Z13oAAAAAFjJB1C70NxoZzJKx/dFR60QBAAAATJcgahcGms2OjqhkxDpRAAAAANMmiNqF/mZjhyBqeGTzPFYDAAAAsLAJonahFUR1jObpiAIAAACYNkHULgw0GxncYTTPGlEAAAAA0yWI2oXxo3mCKAAAAIDpE0TtQiuI6hvbF0QBAAAATJ8gahf6m40M7hBEWSMKAAAAYLoEUbuwpDF+NM9b8wAAAACmSxC1C41Skmb/2L6OKAAAAIDpE0TtRqMsGdu2RhQAAADA9AmidqO5Q0eUIAoAAABgugRRuyGIAgAAAOgOQdRu9PR0jOaNWiMKAAAAYLoEUbuxWEcUAAAAQFcIonZjSc+ibMniJMnwsCAKAAAAYLoEUbsx0GxmKL1Jkq06ogAAAACmTRC1G/3NxlgQNTyyeZ6rAQAAAFi4BFG70d9sZDB9SZLhEYuVAwAAAEyXIGo3+huNDLWDqFFBFAAAAMC0CaJ2o3M0r1ojCgAAAGDaBFG7MdAxmpc6mFpH57cgAAAAgAVKELUbrY6ovrH9EeN5AAAAANMiiNqNgWZzbDQvSUZGBVEAAAAA0yGI2o3Ot+Ylycjw5nmsBgAAAGDhEkTtxk6jeTqiAAAAAKZFELUbAx1vzUuSUW/OAwAAAJgWQdRujB/NGxZEAQAAAEyLIGo3BsaN5umIAgAAAJgeQdRu9Dd2HM0bGbFGFAAAAMB0CKJ2o7/Z3KEjanjEW/MAAAAApkMQtRuLGiUjZcnY/qiOKAAAAIBpEURNQWlsD6JGrBEFAAAAMC2CqCloNAVRAAAAADMliJqCZrN/bNti5QAAAADTI4iagsYOQZSOKAAAAIDpEERNweIeo3kAAAAAMyWImoLFjf6MpiRJhgVRAAAAANMiiJqCgUXNDKU3SbJVEAUAAAAwLYKoKehvNDKUviTJ8PDmea4GAAAAYGESRE3BQLORwW1BlLfmAQAAAEyLIGoK+puNsdG8kVGjeQAAAADTIYiaglYQ1eqIGrVGFAAAAMC0CKKmYKAjiMqo0TwAAACA6RBETUF/s5nB9mhe6nBGR7fMb0EAAAAAC5Agagp26IhKMmLBcgAAAIA9Joiagv6dgijrRAEAAADsKUHUFPQ3tr81L9ERBQAAADAdgqgpGGg2tq8RlWRkZPM8VgMAAACwMAmipmBJs5GhLBnb1xEFAAAAsOcEUVPQWqy8oyNq1BpRAAAAAHtKEDUF/c3muDWiBFEAAAAAe0oQNQX9zUYGdxjNE0QBAAAA7ClB1BQsaRRvzQMAAACYIUHUFJRSUpo6ogAAAABmQhA1RY1G/9i2IAoAAABgzwmipqjR0RE1ajQPAAAAYI8Joqao2dERNTyyeR4rAQAAAFiYBFFTtKhnYGxbRxQAAADAnhNETVFP0xpRAAAAADMhiJqixWVLhtOTJHn40e/lkUe/M88VAQAAACwsgqgpuO/+D2XzQ1/OYPqSJINbfpobbvi3ufPOP0+tdZ6rAwAAAFgYBFG78cgjN+XOO/8sfRnMUHqTJDUlSXLf/f+Qn/zks/NZHgAAAMCCIYjajXXrPpIk6e0IonY4/8A/znVJAAAAAAuSIGo3Ht/8gyRJb4YymCVJkpLt43iPP/4v81IXAAAAwEIzZ0FUKWVtKeUHpZS7SilvmeD8qlLKV0opN5dSvltKObPj3Fvb3/tBKeWFU71nN/T0HJQk6ctgHslTkiTNjI6dX9Rz4Gw8FgAAAGCfMydBVCmlmeSSJC9KckyS80opx4y77I+TfKLWemKSlyf5m/Z3j2nvH5tkbZK/KaU0p3jPGftX/+rsJK3RvI1ZPul5AAAAAHZtWkFUKeV5pZRf3oOvnJLkrlrrPbXWLUk+nuTccdfUJNvaiw5Ksr69fW6Sj9dah2qtP0xyV/t+U7nnjD39aS/JgQeemL4MZtO4IGrJklVZterfdfuRAAAAAPukKQVRpZSvllKe195+U5JPJfmnUsrFU3zOiiT3d+yvax/r9M4kF5RS1iW5Ksnv7+a7U7nntvp/r5RyQynlhg0bNkyx5JZmszcnnnBZViw7LZtyyNjxpxx0Sn7xF//fLF588B7dDwAAAGB/NdWOqDVJrmtvX5jkV5M8J8l/nOL3ywTH6rj985JcVmtdmeTMJP9YSmns4rtTuWfrYK1/X2s9qdZ60iGHHDLRJbvU07M0/3rlb+4wmnfwstPSu3jnUT0AAAAAJtYzxesaSUZLKUck6am1fj9JSilTbQdal+QZHfsrs330bptXp7UGVGqt15VS+pIs3813d3fPrhloNvJQlo3tDw7O2qMAAAAA9klT7Yj6VpL3JPnLJJ9OknYotWmK3/92kiNLKatLKYvTWnz8M+OuuS/J89v3PjpJX5IN7eteXkrpLaWsTnJkkuuneM+u6W828rMcnNH2TzYkiAIAAADYI1MNol6ZZDDJD5K8vX3smCT/fSpfrrUOJ3ldki8kuT2tt+N9v5TyrlLKOe3L3pjk35VSvpPkY0leWVu+n+QTSW5L8vkkr621jkx2zyn+PXtsoNnMSOnJz/LUJMng0IOz9SgAAACAfVKpdcJllfZZJ510Ur3hhhv2+Hs/HtqaE771/byj/qc8Mz9IszmQXzn9OylloqWqAAAAAPYfpZQba60n7e66qb417w2llBPa26eUUu4ppfxLKeWUmRa6UPQ3Wz/VtgXLR0Y2Z3j40fksCQAAAGBBmepo3huT3Nve/i9JLknyX5O8bxZq2iv1N1o/1aZsf+ue8TwAAACAqZtqEPWUWuvDpZSlSU5I8p5a698lOWr2Stt71FrzuQ0PpyTZ1O6ISpLNTzwwf0UBAAAALDBTDaLWlVKek+SlSb5Rax0ppRyQZGT2Sts71Frzhz+4P//+th+lZvtoXpL8/Q+/my2jo/NXHAAAAMAC0jPF6/4oyWeTbEnyG+1jZyf59mwUtTf5/MZH8rEHHxrbf6gjiNq0eV0+9MDGXPiMQ+ejNAAAAIAFZUodUbXWz9VaD621rqy1bgufPp3kxbNX2t7h8o4QKtmxI2p5Nu50HgAAAICJTbUjKqWUI5K8PMmKJA8kuaLWevdsFba3WDe4ZYf9zVmawfSlL4NZlo07nQcAAABgYlPqiCqlnJnkO2ktVP5EkmclubmUctYs1rZXOKx38Y4HShlbsHxZNuaw3kXzUBUAAADAwjPVjqi/SPLiWuuXth0opfxakvckuXI2Cttb/PZhB+dLDz26w7GNWZ4VWZeD81B++2kHzVNlAAAAAAvLVN+atyrJV8cd+3r7+D7tRcsPym897ak7HNvWEdXIaM4/pM5HWQAAAAALzlSDqO8k+YNxx16f5LvdLWfv0ygl7z1qVd5/9Koc2d+bJNmUQ8bOj255cL5KAwAAAFhQpjqa9x+TfK6U8gdJ7kvyjCTDSc6ZrcL2Jo1S8ptPOzirl/TmrJvuzKYsGzs3OLh+HisDAAAAWDimFETVWm8rpfxCklOTHJZkfZJv1Vr3q1fGrehrLVy+bTQvSQaHdEQBAAAATMVUO6JSa92ajnWiSimLSyn31FqPmI3C9kaHLu7JolKysW4fzdMRBQAAADA1U10jaiIlyeFdqmNBaJSSp/UuykMdo3lDgigAAACAKZlJEJUk+90r41b0LspwWZSH85QkyeCQIAoAAABgKmYaRO13VrbXidrYfnOe0TwAAACAqdnlGlGllA9O97v7qsN6FyVpLVj+87kzw8OPZnj48fT0LJ3nygAAAAD2brsLkx7Yzfk/71YhC8Vkb85b2nPkfJUEAAAAsCDsMoiqtb5trgpZKCYKooYG12fpgCAKAAAAYFesEbWHVrRH8zZ2dkRZJwoAAABgtwRRe2jC0TxBFAAAAMBuCaL20IE9zRzQbGRT+615STI4JIgCAAAA2B1B1DSs6FucR3NgtqY1pjc4+OA8VwQAAACw99vdW/OSJKWU35nk1FCSdUmur7Vu7VpVe7nDehfljs2D2ViX5+l50GgeAAAAwBRMKYhK8ntJTk6yKa3gaUWS5UluTnJ4kq2llHNrrTfNRpF7m5Ud60Q9PQ9maOjHqXU0pWgwAwAAAJjMVJOTm5K8pdZ6WK31lFrriiQXJ/nnJIcl+Yck75+lGvc6K3pbQdTG9jpRtW7Nli0b57MkAAAAgL3eVIOoVyR537hj/z3J79RaR5P8RZJju1nY3uywvtbaUA9l2dgx43kAAAAAuzbVIOqnSV407tjaJBva271J9ps1orZ3RC0fO+bNeQAAAAC7NtU1ov4gyRWllJuT3J/kGUlOTPKy9vnnJvlA98vbO61od0Rtao/mJTqiAAAAAHZnSkFUrfXqUsrPJzkrrTWhvpzkt2qtP22f/0KSL8xalXuZp/cuSklrsfJtBFEAAAAAuzbVjqi0Q6cPzWItC8biRiOHLu7JpqHtQdSQIAoAAABgl6YURJVSfi7JnyY5IcnSznO11iNmoa693oq+xfnJluE8Vg/MAXnUGlEAAAAAuzHVjqjL01ob6j8neWL2ylk4VvQuzk15IhuzvBVEDT443yUBAAAA7NWmGkStSXJ6rXVkNotZSA4bW7B8WVbnnmzd+lBGRgbTbPbNc2UAAAAAe6fGFK+7Nsnxs1nIQrOyd3GSHd+cNzSkKwoAAABgMlPtiLozyRdKKZ9M8uPOE7XWd3W9qgVgRbsjauO4N+f196+er5IAAAAA9mpTDaIOTvKFJAe0P9vUrle0QBw21hG1YxAFAAAAwMSmFETVWl8x24UsNCvG1ojaPponiAIAAACY3KRBVCllZa11XXt71WTX1Vrvm43C9nbLF/Wkt1GycaSjI2pIEAUAAAAwmV11RN2e7WN496Y1hlfGXVOTNLtf1t6vlJLDehfl3ieekuH0pCfDOqIAAAAAdmFXb807qGN7UZLF7f92fhbPXml7vxW9i1NLIw9lWRJvzQMAAADYlUmDqFrraMf2yGSfuSlz77Sib9uC5a0ganBwfWrdb9dvBwAAANilKS1WXkr5uSR/muSEJEs7z9Vaj5iFuhaE7QuWt9aJGh0dytatD2Xx4mXzWRYAAADAXmlKQVSSy5Pcn+Q/J3li9spZWFb0tjqiNqZjwfLB9YIoAAAAgAlMNYhak+T0/X0Ub7ztHVGHjB0bHFqfA7NmvkoCAAAA2GvtarHyTtcmOX42C1mItnVEbRrXEQUAAADAzqbaEXVnki+UUj6Z5MedJ2qt7+p6VQvEYe2OqM7RvKFBb84DAAAAmMhUg6iDk3whyQHtzzb79SviBprNPLWnmYe2dnREDQmiAAAAACYypSCq1vqK2S5koVrRtzi3DvfniTqQ/mw2mgcAAAAwiUmDqFLKylrruvb2qsmuq7XeNxuFLRSH9S7KrY8/mY1ZllWCKAAAAIBJ7aoj6vZsH8O7N60xvDLumpqk2f2yFo4VfdsWLD8kq3Jftmz5aUZHh9Jo9M5zZQAAAAB7l129Ne+gju1FSRa3/9v5WTx7pS0MK3onWLB86CfzVQ4AAADAXmvSIKrWOtqxPfL/s3fn0XFf9f3/n3dG+25Zkhd5jRMnsZPYSZywBVpK+BLCEuiXQsLvS6G0hPbLVuBLC7QUSkvLUgjQUgq0hLUsBVpCCftaQkjiJHbiLd4X2bIt2dr3mbm/PyTLkq0tIM1I1vNxjk9m7r1z9R4vnMPr3Pv+jPcrO2XOXmdPRI1oWO71PEmSJEmSpPNMqVl5CCEJvBr4LaCGEVf0Yoy/MzOlzQ1nTkSdonZ4zCBKkiRJkiTpfBNdzRvpQ8DrgfuBJwDfBpYBv5ihuuaMMyeimj0RJUmSJEmSNKGpBlEvAm6KMX4QSA/99xbgaTNW2RyxqCCfZDjnal5fYw4rkiRJkiRJmp2mGkSVAIeGXneHEIpjjDuBa2amrLkjLxFYXJBPC9Vkhn47+zwRJUmSJEmSdJ6pBlG7gE1Drx8E/iqE8FbAxIXB63mZkKSVBYAnoiRJkiRJksYy1SDqjcCZp+i9GXgS8HvAH89EUXPNmYblZ/pE9fYeI8aYy5IkSZIkSZJmnUmDqKEn5q0FtgHEGB+LMf52jPHaGONPZ7i+OWHpOQ3L0+kuUqn2XJYkSZIkSZI060waRMUY08A/xhj7slDPnHTmRNQpaofHfHKeJEmSJEnSaFO9mvftEMLNM1rJHLZs6ETU6CfnGURJkiRJkiSNlDfFdQngGyGEXwBHgOEGSDHGV85EYXPJ0nN6RIEnoiRJkiRJks411SBqD/CBmSxkLqsfPhF19mpeX69PzpMkSZIkSRppwiAqhHBbjPFLMcZ3ZKuguagqL0lJMsGp1MLhMa/mSZIkSZIkjTZZj6hPZKWKOS6EQH1hPl2U0UcR4NU8SZIkSZKkc00WRIXp+kEhhJtCCI+FEPaGEN46xvwdIYQtQ792hxBah8afPmJ8SwihN4TwgqG5z4QQDoyY2zhd9T5e9YUFEMJwnyiDKEmSJEmSpNEm6xGVDCE8nQkCqRjjjyf7ISGEJPAx4JlAA/BACOGuGOOOEfu8ccT61wFXD43/BNg4NF4N7AW+P2L7t8QYvzZZDTOtvuhsw/J6GujrO0EmkyKRmGobLkmSJEmSpAvbZClJIfBvjB9EReCiKfyc64G9Mcb9ACGELwO3ADvGWX8b8M4xxl8EfCfG2D2Fn5lVZxuWn3lyXob+/pMUFS3NXVGSJEmSJEmzyGRX87pijBfFGFeP82sqIRRAPXBkxPuGobHzhBBWAquBsU5a3Qp86Zyx94QQHhm62lc4zp63hxA2hxA2NzU1TbHkx2dp4eCJqJFPzvN6niRJkiRJ0lmTBVHTZawTVXGctbcCX4sxpkdtEMIS4ErgeyOG3wZcBlwHVAN/PtaGMcZPxhg3xRg31dbWjrXkN7Zs6ERU8/CJKIMoSZIkSZKkkbLVrLwBWD7i/TJgvJRmrFNPAC8G/jPGOHBmIMbYGAf1AXcyeAUwJ+oLz72aB719jbkqR5IkSZIkadaZMIiKMZZP0895ALgkhLA6hFDAYNh017mLQgiXAguAe8fY4zbOCaiGTkkRQgjAC4Bt01Tv47Zk+GqeJ6IkSZIkSZLGkpVHusUYUyGE1zJ4rS4JfDrGuD2E8G5gc4zxTCh1G/DlGOOoa3shhFUMnqj62TlbfzGEUMvgya0twB/P3LeYWFEyQU1+Hqf7Fw6P9RlESZIkSZIkDctKEAUQY7wbuPucsb865/27xvnsQcZobh5j/J3pq/A3V1+UT/NAira4gEpa6O0ziJIkSZIkSTojW83K54UzfaKahq7neTVPkiRJkiTpLIOoaVRfNLpPVCrVTirVkcuSJEmSJEmSZg2DqGk05pPzen1yniRJkiRJEhhETav6osEgqnlkEGWfKEmSJEmSJMAgalrVF46+mgfQ54koSZIkSZIkwCBqWp05ETUyiOrpPZqrciRJkiRJkmYVg6hpVFuQR34InKJ2eOzQoU+wffub6ek5nMPKJEmSJEmScs8gahoFoCbRQTsVDJA3NJrh+In/4oHNv0t398EcVidJkiRJkpRbBlHTqK39IapSByGEUdfzAAYGWti77/25KUySJEmSJGkWMIiaRseP38VCmgFoHnE974zmph+SSnVluyxJkiRJkqRZwSBqGg0MtLCQUwA0UXfefCRNKtWe7bIkSZIkSZJmBYOoaVRaejELaQLgEKvPm8/Lq6KgoOa8cUmSJEmSpPnAIGoaLV3yImpCKwD7WXPefP3Sl5BI5Ge7LEmSJEmSpFnBIGoaFRUt5fo1rwTgMKvIEIbnqqufyurVb8hVaZIkSZIkSTlnEDXNNix9BgADoYBTYQkAiUQRG676FMlkYS5LkyRJkiRJyimDqGlWnpekIm/wt7Ux7yoAMpleenoO57IsSZIkSZKknDOImgH1hQUA7Ipn+0S1dzyaq3IkSZIkSZJmBYOoGbB0KIjamlo1PNbe/kiOqpEkSZIkSZodDKJmQH3R4JPxjrCCEAZDqQ5PREmSJEmSpHnOIGoGLCsaDJ/SIQ+KLwWgo2MHmUwql2VJkiRJkiTllEHUDFhamD/8uqfwMmCwYXlX995clSRJkiRJkpRzBlEzoH7oRBRAc97a4dcd7V7PkyRJkiRJ85dB1AwoCmH49Tc7lw6/bu+wYbkkSZIkSZq/DKKm2Y9OtfPCh/cMv7+3p5ZeigBo80SUJEmSJEmaxwyiptHJvgH+aNsBeuPZsRiSHGQ1AO0dO8lk+nJUnSRJkiRJUm4ZRE2jLzWepicTzxvfzxoAEqRoa38s22VJkiRJkiTNCgZR02hHV8+Y4we4ePj1sdYt2SpHkiRJkiRpVjGImkaVeckxx/cNnYgCSHVtz1Y5kiRJkiRJs4pB1DR6fl3VmOMnWUwXJQD0dm3LZkmSJEmSJEmzhkHUNHpKVdmYYVQMCY6EwVNRXV17SKfHvsInSZIkSZJ0ITOImkYhBD52+Ur+bPViagvyhseLEoFNi68HIMY0HZ07clWiJEmSJElSzhhETbP8ROBNqxaz5cnreXp1OQC9mUhN5VXDazraH81VeZIkSZIkSTljEDVDkiHwxMqy4feHwtkn57UbREmSJEmSpHnIIGoGbagoHn79SG8l+fnVALR3GERJkiRJkqT5xyBqBl1VXjL8emtnLxXlVwDQ3b2fVKojV2VJkiRJkiTlhEHUDKrOz2NFUQEAWzu6Ka840ycq0tGxPXeFSZIkSZIk5YBB1AzbMHQq6ljfALFo3fC41/MkSZIkSdJ8YxA1wzaUn+0TtT+sGX5tw3JJkiRJkjTfGETNsI0VI/pE9ZZSWLAIgA6DKEmSJEmSNM8YRM2wUQ3LO3oor7gSgJ7ewwwMtOaqLEmSJEmSpKwziJphFXlJ1hQXAkMNy8uvHJ7zep4kSZIkSZpPDKKyYMPQ9bym/hSposuHxztsWC5JkiRJkuYRg6gsGNmwfB8XDb/2yXmSJEmSJGk+MYjKgg0j+0T1FFFUtAyA9vZHclWSJEmSJElS1hlEZcGVZcWEoddb2rupGOoT1dd3nL6+ptwVJkmSJEmSlEUGUVlQmpfkkpIi4PyG5faJkiRJkiRJ84VBVJZsqBjsE9WSStNTdNnwuE/OkyRJkiRJ84VBVJZsHNEnak+0YbkkSZIkSZp/DKKyZGQQtbU7SUnJamCwYXmMMVdlSZIkSZIkZY1BVJasKysmOdSxfGv72T5RAwOn6OtrzGFlkiRJkiRJ2WEQlSXFyQSXlQ42LH+ks5vy8iuG57yeJ0mSJEmS5gODqCzaMHQ9rz2VobPAhuWSJEmSJGl+MYjKog0j+kQ9Fldz5re/wyBKkiRJkiTNAwZRWTQyiNraGSktvRgYvJpnw3JJkiRJknShM4jKosvLisgPgx3Lt3Z0UzHUsDyVaqOn53AuS5MkSZIkSZpxBlFZVJhIsK7sTMPyHspGNCzvsGG5JEmSJEm6wBlEZdmZ63nd6QxtoxqWP5KrkiRJkiRJkmgAgfMAACAASURBVLLCICrLNo7oE7UzvZwQ8gBo79iWq5IkSZIkSZKywiAqyzZUnA2itnSmKSu9FICOjm3EmM5VWZIkSZIkSTPOICrL1pYUUZQ427C8vGKwYXk63UV394FcliZJkiRJkjSjDKKyLD8RWF9WDMD2zh5Kys42LN9/4KN0de3NVWmSJEmSJEkzyiAqB840LO/NRDYf/u/h8ZMnv82v7nsW23f8PzKZ/lyVJ0mSJEmSNCMMonJgw4iG5Vt7i8+bP378P9m77/3ZLEmSJEmSJGnGGUTlwMYRDcv3ccmYa44e/RIDA+3ZKkmSJEmSJGnGGUTlwMUlhRSHwSfkHWDNmGsymV46OrdnsyxJkiRJkqQZZRCVA8kQuLSwC4BDrCQ1zh9DIuRnsyxJkiRJkqQZZRCVI1dXVgOQDvkcYcV58/n5C6mouCrbZUmSJEmSJM2YrAVRIYSbQgiPhRD2hhDeOsb8HSGELUO/docQWkfMpUfM3TVifHUI4b4Qwp4QwldCCAXZ+j6/qesW1g+/3sfa8+YvuuhPSSTmzNeRJEmSJEmaVFaCqBBCEvgY8GxgHXBbCGHdyDUxxjfGGDfGGDcC/wh8Y8R0z5m5GOPzR4y/D7gjxngJ0AL84Yx+kWm0ofzs0/IOhtFBVF3dc1lW/9JslyRJkiRJkjSjsnUi6npgb4xxf4yxH/gycMsE628DvjTRhiGEAPwO8LWhoc8CL5iGWrNidXEh5cnB3/4Tpc9i/bo7hudSqbZclSVJkiRJkjRjshVE1QNHRrxvGBo7TwhhJbAa+PGI4aIQwuYQwq9CCGfCpoVAa4wxNYU9bx/6/Oampqbf5HtMm0QIbCgvAeCx7n4qa59Lefl6AFpafkUq1ZHL8iRJkiRJkqZdtoKoMMZYHGftrcDXYozpEWMrYoybgJcCHw4hrHk8e8YYPxlj3BRj3FRbW/t46p5RGyoGg6h0hB2dPdTUPBOAGAc4dernuSxNkiRJkiRp2mUriGoAlo94vww4Ns7aWznnWl6M8djQf/cDPwWuBpqBqhBC3hT2nJXOnIgC2NLRTW3NjcPvm5p/mIuSJEmSJEmSZky2gqgHgEuGnnJXwGDYdNe5i0IIlwILgHtHjC0IIRQOva4BngLsiDFG4CfAi4aWvhz45ox+i2k2smH5lo5uysouo6ho8HbhqVM/JZMZyFVpkiRJkiRJ0y4rQdRQH6fXAt8DdgJfjTFuDyG8O4Qw8il4twFfHgqZzrgc2BxC2Mpg8PTeGOOOobk/B94UQtjLYM+of5vp7zKdVhQVsCAvCcDW9h5CCNQMnYpKpdppbb0/l+VJkiRJkiRNq7zJl0yPGOPdwN3njP3VOe/fNcbnfglcOc6e+xl8It+cFIYalv+0pYM93b10pdLU1txIQ8NngcHredXVT8lxlZIkSZIkSdMjW1fzNI4zDcsj8GhnD1VV15GXVwFAc9MPGX04TJIkSZIkae4yiMqxhfnJ4de3bz/I+w82U7bgtwDo7TtGZ+fOXJUmSZIkSZI0rQyicuh7zW28e+/ZB/2d7E/x4UMn+EjL5cNjPj1PkiRJkiRdKAyicqQrleZ1Ow6RGmPu56mrSJMPDF7PkyRJkiRJuhAYROXIt5paaU9nxpzrDcVsYz0AHZ3b6e09NuY6SZIkSZKkucQgKkeO9PZPOP8g1w2/9nqeJEmSJEm6EBhE5cjSwoIJ5x8aEUR5PU+SJEmSJF0IDKJy5Hl1VZQmx//tX125nPLyKwFoab2PgYH2bJUmSZIkSZI0IwyicqQiL8k/XLp8zD+AsmSCD166nNqaGwGIMcWpUz/Nan2SJEmSJEnTzSAqh164aAHfuuYSnltbSWXe2T+K59dVcXlZMbW1zxwes0+UJEmSJEma6wyicuzaylL+9YrVPPKUKygbuqp3T0snMUZKS9dSVLQcgFOnfkYmM3GDc0mSJEmSpNnMIGqWKEwkeMbCCgAO9fazs6uXEAK1tYPX89LpTlpa7stliZIkSZIkSb8Rg6hZ5Nk1lcOvv9PUBjDcJwq8nidJkiRJkuY2g6hZ5BkLKygIAYDvNA8GUZWVm8jLqwKgufmHxBhzVp8kSZIkSdJvwiBqFinPS3LDgjIAtnX2cLinj0Qij5qa3wagr+84HR3bclihJEmSJEnSr88gapa5ubZq+PV3m89cz/PpeZIkSZIkae4ziJplnlVTQRh6feZ6XnX1U0kkCoDB63mSJEmSJElzkUHULFNbkM91laUA3NfaRXN/iry8UhYseDIAnZ276Ok5kssSJUmSJEmSfi0GUbPQTUNPz8sAPzjl0/MkSZIkSdKFwSBqFnr2UBAF8J2mwSCqpuYZw2PNzT/Kek2SJEmSJEm/KYOoWWh1SSGXlRYB8LOWDrpSaQoL66io2AhAS8t9NB6/i/7+07ksU5IkSZIk6XExiJqlzpyK6stEfnK6gxgzJBPFQ7MZdux4I/fc8xQe2/1uMpmB3BUqSZIkSZI0RQZRs9Sza0dcz2tuY/+Bj9DSeu+oNZnYT0PDZ3ls97uyXJ0kSZIkSdLjZxA1S11ZVkx9YT4AP2huY/+hO8dde+zYV+npOZqt0iRJkiRJkn4tBlGzVAhh+FRUezrD9rhmgtUZTp/+eXYKkyRJkiRJ+jUZRM1iI5+et5nrJ1wbY2amy5EkSZIkSfqNGETNYk+oLKM6PwnAg+EJZAjjrq1aMHFQJUmSJEmSlGsGUbNYXiLwzIWDp6JaqOYAY1/Pq6m5kbLSS7JZmiRJkiRJ0uNmEDXLjbyet7P0ZefNh5Dk0rXvzmZJkiRJkiRJvxaDqFnut6rLKU4M/jFt5lqe9MQfsnbtO6lecAMAMaZpav5eLkuUJEmSJEmaEoOoWa44meDp1eUA7Onu42hcwvJlv8/l695HCIP9oxoaPmezckmSJEmSNOsZRM0Bz649ez3vu81tABQVLqa29iYAursPcOr0z3NSmyRJkiRJ0lQZRM0BNy6sIDn0wLy7m9qGx5cvf/nw64Yjn812WZIkSZIkSY+LQdQcsCA/jydXlQHwcEc3jX39AFRWXEN5+ZUAnDr9c7q69uesRkmSJEmSpMkYRM0RN9WMvJ7XDkAIgeXLRpyKavhc1uuSJEmSJEmaKoOoOeLZI4OoEdfzFi26mYKCGgAaj3+dVKoj67VJkiRJkiRNhUHUHLG0qICN5SUA/E9LB58/2szR3n4SiULql74UgHS6m2ONX8tlmZIkSZIkSeMyiJojUplI/lDD8gzwlt0NXHfvDt6w8zA1i19CCPkANBz5HDGmc1eoJEmSJEnSOAyi5oi/3X+MB9q7R41lgK8cP83bD/axqO5mAHp6D9N86qfZL1CSJEmSJGkSBlFzwKn+FHc2NI87/7UTLSRqbxt+33Dks9koS5IkSZIk6XExiJoD7mvrpC/GCdc8MLCaioqrATjdcg+dXXuyUZokSZIkSdKUGURdICKwfNnvD79vaPhc7oqRJEmSJEkag0HUHHB9ZRkFIUy45qkLyqmru4mCgjoAGhv/k4GBtmyUJ0mSJEmSNCUGUXNATUEeL69fOO78C+uquKikkESigGX1LwUgk+nhWONXs1WiJEmSJEnSpAyi5oi/WlPPH9TXkDxnPAG8/aKlw+/r628jhAIAGho+T4zp7BUpSZIkSZI0AYOoOSI/Efj7tcvY/OR1fPTyFTyvthKADPDV46eH1xUU1LB40XMB6O09SlPzD3NRriRJkiRJ0nkMouaYJYUFvHhxNR+4dDnFicE/vs8ca6Yvkxles2xE0/J9+z5Iw9F/p71jW9ZrlSRJkiRJGskgao6qys/jJUuqAWjqT/FfJ1qH58rLr6CocPC6Xnf3Ph577B088MAtPPjQbfT1ncxJvZIkSZIkSQZRc9irltUMv/5kw0lijAAcPfrv9PYdO299a+v9bNn6h/aNkiRJkiRJOWEQNYetKSnixoUVAGzv7OWXrZ1kMikOHvrncT/T2bmD5uYfZ6tESZIkSZKkYQZRc9yrl9UOv/5kQxM9PQfp6zs+4WdOt9w702VJkiRJkiSdxyBqjrthQRmXlxYB8P3mdg72Tv6ZQJjhqiRJkiRJks5nEDXHhRB41fLBU1ER+PdThRQVLZ/wM9ULn5qFyiRJkiRJkkYziLoA/G7dAhbm5wHwpeMt1Kx4/bhrS0rWsrD6adkqTZIkSZIkaZhB1AWgKJng5fULAehOZ/hx5gbWXvJXJJNl561NJguyXZ4kSZIkSRJgEHXBeMXSGgrCYO+nfzvaxJL63+epN9zLVVf+C+su/xDl5VcC0NGxjeMn7splqZIkSZIkaZ4yiLpA1BXm84JFVQA09A7wneY2kskSamufyZIlt3D5Ze+BoSbl+/Z9gHS6O4fVSpIkSZKk+cgg6gJy+7La4defamgaNVdevp6lS34PgL6+4xw89Ims1iZJkiRJkmQQdQG5oryEJ1cN9oW6v62Lh9tHn3q6aM2bh/tGHT78KXp6jma9RkmSJEmSNH8ZRF1gXr18/FNRhQU1rF71fwHIZPrYu+99Wa1NkiRJkiTNbwZRF5gbF1awqnjwyXh3nWyhsa9/1Pzy5a+guHgFACdPfpvW1s1Zr1GSJEmSJM1PBlEXmGQI/NFQr6hUhDsbmkfNJxKFXHLx24bf797zN8SYyWqNkiRJkiRpfjKIugDduria0uTgH+0/Hj5J/U+28NwHd/Otk60A1NQ8kwULngRAR8c2Ghu/kbNaJUmSJEnS/GEQdQE61jdAKhMBiEAa2Nzezau2H+RDB48TQmDtJe/gzB//vv3/QCrVmbN6JUmSJEnS/GAQdQF6596j9MU45twHDhznYE8fZWWXUl9/KwD9/U0cPPTxbJYoSZIkSZLmIYOoC8zJvgF+crpj3PkIfO14CwAXrf5T8vLKATh8+NP09BzORomSJEmSJGmeMoi6wDQPpCZd09Q/AEBBwUJWr3o9ADH286v7buInP72c++5/Dg1H/90m5pIkSZIkaVplLYgKIdwUQngshLA3hPDWMebvCCFsGfq1O4TQOjS+MYRwbwhhewjhkRDCS0Z85jMhhAMjPrcxW99ntqovzKcghAnXrC4uHH69aNFzCSEfgEymj0ymn87OXTz22DvYsfMtxHGu+EmSJEmSJD1eWQmiQghJ4GPAs4F1wG0hhHUj18QY3xhj3Bhj3Aj8I3DmUW7dwO/HGNcDNwEfDiFUjfjoW858Lsa4Zca/zCxXmZ/HLYuqxp0vDPB7i6uH3x86/EliHBhz7fHj/8Wp0z+b9holSZIkSdL8lK0TUdcDe2OM+2OM/cCXgVsmWH8b8CWAGOPuGOOeodfHgJNA7QzXO6e9++J61pcVjTm3qbKMhQV5AMQYaWz8zwn3amz8xoTzkiRJkiRJU5WtIKoeODLifcPQ2HlCCCuB1cCPx5i7HigA9o0Yfs/Qlb07QgiF535m6HO3hxA2hxA2NzU1/brfYc5YkJ/Ht65Zy/vWLuOpC8rYWF5MaXLwj/qe1k42t3UBEGOKVKp1wr36+5tnvF5JkiRJkjQ/ZCuIGqtp0XjNh24FvhZjTI/aIIQlwOeBP4hnu2i/DbgMuA6oBv58rA1jjJ+MMW6KMW6qrZ0fh6lKkgleXl/Df2y8mO9uupQPX7ZieO5tuxtIx0gikU9R4dKJ9yleOdOlSpIkSZKkeSJbQVQDsHzE+2XAsXHW3srQtbwzQggVwLeBv4wx/urMeIyxMQ7qA+5k8AqgxvDc2kqeuqAMgEc7e/j8sVMA1NffNuHnJpuXJEmSJEmaqmwFUQ8Al4QQVocQChgMm+46d1EI4VJgAXDviLEC4D+Bz8UY/+Oc9UuG/huAFwDbZuwbzHEhBN5zyTLyhs6mvXd/I6f6U6xY8UcsrH7auJ/r6Tky7pwkSZIkSdLjkZUgKsaYAl4LfA/YCXw1xrg9hPDuEMLzRyy9DfhyjHHktb0XA08DXhFC2DL0a+PQ3BdDCI8CjwI1wN/O+JeZw9aWFnH7sjoAWlNp/n5/I4lEAVdd9SnWr7uDhdVPo7z8SqqqnjD8mZ273k5396FclSxJkiRJki4gYXTmc+HbtGlT3Lx5c67LyJnOVJob7tvF8f4BAnD3tWu5uqLkvHV79v49hw//KwDl5Vew6dqvkkiM2QtekiRJkiTNcyGEB2OMmyZbl62reZolyvKSvPPiwQblkcHG5Zkxwsg1F72ZiorBg2cdHdvYu+8D2SxTkiRJkiRdgAyi5qEX1FXxpKpSALZ0dPOlxtPnrUkkCrhi/UfIyysH4MiRO2lq+mFW65QkSZIkSRcWg6h5KITA312yjORQ4/L37D9Gy0DqvHXFxcu4/LL3Dr/fsfPP6O0d72GHkiRJkiRJEzOImqcuLyvmlfU1AJweSPO+A8fHXFdXdxPL6l8GQCrVxrbtbyCTGchanZIkSZIk6cJhEDWPvWX1Emry8wD43NFm7jrRwi9aOjjW2z9q3cUXv42ysnUAtLU9xP4DH8l6rZIkSZIkae7zqXnz3FcaT/OGXYdHjQXgWTUVvH/tcuoK8wHo7j7A/Q/cQjrdBUBx8UoGBtooKlrMkiUvYln9S32qniRJkiRJ85RPzdOUrCsrIpwzFoHvNrfzoi376E5nACgpWc2la/96eE1PzyFSqVY6O3exZ8/fsnXr7WQy/UiSJEmSJI3HIGqe+/ChE4x3Jm53dy9fO372iXqR9Lj7nG75BUePfXmaq5MkSZIkSRcSg6h5LBMj329un3DN90bMNzZ+fcK1jY1fm5a6JEmSJEnShckgah6LwMAkPcL6Mpmzr/vGfrLe2fkT01GWJEmSJEm6QBlEzWPJELimomTCNddVlg6/Li5aPuHaoqJl01KXJEmSJEm6MBlEzXOvWVE37lwAfndR1fD7pUtfPOFeFeVXTVdZkiRJkiTpAmQQNc89p7aKd61ZSn4499l5g1f33nvgOHHo+l5d3c0sWfKicfc61vhlTrfcO1OlSpIkSZKkOS7ESXoEXWg2bdoUN2/enOsyZp2TfQN882QrpwdS1BXk8dFDJ2nsHwDgnWuW8idDJ6dijJxs+i7Hjn2F3t6jFBQsIpko5NTpnwKQTJZyzdVfoKLC01GSJEmSJM0XIYQHY4ybJl1nEKWxPNjWxQse3stAjCQDfG3jxTypqmzMtTFGdu16O8cavwpAXl4V117zJcrK1mazZEmSJEmSlCNTDaK8mqcxXVtZyl9fvBSAdIRXbz/Iib6BMdeGELjssr+lru5mAFKpVh7e8nJ6eg5nrV5JkiRJkjT7GURpXH9QX8PvLloAwMn+FK/efpCBzNgn6EJIsn7dB1lY/TQA+vtP8vDDL6ev7yQA8+3knSRJkiRJOp9X8zShrnSamx/cw2NdvQD8yfJa3nlx/bjr0+keHt7yctraHgSgoKCOvLxyurv3k5dXyeLFz2PVyv9LYeH4T+uTJEmSJElzi1fzNC1Kk0n+7YpVlCYH/6p8/EgT325qHXd9MlnMhqv+lbKyy4HBk1Hd3fuASCrVSkPD59m8+X/T13ciG+VLkiRJkqRZxCBKk7q4pIg7Llsx/P41Ow7xvx/ew9Pv38VLt+7jWydbyYw4WZefX8HqVa8fd7/evmPs23/HjNYsSZIkSZJmH4MoTcnz66q4fVkNAL2ZyD2tXezs6uXHpzt41faDvHbn4VFhVPOpH0+434kT3yKTGbv5uSRJkiRJujAZRGnKrq4oHXfuGyda+Orx08PvB/pPTbhXJtNLOt09bbVJkiRJkqTZzyBKU/blxtMTzn/h2NnwqaRk9YRr8/Iqycsrn5a6JEmSJEnS3GAQpSk71Ns3yXz/8OulS1/CRH+90ukeWlruna7SJEmSJEnSHGAQpSlbVJA/5fnS0jVcdum7gTDm2hj72bL1lRw79h/TWaIkSZIkSZrF8nJdgOaOlyyu5r62rnHnLy8tGvW+vv42Kio20HD0i3R17SE/r5K6uptpab2fxsavEmOKnbveSk/PIS666E2EYC4qSZIkSdKFLMQRTzqbDzZt2hQ3b96c6zLmpIFM5BWPHuBHp9vHnE8AH1+/klvqFky4T4yRw4c/yd597x8eW1T3XC6//P309TXS1r6FZKKI6uqn2EdKkiRJkqQ5IITwYIxx06TrDKL0ePRnMnz26Cm+2HiKI739LC3MZ2lhAT9r6QAgGeAT61bx3LqqSfc6cfJudux4M5nMYG+pvLwqUqnW4flEopjVq17DypV/TAhjX/GTJEmSJEm5ZxA1DoOomfH+A4186OAJAPICfHL9Km6unTyMamt7iK2P3M7AQMu4ay655C9ZsfwPpq1WSZIkSZI0vaYaRNmUR9PiLasW84aViwBIRbh9+0G+19w26ecqK69h7SXvnHDNoUP/MnxqSpIkSZIkzV02K9e0CCHw1tWLScfIPx0+SSrCH207yCfWr6Q/E3msq5eq/CTPq61iaVHBqM92de2ecO/+/mY6O3dRUXHVTH4FSZIkSZI0wwyiNG1CCPzFRUtIx8jHjzQxECOv3HZw1Jp37z3G/1u9mD9duWhE36f5dT1UkiRJkqT5yqt5mlYhBP5qzVJetmThmPNp4H0HjvMfJ872hFpQ/ZTJdqWra+/0FSlJkiRJknLCIErTLoTAyuKCCdf88+GTw68XVD2RysqJ+plFdux8C9u3v4lUqmOaqpQkSZIkSdlmEKUZsbm9a8L5XV29tKfSwGBwteGqf2Fh9dNGrQnkU1Fx9fD74ye+yX33P4eW1gcA6OjYyaHDn+LQ4U/R0bFjmr+BJEmSJEmabvaI0owoSEyeceYN94iC/PwFbNx4J52du2lrf5hkooiFC59Gfv4CTp++hx07/4y+vuP09h7loYdeSnHxCnp6Do7ar6bmGaxf9yHy8sqm++tIkiRJkqRp4IkozYhnLayYcL4yL0lXOn3eeFnZWuqXvoTFi28hP38BANXVT+EJ13+butpnD63KnBdCATQ3/4idu972m5YuSZIkSZJmiEGUZsTz6qq4qqx43Pm2VJpnbd7N1o7uKe2Xn1/FFVf8Ixev+fMJ1508eTfd3QcfT6mSJEmSJClLDKI0IwoSCb68cQ3Pr6sa9ZdsVVEBywrzATjWN8AtD+3h68dPT2nPEAIFBTWTrmtt2/zrlCxJkiRJkmaYPaI0Y6rz8/jk+lWcuHiAvd29VOXnsa60iM50htfuPMT3mtvpzURes/Mwj3b28Berl3Bfexeb27ooSiR4Vk0lq0sKR+0ZwuR/ZTPp3pn6SpIkSZIk6TcQYoy5riGrNm3aFDdv9sRMrmVi5B8OHudDB08Mj5UlE3SmM6PW/f7Shfz92mUkhxqb9/ef4hf3PIUYB8bdO5ksY81Fb6S+/v8jkcifmS8gSZIkSZKGhRAejDFummydV/OUE4kQ+LPVS/i3K1ZRMvSEvXNDKIDPHTvFPxw4Pvy+oGAhK5a/csK90+lOdu/5G+5/4HmcPv1Lensb2b37b7jnnqfy8/+5nq2P3E5Ly6+m9wtJkiRJkqRJeSJKOXfn0Wbetrth3PmKZIKHn7Ke0mQSgBgzHDjwUQ4f+TTpdBcAyWQp9fUvJRA4fOQzxNg//PkQCka9P+OyS99Dff2t0/xtJEmSJEmaf6Z6IsoeUcq5Y73nh0QjtaczPNbZyzWVpQCEkOCii/6UFSv+iLb2LQBUVmwgL68cgKVLX8yePX9H86kfA4wZQgE8tvtd1NTeSOEUGqBLkiRJkqTfnFfzlHP5iTDpmrwx1uTllbGw+gYWVt8wHEIBlJSsZsOGT3H55e+fcM8YBzhx/K7HX7AkSZIkSfq1GEQp5565sHLSNR89dIKm/vEblI+ltOSiSdf09h59XHtKkiRJkqRfn0GUcu7qihKeXTNxGPXfTW381v27+MaJFs70NcvEyP7uPvZ395Eeo9dZUdFSYOLTVo3Hv0FDwxfJZPqGx1KpLk6c+G8aGr5AS8t9zLc+apIkSZIkzRSblWtW6ElneOfeo3y58TT9Q38n6wry+N+LFvDtpjYOj+gj9b8WVvCkqlLuPHpqeHx5UQFvXLWIly5ZOGrfrVtfNdwraiKFhYtZueJ2Qshn7773kU53Ds+Vlq7lyis+Rmnp5CesJEmSJEmaj6barNwgSrPKqf4U2zp7KEwErqkooSCRoCud5r37G/nXhmYm+9v6rjVL+eMVdcPve3uP8eBDt9Hbe+5T+ZJUVV1HW9tmYkxNWldh4SKe+ITvjepFJUmSJEmSBhlEjcMgau66v7WTN+w6zIGe8Z+yV5JIsPUp6ynPSw6PDQy00NDwRZqavk8600tlxUaWL38F5eXr6Ok5wsFDH6ex8euTBlJrL3kHy5e/Yrq+jiRJkiRJFwyDqHEYRM1tXz9+mtfsPDzhmk+sX8ktdQse175dXXv51X3PmnBNTc2NbLjqE49rX0mSJEmS5oOpBlF52ShGmi4DUwhOWwfSj3vfoqL6Sde0tT7IiZPfobbmRhKJfABaWu7jWON/0NfbSFFRPUuXvpiqqkn/3UmSJEmSNC8ZRGlO2VBeMumaDx48TlEiwYsWLyAZBp+aF2PkUG8/vZkMq4oKKUqOfmBkMllMVeV1tLY9MO6+A6kWtm17LYUFi1i69Fb6+o9z7NhXRq1pPP51Vq54NWvWvIUQJn5inyRJkiRJ841X8zTnvHjLXn7e0jnpustKi3j7RUsIwN/tb2RnVy8AC/KSvHJZDW9cuZi8xNmw6PTpe3h4yyuAzHl7JRKFZDJ9U65xw1Wfoqbmd6a8XpIkSZKkuWyqV/MSky2QZpuPrVvJVWXF541fWlLI82urOBMt7erq5fcfPcDLHj0wHEIBtKTSfPDgCd6y+8ioz1dXP4UrrvgIBQU1o8YrK6/liU/4Addf998sXfoSEonzf/a5Go5+4fF/MUmSJEmSLnCeiNKclI6RH55q56enOwD4rQXl3LiwgrxEYGdnD3+3v5EfnGqfdJ+fXHcpl58TamUy/bS03MvAQBulpZdQXn75xIynHgAAIABJREFUqPmBgXbu+eVTSafHP5WVn1/NU578M5LJs1cJOzp3cfTol+ju2kt+QTWLF91CTc3vEIJ5sCRJkiRpbvOpeeMwiJo/vnr8FK/feWTCNW9etYi3rF7yuPe+7/7n0Nm5a8I1yWQJtTXPZPHiW+juPsTuPe8GRv97q619Fles/8hw83NJkiRJkuYin5qnee/i4qJJ12zt6CaViaN6RQEc6umjNZVmZVEBVfnn/zNZvOj57J0kiEqnuzl+4pscP/HNcdc0NX2PI0fuZOXK2yetVZIkSZKkuc47QbpgXVJaRHFi4r/iPzzVwRN+tYOPHz5JeyrNQ+1dPOfB3TzhVzt51ubdbLhnO2/cdZiOVHrU55Ytexnl5VeOuWdZ2eWsWPFqioqWTanOhqP/PuZ4X18Tp079jJbWB8hkBqa0lyRJkiRJs5lX83RBe8eeBj7V0DyltcWJQH8mkh5j7tqKEv7r6kvIH3FyKpXq4OChf+HYsa8yMHCagoIali59CatW/jHJZAkxRtraHuTRba+nv//EhD97w1Wforr6BhKJAlKpLh7b/U5OnPgWMaYAKCio4+I1f8aSJS+c8neXJEmSJClb7BE1DoOo+aU3neFPdhziO81to8brC/P5m0vq+X5zO9840UL/FP4d/Mu6lbxg0YLzxmOMxDhACPmEEM6bf3Tb6zl58tuT7p9MlrFw4W/T1bWHrq7Hxlyzfv2HWbzoeZPuJUmSJElSNhlEjcMgav6JMbK5vZu7m1rpzUSuqSjhebVVFCUHr+2d7BvgXxua+OjhkxPu87zaKj51xarzxjMx0p3OUJJMkBgjiGpq/hGPPDI9PaCKi1fxpCf+4Lwn7aVSHbS3PwIEKis3jnpanyRJkiRJM81m5dKQEALXVZZyXWXpmPN1hfm8fuWiSYOoX7Z28s2TLTxrYSVFyQRtAyk+dPAEXz5+mrZUmgV5SW5dUs2bVi2mPC85/LmahU+ntvYmmpq+e96ehYX1rL3kL2ht20xT0w/o7Z34KX89PQdpb99GZeVVAMSYZt/+O2g48lnSmW4A8vLKWbnidlau/JMxT2hJkiRJkpQrnoiSGDw19bT7d7Gnu2/StZV5SW6uqeS+tk729/SfN7+xvIRvXH0xJcmzp5YymRRHGu6koeGL9PYeIZksY/HiW1i96nUUFtYO17B9x5s4ceKuSSpIUFmxgerqG+js2jNmwAWwevUbuGj16yf9PpIkSZIk/aa8mjcOgyiN54vHTvHmx8Y+kZQAMo9jr7++eCmvXl435lwm0z9uP6kTJ7/Dtm2vfRw/aXyJRDFPveFe8vLKR/zsFKdO/Zim5h+RyfRTVXktixe/gLy8smn5mZIkSZKk+cmredLj9NIl1Rzo6eOfzrmiV5WX5NNXrGZBfpKvHD/N10+00NSfmnCvb5xoOS+I6kqnuetkK3u7+6jOz+MFdVXUFxWMWlNbcyMlJWvo7t435r4LFtxAzPTR1v7w8BP1xpPJ9HCk4QusXPGHJBIFDAy0s/WRP6St7aHhNSdO3MWBg//Exo2fobzssgn3kyRJkiTpN+WJKOkc+7v7+M8TLbSmUlxaWswL66ooHdHzKZWJXHHPo7Smxj8jlR8Cr11Rx821lVxRVswvWzv5o20HaUmlh9ckgD9fvYQ3rFo06rM9PQ088ugf09m5c8RoghXL/4CLL34rISRIpbrYu+/9HD36hUm/TyJRSEXFRgYGWujq2j3mmqKiep70xB+RSOSPGs9k+mlpuY9Uqp2ysssoLV0z6c+TJEmSJM0/s+5qXgjhJuAjQBL41xjje8+ZvwN4+tDbEqAuxlg1NPdy4C+H5v42xvjZofFrgc8AxcDdwBviJF/IIErT4cVb9vLzls4prV1SmE9T/wCpcf5m/su6lbxg0YJRYzFmaGm5l/b2R0gmi6mt/V8UFS0dtaazczf33f/sX6v+sVx5xT9TV/es4ffHT3yLPXv+lv7+5uGx6gU3sG7dBygsHPvaoSRJkiRpfppVQVQIIQnsBp4JNAAPALfFGHeMs/51wNUxxleGEKqBzcAmIAIPAtfGGFtCCPcDbwB+xWAQ9dEY43cmqsUgStPh7qZWXrnt4Ljz+SEwMMV/W1eVF/P9TZeOGuvLZLi7qY0t7d2UJBPcXFvJleUl531269ZX0Xzqx2PuW1X1BMrL19PWupn2jm1M1uWqqKieRXXPobziKtKpLnbu+vMx15WWXsL1132TRKJw1HiMkc7OXfT3n6SoaDmlpRdN+PMkSZIkSReO2dYj6npgb4xxP0AI4cvALcCYQRRwG/DOodfPAn4QYzw99NkfADeFEH4KVMQY7x0a/xzwAmDCIEqaDs+uqeQ1K+r42Dn9pADetGoRr1lex09Od/Cd5jb+62QL6QkyqUc6evjJqXaeVFVGUTLBrq4e/s8j+2noHRhec8ehE7xo0QLuuGwF+YmzTc7Xrfsg27a/ntOn/2fUnrW1N7F+3T+QTBYD0NK6mYceesmE36m39yiHDn9y0u/e1bWHkye/y+LFtwyPtbc/yq5df0FH5/bhsaqq67n8sr+npGTVpHtKkiRJkuaHbAVR9cDIx5E1AE8Ya2EIYSWwGjhzzGOsz9YP/WoYY3ysPW8HbgdYsWLF469eOkcIgXesWcrNNZV8qfE0R/v6WV5UwG1LFnJ1xeDJpefWVfHcuiraBlL88HTHhPvd9sh+ihOBTZWlbG3vpj19/umlr51oYVlRAW+9aMnwWH5+BVdv/Azt7Y9w+vQvISRYuPBp5zUer6q8huLiVfT0HJygiiSQnmD+rAMHP0aMKcrK1hFCkoce/j+k06OvKra23s+DD72UJ1z/LQoKFo6ay2QGOHXqZ3T3HKAgv4ba2htHPd1PkiRJknRhylYQdf5z6gev2Y3lVuBrMcYz/494vM9Oec8Y4yeBT8Lg1byJS5Wm7trKUq6tLJ1wzc11VZMGUQA9mcj/TNJ36s6jzbxh5SKKk4nhsV+0dPBPh0r5ResTSRD47bYC3rCya1RdISRYe8lfsvWRVzNW2LRq5Z+watVr6ezcwemWX7F//wcnrKO7ex87dv7Z0LsE41376+8/QcPRf+ei1a8bHmtre5hHt72Ovr7G4bHk7lIuXfvXLFnywgl/riRJkiRpbstWENUALB/xfhlwbJy1twKvOeezv33OZ386NL5sintKOfO7dQv4t4Ymtnf2njdXFODdlyzjUG8//3O6g0c6eybcqy2V5t37jvKc2iquLi/h+6fa+b87Do1IYCPfP9XOT053cOeVq7lxYcXwTE3N07l642fYf+DDtLU9CEBx0QpWrHwV9UtvI4RAZeU1VFRczYkTd9HVtWeK33Di3lMNDZ8lL1lKaenF5OVX8vDDryCdGR24pdNd7Nj5FgoLF1Fd/eRRcx0d2znS8Hk6OraTTJawqO5mli79PZLJ83tmSZIkSZJmt2w1K89jsFn5M4CjDDYrf2mMcfs56y4FvgesPvP0u6Fm5Q8C1wwte4jBZuWnQwgPAK8D7mOwWfk/xhjvnqgWm5UrF071p3j7ngb++2Tr8HmkDeXF/P0ly7hmxMmlt+9u4NNHm8fe5BxJBo8AjhcDLS3M5/4nriNvRE+ptoEUnz92ip83NZDO9LGuahmvXLaI1SWjG48fP34X23e8ccx9CwuXcsX6O+juPkhn504ajn6BGFNTqnky1dVP5eqNnxl+39j4dXbsfCvnfsuyssu45uovkp9fdd4efX0naG17iERIUlX1RPLzK85bI0mSJEmaXrPqqXkAIYSbgQ8z+P+fPx1jfE8I4d3A5hjjXUNr3gUUxRjfes5nXwm8fejte2KMdw6NbwI+AxQz2KT8dXGSL2QQpVxq6h/gYE8/1flJ1pQUnTf/UFsXNz801ZNIk/vEupU8v66KEAINvf387sN7OdzbP2pNUSLwmStX89vVowObI0c+w959HyCTOXuSq6zsMq684mOjGpBvfeTVNDf/cNpqrq5+KiUlq8nPr+bAgY8yXtS2ZMnvse7y9w6/T6f72L37XRxr/Dpnrh8mEkWsXPEqVq9+PSEkxtxn8H8y4rjzkiRJkqTJzbogarYwiNJsFmPkVdsP8t9NbefNJYCPr19JWTLJA21d3HWyhf09/edvco7agjyuKithb3cvh3rHXl+Vl+TBJ6+jNJkcHmtPpbnz0H72Hv8BMd1BYclabl71OzyjpnLUZ1tbN/PgQ7cxVmAUQiEbN3yKTKaPrq49HDr8aQYGpnbiazIh5HHtNV+hrGwtyWQJ27a/kRMn7hpz7erVfzqqTxUMPunvwMF/4tSpnxFjiqqq61i54nZqap4+LfVJkiRJ0nxiEDUOgyjNdv2ZDH+3v5EvHDtF59DT89aWFPGONUt45ogQ6OenO3jx1n3T9nM/cOkyXra0Bhi8SvjCh/eyu/v8vlZvXb2YP121eNTY8eN3seuxvyCd7h4ey89fwPp1d7Bw4VOHxw4d+iR7971v3BoSiWIymT4m6zt1rvz8BQwMtIw7n0yWcsNT7hl+Mt/plnvZsuWVxHh+MHfZpe+hvv7WUWPpdA+NjV/nxMm7Sac7KS9bz7JlL6O8fN3jqlOSJEmSLlQGUeMwiNJc0ZVOs7e7j5JEgotLCglh9IMiMzFyw3272N/TN+bnlxbm84TKUh7p6GHfOGvOtbyogMtKi/j/2Tvv8Diqc/9/Zrbvqq1675Ity02WC7bBdAjNIbTQIdQbSACHUAMJEEIgAQIhlFxCC4RieuimF+Pei5rVe2+r7TPn98dKK612V+TekPxyk/k8jx95jr575pwzs6vd777vezo83ojF1Sf4dMksymIsweOaMTc/r6mGoY+xM0gPacQlHcWvZhWRZTYGdT7fMJs2nxCyY94EsmxiceXL2GzFuFyt1Df8jt7e9/+mcf8tWK1FxMTMxmzOpLPzNXy+/og6WTZz8MqvMRjig2PeseM8Rh37piuZU3YPGRmnhLQKIRga2kRn1+t4vb1YLPlkZZ5JTEzptzYXDQ0NDQ0NDQ0NDQ2NfzU0IyoKmhGl8e/E3lEnZ+yqZ8CnhLRnGA28UlEUrEO1fWSM47d9e7WnTkqJ586SbFKNelrdXo7dWsugXwnT5ZqNrFtcSoJhcoNOl6uFzXt+gt+xI9gmjLlUzPkVSVN2zBsba2DjpqOjjiEmZjZ2+wpcrhZGRnbj9fZ8S7ODpKQjSE4+HLMpnY7Ol+ntXRdRJ0l6Viz/FLM5MzAPoVJd/TM6OtdOV1Jacis5OReE9THqqKaj40VcrlZMxlQyMk4lIeEbX7s1NDQ0NDQ0NDQ0NDT+pdCMqChoRpTGvxu9Xh/PdvTz1aADGTgsMZZzMpOwTzF/AI7dWsOuUVfEPnTAqsRY6p2esGLmM2HTyZhliX5fuAk1wc8KM/hxXlrw+KHmbn7V0EmWaCGdToZJoJ4STkxN5NE5eSG7/DU2PkRD4wNhfRqNaSyuXIvFkg3A6Oh+Nm85aYaRSuh0lpDUwW+LlORjyc4+B6MxhYHB9dTV3RlVu2Tx68TFzQ8eR0tVzM4+j9KSX4RFwY2O7qOt7TlGHVUY9HGkpZ1Ievp3kWVTWB8aGhoaGhoaGhoaGhr/TDQjKgqaEaXxn8quUSen7TjAqBJef+neWTmcm5kEwIDPR8X6/Xi+pdeGeL2OszISyTUb8aqC2+o7omrvLMnikuyU4HGD08Ovd/yF+Z43KKABFxY2s5zUrEv4aem8EKNm565L6O//NGK/uTkXU1x8E37/CENDm9m957++lbn9T0lIWE5R4TUYjcm4XK3s3HVhVO2cOfeRkX5y8LijYy1V1TcDodclLq6CioVPo9fHhLQPDG6gpeVPDA1tRZaNJCcfSX7e5VitBRHPJ4SK09mIQMVqKUCW9RF1GhoaGhoaGhoaGhoakdCMqChoRpTGfzIHnG4eau5hXd8wHiFYGmfjitxUViXGhuiur2nlzx2RaygBXJSVzIhfocHlYceIk2/rVSTDpOcv84vINBkwSRKrttTQGiVCa7pp5fAM8+a2H5PuXh9sU5Hoi/0upy+6G73OEGzfvuM8Bge/jtiv0ZjCvLmP4PX14nQ2U1//G6abP/8MzOY8Zs/6BQZDIqrqZtv2s4lWxD0n+0JKS28NHnd0vEJV9Q1hOp0uhkUVzxEXNy+kvbPzNRoaf4/b3QqAyZhGXt5lZGdfEBaVBeB2d9Lb9xGK4iQudh52+/KIOg0NDQ0NDQ0NDQ2N/xw0IyoKmhGlofHNDPj8nLLjANVj4QXLp6fanb2znk8GR7/1MRglCe8Mr0/pRj1bl89BL8sA/HBfE6/3DJElWpjNflR07KKCASmZn+an89OCyZ3+XK52Pt1yFiZ/e0ifPimGyoVPk2KvCLbt3nPljEXTi4puQJIkvJ5e2tqfG9/175+LJBkpKb4RozEZSTawd+/VEXcEBIiNncuSxW8EjaO29uepqbk1orag4GoKC64KHgshqK//Dc0tTwCT6ZgxMWXMn/dYMFVyAlX10d39Fl1db+D1DWK1FpCddQ52+7Koc/F4+/C4OzCZ0jCZ0qLqNDQ0NDQ0NDQ0NDT+tdCMqChoRpSGxt/GqF/h6fY+3ugZZMSvMifGzMVZKWHRU18OjHL6rvqIfRgkiTcXFWOWZVpcXn7d2EHN2Ldn1BgkiTSTngS9nr2OyPWvAKyyxK6Vc4nV6wB4oKmLBxoaOZRPWMg2dChUM4ePOYYj0gp4tDw/+Fins5UvtpyGQekL6zcx+4dUlP40eFxb9ytaW5+MOo6M9NOwWLLxevvp7nkHn2/gfzHrv5+UlGOxWHLR6Ww0N/8RVY28dpJk4OCV6zEaA2mbzS1/4sCBX0fUWq1FLFv6DrIciDxTVQ+7dl3GwOBXYdrCgjUUFPwopM3t7qCm9nb6+j5hIvIrKelQSkt+gdWaF9aHx9tHZ+erjDlq0RviSU9bTXz8wqhzFkLF6+1Flk0YDAlRdRoaGhoaGhoaGhoa/zs0IyoKmhGlofHt8+f2Pm6paw+JYLLpZP5QlstxKZMf+t/qGeLSfU1R+zkkIYZ5sVY6PF7WDzro9fm/tTHG63VkmQwkGvRsHHbgn+Gl74ulsym1BXYcfL6jnzuqd3ESr7OMr7HgoolCPuAE+qyH8enSWZjGo7I8nj4+2vQ9zP7wOlg+2xKOWfJs0Khpb3+R6pqfIYBISW3JyUdity/H5x2gt+8jxsZq/94l+F9hMCRiNmWg08cwPLwjaqQVBKLD0lKPQ6+PpbX1zzQ2PRhVu3jxa8THLQDA5xtk85aTcbvbwnRGYypLl7yJyZQabOvr+5Q9e38cZqBlZpzB7Nm/QpLkYJsQgra2P9PS+gRudyACLiF+CYVF12JPWBJ2PlX10dv3IQMDAQMt0b6SlJRjgtdtOkII3O52VNWNxZKLLBujzllDQ0NDQ0NDQ0Pj3xnNiIqCZkRpaPxj6PX6eL17kC6PnzyLkZNTE4iftnOfIgSX7G3ivb7hsMeXWE28taiEhPHHfNQ/wrm7G6KeL14vMy/GSrfXR4vbi0f99l7LjJJEmslAkkFH9Zgb9wx931WcxTlZSZhkmde7B7lh3y5O40VW8iVm3AyRwCccxTucyntL51EWYwHAr7h59eszSfbtCetzUM7m2INeJ9acCMDIyB62bD05qmllt68kJ/t8fL4h+ge+pKfn7W9jGf6h2GwlpKYch14fw+DQZvr6Poqqzc29lJLiG4FA5NSGjUejquFpowAlxT8jN/ei4HFd3V20tD4RppMkPQsXPEli4spgm8fTw86dF+IYq5k21lIqFj4dlirY3/85B+rvxeHYDwRMu+zs8ynIvwJJ0oVohVDo7f2Iru6/4vcNYbOVkJV1NjExpRHnIYTK8PB23O4OzJYs4uMWaXW4NDQ0NDQ0NDQ0/qXRjKgoaEaUhsb/X3yq4Im2Xv7c0U+Dy0OyQc/p6XauykvDPsW4UoXglB0H2Dg8FtaHBDwzr4BjkuMB2OdwceSWmjDdVP1sm5kBn59erz9Kye+/D6tOxqcKfBOvqUJFh4KCHsYNhFX2GK7MTSNer2Pb8Bi319Wzmtc4kg+JYwQXFr7iUF7lDNYUl3FFbiAKyK8K7t9wA5WeV8PO208SmeV/4eC0EgAUxcXnX61EKOFmH0BM/FIWzn0An28Ix1gt+/ZdE9Hgmmgzm7NQVS8+3/CM0VD/aAwGOzqdDUVx4fNFL6Sv1ydQXHwjer0Nv2+U6pqbo2pttlKWLX03aPBs2342Q0ObImoTEpZSueiF4HFf36fs2n0ZkQrIZ2acQVnZZAqjqnrZveeKCLs6ypTN/hWZmWeEtA4P72B/1XU4nY3BNqu1mPI5vyUubn6IVgiV9o4XaW//C2Nj9RgNiaSlryY/73IMBnvY2Pz+Ubq63mRkZBeyzkpqyrEzFpv3+YYZHtmBhEx8/KKw3RmnM/GeQjPNNDQ0NDQ0NDT+89CMqChoRpSGxr8OQogZP7CO+BVurG3jze7BYGnsLJOBXxRnsTo1tM7Pebsb+LB/JGI/l+ekcHtxFgADXh8Lv96Hd4aXvrkxFjyqSo/Xz7BfiS78thACCYFACppWsTqZU9LsxOl1dHq8vNI1SCnVLGEjuTQhIdhPOZ9wLLPiM3mrMhBZ41cFF3z9F8713okRX8hp+khmU/KD3D//oGDbe1suxjj6WUQzymWexwnLX0eSJIRQWb/hKDzu5vDhjz82LW01Bn08Pv8IPT3vIoQvTPuvhtGYil4fhyRJjI3VzajNy/svbNZCZNlC3YFf4fF0RdUuW/peMNqpsfEhGhofiKKUOWjZB9hshQA4nY1s3rIaRXGGKfW6WJYufTtYFF4Iwf6q6+jqej1Ma7UWUrnoJYzGxGDbyMhudu66OKw2WXLSEcyd+wd0OlOwTVX91Df8lra2Z4MF+HU6G7m5l1CQ/6OQ9EcAh6OGxqY/0Nf3CUL4iI9bRG7epaQkHxk2NiFU+vo+obvnbfz+EWJss8nKOguLJSfiCimKi96+j/B6erFYsklKOmzGFEifbwi3ux2DMQmzKT2qTkNDQ0NDQ0ND49tFM6KioBlRGhr/9+jy+KhyuIjR66iItaKXw82rUb/Cj6uaeb9v0oySgfMyk/hVSXbIY2470M5jrb0Rz3V4YiwvLCgCAh/0D91cTa0zeoH1Q+wxxOh0DPr8bBke459gW0XEKEnE6GV0SPT6/BiEh1S6SaQfAz6GsNNKLn7JxM2FGaSbDJhkiZuqavmu8hSzqMaEGzNuTLjpIJPfy7fy+cpDgumSD+16ljn9t4Wcd8KEqpHmc/7BLxMzrt20/w4cXc+EjXNCX1R6F4lxZfj9Dhoaf8/w8JaoczMYErFYcvD7x3C726Km5f1roUOvtyLLFrzefpjhzoiJmUNS4iHIOjP9fZ8xMrorqjYtdTWFhVchyyaGh7ezd9/VUbXZ2Rcwq/TnACiKk683HI7XG150HyAn5weUltwSPK6u+QXt7c9F1Obn/4iiwjXB4+HhHWzfcV7EovelJT8nJ+eC4LGqesajwz4L0UmSnvLyB0hLPS6kvafnA6qqb8Lvn4zwMxnTKC9/ALt9aYjW5xuktvaXdPe8gxCB+nJ2+wpKS26NmALZ0/sBra3PMDq6F53ORmrqceTnXR5xt0avd4COjrUMDW1CkvQkJR9ORvrJ6HTWiGs0MrqX/r7PEMJPQsIS7PYVUU13VfUyNLQFRRkjJqYsqiE3gd8/is83iNGYik5nnlGroaGhoaGhofHPRDOioqAZURoa/97UjLnZMOTAIEkclhhLljk8csKvCn5W18azHf0hiVVHJcXxyJw84vST9X3e7hnikigF1hfEWnhnUWnQ5LqhppVnOqKnjX0/PZHyGDNDfoWXuwZpdf//S3X7RoQASSJOJ5No1GOWZWrH3MSJAXJpJpYRTHgw4KWXVHazkONTUzkkMQaLLPOXtjbKRh4hizZMeDDiwYgXI142s4x5s+7gnKwUAN5vXo+u/gJkIv89Gs17jJOLjgagpXsddft+GLVelmpdyILCS1GUMQYGN9HVFZ7OOImE1VqIqnrw+YZQFMfft2b/csjExpYjyyb8viHGnAeiKiXJQEH+j9DpbaiKm/qG+yDK9ZAkYzDaSpIM7Nz1A8bGIqfGBnZe/AqjMRmA+ob7aWp6OKp2xfJPMJszARge3snWbWcQycTTyRaWLXsXiyUXCERNbd12Gg5HdZhWr49nyeLXQ3ZfjBalZjSmsbhybTDqDAKm0s6dF+LzDYZoLZZ8FlU8h9mcEWxTFDf79l9Lb+/7Idq4uAXMn/dHTKaUkPaurjepO3BXiEGYnHwUc8ruDkutdLlaqTtwN729HwIKOp2V9PRTKC76KXp96G6mquqnveN5OtpfxOVuxWhMISPjVHJzfhDRPBsdraKt/Vkcjmr0uljS0k4kPX01smwK0yqKi+7utxka3oosGUhOPpKkpFVhddEmcDhq6B/4AiFUEu3Lw9JLp/c9OLgRRXESFzcveH2j4XK14fMNYLHkRExFDV0TH37/CHp9XNTNBzQ0NDQ0NDT+PjQjKgqaEaWhoTFBq9vLZwMj+FTB8oSYYCHx6bzUOcAd9R30T9nF7+ikOB6YnUuScbKuVbvby3Hbaunxhu/2tyDWwpsVJZh1gZSm93uHuXBvY5hugpsK0jklPZFRv8ILnf083hY5kgXAIktUxtlwKCptbi993+Jug/9I9BKYZRmvKpBUJ0n0hxhWenx0kI1izOb4lHjMskS1w4U6+C45tGDAixEf+vGfChLrzRdx25zFmCSJPq+T2j0/IJm+YH8GfOjwo0Ow03AMPzn4ESRJwudz8OGXB2EiPKoHwIuRivJ7A1rvAFW1v0Qm+jrbbLPQ62NQFBcORxXRTJ3/BAyGRIzGJCTJwNhYDUJEjw6LjZ1HQsJiZMlIb99HOJ31UbVJSYeTnXU2kmSgf+BzWlufiqpNS1vN7Fl3IEkGXK5WNm0GE3pRAAAgAElEQVT+TlRtSsqxzJ/3CBAwdDZsPAq3uzWiNtG+koqKPwePq6pvpqPjpYja+PhFVC5aG4yM6un9gD17roiojYudT2Xly8hy4PXF7e5ky9ZT8Hp7ImoXLXoxmFophMKePVfS2/dhuDZuIYsqng0xozo6Xqaq+mam1zuLj6tg4cJn0OttwbaxsQPs2HkhHk9niDYhfgkLFjweYogpipv9VdfR0/NuiDbRfjBz5z6EwRAX0t7W/jz19feGRL+lJB9NWdk9GAzxIdqR0b3U1t7B8PA2IBBRl5Z6AqWlP8dgCE3b9vlGaGh8gM7OV1EUx7iBdzKFBWtCUlcDayfo6nqdtrZncYzVYjAkkJZ2Enm5l4VpIWDgtbY9zfDwDmTZRErKMWRnnRNR6/MN09H58uSOnIkrycw4I2xugXGoDAx8SU/vOlTVTXxcBenp3w0zHCdwuVrp7n4bnz+wEUJa6gnodJH/nk2kunrcnZjNWSQnHxWSljt9PUZGd+Mcq8dgsJOYuHLGtFiPpxeHYz+yzkp83MIZDT9V9Y6nQ8vYbCXBez163z2oqhezOSOq6TnZtwdFcQfTrr+JbyoToKGhoaHxt6MZUVHQjCgNDY3/DR5V5etBBw5FZW6MhQJr5DfuzS4Ptx/o4P2+YVTAIsucnm7nlqLMkEgrIQTX1bTxXGd4BNURibE8Pa8AoxwwrUb9Cis2VtEbxWC6uzSbC7MCEScdbi9LNuyPmgiWbtRzT2kOLlWl0eXhnsbodY4AiiyBeY4qSkSD7f8qsvCjoiPeoMciy4Bg1DNEMj3o8WPAN25c+dHjp5VcypJmkW02IgE7Oj6jUNSim6KZ+OfCTHzOT6iMj8MoS6yrephc3yb0+MY1gZ86/Bjx4U2/hsMyK5BVDxtq7ifWtTFqdJjXVEpR6sGoqoeuvvUonqaocxTImIyJqKoHv9/JTOmBGgEmIu2s1iJk2YSiOHG5mmbUpqR8B6PBjir8dHa+QiTTcUKbm3MxVlsREjoaGn83Y52x4qIbxgvJ62lqfpSenneiaktLbycr8/tIkp7u7rfYt39N1KjBosJryc8PGGAuVysbNh4VTGWcPt6pKZuq6mfjpmNwuZojatPTTqa8/L5g+/6qG+jsfCVkHBP/T0o6jIULJnez7Ox8nf1VP43Yb3x8JZWLXgzWJXOM1bF166koylhQM/EzNqacysqXg8aKojjZtu1MRh37wtbBai1mceXLIYZYTc1ttLU/G6a1WHKprHwZ03hkH0B3z3uBzR6mrZ3ZlMmiRS9isWQF28bGGtix81w8nu4QrdGYxqKKZ7HZioJtiuJk9+7/YmBw/TRtMgsWPEFc7NzJNRKChsbf0dT0CFPvO4PBzvx5j5GQEPo5oLfvY/bvvx6/f2iKNpHyOfeRlLQqROt0NrJ33zWMju4NGcOs0jtITT02ROv3j1Fbextd3X8NrofRmEpx0fVkZHwvRCuEoKX1CZqb/xisV2cypZOffyXZWWcznf7+L6hvuC84DpMpndzcS8jJvjDMPHI6G6lvuJ/e3nUI4cdkyiAn+3xycy8OM6/8/lGamv9IZ+creL29mM3ZZGWeSW7uRWGRgEKodHa+Qlv78zidjRiNyWSkf4+cnB+EGLUTDA5uorXtaUZH96HTWUlNPYGc7PPCTNLAmJtpbXuGoaEtSJKO5KTDyM4+NxhFOn3M7R0v0d//GUINpP1mZZ8TsRaeqvrp7fuQnp53UfwOYmLLyco8K+S+nMrw8E46u17H6+3Fai0gM+N0rNb8iFq3u4OOzldxu1owmlLJSP8eNltxRK3f76C7+y1GHdXo9YFoy9iY2RG1Qij093/O0NBWJNlASvKRM0ZQjozuZaD/C4RQsNuXEx9fGdVQdLs76e37CFVxEhs3H3vCQVG1fr+Dvr5P8PkGsFoLSUxcGdX8FEJhYGA9LncbZlMGiYkHz2jAjozuZcwRMLnt9pVRTWAAl6udkdHd6GQTCQnLIt5rE/h8IwyPbAcgPm5RmNE/FVX1MjyyC1VxExs7B6MxKapWCIHDUR0wua2FEVPXQ8fchsfThdmcGYxujj7mQVyuVgwG+zempCuKG6erCZ1swWLJndE4FkLB5WoBpHGtPINW4PF0jZvcWd9oiPt8Q/j9Dkym1BlN+cCYXfh8QxiNiRGji6eiqj58viH0+rgZ74mJMfv9w8iy+W9Kz/f7x5Ak3d+kVVUvQohvHENgHCpC+CLOTTOioqAZURoaGv8Mhn1+Bv0KqUYDVl3kP4JCCN7qHea5jj6aXV7STAa+n57IGemJGKbVwdo76uSCPY20eyYLgEvAj3NTuakwI+SP8j0NnfyuOfQDD4AOeHpeAUcnT34Df/auej4ZGI04vvkxFj5YXBrs+/httWwfCS+iHTxvaRbpJiMuReXO+g7aPNGLlRdYjOSZTbhVld0OF07lH7GX4f9dJKGMm1UKepTxKC4Fl2wny2JFL0n0exzE+RqCOt0UnR6FHjmHxWnzMUgSbY4ezCPvY8CHCS+G8X9GfOjw4cZGedaJmCUfDncPrr7XxvvxB/uXx88hIUhMXIVVp0fxjzA8+CUSCjICGRUJEfJPb0jCKJtQhQevN1CbTYs9+P+HJOkxm7ORJD0+30BY8fpQAh+MJVmP19vP8HDk908BI0giK+vcYCRgW9szE1swRNBCQf5VmM1ZIEnUHbgbf4RxBGvKFV5HfHwFkqSjofFBBge/jjriosLrSU8/CSSZjvaXaGz6fVRtfv7VFBVeBcDQ0Fa2bf9+VAMvM/NMymb/CghEN321fmXEumgASYmHsXBhwGgTQrB5y2ocjv0RtTbbbJYtfTv4OltdfQvtHS9EHIfBmMrK5Z8FPyR0dLxCVfUNEbWyLoYVyz8Ommejo/vZvPUUiLCJhCSZWLr0TWJsJePzG2Hj5uPxejoj9C2zqOJZ7PaDgvPbsfMCBgfXRxxHefkDpKedFDyeKT23pPhn5OZeFDzu7f2I3Xt+CKhhfefmXkJJ8U3BY6ezkc1bT0Xxh+8Ym5a2mvI59wfX2O93sGX7WTgjXJME+0oqFjwRNBOEEOzb/1O6u98I09pi5rJ40V9CdhNtbX2G2ro7wrRGcx5LK18KSc8dGNzAzl2XIKbVPdQZklmy6IXgJhYQ+IC/ZfvZ+DztIVpJF0PlwqeIj18UbFMUF9t3XcLI0MbQQUhG5s99iJSUo4JNQghq6n5Je9v0mo46Zs+6g6ysM0NaAxGUP2P6FxsF+VdTOP5cmmBoaCs7dl+OOsX4BMjIOIuy2XeEGARudwfbdl6E2xm6aUhi8tHML38w5IOxojjZtfcaBvs/DtHGxC9l0fxHQww/IQR1B35DS+sTSFPGbLaVsWj+YyFp2IH5vUpV7e2gTu7YbDDnUjHvYWJj54TNb+feNSjejmCbbEhjQflvSUxcGaJ1udrYufdqnKM7g22S3k5Z6a1kpH83ROv3O9hbdTN9ve8iTRjMso3iwqvJzbko5P2eEAp19ffT0vo0kgjcR0IykZtzISVF14YZaO0da6k+cB/4A1H2Aj1p6acwZ9bPw6IoBwY3sKf6NvyuA+NamcTko5k7+86wqE+ns5HdVbcwNjx5z8UkrGR+2Z1hKdY+3xD7am6nr+ddpPHIcnNsBfNm/YK4uHkhWlX1UdfwO1rb/oKkBson6C2lzC29Kcw8F0LQ1v48tY2PgC/wJY9kyqWs8KowQxygr/8z9tXdi99ZFXi8IZXi3IvJy704zOgaHa1ib+3dOIcDEa1CF09u1tkUF14VZkh5PN3sq72Hgb73kIQXVQpE4ZaVXB8W1aooTmrrH6StYy2yOoKQDCQmH8eckuvCjDwhVJpbnuZAy1NIvg4EMjH2w5hb8lNiYmaFza+7+132NzyC6grMzxi3jLlF14TV1wQYGt7G3gMP4B7egIRAtpZTXnhl2JcOAE5nE/sOPMBw/wdIwgumQkrzLyI788zgumlGVBQ0I0pDQ+P/Kl5V5YO+EfY7XMTqdZyUmkBOhBpYQgie6+zn4ZYemlyBOlRL421cX5DOwfbQP4K9Xh9n7WpgryP0A1WBxciLC4rIs0y++ds05OD0nfV4I/zdOCsjkd/Nnnyz8WrXAFdWtUSch1mW+HJZWXDsz3b0cV1NW9R5X5+fzpkZiXhUwSvdA9zXFG6yTZBnNnJ+VjIeVWXniJN1UXZShMCHmnkxFlRgwOenY8I4i7CLoca3iyRUZPwhppgJD5Jkwm7QYcCP19uLjRH045FjAZ0/GIWmSBZSLAno8TPm6sAiBqeYd/6gMadHQUWH3ZKCLBTc3n706tC4uaYGfwb+BQw1o86GhIJQ3UjCHXhjhho1Uk3j/zaSpCPwfji6IS4AsymQFqYoY2E1w6bqAOJi545H1blwOPZHNGkm2hISlmE0JiGEQk/vh0gzjCPRfnAg+kSSaet4FaGEGy8TJNhXkGRfAZJMe9fbuMcim2EA1thK8rJOA2T6+r+kt/ftqFqDdRZlxdciIeFw1FLf8NuoBh6GNCrnPYQk6fD7h9mx69IQQ2AqQrKwbPFadDoLAti4/Tzwdobrxn8uXfzmeISPzLY9P2Fs6LOIWgmoWPgcdvsyQKK28SHamh6Mqp01626ys04HJsywy6MblLlXUFZ8LRCIMFy/4Yio1y8u+USWzA+cV1U9fPrVIeCPXFfSYFvAqmWvBY+/2noWnpHNEbUYUjl85efBD8R7qu+gp+OZyPecZOKQlV8EDcq2jlepqb4+yv0psWzJm8TGlgMwMrKHzVu/N2mOBHWBx86d+whp4x9cfb4hPlt/GLI6GlGbV3gDxfmXBdqEymcbT0R11UTUJqWfxcI5dwbbt+xZw0jvXyNGW5riV3Bw5WRUY13Tf9PScE/E+UmmAg5b/l7QdOzt+5zduy8iEqougUOXrwtGDzmdzazfdAKyCDejVcnI8iVvBM0BRXHy6YbjkLxtYeMQSMyf/ydSkw8bXwvBV9vOxzvydcQx55fcRlHOecHjXdW/oq/jyYhjTsq8kIWzbw0et3S8Rl31dRG15oTDWVHxeNBIGBrawZbtZyHjC4s8xTKLw5e9HoyE8Xi6+XzjSeiU/jCtok/hsIPeDkb4qaqHTzedCq6qcK1kYfniV4mNnTRVvt71I1z9701Zr4BWRWbB/D+Rmnxo8Hf76h+mq/n+iPPLK76d4txzg8cdPevYv/eK8fd6oetsz/wBi2ZPbt4yOlrFhq1noBPOMK0x4XAOnrJuXm8/n246Gb2vg+kIy2wOX/pK0PBTVR+fbT0H4dgWpvXr0zhs2RuYTKnBtk17b8TR83K4VrKyfPFLxE0xSqubnqK9IfCcmTpmFZnyuY+RmTq5o3FX3+fs2X1ZxHITWYW3MDv/B8Fjh+MA67eejl4Nf28dm34uS+fcDmhGVFQ0I0pDQ+M/BSEEPV4/RlnCbogebuxTBe/1DfPpwAiKEBxsj2V1SkKwntVUtgyPcVdDBxuGAt8Wphr1XJKdwpW5qehCvqUT3NvUxf1N3SFvWWN0Mv9dns8RSZNh415V5dzdDXwxGF4sfGm8jZcWFGEZH4tTUTlqSw0NrvCdDGXghQVFHJoYMNtcisrSDfujpjSekW7n92V5wTFUbthPb5T0w0S9jo+XzkJCwqUofHf7AXpmqMV1XX4aWWYjXlXw28auyTEIgYSKmPIt5fwYC7NizPhVwecDowz4FRAqsYyiQ8GLESc2kCSsskyqSY9PFfT5/HhUgSwCRoqKjPoNtVM0/k6EGDevAgbXRJTY1Ki1wE91yrESoT309/J4H4H+JjVyyONCtdKUtqkRa5OPU6aMU52iUdHBuOnmH/+nBs8hBU25QJuGxn8aAhm9zhqoCeh3BaM2omnNpjQkJLz+UVQlcoRxQAs2axGSpMPnd+D1hH9YndBJQIxtNjq9FVXxMOrYF93sA6y22ZjHo636BtYjz/DcNVpLSLAVgyTR2fclOjX6mCVTLun2JQFt/2bwRv6CCUDRJZOXfhxI0D+8H9do+AfsCfyShcLsc5AkmTFXG32970bVqsjk516MTmfF73fQ3PrkeMRtKBPrk519IWZTGkKo1DT+AX0Es2iC5LRTSYyfC0jsa3wCgy9yPUAAi/1I8tOOAiSq215DOKIYg4CwVjA3P2AYtfZ9xUjPa1G1HkMui2fdAMDIWD3NjfdHvdZeKZbK8t8gywb8/mF2778u6uu0gsz88gcwGhIQQmHTnjWY1KEwXTDytOR24sdNoA37f4nJHZ7SPEFK9hXkpR0JksSO+j+hDEa/foakU6govBCA+q519Lf+IarWZzuIg8t/AcDgSA211ddEXQuXPofDK59EkiT8/hE2bD0DfZTnqgcbq5a+hk5nRgiVTzZ/H4saXnNx4v3qooq1WK2BNNaPtl+NxRXuHQSje2f/juzkFQCsr34Ate+FqPOLy17DwoKAIVbV9ia9jXdEnZ+wn8Jh8wI7H/cN72PfrnOiap3mRRy/LFCv0usd5IsNR6AnclaCS07luEM+CX4B8+6XR2FVI78W+TFw2MqvghFwb2+6AKszPCJ5YlzzF71BSsI8zYiKhmZEaWhoaPz99Hv9OFWVDKMhuGtgJJpcHl7rHmTA56fIaubUNHtIrawJPKrK4629PN85QJvbS7rJwFkZiVyekxqW2tju9vKjquagGQYBQ+zOkmxWp4bW4Ng+PMbZuxsY8od+A784zsrzC4pCxvJ+7zAX720M+65eBzw+N5/jUyb7Xts1wFVRIr6OSYrjmXkFwW/IXujsZ0115De3sTqZrw8qI8UY+EZ2pkgygD+V53Pi+BzX9Q1z/p7oBe/PSrdzS1EWihBsGxnjB3ubgEDaXzxD+DAxJgVSSrLNBn5Tmo0ioM3t5aa6QPpHkuglgzYEOtrIYViyIwNX5aVhkWVG/AqPtvYgCR/ZtJJC4I3dIIm0kYtHMrMiwUaa0YBPCN7vG0ZW3aTTSSwjCGQ8mOgnmSHspJoMpBoN+IWg3ukB1UMsI0iIoD3iw4gLC7IkY5QlFAE+IbRYpX8E48apjqmRY5Nm16RhNfm7qSbW1LbQyDM1TD9z29/2TwprE9PalCjt0dom+hRh/UuIkJ/h2sn+pkbUTWg1NDQ0NDT+r9NPIkkEUuudiedw0sI7NCMqGpoRpaGhofHvQZXDRc2YmwSDjhUJMcHi7tMZ8Pl5sXOAbSNjmGSZ45LjOS45PqKBtnnIwe9bevhivG7WIfZYrspLZVlCTJj2+c5+ft3QGYyiMkgSZ6TbubMkOxjBBYHosHsau3iwOTQ6LNGg48m5BRw0pW8hBNfWtPJ8Z3i9nAuzkvl1SVbQ4BJCcNm+Zt7qDf+GM9ds5O1FJaSaJuucXLS3iff6wtN4JAK1w46dUjvssn1N/LUnvF+A8zOT+M2sycKia6pbeCHCeAEKLSa+XDY7GC13+4F2Hm0N1InSiUBReDdmkCR0wNcHlQXTQR9u6eGX9R0gBMXUkcAAvaTSTAFIEo/NyePkNDsAr3QN8KOqFqxilJV8SQrdDJPARlbSL6VydnoiNxRmoAjBxiEHV1S1kClaWcGXZNKBCzO7Wch2llBoi+O3s3JQgBaXh6urWykWNSxlAxl0oiDTRCEbWcGgnM2vS7MxyBIjfoVf1LVTLPaxkG2YcaOOG23VzGE3FfwgOzVoyj3S0k22so9ZVAMErZBhEtjOYirsqZTaLKhC8HbPEPG+GnJpDrFNFHTUU4LFksW8GCsqgu3DTvTeRpLpnWJ/SON9xzOky2XOeEpqq8uL4u0KmoJTtX709EspZJkDaVKjfgWPbwQz7pBERTH+040Fk84IEvhVgV/1B6L/JvqcoWDsfyRCRDWvphtXhJhdYorJFcn0Ykofof1NPDa8LzHt3KH/l8ft+UjnZ/IKRxhDpLEQNqbQPqaOJfLjCdNOzmt6X9G0f8u/iTlP/Tk5rtBzyDOcK/TxkX4/9fFMGXf4eSbmH3o+Qv6vGZ0aGhr/LKooo4xAHapWy7FcuPwRzYiKhmZEaWhoaGh8ExN/G79pS2+vqrJ9xIlbVZkbYyXZGD0Fstnl4Y3uIQb9fmbZzKxOTcCmC48OE0LwYf8IL3YO0O7xkm02ck5GEocnxoaNx68K/tTWy9MdfTS5vMTpZU5JS+Ta/LRglNUEbkXlzoYO/tIxgEsNfFAptJi4tSiD41JCI8nGFIVrqlpDTC4JODMjkXtKs0NMvzG/wkV7m/h8MDS9I8ds5IUFhRRbJ3dqGVMUztnVwMbhsRCtDNw/O4czMyZ373EpKmftqg/TApyYEs8fy/ODBpdPFZy5q571Q+HpndlmA+8uKg0x5c7d3cjHA+E1DvQSvLigKKSW2hX7m3mtO1APSC98AQNIClzn6/LTubZgcseqW+vaeLytD53wkU8jOhSayccjWVhlj2HtwsmdpX7f3M1dDZ3EiWEWsQULTpoopIpyko0GNh5URsx4xN5LnQNcXd1CjmjiYD4ngSG6yOBzjmBQSmbd4lLmxVoB+HJglNN3HmA+OzmcD0mjm0HsfMHhbOYg7p2dzzmZgXWuG3OzalMVK/mcY3iXPJpxYmUDK/krp7A6qyRoOg75/FSs38cq9W2O568kEyh220ARr3AmxK3kg8WBtA6/Kli6YR+LPWs5kdexEdjkwI2J9zmeD+Rz2LRiLhZdIKnk1O21ZI0+z/H8dbymlwxIHKCEx/khjy9cQqnNjCoEN9e1I3r/wjG8iwxBW2AMC89yEacVHcEx46m/T7T10tLxMsfwHtOtkS84gtjUM7goOxkV+Kh/mE3Nb3EM7wUtkglLoIU89tou5PqiPAQBE/ythg84knUEapxMWgVOrKzTncV1pfPQSRL9Pj/P1H3OkXwwTRsYx2ccxekFB2M36vGqKk/Wb+Vg9QNkJqL8JvuuZg4lGccGn1PPt1RT4XsPHWqYdoBk/PaTWBIfhwA+6G4l3/U+epQwC8OLiVbL0RyanI5AsHlwkATHOkx4plgpk+Ou0y3h0LRiBIK6MRdi+BOsOKfZIoHHNEuFLEqZj06W6PX46B/8mjhGgjbLhFYgMUAimfZKYvU6XIpK48AuEulDIE1Zu0C/bkzoYipJNRlRBdQM1pEsJnahDLdnHKa5ZFttCAH1Ix0kqu0h85pAIDGszyXHloQQ0OYcJNbfipjye4LjgBEpicyYNISAXo8Lk68lwhgCxy6s2K0ZyBKM+PzgbRu/zqHnB/Cjx2BKxyjLeFUVj6cH3ZR43TALSp+EWadDAGOeIQz4xnsWwfWbGIki2bDq9QgBLr9jXMu0KxKYgR8jJp0BGRWv4kWHb8ooJ1YicHVUZAyyHhAoaiD1V4StMEzU5ZGD11UN9hNquEWcacj/mTLmqcbf5BWYnEv4FZx8/PTzTjcDQ69OtHOHjnn670PHQdh4p543dFxTzz3zWCPNNfKYYOp1jtQWTTt1Xn+LNtp8pj5m5vbp/UfX/8/GP1P75O9CzzfTPMPHF+l33/T76fdI6LnCNdPHGjrG6dpI85je/j/TuDGzgsAur7Vxl/LDxTdqRlQ0NCNKQ0NDQ+PfEZ8q0EvfbJ6N+BVqx9xYdTKzbWbkGfQHnG7WDzqQJVhljw0pXj8VIQTrhxy83zeMVxUsjbdxYpQ6Yz5V8EbPIG90DzGqKJTZzFyYlUxZjCVM61ZUnmrv46WuAbo9PnIsRs7NSOKczKSQmmQQMK4eaO7m2Y4+BnwKZlni5FQ7NxZmkG4yhGl/Wd/BC52TplyZzcxtxVnBGmNTx3BLXTsvdvXjH3/LZJFlrshN4dr89JD186mC62tbwyLEDrXH8sfyPBKm1GpThOCGmjae6wwtVpxq1PPs/EIWjBtLE+v78wPtPN7WF6LVAb+ZlRM0liaItnPmaWl2fl+WGzLmx1p6uK0+vD5Emc3MaxXFIfXlXuoc4JrqFhAKSfTjR8+QlEicTuaVimLmTxnzur5hLtrbiE51U0QdMir1FOOSbDw4O5fvZ0zuurR1eIxTdx7AqAxRwTZMuGmgmHpKOD8rOSQCr97p5vhtdRh9HazgS2IZpZNMvuYQZsUl8+aiYkzjRmmv18cxW2rRe+o4lE9IoYcBkvicIxk1lvLh4lIyxzdNcCkq39lWi9+xm6N5n2xaGcPG1xzCBlbxYkUZK+wxwetx+s56+gY3cRxvUUItXoxsZRnvchI/KV3AJdmTu6NdXdXC7s4v+C6vUs4eAPYzlzc5lfKMVTw0XqsO4L7GLt5q/JgzeD4YLTdAIu9yEi0xp7FuSVnw3l/bNcBD+9dxPk9SSD0AfnRs4GDe0F/KhwctDhrSG4YcrNm+jsv4A3k0B8/XSCGPcyWPVh7NovjA9vDNLg9nbPyIH4p7yWEyVXiUGJ7hEk6ffTZnj99zY36FYzd8ykW+u0L6VZF4nxMxZ/2EX83KDa7bSZs2stp5G0UcCLnf9lPOl3G3s3ZxRbDtsl17mdd/M3MIrVXTTxL/bbiNt1d8J/gac0ddE+bWG6lkS4jWh4FHpTX8/qALg69fz7b3UFvzMw7jE6bzBqdy8oJbOHLczPxqcJSXd9zOal4P0+6iAmPhA/woP7DzWovLw8833Mt5hBeP7iKd7SkPcd+8wO52TkXlkq/+yIXK/WE1flyYWWu9i2eWrUaSpEBE68ZXOMN1K8YIdV/+LF/JIwdfjW3cuL5252esGriKGMJN/A/5Dhctu49SW8DMvLd2LzltFweN5alUU0benKc4MT1wL7/c0c5o9fnk0RTUTNhQ/STRmfMkV5fMBWDb0ChfbL+A+ewK03ox8Jn9Ae6p+A4QeJ4++NWPOZwPw8YggDeMa3hwxRVIkoRXFdz45S85Xn1uimby4/nH0gncsPIeYvQ6hICbNz/F0a57Q/qb+EhfxRyOqHyaWbbA3567931IZf/16McNvwnrRkKijyQspc9wXFqgbtAzTXtJab0cK+5gvxN4MNOY/iCXFi9GAJ/1djFY8wOS6Av2N8jykmMAACAASURBVJUNsTdy3fzArm4NThdf7biYfBrCtAKJrwync8OSawAYU1Se3HwdFWLTFDtw0hrcKS3hgiW/xaYP1AK6f8t9HOT/a4h2Yt06yGTh/McpsgVew3+/ey2VY49NGeXkOMawIBU+yhGpgfv+mbqvKO6/Y4o5MalVkGhK+SVnFwZ2anu/sx5LyzUY8YZpAXbZLuGyuaciBOwfGaK1+ockMBSc/1R26I/h8oqrkSSJAa+Pj3ZeQ+74vTldWyfN4ZSKX2PRySiq4M/b72SO2BJci6nj6COJ8vI/kGMJrMUTu//EfO/bU9ZtEi9GpLz7WZ4SuC+er32f0pH/jrDGAdqSb+J7eYH74p22PaR03x62bhOPrbadw7mzv4cAtg504Wm8FtP4uk13b6r0h3Lu/CsB6PR42LtvDYn0R1yLRqmEExfcgUmW8Cgq7+y+jTxRRyQGsTOn7D4yzBaEEDy/52FmK6GF9BMYJJMOPBjRl73NMRlFmhEVDc2I0tDQ0NDQ+PdFEYJhv0KMTo6arjnBqF/hgNNDrF6myGKa0cTr9vjYMjyGQZZYnhATsdbZBI1OD58MjOAXgoMSYkJMpelUOVz8tWcIh6IwL9bKSSkJIamdU9k+MsbLXYP0eH0UWkycnZFEgTWyObhxyMFzHf00u7ykmfR8Pz2Ro5LiIs5x/eAoT7b3BXfkPDnVzvmZScGIrKl8PejgkdYeNg87MEoyxybHcWVuGoURxrFpyMEDzd18PjCKChwUb+OqvLSQzQom2DHi5O6GzmBkXabJwKXZKVyekxJmllY5XPz8QDtfjm9wYJIlTkuz84virLDr0uzycH1NW0jE3vIEG/eU5gQ/jE/Q7fGxprqFTwYmtdlmA3eVZHPMlNRVCBi611S18O6UdFerLLMmP40f5aaGrLNbUflpTSuvdg8iicCHXFXScWqanXtn5YRcb0UIfl7XzpPtfcSKYQx4GSCR+XExPD2vgAzT5E6pQgh+19zNvY1dpIoOYhiliwyspkSemltA5bixNMFzHf3cVNNKlqgniT76SKFTKuS304xBCNTM+699jeSp+8mknVHi2MVCLsrJ5rbizJD5bR0e49xdB8jy76SIejyY2MYSyhOLeGpeQUidvwNON9/fUUeSZyvl7EFFYjcVeCyLeLmimKwpO8H2en2cvqMO09hGKtmCHh81zGafbhVPLSxnyZT5ORWVc3bV4Rz6iuWsx8oYreTxGUfxs7LKkGhLRQiu2NdEU8+nrOIT7AzSQxqfchSH5R7KbUWh8/tlfQdfNK/jSNaRTgfDJPAVh2JIPIGn5k8anwBPt/fxTM37HMs7FNCACwubWUGt5WReqKwIiVRd1zfMXXs+4DviDcrYh4KOHVTyuf4UHq44NBjlCLBr1Mma7R9ylPIyFWxDRqGaObzLyfx47snB2oEQqPN30dbPOMT7HEvYhAE/HWTyASewqOACflKQEdSO+hXO37aRBWNPcBDrMeBnDBufciTOlEt5dG5Z8PnnVwWX7NpJ9uBjrORzjPjwo2MrS9luu5xnKleEvGbcVH0Af8eDHMbHmAlsMFJHKe/oL+IPS1aHfLHxWHMH++of4GjeC0ZQ9pLC69LZXL3woqAJDPBuzyCv7n2Ik3iNWALPVScW3ucEls+6lnOzJnca2zni5LfbHud74s/Yxw0NBZmNrETNuonbZk3u0Nbu9nLt5uf5nv/REGOuhtlsj7+JPy5aGTSBx/wKl295m2Nd95LJpJHfQypvmX7CI0u/F/ziQRWCH+74kiVDd1BAY1A7hpVXdRdzy+L/omTKa9FtVXtI7LyFcvYG2/zoeJ+TOGn+rRyRPHmtn2xppfvAjSxlI1PZzDJSiu7mkrzJ3Yw/7hvko923cAQfMHUH2EYKaUq7mzvLK4NtVQ4XD2/9NavVF0KM0kHsfBhzGw8vPiFY3qDX6+PmTY9xiu/hEKPUg4nXDFfxm4MuIXF8LTyqyo82r+Uk513YphilKhLvyN/n8sU/oywmcN8LIbhu1ycsHbiJJEK/sNnASpbNu5+jU5KDbffW7ia57eoQ8xzgACVQ8BCXF5QE29a2t9Jdc2WYyd1HMnuS7+XO+YdMnmtwiI92rGEFX4RoXZh5z3orDy79fnAtmpxu/rD5No5TQ3e3U5F4U385dyxfE1wLh1/hxo0Psdr7cJgZ/al0HOcu+S2zx7+gU4VgzZaXOMZxR/C5NMFuFjBv3mMclTJ539++9zPm9vyEBELLMbSTjSf/US4tnNxh70+N+zA2/pAs2kO0I8SyM+U+bp83ucPehz1d1Oy9lDJCd2D1oed9683ct+x8dJKkGVHR0IwoDQ0NDQ0NDY1/Hn418F5zpo0NJhjxK7gUlRSjfsZoPQgYRwM+P9lmI7EzGIMArW4vbW4vmSZD1Mi+CZpcHmrH3CTodVTG28Ki76bS6PSwfbz+3KrE2BkNyiaXhy/HDbGDE2KjmogQiK5Z1z+CS1GpjLOxPMEW1Shtd3v5a88Qgz4/s2MsHJ8cHzEaEQJr9kr3IB3uQNrvaen2sDTeCXq9PtZ2DXLA6SbRoOfUNDtzIkQuAgz6/LzcNcD2ESdWncyJKQkclhgb8RqO+hXWdg2wftCBJMERiXF8L80etjEFBCLVXuse5IPxaMsl8TbOy0wKptpOxacK3uwZ5I2eIUb8M0dbqkLwbu9wSLTleZlJHGoPT4EG+KR/hD939NHg9JJs1HNGup1T0xIxRLinNw05+FNbH3scTmJ0OlanJnBBZhLxEXav3edw8WhLDxuGHBhkiaOS4rgsO4XcCPdok8vDIy09fNQ3jE8IDkqI5Ye5KSyKs4Vpezw+Hm7t4Z3uflx+N7Ni7VyakxKWhg2B6/FISw+vd3Ux5h0mxZLMOVlpXJCZHPac9aoqj7f18VJ7G8OuHqymRL6bkcOVualhz0EhBM919vNMayvDY82gi2FVWilr8tPJnmI4TvBWzxB/bG5hYLQGgZ5ZyXO5Jj+LhXHhRv5Xg6P8vrGVrqFdyKjEx83lyvz8MMMYYO+ok3sbWmnp34IRD4q5lLNyZ/ODrOSwa93q9nLXgVYO9G7AIkZw6nM4Mnsxa/LSwp5Tgz4/v65vZ3fXemLUHkblZOakreTmouyIqfH3NXayvv1r4pUmxoghKWkVNxYXMWuaIa4KwR9be3mneQN23z68GFFjD+bq4vKQtPEJXu4a4C8Nm0lwBz7fDporOadwGWekJ4ZpPxsY4Q9124kZ+xIjXnr0ZRyaczhX5aeHvc7tHnXym9pd6Ic/JoZReqQ8CjOO5dbi/GD03QQtLg931u7H3f8eifTTTwoxycdxa2lZMOp0giGfnztqa+nteYdU0cYocbhjj+GnsytDomohYFzdfaCJ2o63yVJr8WCiy7yKi0sPC7vWQggebu5gfcvb5Pl3oyLRql/MsYXHcUFWati1fqWzj1cPvEeebyN6fLTIZZRlfZfrigrC7vsv+kf479qPyHR9go0x2qV8YlNO5hez54a95lc5XNyz/3OSHO9hZ5BeUnHFr+aWOcvCntedHi+379uEaejNoMndZT2Wq+YcFfa8dvgVbq/eyVjva+SJelxYaTCu4szSkzg+NTQq2q8KfnOglqb2tRSLfSjI1OqWsCL/VC7JzQlZCyEET7S0saFpLaXKFnQo1EtzyMw8gxtL5oS9xr3V3cvrta9Q6vsSMy5aKEBKOo2fz1kafI3TjKgoaEaUhoaGhoaGhoaGhobG/x4hxDemgk+gCBGo/PY36L2qioz0NxnXbkVFhYgm5nRciopHVYnX675xHG5FZVRRsOv13zgOj6oy4PNj1+ujGsATeFWVHq+fOL1uRtMaAmvW5vZikeWIxutUVCFodQfStnLMxhlNfCEEbR4fbkUlz2L8xsjhTo+XIZ9CjtkYMUp2Kr1eH90eH+km44w1MyFgSDW7vSQa9OREMCanMuZXqHN6sOpkSqwzRy97VJUqhxtZgjKbJaJZPIEiBPsdLryqYLbNHGawTeX/tXfn0ZZU1R3Hv1tmBAFBlElbZZAsxSE4ZQkiGGKIoigYXYpxTKLLiDFqNCbSSpyCQaMJsiIqTjESNYhxAMTglAgKStPIoGgrg6gIAm0HtOHkj1Pvdb2qvXfVk+7b1c3vs9Zdfe97vz7vVO1b55yqd+99pRQu/dUt3LT6Nu6/9ZaD23fFqlv46a2r2WOrzQe376pbfs2K/7uVnTbflH223jLdvp//+jdcuvIWttl0E/bbdqv0FyU3rb6NZTevYhMzHrrt1unz85bb6meeri6FB2+7lXvhfM7q2wsX3PQrVt52O/tus+WCV+qCLkSFdCFKRERERERERGTtGnshSn9LV0REREREREREZkIXokREREREREREZCZ0IUpERERERERERGZCF6JERERERERERGQmdCFKRERERERERERmQheiRERERERERERkJnQhSkREREREREREZkIXokREREREREREZCZ0IUpERERERERERGZCF6JERERERERERGQmdCFKRERERERERERmQheiRERERERERERkJnQhSkREREREREREZkIXokREREREREREZCZ0IUpERERERERERGZCF6JERERERERERGQmdCFKRERERERERERmQheiRERERERERERkJnQhSkREREREREREZkIXokREREREREREZCZ0IUpERERERERERGZCF6JERERERERERGQmdCFKRERERERERERmQheiRERERERERERkJnQhSkREREREREREZkIXokREREREREREZCZ0IUpERERERERERGbCSinruw8zZWY/B37U+fJOwHWLaGYx+Y05O5V+bGjZqfRjCtmp9GMK2an0YwrZqfRjQ8tOpR9TyE6lH1PITqUfU8hOpR8bWnYq/ZhCdir9mEJ2Kv2YQnYq/djQslPpxxSyU+nH2sjep5Ryj8H/XUq509+Ab62r/MacnUo/NrTsVPoxhexU+jGF7FT6MYXsVPqxoWWn0o8pZKfSjylkp9KPKWSn0o8NLTuVfkwhO5V+TCE7lX5MITuVfmxo2an0YwrZqfRjXW5f96a35omIiIiIiIiIyEzoQpSIiIiIiIiIiMyELkRV/7oO8xtzdir92NCyU+nHFLJT6ccUslPpxxSyU+nHhpadSj+mkJ1KP6aQnUo/ppCdSj82tOxU+jGF7FT6MYXsVPoxhexU+rGhZafSjylkp9KPdbl9C9zpPqxcRERERERERETWD70iSkREREREREREZkIXokREREREREREZDbuyJ/c2xhuwBOAy4DvA68ZyL4f+BmwfCC3B/DfwCXAxcAxSXZL4Dzgwib7hhF93gT4NvBfI7IrgIuA7zDwJxaB7YFPAJc2fX90kNunaW/udhPw8qTdv2y2bTnwMWDLJHtMk7vYa9OrAXB34Czge82/OyTZo5q2bwf2H2j3+GZfLAP+E9g+yR7X5L4DnAnsOvScAV4JFGCngX4sBa5u7e/DsraBv2ie0xcD/5C0+/FWmyuA7yTZhwDfmHseAY9Isg8G/rd53n0GuFt2XHj1S7K9+iXZqH5RvlfDKOvVMGm3V7+s3W79knZ79Uuyvfol2ah+7lgF3Bc4t6nfx4HNk+xLqWPt/PM+yX602Q/Lqc+zzZLs+5qvLaOOYdtE2dZ+fjewcqAPpwA/bO3nhwzkDXgTcHmzX1+WZL/aavca4LQkewhwQZP9GrBnkj24yS4HPghsGs0dXu2SbK92SbZXuyTbq92Y+a5dv6Rtt35Btle7JNurXZLt1S7JZrVbQWcuJ577vGw093nZaOz0su7cF+Wj+S9oeyn+3Oe2iz/3ee1Gc5+XjeY+LxuNnb21VVS7JB/Vz8tG9fOy0dolXA86tfPajWrntuvVLmk7qp+XjernZXv1I1jvevVLst66JcpGtYvy3rolXaOzcN0StdurX9Zut35Ju966Jcp665Yo6x57Td965yAEc1+Qdee+IOvOfUHWnfu8bDTvBe2egr9u8bLZvOfl3bkvyLpzX5B15z6cc0Liec/LRuOml3WPvSQfjZ3heSz9sdNrdyn+2Om2iz/vee1G46aXjcZNLxsee2Nuo4Mb4426CLwCuB/15OlC4HeS/IHAwxi+ELUL8LDm/rbUA9xtlzoIzA0+m1EHxkcNtP8K4N8YfyFqp6Fck/0g8MLm/ubtg3BgH14L3Cf4/m7UAXGr5vGpwHOD7AObJ/jWwKbAF4G9hmpAnfRe09x/DfC2JLsvdSI7h4WDkpc9lDWD4dsG2m1Pei8DTsqeM9SLAGcAP2LhxOa1vRR45ZjnI/C4Zr9t0TzeecxzF/hH4PVJu2cCf9jcPww4J8l+E3hsc//5wHHZceHVL8n26pdko/pF+V4No6xXw6TdXv2SbK9+WR+69Uva7dUvyUb1c8cq6jH9jObrJwEvTrIPBZbQGpeS7GHN94y6WMnabdfuBOrzKBxbgf2BD7PmQlTU7inAkc7xEuWfB3wIuEurfoNjPPBJ4DlJu5cD+zZff0nTLy/7e8CVwN7N198IvKD1cxbMHV7tkmyvdkm2V7sk26td1rZXv6Rtt35Btle7rA/d2iXt9mrnZamvUM9q5+33aO7zstHc52WjsdPLunNflPfGzqTtpfhzn5eN5j63D92xM2k3mvu8bDR29tZWUe2SfFQ/LxvVz8tGaxd3PRjUzms3qp2XdWuX9SOon9d2VD8v69av9bPm17tZ/ZysW7sg69YuyYfHXzcb1S9o161fkA3r5/XBq13Qrlu7IBsde+45CP66Jcp665Yo661boqy3bgnPmeivW6J2T6Ez7yVZd97L+tFqc27dErXtrVu87PNx5j6Cc0L8c4Yo650zRNlo3Izy3jlDeB5L/5whancp/XOGKOudM4w5l547Z4ja9c4Zomw6bg7d7uxvzXsE8P1Syg9KKb8G/h14chQupXwFuH6o0VLKT0opFzT3b6ZeZd4tyJZSysrm4WbNrURtm9nuwB8BJw/1YzHM7G7Uiwrva/r161LKL0f810OAK0opP0oymwJbmdmm1CfwNUFuX+AbpZRVpZTVwJeBI9qBoAZPpi4qaP59SpQtpVxSSrms+4OD7JlNP6BeGd49yd7UenhXmhomz5l3AK+mU+uxz7Ek+2LgraWUW5vMz4baNTMDnk6dNKNsof52EGA7mhoG2X2ArzT3zwKe1mSj46JXvyjr1S/JRvWL8r0aDhzLC2q4yOM+yvbqN9Ruu35Jtle/JBvVLxqrDqb+Ng/W1M/NllK+XUpZ0dkXUfZzzfcK9ZU/uyfZm1r7Yqs1zfazZrYJ9Tdfrx7qA4Ek/2LgjaWU25vcz4baNrNtm314WpL16udlbwNuLaVc3nx9vn7duaPZV73aedlmW3q1S7K92iXZXu2ytr36RdlIkO3Vbqjddu2SrDt2OtkdCWqXcOc+jzd2Jll37Ayy7tw3wJ3/7iB37st0576AW79Ab+xM1lZu7aK8V78k26tfku3Vb2A9uKB2i1k7Jlm3dkNtt+uXZHv1S7Lu3NfSXu8OHXvz2RHHXjs75thr54eOv+4aPTv2xqznvezQsddrNzn22tmhY6+dzWrXPQf5CcHc52Sviea+IOvOfUE2mvt62Wje87JOP7OsO+8Ntd2d+4JsVL9u9lf4c190Tugde242OPaibHTsRXnv2MvOY7vH3uA571Cf8Y+9tN3OsRdlvdpF2aFxM3VnvxC1G/Uq7JyrCE4cf1tmtoR6Nf3cJLOJmX2H+hans0opYRZ4J/WJfPvILhTgTDM738z+NMndD/g58AEz+7aZnWxmdx3R/jNIFnGllKuBtwM/pg7+N5ZSzgziy4EDzWxHM9uaNW9hGnLPUspPmp/3E+oV4bXt+cDns4CZvcnMrgSeRb3SHOUOB64upVy4iJ//UjNbZmbvN7MdktzewAFmdq6ZfdnMHj6i7QOAn5ZSvpdkXg4c32zf24HXJtnlwOHN/aNwatg5LtL6jTmGRmTd+nXzWQ3b2aEaOv0I69fJpvULts+tXyeb1q+TDevXHauoryj9ZWvynh9DFzOuZVkz2ww4GvhCljWzD1B/O/oA6svXo+xLgdPnnnMj+vCmpnbvMLMtBvL3B/7YzL5lZp83s71G7IsjgLNbi1Iv+0Lgc2Z2VbMv3hrU4zxgMzPbv2n7yFb9unPHjgS1c7KZMNutXZT1apfk3fol/fDq52Xd2mXbR6d2QdatnZO9jrh24M/l0dg5dt4fk22PnW42GTd7+WTsjPrhjZ1eNho7s+3rjp1eNho7vaw3dkZrq6h2i1mLjcnO1S/MOvVzs0Htsj50axdlo9oNbV+7flHWq1+UHVq7tNe7Q+vOdG08MhutOxfkk+NvQXZo3eL0I1t3trND605v+6J1Zzs7tO5sZ93aeecgwPk4c99izleGsu25L8t2574k25v3BvqwYN5Lsu68N2JfzM99SbY39wX1OBV/7ovOCb1jbzHnj2Oy7WMvzDvHnpsNjr2sH91jL8p6x97Q9rWPvSjrHXtRdvCcL1UW8fKpje3W7LCTW4+Ppg4E2f9ZwsBb81rZbagD3lNH5renfmbLA4PvPxE4sbl/EOPemjf3ntWdqW89PDDI7Q+sBh7ZPP4nBl5eR30583XUQSHK7AB8CbgH9Tf2pwHPTvIvoL5P+CvUlzm+Y6gG1Aml/f0bhuqF8xLpJPs66vuFbczzgHrAvsHLUq/+nwts1zxeQf/l0d3tuyf1Jch3ob6X+/1JdjnwLurLgh9BfQmsDWzfe4C/GujDu4CnNfefDnwxyT6A+rLO84FjgV9kx8VA/dxjKKhflO3Vb+j4dGo4nx2qobN9Wf262ax+0fZ59eu2m9Wvm03r12TmxqoDqK8qnfv6HsBFQfaBra8t2GcD2fcC7xyZ3QQ4EXhekD2Q+lkFcy+/Xpm1S337ogFbUH/z9vqB/Mq5WjTPla+O6PPn52qTtPsp1ozNr6I1bznZR1M/x+E84O+pn0HUmzuoY3Kvdl6287PmazciO1+7EdkFtQv6vKtXv6htr35Jtle7EX2er13Sbq92SbZXu9bP6s3lBGOnl21lzmHhW7uy7IKxM8sG46bXZ3fsDLLu2Blk3bFzYPsWjJ1Bu+7YGWR7YyfB2iqpXboWY+FbTIay8/UbyrbrF2SP92qXbF+vdkk2qt3Q9s3XL2m7V78kG859dNa7Uf28bHTsDWSjdUu47qZ//M1nGV63dLcvW7d0s9m6Jdo+b93SbTdbt3Szbu3wz0GOxp/70vMVFo5XQ9n23DeUnZ/7guxz8Oc9t138eS/KumuWEX1uz31R297cF2XduQ/nnJB47AzPH+nPe1nWO+dLz01pHXtBn6N5z8tG856XjcbObPu6857XbjTvednBc4bsNjq4Md6aJ/4ZnSfSawf+zxJGXIhqDrAzgFcssk/HErwvG3gL9cr9CuoV9FXARxbR9tKk7XsBK1qPDwA+O9Dek4EzBzJHAe9rPX4OzQJ8RH/fDLxkqAbUD2nbpbm/C3DZUL0YeSEK+BPqh7BtPfZ5QH2/+nIvCzyI+uqFFc1tNfU3A/ca2XZ327uPvwAc1Hp8BXCPZPs2BX5KfdtT9nNuZM3CwoCbRvZ3b+C87LiI6udlo/pF2aR+6fHZrmE3m9VwRLtLonaz+iXb16tf0K5bvxH9XVC/zveOpS4urmPNAmnBmNrJvrL1eAXB57a0s83902g+v2Co3eZrj8X/LJ9jm9u1rdrdTmtBOtDuQV677Tz1gy6XtPbzjQPbtyP1ZNX9Aw6tfXxF62v3Br47ss+HUn/b6M0dH/VqF2Q/0mpzvnZZtlu7oXa7tQvyN3j1G9n2QdSLWW7Wq93A9i2oXZD9rFe7kf09FDg1eF4spT7fwrmvm209Pgfnc2q6WYKxM2q3+dqCuc/J/x0D81/S9hKv7da+COe+YPvcuc9pN5z7Bvq7N/Wkyl1bRbWL8l79smy3fkPttusXZM8OavegEe0uSdr9bFS7ge1bUL+k7V79Ru6L7tplwXo3qp+XzY49L9ut3VA+Ov7aWQbWngPtLonabR5n605v+6J1Z7fdbN2Z9Xe+dvjnIO/Bn/vS8xUWzn1hlv7cN3geRDP3Bdkf4s97Y9o9KGn3RII1y8D2dee+aB97c9+YPrtzH805IePmvQXnj+Tz3nyWgXnPa9s79jrZYxg373ntLknafQnj5r329g3Ne3Ptjpn3vP6G5wzR7c7+1rxvAnuZ2X3NbHPqyzxPv6ONmplR33t+SSnlhIHsPcxs++b+VsDjqQNDTynltaWU3UspS5q+fqmU8uyk7btafR8vzcuOD6UuBry2rwWuNLN9mi8dQh00Ms9k+KXHPwYeZWZbN/vlEOpn0UR93rn5997Uq/NjXtp8OnXwoPn30yP+zyAzewLw18DhpZRVA9m9Wg8PJ67hRaWUnUspS5o6XkX9wOhrk7Z3aT08gqCGjdOo79vGzPZmzW+NIo8HLi2lXJVkoL4/+LHN/YOpf60i6u9cDe8C/C31qnl2XPTqt8hjyM1G9UvyvRp62aiG1MHda7dXv2T7ovpF+2JB/ZJ2e/VL9kNUP2+suoT6Spwjm/8+V7/R41qUNbMXAn8APLM0n18QZC8zsz1b2/+k5v972fNLKfdq1W5VKWXPpA+7tNp9Cs2xl2zffP2a/X35wL44inrh5ZaBfbxd83wA+H3gkqTPc/Xbgvr8PymYO57l1W4x80yU9WrnZYGjvdolbe/g1S/pR69+yfb1ajewLxbULti+J3u1S/rbq13zOJrLvbFz9LwfZb2xM8m6c1+Q/2Ywdt4ctO2NndH2eWPnqmRfdMfOqF1v7Iz2RW/sTNZW7rplMWuxKOvVL8n26hdkLwjWLhcF7fZql2ybO+8N7IsF9Uuyvfol+8Kd+xrd9W627hyzNnazI9ad3Xy29pzPjlh7dtvN1p3d7cvWnd6+iNad3Wy27uz2N6qddw7yXZy5L8hG5ytu1pv7kqw393nZ8K0DDwAABhtJREFUE7x5L2nXW7dE29ab97I+N99bMPcl+7g39yV9juY+75zQPfaCrMvLZsdekI/mvm72Q9GxF7TrHnvB9rnHXrIvesdekHWPvaC/2bg5bOwVq431Rn2P4+XUq4ivG8h+jPqe1t80T6QXBLnHUD8/YO7POs7/+UUnux/1rRPLmidb760fwf87iIG35lHfB38ha/6899D2PYT6ZxqXNU/uHZLs1tQr4tuN6OsbqAfocupffNgiyX6VOoBdCBwypgbUq/NnNwfK2cDdk+wRzf1bqRcOzkiy36d+hthcDU9Ksp9stm8Z9c9X7jbmOUP/5dFe2x+mvmx4GXUA3iXJbk797f5y6ssnD876Qf0rFn8+Yh8/hvqyywupLzH93SR7DPWYupz6eShzV9Xd48KrX5Lt1S/JRvWL8r0aRlmvhkm7vfol2V79sj5065e026tfko3q545V1HHmvGZ//wf15eBR9mVN/VZTJ7qTk+xq6rg817fXe1nqy5e/3uzj5dRX+twtardTu5UD2/alVrsfYc1fqYvy21N/C38R9bdqD876Qf0t3ROG5gPq8/6ipn7nNPs8yh5PXfBdRudPB3fnDq92SbZXuyTbq52XjWo3dr7Df2tlux9u/YJsr3ZZH7q1S9rt1S7JurUjmMvxx84o642dUbY3dibZaO4bXH+wZuyM2vbGzijrjZ1hH+iPnVG73tgZZaOxs7e28mrX6ouXj9YuXjaa+7xsVL90PcjCV4d47UbrFi/rrluyfnTrl7QdrV28bFS/3no3ql+QjWrnZd3aJfmofukavVM/r92ofl42Wne6fQhq57Ub1c7LurVrvtc7ByGY+4KsO/cFWXfuC7Lu3Odlo3kvaDdat3jZcN6L+oEz9wVtu3NfkI3mvt45IfGx52WjY8/LZseel4+OvaHz2BWsOfa8dqNjz8tGx57bB/xjz2s3Ova8bHjsjbnNDbIiIiIiIiIiIiLr1J39rXkiIiIiIiIiIjIjuhAlIiIiIiIiIiIzoQtRIiIiIiIiIiIyE7oQJSIiIiIiIiIiM6ELUSIiIiIiIiIiMhO6ECUiIiKygTKzYmZ7ru9+iIiIiIylC1EiIiIia4mZrTCz/zOzla3bP6/vfomIiIhMxabruwMiIiIiG5knlVK+uL47ISIiIjJFekWUiIiIyDpmZs81s6+b2bvN7EYzu9TMDml9f1czO93Mrjez75vZi1rf28TM/sbMrjCzm83sfDPbo9X8483se2Z2g5n9i5lZ8//2NLMvNz/vOjP7+Aw3WURERMSlV0SJiIiIzMYjgU8AOwFPBT5lZvctpVwPfAy4GNgVeABwlpn9oJRyNvAK4JnAYcDlwH7Aqla7TwQeDtwNOB/4DPAF4DjgTOBxwObA/ut6A0VERESGWCllffdBREREZKNgZiuoF5pWt778KuA3wJuB3Uqz+DKz84B3A+cAK4DtSyk3N997C7BLKeW5ZnYZ8OpSyqedn1eAA0opX2senwpcUEp5q5l9CLgFeGMp5ap1sLkiIiIii6a35omIiIisXU8ppWzfur23+frVZeFvAH9EfQXUrsD1cxehWt/brbm/B3BF8vOubd1fBWzT3H81YMB5ZnaxmT3/t9weERERkbVGF6JEREREZmO3uc9vatwbuKa53d3Mtu187+rm/pXA/Rf7w0op15ZSXlRK2RX4M+BEM9vzt+u6iIiIyNqhC1EiIiIis7Ez8DIz28zMjgL2BT5XSrkS+B/gLWa2pZntB7wA+Gjz/04GjjOzvazaz8x2HPphZnaUme3ePLwBKMBta3ujRERERBZDH1YuIiIisnZ9xszaF3zOAj4NnAvsBVwH/BQ4spTyiybzTOAk6qujbgCOLaWc1XzvBGAL6geP7wRcChwxoh8PB95pZts1P++YUsoP78iGiYiIiNxR+rByERERkXXMzJ4LvLCU8pj13RcRERGR9UlvzRMRERERERERkZnQhSgREREREREREZkJvTVPRERERERERERmQq+IEhERERERERGRmdCFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ+H9/niiZpk4gWAAAAABJRU5ErkJggg==\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAABKIAAALMCAYAAADXShqaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xmc5XV5J/rPc051V3UViMriCC2CI0bFVlTQGBTnxklsgdF4nUxkJIm5JjF3NDrEEMxczeJkMnOzR6Nm4oxLFhK8xsyYCOISjctoEBAXFkcgCA2YNK0gWxVdVd/7xzldnC6ququ7TlV1db/fr9d58dvPU6f56/N6nu+vWmsBAAAAgJXWWesCAAAAADg0CKIAAAAAWBWCKAAAAABWhSAKAAAAgFUhiAIAAABgVQiiAAAAAFgVgigAYN2oqm5V3VNVxw/zWg5sVfWTVfWpta4DAFg+QRQAsGL6QdCuz2xV3T+w/4p9fV5rbaa1dlhr7eZhXruvqurXquq9w37uEr+7U1UXVNX1/d/zm1X1n6pq4yp9/7/s/1veM+9z2mp8PwCwvo2sdQEAwMGrtXbYru2quinJT7bWPr7Y9VU10lqbXo3a1rG3J/n+JK9IckWSJyZ5b5InJfk/h/lFe/j3uLm1dsIwvwsAODToiAIA1ky/s+iiqvrzqro7yblV9Zyq+kJV3VlVt1fVW6tqQ//6kapqVXVCf/9P++cvqaq7q+rzVXXivl7bP/+iqvrfVXVXVb2tqj5XVa/cj7/p5Kr6u379X62qswbOnV1V1/a/f1tVndc/fkxVXdy/59tV9elFnv3EJK9Ock5r7e9ba9Otta8l+ddJzq6qM6rq9Kq6tao6A/f9cFVd2d/uVNV/qKobquqOqvqLqnpE/9zj+7/ZT1TVzUk+uh9//2f7HVqX93/Lv9r1/P75l1bV1f2/9W+r6nsGzj22qv5HVW3v1/b7uz+6frd/341V9YMDJ15VVTf1f9cbq+rl+1o3ALA6BFEAwFp7aZILkxyR5KIk00len+SoJKcn2Zpe+LKYf5vkzUkemeTmJP9xX6+tqmOSvD/J+f3v/Yckz9rXP6Q/Hvc3ST6c5Ogk5yW5qKoe37/kPUle1Vo7PMlTk/xd//j5SW7s3/PPkrxpka/4l0n+obV25eDB1tpNSb6Y5AeS/K8kO5M8f97ffWF/+7wkZyU5I8nmJPckeeu87zkjvU6rs7J/fqz/OTZJJfndJKmqJyX5kyQ/m97f+vEkH6qqDVU1kt7vdn2SE5I8Jr1/k12+L8lXkxzZf95/7z/zYUl+J8kP9H/X05N8ZT/rBgBWmCAKAFhrn22t/XVrbba1dn9r7YsD3T43Jvmj7B6qzPeB1trlrbWdSf4sySn7ce3ZSa5qrf3P/rnfTXLHfvwtpyfZmOQ3W2s7+2OIlyTZ1aGzM8mTq+rw1tq3BwKlnemFNse31h5orS3YEZVeSHb7IuduT3JUa60l+Ysk5yRJVT08yQv7x5LkZ5L8h9bara21ySS/muSHBzuokvxya+2+1tr9i3zX8f3OpMHP6MD597XWrmmt3Zvkl5K8vKqq/zt8qLX2t/3f+b+kF0A+O8lz+n/fBa21e/v/L3xu4Jk3tNbe3VqbSfK+JJur6qj+uZbkKVU11lq7vbV2zSJ1AwBrTBAFAKy1WwZ3quqJVfXhqvpWVX03yVvSCygW862B7fuSHLbYhXu49tjBOvphzrYl1D7fsemtn9QGjn0zyXH97ZcmeXGSm6vqU1X17P7x/9K/7hP9kbnzF3n+HUkevci5R+fB8OzCJC/rjzS+LMnft9Z2/T3HJ/nrXQFSel1GSXLMwLN2+zdZwM2ttYfP+0wtcv83k4ym14V2bH8/SdJam03vdz4uvQ6om/pB00Lm/9slyWGtte+mF7q9Jsm3qupvquoJe6kfAFgjgigAYK21efv/NcnXkjy+tfaw9DpqaoVruD29MbUkvcWI8mB4tC9uS/KY/v27HJ/k1iTpd3q9OL3Q52/S71JqrX23tXZefwHwH0pyQVUt1AX2iSQnVtUzBg/218E6rX8+rbWvpBfcvDC7j+UlveDnB+aFSGOttbmgZ16Qtj8eM7B9fJKpJN9O7/d57EDdnfR+91vTC68eW1Xdff2y1tolrbV/mV4Yd316/w8BAAcgQRQAcKA5PMldSe7trym0p/WhhuVvkjyjqv5Vf62i16e3htGedKtqbOAzmt76TNNJ3tBf9+j7k5yZ3jpRm6rq31bVw/pjaXcnmU2S/vf+836AdVeSmV3nBrXWrk3y35L8eVU9q6q6VfWUJB9Icklr7VMDl1+Y3npQz+mf3+UPk/x6VR3f/+5jqurF+/BbLcWP9TvbJtIb/Xt/P9x6f5IXV9W/6HdrnZ/e7/D3ST6fZEe/tvH+73X63r6oqh7d//3GkzyQ5N4s8NsBAAcGQRQAcKB5Q5IfTy+g+K/pLWC+olpr/5jkR9Jb9HpHkn+e5EvpdfIs5twk9w98vt4fT/tXSV6S3pjcW5P829baN/r3/HiSb/ZHDl/Vf0aSfE+Sv01v4fDPJfn91tpnFvne/zu9NZL+PL3Q5ZIkH0vyb+Zdd2GS70/ysdbadwaO/06Sj6Q3Bnh3euHZaXv4OxdyfFXdM+/zQwPn/yTJn6bXadZN8u+TpLV2df83eGeS7ektRP/i/npa0+mt1fWk9Lqjbk7vbYB7000v0Lo9vX+770tvTA8AOADV8juvAQAOLv3xsNuS/Os9BEIsoKo+m+S/tdbeu9a1AAAHHh1RAABJqmprVT28P2L35vTeZHfZGpcFAHBQEUQBAPQ8N8mN6Y2MvTDJS+e9CQ4AgGUymgcAAADAqtARBQAAAMCqGFnrAlbbUUcd1U444YS1LgMAAADgoHHFFVfc0Vo7em/XHXJB1AknnJDLL798rcsAAAAAOGhU1TeXcp3RPAAAAABWhSAKAAAAgFUhiAIAAABgVRxya0QBAAAAh5adO3dm27ZtmZycXOtS1r2xsbFs3rw5GzZs2K/7BVEAAADAQW3btm05/PDDc8IJJ6Sq1rqcdau1lh07dmTbtm058cQT9+sZRvMAAACAg9rk5GSOPPJIIdQyVVWOPPLIZXWWCaIAAACAg54QajiW+zsKogAAAABYFdaIAgAAAA4pWz77tWzfOT205x29YSRffe5Thva8g5mOKAAAAOCQMswQaiWeN2hmZmbFnr0WBFEAAAAAK+zee+/NWWedlac97Wl5ylOekve973354R/+4bnzn/rUp3L22WcnSQ477LC84Q1vyNOe9rR8/vOfz8UXX5wnPvGJeeYzn5nXve51c9ct5Fd+5Vfy4z/+43ne856Xxz72sfngBz+YX/iFX8iWLVuydevW7Ny5M0nyxje+MU9+8pPz1Kc+NT//8z+fJNm+fXte9rKX5bTTTstpp52Wz33uc0P/HYzmAQAAAKywj3zkIzn22GPz4Q9/OEly11135c1vfnPuvffeTExM5KKLLsrLX/7yJL3Q6tnPfnZ++7d/O5OTkznppJPy6U9/OieeeGLOOeecvX7XDTfckE9+8pO55ppr8pznPCd/+Zd/md/4jd/IS1/60nz4wx/O8573vPzVX/1VrrvuulRV7rzzziTJ61//+px33nl57nOfm5tvvjkvfOELc+211w71d9ARBQAAALDCtmzZko997GO54IIL8pnPfCZHHHFEtm7dmr/+67/O9PR0PvzhD+clL3lJkqTb7eZlL3tZkuS6667L4x73uJx44olJsqQg6kUvelE2bNiQLVu2ZGZmJlu3bp2r4aabbsoRRxyRsbGxvOpVr8oHP/jBjI+PJ0k+/vGP57WvfW1OOeWUvPjFL853v/vd3HPPPUP9HXREAQAAAKywJzzhCbnyyitz8cUX501velNe8IIX5OUvf3n+4A/+II985CNz6qmn5vDDD0+SjI2Npdvt7vd3jY6OJkk6nU42bNiQqprbn56ezsjISC677LJ84hOfyAc+8IH8wR/8Qf72b/82s7Oz+cIXvpCxsbHl/8GL0BEFAAAAsMJuu+22jI+P59xzz83555+fK6+8Ms9//vNz5ZVX5l3vetfcWN583/M935Mbb7wxN910U5LkoosuWnYt99xzT+66666ceeaZ+d3f/d18+ctfTpL84A/+YN72trfNXXfVVVct+7vm0xEFAAAAHFKO3jAy1DfdHb1h7/HKV7/61Zx//vlzXUrvfOc70+12c/bZZ+e9731v3ve+9y1436ZNm/KOd7wjW7duzcTERE477bRl13v33XfnJS95SSYnJ9Nay+/8zu8kSd761rfmNa95TZ761Kdmeno6Z5xxRv7wD/9w2d83qFprQ33gge7UU09tl19++VqXAQAAAKySa6+9Nk960pPWuoz9ds899+Swww5Lay2vec1rctJJJ+W8885bs3oW+j2r6orW2ql7u9doHgAAAMAB7F3veldOOeWUnHzyybnrrrvy6le/eq1L2m9G8wAAAAAOYOedd95DOqDe85735Pd///d3O3b66afn7W9/+2qWts8EUQAAAADrzE/8xE/kJ37iJ9a6jH1mNA8AAACAVSGIAgAAAGBVCKIAAAAAWBWCKAAAAABWhcXKAQAAgEPKZz777DzwwB1De97GjUflec/9+6E972CmIwoAAAA4pAwzhFqJ5813wgkn5I47VvY7VosgCgAAAIBVccgGUTMzU7nllvfmsi++JJ/93Om58spX5B//8W/SWlvr0gAAAICDzL333puzzjorT3va0/KUpzwlF110US6++OI88YlPzDOf+cy87nWvy9lnn50k2bFjR37wB38wJ598cn7yJ39yj1nFTTfdlCc+8Yl55StfmSc84Ql5xStekY9//OM5/fTTc9JJJ+Wyyy5Lkvzd3/1dTjnllJxyyil5+tOfnrvvvjtJ8pu/+Zs57bTT8tSnPjW//Mu/vOK/wyEZRM3MTOaqL78y//sb/zF33/21TE19K9+58wv52tWvz9e//mZhFAAAADBUH/nIR3Lsscfmy1/+cr72ta9l69atefWrX51LLrkkV1xxRbZv3z537a/+6q/muc99bq6++uq89KUvzc0337zHZ19//fV5wxvekOuuuy7XXXddLrzwwnz2s5/Nb/3Wb+XXf/3XkyS/9Vu/lbe//e256qqr8pnPfCabNm3KRz/60XzjG9/IZZddlquuuipXXHFFPv3pT6/o73BIBlE33/Lfc+edly147tbb/jw7dnxqdQsCAAAADmpbtmzJxz72sVxwwQX5zGc+k3/4h3/I4x73uJx44olJknPOOWfu2k9/+tM599xzkyRnnXVWHvGIR+zx2SeeeGK2bNmSTqeTk08+OS94wQtSVdmyZUtuuummJMnpp5+en/u5n8tb3/rW3HnnnRkZGclHP/rRfPSjH83Tn/70POMZz8h1112Xb3zjGyvzA/Qdkm/Nu/22D+zx/G23fyBHHfV/rFI1AAAAwMHuCU94Qq688spcfPHFedOb3pQXvOAFQ3v26Ojo3Han05nb73Q6mZ6eTpK88Y1vzFlnnZWLL744p59+ei699NK01vKLv/iLefWrXz20WvbmkOyImpy6fY/npyZvW6VKAAAAgNW2ceNRq/682267LePj4zn33HNz/vnn53Of+1xuvPHGuY6liy66aO7aM844IxdeeGGS5JJLLsl3vvOdZdd4ww03ZMuWLbngggty2mmn5brrrssLX/jCvPvd784999yTJLn11lvzT//0T8v+rj05JDuixkYfnfsnF5+vHB07dhWrAQAAAFbT857796v+nV/96ldz/vnnp9PpZMOGDXnnO9+Z22+/PVu3bs3ExEROO+20uWt/+Zd/Oeecc05OPvnkfN/3fV+OP/74ZX//7/3e7+WTn/zk3Pjei170ooyOjubaa6/Nc57znCTJYYcdlj/90z/NMcccs+zvW0wdagtzn3rqqe0DH/i/csONv73oNU972n/PUUf+i9UrCgAAAFgx1157bZ70pCetdRkPcc899+Swww5Lay2vec1rctJJJ+W8885b67L2aqHfs6quaK2durd7D8nRvMc85lV5+MOfveC54457RY585PNXuSIAAADgUPOud70rp5xySk4++eTcddddq7pW01o5JEfzut3RPP2U9+TW296fG274jczM3Jeqbk4++fdzzNFbU1VrXSIAAABwkDvvvPOW3AG1Y8eOBRc4/8QnPpEjjzxy2KWtmEMyiEqSTmc0j9n8o7lj+8fz7e98NknlUce8aK3LAgAAAFZAa21dN54ceeSRueqqq9a6jCx3iadDcjRvULe7KUnS2nRmZx9Y42oAAACAYRsbG8uOHTuWHaIc6lpr2bFjR8bGxvb7GYdsR9Qu3e743PbMzP3pdDauYTUAAADAsG3evDnbtm3L9u3b17qUdW9sbCybN2/e7/sFUbsFUfdmw4Yj1rAaAAAAYNg2bNiQE088ca3LIEbzHtIRBQAAAMDKEETN64gCAAAAYGUIovqLlSc6ogAAAABW0iEfRHV264i6bw0rAQAAADi4HdJB1M33T+UT39k5t//dB+5ew2oAAAAADm6HbBD1tm/+Y773C9fmg9un5o790tdvyCXb71zDqgAAAAAOXodkEHXJ9jvzn268PbNJpjI6d7za/fnpq7+Z6++bXLviAAAAAA5Sh2QQ9Ye3bJ/bnszY3PZYJrOztbxn2x1rURYAAADAQe2QDKK+9N0HFyWfGgiiRtMb0/vS3RYtBwAAABi2QzKI2tR98M+e3C2I6o3kjXcOyZ8FAAAAYEUdkonLWUcfMbc9uEbUWD+IOnPgPAAAAADDcUgGUa9/7KPy8JFukoeO5j1xYiw/8uhHrlVpAAAAAAetQzKIeuym0fzPZ5yU5z3isN06ok4cnckHTnl8JrrdNawOAAAA4OB0SAZRSfI9E2P5/055fJ5xxMOyMyNJkpPHk6M2jqxxZQAAAAAHp0M2iNrlERtG5sbzZma8LQ8AAABgpaxaEFVVW6vq61V1fVW9cYHzx1fVJ6vqS1X1lao6s3/8FVV11cBntqpO6Z/7VP+Zu84ds691jXc7c2/O2zktiAIAAABYKasyh1ZV3SRvT/IDSbYl+WJVfai1ds3AZW9K8v7W2jur6slJLk5yQmvtz5L8Wf85W5L8j9baVQP3vaK1dvn+1jbR7cytEzWtIwoAAABgxaxWR9SzklzfWruxtfZAkr9I8pJ517QkD+tvH5HktgWec07/3qEZ7Iiamb1/mI8GAAAAYMBqBVHHJbllYH9b/9igX0lyblVtS68b6mcXeM6PJPnzecfe0x/Le3NV1UJfXlU/XVWXV9Xl27dv3+3cRLc7t0bUrI4oAAAAgBVzIC1Wfk6S97bWNic5M8mfVNVcfVX17CT3tda+NnDPK1prW5I8r//50YUe3Fr7o9baqa21U48++ujdzo0PjOZl9v60NjvEPwkAAACAXVYriLo1yWMG9jf3jw16VZL3J0lr7fNJxpIcNXD+5ZnXDdVau7X/37uTXJjeCOA+6QVRY3P7MzPG8wAAAABWwmoFUV9MclJVnVhVG9MLlT4075qbk7wgSarqSekFUdv7+50k/yYD60NV1UhVHdXf3pDk7CRfyz4aXCMqsU4UAAAAwEpZlbfmtdamq+q1SS5N0k3y7tba1VX1liSXt9Y+lOQNSd5VVeelt3D5K1trrf+IM5Lc0lq7ceCxo0ku7YdQ3SQfT/Kufa1tvDMwmhfrRAEAAACslFUJopKktXZxeouQDx77pYHta5Kcvsi9n0ryvfOO3Zvkmcuta2LeaN60IAoAAABgRRxIi5WviflrROmIAgAAAFgZgqhuN5MDo3kWKwcAAABYGYd8EDV/NG9m5t41rAYAAADg4HXIB1HzR/N0RAEAAACsjEM+iJrodjK5WxBljSgAAACAlXDIB1G9jqjBNaIEUQAAAAAr4ZAPojZWZWfpiAIAAABYaYd8EFVV6XQ3ze0LogAAAABWxiEfRCVJpzM+t22xcgAAAICVIYhKMtIdDKJ0RAEAAACsBEFUkq4gCgAAAGDFCaKSjHY3ZTaVJJmZFUQBAAAArARBVJJNI91MZTRJMjMtiAIAAABYCYKoJOPdTqYyliTZaTQPAAAAYEUIopJMdDtzHVHTgigAAACAFSGISjLe7Way3xE1M3P/GlcDAAAAcHASRGVXR9SuIEpHFAAAAMBKEEQlGe88OJrXvDUPAAAAYEUIotJbrHzXaF7adGZnH1jbggAAAAAOQoKo7D6al1gnCgAAAGAlCKLS64jaPYgyngcAAAAwbIKo7OqIGp3bF0QBAAAADJ8gKsl4t5tJQRQAAADAihJEZVdH1Ka5fWtEAQAAAAyfICq71oga7Ii6dw2rAQAAADg4CaKyQBA1qyMKAAAAYNgEUemN5k0OjuZNWyMKAAAAYNgEUUnGO/M7ogRRAAAAAMMmiEqy6SFrRBnNAwAAABg2QVSSTlVaZ3xu32LlAAAAAMMniOrrdAfWiNIRBQAAADB0gqi+bndibntmxhpRAAAAAMMmiOrr7tYRJYgCAAAAGDZBVN/G7uAaUYIoAAAAgGETRPWNjWzMzowkSWatEQUAAAAwdIKovoluJ1MZS5Ls9NY8AAAAgKETRPWNd7uZymiSZHraaB4AAADAsAmi+sY7nUz2O6KmrREFAAAAMHSCqL7B0byZWWtEAQAAAAybIKpvvPtgR9SsjigAAACAoRNE9Y13O3NrRGX2/rQ2u7YFAQAAABxkBFF9uwVRSWZnJ9ewGgAAAICDjyCqrzeat2luf8Z4HgAAAMBQCaL6Jrrd3TqiBFEAAAAAwyWI6ps/mjcz4815AAAAAMMkiOqb6HYyZTQPAAAAYMUIovp6a0QZzQMAAABYKYKovl5H1NjcviAKAAAAYLgEUX3jHUEUAAAAwEoSRPU9dLFyQRQAAADAMAmi+nprRA12RHlrHgAAAMAwCaL6NnY6manBIOreNawGAAAA4OAjiBrQ6W6a29YRBQAAADBcgqgB1Rmf256ZtUYUAAAAwDAJogaMjAwEUdOCKAAAAIBhEkQNGOkOdkQZzQMAAAAYJkHUgA2dscymklisHAAAAGDYBFEDJkY25IGMJkmmjeYBAAAADJUgasB4t5PJjCVJpr01DwAAAGCoBFEDJrqdTO3qiDKaBwAAADBUgqgB4wNB1IyOKAAAAIChEkQN6I3mbUqSzMxYIwoAAABgmARRAwZH89qsIAoAAABgmARRAwZH89KmMzu7c20LAgAAADiICKIGTHS7c6N5iXWiAAAAAIZJEDVgt46oJDPenAcAAAAwNIKoAeOdTqYyNrevIwoAAABgeFYtiKqqrVX19aq6vqreuMD546vqk1X1par6SlWd2T9+QlXdX1VX9T9/OHDPM6vqq/1nvrWqajk1TnQ7mdQRBQAAALAiViWIqqpukrcneVGSJyc5p6qePO+yNyV5f2vt6UlenuQdA+duaK2d0v/8zMDxdyb5qSQn9T9bl1NnbzRPRxQAAADASlitjqhnJbm+tXZja+2BJH+R5CXzrmlJHtbfPiLJbXt6YFU9OsnDWmtfaK21JH+c5IeWU+RDgqjZ+5bzOAAAAAAGrFYQdVySWwb2t/WPDfqVJOdW1bYkFyf52YFzJ/ZH9v6uqp438Mxte3nmPnnoYuWCKAAAAIBhOZAWKz8nyXtba5uTnJnkT6qqk+T2JMf3R/Z+LsmFVfWwPTznIarqp6vq8qq6fPv27YteN9HtZnK30TxBFAAAAMCwrFYQdWuSxwzsb+4fG/SqJO9Pktba55OMJTmqtTbVWtvRP35FkhuSPKF//+a9PDP9+/6otXZqa+3Uo48+etEirREFAAAAsHJWK4j6YpKTqurEqtqY3mLkH5p3zc1JXpAkVfWk9IKo7VV1dH+x81TV49JblPzG1trtSb5bVd/bf1vejyX5n8spclOnjOYBAAAArJCR1fiS1tp0Vb02yaVJukne3Vq7uqrekuTy1tqHkrwhybuq6rz0Fi5/ZWutVdUZSd5SVTuTzCb5mdbat/uP/ndJ3ptkU5JL+p/9VlWp7ngy09sXRAEAAAAMz6oEUUnSWrs4vUXIB4/90sD2NUlOX+C+v0zyl4s88/IkTxlmnZ3OJkEUAAAAwAo4kBYrPyB0u+Nz24IoAAAAgOERRM0zGETNWqwcAAAAYGgEUfNsGAiipmfuXcNKAAAAAA4ugqh5No2MZrq/dJaOKAAAAIDhEUTNM97tZDJjSZKd1ogCAAAAGBpB1DwT3U6mMpokmZ42mgcAAAAwLIKoecYHgyijeQAAAABDI4iaZ1O3k8lsSpLMzBrNAwAAABgWQdQ8g6N5s9aIAgAAABgaQdQ8450HFyvP7GRam13bggAAAAAOEoKoeSZGunMdUUnL7OzkmtYDAAAAcLAQRM0z3ulkaldHVJIZ43kAAAAAQyGImmeiOzCal2TGm/MAAAAAhkIQNc/4wGLliY4oAAAAgGERRM3TC6KM5gEAAAAMmyBqnoeO5gmiAAAAAIZBEDWP0TwAAACAlSGImmei281UNs3tW6wcAAAAYDgEUfPoiAIAAABYGYKoecY7nUwKogAAAACGThA1z0inMtsZHM0TRAEAAAAMgyBqAR1BFAAAAMDQCaIW0OmMz23PzAqiAAAAAIZBELWA7shAEOWteQAAAABDIYhawIbuxNy20TwAAACA4RBELWBDdyyzqSSCKAAAAIBhEUQtYHxkJA9kNIkgCgAAAGBYBFELmOh2MtUPonZOC6IAAAAAhkEQtYDxbieTGUuSTOuIAgAAABgKQdQCBjuiBFEAAAAAwyGIWsB4p5OpfkfUrCAKAAAAYCgEUQsYHM1rs/evcTUAAAAABwdB1AImut250by0nZmd3bm2BQEAAAAcBARRCxjvPjialyQzM7qiAAAAAJZLELWAiYHRvCSZmbVOFAAAAMByCaIWMD7w1rwkmZkWRAEAAAAslyBqAeM6ogAAAACGThC1AGtEAQAAAAyfIGoBDxnNm7l3DasBAAAAODgIohYw0e3qiAIAAAAYMkHUAsY789aI0hEFAAAAsGyCqAVMPGQ0T0cUAAAAwHIJohYw2qk8MNARNTvjrXkAAAAAyyWIWkBVpbrjc/vTgigAAADGmaJ+AAAgAElEQVSAZRNELaLT2TS3PWs0DwAAAGDZBFGL6A50RM3oiAIAAABYNkHUIjaMCKIAAAAAhkkQtYgRHVEAAAAAQyWIWsSmkdFMZySJxcoBAAAAhkEQtYjxbidTGU0iiAIAAAAYBkHUIjZ1OpnMWJJkpyAKAAAAYNkEUYuYGOiImpkWRAEAAAAslyBqEePdTiazKUkyMyuIAgAAAFguQdQiJrrduY6o2Zn717gaAAAAgPVPELWIwcXKM3t/WmtrWxAAAADAOieIWkRvNG+sv9cyOzu5pvUAAAAArHeCqEX0Fisfm9uf8eY8AAAAgGURRC1it9G8CKIAAAAAlksQtYiJ3UbzBFEAAAAAyyWIWsR4x2geAAAAwDAJohYxMdI1mgcAAAAwRIKoRYx35o/m3b+G1QAAAACsf4KoRYx7ax4AAADAUAmiFjEhiAIAAAAYKkHUIjZ1O5kcXCNq1mgeAAAAwHIIohbRrUo6m+b2Z6bvXcNqAAAAANY/QdSeDAZROqIAAAAAlmXVgqiq2lpVX6+q66vqjQucP76qPllVX6qqr1TVmf3jP1BVV1TVV/v//f6Bez7Vf+ZV/c8xw6y52x2f27ZGFAAAAMDyjKzGl1RVN8nbk/xAkm1JvlhVH2qtXTNw2ZuSvL+19s6qenKSi5OckOSOJP+qtXZbVT0lyaVJjhu47xWttctXou4RQRQAAADA0KxWR9SzklzfWruxtfZAkr9I8pJ517QkD+tvH5HktiRprX2ptXZb//jVSTZV1WhWwUh3Ym5bEAUAAACwPKsVRB2X5JaB/W3ZvaspSX4lyblVtS29bqifXeA5L0tyZWttauDYe/pjeW+uqhpizdnYHZvbFkQBAAAALM+BtFj5OUne21rbnOTMJH9SVXP1VdXJSf7fJK8euOcVrbUtSZ7X//zoQg+uqp+uqsur6vLt27cvuaDxkZFMphdGTQuiAAAAAJZltYKoW5M8ZmB/c//YoFcleX+StNY+n2QsyVFJUlWbk/xVkh9rrd2w64bW2q39/96d5ML0RgAforX2R621U1trpx599NFLLnq828lUelOA09OCKAAAAIDlWK0g6otJTqqqE6tqY5KXJ/nQvGtuTvKCJKmqJ6UXRG2vqocn+XCSN7bWPrfr4qoaqapdQdWGJGcn+dowi54YDKJ0RAEAAAAsy6oEUa216SSvTe+Nd9em93a8q6vqLVX14v5lb0jyU1X15SR/nuSVrbXWv+/xSX6pvxbUVVV1TJLRJJdW1VeSXJVeh9W7hln3eLdjNA8AAABgSEZW64taaxentwj54LFfGti+JsnpC9z3a0l+bZHHPnOYNc430e1mqh9Ezc7cv5JfBQAAAHDQO5AWKz/gDK4R1WZ1RAEAAAAshyBqDwZH89J2ZnZ2em0LAgAAAFjHBFF70OuIGpvbn501ngcAAACwvwRRezDeGeiISjI9c+8aVgMAAACwvgmi9mBiYI2oxILlAAAAAMshiNqD+aN5OqIAAAAA9p8gag8mut3dRvNmdEQBAAAA7DdB1B6MP2Q07741rAYAAABgfRNE7cFDR/MEUQAAAAD7SxC1BxPd3d+apyMKAAAAYP8JovZg/mieNaIAAAAA9p8gag82VmW6Bhcr99Y8AAAAgP0liNqDqkp1x+f2dUQBAAAA7D9B1F50OoNBlDWiAAAAAPaXIGovuoIoAAAAgKEQRO1Fd2TT3LbRPAAAAID9J4jai427rRFlsXIAAACA/SWI2ovRkdFMZyRJMm00DwAAAGC/CaL2YqLbyVRGkyTTRvMAAAAA9psgai/GB4KondNG8wAAAAD2lyBqL8a73UxmLIm35gEAAAAshyBqL3qjef0gatZoHgAAAMD+EkTtxXinM9cRNasjCgAAAGC/CaL2YnCx8szen9ba2hYEAAAAsE4JovZifGA0L2mZnZ1c03oAAAAA1itB1F5MdB8czUssWA4AAACwvwRRezE+OJqXZGbGguUAAAAA+0MQtRfj3e7AaF4yM3PvGlYDAAAAsH4JovbioaN5OqIAAAAA9ocgai8eOppnjSgAAACA/SGI2guLlQMAAAAMhyBqL3odUYIoAAAAgOUSRO3FeGfeaN6sNaIAAAAA9ocgai82Gc0DAAAAGApB1F50qpLOprl9QRQAAADA/hFELUG3Mz63LYgCAAAA2D+CqCWoriAKAAAAYLkEUUuwoTs4mmexcgAAAID9IYhagu7IxNz2zMy9a1gJAAAAwPoliFqCjd3Bt+bpiAIAAADYH4KoJZgYGclkRpMk09aIAgAAANgvgqglGO92M5VeV9TOaaN5AAAAAPtDELUEE91OpuY6oozmAQAAAOwPQdQSjHc6cx1RFisHAAAA2D+CqCUY73Yy2Q+iZnVEAQAAAOwXQdQSjA+M5rVZQRQAAADA/hBELUEviOp1RKU9kNnZ6bUtCAAAAGAdEkQtwcTAaF6SzOqKAgAAANhngqglGO9250bzkmRm5r41rAYAAABgfRJELcH8jihBFAAAAMC+E0QtwW5rREUQBQAAALA/BFFLMPjWvCSZmbFGFAAAAMC+EkQtwUNH8+5dw2oAAAAA1idB1BKMd+aP5umIAgAAANhXgqglmHjIaJ6OKAAAAIB9JYhagvFud95ono4oAAAAgH0liFqCDZ3KTG2a2/fWPAAAAIB9J4haouoKogAAAACWQxC1RJ3O+Ny2IAoAAABg3wmilmhksCNq1hpRAAAAAPtKELVEnZGBjqhpHVEAAAAA+0oQtUQbuwNB1KwgCgAAAGBfCaKWaGNVptNNktx997W5775/WOOKAAAAANYXQdQS3H77B3PfnZ/KVMaSJPdNbsvnv/ADuemmd6xxZQAAAADrhyBqL+6+++pcc+0FGWv3ZSqjSZKWStJyw42/ne3bP7a2BQIAAACsE4Kovbhl2x8nmc1opjLZ74jqBVE9N9/ynjWqDAAAAGB9EUTtxd13X50kGc3k3GhepT3kPAAAAAB7Jojai27/bXmjmcxdOSJJ0sns3PmR7sSa1AUAAACw3qxaEFVVW6vq61V1fVW9cYHzx1fVJ6vqS1X1lao6c+DcL/bv+3pVvXCpzxyGY455UZJkLFPZkaN63zt4/lFnLnAXAAAAAPOtShBVVd0kb0/yoiRPTnJOVT153mVvSvL+1trTk7w8yTv69z65v39ykq1J3lFV3SU+c9mOffS/ycTEEzKaybkgapeNGx+Vxx7/U8P+SgAAAICD0n4FUVX1vKo6fR9ueVaS61trN7bWHkjyF0leMu+aluRh/e0jktzW335Jkr9orU211v4hyfX95y3lmcs2MjKRZzz9z/LPjnjqbkHU4YdvyanPvCijo48a9lcCAAAAHJSWFERV1aeq6rn97Z9P8sEkH6iqC5b4PccluWVgf1v/2KBfSXJuVW1LcnGSn93LvUt55q76f7qqLq+qy7dv377Ekh+0ceMj84THnrtbEHXM0VuzadNj9vlZAAAAAIeqpXZEbUnyhf72q5P8iyTPTvLvhljLOUne21rbnOTMJH9SVUMZHWyt/VFr7dTW2qlHH330fj1jvNPJHQNB1OTU7cMoDQAAAOCQMbLE6zpJZqvqcUlGWmtXJ0lVPXKJ99+aZLB9aHP/2KBXpbcGVFprn6+qsSRH7eXevT1zaMa7nXwnR87tT03etoerAQAAAJhvqR1H/yvJ7yX5jSR/lST9UGrHEu//YpKTqurEqtqY3uLjH5p3zc1JXtB/9pOSjCXZ3r/u5VU1WlUnJjkpyWVLfObQjHc7ma4N+U4enkRHFAAAAMC+WmoQ9cokk0m+nuSX+seenORtS7m5tTad5LVJLk1ybXpvx7u6qt5SVS/uX/aGJD9VVV9O8udJXtl6rk7y/iTXJPlIkte01mYWe+YS/559Nt7t/VTf7o/nTeqIAgAAANgn1Vpb6xpW1amnntouv/zyfb7vjgem85TPfS2vb7+ZZ/WXy3r+GV/JyMjEsEsEAAAAWFeq6orW2ql7u26pb817fVWd0t9+VlXdWFXfqKpnLbfQ9WJXR9Tgm/OmjOcBAAAALNlSR/PekOSm/vZ/SfL2JL+V5K0rUNMBaVOnUsnub84zngcAAACwZEsNoh7eWruzqg5LckqS32ut/dckT1y50g4cN943lZd+6fq07N4R9b/vvGnNagIAAABYb0aWeN22qnp2kpOTfKa1NlNVhyeZWbnSDgw7HpjOy750fW5/YGdvfyCIev/N1+ToR0/m8eNja1UeAAAAwLqx1I6oX0jy10nekuTX+sfOTvLFlSjqQPLeW++YC6GS3YOoh7Xt+YNv/tNalAUAAACw7iypI6q19jdJjpl3+K/6n4Pax3d8d7f97+aITGckI5nOI7MjH5h3HgAAAICFLXU0L1X1uCQvT3JckluTXNRau2GlCjtQTLe2236rTna0I/Oo/GOOyvaHnAcAAABgYUsazauqM5N8Ob2Fyu9L8rQkX6qqs1awtgPCsx8+8ZBju8bzHpkd+d4jxle7JAAAAIB1aakdUf85yQ+11j6x60BVfX+S30vy4ZUo7EDxquOOzp/dtiP3zz7Y+bQriNqYnfmZRy+5qQwAAADgkLbUxcqPT/Kpecc+3T9+UDtxfDR/vOVxOXLDg4HT4ILlJ49aIwoAAABgKZYaRH05yb+fd+x1Sb4y3HIOTM975OG54jlPzmsf01uvfTCImpy6ba3KAgAAAFhXlhpE/bskr6mqW6rqc1V1c5LX9o8fEsa6nbzsnz0iybwgalIQBQAAALAUS1rgqLV2TVV9T5LTkxyb5LYk/6u19sBKFneg2Ty2McnuQdTU5O1rVQ4AAADAurLklbZbazszsE5UVW2sqhtba49bicIORIePdHN4t5Md04OjeYIoAAAAgKVY6mjeQirJCUOqY904bmxj7q/x3J/xJMmkjigAAACAJVlOEJUkbShVrCPHjfbG8+7oj+dNWSMKAAAAYEmWG0Qdco4b25BkIIh64J8yO7tzLUsCAAAAWBf2uEZUVb17f+89WO3qiPr23ILlLVNT/5hNmzavXVEAAAAA68DewqRb93L+14dVyHpx7LyOqCSZnLxNEAUAAACwF3sMolprb16tQtaLBzuijpw7NuXNeQAAAAB7ZY2offTgGlFHzx2btGA5AAAAwF4JovbRo0c3pJLsGOiImtQRBQAAALBXgqh9tLHTyTEbR/KdHJmWSqIjCgAAAGApBFH74djRjZmuDbkrD09ijSgAAACApdjbW/OSJFX1Y4ucmkqyLcllrbWdQ6vqAHfc2IZ86e7em/Menu/oiAIAAABYgiUFUUl+OslpSXakFzwdl+SoJF9KckKSB6rqh1prV65EkQea48Z6b87bkaPy+Hwj09PfzfT0PRkZOWyNKwMAAAA4cC11NO/KJG9srR3bWntWa+24JBck+fskxyZ5d5K3rVCNB5zjRntvzrNgOQAAAMDSLTWI+tEkb5137G1Jfqy1NpvkPyc5eZiFHcge7Ig6eu7YlPE8AAAAgD1aahD1T0leNO/Y1iTb+9tjSWaGVdSB7tjRXhB1R46aOzY5qSMKAAAAYE+WukbUv09yUVV9KcktSR6T5OlJfqR//nuTvGP45R2YNo/1RvO+PRhETemIAgAAANiTJQVRrbVLqurxSc5Kb02ov03yw621f+qfvzTJpStW5QHmyA0jGe1U7pgZ7IgSRAEAAADsyVI7otIPnd6zgrWsG52qPHp0Q75538MynZGMZDpTRvMAAAAA9mhJa0RV1WOr6o+r6itVdePgZ6ULPFAdN7oxrTrZ0R/PM5oHAAAAsGdL7Yi6ML21of6fJPetXDnrx7H9daJ25Mg8Kt/K1NS30tpsqpa6/jsAAADAoWWpQdSWJGe01g6ZN+Ptzea5N+cdnSSZnX0gD+z8dkY3HrWn2wAAAAAOWUtt3/lskqeuZCHrzXFjvSBqx8Cb86YsWA4AAACwqKV2RH0jyaVV9YEk3xo80Vp7y9CrWgeOHX1wNG+Xycnb87CHyesAAAAAFrLUIOqRSS5Ncnj/s0sbekXrxIMdUUfPHbNgOQAAAMDilhREtdZ+dKULWW+OW6Ajamry9rUqBwAAAOCAt2gQVVWbW2vb+tvHL3Zda+3mlSjsQHfYSDdHjHTz7Z0PrhE1aY0oAAAAgEXtqSPq2jw4hndTemN4Ne+alqQ7/LLWh+NGN+Sa6fHc3yayKfdmckpHFAAAAMBi9vTWvCMGtjck2dj/7+Bn48qVduA7tr9O1Pb+eJ6OKAAAAIDFLRpEtdZmB7ZnFvusTpkHpgfXieotWP7AA9szO/vAWpYEAAAAcMBa0mLlVfXYJP8xySlJDhs811p73ArUtS48+Oa8XQuWt0xN/WM2bXrM2hUFAAAAcIBaUhCV5MIktyT5f5Lct3LlrC/zO6KS3nieIAoAAADgoZYaRG1JcsahPoo330M7omLBcgAAAIBF7Gmx8kGfTfLUlSxkPTp2riPqqLljUxYsBwAAAFjQUjuivpHk0qr6QJJvDZ5orb1l6FWtE48e3ZjK7kGUjigAAACAhS01iHpkkkuTHN7/7NKGXtE6sqFTedTGDdk+9ci0VCotkzqiAAAAABa0pCCqtfajK13IenXc2IZ864ENuas9Ig/PtwVRAAAAAItYNIiqqs2ttW397eMXu661dvNKFLZeHDe2MVd8977ckSPz8Hw7U0bzAAAAABa0p46oa/PgGN5N6Y3h1bxrWpLu8MtaP3YtWH5Hjsrj841MT9+d6em7MzJy+F7uBAAAADi07OmteUcMbG9IsrH/38HPxpUrbX3YPNb7Cb49uGD5pK4oAAAAgPkWDaJaa7MD2zOLfVanzAPXYEfULpNT1okCAAAAmG9Ji5VXVTfJq5M8P8lRGRjRa619/8qUtj4c1++I2jEQRE3piAIAAAB4iD2N5g36nSSvS3JZkmcn+XCSzUk+u0J1rRvHjT40iPLmPAAAAICHWmoQ9a+TbG2t/XaSmf5/X5LkjBWrbJ04ckM3Y53Kjhw9d8xoHgAAAMBDLTWIGk/yzf72fVW1qf3/7N13lORVnf//161cnXOenGdggrSDIIigAmICAwLm7H4VFRe/YljToqu7urhr5ofCV0UMrO7iqiAGUIJAT07MTE/sNNO5q7uru7rC/f3R1TXV09VhpLuqw/Nxzhw+de+tT78/zAzn8Dr3vj/WHpD0vJkpa+4wxqjK61FAeQpruF8UzcoBAAAAAADGmmoQ9ayk2vj1NkmfMcbcJomtP4o3LDcmcTyPHlEAAAAAAABjTTWIukXSyFv0/lHSRZLeIOn9M1HUXHOmYXmxJGkwdEpJLx0EAAAAAACApvDWvPgb81ZL+pkkWWsPSnrxzJY1t1T7ho/kjeyIsnZIQ0Md8npLJ/oaAAAAAADAgjLpjihrbVTSN6y1oTTUMyelenNeKMTxPAAAAAAAgGRTPZr3G2PMNTNayRx29o4oSRocpH0WAAAAAABAskmP5sU5JP3SGPOYpAZJdmTCWvvOmShsLqmK74hqJ4gCAAAAAAAY11SDqMOS/m0mC5nLqr3DO6I6k4MojuYBAAAAAACMMmEQZYy50Vp7n7X2n9JV0FyU7XKq0OVUR5gdUQAAAAAAAOOZrEfU99JSxTxQ5XNr0PgVVLYkmpUDAAAAAACcbbIgykzXDzLGXG2MOWiMqTfG3JZi/g5jzM74r0PGmO74+OVJ4zuNMYPGmGvjc/cYY44lzW2ernrP1cib89rix/PYEQUAAAAAADDaZD2inMaYyzVBIGWt/dNkP8QY45T0LUkvk9Qo6RljzAPW2v1J97klaf3NkrbEx/8saXN8vEhSvaTfJ93+Y9ba+yerYaZV+YaDqA6VaIlOaGioTbFYSA6HN8OVAQAAAAAAzA6TBVFeSd/X+EGUlbR8Cj9nq6R6a+1RSTLG/FTSayTtH2f9jZI+m2L89ZJ+Z60NTuFnptVIw/KOpIblodBp+f2LM1USAAAAAADArDLZ0bx+a+1ya+2ycX5NJYSSpGpJDUmfG+NjYxhjlkhaJinVTqsbJN131tgXjTG740f7Um4/Msa81xhTZ4ypa2trm2LJ56YmaUfUCI7nAQAAAAAAnDFZEJUJN0i631obTR40xlRKOl/SQ0nDn5C0VtLzJRVJ+niqG1pr77TW1lpra0tLS2ek6KoUO6IGB2lYDgAAAAAAMCJdzcqbJC1K+lwTH0sl1a4nSbpe0q+steGRAWttix0WknS3ho8AZkR1qh1RIXZEAQAAAAAAjJgwiLLW5k7Tz3lG0ipjzDJjjEfDYdMDZy8yxqyVVCjpyRT3uFFnBVTxXVIyxhhJ10raO031nrMKj1sOndUjih1RAAAAAAAACZM1K58W1tqIMeaDGj5W55T0A2vtPmPMFyTVWWtHQqkbJP3UWmuTv2+MWarhHVWPnnXre40xpRreubVT0vtn7ikm5nIYVXjdOj1YJCsjI8uOKAAAAAAAgCRpCaIkyVr7W0m/PWvsM2d9/tw43z2uFM3NrbVXTF+Fz12116PmUFjdtkiF6qBZOQAAAAAAQJLZ2Kx8zqryDTcsb1exJJqVAwAAAAAAJCOImkbV3tENy6PRPkUivZksCQAAAAAAYNYgiJpGIzuiRr05j+N5AAAAAAAAkgiiplXNWTuiJIIoAAAAAACAEQRR06g61Y6oEH2iAAAAAAAAJIKoaVUV3xHVzo4oAAAAAACAMQiiplGR2ymfw4zaEXXy5J06cOATGhhoymBlAAAAAAAAmUcQNY2MMSo1AfUqT0MaPqZnbVTNLT9X3bbXamDgZIYrBAAAAAAAyByCqGnU07NT+ZGjkjHqTNoVJUlDQ+2qP/JvGaoMAAAAAAAg8wiiptGpU/+tYrVLGt0nakRb2+8VifSnuywAAAAAAIBZgSBqGg2FO5OCqNIx89ZGFIn2prssAAAAAACAWYEgahplZy1PBFEntGzMvMuVJ4+7KN1lAQAAAAAAzAoEUdOoqup6lZguSdJRrUw573B40l0WAAAAAADArEAQNY18vipduPydkqQTWqKYTGKuoOBCLV/2kUyVBgAAAAAAkHEEUdNsY83VkqSw8apDFZIkh8OnzZvukdPpz2RpAAAAAAAAGUUQNc2ynA4VuZ2SpGb3RklSLDaowcHGTJYFAAAAAACQcQRRM6DaO9wH6kBsRWIs0Ls7U+UAAAAAAADMCgRRM6DK55Yk7YmeeXNeb2BPpsoBAAAAAACYFQiiZsDIjqgGLZIxw9eBXoIoAAAAAACwsBFEzYBq33D4FDVuyb9aktTbu0+xWCSTZQEAAAAAAGQUQdQMqPa6E9f93nWShhuWB4NHMlUSAAAAAABAxhFEzYCRHVGS1O5clbgOBGhYDgAAAAAAFi6CqBnQF4kmrn/WU5m4pk8UAAAAAABYyAiiptl9LR26affRxOdd4XINyitJ6u7ZlamyAAAAAAAAMo4gahqdGAjpYwcbZJPGrHHquJZLknr7nlUsFspMcQAAAAAAABlGEDWN7mvpVMSOHT+qFZIkhyLqCjyb5qoAAAAAAABmB4KoaXR0IPVup2Nambg+1c3xPAAAAAAAsDARRE2jUrcr5fiR+I4oSYoG96WrHAAAAAAAgFmFIGoavaGiKOV4qyrUryxJ0kDf3nSWBAAAAAAAMGsQRE2jzXlZem9N6ZhxaxxqMqskSf39hxWNDqS7NAAAAAAAgIwjiJpmn19Zpf9ct1gbc/2JMZ/D6KKq50uSrI2qt29/psoDAAAAAADIGIKoaWaM0fUVRfp97Rq9rrxQkjQYs/LknJ9Y0xvYk6nyAAAAAAAAMoYgagZtzs1KXB+1ZxqWBwiiAAAAAADAAkQQNYM2550JonYM5sntHm5mHugliAIAAAAAAAsPQdQM2pDjl9MMX+/sHVBe3vDxvGDwqCKR3gxWBgAAAAAAkH4EUTMoy+nQ2myfJGl3b1A5iT5RVr29+zJXGAAAAAAAQAYQRM2wkT5RvdGY+r1rE+OBwO5MlQQAAAAAAJARBFEzLLlP1OHkhuX0iQIAAAAAAAsMQdQMS35z3o6BLHm9FZJ4cx4AAAAAAFh4CKJm2Npsv3yO4Y7lO3uDys09T5I0ONigcLgrk6UBAAAAAACkFUHUDHM7jDbk+CVJe/sGlJ17fmKOXVEAAAAAAGAhIYhKgy3xPlGhmFW3e01inD5RAAAAAABgISGISoPkPlHPxpYnrnvZEQUAAAAAABYQgqg0SH5z3vagWz7fIknsiAIAAAAAAAsLQVQaLPd7lesc/le9MxBUXt5wn6hQ6JRCodZMlgYAAAAAAJA2BFFp4DBGm+LH8w4GB+XLOS8xx64oAAAAAACwUBBEpcnI8byoldqcqxLj9IkCAAAAAAALBUFUmiQ3LN8XXZa4DvTuzkQ5AAAAAAAAaUcQlSbJDct3BB3Kyhp+e14gsEfW2kyVBQAAAAAAkDYEUWlS7XWrxO2SFG9YnrtRkhQOd2pwsDmTpQEAAAAAAKQFQVSaGGMSu6KODoTkyl6fmOulYTkAAAAAAFgACKLSaEtSn6gW5+rEdSBAnygAAAAAADD/EUSlUXKfqN3hxTLGKUkKsCMKAAAAAAAsAARRaZT85rwd/TFlZ6+SNHw0z9pYpsoCAAAAAABIC4KoNCr2uLTI55Ek7ewNKjf3fElSJNKrgYETmSwNAAAAAABgxhFEpdnIrqiWUFjyn2lYHghwPA8AAAAAAMxvBFFpltwnqsGxKnFNnygAAAAAADDfEUSl2eZcf+J6V7hKxrglSb3siAIAAAAAAPMcQVSabcrNkolf7+iLKCdnrSSpt2+frI1mrjAAAAAAAIAZRhCVZjkup1ZmeSVJOwNB5cUblkejQfX3H8lkaQAAAAAAADOKICoDRvpEdUWiCnnXJMbrj/ybegK7ZK3NVGkAAAAAAAAzhiAqA0benCdJf238a+K6o+NPqqt7rfbs+QdFo4OZKA0AAAAAAGDGEERlwJakN+dtD/0RqHQAACAASURBVFeOmW9rf1iH67+YzpIAAAAAAABmHEFUBmzI8ctlho/fHdHqlGtamu9XONyVzrIAAAAAAABmFEFUBngdDq1090mSjmuZUnWEitkh9fYeSG9hAAAAAAAAM4ggKkPWe4OSpJDxq0PFKdc4nb50lgQAAAAAADCjCKIyZGvJ0sT1AW0YM+/1lCs3d2MaKwIAAAAAAJhZaQuijDFXG2MOGmPqjTG3pZi/wxizM/7rkDGmO2kumjT3QNL4MmPMU/F7/swY40nX8zxXF5YsTlzXa82Y+RUrbpXD4UpnSQAAAAAAADMqLUGUMcYp6VuSXi5pvaQbjTHrk9dYa2+x1m621m6W9A1Jv0yaHhiZs9a+Omn8K5LusNaulNQl6V0z+iDTaFWWT36HkSQdNaODqJKSl6qy8rWZKAsAAAAAAGDGpGtH1FZJ9dbao9baIUk/lfSaCdbfKOm+iW5ojDGSrpB0f3zo/0m6dhpqTQuXw2hjbpYkqcEs03mbfiKHwytJCgaPyNpULcwBAAAAAADmrnQFUdWSGpI+N8bHxjDGLJG0TNKfkoZ9xpg6Y8zfjDEjYVOxpG5rbWQK93xv/Pt1bW1tz+U5ptXmeBAVtlKT+zyVFF8hSQoGj6mv/2AmSwMAAAAAAJh2s7FZ+Q2S7rfWRpPGllhrayXdJOnrxpgV53JDa+2d1tpaa21taWnpdNb6nGzOy0pc7wwEVVZ2deJza+vvMlESAAAAAADAjElXENUkaVHS55r4WCo36Kxjedbapvg/j0p6RNIWSR2SCowxIx29J7rnrLQlOYjqDaq4+PLE8bzW1t9xPA8AAAAAAMwr6QqinpG0Kv6WO4+Gw6YHzl5kjFkrqVDSk0ljhcYYb/y6RNILJe23wynNnyW9Pr70bZL+Z0afYpot8XlU6HJKknYGBuRyZau4+DJJw32i+vsPZ7I8AAAAAACAaZWWICrex+mDkh6SdEDSz621+4wxXzDGJL8F7wZJP7WjtwKtk1RnjNml4eDpy9ba/fG5j0v6qDGmXsM9o74/088ynYwx2hTvE3U4OKi+SFRlpS9PzHM8DwAAAAAAzCeuyZdMD2vtbyX99qyxz5z1+XMpvveEpPPHuedRDb+Rb87anJelR7p6ZSXt7h3Q1pIr5HB4FIsNqbXtd1q+/MOZLhEAAAAAAGBazMZm5QuKy5jE9Y27jugjhzrkzb9EktTff1j9/fWZKg0AAAAAAGBaEURl0I+bO/TV46cSn0PW6v7TXfpmz5kNYKc5ngcAAAAAAOYJgqgM6QxH9KnDjSnnnoxdoEj81GQbQRQAAAAAAJgnCKIy5Net3QrFbMq5AZOtXdosSerrP6j+/qPpLA0AAAAAAGBGEERlSOtQeML5p3XRmbVt7IoCAAAAAABzH0FUhiz3eyec367nS2b4eF4rx/MAAAAAAMA8QBCVIdeUFqjI7Rx3fmtRhYqLht+e19d3QMHgsXSVBgAAAAAAMCMIojLE73Tozg1L5XeYMXNlHpe+unaRyspenhhrbX0wneUBAAAAAABMO4KoDLqkMFePbF2r9y8q1cqsM0f1rinJV43Po9KSl8pwPA8AAAAAAMwTBFEZtsTv1edWVuuPz1+jHOfwb8cjXb2y1srtLlBR4cWSpN6+fQoGT2SyVAAAAAAAgOeEIGqW8DocuqI4T5J0fGBIB4ODkqSysmsSa1rbOJ4HAAAAAADmLoKoWeSqeBAlSb9vD0iSSktfKmOGm5pzPA8AAAAAAMxlBFGzyEuK8+SM9y5/sL1HkuR2F6pw5Hhe7x4NDDRkqjwAAAAAAIDnhCBqFilwu3RRfo4kaXsgqNOhsCSprPTqxBqO5wEAAAAAgLmKIGqWuaokP3H9cMfI8bwrOZ4HAAAAAADmPIKoWebKkjN9okaO53k8RSoouFCSFAjs0sBAU0ZqAwAAAAAAeC4IomaZJX6v1mX7JEl/7epVfzQqSSore3liTRvH8wAAAAAAwBxEEDULXR0/nheKWT3a2StJKiu9UiO/Xac5ngcAAAAAAOYggqhZKLlP1EPtw32iPJ4S5ec/T5IUCOzQM3Wv09Fj31BoqD0jNQIAAAAAAJwrgqhZaGOuXxUetyTp4Y4eRa1VMHhc/f2HEmsCgZ06duzreuqpqxTo3ZupUgEAAAAAAKaMIGoWchiTaFreGY7qmZ5+7dt/qyKRwJi14XC39u69WdZG010mAAAAAADAOSGImqWSj+f9uvmoAoEd464dGDipzs7H01EWAAAAAADA340gapa6pDBH2c7h356Hu4ZkJ1kfDB6b+aIAAAAAAACeA4KoWcrrcOjyolxJ0skhl1pUPeF6t6coHWUBAAAAAAD83QiiZrHk43m7XC8Zd53LlavSkvHnAQAAAAAAZgOCqFnspcV5cprh673eV8gYV8p1a1Z/Xk5nVhorAwAAAAAAOHcEUbNYodulrfnZkqRdQZeWnP8zlZS8VJI7aZVRfn5tRuoDAAAAAAA4FwRRs9zV8eN5VtJTQ4u0aeP3dMXlB7Rm9RfiK6yam+/LWH0AAAAAAABTRRA1yyX3iXqoo0eSZIxRRcVr5HTmSJKamn+uWGwoI/UBAAAAAABMFUHULLfU79WabJ8k6S+dvQpGY5IklytHlRXXSZLC4Q61tj2UsRoBAAAAAACmgiBqDhg5njcQs/pLZ29ivLr6psR1U+O9aa8LAAAAAADgXBBEzQFXFeclrkeO50lSTs5qFRRslSR19zyjvr6Daa8NAAAAAABgqgii5oDNeVkq87gkSb9vDyhqbWKupvpNievGpp+kvTYAAAAAAICpIoiaAxzG6Mri4eN5HeGItgeCibnS0ivl8ZRIkk6d+pUikb6M1AgAAAAAADAZgqg54qqSM8fzHmw/czzP4fCoqvJ6SVI02q9Tpx9Ie20AAAAAAABTQRA1R1xamCu/Y/i36/dJQZQkVVffqJHfyqbGH8smHd0DAAAAAACYLQii5gif06HLinIlSYeDIb1n7zH98nSXQrGYfL4qlZRcIUnq6z+onp5tmSwVAAAAAAAgJYKoOaJ9KKK9fWd6Q/26rUf/Z/8JveSZg2oYHDqrafm9mSgRAAAAAABgQgRRc8RHnj2pxsHwmPH6YEjv2XtchYUvlN+/WJLU2vo7DQ21p7tEAAAAAACACRFEzQFHgyH9oSMw7vzO3qDqAgOqrr5JkmRtWM3N96erPAAAAAAAgCkhiJoD9vYNTLpmT9+AqipfL4fDI0lqav6JrI3OdGkAAAAAAABTRhA1B+Q4J/9tynE65XYXqrzslZKkwcEmdXQ8OtOlAQAAAAAATBlB1BxwcUGOitzOcee9DqMrS/IkSdU0LQcAAAAAALMUQdQc4HM69E8rqsadv3VphQrdLklSXt4m5eZukCR1dDyqgYGTaakRAAAAAABgMgRRc8SNlcW6c8NSrc7yjZl7WXw3lCQZY1RT/eb4J6umpvvSVCEAAAAAAMDECKLmkFeXFejRrWtUd9F6/fPKMzuk/r+GtlHrystfKZczV5J0suFubd/xVh05+u8aHGxOa70AAAAAAADJCKLmGGOManwevb26VJVetyTp/tNdah+KJNZYG5XD6Ytfh9XV9biOH/+Wnvzblero+GtG6gYAAAAAACCImqPcDqN3VpdIkkIxqx81tyfmDtd/WUNDbWO+E4sNaM/eDyocDqStTgAAAAAAgBEEUXPYm6uK5XcYSdLdTe0aisUUifTq1KlfjfudaLRPp07/d7pKBAAAAAAASCCImsMK3S69oaJIktQ6FNH/tHZrcLBZsVhowu/199enozwAAAAAAIBRCKLmuPfUlCau72xok9OVN8HqYW53wUyWBAAAAAAAkBJB1By3KtunK4qG35C3p29AOwdzVVBw4YTfqSh/dTpKAwAAAAAAGIUgah5436KyxPWdDW1averTcjqzU64tLr5C2dkr01UaAAAAAABAAkHUPPCiwhytyfZJkh5s71Gna4VqL7hfZaUvlzGuUWv7+w9P2kMKAAAAAABgJhBEzQPGGL033ivKSrqrsU05Oat1/vnf1Isv26cXX7ZX5fHjeIODDWpo/GEGqwUAAAAAAAsVQdQ88dryQhW5nZKkn7R0KhCJSpIcDpecTr9WLL9VDodHknT8+LcUDndlrFYAAAAAALAwEUTNE36nQ2+rKpEk9Udjuq+lY/S8v1qLFr1TkhSJ9OrosW+kvUYAAAAAALCwEUTNI2+vLpHbGEnSXY3tilo7an7pkvfL7S6SJDU13atg8FjaawQAAAAAAAsXQdQ8Uu5169ryAklSw+CQHmzvGTXvcuVq+bKPSJKsjai+/itprxEAAAAAACxcBFHzzEjTckm6s6FtzHxV1RuVlbVSktTW/rC6up5KW20AAAAAAGBhI4iaZ87PzdJFBdmSpKd6+rUzEBw173C4tGrlbYnPh+u/JGtjaa0RAAAAAAAsTARR81DyrqivHGvRE1196gpHEmPFxS9WYeHFkqTe3r06dfqBtNcIAAAAAAAWHoKoeejKknyVul2SpD939uq1O+u1+fF9+tjBBgWjMRljtGrlJyUNNzY/cuSrikYHM1gxAAAAAABYCAii5qGH2wNqS9oBJUkha/Wj5g69Z+9xWWuVm7tOlZWvG54Ltaih4QeZKBUAAAAAACwgBFHzjLVWXzraMu78HzsDeqqnX5K0fPktcjj8kqTjJ76r0FB7WmoEAAAAAAALE0HUPHN8YEiHghMfs3uwvUeS5PNWaMnid0uSotF+bdt2vXbseKsOHvqc+voOznitAAAAAABgYSGImmdCU3gDXihmE9c1Ne+Qw+GVJA0MnFBn1+NqbPyRnnr6FWpsvHfG6gQAAAAAAAtP2oIoY8zVxpiDxph6Y8xtKebvMMbsjP86ZIzpjo9vNsY8aYzZZ4zZbYx5Y9J37jHGHEv63uZ0Pc9stczvVZHbOeGarfnZietTp/5LsVgoxSqrg4c+q96+Z6e5QgAAAAAAsFClJYgyxjglfUvSyyWtl3SjMWZ98hpr7S3W2s3W2s2SviHpl/GpoKS3Wms3SLpa0teNMQVJX/3YyPestTtn/GFmOa/DoffWlI47X+Fx6ZrS/MTnxqYfT3A3q6amn0xjdQAAAAAAYCFL146orZLqrbVHrbVDkn4q6TUTrL9R0n2SZK09ZK09HL9ultQqafykBfrQknK9vbok5VyxxyW3MZKkWCyigYETE96rv79+2usDAAAAAAALU7qCqGpJDUmfG+NjYxhjlkhaJulPKea2SvJIOpI0/MX4kb07jDHece75XmNMnTGmrq2t7e99hjnDYYy+vLpGT1y4Tp9aXqmPLC7T6qzhfzX7+gb1s1OdkiRjnHI6cya8l8ddNOP1AgAAAACAhWE2Niu/QdL91tpo8qAxplLSjyS9w9pER+5PSFor6fmSiiR9PNUNrbV3WmtrrbW1paULZzPV8iyvbl5SrttWVOk7G5YmfrNvP9KinnBExhhVVEy0MU2qqHj1zBcKAAAAAAAWhHQFUU2SFiV9romPpXKD4sfyRhhj8iT9RtKnrLV/Gxm31rbYYSFJd2v4CCBS2JDjTxzX6whH9K/HTkmSli29WT5fzTjfcsjnWzTOHAAAAAAAwLlJVxD1jKRVxphlxhiPhsOmB85eZIxZK6lQ0pNJYx5Jv5L0Q2vt/Wetr4z/00i6VtLeGXuCeeDjyypU7HZJku5uatf+vgF5vaWqveAXqq66UU7n8Nv0XM7c+Ddi2rvvw4pGgxmqGAAAAAAAzCdpCaKstRFJH5T0kKQDkn5urd1njPmCMSb57NcNkn5qrbVJY9dLepGktxtjdsZ/bY7P3WuM2SNpj6QSSbfP+MPMYflulz61olKSFJP0iUONstbK6y3T2rW367IX7dSLL9unSy+tU2HhxZKkYPCIDh76QgarBgAAAAAA84UZnfnMf7W1tbauri7TZWRMzFq9cvthbQ8M73L61rrFel3F2IbkoVCrnnr6FQqHhxubb1h/B/2iAAAAAABASsaYbdba2snWzcZm5ZhBDmP0pVU1MvHPnz/SrN5IdMw6r7dMG9Z/LfH52YOfVjB4PD1FAgAAAACAeYkgagHanJelt1QVS5JahyL62vFTKdcVF79ISxa/T5IUjfZr774PKRYLpa1OAAAAAAAwvxBELVC3La9UocspSbqrsU0H+wdTrlu+/Bbl5W2RJPX27lN9/b+mrUYAAAAAADC/EEQtUEVulz6xfLhxecRKn4o3Lj+bw+HWeRu+LpcrT5LU0HiP2tr+kNZaAQAAAADA/ECz8gUsaq1eXndIu/sGJEnXlxeqyufRiiyvXllaIL/zTE7Z2vqg9uz9gCTJ5cpTdfVNikQC8norVFlxnXy+qow8AwAAAAAAyLypNisniFrgtvX06xXbD48ZL3a7dNd5S3VRQU5i7NmDn1FT070p7uLU6tWf1qKat85gpQAAAAAAYLbirXmYkpZQOOV4Rziit+w+qpbQUGKsvOyV49wlqkOHPq/OzidmoEIAAAAAADBfEEQtcN9uaB13ri8a0w+bOhKfm5p/MuG9Ghrvma6yAAAAAADAPEQQtYBFrdX2QHDCNc/09CeuA4HdE66dbB4AAAAAACxsBFELmJHkNmbCNV7HmT8iTmf2hGudTv90lAUAAAAAAOYpgqgFzGGMrizJm3DN1aVn5stKr5pwbVHRi6alLgAAAAAAMD8RRC1wH11aIb8j9a4oI2l1li/xuabmLfL7F497r66uJxUOd093iQAAAAAAYJ4giFrgNuT49YvNK3V+zpljdSOxlJX0f/afUMdQRJLkdufrguf9VGVl10hyJlY7nTmSpGDwiHbuepcikTN9pQAAAAAAAEYYa22ma0ir2tpaW1dXl+kyZh1rrQ4GB9U5FNUSv1ufONSk33cEJEmXFubovo0r5EraORUOd2kwdFpeT6kkq23bb1AweEySVFh4sTZvuksOhzcTjwIAAAAAANLMGLPNWls72Tp2REGSZIzR2my/Li7MUbXPq2+uX6Ll/uEg6a9dffrysZZR693uQuXmrJXHUyyPp0RbNv9QXm+FJKmr6wnt3XeLYrFI2p8DAAAAAADMXgRRSCnP5dT3z1uqLOfwH5FvnmzV/7aO3//J56vSls0/lNtdJElqa3tIzx78lKyNpaVeAAAAAAAw+7kyXQBmr3U5ft2xdpHet++EJOnDz57Uqmyf1mT7Uq7Pzl6hzZvv1vbtb1I02qeWlvvlMB75/IsV7D8st7tA5eWvUl7e+el8DAAAAAAAMEvQIwqT+nx9k77T0CZJWuH36ne1q5Xnco67vqvrae3c9XbFYqGU89VVN2rNmi/IGDbkAQAAAAAwH9AjCtPmU8ur9MKC4TfjHRkI6eb9J/SXzoB+0tKhRzoDisRGh5mFhVu1Zs3t496vqfk+NTb+cEZrBgAAAAAAsw9H8zApl8PoexuW6qq6g2oKhfVQR0APxd+oJ0nVXrf+Y91iXVKYmxgbHGiY8J4nG+5RTc3bZIyZcB0AAAAAAJg/2BGFKSnxuHT7quqUc02hsN68+6gO9g8mxnr79k14v8HBBkUivdNaIwAAAAAAmN0IojBlj3f3jTs3GLP6zsnWxGenM3uSuznkcHinqTIAAAAAADAXEERhyh7tnHgH01+6zsyXl718wrVud76i0f5pqQsAAAAAAMwNBFGYESUlL1FR4QvHnQ+Hu1S37XXq7z+SxqoAAAAAAEAmEURhyl6U1Iw8lYvycxLXxji1ceOdWrzoXXI6z4zn59fK5xvuNTUwcFJ1216vzs7HZ6ZgAAAAAAAwqxhrbaZrSKva2lpbV1eX6TLmpBMDIb3kmYPqi8ZSzm/JzdIvt6yU3zk634xGBxUKnZLLlSuPp1jhcLf27PmAurr/JkkyxqU1qz+v6uobFI2GNDBwXA6HV37/Et6qBwAAAADAHGCM2WatrZ10HUEUzkVdT78+sP+ETgwOJcackqLx65cW5+kH5y2VxzHxZrtYLKyDBz+j5pafJ8by8raov/+ootEeSVJ29mqtXPExlZRcMd2PAQAAAAAAphFB1DgIop67qLV6oqtPDaEhVXrcqvZ69Lpd9WobikiSXlVaoO+sXyKXY+LdTNZanWz4vurrvyxpvD+HRhvP/65KS186vQ8BAAAAAACmzVSDKHpE4Zw5jdGlRbm6qbJYlxfnaXWOTz/ftEIFLqck6ddt3frowZOKTRJyGmO0ZPG7tWrVP02wyqr+yJdlberjgAAAAAAAYO4giMK0WJfj132bVign3h/q56e69MnDTZrKjrtoJDDhfDB4TH39h6alTgAAAAAAkDmuTBeA+WNLXpZ+vHG5btx1RAMxq3ua2hWzVsVupw4FQypwOXVdeaFeWJAzqgl5NBqc9N6xKawBAAAAAACzG0EUptULCnJ0z/nL9ZbdRzVkrX7Y3DFq/t6WTl1bVqBvrjvTQyo3b+Mkd3XI662YoYoBAAAAAEC6cDQP0+6yolx9aVX1uPP/3dqt7za0Jj6XlrxUft/iCe4Y0/Ydb1Ff38FprBIAAAAAAKQbQRRmREMoPOH83U3tif5RDodbmzbdJZ9vbHjlcHglSQMDx/VM3WvV0vKr6S8WAAAAAACkBUfzMCP29w1MON8UCqsnElWBe/iPYHb2Cl30gofV2vqgenp2yOH0qrT0Svl9S7R//z+qs+sxxWKD2n/gVvUEtmvVyk+pp2ebOjsflyQVFb1QhYUXyRiyVQAAAAAAZiszlbeazSe1tbW2rq4u02XMex/cf0L3n+4ad94l6chlG+V1TB4cWRvV0WP/qePHv5kYczqzxjQ5z89/njZtvFNud+HfXTcAAAAAADh3xpht1traydaxfQQz4tryicMgl8PoQN/glO5ljFMrlt+iTRvvksuVLyn1m/Z6erZr3/5/PPdiAQAAAABAWhBEYUZcUZSrq0vyxp0fjFm9Zsdh/aylc8r3LCm5XOed940J13R0PKq+/sNTvicAAAAAAEgfgijMCIcxunPDUt26tEKlnuE+UC4jXVWcpxcV5kiSQjGrDz97Up861KhwbGpHRIdCpyZdE+jZ9fcXDgAAAAAAZgzNyjFjPA6Hbl1WoVuWlqszHFG206ksp0Mxa/X1E6f1b8dOyUr6flO79vUN6J9XVevXrd3aFgjK6zB6eWm+Xl9eJL/zTF7qcPgm/8HGzNxDAQAAAACAvxvNypExv2/v0Qf2n1BvNCZJMpLO/tO4LtunX2xeqZL4rqpwOKDHHr9Isdj4/aX8/sVat/YrKizcOkOVAwAAAACAZDQrx6x3ZUm+HqxdrZV+r6SxIZQkHegf1CcPNyY+u915Wrb0gxPed2DgpLbvuFHPPvtpRSK9kqRIpFetrQ/p1KkHFAwen65HAAAAAAAA54CjecioFVk+vbOmRJ883DTumt+0dqttVVilHrckacmS98vpzNLxE9/V0FCrJMnjKVV19U3q7d2n9vY/SJKamu9Te8efVFT4QrW2PaRotD9xz9KSl2ndun+V2z1+Q3UAAAAAADC9CKKQcY2D4Qnno5KODwwlgihjjBYtepuqq29Sf/8hSVJ29io5HB5Za9Xa9jsdPPg5hcMdCoVOq+XUL8fcs639YYX3vF/P23KvDD2lAAAAAABIC47mIeOKPZPnoX7H2D+qDodbubkblJu7QQ6HR9JwSFVedo0uesHvVVFx3YT37O5+St3dT/19RQMAAAAAgHNGEIWMu66sQM5J1rxj7zH9tbN3yvd0uwu0aNHbJ13X0fnYlO8JAAAAAACeG4IoZFyVz6PblldOuKZhcEhv2HVEHzvYoN5IVJLUPDikn5/q1M9PdappcGjsl6bwRsi+vgOKxVJ8FwAAAAAATDtjp/A/6/NJbW2traury3QZSOGB1m5952SrdvQG5XUYXV2Sr7dVFevupg79uq07sa7K69babJ8e7exVND7mkHRDZZH+ZXWNvPFjfLFYSI89fonC4c4Jf67XW6klS96nqso3KBrt14mTd+rUqQcUiXQrO3uVaqrfosrK19NLCgAAAACAcRhjtllrayddRxCF2SZqrRzSqODnf1u7dduhRrWHIxN+902VRfra2sWJzydO3qX6+n+Z0s91u4tlbVSRSPeYueqqG7VmzT8TRgEAAAAAkMJUgyiO5mHWcRozJvB5ZVmB/nLhWl1Tkj/hd+9rGX1Mb/Gid2nZ0g/JYTyj1hUVXaoLLrg/vtNpuFl6ONyRMoSSpKbm+9Td/czf8zgAAAAAACBu8teVAbNEkdulV5Tm67ftPeOuiUl6vLtP11cUSRreVbV8+YdVU/MWdXT+RbHooPLyNys3Z60kqSB/i5Yt/ZBOnPiumpp/MuHPbzn1SxUWbp225wEAAAAAYKEhiMK80zYUHjPm8RSpsuLalOv9/mqtXPl/Jw2iuruf0cBAo/z+GkmStVGdOvWAmlt+oVCoRT5ftaoqr1d5+StlDJsNAQAAAAA4G0EU5pSLC3PklBJNylP50tEWNQ6G9dGl5Sr1uCVJPeGIHu3q02AspuflZWlllm/Ud5zObLndxQqHO8a978DAcT3x5GUqKNiqiopr1d72J7V3/CFp/qS6up5UR8cjWr/+q4RRAAAAAACchSAKc0ql16M3VRXrh83jB0ZRK93d1K6fnerUP9SUSkb69sk2DcRiiTUvK87TN9YtVoF7+K+AMQ5VV71Rx098e9IaurufVnf30+POnzr9PyoufrEqKl59Dk8GAAAAAMD8x5YNzDm3r6rWW6uK5Uwac2r4jXm3r6xSqWc4XApGY/raidP62vHTo0IoSXq4I6C37zmm5LdGLl36ARUWvCDlz1yx/GNau/ZLKsh//pRqbGr+6Tk9EwAAAAAAC4FJ/h/xhaC2ttbW1dVlugxMg5bQkJ7o6pM0fGSv0jv8Zrz+SFTfa2zTt062qj8am+gWun/zCl1SmJv4HIuFdfr0Azp1+tcKh7uUk71a1dVvUn7+5sSaovDb3wAAIABJREFUYPCEnn7mVYpG+8e9r9tdqksveWLM8byhoU4NDJyUx1Mkv3/xOT8zAAAAAACzkTFmm7W2dtJ1BFGYrx7tDOiNu45OuOZDi8v0yRVV53zvZ565ToHe3ROuycpapurqN6my4nWSojp46Atqbf2trI1IkvLytmjN6s8oL2/jOf98AAAAAABmk6kGURzNw7xV6J68Bdpfunp1YiA0aqw7HNFPWjr07ZOt+mNHQNEUYW1l1RsmvXcweEyHD9+uxx6/WE8++TKdPv1AIoSSpEBgh7Ztv0l9fQen8DQAAAAAAMx9NCvHvLU226cSt0vt4ci4a3b2Duiivx3Qy0vz9d6aUu3pDer2oy0ajJ0Jn5b5PfrBecu0LsefGKuqfIPa2/+ojo5HxtwzP79WXk+p2toflrURxWIDisUGUv78WGxAx45/U+ef941R4339h9XY+CMFArvldGaprPRKVVa+QS5X9jn+WwAAAAAAYPbgaB7mte81tOqz9c0p51xGikzxj3+p26W/Xrg28ZY9abifVHPzz9Tc8gsNDrbI76tWVdX1qqx8gxwOl0KhVjU1/0wnTnxHsVhogrs7deHWXys7e5WMcai19UHt3ffhUbunJCk7e5Wet+VeeTzFUysaAAAAAIA0oUfUOAiiFhZrre44cVr/ceK0Qkm7nK4oytW/rVmkP3UGdGdDmw4HJwqKhn1uRZXev7jsnGvYvv3N6up+ctJ1TmeOcnPXq7t7m6RoyjXl5a/WeRvuOOcaAAAAAACYSQRR4yCIWpi6whE90tmrgVhMF+Rla022LzFnrdXv2nv0zr3HJ7zHS4pyde+mFaPGotbqia4+NYWGVOPz6OKCHDmMGbXmyNF/1/Hj35qW5zDGpUsveUpud8G03A8AAAAAgOkw1SCKHlFYEArdLl1XXphyzhijy4vyJr3Ho129+siBk3p9RaEuLsjR0z39uvnASTUMDiXWLPV79K11S3RB/pleTtVVN+rkyR+M2yeqqvIGyVgFArvijcvHD4etjSgQ2K3i4hclxmKxIbW2PaSuridl5FBR8aUqKX6JHA7+egMAAAAAZhd2RAFxr99Rr8e6+6a0ttzjUsdQRKnaoOc4HfrD89doqd+bGOvsfFx79t6sSKQnaaXR8mUf1rJlNydGjhz5dx0/MdnuKaOiwheqsvJ1ys09T7v3/IOCwfpRK3Jzz9PmTT9I2U/KWqtQqEWx2JB8vhoCKwAAAADAc8bRvHEQRGE8T3b36fU76lN2Zyp0OeVzONQyFJ7Svd5ZXaIvra4ZNRaJ9Ol0628U7D8it7tQ5eWvkt8/es3AQKOeePJySbEpVu0Yd21x8WXavOkHo8ba2/+sI0e/pr6+A5Ikj6dMixe/S4sXvVPGOKb4MwEAAAAAGG2qQRT/5wnEXVSQo7vPX6Yan3vU+IsLc/XnrWtVd/F63b95ha6vKJQZ5x4j/tgRGDP2dK/0ha4L9b6u1+pj3S/Tf3f7FY6NDoL9/hqtWHFrynu6XPlauvRm5eVtShodP7Dq6HhUvb2HEp9b2x7Srt3vSYRQkjQ01Kr6+n/R4fp/meSJAAAAAAB47tgRBZwlaq2e6elXVziiNdl+Lc/yjlmz5fG9ahlKdTDvjA05Pl1TUqBrSvP1UFuPvnz81Jg1lxbm6EfnL5fPOToTbmt7WCcb7lYgsEtOZ5bKSq/SkiXvT+yg6u+v16FD/6zOrscmeRojn69KPt8i9fbuUTTaP+66iy/6s/z+RaNGA4HdOn36fxWOBJSbs1YVFdfJ7c6f5GcCAAAAABaaWXc0zxhztaT/kOSUdJe19stnzd8h6fL4xyxJZdbagvjc2yR9Oj53u7X2/8XHL5B0jyS/pN9K+rCd5IEIojAdbnn2pO5r6ZyWe926tEK3Lqs45++1tPxK+w+k3j3191i8+N1aueLjMsYha6M68Oyn1NLyi1FrnM4cbTz/Oyoqunjafi4AAAAAYO6bVUGUMcYp6ZCkl0lqlPSMpButtfvHWX+zpC3W2ncaY4ok1Umq1fDrxLZJusBa22WMeVrShyQ9peEg6j+ttb+bqBaCKEyHQ/2DurLuoAZjY//+eIxRbX626nr6NTSFv1+VXre2X7Rexpw58DcUi+m3bT3a0RtUlsOhV5YVaEOOf9T3wuFuPfb4xYrFQinva4xHBQVbNTjYqMHBBlmbqvvVaC5XgfLzt8jaqDo7/5JyjdOZrYsv+pM8npLEmLUxtbU/rJaWXyoUOi2/f5Gqq25QYeHFo54LAAAAADA/TTWIStfrsrZKqrfWHpUkY8xPJb1GUsogStKNkj4bv75K0sPW2s74dx+WdLUx5hFJedbav8XHfyjpWkkTBlHAdFid7dOPNy7Xhw+cVFPoTAPzRT6PvrlusS4syFFvJKo/dgT06cNNag+Pf4yvJRTWnY1turokX0v8Xh3sH9Sbdx9Vw+BQYs0dJ07rDRWFumPNYrkcw8GO212g5cs+rPoj/5ryvmvWfE7VVW+UJIVCrXrs8UuklK3Yz4hEutXR8ecJ10Sj/Wpu+S8tXfI+SZK1Ue3bf6tOn34gsaa3d49aW38b32V1G2EUAAAAAEBS+oKoakkNSZ8bJV2YaqExZomkZZL+NMF3q+O/GlOMp7rneyW9V5IWL1587tUDKVxSmKunXrBef+nqVUsorGqfW5cW5soZD11yXU5dW16ohzsC+q/TXRPe67P1zfpsfbNW+L06PRRWX3RsE/JfnOpSjdejjy+vTIwtWfI+ud2FOnb82xocHP5rkpW1QsuXfUjl5a9MrPN6y1RZeZ1aWu5P+fPd7mKVFF+unsBOBYP1kz770aNfV0fHo8rOXqlIuEenW/835bqTJ+9SUeHFKi6+bMxcf3+9gsFj8nhKlJe3ibf2AQAAAMACkK4g6lzcIOl+O5VzRFNkrb1T0p3S8NG86bov4HIYXVGcN+Ga6yuKJg2iRhwZSH3MbsTdTe360JJy+ZOam9e5rtR/uM5TqxpkZbTIuUT/6KpU+VnfXb3qMxocbFJX15Ojxn3eKm3efI+ys1dIkgYGGvXEk2ODo2TWDqm7+yl1dz816TM1Nd03KogaGGjQ/gP/V93dTyfGsrKWae2aL6qwMGU+rXC4W/399XI6s5WTs5YdVgAAAAAwR6UriGqSlPw6rpr4WCo3SPrAWd998VnffSQ+XjPFewIZ86LCHL2pskj3pmhuvibbp++tX6K6QFB/6OjRHzoCikwQlXZHovr04UZdVZKv5+Vl69dt3frEofjGQDMcPbX3Degtu4/q2+uX6LrywsR3Xa5sbdn8Q3V2PaH2tj8oFgupoKBWZWWvkNPpS6zz+2tUXHyZOjoeHbcOn69aoVCrrA2Pu2ZEe8eftWfPB5Wfv0XZ2at04MAnFRpqGbUmGDymnbveodoL7ldu7vrEeDQa1KHDt+tUy68Us8NHFbOylmnVyk+qpOSKSX82AAAAAGB2SVezcpeGm5W/RMNh0TOSbrLW7jtr3VpJD0paNvL2u3iz8m2Snhdftl3Dzco7UzQr/4a19rcT1UKzcmRCzFr9/FSn7mnq0OHgoIrcLr2+vFDvX1SqfPeZPPhz9U36bkPblO9rNNzBP5USt0vbL14vj+PM7qn2oYjubmrT79sDCsWstuZn692LSrQ2e3Qj9N7efarb9kbFYgNj7ltcfJk2bbxL1sY0ONigbdtv0tBQ65RrnkhZ2TU6/7xvSBpugL5z5zvU2fVYipUObd70AxUXXzpq1FqrnsB2dXc9LWOcKi5+sXJyVk9LbQAAAACA8c2qt+ZJkjHmGklfl+SU9ANr7ReNMV+QVGetfSC+5nOSfNba28767jslfTL+8YvW2rvj47WS7pHk13CT8pvtJA9EEIXZbFtPv16x/fC03e8rq2v0pspiuRxGxwdCum57vVqGRu9ichuj75+3VFeW5I8aDwT2qL7+y+rq/pskyeXKVVXVG7V82UfldHoT644f/7aOHP3auDW4XQUKR7qnXHNOzrr4G/msOjtThVDDcnPP09bn/0/i89BQp/bs/eCY44Ll5a/SurVfGVUzAAAAAGB6zbogarYgiMJsZq3Vu/cd12/aesbMOSR9b/0SFXlc2h4I6penu3Sgf3DSe/odDp2X41fj4NCYEGpEntOh7RdvUI7LmRjrCkd0Z0Ob/nD6mMKRXlXkLNI7FlXrqrMCq2g0qO073qxAYNeY+xYXX66N539X4XCHenp26OChL2ho6PSkNU/V4sXvVX7eJvmzlunQwc+pu+fplOuqq2/S2jX/PGosHO5RU/NP1dH+Z1kbUUHBVtXUvFk+X9W01QcAAAAACwVB1DgIojDbhWIxfeloi37c3KH++NvzVmV59U8rqkbtWvpbd5+u3TH5G+6m6vZVVXp3TZkkqW0orNdsr9fRFM3Tb1lSPurNfZIUifTr5Mm7dKL5vxQNt8rtrdGymhtVU/MWORyexLojR76q4ye+M24NTme2nM4sDQ11SBr75sC/n1OXXvJEfKeVNDBwUtu3v0mDoeYxP3/zph+ooGDsfzvD4YA6Oh5RJNqn3NwNysvdSNN0AAAAAIgjiBoHQRTmiv5IVIeDIWU5HVqV5R0Telhr9dK6g9rXl3pX1Eq/Vy8szNHu3gHt7RtQeAp/18s8Lq3N9un0UFgH+8d/g98fn79GG3LO9JXaEQjqH589qf1JO7S25mfrjrWLtCLrTCP0wcEW/e3/Z++8w+Oozr59z2xfrbTqXbKaLclVsi3b2JQ49N5LCKGGkkAooQQSEggkJHQCARISQg8l9BZwaAaMe5ObJEuyeu/avjtzvj9WWu16d+W8+ZK8ecPc16VrtWd/e/bMmdkyv3me56w/BkVxxOhVorrqGVJTVyCEQnv70+xtvOuAY/57sVpLsCdVYbWW0N3zFi5X7BRIozGTFctXRxhobW1P0tT8AKo6tX1JSdXMm/twzAgqj6eLnp638PkGsFiLyM46CYPBHqXT0NDQ0NDQ0NDQ0ND4b0EzouKgGVEa/000u7yctb2RDk9kyt1Mq4lXqkrJMQXNlD3jblZuqv+nve4xaUncVpZHgdlIq8fL0ZsacCjREUxZRj0f1ZSTYTSE2kZGNrOx9irkwFSBc1WyMGvW7czIOyPUpihu1q47HK83dipf0YwrSUk9CJezmd6+dxkZiZ2W949QUnI9uTlnYDSm093zOnv2/CimzmotZemSd5DlqfpTra1P0Nh0L+ERXTqdlTmzHyQj44ioPoQQjI1txe1ux2jMICVlKZKki9JpaGhoaGhoaGhoaGj8J6MZUXHQjCiN/zacisJrPcN8OeJABlamJnFyZjJmnRyhO2FzA5vGXDH70EsSR6Ql0uTy0uzyovydr62XgjWoxmOYUJPcUJTNDcXZQNB0ubG+gxe7eqliM9l0M4adTSxlrj2Dl6pKSNBNmTAORz3btl+G19sR0Wd+3vnMmvVTJCm4jS5XK2vXHU78NQRlZNkUcxXA6ZAkAyAQIhBXU1F+F7m5ZyFJEv39f6N2xxVx+1q29K9YrcWhtnFHHbt3/RCHc8okNJvzqaz4Fampy6P6GHfU0dnxPOOOOgz6RLKyTiAr68SI6K39mfyM19IINTQ0NDQ0NDQ0NDT+lWhGVBw0I0rj68puh5vTtjYyEoi0mWTgN5WFnJmdCsBoIMCCNbvwqP+czwarTubYdDsFZiPOgMIfOgfiam8syub6CdMKYMuokwtq91LkX0sxzbixsIFlHJw9h99UFKKXp8yVPXU/pqvr5Zj9lpXdTGHBd/F6exgY+Jj6htv+Kds2iSTp0evtKIozIn1vf3JyzqKy4pdIkozX28f6Dcfh9w9H6WTZxOJFr5GYWBlq6+p+lT17bmH/2ll2+yKqFjyFXp8Q0T4w8AmtrU8wMroZWTaQnn44RUVXkmiriDk2IVRcrn0IoWC1liDL+v/BDGhoaGhoaGhoaGhofN3RjKg4aEaUxteZNreXx9v7+XBgFK8qWGpP4IqCDJYk2yJ0Nzd08PQ0htGVBZmMKwr73F6+HHbEjUP6n5Ksl3moYgZ5ZgN2vY6jN9UzHIgdbRUeaQXg8Ht5fvPtFLrexIQv2IaN/pQLuGjBtejkYPSUEIJ164/B5Ypd6F2SjBQWXITP14/TtY+xsa3/pK2b7F+PyZiJQMXr7YmrS08/gtmVd6PTJeD19rJ23eFxI7MKCi5m1syfhO63dzxHQ8PtUTpZtlBd/QzJ9kUR7d3dr9G87xE8nnYATMYsZsy4jPz8C2JGUqmqj5GRTSiqm0TbbMzmnCiNhoaGhoaGhoaGhsbXC82IioNmRGloHJgRf4AztjWx0xGdynZ7aS5XFGaG7l+0Yx9/HRiN25degsC/4GMmSS+zdmklaRP1p767cx/v9o9iFU6KaURBTxNl+CUT187I4uawlf4GhzayefsF6ERkQXaBREn5rymZqFV1INMKwG5fjE424w8MMz6+m/jpgf8/6GCahElZNlJaehNGQxqSJLNr1/UIYptWtoRylix5L2QwdXS8QH3Dz2Jqi4uupqTkmoi2ru5XaWy8B79/cPLVyco8joqKX6DXJ0ZohRAMDX0eLNzuH8JqLSEv71vYEmbG3RZFcePzDWE0pqDTWePqNDQ0NDQ0NDQ0NDT+s9CMqDhoRpSGxt+HU1F4vmuQN3tHGAsoVNrMXJyXwfKUyOipjaNOTt6yl1hxS2ZZ4uPF5dgNeto9Pn66tyNunap/FJtOJs2gp9Xji6uxSBLbD55Lkj5Yf+rnjZ2827aJk3iNaragQ6GOSt7hVLLTlvPnBaWh5/b2vsvOXZFmjAAkwJRQyfKaN5DloBm2e8+P6O5+Ne44kpKq0OtseLw9uFzN7J9m9+8iLW0lFkshep2NtvYn46YTSpKBg1eswWhMA6Cn9x127bo2pjY5eSkLq18IGVxCKOzafT29ve/s3yvl5XeQn3duRKvPN0RT07309L6NqnqQJSOZWcdTVvojTKaMqNdTVS99/atwOvdi0NvJzDxOi8zS0NDQ0NDQ0NDQ+F9EM6LioBlRGhr/fN7sHeaG+vaIlfPSDHoenz2DQ1OnomRWD41z9vamuP2ckZXCYnsCnR4f7/eP0uT2xtX+TzFJEpkmA8l6Hbsc7mktoL8umkl1UrDm0is9Q7yx+0nO4XlsOEKaWhawynojby1ZinEi7c/pauGL9adgEOMhs2ry1mcs5shlb4VqOe1t/DVtbX+IOwaLuRC7fSEBxcH4+M5p0/j+lZhMuSRYizEYUhgYXI2ijMfVzp/3e9LTv4kkybS1P8Xevb+Io5RYuuQ9bLZyAAIBB5s2n4HTuTdKabHMoGbx6xgMyaG20bHt7Kj9Hl5f+IqKOkqKf0BR0VVR6YRDQ1/R1v4ko6Nb0clm0jOOZEbhZVgseVGvJ4RgZGQDQ0NfApCauoLk5KXTFnv3+gZQFS8mU5ZWW0tDQ0NDQ0NDQ+Nri2ZExUEzojQ0/jU4Agrv9Y/S6/NTaDZyTLo9auU+IQQ/bezkjx3R9adWJNt4YX5J6DnrRhycsjV+Sly2Uc+SZBvdHj8NLg+jgb93rb8DIwHpRj0ZBj373F7cqkAWAbLpwoqbIdIYktIB+NXMPM7KScUqy7zYM8QDe9ZwPk8yh50ABNCxkWU8yyW8sriGqqRgupnT3cnn647FKJxRr6+gp6r6ZTJTqgAYG6tl46ZTQ6bW/qSkrKCw4CL8gVGGhr6kp+eNf9pc/CPodAkoiofp0gmTkqrIzj4Fvc7G0PCaacdcXPQDSkqCkVg+3xBr1x1JIDASUzu78l5yck4L3e/sfJG6+lujdAZDMgsXvhSRJuj3j1K743uMjKyP0CYnL2H+vMcjzDCA4eF1NDXdx+hEHTGTMYuCwosoLLgktKLjJEKoDA6upqf3bfz+YWwJs8jL+1bEKorhKIqLvr4P8Xq7MZvzycg4Ep3OEm+KUBQ3Xm8Per0dozE1rk5DQ0NDQ0NDQ0PjX4VmRMVBM6I0NP53EULwwcAoz3YN0uzykm7Uc1Z2KufkpGKS5Qjdd3e18F5/dP0pgyTxalUpSyeKrO9xuFm5sT7ua8rAoiQrIwGFbq8/InLrn4VBklARKBMfqQbhw4gXH0b8kgmAGruVy/IzSdbr2DLm5IXmr7iC31JAG/JEbalR7PyB73FMyUlcU5QFgE9VeWDNDdT434p63SFSSal8jiNygqvhKYqX1WsORgSGYo7TlnwIC+c9iN8/gsNRz46dV8Y0uCbbDPpkAoojbqH0fweSpCfBWopOn4DfN4zLvS+u1mTKZc7s+9HrbajCz6ZNZ0GcelnJyUtZtPDPofvbay9jYODjmNq0tJVULfhj6P7g0Jds335JzHnJz/sO5eW3h+6rqp+du66mv3/Vfkodsyt/HWGcAfT1f8iePT8iEJiKPtPrk5kz+z7S01dGaAMBB41N99LT/TqKGkx7TUs9lLKZP45Zi8vr7aOz62XGx2qRdRYyM44mI+PouJFcLlcrwyPrkJBJTV2B2ZwbU6ehoaGhoaGhoaGhGVFx0IwoDY3/O/hUlXv29fBs5wBjE+bRoiQrPy3NZdl+K/1NVzT9qsJMbi0NnkA7Awrz1+zEqcb/7FtiT8ClqPR4/Qz4/w0GjBDo8SOj4sMEkoRVljk2w45NJ9Pj9fPhwChlNFDFFgpoRUJlLxWsZiUFtlzeWTQTiyzhE4Jz1vyFS/x3YiGy2Hw3OXyZ8gC/q14Sant7/cUkOFfHNKPGTfM4eXkwUsnnG2LdhmMI+GMbXAB2+0L0+iQCASejo5v536qB9T8hKakag8GOUBWGhr+YVltWdgtWywwk2UR9/c9CqwzGYtnSVSQkBGuNtbQ8RlPz/XGUOg5a9mEoMmpsfCebNp0e0+CSJANLat4KpTSqqp8tW789MdeR6PV2aha/FhFxNTT0FbU7LkdRIuu02e2LqVrwJHr91HtKUdzsqbtlvxpfMrm5Z1I+63Zk2RjRh9PZSEvL4wwMfoyq+rDbFzGj8FLS0g6NGpsQCn39H9Lb+x6BwCg2WyX5eedOEx3mZXDwU7zeXiyWQlJTD5k2BTIQcOL19mI0pmAwpMTVaWhoaGhoaGho/HPRjKg4aEaUhsb/PTyKSqvHR6JOJtdsjKlxBhSuq2/n7b6plC29BBfnZXBbWS66sBo/D7T0cM++2DWXTs5M5vdzioBgVNYh6/fQ6I5fCP2QFBvJej1jAYU1I+P/khUC/150EpgkGZeqohMB7AxjZxQDfsZJpI9sFEnPFQUZ5JgM6CWJexqbOUZ9hVIaMePBhAcTXgZI53fS9XyyfAWZpuCcP7rpfirGHov52t3kc/yKD0gxBdPHvtpxC+7+V6J0k4ZXQdF1pCbNIaA4aG39Aw7HrrjbJeusGPR2FMURESX0n4wkGTEY7MiyGa+3e9qIMputguTkGiTJwODg59Ou0picfBCFhRchyyZGhjfS0vrbuNrsrFOYMydogAUC46z56lACgbGY2tzcc6is+GXo/s6d19Db925M7f4RX2NjtWzZeh6KEp1mWl5+Z0RhekXxUFt7OUPDX0boJEnP7Mp7yc4+KaK9v/8j9tTdjN8/HGozmXKYO+chkpMjf+P4/WM0Nt1NT8+bEwX4JdLTv8nMsltimlwjI5to73iG8fFd6HU2MjOPJT//vKjVHyf77u55jZGRDUiSnrS0w8jKPBGdzhRzjpzORgYGP0MIhZTkGpKSquPWGVNVP6OjW1AUJzZbxQGjzvz+Efz+YYzGzFDNuekQQkxb40xDQ0NDQ0ND45+FZkTFQTOiNDT+u2l2eVk36sAgSRyWkkimyRClEUJwX0sPj7X141aDUTs64KycVO6amY8lrLbVm73DXLG7NeZrzbGZ+WBROQY5eJL3s72dPNHRH3dsl+SlMyfRwqg/uCLhP7MY+78KGUjQyVh1Mn2+AHYxRCa9EaaVDyO1VHFQWhbLk21YdDKvd7Uxz/En0hjAhBcTXox4MeGjjnLySn/JhfmZmGSJ91o+xbLv0rhjGCl4iNNnnghAU8drtDTcFFerGIuYlX8miuJgZHRLVL2nSSa/+XSyCVX9z98P/yh6fTI6nRlV9eP3D06jlMnP+zY6vQ0l4KCj87lptDoWLHgSszkbWTJTu+MKnM662L1KRlasWBOqW9XYdC+trb+LqZUkPQct+yRURH5srJZNm8+MaeLpdAksXfI+Fks+EIya2rzlbMbHd0RpDYZUaha/GVGcvqPzz9TX/zRKa7WWsGjhixiN6aE2h6OerdsuwOeLfG8nJMykuuq5iFUdVdXLnj0/pqf3zQhtcvIS5s19NKp+V0/vOzTu/VVY4X2JjIyjqaz4ZVRNMperlb2NdzEw8AmgIstmsrNPpqz0ZgyGpAitqnppa3+ars6XcHvaMBozyck5naIZl8c02kZHt9De8SwORx16fSJZmSeQm3tWzLpkPt8Q3d1/YWR0M5JkICP9cLKyjkeW45tyg4OfI1BJSVlGUuLcmDoIGpXDI+tQFBdJiXOxWArjaoN6N4HAOAZD6gEXCggEHPh8AxiNaTHnYH+EUKNqvWloaGhoaGhMj2ZExUEzojQ0NCYZ9QdYO+LELwQ19gSyY5hWAM91DfCLpu6IguiHpth4dPYMMoxTz+n3+Tl+817aPNERVMuTbby8oDRkWn08OMa3a5vjju2XM/M4PSuFcUXlxa5BHmjtjatN1MkclpqIU1Fpcnljvv5/KtLEn054SWJswrDyYcCHER8DZOA15HNEWhIWncxepwvDyHtk0IcBP0Z86PFjIICMwjrTt7i+fCkmWabHPUJP/fex4cRAAD0BDPgw4MeAj2a5iisPC9aIcnt6+XztNzDgjznOAHpmz/oZsqTD7x9hb/MDyNMUY7dYitHrrSiKB5cr/kqRXwcMhjSMxlQkSY/T2YAQ8ectMXEuyfbFyLKR/oFPpo0Oy0g/krz885AlA4NDn8c1uABycs6isuIuJEnC7e7gq7UH8P0DAAAgAElEQVTfJF4x/ezsU5gzOxhJJoTC2nVH4nbHNqPTUg+lquqp0P36+tvjmnjJyUtZWP1CKDqpv/8jandcHlOblFTN4kUvI0k6ADyeLjZuOhWfL3qhh8TEuSxa+DI6nRkAVQ1QW3spg0OfR2ltttksWvhiRBpme8ezNDT8PLpf2xyqq5+PMLnGxmrZtv0i/P7IxQJstkqqq57BaEwLtSmKlz17boqKrEtLO4y5c34TZQZ1dP6Zpqb7CASmUqwz0o+ksvJuDAZ7hNbtbqex6V76+z9EiAB6fTJ5uWdTXHx1aB4m8fkGaWz8Nb2976IKH5KkJzPzWGaW3YLJlBWhFUKls/NFOjqfw+nci16fRFbWSRQXfT9KOzkfbe1PBVfk1JnJSD+C/IILMYUZmZP4/SN0db0SsSJnbu7ZUYbj5DgGhz6nv38VquIlyV5FTvYpcQ00v3+U/v6/4Q8Mk5Awk7TUQ0LHzv5Mprp6vN2Yzbmkp62MSrWdGodg3LELl7MJgyGVlJRlyHLs70kI1qAbd+xGJ1ux26vi9gugqj4czgYkZBISZk1rJAoh8Hq7UVUvZnPBAU1HVfWhql50OpsWDaihoaHxb0YzouKgGVEaGhr/CC5F5YvhccYDCnMTLVQkxF7BrNfr5+593bzeO4xHFaQadJyXk8a1RdlYdZHF2G9r7IoZQXV8hp3fzy5CP2FaORWFQ9fX0emNbZI8XFnIWdnBSIt+n5+FX+3GH+ezvchi5LHKGbhUlRaXlxsaOqbd7kVJVvSSxGhAoc7pmVb7fxGdBGZZRgb8AQd2RvczrPzo8dNPFsUps8g1GdFJEjt7PmeGaJrQBnU6FHQE8GImIe9Kqux2DJLEJ/VPkB/Yih4F3YRuykDz4824ksMLlmAgwJf1j2B3/g09AWREVN0ut/UgqgqOQ1W97Ov+gIAj/veZIllISaxAVd14PN0RJ/hfTyRk2YAQIMR0Zq2EPakaSTYQCIzjcOyOqZpMM83KOhmjMRWhBujofIFYtdEmtUUzrgymCUo6mpvvxePpiqstK7uZlORlSJKeltbH6Ot7P+6IZ5b9hNzcs5AkPb29b7On7pa4q2wWF19DSfHVALhc+1i77qiYYwbIyzuPivKgSaWqPr5auxKvN3Zac0bGUcyf93jo/p49t9DV/UrEOCb/T0/7JgsW/CGk7e5+g917bog5D3b7IhYtfCkUnRQ05U6LilADSElZTtWCp0JGRSAwzsZNp8c0g83mAmoWvx6KUhNCsKfuZrq7X43Smkw5LF78KmZTdqitp+dtdu2+nv3nzmTKYtHClyKiuZzOJrZsPQ+fry9CazRmUF39XMTCAoriYnvt5QwPfxWhNRjSqKr6U1REWXvHczQ2/noiHTWIxTKD+fMeD9WTm2Rg8DN2774Rf1itP6MxnTmzHyA1dUWE1uVqZdfuHzI2ti1Mm0lF+R1kZBwZoQ0EHNQ33EZPzztMGrxGYwZlpTeSk3N6hFYIQVv7H2ltfSI0DpMpm6KiK8nL/VaUcTQw8ClNzffjcOwJjaGw8BIKCy6OilhzuVpobn6Qvv4PEcKP2ZxHfv75FBZcFGXMBQIO2tr+SFf3q/h8/ZjN+eTlnUNB/gVRBpoQKt09r9PZ+SIu1z6MxnRysk+loOACdDor+zMyupmO9mcZG9+JXp9AZsZx5Od/O6aR6HZ30NHxLCMjG5EkHWlp3yAv79yYq58qipuenjcn0n79JCcvJTfnzJja4Eqtn9Hb9x5KwEli4hxyc8+KaagCjI5uo7vnDXy+fqyWInJzz8JqLYqp9Xi66e55Hbe7HZMxg5yc06ap8eehr+/9ULRlZubxJCSUxNQKoTI8vJaRkU3IsoH09MOjjuFwHI6GoOEuJqItk+bH1fp8QwwOfkpAcZGUNJ+kxPlxTcpAYJz+/o/w+4exWotJSzs0rrErhMrI6GY87g5M5mxSkpfE1U6O2enci95gJyV56QGM3V7GxnYgyyaSk2uiTPbIMTsZG9sOQFLSgmnTtlU1wLhjF6rixWYrjzL6I7dP4Ha34PcPY7EUHXBVXo+3B6+3F7MpB5Mpc1qt3z+Gx9OBwZB8wJR0VfXidrej01kPqJ00rpFkTMasA5rRfv8oqvBjNKQdUKsobgKKE6Mh9YARs6oaQFGc6PW2aY+J4JhVAgEHOp31gEa7EAJVdSNJhmmPn6lx+JEk+YBjmBwHiL9LOzmWWHOmGVFx0IwoDQ2Nfwd+VeBQFJL0uoj6VOEIIfhkaJznuwZpdXvJMhk4OzuVEzOTo57T5PJw0Y4WGlxTJxsmSeLG4myumhH5w/L37X3c1hh9gmuUJF6uKuWgsELvl+1qiairFc5ByQm8UT11gnTq1r2sHYmuAzTJH+YUkWnU41JUfrK3g+ZpamtVJpgpsZpwKyqbRp2hYvQaU0hCRT9hXOlRkFHxyIlkmcwYJIlhnxtboC0UDRbUBdARQI9CjzSDJZmzMMgSXc4hzGOrJh4LoJ+IJps0xMaxs7DgFMxygHF3P86+F9Djn9AqoT51BJBQyUg/EqssoQaGGRn6BBk19CchkCdOziUEen0KRp0JIXxR5oEWq/C/gYzJmIEk6QgExgko09Vdk0m2L0KS9fh8w3FTMCdNo+zs0zAYklFVL52df0bEMFMntcXF10ycUMjsbfxVzIUQJrUzy24lJWUJkqSned/D9Pd/EFdbUf5LMjKOQADtbU/T2vZ4lHaSgoJLKCm+GknSMTKyiW3bL4xr4OXknMHsyrsB8PuH+WLNwQg1tjmfmnoI1VVPB8clBOs3nIjTuSem1ppQzrIl74V+zNfV3Upn14sxx2EwZrLioM9Ctcn6+j5kx87vx+xXb0hn+bJVoZPM8fE9bNh0KojoCxqSZGLpkrdJSCgDgifia9cfh8/bFWMcOhZWP09KypLQ9m3ddgHDw2tijnnOnIfIzjoxdL+p+QFaWh6NOeaZZT+hsPDi0P2+/g/ZseNKQET1XVBwMbNm/iR03+Xax4ZNZ6AEor/PsrNOYfbs+0JzHAg42bTlXJyOnVHalNTDqJr/ROhkUAjB7j0/oqfntShtQuI8Fle/EHHSHy/t12QpYcmiyLTf4ZGNbN12MUKNXEBCb8yiZuELEeaO19vLhi3fxrffqrGyPplF1c9EGJSK4mVr7eWM7r8Ah2ymat5jpKUdFmoSQlDfcAednc/uN2IdFeV3kJd3TkRrd/dr7K77MYSlTAskSouvo7j4ygjt6Nh2tmy/DNUfGcmZl3ch5bNujTiB9Xp72bztUtzOyHqR6ZknMm/23RGpv4rioXb3DQz1/zVCm5RyMFVzH4mI4hRC0Lzvt+xrfQwp7AKENXEhC+f/NsqY6+x6hbqGOyFsn+hN+VTPfzTKBB4bq2Xrzh8S8EztE50pnwVz7iMluSZC6/F0s23nD3GObQi1SYYM5sy6jaysYyO0iuJi556f0t/3NtKk0a2zM7PkWgoLzo/QCqHStO+3tLT9EUkN/j4TcgIzCi6mrOTqKLOku+ctdu+9B/zBCwpCMpKTczaVs26JSq8eGd1Mbd3t+J3BizECPWmZxzO3/PaodHCXq5XauttwjnwxoZVISjmUeRV3RqTFQ9CA2t3wC/r73gntE3NiDfMrfkZi4uwIraoGaNz3MK0dzyMrwYtpeuts5s26Oco8B+jseo265kfAF1xMRjKXMqf0WrKyjovSDg+vZ8fe+/A7tgTHbCygvPhy8nPPiTJXnM4mdjTcjWP4MyQUVH06xQXfoWTGFVHGkc83yO7GB+jvfRtZuFDlJHJzz6K85Ooog1BRvOzd9yjtnS8iK0Ookpn0zBOZXfbDKCNPCEF7xws0tP4JydeKQE9i2hHMLfthaHGccPr7P2Zn82Oozm0IZMz2Fcwruwa7vTpKOzZWy47Gh3GPfAEo6GzVzC35Phn7rdQM4HZ3srvpYYYG/oqkupAsFVQUXUxu9qmhedOMqDhoRpSGhsb/VVQhWDPsYJfDTZJexzEZdlINsa+cvNs3wqNtfWwdd6EDjkxP4rqibBYkRl69HQ8oXLxzH18MOyLaFyRaeH5+SUTq4Y5xF6dubcQRwzS6oiCD28umfmy81z/CJTtbYo7NppP5cmllKBXyz92D/LAu/gp0t5Xmck5OKh5V5S89Q9zVHDsiA6DUYuLSggx8qsrWMRdvxDHZIFj/anmyjYAQDPgDNLomakUJgQ4FgYSKDFpqx78MvfBNGGLBlExZ0pGsByN+FP8QFhyYJlI1g3+B0K0qmUg3J6ITfjzeXsxidMI4C0QYckHzTCLRlIxO+PH7h9ELZyiCTYcyEX2mhm51kgEJBUkEYOJxja83AjAaUpAkHYriRYlj4E0aJraECmSdGUVx43TWxzW4AJLtNRiN6cEVJQc+mjr5jEFa6jdITKwMRsp1voTwx69LmJZ+FOkTJ2ttXa/hdtTG1SYk1TAj70xApn/wc/r73o6rNVgrmF12PSDhcNTT1Hxv/O0zZLJw7sPIsg5/YIxt2y9DipMWKyQLNYteRK9LAATrtnwH/NFp6ZPvxprFb2Ax5yNJOjbvuB7nyKcxtRKwsPrFkHnW0Pxb2lsejKstr7iX/NzTgODJXO2Oy+JuX17hlVSU/RAInqCtWfuNuPvPnnEyi+c9AAQjDD/58hCkQHS6LYDBVs2hS6ai877cfD7e0TUxtRhyWLnis9AJ8Y6Gu+jreDLmmFXJwmErVofSaNu7XqOh7qaYWoHE0pq3SEycA8D4+C7Wbzwl7vbNnfs4WZlHAUGz4bOvViIrkd/BoejQ0lspnXFRsE0IVq8/GcW1K6Y2Lec7VFXeHmrfuPNGxvpejxltaUo+lIMXTqVLN7U9Q0vjHTH7lcyz+Mayd0Lz1j/wGbW1l8TcNlVn57CDVoWMRLe7gy/WH49OdURpFcnMiiVvh8wBRXHz6drjJ8yDyHkWyCxY8BQZaQeH5mLNlovwjn4Rc58Uz7qTkvypBUC2N9zNQMcTMcecnncpC8pvDt1v736bhj3XxdSaU45kRfVUevvY2A7WbT4bnfBGj8M6h5VLXg1FDnq9faxedxI6pT+knbxV9Nl8Y9nboeNNVb18uuFMCNvXIa1kZXnNa9hss0KPfbX9GtyD70ZpVWQWzH+KzPSDQ4/tbv493S33RGkBimbdRWn+2aHHuvs/Y+eOy5BRorYvNf97VM+aitB1OPayZuMZ6EX0vjamHM3BVY+GDBi/f4RP1p+G3hedzi+s81lZ81LoQoKqBvhs0/kIR3QtU78+h5XL3oxI816/66c4ev8cpQ1INpbXvEJiWPRgfevzdDTdFqVV0TN33hPkZEyZ0b2DX7F9+8XoYpSmyC+7k/LCqePN6dzHFxvPxKAOR2mTci6ipvJWQDOi4qIZURoaGl8nfKqKTpLiRmVB8IfPulEnnw6OoQAHJ9s4LDUROcZz9jjc3NfSw4cDowQEzLSauLwgk2/npEZdQfp9ex+/aOqOSBNMN+j549wiloVFZQVUwUU79/G3wegV3Q5OtvHCghJMcvCqnltROWpTPXtd0QXGdcDLVaUcnJIY2vbl6/fQ4Ymd0nhBbhp3lxcAwQi2Jet20x0n/THToOOLpZUoE2M4YfNeun2xtQC3luRQYDESUAV3NXeH0ioloSKjoKAPGVxL7QlU2iz4VJVVA6MM+BUQAjNu9ASCcUtSMCTfppPJNRkJCEGP149LVScip4JXpxV0mnn2r0AEo7yCxpUaMrD0YWbW1F8gLDpNmYgUm9wzk5rJ/yf7U5AntLpQdJkS8fypx8Lbw/+UsOdGP18Xdl8P6FCQJgw7OTSG8Mi2yf+ZuD/5+Nfrd6PGfxMSkiSjCmXaaEyBDoPehiTJ+AIOpBhRZFNaGas5HyTw+kdRp0mBFkCirRJJ0k2kJbXF1UkEU6x0ugRU1cvo6OZpzUxbYhUWcyYIQe/AJ9PWMDRaK0ixlYEk0dX/GTo1flSkZCokO3UJINE9tAG8sevlAQT0GRTnnICETP/oDtxh0T9RWimBshkXISHjdLfT1/tGXK2CjrKiK9HpzAQCDppbH0eaJtqysPAKzOZsECq7Gu/HIOJHcmfkfIvUpGAUzs7mJzD4418Us6QcTlH2UUhI7Ol4AzG+Nq5WTahhQfGFgERb/+eM9r4UV+s1FLOk8scAjDqaaGn+ddx97ZOSqJn3ILJswB8YY/vOq9HFMQYVdFTPexSDISVY73D7DzCp0YuWhCJPy+8mOTFoAn2x83bMnu1xx5xZeB1FWYcDsLnxCZThaYzr9DNZWBKMdGzs/pDB9ofian0JKzh07u1IksTQWB31u6+KOxcuQxGHL3oaSZKDKwNvPBUDsSPxvdhYedD76HQmhBB8uO40EpTozAEIJlsvrXkPy0QK4AdbrsLqjDaBQ2n0s39HXvpyJEni8z33ofY9E1ebXPhjqorPA2B3+xv0Nf8k7vaJtLP45rxganz/yE52bjszrtZpWcKJy54HJHy+ET5dsyLuXLh0uZxw6GokSUYIlXe+WElCIHaZDj8mvnnwVxiNwVqG76y/CKszuv7k5LgWLHqHdPtszYiKh2ZEaWhoaPz/41cFPqGSoJs+j7zP6+ft/hEGfQHKrCaOz0jGrIvOq/ergue6Bnihe5AOj58ck4Fv5aRyYV56yISapNfr59q6Nj4dmvrxnGcycNesfI5Oj6x1sMfh5tztzVGm0crURJ6cWxxRt+vTwTHO37Evqr6WQZJ4el4xh6dNhaK/3jvM9+Ospnhcup0/zZtKqZhu5cVUg44vl1aGItum0wI8NbeIYzOCPwg+GRzj3GkK3l+Yl8atJbn4hWDTqJPv7JhIHxACE55gvJAUjEorNBu5d1Y+CtDh8XHTRO0wq3CQQxcqMl3k4pWsyMAPi7IwyzIOReWR1l4k4WMGLWTRg0BiiDRaKcIjWTksJZEskx6/Kni3fxRJdZNDF3ZGEMi4sTBIOsOkkmM2kmU0oApBvcuDUNwkM4KECNkifow4sKGT9Vh1MgEhcCvqRG0DENpKZ/9ahAiZVDrUsOizcAMu/L6CHn/I4Jo0vKQoI02N0EyaevH7nboVYael+/8fTBAl1DYV/abGNOBim3Gx/qZ7njhAX1PpqxoaGhoaGv8ugr9wpaiWaGt3Ehlp4ntVxDC4h0khnaDB6Uo9jxOrfq4ZUfHQjCgNDQ2N/w6aXV4anB7sBh1L7Alxo75cisqbfcNsHnVilmWOzbCzIjn2ako7xl38tq2P1RMm1yEpiVw1IzMqpRHglZ4hftXcHYqiMkkS5+Sk8vOyvCiz7fG2Pn7V3I0v7Ds332zgT3OLmR/WtxCCHzV08GxX9FXLS/PTuaMsLzRuIQQ/2NPGq73RIdKlFhNvLZxJunGqzsl3d7XwXn/01XoJeGZeMUeFmXiX72rhrThpjRfmpfPrWfmh+zfWt/NcjPFCMGLusyUVoX1zZ1MXj7YFCzZLImgCBCbMML0EXy2tpNASDFufrHUmCZVZ1JHCMH1k0kzwSv4Tc4o4KTNoyk0ag8limIP5lCx6GcXOGg6lW8rngtxUbi7JRRGwadTBhTtbmCV2cxifkEc7LhLYzkI+5xuUJKbzQEUhiipo83i5YncbFWInB7GGdPoRSLRSxFcczKCuiHtm5WOUZcYDCrc0dFAhtlLNFoz4JowzA3XMZjM1XFqQTbbJgF8VPNLaQ6FSSzm7ISwxcJRkNrKUxanZVCRYUITgrb5hknx1FLEvzMoIxjPtpQKbNZ95Nisqgs2jLiTvPrInTMGphEOZYVIY1RUyL9GKCrS5vfi8PSQzHKX1Y2BAymKGxYIKjAUU3L4xrLgixjCpd2PBog+magSEwK/4kSdq+whk1L+z+OnXCUlMmVZSmEEV3hYZBTdZLy48Ak+diGpTIiIjpFDkmghrY7/HRcRelEL3I2u9TY5pUh/rNmj5qaHTlci+iNu/FPH8qedG9knYWCP7kcNeM7pO3fRjme6PA4xncn+Fjyt8rvavlTd9n/HGEWvOvl7nbRoaGv/Z7KGSSoI1ENstR3HhQY9rRlQ8NCNKQ0NDQ+OfRUAVbBt34VFV5tgspMSp2QXBFQ3f6x9lxB+gPMHMEWl2DHK0GSaE4LOhcV7sGaLb4yfPbODbOWkcnBJtnilC8FzXIE93DtDo8pBq0HN6Vgo/mJEVVT/Mq6r8sqmb57sGcanBE6Qyq4mfleZGmFAQNO9+WNfGm2FmlAycm5PGXbPyMIZFqbkUlct2tfDRfqmVxRYjLy4opcgyVQDVrah8p7aZL0ciay3ogN9UFnJG9tSKPF5V5bza5qj6ZQCnZ6XwSGVhKH00oAq+XdvM6uHoFJMZZiPvLpoZqnc2nSlnkCRerSplaVjq6FW7W0Nmn0U4UdHhnUiVvLk4m2uLplZSu72xk9+192MQXmbSgA6FJspwSTZWpiby4oKpgqKPtvVxZ1MXdjHMYjZgxk0LxexmHhkmI2uXVpKgD5o3r/YMcdWeNnJFB4fwGUmM0ks2n7OSMSmVv9WUM8cWXMnzy+FxztjWRLnYzUo+Iosehknlc1ayjYU8UFnIuTnBeh1NLg+HrK9joVjP0bxHIa24sLKOFbzHSZyeVxxKXx3xB6j+ajfVympO4E1m0IKKxA6qeJ0zsSVV88HiWaH9sXTtLmq8L3ECb2LBDYAHIx9xDO/J57Nh+Twsehkh4LQte8kff55jeRs9aug0vIkyfs+V/Km6hplWMwL4cX0biQOPchBrwqKfJPzoeInvcFzpyRyVbkcAT3b009D5FifwJhIipFWQWcWxZGSdwnfzMxDAx4NjrG55l+N4m8kaJBC0GZopZY/tQm4uLUQlGGn5etMqjub9kGby17QDGx/qzuWW8nnoJYkhf4AnGz7jSN4nWDtlKlpLReJjjuHc0kNINujxqSpPNG7gUPV95IkowPCorjpmMyv3OMom5uLl1lqq/JOF2yPtjlHseJJPoybFjhDwQW87xe730U/UvwuPFPNhpN1yFIdl5ICADcND2B0fYMK3n30S1O/VLeUb2cHC5g1ON+rIR1hxhnRAyK5plUpZmDkfvSTR5/MzOLSGJMaItMqC2iFSyU1djE2vw62oNA9uC5m/k3M3OR4PZnSJi8k0mlAR1A/Vky56oywfJval0zSXfGvwfd042kGq2hn22pGRdKP6GRQkpCGAducQSYG2iDkLH/OYlEquLQeBoN/rxuRr3W+sU2NxYyHFmocswXggAN72kCoyig8C6DGacjHKEl5VxeftCdUuJGwLQ2PXp2HR6RFCxeUbwoAvNG9TzwjeBuQkLDoDAnD7HejxE1ltaSpmQsGIWRf8LvEpPvR4w1473EgLzrNB1gECVVWQJ2rOTGlFxKvIyEiSIHguOmU4To5l/zjH8G3e39zbP/5Rjhrf1BFMVK9EtIWP4u8zB9Wo8YXv+f1jUPY3iWO1x2uL7i8yBjRyjiLbpn497N8WWxNr7JHbFdnf1GOxt3O6x6Zvj71dscYbfbzEen70vor3fyyinx97f4Vv199D9OvGntfI/mNdaIg1X9F9xn7u39d3+OMObCwjmKbakPRdvrf4Fs2IiodmRGloaGhofJ1xBBQaXB6sOplyq3na5Yr3ubysGXEgA4ekJlJgNsbUCSHYMOrkw4ExfEKlxp7Asen2CMNqkoAqeLd/hDf7hhkLqMy2mTk/N51ZCdFLU3tVlee7Bnm5Z4her59Cs4nzctM4MzslqoaZR1F5pK2X57oG6fMFsMoyp2encGNRNpmmyCWOfarKPft6eLZzILRi46IkKz8tzY2oXzY5hp/t7eTP3UOhtM0EncxVhZlcOyNyaeiAKvjx3g6e7xqMSLw6PDWJx2YXYg8zB1Uh+MneTp7qjCxWnGsy8Nz8kpCxNDm/dzR18Xh7ZGFqvQT3lRdwzoSxNMl9+3q4ryW6qP852ak8UFEQMXd/7Ojn1r2dUdq5NguvVZVGjPmVniGu2dOGAMzCjYIOv2QkSSfzWnUZ88Ki+1YNjHLxzn3oVDczaUBGpZGZuCQbv6ko5OycKdNx86iT07c1YlBGqGYzZjw0UUYTM7kgLz1khkEwEvLYzQ1k+ndxMKtJZJxucvmUI5hhL+KN6rLQcTfgC3D0pnp8ng4O5VPS6WdowpRTjYWsqplFjil4THsUleM2NzDuqONwVlFAGw5srOUQtkjLeKW6PHRsCCE4a3sT3UNbOJZ3KKMBPwY2sZQPOIEbZs3lu/kZoTFfX9fG5q4vOJnXmMMOAHYzj7c4jarcQ3mwojCkfbClh7eaP+ZMXqRi4irzIGm8z4l02M5gVU1FaP+92jPEk7tf50L+QDpTx9EOFvBn/TW8v2xFKCpy3YiD67d8wGU8Qj5T9UBamcEfuIpHFx/NwqTgik6tbi9nr1vF98V95IVpnSTwDJdweuV5ITPTGVA4eu1qLvLfRRH7QloVib9yItb86/jFrMLQvJ24fj0nuW6jlMaI4203c/gy+Q5eXlgVaru8djezB37M3Ik5m6SfDJ403M67K44KpW7fubcFS/tNVLM5QutHz++ka/nNsouZMWGKP9/ZT1P9TRxMdK2TdziFkxbcxjcnUrHXDI/z6tbbOIG3orS1LMBY8huuLAoen+0eHz//6ld8i/1XoIM+Mtmc8Qj3zQuem7kVlcu+fIzzlYeiopy8mHjF8gueWhZchUoIwWXrXuQM9+3oY6TGvCRfykOH3BhKlb9x28ccNnRNyAAO51MO54KlDzNz4vP2/oad5HdcEnH8TFJHJUVznuL4rOCx/JeuThx151HIVG2rScNmiFS6Cv7ENTPnAbB5ZIw1W77DHHZGaf3oWZ18P79aeAIQvEjz2Jff42A+jRoDwDumH/DAQVcjSRI+VfDjL27jGPWliH6DtxKrpaO5ccWDJOiDx8VP1j/B4e4Hw7TSxDgk9jKLQxY9zyxb8HPr1zs/YvHgDaHIwiktDJCBZdYzHJcdXJTl6eZtZHdcgZnJlVslIrAAACAASURBVDOnPlO9GNiX8zCXli1FAJ/2dTNWfz4pDIf6DR/7+sSbuGHBGQA0u9ys2XIhM2iNsnokJNYYTuemmutBAldA5ekNP2Se2LiffRf8v1ZaxPk1D2DV6xBC8ODGe1gSeC9qfgUSvWQzf8GTlFqDnwGP1L7AQucfQ32Fj9mNGUp+xzczg+/rZ/Z+TungHaHjeH97qCXzTr5VvBSAD7qbsLRdjXHCpNw/Haw24RIum3sWAsHu0VE66i7HzmhoDqbmArbpj+Sy6uuQJIlBn59Pt/2AfNojtJM0SZWcUn03Fp2MIuC5zbdTKbaEbd8UI6RQMedxCqzB4+JP23/PXN9fY2r9GGDG/SzPCM7Fn+vfYeb4kxH7I5zOtJs4uXgZCHivfRuZfXeGzVUk9Qnn8u2K0wHYNNSNb9/1GELzFkmd/hC+Pf8HAHR7PezadW3M4w2gTSrl2AW/wCxLeFTBX7ffSqFoChvHlH6MJMorHyLHbEEgeGnHI8wKrAvbPkhmmFy68GNAV/kuR+aUaUZUPDQjSkNDQ0ND478XIQQORcWqk6ct0g/Bk8E2jw+bTiYvjsk2Sb/Pz+ZRF3pZYpk9AZs+fqpZu8fHp4Nj+IVgebKNyjBTaX8anB7e7R/BEVCZl2jhuAx7VF20SWrHXbzaM0y/z0+RxcS3clJDqYz7s2nUyfNdg7S4vWSbDJydnco3UhNjGo8bRhz8qXMgtCLnKZkpnJubGrMG3NoRB79r72PDiBODLHFMup3vFWRSbI0ex/oRBw+19rJ6aBwVWGZP4JoZWaxMS4rSbhtzcfe+7lDttzyTgcsKMrg0PyPKdKx3eri9sTOktcgyZ2an8NPSXBL32y+tbi+3NHTwSVhNuUNSbPxqVj5l1kjzs9/n54b6dlYNjIV+6BdZjNw1Mz9kTEwyFlC4rq4tIrIuQSdz3YwsrizMjFyaXlW5paGDl7qHQrXMJEnmrOxU7i7Pj9jfqhDc3tjFHzv6SRCjGPAzTArV9kT+NLc4tNroJA+19HBfcydl1JGAg04KEKZCnppbzEJ75FLhL3QNcnN9O4WigTQGGSCDDqmM+yoLOSssGhGCRuIVO5spUneSQyfjJLGNRVxSmM/PSnMjtm/zqJPztjeRE9hOKXvxYmYTS1iQVsyf5hZjCUtVbnJ5OHvrXlK8m5jDTgQStVThsyzkL9Vl5Ia9Dwd8Ac7a2oDs3MhCNmLATwMV7NEfwjMLKlkUtn0uReW87Y24R1ZzEGuw4qKNGXzGkdw6e3HE9ilCcNWuFlr7PuJQPgml/X7KkRxW+M2o7ftFYydftH3I4awimy5GSeZLDkOXegJPz58ZYbg/0znA8/XvcgzvUsQ+PJhZx3IarKfz/MJFESvR/m1glF/veJ9jxRtUsAsFPdtYyKf603l04eERZnTtuIubtrzLUcorzGMbMoJGZvK+dCqXzT2b4ydqBwJ0enxcuuljDvU9yyI2IiPoJ4MPOY75xd/luuLckHY8oHDB5nXMdz7JMtZgIICTBD7lcNwZl/HY3IqIyNPLtm8nb/hRlvNFcHVTZDZTw+aEK3h60YqIz8Wf1DVA1/0czOqQ8dBCMe8aLuGhxaeFjEGA37d1Ud94L0fwIaaJqKthUnhTOocrq6/goLALBB/0jfDmzvs5njewTphtXkz8jWOoKf8R5+ZlhbTbx138ZtMjnCJeIIGpouWbqcGTexs/q6iMmLfrNr7Eqf7HyaQv1L6H2Wy138wTC5eHvlOcAYXvbXyT49z3ksHUBYIhUnnHdB0PLzmL5AkTXxWCq7auZunIbREmsAcTb8gXcfOSH0R8Ft1Rt530rh9TTl2oTUViFcdz7Pyf8830qX39VFsrA403snA/A3Yri0gpu4dLCotCbR8PDLG69kccxicR2nYKaM66hzvmLAm17R538YdNd3CceDXCKB0jkVW223hk8SnoJyK6+31+bl33KKcGHscwsXAKgA8Dbxh+wK+XXR6K0PaqKldveIkTXXdFGaXvS2dwac3tVEwc90IIbtz2EcuGbwmZKpNsYBk18x7miIypizAPNGwlo+NqcoksQt5CMWrRb7m0pCLU9pfOFgbrv8dMGiK0I9jZnn4fd8z/Zqht3fAwq7dexWLWRWh9GPir9Sc8sOS80Fy0uDw8seFmDlffYX/e01/ETw+6ORQ57wgo3Lr2QY73/y7KjP5COoJzah6kYsIkVYXgho1/5kjHnaH30iS7mUPlvD9wRMbUcX/njo+Z1389iURGifeQjafod1xSMi/U9mRzLeaW75FN5MUrBwlsS7+X2+YfHWr7W28nTbu+GzVvAXSsst7EPUsvQSdJmhEVD82I0tDQ0NDQ0ND49xFQg7819TFSUfdnPKDgUVXSDPqYK3eG0+/zM+xXyDMZQmmM8ejy+Oj0+sk2GeJG9k3S7vGx1+kh2aCjKtE67Tha3F62jrkwyRKHpCRGGWHhdHh8fDmRProiJX6EIQRPij8cGMWjChYlWVliT4gbvdjt9fF23wgjfoXyBDPHTmNm9nn9vNo7TLfXR57JyOnZKRHmSDiDvgB/6Rli72Tab3YKFQmxTdVRf4C/9A6zdcyFVSdzfIadQ1Nir77qCCj8pXeYL4fHkZFYmZbIqZkpEYbVJJ6JGn+rBsbwqoIl9gTOzU2NOWa/Kni7b5g3+0YYDyhU2ixckJcWc8yqEHw4MMpLPUP0eP3MsJg4LyeNQ2KkQENwMYtnuwZpdntJN+g5KzuV07JSYqZXbxx18seOfnaMu7HpZE7MTOaCvHSSYhwbux1uft/ez1cjDgySxBFpSVxakBHz2Gh1e3msrY9PBoZQhcLilFS+V5BJVVJ0DcN+n5/H2vp4v7cXv+KmxJbJpYWZUQt6QPA993h7H2909+L0jZBhSee8/GzOz02LMvN9qsofOwZ4pbOdcU8vZmMaJ+UU8v3CzKhjXwjBC91DPNe+j3HnPiSdjYMy53BtcTb5Mbbv3b4R/tC6j9Hx3Qj0lKRXc21xfswajWuGx3lkXwv9I1uQESQlzefyotKoNPPJOb6/eR/dA19hxEvAXMEZMxZwQW5a1L5u9/j4VVMHDX3rMIsxXPoCjsxfyHUzsqJqP474A9zd3MHO7s9IVPtwyGnMylrJj0oLoo5Pr6pyf3MX6zo/J1lpwYUNe+pKfjRzZlQ0sCoET7T18UH7l6T4duHDiJJ4CFeVzQ+tChzOqz1DvNS8hmRP8Px2xLyYs0tWcOZ+5jLA6qFxfrd3HYnOLzDgpU9fycEFR3FVUU7Uvq4dd3Ff/WbMY38jgXH6pBkUZh/PrTNLoj5v2z0+flG/A//ge6QwyBAZWNKP59ZZcyLMZQh+VtzR0MBg39tkig4cJOFMPIobKpZERNVC8Hj7dWMzjV1vk6/uxYuJbvNhXDRzJUeGma8QPN4eb+tkfcubFCrbEUi06RdzeMlJnJ+XFbWvX+3u453Gtyjyr8OAn1a5ktK807mhdGbUd9Xng2M8Vf8e+Z5PsOKiQyrCmnEqP62ojnpf73G4eXD3KtIdfyWFIfrJwmE/mVvmHBr1vu7x+vnFri8xj7wZMrm7rEdxZeVxURcSHAGFO+s24+l7jRk04cZCk/FQzph1KsdlpkdoA6rg3sbddHa+TKnYiYKevbpF1BSdwyWFhRFzIYTgybZWNu17kVnqpmBJAWk2WbnncNPMeVGfce/09PFOw0uUBz7HgptWSlDTzuTW2ctDEdSaERUHzYjS0NDQ0NDQ0NDQ0ND496AIgQzTpoJP4lNVdJJ0wIhWCBo8qiCmiRlL61UFiTr5gOPwKCrjikKKXn9AA92rqoz4Fex6XcxVgcPxq4I+n59EvS6mMRmOIgSdHh8WnRzXLJ5ECEG7xwdAvtk4rXkuhKDL68ejqhSYjTFT6MPp9voY8SsUmI3TRgJD0ADt9frJNhlDacHxGPEHaPf4SDHoYxqT4TgVhUaXF6ssU2Y1Tbv/vKpKndODDFQkWGKaxZMoQrDb4canCioSzNNe0BBCUOf0MBZQKLOaSTvA9jW7vPR4/RRajAfcvk6Pj31uLxlGA+UxyhSEM+ALUOd0Y9PpmJ9omXZfjwcUto+70EsS1UnWuBcoIHjMbx13EVAF8xMtEWn5+6MIwZYxF+MBhdk2S1SkrmZE/b/27jzakqq64/h3yySgDDIokzYRULIUjcEpCxDBGEOMoIKR5YSASXQZMA5EY4KtxCkYNGqQpaioOBE1qHFgkOAUAwoyNDII2sggKoIMIQ7IyR+n3ut6VXvvqifdt6ub32etu/re93593qnat845Ve/e+wK6ECUiIiIiIiIisnKNvRA1fPlYRERERERERERkJdCFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ0IUoERERERERERGZCV2IEhERERERERGRmdCFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ0IUoERERERERERGZCV2IEhERERERERGRmdCFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ0IUoERERERERERGZCV2IEhERERERERGRmdCFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ0IUoERERERERERGZCV2IEhERERERERGRmdCFKBERERERERERmQldiBIRERERERERkZmwUsrq7sNMmdnPgKs7X94SuHERzSwmvzZnp9KPNS07lX5MITuVfkwhO5V+TCE7lX6sadmp9GMK2an0YwrZqfRjCtmp9GNNy06lH1PITqUfU8hOpR9TyE6lH2tadir9mEJ2Kv1YGdkHlVK2GvzfpZR7/A34zqrKr83ZqfRjTctOpR9TyE6lH1PITqUfU8hOpR9rWnYq/ZhCdir9mEJ2Kv2YQnYq/VjTslPpxxSyU+nHFLJT6ccUslPpx5qWnUo/ppCdSj9W5fZ1b3prnoiIiIiIiIiIzIQuRImIiIiIiIiIyEzoQlT13lWYX5uzU+nHmpadSj+mkJ1KP6aQnUo/ppCdSj/WtOxU+jGF7FT6MYXsVPoxhexU+rGmZafSjylkp9KPKWSn0o8pZKfSjzUtO5V+TCE7lX6syu1b4B73YeUiIiIiIiIiIrJ66BVRIiIiIiIiIiIyE7oQJSIiIiIiIiIis3F3/uTe2nADngJcDlwJvHog+wHgp8CygdwOwH8B3wMuAY5MsvcGzgUubLKvH9HndYDvAv85IrscuBi4gIE/sQhsBnwKuAy4FHh8kHtI097c7VbgZUm7f9ts2zLg48C9k+yRTe4Sr02vBsD9gDOA7zf/bp5kD2ravgvYfaDdY5t9cRHwH8BmSfaYJncBcDqw7dBzBngFUIAtB/qxFLiutb/3y9oG/qbp9yXAPyftfrLV5nLggiT7SOB/5p5HwGOS7COAbzXPu88Dm2THhVe/JNurX5KN6hflezWMsl4Nk3Z79cva7dYvabdXvyTbq1+SjernjlXAjsA51DH0k8D6SfalTW7+eZ9kP0odm5dRn2frJdn3N1+7iDqG3SfKtvbzO4HbB/pwEvDD1n5+5EDegDcCV1DH0COS7Ndb7V4PnJpk9wXOb7LfAHZKsvs02WXAh4B1o7nDq12S7dUuyfZql2R7tRsz37Xrl7Tt1i/I9mqXZHu1S7K92iXZrHbL6czlxHOfl43mPi8bjZ1e1p37onw0/wVtL8Wf+9x28ec+r91o7vOy0dznZaOxs7e2imqX5KP6edmofl42WruE60Gndl67Ue3cdr3aJW1H9fOyUf28bK9+BOtdr35J1lu3RNmodlHeW7eka3QWrluidnv1y9rt1i9p11u3RFlv3RJl3WOv6VvvHIRg7guy7twXZN25L8i6c5+Xjea9oN2T8NctXjab97y8O/cFWXfuC7Lu3IdzTkg873nZaNz0su6xl+SjsTM8j6U/dnrtLsUfO9128ec9r91o3PSy0bjpZcNjb8xtdHBtvFEXgVcBv0c9eboQ+P0kvxfwKIYvRG0DPKq5f1/qAe62Sx0E5gaf9agD4+MG2n858DHGX4jacijXZD8EHN7cX799EA7swxuABwXf3446IG7YPD4FOCTIPqx5gm8ErAucSWvRHtWAOum9urn/auCtSXZX6kR2NgsHJS/7ZFYMhm8daLc96R0BnJA9Z6gXAU4DrmbhxOa1vRR45ZjnI/DEZr9t0DzeesxzF/gX4Oik3dOBP23u7wecnWS/DTyhuX8ocEx2XHj1S7K9+iXZqH5RvlfDKOvVMGm3V78k26tf1odu/ZJ2e/VLslH93LGKekw/u/n6CcCLk+wfAEtojUtJdr/me0ZdrGTttmt3HPV5FI6twO7AR1hxISpq9yTgQOd4ifIvBD4M3KtVv8ExHvg08Pyk3SuAXZuvv6Tpl5f9I+AaYJfm628ADmv9nAVzh1e7JNurXZLt1S7J9mqXte3VL2nbrV+Q7dUu60O3dkm7vdp5Weor1LPaefs9mvu8bDT3edlo7PSy7twX5b2xM2l7Kf7c52Wjuc/tQ3fsTNqN5j4vG42dvbVVVLskH9XPy0b187LR2sVdDwa189qNaudl3dpl/Qjq57Ud1c/LuvVr/az59W5WPyfr1i7IurVL8uHx181G9QvadesXZMP6eX3wahe069YuyEbHnnsOgr9uibLeuiXKeuuWKOutW8JzJvrrlqjdk+jMe0nWnfeyfrTanFu3RG176xYveyjO3EdwToh/zhBlvXOGKBuNm1HeO2cIz2PpnzNE7S6lf84QZb1zhjHn0nPnDFG73jlDlE3HzaHbPf2teY8Briyl/KCU8mvgE8D+UbiU8jXgpqFGSyk/LqWc39y/jXqVebsgW0optzcP12tuJWrbzLYH/gw4cagfi2Fmm1IvKry/6devSym/GPFf9wWuKqVcnWTWBTY0s3WpT+Drg9yuwDmllDtKKXcCXwWe0Q4ENdifuqig+feAKFtKubSUcnn3BwfZ05t+QL0yvH2SvbX1cGOaGibPmbcDR9Gp9djnWJJ9MfCWUsqvmsxPh9o1MwOeRZ00o2yh/nYQYFOaGgbZXYCvNffPAJ7ZZKPjole/KOvVL8lG9YvyvRoOHMsLarjI4z7K9uo31G67fkm2V78kG9UvGqv2of42D1bUz82WUr5bSlne2RdR9ovN9wr1lT/bJ9lbW/tiwxXN9rNmtg71N19HDfWBQJJ/MfCGUspdTe6nQ22b2SbNPjw1yXr187K/BX5dSrmi+fp8/bpzR7OverXzss229GqXZHu1S7K92mVte/WLspEg26vdULvt2iVZd+x0slsQ1C7hzn0eb+xMsu7YGWTduW+AO//dTe7cl+nOfQG3foHe2JmsrdzaRXmvfkm2V78k26vfwHpwQe0Ws3ZMsm7thtpu1y/J9uqXZN25r6W93h069uazI469dnbMsdfODx1/3TV6duyNWc972aFjr9ducuy1s0PHXjub1a57DvJjgrnPyV4fzX1B1p37gmw09/Wy0bznZZ1+Zll33htquzv3Bdmoft3s/+LPfdE5oXfsudng2Iuy0bEX5b1jLzuP7R57g+e8Q33GP/bSdjvHXpT1ahdlh8bN1D39QtR21Kuwc64lOHH8XZnZEurV9HOSzDpmdgH1LU5nlFLCLPAO6hP5rpFdKMDpZnaemf1lktsR+BnwQTP7rpmdaGYbj2j/2SSLuFLKdcDbgB9RB/9bSimnB/FlwJ5mtoWZbcSKtzANuX8p5cfN/RuA+4/4P4t1KPClLGBmbzSza4DnUK80R7n9getKKRcu4ue/1MwuMrMPmNnmSW4X6j48x8y+amaPHtH2nsBPSinfTzIvA45ttu9twGuS7CWsuKB7EE4NO8dFWr8xx9CIrFu/bj6rYTs7VEOnH2H9Otm0fsH2ufXrZNP6dbJh/bpjFfUVpb9oTd7zY+hixrUsa2brAc8DvpxlzeyD1OfPQ4F3JdmXAp9rPeeG+vDGpnZvN7MNBvIPBv7CzL5jZl8ys51H7IsDgK+0FqVe9nDgi2Z2bbMv3hLU41xgXTPbvWn7wFb9unPHFgS1c7KZMNutXZT1apfk3fol/fDq52Xd2mXbR6d2QdatnZO9kbh24M/l0dg5dt4fk22PnW42GTd7+WTsjPrhjZ1eNho7s+3rjp1eNho7vaw3dkZrq6h2i1mLjcnO1S/MOvVzs0Htsj50axdlo9oNbV+7flHWq1+UHVq7tNe7Q+vOdG08MhutOxfkk+NvQXZo3eL0I1t3trND605v+6J1Zzs7tO5sZ93aeecgwHk4c99izleGsu25L8t2574k25v3BvqwYN5Lsu68N2JfzM99SbY39wX1OAV/7ovOCb1jbzHnj2Oy7WMvzDvHnpsNjr2sH91jL8p6x97Q9rWPvSjrHXtRdvCcL1UW8fKpte1GfbKf2Hr8PODdA/9nCQNvzWtl70Md8J4xMr8Z9TNbHhZ8/6nA8c39vRn31rztyoqX610I7BXkdgfuBB7bPP5XBl5eR305843UQSHKbA6cBWxF/Y39qcBzk/xhzT77GvAe4B1DNaBOKO3v3zxUL5yXSCfZ11LfL2xjngfUA/b1XpZ69f8cYNPm8XL6L4/ubt/9qS9Bvhf1vdwfSLLLqCdyRn3F3w/n+p1s33uAVwz04Z3AM5v7zwLOTLIPpb6s8zzgdcDPs+NioH7uMRTUL8r26jd0fDo1nM8O1dDZvqx+3WxWv2j7vPp1283q182m9Wsyc2PVHtRXlc59fYfucwxnXOvus4Hs+/DHAS+7DnA88MIguxf1swrmXn59e9Yu9e2LBmxA/c3b0QP52+dq0TxXvj6iz1+aq03S7mdYMTa/ita85WQfT/0ch3OBf6K+x783d1BfGt6rnZft/Kz52o3IztduRHZB7YI+b+vVL2rbq1+S7dVuRJ/na5e026tdku3VrvWzenM5wdjpZVuZs1n41q4su2DszLLBuOn12R07g6w7dgZZd+wc2L4FY2fQrjt2Btne2Emwtkpql67FWPgWk6HsfP2Gsu36Bdljvdol29erXZKNaje0ffP1S9ru1S/JhnMfnfVuVD8vGx17A9lo3RKuu+kff/NZhtct3e3L1i3dbLZuibbPW7d0283WLd2sWzuCcxD8uS89X2HheDWUbc99Q9n5uS/IPh9/3ou2zZv3oqy7ZhnR5/bcF7XtzX1R1p37cM4JicfO8PyR/ryXZb1zvvTclNax52TfSzzvedsXzXteNho7s+3rznteu9G852UHzxmy2+jg2nhrnvindZ5Irxn4P0sYcSGqOcBOA16+yD4dTfC+bODN1Cv3y6lXge8ATl5E20uTth8ALG893hP4wkB7+wOnD2QOAt7fevx8mgX4iP6+CXjJUA2oHwy4TXN/G+DyoXox8kIU9X3O3wI2Gvs8AB7Y6d98Fng49dULy5vbndTfDDxgZNvdbe8+/jLwxNbjq4Ctku1bF/gJ9W1P2c+5hRULCwNuHdnfXYBzs+Miqp+XjeoXZZP6pcdnu4bdbFbDEe0uidrN6pdsX69+Qbtu/Ub0d0H9Ot87mrq4uJEVC6QFY2on+8rW4+UEn9vSzlIntVNpPr9gqN3ma3vhf5bP0U17N7RqdxetBelAu3t77bbz1A+M3LG1n28Z2L4tqSer7h9waO3jqzrPze+N7POTqb9t9OaOj3q1C7Int9qcr12W7dZuqN1u7YL8zV79Rra9N/Vilpv1ajewfQtqF2S/4NVuZH+fDJwSPC+WUp9v4dzXzbYen43zOTXdLMHYGbXb2r5oLlgK/CMD81/S9hKv7da+COe+YPvcuc9pN5z7Bvq7C/Wkyl1bRbWL8l79smy3fkPttusXZL8S1O7hI9pdkrT7hah2A9u3oH5J2736jdwX3bXLgvVuVD8vmx17XrZbu6F8dPy1swysPQfaXRK12zzO1p3e9kXrzm672boz6+987fDPQd6DP/el5yssnPvCLP25b/A8iGbuC7I/xJ/3xrS7d9Lu8QRrloHt68590T725r4xfXbnPppzQsbNewvOH8nnvfksA/Oe17Z37HWyRzJu3vPaXZK0+xLGzXvt7Rua9+baHTPvef0Nzxmi2z39rXnfBnY2sx3NbH3qyzw/d3cbNTOjvvf80lLKcQPZrcxss+b+hsAfUweGnlLKa0op25dSljR9PauU8tyk7Y3N7L5z96kH97Kg7RuAa8zsIc2X9qUOGpmDGX7p8Y+Ax5nZRs1+2Zf6WTRRn7du/n0g9er8xwbah1qzFzT3XwB8dsT/GWRmT6G+deJppZQ7BrI7tx7uT1zDi0spW5dSljR1vJb6gdE3JG1v03r4dIIaNk6lfngdZrYLK35rFHkScFkp5dokA/X9wU9o7u9D/WsVUX/nangv4B+oH+CXHRe9+i3yGHKzUf2SfK+GXjaqIXVw99rt1S/Zvqh+0b5YUL+k3V79kv0Q1c8bqy6lvhLnwOa/z9Vv9LgWZc3scOBPgINL8/kFQfZyM9uptf1Pa/6/lz2vlPKAVu3uKKXslPRhm1a7B9Ace8n2zdev2d9XDOyLA6kXXn45sI83bZ4PzH0t6fNc/TYA/o764bXe3PEcr3aLmWeirFc7Lws8z6td0vbmXv2SfvTql2xfr3YD+2JB7YLt29+rXdLfXu2ax9Fc7o2do+f9KOuNnUnWnfuC/LeDsfO2oG1v7Iy2zxs770j2RXfsjNr1xs5oX/TGzmRt5a5bFrMWi7Je/ZJsr35B9vxg7XJx0G6vdsm2ufPewL5YUL8k26tfsi/cua/RXe9m684xa2M3O2Ld2c1na8/57Ii1Z7fdbN3Z3b5s3enti2jd2c1m685uf6Paeecg38OZ+4JsdL7iZr25L8l6c5+XPc6b95J2vXVLtG29eS/rc/O9BXNfso97ApFKmgAABgdJREFUc1/S52ju884J3WMvyLq8bHbsBflo7utmPxQde0G77rEXbJ977CX7onfsBVn32Av6m42bw8ZesVpbb9T3OF5BvYr42oHsx6nvaf1N80Q6LMjtQf38gLk/6zj/5xed7G7UP998UfNk6731I/h/ezPw1jzqXwO8kBV/3nto+x5J/TONFzVP7s2T7MbUK+Kbjujr66kH6DLqX3zYIMl+nTqAXQjsO6YG1M86+UpzoJwJ3C/JPr25/yvqhYPTkuyV1M8Qm6vhCUn20832XUT985XbRdnO9ixn4cujvbY/Qv2zmBdRB+Btkuz61N/uL6P+KdR9sn5Q/4rFX4/Yx3tQX3Z5IfUlpn+YZI+kHlNXUD8PZe6quntcePVLsr36JdmoflG+V8Mo69UwabdXvyTbq1/Wh279knZ79UuyUf3csYo6zpzb7O9/p74cPMoe0dTvTupEd2KSvZM6Ls/17WgvS3358jebfbyM+kqfTaJ2O7W7fWDbzmq1ezIr/kpdlN+M+lv4i6m/VXtE1g/qb+meMjQfUJ/3Fzf1O7vZ51H2WOqC73I6fzq4O3d4tUuyvdol2V7tvGxUu7HzHf5bK9v9cOsXZHu1y/rQrV3Sbq92SdatHcFcjj92Rllv7IyyvbEzyUZz3+D6gxVjZ9S2N3ZGWW/sDPtAf+yM2vXGzigbjZ29tZVXu1ZfvHy0dvGy0dznZaP6petBFr46xGs3Wrd4WXfdkvWjW7+k7Wjt4mWj+vXWu1H9gmxUOy/r1i7JR/VL1+id+nntRvXzstG60+1DUDuv3ah2XtatXfO93jkIwdwXZN25L8i6c1+Qdec+LxvNe0G70brFy4bzXtQPnLkvaNud+4JsNPf1zgmJjz0vGx17XjY79rx8dOwNnccuZ8Wx57UbHXteNjr23D7gH3teu9Gx52XDY2/MbW6QFRERERERERERWaXu6W/NExERERERERGRGdGFKBERERERERERmQldiBIRERERERERkZnQhSgREREREREREZkJXYgSEREREREREZGZ0IUoERERkTWUmRUz22l190NERERkLF2IEhEREVlJzGy5mf2fmd3eur17dfdLREREZCrWXd0dEBEREVnL/Hkp5czV3QkRERGRKdIrokRERERWMTM7xMy+aWbvNrNbzOwyM9u39f1tzexzZnaTmV1pZi9qfW8dM/t7M7vKzG4zs/PMbIdW808ys++b2S/M7N/MzJr/t5OZfbX5eTea2SdnuMkiIiIiLr0iSkRERGQ2Hgt8CtgSeAbwGTPbsZRyE/AJYBmwLfBQ4Awzu6qUchbwcuBgYD/gCmA34I5Wu08FHg1sApwHfB74MnAMcDrwRGB9YPdVvYEiIiIiQ6yUsrr7ICIiIrJWMLPl1AtNd7a+/CrgN8CbgO1Ks/gys3OBdwFnA8uBzUoptzXfezOwTSnlEDO7HDiqlPJZ5+cVYM9Syjeax6cA55dS3mJmHwZ+CbyhlHLtKthcERERkUXTW/NEREREVq4DSimbtW7va75+XVn4G8Crqa+A2ha4ae4iVOt72zX3dwCuSn7eDa37dwD3ae4fBRhwrpldYmaH/o7bIyIiIrLS6EKUiIiIyGxsN/f5TY0HAtc3t/uZ2X0737uuuX8N8ODF/rBSyg2llBeVUrYF/go43sx2+t26LiIiIrJy6EKUiIiIyGxsDRxhZuuZ2UHArsAXSynXAP8NvNnM7m1muwGHASc3/+9E4Bgz29mq3cxsi6EfZmYHmdn2zcObgQLctbI3SkRERGQx9GHlIiIiIivX583st63HZwCfBc4BdgZuBH4CHFhK+XmTORg4gfrqqJuB15VSzmy+dxywAfWDx7cELgOePqIfjwbeYWabNj/vyFLKD+7OhomIiIjcXfqwchEREZFVzMwOAQ4vpeyxuvsiIiIisjrprXkiIiIiIiIiIjITuhAlIiIiIiIiIiIzobfmiYiIiIiIiIjITOgVUSIiIiIiIiIiMhO6ECUiIiIiIiIiIjOhC1EiIiIiIiIiIjITuhAlIiIiIiIiIiIzoQtRIiIiIiIiIiIyE/8PNRi49G6D8RwAAAAASUVORK5CYII=\n", "text/plain": [ "
      " ] @@ -286,7 +267,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -298,7 +279,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -307,13 +288,13 @@ "Text(0.5,0,'Epochs')" ] }, - "execution_count": 34, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABJwAAALMCAYAAAChcKgRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xt4ldWd//33SnYISTgICagUKWBQBILBAmqhWusU8dD0YMcjrR11hjr20R8dKfKbGW3n96j9XXWm1kfGVltbO7WFuZx26LRAKVQraC2KxhPQcqxyaIUAARICOaznj+xsAiQQSMhpv1/Xlct73/faa617y1+fa63vCjFGJEmSJEmSpLaS0dETkCRJkiRJUvdi4CRJkiRJkqQ2ZeAkSZIkSZKkNmXgJEmSJEmSpDZl4CRJkiRJkqQ2ZeAkSZIkSZKkNmXgJEmSOp0QQmYIYV8IYUhbtlXnFkK4PYTwfEfPQ5IktZ6BkyRJarVk4NPwVxdC2N/o880n2l+MsTbG2CvG+G5btj1RIYT/N4Twg7but4VjZ4QQZoUQ1iV/zz+FEB4IIfRop/H/Kvn/ct8RfxPaY3xJktS1JTp6ApIkqeuLMfZquA4hbAJujzEuaa59CCERY6xpj7l1YXOAjwE3AyuBkcAPgPOAz7TlQMf4//FujHFoW44lSZLSgyucJEnSKZdcKTQvhPCTEMJeYFoI4eIQwsshhN0hhG0hhEdDCFnJ9okQQgwhDE1+/lHy+cIQwt4Qwu9CCMNOtG3y+ZUhhD+GEMpDCP9fCOHFEMIXTuKdRocQfpuc/1shhKsbPbsmhLA6Of7mEMKM5P2BIYQFye/sDCG80EzfI4HpwI0xxt/HGGtijG8DnwWuCSFcEkKYHELYEkLIaPS9vw4hvJa8zggh/O8QwvoQwo4QwtwQQr/ks8Lkb/Y3IYR3gcUn8f7LkyuuXk3+lj9r6D/5/FMhhHeS7/qbEMK5jZ59MITw3yGE7cm5fevwrsM3k9/bEEKY0ujBbSGETcnfdUMI4YYTnbckSWofBk6SJKm9fBr4MdAXmAfUAHcDBcAkYCr1IUtzbgL+GegPvAv8nxNtG0IYCPwnMDM57kZg4om+SHJb2y+AXwIDgBnAvBBCYbLJ94HbYoy9gbHAb5P3ZwIbkt85IznHpvwVsDHG+FrjmzHGTcArwMeBF4Fq4NIj3vvHyesvA1cDlwCDgQrg0SPGuYT6lVNXc3I+n/wbBATgmwAhhPOAHwH/D/XvugT4nxBCVgghQf3vtg4YCpxF/f+TBh8G3gLyk/19L9lnH+DfgI8nf9dJwJsnOW9JknSKGThJkqT2sjzG+D8xxroY4/4Y4yuNVu9sAJ7g8PDkSM/GGF+NMVYDzwDFJ9H2GqA0xjg/+eybwI6TeJdJQA/gGzHG6uT2wYVAw4qbamBUCKF3jHFno+ComvpwZkiM8WCM8bdH9VyvANjWzLNtQEGMMQJzgRsBQginAVck70F9ePe/Y4xbYoxVwFeB6xqviALujzFWxhj3NzPWkORKo8Z/2Y2ePx1jXBVjrADuA24IIYTk7/DzGONvkr/z14E+wIXAxcn3mxVjrEj+W3ixUZ/rY4xPxRhrgaeBwSGEguSzCIwJIfSMMW6LMa5qZt6SJKmDGThJkqT28l7jDyGEkSGEX4YQ/hxC2AP8C/VBRHP+3Oi6EujVXMNjtB3UeB7J0GZzC+Z+pEHU1zeKje79CfhA8vrTQAnwbgjh+RDChcn7X0+2W5rc6jazmf53AGc28+xMDoVkPwauTW5FvBb4fYyx4X2GUL+qaHcIYTf1q4YiMLBRX4f9P2nCuzHG0474O9DM9/8EZFO/qmxQ8jMAMcY66n/nD1C/omlTMlBqypH/7wB6xRj3UB+u3Qn8OYTwixDCOceZvyRJ6iAGTpIkqb3EIz5/B3gbKIwx9qF+hUw4xXPYRv32MqC+WBCHQqITsRU4K/n9BkOALQDJlVsl1Ic7vyC56ijGuCfGOCNZiPtTwKwQQlOrupYCw0IIFzS+maxTNSH5nBjjm8l3uoLDt9NBfcDz8SPCop4xxlSgc0RgdjLOanQ9BDgA7KT+9/lgo3lnUP+7b6E+pPpgCCHzRAeLMS6MMf4V9aHbOur/DUmSpE7IwEmSJHWU3kA5UJGs+XOs+k1t5RfABSGETyRrCd1NfY2hY8kMIfRs9JcNvER9Dap/SNYl+hhwFfCfIYScEMJNIYQ+ye1ke4FagOS4ZyeDqvLk/aNW+sQYVwPfBX4SQpgYQsgMIYwBngUWxhifb9T8J9TXkLo4+bzBt4EHQwhDkmMPDCGUnMBv1RKfT65UywO+BvxnMsT6T6AkhPDR5OqrmdT/Dr8HfgeUJeeWm/y9Jh1voBDCmcnfLxc4SH1NquZWSUmSpA5m4CRJkjrKPwC3UB9EfIf6QuKnVIzxL8D11BefLgPOBl6nfmVOc6YB+xv9/SG5rewTwCep3972KHBTjPGPye/cAvwpuVXwNuBzyfvnAr8B9lFf9PtbMcblzYx7B/U1jH5CfbiyEPg1cN0R7X4MfAz4dYxxV6P7/wYson773l7qQ7IJx3jPpgwJIew74u9TjZ7/B/XFwbcBmcD/AogxvpP8DR4HtlNfEL4kWe+qhvpaWudRv9rpXepP3zueTOqDq23U/7/7MPClE3wfSZLUTkLrV1JLkiR1TcltXVuBz8YYl3X0fLqSEMJy4Lsxxh909FwkSVLn4wonSZKUVkIIU0MIfZNb4/6Z+q1xKzp4WpIkSd2KgZMkSUo3k4EN1G+Fmwp86oiT1yRJktRKbqmTJEmSJElSm3KFkyRJkiRJktpUoqMncCoUFBTEoUOHdvQ0JEmSJEmSuo2VK1fuiDEOaEnbbhk4DR06lFdffbWjpyFJkiRJktRthBD+1NK2bqmTJEmSJElSmzJwkiRJkiRJUpsycJIkSZIkSVKb6pY1nCRJkiRJ0omrrq5m8+bNVFVVdfRU1IF69uzJ4MGDycrKOuk+DJwkSZIkSRIAmzdvpnfv3gwdOpQQQkdPRx0gxkhZWRmbN29m2LBhJ92PW+okSZIkSRIAVVVV5OfnGzalsRAC+fn5rV7lZuAkSZIkSZJSDJvUFv8GDJwkSZIkSZLUpqzhJEmSJEmSmlS0/G22V9e0WX8DshK8NXlMm/WnzssVTpIkSZIkqUltGTadiv46uxgjd911F4WFhYwdO5bXXnutyXYrV66kqKiIwsJC7rrrLmKMAMycOZORI0cyduxYPv3pT7N7924ANm3aRE5ODsXFxRQXF/PFL36x3d6ppQycJEmSJElSl1ZbW9vRU2jSwoULWbt2LWvXruWJJ57gjjvuaLLdHXfcwRNPPJFqu2jRIgA+/vGP8/bbb/Pmm29yzjnn8NBDD6W+c/bZZ1NaWkppaSnf/va32+V9ToSBkyRJkiRJ6hQqKiq4+uqrOf/88xkzZgxPP/001113Xer5888/zyc+8QkAevXqxX333ceFF17I7373OxYsWMDIkSOZPHkyd911F9dcc02z43z1q1/llltuYcqUKQwdOpSf/vSnfOUrX6GoqIipU6dSXV0NwL333suoUaMYO3Ys99xzDwDbt2/n2muvZcKECUyYMIEXX3yx2XHmz5/P5z//eUIIXHTRRezevZtt27Yd1mbbtm3s2bOHiy++mBACn//85/nv//5vAKZMmUIiUV8N6aKLLmLz5s0n8at2DAMnSZIkSZLUKSxatIhBgwbxxhtv8Pbbb/OpT32Kl19+mYqKCgDmzZvH9ddfD9SHU2PGjOH3v/8948ePZ/r06SxcuJDly5ezffv24461fv16fvnLXzJ//nymTZvGZZddxltvvUVOTg6//OUv2blzJz/72c945513ePPNN/mnf/onAO6++25mzJjBK6+8wn/9139x++23NzvGli1bOOuss1KfBw8ezJYtW45qM3jw4GO2AXjqqae48sorU583btzIuHHjuPTSS1m2bNlx37e9GThJkiRJkqROoaioiCVLljBr1iyWLVtG3759mTp1Kv/zP/9DTU0Nv/zlL/nkJz8JQGZmJtdeey0Aa9asYfjw4QwbNgyAG2+88bhjXXnllWRlZVFUVERtbS1Tp05NzWHTpk306dOHnj17cvvtt/PTn/6U3NxcAJYsWcKXvvQliouLKSkpYc+ePezdu7fJMRpqMTUWQjjhNg888ACJRIKbb74ZgDPPPJN3332X119/nX/7t3/jpptuYs+ePcd95/bkKXWSJEmSJKlTOOecc1i5ciULFixg9uzZTJkyheuvv545c+bQv39/JkyYQO/evQHo2bMnmZmZQNOhzfFkZ2cDkJGRQVZWVirkycjIoKamhkQiwYoVK1i6dClz587lscce4ze/+Q11dXX87ne/Iycn57hjDB48mPfeey/1efPmzQwaNOioNo23yh3Z5umnn+YXv/gFS5cuTc0xOzs7Nf8PfehDnH322fzxj39k/PjxJ/w7nCqucJIkSZIkSU0akNW261SO19/WrVvJzc1l2rRp3HPPPbz22mt89KMf5bXXXuPJJ59Mbac70siRI9mwYQObNm0C6rfetda+ffsoLy/nqquu4pFHHqG0tBSor6v02GOPpdo13G9KSUkJP/zhD4kx8vLLL9O3b1/OPPPMw9qceeaZ9O7dm5dffpkYIz/84Q9Tq7gWLVrE//2//5ef//znqRVWUF9HqqFQ+oYNG1i7di3Dhw9v9Tu3JVc4SZIkSZKkJr01eUz7jvfWW8ycOTO16ujxxx8nMzOTa665hh/84Ac8/fTTTX4vJyeHf//3f2fq1KkUFBQwceLEVs9l7969fPKTn6SqqooYI9/85jcBePTRR7nzzjsZO3YsNTU1XHLJJc2eEnfVVVexYMECCgsLyc3N5fvf/37qWXFxcSqsevzxx/nCF77A/v37ufLKK1O1mr70pS9x4MABPv7xjwP1hcO//e1v88ILL3DfffeRSCTIzMzk29/+Nv3792/1O7elcDLLzjq78ePHx1dffbWjpyFJkiRJUpeyevVqzjvvvI6exknZt28fvXr1IsbInXfeyYgRI5gxY0ZHT6vLaurfQghhZYyxRfv23FInSZIkSZK6vCeffJLi4mJGjx5NeXk506dP7+gppTW31EmSJEmSpC5vxowZR61o+v73v8+3vvWtw+5NmjSJOXPmtOnY7TVOV+KWOkmSJEmSBHTtLXVqW26pkyRJkiRJUqdi4CRJkiRJkqQ2ZeAkSZIkSZKkNmXRcEmSJEmS1KRlyy/k4MEdbdZfjx4FfGTy79usP3VernCSJEmSJElNasuw6VT019jQoUPZsePU9d9SvXr1OqH2ixYt4txzz6WwsJCvf/3rTbY5cOAA119/PYWFhVx44YVs2rQp9eyhhx6isLCQc889l1/96lep+7feeisDBw5kzJgxJ/UerWXgJEmSJEmSdArV1tY2e//OO+9k4cKFrFq1ip/85CesWrXqqHbf+9736NevH+vWrWPGjBnMmjULgFWrVjF37lzeeecdFi1axN///d+nxvrCF77AokWLTt1LHUe3D5wqKzex5g/38dJLl/HiS5eyevVs9lWs7ehpSZIkSZKkI1RUVHD11Vdz/vnnM2bMGObNm8eCBQsYOXIkkydP5q677uKaa64BoKysjClTpjBu3DimT59OjLHZfjdt2sTIkSO5/fbbGTNmDDfffDNLlixh0qRJjBgxghUrVgDw29/+luLiYoqLixk3bhx79+4F4Bvf+AYTJkxg7Nix3H///S16l+eff57LLruMm266iaKioibbrFixgsLCQoYPH06PHj244YYbmD9//lHt5s+fzy233ALAZz/7WZYuXUqMkfnz53PDDTeQnZ3NsGHDKCwsTL3LJZdcQv/+/Vs011OhWwdO5eWvs+KVErZseYb9Ve9SVbWZrdv+k1de+SQ7d77Y0dOTJEmSJEmNLFq0iEGDBvHGG2/w9ttvM3XqVKZPn87ChQtZvnw527dvT7X92te+xuTJk3n99dcpKSnh3XffPWbf69at4+677+bNN99kzZo1/PjHP2b58uU8/PDDPPjggwA8/PDDzJkzh9LSUpYtW0ZOTg6LFy9m7dq1rFixgtLSUlauXMkLL7zQovdZsWIFDzzwQJOrlgC2bNnCWWedlfo8ePBgtmzZcsx2iUSCvn37UlZW1uLvd4RuGzjFWMeq1TOpra046lld3QFWrZpJXV11B8xMkiRJkiQ1paioiCVLljBr1iyWLVvGxo0bGT58OMOGDQPgxhtvTLV94YUXmDZtGgBXX301/fr1O2bfw4YNo6ioiIyMDEaPHs3ll19OCIGioqJUTaRJkybx5S9/mUcffZTdu3eTSCRYvHgxixcvZty4cVxwwQWsWbOGtWtbtnNq4sSJqbk3palVWSGEFrdr6fc7QrcNnHaXr6SycmOzzw8c/As7dy5rxxlJkiRJkqRjOeecc1i5ciVFRUXMnj27ye1ljZ1IuJKdnZ26zsjISH3OyMigpqYGgHvvvZfvfve77N+/n4suuog1a9YQY2T27NmUlpZSWlrKunXruO2221o0Zl5e3jGfDx48mPfeey/1efPmzQwaNOiY7WpqaigvL6d///4t/n5H6LaB04GqbcdtU1W1tR1mIkmSJElS19SjR0G79rd161Zyc3OZNm0a99xzDy+99BIbNmxIrUCaN29equ0ll1zCM888A8DChQvZtWtXq+e3fv16ioqKmDVrFuPHj2fNmjVcccUVPPXUU+zbtw+o3972/vvvt3osgAkTJrB27Vo2btzIwYMHmTt3LiUlJUe1Kykp4emnnwbg2Wef5WMf+xghBEpKSpg7dy4HDhxg48aNrF27lokTJ7bJ3For0dETOFWye5553DY9e3aO1E+SJEmSpM7oI5N/367jvfXWW8ycOZOMjAyysrJ4/PHH2bZtG1OnTqWgoOCwMOX+++/nxhtv5IILLuDSSy9lyJAhrR7/kUce4bnnniMzM5NRo0Zx5ZVXkp2dzerVq7n44osB6NWrFz/60Y8YOHBgq8dLJBI89thjXHHFFdTW1nLrrbcyevRoAO677z7Gjx9PSUkJt912G5/73OcoLCykf//+zJ07F4DRo0dz3XXXMWrUKBKJBHPmzCEzMxOo3374/PPPs2PHDgYPHszXvva1Fq/MagvhWFXcu6rx48fHV15Zwcu/v4LKyg1NtsnOPoMPX/w8GRlZ7Tw7SZIkSZI6p9WrV3Peeed19DQOs2/fPnr16kWMkTvvvJMRI0YwY8aMjp5Wt9fUv4UQwsoY4/iWfL/bbqkLIYNR532DzMxeRz3LyOjJqFEPGzZJkiRJktTJPfnkkxQXFzN69GjKy8uZPn16R09JLdBtt9QB9O1bzIUT/4d333uKrVufpa5uPwAfumAeffqM6eDZSZIkSZKk45kxY0aLVzSVlZVx+eWXH3V/6dKl5Ofnt9mcTnSc9ppXZ9KtAyeAnJwhnHvOV8nMyOFP7z6RvNv9thFKkiRJkpTu8vPzKS0t7XTjtNe8OpNuu6XuSLl5Z6euKyrXd+BMJEmSJEmSure0CZzycgtT15UV6zpwJpIkSZIkSd1b+gROrnCSJEmSJElqF92+hlODRKI32T1O58DBv1BRYeAkSZIkSdLx/HHyR6jdsaPN+sssKOCc5cvarD91XmmzwgkO1XHav/9P1NVVd/BsJEmSJEnq3NoybDoV/TU2dOhQdpzC/luqV69eJ9R+0aJFnHvuuRQWFvL1r3+9yTYHDhzg+uuvp7CwkAsvvJBNmzalnj300EMUFhZy7rnn8qtf/QqA9957j8suu4zzzjuP0aNH861vfSvV/qtf/Sof+MAHKC4upri4mAULFpz4S7ZAWgVODdvqYqxh//4/dfBsJEmSJElSOqitrW32/p133snChQtZtWoVP/nJT1i1atVR7b73ve/Rr18/1q1bx4wZM5g1axYAq1atYu7cubzzzjssWrSIv//7v6e2tpZEIsG//uu/snr1al5++WXmzJlzWL8zZsygtLSU0tJSrrrqqlPyzukVODUqHO62OkmSJEmSOpeKigquvvpqzj//fMaMGcO8efNYsGABI0eOZPLkydx1111cc801AJSVlTFlyhTGjRvH9OnTiTE22++mTZsYOXIkt99+O2PGjOHmm29myZIlTJo0iREjRrBixQoAfvvb36ZW/owbN469e/cC8I1vfIMJEyYwduxY7r///ha9y/PPP89ll13GTTfdRFFRUZNtVqxYQWFhIcOHD6dHjx7ccMMNzJ8//6h28+fP55ZbbgHgs5/9LEuXLiXGyPz587nhhhvIzs5m2LBhFBYWsmLFCs4880wuuOACAHr37s15553Hli1bWjTvtpJWgVPuYYXDPalOkiRJkqTOZNGiRQwaNIg33niDt99+m6lTpzJ9+nQWLlzI8uXL2b59e6rt1772NSZPnszrr79OSUkJ77777jH7XrduHXfffTdvvvkma9as4cc//jHLly/n4Ycf5sEHHwTg4YcfZs6cOZSWlrJs2TJycnJYvHgxa9euZcWKFZSWlrJy5UpeeOGFFr3PihUreOCBB5pctQSwZcsWzjrrrNTnwYMHNxkMNW6XSCTo27cvZWVlLfr+pk2beP3117nwwgtT9x577DHGjh3Lrbfeyq5du1r0LicqrQKnw1c4GThJkiRJktSZFBUVsWTJEmbNmsWyZcvYuHEjw4cPZ9iwYQDceOONqbYvvPAC06ZNA+Dqq6+mX79+x+x72LBhFBUVkZGRwejRo7n88ssJIVBUVJSqiTRp0iS+/OUv8+ijj7J7924SiQSLFy9m8eLFjBs3jgsuuIA1a9awdu3aFr3PxIkTU3NvSlOrskIILW53vO/v27ePa6+9lkceeYQ+ffoAcMcdd7B+/XpKS0s588wz+Yd/+IcWvcuJSqvAqUePAhKJ+h+40i11kiRJkiR1Kueccw4rV66kqKiI2bNnN7m9rLGmwpnmZGdnp64zMjJSnzMyMqipqQHg3nvv5bvf/S779+/noosuYs2aNcQYmT17dqrm0bp167jttttaNGZeXt4xnw8ePJj33nsv9Xnz5s0MGjTomO1qamooLy+nf//+x/x+dXU11157LTfffDOf+cxnUm1OP/10MjMzycjI4G//9m9T2wnbWloFTiEE8nLrt9VVVK4nxroOnpEkSZIkSZ1XZkFBu/a3detWcnNzmTZtGvfccw8vvfQSGzZsSK1AmjdvXqrtJZdcwjPPPAPAwoUL22Rr2Pr16ykqKmLWrFmMHz+eNWvWcMUVV/DUU0+xb98+oH572/vvv9/qsQAmTJjA2rVr2bhxIwcPHmTu3LmUlJQc1a6kpISnn34agGeffZaPfexjhBAoKSlh7ty5HDhwgI0bN7J27VomTpxIjJHbbruN8847jy9/+cuH9bVt27bU9c9+9jPGjBnTJu9ypMQp6bUTy80rpHzP69TVVVFVtZWcnMEdPSVJkiRJkjqlc5Yva9fx3nrrLWbOnElGRgZZWVk8/vjjbNu2jalTp1JQUMDEiRNTbe+//35uvPFGLrjgAi699FKGDBnS6vEfeeQRnnvuOTIzMxk1ahRXXnkl2dnZrF69mosvvhiAXr168aMf/YiBAwe2erxEIsFjjz3GFVdcQW1tLbfeeiujR48G4L777mP8+PGUlJRw22238bnPfY7CwkL69+/P3LlzARg9ejTXXXcdo0aNIpFIMGfOHDIzM1m+fDn/8R//QVFREcXFxQA8+OCDXHXVVXzlK1+htLSUEAJDhw7lO9/5TqvfoynhWFXcu6rx48fHV199tclnf3r3Sdat+zoA55//PQryP9qOM5MkSZIkqfNavXo15513XkdP4zD79u2jV69exBi58847GTFiBDNmzOjoaXV7Tf1bCCGsjDGOb8n302pLHRxeONw6TpIkSZIkdW5PPvkkxcXFjB49mvLycqZPn97RU1ILpN2Wurw8T6qTJEmSJKmrmDFjRotXNJWVlXH55ZcfdX/p0qXk5+e32ZxOdJz2mldnknaBU8+eHyAjoyd1dVVUVLrCSZIkSZKk7iI/P5/S0tJON057zaszSbstdSFkkJs7HICKivV0xxpWkiRJkiRJHSntAieAvLyzAaip2U11dVkHz0aSJEmSJKl7Sc/AKffs1HWFhcMlSZIkSZLaVNrVcALIbVw4vHI9/fpd2IGzkSRJkiSpc/rGN75BRUVFm/WXl5fHzJkz26w/dV6ucPKkOkmSJEmSmtSWYdOp6K+xoUOHsmPHjlPWf0v16tXrhNovWrSIc889l8LCQr7+9a832ebAgQNcf/31FBYWcuGFF7Jp06bUs4ceeojCwkLOPfdcfvWrX6XuDx06lKKiIoqLixk/fvxJvUtrpOcKp9yhhJBJjLVUuqVOkiRJkiSdQrW1tWRmZjZ5/8477+TXv/41gwcPZsKECZSUlDBq1KjD2n3ve9+jX79+rFu3jrlz5zJr1izmzZvHqlWrmDt3Lu+88w5bt27lr/7qr/jjH/+YGuu5556joKCgXd7xSGm5wikjowc5OUMAqKh0hZMkSZIkSZ1BRUUFV199Neeffz5jxoxh3rx5LFiwgJEjRzJ58mTuuusurrnmGgDKysqYMmUK48aNY/r06cc8hX7Tpk2MHDmS22+/nTFjxnDzzTezZMkSJk2axIgRI1ixYgUAv/3tbykuLqa4uJhx48axd+9eoH5r4YQJExg7diz3339/i97l+eef57LLLuOmm26iqKioyTYrVqygsLCQ4cOH06NHD2644Qbmz59/VLv58+dzyy23APDZz36WpUuXEmNk/vz53HDDDWRnZzNs2DAKCwtT79LR2i1wCiFMDSH8IYSwLoRwbxPPPxhCWBpCeDOE8HwIYXCjZ7UhhNLk38/bYj65yW11Bw78mZqavW3RpSRJkiRJaoVFixYxaNAg3njjDd5++22mTp3K9OnTWbhwIcuXL2f79u2ptl/72teYPHkyr7/+OiUlJbz77rvH7HvdunXcfffdvPnmm6xZs4Yf//jHLF++nIcffpgHH3wQgIcffpg5c+ZQWlrKsmXLyMnJYfHixaxdu5YVK1ZQWlrKypUreeGFF1r0PitWrOCBBx5g1apVTT7fsmV2NP+DAAAgAElEQVQLZ511Vurz4MGD2bJlyzHbJRIJ+vbtS1lZ2TG/H0JgypQpfOhDH+KJJ55o0XzbUrsETiGETGAOcCUwCrgxhDDqiGYPAz+MMY4F/gV4qNGz/THG4uRfSVvMKe+wwuEb2qJLSZIkSZLUCkVFRSxZsoRZs2axbNkyNm7cyPDhwxk2bBgAN954Y6rtCy+8wLRp0wC4+uqr6dev3zH7HjZsGEVFRWRkZDB69Gguv/xyQggUFRWlaiJNmjSJL3/5yzz66KPs3r2bRCLB4sWLWbx4MePGjeOCCy5gzZo1rF27tkXvM3HixNTcm9LUqqwQQovbHev7L774Iq+99hoLFy5kzpw5LQ7J2kp7rXCaCKyLMW6IMR4E5gKfPKLNKGBp8vq5Jp63qcaFwystHC5JkiRJUoc755xzWLlyJUVFRcyePbvJ7WWNNRXONCc7Ozt1nZGRkfqckZFBTU0NAPfeey/f/e532b9/PxdddBFr1qwhxsjs2bMpLS2ltLSUdevWcdttt7VozLy8vGM+Hzx4MO+9917q8+bNmxk0aNAx29XU1FBeXk7//v2P+f2G/w4cOJBPf/rT7b7Vrr0Cpw8A7zX6vDl5r7E3gGuT158GeocQ8pOfe4YQXg0hvBxC+FRTA4QQ/i7Z5tXGS+yac9gKJwuHS5IkSZJ0lOMFJm3d39atW8nNzWXatGncc889vPTSS2zYsCG1AmnevHmptpdccgnPPPMMAAsXLmTXrl2tnt/69espKipi1qxZjB8/njVr1nDFFVfw1FNPsW/fPqB+e9v777/f6rEAJkyYwNq1a9m4cSMHDx5k7ty5lJQcvbGrpKSEp59+GoBnn32Wj33sY4QQKCkpYe7cuRw4cICNGzeydu1aJk6cSEVFRar+VEVFBYsXL2bMmDFtMueWaq9T6pqKHI9c93UP8FgI4QvAC8AWoCb5bEiMcWsIYTjwmxDCWzHGw1KiGOMTwBMA48ePb75SWFJu7vDUdUWlgZMkSZIkSUeaOXNmu4731ltvMXPmTDIyMsjKyuLxxx9n27ZtTJ06lYKCAiZOnJhqe//993PjjTdywQUXcOmllzJkyJBWj//II4/w3HPPkZmZyahRo7jyyivJzs5m9erVXHzxxQD06tWLH/3oRwwcOLDV4yUSCR577DGuuOIKamtrufXWWxk9ejQA9913H+PHj6ekpITbbruNz33ucxQWFtK/f3/mzp0LwOjRo7nuuusYNWoUiUSCOXPmkJmZyV/+8hc+/elPA/Urom666SamTp3a6vmeiHCsKu5tNkgIFwNfjTFekfw8GyDG+FAz7XsBa2KMg5t49gPgFzHGZ5sbb/z48fHVV1897ryWvziJAwf+TE7OUD588dLjtpckSZIkqTtbvXo15513XkdP4zD79u2jV69exBi58847GTFiBDNmzOjoaXV7Tf1bCCGsjDGOb8n322tL3SvAiBDCsBBCD+AG4LDT5kIIBSGEhvnMBp5K3u8XQshuaANMApou736C8nLrt9Xt3/8udXUH2qJLSZIkSZLUhp588kmKi4sZPXo05eXlTJ8+vaOnpBZoly11McaaEMKXgF8BmcBTMcZ3Qgj/ArwaY/w58FHgoRBCpH5L3Z3Jr58HfCeEUEd9QPb1GGPbBE55hezctRyoo7JyE716ndsW3UqSJEmSpDYyY8aMFq9oKisr4/LLLz/q/tKlS8nPz2/iGyfnRMdpr3l1Ju1Vw4kY4wJgwRH37mt0/Sxw1Da5GONLQNGpmFNu3qGT6ioq1xs4SZIkSZLSXozxhE5/60zy8/MpLS3tdOO017zaSluUX2qvLXWdUsOWOvCkOkmSJEmSevbsSVlZWZsEDuqaYoyUlZXRs2fPVvXTbiucOqO8RiucKivWdeBMJEmSJEnqeIMHD2bz5s1s3769o6eiDtSzZ08GDz7qHLcTktaBU48e+WRl9aO6ehcVla5wkiRJkiSlt6ysLIYNG9bR01A3kNZb6gByc+tXOVVWbiDG2g6ejSRJkiRJUteX9oFTw7a6uroDVFVt6eDZSJIkSZIkdX3dfktdjJH9paVUvPgS1NWRe9GF5E6YkKq4f2Th8JycIR01VUmSJEmSpG6hWwdOtfsq2PK//hcVy5cfuvnv/07O+A9x1mOPkXnaaYcVDq+oXEcBl3XATCVJkiRJkrqPbr2lbts//9PhYVPS/ldXsuWemQDkHrHCSZIkSZIkSa3TbQOng5s3s3fRr5p9XrF8OVV/+CM9e55JRkZO/b2Kde01PUmSJEmSpG6r2wZO+994A2I8dpvXXyeEDPLyhgNQWbmOeJzvSJIkSZIk6di6beAUevRocZuGwuE1NXs5eHD7KZ2XJEmSJElSd9dtA6e8iy4i5OQ03yCRoNclHwEgt3HhcLfVSZIkSZIktUq3DZwye/em4ItfbPZ5/t98gURBAXBohRNARaWFwyVJkiRJkloj0dETOJXy/+5vyeiZzfZvf4e6XbvqbyYyGfClL5H/d3+XapeXdyhwqvSkOkmSJEmSpFbptiucAEII9L/lFkb89nky+50GQPbwsyn44hcJGYdePSdnCCHUZ28VlW6pkyRJkiRJao1uHTg1yOjRg8SZZwJQs3Pn0c8zssjJGQpAhSucJEmSJEmSWiUtAieARH59vabanTuJtbVHPc9LFg4/ePB9amr2tuvcJEmSJEmSupP0CZySBcKpq6O2oZ5TI3m5nlQnSZIkSZLUFtIvcAJqduw46nnjwuFuq5MkSZIkSTp5aRQ45aeua7YfHTjl5jVa4WThcEmSJEmSpJOWNoFTZuMVTmVNrHDKHZ66rnSFkyRJkiRJ0klLm8ApUTAgdV3bxJa6zMxcevb8AOAKJ0mSJEmSpNZIo8Dp2Fvq4FDh8P37N1Nbe6Bd5iVJkiRJktTdpFHg1HhLXVmTbXJThcPrqNy/sR1mJUmSJEmS1P2kTeCU0acPISsLaPqUOji0wgmgomJtu8xLkiRJkiSpu0mbwCmEkCocXrNje5Nt8lIrnCwcLkmSJEmSdLLSJnCCQ9vqanc0vaUuL6/RCqdKAydJkiRJkqSTkZ6B0+7dxIMHj3qeldWPrKz+AOwpL6Wqamu7zk+SJEmSJKk7SLPAqdFJdTt3HvV8y5afUFOzB4CqA1t58aVLeOvtu6ip2dtuc5QkSZIkSerq0ipwymx8Ut0R2+q2bfspa/7wT8RY0+hu5P33f8mbb36RGGM7zVKSJEmSJKlrS6vAKXFY4HSocHiMtWzY+K1mv7dr98vs2v3yKZ2bJEmSJElSd5FmgdOA1HXtjh2p64rKDVRVbT7md8vKnj9V05IkSZIkSepW0ixwalTDqdGWuhhrj/vdlrSRJEmSJElS2gVOjbfUHVrhlJc7nKys/Ka+ktLvtAtP2bwkSZIkSZK6EwMnICOjBx/84N82+71eeeeSn3/ZKZ2bJEmSJElSd5FWgVNGXh4hJwc4vIYTwJCzbueDH/wiISQOu5+TM4zzi58iI+Pw+5IkSZIkSWpaWgVOcGiVU80RgVMIgcKzZzLpw8sYctZtqfsDB06lZ/YZ7TpHSZIkSZKkrszA6QjZ2QMZNuxuGn6avXvebK+pSZIkSZIkdQtpGDjVFwev27ePuqqqptsk8ujV6xwAyve8QYx17TY/SZIkSZKkri7tAqfMwwqHlzXbrk+f8wGord1HZeWGUz4vSZIkSZKk7iLtAqfGJ9XV7tjebLu+fYpT1+V7Sk/pnCRJkiRJkrqT9Auc8huvcGq6jhMcWuEEsGfPG6d0TpIkSZIkSd1J+gVOA1q2pS4vr5DMzDzAwEmSJEmSJOlEpF/gVNCyFU4hZNK79xgA9u37A7W1TRcYlyRJkiRJ0uHSLnDKPGxLXfM1nOBQHacYa9i79+1TOi9JkiRJkqTuIu0Cp0RBfuq6tqz5LXUAffpax0mSJEmSJOlEpV3glNGzJxm9ewNQs735LXVweOFwT6qTJEmSJElqmbQLnAAS+fWrnI5VwwmgZ/YZZGefAcCePW+e8nlJkiRJkiR1B+kZOCULh9eUlRFjPGbbhlVOVVWbOXjw2AGVJEmSJEmS0jRwyhxQHzjF/fupq6g8Ztu+h22rs46TJEmSJEnS8aRl4JRodFJd7XFOquuTPKkOYE+5dZwkSZIkSZKOJz0Dp4JDgVPNcU6q6917DA0/kyfVSZIkSZIkHV96Bk4DGgVOxzmpLpHIo1feCAD27H2TGOtO6dwkSZIkSZK6urQMnDKTp9TB8U+qg0OFw2tq9lJZufGUzUuSJEmSJKk7SMvAKVEwIHVdU9aCwKlvozpOe6zjJEmSJEmSdCzpGTgNaFw0/PiBU99GhcM9qU6SJEmSJOnY0jNw6t8/dX28Gk4AeXmFZGbmAq5wkiRJkiRJOp60DJxCVhaZp50GHP+UOoAQMunduwiAffv+QG1t1SmdnyRJkiRJUleWloETHNpW15Ki4QB9k4XDY6xh7753Ttm8JEmSJEmSurq0DZwy8+sDp9odO4gxHrd9n0Z1nPaUW8dJkiRJkiSpOWkbOCUK6gOnWF1N3Z49x23fp8/Y1HW5dZwkSZIkSZKalejoCZxqf/7zn3nxxRdZv349MUaGDRvGpEmTUoET1G+ry+zb95j99Ox5Jtk9TufAwb+wx5PqJEmSJEmSmtWtA6eNGzfyzDPPUFNTk7q3atUq1qxZw3VZWal7Ndt3kH322cftr0/f89m+fTFVVZs5eHAHPXoUHPc7kiRJkiRJ6abbbqmrq6tj/vz5h4VNjZ+9tmFj6nNNWcsKhx9Wx2nPm62fpCRJkiRJUjfUbQOnP/3pT+zevbvZ5+UZh1699gRPqgPrOEmSJEmSJDWn2wZOe/fuPebzAz2zU9c1LQycevceAwTAk+okSZIkSZKa020Dp379+h3z+f6cnNR1zY6yFvWZSPQiL28EAHv2vkGMdSc/QUmSJEmSpG6q2wZOgwcPZuDAgc0+zxk4EJLb6lq6wgmgb7KOU03NXiorN7VqjpIkSZIkSd1Rtw2cQgh85jOfITc396hn2dnZXPvXf01m//7AiQVOfRrVcdpjHSdJkiRJkqSjdNvACeCMM87gi1/8Ih/+8IdT9/r3788dd9zBWWedRaKgAGh50XCAPn0PnVRXvsc6TpIkSZIkSUfq1oETQJ8+fZgyZQqJRCL1+bTTTgNIBU41O3cSa2tb1F9ebiEZGfX1n1zhJEmSJEmSdLRuHzg1yEkWCd+/f3/qXiI/v/6itpba3btb1E9GRoI+fYoA2LdvDbW1VW07UUmSJEmSpC4u7QKnysrK1L3EgILUdUtPqoNDdZxirGHfvlVtNENJkiRJkqTuIe0Cp8YrnDILGgdO21vcV8NJdWAdJ0mSJEmSpCOlTeDUcFpdTU0N1dXVACTyDwVOJ1Q4vPFJdeXWcZIkSZIkSWosbQKnhhVOcGiV08luqcvOPoMePQYCrnCSJEmSJEk6UnoHTodtqWv5CqcQAn2Tq5yqqt7j4MGWh1WSJEmSJEndnYFT0okETgB9GtVx2rPnzVbOTpIkSZIkqftIy8Cp4aS6jL59ISsLgNqyEwyc+h6q41S+xzpOkiRJkiRJDdIycGpY4RRCIJGfD0DN9hMMnHqPAQIAu3a+RE3NvraZqCRJkiRJUheXNoFTwyl1cChwgkPb6k50S93+qi1kZPQEoHzPa7ywbCKr1/wjNTV722C2kiRJkiRJXVfaBE5NrXACUiucanfvJlZXt6ivior1vLbyBurqDvUT4wG2bp1LaenfUFd3sI1mLUmSJEmS1PWkfeCUOSBZODxGanbualFfGzZ+i5raplcyle95nfffX3TyE5UkSZIkSeri0jJwaigaDkeeVLf9uP3EWMv27b8+Zpv3319wEjOUJEmSJEnqHtImcMrKyiKRSABHbqk7FDjVtqCOU4w1xHjsLXO1tZXHfC5JkiRJktSdpU3gBIcKhx8WOA1ovMKp7Lh9ZGRkk5c34phtevcZe5IzlCRJkiRJ6vrSKnBq2FbX1Cl10PKT6oacdVuzzzIysvnAoBtPcoaSJEmSJEldX9oHTpnJU+qg5YHTmWd+liFD/vao+yFkUjRmDjk5H2jlTCVJkiRJkrqutAycampqqK6uBiAxYEDqeW1ZywKnEAIjCu/logt/xQeH3EFmZv1WvUSiD/n5l7bxrCVJkiRJkrqWtAyc4NBJdRl5eYSePQGo2d6ywKlBXl4hhYX3cPrAawCort7Fvn2r22i2kiRJkiRJXVPaBk4N2+pCCCSS2+pauqXuSP3zP5K6Ltu5vBUzlCRJkiRJ6vrSKnBqOKUOmi4cXlN2/FPqmtK/34eBAMDOnctOfoKSJEmSJEndQFoFTk2tcALIHFAfONXt2UPdgQMn3G9W1mn06V0EwO7dK6mtrWzlTCVJkiRJkrouAycgkV+Quq492W11/ScDEONBdu1ecZIzlCRJkiRJ6voMnDi0pQ5asa2u/6E6Tjut4yRJkiRJktJY2gZODafUASQGNAqcTnKFU9++xWRm5gEGTpIkSZIkKb2lbeB0WA2n5Cl1ADXbTy5wysjoQb9+FwFQUbGWqgN/PslZSpIkSZIkdW3tFjiFEKaGEP4QQlgXQri3iecfDCEsDSG8GUJ4PoQwuNGzW0IIa5N/t5zsHFq2pe7kAic4VMcJXOUkSZIkSZLSV7sETiGETGAOcCUwCrgxhDDqiGYPAz+MMY4F/gV4KPnd/sD9wIXAROD+EEK/k5lHVlYWWVlZwBGB04ABqeuTLRoOkG8dJ0mSJEmSpHZb4TQRWBdj3BBjPAjMBT55RJtRwNLk9XONnl8B/DrGuDPGuAv4NTD1ZCfSsMrp8FPqWr+lrr7vofTs+QEAdu58kRjrTrovSZIkSZKkrqq9AqcPAO81+rw5ea+xN4Brk9efBnqHEPJb+N0WaypwysjJISOvvuD3yZ5SBxBCSG2rq67eyd59q066L0mSJEmSpK6qvQKn0MS9eMTne4BLQwivA5cCW4CaFn6XEMLfhRBeDSG8un379mYn0hA4VVZWEuOhbhrqOJ3sKXUN+jfeVlfmtjpJkiRJkpR+2itw2gyc1ejzYGBr4wYxxq0xxs/EGMcB/5i8V96S7ybbPhFjHB9jHD+gUU2mI+Xm5gJQW1tLdXV16n5mWwVO/S6mISPbuXNZq/qSJEmSJEnqitorcHoFGBFCGBZC6AHcAPy8cYMQQkEIoWE+s4Gnkte/AqaEEPoli4VPSd47Kcc7qS5WVlJXUXGy3ZOVdRp9+owFYHf5SmprK0+6L0mSJEmSpK6oXQKnGGMN8CXqg6LVwH/GGN8JIfxLCKEk2eyjwB9CCH8ETgceSH53J/B/qA+tXgH+JXnvpBwvcILW1XECUnWcYqxm167ft6ovSZIkSZKkribRXgPFGBcAC464d1+j62eBZ5v57lMcWvHUKs0HTo1Oqtuxgx5Dhpz0GP37f4RNm+YAsHPXixQUXHbSfUmSJEmSJHU17bWlrtNoHDhVVh7a7pbZeIVTK+s49e1TTGZm/al3O3daOFySJEmSJKWXtA6cmt1S18rAKSMji379LgagomItVVXbWtWfJEmSJElSV5J2gVPDKXXQfOBU28rACQ7VcQLYufPFVvcnSZIkSZLUVaRd4NSyFU6tKxoOkH9Y4LSs1f1JkiRJkiR1FQZOSZn5hxcNb/04Q+nZczBQXzg8xrpW9ylJkiRJktQVGDglZfToQUbfvkDbBE4hhNS2uurqXezd+06r+5QkSZIkSeoK0i5wSiQSZGVlAYefUgeHttW1RQ0nOLKOk6fVSZIkSZKk9JB2gRMcWuXUeIUTHAqcanbsIMbY6nH697uYhp/YwEmSJEmSJKWLtAycGk6qOypwStZxigcPUrd3b6vHyco6jT59xgKwu3wltbWVx/mGJEmSJElS15eWgVPjFU6NVzIlBrTtSXVwaFtdjNXs2vX7NulTkiRJkiSpM0vrwKm2tpbq6urU/cyCxoHT9jYZK7//R1LXbquTJEmSJEnpIK0DJzh8W10i/1Dg1FaFw/v0OZ/MzF4AlBk4SZIkSZKkNJD2gVPjk+pOxZa6jIws+vW7KDnWOqqqtrZJv5IkSZIkSZ1VWgZODUXD4YgVTodtqWubFU5w5La6F9usX0mSJEmSpM4oLQOnZrfUnaLAqaFwOMDWbc+yd++qw4qVS5IkSZIkdScGTo0Cp8x+/SAEAGrK2i5wqq2tIoQsAMrLX2XFK5/glVc/yd69q9psDEmSJEmSpM7CwKlR4BQSCTL79wegdnvbBE5VVdt4vfRmYqw+7P7eve/w2uvT2L9/c5uMI0mSJEmS1FkYODUKnODQtrq22lL33nvfp7p6d5PPamrKefe9p9pkHEmSJEmSpM4i7QOnxqfUASTy8wGo2bmTWFfX6rF2lD1/zOdlZc+1egxJkiRJkqTOJO0Dp6NWOA1IFg6vqaG2vLzVYx25le6o53U1rR5DkiRJkiSpM0nLwCmRSNCjRw/g8MCpds8eDm7Zmvq87Z/+mapVrSvsfdppE1v1XJIkSZIkqatJy8AJDq1yagicqt9/n41//dfsf/XVVJt9S5ey8a+vo/wXvzzpcYacdWvqhLojhZBgyJBbT7pvSZIkSZKkzsjAKRk4/eWBB6n+07tHN6ytZds//iM1O3ee1Di9ep3L2KLHSSROO+rZyHMfoHfv0SfVryRJkiRJUmeV9oFTZWUl1WVl7F2ypNm28cAByn/+85Meq6DgMiZPWs7oUd8kP/9jqft1dVUn3ackSZIkSVJnlfaBU11dHZWbN0Nt7THbV7+3uVXjZWbmcMYZJYwc+X+AAMBf3l/Qqj4lSZIkSZI6o7QNnHJzc1PX1b16Hbd94vTT22TcntlncFrf8QDs3r2CAwf+0ib9SpIkSZIkdRZpGzg1rHACqM7NJe+SjzTfODOTviWfaLOxB55+VfIq8v77i9qsX0mSJEmSpM7AwIn6wuFn/OM/kpmf32Tb0++9l6wzzmizsQcOmIrb6iRJkiRJUndl4ER94NTjgx9k2H89S/9bPk/Iy0s9O/2r99P/c9PadOzs7IGcdtpEAMrLX6XqwJ/btH9JkiRJkqSOZOBE/Ul1AFlnnMHps2dz+le+knqW6Nv3lIx/+sCrUtfvv7/wlIwhSZIkSZLUEdI2cGpcNHz//v2HPcsePix1fWDDhlMy/oABV9Dw87/vtjpJkiRJktSNpG3gdOSWusZ6nH126vrg+lMTOGVnD6Bfalvda1RVbT0l40iSJEmSJLU3AyeODpwy+/UjM7mV7sDGjadsDgNPvzp17Wl1kiRJkiSpuzBw4ujAKYRAj+HDATi4cSOxru6UzGHggCk0/C/wtDpJkiRJktRdpG3glJmZSY8ePYCjAyeAHmfXB06xqoqabdtOyRx69CigX7+LANiz53X2799ySsaRJEmSJElqT2kbOMGhVU4Np9Q1lj1seOr6VBUOhyNOq9vuaXWSJEmSJKnrS+vAqeGkuiZXODU6qe7gKQycBgyYQgiZgKfVSZIkSZKk7iGtA6eGFU779+8nxnjYs+zhjVc4nbrC4T165NPvtIsB2LPnDfbv33zKxpIkSZIkSf8/e/cdHfdV5///daeo92JbluUaO65x7OAUQtiQUFJomx5SNkuoCwR+CywsbWkBloUvLKFsIGGzBBJIMQlhgcDG6T2xE9e4Wy6yLVm9Tr2/PzQajR1pNJLm8xlJ83ycM8cffe6dz+etxHBOXufe94UbCJwkRaNRBYPB48b8s2bJ+P2SpODu3Y7WMW16wrY6VjkBAAAAAIBJjsAp5nUn1Xm9ypk7V5IU2OvcCiep/7S6gW11nFYHAAAAAAAmOwKnmKH7OPVvq4s0NyvS1uZYHX5/ucrL3yhJ6uzcpJ6eesfeBQAAAAAA4DQCp5ghT6pbkNDHyeFVTtOnXRy/bmzktDoAAAAAADB5ZXXgNHBKnTTMCqd5g4GTkyfVSQOn1fX3jKKPEwAAAAAAmMyyOnAaeUvdvPh1wOHAye8vVUXF2ZKkzq4t6ulxdkUVAAAAAACAUwicYoYKnHLnDQZOwT3OB0DTpyWeVse2OgAAAAAAMDkROMUMFTh5Cgrkm1kjSQrs2e14PVVVb4tvq+O0OgAAAAAAMFkROMUM1TRcknJjfZxCBw4qGgw6Wo/fX6LKinMkSV1d29Td7ew2PgAAAAAAACcQOMUMtcJJknLmxxqHR6MK1dc7XtO0hG11DYfvVSQydBAGAAAAAAAwUWV14OT1epWbmytp+MApd8HgSXWB3c6vOKqsPEcD/1r27/+5Hn/8VG3c9DGaiAMAAAAAgEkjqwMnaXCV07ArnOYNBk7Bvc4GTtZGtXXb5yRFB+8poqamv+illy9XT4/zK6wAAAAAAADGi8BphMApd/7gSXUBh0+qO3ZsnZqbHxtyLBRq1Z69P3T0/QAAAAAAAOlA4JQQOFlrXzfuraqSp6REkhTc4+wKp6ONf0w63tj4Z0WjIUdrAAAAAAAAGC8Cp1jgFI1GFQgEXjdujFHuvP5VToG9e2Wj0dfNSZdwuCPpuLUhRaPOnpQHAAAAAAAwXlkfOBUUFMSvRzqpzvb0KHz0qGO1FBUuTjqel1cnr7cg6RwAAAAAAIBMy/rAaWCFk5QscErs4+Tctrra2qtljH/Y8bpZ18sY49j7AQAAAAAA0oHAKYXAKXfBgvh1cLdzgVN+fp2WLfvBkKFTcfFK1dX9g2PvBgAAAAAASBdfpgvItJRWOM1LWOG019nG4dOnXajSkpVqaLhXHR0b1dzyuCSraLRP5IMAAAAAAGAyyPoEI6XAqa5O8vevOgru2et4TXl5MzV//id16qm3q6rqfElSd/d2dXRscPzdAAAAAAAA40XglBA49fT0DDnH+HzKmTNbkhR0sIfTUGprr45fHzx0l6vvBgAAAAAAGIusD5xSOaVOknLn9Z9UF25qUqSz0/G6BlRWnKO8vFpJUmPjnxQKtWGe2CwAACAASURBVLv2bgAAAAAAgLHI+sAplS11kpQzf3782s1VTsZ4NXPmlZKkaDSgw0fWuvZuAAAAAACAscj6wCkvLy9+nXSF0/yExuEu9HFKNLPmchnT39/90KHfylrr6vsBAAAAAABGI+sDJ6/Xq9zcXEkjrXBaEL8O7tnteF2JcnOnxZuH9/TsUlv7S66+HwAAAAAAYDSyPnCSBrfVJQ2c5mVuhZMk1c4cbB7ecOhu198PAAAAAACQKgInDTYOH+6UOknyFhXKN326JPdPqpOkioqzlZ/Xf1JeY9OfFQq1ul4DAAAAAABAKgicdPwKp2T9kXIX9DcODx44IBsKuVLbAGM8Cc3Dgzp8mObhAAAAAABgYiJw0mDgZK1VIBAYdl7OvNhJdeGwgvv3u1HacWpmXjbYPLyB5uEAAAAAAGBiInDSYOAkjdQ4PLGPk/vb6nJzqlRd/XZJUk/PHrW1Pe96DQAAAAAAACMhcFLqgVPu/Pnx62AGGodLUu3Mq+LXh2geDgAAAAAAJiACJx0fOCVrHJ4zf0H8Orhnt6M1Dae8/Czl58+RJDU2PaxgsDkjdQAAAAAAAAyHwEmDp9RJyVc4+aZVy1NYKEkKZGiFkzGe+Cona0M6fPj+jNQBAAAAAAAwHAInpb6lzhijnNi2uuCePRlr2l1Tc6mMyZE00Dw8mpE6AAAAAAAAhkLgpNQDJ2mwj1O0u1vhxkZH6xpOTk6lpsWah/f21qu19dmM1AEAAAAAADAUAieNLnDKOa5xuPsn1Q2orb06fn2o4bcZqwMAAAAAAOBEBE4abeA0L34dyGDgVFZ2hgoK+sOvpqa/KhA8lrFaAAAAAAAAEhE4KfVT6iQpd0HiSXWZaRwu9feTGmweHtaO7V9TY+NfFA53ZqwmAAAAAAAAicBJkuTxeJSXlycphRVOdXWSzydJCuzZ7XhtyVRVvVUD/wobm/6kTZs/pqeeeqP21d+asYbmAAAAAAAABE4xA6ucRgqcjN/fHzopsyucrLXatu1zko4/oS4S7dHu3d/VwUO/zkxhAAAAAAAg6xE4xaQaOEmDjcPDR48q0tXtaF3DaW19Vm3tLw47vm/fTxWNhlysCAAAAAAAoB+BU0xi4BSNRpPOzU08qW5vZhqHt7Q8lXQ8GGxUd/dOl6oBAAAAAAAYROAUMxA4WWsVDAaTzs1JDJwyeFLdyOjjBAAAAAAA3EfgFFNQUBC/HvGkuvnz4teBDPVxKq94Y9LxnJwqFRYudKkaAAAAAACAQQROMQMrnKQUTqqbACucKsrfqNKSVcOOz5nzEXk8OS5WBAAAAAAA0I/AKWY0gZO3uFi+6mpJUiBDgZMxHp1yys9VUXHO68Y8njzNrLkiA1UBAAAAAAAQOMWNJnCSBlc5Bffvlw1l5jS4nJwKrTr1Dp1x+p+0ePG3VFlxriQpGu1Tw+F7MlITAAAAAAAAgVPM6AOnWB+nUEjBgwedKislRUUnq3bmlVq85GYZ45ck7d9/m6LR5M3PAQAAAAAAnEDgFDPawCl3/oL49UQ5qS4vd4ZqZvy9JCkQOKIjR/6Q4YoAAAAAAEA2InCKGc0pdVLCCidlro/TUObM+ZAkI0mq33+rrI1ktiAAAAAAAJB1CJxiRr/CKfGkur2O1DQWBQXzNG3ahZKknp49amr6W4YrAgAAAAAA2YbAKSYvLy9+nUrg5JsxQya2KmqibKkbMHfOR+LX++p/JmttBqsBAAAAAADZhsApxuPxxEOnVAInY4xy5/Vvqwvs2TOhQp3i4mWqqDhHktTZuVmtrc9kuCIAAAAAAJBNCJwSDGyrSyVwinR2ygb7T4GLdnZq76WXqv3BBydM8HTiKicAAAAAAAC3EDglGGgcPlLgFGlvV/37rlFg5874vcDWbWr43Od15GtfmxChU1nZGSopWSVJam19Vu0dr2a4IgAAAAAAkC0InBIkrnCKRqPDzjv2058dFzYlavvt79Tz/POO1DcaxpjjVjnV72OVEwAAAAAAcAeBU4KBwMlaq0AgMOQca63aHngg6XPaf//7tNc2FlVV56mwcKEkqenY39TdvSvDFQEAAAAAgGxA4JRgIHCSht9WZ4NBRdvbkz4n1NiY1rrGyhiP5sz+cPzn+vpbM1gNAAAAAADIFgROCVIJnExOjrzVVUmfkzOrLq11jcf06e9UXl6tJOnI0T+or68hwxUBAAAAAICpjsApQUqBkzEqv/yKpM8pu+LytNY1Hh6PX7Nnf1CSZG1Y9ft/keGKAAAAAADAVOda4GSMucAYs90Ys8sY8/khxmcbYx41xmwwxmw0xlwUuz/XGNNrjHkl9vkvp2ocOKVOknp6eoadV/mhD6pgzZohx6o/eZPyV6xIe23jMbPmMvn9FZKkhoZ7FAw2Z7giAAAAAAAwlbkSOBljvJJ+IulCSUslXW2MWXrCtC9Jusdau0rSVZJ+mjC221p7auzzETkklRVOkuTJy9Ps22/TjG98XXmnnhq/X7Bmjao++lGnyhszrzdfs+v+UZIUjfbp5Zev1MaNH9HBg79RONyd4eoAAAAAAMBU49YKp9Ml7bLW7rHWBiX9VtJ7TphjJZXErkslud5sKNXAServ5VR++eWa99u75ZtZI0kK1tfLWutojWNVUfFmSUaS1NO7V03H/qbtO76i51+4SL29BzJbHAAAAAAAmFLcCpxqJSWmGgdj9xJ9VdK1xpiDkv4k6RMJY/NiW+0eN8acM9QLjDEfMsa8ZIx5qampaUxFjiZwSlSwarUkKdzYqHDDxGvKba3Va699Qf2Z3vH6+g5q85ZPTdigDAAAAAAATD5uBU5miHsnJhxXS7rDWjtL0kWS7jTGeCQdljQ7ttXunyXdZYwpOeG7stb+3Fr7BmvtG6qrq8dU5FgDp/xVq+LXPes3jOndTuro2KDOri1Jxl9RZ+cmFysCAAAAAABTmVuB00FJdQk/z9Lrt8zdKOkeSbLWPispT1KVtTZgrW2O3X9Z0m5Ji5woMi8vL349qhVOqwcDp94N69NaUzp0de8ccU539y4XKgEAAAAAANnArcDpRUkLjTHzjDE56m8K/ocT5uyXdL4kGWOWqD9wajLGVMeajssYM1/SQkl7nCjS4/HEVzklO6XuRLmLFskTO+FuIq5w8vvKRpzj85e6UAkAAAAAAMgGrgRO1tqwpI9LeljSNvWfRrfFGPN1Y8y7Y9M+LemDxphXJd0t6Qbb31jozZI2xu7fJ+kj1toWp2odCJxGs8LJ+HzKW3mKJCmwY4ciXV2O1DZWlZVvls/3ul2IcX5/hSrK3+RiRQAAAAAAYCrzufUia+2f1N8MPPHeVxKut0o6e4jv3S/pfscLjBlL4CT1Nw7vefY5KRpV76uvqujs1/0qGeP15mvRwi9r67bPDjk+b+5N8npzXa4KAAAAAABMVW5tqZs0BgKnvr4+RaPR1L+3enX8uncCbqurqblEK0+5TaUlq1431tW1NQMVAQAAAACAqcq1FU6TxUDgZK1VIBA47uS6pN87daVkjGStejdMvMBJkqqq3qKqqrcoEulRJNKrF158twKBI2o4fJ/qZr9fRYULM10iAAAAAACYAljhdIKCWPNvaXTb6rxFRcpd1H94Xu+rr8pGImmvLV283gLl5FRq/rxPxu5EtXv39zJaEwAAAAAAmDoInE6QmzvYy6ilZXS9yfNX929Xi3Z3K7BjR1rrcsKMGZeoMLaq6dix/1Nb20sZrggAAAAAAEwFYwqcjDHnGGPemO5iMm3jxo16/vnn4z//5je/0YMPPqhgMJjS9wsS+jj1TNBtdYk8Hp8WLBhsJL5z13fUfzAgAAAAAADA2KUUOBljHjPGnBO7/oyktZLuN8Z8zsni3LRlyxatXbtWgUAgfs9aqw0bNujee+9NKYjJXzXYkHsiNg4fSlXleSorXSNJ6ujYoKamv2a4IgAAAAAAMNmlusJphaRnY9cflnSupDMk/ZMDNbkuGo3qkUceGXZ8586d2r9//4jP8dfWylddLUnqXb8+bfU5yRijk04azA137/kPRaOhDFYEAAAAAAAmu1QDJ4+kqDFmviSftXaLtXa/pArnSnNPc3PziP2atm/fPuJzjDHKj22rCzU0KHT0aFrqc1pp6SpVV18gSerp2auGw/dmuCIAAAAAADCZpRo4PSPph5K+K+n3khQLn5odqstVkRROlEtljiQVrE7YVjcJ+jgNOGnBZ2SMV5K0d+9/KhzuznBFAAAAAABgsko1cLpBUp+k7ZK+Eru3VNItDtTkuqqqKuXn5yedM3v27JSeldjHqWeSbKuTpIKCeZo58ypJUjB4TPsP/DLDFQEAAAAAgMkqpcDJWttkrf0Xa+0XrbVdsXt/tNZ+39ny3OHz+XTmmWcOO15ZWanFixen9Ky8JUtk8vIkTZ7G4QPmzbtJXm+BJGn//l8oGDyW4YoAAAAAAMBklOopdZ80xpwauz7dGLPHGLPDGHO6s+W555xzztGaNWted7+yslLXXnutvF5vSs8xfr/yV6yQJPVt26ZoT09a63RSbk6VZs/+oCQpEunW3r0/znBFAAAAAABgMkp1S92nJe2LXX9H0k8kfV/SjxyoKSM8Ho8uvvhi3XTTTZo3b178/t///d+rvLx8VM+Kb6uLRNS7aXM6y3Tc7LoblZNTJUk61HC3WlqeU3f3LkWjwQxXBgAAAAAAJotUA6cya22bMaZI0qmSfmitvVVSavvMJpGKigqdcsop8Z9HOr1uKPnHNQ6fPH2cJMnnK9S8uTdJkqwNa8Mr1+i559+hp54+W3v3/ljWptY8HQAAAAAAZC9fivMOGmPOkLRM0pPW2ogxpljSlEwfKisr49fNzaM/iK/g1FPj15OpcfiAnNyq190LhVq0Z+8PFAge1eKTv5GBqgAAAAAAwGSR6gqnf5H0kKSvS/pm7N47Jb3oRFGZNt7AyVtWppwFCyRJva+8KhuNpq02p1kb1a5d3x12/NChu9TdvdvFigAAAAAAwGST6il1f7TWTrPWzrLWDoRMv5f0XudKy5yCggLlxU6aG0vgJEkFsW110Y4OBXdPnoCmq2u7env3JZ3T2PQXd4oBAAAAAACTUqornGSMmW+M+YIx5ifGmC9IqrXW9jlYW8YYY+KrnFpaWmStHfUz8letjl/3rN+QttqcFol0pzBn8py8BwAAAAAA3JdS4GSMuUjSq+pvGN4jaaWkDcaYix2sLaMGAqdAIKDu7pFDmBMVHNc4fPIEToWFi+Tx5CadU1K8wqVqAAAAAADAZJTqCqdvS3qvtfYKa+1nrbVXqn873bedKy2zxtvHyT9njrwVFZKknkl0Up3fX6KamsuGHc/JmaGqqvNdrAgAAAAAAEw2qQZOsyU9dsK9J2L3p6TxBk7GGOWv6l/lFKrfr/CxY2mrzWkLT/qCKivPHXIsN7dSxnjdLQgAAAAAAEwqqQZOr0r61An3bpK0Mb3lTBzjDZykE7bVvfLKuGtyi9ebp5Wn3KbVq+5WXd0/aubMq5SXVydJ6uzcosOH789whQAAAAAAYCLzpTjvnyT90RjzKUn7JdVJCkt6t1OFZVpFbDucNPbAaWCFk9TfOLz4rW8dd11uMcaovPx0lZefLqk/aHrhxfdKimrnru+oqup85eRUJH8IAAAAAADISimtcLLWbpV0sqTrJP1E0vWSFltrNztYW0bl5uaquLhY0tgDp7xly2T8fklS7/rJ08dpKMXFy1RX9w+SpHC4Tbt2/3uGKwIAAAAAABNVqlvqZK0NWWsfs9beZa19TJKMMXscq2wCGNhW19LSomg0Ourve3Jzlbd8uSSpb8sWRQOBtNbntvnzPqXc3BmSpMOH71Nr6wsZrggAAAAAAExEKQdOQzCS5qapjglpIHCKRCJqb28f0zPyY32cbCikvi1b0lZbJvh8RVq08Cvxn1/b/mVFo8EMVgQAAAAAACai8QROkmTTUsUElZbG4Ql9nCb7tjpJqq5+uyor3yJJ6unZpf37b89wRQAAAAAAYKIZb+A0paUjcDqxcfhkZ4zRyYu+Ko8nT5K0d98t6u3dn+GqAAAAAADARJL0lDpjzC/H+t2pIB2Bk6+yUjlz5ihYX6/eDRtkrZUxJl0lZkR+/izNm3eTdu/+rqLRgLbv+KpWnnL7pP+9AAAAAABAeoy0wulQkk+9pG85Wl2GlZWVxUOUsQZO0uAqp0hrq4L79qWjtIybXfd+FRYulCQ1Nz+upqaHM1wRAAAAAACYKJKuUrLWftmtQiYin8+n8vJytbS0jC9wWr1K7Q88IEnqXb9BufPmpavEjPF4/Dr55G9o/fqrJEk7dnxdJSUrJUk5OVXyePyZLA8AAAAAAGQQPZxGMLCtrq2tTaFQaEzPKFi9On7ds2HyNw4fUF62RjU1l0uSAsGjevqZN+npZ96kp54+S7t3f1/RaCDDFQIAAAAAgEwgcBpBYh+n1tbWMT0jZ/58eUpKJEndTz+jwJ69aaltIphZc6Wk43s3hUKt2lf/U23a9HFZG81MYQAAAAAAIGMInEaQjsbhiX2bwocPa89FF2nf+65RYNeu8ZaXcfsP3CbJDjl2rHmdmpsfd7cgAAAAAACQcQROIxhv4BQ62qj6665XtKPjuPu969er/rrrFTp8eNw1ZkokEtCxpr8lnXO08Y8uVQMAAAAAACaKpE3DBxhjrh9mKCDpoKQXrLVja3A0wVVUVMSvxxI4td75K0WG+V6ktVUt//MrTf/858ZcXyZFoz2yiiSdEw53uVQNAAAAAACYKFIKnCR9SNIaSc3qD5hqJVVJ2iBprqSQMeY91tqp0xE7pqSkRD6fT+FweEyBU+ejj40wvm7SBk4+X5lyc2sUCAy/SquoaLGLFQEAAAAAgIkg1S116yV93lo701p7urW2VtLnJD0vaaak2yX92KEaM8rj8cRXOY0lcLLB4Ajjk3dhmDFGdbOGW/wmSV7VzrzStXoAAAAAAMDEkGrgdJ2kH51w7xZJ19v+Y8i+LWlZOgubSAb6OHV3d6u3t3dU3y1YvWqE8dVjrmsimD37Rs2Y8d5hRq0CgUZX6wEAAAAAAJmXauDUKOnCE+5dIKkpdp0rafIu1RlBYuPwlpaWUX234oYbJK936EGvt398EjPGq6VLvqfVq3+r2tprNH3aO1VV9dbYaFRbtn5akUhPRmsEAAAAAADuSjVw+pSku4wxjxtjfm2MeVzS3ZI+GRs/S9LPnChwIhjPSXV5S5ao9gf/T56ioteNzfzOt5W/Yvm468s0Y4zKy9Zo8clf1/Ll/6lTVvyXqirPkyT19u7Tzl3fyXCFAAAAAADATSkFTtbaP0s6SdIdkrZJ+h9JJ8Xuy1r7sLX2y04VmWnjCZwkqeTtb9dJjz2mmpu/qbyVK+P3vQkn4E0lxhgtXvJt+f39v9+hQ7/RsebHMlsUAAAAAABwTaornGStbbTW/re19mZr7S+ttVnTnGe8gZMkeYsKVXbppZr2z/8cv9f1yLpx1zZR5eZUacnim+M/b9v2eQWDo9uOCAAAAAAAJqeUAidjzBxjzK+MMRuNMXsSP04XOBEUFBQoLy9P0tgDp/izTlstb2mpJKnz0UdlrR13fRNVdfXbVVNzuSQpGGzSa9u/OKV/XwAAAAAA0C/VFU53ScqR9EVJHzzhM+UZY+KrnJqbm8cVmhifT0Xn/p0kKXz4sPq2bk1LjRPVooVfUl5enSSpqemvOnJkbYYrAgAAAAAATks1cFoh6Rpr7UPW2kcSP04WN5EMBE7BYFBdXV3jelbReefHr7vWPTquZ010Pl+Rli39niQjSdq+4+vq7T2Y2aIAAAAAAICjUg2cnpJ0ipOFTHTp6OM0oPDss2X8fklS57qp28dpQFnZGzRnzoclSZFIl7Zu/YwikYBCoXZZG8lwdQAAAAAAIN18Kc7bKelhY8x9ko4kDlhrv572qiagEwOnuXPnjvlZ3qJCFZx1prqfeFKBbdsUOnRI/traNFQ5cc2f90k1Nz+urq5tamt/UY8/sVLWhuTzlWpmzWWaN+8m+XxFmS4TAAAAAACkQaornCokPSypWNLChM9JDtU14aRzhZMkFSdsq+uc4tvqJMnjydHik7+hga111oYkSeFwu/YfuF0bNlynSKQ3gxUCAAAAAIB0SSlwstZeN8zneqcLTJfucERd4bFv36qoqIhfpyNwKnrLW+LXXY9O/W11ktTc8pSkoRuud3RuVEPD79wtCAAAAAAAOGLYwMkYMyvhevZwH3fKHLvHWjr0rpd3asGTm3TSk5v0jpe26y9N7aN+Tm5uroqLiyWlJ3DyT5+mvBUrJEndL7yoSEfHuJ850R09+oek40eOPuRSJQAAAAAAwEnJVjhtS7jeJ2lv7M/Ez15HqkqTB4626upX9+jFju74vVc7e3XD5r36TcPoQ6OBbXUtLS2KRqPjrq/4/PP6L8JhdT3x5LifN9GFQm1Jx8Ph0QeBAAAAAABg4kkWOJUmXPsl5cT+TPzkOFfa+ASiUX1p56FhNnBJ/7brkLpHucVuIHCKRqNqa0senqSi6Lzz4tdd6x4Z9/MmuqLCRUnHCwuypiUYAAAAAABT2rCBk7U2mnAdGe7jTpmj91Rrl46FwsOOd0Wi+lvz6LaxpbtxeO7ChfLP6t+52PXEk7LB4LifOZHNmpW85VdNzRUuVQIAAAAAAJyUUtNwY8wcY8yvjDEbjTF7Ej9OFzhW7SmsXmob4wonKT2BkzEmvq0u2tWl7hdfHPczJ7Lq6rdrzuwPDTt+tPEPsna4NWkAAAAAAGCy8KU47y5JByR9UVKPc+Wkz+LCvLTMSZTuwEmSis47Xy3/8ytJUtcj61R09tlpee5EZIzRSSd9TtXTLtDhhnsVCByV31+uY83rFAq16ujRh1Raulp1I6yEAgAAAAAAE1uqgdMKSW+eyFvoTrS0KF9nlhbqufbuoccL83RGaeGonlleXi5jjKy1aQucCk5bLU9pqaLt7ep89FFN//KXZIxJy7MnqtKSlSotWRn/ubXtRW3YcI2sjWjnzm+ppHi5SktXZ7BCAAAAAAAwHiltqZP0lKRTnCzECT9eOkfz83Nfd782169fLJ876mDH6/WqvLxcUvpWOBmfT0V/92ZJUvjwYQW2bRvhG1NPedkanbTgc5Ika0PatPkTCgaPZbgqAAAAAAAwVqkGTjslPWyM+akx5iuJHyeLG69ZeTn6vzUn6+aFtfF7y4vy9fjpi7WgYHTb6QYMbKtrb29XKBRKS53F550fv+58ZF1anjnZ1NW9X9OqL5QkBQJHtHnLpxSNDt/0HQAAAAAATFypBk4Vkh6WVCxpYcJnwp9jX+D16P21VfLFFjNV+L0q8nnH/LzEPk4tLS3jLU+SVPimN8n4/ZKkznXZGTgZY7RkyXdUUDBfktTa+qz27P1BhqsCAAAAAABjkVIPJ2vtdU4X4iRjjEp9PjWHwqM+me5EJzYOnz59+njLk7eoUAVnnanuJ55UYNs2hQ4dkr+2duQvTjE+X5FWrPipXnrpEkUiPaqv/y+Vlpyq6uq3Zbo0AAAAAAAwCsOucDLGzEq4nj3cx50yx68stqqpPZTewCldis87L37due7RtD13sikqXKgli78d/3nL1k9r22tf0sZNH9W2176g1rYXZa3NYIUAAAAAAGAkybbUJXav3idpb+zPxM9eR6pyQKk/FjileYVTuhS9ZTBw6no0O7fVDZg+/Z2qm3WDJCkS6VZDw91qavqrGhp+p/Xrr9Jrr31B1kYzWyQAAAAAABhWssCpNOHaLykn9mfiJ8e50tKr1DcYOEXHsUKmuLhYPl//TsR0Bk7+6dOUt2KFJKn7hRcV6ehI27MnoxkzLhl2rOHwPTp06G4XqwEAAAAAAKMxbOBkE5aQWGsjw33cKXP8BrbUWUmd41jl5PF44quc0hk4SVLx+bFVTuGwup54Mq3PnmwOH7k36fiBg79yqRIAAAAAADBaKZ1SZ4zxGmP+yRjzO2PMI8aYdQMfpwtMl1L/YH/0dDUO7+npUW9v77ielei4bXXrHknbcyejrq7tScd7enZpEuWdAAAAAABklZQCJ0n/T9JNkl6QdIak/5U0S9JTDtWVdgMrnKSJ28cpd9FC+Wf192rveuJJ2WAwbc+ebHy+kqTjXm+RUv/rCwAAAAAA3JTqf7FfJukCa+33JUVif75H0psdqyzNShMDpwl6Up0xJr6tLtrVpe4XX0zbsyebGdPflXS8quqtMsa4VA0AAAAAABiNVAOnAkn1seseY0y+tXabpNXOlJV+A6fUSenbUidJLS0t43rWiYrOOz9+3fnIpNmxmHbV1ReoovzsYce7u3coEulzsSIAAAAAAJCqVAOn1yS9IXb9sqSvGGM+L6nBkaocMBm21ElS/qpTZfLyJEltd92l7WeepSM3f0vhNL9novN4fDrllF9o7pyPyu+viN31yectliR1dW3V1q2fUUJvewAAAAAAMEGkGjj9f5IG/sv+05LOknS5pI84UZQTErfUtYXC43pWQUGB8vPzJaU3cLLW6shXvybbN7hyJ9rWptY779S+K69SuKkpbe+aDLzeXC1Y8Bmd86bndM6bXtC5f7dRp5/+R+XkVEmSGpv+rN27v5fhKgEAAAAAwIlGDJyMMV5JiyRtliRr7XZr7bnW2tOstY85XF/alCWcUjfeFU7S4Cqn5uZmWWvH/TxJ6nn+ebWvXTvkWOjgQTXd8uO0vGeyMcarnJxKeb25ys+fpVNO+bk8nv5VYPX7b9Whht9luEIAAAAAAJBoxMDJ9p89f4u1NuBCPY4pTeOWOmkwcAoGg+rq6hr38ySp/YEHk4//8SHZyPhrn+xKS1Zq2dLvS+pvGr59+5fV3DJpDkwEAAAAAGDKS3VL3f8aYy5ytBKHJfZwGm/TcMmZPk4j9WmyPb2K9tIoW5KmTbtAJ530OUmStRFt2vQxdXXtyHBVAAAAAABAknwjT5HUH0ytTQCsvAAAIABJREFUNcY8JemApPgeMmvt+50oLN0KvR55jRSxUnso/YHT3Llzx/3MnLlz1f3kk8OOe6ur5CksGPd7porZdR9Qb0+9DjXcrUikS6+8+o+aPv096unZLZ+3UNXT3qHqqreqf1coAAAAAABwS6qB005J/+FkIU4zxqjU51VLKKK28PiahkvOrHAqv+Jytf7619IwPaHKr7xKxpi0vGsqMMZo0aKvqrfvoFpanlQgcET7998aHz9y9EGVl52plSt/Ia+XoA4AAAAAALck3VJnjLlakqy1Xx7u406Z6VHm68/X0tHDqaKiIn6drsApd+FCTf/SF4ceW7RIlR/6YFreM5V4PD4tPvlbGu6vcmvbc9q1+7vuFgUAAAAAQJYbqYfTrSOMTyoDjcPTsaUuJydHJSUlktIXOElSxTXXaO7996n0skuVe/LJ8fveinJ5cnLS9p6p5FjzI5Kiw44fbrhf4XC3ewUBAAAAAJDlRgqcptT+rTJ/LHAKRxQdZtvaaAxsq2tpaVEkjafH5S9bppnf/KbmP/iA8leulCT1PPe8gvv3p+0dU0l31/ak45Foj/r6DrpUDQAAAAAAGClw8hpj3mKMOW+4jytVpsnACqeopK7I8CtiUtHV1aXu7v5VM9FoVLfccouee+45RaPje+6Jyq64PH7ddv/atD57qvD5y1KYU+pCJQAAAAAAQBo5cMqVdHuSz22OVpdmA4GTJLWFxt44vKurS7fddpsaGxsHn9fWpr/85S9au3atbBpWTw0oueACeQr6G163r10rm4aG51PNjOnvSjpeUDBfebkzXKoGAAAAAACMFDh1W2vnW2vnDfOZ70qVaVLmHzyUbzyNwx977DG1tbUNObZ582bt3LlzzM8+kaewUCXvfKckKdzUpK7HH0/bs6eKoqKTNWvWdcOO9/TsU2PTwy5WBAAAAABAdhspcJpSElc4jTVwikaj2rhxY9I5I42PVtnlCdvq7r0vrc+eKhYt/IoWLfo35efNjt3xqKBgQew6qs2bP6Xm5iczVR4AAAAAAFklu5qGH7elbmyBUyQSUTAYTDpnoLdTuuQtX6bcxYslSV1PPKHQkSNpff5UYIxHdbOu11lnrdObz3lZ5/7dJp115l81Z85HJEnWBrVx00fU1vZShisFAAAAAGDqSxo4WWuL3SrEDQOn1EljX+Hk8/lUUlKSdE5VVdWYnj0cY4zKLr+s/4doVO2//31anz+VGGPk95fJ682TJC2Y/xnV1l4rSYpG+/TKqzeqo3NzJksEAAAAAGDKy9otdW1jDJyMMVqzZk3SOaeddtqYnp1M6bveJZObK0lqu+9+2TSfhjdVGWN08qJ/04wZ75UkRSJdeuWVG9TVnb4+WwAAAAAA4HhZFTgd1zR8HKfUnXXWWVq0aNGQYxdeeKFmzEj/iWjekhKVXPAOSVLo0CF1P/Ns2t8xVRnj0ZLF/67q6rdLkkKhVq1ff402bvq4nn7mXD3z7Fu0fcdX1dt7IMOVAgAAAAAwNWRV4JSOFU5S/7a6q666SldccYUWLlwYv19XV6czzjhjXDUmc1zz8PtoHj4aHo9Py5f9UBUV50iSQqFmNTX9WX19B9Tbu18HD96pF154lzo60tvwHQAAAACAbJRVgVNZmgInSfJ4PFq6dKmuueYaVVdXS5IaGxsVdXCrW/5ppyln3jxJUucjjyjc0uLYu6YijydXy5f9QMb4hhwPRzq1ddu/yFrrcmUAAAAAAEwtWRU4FXo98sbO3Wsf4yl1Q5kzZ44kKRAIqLGxMW3PPZExRmWXxZqHh0Jqf+BBx941VbW2viBrh99O2d29Ux0dr7hYEQAAAAAAU09WBU7GmPi2urbw2Hs4nWj27Nnx6/r6+rQ9dyil732P5PdLktruvZfVOKPUF2hIYc5hFyoBAAAAAGDqyqrASZLKfP3bqdrHuaUuUWLgtH///rQ9dyi+ykoVn3eeJCm4d696X37Z0fdNNXl5M0eek1vjQiUAAAAAAExdWRc4DaxwSueWurKyMpWWlkrqD5ycXnV0XPPwe2kePhpVlecqJ6dq2HFj/MrJmeZiRQAAAAAATD1ZFziV+WOBUziiaBqDoYFVTp2dnWptbU3bc4dS+Maz5J/Zv1Kn4+GHFenocPR9U4nHk6ulS74njyd3yHFrQ1q/4X3q7T3gcmUAAAAAAEwdWRc4DaxwikrqiqTvRDk3t9UZj0ell10qSbJ9fWr/4x8dfd9UU1l5jta84QHNrLlCeXl1ys+fo5kzr1RR4WJJUl/fQb28/ir19OzNcKUAAAAAAExOWRs4SVJbyJnG4U4HTpJUdsklkqf/X1/bvffRPHyUiooWacmSb+vsNz6mN561TksWf0unnfZblZauliQFAkf08vqr1dW9M8OVAgAAAAAw+WRd4FTm98Wv09k4vLq6Wnl5eZLcCZz8M2ao6M1vliQFtm1T3+Ytjr9zqvP5inXqyjtUVnaGJCkYbNL69e9TZ+c2SVI43KlgsIVwDwAAAACAEbgWOBljLjDGbDfG7DLGfH6I8dnGmEeNMRuMMRuNMRcljP1r7HvbjTHvGE8diSuc0hk4eTye+CqnY8eOqbu7O23PHk7Z5ZfFrxs+/3kd/fZ31PPSSwQi4+DzFerUlberouIcSVIo1KKX11+p559/lx5/4lQ9+dQaPff829XQwKoyAAAAAACG40rgZIzxSvqJpAslLZV0tTFm6QnTviTpHmvtKklXSfpp7LtLYz8vk3SBpJ/GnjcmZcdtqUtf4CS5v63OVzMzvq0uuHu3Wv7nf1R/7XU6+LGPKxoIOP7+qcrrzdfKU25VVdX5kqRIpFtd3Vvj4z09e7Tttc9p374fZ6pEAAAAAAAmNLdWOJ0uaZe1do+1Nijpt5Lec8IcK6kkdl0qqSF2/R5Jv7XWBqy1eyXtij1vTEr9zqxwkqQ5c+bEr50OnKLBoA5+/GNS9PWNz7vWrVPj97/v6PunOo8nV8uX/UgeT/6wc/bsvUWBwFEXqwIAAAAAYHJwK3CqlZR4zvzB2L1EX5V0rTHmoKQ/SfrEKL6bsuOahqc5cKqpqZHP198jqr6+Pq3PPlHnX/+mcMPhYcfb7r1PkS7nt/VNZd3dOxSN9iaZEVFj459dqwcAAAAAgMnCrcDJDHHvxAY4V0u6w1o7S9JFku40xnhS/K6MMR8yxrxkjHmpqalp2EISt9S1p/GUOkny+Xyqre3Pwg4fPqxgMJjW5yfq25K8Sbjt7VVw7x7H3p8NQqH2keeEO12oBAAAAACAycWtwOmgpLqEn2dpcMvcgBsl3SNJ1tpnJeVJqkrxu7LW/txa+wZr7Ruqq6uHLaQ04ZS6dK9wkgb7OFlrdfDgwbQ/f4CnoCAtczC8wqKFGul/IkWFi9wpBgAAAACAScStwOlFSQuNMfOMMTnqbwL+hxPm7Jd0viQZY5aoP3Bqis27yhiTa4yZJ2mhpBfGWkiZQ6fUDXCrj1Px29+edDx34UnKmT/fsfdng7zcGZpWnfxQxGPHHlE0GnKpIgAAAAAAJgdXAidrbVjSxyU9LGmb+k+j22KM+box5t2xaZ+W9EFjzKuS7pZ0g+23Rf0rn7ZK+oukj1lrx5wUFXk98sY26bWn+ZQ6SZo1a5aM6X+Bk32c8k5epNLLLh12fNpnPxuvA2O3ePE3VVy8Ytjxw0fu18ZNH1Y4TL8sAAAAAAAG+Eaekh7W2j+pvxl44r2vJFxvlXT2MN+9WdLN6ajDGKNSn1ctoYgjW+ry8vI0ffp0HTlyRAcPHlQkEpHX6x35i2NQ87WvyV9To9Zf/0aR1tbBAWOUE9vah/Hx+8v0htPuVVPTX3Xs2DpFbVBlpW9Qfn6dtmz9Z4XDnWpuflzrN7xPK1fertycqkyXDAAAAABAxrm1pW5CGTiprj2c3qbhAwb6OIVCIR05csSRd0iS8XpV/bGP6aTHH9P8//2jqj/96f4Ba3Xs579w7L3ZxuPxa/r0i7Vs2fe1Yvktqqv7B1VVnafTVv9OubkzJEmdnZv18kuXq6dnrySpr69BXV3bFYn0ZLJ0AAAAAAAyIksDp/6FXU70cJKO7+Pk5La6AZ6cHOUuWKDKf7hevpk1kqT2P/xBwYOHHH93NisqOllvOO0+FcYah/f27deLL/29nn3uQj39zDl6/oWL9ORTZ2rHjm8oEunLcLUAAAAAALgnKwOnsvgKp4istWl//uyE7WxONg4/kcnJUeUHPtD/Qzis5ttY5eS0vLwanbb6dyorO0OSFA53qqdnR3w8EunWgYN3aNOmj8raaKbKBAAAAADAVVkZOJX6+wOniJW6IukPAYqLi1VeXi6pP3ByItQaTtmll8pb3d9HqP3+tQodPerau7OV31+iVaf+t/z+ymHnNLc8oZaWp12sCgAAAACAzMnOwMk32MTbicbh0uC2up6eHjU3NzvyjqF4cnNV+f4bJUk2FFLLL3/p2ruzWSTSp1Ao+b/nxqY/u1QNAAAAAACZlZWBU1lC4NQecrZxuOROH6dE5VdeIW9shVXr7+5R2MXAK1ul0hycBuIAAAAAgGyRlYFTqd8Xv3ZqhVOm+jhJkqegQBU33CBJsn19arnjDlffn41yc6fFT6wbTl5enUvVAAAAAACQWVkZOB23wsmhwKmyslKFhYWS3A+cJKn8mvfJU1IiSWr9zV2KtLW5XkM2McarulnXJ51z6NCdam55yqWKAAAAAADInKwMnEqP21LnTOBkjImvcmptbVVHR4cj7xmOt6hIFddeK0mK9vSo5c5fu/r+bDR79gc0c+aVr7tvTP/ft3C4U6+88o/af+C/XW0kDwAAAACA27IycCrzO980XMrstjpJqrj+OnkKCiRJLXfeqUhXl+s1ZBNjvFqy+Fs6fc1DmjPnI6qtvVaLT/6m3nT28wlBVFQ7d35T2177V0WjAUmStVaRSIAQCgAAAAAwZfhGnjL1uLGlTnp94LR8+XLH3jUUb1mZyq95n5p/cZuiHR1q/c1dqvrwh1ytIRsVFy9VcfHS4+4tPvlmFRUt0c6d35C1ER0+fK+6u3eoqGiJGhv/rHC4Xbk50zWz9irNmf1heb25GaoeAAAAAIDxy8oVTsc1DXfolDpJmjFjhvx+vyT3T6obUHHDDTJ5eZKkljvuULSHk9IywRijulnX6dSVd8jnK5MkdXS8qoaG3yocbpckBYJHtXfvf+rVjR9QNBrKZLkAAAAAAIxLVgZObq1w8nq9qqvrP5ns6NGj6uvrc+xdw/FVVqrsisslSZHWVrXec4/rNWBQRcUbdfqa38vvrxh2TmvrMzpy9EEXqwIAAAAAIL2yMnAq8nrkNf3XTgZO0vHb6g4cOODou4ZTeeONMrGVVi23/1LRQCAjdaBffv5seTzJt8wdOULgBAAAAACYvLIycDLGxE+qa3PolLoBmW4cLkn+6dNVeuklkqRwU5P23/gBHfn6N9S5bp1sxNnfH0Mb2EY3nFCo1aVKAAAAAABIv6wMnCTFAyenVzjNmjVLHk//P+ZM9XGSpLJLLolf9770klrvuksH/+lj2nf5FQo3N2esrmxVULAg6bjPV+JSJQAAAAAApF8WB079jcPbws41DZeknJwc1dTUSJIOHTqksMPvG07jf3xvyPt9W7eq4bP/4nI1mFV7bdLxtrYXtGfvLbKWFWgAAAAAgMknawOnsoQVTtZaR981sK0uEomooaHB0XcNpW/bNvW8+OKw493PPKPAzp0uVoSamks0s+aKJDOs9u79oV559UYFg6xAAwAAAABMLlkbOJX6+wOniJW6IlFH3zVnzpz4dSa21fVt2TLinN4U5iB9jPFo8eJvadWpv9KM6e9RefkbNXPmlVrzht9ryeLvxJuKt7Q8qRdefLfa2l5SKNShgwd/rR07b1Z9/a3q6zuc4d8CAAAAAICh+TJdQKYM9HCSpLZwRMUJP6dbXV1d/DoTjcM9BQVpmYP0MsaoouJsVVScfdz9kpJTVFyyQps2fUy9vfsUCBzRy+uvljFeWRuKz9u1+/taeNLnNXv2+90uHQAAAACApLJ2hVNZQsDUHnK2r1J+fr6Ki4slSbt27dIdd9yhjRs3Khp1dmXVgMJzzpHJzx923FNYqKKzzx52HO4rLlqs09c8oGnTLordiR4XNvWLaOeum3Xs2Dq3ywMAAAAAIKmsDZxK/YOLu9ocPKkuGo3qgQceUGdnpyTJWqt9+/Zp7dq1Wrt2rSuhk7e4WNWf+MSw4+XXXCNPYaHjdWB0fL5iLV/2I5WVnZ50Xv3+21yqCAAAAACA1GRt4HTcCicHA6fNmzdr48aNw469+uqrjr07UeX7/1E13/yG/LNmvW6sb8sWxxunY2yMMYpGA0nndHS84lI1AAAAAACkJmsDp9LjttQ5FzitX79+XOPpVHbZZVrw14e14C9/1vw//a9yTz5ZktT99NPqWse2rInK48lLOm5MjkuVAAAAAACQmqwNnMr8xzcNd0pra2vS8ba2NsfePRTj8Shn7lzlzp+vGV/6Yvz+0e/8u6KB5CtpkBnV1W9LOh6N9unIkQddqgYAAAAAgJFlbeBU6tKWupKSkqTjA83EM6FgzRqVXHShJCl04IBa/vuOjNWC4c2suVwFBQuGHbc2pC1b/1mbt3xKoVCHi5UBAAAAADA0AidJbQ6eUrdq1apxjTtt2mc/K5PXv2Xr2M9/rtDRoxmtB6/n8xVp9eq7NK36QkmDf29LS9eoZsalkowk6ejRh/TCCxertfV5BYLHtG/fz7Rp8ye0bdu/qrn5SVnrzqmIAAAAAAD4Rp4yNZUlnFLn5AqnlStX6rXXXtOOHTteN7ZgwQKtXr3asXenwl9To8oPfVDHfnSLbE+PGr/3fdX+x3czWhNeLzenSitW/FjB4DH19h5UTk618vNrJUkzZrxXW7d9VoHAEfUFGrR+wzUyxitrB4PUhsP3qKrqrVqx/EfyeHIz9WsAAAAAALJE1q5wKvJ64r+8k4GT1+vVlVdeqYsvvlg1NTXy+QaDrjPOOENerzfJt91R+f73y1/bH150PPSQelxsZI7RycmpUmnpqfGwSZIqKt6oM07/X02bdlHsjj0ubBpw7Nj/ac/eW1yqFAAAAACQzbI2cPIYE99W1+bgKXVSf+i0Zs0affjDH9a1114bv//aa685+t5UefLyNO1z/xL/+eg3b5aNOPvPBOnl95dp+bIfadq0dyadd+jQ3YpGgy5VBQAAAADIVlkbOElSaeykOidXOJ1o9uzZKioqkiRt27ZNkQkS7BS/7W0qOPNMSVLf1q1qu//+DFeE0TLGyONJvks2HG5TIHDEpYoAAAAAANkquwOngRVOYeeahp/I4/Fo6dKlkqTe3l7t2bPHtXcnY4zR9C/8qxTb4tf0gx8q0sGJZ5ONz5f8VERJ8noLXagEAAAAAJDNsjpwKov1U2oPR2Stde29y5Yti19v2bLFtfeOJG/RIpVffbUkKdLaqmM/+UmGK8JoTZ+efEudJG3c9FH19Ox1oRoAAAAAQLbK6sBpYEtdxErdEfeOjK+rq1NxcbGk/j5OYRdXWI2k+hMfl7esTJLUcuevdfR739Ox/7pV3c8972ooh7EpLVmtGdPfm3ROe/vLev6Fi1W//xeytn9LZzDYrGPH1qm5+QlFIj1ulAoAAAAAmMKSN3yZ4sp8gyfEtYUjKvK5c2Kcx+PRsmXL9Nxzz6mvr0979uzRokWLXHn3SLylpaq66RM6+vVvSNGoWm67PT6Wt3y5Zv3kx/JPn57BCpGMMUZLl35XhYULdfDgrxQIHpVkVFl5rmpqLlN9/X+ps3OTotGAdu36jhqP/kl5+bPV1PSwrA1Jkny+Ys2d81HNnv0hGWMy+wsBAAAAACalrA6cShMCpvZwRLNcfPdA4CT1b6ubKIGTJIWODN1Uum/zZh34yEc17/77ZDxZvThuQjPGq7lzP6I5cz6oYLBFXm++fL7+RvXVVW/V/gO3a+/eHyoaDaqjc6M6Ojce9/1wuFO7dn9XktGcOR/KwG8AAAAAAJjssjo1SAyc2kLubmurra1VSUl/g+eJtK0u2tOjtrvuHnY8sG2bup9+2sWKMFbGeJWbWx0PmyTJ4/Fp7pwP6/Q1D6moaEnS7++r/5kikV6nywQAAAAATEFZHTiV+QcXeLWHI66+e2BbnSQFAgHt3r3b1fcPp2/7dkW7upLO6XnpZZeqgVMKC0/SjOnvSTonHO5Qe/t6lyoCAAAAAEwlWR04lZ7Qw8ltiafVbd682fX3D8X4c0ae48vqnZhTyMiN8geaigMAAAAAMBpZHTglNg1vD7n/H9a1tbUqLS2VJG3fvl2hUMj1Gk6Ut/hk+UZoCl70lre4VA2cVFZ+5ohzDhz8lfr6GlyoBgAAAAAwlWR14FTqP75puNuMMfFVTsFgULt27XK9hhMZn0/VN9007Li/rk75y5cNO47Jo6T4FJWXn5V0TnPzo3ru+Xdo//7bFY329xnr62vQoYbf6eChu9Tdnfm/swAAAACAiSer90aVZXhLnSQtX75czzzzjKT+0+qWLEneyNkNZZdeIhsJq+lHtyhy7NhxY6EDB9T19NMqOvvsDFWHdDHGaMXyH2vT5k+otfWZxBFVVJyj3t569fbWKxLp0c5d39Lhw2uVXzBHTU1/U+J2vOqqt2np0u8d15wcAAAAAJDdsjtwSmwa7vIpdQNqampUXl6u1tZWbd++XcFgUDk5I/dRclr5FVeo7L3vVe+rryoaCCrUeFRHvvBFSdLhL3xR8//woLyx7YCYvPz+Mq1edac6Ojaqre1FGeNTZeW5KiiYo0gkoPr9P1d9/U8VjQbV1f2aurpfe90zmo79TVu2florT7k1A78BAAAAAGAiyuotdUVeT/wfQKZWOCVuqwuFQhNiW90Ak5OjgjVrVPSms1V+ySUqefe7JEnho0d15OabM1wd0qmk5BTNnn2j6ur+QQUFcyRJXm+u5s/7hM44/U8qKzs96fePHfs/dXa9PowCAAAAAGSnrA6cPMbET6rLRA+nAYmn1W3ZsiVjdYxkxpe+FG8o3vGHh9Txl4czXBHcUFAwT7PrPjDivNbWZ12oBgAAAAAwGWR14CQNNg7PxCl1A2bMmKGKigpJ0o4dOxQMBjNWSzLekhLVfGtwZdORr35V4aamDFYEtxgz8v9VRMI9LlQCAAAAAJgMCJxiK5z+f/buOzrO6tr7+PeZplHvkiVbzZKNey8YG4gJob60EExCCJhOCEkgCSSQcpPce9NuCgkk1CSkgGmhGGICpmNj3HCRe5dkyepdo9G05/3DtixZ0sg20ozK77NWVjzn7DmzH2OEtXXOPuE6Ugddj9Xt2rUrbLn0Jmb+fBK//GUA/PX1HPrBDzFNM8xZSX9LSJiF1RIVNKao+BGKip8gEBiYBVMREREREREJnWFfcEqwHW4c3uDzhbVwMliO1QGkfefbOHJzAWh+/33qX3ghvAlJv7PZYsnKvjFojN/vYs+en/Px6guprn4Xd1s5O3f9lA9XzOO996ey/pNrqKp6M0QZi4iIiIiISDgN+4LT0SN1PhNc/kAv0f0nPT2dlJQUAHbv3k1bW1vYcumNJTKSzF/9EqyHf+8qf/4LPCUlYc5K+tvovG+Sm/s1LBZnh1ErI9KvICf7diyWw7crtrYeYNPmm/noo4UcPPg3PJ5K/P5m6utXs7nwq+zf/1B4HkBERERERERCZtgXnBKOHKmDgXOszufzDehjdQCRU6aQctutAARcLsq+dx+thYW0fPwx3orKMGcn/cEwLOSP/hYL5n/E5Ml/YtKkB1kw/0MmTvw1BQX3cPrcN0lNvaA93jS7P1q3b//vcLn2hyptERERERERCYNhX3CK71BwCudNdTC4jtUBpNx+O84JEwBoXb+eA1ctonjxDexZuJCDd92Nr64uzBlKf7Db40lLPZ/0tIuIiEhvH4+MzGLK5D8yefLDva5x6NCL/ZmiiIiIiIiIhJkKTh0KTnVeXxgzgbS0NFJTU4HDx+rcbndY8+mN4XCQuPj6rhOBAE3/+Q8lN92MOUBv3JP+Ex2V32uMu0274ERERERERIayYV9wSrDb2n8d7h1OcGyXk9/vZ+fOnWHOpncNL73U45x72zYa31wewmxkIIiISMdiOILG1NWtpL5+XYgyEhERERERkVAb9gWn+AHSw+mojsfq3njjDZ5//nl27NhBIBC+huY9CbhcuFZ9HDSm+Z23Q5SNDBQ2WwzpIy4NGtPWdoj1n1zNps230dKyl0CgjYMHn2Ldui/w0UcL2bjxBqqqlof15kgRERERERE5dbbeQ4a2jk3DG7zhLziVdLjtzeVysXXrVrZu3cppp53GokWLsFqtQd4dWqav9yOIAR2pG5bGFHyPpqYtNDfvOG7GICoqD5drHwDV1W9RXf0uDkcyHs+xY3at7mJqaj8gK+sGxhR8H8MwQpi9iIiIiIiIfFra4WQfOE3Dq6urefXVV7ud27lzJytXrgxxRsFZYmNxFATv1xM1Y2aIspGBxG5PZNbM5xk75kckxM8mJmYCmRmLmDN7KafPfZMpUx4jOnrMkWh/p2JTRyUlf6Wu7qPQJS4iIiIiIiJ9QgWnAXSkbt26dUGPEK1du3ZAHTEyDIPkm27uOcBiIfaz54QuIRlQrNYosrKuZ+bMZ5g751XGj/85sbETMAyD1JTPMmf2a4wb9zN6+zJUduj50CQsIiIiIiIifWbYF5w6H6kL7y111dXVQeebmprwDLAjavGXX0bqN78B3R31CwQo/+//wfSH/6iiDDwWi43MjEVA8P5kra6SoPMiIiIiIiIy8Az7glOszcrR7jDh3uEUFRUVdN5ms2GzDay2W4ZhkPLVr1Lwztuk3/c9Uu64g4yf/S/2UaMAaFmxgqrf/yHMWcpAZRgGEREZQWOaW3Zx6NCLmGbnfz/9fjetraX4/a7+TFFEREREREROwcCqXoSBxTCIt1mp9/lGZx5KAAAgAElEQVTD3sNpypQpbN68ucf5yZMnD6im4R3Z09NJuv769tfOiZM48MUvYra2UvPYYzgnTCDugvPDmKEMVJkZV7H/QM9FyUDAxbbt93Cg6GFG532TxMT57Nv3Gw6Vv0wg0Iph2ElPu5iCgnuJiEgPYeYiIiIiIiLSk2G/wwmO9XEK9y11+fn5TJw4sds5p9PJZz7zmdAm9Ck4TxtL5s/+t/112f3307Z7dxgzkoEqJ+cW4uKmdzsXGZkLR/Ygulz72LL1m6z8aD6lZUsIBFoBME0v5RUvs2791Xg8NSHKWkRERERERIJRwYljN9WF+0idYRh8/vOf57zzziMhIaHTXEREBLGxsWHK7NTEXXghyTffBIDpclFy5534GxvDnJUMNFZrFDOm/4OCgvuIiRmPw5FKQsIcJk74HfNOf4u5c5aRlnphe3wg0NbtOm53CcXFfw5V2iIiIiIiIhKEMZBuPesrs2bNMtetW3fC8Ys27uGDumZsBpScPRXDMHp/Uz8zTROfz8fSpUspLCwEYNGiRUyYMCHMmZ0c0++n5JZbafno8NX20WefRdbDD2NYVOuUk9PUtJVPNlyLz9dz0dLpHMX8M94PYVYiIiIiIiLDh2EY603TnHUisfquH4g/0ojbZ4LLH/zGrFAxDAO73c68efPax1atWhXGjE6NYbWS+ZtfYx85EoCW9z+g+qGHaNu3n5ZVq/AUFYU5QxksYmMn9tqjyeOpxTQHxr/DIiIiIiIiw5kKTkCC/Vgj7nAfqzteZmYmubm5AJSUlFBSMviuiLclJjLqoQcxnE4Aqv/0MPsuuojiG25k7/kXUHT9YhWe5ITExIwPOh8IuFi95mIqKl5rv9XO7S5j374H2Lz5drZtu5eamg8Zijs7RUREREREBhIVnDjWNBwI+0113Rnsu5wAnOPHk/atu7udc61eTdFXrsNXo4bPElzWqOt6jWlp2cWWrd/k49UXsGvXf7Pyo3PYf+BBqqqXc6j8X2zctJjCLV8jEPCGIGMREREREZHhSQUnOhec6sN8U113xowZQ3JyMgDbt2+nrq4uzBmdGk9RcY9zvspK6p56KoTZyGAUHz+d08b+lO6+dGVmXE1Kyrntr12ufZQcfBLoWliqqnqDA0WP9F+iIiIiIiIiw5wKTnQ+Utfg84Uxk+5ZLJb2XU6mafLxxx+HOaNT0/TuO8Hn3w4+LwIwatSXmXf6W+Tm3EF6+iVkZ9/M3DnLGD/+Z0yd8ihzZr9GWtpFva5TWvrP9mN3IiIiIiIi0rdUcOJY03AYeD2cjpo6dSpRUVEAbNiwgdbW1jBndPJMb/AjTKbHE6JMZLCLisohP//bTJr4AGMK7iMm5rT2udjY8Uye9CCJiQuCruHxVOPxDs7dgiIiIiIiIgOdCk5AQsceTgPwSB2A3W5n9uzZAHg8Hj755JMwZ3TyoqbPCDrvnDE9RJnIcBDpzOw1Zt++B2htPXbU0zT9VFe/w569v2bf/j/Q1LS1P1MUEREREREZslRwAuLtA7tp+FGzZ8/Gaj2c6+rVq/H7B26u3Um6YTFYev4jZ42JDV0yMuSlp1/Sa0xZ2RI+WvVZCrd8g6qqd1i95mI2bb6FoqKH2b//96xZeymFW75OINAWgoxFRERERESGDhWc6LzDaaAeqQOIiYlhypQpADQ2NrJ16+DafRE1fTqZv/wFRqSz2/m6v/2N+n+9GOKsZKhKTJxHWtrF3c5ZLE7s9sQjrwJUVv6bzYW30NKyu0tsZeUydu/5ZT9mKiIiIiIiMvTYeg8Z+jreUjeQdzgBzJs3jw0bNgCwatUqJk+ejGEYYc7qxMVfcgkxZ59N4+v/wVdRjn3kSAIeLxU/+QkAh370I6yJicSeszDMmcpgZxgGEyf8ltiY8Rw8+A/aPBUYWElJ/RwF+d8hIiKT8vKXKCp+nNbWA0HXKit7ltF5d2G3x4UmeRERERERkUFOBScgzmbFAEyg3jvwbqnrKC0tjYKCAvbs2cOhQ4c4cOAAeXl54U7rpFjj4ki8elGnMbOlmcpf/wb8fkrvvpvsv/yZqJkzw5ShDBUWi43c3K+Sk3MbXm89VmsUVuuxHXYjR36RzMyr2LLlLiqrlvW4TiDgprl5O4mJc0ORtoiIiIiIyKCnI3WAxTDadzkN9B1OcHiX01GrVq0KYyZ9J+mmm0havBgAs62Nktu/invnrvAmJUOGYVhwOJI6FZuOzVmJjZ3Q6xrFJX+lucORu0DAQ3nFq2zbdi9bt32HQ4dexO9XrycRERERERFQwandYCo4jR49mvT0dAB27dpFdXV1mDP69AzDIO3ee4i/7FIAAk1NlNx8M56DpQCYPh+maYYzRRnCUlLP7TWmuno5q1dfwIaNiykvf4U1ay5j69a7OFT+L8rLX2Lb9ntYveYiWlsPhiBjERERERGRgU0FpyOO3lRX7x34BSfDMIbkLifDYiHjf/6H6LPPAsBXVcWBq65i92fPZcekyew6fR7l//szfLW1Yc5UhpqY6DFkjLiyx3mb7VjvptraD9m67Vu0uLruwGttPUDhljtVHBURERERkWFPBacjEjrscBoM3yxOmjSJmJgYADZt2kRLS0uYM+obht3OqN/9DufUqQD46+rwlR7e5RRoaKDuH//gwJe+pKKT9Llx4/6XnOzbsFqj28eczpFMmPAbzlywhokTfkdc7JRe12lqKqSxcUN/pioiIiIiIjLgqeB0RLztcP90r2niCgTCnE3vbDYbc+cebmDs8/n405/+xBNPPMHKlStxu91hzu7TsURFkfTla3qc9xYVU/3wIyHMSIYDi8VOQcG9LJi/ilkzn2fO7KWcMe9dMkZcjsViZ8SIS5k160Xycr/Z61qNjVu7HTdNc1AUtEVERERERD4t3VJ3RMKRI3UADV4/0VZrkOiB4egOJ4CWlhZaWlo4ePAg69evZ/HixcTFDd4r3JveejvofMPSpaTffx+GYYQoIxkubLZo4uNndDtnGAZxcZN7XePAgYcImG1kZnwBuz0Bl2s/+w/8kaqqN/H7W4mLm0J29k2kp13U1+mLiIiIiIgMCNrhdMTRpuEwOBqHt7a2smxZ99e419bW9jg3WPh7OTIXaGiAQbATTYaexMQzsNkSgsZ4vNXs2fNzVqw8g82bv8qatZdRXv4Sfn8LEKCxcSNbtnyd/fsfCk3SIiIiIiIiIaaC0xEdC071g6DgVFhYiNfr7XF+586dNDU1hTCjvuUYPTrovC0rC2MQ7EKTocdqjWDMmPt6nI+NnYhh2AEIBNqoqn7zSKGpq337H6C1tbhf8hQREREREQknFZyOOP5I3UBXV1cXdN40TRoaGkKUTd9LuHpR0HnDaiXQ2hqibEQ6y8z4ApMn/YmYmHHtY1FR+Uyc8FvmzF7K/PkryB/9bRyOtF5WMikvX9q/yYqIiIiIiISBejgdcbRpOEC9zxfGTE5Mx/5NnyZmoIqcOJH0+++j4mc/73bee+AAJbfeRtYjD2OJju42RqQ/paWdT2rqeXg81UAAhyOtvadYhCOF3Nw7SEpawNp1VwRdp75hDX5/G1ZrRPuYy1VERcVSPN46oqMLGJF+CTZbbH8+joiIiIiISJ9SwemIhEHWw2ny5Mm8/fbbBHroY5SXl0dCQvA+MwNd0nXXETlzJvXPPodn3z6syclEzZlNzSOP4quqwrV2LcU330LWY49ijdU34xJ6hmEQEZHa43xkZDaGYcM0ey5i19auZMXKeWSMuIKMzEVUVv6bAwf+2Clm755fMWnyQyQnLeiz3EVERERERPqTMRSv6J41a5a5bt26k3rPpiYX56/bBcDdOel8d3RGf6TWp1avXs3rr7/e7dzixYvJzc0NbUIh4ikqomjxDfgOHQLAOXky2U88jjU+PsyZiXS1deu3Ka94+VOvY7FEMu/0N3E6M/sgKxERERERkZNnGMZ60zRnnUisejgdkTDImoYDzJ07l2uvvZb8/HwiIiJwOBztcxs3bgxjZv3LkZNDzj/+gX3UKADchYWHC1B1dQTcbtw7d+EpKWEoFlNl8Bk79gedej0dZRgOThv7U7KzbsJuT+x1nUCgldKyZ/ojRRERERERkT6nHU5H1Ht9jFuxBYDPpyfypwk5/ZFav/J4PDz00EM0NjYCcOutt5KZOXR3Q3gPHaJ48Q14iooAsCYmEvB4MFsO3wgWMWE86ffcQ/S8eeFMUwS/3015+UtUVr2B399KfNxURo68hqioXODwbXYVFcvYtv07QddJiJ/LzJlPdxozTZPGxg20uktxRmQQHz+zvZeUiIiIiIhIXzqZHU4qOB0RME1GvrcJEzgnKZanp+b3T3L9bPPmzbz44osAZGdnc8MNNwzpbz69lZUU33ADnr37ug+wWsl+4nEVnWTAM00/7743EdP0BokySE39HBkZXyA56Wyam7exbfu9tLTsbo+IispnwvhfER8/rf+TFhERERGRYUVH6k6BxTCIO3KsbjA0De/J5MmTGXXkqFlxcTHbtm0Lc0b9y56WRvq99/Yc4PdT+evf6HidDHiGYSU15dxeokyqqt5k8+ZbWbFyHuvWL+pUbAJwufayYeP1uFwH+i1XERERERGR3qjg1EH8ECg4GYbB+eef3/56+fLleL3BdkwMfi2r1wSdd2/dire0LETZiJy6vLxvYLVGdTsXETGCqKhjOy+93toed0P5/c0Ul/ylX3IUERERERE5ESo4dXC0cXi9d/AWnACysrKYPHkyAPX19axevTrMGfWvgKul1xiz1RWCTEQ+nZiYscyY/jTx8cd2qBqGnYwRVzJ3zuucPvcNZs16kZEjr6G3L9/V1e93O+5yHaCq+m3q69dhmoP7a52IiIiIiAxctnAnMJDE24/tcDJNc1D3Pjr33HPZvn07Pp+PDz74gKlTpxIbGxvutPpF5OTJ1D/zbI/zRqQTe3Z2CDMSOXVxcZOZNfNZWltL8frqiHRmYbfHt8/Hx00lPm4qNTUf4naX9LhOW1spO3b+kBEjLic+bgYeTyXbtn+X2toP22OczixOO+3HpCR/pj8fSUREREREhiHtcOrg6JE6r2niCgTCnM2nEx8fz/z584HDt9e9++67Yc6o/8RddBG21NQe581WNzWPPqo+TjKoREaOJC52UqdiU0dJSWf0soJJaenTrF+/iI9WLeTj1Rd3KjYBuN0lbN58G/X1J3fJgoiIiIiISG9UcOogwXZsw1fDID9WBzB//vz2XU2ffPIJhw4dCnNG/cMSGUnWY48GLTpV/+lhyr77XQIeTwgzE+k/2Vk3YrE4e5i1YrMltL9yu0vw+eq6jTRNH/v3P9gPGYqIiIiIyHCmglMHR4/UweBuHH6Uw+Hgs5/9bPvrN954Y8ju8nGOH0/+m2+Q8bOfkXjNl0i++SZyn3+Okb/9DYbDAUDj0lcpuelm/A0NYc5W5NOLji5g6pTHcDjSOo07HKlMm/pnzlywmmlT/0J6+qX09qW+tm4FgUBbpzHTNGlo2EBxyV8pLV1CW1tFXz+CiIiIiIgMYcZQLEDMmjXLXLfu5I+IPFhUwf/uO7wL6KXpBcxLiOnr1EIuEAjwxBNPUFZ2+Ja2q6++mvHjx4c5q9ByrV/PwTu+1l5ocuTlMerRR/CVl+NavQasFmLOPIvIyZPCnKnIyQsEPNTUfIDbXYrTmUFy8mewWBydYjZsvIHa2g+CrpOS8llGjLiclOSF+HzNFG75Gg0N69vnDcNGdtaN5Offg2HoZxUiIiIiIsORYRjrTdOc1XukCk6d/L20mnt3HQTgyUl5XJDafe+UwaaoqIi//vWvACQmJvKFL3wBq9VKamoqVqu1l3cPDZ4DByi+7Ta8RcWHB2w28Pk6xcQsXMjI3/waS1T319KLDFZFxY+zZ88vTijWao3GMGz4fN3vBMzPv5fcnNv6Mj0RERERERkkTqbgFLIfUxuGcYFhGDsNw9hjGMb3upn/nWEYG4/8b5dhGPUd5vwd5pb2V44dj9TVH1eMGMxycnKYOHEiAHV1dTz++OM88sgjPPDAA6xZs2bIHrPryJGbS+4zzxA5ffrhgW7++Ta/+y7lP/lJiDMT6X+ZGVdhtyf1OB8RMaL9135/S4/FJoDi4ie6HL8TERERERE5XkgKToZhWIE/AhcCE4AvGYYxoWOMaZp3m6Y5zTTNacCDwIsdpluPzpmmeWl/5dmpafgQ6OHUUVxcXJexpqYmli1bxkcffRSGjELPlphI6r33BI1pePU1vOXlIcpIJDTs9gSmT/sbkc7sTuMWSwRjx/yI+WesYM6cf5OTcztWa2zQtbzeWpqadxw3Vk9R0WNs2LiYDRsXU1T8BF5vY58/h4iIiIiIDB623kP6xBxgj2ma+wAMw3gGuAzY1kP8l4D/ClFu7eJtHXY4DYFb6o5yuVysXbu2x/n33nuPmTNn4nT2dOPV0NG2tac/ckcEArRu2ox9xIjgcSKDTGzsBE4/fTm1tR/Q0rIbuz2R1NTzsNsP32YXGzOO2JhxmCYUFz8SdK2NGxeTlnoBaWkX4XCksnHTDXg8le3ztbUfUlLyF2ZMf4qoqLx+fS4RERERERmYQnWkbiRQ0uH1wSNjXRiGkQPkAe90GHYahrHOMIyPDcO4vIf33XokZl1VVdUpJZkwxG6pO2rv3r34ghwR9Hq97N27N4QZhY/hsPceE+HoNUZkMLJYbKSknENOzm1kZi5qLzZ1lJLymV7X8fkaKTv0HBs3LWbN2ks6FZuOamurYMvWbw6LI7siIiIiItJVqApORjdjPX0X8kXgBdM0O1Z8so80pboGeMAwjPwui5nmY6ZpzjJNc1ZqauopJdlxh9NQKjh5vd4+iRkKYs46C3pplG66XCHKRmTgSYifRVLigh7nExNOx+FI6TAS6DG2qWkrjY2buoybpklLy16amrbh97s/TboiIiIiIjJAhargdBDI6vB6FFDWQ+wXgSUdB0zTLDvy//uA94DpfZ8ixA3RI3UjR3a7mayTUaNGhSCT8LOPGEHStdcGjSn91rep/O3vMP1D58+AyIkyDIPJk/9IevoldPxPhM0Wz9ix/8WMGU+xYP5HzJj+NAkJp/e63qFDL+DzNbW/rqp6k49Xn8/Hq89jzdpLWLFyHnv3/ZZAYOhc1CAiIiIiIqHr4bQWGGMYRh5QyuGi0jXHBxmGcRqQCKzqMJYIuEzTbDMMIwWYD/yqP5K0GgZxNguNvgANQ+iWuvT0dEaPHs2+ffu6nc/MzCQlJaXbuaEo7d57sMTEUPv3vxNoOvyNsCUxkYj8fFrXrQOg5rHHcG8pJPPXv8aW1PPtXiJDkc0Ww6SJD1CQfy+NTYVYLU4SEuZitR7u82YYVhIT55Ltu576+o+DrlVatoSyQ/8iKWkezohRlJY91Wne52vkwIE/0tZWyYTxv+i3ZxIRERERkdAKScHJNE2fYRh3Am8AVuAvpmluNQzjp8A60zSXHgn9EvCM2bnpx3jgUcMwAhz+cfsvTNPspfPzqUuw2Wj0eYbUkTqAK6+8kqeffprS0tIuc42NjbhcLqKiosKQWegZViupX7+T5JtuxL19O1gsOCdOxOJw0PDavzn0wx9itrbS8tEq9l/5BTL++79p3byJ5nffw/T5iJo+ncSvXEtEnpohy9DmdGbidGb2OJ+UdDZ2eyJeb13QdUzTQ03N+0FjDh16nuzsm4iJHnNKuYqIiIiIyMBiDMWGrrNmzTLXHdmpcrLOW7uTzc2tpDpsFM6f1MeZhVcgEGDfvn3tO53Ky8vbfz1hwgSuuuoqDKO7dlvDi3vXLkq/8U08Bw70GGNERDDqoYeIObPnXjciw0FFxWts2XoXXdvyGYwpuB+fv4Wqqjdpbu795wR5uXcxevTX218HAh7KK5ZSUfEaXm89MdFjGTXqWuLipvTtQ4iIiIiIyAkxDGP9kR7bvceq4NTZVRv38GFdMw7DoOjsKUO6AON2u3n44YdpaGgA4PLLL2fatGlhzmpg8Dc3c+i++2ha/laPMZb4eMa89y6WyMgQZiYy8NTWfsSBooepq1sFGCQlnkFu7ldJTDzW46mi8nW2bLkz6DpWaxQj0i8jJeWzxMVPp7DwDurrV3eJGzv2x2SN+kpfP4aIiIiIiPTiZApOoerhNGgcvanOY5q0BkyirEO34OR0Orniiit48sknAVi2bBk5OTkkJiaGN7EBwBoTQ+q3vhW04BRoaKBp+XLiL700hJmJDDxJSWeQlHQGgYAPwzAwjK43QSYnLcBicRII9Hwrnd/vorRsCaVlSzAMG6bZfS+9Xbt+QlLiGURHd7mwVEREREREBohQ3VI3aCTYjtXghlLj8J7k5uYyf/58ADweDy+99BKBQM/XnA8n3tKeLlI8xlNSEoJMRAYHi8XWbbEJwGaLJTNzUY/vtVpjsNuPNejvqdh0ZJaysme7jHq9dRw8+E/27P01paVLOt2OJyIiIiIioaUdTseJtx/7Zqne6ycjIozJhMjChQvZu3cv5eXlFBcXs3LlSs4888xwpxV2trTUXmMMHacTOWEF+d+jra2cqqo3O41HOrOZNu0vREZm09C4kYqK1zh48O9B16qs+g8JiXNJSjwDqzWSQ4deZMfOHxAItLXH7N7zMyaM/zVpaef3y/OIiIiIiEjPVHA6TrTl2KavvS4342OGfkHBZrPx+c9/nsceewyfz8e7775Lfn4+mZk93041HESMGYNz4kTcW7f2GFP7xBNE5OURu3BhCDMTGZys1ggmT/oTjY2bqKpeTsDvJj5+Bqmpn8NicQCQED+TuNjJlJU9F/T4ndtdyubNt2KxOIiJmUBj48YuMX6/iy1bv8HsqFeIjRnXb88lIiIiIiJd6UhdB/8sq+Gh4or217dsLeL2rQdo8vnDmFVopKWlce655wKHb7N78cUX8Xg8Yc4qvAzDYMRPf4IlJqbHGH9tHQe/egeHfvxjAq2tIcxOZHAyDIP4+GkU5N/D2LE/JD394vZi01EWi4P09EtOaL1AwNNtseko0/RRUvLXbsZNGho+oazsBaqr3+m0M0pERERERD493VJ3xPPltXx9e3G3cwsSYnh+Wv6QvrEODheannrqKfbu3QvArFmzmDRpEj6fj4yMDKKjo8OcYXh4Dhyg5s9/pund9zC9XqJmzCDxmi/R9OZy6p9/vj3OkZdH+vfvp+XDD2lc9jr+5macp51G0nVfIfbCC4f8nx+RvtTWVsm69Ytwu7v2ScvM/BIZGVdQU/0u1TXv0dy8PehadnsKs2e9QGRkFgAtLXvZsvWbnd5ntycz7rT/1vE7EREREZEgTuaWOhWcAL9pMu/j7RS7e97R869p+cxPjO2L9Aa0xsZGHn74YVqP261jtVqZNm0aF1xwAXa7PUzZDTxNb73FoR/8EH99fdC45FtuIe3b3wpRViJDg8dTzYGiR6moWIrX20hMzBhGjbyOjIwrOxVwV6ycT1tbea/rRUWNJjFhLhWVy/D5GrqJsDJj+j9JTJzTh08hIiIiIjJ0nEzBSUfqgD2utqDFJoC3ahpDlE14xcXFMWnSpC7jfr+f9evX869//SsMWQ1cseeeS97SV4jupcl6zeOP496xI0RZiQwNDkcKY8d8nzMXrOachduZM3spmZlf6LJbMDX1xHYluVz7KC1b0kOxCcDPgaI/9fDe/ZSXL6Wq6i38ftfJPIaIiIiIyLCkghOHdzj1HhOCRAYAv9/PjiCFkR07dlBWVhbCjAY+e1oaIx/4HViC/+vU8NLLIcpIZHjJyb4Zuz2x2zmHI5WCgvtITT0Pq7XnfmxH1dauoK2tqv2111vHpk23sOrjc9m67W42F97GipVncPDgP/ssfxERERGRoUi31AEFURGk2G1Ue309xpyR0Ps3KkNBRUUFTU1NQWN27tw57G+wO16gsRECgaAxvqqqoPMicmqczkxmzFjCzp0/or5+Tft4UuICxo37byIjs4GbCQS8rFv3BZqatwRZzWTFytOJjZlIYtIZVFe/g8u1t1OEz9fEzl3/hdUaRUbG5/vnoUREREREBjkVnACHxcId2Wn8dG/3O3fGRzs5NzkuxFmFh9/f+418gV4KK8ORNTkZS1QUAVfPR218NdWYXi+GemCJ9LmY6DHMnLEEl6uItrYKnM5MIiNHdYqxWOykpV/US8HpsKbmrTQ1bw0as3//g4wYcTmG0Xl3Y3PzTurqV2NgISnpTKKick7+gUREREREBjkdqTviq1mp3Jmdhu24i8TGREbw1JTR2CzD44ax9PR0IiIigsZkZ2eHKJvBw+JwEH/55UFjXKvXsH/R1bRuCf5NrIicuqioHBIT53QpNh01MnMRDkdqD+82GJl5DUlJZ2KxBP86CNDqLqaubhVHL9/w+ZrZtPk2Vq+5iF27fsLOXf/Fqo/PYdu2ewkE2k71kUREREREBiXdUnec8jYvvz1Qzt/LagD4SX4mt2Wn9WV6A97bb7/Nhx9+2O2c1WrlrrvuIjZ26N/Yd7L8zc0U33Aj7sLCrpM2G/iOHNm0WEhavJiUr91BywcfUP/yy/iqqnBkZZOw6Cpi5s8PbeIiw0xzy262bvkmzS0728fs9iROO+2npKddCIDf38aWrd+guvqtXtdzOkeSmHgGTU3baO5hV9TIkV9m3Gk/7ZsHEBEREREJk5O5pU4Fp24cdHuYtWobAF9IT+ShCcPrOITf7+f111+np9/DrKwsrr/+emw2ncg8XsDtpuHlV2hctgx/cxPOceNJvOYarLExHPrRf+Favbo91oiMxGxt7bJG0k03kvad73S5iUtE+o5pmjQ0rMfl2ofdnkRyctddTZVVb1BYeEeffJ5h2Jg/fyURjpQOOfipqXmfquq3MU0fCfEzSU+/BKs1sk8+U0RERESkr6ng9CkLTqZpMnHlFmq9fsZGOflg7rg+zG7wqK2tZefOnfh8PtLS0li+fDnV1dUATJkyhSuuuEJFkZNgmiYNL75IxSbJe/IAACAASURBVC9/dbjJeBBZTzxBzALtdBIJp0DAx9p1l9Hc3P3Nnelpl2Lip65uFV5vba/r5eTcTk72rdjt8fh8TWzcdDMNDZ3/W+WMyGTatL8SHV3QJ88gIiIiItKXVHD6lAUngC9u3Mt7dU1YgN1nTSbaau2b5Aax2tpaHn/8cVqP7Mo555xzOOuss8Kc1eDjq6pi70UXEwhyG2Dseecx6g+/D2FWItKdtrZKCrd8vVNhyGKJIC/36+Tk3I5hGJhmgL37HqCo6I8nsKJBbOwE/P5WXK593UZERuYy7/Q3MQz9d0dEREREBpaTKTjpTFQPpsRG8l5dEwFgW7Ob2fHR4U4p7JKSkvjiF7/I3/72NwKBAO+88w5JSUlMmjQp3KkNKrbUVMxebgP0FBeHKBsRCSYiIo1ZM5+lsXEzjY2FWK1RpKQsxG5PaI8xDAsjM6+iqOhPQG8/xDFpagp+cUBr6wFqat4nJeWc9jGPp5aSkr9QUflvfL4mYmLGkzXqelJTz/0UTyciIiIi0n90S10PJsdGtf96U1PPV90PNzk5OVx66aXtr19++WUOHjwYxowGJ1tKStB5f10dvtrej+iISGjExU1h1Kgvk5FxRadi01GRkVlkZlzV4/tHjPg8o0d/i8TEMzAMR6+ft//AQ1RULqPNU01bWyXr1l3JgaKHaW0txuuto67uIzYX3sa+/X/4VM8lIiIiItJfVHDqwZTYY01bC5u6NnYezqZNm8aZZ54JgM/nY8mSJdTX1+P1emlpaSEQCIQ5w4Ev/vLLgs77KirYe/4F1P79H5heb/u45+BBmleuxL1zF0PxOKzIYHbaaT9h1KivYBj29jGLJZLc3DuZMP6X5OV+jRnT/8H0aX/vda3Gxk1s2fJ1VqyYy6qPP0eru/tdj/v3/57m5p1dxk3TpKFxE1VVy7udFxERERHpb+rh1APTNBm/Ygv1Pj/jo528O2d4Ng7vSSAQ4Pnnn2f79u0ARERE4PF4ME2TmJgYZs+ezYIFC7Cq91W3Ai4XRdcvxl1Y2GXOcDgwPZ721xFjCki+7TYaXnqZlpUrj41PGE/Gj39M5JQpIclZRE6Mx1NNQ8MnYFhJiJ+N3R7Xad40/Xy06hzc7r7ZHZo16kbGjv1+++v6hvVs334/Ltee9rH4uOlMmPB/REXl9clnioiIiMjwpKbhfVBwArhq4x4+rGvGasCeM6cQadWGsI48Hg9PPPEElZWV3c6PHz+eRYsW6Sa7HgRaWqh58kkaXnoZX3U1jqxRJCy6mrhLL6H2iT9T++STnXY3dceIiiLv+eeIyM8PUdYi0hcqq96gsPBrdNfzKTv7NtLTLqCufg21NR9SW7ci6FqGYScpaQEJ8bOIcKazffsPME13lziHI425c17D4Ujuq8cQERERkWFGBac+Kjj9dE8Zfyo5XExZNmMMM9Q4vIunn36aXbt29Th/7bXXUlCg671PhefAASp+8Uua33svaFz8ZZeR+ctfhCYpEekz1TXvsW/fb9ubiDsjMsnOuYVRI7/SXqj3+9t4/4MZ3RaQTsXovLvIy/t6p7G6ujWUli2htbUIhyOVjBFXkJp6HoahH7KIiIiISGe6pa6PdOzjtLm5VQWn4/h8Pvbs2RM0prCwUAWnU+TIzSXrkYfZ+/8uwRPk97npnXdCmJWI9JWU5M+QnHQ2Hk8lgYAXpzMDw+h8DNlqjSAz4wpKy5b0uE5kZDatrSd2s2XZoX+RmnYB0VH5GIaFffseYP+BBzvFVFe/RVrqhUyc+AAWi/6aICIiIiKnRn+TDGJKh5vqNuumui48Hk+vDcLd7r75qfxwZomOCjofcLsxTVNHF0UGIcMwiIhIDxqTn38PDY0baG7e0WVu7JgfkZV1PR5PLQ0Nn1C45U5Ms+ejuG53CatXX4DNFkdUVB6NjZu6jausep2Esjlkjbquy1xLyz6am7dhtUaTmHgGVmtEL08pIiIiIsOR9ssHkRvpIPZI3ybdVNeV0+kkNjY2aExaWlqIshm6ombMDB7g9VL0xS/RsmZN5+HKSppXrqR10yZMv78fMxSR/mS3xzNzxnOMGfMD4uKmERWVT3ra/2PmjGfJyroeAIcjidTUc0lOPvuE1vT5GnssNh118OBTnV57PNVs2LiYj1d/ji1bv8mmzTez8qP5lJW9cGoPJiIiIiJDmnY4BWExDCbFRrKqvoUdLW7aAgEiLKrRHWWxWJg9ezbvBDnSZdHv16eW+OVrqFuyBDPIbrHWTZsovu56os8+i5TbbqPu6SU0vv46HCk02UeOJP2+7xF77rmhSltE+pDNFk121g1kZ90QNC4n+xaqq98Buu4+tVic5OV9k1bXfhoaN9DSsjvoWi7XHjZuupG4uOnExU5iz97/o6VlZ6cYr7eO7Tu+i9UWTXrahSf9XCIiIiIydKlpeC/+a08pj5ZUAfDGrLFMjQ1+vGm48fv9/Otf/2Lbtm09xpx//vnMmzcvhFkNPS2rVlH6rW/jr6trHzMiIki87jrcGzfiWru290UMg6xHHyHmrLP6MVMRCbfyilfZseMH+P3N7WMORxqTJv6exMQ57WOfbLieul5uwDtR0dFjmTtnWaejvTU1H1JU/Bj19WsxDBspKQvJzbmD2NjxffKZIiIiIhJ6uqWuDwtO/yqv5WvbDzdj/b/TRvGVzJQ+WXcoMU2Tffv2UVhYiNvtJi0tDbvdzttvv90eo6LTpxdwu2lavhxPUTG2lGRizz8fW2IipmnS8uGHVP7mt7Tt3Bl0DeekSeS98HyIMhaRcPH5WqiqXo6nrZLIyBxSUhZisTg6xVRU/JstW7/R4xp2eyJeb12P88cbO/YnpCSfjdM5ikOHXmD7ju91ibFYnEyb9iSJCbNP/GFEREREZMBQwakPC067W9ycueZwo9brMpP51WlZfbLucLBu3Tpee+219tcqOvUvMxCgePENuI7r5XS8gg/ex67eWiLDnmn6KSz8GlXVy7vMxcSMZ+aMJZimSWPTZvbs+SXNzT3vZO3IbkvA62uku2N9AFFRBZw+9z+ddkP5/a2UVyylpuY9TNNPYsJcMjK+gN0ef0rPJiIiIiL942QKTurh1IvRURFEWS24/AE26aa6kzJr1iwMw+DVV18F4I033sA0Tc4444wwZzY0GRYL9pEje43zVVSq4CQiGIaVSZMe5GDpPyktXUJraxEORwoZI64gJ+c2bLbDl0IkJy3Ak13Ntm3fPqF1vb76oPMu1x7q61eTmHg6AO62cjZsuA6Xa297THX12xQVP8b0aX8nJua0U3xCEREREQkn7XA6AZd9spvVDS04DIO9Z03BbtH18ydj/fr17UUngIKCAmpra2lsbCQxMZEZM2Ywe/ZsbDbVPz+t+hde4NAPfhg0xnA6Sbz6apJuuhF7Whreikrqn30W17p1GDYbMZ/5DPGfvwJrTEyIshaRgS4QaGP1motxufZ3O5+beycORwqNjZuoqXkfr7e21zWjokYTFzuFxsZNuFq7XzcyMod5py/HMKydxltbS6isegO/30Vs7ERSkj/TJUZERERE+p6O1PVxwekHuw/yxMFqAN6aNZZJahx+0j755BOWLl3a43xBQQFf+tKXsFr1DcOnEWhtZe9FF+E7VN5rrBERQczChTR/+CFmS0unOXt2Njl/exJ7RkZ/pSoig0xraylbttxJY9Pm9jGLxcno0XeTk31z+1hF5ets2XJnn33utKl/ITn5bABMM8CePb+guOQvwLG/v0RG5jJ1yuNER4/us88VERERka5UcOrjgtNz5bV840jj8N+Oy+KajOQ+W3s4ee6554LeZnfppZcyY8aMEGY0NLXt3cvBO76Gp6jo2KDFQtJ112FNSqL2r3/tdNtdT6LPmEf2X/7Sj5mKyGBjmiYNjZ/Q3LQdqy2GlOSFXfosBQIeVn50Nh5PZbdrOJ2jiI7Op7Gx8IR2QjkcKSQlLSA2dhIu1wFKS//Zw7ojOX3um1itzuPy8VJXtwqPp5qoqDzi4qZ16h8lIiIiIidOPZz62OSYyPZfb25q5Rpt+jglNTU1Qec3btyoglMfiMjPZ/Syf9P8wQe4t2/HGhND7Oc+175bKenL11D3zLNUP/IwgabmHtdp+WgVnqIiHDk5oUpdRAY4wzBIiJ9JQvzMHmMsFgeTJz3Ixk034vd33j3pdI5ixvSniIwchWmaVFa+zpatXw/6mR5PNeXlL1Ne/nLQOLe7lMrKZWRkfL59rLr6Hbbv+H6n4ldMzHgmTXyA6OiCoOuJiIiIyKejgtMJGBPlJNJi0BowKVTj8FPW1NQUdL65uefih5wcw2olduFCYhcu7DJniY4m+aYbCbS2Uv3QQ0HXadu/XwUnETlpCQmzOH3uG5SWPkVd/Vosho2UlHPIyLgKuz0OOFy8Sks7H+fekbjdpT2sZMFmi8Hnazyhz923/wHc7lJiYydimn42F34N8HeKaW7ezicbrmXunGU4HEld1mhu3kWru4SIiHRiYyZqN5SIiIjIKVLB6QTYLAYTYiJZ3+hiW3MrvoCJTY3DT1pSUhIuV88Fu/h4XX8dSrb03m+qq/rdA+D3E7NwIYbFgunx0Pjmcpo/eB98fqLmzCb+kkuwREeHIGMRGUyczgzy878TNMYwrJw29sds2nw7xxeGAMaO/RGjRl6L211KTe0H7NwZ/FIEt7uUffsf6DU3j6eKsrJnyM29o32spWUf27ffS0PjhvaxmJhxjB/3c+LipvS6poiIiIh0ph5OJ+i+XQf5a+nhxuHvzj6N8R2O2cmJ2bBhA6+88kqP8zExMSxevJiUlJQQZjV8+Rsa2H32ZzDd7l5jHbm5JCy6ivqXX8Gza1enOVtGBtl/foKI0WrWKyKnpq5uDfsPPEhd3UcAxMZOJjfnq6Slnd8eY5oma9ZeQnPz9j75TIcjjbzcO4mJHYfDkcr69YvweKq6xFmt0cyZ/QpRUXld5lpbS2lp2YXNHkd83DTdlCciIiJDnpqG90PB6elDNXxrRwkAvx+XzdUZXbfhS3CBQICXX36ZzZs39xjjdDq5+uqrycvr+hd76Xt1zzxL+Y9/3HXCZiOioIC2HTtOaB1Hfj6jX12KYbH0bYIiMqz4/W2AH6u1+9tgq6vfYdPmW+l4Q91RiQmnM2nSH2hu3kFj01b27v0/INAneWVmLGL8+J+3v/Z4ati+436qq99uz8XpHMXYMT8kNfXcPvlMERERkYHoZApO+u7wBE2NPfaX38Jm9XE6FRaLhcsvv5xFixYxduxYRowYwbhx47jyyivJOdInyO12849//IMNGzb0spr0hcQvXk3Wo48QNWsWWCwYdjuxnzuX3GefYfTLL5H7/PPEXngB9FJI8uzdS8uqVSHKWkSGKqs1osdiE0BKyjlMnvRHnM5R7WOGYSNjxJVMmfIoDkcySUnzyc25leTkM/ssr/KKVyktXUJDwwY8njo2bryB6uq36Fj4crsPsrnwq9TWrux2jdbWUioqXqOq6k18vuA9DUVERESGAu1wOkHegEnBh5tpC5jMiY9m6Ywxfbr+cOfz+XjttdfYuHFj+9iCBQs4++yz2b17N9XV1URHRzN+/Hiionr+ZkROnRkIgGF02yC37plnKP/xT4K+P/U73yHl5pv6Kz0RkXam6aepaSs+XzMxMWNxOLoexa6rX8snn3yZ7npD2WyJTJ3yKG1t5TQ1b+fgwb93uVHvVMXHz2LWzGfbX/t8LezYcT8Vlf/maIHKaokiN/cOcnJuV1NyERERGVR0pK4fCk4AF6zbxcYmF1FWC7vPnIxVf0nsU6ZpsmLFCt5+++32MZvNhs/n6/T6/PPPZ/bs2eFIcdhqWb2G4uuvDxpjTUgg+Zabif/857ElJuKtqKDm8SdofOM/BFpcRE6YQOL11xH3uc+FKGsRGe4qK//Djp0/xOutbR+Lispn0qQ/EBszrn1s584fc7D0H332uaPz7iYubioxMWPZtv0+amvf7zZuTMH9ZGd3LdQ3N++itnYFJgGSEs8gNnZCn+UmIiIi8mmo4NRPBad7d5bw97IaAD6YM46x0c4+/wyBrVu38tJLL3UqNB3vmmuuYezYsSHMangzfT72fPZcfBUVvcYaERHELPwMro9X46+v7zKf8o2vk3rHHd28U0Sk7/n9bdTWvo/HU0NUVB4JCXMwjM7HhF2u/axe8/8IBLq7RMHCxAm/BsNCc/NODh16odvm4qfCbk9k/hkrsVojjuTqZtv2e6isXNYpLjl5IZMm/g6bLbZPPldERETkVKmHUz+Z0rGPU5P6OPWXiRMnMnny5KAxK1asCFE2AmDYbKTfdx/0sKvPOXEiht0OgNnWRtN/3ui22ARQ/YcHadu/v99yFRHpyGqNIDX1PEaO/BKJiad3KTYBREXlMXXKY9jtSce9N4ZJEx9gxIjLGJF+CQX53yE3t+8K5l5vHdu2f4eysudoaNjA9u3f61JsAqipeZdt2+7pMm6aAWpqV7Bjxw/Yuu3blBz8u/pDiYiIyIBhC3cCg8nk2Mj2X29uauXKEWFMZoirq6sLOl9cXEwgEMCiW9FCJu6C87FEP0b1Qw/RumkTcPh2upTbbiX+0kvx1dZS//wL1D39dK87oRpeeYW0u+4KRdoiIickKWk+889YQXX1clpbDxIRkU5q6nnYbNGd4jJGXMGBA4/g8XT/dW5Mwf1ERY2muXkn5RVLaWnZGfRzKyuXdVtkOl5V9XJaWvYQHV0AHN4NVVj4VWpqP2iPKS9/mf37/8DUqX8mPm5qr2uKiIiI9CcVnE7CuGgndsPAa5ps1k11/cpqtQadN3pobi39K+bMBcScuQB/fT2m3481Kan9n4MtKYmU224l5pyF7L/k0qDruNavJ+B2Y3EeO5batn8/ja++hq+mBkduLvGXX4YtMbFfn0dEpCOrNYL09P8XNMZmi2X6tCfZXHg7ra1F7eOGYWN03t3tPZlSUhYSnzCTTz75Yp/lV1j4NZKSzyQ6qoCa2hWdik1Heb11bNp0C/PPeK/TjX9+v5uysmcoL38Fj7eO6Oh8Ro28lpSUhX2Wn4iIiEhHKjidhAiLhXHRTgqbWylsaiVgmlhU9OgXY8eOZe/evT3OW61WDh48SFZWVgizkqOsCQk9ztkzMjEcDkyPp8eY1rXr2H3W2cRfcgnxX7iS5rfepvqPf+wUU/X735P5f79Sk3ERGXBiYsYy7/Tl1NR8QHPzDmz2eNJSz+tyW15C/CwSEuZSX7+623Vycu4kPe0CWlp2U1n5H6qq3wj6uS2uPbS49vSan9dbQ3n5vxk58irg8E15GzZeT2PjhvYYt7uEmpr3yM35Kvn53+l2HZ+vCY+nGocjRf2jRERE5KSpafhJ+vaOYp46dPi2m4/mjmd0VES/fM5w19bWxqOPPkptbW2PMYZhsHDhQhYsWKCjdQNM2f3fp+HFFz/1OobdTt7SV4jIy+uDrEREQs/rrWfr1rs77UYyDBtZWTdQkH9ve08pr7eeFSvPIBBoC7KaBQic0Ocahp3Y2IlER42m1V1Gff3HPcbOmvk88fEz2l+3tVWxZ8/Pqaj8N6bpwzBspKVdyJiC+4iISD+hzxcREZGhSbfU9WPB6cnSar636yAAj0zI4fJ0HfnpLw0NDSxdurTTTqekpCRiYmIoLi5uH8vLy+OKK64gLi6O+vp6Dh06hMPhICcnB5tNm/jCwV9fT9H1i2nbeVzvEquV1Lvvwl9bR8PLL+MPUlA8KvG6rzDi/vv7KVMRkdBoat5BQ8MnWCwOkpPOJiIitUvM3n2/5cCBP3bzbhg16joK8r+Hq3U/dbWr2L3nf/ost+TkzzJp4m+w2WLxehtZu+4KWlsPdImLjMxm9qwXsds7/90nEPBRU/MuLS17juz2ugCHI6nL+0VERGTwU8GpHwtOnzS2cNH63QDckZXGjwoy++Vz5JiamhpqamqIjo4mM/Pw7/e6dev4z3/+g9/vByAyMpKUlBRKSkra3xcVFcW5557LjBkzul1X+lfA5aL+pZdoev0/+F0tRE6cSOKXv4xz3DgATI+Hxnfeoeyuu4OuEzF+PKNf6rxbyt/YSOOyZXhKSrCnpxN38cXYkpP77VlERELBNAPsP/AQxcVP4Pe3AGCxOMkadR2jR38bi8V2JM5k7brLaWra0uNa0VEFuNvK8PtPvOekw5GKxRKB232wx5i83K8zevSxSx8am7ZQWHgHbndp+5jFcJBfcC/ZWTec8GeLiIjI4KCCUz8WnFr9AQo+3IzfhAUJMbwwvaBfPkd6V1FRwfPPP091dXXQuCuvvJLJkyeHKCs5WTtnziLQ0hI0xjl5MvGXXkrcxRfhWr+eQ9/9HgHXsW+iDLud9B/+gMRFi/o7XRGRfufzNdPQsAEIEBc3Hbs9rktMXd1qNmy8DtP0dZnLzPwi48f9L6YZwO0+xOo1F+H3N/dJbjZbAmMKvkdUVB52exLr1l+Fz1ffbeykSQ+RnnZhp7HGpi2UlDxJY+MmLBYnqannkTXq2i67pkRERGRgUsGpHwtOAOes2cG2FjfxNis7FkzSbWlh5PF4eO6559izp+cmqklJSdx5553q8zRAnVS/J6sVAgHo4etW9pNPEn363D7MTkRk4KqvX8fefb+hvn4NAA5HGtlZi8nOvqW9NxTA7j2/oLj48R7XSU+/jEDAjcu1j5aW3X2WX2zsJObMfqX9dXnFq2zd+m3A3ynO6RzFzBlLcDq77hr3+VpobNoMQHzc1E4374mIiEjonUzBSQ1uTsHk2Ci2tbhp8PkpdnvIiVTj8HBxOBwkJycHLTjV1tZSV1dHso5cDUgpt99G09tvE2ho6DJny8zEkZODa/Xqw4Umv7+bFY6pVcFJRIaRhIRZzJyxBI+nBr/fTUREevuxu47ycr9GXd1HNDVt7TKXlXUjY8d8v/31ps23U129vE/ya2rawrr1i4iOyscRkUZR0aMcX2wCcLsPsnPXj5k65bH2MdMMsH//gxSX/Ln9eKHVGkNO9i3k5t7RqaAmIiIiA5MKTqdgSmwkz5Yf/vXmplYVnMIsEOj9xp4TiZHwcGRnk/v0U1T87Oe0rFwJgOFwEHfxxaR/77tY4+PxVlbS+Nq/qXrgAUyPp8e1XGvXYvr9GFZr+1jb7t3ULVmCe+curHFxxF10EXEXXoChhvIiMkQ4HMF/oGKzxTJj+hJKS/9JefkreLx1REfnM2rktaSmnt8pNjvrxqAFp9PG/gSbLQ6Xaz9lh56jra086Gc3NKynoWF9r89QXf02tbUfER8/Has1kr17/4+i4sc6xfj9zezb/zsCpof80d86bs5NecUrVFUtJxBoIz5+Ov+fvfsMjKpKGzj+v9MnmUwySSY9IYEUQg0QARGUVURFXTsothVdy9p17bv2VdfXXcvqqqi4uqgoYO+4FpTQSygBQnrvZTIlU+/7ITAwzAxYAgQ4v2/33GfOPTfilOee85zUlIvQ6ZL3e21BEARBEA4MsaTuV1jdbePMdX1Tzm/MSOC+IaJw+KG0ZcsWFi5cuM+YCRMmcOKJJ6LViuTgQOZpbcXT0Yk6JRllVFTQ+crzL6B3c/giuQAqs5moU0/FeNppOCsqaLr//r7ZUXuImDCB9JdfQqHT9ev4BUEQjgS1dW9SWvoIsOd7p4Kc7HvIyJjjb2loeI+t2+4J249SGYEse/D5wj8oCEWjMeNytYY9r1BomXzcMn/dJ5ernfXrL8VqC9wZVamMYPSoVzCZJgb1IcsyVus23J4uIiOyQ+4aKAiCIAhCMFHD6QAnnGxeLzlLN+EDppqiWFAw5IBdS9g/r9fLSy+9RGtr+C+nANHR0Zx++unk5uYC4HQ6aWpqQpIkkpOTUavVB2O4wm/Q9tJLtD7zbL/0FffHq0i4/fZ+6UsQBOFI43DU0tj4Pr299Wh1ySQnnUtExKCAGK+3l9Vrzg5Z90mSVIwd+zbRxgJ6e5uorn6R+oZ3+m180dGFxJqORa/PoKFxMV1dK0LGqdUmjpu0NKD2U0fHMkp3PLLHuJUkJJzC0LyHRfFyQRAEQdgPkXA6wAkngONXbqPU3kusWsmW40Th8EOtu7ubd999l4aGBn+bJEnk5uZSU1ODw+Hwtw8fPhyj0ci6detwOp0AREREMGXKFCZOnCj+Ww5g3q4uKs87H3d9fdA5hcGAYcoUrEVFIetBBcVHR5O77KegpXVeqxXn9u1IGg26/Hyx9E4QBGEfnM5Wtm67m/b27/1ten0GebkPERd3/O44VxvLlk1Glt0h+4mIGEJc3Ak4HNVYLJtwuVr6bYxDhtxBRvocFAoNXd1rWbdudsjd/aKiRlA4biEKhcbfJss+WtuW0NiwiF5nE3pdKikps4iLmyq+LwiCIAhHJZFwOggJpxtKqlnU3AnAmmOHkabT7OcVwoEmyzI1NTU0NDSg0WjIy8vDYDBgtVr56quv2LRp0377mDZtGpMnTz4IoxV+LXdDA00PPYx16VL/bnUR48eTdP9f0WZnI7tc2FasoPHBh/DskYAMJeX/niTqlFNQaDTIbjctTz9D5zvvIO9MUKqSkjDffDMx55x9wO9LEAThcOZw1GKzlaFWx2A0jg5Z1Lu+fgHbtt8X1K7RxDNu7AIiIrIAsNkqWLHy5H4eoQKdNgmP14rHYwkbNXzY0yQl/R7oSzaVlNxBU/OHQXGpqZeQl/tgUNJJlmW6ulZht1eg1sQSFzsVpVIs5xcEQRCOHCLhdBASTnNrW7i/rO/H7LwRmcwwxxzQ6wm/XVlZGR9//DEWS/gvmlqtlttuu03UejoMuJuacDc0oEpIQJOWFnS+8cEH6Vrw7n77UURGYjjheNwtLTjWhC5sm/z44yLpJAiC0A86O1dSUzsPi2UDCoWOBPMppGfMQadNCojbuOk6Wlu/DtmH2TyDvLwHcDiq6epcQ3nFk/02Pq0mkYSE09Dp03A4aqmreyNs7KiRL2I2T/cfW63b2bzlhRHfbAAAIABJREFU5oAlhmq1ibzcB0lMPCNsPz6fC0lSiZ33BEEQhMOCSDgdhITTii4rZ68vA+DWQYncNVjsgnI4WLlyJV988cU+Yy666CLy8vIO0oiEA8W+bj3Vs2f3S1+qxESy//eNWF4nCIJwkHg8PWzefBPtHUsD2uPjTmT48GdQqSL9batWn01PT7hZzBJJSWfjdrVjd1TjcFT32xijDCMYPvxpdLpUvF4bK1edhsvVFiJSwZgxbxJrOtbfIssyjY2Lqa2dh9W2HUnSYDafzOCsm4mMFLVBBUEQhIHrlyScxK+nX2mEQY8EyEBxj/1QD0foRy7XL9tNRxiYIsaOIebCWSFnOSnNZpL++hcc6zfQ8803uGtr99mXp7kZx6ZNRIwZ42/rLSmh7ZVXsP20DHw+Io6dSPxVV6EvKOj3exEEQTjaqFRRFBS8jsWykY6OIgBiY4/DaBwZFJuX+yDr1l+Cz+cIOjdk8J/JzLwW6EvyrFh5KnZ7Wb+Msce62b/0T6mMxOu1hYn0UV31YkDCqbz8Sapr5vqPZdlFS8tndLQvZey4BUQZhgb04PH00Ni4mM6ulYCC+LgTSEw8E6VS3y/3IgiCIAgHgpjh9Cv5ZJlxy0todLpRSxKXJMdyWWo8+QbxwT+QNTY28vLLL+8zxmg0Mm3aNEaMGIFCsXt6u8PhwOv1EhkZKQqFHiZkWaZr4UI657+Fs7QURVQUxhkziL/uWtRJSf6YpgcfouvdfS+/U0ZHYzjpJAxTT0BSKqm/9TbkvZOTSiVpzz5D1LRpB+qWBEEQhBB6rNuoqnye1rZvkGU3xqhRZGRcRWLi6QFxDY2L2Lr1rpB9qNUxjD/mUzxeK72OOrZuuw+Xq7nfxmg0jkanS0WpjKSxcWHYOJNpEmPH/Nd/bLOVsX7DZTidgWOJiBjMmII30emCZ9k7XW10tC/FJ7uJiS4Us6YEQRCEfiOW1B3ghJPHJ3NNSRWftQbuhCUBT+alcWlK/AG7tvDbvfHGG1RWVu43Ljk5menTpyPLMt999x21O2fBxMbGMnnyZMaMGSMST4cR2edDUoSuj2H98Sdq//jHfrmO0mQi+/vvUIg6YIIgCAedLPuQZS8KhTrMeZnKqueprPwX4PW3azWJjBr1EkbjKH9bdc2rlJU9HvZaCQmno9HE0dtbT2fHcry+/pvxnpl5IwZDHjptEltK/ozDURUyzmQ6lrFj5vuPZdlHWfmT1Na+HrATX3z8NIYPewqVKipkPy5XOx5PDzpdMgqF+PwSBEEQwhMJpwOccHqhpoVHykPvfiUB3x6TJ2Y6DWB2u50FCxZQU1MT0D548GAiIyN/1m52AFOnTmXq1KkHYITCwSb7fFSeex7ObdtCntdkZ+Pt7sLbGqo2R7DUfz2H8eTdOyx52troeONNepYswdfbi37kCGIvv5yIwp/1Pi0IgiD0s97eRlpavsDt6cIQmYfZfDIKReCOw16vkw0bLqere3XQ6+Nij2fUqLn+xFZ19VzKyv8e9npKZSQg4fVa+/U+AEaNepVY07EolToqKp6lsuq5kHGxsVMYU/CfgDaLZRNl5X+ns3M5ACpVNKmpsxmcdVPQ30MQBEEQQCScDmjCSZZlxq/YSm1v+Do/c1LjeSw3eNcsYeCQZZmamhoqKytRKBRkZ2eTkpIC9C27+/rrr/c7C0qSJG699VaMRuPBGLJwgLkbG6m97k9BSSfjGWeQ8tjfQKWid0sJ7XPn0rNkyT77UmdkEHPuOUQeNxlFdDQ1l12Gp6kpKC7p4YcwzZzZr/chCIIg9B+vt5fa2tdpaFyE09mETpdGasos0tIuCUjIuN1drFw1I2jZWx8l48a+TXT0ODweC1XV/6am5tV+H6tabcLt7gZ8YWMKC98n2jgaAEvPZtauvTBk7av4+GmMGvlSwExur9dJff18Gpvex+lsQa8fRGrKhSQnnyt22BMEQTiKiITTAUw49Xp9ZC7duM+Y400G3ivIPiDXFw4OWZZZsWIFX3311T7jTj31VCZOnHiQRiUcaLLPh23ZMuzr1qHQajH87kR0ebkBMY7iYqpmXfjzO1Wrwe0OfU6lIvt//0OdmBA8FlkGtxtJI54wC4IgHA5stgq2lNxCT88Wf5tGY2Zo3sOYzdP9bW53F8uKpuD1hl6CZzbPIDXlAnp7G2hr+5a29v/12xhVyigiDTlotclYLBvo7a0PGzum4L/Exk4C+hJvG4qvoKtrVVBcUuJZDBv2VFDSqce6jYb6BdgdVWg1ZpKSzsFkOlaUIxAEQTjMiV3qDiCNQiJSqcDmDf/0KFYt/qyHO0mSMJvN+43r6uo6CKMRDhZJocAwZQqGKVPCxuhGjUI7dGjY5XeSRoPs8YBv53tEuGQTgMeD5ZOPibvqKn+Tu6WFtn//G8snn+Kz2VCnpWGaPZvYyy5FUon3FkEQhIEqMnIwxxR+hKVnI3ZbORpNHCbTpKB6Ump1DCOGP8emzdfj8zkDzhmNYxiW/5i/1lJi4un8+NOxIWchASiVBpKSzsLpbMZmKwtb62kXj7eH7u51P+t+tm2/D7P5FHTaJLotm0ImmwCamj/CbJ5OQsKp/raamnnsKPtbQFxj0/ukJM9k6NC/BSWnvF4nLa1fYO0pQakykJBwGobInJ81TkEQBGHgEjOcfoU7ttfy34b2sOffHJnF9PjoA3Z94eCwWCw8/fTT7Ov/EUmSGD16NFOmTCEuLg6fz0dpaSmbN2+mt7eXpKQkxo0bh8lkOogjFw603tJSaq6Yg7c98H1AYYwi47XX0AwahG35Cro//RTrfpbfKWNjiT77bCInTkCdkUHtnCtxNwTXiIs65RRSn3laPBkWBEE4Qjgc9dQ3vEOPZSNKZQQJCaeRkHBaUO2kurq32F56f4geFIwc8TwJCacA4PXa+WnZJDyenrDX1OnScLs78Xpt/XkrREQMISvzBrS6ZNyudjZtvj5s7NChj5GaMst/bOnZTHHx1UE7AqamXkJe7gMhl+tZraV0da9BIamIjZ0Scqc+QRAE4cAQS+oOcMKp2elmxtpS6p3BMxdOjI1i/qjBKMSPwiPCe++9R0lJyX7jJEli2LBhWCwW/252uyiVSs4//3zy8/MP1DCFQ8DT1kbnOwuw/fQTsiwTOXEiptkXoU5K2h3T3s6OyVPg577PStI+Y9NeepGovQrVy7JMb3Ex7oYGVEnJ6McUiKSUIAjCEaal5Suqqv9NT89mAGKijyEr60ZiY48LiKuq+jflFf8I2Ud8/EmMHjUXWZZxu7tYvuJkPJ7OAz72vel1GYwc9SJaTQKSpGL5imm43aEf5GYPuYtBg672H3s8PWwpuZ22tj2XGSpJS5tNTvZfUCiCZwJ7PFa6utcgy16ijQVoNHH9fUuCIAhHFZFwOsAJJ4Amp5t/VjWxuLkzYHndtWlmHsxJPaDXFg4eu93O/PnzadhrxklkZCRjxoxh48aNWCyW/fajUqm46aabRIHxo1Dt9Tdg/V+Y+huShKRWI7vCb0Kwp6jp00l77ln/sWPzFhrvuRvnjjJ/m2bIEFIefwz9qFGhuhAEQRAOYx6PFUlSoFRGhDwvyzKVlc9SXfMKPl/vzlaJxITTGTr0MVSqSH9sZdULVFT8M2Q/CkUk48d/jOxz0utsZOvWe3C5Wvr7dpAkFbLsCXterY5j4oQlaDR9Kwc2bLiC9o6lIWMHZVxNdvZd/mNZlqmqeoHqmrn+GV2SpCY15UJycu4NuQufLPuwWrfh9dqJjMxGrY75LbcnCIJwRBIJp4OQcNrFJ8t0ub1MWrmVLo8Xo0rBumOHY1ApD8r1hQPP6/Wybds2SktL8Xq9ZGRkMGrUKHQ6HR6Ph40bN/Ljjz/S2bnvp4S/+93vOOGEEw7SqIWBwt3YSPXFlwQvk5Mkkh99FOOZZ+DYsAHbsmW0vzx3v/3pRowgorAQTVYWzU8+iWwLXhahMBjIen8xmoyM/roNQRAE4TDidnfT0VmE7HMTHT0WvT5492Sfz8PWrXfS1PxRQLtKGcXIUS8SazrW31ZT+zo7djwa9nqDs25Dp0vC6Wymrv5tnM7G/rsZ+mpVqdXR+yxyLkk6pkxejlrd93CvsvJ5KiqfDhmbnHQew4Y9GdDW1vYtpTv+5q+DpZA0JCWfS27OfWETfL3OJtyudnS6VJGcEgThqCESTgcx4bTLY+UNPFfT9+Tn0ZxUrkrbf8Fp4cjR2dnJs88+u8+Y/Px8Zs2atc8Y4cjk6eykc/5b9Hz9NT6HA/2oUcRedin6ggJ/jCzLlJ80LWT9pl8j5qILSX7ggYA2+5o1dL79Ds6yMpQxMRjPOJ3os89GIXbCEwRBOCrJsozFsoHm5k/xeHowROWTnHROUPLE53OxcdO1tLf/ENRHaurF5OU+5F/O3dj4ASVb/xz2mibTRIxRo3A6W+joXB5Uu+m3UCj06HTJqNWxWCzrkWVv2NhjJ35LRMQgANrbf2BD8VVA8KZAsbFTKBj9esBydau1lNIdD9PZuRzom6mVmHAGubl/Qa0OXbfT7e7G6WxCo4kXy/oEQTisiYTTIUg4NTndHLO8BLcsk6HTsHxiPkpRR+Wo0dvby9///vd9FhhXKBSMGzeO8ePHYzabcTqdFBUVsWHDBnp6eoiNjfWfVyrFDLmjUfurr9LyVOjaGwC6kSNwlu5AdjrDxuyiiIkh4+WX0OXnI2k0YfvWF44j45VXUOj1v2nsgiAIwpHN5/PQ1PQBjY2Lcbpa0evTSU29CHP89IBkjM/nZkPxHDo7i4L60OsHUThuERpNLADt7T+yofgPYa+p0ZiJiS7E6WrGZivH4+nut/tRq2OIiBiMRmOmq2s1bndH2NgxBW/662XZ7VWsXnNuyLEYDPkUjluIUrn7M9XlaqN0x99oafl85/JBBfHxJ5Kb8xf0+vSQ1/N67TgctahURlEQXRCEAUcknA5BwgnghpJqFjX3Lat6bUQmp5vF1NqjyTvvvMP27dt/VmxmZibd3d0hl+Hl5eUxa9YsFIrgXVmEI5vsdlN/2230LPkm8IQkkfTAA5gunIXP5aJ38xZq/3Qdvq79f/GWtFo0gwfj3Lo1bEzcNdeQcOstgWORZRzr1uEo3oik0xJ14okBBdEFQRAEIRyv10l19YvUNyzA5WpFqYwkKelsBmfdhEYT74+TZR8biufQ0fFjUB+SpGJMwRuYTBOBvllFK1edto+rKjEaR+ByteF0Nu+zNtQvpVabiIoagVZjxtKzGZutNGxsXt4jpKXOBvqKnK9ecx52e3lQnEaTyPhjPkSrTfC3eb0Oysr/j8bGhXi9dgCiowvJybmXaOPokNfzenux28uRFBoiI4aE3NVPEAShP4mE0yFKOG3qsXPymr4PoPHRkXw8Nuegj0E4dNra2njttddwOBxB5xITE7FardhC1NsJ5bzzzmPkyJH9PUThMCD7fFi//57uTz7B29mFdnAWMbMuRJeXGxDX+Nf76Vq4sF+uqTCZyF32E9LOJKe7uYW6m26kt3jjHkEKYi+7jIQ77/DHCYIgCMK+yLKM12tHqdQhSaFnb3u9DnaUPUFj4yJ/oXODYSg52fcG7cK3adMNtLR+EbKfwVm3kpV1AwBOZzs/LZsEhE86abUpeDwWvF7rr7iz8JRKA6aY8Wg08TgctXR2LQ8bm5F+JTk59wI7k28b/kBH57KgOIVCx7ixCzAad3839Pk8VFU9T23dG3g8fRvY6PWDGDLkzyQmzAh5Pa/XSU/PJmRkjFHDw9amEgRB2BeRcDpECSeA89aXsayr74Pr87E5jI2O3M8rhCNJe3s73333HSUlJfh8PmJiYpgwYQITJkzA5/OxZcsWVq1aRX19+KKXAEOGDOHSSy89SKMWDkfO8nIqzzsfubc3+KRaTcJtt+Jpasa+YT29GzfBft7rtcOGEVEwGt3wEbS99hruioqQceZbbiH+2muC2n1OZ98sKklCm58v6kIJgiAIv4jH04PdXolSaSAiIitgmd4uXm8v27ffT2PTh0BffSaFQs+gjD+SlXVTwGu2lz5IXd1/Q14rLvZ4CgpeB8Dl6mLFypP3uaROpYrC47ERqsbTbyFJamJjj0OjicfrsYVNpu09ZoCt2+6joWFByNjhw58hKfFM/7Esy9TUvkpV1Ut4PF1A3z2lp88hK/OGkLOivF4HnZ0r8Pp6MUaNDFl4XhCEo5NIOB3ChNPXbd1ctqkSgLMSYnh5eOYhGYdwaHm9XjweDxqNJuQXpkceeQSvN3why8jISObMmUNc3O6ikpWVlaxZs4aOjg4MBgOjR49m2LBhYundUcy2fDn1d96Jt7XN36aMjyfl8ccxTJnsb6u/+24sH34UqotfTBkTQ/YP36PQaoG+L7Edr71G+6uv4e3q+xKrNJmI++Mfib3iDyH//QuCIAjCb9Hb20i3ZT0KSUVMzET/znR78vlcbNv+VxobFwW0x8WdwPBhzwS8prr6ZcrKn9y7CwC02iSOnfgNCoUGt7uTdesvx2bbFnZsCoUWn2//tRZ/jZiYiWi1CUiSkqamD8LG6bQpTJr0vX9WWWXVC1RU/DNkbEbGH8nJvjugra5uPuUV/9yjTpWE2XwK+UMfD/m39nh6aGv7Fpe7g8iIbGJjjxNL+wThCCYSTocw4eSTZaas3Ea5w4lSghUTh5GuE0/6hUAvvvgizc3735UlIyODgoIC2traKCoKLr6Zn5/P+eefL4qMH8Vklwvrjz/ibmhElZSI4YQTgmYX2desofqS8DPmVAkJeHt6kEMsBw0l+pxzMEydim74cLoWL6b9xRdDxplvvon4664LHrMs42luRu7tRZ2aiqRW/6zrCoIgCMIvZbdX096xFFn2YIqZQFTUsKAYWfZRuuMR6ureDGjX6dIZPWouBsPuZe3NLZ+zefONIa8lSRomTvgCnS4Vl7udbdvuo739+7BjkyRVv9aa2lNkZA56XToqlZHm5k+RwywvlCQVx036Ca22b4fthoZFbN12V8jYmJgJjB3zVsDDpIaG9yjd8She7+6yEXp9JiNHvkCUYWhQHy5XO03NH9PbW49Om0xi0llo96jrJQjCwCcSTocw4QTwn/o27i6tA+DadDMPZqcesrEIA9Pq1av57LPP+qWvU089lYkTJ/ZLX8KRq+nhR+h8++2gdk1WFoPefgtlVBTO8gpa/vEPbEuX9ss1Jb2OnB9+QGnc/TTU+uNPtD79NL0lJQAozfHEXnYZcVdeKWpDCYIgCIeU3V5JS8uXeLw2jFEjiI8/CYUi+KFIZeXzVFQ+A+z+HaVURDB8xDOY40/yt1ksm1i95lzCLcUbPfo1Yk3H4XZ3UFX9MnV1b+xjdAokSXFAElRqdSx6fQYadSydXSv8BctD2XPHvra27yjeeFWYPuM4duJXqNUmf1tj4/ts3XYfsuzyt0mSmqF5j5CSckFQH3Z7NQ0N72Kz7UCtjiEx8ffExk4Ws6cF4RATCadDnHCyeb0UFpXQ6fESpVSwbtJwolRiBoqwm8/nY/HixWzZsiXo3MSJE0lMTKS4uJiqqqr99mU2m7n++utDXsNms6HRaNDuXP4kHL1kWcbyySd0vPUWrrJylDExGM84g7g5V6CMjvbH2deupfriS/rtujEXziLm/AvQ5mRjW76cuj9dD77gL96m2bNJuv+vIftw19fj7e5GnZGB0mDot7EJgiAIwq/lcNTQ1PQRLnc7EfoskpLOQq0O3qG6L8lyL7Ls3qNVQXb2XQzK2J2scbnaKVp+Ytgi5jnZ95GefgUej4Xm5s/ZXvqXfYxOQqUy7rEkrv9IkhKtJhG1JhaHo8ZfsDyUrMwbdtbWUtJtKWbNmvMJnXyTGDd2ATExu3+/NjV9RMnWO4MSbAkJMxg+7GkUClVAu8Wykdra/9BtWY9Socdsnk5a2qVoNHGE4/X2AhJKpfieLAi/hEg4HeKEE8DjFY08W923ZOrh7BSuTk/YzyuEo43P52Pbtm1s2LABq9WKyWRi3LhxDB482B/T2dnJvHnz6OnpCduPJElceeWVpKamIkkSXq+XoqIiVq5cidXa96UlJyeHk046iSSxrb2wH7IsU/vHq7H99FPI84n3348uL5feLVuwfPEFjvUbfl7HSiWSQoHsdocNGfzF52izsvzHjuJimh9/AseGvmtIOh3Rv/89CXfeidIgNmQQBEEQDg9OZzONjR/g6K1Fq00iOels9Pr0oLjOzlVs3HRtUKIoNfVi8nIf9NdFkmUvK1bOwG4vC3m91NRLGJr3ED6fm56eEtasPXef49NpU/B4rftMHv16Emp1DD6fK2DZ3d5MMRMZOvRR1GoTLlcHK1edFnY2V/aQOxk0aPcGJk1NH7Gl5M/snczSaVMYO3YBen3gapP29h+orHqB7u61AMREH0Nm1g3ExU4mFJ/PQ0/PZnw+JwZDfsg6VoJwNBEJpwGQcGpyujlmeQluWSZdp2H5hHxUCjH9U/jl3n77bUpLS/cbFx0dTX5+Pq2trZSXlwed12g0XHHFFSQnJx+IYQpHEJ/NRtPDD9P96Wews7i90mTCfPNNmC680B/nLCuj4owzw3Xzi0WdfDJxV1+NNnsIzooKqi++JOQufPrCcQz6z3+QVIFPN2WfD8e6dbibmlGnpqAvKBDT7gVBEITDisfTQ1PTx1ht21CpjCQmnEFUVH5QnMNRw4biOdjtlQHt8fHTGDH8uYBZO5s330xzy6chrxcXdwIFo+ftvLadouUn7HPHvsjIXEDG5WrfZ9xvI7HncsW9qVTR5A99ArXGBEisX39pwDK9PcXHncjo0a/4jxubPqSk5PaQ1xwx/BkSE88IaG1sfJ/yin/gdDYBoFDoSEmZRU72XSgUwTOjXK42Wlq+wuPpJjIyl7i4qUGzsQThcCcSTgMg4QRw49ZqFjZ1AvDK8EzOTAieZisI+7NlyxYWLlzYL30NHjyYyy67rF/6Eo587uYWerdsQaHToh83zr8z3Z5qrrkG2w+haz7pJ07AdO659G7dhn3VKnpDLCENSZKQdLp9FjFPfe5ZjNOn+4/t69bReM+9uKqr/W3anGySn3gC/fDhP++6giAIgnAY8fnctLX9j67utSgUWszx0zAaRwc9bPF4bGzecjPt7d8FtMfETGDUyBdRq3cvra+pmceOsr+FvJ7BkM/4Yz7y7363bv3ldHaGnhENYDSORq024XZ30tNTsteywoMnI/1KdLpUlMpItpc+hM8XukaVWh3L5ON+8ieSwienICHhdEaOeC6grbp6LuUV/wy4T50unVEjXwyZNOzqWkND4yKcziZ0ulRSU2ZhNI76tbcpCAeNSDgNkITT5h4709b0zUwpNEbw6bjc/bxCEIL5fD4WLlzI1q1bg84lJiYyefJkduzYwfbt23E6978N75133klERIT/uL29nZUrV1JdXY1SqSQ3N5fCwkIMolaO8DN4u7upu+FG7KtXB7RHTJxI2nPP+guGe7u7KZ08BfaxpO6X0ObmEn/D9Wizs5E9HqpmXRgyQaUwGhn84QeoU1IC2j3t7XR/+CHOykpUpliif38m2pycfhmbIAiCIAxEFstG2tuXIiMTazqW6OhxQckpWZapqHya6uqXA5a0RUePZeSI59FqE/1tXV1rWLf+4pBL3/S6DMaP/wSVqu/7ZHnFP6mqeiHs2KKjj8EYNRy3u4uOziJcrpbferu/ijFqFBGRQ1CromloXBS2phbA+GM+8e962Les77aQcX0F1L/xL8WTZZkdZY9RWzsvKDYr62YGZ90U0CbLXurq36a+/i3s9krUahNJSWeTOejakHXDAFyuDqzWbShVkRijRviThILQH0TCaYAknADOX1/GT119b1Sfjs2hMFrUHRF+Oa/Xy5o1a1izZg0dHR0YDAZGjx7NpEmT0Ol0AHg8HubPn7/fQuPp6ekMGzaM3Nxcurq6eOedd/B4Ar8oREVFcfnllxMfL7apFfZPlmUca9ZgW74ckIg8bhL6sWODvsQ2PvAgXe++G7IPldmM+ZabcZZX0FtSgn3Fip8/AEmCfXyWxV5xBYl33ek/tixZQsMddwYt14u76krMt98uluEJgiAIRz2nq432tu/weu0YjaNDzpwCaGv7lm3b78fpbPS3xcSMZ1j+/6HXp/nbXK52Vq0+KyBuF602kWMKP0KrNe/sM/zud7v6T0o8C7e7k/aOH+nqWvlbbvVXUyh0aLWJqFUx2Ow79rm7X0b61WRkXIFKFU1b+//YvPnGsLEFBW/460nJssyWkltpbv4kKC4iYgiF494N2AnQ67VTWvoIjU3v+xOBOl0q2dn3kJhwWlAfsuyjvWMpnZ0rkFAQF3cCMTHjxXchYZ9EwmkAJZyWtHVz6aa+tdVnmmN4ZUTmoR2QcEQrKiri66+//tnxkiQR7j0gIyODOXPm9NfQBAGfw0HdjTcFFSRXJSaS/spcdLl9s0BlWab81FNxV9f0y3UVERHEzJyJZnAWiohIGu6+GzyhC5EmP/44MeecHdBm+fprOt54k95Nm1BERBA1fTrx11yNOjU1ZB+CIAiCcDTx+Tx0da/G7e4kMmIIBkNeyDiHo47tpQ/S3v49fTWaJOLijicv90H0+gx/nCz72LT5elpbg7/TajRmCsct9hcCd7raWLZsctjlejHRx5Cdcw9udyfd3eupqnr+t95uP1AQere+PpGRuQwadA1qVTQ22w7Kyv8eNjY9fQ65OfcBfd+fNhRfQUfHjyFjR418EbN5dzkCp7OV4o1X0dOzOSDOZJrEqJH/RqWKCmh3Opuprf0PrW3fIstuYqLHkZ4+J+RywV3jcThq8Hh7iNAPCupPOHyJhNMASjj5ZJnjV22jzO5EAn4XG0VOhI4Lk2PJN+gP9fCEI4zNZuPZZ5/F5QpdONFgMPh3rvs5rr/+esxms/+4vb2doqIiduzYgc/nIzMzk0mTJpGy13IlQQhHlmXsK1fSs+QbZJcTfcEYjDMQBs1RAAAgAElEQVROQ6EPfD/sfOcdmh56OGQfUkQEKU88jqe1FVd5OZ0LF/XbUj1NXh5DPvrQf9z20ku0PvNsUJwyNpZBb80P2FVvF2dZGbZVq5AUCiInT0aTlhYUIwiCIAhHK6ezmV5nE1ptIjpt6B2UfT431TVzqa97C6erGYVCQ4L5NAYPvi1g5hRAQ8NCtm67h70LjWs0CYwbu4CIiEFA33eQVat/j9VaEvKaxqhRjB37Hl5vD729jaxdNxOfL3jzkl0iIgYjyz7c7i48nq5f8BfoP5KkJjFhBiq1EbfbQnPzR2FjIyKymTjhS/8D57XrLqS7O/Rv5oSEGYwc8S//sc1Wztp1F+F2t+91fRUjRzyP2XxyQHtn12p27HjUn8xSKPSkJJ9PdvZdKJXBv4Gdzhaamz/B6WpBrx9EYsIZYjfAAUwknAZQwgngik0VfNEWvM3o3VlJ3JIptqkX+ld5eTnvvvtuUNIpLy+PCy64AIfDQVlZGUVFRbS2tu6zr8zMTEaPHk1WVhY2m40333wzqE6UQqFg5syZDB06tN/vRTh6ybJMyxN/p+ONNwLaFUYjac//i8jx4/1t9XfcieWT4Knmv5YqORlN5iBUcfFYPg29qw9A5PFTyJg713/stdpovOduepZ8sztIkog+9xySH3gASaPptzEKgiAIwtFAln14PD0olXoUivCfo11da6ipnYelewMKpQ6zeToZ6XPQahMC4uz2StatvzRoaZ9Ol8qYgv/6k1MAZWV/p7pmLqFERuYyYfyn/tpIa9ZeRHf3qrDji4ubilaTgNvTTXv7D/tMZB1ISmUkarUJSVLjcFTuM3b4sOeIjByMShXF5i03YbEUh4xTqaI4btIyVKq+0jHd3etYu252yFlnsabJFBS8jiQp/G21df9lx45HA2qBKZWRDB/2T8zmaQGvl2Ufzc2fUFf/Ng5HNRqNmeTkc0lNmR2wM+OenM4WrLZSVMpIjMZRop5VPxAJpwGUcFra0cPM4uAt6nd5a9RgTooT2Vuhf1mtVtavX09TUxNarZbhw4czePDggPXY69ev56OPwj8F2ZtSqcTr9YY8p9frufXWW9Hs9YPa4/FQXV2Ny+UiKSkJk8kU8vWCEI6zvBzLZ5/j7e5Gm5uL8fTTURoCa+H1bttG1QUzkUPMcpL0ejIXvAOyjKuigtbn/oVrP3XOfonkJx5HP3w46vR0Gu64IzDZtAfT7ItIuv/+wHsrK6Pt5blYv/sO2e0mYtxYYq+8EsNxx/Xb+ARBEARBCOTx9NDYuJiOziJAIjb2OJKTzvUXON/F53Ozbdt9NDYtDmg3GIYyetQr6HS7Z/h3dq5i/YZLQxZQNxiGckzh+/7d7yorn6ei8umw40tNvZiY6EI8HgsNDQvpsW4OGztQmEyTiDYWoFJFUd/w3j6TWQWj5xEXdwIA7e0/sqH4DyHjJEnNhPGfEhmZDfQ9jNy67R4aG4N3746OLmRMwX8CZk95PD1s2/5Xmps/B/p+w+h0aeTk3EuC+ZSgPjweK01NH9HVtQpJoSI+7kTM5ukoFOqw9+L1OpBlb9C/nSOdSDgNoITT5Zsq+CrE7KZdToyN4u3RQw7iiAShT29vL//85z/DLr9TKBT4fOHXl+/t3HPPZdSo3Vu5btiwgSVLlmCz2fxt+fn5nHnmmQG75AlCf7AuXUrDvffhbWvzt6mSkkj9vyeJOOYYf1vXBx/SeM89YftRJSai0Olw1deHrfP0qyiV5PzwPaqdhfgdxcVUXzEH2R5cYDTpkYcxXXBBULtj02YsX3yBr6cHbf5Qos88E2WUqIcgCIIgCAeS1baDttb/4fP1Eh09ltjYyQEzdHZpa/+e0tKHcDh21aCUiI8/ifyhf0Oj2b0Rj8fTw9p1F2K1bgvqw2gcw9gx81Eq+zYFslg2snrNOWHHNnjwn0lLnY3H001T0ydUVP4zbKwkqYmKGoHH04PL1YzH0/Mz/wL9S6HQotUmolJF4XDU4fF0h42NizuRrMw/oVQZ6LFspGTrnWFjhwy+nczMPwF9O/utXXcR3d1rQ0RKjB79KvFxU/0tVtsONmy4HKezOSAyKmoEBaNfR6OJDWjv7FxBReVz/oL1UYbhDMq8LmRhdoDe3gZaW7/G67UTFTV857+hw3emlUg4DaCE0zHLS6jtDf2DHiBZq2b9pOEHcUSCsNumTZt4//33gwqHazQaLr30UvR6PRUVFWzcuJG6urp99qXVasnOziYjIwO3280334Se6ZGWlsacOXNQKII/qAF8Pl/Yc4KwL7LLhXXpUtzNzahTUjBMnoykVgfFVF96GY7i4Gnhiuhosha+hyYjA9ntpuXZ5+h49dV+G58qJQXd0KGo09KwfPkl3pbQWz5LWi05P3yPMqZvq2PZ66Xxr/fT/f77geM1Gkn/9wtEFAZ/3suyTO+mTXhaW1Gnp/sLsguCIAiCcODIsg+LpRi3p5vIiBx/cfO9ud3dVFY9T2PjYjyebtTqWFJSZpI56E/+pWm7VFe/TFn5k0F9xMedyMiRL/iXGvp8TlasOBVHb+hNV4YMvoPMzGuBvoLhy4omh5yRBX01lzIHXYfP58DhqKO5pf9KFxwoSmUESYlnoVQZcDlbaNpHPSuDYTjjj/loZz0rL8tXTMfhqAoZazZPZ9TIF/3Hra3fsHHTn9g1a2pPuTn3k55+uf9YlmXKK56iunouexaKj4jIZvSol4iICKwF6vU6qKt7k8amD3G52oiIyCI1dTZJiWeF3DlQlmW6LetwOGrRahMxxYw/KIkskXAaQAmnaau3s9nqCHs+L1LHD+NF7Rvh0KmpqaGoqIiqqiqUSiW5ubkcd9xxxMfvfhLT2NjIyy+/3G/XnD17Nrl7/AC22+38+OOPFBcXY7fbiY2NpbCwkAkTJqBUHr7Zf2Fg8lqttDz1FN0ffYTs6AVJInLKZBLvugvtkN0zTj3t7ZSdeBLyXnXLdomYOJGoadNw19ZgW74CZ2lpv40xasYMon9/JuqUFCxfL6H9+dC76iiiohjy9Veo9liual+3nsa//hVX+e7l3PqCApIffyxkkXMA2ePB292NIioKhag1JQiCIAgHhSz78Pl6USh0IWdN7dLdvY66+rex2yvQqONISjqLhITTgpILdnslxRuvxm6v2KNVIi3tMnJz/hJwjbLyp6iufpFQ9k6crF5zPhbL+pCxCkUEhePeA/oKqG/afNM+i6hrtSkoldqdM63awsYdeEpUKgOSpAoqhr63IUPuRK9LQ6mMoGTrXWHjFQotk48rQq3ue2hYU/s6O3Y8GjJWr8tgwoQv/bWnPB4b6zdchsWyISg2JXkmQ4c+FpB06ukpYUvJbdhsO/xtOl06w/KfwGSaGNRHt6WYuro3sVq3o1IZSUw8g+Sk88LWvnK7O7H0bEEhaYiOHu1fEgoi4TSgEk7/qm7mbxWNYc/fnpnIHVnJB3FEgvDLybLM3LlzaWwM/W9ZqVQSERFBT8/Pm5qbkZHBySefTFJSEm63m3nz5tHWFvyBM3ToUGbOnClmPAkHhM9ux93UhDImBlVsbMgYy5dfUn/7n2Gv+mXavDwy/vO6P9Hjqq2lfPopEO4zVZJQmkx4Ozr69R4A4m+4gfg/XYekUOAsK6PygpnIjuAHHSqzmayPPgy4V5/dTusLL9C1aBG+bguSVotxxgzMt9yCOjEhqA9BEARBEAY2WfbS3r4Ui2UjSlUE5vjpAcXQd8fJ1NTMpbrmVdzuvu8nWm0yWVk3kpoyKyDWat3OuvWzcbsDE0mSpGL48KdJTJjhb6uueZWyssdDjk2limHSsd/5d6Dbuu0+GhoWhL2X+PhpGKNG4PFaaWn5kt7efa+4GAi02hT0+jSUigg6u1bss0D8oEHXk5hwKkplJPX1b1FT+1rY2NGjX/MvA+ztbWTlqjNCJvYUCh3HFL6PwZDnb6urm8/20geCYo3GMYwp+E9ADSqfz0npjsdobHgPn9y3UkutjmXw4FtJS50NiITTgEo4WT1ezli3g2220P/Qzk808a/8jJBT5ARhIGlubuaNN97AvlfNGZVKxUUXXcSQIUPo6upi69atfPXVVz+rT4VCgU6nC+pzTzNnzmTYsGH+Y6/Xy7p161i7di1dXV1ERUVRUFDA+PHjUavDF/UThF/LWVZG59tv49i0GUVEBFGnTCfmnHNQ6AO39W24976gZW+7xF17DQm33ILPbse+di21f7y6X8coqdWokpPx2e0Bdaz2Fn/TjZj/tLO+gctF9RVzcKwNrm+gTkkh8713/TWndun59js6/vsmzq3bUERGYjztVGKvuAJVXFy/3o8gCIIgCAeHz+fEatuBhJLIyBwUClXIuN7eBmpq5/XVs5JdxMQcQ0b6HIzGUQFxsuxje+lD1NfPD2hXq+MYPWou0dEF/jaHo4ZVq88JmTjR6zMYf8zHqFR99Srb2r+nuPjKsPcxOOtmkpMvwOO10tT0IdXVL4WNVSj0xMQU4vVYcfTW43KFLnMwkGi1iZhMk1ApDVh6NoWcCbVLfPx08oc+glJpoLe3gRUrT2HPJX17Sk+fQ27Off7jTZtvoqXls5CxQ/MeJTX1IpFwGkgJJ4AOt4cnKhpZ1NSJ3edDK0nIgGvn3/6BISlclyGeJAsDn8ViYeXKlezYsQOfz0dmZiYTJkzAbDb7Y3w+H88++yzd3eELAP4S6enpXHzxxeh0Onw+HwsXLmTr1q1BcYMGDeKSSy4JmXTy+Xw4HA40Go1ISgkHjOxy0fzEE3S+t9BfcFzS6Yj9w+WYb7oJaY+ZejVzrsRWVBSyH4XRSNLDD+FtacVZWUHXO+Gf/P1SklZLxLhxqJKS8HZ2Yv3uu7CxsZdfRuIeBdbbXnqJ1meeDYpTp6Qw6O23UCclBbQ7Kyvp/O9/sa9Zi6RSYZh6AqbZs4OSWIIgCIIgHHms1u00t3yOx9NDlGEYiYkzUCqDNw6yWrezvfRBurpW7WzpK7Y+NO9htNpEf5wsy2wvfTAokQV9u+SNHvWqf3mY1+tg+YqTcTpDr87IzX2Q9LRLAXC62li27Liw9azU6jiG5j2E1+vA7qimqip0mYNdVKoYZNmF1xv+gfrBJQHhcz6SpCY1dTYqVRRej43autfDxmo0Zo6b9CNKpUYknAZSwmkXp89Hl9tLtErJ6m4bF24sxyv3/RN4Y2QW0+OjD/UQBaFfrFq1is8//zzkuZiYGE466SQaGxtpaGig6mduUW8ymdDr9TQ0NISNOfnkkzlujy3lvV4vRUVFrFq1ip6eHhQKBUOHDuXEE08MqFElCP3J09aGfd06JKWSiMJClNHB7+3uhgaqL70Md319QLuk15P+4otETpzgb6uZMwdb0fKw14ucNAnZ48Hd2Ii7trbf7kPSaPoSRAkJIEHL34MLlu5iPP10Uv/xlP/YumwZddffgNwbOLtXZTaT8cYbaAcH1pLydHbS9e67WH9Yiuz1EnFMIbGzZ6NODV1sVRAEQRCEI4vDUYvT1YJelxaQaNqTLMu0tn5NfcM72O1VaDXxJCWfS0ry+f7i6bvYbOUUb7x6r2LgCjIHXcPgwbcHrDCqrHqBiopQO/wpGDXyBczm6f6W9Rv+QEfHjyHHp9dncuzEr5EkJT6fh1WrTsdmLwt7z8lJ56PVmvF4bTQ2LhpASarwCsctIiZmrEg4DcSE097erG/jztK+daiRSgWfjM1hmEG/n1cJwsAnyzI//PADS5cuxefbPX0zKSmJWbNmYdqjwPFzzz1HRz/VtYmOjuaqq67CYOhbh7xo0SK2bNkSFKfX67nyyitDJp2ampqora1FpVKRnZ1NlNh2XjhAvBYLXYsWY/32W2S3G33hOEwXzUaTFphkcWzaRPWllwUlbwAMJ5xA2ksv+r801Vx7Hbbvvw97TUVkJD6n0z8Dq99IEuZbb0GTloYiOpr6W2/DZ7GEDNWPHUvm22/5j11VVVRf/gc8zYFbESsiI0l/ZS4RY8cGtPtsNroWL8by5Vf4bDZ0w4djung2+uFix1dBEARBEHbrq2f1Az09W1CqDCSYT0GnSwkRJ9PYuJjqmpew2ysBiDaOIWvwLcTFTg6IdTqbWbf+Mux7JZI0GjNjCt4IqJ3U2rqEjZuuDTm2mJjxjB3zlr+Qe1XVvymv+EfYexk9+j9EG0fg8dgor3ia5uYPw8bq9Zk7lwza6LYU43SGf2D/S40d8w6xsRNEwulwSDgB/HVHHa/U9dXbSNWq+bIwF7NGLPkRjgxWq5Xt27fjdDpJSUlh0KBBQfXKVqxYwZdffhm2j4KCAmw2G83NzVjC/IDdm06nIyoqitbW1rAx+fn5zJq1uyCi3W7n/fffp6xs94eHQqFg4sSJTJs2TRQuFw4pR3Exzf/3fzjW9NVbUhgMxMycifmWmwN2lbOvXUv1pZeBL3idvqTXM/iTj1EnJ+Npb6fxgQewfff9wbqFAPE33YR++DCUcfE03X8/vSUlIeNUiYlkf7MEaedSWE9nJzWXXY5zx47AQIWC5L/9jZhzzg5o9vX20rlgAd0ff4y3rR1NZiamC2cRddpp+6ydKHs8oFSK+oqCIAiCcBSRZRmXux2FpPLvNBeK19tLc8untLd9jyx7iDFNIDnpPH8x9D01N3/GjrLH91jepyQp8Qzy8h7y16fq69NJ8cYr6ewMntWekX4lOTn3+o8djjpWrpqB12sLipUkNYWFizBGjQDAYtnI6jXnhL2XBPMMhgy5DY/XRlfnSnaUPRY2VqmM2LkLn3HgJZwkSToVeBZQAq/KsvzEXuefBn638zACSJBlOWbnucuBv+w896gsy2/s61qHU8LJK8tcurGCbzv6dvcqNEawqCAbnVL8uBWODl6vl8WLF1MS4gfntGnTmDx591OFN998k4qKiqC4X+uss84iISGBuLg43nnnHaqrq0PGTZ06lalTpwa0tbS0UFRURHl5ObIsk5WVxaRJk0hOFrtOCgeOu7kFn82KOiUFhU4XMqb7s89oeuBBfFarv01lNpPyj6eIHD/e3+bYvIWqCy4Iu7Ne4v33Ezn+GNzNzXS+9TbWb7/t35v5mUyzZxM5ZTKquDjaX5tHT7hNCVQqsv/3DerEvmn4PrudmiuvwrE+eBvnmIsuJOn++wMSSr7eXtpfe42u9xbiaW5GGRtLzLnnEHfNNSjDzHT0Wm24a2tQGo1i+Z8gCIIgCCH5fB4sPcV4PTYMhqFotaHrN/t8LhobF9PY9AEuVzsREZmkpV5MXNzvgh6CdXevY/OWWwN27tNo4skf+gTx8b8LiN1R9jg1Na8GXU+nS6dw3Hv+8ciyj9VrzqWnZ1PI8Q3KuJrs7LsGXtFwSZKUQClwMlAHrAYukmU55CNNSZJuBMbIsjxHkqRYYA1QSF+1q7XAOFmWO8Nd73BKOAH07NzJbvvOnezOSYjh6jQzpXYnJrWS401RIgElHNF8Ph+lpaUUFxdjs9mIjY2lsLCQtLS0gLht27axYEH4Asp5eXloNBpaW1tpbm6mv97ftFott99+O5qdM0mqq6uZP38+brc7IE6pVHLhhReSk5MT0C7LMvX19WzduhW3201qairDhg0TBcyFA8Zns9HzzTd42tpQp6UT9bupSBpNUFznggU0PfxI0Iwo0yWXkHjfvf4vN666OspPORW83pDXM0ybhmnWTDwtrVh/+IGer7/u/5v6GSKPP57oM89AaYqlZ8kSut59N2xs+iuvYJjSl9CWXS5q/ng19pUrg+K0Q4cyaP5/URr22DLYbqflqafo+uADZEffZ7d+9GgS77kbfUFBUB/Qt9uhff16FBoNkZMni539BEEQBEH4TWTZS0fHTzgcdWi1icTFHR9Uy6ovTqa5+RPq6t7EatuOSmUkMfEMBmVcjUYT+H3E6Wxh46ZrsViKA9pTkmeSl/cwCoV6QCacjgUelGX5lJ3H9wDIsvx4mPgi4AFZlpdIknQRMFWW5Wt2nnsZ+F6W5XfCXe9wSzgBVDucnLa2lA538Jf5WLWSx3LSODvRFOKVgnD0kGWZzz77jFD/f+fn53P++eejVCoB+Pbbb1m6dGm/XdtsNpOUlERMTAxr167Fbg9d1C8yMpJbb70VlapvS1mPx8MHH3wQVEvKaDRy8cUXk5gYXBTR4/FQUVGBw+EgPj6elJQUsbRHOGCcFZV0LV6Eq7oaVXw80WedRcSYMUFxnQvepemhh4JmRGlzc8l44z+odtZm81ptlJ1wAj5b8DRvAHVGBuZbb8Hb3kHvls10fxC+BsGBpM7KJObcc1GZTDhKSuh6O+zXCuJvvAHz9dcDIHu91Fx5FfYVK4LiJK2WQfPnox85wt/m7e6m4a67se5RW0tSq4m94grMt9wcsHshgM/louerr7H99BOy7CNy4rEYZ5wWdkabIAiCIAhCf5Jlma6uVXR3r0Wh0BIffyIREbs3fBmICafzgVNlWb5q5/GlwARZlm8IETsIWAGkybLslSTpz4BOluVHd57/K+CQZfmpvV53NXA1QEZGxrhwS2MGsh/aLczaGHq5kAQsGD2EE2JFAWPh6CbLMtu3b2ft2rV0dXURFRVFQUEBI0aMCKiz1N3dzXPPPYc3zIyMMWPGkJ6eTnt7O9u3b6etra3fxjhhwgTy8/OJiYlh1apVFBUVhYwzGo3ceOONATOdSkpK+Oyzz7Dt8WM9JSWF8847j7gwMyJaW1vp7u4mOjoas9ncb/chCHtzbNhAx/y36N26FYUhEuNpp2G64AIUkZEBcZbPP6f+jjuDZkQpDAYyXp+HfuRIoO//54oZp+OqrAx5PUmvJ/nBB/DabLjrG+iYNy/sEsADSdLrMZ56KsqYGDzt7Vg+/jhsbOTkyWS8+grQd381V8wJmZwCMN9yM/HX7i4m6m5pofbKq4JqVKkzMsiYNy+ooLy7sZH2V16lZ8kSfL296EeOIPYPf8Bw/PEhryfLMr0lJXg7u9BmZYplgIIgCIIg/GIDMeF0AXDKXgmn8bIs3xgi9i76kk037jy+A9DulXCyy7IctoT74TjDCWBeXSv37qgPe35idCQfjs0Je14QhEBbt25l0aJFQUmnoUOHcsEFF/hnQ9XV1fHqq8HrmneRJAm1Wo3L5er3Mc6YMYPCwkIUCgWVlZW8+eabIZcCRkdHc91116HbY5ZDa2srH3/8MbW1tf62tLQ0fv/735OQEHpteGdnJ21tbej1elJSUkQxdOGAsa9fT8e817GvWYOkUmGYOpW4K+egycwMjFuzhpqr/hi8C59CQcqTTxJ9xun+probb6RnyTdhr5n82N+Q1Gq8HR20/uv5gDpWB5MmOxuVyYSMjGN1+O8jksHAkM8/QxUfj6RQUH3FFdiXh05O6UaMIHPhe/7Zjs7KSqovuRRve3tQbOI9dxN7+eUBbbYVK2l65BFc5eU7Ly5hOOEEkh5+CHWI9wtXVRVdH3yIu7EBdVIy0eecjTYrKyhuF1mWkV0uJLU6aNaWIAiCIAhHjoGYcPrZS+okSVoPXC/LctHO46NiSR3A5Zsq+Kpt37twVR4/Cr2o5yQIP1t3dzfr1q2jubkZnU7HiBEjGDJkSNAStbfeeuv/2Tvv+KjKtP1/z5mWSTJJJj0hpBB6R7oCroBIUUDp0hSxLaK46zZdd9XVfdd1V8Uu9gI2UFdBwYpSRKS3QEIS0nsyvc85vz8mOWQyM3F33333t65zfT4hnJNrnvM8zzlz5jzX3Pd1U9K18lU7pk2bxvjx43E4HBQVFbF169Z/aR9VKhUGgwGn04nb7Y7Iu+yyyxg/fjwAFouFZ599NigSqgOxsbHceOONJCYmKvusVisffvghxcXFyr7k5GRmzZpFYWFh2OM1NjZSXV2NRqOhd+/e6PX6f3aIUUTRLVxniml57jlsX3+N7PMRN3o0KauvI3b06CCep7qGiquvxtfYGNJG2rp1pN50o7Ld9NRTND/2eMRjZv3hD2jz8/C1tdH89NO4i07/6wb0j0IUEWNjv1cgS117C/qhw1AlGGh48M84Dx0KT1Sp6P3pJ2iyA6WfncdPULF0KXIY0VxbWEjB5ncQO72/W199lYY/PRjs7SUIpN9xBynXrQp6vezz0frqa7S98QbeqiqE2FgSZ80kdc0aNJmZIceTZRnnoUM4jx5DiNFhmDw5LC+KKKKIIoooovjPxH+i4KQmYBo+BaghYBp+tSzLJ7vw+gE7gAK5vWPtpuEHgQvaaYcImIa3RjpeVHCKIooo/lG43W4+/PBDTpw4oezTaDRMmjSJCRMmKAKVLMs89dRTNDU1hW0nLi6OqVOnYrVaaWhoCPFu+t9AFEWSk5OJj4/HZrN1mwY4btw4pk+fDoDX62XDhg1h+yyKItdccw25ubnKPofDwZYtWyjtiIQA1Go1EydOZNKkSSFinSRJlJSUUFZWhiAIFBYWUlhYGI2eiuL/BN6GRlpffBHL9u1IdjsxgwaRvHIFhsmTg3iSy0XV6utxhHkeMC5fTsadv1GuZfvevVSuui7iMbP++ADxF1+M32Si9bXXML0Z2YwctRp1aip+sxnZ6fznBvkvgK5vX/TDhyMa4rF98WXEtEWAjHt+T/LixQDYv91PZZfoqM7o+cLzxF90ERC4H9b+/A4sH30UwlNnZpL/5htBYpK3oZHqW9fiOnrsPFEUSV6xgvRf/iIkMspVXEzriy9h/+YbEATiLryQlFXXouvdO2zffG1t2HfvQfa40Q8bFpEXRRRRRBFFFFH88/iPE5wABEGYCTwKqIAXZVl+QBCE+4ADsix/0M65h4Bf06+7vHYVcGf75gOyLL/U3bF+qILT89VN/LablDq1AM8OymdWWtK/sVdRRPHjgslkorq6GpVKRUFBQVAKWwfq6up45ZVXcHVJAdJqtSxbtkwRb2RZ5umnn6YxTDRGB/r06YPP58NsNtPaGlFH/4chCAIZGRnExcXhdruprq6OyC0sLGT58uVAQDx66aWXgvTJG6MAACAASURBVNL0OmP69OmMGzdO2bbZbGzcuJG6urogXo8ePbj66quJ6+LtY7fb+e677zh9+jQ+n4+cnBzGjRtHZjcRDmazGafTidFoRKfTfe/Yo4iiA5LHg+mddzB/8AH+5ha0+fkkLV6EYerUEOG06bHHaX7qqZA2khYuJPPeexS+r62N0mmXIVmtYY+ZcedvSF6xAgD7/v1Urogs3qDVYpgyGclmx1tdhaf83D830H8BhNhYVPHxSHZ7RMN3CAhZqTffhBgfj/tsKY0PPhiRmzjvKrIfeAAImK2Xz1+Au6goLLdrhJr9m2+ouulm5C5Rn0JMDD03PEvcmDHKPlmWaX7iSVqeey4oiivu4kn0ePBBVEnBz01+m42211/HvHUrktmCrk8fjMuWhoiWHZDcbmw7v8LXUI8mJ4f4iRMRolVGo4giiiii+JHiP1Jw+nfihyo4WX1+frL/NDVub7e8+RlGHujTg0SN+t/UsyiiiKIrzGYz3377LWfPngWgoKCAsWPHkpycHMQrKSlh06ZNYX2ZBg8ezPz585XtF198kcrKyojH7EiRs9lsEc3Q/1kkJiYq4lBtbW1Enl6v57bbblOEuFdeeYXyCJETvXv3ZtmyZcp2W1sbL7/8MmazOYgniiLz589n4MCBQftramrYsWOHMicajYbhw4czderUsMKTxWLh2LFjWCwWkpKSGDp0KPGdStmHg9/vRxTFaBXAKICA75Tpnc14q6tRp6eTeOVc4i68MOT6cBw8SPWaW/CbTEH7jUuXknHXnUqkjizLVF1/A/bdu8MeL/0Xd5ByXSCySnK7OXvxT0LaVKDVknnnb5A9XryNjbS+8ML/FwP1fwiCgH7kBaji4vHb7Ti7eTYT4+Lo+fxzqJKSELRazi1egj9CJKmmRw8KP9mB0O7D1/LCCzQ+9Jew3NhRo8h97VXlHPpNJiqWrwgxZgdIuelG0tetC9pn3bmTujvvwt/pCwF1ejrZDz1E3NgxQVzZ76ftzTcxvfkm7rJyVMlGEmfPJvX660NELzhfkdB14gRCrJ6Eyy4jpn//iHMk+/14q6tBpULTo0f0vhVFFFFEEcX/F0QFpx+o4ARQ6nBx48kKTtjOh+EnqkSGGGLZbTrv7ZCt0/BI/1z6xul4taaFQxYHepXAjNQk5mYkoYumskQRxX8MSkpK+PTTT5VIJ51Ox+jRo7nkkksU43KA0tJSXnvttbBtGAwGfvrTn6LX65Flmc2bN3ebrhcTE4NKpcLhcIQVu/630Gg06HQ6bN/jOTNnzhwyMzOJiYnhgw8+iChOaTQafvaznyk+UXV1dbz44ot4vaECfF5eHitXrgxK2fvuu+/4+OOPkTp5zqhUKmbPns2wYcOCXu/3+9m3bx8HDhygra2NmJgYhg4dyqRJk8IKVH6/n5KSEmpra9FqtQwYMCBixcAOeDwexWw+iv9O+G12LNu24T5zBjHBQOKsWej6hBb28Nts1N31W6w7dij7hJgYUm64ntSbbw4SDUzvvU/db34T9ngZd91F8vLzAm71utuxbt8evnNaLYXbtiJoNEhWK9W3rcNTFr4KLoC2IB91Sip+mw13aSmEed/9p0E/ciTanj0RdDrM778fEgnVGZl/fIC4MWMQY2NpenQ9prffjsjN37wZ/eBBALhOneLcosXIYeZDiImh1/vvKSb4sixTe8cvsGzbFsLV9upF3sbXURuNyj7XmWKqbroJX5fo0MQrryTrD/chqM9/qSjLMqa33qJ5wwZ8tQG+tqCAtLW3kDBzZsjx/DYb5i1bsH65E9nnJXbEBRivXoImKyvsmP02G7avvkKy2ogZ0J+YoUO7FbNknw/J6USMi4saxEcRRRRR/AgRFZx+wIITBB4sDlgcnLG7MGpUTE5OQK8S+bLFwu2nq6j3nH/wUQO+Lq8fGq/n7eGFJEUjoKKI4j8GsizT2tqK1+slJSUlohBx7NgxPvroo6B0vfT0dBYsWEBaWpqyr76+ng0bNgQJLB0QRZHVq1eTnZ2NJEns3LmTr7/+OmLfNBoNiYmJ2O12nP8fPWcKCgrIzc0lJiaGI0eO0NDQEJG7aNEiBgwYAEBZWRmvvvpqWJ4gCKxevZoe7eXfJUni7bff5vTpUINoo9HIddddFyQ6NTc3s2nTppB0x9GjRzNjxowQn6rTp0+za9cuamoC6dH5+flcfPHFFISp7tXhfXXmzBkkSaJnz54MGTIErVYbdiyyLFNfX4/L5SI9PT0kXTGK/2x4KitxHj2KoNESd+F4VAkJYXnWzz6j6cmnlNQzbe9CUm+6OahaH4CvqYmK5SvwnDsX3IBKRY+/PETCjBnKLssnn1Bz621hjycmJCiV8gCan3mWpkcfjTgOw8yZxE+aiNQuuDkPH448aFFEjIlBcjgic/7DoOnRA/2wYQixepwHDobObyfET51C6k03I+pjcBw5Qv1dv43ITV65kozfBBwjJJeL0ukz8NXXh+WmrllD2tpblO3mZ56h6dH1YblZD9xP0rx5yra3vp6KFSvxdomWFWJjyd3wLLGjgtcHrRs30vTXh4POUczQoeQ88jCa9vtmB3ytrTQ9/jiWDz5EsttRJSeTtGABqTfdGGQ63wHX6dOY3n4bT0Ul6tRUEufOIXbcuLBilizLuI4fx1V0GjEujviLJ6EyGMKOuYPvb2lBUKvDRo9FEUUUUUTxf4eo4PQDF5y6g8nr466SGrY0tHXLm59h5ImBef+mXkURRRT/Sni9XkpKSnA4HKSlpZGbmxv2Ab2oqIj3338/qLKdVqtlzpw5DBo0SNnncDh44okncERY9C1YsEDhnz17ltdffz1i39RqNYWFhTidTkwmExZL94UO/q/QUdlPp9NhsVi6Fcpyc3O56KKL0Ol01NTU8Omnn0bkjhkzhpntEQNer5cnn3wSU4QUpylTpjBx4kRl+8CBA2ErGAqCwMKFCxWBDMDlcrFx48YQr6yEhASWL18eJC5CIEpu+/bttLS0AAFRcfDgwcycOTPEZ8zn83HgwAGOHj2K1WolOTmZkSNHMmTIkLBG7g6Hg2PHjtHa2kp8fDxDhgzB2CkSoys6DPF1Oh09evT4XnN4SZIQBCGa/vMPQJblQAqXJKFKTY04d36bDdPb72D95BMkpxP90KEYly0jpl/fEG7rK6/Q+Je/BkXrqNPTyXn8MfSdogD9VivnFi0OGxGlzcsj/603lQW+u6ycsssvD65m1wkdvkyyJGHfu5eq1ddHHLOYYMC4dCmyw4m7vBx7NyL5DxIqFfqhQxH1MfjaTBG9rAAEvZ6Mu3+LKi4e2eel9pe/gghp1GJSEn2+2onYnmZcufr6iCmcqtQUen/+ucI1b91G7R13hOVq8/Io+OBvCtdvMnFu8ZKwAlzsqFHkvvgCQiexvOXll2n8U6i/V+L8eWTdd19QZJS3ro6adbfjPHo0aA7Sb1+n+KF1QJZlzO++R8vzzytG+Prhw0lde4tiZt8ZvpYW2t54E9vXX4PPR+yYMRiXLUWbkxN23O6yMqw7digFCQxTpgSNq2tf3KdP42tuQZufh7Znz7C8znzZ60XQaKL3wyiiiOIHjajg9F8sOHXgl2eqeLW2JeLfNYLA0YsGkRyNcooiiv9quN1uTp06hdlsJiEhgUGDBoX1N6qtreWtt94K8k9SqVRMmTKFCy+8UNknyzIvvfRSRC+pGTNmMHbsWACcTid//etf8fm6xlmeb3/SpEmKKfqxY8fC8v6TIAgCOTk5aDQaXC5Xt35WWq2WK6+8UhF8Nm7cGHEuDAYD69atU1Iot2zZwvHjx8Nyk5OTueWWWxQhp7y8nFdffTVsamTPnj259tprFa7X62Xjxo2cC7MoHDZsGHPnzg1a6BQVFfHuu+8GpS4KgsCUKVOYMGFC0OtdLhdbt27l5MmTSl8SExOZPn16kJjWgZKSEnbv3k1lZSWiKNK3b18mTZpEVpi0HpfLxZEjRygpKUGSJPLz8xk5cmRED66WlhZOnTqFx+MhKyuLfv36BaWndoYkSdTU1GC320lJSQkR87rC5/Nht9vR6/URo81+qPC1tGDZsQN/Wxu6wkIMkyeHXUz7Wlpo/MtfsWzbhuzxIGg0JMycSfodP0fdZf5MW7ZQd/fvQkSn+MmTyVn/qGKuLcsyVTfciH3XrrB9y/rjH0m66soA1++ndNpleGvCF1LR5OZS8Lf3kV2ugBCyaDFSN+J3wuWzUCUkIjnsmD/cGlG8+cFCrUbU6xE0miCvqXCIHTcOXZ8+CDodpi1bkNoif4GZfMMNJEydgqDT0fb6RkzvvBORm3X/H0hq9yR0Hj3KuUWLI3Iz/3AfxgULgICPVfncKyOmfGb96X9ImjtX2Y4YgSeK5Dy2HsPUqcoud1k5lStX4uviBRYu2kuWJBruv5+2TW8EcTXZ2eQ88zQxfYNFXMehw9Tfcw/u4mJlX9yECWT94b6Q1EXJ5aLluecxvfMOvsZGVElJJF51Fak33Rg20tFVVETbpk24zhSjSkggYeZMEi+fFfa9Kvt8WL/8EuehwwjtRQj0Q4eGzk87/CYTjoMHkSWJ2AsuQP096eG+1lZkpxN1RkZQmmcUUUTx40ZUcPoRCE6/P1vDs1XhzTQ7sGV4IRcZg8ORyx1udrVZEQSYZDSQp49WfIoiih8L/H4/xcXFNDU1odfrGThwYNi0LIfDwfvvv09xpwdpjUbDpEmTmDBhQpBgsXv3bj777LOwx5s2bVqQmPXSSy9RUVERliuKoiKcuFwu3n33XezdVMpKSEhAq9XidruxRqgW9p8Gg8FAbGwsoiiGVPXriuHDh5OVlYVGo2H37t3dVjCcO3cu/fv3R61Ws2/fvojnA4JTEZuamnj66afDpmV25X5f9cKlS5fSp5N/UaRoL7VazbJly8hv972BgJn8K6+8EhJJptfrWbZsmZIO2dGPHTt28O233wZxjUYjS5YsIT09PWh/eXk5W7duVSLDIOABNnfu3JAoLqfTyRdffMGRI0fwer2oVCoGDx7M1KlTMXRJ7ZFlmaKiIvbv309DQwMxMTEMHjyY8ePHExsbGzJus9nMgQMHqKqqQqVS0a9fP4YPHx5W0OpItSwtLUWWZQoKCroV1CwWC2fOnMHr9dKjR4+IEZEdbVdWVirRk98nvlmamjDV1JCUlUVCRkZEnvvsWVrffAtLeRkxCQkkX3458ZdcEuLvI9nt1N93H+at2xTRR2U0knbbrRgXBwsUjkOHqbr++pCKeaLBQO4LzwctqlteejlitbzYsWPJffklZU7qH/gjbRG88gDyt2xGm5eP7HRQ/8c/Yv04gk8WoB81irgxY5BcLmw7d3brk4UoIsbHIzudYT2hfvDQaNCkpyNotfhaWroVAEWjkaTZsxF0Ojzl5Vi7iTpVZ2SQ/ec/I2g1SHY7VTfeFFEw1GRnU/jpJ4qZ/LlFi4OipoLaTUuj9+efKSJO84bnaHr44Yh9KNz+sZI26DpzJuDt1aVSLbSLoe9uQdUulsseD5Wrr8exf38IV9evH3kbX1e4AG3vvEP9734fUhBAP2okuRs2IHa6v3iqq6m64caQ685w6VSy//IXJToNAiJu06OP0vrqa+f9ztRqjIsWkfHrX4VUXXQcPEjjI4/gPHBQmS/jiuWkXHddyPvab7PR+uqrWD74EH9bG9rCQoxLlpBw+ayw9yLHoUOYNm/BW1uLJiuLpHlXhaR6dkCy27F8/DHus6WokhJJmDWr20gyd1kZzqPHEPUxxF14YcTUZQicG3dpKQC63r2/t/Kk7PEg+/1h00ejiOLHhqjg9CMQnNafa+B/yrtfsMSKInMzkrg6K4WBcTHcUVzNu51S8QRgXoaRv/TrSYwqavoYRRRRBKO5uZnq6mo0Gg2FhYUhqVvQ7jl34AC7du1S0usSExOZNGkSI0eODOI2NTXx0ksvhU3tmz17NhdccIGyvX//fj766KOw/dJqtaxdu1YRAXbs2ME333wTcRwDBgygV69euN1uTp482a3YI4oiWq0Wj8cTUYj5IUOv15OdnY1araaxsZG2bqIbjEYj48ePV7j79u2LyE1PT2fJkiWoVCq8Xi9PPfVUxEqKqamprFmzRlmIdCdEJiYmcuuttypiy549eyKmRCYkJLB27VrFH62uro7nn38+bD8SExO56aabFJN6j8fDiy++SH0YTx2j0cjq1auDxNnPP/+cXWEidZKTk1m1alVQZNa5c+fYtGkTHo8nZB5WrlwZJGbZ7XY2bdqkeIB1ID09nWXLlpHQafEkyzKfffYZe/fuDYp+y87OZtGiRUpVyw6UlJSwdevWoCjH/Px85s6dS1IXDxyTycTHH39McXGx0nbfvn2ZMWNGiFDn8/nYvXs3Bw4cwGazIYoiAwcOZMqUKWFTM4uLi9mz8ytq6uvQqFT0HzSIiZMmhVT4BKg/eZJdW7ZQ1S465cXHM2HefDIGBkfUybJM3VNPcfCzz6nOSMevUpHa3MzQ9Az6PXB/0KLTb7VybuVKqlpaqMjLw6PRkmg206usjPzrV5N6880K111eTvm8+dgEOJefj0MfS5zdTv65c8RrtfR6/z0lmsV5/DjnFizEp1JR3TMHiyEBndtNbmUlepeL9Dt+Tsrq1QDYDxygctlyZKDVaMSSkIDa5yO9qQmdx4M6J4f0deuQ3S7cJSW0vvwKAB61GpdejyDLxDocqNrvUfoRwwEBv9mEpyyQZiYDsiAgyDI/puQtQadDiIlBEEX8ne5vHe+QznMRM2xYILVOpcK6Y0e3xvMJs2cTN3YMgkZD66ZNuI6EF7IAUm68gaQFCxDUaswffUTTnx+KyE1dewtpa9YA4KmooHTmrIiCWvKqVWT88heB8fj9lM2Zg+dsaViu8eqryfzd3cp2w0MP0frCi2G5SQsXknXfvcq2ff9+KlddB2Eidrty/RYLFcuWB0V6de5Dxt2/Ve71sizT9PDDtDz3fOjYrrmG9F/9Mkigcnz3HdVrbw2u3CkIpNx4A2m33RbE9ZtM1P7q19i++uo8Va8n9aabSLnh+iCuLMu0vfYazc89h7+pGQgIaik33Yjx6qtDRDLnyZM0P/FkoG1JQtevHymrriVh9uwQrt9mo+31jVi2bcNvNqPr0wfj0qUYJl8SMmZZlnF8+y2md9/FV9+ApkcPkhbMJ7bTs1Bn+FpaMP/tAzzlZaiMySTOvgJd795hubIk4di/H+fx44ixsRimTEGTmRmWCwFPQMehwwgqkdjRo1F1+fwIatvvx332LLLHi65Pb8Qwz4adIdntSA4HquRkRQyO4oePqOD0IxCcKpxuxu0r4u89ewaViNUffvG0JCuZR/rn/us6F0UUUfzo4Pf7aWlpQRAEUlJSIvr6mM1m9u7dy+nTp/H5fOTk5DB+/PigiBcIPIht3749JJIlJiaGRYsWBZlwW61WNmzYEDbSyWg0cv311ytRJ3V1dTz77LMRx9E5Kuv7/KwMBgMTJ07E6/XS2NjI0QjfpHcgKSkJSZJwu91Bvls/RhgMBrRarWKm3x3y8/MxGo2IosixY8fCVi7swNChQ8nLy0OlUrF///5uUyLHjh3L8OHDUalUHD9+PKyA1IGLLrqIqVOnIggCtbW1bNiwISJ3xIgRzJkzBwgIWY8++mhE/7R+/fqxZMkSZXvjxo2UlJSE5fbs2ZNVq1YpC5zuxLf09HRuvPFGRairrKzk5ZdfDiuiGo1GbrrpJiUV1263s2HDhiBhqgMJCQnccMMNiqAmSRJvvfUWZ86cCeHGxcWxevXqINHpm2++YUenan0diImJ4dprryWjUxRVRUUFGzduDBHqtFoty5cvp2enKAebzcbLL79Mc3NzEFetVrNkyRIKCwuVfX6/ny1vv82pLn0WBYH5CxYwcODAoP1fb97Ml8ePI3daWIqSxGVjxzJ2VrCR+7Enn2RbVRXuTgswQZIYbbMz48E/BaUXnlm3jh2STFtKchB3QNFpZqxdi2HSRIV7dtFi9uhjOJefD+39ECSJPiUlXGRMJu+pJxVu2Zy5HBMETg0aiLc9ekf0+elVVsqIU0UUvvUWoj4Gyemi+tZbqbNaKSssxB2jQ+X3o/L5SGtqJreiAuP06egKeyG7PZjefRen2UxLSgoenRaV34/o9xPrcJBgsaJOSECTnobk9uCrr0fyeJBEEUGWEf8L1xr/G0gd51CWEUQRdXo6glqN32pFMpvxqlS49Hq87deLzu1G73AgqtUYpk5F0GrwN7dg37sXj0ZDW1ISjrjAZ1y8zYaxtQ21IJC2di2iIR7Z46HxL3/FCzRkZNCWbEQSRZJMZrLq6tD6fPR4bH1AkFCpqL3jFzjLy6nNzqYuOwufWkOiyUT+uXPEOZ3kb9lCzMABCIJAw58epOXll2nIyKAiPw+3VkeC1ULh2VIMNhu5r7xC3NgxAFi//JLqm3+KKTGR8l4FOGJj0TucFJSXYzSZ6PHYehKmTQMCAkjp9BnY/X7KexVgTkxE6/aQV1FBaktLcAquLFOxbDmWo0epzMulJSUFlc9PTk016Q2NZP72tyQvW6rMf/PTT9Pw2OPUZmfT0H7PyayvJ6uujsyf/0wRhiEQaVlx7bU0x8dT06NHQMxuaaZHdQ0Zt6wh7ac/Vbh+k4mKFStprq2lKrcnbq2OJLOJnpVVZN54A2m33qpwZVmm4Q/3U795M5V5uYqYnVtZSdaqVaTfvi7omrF+/jkVv/gl1WmpQWJ2jxXLSfvZz4KEL091DVW33EKFyURbsjEwF3V15C5eFMKVPR7q//hHyj77nMaUFARZJqu1lYJ580i77daQaDbzhx9S/vQzVIoikiiS4XDQe84cUn96c4iY5Coupnr9YxSXleFVq0kB+s2cSer1q0PSM/02Ow3PP8fp3XuweTwYY2PpP20aqcuXhUSeybKMZccnFP3tfdosFuJi9AyYMpnU+fPDpp26y8oo2bSJupoadFot/SdNIu2KK8Jy/TY71e+9y7kTJxFVIn3Hjyd9+vSw0W+yLNOyZw8lu3cj+SUKLriArGmXRhTVnDW1FH20DY/TSc7AgfSYPDlilU/J7aZ4+3asLa2k9yog9+KLu/V9q/zuO5rPncOYlUXehRd2663ZUlVFbVERsUlJFIwa1S3XbrVSXVSEWqcjb9Ag1J3OW1Rw+hEITgAPldfx13OhVZxiBIG5GUa+arNS5/7+sG0R+G78QHrEnH/j+WWZT5stbG8245IkRiXGsSDDSGLUEyqKKKL4N6KxsZHjx4/jdDpJT09n6NChYSOtWltb+fjjj5XFuiAI9O/fnxkzZgRFhQDs27eP7WHKyQ8cOJB58+Ypi3RZltm4cSNnz54N27fFixfTv39/ILDwfuqpp0IWvB3Iz8/nmmuuUbiPPfZYRDNygPnz56PT6fB6vWzdujWiYAGQk5NDUlISPp+PsrKykAV6Z3T+pjmKfxyiKAaMf79n/tLS0lCpVLhcrm7PMwQih7RaLV6vN6xw0xkjRoxQrue9e/d2K76NHTuW7OxsRFFk9+7d3VZ9HDduHAMHDkQQBA4dOsThbirPjRo1iokTJyIIAmVlZbz//vsRuUOGDOGqq65CEATMZjOPPvpoxLnr+h5Zv359WNELQn3O3nzzzbCVJyEgZt1+++2KoLZr1y4+//zzsFyVSsXatWuViK/S0lJe6yb9bvXq1eS0m0+bzWaefPLJiO+/hQsXKmKW3+/nmaefpqm5OZA61WUhMX36dMaNG6dsv/P665w8ezYsd9wFFzB99mxl+6t3NvPlyRPBB29/XR+DgaU//7my+8Q777Dl+HHkzguOdm6a1cqNDzyAuv1+W/n222w8dCggpnX0o/23wWJh1aJFGNv9/Vree4/Xv/yStvaoNUGSEP1+RFkm1m7ningDPa+7DtnrwbpzJ19v30FjRjqiJAWELElC9EtovB4GNTSQc/XV4PXiLjnLuW++oTktVeGoJD+iX0KU/KQ2t5A0cACBaC8z9spKXHp9gNvlR/VfGMH6b4dKBX4/flHEp1YjCwKSKLb/FtC53cTE6NHk5CCoVHgqKrD6fNgMBoUniwKSIJJgsZCs1RI/YQKCRo3r9Gmay8qpzc5CUqmQBFHhJrW1ke9ykbJ6daDdqkqqN2+huF8/PDptMNdkYnBNDTl/+h9EnQ7J6aT0jl9weMhgLImJgT609zvBbGb0seP037QJVWICiCLl161mtzGJmpwchSsLAnF2Oxd/vYsR77+HJjsbgLp77+Prkyco6iJa61wuLtq9h9HPPoO+vUCLees29jzxBAdGjUJSnxcoVF4vY7/dz4W/u5v4SZOAQOTbvpXXsGfc2BAxe/iRI1y8apXicyZ7PBxduIgv8vOxJHWKUpJl+p05w6XTLiN11bXK7vI772JHSzP1XXzHcisqmD5kKNk/u13Z1/ree2zbvIXS3oVB96GM+nqmZ2ZScM89yj5XcTE77votR/v1CxpfUlsbl2m09H/oz8rziN9mZ/cta9ibmRk0vjibjcluDxc8+ogi4siyzIl77+UTiwVrp2c7ncvFRRYLEx55JCiVtPb993nv009p6pQ+rvZ6GdXayqUP/QVV/PnIZevp02x5dD3nMtKV8Yl+P4NaWph9//1oOkXh+qw2Pvjd3ZyIi1fGJ0gShW1tzPv1r9F3sgKQZZmvHnqIvW1teDr1LcdkYv5Pf0pSF3+4E1u2sP3bb7F1Sp9Ns9mYd/XVZHaJfqs5fJj33nyLZv35eUtyurhi5gwKL744iGttamLzE09Q0encxXk8TLvoIoZ1+fLE6/XyweOPc9JkQmr/fIjx+ZgwZAgTFi0KjDcqOP04BCdZltnS0MYzVU2csDlRC3BZaiI/z89kYLwevyzzZauVP5fVcczWfanz63ukcntBJskaNVafn+XHythnDvZNSNGo2TSsF8MMof4UAJIs45VldN9TsSiKKKKI4v8KVqsVm82GwWCIaDgNEtCRRwAAIABJREFUgUingwcP0tzcTGxsLMOGDaNPnz4h3/R4PB527NjBkSNHlNSspKQkpk2bFhIJ0djYyKuvvorNZgvan5yczMqVK4NSnIqKinjrrbfC9m3ChAlM7WR82116YXx8PLfccosiwu3du5dPPvkk4riXLFlCv3798Pv97Nu3r9uKfb1792bMmDH4/X7Onj3LwYMHI3K1Wi3Dhg1DkiRaWlrCmpZ3Rkf6lNvt7tarK4r/Dvy9QmdcXBwqlQq/3/+910VKSgo6nS6QTvc9nmjp6ekkJSUhCAKlpaURzf0BMjIyyM7ORhAEzp49220lztTUVPr164cgCFRUVET0OINAGueYMWMCUSENDd1GROp0Oi699FJUKhU2my2iQAYBEXT27NnodDr8fj8ffPBBt6LzFVdcQXJyMoIgsG3bNpqawviBtgtJ06ZNIz8/H0EQ2LVrF6dOnYrY7oQJExgxYgSCIHD86FG+7JTa1BXDBgzg0lkBf5/aqio2vvlmWDENoGdCAstvuQVBEHDYbDz6yCPBAlknJPj9rLv3XkRRRJIkHrnzTqwR0n3UXi8/u/VWdHHxSB43b9z1WxriYsOKU6IkMX3AQFIGDgCvlwMvvURVjL6TeOVHkGREyY8oyfRRiaQPH4Hs9VK/Zw/1KlX4dmUJg9tDQo8eyF4v7oYGHF04giy3c//71ms/ZEjtwpNKFAMCh0qF127Ho9EEiVgd/wdIEkV0OTmgErGVl9OqizkvYontrxEC13aWz4dx5EhQq7CdPEWp34ckqhSecgxBJNdsJnfObARRhbP0LIfKz+GKiQniyQLIgkjPpiaGXbcKQaPFbzLx1Qd/oy05GZnzfZABWRTJbGhkytpbEA0GkGW+/PNDlGVnKX/vzDW2trLwxhvR9sxFEAX23Hcf+zoJep1/Yu12rps3j8Tx40EUOfnoera0tYZ9X2s8HlaOHEXOvKsAqP/0U1784osg4aYDgiQxPyODQe0pqq7aWp5+6CHMiYlh7y9TBJGJv/8dEEgTfHHd7VSlJIfljnA4mPPnPyvbf7vzTg53jZBqf12B2cyKhx9WPvf2Pf0027t+4dPOTbFaubmTsF/2xZe8/uUXSJ2jpNq5sU4XN992K4Z2gdNUUcHTzzyLW6cN6bPa5+O6BQvIaq9A6/N4eOr3v6dVpwsdnyyzaMIEBlx6qbLrtT/8gdIIqb2X9u3LRYHU06jg9GMQnDrDLUmoBQFVmA/rt+tbubUofMWprhgQF4NXljnrCJ/ukaXTsG/cgCBRqcbl4a/n6nmvwYRTkijQa1nVI41VOalh+xNFFFFE8UODw+GgsbERrVZLZmZmxBBkp9PJkSNHKC8vRxRFevfuzdChQ8OaQ5eUlPDFF18oi+XExEQuuugiRo8eHeI38dlnn7Fnz56g1ycmJrJkyRIyO/ky+Hw+3njjDUpLQ309LrjgAq644gqlbbfbzfPPPx92wanX67nhhhuUdCiPx8MTTzwRcfE9c+ZMxowZo/Rh/fr1Ec3cCwoKWLlyJRCIZHn88ccjekmp1WrWrFmDWq3G7/fz+uuvR4wiA5g8eTJpaWlIksTOnTvDL6bbkZubS15eHpIkcfr06SBj8a7Q6/Xk5OQgSRJNTU3dihAQSD2TZRm3293t4j+KKKL41+PvFTjVarVyL/d4PBFFLwgUzlDSTm22bi0tNGo18e3ebFazGV830VQqQSA5NRVBELCaTDgj3S/avbgyM7MQBHCYzZhttoAnlQxCe48EWVa2U1NSUIkiXpcLk9l8nqf8bvf3kmUMsbFodTrk9qqy59uSEeTAQTr8wHSCiFYfiHRzWANfsCjttbfd0Q+VLKNSq0GWkLze4D7IstJv2sehtPWP7usyD13/1HHMoO0uvG5f+0/0pbv9/+hr/i7u9/T77/nb/7q9CH/v/vVy4K8CIb9lBFSyjCZWD4KI1+nE1y50IQCCEPDXbxfVVJKEISUVRAG32YxdkoLaEuB8u5JEeo8eCGoVbrOZJrs90JOO9pW+gCBDz545qGNj8btclFdVBUQ3hMA9Qzg/LFkQ6JWRQVxGBjJQdOQI3vYIQIROM9X+//zERNL690cQBE7t3Yulg9vp+B3bOTothe22D2f27KHW51eOHejL+XbTgREzZgS+EDl0iJONjUpbstA+F+3bBp+PKQsXgijSWlnJzgMH2udYUPrtVatBFNF7PPz83nvRaLVRwenHJjh1hzq3h1F7T/GvKgL8+IBcFmQGvp2udnm4/GAx9Z7QbwvnZxh5fEBwtRyvJPNmfQtv1rVS5/bSM0bL1VkpzM80RhSnbD4/tW4vRo2KNG33FSSiiCKKKH5osFqt+P1+EhISus2lb2tr49SpU7jdbjIzMyNWLvP7/Rw+fJgjR45gs9kwGo2MGjVKSZnqDLvdzvbt2zl58qTi79OrVy9mzJgRUsGsqamJN954I8hzSRAEJkyYwOTJk4Parqqq4vXXXw/xqjIajaxcuTLIpLq0tJRNmzaFNffuaiZfUlLCpk2bwi4ke/XqxfLly5V+dJcOpdPpWLNmjZKeVlVVxQsvvBCWC6HV/Z588smI3DFjxjBz5kwgcG4feeSRiAb0GRkZXHfddYq3V3cpWSqVimXLlqHVavH7/bz99tsh0XSdcfHFF5Oamoosy+zcubNbr6zevXsr4tuJEye6FeqMRiOFhYVIkkR1dTWN7Q+x4aBWqykoKECWZUwmU7diIQQikQRBwOVyRUyn60BMTIwSyeIKUy0siiiiiCKKKKL41yDBZMLS/uy2fMoUek+aFBWcooJTMH5dXM3LNeEf9GanJTLEEMtek41vTDZcUvfXhADk6bUU6HVUOT2cdUY2v908vJAJxsC3PF5J5toT5XzWEvrN8Jz0JJ4emIfYacFi8fn5Q2kt79S3Kn26JNnAPb170C8uNETa4vOzub6Vo1YnsSqRy9MSuTApPqLJms3n56jVgSgIDDfEoo9W6osiiih+pHA4HJjNZuLi4kI8rzpDkiRKSkqor69Hp9MxYMCAkGpoHbBYLBw8eJBz586hUqno27cvw4cPD+vBVV1dza5duygpKUGSJPLy8pgwYQJ9+vQJ4Z45c4YdO3YoAopKpWLYsGFMnz49JJLs8OHDfPTRR0FeRwkJCSxYsCDIdBoCqYsff/xxiJg1adIkJk+eHLQvkvl1VlYWK1euDBpjJK5Go2HFihVB/fjuu+/Ytm1bCBdgypQpTJw4Udk+evQo7733XlhuQUEBK1asUD7/zpw5wxtvvBGWGxsby5o1a5QqfN8nvl177bXk5eUBAe+0J554IqKgdumll3LRRRcBgei/Rx55JKKgNnDgQBYuXAgERNP169dHjCRLSkri1ltvVQTa5557LqSyXwdEUWTdunUYDAZkWWbz5s3dpobNnz+f/Pz8gO/GV1/R3fPk2LFjFUH0xIkT3RrP5+XlcckllyDLMjU1NXz22WcRufHx8YpoabFYwnrOdUAQBC677DKluuann34asUIkBATRjgIGu3fv7las6927N9nt6RtHjx7tVgRMT09XijmUlZV1K1rGxsbSt923pKGh4Xsrh3akLVosFqqrqyNyITDPGo0Gt9vdbYojBLzW9Hq9Ip52h440bVmWaWpq6naOdTqdktJtsVi69VoTRVG5Nl0u1/dGRHYUwPB6vd22C4F7jCAISJLUbRopBK4jQRDOe9R1E+0VRRRR/PjQWXBaPHYcA2bOiApOUcEpGF5J5t7SGl6tacHTfs61gsC1PVK5uzAbtRj4UDlhdTD1QGhp038WfWJ1LMhMJkOr4ZjVwQsRRC+ApwbmcVVGIH3DLUlcefgshyyhRrmJahXbR/alIPZ8/u4hs51lx8to9QY/AExPTeDZQflBKYB+Weah8nqeq27C3l65z6hWcWteBjf1TAsRqIpsTtZXNPBJiwWfJDM6MY5bctO5JCV0UeaWJLbUt/G3RhMWn59B8XquzUllULw+7JiL7S7eqW+lweMlX69jUWZykHl7Zzj8EtuaTJQ53KTrNMxOSyJFG97EXZZljlidnLA5MKhUTE5JIEEduRRpk8fLAbMdtSAwPime+G64bknimNWJT5YZEq/vlivLMmVONza/RG+9jrhuuAAtHh+tXh/ZOs33cp1+iRavD6NGRdz3lFmVZJlWr584lfh3CYt2vx+1IPxdfmQ+SUYGNOL3P5jJcoArRh/ioogiLDoWO91FekFA+GpoaMDtdpOenq4swsLB5XJx+vRpbDYbKSkp9O3bN2xkGEBLSwuHDx+mra0Ng8HA8OHDg1IWO6OiooL9+/fT0NBATEwMgwcP5oILLgibPllUVMTevXupqqpCpVLRr18/Jk2aFLbto0eP8tVXXymCWmJiIhMnTmTkyJEhn08HDx7k888/V0zlBUFg0KBBXH755SHC3sGDB9m+fXvQItVoNLJw4UKyuhjGHj58mK1btwYtqkVRZNasWYwcOTKIe/z4cd57770Q0Wnw4MFceeWVQXN9+vRp3nnnnZDFekpKCtdccw2G9jQkgPLycjZu3BiyUNZoNCxdupT8TpUta2trefnll8Mu1jtXnoTAOX7++edxOkO9Lfv378+iRYuUebbZbGzYsCGs8JWcnMz111+PXh/4fPd4PDz33HNhhRaNRsPq1auVKnySJPHyyy9TWRne7mDevHkMGTJE2d68eTMnTpwIyx03bhzTp09Xtj/99NOQ9NsO5OXlcc011yjj+0f84YqLi9m0aVNYriiK3Hjjjcr4GhoaeOaZZyKmtM2fP5/BgwcDgSjL9evXRxRaOnva+f1+HnvssYjCV+/evVm2bJmy/eKLL0ac44SEBG677Tbl+nz//fc5cuRIxPGtW7dOEeO//vprvvjii7BcgBUrVtCrVy8gIERu3rw5Iveyyy5j/PjxAP9QBUy73c7DDz8cUfjKzs7mhhtuAALz9uijj0ZMc46JieHnP/85mvZKXN3NG8CaNWtISUlBlmU+/PDDiPMGgXPdIS7u3buXnTt3RuROnDhRmYvi4uJAQYIIwle/fv24/PLLAWhubuaVV16JyE1JSWHZsmVK9OTzzz2HL8K8adRqVl13HRqNBlmWee2VV7CEiyRtP9bixYuV1PMP//Y3qsNVRm3nTp06Vbku9nz9NScjFDoAGD5sGKNGjwbgxPHj7OtSsbczcjMzmTJjBgA1NTUBH8cIc2HQarly8WIEQcBms7Gl49oMwxWBBYsWodFo8Pl8bHnjDbzhnl/bjzVjxgyMRiOyLLPtrbewdJNKOnbECAr69UOWZXZ98AG1Ye7HHeibns7wn/wEWZY5+uWXFHcTKZuu1jBp7hxkWebcsWMcjFD1FUDv8zF93jyQZVpravh6/34lbRT5fIqqQKAy6SWTJ6PWaPA4HHz91VdKWlrn1NSO/18wcCDxKSnIksT+nV/hbV97nOe1p54C+cnJpPXqBbLMqX37cKhUXdoN/CPIMilqDdkD+oMsU3X8BObOKbLt3I52Y/0SOf37AzItZeW0ej3B7Sn/B7XfT06vAmQZHE2NNDscwfMANKek4IyLQ+31cvvatcRnZf3dgpPqnk6O8v8t2LBhwz0dN9koAlAJApNTEliWncLIhDjmZCRxf58cZqYnBS1+07RqtjaZafFG/iZkbGIcoiBg9X1/kl6r18+uNhvbm80ctkausgRw0Gyn2evjqNXBm3WtfNYa/oPRLclY/H5mpgVUVrvPz8xDJSFiE8BZhxuPJHNx8vmH2N+dreGpqia8nR6EXJLMV21WYkSRsUnnjYYPmO3MPVzCCZsLryzjB6pcHrY0tJGuVTMs4fwix+bzs/BIKS/VtlDh8lDv8XLM5uT12hayYzQM6WK2/pfyeq4/eY79ZjsnbS72mGy8VNNMtk7D4C7cr1utXHGohHcbTewz2/m8xcIL1U2katUhJu41Lg9XHyvjoXP1fNpiYWuTmReqm4lViYxMjAviuvwSvymp5paiCt5rNPFeoykgCsqB89zVR+aFmmauPX6OF2qaeau+lRdqmnH4JS5Mig8RUb5qtbLqRDkPltfzem0Lz1c30+z1cmFSvCJwdqDU4eLWokp+caaKF2ua2VDdRLXLy7ik+BDRx+T18buzNaw5VclTVY08U9lEqcPNBQlxIeKXT5J5vKKRm09V8Odz9TxR0cApu5NB8XqSw1RcfLehjTWnKrj7bC3rKxo4ZLHTKzaGLF1oKuc3Jht3nK5i3elKHq5oYI/JSqZOQ74+1MjwrMPF3SU13FJUyZ/L6/m0xYxBpaJ/GCGyxePjwfI61p6q5P7SWj5oClS4GmbQhyw2XX6Jp6saWXe6kvvO1vJWfSt2n8QwQ2yIACbLMm/Vt3LHmWruLqnhtdpmmjw+Bhv0YUW4na0Wfn2mmjtLqnmuuolzTjf94/UkhhECT1gd/P5sDb8qrubpykZO2pwU6HVhU2CrXR4eLKvjl2eqeKyikW9NNjK0GnrqQxfpZq+PJyob+cWZKh4+18AXLRbi1SJ9YnUhc+GRJF6paeZXZ6p5sLyeDxtNCMCgeH3ItSnLMh80mbizuIYHymp5u74Vu19icLw+rHC4p83K787WcG9pLa/VtNDo8TEgTk9smHkrsjm5v7SOu89W80J1M2VON31idSSFud5qXR7+cq6eu4preLqqkaNWJz1jtGSEud4sPj9PVzbym+JqHq9oYE+bjRSNmrww15tHknittoU7S6p5+Fw9nzRb0IkC/eJiQuZNlmU+bDJzV3E1D5bX8X6DCa8sMzA+BnWYB8s9bVbuOVvLA2V1vFHXQpvXx4C4GGIizMUDpXXcU1rDKzXNVLk89I2NwRDmGqpxeXj4XD2/O1vD8zXNFNmc5Ot1pIYR1c1eH89WN/GnejObLB4OOzxkaNXkhBHr3ZLEm41mHrf6eE/WcVzUYVCrKQxzDcmyzKc2Ny/5dXwcm8LpxFTi4uMZEBd6DQGckFW8rU3ii7SenM3OJykzmyEJcWjDiGWN+ni2JWawK6cvlb0HkdynH6MyUsOK5d6kZHan5bLPmEVVXh8SLhjDhN69SA4zF/q0dI5k9+KgPoHajBzUI8cxZcxoesaF3luSMzIp7VnIEUFLY2IK3n6DuWTaNIZmpIVwMzIzacgt5LgPWvVxOHvkMfKymUwZ2C9kLjIyMnDk9uKky4tZpcaZnEbeRRNZOPkn6Lqc69TUVFT5vThpc+HwS7jj4kkYPJylc+aQYgg2+TcajSQWFHLKbMPldOJXaxDzejF37lz65wZHpxkMBjIK+3CqzYTPYgFkfKkZTJg6jUvGjA7ixsbGkt+3H6eaWvGYA/cJf4ye3qPGsPiKy4PETq1WS9/+/SlqbMZtMrX7fwgk9+nLtYsXER93/jNVpVIxYMAAihuacLS10TFL2tQ0lixcQM9OVYs6qmiWNTZhbW1RuIIuhkunT2fUiBFBfe7duzfVLa20NTUpXASBQSNHcfn0y4L6nJ+fT7PVSlN9fVAb6T1zWbpoUZAgmp2djcvrpaZLJFCMwcCKZcuCUl9TUlJQa7SUlZcFcUW1mquuvFJZSENArDIajZwpLlYWLB24aMJExo0dq7z/tFotWVlZnCwqQu6yQO3Vuw9XXH7+nIiiSF5eHsdPnsLfRYhMSklh8aJFis8SBAS2k0VFeLqk9mp0OpZefXXQ+HJzczlTUoKjq1m9IDB37lwlqq9j3iqrqjCHqUA5btw4Ro8+f82lpqbS2toaNvW0oFcvpk+frozPYDDg8/nCij1JycnMnzdPOX9arZa4uDiKi0O/JNZqdSxcuFARcEVRJDU1lRMnT4acD6F9fJ1F58zMTI4dP44URpSZOHEigwcPRhAERFEkKyuLEydPhhUM+/btyyWXXIJarUalUpGVlUVxcXHYggBpaWnMmTMHvV6PRqMhPT2d6urqIH+/gNMPxOj1LFywgKSkJHQ6HUajEavVGhQp18EVBIF58+aRnZ1NTEwM8fHxqNRqysrKunYBCAjUAwYMIDY2lri4OBKTkjh58mQoURAYMWIEEyZMID4+nvj4eDIyMzl69GhohJggkJOTwxVXXEFiYiIGg4Gcnj05dvQoXp8vpOpjgsHAwkWLSE1NJSEhgdy8PE6dPBkQybtw1SoVi5YuVarV5uTkUFlejsliCSs6zZ03j759+2I0GsnIyMBqMlEXoYLpxRMnMnrMGJKTk0lNTUWtUlFaXh5Maj/GgF69uGzWLFJSUkhNTcWYmsqJrpGkHRUw4+NZuGIF6enppKWlkV1QwJGDBxVz9c7cGEFg2U03kZOTQ3p6OvkDBnB47158neagA6Iss2jFcvr060dGRgaFAwdyfNcuXJ3a64xZP/kJwy+8kMysLAr69aP8228xyXLA40ilQlapAhUSVSpG5udzyYIFZBcU0LNvX1pOFVHt8+FTq/Fptfg0msCPTkcPvZ6r1q0jp39/eg4YgK+5idMOBx6dLvATE4NHp8Ot1xMjiiy/+27yR4wgd9gwYtVqDjY344qNxRUbi7PTj1+rZcmttzJgyhTyR48mPSeHveXl2BMMOOLjlR97fDyO2FhmXT6LUUuX0uvCC8kfeQG7jh6lzWjElmDAlpCAtdPPmL59ueSOn9P74ovpN3Uq3+zZQ0NaGpaEBCxJSVgSE/FpNCAIDJFkhs6Zzb333lt3zz33RFbIOyEqOP3IEKdS0Tcuhr5xMWEXS4IgUKjX8V5jG+F06Z/2TOeZQfnc0DONNblpvFrb8r0peH8vbH6J/WY7u9psnLR378dwyubitdpmXqtt4fnqJhrCeEh14KjVgUoQOGV38Y3JyqMVkf0mDprtXJaaiF8OLFRWnygP608F8HWblZU9UpXF+n2ltWxtCv3WTQY+b7EwP8NIYvuic1uTiV8Xh4ZvS8AnzRampSYoi84Kp5s5h0uw+YPPiB/4tMXCqMRYReRwSxKzDwUEss7wtlcszNNrg6Kt1hZV8FZ98Ln2yjK7TbYQ8W1DdRO/O1uLs9MDoVeW+dZsp83rZ2qniK/dbVauPlYadF68sswhi4NTNidz05OUh80Kp5tZh0o42anPPhmO2Zx83WZlQUayIlDZ/X6uOlzKpy0WfO0PTX7glN3FR01mrswwKte1LMvcXFTBCzXNytzJQInDzZaGNmakJQaJTuvPNfCbkmqaOomt5U4P79S3MjYxPkgQ2d5kZtmxMspd5x+wql1etjS0ka/XMrDTHJ+yObniYAlH2qPCABo8PrY2mRGAC43n57jJ4+WKQyV81mLFIUnIQIvXx+ctFiqcHmakJirz5pEklh4rY2NdKyafHz9g8vnZbbLxrcnG3HSjMm+yLPOr4oAQU+/x4pNlrO3vt+1NZuakG4PuBy9UN/HTU5Wcc3lwSzJ2v8RRq5MtDa1cmpIYFFn3ZYuF+UdKOWl34ZQkHJJEkd3FG3WtjEqICxJEiu0uZh0qZq/JjtUv4ZQkSp1u3q5vJVsXLMq2eHzMPlzC1iYzbT4/TkmiyuXhg0YTbklmUicR2SNJrDhWzgs1zTR4fDgkiXqPl09aLJyxu7g87bywLssyd5XU8IfSOqpcHux+iWavj6/brHzVamVuelKQYPBSTTM3nKzgrCMQqdfm8/Ot2c4HTW3MSksKihz8ssXCvCOlHLM5sfgkzD4/R6wO3qprZaLRECRcnmmfi91tNtp8fiy+jnlroV/7PboDzR4fsw+V8EGTmWavD6s/MG/vNLQRIwqM6fQ+dXeaizq3F6s/MG9bm8zUub1MS0kIMtT9VXE195fVUenyYPUH5u3zFgvfme3MSU8KEoefq2riplMVlDjcmH1+mrw+dpsCXyhckZ4UJJ580mxmQftcmHx+Wr1+DlgcbG5oY3KyIUiMPGVzMutgCbtNNlq9fkw+P8dtTjbVtjDcEBsUzdro9nLFobN82GTm/7V35nFyVWXe/55au7t63/fse4gQIBAEZEBRAUFQcEEdBFFxHJ3xVWccfRV13EYdd9kVRVRGRkUEWZVFCQESlixkJQmdpJPuTu9bVVfVff+4t6trOedUxWnHyuvz/Xz6k15+OXXuee75Peeee+65vbE4Q/EEO8Yn+fmhfhpCAY5PuwkwkUjy9hde4odeWwzFE+ydiPHrnkFG4knOqq3IaIt/3tbFl/cc4uXJGEPxBAeiU9zfN8ym0Qne0FCdsc/gd/cd5oMvvsyu8SiD8QSHY3EeHRjh4f5hLmqszpiAu6d3kLc8v5tNo5MMxBP0Tbnn0K8OD/Da+ipq0nzoueFxLty4kyeGxjis/BxWfp4bneBn3f2sqY7QkTapdnAyxgUbd/K7/hG6AiUcCJexNZbgZ91H6Mjy+rGEe0Pk9p4hXiopZ295Ddv9YX7ZM0jCcVKPvoO7IvQftu7jGwf62V5Wxa6aRrZHqrl/aIKdY1HOb6jKmHT6+p5DfHxvL5vLa9nW0MaLNU08NqV4bGA0pz/96vAAV+7o5tnKejY3z2FzUyfrgxHu6R/hvKz+9PTQGJdtO8C6inqeb1/Ac+0L2FDZwF1DE5xeU05LeKYt9k1EuWzbAR6O1LOxczEbOpfwbGMHv43BwrKSjMfwh+MJLt9+gLtKa3mufRGb2hbwdOdiHgyUU+L3Z+S9eNLhwy8d4tZgFZvb5rOjqYOn5i7j0epmDifdNwOnT1x+bf8RvkaEbS1z2FvXwvMdC3m0dSFbkn4uaqzJmNC+s2+YT0TDbGvu5GB1Hdua5/DIgpU8HYhwQUN1xg2UJ4fH+dCYjy2NHfSVV/NSQyt/XHgcj1U2ck59FY1p/WnPZIx/GIaN9W0MlZZzoKaBp+Yt46GWBZxQW8mCspm2GIwn+NBggj/VtjAWLqWnooZNbfN5YP5xtNZUZ9ykiiWTfGowzu8qm5gIhhiIVLKrsY2Hl5xAoraBV2d5y3eG49xeVk80EGI0XEpXbROPLT6eg82dXNhUnTGhfddEkm/5q4kGgkwGQvRU1rB+3nI2zV3KG5uaAsZaAAAgAElEQVTrMvLT0zGHf6eCsUCQpM9Hf1kFL3QsZN2iV3Bee1NGftqbgE8nIwz4gyjHYTRcyvamTh5degKnz+nIyE+DDlybiHDAFyCQSBANhthX28yjS45n8cJFrErLT1Mo/sMp40XHTzg+RdLn43BlLevmr6R0+XGclbUC/hYV4U9xHyVTMXyOw2BZORs7F3Nk1Ylc2Jz5gp37/BF+GVOUxaIEE3HGQyVsaZ3HjuPW8KY5LRne8nywjJujfkpik5TFokz5A+xqaGfDyjVctGR+xo2O/aFSvhUL4I9OUjE5jqNgf00j65eu5u9WrcyYsB8Nhfl6PMzU5CRVE6P4gL5IJevnr+C4k9ewIq0tEsEg3yHC4NgY1ROj+ByHsVAJz3UspPrUMzijfuaRa5/fz4+D1bw0Mkr1+Ah+x2HK52db8xxGTjmTN7Q3pbxFKcVDkTrWD4xQMz5CwBsT7a1rZu/qV3Lp4nkZ3rK1sp67+0epHh8mlEiggMMVNWw6bg2XHL8yw1t6qur48eAEVSNDhBPumG+4pIxnFx7HuaedmuEtE+WV3DCWpHR4kNIpd8w3GQjyQsciTjzrbJak+awqi3DDVAAG+olMuZOcCeVjR1MHTWe/hrX1NSltIBTmJ/4II319VES91T1K8XJNI9HTX835c1pT/cnv9/NgWR27e3qonhhzJ52Voi9SSdeaM7lsxZKUtyil2FLXzLqDPdSOjaQmqEdDJWxbeTKXnnJShrf0NLby2+4+6kYGmG7NKZ+fLfOWc/6rz6YxbdwSrW/kpz1D1A704fOWxjhKsaN5DmvOfwOLymfOC39NLT8ailLVcwh/aodzRXdFDW0Xvok1DbUpbbi8nJ/GQ5Ts35ehHQuEiL32Qs6b15ExQX1/pJapXTsJpO0+nnDg4Mmv4tKTj095i8/nY1PLHHq2bCGYvlN5MsmBuUt40wXnZXhL74KlbN/4HMH0q6Kkw0B5DedecQX1aW2RWL6SZ57aQDARz5hYjCfg+Pd+gHkVM95ZsnQZjz/zHMHoBEyfs0oRmIzSctnlnNA2M9lbMW8+j72wjcDIIE7amCo0OUlozemcc8rMgqLypibW7z2I6jlEIjDT1wOxGL76Fs5/62UpbwmVl7NjeIKp3buYSrvp4EskKIsmeN1HP0KJ93nK7+dIOMLIcxuJlqbdvFKK2r5+zvj4x6ktjxzVhJM8UidoeXJwlC+/1M2TQ+5diI6SENd0NPDutvqMwdVX93Tz9b36mXEfcMfxCwgoxeHoFJ/dfYDuqP0Z8mON6oCf6qAfP4o9E1HtJN0080tDLImU4lfu6pgjmhVZ0yyJhDm7thK/UvxpYNS6Omx+aYgr2xtQwAsjE9xxyLwxbGMwwOcWteJTPg5FY3x6l2bpr0eZT3H9irmU+X1Ekw7v3bI39QhiNgq49bh5NIeDKOAft77M9nHzpOF3l3VyQmUZCsWXXjrI3ZqJumk+vaCFi5tqUChuP3iEr+49ZNS+p62e/zPPfURl/eAoV2zea9SeW1fJ9SvmAnA4GuOV67cZ47ekLMzDJ7t7SMSTDiev20KPIX6Vfh/PvnJF6uL7kmd38cSgfmNfBaw7dVlqwvATO/bzQ8tjpz9bNT/1KOetB/q0k5bT/PuiNt7T7q5ceGJglEue22XUXtFWz5cXtwNwKDrFSeu2EDekhtOry7nzhIWA+6juSeu2GCd828JB1p+6PDVpcfGzO1k3qH/VeUgpNp62IrWi5V+2d/Gjg+a3hv3uxMWc4E0u3LK/l0/u1O/fAu759mbvRQeP949w6fO5b3Cb5sNzmvjEfDf575+Mcco68wsXzm+o4paV7p4l0WSSE5/YSp9hdejishIeXbMk5Z8XbNjBM5pHhgEq/D6ePW1FalD4z9te5mfd+n6tgMfWLGWRd0F93cs9fHa3uV//6Lh5vNa7AHjoyDDveEF/lxfgk/Nb+Mc57iMye8ajvHL9i8Y+cllzDd9e5q4AGE8kOfGJLQwYVsGurizj3hPdRywcx+HcZ3awaVS/nL4uGGDD2uWpi6wPbN3HLw/r32jnB55cuzw1KfONvYf4yh6zX/ziFQs4w5u4/G3PIO/Zsteo/cKiNq7y+tO2sQnOemq7Ufvutnq+5PWn4XiC1U9syblhME16f3IchzOf2sZOw1tiW8NBnkrrT1du2sO9fXrvDCnFhtOWpyb2vvRSN9/ap8/VAPesXpSaXLjzUD8ftLzZ9htLO3hbSx3g3tB5reUx/H/obOT/LnD3ADoSi3Piui3Gm1Tn1lXy41XuKpmE47D2yRd5eVL/mNX80jB/PGVp6uL0bc/v5g+GVdFlPh/PnrY8dcPn0zsPcON+8/5Cvz95SeqmwW0H+/jYdrPP3rBiDhc1uheRTw6O8sZnzT770bnNfNTLT4eiU5y8bmvGSut0Lmqs5gYvP8WSSU5et9Xos8sjJan8BPDGjTtTY7dsqgJ+Np62PJWfPra9i9sMPquAP52yjPnehO+NXT3WMcNtx83jNZ63/OHIMG+zeMunF7Tygc5GwJ0sXPuk2Vve1lLLN5Z2Au4k8mqLt5xUWcZv07zltRt28MKI3lvqgwE2nLY8tYr6g1v3cafBWwIK1p+6PLXlwbf2HuZLe8z7TqXvX3pv7yBXWsYiX1rczrvb6gH3RsRZT20zvgHvyrZ6vuh5y0g8wQkWbzmjppxfHD/jLa96ajs7psdlzsxFPeTm6qs27+Ge3pk31SknSdLnnjPZ3vLll7r5puctvmSSQDJBzB9IlX3v6kWs9rzlvw/18w+et/iSScLxGNFAiKQXg28u7eCtGm/xJZOUxSaJBYLEAu7npntL/1Sc1U+43qKcJBWTE8R9fsbDbl40eYtyHCon3L4yVBoB76b74zpvcRyqJkYJJJMMlpaT8PvN3uI4VE2MEY7HGCotJxp0zxuTt1RMjBGJTjBaUsZoiTuuMXlLZHKcqokxJoMh+iOVoJTRW0pjk9SNDjPl99NTWYOjfDnesmbdixyKTRGaitE0MoCD4lBVLXF/gBXlJTx0Uq63BBJxWoaO4Esm6amsYSJUYvQWXzJJ62AfocQURyJVDJWVG71FOQ4tg32UxSYZLo3QU1EDSum9xXFoGu6namKM8VCYAzUNOMpn9Ja6kUEaRgaJBoN01TYR9weM3lI1NsKCnv3EfX52tnQyESoxekvFxBir9u7An0ywtWMBfZU1Rm8pmxxn7fYXKI1F2dnSyfa2uQR8SustpZMTnLXpaSonx+murufxFSfg+PxabymZnODcZ9dROzrMcGmEB084lZFIhdZbSiYneN0zf6R56AjRQIhHV5zI7vY5Wm+Jj45x/tOP0XHkMEmfj+fmLOKPq07mlXWV/OL4hSilZA8nmXCaHQan4kSTDg2hgPaRgmgyyZWb9vJwf+b+Bn7gG8s6uax5ZgY73wXyFxa1cVp1OYPxBDd09XBfn/m102U+HwvKwowmEhyOuisaBKFYUGlf+R489SsIen0r32rBgIKwz4fCvai3nfU+SD26NJ5IGi9spqkJ+FEKJhNO3v5UG/TjQzHlOAzlebS2JuAn5FMkHIyTMdNUBnypwcqh6JT1tdMRv49q7/h6YnHr8YWVoiHsDgj7Ywnr8fkgdad3KJ7Ie3wdJUF8KMYTyYzVcTpawkHCPsVU0uFA1L7Za0MwQEXATxKHfRMxa1tUB/zUeQPelydj1rYo8/loLXEH7N3RKeMkMkBQkVoBcCQWN17kgXuuzy8N41PuJItt1SnAnJIQIZ8imnSMkwrTtISDlPt9JB3YbXlJBbgTVNNt8dLEpHHiFNyJvek72fujMcYtbRH2KeaUuG3RG5vK2xbTj30OTsXztsX80jBBn2IikczbFu0lQcr9fuKOwy7DxNQ0jaFAahXnzrFJqxdVBfypFXj7JqJMWLyo1KdS58Wh6BSDlrbwQ2oydKCAtlhYFiagFGPeyjwbnSUhIn4fUwW0RXMoSE3QvXe+Y2zS6p01AT/NXlvsmYhafTni99Hp+cVBbxWdiYCCRd7qoiNTcXrytMXisjB+b/uC/Xn8Ym5piFKfj1jSydtHWsJBqgNuW2wfm7R6S13Qn1o9tXs8mtoDVEe535ea7N3vrZg0EVKKBWXT/SluzQ0KWBwpwYfrLfm8c35piLDPx2QyyZ4J+znUFg5SGfCTdGDHuL0tGkIBGqb703jU6rOVAR/tnre8PBkzTgoBlPgU86f7U2xKuz3END5wH40GBuIJuvO0xcKyMCHl5qe9BfSncr+PuOOwI09/agrN+Oz28UkSloarDvhp9frT3omYNf+W+nzM81aTd0ftPuuH1MrF/qm48WmEaRZ73jJagM/OSfMW0w2AaZpDQWqDM/3J5i21ad7yUgHeMsfrTwcK8JbFnrf0FeAtS8pK8CtSK6BtzJv2lgJ8ttXzliSFeEuAJu8G4+6JKFFLW1SkeUtXAd6y0POWngK8ZYnnLUMFeMuC0jBhn2IymeSlPN7S7nlLAjfn/P/mLetPXcbcshKZcJIJp/89Eo7DfX1D/OrwAEPxBEsjJbyztT7jkRBwV0NcuXkPD2reUndxYzXfS3tL3Z7xKK96aptxcPPttMms54bHed0G8x3WxWVhvri4ncmkw86xCT6723xXSuHeqQ8oxcBUwnj3eJrWcJAyvzvIy5fABEEQBEEQBEEQBOFY5c7jF3BGbWXBE076V1wJwlHgV4rzG6o5v6Haqgv6FD9cOY87DvXz8+5+DkZjdJSEeEdrHZc01WSsoJpXFubGFXO5ZuvenDutH+xs5NKmmeegj68s49LmGn5xKHcpdFAp/mNJB6d6ezK8uq6SR/pHeXRAv/T+0uYavrVsZmPItz63m0cM2vpggCdOWZZ61ON9W/ZyV0/uBpLT3HfiIlaUlxF3HK7dtZ8fHTQ/+vbPcxp5Y1MtScfhZ9391uX/Z9aU84HORpIOPDE4wndfNmubQ0H+bX4zDoq9E1G+YXnEIqDgY3ObKfH7GJ5K8I19h613bi5vqaUpHCSRhBv391jvkL+qppylkVIc4JeH++mzzLovLSvh5Gp3OfbjAyPstdxVaAgGUhvEbxmd4EXLXmBhn+I1dZUoFAcmY2zMs6n9WbUVhH2K4XjC+FjYNCdUlFEddFch/HFg1HpnY0FpmNaSII4DTw2NWe8gN4UCzPPuVmwdc/cJMlHh97GsvBTHgX2TUesdr4CCVeVloNzVG12T9jsbKyIlBHyK0Xgy7930+aVhIn737tj2PHuztYa9VQgObCvgTmFTgasQynw+OktDOED3ZIxhyx2hgIL5pTN3TfOtyppfGsav3D3o8t0Rag8HKSlwgrohFKDS765w2ptnhVNV2gqnrjwrnEp9Ppq91V75VocGFKm7iv1T+Vd7dZaE8CkYjSfztltLOEhIKWJJh+6Yvd3qgwHK/D6SjpN3pUdVwJ/a2P7AZMy6qqfM50s9wtkTm7KeQ0GlUnemB6bi1ruKCmgrCaJQjMYT1rv04LZFQCmiyWTeO9P1wQAlfkXScVfU2KgK+KkI+MDT2vpTmc9HbchbMRiNW30opBSN3jk0MJWwrpJTkFrdMJpI5j2HmkNB/Mpd7Wl7eQm4bRH2KRKOk3d1Q1XAT7mXqw/O4urJkFI0eOdQ/1QiY4/DbHyQWkU2Ek9YfQhP6wMmkknrnWlw/SKkFHHHybuKrCbgp8zv7sCS7xwq9/tS/elwbMq6YjDsU9R7PnRkKm7tT35I9aeheMLan2CmLca9ffRsNIYCBD1vybfqdLotkpDXvyv8vtQ+QIeiU1ZvKfGplCf3xeJELedQQEGTt4psMG7vT+D2J0Vh/akpFPC8xcnrybVBP6U+HwnHXQ1ho9Lvo8JbJdedpz+V+VRqn7remN1bgkrRGJrxFlt+UrjnhcJdqTNcYFtMJpPWLS3AXX1T4iusP017SyFt8ed7S9w6pk73luF4wroCCGZ8thBvmfbZKcfJm5+qA34iBfYn8ZYZ0r2lO0+uLhZvaQ3nvqTFhqxwEoqavlicXxzqZ9f4JPWhIJc01WRs/jlNPOnw7ZcPc8v+vtQg9ZSqCJ+c35KxoS64FwtXbd6bs6/OefVVfHf5nIwN5A5Mxrjk2V3sy7o4jPh93Hbc/IwNn/dPxnjDhp3ai6f3dzRw7cKZN9Qcjk7xumd2aLUry0v5zepFqXoMTcV5zTM7tBeo5X4f9564OLWabCrp8NpntrPVcGGf/kw4wGXP7eKxAf3+Qtl1tu0js7Y6wi/d53kB+94pjaEA605dlnp06peHB/jA1n1abVApfn/yktQjGc8MjXHBRvMrTn+4ci6v9yY+D07GWLv+ReMy3Y/NbU7t9zSZSLLmya3GZHpBQxU3e3v1OI7D+Rt3stGw/86isjCPrpl5/t+2L0TE72PD2uWpjT2/ufcQX7bsOfPQSYtTbzC8u2eQqy17zly3fA4XexOzW0cnOPtp854z/zSniX/19i0amIpz4hNbjQO98+qr+MFxblskHIdXrn/ROAmY3Ra2fVYifh8b1/7l91lZNzjKxZZ9Vv7P3CY+Ns9ti4OTMdY8udU4EHpDQzU3rZwLuI8Xn7RuK72Gc2hppIQ/pO2z8oYNO3l6WD9x+T/Zw+n7L/fwOcseTul95MG+Id65aY9R+2/zW/iQt4fTS94eTqYRw5ubavjucneyfiyRYPUTW40DllUVpdx/4mKUUjiOwzlPmz2rJuBn42krUi9nsE3s+3D3RJt+3Mu2xyDAT1fN52xvT7RfHx7g/QYfAvjswlbe1+HuC7FldIJzLP3pHS11fG2p+xa1wak4Jzyx1TgRcUpVhLtWLwLczbpfuf5F42NAjaEAz6xdntoo9x0vvMRDmhXD4A40n167PPXI4Gd3HeC6LnN/+tUJC1nr5cqfdh/hI9u6jNr/WNzOu7x9IfJ58tXt9Xx+kbsvRG9sihOf2Gq84DyrpoKfH78AcHPZmie3GgfqHSUhnjx1WWpDVNteeWGfYuPaFanNpPPtD3ffiYtTm8/fvL+XT1n2h/vOsk4u9VZbP9Y/wmWW/eE+1NnIv3l7znRNxjjVsj/c6+ur+KHns5OJJKvXbTFeGC4qC/PYmqUpb3n9MzuM+z5G/D6eO21F6nHrD724j//S3Kyb5pE1S1jqvfHwO/sO84WXzCvEb14xlwsaXW+5v2+Iv7d4y7/Ma+af57r5d9f4JGesN+9bdElTDd+f9pZ4guOf2GK8oD6uvJQHTprxlr97ejvbDN5S7XnL9Fjr6s17ubvX7C1PpO23+B97uvlPi7fcvmo+53je8qvDA1xj8ZZrF7Tyfm/PmU0j47zGsifa5S21fN3bc2ZgKs4JT5j3RFtTFeE3ad5ymiVXZ3vL5c+/lLNVxjTZ3nLtrgNcb/GWXx6/MDVW/unBI3xku9lbvrK4nb/3vOXpoTHeUKC39ESnOGnd7HhLe4m7n1VB3uLtPVmot6TvPfmX9JZT1m01Tlr8Jb3l2dNWpCZOisVbTl+/zagtBm/J3sv1Ky91WxcDFJO3HM0eTvKWOqGoKfP7OLkqwmvrqzijpkL7imwAn1KsrS7nve0NvKWllg/PaeKq9obUJmzplPp9XNZcw1m1lXSWhHhVbQWfWdDK1R2NOa9Erwz4eUtLLfWhALGkQ7M36fWtZZ2sqCjN0b6xqYZY0mHfZJSEA8dVlPGpBS1c09GYsdl6ecDPeQ1VdEen2D0excGdtX5Lcy3fWz4n47XhJX4f5zVUsXN8MmOwsKq8lBtXzmNlWj38SvH6+io2j05kTFBVB/z8+6L2jD21AF5TV8mW0YmMchXwztY6PruwLePNKWfWVLBrfDLnefaTKyPcsnIekbQ6r6mKcCg6lbMRcEsoyO2r5tNeMvNmmKWREuIOrM/a5DTsU3xv+RzWpk3qtZaEqAsF+MOR4ZzB6UfmNnFF28zrvSsCfpZFSrm3dyhnUH9+QxWfX9SeOr6AT7GmKsI9vUM5E1Qry0u5wds8Hdw3gLyyppz7+oZyVhg1hgLctmp+xhu41laX82j/SM5kVtinuHHF3Iy3vayuLOP5kXHtBefnFramJgrAfVS0Nxbnec2GqJe31PKhOU2pc64hFCTi92lX9q2tjvDVJR2pc7/U7+6Pdm/fYM6AZV5piJtWzktNhPiUYnVlGXf3DObcZakK+PnhcfMy3vZySnU59/YO5iT0oHJjnf6WupOqIjw2MKK9u/jJ+S2pwQfAikgp28Zyz01wVy1+OK0tOkpCjCeS2smeU6oifCWtLSoC7v4luseA55SEuGHlTF8NKOVOFvcO5uxnURXw84Pj5tGc1hYnVkW4u2cwZyIioOD7y+dknBcnVUZ48MiQdlD4qfktGefFqopSNgyPaSeo39pcywc7Z7xofmmYw9E4L2g27F5bHeGLi9pTbVETDBjPoXmlIa5fPjflASGfj4VlYe7pzT2Hqr22mH7zjVKK4yvL+M3hgZxzKKgU162Yw9K0twCdVFXGvb1D2rvZn13Yyjl1M29EOr6yjCcGRrWrON7VWsfV7Q2ptlgcKeGl8ah2AHlmTTmfW9SWevNNYyiIg6Nd6bioLJxx46LE76OjJMh9fUM5nlUb9HPLynmpvKaUYlV5KXf1DObc+Q4pxY0r52a8YezEyjJ+2zOovYP7pcXtGW+eW10Z4RGND4F74TY9gQSwLFLKlrEJdmv606vrKvnUgtaUd7aWhBhLJLQb4C+LlPCtZZ2pVcARv9ufHtD0p8ZQgJtXzkuthPArxYqIvj+V+BQ3r5iX8Yax1ZVl/EbTnxTwtSUdGTeeTqyK8GDfsHYF1T92NnJZy0yeXFleyrPD49r9b85vqOLj81pSE+qdJSH6DJ68qqKUry/tTG0iWxXwUxX083vNBHxrOMiNK+em7v4HfIrFZSX8tncopz+V+33cvHJexlhndWUZv/be5pmOH/jWsk5ekfYmxxMrI9zXN6Tdh+vj85p5Q9oNqlUVZawfHNWuMnxTU6bPLigN0zUZy3j77DQnV0b4cprP1norKB7X3PzqLAlx/Yq5qZwT8vmYVxrm3t7BnP5UFfBz88q5KZ9VSvGKijJ+0zOQMxHh+uxclqV5y4mVZdyjyU/gbnL+mrS3rR1fUcbjA6PalUCXt9Tyvo4Zb1kSKWHneFS7wvf06nI+v3jGW5rCQRKOo930fWFZmO8tnxmLlPp9tJWEuF/jLTUBP7ccN5f60IzPHldeym8M3nLDirksTLuZe1KVm9d1KyK+sLidM7K85ff9w9qbLe9pr09NIIHrLZtHJ7Qro8+ureDTC2bGnW0lIUYSCTZovGVppIRvp3tLwE99KKDN1Q3BXG9ZbvGWm1bMLdhbvro001tWW3L1BzsbeUuWt2wcHs+5oQ3ujb1/mZ/pLabxns5bKgN+7c291nCQG1bMSd3YC/gUiwzeEtF4ywmVZdx1FN7yu1nwlks03vLyZIytBm/50pL21MRpbTBAyOIt162YGcMdrbesqnD7k85bvrd8Tsbbqo/GW044Cm9ZHClhl8FbXlldzr8X6C0LSt1xy/RCgEK9Rd5SJyuchGOIoak4/VMJGsOBjNeK63h5Isq+iRgN4QBLykoyJrGy2TY2weaRCSoCfs6oqchYuZXN5pFx980TSnF2bQWdack2m00j4/z+yAhTjsNp1eWsrY4Y67F1dCI1aFlVUcoFDZmvC09nx9gkdx7qp28qzvzSMG9pqc2YuEln70SUn3X38/JElKZwkLc012YMHNM5OBnj9u4jbB2dpCrg541N1byqpkJb575YnJ92H+GpoTHCPsVr66u4qLE6lcjTGY4nuKO7nz/0u5Nfp9dU8PaW2oxXnE8zmUjyy8MD/LZ3kPFEkhMqy7iirT5jUDNNwnG4u2eQ/z48QP9UnEVlJfx9W33qrlg6juPw+/4Rftp9hP2TMVrDId7eUpvxeup0nh4a49YDfWwfm6Qm6Ofiphre3FST8WrhabaNTXDL/j42Do9R4vNxXkM172ipTQ1U0tk/GeOW/b2ppP6q2gqubKvXTvj2T8W59UAf9/UOMZFMcnJVhPe0N2Qk52nGE0luP3iEX/UMMDSVYGl5Ce9uq8+4kE5vtzsPDfDzQ0fojk7RURLi8pY6LmysznnhgeM4PHhkmFsP9LF7PEpdKMCbm2p4e0ud9vx8emiMm/b38tzwOBG/jwsaqrmyvV4b6+1jk9zU1csfB0fwozi7roKr2xu0fao7GuPGrl4ePDJMLOmwpirC+zoaMibephmOJ/jh/j5+3TPAcDzByopSrmpr4Mza3LaIJZP85OAR7jjUT28sTmdJiHe11fNGQ1vc1TPIjw8eYc9ElIZQgMuaa3lna532vH+sf4Sb9veyeXSCcr+PixpruMrQFi+MjHNDVy9PDo4SUIrX1Ffyvo7G1GN66eybiHJDVy8PHRl23yJUXc77OxpSK/rSORKLc/P+Xu7uHWQ07nrL1e0NqTfOpTORSHLrgT7uPOy2xfzSMH/vtUV2H0k4Dr841M9PDh7h5ckYzaEgb2lx20LXRx7oG+IH+/vYOjZBVcB97f3V7fXaPrJhaIzrunp4ZmickE/xuvoq3tehvymya3yS617u4ZH+EZLA6TXlfKCjUetxPdEpbtjfyz1p3vLe9gZeqekjo/EEPzjQl/KWhWVhrmir58KG3LaIJx1+2n2En3b3u95SEuTtLXVc3lKXc2PGcRzu6R3ih2necklTDe9pb8i4eTLNusFRbuzqZUPKW9y2aNEs0X9xdILrunp4fGAUhXvj45rORu1K54OTMa7v6uV3fUPuisNKtz+dkrXKGdz+dFNXL7/qGWBwyt178sr2el5fX5XTFrFkktsOHuHn3f10R6do97YCeGtzbeqtXult8eueQX50oI+d41HqPW+5sr1em98f6x/hxnRvaazm/R0N2ty3aWSc67t6+ePACH6lOLu2kms6GzImIad5eSLK9V293N83xJTjcEpVOdd0NKTeFpZO/7AiznsAABz8SURBVFScm7p6uatn0PWW8lKubK/n3LSLoGkmE0l+dLCPO7r7ORyLM7fUbYu3NNfmeEvScbjz8AC3HTjCSxNRGj1vuaKtPrViMZ2Hjwxz8/5eNo1MUBHwef2pIbV6JJ1nh8e5vquHdYOjBJXiNfVVXNPRoM2pe8ajXNfVw8NHhol745b3dzbyCo239MamuLEr11um3w6bzngiyQ/293Ln4QF6Y3EWlIV5V2sdb2qq0XrLHd393DbtLeEAb22u411tep+9r3eIWw70Zoxb3tvekFoNnc7TQ2Nc39UzM26pq+L9nXqf3Tk2yfe7PG9xHM7w+tMKjbccjk5xfVcP9/QOpbzlfR0N2vw7Gk9wy37XZ/unEizyvOUijc9Oed5ye/cRDkxO0RoO8vbWOt5h8Jbf9g7xgwO97BiLUuuNW642eMsTA6PcsL+HjcPjKW95v8Vbvt/Vw+P9oyg17S0NqVU36RyYjHF9V4/nLU7KW07VeMvQVJyb9vfxq8MDDMZnvOU8g7f8OMNbgryjtY63NdcZveXWA33sGo9SFwxwaXMNV7bVZ9zwneZoveW6rl7+5HnL39VW8IHORq237PO85QHPW9ZURbimozH1ltN0pr3l1z0DjMSTrCgv5SqLt9x6oI87DvXTE4szpzTEOy3e8otDA9x2sI89E7HUuOXdBm956MgwN3e545aKgI8LG2t4r8FbNg6Pcb03bgkqxavrKvlAZ2Neb5m+JrpmFr3lF4cH6EuNW4rPW+QtdTLhJAiCIAiCIAiCIAiCMKsczYSTecmDIAiCIAiCIAiCIAiCIPwZyISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKvIhJMgCIIgCIIgCIIgCIIwq8iEkyAIgiAIgiAIgiAIgjCryISTIAiCIAiCIAiCIAiCMKsox3H+2nWYdZRSvcA+zZ/qgb4Ci/lLaYulHseatljqUQzaYqlHMWiLpR7FoC2Wehxr2mKpRzFoi6UexaAtlnoUg7ZY6nGsaYulHsWgLZZ6FIO2WOpRDNpiqcexpi2WehSDtljqUQza/+16zHEcp6Gg/+04zt/MF/DMX1tbLPU41rTFUo9i0BZLPYpBWyz1KAZtsdTjWNMWSz2KQVss9SgGbbHUoxi0xVKPY01bLPUoBm2x1KMYtMVSj2LQFks9jjVtsdSjGLTFUo9i0BZTPbK/5JE6QRAEQRAEQRAEQRAEYVaRCSdBEARBEARBEARBEARhVvlbm3C6sQi0xVKPY01bLPUoBm2x1KMYtMVSj2LQFks9jjVtsdSjGLTFUo9i0BZLPYpBWyz1ONa0xVKPYtAWSz2KQVss9SgGbbHU41jTFks9ikFbLPUoBm0x1SOD/y83DRcEQRAEQRAEQRAEQRD+evytrXASBEEQBEEQBEEQBEEQ/sLIhJMgCIIgCIIgCIIgCIIwu/xPXnF3LH0BrwO2A7uAf7XofgD0AJsLKLMD+APwIrAF+LBFWwI8BTzvaT9bQPl+4Fngt3l0e4FNwHMU8NpCoBq4E9jm1X2tQbfEK3P6axj4J0u5/+wd22bgZ0CJRfthT7clu0xdDIBa4EFgp/dvTR79pV7ZSeCkPNqvem3xAvAroNqi/bynew54AGjNd94AHwUcoN5S7rXAgbS2Ps9WLvCP3vm8BfiPPMd3R1q5e4HnLNrjgSenzyVgjUX7CmCdd+7dDVTa+oUuhhZtTvws2pz4WbQ58TNpdfGzlGuKn7Hs7Bhays6Jn0WbEz+LNid+GHwKmAes92J3BxCyaD+I67Opc97mgcDtXjtsxj3PghbtLd7vXsD1sHKTNu1zvwOM5qnDrcCetHY+3qJVwBeAHV6bfsiifTytzIPAr/PU4xxgo6f/I7DQoj3b024GfgQETLlDFz+LVhs/gzYndhZtTuzy5br02FnKzYldHn1O/CxabfwM2pzYWbS22O0lK59jyH8GrSn36bSm3KfTmnJfjtaS+3TlXoveO7XlYs59urJNuU+nNeU+ndaU+3LGVpbY6bSm2Om02thZ9Kb4GceDmvjpyjXFT1uuLn6Gck2x02lNsdNpTbHTjnd18bNodeMWk1Y3bjFpdeMW6/iczHGLqVxT7IxlZ8fPUrZu3GLS6sYtJq0pfjnXHxjynkFrGrfotLa8p9Nrc59Oa8p9hnJvRZP7DFpt3jNoTeMWndaW93R6be5Dc02I2Tt1WpN36rSmvKfTmnzTdg2b7Zu6cq9F0/dsZaP3Tl3ZJu/UaU3eqdNq+16hXwULj+Uv3MHebmA+7oXS88Byg/ZMYDWFTTi1AKu97ytwO7KpXMWMyQRxTfDUPOV/BPgphU041eerb5r+R8B7vO9DpA1U8rThIWCO4e9tuMZX6v38X8AVBu1K70QuAwLAQ8AiWwxwE9u/et//K/CVPPpluAnrETLNR6c9lxnT+8p02QZtZdr3HwKut503uBf79wP7mDEfXbnXAh8t5HwE/s5rs7D3c2Oh5y/wdeDTlrIfAF7vfX8e8IhF+zTwKu/7K4HP2/qFLoYWbU78LNqc+Fm0OfEzaXXxs5Rrip9JnxNDWz2y42cpNyd+Fm1O/DD4FG5/fqv3++uBayzaE4C5ZPmSRX+e9zeFOyixlZ0ev//EPY+M3gqcBNzGzISTqdxbgTdntbVJ+27gx4AvLXZ5/R34b+BdecreASzzfv8Br1467WlAF7DY+/3ngKvSPisjd+jiZ9Fq42fQ5sTOos2JnUmri52l3JzY5dHnxM9WD138DOXmxE6nxV1Zboudrt21+c+gNeU+ndaU+3RaU+7L0eq801Lutei9U6e15T5tPbK901K2KffptKbclzO2ssROpzXFTqfVxs6iN8VPOx40xE9Xril+Oq02fqY6GGKnK9cUO51WG7usz0uNd03xM2i18TNojfHTaLWx02lNsTOUq42dRW/sf7p66OJnKFcbP4NWN27RXn+gH7eYtDl5z6LV5j2LXjduMV4zkTtuMZV7K7njFpNWN27Je92Gl/cs5WrznkF/JZrch+GaEP01g0mru2YwaXXXDCat7prBeA1L7jWDqdxr0fumSa+7ZrBeS6f3PUu5umsGkzavd9q+/lYeqVsD7HIc5yXHcWLAz4GLdELHcR4D+gsp1HGcbsdxNnrfj+DOGrcZtI7jOKPej0HvyzGVrZRqB84Hbi6kLoWilKrEnTy4xatXzHGcwQL+6znAbsdx9lk0AaBUKRXAPVEPGnTLgCcdxxl3HCcOPApcPP1HQwwuwh084P37RpvecZwXHcfZnv3BBu0DXj3Aneltt2iH036M4MXQct58A/g4abE+ynNMp70G+LLjOFFP01NI2UopBVyGmyBNWgd3pQtAFV4MDdolwGPe9w8Cb/K0pn6RE0OTVhc/izYnfhZtTvzy9OOM+B1Nn8+jz4lhvrLT42fR5sTPos2Jn8Wnzsa9MwczsdNqHcd51nGcvZq2MOnv9f7m4K7kabdoh9PaonSm2FytUsqPeyfr4/nqkF3XPNprgM85jpP0dD35ylVKVXht+Os8Zevip9MmgKjjODu836f6X3bu8NoqJ346rVc3bfwM2pzYWbQ5sTNpdbEzaW0Y9Dnxy1d2dvwMWq13arR1GGJnwZj/stF5p0WrzX0GrTb3WcjJfbOAMffZyM59BrTxM5DjnZaxVU7sTFpd7Cxabews+pz45RkPZsTvaMaOFm1O/PKVmx47izYndhatdtySRfp4N1/fS2kL6Hvp2nx9L12br+9lj89tfa+QsbxJn6//5ZRt6Xvp2nx9L11ril/29Uc3hryn0R405T2DVpv3LHpt7tNpTblPp9XU1abV5j1budl5z6C1xS5bP4Y+95muCXV9T6s19D2TVtf3TFpd37Ndw2b3Pev1rgaTXtf3rGVn9T2TVhc/k7YQ7zTytzLh1IY7qzrNfiwXiX8OSqm5uDPk6y0av1LqOdxHkx50HMeoBb6Je9ImC/h4B3hAKbVBKfXePNr5QC/wQ6XUs0qpm5VSkQI+461YBmuO4xwAvga8jGv0Q47jPGCQbwbOVErVKaXKcGdVO/J8fpPjON3eZ3Xjzu7+JbgS+J1NoJT6glKqC7gcd+bYpLsQOOA4zvMFfvYHlVIvKKV+oJSqsegWA2copdYrpR5VSp1cYPlnAIcdx9lp0fwT8FXv+L4GfMKi3Qxc6H1/KZoYZvULawwL6UMFaHPil621xS9dmy9+mjpY45elt8bQcHza+GVprfHL0mrjl+1TuKtDB9MSdMo/j9LTrHqlVBB4J3CfTauU+iHu3c6luMvOTdoPAr+ZPucKqMMXvPh9QykVtmgXAG9RSj2jlPqdUmpRAW1xMfBw+uDFoH8PcK9Sar/XFl82xOQpIKiUOskr7s3M9L/s3FGHIX4arQ2jNjt2Jq0udgatNnaWOuTEzqLXxs92fOTGT6fVxk6j7cMcO9Dnc5N3Hk3uz6dN906t1uCdOVqLd5rqoPNOndbmm7bjy/ZOndbknTqtzjtNYytd7I5mHFaINj12Rr0mflqtIX62emTHz6TVxS/f8aXHzqTVxc6kzTtuIXO8m2/saR0bF6jVjTsztIa+l6PNN27R1CHfuDNdn2/sqTs+07gzXZtv3JmuzYmf7voD2IAm7x3NtUo+bXbes+mzc59Fm5P78tQjI/dZtDl5r4C2SOU9i1ab9wwx+S/0uc90Tajre0dz/ViIdrrvGbWavqfVGvqerQ66vmfS6/pevuNL73smra7vmbSFeKcZ5yiWQx2rX17D3Jz28ztxO7xJP5cCHqlL05fjmtslBeqrcfdUWWn4+wXA973vzyL/I3XTz5Q24j4ueKZFexIQB07xfv4WeZbF4S5F7sPt/CZNDfB7oAH3DvyvgXdY9FfhPsf7GO4SxW/YYoCbONL/PlBIzNAsbbZoP4n7PK8q5FzA7Zif1WlxZ/PXA1Xez3vJfLwo+/iacJcN+3Cfs/6BRbsZ+Dbuct41uMtW89YZuA74P3na+du4K13AnRl/yKJdirsccwPwGeCIrV/YYpitzRM/k1YXP2Pf1MQvpS0gftnHZoyfQW+MoeX4dPHLLtcWv2xtvvhN+9QZuCtEp3/fAWwyaFem/S6jzQrQ3wR8s0CtH/g+8G6D9kzc/QSml02P2srFfexQAWHcO2nZy//TtaPTcfDOlccLqO/vpuOSpx6/ZMabP0Za3tJo1+LutfAU8O+4ewTl5A5cT86Jn06b9Vmp+BWgTcWuAG0qdob6tupiZyrXFDuLPid+BdQ5FT9LuTmxs2hzYpf2WTn5HIN36rRpmkfIfCzLps3wTps22zsN9dV6p0Gr9U6D1uabtuPL8E5D2VrvNGhzvBPD2EoXO5NWF7sCtNmxyzvGm46fQftVXfwsx5cTP4tWF798x5eKnaXcnNhZtPnyXsZ4Vxc/k9bU9/JodeMW45ib3HFLSkv+cUv2seUbt2Trbf3PdHy6cUt2ubZxS7ZW1/d01x/vRJ/3rNcqZPpVPm3GmKUAfXru02nfhT73actFk/ssWl3ey1ff9LxnKlc7ZrHotbkPzTUh5rxnvH4kN+/ZtNneme+6ND3v6epryns6re2aT6fX9r08x5ed93TlmvKeTmv1znxfBQuP5S/vBL8/66T5hEU/lwInnLyOdD/wkaOs02cwPDsNfAl3Nn4v7oz4OPCTAsu91lSu9/dmYG/az2cA9+Qp8yLggTyaS4Fb0n5+F95Au4A6fxH4gC0GuBultXjftwDbC4kZBU44AX+PuxlaWaHnAu7z5Jt1WuA43NUIe72vOO5Mf3MB5WYfe/bP9wFnpf28G2jIc3wB4DDu40q2zxpixnwVMFxgWywGnrL1C1MMdVpT/ExaXfxs5WbHL1tri18B5Wa3qa4ttDG0HF9O/AzlauNXQJ0z4pf2+8/gDiL6mBkEZfhplvajaT/vxb6nSkrvff9rvP0F8pXt/e5V6Pfa+Yz3dSgtfknSBp95yj3LUu5HcTebnJvWxkN5jq0Od1Bse4nCdDvvTvtdJ7C1wDqfi3v3UJc7btfFz6D9SVqZqfjZtNmxy1dueuwM2gFd7AosNxU7k14XvzzHlxE/g/YeXewKrPO5wH8Zzotrcc85a/5L16b9/AiafWSytRhyn6lc73cZuS9L+3+x5L485c61lPtR8uQ+w/Fpc5+mbGPuy1PnxbgXT9qxlS52Jq0udjatLnb5yk6Pn0H7sCF+xxVQ7lxLufcY4rfCcnwZsbOUmxO7AtshJ++RNd7Vxc+ktfU9nVYXP1u5ur6XriX/uNNW7lxyx43ZbWHsf4bjM407s8u1jTttdZ7ue7rrj+vQ5z3rtQqZec+oRTNmyVe297vp3KfT7kGf+wop9yxLud9Hn/dsx5ed90xtrB2zFFhnbe7DuyaksLyXcf2IPe+ltOTPe7rrUlPe+yLuJtuF5L2817uGtsib+7KOL1/emy63kLynq7P2msH29bfySN3TwCKl1DylVAh3eeZv/qeFKqUU7vPhLzqO8595tA1KqWrv+1Lg1bgGkIPjOJ9wHKfdcZy5Xl1/7zjOOwzlRpT7nC3ecuFzcRO+FsdxDgFdSqkl3q/OwR0Y23gb+ZcMvwycqpQq89rlHNy9YrQopRq9fztxZ9vzlf8bXIPA+/euPPqCUUq9DvgX4ELHccbzaBel/Xgh5hhuchyn0XGcuV4c9+Nu3HzIUG5L2o8XY4khbpI72/t/i5m5A2Tj1cA2x3H259EdxE2IeJ9hfPwuLYY+4FO4s+C2fpETw6PsQ1qtLn4WbU78dFpT/HANXFeuNn6W4zPF0NQWGfGzlJsTP0tb5MTP4FMv4q6qebP3X6djV7CneRqtXin1HuC1wNscb38Bg3a7Umph2vG/wfv/Ou0Gx3Ga0+I37jjOQksdWtLKfSOw2XJ8qdh5bb0jT1tcijsJMpmnLV4EqrzzAeA1wIuWOk/HL4x7/l9vyB2X6+J3NHnGpNXFTqcF3qmLnaHcGl3sLHXIiZ2tzrr45WmLjPgZju8iXewsdc6JnfezKZ/rvLPg3G/SGrzTpNV5p077tME7Rwzl5nin5di0vpmnLbK906TVeaepLXK80zK2yond0YzDTFrTuMWiz4mfQbvRMHbZZCg3J36W49PFb6ulLTJiZyk3J3aWdtCOW9LIHu/axp6FjI21WlP8DFrbuDOlLWDcmV1uvnFn9vHZxp66tjCNO7O1tnFndp118dNdf2xFk/cMWtO1ilary3t59Dm5z6D9T13us5Sry32m48vJe3naInvcYmrjnLyXpy1MuU93TajtewatFp3W1PcMWm3f02h/bOp7hnKNfc9wfKbcZ2qLnL5n0Gr7nqHO+bzTztHMTh3LX7jPIO7AnRX8pEX3M9znTae8E+Yqi/Z03Of7p1+ZmPFqwyztKtxHHl7wTqxPF1jvs7A8Uof7nPrzzLwy23hsaf/neNzXH77gncQ1Fm0Z7ix3VQHlfha3M27GfcNC2KJ9HNesngfOyRcD3Nn2h73O8DBQm0d/sfd9FHeS4H6LdhfuHl/TMbzeov1v7/hewH0tZFsh5w2Zd0105d6Gu9z3BVyTbbFoQ7h36jfjLnk8O9/5i/s2i/cX0M6n4y6XfB53eeiJFu2HcfvUDtzntqdnybX9QhdDizYnfhZtTvws2pz4mbS6+FnKNcXPpM+Joa0e2fGzlJsTP4s2J34YfArXZ57y2voXuEu4TdoPebGL4yaz6SXWJn0c15en6/ZpnRZ36fGfvHbejLtyp9JUblb8RvPU4fdp5f4E9/FDk7Ya9676Jtw7ZK+w1QH3jtvrCskHuOf9Ji9+j3jtbtJ+FXdwt52s1/Jm5w5d/CxabfwM2pzY6bSm2BWS69A/Dpleh5zY5dHnxM9WD138DOXmxM6i1cYOQz5H750mrc47TVqdd5q0Ou/MO/5gxjtN5eZ4p0WrzX22epDrnaaydd5p0ppyX87YShc7i9Y0btFpteMWi940drGOB8kcu+jKNeU+ndYUP20dsmNnKdc0btFptbHz9DnjXUv8dFpT/HRa07hTpzXFzjo+z4qdrlxt7Cx6U/y09TDET1euKX46ranv5Vx/YMh7Bq1p3KLTGvOeQa/NfTqtKfcZytXmPoNWm/dMdUA/btGVa8x7Br0p9+VcE2Luezqtqe/ptKa+p9Oa+p7xGlbT93Tl2vqeTm/qe9p6oO97unJNfU+nNXpnIV/THVUQBEEQBEEQBEEQBEEQZoW/lUfqBEEQBEEQBEEQBEEQhP8lZMJJEARBEARBEARBEARBmFVkwkkQBEEQBEEQBEEQBEGYVWTCSRAEQRAEQRAEQRAEQZhVZMJJEARBEARBEARBEARBmFVkwkkQBEEQBKGIUUo5SqmFf+16CIIgCIIgHA0y4SQIgiAIgnAUKKX2KqUmlFKjaV/f/WvXSxAEQRAEoZgI/LUrIAiCIAiCcAzyBsdxHvprV0IQBEEQBKFYkRVOgiAIgiAIs4BS6gql1J+UUt9RSg0ppbYppc5J+3urUuo3Sql+pdQupdTVaX/zK6X+TSm1Wyk1opTaoJTqSCv+1UqpnUqpAaXU95RSyvt/C5VSj3qf16eUuuN/8ZAFQRAEQRCMyAonQRAEQRCE2eMU4E6gHrgE+KVSap7jOP3Az4AtQCuwFHhQKfWS4zgPAx8B3gacB+wAVgHjaeVeAJwMVAIbgLuB+4DPAw8AfweEgJP+0gcoCIIgCIJQCMpxnL92HQRBEARBEI4ZlFJ7cSeU4mm//hgwBXwRaHO8AZZS6ingO8AjwF6g2nGcEe9vXwJaHMe5Qim1Hfi44zh3aT7PAc5wHOeP3s//BWx0HOfLSqkfA5PA5xzH2f8XOFxBEARBEIQ/C3mkThAEQRAE4eh5o+M41WlfN3m/P+Bk3s3bh7uiqRXon55sSvtbm/d9B7Db8nmH0r4fB8q97z8OKOAppdQWpdSVf+bxCIIgCIIgzCoy4SQIgiAIgjB7tE3vr+TRCRz0vmqVUhVZfzvgfd8FLDjaD3Mc55DjOFc7jtMKvA/4vlJq4Z9XdUEQBEEQhNlDJpwEQRAEQRBmj0bgQ0qpoFLqUmAZcK/jOF3AE8CXlFIlSqlVwFXA7d7/uxn4vFJqkXJZpZSqy/dhSqlLlVLt3o8DgAMkZvugBEEQBEEQjhbZNFwQBEEQBOHouVsplT6x8yBwF7AeWAT0AYeBNzuOc8TTvA24Hne10wDwGcdxHvT+9p9AGHcD8HpgG3BxAfU4GfimUqrK+7wPO46z539yYIIgCIIgCLOBbBouCIIgCIIwCyilrgDe4zjO6X/tugiCIAiCIPy1kUfqBEEQBEEQBEEQBEEQhFlFJpwEQRAEQRAEQRAEQRCEWUUeqRMEQRAEQRAEQRAEQRBmFVnhJAiCIAiCIAiCIAiCIMwqMuEkCIIgCIIgCIIgCIIgzCoy4SQIgiAIgiAIgiAIgiDMKjLhJAiCIAiCIAiCIAiCIMwqMuEkCIIgCIIgCIIgCIIgzCr/D4vulOoifivTAAAAAElFTkSuQmCC\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAABJwAAALMCAYAAAChcKgRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xt4VeWd//33nSMkAYQAVgEFDMcQjAp4QO2MTgUPE4dqK1RaHemMbe3j/NCh6EyrdQ7q0zoddWRsa2u105+FeZx2cCogSlXAjkWhsQqkgkCRQwsECJCEkMN6/shmEyAJIedkv1/Xlcu117r3vb5ry1+f676/K0RRhCRJkiRJktRakjq6AEmSJEmSJHUvBk6SJEmSJElqVQZOkiRJkiRJalUGTpIkSZIkSWpVBk6SJEmSJElqVQZOkiRJkiRJalUGTpIkqdMJISSHEA6FEM5pzbHq3EIIXwwhvNHRdUiSpJYzcJIkSS0WC3yO/tWEEMrrfL71dOeLoqg6iqKsKIq2tubY0xVC+KcQwnOtPW8T750UQpgbQtgY+z1/H0L45xBCWjvd/89i/y8PnfA3sT3uL0mSuraUji5AkiR1fVEUZR09DiFsAb4YRdFrDY0PIaREUVTVHrV1YfOAq4BbgdXAaOA5YAzw6da8USP/P7ZGUTS0Ne8lSZISgyucJElSm4utFFoQQvhpCOEgMDOEcGkI4e0Qwv4Qws4QwpMhhNTY+JQQQhRCGBr7/JPY9cUhhIMhhP8NIQw73bGx69eGED4MIZSEEP4thPBWCOH2ZjxTbgjhzVj974cQrq9z7YYQwvrY/beFEGbHzg8MISyKfWdvCGF5A3OPBu4EZkRR9OsoiqqiKPoAuBm4IYRwZQhhcghhewghqc73PhNCWBM7Tgoh/F0I4aMQwp4QwvwQQt/YtZzYb/aXIYStwNJmPP/K2Iqrd2O/5c+Pzh+7Pi2EsDb2rL8MIYyqc+3cEMJ/hxB2x2p74vipw7/GvrcphHBNnQuzQghbYr/rphDC9NOtW5IktQ8DJ0mS1F6mAS8AfYAFQBXwN0B/YDIwldqQpSGfA74B9AO2Av94umNDCAOB/wTmxO67GZh0ug8S29b2C+BlYAAwG1gQQsiJDfkRMCuKol7AeODN2Pk5wKbYdz4BfL2BW/wZsDmKojV1T0ZRtAV4B/gU8CugEvjkCc/9Qux4NnA9cCUwGDgEPHnCfa6kduXU9TTPF2J/ZwMB+FeAEMIY4D+A/4faZ30NeCmEkBpCSKH2d9sIDAWGUPv/5KjLgPeB7Nh8P4zN2Rv4DvCp2O86GfhtM+uWJEltzMBJkiS1l5VRFP1PFEU1URSVR1H0Tp3VO5uA73N8eHKiF6MoejeKokrg/wL5zRh7A1AYRdHC2LV/BfY041kmA2nAt6MoqoxtH1wMHF1xUwmMDSH0iqJob53gqJLacOacKIqORFFU7wonasOwnQ1c2wn0j6IoAuYDMwBCCGcAU2LnAL4E/F0URdujKDoMPAR8pu6KKODBKIrKoigqb+Be58RWGtX9S69z/fkoitZFUVQKPABMDyGE2O/wUhRFv4z9zo9SGzReDFwae765URSVxv4tvFVnzo+iKHo2iqJq4HlgcAihf+xaBIwLIfSIomhnFEXrGqhbkiR1MAMnSZLUXj6u+yGEMDqE8HII4Q8hhAPAP1AbRDTkD3WOy4CshgY2MvbsunXEQpttTaj9RGdT298oqnPu98Cg2PE0oADYGkJ4I4Rwcez8o7Fxy2Jb3eY0MP8e4KwGrp3FsZDsBeCm2FbEm4BfR1F09HnOAf7naFBE7aohgIF15jru/0k9tkZRdMYJfxUNfP/3QDq1q8rOjn0GIIqiGmp/50HUrmjaEguU6nPi/zuArCiKDlAbrt0F/CGE8IsQwshT1C9JkjqIgZMkSWov0Qmfvwd8AOREUdSb2hUyoY1r2Ent9jKgtlkQx0Ki07EDGBL7/lHnANsBYiu3CqgNd35BbNVRFEUHoiiaHWvE/RfA3BBCfau6lgHDQggX1j0Z61M1MXadKIp+S21AM4Xjt9NBbcDzqRPCoh5RFMUDnRMCs+YYUuf4HKAC2Evt73NunbqTqP3dt1MbUp0bQkg+3ZtFUbQ4iqI/ozZ020jtvyFJktQJGThJkqSO0gsoAUpjPX8a69/UWn4BXBhC+PNYL6G/obbHUGOSQwg96vylU9s/qQq4N9aX6CrgOmr7OPUMIXwuhNA7tp3sIFADELvvebGgqgSoPnqtriiK1gM/AH4aQpgUQkgOIYwDXgQWR1H0Rp3hL1Dbr+nS2PWjvgs8HEI4J3bvgSGEgtP4rZriC7GVapnUbtn7z1iI9Z9AQQjhT2Krr+ZQ+zv8GvhfoDhWW0bs95p8qhuFEM6K/X4ZwBGglHp+O0mS1DkYOEmSpI5yL3AbtUHE96htJN6moij6I3ALtc2ni4HzgN9QuzKnITOB8jp/v4ttK/tz4EZqt7c9CXwuiqINse/cBvw+tlVwVmwOgFHAL6lt4P0W8EQURSsauO+Xqe1h9FNqw5XFwKvAZ08Y9wJwFfBqFEX76pz/DrCE2u17B6kNySY28pz1OSeEcOiEv7+oc/0/gJ9Qu3IsGfg/AFEUrY39Bk8Du6ltCF8Q63dVRW0vrTHUrnbaSu3b904lmdrgaie1/+8uo3Z7nSRJ6oRCy1dSS5IkdU2xbV07gJsbCX5UjxDCSuAHURQ919G1SJKkzscVTpIkKaGEEKaGEM6IbY37BrVvjlvVwWVJkiR1KwZOkiQp0VwObKJ2q9cUYNoJb16TJElSC7mlTpIkSZIkSa3KFU6SJEmSJElqVSkdXUBb6N+/fzR06NCOLkOSJEmSJKnbWL169Z4oigY0ZWy3DJyGDh3Ku+++29FlSJIkSZIkdRshhN83daxb6iRJkiRJktSqDJwkSZIkSZLUqgycJEmSJEmS1Kq6ZQ8nSZIkSZJ0+iorK9m2bRuHDx/u6FLUgXr06MHgwYNJTU1t9hwGTpIkSZIkCYBt27bRq1cvhg4dSgiho8tRB4iiiOLiYrZt28awYcOaPY9b6iRJkiRJEgCHDx8mOzvbsCmBhRDIzs5u8So3AydJkiRJkhRn2KTW+Ddg4CRJkiRJkqRWZQ8nSZIkSZJUr7yVH7C7sqrV5huQmsL7l49rtfnUebnCSZIkSZIk1as1w6a2mK+ze+655/jqV7/a5PFRFHH33XeTk5PD+PHjWbNmTb3jVq9eTV5eHjk5Odx9991EUQTAnDlzGD16NOPHj2fatGns378fgC1bttCzZ0/y8/PJz8/nS1/6Ussf7hQMnCRJkiRJUpdWXV3d0SWclqqq+oO3xYsXs2HDBjZs2MD3v/99vvzlL9c77stf/jLPPPNMfOySJUsA+NSnPsUHH3zAb3/7W0aOHMkjjzwS/855551HYWEhhYWFfPe73239hzqBgZMkSZIkSeoUSktLuf766zn//PMZN24czz//PJ/5zGfi19944w1uuOEGALKysrj33ns5//zz+d///V8WLVrE6NGjueiii7j77rvj4+rzzW9+k9tuu40rrriCc889l5/97Gd87WtfIy8vj6lTp1JZWQnAfffdx9ixYxk/fjx/+7d/C8Du3bu56aabmDhxIhMnTuStt95q0rPdfvvtfOlLX+Liiy/ma1/7Wr1jFi5cyBe+8AVCCFxyySXs37+fnTt3Hjdm586dHDhwgEsuuYQQAl/4whf47//+bwCuueYaUlJquyddcsklbNu2rUm1tQV7OEmSJEmSpE5hyZIlnH322bz88ssAlJSU8I1vfIPS0lIyMzNZsGAB06dPB2rDqYsvvph/+Zd/4fDhw4wYMYLly5czbNgwZsyYccp7ffTRR7z++uusW7eOSy+9lP/6r//iW9/6FtOmTePll1/miiuu4Oc//zlFRUWEEOLb0/7mb/6G2bNnc/nll7N161amTJnC+vXrm/R827Zt41e/+hXJycn1Xt++fTtDhgyJfx48eDDbt2/nrLPOOm7M4MGDTxpzomeffZZbbrkl/nnz5s1ccMEF9O7dm3/6p3/iiiuuaFLNzeUKJ0mSJEmS1Cnk5eXx6quvMnfuXFasWEGfPn2YOnUq//M//0NVVRUvv/wyN954IwDJycncdNNNABQVFTF8+HCGDRsG0KTA6dprryU1NZW8vDyqq6uZOnVqvIYtW7bQp08fevTowaxZs/jZz35GRkYGAK+99hpf/epXyc/Pp6CggAMHDnDo0KEmPd9nPvOZBsOm1vTP//zPpKSkcOuttwJw1llnsXXrVn7zm9/wne98h8997nMcOHCgTWtwhZMkSZIkSeoURo4cyZo1a1i0aBFf//rXufrqq5k+fTpPPfUU/fr1Y8KECfTq1QuAHj16tCi8SU9PByApKYnU1FRCCPHPVVVVpKSksGrVKpYtW8aLL77IU089xS9/+Utqamp4++236dGjx2nfMzMzs9HrgwYN4uOPP45/3rZtG4MGDTppTN2tcieOee655/jFL37BsmXL4s+Unp4ef96LLrqI8847jw8//JAJEyac9jM0lSucJEmSJElSvQaktu46lVPNt2PHDjIyMpg5cyZz5sxhzZo1fPKTn2TNmjU888wz8e10Jxo1ahSbNm1iy5YtACxYsKDFtR46dIiSkhKuu+46/vVf/5X33nsPqO2T9G//9m/xcYWFhS2+11EFBQX8+Mc/Jooi3n77bfr06XPcdjqoXa3Uu3dv3n77baIo4sc//nF81deSJUv41re+xUsvvRRfkQW1faeONlbftGkTGzZsYPjw4a1Wd31c4SRJkiRJkur1/uXj2vd+77/PnDlz4quOnn76aZKTk7nhhht47rnneP755+v9Xs+ePfn3f/93pk6dSmZmJhMnTmxxLQcPHuTGG2/k8OHDRFHEd77zHQCefPJJ7rrrLsaPH09VVRVXXnllq7317brrrmPRokXk5OSQkZHBj370o/i1/Pz8eLj17//+79x+++2Ul5dz7bXXcu211wLw1a9+lYqKCj71qU8BtY3Dv/vd77J8+XIeeOABUlNTSUpK4rvf/S79+vVrlZobEqIoatMbdIQJEyZE7777bkeXIUmSJElSl7J+/XrGjBnT0WU0y6FDh8jKyiKKIu666y5GjBjB7NmzO7qsLqu+fwshhNVRFDVpH55b6iRJkiRJUpf3zDPPkJ+fT25uLiUlJdx5550dXVJCc0udJEmSJEnq8mbPnn3SiqYf/ehHPPHEE8edmzx5MvPmzWvVe5/ufdqrro7kljpJkiRJkgR07S11al1uqZMkSZIkSVKnYuAkSZIkSZKkVmXgJEmSJEmSpFZl03BJkiRJklSvFSsv5siRPa02X1paf664/NetNp86L1c4SZIkSZKkerVm2NQW89U1dOhQ9uxpu/mbKisr67TGL1myhFGjRpGTk8Ojjz5a75iKigpuueUWcnJyuPjii9myZUv82iOPPEJOTg6jRo3ilVdeiZ+/4447GDhwIOPGjWvWc7SUgZMkSZIkSVIbqqqqqvd8dXU1d911F4sXL2bdunX89Kc/Zd26dSeN++EPf0jfvn3ZuHEjs2fPZu7cuQCsW7eO+fPns3btWpYsWcJXvvIVqqurAbj99ttZsmRJ2z3UKXT7wKmsbAtFv3uAX/3qT3nrV59k/fr7OVS6oaPLkiRJkiRJJygtLeX666/n/PPPZ9y4cSxYsIBFixYxevRoLrroIu6++25uuOEGAIqLi7nmmmvIzc3li1/8IlEUNTjvli1bGD16NLfffjsjR47k1ltv5bXXXmPy5MmMGDGCVatWAfDmm2+Sn59Pfn4+F1xwAQcPHgTg29/+NhMnTmT8+PE8+OCDTXqWN954gyuuuIKCggLGjh1b75hVq1aRk5PD8OHDSUtLY/r06SxcuPCkcQsXLuS2224D4Oabb2bZsmVEUcTChQuZPn066enpDBs2jJycnPizXHnllfTr169JtbaFbh04lZT8hlXvFLB9+/+l/PBWDh/exo6d/8k779zI3r1vdXR5kiRJkiSpjiVLlnD22Wfz3nvv8cEHHzB16lTuvPNOFi9ezOrVq9m9e3d87EMPPcTll1/O2rVrmTZtGlu3bm107o0bN3LvvfdSVFREUVERL7zwAitXruSxxx7j4YcfBuCxxx5j3rx5FBYWsmLFCnr27MnSpUvZsGEDq1atorCwkNWrV7N8+fImPc+aNWt44okn+PDDD+u9vn37doYMGRL/PHjwYLZv397ouJSUFPr06UNxcXGTv98Rum3gFEU1rFs/h+rq0pOu1dRUsG7dHGpqKjugMkmSJEmSVJ+8vDxeffVV5s6dy4oVK9i8eTPDhw9n2LBhAMyYMSM+dvny5cycOROA66+/nr59+zY697Bhw8jLyyMpKYnc3FyuvvpqQgjk5eXFeyJNnjyZe+65hyeffJL9+/eTkpLC0qVLWbp0KRdccAEXXnghRUVFbNjQtJ1TkyZNiteeaLpt4LS/ZDVlZZsbvF5x5I/s3buiHSuSJEmSJEmNGTlyJGvWrCEvL4+vf/3rvPTSS602d3p6evw4KSkp/jkpKSneY+m+++7jBz/4AeXl5UyePJmioiKiKOL++++nsLCQwsJCNm7cyKxZs5p0z8zMzEavDxo0iI8//jj+edu2bQwaNKjRcVVVVZSUlJCdnd3k73eEbhs4VRzeecoxhw/vaIdKJEmSJEnqmtLS+rfrfDt27CAjI4OZM2cyZ84c3nrrLTZt2hRfgbRgwYL42CuvvJIXXngBgMWLF7Nv374W1/fRRx+Rl5fH3LlzmThxIkVFRUyZMoVnn32WQ4cOAbXb23bt2tXiewFMnDiRDRs2sHnzZo4cOcL8+fMpKCg4aVxBQQHPP/88AC+++CJXXXUVIQQKCgqYP38+FRUVbN68mQ0bNjBp0qRWqa2lUjq6gLaS3uOsU47p0ePsdqhEkiRJkqSu6YrLf92u93v//feZM2cOSUlJpKam8vTTT7Nz506mTp1KZmYmEydOjI998MEHmTFjBrm5uVx22WWcc845Lb7/448/zuuvvx7fdnfttdeSnp7O+vXrufTSSwHIysriJz/5CQMHDmzx/VJSUnjqqaeYMmUK1dXV3HHHHeTm5gLwwAMPMGHCBAoKCpg1axaf//znycnJoV+/fsyfPx+A3NxcPvvZzzJ27FhSUlKYN28eycnJQO32wzfeeIM9e/YwePBgHnrooSavzGoNobEu7l3VhAkTonfeWcXbv55CWdmmesekp3+Cyy59g6Sk1HauTpIkSZKkzmn9+vWMGTOmo8s4zqFDh8jKyiKKIu666y5GjBjB7NmzO7qsbq++fwshhNVRFE1oyve77Za6EJIYO+bbJCdnnXQtKakHY8c+ZtgkSZIkSVIn98wzz5Cfn09ubi4lJSXceeedHV2SmqDbbqkD6NMnn4sn/Q9bP36WHTtepKamHICLLlxA797jOrg6SZIkSZJ0KrNnz27yiqbi4mKuvvrqk84vW7aM7OzsVqvpdO/TXnV1Jt06cALo2fMcRo38JklJPdi69Znak6Fja5IkSZIkSa0vOzubwsLCTnef9qqrM+m2W+pOlJmREz8uK/2oAyuRJEmSJEnq3hIncMo8L35cWrqxAyuRJEmSJEnq3hIocDq2wqm0zBVOkiRJkiRJbaXb93A6KiWlF2lpAzlyZBelbqmTJEmSJOmUPrz8Cqr37Gm1+ZL792fkyhWtNp86r4RZ4QTHttWVl2+hpqayg6uRJEmSJKlza82wqS3mq2vo0KHsacP5myorK+u0xi9ZsoRRo0aRk5PDo48+Wu+YiooKbrnlFnJycrj44ovZsmVL/NojjzxCTk4Oo0aN4pVXXgHg448/5k//9E8ZO3Ysubm5PPHEE/Hx3/zmNxk0aBD5+fnk5+ezaNGi03/IJkiswCnWODyKqigv39rB1UiSJEmSpERQVVVV7/nq6mruuusuFi9ezLp16/jpT3/KunXrThr3wx/+kL59+7Jx40Zmz57N3LlzAVi3bh3z589n7dq1LFmyhK985StUV1eTkpLCv/zLv7Bu3Trefvtt5s2bd9y8s2fPprCwkMLCQq677ro2eeaECpwy6jYOL7NxuCRJkiRJnUlpaSnXX389559/PuPGjWPBggUsWrSI0aNHc9FFF3H33Xdzww03AFBcXMw111xDbm4uX/ziF4miqMF5t2zZwujRo7n99tsZOXIkt956K6+99hqTJ09mxIgRrFq1CoA333wzvvLnggsu4ODBgwB8+9vfZuLEiYwfP54HH3ywSc/yxhtvcMUVV1BQUMDYsWPrHbNq1SpycnIYPnw4aWlpTJ8+nYULF540buHChdx2220A3HzzzSxbtowoili4cCHTp08nPT2dYcOGkZOTw6pVqzjrrLO48MILAejVqxdjxoxh+/btTaq7tSRU4JSZcSxwKrOPkyRJkiRJncqSJUs4++yzee+99/jggw+YOnUqd955J4sXL2b16tXs3r07Pvahhx7i8ssvZ+3atUybNo2tWxvfybRx40buvfdeioqKKCoq4oUXXmDlypU89thjPPzwwwA89thjzJs3j8LCQlasWEHPnj1ZunQpGzZsYNWqVRQWFrJ69WqWL1/epOdZs2YNTzzxBB9++GG917dv386QIUPinwcPHlxvMFR3XEpKCn369KG4uLhJ39+yZQu/+c1vuPjii+PnnnrqKcaPH88dd9zBvn37mvQspyuxAqe6K5wMnCRJkiRJ6lTy8vJ49dVXmTt3LitWrGDz5s0MHz6cYcOGATBjxoz42OXLlzNz5kwArr/+evr27dvo3MOGDSMvL4+kpCRyc3O5+uqrCSGQl5cX74k0efJk7rnnHp588kn2799PSkoKS5cuZenSpVxwwQVceOGFFBUVsWHDhiY9z6RJk+K1d4RDhw5x00038fjjj9O7d28AvvzlL/PRRx9RWFjIWWedxb333tsm906owCktbSDJybXNu9xSJ0mSJElS5zJy5EjWrFlDXl4eX//613nppZdabe709PT4cVJSUvxzUlJSvMfSfffdxw9+8APKy8uZPHkyRUVFRFHE/fffH+95tHHjRmbNmtWke2ZmZjZ6fdCgQXz88cfxz9u2bWPQoEGNjquqqqKkpITs7OxGv19ZWclNN93Erbfeyqc//en4mDPPPJPk5GSSkpL4q7/6q/h2wtaWUIFTCIHMzNrG4WVlm4iimg6uSJIkSZKkziu5f/92nW/Hjh1kZGQwc+ZM5syZw1tvvcWmTZviK5AWLFgQH3vllVfywgsvALB48eJW2Rr20UcfkZeXx9y5c5k4cSJFRUVMmTKFZ599lkOHDgG129t27drV4nsBTJw4kQ0bNrB582aOHDnC/PnzKSgoOGlcQUEBzz//PAAvvvgiV111FSEECgoKmD9/PhUVFWzevJkNGzYwadIkoihi1qxZjBkzhnvuuee4uXbu3Bk//vnPf864ceNa5VlOlNIms3ZimRnnceBAIdXVZVRU/IEePc7u6JIkSZIkSeqURq5c0a73e//995kzZw5JSUmkpqby9NNPs3PnTqZOnUpmZiYTJ06Mj33wwQeZMWMGubm5XHbZZZxzzjktvv/jjz/O66+/Ht92d+2115Kens769eu59NJLAcjKyuInP/kJAwcObPH9UlJSeOqpp5gyZQrV1dXccccd5ObmAvDAAw8wYcIECgoKmDVrFp///OfJycmhX79+zJ8/H4Dc3Fw++9nPMnbsWFJSUpg3bx7JycmsXLmS//iP/yAvL4/8/HwAHn74Ya677jq+9rWvUVhYSAiBoUOH8r3vfa/Fz1Gf0FgX965qwoQJ0bvvvlvvtd///nts/OhbAOSf/yOys69sz9IkSZIkSeq01q9fz5gxYzq6jOMcOnSIrKwsoijirrvuYsSIEcyePbujy+r26vu3EEJYHUXRhKZ8P6G21AFkxLbUAZSW2ThckiRJkqTO7JlnniE/P5/c3FxKSkq48847O7okNUFCbqk7qrTUxuGSJEmSJHVms2fPbvKKpuLiYq6++uqTzi9btozs7OxWq+l079NedXUmCRc49ew5hKSkNGpqjlBW6gonSZIkSZK6i+zsbAoLCzvdfdqrrs4k4bbUhZBMRs9hgFvqJEmSJEmS2kLCBU4AGZm12+oqK/dSWdny1yZKkiRJkiTpmIQMnDIz6jQOd1udJEmSJElSq0q4Hk4AmZnHNw4/44wmvdFPkiRJkqSE8u1vf5vS0tJWmy8zM5M5c+a02nzqvBJyhVNGZp0VTvZxkiRJkiSpXq0ZNrXFfHUNHTqUPXv2tNn8TZWVlXVa45csWcKoUaPIycnh0UcfrXdMRUUFt9xyCzk5OVx88cVs2bIlfu2RRx4hJyeHUaNG8corr8TPDx06lLy8PPLz85kwof0X2iTkCqfapuFJQA1lpRs7uhxJkiRJktSNVVVVkZJycgRTXV3NXXfdxauvvsrgwYOZOHEiBQUFjB079rhxP/zhD+nbty8bN25k/vz5zJ07lwULFrBu3Trmz5/P2rVr2bFjB3/2Z3/Ghx9+SHJyMgCvv/46/fv3b5dnPFFCrnBKTk6nZ8/BgCucJEmSJEnqLEpLS7n++us5//zzGTduHAsWLGDRokWMHj2aiy66iLvvvpsbbrgBgOLiYq655hpyc3P54he/SBRFDc67ZcsWRo8eze23387IkSO59dZbee2115g8eTIjRoxg1apVALz55pvk5+eTn5/PBRdcwMGDB4HarYUTJ05k/PjxPPjgg016ljfeeIMrrrii3gDpqFWrVpGTk8Pw4cNJS0tj+vTpLFy48KRxCxcu5LbbbgPg5ptvZtmyZURRxMKFC5k+fTrp6ekMGzaMnJyc+LN0tHYLnEIIU0MIvwshbAwh3FfP9XNDCMtCCL8NIbwRQhhc51p1CKEw9vdSa9RztHH44cPbqa4ua40pJUmSJElSCyxZsoSzzz6b9957jw8++ICpU6dy5513snjxYlavXs3u3bvjYx966CEuv/xy1q5dy7Rp09i6dWujc2/cuJF7772XoqIiioqKeOGFF1i5ciWPPfYYDz/8MACPPfYY8+bNo7CwkBUrVtCzZ0+WLl3Khg0bWLVqFYWFhaxevZpfnNQWAAAgAElEQVTly5c36XnWrFnDE088wYcffljv9e3btzNkyJD458GDB7N9+/ZGx6WkpNCnTx+Ki4sb/X4IgWuuuYaLLrqI73//+02qtzW1S+AUQkgG5gHXAmOBGSGEE+O9x4AfR1E0HvgH4JE618qjKMqP/RW0Rk0ZdRuHl21qjSklSZIkSVIL5OXl8eqrrzJ37lxWrFjB5s2bGT58OMOGDQNgxowZ8bHLly9n5syZAFx//fX07du30bmHDRtGXl4eSUlJ5ObmcvXVVxNCIC8vL94TafLkydxzzz08+eST7N+/n5SUFJYuXcrSpUu54IILuPDCCykqKmLDhg1Nep5JkybFa29vK1euZM2aNSxevJh58+Y1OSRrLe21wmkSsDGKok1RFB0B5gM3njBmLPDL2PHr9VxvVUdXOAGUlbqtTpIkSZKkjjZy5EjWrFlDXl4eX//613nppVbZ5ARAenp6/DgpKSn+OSkpiaqqKgDuu+8+fvCDH1BeXs7kyZMpKioiiiLuv/9+CgsLKSwsZOPGjcyaNatJ98zMzGz0+qBBg/j444/jn7dt28agQYMaHVdVVUVJSQnZ2dmNfv/ofwcOHMi0adPafatdewVOg4CP63zeFjtX13vAp2PH04BeIYTs2OceIYR3QwhvhxD+or4bhBD+Ojbm3bpL7BqSWXeFk43DJUmSJEk6yakCk9aeb8eOHWRkZDBz5kzmzJnDW2+9xaZNm+IrkBYsWBAfe+WVV/LCCy8AsHjxYvbt29fi+j766CPy8vKYO3cuEydOpKioiClTpvDss89y6NAhoHZ7265du1p8L4CJEyeyYcMGNm/ezJEjR5g/fz4FBSdv7CooKOD5558H4MUXX+Sqq64ihEBBQQHz58+noqKCzZs3s2HDBiZNmkRpaWm8/1RpaSlLly5l3LhxrVJzU3Wmt9T9LfBUCOF2YDmwHaiOXTs3iqLtIYThwC9DCO9HUXTcsqQoir4PfB9gwoQJDXcKi8nMPLbCycbhkiRJkiSdbM6cOe16v/fff585c+aQlJREamoqTz/9NDt37mTq1KlkZmYyceLE+NgHH3yQGTNmkJuby2WXXcY555zT4vs//vjjvP766/Ftd9deey3p6emsX7+eSy+9FICsrCx+8pOfMHDgwBbfLyUlhaeeeoopU6ZQXV3NHXfcQW5uLgAPPPAAEyZMoKCggFmzZvH5z3+enJwc+vXrx/z58wHIzc3ls5/9LGPHjiUlJYV58+aRnJzMH//4R6ZNmwbUroj63Oc+x9SpU1tc7+kIjXVxb7WbhHAp8M0oiqbEPt8PEEXRIw2MzwKKoigaXM+154BfRFH0YkP3mzBhQvTuu++esq4VKy/lyJFdZGTkcOklrzTpWSRJkiRJ6q7Wr1/PmDFjOrqM4xw6dIisrCyiKOKuu+5ixIgRzJ49u6PL6vbq+7cQQlgdRdGEpny/vbbUvQOMCCEMCyGkAdOB4zZihhD6hxCO1nM/8GzsfN8QQvrRMcBkYF1rFHV0W115+RZqaipbY0pJkiRJktSKnnnmGfLz88nNzaWkpIQ777yzo0tSE7TLlrooiqpCCF8FXgGSgWejKFobQvgH4N0oil4C/gR4JIQQUbul7q7Y18cA3wsh1FAbkD0aRVHrBE4ZOezb979EURXl5VuP6+skSZIkSZI63uzZs5u8oqm4uJirr776pPPLli0jOzu7nm80z+nep73q6kzarYdTFEWLgEUnnHugzvGLwEnb5KIo+hWQ1xY1ZWQOjx+Xlm00cJIkSZIkJbwoigghdHQZzZKdnU1hYWGnu0971dVaWqP9UnttqeuUMjOOBUxlpTYOlyRJkiQlth49elBcXNwqgYO6piiKKC4upkePHi2apzO9pa7dHfemOgMnSZIkSVKCGzx4MNu2bWP37t0dXYo6UI8ePRg8+KT3uJ2WhA6c0tIGkpycRXX1IUrLNnZ0OZIkSZIkdajU1FSGDRvW0WWoG0joLXUhhPgqp7KyTURRTQdXJEmSJEmS1PV1+xVOUXU1h5Yvp/StX0FNDRmXXEyvq64ipNQ+embGeRw4UEh1dRkVFX+gR4+zO7hiSZIkSZKkrq1bB05V+/bx8Ze+xOH3fhs/t++FF0gfOZIhzzxD6pkDj3szXWnpRgMnSZIkSZKkFurWW+p23nf/cWHTURUffsj2e+4BIKNu4/AyG4dLkiRJkiS1VLcNnCo2b+bQm282eL189WrK3/+AzIxjK5zKfFOdJEmSJElSi3XbwOnw2nWnHvPB+/TsOYSkpDTAFU6SJEmSJEmtodsGTkkZPZswJoMQksnoWfvKx9LSjW1dliRJkiRJUrfXbQOnzEsuIalXrwavh/R0sj75SQAyYo3DKyv3Ulm5r13qkyRJkiRJ6q66beCUlJHBgNn/p8Hr/b/yFZLPOAOAzIw6jcPt4yRJkiRJktQi3TZwAuj3uc9x1qOPkDJkSPxcSE/jE998kOy//qv4uczMY43D3VYnSZIkSZLUMt06cAI44y/+gpxXlpDcPxuAtHPPpe/06YQQ4mMyMuuscLJxuCRJkiRJUot0+8AJICQlkTrwTACq9hSfdL22aXjtT1HmCidJkiRJkqQWSYjACSB5QH8AqvftI6qqOv5acjo9ew4GXOEkSZIkSZLUUgkTOKX0rw2ciCKq9u496frRxuGHD2+nurqsPUuTJEmSJEnqVhIncMruHz+u3rPnpOsZdRuHl21ql5okSZIkSZK6o8QJnPofC5yqik/u43R0hRNAWanb6iRJkiRJkporcQKnAXUCp90nr3DKrLvCycbhkiRJkiRJzZYwgVNydnb8uKqeLXWZmcdWONk4XJIkSZIkqfkSJnBK6T8gflxdfHLglJLSi7S0gQCUuqVOkiRJkiSp2RIncDrFljqAzIzhAJSXb6GmprJd6pIkSZIkSepuEiZwSsrKIqSlAfVvqQPIiG2ri6Iqysu3tlttkiRJkiRJ3UnCBE4hhPib6up7Sx2c0Di8zMbhkiRJkiRJzZEwgRNA8tHAqYEVTpkZxwKnMvs4SZIkSZIkNUtCBU5HVzjVlJRQc+TISdePe1OdgZMkSZIkSVKzJGTgBFBdz7a6tLSBJCdnAXDw0HpqaqrarTZJkiRJkqTuIsECp+z4cX3b6vaXvAtEAJSW/o6Vb01my5bvEkXV7VWiJEmSJElSl5dQgVNynRVOJwZO+/a9zW9+M5Pq6tL4ucrKPXy06dsU/e4b7VajJEmSJElSV5dQgVNKA4FTFEVs2PgwUVT/FrodOxZw8FBRm9cnSZIkSZLUHSRs4FRdJ3AqL9/KwYNrG/3urj++3GZ1SZIkSZIkdScJGzhV7TnWNLzuNrqGVFUfapOaJEmSJEmSupvECpyy628a3rPnuSQnZTT63V5ZuW1WlyRJkiRJUneSUIFTUmYmIaM2WKobOKWkZHLW2Z9p8Htpaf0588zr27w+SZIkSZKk7iChAic4tq2u+oS31OWc9zWysz950vjk5F6cP/4ZkpN7tkt9kiRJkiRJXV3CBk5VJwROyck9OH/8D7kg/8dkZ/9J/PyQIbfRu/f49ixRkiRJkiSpS0u8wCnWx6mmtJSa8vLjroUQ6NdvMrljH4ufO3SoqF3rkyRJkiRJ6uoSL3AaUOdNdcXF9Y5JTe1Lz57nAnDgQCFRFLVLbZIkSZIkSd1BwgVOyf3rBE67dzc4rnfv8wE4cmQPFRU727wuSZIkSZKk7iLhAqeU7DqB0wl9nOrqEwucAEoOFLZpTZIkSZIkSd1J4gVOdbbUVTewpQ6gd+/8+PGBEgMnSZIkSZKkpkq8wOm4LXUNr3Dq1WsMIaQCUHLgvTavS5IkSZIkqbtIvMAp9pY6aHxLXVJSOr2yxgBw8OAH1NRUtXltkiRJkiRJ3UHCBU7HNQ0vbjhwAujdp7aPU03NYUpLP2zTuiRJkiRJkrqLhAucktLTSerVC4DqRrbUwfF9nGwcLkmSJEmS1DQJFzjBsT5OjW2pg+PfVHfAPk6SJEmSJElNktiBU3ExURQ1OK5nz6GkpPQBDJwkSZIkSZKaKiEDp+T+tY3Do8OHqSktbXBcCIHevccDUFq6kaqqg+1SnyRJkiRJUleWkIFTSv8B8eOq3bsbHdsn3scp4sCB99uwKkmSJEmSpO4hQQOnY2+qqy4ubnRsb/s4SZIkSZIknZYEDZyy48enahx+dEsdGDhJkiRJkiQ1RYIGTsdWOFXtbjxwSkvLpmePcwAoOfBeo03GJUmSJEmSlKCBU3LdwKm48cAJoHef2m11R47soqJiZ5vVJUmSJEmS1B0kZOB03AqnU2ypg+P7OJW4rU6SJEmSJKlRiRk49esXP64+xZY6qPumOvs4SZIkSZIknUpCBk4hNZXkvn0BqDrFW+oAsrLGEkIqYOAkSZIkSZJ0KgkZOMGxN9U1ZUtdcnI6WVmjAThw4H1qaqratDZJkiRJkqSuLGEDp6ONw6uKi5v05rmj2+pqasopLd3QprVJkiRJkiR1ZQkbOKX0H1B7UFlJTUnJKcfXbRzutjpJkiRJkqSGJUTgVFNTw+7du9m1axfV1dUApGRnx6+f7pvqDJwkSZIkSZIaltLRBbS1NWvW8Oabb1ISW8XUq1cvLr/8cs7rf3zglJ6T0+g8GRlDSUnpTVXVAUoOFLZpzZIkSZIkSV1Zt17h9Otf/5qXXnopHjYBHDx4kMWLF/NRnbfTVe059ZvqQkiKr3IqLd1AVdWh1i9YkiRJkiSpG+i2gVNFRQW//OUvG7z+3u9/Hz+u2rO7SXMe21YXcfDgBy0pT5IkSZIkqdvqtoHT5s2bqaioaPB6aVpa/Li6CT2cAPrU6eNUYh8nSZIkSZKkenXbwKmysrLR64d79IgfN2VLHZzYONw+TpIkSZIkSfXptoHT2Wef3ej1I2lpkFT7+E15Sx1AWlo2PXoMAeBAiSucJEmSJEmS6tNtA6fs7GxGjRrV4PVzhg0jObsfAFXFTVvhBMe21VUc+SOHK/7QsiIlSZIkSZK6oW4bOAHceOONDBky5KTzn/jEJ7j55ptJye4PNL1pOJywrc5VTpIkSZIkSSfp1oFTRkYGf/mXf8mMGTPi5wYNGsRf//Vf06tXL1L61wZO1cV7iaqrmzRn7z72cZIkSZIkSWpMtw6cAJKSkhg1ahRpsbfSJScnkxTr3XQ0cKKmhur9+5s0X6+sXEJIAXxTnSRJkiRJUn26feB0VM+ePQEoLy+Pn0vpnx0/bmrj8OTkHmRljQbg4MH3iaKmrYySJEmSJElKFAkdOCUfXeEEVO1uWuAEx/o4VVeXUVq6sZUqlCRJkiRJ6h4SMnCKogiAlP4D4teri5seOPWp0zi8xD5OkiRJkiRJx0m4wKm6uprKykqgeVvqAHr3zo8fHygxcJIkSZIkSaor4QInOLatLqWZW+oyMoaRktILgAMHf9tKFUqSJEmSJHUPCRk4lZWVAScETsXFTZ4rhCR69xoPwKFDH1JVVdpKVUqSJEmSJHV9CRk4HV3hlNSnD6SmAlC1Z/dpzdc73sephoMHP2iVGiVJkiRJkrqDhA6cQgikZNf2cao+jR5OAL371OnjZONwSZIkSZKkuIQJnDIyMuLHRwMnOLatrmpP07fUQd0VTlBywD5OkiRJkiRJRyVM4FTfCifg2AqnffuIYm+va4qkkEpycm3j8N27X2HVO3/Bjh0vEkVRK1UsSZIkSZLUNSV84JQ8oE7j8L17mzRXZeV+3l39WaqrD8bORBw8+D7ri+ZS9Lu/N3SSJEmSJEkJLeEDp+PeVNfEPk6bN/8bZWUb6722Y8cC9u37VTOrlCRJkiRJ6voMnLKPBU5NaRweRRE7//CzRsfs3Nn4dUmSJEmSpO4sYQKnlJQUUlNTgRMCpwGnt8KppqaCqqoDjY6pOLKrmVVKkiRJkiR1fQkTOMGxVU71NQ2Hpr2pLikpnbS0gae4zznNrFCSJEmSJKnrS8jAqaysLH4u+TR7OIUQGHT29EbHnOq6JEmSJElSd5aQgVN5eXn8TXIpAwbEr1ft2d2kec4990v07XtpvdfOGz6H3r3zWlipJEmSJElS15VQgVNGRgYA1dXVVFZWApCUmUlIT68934QtdQDJyenkn/8jxoz5fznjjEuBZADS0gYydOiXWr9wSZIkSZKkLiShAqf63lQXQiAltq2uKVvqjkpKSuXss27mogt/wsCBUwA4cmQXZWVbWq9gSZIkSZKkLijhAyegWYFTXdn9rogf7927spnVSZIkSZIkdQ8GThxrHF5z8CA1FRWnPW+/fpfHj4v3rmhBhZIkSZIkSV2fgRPHVjgBVDdjlVOPHmeTkZEDwL59b1NTU9mCKiVJkiRJkro2AyeOD5yav62udpVTdfUhSg4UNrNCSZIkSZKkrs/ACUjpnx0/ripu2pvqTtQvu24fJ7fVSZIkSZKkxGXgxLEeTgBVu5u3wqnvGZMIIQ2wcbgkSZIkSUpsCRs4lZWVxY+P31K3u1lzJydncMYZFwFw4MBvqazc18wqJUmSJEmSurZ2C5xCCFNDCL8LIWwMIdxXz/VzQwjLQgi/DSG8EUIYXOfabSGEDbG/25pbQ5OahjdzSx1Av35Ht9VF7N37q2bPI0mSJEmS1JW1S+AUQkgG5gHXAmOBGSGEsScMewz4cRRF44F/AB6Jfbcf8CBwMTAJeDCE0Lc5daSmppKamgqcEDhl1+nh1MwtdQDZ/er2cXJbnSRJkiRJSkzttcJpErAxiqJNURQdAeYDN54wZizwy9jx63WuTwFejaJobxRF+4BXganNLeToKqe6gVNSRgZJmZlA899SB5CVNZrU1NrwqnjvCqIoavZckiRJkiRJXVV7BU6DgI/rfN4WO1fXe8CnY8fTgF4hhOwmfpcQwl+HEN4NIby7e3fDfZjqC5wAkmNvqmvuW+pqa0giu9/lAFRU7KSs7KNmzyVJkiRJktRVdaam4X8LfDKE8Bvgk8B2oLqpX46i6PtRFE2IomjCgAEDGhzXUOCU0r/2Oy1Z4QR1+zjVrnKSJEmSJElKNO0VOG0HhtT5PDh2Li6Koh1RFH06iqILgL+PndvflO+ejqOBU1VVFZWVlfHzRxuHR2Vl1JSWNnd6+vWbHD+2j5MkSZIkSUpE7RU4vQOMCCEMCyGkAdOBl+oOCCH0DyEcred+4NnY8SvANSGEvrFm4dfEzjVLg2+qq9s4vAXb6tLTB5KVNRqAfft+TU1NRbPnkiRJkiRJ6oraJXCKoqgK+Cq1QdF64D+jKFobQviHEEJBbNifAL8LIXwInAn8c+y7e4F/pDa0egf4h9i5ZmkwcBrQP37c8m11tX2camrK2b9/dYvmkiRJkiRJ6mpS2utGURQtAhadcO6BOscvAi828N1nObbiqUUaCpyS665wamHglN3vSrZu/QFQu62uX7/LWjSfJEmSJElSV9KZmoa3i7qBU1lZWfz4aNNwaHng1KfPBJKS0gH7OEmSJEmSpMST0IFTQ1vqqlsYOCUnp3PGGZMAOHhoLRVHWjafJEmSJElSV5JwgVNGRkb8uMGm4Xua3zT8qOx+V8aP9+19q8XzSZIkSZIkdRUJFzg12MOpf+s1DYdjjcMBiveuaPF8kiRJkiRJXYWBU0xSWhpJffoArRM4ZWaOID3tTKC2j1MURS2eU5IkSZIkqSswcKrj6La6lvZwAgghxFc5HTmym0Olv2vxnJIkSZIkSV1BwgVOqamppKSkAPUETrFtdVV79rTKiqR+2VfEj/e6rU6SJEmSJCWIhAuc4Ngqp4YCp+jIEWoOHmzxffr1nQwEAPYWr2zxfJIkSZIkSV2BgVMdyf1b9011aWn96NUrF4D9Jauori4/xTckSZIkSZK6PgOnOlL6D4gfV+3Z3Sr36tevdltdTc0R9u9/p1XmlCRJkiRJ6swSOnCqrKyksrIyfv7oljponcbhANn96vZxcludJEmSJEnq/hIycMrIyIgf113llNLKW+oA+vS5gOTk2vsV2zhckiRJkiQlgIQMnI6ucIITA6djK5yqWmmFU1JSGn3PuASA0tIPOVzxh1aZV5IkSZIkqbMycKoTOCW3QeAE0C/bbXWSJEmSJClxGDjVXeHUrx+EAEBVcesFTnX7OO3c8SIHD64liqJWm1+SJEmSJKkzMXCqEziFlBSS+/YFoHp36wVOVdWHCSEVgP0l77DqnQLeeedGDh5c12r3kCRJkiRJ6iwMnOoETnCsj1Nrbak7fHgHhYUziaLK484fPLSWNb+ZSXn5tla5jyRJkiRJUmdh4HRS4FT7prqqvXuJampafK+PP36Oysr99V6rqiph68fPtvgekiRJkiRJnYmB0wmBU7xxeFUV1SUlLb7XnuLXG71efIrrkiRJkiRJXY2BU53AqfrgQSp37Ih/3vmNb3C4qKhF94qiqsav1zR+XZIkSZIkqatJyMApNTWV5ORkAMrKygCo3LWLLTd/hvJ3V8fHHXptGZtvupkDixY1+15nnHFx49f7Tmr23JIkSZIkSZ1RQgZOIQQyMjKAYyuc/vjwIxz5/e9PHlxdzY77/46qvXubda9zhvxl/A11J9eRwjlDZjVrXkmSJEmSpM4qIQMnOLatrry8nKq9ezn46qsNjo0qKih56aVm3ScraxTj854mNbXvSddGj3qEXr3GNmteSZIkSZKkzsrAqbycqj/8AaqrGx1f+fG2Zt+rf/8/ZfJlK8nNfZzs7Kvi52tqyhv5liRJkiRJUteU8IFTZWUlUd+TVx+dKGXgwBbdLzm5B584888ZPfof4+f+uOvlFs0pSZIkSZLUGSV84ARQlZVF5hVXNDw4OZk+BX/eKvftkf4J+vS5CID9+1dRUbG7VeaVJEmSJEnqLAycqN1W94m//zuSs7PrHXvm3K+RetZZrXbvMwdeFzuK2LV7SavNK0mSJEmS1BkYOFEbOKUNHcqwF/8/+n7+84Q61z7xzW/S7wtfaNV7Dxg4FQgA7Nq1qFXnliRJkiRJ6mgGTtQGTgCpZ53FJ/7+7zjza3Pi15Kb0N/pdPVI/wRn9JkAwP7971BRsavV7yFJkiRJktRRDJw4FjgdlTZsePz4yOZNbXL/gQOvjR25rU6SJEmSJHUvCRs4ZWRkxI/LysqOu5Y2fFj8uGJTWwVOdbbV/dFtdZIkSZIkqftI2MCpsRVOKQMGkJSVBcCRTZvb5P7p6WdyxhkTAdhf8i6HK/7QJveRJEmSJElqbwZOnBw4hRBIG167re7Ipk1EUdQmNQys87a63bvcVidJkiRJkroHAydODpwA0ocNBaCmrIyqXW3T1HvggGPb6v64a3Gb3EOSJEmSJKm9JWzglJqaSnJyMlB/4HRc4/A26uOUnj6AM86YBECJ2+okSZIkSVI3kbCBUwghvsqp3sCpbuPwzW3TxwngzPi2OtjlKidJkiRJktQNJGzgBDQaOKUPr7vCqe0CpwEDpnD0f8OuXb6tTpIkSZIkdX0GTtQfOKWecw7Etty11ZY6OLqtrvZtdSUlazh8eGeb3UuSJEmS/n/27jy87rLO///zPic52dMmTUv3jS5A2UFwAVzQERl1GEQFR/i6DDIjrjiLouN3fs44zlcdL2ccHFHHkXFEx0FcUVBxARwFSgulpRTolu5N0qZJmvWc8/n9kZM0LU1y2pwly/NxXefqJ5/7Pp/POy3+4eu67/ctSYVg4AT09vaSTCaPGoslEiTmzwfyu6UO4JRZfzh4vb/J0+okSZIkSdLEZuCU0d3d/bzxRGZbXXLPHtKHD+etjpmzhmyr23dP3t4jSZIkSZJUCFM6cKqsrBy87uzsfN74UY3Dt23LWx1liQbq6i4G4FDbWrq7d+ftXZIkSZIkSfk2pQOnoSucitk4HGCWp9VJkiRJkqRJwsAp43iBU2LJkRVOvVvz1zgcYNbMP2Dgn2OfgZMkSZIkSZrADJwyRguc8t04PJFooK7uhQC0ta2lq2tXXt8nSZIkSZKULwZOGccLnErq6ojX1QH531IHcMrQbXVNrnKSJEmSJEkTk4FTxvECJzhyUl3vtm1EqVRe65k58w8IIQ7A/v0/yeu7JEmSJEmS8sXAKWO4wKksc1Jd1NND3549ea0nkZhB3fSBbXVP0NW1M6/vkyRJkiRJygcDp4xhVzgtGXpSXX4bh4On1UmSJEmSpIlvSgdOiUSCWKz/r2D4LXVDGocXIHBqaHglA/8sz23+Rx548EI2PfMJenqb8/5uSZIkSZKkXJjSgVMIYXCV0/Bb6oaucMpv4/Aoiti8+dNAevBeX99Bdu68g9Wrr6Gnpymv75ckSZIkScqFKR04AVRWVgLQ2dl53PHSefMIpaVA/rfUHTz4v+zZ+93jjnV372DL1s/n9f2SJEmSJEm5MOUDp9FWOIV4nMTiRQD0bNuW11r27v3+iOP79v2QKMrvSXmSJEmSJEljZeCUCZx6e3tJpY4f5gw0Dk81N5M6dChvtfSO0qcpleoklTp+MCZJkiRJkjReGDhlc1LdkMbhvVvz18eponLJiOOJxCzi8aq8vV+SJEmSJCkXDJyyCJyGNg7vyWPj8HlzrwXC8OPzriOE4cclSZIkSZLGAwOnbFY4LRlyUt3W/DUOr65ewcoVf3vcsaqq5SxedFPe3i1JkiRJkpQrJcUuoNiyC5yObHXL5wongPnz30rttHPZtfObtLVvoKNjAwCxWAWxWFle3y1JkiRJkpQLrnDKInCKV1dRcsopAPRuyd8KpwG1NWdy+umf4uKLfkh93SUAtLevo719Q97fLUmSJEmSNFYGTlkETnCkcXjvjh1EfX15r2vAvHlvGbzetetbBXuvJEmSJEnSyZrygVNlZeXgdWdn57Dzygb6OCWT9O7Yme+yBjU0vIJEYiYAe/f9kGSyo2DvliRJkiRJOhlTPnDKeoXTkD5O+WwcfqxYrJS5c94IQCp1mH37flSwd0uSJEmSJJ0MA6cT3FIH0FOAPk5DzchMEUkAACAASURBVJ17LRCA/m11URQV9P2SJEmSJEknYsoHTolEglis/69hpMCpbOnSwevePJ9Ud6yKinnMmPFSANo7NtDe/mRB3y9JkiRJknQipnzgFEIYXOU0UuBUcsophEy/p0KcVHeseXOvG7y2ebgkSZIkSRrPpnzgBGQVOIVYjLLFiwHo2bq14NvaZsx4GWVlswHYu+9HJJPtBX2/JEmSJElStgycyC5wAkhkttWl29pItbTkva6hYrES5s59c//7013s2fv9gr5fkiRJkiQpWwZOHAmcenp6SKVSw84rZuNwIHNaXf8/2W6bh0uSJEmSpHHKwImjT6rr7u4edt5RjcO3bstnScdVXj6HhoZXANBxeBOH2tYUvAZJkiRJkqTRGDhxdOA00ra6xJKhJ9UVfoUTwLx5Ng+XJEmSJEnjm4ETUJk5fQ6gs7Nz2HmJRQshBAB6thYncJpRfynl5fMA2L//J/T1tRalDkmSJEmSpOEYOJH9CqdYeTml8/rDnt4tW/Ne1/GEEGfe3GsBSKd72LP3e0WpQ5IkSZIkaTgGTmQfOMGRxuF9u3aRHqHfUz7NmfNGQigB+rfV2TxckiRJkiSNJwZOnFjgVDbQxymK6N2+PZ9lDV9D2UwaGl4JQGfnZlpbHy1KHZIkSZIkScdj4MSJrnAqfuNwgPnz3jJ4vWv3nUWrQ5IkSZIk6VgGTpzgCqfMljqAniIGTnV1L6KiYiEA+/ffR2/vgaLVIkmSJEmSNJSBE2NY4bR1W75KGlUIscHm4VHUyzPP/h1NTb8gmTxctJokSZIkSZLAwAmAsrIyQgjA6IFTvL6e2LRpQHG31AHMnHkF0F/3vn0/ZN2TN/HQb19MY+O/20hckiRJkiQVjYETEEIYXOU0WuAUQqBsSf+2up6tW4nS6bzXdzxRFPH007cCRwdLqVQHzz73D+za/a2i1CVJkiRJkmTglFFZWQmMHjjBkW11UVcXyX378lrXcFpbH+Zg6++HHd+27TbS6WQBK5IkSZIkSepn4JQxsMKps7Nz1LmJJYsHr4vVOLzlwIMjjvf07OXw4WcKVI0kSZIkSdIRBk4ZA4FTT08PqVRqxLllQxuHb9ma17qGFY2+lS+iONv9JEmSJEnS1GbglDH0pLru7u4R5yaWDD2prjgrnOrqXjTieGnpDKqrlheoGkmSJEmSpCMMnDKGBk6j9XFKLJgPJSUA9BRphVN9/SXU1pw97PiiRTcSi5UVsCJJkiRJkqR+Bk4ZJxI4hdJSEgsXAtC7tTiBUwgxzjnnK9RNf+HzxmKxcubNfUsRqpIkSZIkSTJwGnQigRNAYukSAJL79pHqOJy3ukasIdHA+ed/k4te8ENWrvgE9fWXApBOd7N7z3eKUpMkSZIkSZKBU8aJBk5lR/VxKlLj8IyamlXMn/8nnH7apwihFIDGxq+STvcWtS5JkiRJkjQ1GThlnPgKp+I3Dj9Wefkc5sz+YwB6evayd+8PilyRJEmSJEmaigycMk54hVNmSx1Az5bxETgBLFr0Lgb+Wbc33k4UpYpbkCRJkiRJmnIMnDIqKysHr7Na4bTkSODUW6ST6o6nsnIJs2ZdAUBn51b2N/2syBVJkiRJkqSpxsApY+gKp87OzlHnx2triTc0AONnS92AxYv+bPB6+7Z/I4qiIlYjSZIkSZKmGgOnjLKyMkIIQHYrnADKMqucerZuI3nwYN5qO1E1NauYUX8ZAO0dGzhw4MEiVyRJkiRJkqYSA6eMEMLgKqdsAqfup56iZ1tmK10yybOXvZTdt36UVHt7PsvM2qJFfz54vW37l4pYiSRJkiRJmmoMnIbINnDqefZZtl9/A6mm5iM3+/o4dPfdNL7zT0n39uazzKxMn/4Cpk07H4DW1oc5dGhNkSuSJEmSJElThYHTENkGTk23fZH04cPHHetet472n/4057WdqBACixe9e/BnVzlJkiRJkqRCMXAaYiBw6u7uJp1OH3dOlErRfv/9Iz6n7ec/z3ltJ2PGjJdRXX0aAM3N99PRsanIFUmSJEmSpKnAwGmIoSfVdXd3H3dOlEpBX9+Iz4k6s2s6nm8hBBYtvGnwZ1c5SZIkSZKkQjBwGmJo4DTctrpYIkHZypUjPqf87LNyWtdYzJp1JRXlCwHYt+/HdHU1FrkiSZIkSZI02Rk4DZFN4ARQ/7a3DTsWysupe/Obc1nWmMRiJSxcdGPmpzTbG79S1HokSZIkSdLkV7DAKYRwRQhhUwjhuRDCh48zvjCE8KsQwtoQwroQwpWZ+4tDCF0hhMczn7ztC6usrBy8HilwmnbVHzHjppuedz8kEsz/13+ldM6cvNR3subMfgOJxEwA9uy5i56epiJXJEmSJEmSJrOCBE4hhDhwG/Aa4AzguhDCGcdM+xjwnSiKzgOuBb44ZGxzFEXnZj5/lq86h65w6uzsHHZeCIFZH/wAp977U+pvvHHwfsUFF1B9yUvyVd5Ji8fLWLjgHQCk0708uvqPeWzNW9i27d/o7T1Q5OokSZIkSdJkU6gVThcBz0VRtCWKol7g28AfHTMnAmoz19OA3QWqbVC2W+oGJBYv5pQP3ULZ8mUAdK9fTzTM6XbFNn36C4AAQE/PHlpbH2bzls/y8CNX0nH42eIWJ0mSJEmSJpVCBU7zgB1Dft6ZuTfU3wJvDSHsBH4CvHfI2JLMVrvfhBAuPd4LQgjvCiGsDiGsbmo6uS1jJxo4DX7v3PMASLe307t580m9O5+iKGLj0x+hP9M7Wm9vExs2fJAoev6YJEmSJEnSyRhPTcOvA74eRdF84ErgGyGEGLAHWJjZancLcGcIofbYL0dR9OUoii6MoujCmTNnnlQBJx04nXfe4HXn2rUn9e58am19lMMjrGLq6NhIW9v4q1uSJEmSJE1MhQqcdgELhvw8P3NvqHcC3wGIouh3QDnQEEVRTxRFLZn7jwGbgRX5KPJkA6fK848ETl1rxl9w09m1dfQ5naPPkSRJkiRJykahAqdHgeUhhCUhhAT9TcF/eMycRuBygBDC6fQHTk0hhJmZpuOEEJYCy4Et+SiyrKyMEPr7HJ1I4FS6aBHxurr+743DFU6J0vpR55QmZhSgEkmSJEmSNBUUJHCKoigJvAe4D9hI/2l0G0IInwghvD4z7UPAjSGEJ4BvAW+L+hsLXQasCyE8DtwF/FkURXk5Wi0Wi1FeXg6cWOAUQhjcVte7fTvJA+Pr5Lf6+ssoHSF0SiRmUl/34gJWJEmSJEmSJrOSQr0oiqKf0N8MfOi9jw+5fgp4yXG+913gu3kvMKOiooKurq4TCpygf1tdxy9/CfSvcqq5/PJ8lHdS4vEyTlv5dzy5/n1A6nnjS5d8kFgsUfjCJEmSJEnSpDSemoaPC5WVlcCJrXCCoxuHj8dtdbNmXcH553+TGTNeSgilQHxwrPXQI8UrTJIkSZIkTToFW+E0UQw0Du/q6iKdThOLZZfJla9aBaWl0NdH59rH81niSaub/gLqpr+AKIpIp3v4/cN/QHf3Lvbu/QELF/wpNTWnF7tESZIkSZI0CbjC6RhDT6rr7u7O+nux8nLKz+gPbLqffJKotzfnteVKCIF4vJylS2/J3InYvPnTRa1JkiRJkiRNHgZOxxgaOJ1wH6fzzgcg6u2l+6mnclpXPsw+5fVUV58BQMuBBzhw4H+LXJEkSZIkSZoMTipwCiFcGkJ4XoPvyWDglDqA1tbWE/ru0D5O43Vb3VAhxFh26l8N/vzc5v9HFKWLWJEkSZIkSZoMsgqcQgi/DiFckrn+C+Bu4K4Qwl/ns7hCW79+PQ8//PDgz9/85jf58Y9/TF9fX1bfrzjv3MHr8dg4/Hjq6y+hru7FALS3r2ff/nuKXJEkSZIkSZrosl3hdBbw+8z1TcDLgIuBd+ehpqJ46qmnuOuuu47q25ROp1m9ejX/8z//QxRFoz6jdNYsSufPB6Bz7ZqsvlNsIYSjVjlt2fw50unx239KkiRJkiSNf9kGTjEgHUJYCpREUbQhiqJGoD5/pRVOOp3m/vvvH3b8mWeeYceOHVk9a2BbXaqpmb5du3JSX77V1p7FKae8DoCu7kZ27fpWkSuSJEmSJEkTWbaB0/8Cnwc+DXwPIBM+teSproJqaWmhpWXkX2XTpk1ZPWsibqsDOHXphwihFICt2/6VZLK9yBVJkiRJkqSJKtvA6W1AN7AJ+Hjm3hnAF/JQU8GlUqlR5ySTyayeVTmkcfhECpwqKhYwf96fANDXd4DtjV8pckWSJEmSJGmiyipwiqKoKYqiv4qi6KNRFHVk7v04iqJ/ym95hTFjxgwqKipGnLNgwYKsnlW2YgWxykoAOtdMnMAJYPHidxOPVwPQ2Pg1enr2F7kiSZIkSZI0EWV7St37QwjnZq4vCiFsCSE8G0K4KL/lFUZpaSkXX3zxsOP19fWcdtppWT0rxONUnHsOAD3PPEOqoyMnNRZCIjGDRYveBUA63cXWrf9S5IokSZIkSdJElO2Wug8B2zLX/wjcBnwWmDSJxGWXXcYFF1zwvPv19fW89a1vpaSkJOtnVZyb2VaXTtO9bl2uSiyIhQveTiIxC4Dde75DS8sDdHRsIpXqKXJlkiRJkiRposg2cJoeRVFrCKEaOBf4fBRFtwPZLfuZAGKxGK973et4z3vew+LFiwfvX3311dTXn9hhfBVD+jhNtG118XglS5e8H4AoSvH4E2/n4Ueu5KHfvpgtW/6ZKBq935UkSZIkSZrasg2cdoYQLgbeBDwYRVEqhFADTLr0oaGhgXPOOWfw5wMHDpzwMyrOPQdCACZW4/ABJaW1z7uXTLayddu/8PTTHytCRZIkSZIkaSLJNnD6K+BHwCeAv8/cey3waD6KKrYZM2YMXre0tJzw9+M1NZQtXw5A1xNPEGVxCt54EUVpNm/+zLDju/d8h47DzxawIkmSJEmSNNFke0rdj6MomhVF0fwoigZCpu8BV+WvtOIZa+AER7bVpTs66Hluc07qKoSOjk10dTWOOKdp/30FqkaSJEmSJE1E2a5wIoSwNIRwawjhthDCrcC8KIq681hb0VRWVlJeXg6cfOBUef6RPk5da9fkpK5CSKU7s5jTVYBKJEmSJEnSRJVV4BRCuBJ4gv6G4Z3AOcDaEMIf5rG2ogkhDK5yamlpIYqiE37G0MbhE6mPU1XlcmKx8hHn1NacXaBqJEmSJEnSRJTtCqdPAVdFUfSmKIr+MoqiN9O/ne5T+SutuAYCp97eXjo6Ok74+6ULFhDPPKNz7eM5rS2fSktrmTv3jcOOl5XNoaHh8gJWJEmSJEmSJppsA6eFwK+PufdA5v6k1NDQMHh9MtvqQghUnHcuAH2NjSSbm3NWW74tO/UjNDS88rhjpaXTCSHrnZiSJEmSJGkKyjY5eAL4wDH33gesy20540cuGodXnnf+4HXnBNpWF4+XcfZZX+KCC77DwoU3Mm/eW6msWAJAR8dGdu3+dpErlCRJkiRJ41lJlvPeDfw4hPABoBFYACSB1+ersGLL5Ul1AF1rH6f2Va8ac12FEkJg+rQLmD7tAqD/9LpHHn09UZRk8+ZPM7PhVZSVzSxylZIkSZIkaTzKaoVTFEVPASuB64HbgBuA06IoWp/H2oqqvr5+8PpkA6fyVWcQSkuBidU4/Hiqq1eycME7AUgm23n2uU8WuSJJkiRJkjReZd2MJ4qiviiKfh1F0Z1RFP0aIISwJW+VFVkikaC2thaA5pPsvxQrK6N81SoAutevJ93Tk7P6imHJkvdSXj4fgH37fkRLy4NFrkiSJEmSJI1HY+n+HIDFOapjXBrYVnfw4EFSqdRJPaPi/P4+TlFfH90bnspZbcUQj1ewcsXfDv686ZmPk0p1F68gSZIkSZI0Lo31uLEoJ1WMUwOBUzqdprW19aSeMXBSHUz8bXUADQ0vZ+bMKwDo6mpk2/Z/K3JFkiRJkiRpvPF8+xHk5qS6IY3DH5/4gRPAihV/QzxeDcD27bdz+PDmIlckSZIkSZLGkxFPqQshfO1kvzsZ5CJwKmlooHThQvoaG+lcs5Yoiggh5KrEoigvm82pSz/IM8/+HVHUx9Ob/obzz/vmhP+9JEmSJElSboy2wmnXCJ/twD/ktboiy0XgBFCZ2VaXammhb8eOMdc1Hsyffz01NWcC0Nr6MHv33l3kiiRJkiRJ0ngx4iqlKIr+plCFjEfTp08nFouRTqfHFDhVnHceh37wQ6C/j1Ni4cJclVg0IcQ5beXf8+jqq4E0zz73j1RVrQQiKisXU1JSU+wSJUmSJElSkdjDaQTxeJy6ujpgbCucKob0cepcMzn6OAHU1p7F/PnXA9DXd4BHV/8Rj66+igcfupiNT99KMtlR5AolSZIkSVIxGDiNYmBbXVtbG729vSf1jLJly4hV9zfZ7nz4YVKHDuWsvmKbNeuK591Lp3vYvfu/efyJd5JOJ4tQlSRJkiRJKiYDp1E0NDQMXh84cOCknpFsbiGUlwPQu20bz7z4Jez8wAfp27s3JzUW0/bttw87dujQapqbf1HAaiRJkiRJ0nhg4DSKsTYOT7W2sv3660k1Nw+5maL93nvZ/tbrSR48mIsyiyKV6qKl5TcjztnfdG+BqpEkSZIkSePFiE3DB4QQbhhmqAfYCTwSRVFfzqoaR8YaOB345jfpa2w87ljfzp0c/MY3mPm+9510fcWUTvcC0YhzUqmuwhQjSZIkSZLGjawCJ+BdwAuAFvoDpnlAA7AWWAz0hhCuiqJoTT6KHIttXT18aUcTvznQRjqCS+qquWnBLFZUlWf1/bEGTu33/WzE8bZ775uwgVNJSS0VFYvo6to+7JxptecWsCJJkiRJkjQeZLulbg3w4SiK5kZRdFEURfOAvwYeBuYCXwO+kKcaT9qaQ4d51aOb+PquZrZ29bK9u5dv7jnAH6zexEMH27N6RnV1NYlEAoDmodvispTu7BzT+HgWQmDhwj8dYbyUuXPfWMCKJEmSJEnSeJBt4HQ98C/H3PsCcEMURWngU8CqXBY2Vuko4r0bG2lPpZ831p2OeM9TjfSlR94OBv2hysAqp5aWFqJo9O8MVb5q5L+W0cbHu3lzr2PhgncedyyK+mjveLrAFUmSJEmSpGLLNnDaD7zmmHtXAE2Z63IglauicuGRQ4fZ3NUz7Pje3j5+daAtq2cNBE7d3d10nuCKpPobrh95/PqRx8e7EALLl9/KCy/+OUuWvJ/58/8Pc+deNzi+8am/oq+vtYgVSpIkSZKkQsu2h9MHgP8OIawFdgALgPOAN2fGXwh8MfflnbzdPaP3MN+VxRx4fh+nqqqqrOuovOACZv/fj7P37z8JqaMzuVkf/jBVL7w462eNZ1VVS1m65EgvqnSqi737vk9P7z6e3vRxzlz1z4QQilihJEmSJEkqlKxWOEVR9FNgGfB1YCNwB7Asc58oiu6Louhv8lXkyZhbVjrqnHlZzIGxNw6vu+46lv38ZzS8770klp06eL/8tNNO+FkTxcqVf0t52VwA9u+/h337flTkiiRJkiRJUqFku6WOKIr2R1H0H1EUfTKKoq9FUbQ/n4WN1UXTqji1omzY8dmJUl5eX5vVs8YaOAGUzp3LzHe/m9kf/ejgvfb77z+pZ00EJSU1nHHGZ4D+VU2bnvk43d27i1uUJEmSJEkqiKwCpxDCohDCf4YQ1oUQtgz95LvAkxULgS+cvpDq+PN/xfJY4F/PWEhpLLstXrkInAZUXnghsdr+oKv9/l+ccBPyiaSu7oUsXPAOAJLJdp7a+Ff095iXJEmSJEmTWbYrnO4EEsBHgRuP+Yxb50+r4hcvWMl1s+sH7y2tKONnF67kkrqarJ9TXl4+2LdprIFTKC2l+qUvBSC5ew89T0/uU9yWLv0QVVUrADh48Hfs2HlHkSuSJEmSJEn5lm3gdBbwJ1EU/SiKovuHfvJZXC4srijj0ysXDP68oqqMFVXlJ/ycgVVOLS0tpNNjW6VTc/krBq/bfzHu/wrHJB4vY9UZnyOEBACbN3+ajo5NRa5KkiRJkiTlU7aB00PA2fksJJ9KY4GKWP+v2pY8ubBoIHBKpVK0tbWNqZ6qSy4llPY3LJ/MfZwG1NSczqlLPwhAOt3Lhqc+RHf3Xjo6NtHXd6jI1UmSJEmSpFwryXLes8B9IYS7gL1DB6Io+kTOq8qDaSVxunrTtCVTJ/X9hoaGweuWlhamT59+0rXEq6uofPGLOPybB+h5+ml6d+4iMX/eST9vIli48J00t/yK1tZH6OjYyG//9yUAhFDCrJlXsHz5xygrm1nkKiVJkiRJUi5ku8KpHrgPqAGWD/ksy1NdOVdbEgfg0EkGTrlsHA5Q84rLB687fjn5VzmFEGfZso8wcGrdgChKsm//j3lszbX09Y1t5ZgkSZIkSRofsgqcoii6fpjPDfkuMFdqS/p/1fZxEzi9HEJ/+NJ+/y/H/LyJoGn/T4Hjn8rX1bWNXbvuLGxBkiRJkiQpL4YNnEII84dcLxzuU5gyx25ghVNbMkU6On7oMZK6ujpCJiBqbm4ecz0lM2dScc45AHSuXk3y4MExP3O827//3pHHm35SoEokSZIkSVI+jbTCaeOQ623A1syfQz9b81JVHkzLBE5p4HDqxBuHl5SUDPZtysUKJ4DqgdPqUik6fvObnDxzPEumDo84nkp1FqgSSZIkSZKUTyMFTtOGXJcCicyfQz+J/JWWWzWZwAk46cbhA9vqWltbSSaTY6/p8lcOXndMgW11tbVnjTheUzPyuCRJkiRJmhiGDZyiKEoPuU4N9ylMmWM3LYeBE8CBAwfGXFPZ0iUkli4FoOOhh0h3d4/5mePZggVvH3F8zuyrC1SJJEmSJEnKp6yahocQFoUQ/jOEsC6EsGXoJ98F5krtkMBpvJxUB1CT2VYXdXVx+H9/l5Nnjlcz6i9h+fKPMdx/djt2fI0hOackSZIkSZqgSrKcdyewA/goMCEb7eR6hVPuAqfLafnKVwFo/+X9/afXTWILF7ydmQ2vYu/e79Hds5dEaQN7991Nd/duWg48wNZtt7F0yXuLXaYkSZIkSRqDbAOns4DLJtIWumPVjtPAqfzss4nPbCDV1EzHL39FlEoR4vHRvziBVVTMZ8mQUGnWrFez+rFrSKd72Lr1n5lWew4zZlxWxAolSZIkSdJYZLWlDngIODufheRbLrbU1dbWUlLSn9HlKnAKsRg1L+/fVpc6cICuxx/PyXMnkpqaM1i58hOZnyLWb/ggXV27ilqTJEmSJEk6edkGTs8C94UQvhhC+PjQTz6Ly6VcbKmLxWLU19cDuQucAGpeefngdfsUOK3ueObOuYa5c98MQDLZypPrbyad7ilyVZIkSZIk6WRkGzjVA/cBNcDyIZ9leaor52pyEDjBkW11hw8fpqura8x1AVS+8IXEKisBaL//F0RRlJPnTjQrlv9famrOBKC9/Umeefbvi1yRJEmSJEk6GVkFTlEUXT/M54Z8F5gruVjhBNDQ0DB4feDAgTHVNCCWSFB1WX/Por7tjfRu3pyT50408XgZZ515GyUl0wDYtetO9uy5u8hVSZIkSZKkEzVs4BRCmD/keuFwn8KUOXa56OEE+WkcDv2n1Q1o/8X9OXvuRFNRMZ9Vqz4HBACe3vRR1j15M4+teQvrnryZpqafE0Xp4hYpSZIkSZJGNNIKp41DrrcBWzN/Dv1szUtVeVAeC5SG/hCjPXnygUW+Aqfql14GmYbk7fdP3cAJoGHGy1iy+D0ApNO9NDXdS2vrwzQ13cu6J/+MJ9e/l3Q6WeQqJUmSJEnScEYKnKYNuS4FEpk/h34S+Sstt0IIg6uccrXCqbm5ecx1DYjX1lJ10QsA6H7ySfr27cvZsyeiGQ0vH3asqeledu68o4DVSJIkSZKkEzFs4BQN2bcURVFquE9hysyNgT5OY+nhVFlZSUVFBZDbFU4A1UO21XX8cmqeVjdg9+7vjDi+c9d/FagSSZIkSZJ0orJqGh5CiIcQ3h1C+O8Qwv0hhF8OfPJdYC7VlPT/umMJnODIKqeWlpacnihX84pXDF5P5T5OAJ2dW0Yc7+pqZILlnZIkSZIkTRlZBU7A54D3AY8AFwP3APOBh/JUV14MXeE0lqBoIHDq6+ujvb09J7UBlM6ZQ/mqVQAcfuQRUjl89kRTWlo/4nj/SXbZ/ucrSZIkSZIKKdv/x34NcEUURf8EpDJ//hFwWd4qy4OBHk69UUR3euyBE+RjW11mlVNfHx0PPJDTZ08kc2ZfNeJ4Q8MrCZkm8JIkSZIkaXzJNnCqBLZnrjtDCBVRFG0Ezs9PWfkxsMIJoD1HjcNzHTjVXP7KweuD3/o2XY8/TpQ++VP1JqqGhsuZOfPVw463tT1BMjl1V4BJkiRJkjSeZRs4PQ1cmLl+DPh4COHDwO68VJUnNUMCp1ydVJfrwCleN51QXg5A1+rVbLv2Oja/5jV0PDihdi+OWQgxzlz1LyxfdisVFQsBiMdrKS3t/7vv7HyO9evfRzqdLGaZkiRJkiTpOLINnD4IDCyz+RDwIuCNwJ/lo6h8GbrCaSyNw+vrj/QXymXglO7upvEd7yDq7j7qft/2Rnb8+Z/TuWZtzt41EcRiJSxc+E5e/KJf8YqXP8NLL1vDxRfdQ3nZXABaDjzAM8/+fzlt3C5JkiRJksZu1MAphBAHVgDrAaIo2hRF0cuiKLogiqJf57m+nKrN0QqnRCJBbW0tkNvAqe2ee+h9bvPxB5NJmr/4xZy9a6IJIU4IgbKymZxzzleJx6sB2LXrTnbs+FqRq5MkSZIkSUONGjhF/WfPfyGKop4C1JNXtTla4QRHttUdPHiQVGpszxrQ8etfjzh++Le/Jd3bm5N3TWTV1Ss568x/pT8LhWef+xT7m+4rclWSJEmSJGlAtlvq7gkhXJnXSgogV1vqABoaGgBIp9O0traO6VkDor5R+hFFEeQo3JroZsy4lJUrPpH5KWLDhltoNrhz7AAAIABJREFUa1tX1JokSZIkSVK/kiznxYC7QwgPATuAwaY5URS9Ix+F5UOuttTB8xuHD/35ZFVeeMGIq5zKV60iVlEx5vdMFvPmXUtX13a2N36ZdLqbx594JzNnvorOzm3E45XMmnkFs2e/jlisrNilSpIkSZI0pWQbOD0LfCafhRTC0MCpPceBUy5Me8MbaPn3r5E6ePD477zxxpy8ZzI59dS/pLOrkaame+nrO8Du3f89ONbS8it27fom5557B6WltUWsUpIkSZKkqWXEwCmEcF0URd+KouhvClVQPuVrhVNzc/OYnjWgpK6OBV/9Crve+z76du8+aqz6FS+n9opX5+Q9k0kIMZYv+yhNTT/jyEGKR7S1r+O55z7F6ad/qvDFSZIkSZI0RY3Ww+n2glRRILns4TRt2jRisf6/vlyeVFexahWn3ncv82/7V2a8612QeUfvtu1EUTTKt6empub7OF7YNGDvvh+QTLYXriBJkiRJkqa40QKnUJAqCqQqHhv8hdqSwwcU2YjH49TX1wO5DZwAQmkpNZdfzqxbPkjNq14FQO+WLXStXZvT90wWnYc3jzieTvfQ3b17xDmSJEmSJCl3Rguc4iGEl4cQXjHcpyBV5kgshMFtdWNd4QQMBk7t7e1s3bo1LyuQpl9zzeB16//clfPnTwalifrR55TWFaASSZIkSZIEozcNLwP+neFXOkXA0pxWlGe1JXEOJVNj7uG0YcMGtm7dOvjzHXfcwZw5c7jmmmtycmLdgKoXv4iSuXNI7t5D2733csqtHyFeU5Oz508Gs0+5im3bbht2vLJiKWVlswpYkSRJkiRJU9toK5wOR1G0NIqiJcN8JlTYBFBb0v8rt6dOPnDaunUrd911F319fUfd37NnD3fccQfd3d1jqnGoEI8z/eo3ABB1ddF2z09y9uzJoqpqKYsX/fmw451dW9i9x9VhkiRJkiQVymiB06QzsKVuLCucHnjggWG3z7W1tbE2x72Wpl/9xxD6F5m13mVwcjxLl36IM874J2pqVgEQi5VTW3vO4PjGjR9h337DOkmSJEmSCmFKNQ2HIyfVdabS9KVPvOdSKpU6aivd8WzePHIT6xNVOncuVZdcAkD3+vV0b9yY0+dPBiEE5sy+iote8ENe8fJneNlL1/OCC+9m2al/nZmRZsOGD9Lc/Kui1ilJkiRJ0lQwYuAURdGkaxY0sMIJctM4vFCOah5+13eLWMn4F0KckFkRtmjRu1i8+GYAoijJk+tv5uDB3xezPEmSJEmSJr0pu6UOTq6PUzweZ/HixSPOWbo0962tal7+MuKZU/EO/ehHpHPYJ2qyW7rkgyyY/zYA0ukenlj3Lg4dery4RUmSJEmSNIlN6cDpZPs4XXrppcOO1dTUcN55553Uc0cSEgmmXXUVAOm2Ntp//vOcv2OyCiGwfPnHmDvnTQCkUodZ+/gNrH7szfzq16v49W/O5sn176W9fUORK5UkSZIkaXKYcoHTtKFb6vpOLnA69dRTufrqq6moqDjqfkVFBTfccMPz7ufK9GveMHjd+j82Dz8RIQROO+3vmTXrD4H+0OnQodWk092kUofZv/8nPLr6Gg4c+N8iVypJkiRJ0sQ35QKnmqGB00lsqRtw9tlnc8stt/DGN76ReLz/mdOnT2fmzJljrnE4ZUuXUnHBBQB0PvIIvdu25e1dk1EIcU5b+XeEUHLc8Sjq5emnP0oUpQtcmSRJkiRJk8uUC5ym5WBL3YDS0lJWrVrFwoULAdi7dy/dee6tdFTz8O/endd3TUYHDjxIFCWHHe/qbqS1dXUBK5IkSZIkafKZ0oHTyW6pO9ZA4BRFETt37szJM4dT++o/IFZdDUDr979H1NeX1/dNNj29TaPO6e1rLkAlkiRJkiRNXlMucMrVlrqhFi1aNHi9ffv2nDxzOLHKSmpfm+lD1NRMxwMP5PV9k01lxeIs5iwadY4kSZIkSRrelAucjlrhNMYtdQPmz59PLNb/V9nY2JiTZ45k+jVvHLy2efiJqa+/lPLyecOOh1AybI8nSZIkSZKUnSkXONXmsIfTgEQiwZw5cwDYuXMnyeTwPYJyoXzVGZSdfjoAHQ88QN++fXl932QSi5Vw5plfoKSk9rjjUZTksTVvoa19fYErkyRJkiRp8phygVNN/Ejg1J6jwAmO9HFKpVLs3r07Z889nhAC0695Q/8P6TSHvve9vL5vsplWew4XX/xTFi96N9Nqz2PatAtZsvh91NW9BIBkspW1a9/KoUNrilypJEmSJEkT05QLnEpjgcp4/6+dqxVOcCRwgsJsq5v22tcSysoAaL3ru0TpdN7fOZmUl83m1FM/xIUX3sWFF/w3S5e+n3PP+SozZ74agGSynbWP/x8OHvx9kSuVJEmSJGnimXKBExzp45SrHk5wdOCU78bhAPFp06i9oj8c6du5k86HH877Oye7WCzBmav+hVNOeT0AqVQnjz/xDlpafkM63cehQ2s5ePARksn2IlcqSZIkSdL4VrDAKYRwRQhhUwjhuRDCh48zvjCE8KsQwtoQwroQwpVDxj6S+d6mEMKrx1rLwLa6XK5wqqqqoqGhAYAdO3aQLsCKo+nXXDN4vesv/pId776Z1ru/R7q3N+/vnqxisRJWnfFZ5s55EwDpdA+PP3EjDz50Masfu4Y1a6/joYdezLPPfYp0uq/I1UqSJEmSND4VJHAKIcSB24DXAGcA14UQzjhm2seA70RRdB5wLfDFzHfPyPy8CrgC+GLmeSdtYIVTezK3odCiRYsA6O7uZv/+/Tl99vGkk0kIAYBUSwsdv/wle269lW3XXkuqtTXv75+sQohz2mmfZP78GzJ3UiSThwbHU+lOGhu/ytObPlacAiVJkiRJGucKtcLpIuC5KIq2RFHUC3wb+KNj5kTAwNFh04CBztt/BHw7iqKeKIq2As9lnnfSaodsqUtH0VgedZRC9nFKdRxm1/s/AMepv+epjez9h3/I6/snuxBiLDv1w8Ri5cPO2bPnLg4f3lLAqiRJkiRJmhgKFTjNA3YM+Xln5t5Qfwu8NYSwE/gJ8N4T+O4JmVbaHzhFQEcqd6ucCtnHqe0n95Buaxth/KckDx7Maw2TXXv7k6TT3SPOaW65v0DVSJIkSZI0cYynpuHXAV+Pomg+cCXwjRBC1vWFEN4VQlgdQljd1NQ04tya+JHH5rJx+PTp06mt7V+k1djYSJTD1VPH6t2ydeQJySR9O3aMPEcjSqd7Rp+TGn2OJEmSJElTTaECp13AgiE/z8/cG+qdwHcAoij6HVAONGT5XaIo+nIURRdGUXThzJkzRyxmoIcT5DZwCiEMrnJqb2+nNY99lOLTp48+p64ub++fCmpqVhGLlY08p/bsAlUjSZIkSdLEUajA6VFgeQhhSQghQX8T8B8eM6cRuBwghHA6/YFTU2betSGEshDCEmA58MhYiqkdEjjl8qQ6ONI4HPK7ra72tX842DD8eCrOPZfEggXDjmt0paXTmTv3TSPO2b79S/T1Db+1UZIkSZKkqagggVMURUngPcB9wEb6T6PbEEL4RAjh9ZlpHwJuDCE8AXwLeFvUbwP9K5+eAu4Fbo6iaEwp0dDAqT3HgVOhGocn5s+n4eabjz8Yi3HKRz+at3dPJcuXfYRTZr32efcHDkpsbX2Yx9a8me7uPYUuTZIkSZKkcaukUC+Kougn9DcDH3rv40OunwJeMsx3Pwl8Mle15HOF08yZMykvL6e7uzvvjcNnvudmEosW0vIf/0HPUxuPDKTThPh4as81ccViZZx55j+zuOPdNDffTzrdx/TpF1JWNpcnnngHXd2NHD78DKsfu4Zzz/ka1dUri12yJEmSJElFNyVTiWl5DJxisdjgKqeWlhY6Ojpy+vxjTXvd61h6992ctu4J5n7uc4P3m790e17fO9VUV69k8eJ3s3Tp+6mvfwlVVUu44ML/oabmTAB6evby2Jo3c/Dgw6TTvRw48Fv27f8pnZ2jNHeXJEmSJGkSmpKBUz631MHR2+p2FOikuJBIUHvFq0mceioA7T/7GT3PPVeQd09VZYkGzj/vTmbMeCkAyWQ7ax+/ngcevIi1j9/A+vXv4Xe/fyWPP/EOenubi1ytJEmSJEmFM+UDp1yvcILCNQ4/VojFaLjpXYM/N3/5ywV791RVUlLF2Wfdzpw51wAQRSlSqfaj5rS0/Ia1j7+NdLq3GCVKkiRJklRwUzJwGrqlri0PgdOcOXMoKelvj5XPxuHHU3vllZRmTqdr+/E99Bb4/VNRLFbKaSs/RSIxc9g5HR0baWr6eQGrkiRJkiSpeKZk4FST5xVOJSUlzJ8/H4A9e/bQ09OT83cMJ5SUMOPGP+3/IZ2m5StfKdi7p7Jk8gC9vU0jzmlp+XVhipEkSZIkqcimZOBUHgskQgDy08MJjvRxiqKInTt35uUdw5l21VWUzJ4NQOv3f0Df7t0Fff9UFEXpnMyRJEmSJGkymJKBUwhhsI9TPlY4wdGNwwvZxwkglkgw453v7P+hr4+Wf/9aQd8/FSUSM6msXDrKnIYCVSNJkiRJUnFNycAJjvRxykcPJ4AFCxYQMquoCt3HCWD6G68hPmMGAK133UWyaeTtXhqbEAKLF/35iHMad/wHO3b+J1EUFagqSZIkSZKKY8oGTjWDgVN+tjmVlZUxO7OtbefOnSSTyby8Zzix8nJmvP1tAEQ9PbR8/esFff9UNGfO1Sxb9mFisbKj7peUTMtcpXjmmf+Pp5++lXS6cH29JEmSJEkqtCkbOA1d4ZSvFSeLFi0CIJlMsmfPnry8YyTTr72O2LT+sOPgt75N8uDBgtcw1SxaeCOXvOS3nH7a/2P58o9x/nnf5NJLVrNi+d8QQv9/c7v3fIc1a99KT0//qrPOzu20tq6mu2dvMUuXJEmSJClnpmzgNNDDqS+K6ErnJ3Aa2sepGNvq4tVV1N9wPQBRZycHv/GNgtcwFZWW1jF37jUsXPB26upeSCwWY8GCt3HuOf8xuNrp0KE1PPLoa/n9w6/ld79/BY+teTO//e0lPPHEjXR3Fz6clCRJkiQpl6Zw4HTkV8/3SXVQ+MbhA+rf+lZiVVUAHPivb5Jqby9KHYL6+pfwggu/R1XVcgB6e5s5fHjjkBkRzS2/ZM2at9DX11acIiVJkiRJyoEpHDjFB6/zdVJddXU1MzKNuxsbG0mn89MvaiTxadOoe8tbAEi3tXHwzm8VvAYdUVm5iAsvuIvy8nnDzunqbmT37m8XsCpJkiRJknJrygZO04YETvk6qQ6OrHLq7u6mubk5b+8ZSf3b/g+hvByAA1//OunOzqLUoX4lJdUkk10jzmlq+lmBqpEkSZIkKfembOBUU6DAaaBxOBRvW13JjBnUvflNAKQOHmT7DTew+68/zKEf/pB0b29Raprqomjkv/d02n8XSZIkSdLENWUDp0KvcILiNA4fUHvllYPX3es3cOgHP2D3X/01W//4avr27StaXVPVtGnnjTgej1fl7fRESZIkSZLybcoGToXo4QRQV1dHdXU1ULwVTlEUse8fPnXcsd7Nm9n9ob8ocEVauPBPRxxvPfQITz31F6RSbn+UJEmSJE08Bk7kd4VTCGFwW11bWxutra15e9dwutevp+uJJ4Yd71y9mu6nny5gRZpRfwkrV3yCEEqOuh9CKRAA2Lvv+zy6+g10dm4tQoWSJEmSJJ28ktGnTE6F2lIH/dvqNmzYAPSvcpo+fXpe33esbMKk7qefpvy00wpQjQbMn/8nzJz1avbt+xG9PfupqFjIKae8lvb2Dazf8H56e5s5fPgZHnn0Ks44/dNMm3Y+e/beTVfnNhJlM5l9ylVUVS0t9q8hSZIkSdLzTNnAqVBb6uDoxuGNjY2cc845eX3fseI1NaPPyWz7U2GVJRpYuODtR92rq3shF73gRzy5/r0cOrSaVKqDJ9e/m/4FienBedu23caSJR9g6ZL3FrZoSZIkSZJG4ZY68r/CadasWZSVlQGwadMmnnnmGVKp/L5zqKpLLiVWVTXseGxaLVUveUnB6tHoyspmcf55/8XCBe8ccjf9vHlbt36efft/WrjCJEmSJEnKwpQNnKriscFfPt+B0/bt2wcDpo6ODu68804+//nPs2XLlry+d0C8uopZfzl8Y/C6t/wJsYqKgtSi7MVipSxffiv19ZeOOG/Hjv8oUEWSJEmSJGVnygZOsRAGVznlM3Bqbm7mzjvvJJlMHnW/vb2dO++8k/379+ft3UPVXXst8z73T5QtX/68sc6HHyaKooLUoROX7Ds04nh7+5MFqkSSJEmSpOxM2cAJjmyry2cPp9/97nf09fUddyyZTPK73/0ub+8+Vu2VV7Lkhz9g+YMPsOyB31Bx7rkAdK1ZQ9uPflSwOnRi4vHKEcdjsbICVSJJkiRJUnYMnID25PN74+TK1q0jH2k/2niuhRAomTmT0lmzOOVjH4MQANj/mc+S6jhc0FqUnVmzrhxxPJk8zLZtXyKKCtcXTJIkSZKkkRg4kd8VTiET6IxHFWeuYvo11wCQbGqi5Uv/VuSKdDxz5lxNdfXpI8xIs3nLZ3hszXV0dTUWrC5JkiRJkoYzpQOnaZnAqSudpi+dnx5Gp5566pjG823mBz9ArLYWgJY7/pOeAq+40uji8QrOP++/mDPnGmIhkbkbmDHjFSxe9O7BLXWHDj3Gw4/8Ibt2fZuWlt/yxLqb+O1vL+X3D7+GrVu/QN8ovaAkSZIkScqVMBmbRV944YXR6tWrR533vo3b+c7egwBseMmZzEiU5LyWAwcOcPvtt9PT0/O8sUQiwU033cSMGTNy/t4TceAb/8W+T34SgKrLLmXB7beP65VZU1ky2U5Pzz5KS+tJJOoB6Dj8LE899SHa2zeM+N3KyiWcf/63KUs0FKJUSZIkSdIkE0J4LIqiC7OZ6wqnjHydVFdfX8/1119/3FDp9a9/fdHDJoC6664dPL3u8AMP0vHrXxe3IA2rpKSGqqplg2ETQHXVci684C4WL76Zkf4n3dm5leee+8cCVClJkiRJmuqmdOBUOyRwymcfp/nz53PzzTfztre9jfPPP3/w/oEDB/L2zhMRSkr6G4hn7PvUP5I+zoosjV+xWIJTl97C7NlXjThv/757SCZtDi9JkiRJyi8Dp4x8rXAaEIvFWLx4Ma961auIxfr/2jdsGHkLVCFVXXwRNa+5AoC+xkYOfP2OIlekkzPyiYvpqJe+vpYC1SJJkiRJmqoMnDLyHTgNqKioYNmyZQDs27ePpqamgrw3G6f85V8SyssBaP7Sl+jbu7fIFelElZfNGWVGAEoLUYokSZIkaQqb0oFTIXo4Hc+qVasGr9evX1+w946mdO5cGm56FwBRVxf7P/PZIlekEzVnzhvoD5WGE7H6satpar6/UCVJkiRJkqagKR04FaqH07FWrlxJPN7/7g0bNjCeTgqsf8c7KJ0/H4C2e+6h8U9vZNctt3DgjjtItbUVuTqNprJyCcuX3XrcsRD6/5vr7d3PunXvYsNTH6KvrxWAtrZ1bG/8Kjt2fJ3Ozm2FKleSJEmSNEmVFLuAYirGljqA8vJyVqxYwcaNG2lubmbfvn3Mnj27YO8fSaysjJkf+AC7/+IvADj80EMAtP3kpzR/6XYWfPnLVJx1ZjFL1CgWLnwH1dWnsWPnHXS0P0W8pJpTZl3J7Nl/zNZtX2DPnrsA2Lv3+xxoeZDSRAOHD2868oBn/445c67htJV/Tyzm9jtJkiRJ0omb0oFTsbbUQf+2uo0bNwL9q5zGS+AE0PHgg8e9nzp4kJ0338ypv/g5sUSiwFXpRNTXv5j6+hc/7/4Zp/8/Zs16DU8//VF6evbS29dC73GaiO/Zcxcl8WpWrPibQpQrSZIkSZpk3FKXUcgtdQArVqygtLR/9cj69evHzba65MGDtN1zz/Dj+/fT/rOfF7Ai5VrDjJfxwovvpWHGK0act2v3twa33EmSJEmSdCKmdOBUEz8SOLWnChs4JRIJVqxYAcDBgwfZvXt3Qd8/nN4tWyCZHHFOz6anC1SN8qWkpIbp0y8ccU463UNb27oCVSRJkiRJmkymdOBUEgtUxfv/Cg71FTZwAjjzzCO9kDZs2FDw9x9PrKYmizm1BahE+RZCNv2ZRjrxTpIkSZKk45vSgRMc6eNU6B5OAMuWLSOR6YU0Xk6rK1u+nLLly0acU/uaKwpUjfJpxoyXjjpn85bP0Na+vgDVSJIkSZImkykfONUMBE4F3lIHUFpaymmnnQbAoUOH2LlzZ8FrOFYIgVNuvRVKjt9PvnT+PErnzy9wVcqHqqpTmX3KVSPOaW/fwKOP/jHPPPtJksnDAHR0bGLb9tvZuu02WltXj4ugVJIkSZI0vkzpU+qguCucoH9b3bp1/X1y1q9fz4IFC4pSx1BVL3oRi+74Ok3//C90PvJI/82SEkgm6du5i4Pf+hb1b3lLcYtUTpx++j8QL6lm9+7/Jor6gP7+TnPnvIm2tnW0HnoUSLNjx9fYv/8nlJfP49Chx456xvTpF3HWmbeRSNQX4TeQJEmSJI1HYTKuTrjwwguj1atXZzX3reu28IuWNgKw62XnEAuF7VmTTCb57Gc/S3d3N9XV1dxyyy3EYuNn4VmqrY2op4fevXvZft1bIJkklJez5Ht3U7ZkSbHLU4709h6gre0JQqyU6dPOJx6vJIoi9uz5Ls8+9ymSyZFPq6urexHnn/dfBapWkiRJklQMIYTHoiga+QSqjPGTbBRJbWaFUwR0pNIFf39JSQmnn346AB0dHTQ2Nha8hpHEa2spmTmTyrPOouHP/wyAqLub3X/9YaJRTrPTxJFI1NPQ8HJm1F9CPF4J9G+vnDv3Gl70wp8xa9aVI37/4MHfeaKdJEmSJGmQgVMmcAI4VMRtdQPWrx+/DZobbrqJ8rPPBqB73Tqab7+9yBWpEBKJGf8/e/cdH2lV9n/8c0/JpM2kZ9OzPcv23qUqvYvSBFEEFbCAqPj8LNifx0dB8REElSIiIB2lKbC0ZVu2ZftmS7LJbja9zSSZTLl/f2Q3m5BksrskMynf9+vly9z3uebMdWfZMlfOuQ4ZGaF7PQE0NBzfqkIREREREREZ+UZ9wSmhS8EpUn2cxo4dS2xsx6qS7du3E4hAA/PjYdhsZP3Pf2NERwNQc/8DtG4ZugUyGTgWo/92bz5/cxgyERERERERkeFg1BecnNZj34JGX2QKPVarlalTpwLQ0tJCSUlJRPI4Ho5x40j/zrc7LgIBDn33uwRbWyOblAy6xMQF2GzOkDGlpQ+wZ++vO0+zO8o0TYJB32CmJyIiIiIiIkPMqC84JdiPrXBqjuDKomnTpnV+PZS31QEkXX01ccuXA9C+bx9Vv7knwhnJYLNaYxk79taQMabpo7T0AVat/iQVFS/g8exj27Zv8c6701jxzhTWrDmfQ4eeZSQeVCAiIiIiIiLdjfqC01Do4QSQn59PfHw8ADt27MA/hBtyG4ZB5s9/jjUhAYD6v/0N98qVEc5KBlte7peYNPH/Ybcndd6zWmLJy/0SEyd8r3MFVHt7Fdt33MnqNedyuPJFgkEvAG7PLnbs/C579vwyIvmLiIiIiIhI+PTfmGWEc1kj38MJwGKxMG3aNNasWUNbWxv79u1j8uTJEcunP/Yx6WTc/SMO3n4HAIe+810SPn05eL04Jk/Gdd55WI70pZKRwTAM8vK+SHb2tTQ1bcY0/bhcMzoLTZmZl7J3370cOvQ0Hec+9v776UDZX8jM/DTx8QXhS15ERERERETCatSvcBoKTcOP6rqtbtu2bRHM5Pi4zjsP10UXAhCoraXuoT9R99hfqfh/32fPmWfRsn59hDOUwWC1OkhKWkhy8tJufZ2iolI5ZcrPmTP7r/3OcfjwS4OZooiIiIiIiETYqC84uexDY0sdQE5ODi6XC4CdO3fi8w39RsvRU6f1ej/Q0EDZV76Kv64uzBlJpDkcGf3GtLfXhiETERERERERiZRRX3AaSiucjm6rA/B6vezZsyei+fTHNE0annqqz/FgczMNzz4XxoxkKIiOzsBqCb2dsqb2baqqXlcDcRERERERkRFq1BecnEOkh9NR06dP7/z69ddf57XXXqO8vDyCGfUt2NREe2lpyJjWos1hykaGCqs1lozMy0LG+Hx1bNl6K4Xrr6C+fi0+XyN79vyKDz5YytsrJrNq9TmUlT1KMDh0m+eLiIiIiIhI30Z90/BoqwWHxcAbNIdEwampqanz68bGRtasWcOaNWuYO3cuF154IRbL0KkRGlFRYBgQYpWKxREdxoxkqJg44Tu43TtobNzwkRE7Ca4ZNDZ13G9q2sSGjVdjtcYRCHg6o1pa9rC7+Kc0NK5n+rTfYRhD5797ERERERER6Z8+xQGuI9vqIt3DqaGhgWeffbbXsQ0bNrBu3bowZxSaJSaGuE8sDxnj/NSnwpSNDCU2Wzxz5zzBtKn3kJr6SZKSlpCXdxNLl/yb+fOfYf6850hMXNQZ37XY1FVV1avU1LwZrrRFRERERERkgKjgBLiObKtrjnDBaf369QQCfeewevXqIdfzJu1rX8dwOHoftFhwTJwQ3oRkyLBYosjIuIRZMx9k7py/MWniXcTE5AGQkDCbuXOeYNbMP2MY1pDzVBx+IRzpioiIiIiIyABSwYmhs8Lp8OHDIcfr6+tpb28PUzbHJ2bGdPIeeYToGTOO3Ty67S8Y5ODtdxBsaYlMcjKkGYZBSsrpmGbo33fetoowZSQiIiIiIiIDRQUnjp1U1+QPRHQFkaOvlUJHWCwWbLah13Yrdu4cxj3zDya88Tpjn/kHk95/j5i5cwHwFhdTcffdQ25llgwNhmEQEzM2ZEyzeyf79v8ev7+52/22tkPU16+lpaVk8BIUERERERGRk6KCE+A8UnDym9ASDEYsj2nTpoUcnzp1KlZr6O1HkRSVn0/MjBnYUlLIvvderCkDmhWCAAAgAElEQVQpADS9/E8ann46wtnJUJWTfU3IcdP0sX//b1n54emUlj6I272HjZtuYOWHn2DDxqtZtfosCtdfidu9K0wZi4iIiIiISH9UcOLYCieAZn/kCk4FBQWMHz++1zG73c7pp58e3oQ+BvuYdLLvuadze13lz39Ba1FRhLOSoSgn53rSUntvLp+YuBCrNRYAv7+BPXt/xZq151FX9363uMbGQtZvuIbW1vJBz1dERERERET6p4ITx3o4QWT7OFksFq6++mqWLl1KdHR0tzGXy0XKkRVDw0XcooWk33E7AKbPR/k3vom/vj7CWclQY7HYmTHjD0yf/ntSU8/C5ZpFxphLmTf3aebNfZKlS94hL/dGLJajW057Lwr7/Q2UHvhT+BIXERERERGRPhkjsbfO/PnzzcLCwuOO/23JYf57f0fD7n/OncSChLjBSu24+f1+mpubeeWVV9izZw8A119/fZ8roIYq0zQp/9rXcL/5FgBxy5aR+9CDGEN4a6AMTV5vJWvWXoTPV9tnjMORwfJlK8OYlYiIiIiIyOhhGMZ60zTnH0+sVjjRfYVTU4RPqjvKZrORlJTEsmXLOu+tWrUqghmdHMMwyPrlL7Hn5wHgWbmSmj/cT6C5mfaSEgJuT4QzlOHC4RhDVFToVX4+XyOBQFuYMhIREREREZG+qOBE9x5OQ6XgdNTYsWPJyMgAoLi4mOrq6ghndOKsTic5992HcWSbYM3997N70WL2nnsexUuXcuiu7+Gvq4twljIcJCTMCTkeDLby4arTKD3wZwKBFqDjlLudO79PYeEVbNr8RSoqXiAY9IUjXRERERERkVFLBSeOnVIHke3h1BvDMFiyZEnn9erVqyOYzcmLLigg/dvfPnbjyGmAZns7jS++SOl11xNwuyOUnQwXuTk3YBi2kDHt7TXs2fNLVn54GkVbbmXt2gs5eOhJGps2Ulv7Ltt33MnGTZ8nEGgNU9YiIiIiIiKjjwpOfPSUuqFVcAKYNm0aTqcTgM2bN+PxDM9taP7Kyj7H2vfupeGpp8KYjQxH8fGTmTH9/7Bau/dZs1gcTJzwXXKyr8MwogDw+eqorn4d6NmnrqFhDfv2/y4cKYuIiIiIiIxKKjgxdE6p64vNZmPhwoVARzPxE2mIPpQ0vfF66PHXQo+LAKSlfYrly1YypeDn5Od/lcmTf8SypR+Qn38zBQV3s2zpO+TmfgHDCN2Y/tChfxAM+sOUtYiIiIiIyOiighNDu4fTUfPmzcNutwOwdu1a/P7h90E52E+D8OAwXbkl4WezOcnOvoqJE+4kN+d6oqKSO8ccjjFMnvR9kpM/EXIOv78Rn69+sFMVEREREREZlVRwYmieUvdRsbGxzJ49GwCPx8OWLVsinNGJi546tZ/xU8KUiYwG0dHZ/cbs2n03Tc1bO68DAS8Vh19k564fsrv4Z9TVr8I0e27JExERERERkdBUcALirJbOb8RQ3FJ31KJFizq/Xr169bD7IJx8/XUhx61p6WHKREaDzIxL+42prn6ddesuYePG6zlU8SyrVp/N9u3f4uDBJygre4SNGz/Hps1fUINxERERERGRE6SCEx0nwR3dVjdUVzgBpKamUlBQAEBlZSX79++PcEYnJv4TnyD923eCYfQ6Xv/YYzS+/HKYs5KRKiFhLtnZn+t1zGZzER2d03ldV7+SHTu+i9db3iO2ru59dhf/dNDyFBERERERGYlCny8+ijhtVur9gSFdcAJYvHgxu3btAmDVqlWMHz8+whmdmJQbb8R51lk0vPgi/sOV2LOzwWql5r77ADj0vf/CEh+P88wzI5ypjAQFk+/G5ZxOWflfcbt3YrO5GDPmQsaNvYWoqFSqq9+k9MCDNDVtDjlPRcXzTJzwbez2pDBlLiIiIiIiMryp4HTEcFjhBDB27FgyMjI4fPgwxcXFVFdXk5aWFum0TkjU2LGkf/Ob3e4ZFgvVv/0tBAIc/Obt5D70EHGLF/Uxg8jxMQyDrKzPkJX1GUzTxPjI6rr09HNISzubbdvvpLLyxT7nMU0fbvcukpIWD3bKIiIiIiIiI4K21B3hGiYFJ8MwWLJkSef16tWrI5jNwEn58s0kf+ELAJjt7ZTfcgutW7b28yqR4/fRYlPX+/Hxk/t9fUnJAzQ2buq8DgRaOXjwKbZsuY2iLbdQVv44fr97wPIVEREREREZzlRwOuJowak1aNIeDEY4m9CmTZuG0+kEYPPmzXg8nghn9PEZhkH6d75NwhWfBiDY0kLZTTfh3bOHgNuDt7gYf01NhLOUkSo97ex+Y+rqP6Bw/adZV3gFZeWPs3rN+ezc9f+oqn6N6uo32L37blavOZeWluHVW01ERERERGQwqOB0xNGCE0CTf2gXnGw2GwsXLgTA7/dTWFgY4YwGhmEYZP74xzjP7vjwH2hoYP9nPsvupUvZd9HFFC//BAdu/BLePXsinKmMNLGx48jJ6fsUxaioYycoNjVtZPfuu2lrO9AjzuutYMvW24bdCZIiIiIiIiIDTQWnIxK6FZyG9rY6gHnz5mG32wFYu3Ytfr8/whkNDMNqJevX/0vs0o5tg2ZrK7S3d457Vq6k5JpraS8piVCGMlJNnvRDJoy/E7s9pfNeXNxkZs58iOXLVjJr5p9ISlra7zxu904aGkdGEVhERERERORkqeB0hNN27FvROAwKTrGxscyePRsAj8fD008/zbvvvktlZWWEM/v4LFFROM8+p8/xYFMTNQ/8MYwZyWhgGBbGjv0qy5d9wOJFb7B0yQoWLXyVtNSzMAwLqalnMnfO40wY/+1+53I3b+/1vt/vpr29RiugRERERERkxFPB6YiuK5yah0HBCaCgoKDz6+LiYlasWMEDDzzACy+8QCAwPJ6hL54VK0KON73xhj60y6CwWKKIi5tITExer43G451T+p1j77572Lv3N7S1HQKgqamIjZtu4N33ZvH+B4v4cNXplJU9imkO7e27IiIiIiIiJ8sW6QSGiq49nIbDCqdAIMBrr73W69jmzZtxOp188pOfDHNWAyfY0hJy3Gxrg2AQrNaQcSIDLTlpCXZ7Cj5fbZ8xgYCbktL7KSn9I4kJc2ls2oxp+jrH29rK2V38U1paD1Aw+YfhSFtERERERCSstMLpiOHWw2nXrl3U1vb9gXfdunW0d+l9NNxET58ecjxq8iQMFZskAiwWx5EiUc/VTwCJiYuxWuOPXAVpaCzsVmzqqrz8Mdye4sFJVEREREREJIJUcDrCOcwKTuXl5SHHvV4v1dXVYcpm4CVdfRVGVFSf44G6enwVFWHMSOSYMWMuZPash0lImNd5z+WcycwZDzBv7hMsX7aSgoKfEhs7vt+5qipfGcxURUREREREIkJb6o4YbiucbLb+f+mOJ2aoisrLI/u393Lwjm91bJ/7iEBNDaWfu468Rx8hKjc3AhnKaJeSciopKafi93uAIDabs3PMZosnJ/saXM6ZrCu8JOQ8dfWryfU1YrcndN5zu3dzuPJlfL564mInkJl5GXZ70mA9ioiIiIiIyIAbvhWJATbcejhNmTKF9957r8/x5ORk0tLSwpjRwHOeeSYTV7xN40sv0b5vP9bkJOLPOIPKn/6Mtq1b8R082FF0euQRHOPHRTpdGaVstrg+x2Jj87FYogkGexZNj2psXMcHK5cwJv0CsrKuoqr6DcrK/tItZt++e5k+/T5SU88YsLxFREREREQGk7bUHeEaZiucsrKymDZtWp/jp556KhbL8P/ltSUlkXLDDWT+5Mekf/ObxM6aRd4jDxMzdy4A/spKSq+7jrZduyOcqUhPNpuTzMxP9xsXDHqpOPw86zd8tkexCSAQbGHL1ltpbQ29lVZERERERGSoGP4ViQHitA6vghPAZZddxuLFi7Hb7T3GampqIpBReFidTvL+/CdiFy8GIFBby4Hrr6d161ZaCgupfeRR6p/+B/5h3MNKRo5JE+8iMXFRj/s2q5OZM/7I+PF3EB2d3e88waCXg4eeHIwURUREREREBpxhmmakcxhw8+fPNwsLC0/4dRPfK8IdCLI4IY4X504ahMwGR1tbG1VVVfj9fp5++mm8Xi9Wq5VbbrmFlJSUSKc3aIJtbZR/4xt43j2ytdBigWDwWIDNRurNN5H6ta9hGL2fKCYSDqYZoKZmBVXVrxMItJLgmkVm5hVERSV3GX+boi1fCTlPYsIC5s176iNzmzQ2rqet7RCO6EwSE+ZhGPpZgoiIiIiIDDzDMNabpjn/uGJVcDpm7ofbOOT1cUpcNCsWThmEzAbf6tWref311wGYPHky11xzTYQzGlxmezsHv/Utmv/zZp8xY/7reyRff30YsxI5caYZYMU70zHN9hBRBikpp5OV+RlSU8/A7d7Jtu130tKytzMiNnYcU0/5XxIS5gx+0iIiIiIiMqqcSMFJPwbv4mgfp+Gypa43CxYs6GwWvnv3boqLiyOc0eAyoqJwnnteyJjaP/8F0+8PU0YiJ8cwrKSnnd1PlElt7Qq2bL2F9z9YTOH6z3YrNgG0tOxn46bP09JSMmi5ioiIiIiI9EcFpy4SRkDByWq1cu6553Zev/766/hHeLGldf36kOP+qiraDxwIUzYiJ2/cuK9jszp7HYuOziE+/pTOa7+/EdP09RobCHg40EvzcRERERERkXBRwakL55GCU3MgSGAYbzWcMGECU6Z0bAmsra1l7dq1Ec5okHU5YbAvxgg4sU9Gvri4Ccyb9zTJScs671ks0WRnXc2iha+waOG/WLjgZXJyrqe/P75rat7tcc80TZqaijhU8SzVNW8RCHgH+hFEREREREQAsEU6gaEkoUvhwu0PkGAfvt+ec845h+LiYgKBAO+88w4zZszA6ex95cRwF/+JU6n/6+N9jlvi4rBlZoYxI5GTFx9fwJw5f8XrrcbnbyDakYXNFtc57nROo8A5jZqad2hr63vlntd7iO3bv82YjEtITlpCa2s527Z9k6bmos4Yuz2JyZN+SEbGxYP6TCIiIiIiMvpo2UcXri4Fp8ZhvK0OICkpiWXLOlZJtLe389Zbb0U4o8ETt2wpMfPn9Tke9Hgov+VWAm53GLMS+XgcjjTi4yZ1KzZ1lZy8tJ8ZTCoOP8+mTZ/ng5VLWbvuom7FJgCfr55t2++gtvb9AcpaRERERESkgwpOXXQtOA3nPk5HLV++HJfLBcCmTZsoLy+PcEaDw7BYyL3/fpznnguG0Xnf4nRiREcD4Fm5ktLPXYevsjJSaYoMqLzcG7FYovsYtRIVldZ51d5eQyDg6SPWpKTkDwOen4iIiIiIjG4qOHXRveAUjGAmAyMqKoqzzz526tVrr71GMDj8n6s3VpeLnN/ey8Q3/0PO//2evIf/wqSVHzDumX9gy+rYTufduZOSK6+ibdfuCGcr8vHFxY1n1qw/44ga0+1+VFQ6s2c/zPJlHzJ3zt/JyroSwwi9PbihcR2BQFu3e8Ggn+rq/7Bn76/Zv//3uN27BvwZRERERERk5DLMYdwcuy/z5883CwsLT/h1fztUy527ygB4dPo4zk1LGOjUws40TR599FFKS0sBuOSSS5gzZ06EswovX1UVZV/5Ct7tOwCwxMeT9atf4TtcQcuatWAxiP/EqbguOB+LwxHhbEVOTDDoo67ufVrbDhLtyCQl5VQslqhuMZs230Rt7dsh50lImEfGmItJTz8Xv9/D5qIv0dKyr1tMZsblTJnyCywW+4A/h4iIiIiIDH2GYaw3TXP+ccWq4HTMy1UN3LytBIDfTcnjyszkAc4sMg4fPsyDDz6IaZrExMSwbNky7HY748ePJy0trf8JRoCgx0P57bfjea/vXjVREyaQ9/DD2MekhzEzkcFXVvYYu4t/clyxBlYMSxTBYGuv4/n5X2XihDsHMj0RERERERkmTqTgFLYtdYZhnGsYxi7DMPYYhnFXL+P3Goax6cj/dhuG0dBlLNBl7OXBytFlO/btGAk9nI7KyMjoXNXU2trKm2++yWuvvcYf/vAHnnnmGXw+X4QzHHyWuDhy77+fhM9c0WdM+969HLrru2HMSiQ8MjMvx+Ho+6TGuLjJnV+bBPosNgGUlz9OINAyoPmJiIiIiMjIE7qxxwAxDMMK/AH4FFAOrDMM42XTNLcfjTFN8/Yu8V8Duu77ajVNc/Zg5znSmoZ31dLS+wfEbdu2YbPZuOyyy8KcUfgZNhuu88+n8Zln+4xpWbUa7969OCZMCGNmIoPLZnMyd87jbN36DZrd27rcT2Dy5B+SmXEpbW0VVFW9SumBP9HeXt3nXIGAm8bGzSQnL+m819p6gANlj1Jf/yEAycnLyc25gZiYnMF7KBERERERGdLCUnACFgJ7TNPcB2AYxlPAJcD2PuKvBn4Uptw6JYzQglNdXR07d+7sc7yoqIgzzjiDxMTEMGYVGe3Fxf3GeIuLVXCSESc2dhwLFrxEU9NG3J5i7PZEUpJPw2rtOOkuOjqTvLwb8fkb+z21btPmG0hJPpX09HOJikpny9Zbu52C5/EUU3HoWWbPeYwE16xBfS4RERERERmawrWlLhso63JdfuReD4Zh5APjgK4dbqMNwyg0DGO1YRiX9vG6m4/EFFZX9/3T+VC6rnBqHEEFp6MNw/timiYHDhwIUzaRZU3ovxG81eUKQyYi4WcYBgkJc8nOupL0tHM6i01dpaV+st95TNNPTe3bbN/xHTZtvqFbsekof6CZ7du/hWmOzJMxRUREREQktLD1cDoBVwHPmqbZteKTf6Qp1TXAbw3D6LH8xDTNh0zTnG+a5vyTbYTttB4rODUHRk7ByWLp/5fZMIwwZBJ58WecgRHT80N2V55VqzFH0K+/yIlwuWaSnn5+n+Pp6RcSH3/Kcc3V0rKfhoa1Pe4HAl7q6lZSU7MCr/fkfkAgIiIiIiJDW7i21B0Ecrtc5xy515urgFu73jBN8+CR/99nGMY7dPR32jvQSUZbLURbDNqCJo2+kVNwGDduHBaLhWCw95UGFouF8ePHhzmryLC6XKTfeSeVP/1ZnzG1f/oTrVu2kP3r/8WWmhrG7ESGhmlTf43DMYaDB5/qbCAeHZ3LxAnfZsyYC4COYtKePb+iuubfIecqP/h3HI5MYmPzO67Ln2Df/nvx+eoBMAwbGRmXUjD5bqzWmEF8KhERERERCadwFZzWAZMMwxhHR6HpKjpWK3VjGMYUIAlY1eVeEtBimqbXMIxUYBnwq8FK1Gmz0tbuH1E9nFwuFwsWLGDNmjW9jufk5BAXFxfmrCIn+dprsaWkUPPgQ3h37AAgZs4cYubNo/6JJzBbW2lZvZr9l11O9r33EDt/PqZp4q+qBr8PW0YGRpfVcCIjjcXiYPKk7zN+3Ddwe3ZjtUQTH38KhnFstWRs7Diysj7bb8GpquoVqqpeIT5+Cg5HFrW1b3cbN00/FRXP4vPVM2vmQ4PyPCIiIiIiEn5hKTiZpuk3DOM24A3ACjxsmuY2wzB+AhSapvnykdCrgKdM0zS7vPwU4EHDMIJ0bAH8766n2w20BJuV6nY/TSNsS9XZZ5+N1Wpl7dq1+P3+bmPl5eUcOnSIrKysCGUXfq5zz8V17rkEmprAYsEaHw9A4qWXUP71b9C+bx/+6mpKP38DCZddStu27Z3FKVtWJilfvJGka68ZNVsRZXSy2ZwkJszrczw5eTkORwZe7+F+53K7d+J29314QU3NWzQ1FeFyzTypXEVEREREZGgxutd2Rob58+ebhYWFJ/Xa89fvZkNTC8l2K9uXzxjgzCKvpaWFsrKO/u1VVVW89dZbAKSmpnLzzTcTFRUVyfSGhKDHQ8UPfkjTq6+GjEu5+WbS77g9TFmJDE11dSvZXHQTwaC3232LJZppU+/F72+kuvrf1Na9h2n6+5ilw9j8W5gw4Vud18FgO4cPv8jhyn/i9zUSFz+JnOzrSEiYPSjPIiIiIiIioRmGsf5Ij+3+Y1Vw6u6qTXt5p74ZqwHlp80a0StYgsEgTzzxBHv3drTDmj9/PhdeeGGEsxoaTNOk/m9PUPnzn/cdZLEw8a03sWdmhi8xkSHI7d5F6YE/UV/3IRgGyUnLyMv7EvHxkztj6uo+ZOOm60LOYxh20tLOJi3tUyQlLmTr1m/Q0LiuR9zkST8kN/fzA/4cIiIiIiIS2okUnMLVw2nYcNk7evMETGgJBokbwb16LBYLl156Kffffz+tra0UFhYyadIkCgoKIp1axBmGQczMfla4BYM0//vfJH9eH3xldIuPL2Da1F+HjElImIPN5sTvb+4zxjR9nT2fOnZQ937Qwe7in5KcvIy4uIkfI2sRERERERlMlv5DRpcE27EC00hqHN4Xp9PJJZdc0nn90ksv4Xa7I5jR0BE4ju9DwOMJQyYiw5/VGkNuzg19jtvtKURHdz3MtPdiUweTg4ee7nG3sWkzO3f9gM2bb2LX7rtpbh60dn8iIiIiItIPFZw+wtWl4NQ4CgpOAFOmTGHu3LlAR4+nl156iZG41fJEOSZNgn5WuFlidIy7yPEaN+5r5ORcB3TfquxyzWHRwn+xdMkKFi18lfz8r/Q7V2XlKxw69CxebyWmabJnz68oLLycgwf/Tk3t25SXP87adRdRUnL/ID2NiIiIiIiEoi11H+HqUmBo8o2OghPAueeeS0lJCXV1dRQXF7Nu3ToWLlwY6bQiyp6ejuu882j617/6jKn6zT1gQvINn8ewqH4rEophWCmYfDd5uV+ipvZtgoE2EhLmkpAwr7NfXnx8AeNjv0lZ2aMEg219ztXeXsmOnd8FwOHIwus91Gvc3n2/weWaTXLy0oF/IBERERER6ZM+IXfh8QfY6m7pvL7vQBXFnr4/8IwkUVFRfPrTn8ZypGjy73//m6qqqghnFXkZP/ohMfN6HgtvOBwdX/j9VP3qV5R99av46+vDnJ3I8BQTk0NuzvXk599MYuL8HoczWCx2MsZcfNzz9VVsOqr84OM97gUCbRw+/BJ7991DWdmjeL3Vx/1+IiIiIiLSP51Sd0RNu5/LN+5hd0v3ApPdMHhwWj7npyUOZIpD1nvvvcfbb78NQEJCAtnZ2QSDQXJycpgzZw5xcXERzjD8zEAA9/vv417xDqbPR+y8uTjPO4/GZ56h8te/AZ8PAFt6Ohk/+THeHTtoevU1As3NRE+ZQtJ1nyN+2bIIP4XI8OL1VrN+/WdpbTvQYyw7+3Pk5d5Abe171Na9T23tipBz2Wwupk27l6TExVit0dTXr2XL1lvx+eo6YwzDxsSJd5GX+4UBfxYRERERkZHiRE6pU8HpiK9uK+GFqoZex2IsBuuXTiPZPvJ3IAaDQR5++GHKy8t7jEVHR3PttdeSm5vbyytHp9YtWzn4rW/hO9DzQ3FXad+6g9SbbgpTViIjQ3t7LaUHHqLy8D/x+RuJj5tMTs51ZGRc1m1V1IerPklr6/5+57NYHLhcs2ls3IhptvcaM3PGA6SlnT1gzyAiIiIiMpKcSMFJW+qAOp+ff1b3XmwCaA2aPF85OrZLWSwW4uPjex1ra2vjqaeewndkRY9AzIzpjHv+OVznnx8yrvo39+DduzdMWYmMDFFRKUya+D2WL/+QM07fxoIFL5CZeXmPLXgZGZf0MUN3waCXhoY1fRabAEoP/KmX1/moqn6DPXt/TUnJ/Xg8+07sQURERERERiEVnIDytnb8/Sz02tfiDU8yEebxeNi9e3fI8e3bddR4V9b4eDJ+/jOwhV4B1/jCC2HKSGR0ycu9gbi4Sb2OxcdPZ/asx8jP/yrx8VP7nauxcQMNDesIBv0AeDx7Wb3mbLZsuYXS0gfYu+83rF7zKXbu+gGmOXoOlhAREREROVEjf4/YcUiL6v/bkH4cMSNBTU0NwWAwZIyaifcUbGgAvz9kjO9QRZiyERldbDYn8+Y+yb59v6Pi8PMEAh5sNhdZmZ9h3LivYbM5SUlZzsQJd7JhwzXUN6wJOd/6DVdhtcaTmLiAxsYN+P2NPWIOHvw7DkcG48beOliPJSIiIiIyrI2OKko/Mh1RnJoUz3v17l7HLcDlY5LCm1SExMTEDEjMaGNNSsKIjsZs6/tUQ19lJcH2dixRUWHMTGR0sNuTKCi4m0mTvn+k4BSPYVh7xKWnX9BvwQkgEHD324y8rOxR8vNuwmI59nvaNIPU1r1Hff1qDMNKSvJpJCYu6LENUERERERkpNOWuiN+NimHJFvPDycA/zU+k7wYR5gzioy0tDTGjBkTMmbatGlhymb4sERH47rwgpAxrevXs//Sy/CsWdtjbCQ27xeJBIvFht2e0GuxCSAz8zJiYyf0OmYYDiZN/AE5Odf3GdOVz1fHoYpn8Ps9QMfJeusKL2Pz5hs5cOBPlJb+kQ0br2bjpuvx+5tP/qFERERERIYhnVLXRWmrl9+XVvF8ZR0twY7vy1dy0rh7UvZApziklZSU8PjjjxMI9N6f5Oqrr6agoCDMWQ19/vp6Sj93He29NQe328B3bMtdwiWXkHLLV2l89lkaXnyRQHUN9txcEj/7GVI+/3kMrYISGTRebzU7d32fmpq3gI4/6+Pjp1JQcDeJCfM647bv+C8qKp7udz7DsJPgmk1b2yHavAd7jUlPv4AZ0+8bkPxFRERERCLlRE6pU8GpFxuaPJy/vhiAr+am8aOJo6vgBFBeXs6KFSvYe6R4EhcXh8fT8VN8u93OjTfeSEZGRiRTHJICzc3U//1Jml59lWBzM45TTiH5c9cSlZfH4Z/9HPeKLlt0LBbopV9W3KmfIPf++zH6aUIuIh9PW9shWlpLibInExc3uce2t/r6NWzYeM0AvZvB0iXvEhNz7O+TQMBLZeU/qal9CzPoJyFhHllZnyUqKnmA3lNEREREZGCp4PQxC05tgSAT3y/Cb8KyxHiemzNxALMbXtrb2wkEAkRHR/P666+zZk1H7xOXy8VNN92E0+mMcIbDS/Nbb3H4Zz/HXxG6gXjW//w3CZcc37NmIzYAACAASURBVFHvIjI4TNNkc9FNffZymjDh2wQDbdTVfUBj0yaOrpbqS1bWVeTl3khs7Dh8vlo2brwet2dXtxi7PZHZsx7B5Zo5UI8hIiIiIjJgVHD6mAUngLPW7WSbuw2XzcKu5TPU8BUIBoM8+eSTFBd3rP7Kzs7mhhtuwG63Rziz4SXo8VB82ukE3b03qQeIW7qUvIf/EsasRKQ3gUAbxXt+QUXFswSDXgBiYvKZNPEu0tLO7ow7UPYYxcU/Oa45o6LSMQwLXu/hXscdjgyWLlnRrRn5sXxaCQQ82O3JGIbaMIqIiIhIeJ1IwUn/Wu3DTGcsAE3+ICWt7RHOZmiwWCxcccUVpKenA3Dw4EFefPFFgr1sC5O+WeLiMPv5nvlqa8OUjYiEYrVGM6XgJyxftpp5c59m4YJ/smTxm92KTQBj0s/HMI5vG2x7e1WfxSYAr/cw1TVvdbvn9hRTVPQV3nl3Ju9/sIiVH36CkpI/Egz6+5hFRERERCSyVHDqw9GCE8Dm5pYIZjK0OBwOrrnmGuLi4gDYtm0b7777boSzGn4cY8eGHA9UVdG2a3d4khGRftntLhIT5+N0Tu11ZZHDkUZe3k19vn7c2K9zypT/ISPjUuz2/ns07dt3L6UH/kxj4yaamraxvvAzVNf8B+goVnu9h9m773/ZseM7OuVSRERERIYkbanrw4ZGD+dv6Ng6dktuOj+cmDUQqY0YZWVlPProo50n2V1wwQUYhkFraytjxoxh4sSJWCyqZ/al/h//4PAPfxQ6yGol+XPXknrbbViP9Mpq27Wb9tISbKlpxMyehaHvsciQYZomBw48ROmBP+Pz1QHgcGQybtzXyM66sjOusbGIwvWXncDMFo4Wmnozd+5TJCUu6HYvGPTT0LAGr7eSmNh8ElxztTVcRERERD429XAagIJT65HG4QETlifG8+wobhzely1btvDcc8/1OpaUlMSVV16pk+z6YAaDHLrrLppe/mePMVt6Ov6qqs5ra2oqKV/4Ak3//jdtmzd33o/Kzyfjpz8hbuHCsOQsIscnGPTi9hRjYCUubhIWS/etdqZpsnrN2bS07BuQ98vMvIKpp/xP53Vt3Qfs2HEXXu+xwwni4iYzbeo9OJ2nDMh7ioiIiMjopILTABScAM5cu5PtnjYSbFZ2Lp+unw734umnn2bHjh29jsXFxXHbbbcRExMT5qyGB9M0cb/zDo0vvIi/poao3BwSr7ySmNmzaXjuOap/cw+BhoaQcxgOB2OfforoKVPClLWIDIS6upVs2nwjpunrMTZxwnfJyLiMxsb11NS+Q0XFM/3OFx9XQGLSQqIdWezZew/Qc167PYlFC1/F4UgfiEcQERERkVFIBacBKjh9c8cBnjrcsS1i9eJTGBvj+NhzjjT3338/VV1W43zUOeecw5IlS8KY0cjhr6+n+ne/o+Gpp0PGuc4/n+x7fhOmrERkoDQ2bmT//vuorXsfMHHGTyM//2bGjLmwMyYY9PHByiX4fPUD8p5jx97KhPF3dF6bpkl19b85eOhJWltKiXKkkpFxGVmZV/R6Sp6IiIiIjG4nUnA6viN1RqmZzhieOnKQ0ObmFhWcPqKtrS1ksQmgtLRUBaeTZEtKIvPuu2ndtBnvzp19xrnVtF1kWEpImMPs2Y8QDHoxzQBWa2yPGIvFTnb2NZSU/KHXOQzDRkLCfJqbtxAIePp9z4qK50hJPhWncwYWSxS7d/+Y8oOPd463th2gsXED1VWvM2vWn7BY9PeeiIiIiJwcFZxCmNXlpLqi5lYuSU+KYDZDz/FsMbRarWHIZGSzOEJ/4Av6fJimqS2fIsNUf0WdcWNvw+3eRU3Nmx95XTQzpv8fqalnEAz6cbu3s37DVQSD3j7n8noPs37DlRiGnZiY3D77SNXVr6Ss7FHy87/c7b5pmjQ0rKHZvQObNY7U1LOIiko5zicVERERkdFEBacQpsbHYDUgYEJRc0uk0xlyHA4H+fn5lJaW9hkzbty4MGY0MsUuXEhrl2bhPbS3U/alm0j/zneILpjceds0Tcy2Ngy7HcOm3+oiw5XFEsXMGQ9QV7eSyqp/4fc344w/haysz+JwjDkSY8PlmklKyhlUV7/e75ym6eu3aXn5wae6FZxaWw9QtOUW3O5jffsMw874cd8gP/8rKnqLiIiISDfq4dSPM9buZIenjUSblR1qHN5DSUkJjz32GH39d5SVlcV1112nxuEfg6+ign0XXUzQ7Q4daLGQeMUVpN52K81vvkn9Xx+nvaQEw27HefbZpN52Kw4VAEVGtKamIgrXfwbT9PcYs1rjmTTxe7S0ltDYuIHGxg1A6H8DOJ3Tcblm43ROZ//+33U7+a6rU6b8kqyszw7EI4iIiIjIEKam4QNYcPrGjgM8faRx+JrFp5CvPk497Nq1i9dee42GLieq2e12fL6OU5IyMzO57rrriI3t2Z9Ejk/Lxo0cvP0O/IcPd96zxMeT9LnP4X73XbxdTwq02cDf88OmxeVi7JN/xzFhQjhSFpEIqal5m+077sLnq+28FxOTx/Rpv8Plmtl5b/Pmm6mpfWtA3jMmJo8li9/CMCyd9+rqV3Gg9CEaGgsxDBupKWeQn/8V4uMnh5hJRERERIYyFZwGsOD0l/Jq/l/xQQAemjaWi9MTB2TekSYYDFJeXk5rayvp6elYLBYee+wx6uo6inUZGRlcf/31Kjp9DKbPh/v992kvKcWWlorzzDOxxMVhBgI0vvQy1ffei7+6OuQc8aefTu4fHwhTxiISKcFgO7W17+FtryI2Jp+kpCXdikEA1TVvUVR0c59zREfn4PPVEQgc35byCRO+Q2rK6cTFTeLw4RfZvuM7fHQFlcUSw5w5j5GYMO+En0lEREREIk8FpwEsOBU2erhwQzEAt+Wl8/0JWQMy72jQ1NTEY489Rm1tx0/Zx4wZw/XXX09cXFyEMxuZgi0tlH7hC7RtLuo7yDCYvHoV1oSE8CUmIkOSaZrs2HkXFRXP9hhLSJjPnNmPYhh2PJ5idu3+EY2N649rXqs1/kiRKtjreFzcZBYtfLXbFnW/v5lDFc9SW/MOpuknMWkx2VlX4XCkndSziYiIiMjgOJGCk6X/kNFtanxM5zdJjcNPjMvl4oYbbiA1NRWAyspKHn30Uerq6igqKuKDDz5g69atnVvv5OOxxMbiGDs2dJBp4qusDEs+IjK0GYbBKVN+ybSp95CYsICoqDTi46cyadL3mTP7r1itMVgsNpzOU8jLu/G45w0E3PRVbALweHZT37C287q1tZw1ay+iuPhn1NV/QH3Davbv/y2r15xDU1OIArqIiIiIDGla4XQcTl+7k51qHH7S3G43jz32GNVHtnsZhtGtyXhsbCyXX345EydOjFSKI0bNQ3+i+p57QsZYXC6Srr6a5M9diy0tjZYNG6j9y8O0FBZi2GzEn34aKTd+Ccd4NRgXkQ7BoJ/C9Z+muXlrr+OTJ/0IqzWWxqaN1NS8SXt7Tb9zxsTk43ROp6mpiLa2sl5joqOzWbL4bSyWYydtmmaQ2tp3qap6DX/Ag8s5ncysz+CISj25hxMRERGR46YtdQNccPr6jlL+cbgeUOPwk+V2u/nLX/5CfX19r+M2m40vf/nLpKVp+8TH4a+uZs+nzsZsa+s31oiKImbOHFrWroWP/DlgiYsj75GHiZk5s49Xi8ho095ex46dd1FTc6zReFRUKpMm/hcZGZd03qusepWtW782YO87c8ZDpKWdBUAg0MaWLV+ltu69bjFWaywzZ/yR5ORlA/a+IiIiItKTCk4DXHD6c3k13z/SOPxP08ZykRqHn5Qnn3ySXbt29Tk+b948LrroojBmNDI1v/UWB2+/A7O9vdv96DlziJk5g8bnnifodvc7j6OggHEvvqAVfSLSTWvrAZqbd2CzxZOYuACLJarbeDDoZeWHp/a5ysnhyCY+fjLNzVtpbw990AF0NBpPSJiNM34qHk9xj2LTUVZrPMuWvovd3v3vaK+3msqqf9HeXkNs7DjGpJ+P1aoDLEREREROxokUnGz9h8gs57F/mBY1t6jgdJIOHjwYcry0tDRMmYxszrPOYvyrr9Lw9NO07diBJT4e13nn4TzrTAybjbTbbqPhH89Q8+CDBJua+pzHu2sX3h07iJ46NYzZi8hQFxOTR0xMXp/jFouD6dPuY3PRl3qccOdwZDJv7t86X19d/R+Ktnwl5PsFg63U16+ivn5VyLhAwE3F4RfIy/1C570DZY+wZ89/Y5r+znvFxb9gxvTfazWUiIiIyCBTwek4TDvSODwIFDW3RjqdYctiCd2jvr9xOX5ROdmkf+uOXsesTicpN36RoNdLzX33hZyn/eBBFZxE5IQlJS1i0cLXKD/4OA0N6zAMG6kpZ5KdfVW3FUipqWcRE5NHa+uBXucxsBIbO46W1v2YZqDf992373c0Nm7E5ZxGIOBlf8nvesT4/Y1sLvoyixe9TkxMTrcx0zRpatpIa2s5Dkc6iYkLMQz93SQiIiJyMlRwOg6xVguT4qLZ5WmjqLkF0zS1zegkTJ48mVBbHZOSksKYjTjG5vcbc/iHP6K9pISkK6/E6nIRaGig/h/P4H7vXfAHiF24kKRrrsaekRGGjEVkOImJyWHSxO+FjDEMC1MKfsamzTdimj1PLJ0y5edkZX2GQMBLQ+M6Nm36fMj5AoFmqqpeoarqlZBxwWArBw8+wcSJ3+2819y8g23b78Dj2X3sGaLzOOWU/yYpaVHI+URERESkJ/VwOk5f21HKM0cah69dfAp5ahx+wurq6njwwQfxer19xnzqU59i6dKlKuiFQdDrZc8ZZxKoq+s31hIXh/O8c3G/+x6B6u49Vyzx8eQ+9BCxc+cMVqoiMsI1NRWxv+QP1Na+i2kGSExcwNj8L5OSclq3uA0brqW+YXWf89jtSfh8vR9O8VE2m4vs7GtxOqfhcKSzadNNBAKNPeIslmgWzH+B+PjJ3e6bZpD6+lW43buw2VykpX2yR/8oERERkZFGTcMHoeDUtXH4n6eN5UL1cTophw4d4qWXXqKysrLzntPppLm5ufN6wYIFnHfeedpiFwaeDz+k7JZbe5xqZ01OJm7JEprffBMzRIHwKFtGBhP/828Mu32wUhWRUaDj3yRmn9vYmpq3sn79VQSDPbe3Z2V+llNO+SVebxXNzdso2vLVXldNnYyMjMuYNvXXndcezz62bL0Fj6e4857F4mDC+DvJy/vigLyniIiIyFCkgtMgFJzWNri5eOMeAL6el85/Tcga0PlHE9M0qaiooLm5mYSEBDIyMtiwYQP//Oc/OfrfY0FBAZ/+9KeJiorqZzb5uNpLS6n72xO0FBZi2GzEn34aSVdfjS05GX9tLfVPPEHdXx/v92S7nD/8H86zzgpT1iIyWjU372DfvnuoqV0BmDgcmeTm3kBe7he7Faq2brudysqXB+Q9LZYo8vO/SnzcZKKjc9hc9GXa2w/3Gjtt6r1kZFzca96NjRuwWKJISTkNhyN9QHITERERCScVnAah4OQJBJj03haCwOlJTp6aPWFA5xcoLi7mmWeeob29HYDs7GwuvPBCdu3aRU1NDbGxscyaNYvs7OwIZzr6NPzrX1Tc+e2QMam33Ubabbf2uO+rqCDY0oI9JweLQ1tRRWRgBAItBAIt2O3Jva6IcnuKKSy8vMdJeQCxsROYO+cJWtsO0Ny8jX37fovf33M73cmIi5vMooWvdm4N9/nq2brtdurq3u+MMQwbubk3MHHCd9WUXERERIYVFZwGoeAEcOqanexuaSPZbmXbsunqMzQIKioqeOKJJ3CHWE2zYMECzj//fH3/w6hl40ZKr74mdJDNRsLFF5N09dXEzJhOS2EhVf/7a1o3bwbA4nKRdNVVpN12K4ZWrolIGDQ1FbFr149oai46cscgNfUsphT8DIcjrTNud/HPKSt7eMDeNyvzShISZhMXN4XiPT+nsbH3f5OMG/dNxo/7Wq9519a9D2aQpOSlJLjm6u88ERERGRJUcBqkgtNt20t5trKjGem6JVPJjdaH5sHQ0NDAY489Rn19341fL7roIubNmxfGrEY3Mxhk3/kX0F5SclzxUePH015aCoGex5g7zzmH7N/eqw9PIhI2Hs8+2ttriInNI9rR81TN1tYy1qy9gEDA02PMMOzMmvUwFsOK27ObsrJHaG0tHZC8bDYny5d9iNUaC4Df72Hrtm9QW7uiW1xS4mJmzLgfuz1hQN5XRERE5GSdSMFJ67hPwCxnbOfXRc09l+jLwEhMTGT8+PEhY9asWROmbATAsFjI+MmPMfrYEuc89xxsmZmd1+379vVabAJofuMNWjduHJQ8RUR6Exc3nqSkhb0WmwBiYnKZPfsRoqO7b9m221OYOeMBUpKXkpS0iNyc6xg37uv9vNvxF9P9/ma2bP0aBw78hdra99m2/Vs9ik0A9Q2r2bb99h73A4EWysoeY/2Ga1hXeDm7dt2Nx7P3uN9fREREZDDZIp3AcDLTGdP5dVFzKxek6aS6wVJTUxNyvKqqimAwqJPswihu4ULGPv0UtQ89hPuddzEDAWIXLiTlxhuJW7wI0+/H/d571D36GC1r14acq/mNN4idOzdMmYuI9C8xYR5LFr9NXf0HtLWW43Ckk5JyGhZL90L7mPTzKC19EI9ndy+zGMyc8RDRMVm43Ts5ePDvNDauD/m+tbXvUFv7Tr/51da+i9u9i/j4AgDa22vZsPHabiflNTVt5uChp5gx/T7S0s7ucy7TNLXKVERERAadCk4nYHp8DAZgohVOgy06OjrkuM1m0z+WIyB6yhSy77mn1zHDZsN55plEjRvHvvPODzmP+/33cV1wAdEzZmAYBsG2NhqefpqGl14iUFNL1LhxJF11Jc5zz9Wvs4iEjcViIzXl9H5iHMyZ8zg7tn+b2rr3Ou87osYwafIPSEs7EwBn/BRiY8dRWHj5gOW3afONJCbMIzZ2PHX1H3QrNh1lmj62bruD5cvex25P6rzf3l5DSckDHK58CZ+vntjYCeRkX0tOzucwDOuA5SgiIiJylHo4naBPrNlBcYtXjcMHWVFREc8//3yf4xaLhYsvvphZs2bp12CICba3s+fU0wg0NPQb65g0EddFF9P8n//QtmVLj/Gka65mzA9+oF9jERmSWlr243bvxmZzkpi4AIvF3iNm0+Yb+1zBNHHC90hPPxePZzeHK/9FZeVLA5bb+HF3MHbsLRiGgbe9hvXrP0Nr64EecWPGXMy0qff0+udsa2sZbW2HiI7OJCYmb8ByExERkeFLTcMHseDUtXF44ZKp5Khx+KAIBAL89a9/pbQ0dGPWGTNmcOGFF+Loo7eQREb1ffdRc/8DvQ8aBpzAnzu5f/4z8cuXDVBmIiLh5fd72Lnr+1RW/gsIAmC1xjE2/xby87/cWejx+5t5/4MlBIOtfc5ls7nw+5uO+71ttkTi4ibg8zXS0rKnz7jZsx4mJeW0zuuWlv3s3PUD6utXdd5LTFzElIKfERcXuseiiIiIjGwqOA1iwemhsip+uOcQAA9PH8v56uM0aNrb21mxYgUbNmzA6/ViGAYFBQW4XC7WdukRlJyczBVXXEFGRgb79+/n0KFDREVFMWXKFBISdKJPJJg+H4fu+h5Nr7zS7b41OZmcP/wfgYYGGp9/nuYV74DfH3Iu1/nn9bmNT0RkuGhrO0Rj40YsFgdJSYuw2Zw9Yvbv/z/27b+319fn5X2JiRPuwuero75hHVu33jpguSUkzGNKwU+JiRmL39/E2nUX095e1SMuKiqVBQte6tF8PRj0U1v3Lh53MXZ7Iunp53TbziciIiIjhwpOg1hwWt3g5tKNHT8l/Gb+GO4an9nPK+Tj8vv9uN1uYmJiOlcy7d+/n+eeew632w10bLGLjY3tvAYwDIOlS5fyyU9+UluyIsA0TdqKimh6/Q2CHg/R06aRcOEFWOLiOmN8NTXsWf6JkPPYMjLIe+RhHOPGdczb3k7dE3+n4Zln8JWVYRszhoTLLyPlhhuwxMaGnEtEZCgzTZMDZX+mpOSP+P0d25JtNie5uV9k3NjbMIxjB2Vs3Hg9dfUr+5jJIDXlDNq8FbS07CMY9B5nBhZstviQq6jy8r7EpInf67xubt5G0ZZbaGsrPzaLEcWEid8hL/cLx/m+IiIiMlyo4DSIBSePP8DE97dgAmckO3ly1oRBeR/pn8fj4cUXX6S4uGfT1K7OPvtsli5dGqas5ETtOfMsfIcO9RsXPX06rvPPo/m992hdvabn+KyZ5D/6KJaYmF5eLSIyfAQCbTQ3b8U0g7hc07FaexbTPZ49FK6/srMw1dWECd9hbP6XgY7VRys/XE57e/WA5Ga1xjN+/DeJi51AVFQaGzZeh99f32vsjOl/ID393G73Ghs3cKDsEZoaN2GxxpCWdja5uTfgiEodkPxERERkcKngNIgFJzjWODzFbmPrsmlaPRNBpmnywgsvUFRU1GdMfHw8t99+O1arTuEZiqrvv5+a+34/IHOl3XEHqTffNCBziYgMda2tBygp/SNVVW8QDLbgcs4iL+9G0tI+1S2upOR+9u77TZ/zjM2/lUCwBY9nD/V1H2ISGJD8nM7pLFxwrBF6RcVzbN/xXTrO+z3G4RjDvLlP9dqYvLX1IPUNqzAwSEpaQnR01oDkJiIiIifnRApOtsFOZiSa6YyluMVLrc/PIa+PbDUOjxjDMPptGO52u6mrqyMtLS1MWcmJSPniF2n5cBUtvRSJE6+8kpjZs2l65RU8q1ZBIPSHoMYXX1TBSURGjZiYPE6Z8gtOmfKLkHF5eV+ioXEDtbUreoxNnvQDcnNv6Lzetu1bHK58cUDya27eyrrCK4iLHUdUVBqlB/7MR4tNAF5vJbt2/5jZs/7SeS8Y9LJz1w+pqHieo83WwUJW5hUUFPwYi0X/9hIRERnqVHA6CTOdMTx35KS6ouYWFZwizGKx9Bvj76cxtUSOJTqa3If/QsM/nqHx5ZcJ1NQQNXYsiVddifNTn8IwDBIvuxR/TQ37Lr2MQE1Nn3O1l5XhWbuW2PnzMSwWTNOk6V//ou7xv+HdvRur04nrggtI+dKN2FK1fUNERgeLJYpZMx+kqvrfHD78Ij5fPXGxE8jOuRaXc3q32NzcGzhc+U/odZWT8f/Zu88AqaqzgeP/e6fPTtnZ3ndZYJfeEQQVwdjQEBsqKvYaS9TkNWpi1MTEEmPX2FERK3aiKHYB6bDUZSnbe5tt0+fe98PCwDAzC+pSPb9v99xnzj2XZXdmnnvOcxgy+ElkjR5X11aqql/H4+l5SXR7+2ra21fvdYzNzd9S3zAfu204BkPqjmTT3D2iFGpq30GSNAwYcF/YGbe7kvKK52ls/BJF8WC3jyQn+woSEsQup4IgCIJwsIgldT/Dj85OztxROPyW3FT+LAqHH1Tbtm1j9uzZPcZYLBZOOOEEhg8fHkpQqaqK2+1GkiRMou7PYaHyut/T+U3kE/o9adPTsZ9+Ov6GBto/+ijivC4zk9w33kCXmrI/hikIgnBYq6+fx6ZNdxBUXKE2WTYxcMC/SEubFmqrqXmHTcV3ROsCAI3GjKoqKIrnJ11fkvSoqq+HCA0TJ34f2i2vs7OElatmRK1nNaDwPjIzZ0S0B4MeWlp+wO9vw2IpwGodKkokCIIgCMI+EDWc9nPCqTMQpP+OwuFTEqy8IQqHH1SKovDaa69RVla219j09HROOeUU2tvb+eGHH2ho6N72OSsri0mTJtG/f//9PFrhl+j87jsqr7m2V/qyn3EGGQ/c3yt9CYIgHGn8/nbqG+bh8dRgNKSTmno6Op09LCYY9LBs+e9wubZGvF6StIwaOQe7fRReXwNlZc9SXd3zw6GfwmTMwWobismUTUPDZ7jd5VHjJEnHMRMXot+tKHlt7QeUbLkvLEFls41gyODHMJmye22MgiAIgnAkEgmn/ZxwAjhm6Sa2urwk6bSsE4XDDzqv18u8efNYv349O/9PWywWJk2aRH19PStXrmRf/q+fc845DBkyZK9xwsGhqioNDzxIy6uvRpyznnYa8WeeSfu8eXR88QWKyxWlh10kvZ6C5cuQ91IDTBAEQYjN621gU/HtNDd/F2ozGXMoLLyHxMRJoTafr5mFi46JOXPJYhlISvIpuD2VtLWtwuXa3mtjTE+fTk72ZZhMObS2LqFo7ZVR40zGHMaN+1/YroDBoJvKylnU1M7F663DaMwiI+NcsrMuFnWkBEEQhF8lkXA6AAmn328s5/0ddZxWHT2IDFHH6ZDQ0dFBbW0ter2e7Ozs0M509fX1fP7552zf3vMHWLGj3aFPVVW6Fi/G+e5c/JWVaFNTsZ95BtYTTkDasVxScbspv/gSPOvW9dhXxr8fwnbqqUja7nJ2XcuW0fLyLNyrVyPp9VimTCHxyivQZ4sn3oIgCD1xucpxubah08Vjs41AkiLrK9bWvr9jlzolrN1gSNuxS13331qPp4ZFiydFxO1Oo4kjGOz6yeOUJB2q6o95fvcleMGgh9VrLqGtLfIzZULCsQwf9gKyrAtrDwQ6qa+fh8u1HZ0ugbS0aWJnPUEQBOGIIhJOByDh9GxFA/ds6y6U+cqQPpySbN/LK4SDTVVV5s+fz9KlS3uMmzlzJn37imWSh7uGhx+m+cWX9hqnSUzEdtpU5DgLzf/9b8R52WYjd/ZrGAsL98cwBUEQflXa2tZQUfky7e1FyLKRlOSTyM6+JGzJG8Cm4jupqXk7ah9ZmTMpKLibQKCNtrbVMWcs/RwajQVH/FEYTZm43dU0N38dM3bggPvJyDg3dNzcspD1628kEGjfLUqmb/6t5OVdF7WPYNCL212ORhOHyZTZW7chCIIgCPvNT0k4iV3qfqZh1l3TrYs6XCLhdBiQJAmHw7HXOI/npxU3FQ5N9rPPpvnlWaDEfkIOEGxupvW12HVFlPZ26u6+h7y33ow4jNfAUQAAIABJREFUpyoKvvJyUFX0OTmhmVKCIAhCdHb7CIban9hrXGHBPUjIVNe8w84d8yRJS0bG+fTv/xckSUKniycpaTKJiZNpbo6+oYQsG8jPvxWvtx5X1zaaW76LGrdTMNhJUw9Jpt2VlT2DTufAaMpGQsPatdeiKO49ohS2bX8YkzmX1JSpu1oVP6Wlj1NVPSeUoLJah9Cv3+0kOI7ep+sLgiAIwqFOzHD6mTp2FA4HOCHBxpzh+fv1ekLvKC0t5dUo9X92V1hYyOmnn47Vaj1AoxL2l9a33qLu3r/DHn/njMOHk3LzH2j/4gs6Pv2MYFvbXvvK//RTDPl9QsfODz6k6emn8VdVAaBNSyPx6qtwzJgharoJgiD0Eo+3DmfrMgAcjvEYDJG7i3o8NaxcNQOPpyqsXZJ0DB3yJMnJJ4balq84i/b2opjX02qtBAIdvTT6XazWIRw1tnvXVFVV2bDhZuob5kXESZKWEcNnkZAwIay9rW0N5RXP09q6BEnSkJh4HLk512CxFPT6WAVBEAShJ2JJ3QFIOAVVlVGLN1DvC6CV4OwUB5dlJTPCZt77i4WDRlEUnn322dDudLHo9XqOPfZYjj76aGRZZvv27WzevJlgMEhOTg6DBw9Gp9P12IdwaHCvW0/rG2/g3bwZ2WbDNvVU7Gecgazvrrum+nxU33YbHfM/77EfXW4u1uOPxzzuKLzbt9P48H+ixiXffDNJ117T6/chCIIgxOb3t1Fd/SaNTV+gKF7s9lFkZV2MJS5899mmpq8pWntV1D6MxmzGHfU/JEmL11tD0dprcLm29doYtdp4TMZMNNo4nM5lMeOs1qEcNfbD0HFD4+esX38TqhoIi5NlEyNGzMIRPzaiD7e7kqbmb1HVAPH20dhsw3rtPgRBEIRfN5Fw2s8JJ6+icNm6Ur5uiXwC9vd+GVydHfn0TTh0NDU1MXv2bNr2mNVis9lQVZWOjl0/1/j4ePR6fUSCKj4+nosuuoikpPCaE8Lhyfnee9T+5a+90pdkMND/u2/RxMf3Sn+CIAhC76qtfY+SLfeF1VqyWocydMgTmEw5obaa2rls2vTnmP3k5V6P3pCMx11JXf1H+HxNvTbG5OSTiTPno9MnsW3bIyhK9ALpJlMeR49fECrSrigBSrbcS3X1m8Cuz/gOx9EMHfIkOl1kaQFVDdLWtppAoAOLpVAUORcEQRB6JBJO+znh9FhZHQ+U1sU8/+WYAoZYxUynQ5nX62XdunWUlpYiSRL9+/dn0KBBqKrK4sWLWbhwIYFAoMc+EhMTuf7665HlyJ14hMNLsLOLrZMno3REX0Yh2+2oXi/qPtb3Sn/gfuLPOCN0HGhupuW12XR88QWKx4Np6FASLrkY8+jRvTJ+QRAE4acJBt00N39PINBGXFwBNtvwiOXQqqqwcdP/UVf3YcTrs7JmUtD/7tBrqqvfonjzX2Jez2BIQ69PwuOpwe9v6dV7yc//I0mJkzGZsikre5ryiuejxsXHH8WokW+E3WdD4+dsKbkPj7dmR4tEcvLJDBzwT3S66A9OVFVBUTzIskksIRcEQfgVEgmn/ZhwUlWV0T9upMYbe0vdSzISebBQbKN+OHM6nXz22Wds3ry5x7gLLriAggJRP+FI0PHNN1Tf9AdUf/jvtjYjndzXXkOXkoJ7/XpaZr9Ox2ef9djXzqV7lkmT0OfkUHHlVQRqa8ODJIm0v9+LY/r03r4VQRAEoZeoqkpT0wJqaufi9dZhNGaSkXEeiQmTwpItwaCHFSvPprOzOKIPjcbMmDHvh5b3VVXNYXPJ3w7YPexu1Mi3cDi6l+A1N3/PmqIrgMjNNWzWYYwe/S6yvGszDL+/le2lT1BX9wGBQAd6fTIZGeeRl3sdGo3xQN2CIAiCcJCJhNN+TDi5gwp9vl/bY8yxDgvvjui3X64vHDjbtm1j9uzYu5cBTJ48mUmTJh2gEQn7m3frVlpmv4579WokvR7rCVOIP/98tLvtbugtLWX7qVN76GUPshx7pzytln5ff4UuRSzDFQRBONz5fC2UlNxLQ+P8UL0lu20kBYX3YLMOCcUFgy4W/zgFn68xaj+5OdeSk3MZHm8dDQ2fU17+TK+NUZI0GAwZGAwpdHVtJRCIvWnG0CHPkJJyMtBdI2vFyulRa1rFx49j5IhXkGV9WLvTuYLq6jdxucsx6JNISz+T5KQTQ8v/BEEQhMPTT0k4iT28fyKDLGHVyHQEY2+1nqwXxaSPBHq9fq8xW7ZsYdiwYTgc4TUR2tvb8Xg8oRpQwuHB0K8f6ffe03NMnz5Yjj+ezm+/jXpek5AAikLQ6exuiJVsAggEaP/kExKvuCLUpHR10fLGG7R//AkBZyuG/L44ZszAevJJYumCIAjCIUyvT2DIkMfx+ZpxuyvQ6RIwm3Mj4jQaM8OHv0hR0ZURSaeUlNPIz78ZWdah1ydhiSukru59vN7opRxMplxysq/A7amgvW0tzrbYxcihu16Tx1OJx1O51/spLr6TuroPMBjT6eosiVlA3elcSl3dx2RknBNq2176BKWlj4fFNTYtICVlKkMGP4YkacLO+f1Oaus+oLNjIxptHCkppxFvHyPe9wRBEA5zYobTz3BHSRWzqmMXhnxzWD6TE2377frCgaEoCo8//nhEcfE9ybLMsGHDOPbYY3G5XHzxxRdUVnZ/kNPr9YwePZopU6aIXe2OIIHWVqquvQ53UfjW2oYBA8h+/jm0iYm4i9bifG8ube+932Nf2pQUHDPOJ27CBHS5uVRefgWejRsj4hIuuYTUO27v1fsQBEEQDp5g0EV9/f9o71iLRmMmJWUqNuuwiCSL07mCNUWXEwyGFw7X6RIZNfJ1LJaCHf15WbhoAoGAM+Y1rdZhBIMdeDx1KIq71+7FYEgnJ/syDMZ0/L7WHpcMFhTcQ3bWzNBxa+tS1q69hkAwvI5iSspUBg96BFkO//ykKAGam7/B6VyOJOtISpyM3T5aJKcEQRAOELGkbj8nnBp9fqat2kKp2xdxbkqClTnD8sWb3hFi48aNvPPOO1HP6XQ6/LvV+9n5M4/2O9W/f39mzJghCowfQVRFoeuHH+hcuAgUhbijx2M5/ngk7a6Jo4GmJrYcN6nnWU67kfR6VF/k35Wdcme/hnls+PbXqqriLS7GX1uLLi0Nw8CB4u+PIAjCEcbtrqKq6jVaW5eAJJOYeBxZmTMxGJLD4ioqXmbL1n9G7SMlZSpDhzwJdD9U+3HJlB5nOmk0ZoJBV+/dxA4GQxqDBz2CwZCCLBtZuvTUiGTTTn3ybiQ//+bQscdTw5qiK+jqKgmLS0ycxNAhT6HRRG7a09W1neaW71DVAI748dhsQ3v3hgRBEH5lRMJpPyecAJp8AZ4sr+e9+laa/YHQxrOnJNl4ZWj+fr22cGBt3ryZr7/+mvr6egCsVisTJkxgzJgxrF69moULF9Le3r6XXmDmzJn07dt3fw9XOMRUXn8DnV99Ff2kJHXXeAoG96kv2xlnkPnA/aFjz+bN1N75FzwbNoTaDIMGknHffRgHDfpF4xYEQRAOP6qqUlH5ImVlTxMIdCdxJElLevrZFPS/G43GEIqtrnmb4uI7o/ZjMKQxftwXSJKM11tHUdFVuNylB+QedqfRWBkz+h2MxnRkOY4VK8+go2ND1Nj0tLMZNOih0LGieNlUfGfELoMJjokMGfIkOp09oo9g0EVLy0ICQRdW6+BQoXdBEARhF5FwOgAJp90FFYXJy0socXVvmf7dUQMojBO7dRxJVFWlra2NYDBIfHw8Gs2u2gOBQIAVK1Ywf/78HvsYNWoU06ZN299DFQ4x/poayi66iEBN5C516ff9A+tJJ+FaupTOHxbifPvtnjvTaIg7ZiLmkaPQ5+VSe/c9KFGWfMo2G33efx99VmYv3okgCIJwuAgGXTidy1EUPzb7CAz6pIgYVVUpK3uK0rKnQkXOAczmPgwb+ixxcbs2wKmpncumTX+Oeb1BA/+DXp+Ax1tLefnzuN1lvXo/ALJsQFG8PURoOOqoeVji+iFJMsWb/0Z19ZyokQkJxzJyxCthbVVVc9i2/d+hRN3OuMGDHkYf5d8PumdP+XxNmMw5GA1pP/WWBEEQDksi4XSAE04A79a1cOOmCgCmpzl4cmBkkUjhyNXW1sajjz7aY0xaWhqXXnopRuOuZGQwGKSsrIyOjg4SEhLIzs4Wy6GOQIGWFlpff52OBQtQXG6Mw4aScPHFmEeODMWoqsqWY44l2NzcK9d0XHQRaX/9S1hb56JFtL7xJt6tW9DEx2M/7XTiz52ObBQJckEQhF8rr7eBxsYvCAQ6sFgHkphwbERRb1VVKd78V2pq3op4fb++t5Gbe03ouL5+Hus3/CHm9ZIST8AePxqvt57mpm9weyp672bontGl0yXsKMge+3vOmDHvY7cNB6C27kM2bvxj1DirZTBjxryPLO9aMt/evpbNm++mvWPnztUSSUknMKDwHxgM0XeedburcHsqMRrSMJv7/Kx7EwRBOBSIhNNBSDj5FZWjl26kyuNHK8GP4weRbRS7k/1aKIrCI488QmdnZ49xer2eESNGcNRRR9HW1sZHH30UthwvJSWFs846i7Q08ZTs16jhkUdpfv75mOd1mZn4q6v3qS/Z4SDnuWcxDhiApNfT+PTTND35VESccfgwcl56GY0l7mePWxAEQTjyqapKq3MJtbXv4fM2YjLnkJFxHjbrkD3igqxd93uamr6M6MMSV8jo0W+j1VoBaHUuZ9Wq82Ne02TKISlxCl5fAx0d63G7ezc5pdcnodcn43KV9VhEfejQZ0hJPhmAzq4trFhxVtT6VmZzPmPHfIhWu+s91e2upLj4r7S0Lgy12e2jGFB4HxZLYdTrBQIduN0VaLU2TKbsn3t7giAI+4VIOB2EhBPArOom7iipAuDyzCT+VZB1wMcgHDwLFy7kyy8jP1z9VGazmeuuuw6r1doLoxIOJ0pXFxWXXxGx+x1A4jXXkHLLzQSdTlyrV1Pz59tR9qF2mKTXo+/TB+/mzTFjEq+8gpQ//Smi3V9Tg3v9emSTCfOYMcgm00+7IUEQBOFXSVH8VFW9RnXNm7hc5ej1SaSnn0VuzjXodLt2clZVlfUbbqKh4dOIPmTZyOhRb2KzDQPA7a5m8Y+TiDVrSZZNpCSfhM/XhMtdhsezbw9o9oUsGzGb8zEYkunqKsXTw6ysgoK7yc66GACfr4Vly6fh9dZGxGm18Rw19sOwhFIw6GLL1georX0PReku1WGzjaB//zuJt4+Oej2vr4mO9rVIsp54+xg0GjFrWRCE/UsknA5SwskdVBj740aa/AGMssTyoweRrNft/YXCEUFRFD755BNWr14d1m4ymZg2bRq1tbWsWLECl2vvO74cd9xxTJkyZX8NVTiEKR4Pznfn0vbJJwRbWzHk5+O4YAaWSZPC4mr+8hfa3nu/V64p2+0U/LgYaccuisHOLur+9jfa588P7bAn220k33gTCRdd2CvXFARBEAToTk6VlT1NVfUc/P4WAByOCfTr+3+hZNNOJSX/oLLqlaj9DBzwABkZ0wEIBLpYuOhogsGumNeNjz+KQKATr7cWv7+1d26G7uSU1ToEvT4Rr6d2t2V3kTIzL2RA4d+B7plhq1dfTKtzSZQ+DYwe9VbYv0cw6KGk5F5q694P1eDSau3k97mJrKxLopZoCAQ6aWtfA6qCzTYiLPknCIKwr0TC6SAlnACeLK/nn9u7n2L8ITeVO/LTD8o4hIOnpqaG9evX4/V6SUtLY+jQoaG6TYFAgPXr1/PRRx/R0+9eRkYGV1999YEasnAY8pSUUHbOdFSfL/KkVkvyzX8g2NiEu6ioe8bUXv7Wm0aMwDxmNMYRI2h95VVcMf6Gpv3j7zimT49oV9xuPJs2AWAcNEjUhRIEQRB+EkXx4fXWo9HEodcnRI1RVYWy8v9SWTkrlCQyGrPJz7+Z9LQzwmLLyv7Ltu0PR+0nPX06gwY+EOpz8eIpeLyVMcdmNGajKB58viZ6qgv1U0losNtHoTckEwx6aG7+OmbsnoXO1677PY2Nn0eNLSi4h+ysmaFjVVUoLXuaiooXQkk4WTaRnTWT/Pw/htWn2ikQ6KClZRHBoBubbRhxcWKnZUEQuomE00FMOLUHgoxevIGOoIJNK7Py6MFYtZq9v1D4Vbn//vvxemPvtGI0Gjn//PPJzc0NPaHy+Xxs3LiRlpYWLBYLgwcPJi5O1N35Nev87jtq/nw7Qacz1Cbb7WTc/y+su82Qq7nzTtre/6BXrqlNS6PflwuQtN0fTlVFofn5F2ie9TJKW3toDElXXUnCFVeIIviCIAhCrwsGvbhcW5EkLXFx/SKKnEP3cr3yiucpL/9vaOc5WTaQmTGDfv1uR5Z3rUKoqXmXTcW3R71WXFwBR439GFnWoSgBVqycTkcPs5a0WhuqGuxxdtXPZbUORq9PBiSam7+JGafTJXLMxIXIcnc92W3bH6WsLLKOI0BW1iUUFvwtdKyqKhWVL1Fa+kTYPSQmHs/gQQ+j0zki+vB666mv/x9+fwvmuH6kJJ8ilvYJwhFMJJwOYsIJ4F/baniiogGAv+Snc2Nu6kEbi3Boevvtt9m0YzZIT1JTUxk/fjxxcXF88MEHuN27ClpqNBpOPfVUxozZp9914QileDx0fPUVgdpatGnpWH9zQsTsItfq1ZTPuCBmH5qkJIJtbeD379M1Ey69FMtxx2IYMICW116j+dnnosYl3XADyTdcH33cPh8EAshm8z5dUxAEQRB+jmDQhbNtFaoawG4bgU4XHzWuouIltm1/NKx4uN0+hqFDnsBg2PVZvrHxS9auuyZaF8iyifHj5mMyZREMutm46XYaGubFHJssmwA1VK+pNxn0qRiNGWh18TQ3fw8EY42CYyYuDN1jVfUbbN58V9RIu20ko0e/gyTJobay8ufYtu0/Yf3rdIkMG/oM8fGRn1Hd7kpqa9/D46nGYEwnPe1szGaxu7cgHE5EwukgJ5wafX7G/rgRj6KSrNeybPwgTBp57y8UfjWqq6t56aWXUHbUx9mdJEk9Lrfb00UXXUS/fv16c3jCEaj+wYdomTUrot1QUEDu7NeQDAY8GzbQ+PjjuJYt75VrSkYj/b/7Fo3dHmpzr99A01NP0fn996AoGPr3J+GKy7H/7ndiNpQgCIJwUAUCHTQ1fUMw2IXVOgSbbWjUuIrKWWzd+kCodhJ0FwEfOuQJEhImhto6u7awbNlvUdVoD3QkRo18g/j4sQSDXZSWPkVF5QsxxyZJWjQaC4GAM2bMz6XVWjEaM9FpHbS1r+4xATZ82AskJXXPoq6vn8f6DX+IGqfRWDh6/AIMhpRQW2XVa5SU/APY/fOvTP9+d5CTc3lEH23tRVRXzaGraws6XTypqdNITT09bHbanhTFjyRpwpJigiD0LpFwOsgJJ4A7SqqYVd0EwAMFWVyamXRQxyMcejZv3szHH39MV9eu6coOh4Ozzz4bn8/HkiVLKCkp2Ws/+fn5XHzxxWFt1dXVLF26lLq6OgwGA4MHD2bUqFHo9fpevw/h8KCqKh1fLKB1zhy827ahiY/HfvppOGbORGOxhOI6v/+eyqujP7n9ORwXXYhjxgz0eXm4i4qouOxy1CjLSZNuvIHk6yNnQyldXXT+8ANBZxuGggJMI0eIxJQgCIJw0Hm9jdQ3zMPna8ZsziM1ZSoaTeSs3camr9iw4VaCwc5QmywbKCz8Oxnp54TafL4WflxyAoFA9B1oC/rfRXb2pShKgMamr1i//vc9jE7GZMrG52sOu25v0Wqt6LQOvL7GsBlhe8rMnEl+n5vQ6ew4nStZtXpGzNiRI14LS9ZVVr5CyZZ/RMQ5HBMYPuxFNBpDWHtd3cdUVL5IR8cGJElPcvJv6JN3IxZLQdTrqaoS2snQaMwUCSpB+AlEwukQSDhVuL0cvXQTQRVyjHoWjxuIVhZfkoRwgUCArVu30tnZicPhoE+fPsjyrje85uZmXn755bCk1J4kSeLss88mNzcXq9XKypUr+eSTTyLiUlNTueSSSzCLJUxCD1RFoeyc6Xg2box6PuHqqzCPHIm3uJiOr77Cs37DPvUrmU2gguqO8cFUlun35QJ0GRmhJud771F//wMonbs+LBsGDSTr0UfR54rp94IgCMLhIRDopKHhU9zuSgyGNFJTp0atheR0rmDtumsjds3LyrqEgv53hR64qKrKipXTaW9fHdEHQF7udfTt+ycAurpKWbL0RHoqdm425xMMdOHzN6GqsZbe/RIykqSJMdOrm802nIL+f0Wnc+D1NrFq9fkxY/vk3Uh+/s2h49LSJ9le+lhEnEZjZtTINyJmqtXUzqWs7Gnc7goATKY8+vS5MaLw/E7BoAdn2wqUoAebbVjYrC1B+DUSCadDIOEEcMPGcubWd79hPD0wh7PTou+4IQg9mTVrFuXl5fsUa7fbaWtri3l+1KhRTJs2rbeGJhyhAo2NVP3hZtyrVu1q1GpJvPQSkm+9FWlHUtS7fTvbp57Wa9e1nnwSCRddhC43F/fatVTfcGPUOF1GBvmffIy8R9F8X3k5bR99hL+uHl1mBvFnnIEuM7PXxicIgiAI+1sg0EV9wzw6O4vRam2kpp6OJa5/RJzP18S6dTfgbNt9GbxEZuaFFBb8LayQ+oaNf6KuLvrmIUlJv2H4sO5ajMGgn8WLj8Xnb4w5Ppt1OJIk4/c7cblLf95N/kKybKJv3z+i08ajqAGKi+8gVkLNbh/FmNHvho4rKmexZct9UWP33N0PupcBbt/+GIHAzs/XGtLTfkdh4b0RM9oUJUBD42fU18/D73disRSQmXkhVsuAn32vgnAoEgmnQyThVNzl5vhlmwEYEGfk67GFyGIpiPATLVq0iAULFvRKX1qtlttuuy1iaZ3f76epqQlZlklOTg6bZSX8OqmqimftWtxFRUhGI5bjj0eXEvlEr/K639P5TfSdcszjx2ObeiqeDRtxLV+Ob/v2fR+ALEOUGmc7pd39Nxwzdk3Nb37pZRoefhh2f0/TaEi76y4c55+379cVBEEQhMOEqqq0t6/G2bYKWTaQlDgZkykrIq67gPltNDR8GtaemHg8QwY/hlZrDbVVVs2mpOSeqNezWYcxZsx7oeVnRWuvoanpy5jjS3Acg96QhN/fSmvr0v1SHH1fJCediNGUjUZjprz8eVTVFzVOo7FwzMTFaLXdD7Sqq9+kePNfo8YmJU5h+PBdNbeCQS9r115NS+vCPSJlBgy4j8yM8M8iqqrS1PQlNbVz8XprMRqzyMw4j4SE42KWDlDVIH6/E63WgiwbosYIwoEgEk6HSMIJ4NJ125nf1L0W+7WhfTgpyb6XVwhCOI/Hw3PPPUdra2vEOZ1OxxlnnEFbWxsVFRVs2bKFYLDnqdBTp05lxIgR6PV6FEVh4cKF/Pjjj6Ed8OLj45k8eTLDhw/fL/cjHFmC7e1U3fQHXEuWhLXHTZxI5uOPhepDBZqb2XLscT0mkX4KTWIi9tNPQ5/fF6Wzk4Z//zt6oCSR+8YczCNHhjX76+pwvv8+vtIyNI547NN+h2nI4F4ZmyAIgiAcijo7N9PSshAVhQTHBKzWyPc9VVUpL/8vpWVPoSi7ai46HBMYMvhR9PpddWk7OjawYuW5URNJlrhCxox5H42me+fc0tKn2F76aMyxJSYcj90+Ar/fSVPT17g9Fb/kVn82kykPszkXrdZOY+OCHmtUjRjxKo74cciyjm3b/kNZ+TMxImWOHv8FZnMfoLt+1MZNt0WddZaVNZOC/neHJZ2CQS9lZU9RXfMWfn8LkqQjNeU0+vb9I0ZjRkQfAH5/G51dJWg1ZiyWgaJGldCrRMLpEEo4rWrrYuqqLQCMsJp4pDCbNKOeBJ32II9MOJy0tbXxySefsHXr1lBbeno6p59+Opm7LRn64osvWLx48V7702q19O/fH7/fH9bn7qZNm8aoUaN++eCFI56qqrhXr6Hrx8UgSVgmTMA4fHjEE7rqP/6J9v/9L2ofst1O6p13EqipxrNlKx2ffho17uewTZ1K5iP/CR23f/opNX++HdUfXkvCcdFFpP7lzqhPFoMdHXi3bkU2x2Eo6C8KlwuCIAhHNL+/jebm7wgqbmy24TGXhTmdK9hcci+dnTtrP8qkJJ9MYeG96PWJu/XXzoqVZ+FyRS7DM5lyGDvm/VBdq5bWH1m9+qKYY0tOPonMjBmh5FR9Q2Tt0gNJknQ7diyM/b06IWES2Vkz0esTcTqXsWXr/TFjhw19luTkE4HuZXpFRVdEmTkFen0qY8e8h9GYHmoLBt1s2fovamvmouyYyWUy5dC/3x0kJ58U0Ucw6KGh4X+0ti4BSSYxcRLJSSf2uBOgIIiE0yGUcAL47coSlre7Qsca4JRkO3/vl0mmUewaJuw7p9NJS0sLFouFlCjLmxoaGnjmmVhPV36auLg4brnlFrTa8OSooig0NTWhKApJSUkR5wUhlqDTScUVV+LZEF5oXLZayX7uWcy7JTjLzjsfd1FR7M4kKXz5XE80GuKOPhpDYQGaeAeNjz4ac6ZV2j/+jmP69NCx4vXS8NC/cb73Hqqn+wmuPi+PlD/fhnXy5H27viAIgiAcwVRVpatrC35/K2Zzn5hFtb2+JrZuuZ/6hv+hqn4kSUtKylT697sj7DWqqlJcfCc1te9E9GEy5TB69LsYdsy0CgQ6WbT42Ji7+6WmTKOg4C78/lba2laxqfj2Hu9Fq7ESCHbSU/JofzMZc8jKugit1kZXVwkVlS/HjM3IOJ+BA/4JdP+7Fa29kubmb6NESgwf9jxJSVNCLW53BavXXBIqnr6T1TKYESNeDpvNBuDx1FJZOYum5q9RlADx9tHk5FwedaZc93iCtLWvIeBvx2IpjDkbSzj8iITTIZRwavUHmLS0mAZ/IOJcllHH/NGFJOnFF3ah93z77bd8++23Ee0Y3clAAAAgAElEQVQ2m43TTz+dyspKNm7cSHNz8177mjlzJn379g0dr1mzhm+//Ran0wmA2WxmwoQJTJw4Ucz4EPaJ4vPRMX8+HV9+heL1YB45kvhzzkGbFP6hpvP776m85tqoSSVdbg55b79NoL4e37Zt1N5zL0p79A+aP5U+L4/8Tz5G0ulQVZWq318fvUaVLJP932ewTJoUcUpVVQINDUiyjCYpSfxuCIIgCMJuAoEOfL4m9PqksPpRu1NVhZrad6mqmk1X11Z0unjSUqeRm3tN2MwpgJaWRRStvSZi+ZvFMohRI2ej08WH2laumoHTuSzqNR2OCYwaORtVDeL1NrBk6ckEg7F3inY4JqDRmAkGOml1LuXgJKlkkpKmoNM5CAbdNDTMixkZZy5g3LhPkSQJVVVZtnzabjPTwiUmTmLE8F2Jrs6uLaxadQF+f0tYnCRpGTL4CVJSTg5rb2z8gpKSf+Dx1oTGmZx8EgMH/DPs5xHqv3MztbXv4fU1YjLlkJE+PWo9MuHQIBJOh1DC6eHSOh4uq4t5/pbcVP6cnx7zvCD8HJs2bWLp0qXU1tZiMBgYMmQIEyZMwLKjno6qqsyfP5+lS5f22I9GoyEvL4++ffvi8Xj4/vvvo8ZNnDiRE088sdfvQ/h1a/tkHvX/+hfB3eqXmUaOJPPhf4ftPtfwn0dofuGFaF0AoHE4CDqd+z4jSpbRpqWisdnxFhfHDDMUFtLnww/CEkrODz6k+bnn8JWVdccMHEjyDddjPeGEfbu2IAiCIAg/mcdTQ3X1m7S1r0EjG0lOPpnU1N+i0YQX1/Z661mz5jI6uzaHtVssgxgxYlZo5hTA9tInKC19POr19iyg3tNOgAB98m5Cp3fg8zVTVTV7t13vDixZNqDVWpEkLV5v7O+oAIUFf8diKUSrtbFx0210dKyLGtddbH0RWm3394zm5u9ZU3QFEDmb3GYdxujR7yLL3RMuVFVl+/ZHotS/0jBwwH1kZJwb1qooAaqrX6e65i1crjL0+iTS088iN+eqmMlLl6t8x66PFuLjxyLLYoXRLyUSTodQwun4ZcUUd8XekaG/2cAP4wYewBEJQrfS0lJeffXVXulLkiRuueUWbDZbqM3lcrFkyRI2btyI1+slIyODcePGkZ+f3yvXFH4dFK+XrsWLUdrbMfTvj3HQoIiYQEsLZedMx19TE3FOn59P3ttvIWm1eLduo/auu3pMIv1UqXf9Fcsxx6DLzqZl1isxi5dnPPQg9mnTwsfd2EjL63Po/OYb1EAA06iRJFx8McaCgl4bnyAIgiAI4RTFT2PTl7S2LAIgIeFYkpJOCCVBdlJVhS1b/kll1Wvsnjyx20czdMjTGAzJoTaPp5YVK6fj9dZGXC8j/VwGDPhX6AFVecWLbO2hhlNe3g044scRCLRTVTWbVueSmLEgcTCX/+1kt4/Bah2MVmuhtvb9qP8OOw0Z/CSpqVMBqG/4jPXrb4gRKTF2zAfYbEOB7p/HuvU30Nj4eUSkxTKI0aPeCEs6+XwtbCq+naamr0JtOl0i/fvfSXraGRF9eDw1VFW/gdO5DEnSkpQ0hYz0c9HpbBGxO/n9bahqAJ0u4Vc1o10knA6hhNNRP26kwhN9602ADIOOVRPEzkjCgaeqKs8//zy1tdHfEJKTk5Flmfr6+n3qb+zYsUyePBmz2Ux7ezsvv/xyaOnd7k455RTGjx//i8YuCHvy19VR/+CDdCz4EgIBJJ0O29SppNz2f2gTd029b/vkE2r+77aY/ej79EGXmYm/uhpfefk+76onmc2obnfMWVSaxET6f/M1kr77qZq3tJTymRcTbGoK70enI/OJxyPqQ6mBAB1ffkX7Z5+hdLRjGDAQx3nnos/N3afxCYIgCILw83g8NTQ1fY2ieLHbR2KzjYyaXPB66ykte5r6+o8JBDqIi+tPVuZMMjNnhO0SFwx6WL3mYtraVkb0kZBwLMOHvRAq2t3ZWcLSZacRbbYQdM9CysiYjt/vpKb2PbZvfzjmfUiSDrt9JMFgF15vPT5fU8zY/U2WTWi1VgKB9qi7HO5kt48hN+dKtForbe1FbNv2UMzYvLwb6Jt/C9A9E2rFyrPp6FgfNXbo0GdISd61DNDpXMGaoisIBjvD4ozGLEaNnBOxvK+5+Xu2lz5Be/tqAMzmfuTlXRc1kaUoAZqavqS+4VOCQRdW62AyM84PK/a+J1VVUdVgRBL0UCESTodQwunK9aXMa4w9ZfLUJDuzhvY5gCMShF3a2tqYM2cODQ0NYe39+vXj3HPPRa/X09nZycKFC1mypKenK7vEx8ejKArtMWrqSJLETTfdhMPhCGtXFIXq6mq6urpITEwkOTk56usFoSfB9nYCzc1ok5PR7FhCujvV76f8sstwr4j8kKdJTCTv7bfRZ3Uv13N+8AG1d9zZa2OzTJ6MaeRIdOlpNL34Ir7NJVHjZJuN/t9+g2w2A6B4PFRedx2uH/f4HdRqyXz439hOOSWij2BHBx1ffEGgsRFddjbW3/wG2WCIiBMEQRAEofepqhKWZNpTMOimsnIWNbXv4vXWYzRmkpFxHtlZF0cs+aqpncumTXcCwbD27plT/wxdJxj0smTpiXg81VGvmZ9/K33yrgfA729l4aKJKIo3aqxGY6Fv/q0Egy5c7jJqa+fu660fNLJkICn5BLRaKz5fM01NX8aMNZv7MnrUu2i1cYDCosWT8PkaosY6HEczauTroeOGhvmsW38D0WaW9et3O7k5V4WOg0EXRUVXRcxSk2UjQ4c+TVLi8WHtPl8LZeXPUFv7AYGAE5Mxh8zMGWRnXxZ150BVDdLSshC3uwqDIZXExOMOyJJBkXA6hBJOy5ydTFsdfdt5gNeH9uE3SfYDOCJBCBcMBtmyZQvl5eVoNBoKCgrIzs4Oe3LT1tbGY489Rm/9vTjuuOOYMmXXLhmlpaXMmzcvrJB5bm4uZ5xxRkRiShB+KcXlovHxJ3C+9x5KZydoNFinTCHl//6EPidnV5zXy7ZTTyVQE30WoP3sszEOGoi3uJjOHxYSqOu5FsJPYT/rLGwnn4Q2JQXn3Lm0znkjapyk09H3ywXoUlNDbW0ff0zt3fd0z7jaQZOQQOYj/yEuyuxCNRCgc+FC/BUVaJOSsEyejGwy9dq9CIIgCILwy7hc5dTUvI3LtR2dPoG0tDOJt4+JmGnV1bWNorVX4XaXh7VnZl5IYcHdSJIm1FZe8QJbtz4Q9XqDBv0nbLbOihXn0LZjNs+eZDmOcUd9AqgEAp0Urb0any/2CgmLZQBarZ1AoIPOzs3smUg70CRJh6r6e4zJyb4CkykXjSaOki33EQi0Ro2TJT3HHLMYna77+8vmkr9TVRW9hIlGY2bihO9DsX5/KytWTsflKo2ITUr6DcOGPhP282trW8X6Dbfg8VSF2vT6JAYOeICkpMidlFtaFlNV9RqdnZvR6mykpp5OZsYFO5JukdzuCtra1iDLehyOCWFLC0XC6RBKOAHMrmnijpIqAlH+qc9JdfDkwJxf1ZpP4fA0d+5c1q+PPi01MTGRYcOGUVtbS2VlJV1dsXf0ANBqtfTv35/s7GzMZjMff/wxSpSlS3a7nWuvvRbTHl9+29vbWbNmDU6nE6vVyvDhw0lISPj5Nyf8Kik+H8HGRmS7PepsKADv1q1UXnsd/qqqsPb46dNJu+duJE33G79r+XLKZ16838ccjf3ss0i67vfoUlNwrV5NxSWXRl3aJxmN5H/8UVhSzV1URPUtt4bVv5JtNtLvvQfbqaceiOELgiAIgtCLFCVAc/PXtLevRaOJIzn5ZOLiotdQramdS1nZf3G7ywCwWAaS3+cPJCeHbwbUvUvdDPz+8ERLtF3qqmveprg4+gxxgyGN8eO+CCU5Srb8k8rKl6PGAqSnn4vdNoxAoIPa2vfocsWeyHGo0Gji0GntSLJhR+IvdnmG5OSTSUk+Ba3WSl3dh9T3sMvgkCFPkZrS/dnM7a5i6bKpUXdRlCQdY8bMxWYdEmorL3+OrVGWI1osAxk1cg463a4JMIFAJ5uK76Ch4dNQmyybyMu9lry865EkSSScDrWEE0C1x8fculaqvD7sWpk3a1tp9gcAeGxANuenJ+6lB0E4uLxeL2+99RalpeFZ99TUVC688MJQwXCPx8ODDz7Ya7OhTjrpJCZMmBA6XrNmTdQE1YknnsjEiRN75ZqCsDvV76fjq6/xrF+HZDZjO+kkDP36hceoKqVnnY1306aofZiPmUj63/6Gv7YOd9EaGh95tPcHKklIOh2qL3bdwPjzziXtnnuQJAl/XR3bfzsNpaMjMlCWyZ39GubRo8Oa2xcsoPW12XiKi5Hj4rCdcgqJV1yOViyBFQRBEITDkqqqeL11SJIGvT455kQIj6eWyspZNDV/jaL4ibePITvnsrDExs7+SsueoqzsKVQ1EGo3m/swbOizxMXt+gzl9TayfMWZUYuMW62DGT3qbTSa7gfPra1LWLX6wpj30b/fXWRkTCcQ7KCu9gO29VDPSqOxkJQ0pXvJoKsUl2tbzNhDhVYbj90+Aq3GQmdXCV1d0UszACQ4jqWw8B40Wgsedw0rVp4ZMzYrcyaFhfcA3T+7oqLLaW6JvjP5ziWDIuF0CCac9rSwtYPpa7ahAiZZZv6YAgrjjAd7WILQI1VVKS8vZ8uWLQSDQfLy8igoKECWw9eo9zQbCrpnLrW17dt2sGazmTFjxpCYmIiqqnz44YcxYy+44AIK9tjhS1EUtmzZQkNDA0ajkYEDB2KJMZtFEH4JX2UlFZdfgb+yMqzdMHAgOS++EFa8vPTc8/CsXRu1H8lgIO3ee1E6OvBVV9P66qsxi5H/LDod2sRE1GCQYGNjzDDL8ceT/ex/Q8eNTz1N01NPRcRp09PJm/M6uoyMsPauJUtofuFFXCtXImm1WCZNIvGaq8UufIIgCILwK+D1NtDY+AWBQAcW60ASE44NWxK2k8dTw9atD9LQOB9VDaCRzaSln0Hf/P+L2CFu2/ZHKCt7OqKP5OSTGTL4iVCR7WDQy9Jlp+B2V0Qd28AB95ORcS4Afn87CxdNQFHcUWONxmwGDXyIYLCTrq7tbN0We4dBAJOpD7KsJRh0xayndSiRJA2pqdPQai0E/B3U1cf+rqXTOZg4YRFarVEknA71hBPAQ6W1PFLWvb51QJyRT0cXYNbELi4nCIeLjo4OZs2aRUtLS8S50047jbFjx9LR0UFFRQXvvvtur103Pz+fiy/etaypvr6et99+O2wcsiwzefJkjj322F67riDspHg8tH82H9eSJaDRYDnuWKwnnICkCy/06Nm8mfKZF6PsWVxfoyHjoQexn3ZaqKnqxhu7d9+LIf7c6YCEv66WrkWLIdh7tRBMY0ajz8pGjjPHrCMFYJt6KpmPPBI6dr7/AbV/+UtEokwyGsl5+SXMo0aFtauqiqeoiI7vvoNAEPNRRxE3cQKSHPs9MdDSguJyoUtNjfj3FQRBEATh8BIIdOL3O9Hrk9BoYk/EaG1dQnXNW7hcZRj0yaSln0VK8skRRdrd7grWrruezs6NoTZZ0tOnzx/Izb0mbCZXTc07bCq+I+JasmxkxIhXcMSPDbWtKbqC5uZvo47NbO7H+HGfhcayevUltLQujHkvBf3/hk4Xjz/QTmnpE/j9kd+ddpIkbdiMsYNl9Ki3cTjGioTT4ZBwCqoq56zZyo/O7rWXF6Yn8J8BOXt5lSAcHtxuN8uWLWPjxo14vV4yMjIYN24cuXts4/7qq69GLNP7uSRJYvDgwaSlpZGYmMi8efNi1pM644wzGDFiRER7IBCgpaUFrVaLw+EQ9dWE/cZXVU3LrFl0fPM1qt+PefQYEi+7FNPw4eFxZWWUzbiAYGtkgcr4c88l/e/3ho5r7riTtg8+iHlNw4ABaBzxBJua8G4v7b3klCSRcOUV6NLSkM1mau++B2Is7TP070efjz8O/W4pLhdVt9xC13fh07eNgweT/ex/I5bruYuKaPjPI7iWLQO6C6I7LryApGuuQdJG3z5Y8XgItrejdThEckoQBEEQfiVUVcXpXEZH50a0GgtJSSeg10ev+9rc/APlFc/jdC5HkrQkJU0mL/f3WK0Dw+K8vibWrLmUzs7wMgoGQzojR7xKXFzfUFtHZzErV55HMNgZcb3dl7IBVFa+QsmWf8S8lzFj3sdmHUow6GLr1oeorpkTMzYurgCHYzzBoAunc3lEEflfQiScOHwSTgC1Xh8nLN9Mi7/7Q/8zg3I5K1XsyiX8emzbto3Zs2dHPWcwGLj88svx+Xw0NTXx1Vdf0dkZ+Qf750hKSuL666/f9aVXUVi0aBE//vgjLpcLgOTkZE444QQGDBgQtQ9FUWhpaUFVVRISEtBoIqcJC0Jv8FVW0vTU07TPn4/q9aLv0wfHRRfimDEjbBaQd/t2yqafixIl0apNS6PPB++j3bHzY9Ozz9L42OMxrylbLKh+P6o3+pbJv0TSDTcQN34cuqwsGh55lPaPP44aZxo5ktw35oR+T91r1lB+yaVRx2T77W/JeOjBsCSxv6aGhkcfo2P+fFS/H9luI/6cc0i+4QaxE58gCIIgCD+LovhoaJhPc/N3KKofh2M8aam/i7rjW2fXFkq3P05j0wJUNYDJlEt29mVkZV4U9plFVYOs33ALDQ3/i+ijX787yM25MnTs8dSydNnpBALOiFhZNjJ2zPtYLIUAdHRsYtny02PeS1ra2fTvd9uO5NQKNm76v5ixWq2NYyYuRqs1H3oJJ0mSTgEeBzTAi6qqPrDH+UeBnfv3mYEUVVXjd5y7BPjrjnP3qaoafW/BHQ6nhBPA183tXLB2OwBxGpkFYwrJNxsO8qgE4cBZvXo1n376KX7/ri1JbTYb06dPJzs7O9S2aNEiFixYELMfnU4X1sfeHHPMMeTl5ZGRkcE333zD8uXLo8ade+65DBo0KGLM3333HU6nMzTeY445hrFjx0adFaWqKk1NTfj9fpKSktDr9fs8TkHYSQ0GUQMBJL0+5uw799q11N71N7ybN4fazGPHkv7P+8J2qAu0tlI67XcEotRxkvR68t56E8OAAQQam2h87LEeZ07tT0k33IBxQCGSwUj9w//Gtzl2kcy8t98KzRDz19dTdu55BOojt2Y2jx1Lzssvhc12Un0+Wua8gfOdd/BXVaFNScF+1pkkXnYZstkc9XqK242/pgbZakWXkvIL71QQBEEQhCOVovhQFC8ajSXmZzhVVWlq/pq6ug/x+Zowm/LIzJyBzTYsIrajYyMbNt5KV9eWUFt3vakHcDjGh8VuL32C0tLIh4xmc19Gj3orNOtLVVVWr76IVueSqOPLz7+VPt071R1aCSepuzpYCXAiUAUsB2aoqroxRvyNwEhVVS+XJCkBWAGMAVRgJTBaVdXItQU7HG4JJ4D7ttXwVEUDAEMsJuaO6EuzP4BdqyVJH32JgCAcSTweD8XFxXR2dpKYmEhBQUHEjCGPx8OLL75IU1NTxOutVitXX301ALW1tXz11VfUR/mi+XPEx8dz0003hYqjL1u2jE8//TRq7JQpUzjuuOPC2kpKSliwYAGNO77Y6/V6xowZw5QpU9DGWAIkCL+Eqqp4N20i0NSELisbQ36fqHHe7aXU/PnPeNatC7XpsrJIv+8fxI3f9WHFX13N1pNOjrkEz3baaSRceimB5iY6v/sO55tv9e4N7SNdZibmcePQJiXhWrEC96pVMWMzHnwA++9+B4AaCFB5/fURy/oAjMOHkTtrVljSSfF4aHz0MZxz54Zmk5nHjiXlttswDR0S0Qd073boq6xC0uvRZWaI5bqCIAiCIPwiqqrS1r4Kj7sKvSEFR/xRUQuzAzQ1fU1l1Wt0dhaj1dpITf0t2VmXRBRm9/vb2LDxjzQ3fxNqkyQdOTlX0Df/j0iSfEgmnI4G7lFV9eQdx3cAqKoatcS7JEmLgbtVVV0gSdIM4HhVVa/Zce454FtVVd+Mdb3DMeHkV1TOXL2FFe3dS3m0EgR2/GiOd1i5u18GAy1i+r8gdHV18fnnn7NhwwaCwSCSJFFYWMjJJ5+Mw7FrOeq6det47733eu26GRkZJCQkYDKZWLVqFcEYX7y1Wi233nor5h1fTktKSnjzzTeJ9rd28ODBTJ8+PWo/bW1tuFwuHA4HRqPYwVLYf3Ymp3zl5WiTkjCNHh21WHfru+9S97e7IwqBGwYMIPeVWWji4wFQfD62nnACwcbIxDB079qXeNml+KqqcK9cRdeiRb1/U/tAk5yMfepUtCnJ+MorcL7zTszY5FtvJenqqwBQFYXKq66OOm7JaCTvjTkYd5sRqSoKzS+9RMurrxHckSw3FBSQfMvNWCdPjuhDVVVcy5bTtXAhqArm8UcTN+HoHguoC4IgCIIg9KbOzhLa2lYhywYSE49Fr08KnTsUE07nAKeoqnrljuOZwDhVVW+IEpsLLAGyVFUNSpL0J8Coqup9O87fBbhVVX14j9ddDVwNkJOTM7q8vPcKYx0oZS4vxyzbFEo07c6mkZk3uoCCOPHFUxCge7ZTe3s7FosllNzZXSAQ4MUXX6Suri7inCRJTJ8+Hb1eT01NDUVFRTQ3N/fa2BITE0lJSSEuLo6NGzeGakJFc9VVV5GZmRk6rq2tZf78+ez8G6bRaBg2bBgnn3xy1MST0+mkqKgIp9OJ3W5n+PDhYYk3QehN7nXraX39dTzFxcgWC7ZTTiH+7LMilpy5li+n4pprUff4v6/NSCf31VfR71gqq3g8bJl0PEpbW/QLyjKpd9yOZDSidHbR+PhjqJ7erym1N3JcHLZpv0VjtxNoaqZt7tyYsXGTjiPnuedCx3X3/ZPW11+PDJQkMp94HNuJJ4aagu3tVN1wY6gg+k7GYcO6C6gnhBc6Dba10frmW3QsWIDi8WAaMgTHxTMxDR4cc3yB1laCTmd3gXdRx0oQBEEQhJ/ocE84/ZnuZNONO473KeG0u8NxhhPAvAYnV24oi3l+Wko8zw/OO2DjEYTDXVdXF5988gnFxcWhtvj4eE499VQKCwtDbcXFxbz1Vs9LgGRZRlGUXh9j3759Oe6440hNTaW9vZ0XX3wRX5TdvTIzM7nsssvCluAtX76cTz/9NGz2lCRJnHLKKYwbNy6ij87OTtasWUNjYyMmk4lhw4aRkZHR6/ckCNC9DK/1zTdxLV8BOi3W448nfvp0NHZ7WFzLnDnU/+O+qH0kXHIxqXfs2qa44T+P0PzCC1FjZbudvvM/g0CAQFMTVTfehL+qqvdu6CeImzgBbUoqkk7X48wpXXY2fT+fH5q9VHXjjXQs+DJqrHn8eHJfmRU69tc3UD5zJv6KivBAjYaMB+7H/tvfhjV7t26l/sGH6PrhBwBksxn7mWeScustyHGRRU7VQICuH3/EX12DLj3t/9k77/go6vz/P2e2JZtk03tIAknoTQhYUFRUEEQUBEEQiSAWsJ3end+788565+mdnqcHFmxYQBSQYqGjiFTpEHpCek82yW62zszvj90M2exu9L5f736WfT0eeSQzee/MZz7zmc/OvOb1fr2JGDEiaBVAAMlixVVRgSYmGl1yctC4EEIIIYQQQgjhp40fI+H0vVPqBEE4AMxXFGWHd/kXkVIHcOexc6yp9Xeab4dWgOKRg9CJId+HEEL4d2A2m6mtrSUsLIyMjAzVi6kdkiSxcOHCoCqnIUOGcP3112O32ykvL+eDD4KXIQVPWp3b7f632/ldpucTJ05kkNcQuaSkhLfffjto7KxZs+je/bxvz8mTJ1m+fLnf9vPz8xk3bpxfn7S0tLBv3z7Ky8vR6XT07t2b/v37d+k55XK5EAQh5EsVwr+Npo8+on7BQtXgW4w2ETdrFgl33+2TSqY4nVQ8/LAfKSOaTHR7ZSHGoUPVdebly6l69I+BdygIZH3wvkexVFdHzbPP4Th+PHDsfxiahAS0sTEIOj32woDWlipS//w0Yf0HoDFFUf3U01i2bAkYJ+j15G7dgjY+HgBHcTHnpk5Dbmnxiw3PH0rW22/7GKi37d1LxSOP4K6sUtdpU1JI++tfibjIl8yWLFZq//Y3mlevRrHbAY+fVfLv/scntbAj3PX12I8fRzAYMA4ejBAqohBCCCGEEEIIPxn8GAknLR7T8KuACjym4dMVRTnWKa43sA7orngb5jUN3wcM8Ybtx2Ma3hhsfz9VwmnGoSI2N/rfDHbE2csGEKENlV4PIYQfGvX19bz//vtq1bl25ObmcvPNN/tUlVuyZAmnTgWulJWZmcns2bNxOp2YzWYWLVr0b1XO6wqiKBIREYHBYMBisWD3PtwFQrdu3bjxxhuJiIjAZrOxYMGCoCTYuHHjGD58uLpcXFzM0qVL/ZRWqamp3HbbbYR3SsM5c+YM27Zto9SrtMjKymLkyJHk5OQE3F9dXR0nT57E7XaTkZFBjx49/AivEH55UNxu7CdPgiRh6NkTMYh3maIotO3dS+u69chWK2H9+hF9wwQ/5ZSiKNQ89TRNS5b4bkCjIfXJJ4m5aZK6yvLVV5TddXfQtiX/8VEiL78CqdlM05KlNHflDyeKHp+r/1IV4GAIHzaMyEsuRowy0fzpp9gPHgwam/b834m+7jrAQ04VT7oJxWbzixPCwui+YjkG77WtuFyUzCoIaM4uGI1kf7iUsJ491XWyzUb100/TvHoNeOcjTUICSb96kJibbvLbhrupCfOyj7Du3AmCQMQllxAzZTLaIGnD7oYGLF9/jeJwEj54MGG9egaMCyGEEEIIIYQQ/vf40RFOAIIgjANeBDTAW4qi/FkQhCeBbxVFWeONeRxP+tz/dPrsbOD33sU/K4oS/LU+P13C6dmiKv5REryqllaADwfmcGlc1H+xVSGE8MuB2+2msLCQsrIyNBoNvXv3Jisry6+aVFtbG0uWLKG8U7pOcnIyt956K1FR56/RTZs2sX379oD7i4yMZPTo0dedCrYAACAASURBVNTX11NdXR2UxPq/QhCEgKblHdsxbdo0jEYjWq2WBQsW4HAE9skZOHAgkyadf1A/fPgwK1euDBg7efJk+vc/X7FLkiQ+/fRTDhw44BOXnJzM9OnTie5EGICnr0+cOIHNZiMpKYmcnJyg5FRDQwMHDx6kpaWFmJgYBg8eHPKzCgHbwYM0r1mDu74BfXY2MZNvQp+Z6ROjKAq1zz5H4zvv+H3edP31pD37V1Vp5W5o4OzoMWp1us5IeuQR4mbeitTURPPnn1P7zF+DN04Q0Hfvjmy3ITWZA5I8/w2IRiOGvDzEaBPOkhJcJaVBY40jLiH+9tkIWi1t3+6l/l8LgsZGXXMNGS+/BHj6uPzue7B89VXA2I5VAwHsp05RevtspE7KU01CAlmL31FJr/Zt1/3znzS++RZKB4I/YsQI0v7+Nz+Cyt3YSMOiN2j59FOklhYMeXnEzphB9I03BKweKJnNtHzxBa7qGnTpaZjGjkUTFboXCyGEEEII4ZeJHyXh9N/ET5VwqrA7GbH7OHa563MyMy2eP+akYQopnUII4f8bZFnm7NmznDlzBkVR6NGjB3l5eWg0vtelJEmsWrWKIx3KzgNER0czY8YMkpKS1HULFy6ktrY26D5NJhPh4eE4HA6am5u7JJH+UxAEgauvvpqoqCi0Wi2rVq0K6DkFYDQaeeihh9QUu67It+TkZO666y4fMmnXrl1s2rTJR5kVFxfH1KlTSe7kEfPNN9+wceNGv7aOGzeOYcOGBdxnW1sbZrMZo9FIjLfCWjDIsozFYkGn0/kpvEL4eUBRFNp27qTpo49xlZejTUoiZtJEIkeN8qsQZ929h/J770VubfVZH3PzzaQ8/pgar7jdFF03HmeQQiYxUyaT+tRTgCc17fTIkX5G6ypEkfi77gK3C1dNLS1r1vwfj/i/g/DBg9HEx4MsYdn6ZdA4bUoKPT5dq/pJFV1/Pc4zZwPGGnr3pvsnK1VyqH7RIuqefyHw/ocOJev999RYd10d56bPwFVW5hcbO306KX/yTcNsXr2aqsceV9MFwWMin/rXZ3wM38FTkbB51WqaPvwQZ3ExmthYoidMIK5gFprISL/9yU4nrRs2Yj9yBDHCSNTo0YT17h2siwCQWlsRRDGg71YIIYQQQggh/DcQIpx+ooQTwKaGFu48eo62TubEWWF6SuznH+pS9Dqe7ZVBpEbktbI69rW0Ea4RGJcQw93dEkkLC/khhBDCjwmVlZUcP34cl8tFeno6ffr08fM6OnjwIKtWrQr4eZ1Ox/z581Vi5Msvv+TLL78Mur/s7GySkpKwWq0UFxd3WSnvP4m4uDiioqLQ6XQUFRV1abw+efJk+vXrhyAIHD16lOVBKoFFRkZy7733qlX7Tp8+3aWv1uzZs8nsoGhpa2tj3bp1HD16VG1PVlYWY8eOJSUlxeezsiyzc+dOdu/eTYvX/6ZHjx5cddVVPtUF29Hc3MzevXspLi5GFEVyc3PJz88noouHQ4vFgiRJREVFhVILf0KQWlpoXr0Gx6lTiKYoTOPGBawO5ygqpmzuXFwVFT7rI0aMIOPll3wq/DW89Ta1zz0XcH8J8+4h8f771eXSO+/Euu3rwI0TRTLffgtBr0dubaXy0T8idUFmi1FRIAgeAu3/932hICAYDD4ETyBEXX01uvQ00Gppev8DlCCqTIDkP/2JiGH5COHh1L74Iq2ffhY0NmvJEoxDLgCgbf9+Sm6dCYHmLa2W7iuWE+YtQKEoClV//CPNy/3TLQ29e5P13rs+qij7yVOU3X037qoqn9joiRNJfepJP4P2lnXrqX/9NRyFHq+x8KFDSZh3D5EjRvjtrz0V0bJ1K4rLRfjQocTdOgN9VlbAY1YUBcfp08gWC/ru3YOmLHaMddfVoe/WzU8tGEIIIYQQws8fIcLpJ0w4ATQ43XxU3chJq51YnYZJybEMiDLyaa2Z350up87ZtRlxok7LqiG55BgD+1+EEEIIP04oisLmzZv9VEAGg4EpU6aQm5urrmtra2PRokU0NTX5bScmJoa5c+eqJEdhYSEfdVEpKyUlhd69e2O1WikpKelSZfWfhkajITIyEqvV2qXxem5uLunp6ciyzJEjR/y8tzqiV69eTJs2DUEQcLlcvPnmm1RXV/vFGQwG7rjjDhITE9V1a9euZd++fX6xWq2WWbNm0a1bN3VdeXk57733nl86YmRkJAUFBSQkJPisP3PmDFu2bKGyshLwnLcRI0aQn58fMK2nvr6ewsJC7HY7KSkp9OnTB10Ho+eOMJvNHDlyhNbWVuLi4hgwYECXpJfT6aS1tRWj0RhScP0HIDsctK5bR9v+AwgGPVGjrsJ44XC/86woCuYPP6T+lVdxe69DTXw88XPmEHd7gU+8s7SUc9NnINXX++0v+Y+PEjdjhrrc9NFHVP/psYBtE/R6enz+GfqMDBRJouqxx2kOQvYChA8ZgunaMSguN5Zt22jbvTv4gWu1iAZD0PTDHyM08fGE9eqJEG7EfuwY7gBzRTuMl1xC4r3zESMisBcep6pDRcXOiL9jDkm//jUAst3O2WvHBt12wvz5JN53vpBz45Il1Dz5lH+gKJL+j39gGjNaXeUsL6dk5m1+RJYQFkbGgn/5EVTWHTuoeeYZHKfPeOJ0OkwTrif5d79HE+k7Z9iOHqP6scewHztvwRpxycWkPPkU+gx/Al62WmlZtw5nSSnahARM48ai7TQPtqPdm81x/DhiZCSRo0Z1SXyBpx8RRcTvYTov2+0gSSFlWAghhBDCD4AQ4fQTJ5y6QpPLzRNnKvmwOqhnOgCXx0axbHBgw94QQgjhx436+nqOHDlCW1sbiYmJDBgwICAJ0NLSwoYNGygsLESWZURRpE+fPowePdrHD0mWZd577z2Ki4v9thEWFsacOXNUkqWpqYl//vOfQduWmJjINddcg91up6qqip07d3Z5LOHh4bjd7h/MOP1/C51OR0xMDIIgdEmo9ezZk7FjxyIIAnV1dV0qp7p168acOXMAT+rkSy+9RHNzc8DYtLQ05s6dqxIGJ06cYNmyZQHTIi+77DKuuuoqdVlRFNavX8+uXbt84qKiorjllltIS0vzWb9jxw42btzos22tVsvEiRPp10mBY7PZ2LhxI4cPH8btdiMIAr169WL06NHExcX5ta2kpIQ9e/ZQVVWFwWCgX79+5Ofnq2qzjpAkiVOnTlFaWoooivTs2ZPMzMyAZFoIvlBcLhxnzqDIMmF5eUGruLlqaml8+21aN2xAttkIGziA+IICIi6+2Hd7ikLNM8/Q9O57PuuFsDDSX3ieqFGj1HXO0lKKJ04KSBIJRqPHNNxbAdNVU8PZa8cG9Z5KeeIJYqfejOx00vLpZ1T9/vcB4wDQ6Yi64gpkhx1XRSXOs4HT6X6y0GgwDh2KEGZAamj0IW06QwgPJ/Xpp9GYolBkmfL7H4AgCi5tYiK5WzarVQZLCm6nrdNcoTYhJobcrVsQvd8nbXv3UlJwO0iSX6wxP5/Mxe8geNPEHcXFnJtyM7LF4t+GtFR6rFyJpkNqsuXr7VQ8/LBPZURBpyPpd/9D3PTpPp93lpVRfv8DPpUiBb2ehHvvJX7uHX5zRsv6DTQsWoT96FGPmfyIESTMm6cq0zrCumcP9QtfUfskrF8/4u+804ek6wjZ6cT27bfIbW0YevcJSKS1Q1EUnGfPevzhsjLRpaYGjQWQ29pw19ejiYsLmGIZQgghhPBTQYhw+hkTTu349YlS3q/qmnTad3Ff0kOpdSGE8LOHzWbDYrEQGRkZVJ3idDrZsmULBw4cUBU4ubm5jB492sdHCmDr1q18FcDYV6fTUVBQoKaSKYrCa6+9FlAtBL4ki8vl4qWXXqK1k+dNR+Tm5iJJEq2trdQHUG38GKHX69FqtSiKgu07DJ/z8/NJSEjAYDCwcePGoGmOgiDw4IMPqqThrl27WLduXcBYo9HI/fffrxI+p06dYknnqmxeiKLIXXfdpfpfuVwu3nrrLao6KSHAo8qaO3euD3G5Z88ePv/8c7/YxMRECgoKfBRUZrOZDz74gLq6Op/Y3NxcpkyZgsFg8Fnf1tbGvn37OH36NLIsk52dzbBhwwIaySuKQlFREYWFhTidTlJTUxk8eDDGDqlp7ZAkicOHD3P48GEsFgvx8fHk5+f7qAU7w2az0draSkRERJeqMIDq6mpqamoIDw+ne/fuQRVn/0koikJdXR0ul0sdX8FgP3GC5rVrkZrMGHJ6EH3jjWjj4/3ibAcPUvHIIz7m4brMTNL++gzGIUN8Yi3ffEPFAw/6ERFxBQUkPfJblSxQJIniiZNwBCmOkHD/fSTOmwd4FGFnrrgSKYCCE0CMj6P70qUgy0gWK6Vz5iAHIXsBom+ahDY2FqnNhvnjj6ELAlzQ60FRfMzHf+wQjEYPiSQIARVvHRFx2aWEDxiIGBFB07JluEqDG8Qn/vY3RAwZgqIo1L/2GtYvAxu+AyQ+9BAJd84FPKRl0fUTgqY5dlv0OpGXXQZ4znXR9ROCtiP16aeImTxZXW589z1q/vIX/0Ctlm6vvkrkpecVXK1bt1J+730BCbXkRx8l7tYZPuvMq1ZR+7e/nzeqFwSirr6K1Kee8iHTAGyHD1P9xJPniUNBIPKKK0h5/HF0yb7fqe6mJmqff56WtZ96+kSrxTR6NEm//Q26TmncAPbjx2lashT7qZNoTNGYxo0j+rpxAYlnRZax7tyJ7cBBBL2eqFFXYuhiflMUBamxEWQZTULCD/oCQPFeN4JOF3qxEEIIP3OECKdfAOH02OkKXiuv6zLmzf5ZXJfoK0c+ZrGxrbEVAbg8Loo+kaHUiRBC+CXB5XLR0tJCeHh4wAd08Nw0Hj58mB07dlBTU4MoivTq1YvLL7/cz+Oovr6ed999V/U3akd0dDSzZs3yUcl05VHVt29fbr75ZnX57bffpiSI0TLAyJEj6d27N6Iosnv3br/Kdx2RkpKCXq+nqampS8Lrx4Tw8HBMJhN6vZ6qqqou0wt79OihqpyOHDkSVGUFHq+q/Px8dDodxcXF7O4iHSo/P5/x48cD0NjYyMsvvxzUqH7w4MHceOONgGf8vP766wGJLIBBgwYxceJEdbmhoYHFixf7jSGDwcCMGTN8/Lfcbjcff/wxJ0+e9Iu95ZZbyM7O9oldunQpZwMoZS699FKuvvpqn3UWi4X169dz7NgxZFlW1V5jxozxq3bY3NzMypUrfcao0WhkzJgxDBo0yG9/jY2N7Nmzh7KyMrRaLb169WLIkCEBlWEAra2tnDt3DkVRyMrKCki8AZw8eZKNGzeqBK1er2fo0KFcddVVfh5xiqJQUlLC4cOHsdlsJCQkMGTIkICVHCVJYv/+/ezbvp0WqxWTMYIhl45g6NChfoURANzNzRR+9BG1FRWEG40MvP56ovr08YtzVVVRPv9ebIWF2MPC0EgSereb2OnTSf7971Q1DXhULGUPPURVSjLVyZ55J6W6mtSaGjJf+idRHVSAje++S81fngnYRxGXXEy3N99UH4Brn3+ehkVvBIxFEOjx6VoMOTkobjfVTz2NedmywLFAxGWXYRw6FNlqpXXTJpwBVKQdty0YDB7C4Wd4740goEmIR2OKRm5pwV0X/B5Vl5lJ7NSpIAjYjh2j9bPgnlqaxETS//43xPBwZIeT0tmzgxKG2m7dyFm7BkEUUSSZM+PGIlVVIwMtJhOKIGBqbUUjywhhYeR99SUa77XVsm49FQ8+iC0sjIr0dNw6LbGNTSTV1mIcNIisJR+cV3udPk3x1GkBDf712dlkL1+upiPKVivnbpkekGjVpqXS/aOPfNIMzcuXU/Wnx/x8w4z5+XR7/TUfzzdXdTXl8+ZjLyz0iY2+4QZSn35KVb21o2XjRuoXLMRx4oSnrbk5JNx1N9HXj/drm7uujoZ33qF14yYUm42wAQOImzWLiAuH+8W6amupX7CQlrVrkdva0KWnEzt9OnGzbvPzIlMUBev27ZiXr8BVWYkuJYWYyTcRMXJkQJLKXV9P86pVOE6fQRMTg2n8eMIH9PeLa4f95ElsBw8hhocRcdll3+lH5q6rQxCE70W+SWYzstOJNiHBr5BEZzjLy5HMzeizMkMVLUP4WSJEOP0CCKc3yut49HTFd8aNjI3k1rQELouJ4IETZWxo8L2hH5cQzct9M4kIcAMZQgghhOByuRBFMeBDZjvsdjsHDx6kqKgIgJycHAYNGhTwYfrAgQNs2bJFJX60Wi1Dhgzhmmuu8VGHFBcX8+677wYkOBITE7nzzjvVeKvVyqJFiwL6OCUkJHDHHXeobVm/fn2XaYBGo5H+/ft7UpBqaijt4u2/IAhkZGSoFey6Inp+qhAEQT33XZFegiCQk5ODTqfDbrcHTN/sGDtt2jTi4+MJDw9n6dKllJeXB4w1mUw88MADahs2btzIN998EzA2LCyMBx54QFX5bd++nU2bNgVtR0FBgUpQ2e123njjjYDKuqioKO68806ivA8NLpeLV199lYZ2FUQn3HLLLfTyGkkDnD17lqVLl/r1X2xsLAUFBT5kkiRJrF+/nm+//VY1tBcEgcGDB3Pdddf5kEgnT55k6dKlAdvQt29fpkyZoj5AybLMmjVrOHjwoE+cKIrceOONDBw40KcNH330kR+pB5CXl8e0adN85oO6ujo+/vhjn1RVg8HAtddeywUX+KY4KYrC7t272fHlV7TYPYrArNRUrho71odYBA/p9t6iRdR2IiKTo6OZOXcukR1SkhRFofq119m3bh0VSYlIGg3xDQ0MSksj76mnfB74ZJuNkrlzKaqopDQrE6dOT3RLMzlnz5Lz4K98VC+uykqKb5qM2e2iOLs7NmM4EVYr2cXniImMpPvKFapCrG3/AYpnzOBsTg5nc3NojYpC73SSfe4cvY+fIOOeu0m45x6PV9GePZTOKqAyJYVz3bNpjYrC4HCQWlVNVkkJkQkJJMy7B8Vmx3HyBOaPl9MWFkZzTAwunQ5RljC22YhqbUXndhPWry9otMitrTiLi1EAWRSRRRFRlhFlmZDexAOXVosiCOhcLgw9emDIzUU0mWjdtIlKg56K9AwcYQbc3jijxUr/Y8fodvddGAcPBkGk/pXzKXpt4WE4dXrCbTYMXiIs7o47iLlhAmg0mFetovH1RQAogCSKHsLL256O3l7O0lLOjh0HkkRrVCTm6Bj0LieJtXWIikLc7Nkk//Y3nm3JMsWTJ6sm8p0RV1BA8v88oi6bP1kV1GMs+Q9/IG7mreqys7yckhm3YrZaKE/PQNJqiGto9BC+jz9G7LRp5/uztpaSabfgqqz0HJ9Gg0aSEICoMWNI/8cL56t2Kgo1Tz1F05KltERF0RZhxGhtw9TaSsyUKaQ8+YQP6dO6ZSsVDz2EbLerY1kAYqZNJeVPf/IhfdyNjZT+5jccr6yiMT4OjSSRUV1D74kTSbzvXv+0zM8/p27hQrUSpiEvj4T58zFdO8avf6q2bWPbyk84p9chiyIJVisXDhjIgADpnraDB6l+5hksR4/h1moxALETJpD0yCN+nmgAksVC6/oNuGqq0aenE3XNNT6kYmfYT57CUXQWbVw8xvyhPiR9R8gOBy3r1tF65Chao5GYMaMDFrVQ++PgIcp2fIMgasgedSXGnj2DxoJHtac4XWiTEr+TqJNtNuS2NjSxsd9J1P2n4HK5MJvN6PX6oC9w/tOQJImKigqcTifJycnqPcVPFSHC6RdAODW63AzZcQy7/P3On14QcAY515OSY1nYN3DlkhBCCCGEHxqSJFFZWYnb7SY5OTmo0urEiRN8/vnnPsqX3NxcbrjhBr8v6paWFtavX8/x48eRZRmNRkP//v0ZPXq0T1pUU1MTCxYsCEqe3HTTTQwYMADwkBD//Oc/g6bKXXTRRVx77bWAJ2XxhRdewB6kspbJZGLy5Mk4nU7MZjOffvppkN45H68oCg6HA6fT2WXszx2pqamYTCYANeUuGJKSkoiOjkaWZUpKSrokyeLj4xkwYAAGg4GysjIKO6kEOmLAgAEMHz4cWZY5efIkO3bs6LINt912GwaDAVmWefHFF4OOoby8PGZ0MPf+4osvgqrOOqvIFi5c6Jey2BF33HEHGRkZQNdpmaIoMm/ePNXUfv/+/axZsybodsePH09+vuce0263s2DBgqDKwenTp9Ozw4PLunXr/LzIwFMsYObMmT4KtXfeeYdz584F3G5OTg4zZ85Ul61WK4sXL/bzZ9PpdEyfPp3uXs8p8Mw/Hy9bxolOihNREJg4aZJ6/bfjq9Wr2bp/P3R4qBIUhdGXXMLFY84/nCqKwpJnnuF0gOs12unkzt/+lghvWpaiKHz2yCN8G2DuM9jtzLjySjK921Zkma2zZrGtRw+fNgCIksToujouevVVdbunJ09ma1w8VekdvN0UhdjGRi7du5e+ixaBICK1tnDy93/gSEYGGllG53KhkSQQQEHA1NJM3+49MA4eBIKAedVq2srLqUpJwRwbAwiE2e3ENTYQ19iEPjkZTWwscnMzrpqagGlsIQSGGBGhqt/cbW20GY04DAYkjQZZFAEFU0sLRrcb07XXIhg8PmCWrVtxabU0xsVhjYhAEQSiWlqIa2xEq9GQ9pc/o4mJBY1Ixf0P4GxroyotlZrkZA9xUl9PZkkpWr2evG1fofXOs6V338O2lhbO9MzzaafJ3MxlO75h8Nq16Lyp2VV/eoyqNWs42r8/pVmZuHU6jFYruafP0OvkSTIXLiDqyisBj2rx+B8fZe+wYdR3KM4RX1/PsD176fPYn4i+7jrAQ/aeHncdp7IyOZ2XhzUyEq3LRWZpKf2OHqP7ww+r5LAiyxwsuJ0NKcnYOl1TaRUV3HDxxSTPnq2ua1q6lOonnvR81ruu/cpKeepJYqdMUWNLN2zg/a1bcQZIVb5Cr+eKDr50tmPHOHrnXRzu1ZPyjAwUUSTMZiP3zBmGhIXTffE7Poov8+efs+u11yhKS6PNaCTCaiW3sooLf/UgJm+ftcNeUsKWZ5/luCGMVpOHoO5RX8+o6dNJ7ODBB2A7cYKtTzzJ8dQUWk0mBFkmraKSixMT6ff0Uz5tcDc38/ljj3M0PEw9xjCbjaEIjHricTSdXhxad+zg+GuvUWy1IosiKQj0nzSJuKk3+xFPTUePsuXddzmrKLh0OmKtbQzt0YPh9873I8raGhrY9PrrHG9pwabXE2W3MzAllSvvvgttJ6sIRVFo3raNw6tW02RpJcJgYOCoUSROmOCnqHO73Wz69FP2HzqkPgenmkyMvekmMgNU7WyprWPnR8soq65GFAR65uQwbMoUDEHS6x1FRVi//hrFLWEcPozwTt8f7Th69CgbNmxQ72cFQaB///5cd911AV/ONjQ0cHDXLixNTSRkZDB42LCAKf6yLHPo0CG+3b0bs9lMVFQUF+TnM3ToUD+VM3heEO3Zs4fa6mrCjEYGDBhAnz59Ar5UdrlcHDt2jNLSUlWZ3aNHD/U8hwinXwDhBLCqpon5hSV0/kpPN+iYk5HAp3XN7G/57lLoArD7oj5khp+fTCVFYWN9C1/UN+OQZYaaIrg5JZZonf/gDSGEEEL4T0GWZUpLS7Hb7SQmJhIfwG+mI+x2O1arlcjIyKBeNqdPn2b58uV+1eSuuOIKrrjiCp91JSUlLF261I9Iys3NZerUqT6qrGPHjrF8+XI/VZZWq2XGjBk+D73Lly/n6NGjAdsXFxfH/Pnz1RuAN998k7KysqDHPHbsWLKyshAEgW3btnGsCzPiwYMH06NHD1wuFwcOHAiqLAKPYig9PR1JkmhoaPjJpCP+mCAIQtA0xHYMGDCAyMhIVf3TVXxOTg4ul4vW1taAFSo77zssLAy9Xo/FYkHqggAwmUzExsYiSRI1NTVdmvwbDAZyc3PR6/U0NDR0qQJMSEhQlVmtra1dVsuMi4tj7NixuFwuamtr+fLLL7s8vokTJ5Keno7RaGTt2rUcPx5Y6REeHs6vfvUr9F7/m6+++oqtW7cGjBVFkfvuu09NMzx9+nSXhQNmz56tKrO6UpwBXHzxxYzxkkiNjY289NJLQWM7EpFOp5Pn//53HEGI51iTifsefBDRqxxY+c5iDp8r9qTtdXoATAgLY95vf6vGvvXsc5TaAtwnej87c8oUcryqiLpt23hnzRqsAcyu4xsbmXP//Ri93kHVb77FsoMHsEZGonc60blcCF51isFu53JBJGPuHSArtOzZzYa939JmNCIonhhRktFKbjSSRG5dPT1vugnF4cBx8iQnjh1T0zE1koTQ4XqJam0lbcAARGM4bouVs+fO4dZokDQa3DodCqD1fk6UJbSKgqIoqhpMEjXq34KiYLS1Ed5mw9jWht7p/PmrxLRaBI0Gt8uFW6tFFkUUQUARBARFQVAURFkmTBQ9flKShOQlHtrPg9ChPwH0oogmLg5Bq8XV0ECzTouk0aox7dtHUUh2OonKH4oganCcPUNFcwuWyEhPjCggCyKKKCBKMt0bG0mafguiToe9tIxdJ05gDwvzxKnnU0TWaEitb+DKhx9CNEaguN2U3XUXDWFhnM7LpTYpCUUQSKyto+epUyS4XPRYvRoxPAwFWPzYY9R2GvNqX0gSsydOJDo7G0VROPnoH9kSFYlbp/MwWQK4dDqcej2plVXcPGUK0V4FlfXAAZYuWEC596VAR2SfK2Ha739HmJesd7e08N7//I6SpES/2KjWVgpuvpn4Cy8EPGqij+69lxMdKuiqp9flYlJ8PH0fekhdt+LXv+ZIEAP7i2WZMU8+qS43bdrMig+X+rU5usnMjd2z6f7gg+q6+kOHePuDD7AGINUvcLqY8OenVeKiraGRRc89S1MAD9IMu53bn3wSjXf+VhSFg399lvXmJuwd4jUuFyOsVq587jnV60xRFN5fsICzHZXL3rlNVBRm3XYbWTnnC2ydVrRVEgAAIABJREFU272bJavX4NT7pqKa7A5unz+P2A59KjudnH30UQ4UF1OZmoYiesbQoMgI+v7tb2qqLnhenn744Ye+B+ZtR2a3bhTcfrs6JwNs+uQTth865BOuVRQmTphAv6FDz7dBllm+dCmFp0/79Vt2aiq3zpnjQzodPHiQ1atW0fkOIzstjRm33+5zP1tXV8d7b75JS6d73+y0NKYXFKDX60OE0y+FcAI40trGG+X17G+xEi6KjE2MpiA9gVgvMXTMYuOx0xVsN/tXFemIgrR4fpWdQrJBR6tb4tbDRexu9q1QE6/TsmRQDwZF+U8esqJwoKUNs1uiZ0QY3UJm5SGEEMKPGHa7ncOHD9PQ0IDR+5YnUFU28BhaHzx4kIqKCvR6PX379iUnJ8fnBqEdJSUlfPPNNxQXFyOKIrm5uVx66aWkdqpeZLPZWLJkiR+RZDKZuPXWW32M3EtKSli8eHFAZU9WVhazZs1S22I2m3n99dcDGpJ3Ti+sqanh1VdfDUpwTJ48mf79PV4Z9fX1/Otf/woYBx7SZOLEiciyTENDA6+88krQWL1ez8UXX4zD4aC+vp4zZ84Ejf3fQhTFLpVQIfyykJycrKZRnD17tkvyLTExkdTUVARBoKioqEuiNSIigoSEBNxuN/X19X4kdme0k+But7vLNrTHCoKAJEnfWeUzMTGRyMhIRFEM6FnWEUlJSYiiiMPh+E7S0mQy0aNHDyIjI6moqAicKut9cLrmmmsYNGgQiqKwf/dutm7fHnS7PbtlMuLqq5AkiTOnT7OjqzRnrZZrvKoXc1MTX23bFpBMAw8R0CM3F4fDQXNz8w9KkouyjEmvJ1KjwVFVhSzLaF0udN4fQVYQFRlBVsgQBboNHoygKFR+s4MaUVD/L0qySlgIKEQ5XcRkZuJ2u7FUV2MTzlt8CYDW7ULv9OxD63arSjTxB35+UwC3VovDYMCp1+M06HHoDTgNeiTRkyandbs9RKDb83egdaIsowjnSR9ZFM+TQILou95LZLWnfao/HfpJTQlVFA9xqNXi1mlxaXX+v7VaXDrPb73LSWSrhajWViItFvT/ZhEABbAZw2mOjsYcHUNzdDTN0dG49Dr0TicGhwO904ne4fnb4HSof3f8f3vbZVFE1GoJS09HYzLRVFlJTViYh5Ay6HHpPOl6CqCIIr0QyJt4I2i0FG/dwv5WC4rg+Z8idBhPskyGzUb+xEkoLhd1e/awv6YGUZYQvTGKcJ6sC3M4GHjllShaLW119Rw4exZZFFBEERkvuYji3YfMhAkTCE9LA52ODS+8QHFyMpIoInlTThXvdRhtNjP7wQfRe30+lz3xBMXee432mHZiURFFbhs5kqyrrgJB4PPnnmNvIBWw9zq/NjubiwoKAKjcvJm3tmzxkHoB4m9MTmHwvHsAOHPoEO9/8knQ+SJVELjrsccAcDud/OOxx7AGeVGZ4XRyR4diBYWPP85qux1HJ3WSxu1mdGMTFy5c4G2SwoLnn6c+QHXP9nZ1VAIf/OorVm3dGrDNoixzz913k+j17Dy8dy8rP/vMP9a7fPUll3DpaE81zsbGRl7+5z/Vc9E59uKBAxkzaRLgUQG//OyzmDu/5PDGDuzRg0m33RYinH5JhNP3wfLqRu49HvwNZEfkGg0oCpy1Bb5pSjXo2HVRHwwdHrQ2N7Twh9PlnLN5BqYAjE4w8fde3UjU//cr9oQQQggh/BQgyzKnTp3i5MmTSJJEVlYW/fv3D6jMKioqYt26dWrKkEajYdCgQYwZM8Yvvr6+ni+++EJ98BRFkT59+jB27Fgf3xuAw4cPs3r1ar+H35EjR3LllVf6SOS//vprNm/e7Ne2mJgYZs+eraa9QfDUKYAJEyYwxFvpTJZlFixYENQPSa/X89BDD6HX65FlmcWLF3ep9pozZw4ZGRkIgsDmzZv5+uuvg8aOHz+elJQUHA4H27Zt69KkPjY2lgsuuABRFKmsrOwy/S48PJxevXrhcDioq6v7wSsu6nQ6jEYjLS0tXaqhkpKS0Ov1OJ1Ov1SzzhAEAZ1OhyiKOJ3OEFkXQgg/cmgEAdmr8FJ/ZMWjElMUREUhMT4erSDislpobmlR/y8qCrIgIGm0KoEjeRVNP1cY7HYPAWVp9SGiolpbkUWR5pholVQ6Ty79cC/PBVn2I9d8CDVZ8aSzesm6djKnqx/Ah0BQfuDzJ8iyqlDUOV3oXU50XvJT73IiSjKSRoOk9agIO/64tVqf5fZj8mtzp3Xtv7VuNxq3G62X1NQLIjpZBpsNQZYQJQmN5Ok7t1aDW6vDpdMia7SI0SacoohNlnF52xKIcEJR0AI6RUFwu1G8hGn7j0aSPG3wkqs5ySlER5swSDKn9uzGoTdAB+WdJIo49XpkQaBn//7YoqJoaGujpLoat0aD0oUXapheT5heT5vZ7MlcUklpb9qnt/1ROh3xGRm4JYnqigo1TbBjTDs0QEJyMoIg0Go2Y7Hb/fvBuywqCr379UMURVqamiitCO4TLcoyDz/yCJGRkSHCKUQ4nUetw8WQncdw/0Cn+pm8dGalJyAKAjvNFqYcPBNw230jwvgiv6cPOeWUZZZUNbK0qoEqh4tuYXpmpMUzNSUOTRDTuXqnm1K7g3idlqzw4OWeQwghhBB+zlAUhbq6OhwOB/Hx8UG9r9rR0tKCxWIhOjo6YO5/x7iDBw/S2NhIREQEgwYN8lFYdcTp06fZtWsX1dXVGAwG+vXrx0UXXeS3fVmW2b59Ozt37lT9i2JiYhg1apSPQTXAuXPneP/99/38lgRBYOLEiT7x5eXlvPPOOwG9mTpXv3M4HLzzzjsBq+UNGjSIG2+8USXUKisrWbRoUVACZ8aMGeTlefxMnE4nCxcuDGhSDzBlyhT6edOQrFYrL7zwQlA1S3JyMrfccgtutxuz2cwHH3zQJYk0b9489dx0RajFxMRw7733qnL6VatW+RmGd8TNN99M3759ATh06BCffPJJ0NgJEybQr18/nE4n+/bt6zL1LTY2lvz8fNxuNxUVFZwKUKmrHaIoMmrUKHVcf/bZZ0H7TaPRMGLECJxOJ01NTQENzv9b0Ov1aLVaXC5Xl0okjUZDeno64Lnmgo2fdqSnpyOKIna7vUuvLvCoob6Paqq9HeHh4ej1esxm88+SXAwLC8NgMHxnIYeLLrqIhIQERFFk69atgRVR3rf6SUlJapXX79PPIfz8oPEqzNxabXASI4RfFHReNZsiCLQZjT846fdjQlRzM61epfD00WPoNeKSEOEUIpx88diZCl4rC3zDMi0llj6R4ewwW9jeZMEqfffNh04QSA/T0eSSaHYH/+L9V59MJqd40lRcssJtR4rY2uj/hX59Ygyv9svyIZ0anG4ePV3O2jqzSmgNM0Xw557pDAyQ1ldmd/JuRT2HWtswakTGJ8YwISkGfZCLv9nlZn9LGxpBYGi0sctKfYqiUGJ34pIVssMN6MTQl0wIIYQQwnfB7XZTV1eHRqNRH+wCobq6mu3bt3Pq1ClkWSY7O5sRI0b4+F61o6Kigs2bN6tVESMjIxk+fDiXXnqp3/YdDge7du3i8OHDWCwW4uPjyc/PZ/DgwX6xBw8eZO3atT4Pk4IgcPXVVzNixAif2MbGRj7++GMfMkuv13PNNdcwbNgwn9i9e/fyWYCy73q9noKCAtLSzps7f/755+zZsydgHw0cOJBJXsk7eGTvq1ev5vDhwz5xMTExzJgxg8QOprxms5lFixZhtfqmyoPHj2z69Olqf8iyzMqVKwN6jPXt25fJkyersQ6Hg1deeSUgeSIIAjNnzqRHjx6Ax4D0xRdfDNgGgCFDhjBhwgR1+ZtvvmHjxo0BY6+99louuugidfn111+nsrIyYKwgCDz44IOqEf+KFSu69DmbOnUq2V5Plq+++iqoiTvA1VdfzaWXXgp4xuWiRYu+V6zFYuHFF18MamrfkTyVJImXXnopKHmSlpbG3LlzEQQBt9vNiy++iCVQ+oYX999/v5o+3JWflV6v57777lPbu2LFii7VerGxsWRnZyMIAqWlpV3Gmkwmhg8fjkajoba2lgMHDnQZ2+591dzczIYNG4LG6nQ6HnroIbVS5YoVKzhy5EjA2IyMDObMmaOSzoWFhUE9xrp168btXp8VRVFYtGhR0PEGkJKSQnp6OrIsU1RU1CXxFRERQf/+/dW+6CrFOCwsjFxvymB9ff13pkT+O0hKSiIxMZHw8HCOHjmCPVB6qJd8u+iii4iKisLpdLJr586g/mLgSffMyspCFEWKioq6HBfxcXHk5OaqRR+6Ilr1Wi0jr7gCg8GAzWZjy5YtQVOndIJAXFISjY2N35meCh7yOykpieTkZM4dOEBzkO8tjdvN7LlzUTQarFYrK5ctw9EFgRsdFUV0bCyyLFNdVobbq2BRfci8y+3Km6joaERBwNrUhMvrgSYoilfOcv5YRSA5PR0EAUuTmWZbm0cF1b5jb+oiXuVbXs+e6DQabM3NnCsv9/HfalfUtKuooqKjcSkKdrcbhyx/L0JFoyggSX6pkuePD7QC6A1hCCjY29o8llcKnnQ+QBY1qnpK9qqU/NLBgqA99VTndPopsxDArdGqqiy3Vuu3LHWlivpfQOtyEWGxEmm1EGG1YnA4cGl1uPQ6nDq9z+92zy/5O6rHi6oX3Xk1lijLKHj8zjoq4QIp5lTF2b9xjCazmRZv4YuZF19C7rVjQoRTiHDyhaQoPFdczaKyOtq8k2GERuTOjER+0z0F0Tvgjra2cfW3wd9A/rtI0Gm5Kt5EtzA952x2ltcEf6O3oE8mN3nJKZskM37/KY5Z/Cs+RWpEPhvak14R5/NmNzW0MOdoMY5OVfuGmox8OCiHKO35C9ctK/y5qJK3K+rVKn9RGpF7M5O5PyvJr8rC53Vmnimq4nSb58s3Ua/l7m5J3NMtUe23dljcEu9U1LOm1kyzW6J/VDiz0xMYEetf+lJRFHaYLXxU3USt00VWuIFbU+PoH4BMAw9BtrLWzLk2B4l6LZOSY0kL4pUlKwrfNFk4YrERpRW5NiG6y/TGU1Y7u5staAWBK+KiSDUElxG3uiV2mS1ICgyNNna5XYcss9tsxSJJ9I8M9zGm7wxFUThhtVPndNPdaPhOH7AGp5tKh5NkvY4kQ9epm1ZJoszuJFqr6fLYwDM+yh1O9IJAqkHXZblXRVGod7lRFM+4+K7SsFa3hF1WiNNpvjPWLsm0ShJxOm1Q9V87JEXBKslEaMTvjAXPedELwne2ATz9IQr4jfVAkL3fJ98nVlE8NxXfJ7Y9/vu0N4RfDmw2G06nk6ioqKBE1r+LlpYWDh06RFNTEyaTiYEDBwb19lIUhbKyMmpqaggLC6Nnz55dGtXv2LGDsrIyNBoNvXv35tJLL/UhhcBDLnzxxRfs27fPR+k0aNAgxo8f72Pq2Y7q6moKCwtxuVykp6fTu3fvgNVpmpqa2Lx5M4WFhciyjNFoZOjQoVx++eV+8bIsc/ToUQ4cOIDZbCY6OpoLLriAAQMG+PV1U1MTK1eu9El1jIiIYNy4carSqx3l5eV88MEHflX7srKymD59ul//7du3j23btqkP7DExMVx++eVccMEFfttdvHhxwIfJjkQPeMjCN998MyDx1a9fPyZPnqzONVarlUWLFgUk1Dp7ogFs376dTZs2+cXm5eUxdepUn34+evQoK1eu9FMYJSUlUVBQ4KNgLCsr4/333/fziIqMjGTWrFk+46grhdqwYcO4zuuHBJ7xtnLlSj8CTq/XM23aNJUsBDhy5AgrVqwIuF3wrYr4Xf5wHVWAkiTx+uuvU1NTEzC2o5ccwLJly4IaxHcu/OBwOFi2bJlKTrcjOTmZGTNm+KQBgyfFeMOGDSphJwgCffr04frrr1dJLIDjx4+zbNmygG3QarXMmzdPnTeKiop49913A8YCTJo0SVVxOhwOFixY4FOZtSM6VlG1WCy88MILQRVqKSkp3HnnncB5sjdYtUy9Xs/DDz+sXn/Hjh3j448/Dhibl5fH9OnT1WukK2JYEATuueceVZVZVVXFokWLArZZFEXmzJmjqgDr6upYuHCh/xjykkqjRo1i5MiR6upVy5dzsCNJ7o0TgOlehaqiKBQWFgY9tvbP3H777WR5K5g1VFTw1sKFWDvNvYIsc8OIEQzuUKly27ZtQYkvjSBw3wMPEON9YP9m/Xo27twZ1H9nwtVXM8Q7b5UWFvJWIDLUGzsyN49R3op9bW1t/OOvz+Lq/GLcGzsoI4OJd9yhrn798cfxo069sXkGAzN+9zvAc53+7YknsAf6rvXG3zV3LsmpqYiiyMfPPMOxYJ52isK8mTNJ8hYZ2Pbqq2yprg7aFzfn59N3/Hjcbjcb33mH3V0UO8nT65nhrRzYtHcvr6xYgUuvV9NO27crAINaWxn3l2cAhdbycha+8855Qq0jCadAmNPBiGuvxWa3U1dTy6niItVvq50Ia0+/07lc9ExMJDcri0hZ5sy771GZkODjq9XOBuqdTgakphDeqxcoULhlC80GT1qeJGpQRMHDLXrTL9MFyB42HEEQqPzmG0o0ompQr7ZX9vjDJdsdZAwfDpJE49GjVLic3rTa8x5yipfkC3M4SO7ZC0VRsNfWUuM1C1e815AiwJEBA2iJiSGytZX7HvwVYZndQoRTiHAKjBa3xL5mK4IAQ00RPkQMeG6gr9h7kpPWwKW9AcYnRtPkkii1Oyiz/3smfF0hSa/lhqQYTFoNp60O1tQFJ6duSIzmtf6eN9+NLjfDdhYGVWbNTIvnb73OVxZ45GQZiysD+4X8tnsKD2WnqMurapq4uzCwr8ddGYk8kZeuLje63Ew8cCZg3z2Rm8Zd3c6nqCiKwiOnynk3QDv+mJPG/EzfdJYN9c3cU1jic4wab+zdnWKL2xzcfrSYEx3aoRMEHspO5sGsZJ+H9la3xH3HS1hXf/7mRgPcnpHAE7npPuSFoii8cK6GBaW1KmmpEwSmp8bxZF66T+okwLKqRp46W0m9y/MGVwDGJkTzfO9uqql9O/Y1W/ntqTIfgnFUXBTP9+7mRxDVOFw8erqCz+s8ec4CMCrOxNN56XQ3+j6s2CSZvxRVsqSqUe27i6IjeCw3nQtMvsSerCi8XlbHq2V1VDs943pAZDi/65HKqHjfm1KAL+rMPH+uhqMWzw1cT2MYv8pOZmJyrF/swZY2ni2u4svGVhQgM0zPXd0SmZ2e4EeilNgc/LWoik/rmnEpCvE6LbelxfNgdrJfH7e4JZ4/V82yqkbMbolorYapKXE8nJ3sV1FSUhTeKK/j7Yp6ztmcRGhEJibF8nD35IAk3KqaJhaW1XK41YZeELg2MZqHspPpHeFfRWRHk4WXSmr42uxRL46MjeKBrGQuivGvfHLCauMf52pYX9+MQ1YYYjIyLzOJ6xJj/GKrHS5eLKnhk5ommt0SvSLCmJ2ewMy0eD+iyipJLCyt5cOqRp903TszEgnT+PabpCgsrqhncWUDZ9vsxOu0TE6J497MJL+xCbCm1syisjoOtrYRoRG5LjGa+7OSA6b47jZb+FdpLd+YLYjAqHgT92cmBSSST1rtvFxSw4aGZlyywvDoSOZnJjEyzp+grnW4eLm0htW1ZlrcEv0iw7kjI5Ebk2L8xpBVknitrI6PqhupcbjJDtdza1o8s9IS0Ha6CZUUhfcqG3i3op5im5MkvZabU+K4u1siEVr/N2yf1pp5o7zOQ2ZrNExIimF+ZhLJAUjfXWYLC0tr2eUls6+Jj+bezCTyIvxLAJ+02vlXaQ2bG1pwKwqXxEQyPzOZYdH+qYA1DhcLS2tZW2fGIkkMjDQyt1siYxKi/WJb3RKvltWyvLqJOpebnHADt6XHMyPVfwy5ZYV3K+t5v7KBUruTFIOOqSlx3JGRSHinMaQoCqtrzbxVUc9xiw2TVsONybHM65ZEvN5/DO1osvBKWS3fNlvRiwJjEqKZn5kUcAwdt9h45UQxZ4qKAZms7Gzu7pMTsGBHpd3JwrJaPqtrxibJXGAycle3RK6I85+zWtwSr5XV8kllHa12O5nRJgoykpicHOs3hpyyzOKKBpZUNVDhcJJm0HNLahwF6Ql+85CiKHxSa+bDY6cwN9SjM4RxWZ9e3JOdErCy7ebKOlbs3I29phq0WlJz87jroqFkGv3HxdHWNhaUVLO/ogYBGJqRyvysZPpG+s9DB0vLWLFpC5SVoFFk7LEJDLv4Ym4aPtQv9lxdHe99vh7XuSJERcYZbiR70AXMuWaUX3noerOZN9Z8RlvxGY/3jSgSlduTO68fT3SU7xynKAqLDxXy7bd7EVqakQ1hZPbpxz2XXogpwMuZtaeK2LRzF0p9HbJWS3xOHndcPoJuUf7jfmdFDcu/3o6zsgIEAWO3LGZcfikDE/0J0S379vPl1q2IFs+cLOn19B6Szy2jr/YjDOsdLl7Zf4SzhYXgcmKIT+C6C4cxIdO30IGiKHy+fj17O3uzCQJjx47lwuHDfVYfPnyYT1avRumUfnbJyJGM7lTGvbW1lTc++pjmsg5+o3oDV159FZd32q7T6WTxyk8oP3GCdg2HImoYOPxCJgY4vtNWG6/vO0J50VmQZeIzunH78AsYGhN4nl1YUs3O02dwO5ykpKRS0DOLawLMLV9u386Xmzefd/gGRL2eaVOm0NObftuOrV9+xVdf+ivJhuQP4/rrxvlcf/X19byxZCn2xg73hxoNI0ddxagRl/h8fteuXaxbt85vu1qdjoJZs1QCEDznY+XKlZ7+8q5r3+t1113np8z8at9+tmzejNDmIWZlUSS9Tz9uv+F6teIjeIiIDz/+mNMnTvi1Y/z48eTn+z6PHiksZOUnq1Bc51VRgk7HjTfcwKAOxCJ4COe1n37q08cAPfJ6MmPaVJ9rVZZl3t+wiTPf7kV0e+7hlOhYxl07hgv79PaJW7hwYVClVXJyMnfffbfPOTlV38iqJUtxVnuoGTE8gqHXT2DMYN/2SpLEh8tXcPq4r8efIIpMvukmHwLe7Xbz3qJFlAQgWntmZjKtoMBnLH+1ciVbO6lZAbJNJmY+8IBPX5z2VkXrnH+SEh3N7fPm+RD7luZm3vrHizR2ql2WqtMx6+GHfQj1fTt2sDaIwnBQTg4TZ85Ul20tLSx69lkaA7wkuSovj8u8VTgBJJeLDx5/nKIAsUMNBq73kl4AbVYrC559FqsonieovL81sszc2bNJyc5W4w88+ihfSBLOTi8zssvKuPnXv8HYYWxsfv4Fvm71J3tFSWLiwIEMmDIF8MyFL//5zzQGUajqFYVf/+EP6nVS98orbNm6leN9+vgQaqbmZi4/VsgFqz5B9L5gKFmxgg++/dbT3k4EXGyTmTn3zifSO7/Yjh3jo+eeoziAEjytooKpt84k+nIPKesoK2fZH/5AUQ//2DCbjSnJyeR4qxdKFivL5s/nVPfz/dixr69qaOCyf/0rZBoeIpz+b/i6sZXph4twBRgb87ol8adcj/xfURSu2HOCk23Bq7JowG/C+6GgEwSMGo+8uaWLNEANcGe3RGK0WuyKzIvnavxKQrYjTBR4f2APIjUaUBRuO1JMrSvwhAKwY3hvuhs9lWR+faKM96sCE1kCsP3C3uR4b6iXVTXywIngRu6rL8jlQu/D+pk2O6P2nFSN4Trj3QHdGe29GbJJMpfvOUGpPbC8+W+9MpiZlqAuTz90li0BUhwB7s1M4tGc86keL5yr5rni6oCxU1JieblPlrq8ptbMncfOBYwdajKyZkieSmYdt9gYt+8UNtn/+LqH69mY34tI74Nvi1vi2m9PURTA1D5Jr2V9fk+VPJEVhVsOFfFVk//xhYsCq4fk+aRmPna6gtfK/eXbIvD2gO4+D7MfVTdyfxAj/j/npTMn4/zb5m+brdx08Iyf+g5gbkYCT+WdvyEsszu5bt8pap3+Y25UXBTvDeyh9ptVkph44AyHW/3fWPaLDGP1BXlqvymKwrzCEj75f+2deZhdRZn/P3W33vctnc7S2TcSIIR9U9xxARFURBSB0XFGccaFGXVGcV9HHZ1RRnZcQEFkEZFFRUAggSQEspB966ST9L5v997z++Oce/vce6rq3jg9cvPz/TxPP+nb/U11nXpPvW/Ve+pUHQkmcacXRXnwpAUZSafv7jnENzS2LguHuOfE+RkT3weO9PKhTXvI7oFh4MbjWnmTL5G0vn+Yi1/YoU0Of27edP7BlzxtHxvnzWu3c3AsmNS+rLmWby+aOXmkbiLJJS/sYG1/8GS2s2vK+dmKuelXax3H4R+37OOew8FXERaUFnHfygXU+ibJpvu+Nhrm/pULmO+bJN93pIcPb9obaIuikOJnK+Zylm+l47r+IS55Yae2Lb6zaCbvmV6X/nxgdJy3rdvOAU1b/OOsRv7d10+HEgkuXr+T9QPBtji/voobjmtN30NJx+HvN+/lfs19saKihHtOmJ++hwC+ubud7+wJDo6nF0W5b+WCjFWJvzrUzUe37Au0RVk4xF3Hz2OlL5G0pneQd2/YlU5kpwgDP1w2mwsaJ5O4+0bGuGDdDtrHg23xmbnNXDO7Kf15IO72kVRS2M/FTTX8YMms9D2UcByu3riHhzqDr76cWlXGncfPy0g6fWnnQf57X3Aj7tnFMR5YuSBj1eUd7V18/OX9gbhTFQnz6xPnZyRP/twzwGUv7kqvvk0RVYqbj2vNmPjuGh7jgnXb6dDEqC8vaOFqnx/qnYhz4fodGQ8iUrx/eh1fXzgj3RYTSYf3v7RLGxvOrangJyvmZLyq/m/b27ixLTh50/WnWw908q/bgk+n66IR7ls5P6M//bGrn/e/tDsQ+2LK7U9n+xKzW4dGuXDddnriifTT69RT528vmsl7ff2pczzOheu3s2N4jFAySSSZYDzsvj7x4ZkNfH7+5IOksWSSyzbs4qneQaLxCUomxhmOFREPR3hDfSUMhMpPAAAgAElEQVQ3LZuTTuI6jsMnt+7nZ+3dges7rryEe06cT6WvP12/7wjX7Qy+ljUtFuG+lQsykpEPd/Zx1cbdgf0yS0Mh7jx+Lqf4kvsvDQzzjvU76I8nqBkeIJRM0ltWgRMK833fNgfgJm/fum67dszw8dYmrp0zmXQaTiR51ws72dPWxqLD+ygdH6W/uIwtzbN5VetMfrR0djqJm3QcPrplHw/ta2fh4X1UjQwxEitia9NM5k1r4q4T5mVsY5CKOTVD/dQP9jEejtBW00hzWTH3n7ggYzX3vYd7+IfNeykdHWZaXxfJUIgD1Q3Eiou564T5nOB7mLS2b4h3bgj62YiC/1nWmvGgY//oOBes08ecz85t5qM+3zIYT/COF3awo7Ob+UfaKBkfo6+0nB0NLVw0o4nvLZ6Z4Vs+uGkPa3btYXH7XipHhxiKlbB12ixmz57NncfPy3go8rVd7fznnkNM7+2kbqiP8UiU3XXNzKis4L6V8zNWlf+ivZvvPrWa4/dvp6m/m2QoxJ66ZrbPXcKtZ5/EEp9vebpnkE/98c+s2P0yTQNu/Ossr2LD7EV8/ryzeK3vwdru4THetm47nePjNPb3EknG6SqvYjRapPUtb1+3naH9e1lweD8lE+P0lFWwubmVCxbO5WsLWjJ8yxUv7eaJI13M7TxI+egIg0Ul7GqYzpkNtdye5Vv+fXsbd2/dxdL2PdQMDTAajbGjcQaxma3ce1Kmb7ntQCf/sq3NPU1wdIh4KMxgUQl1sWjAt/xm206evOuXlExkjiWHo0Wc965LeeP81vTPtg6N8vb12+meCM5m/mPRTC7L9i3rtjFy8ADzj+ynKD5Bd2klLzfP5vIFswO+5b3rt9O9ZRMr9m+ndHyMoaISNsxawOxlx3Hj8nkZvuVTW9t49OVtnL1lPVWjQ4yFozw3bynRBYu5Z+WCgG/51uadLN+/gxndRxiLRNk0Yy4j02dx76pFQd/y0i5a2/ezrG0nSUK80LqQrsbp3HnCvAzfsnFgmE89+Bgnbt1AzHHbIwlsmj6XS99yPhf75heHxya4cPUmlqx5kjmd7ShgNBzhiSUrecuZpwZ8y6XrtlHz+KMsbt9LSEEcxbrWRdSd99qAb/mnZ1+k9OH7KfbNMBNJ2HnqudzwxnMyfcuug+y66WZO276RZCxCOB6nq7iCR9/yDq5/y3mZvuVQN4/ecCNzug8xWFWJSiYp6x/gxbnL+PgV78nwLY9t2caTd/48+Eaa4zDv9W/i8jMnX/neNzTCAx//F0577inaWmYQj0ao6e4hNDbOnq98kyvOm3xlf2Aizv98/suUdR/isHfiXyiRoGV/G31nvYZ/vuqyDN/ynR/ezNIH7mZf62yGysooGRlh1t69bDn9VVzz6U9k+JbbbvoJ1ffexc758+ivqiIyMcGsffsIx+F1N1xPY/nkuOzBex5g+Ke3snPBfEa91Z3VPT3M3LOPpd/+NksWzJOEkySc/vc82zvIN3a380yv+2RjVnGMv5/ZwAeyVmT86lA3/2iYeNdHIzx5yiLGHDeQ/+PmPeybwhVRhYICYiGlTSj4aYq5m56HFbw0MMKgJUk2qzjGmTXlRJTi+b4htlhWnM0sjnFZs7vp+ksDI9aVYbWRMNd6r1AeGBvnP/eaTy6KKsU3Fs6gLBJiPOHwyW37rdf47UUzmVYUBcfh2m37OThmTtRd2zqNVVVlhJQ7oX+6V7+nB8Dfz2zg7U01hIA727u56YD5/f93NtVwTWuTm+DrGeRfNBObFKdXlfG9JbMAd0J/0Qvmo6RnFEe5/0Q3STaWSPK657fSZ7BfiVI8ceri9OrBd27YqU0KpXjopAUs8AZC125t454j5j0Zfrh0Fm+or0KhuHF/B1/dHdwMOcWn50zjI94A+fGufi57SXOctcelzbV8d7HbFvtGxjjt2S2BREGKkyvLeOAk9+nKaCLJiU9vcid5GhpiEdaevjQ9gHzj89t4QZMIATe58PwZS9OJr2u27OWXh8xt8esT53O6NxD6r72H+fIuc1v4J5yPdPbxPktb+Fcu7hoe44zV+lc3AF5dW8Edx88D3AHTiU9vMu5pN7s4xjOnLSGkFI7j8Nrnt2pfGQY38b3+jGXp1VYf3rRHmyxM8diqhekVVKZkYYrrl87mQm8V3m+O9HK1ITEM8InWJj7lDQpfHhrhVWvMGzO/rbGaHy9rBdxEzwlPbzKuOl1SVswfTl6E8trinDUvp19ZzqYyHGL9mcvSA8grX9rNbzVJIXB98Z9PXcJcb6XjV3ce5PuapFAKf7I+12mu/pWnGwaGeYPl1fN3T6tN+5buiTgnPr3J6DtXVZbym5Pc45ATjsPpz24xPjCoj0ZYe8bS9AqjSzfs1O6JCG5/WnP6Ulq8wbQpoZ7i7hPmpROiPz3YxSe3mk8C/NrCGXygxZ1YrO4d5IL15j1nrmyp56sLvdesxiZY9cxm7cMsgHNqyvnlCe4rFhNJh1XPbOKwJvkO0FIUZfVpS9MTsgvXbefZPn0ciSnF2jOWpifq127dr11dnOK3Jy1gZaU78L5hfwf/vsN8Ws9/Lp7Fu5rdBM7j3f28e8Muo9b/EGfvyBinW/zsG+oruW25+zrbSCLJSoufnVdSxJOnLk77ljeu3cYGQ8wpDYV44cxl6cnpRzbv5W5N8j3FH09elE5afH/vYb5q8bM3LGvlrY1uAuehjl4+sHGPUetfTb5taJRz1gRXx6S4sLGa6z3fMuj5FtP46bjyEh5dtTDtW85ds5Vtw3o/WxUJs/6MZZR6E7KrNu7mwQ6zb3n61CXpVdTf2NXOd/fqXwEE+OmKuekEzj1egszE5+dN58Oeb3lxYJjXW3zLe5pr+Y4Xq3s835JOTqf6lTdG98fqhONwxrNb2Ov5llh8HOXAWNT1D9m+xfYwMgw8d/rS9ETdtkcswK9OmJfeVuJnB7v4hMW3fH3hDK7wfMua3kHeZvEtV7XU85W/0Lec/MxmegcHWdq+hxk9R3BQtNU2sqV5NnXl5aw5fWn6wUwu37LujGXUe6taC8G35BrD+X1LrjGc37cAvOH5rWwYGEE5SeoG+wk5SbrLKomHI6+Ib6keHqDSS2Z3lFeDUhm+ZfvQKGenfIvjUDE8xGgsxoR337+9sZof+XzLiU9vYiDh7jVV19fDRCRCf7nbj7N9y6ue28qhjg5O2rOV1q52lONwsLqeta1LGKut1/qW1gP7OWf9akrHRtk+cw5PnHgK8WhU61tmHG7ndc89ReXwIO21DTx86jn0VVRqfUtNfy+vW/0UzZ1H6Kyu4dFTz+ZIbb3Wt7QcaeetT/6euW17GSwt44+rzuDPx6/i3S31Ad/ScLCNt//xYRYc3Ec8FOL5xSu4/5zXsri5iQdOWiAJJ0k4TR29E3HGkg4NsYhxr5X/3neEr+9qz3D0M4tj3Lp8Dst8T1dSTx9MfGvhDM6sqaAvnuBH+45YEyeVkTAnVJQwnEiyZ2Q8/dqWIAh/Gf7enSsqhFP/wcm9gjGsIORtA5nrpMwwpAd5phV9fm005L5bPpp0rHUOQfopz1gimbPOZZ52POkYB7B+rQLijhNYlZJNaThEREHCIefhDMUhRcybTPfH7doipSjx6tEXTxgHmuAmkisi7vUNxJPW6wsB1VF38DicSOa8vppIGKVgLOnkvL6qSJiwgrjjrly0UR4OpdtC95TZT0lIpVci9UwkrPdFTCnKIyEUiv54wtoWYUgnAAcTiZxtUe9pR5NJ68MFcFfLhZViIunQm6MtKiMhikIhkg505Yh7ZeFQ+l7uGI9b26I4pNIThZ6JOBMWcUSRXlnQH7e3hcJNPIN7D+Vqi7pohLB3D9kOJAGojoSJhRQJx6Erx31RHg6lB/+6FaR+Snxt0TURt/qtqFLUen2kL0dbhJhsi8FEMmcfaYhGCCnXv+Vqi9pImGhIEc+jLSrDIUrCIRxyt0VpKJT2F53jcavvjClFjdcWvfGE9QFVGNKT9IFEkuEcbdEYi6CAkaST01/URcNElGLCcXL6i6pImOKQIonbR2yUhUOU+/qTrcZFIUV1uj8lrPEsrCb9xUA8GVjpmU1TLIJCMZxIWFf3g9ufIsqNZaakQorqSJiikCLpoF016afc51uO5OFbqr29y7om4lY/G1WKOq8t+uJx7cr3FAr3vgA3nubyLfU+35LLz9akfQs55xcV4RBl4TAOTs62KAkpqry26JyYeMV8S2MsQgjFaDKZsy1qo2FiKkTc27PURmUkTGkohINjfFiQojQcotJ7kNQxPmH1LUVKpeNvTzye07ekHi4MJBI52yLVn0aSyZx+ti4aJqpCTDjJnH62KhKmJBQi6d0XNsrCISq8tjgyPpHTt9R491D3RNzqWyIK6r3XFPvjiSn1LX8+dTHzy0ok4SQJp78uHeMTPNjRR99EgsXlxbymtjKwV0g86XD1pt0Z+wWluLiphu8vmZVOau0fHeec1VuMweZ/fK9Z5Ho6tqqylG8smslIIsmWwRE+ZUl6hYAPzWygNBxiIJ7ghrZOa/BYVVlKZSTMSNLh2d5Bq7YopCgKKeIOOQdXgiAIgiAIgiAIglBI3H3CPM6urcw74RTczVEQ/gIaYtH0MlgTkZDixmVzuOtwN79o7+agt7Hve6fXcUFjdcYKKneF1Fz+buPujCxrCHcptn9Pj4VlxfzjrEbtfhqV4RDfXDQzvUfGqqoyftfZz++79SeAvLu5NuMd6954wvhaT2p/itSKjI9t2ccvDgX3bkjxyKpF6ZP1cr3q8aUF07mwsYa44/DTA138h2Xp9uvrKrlmdhNJx+GZ3kG+ZnmdZnZxjC/NbwEF+0bG+TfLct4ipfjawhaioRD98QSf334AW47+Y7MbaS6KkXQcvrm7nV7LqoxLmmpYXlFC0oGbD3QaXyEBOKWqjLNrykk67rvmm62vF0Y539uTYX3/MGsMy6DBfaLwLm8vi70jY/zesHwc3PvuoqYaikMheuJx47L7FK+uraAuGmHCcXjgSK/1acWJFaW0lrjLex/p6rc+jZlXUsSyihIcB1b3DVqfmtRFI5xcVYoDvDw4ml5Kr6MopDiruhwHODg2od3nJYUCzqwuJxpS9E4ktHsF+VlZ4SZlx5xk+hVdEwtLi2guiuHg8EzvkPVJaEtRlDnePgQvDgxbn8ZUR8JpH7BjeNTabkVKsbLKfT3t4OiEtd0UsKqyjLByV4XY9rMDd0l2WTjEaDJpfOUlxbySIupiEZKOw7r+Yes91FwUpcXbM2jT4Ij1qXB1JMw8b+n2ruEx61Pv4pBKr1I9ODZBu2ZvkxQKOL6ilJDXFrtHzO0G7mt1xaEQI8mk9X4Ddy+3qojbFi8NjlgT+81FUabFJtvC9vSvNhpmdrHbFjtHRq0ryUpCikVlqbYYt95DIWB5havtnojnPFxjcVkxsZBiOJFkR457aE5JjIpImHjSsfpCcPfWqo9F3JNvhkasT9Nro2FmeK/I7Bwes/qhsnAofQ8dGJ2wrrQKK1hWVgLKXSGj2x/Hz5KyYqJKMZBI5LyH5pUUURYJMZF0rK+dg+sv6ry22DQ4Yn2aXh8NM917tXf78Ki1P5WHQ8z1/ND+0XFrf4oqWOLdQ0fGJziU46n30rJiIkrRF09Y/RDA/NIiSkMhxpIOWw2vkKWYWRylJhJx917J0Z8aohH3VXlg2/CodWVBZTiU3htm7+i4dSVSkVLp8dCh8Qlrf1LA0vJiwih64wnreAHcsVnKt+TqT7OKY1RHwiRw2Dw4am2LpliEJs+3bBkatcan6kiYWV5/2j0y5r6mY6A4pFjovVZ/cGzCuookhNsWCkX3RFy7r5+fRWXFFCnFUCLJTs3el35aS2JUhsPEndy+ZVosSkMs4o4vcviWmsikb9k1YvctpSGfbxkbt65QCwNLyktQuKuQbPEJXD8bVYrBPHzL3JIiysIhxh3HeogSuH62Luq2xebBEWusrvP5lh15+JY5Pt9iW4kUVaQPdDkyPpFzddHSsmLCyl3Zm8u3zCspojQcYiyZZFuO/jSjKEpNNEIyj/70f+lbFnq+5XCeviXk+Zb9U+xbqiJhEo4bn3L5lkbPt7ycw7dU+XzLnjx8S2rLjvYp9i25ThLPRlY4CQXNYDzBr4/0sGN4jPpohAubarQ3ueM43Hmomx/v72DL0CgxpTi/oYpPtE4LnIjUNxHnQ5v28njWZtIXNlbzvcWzMjZYG4on+MDG3TzRM5ihbS2JcceKeRmnonWMT3DBuh3aDa0/PaeZj7VObjjZMxHnreu2ax3WGdXl3Hn85CbHg/EEb1q7TbvHSVUkzMOrFtLqOeJ40uGt67YbkwC3ZW1+bduz4FOt0/jEnMkT+z67rc24f9Jraiv52fGTxyebNoYFN3j98ZRF6et7tLOPyw176pSFQ/zplMXpwcpGb+8UU6i58/i56dOZOsfjnP7sZqMz9p8cGE86nLl6izHw+vdkAbh4/Q6e6h3UaldUlPDwSQvTe53Z3umviYR5/vSl6dPAbmzr4N+265OAEQV/OmVy4/nfd/Vz2Yvmd/r9tt4+NMq5a142DoT+dc40/sl7530wnuCkZzYblxZf1FTDD5e6G8TnOtXS/8472O+3mkiY589Ymt6r5+u72vmeIdEaAp70bcJ/l7dBtQn/Xj3r+4d501rzHhn+zWE7xic4+ZnNxqXs75xWw/e9zfLjSYfTVm+mzZBcOKGilIfcd97d//vCjoBfSVEXjfD86UvTr4Z9ZlsbNxv6Xhh4+rQl6cHYzW0dfMZwDwHcsWIur/be/3+qZ4CLLXuXfXH+dD7o9ZH9o+Oc/uxm48TCfxroaCLJqmc2Gwc3p1aVcd/KyVOczl+7jXWaDd/BHYg959sHzLa3V1Qp1py+JL0PWK69ve49cX76JMWHO/t4v2Vvr28unMH7vIcqO4ZHOXv1y8YBpP8wgMF4gpVPbzImRM+tqeAXJ7j7gDmOw6uf22pMwM0odvctSj3kuHrjbn5j6E9FIcXa0yf3Fsm158xDJy1Mn9z568M9fNiy58z3l8zinV6y/qWBYV5n2XPmmlmNfMbbW6RnIs7KpzcZJ1lvrK/kVm9vkYTj+uQ9honhgtIi/nTK5N4il23YZXyQVB4Ose6Myb1Fcu05499bJNeeM/69RXLtOePfW+TwmOtbTAlR/94iE0mHU57dbJxQZ/vZt6/fbkzuV0fCrPPtLWKLTyHcPdFS4xxbfILMfYv+1D3AuzaYfYt/b5F93n5Wprh+WXMt/+HtLTKSSHLSM5uMiQj/vkWO4/CmtduN+wY2xFw/m9q36KNb9nKXwbdEFKw+bXJPtB/sPcxXLL7Fv2/R7zr6uGKj2bf49y3a5sVqk2/x71uUseeMBv++RVPtW54/fXJPtFy+xb9vUS7f4t+3KJdv+eisRj77F/oW/35W2cwvLeIJn2+x7WdVFg6xXnwLUJi+Ze/IGGdYfIt/TzTxLUfvW45mD6fwddddl4/umOLHP/7xdR/84Adf6WoIU0AsFOL4ilJeXVvJqdXlVGmO6QZQSrG8opQrWur56OxGPtE6jbc21miPpy4Oh7h4Wi2vq69kTkkR59VW8IUFLVzREjwyPBYKcXFTDadXl1MdibC8wj1y+msLZ1CfdcRxWTjMRU01lIRCdIxPEFWKU6vK+PKCGVzqO8UCoCQc4oLGGkaSSXYOjzLuONRHI3xoZgPfXDQzI+kVC4V4a2M1baMT7ByezJKfUV3ODctaMxJqIS/Rtnt4LCOZ1RSL8O1FM3lLY+bR86+rq+Lg2Dgv+55EFIcUH53VxCfmTMvYIP7smnIOj08ETnx6bV0lP1o2O+Oo7OMrSgjhbnjud93Ly0u4fcUcan1tN6+0mLpYhKe6BzK0ddEItxw3h2UVk/uANRZFmVtaxKOd/RnaMPClBS1c1DR5+k5pOMSqyjJ+19kXeHLyvul1/Muc5vT1hZTinNoKHunsCzjYs6rL+cGSWRknp5xTW8Hj3QOBCXVrSYzbl8+l2ndyyhnV5TzXPxRIRFSGQ9y+Yi5zfCenHF9RStvoeGAz6YhyB2Jn+k45m1taRNJbCZTNNbMa+YDvFJm6WITmoiiPdvUHgs3r6yr58oIZ6YFmLBRiRUUpD3b0BpILy8qLuX5ZazoRopTi9OoyHjzSF3g3vCkW4fblc6nz2fqM6nJ+3zUQWBFRElLcfNwc5vvu5VWVZazrHwoMChXuJuDn+o59X1pWTPvYBC9pTiP70IwGrpoxedhBc1GUopDiSU2y53V1lXxxQUu6LcrCYeaXFvNQZ18gWbesvJgfLZ1si5BSrKos44EjvYxlDfSaYhFuWzEnoy1Oqy7nd519gSd1paEQNy9vTSfTwF3l91TPIIeyTmZLtYX/tK7l5aVsHR7VJqj/fmYDV/gOfphVUsRYMqldCfjG+ko+N2+yLaoiYaYVRXm0M3gPLS8v4QdLJpP1kZBiRUUJDxwJ3kPNsSg3L5+T3o8hdX0PdvQGnnyXhUPcvHxOOqHuasv5Q3d/oO+FgO8snpVOIAGcUFnKiwMj7NY8BPjn2U28u3nSL88rKaJ7IqEdQL6loYrPzJuenoDURiNUR8PaSciJFaV8b8mstD+MhUIsKivmNx3Be6ilKMqNx7Wm/YVSipOr3HsoO8lZEQ5x6/K56eQ7wKlV5TzS2R9YVRMGfrBkdsZJgCdVlrGmb1CbEL12zjQuappcNbyorJi20Qk2afrTO5pq+GTrZGxosvSnU6rK+NbimWnfWRIOMbe0mIc6g6s+ZxfHuGFZa/qwhZBSrKws1fanqkiYW5bPyThZ89TqMh7S9KeoUvxo6ez0pvrgThye6h3UTrI+N296epUswLLyEnYMj2mT6pc11/KRWY3ptmgpjpFwHO0Gw2dVl/O1hTPSY4xyb9XHw519gf60oLSI65e1ppPvYaU4vqKU33T0Bp5810Uj3Lp8Dg2+UxFPrSrntx19gT1tikKKHy9rTa8sAji5qozHuwe0qwC+sqCF83wnly0vL2XT4Ih2pcxVLfVc7fOzs4tjDMaT2pNDz6utyPCzVdEIjUURHusKJgyXlBXzX0tnp/1sNKQ4rtxti2zf0hiLcMvyOen9xVL96TdHegPxqSSkuOm4OczN8LPlPNbVH4hPIeBbi2dmxN8TKktZ3z+sTVp8ZFYj7/Wd1jWvtIiO8bh2VeubG6r47NxJ31IXi1AZCWkPA1hRUcL3l8zO8C2LLb7lpuPmZPiWVV5/yvYt5eEQt2h8y8OdfYFVNWHg+0tmc5LPt6ysLGN17yBtmv70qdZpvMN3KuKismL2jY6zWXNgxkVNNXxqTqZviSqlfbh3cmUZ3/5f+JaTKku5/0hvIClTFQlzy3FzaPa3RXU5v+3sDayAjSrFDzW+5UlNrIaj8y3vaa7lo3n6ljOry/l6nr5lfmkR/6PxLQ8cCfqW2mhY61se7OgN+JaYUvz4uMLzLdXRCA0xs2/57yzfsqy8hAc6ekn8lXyLAr59DPuWL3zhC+3XXXfdjwMFapAVToLwCpNwHEYTSUrDoYwEj46O8Qn2j47TEIvmXM64b2SMLUOjlIdDnFLlvgJl4sDoOM/1DRENKc6sLs9ImGSzd2SMP3YPEHcczqguzzjSW1ffRzr7GUwkWFFRymlVZcZr7ByP80BHL53jE8wrLeZN9VUZx5Bnl3v3oR72jY7TXBTlHU016acD2fTHE9x9qJuXh0apjIS5sLE6Y4DgZziR5N4jPTzfN0QsFOKN9ZWcU1Oh3TB/LJnkwY4+/tjdj+PAWTXlXNBYo61zwnF4tLOf33T0MpJMckJFKZc216VXH/hxvEHFrw710D0RZ0FZMe9prs04xtbP+v5h7mjvom10nJbiGJdOq82YaPp5eWiE2w90sW1olJpohIuaqnlDfZX2+tpGx7n9QCfr+ocpDoc4v6GKixprMpKhKbrG4/y8vYsnvFWD59ZUcNn0uoykQorhRJK7DnXzu84+RpNJVlWW8f6W+ozBbop40uGBjl5+fbiHvniCRWXFXNFSr73nHMfh990D3NHexaGxCWYUx3hvcx1n1ZRr77nn+4a47WAnO4bGqI9FeEdTDW9trE4PVPxsHxrllgOdvDAwTFk4xFsaqrlkWm36SZ6fg6Pj3Hqgkz/3DhJWilfXVnD59Hqtrfsm4vysvZtHOvsYdxxOrSrjipZ6ra1HE0nuOtzNvYd7GUgkWFZewgda6lmhuZcT3qucvzjUTcd4nNklMS6fXse5NRXatniie4DbD3aye2SMxliUd06r5W2GtnhxYJib2zrZODhCeTjEBU01vMvQFntGxri5rZNnegeJKMXr6yt53/R67YOAjvEJbj/QxWNd/e5pbTXlXGloi6F4gp+3d3P/kd60b7lqhr4t4kmHXx/p4e5DPXSMTzC3tIj3T6/PSNKlcByHx7r6+Wl7F/tGxplWFOVdXlvo+sjaviFuPtDJ5sERqiJhLvTaQucDtg2NcmNbB8/1DVEUCvGG+kre31Kfcax3ivaxcW5p6+SP3QM4uL7lypZ6Zmnaoj+e4PYDnfy2s889KbGylKta6rU+bjyZ5K5DPdx9uJvuiQQLSou4oqU+fTpddlv8trOPnx3s4sDYBNOLorynuY43N+j9xdM9g9xyoJOtQ6PURsO8vamGdzfXZjyISLF5cIQb2zpc3xIK8eaGKi6fXqeNO/tHx7m5rYMnegZQKM6treDKlnqtv++ZiHP7gS4e8nzLyVVlXNlSn3E0fIqRRJI72ru490gvvd7ekx9oqU+ffOkn6Tjcd6SXO9u7OTg2nt4K4E31Vcb+dOuBTnYMj1EXC3NJUy0XT6vJeGiR4sWBYW5s6+CF/hHXtzRW877pdenJsZ89I2Pc1NbBUz2ubzmvtoIrZzSkX0vx0zke59YDnTzS1eeuYqgq4+oZDYEV3wBDiQR3tHdz7+Ee+uNJjqso4cqWelZp4kiqP/2ivS6Ij1IAABzcSURBVJvD4xO0lhRx+fQ6XldXGWiLlE/+ycFOdg1P+pZ3NNUEHu4BrOsb4qYDnbw0MEJFJMQFjdVc1lyXXgHsZ+fwKDe2dfJs7yBRpXhtfSUfaKlPr7rxc3hsglsOdPL7rn4mvHHL1TMa0qdZ+hmMJ/jpwS7u7+hlMJ5kRUUJV81oSK/+8zORdLjrcHfat8wrLeJ90+szJtL+tni40/Ute0fGaPZ8y4VNNVo/u7p3MOBb3tNcp/WzWz3fsqZviKKQ4g11VVzRovezB0fHuflAJ493D5B0HM6uqeDKGXo/2zcR5/aDXTzY4fqWlZWun12u8S1jySS/PNTNrw710DURZ2FZMVcY/GzScXiwo4+fe+OW6UUx3jO9lrc26P3sUz0D3HKgk21Do9RGI65vmVarHYtsGhzhprYO1vYPUxxSvLnB7U8m33JTWwdPdA+gFJxbU8mVM/Rjke6JOLcd6OQh7wHmqspSrp7RYPUtvz7cS288zuKyEq6cofctCcfh/iO93NHeRXtq3DK9jvMNvuVP3QPcdqCT7cOj1EUjXDKtlksMvmXDwDA3ZfmWy6fXpVdj+dkzMsaNbR38uWdy3HKVwbd0jE9w64FOHu3sZ9yZ9C0Lp8C33OP5liPjE8wuKeJ9Ft/yWFc/PznYxe6RMRpiUd45rYaLm2qNvuXGA51s9HzL2xqrea/Bt+zI9i117n0xFb7lJwe7eOAofMtdh7rpHI+nxy023/KTg13sGx1jWlGUd/+VfIucUicJJ0EQBEEQBEEQBEEQhCnlaBJO+uUDgiAIgiAIgiAIgiAIgvAXIgknQRAEQRAEQRAEQRAEYUqRhJMgCIIgCIIgCIIgCIIwpUjCSRAEQRAEQRAEQRAEQZhSJOEkCIIgCIIgCIIgCIIgTCmScBIEQRAEQRAEQRAEQRCmFEk4CYIgCIIgCIIgCIIgCFOKJJwEQRAEQRAEQRAEQRCEKUUSToIgCIIgCIIgCIIgCMKUIgknQRAEQRAEQRAEQRAEYUqRhJMgCIIgCIIgCIIgCIIwpUjCSRAEQRAEQRAEQRAEQZhSJOEkCIIgCIIgCIIgCIIgTCmScBIEQRAEQRAEQRAEQRCmFEk4CYIgCIIgCIIgCIIgCFOKJJwEQRAEQRAEQRAEQRCEKUUSToIgCIIgCIIgCIIgCMKUIgknQRAEQRAEQRAEQRAEYUqRhJMgCIIgCIIgCIIgCIIwpUjCSRAEQRAEQRAEQRAEQZhSJOEkCIIgCIIgCIIgCIIgTCmScBIEQRAEQRAEQRAEQRCmFEk4CYIgCIIgCIIgCIIgCFOKJJwEQRAEQRAEQRAEQRCEKUUSToIgCIIgCIIgCIIgCMKUIgknQRAEQRAEQRAEQRAEYUpRjuO80nWYcpRSHcBeza/qgc48i/m/0hZKPY41baHUoxC0hVKPQtAWSj0KQVso9TjWtIVSj0LQFko9CkFbKPUoBG2h1ONY0xZKPQpBWyj1KARtodSjELSFUo9jTVso9SgEbaHUoxC0f+16zHYcpyGv/+04zt/MF/D8K60tlHoca9pCqUchaAulHoWgLZR6FIK2UOpxrGkLpR6FoC2UehSCtlDqUQjaQqnHsaYtlHoUgrZQ6lEI2kKpRyFoC6Uex5q2UOpRCNpCqUchaAupHtlf8kqdIAiCIAiCIAiCIAiCMKVIwkkQBEEQBEEQBEEQBEGYUv7WEk4/LgBtodTjWNMWSj0KQVso9SgEbaHUoxC0hVKPY01bKPUoBG2h1KMQtIVSj0LQFko9jjVtodSjELSFUo9C0BZKPQpBWyj1ONa0hVKPQtAWSj0KQVtI9cjg/8tNwwVBEARBEARBEARBEIRXjr+1FU6CIAiCIAiCIAiCIAjC/zGScBIEQRAEQRAEQRAEQRCmlv/NEXfH0hfwRmArsAP4V4vuZuAIsDGPMmcCfwQ2A5uAj1m0xcAaYIOn/UIe5YeB9cBvcuj2AC8BL5DHsYVANXA38DKwBTjdoFvklZn66gf+yVLuP3vXthG4Ayi2aD/m6TZll6mzAVALPAps9/6tyaG/xCs7CazKof2W1xYvAr8Gqi3aL3m6F4BHgOm57hvgE4AD1FvKvQ444Gvr823lAh/16rwJ+GaO6/uFr9w9wAsW7QnAs6l7CTjFoj0eeMa79x4AKm39QmdDizZgP4s2YD+LNmA/k1ZnP0u5JvsZy862oaXsgP0s2oD9LNqA/TD4KWAOsBrXf/4CiFm0H/F06Xve5gOBn+H65o2491nUor3J+9mLuD6s3KT1/d3vA4M56nArsNvXzidYtAr4CrAN139eY9E+6SvzIHBvjnq8Bljn6Z8C5lu053najcBtQMQUO3T2s2i19jNoA7azaAO2yxXr/LazlBuwXQ59wH4WrdZ+Bm3AdhatzXZ7yIrnGOKfQWuKfTqtKfbptKbYF9BaYp+u3OvQ+05tuZhjn65sU+zTaU2xT6c1xb7A2MpiO53WZDudVms7i95kP+N4UGM/Xbkm+2nL1dnPUK7JdjqtyXY6rcl22vGuzn4WrW7cYtLqxi0mrW7cYh2fkzluMZVrsp2x7Gz7WcrWjVtMWt24xaQ12S8w/8AQ9wxa07hFp7XFPZ1eG/t0WlPsM5R7K5rYZ9Bq455Baxq36LS2uKfTa2MfmjkhZt+p05p8p05rins6rclv2uaw2X5TV+51aPqerWz0vlNXtsl36rQm36nTavtevl95C4/lL9zB3k5gLu5EaQOw1KA9B1hJfgmnZmCl930Fbkc2lauYdDJRXCd4Wo7yPw78nPwSTvW56uvT3wZc7X0fwzdQydGGh4DZht+34Dq+Eu/zL4ErDNrjvBu5FIgAj5HppAI2wA1s/+p9/6/AN3Lol+AGrMfJdD467euZdHrfSJVt0Fb6vr8GuN523+BO9h8G9jLpfHTlXgd8Mp/7EXi112ZF3ufGfO9f4D+Az1nKfgR4k/f9+cDjFu1zwLne91cCX7L1C50NLdqA/SzagP0s2oD9TFqd/Szlmuxn0gdsaKtHtv0s5QbsZ9EG7IfBT+H253d7P78e+LBFeyLQSpZfsujP936ncAcltrL99vsO7n1k9K3AKuAnTCacTOXeClyc1dYm7QeA24GQz3Y5/TvwK+B9OcreBizxfv4PXr102jOA/cBC7+dfBK7y/a2M2KGzn0WrtZ9BG7CdRRuwnUmrs52l3IDtcugD9rPVQ2c/Q7kB2+m0uCvLbbbTtbs2/hm0ptin05pin05rin0Brc53Wsq9Dr3v1GltsU9bj2zfaSnbFPt0WlPsC4ytLLbTaU2202m1trPoTfbTjgcN9tOVa7KfTqu1n6kOBtvpyjXZTqfV2i7r76XHuyb7GbRa+xm0RvtptFrb6bQm2xnK1drOojf2P109dPYzlKu1n0GrG7do5x/oxy0mbSDuWbTauGfR68YtxjkTwXGLqdxbCY5bTFrduCXnvA0v7lnK1cY9g/5KNLEPw5wQ/ZzBpNXNGUxa3ZzBpNXNGYxzWIJzBlO516H3mya9bs5gnUv7+56lXN2cwaTN6TttX38rr9SdAuxwHGeX4zjjwJ3ABTqh4zhPAN35FOo4TrvjOOu87wdws8YtBq3jOM6g9zHqfTmmspVSM4A3AzfmU5d8UUpV4SYPbvLqNe44Tm8e//U1wE7HcfZaNBGgRCkVwb1RDxp0S4DVjuMMO44TB/4EXJT6pcEGF+AOHvD+vdCmdxxni+M4W7P/sEH7iFcPcDO9Myzaft/HMjwbWu6b7wLX4rP1Ud5jOu2Hga87jjPmaY7kU7ZSSgHvxA2QJq2Du9IFoArPhgbtQuAJ7/tHgXd4WlO/CNjQpNXZz6IN2M+iDdgvRz/OsN/R9Pkc+oANc5Xtt59FG7CfRRuwn8VPnYf7ZA4mbafVOo6z3nGcPZq2MOl/6/3OwV3JM8Oi7fe1RclksUGtUiqM+yTr2lx1yK5rDu2HgS86jpP0dEdylauUqvTa8N4cZevsp9MmgHHHcbZ5P0/3v+zY4bVVwH46rVc3rf0M2oDtLNqA7Uxane1MWhsGfcB+ucrOtp9Bq/WdGm0dBttZMMa/bHS+06LVxj6DVhv7LARi3xRgjH02smOfAa39DAR8p2VsFbCdSauznUWrtZ1FH7BfjvFghv2OZuxo0Qbsl6tcv+0s2oDtLFrtuCUL/3g3V99La/Poe35trr7n1+bqe9njc1vfy2csb9Ln6n+Bsi19z6/N1ff8WpP9sucf7RjinkZ70BT3DFpt3LPotbFPpzXFPp1WU1ebVhv3bOVmxz2D1ma7bP0Q+thnmhPq+p5Wa+h7Jq2u75m0ur5nm8Nm9z3rfFeDSa/re9ays/qeSauzn0mbj+808reScGrBzaqmaMMySfxLUEq14mbIV1s0YaXUC7ivJj3qOI5RC3wP96ZN5vHnHeARpdRapdQHc2jnAB3ALUqp9UqpG5VSZXn8jXdjGaw5jnMA+DawD9fR9zmO84hBvhE4WylVp5Qqxc2qzszx95scx2n3vj8ENOVR57+EK4GHbAKl1FeUUvuBy3AzxybdBcABx3E25Pm3P6KUelEpdbNSqsaiW4jbfquVUn9SSp2cZ/lnA4cdx9lu0fwT8C3v+r4NfNqi3cRk4vYSNDbM6hdWG+bTh/LQBuyXrbXZz6/NZT9NHaz2y9JbbWi4Pq39srRW+2VptfbL9lO4q0N7fQE67T+P0qdZ9UqpKHA58DubVil1C+79sxj4gUX7EeB+3z2Xqw5f8ez3XaVUkUU7D3iXUup5pdRDSqkFebTFhcDv/YMXg/5q4LdKqTavLb5usMkaIKKUWuUVdzGT/S87dtRhsJ9Ga8OozbadSauznUGrtZ2lDgHbWfRa+9muj6D9dFqt7TTaTsy2A308N/nOo4n9ubR+36nVGnxnQGvxnaY66HynTmvzm7bry/adOq3Jd+q0Ot9pGlvpbHc047B8tH7bGfUa+2m1BvvZ6pFtP5NWZ79c1+e3nUmrs51Jm3PcQuZ4N9fY0zo2zlOrG3dmaA19L6DNNW7R1CHXuNOvzzX21F2fadzp1+Yad/q1Afvp5h/AWjRx72jmKrm02XHPps+OfRZtIPblqEdG7LNoA3Evj7ZIxz2LVhv3DDb5JfrYZ5oT6vre0cwf89Gm+p5Rq+l7Wq2h79nqoOt7Jr2u7+W6Pn/fM2l1fc+kzcd3mnGOYjnUsfqFe1Pf6Pt8OfBfFn0rebxS59OX4zq3i/LUV+PuqXKc4fdvAX7off8qcr9S1+JMLrHbAJxj0a4C4sCp3uf/JMeyONylyJ24nd+kqQH+ADTgPoG/F3ivRX+V12ZPAD8CvmezAW7g8P++Jx+boVnabNF+Fvd9XpXPvYDbMb+g0+Jm81cDVd7nPWS+XpR9fU24y4ZDuO9Z32zRbsSdrCnc1Xu786mz186fyNHO38dd6QJuZvwxi3Yx7nLMtcDngS5bv7DZMFubw34mrc5+xr6psV9am4f9sq/NaD+D3mhDy/Xp7Jddrs1+2dpc9kv5qbNwV4imfj4z+/5C49Oy2ywP/Q1k+QGLNgz8EPiAQXsO7n4CqWXTg7ZycV87VEAR7pO07OX/fu1gyg7evfJkHvV9KGWXHPW4h0nf/Cl8cUujPR13r4U1wJdx38EPxA7cZd0B++m0WX8rbb88tGnb5aFN285Q3+k625nKNdnOog/YL486p+1nKTdgO4s2YDvf3wrEcwy+U6f1aR4n87UsmzbDd9q02b7TUF+t7zRotb7ToLX5Tdv1ZfhOQ9la32nQBnwnhrGVznYmrc52eWizbZdzjJeyn0H7LZ39LNcXsJ9Fq7NfrutL285SbsB2Fm2uuJcx3tXZz6Q19b0cWt24xTjmJjhuSWvJPW7JvrZc45Zsva3/ma5PN27JLtc2bsnW6vqedv6BPu5Z5ypk+qtc2owxSx56f+zTad+HPvaZri8Q+yxaXdzLVV9/3DOVqx2zWPTa2IdmTog57hnnjwTjnk2b7TtzzUv9cS9b+2PMcU93bbY5n06v7Xs5ri877unKNcU9ndbqO3N95S08lr+8G/zhrJvm0xZ9K3kmnLyO9DDw8aOs0+cwvDsNfA03G78HN6s7DPw0z3KvM5Xr/X4asMf3+WzgwRxlXgA8kkNzCXCT7/P78AbaedT5q8A/2GyAu0Ffs/d9M7A1H5uRZ8IJ913kZ4DSfO8FYFZWHdNaYDnuaoQ93lccN9M/LY9ys689+/PvgFf7Pu8EGnJcXwQ4jPu6ku1v9THpfBXQn2dbLATW2PqFyYY6rcl+Jq3OfrZys+2XrbXZL49ys9tU1xZaG1quL2A/Q7la++VR5wz7+X7+OdxBRCeTg6AMf5ql/aTv8x7se6qk9bjB6168/QVyle397Bz0e+18zivvkM9+SXyDzxzlvspS7idxN22c42vjvhzXVo87KLYdopBq551Z9+fmPOv8etynh7rY8TOd/Qzan/rKTNvPps22Xa5y/bYzaHt0tsuz3LTtTHqd/XJcX4b9DNoHdbbLs86vB35puC+uw73nrPHPr/V9fhzNPjLZWgyxz1Su7/p08fY64N+xxL4c5bZayv0kOWKf4fq0sU9TtjH25ajzQtzJk3ZspbOdSauznU2rs12usv32M2h/b7Df8jzKbbWU+6DBfsss15dhO0u5Advl2Q6BuEfWeFdnP5PW1vd0Wp39bOXq+p5fS+5xp63cVoLjxuy2MPY/w/WZxp3Z5drGnbY6p/qebv7xI/RxzzpXITPuGbVoxiy5yvZ+lop9Ou1u9LEvn3JfZSn3h+jjnu36suOeqY21Y5Y866yNfXhzQvKLexnzR+xxL60ld9zTzUtNce+ruJts5xP3cs53DW2RM/ZlXV+uuJcqN5+4p6uzds5g+/pbeaXuOWCBUmqOUiqGuzzz/v9toUophft++BbHcb6TQ9uglKr2vi8BXofrAAI4jvNpx3FmOI7T6tX1D47jvNdQbplSqiL1PW4H3miqh+M4h4D9SqlF3o9egzswtnEpuZcM7wNOU0qVeu3yGty9YrQopRq9f2fhZtt/nqP8+4H3e9+/H7gvhz5vlFJvxH3l4W2O4wzn0C7wfbwAsw1fchyn0XGcVs+ObbgbNx8ylNvs+/h2LDbEDXKv9v7fQiafANl4LfCy4zhtOXQHgXO978/DPR1Ci8+GIeDfcDfTs/WLgA2Psg9ptTr7WbQB++m0JvvhOnBduVr7Wa7PZENTW2TYz1JuwH6WtgjYz+CntuCuqrnY+68p2+Xt0zyNVq+Uuhp4A3Cp4+0vYNBuVUrN913/27z/r9OudRxnms9+w47jzLfUodlX7oXARsv1pW3ntfW2HG1xMW4SZDRHW2wBqrz7gdTPLHVO2a8I+BfcjWR1seMynf2OJs6YtDrb6bTA5TrbGcqt0dnOUoeA7Wx11tkvR1tk2M9wfRfobGepc8B23mdTPNf5zrxjv0lr8J0mrc536rTPGXzngKHcgO+0XJvWb+Zoi2zfadLqfKepLQK+0zK2CtjuaMZhJq1p3GLRB+xn0K4zjF1eMpQbsJ/l+nT222xpiwzbWcoN2M7SDtpxi4/s8a5t7JnP2FirNdnPoLWNO9PaPMad2eXmGndmX59t7KlrC9O4M1trG3dm11lnP938YzOauGfQmuYqWq0u7uXQB2KfQfsdXeyzlKuLfabrC8S9HG2RPW4xtXEg7uVoC1Ps080JtX3PoNWi05r6nkGr7Xsa7W2mvmco19j3DNdnin2mtgj0PYNW2/cMdc7lO+0cTXbqWP7CfQdxG25W8LMW3R2475tOeDfMVRbtWbjv96eOTMw42jBLuwL3WOQXvRvrc3nW+1VYXqnDPXlvA5NHZhuvzfd/TsA9/vBF7yausWjLcLPcVXmU+wXczrgR94SFIov2SVxntQF4TS4b4O5D8nuvMzwG1ObQv937fgw3SfCwRbsDd4+vlA2vt2h/5V3fi7jHQraYtFnXtIfJpya6cn+Ce9Tki7hOttmijeE+qd+Ie7zoebnuX9zTLP4+j3Y+C3e55Abc5aEnWbQfw+1T23Df205lybX9QmdDizZgP4s2YD+LNmA/k1ZnP0u5JvuZ9AEb2uqRbT9LuQH7WbQB+2HwU7h+Zo3X1nfhLuE2aa/xbBfHDWapJdYmfRzXL6fq9jmdFnfp8Z+9dt6Iu3Kn0lRulv0Gc9ThD75yf4r7+qFJW437VP0l3Cdkx9vqgPvE7Y35xAPc+/4lz36Pe+1u0n4Ld3C3laxjebNjh85+Fq3WfgZtwHY6rcl2+cQ69K9D+usQsF0OfcB+tnro7GcoN2A7i1ZrOwzxHL3vNGl1vtOk1flOk1bnO3OOP5j0naZyA77TotXGPls9CPpOU9k632nSmmJfYGyls51Faxq36LTacYtFbxq7WMeDZI5ddOWaYp9Oa7Kftg7ZtrOUaxq36LRa23n6wHjXYj+d1mQ/ndY07tRpTbazjs+zbKcrV2s7i95kP209DPbTlWuyn05r6nuB+QeGuGfQmsYtOq0x7hn02tin05pin6FcbewzaLVxz1QH9OMWXbnGuGfQm2JfYE6Iue/ptKa+p9Oa+p5Oa+p7xjmspu/pyrX1PZ3e1Pe09UDf93TlmvqeTmv0nfl8pTqqIAiCIAiCIAiCIAiCIEwJfyuv1AmCIAiCIAiCIAiCIAh/JSThJAiCIAiCIAiCIAiCIEwpknASBEEQBEEQBEEQBEEQphRJOAmCIAiCIAiCIAiCIAhTiiScBEEQBEEQBEEQBEEQhClFEk6CIAiCIAgFjFLKUUrNf6XrIQiCIAiCcDRIwkkQBEEQBOEoUErtUUqNKKUGfV//9UrXSxAEQRAEoZCIvNIVEARBEARBOAZ5q+M4j73SlRAEQRAEQShUZIWTIAiCIAjCFKCUukIp9Wel1H8ppfqUUi8rpV7j+/10pdT9SqlupdQOpdTf+X4XVkp9Rim1Uyk1oJRaq5Sa6Sv+tUqp7UqpXqXUfyullPf/5iul/uT9vU6l1C/+ipcsCIIgCIJgRFY4CYIgCIIgTB2nAncD9cBFwD1KqTmO43QDdwIbgenAYuBRpdROx3H+AHwcuBQ4H9gGrACGfeW+BTgZqATWAg8AvwO+BDwCvBqIAav+ry9QEARBEAQhH5TjOK90HQRBEARBEI4ZlFJ7cBNKcd+PPwVMAF8FWhxvgKWUWgP8AHgc2ANUO44z4P3ua0Cz4zhXKKW2Atc6jnOf5u85wNmO4zzlff4lsM5xnK8rpW4HRoEvOo7T9n9wuYIgCIIgCH8R8kqdIAiCIAjC0XOh4zjVvq8bvJ8fcDKf5u3FXdE0HehOJZt8v2vxvp8J7LT8vUO+74eBcu/7awEFrFFKbVJKXfkXXo8gCIIgCMKUIgknQRAEQRCEqaMltb+SxyzgoPdVq5SqyPrdAe/7/cC8o/1jjuMcchzn7xzHmQ58CPihUmr+X1Z1QRAEQRCEqUMSToIgCIIgCFNHI3CNUiqqlLoEWAL81nGc/cDTwNeUUsVKqRXAVcBPvf93I/AlpdQC5bJCKVWX648ppS5RSs3wPvYADpCc6osSBEEQBEE4WmTTcEEQBEEQhKPnAaVUwvf5UeA+YDWwAOgEDgMXO47T5WkuBa7HXe3UA3zecZzHvN99ByjC3QC8HngZeHse9TgZ+J5Sqsr7ex9zHGfX/+bCBEEQBEEQpgLZNFwQBEEQBGEKUEpdAVztOM5Zr3RdBEEQBEEQXmnklTpBEARBEARBEARBEARhSpGEkyAIgiAIgiAIgiAIgjClyCt1giAIgiAIgiAIgiAIwpQiK5wEQRAEQRAEQRAEQRCEKUUSToIgCIIgCIIgCIIgCMKUIgknQRAEQRAEQRAEQRAEYUqRhJMgCIIgCIIgCIIgCIIwpUjCSRAEQRAEQRAEQRAEQZhS/h8eZnJxoULcXgAAAABJRU5ErkJggg==\n", "text/plain": [ "
      " ] @@ -345,7 +326,7 @@ "sns.pointplot(x3, dataplot3['sgd_mse_lr_0.001'], color=color[8])\n", "sns.pointplot(x3, dataplot3['sgd_mse_lr_0.0025'], color=color[3])\n", "sns.pointplot(x3, dataplot3['sgd_mse_lr_0.005'], color=color[7])\n", - "color_patch1 = mpatches.Patch(color=color[9], label=\"svrg_mse_0.025\")\n", + "color_patch1 = mpatches.Patch(color=color[9], label=\"svrg_mse_lr_0.025\")\n", "color_patch2 = mpatches.Patch(color=color[8], label=\"sgd_mse_lr_0.001\")\n", "color_patch3 = mpatches.Patch(color=color[3], label=\"sgd_mse_lr_0.0025\")\n", "color_patch4 = mpatches.Patch(color=color[7], label=\"sgd_mse_lr_0.005\")\n", @@ -357,21 +338,21 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 2", + "display_name": "Python 3", "language": "python", - "name": "python2" + "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", - "version": 2 + "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.15" + "pygments_lexer": "ipython3", + "version": "3.6.4" } }, "nbformat": 4, diff --git a/example/svrg_module/linear_regression/data_reader.py b/example/svrg_module/linear_regression/data_reader.py index d56ae03a5..23847d531 100644 --- a/example/svrg_module/linear_regression/data_reader.py +++ b/example/svrg_module/linear_regression/data_reader.py @@ -15,21 +15,35 @@ # specific language governing permissions and limitations # under the License. +import bz2 +import os +import shutil +import mxnet as mx import numpy as np from sklearn.datasets import load_svmlight_file + # Download data file -# from subprocess import call # YearPredictionMSD dataset: https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd -# call(['wget', 'https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/YearPredictionMSD.bz2']) -# call(['bzip2', '-d', 'YearPredictionMSD.bz2']) -def read_year_prediction_data(fileName): +def get_year_prediction_data(dirname=None): feature_dim = 90 + if dirname is None: + dirname = os.path.join(os.path.dirname(__file__), 'data') + filename = 'YearPredictionMSD' + download_filename = os.path.join(dirname, "%s.bz2" % filename) + extracted_filename = os.path.join(dirname, filename) + if not os.path.isfile(download_filename): + print("Downloading data...") + mx.test_utils.download('https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/regression/%s.bz2' % filename, dirname=dirname) + if not os.path.isfile(extracted_filename): + print("Extracting data...") + with bz2.BZ2File(download_filename) as fr, open(extracted_filename,"wb") as fw: + shutil.copyfileobj(fr,fw) print("Reading data from disk...") - train_features, train_labels = load_svmlight_file(fileName, n_features=feature_dim, dtype=np.float32) + train_features, train_labels = load_svmlight_file(extracted_filename, n_features=feature_dim, dtype=np.float32) train_features = train_features.todense() # normalize the data: subtract means and divide by standard deviations diff --git a/example/svrg_module/linear_regression/train.py b/example/svrg_module/linear_regression/train.py index b3d942973..6b6574c96 100644 --- a/example/svrg_module/linear_regression/train.py +++ b/example/svrg_module/linear_regression/train.py @@ -19,7 +19,7 @@ import argparse import mxnet as mx from common import create_lin_reg_network, create_logger -from data_reader import read_year_prediction_data +from data_reader import get_year_prediction_data parser = argparse.ArgumentParser() parser.add_argument('-e', dest='epochs', help='number of epochs for training phase', type=int, default=100) @@ -37,7 +37,7 @@ logger = create_logger() kv = mx.kvstore.create(args.kv_store) -feature_dim, train_features, train_labels = read_year_prediction_data('YearPredictionMSD') +feature_dim, train_features, train_labels = get_year_prediction_data() train_iter, mod = create_lin_reg_network(train_features, train_labels, feature_dim, args.batch_size, args.updateFreq, ctx, logger) diff --git a/example/mxnet_adversarial_vae/README.md b/example/vae-gan/README.md similarity index 100% rename from example/mxnet_adversarial_vae/README.md rename to example/vae-gan/README.md diff --git a/example/mxnet_adversarial_vae/convert_data.py b/example/vae-gan/convert_data.py similarity index 100% rename from example/mxnet_adversarial_vae/convert_data.py rename to example/vae-gan/convert_data.py diff --git a/example/mxnet_adversarial_vae/vaegan_mxnet.py b/example/vae-gan/vaegan_mxnet.py similarity index 100% rename from example/mxnet_adversarial_vae/vaegan_mxnet.py rename to example/vae-gan/vaegan_mxnet.py diff --git a/example/vae/VAE_example.ipynb b/example/vae/VAE_example.ipynb deleted file mode 100755 index e7ec03afd..000000000 --- a/example/vae/VAE_example.ipynb +++ /dev/null @@ -1,1204 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "import mxnet as mx\n", - "import numpy as np\n", - "import os\n", - "import logging\n", - "import matplotlib.pyplot as plt\n", - "import matplotlib.cm as cm" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Building a Variational Autoencoder in MXNet\n", - "\n", - "#### Xiaoyu Lu, July 5th, 2017\n", - "\n", - "This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Prerequisites\n", - "\n", - "To complete this tutorial, we need following python packages:\n", - "\n", - "- numpy, matplotlib " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 1. Loading the Data\n", - "\n", - "We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "mnist = mx.test_utils.get_mnist()\n", - "image = np.reshape(mnist['train_data'],(60000,28*28))\n", - "label = image\n", - "image_test = np.reshape(mnist['test_data'],(10000,28*28))\n", - "label_test = image_test\n", - "[N,features] = np.shape(image) #number of examples and features" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADFCAYAAACxSv92AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFzNJREFUeJzt3Xuw1XW5x/HPg8rdGMEtoYfj9gJqOZyNbdBGOYOCRFp5\noRwtHUpHgkzFQHFwTNMsrGMKURYIgdnxWIi3xjlSdPIymYlhBwEvpRBX2YxQQolcnvMH68wQ6/nF\nWnvd9lrf92umYe/P/u7fen6sZ+8efq7fd5m7CwAAAEhRp1oXAAAAANQKwzAAAACSxTAMAACAZDEM\nAwAAIFkMwwAAAEgWwzAAAACSxTAMAACAZDEMAwAAIFkMwwAAAEjWwaV8s5mNljRd0kGS7nP3af9s\n/eGHH+7Nzc2lPCSgl156abO7N1XzMeldlGrVqlXavHmzVfMx6VuUA79zUa8K7d12D8NmdpCk70k6\nW9JaSS+a2ePuviLre5qbm7VkyZL2PiQgSTKz1dV+THoXpWptba36Y9K3KAd+56JeFdq7pbxMYqik\nP7r7m+7+vqT/knReCccDAAAAqqqUYfgoSWv2+XxtLvsHZjbOzJaY2ZK2trYSHg6oLnoX9Yi+Rb2i\nd1ErFb+Bzt1nuXuru7c2NVX1JUdASehd1CP6FvWK3kWtlDIMr5PUf5/P/yWXAQAAAHWhlGH4RUkD\nzOwYM+ss6WJJj5enLAAAAKDy2r2bhLvvMrMvS3pKe7dWm+vuy8tWGQAAAFBhJe0z7O5PSnqyTLUA\nAAAAVcU70AEAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAA\ngGQxDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9e6AACNbc2aNWE+ffr0ML/77rvz\nsuuuuy5ce+2114Z5//79C6wOAJA6rgwDAAAgWQzDAAAASBbDMAAAAJLFMAwAAIBkMQwDAAAgWewm\nUWF79uwJ8x07dpTl+PPnzw/z7du352UrVqwI195zzz1hPnXq1DCfOXNmmHfr1i3M77rrrrxswoQJ\n4VrUr3Xr1oX54MGDw3zr1q1hbmZ5WVaPZvV/W1tbmAMd2cqVK8N85MiRYf7yyy+HeVNTU9lqQppm\nz54d5uPHjw/zrFnntddeC/OBAwe2r7AKKWkYNrNVkt6VtFvSLndvLUdRAAAAQDWU48rwme6+uQzH\nAQAAAKqK1wwDAAAgWaUOwy7pl2b2kpmNixaY2TgzW2JmS3gdH+oJvYt6RN+iXtG7qJVSh+Ez3L1F\n0sclXWVm/77/Anef5e6t7t7Ki/pRT+hd1CP6FvWK3kWtlPSaYXdfl/tzk5k9ImmopGfKUVg1/eUv\nfwnz3bt3h/kf/vCHvGzRokXh2qw75mfNmlVgdeXT3Nwc5pMmTQrzOXPmhHmvXr3CfNiwYWF+1lln\nHbg41I3Vq1eH+fDhw8N8y5YtYR7tGiHF/dWlS5dw7aZNm8L8zTffDPOjjz46zA866KAwT9kbb7yR\nl2U9l0OHDq10OUl44YUXwnzEiBFVrgQpWbx4cV72la98JVzbqVNx11Czfs93NO2+MmxmPczs0P//\nWNIoSa+UqzAAAACg0kq5MtxX0iO5qf9gSf/p7v9dlqoAAACAKmj3MOzub0r6tzLWAgAAAFQVW6sB\nAAAgWQzDAAAASFY53oGubqxduzbMW1pawjzrzul6EN3xmbU7RLdu3cL8iiuuCPMjjjgizHv27Bnm\nbJHTse3cuTPMs3aNGD16dJivWbOmLPVEP4933HFHuPaMM84I8wEDBoR51i4uWb2esugO81dffTVc\ny24SxXH3MI928JCk119/vZLlIHFRf7333ns1qKR2uDIMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACS\nxTAMAACAZCW1m0SfPn3CvG/fvmFei90kRo0aFeZZtS9cuDDMu3TpkpcNHz683XWhcV1//fVhPnPm\nzCpXstfTTz+dl23fvj1ce8EFF4R51s/F0qVL219YYmbMmJGXZf1+QnG2bdsW5t/85jfD/Nprrw1z\ndupBMVasWBHmt956a8HHOOWUU8J80aJFYd6jR4+Cj11LXBkGAABAshiGAQAAkCyGYQAAACSLYRgA\nAADJYhgGAABAspLaTaJbt25hPm/evDBfsGBBmH/0ox/Ny8aMGVNULWeccUaYP/bYY2HeuXPnMN+4\ncWOYT58+vah6kIY1a9bkZQ888EC41t2LOnbWzg5ZPxuXXnppmPfv3z8vO+mkk8K1U6ZMCfOsn91i\nzyllu3fvrnUJDWv8+PFFrc/qfyDyxz/+MczPOeecMH/nnXcKPva0adPCvFevXgUfoyPiyjAAAACS\nxTAMAACAZDEMAwAAIFkMwwAAAEgWwzAAAACSdcDdJMxsrqRPSNrk7ifnst6SHpLULGmVpIvcfUvl\nyqysIUOGhPmgQYPCPNrZ4YYbbgjXfutb3wrz22+/veBj/zMf/OAHwzzrPe6RhnXr1oX54MGD87Kt\nW7eGa80szD/3uc+F+ezZs8N8xYoVRa2/+OKL87Lu3buHa4888sgw79Qp/nf+j3/84zC/8cYbwzza\n2aLRrF+/PsyzegilK+bufUk6++yzK1QJGtF9990X5tFuQlkuvPDCMD/zzDPbVVNHV8iV4XmSRu+X\n3ShpsbsPkLQ49zkAAABQVw44DLv7M5L2/2fseZLm5z6eL+n8MtcFAAAAVFx7XzPc19035D7eKKlv\n1kIzG2dmS8xsSVtbWzsfDqg+ehf1iL5FvaJ3USsl30Dne9/SKfNtndx9lru3untrU1NTqQ8HVA29\ni3pE36Je0buolfa+HfPbZtbP3TeYWT9Jm8pZVEfRpUuXgtcedthhRR17xowZYT5s2LAwz7qZCWnb\nvHlzmN95551hvmVL/n2uffvG/2HnmGOOCfMJEyaEedbNny0tLUXllfS3v/0tzL/97W+HedbPaSNZ\ntGhRmGf9XaFw27dvD/Nly5YVdZw+ffqUoxw0mGJ/n2XdWBz1V9ZN/o2qvVeGH5c0NvfxWEmPlacc\nAAAAoHoOOAyb2YOSnpd0gpmtNbMrJE2TdLaZvSFpZO5zAAAAoK4c8GUS7n5JxpdGlLkWAAAAoKp4\nBzoAAAAki2EYAAAAyWrvbhLYz8SJE8P8d7/7XZg/8sgjYb58+fIwP/nkk9tXGBrCrl27wnzy5Mlh\n/sADD4R5r1698rKnnnoqXHv88ceH+c6dO8O8nr311lu1LqFmXnnllYLX1mIHkHp20003hXnWW2AP\nGjQozLN2akEatm7dGubnnXdeWY5/66235mUnnnhiWY5dL7gyDAAAgGQxDAMAACBZDMMAAABIFsMw\nAAAAksUwDAAAgGSxm0SZZN3tO2vWrDBfvHhxmGfdHXr++eeH+emnnx7mF1xwQV5mZuFadHx//vOf\nwzxr14gsv/3tb/OygQMHFnWMbt26FbUejePUU0+tdQlVs2PHjjB/6aWXwjz6Xf/QQw8V9ZgzZswI\n865duxZ1HDSWZ599Nsx/85vfFHWcz3zmM2H++c9/vtiSGg5XhgEAAJAshmEAAAAki2EYAAAAyWIY\nBgAAQLIYhgEAAJAsdpOosN69e4f5U089FeajR48O83vuuaeofO7cuXnZmDFjwrU9e/YMc3QcV111\nVZi7e5hHu4lIxe8cUa/27NkT5p06xf/+z/p7xD/aunVrRY+/fv36MI+ez6effjpc+9Zbb4X5+++/\nH+bf/e53w3z37t1h3qNHjzAfNWpUXpa1C8TOnTvD/KSTTgpzpOHFF18M87FjxxZ1nE9+8pNhPnv2\n7DBntxKuDAMAACBhDMMAAABIFsMwAAAAksUwDAAAgGQxDAMAACBZB9xNwszmSvqEpE3ufnIuu1XS\nlZLacsumuvuTlSqyEQ0dOjTMly9fHubXXXddmP/sZz8L88svvzwv+9Of/hSuvf7668P80EMPDXNU\nztKlS8P8mWeeCXMzC/Os96BPRdauEVl/X62trZUsp0Pr3r17mEd/V5/61KfCtSeccEJZann++efD\nPNrt4+CD4//7ytod59RTTw3zyZMnh/mwYcPCvKWlJcyjXSb69+8frt2+fXuYNzU1hTkaS9auLKed\ndlpZjn/88ceHedZOKCjsyvA8SdF+X3e7e0vufwzCAAAAqDsHHIbd/RlJ71ShFgAAAKCqSnnN8NVm\n9r9mNtfMDstaZGbjzGyJmS1pa2vLWgZ0OPQu6hF9i3pF76JW2jsM3yvpWEktkjZIuitrobvPcvdW\nd2/l9VCoJ/Qu6hF9i3pF76JW2jUMu/vb7r7b3fdImi0pvhsMAAAA6MAOuJtExMz6ufuG3KcXSHql\nfCWlrV+/fmE+b968MB8/fnyYjxw5Mi+74447wrWvvfZamD/00ENhjsp57733wnzHjh1hfuSRR4b5\nueeeW7aaOoJdu3aF+YwZM4o6zqc//ekwnzp1atE1NYrbbrstzI877ri87Ne//nVFaxkwYECYf/az\nn83Lsu6YP+aYY8paU6GefDL/PvKNGzeGa0888cRKl4MO7K674v+YnrULTrGmTJlSluOkpJCt1R6U\nNFzS4Wa2VtItkoabWYskl7RK0hcrWCMAAABQEQccht39kiCeU4FaAAAAgKriHegAAACQLIZhAAAA\nJIthGAAAAMlq124SqL6uXbuG+fDhw8P8oIMOysuy7sh/9NFHwzxrl4kTTjghzFF9WX3Rs2fPKldS\nPlGf3nvvveHaG264Icybm5vD/Kabbgrzzp07F1ZcQsaOHVtQhr1+/vOfF7z28ssvr2Al6EjWrVuX\nly1YsKAsx/7CF74Q5uzRXDyuDAMAACBZDMMAAABIFsMwAAAAksUwDAAAgGRxA10Hs379+jBfuHBh\nmD///PNhnnWzXGTIkCFhPnDgwIKPgdq47LLLal1Cu0U3lkjSnXfemZd9//vfD9dm3UAye/bs9hcG\nVNiFF15Y6xJQJa2trXnZ5s2bizrGxz72sTCfOXNmu2pCPq4MAwAAIFkMwwAAAEgWwzAAAACSxTAM\nAACAZDEMAwAAIFnsJlFhbW1tYf69730vzH/0ox+F+dq1a0uuJXqLZin7rWvNrOTHRHHcvah83rx5\nYX7zzTeXq6SSPfjgg2F+9dVXh/mWLVvysmuuuSZce/fdd7e/MACosE2bNuVlnToVdx1yypQpYc7b\nyJcPV4YBAACQLIZhAAAAJIthGAAAAMliGAYAAECyGIYBAACQrAPuJmFm/SXdL6mvJJc0y92nm1lv\nSQ9Japa0StJF7p5/G3gD2rZtW172xBNPhGtvu+22MH/99dfLWtP+zjrrrLxs2rRp4dqPfOQjFa0F\nhcvawSMrz9plJKvvrrjiirzs0EMPDdcuX748zH/4wx+G+bPPPhvmq1atCvPjjjsuzC+++OK8LGs3\nCaAjy9oFZvXq1WF+7LHHVrIcVNDkyZPDfM+ePSUfe9CgQSUfA/9cIVeGd0ma5O4fknSapKvM7EOS\nbpS02N0HSFqc+xwAAACoGwccht19g7v/Pvfxu5JWSjpK0nmS5ueWzZd0fqWKBAAAACqhqNcMm1mz\npMGSXpDU19035L60UXtfRhF9zzgzW2JmS7LegALoiOhd1CP6FvWK3kWtFDwMm1lPSQ9Lmujuf933\na773hVHhi6PcfZa7t7p7a1NTU0nFAtVE76Ie0beoV/QuaqWgYdjMDtHeQfgn7r4wF79tZv1yX+8n\nKf89BwEAAIAOrJDdJEzSHEkr3f07+3zpcUljJU3L/flYRSqsgu3bt4f5mjVrwvzSSy/Ny5YuXVrW\nmvY3atSoMP/a174W5kOGDMnLsnYkQP3avXt3mGftJjFnzpy8rHfv3uHaZcuWtb+wfXz84x8P89Gj\nR4f5l7/85bI8LlBrWb9zy7HDAGpj3bp1Yb5gwYIw79Qp/5pjly5dwrW33HJLmPfo0aPA6tBeBxyG\nJZ0u6TJJy8zs5Vw2VXuH4J+a2RWSVku6qDIlAgAAAJVxwGHY3Z+TlHVJcUR5ywEAAACqh3egAwAA\nQLIYhgEAAJAshmEAAAAkq5Ab6OrO3//+9zCfOHFimD/33HNh/uqrr5atpv2dc845Yf7Vr341zFta\nWsL8kEMOKVtNqL0Pf/jDYT5y5Mgw/+Uvf1nU8deuXZuXZd0dneWII44I8wkTJoT5zTffXNTxgUb3\nq1/9KsxHjOA2nI5u27ZtYV7M79Hm5uYwnzJlSntKQhlwZRgAAADJYhgGAABAshiGAQAAkCyGYQAA\nACSLYRgAAADJqpvdJFatWpWXfeMb3wjXZt1hv3r16nKW9A+6d+8e5rfffnuYf+lLXwrzzp07l60m\n1J8PfOADYZ71vvf3339/mF9zzTUl1/L1r389zK+88sow79OnT8mPCTQSd691CQAKwJVhAAAAJIth\nGAAAAMliGAYAAECyGIYBAACQLIZhAAAAJKtudpN4+OGH87I5c+aU5dinnHJKmF9yySVhfvDB+X9t\n48aNC9d27dq1/YUBOT179gzzrF1JsnIAlTFmzJi87Ac/+EENKkElHXXUUWF+7rnnhvkTTzxRyXJQ\nJlwZBgAAQLIYhgEAAJAshmEAAAAki2EYAAAAyTrgDXRm1l/S/ZL6SnJJs9x9upndKulKSW25pVPd\n/clKFTpp0qSCMgAAqm3EiBF52Z49e2pQCSop62bmRx99tMqVoJwK2U1il6RJ7v57MztU0ktm9ovc\n1+529/+oXHkAAABA5RxwGHb3DZI25D5+18xWSor3FgEAAADqSFGvGTazZkmDJb2Qi642s/81s7lm\ndljG94wzsyVmtqStrS1aAnRI9C7qEX2LekXvolYKHobNrKekhyVNdPe/SrpX0rGSWrT3yvFd0fe5\n+yx3b3X31qampjKUDFQHvYt6RN+iXtG7qJWChmEzO0R7B+GfuPtCSXL3t919t7vvkTRb0tDKlQkA\nAACU3wGHYTMzSXMkrXT37+yT99tn2QWSXil/eQAAAEDlFLKbxOmSLpO0zMxezmVTJV1iZi3au93a\nKklfrEiFAAAAQIUUspvEc5Is+FLF9hQGAAAAqoF3oAMAAECyGIYBAACQLIZhAAAAJIthGAAAAMli\nGAYAAECyGIYBAACQLIZhAAAAJIthGAAAAMliGAYAAECyzN2r92BmbZJWV+0BK+9wSZtrXUSVdKRz\nPdrdm6r5gPRu3epI50nflkdHek4rqSOdJ71buo70fFZaRzrXgnq3qsNwozGzJe7eWus6qiGlc01B\nKs9nKueZklSe01TOMxUpPZ/1eK68TAIAAADJYhgGAABAshiGSzOr1gVUUUrnmoJUns9UzjMlqTyn\nqZxnKlJ6PuvuXHnNMAAAAJLFlWEAAAAki2EYAAAAyWIYLpCZzTWzTWb2yj5ZbzP7hZm9kfvzsFrW\nWA5m1t/M/sfMVpjZcjO7Npc33Lmmgt5tvHNNAX3beOeaCnq3/s6VYbhw8ySN3i+7UdJidx8gaXHu\n83q3S9Ikd/+QpNMkXWVmH1Jjnmsq5onebbRzTcE80beNdq6pmCd6t67OlWG4QO7+jKR39ovPkzQ/\n9/F8SedXtagKcPcN7v773MfvSlop6Sg14Lmmgt5tvHNNAX3beOeaCnq3/s6VYbg0fd19Q+7jjZL6\n1rKYcjOzZkmDJb2gBj/XBDX080nvNqyGfi7p24bW0M9nvfcuw3CZ+N496hpmnzoz6ynpYUkT3f2v\n+36t0c41dY32fNK7aWi055K+TUejPZ+N0LsMw6V528z6SVLuz001rqcszOwQ7W3sn7j7wlzckOea\nsIZ8PundhteQzyV9m4SGfD4bpXcZhkvzuKSxuY/HSnqshrWUhZmZpDmSVrr7d/b5UsOda+Ia7vmk\nd5PQcM8lfZuMhns+G6l3eQe6ApnZg5KGSzpc0tuSbpH0qKSfSvpXSaslXeTu+79ovq6Y2RmSnpW0\nTNKeXDxVe18H1FDnmgp6l96tR/QtfVuv6N36612GYQAAACSLl0kAAAAgWQzDAAAASBbDMAAAAJLF\nMAwAAIBkMQwDAAAgWQzDAAAASBbDMAAAAJL1f1iM7ql26a6bAAAAAElFTkSuQmCC\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "f, (ax1, ax2, ax3, ax4) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n", - "ax1.imshow(np.reshape(image[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax2.imshow(np.reshape(image[1,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax3.imshow(np.reshape(image[2,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax4.imshow(np.reshape(image[3,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "model_prefix = None\n", - "\n", - "batch_size = 100\n", - "nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},\n", - " batch_size = batch_size)\n", - "nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},\n", - " batch_size = batch_size)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 2. Building the Network Architecture\n", - "\n", - "### 2.1 Gaussian MLP as encoder\n", - "Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:\n", - "\n", - "\\begin{align}\n", - "\\log q_{\\phi}(z|x) &= \\log \\mathcal{N}(z:\\mu,\\sigma^2I) \\\\\n", - "\\textit{ where } \\mu &= W_2h+b_2, \\log \\sigma^2 = W_3h+b_3\\\\\n", - "h &= \\tanh(W_1x+b_1)\n", - "\\end{align}\n", - "\n", - "where $\\{W_1,W_2,W_3,b_1,b_2,b_3\\}$ are the weights and biases of the MLP.\n", - "Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.\n", - "\n", - "\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "## define data and loss labels as symbols \n", - "data = mx.sym.var('data')\n", - "loss_label = mx.sym.var('loss_label')\n", - "\n", - "## define fully connected and activation layers for the encoder, where we used tanh activation function.\n", - "encoder_h = mx.sym.FullyConnected(data=data, name=\"encoder_h\",num_hidden=400)\n", - "act_h = mx.sym.Activation(data=encoder_h, act_type=\"tanh\",name=\"activation_h\")\n", - "\n", - "## define mu and log variance which are the fully connected layers of the previous activation layer\n", - "mu = mx.sym.FullyConnected(data=act_h, name=\"mu\",num_hidden = 5)\n", - "logvar = mx.sym.FullyConnected(data=act_h, name=\"logvar\",num_hidden = 5)\n", - "\n", - "## sample the latent variables z according to Normal(mu,var)\n", - "z = mu + np.multiply(mx.symbol.exp(0.5 * logvar), \n", - " mx.symbol.random_normal(loc=0, scale=1, shape=np.shape(logvar.get_internals()[\"logvar_output\"])))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.2 Bernoulli MLP as decoder\n", - "\n", - "In this case let $p_\\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:\n", - "\n", - "\\begin{align}\n", - "\\log p(x|z) &= \\sum_{i=1}^D x_i\\log y_i + (1-x_i)\\log (1-y_i) \\\\\n", - "\\textit{ where } y &= f_\\sigma(W_5\\tanh (W_4z+b_4)+b_5)\n", - "\\end{align}\n", - "\n", - "where $f_\\sigma(\\dot)$ is the elementwise sigmoid activation function, $\\{W_4,W_5,b_4,b_5\\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# define fully connected and tanh activation layers for the decoder\n", - "decoder_z = mx.sym.FullyConnected(data=z, name=\"decoder_z\",num_hidden=400)\n", - "act_z = mx.sym.Activation(data=decoder_z, act_type=\"tanh\",name=\"activation_z\")\n", - "\n", - "# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension\n", - "decoder_x = mx.sym.FullyConnected(data=act_z, name=\"decoder_x\",num_hidden=features)\n", - "y = mx.sym.Activation(data=decoder_x, act_type=\"sigmoid\",name='activation_x')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### 2.3 Joint Loss Function for the Encoder and the Decoder\n", - "\n", - "The variational lower bound also called evidence lower bound (ELBO) can be estimated as:\n", - "\n", - "\\begin{align}\n", - "\\mathcal{L}(\\theta,\\phi;x_{(i)}) \\approx \\frac{1}{2}\\left(1+\\log ((\\sigma_j^{(i)})^2)-(\\mu_j^{(i)})^2-(\\sigma_j^{(i)})^2\\right) + \\log p_\\theta(x^{(i)}|z^{(i)})\n", - "\\end{align}\n", - "\n", - "where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\\mathcal{L}$(minus ELBO) for MXNet to minimize." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# define the objective loss function that needs to be minimized\n", - "KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)\n", - "loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) \n", - " + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL\n", - "output = mx.symbol.MakeLoss(sum(loss),name='loss')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 3. Training the model\n", - "\n", - "Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. \n", - "\n", - "To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# set up the log\n", - "nd_iter.reset()\n", - "logging.getLogger().setLevel(logging.DEBUG) \n", - "\n", - "# define function to trave back training loss\n", - "def log_to_list(period, lst):\n", - " def _callback(param):\n", - " \"\"\"The checkpoint function.\"\"\"\n", - " if param.nbatch % period == 0:\n", - " name, value = param.eval_metric.get()\n", - " lst.append(value)\n", - " return _callback\n", - "\n", - "# define the model\n", - "model = mx.mod.Module(\n", - " symbol = output ,\n", - " data_names=['data'],\n", - " label_names = ['loss_label'])" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:root:Epoch[0] Train-loss=375.023381\n", - "INFO:root:Epoch[0] Time cost=6.127\n", - "INFO:root:Epoch[1] Train-loss=212.780315\n", - "INFO:root:Epoch[1] Time cost=6.409\n", - "INFO:root:Epoch[2] Train-loss=208.209400\n", - "INFO:root:Epoch[2] Time cost=6.619\n", - "INFO:root:Epoch[3] Train-loss=206.146854\n", - "INFO:root:Epoch[3] Time cost=6.648\n", - "INFO:root:Epoch[4] Train-loss=204.530598\n", - "INFO:root:Epoch[4] Time cost=7.000\n", - "INFO:root:Epoch[5] Train-loss=202.799992\n", - "INFO:root:Epoch[5] Time cost=6.778\n", - "INFO:root:Epoch[6] Train-loss=200.333474\n", - "INFO:root:Epoch[6] Time cost=7.187\n", - "INFO:root:Epoch[7] Train-loss=197.506393\n", - "INFO:root:Epoch[7] Time cost=6.712\n", - "INFO:root:Epoch[8] Train-loss=195.969775\n", - "INFO:root:Epoch[8] Time cost=6.896\n", - "INFO:root:Epoch[9] Train-loss=195.418288\n", - "INFO:root:Epoch[9] Time cost=6.887\n", - "INFO:root:Epoch[10] Train-loss=194.739763\n", - "INFO:root:Epoch[10] Time cost=6.745\n", - "INFO:root:Epoch[11] Train-loss=194.380536\n", - "INFO:root:Epoch[11] Time cost=6.706\n", - "INFO:root:Epoch[12] Train-loss=193.955462\n", - "INFO:root:Epoch[12] Time cost=6.592\n", - "INFO:root:Epoch[13] Train-loss=193.493671\n", - "INFO:root:Epoch[13] Time cost=6.775\n", - "INFO:root:Epoch[14] Train-loss=192.958739\n", - "INFO:root:Epoch[14] Time cost=6.600\n", - "INFO:root:Epoch[15] Train-loss=191.928542\n", - "INFO:root:Epoch[15] Time cost=6.586\n", - "INFO:root:Epoch[16] Train-loss=189.797939\n", - "INFO:root:Epoch[16] Time cost=6.700\n", - "INFO:root:Epoch[17] Train-loss=186.672446\n", - "INFO:root:Epoch[17] Time cost=6.869\n", - "INFO:root:Epoch[18] Train-loss=184.616599\n", - "INFO:root:Epoch[18] Time cost=7.144\n", - "INFO:root:Epoch[19] Train-loss=183.305978\n", - "INFO:root:Epoch[19] Time cost=6.997\n", - "INFO:root:Epoch[20] Train-loss=181.944634\n", - "INFO:root:Epoch[20] Time cost=6.481\n", - "INFO:root:Epoch[21] Train-loss=181.005329\n", - "INFO:root:Epoch[21] Time cost=6.754\n", - "INFO:root:Epoch[22] Train-loss=178.363118\n", - "INFO:root:Epoch[22] Time cost=7.000\n", - "INFO:root:Epoch[23] Train-loss=176.363421\n", - "INFO:root:Epoch[23] Time cost=6.923\n", - "INFO:root:Epoch[24] Train-loss=174.573954\n", - "INFO:root:Epoch[24] Time cost=6.510\n", - "INFO:root:Epoch[25] Train-loss=173.245940\n", - "INFO:root:Epoch[25] Time cost=6.926\n", - "INFO:root:Epoch[26] Train-loss=172.082522\n", - "INFO:root:Epoch[26] Time cost=6.733\n", - "INFO:root:Epoch[27] Train-loss=171.123084\n", - "INFO:root:Epoch[27] Time cost=6.616\n", - "INFO:root:Epoch[28] Train-loss=170.239300\n", - "INFO:root:Epoch[28] Time cost=7.004\n", - "INFO:root:Epoch[29] Train-loss=169.538416\n", - "INFO:root:Epoch[29] Time cost=6.341\n", - "INFO:root:Epoch[30] Train-loss=168.952901\n", - "INFO:root:Epoch[30] Time cost=6.736\n", - "INFO:root:Epoch[31] Train-loss=168.169076\n", - "INFO:root:Epoch[31] Time cost=6.616\n", - "INFO:root:Epoch[32] Train-loss=167.208973\n", - "INFO:root:Epoch[32] Time cost=6.446\n", - "INFO:root:Epoch[33] Train-loss=165.732213\n", - "INFO:root:Epoch[33] Time cost=6.405\n", - "INFO:root:Epoch[34] Train-loss=163.606801\n", - "INFO:root:Epoch[34] Time cost=6.139\n", - "INFO:root:Epoch[35] Train-loss=161.985880\n", - "INFO:root:Epoch[35] Time cost=6.678\n", - "INFO:root:Epoch[36] Train-loss=160.763072\n", - "INFO:root:Epoch[36] Time cost=8.749\n", - "INFO:root:Epoch[37] Train-loss=160.025193\n", - "INFO:root:Epoch[37] Time cost=6.519\n", - "INFO:root:Epoch[38] Train-loss=159.319723\n", - "INFO:root:Epoch[38] Time cost=7.584\n", - "INFO:root:Epoch[39] Train-loss=158.670701\n", - "INFO:root:Epoch[39] Time cost=6.874\n", - "INFO:root:Epoch[40] Train-loss=158.225733\n", - "INFO:root:Epoch[40] Time cost=6.402\n", - "INFO:root:Epoch[41] Train-loss=157.741337\n", - "INFO:root:Epoch[41] Time cost=8.617\n", - "INFO:root:Epoch[42] Train-loss=157.301411\n", - "INFO:root:Epoch[42] Time cost=6.515\n", - "INFO:root:Epoch[43] Train-loss=156.765170\n", - "INFO:root:Epoch[43] Time cost=6.447\n", - "INFO:root:Epoch[44] Train-loss=156.389668\n", - "INFO:root:Epoch[44] Time cost=6.130\n", - "INFO:root:Epoch[45] Train-loss=155.815434\n", - "INFO:root:Epoch[45] Time cost=6.155\n", - "INFO:root:Epoch[46] Train-loss=155.432254\n", - "INFO:root:Epoch[46] Time cost=6.158\n", - "INFO:root:Epoch[47] Train-loss=155.114027\n", - "INFO:root:Epoch[47] Time cost=6.749\n", - "INFO:root:Epoch[48] Train-loss=154.612441\n", - "INFO:root:Epoch[48] Time cost=6.255\n", - "INFO:root:Epoch[49] Train-loss=154.137659\n", - "INFO:root:Epoch[49] Time cost=7.813\n", - "INFO:root:Epoch[50] Train-loss=153.634072\n", - "INFO:root:Epoch[50] Time cost=7.408\n", - "INFO:root:Epoch[51] Train-loss=153.417397\n", - "INFO:root:Epoch[51] Time cost=7.747\n", - "INFO:root:Epoch[52] Train-loss=152.851887\n", - "INFO:root:Epoch[52] Time cost=8.587\n", - "INFO:root:Epoch[53] Train-loss=152.575068\n", - "INFO:root:Epoch[53] Time cost=7.554\n", - "INFO:root:Epoch[54] Train-loss=152.084419\n", - "INFO:root:Epoch[54] Time cost=6.628\n", - "INFO:root:Epoch[55] Train-loss=151.724836\n", - "INFO:root:Epoch[55] Time cost=6.535\n", - "INFO:root:Epoch[56] Train-loss=151.302525\n", - "INFO:root:Epoch[56] Time cost=7.148\n", - "INFO:root:Epoch[57] Train-loss=150.960916\n", - "INFO:root:Epoch[57] Time cost=7.195\n", - "INFO:root:Epoch[58] Train-loss=150.603895\n", - "INFO:root:Epoch[58] Time cost=6.649\n", - "INFO:root:Epoch[59] Train-loss=150.237795\n", - "INFO:root:Epoch[59] Time cost=6.222\n", - "INFO:root:Epoch[60] Train-loss=149.936080\n", - "INFO:root:Epoch[60] Time cost=8.450\n", - "INFO:root:Epoch[61] Train-loss=149.514617\n", - "INFO:root:Epoch[61] Time cost=6.113\n", - "INFO:root:Epoch[62] Train-loss=149.229345\n", - "INFO:root:Epoch[62] Time cost=6.088\n", - "INFO:root:Epoch[63] Train-loss=148.893769\n", - "INFO:root:Epoch[63] Time cost=6.558\n", - "INFO:root:Epoch[64] Train-loss=148.526837\n", - "INFO:root:Epoch[64] Time cost=7.590\n", - "INFO:root:Epoch[65] Train-loss=148.249951\n", - "INFO:root:Epoch[65] Time cost=6.180\n", - "INFO:root:Epoch[66] Train-loss=147.940414\n", - "INFO:root:Epoch[66] Time cost=6.242\n", - "INFO:root:Epoch[67] Train-loss=147.621304\n", - "INFO:root:Epoch[67] Time cost=8.501\n", - "INFO:root:Epoch[68] Train-loss=147.294314\n", - "INFO:root:Epoch[68] Time cost=7.645\n", - "INFO:root:Epoch[69] Train-loss=147.074479\n", - "INFO:root:Epoch[69] Time cost=7.092\n", - "INFO:root:Epoch[70] Train-loss=146.796387\n", - "INFO:root:Epoch[70] Time cost=6.914\n", - "INFO:root:Epoch[71] Train-loss=146.508842\n", - "INFO:root:Epoch[71] Time cost=6.606\n", - "INFO:root:Epoch[72] Train-loss=146.230444\n", - "INFO:root:Epoch[72] Time cost=7.755\n", - "INFO:root:Epoch[73] Train-loss=145.970296\n", - "INFO:root:Epoch[73] Time cost=6.409\n", - "INFO:root:Epoch[74] Train-loss=145.711610\n", - "INFO:root:Epoch[74] Time cost=6.334\n", - "INFO:root:Epoch[75] Train-loss=145.460053\n", - "INFO:root:Epoch[75] Time cost=7.269\n", - "INFO:root:Epoch[76] Train-loss=145.156451\n", - "INFO:root:Epoch[76] Time cost=6.744\n", - "INFO:root:Epoch[77] Train-loss=144.957674\n", - "INFO:root:Epoch[77] Time cost=7.100\n", - "INFO:root:Epoch[78] Train-loss=144.729749\n", - "INFO:root:Epoch[78] Time cost=6.242\n", - "INFO:root:Epoch[79] Train-loss=144.481728\n", - "INFO:root:Epoch[79] Time cost=6.865\n", - "INFO:root:Epoch[80] Train-loss=144.236061\n", - "INFO:root:Epoch[80] Time cost=6.632\n", - "INFO:root:Epoch[81] Train-loss=144.030473\n", - "INFO:root:Epoch[81] Time cost=6.764\n", - "INFO:root:Epoch[82] Train-loss=143.776374\n", - "INFO:root:Epoch[82] Time cost=6.564\n", - "INFO:root:Epoch[83] Train-loss=143.538847\n", - "INFO:root:Epoch[83] Time cost=6.181\n", - "INFO:root:Epoch[84] Train-loss=143.326444\n", - "INFO:root:Epoch[84] Time cost=6.220\n", - "INFO:root:Epoch[85] Train-loss=143.078987\n", - "INFO:root:Epoch[85] Time cost=6.823\n", - "INFO:root:Epoch[86] Train-loss=142.877117\n", - "INFO:root:Epoch[86] Time cost=7.755\n", - "INFO:root:Epoch[87] Train-loss=142.667316\n", - "INFO:root:Epoch[87] Time cost=6.068\n", - "INFO:root:Epoch[88] Train-loss=142.461755\n", - "INFO:root:Epoch[88] Time cost=6.111\n", - "INFO:root:Epoch[89] Train-loss=142.270438\n", - "INFO:root:Epoch[89] Time cost=6.221\n", - "INFO:root:Epoch[90] Train-loss=142.047086\n", - "INFO:root:Epoch[90] Time cost=8.061\n", - "INFO:root:Epoch[91] Train-loss=141.855774\n", - "INFO:root:Epoch[91] Time cost=6.433\n", - "INFO:root:Epoch[92] Train-loss=141.688955\n", - "INFO:root:Epoch[92] Time cost=7.153\n", - "INFO:root:Epoch[93] Train-loss=141.442910\n", - "INFO:root:Epoch[93] Time cost=7.113\n", - "INFO:root:Epoch[94] Train-loss=141.279274\n", - "INFO:root:Epoch[94] Time cost=7.152\n", - "INFO:root:Epoch[95] Train-loss=141.086522\n", - "INFO:root:Epoch[95] Time cost=6.472\n", - "INFO:root:Epoch[96] Train-loss=140.901925\n", - "INFO:root:Epoch[96] Time cost=6.767\n", - "INFO:root:Epoch[97] Train-loss=140.722496\n", - "INFO:root:Epoch[97] Time cost=7.044\n", - "INFO:root:Epoch[98] Train-loss=140.579295\n", - "INFO:root:Epoch[98] Time cost=7.040\n", - "INFO:root:Epoch[99] Train-loss=140.386067\n", - "INFO:root:Epoch[99] Time cost=6.669\n" - ] - } - ], - "source": [ - "# training the model, save training loss as a list.\n", - "training_loss=list()\n", - "\n", - "# initilize the parameters for training using Normal.\n", - "init = mx.init.Normal(0.01)\n", - "model.fit(nd_iter, # train data\n", - " initializer=init,\n", - " # if eval_data is supplied, test loss will also be reported\n", - " # eval_data = nd_iter_test,\n", - " optimizer='sgd', # use SGD to train\n", - " optimizer_params={'learning_rate':1e-3,'wd':1e-2}, \n", - " # save parameters for each epoch if model_prefix is supplied\n", - " epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),\n", - " batch_end_callback = log_to_list(N/batch_size,training_loss), \n", - " num_epoch=100,\n", - " eval_metric = 'Loss')" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcZGV97/HPt6v3bfZ9BmaEEdlRRwSjUSO5EGIC90oM\nMYkYt/iSxJhrouISlStJNIkmvhKNxusVIooaF9C4gUaMMYADso6AA8MwM8zSs/Z0Ty+1/O4f53RT\n9PSpGmqmunq6v+/Xq15z6jmnqp7ndM3zq2c5z1FEYGZmVoumRmfAzMyOXw4iZmZWMwcRMzOrmYOI\nmZnVzEHEzMxq5iBiZmY1cxCxY0rSP0t677E+diaQ1CHpG5IOSPpyAz7/XZI+fbTHSnqJpK3HNneZ\n+XhM0gVT8VlWm+ZGZ8CmD0mPAa+PiFtqfY+IeFM9jp0hLgOWAAsiojDVHx4Rf1mPY58OSauBTUBL\nI86BHXtuidgRkzSrfnTUobwnAg/XUnnOtnNvxw8HEQNA0r8CJwDfkDQg6e2SVksKSa+T9Djwg/TY\nL0vakXbL/EjS6WXv81lJH0y3XyJpq6S3SdolabukP6jx2AVpV1C/pJ9K+qCkH1cozwsl/UTSfklb\nJL0mTf+hpNeXHfea8vdJy3ulpF8Av5D0CUl/O+G9b5T0v9Pt5ZK+IqlP0iZJb8nIzweAvwB+Oz2/\nr5PUJOk9kjanZb5O0pz0+EnP/YT3HDtnby87Z5dKuljSw5L2SnpX2fHvl/S5Ce9/haTHJe2W9O7J\njq1wjt+Vvu4xSb9blv7rkn6W/q22SHp/2ct+lP67Pz0P56eveYOkn0s6KGmDpOeUveYcSfem37cv\nSmov+6yXS7o7/Tv/RNJZZfveIWlb+p4PSXpZpfJYjSLCDz+ICIDHgAvKnq8GArgO6AI60vTXAj1A\nG/D3wN1lr/ks8MF0+yVAAbgaaAEuBg4B82o49ob00QmcBmwBfpxRjhOBg8DvpO+1ADgn3fdDki67\nsWNfU/4+aXlvBuYDHcAvp5+ldP88YAhYTvIj7E6S4NAKPAN4FLgwI1/vBz5X9vy1wMb0dd3AV4F/\nrXTuJ7zf2Dn7i7ScbwD6gM+nf5/T07yumfj5Ze//L2k5zwZGgFMny2vG534k/Q68GBgETinbf2Z6\nfs4CdgKXTvjc5rL3+y1gG/A8QMDJwIll38k70vM9H/g58KZ037OBXcDzgRxwRXp8G3BK+ndbXva5\nJzX6/9hMfLglYkfi/RExGBFDABHxmYg4GBEjJJXN2WO/oCeRB66OiHxEfAsYIPkPfsTHSsoBrwDe\nFxGHImIDcG2F/L4KuCUivpC+156IuPtplPevImJvWt7/JKn0XpTuuwz474h4gqTSWxQRV0fEaEQ8\nSlIpX36En/O7wEci4tGIGACuAi6f0HX1lHM/iTxwTUTkSYLsQuAf0r/PA8AGkgCR5QMRMRQR9wD3\nVDl2ovdGxEhE3Ar8O/BKgIj4YUTcFxGliLgX+AJJoMnyeuDDEfHTSGyMiM1l+z8WEU9ExF7gG8A5\nafobgU9GxO0RUYyIa0kC4XlAkSSYnCapJSIei4hHnkbZ7Ag5iNiR2DK2ISkn6a8lPSKpn+SXHySV\n12T2xFPHAA6R/Op+OscuIpkEsqVsX/n2RKuAo6kwxt87IoKkcv6dNOlVwPXp9onA8rQrZb+k/cC7\nSAbPj8RyoLyy3ExSzvLXVyonJOesmG6PBZqdZfuHyD7fADvKtiv9bSbaFxGDZc83k5QHSc+X9B9p\nF98B4E1kfz+g+t8rK48nAm+bcP5XkbQ+NgJvJfmRs0vSDZKWH2HZ7GlwELFyWUs6l6e/CrgEuACY\nQ9JNAEk3RL30kXSfrCxLW1Xh+C3ASRn7Bkm6xMYsneSYiefhC8Blkk4k6Tr5StnnbIqIuWWPnoi4\nuELeyj1BUhGOOYGknOVBYLousz1PUlfZ8xNIygNJd9pNwKqImAP8M09+PyYrT6W/VyVbSFph5ee/\nMyK+ABARn4+IF5Kc4wA+VMNnWBUOIlZuJ0n/fCU9JF0Ge0gq47pMBS2X/tL+KvB+SZ2SngW8usJL\nrgcukPRKSc3poPxYF8jdwP9K3+dk4HVH8Pk/A3YDnwa+GxH70113AAfTAdyOtJV2hqTnHWHRvgD8\nqaQ1krpJzuUX4/iZ+voBSa2SXgS8HBi79qUH2BsRw5LOJfnhMaYPKPHU79mngT+T9FwlTk4DdjX/\nArwpbflIUlc6qN8j6RRJvyKpDRgmaZGVjrK8NgkHESv3V8B70q6BP8s45jqSrottJP3tt01R3v6I\npOWzA/hXkgp4ZLIDI+JxkoH5twF7SQLHWF//R4FRkoB5LU92TVXzeZLW1+fLPqdIUnmeQ3Ltw1ig\nyRofmugzaVl+lL5+GPjjI3xto+0A9pG0Pq4nGex+MN33ZuBqSQdJBv2/NPaiiDgEXAP8V/o9Oy8i\nvpymfZ5kQsTXSQbRK4qI9SSTCf4xzctGkokSkIyH/DXJ32QHsJhkzMmOsbEZJ2bHFUkfApZGxBWN\nzovZbOaWiB0XJD1L0llpt8W5JN1QX2t0vsxmO18Fa8eLHpIurOUkXVF/B9zY0ByZmbuzzMysdu7O\nMjOzms347qyFCxfG6tWrG50NM7Pjyp133rk7IhZVO27GB5HVq1ezfv36RmfDzOy4Imlz9aPcnWVm\nZkfBQcTMzGrmIGJmZjVzEDEzs5o5iJiZWc0cRMzMrGYOImZmVrMZf52Imdl0FxEM5YsMjBTIF4NS\nKShNWJKqFFAsxfijFEEEyb/pewznS/QNjNB3cIQ9AyP8+YWnINXzfnEOImZmNRkaLbJp9yAb+wZ4\nYv8QAnJNoiXXRE97M3M7W+hsbWbf4Cg7+4fZdXCEwZECh0aLHMoX2TswOl7hHxzOUzrGyxi25pp4\n80tPprutvtW8g4iZHbcigpFCiUOjxfEKenC0wKGRIvliiXyxRKEU5IslRgslRoslhkaTX/wDwwX6\nh/PsP5TnwFCeUgQdrc10tuTIF0vs6B9mZ/8w/UMFJGhKf9EXI2kpFJ5mrZ9rEl2tOTpbm+lszTG/\nq5W1i7t5wUkLmNPRQldbM12tOVqbm5BETmKsEREBTU2Qa2qiuUk0pflpSo+RQIi25iYW9bSxqKeN\nOR0tdW+FgIOImTVARDA4WmTPwAgDIwVyTUmlOVossat/hO0Hkgp8d/pL/cBQnuacaM01kWsSuwdG\n2dU/TN/ACPlibT/hO1py9HY0M7ejlTkdLTQ3NXFgKM+OA0PkmppY2tvGWSvnMKejlSDpOooImtK8\ndrbmWLOwm5MWd7FyXiciCTCjhRIHhwscGMozOFJgbmcLS3rbmd/ZSlNT/Sv1qeYgYmZV5Ysltuw9\nxKHRIqVI+uQPjRY5MJSnfyhPrkn0tLfQ097MwEiB7fuH2H5geDwY7CrrsilFMDRaZKRQ/Zbn87ta\nWdjdytyOVobzJfqHChRKwcLuVk5atJBFPW30tDeP/8Lvamumsy1HZ0uOtpYczU0i1yRam5tozTXR\n2txER2uOrtZmcnWs0Bd2t9XtvacbBxGzWSZfTLp/BkYK7DgwzI4Dw/QdHGak8GSXz6HRIodGC/QP\nFXikb4BH+wYZLVav9Mu1NjextLedpb3tnL68l96OFnJKumLaWnIs6GplQXcb3W258UHj5iaxuLed\npXPaWdzTRkvOE0inOwcRs+NERFAKGBgpsHdwlL2DoxwYGqV/KOnb3z0wyo4DSQtg98Ao/Wkr4VC+\niEj60IOo2v0jQVfab9/d1syJCzp58SmLeObiHrrbm5NA0JQc09vRQm9HC6VScHC4wMHhPJ2tzSyb\n286CrtYp6ZO3xnIQMWuAQ6MFdvaPsKt/mEP5IqOFEiOFEvsGR9k9MMLugWRc4In9Q2zfP8zgaKHq\n7B0JFve0sbS3nRVz2zl1WQ9zOlrobM0l/fnpcZ0tOTrSALFkTjvL5rSzqLuNjtYcLblk4NaVvx0p\nBxGzGo0WSmw/METfwRGG8kWGRosM5Yv0DxfoH8ozMFJI+/6LDI4U6Ts4wq6Dw+zqH+HgSCHzfZsE\n87vaWDqnjdULunjBSQvpbmseH9Dtaktm9szvamVuZzIo3NvenAwOu/vHppiDiM1qhWKJx/ceon+4\nMH4B19iA8dig8Vg3zYGhZDro/qHRNCCMEBVaBy050d6So70lR2drjkXdbZyytIcXrV3Ekt52lvS2\nsbinnc62HG3NTbQ1NzG3s5V5na11HfQ1O5YcRGzGyhdLbN5ziI27DvKLnQM80jfAUL4IJPPut+4b\nYmPfAKNVZgm15ERve9L3P7ezhUXdbTxraS8r5nawYl4HS3rb6WpNgkVHa46e9mZ621tob8lNRTHN\nGspBxGaEgZEC927dzz1bDnDftv08vHOAzXsGnzKIvGJuBz3tyVc+ApbOaeeFaxeydnE3C3va0ou3\nkusH5nS0JN1EHS20pRd/mdnhHETsuFQqBRu29/PDh3bxw4f6uOvxfeMDzyfM7+SUpT386mlLOHlR\nN2uXdHPy4m46W/11NzvW/L/Kjhtb9x3iBw/u4icb93D7pj3sO5QH4MwVc3jzS05m3ep5nL1yLvO6\nWhucU7PZw0HEpp3hfJHNew6ND25v3DXAd+7fzj1bDwBJt9TLTl3C+c9YwIueuZDFPe0NzrHZ7OUg\nYg1RLAV3b9nPY7sHKaaL2W0/MMTtm/Zy95b9hw12n7VyDu+46FlcdMZS1izsalCuzWwiBxGrq4jg\nwFCeXQdHxpfYWL95Lz94cBe7B0afcmyuSZyxvJcrzj+RM1fOZX5nK70dzel0WLc2zKYjBxE7pvYf\nGuXf79vON+/ZzqO7B9gzMHrYktk97c289JTFXHDaEs5aMYeW5uQq6Z72Zg9+mx1n/D/Wjtqh0QI3\nb9jJN+7Zzq0P7yJfDE5a1MWLn7mIhd1tLOhO7m8wthjfsrntXljPbIZwELGnpVgKHukb4OGdB9m4\na4Cfb+/nRw/vZihfZElvG1ecv5pLn72C05f3+toKs1nAQcQqigju3XqA7z+4i7s27+PuLfsZSNd9\nkpJrMl7x3BX8xlnLed7q+TPypjtmls1BxIAkWPxi1wC3b9rL4EiB4XyRXQdH+MHPd7Gjf5gmwanL\nern02cs5Z9U8Tl3Ww0mLur20h9ks5yAyi/UdHOG+bfu5/dG9fG/DTjbtHnzK/u62Zl5w0gL+/PRT\neNmpi5nb6Yv4zOypHERmmYPDeT5566N85a6tbD8wDEBzkzj/pAW87oVrePEzF7Ggu5X25py7psys\nKgeRWWK0UOKGnz7OP9zyC/YMjnLBqUt43QvXcNbKuZy+vJeuNn8VzOzpc80xQ5VKwc6Dw6x/bB83\nb9jJfzy4i4MjBZ6/Zj6fufhUzl41t9FZNLMZwEHkOBAR4/fV3j3w5O1TB0cKjOST26oeHM6z91Ce\nfYOjbD8wxJZ9Q+NLhyzoauXXzlzKb5y9nBeevNBTb83smHEQaaBiKXhszyAbnujnkb4BettbWDan\nnYU9bTy88yA/3bSXux7fz44Dw4wWs2+c1KRkEHxBdxvzOltYu7iHC05dwsr5nZy6tIdnnzDPd8oz\ns7pwEJliEcFPH9vHl9dv4dv37xi/5mIyi3raWHfiPC4+cxnzu1qY19nKwu625NHTSk97csOk5ia5\ndWFmDeEgMkW27R/ia3dt5St3bWPT7kG625q5+MylnLtmAacu6+Hkxd0MjhR5Yv8QfQMjPGNhFyfM\n73RwMLNpzUGkjvYOjnLzhh18457t/Ncju4mAc1fP58qXnszFZy49bLHBtuYc831DJTM7jjQkiEj6\nG+A3gFHgEeAPImJ/uu8q4HVAEXhLRHw3TX8u8FmgA/gW8CcREYe/e2Pt6h/mext28p37d/Dfj+6h\nWApWze/gLb+yllc8ZyUnLOhsdBbNzI6ZRrVEbgauioiCpA8BVwHvkHQacDlwOrAcuEXSMyOiCHwC\neANwO0kQuQj4dkNyP8Hjew7x3Qd28N0HdnDn4/uIgDULu3jTi5/Br52xzIsRmtmM1ZAgEhHfK3t6\nG3BZun0JcENEjACbJG0EzpX0GNAbEbcBSLoOuJQGBpEtew/x9Z9t49/v286DOw4CcNqyXv70gmdy\n0RlLWbu424HDzGa86TAm8lrgi+n2CpKgMmZrmpZPtyemT0rSG4E3ApxwwgnHMq/8ZONuPnLzw6zf\nvA+A562ex3t+/VQuPH0pq+a7q8rMZpe6BRFJtwBLJ9n17oi4MT3m3UABuP5YfnZEfAr4FMC6deuO\n2bjJjgPD/OHn7mRORwt/fuEpXHLOclbOc+Aws9mrbkEkIi6otF/Sa4CXAy8rGyDfBqwqO2xlmrYt\n3Z6YPmUignd+9V7yxRKfe93zWb2wayo/3sxsWmrIPUolXQS8HfjNiDhUtusm4HJJbZLWAGuBOyJi\nO9Av6TwlAw2vBm6cyjz/251b+eFDfbzjomc5gJiZpRo1JvKPQBtwczr4fFtEvCkiHpD0JWADSTfX\nlenMLIA38+QU328zhYPqOw4Mc/U3N3Du6vlccf7qqfpYM7Npr1Gzs06usO8a4JpJ0tcDZ9QzX1k+\n9J0HyRdLfPiys3yPDTOzMg3pzjrebN4zyLoT57sby8xsAgeRI1AoBc05t0DMzCZyEDkC+WLQ3ORT\nZWY2kWvGI1AolmhxS8TM7DAOIkcg6c7yqTIzm8g14xHIF0u0eFaWmdlhHESOQKHogXUzs8k4iByB\nQqnk7iwzs0m4ZjwC+WK4O8vMbBIOIkegUHRLxMxsMq4Zj0DeFxuamU3KQeQIFIolWnyxoZnZYVwz\nVlEqBaXALREzs0k4iFSRL5UAaPGYiJnZYVwzVlEoJjddbPbsLDOzwziIVDEeRNwSMTM7jGvGKsa6\ns9wSMTM7nINIFU+2RBxEzMwmchCpIl9MB9Y9xdfM7DCuGasoltwSMTPL4iBSRWFsTMQD62Zmh3HN\nWEU+HRPxAoxmZodzEKnCU3zNzLK5ZqxifIqvx0TMzA7jIFJFYbw7y6fKzGwi14xVFIpuiZiZZXEQ\nqSKfTvFtcRAxMzuMg0gV4y0Rd2eZmR3GNWMVeS97YmaWyUGkioLvJ2Jmlsk1YxW+n4iZWTYHkSrG\nF2B0S8TM7DCuGasoeAFGM7NMDiJVeHaWmVk214xVjC/A6JaImdlhHESq8FLwZmbZXDNWkffsLDOz\nTA4iVXiKr5lZNgeRKsa6s3IOImZmh2lIEJH0fyTdK+luSd+TtLxs31WSNkp6SNKFZenPlXRfuu9j\nkqakVi+UgpacmKKPMzM7rjSqJfI3EXFWRJwDfBP4CwBJpwGXA6cDFwEfl5RLX/MJ4A3A2vRx0VRk\ntFAseXqvmVmGhtSOEdFf9rQLiHT7EuCGiBiJiE3ARuBcScuA3oi4LSICuA64dCrymi+GLzQ0M8vQ\n3KgPlnQN8GrgAPDSNHkFcFvZYVvTtHy6PTE9673fCLwR4IQTTjiqfBZKJS95YmaWoW61o6RbJN0/\nyeMSgIh4d0SsAq4H/uhYfnZEfCoi1kXEukWLFh3VexWK4ZlZZmYZ6tYSiYgLjvDQ64FvAe8DtgGr\nyvatTNO2pdsT0+suXwy3RMzMMjRqdtbasqeXAA+m2zcBl0tqk7SGZAD9jojYDvRLOi+dlfVq4Map\nyGuhVPKYiJlZhkaNify1pFOAErAZeBNARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdefuLDOzbA0J\nIhHxigr7rgGumSR9PXBGPfM1mXzRA+tmZllcO1ZRKHmKr5lZFgeRKvK+2NDMLFPV2lHSGZKuk7Q+\nfVwr6aypyNx0UCiG7yViZpahYhBJr+n4GvBD4LXp41bgK2PXe8x0hZJbImZmWaoNrF8N/GpEPFaW\ndq+kH5BMsZ2SabaNlC8G7S1uiZiZTabaT+zmCQEEgDStpR4Zmm687ImZWbZqtWNB0mGLT0k6keQ6\njhnP14mYmWWr1p31PuAWSX8J3JmmrQPeCbyjnhmbLvJFX7FuZpalYhCJiK9L2gS8DfjjNHkD8MqI\nuKfemZsOCqXwwLqZWYaqV6ynweLVU5CXaang+4mYmWWqNsV3oaT3SXqLpG5Jn0iXc79R0slTlclG\nKpRKtLglYmY2qWq14+eBNtLVdIFNwGUkt7T9dH2zNj24JWJmlq1ad9aSiHhXuvz65oj4cJr+oKQr\n65y3acELMJqZZatWOxYB0vua756wr1SXHE0zycC6WyJmZpOp1hJ5hqSbAJVtkz5fU9ecTRNJd5Zb\nImZmk6kWRMrXx/rbCfsmPp+R8qWSF2A0M8tQ7TqRW7P2SfoiyWKMM1axFETg60TMzDIcTe14/jHL\nxTSVLybDPp6dZWY2Of/ErqBQCgB3Z5mZZajYnSXpOVm7mAWr+BbGWiLuzjIzm1S1gfW/q7DvwWOZ\nkekoX3RLxMyskmoD6y+dqoxMR4XS2JiIWyJmZpOptnbW28u2f2vCvr+sV6ami0LaEvHFhmZmk6v2\nE/vysu2rJuy76BjnZdoZm53lZU/MzCZXrXZUxvZkz2ecsdlZnuJrZja5akEkMrYnez7j5D07y8ys\nomqzs86W1E/S6uhIt0mft9c1Z9NAwbOzzMwqqjY7KzdVGZmOxmZn5TywbmY2KffTVPDkdSI+TWZm\nk3HtWEGx5Cm+ZmaVOIhU8OQCjD5NZmaTce1YgQfWzcwqcxCpYHzZE0/xNTOblGvHCrwAo5lZZQ4i\nFXgBRjOzylw7VpD3AoxmZhU5iFRQ8HUiZmYVuXas4MnuLLdEzMwm4yBSwfjAumdnmZlNqqG1o6S3\nSQpJC8vSrpK0UdJDki4sS3+upPvSfR+TVPfmwfg91t0SMTObVMOCiKRVwP8AHi9LO43kRlink9z0\n6uOSxhaB/ATwBmBt+qj7TbF8PxEzs8oa2RL5KPB2nnpfkkuAGyJiJCI2ARuBcyUtA3oj4raICOA6\n4NJ6Z3D8zobuzjIzm1RDakdJlwDbIuKeCbtWAFvKnm9N01ak2xPTs97/jZLWS1rf19dXcz4LxaBJ\n0OQpvmZmk6p2U6qaSboFWDrJrncD7yLpyqqLiPgU8CmAdevW1XwHxnyp5AsNzcwqqFsQiYgLJkuX\ndCawBrgnHRtfCdwl6VxgG7Cq7PCVadq2dHtiel0VikGLWyFmZpmm/Gd2RNwXEYsjYnVErCbpmnpO\nROwAbgIul9QmaQ3JAPodEbEd6Jd0Xjor69XAjfXOa6HoloiZWSV1a4nUIiIekPQlYANQAK6MiGK6\n+83AZ4EO4Nvpo67ypfCSJ2ZmFTQ8iKStkfLn1wDXTHLceuCMKcoWMNYScRAxM8vivpoKCqXwvUTM\nzCpwDVlBoRi+l4iZWQUOIhUUPMXXzKwi15AV5IseWDczq8RBpIJCseR7iZiZVeAasoJCKTw7y8ys\nAgeRCvLFkhdfNDOrwDVkBYWiWyJmZpU4iFSQL4VnZ5mZVeAasoJCseQFGM3MKnAQqcDdWWZmlTmI\nVOD7iZiZVeYasgLfT8TMrDIHkQp8PxEzs8pcQ1aQL3kBRjOzShxEKigUS14K3sysAteQFXh2lplZ\nZQ4iFeRLXoDRzKwS15AVFIpBzrOzzMwyOYhkiAgKJU/xNTOrxEEkQ7EUAJ7ia2ZWgWvIDIXxIOKW\niJlZFgeRDPliCcD3EzEzq8A1ZIZC0S0RM7NqHEQy5EtJS8RjImZm2VxDZhhriXh2lplZNgeRDE92\nZ/kUmZllcQ2ZYaw7ywswmpllcxDJMN4S8ewsM7NMriEzjE3x9ewsM7NsDiIZxi42dHeWmVk2B5EM\nhbGWiLuzzMwyuYbMkPfFhmZmVTmIZCiMz87yKTIzy+IaMsOTs7PcEjEzy+IgkmF8AUa3RMzMMrmG\nzOCl4M3MqnMQyTB+nYi7s8zMMjmIZPAV62Zm1TWkhpT0fknbJN2dPi4u23eVpI2SHpJ0YVn6cyXd\nl+77mKS6NhGK7s4yM6uqkT+zPxoR56SPbwFIOg24HDgduAj4uKRcevwngDcAa9PHRfXMXN5TfM3M\nqppuNeQlwA0RMRIRm4CNwLmSlgG9EXFbRARwHXBpPTPiKb5mZtU1Moj8saR7JX1G0rw0bQWwpeyY\nrWnainR7YvqkJL1R0npJ6/v6+mrK3JMLME63OGtmNn3UrYaUdIuk+yd5XELSNfUM4BxgO/B3x/Kz\nI+JTEbEuItYtWrSopvfwAoxmZtU11+uNI+KCIzlO0r8A30yfbgNWle1emaZtS7cnpteNF2A0M6uu\nUbOzlpU9/Z/A/en2TcDlktokrSEZQL8jIrYD/ZLOS2dlvRq4sZ55HFuA0S0RM7NsdWuJVPFhSecA\nATwG/CFARDwg6UvABqAAXBkRxfQ1bwY+C3QA304fdVMolcg1iTrPJDYzO641JIhExO9X2HcNcM0k\n6euBM+qZr3KFYnhmlplZFe7wz5Avhq8RMTOrwrVkhkKp5KvVzcyqcBDJkC+GZ2aZmVXhWjJDoVjy\nzCwzsyocRDIUSuHuLDOzKhxEMuSLJVrcnWVmVpFryQyFolsiZmbVOIhkKJRKHlg3M6vCtWSGvFsi\nZmZVOYhkSFoiDiJmZpU4iGRIxkR8eszMKnEtmaFQCl8nYmZWhYNIhkLRA+tmZtW4lsyQLMDoloiZ\nWSUOIhk8xdfMrLpG3ZRq2nvR2kUsm9Pe6GyYmU1rDiIZ3vvy0xqdBTOzac/9NWZmVjMHETMzq5mD\niJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImJlZzRxEzMysZoqIRuehriT1AZtrfPlCYPcxzM7xYDaW\nGWZnuWdjmWF2lruWMp8YEYuqHTTjg8jRkLQ+ItY1Oh9TaTaWGWZnuWdjmWF2lrueZXZ3lpmZ1cxB\nxMzMauYgUtmnGp2BBpiNZYbZWe7ZWGaYneWuW5k9JmJmZjVzS8TMzGrmIGJmZjVzEJmEpIskPSRp\no6R3Njo/9SJplaT/kLRB0gOS/iRNny/pZkm/SP+d1+i8HmuScpJ+Jumb6fPZUOa5kv5N0oOSfi7p\n/Jlebkl/mn6375f0BUntM7HMkj4jaZek+8vSMssp6aq0fntI0oVH89kOIhNIygH/BPwacBrwO5Jm\n6m0OC8CgrOS/AAAEb0lEQVTbIuI04DzgyrSs7wS+HxFrge+nz2eaPwF+XvZ8NpT5H4DvRMSzgLNJ\nyj9jyy1pBfAWYF1EnAHkgMuZmWX+LHDRhLRJy5n+H78cOD19zcfTeq8mDiKHOxfYGBGPRsQocANw\nSYPzVBcRsT0i7kq3D5JUKitIyntteti1wKWNyWF9SFoJ/Drw6bLkmV7mOcAvA/8XICJGI2I/M7zc\nJLcA75DUDHQCTzADyxwRPwL2TkjOKuclwA0RMRIRm4CNJPVeTRxEDrcC2FL2fGuaNqNJWg08G7gd\nWBIR29NdO4AlDcpWvfw98HagVJY208u8BugD/l/ajfdpSV3M4HJHxDbgb4HHge3AgYj4HjO4zBNk\nlfOY1nEOIoakbuArwFsjor98XyRzwGfMPHBJLwd2RcSdWcfMtDKnmoHnAJ+IiGcDg0zoxplp5U7H\nAC4hCaDLgS5Jv1d+zEwrc5Z6ltNB5HDbgFVlz1emaTOSpBaSAHJ9RHw1Td4paVm6fxmwq1H5q4Nf\nAn5T0mMkXZW/IulzzOwyQ/Jrc2tE3J4+/zeSoDKTy30BsCki+iIiD3wVeAEzu8zlssp5TOs4B5HD\n/RRYK2mNpFaSAaibGpynupAkkj7yn0fER8p23QRckW5fAdw41Xmrl4i4KiJWRsRqkr/tDyLi95jB\nZQaIiB3AFkmnpEkvAzYws8v9OHCepM70u/4yknG/mVzmclnlvAm4XFKbpDXAWuCOWj/EV6xPQtLF\nJP3mOeAzEXFNg7NUF5JeCPwncB9Pjg+8i2Rc5EvACSTL6L8yIiYO2h33JL0E+LOIeLmkBczwMks6\nh2QyQSvwKPAHJD8kZ2y5JX0A+G2SmYg/A14PdDPDyizpC8BLSJZ83wm8D/g6GeWU9G7gtSTn5a0R\n8e2aP9tBxMzMauXuLDMzq5mDiJmZ1cxBxMzMauYgYmZmNXMQMTOzmjmImE1Tkl4ytsqw2XTlIGJm\nZjVzEDE7SpJ+T9Idku6W9Mn0XiUDkj6a3svi+5IWpceeI+k2SfdK+trYPR4knSzpFkn3SLpL0knp\n23eX3QPk+vTKa7Npw0HE7ChIOpXkiuhfiohzgCLwu0AXsD4iTgduJbmCGOA64B0RcRbJSgFj6dcD\n/xQRZ5Os7zS2+uqzgbeS3NvmGSRrf5lNG82NzoDZce5lwHOBn6aNhA6She5KwBfTYz4HfDW9p8fc\niLg1Tb8W+LKkHmBFRHwNICKGAdL3uyMitqbP7wZWAz+uf7HMjoyDiNnREXBtRFz1lETpvROOq3V9\noZGy7SL+P2vTjLuzzI7O94HLJC2G8ftan0jyf+uy9JhXAT+OiAPAPkkvStN/H7g1vavkVkmXpu/R\nJqlzSkthViP/qjE7ChGxQdJ7gO9JagLywJUkN306N923i2TcBJIluf85DRJjK+lCElA+Kenq9D1+\nawqLYVYzr+JrVgeSBiKiu9H5MKs3d2eZmVnN3BIxM7OauSViZmY1cxAxM7OaOYiYmVnNHETMzKxm\nDiJmZlaz/w/UuBmpwDVyRQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "ELBO = [-training_loss[i] for i in range(len(training_loss))]\n", - "plt.plot(ELBO)\n", - "plt.ylabel('ELBO');plt.xlabel('epoch');plt.title(\"training curve for mini batches\")\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set. " - ] - }, - { - "cell_type": "code", - "execution_count": 80, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "arg_params = model.get_params()[0]\n", - "\n", - "# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch\n", - "# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)\n", - "# assert sym.tojson() == output.tojson()\n", - "\n", - "e = y.bind(mx.cpu(), {'data': nd_iter_test.data[0][1],\n", - " 'encoder_h_weight': arg_params['encoder_h_weight'],\n", - " 'encoder_h_bias': arg_params['encoder_h_bias'],\n", - " 'mu_weight': arg_params['mu_weight'],\n", - " 'mu_bias': arg_params['mu_bias'],\n", - " 'logvar_weight':arg_params['logvar_weight'],\n", - " 'logvar_bias':arg_params['logvar_bias'],\n", - " 'decoder_z_weight':arg_params['decoder_z_weight'],\n", - " 'decoder_z_bias':arg_params['decoder_z_bias'],\n", - " 'decoder_x_weight':arg_params['decoder_x_weight'],\n", - " 'decoder_x_bias':arg_params['decoder_x_bias'], \n", - " 'loss_label':label})\n", - "\n", - "x_fit = e.forward()\n", - "x_construction = x_fit[0].asnumpy()" - ] - }, - { - "cell_type": "code", - "execution_count": 78, - "metadata": { - "collapsed": false, - "scrolled": true - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuUXGWZ7/HfkyvkSkKHJARICDAkxMkBaUm4CZrAwQku\nRAdnogzxNjhneRhdOjOiZ80ZdIYj5yxvzNHlDB4QFEcFuSsjIANyRxIMQhJCQi6E0Em6SUI693R4\nzx+1w5Sp56F7d1d1d9X+ftbKSvev3q7au/dT1W9X7+fdllISAAAAUEQD+noDAAAAgL7CZBgAAACF\nxWQYAAAAhcVkGAAAAIXFZBgAAACFxWQYAAAAhcVkuB8ws/9nZl/u6+0AesLMpphZMrNBwe1LzOzc\nXt4soFPULuoRdVs9TIYlmdn2sn9vmtmuss8/WuvHTyl9KqX0v2r9OKgfZrbGzOb29XZUU0ppRkrp\n4b7eDtQWtYt6RN0Wm/vbRNGklEYc+NjM1kj6VErp19F4MxuUUurojW0DaoU6Rr2idlGPqNv+i3eG\nu8DM/snMfmZmPzGzdkmXmtnNZnZV2Zi52UT6wOdHmdkdZtZqZqvN7DNvc/9v3deB+zGzL2Vf+5qZ\nvd/MLjSzFWa22cz+ruxrTzezp8xsq5m1mNk/m9ngstvfZ2YvmdkbZvZ/zexxM/tY2e2fMrMXzWyL\nmf27mR1dpW8baiSrhcXZMX/CzGaW3Xalmb1sZu1mttTMLi677WPZ8f+Wmb0u6aose8zMvp7VwGoz\ne1/Z14w2s+uz2lqfPRcGZrcNzL6uzcxWSZrXyXa/9c6LmV1lZrdmtd9uZs+b2R9ldb/JzNaZ2fll\nX/txM1uWjV1lZp8+6L7/LtvG17KaTmZ2fHbb0Gw7XzGzjWb2L2Z2aM+OArqD2qV26xF12/h1y2S4\n6y6W9G+SRkv62dsNNLMBkn4h6RlJkySdJ+lvzWxOFx/rKJWOzZGS/lHS9ZL+XNIpks6V9FUzOyYb\n2yHps5KaJJ0p6QJJn8624whJt0j62+z21ZJOK9vOD2W3XSRpnKSns31EP2Vmp0i6QaVjfLikf5V0\nt5kNzYa8LOlsler0K5JuNrOJZXcxS9IqSeMlXV2WLVepRv6PpOvNzLLbblSpxo5Xqf7Ol/Sp7La/\nlHRhljdL+tOcu/N+ST+SNEbS7yTdp1LdT5L01WzfDtiUPdYoSR+X9C0ze2f2PblA0uclzc2289yD\nHucaSX8k6eTs9kmS/mfObUUPUbvUbj2ibgtStykl/pX9k7RG0tyDsn+S9B8HZTdLuqrs87mS1mQf\nnylp1UHj/17S94PHfOu+svvZLmlg9vkYSUnSqWXjn5N0YXBffyPp1uzjT0h6tOw2k9Qi6WPZ5w9I\nWlB2+yBJeyRN6uvjUPR/Xh1m+fck/eNB2XJJ5wT3s1jSRdnHH5P0ykG3f0zSyrLPh2X1NkGlF+89\nkg4tu32+pIeyj/9D0l+V3XZ+9rWDOtsnSVdJeqDstvcfVPcjs/s6LLivOyV9Nvv4BklfK7vt+Oxr\nj89qfoek48puP13S6r4+xo36j9qlduvxH3Vb7LrlnOGuW5dj7GRJx5jZ1rJsoKSHu/j1bSml/dnH\nu7L/N5bdvkvSCEkys2mSviHpVJWeVINUeodXKr2z/NZ2p5SSmb160HZ+18yuLcveVOmd6fVd3Fb0\nrsmSFpjZFWXZEJWOtczsMpV+Y5+S3TZCpXcfDvDqeMOBD1JKO7M3KEZIGitpsKSW/3zTQgPK7uMP\n6kvS2pz7cnBNe3U/QtLW7M+I/6DSuw0DVKr158u2Y2HZfZVv07hs7KKyfTCVno/oXdQutVuPqNsC\n1C2T4a5LB32+Q6UDfsCEso/XSVqRUppe860q/VnjKUl/llLabmZ/o9KfNqTSu8Dl5wCZSn+uKN/O\nv08pve1pH+hX1km6OqV09cE3mNlkSd+XNEfSkyml/Wa2WKUXogMOruPOHmuPpKbkN320SCo/x/wY\nZ0yPZX+OvE3SZZLuSintM7M79Z/71aLSL3AHlG9Tm0ov8jNSSvyC17eoXWq3HlG3BahbzhnuvsWS\n5pnZmOz8oL8uu+1JSXvN7Atmdkh20vsfm9mpNdiOkZLekLTDzKYrO1848wtJ77RSA94glc4tHld2\n+79I+h/Z18nMDjOzvOcgoXYGZ/Vz4N8glV54/8rMZlnJcDObZ2YjJQ1X6YW3VSo1QEh6R3cfPKXU\nIul+Sd8ws1FmNsDMjjOzc7Iht0j6ays1i46RdGUP9vXtDJE0VKX96sjesTi/7PZbJH3czKab2TCV\nTkk6sA9vqvQ9+1Z2Dr3MbJKZ/dcabStKqN0Sare+ULclhatbJsPdd6OkZSr9meJXkn564IbsN7o/\nUalZbY1Kvyn9q0onolfbFyQtkNSePcZb7/KmlDZK+jNJ35T0uqTjVDppfk92+63Zbbea2TZJv5fU\nrwu2YO5V6TfsA/+uSiktVKmJ4juStkhaqdI5aEopLVXplJknVfpz2B9LeryH23CZSi+MS7PH+7mk\nA80h31epAeM5Sc9Kur2Hj+VKKbWr9MvmLdk2fETS3WW3/7ukf5b0kErfj6eym/Zk/3/xQJ7V+a8l\nnViLbcVbqF1Ru3WIulUx69ayk5tRAFZanuU1SX+aUnq0r7cHqIXsLx0vSBoa/KkR6JeoXdSjRqhb\n3hlucGZ2QXb6w1CV/pSxT9Jv+3izgKoys4uttLblGEn/W9I99fqijGKhdlGPGq1umQw3vrNUWuOw\nVaVTIC5OKe15+y8B6s6nVVoX82VJ+yX9t77dHKDLqF3Uo4aqW06TAAAAQGHxzjAAAAAKq0eT4ex8\n1OVmttLMarXEBwAAAFAT3T5NIluZ4CVJ50l6VdIzkuZnS424mpqa0pQpU7r1eMABixYtakspjet8\nZPVQu+ipNWvWqK2tzTofWT3ULaqB11zUq67Wbk+uQHeaStfXXiVJZvZTSReptDaea8qUKVq4cGF0\nM9AlZpb3EpQ9Ru2ip5qbm3v9MalbVAOvuahXXa3dnpwmMUl/eD3qV/WHl/oFAAAA+rWaN9CZ2eVm\nttDMFra2ttb64YCqoXZRj6hb1CtqF32lJ5Ph9ZKOLvv8qCz7Ayml61JKzSml5nHjevWUI6BHqF3U\nI+oW9YraRV/pyTnDz0g6wcyOVWkS/OcqXb8aAAAAVdAX14Mw69Ve3z7X7clwSqnDzP67pPskDZR0\nQ0ppSdW2DAAAAKixnrwzrJTSvZLurdK2AAAAAL2KK9ABAACgsJgMAwAAoLCYDAMAAKCwenTOMAAA\nQNFFKz54+ZtvvumO7ejocPPdu3e7+bZt29x88+bNbv7GG2+4+cCBAyuyCRMmuGPHjh3r5sOGDXPz\nwYMHu/mAAf57sX21igXvDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsFhNAgAAoEy0\nOkQkWiFi//79Fdm+ffvcsVu2bHHzVatWufmDDz7o5kuWLMl1/+PHj6/IZs6c6Y6dO3eum0+dOtXN\nq7HKRG+sMME7wwAAACgsJsMAAAAoLCbDAAAAKCwmwwAAACgsGugAAHUnb4NTXn11WVj0b3nrzhsf\nXXY5urzyokWL3Pzpp5928xUrVrh51MzmXXo5uuzykCFD3LyWou95NZ+jvDMMAACAwmIyDAAAgMJi\nMgwAAIDCYjIMAACAwmIyDAAAgMLq0WoSZrZGUruk/ZI6UkrN1dgoAMVV61UCelsRViWoRof92+Xe\nJW2jy99GvMu8vl0eHbe8216Nx8ybo+eq9b3Ncz/t7e1u/tprr7l5W1ubm0erRkybNs3NZ82aVZGd\neuqp7thx48a5ebTKRFTrefRGnVdjabX3pJT8IwIAAAD0Y5wmAQAAgMLq6WQ4Sfq1mS0ys8ursUEA\nAABAb+npZPislNLJkt4n6TNm9u6DB5jZ5Wa20MwWtra29vDhgN5D7aIeUbeoV9Qu+kqPJsMppfXZ\n/5sk3SHpNGfMdSml5pRSc3TiNdAfUbuoR9Qt6hW1i77S7QY6MxsuaUBKqT37+HxJX63algENIk+X\ned5O9aijPk+nfX/rVM/zuNXa9lrua72ujlGNuo3qsKOjw82jbvrNmzdXZNu3b3fH7tu3z82jYxx1\n3kf5oEH+j00vP/TQQ92xUR515Odd2YJVJmon77Hw7Nq1y81feuklN3/llVfcPKrRGTNmuPmcOXPc\nvLm5ciGwsWPHumOHDx/u5tHzIu/KKX2lJ6tJjJd0R7ZDgyT9W0rpV1XZKgAAAKAXdHsynFJaJem/\nVHFbAAAAgF7F0moAAAAoLCbDAAAAKCwmwwAAACisalyOud956qmn3Pzaa69180mTJrl51PG7YMGC\niizqvIxy1K9qrPiwd+9ed+y2bdvcPFpzc+vWrW4eddR79u/f7+YDBw5087wrB+S9f68refDgwe7Y\nqLN5woQJbj5mzBg3j+4/6oRuJHnqOe+x37Nnj5u3tLS4+SOPPOLmzz//fEX2+uuvu2Oj59bQoUPd\nfPz48W4eLe01atQoN/dq68QTT3THHnPMMW5e7x35qOStnLJ27Vp37GOPPebmmzZtcvPode7MM890\n89mzZ7t5U1NTRZb3NTHvyj79rXYb/5UeAAAACDAZBgAAQGExGQYAAEBhMRkGAABAYTEZBgAAQGE1\n5GoS3moPkrRixYqq3P/VV19dkY0ePdodG3Vv1oMpU6a4+Ze+9CU3jzqkG03UUe91DUt+R/2rr77q\njn344Yfd/Omnn3bzDRs2uPmWLVvcfOfOnRVZtBJA1DUcrQ7h3ffb3f+QIUPc3OvWj1YC8LqgJens\ns89280svvdTNo5UDvI7n/tYFXStenUfHMqr9aMWHX/ziF25+//33u/mqVasqssMOO8wdG71uRcc4\nqqFI9HPEq4vo+3X44Ye7+YgRI3JtC3pf3hVV2traKrI777zTHfvcc8+5ebSy1fTp0938tNNOc/Po\nORC9vnry7n80vr/hnWEAAAAUFpNhAAAAFBaTYQAAABQWk2EAAAAUFpNhAAAAFFZDriYRdWouXrzY\nzWfMmOHmS5YscXOvs/+uu+5yx953331ufuyxx7r56tWr3Twv7xr3EydOdMeuW7cu131H3dpf/OIX\nc91PvYq6Y6NVFnbs2FGRrVy50h3729/+1s2ja9ZHq0Z4K1hIfl14mRRfmz7v/kf3f8ghh7i5t0/R\nfkai1Q3mzp3r5lF3/8CBA3M9bn8WHbc83d55V1JpaWlx86VLl7r5+vXr3fzII4+syM455xx3bN5O\n+qhTv7W11c1/+ctfurn38yLq0m9ubnbzeum8L7Jo1YRt27a5+e23316RRfMC72eFJE2bNs3N58yZ\n4+be80WK6zFaOcgTvc5HeXTfUe6tyhI9L6q5sg/vDAMAAKCwmAwDAACgsJgMAwAAoLCYDAMAAKCw\nOp0Mm9kNZrbJzF4oy8aa2QNmtiL7f0xtNxMAAACovq6sJnGjpO9I+mFZdqWkB1NK15jZldnn/WYp\ngeh63VEemTlzppvPnz+/IrvmmmvcsWvWrHHzaDWJVatWdW3jOjFkyJCKLFpNItqWqJs66mxtdJ11\nekedrV7X7OjRo92xRxxxhJtHtbt9+3Y3j+7f6zKeMGGCOzZaYSFaZWLnzp25xkfdxIsWLarI7rnn\nHnfs5s2bc21L3q7kanQrV7Pjub+KOuzzHocTTzzRzWfPnl2RXXDBBe7YqG6jVSPyruAQrZzx4osv\nVmQjRoxwx0arBrCaRP8RHYvdu3e7+RNPPOHmt956a0W2YcMGd2w055g3b56bH3/88W4+bNgwN8+z\ngkP0nI7qf9euXW4erRoUvS56qw/1xmtop+8Mp5QekXTwT5yLJN2UfXyTpA9UebsAAACAmuvuOcPj\nU0oHFpDcIGl8lbYHAAAA6DU9bqBLpb8lhH/bMbPLzWyhmS2M/uwO9EfULuoRdYt6Re2ir3R3MrzR\nzCZKUvb/pmhgSum6lFJzSqk5ugIQ0B9Ru6hH1C3qFbWLvtLdyzHfLWmBpGuy//1rERdIdJJ43maz\nvE1+eXiXkZaktrY2N581a5abn3/++VXbpnpy4CT+PI1ykt/MMHXqVHfsBz7gn35/xhlnuHnU5BE1\nS3rNcsOHD3fHRo1vkaixIhI1/3n5vffe646NtnHSpEluPnbsWDfPcznSelWNJpTo+xRdtjo6Pk1N\nTW4e1e2ZZ55ZkUXNplGjXLSN+/btc/OoPqNLRntNUVu3bnXH5tUbl6Mtquh7G9XF6tWr3fynP/2p\nm3tN9FHtfuhDH3Jzr/4ladSoUW4ePU+jffUupRw1xG3a5L/3Gc0joktARz8DvX3ymuqkfA2BnenK\n0mo/kfSkpBPN7FUz+6RKk+DzzGyFpLnZ5wAAAEBd6fSd4ZRS5TpiJXOqvC0AAABAr2r8vw0CAAAA\nASbDAAAAKCwmwwAAACis7q4mgX7Ou9znxRdf7I6NLrv47W9/282jbu2iiDpVo25179LY0SVjR44c\n6ebRZTcj0THyuvur1ZEb1VHeS/W+8sorFVnU2Rx1U0fd19H3vZpdyY0sb+1H9Tx+vH+dpqjz3Luf\n6DEjXse8JG3bts3Nly5d6ubLly938z179lRk0eol0epD6H3RCgvt7e1u/qtf/crNf/e737m595p7\n7rnnumPnzPFbsaLXuUh0yei9e/e6uff6unbtWnfsM8884+be67YUrygTfQ9OP/30iizaf+9na3fx\nzjAAAAAKi8kwAAAACovJMAAAAAqLyTAAAAAKi8kwAAAACovVJBrUjTfeWJFt2LDBHRt12E+ePLma\nm9Tw8nTaR6sXRNdgz9t9HnXa13J1hOi+Ozo63HzlypVu/vjjj1dkXqe+5HceS9LZZ5/t5tFqBdHx\nKLI8tRJ9/6LVJI444gg3jzr4t2zZUpFFxzLqMI9qKKrDhQsXunlra6ube9+v0aNHu2Or1QUfrYTA\nKiiVou9VtMJCtDrEww8/7ObR6jhTp06tyObNm+eOHTNmjJtHK6Fs3brVzTdv3uzm0fOrpaWlIluy\nZIk79qWXXnLzaPWJaDWh6Pl45JFHVmTTpk1zx7KaBAAAAFAFTIYBAABQWEyGAQAAUFhMhgEAAFBY\nTIYBAABQWKwmUedefvllN//85z/f5ft48skn3XzChAnd2qZGV41O7aj7Pup4jlaHqEY3eXQfeUVd\nw95KAJJ02223ufnq1asrsqgr/4ILLnDzo446ys2j1TqKLKoVry6isXlXk2hqanJzr6tdkp577rmK\nbN26de7Y6BhHdbh8+XI3f+GFF9w86tT3RN+vaHWA6LkYPbei14Wi876P0fd806ZNbn7rrbe6eVQX\n0TGaMWNGRRa9Pu3bt8/No5qLanfjxo1uHm2j972Jvl/bt2/P9ZjRqhGjRo1yc29ViuOOO84dW83V\nVHhnGAAAAIXFZBgAAACFxWQYAAAAhcVkGAAAAIXV6WTYzG4ws01m9kJZdpWZrTezxdm/P6ntZgIA\nAADV15XW6hslfUfSDw/Kv5VS+nrVtwi53HPPPW7udaVecskl7ljv2umonjydrXm7YPOsBFCt+4jy\nvXv3uvkjjzzi5vfff7+bd3R0VGSzZs1yx77nPe9x82HDhrl5tOoBKnl1kXe1k+g4RCvVrFy50s1f\nffXVisxbdUSKj/HOnTvdfMOGDW6+detWN4868r189+7d7tjouRJ18EcrZFTjed6IvO9LtKrBQw89\n5OZPPPGEm0d1MXbsWDc/9thjK7KoRl9//XU3X7ZsmZu/9NJLbu69hkrxz/qjjz66IotWfIlWcdm1\na1euvL293c29ba/Gz7nOdPqTIaX0iKSuryUDAAAA1ImevE1yhZn9PjuNYkzVtggAAADoJd2dDH9P\n0lRJJ0tqkfSNaKCZXW5mC81sYWtrazcfDuh91C7qEXWLekXtoq90azKcUtqYUtqfUnpT0vclnfY2\nY69LKTWnlJrHjRvX3e0Eeh21i3pE3aJeUbvoK92aDJvZxLJPL5bkX58QAAAA6Mc6XU3CzH4i6VxJ\nTWb2qqR/kHSumZ0sKUlaI+nTNdxGKL5m+R133OHmQ4cOrci+9rWvuWO5vn119EW3dt5u2mp030Yd\n71Fn8/XXX+/mLS0tbn7kkUdWZB/84Afdscccc4ybDx482M3RM3lrfMiQIW7uHWNJOuWUU9zce43a\ntGlTrm2JVraIVmqIVh9YtWqVm3td8HlWnqi16Llfz6tMRPvkHYstW7a4Y5999lk3j07TiFYCiV5z\nvOfAtm3b3LFtbW1uHtVctFJD9Lo4Y8YMNz/ssMMqsqhGo1VZtm/f7ubRMRo1apSbT5w4sSKLnqPV\nrN1OJ8MppflO7P90AwAAAOoIi24CAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsDptoEP/EHXkP/ro\no27+kY98pCKLrksOeKJu4s2b/auzf/e733Xzp556ys2jlQYuvPDCiuy9732vO/bQQw918wED/N/z\n67lzvh5FK9WMHDnSzU844QQ3Hzt2bEUWrQ6Qd1ui7vj77rvPzZ955hk397rmo475qPajuqWe8/Fe\nu6IVHKKVmqJ6ifJo1QRv1ZxodYhoNYnXX3/dzaPXv2iN5mi8tyrFY4895o6NXs+j55H33JWkmTNn\nuvmUKVMqst5YHYh3hgEAAFBYTIYBAABQWEyGAQAAUFhMhgEAAFBYNND1M4sXL3bzK664ws29yyhK\n0le/+tWqbRPqT57GmqhRLrrsaNRYdPfdd7v57t273fz000938wULFlRkUUNIdJlO9K6o3qLGr6gh\nJmqs8xp/JkyYUJVtaW9vd/OVK1e6edQQ5DXFRdsYXRo6as6qRqNckZrtvH2NmhabmprcPHrNiV4X\nowY9rxEt76XEd+zY4ebRPkWv6dHjrl69uiL7+c9/7o7duHGjm0c1PXv2bDe/5JJL3Nw7Hr1xOWbe\nGQYAAEBhMRkGAABAYTEZBgAAQGExGQYAAEBhMRkGAABAYdGK3Ue8yx9K0vz58918//79bv7Rj37U\nzbn0cjFE3bTRpUE9UefxunXr3PyGG25w8+iSoVH3/WWXXebmxx13XEXWG93EqL68KztE3fHe8c9T\n4283vqOjw82jFS+iSyx7+dFHH+2OPeSQQ9y8WvVc9OeFtypHtGpEc3Ozmy9btszNo8vRR69/ixYt\nqsiWLl3qjo1WE4lqNO8KGVHtepc237p1qzt20qRJbh6tGnH55Ze7+YwZM9x86NChFVn0elFNvDMM\nAACAwmIyDAAAgMJiMgwAAIDCYjIMAACAwmIyDAAAgMLqdDUJMzta0g8ljZeUJF2XUrrWzMZK+pmk\nKZLWSPpwSqmyJbHgok79efPmufny5cvdfPr06W7+la98pXsbhkLyOup37tzpjr3lllvcfPHixW4e\ndfyed955uXKvQ7ro3fGNJu/x9Gor72oS0Wtx1Km/Y8cONx82bJibH3744RXZmDFjurh16I6ojrxV\nGYYPH+6OnTVrlpuvX7/ezbdv3+7mzz77rJt7KzVEr7l5azpaTSK6H2+lBkk64YQTKrJzzjnHHTtz\n5kw3j8ZPnjw517Z4z/XeeP3vyjvDHZK+kFI6SdJsSZ8xs5MkXSnpwZTSCZIezD4HAAAA6kank+GU\nUktK6dns43ZJyyRNknSRpJuyYTdJ+kCtNhIAAACohVznDJvZFEmnSHpa0viUUkt20waVTqPwvuZy\nM1toZgtbW1t7sKlA76J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW0rvy2VTlBxT1JJKV2XUmpOKTWP\nGzeuRxsL9CZqF/WIukW9onbRV7o0GTazwSpNhH+cUro9izea2cTs9omSNtVmEwEAAIDa6MpqEibp\neknLUkrfLLvpbkkLJF2T/X9XTbawzkXXMX/44Ydz3c+PfvQjNx87dmzeTUIBRN3E+/btq8hefPFF\nd+xvfvMbN4+68qdMmeLmn/zkJ9086rSv5XXoo+8Lq1X0b3mOT3SMo3zPnj1uHq0yEdXtoYceWpHt\n3r3bHes9D6X8qwnA59XLoEH+dKepqcnNL730Ujd/5zvf6eYPPfSQm7/wwgsV2Zo1a9yx0XwhqosJ\nEya4+SmnnOLmZ5xxhptPmzatIhs9erQ7dtSoUW7u1b8Uf9+j53RfvRZ3OhmWdKakv5D0vJkdWFPp\nyypNgm8xs09KWivpw7XZRAAAAKA2Op0Mp5QekxRN1edUd3MAAACA3sMV6AAAAFBYTIYBAABQWEyG\nAQAAUFhdaaBDF7zxxhtuPnv27Fz3c/PNN7t51B2KYou6jPfv3+/mW7Zsqcgef/xxd2zU2Tx+vHt9\nHc2fP9/No+7rwYMHu7nXTRx1GOftvq/G/bDyRO3kOT55j0Pe50q0asrOnTvdfO/evRVZW1ubOzZa\nwSLvShjUYtdF36uBAwe6+YgRI9z8Xe96l5tHP6O9lUOi1USiPKrRaKWGaGWHaF+9FXyiVX2i72Pe\n8f0N7wwDAACgsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaJKfvCDH7j5qlWrct3PWWed\n5eb10pGJ2oi6yaOO96hb/bXXXqvI1q9f745tampy85kzZ7r5vHnz3DzqbM5T03lXjch7Pzy/+jfv\n+ORdeSHqyG9vb3fz1tZWN9+2bZubjxw5ssuPGeXVqnP0XPSaEOVDhgzJlXvq+fjX+2so7wwDAACg\nsJgMAwAAoLCYDAMAAKCwmAwDAACgsJgMAwAAoLBYTaIbVqxYUZFdddVVvb8hqAt5OoTzrhrR0dHh\n5lHH+8qVKyuyXbt2uWOPOOIIN49Wk5gwYYKbR13GfbGyQ713PKNzUV1Fz5V9+/a5+dChQ9382GOP\ndfMxY8ZUZFOnTnXHjhgxws0HDPDfn6Jui4Hj3Hd4ZxgAAACFxWQYAAAAhcVkGAAAAIXFZBgAAACF\n1WkDnZkdLemHksZLSpKuSylda2ZXSfpLSQeuWfnllNK9tdrQ/uTRRx+tyKKGpcj06dPdPLp0LepX\nNS4lGzX/7Ny5082jSyyvXbu2Itu9e7c7dtiwYW4eNcoNGtT7/bi1bjihoaVxRPUZ1XN0efF3v/vd\nbu41xU2u/Qu7AAAEuElEQVSaNMkdO3r0aDePtjHvpYEB5NOVn14dkr6QUnrWzEZKWmRmD2S3fSul\n9PXabR4AAABQO51OhlNKLZJaso/bzWyZJP/XXQAAAKCO5Dpn2MymSDpF0tNZdIWZ/d7MbjCzykUW\nS19zuZktNLOFra2t3hCgX6J2UY+oW9Qrahd9pcuTYTMbIek2SZ9LKW2T9D1JUyWdrNI7x9/wvi6l\ndF1KqTml1Dxu3LgqbDLQO6hd1CPqFvWK2kVf6dJk2MwGqzQR/nFK6XZJSiltTCntTym9Ken7kk6r\n3WYCAAAA1deV1SRM0vWSlqWUvlmWT8zOJ5akiyW9UJtNrG9nnHGGmz/wwANuzmoSxRZ1h0eXaR08\neLCbH3744W5+0kknVWTRpWG9y8tK0jve8Q43HzlypJvn7Xj3VtSo9SWd6cpvHNFzZciQIW4e1Xm0\n4kNUcwMHDuxS9nY5q0YAfaMrq0mcKekvJD1vZouz7MuS5pvZySott7ZG0qdrsoUAAABAjXRlNYnH\nJHm/lhZiTWEAAAA0Lq5ABwAAgMJiMgwAAIDCYjIMAACAwupKAx0O8olPfKJLGRDJ2zUerRoR5cOH\nD3fzyZMnV2RRd3yU5xV19+fpkKfLHgfLWxNRHq3sUK3tAdD/8c4wAAAACovJMAAAAAqLyTAAAAAK\ni8kwAAAACovJMAAAAArLqtUx3qUHM2uVtLbXHrD2miS19fVG9JL+tK+TU0rjevMBqd261Z/2k7qt\njv50TGupP+0ntdtz/el41lp/2tcu1W6vToYbjZktTCk19/V29IYi7WsRFOV4FmU/i6Qox7Qo+1kU\nRTqe9bivnCYBAACAwmIyDAAAgMJiMtwz1/X1BvSiIu1rERTleBZlP4ukKMe0KPtZFEU6nnW3r5wz\nDAAAgMLinWEAAAAUFpPhLjKzG8xsk5m9UJaNNbMHzGxF9v+YvtzGajCzo83sITNbamZLzOyzWd5w\n+1oU1G7j7WsRULeNt69FQe3W374yGe66GyVdcFB2paQHU0onSHow+7zedUj6QkrpJEmzJX3GzE5S\nY+5rUdwoarfR9rUIbhR122j7WhQ3itqtq31lMtxFKaVHJG0+KL5I0k3ZxzdJ+kCvblQNpJRaUkrP\nZh+3S1omaZIacF+LgtptvH0tAuq28fa1KKjd+ttXJsM9Mz6l1JJ9vEHS+L7cmGozsymSTpH0tBp8\nXwuooY8ntduwGvpYUrcNraGPZ73XLpPhKkmlZTkaZmkOMxsh6TZJn0spbSu/rdH2tega7XhSu8XQ\naMeSui2ORjuejVC7TIZ7ZqOZTZSk7P9Nfbw9VWFmg1Uq7B+nlG7P4obc1wJryONJ7Ta8hjyW1G0h\nNOTxbJTaZTLcM3dLWpB9vEDSXX24LVVhZibpeknLUkrfLLup4fa14BrueFK7hdBwx5K6LYyGO56N\nVLtcdKOLzOwnks6V1CRpo6R/kHSnpFskHSNpraQPp5QOPmm+rpjZWZIelfS8pDez+MsqnQfUUPta\nFNQutVuPqFvqtl5Ru/VXu0yGAQAAUFicJgEAAIDCYjIMAACAwmIyDAAAgMJiMgwAAIDCYjIMAACA\nwmIyDAAAgMJiMgwAAIDCYjIMAACAwvr/nULp1p8Yt+AAAAAASUVORK5CYII=\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "# learning images on the test set\n", - "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n", - "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax1.set_title('True image')\n", - "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax2.set_title('Learned image')\n", - "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax3.set_title('Learned image')\n", - "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax4.set_title('Learned image')\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 37, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[('loss', 139.73684648437501)]" - ] - }, - "execution_count": 37, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# calculate the ELBO which is minus the loss for test set\n", - "metric = mx.metric.Loss()\n", - "model.score(nd_iter_test, metric)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## 4. All together: MXNet-based class VAE" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "from VAE import VAE" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One can directly call the class `VAE` to do the training:\n", - "\n", - "```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,\n", - "batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,\n", - "initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```\n", - "\n", - "The outputs are the learned model and training loss." - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:root:Epoch[0] Train-loss=377.146422\n", - "INFO:root:Epoch[0] Time cost=5.989\n", - "INFO:root:Epoch[1] Train-loss=211.998043\n", - "INFO:root:Epoch[1] Time cost=6.303\n", - "INFO:root:Epoch[2] Train-loss=207.103096\n", - "INFO:root:Epoch[2] Time cost=7.368\n", - "INFO:root:Epoch[3] Train-loss=204.958183\n", - "INFO:root:Epoch[3] Time cost=7.530\n", - "INFO:root:Epoch[4] Train-loss=203.342700\n", - "INFO:root:Epoch[4] Time cost=8.887\n", - "INFO:root:Epoch[5] Train-loss=201.649251\n", - "INFO:root:Epoch[5] Time cost=9.147\n", - "INFO:root:Epoch[6] Train-loss=199.782661\n", - "INFO:root:Epoch[6] Time cost=8.924\n", - "INFO:root:Epoch[7] Train-loss=198.044015\n", - "INFO:root:Epoch[7] Time cost=8.920\n", - "INFO:root:Epoch[8] Train-loss=195.732077\n", - "INFO:root:Epoch[8] Time cost=8.857\n", - "INFO:root:Epoch[9] Train-loss=194.070547\n", - "INFO:root:Epoch[9] Time cost=9.216\n", - "INFO:root:Epoch[10] Train-loss=193.186871\n", - "INFO:root:Epoch[10] Time cost=8.966\n", - "INFO:root:Epoch[11] Train-loss=192.700208\n", - "INFO:root:Epoch[11] Time cost=8.843\n", - "INFO:root:Epoch[12] Train-loss=192.191504\n", - "INFO:root:Epoch[12] Time cost=8.152\n", - "INFO:root:Epoch[13] Train-loss=191.842837\n", - "INFO:root:Epoch[13] Time cost=6.180\n", - "INFO:root:Epoch[14] Train-loss=191.310450\n", - "INFO:root:Epoch[14] Time cost=6.067\n", - "INFO:root:Epoch[15] Train-loss=190.520681\n", - "INFO:root:Epoch[15] Time cost=6.058\n", - "INFO:root:Epoch[16] Train-loss=189.784146\n", - "INFO:root:Epoch[16] Time cost=6.046\n", - "INFO:root:Epoch[17] Train-loss=188.515020\n", - "INFO:root:Epoch[17] Time cost=6.062\n", - "INFO:root:Epoch[18] Train-loss=187.530712\n", - "INFO:root:Epoch[18] Time cost=6.088\n", - "INFO:root:Epoch[19] Train-loss=186.194826\n", - "INFO:root:Epoch[19] Time cost=6.491\n", - "INFO:root:Epoch[20] Train-loss=185.492288\n", - "INFO:root:Epoch[20] Time cost=6.182\n", - "INFO:root:Epoch[21] Train-loss=184.922654\n", - "INFO:root:Epoch[21] Time cost=6.058\n", - "INFO:root:Epoch[22] Train-loss=184.677911\n", - "INFO:root:Epoch[22] Time cost=6.042\n", - "INFO:root:Epoch[23] Train-loss=183.921396\n", - "INFO:root:Epoch[23] Time cost=5.994\n", - "INFO:root:Epoch[24] Train-loss=183.600690\n", - "INFO:root:Epoch[24] Time cost=6.038\n", - "INFO:root:Epoch[25] Train-loss=183.388476\n", - "INFO:root:Epoch[25] Time cost=6.025\n", - "INFO:root:Epoch[26] Train-loss=182.972208\n", - "INFO:root:Epoch[26] Time cost=6.014\n", - "INFO:root:Epoch[27] Train-loss=182.561678\n", - "INFO:root:Epoch[27] Time cost=6.064\n", - "INFO:root:Epoch[28] Train-loss=182.475261\n", - "INFO:root:Epoch[28] Time cost=5.983\n", - "INFO:root:Epoch[29] Train-loss=182.308808\n", - "INFO:root:Epoch[29] Time cost=6.371\n", - "INFO:root:Epoch[30] Train-loss=182.135900\n", - "INFO:root:Epoch[30] Time cost=6.038\n", - "INFO:root:Epoch[31] Train-loss=181.978367\n", - "INFO:root:Epoch[31] Time cost=6.924\n", - "INFO:root:Epoch[32] Train-loss=181.677153\n", - "INFO:root:Epoch[32] Time cost=8.205\n", - "INFO:root:Epoch[33] Train-loss=181.677775\n", - "INFO:root:Epoch[33] Time cost=6.017\n", - "INFO:root:Epoch[34] Train-loss=181.257998\n", - "INFO:root:Epoch[34] Time cost=6.056\n", - "INFO:root:Epoch[35] Train-loss=181.125288\n", - "INFO:root:Epoch[35] Time cost=6.020\n", - "INFO:root:Epoch[36] Train-loss=181.018858\n", - "INFO:root:Epoch[36] Time cost=6.035\n", - "INFO:root:Epoch[37] Train-loss=180.785110\n", - "INFO:root:Epoch[37] Time cost=6.049\n", - "INFO:root:Epoch[38] Train-loss=180.452598\n", - "INFO:root:Epoch[38] Time cost=6.083\n", - "INFO:root:Epoch[39] Train-loss=180.362733\n", - "INFO:root:Epoch[39] Time cost=6.198\n", - "INFO:root:Epoch[40] Train-loss=180.060788\n", - "INFO:root:Epoch[40] Time cost=6.049\n", - "INFO:root:Epoch[41] Train-loss=180.022728\n", - "INFO:root:Epoch[41] Time cost=6.135\n", - "INFO:root:Epoch[42] Train-loss=179.648499\n", - "INFO:root:Epoch[42] Time cost=6.055\n", - "INFO:root:Epoch[43] Train-loss=179.507952\n", - "INFO:root:Epoch[43] Time cost=6.108\n", - "INFO:root:Epoch[44] Train-loss=179.303132\n", - "INFO:root:Epoch[44] Time cost=6.020\n", - "INFO:root:Epoch[45] Train-loss=178.945211\n", - "INFO:root:Epoch[45] Time cost=6.004\n", - "INFO:root:Epoch[46] Train-loss=178.808598\n", - "INFO:root:Epoch[46] Time cost=6.016\n", - "INFO:root:Epoch[47] Train-loss=178.550906\n", - "INFO:root:Epoch[47] Time cost=6.050\n", - "INFO:root:Epoch[48] Train-loss=178.403674\n", - "INFO:root:Epoch[48] Time cost=6.115\n", - "INFO:root:Epoch[49] Train-loss=178.237544\n", - "INFO:root:Epoch[49] Time cost=6.004\n", - "INFO:root:Epoch[50] Train-loss=178.033747\n", - "INFO:root:Epoch[50] Time cost=6.051\n", - "INFO:root:Epoch[51] Train-loss=177.802884\n", - "INFO:root:Epoch[51] Time cost=6.028\n", - "INFO:root:Epoch[52] Train-loss=177.533980\n", - "INFO:root:Epoch[52] Time cost=6.052\n", - "INFO:root:Epoch[53] Train-loss=177.490143\n", - "INFO:root:Epoch[53] Time cost=6.019\n", - "INFO:root:Epoch[54] Train-loss=177.136637\n", - "INFO:root:Epoch[54] Time cost=6.014\n", - "INFO:root:Epoch[55] Train-loss=177.062524\n", - "INFO:root:Epoch[55] Time cost=6.024\n", - "INFO:root:Epoch[56] Train-loss=176.869033\n", - "INFO:root:Epoch[56] Time cost=6.065\n", - "INFO:root:Epoch[57] Train-loss=176.704606\n", - "INFO:root:Epoch[57] Time cost=6.037\n", - "INFO:root:Epoch[58] Train-loss=176.470091\n", - "INFO:root:Epoch[58] Time cost=6.012\n", - "INFO:root:Epoch[59] Train-loss=176.261440\n", - "INFO:root:Epoch[59] Time cost=6.215\n", - "INFO:root:Epoch[60] Train-loss=176.133904\n", - "INFO:root:Epoch[60] Time cost=6.042\n", - "INFO:root:Epoch[61] Train-loss=175.941920\n", - "INFO:root:Epoch[61] Time cost=6.000\n", - "INFO:root:Epoch[62] Train-loss=175.731296\n", - "INFO:root:Epoch[62] Time cost=6.025\n", - "INFO:root:Epoch[63] Train-loss=175.613303\n", - "INFO:root:Epoch[63] Time cost=6.002\n", - "INFO:root:Epoch[64] Train-loss=175.438844\n", - "INFO:root:Epoch[64] Time cost=5.982\n", - "INFO:root:Epoch[65] Train-loss=175.254716\n", - "INFO:root:Epoch[65] Time cost=6.016\n", - "INFO:root:Epoch[66] Train-loss=175.090210\n", - "INFO:root:Epoch[66] Time cost=6.008\n", - "INFO:root:Epoch[67] Train-loss=174.895443\n", - "INFO:root:Epoch[67] Time cost=6.008\n", - "INFO:root:Epoch[68] Train-loss=174.701321\n", - "INFO:root:Epoch[68] Time cost=6.418\n", - "INFO:root:Epoch[69] Train-loss=174.553292\n", - "INFO:root:Epoch[69] Time cost=6.072\n", - "INFO:root:Epoch[70] Train-loss=174.349379\n", - "INFO:root:Epoch[70] Time cost=6.048\n", - "INFO:root:Epoch[71] Train-loss=174.174641\n", - "INFO:root:Epoch[71] Time cost=6.036\n", - "INFO:root:Epoch[72] Train-loss=173.966333\n", - "INFO:root:Epoch[72] Time cost=6.017\n", - "INFO:root:Epoch[73] Train-loss=173.798454\n", - "INFO:root:Epoch[73] Time cost=6.018\n", - "INFO:root:Epoch[74] Train-loss=173.635657\n", - "INFO:root:Epoch[74] Time cost=5.985\n", - "INFO:root:Epoch[75] Train-loss=173.423795\n", - "INFO:root:Epoch[75] Time cost=6.016\n", - "INFO:root:Epoch[76] Train-loss=173.273981\n", - "INFO:root:Epoch[76] Time cost=6.018\n", - "INFO:root:Epoch[77] Train-loss=173.073401\n", - "INFO:root:Epoch[77] Time cost=5.996\n", - "INFO:root:Epoch[78] Train-loss=172.888044\n", - "INFO:root:Epoch[78] Time cost=6.035\n", - "INFO:root:Epoch[79] Train-loss=172.694943\n", - "INFO:root:Epoch[79] Time cost=8.492\n", - "INFO:root:Epoch[80] Train-loss=172.504260\n", - "INFO:root:Epoch[80] Time cost=7.380\n", - "INFO:root:Epoch[81] Train-loss=172.323245\n", - "INFO:root:Epoch[81] Time cost=6.063\n", - "INFO:root:Epoch[82] Train-loss=172.131274\n", - "INFO:root:Epoch[82] Time cost=6.209\n", - "INFO:root:Epoch[83] Train-loss=171.932986\n", - "INFO:root:Epoch[83] Time cost=6.060\n", - "INFO:root:Epoch[84] Train-loss=171.755262\n", - "INFO:root:Epoch[84] Time cost=6.068\n", - "INFO:root:Epoch[85] Train-loss=171.556803\n", - "INFO:root:Epoch[85] Time cost=6.004\n", - "INFO:root:Epoch[86] Train-loss=171.384773\n", - "INFO:root:Epoch[86] Time cost=6.059\n", - "INFO:root:Epoch[87] Train-loss=171.185034\n", - "INFO:root:Epoch[87] Time cost=6.001\n", - "INFO:root:Epoch[88] Train-loss=170.995980\n", - "INFO:root:Epoch[88] Time cost=6.143\n", - "INFO:root:Epoch[89] Train-loss=170.818701\n", - "INFO:root:Epoch[89] Time cost=6.690\n", - "INFO:root:Epoch[90] Train-loss=170.629929\n", - "INFO:root:Epoch[90] Time cost=6.869\n", - "INFO:root:Epoch[91] Train-loss=170.450824\n", - "INFO:root:Epoch[91] Time cost=7.156\n", - "INFO:root:Epoch[92] Train-loss=170.261806\n", - "INFO:root:Epoch[92] Time cost=6.972\n", - "INFO:root:Epoch[93] Train-loss=170.070318\n", - "INFO:root:Epoch[93] Time cost=6.595\n", - "INFO:root:Epoch[94] Train-loss=169.906993\n", - "INFO:root:Epoch[94] Time cost=6.561\n", - "INFO:root:Epoch[95] Train-loss=169.734455\n", - "INFO:root:Epoch[95] Time cost=6.744\n", - "INFO:root:Epoch[96] Train-loss=169.564318\n", - "INFO:root:Epoch[96] Time cost=6.601\n", - "INFO:root:Epoch[97] Train-loss=169.373926\n", - "INFO:root:Epoch[97] Time cost=6.725\n", - "INFO:root:Epoch[98] Train-loss=169.215408\n", - "INFO:root:Epoch[98] Time cost=6.391\n", - "INFO:root:Epoch[99] Train-loss=169.039854\n", - "INFO:root:Epoch[99] Time cost=6.677\n", - "INFO:root:Epoch[100] Train-loss=168.869222\n", - "INFO:root:Epoch[100] Time cost=6.370\n", - "INFO:root:Epoch[101] Train-loss=168.703175\n", - "INFO:root:Epoch[101] Time cost=6.607\n", - "INFO:root:Epoch[102] Train-loss=168.523054\n", - "INFO:root:Epoch[102] Time cost=6.368\n", - "INFO:root:Epoch[103] Train-loss=168.365964\n", - "INFO:root:Epoch[103] Time cost=10.267\n", - "INFO:root:Epoch[104] Train-loss=168.181174\n", - "INFO:root:Epoch[104] Time cost=11.132\n", - "INFO:root:Epoch[105] Train-loss=168.021498\n", - "INFO:root:Epoch[105] Time cost=10.187\n", - "INFO:root:Epoch[106] Train-loss=167.858251\n", - "INFO:root:Epoch[106] Time cost=10.676\n", - "INFO:root:Epoch[107] Train-loss=167.690670\n", - "INFO:root:Epoch[107] Time cost=10.973\n", - "INFO:root:Epoch[108] Train-loss=167.535069\n", - "INFO:root:Epoch[108] Time cost=10.108\n", - "INFO:root:Epoch[109] Train-loss=167.373971\n", - "INFO:root:Epoch[109] Time cost=11.013\n", - "INFO:root:Epoch[110] Train-loss=167.207507\n", - "INFO:root:Epoch[110] Time cost=11.427\n", - "INFO:root:Epoch[111] Train-loss=167.043077\n", - "INFO:root:Epoch[111] Time cost=10.349\n", - "INFO:root:Epoch[112] Train-loss=166.884060\n", - "INFO:root:Epoch[112] Time cost=13.129\n", - "INFO:root:Epoch[113] Train-loss=166.746976\n", - "INFO:root:Epoch[113] Time cost=11.255\n", - "INFO:root:Epoch[114] Train-loss=166.572499\n", - "INFO:root:Epoch[114] Time cost=10.037\n", - "INFO:root:Epoch[115] Train-loss=166.445170\n", - "INFO:root:Epoch[115] Time cost=10.406\n", - "INFO:root:Epoch[116] Train-loss=166.284912\n", - "INFO:root:Epoch[116] Time cost=10.170\n", - "INFO:root:Epoch[117] Train-loss=166.171475\n", - "INFO:root:Epoch[117] Time cost=10.034\n", - "INFO:root:Epoch[118] Train-loss=166.015457\n", - "INFO:root:Epoch[118] Time cost=10.047\n", - "INFO:root:Epoch[119] Train-loss=165.882208\n", - "INFO:root:Epoch[119] Time cost=10.008\n", - "INFO:root:Epoch[120] Train-loss=165.753836\n", - "INFO:root:Epoch[120] Time cost=10.056\n", - "INFO:root:Epoch[121] Train-loss=165.626045\n", - "INFO:root:Epoch[121] Time cost=10.704\n", - "INFO:root:Epoch[122] Train-loss=165.492859\n", - "INFO:root:Epoch[122] Time cost=10.609\n", - "INFO:root:Epoch[123] Train-loss=165.361132\n", - "INFO:root:Epoch[123] Time cost=10.027\n", - "INFO:root:Epoch[124] Train-loss=165.256487\n", - "INFO:root:Epoch[124] Time cost=11.225\n", - "INFO:root:Epoch[125] Train-loss=165.119995\n", - "INFO:root:Epoch[125] Time cost=11.266\n", - "INFO:root:Epoch[126] Train-loss=165.012773\n", - "INFO:root:Epoch[126] Time cost=10.547\n", - "INFO:root:Epoch[127] Train-loss=164.898748\n", - "INFO:root:Epoch[127] Time cost=10.339\n", - "INFO:root:Epoch[128] Train-loss=164.775702\n", - "INFO:root:Epoch[128] Time cost=10.875\n", - "INFO:root:Epoch[129] Train-loss=164.692449\n", - "INFO:root:Epoch[129] Time cost=8.412\n", - "INFO:root:Epoch[130] Train-loss=164.564323\n", - "INFO:root:Epoch[130] Time cost=7.239\n", - "INFO:root:Epoch[131] Train-loss=164.468273\n", - "INFO:root:Epoch[131] Time cost=10.096\n", - "INFO:root:Epoch[132] Train-loss=164.328320\n", - "INFO:root:Epoch[132] Time cost=9.680\n", - "INFO:root:Epoch[133] Train-loss=164.256156\n", - "INFO:root:Epoch[133] Time cost=10.707\n", - "INFO:root:Epoch[134] Train-loss=164.151625\n", - "INFO:root:Epoch[134] Time cost=13.835\n", - "INFO:root:Epoch[135] Train-loss=164.046402\n", - "INFO:root:Epoch[135] Time cost=10.049\n", - "INFO:root:Epoch[136] Train-loss=163.960676\n", - "INFO:root:Epoch[136] Time cost=9.625\n", - "INFO:root:Epoch[137] Train-loss=163.873193\n", - "INFO:root:Epoch[137] Time cost=9.845\n", - "INFO:root:Epoch[138] Train-loss=163.783837\n", - "INFO:root:Epoch[138] Time cost=9.618\n", - "INFO:root:Epoch[139] Train-loss=163.658903\n", - "INFO:root:Epoch[139] Time cost=10.411\n", - "INFO:root:Epoch[140] Train-loss=163.588920\n", - "INFO:root:Epoch[140] Time cost=9.633\n", - "INFO:root:Epoch[141] Train-loss=163.493254\n", - "INFO:root:Epoch[141] Time cost=10.668\n", - "INFO:root:Epoch[142] Train-loss=163.401188\n", - "INFO:root:Epoch[142] Time cost=10.644\n", - "INFO:root:Epoch[143] Train-loss=163.334470\n", - "INFO:root:Epoch[143] Time cost=9.665\n", - "INFO:root:Epoch[144] Train-loss=163.235133\n", - "INFO:root:Epoch[144] Time cost=9.612\n", - "INFO:root:Epoch[145] Train-loss=163.168029\n", - "INFO:root:Epoch[145] Time cost=9.578\n", - "INFO:root:Epoch[146] Train-loss=163.092392\n", - "INFO:root:Epoch[146] Time cost=10.215\n", - "INFO:root:Epoch[147] Train-loss=163.014362\n", - "INFO:root:Epoch[147] Time cost=12.296\n", - "INFO:root:Epoch[148] Train-loss=162.891574\n", - "INFO:root:Epoch[148] Time cost=9.578\n", - "INFO:root:Epoch[149] Train-loss=162.831664\n", - "INFO:root:Epoch[149] Time cost=9.536\n", - "INFO:root:Epoch[150] Train-loss=162.768784\n", - "INFO:root:Epoch[150] Time cost=9.607\n", - "INFO:root:Epoch[151] Train-loss=162.695416\n", - "INFO:root:Epoch[151] Time cost=9.681\n", - "INFO:root:Epoch[152] Train-loss=162.620814\n", - "INFO:root:Epoch[152] Time cost=9.464\n", - "INFO:root:Epoch[153] Train-loss=162.527031\n", - "INFO:root:Epoch[153] Time cost=9.518\n", - "INFO:root:Epoch[154] Train-loss=162.466575\n", - "INFO:root:Epoch[154] Time cost=9.562\n", - "INFO:root:Epoch[155] Train-loss=162.409388\n", - "INFO:root:Epoch[155] Time cost=9.483\n", - "INFO:root:Epoch[156] Train-loss=162.308957\n", - "INFO:root:Epoch[156] Time cost=9.545\n", - "INFO:root:Epoch[157] Train-loss=162.211725\n", - "INFO:root:Epoch[157] Time cost=9.542\n", - "INFO:root:Epoch[158] Train-loss=162.141098\n", - "INFO:root:Epoch[158] Time cost=9.768\n", - "INFO:root:Epoch[159] Train-loss=162.124311\n", - "INFO:root:Epoch[159] Time cost=7.155\n", - "INFO:root:Epoch[160] Train-loss=162.013039\n", - "INFO:root:Epoch[160] Time cost=6.147\n", - "INFO:root:Epoch[161] Train-loss=161.954485\n", - "INFO:root:Epoch[161] Time cost=9.121\n", - "INFO:root:Epoch[162] Train-loss=161.913859\n", - "INFO:root:Epoch[162] Time cost=9.936\n", - "INFO:root:Epoch[163] Train-loss=161.830799\n", - "INFO:root:Epoch[163] Time cost=8.612\n", - "INFO:root:Epoch[164] Train-loss=161.768672\n", - "INFO:root:Epoch[164] Time cost=9.722\n", - "INFO:root:Epoch[165] Train-loss=161.689120\n", - "INFO:root:Epoch[165] Time cost=9.478\n", - "INFO:root:Epoch[166] Train-loss=161.598279\n", - "INFO:root:Epoch[166] Time cost=9.466\n", - "INFO:root:Epoch[167] Train-loss=161.551172\n", - "INFO:root:Epoch[167] Time cost=9.419\n", - "INFO:root:Epoch[168] Train-loss=161.488880\n", - "INFO:root:Epoch[168] Time cost=9.457\n", - "INFO:root:Epoch[169] Train-loss=161.410458\n", - "INFO:root:Epoch[169] Time cost=9.504\n", - "INFO:root:Epoch[170] Train-loss=161.340681\n", - "INFO:root:Epoch[170] Time cost=9.866\n", - "INFO:root:Epoch[171] Train-loss=161.281700\n", - "INFO:root:Epoch[171] Time cost=9.526\n", - "INFO:root:Epoch[172] Train-loss=161.215523\n", - "INFO:root:Epoch[172] Time cost=9.511\n", - "INFO:root:Epoch[173] Train-loss=161.152452\n", - "INFO:root:Epoch[173] Time cost=9.498\n", - "INFO:root:Epoch[174] Train-loss=161.058544\n", - "INFO:root:Epoch[174] Time cost=9.561\n", - "INFO:root:Epoch[175] Train-loss=161.036475\n", - "INFO:root:Epoch[175] Time cost=9.463\n", - "INFO:root:Epoch[176] Train-loss=161.009996\n", - "INFO:root:Epoch[176] Time cost=9.629\n", - "INFO:root:Epoch[177] Train-loss=160.853546\n", - "INFO:root:Epoch[177] Time cost=9.518\n", - "INFO:root:Epoch[178] Train-loss=160.860520\n", - "INFO:root:Epoch[178] Time cost=9.395\n", - "INFO:root:Epoch[179] Train-loss=160.810621\n", - "INFO:root:Epoch[179] Time cost=9.452\n", - "INFO:root:Epoch[180] Train-loss=160.683071\n", - "INFO:root:Epoch[180] Time cost=9.411\n", - "INFO:root:Epoch[181] Train-loss=160.674101\n", - "INFO:root:Epoch[181] Time cost=8.784\n", - "INFO:root:Epoch[182] Train-loss=160.554823\n", - "INFO:root:Epoch[182] Time cost=7.265\n", - "INFO:root:Epoch[183] Train-loss=160.536528\n", - "INFO:root:Epoch[183] Time cost=6.108\n", - "INFO:root:Epoch[184] Train-loss=160.525913\n", - "INFO:root:Epoch[184] Time cost=6.349\n", - "INFO:root:Epoch[185] Train-loss=160.399412\n", - "INFO:root:Epoch[185] Time cost=7.364\n", - "INFO:root:Epoch[186] Train-loss=160.380027\n", - "INFO:root:Epoch[186] Time cost=7.651\n", - "INFO:root:Epoch[187] Train-loss=160.272921\n", - "INFO:root:Epoch[187] Time cost=7.309\n", - "INFO:root:Epoch[188] Train-loss=160.243907\n", - "INFO:root:Epoch[188] Time cost=7.162\n", - "INFO:root:Epoch[189] Train-loss=160.194351\n", - "INFO:root:Epoch[189] Time cost=8.941\n", - "INFO:root:Epoch[190] Train-loss=160.130400\n", - "INFO:root:Epoch[190] Time cost=10.242\n", - "INFO:root:Epoch[191] Train-loss=160.073841\n", - "INFO:root:Epoch[191] Time cost=10.528\n", - "INFO:root:Epoch[192] Train-loss=160.021623\n", - "INFO:root:Epoch[192] Time cost=9.482\n", - "INFO:root:Epoch[193] Train-loss=159.938673\n", - "INFO:root:Epoch[193] Time cost=9.465\n", - "INFO:root:Epoch[194] Train-loss=159.885823\n", - "INFO:root:Epoch[194] Time cost=9.523\n", - "INFO:root:Epoch[195] Train-loss=159.886516\n", - "INFO:root:Epoch[195] Time cost=9.599\n", - "INFO:root:Epoch[196] Train-loss=159.797400\n", - "INFO:root:Epoch[196] Time cost=8.675\n", - "INFO:root:Epoch[197] Train-loss=159.705562\n", - "INFO:root:Epoch[197] Time cost=9.551\n", - "INFO:root:Epoch[198] Train-loss=159.738354\n", - "INFO:root:Epoch[198] Time cost=9.919\n", - "INFO:root:Epoch[199] Train-loss=159.619932\n", - "INFO:root:Epoch[199] Time cost=10.121\n" - ] - } - ], - "source": [ - "# can initilize weights and biases with the learned parameters as follows: \n", - "# init = mx.initializer.Load(params)\n", - "\n", - "# call the VAE, output model contains the learned model and training loss\n", - "out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200) " - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "# encode test images to obtain mu and logvar which are used for sampling\n", - "[mu,logvar] = VAE.encoder(out,image_test)\n", - "# sample in the latent space\n", - "z = VAE.sampler(mu,logvar)\n", - "# decode from the latent space to obtain reconstructed images\n", - "x_construction = VAE.decoder(out,z)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAADSCAYAAACvmc1VAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuU3GWd5/HPl9xJQsilE5JAEgIBQgQCtshNyQpBRVE5\nq86gO6COg7PHdeTozIjumRWdYWX3eJ11jw4uCOIFdcHb6OhGRA0YkA6JQIgxkgSTkPu1cyfJs3/U\nL0yR+n5J/7qrurvqeb/OyUn3p55UP7/+fav6SXU935+llAQAAADk6Li+ngAAAADQV1gMAwAAIFss\nhgEAAJAtFsMAAADIFothAAAAZIvFMAAAALLFYrgfMLP/Y2Yf6+t5AD1hZtPMLJnZwOD2JWY2p5en\nBRwTtYtmRN3WD4thSWa2q+rPYTPbW/X5Oxv99VNK700p/fdGfx00DzNbZWZX9vU86imlNCul9Mu+\nngcai9pFM6Ju8+b+byI3KaURRz42s1WS3ptS+nk03swGppQO9sbcgEahjtGsqF00I+q2/+KV4S4w\ns38ys2+b2bfMrFPSfzKzr5vZLVVjriwW0kc+P9nMvmdmm8xspZm9/yXu/4X7OnI/ZvbR4t8+Z2bX\nmNkbzWy5mW01s7+v+rcXm9kjZrbdzNaZ2T+b2aCq219vZn8wsx1m9r/M7GEze1fV7e81s9+b2TYz\n+zczO6VO3zY0SFELi4tz/hszO7fqtpvN7Bkz6zSzp83s2qrb3lWc/8+Z2RZJtxTZQ2b26aIGVprZ\n66v+zSgzu6OorbXFY2FAcduA4t9tNrMVkt5wjHm/8MqLmd1iZt8tar/TzJ40szOKut9oZqvN7Kqq\nf/tuM1tajF1hZu876r7/vpjjc0VNJzM7vbhtSDHPP5nZBjP7spkN69lZQHdQu9RuM6JuW79uWQx3\n3bWSvilplKRvv9RAMztO0r9KekzSZElzJf2dmV3Rxa91sirnZpKkf5R0h6Q/l3S+pDmSPmlmU4qx\nByV9UNI4SZdKep2k9xXzGC/pO5L+rrh9paQLq+b5H4vb3iypTdKjxTGinzKz8yXdqco5HivpXyT9\n0MyGFEOekfQqVer0E5K+bmYTq+7ilZJWSJog6daqbJkqNfI/Jd1hZlbcdpcqNXa6KvV3laT3Frf9\nlaQ3Fnm7pLeWPJxrJN0jabSkRZJ+pkrdT5b0yeLYjthYfK0TJL1b0ufM7ILie/I6SR+SdGUxzzlH\nfZ3bJJ0haXZx+2RJ/63kXNFD1C6124yo20zqNqXEn6o/klZJuvKo7J8k/eKo7OuSbqn6/EpJq4qP\nL5W04qjx/yDpK8HXfOG+ivvZJWlA8floSUnSy6vG/07SG4P7+ltJ3y0+fo+k+VW3maR1kt5VfD5P\n0g1Vtw+UtF/S5L4+D7n/8eqwyL8k6R+PypZJujy4n8WS3lx8/C5Jfzrq9ndJ+mPV58cX9XaSKk/e\n+yUNq7r9OkkPFh//QtJfV912VfFvBx7rmCTdImle1W3XHFX3I4v7OjG4r+9L+mDx8Z2SPlV12+nF\nvz29qPndkk6ruv1iSSv7+hy36h9ql9ptxj/Ubd51y3uGu251ibFTJU0xs+1V2QBJv+ziv9+cUjpU\nfLy3+HtD1e17JY2QJDM7S9JnJL1clQfVQFVe4ZUqryy/MO+UUjKzNUfN83+b2ReqssOqvDK9totz\nRe+aKukGM/tAVTZYlXMtM7telf+xTytuG6HKqw9HeHW8/sgHKaU9xQsUIySNkTRI0rp/f9FCx1Xd\nx4vqS9KzJY/l6Jr26n6EpO3FrxE/rsqrDcepUutPVs2jo+q+qufUVoxdWHUMpsrjEb2L2qV2mxF1\nm0HdshjuunTU57tVOeFHnFT18WpJy1NKMxs+q8qvNR6R9GcppV1m9req/GpDqrwKXP0eIFPl1xXV\n8/yHlNJLvu0D/cpqSbemlG49+gYzmyrpK5KukLQgpXTIzBar8kR0xNF1fKyvtV/SuORv+lgnqfo9\n5lOcMT1W/DryPknXS/pBSul5M/u+/v241qnyH7gjque0WZUn+VkpJf6D17eoXWq3GVG3GdQt7xnu\nvsWS3mBmo4v3B/1N1W0LJB0wsw+b2dDiTe/nmNnLGzCPkZJ2SNptZjNVvF+48K+SLrDKBryBqry3\nuK3q9i9L+q/Fv5OZnWhmZd+DhMYZVNTPkT8DVXni/Wsze6VVDDezN5jZSEnDVXni3SRVNkBIell3\nv3hKaZ2k/yfpM2Z2gpkdZ2anmdnlxZDvSPobq2wWHS3p5h4c60sZLGmIKsd1sHjF4qqq278j6d1m\nNtPMjlflLUlHjuGwKt+zzxXvoZeZTTaz1zZorqigdiuo3eZC3VZkV7cshrvvLklLVfk1xU8l3Xvk\nhuJ/dFerslltlSr/U/oXVd6IXm8flnSDpM7ia7zwKm9KaYOkP5P0WUlbJJ2mypvm9xe3f7e47btm\ntlPSE5L6dcFm5ieq/A/7yJ9bUkodqmyi+KKkbZL+qMp70JRSelqVt8wsUOXXYedIeriHc7helSfG\np4uv938lHdkc8hVVNmD8TtLjku7v4ddypZQ6VfnP5neKObxD0g+rbv83Sf8s6UFVvh+PFDftL/7+\nyJG8qPOfSzqzEXPFC6hdUbtNiLpVnnVrxZubkQGrtGd5TtJbU0rz+3o+QCMUv+l4StKQ4FeNQL9E\n7aIZtULd8spwizOz1xVvfxiiyq8ynpf02z6eFlBXZnatVXpbjpb0PyT9qFmflJEXahfNqNXqlsVw\n67tMlR6Hm1R5C8S1KaX9L/1PgKbzPlX6Yj4j6ZCk/9y30wG6jNpFM2qpuuVtEgAAAMgWrwwDAAAg\nWz1aDBfvR11mZn80s0a1+AAAAAAaottvkyg6E/xB0lxJayQ9Jum6otWIa9y4cWnatGnd+nrAEQsX\nLtycUmo79sj6oXbRU6tWrdLmzZvt2CPrh7pFPfCci2bV1drtyRXoLlTl+torJMnM7pX0ZlV647mm\nTZumjo6O6GagS8ys7CUoe4zaRU+1t7f3+tekblEPPOeiWXW1dnvyNonJevH1qNfoxZf6BQAAAPq1\nhm+gM7MbzazDzDo2bdrU6C8H1A21i2ZE3aJZUbvoKz1ZDK+VdErV5ycX2YuklG5PKbWnlNrb2nr1\nLUdAj1C7aEbULZoVtYu+0pPF8GOSZpjZqWY2WNKfq+ra1QAAAEB/1+0NdCmlg2b2XyT9TNIASXem\nlJbUbWYAAABAg/Wkm4RSSj+R9JM6zQUAAADoVVyBDgAAANliMQwAAIBssRgGAABAtnr0nmEAAPpC\nSqmh92/Wq1fORotqZJ0ePnzYzQ8ePOjmhw4dcnNvjlH9H3ec/xrqwIH+cjIaH91/Xz3ueGUYAAAA\n2WIxDAAAgGyxGAYAAEC2WAwDAAAgWyyGAQAAkC26SQAAAFQp2wWizPiy9x11gdi3b5+bb9++3c23\nbt3q5gcOHKjJBg8e7I4dNWqUm48ePdrNhw0b5uZluk/0RocJXhkGAABAtlgMAwAAIFsshgEAAJAt\nFsMAAADIFhvoALScel0ClUvy9g+NvvRyPTRyA1WkTH3mXsv12hBXj/uJLqMcbZTbvXu3m69evdrN\nFy1a5OYrVqxwc28j3oknnuiOPf3009181qxZbj5p0iQ3HzFihJsPGjTIzT31rGleGQYAAEC2WAwD\nAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABkq0fdJMxslaROSYckHUwptddjUgBQrdHdBOpx/znv1u+L\nbg+N7shQtpuA1yEg6g5Q9r6jOXqXro3yAQMGlLrvXNSrjsqc06ibxP79+9183bp1bv7oo4+6+WOP\nPebmGzZscHPv0sjjx493x0bdHqLuE0OHDi11P16dRrVb9vHyUurRWu0/pJQ21+F+AAAAgF7F2yQA\nAACQrZ4uhpOkn5vZQjO7sR4TAgAAAHpLTxfDl6WUZkt6vaT3m9mrjx5gZjeaWYeZdWzatKmHXw7o\nPdQumhF1i2ZF7aKv9GgxnFJaW/y9UdL3JF3ojLk9pdSeUmpva2vryZcDehW1i2ZE3aJZUbvoK93e\nQGdmwyUdl1LqLD6+StIn6zYzIEP1uO59d+6nL+bSyLmX2WUvldt9XLb7QLPu1i9zHup1jqNd9l4e\njY06OETjo7mUvR9vfHQfkWjXfNkd+V53gLIdKZq1bqW+6W4S8erl+eefd8dGr4ZH3SGibhIrVqxw\n82HDhrn5mDFjujw2mvvGjRvdPOoycfzxx7v54MGDa7LeqNGedJOYIOl7xWQGSvpmSumndZkVAAAA\n0Au6vRhOKa2QdF4d5wIAAAD0KlqrAQAAIFsshgEAAJAtFsMAAADIVj0ux9zvPPLII27+hS98wc0n\nT57s5tFuyhtuuKEm83ZjvlSO1lNm53y0y/zAgQNuvmfPHjffvXt3qfH79+/vUvZSc9m7d6+bR7uM\no/vp7Ox08127dtVk0Ryj3fSnnXaam8+cOdPNx48f7+bezuZox3+z7r4vU7f16uAQ1Yp37iVp27Zt\nNdnmzZvdsTt27HDzgwcPunnZziNepwbJP9aoPqOfLdHO+9GjR7v5yJEj3TyaO7quXt10vMdGVKNL\nlixx8wULFrj50qVL3TyqizPOOMPNTz311Josqrl9+/a5efQzZ+vWrW4e1foJJ5xQk3nPw/XGIwYA\nAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQrZbsJuF1e5Ck5cuX1+X+b7311pps1KhR7tiL\nLrqoLl+zL0ybNs3NP/rRj7r5lClTGjib/qPM7nvJ300c7chdv369m0e1G12Dfu3atW7uXT8+uqb8\n6tWr3TzaCR0df7QTOOrK4HUaiHYqDxkyxM1f/vKXu/mb3vQmN7/iiivcfOzYsTVZNO/+rmzdet0R\noq4RUaeGqM63bNni5suWLXPzxYsX12RR3UbnJ9odP2LECDePOj5EHTK8LivR8Uc76WfMmOHmUZ0f\nf/zxbt6snU2aQZnHi+R301mzZo07dv78+W6+aNGiLs6uIuoacfHFF3d5fFRDzz33nJtHx7R9+3Y3\njx4b3vcx+t7W87mYV4YBAACQLRbDAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WrJbhLf//733dzb\nkSxJs2bNcvPoOuGPPvpoTfaDH/zAHfuzn/3Mzb1rgUvSypUr3bysgQNrT+3EiRPdsVHXgEjUZeIj\nH/lIqftpNfXoJrFu3To3f/LJJ9086jIR1ZF3rqNd+bt373bzQYMGubl3TXkp3vE+cuRIN9+5c2dN\nFu1I9jpPSHEngKjLhLfjO3dePZepcSk+b1E9P/jgg27udU2JznH0/DRu3Dg3P+mkk9zcew6VpG3b\ntrm59ziKHs/RfYwZM8bNo049xx3nv57ldQLIqcOEd6xR7ZYV3U/UUWXz5s012cMPP+yO9dYWktTZ\n2enmF154oZtfcsklbt7e3u7mXneT6GfUrl273Dyqxeg5OuoQUeZ5p554ZRgAAADZYjEMAACAbLEY\nBgAAQLZYDAMAACBbx1wMm9mdZrbRzJ6qysaY2TwzW1787V/nEgAAAOjHutJN4i5JX5T0tarsZkkP\npJRuM7Obi8/7TSuBmTNnlsoj5557rptfd911Ndltt93mjl21apWbR90kvF3T3TF48OCaLOomEc1l\n06ZNbn7WWWd1f2KQFF9TfejQoW4edV4YNWqUm0+aNMnNvc4Op5xySqmvOX78eDefPn26m0+ePNnN\no93EHR0dNdm9997rjt26daubR4YPH+7mQ4YMcXNvh3Quu/LLHGfUTcLbSS9Jjz/+uJtHz5fe4+WM\nM85wx1566aVuHj3PjRgxws2j3fR/+tOf3Nx77o66Ruzfv9/No533UQeXqONFmbrNpZ7rJXreiros\nLFq0qCZbsGCBO3bHjh1ufs4557j5q1/9ajd/xSte4ebRc7f3+Ioe01Fnhz179rh5szjmK8MppV9L\nOvonzpsl3V18fLekt9R5XgAAAEDDdfc9wxNSSkcaKK6XNKFO8wEAAAB6TY830KXKa+ZhR2Qzu9HM\nOsysI/q1O9AfUbtoRtQtmhW1i77S3cXwBjObKEnF3/4lrCSllG5PKbWnlNrb2tq6+eWA3kftohlR\nt2hW1C76Sncvx/xDSTdIuq34278WcUaijU9lN5uV3eRXRnSpx2iTyytf+Uo3v+qqq+o2p1YSbUTx\nNid4GxwlacIE/x1Hs2fPdvOxY8e6ebQRw9tYE10uOdpwGeXRXKJNPlu2bHFzbxNVtJkj2ogYbdo7\n88wz3TzaWBfdv6dZNyKV2VgVjY0uRRtdjnj9+vVuHm1OOvvss2uyuXPnumNnzJjh5tFm0+iYog1R\n0WZLb3x0qfNog2t0ienoMVpmAx180fmPajHa5Bhtfp83b15NtmbNGndstF64/PLL3TzaLBptlIs2\nYnrPr7t373bHRptco8d6dBn06PveV7XbldZq35K0QNKZZrbGzP5SlUXwXDNbLunK4nMAAACgqRzz\nleGUUm0fsYor6jwXAAAAoFfxuxQAAABki8UwAAAAssViGAAAANnqbjcJ9HPeTtBrr73WHRvtmv38\n5z/v5tGO59zVo5tEtOM96qYQ7TKPduR65y7qhBJdpjYaH33NaO5RB4Knn366Juvs7HTHRpeMPv/8\n89389NNPd/McLscczTu6vGqZ44wuXbx69Wo33759u5tHXT0uuOCCmiw6l6NHj3bzqPNC9PwXfV+i\nTi1eh4CoU090aeioU0v0nBs95pq1RhupbNeI6Hlrw4YNbv7jH//YzX//+9/XZFHXoAsvvNDN58yZ\n4+YnnXSSm0e1Hh2T9/y6fPlyd+zChQvdPOqcUrYWvZ+XvVHnvDIMAACAbLEYBgAAQLZYDAMAACBb\nLIYBAACQLRbDAAAAyBbdJFrUXXfdVZOtX7/eHTt27Fg3nzp1aj2n1DLqsSs/2h0bXTs+6poQdXaI\n7sfrmuDt3pXiOZa9dvzzzz/v5g8//LCb/+IXv6jJos4TZ555ppu/5jWvcfPx48e7edTdg135LxbV\n+K5du9w86qawd+9eN4+6KZx88sk1WVT70RwPHDjg5lEnjC1btrj5U0895eZPPvlkTbZnzx53bNQ1\n5oQTTnDzqDtA2cdizqK6KNNhQZIeeOABN1+wYIGbe89/USeUyy67zM3b2trcPHp+imrd6zIlSStX\nrqzJvOdhSXr88cdL3Xe0voh+RpV5zq1HJ5wjeCQBAAAgWyyGAQAAkC0WwwAAAMgWi2EAAABki8Uw\nAAAAskU3iSb3zDPPuPmHPvShLt9HtAs2uu45Gifq7BDtnPe6Q0jldp9HO3IjZcevW7fOze+99143\n9zoQRLvvr776ajc/77zz3Dz6PtI1omsOHz7s5lE3iWinfrSTPLJx48aarGznhairSdQ1YMWKFW4e\n7aZfs2ZNTRbV7ejRo908qs9I2cdiLrzvS1SL+/fvd/MlS5a4+bx589z82WefdXOvc0TUTSKql7Ld\nIaJajzqkLFy4sCZbtmyZO3b16tVuHtVi1Amov3VI4ZVhAAAAZIvFMAAAALLFYhgAAADZYjEMAACA\nbB1zMWxmd5rZRjN7qiq7xczWmtni4o+/iwUAAADox7rSTeIuSV+U9LWj8s+llD5d9xmhlB/96Edu\n7u0mfdvb3uaOnT59el3nlKuoI4G3yzYaG+2kLbv7PlKma0I0Nsr37dvn5j/+8Y/dvKOjw829XcZz\n5sxxx771rW9182i3ftStI2dl6jbaMR7tDD/xxBPdfMyYMW4e7Tz3aiXqpBOd46gTRnRMW7dudfOn\nnnrKzb2OGuPGjXPHjh8/3s0HDx7s5n21w75Zeec66rCwfv16N49+tj7xxBNuHtWRd64nTJjgjo06\nW0RdI6JOKNGxep16JGnDhg012Z49e9yx0WM06mwUdciInjP6yjEfYSmlX0vynxUAAACAJtaT/25+\nwMyeKN5G4b8MAwAAAPRj3V0Mf0nSdEmzJa2T9JlooJndaGYdZtaxadOmbn45oPdRu2hG1C2aFbWL\nvtKtxXBKaUNK6VBK6bCkr0i68CXG3p5Sak8ptbe1tXV3nkCvo3bRjKhbNCtqF32lW4thM5tY9em1\nkvwdBQAAAEA/dsztfGb2LUlzJI0zszWSPi5pjpnNlpQkrZL0vgbOEYp3h37ve99zc29n56c+9Sl3\nLDvsG8vbrV+2m0SZHf9lld2pfujQITePdvffc889bh7tnJ45c2ZNdtNNN7ljp06d6ubRrvyyHTLw\nYtH3Keoacd5557l5dH62bNni5l7NRTvso84rQ4cOLZVH9blt2zY390yZMsXNX/ayl7l51AUlOqbo\nuTuXeo6e/7yOB9u3b3fH/uY3v3Hz3/72t26+d+9eN584caKbn3rqqTVZ1GGhbNeIqONJtF7YuHGj\nm+/cubMmi7pGRLV48sknu7l3/JI0bNgwN+8rx1wMp5Suc+I7GjAXAAAAoFfRvBAAAADZYjEMAACA\nbLEYBgAAQLZYDAMAACBb/evi0AjdcYe/Z3H+/Plu/o53vKMmmz59el3nhO4r200i2jVdj24S0VwO\nHz7s5tGO/y9/+ctuvmTJEjePdl/feOONNdkrXvEKd2zUCaBshwzU8upi4ED/R8a4cePc3OtqI0mn\nnHKKm0e75g8cOFCTRbVftmNItIN/5cqVbh51U/E6QVx00UXu2BkzZrj58OHD3Tz6vpfpGtGKHSai\nGti3b19NFp3PhQsXurnXYUGKa93rgiP53RSijgxRB5Ooa8SGDRvcfMeOHW6+evVqN1+zZk2X7zvq\neBJ1jjnnnHPcPKr1vnru5icGAAAAssViGAAAANliMQwAAIBssRgGAABAtthA188sXrzYzT/wgQ+4\neXQZ1E9+8pN1mxPqr+wmn3pslItE971nzx43/+lPf+rm999/v5tHl9597Wtf6+Zvf/vba7Lo0p1s\nlGucMhvoovMQbaCLNuFEm9O8Go3GRpeRjTYnrV27ttT4aOPPhAkTarK5c+e6Y6PjL7tRrhU3xZUR\nbfL1Nr/98Y9/dMdGm83Gjx/v5tHP3GhjnbeZ77nnnnPHRpeMXrFihZtHtbtq1apS470NqtElo9vb\n2938yiuvdPOpU6e6efRzoa9qmp8kAAAAyBaLYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGzRTaKP\n7N27182vu+46N492Tr/zne90cy693FrqtcPW25Uf7Zr/3e9+5+bf/OY33Ty6ZOj555/v5tdff72b\ne7u1c981319EXSOi8zNgwAA3j7omRJ1NvK4B0XNimY4Ukr/bX4ovxztmzBg3v+CCC2qy6HmYrhHl\nlKkLyX9Oi7rjRB1Pom4K0bmLulIsWrSoJos6nqxbt87No04YmzdvdvPoEuPR3L2OD1F3iGuuucbN\nzzzzTDePOqdEzw1erfdG/fPKMAAAALLFYhgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbx+wmYWan\nSPqapAmSkqTbU0pfMLMxkr4taZqkVZLenlLa1ripNqdot+sb3vAGN1+2bJmbz5w5080/8YlPdG9i\nyJJ3DfpnnnnGHXvfffe5+cKFC928ra3Nza+++mo3nzVrlpt7O55z303f35U9P9H46PmyzH1E+fPP\nP+/mq1evLjUXb+e9JJ177rk12ciRI92xZUXdFHhc+AYNGlSTRc9PUdeIzs5ON4+6UmzYsMHNt22r\nXRZFXSCijjxRd4hI1MEh6uzjdYiYM2eOO3bSpElufvzxx7u5dy6k8p1pGq0rrwwflPThlNLZki6S\n9H4zO1vSzZIeSCnNkPRA8TkAAADQNI65GE4prUspPV583ClpqaTJkt4s6e5i2N2S3tKoSQIAAACN\nUOo9w2Y2TdL5kh6VNCGldKRD9HpV3kbh/ZsbzazDzDo2bdrUg6kCvYvaRTOibtGsqF30lS4vhs1s\nhKT7JN2UUnrRpXlS5Q1N7puaUkq3p5TaU0rt0Xt2gP6I2kUzom7RrKhd9JUuLYbNbJAqC+FvpJTu\nL+INZjaxuH2ipI2NmSIAAADQGF3pJmGS7pC0NKX02aqbfijpBkm3FX//oCEzbHLR7tBf/vKXpe7n\nnnvucfMxY8aUnRIy4HWNkKS1a9fWZPPmzXPH/vznP3fzaBfwq171Kjd/05ve5ObDhw93c3bIt76o\nO0KUl3Ho0CE3j7oDbNmyxc2j+pw4caKbn3XWWTVZtJM+UrZrRC5dJqLjiZ6LvC4eUReQqFPD0KFD\n3Tx6+0bUTcGb4759+9yxkehV8unTp7v5RRdd5OZz58518xkzZtRkJ5xwgjvW6/YjSQMGDHDzsufO\nG98b9XzMxbCkSyX9haQnzWxxkX1MlUXwd8zsLyU9K+ntjZkiAAAA0BjHXAynlB6SFC3Lr6jvdAAA\nAIDewxXoAAAAkC0WwwAAAMgWi2EAAABkqysb6NAFO3bscPNoV2fk61//uptH1xRH3qKd8xs3+p0O\nH3rooZrsV7/6lTs26oTi7TyWpCuu8LcQRDueo93HaH316BoR3cfBgwfdfOfOnW4ejR87dqybT5o0\nyc29DgaRehw/4o4EXmeHKVOmuGOjriHTpk1z8127drl51CFi7969NVnUHWHIkCFuHnWNmjDBvdZZ\nWLvRsXodIsp2h4hE4+vRIaKeXSZ4ZRgAAADZYjEMAACAbLEYBgAAQLZYDAMAACBbLIYBAACQLbpJ\n1MlXv/pVN1+xYkWp+7nsssvcvNWuNY9yDh8+7OadnZ1u/oc//MHNFy5cWJOtWrXKHTt69Gg3b29v\nd/OLL77Yzb2d3VK8ExytI+qaUI88Grt//34393b1S9LgwYPdfNy4cW7e1tZW6n7KKNtlIvefC9Hx\ne+fixBNPdMeOGDHCzaOuIVH3keg52ptjVCuDBg1y86izQ/QcGuVl6qXRHU/KzKU36pyfRgAAAMgW\ni2EAAABki8UwAAAAssViGAAAANliMQwAAIBs0U2iG5YvX16T3XLLLb0/EbScsjvk169f7+ZLly51\n87Vr13Z5LtOmTXPzWbNmuXm0+z73He85KNsdoh6i3ftRPmTIEDc/+eST3TzqGjB27Fg39zoERLVP\n14j6KPMGNBzJAAAFd0lEQVR9iTo1DBzoL4OGDh3arTn1RL3Ocz3uJ7ea45VhAAAAZIvFMAAAALLF\nYhgAAADZYjEMAACAbB1zA52ZnSLpa5ImSEqSbk8pfcHMbpH0V5I2FUM/llL6SaMm2p/Mnz+/Jtu5\nc2ep+5g5c6abDxs2rFtzQnOJNtAcOnTIzaMNdNHlmKPxXn1NnTrVHTtlyhQ3P+2009w82ogSib4H\n3saNem3Eym1TSH9V9jx4l5eNLlEbXeo2urx4dD9RzUWXFx81alRNFm3aqsflcrszPhf97VK/6P+6\n8tProKQPp5QeN7ORkhaa2bzits+llD7duOkBAAAAjXPMxXBKaZ2kdcXHnWa2VNLkRk8MAAAAaLRS\n7xk2s2mSzpf0aBF9wMyeMLM7zcz9HZSZ3WhmHWbWsWnTJm8I0C9Ru2hG1C2aFbWLvtLlxbCZjZB0\nn6SbUko7JX1J0nRJs1V55fgz3r9LKd2eUmpPKbW3tbXVYcpA76B20YyoWzQrahd9pUuLYTMbpMpC\n+BsppfslKaW0IaV0KKV0WNJXJF3YuGkCAAAA9deVbhIm6Q5JS1NKn63KJxbvJ5akayU91ZgpNrdL\nLrnEzefNm+fmdJPIW7SzOerUMGbMGDefPXu2m3s77Xfv3u2OjbpJTJ7sbxmIds6XvVSvl7PLvn/r\ni8vIRmNHjhzp5tFz60knneTmUceHKPceo1Gniug+ItQz0Fhd6SZxqaS/kPSkmS0uso9Jus7MZqvS\nbm2VpPc1ZIYAAABAg3Slm8RDkrz/lmbRUxgAAACtiyvQAQAAIFsshgEAAJAtFsMAAADIVlc20OEo\n73nPe7qUAZFod3i0+3zEiBFuPnz4cDefPn26m19++eU1WdTVIVKvzg716ByA/q2R561spwavk0qj\nUbdAc+CVYQAAAGSLxTAAAACyxWIYAAAA2WIxDAAAgGyxGAYAAEC2rOxO8h59MbNNkp7ttS/YeOMk\nbe7rSfSS/nSsU1NKbb35BandptWfjpO6rY/+dE4bqT8dJ7Xbc/3pfDZafzrWLtVury6GW42ZdaSU\n2vt6Hr0hp2PNQS7nM5fjzEku5zSX48xFTuezGY+Vt0kAAAAgWyyGAQAAkC0Wwz1ze19PoBfldKw5\nyOV85nKcOcnlnOZynLnI6Xw23bHynmEAAABki1eGAQAAkC0Ww11kZnea2UYze6oqG2Nm88xsefH3\n6L6cYz2Y2Slm9qCZPW1mS8zsg0XecseaC2q39Y41B9Rt6x1rLqjd5jtWFsNdd5ek1x2V3SzpgZTS\nDEkPFJ83u4OSPpxSOlvSRZLeb2ZnqzWPNRd3idpttWPNwV2iblvtWHNxl6jdpjpWFsNdlFL6taSt\nR8VvlnR38fHdkt7Sq5NqgJTSupTS48XHnZKWSpqsFjzWXFC7rXesOaBuW+9Yc0HtNt+xshjumQkp\npXXFx+slTejLydSbmU2TdL6kR9Xix5qhlj6f1G7LaulzSd22tJY+n81euyyG6yRV2nK0TGsOMxsh\n6T5JN6WUdlbf1mrHmrtWO5/Ubh5a7VxSt/lotfPZCrXLYrhnNpjZREkq/t7Yx/OpCzMbpEphfyOl\ndH8Rt+SxZqwlzye12/Ja8lxSt1loyfPZKrXLYrhnfijphuLjGyT9oA/nUhdmZpLukLQ0pfTZqpta\n7lgz13Lnk9rNQsudS+o2Gy13PlupdrnoRheZ2bckzZE0TtIGSR+X9H1J35E0RdKzkt6eUjr6TfNN\nxcwukzRf0pOSDhfxx1R5H1BLHWsuqF1qtxlRt9Rts6J2m692WQwDAAAgW7xNAgAAANliMQwAAIBs\nsRgGAABAtlgMAwAAIFsshgEAAJAtFsMAAADIFothAAAAZIvFMAAAALL1/wEufHLDkoXM+QAAAABJ\nRU5ErkJggg==\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4, sharex='col', sharey='row',figsize=(12,3))\n", - "ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax1.set_title('True image')\n", - "ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax2.set_title('Learned image')\n", - "ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax3.set_title('Learned image')\n", - "ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - "ax4.set_title('Learned image')\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 78, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEICAYAAABLdt/UAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXuUZXV157/7vmjurQbjacQRqFuK4izGqIlE42saLZNg\ngxLHLJUU0s7MsuTmMcSBPLBWBieZMq9ZQSfIYI3iOH1vjMwYHzE4JATGrPggNAhMRHTQrioUH3QB\nAl1IN1V7/rj3FKdOncfvnHve9/tZa6/uuvc8fufcc75nn/3bv/0TVQUhhJDqUMu7AYQQQpKFwk4I\nIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk4IIRWDwk5yRUSWReR1ebeDkCpBYSelQURURJ6b0LbOFpHv\nJLEtQooGhZ0QQioGhZ0UBhF5qYh8WUQeFpHvichVItIafff3o8XuFJHHROSto8/PE5E7Rut8SURe\n6NjesohcJiJ3iciPROQTIrJLRDoAPg/gWaNtPSYiz/Jozz4RuVtEHhWR74rIZaPPzxaR74jIe0Tk\n8Gg/c471zhWRr4rIIyJyn4i817XdV43a+vDo+3eMPj9ORP6ziKyKyA9E5BoROT7Rk0wmAgo7KRIb\nAN4NYA+AlwOYBfArAKCq/3K0zItUdUpVPyEiPwXgWgDvAmAB+BCAz4rIcY5tvgXAOQCeDeCFAN6h\nqkcAvB7A/aNtTanq/R7t+QiAd6nqbgAvAHCT47tnjtp5CoD9AJZE5Pmj744AuAjA0wCcC6AnIr8I\nACLSxfCh8mcATgLwYgB3jNb7QwBnjD577mjb/8Hw3BGyBYWdFAZVvU1Vv6KqT6rqMoZCvTdglXkA\nH1LVW1R1Q1U/BuAJAD/rWOa/qOr9qvoggL/CUDRNOQbgTBE5QVUfUtXbXd//rqo+oapfAPDXGD5E\noKr/R1X/r6puqupdAD7uOI5fBnCjqn5cVY+p6pqq3iEiMjqed6vqg6r6KID3AXhbhPYSAoDCTgqE\niJwhIp8Tke+LyCMYCtuegFW6AC4dhTQeFpGHAZwGwBlW+b7j/+sApiI06c0A9gFYEZEviMjLHd89\nNPL8bVbs/YrIy0TkZhF5QER+BOBix3GcBuBbHvs6CUAbwG2OY/nfo88JiQSFnRSJ/wrgHgDPU9UT\nALwHgAQsfx+ARVV9msPaqvpxg32FljVV1VtV9XwAzwDwaQDXOb7+iVGs3mYagB3O+XMAnwVwmqqe\nCOAax3HcB+B0j90dBvA4gH/hOJYTVTXKg4gQABR2Uix2A3gEwGMi8s8B9Fzf/wDAcxx//zcAF488\nZBGRzqjjcrfBvn4AwBKRE72+FJGWiMyJyImqemzUrk3XYv9xtNyrAZwH4H86juNBVf2xiLwUw/CL\nzQDA60TkLSLSEBFLRF6sqpuj47lSRJ4xasMpIvILBsdCyDYo7KRIXIahCD6Koch9wvX9ewF8bBSq\neIuqHgTwTgBXAXgIwL0A3mGyI1W9B8PY97dH29uRFQPg7QCWR2GhiwHMOb77/mif92Mo1hePtgkM\nO3x/T0QexbDzc8vTV9VVDMM7lwJ4EMOO0xeNvv7t0TF8ZbTPGwHYHbKEGCOcaIOQaIjI2QD6qnpq\n3m0hxAt67IQQUjEo7IQQUjEYiiGEkIpBj50QQipGI4+d7tmzR2dmZvLYNSGElJbbbrvtsKqGDlrL\nRdhnZmZw8ODBPHZNCCGlRURWTJZjKIYQQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QQioGhZ0QF4PB\nADMzM6jVapiZmcFgMMi7SYREIpd0R0KKymAwwPz8PNbX1wEAKysrmJ+fBwDMzc0FrUpIYaDHToiD\nhYWFLVG3WV9fx8LCQk4tIiQ6FHZCHKyurkb6nJAiQmEnxMH09HSkz6sK+xnKDYWdEAeLi4tot9vb\nPmu321hcXMypRdlj9zOsrKxAVbf6GSju5YHCToiDubk5LC0todvtQkTQ7XaxtLQ0UR2n7GcoPxR2\nQlzMzc1heXkZm5ubWF5eLrSopxEyWVnxrjPl9zkpHkx3JKSkpJWaWa/XsbGx4fk5KQf02EnpKXtH\nX9z2RwmZRNmHl6gHfU4KiKpmbi95yUuUkCTo9/vabrcVwJa1223t9/uxt9ftdlVEtNvtxt6O6fZ7\nvV7s9ovItvVsE5Ed+3Tvo9VqqWVZnsfZ7XY9t9vtdhM9FyQ6AA6qgcZS2EmpSVKEkn5ImGzfT5xN\n2m967H7L+R1n2ueBxIfCTiYCU6/VhLQ9VROBjdJ+UwH2O0dBx5n2mwuJB4WdlB4TcUlSjJN8SETZ\n/jjtDzpH9ndJPkxIvlDYSakx9UaTDBuEPSScQlmv17e+M92X3/bdgp9E2MPrvCT1MCH5QWEnpcLt\neVqWFSpCtvj1er0toa3X69rr9Yz2EeUhESSUpkLst/1er7ejQ3XcMEiQp25ZljabTd/jZAimuFDY\nSWmI4106RTFJz77f7297qFiWZRTS6HQ6RsfZ6XS21qnVajseQmHtNBXesLCS13bYaVp8KOykNESJ\nA7vN9tS9vFKncDkF1b2cEz9xM2mL35uCvV23lwwM0w6dHrrf8djiayq8cfoekuqvoNefHhR2Uhqi\ndCqmYVNTU1seq5+w+n3uXsaPoIeX6fHXarVA4XUKqmVZ2mq1PB8CfsKbROcxvf50obCTwuEnKH6i\nNzU1ZSSmSYh7o9Hw9KjdAhW2HT8PNe2Hl5egNpvNHYOQvN4c6vV6YJ9GFI897ZTRSYfCTgqF3+Cc\nXq/nOzIyTGiBYWw7K48/rEPX2XZnTNx0vXEs7E3D/jfOuQoKMblJO2V00jEVdhYBI5ngVddEVXHN\nNdcAAI4//vit72u1Go4ePWq03SNHjiTb0ADW1taMljt69CguuugifPGLX8RHPvIR42MZh7D6Lva/\nQ22IxvXXX2+87PT0tGcVyEmbqCRvWASMZILf1HK2uDtFc3NzM6tmpcbm5iauueaaQFG3LAu9Xg+t\nVivDlkUnyrSAnKikGCQm7CJSF5GvisjnktomqQaDwQC1mv+lFseLLANBxyUiOHz4MK6++mrs3r07\nw1ZFJ4q3zYlKikGSoZhLAHwdwAkJbpOUHLtmOEu+bkdVISJ5NyOUVqsV2duem5ujkOdMIh67iJwK\n4FwAH05ie6Q6eMXW42BP8hDk+ZPk2b17N0W6hCR1l7wfwG8B8A2Oisi8iBwUkYMPPPBAQrslRSco\nPtvtdo238/znPx8iUon4e5l48MEH824CicHYwi4i5wH4oareFrScqi6p6lmqetZJJ5007m5JSQiL\nz1qWZbSdu+++u7Kx+LRIYio7ZrOUkyQ89lcCeKOILAP4CwCvFZF+AtslFcArS8JmZWUFjzzySOGz\nQsqIiBj3a9gPAHfMn9ks5WVsYVfVy1X1VFWdAfA2ADep6oVjt4yUnsFgsBVj9/Mejx07ht27d29l\nUZBkiPJ2Mz8/D1XFgQMHtmWz7N+/H5dccglEBCKCPXv2lG4+2YnFZBSTqQE4G8DnwpbjyNP8SLpA\nk9/2olZstOn1epmMIqU9ZV7D/YOKlrHuS36AJQWImzQmfvbbXtSZe5xtSKr+C83c3AT9fqz7kh8U\ndrIDv5olzhs1ikcfVPApak0Suw302LO3Wq224zcP+v1Y9yU/KOwThIkY9/v90Bs16iQPQduLWmPd\nbgO99fyt3W4nVu2RJAso7NXFXXfbb5ozJyav1kEeeL/f31Hf288sy/IUhiDBsCfGyFvUaE/9Hoyx\nFw9Q2MuN3xRtpp2Sbq8q6NXaLssaVHLVtPRss9n0fABYlqW9Xs9zO37r0PIzu9/D6xok+QEKe3np\n9/vaaDQ8BdBUYN1x0HE9dpN91ut138kxvGb0sduZt4hNkrVarW2Tb5j0u5DiAENhZ+GNAnLJJZfg\nySef3PH5sWPHjGuCP/3pT9/2d9BAk5WVFczMzGDfvn2eJVf37dtntM+NjQ089thjnt+tra15lrAd\nXqvED9Pcfnu5sNGm1157LQ4fPozNzU0sLy/jAx/4AMvsVhET9U/a6LEHgwQ8s2azuW1+S5N12u32\ntomVu92u9nq9SPnotOTNfqvxe7up1Wqh85ba5gUnny4PYCgmO5K+MZIUhKixa/crODs0i2F2fNuv\n/yJs/liv35aUDzAUkw12vfGVlRWoKlZWVjA/Pz/W0GvTwlhhqGrkadnc05p5TXOWBZ1OJ5f9FhU7\nBHfttdfuuD7W1ta2rrnFxUU0m03PbZiG1EgFMFH/pK1KHnsas7L7DefOyuwQjF9HaBb7z2IC6LKZ\nSSe3ff10Op0d348zypgUAzAUkw1RZ2V3xrztwTi2kLo/n0SLkvkzaWZfUybXnJ/41+t1xtJLDCjs\n2RDFY49aGGvSzC8lkja0MI/djrVH6SynuJcLUNizIUphLXZExhMs2lNZTvY15xWqazQakUN47FAt\nF6CwZ4c7K8adMmiailZVs9Mow4aoT+r5MbF6vb5tYFFS/R/ukCFTH4sNKOz5EOTBFzl2nFaIyCkO\nvV5vm3hPTU1tewhWsW/Br+ZKUczpsSdd1pkkDyjs+eAXTiiyN2qLb9LbdXrjXqLRarW0Vqvlfvxp\nn9u82+BnbtFOI8OLJAso7PlQZAEPu7nTEjbV8ePntlef9/lK27I6RhHZKv4Wdu2y/npxAAco5UOZ\nZnW3LAtLS0sAgJmZmVT2sbq6uu3fOIgIFhcXS3Vu49BsNnHxxRcbD84aZ45YVcX111+/7TO/85vm\neR8MBpiZmUGtVsPMzAznVE0KE/VP2qrssff7/dLEikVEZ2dnY8fXTbxLO2963HMSpWRx2c3rvLqr\nfdod0uOcD6+O0yxj7IzpRwcMxeRD1aZ2ExE988wzdwhz1gXC7Bu+6J3QaZk9EYnXpOHOz6OcG7+x\nFlllxTCmHx1Q2POhLN66qXU6nR2DhsYR2XE6S22hKXKHZFpmEuf2+z28JjJJ2jOO80CIEtNnGuYQ\nUNjzIW8ByMosy8olLFLmUIxX/RZTC/Ni/cJUzpm3kp4NyfmQdYu0+8HhJcymHjtDNk8BCns+VD19\njxbf4oaQTETMpDBYkuJo0t8Rtm+vUJ7XA8HvLdgvPFVlQGFPD7/XwjJ1nNLKYU7POigcERbWiOId\nm4ilSTjMZN9B+4vaWT4JXjwo7MkR9MrpvKCyjv3mldfdbrcr04FpdwLn3Q4/c86EFeTdhgl30LXi\nvM7d+7DXM32QRN130AMkzv1kWVZ6QlAAQGFPBlOvIesBNPV6XWdnZzMXmnq9rv1+v9BiaGp5PZCT\nNNNQS9CIaHuZsIe1U+SjLGvSt+DlbY9zP1U1NAMKezJEuemzEog8PXWTKdho2ZkzgySogzSsZMQ4\nHbtJmTssNO41Zl+vvV5vK0Rar9efGnF7++2qtZpqibx8ZCXsAE4DcDOAuwF8DcAlYeuUSdijiGin\n0yl0wacwC+ofiPMqTkvfnGIY5LWnUQsoDXM+iJII9/k9sHoXX6z6ileonnwyhd1zA8A/A/DTo//v\nBvBNAGcGrVMmYY/qNRSxsFWYCDvjnL7ezZjnhZaOtVqtbbNveZmdPZJ3W02s0Whk8vZwkYjq6aer\nXn45hd1og8BnAPxc0DJFFXavHvqqDGP388btmHmcc1WF80KrlnWBoax52N7RMlOAfhfQd550kt75\npjdR2EM3BswAWAVwgsd38wAOAjg4PT2d/hmIgN9rnzNGV4XQg58Qx00Tm9Th/WW0SUnDbQH6spHt\nPe44fbmIXgfoOqDPGS3zx4DeOPr/f2o09PGpqRRUJR2QtbADmAJwG4B/FbZskTz2MM+z2+2W5jU2\nyOw3EL8b3KQ+R1D+vv25ZVkU+4Jald6wOp1OqLPV6/X0g69/vW4AetHoszMAPQLoC0Z/XwHoWq2W\nssokB7IUdgBNADcA+PcmyxdJ2MNEW0Qq4a2bTD8XNDAlSo6zyXmlZWtFz9e37Qp4h1F+Ica2ngfo\nQ4B++pRTthyazwP6Qdf+HgAy05txQYadpwLgfwB4v+k6RRL2MNGumscelM/s/Nt0dh2/5csgIpNk\ns7OzRtdx3iGbKzAU45e57ISI25kC9GuA/gOgxzca2u/3df8znqGPAHo6oCeO7A8AfbBWU33oIdUf\n/zgPCYoEMhT2V41O5l0A7hjZvqB1iiTsQRe7M1WsKq+xXtUa/R5uzvCMyVuLPeqvSueLlq1dgaEH\nHXW9CwA9BOjG6N9bAL0f0GeOvu92u3rrhRd6vg1s2e//fk4qZA44QMmMsKp4zuW8vJmXAPpRQO8Z\nXVQfLcDNEWbNZnPbjPd+yzkHv5i+tbBDlRZkXfgL617EE/YLAH3Mta1NQN/rWk7vu0//5j3v0bee\nfLKeDegbTzhB//y44/RhQN968sn6qfe/Pw8JigQo7OaYFD7yE6x/B+j/A/QAhilUHy3AzWN0gzm8\n8bA6I6rmoRWKOi3InFkrtjmzVq4A9BiG4n4U0NsBfVPINg/B+0Fxv2Mfr6jVVH/0o233c7vd3vYg\nKUMRMVDYkyMotCCO/9+K+ML+CkC/AujjgH4b0F9P+QZzeuN+ou0cnFSFfgZa8excYFvWyhyg7wb0\nbEDfAOjnMBTpIHHfgP9bwDa7+eYd1/MV2P6GYJIdliegsCfDp6680vdC2eu6wOIK++mAPgroxwF9\nDaC/jaHX8m9TvKGcxaNMUiCrkBlEK5bZWStXhSz3JQw9d7/vD8H7/jzkWMZdesFvWyYzVeUJKOzj\n0+/39WnHHx/46ui8KOIK+zWAfgPQuuOzDwK6mtINFaVT2MbEY3d3ytJofubMWmmELHsZhl55zed7\nrxj7Y6PP7WXst8+wa96efL2o1SFBYY+OycTA7ldHp/kJe9d10TltL6ArGKZdOdd59ej7F3hsbxwT\nEe31ekaTgtTr9W3nxuuGsAeKmJRypdFs+yS2Z60E2aUIFnZgZ1bMBa7v45RoLmLMHRR2M6L82GGv\njn7CHtRhdCaGAr7ftc6e0ee/lMJNFWW+Uq9z5Rxhaos689ZpfuYW3esAfQLQVxqu/yVAb0ugHfa1\nG3WdIgEKezhR8q1NXh1NQzFOr/9ZGAr4+a5l6qPP35njDel3UVeldg4tffNLRbwJO52d3YDeDOiv\nADoL6C8C+tcY3itvSKg9UQdgFS3mDgp7OH6eutdrncmro4mwu73+Igu710z2/X6fok4ztkPwD0O6\nbS+gHwb0Wxi+zT4G6N8Dek5CbYlz3dJjL5GwB4VfvDyMJzDMUgl7dQwTdi+vvz3ahztmn1YoJup8\npfacmzZMe6RFMb9UxI0CtC3MosTYTScBHxdQ2L0JC78c8rkQH4H3q+MeQN88snsxfMV8M6BXwtzr\nXwH0fa7PXjXabxKdp/brZ9wa851OJ1Z8kkY7BO/76VAB2ua2Vqu1rd8oiqgHzTebJKCwexPmcRoP\ndsDw1XGv4bJBXv81gH4d23v9/wxDwU/qovUSd/tcULBpaZlJKmIW5jerWRLpjSYjt5MCFHZvgkSs\nXq8n4mH4bcPP67cHKA0wHHH3mxgOp05rgJK73G5Z5sOkldPCUhGzsE6n4/uW6tWXlISmpNHxCgq7\nN35PV3uKuMUzzxzbw4jq9QNDT/4WDEsKHEL6JQVss18ZmYNOm2Rz9yUloSn02DMkKFWv2Wxqr9cb\n28M4hPG9/iyNg4totKHFCckwxp6TsGcdTy5KXDGKMc5OK5o1Gg1tNpuZ79dLlMOyXpgVk7Gw5zXp\nQxHiilHMz2PftWtX7m2jldPGdRbsN+g8Umydnaq9Xi8zjzwMUNiHMO96PPPLJqDRgszus+p0OmNt\nx7KsWPdwkp6+yQxjWQFDYW+gwgwGA6ysrOTdjFKzubmZdxNICdnY2MCFF1449nbW1tawtrYWeb1j\nx46NvW+boZ7uZHV1NbF9JE0t7wakxWAwwPz8fN7NqCyWZaFer+fdDEJyY3p6Ou8m+FJZYV9YWMD6\n+nrezagk3W4Xhw8fxq5du/JuCiG50G63sbi4mHczfKmssBf5NaksNBrekbp9+/YBAI4cOZJlcwhJ\njShvn5ZlYWlpCXNzcym2aDxKI+yDwQAzMzOo1WqYmZnBYDAIXL7Ir0lZUavF/3kty8KJJ57o+d11\n110Xev4JKSoisuOzjY0Nz8+9ePzxx7f9HVWbMsGkhzVpi5oVE2cAQF5pjkWzqFktlmXp7Oxs5LrV\nNFqZbNxsL+ecwVmmQqJK6Y4mQ3a9Bgi4Z/zJ+2Iqg3HeUlrVLYnBeHYdmCzLCahWTNjDiuyEPTXp\nvdNotCTNFu4sC4CpVkzYw56KYYW9OEiJRptMc761x/XU3es5nUZ67GMIe5hHHvSD0VOn0SbT3OIa\ndxtBdWD8JnHv9XqRNM4UZCnsAM4B8A0A9wL4nbDl45QUCDq5YR45h8XTaNWzoP4grzK8Ubdv0gla\nWY8dQB3AtwA8B0ALwJ0AzgxaJ+laMf1+P5cqcDQaLT+zLCswKcI9gYZJAoUdsnE7j36OZWVj7ABe\nDuAGx9+XA7g8aJ0khZ0xdBptsi0sFOtMojDJ+vLSGL9QcJU99l8C8GHH328HcFXQOkkJO7NdaDSa\nqdkleIMcwVqttkNngsQ7Sh57EjXbUTRhBzAP4CCAg9PT05EPyAt66skac9hpVTdbdMOWixJuMRHs\npAYyYRJCMZz1J3njOaVV3UxHVScZbkkqZANDYU+iVsytAJ4nIs8WkRaAtwH4bALbDYX1YJJneO0Q\nUl02NjaMlltfX8fCwgL27du3o45M1OqOfkUJUytWaKL+YQZgH4BvYpgdsxC2fJIxdhTAA6DRJsXK\n0KeV9FunO+NORCLnqZfRY4eqXq+qZ6jq6apa3CLFhJCx2L9/f2AVxLwnXxERHDhwAO12O7Ftumdj\nUlVcf/31kbaxuLi4o02p1nQ3Uf+kLSmPnZ2nNFp21ul0Au85u+NwdnY213baBQDT3EecPPVSZcXE\nsSw7T8vw6kijld2cQuc3zD4rM818GcfymMhaNeNQTF6EdZ52u13s378/o9YQMrmo6tYkE0tLS7m2\nxe70tCwrcLl6vY7Z2Vl0u91I2y/6tHgAUGqPPWiAUliqEo1Gq66JiJHXbuLdt1ottSxrrBBKUmAS\nQjGq20sK2PmpJoMLaDRaeaxWq0W6ly3LUlWz+jDdbtd3uVqtlquQu8GkCHsY9NhptGpYlDdwu7qj\nSdkR27t3j7xutVqFEnVVCvsWXj9s3KHznF6PRsvP/GqzBC2vOuzMDRptam/XeX+7K0MWBVDYn8Jv\nPlT3D9npdHx/fLvXP++LO+qNoJp/lgKNloQ5a7OYOllhy7Xbbe31eplOSD0OoLCbY/KKFzYNn22m\nk3o4Lzg7dmhXnwu7GE3ije4LkyGp9OxdgP4doD8E9GFA/wHQnytAu6pm9j2YVFVX5+xIQfsrEqCw\nm2FykZhMjN3pdELnQbTNZDhy0Pr2G4dfSElEtt4+7FfQoLcR2ni2AuiHAD0f0NcB+t8B3QD0DQVo\nW1XM9qyTdFBsgpykonntoLCb4Xeh1Ot13/SmsBFkfuIfpcaEiRfBEEsxzPL47IuA3lSAtpXZnFlu\nXuGSsX+3kBmYgOKFZEBhNyOtqa3GHT7s9XBwPxgYXsnGusDwVvGwvT7rXAXoPQVoe9mt2WzmnrRQ\npJAMKOxmFDm+1uv1djx4nB4Ec/SzsRagL3PZdYCuA/ocn3VuA/SzBWh7auek1drRH+TMJPG6dr2s\nVqsVfoIXd7kE+02iXq9HrvI4LqCwm5HUzCZpEPbQoceej52LYQz9Ip/v/zWG3vzZBWhrUtZoNHxH\nX7rfTp2xcGc4xa+fJywjLW+zw7JTU1Oe32cp7qCwm5NE1bU0MJmSi0XOsrXnAfoQhqEWr+9/GtDH\nAL2yAG1N0vxGYJpcg/ZAnyAPvihvn3HaUa/XM9MEUNjLj59HblnW1oPI2QFke0dFuUmqZlOAfg3D\ndMaGx/fPBvR7gH4G0FoB2pu02cP0Ta5Rt3U6Hd9BQqZT1aVtduZNnJh+VmASqjtWHa/i/K1WC488\n8ghWVlagqlhbW8Pjjz+Ofr+PJ598EqqKAwcOoNFo5NTq6nABgEMANkb//h2An8Bw9vYnXcueBOAG\nACsYzg25mV0zM2NtbW3HZ6ZTux05csR3SjrTqeqSwLIsdLtdiAgsy4JlWRARdLtdLC0t4eqrr8bU\n1FSkbeY9uYgnJuqftNFjN8cdJvLzJpydvVFG5tG87QIMQyrqsE1A3+uxbAfQWwH9JqB7CtD2NM1N\nmfp5gvrOnPdZnG1nFcIFQzHVJOjCq9VqOjs7y7h7AnYI20XdtvuxPTtmN6A3APoEhg8Dd/ZM3seR\npDlDMXmWxHaP1Lbb4b436vX61kjwoAyWKH1VjUbD9x7MIukCFPbq4PQmihKPrLptwFvY3bY35Ht7\nez0MvfoHAT0C6F2jz/I+TlNzVjrMu9PeT0Cd94llWTsmofZbz/QBVavVth4OeaVJg8JeDfK+iSbV\nDsFbqA/F3N57RnYeoK8FdBHDh8elBTjWMJuamopUd8jZue83anTcDJkwAY0ivFHCL/bDIa2BjWGA\nwl58TEoT0EPPx7xi7I+NPk9qH31A7yzAsYaZPeLZNAbtJW5eue5ujxoYvhn45YuH7cNJ0LpuooaU\nut0uPXYvo7CHD4yip56/XYChh74x+jdI1LvwD8fs9VnnA4B+vQDHmbSZiptfDfRxPfagqe68cs6j\n3mv2xBx5DGwEhb3YBD3x6amXz0zLDtQxzKI5B8OBTr9WgLYnbe5JKqIOAAzzoG1hjbu+F+74fFBW\nmf1QyWNgIyjsxSbIK/F6RaWVy7zKDpyM7Z787xWgnWmaZVm+8XVnhopXmMbPgzapkBp0b0UJlRSx\n3Ago7MXGz6vgqNHym1/ZgTqgL8EwNHMFoI8D+psFaG+aFtTJ6Dcnqbv2utck9W5MMsfCPP2w7Rah\n3Ago7MXG74LO+0akjWdhZQec9rsYhmqOL0C78zA7e8bruzDP2h06CasQKSI6OztbKJGOAyjsxcOv\nCp7zQsv7ZqONZ5/EcBDTMw2WPRfDkMzpBWh30Swo68W0s9M5WY7fvKZe92CRQRbCDuBPANwD4C4A\nnwLwNJN/GYuVAAAQmklEQVT1JlHYTeN1LAVQHnNnzVyH4QjUVxqu/z4MwzGT6rEHWZDHbpqe6Hw4\nmIY+846hh4GMhP3nATRG//8jAH9kst4kCrvpKye99nKYXy2Zm7AzO2Y3oP8I6K9iOCfq6wH9U0CP\nAvoHBTiWollYB6lpP5Tz3oqy/7Rz0ccBWYdiALwJwMBk2UkU9rCeeuerIOcyLb4dwnZRD7K9gC4B\n+g0Mywk8AOiXAJ0rwHEU1bzEPUp9GveYkChJCWmPHh0H5CDsfwXgwoDv5wEcBHBweno69ROQBVF6\nzKNekEWeUYbmX0tmowBtq4o5s1jC4urOuVHdk2BHHRMyER47gBsB/JOHne9YZgHDGLuY7LQKHnvU\nHNd+v79Vac7kgs77pqIF2yF4C/uhArStSmZXlAxyjGynKs5o7aLlqYeBrDx2AO8A8GUAbdN1qiDs\nUdO0GDuvlmVRS4Y2tKCEApMOUj+zHwjMitkp6ucAuBvASVHWq4KwR63uFvWioxXfotSSmUQzfUMd\nx5yOVJwqjUEUUfSRkbDfC+A+AHeM7BqT9aog7FE9doZXaEW2tK7PNNN33eIcxXnq9XqBwl3EcgKq\nGYZi4lgVhN3kh+cUdbSimt3Z6BzAk8Z+ugHTObotyr3i5UFHyX5pNps7Rqs679+4I2LTBhT29Al7\n4rOYF62I5iWKaYYKe71e6JB/u00mnZ9B4jruA8redl4TaYQBCnu+MKZOK7o5y+uaerpx4+adTmfL\nIw8a7el0lrxSfk3CIb1eL3ZoyRZueuwUdk8YU6eVwZrNpvHAn3a7rbOzs7H3ZYuySaeklziblOy1\niTtPsC3cjLFT2D2hx04ri9nC5xbSVqu1FYe3LMtoyrowM/F4g2LlcTxmL5EOi7Hb601kVkxcmwRh\nZ4ydVkazBdUpZFEG/oSFakxi1EFOUdwYt5dIF1G4wwCFPX+YFUPL26KGIoCdXnGUa9iyrMCHgInH\nHRTGdJbiLYMQJw0o7MWCk1PT8jD72ov69ui8bqOsZ9d38XoY+HWSukU6TqGvSQEU9uIRpTodjZaE\nOa89U8+7Xq9vrRdnmL77eneLd1jHZBQnKO8slawBhb24MGOGlpXFzVePc62aetAmqYTuh4LfPvPO\nK88aUNiLC+PutKzMK9MjzBt2CqzpgyBKzDvO4J+i5pVnDSjsxYXCTsvS7NK3NmEhQefApbAYe5w4\ndxyRLmpeedaAwl5cyh6KyaJqHy1Z86qrEua52wLvN+mLcyKMKHjt274nLMvaVsMmSl55GdMXowIK\ne3EpcweqPfqPOfrlsnq9rr1eb0v4TB/O7Xbbd2CS+00gCs63hiBHx9QrnxSPHhT24tLv90OLIjlt\ndna2MA+DXq/HUBJNgWQ6Lk2ua5M4+qTE4GEo7DWQzJmbm8Pu3buNl//yl7+M5z73uSm2KJxOp4N+\nv4+rr74aa2trubaFFIPp6emxt7G6uprqMibrVhEKe8YMBgPMzMxEEsf19XXcdNNNKbYqnCNHjuDC\nCy9Eo9HItR2kGLTbbSwuLgJ46pqu1WqYmZnBYDAw3o7Jw2GcZZJ4+JQSE7c+aZvUUAxHn5bf7Jht\n0DKNRiP3dqZpUQYbjXtPMMa+HTDGXjzC4ontdjvx+PXs7OxWrRBms4xvNkH1VyzL8u1cnpqaKnwf\nRVD73DHrJGLbzmyWoKyYKNthVgyFPTOCev+7o+nJkr7p3Z20ZU+1zNtsxtnGrl27cj+OIPMTa6/0\nxqLONFRVkQeFvXgEeTdlDdOIyEQ9LMJ+yyKa14QVfsva3rLf926xLGI2SpXDMqCwF4+gCy5roWBY\nJro5xcpvbs1Go1HIUItXLXL321yr1Yp0Lbbbbe31eoUT0SI+bJICFPZi4veKOElebxnNLVZ+4mGP\n1owyTiFtswXNfe05ByyFdYgGbbtoYY+ihoeSABT2clGmV/tJM/vtxunt+i1rx6HzGJnbaDR27Nc5\nz2iYZ93r9bZ1tPuVEnAfb9Ggx66gsBeEuDH2Xbt2RZ4hp8yWtyfcarUCRbvb7eb2kG42mzu8cPvv\noPaq+oeWwqyIYskYu4LCXiA4EUe5zRaPPNvgFFoTZ8H2uOM4B0UWy6KFh5ICFPbywnj75JplWWO9\nldhC3e/3jcTafhDEaWdVxLJMgLViyovfMGgRybgl43Pccceh3W4nuk3LshLfZt602230+30cPnwY\n1157LSzLirWd6elpDAYDzM/PY2NjI3SfdlmAer1uvA/LsnD48GHMzc3FaiPJABP1DzMAl2L4JN9j\nsjw99mD8YoRx46B5W6/XM+qIM7FWqxUpnbBIw/ud4YFOp7PVKWuX1HUTNW3SvkZMPXV3x6nJ+Sty\n+GUSQFahGACnAbgBwAoo7InhFyMsYwy+2+0m1sEbNdtkdnY29+O3z4HztzXp3PNarlarbT0Q7MwV\nZ0dpWEw9SJidDwX7YVPVWHVZyVLY/xeAFwFYprCnTxlHqCbVZxDn4ZDkQKywUZl+1mw2t9U/8fPE\nvTJM3HVU3PF3p1CHPfTr9TqFueRkIuwAzgfwgdH/A4UdwDyAgwAOTk9PZ3AKqkvYzV40S8JjzzKk\n4jeaMs6I0k6nY/z7hOWEh+VnJzETESk2iQk7gBsB/JOHnQ/gFgAnqoGwO40eezBRX3/dQl+kaets\nQQnqH7AsS2dnZ32FaWpqKrNjsgcYef0GUb31qOmrYTnhYSMq/fZFT706JCbsvisCPwnghyNBXwbw\nJIBVAM8MW5fC7k8SgytMU93SMHsaP6+HklcM1/7cS7Q6nU5udXS84t1RzqktpqYPA5PfOMxjr/LA\nHDIkdWHfsSF67ImQ1HDoOB6m2+yJq3u9nnGsul6vh9YiidLOvPoTnIIYt1/DJHzjdW783thMhJud\nndWGwl5S4hQw8ruZ/TzhKGYLdBJxfDsdz9nWtCshhpUACDt21fEykYI6XP06S/1SXe122G8OFO7J\nI3Nhj2IUdn9MPPYomRLOUEbcDJEkR8KmPaq2Xq/r1NTUNmGNm0dvP0zHabP91uPehl+IxO/3N12f\nVBsKe0kJe902DQv4hW6KWCs8bWs2m7HeOJLw2O3QlD3lm/2ZvX3TGYmi/MakulDYS0xQnNRUZPxC\nN3mLbFnM/TBN4k3DKyxkWuc9ym9MqoupsLNWTAGZm5vD8vIyNjc3sby8vK0mx+rqqtE2/OrNhJFH\nPRrLsmLXRhmXer2OXq+Hfr+PbrcLEUG328XS0tK2855EbZqjR4/i2LFj2z5bX1/H/v37MRgMAACL\ni4s79uX3m8T9jckEYKL+SRs99viYeHRB8dewGei9BuckGRcP6w9Iaj9xzF2xMEo20LjmPg/O32lq\nampHO5rN5o5+FGbCVB8wFFNNvGLsziHrdizX7yYPmuvSuYxTKExqkEQRTz8RCppSrl6vez4Ukiou\n5hbMKAXXknrwBeWju83+zZi7PllQ2CvMOHnOQesHkUTqpC2CfoS9jXg9FNKondM1LIHgbEdSndIm\n58HZzqBl6b1XDwr7BJLU4KYo247rlXoR9uDweygk9dBx7ifqcSQ1gXWUYxGR0GXpvVcLCvsEkubs\n7EkIZ9jbQ9g+/HL5owz1N91P2Da96qcnVcrBmYcf1k6TBy7TIqsDhX0CKaLHHhTyiRpGsTs30yxd\nbNeJCYux+53TpNoW5v3bD8ko85qS8kNhn0DS7EiLI1hhD5Q4DwsRMfZo45hlWVvtCxJ30xIPcT14\nd39CUN2dsEJp9NirA4V9Qkkz9W3cTls3RZy02y3YYemhJiWV46SPxvGymSFTfSjsJFPiPFCS6pC1\nzV1Z0inKziH9Qeb2bsPW8RNOdz0frxTUoDeCuF42c9qrDYWdpIapeIQtFxbeierRuwcYebUnaH9e\nIh03Q8bLS/fqcI1SIIwQCjtJhSi58lFz6t2ebVDqn58HHiaKJp60E5O3CnfYxG8de3amoDbRyyZB\nUNhJKphm3iSVoRPk0aaZBWRj0mns3l+Ql8+OTDIOFHaSCqa58knm1Pt5tEnn7Qd1DtsPEZOwSZCX\nz9RDMg4UdpIKWXvsSbTFhCTLMQSV+aXHTsaBwk5SIekYexZtMSHpBxE7RUkaUNhJaiSVFZNlW8JI\noxwDO0VJ0pgKuwyXzZazzjpLDx48mPl+STUYDAZYWFjA6uoqpqensbi4uG1SjDjMzMxgZWVlx+fd\nbhfLy8tjbZuQpBCR21T1rLDlOIMSKRWDwQDz8/NYWVmBqmJlZQXz8/NbMxDFxWvmona7jcXFxbG2\nS0geUNhJqVhYWMD6+vq2z9bX17GwsABgKPwzMzOo1WqYmZkxFvy5uTksLS0FTo9HSFlgKIaUilqt\nBq9rVkRw4MABzM/PbxP+drtNgSaVgaEYUkn8JnCenp4O9eYJmRQo7KRUBMXCV1dXPdfx+5yQqkJh\nJ6UiKBYe5M0TMklQ2EnpmJubw/LyMjY3N7G8vLwVP2dmCyFDxhZ2Efl1EblHRL4mIn+cRKMIiQMz\nWwgZMlZWjIi8BsACgHNV9QkReYaq/jBsPWbFEEJIdLLKiukB+ENVfQIATESdEEJIuowr7GcAeLWI\n3CIiXxCRn/FbUETmReSgiBx84IEHxtwtIYQQPxphC4jIjQCe6fHVwmj9pwP4WQA/A+A6EXmOesR3\nVHUJwBIwDMWM02hCCCH+hAq7qr7O7zsR6QH4y5GQ/6OIbALYA4AuOSGE5MS4oZhPA3gNAIjIGQBa\nAA6P2yhCCCHxGTcrpgXgWgAvBnAUwGWqepPBeg8A2FkjNX32YHIfPDz2yYTHXi26qnpS2EK5FAHL\nCxE5aJIqVEV47Dz2SWOSj50jTwkhpGJQ2AkhpGJMmrAv5d2AHOGxTyY89glkomLshBAyCUyax04I\nIZWHwk4IIRVjYoVdRC4VERWRPXm3JStE5E9GJZbvEpFPicjT8m5T2ojIOSLyDRG5V0R+J+/2ZIWI\nnCYiN4vI3aOS2pfk3aasEZG6iHxVRD6Xd1uyZiKFXUROA/DzACZtzrS/BfACVX0hgG8CuDzn9qSK\niNQBfBDA6wGcCeACETkz31ZlxpMALlXVMzGs5fSrE3TsNpcA+HrejciDiRR2AFcC+C0AE9VzrKp/\no6pPjv78CoBT82xPBrwUwL2q+m1VPQrgLwCcn3ObMkFVv6eqt4/+/yiGAndKvq3KDhE5FcC5AD6c\nd1vyYOKEXUTOB/BdVb0z77bkzL8B8Pm8G5EypwC4z/H3dzBB4mYjIjMAfgrALfm2JFPej6Hztpl3\nQ/IgtLpjGQkpNfweDMMwlSTo2FX1M6NlFjB8VR9k2TaSPSIyBeCTAH5DVR/Juz1ZICLnAfihqt4m\nImfn3Z48qKSw+5UaFpGfBPBsAHeKCDAMRdwuIi9V1e9n2MTUCCqzDAAi8g4A5wGY9aqbXzG+C+A0\nx9+njj6bCESkiaGoD1T1L/NuT4a8EsAbRWQfgF0AThCRvqpemHO7MmOiByiJyDKAs1S1ahXgPBGR\ncwD8KYC9qlr5mvki0sCwk3gWQ0G/FcAvq+rXcm1YBsjQc/kYgAdV9Tfybk9ejDz2y1T1vLzbkiUT\nF2OfcK4CsBvA34rIHSJyTd4NSpNRR/GvAbgBw87D6yZB1Ee8EsDbAbx29FvfMfJgyQQw0R47IYRU\nEXrshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMSjshBBSMf4/1HOo5p+uW1YAAAAA\nSUVORK5CYII=\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAC3CAYAAAD3oFO8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmQXeV55/HfA4jFyAIJLQgEEmKRWAIiFhgHs4Ultkmw\nQ7m8ToaUE5OZSjLOjCsVT6pmxjVVmXFSWWpqqiYTMvaAPTF2cELhcpwhBisDGErQCAm0IIRAWAK0\ngQFhK2x6549uOfee99fdr+7St889309Vl3Qenb73vec55/Sr2+9zn0gpCQAAAGiiwwY9AAAAAGBQ\nmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxmAwXilF/GBEvjX39YUTE\noMeFQxcRV0XEqoh4NSK2DXo86FxE/G5ErI+IfRHxbET87qDHhM5ExL+NiGci4rWIeCEi/iwijhj0\nuNCZiDgyIjZFxI5BjwWdiYgvRsRbEfF6y9fSQY+rH5gMl7tZ0kckXSDpfEm/JOk3BjoidOrHkr4i\niYlT/YWkfylptqQPSPqtiPjEYIeEDn1b0kUppVmSztPovfbfDHZI6MLvStoz6EGga99MKc1s+Xpm\n0APqBybDFRHx8cr/gt6IiH+UdJOkP0kp7UgpPS/pjyX96iDHiomNl8uU0sMppa9JGsqLehhNkMs/\nSimtSSm9nVLaLOkuSZcOerwY3wS53JpSeungbpIOSDpjgEPFBCb4WamIOE3Sv5D0Xwc6SBSZKJdN\nwWS4IqX00/8FSTpJoxOm2yWdK2ldy67rxmKYpibIJWqmJJdjy5Yuk7RhAENEoYlyGRGfiojXJO3V\n6DvDfzG4kWIik1yT/13S70vaP6jxodwkufyliHg5IjZExL8e3Cj7i8nwOCLiMElfl/SPKaW/kDRT\n0qstu7wmaSbrhqc/k0vU1CS5/KJG72n/e6rHhUPncplS+vrYMomzJP1PSbsGOEQUqOYxIn5Z0uEp\npTsHPDQcInNN/rWksyXNk/RZSf8xIj45wCH2DZPh8f2BpHfrn9esvS5pVsu/Hyfp9ZRSmuqB4ZBV\nc4n6srmMiN/S6Nrh61NKbwxiYDhk416XKaUtGn2H/39M9aBwyH6ax4g4VtIfiXttXbVdkymljSml\nF1JK76SUHpT03yR9dJAD7BcqdY2xApxParSY462x8AaN/tru4bHtC8SvY6e9cXKJGhovlxHxGUlf\nkHR5SonK9RoovC6PkHT61I0Kh6qax4g4V9ISSfeP/dL0SEnHRcROSZeklLYNaqyYWOE1mTS6nn/o\n8M5wRURcqNH1Th9JKbVWwn5V0r+LiJMj4mRJn5d06wCGiELj5TIiDouIoyXNGN2MoyPiyEGNE5Ob\nIJeflvRfJF07rFXOw2aCXP56RMwf+/s5kv69pHsHM0pMZpw8rpd0iqQVY1+/rtGlLiskbR/EODG5\nCa7JD0fE7LGPlr1Y0uc0WqQ8dHhnOPdhjX5M0wMty4Hvl/QhSUslPTEW+1+iuGO6Gy+XfyhpVct+\n+yX9P0lXTuXgcEjGy+VySSdIeqQl/n9SSv9qykeIUuPlcqekP4iImRr9SK47JP2HgYwQJWweU0of\nPLgRES9LOpBS2jmA8aHceNfkKxr9GNKjJO2Q9KWU0m0DGWGfBUteAQAA0FQskwAAAEBjMRkGAABA\nYzEZBgAAQGMxGQYAAEBjdTUZjogPRMTmiHg6Ir7Qq0EBAAAAU6HjT5OIiMMlPSXpWo1+5MYjkj6Z\nUto43vfMnTs3LVmypKPnQ/ceffTRvSmleb14LHI5ONu2bdPevXt79sHn5HJweplL8jhY3F+HB7kc\nHqW57OZzhi+W9PTBD7qPiG9o9LPqxp0ML1myRCMjI108JboREc/16rHI5eCsXLmyp49HLgenl7kk\nj4PF/XV4kMvhUZrLbpZJnKz2jjI7xmLVgdwcESMRMbJnz57qP6NGyOXwIJfDgTwOD3I5PMhl/fS9\ngC6ldEtKaWVKaeW8eT35rQMGhFwOD3I5HMjj8CCXw4Nc1k83k+HnNdqD/KBFYzEAAACgFrqZDD8i\n6cyIOC0ijpT0CUnf7s2wAAAAgP7ruIAupfR2RPyWpLslHS7pKymlDT0bGQAAANBn3XyahFJK35X0\n3R6NBQAAAJhSdKADAABAYzEZBgAAQGN1tUwCAA5VaddLt5+LRZQ1cHP7lX4vyvJWmrNSpTkjj4em\nm5x0+vi9PDd6fV7U+fypUy6n4l7daS55ZxgAAACNxWQYAAAAjcVkGAAAAI3FZBgAAACNRQHdgPS6\nqASYrjotvDpw4EDRfu56OOww/p/fjV4Wy7k8Ok0oduq3bopTe/mcLuel13OJ0gIrdx+oy71hEIVx\nTje5rMa6KXx0eevlvaAeZwUAAADQB0yGAQAA0FhMhgEAANBYjVsz3M2aqrfffjuLvf7661lsx44d\nWWzfvn1t2yeddFK2z/z587PYUUcdlcX6vXamLqYil9u3b89ir776atv2okWLsn0WLFiQxYYtl51+\nGHuv1weXHq9OPyi+9DyrQzOIXjY8GVSDDad0XXKnBp3H6bIW2MXeeeedLPbmm29msVdeeSWL7dmz\nJ4u99NJLk47tXe96VxZz99y5c+cWfe8RR0zdVGi65NJdM2+99VYW+/GPf5zFXI5crDrvOfroo7N9\njjvuuCw2b968ov3c4x1++OFZrATvDAMAAKCxmAwDAACgsZgMAwAAoLG6WigTEdsk7ZP0jqS3U0or\nezEoAAAAYCr0YtX4VSmlvT14nIEpLRRwRVc//OEPs9gdd9yRxaqFAldeeWW2zzXXXJPFZsyYkcUG\nXcwxDFzxwAsvvJDFvvWtb2WxXbt2tW1fffXV2T4ul0ceeWTR2EoLx/qlm2YKLlYtsHHXkYu5whx3\nPbiYK0wsKf5x4y/9AHhXuFG6Xz90WuAoleWxmwYbpcfAHfuS886dO6XFqqWNGvp9TU5WQFV6n+hl\nAxV3nf7TP/1TFnNF5A8++GAWe/TRR7PYzp0727bfeOONbJ/jjz8+i5111llZ7Kqrrspi559/fhZz\nxVm91Ekue/lcpUWOe/fmU7knnngii/3gBz/IYlu3bs1i1aI6V6i4cOHCLHbBBRdkscsuuyyLuZzP\nnDkzi5VgmQQAAAAaq9vJcJJ0T0Q8GhE392JAAAAAwFTpdjL8/pTSCkkflPSbEXF5dYeIuDkiRiJi\nxH2mIOqDXA4PcjkcyOPwIJfDg1zWT1eT4ZTS82N/7pZ0p6SLzT63pJRWppRWug9SRn2Qy+FBLocD\neRwe5HJ4kMv66biALiKOlXRYSmnf2N+vk/SfezayHuhmUbr7XrfgfNOmTVlsw4YNkz6+6/TiijRw\naEoLQdzxf/rpp7PY+vXrJ3089/iuQGg6Fz62voaS4inJF9O4a6TasW/37t3ZPtWixPEe/9hjj81i\nxxxzTBZzRXUuJ9WCDvd97jldzHVDcgUjredBLwtnqo9XmkcXc9fHT37yk7Zt15mquk91TAe5Y+UK\nTCc7fhM9R5XrAOli7hwovcanUun9pOR4lf68c13GNm7cmMXuv//+LLZ69eos9swzz2Sx6nnlzkV3\nrmzevDmLufG6n7MXXXRRFuuXXhY+uv3c9bx///4s9uyzz2axe+65J4s9/PDDWcwV1b388stZrHo/\ncK/JFby5YjzXwfDGG2/MYsuWLctiJbr5NIkFku4cS+IRkr6eUvq/XTweAAAAMKU6ngynlJ6RlH/+\nBQAAAFAT/F4eAAAAjcVkGAAAAI3Viw50A9Hvji2u+MQtQnfFcq4r3amnnjrhtuSLOaZz0VWv9LqI\nqOTxXVHG448/nsVckcHixYvbthctWpTt4wo8uil46bfWY1RaLOc6Q/3oRz/KYtu2bWvbdsfZffyQ\nK55ynafe/e53F32vK4yq7ucqv+fPn5/F3LXqTHUuW+9b7hx3eXT3NZfH7du3t2274qfXXnsti7lj\nNWfOnCw2e/bsou8tKTR2RZXuOd35VNpJsN/3rV4p+fnmzgGX33/4h3/IYiMjI1nMFUC5c8rda6rF\nlS4f7t5TPT8l6aGHHspis2bNymLVe3o/dXNPKOku5wpb165dm8XuvPPOLOaK5aodASV/b3H312pn\nP/d9roPhli1bsti73vWuLLZgwYIs5jraleCdYQAAADQWk2EAAAA0FpNhAAAANBaTYQAAADRWLQro\n+l0s57gCOtdh5cknn8xirlPKe97znrZtV3TlFqC7wo06F9UNoujEPafL0WOPPZbF9u7dm8VWrFjR\ntn3yySdn+5R2zpouuWw9RqUFpa6IxR2vp556qm3bFZi689wVsy1dujSLuQI6V5jjCniqxSbuGncF\nN6Xn8VTmPKXUlid3DFyxiuvS5e5r69ata9t2XQNdkYu717kOdO7+516Dy1E1j66A1Z3Dbhwu5gz6\n2u3mZ1m1YM51kbv99tuz2AMPPJDFSq8ZVzReLbBy3+uKPqv3FMkX0D3//PNZzN3nXSfZfum0s5xU\nlss1a9Zk+9x2221ZzOXSFd+5ItPTTz89i7n7dbW7nOtQ6c49lzeX8+o9SZIuvfTSLFaCd4YBAADQ\nWEyGAQAA0FhMhgEAANBYTIYBAADQWLUooOs3t1DdLdp3HXncwnu3yP2ss85q2z7hhBOyfVynHRya\nkg49Ut4VTfKL8V3HnGoRl+ucVdIlazppLQYqLfpyx8YVY1WLIdy1ddppp2Wxiy++OIudeOKJWcwV\nXrlCDZeTasHfm2++me1z0kknZTGntNh1smPdjckKId3rc4Vw7lp4+umn27ZdkZorrKkWnErS3Llz\ns5jLoyvocbndvXt327Z7na64z3UX7KY4sl/cmEoLrFzRZLVzqiuwuvvuu7OYKzx2hVPnnXdeFnPX\nuLu2qo/nuhq682fVqlVZbMeOHVmseh5L/nyfSt3k8oknnmjb/upXv5rtc88992Sxffv2ZTHXve2S\nSy7JYmeffXYWW7JkSRarFki6TqPu5+d9992Xxdz3ug6xzz33XBYrUa+f2AAAAEAPMRkGAABAY006\nGY6Ir0TE7ohY3xKbExHfi4gtY3/m73MDAAAA01zJO8O3SvpAJfYFSfemlM6UdO/YNgAAAFArkxbQ\npZTui4gllfCHJV059vfbJP2jpN/r4bjalBYqdNrdrLTQZPXq1Vls586dWcwVBVSLgVyXo+ncoaxX\nOn093XSucwVb3//+97NYtQhH8oWO73//+9u2S3NZF6XFf66YwxWZvvjii23b1a5Eku8s564j17HK\nHWtXNOmKsV544YUsVuUKf5zS67ef58ZkxXnuvuYKTlw3r2rBpDsuF154YRZbvHhxFjv22GOzmFN6\nLlavXVfo5c6dZcuWZbFBdMmcTDddITdv3pzFvva1r7Vtu2K50vvhFVdckcV+4Rd+IYudccYZWcx1\nj6x273QFdO68cMVUroDu1VdfzWKdFl31isulu1a3bt2axb7+9a+3bd97773ZPq4zqLu/XnvttVns\n53/+57NY9QMBJN91sPqhAK4LqPs54rqUuvPRFf+67y3R6ZrhBSmlgz/hdkpa0OHjAAAAAAPTdQFd\nGv0v67j/lY6ImyNiJCJG3EdjoD7I5fAgl8OhNY/u3R/UB9fk8CCX9dPpZHhXRCyUpLE/8/evx6SU\nbkkprUwprXSfR4j6IJfDg1wOh9Y8us9eRX1wTQ4Pclk/nTbd+LakmyR9aezPu3o2ogFw63Vc84CR\nkZEs5ta7uLVR1Q+krq6LksrXEtZ5PWq/ubV1bs3Z/fffn8XcejuXy+p6KdcspddrRfud88ke310j\nL7/8chZza4ar67rch6y7tYjHHHNMFnPXTen64C1btmSx6gfwu8YM7rGc6bDmf7Ln279/fxZz66Zf\nf/31LFZdG++ujUWLFmUxt1bXNdhweXTXs1t3WD3H3Ll5yimnZDHXOGY6Km0M5dbIfvOb38xi1ZoJ\n91uF448/Pov94i/+Yhb7+Mc/nsWWL1+exdy15c7X6r3GNXdxuZwzZ04Wc2uL3X3erTHvl9LGUK4e\n6Y477shi1fXe7p1o9x/lG264IYt95CMfyWJufbBb6+1+DrrXVeXWLpfm0jUOcWvCS5R8tNrtkh6S\ntCwidkTEr2l0EnxtRGyRdM3YNgAAAFArJZ8m8clx/unqHo8FAAAAmFJ0oAMAAEBjMRkGAABAY3Va\nQFdbpYvXq4U1krRu3bos5hZ1X3DBBVmsuiC810VXFNWNcoVeroHAhg0bspgrzrrsssuyWLUAzJ0D\nw5Y3d424D0F3H3heLQQ555xzsn1ckYzjCp5cgaRrNLB+/fos9uSTT7Ztn3nmmdk+JUUg05G717nC\nOFds5ooGq+e9K2JyhTWuWM5xTQZcMYwrEqs2TXCFze7x3f1iOiot8r7zzjuz2KpVq7JY9Zp0jXCu\nv/76LPbpT386i5177rlZzBW/lhTLudhRRx2V7eOatrgCPXdPd/cQd270i3vNrhDMNUJxsWpTI3cN\nulx+9KMfzWJnn312FnPnhsulu99U93P3eff4bj/3c9Ydy07v17wzDAAAgMZiMgwAAIDGYjIMAACA\nxmIyDAAAgMZqXAGd4xbPP/jgg1nMdelZuHBhFrv88suzWLUIwC0Gd+pSYDUIpV2Z7rvvvizmOg4t\nWLAgi1177bVZrLq4f9hy6Y6rK0iqFm5IvptZtcDNFWe5a8sVUbhrdevWrVnsoYceymJr1qzJYtWC\nP3c9u4Ibd565wg1XwDNVXHGJKzZ0RWqu0K76eO71uvy4fLv9XOesTZs2ZTGX72rMFVOV5tHF3LF0\n10kvtT7+T37yk+zfq13kJOnv/u7vsti2bduyWPW8dIXCn/jEJ7KYK351x9rdE93xKrl3lhYou3PK\nnaOuoNO9hl5qfe1unO48//u///ss5s79ai6vuOKKbB/XJdAVPrrCRFfs75RcI6U/A93PG3ddukJN\n9xpK8M4wAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGqtxBXRuEb/rwPTAAw9kMbeA23Wbc12sSgpp\n6lJgNZ25Tj733ntvFnOFFZdcckkWO+OMM7JYSUFBnXPprhFX9OGKetx+1WKsp556KtvHdZlyRS0u\nv65ob8uWLVnMdZXcv39/27YrMPvRj36UxVxRWGnxSeu50c8iLHeOu3G7/VwBS7XjoCtIdI/vjoE7\npi6PrqPhE088kcWqxXcnnHBCto87X9143X6lHbF6JaXU9vPGnc/f+c53spjrrOkKB6udxq677rpJ\n95Gko48+Oot1022zpGuZK8xy548r6HWP7zq0nXrqqVmsV1JKba/BdQ50hY+PPPJIFnPX5fLly9u2\nP/jBD2b7uGI5d38t7Yxbkjcpv0ZcLnft2pXF3L3A3afcdb5o0aIsVoJ3hgEAANBYTIYBAADQWJNO\nhiPiKxGxOyLWt8S+GBHPR8Tasa8P9XeYAAAAQO+VvDN8q6QPmPifpZRWjH19t7fDAgAAAPpv0qqu\nlNJ9EbGk/0OZGm4Rtis6WL9+fRZzRXCu6GrOnDlZrLq4vM4FVtOFy+XmzZuz2GOPPZbFXKGA68I0\na9asLDbsuXTFES7mCsbmzp2bxaqFLS5HrvjFXW+uuKm0s5jbr1rQ4TqvuQ5trtDOnSuu4Kjfncsm\neh6Xs5kzZ2YxVzC2ffv2tu033ngj2+fxxx/PYu5ac4VArvjS5cwVT1W/1+XRFcu53LrXNdUd6A4c\nONB2jFatWpXt47qkutfjfh69973vbdt+3/vel+3jCs26udeVHq/qsXavaWRkJIs999xzWcydx2ed\ndVYWu+iii4rG1omUUtt57Ap5f/CDH2Qx15XzuOOOy2I/93M/N+G25O9N/SwAPajk/rpx48YstmPH\njizmiqyXLVuWxVx+S3RzNH47Ih4fW0Yxu4vHAQAAAAai08nwn0taKmmFpBcl/cl4O0bEzRExEhEj\nrvc86oNcDg9yORxa8+jeSUJ9tObSffwW6oPrsn46mgynlHallN5JKR2Q9JeSLp5g31tSSitTSivn\nzZvX6TgxDZDL4UEuh0NrHt0SFdRHay7d56eiPrgu66ejyXBELGzZ/GVJ+QJbAAAAYJqbtIAuIm6X\ndKWkuRGxQ9J/knRlRKyQlCRtk/QbfRxjx9yCfVe48f3vfz+LuWIL9w7a5ZdfnsVmzJhROkQUKu2K\n5jr5vPLKK1mMXB4aV7zgOi2uWLEii1VzVy3EknwuHVfU44pDSlW7T7rXWVpo4gr+3Pf2q+AyItoe\n2xWuuXepzjnnnCzmChW3bdvWtu0K2Vxhoet0dcwxx2Sx0kI+t6ynWjxVWqzl9ivNYz8LkN555522\nc9MVjLnj4F6P68i1cuXKtm13XnTTjcxxRYiuCLp6v3YdRN3PbFdod+KJJ2YxV/R+/vnnZ7FeOXDg\nQFunS1f46Irq3LFxnfLe8573tG27n22luSzlculi1YI517Vy9erVWcx1Gl28eHEWqxaCStJJJ52U\nxUqUfJrEJ034yx09GwAAADCN0IEOAAAAjcVkGAAAAI3FZBgAAACNNema4emqZNG+W4DuOl09/PDD\nWcwtLq8uVJekpUuXZrGp6OwyTEpy6Rbnu89vfOCBB4qes1pAIvnihGHrLlfCnb+uc5k792+44YYs\nVi18ePbZZ7N9XMGqK2RyBXSu8Mp1OnKFXM8880zbtis2WrBgQRZzH33ljtFkxVi9Pr9aH88Vn5UW\nFLn9nnzyybbt3bt3Z/u4e6477q6T1vHHH5/FXAc6VwRdzaMrhHTP6XLrxuGOZT/v82+99ZZ27dr1\n0+0tW7Zk+7iiU3e+LV++PItVr133+hx3Hy49h13e3Dl01113tW1/61vfyvbZunVrFnP3BnduX3/9\n9Vls/vz5WaxXDhw40FZY6orlXOGpy4nLZbWQ2Z37pTlyP4vdNe3OPVeo/tBDD7Vtf+Mb38j2Wbdu\nXRZz9wzXWe+KK67IYp0WVDNrAwAAQGMxGQYAAEBjMRkGAABAY027NcOlH+DtVNczufVma9euzWI7\nd+7MYm4dolt/5NYpYXydfhj+22+/ne2zcePGLPb8889nMZfLK6+8Mou5dUpN5NZCHn300VnMfbi7\nW6+1cOHCtu3WtZAHuTVzbv2jWw/nuPOg2mBDyu8Z7gPblyxZksVmz56dxVyDlqmuH2hdG+jG49bN\nuvPeraWtrld0TTfeeOONLObOHRdz6xrduVJduyzl+XDXvKsJqJ6b442t100LJlNt1OCaoDju+nBr\n2avnvVtj77jXXNJsQZI2bNiQxb7zne9ksWrjJLeu2J2zF198cRb72Mc+lsWWLVuWxUrXTHfiwIED\nbflzzUHcz0XXmMbNN6o5cddg6X3IrQ929Rw//OEPs9iqVauyWDW/mzZtyvZx56xbH3zjjTdmMXdN\nd9ooi3eGAQAA0FhMhgEAANBYTIYBAADQWEyGAQAA0FjTroCulFtwXo3t27cv2+exxx7LYq7QzhVW\n/OzP/mwWc0U+TWzU4HRTDFn9XlcUsHr16izmcumKot73vvdlsakukplOWl+nK7YojbnroVqc4opf\nWouFDnLFHK5Yx32vG5srkqk2WHDNNFzRmSvSmA7nSusY3Pnsjot7La6IrFq84woo3fVXcq+WfJGY\n28+dA9Vz6uSTT872qTZ/kXxxmTtGLrf9zHdEtOXFFaa6a81dMy+88EIWW7NmTdu2a4jhntPtV1rk\n6O7Xjz/+eBarFma6QjLXbOGzn/1sFrvwwguzmLv/9LvQdbJmOC6X7loqOdbuena5dD9TX3rppSzm\nPnTg0UcfzWKueVl1vK5YzuXyM5/5TBY799xzs5i7T3WaS94ZBgAAQGMxGQYAAEBjMRkGAABAY006\nGY6IUyJiVURsjIgNEfG5sficiPheRGwZ+zP/FHoAAABgGispoHtb0udTSmsi4t2SHo2I70n6VUn3\nppS+FBFfkPQFSb93KE/eTYGVHWilS9mLL76Y7eM6p7gF16effnoWc0UZU91hqimqRTJ79uzJ9nn2\n2WezmCtO+Jmf+ZksRi67V1poVL3OXQGLK/pwBR6ugMft54qx3HirRWGugK60W9p0U1r05e7DJbkt\nLVJ0RV0uZ65gyHUmdN0oq3k75ZRTsn1cwZ8rMpwOZsyYofnz5/90++yzz872efrpp7OYO14jIyNZ\nrHr8169fn+3jzhVXlO46EbpCr1deeaXoOZYuXdq27QqsbrrppixW7ZAo+eK7qS6UPuyww9o6IrpC\nTtdZznXGffDBB7NY9fpynf5c4ZrrhOfmTNu2bctiritgyQcRXH311dk+LpfufO93Lif96Z9SejGl\ntGbs7/skbZJ0sqQPS7ptbLfbJH2koxEAAAAAA3JIb4VFxBJJF0paLWlBSungfyN2Ssqb2Y9+z80R\nMRIRI+7dPdQHuRwe5HI4kMfh0ZrLl19+edDDQRfIZf0UT4YjYqakv5H0Oymltt/FpNHfs9k1Dyml\nW1JKK1NKK92vqVAf5HJ4kMvhQB6HR2su58yZM+jhoAvksn6KJsMRMUOjE+G/Sin97Vh4V0QsHPv3\nhZLyRSQAAADANDZpAV2Mrkb+sqRNKaU/bfmnb0u6SdKXxv68qy8jHIcr+qgu4HYLv91if7d4vbqI\nX8q7VUkUXR3Uy25zUp7L7du3Z/u4gozZs/MPNXGda1znqToURU2F0uNQmvPqNVLavc3F3nzzzaLn\ndB2dWotYDqoWYLjzwunmfJ9u51mnRXXuGJd2KnRFcO453XO4+/Cpp57atu26TrriPqe0Y14/83jE\nEUdo7ty5P92+/PLLs32eeeaZLLZu3bos5u6TjzzySNv2U089le3jiq5Kc+SutQsuuCCLnXnmmVms\n+lpdF7nWY3OQy29pkW8/HX744W1dLC+99NJsn82bN2exVatWZbG9e/dmsbvvvrttu5pbyRcBu8JW\nd126/Lqfs+edd14Wu+qqq9q2XQGd6/brzr1+57Lk0yQulfQrkp6IiIN9+X5fo5Pgv46IX5P0nKSP\n9WxUAAAAwBSYdDKcUnpA0njT73yaDwAAANQEv+MHAABAYzEZBgAAQGOVrBkeOLdo3y3+3r9/f9v2\njh07sn19fRF3AAAH/klEQVSqnc0ktXX6Oei0007LYt10oqq+hulWRDNILifVQqnnnnuu6PtOPPHE\nLOZy6YoCSpHLQ1PSuaw0H+4adEVvrQUrE31v9TxzHY163SmzrkryWHotuGInV8jsrmfXPbJ6L5g1\na1bR2NzPkdJ897trWWunxmuuuSbbx3VZdMWFTz75ZBYrOe9dPlyHxkWLFmUxVyznCseqhY/ueUsL\n46ZrMfthhx3Wdu9ZuXJlto8rcnSv2xXHVb/XnReuMM7dN9315grQXS4vu+yyLFbtBFnaRa6bXPat\nAx0AAAAwrJgMAwAAoLGYDAMAAKCxmAwDAACgsaa8gG6y4oTS7j8uVi2iaC1AOMgVX7hCAddNxT1e\naREJRVajOs2lK4hxhXFuv+XLl2ex6dqtaLoqzVuJXl8frgiq2sFwvOeoFnS4IrvSTmt1ve57WSBY\negzc8XOdCV0nKtfdrHrdu+u7m1xMdR4jou14uI56n/rUp7KYK85au3ZtFtu1a1fbtjsHXBHq4sWL\ns5i7v7qiutJrq3qsS499aSfFQeSy9XXOmzcv2+eGG27IYu64rl69Oott2bKlbfu1117L9nHH2XV+\nc4WPK1asyGILFizIYu5arRbHlRbGDeK+yTvDAAAAaCwmwwAAAGgsJsMAAABoLCbDAAAAaKwpL6Br\nXRjdTeGG61pS7b5z4403Zvt86EMfymKucMMVy1F01R/uGFaLZK677rpsn6uuuqrosdzCfpfzuhZA\nDUovC1a6KbxyhTmuU9YZZ5yRxaqduJYsWZLt47p6ufPH3ZOm2znV72K5bh6/NLeumKz6vK5IyeXR\n3dNL89hvrc/pzrc5c+Zksfe+971ZzBXVuaLTQx3TQS5vve4q1qluiu/6NQ6XS1es6PLmitmq3eVc\nbrvJWzcFxINABzoAAADgEDEZBgAAQGNNOhmOiFMiYlVEbIyIDRHxubH4FyPi+YhYO/aVrz8AAAAA\nprGSNcNvS/p8SmlNRLxb0qMR8b2xf/uzlNIf93JApWvs3Nq06vovt1bUrc3pxnRZJ1MXpceruq7K\nrbOainFgVL8/0N6tQXNr1Y499tgs5q5ztz7UrQeurpFzDR3c45c0CxgvNki9Xufb6XOWru1362Ld\nusZTTz21bds1Upo9e3bRc06Xtd+TPX7p87vXg6nVz1y6cxiHbtLJcErpRUkvjv19X0RskpS3cQMA\nAABq5pDWDEfEEkkXSjrYE/C3I+LxiPhKROT/7QYAAACmseLJcETMlPQ3kn4npfSapD+XtFTSCo2+\nc/wn43zfzRExEhEje/bs6cGQMSjkcniQy+FAHocHuRwe5LJ+iibDETFDoxPhv0op/a0kpZR2pZTe\nSSkdkPSXki5235tSuiWltDKltNJ99iPqg1wOD3I5HMjj8CCXw4Nc1s+ka4ZjdGX3lyVtSin9aUt8\n4dh6Ykn6ZUnrD/XJuylAmG6FKU1Xmo9eFl11Mw50r9NjXfp9pUV1rnGCa5pTcu51UyhV13Ov39df\nacwVB7nczpo1a9LHc+dOaayueQTQuZJPk7hU0q9IeiIi1o7Ffl/SJyNihaQkaZuk3+jLCAEAAIA+\nKfk0iQckuf8qf7f3wwEAAACmDh3oAAAA0FhMhgEAANBYJWuGgZ6hOAWl6tDRDZMrLZZzxWy9fE4A\nGA/vDAMAAKCxmAwDAACgsZgMAwAAoLGYDAMAAKCxwnVl6tuTReyR9NyUPWG7uZL2Dui5e6EX41+c\nUupJb0hy2ZVux9+zPEoDzWXd8yhNo1xyTXaNXI6qey75WfnPyGVhLqd0MjxIETGSUlo56HF0qu7j\n76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4upHD/LJAAAANBYTIYBAADQWE2aDN8y\n6AF0qe7j76W6H4u6j79XhuE4DMNr6IVhOA7D8Bp6oe7Hoe7j76W6H4spG39j1gwDAAAAVU16ZxgA\nAABoM5ST4Yj4SkTsjoj1LbE5EfG9iNgy9ufsQY5xIhFxSkSsioiNEbEhIj43Fq/Na+gF8jg8yOXw\nIJfDg1wOh7rnURp8LodyMizpVkkfqMS+IOnelNKZku4d256u3pb0+ZTSOZIukfSbEXGO6vUaeuFW\nkcdhcavI5bC4VeRyWNwqcjkMblW98ygNOpcppaH8krRE0vqW7c2SFo79faGkzYMe4yG8lrskXVvn\n10Aem51HcjlcX+RyeL7I5XB8DVMeB5HLYX1n2FmQUnpx7O87JS0Y5GBKRcQSSRdKWq2avoYeq+Ux\nII9WLY8DubRqeRzIpVXL40AuM7U9BoPIZZMmwz+VRv+LMe0/RiMiZkr6G0m/k1J6rfXf6vIa+qku\nx4A8Tq4ux4FcTq4ux4FcTq4ux4FcTqxOx2BQuWzSZHhXRCyUpLE/dw94PBOKiBkaPSH+KqX0t2Ph\nWr2GPqnVMSCPE6rVcSCXE6rVcSCXE6rVcSCX46rdMRhkLps0Gf62pJvG/n6TRtejTEsREZK+LGlT\nSulPW/6pNq+hj2pzDMjjpGpzHMjlpGpzHMjlpGpzHMjlhGp1DAaey0Evku7Hl6TbJb0o6S1JOyT9\nmqQTNFqJuEXSPZLmDHqcE4z//Rr9VcDjktaOfX2oTq+BPJJHcjmcX+RyeL7I5XB81T2P0yGXdKAD\nAABAYzVpmQQAAADQhskwAAAAGovJMAAAABqLyTAAAAAai8kwAAAAGovJMAAAABqLyTAAAAAai8kw\nAAAAGuv/A7waJ9gxyNuaAAAAAElFTkSuQmCC\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "z1 = z[:,0]\n", - "z2 = z[:,1]\n", - "\n", - "fig = plt.figure()\n", - "ax = fig.add_subplot(111)\n", - "ax.plot(z1,z2,'ko')\n", - "plt.title(\"latent space\")\n", - "\n", - "#np.where((z1>3) & (z2<2) & (z2>0))\n", - "#select the points from the latent space\n", - "a_vec = [2,5,7,789,25,9993]\n", - "for i in range(len(a_vec)):\n", - " ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro') \n", - " ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]), \n", - " xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)\n", - "\n", - "\n", - "f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6, sharex='col', sharey='row',figsize=(12,2.5))\n", - "for i in range(len(a_vec)):\n", - " eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)\n", - " eval('ax%d' %(i)).set_title('z%d'%i)\n", - "\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right." - ] - } - ], - "metadata": { - "anaconda-cloud": {}, - "kernelspec": { - "display_name": "Python [Root]", - "language": "python", - "name": "Python [Root]" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.12" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/include/dlpack b/include/dlpack new file mode 120000 index 000000000..e19164b88 --- /dev/null +++ b/include/dlpack @@ -0,0 +1 @@ +../3rdparty/dlpack/include/dlpack \ No newline at end of file diff --git a/include/dmlc b/include/dmlc new file mode 120000 index 000000000..869c40b0e --- /dev/null +++ b/include/dmlc @@ -0,0 +1 @@ +../3rdparty/dmlc-core/include/dmlc \ No newline at end of file diff --git a/include/mshadow b/include/mshadow new file mode 120000 index 000000000..0ff1a4b9e --- /dev/null +++ b/include/mshadow @@ -0,0 +1 @@ +../3rdparty/mshadow/mshadow \ No newline at end of file diff --git a/include/mxnet/base.h b/include/mxnet/base.h index 4b001d54a..3b226d757 100644 --- a/include/mxnet/base.h +++ b/include/mxnet/base.h @@ -102,9 +102,9 @@ /*! \brief major version */ #define MXNET_MAJOR 1 /*! \brief minor version */ -#define MXNET_MINOR 3 +#define MXNET_MINOR 5 /*! \brief patch version */ -#define MXNET_PATCH 1 +#define MXNET_PATCH 0 /*! \brief mxnet version */ #define MXNET_VERSION (MXNET_MAJOR*10000 + MXNET_MINOR*100 + MXNET_PATCH) /*! \brief helper for making version number */ @@ -226,11 +226,11 @@ struct Context { /*! * \brief get the free and total available memory on a GPU * \param dev the GPU number to query - * \param free_mem pointer to the integer holding free GPU memory - * \param total_mem pointer to the integer holding total GPU memory + * \param free_mem pointer to the uint64_t holding free GPU memory + * \param total_mem pointer to the uint64_t holding total GPU memory * \return No return value */ - inline static void GetGPUMemoryInformation(int dev, int *free, int *total); + inline static void GetGPUMemoryInformation(int dev, uint64_t *free, uint64_t *total); /*! * Create a pinned CPU context. * \param dev_id the device id for corresponding GPU. @@ -341,8 +341,8 @@ inline int32_t Context::GetGPUCount() { #endif } -inline void Context::GetGPUMemoryInformation(int dev, int *free_mem, - int *total_mem) { +inline void Context::GetGPUMemoryInformation(int dev, uint64_t *free_mem, + uint64_t *total_mem) { #if MXNET_USE_CUDA size_t memF, memT; @@ -361,8 +361,8 @@ inline void Context::GetGPUMemoryInformation(int dev, int *free_mem, e = cudaSetDevice(curDevice); CHECK_EQ(e, cudaSuccess) << " CUDA: " << cudaGetErrorString(e); - *free_mem = static_cast(memF); - *total_mem = static_cast(memT); + *free_mem = static_cast(memF); + *total_mem = static_cast(memT); #else LOG(FATAL) diff --git a/include/mxnet/c_api.h b/include/mxnet/c_api.h index 13dea9492..ce773b8ad 100644 --- a/include/mxnet/c_api.h +++ b/include/mxnet/c_api.h @@ -441,6 +441,7 @@ MXNET_DLL int MXGetGPUCount(int* out); /*! * \brief get the free and total available memory on a GPU + * Note: Deprecated, use MXGetGPUMemoryInformation64 instead. * \param dev the GPU number to query * \param free_mem pointer to the integer holding free GPU memory * \param total_mem pointer to the integer holding total GPU memory @@ -448,6 +449,15 @@ MXNET_DLL int MXGetGPUCount(int* out); */ MXNET_DLL int MXGetGPUMemoryInformation(int dev, int *free_mem, int *total_mem); +/*! + * \brief get the free and total available memory on a GPU + * \param dev the GPU number to query + * \param free_mem pointer to the uint64_t holding free GPU memory + * \param total_mem pointer to the uint64_t holding total GPU memory + * \return 0 when success, -1 when failure happens + */ +MXNET_DLL int MXGetGPUMemoryInformation64(int dev, uint64_t *free_mem, uint64_t *total_mem); + /*! * \brief get the MXNet library version as an integer * \param pointer to the integer holding the version number diff --git a/include/mxnet/c_api_error.h b/include/mxnet/c_api_error.h new file mode 100644 index 000000000..0c6ea03fa --- /dev/null +++ b/include/mxnet/c_api_error.h @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2018 by Contributors + * \file c_api_error.h + * \brief Error handling for C API. + */ +#ifndef MXNET_C_API_ERROR_H_ +#define MXNET_C_API_ERROR_H_ + +/*! + * \brief Macros to guard beginning and end section of all functions + * every function starts with API_BEGIN() + * and finishes with API_END() or API_END_HANDLE_ERROR() + * The finally clause contains procedure to cleanup states when an error happens. + */ +#define MX_API_BEGIN() try { on_enter_api(__FUNCTION__); +#define MX_API_END() } catch(dmlc::Error &_except_) { on_exit_api(); return MXAPIHandleException(_except_); } on_exit_api(); return 0; // NOLINT(*) +#define MX_API_END_HANDLE_ERROR(Finalize) } catch(dmlc::Error &_except_) { Finalize; on_exit_api(); return MXAPIHandleException(_except_); } on_exit_api(); return 0; // NOLINT(*) +/*! + * \brief Set the last error message needed by C API + * \param msg The error message to set. + */ +void MXAPISetLastError(const char* msg); +/*! + * \brief handle exception throwed out + * \param e the exception + * \return the return value of API after exception is handled + */ +inline int MXAPIHandleException(const dmlc::Error &e) { + MXAPISetLastError(e.what()); + return -1; +} + +namespace mxnet { +extern void on_enter_api(const char *function); +extern void on_exit_api(); +} +#endif // MXNET_C_API_ERROR_H_ diff --git a/include/mxnet/engine.h b/include/mxnet/engine.h index 11e64edfc..e02b995d6 100644 --- a/include/mxnet/engine.h +++ b/include/mxnet/engine.h @@ -25,7 +25,6 @@ #ifndef MXNET_ENGINE_H_ #define MXNET_ENGINE_H_ -#include #if DMLC_USE_CXX11 #include #include diff --git a/include/mxnet/ndarray.h b/include/mxnet/ndarray.h index e877d35db..4ba13ca64 100644 --- a/include/mxnet/ndarray.h +++ b/include/mxnet/ndarray.h @@ -103,7 +103,18 @@ class NDArray { bool delay_alloc = true, int dtype = mshadow::default_type_flag, std::vector aux_types = {}, std::vector aux_shapes = {}, TShape storage_shape = TShape(mshadow::Shape1(0))); - + /*! + * \brief constructs a new dynamic NDArray whose shape is unknown, + * hence the NDArray is inherently lazily created + * \param ctx context of NDArray + * \param dtype data type of this ndarray + */ + explicit NDArray(Context ctx, int dtype = mshadow::default_type_flag) { + ptr_ = std::make_shared(TShape(mshadow::Shape1(0)), ctx, true, dtype); + dtype_ = dtype; + storage_type_ = kDefaultStorage; + entry_ = {nullptr, 0, 0}; + } /*! * \brief constructing a static NDArray that shares data with TBlob * Use with caution: allocate ONLY ONE NDArray for each TBlob, @@ -157,7 +168,20 @@ class NDArray { : ptr_(std::make_shared(stype, data, aux_data, dev_id)), shape_(shape), dtype_(data.type_flag_), storage_type_(stype), entry_({nullptr, 0, 0}) { } - + /*! + * \brief initialize the NDArray, assuming it is not assigned a meaningful shape before + * \param shape the shape of the NDArray + */ + void Init(const TShape &shape) { + ptr_->Init(shape, this->dtype_); + this->shape_ = shape; + } + /*! + * \brief set the correct shape of NDArray directly from the storage_shape of its own chunk. + */ + void SetShapeFromChunk() { + shape_ = ptr_->storage_shape; + } /* * This indicates whether an array is a view of another array (created by * reshape or slice). If an array is a view and the the data is stored in @@ -960,7 +984,13 @@ class NDArray { #endif } } - + /*! \brief initialize the shape and dtype, assuming it is not initialized before. */ + void Init(const TShape &shape, int dtype) { + auto size = shape.Size(); + storage_shape = shape; + shandle.size = size * mshadow::mshadow_sizeof(dtype); + this->CheckAndAlloc(); + } inline void CheckAndAlloc(const TShape &shape, const std::vector &aux_shapes, int dtype) { // calculate size, perform allocation diff --git a/include/mxnet/random_generator.h b/include/mxnet/random_generator.h index 6e37efd40..e7b419309 100644 --- a/include/mxnet/random_generator.h +++ b/include/mxnet/random_generator.h @@ -31,6 +31,7 @@ #if MXNET_USE_CUDA #include +#include #endif // MXNET_USE_CUDA namespace mxnet { @@ -54,7 +55,6 @@ class RandGenerator { public: typedef typename std::conditional::value, DType, double>::type FType; - explicit Impl(RandGenerator *gen, int state_idx) : engine_(gen->states_ + state_idx) {} @@ -63,6 +63,10 @@ class RandGenerator { MSHADOW_XINLINE int rand() { return engine_->operator()(); } + MSHADOW_XINLINE int64_t rand_int64() { + return static_cast(engine_->operator()() << 31) + engine_->operator()(); + } + MSHADOW_XINLINE FType uniform() { typedef typename std::conditional::value, std::uniform_int_distribution, @@ -78,7 +82,7 @@ class RandGenerator { private: std::mt19937 *engine_; - }; + }; // class RandGenerator::Impl static void AllocState(RandGenerator *inst) { inst->states_ = new std::mt19937[kNumRandomStates]; @@ -137,6 +141,10 @@ class RandGenerator { return curand(&state_); } + MSHADOW_FORCE_INLINE __device__ int64_t rand_int64() { + return static_cast(curand(&state_) << 31) + curand(&state_); + } + MSHADOW_FORCE_INLINE __device__ float uniform() { return static_cast(1.0) - curand_uniform(&state_); } @@ -189,6 +197,10 @@ class RandGenerator { return curand(&state_); } + MSHADOW_FORCE_INLINE __device__ int64_t rand_int64() { + return static_cast(curand(&state_) << 31) + curand(&state_); + } + MSHADOW_FORCE_INLINE __device__ double uniform() { return static_cast(1.0) - curand_uniform_double(&state_); } diff --git a/include/nnvm b/include/nnvm new file mode 120000 index 000000000..779dd4459 --- /dev/null +++ b/include/nnvm @@ -0,0 +1 @@ +../3rdparty/tvm/nnvm/include/nnvm \ No newline at end of file diff --git a/julia/.gitignore b/julia/.gitignore index d6791c849..3687ed485 100644 --- a/julia/.gitignore +++ b/julia/.gitignore @@ -7,7 +7,4 @@ data deps/src deps/usr deps/deps.jl -docs/_build -docs/build/ -docs/site/ .vscode diff --git a/julia/README.md b/julia/README.md index a4299575f..91a398146 100644 --- a/julia/README.md +++ b/julia/README.md @@ -3,7 +3,7 @@ [![MXNet](http://pkg.julialang.org/badges/MXNet_0.6.svg)](http://pkg.julialang.org/?pkg=MXNet) -MXNet.jl is the [dmlc/mxnet](https://github.com/apache/incubator-mxnet) [Julia](http://julialang.org/) package. MXNet.jl brings flexible and efficient GPU computing and state-of-art deep learning to Julia. Some highlight of its features include: +MXNet.jl is the [Apache MXNet](https://github.com/apache/incubator-mxnet) [Julia](http://julialang.org/) package. MXNet.jl brings flexible and efficient GPU computing and state-of-art deep learning to Julia. Some highlight of its features include: * Efficient tensor/matrix computation across multiple devices, including multiple CPUs, GPUs and distributed server nodes. * Flexible symbolic manipulation to composite and construction of state-of-the-art deep learning models. @@ -50,7 +50,7 @@ labels = reduce( labels .= labels .+ 1 # Now we use compute the accuracy -pred = map(i -> indmax(probs[1:10, i]), 1:size(probs, 2)) +pred = map(i -> argmax(probs[1:10, i]), 1:size(probs, 2)) correct = sum(pred .== labels) accuracy = 100correct/length(labels) @printf "Accuracy on eval set: %.2f%%\n" accuracy diff --git a/julia/REQUIRE b/julia/REQUIRE index 5a76dc543..b53f0c3cc 100644 --- a/julia/REQUIRE +++ b/julia/REQUIRE @@ -3,5 +3,4 @@ Formatting BinDeps JSON MacroTools -TakingBroadcastSeriously Reexport diff --git a/julia/deps/build.jl b/julia/deps/build.jl index bdc33be8c..7a37803f3 100644 --- a/julia/deps/build.jl +++ b/julia/deps/build.jl @@ -15,7 +15,9 @@ # specific language governing permissions and limitations # under the License. -import JSON +using JSON +using Libdl +using LinearAlgebra ################################################################################ # First try to detect and load existing libmxnet @@ -26,17 +28,17 @@ curr_win = "20180211" # v1.1.0 if haskey(ENV, "MXNET_HOME") MXNET_HOME = ENV["MXNET_HOME"] - info("MXNET_HOME environment detected: $MXNET_HOME") - info("Trying to load existing libmxnet...") + @info("MXNET_HOME environment detected: $MXNET_HOME") + @info("Trying to load existing libmxnet...") # In case of macOS, if user build libmxnet from source and set the MXNET_HOME, # the output is still named as `libmxnet.so`. lib = Libdl.find_library(["libmxnet.$(Libdl.dlext)", "libmxnet.so"], [joinpath(MXNET_HOME, "lib"), MXNET_HOME]) if !isempty(lib) - info("Existing libmxnet detected at $lib, skip building...") + @info("Existing libmxnet detected at $lib, skip building...") libmxnet_detected = true else - info("Failed to load existing libmxnet, trying to build from source...") + @info("Failed to load existing libmxnet, trying to build from source...") end end @@ -44,33 +46,35 @@ end CUDAPATHS = String[] if haskey(ENV, "CUDA_HOME") push!(CUDAPATHS, joinpath(ENV["CUDA_HOME"], "lib64")) -elseif is_linux() +elseif Sys.islinux() append!(CUDAPATHS, ["/opt/cuda/lib64", "/usr/local/cuda/lib64"]) end -if is_unix() +if Sys.isunix() try - push!(CUDAPATHS, replace(strip(readstring(`which nvcc`)), "bin/nvcc", "lib64")) + push!(CUDAPATHS, replace(strip(read(`which nvcc`, String)), "bin/nvcc", "lib64")) + catch end end HAS_CUDA = false HAS_CUDNN = false let cudalib = Libdl.find_library(["libcuda", "nvcuda.dll"], CUDAPATHS) - HAS_CUDA = !isempty(cudalib) && Libdl.dlopen_e(cudalib) != C_NULL + global HAS_CUDA = !isempty(cudalib) && Libdl.dlopen_e(cudalib) != C_NULL end -if !HAS_CUDA && is_windows() +if !HAS_CUDA && Sys.iswindows() # TODO: this needs to be improved. try run(`nvcc --version`) - HAS_CUDA = true + global HAS_CUDA = true + catch end end if HAS_CUDA # then check cudnn let cudnnlib = Libdl.find_library("libcudnn", CUDAPATHS) - HAS_CUDNN = !isempty(cudnnlib) && Libdl.dlopen_e(cudnnlib) != C_NULL + global HAS_CUDNN = !isempty(cudnnlib) && Libdl.dlopen_e(cudnnlib) != C_NULL if HAS_CUDNN && !haskey(ENV, "CUDA_HOME") # inference `CUDA_HOME` ENV["CUDA_HOME"] = dirname(dirname(cudnnlib)) end @@ -78,13 +82,13 @@ if HAS_CUDA # then check cudnn end if HAS_CUDA - info("Found a CUDA installation.") + @info("Found a CUDA installation.") if HAS_CUDNN - info("Found a CuDNN installation.") + @info("Found a CuDNN installation.") end - info("CUDA_HOME -> $(get(ENV, "CUDA_HOME", nothing))") + @info("CUDA_HOME -> $(get(ENV, "CUDA_HOME", nothing))") else - info("Did not find a CUDA installation, using CPU-only version of MXNet.") + @info("Did not find a CUDA installation, using CPU-only version of MXNet.") end # propagate more build flags from ENV @@ -98,26 +102,26 @@ function get_cpucore() if haskey(ENV, "TRAVIS") # on travis-ci 2 else - min(Sys.CPU_CORES, 32) + min(Sys.CPU_THREADS, 32) end end using BinDeps @BinDeps.setup if !libmxnet_detected - if is_windows() + if Sys.iswindows() if Sys.ARCH != :x86_64 - info("Prebuilt windows binaries are only available on 64bit. You will have to built MXNet yourself.") + @info("Prebuilt windows binaries are only available on 64bit. You will have to built MXNet yourself.") return end - info("Downloading pre-built packages for Windows.") + @info("Downloading pre-built packages for Windows.") base_url = "https://github.com/yajiedesign/mxnet/releases/download/weekly_binary_build_v2/prebuildbase_win10_x64_vc14_v2.7z" if libmxnet_curr_ver == "master" # download_cmd uses powershell 2, but we need powershell 3 to do this run(`powershell -NoProfile -Command Invoke-WebRequest -Uri "https://api.github.com/repos/yajiedesign/mxnet/releases/latest" -OutFile "mxnet.json"`) curr_win = JSON.parsefile("mxnet.json")["tag_name"] - info("Can't use MXNet master on Windows, using latest binaries from $curr_win.") + @info("Can't use MXNet master on Windows, using latest binaries from $curr_win.") end # TODO: Get url from JSON. name = "mxnet_x64_vc14_$(HAS_CUDA ? "gpu" : "cpu").7z" @@ -144,8 +148,7 @@ if !libmxnet_detected ################################################################################ blas_path = Libdl.dlpath(Libdl.dlopen(Base.libblas_name)) - - blas_vendor = Base.BLAS.vendor() + blas_vendor = LinearAlgebra.BLAS.vendor() ilp64 = "" if blas_vendor == :openblas64 @@ -154,18 +157,18 @@ if !libmxnet_detected FORCE_LAPACK = false if blas_vendor == :unknown - info("Julia is built with an unkown blas library ($blas_path).") - info("Attempting build without reusing the blas library") + @info("Julia is built with an unkown blas library ($blas_path).") + @info("Attempting build without reusing the blas library") USE_JULIA_BLAS = false elseif !(blas_vendor in (:openblas, :openblas64)) - info("Unsure if we can build against $blas_vendor.") - info("Attempting build anyway.") + @info("Unsure if we can build against $blas_vendor.") + @info("Attempting build anyway.") USE_JULIA_BLAS = true else USE_JULIA_BLAS = true FORCE_LAPACK = true end - info("USE_JULIA_BLAS -> $USE_JULIA_BLAS") + @info("USE_JULIA_BLAS -> $USE_JULIA_BLAS") blas_name = blas_vendor == :openblas64 ? "openblas" : string(blas_vendor) MSHADOW_LDFLAGS = "MSHADOW_LDFLAGS=-lm $blas_path" @@ -188,7 +191,7 @@ if !libmxnet_detected @build_steps begin BinDeps.DirectoryRule(_mxdir, @build_steps begin ChangeDirectory(_srcdir) - `git clone https://github.com/apache/incubator-mxnet` + `git clone https://github.com/apache/incubator-mxnet mxnet` end) @build_steps begin ChangeDirectory(_mxdir) @@ -199,15 +202,12 @@ if !libmxnet_detected `git checkout origin/$libmxnet_curr_ver` end `git submodule update --init --recursive` - `git -C mshadow checkout -- make/mshadow.mk` - - # copying on changed, make travis caching happy - `../../cpcblas.sh` - - `sed -i -s "s/MSHADOW_CFLAGS = \(.*\)/MSHADOW_CFLAGS = \1 $ilp64/" mshadow/make/mshadow.mk` + `git -C 3rdparty/mshadow checkout -- make/mshadow.mk` + `cp -v ../../cblas.h include/cblas.h` + `sed -i -s "s/MSHADOW_CFLAGS = \(.*\)/MSHADOW_CFLAGS = \1 $ilp64/" 3rdparty/mshadow/make/mshadow.mk` # Copy config.mk, always override the file - if is_apple() + if Sys.isapple() `cp make/osx.mk config.mk` else `cp make/config.mk config.mk` @@ -239,7 +239,7 @@ if !libmxnet_detected # Force enable LAPACK build # Julia's OpenBLAS has LAPACK functionality already if FORCE_LAPACK - if is_apple() + if Sys.isapple() MSHADOW_LDFLAGS *= " -framework Accelerate" end `sed -i -s 's/ADD_CFLAGS =\(.*\)/ADD_CFLAGS =\1 -DMXNET_USE_LAPACK/' config.mk` diff --git a/julia/docs/.gitignore b/julia/docs/.gitignore new file mode 100644 index 000000000..8a6e014fb --- /dev/null +++ b/julia/docs/.gitignore @@ -0,0 +1,6 @@ +Manifest.toml + +_build +build/ +site/ +venv/ diff --git a/julia/docs/Makefile b/julia/docs/Makefile index 57c623889..e42b8cdcc 100644 --- a/julia/docs/Makefile +++ b/julia/docs/Makefile @@ -16,5 +16,14 @@ # under the License. all: - julia --color=yes ./make.jl - mkdocs build + julia --color=yes --project=./ -e \ + 'using Pkg; \ + Pkg.develop(PackageSpec(name="MXNet", path = joinpath(pwd(), "..")))' + julia --color=yes --project=./ ./make.jl + pip install --user pygments mkdocs mkdocs-material python-markdown-math + ~/.local/bin/mkdocs build + +clean: + rm -rvf venv + rm -rvf build + rm -rvf site diff --git a/julia/docs/Project.toml b/julia/docs/Project.toml new file mode 100644 index 000000000..a4b243b0f --- /dev/null +++ b/julia/docs/Project.toml @@ -0,0 +1,7 @@ +[deps] +Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4" +DocumenterMarkdown = "997ab1e6-3595-5248-9280-8efb232c3433" +MXNet = "a7949054-b901-59c6-b8e3-7238c29bf7f0" + +[compat] +Documenter = "~0.21" diff --git a/julia/docs/make.jl b/julia/docs/make.jl index 6e3705a95..3e541c636 100644 --- a/julia/docs/make.jl +++ b/julia/docs/make.jl @@ -15,15 +15,13 @@ # specific language governing permissions and limitations # under the License. -using Documenter, MXNet +using Documenter +using DocumenterMarkdown +using MXNet makedocs( - modules = MXNet, - doctest = false -) - -deploydocs( - deps = Deps.pip("pygments", "mkdocs", "mkdocs-material", "python-markdown-math"), - repo = "github.com/dmlc/MXNet.jl.git", - julia = "0.6", + sitename = "MXNet.jl", + modules = MXNet, + doctest = false, + format = Markdown(), ) diff --git a/julia/docs/mkdocs.yml b/julia/docs/mkdocs.yml index 242817308..22cb71869 100644 --- a/julia/docs/mkdocs.yml +++ b/julia/docs/mkdocs.yml @@ -41,7 +41,7 @@ markdown_extensions: docs_dir: 'build' -pages: +nav: - Home: index.md - Tutorial: - Digit Recognition on MNIST: tutorial/mnist.md diff --git a/julia/docs/src/tutorial/mnist.md b/julia/docs/src/tutorial/mnist.md index 76430fd1b..916e46deb 100644 --- a/julia/docs/src/tutorial/mnist.md +++ b/julia/docs/src/tutorial/mnist.md @@ -252,7 +252,7 @@ labels = reduce( labels .= labels .+ 1 # Now we use compute the accuracy -pred = map(i -> indmax(probs[1:10, i]), 1:size(probs, 2)) +pred = map(i -> argmax(probs[1:10, i]), 1:size(probs, 2)) correct = sum(pred .== labels) @printf "Accuracy on eval set: %.2f%%\n" 100correct/length(labels) ``` diff --git a/julia/examples/char-lstm/lstm.jl b/julia/examples/char-lstm/lstm.jl index fc4bcc4b6..6f6640e95 100644 --- a/julia/examples/char-lstm/lstm.jl +++ b/julia/examples/char-lstm/lstm.jl @@ -145,7 +145,16 @@ function mx.update!(metric::NLL, labels::Vector{<:mx.NDArray}, preds::Vector{<:m nll = 0.0 for (label, pred) in zip(labels, preds) @mx.nd_as_jl ro=(label, pred) begin - nll -= sum(log.(max.(broadcast_getindex(pred, round.(Int,label+1), 1:length(label)), 1e-20))) + nll -= sum( + log.( + max.( + getindex.( + (pred,), + round.(Int,label .+ 1), + 1:length(label)), + 1e-20) + ) + ) end end diff --git a/julia/examples/char-lstm/seq-data.jl b/julia/examples/char-lstm/seq-data.jl index 3489e5bc3..0df110322 100644 --- a/julia/examples/char-lstm/seq-data.jl +++ b/julia/examples/char-lstm/seq-data.jl @@ -16,13 +16,14 @@ # under the License. # Simple data provider that load text -using Iterators +using Base.Iterators using MXNet -function build_vocabulary(corpus_fn::AbstractString, vocab_fn::AbstractString; max_vocab=10000) +function build_vocabulary(corpus_fn::AbstractString, vocab_fn::AbstractString; + max_vocab = 10000) if isfile(vocab_fn) - info("Vocabulary already exists, reusing $vocab_fn...") - vocab = Dict{Char,Int}(w => i for (i,w) in enumerate(readstring(vocab_fn))) + @info("Vocabulary already exists, reusing $vocab_fn...") + vocab = Dict{Char,Int}(w => i for (i,w) in enumerate(read(vocab_fn, String))) else # count symbol frequency dict = Dict{Char,Int}() @@ -99,8 +100,8 @@ function mx.eachbatch(p::CharSeqProvider) for idx_batch in partition(idx_all, p.batch_size*p.seq_len) for i = 1:p.seq_len - data_jl[i][:] = 0 - label_jl[i][:] = 0 + data_jl[i][:] .= 0 + label_jl[i][:] .= 0 end for (i, idx_seq) in enumerate(partition(idx_batch, p.seq_len)) diff --git a/julia/examples/char-lstm/train.jl b/julia/examples/char-lstm/train.jl index 57bfeb6b6..7dbdefd42 100644 --- a/julia/examples/char-lstm/train.jl +++ b/julia/examples/char-lstm/train.jl @@ -31,7 +31,7 @@ lstm = LSTM(LSTM_N_LAYER, SEQ_LENGTH, DIM_HIDDEN, DIM_EMBED, #--data # load data -text_all = readstring(INPUT_FILE) +text_all = read(INPUT_FILE, String) len_train = round(Int, length(text_all)*DATA_TR_RATIO) text_tr = text_all[1:len_train] text_val = text_all[len_train+1:end] diff --git a/julia/examples/char-lstm/visualize.jl b/julia/examples/char-lstm/visualize.jl index e2a2c87c9..dd4839400 100644 --- a/julia/examples/char-lstm/visualize.jl +++ b/julia/examples/char-lstm/visualize.jl @@ -15,8 +15,8 @@ # specific language governing permissions and limitations # under the License. -include(joinpath(dirname(@__FILE__), "config.jl")) -include(joinpath(dirname(@__FILE__), "lstm.jl")) +include(joinpath(@__DIR__, "config.jl")) +include(joinpath(@__DIR__, "lstm.jl")) using MXNet diff --git a/julia/examples/imagenet/ijulia-pretrained-predict/Prediction with Pre-trained Model.ipynb b/julia/examples/imagenet/ijulia-pretrained-predict/Prediction with Pre-trained Model.ipynb index 2d5d0ee7d..3ef8040c3 100644 --- a/julia/examples/imagenet/ijulia-pretrained-predict/Prediction with Pre-trained Model.ipynb +++ b/julia/examples/imagenet/ijulia-pretrained-predict/Prediction with Pre-trained Model.ipynb @@ -181,7 +181,7 @@ "classes = open(joinpath(model_dir, \"synset.txt\")) do s \n", " map(x -> replace(strip(x), r\"^n[0-9]+ \", \"\"), readlines(s))\n", "end\n", - "println(classes[indmax(pred)])" + "println(classes[argmax(pred)])" ] }, { diff --git a/julia/examples/mnist/mlp-test.jl b/julia/examples/mnist/mlp-test.jl index 1af84ed3b..7a24f9281 100644 --- a/julia/examples/mnist/mlp-test.jl +++ b/julia/examples/mnist/mlp-test.jl @@ -21,7 +21,7 @@ module MNISTTest using MXNet -using Base.Test +using Test include("mnist-data.jl") @@ -56,7 +56,7 @@ function mnist_fit_and_predict(optimizer, initializer, n_epoch) end mlp_load = mx.load("$cp_prefix-symbol.json", mx.SymbolicNode) @test mx.to_json(mlp_load) == mx.to_json(mlp) - mlp_load = mx.from_json(readstring("$cp_prefix-symbol.json"), mx.SymbolicNode) + mlp_load = mx.from_json(read("$cp_prefix-symbol.json", String), mx.SymbolicNode) @test mx.to_json(mlp_load) == mx.to_json(mlp) #-------------------------------------------------------------------------------- @@ -68,13 +68,13 @@ function mnist_fit_and_predict(optimizer, initializer, n_epoch) for batch in eval_provider push!(labels, copy(mx.get(eval_provider, batch, :softmax_label))) end - labels = cat(1, labels...) + labels = cat(labels..., dims = 1) # Now we use compute the accuracy correct = 0 for i = 1:length(labels) # labels are 0...9 - if indmax(probs[:,i]) == labels[i]+1 + if argmax(probs[:,i]) == labels[i]+1 correct += 1 end end @@ -88,32 +88,32 @@ function mnist_fit_and_predict(optimizer, initializer, n_epoch) end function test_mnist_mlp() - info("MNIST::SGD") + @info("MNIST::SGD") @test mnist_fit_and_predict(mx.SGD(η=.2), mx.UniformInitializer(.01), 2) > 90 - info("MNIST::SGD::η scheduler") + @info("MNIST::SGD::η scheduler") @test mnist_fit_and_predict(mx.SGD(η_sched=mx.LearningRate.Inv(.25)), mx.UniformInitializer(.01), 2) > 90 - info("MNIST::SGD::momentum μ") + @info("MNIST::SGD::momentum μ") @test mnist_fit_and_predict(mx.SGD(η=.1, μ=.9), mx.UniformInitializer(.01), 2) > 90 - info("MNIST::ADAM") + @info("MNIST::ADAM") @test mnist_fit_and_predict(mx.ADAM(), mx.NormalInitializer(), 2) > 90 - info("MNIST::AdaGrad") + @info("MNIST::AdaGrad") @test mnist_fit_and_predict(mx.AdaGrad(), mx.NormalInitializer(), 2) > 90 - info("MNIST::AdaDelta") + @info("MNIST::AdaDelta") @test mnist_fit_and_predict(mx.AdaDelta(), mx.NormalInitializer(), 2) > 90 - info("MNIST::AdaMax") + @info("MNIST::AdaMax") @test mnist_fit_and_predict(mx.AdaMax(), mx.NormalInitializer(), 2) > 90 - info("MNIST::RMSProp") + @info("MNIST::RMSProp") @test mnist_fit_and_predict(mx.RMSProp(), mx.NormalInitializer(), 2) > 90 - info("MNIST::Nadam") + @info("MNIST::Nadam") @test mnist_fit_and_predict(mx.Nadam(), mx.NormalInitializer(), 2) > 90 end diff --git a/julia/examples/mnist/mlp.jl b/julia/examples/mnist/mlp.jl index 20facc9b7..86111ee68 100644 --- a/julia/examples/mnist/mlp.jl +++ b/julia/examples/mnist/mlp.jl @@ -70,6 +70,6 @@ labels = reduce( labels .= labels .+ 1 # Now we use compute the accuracy -pred = map(i -> indmax(probs[1:10, i]), 1:size(probs, 2)) +pred = map(i -> argmax(probs[1:10, i]), 1:size(probs, 2)) correct = sum(pred .== labels) @printf "Accuracy on eval set: %.2f%%\n" 100correct/length(labels) diff --git a/julia/src/MXNet.jl b/julia/src/MXNet.jl index 03c3cb89b..febd80cc8 100644 --- a/julia/src/MXNet.jl +++ b/julia/src/MXNet.jl @@ -15,8 +15,6 @@ # specific language governing permissions and limitations # under the License. -__precompile__() - module MXNet using Reexport @@ -26,16 +24,20 @@ using Reexport export mx module mx -import Base.Iterators: filter +using Base.Broadcast: Broadcasted, DefaultArrayStyle +using Libdl +using LinearAlgebra +using Markdown +using Printf +using Statistics +using Random using Formatting using MacroTools -using TakingBroadcastSeriously: @unfuse -import TakingBroadcastSeriously: broadcast_ # Functions from base that we can safely extend and that are defined by libmxnet. -import Base: round, ceil, floor, cos, sin, abs, sign, exp, sqrt, exp, log, norm, - transpose +import Base.Broadcast: broadcasted +import Base.Iterators: filter ############################################################################### # exports @@ -140,9 +142,9 @@ include("base.jl") include("context.jl") include("util.jl") -include("broadcast.jl") include("ndarray.jl") + include("random.jl") include("autograd.jl") @@ -150,6 +152,8 @@ include("name.jl") include("symbolic-node.jl") include("executor.jl") +include("broadcast.jl") + include("metric.jl") include("optimizer.jl") include("initializer.jl") diff --git a/julia/src/autograd.jl b/julia/src/autograd.jl index 72fb82ba1..8b5edae57 100644 --- a/julia/src/autograd.jl +++ b/julia/src/autograd.jl @@ -43,7 +43,7 @@ function _set_recording(state::Bool)::Bool prev[] end -_set_recording(::Void) = nothing +_set_recording(::Cvoid) = nothing """ Set status to training/predicting. @@ -63,7 +63,7 @@ function _set_training(train_mode::Bool)::Bool prev[] end -_set_training(::Void) = nothing +_set_training(::Cvoid) = nothing ############################################################################### # Public API @@ -91,7 +91,7 @@ function is_training()::Bool state[] end -@inline function _record(f, is_record::Union{Void,Bool}, train_mode::Union{Void,Bool}) +@inline function _record(f, is_record::Union{Cvoid,Bool}, train_mode::Union{Cvoid,Bool}) # Port from Python's `_RecordingStateScope` context manager # __enter__ prev_is_record = _set_recording(is_record) @@ -211,12 +211,12 @@ Compute the gradients of heads w.r.t previously marked variables. - `head::NDArray`: output NDArray -- `head_grad::NDArray` or `Void`: gradient coefficient with respect to head. +- `head_grad::NDArray` or `Cvoid`: gradient coefficient with respect to head. - `heads::Vector{NDArray}`: a list of output NDArray - `head_grads::Vector`: a list of gradient coefficient with respect ot heads. - the element should be `NDArray` or `Void` + the element should be `NDArray` or `Cvoid` - `retain_graph::Bool`: whether to keep the graph after backward. e.g: If you want to differentiate the same graph twice, @@ -227,10 +227,10 @@ Compute the gradients of heads w.r.t previously marked variables. backward!(head::NDArray, head_grad::NDArray; kws...) = backward!([head], [head_grad]; kws...) -backward!(head::NDArray, head_grad::Void = nothing; kws...) = +backward!(head::NDArray, head_grad::Cvoid = nothing; kws...) = backward!([head], head_grad; kws...) -function backward!(heads::VecOfNDArray, head_grad::Void; +function backward!(heads::VecOfNDArray, head_grad::Cvoid; retain_graph::Bool = false, train_mode::Bool = true) @mxcall( :MXAutogradBackwardEx, @@ -262,10 +262,10 @@ function backward!(heads::VecOfNDArray, head_grads::Vector; ograd_handles = map(head_grads) do x if x isa NDArray x.handle - elseif x isa Void + elseif x ≡ nothing # faster than `x isa Cvoid` in Julia 0.7 MX_handle(C_NULL) else - throw(ArgumentError("element of head_grads should be NDArray or Void")) + throw(ArgumentError("element of head_grads should be NDArray or Cvoid")) end end @assert length(output_handles) == length(ograd_handles) diff --git a/julia/src/base.jl b/julia/src/base.jl index ce1c183ea..61779d194 100644 --- a/julia/src/base.jl +++ b/julia/src/base.jl @@ -27,7 +27,7 @@ Base.show(io::IO, e::MXError) = print(io, e.msg) ################################################################################ const MX_uint = Cuint const MX_float = Cfloat -const MX_handle = Ptr{Void} +const MX_handle = Ptr{Cvoid} const char_p = Ptr{UInt8} const char_pp = Ptr{char_p} @@ -50,7 +50,10 @@ const grad_req_map = Dict{Symbol,GRAD_REQ}( const MXNET_LIB = Libdl.find_library(["libmxnet.$(Libdl.dlext)", "libmxnet.so"], # see build.jl [joinpath(get(ENV, "MXNET_HOME", ""), "lib"), get(ENV, "MXNET_HOME", ""), - Pkg.dir("MXNet", "deps", "usr", "lib")]) + joinpath(@__DIR__, "..", + "deps", "usr", "lib")]) +const LIB_VERSION = Ref{Cint}(0) + if isempty(MXNET_LIB) # touch this file, so that after the user properly build libmxnet, the precompiled # MXNet.ji will be re-compiled to get MXNET_LIB properly. @@ -65,8 +68,7 @@ function __init__() # TODO: bug in nnvm, if do not call this, call get handle "_copyto" will fail _get_libmx_op_names() _populate_iter_creator_cache!() - - global const LIB_VERSION = _get_lib_version() + _get_lib_version!() atexit() do # notify libmxnet we are shutting down @@ -87,8 +89,8 @@ macro mxcall(fv, argtypes, args...) f = eval(fv) args = map(esc, args) quote - _mxret = ccall( ($(Meta.quot(f)), $MXNET_LIB), - Cint, $argtypes, $(args...) ) + _mxret = ccall(($(QuoteNode(f)), $MXNET_LIB), + Cint, $argtypes, $(args...)) if _mxret != 0 err_msg = mx_get_last_error() throw(MXError(err_msg)) @@ -98,38 +100,38 @@ end """ Get libmxnet version + +This function will changes the global variable `LIB_VERSION`. """ -function _get_lib_version() - ver = Ref{Cint}(0) - @mxcall :MXGetVersion (Ref{Cint},) ver - ver[] +function _get_lib_version!() + @mxcall :MXGetVersion (Ref{Cint},) LIB_VERSION + LIB_VERSION[] end ################################################################################ # Handle types ################################################################################ -macro mx_define_handle_t(name, destructor) - name = esc(name) - quote +function mx_define_handle_t(name, destructor) + @eval begin mutable struct $name - value :: MX_handle + value::MX_handle function $name(value = C_NULL) hdr = new(value) - $(if destructor != :nop - :(finalizer(hdr, delete!)) + $(if destructor != nothing + :(finalizer(delete!, hdr)) end) return hdr end end - $(if finalizer != :nop + $(if finalizer != nothing quote function delete!(h :: $name) if h.value != C_NULL - @mxcall($(Meta.quot(destructor)), (MX_handle,), h.value) + @mxcall($(QuoteNode(destructor)), (MX_handle,), h.value) h.value = C_NULL end end @@ -142,16 +144,16 @@ macro mx_define_handle_t(name, destructor) Base.convert(t::Type{MX_handle}, obj::$name) = Base.unsafe_convert(t, obj) Base.cconvert(t::Type{MX_handle}, obj::$name) = Base.unsafe_convert(t, obj) - function Base.isnull(obj::$name) obj.value == C_NULL end + MX_handle(x::$name) = Base.convert(MX_handle, x) end end -@mx_define_handle_t(MX_NDArrayHandle, MXNDArrayFree) -@mx_define_handle_t(MX_OpHandle, nop) -@mx_define_handle_t(MX_SymbolHandle, MXSymbolFree) -@mx_define_handle_t(MX_ExecutorHandle, MXExecutorFree) -@mx_define_handle_t(MX_DataIterHandle, MXDataIterFree) -@mx_define_handle_t(MX_KVStoreHandle, MXKVStoreFree) +mx_define_handle_t(:MX_NDArrayHandle, :MXNDArrayFree) +mx_define_handle_t(:MX_OpHandle, nothing) +mx_define_handle_t(:MX_SymbolHandle, :MXSymbolFree) +mx_define_handle_t(:MX_ExecutorHandle, :MXExecutorFree) +mx_define_handle_t(:MX_DataIterHandle, :MXDataIterFree) +mx_define_handle_t(:MX_KVStoreHandle, :MXKVStoreFree) ################################################################################ # MXNet Params @@ -183,8 +185,8 @@ dump_mx_param(val::Float64) = @sprintf("%.16e", val) dump_mx_param(val::Float32) = @sprintf("%.8e", val) dump_mx_param(val::Float16) = @sprintf("%.4e", val) dump_mx_param(val::Irrational) = @sprintf("%.16e", val) -dump_mx_param(shape::NTuple{N, <:Integer}) where N = - string(tuple(flipdim([shape...], 1)...)) +dump_mx_param(shape::NTuple{N,<:Integer}) where N = + string(reverse(shape)) """ @@ -248,11 +250,11 @@ function _defstruct_impl(is_immutable, name, fields) name = esc(name.args[1]) end - field_defs = Vector{Expr}(length(fields)) # :(field2 :: Int) - field_names = Vector{Expr}(length(fields)) # :field2 - field_defaults = Vector{Expr}(length(fields)) # :(field2 = 0) - field_types = Vector{Expr}(length(fields)) # Int - field_asserts = Vector{Expr}(length(fields)) # :(field2 >= 0) + field_defs = Vector{Expr}(undef, length(fields)) # :(field2 :: Int) + field_names = Vector{Expr}(undef, length(fields)) # :field2 + field_defaults = Vector{Expr}(undef, length(fields)) # :(field2 = 0) + field_types = Vector{Expr}(undef, length(fields)) # Int + field_asserts = Vector{Expr}(undef, length(fields)) # :(field2 >= 0) required_field = Symbol[] for i = 1:length(fields) diff --git a/julia/src/broadcast.jl b/julia/src/broadcast.jl index fee960a46..7c68fab50 100644 --- a/julia/src/broadcast.jl +++ b/julia/src/broadcast.jl @@ -15,22 +15,17 @@ # specific language governing permissions and limitations # under the License. -using TakingBroadcastSeriously: Broadcasted, unwrap +struct NDArrayStyle{N} <: Broadcast.AbstractArrayStyle{N} end +NDArrayStyle(::Val{N}) where N = NDArrayStyle{N}() +NDArrayStyle{M}(::Val{N}) where {N,M} = NDArrayStyle{N}() -for f in :[%, - tan, asin, acos, atan, - sinh, cosh, tanh, asinh, acosh, atanh, - min, max, - hypot].args - # copy from TakingBroadcastSeriously - @eval Base.$f(a::Broadcasted...) = Broadcasted(broadcast_($f, unwrap.(a)...)) - @eval Base.$f(a::Broadcasted, b) = Broadcasted(broadcast_($f, unwrap(a), b)) - @eval Base.$f(b, a::Broadcasted) = Broadcasted(broadcast_($f, b, unwrap(a))) -end +# Determin the output type +Base.BroadcastStyle(::Type{<:NDArray{T,N}}) where {T,N} = NDArrayStyle{N}() -for f in :[σ, sigmoid, relu, softmax, log_softmax].args - # copy from TakingBroadcastSeriously - @eval $f(a::Broadcasted...) = Broadcasted(broadcast_($f, unwrap.(a)...)) - @eval $f(a::Broadcasted, b) = Broadcasted(broadcast_($f, unwrap(a), b)) - @eval $f(b, a::Broadcasted) = Broadcasted(broadcast_($f, b, unwrap(a))) -end +Base.broadcastable(x::NDArray) = x + +# Make it non-lazy +broadcasted(f, x::NDArray, args...) = f(x, args...) +broadcasted(f, y, x::NDArray, args...) = f(y, x, args...) +broadcasted(f, x::NDArray{T,N}, y::NDArray{T,N}, args...) where {T,N} = + f(x, y, args...) diff --git a/julia/src/callback.jl b/julia/src/callback.jl index 06e431de0..39d8f2552 100644 --- a/julia/src/callback.jl +++ b/julia/src/callback.jl @@ -96,7 +96,7 @@ function speedometer(;frequency::Int = 50) cl_tic = time() else speed = frequency * state.batch_size / (time() - cl_tic) - info(format("Speed: {1:>6.2f} samples/sec", speed)) + @info(format("Speed: {1:>6.2f} samples/sec", speed)) cl_tic = time() end end diff --git a/julia/src/context.jl b/julia/src/context.jl index c97522b3b..71aee3020 100644 --- a/julia/src/context.jl +++ b/julia/src/context.jl @@ -17,17 +17,22 @@ @enum CONTEXT_TYPE CPU=1 GPU=2 CPU_PINNED=3 +Base.convert(::Type{CONTEXT_TYPE}, x::Integer) = CONTEXT_TYPE(x) + """ Context(dev_type, dev_id) A context describes the device type and id on which computation should be carried on. """ struct Context - device_type :: CONTEXT_TYPE - device_id :: Int + device_type::CONTEXT_TYPE + device_id::Int + + Context(dev_type::CONTEXT_TYPE, dev_id::Integer = 0) = new(dev_type, dev_id) end -Context(dev_type :: Union{CONTEXT_TYPE, Int}, dev_id :: Int = 0) = - Context(convert(CONTEXT_TYPE, dev_type), dev_id) + +Context(dev_type::Integer, dev_id::Integer = 0) = + Context(convert(CONTEXT_TYPE, dev_type), dev_id) Base.show(io::IO, ctx::Context) = print(io, "$(ctx.device_type)$(ctx.device_id)") @@ -39,9 +44,9 @@ Get a CPU context with a specific id. `cpu()` is usually the default context for operations when no context is specified. # Arguments -* `dev_id::Int = 0`: the CPU id. +* `dev_id::Integer = 0`: the CPU id. """ -cpu(dev_id::Int = 0) = Context(CPU, dev_id) +cpu(dev_id::Integer = 0) = Context(CPU, dev_id) """ gpu(dev_id) @@ -49,6 +54,6 @@ cpu(dev_id::Int = 0) = Context(CPU, dev_id) Get a GPU context with a specific id. The K GPUs on a node is typically numbered as 0,...,K-1. # Arguments -* `dev_id :: Int = 0` the GPU device id. +* `dev_id::Integer = 0` the GPU device id. """ -gpu(dev_id::Int = 0) = return Context(GPU, dev_id) +gpu(dev_id::Integer = 0) = Context(GPU, dev_id) diff --git a/julia/src/deprecated.jl b/julia/src/deprecated.jl index 12c5345aa..32819810e 100644 --- a/julia/src/deprecated.jl +++ b/julia/src/deprecated.jl @@ -44,26 +44,26 @@ # @deprecate make `randn` exported accidentially # so we make the depwarn manually function randn(μ, σ, dims::NTuple{N,Int}, ctx::Context = cpu()) where N - warn("mx.randn(μ, σ, dims, ctx = cpu()) is deprecated, use " * - "mx.randn(dims...; μ = μ, σ = σ, context = ctx) instead.") + @warn("mx.randn(μ, σ, dims, ctx = cpu()) is deprecated, use " * + "mx.randn(dims...; μ = μ, σ = σ, context = ctx) instead.") mx.randn(dims...; μ = μ, σ = σ, context = ctx) end function randn!(μ, σ, x::NDArray) - warn("mx.randn!(μ, σ, x::NDArray) is deprecated, use " * - "mx.randn!(x; μ = μ, σ = σ) instead.") + @warn("mx.randn!(μ, σ, x::NDArray) is deprecated, use " * + "mx.randn!(x; μ = μ, σ = σ) instead.") randn!(x; μ = μ, σ = σ) end function rand!(low::Real, high::Real, x::NDArray) - warn("rand!(low, high, x::NDArray) is deprecated, use " * - "rand!(x, low = low, high = high) instead.") + @warn("rand!(low, high, x::NDArray) is deprecated, use " * + "rand!(x, low = low, high = high) instead.") rand!(x, low = low, high = high) end function rand(low::Real, high::Real, dims::NTuple{N,Int}, context::Context = cpu()) where N - warn("rand!(low, high, dims, x::NDArray, context = cpu()) is deprecated, use " * - "rand!(dims..., x; low = low, high = high, context = cpu()) instead.") + @warn("rand!(low, high, dims, x::NDArray, context = cpu()) is deprecated, use " * + "rand!(dims..., x; low = low, high = high, context = cpu()) instead.") rand(dims...; low = low, high = high, context = context) end @@ -75,86 +75,97 @@ end @deprecate clip(x; a_min = 0, a_max = 0) clip(x, a_min, a_max) function broadcast_plus(x::NDArray, y::NDArray) - warn("broadcast_plus(x, y) is deprecated, use x .+ y instead.") + @warn("broadcast_plus(x, y) is deprecated, use x .+ y instead.") x .+ y end function broadcast_add(x::NDArray, y::NDArray) - warn("broadcast_add(x, y) is deprecated, use x .+ y instead.") + @warn("broadcast_add(x, y) is deprecated, use x .+ y instead.") x .+ y end function broadcast_sub(x::NDArray, y::NDArray) - warn("broadcast_sub(x, y) is deprecated, use x .- y instead.") + @warn("broadcast_sub(x, y) is deprecated, use x .- y instead.") x .- y end function broadcast_minus(x::NDArray, y::NDArray) - warn("broadcast_minus(x, y) is deprecated, use x .- y instead.") + @warn("broadcast_minus(x, y) is deprecated, use x .- y instead.") x .- y end function broadcast_mul(x::NDArray, y::NDArray) - warn("broadcast_mul(x, y) is deprecated, use x .* y instead.") + @warn("broadcast_mul(x, y) is deprecated, use x .* y instead.") x .* y end function broadcast_div(x::NDArray, y::NDArray) - warn("broadcast_div(x, y) is deprecated, use x ./ y instead.") + @warn("broadcast_div(x, y) is deprecated, use x ./ y instead.") x ./ y end function broadcast_mod(x::NDArray, y::NDArray) - warn("broadcast_mod(x, y) is deprecated, use x .% y instead.") + @warn("broadcast_mod(x, y) is deprecated, use x .% y instead.") x .% y end function broadcast_power(x::NDArray, y::NDArray) - warn("broadcast_power(x, y) is deprecated, use x.^y instead.") + @warn("broadcast_power(x, y) is deprecated, use x.^y instead.") x.^y end function broadcast_equal(x::NDArray, y::NDArray) - warn("broadcast_equal(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_equal(x, y) is deprecated, use x .== y instead.") x .== y end function broadcast_not_equal(x::NDArray, y::NDArray) - warn("broadcast_not_equal(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_not_equal(x, y) is deprecated, use x .== y instead.") x .!= y end function broadcast_greater(x::NDArray, y::NDArray) - warn("broadcast_greater(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_greater(x, y) is deprecated, use x .== y instead.") x .> y end function broadcast_greater_equal(x::NDArray, y::NDArray) - warn("broadcast_greater_equal(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_greater_equal(x, y) is deprecated, use x .== y instead.") x .>= y end function broadcast_lesser(x::NDArray, y::NDArray) - warn("broadcast_lesser(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_lesser(x, y) is deprecated, use x .== y instead.") x .< y end function broadcast_lesser_equal(x::NDArray, y::NDArray) - warn("broadcast_lesser_equal(x, y) is deprecated, use x .== y instead.") + @warn("broadcast_lesser_equal(x, y) is deprecated, use x .== y instead.") x .<= y end function broadcast_maximum(x::NDArray, y::NDArray) - warn("broadcast_maximum(x, y) is deprecated, use max.(x, y) instead.") + @warn("broadcast_maximum(x, y) is deprecated, use max.(x, y) instead.") max.(x, y) end function broadcast_minimum(x::NDArray, y::NDArray) - warn("broadcast_minimum(x, y) is deprecated, use min.(x, y) instead.") + @warn("broadcast_minimum(x, y) is deprecated, use min.(x, y) instead.") min.(x, y) end function broadcast_hypot(x::NDArray, y::NDArray) - warn("broadcast_hypot(x, y) is deprecated, use hypot.(x, y) instead.") + @warn("broadcast_hypot(x, y) is deprecated, use hypot.(x, y) instead.") hypot.(x, y) end + +# Introduced by https://github.com/apache/incubator-mxnet/pull/12845 +import Base: sum, maximum, minimum, prod, cat +@deprecate sum(x::NDArray, dims) sum(x, dims = dims) +@deprecate maximum(x::NDArray, dims) maximum(x, dims = dims) +@deprecate minimum(x::NDArray, dims) minimum(x, dims = dims) +@deprecate prod(x::NDArray, dims) prod(x, dims = dims) +@deprecate cat(dims, As::NDArray{T}...) where T cat(As..., dims = dims) + +import Statistics: mean +@deprecate mean(x::NDArray, dims) mean(x, dims = dims) diff --git a/julia/src/executor.jl b/julia/src/executor.jl index 4bf4339d6..29c21c8f4 100644 --- a/julia/src/executor.jl +++ b/julia/src/executor.jl @@ -28,7 +28,7 @@ mutable struct Executor handle :: MX_ExecutorHandle symbol :: SymbolicNode arg_arrays :: VecOfNDArray - grad_arrays :: Vector{Union{Void,<:NDArray}} + grad_arrays :: Vector{Union{Cvoid,<:NDArray}} aux_arrays :: VecOfNDArray outputs :: VecOfNDArray arg_dict :: Dict{Symbol} @@ -73,17 +73,17 @@ function _get_ndarray_inputs(arg_key::AbstractString, args::Dict{Symbol}, args_vec = map(arg_names) do name arr = get(args, name, nothing) if !allow_missing - @assert(!isa(arr, Void), "Must specify all arguments in $arg_key ($name is missing)") + @assert(!isa(arr, Cvoid), "Must specify all arguments in $arg_key ($name is missing)") end arr end # help the type inference if allow_missing - args_vec = Union{NDArray,Void}[args_vec...] + args_vec = Union{NDArray,Cvoid}[args_vec...] else args_vec = NDArray[args_vec...] end - args_hdr = MX_handle[(isa(x,Void) ? MX_handle(0) : x) for x in args_vec] + args_hdr = MX_handle[(isa(x,Cvoid) ? MX_handle(0) : x) for x in args_vec] return (args_hdr, args_vec) end @@ -115,12 +115,12 @@ function bind(self::SymbolicNode, ctx::Context, args; aux_args_hdr, aux_states = _get_ndarray_inputs("aux_states", aux_states, list_auxiliary_states(self), false) if isa(grad_req, GRAD_REQ) - reqs = MX_uint[grad_req for i=1:length(args)] + reqs = MX_uint[MX_uint(grad_req) for i=1:length(args)] elseif isa(grad_req, Vector{GRAD_REQ}) @assert(length(grad_req) == length(args)) - reqs = MX_uint[grad_req...] + reqs = MX_uint[MX_uint.(grad_req)...] elseif isa(grad_req, Dict{Symbol, GRAD_REQ}) - reqs = MX_uint[get(grad_req, name, GRAD_NOP) for name in arg_names] + reqs = MX_uint[MX_uint(get(grad_req, name, GRAD_NOP)) for name in arg_names] end ref_hdr = Ref{MX_handle}(0) @@ -129,7 +129,7 @@ function bind(self::SymbolicNode, ctx::Context, args; MX_uint, Ptr{MX_handle}, Ref{MX_handle}), self, ctx.device_type, ctx.device_id, length(args), args_hdr, args_grad_hdr, reqs, length(aux_states), aux_args_hdr, ref_hdr) - args_grad = convert(Vector{Union{Void,NDArray}}, args_grad) + args_grad = convert(Vector{Union{Cvoid,NDArray}}, args_grad) executor = Executor(MX_ExecutorHandle(ref_hdr[]), self, args, args_grad, aux_states) end @@ -145,7 +145,7 @@ function simple_bind(self::SymbolicNode, ctx::Context; grad_req::Union{GRAD_REQ,Dict{Symbol,GRAD_REQ}} = GRAD_WRITE, kwargs...) arg_shapes, out_shapes, aux_shapes = infer_shape(self; kwargs...) - @assert(!isa(arg_shapes, Void), "Information not enough to perform complete shape inference") + @assert(!isa(arg_shapes, Cvoid), "Information not enough to perform complete shape inference") arg_arrays = NDArray[zeros(shape, ctx) for shape in arg_shapes] arg_names = list_arguments(self) @@ -228,7 +228,7 @@ julia> x = mx.Variable(:x) MXNet.mx.SymbolicNode x julia> exec = mx.bind(x + 1, mx.cpu(), Dict(:x => mx.ones(2,3))) -mx.Executor Ptr{Void} @0x000055c3dee9eb30 +mx.Executor Ptr{Nothing} @0x000055c3dee9eb30 julia> print(exec) Symbol Outputs: diff --git a/julia/src/initializer.jl b/julia/src/initializer.jl index 95dbeb31f..d4b35c0ce 100644 --- a/julia/src/initializer.jl +++ b/julia/src/initializer.jl @@ -143,6 +143,10 @@ NormalInitializer(; mu=0, sigma=0.01) = NormalInitializer(mu, sigma) _init_weight(i::NormalInitializer, name::Symbol, x::NDArray) = randn!(x, μ = i.μ, σ = i.σ) +@enum XavierDistribution xv_uniform xv_normal +@enum XavierRegularization xv_avg xv_in xv_out + + """ XavierInitializer @@ -161,10 +165,6 @@ used by various libraries. * [K. He, X. Zhang, S. Ren, and J. Sun 2015]: `mx.XavierInitializer(distribution = mx.xv_gaussian, regularization = mx.xv_in, magnitude = 2)` * caffe_avg: `mx.XavierInitializer(distribution = mx.xv_uniform, regularization = mx.xv_avg, magnitude = 3)` """ - -@enum XavierDistribution xv_uniform xv_normal -@enum XavierRegularization xv_avg xv_in xv_out - struct XavierInitializer <: AbstractInitializer distribution :: XavierDistribution regularization :: XavierRegularization diff --git a/julia/src/io.jl b/julia/src/io.jl index e5f439507..32f7fece7 100644 --- a/julia/src/io.jl +++ b/julia/src/io.jl @@ -24,13 +24,12 @@ The root type for all data provider. A data provider should implement the follow * [`provide_data`](@ref) * [`provide_label`](@ref) -As well as the Julia iterator interface (see [the Julia manual](http://docs.julialang.org/en/stable/manual/interfaces/)). +As well as the Julia iterator interface (see +[the Julia manual](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-iteration-1)). Normally this involves defining: * `Base.eltype(provider) -> AbstractDataBatch` -* `Base.start(provider) -> AbstractDataProviderState` -* `Base.done(provider, state) -> Bool` -* `Base.next(provider, state) -> (AbstractDataBatch, AbstractDataProvider)` +* `Base.iterate(provider[, state]) -> (AbstractDataBatch, AbstractDataProvider)` """ abstract type AbstractDataProvider end @@ -395,7 +394,11 @@ end Base.eltype(provider :: ArrayDataProvider) = ArrayDataProviderState -function Base.start(provider :: ArrayDataProvider) +struct ArrayDataBatch <: AbstractDataBatch + idx :: UnitRange{Int} +end + +function _start(provider::ArrayDataProvider) if provider.shuffle # re-shuffle all data idx_perm = randperm(provider.sample_count) @@ -406,13 +409,9 @@ function Base.start(provider :: ArrayDataProvider) return ArrayDataProviderState(1) end -Base.done(provider::ArrayDataProvider, state::ArrayDataProviderState) = - state.curr_idx > provider.sample_count - -struct ArrayDataBatch <: AbstractDataBatch - idx :: UnitRange{Int} -end -function Base.next(provider :: ArrayDataProvider, state :: ArrayDataProviderState) +function Base.iterate(provider::ArrayDataProvider, + state::ArrayDataProviderState = _start(provider)) + (state.curr_idx > provider.sample_count) && return nothing idx = state.curr_idx:Base.min(state.curr_idx+provider.batch_size-1, provider.sample_count) return (ArrayDataBatch(idx), ArrayDataProviderState(idx.stop+1)) end @@ -480,12 +479,12 @@ end function MXDataProvider(handle :: MX_DataIterHandle; data_name :: Symbol = :data, - label_name :: Union{Symbol,Void} = :softmax_label, + label_name :: Union{Symbol,Nothing} = :softmax_label, kwargs...) # for convenience, we ignore the rest keyword arguments # init iterator, load the first batch and get shapes @assert(_iter_next(handle), "Failed to load the first batch in MXDataProvider") data_shape = Tuple{Base.Symbol, Tuple}[(data_name, size(_get_data(handle)))] - if !isa(label_name, Void) + if !isa(label_name, Nothing) label_shape = Tuple{Base.Symbol, Tuple}[(label_name::Base.Symbol, size(_get_label(handle)))] else label_shape = Tuple{Base.Symbol, Tuple}[] @@ -504,10 +503,9 @@ end struct MXDataBatch <: AbstractDataBatch end -function Base.eltype(provider :: MXDataProvider) - MXDataBatch -end -function Base.start(provider :: MXDataProvider) +Base.eltype(::MXDataProvider) = MXDataBatch + +function _start(provider::MXDataProvider) if !provider.first_epoch _reset_data_iter(provider.handle) else @@ -516,7 +514,8 @@ function Base.start(provider :: MXDataProvider) return MXDataProviderState(true) end -function Base.done(provider :: MXDataProvider, state :: MXDataProviderState) + +function _done(provider::MXDataProvider, state::MXDataProviderState) if provider.first_batch state.has_next = true provider.first_batch = false @@ -525,8 +524,10 @@ function Base.done(provider :: MXDataProvider, state :: MXDataProviderState) end return !state.has_next end -function Base.next(provider :: MXDataProvider, state :: MXDataProviderState) - return (MXDataBatch(), state) + +function Base.iterate(provider::MXDataProvider, state::MXDataProviderState = _start(provider)) + _done(provider, state) && return nothing + MXDataBatch(), state end function get_data(provider :: MXDataProvider, batch :: MXDataBatch) @@ -574,7 +575,7 @@ function _populate_iter_creator_cache!() end end -_get_iter_creator(name :: Symbol) = _iter_creator_cache[name] +_get_iter_creator(name::Symbol) = _iter_creator_cache[name] function _define_data_iter_creator(hdr :: MX_handle) ref_name = Ref{char_p}(0) @@ -611,7 +612,7 @@ function _define_data_iter_creator(hdr :: MX_handle) end defun = quote - @doc $f_desc -> + @doc $f_desc function $iter_name(; kwargs...) arg_keys = String[string(k) for (k,v) in kwargs] arg_vals = String[dump_mx_param(v) for (k,v) in kwargs] diff --git a/julia/src/kvstore.jl b/julia/src/kvstore.jl index ac0367144..000684d5f 100644 --- a/julia/src/kvstore.jl +++ b/julia/src/kvstore.jl @@ -45,10 +45,10 @@ For distributed training, `KVStore` also supports a number of types: """ mutable struct KVStore handle :: MX_KVStoreHandle - updater_c :: Ptr{Void} + updater_c :: Ptr{Cvoid} updater :: Function - KVStore(hdr::MX_KVStoreHandle) = new(hdr, Ptr{Void}(0)) + KVStore(hdr::MX_KVStoreHandle) = new(hdr, Ptr{Cvoid}(0)) end function KVStore(kv_type::Symbol = :local) @@ -251,7 +251,7 @@ barrier(kv::KVStore) = @mxcall(:MXKVStoreBarrier, (MX_handle,), kv) # extra handle parameter of the API to pass the updater object around. Fix this when someday # full closure cfunction is supported in Julia. function _kvstore_update_wrapper(key::Cint, nd_recv::MX_handle, nd_local::MX_handle, - updater::Ptr{Void}) + updater::Ptr{Cvoid}) updater_func = unsafe_pointer_to_objref(updater) updater_func(Int(key), NDArray(MX_NDArrayHandle(nd_recv)), NDArray(MX_NDArrayHandle(nd_local))) @@ -291,9 +291,9 @@ julia> x """ function setupdater!(kv::KVStore, updater) kv.updater = updater # keep a reference to the julia object so that updater_c is kept valid - kv.updater_c = cfunction(_kvstore_update_wrapper, Void, - (Cint, MX_handle, MX_handle, Ptr{Void})) - @mxcall(:MXKVStoreSetUpdater, (MX_handle, Ptr{Void}, Any), + kv.updater_c = @cfunction(_kvstore_update_wrapper, Cvoid, + (Cint,MX_handle,MX_handle,Ptr{Cvoid})) + @mxcall(:MXKVStoreSetUpdater, (MX_handle, Ptr{Cvoid}, Any), kv, kv.updater_c, updater) end @@ -336,7 +336,7 @@ julia> W ``` """ function setoptimizer!(kv::KVStore, opt::AbstractOptimizer) - if ismatch(r"dist", string(get_type(kv))) && _isworker() + if occursin(r"dist", string(get_type(kv))) && _isworker() # TODO error("not implemented") else diff --git a/julia/src/metric.jl b/julia/src/metric.jl index 772eb3b3e..f1cdc68d9 100644 --- a/julia/src/metric.jl +++ b/julia/src/metric.jl @@ -47,16 +47,16 @@ Update and accumulate metrics. * `labels::Vector{NDArray}`: the labels from the data provider. * `preds::Vector{NDArray}`: the outputs (predictions) of the network. """ -function update!(metric::T, labels::VecOfNDArray, preds::VecOfNDArray) where T <: AbstractEvalMetric +update!(metric::T, labels::VecOfNDArray, preds::VecOfNDArray) where T<:AbstractEvalMetric = _update!(metric, labels, preds, hasNDArraySupport(metric)) -end function _update!(metric::T, labels::VecOfNDArray, preds::VecOfNDArray, - ::Val{true}) where T<: AbstractEvalMetric + ::Val{true}) where T<:AbstractEvalMetric if length(labels) != length(preds) - Base.warn_once( - "The number of labels ($(length(labels))) does not correspond to the\ - number of outputs ($(length(preds))). The calculated metric might not be accuracte.") + @warn( + "The number of labels ($(length(labels))) does not correspond to the " * + "number of outputs ($(length(preds))). The calculated metric might not be accuracte.", + maxlog = 1) end for (label, pred) in zip(labels, preds) _update_single_output(metric, label, pred) @@ -64,11 +64,12 @@ function _update!(metric::T, labels::VecOfNDArray, preds::VecOfNDArray, end function _update!(metric::T, labels::VecOfNDArray, preds::VecOfNDArray, - ::Val{false}) where T<: AbstractEvalMetric + ::Val{false}) where {T<:AbstractEvalMetric} if length(labels) != length(preds) - Base.warn_once( - "The number of labels ($(length(labels))) does not correspond to the\ - number of outputs ($(length(preds))). The calculated metric might not be accuracte.") + @warn( + "The number of labels ($(length(labels))) does not correspond to the " * + "number of outputs ($(length(preds))). The calculated metric might not be accuracte.", + maxlog = 1) end for (label, pred) in zip(labels, preds) @nd_as_jl ro=(label, pred) begin @@ -124,19 +125,19 @@ To calculate both mean-squared error [`Accuracy`](@ref) and log-loss [`ACE`](@re ``` """ mutable struct MultiMetric <: AbstractEvalMetric - metrics :: Vector{mx.AbstractEvalMetric} + metrics :: Vector{mx.AbstractEvalMetric} end function update!(metric :: MultiMetric, labels :: Vector{<:NDArray}, preds :: Vector{<:NDArray}) - for m in metric.metrics - update!(m, labels, preds) - end - nothing + for m in metric.metrics + update!(m, labels, preds) + end + nothing end function reset!(metric :: MultiMetric) - map(reset!, metric.metrics) - nothing + map(reset!, metric.metrics) + nothing end get(metric::MultiMetric) = mapreduce(get, append!, metric.metrics) @@ -154,21 +155,21 @@ and log-loss [`ACE`](@ref) for the second output: ``` """ mutable struct SeqMetric <: AbstractEvalMetric - metrics :: Vector{AbstractEvalMetric} + metrics :: Vector{AbstractEvalMetric} end function update!(metric::SeqMetric, labels::VecOfNDArray, preds::VecOfNDArray) - @assert length(metric.metrics) == length(labels) - @assert length(metric.metrics) == length(preds) - for (m, l, p) in zip(metric.metrics, labels, preds) - update!(m, [l], [p]) - end - nothing + @assert length(metric.metrics) == length(labels) + @assert length(metric.metrics) == length(preds) + for (m, l, p) in zip(metric.metrics, labels, preds) + update!(m, [l], [p]) + end + nothing end function reset!(metric::SeqMetric) - map(reset!, metric.metrics) - nothing + map(reset!, metric.metrics) + nothing end get(metric::SeqMetric) = mapreduce(get, append!, metric.metrics) @@ -204,7 +205,7 @@ function _update_single_output(metric::Accuracy, label::Array, pred::Array) for i in 1:size(labels, 1) label = labels[i, j, 1, sample] klasses = view(pred, i, j, :, sample) - klass = indmax(klasses) - 1 # Classes start at 0...k-1 + klass = argmax(klasses) - 1 # Classes start at 0...k-1 metric.acc_sum += klass == label metric.n_sample += 1 @@ -213,7 +214,7 @@ function _update_single_output(metric::Accuracy, label::Array, pred::Array) end elseif ndims(pred) == 2 # 1-dimensional case for sample in 1:size(label, 1) - klass = indmax(view(pred, :, sample)) - 1 + klass = argmax(view(pred, :, sample)) - 1 metric.acc_sum += klass == label[sample] metric.n_sample += 1 end @@ -237,7 +238,6 @@ Mean Squared Error. Calculates the mean squared error regression loss. Requires that label and prediction have the same shape. """ - mutable struct MSE{N} <: AbstractEvalMetric mse_sum :: Vector{NDArray{MX_float,N}} n_sample :: Int @@ -269,7 +269,7 @@ function reset!(metric::MSE{N}) where N metric.n_sample = 0 end -doc""" +@doc doc""" NMSE Normalized Mean Squared Error diff --git a/julia/src/model.jl b/julia/src/model.jl index 109cb35e3..cb5f95e3c 100644 --- a/julia/src/model.jl +++ b/julia/src/model.jl @@ -38,7 +38,7 @@ mutable struct FeedForward <: AbstractModel arg_params :: Dict{Symbol} aux_params :: Dict{Symbol} - pred_exec :: Union{Executor,Void} + pred_exec :: Union{Executor,Cvoid} # leave the rest fields undefined FeedForward(arch::SymbolicNode, ctx::Vector{Context}) = new(arch, ctx) @@ -53,9 +53,9 @@ piece. function _split_inputs(batch_size::Int, n_split::Int) @assert(batch_size >= n_split) per_split = floor(Int, batch_size / n_split) - counts = Base.zeros(Int, n_split)+per_split + counts = Base.zeros(Int, n_split) .+ per_split extra = batch_size - Base.sum(counts) - counts[1:extra] += 1 + counts[1:extra] .+= 1 cum = [0, cumsum(counts)...] idx = [cum[i-1]+1:cum[i] for i = 2:length(cum)] @@ -118,7 +118,7 @@ function init_model(self::FeedForward, initializer::AbstractInitializer; overwri arg_params[name] = self.arg_params[name] continue else - warn("Shape mismatch for $name. Overwriting with new one.") + @warn("Shape mismatch for $name. Overwriting with new one.") delete!(self.arg_params, name) end end @@ -131,7 +131,7 @@ function init_model(self::FeedForward, initializer::AbstractInitializer; overwri aux_params[name] = self.aux_params[name] continue else - warn("Shape mismatch for $name. Overwriting with new one.") + @warn("Shape mismatch for $name. Overwriting with new one.") delete!(self.aux_params, name) end end @@ -156,7 +156,7 @@ function init_model(self::FeedForward, initializer::AbstractInitializer; overwri end function _setup_predictor(self::FeedForward, overwrite::Bool=false; verbosity::Integer = 1, data_shapes...) - if !isdefined(self, :pred_exec) || isa(self.pred_exec, Void) || overwrite + if !isdefined(self, :pred_exec) || isa(self.pred_exec, Cvoid) || overwrite if !isdefined(self, :arg_params) || !isdefined(self, :aux_params) @assert(false, "Model weights not defined, please init or train the model, or load from file") end @@ -164,7 +164,7 @@ function _setup_predictor(self::FeedForward, overwrite::Bool=false; verbosity::I # the predictor use only the first device self.pred_exec = simple_bind(self.arch, self.ctx[1]; grad_req=GRAD_NOP, data_shapes...) dbg_str = mx.debug_str(self.pred_exec) - verbosity >= 1 && info(string("TempSpace: ", split(dbg_str, ['\n'])[end-2]..., " on ", self.ctx[1])) + verbosity >= 1 && @info(string("TempSpace: ", split(dbg_str, ['\n'])[end-2]..., " on ", self.ctx[1])) copy_params_from(self.pred_exec, self.arg_params, self.aux_params) else # make sure the new setup is compatible with the existing one @@ -224,7 +224,7 @@ function predict(callback::Function, self::FeedForward, data::AbstractDataProvid predict(self, data; overwrite = overwrite, callback=callback, verbosity = verbosity) end function predict(self::FeedForward, data::AbstractDataProvider; - overwrite::Bool = true, callback::Union{Function,Void}=nothing, verbosity::Integer = 1) + overwrite::Bool = true, callback::Union{Function,Cvoid}=nothing, verbosity::Integer = 1) data_shapes = provide_data(data) data_names = [x[1] for x in data_shapes] _setup_predictor(self, overwrite; verbosity = verbosity, data_shapes...) @@ -235,7 +235,7 @@ function predict(self::FeedForward, data::AbstractDataProvider; for batch in eachbatch(data) load_data!(data, batch, data_arrays) forward(self.pred_exec, is_train=false) - if isa(callback, Void) + if isa(callback, Cvoid) # no callback, accumulate the data and return at the end for (o_list, o_nd) in zip(output_list, self.pred_exec.outputs) push!(o_list, copy(slice(o_nd, 1:count_samples(data, batch)))) @@ -249,7 +249,7 @@ function predict(self::FeedForward, data::AbstractDataProvider; end end - if !isa(callback, Void) + if !isa(callback, Cvoid) # callback exists, do not accumulate data return nothing end @@ -264,7 +264,7 @@ function predict(self::FeedForward, data::AbstractDataProvider; end # concatenate along mini-batches - output_arrays = [cat(ndims(x[1]), x...) for x in output_list] + output_arrays = [cat(x..., dims = ndims(x[1])) for x in output_list] if length(output_arrays) == 1 # only 1 output, return it directly, instead of a list output_arrays = output_arrays[1] @@ -279,7 +279,7 @@ function _init_model(self::FeedForward, data::AbstractDataProvider, end function _create_kvstore(kv_type::Symbol, num_device::Int, arg_params::Dict{Symbol}, verbosity::Int) - if num_device == 1 && !ismatch(r"dist", string(kv_type)) + if num_device == 1 && !occursin(r"dist", string(kv_type)) return nothing else if kv_type == :local @@ -289,7 +289,7 @@ function _create_kvstore(kv_type::Symbol, num_device::Int, arg_params::Dict{Symb else kv_type = :local_allreduce_cpu end - verbosity >= 2 && info("Auto-select kvstore type = $kv_type") + verbosity >= 2 && @info("Auto-select kvstore type = $kv_type") end return KVStore(kv_type) end @@ -298,7 +298,7 @@ end @defstruct TrainingOptions ( initializer :: AbstractInitializer = UniformInitializer(0.01), n_epoch :: Int = 10, - eval_data :: Union{Void,AbstractDataProvider} = nothing, + eval_data :: Union{Cvoid,AbstractDataProvider} = nothing, eval_metric :: AbstractEvalMetric = Accuracy(), kvstore :: Union{Symbol,KVStore} = :local, force_init :: Bool = false, @@ -364,25 +364,25 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData kwargs...) opts = TrainingOptions(; kwargs...) - opts.verbosity >= 1 && info("Start training on $(self.ctx)") + opts.verbosity >= 1 && @info("Start training on $(self.ctx)") batch_size = get_batch_size(data) num_dev = length(self.ctx) slices = _split_inputs(batch_size, num_dev) # initialize parameters - opts.verbosity >= 2 && info("Initializing parameters...") + opts.verbosity >= 2 && @info("Initializing parameters...") arg_names, param_names, aux_names = _init_model(self, data, opts.initializer, opts.force_init) # setup kvstore kvstore = opts.kvstore if isa(kvstore, Symbol) - opts.verbosity >= 2 && info("Creating KVStore...") + opts.verbosity >= 2 && @info("Creating KVStore...") kvstore = _create_kvstore(kvstore, length(self.ctx), self.arg_params, opts.verbosity) end update_on_kvstore = true - if isa(kvstore, Void) || ismatch(r"local_allreduce", string(get_type(kvstore))) + if isa(kvstore, Cvoid) || occursin(r"local_allreduce", string(get_type(kvstore))) update_on_kvstore = false end @@ -407,13 +407,13 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData end end - train_execs = Array{Executor}(num_dev) + train_execs = Array{Executor}(undef, num_dev) for i = 1:num_dev data_shapes = Dict(map((x) -> x[1] => tuple(x[2][1:end-1]...,length(slices[i])), provide_data(data))) label_shapes = Dict(map((x) -> x[1] => tuple(x[2][1:end-1]...,length(slices[i])), provide_label(data))) train_execs[i] = simple_bind(self.arch, self.ctx[i]; grad_req=grad_req, data_shapes..., label_shapes...) dbg_str = mx.debug_str(train_execs[i]) - opts.verbosity >= 2 && info(string("TempSpace: ", split(dbg_str, ['\n'])[end-2]..., " on ", self.ctx[i])) + opts.verbosity >= 2 && @info(string("TempSpace: ", split(dbg_str, ['\n'])[end-2]..., " on ", self.ctx[i])) copy_params_from(train_execs[i], self.arg_params, self.aux_params) end @@ -441,12 +441,12 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData updater = getupdater(optimizer) end - if !isa(kvstore, Void) + if !isa(kvstore, Cvoid) if update_on_kvstore set_optimizer(kvstore, optimizer) end - opts.verbosity >= 2 && info("Initializing KVStore...") + opts.verbosity >= 2 && @info("Initializing KVStore...") # init kv with gradients for idx = 1:length(param_arrays) param_on_devs = param_arrays[idx] @@ -469,7 +469,7 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData # invoke callbacks on epoch 0 _invoke_callbacks(self, opts.callbacks, op_state, AbstractEpochCallback) - opts.verbosity >= 2 && info("Start training...") + opts.verbosity >= 2 && @info("Start training...") for i_epoch = 1:opts.n_epoch time_start = time() reset!(opts.eval_metric) @@ -506,7 +506,7 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData end # gradient synchronization - if !isa(kvstore, Void) + if !isa(kvstore, Cvoid) # push gradient, priority is negative index push!(kvstore, idx, grad_arrays[idx], priority=-idx) if update_on_kvstore @@ -543,17 +543,17 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData time_stop = time() metric = get(opts.eval_metric) - opts.verbosity >= 2 && info(format("== Epoch {1:0>3d}/{2:0>3d} ==========", i_epoch, opts.n_epoch)) + opts.verbosity >= 2 && @info(format("== Epoch {1:0>3d}/{2:0>3d} ==========", i_epoch, opts.n_epoch)) if opts.verbosity >= 3 - info("## Training summary") + @info("## Training summary") for (name, value) in metric - info(format("{1:>18s} = {2:.4f}", string(name), value)) + @info(format("{1:>18s} = {2:.4f}", string(name), value)) end - info(format("{1:>18s} = {2:.4f} seconds", "time", time_stop-time_start)) + @info(format("{1:>18s} = {2:.4f} seconds", "time", time_stop-time_start)) end # evaluation on validation set - if !isa(opts.eval_data, Void) + if !isa(opts.eval_data, Cvoid) # because we are re-using the memory allocated for the training network, # the batch_size of the validation dataset must be the same as the training # batch_size @@ -577,9 +577,9 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData end if opts.verbosity >= 3 - info("## Validation summary") + @info("## Validation summary") for (name, value) in get(opts.eval_metric) - info(format("{1:>18s} = {2:.4f}", string(name), value)) + @info(format("{1:>18s} = {2:.4f}", string(name), value)) end end end @@ -603,7 +603,7 @@ function fit(self::FeedForward, optimizer::AbstractOptimizer, data::AbstractData _invoke_callbacks(self, opts.callbacks, op_state, AbstractEpochCallback; metric=metric) end # end of all epochs - opts.verbosity >= 1 && info("Finish training on $(self.ctx)") + opts.verbosity >= 1 && @info("Finish training on $(self.ctx)") nothing end @@ -613,13 +613,15 @@ save_checkpoint(self::FeedForward, prefix::AbstractString, state::OptimizationSt function save_checkpoint(sym::SymbolicNode, arg_params::Dict{Symbol}, aux_params::Dict{Symbol}, prefix::AbstractString, epoch::Int) save("$prefix-symbol.json", sym) - save_dict = Dict{Symbol, NDArray}(map((x) -> Symbol("arg:$(x[1])") => x[2], arg_params)) + save_dict = Dict{Symbol,NDArray}( + Symbol("arg:$(x[1])") => x[2] for x in arg_params + ) if !isempty(aux_params) merge!(save_dict, Dict(map((x) -> Symbol("aux:$(x[1])") => x[2], aux_params))) end save_filename = format("{1}-{2:04d}.params", prefix, epoch) save(save_filename, save_dict) - info("Saved checkpoint to '$save_filename'") + @info("Saved checkpoint to '$save_filename'") end function load_checkpoint(prefix::AbstractString, epoch::Int) @@ -656,7 +658,7 @@ end function load_checkpoint(self::FeedForward, prefix::AbstractString, epoch::Int; overwrite::Bool = true, allow_different_arch::Bool = false) if isdefined(self, :arg_params) && isdefined(self, :aux_params) && !overwrite - info("model weights already exists, skip loading... (call with overwrite=true if needed)") + @info("model weights already exists, skip loading... (call with overwrite=true if needed)") return self end diff --git a/julia/src/ndarray.jl b/julia/src/ndarray.jl index 9e47150a1..dad9b59e8 100644 --- a/julia/src/ndarray.jl +++ b/julia/src/ndarray.jl @@ -61,22 +61,22 @@ function fromTypeFlag(T::TypeFlag) end # create a NDArray handle of specific shape -function _ndarray_alloc(shape :: NTuple{N, Int}, ctx :: Context, delay_alloc :: Bool) where N +function _ndarray_alloc(shape::NTuple{N,Int}, ctx::Context, delay_alloc::Bool) where N h_ref = Ref{MX_handle}(0) - shape = flipdim(MX_uint[shape...],1) + shape = collect(reverse(MX_uint.(shape))) @mxcall(:MXNDArrayCreate, (Ptr{MX_uint}, MX_uint, Cint, Cint, Cint, Ref{MX_handle}), - shape, length(shape), ctx.device_type, ctx.device_id, delay_alloc, h_ref) + shape, N, ctx.device_type, ctx.device_id, delay_alloc, h_ref) handle = MX_NDArrayHandle(h_ref[]) return handle end # create a NDArray handle of specific shape type -function _ndarray_alloc(:: Type{T}, shape :: NTuple{N, Int}, ctx :: Context, delay_alloc :: Bool) where {T <: DType,N} +function _ndarray_alloc(::Type{T}, shape::NTuple{N,Int}, ctx::Context, delay_alloc::Bool) where {T<:DType,N} h_ref = Ref{MX_handle}(0) - shape = flipdim(MX_uint[shape...],1) + shape = collect(reverse(MX_uint.(shape))) dtype = toTypeFlag(T) @mxcall(:MXNDArrayCreateEx, (Ptr{MX_uint}, MX_uint, Cint, Cint, Cint, Cint, Ref{MX_handle}), - shape, length(shape), ctx.device_type, ctx.device_id, delay_alloc, dtype, h_ref) + shape, N, ctx.device_type, ctx.device_id, delay_alloc, dtype, h_ref) handle = MX_NDArrayHandle(h_ref[]) return handle end @@ -113,51 +113,53 @@ mutable struct NDArray{T,N} NDArray{T,N}(handle, writable = true) where {T,N} = new(handle, writable) end -NDArray(x::AbstractArray{T}) where {T<:DType} = copy(collect(x), cpu()) -NDArray(x::Array{T}) where {T<:DType} = copy(x, cpu()) +NDArray(x::AbstractArray{<:DType}) = copy(collect(x), cpu()) +NDArray(x::Array{<:DType}) = copy(x, cpu()) NDArray(::Type{T}, x::AbstractArray) where {T<:DType} = copy(convert(AbstractArray{T}, x), cpu()) NDArray(handle, writable = true) = NDArray{eltype(handle), ndims(handle)}(handle, writable) # type aliases -const NDArrayOrReal = Union{NDArray, Real} +const NDArrayOrReal = Union{NDArray,Real} const VecOfNDArray = AbstractVector{<:NDArray} -@unfuse NDArray - function Base.show(io::IO, x::NDArray) - print(io, "NDArray ") - Base.showarray(io, try_get_shared(x, sync = :read), header = false) + print(io, "NDArray(") + Base.show(io, try_get_shared(x, sync = :read)) + print(io, ")") end # for REPL -function Base.show(io::IO, ::MIME{Symbol("text/plain")}, x::NDArray{T, N}) where {T, N} +function Base.show(io::IO, ::MIME{Symbol("text/plain")}, x::NDArray{T,N}) where {T,N} type_ = split(string(typeof(x)), '.', limit=2)[end] - size_ = N == 1 ? "$(length(x))-element" : join(size(x), "×") - println(io, "$size_ $type_ @ $(context(x)):") - Base.showarray(io, try_get_shared(x, sync = :read), false, header = false) + n = length(x) + size_ = N == 1 ? "$n-element" : join(size(x), "×") + print(io, "$size_ $type_ @ $(context(x))", (n == 0) ? "" : ":\n") + Base.print_array(io, try_get_shared(x, sync = :read)) end -Base.unsafe_convert(::Type{MX_handle}, obj::NDArray) = - Base.unsafe_convert(MX_handle, obj.handle) -Base.convert(T::Type{MX_handle}, obj::NDArray) = Base.unsafe_convert(T, obj) -Base.cconvert(T::Type{MX_handle}, obj::NDArray) = Base.unsafe_convert(T, obj) +Base.unsafe_convert(::Type{MX_handle}, x::NDArray) = + Base.unsafe_convert(MX_handle, x.handle) +Base.convert(T::Type{MX_handle}, x::NDArray) = Base.unsafe_convert(T, x) +Base.cconvert(T::Type{MX_handle}, x::NDArray) = Base.unsafe_convert(T, x) + +MX_handle(x::NDArray) = Base.convert(MX_handle, x) ################################################################################ # NDArray functions exported to the users ################################################################################ """ - context(arr::NDArray) + context(x::NDArray) Get the context that this `NDArray` lives on. """ -function context(arr::NDArray) +function context(x::NDArray) ref_typeid = Ref{Cint}(0) ref_devid = Ref{Cint}(0) @mxcall(:MXNDArrayGetContext, (MX_handle, Ref{Cint}, Ref{Cint}), - arr, ref_typeid, ref_devid) - return Context(ref_typeid[], ref_devid[]) + x, ref_typeid, ref_devid) + Context(ref_typeid[], ref_devid[]) end """ @@ -168,7 +170,7 @@ end Allocate memory for an uninitialized `NDArray` with a specified type. """ empty(::Type{T}, dims::NTuple{N,Int}, ctx::Context = cpu()) where {N,T<:DType} = - NDArray{T, N}(_ndarray_alloc(T, dims, ctx, false)) + NDArray{T,N}(_ndarray_alloc(T, dims, ctx, false)) empty(::Type{T}, dims::Int...) where {T<:DType} = empty(T, dims) """ @@ -198,9 +200,9 @@ Base.similar(x::NDArray{T}) where {T} = empty(T, size(x), context(x)) Create zero-ed `NDArray` with specific shape and type. """ function zeros(::Type{T}, dims::NTuple{N,Int}, ctx::Context = cpu()) where {N,T<:DType} - arr = empty(T, dims, ctx) - arr[:] = zero(T) - arr + x = empty(T, dims, ctx) + x[:] = zero(T) + x end zeros(::Type{T}, dims::Int...) where {T<:DType} = zeros(T, dims) @@ -234,32 +236,24 @@ ones(dims::Int...) = ones(dims) ones(x::NDArray)::typeof(x) = ones_like(x) Base.ones(x::NDArray)::typeof(x) = ones_like(x) -import Base: size, length, ndims, eltype +import Base: length, ndims """ size(x::NDArray) - size(x::NDArray, dims...) + size(x::NDArray, dims) Get the shape of an `NDArray`. The shape is in Julia's column-major convention. See also the notes on NDArray shapes [`NDArray`](@ref). """ -function size(x::NDArray) +function Base.size(x::NDArray) ref_ndim = Ref{MX_uint}(0) ref_shape = Ref{Ptr{MX_uint}}(0) @mxcall(:MXNDArrayGetShape, (MX_handle, Ref{MX_uint}, Ref{Ptr{MX_uint}}), x, ref_ndim, ref_shape) - tuple(map(Int, flipdim(unsafe_wrap(Array, ref_shape[], ref_ndim[]),1))...) + tuple(map(Int, reverse(unsafe_wrap(Array, ref_shape[], ref_ndim[])))...) end -function size(x::NDArray{T,N}, dim::Int) where {T,N} - if dim > N - 1 - else - size(x)[dim] - end -end - -size(x::NDArray, dims::Int...) = map(d -> size(x, d), dims) +Base.size(x::NDArray{T,N}, dims::Integer) where {T,N} = (dims > N) ? 1 : size(x)[dims] """ length(x::NDArray) @@ -289,25 +283,23 @@ end Get the element type of an `NDArray`. """ -function eltype(x::Union{NDArray, MX_NDArrayHandle}) +function Base.eltype(x::Union{NDArray,MX_NDArrayHandle}) dtype_ref = Ref{Cint}(0) @mxcall(:MXNDArrayGetDType, (MX_handle, Ptr{Cint}), x, dtype_ref) if dtype_ref[] == -1 # x->is_none() - warn("Eltype of $x is not defined") - Base.show_backtrace(STDOUT, backtrace()) - println() - Float32 - else - fromTypeFlag(TypeFlag(dtype_ref[])) + # TODO: unit test for this branch + throw(MXError("Eltype of $x is not defined")) end + + fromTypeFlag(TypeFlag(dtype_ref[])) end @inline _first(x::NDArray) = try_get_shared(x, sync = :read) |> first Base.first(x::NDArray) = _first(x) -Base.endof(x::NDArray) = length(x) +Base.lastindex(x::NDArray) = length(x) """ slice(arr :: NDArray, start:stop) @@ -456,7 +448,7 @@ Copy contents of `src` into `dst`. function copy!(dst::NDArray, src::NDArray) @assert(dst.writable) if dst.handle == src.handle - warn("Copying an NDArray to itself") + @warn("Copying an NDArray to itself") return end @@ -466,7 +458,7 @@ end function copy!(dst::Array{T}, src::NDArray{T}) where T<:DType @assert size(dst) == size(src) - @mxcall(:MXNDArraySyncCopyToCPU, (MX_handle, Ptr{Void}, Csize_t), + @mxcall(:MXNDArraySyncCopyToCPU, (MX_handle, Ptr{Cvoid}, Csize_t), src, pointer(dst), length(dst)) dst end @@ -478,7 +470,7 @@ function copy!(dst::NDArray{T}, src::Array{<:Real}) where {T} @assert dst.writable @assert size(dst) == size(src) src = convert(Array{T}, src) # this might involve copying - @mxcall(:MXNDArraySyncCopyFromCPU, (MX_handle, Ptr{Void}, Csize_t), + @mxcall(:MXNDArraySyncCopyFromCPU, (MX_handle, Ptr{Cvoid}, Csize_t), dst.handle, pointer(src), length(src)) dst end @@ -487,7 +479,7 @@ function copy_ignore_shape!(dst::NDArray{T}, src::Array{<:Real}) where {T} @assert dst.writable @assert length(dst) == length(src) src = convert(Array{T}, src) # this might involve copying - @mxcall(:MXNDArraySyncCopyFromCPU, (MX_handle, Ptr{Void}, Csize_t), + @mxcall(:MXNDArraySyncCopyFromCPU, (MX_handle, Ptr{Cvoid}, Csize_t), dst.handle, pointer(src), length(src)) dst end @@ -501,8 +493,10 @@ end Create a copy of an array. When no `Context` is given, create a Julia `Array`. Otherwise, create an `NDArray` on the specified context. """ +copy + # Create copy: NDArray -> Julia Array -copy(x::NDArray{T,D}) where{T,D} = copy!(Array{T,D}(size(x)), x) +copy(x::NDArray{T,D}) where{T,D} = copy!(Array{T,D}(undef, size(x)), x) # Create copy: NDArray -> NDArray in a given context copy(x::NDArray{T,D}, ctx::Context) where {T,D} = @@ -538,27 +532,27 @@ end """ hcat(x::NDArray...) """ -Base.hcat(xs::NDArray{T}...) where T = cat(2, xs...) +Base.hcat(xs::NDArray{T}...) where T = cat(xs..., dims = 2) """ vcat(x::NDArray...) """ -Base.vcat(xs::NDArray{T}...) where T = cat(1, xs...) +Base.vcat(xs::NDArray{T}...) where T = cat(xs..., dims = 1) """ - cat(dim, xs::NDArray...) + cat(xs::NDArray...; dims) -Concate the `NDArray`s which have the same element type along the `dim`. +Concate the `NDArray`s which have the same element type along the `dims`. Building a diagonal matrix is not supported yet. """ -function Base.cat(dim::Int, xs::NDArray{T}...) where T +function Base.cat(xs::NDArray{T}...; dims) where T ns = ndims.(xs) - d = Base.max(dim, maximum(ns)) + d = Base.max(dims, maximum(ns)) xs′ = map(zip(ns, xs)) do i n, x = i (d > n) ? reshape(x, -2, Base.ones(Int, d - n)...) : x end - concat(xs′..., dim = d - dim) + concat(xs′..., dim = d - dims) end """ @@ -633,11 +627,7 @@ added together. Note at least the first or second argument needs to be an +(x::NDArray, y::Real) = _plus_scalar(x, scalar = y) +(y::Real, x::NDArray) = _plus_scalar(x, scalar = y) -broadcast_(::typeof(+), x::NDArray, y::Real) = x + y -broadcast_(::typeof(+), x::Real, y::NDArray) = x + y - -broadcast_(::typeof(+), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = x + y -broadcast_(::typeof(+), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(+), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_add(x, y) """ @@ -665,16 +655,12 @@ import Base: - Subtraction `x - y`, of scalar types or `NDArray`. Or create the negative of `x`. """ --(x::NDArray) = _mul_scalar(x, scalar = -one(eltype(x))) +-(x::NDArray) = _mul_scalar(x, scalar = -one(eltype(x))) -(x::NDArray, y::NDArray) = _minus(x, y) -(x::NDArray, y::Real) = _minus_scalar(x, scalar = y) -(y::Real, x::NDArray) = _rminus_scalar(x, scalar = y) -broadcast_(::typeof(-), x::NDArray, y::Real) = x - y -broadcast_(::typeof(-), x::Real, y::NDArray) = x - y - -broadcast_(::typeof(-), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = x - y -broadcast_(::typeof(-), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(-), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_minus(x, y) """ @@ -703,12 +689,9 @@ Elementwise multiplication for `NDArray`. *(x::NDArray, y::Real) = _mul_scalar(x, scalar = y) *(y::Real, x::NDArray) = _mul_scalar(x, scalar = y) -broadcast_(::typeof(*), x::NDArray, y::Real) = x * y -broadcast_(::typeof(*), y::Real, x::NDArray) = x * y - -broadcast_(::typeof(*), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = +broadcasted(::typeof(*), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = _mul(x, y) -broadcast_(::typeof(*), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(*), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_mul(x, y) """ @@ -718,6 +701,9 @@ Matrix/tensor multiplication. """ *(x::NDArray{T}, y::NDArray{T}) where T = x ⋅ y +LinearAlgebra.adjoint(x::NDArray{T,1}) where T = transpose(x) +LinearAlgebra.adjoint(x::NDArray{T,2}) where T = transpose(x) + """ div_from!(dst::NDArray, arg::NDArrayOrReal) @@ -767,15 +753,13 @@ of the same shape. """ /(x::NDArray, y::Real) = _div_scalar(x, scalar = y) -broadcast_(::typeof(/), x::NDArray, y::Real) = _div_scalar(x, scalar = y) -broadcast_(::typeof(/), y::Real, x::NDArray) = _rdiv_scalar(x, scalar = y) - -broadcast_(::typeof(/), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = +broadcasted(::typeof(/), y::Real, x::NDArray) = _rdiv_scalar(x, scalar = y) +broadcasted(::typeof(/), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = _div(x, y) -broadcast_(::typeof(/), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(/), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_div(x, y) -function broadcast_(::typeof(/), x::NDArray{T}, y::Real) where {T<:Integer} +function broadcasted(::typeof(/), x::NDArray{T}, y::Real) where {T<:Integer} @assert(round(T, y) != zero(T), "Integer divided by zero") _div_scalar(x, scalar = y) end @@ -807,52 +791,50 @@ import Base: % Elementwise modulo for `NDArray`. """ -%(x::NDArray, y::Real) = _mod_scalar(x, scalar = y) - -broadcast_(::typeof(%), x::NDArray, y::Real) = _mod_scalar(x, y) -broadcast_(::typeof(%), y::Real, x::NDArray) = _rmod_scalar(x, y) +%(x::NDArray, y::Real) = _mod_scalar(x, y) -broadcast_(::typeof(%), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = +broadcasted(::typeof(%), y::Real, x::NDArray) = _rmod_scalar(x, y) +broadcasted(::typeof(%), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = _mod(x, y) -broadcast_(::typeof(%), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(%), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_mod(x, y) -import Base: ^ - # document of `.^` is merged into SymbolicNode's -broadcast_(::typeof(^), x::NDArray, s::Real) = _power_scalar(x, scalar = s) -broadcast_(::typeof(^), s::Real, x::NDArray) = _rpower_scalar(x, scalar = s) +broadcasted(::typeof(Base.literal_pow), ::typeof(^), x::NDArray, ::Val{s}) where {s} = + _power_scalar(x, scalar = s) +broadcasted(::typeof(^), x::NDArray, s::Real) = _power_scalar(x, scalar = s) +broadcasted(::typeof(^), s::Real, x::NDArray) = _rpower_scalar(x, scalar = s) -broadcast_(::typeof(^), ::Irrational{:e}, x::NDArray) = exp(x) -broadcast_(::typeof(^), x::NDArray, s::Irrational) = _power_scalar(x, scalar = s) -broadcast_(::typeof(^), s::Irrational, x::NDArray) = _rpower_scalar(x, scalar = s) +broadcasted(::typeof(^), ::Irrational{:ℯ}, x::NDArray) = exp(x) +broadcasted(::typeof(^), x::NDArray, s::Irrational) = _power_scalar(x, scalar = s) +broadcasted(::typeof(^), s::Irrational, x::NDArray) = _rpower_scalar(x, scalar = s) -broadcast_(::typeof(^), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = +broadcasted(::typeof(^), x::NDArray{T,N}, y::NDArray{T,N}) where {T,N} = _power(x, y) -broadcast_(::typeof(^), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = +broadcasted(::typeof(^), x::NDArray{T,N}, y::NDArray{T,M}) where {T,N,M} = _broadcast_power(x, y) ############################################################################### # comparison ############################################################################### -broadcast_(::typeof(==), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(==), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_equal(x, y) -broadcast_(::typeof(!=), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(!=), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_not_equal(x, y) -broadcast_(::typeof(>), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(>), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_greater(x, y) -broadcast_(::typeof(>=), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(>=), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_greater_equal(x, y) -broadcast_(::typeof(<), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(<), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_lesser(x, y) -broadcast_(::typeof(<=), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(<=), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_lesser_equal(x, y) @@ -862,10 +844,10 @@ broadcast_(::typeof(<=), x::NDArray{T}, y::NDArray{T}) where {T} = import Base: min, max -broadcast_(::typeof(max), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(max), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_maximum(x, y) -broadcast_(::typeof(min), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(min), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_minimum(x, y) """ @@ -894,7 +876,7 @@ fill(x, dims::Integer...) = fill(x, dims) import Base: hypot -broadcast_(::typeof(hypot), x::NDArray{T}, y::NDArray{T}) where {T} = +broadcasted(::typeof(hypot), x::NDArray{T}, y::NDArray{T}) where {T} = _broadcast_hypot(x, y) """ @@ -985,7 +967,7 @@ macro nd_as_jl(m_args...) wait_statements = Expr(:block, [:(_wait_to_read($v)) for v in nd_ro]..., [:(_wait_to_write($v)) for v in nd_rw]...) clear_statements = Expr(:block, [:($v_orig = nothing) for v_orig in rw_origs]...) - let_assignments = [:($v = try_get_shared($v)) for v in nd_all] + let_assignments = Expr(:block, [:($v = try_get_shared($v)) for v in nd_all]...) sync_statements = map(rw_origs, nd_rw) do v_orig, v quote if !is_shared($v, $v_orig) @@ -996,10 +978,10 @@ macro nd_as_jl(m_args...) end sync_statements = Expr(:block, sync_statements...) - let_statement = Expr(:let, quote + let_statement = Expr(:let, let_assignments, quote $stmts $sync_statements - end, let_assignments...) + end) m_body = quote $wait_statements $save_statements @@ -1015,8 +997,8 @@ end # pointers from CPU) leads to undefined behavior. import Base.pointer function pointer(arr :: NDArray) - pdata = Ref{Ptr{Void}}(0) - @mxcall(:MXNDArrayGetData, (MX_handle, Ref{Ptr{Void}}), arr, pdata) + pdata = Ref{Ptr{Cvoid}}(0) + @mxcall(:MXNDArrayGetData, (MX_handle, Ref{Ptr{Cvoid}}), arr, pdata) return convert(Ptr{eltype(arr)}, pdata[]) end @@ -1146,18 +1128,6 @@ end const _ndsig = Dict{Symbol,Expr}() const _nddoc = Dict{Symbol,Any}() -function _autoimport(name::Symbol, sig::Expr) - if name == :broadcast_ - name = _broadcast_target(sig) - end - - if isdefined(Base, name) - :(import Base: $name) - else - :() - end -end - _isinplace(name::Symbol) = endswith(string(name), "!") _writable(name::Symbol, x) = @@ -1178,7 +1148,7 @@ _broadcast_target(sig::Expr) = sig.args[2].args[].args[end] Generate docstring from function signature """ function _docsig(fname::Symbol, sig::Expr, opname::String) - if fname !== :broadcast_ + if fname !== :broadcasted get(_nddoc, fname, " $sig") * "\n" * _getdocdefine(opname) else name = _broadcast_target(sig) @@ -1205,11 +1175,12 @@ function _docsig(fname::Symbol, sig::Expr, opname::String) end end + macro _remap(sig::Expr, imp::Expr) - fname = (sig.head == :call) ? sig.args[1] : sig.args[1].args[1] # case of `where` - opname = string(imp.args[1]) + d = splitdef(:($sig = $imp)) + @capture d[:name] (M_.fname_|fname_) - import_expr = _autoimport(fname, sig) + opname = string(imp.args[1]) if isa(imp.args[2], Expr) && imp.args[2].head == :parameters ndin = imp.args[3:end] @@ -1257,8 +1228,7 @@ macro _remap(sig::Expr, imp::Expr) func_def = Expr(:function, sig, func_body) esc(quote - $import_expr - @doc $docstr -> + @doc $docstr $func_def end) end @@ -1271,32 +1241,37 @@ macro _remap(sig::Expr, imp::Symbol) end) end -_ndsig[:reshape] = :(reshape(arr; shape = dim, reverse = !reverse)) -@_remap reshape(arr::NDArray, dim...; reverse = false) reshape -@_remap reshape(arr::NDArray, dim; reverse = false) reshape +_ndsig[:reshape] = :(reshape(x; shape = dim, reverse = !reverse)) +@_remap Base.reshape(x::NDArray, dim...; reverse = false) reshape +@_remap Base.reshape(x::NDArray, dim ; reverse = false) reshape -@_remap mean(arr::NDArray) mean(arr) -@_remap mean(arr::NDArray, region) mean(arr; axis = 0 .- region, keepdims = true) +Statistics.mean(x::NDArray; dims = :) = _mean(x, dims) +@_remap _mean(x::NDArray, ::Colon) mean(x) +@_remap _mean(x::NDArray, dims) mean(x; axis = 0 .- dims, keepdims = true) -@_remap sum(arr::NDArray) sum(arr) -@_remap sum(arr::NDArray, dims) sum(arr; axis = 0 .- dims, keepdims = true) +Base.sum(x::NDArray; dims = :) = _sum(x, dims) +@_remap _sum(x::NDArray, ::Colon) sum(x) +@_remap _sum(x::NDArray, dims) sum(x; axis = 0 .- dims, keepdims = true) -@_remap maximum(arr::NDArray) max(arr) -@_remap maximum(arr::NDArray, dims) max(arr; axis = 0 .- dims, keepdims = true) +Base.maximum(x::NDArray; dims = :) = _nd_maximum(x, dims) +@_remap _nd_maximum(x::NDArray, ::Colon) max(x) +@_remap _nd_maximum(x::NDArray, dims) max(x; axis = 0 .- dims, keepdims = true) -@_remap minimum(arr::NDArray) min(arr) -@_remap minimum(arr::NDArray, dims) min(arr; axis = 0 .- dims, keepdims = true) +Base.minimum(x::NDArray; dims = :) = _nd_minimum(x, dims) +@_remap _nd_minimum(x::NDArray, ::Colon) min(x) +@_remap _nd_minimum(x::NDArray, dims) min(x; axis = 0 .- dims, keepdims = true) # See https://github.com/dmlc/MXNet.jl/issues/55 -@_remap dot(x::NDArray, y::NDArray) dot(y, x) +@_remap LinearAlgebra.dot(x::NDArray, y::NDArray) dot(y, x) # See https://github.com/dmlc/MXNet.jl/pull/123 -@_remap transpose(arr::NDArray{T,1}) where T reshape(arr; shape = (1, length(arr)), reverse = true) -@_remap transpose(arr::NDArray{T,2}) where T transpose(arr) -@_remap permutedims(arr::NDArray, axes) transpose(arr; axes = length(axes) .- tuple(axes...)) +@_remap Base.transpose(x::NDArray{T,1}) where T reshape(x; shape = (1, length(x)), reverse = true) +@_remap Base.transpose(x::NDArray{T,2}) where T transpose(x) +@_remap Base.permutedims(x::NDArray, axes) transpose(x; axes = length(axes) .- tuple(axes...)) -@_remap prod(arr::NDArray) prod(arr) -@_remap prod(arr::NDArray, dims) prod(arr; axis = 0 .- dims, keepdims = true) +Base.prod(x::NDArray; dims = :) = _prod(x, dims) +@_remap _prod(x::NDArray, ::Colon) prod(x) +@_remap _prod(x::NDArray, dims) prod(x; axis = 0 .- dims, keepdims = true) _nddoc[:clip] = _nddoc[:clip!] = """ @@ -1362,23 +1337,23 @@ julia> mx.expand_dims(x, 2) @_remap expand_dims(x::NDArray, dim) expand_dims(x; axis = -dim) # trigonometric functions, remap to keep consistent with Base -@_remap broadcast_(::typeof(sin), x::NDArray) sin(x) -@_remap broadcast_(::typeof(cos), x::NDArray) cos(x) -@_remap broadcast_(::typeof(tan), x::NDArray) tan(x) -@_remap broadcast_(::typeof(asin), x::NDArray) arcsin(x) -@_remap broadcast_(::typeof(acos), x::NDArray) arccos(x) -@_remap broadcast_(::typeof(atan), x::NDArray) arctan(x) +@_remap broadcasted(::typeof(sin), x::NDArray) sin(x) +@_remap broadcasted(::typeof(cos), x::NDArray) cos(x) +@_remap broadcasted(::typeof(tan), x::NDArray) tan(x) +@_remap broadcasted(::typeof(asin), x::NDArray) arcsin(x) +@_remap broadcasted(::typeof(acos), x::NDArray) arccos(x) +@_remap broadcasted(::typeof(atan), x::NDArray) arctan(x) # hyperbolic funcs, remap to keep consistent with Base -@_remap broadcast_(::typeof(sinh), x::NDArray) sinh(x) -@_remap broadcast_(::typeof(cosh), x::NDArray) cosh(x) -@_remap broadcast_(::typeof(tanh), x::NDArray) tanh(x) -@_remap broadcast_(::typeof(asinh), x::NDArray) arcsinh(x) -@_remap broadcast_(::typeof(acosh), x::NDArray) arccosh(x) -@_remap broadcast_(::typeof(atanh), x::NDArray) arctanh(x) +@_remap broadcasted(::typeof(sinh), x::NDArray) sinh(x) +@_remap broadcasted(::typeof(cosh), x::NDArray) cosh(x) +@_remap broadcasted(::typeof(tanh), x::NDArray) tanh(x) +@_remap broadcasted(::typeof(asinh), x::NDArray) arcsinh(x) +@_remap broadcasted(::typeof(acosh), x::NDArray) arccosh(x) +@_remap broadcasted(::typeof(atanh), x::NDArray) arctanh(x) # activation functions -_nddoc[:σ] = _nddoc[:sigmoid] = doc""" +@doc doc""" σ.(x::NDArray) sigmoid.(x::NDArray) @@ -1390,10 +1365,12 @@ Computes sigmoid of x element-wise. The storage type of `sigmoid` output is always dense. """ -@_remap broadcast_(::typeof(σ), x::NDArray) sigmoid(x) -@_remap broadcast_(::typeof(sigmoid), x::NDArray) sigmoid(x) +function σ end +const sigmoid = σ +_nddoc[:σ] = false +@_remap broadcasted(::typeof(σ), x::NDArray) sigmoid(x) -_nddoc[:relu] = doc""" +@doc doc""" relu.(x::NDArray) Computes rectified linear. @@ -1402,9 +1379,11 @@ Computes rectified linear. \max(x, 0) ``` """ -@_remap broadcast_(::typeof(relu), x::NDArray) relu(x) +function relu end +_nddoc[:relu] = false +@_remap broadcasted(::typeof(relu), x::NDArray) relu(x) -_nddoc[:softmax] = doc""" +@doc doc""" softmax.(x::NDArray, [dim = ndims(x)]) Applies the softmax function. @@ -1416,10 +1395,12 @@ and the elements along the given axis sum up to 1. softmax(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}} ``` """ -@_remap broadcast_(::typeof(softmax), x::NDArray) softmax(x; axis = -ndims(x)) -@_remap broadcast_(::typeof(softmax), x::NDArray, dim::Int) softmax(x; axis = -dim) +function softmax end +_nddoc[:softmax] = false +@_remap broadcasted(::typeof(softmax), x::NDArray) softmax(x; axis = -ndims(x)) +@_remap broadcasted(::typeof(softmax), x::NDArray, dim::Int) softmax(x; axis = -dim) -_nddoc[:log_softmax] = """ +""" log_softmax.(x::NDArray, [dim = ndims(x)]) Computes the log softmax of the input. @@ -1435,8 +1416,10 @@ julia> mx.log_softmax.(x) -1.41703 -0.41703 -2.31703 -2.31703 -0.41703 -1.41703 """ -@_remap broadcast_(::typeof(log_softmax), x::NDArray) log_softmax(x; axis = -ndims(x)) -@_remap broadcast_(::typeof(log_softmax), x::NDArray, dim::Int) log_softmax(x; axis = -dim) +function log_softmax end +_nddoc[:log_softmax] = false +@_remap broadcasted(::typeof(log_softmax), x::NDArray) log_softmax(x; axis = -ndims(x)) +@_remap broadcasted(::typeof(log_softmax), x::NDArray, dim::Int) log_softmax(x; axis = -dim) ################################################################################ # remapping to solving type unstablility @@ -1570,7 +1553,7 @@ julia> mx.broadcast_axis(x, 3, 2) """ @_remap(broadcast_axis(x::NDArray, dim, size), broadcast_axis(x; axis = ndims(x) .- dim, size = size)) -@_remap(broadcast_axes(x::NDArray, dim, size), +@_remap(Base.broadcast_axes(x::NDArray, dim, size), broadcast_axes(x; axis = ndims(x) .- dim, size = size)) ################################################################################ @@ -1619,7 +1602,7 @@ Upon calling, the output arguments will be automatically initialized with empty Those functions always return the output arguments. If there is only one output (the typical situation), that object (`NDArray`) is returned. Otherwise, a tuple containing all the outputs will be returned. """ -function _get_ndarray_function_def(name :: String) +function _get_ndarray_function_def(name::String) func_name = Symbol(name) func_def = quote @@ -1763,11 +1746,13 @@ macro _import_ndarray_functions() func_def, func_def2 = _get_ndarray_function_def(name) func_name = Symbol(name) - expr = quote - # TODO the explicit exclusion of take will no longer be necessary when it is removed from Base - $((isdefined(Base, func_name) && func_name ≠ :take) ? :(import Base.$func_name) : :()) + + import_expr = _import_expr(func_name) + + quote + $import_expr $func_def - @doc $desc -> + @doc $desc $func_def2 end end @@ -1777,4 +1762,4 @@ macro _import_ndarray_functions() end) end -@_import_ndarray_functions() +@_import_ndarray_functions diff --git a/julia/src/nn-factory.jl b/julia/src/nn-factory.jl index b5134f9d2..cb5df3722 100644 --- a/julia/src/nn-factory.jl +++ b/julia/src/nn-factory.jl @@ -38,7 +38,7 @@ fully connected layers. Returns the constructed MLP. """ -function MLP(input, spec; hidden_activation::Symbol=:relu, prefix=gensym()) +function MLP(input, spec; hidden_activation::Symbol = :relu, prefix = gensym()) spec = convert(Vector{Union{Int,Tuple}}, spec) n_layer = length(spec) diff --git a/julia/src/optimizer.jl b/julia/src/optimizer.jl index 6436878df..46726500f 100644 --- a/julia/src/optimizer.jl +++ b/julia/src/optimizer.jl @@ -73,6 +73,8 @@ OptimizationState(batch_size::Int) = OptimizationState(batch_size, 0, 0, 0) module LearningRate +using Markdown + import Base: get import ..mx: AbstractLearningRateScheduler, OptimizationState, update! @@ -100,7 +102,7 @@ end get(f::Fixed) = f.η -doc""" +@doc doc""" LearningRate.Exp(η₀; γ = 0.9) ```math @@ -122,7 +124,7 @@ end get(a::Exp) = a.η₀ * a.γ^a.t -doc""" +@doc doc""" LearningRate.Inv(η₀; γ = 0.9, p = 0.5) ```math @@ -156,6 +158,8 @@ using .LearningRate module Momentum +using Markdown + import Base: get import ..mx: AbstractMomentumScheduler, OptimizationState @@ -194,7 +198,7 @@ end get(f::Fixed) = f.μ -doc""" +@doc doc""" NadamScheduler(; μ = 0.99, δ = 0.004, γ = 0.5, α = 0.96) Nesterov-accelerated adaptive momentum scheduler. diff --git a/julia/src/optimizers/adadelta.jl b/julia/src/optimizers/adadelta.jl index 2b0cd0060..7a45dd045 100644 --- a/julia/src/optimizers/adadelta.jl +++ b/julia/src/optimizers/adadelta.jl @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -doc""" +@doc doc""" AdaDelta(; kwargs...) Scale learning rates by the ratio of accumulated gradients to accumulated diff --git a/julia/src/optimizers/adagrad.jl b/julia/src/optimizers/adagrad.jl index 4236cb8cd..6ddcb3628 100644 --- a/julia/src/optimizers/adagrad.jl +++ b/julia/src/optimizers/adagrad.jl @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -doc""" +@doc doc""" AdaGrad(; kwargs...) Scale learning rates by dividing with the square root of accumulated diff --git a/julia/src/optimizers/nadam.jl b/julia/src/optimizers/nadam.jl index fdcd1ea7e..522e9194c 100644 --- a/julia/src/optimizers/nadam.jl +++ b/julia/src/optimizers/nadam.jl @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -doc""" +@doc doc""" Nadam(; kwargs...) Nesterov Adam optimizer: Adam RMSprop with Nesterov momentum, diff --git a/julia/src/optimizers/rmsprop.jl b/julia/src/optimizers/rmsprop.jl index 835114268..184457525 100644 --- a/julia/src/optimizers/rmsprop.jl +++ b/julia/src/optimizers/rmsprop.jl @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -doc""" +@doc doc""" RMSProp(; kwargs...) Scale learning rates by dividing with the moving average of the root mean diff --git a/julia/src/optimizers/sgd.jl b/julia/src/optimizers/sgd.jl index dfd3d954b..6af809482 100644 --- a/julia/src/optimizers/sgd.jl +++ b/julia/src/optimizers/sgd.jl @@ -15,7 +15,7 @@ # specific language governing permissions and limitations # under the License. -doc""" +@doc doc""" SGD(; kwargs...) Stochastic gradient descent optimizer. @@ -69,7 +69,7 @@ SGD create_state(sgd::SGD, ::Int, W::NDArray) = isa(sgd.μ_sched, Momentum.Null) ? nothing : zeros(size(W), context(W)) -function update!(sgd::SGD, ::Int, W::NDArray, ∇::NDArray, ::Void) +function update!(sgd::SGD, ::Int, W::NDArray, ∇::NDArray, ::Nothing) η = get(sgd.η_sched) normgrad!(sgd, W, ∇) @inplace W += -η * ∇ diff --git a/julia/src/random.jl b/julia/src/random.jl index b9b32a42e..e18e906a5 100644 --- a/julia/src/random.jl +++ b/julia/src/random.jl @@ -76,8 +76,13 @@ randn(dims::Int...; μ = 0, σ = 1, context = cpu()) = randn!(empty(dims, context), μ = μ, σ = σ) """ - srand(seed::Int) + seed!(seed::Int) Set the random seed of libmxnet """ -srand(seed_state::Int) = @mxcall(:MXRandomSeed, (Cint,), seed_state) +seed!(s::Int) = @mxcall :MXRandomSeed (Cint,) s + +function srand(s::Int) + @warn "`mx.srand` is deprecated, use `mx.seed!` instead." + seed!(s) +end diff --git a/julia/src/symbolic-node.jl b/julia/src/symbolic-node.jl index 508f9d449..8b7a8be09 100644 --- a/julia/src/symbolic-node.jl +++ b/julia/src/symbolic-node.jl @@ -19,20 +19,21 @@ SymbolicNode SymbolicNode is the basic building block of the symbolic graph in MXNet.jl. +It's a callable object and supports following calls: - (self :: SymbolicNode)(args :: SymbolicNode...) - (self :: SymbolicNode)(; kwargs...) + (s::SymbolicNode)(args::SymbolicNode...) + (s::SymbolicNode)(; kwargs...) -Make a new node by composing `self` with `args`. Or the arguments +Make a new node by composing `s` with `args`. Or the arguments can be specified using keyword arguments. """ mutable struct SymbolicNode handle::MX_SymbolHandle end -const SymbolicNodeOrReal = Union{SymbolicNode, Real} +const SymbolicNodeOrReal = Union{SymbolicNode,Real} -@unfuse SymbolicNode # for broadcasting +# @unfuse SymbolicNode # for broadcasting Base.unsafe_convert(::Type{MX_handle}, obj::SymbolicNode) = Base.unsafe_convert(MX_handle, obj.handle) @@ -40,31 +41,31 @@ Base.convert(t::Type{MX_handle}, obj::SymbolicNode) = Base.unsafe_convert(t, obj Base.cconvert(t::Type{MX_handle}, obj::SymbolicNode) = Base.unsafe_convert(t, obj) """ - deepcopy(self :: SymbolicNode) + deepcopy(s::SymbolicNode) Make a deep copy of a SymbolicNode. """ -function Base.deepcopy(self :: SymbolicNode) - ref_hdr = Ref{MX_handle}(0) - @mxcall(:MXSymbolCopy, (MX_handle, Ref{MX_handle}), self, ref_hdr) - return SymbolicNode(MX_SymbolHandle(ref_hdr[])) +function Base.deepcopy(s::SymbolicNode) + ref_hdr = Ref{MX_handle}(C_NULL) + @mxcall(:MXSymbolCopy, (MX_handle, Ref{MX_handle}), s, ref_hdr) + SymbolicNode(MX_SymbolHandle(ref_hdr[])) end """ - copy(self :: SymbolicNode) + copy(s::SymbolicNode) Make a copy of a SymbolicNode. The same as making a deep copy. """ -function Base.copy(self :: SymbolicNode) - Base.deepcopy(self) -end +Base.copy(s::SymbolicNode) = Base.deepcopy(s) -function (self::SymbolicNode)(args :: SymbolicNode...) - s = deepcopy(self) + +function (s::SymbolicNode)(args::SymbolicNode...) + s = deepcopy(s) _compose!(s, args...) end -function (self::SymbolicNode)(;kwargs...) - s = deepcopy(self) + +function (s::SymbolicNode)(; kwargs...) + s = deepcopy(s) _compose!(s; kwargs...) end @@ -82,7 +83,7 @@ macro _list_symbol_info(self, func_name) end """ - list_arguments(self :: SymbolicNode) + list_arguments(s::SymbolicNode) List all the arguments of this node. The argument for a node contains both the inputs and parameters. For example, a `FullyConnected` node will @@ -91,24 +92,20 @@ list all the arguments for intermediate nodes. Returns a list of symbols indicating the names of the arguments. """ -function list_arguments(self :: SymbolicNode) - @_list_symbol_info(self, :MXSymbolListArguments) -end +list_arguments(s::SymbolicNode) = @_list_symbol_info(s, :MXSymbolListArguments) """ - list_outputs(self :: SymbolicNode) + list_outputs(s::SymbolicNode) List all the outputs of this node. Returns a list of symbols indicating the names of the outputs. """ -function list_outputs(self :: SymbolicNode) - @_list_symbol_info(self, :MXSymbolListOutputs) -end +list_outputs(s::SymbolicNode) = @_list_symbol_info(s, :MXSymbolListOutputs) """ - list_auxiliary_states(self :: SymbolicNode) + list_auxiliary_states(s::SymbolicNode) List all auxiliary states in the symbool. @@ -120,19 +117,18 @@ Most operators do not have Auxiliary states. Returns a list of symbols indicating the names of the auxiliary states. """ -function list_auxiliary_states(self :: SymbolicNode) - @_list_symbol_info(self, :MXSymbolListAuxiliaryStates) -end +list_auxiliary_states(s::SymbolicNode) = + @_list_symbol_info(s, :MXSymbolListAuxiliaryStates) """ - get_internals(self :: SymbolicNode) + get_internals(s::SymbolicNode) Get a new grouped `SymbolicNode` whose output contains all the internal outputs of this `SymbolicNode`. """ -function get_internals(self :: SymbolicNode) +function get_internals(s::SymbolicNode) ref_hdr = Ref{MX_handle}(0) - @mxcall(:MXSymbolGetInternals, (MX_handle, Ref{MX_handle}), self, ref_hdr) + @mxcall(:MXSymbolGetInternals, (MX_handle, Ref{MX_handle}), s, ref_hdr) return SymbolicNode(MX_SymbolHandle(ref_hdr[])) end @@ -166,37 +162,38 @@ function get_children(x::SymbolicNode) end """ - get_attr(self :: SymbolicNode, key :: Symbol) + get_attr(s::SymbolicNode, key::Symbol) Get attribute attached to this `SymbolicNode` belonging to key. -Returns the value belonging to key as a `Nullable`. +Returns the value belonging to key as a `String`. +If not available, returns `missing`. """ -function get_attr(self :: SymbolicNode, key :: Symbol) +function get_attr(s::SymbolicNode, key::Symbol) key_s = string(key) ref_out = Ref{Cstring}() ref_success = Ref{Cint}(-1) @mxcall(:MXSymbolGetAttr, (MX_handle, Cstring, Ref{Cstring}, Ref{Cint}), - self, key_s, ref_out, ref_success) + s, key_s, ref_out, ref_success) if ref_success[] == 1 - return Nullable{String}(unsafe_string(ref_out[])) + unsafe_string(ref_out[]) else - return Nullable{String}() + missing end end """ - list_attr(self :: SymbolicNode) + list_attr(s::SymbolicNode) Get all attributes from a symbol. Returns a dictionary of attributes. """ -function list_attr(self :: SymbolicNode) +function list_attr(s::SymbolicNode) ref_sz = Ref{MX_uint}(0) ref_strings = Ref{char_pp}(0) @mxcall(:MXSymbolListAttrShallow, (MX_handle, Ref{MX_uint}, Ref{char_pp}), - self, ref_sz, ref_strings) + s, ref_sz, ref_strings) narg = 2*ref_sz[] strings = unsafe_wrap(Array, ref_strings[], narg) out = Dict{Symbol, String}() @@ -209,17 +206,17 @@ function list_attr(self :: SymbolicNode) end """ - list_all_attr(self :: SymbolicNode) + list_all_attr(s::SymbolicNode) Get all attributes from the symbol graph. Returns a dictionary of attributes. """ -function list_all_attr(self :: SymbolicNode) +function list_all_attr(s::SymbolicNode) ref_sz = Ref{MX_uint}(0) ref_strings = Ref{char_pp}(0) @mxcall(:MXSymbolListAttr, (MX_handle, Ref{MX_uint}, Ref{char_pp}), - self, ref_sz, ref_strings) + s, ref_sz, ref_strings) narg = 2*ref_sz[] strings = unsafe_wrap(Array, ref_strings[], narg) out = Dict{Symbol, String}() @@ -232,7 +229,7 @@ function list_all_attr(self :: SymbolicNode) end """ - set_attr(self:: SymbolicNode, key :: Symbol, value :: AbstractString) + set_attr(s::SymbolicNode, key::Symbol, value::AbstractString) Set the attribute key to value for this `SymbolicNode`. @@ -242,15 +239,15 @@ Set the attribute key to value for this `SymbolicNode`. the attributes of a `SymbolicNode` that is already been used somewhere else might cause unexpected behavior and inconsistency. """ -function set_attr(self :: SymbolicNode, key :: Symbol, value :: AbstractString) +function set_attr(s::SymbolicNode, key::Symbol, value::AbstractString) key_s = string(key) value_s = String(value) - @mxcall(:MXSymbolSetAttr, (MX_handle, Cstring, Cstring), self, key_s, value_s) + @mxcall(:MXSymbolSetAttr, (MX_handle, Cstring, Cstring), s, key_s, value_s) end """ - get_name(self :: SymbolicNode) + get_name(s::SymbolicNode) Get the name of the symbol. @@ -262,15 +259,15 @@ Get the name of the symbol. julia> mx.get_name(y) :fullyconnected0 """ -function get_name(self :: mx.SymbolicNode) - name = Ref{mx.char_p}(0) +function get_name(s::mx.SymbolicNode) + name = Ref{mx.char_p}(C_NULL) success = Ref(0) - @mxcall(:MXSymbolGetName, (MX_handle, Ref{char_p}, Ref{Int}), self.handle.value, name, success) + @mxcall(:MXSymbolGetName, (MX_handle, Ref{char_p}, Ref{Int}), s.handle.value, name, success) @assert success[] != -1 str = name[] if str == C_NULL # e.g. the symbol returned via get_internals - string(self.handle.value) + string(s.handle.value) else Symbol(unsafe_string(str)) end @@ -279,16 +276,6 @@ end Base.show(io::IO, sym::SymbolicNode) = print(io, "$(typeof(sym)) $(get_name(sym))") -import Base: print - -function print(io::IO, sym::SymbolicNode) - out = Ref{mx.char_p}(C_NULL) - @mx.mxcall(:MXSymbolPrint, (mx.MX_SymbolHandle, Ref{mx.char_p}), sym.handle, out) - print(io, unsafe_string(out[])) -end - -print(sym::SymbolicNode) = print(STDOUT, sym) - """ print([io::IO], sym::SymbolicNode) @@ -298,7 +285,7 @@ Print the content of symbol, used for debug. julia> layer = @mx.chain mx.Variable(:data) => mx.FullyConnected(name=:fc1, num_hidden=128) => mx.Activation(name=:relu1, act_type=:relu) -MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Void} @0x000055b29b9c3520)) +MXNet.mx.SymbolicNode(MXNet.mx.MX_SymbolHandle(Ptr{Nothing} @0x000055b29b9c3520)) julia> print(layer) Symbol Outputs: @@ -322,25 +309,32 @@ Attrs: act_type=relu ``` """ -print +function Base.print(io::IO, sym::SymbolicNode) + out = Ref{mx.char_p}(C_NULL) + @mx.mxcall(:MXSymbolPrint, (mx.MX_SymbolHandle, Ref{mx.char_p}), sym.handle, out) + print(io, unsafe_string(out[])) +end + +Base.print(sym::SymbolicNode) = print(STDOUT, sym) """ - grad(self :: SymbolicNode, wrt :: Vector{SymbolicNode}) + grad(s::SymbolicNode, wrt::Vector{Symbol}) Get the autodiff gradient of the current `SymbolicNode`. This function can only be used if the current symbol is a loss function. # Arguments: -* `self::SymbolicNode`: current node. +* `s::SymbolicNode`: current node. * `wrt::Vector{Symbol}`: the names of the arguments to the gradient. Returns a gradient symbol of the corresponding gradient. """ -function grad(self :: SymbolicNode, wrt :: Vector{Symbol}) - hdr_ref = Ref{MX_handle}(0) - keys = String[string(key) for key in wrt] +function grad(s::SymbolicNode, wrt::Vector{Symbol}) + hdr_ref = Ref{MX_handle}(C_NULL) + keys = string.(key) - @mxcall(:MXSymbolGrad, (MX_handle, MX_uint, char_pp, Ptr{MX_handle}), self, length(keys), keys, hdr_ref) + @mxcall(:MXSymbolGrad, (MX_handle, MX_uint, char_pp, Ptr{MX_handle}), + self, length(keys), keys, hdr_ref) return SymbolicNode(MX_SymbolHandle(hdr_ref[])) end @@ -403,7 +397,7 @@ function _build_shapes(shape_size::MX_uint, shape_ndim::Ptr{MX_uint}, shape_data shape_data = unsafe_wrap(Array, shape_data, shape_size) shapes = map(1:shape_size) do i my_shape = unsafe_wrap(Array, shape_data[i], shape_ndim[i]) - tuple(flipdim(Int[my_shape...],1)...) + tuple(reverse(Int[my_shape...], dims = 1)...) end convert(Vector{Tuple}, shapes) end @@ -458,18 +452,18 @@ function infer_shape(self :: SymbolicNode; kwargs...) sdata = MX_uint[] indptr = MX_uint[0] for (k,v) in kwargs - append!(sdata, flipdim([v...],1)) + append!(sdata, reverse([v...], dims = 1)) push!(indptr, length(sdata)) end keys = AbstractString[string(x[1]) for x in kwargs] _infer_shape(self, keys, indptr, sdata) end -function infer_shape(self :: SymbolicNode, args :: Union{Tuple, Void}...) +function infer_shape(self :: SymbolicNode, args::Union{Tuple, Cvoid}...) sdata = MX_uint[] indptr = MX_uint[0] for arg in args - if isa(arg, Void); continue; end - append!(sdata, flipdim([arg...],1)) + if isa(arg, Cvoid); continue; end + append!(sdata, reverse([arg...], dims = 1)) push!(indptr, length(sdata)) end keys = Ptr{char_p}(0) @@ -528,12 +522,12 @@ function infer_type(self :: SymbolicNode; kwargs...) _infer_type(self, keys, types) end -function infer_type(self :: SymbolicNode, args :: Union{Tuple, Void}...) +function infer_type(self :: SymbolicNode, args :: Union{Tuple,Cvoid}...) types = Cint[] keys = Ptr{char_p}(0) for arg in args - if isa(arg, Void); continue; end + if isa(arg, Cvoid); continue; end push!(types, toTypeFlag(arg)) end _infer_type(self, keys, types) @@ -548,7 +542,7 @@ indicating the index, as in the list of [`list_outputs`](@ref). """ function Base.getindex(self :: SymbolicNode, idx :: Union{Base.Symbol, AbstractString}) idx = Symbol(idx) - i_idx = find(idx .== list_outputs(self)) + i_idx = findall(idx .== list_outputs(self)) @assert(length(i_idx) > 0, "Cannot find output with name '$idx'") @assert(length(i_idx) < 2, "Found duplicated output with name '$idx'") Base.getindex(self, i_idx[1]) @@ -582,8 +576,8 @@ end +(s::Real, x::SymbolicNode, ys::SymbolicNodeOrReal...) = +(x + s, ys...) -broadcast_(::typeof(+), x::SymbolicNode, ys::SymbolicNodeOrReal...) = +(x, ys...) -broadcast_(::typeof(+), s::Real, x::SymbolicNode, ys::SymbolicNodeOrReal...) = +(x + s, ys...) +broadcasted(::typeof(+), x::SymbolicNode, ys::SymbolicNodeOrReal...) = +(x, ys...) +broadcasted(::typeof(+), s::Real, x::SymbolicNode, ys::SymbolicNodeOrReal...) = +(x + s, ys...) import Base: - @@ -600,8 +594,8 @@ s::Real - x::SymbolicNode = _rminus_scalar(x, scalar=MX_float(s)) -(x::SymbolicNode) = 0 - x -broadcast_(::typeof(-), x::SymbolicNode, y::SymbolicNodeOrReal) = x - y -broadcast_(::typeof(-), s::Real, x::SymbolicNode) = s - x +broadcasted(::typeof(-), x::SymbolicNode, y::SymbolicNodeOrReal) = x - y +broadcasted(::typeof(-), s::Real, x::SymbolicNode) = s - x import Base: * @@ -613,7 +607,7 @@ Elementwise multiplication of `SymbolicNode`. x::SymbolicNode * s::Real = _mul_scalar(x, scalar=MX_float(s)) s::Real * x::SymbolicNode = _mul_scalar(x, scalar=MX_float(s)) -function broadcast_(::typeof(*), x::SymbolicNode, ys::SymbolicNodeOrReal...) +function broadcasted(::typeof(*), x::SymbolicNode, ys::SymbolicNodeOrReal...) ret = x for y in ys if y isa SymbolicNode @@ -625,8 +619,8 @@ function broadcast_(::typeof(*), x::SymbolicNode, ys::SymbolicNodeOrReal...) ret end -broadcast_(::typeof(*), s::Real, x::SymbolicNode, ys::SymbolicNodeOrReal...) = - broadcast_(*, x * s, ys...) +broadcasted(::typeof(*), s::Real, x::SymbolicNode, ys::SymbolicNodeOrReal...) = + broadcasted(*, x * s, ys...) import Base: / @@ -642,9 +636,9 @@ of the same shape. """ x::SymbolicNode / s::Real = _DivScalar(x, scalar=MX_float(s)) -broadcast_(::typeof(/), x::SymbolicNode, y::SymbolicNode) = _div(x, y) -broadcast_(::typeof(/), x::SymbolicNode, s::Real) = _div_scalar(x, scalar=MX_float(s)) -broadcast_(::typeof(/), s::Real, x::SymbolicNode) = _rdiv_scalar(x, scalar=MX_float(s)) +broadcasted(::typeof(/), x::SymbolicNode, y::SymbolicNode) = _div(x, y) +broadcasted(::typeof(/), x::SymbolicNode, s::Real) = _div_scalar(x, scalar=MX_float(s)) +broadcasted(::typeof(/), s::Real, x::SymbolicNode) = _rdiv_scalar(x, scalar=MX_float(s)) import Base: ^ @@ -657,22 +651,24 @@ Operating with `Real` is available. """ ^ -broadcast_(::typeof(^), x::SymbolicNode, y::SymbolicNode) = _power(x, y) -broadcast_(::typeof(^), x::SymbolicNode, s::Real) = _power_scalar(x, scalar=MX_float(s)) -broadcast_(::typeof(^), s::Real, x::SymbolicNode) = _rpower_scalar(x, scalar=MX_float(s)) +broadcasted(::typeof(^), x::SymbolicNode, y::SymbolicNode) = _power(x, y) +broadcasted(::typeof(^), x::SymbolicNode, s::Real) = _power_scalar(x, scalar = s) +broadcasted(::typeof(^), s::Real, x::SymbolicNode) = _rpower_scalar(x, scalar = s) +broadcasted(::typeof(Base.literal_pow), ::typeof(^), x::SymbolicNode, ::Val{s}) where {s} = + _power_scalar(x, scalar = s) -broadcast_(::typeof(^), ::Irrational{:e}, x::SymbolicNode) = exp(x) -broadcast_(::typeof(^), x::SymbolicNode, s::Irrational) = +broadcasted(::typeof(^), ::Irrational{:ℯ}, x::SymbolicNode) = exp(x) +broadcasted(::typeof(^), x::SymbolicNode, s::Irrational) = _power_scalar(x, scalar=MX_float(s)) -broadcast_(::typeof(^), s::Irrational, x::SymbolicNode) = +broadcasted(::typeof(^), s::Irrational, x::SymbolicNode) = _rpower_scalar(x, scalar=MX_float(s)) -function _compose!(node :: SymbolicNode; kwargs...) - name = char_p(0) - arg_keys = AbstractString[] +function _compose!(node::SymbolicNode; kwargs...) + name = char_p(C_NULL) + arg_keys = AbstractString[] # FIXME: can it be String[] ? arg_vals = MX_handle[] - for (k,v) in kwargs + for (k, v) in kwargs if k == :name name = string(v) else @@ -685,20 +681,21 @@ function _compose!(node :: SymbolicNode; kwargs...) @mxcall(:MXSymbolCompose, (MX_handle, char_p, MX_uint, Ptr{char_p}, Ptr{MX_handle}), node, name, length(arg_keys), arg_keys, arg_vals) - return node + node end -function _compose!(node :: SymbolicNode, args::SymbolicNode...) +_compose!(node::SymbolicNode, args::SymbolicNode...) = _compose!(node, char_p(0), args...) -end -function _compose!(node :: SymbolicNode, name :: Union{Base.Symbol, char_p}, args::SymbolicNode...) - if isa(name, Base.Symbol); name = string(name); end - arg_keys = Ptr{char_p}(0) +function _compose!(node::SymbolicNode, name::Union{Symbol, char_p}, args::SymbolicNode...) + if name isa Symbol + name = string(name) + end + arg_keys = Ptr{char_p}(C_NULL) arg_vals = MX_handle[args...] @mxcall(:MXSymbolCompose, (MX_handle, char_p, MX_uint, Ptr{char_p}, Ptr{MX_handle}), node, name, length(arg_vals), arg_keys, arg_vals) - return node + node end """ @@ -844,30 +841,26 @@ end node end -function _define_atomic_symbol_creator(name :: String) +function _define_atomic_symbol_creator(name::String) handle = _get_libmx_op_handle(name) f_desc, key_narg = _get_libmx_op_description(name, handle) f_desc *= "* `name::Symbol`: The name of the `SymbolicNode`. (e.g. `:my_symbol`), optional.\n" - f_desc *= "* `attrs::Dict{Symbol, AbstractString}`: The attributes associated with this `SymbolicNode`.\n\n" + f_desc *= "* `attrs::Dict{Symbol,String}`: The attributes associated with this `SymbolicNode`.\n\n" func_name = Symbol(name) + import_expr = _import_expr(func_name) + func_def = quote - function $func_name(::Type{SymbolicNode}, args::SymbolicNode...; kwargs...) - idx = findfirst(x -> x[1] == :name, kwargs) - if idx > 0 - name = kwargs[idx][2] - else - name = "" - end + function $func_name(::Type{SymbolicNode}, args::SymbolicNode...; name = "", kwargs...) - # XXX: hacky way of solving the problem that the arguments of `dot` should be swapped + # NOTE: hacky way of solving the problem that the arguments of `dot` should be swapped # See https://github.com/dmlc/MXNet.jl/issues/55 if $name == "dot" args = reverse(args) end - # XXX: hacky way of solving the semantic difference of the axes parameter in Julia + # NOTE: hacky way of solving the semantic difference of the axes parameter in Julia # and in libmxnet. # See https://github.com/dmlc/MXNet.jl/pull/123 if $name == "transpose" @@ -876,8 +869,8 @@ function _define_atomic_symbol_creator(name :: String) param_keys = String[] param_vals = String[] - symbol_kws = Dict{Symbol, SymbolicNode}() - attrs = Dict{Symbol, String}() + symbol_kws = Dict{Symbol,SymbolicNode}() + attrs = Dict{Symbol,String}() $(if key_narg != "" quote @@ -936,23 +929,24 @@ function _define_atomic_symbol_creator(name :: String) end # quote func_def2 = quote - @doc $f_desc -> + @doc $f_desc function $func_name(args::SymbolicNode...; kwargs...) $func_name(SymbolicNode, args...; kwargs...) end # function end # quote return quote + $import_expr $func_def $func_def2 end end macro _import_atomic_symbol_creators() - # XXX: those are operators defined for NDArray, we exclude them here + # NOTE: those are operators defined for NDArray, we exclude them here # because the calling convention for the type signature is not strong # enough to disambiguate the method for NDArray and SymbolicNode - const ignored_ops = ["_set_value", "reshape"] # in lowercase + ignored_ops = ("_set_value", "reshape") # in lowercase op_names = _get_libmx_op_names() func_exprs = map(op_names) do name @@ -966,7 +960,7 @@ macro _import_atomic_symbol_creators() end) end -@_import_atomic_symbol_creators() +@_import_atomic_symbol_creators ################################################################################ # Utility macros to chain up symbols @@ -976,7 +970,7 @@ macro chain(layers) last_layer = nothing function _chain_layer(layer, last_layer) - if isa(last_layer, Void) + if last_layer ≡ nothing return esc(layer) else if @capture(layer, f_(x__)) diff --git a/julia/src/util.jl b/julia/src/util.jl index c07c9ec91..a836d3e39 100644 --- a/julia/src/util.jl +++ b/julia/src/util.jl @@ -19,7 +19,7 @@ # Dataset related utilities ################################################################################ function get_data_dir() - data_dir = joinpath(Pkg.dir("MXNet"), "data") + data_dir = joinpath(@__DIR__, "..", "data") mkpath(data_dir) data_dir end @@ -32,7 +32,7 @@ function get_mnist_ubyte() :train_label => "train-labels-idx1-ubyte", :test_data => "t10k-images-idx3-ubyte", :test_label => "t10k-labels-idx1-ubyte") - filenames = Dict(map((x) -> x[1] => joinpath(mnist_dir, x[2]), filenames)) + filenames = Dict((x[1] => joinpath(mnist_dir, x[2]) for x ∈ pairs(filenames))) if !all(isfile, values(filenames)) cd(mnist_dir) do mnist_dir = download("http://data.mxnet.io/mxnet/data/mnist.zip", "mnist.zip") @@ -40,7 +40,7 @@ function get_mnist_ubyte() run(`unzip -u $mnist_dir`) catch try - run(pipe(`7z x $mnist_dir`,stdout=DevNull)) + run(pipe(`7z x $mnist_dir`,stdout = devnull)) catch error("Extraction Failed:No extraction program found in path") end @@ -63,7 +63,7 @@ function get_cifar10() run(`unzip -u cifar10.zip`) catch try - run(pipeline(`7z x cifar10.zip`, stdout=DevNull)) + run(pipeline(`7z x cifar10.zip`, stdout = devnull)) catch error("Extraction Failed:No extraction program found in path") end @@ -149,9 +149,8 @@ function _get_libmx_op_description(name::String, handle::MX_OpHandle) return desc, key_narg end -function _format_typestring(typestr :: String) - replace(typestr, r"\bSymbol\b", "SymbolicNode") -end +_format_typestring(s::String) = replace(s, r"\bSymbol\b" => "SymbolicNode") + function _format_docstring(narg::Int, arg_names::Ref{char_pp}, arg_types::Ref{char_pp}, arg_descs::Ref{char_pp}, remove_dup::Bool=true) param_keys = Set{String}() @@ -191,7 +190,7 @@ function _getdocdefine(name::String) op = _get_libmx_op_handle(name) str = _get_libmx_op_description(name, op)[1] lines = split(str, '\n') - for m ∈ match.(r"^Defined in .*$", lines) + for m ∈ match.(Ref(r"^Defined in .*$"), lines) m != nothing && return m.match end "" @@ -227,7 +226,7 @@ function _sig_checker() return end - warn(_sig) + @warn(_sig) end end @@ -253,3 +252,20 @@ function _firstarg(sig::Expr) end _firstarg(s::Symbol) = s + +const _import_map = Dict{Symbol,Union{Missing,Module}}( + :diag => LinearAlgebra, + :dot => LinearAlgebra, + :norm => LinearAlgebra, + + :shuffle => Random, + + :mean => Statistics, + + :gamma => missing, +) + +function _import_expr(func_name::Symbol) + mod = get(_import_map, func_name, Base) + isdefined(mod, func_name) ? :(import $(Symbol(mod)): $func_name) : :() +end diff --git a/julia/src/visualize.jl b/julia/src/visualize.jl index b41db7e9e..ddbb3c11f 100644 --- a/julia/src/visualize.jl +++ b/julia/src/visualize.jl @@ -30,14 +30,14 @@ import JSON Returns the graph description in GraphViz `dot` language. """ function to_graphviz(network :: SymbolicNode; title="Network Visualization", input_shapes=nothing) - if !isa(input_shapes, Void) + if !isa(input_shapes, Cvoid) internals = get_internals(network) if isa(input_shapes, Dict) _, out_shapes, _ = infer_shape(internals; input_shapes...) else _, out_shapes, _ = infer_shape(internals, input_shapes...) end - @assert(!isa(out_shapes, Void), "Failed to do shape inference, input shapes are incomplete") + @assert(!isa(out_shapes, Cvoid), "Failed to do shape inference, input shapes are incomplete") shape_dict = Dict(zip(list_outputs(internals), out_shapes)) draw_shape = true else @@ -170,7 +170,7 @@ end function _format_graphviz_attr(io::IOBuffer, attrs) label = get(attrs, :label, nothing) - if isa(label, Void) + if isa(label, Cvoid) print(io, " [") else print(io, " [label=$(_simple_escape(label)),") @@ -195,7 +195,7 @@ function _format_graphviz_attr(io::IOBuffer, attrs) println(io, "];") end function _simple_escape(str) - str = replace(string(str), r"\n", "\\n") + str = replace(string(str), r"\n" => "\\n") return "\"$str\"" end function _format_graphviz_node(io::IOBuffer, name::AbstractString, attrs) diff --git a/julia/test/runtests.jl b/julia/test/runtests.jl index 8b46bfda6..4e5f27395 100644 --- a/julia/test/runtests.jl +++ b/julia/test/runtests.jl @@ -15,29 +15,33 @@ # specific language governing permissions and limitations # under the License. +using Test using MXNet -using Base.Test # run test in the whole directory, latest modified files # are run first, this makes waiting time shorter when writing # or modifying unit-tests function test_dir(dir) - jl_files = sort(filter(x -> ismatch(r".*\.jl$", x), readdir(dir)), by = fn -> stat(joinpath(dir,fn)).mtime) - map(reverse(jl_files)) do file + jl_files = sort( + filter(x -> occursin(r".*\.jl$", x), readdir(dir)), + by = fn -> stat(joinpath(dir, fn)).mtime) + foreach(reverse(jl_files)) do file include("$dir/$file") end end -info("libmxnet version => $(mx.LIB_VERSION)") +@info "libmxnet version => $(mx.LIB_VERSION[])" -include(joinpath(dirname(@__FILE__), "common.jl")) +const BASEDIR = joinpath(@__DIR__, "..") + +include(joinpath(@__DIR__, "common.jl")) @testset "MXNet Test" begin - test_dir(joinpath(dirname(@__FILE__), "unittest")) + test_dir(joinpath(@__DIR__, "unittest")) # run the basic MNIST mlp example if haskey(ENV, "CONTINUOUS_INTEGRATION") @testset "MNIST Test" begin - include(joinpath(Pkg.dir("MXNet"), "examples", "mnist", "mlp-test.jl")) + include(joinpath(BASEDIR, "examples", "mnist", "mlp-test.jl")) end end end diff --git a/julia/test/unittest/autograd.jl b/julia/test/unittest/autograd.jl index 96f275b62..8209fe7e0 100644 --- a/julia/test/unittest/autograd.jl +++ b/julia/test/unittest/autograd.jl @@ -17,9 +17,8 @@ module TestAutoGrad -using Base.Test - using MXNet +using Test function checkgradient(f, x, y, ∇) @@ -33,12 +32,12 @@ end # function checkgradient function test_getgrad() - info("AutoGrad::getgrad") + @info("AutoGrad::getgrad") - info("AutoGrad::getgrad::unattached") + @info("AutoGrad::getgrad::unattached") @test nothing == mx.getgrad(mx.zeros(10)) - info("AutoGrad::getgrad::attached") + @info("AutoGrad::getgrad::attached") x = mx.NDArray([1 2; 3 4]) grad = mx.attach_grad!(x) @test eltype(grad) ≡ Int @@ -50,7 +49,7 @@ end function test_mark_variables!() - info("AutoGrad::mark_variables!") + @info("AutoGrad::mark_variables!") x = mx.zeros(4) ẋ = mx.zeros(4) y = mx.zeros(4) @@ -62,13 +61,13 @@ function test_mark_variables!() @test copy(mx.getgrad(x)) == [42, 42, 42, 42] @test copy(mx.getgrad(y)) == [24, 24, 24, 24] - info("AutoGrad::mark_variables!::invalid grad_reqs") + @info("AutoGrad::mark_variables!::invalid grad_reqs") x = mx.zeros(4) y = mx.zeros(4) @test_throws ArgumentError mx.mark_variables!(x, y, :magic) @test_throws ArgumentError mx.mark_variables!([x], [y], [:magic]) - info("AutoGrad::mark_variables!::args length mismatch") + @info("AutoGrad::mark_variables!::args length mismatch") x = mx.zeros(4) y = mx.zeros(4) z = mx.zeros(4) @@ -79,7 +78,7 @@ end function test_record() let x = mx.NDArray([1 2; 3 4]) - info("AutoGrad::record::backward!") + @info("AutoGrad::record::backward!") y = [1 4; 9 16] ∇ = [2 4; 6 8] # gradient is 2x @@ -89,7 +88,7 @@ function test_record() end let x = mx.NDArray([1 2; 3 4]) - info("AutoGrad::record::symbol") + @info("AutoGrad::record::symbol") mx.attach_grad!(x) y = mx.record() do @@ -102,7 +101,7 @@ function test_record() end let x = mx.NDArray([1 2; 3 4]) - info("AutoGrad::record::backward!(retain_graph=true)") + @info("AutoGrad::record::backward!(retain_graph=true)") mx.attach_grad!(x) y = mx.record() do @@ -125,7 +124,7 @@ end # function test_record function test_is_recording() - info("AutoGrad::is_recording") + @info("AutoGrad::is_recording") mx.record() do @test mx.is_recording() end @@ -133,7 +132,7 @@ end # function test_is_recording function test_is_training() - info("AutoGrad::is_training") + @info("AutoGrad::is_training") mx.record() do @test mx.is_training() end @@ -145,7 +144,7 @@ end # function test_is_training function test_pause() - info("AutoGrad::pause") + @info("AutoGrad::pause") let x = mx.NDArray([1 2; 3 4]) ∇ = mx.attach_grad!(x) y = mx.record() do @@ -166,7 +165,7 @@ end # function test_pause function test_train_mode() - info("AutoGrad::train_mode") + @info("AutoGrad::train_mode") let x = mx.NDArray(Float32[1 2; 3 4]) y = mx.train_mode() do mx.Dropout(x, p = 1) @@ -178,7 +177,7 @@ end # function test_train_mode function test_predict_mode() - info("AutoGrad::predict_mode") + @info("AutoGrad::predict_mode") let x = mx.NDArray(Float32[1 2; 3 4]) y = mx.predict_mode() do mx.Dropout(x, p = 1) @@ -190,7 +189,7 @@ end # function test_train_mode function test_backward!() - info("AutoGrad::backward!::with head_grad") + @info("AutoGrad::backward!::with head_grad") let x = mx.NDArray(Float32[1 2; 3 4]), A = Float32[.2 .4; 0 .1] ∇ = mx.attach_grad!(x) y = mx.record() do @@ -200,7 +199,7 @@ function test_backward!() @test copy(∇) ≈ [2 4; 6 8] .* A end - info("AutoGrad::backward!::with head_grads") + @info("AutoGrad::backward!::with head_grads") let x = mx.NDArray(Float32[1 2; 3 4]) ∇ = mx.attach_grad!(x) mx.record() do @@ -216,7 +215,7 @@ function test_backward!() @test copy(∇) ≈ ans end - info("AutoGrad::backward!::ArgumentError") + @info("AutoGrad::backward!::ArgumentError") let x = mx.NDArray([42]) @test_throws ArgumentError mx.backward!([x], [24]) end @@ -224,7 +223,7 @@ end # function test_backward! function test_symbol() - info("AutoGrad::symbol") + @info("AutoGrad::symbol") let x = mx.zeros(4) mx.attach_grad!(x) @@ -234,9 +233,9 @@ end function test_add() - info("AutoGrad::add") + @info("AutoGrad::add") - info("AutoGrad::add::x") + @info("AutoGrad::add::x") let x = mx.NDArray([1 2; 3 4]) y = [1 2; 3 4] ∇ = [1 1; 1 1] # gradient is 1 @@ -245,7 +244,7 @@ function test_add() end end - info("AutoGrad::add::+x") + @info("AutoGrad::add::+x") let x = mx.NDArray([1 2; 3 4]) y = [1 2; 3 4] ∇ = [1 1; 1 1] # gradient is 1 @@ -254,7 +253,7 @@ function test_add() end end - info("AutoGrad::add::x .+ 42") + @info("AutoGrad::add::x .+ 42") let x = mx.NDArray([1 2; 3 4]) y = [43 44; 45 46] ∇ = [1 1; 1 1] # gradient is 1 @@ -263,7 +262,7 @@ function test_add() end end - info("AutoGrad::add::42 .+ x") + @info("AutoGrad::add::42 .+ x") let x = mx.NDArray([1 2; 3 4]) y = [43 44; 45 46] ∇ = [1 1; 1 1] @@ -272,14 +271,14 @@ function test_add() end end - # TODO: info("AutoGrad::add::x .+ y") + # TODO: @info("AutoGrad::add::x .+ y") end # function test_add function test_sub() - info("AutoGrad::sub") + @info("AutoGrad::sub") - info("AutoGrad::sub::-x") + @info("AutoGrad::sub::-x") let x = mx.NDArray([1 2; 3 4]) y = [-1 -2; -3 -4] ∇ = [-1 -1; -1 -1] # gradient is -1 @@ -288,7 +287,7 @@ function test_sub() end end - info("AutoGrad::sub::x .- 42") + @info("AutoGrad::sub::x .- 42") let x = mx.NDArray([1 2; 3 4]) y = [-41 -40; -39 -38] ∇ = [1 1; 1 1] @@ -297,7 +296,7 @@ function test_sub() end end - info("AutoGrad::sub::42 .- x") + @info("AutoGrad::sub::42 .- x") let x = mx.NDArray([1 2; 3 4]) y = [41 40; 39 38] ∇ = -[1 1; 1 1] @@ -306,14 +305,14 @@ function test_sub() end end - # TODO: info("AutoGrad::add::x .- y") + # TODO: @info("AutoGrad::sub::x .- y") end # function test_sub function test_mul() - info("AutoGrad::mul") + @info("AutoGrad::mul") - info("AutoGrad::mul::2x .* x") + @info("AutoGrad::mul::2x .* x") let x = mx.NDArray([1 2; 3 4]) y = [2 8; 18 32] ∇ = [4 8; 12 16] # 4x @@ -322,7 +321,7 @@ function test_mul() end end - info("AutoGrad::mul::x * 2 .* x") + @info("AutoGrad::mul::x * 2 .* x") let x = mx.NDArray([1 2; 3 4]) y = [2 8; 18 32] ∇ = [4 8; 12 16] # 4x @@ -334,9 +333,9 @@ end function test_div() - info("AutoGrad::div") + @info("AutoGrad::div") - info("AutoGrad::div::x ./ 2") + @info("AutoGrad::div::x ./ 2") let x = mx.NDArray(Float32[1 2; 3 4]) y = Float32[.5 1; 1.5 2] ∇ = [.5 .5; .5 .5] @@ -345,7 +344,7 @@ function test_div() end end - info("AutoGrad::rdiv::2 ./ x") + @info("AutoGrad::rdiv::2 ./ x") let A = Float32[1 2; 3 4], x = mx.NDArray(A) y = 2 ./ A ∇ = @. -2 / A^2 # -2 / x² @@ -357,9 +356,9 @@ end # function test_div function test_power() - info("AutoGrad::power") + @info("AutoGrad::power") - info("AutoGrad::power::x.^3") + @info("AutoGrad::power::x.^3") let A = Float32[1 2; 3 4] x = mx.NDArray(A) y = A.^3 @@ -369,7 +368,7 @@ function test_power() end end - info("AutoGrad::power::x.^.5") + @info("AutoGrad::power::x.^.5") let A = Float32[1 2; 3 4] x = mx.NDArray(A) y = A.^.5 diff --git a/julia/test/unittest/bind.jl b/julia/test/unittest/bind.jl index 538b556bf..abaca884b 100644 --- a/julia/test/unittest/bind.jl +++ b/julia/test/unittest/bind.jl @@ -17,7 +17,7 @@ module TestBind using MXNet -using Base.Test +using Test using ..Main: rand_dims @@ -26,7 +26,7 @@ using ..Main: rand_dims ################################################################################ function test_arithmetic(::Type{T}, uf, gf) where T <: mx.DType shape = rand_dims() - info("Bind::arithmetic::$T::$uf::dims = $shape") + @info "Bind::arithmetic::$T::$uf::dims = $shape" lhs = mx.Variable(:lhs) rhs = mx.Variable(:rhs) @@ -77,7 +77,7 @@ function test_arithmetic() test_arithmetic(T, (x,y) -> x .- y, (g,x,y) -> (g,-g)) test_arithmetic(T, (x,y) -> x .* y, (g,x,y) -> (y.*g, x.*g)) if T <: Integer || T == Float16 - warn("Not running division test for $T") + @warn "Not running division test for $T" else test_arithmetic(T, (x,y) -> x ./ y, (g,x,y) -> (g ./ y, -x .* g ./ (y.^2))) end diff --git a/julia/test/unittest/io.jl b/julia/test/unittest/io.jl index 81f2ff79a..cf8d8368d 100644 --- a/julia/test/unittest/io.jl +++ b/julia/test/unittest/io.jl @@ -18,12 +18,12 @@ module TestIO using MXNet -using Base.Test +using Test using ..Main: rand_dims function test_mnist() - info("IO::MNIST") + @info "IO::MNIST" filenames = mx.get_mnist_ubyte() batch_size = 10 @@ -76,7 +76,7 @@ function test_arrays_impl(data::Vector, label::Vector, provider::mx.ArrayDataPro @test batch_size == d2[end] end - info("IO::Array::#data=$(length(data)),#label=$(length(label)),batch_size=$batch_size") + @info "IO::Array::#data=$(length(data)),#label=$(length(label)),batch_size=$batch_size" for (idx, batch) in zip(idx_all, provider) data_batch = [x[[Colon() for i=1:ndims(x)-1]..., idx:min(idx+batch_size-1,sample_count)] for x in data] data_get = mx.get_data(provider, batch) @@ -111,7 +111,7 @@ function test_arrays() end function test_arrays_shuffle() - info("IO::Array::shuffle") + @info "IO::Array::shuffle" sample_count = 15 batch_size = 4 diff --git a/julia/test/unittest/kvstore.jl b/julia/test/unittest/kvstore.jl index 96e1643d8..503a1fdbd 100644 --- a/julia/test/unittest/kvstore.jl +++ b/julia/test/unittest/kvstore.jl @@ -17,7 +17,7 @@ module TestKVStore using MXNet -using Base.Test +using Test using ..Main: rand_dims @@ -34,7 +34,7 @@ function init_kv() end function test_kv_basic() - info("KVStore::basic") + @info("KVStore::basic") kv = init_kv() @test mx.get_type(kv) == :local @@ -43,7 +43,7 @@ function test_kv_basic() end function test_single_kv_pair() - info("KVStore::single") + @info("KVStore::single") kv = init_kv() mx.push!(kv, 3, mx.ones(SHAPE)) @@ -53,7 +53,7 @@ function test_single_kv_pair() end function test_aggregator() - info("KVStore::aggregator") + @info("KVStore::aggregator") kv = init_kv() @@ -64,7 +64,7 @@ function test_aggregator() mx.push!(kv, 3, vals) mx.pull!(kv, 3, vals) for v in vals - @test maximum(abs.(copy(v)) - num_devs) == 0 + @test maximum(abs.(copy(v)) .- num_devs) == 0 end # list @@ -74,7 +74,7 @@ function test_aggregator() for vv in vals for v in vv - @test maximum(abs.(copy(v)) - 2 * num_devs) == 0 + @test maximum(abs.(copy(v)) .- 2 * num_devs) == 0 end end end @@ -95,7 +95,7 @@ function check_setupdater!(f) end # function check_setupdater! function test_setupdater!() - info("KVStore::setupdater!") + @info("KVStore::setupdater!") f(key, Δ, x) = @mx.inplace x += 2Δ g(key, Δ, x) = (x[:] += 2Δ) diff --git a/julia/test/unittest/metric.jl b/julia/test/unittest/metric.jl index 32c4538b6..05e4dbda4 100644 --- a/julia/test/unittest/metric.jl +++ b/julia/test/unittest/metric.jl @@ -18,7 +18,7 @@ module TestMetric using MXNet -using Base.Test +using Test ################################################################################ # Supporting functions @@ -58,7 +58,7 @@ end ################################################################################ function test_ace() - info("EvalMetric::ACE") + @info "EvalMetric::ACE" n_categories = 4 n_observations = 100 labels = convert(Vector{Float32}, rand(0:(n_categories - 1), n_observations)) # MXNet uses Float32 @@ -72,7 +72,7 @@ end function test_nmse() - info("EvalMetric::NMSE") + @info "EvalMetric::NMSE" @testset "EvalMetric::NMSE::update!" begin metric = mx.NMSE() diff --git a/julia/test/unittest/model.jl b/julia/test/unittest/model.jl index dbe97093d..387a0cd55 100644 --- a/julia/test/unittest/model.jl +++ b/julia/test/unittest/model.jl @@ -17,19 +17,19 @@ module TestModel -using Base.Test +using Test using MXNet function test_feedforward() - info("Model::FeedForward::constructor") + @info("Model::FeedForward::constructor") let x = @mx.var x m = mx.FeedForward(x) @test m.arch === x @test length(m.ctx) == 1 end - info("Model::FeedForward::constructor::keyword context") + @info("Model::FeedForward::constructor::keyword context") let x = @mx.var x m = mx.FeedForward(x, context = mx.cpu()) @test m.arch === x diff --git a/julia/test/unittest/name.jl b/julia/test/unittest/name.jl index eea73efb4..1099ec4a7 100644 --- a/julia/test/unittest/name.jl +++ b/julia/test/unittest/name.jl @@ -17,10 +17,10 @@ module TestNameManager using MXNet -using Base.Test +using Test function test_default() - info("NameManager::default") + @info("NameManager::default") name = :_____aaaaa_____ @test get!(mx.DEFAULT_NAME_MANAGER, name, "") == name @@ -32,7 +32,7 @@ function test_default() end function test_prefix() - info("NameManager::prefix") + @info("NameManager::prefix") name = :_____bbbbb_____ prefix = :_____foobar_____ diff --git a/julia/test/unittest/ndarray.jl b/julia/test/unittest/ndarray.jl index 3bacbb009..9ca4ba206 100644 --- a/julia/test/unittest/ndarray.jl +++ b/julia/test/unittest/ndarray.jl @@ -18,7 +18,9 @@ module TestNDArray using MXNet -using Base.Test +using Statistics +using LinearAlgebra +using Test using ..Main: rand_dims @@ -33,7 +35,7 @@ function rand_tensors(::Type{T}, dims::NTuple{N,Int}) where {N,T} end function test_constructor() - info("NDArray::NDArray(x::AbstractArray)") + @info("NDArray::NDArray(x::AbstractArray)") function check_absarray(x) y = mx.NDArray(x) @test ndims(x) == ndims(y) @@ -44,7 +46,7 @@ function test_constructor() check_absarray(1:10) check_absarray(1.0:10) - info("NDArray::NDArray(Type, AbstractArray)") + @info("NDArray::NDArray(Type, AbstractArray)") let x = mx.NDArray(Float32, [1, 2, 3]) @test eltype(x) == Float32 @@ -59,7 +61,7 @@ end # function test_constructor function test_ones_zeros_like() - info("NDArray::Base.zeros") + @info("NDArray::Base.zeros") let x = mx.rand(1, 3, 2, 4, low = 1, high = 10) y = zeros(x) @test sum(copy(y)) == 0 @@ -68,7 +70,7 @@ function test_ones_zeros_like() @test sum(copy(y)) == 0 end - info("NDArray::Base.ones") + @info("NDArray::Base.ones") let x = mx.rand(1, 3, 2, 4, low = 1, high = 10) y = ones(x) @test sum(copy(y)) == 1 * 3 * 2 * 4 @@ -83,7 +85,7 @@ function test_copy() dims = rand_dims() tensor = rand(mx.MX_float, dims) - info("NDArray::copy::dims = $dims") + @info("NDArray::copy::dims = $dims") # copy to NDArray and back array = copy(tensor, mx.cpu()) @@ -95,7 +97,7 @@ function test_copy() tensor2 = copy(array2) @test tensor ≈ tensor2 - info("NDArray::copy::AbstractArray") + @info("NDArray::copy::AbstractArray") let x = copy(1:4, mx.cpu()) @test eltype(x) == Int @test copy(x) == [1, 2, 3, 4] @@ -106,7 +108,7 @@ function test_copy() @test copy(x) ≈ [1., 2, 3, 4] end - info("NDArray::copy!::AbstractArray") + @info("NDArray::copy!::AbstractArray") let x = mx.zeros(4) copy!(x, 1:4) @@ -117,7 +119,7 @@ function test_copy() end function test_deepcopy() - info("NDArray::deepcopy") + @info("NDArray::deepcopy") x = mx.zeros(2, 5) y = deepcopy(x) @@ -129,7 +131,7 @@ function test_assign() dims = rand_dims() tensor = rand(mx.MX_float, dims) - info("NDArray::assign::dims = $dims") + @info("NDArray::assign::dims = $dims") # Julia Array -> NDArray assignment array = mx.empty(size(tensor)) @@ -145,21 +147,21 @@ function test_assign() # scalar -> NDArray assignment scalar = rand() array2[:] = scalar - @test zeros(size(tensor)) + scalar ≈ copy(array2) + @test zeros(size(tensor)) .+ scalar ≈ copy(array2) scalar = rand(Float16) array2[:] = scalar - @test zeros(size(tensor)) + scalar ≈ copy(array2) + @test zeros(size(tensor)) .+ scalar ≈ copy(array2) scalar = rand(Float64) array2[:] = scalar array3[:] = scalar - @test zeros(size(tensor)) + scalar ≈ copy(array2) - @test zeros(Float16, size(tensor)) + scalar ≈ copy(array3) + @test zeros(size(tensor)) .+ scalar ≈ copy(array2) + @test zeros(Float16, size(tensor)) .+ scalar ≈ copy(array3) # NDArray -> NDArray assignment array[:] = array2 - @test zeros(size(tensor)) + scalar ≈ copy(array) + @test zeros(size(tensor)) .+ scalar ≈ copy(array) end function test_slice() @@ -170,7 +172,7 @@ function test_slice() end function test_linear_idx() - info("NDArray::getindex::linear indexing") + @info("NDArray::getindex::linear indexing") let A = reshape(1:30, 3, 10) x = mx.NDArray(A) @@ -201,7 +203,7 @@ function test_linear_idx() @test copy(x[14]) == [14] end - info("NDArray::setindex!::linear indexing") + @info("NDArray::setindex!::linear indexing") let A = reshape(1:24, 3, 2, 4) x = mx.NDArray(A) @@ -217,7 +219,7 @@ function test_linear_idx() @test copy(x[24]) == [42] end - info("NDArray::setindex!::type convert") + @info("NDArray::setindex!::type convert") let x = NDArray([1, 2, 3]) @test eltype(x) == Int @@ -227,7 +229,7 @@ function test_linear_idx() end # function test_linear_idx function test_first() - info("NDArray::first") + @info("NDArray::first") let A = reshape(1:30, 3, 10) x = mx.NDArray(A) @@ -239,12 +241,12 @@ function test_first() end end # function test_first -function test_endof() - info("NDArray::endof") +function test_lastindex() + @info("NDArray::lastindex") let A = [1 2; 3 4; 5 6], x = mx.NDArray(A) - @test endof(A) == endof(x) + @test lastindex(A) == lastindex(x) end -end # function test_endof +end # function test_lastindex function test_cat() function check_cat(f, A, B = 2A) @@ -270,39 +272,39 @@ function test_cat() end let A = [1, 2, 3, 4] - info("NDArray::hcat::1D") + @info("NDArray::hcat::1D") check_cat(:hcat, A) - info("NDArray::vcat::1D") + @info("NDArray::vcat::1D") check_cat(:vcat, A) end let A = [1 2; 3 4] - info("NDArray::hcat::2D") + @info("NDArray::hcat::2D") check_cat(:hcat, A) - info("NDArray::vcat::2D") + @info("NDArray::vcat::2D") check_cat(:vcat, A) end let A = rand(4, 3, 2) - info("NDArray::hcat::3D") + @info("NDArray::hcat::3D") check_cat(:hcat, A) - info("NDArray::vcat::3D") + @info("NDArray::vcat::3D") check_cat(:vcat, A) end let A = rand(4, 3, 2, 2) - info("NDArray::hcat::4D") + @info("NDArray::hcat::4D") check_cat(:hcat, A) - info("NDArray::vcat::4D") + @info("NDArray::vcat::4D") check_cat(:vcat, A) end let A = [1, 2, 3, 4] - info("NDArray::cat::3D/1D") + @info("NDArray::cat::3D/1D") check_cat(:vcat, reshape(A, 4, 1, 1), 2A) end end # function test_cat @@ -313,46 +315,45 @@ function test_plus() t2, a2 = rand_tensors(dims) t3, a3 = rand_tensors(dims) - info("NDArray::plus::dims = $dims") + @info("NDArray::plus::dims = $dims") - @test t1 + t2 ≈ copy(a1 + a2) @test t1 .+ t2 ≈ copy(a1 .+ a2) - @test t1 + t2 + t3 ≈ copy(a1 + a2 + a3) + @test t1 .+ t2 .+ t3 ≈ copy(a1 .+ a2 .+ a3) # test inplace += operation a0 = a1 # keep a reference to a1 @mx.inplace a1 += a2 # perform inplace += @test a0 == a1 # make sure they are still the same object @test copy(a0) ≈ copy(a1) - @test copy(a1) ≈ t1 + t2 + @test copy(a1) ≈ t1 .+ t2 # test scalar scalar = rand() - @test t3 + scalar ≈ copy(a3 + scalar) - @test t2 + scalar + t3 ≈ copy(a2 + scalar + a3) + @test t3 .+ scalar ≈ copy(a3 .+ scalar) + @test t2 .+ scalar .+ t3 ≈ copy(a2 .+ scalar .+ a3) # test small and large scalar t4 = zeros(Float32, dims) a4 = copy(t4, mx.cpu()) scalar_small = 1e-8 scalar_large = 1e8 - @test t4 + scalar_small ≈ copy(a4 .+ scalar_small) - @test t4 + scalar_large ≈ copy(a4 .+ scalar_large) + @test t4 .+ scalar_small ≈ copy(a4 .+ scalar_small) + @test t4 .+ scalar_large ≈ copy(a4 .+ scalar_large) t5 = zeros(Float64, dims) a5 = copy(t5, mx.cpu()) scalar_small = 1e-8 scalar_large = 1e8 - @test t5 + scalar_small ≈ copy(a5 .+ scalar_small) - @test t5 + scalar_large ≈ copy(a5 .+ scalar_large) + @test t5 .+ scalar_small ≈ copy(a5 .+ scalar_small) + @test t5 .+ scalar_large ≈ copy(a5 .+ scalar_large) t6 = zeros(Float16, dims) a6 = copy(t6, mx.cpu()) scalar_small = Float16(1e-5) scalar_large = Float16(1e4) - @test t6 + scalar_small ≈ copy(a6 .+ scalar_small) - @test t6 + scalar_large ≈ copy(a6 .+ scalar_large) + @test t6 .+ scalar_small ≈ copy(a6 .+ scalar_small) + @test t6 .+ scalar_large ≈ copy(a6 .+ scalar_large) let x = mx.NDArray([1 2; 3 4]), y = mx.NDArray([1 1; 1 1]) @test copy(42 .+ x) == [43 44; 45 46] @@ -360,7 +361,7 @@ function test_plus() @test copy(0 .+ x .+ y .+ 41) == [43 44; 45 46] end - info("NDArray::plus::scalar::type convert") + @info("NDArray::plus::scalar::type convert") let x = mx.NDArray([1, 2, 3]) y = x .+ 0.5 @test copy(y) == copy(x) @@ -369,7 +370,7 @@ function test_plus() @test copy(y) == [3, 4, 5] end - info("NDArray::broadcast_add") + @info("NDArray::broadcast_add") let A = [1 2 3; 4 5 6] @@ -392,9 +393,8 @@ function test_minus() t1, a1 = rand_tensors(dims) t2, a2 = rand_tensors(dims) - info("NDArray::minus::dims = $dims") + @info("NDArray::minus::dims = $dims") - @test t1 - t2 ≈ copy(a1 - a2) @test t1 .- t2 ≈ copy(a1 .- a2) @test -t1 ≈ copy(-a1) @@ -409,40 +409,40 @@ function test_minus() @test a0 == a1 # make sure they are still the same object @test a0.handle == a1.handle @test copy(a0) ≈ copy(a1) - @test copy(a1) ≈ t1 - t2 + @test copy(a1) ≈ t1 .- t2 # test scalar scalar = rand() - @test t2 - scalar ≈ copy(a2 - scalar) + @test t2 .- scalar ≈ copy(a2 .- scalar) # test small and large scalar t4 = zeros(Float32, dims) a4 = copy(t4, mx.cpu()) scalar_small = 1e-8 scalar_large = 1e8 - @test t4 - scalar_small ≈ copy(a4 .- scalar_small) - @test t4 - scalar_large ≈ copy(a4 .- scalar_large) + @test t4 .- scalar_small ≈ copy(a4 .- scalar_small) + @test t4 .- scalar_large ≈ copy(a4 .- scalar_large) t5 = zeros(Float64, dims) a5 = copy(t5, mx.cpu()) scalar_small = 1e-8 scalar_large = 1e8 - @test t5 - scalar_small ≈ copy(a5 .- scalar_small) - @test t5 - scalar_large ≈ copy(a5 .- scalar_large) + @test t5 .- scalar_small ≈ copy(a5 .- scalar_small) + @test t5 .- scalar_large ≈ copy(a5 .- scalar_large) t6 = zeros(Float16, dims) a6 = copy(t6, mx.cpu()) scalar_small = Float16(1e-5) scalar_large = Float16(1e4) - @test t6 - scalar_small ≈ copy(a6 .- scalar_small) - @test t6 - scalar_large ≈ copy(a6 .- scalar_large) + @test t6 .- scalar_small ≈ copy(a6 .- scalar_small) + @test t6 .- scalar_large ≈ copy(a6 .- scalar_large) - info("NDArray::minus::scalar::type convert") + @info("NDArray::minus::scalar::type convert") let x = mx.NDArray([1, 2, 3]) @test copy(x .- π) ≈ [-2, -1, 0] end - info("NDArray::broadcast_minus") + @info("NDArray::broadcast_minus") let A = [1 2 3; 4 5 6] @@ -458,6 +458,18 @@ function test_minus() # @inplace x .-= y # @test copy(x) == A .- B end + + @info("NDArray::scalar::rminus") + let + A = [1 2 3; + 4 5 6] + B = 10 .- A + + x = NDArray(A) + y = 10 .- x + + @test copy(y) == B + end end function test_mul() @@ -466,9 +478,9 @@ function test_mul() t2, a2 = rand_tensors(dims) t3, a3 = rand_tensors(dims) - info("NDArray::mul::dims = $dims") + @info("NDArray::mul::dims = $dims") - @test t1 .* t2 ≈ copy(a1.*a2) + @test t1 .* t2 ≈ copy(a1 .* a2) # test inplace .*= operation a0 = a1 # keep a reference to a1 @@ -480,7 +492,7 @@ function test_mul() # test scalar scalar = mx.MX_float(rand()) - @test t3 * scalar ≈ copy(a3 .* scalar) + @test t3 .* scalar ≈ copy(a3 .* scalar) # test small and large scalar t4, a4 = rand_tensors(Float32, dims) @@ -499,26 +511,26 @@ function test_mul() scalar_small = Float16(1e-5) @test t6 * scalar_small ≈ copy(a6 .* scalar_small) - info("NDArray::mul::matrix multiplication") + @info("NDArray::mul::matrix multiplication") let x = mx.NDArray([1. 2]) y = x' * x @test copy(y) == [1. 2; 2 4] end - info("NDArray::mul::elementwise::issue 253") + @info("NDArray::mul::elementwise::issue 253") let x = mx.NDArray([1. 2]) y = x .* x @test copy(y) == [1. 4.] end - info("NDArray::mul::scalar::type convert") + @info("NDArray::mul::scalar::type convert") let x = mx.NDArray([1, 2, 3]) y = x .* π @test eltype(x) == Int @test copy(y) == [3, 6, 9] end - info("NDArray::broadcast_mul") + @info("NDArray::broadcast_mul") let A = [1 2 3; 4 5 6] @@ -541,7 +553,7 @@ function test_div() t1, a1 = rand_tensors(dims) t2, a2 = rand_tensors(dims) - info("NDArray::div::dims = $dims") + @info("NDArray::div::dims = $dims") t2 .+= 2 # avoid numerical instability @mx.inplace a2 .+= 2 @@ -576,7 +588,7 @@ function test_div() scalar_large = 1e4 @test t6 ./ scalar_large ≈ copy(a6 ./ scalar_large) - info("NDArray::div::scalar::type convert") + @info("NDArray::div::scalar::type convert") let x = mx.NDArray([1, 2, 3]) y = x ./ 1.1 @test eltype(y) == Int @@ -589,7 +601,7 @@ function test_div() @test_throws AssertionError x ./ 0.5 end - info("NDArray::broadcast_div") + @info("NDArray::broadcast_div") let A = Float32[1 2 3; 4 5 6] @@ -607,37 +619,36 @@ function test_div() end end - function test_rdiv() - info("NDArray::rdiv") + @info("NDArray::rdiv") - info("NDArray::rdiv::Inf16") + @info("NDArray::rdiv::Inf16") let x = 1 ./ mx.zeros(Float16, 4) @test copy(x) == [Inf16, Inf16, Inf16, Inf16] end - info("NDArray::rdiv::Inf32") + @info("NDArray::rdiv::Inf32") let x = 1 ./ mx.zeros(Float32, 4) @test copy(x) == [Inf32, Inf32, Inf32, Inf32] end - info("NDArray::rdiv::Inf64") + @info("NDArray::rdiv::Inf64") let x = 1 ./ mx.zeros(Float64, 4) @test copy(x) == [Inf64, Inf64, Inf64, Inf64] end - info("NDArray::rdiv::Int") + @info("NDArray::rdiv::Int") let x = 1 ./ mx.NDArray([1 2; 3 4]) @test copy(x) == [1 0; 0 0] end - info("NDArray::rdiv::Float32") + @info("NDArray::rdiv::Float32") let x = 1 ./ mx.NDArray(Float32[1 2; 3 4]) y = 1 ./ Float32[1 2; 3 4] @test copy(x) ≈ y end - info("NDArray::rdiv::type convert") + @info("NDArray::rdiv::type convert") let x = mx.NDArray([1, 2, 3]) y = 5.5 ./ x @test eltype(y) == Int # this differs from julia @@ -645,11 +656,10 @@ function test_rdiv() end end # function test_rdiv - function test_mod() - info("NDArray::mod") - const A = [1 2; 3 4] - const B = [1 1; 3 3] + @info("NDArray::mod") + A = [1 2; 3 4] + B = [1 1; 3 3] let x = NDArray(A), y = NDArray(B) C = A .% B @@ -662,21 +672,21 @@ function test_mod() @test copy(z) ≈ D end - info("NDArray::mod::scalar") + @info("NDArray::mod::scalar") let x = NDArray(A) C = A .% 2 y = x .% 2 @test copy(y) ≈ C end - info("NDArray::rmod") + @info("NDArray::rmod") let x = NDArray(A) C = 11 .% A y = 11 .% x @test copy(y) ≈ C end - info("NDArray::mod_from!") + @info("NDArray::mod_from!") let x = NDArray(A) y = NDArray(B) @@ -694,7 +704,7 @@ function test_mod() @test copy(y) ≈ C end - info("NDArray::mod_from!::scalar") + @info("NDArray::mod_from!::scalar") let x = NDArray(A) C = A .% 2 @@ -702,7 +712,7 @@ function test_mod() @test copy(x) ≈ C end - info("NDArray::rmod_from!") + @info("NDArray::rmod_from!") let x = NDArray(A) C = 11 .% A @@ -710,7 +720,7 @@ function test_mod() @test copy(x) ≈ C end - info("NDArray::mod_from!::writable") + @info("NDArray::mod_from!::writable") let x = NDArray(A) y = NDArray(B) @@ -722,7 +732,7 @@ function test_mod() @test_throws AssertionError mx.rmod_from!(2, x) end - info("NDArray::mod::inplace") + @info("NDArray::mod::inplace") let x = NDArray(A) y = NDArray(B) @@ -731,7 +741,7 @@ function test_mod() @test copy(x) ≈ C end - info("NDArray::broadcast_mod") + @info("NDArray::broadcast_mod") let A = [1 2 3; 4 5 6] @@ -749,13 +759,12 @@ function test_mod() end end # function test_mod - function test_gd() dims = rand_dims() tw, aw = rand_tensors(dims) tg, ag = rand_tensors(dims) - info("NDArray::gd::dims = $dims") + @info("NDArray::gd::dims = $dims") lr = rand() wd = rand() @@ -765,10 +774,9 @@ function test_gd() @test copy(aw) ≈ tw end - function test_saveload() n_arrays = 5 - info("NDArray::saveload::n_arrays = $n_arrays") + @info("NDArray::saveload::n_arrays = $n_arrays") fname = tempname() # save and load a single array @@ -807,7 +815,7 @@ end function test_clip() dims = rand_dims() - info("NDArray::clip::dims = $dims") + @info("NDArray::clip::dims = $dims") j_array, nd_array = rand_tensors(dims) clip_up = maximum(abs.(j_array)) / 2 @@ -819,7 +827,7 @@ function test_clip() @test all(clip_down .<= copy(clipped) .<= clip_up) - info("NDArray::clip!") + @info("NDArray::clip!") let x = NDArray(1.0:20) clip!(x, 5, 15) @@ -828,91 +836,91 @@ function test_clip() end function test_power() - info("NDArray::power") + @info("NDArray::power") - info("NDArray::power::Int::x.^n") + @info("NDArray::power::Int::x .^ n") let x = mx.NDArray([1 2; 3 4]) @test eltype(x) == Int - @test copy(x.^-1) == [1 0; 0 0] - @test copy(x.^0) == [1 1; 1 1] - @test copy(x.^1) == [1 2; 3 4] - @test copy(x.^1.1) == [1 2; 3 4] - @test copy(x.^2) == [1 4; 9 16] - @test copy(x.^2.9) == [1 4; 9 16] - @test copy(x.^3) == [1 8; 27 64] + @test copy(x .^ -1) == [1 0; 0 0] + @test copy(x .^ 0) == [1 1; 1 1] + @test copy(x .^ 1) == [1 2; 3 4] + @test copy(x .^ 1.1) == [1 2; 3 4] + @test copy(x .^ 2) == [1 4; 9 16] + @test copy(x .^ 2.9) == [1 4; 9 16] + @test copy(x .^ 3) == [1 8; 27 64] end - info("NDArray::power::Int::n.^x") + @info("NDArray::power::Int::n .^ x") let x = mx.NDArray([1 2; 3 4]) @test eltype(x) == Int - @test copy(0.^x) == [0 0; 0 0] - @test copy(1.^x) == [1 1; 1 1] - @test copy(1.1.^x) == [1 1; 1 1] - @test copy(2.^x) == [2 4; 8 16] - @test copy(2.9.^x) == [2 4; 8 16] - @test copy(3.^x) == [3 9; 27 81] + @test copy(0 .^ x) == [0 0; 0 0] + @test copy(1 .^ x) == [1 1; 1 1] + @test copy(1.1 .^ x) == [1 1; 1 1] + @test copy(2 .^ x) == [2 4; 8 16] + @test copy(2.9 .^ x) == [2 4; 8 16] + @test copy(3 .^ x) == [3 9; 27 81] end - info("NDArray::power::Int::x.^y") + @info("NDArray::power::Int::x .^ y") let x = mx.NDArray([1 2; 3 4]), y = mx.NDArray([2 2; 2 2]) @test eltype(x) == Int @test eltype(y) == Int - @test copy(x.^y) == [1 4; 9 16] - @test copy(y.^x) == [2 4; 8 16] + @test copy(x .^ y) == [1 4; 9 16] + @test copy(y .^ x) == [2 4; 8 16] end - info("NDArray::power::Float32::x.^n") + @info("NDArray::power::Float32::x .^ n") let x = mx.NDArray(Float32[1 2; 3 4]), A = Float32[1 2; 3 4] @test eltype(x) == Float32 - @test copy(x.^0) == Float32[1 1; 1 1] - @test copy(x.^1) == Float32[1 2; 3 4] - @test copy(x.^2) == Float32[1 4; 9 16] - @test copy(x.^3) == Float32[1 8; 27 64] + @test copy(x .^ 0) == Float32[1 1; 1 1] + @test copy(x .^ 1) == Float32[1 2; 3 4] + @test copy(x .^ 2) == Float32[1 4; 9 16] + @test copy(x .^ 3) == Float32[1 8; 27 64] - @test copy(x.^-1) ≈ A.^-1 - @test copy(x.^1.1) ≈ A.^1.1 - @test copy(x.^2.9) ≈ A.^2.9 + @test copy(x .^ -1) ≈ A .^ -1 + @test copy(x .^ 1.1) ≈ A .^ 1.1 + @test copy(x .^ 2.9) ≈ A .^ 2.9 end - info("NDArray::power::Float32::n.^x") + @info("NDArray::power::Float32::n .^ x") let x = mx.NDArray(Float32[1 2; 3 4]), A = Float32[1 2; 3 4] @test eltype(x) == Float32 - @test copy(0.^x) == Float32[0 0; 0 0] - @test copy(1.^x) == Float32[1 1; 1 1] - @test copy(2.^x) == Float32[2 4; 8 16] - @test copy(3.^x) == Float32[3 9; 27 81] + @test copy(0 .^ x) == Float32[0 0; 0 0] + @test copy(1 .^ x) == Float32[1 1; 1 1] + @test copy(2 .^ x) == Float32[2 4; 8 16] + @test copy(3 .^ x) == Float32[3 9; 27 81] - @test copy(1.1.^x) ≈ 1.1.^A - @test copy(2.9.^x) ≈ 2.9.^A + @test copy(1.1 .^ x) ≈ 1.1 .^ A + @test copy(2.9 .^ x) ≈ 2.9 .^ A end - info("NDArray::power::Float32::x.^y") + @info("NDArray::power::Float32::x .^ y") let x = mx.NDArray(Float32[1 2; 3 4]), y = mx.NDArray(Float32[2 2; 2 2]) @test eltype(x) == Float32 @test eltype(y) == Float32 - @test copy(x.^y) == Float32[1 4; 9 16] - @test copy(y.^x) == Float32[2 4; 8 16] + @test copy(x .^ y) == Float32[1 4; 9 16] + @test copy(y .^ x) == Float32[2 4; 8 16] end - info("NDArray::power::e.^x::x.^e") + @info("NDArray::power::ℯ .^ x::x .^ ℯ") let x = mx.zeros(2, 3), A = [1 1 1; 1 1 1] - @test copy(e.^x) ≈ A + @test copy(ℯ .^ x) ≈ A end let A = Float32[1 2; 3 4], x = mx.NDArray(A) - @test copy(e.^x) ≈ e.^A - @test copy(x.^e) ≈ A.^e + @test copy(ℯ .^ x) ≈ ℯ .^ A + @test copy(x .^ ℯ) ≈ A .^ ℯ end - info("NDArray::power::π.^x::x.^π") + @info("NDArray::power::π .^ x::x .^ π") let A = Float32[1 2; 3 4], x = mx.NDArray(A) - @test copy(π.^x) ≈ π.^A - @test copy(x.^π) ≈ A.^π + @test copy(π .^ x) ≈ π .^ A + @test copy(x .^ π) ≈ A .^ π end # TODO: Float64: wait for https://github.com/apache/incubator-mxnet/pull/8012 - info("NDArray::broadcast_power") + @info("NDArray::broadcast_power") let A = [1 2 3; 4 5 6] @@ -932,7 +940,7 @@ end # function test_power function test_sqrt() dims = rand_dims() - info("NDArray::sqrt::dims = $dims") + @info("NDArray::sqrt::dims = $dims") j_array, nd_array = rand_tensors(dims) sqrt_ed = sqrt(nd_array) @@ -941,7 +949,7 @@ end function test_nd_as_jl() dims = (2, 3) - info("NDArray::nd_as_jl::dims = $dims") + @info("NDArray::nd_as_jl::dims = $dims") x = mx.zeros(dims) + 5 y = mx.ones(dims) @@ -952,7 +960,7 @@ function test_nd_as_jl() end z[:, 1] = y[:, 1] - y[:] = 0 + y .= 0 end @test sum(copy(y)) == 0 @@ -963,7 +971,7 @@ end function test_dot() dims1 = (2, 3) dims2 = (3, 8) - info("NDArray::dot") + @info("NDArray::dot") x = mx.zeros(dims1) y = mx.zeros(dims2) @@ -974,7 +982,7 @@ function test_dot() y = mx.zeros(1, 2, 3) @test_throws mx.MXError dot(x, y) # dimension mismatch - info("NDArray::matrix mul") + @info("NDArray::matrix mul") let A = [1. 2 3; 4 5 6] B = [-1., -2, -3] @@ -997,7 +1005,7 @@ function test_dot() end function test_eltype() - info("NDArray::eltype") + @info("NDArray::eltype") dims1 = (3,3) x = mx.empty(dims1) @@ -1011,7 +1019,7 @@ function test_eltype() end function test_reshape() - info("NDArray::reshape") + @info("NDArray::reshape") A = rand(2, 3, 4) B = reshape(NDArray(A), 4, 3, 2) @@ -1022,7 +1030,7 @@ function test_reshape() @test size(C) == (4, 3, 2) @test copy(C)[3, 1, 1] == A[1, 2, 1] - info("NDArray::reshape::reverse") + @info("NDArray::reshape::reverse") A = mx.zeros(10, 5, 4) B = reshape(A, -1, 0) @@ -1033,7 +1041,7 @@ function test_reshape() end function test_expand_dims() - info("NDArray::expand_dims") + @info("NDArray::expand_dims") let A = [1, 2, 3, 4], x = NDArray(A) @test size(x) == (4,) @@ -1059,44 +1067,44 @@ function test_expand_dims() end # test_expand_dims function test_sum() - info("NDArray::sum") + @info("NDArray::sum") let A = reshape(1.0:8, 2, 2, 2), X = mx.NDArray(A) - @test copy(sum(X))[] == sum(A) - @test copy(sum(X, 1)) == sum(A, 1) - @test copy(sum(X, 2)) == sum(A, 2) - @test copy(sum(X, 3)) == sum(A, 3) - @test copy(sum(X, [1, 2])) == sum(A, [1, 2]) - @test copy(sum(X, (1, 2))) == sum(A, (1, 2)) + @test copy(sum(X))[] == sum(A) + @test copy(sum(X, dims = 1)) == sum(A, dims = 1) + @test copy(sum(X, dims = 2)) == sum(A, dims = 2) + @test copy(sum(X, dims = 3)) == sum(A, dims = 3) + @test copy(sum(X, dims = [1, 2])) == sum(A, dims = [1, 2]) + @test copy(sum(X, dims = (1, 2))) == sum(A, dims = (1, 2)) end end function test_mean() - info("NDArray::mean") + @info("NDArray::mean") let A = reshape(1.0:8, 2, 2, 2), X = mx.NDArray(A) - @test copy(mean(X))[] == mean(A) - @test copy(mean(X, 1)) == mean(A, 1) - @test copy(mean(X, 2)) == mean(A, 2) - @test copy(mean(X, 3)) == mean(A, 3) - @test copy(mean(X, [1, 2])) == mean(A, [1, 2]) - @test copy(mean(X, (1, 2))) == mean(A, (1, 2)) + @test copy(mean(X))[] == mean(A) + @test copy(mean(X, dims = 1)) == mean(A, dims = 1) + @test copy(mean(X, dims = 2)) == mean(A, dims = 2) + @test copy(mean(X, dims = 3)) == mean(A, dims = 3) + @test copy(mean(X, dims = [1, 2])) == mean(A, dims = [1, 2]) + @test copy(mean(X, dims = (1, 2))) == mean(A, dims = (1, 2)) end end function test_maximum() - info("NDArray::maximum") + @info("NDArray::maximum") let A = reshape(1.0:8, 2, 2, 2), X = mx.NDArray(A) - @test copy(maximum(X))[] == maximum(A) - @test copy(maximum(X, 1)) == maximum(A, 1) - @test copy(maximum(X, 2)) == maximum(A, 2) - @test copy(maximum(X, 3)) == maximum(A, 3) - @test copy(maximum(X, [1, 2])) == maximum(A, [1, 2]) - @test copy(maximum(X, (1, 2))) == maximum(A, (1, 2)) + @test copy(maximum(X))[] == maximum(A) + @test copy(maximum(X, dims = 1)) == maximum(A, dims = 1) + @test copy(maximum(X, dims = 2)) == maximum(A, dims = 2) + @test copy(maximum(X, dims = 3)) == maximum(A, dims = 3) + @test copy(maximum(X, dims = [1, 2])) == maximum(A, dims = [1, 2]) + @test copy(maximum(X, dims = (1, 2))) == maximum(A, dims = (1, 2)) end - info("NDArray::broadcast_maximum") + @info("NDArray::broadcast_maximum") let A = [1 2 3; 4 5 6] @@ -1111,18 +1119,18 @@ function test_maximum() end function test_minimum() - info("NDArray::minimum") + @info("NDArray::minimum") let A = reshape(1.0:8, 2, 2, 2), X = mx.NDArray(A) - @test copy(minimum(X))[] == minimum(A) - @test copy(minimum(X, 1)) == minimum(A, 1) - @test copy(minimum(X, 2)) == minimum(A, 2) - @test copy(minimum(X, 3)) == minimum(A, 3) - @test copy(minimum(X, [1, 2])) == minimum(A, [1, 2]) - @test copy(minimum(X, (1, 2))) == minimum(A, (1, 2)) + @test copy(minimum(X))[] == minimum(A) + @test copy(minimum(X, dims = 1)) == minimum(A, dims = 1) + @test copy(minimum(X, dims = 2)) == minimum(A, dims = 2) + @test copy(minimum(X, dims = 3)) == minimum(A, dims = 3) + @test copy(minimum(X, dims = [1, 2])) == minimum(A, dims = [1, 2]) + @test copy(minimum(X, dims = (1, 2))) == minimum(A, dims = (1, 2)) end - info("NDArray::broadcast_minimum") + @info("NDArray::broadcast_minimum") let A = [1 2 3; 4 5 6] @@ -1137,20 +1145,20 @@ function test_minimum() end function test_prod() - info("NDArray::prod") + @info("NDArray::prod") let A = reshape(1.0:8, 2, 2, 2), X = mx.NDArray(A) - @test copy(prod(X))[] == prod(A) - @test copy(prod(X, 1)) == prod(A, 1) - @test copy(prod(X, 2)) == prod(A, 2) - @test copy(prod(X, 3)) == prod(A, 3) - @test copy(prod(X, [1, 2])) == prod(A, [1, 2]) - @test copy(prod(X, (1, 2))) == prod(A, (1, 2)) + @test copy(prod(X))[] == prod(A) + @test copy(prod(X, dims = 1)) == prod(A, dims = 1) + @test copy(prod(X, dims = 2)) == prod(A, dims = 2) + @test copy(prod(X, dims = 3)) == prod(A, dims = 3) + @test copy(prod(X, dims = [1, 2])) == prod(A, dims = [1, 2]) + @test copy(prod(X, dims = (1, 2))) == prod(A, dims = (1, 2)) end end function test_fill() - info("NDArray::fill") + @info("NDArray::fill") let x = mx.fill(42, 2, 3, 4) @test eltype(x) == Int @@ -1176,7 +1184,7 @@ function test_fill() @test copy(x) ≈ fill(Float32(42), 2, 3, 4) end - info("NDArray::fill!::arr") + @info("NDArray::fill!::arr") let x = fill!(mx.zeros(2, 3, 4), 42) @test eltype(x) == Float32 @test size(x) == (2, 3, 4) @@ -1185,19 +1193,19 @@ function test_fill() end # function test_fill function test_transpose() - info("NDArray::transpose::1D") + @info("NDArray::transpose::1D") let A = rand(Float32, 4), x = NDArray(A) @test size(x) == (4,) @test size(x') == (1, 4) end - info("NDArray::transpose::2D") + @info("NDArray::transpose::2D") let A = rand(Float32, 2, 3), x = mx.NDArray(A) @test size(x) == (2, 3) @test size(x') == (3, 2) end - info("NDArray::permutedims") + @info("NDArray::permutedims") let A = collect(Float32, reshape(1.0:24, 2, 3, 4)), x = mx.NDArray(A) A′ = permutedims(A, [2, 1, 3]) x′ = permutedims(x, [2, 1, 3]) @@ -1207,36 +1215,37 @@ function test_transpose() end function test_show() - info("NDArray::show::REPL") + @info("NDArray::show::REPL") let str = sprint(show, MIME"text/plain"(), mx.NDArray([1 2 3 4])) - @test contains(str, "1×4") - @test contains(str, "mx.NDArray") - @test contains(str, "Int64") - @test contains(str, "CPU") + @test occursin("1×4", str) + @test occursin("NDArray", str) + @test occursin("Int64", str) + @test occursin("CPU", str) @test match(r"1\s+2\s+3\s+4", str) != nothing end - info("NDArray::show") + @info("NDArray::show") let str = sprint(show, mx.NDArray([1 2 3 4])) - @test str == "NDArray [1 2 3 4]" + @test str == "NDArray([1 2 3 4])" end let str = sprint(show, mx.zeros(4)) - @test str == "NDArray Float32[0.0, 0.0, 0.0, 0.0]" + @test str == "NDArray(Float32[0.0, 0.0, 0.0, 0.0])" end end function test_size() - info("NDArray::size") + @info("NDArray::size") let A = [1 2; 3 4; 5 6], x = mx.NDArray(A) @test size(A) == size(x) - @test size(A, 1, 2, 3, 4, 5) == size(x, 1, 2, 3, 4, 5) - @inferred size(x, 1, 2, 3, 4, 5) + dims = (1, 2, 3, 4, 5) + @test map(d -> size(A, d), dims) == map(d -> size(x, d), dims) + @inferred map(d -> size(x, d), dims) end end # function test_size() function check_trigonometric(f) - info("NDArray::$f") + @info("NDArray::$f") let A = [.1 .2; .3 .4], x = mx.NDArray(A) B = f.(A) y = f.(x) @@ -1257,7 +1266,7 @@ function test_trigonometric() end # function test_trigonometric function check_hyperbolic(f, A) - info("NDArray::$f") + @info("NDArray::$f") let x = NDArray(A) B = f.(A) y = f.(x) @@ -1283,10 +1292,10 @@ function test_hyperbolic() end # function test_hyperbolic function test_act_funcs() - info("NDArray::σ/sigmoid") + @info("NDArray::σ/sigmoid") let A = Float32[.1, .2, -.3, -.4] - B = @. 1 / (1 + e^(-A)) + B = @. 1 / (1 + ℯ ^ (-A)) x = NDArray(A) y = σ.(x) @test copy(y) ≈ B @@ -1295,7 +1304,7 @@ function test_act_funcs() @test copy(z) ≈ B end - info("NDArray::relu") + @info("NDArray::relu") let A = [1, 2, -3, -4] B = max.(A, 0) @@ -1304,7 +1313,7 @@ function test_act_funcs() @test copy(y) ≈ B end - info("NDArray::softmax::1D") + @info("NDArray::softmax::1D") let A = Float32[1, 2, 3, 4] B = exp.(A) ./ sum(exp.(A)) @@ -1313,20 +1322,20 @@ function test_act_funcs() @test copy(y) ≈ B end - info("NDArray::softmax::2D") + @info("NDArray::softmax::2D") let A = Float32[1 2; 3 4] - B = exp.(A) ./ sum(exp.(A), 1) + B = exp.(A) ./ sum(exp.(A), dims = 1) x = NDArray(A) y = softmax.(x, 1) @test copy(y) ≈ B - C = exp.(A) ./ sum(exp.(A), 2) + C = exp.(A) ./ sum(exp.(A), dims = 2) z = softmax.(x, 2) @test copy(z) ≈ C end - info("NDArray::log_softmax::1D") + @info("NDArray::log_softmax::1D") let A = Float32[1, 2, 3, 4] B = log.(exp.(A) ./ sum(exp.(A))) @@ -1335,15 +1344,15 @@ function test_act_funcs() @test copy(y) ≈ B end - info("NDArray::log_softmax::2D") + @info("NDArray::log_softmax::2D") let A = Float32[1 2; 3 4] - B = log.(exp.(A) ./ sum(exp.(A), 1)) + B = log.(exp.(A) ./ sum(exp.(A), dims = 1)) x = NDArray(A) y = log_softmax.(x, 1) @test copy(y) ≈ B - C = log.(exp.(A) ./ sum(exp.(A), 2)) + C = log.(exp.(A) ./ sum(exp.(A), dims = 2)) z = log_softmax.(x, 2) @test copy(z) ≈ C end @@ -1369,27 +1378,27 @@ macro check_equal(op) end function test_equal() - info("NDArray::broadcast_equal") + @info("NDArray::broadcast_equal") @check_equal == - info("NDArray::broadcast_not_equal") + @info("NDArray::broadcast_not_equal") @check_equal != - info("NDArray::broadcast_greater") + @info("NDArray::broadcast_greater") @check_equal > - info("NDArray::broadcast_greater_equal") + @info("NDArray::broadcast_greater_equal") @check_equal >= - info("NDArray::broadcast_lesser") + @info("NDArray::broadcast_lesser") @check_equal < - info("NDArray::broadcast_lesser_equal") + @info("NDArray::broadcast_lesser_equal") @check_equal <= end # function test_equal function test_broadcast_to() - info("NDArray::broadcast_to") + @info("NDArray::broadcast_to") A = [1 2 3] x = NDArray(A) @test mx.broadcast_to(x, (1, 3)) |> copy == A @@ -1400,20 +1409,20 @@ function test_broadcast_to() end # function test_broadcast_to function test_broadcast_axis() - info("NDArray::broadcast_axis") + @info("NDArray::broadcast_axis") A = reshape([1, 2, 3], 1, 3, 1) x = NDArray(A) @test mx.broadcast_axis(x, 1, 4) |> copy == [A; A; A; A] - @test mx.broadcast_axis(x, 3, 2) |> copy == cat(3, A, A) + @test mx.broadcast_axis(x, 3, 2) |> copy == cat(A, A, dims = 3) - info("NDArray::broadcast_axes") + @info("NDArray::broadcast_axes") @test mx.broadcast_axes(x, 1, 4) |> copy == [A; A; A; A] - @test mx.broadcast_axes(x, 3, 2) |> copy == cat(3, A, A) + @test mx.broadcast_axes(x, 3, 2) |> copy == cat(A, A, dims = 3) end # function test_broadcast_axis function test_hypot() - info("NDArray::hypot") + @info("NDArray::hypot") A = [3 3 3] B = [4, 4] C = hypot.(A, B) @@ -1436,7 +1445,7 @@ end # function test_hypot test_slice() test_linear_idx() test_first() - test_endof() + test_lastindex() test_cat() test_plus() test_minus() diff --git a/julia/test/unittest/operator.jl b/julia/test/unittest/operator.jl index ed8312d91..345dd0f88 100644 --- a/julia/test/unittest/operator.jl +++ b/julia/test/unittest/operator.jl @@ -18,14 +18,14 @@ module TestOperator using MXNet -using Base.Test +using Test using ..Main: rand_dims function test_scalar_op() data = mx.Variable(:data) shape = rand_dims() - info("Operator::scalar_op::dims = $shape") + @info "Operator::scalar_op::dims = $shape" data_jl = 5ones(Float32, shape) arr_data = mx.copy(data_jl, mx.cpu()) @@ -35,7 +35,7 @@ function test_scalar_op() exec_test = mx.bind(test, mx.cpu(), [arr_data], args_grad=[arr_grad]) mx.forward(exec_test) out = copy(exec_test.outputs[1]) - jl_out1 = (4 - ((1+data_jl+1)*2/5) - 0.2) + jl_out1 = @. 4 - ((1+data_jl+1)*2/5) - 0.2 jl_out = 2 ./ jl_out1 @test copy(out) ≈ jl_out diff --git a/julia/test/unittest/optimizer.jl b/julia/test/unittest/optimizer.jl index b068f12ff..cd1e7ebb4 100644 --- a/julia/test/unittest/optimizer.jl +++ b/julia/test/unittest/optimizer.jl @@ -17,7 +17,7 @@ module TestOptimizer -using Base.Test +using Test using MXNet using MXNet.mx.LearningRate @@ -25,7 +25,7 @@ using MXNet.mx.Momentum function test_fixed_η() - info("Optimizer::LearningRate::Fixed") + @info "Optimizer::LearningRate::Fixed" x = LearningRate.Fixed(.42) @test get(x) == .42 update!(x) @@ -34,7 +34,7 @@ end # function test_fixed_η function check_η_decay(x) - info("Optimizer::LearningRate::$x") + @info "Optimizer::LearningRate::$x" η = get(x) @test η == 1 @@ -55,14 +55,14 @@ test_inv_η() = LearningRate.Inv(1) |> check_η_decay function test_μ_null() - info("Optimizer::Momentum::Null") + @info "Optimizer::Momentum::Null" x = Momentum.Null() @test iszero(get(x)) end function test_μ_fixed() - info("Optimizer::Momentum::Fixed") + @info "Optimizer::Momentum::Fixed" x = Momentum.Fixed(42) @test get(x) == 42 end diff --git a/julia/test/unittest/random.jl b/julia/test/unittest/random.jl index 973a4bc32..013e4f609 100644 --- a/julia/test/unittest/random.jl +++ b/julia/test/unittest/random.jl @@ -17,18 +17,19 @@ module TestRandom using MXNet -using Base.Test +using Test +using Statistics function test_uniform() dims = (100, 100, 2) - info("random::uniform::dims = $dims") + @info "random::uniform::dims = $dims" low = -10; high = 10 seed = 123 - mx.srand(seed) + mx.seed!(seed) ret1 = mx.rand(dims..., low = low, high = high) - mx.srand(seed) + mx.seed!(seed) ret2 = mx.empty(dims) mx.rand!(ret2, low = low, high = high) @@ -38,14 +39,14 @@ end function test_gaussian() dims = (80, 80, 4) - info("random::gaussian::dims = $dims") + @info "random::gaussian::dims = $dims" μ = 10; σ = 2 seed = 456 - mx.srand(seed) + mx.seed!(seed) ret1 = mx.randn(dims..., μ = μ, σ = σ) - mx.srand(seed) + mx.seed!(seed) ret2 = mx.empty(dims) mx.randn!(ret2, μ = μ, σ = σ) diff --git a/julia/test/unittest/symbolic-node.jl b/julia/test/unittest/symbolic-node.jl index 507af1733..07ef05f70 100644 --- a/julia/test/unittest/symbolic-node.jl +++ b/julia/test/unittest/symbolic-node.jl @@ -18,7 +18,7 @@ module TestSymbolicNode using MXNet -using Base.Test +using Test using ..Main: mlp2, mlpchain, exec @@ -26,7 +26,7 @@ using ..Main: mlp2, mlpchain, exec # Test Implementations ################################################################################ function test_basic() - info("SymbolicNode::basic") + @info("SymbolicNode::basic") model = mlp2() @test mx.list_arguments(model) == [:data,:fc1_weight,:fc1_bias,:fc2_weight,:fc2_bias] @@ -35,7 +35,7 @@ function test_basic() end function test_chain() - info("SymbolicNode::chain") + @info("SymbolicNode::chain") model = mlpchain() @test mx.list_arguments(model) == [:data,:fc1_weight,:fc1_bias,:fc2_weight,:fc2_bias] @@ -57,7 +57,7 @@ function test_chain() end function test_internal() - info("SymbolicNode::internal") + @info("SymbolicNode::internal") data = mx.Variable(:data) oldfc = mx.FullyConnected(data, name=:fc1, num_hidden=10) @@ -71,7 +71,7 @@ function test_internal() end function test_get_children() - info("SymbolicNode::get_children") + @info("SymbolicNode::get_children") let x = mx.Variable(:x), y = mx.Variable(:y) z = x + y @@ -80,7 +80,7 @@ function test_get_children() @test mx.list_outputs(mx.get_children(z)) == [:x, :y] end - info("SymbolicNode::get_children::on leaf") + @info("SymbolicNode::get_children::on leaf") let x = mx.Variable(:x) @test mx.get_children(x) == nothing end @@ -88,7 +88,7 @@ end # test_get_children function test_compose() - info("SymbolicNode::compose") + @info("SymbolicNode::compose") data = mx.Variable(:data) net1 = mx.FullyConnected(data, name=:fc1, num_hidden=10) @@ -104,7 +104,7 @@ function test_compose() end function test_infer_shape() - info("SymbolicNode::infer_shape::mlp2") + @info("SymbolicNode::infer_shape::mlp2") model = mlp2() data_shape = (100, 100) @@ -118,7 +118,7 @@ function test_infer_shape() end function test_infer_shape_error() - info("SymbolicNode::infer_shape::throws") + @info("SymbolicNode::infer_shape::throws") model = mlp2() weight_shape = (100, 1) @@ -127,7 +127,7 @@ function test_infer_shape_error() end function test_saveload() - info("SymbolicNode::saveload::mlp2") + @info("SymbolicNode::saveload::mlp2") model = mlp2() fname = tempname() @@ -139,37 +139,37 @@ function test_saveload() end function test_attrs() - info("SymbolicNode::Attributes") + @info("SymbolicNode::Attributes") data = mx.Variable(:data) @test mx.get_name(data) == :data result = mx.get_attr(data, :test) - @test isnull(result) + @test ismissing(result) mx.set_attr(data, :test, "1.0") result = mx.get_attr(data, :test) - @test !isnull(result) - @test get(result) == "1.0" + @test !ismissing(result) + @test result == "1.0" data2 = mx.Variable(:data2, attrs = Dict(:test => "hallo!")) - @test get(mx.get_attr(data2, :test)) == "hallo!" + @test mx.get_attr(data2, :test) == "hallo!" conv = mx.Convolution(data2, kernel = (1,1), num_filter = 1) - @test isnull(mx.get_attr(conv, :b)) - @test isa(mx.get_name(conv), Symbol) + @test ismissing(mx.get_attr(conv, :b)) + @test mx.get_name(conv) isa Symbol @test_throws MethodError mx.Variable(:data3, attrs = Dict(:test => "1.0", :test2 => 1.0)) @test_throws MethodError mx.Convolution(data2, kernel = (1,1), num_filter = 1, attrs = Dict(:test => "1.0", :test2 => 1.0)) end function test_functions() - info("SymbolicNode::Functions") + @info("SymbolicNode::Functions") data = mx.Variable(:data) typeof(mx.sum(data)) == mx.SymbolicNode end function test_reshape() - info("SymbolicNode::reshape(sym, dim...)") + @info("SymbolicNode::reshape(sym, dim...)") A = mx.NDArray(collect(1:24)) x = mx.Variable(:x) @@ -181,7 +181,7 @@ function test_reshape() @test size(out) == (2, 3, 4) @test copy(out) == reshape(1:24, 2, 3, 4) - info("SymbolicNode::reshape(sym, dim)") + @info("SymbolicNode::reshape(sym, dim)") A = mx.NDArray(collect(1:24)) x = mx.Variable(:x) @@ -193,18 +193,18 @@ function test_reshape() @test size(out) == (2, 3, 4) @test copy(out) == reshape(1:24, 2, 3, 4) - info("SymbolicNode::reshape::reverse") + @info("SymbolicNode::reshape::reverse") A = mx.zeros(10, 5, 4) x = mx.Variable(:x) - y = mx.reshape(x, -1, 0, reverse=true) + y = mx.reshape(x, -1, 0, reverse = true) e = mx.bind(y, mx.cpu(), Dict(:x => A)) mx.forward(e) out = e.outputs[1] @test size(out) == (50, 4) - info("SymbolicNode::reshape::0") + @info("SymbolicNode::reshape::0") A = mx.zeros(2, 3, 4) x = mx.Variable(:x) @@ -215,7 +215,7 @@ function test_reshape() @test size(out) == (4, 3, 2) - info("SymbolicNode::reshape::-1") + @info("SymbolicNode::reshape::-1") A = mx.zeros(2, 3, 4) x = mx.Variable(:x) @@ -226,7 +226,7 @@ function test_reshape() @test size(out) == (6, 1, 4) - info("SymbolicNode::reshape::-2") + @info("SymbolicNode::reshape::-2") A = mx.zeros(2, 3, 4, 2) x = mx.Variable(:x) @@ -237,7 +237,7 @@ function test_reshape() @test size(out) == (3, 2, 4, 2) - info("SymbolicNode::reshape::-3") + @info("SymbolicNode::reshape::-3") A = mx.zeros(2, 3, 4, 5) x = mx.Variable(:x) @@ -248,7 +248,7 @@ function test_reshape() @test size(out) == (6, 20) - info("SymbolicNode::reshape::-4") + @info("SymbolicNode::reshape::-4") A = mx.zeros(2, 3, 4) x = mx.Variable(:x) @@ -261,12 +261,12 @@ function test_reshape() end function test_dot() - info("SymbolicNode::dot") + @info("SymbolicNode::dot") x = mx.Variable(:x) y = mx.Variable(:y) z = mx.dot(x, y) - z_exec = mx.bind(z, context=mx.cpu(), - args=Dict(:x => mx.ones((100, 2)), :y => mx.ones((2, 200)))) + z_exec = mx.bind(z, context = mx.cpu(), + args = Dict(:x => mx.ones((100, 2)), :y => mx.ones((2, 200)))) mx.forward(z_exec) ret = copy(z_exec.outputs[1]) @@ -275,14 +275,14 @@ function test_dot() end function test_print() - info("SymbolicNode::print") + @info("SymbolicNode::print") io = IOBuffer() print(io, mx.Variable(:x)) @test !isempty(String(take!(io))) end function test_misc() - info("SymbolicNode::Miscellaneous") + @info("SymbolicNode::Miscellaneous") # Test for #189 a = mx.Variable("a") b = mx.Variable("b") @@ -290,7 +290,7 @@ function test_misc() end function test_add() - info("SymbolicNode::elementwise add") + @info("SymbolicNode::elementwise add") let x = mx.Variable(:x), A = Float32[1 2; 3 4] let y = exec(x .+ 42; :x => A)[] @test size(y) == size(A) @@ -329,7 +329,7 @@ function test_add() end # function test_add function test_minus() - info("SymbolicNode::elementwise minus") + @info("SymbolicNode::elementwise minus") let x = mx.Variable(:x), A = Float32[1 2; 3 4] let y = exec(x .- 42; :x => A)[] @test size(y) == size(A) @@ -373,7 +373,7 @@ function test_minus() end # function test_minus function test_mul() - info("SymbolicNode::elementwise mul") + @info("SymbolicNode::elementwise mul") let x = mx.Variable(:x), A = Float32[1 2; 3 4] let y = exec(x .* 42; :x => A)[] @test size(y) == size(A) @@ -412,7 +412,7 @@ function test_mul() end # function test_mul function test_div() - info("SymbolicNode::elementwise div") + @info("SymbolicNode::elementwise div") let x = mx.Variable(:x), A = Float32[1 2; 3 4] let y = exec(x ./ 42; :x => A)[] @test size(y) == size(A) @@ -451,16 +451,16 @@ function test_div() end # function test_div function test_power() - info("SymbolicNode::elementwise power") + @info("SymbolicNode::elementwise power") let x = mx.Variable(:x), A = Float32[1 2; 3 4] - let y = exec(x.^42; :x => A)[] + let y = exec(x .^ 42; :x => A)[] @test size(y) == size(A) - @test copy(y) ≈ A.^42 + @test copy(y) ≈ A .^ 42 end - let y = exec(42.^x; :x => A)[] + let y = exec(42 .^ x; :x => A)[] @test size(y) == size(A) - @test copy(y) ≈ 42.^A + @test copy(y) ≈ 42 .^ A end end @@ -468,61 +468,61 @@ function test_power() x = mx.Variable(:x) y = mx.Variable(:y) - let z = x.^y + let z = x .^ y z = exec(z; :x => A, :y => B)[] @test size(z) == size(A) - @test copy(z) ≈ A.^B + @test copy(z) ≈ A .^ B end - let z = y.^x + let z = y .^ x z = exec(z; :x => A, :y => B)[] @test size(z) == size(A) - @test copy(z) ≈ B.^A + @test copy(z) ≈ B .^ A end end - info("SymbolicNode::power::e.^x::x.^e") + @info("SymbolicNode::power::e .^ x::x .^ e") let x = mx.Variable(:x), A = [0 0 0; 0 0 0] - y = exec(e.^x; :x => A)[] - @test copy(y) ≈ ones(A) + y = exec(ℯ .^ x; :x => A)[] + @test copy(y) ≈ fill(1, size(A)) end let x = mx.Variable(:x), A = Float32[1 2; 3 4] - let y = e.^x + let y = ℯ .^ x z = exec(y; :x => A)[] - @test copy(z) ≈ e.^A + @test copy(z) ≈ ℯ .^ A end - let y = x.^e + let y = x .^ ℯ z = exec(y; :x => A)[] - @test copy(z) ≈ A.^e + @test copy(z) ≈ A .^ ℯ end end - info("SymbolicNode::power::π.^x::x.^π") + @info("SymbolicNode::power::π .^ x::x .^ π") let x = mx.Variable(:x), A = Float32[1 2; 3 4] - let y = π.^x + let y = π .^ x z = exec(y; :x => A)[] - @test copy(z) ≈ π.^A + @test copy(z) ≈ π .^ A end - let y = x.^π + let y = x .^ π z = exec(y; :x => A)[] - @test copy(z) ≈ A.^π + @test copy(z) ≈ A .^ π end end end # function test_power function test_get_name() - info("SymbolicNode::get_name::with get_internals") + @info("SymbolicNode::get_name::with get_internals") name = mx.get_name(mx.get_internals(mlp2())) # no error - @test contains(name, "Ptr") + @test occursin("Ptr", name) end # function test_get_name function test_var() - info("SymbolicNode::var") + @info("SymbolicNode::var") x = @mx.var x @test x isa mx.SymbolicNode diff --git a/julia/test/unittest/util.jl b/julia/test/unittest/util.jl index ddd613ca4..d7f65a3e8 100644 --- a/julia/test/unittest/util.jl +++ b/julia/test/unittest/util.jl @@ -17,19 +17,18 @@ module TestUtil -using Base.Test - using MXNet +using Test function test_getdocdefine() - info("Util::_getdocdefine") - @test contains(mx._getdocdefine("sgd_update"), "Defined in") + @info("Util::_getdocdefine") + @test occursin("Defined in", mx._getdocdefine("sgd_update")) end # function test_getdocdefine function test_firstarg() - info("Util::_firstarg") + @info("Util::_firstarg") @test mx._firstarg(:(f(x, y))) == :x @test mx._firstarg(:(f(x::mx.NDArray, y))) == :x @test mx._firstarg(:(f(x::mx.NDArray, y::mx.NDArray))) == :x diff --git a/julia/test/unittest/visualize.jl b/julia/test/unittest/visualize.jl index 58d111b0f..a5a4f722e 100644 --- a/julia/test/unittest/visualize.jl +++ b/julia/test/unittest/visualize.jl @@ -17,7 +17,7 @@ module TestVisualize using MXNet -using Base.Test +using Test using ..Main: mlp2 @@ -26,7 +26,7 @@ using ..Main: mlp2 ################################################################################ function test_basic() - info("Visualize::basic") + @info("Visualize::basic") mlp = mlp2() diff --git a/make/config.mk b/make/config.mk index d10d5d8ee..81d86e3ca 100644 --- a/make/config.mk +++ b/make/config.mk @@ -99,8 +99,10 @@ USE_LIBJPEG_TURBO_PATH = NONE # use openmp for parallelization USE_OPENMP = 1 -# whether use MKL-DNN library -USE_MKLDNN = 0 +# whether use MKL-DNN library: 0 = disabled, 1 = enabled +# if USE_MKLDNN is not defined, MKL-DNN will be enabled by default on x86 Linux. +# you can disable it explicity with USE_MKLDNN = 0 +USE_MKLDNN = # The following variables influence if/how MXnet is built with nGraph support: # diff --git a/make/config/libmxnet.sym b/make/config/libmxnet.sym new file mode 100644 index 000000000..0ddf63fca --- /dev/null +++ b/make/config/libmxnet.sym @@ -0,0 +1,15 @@ +MX* +NN* +_MX* +_NN* +mx* +nn* +_mx* +_nn* +Java_org_apache_mxnet* +*NDArray* +*Engine*Get* +*Storage*Get* +*on_enter_api* +*on_exit_api* +*MXAPISetLastError* diff --git a/make/config/libmxnet.ver b/make/config/libmxnet.ver new file mode 100644 index 000000000..560549c29 --- /dev/null +++ b/make/config/libmxnet.ver @@ -0,0 +1,19 @@ +{ + global: + NN*; + MX*; + _NN*; + _MX*; + nn*; + mx*; + _nn*; + _mx*; + Java_org_apache_mxnet*; + *NDArray*; + *Engine*Get*; + *Storage*Get*; + *on_enter_api*; + *on_exit_api*; + *MXAPISetLastError*; + local: *; +}; diff --git a/make/crosscompile.jetson.mk b/make/crosscompile.jetson.mk index a1468f449..171f846d2 100644 --- a/make/crosscompile.jetson.mk +++ b/make/crosscompile.jetson.mk @@ -57,10 +57,10 @@ DEBUG = 0 USE_SIGNAL_HANDLER = 1 # the additional link flags you want to add -ADD_LDFLAGS = -L${CROSS_ROOT}/lib +ADD_LDFLAGS = -L${CROSS_ROOT}/lib -L/usr/lib/aarch64-linux-gnu/ # the additional compile flags you want to add -ADD_CFLAGS = -I${CROSS_ROOT}/include +ADD_CFLAGS = -I${CROSS_ROOT}/include -I/usr/include/aarch64-linux-gnu/ #--------------------------------------------- # matrix computation libraries for CPU/GPU diff --git a/make/maven/maven_darwin_cpu.mk b/make/maven/maven_darwin_cpu.mk new file mode 100644 index 000000000..b8bd97fbc --- /dev/null +++ b/make/maven/maven_darwin_cpu.mk @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making maven package +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 0 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -lz -framework CoreFoundation -framework Security -Wl,-exported_symbols_list,$(CURDIR)/make/config/libmxnet.sym,-rpath,'$${ORIGIN}',-dead_strip + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=apple + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 0 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# whether use MKL-DNN library +USE_MKLDNN = 0 + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/maven/maven_linux_cpu.mk b/make/maven/maven_linux_cpu.mk new file mode 100644 index 000000000..07177fbe3 --- /dev/null +++ b/make/maven/maven_linux_cpu.mk @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making maven package +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 0 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -ljpeg -lz -lgfortran -ldl -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/maven/maven_linux_cu90.mk b/make/maven/maven_linux_cu90.mk new file mode 100644 index 000000000..097d252f0 --- /dev/null +++ b/make/maven/maven_linux_cu90.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making maven package +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 0 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_darwin_cpu.mk b/make/pip/pip_darwin_cpu.mk new file mode 100644 index 000000000..bf240ad12 --- /dev/null +++ b/make/pip/pip_darwin_cpu.mk @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -lz -framework CoreFoundation -framework Security -Wl,-exported_symbols_list,$(CURDIR)/make/config/libmxnet.sym,-rpath,'$${ORIGIN}',-dead_strip + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=apple + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 0 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# whether use MKL-DNN library +USE_MKLDNN = 0 + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_darwin_mkl.mk b/make/pip/pip_darwin_mkl.mk new file mode 100644 index 000000000..30b328c64 --- /dev/null +++ b/make/pip/pip_darwin_mkl.mk @@ -0,0 +1,166 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -lz -framework CoreFoundation -framework Security -Wl,-exported_symbols_list,$(CURDIR)/make/config/libmxnet.sym,-rpath,'$${ORIGIN}',-dead_strip + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=apple + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 0 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cpu.mk b/make/pip/pip_linux_cpu.mk new file mode 100644 index 000000000..22bb32b4d --- /dev/null +++ b/make/pip/pip_linux_cpu.mk @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -ljpeg -lz -lgfortran -ldl -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu100.mk b/make/pip/pip_linux_cu100.mk new file mode 100644 index 000000000..514603b94 --- /dev/null +++ b/make/pip/pip_linux_cu100.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-10.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu100mkl.mk b/make/pip/pip_linux_cu100mkl.mk new file mode 100644 index 000000000..da4127f0d --- /dev/null +++ b/make/pip/pip_linux_cu100mkl.mk @@ -0,0 +1,169 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-10.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu75.mk b/make/pip/pip_linux_cu75.mk new file mode 100644 index 000000000..02468aa96 --- /dev/null +++ b/make/pip/pip_linux_cu75.mk @@ -0,0 +1,182 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-7.5 + +# whether use CuDNN R3 library +USE_CUDNN = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu75mkl.mk b/make/pip/pip_linux_cu75mkl.mk new file mode 100644 index 000000000..6895383d4 --- /dev/null +++ b/make/pip/pip_linux_cu75mkl.mk @@ -0,0 +1,166 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-7.5 + +# whether use CuDNN R3 library +USE_CUDNN = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu80.mk b/make/pip/pip_linux_cu80.mk new file mode 100644 index 000000000..2a2b88de7 --- /dev/null +++ b/make/pip/pip_linux_cu80.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-8.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu80mkl.mk b/make/pip/pip_linux_cu80mkl.mk new file mode 100644 index 000000000..40cef4ed2 --- /dev/null +++ b/make/pip/pip_linux_cu80mkl.mk @@ -0,0 +1,169 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-8.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu90.mk b/make/pip/pip_linux_cu90.mk new file mode 100644 index 000000000..88e55180b --- /dev/null +++ b/make/pip/pip_linux_cu90.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu90mkl.mk b/make/pip/pip_linux_cu90mkl.mk new file mode 100644 index 000000000..9c4adf07b --- /dev/null +++ b/make/pip/pip_linux_cu90mkl.mk @@ -0,0 +1,169 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.0 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu91.mk b/make/pip/pip_linux_cu91.mk new file mode 100644 index 000000000..ea6251039 --- /dev/null +++ b/make/pip/pip_linux_cu91.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.1 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu91mkl.mk b/make/pip/pip_linux_cu91mkl.mk new file mode 100644 index 000000000..232ec4797 --- /dev/null +++ b/make/pip/pip_linux_cu91mkl.mk @@ -0,0 +1,169 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.1 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu92.mk b/make/pip/pip_linux_cu92.mk new file mode 100644 index 000000000..01d02cfef --- /dev/null +++ b/make/pip/pip_linux_cu92.mk @@ -0,0 +1,185 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.2 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 0 + + +# MKL ML Library for Intel CPU/Xeon Phi +# Please refer to MKL_README.md for details + +# MKL ML Library folder, need to be root for /usr/local +# Change to User Home directory for standard user +# For USE_BLAS!=mkl only +MKLML_ROOT=/usr/local + +# whether use MKL2017 library +USE_MKL2017 = 0 + +# whether use MKL2017 experimental feature for high performance +# Prerequisite USE_MKL2017=1 +USE_MKL2017_EXPERIMENTAL = 0 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_cu92mkl.mk b/make/pip/pip_linux_cu92mkl.mk new file mode 100644 index 000000000..1abde3d71 --- /dev/null +++ b/make/pip/pip_linux_cu92mkl.mk @@ -0,0 +1,169 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib $(DEPS_PATH)/lib/libculibos.a -lpng -ltiff -ljpeg -lz -ldl -lgfortran -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 1 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = $(DEPS_PATH)/usr/local/cuda-9.2 + +# whether to use CuDNN library +USE_CUDNN = 1 + +# whether to use NCCL library +USE_NCCL = 1 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 1 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/make/pip/pip_linux_mkl.mk b/make/pip/pip_linux_mkl.mk new file mode 100644 index 000000000..96b88850c --- /dev/null +++ b/make/pip/pip_linux_mkl.mk @@ -0,0 +1,166 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +#------------------------------------------------------------------------------- +# Template configuration for compiling mxnet for making python wheel +#------------------------------------------------------------------------------- + +#--------------------- +# choice of compiler +#-------------------- + +export CC = gcc +export CXX = g++ +export NVCC = nvcc + +# whether compile with options for MXNet developer +DEV = 0 + +# whether compile with debug +DEBUG = 0 + +# whether compiler with profiler +USE_PROFILER = 1 + +# whether to turn on signal handler (e.g. segfault logger) +USE_SIGNAL_HANDLER = 1 + +# the additional link flags you want to add +ADD_LDFLAGS += -L$(DEPS_PATH)/lib -lpng -ltiff -ljpeg -lz -lgfortran -ldl -Wl,--version-script=$(CURDIR)/make/config/libmxnet.ver,-rpath,'$${ORIGIN}',--gc-sections + +# the additional compile flags you want to add +ADD_CFLAGS += -I$(DEPS_PATH)/include -ffunction-sections -fdata-sections + +#--------------------------------------------- +# matrix computation libraries for CPU/GPU +#--------------------------------------------- + +# choose the version of blas you want to use +# can be: mkl, blas, atlas, openblas +# in default use atlas for linux while apple for osx +USE_BLAS=openblas + +# whether use opencv during compilation +# you can disable it, however, you will not able to use +# imbin iterator +USE_OPENCV = 1 + +# whether use CUDA during compile +USE_CUDA = 0 + +# add the path to CUDA library to link and compile flag +# if you have already add them to environment variable, leave it as NONE +# USE_CUDA_PATH = /usr/local/cuda +USE_CUDA_PATH = NONE + +# whether use CuDNN R3 library +USE_CUDNN = 0 + +# CUDA architecture setting: going with all of them. +# For CUDA < 6.0, comment the *_50 lines for compatibility. +# CUDA_ARCH := + +# whether use cuda runtime compiling for writing kernels in native language (i.e. Python) +USE_NVRTC = 0 + +# use openmp for parallelization +USE_OPENMP = 1 +USE_OPERATOR_TUNING = 1 +USE_LIBJPEG_TURBO = 1 + +# whether use MKL-DNN library +USE_MKLDNN = 1 + +# whether use NNPACK library +USE_NNPACK = 0 + +# whether use lapack during compilation +# only effective when compiled with blas versions openblas/apple/atlas/mkl +USE_LAPACK = 1 + +# path to lapack library in case of a non-standard installation +USE_LAPACK_PATH = $(DEPS_PATH)/lib + +# add path to intel library, you may need it for MKL, if you did not add the path +# to environment variable +USE_INTEL_PATH = NONE + +# If use MKL, choose static link automatically to allow python wrapper +ifeq ($(USE_BLAS), mkl) +USE_STATIC_MKL = 1 +else +USE_STATIC_MKL = NONE +endif + +#---------------------------- +# Settings for power and arm arch +#---------------------------- +ARCH := $(shell uname -a) +ifneq (,$(filter $(ARCH), armv6l armv7l powerpc64le ppc64le aarch64)) + USE_SSE=0 +else + USE_SSE=1 +endif + +#---------------------------- +# distributed computing +#---------------------------- + +# whether or not to enable multi-machine supporting +USE_DIST_KVSTORE = 1 + +# whether or not allow to read and write HDFS directly. If yes, then hadoop is +# required +USE_HDFS = 0 + +# path to libjvm.so. required if USE_HDFS=1 +LIBJVM=$(JAVA_HOME)/jre/lib/amd64/server + +# whether or not allow to read and write AWS S3 directly. If yes, then +# libcurl4-openssl-dev is required, it can be installed on Ubuntu by +# sudo apt-get install -y libcurl4-openssl-dev +USE_S3 = 1 + +#---------------------------- +# additional operators +#---------------------------- + +# path to folders containing projects specific operators that you don't want to put in src/operators +EXTRA_OPERATORS = + + +#---------------------------- +# plugins +#---------------------------- + +# whether to use caffe integration. This requires installing caffe. +# You also need to add CAFFE_PATH/build/lib to your LD_LIBRARY_PATH +# CAFFE_PATH = $(HOME)/caffe +# MXNET_PLUGINS += plugin/caffe/caffe.mk + +# whether to use torch integration. This requires installing torch. +# You also need to add TORCH_PATH/install/lib to your LD_LIBRARY_PATH +# TORCH_PATH = $(HOME)/torch +# MXNET_PLUGINS += plugin/torch/torch.mk + +# WARPCTC_PATH = $(HOME)/warp-ctc +# MXNET_PLUGINS += plugin/warpctc/warpctc.mk + +# whether to use sframe integration. This requires build sframe +# git@github.com:dato-code/SFrame.git +# SFRAME_PATH = $(HOME)/SFrame +# MXNET_PLUGINS += plugin/sframe/plugin.mk diff --git a/perl-package/AI-MXNet-Gluon-Contrib/META.yml b/perl-package/AI-MXNet-Gluon-Contrib/META.yml index d175c2bd1..f56b10c93 100644 --- a/perl-package/AI-MXNet-Gluon-Contrib/META.yml +++ b/perl-package/AI-MXNet-Gluon-Contrib/META.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + --- abstract: 'Perl interface to MXNet Gluon Contrib' author: diff --git a/perl-package/AI-MXNet-Gluon-ModelZoo/META.yml b/perl-package/AI-MXNet-Gluon-ModelZoo/META.yml index d6d9652a6..35c93845d 100644 --- a/perl-package/AI-MXNet-Gluon-ModelZoo/META.yml +++ b/perl-package/AI-MXNet-Gluon-ModelZoo/META.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + --- abstract: 'Perl interface to MXNet Gluon ModelZoo' author: diff --git a/perl-package/AI-MXNet/META.yml b/perl-package/AI-MXNet/META.yml index e604b7cd0..26e37b572 100644 --- a/perl-package/AI-MXNet/META.yml +++ b/perl-package/AI-MXNet/META.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + --- abstract: 'Perl interface to MXNet machine learning library' author: diff --git a/perl-package/AI-MXNetCAPI/META.yml b/perl-package/AI-MXNetCAPI/META.yml index eb5d9aae8..d870f05fb 100644 --- a/perl-package/AI-MXNetCAPI/META.yml +++ b/perl-package/AI-MXNetCAPI/META.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + --- abstract: 'Swig interface to mxnet c api' author: diff --git a/perl-package/AI-MXNetCAPI/mxnet.i b/perl-package/AI-MXNetCAPI/mxnet.i index 38665748a..b1907f5cd 100644 --- a/perl-package/AI-MXNetCAPI/mxnet.i +++ b/perl-package/AI-MXNetCAPI/mxnet.i @@ -344,6 +344,7 @@ int MXGetGPUCount(int* out); /*! * \brief get the free and total available memory on a GPU + * Note: deprecated, use MXGetGPUMemoryInformation64(). * \param dev the GPU number to query * \param free_mem pointer to the integer holding free GPU memory * \param total_mem pointer to the integer holding total GPU memory @@ -351,6 +352,15 @@ int MXGetGPUCount(int* out); */ int MXGetGPUMemoryInformation(int dev, int *out, int *out); +/*! + * \brief get the free and total available memory on a GPU + * \param dev the GPU number to query + * \param free_mem pointer to the uint64_t holding free GPU memory + * \param total_mem pointer to the uint64_t holding total GPU memory + * \return 0 when success, -1 when failure happens + */ +int MXGetGPUMemoryInformation64(int dev, uint64_t *out, uint64_t *out); + //------------------------------------- // Part 1: NDArray creation and deletion diff --git a/perl-package/AI-NNVMCAPI/META.yml b/perl-package/AI-NNVMCAPI/META.yml index e462637ea..40c49e431 100644 --- a/perl-package/AI-NNVMCAPI/META.yml +++ b/perl-package/AI-NNVMCAPI/META.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + --- abstract: 'Swig interface to nnvm c api' author: diff --git a/python/mxnet/__init__.py b/python/mxnet/__init__.py index e960829e6..374a3b50b 100644 --- a/python/mxnet/__init__.py +++ b/python/mxnet/__init__.py @@ -54,7 +54,6 @@ from . import lr_scheduler # use mx.kv as short for kvstore from . import kvstore as kv -from . import kvstore_server # Runtime compile module from . import rtc # Attribute scope to add attributes to symbolic graphs @@ -82,3 +81,11 @@ from . import gluon __version__ = base.__version__ + +# Dist kvstore module which launches a separate process when role is set to "server". +# This should be done after other modules are initialized. +# Otherwise this may result in errors when unpickling custom LR scheduler/optimizers. +# For example, the LRScheduler in gluoncv depends on a specific version of MXNet, and +# checks the __version__ attr of MXNet, which is not set on kvstore server due to the +# fact that kvstore-server module is imported before the __version__ attr is set. +from . import kvstore_server diff --git a/python/mxnet/autograd.py b/python/mxnet/autograd.py index b3acee27c..afc67e256 100644 --- a/python/mxnet/autograd.py +++ b/python/mxnet/autograd.py @@ -273,8 +273,10 @@ def grad(heads, variables, head_grads=None, retain_graph=None, create_graph=Fals returned as new NDArrays instead of stored into `variable.grad`. Supports recording gradient graph for computing higher order gradients. - .. Note: Currently only a very limited set of operators support higher order - gradients. + .. note:: + + Currently only a very limited set of operators support higher order \ + gradients. Parameters ---------- diff --git a/python/mxnet/base.py b/python/mxnet/base.py index 7276c5234..feb4d70b6 100644 --- a/python/mxnet/base.py +++ b/python/mxnet/base.py @@ -560,7 +560,7 @@ def _as_list(obj): return [obj] -_OP_NAME_PREFIX_LIST = ['_contrib_', '_linalg_', '_sparse_', '_image_'] +_OP_NAME_PREFIX_LIST = ['_contrib_', '_linalg_', '_sparse_', '_image_', '_random_'] def _get_op_name_prefix(op_name): @@ -616,9 +616,13 @@ def _init_op_module(root_namespace, module_name, make_op_func): op_name_prefix = _get_op_name_prefix(name) module_name_local = module_name if len(op_name_prefix) > 0: - func_name = name[len(op_name_prefix):] - cur_module = submodule_dict[op_name_prefix] - module_name_local = "%s.%s.%s" % (root_namespace, module_name, op_name_prefix[1:-1]) + if op_name_prefix != '_random_' or name.endswith('_like'): + func_name = name[len(op_name_prefix):] + cur_module = submodule_dict[op_name_prefix] + module_name_local = "%s.%s.%s" % (root_namespace, module_name, op_name_prefix[1:-1]) + else: + func_name = name + cur_module = module_internal elif name.startswith('_'): func_name = name cur_module = module_internal diff --git a/python/mxnet/callback.py b/python/mxnet/callback.py index e1c171444..bfec794f6 100644 --- a/python/mxnet/callback.py +++ b/python/mxnet/callback.py @@ -113,7 +113,7 @@ def _callback(param): logging.info('Iter[%d] Batch[%d] Train-%s=%f', param.epoch, param.nbatch, name, value) if auto_reset: - param.eval_metric.reset() + param.eval_metric.reset_local() return _callback @@ -164,7 +164,7 @@ def __call__(self, param): if param.eval_metric is not None: name_value = param.eval_metric.get_name_value() if self.auto_reset: - param.eval_metric.reset() + param.eval_metric.reset_local() msg = 'Epoch[%d] Batch [%d-%d]\tSpeed: %.2f samples/sec' msg += '\t%s=%f'*len(name_value) logging.info(msg, param.epoch, count-self.frequent, count, speed, *sum(name_value, ())) diff --git a/python/mxnet/context.py b/python/mxnet/context.py index 075132bec..210c21164 100644 --- a/python/mxnet/context.py +++ b/python/mxnet/context.py @@ -289,6 +289,30 @@ def num_gpus(): check_call(_LIB.MXGetGPUCount(ctypes.byref(count))) return count.value +def gpu_memory_info(device_id=0): + """Query CUDA for the free and total bytes of GPU global memory. + + Parameters + ---------- + device_id : int, optional + The device id of the GPU device. + + Raises + ------ + Will raise an exception on any CUDA error. + + Returns + ------- + (free, total) : (int, int) + The number of GPUs. + + """ + free = ctypes.c_uint64() + total = ctypes.c_uint64() + dev_id = ctypes.c_int(device_id) + check_call(_LIB.MXGetGPUMemoryInformation64(dev_id, ctypes.byref(free), ctypes.byref(total))) + return (free.value, total.value) + def current_context(): """Returns the current context. diff --git a/python/mxnet/contrib/io.py b/python/mxnet/contrib/io.py index 6020b3ee1..3364fcd21 100644 --- a/python/mxnet/contrib/io.py +++ b/python/mxnet/contrib/io.py @@ -37,8 +37,8 @@ class DataLoaderIter(DataIter): dtype : str, optional The dtype specifier, can be float32 or float16 - Example usage: - ---------- + Examples + -------- >>> import mxnet as mx >>> from mxnet.gluon.data.vision import MNIST >>> from mxnet.gluon.data import DataLoader diff --git a/python/mxnet/contrib/onnx/mx2onnx/_op_translations.py b/python/mxnet/contrib/onnx/mx2onnx/_op_translations.py index 7cf856c76..51deb4fca 100644 --- a/python/mxnet/contrib/onnx/mx2onnx/_op_translations.py +++ b/python/mxnet/contrib/onnx/mx2onnx/_op_translations.py @@ -60,21 +60,16 @@ import logging import numpy as np from .export_onnx import MXNetGraph as mx_op - -def import_onnx_modules(): - """ To make sure ONNX is runtime dependency, it is imported used only when needed""" - try: - import onnx - except ImportError: - raise ImportError("Onnx and protobuf need to be installed. " - + "Instructions to install - https://github.com/onnx/onnx") - return onnx +try: + import onnx +except ImportError: + onnx = None def parse_helper(attrs, attrs_name, alt_value=None): """Helper function to parse operator attributes in required format.""" tuple_re = re.compile('\([0-9L|,| ]+\)') - if attrs is None: + if not attrs: return alt_value attrs_str = None if attrs.get(attrs_name) is None else str(attrs.get(attrs_name)) if attrs_str is None: @@ -135,12 +130,39 @@ def get_boolean_attribute_value(attrs, attr_name): """ return 1 if attrs.get(attr_name, 0) in ["True", "1"] else 0 +def get_inputs(node, kwargs): + """Helper function to get inputs""" + name = node["name"] + proc_nodes = kwargs["proc_nodes"] + index_lookup = kwargs["index_lookup"] + inputs = node["inputs"] + attrs = node.get("attrs", {}) + + input_nodes = [] + for ip in inputs: + input_node_id = index_lookup[ip[0]] + input_nodes.append(proc_nodes[input_node_id].name) + + return name, input_nodes, attrs + +def create_basic_op_node(op_name, node, kwargs): + """Helper function to create a basic operator + node that doesn't contain op specific attrs""" + name, input_nodes, _ = get_inputs(node, kwargs) + + node = onnx.helper.make_node( + op_name, + input_nodes, + [name], + name=name + ) + return [node] + @mx_op.register("null") def convert_weights_and_inputs(node, **kwargs): """Helper function to convert weights and inputs. """ - onnx = import_onnx_modules() - name = node["name"] + name, _, _ = get_inputs(node, kwargs) if kwargs["is_input"] is False: weights = kwargs["weights"] @@ -172,20 +194,7 @@ def convert_convolution(node, **kwargs): """Map MXNet's convolution operator attributes to onnx's Conv operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - - num_inputs = len(inputs) - - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[kwargs["index_lookup"][inputs[0][0]]].name - weights_node = proc_nodes[kwargs["index_lookup"][inputs[1][0]]].name - - if num_inputs > 2: - bias_node = proc_nodes[kwargs["index_lookup"][inputs[2][0]]].name - - attrs = node.get("attrs") + name, input_nodes, attrs = get_inputs(node, kwargs) kernel_dims = list(parse_helper(attrs, "kernel")) stride_dims = list(parse_helper(attrs, "stride", [1, 1])) @@ -195,10 +204,6 @@ def convert_convolution(node, **kwargs): pad_dims = pad_dims + pad_dims - input_nodes = [input_node, weights_node] - if num_inputs > 2: - input_nodes.append(bias_node) - conv_node = onnx.helper.make_node( "Conv", inputs=input_nodes, @@ -219,32 +224,26 @@ def convert_fully_connected(node, **kwargs): """Map MXNet's FullyConnected operator attributes to onnx's Gemm operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) + initializer = kwargs["initializer"] no_bias = get_boolean_attribute_value(attrs, "no_bias") - input_node_id = kwargs["index_lookup"][inputs[0][0]] - weight_node_id = kwargs["index_lookup"][inputs[1][0]] - - proc_nodes = kwargs["proc_nodes"] - - input_node = proc_nodes[input_node_id] - input_name = input_node.name + fcnode = [] - weights_node = proc_nodes[weight_node_id] - weights_name = weights_node.name + op_name = "flatten_" + str(kwargs["idx"]) + flatten_node = onnx.helper.make_node( + 'Flatten', + inputs=[input_nodes[0]], + outputs=[op_name], + name=op_name + ) - fcnode = [] + input_nodes[0] = op_name + fcnode.append(flatten_node) - if no_bias == 0: - bias_node_id = kwargs["index_lookup"][inputs[2][0]] - bias_node = proc_nodes[bias_node_id] - bias_name = bias_node.name - else: + if no_bias: data_type = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype('int64')] bias_name = "bias" + str(kwargs["idx"]) tensor_node = onnx.helper.make_tensor_value_info(bias_name, data_type, (1,)) @@ -257,11 +256,12 @@ def convert_fully_connected(node, **kwargs): raw=False, ) ) + input_nodes.append(bias_name) fcnode.append(tensor_node) node = onnx.helper.make_node( "Gemm", - [input_name, weights_name, bias_name], # input (A, B, C) - C can be in place + input_nodes, # input (A, B, C) - C can be in place [name], # output alpha=1.0, beta=1.0, @@ -280,37 +280,14 @@ def convert_batchnorm(node, **kwargs): """Map MXNet's BatchNorm operator attributes to onnx's BatchNormalization operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + name, input_nodes, attrs = get_inputs(node, kwargs) - attrs = node["attrs"] - momentum = float(node.get("attrs", {}).get("momentum", 0.9)) + momentum = float(attrs.get("momentum", 0.9)) eps = float(attrs.get("eps", 0.001)) - data_idx = kwargs["index_lookup"][inputs[0][0]] - gamma_idx = kwargs["index_lookup"][inputs[1][0]] - beta_idx = kwargs["index_lookup"][inputs[2][0]] - moving_mean_idx = kwargs["index_lookup"][inputs[3][0]] - moving_var_idx = kwargs["index_lookup"][inputs[4][0]] - - data_node = proc_nodes[data_idx].name - gamma_node = proc_nodes[gamma_idx].name - beta_node = proc_nodes[beta_idx].name - - mov_mean_node = proc_nodes[moving_mean_idx] - mov_mean_node = mov_mean_node.name - mov_var_node = proc_nodes[moving_var_idx].name - bn_node = onnx.helper.make_node( "BatchNormalization", - [data_node, - gamma_node, # scale - beta_node, # bias - mov_mean_node, - mov_var_node - ], + input_nodes, [name], name=name, epsilon=eps, @@ -327,140 +304,49 @@ def convert_tanh(node, **kwargs): """Map MXNet's tanh operator attributes to onnx's Tanh operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Tanh', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Tanh', node, kwargs) @mx_op.register("cos") def convert_cos(node, **kwargs): """Map MXNet's cos operator attributes to onnx's Cos operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Cos', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Cos', node, kwargs) @mx_op.register("sin") def convert_sin(node, **kwargs): """Map MXNet's sin operator attributes to onnx's Sin operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Sin', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Sin', node, kwargs) @mx_op.register("tan") def convert_tan(node, **kwargs): """Map MXNet's tan operator attributes to onnx's tan operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Tan', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Tan', node, kwargs) @mx_op.register("arccos") def convert_acos(node, **kwargs): """Map MXNet's acos operator attributes to onnx's acos operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Acos', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Acos', node, kwargs) @mx_op.register("arcsin") def convert_asin(node, **kwargs): """Map MXNet's asin operator attributes to onnx's asin operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Asin', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Asin', node, kwargs) @mx_op.register("arctan") def convert_atan(node, **kwargs): """Map MXNet's atan operator attributes to onnx's atan operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Atan', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Atan', node, kwargs) #Basic neural network functions @mx_op.register("sigmoid") @@ -468,58 +354,24 @@ def convert_sigmoid(node, **kwargs): """Map MXNet's sigmoid operator attributes to onnx's Sigmoid operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Sigmoid', - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Sigmoid', node, kwargs) @mx_op.register("relu") def convert_relu(node, **kwargs): """Map MXNet's relu operator attributes to onnx's Relu operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_node_idx].name - - node = onnx.helper.make_node( - 'Relu', - [input_node], - [name], - name=name - ) - - return [node] + return create_basic_op_node('Relu', node, kwargs) @mx_op.register("Activation") def convert_activation(node, **kwargs): """Map MXNet's Activation operator attributes to onnx's Tanh/Relu operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] + name, input_nodes, attrs = get_inputs(node, kwargs) - proc_nodes = kwargs["proc_nodes"] - attrs = node["attrs"] act_type = attrs["act_type"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_idx].output[0] - # Creating a dictionary here, but if this titlecase pattern # mxnet_name.title() act_types = { @@ -534,7 +386,7 @@ def convert_activation(node, **kwargs): if act_name: node = onnx.helper.make_node( act_name, - [input_node], + input_nodes, [name], name=name ) @@ -551,13 +403,7 @@ def convert_pad(node, **kwargs): """Map MXNet's pad operator attributes to onnx's Pad operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - attrs = node["attrs"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_idx].name + name, input_nodes, attrs = get_inputs(node, kwargs) mxnet_pad_width = convert_string_to_list(attrs.get("pad_width")) onnx_pad_width = transform_padding(mxnet_pad_width) @@ -569,7 +415,7 @@ def convert_pad(node, **kwargs): if "constant_value" in attrs else 0.0 node = onnx.helper.make_node( 'Pad', - inputs=[input_node], + inputs=input_nodes, outputs=[name], mode='constant', value=pad_value, @@ -579,7 +425,7 @@ def convert_pad(node, **kwargs): else: node = onnx.helper.make_node( 'Pad', - inputs=[input_node], + inputs=input_nodes, outputs=[name], mode=pad_mode, pads=onnx_pad_width, @@ -591,8 +437,6 @@ def convert_pad(node, **kwargs): def create_helper_trans_node(op_name, input_node, node_name): """create extra transpose node for dot operator""" - onnx = import_onnx_modules() - node_name = op_name + "_" + node_name trans_node = onnx.helper.make_node( 'Transpose', @@ -608,17 +452,10 @@ def convert_dot(node, **kwargs): """Map MXNet's dot operator attributes to onnx's MatMul and Transpose operators based on the values set for transpose_a, transpose_b attributes.""" - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - name = node["name"] + name, input_nodes, attrs = get_inputs(node, kwargs) + input_node_a = input_nodes[0] + input_node_b = input_nodes[1] - input_a_idx = kwargs["index_lookup"][node_inputs[0][0]] - input_node_a = proc_nodes[input_a_idx].name - input_b_idx = kwargs["index_lookup"][node_inputs[1][0]] - input_node_b = proc_nodes[input_b_idx].name - - attrs = node.get('attrs', {}) trans_a_node = None trans_b_node = None @@ -626,14 +463,12 @@ def convert_dot(node, **kwargs): trans_b = get_boolean_attribute_value(attrs, "transpose_b") op_name = "transpose" + str(kwargs["idx"]) - create_helper_trans_node(op_name, input_node_a, 'a') - create_helper_trans_node(op_name, input_node_b, 'b') if trans_a: - trans_a_node = create_helper_trans_node(op_name, input_node_a, 'a') + trans_a_node = create_helper_trans_node(op_name, input_nodes[0], 'a') input_node_a = op_name+"_a" if trans_b: - trans_b_node = create_helper_trans_node(op_name, input_node_b, 'b') + trans_b_node = create_helper_trans_node(op_name, input_nodes[1], 'b') input_node_b = op_name+"_b" matmul_node = onnx.helper.make_node( @@ -660,33 +495,19 @@ def convert_linalg_gemm2(node, **kwargs): transpose_a, transpose_b attributes. Return multiple nodes created. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - name = node["name"] - - input_a_idx = kwargs["index_lookup"][node_inputs[0][0]] - input_node_a = proc_nodes[input_a_idx].name - input_b_idx = kwargs["index_lookup"][node_inputs[1][0]] - input_node_b = proc_nodes[input_b_idx].name + name, input_nodes, attrs = get_inputs(node, kwargs) # Getting the attributes and assigning default values. - if "attrs" in node: - attrs = node["attrs"] - alpha = float(attrs["alpha"]) - trans_a = int(attrs["transpose_a"]) - trans_b = int(attrs["transpose_b"]) - else: - alpha = 1.0 - trans_a = 0 - trans_b = 0 + alpha = float(attrs.get("alpha", 1.0)) + trans_a = get_boolean_attribute_value(attrs, "transpose_a") + trans_b = get_boolean_attribute_value(attrs, "transpose_b") op_name = "transpose" + str(kwargs["idx"]) if alpha == 1.0 and trans_a == 0 and trans_b == 0: matmul_node = onnx.helper.make_node( 'MatMul', - inputs=[input_node_a, input_node_b], + inputs=input_nodes, outputs=[name], name=name ) @@ -696,14 +517,14 @@ def convert_linalg_gemm2(node, **kwargs): node_name = op_name+"_a" trans_a_node = onnx.helper.make_node( 'Transpose', - inputs=[input_node_a], + inputs=[input_nodes[0]], outputs=[op_name+"_a"], name=node_name ) matmul_node = onnx.helper.make_node( 'MatMul', - inputs=[node_name, input_node_b], + inputs=[node_name, input_nodes[1]], outputs=[name], name=name ) @@ -713,14 +534,14 @@ def convert_linalg_gemm2(node, **kwargs): node_name = op_name + "_b" trans_b_node = onnx.helper.make_node( 'Transpose', - inputs=[input_node_b], + inputs=[input_nodes[1]], outputs=[op_name+"_b"], name=node_name ) matmul_node = onnx.helper.make_node( 'MatMul', - inputs=[input_node_a, node_name], + inputs=[input_nodes[0], node_name], outputs=[name], name=name ) @@ -730,7 +551,7 @@ def convert_linalg_gemm2(node, **kwargs): node_name_a = op_name+"_a" trans_a_node = onnx.helper.make_node( 'Transpose', - inputs=[input_node_a], + inputs=[input_nodes[0]], outputs=[op_name+"_a"], name=node_name_a ) @@ -738,14 +559,14 @@ def convert_linalg_gemm2(node, **kwargs): node_name_b = op_name + "_b" trans_b_node = onnx.helper.make_node( 'Transpose', - inputs=[input_node_b], + inputs=[input_nodes[1]], outputs=[op_name+"_b"], name=node_name_b ) matmul_node = onnx.helper.make_node( 'MatMul', - inputs=[node_name_a, node_name_b], + inputs=input_nodes, outputs=[name], name=name ) @@ -759,18 +580,13 @@ def convert_pooling(node, **kwargs): MaxPool/AveragePool/GlobalMaxPool/GlobalAveragePool operators based on the input node's attributes and return the created node. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) + kernel = eval(attrs["kernel"]) pool_type = attrs["pool_type"] stride = eval(attrs["stride"]) if attrs.get("stride") else None global_pool = get_boolean_attribute_value(attrs, "global_pool") - - node_inputs = node["inputs"] - input_node_idx = kwargs["index_lookup"][node_inputs[0][0]] - input_node = proc_nodes[input_node_idx] - name = node["name"] + p_value = attrs.get('p_value', 'None') pooling_convention = attrs.get('pooling_convention', 'valid') @@ -783,26 +599,51 @@ def convert_pooling(node, **kwargs): pad_dims = list(parse_helper(attrs, "pad", [0, 0])) pad_dims = pad_dims + pad_dims - pool_types = {"max": "MaxPool", "avg": "AveragePool"} - global_pool_types = {"max": "GlobalMaxPool", "avg": "GlobalAveragePool"} + pool_types = {"max": "MaxPool", "avg": "AveragePool", "lp": "LpPool"} + global_pool_types = {"max": "GlobalMaxPool", "avg": "GlobalAveragePool", + "lp": "GlobalLpPool"} + + if pool_type == 'lp' and p_value == 'None': + raise AttributeError('ONNX requires a p value for LpPool and GlobalLpPool') if global_pool: - node = onnx.helper.make_node( - global_pool_types[pool_type], - [input_node.name], # input - [name], - name=name - ) + if pool_type == 'lp': + node = onnx.helper.make_node( + global_pool_types[pool_type], + input_nodes, # input + [name], + p=int(p_value), + name=name + ) + else: + node = onnx.helper.make_node( + global_pool_types[pool_type], + input_nodes, # input + [name], + name=name + ) else: - node = onnx.helper.make_node( - pool_types[pool_type], - [input_node.name], # input - [name], - kernel_shape=kernel, - pads=pad_dims, - strides=stride, - name=name - ) + if pool_type == 'lp': + node = onnx.helper.make_node( + pool_types[pool_type], + input_nodes, # input + [name], + p=int(p_value), + kernel_shape=kernel, + pads=pad_dims, + strides=stride, + name=name + ) + else: + node = onnx.helper.make_node( + pool_types[pool_type], + input_nodes, # input + [name], + kernel_shape=kernel, + pads=pad_dims, + strides=stride, + name=name + ) return [node] @@ -812,57 +653,46 @@ def convert_exp(node, **kwargs): """Map MXNet's exp operator attributes to onnx's Exp operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Exp", - [input_node], - [name], - name=name, - ) - return [node] - + return create_basic_op_node('Exp', node, kwargs) @mx_op.register("_copy") -def convert_identity(node, **kwargs): +def convert_copy(node, **kwargs): """Map MXNet's _copy operator attributes to onnx's Identity operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + return create_basic_op_node('Identity', node, kwargs) + +@mx_op.register("identity") +def convert_identity(node, **kwargs): + """Map MXNet's identity operator attributes to onnx's ConstantFill operator + and return the created node. + """ + return create_basic_op_node('ConstantFill', node, kwargs) + +@mx_op.register("InstanceNorm") +def convert_instancenorm(node, **kwargs): + """Map MXNet's InstanceNorm operator attributes to onnx's InstanceNormalization operator + based on the input node's attributes and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + eps = float(attrs.get("eps", 0.001)) node = onnx.helper.make_node( - "Identity", - [input_node], - [name], + 'InstanceNormalization', + inputs=input_nodes, + outputs=[name], name=name, - ) - return [node] + epsilon=eps) + return [node] @mx_op.register("LeakyReLU") def convert_leakyrelu(node, **kwargs): """Map MXNet's LeakyReLU operator attributes to onnx's Elu/LeakyRelu/PRelu operators based on the input node's attributes and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) act_type = attrs.get("act_type", "leaky") alpha = float(attrs.get("slope", 0.25)) @@ -870,25 +700,16 @@ def convert_leakyrelu(node, **kwargs): act_name = {"elu": "Elu", "leaky": "LeakyRelu", "prelu": "PRelu", "selu": "Selu"} - if act_type == "prelu": - alpha_node_index = kwargs["index_lookup"][inputs[1][0]] - alpha_node_name = proc_nodes[alpha_node_index].name - - node = onnx.helper.make_node( - act_name[act_type], - inputs=[input_node, alpha_node_name], - outputs=[name], - name=name) - elif act_type == "selu": + if act_type == "prelu" or act_type == "selu": node = onnx.helper.make_node( act_name[act_type], - inputs=[input_node], + inputs=input_nodes, outputs=[name], name=name) else: node = onnx.helper.make_node( act_name[act_type], - inputs=[input_node], + inputs=input_nodes, outputs=[name], name=name, alpha=alpha) @@ -901,18 +722,13 @@ def convert_softmax(node, **kwargs): """Map MXNet's softmax operator attributes to onnx's Softmax operator and return the created node. """ - onnx = import_onnx_modules() - inputs = node["inputs"] - input_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_idx] + name, input_nodes, attrs = get_inputs(node, kwargs) - name = node["name"] - axis = int(node.get("attrs", {}).get("axis", -1)) + axis = int(attrs.get("axis", -1)) softmax_node = onnx.helper.make_node( "Softmax", - [input_node.name], + input_nodes, [name], axis=axis, name=name @@ -928,16 +744,14 @@ def convert_softmax_output(node, **kwargs): """Map MXNet's SoftmaxOutput operator attributes to onnx's Softmax operator and return the created node. """ - onnx = import_onnx_modules() - inputs = node["inputs"] - input1_idx = kwargs["index_lookup"][inputs[0][0]] - proc_nodes = kwargs["proc_nodes"] - input1 = proc_nodes[input1_idx] name = node["name"] + input1_idx = kwargs["index_lookup"][node["inputs"][0][0]] + input1 = kwargs["proc_nodes"][input1_idx] + softmax_node = onnx.helper.make_node( "Softmax", - [input1.output[0]], + [input1.name], [name], axis=1, name=name @@ -945,21 +759,43 @@ def convert_softmax_output(node, **kwargs): return [softmax_node] +@mx_op.register("LogisticRegressionOutput") +def convert_logistic_regression_output(node, **kwargs): + """Map MXNet's SoftmaxOutput operator attributes to onnx's Softmax operator + and return the created node. + """ + name = node["name"] + input1_idx = kwargs["index_lookup"][node["inputs"][0][0]] + input1 = kwargs["proc_nodes"][input1_idx] + sigmoid_node = onnx.helper.make_node( + "Sigmoid", + [input1.name], + [name], + name=name + ) + return [sigmoid_node] + +@mx_op.register("BlockGrad") +def convert_blockgrad(node, **kwargs): + """ Skip operator """ + return create_basic_op_node('ConstantFill', node, kwargs) + +@mx_op.register("MakeLoss") +def convert_makeloss(node, **kwargs): + """ Skip operator """ + return create_basic_op_node('ConstantFill', node, kwargs) @mx_op.register("Concat") def convert_concat(node, **kwargs): """Map MXNet's Concat operator attributes to onnx's Concat operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - inputs = node["inputs"] - proc_nodes = kwargs["proc_nodes"] - input_names = [proc_nodes[kwargs["index_lookup"][i[0]]].name for i in inputs] - axis = int(node.get("attrs", {}).get("dim", 1)) + name, input_nodes, attrs = get_inputs(node, kwargs) + + axis = int(attrs.get("dim", 1)) concat_node = onnx.helper.make_node( "Concat", - input_names, + input_nodes, [name], axis=axis, name=name @@ -972,18 +808,15 @@ def convert_transpose(node, **kwargs): """Map MXNet's transpose operator attributes to onnx's Transpose operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_idx = kwargs["index_lookup"][node["inputs"][0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_idx].name - axes = node.get("attrs", {}).get("axes", ()) + name, input_nodes, attrs = get_inputs(node, kwargs) + + axes = attrs.get("axes", ()) if axes: axes = tuple(map(int, re.findall(r'\d+', axes))) transpose_node = onnx.helper.make_node( "Transpose", - [input_node], + input_nodes, [name], perm=axes, name=name @@ -991,7 +824,7 @@ def convert_transpose(node, **kwargs): else: transpose_node = onnx.helper.make_node( "Transpose", - [input_node], + input_nodes, [name], name=name ) @@ -1004,21 +837,16 @@ def convert_lrn(node, **kwargs): """Map MXNet's LRN operator attributes to onnx's LRN operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_idx = kwargs["index_lookup"][node["inputs"][0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_idx].name + name, input_nodes, attrs = get_inputs(node, kwargs) - attrs = node["attrs"] - alpha = float(attrs["alpha"]) if "alpha" in attrs else 0.0001 - beta = float(attrs["beta"]) if "beta" in attrs else 0.75 - bias = float(attrs["knorm"]) if "knorm" in attrs else 1.0 - size = int(attrs["nsize"]) + alpha = float(attrs.get("alpha", 0.0001)) + beta = float(attrs.get("beta", 0.75)) + bias = float(attrs.get("knorm", 1.0)) + size = int(attrs.get("nsize")) lrn_node = onnx.helper.make_node( "LRN", - inputs=[input_node], + inputs=input_nodes, outputs=[name], name=name, alpha=alpha, @@ -1035,19 +863,16 @@ def convert_l2normalization(node, **kwargs): """Map MXNet's L2Normalization operator attributes to onnx's LpNormalization operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_id = kwargs["index_lookup"][node["inputs"][0][0]] - input_name = kwargs["proc_nodes"][input_id].name - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) + mode = attrs.get("mode", "instance") if mode != "channel": - raise AttributeError("ONNX currently supports channel mode only") + raise AttributeError("L2Normalization: ONNX currently supports channel mode only") l2norm_node = onnx.helper.make_node( "LpNormalization", - [input_name], + input_nodes, [name], axis=1, # channel only name=name @@ -1060,16 +885,13 @@ def convert_dropout(node, **kwargs): """Map MXNet's Dropout operator attributes to onnx's Dropout operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_id = kwargs["index_lookup"][node["inputs"][0][0]] - input_name = kwargs["proc_nodes"][input_id].name - attrs = node["attrs"] - probability = float(attrs["p"]) + name, input_nodes, attrs = get_inputs(node, kwargs) + + probability = float(attrs.get("p", 0.5)) dropout_node = onnx.helper.make_node( "Dropout", - [input_name], + input_nodes, [name], ratio=probability, name=name @@ -1082,37 +904,21 @@ def convert_flatten(node, **kwargs): """Map MXNet's Flatten operator attributes to onnx's Flatten operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_idx = kwargs["index_lookup"][node["inputs"][0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_idx].name # .output[0] - - flatten_node = onnx.helper.make_node( - "Flatten", - [input_node], - [name], - name=name - ) - return [flatten_node] + return create_basic_op_node('Flatten', node, kwargs) @mx_op.register("clip") def convert_clip(node, **kwargs): """Map MXNet's Clip operator attributes to onnx's Clip operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - input_idx = kwargs["index_lookup"][node["inputs"][0][0]] - proc_nodes = kwargs["proc_nodes"] - input_node = proc_nodes[input_idx].name - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) + a_min = np.float(attrs.get('a_min', -np.inf)) a_max = np.float(attrs.get('a_max', np.inf)) clip_node = onnx.helper.make_node( "Clip", - [input_node], + input_nodes, [name], name=name, min=a_min, @@ -1123,45 +929,48 @@ def convert_clip(node, **kwargs): def scalar_op_helper(node, op_name, **kwargs): """Helper function for scalar arithmetic operations""" - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - scalar_value = [float(node.get("attrs", {}).get("scalar", 1))] - - input_name_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_name_id].name + name, input_nodes, attrs = get_inputs(node, kwargs) + from onnx import numpy_helper + input_type = kwargs["in_type"] + scalar_value = np.array([attrs.get("scalar", 1)], + dtype=onnx.mapping.TENSOR_TYPE_TO_NP_TYPE[input_type]) initializer = kwargs["initializer"] flag = True # If the input value is in initializer, just multiply with scalar input # and create a new initializer for i in initializer: - if i.name == input_node: + if i.name == input_nodes[0]: if op_name == 'Mul': - new_initializer = onnx.numpy_helper.to_array(i) * scalar_value[0] + new_initializer = numpy_helper.to_array(i) * scalar_value[0] elif op_name == 'Sub': - new_initializer = onnx.numpy_helper.to_array(i) - scalar_value[0] + if name.startswith("_rminusscalar"): + new_initializer = scalar_value[0] - numpy_helper.to_array(i) + else: + new_initializer = numpy_helper.to_array(i) - scalar_value[0] elif op_name == 'Add': - new_initializer = onnx.numpy_helper.to_array(i) + scalar_value[0] + new_initializer = numpy_helper.to_array(i) + scalar_value[0] elif op_name == 'Div': - new_initializer = onnx.numpy_helper.to_array(i) / scalar_value[0] + if name.startswith("_rdivscalar"): + new_initializer = scalar_value[0] / numpy_helper.to_array(i) + else: + new_initializer = numpy_helper.to_array(i) / scalar_value[0] + elif op_name == 'Pow': + new_initializer = numpy_helper.to_array(i) ** scalar_value[0] flag = False break # else create a new tensor of the scalar value, add it in initializer if flag is True: - np_arr = np.array(scalar_value) - data_type = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np_arr.dtype] - dims = np.shape(np_arr) + dims = np.shape(scalar_value) scalar_op_name = "scalar_op" + str(kwargs["idx"]) - tensor_node = onnx.helper.make_tensor_value_info(scalar_op_name, data_type, dims) + tensor_node = onnx.helper.make_tensor_value_info(scalar_op_name, input_type, dims) initializer.append( onnx.helper.make_tensor( name=scalar_op_name, - data_type=data_type, + data_type=input_type, dims=dims, vals=scalar_value, raw=False, @@ -1170,7 +979,7 @@ def scalar_op_helper(node, op_name, **kwargs): mul_node = onnx.helper.make_node( op_name, - [input_node, scalar_op_name], + [input_nodes[0], scalar_op_name], [name], name=name ) @@ -1180,7 +989,7 @@ def scalar_op_helper(node, op_name, **kwargs): data_type = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[new_initializer.dtype] dims = np.shape(new_initializer) - new_a_node = input_node + str(kwargs["idx"]) + new_a_node = input_nodes[0] + str(kwargs["idx"]) tensor_node = onnx.helper.make_tensor_value_info(new_a_node, data_type, dims) initializer.append( @@ -1213,6 +1022,13 @@ def convert_minus_scalar(node, **kwargs): """ return scalar_op_helper(node, 'Sub', **kwargs) +@mx_op.register("_rminus_scalar") +def convert_rminus_scalar(node, **kwargs): + """Map MXNet's _rminus_scalar operator attributes to onnx's Sub operator. + Creates a new node for the input scalar value, adds it to the initializer + and return multiple created nodes. + """ + return scalar_op_helper(node, 'Sub', **kwargs) # Convert scalar value into node and pass it as input to mul_node @mx_op.register("_plus_scalar") @@ -1232,6 +1048,21 @@ def convert_div_scalar(node, **kwargs): """ return scalar_op_helper(node, 'Div', **kwargs) +@mx_op.register("_rdiv_scalar") +def convert_rdiv_scalar(node, **kwargs): + """Map MXNet's _rdiv_scalar operator attributes to onnx's Div operator. + Creates a new node for the input scalar value, adds it to the initializer + and return multiple created nodes. + """ + return scalar_op_helper(node, 'Div', **kwargs) + +@mx_op.register("_power_scalar") +def convert_pow_scalar(node, **kwargs): + """Map MXNet's _pow_scalar operator attributes to onnx's Pow operator. + Creates a new node for the input scalar value, adds it to the initializer + and return multiple created nodes. + """ + return scalar_op_helper(node, 'Pow', **kwargs) # Sorting and Searching @mx_op.register("argmax") @@ -1239,21 +1070,14 @@ def convert_argmax(node, **kwargs): """Map MXNet's argmax operator attributes to onnx's ArgMax operator and return the created node. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - - input_node_idx = kwargs["index_lookup"][node_inputs[0][0]] - input_node = proc_nodes[input_node_idx].name - name = node["name"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) axis = int(attrs.get("axis")) - keepdims = int(attrs.get("keepdims")) if "keepdims" in attrs else 1 + keepdims = get_boolean_attribute_value(attrs, "keepdims") node = onnx.helper.make_node( 'ArgMax', - inputs=[input_node], + inputs=input_nodes, axis=axis, keepdims=keepdims, outputs=[name], @@ -1266,21 +1090,14 @@ def convert_argmin(node, **kwargs): """Map MXNet's argmin operator attributes to onnx's ArgMin operator and return the created node. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - - input_node_idx = kwargs["index_lookup"][node_inputs[0][0]] - input_node = proc_nodes[input_node_idx].name - name = node["name"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) axis = int(attrs.get("axis")) - keepdims = int(attrs.get("keepdims")) if "keepdims" in attrs else 1 + keepdims = get_boolean_attribute_value(attrs, "keepdims") node = onnx.helper.make_node( 'ArgMin', - inputs=[input_node], + inputs=input_nodes, axis=axis, keepdims=keepdims, outputs=[name], @@ -1293,25 +1110,7 @@ def convert_maximum(node, **kwargs): """Map MXNet's _maximum operator attributes to onnx's Max operator and return the created node. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - - input_node_list = [] - for node_input in node_inputs: - node_id = kwargs["index_lookup"][node_input[0]] - input_node_list.append(proc_nodes[node_id].name) - - name = node["name"] - - node = onnx.helper.make_node( - 'Max', - inputs=input_node_list, - outputs=[name], - name=name, - ) - - return [node] + return create_basic_op_node('Max', node, kwargs) @mx_op.register("_minimum") @@ -1319,49 +1118,24 @@ def convert_minimum(node, **kwargs): """Map MXNet's _minimum operator attributes to onnx's Min operator and return the created node. """ - onnx = import_onnx_modules() - proc_nodes = kwargs["proc_nodes"] - node_inputs = node["inputs"] - - input_node_list = [] - for node_input in node_inputs: - node_id = kwargs["index_lookup"][node_input[0]] - input_node_list.append(proc_nodes[node_id].name) - - name = node["name"] - - node = onnx.helper.make_node( - 'Min', - inputs=input_node_list, - outputs=[name], - name=name, - ) - - return [node] - + return create_basic_op_node('Min', node, kwargs) @mx_op.register("min") def convert_min(node, **kwargs): """Map MXNet's min operator attributes to onnx's ReduceMin operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + name, input_nodes, attrs = get_inputs(node, kwargs) - mx_axis = node.get("attrs", {}).get("axis", None) + mx_axis = attrs.get("axis", None) axes = convert_string_to_list(str(mx_axis)) if mx_axis is not None else None - keepdims = int(node.get("attrs", {}).get("keepdims", 0)) - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + keepdims = get_boolean_attribute_value(attrs, "keepdims") if axes is not None: node = onnx.helper.make_node( 'ReduceMin', - inputs=[input_node], + inputs=input_nodes, outputs=[name], axes=axes, keepdims=keepdims, @@ -1372,7 +1146,7 @@ def convert_min(node, **kwargs): else: node = onnx.helper.make_node( 'ReduceMin', - inputs=[input_node], + inputs=input_nodes, outputs=[name], keepdims=keepdims, name=name @@ -1386,23 +1160,17 @@ def convert_max(node, **kwargs): """Map MXNet's max operator attributes to onnx's ReduceMax operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + name, input_nodes, attrs = get_inputs(node, kwargs) - mx_axis = node.get("attrs", {}).get("axis", None) + mx_axis = attrs.get("axis", None) axes = convert_string_to_list(str(mx_axis)) if mx_axis is not None else None - keepdims = int(node.get("attrs", {}).get("keepdims", 0)) - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + keepdims = get_boolean_attribute_value(attrs, "keepdims") if axes is not None: node = onnx.helper.make_node( 'ReduceMax', - inputs=[input_node], + inputs=input_nodes, outputs=[name], axes=axes, keepdims=keepdims, @@ -1413,7 +1181,7 @@ def convert_max(node, **kwargs): else: node = onnx.helper.make_node( 'ReduceMax', - inputs=[input_node], + inputs=input_nodes, outputs=[name], keepdims=keepdims, name=name @@ -1427,23 +1195,17 @@ def convert_mean(node, **kwargs): """Map MXNet's mean operator attributes to onnx's ReduceMean operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + name, input_nodes, attrs = get_inputs(node, kwargs) - mx_axis = node.get("attrs", {}).get("axis", None) + mx_axis = attrs.get("axis", None) axes = convert_string_to_list(str(mx_axis)) if mx_axis is not None else None - keepdims = int(node.get("attrs", {}).get("keepdims", 0)) - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + keepdims = get_boolean_attribute_value(attrs, "keepdims") if axes is not None: node = onnx.helper.make_node( 'ReduceMean', - inputs=[input_node], + inputs=input_nodes, outputs=[name], axes=axes, keepdims=keepdims, @@ -1454,7 +1216,7 @@ def convert_mean(node, **kwargs): else: node = onnx.helper.make_node( 'ReduceMean', - inputs=[input_node], + inputs=input_nodes, outputs=[name], keepdims=keepdims, name=name @@ -1468,23 +1230,17 @@ def convert_prod(node, **kwargs): """Map MXNet's prod operator attributes to onnx's ReduceProd operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] + name, input_nodes, attrs = get_inputs(node, kwargs) - mx_axis = node.get("attrs", {}).get("axis", None) + mx_axis = attrs.get("axis", None) axes = convert_string_to_list(str(mx_axis)) if mx_axis is not None else None - keepdims = int(node.get("attrs", {}).get("keepdims", 0)) - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + keepdims = get_boolean_attribute_value(attrs, "keepdims") if axes is not None: node = onnx.helper.make_node( 'ReduceProd', - inputs=[input_node], + inputs=input_nodes, outputs=[name], axes=axes, keepdims=keepdims, @@ -1495,7 +1251,7 @@ def convert_prod(node, **kwargs): else: node = onnx.helper.make_node( 'ReduceProd', - inputs=[input_node], + inputs=input_nodes, outputs=[name], keepdims=keepdims, name=name @@ -1510,25 +1266,7 @@ def convert_elementwise_add(node, **kwargs): """Map MXNet's elemwise_add operator attributes to onnx's Add operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - add_node = onnx.helper.make_node( - "Add", - [input_node_a, input_node_b], - [name], - name=name, - ) - - return [add_node] + return create_basic_op_node('Add', node, kwargs) @mx_op.register("broadcast_add") @@ -1536,25 +1274,7 @@ def covert_broadcast_add(node, **kwargs): """Map MXNet's broadcast_add operator attributes to onnx's Add operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - add_node = onnx.helper.make_node( - "Add", - [input_node_a, input_node_b], - [name], - name=name, - ) - - return [add_node] + return create_basic_op_node('Add', node, kwargs) @mx_op.register("elemwise_sub") @@ -1562,224 +1282,63 @@ def convert_elementwise_sub(node, **kwargs): """Map MXNet's elemwise_sub operator attributes to onnx's Sub operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - sub_node = onnx.helper.make_node( - "Sub", - [input_node_a, input_node_b], - [name], - name=name, - ) - - return [sub_node] + return create_basic_op_node('Sub', node, kwargs) @mx_op.register("broadcast_sub") def covert_broadcast_sub(node, **kwargs): """Map MXNet's broadcast_sub operator attributes to onnx's Sub operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - sub_node = onnx.helper.make_node( - "Sub", - [input_node_a, input_node_b], - [name], - name=name, - ) - - return [sub_node] - + return create_basic_op_node('Sub', node, kwargs) @mx_op.register("elemwise_mul") def convert_elemwise_mul(node, **kwargs): """Map MXNet's elemwise_mul operator attributes to onnx's Mul operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - mul_node = onnx.helper.make_node( - "Mul", - [input_node_a, input_node_b], - [name], - name=name, - ) - - return [mul_node] + return create_basic_op_node('Mul', node, kwargs) @mx_op.register("broadcast_mul") def convert_broadcast_mul(node, **kwargs): """Map MXNet's broadcast_mul operator attributes to onnx's Mul operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - mul_node = onnx.helper.make_node( - "Mul", - [input_node_a, input_node_b], - [name], - name=name - ) - - return [mul_node] - + return create_basic_op_node('Mul', node, kwargs) @mx_op.register("elemwise_div") def convert_elemwise_div(node, **kwargs): """Map MXNet's elemwise_div operator attributes to onnx's Div operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - div_node = onnx.helper.make_node( - "Div", - [input_node_a, input_node_b], - [name], - name=name - ) - - return [div_node] - + return create_basic_op_node('Div', node, kwargs) @mx_op.register("broadcast_div") def convert_broadcast_div(node, **kwargs): """Map MXNet's broadcast_div operator attributes to onnx's Div operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - div_node = onnx.helper.make_node( - "Div", - [input_node_a, input_node_b], - [name], - name=name - ) - - return [div_node] - + return create_basic_op_node('Div', node, kwargs) @mx_op.register("negative") def convert_negative(node, **kwargs): """Map MXNet's negative operator attributes to onnx's Neg operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - - input_node = proc_nodes[input_node_id].name - - neg_node = onnx.helper.make_node( - "Neg", - [input_node], - [name], - name=name, - ) - - return [neg_node] - + return create_basic_op_node('Neg', node, kwargs) @mx_op.register("abs") def convert_abs(node, **kwargs): """Map MXNet's abs operator attributes to onnx's Abs operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - - input_node = proc_nodes[input_node_id].name - - abs_node = onnx.helper.make_node( - "Abs", - [input_node], - [name], - name=name - ) - - return [abs_node] - + return create_basic_op_node('Abs', node, kwargs) @mx_op.register("add_n") def convert_addn(node, **kwargs): """Map MXNet's add_n operator attributes to onnx's Sum operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_list = [] - for input_val in inputs: - input_list.append(proc_nodes[kwargs["index_lookup"][input_val[0]]].name) - - sum_node = onnx.helper.make_node( - "Sum", - input_list, - [name], - name=name - ) - return [sum_node] + return create_basic_op_node('Sum', node, kwargs) # Rounding @mx_op.register("ceil") @@ -1787,42 +1346,14 @@ def convert_ceil(node, **kwargs): """Map MXNet's ceil operator attributes to onnx's Ceil operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Ceil", - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Ceil', node, kwargs) @mx_op.register("floor") def convert_floor(node, **kwargs): """Map MXNet's floor operator attributes to onnx's Floor operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Floor", - [input_node], - [name], - name=name - ) - return [node] + return create_basic_op_node('Floor', node, kwargs) # Changing shape and type. @mx_op.register("Reshape") @@ -1831,16 +1362,12 @@ def convert_reshape(node, **kwargs): Converts output shape attribute to output shape tensor and return multiple created nodes. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) output_shape_list = convert_string_to_list(attrs["shape"]) initializer = kwargs["initializer"] - output_shape_np = np.array(output_shape_list) + output_shape_np = np.array(output_shape_list, dtype='int64') data_type = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[output_shape_np.dtype] dims = np.shape(output_shape_np) @@ -1857,18 +1384,17 @@ def convert_reshape(node, **kwargs): ) ) - input_node_idx = kwargs["index_lookup"][inputs[0][0]] - input_node_name = proc_nodes[input_node_idx].name + input_nodes.append(output_shape_name) not_supported_shape = [-2, -3, -4] for val in output_shape_list: if val in not_supported_shape: - raise AttributeError("Shape value not supported in ONNX", val) + raise AttributeError("Reshape: Shape value not supported in ONNX", val) reshape_node = onnx.helper.make_node( "Reshape", - [input_node_name, output_shape_name], + input_nodes, [name], name=name ) @@ -1880,11 +1406,9 @@ def convert_cast(node, **kwargs): """Map MXNet's Cast operator attributes to onnx's Cast operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - dtype = node["attrs"]["dtype"] + name, input_nodes, attrs = get_inputs(node, kwargs) + + dtype = attrs["dtype"] # dtype can be mapped only with types from TensorProto # float32 is mapped to float and float64 to double in onnx @@ -1894,12 +1418,9 @@ def convert_cast(node, **kwargs): elif dtype == 'float64': dtype = 'double' - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - node = onnx.helper.make_node( "Cast", - [input_node], + input_nodes, [name], to=getattr(onnx.TensorProto, dtype.upper()), name=name, @@ -1912,23 +1433,17 @@ def convert_slice_axis(node, **kwargs): """Map MXNet's slice_axis operator attributes to onnx's Slice operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - axes = int(node["attrs"]["axis"]) - starts = int(node["attrs"]["begin"]) - if node["attrs"]["end"] == 'None': - raise ValueError("Slice: ONNX doesnt't support 'None' in 'end' attribute") - else: - ends = int(node["attrs"]["end"]) + name, input_nodes, attrs = get_inputs(node, kwargs) - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + axes = int(attrs.get("axis")) + starts = int(attrs.get("begin")) + ends = int(attrs.get("end", None)) + if not ends: + raise ValueError("Slice: ONNX doesnt't support 'None' in 'end' attribute") node = onnx.helper.make_node( "Slice", - [input_node], + input_nodes, [name], axes=[axes], starts=[starts], @@ -1944,21 +1459,16 @@ def convert_slice_channel(node, **kwargs): operator based on squeeze_axis attribute and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - num_outputs = int(node.get("attrs", {})["num_outputs"]) - axis = int(node.get("attrs", {}).get("axis", 1)) - squeeze_axis = int(node.get("attrs", {}).get("squeeze_axis", 0)) + name, input_nodes, attrs = get_inputs(node, kwargs) - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + num_outputs = int(attrs.get("num_outputs")) + axis = int(attrs.get("axis", 1)) + squeeze_axis = int(attrs.get("squeeze_axis", 0)) if squeeze_axis == 1 and num_outputs == 1: node = onnx.helper.make_node( "Squeeze", - [input_node], + input_nodes, [name], axes=[axis], name=name, @@ -1967,7 +1477,7 @@ def convert_slice_channel(node, **kwargs): elif squeeze_axis == 0 and num_outputs > 1: node = onnx.helper.make_node( "Split", - [input_node], + input_nodes, [name], axis=axis, split=[num_outputs], @@ -1984,18 +1494,13 @@ def convert_expand_dims(node, **kwargs): """Map MXNet's expand_dims operator attributes to onnx's Unsqueeze operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - axis = int(node["attrs"]["axis"]) + name, input_nodes, attrs = get_inputs(node, kwargs) - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + axis = int(attrs.get("axis")) node = onnx.helper.make_node( "Unsqueeze", - [input_node], + input_nodes, [name], axes=[axis], name=name, @@ -2007,22 +1512,17 @@ def convert_squeeze(node, **kwargs): """Map MXNet's squeeze operator attributes to onnx's squeeze operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - if "axis" in node["attrs"]: - axis = convert_string_to_list(node["attrs"]["axis"]) - else: - raise AttributeError("Missing axis attribute: ONNX currently requires axis to " - "be specified for squeeze operator") + name, input_nodes, attrs = get_inputs(node, kwargs) - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + axis = attrs.get("axis", None) + if not axis: + raise AttributeError("Squeeze: Missing axis attribute: ONNX currently requires axis to " + "be specified for squeeze operator") + axis = convert_string_to_list(axis) node = onnx.helper.make_node( "Squeeze", - [input_node], + input_nodes, [name], axes=axis, name=name, @@ -2035,132 +1535,48 @@ def convert_log(node, **kwargs): """Map MXNet's log operator attributes to onnx's Log operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Log", - [input_node], - [name], - name=name, - ) - return [node] - + return create_basic_op_node('Log', node, kwargs) @mx_op.register("reciprocal") def convert_reciprocal(node, **kwargs): """Map MXNet's reciprocal operator attributes to onnx's Reciprocal operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Reciprocal", - [input_node], - [name], - name=name, - ) - return [node] + return create_basic_op_node('Reciprocal', node, kwargs) @mx_op.register("_power") def convert_power(node, **kwargs): """Map MXNet's _power operator attributes to onnx's Pow operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - node = onnx.helper.make_node( - "Pow", - [input_node_a, input_node_b], - [name], - name=name - ) - return [node] + return create_basic_op_node('Pow', node, kwargs) @mx_op.register("broadcast_power") def convert_broadcast_power(node, **kwargs): """Map MXNet's _power operator attributes to onnx's Pow operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_b_id = kwargs["index_lookup"][inputs[1][0]] - - input_node_a = proc_nodes[input_node_a_id].name - input_node_b = proc_nodes[input_node_b_id].name - - node = onnx.helper.make_node( - "Pow", - [input_node_a, input_node_b], - [name], - name=name - ) - return [node] + return create_basic_op_node('Pow', node, kwargs) @mx_op.register("sqrt") def convert_sqrt(node, **kwargs): """Map MXNet's sqrt operator attributes to onnx's Sqrt operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - - node = onnx.helper.make_node( - "Sqrt", - [input_node], - [name], - name=name, - ) - return [node] + return create_basic_op_node('Sqrt', node, kwargs) @mx_op.register("depth_to_space") def convert_depthtospace(node, **kwargs): """Map MXNet's depth_to_space operator attributes to onnx's DepthToSpace operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - attrs = node["attrs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + name, input_nodes, attrs = get_inputs(node, kwargs) blksize = int(attrs.get("block_size", 0)) node = onnx.helper.make_node( "DepthToSpace", - [input_node], + input_nodes, [name], blocksize=blksize, name=name, @@ -2172,20 +1588,13 @@ def convert_spacetodepth(node, **kwargs): """Map MXNet's space_to_depth operator attributes to onnx's SpaceToDepth operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - attrs = node["attrs"] - - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name + name, input_nodes, attrs = get_inputs(node, kwargs) blksize = int(attrs.get("block_size", 0)) node = onnx.helper.make_node( "SpaceToDepth", - [input_node], + input_nodes, [name], blocksize=blksize, name=name, @@ -2197,13 +1606,7 @@ def convert_square(node, **kwargs): """Map MXNet's square operator attributes to onnx's Pow operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - - input_node_a_id = kwargs["index_lookup"][inputs[0][0]] - input_node_a = proc_nodes[input_node_a_id].name + name, input_nodes, _ = get_inputs(node, kwargs) initializer = kwargs["initializer"] data_type = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype('int64')] @@ -2220,9 +1623,11 @@ def convert_square(node, **kwargs): ) ) + input_nodes.append(power2_name) + node = onnx.helper.make_node( "Pow", - [input_node_a, power2_name], + input_nodes, [name], name=name ) @@ -2233,24 +1638,17 @@ def convert_sum(node, **kwargs): """Map MXNet's sum operator attributes to onnx's ReduceSum operator and return the created node. """ - onnx = import_onnx_modules() - name = node["name"] - proc_nodes = kwargs["proc_nodes"] - inputs = node["inputs"] - attrs = node["attrs"] + name, input_nodes, attrs = get_inputs(node, kwargs) mx_axis = attrs.get("axis", None) axes = convert_string_to_list(str(mx_axis)) if mx_axis is not None else None keepdims = get_boolean_attribute_value(attrs, "keepdims") - input_node_id = kwargs["index_lookup"][inputs[0][0]] - input_node = proc_nodes[input_node_id].name - if axes: node = onnx.helper.make_node( 'ReduceSum', - inputs=[input_node], + inputs=input_nodes, outputs=[name], axes=axes, keepdims=keepdims, @@ -2259,9 +1657,253 @@ def convert_sum(node, **kwargs): else: node = onnx.helper.make_node( 'ReduceSum', - inputs=[input_node], + inputs=input_nodes, outputs=[name], keepdims=keepdims, name=name ) return [node] + + +@mx_op.register("shape_array") +def convert_shape(node, **kwargs): + """Map MXNet's shape_array operator attributes to onnx's Shape operator + and return the created node. + """ + return create_basic_op_node('Shape', node, kwargs) + + +@mx_op.register("hard_sigmoid") +def convert_hardsigmoid(node, **kwargs): + """Map MXNet's hard_sigmoid operator attributes to onnx's HardSigmoid operator + and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + # Converting to float32 + alpha = float(attrs.get("alpha", 0.2)) + beta = float(attrs.get("beta", 0.5)) + + node = onnx.helper.make_node( + 'HardSigmoid', + input_nodes, + [name], + alpha=alpha, + beta=beta, + name=name + ) + return [node] + +@mx_op.register("broadcast_lesser") +def convert_broadcast_lesser(node, **kwargs): + """Map MXNet's broadcast_lesser operator attributes to onnx's Less operator + and return the created node. + """ + return create_basic_op_node('Less', node, kwargs) + +@mx_op.register("broadcast_greater") +def convert_broadcast_greater(node, **kwargs): + """Map MXNet's broadcast_greater operator attributes to onnx's Greater operator + and return the created node. + """ + return create_basic_op_node('Greater', node, kwargs) + +@mx_op.register("broadcast_equal") +def convert_broadcast_equal(node, **kwargs): + """Map MXNet's broadcast_equal operator attributes to onnx's Equal operator + and return the created node. + """ + return create_basic_op_node('Equal', node, kwargs) + + +@mx_op.register("broadcast_logical_and") +def convert_broadcast_logical_and(node, **kwargs): + """Map MXNet's broadcast logical and operator attributes to onnx's Add operator + and return the created node. + """ + return create_basic_op_node('And', node, kwargs) + + +@mx_op.register("broadcast_logical_or") +def convert_broadcast_logical_or(node, **kwargs): + """Map MXNet's broadcast logical or operator attributes to onnx's Or operator + and return the created node. + """ + return create_basic_op_node('Or', node, kwargs) + + +@mx_op.register("broadcast_logical_xor") +def convert_broadcast_logical_xor(node, **kwargs): + """Map MXNet's broadcast logical xor operator attributes to onnx's Xor operator + and return the created node. + """ + return create_basic_op_node('Xor', node, kwargs) + + +@mx_op.register("logical_not") +def convert_logical_not(node, **kwargs): + """Map MXNet's logical not operator attributes to onnx's Not operator + and return the created node. + """ + return create_basic_op_node('Not', node, kwargs) + + +@mx_op.register("size_array") +def convert_size(node, **kwargs): + """Map MXNet's size_array operator attributes to onnx's Size operator + and return the created node. + """ + return create_basic_op_node('Size', node, kwargs) + + +@mx_op.register("log_softmax") +def convert_logsoftmax(node, **kwargs): + """Map MXNet's log_softmax operator attributes to onnx's LogSoftMax operator + and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + # Converting to int + axis = int(attrs.get("axis", -1)) + temp = attrs.get("temperature", 'None') + if temp != 'None': + raise AttributeError("LogSoftMax: ONNX supports only temperature=None") + + node = onnx.helper.make_node( + 'LogSoftmax', + input_nodes, + [name], + axis=axis, + name=name + ) + return [node] + +@mx_op.register("norm") +def convert_norm(node, **kwargs): + """Map MXNet's norm operator attributes to onnx's ReduceL1 and ReduceL2 operators + and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + mx_axis = attrs.get("axis", None) + axes = convert_string_to_list(str(mx_axis)) if mx_axis else None + + keepdims = get_boolean_attribute_value(attrs, "keepdims") + ord = int(attrs.get("ord", 2)) + + onnx_op_name = "ReduceL1" if ord == 1 else "ReduceL2" + + if axes: + reduce_node = onnx.helper.make_node( + onnx_op_name, + input_nodes, + [name], + axes=axes, + keepdims=keepdims, + name=name + ) + return [reduce_node] + else: + reduce_node = onnx.helper.make_node( + onnx_op_name, + input_nodes, + [name], + keepdims=keepdims, + name=name + ) + return [reduce_node] + +@mx_op.register("_sample_multinomial") +def convert_multinomial(node, **kwargs): + """Map MXNet's multinomial operator attributes to onnx's + Multinomial operator and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + dtype = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype(attrs.get("dtype", 'int32'))] + sample_size = convert_string_to_list(attrs.get("shape", '1')) + if len(sample_size) < 2: + sample_size = sample_size[-1] + else: + raise AttributeError("ONNX currently supports integer sample_size only") + node = onnx.helper.make_node( + "Multinomial", + input_nodes, + [name], + dtype=dtype, + sample_size=sample_size, + name=name, + ) + return [node] + + +@mx_op.register("_random_uniform") +def convert_random_uniform(node, **kwargs): + """Map MXNet's random_uniform operator attributes to onnx's RandomUniform + operator and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + # Converting to float32 + low = float(attrs.get("low", 0)) + high = float(attrs.get("high", 1.0)) + shape = convert_string_to_list(attrs.get('shape', '[]')) + dtype = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype(attrs.get('dtype', 'float32'))] + + node = onnx.helper.make_node( + 'RandomUniform', + input_nodes, + [name], + low=low, + high=high, + dtype=dtype, + shape=shape, + name=name + ) + return [node] + + +@mx_op.register("_random_normal") +def convert_random_normal(node, **kwargs): + """Map MXNet's random_normal operator attributes to onnx's RandomNormal + operator and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + # Converting to float32 + mean = float(attrs.get("loc", 0)) + scale = float(attrs.get("scale", 1.0)) + shape = convert_string_to_list(attrs.get('shape', '[]')) + dtype = onnx.mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype(attrs.get('dtype', 'float32'))] + + node = onnx.helper.make_node( + 'RandomNormal', + input_nodes, + [name], + mean=mean, + scale=scale, + dtype=dtype, + shape=shape, + name=name + ) + return [node] + + +@mx_op.register("ROIPooling") +def convert_roipooling(node, **kwargs): + """Map MXNet's ROIPooling operator attributes to onnx's MaxRoiPool + operator and return the created node. + """ + name, input_nodes, attrs = get_inputs(node, kwargs) + + pooled_shape = convert_string_to_list(attrs.get('pooled_size')) + scale = float(attrs.get("spatial_scale")) + + node = onnx.helper.make_node( + 'MaxRoiPool', + input_nodes, + [name], + pooled_shape=pooled_shape, + spatial_scale=scale, + name=name + ) + return [node] diff --git a/python/mxnet/contrib/onnx/mx2onnx/export_model.py b/python/mxnet/contrib/onnx/mx2onnx/export_model.py index e5158051d..f5e4c3b69 100644 --- a/python/mxnet/contrib/onnx/mx2onnx/export_model.py +++ b/python/mxnet/contrib/onnx/mx2onnx/export_model.py @@ -58,6 +58,11 @@ def export_model(sym, params, input_shape, input_type=np.float32, ------- onnx_file_path : str Onnx file path + + Notes + ----- + This method is available when you ``import mxnet.contrib.onnx`` + """ try: diff --git a/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py b/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py index 11847381a..d0d4501d8 100644 --- a/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py +++ b/python/mxnet/contrib/onnx/mx2onnx/export_onnx.py @@ -53,12 +53,8 @@ from __future__ import unicode_literals import logging import json -import numpy as np -from .... import context from .... import ndarray as nd -from .... import io -from .... import module as mod class MXNetGraph(object): @@ -77,7 +73,11 @@ def register(op_name): """Register operators""" def wrapper(func): """Helper function to map functions""" - MXNetGraph.registry_[op_name] = func + try: + import onnx as _ + MXNetGraph.registry_[op_name] = func + except ImportError: + pass return func return wrapper @@ -91,60 +91,6 @@ def convert_layer(node, **kwargs): convert_func = MXNetGraph.registry_[op] return convert_func(node, **kwargs) - @staticmethod - def forward_pass(inputs, sym, arg_params, aux_params, output_label): - """Do a forward pass based on the sym and params to get the shape - of the output using dummy data - - Parameters - ---------- - inputs : json string - - sym : :class:`~mxnet.symbol.Symbol` - MXNet symbol object - arg_params : dict of ``str`` to :class:`~mxnet.ndarray.NDArray` - Dict of converted parameters stored in ``mxnet.ndarray.NDArray`` format - aux_params : dict of ``str`` to :class:`~mxnet.ndarray.NDArray` - Dict of converted parameters stored in ``mxnet.ndarray.NDArray`` format - - Returns - ------- - shape : Shape - Output shape - """ - # if label is not provided, MXNet adds label "softmax_label" by default - # while running load_checkpoint which is not actually a graph input. So ignoring it here - data_names = [graph_input for graph_input in sym.list_inputs() - if graph_input not in arg_params and graph_input not in aux_params - and graph_input != output_label] - - data_shapes = [] - # Adding extra dimension of batch_size 1 if the batch_size is different for multiple inputs. - for idx, input_name in enumerate(data_names): - data_shapes.append((input_name, inputs[idx].shape)) - - # create module, passing cpu context - ctx = context.cpu() - test_mod = mod.Module(symbol=sym, data_names=data_names, context=ctx, label_names=None) - test_mod.bind(for_training=False, data_shapes=data_shapes, label_shapes=None) - - # initializing parameters for calculating result of each individual node - if arg_params is None and aux_params is None: - test_mod.init_params() - else: - test_mod.set_params(arg_params=arg_params, aux_params=aux_params, allow_missing=True) - - data_forward = [] - for idx, input_name in enumerate(data_names): - val = inputs[idx] - data_forward.append(nd.array(val)) - - test_mod.forward(io.DataBatch(data_forward)) - result = test_mod.get_outputs()[0].asnumpy() - - return result.shape - - @staticmethod def split_params(sym, params): """Helper function to split params dictionary into args and aux params @@ -173,15 +119,41 @@ def split_params(sym, params): aux_params.update({aux: nd.array(params[aux])}) return arg_params, aux_params - @staticmethod - def infer_output_shape(sym, params, in_shape, output_label): - """Infer output shape by doing a forward pass using dummy inputs """ - # create dummy input - inputs = [np.random.randn(*input_shape) for input_shape in in_shape] - arg, aux = MXNetGraph.split_params(sym, params) - return MXNetGraph.forward_pass(inputs, sym, arg, aux, output_label) + def get_outputs(sym, params, in_shape, in_label): + """ Infer output shapes and return dictionary of output name to shape + + :param :class:`~mxnet.symbol.Symbol` sym: symbol to perform infer shape on + :param dic of (str, nd.NDArray) params: + :param list of tuple(int, ...) in_shape: list of all input shapes + :param in_label: name of label typically used in loss that may be left in graph. This name is + removed from list of inputs required by symbol + :return: dictionary of output name to shape + :rtype: dict of (str, tuple(int, ...)) + """ + # remove any input listed in params from sym.list_inputs() and bind them to the input shapes provided + # by user. Also remove in_label, which is the name of the label symbol that may have been used + # as the label for loss during training. + inputs = {n: tuple(s) for n, s in zip([n for n in sym.list_inputs() if n not in params and n != in_label], + in_shape)} + # Add params and their shape to list of inputs + inputs.update({n: v.shape for n, v in params.items() if n in sym.list_inputs()}) + # Provide input data as well as input params to infer_shape() + _, out_shapes, _ = sym.infer_shape(**inputs) + + out_names = list() + for name in sym.list_outputs(): + if name.endswith('_output'): + out_names.append(name[:-len('_output')]) + else: + logging.info("output '%s' does not end with '_output'", name) + out_names.append(name) + assert len(out_shapes) == len(out_names) + # bind output shapes with output names + graph_outputs = {n: s for n, s in zip(out_names, out_shapes)} + + return graph_outputs @staticmethod def convert_weights_to_numpy(weights_dict): @@ -224,9 +196,6 @@ def create_onnx_graph_proto(self, sym, params, in_shape, in_type, verbose=False) # Deriving the output_label name. output_label = sym.get_internals()[len(sym.get_internals()) - 1].name + "_label" - # Determine output shape - output_shape = MXNetGraph.infer_output_shape(sym, params, in_shape, output_label) - weights = MXNetGraph.convert_weights_to_numpy(params) mx_graph = json.loads(sym.tojson())["nodes"] @@ -238,6 +207,9 @@ def create_onnx_graph_proto(self, sym, params, in_shape, in_type, verbose=False) onnx_processed_outputs = [] index_lookup = [] + # Determine output shape + graph_outputs = MXNetGraph.get_outputs(sym, params, in_shape, output_label) + graph_input_idx = 0 for idx, node in enumerate(mx_graph): op = node["op"] @@ -290,24 +262,15 @@ def create_onnx_graph_proto(self, sym, params, in_shape, in_type, verbose=False) # If converted node is NodeProto, add it in processed nodes list elif isinstance(converted_node, NodeProto): onnx_processed_nodes.append(converted_node) - if idx == (len(mx_graph) - 1): - # If converted node doesnt have name, use it from output field - if not converted_node.name: - onnx_processed_outputs.append( - make_tensor_value_info( - name=converted_node.output[0], - elem_type=in_type, - shape=output_shape - ) - ) - else: - onnx_processed_outputs.append( - make_tensor_value_info( - name=converted_node.name, - elem_type=in_type, - shape=output_shape - ) + node_name = converted_node.name if converted_node.name else converted_node.output[0] + if node_name in graph_outputs: + onnx_processed_outputs.append( + make_tensor_value_info( + name=node_name, + elem_type=in_type, + shape=graph_outputs[node_name] ) + ) if verbose: logging.info("Output node is: %s", converted_node.name) elif isinstance(converted_node, TensorProto): diff --git a/python/mxnet/contrib/onnx/onnx2mx/_import_helper.py b/python/mxnet/contrib/onnx/onnx2mx/_import_helper.py index f61910f83..cf95bfef0 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/_import_helper.py +++ b/python/mxnet/contrib/onnx/onnx2mx/_import_helper.py @@ -18,12 +18,12 @@ # coding: utf-8_ # pylint: disable=invalid-name """Operator attributes conversion""" -from ._op_translations import identity, random_uniform, random_normal +from ._op_translations import identity, random_uniform, random_normal, sample_multinomial from ._op_translations import add, subtract, multiply, divide, absolute, negative, add_n from ._op_translations import tanh, arccos, arcsin, arctan, _cos, _sin, _tan -from ._op_translations import softplus, shape, gather, lp_pooling +from ._op_translations import softplus, shape, gather, lp_pooling, size from ._op_translations import ceil, floor, hardsigmoid, global_lppooling -from ._op_translations import concat +from ._op_translations import concat, hardmax from ._op_translations import leaky_relu, _elu, _prelu, _selu, softmax, fully_connected from ._op_translations import global_avgpooling, global_maxpooling, linalg_gemm from ._op_translations import sigmoid, pad, relu, matrix_multiplication, batch_norm @@ -37,7 +37,7 @@ from ._op_translations import reduce_sum_square, reduce_l1, reduce_l2, max_roi_pooling from ._op_translations import log_softmax, softsign, lesser, greater, equal from ._op_translations import logical_and, logical_or, logical_xor, logical_not -from ._op_translations import mean, depthtospace, spacetodepth +from ._op_translations import mean, depthtospace, spacetodepth, lpnormalization # convert_map defines maps of ONNX operator names to converter functor(callable) # defined in the op_translations module. @@ -48,6 +48,7 @@ 'RandomNormal' : random_normal, 'RandomUniformLike' : random_uniform, 'RandomNormalLike' : random_normal, + 'Multinomial' : sample_multinomial, # Arithmetic Operators 'Add' : add, 'Sub' : subtract, @@ -139,9 +140,12 @@ 'Softplus' : softplus, 'Tan' : _tan, 'Shape' : shape, + 'Size' : size, 'Gather' : gather, 'HardSigmoid' : hardsigmoid, 'LpPool' : lp_pooling, 'DepthToSpace' : depthtospace, - 'SpaceToDepth' : spacetodepth + 'SpaceToDepth' : spacetodepth, + 'Hardmax' : hardmax, + 'LpNormalization' : lpnormalization } diff --git a/python/mxnet/contrib/onnx/onnx2mx/_op_translations.py b/python/mxnet/contrib/onnx/onnx2mx/_op_translations.py index 368b98d66..dc00feee8 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/_op_translations.py +++ b/python/mxnet/contrib/onnx/onnx2mx/_op_translations.py @@ -29,14 +29,39 @@ def identity(attrs, inputs, proto_obj): def random_uniform(attrs, inputs, proto_obj): """Draw random samples from a uniform distribtuion.""" - new_attr = translation_utils._remove_attributes(attrs, ['seed']) - return 'random_uniform', new_attr, inputs + try: + from onnx.mapping import TENSOR_TYPE_TO_NP_TYPE + except ImportError: + raise ImportError("Onnx and protobuf need to be installed. " + "Instructions to install - https://github.com/onnx/onnx") + new_attrs = translation_utils._remove_attributes(attrs, ['seed']) + new_attrs['dtype'] = TENSOR_TYPE_TO_NP_TYPE[int(new_attrs.get('dtype', 1))] + return 'random_uniform', new_attrs, inputs def random_normal(attrs, inputs, proto_obj): """Draw random samples from a Gaussian distribution.""" + try: + from onnx.mapping import TENSOR_TYPE_TO_NP_TYPE + except ImportError: + raise ImportError("Onnx and protobuf need to be installed. " + "Instructions to install - https://github.com/onnx/onnx") new_attr = translation_utils._remove_attributes(attrs, ['seed']) - new_attr = translation_utils._fix_attribute_names(new_attr, {'mean' : 'loc'}) - return 'random_uniform', new_attr, inputs + new_attr = translation_utils._fix_attribute_names(new_attr, {'mean': 'loc'}) + new_attr['dtype'] = TENSOR_TYPE_TO_NP_TYPE[int(new_attr.get('dtype', 1))] + return 'random_normal', new_attr, inputs + +def sample_multinomial(attrs, inputs, proto_obj): + """Draw random samples from a multinomial distribution.""" + try: + from onnx.mapping import TENSOR_TYPE_TO_NP_TYPE + except ImportError: + raise ImportError("Onnx and protobuf need to be installed. " + + "Instructions to install - https://github.com/onnx/onnx") + new_attrs = translation_utils._remove_attributes(attrs, ['seed']) + new_attrs = translation_utils._fix_attribute_names(new_attrs, {'sample_size': 'shape'}) + new_attrs['dtype'] = TENSOR_TYPE_TO_NP_TYPE[int(attrs.get('dtype', 6))] + return 'sample_multinomial', new_attrs, inputs + # Arithmetic Operations def add(attrs, inputs, proto_obj): @@ -382,6 +407,7 @@ def global_lppooling(attrs, inputs, proto_obj): 'kernel': (1, 1), 'pool_type': 'lp', 'p_value': p_value}) + new_attrs = translation_utils._remove_attributes(new_attrs, ['p']) return 'Pooling', new_attrs, inputs def linalg_gemm(attrs, inputs, proto_obj): @@ -642,6 +668,10 @@ def shape(attrs, inputs, proto_obj): """Returns shape of input array.""" return 'shape_array', attrs, inputs +def size(attrs, inputs, proto_obj): + """Returns array containing size of data.""" + return "size_array", attrs, inputs + def reduce_l2(attrs, inputs, proto_obj): """Reduce input tensor by l2 normalization.""" new_attrs = translation_utils._fix_attribute_names(attrs, {'axes':'axis'}) @@ -667,11 +697,12 @@ def lp_pooling(attrs, inputs, proto_obj): new_attrs = translation_utils._fix_attribute_names(attrs, {'kernel_shape': 'kernel', 'strides': 'stride', - 'pads': 'pad', - 'p_value': p_value + 'pads': 'pad' }) + new_attrs = translation_utils._remove_attributes(new_attrs, ['p']) new_attrs = translation_utils._add_extra_attributes(new_attrs, - {'pooling_convention': 'valid' + {'pooling_convention': 'valid', + 'p_value': p_value }) new_op = translation_utils._fix_pooling('lp', inputs, new_attrs) return new_op, new_attrs, inputs @@ -710,3 +741,37 @@ def spacetodepth(attrs, inputs, proto_obj): new_attrs = translation_utils._fix_attribute_names(attrs, {'blocksize':'block_size'}) return "space_to_depth", new_attrs, inputs + +def hardmax(attrs, inputs, proto_obj): + """Returns batched one-hot vectors.""" + input_tensor_data = proto_obj.model_metadata.get('input_tensor_data')[0] + input_shape = input_tensor_data[1] + + axis = int(attrs.get('axis', 1)) + axis = axis if axis >= 0 else len(input_shape) + axis + + if axis == len(input_shape) - 1: + amax = symbol.argmax(inputs[0], axis=-1) + one_hot = symbol.one_hot(amax, depth=input_shape[-1]) + return one_hot, attrs, inputs + + # since reshape doesn't take a tensor for shape, + # computing with np.prod. This needs to be changed to + # to use mx.sym.prod() when mx.sym.reshape() is fixed. + # (https://github.com/apache/incubator-mxnet/issues/10789) + new_shape = (int(np.prod(input_shape[:axis])), + int(np.prod(input_shape[axis:]))) + reshape_op = symbol.reshape(inputs[0], new_shape) + amax = symbol.argmax(reshape_op, axis=-1) + one_hot = symbol.one_hot(amax, depth=new_shape[-1]) + hardmax_op = symbol.reshape(one_hot, input_shape) + return hardmax_op, attrs, inputs + +def lpnormalization(attrs, inputs, proto_obj): + """ONNX does not have eps attribute, so cannot map it to L2normalization in MXNet + without that, it works as norm operator discussion in PR: + https://github.com/onnx/onnx/pull/1330""" + new_attrs = translation_utils._fix_attribute_names(attrs, {'p': 'ord'}) + axis = int(attrs.get("axis", -1)) + new_attrs.update(axis=axis) + return 'norm', new_attrs, inputs diff --git a/python/mxnet/contrib/onnx/onnx2mx/_translation_utils.py b/python/mxnet/contrib/onnx/onnx2mx/_translation_utils.py index f63c1e9e8..6fd52665c 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/_translation_utils.py +++ b/python/mxnet/contrib/onnx/onnx2mx/_translation_utils.py @@ -94,6 +94,7 @@ def _fix_pooling(pool_type, inputs, new_attr): stride = new_attr.get('stride') kernel = new_attr.get('kernel') padding = new_attr.get('pad') + p_value = new_attr.get('p_value') # Adding default stride. if stride is None: @@ -138,7 +139,10 @@ def _fix_pooling(pool_type, inputs, new_attr): new_pad_op = symbol.pad(curr_sym, mode='constant', pad_width=pad_width) # Apply pooling without pads. - new_pooling_op = symbol.Pooling(new_pad_op, pool_type=pool_type, stride=stride, kernel=kernel) + if pool_type == 'lp': + new_pooling_op = symbol.Pooling(new_pad_op, pool_type=pool_type, stride=stride, kernel=kernel, p_value=p_value) + else: + new_pooling_op = symbol.Pooling(new_pad_op, pool_type=pool_type, stride=stride, kernel=kernel) return new_pooling_op def _fix_bias(op_name, attrs, num_inputs): diff --git a/python/mxnet/contrib/onnx/onnx2mx/import_model.py b/python/mxnet/contrib/onnx/onnx2mx/import_model.py index bf3601a69..0048c748e 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/import_model.py +++ b/python/mxnet/contrib/onnx/onnx2mx/import_model.py @@ -41,6 +41,11 @@ def import_model(model_file): aux_params : dict of ``str`` to :class:`~mxnet.ndarray.NDArray` Dict of converted parameters stored in ``mxnet.ndarray.NDArray`` format + + Notes + ----- + This method is available when you ``import mxnet.contrib.onnx`` + """ graph = GraphProto() @@ -58,6 +63,10 @@ def get_model_metadata(model_file): """ Returns the name and shape information of input and output tensors of the given ONNX model file. + Notes + ----- + This method is available when you ``import mxnet.contrib.onnx`` + Parameters ---------- model_file : str @@ -67,15 +76,13 @@ def get_model_metadata(model_file): ------- model_metadata : dict A dictionary object mapping various metadata to its corresponding value. - The dictionary will have the following template. - { - 'input_tensor_data' : , - 'output_tensor_data' : - } + The dictionary will have the following template:: + 'input_tensor_data' : list of tuples representing the shape of the input paramters + 'output_tensor_data' : list of tuples representing the shape of the output of the model """ graph = GraphProto() + try: import onnx except ImportError: diff --git a/python/mxnet/contrib/onnx/onnx2mx/import_onnx.py b/python/mxnet/contrib/onnx/onnx2mx/import_onnx.py index 3af196f8b..d5071af07 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/import_onnx.py +++ b/python/mxnet/contrib/onnx/onnx2mx/import_onnx.py @@ -19,6 +19,7 @@ # pylint: disable=invalid-name,too-many-locals,no-self-use """ Support import export formats.""" from __future__ import absolute_import as _abs +import numpy as np from .... import symbol from .... import ndarray as nd from ....base import string_types @@ -87,7 +88,7 @@ def from_onnx(self, graph): params : dict A dict of name: nd.array pairs, used as pretrained weights """ - #get input, output shapes + # get input, output shapes self.model_metadata = self.get_graph_metadata(graph) # parse network inputs, aka parameters for init_tensor in graph.initializer: @@ -196,7 +197,11 @@ def _parse_array(self, tensor_proto): except ImportError: raise ImportError("Onnx and protobuf need to be installed. " + "Instructions to install - https://github.com/onnx/onnx") - np_array = to_array(tensor_proto).reshape(tuple(tensor_proto.dims)) + if len(tuple(tensor_proto.dims)) > 0: + np_array = to_array(tensor_proto).reshape(tuple(tensor_proto.dims)) + else: + # If onnx's params are scalar values without dims mentioned. + np_array = np.array([to_array(tensor_proto)]) return nd.array(np_array) def _parse_attr(self, attr_proto): diff --git a/python/mxnet/contrib/onnx/onnx2mx/import_to_gluon.py b/python/mxnet/contrib/onnx/onnx2mx/import_to_gluon.py index 8e1b35441..13ad5b9f8 100644 --- a/python/mxnet/contrib/onnx/onnx2mx/import_to_gluon.py +++ b/python/mxnet/contrib/onnx/onnx2mx/import_to_gluon.py @@ -36,6 +36,11 @@ def import_to_gluon(model_file, ctx): ------- sym_block : :class:`~mxnet.gluon.SymbolBlock` A SymbolBlock object representing the given model file. + + Notes + ----- + This method is available when you ``import mxnet.contrib.onnx`` + """ graph = GraphProto() try: diff --git a/python/mxnet/contrib/quantization.py b/python/mxnet/contrib/quantization.py index 3b0401635..61ad8a3ec 100644 --- a/python/mxnet/contrib/quantization.py +++ b/python/mxnet/contrib/quantization.py @@ -71,11 +71,11 @@ def _quantize_params(qsym, params, th_dict): elif name in params: quantized_params[name] = params[name] elif name.endswith(('_min')): - output = name[: - len('_min')] + "_output" + output = name[: - len('_min')] if output in th_dict: quantized_params[name] = ndarray.array([th_dict[output][0]]) elif name.endswith(('_max')): - output = name[: - len('_min')] + "_output" + output = name[: - len('_min')] if output in th_dict: quantized_params[name] = ndarray.array([th_dict[output][1]]) return quantized_params @@ -513,8 +513,6 @@ def quantize_model(sym, arg_params, aux_params, if not isinstance(calib_data, DataIter): raise ValueError('calib_data must be of DataIter type when calib_mode=%s,' ' while received type %s' % (calib_mode, str(type(calib_data)))) - if calib_layer is None: - calib_layer = lambda name: name.endswith('_output') mod = Module(symbol=sym, data_names=data_names, label_names=label_names, context=ctx) if len(calib_data.provide_label) > 0: diff --git a/python/mxnet/contrib/svrg_optimization/svrg_module.py b/python/mxnet/contrib/svrg_optimization/svrg_module.py index 5d6b5dd57..eecb87cf2 100644 --- a/python/mxnet/contrib/svrg_optimization/svrg_module.py +++ b/python/mxnet/contrib/svrg_optimization/svrg_module.py @@ -37,8 +37,7 @@ class SVRGModule(Module): data_names : list of str Defaults to `('data')` for a typical model used in image classification. label_names : list of str - Defaults to `('softmax_label')` for a typical model used in image - classification. + Defaults to `('softmax_label')` for a typical model used in image classification. logger : Logger Defaults to `logging`. context : Context or list of Context @@ -48,18 +47,19 @@ class SVRGModule(Module): fixed_param_names: list of str Default ``None``, indicating no network parameters are fixed. state_names : list of str - states are similar to data and label, but not provided by data iterator. + states are similar to data and label, but not provided by data iterator. \ Instead they are initialized to 0 and can be set by `set_states()`. group2ctxs : dict of str to context or list of context, or list of dict of str to context Default is `None`. Mapping the `ctx_group` attribute to the context assignment. compression_params : dict - Specifies type of gradient compression and additional arguments depending - on the type of compression being used. For example, 2bit compression requires a threshold. - Arguments would then be {'type':'2bit', 'threshold':0.5} - See mxnet.KVStore.set_gradient_compression method for more details on gradient compression. + Specifies type of gradient compression and additional arguments depending \ + on the type of compression being used. For example, 2bit compression requires a threshold. \ + Arguments would then be {'type':'2bit', 'threshold':0.5} \ + See mxnet.KVStore.set_gradient_compression method for more details on gradient compression. \ update_freq: int - Specifies the number of times to update the full gradients to be used in the SVRG optimization. For instance, + Specifies the number of times to update the full gradients to be used in the SVRG optimization. For instance, \ update_freq = 2 will calculates the gradients over all data every two epochs + Examples -------- >>> # An example of declaring and using SVRGModule. @@ -401,6 +401,7 @@ def fit(self, train_data, eval_data=None, eval_metric='acc', force_rebind=False, force_init=False, begin_epoch=0, num_epoch=None, validation_metric=None, monitor=None, sparse_row_id_fn=None): """Trains the module parameters. + Parameters ---------- train_data : DataIter diff --git a/python/mxnet/contrib/text/embedding.py b/python/mxnet/contrib/text/embedding.py index 277f78222..b7f3fcbc5 100644 --- a/python/mxnet/contrib/text/embedding.py +++ b/python/mxnet/contrib/text/embedding.py @@ -161,7 +161,7 @@ class _TokenEmbedding(vocab.Vocabulary): pre-trained token embedding file, are taken as the indexed tokens of the embedding. - Properties + Attributes ---------- token_to_idx : dict mapping str to int A dict mapping each token to its index integer. @@ -475,7 +475,8 @@ class GloVe(_TokenEmbedding): the resulting representations showcase interesting linear substructures of the word vector space. (Source from https://nlp.stanford.edu/projects/glove/) - Reference: + References + ---------- GloVe: Global Vectors for Word Representation. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. @@ -506,25 +507,6 @@ class GloVe(_TokenEmbedding): embedding vectors, such as loaded from a pre-trained token embedding file. If None, all the tokens from the loaded embedding vectors, such as loaded from a pre-trained token embedding file, will be indexed. - - - Properties - ---------- - token_to_idx : dict mapping str to int - A dict mapping each token to its index integer. - idx_to_token : list of strs - A list of indexed tokens where the list indices and the token indices are aligned. - unknown_token : hashable object - The representation for any unknown token. In other words, any unknown token will be indexed - as the same representation. - reserved_tokens : list of strs or None - A list of reserved tokens that will always be indexed. - vec_len : int - The length of the embedding vector for each token. - idx_to_vec : mxnet.ndarray.NDArray - For all the indexed tokens in this embedding, this NDArray maps each token's index to an - embedding vector. The largest valid index maps to the initialized embedding vector for every - reserved token, such as an unknown_token token and a padding token. """ # Map a pre-trained token embedding archive file and its SHA-1 hash. @@ -564,7 +546,8 @@ class FastText(_TokenEmbedding): representations and text classifiers. It works on standard, generic hardware. Models can later be reduced in size to even fit on mobile devices. (Source from https://fasttext.cc/) - References: + References + ---------- Enriching Word Vectors with Subword Information. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. @@ -610,25 +593,6 @@ class FastText(_TokenEmbedding): embedding vectors, such as loaded from a pre-trained token embedding file. If None, all the tokens from the loaded embedding vectors, such as loaded from a pre-trained token embedding file, will be indexed. - - - Properties - ---------- - token_to_idx : dict mapping str to int - A dict mapping each token to its index integer. - idx_to_token : list of strs - A list of indexed tokens where the list indices and the token indices are aligned. - unknown_token : hashable object - The representation for any unknown token. In other words, any unknown token will be indexed - as the same representation. - reserved_tokens : list of strs or None - A list of reserved tokens that will always be indexed. - vec_len : int - The length of the embedding vector for each token. - idx_to_vec : mxnet.ndarray.NDArray - For all the indexed tokens in this embedding, this NDArray maps each token's index to an - embedding vector. The largest valid index maps to the initialized embedding vector for every - reserved token, such as an unknown_token token and a padding token. """ # Map a pre-trained token embedding archive file and its SHA-1 hash. @@ -687,25 +651,6 @@ class CustomEmbedding(_TokenEmbedding): embedding vectors, such as loaded from a pre-trained token embedding file. If None, all the tokens from the loaded embedding vectors, such as loaded from a pre-trained token embedding file, will be indexed. - - - Properties - ---------- - token_to_idx : dict mapping str to int - A dict mapping each token to its index integer. - idx_to_token : list of strs - A list of indexed tokens where the list indices and the token indices are aligned. - unknown_token : hashable object - The representation for any unknown token. In other words, any unknown token will be indexed - as the same representation. - reserved_tokens : list of strs or None - A list of reserved tokens that will always be indexed. - vec_len : int - The length of the embedding vector for each token. - idx_to_vec : mxnet.ndarray.NDArray - For all the indexed tokens in this embedding, this NDArray maps each token's index to an - embedding vector. The largest valid index maps to the initialized embedding vector for every - reserved token, such as an unknown_token token and a padding token. """ def __init__(self, pretrained_file_path, elem_delim=' ', encoding='utf8', @@ -735,25 +680,6 @@ class CompositeEmbedding(_TokenEmbedding): token_embeddings : instance or list of `mxnet.contrib.text.embedding._TokenEmbedding` One or multiple pre-trained token embeddings to load. If it is a list of multiple embeddings, these embedding vectors will be concatenated for each token. - - - Properties - ---------- - token_to_idx : dict mapping str to int - A dict mapping each token to its index integer. - idx_to_token : list of strs - A list of indexed tokens where the list indices and the token indices are aligned. - unknown_token : hashable object - The representation for any unknown token. In other words, any unknown token will be indexed - as the same representation. - reserved_tokens : list of strs or None - A list of reserved tokens that will always be indexed. - vec_len : int - The length of the embedding vector for each token. - idx_to_vec : mxnet.ndarray.NDArray - For all the indexed tokens in this embedding, this NDArray maps each token's index to an - embedding vector. The largest valid index maps to the initialized embedding vector for every - reserved token, such as an unknown_token token and a padding token. """ def __init__(self, vocabulary, token_embeddings): diff --git a/python/mxnet/contrib/text/vocab.py b/python/mxnet/contrib/text/vocab.py index 9e44acb10..ede2ca535 100644 --- a/python/mxnet/contrib/text/vocab.py +++ b/python/mxnet/contrib/text/vocab.py @@ -63,12 +63,8 @@ class Vocabulary(object): `reserved_tokens` must be of the same hashable type. Examples: str, int, and tuple. - Properties + Attributes ---------- - token_to_idx : dict mapping str to int - A dict mapping each token to its index integer. - idx_to_token : list of strs - A list of indexed tokens where the list indices and the token indices are aligned. unknown_token : hashable object The representation for any unknown token. In other words, any unknown token will be indexed as the same representation. @@ -143,10 +139,16 @@ def __len__(self): @property def token_to_idx(self): + """ + dict mapping str to int: A dict mapping each token to its index integer. + """ return self._token_to_idx @property def idx_to_token(self): + """ + list of strs: A list of indexed tokens where the list indices and the token indices are aligned. + """ return self._idx_to_token @property diff --git a/python/mxnet/gluon/block.py b/python/mxnet/gluon/block.py index 6cb9fc690..704736496 100644 --- a/python/mxnet/gluon/block.py +++ b/python/mxnet/gluon/block.py @@ -326,8 +326,7 @@ def save_parameters(self, filename): References ---------- - `Saving and Loading Gluon Models - + `Saving and Loading Gluon Models \ `_ """ params = self._collect_params_with_prefix() @@ -372,8 +371,7 @@ def load_parameters(self, filename, ctx=None, allow_missing=False, References ---------- - `Saving and Loading Gluon Models - + `Saving and Loading Gluon Models \ `_ """ loaded = ndarray.load(filename) diff --git a/python/mxnet/gluon/contrib/nn/basic_layers.py b/python/mxnet/gluon/contrib/nn/basic_layers.py index c656cd2d4..28fea1592 100644 --- a/python/mxnet/gluon/contrib/nn/basic_layers.py +++ b/python/mxnet/gluon/contrib/nn/basic_layers.py @@ -27,7 +27,7 @@ from ...nn import Sequential, HybridSequential, BatchNorm class Concurrent(Sequential): - """Lays `Block`s concurrently. + """Lays `Block` s concurrently. This block feeds its input to all children blocks, and produce the output by concatenating all the children blocks' outputs @@ -60,7 +60,7 @@ def forward(self, x): class HybridConcurrent(HybridSequential): - """Lays `HybridBlock`s concurrently. + """Lays `HybridBlock` s concurrently. This block feeds its input to all children blocks, and produce the output by concatenating all the children blocks' outputs diff --git a/python/mxnet/gluon/contrib/rnn/conv_rnn_cell.py b/python/mxnet/gluon/contrib/rnn/conv_rnn_cell.py index 09db5470e..b7a19f78b 100644 --- a/python/mxnet/gluon/contrib/rnn/conv_rnn_cell.py +++ b/python/mxnet/gluon/contrib/rnn/conv_rnn_cell.py @@ -255,7 +255,7 @@ class Conv1DRNNCell(_ConvRNNCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_rnn_' + prefix : str, default ``'conv_rnn_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -322,7 +322,7 @@ class Conv2DRNNCell(_ConvRNNCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_rnn_' + prefix : str, default ``'conv_rnn_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -389,7 +389,7 @@ class Conv3DRNNCell(_ConvRNNCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_rnn_' + prefix : str, default ``'conv_rnn_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -519,7 +519,7 @@ class Conv1DLSTMCell(_ConvLSTMCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_lstm_' + prefix : str, default ``'conv_lstm_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -596,7 +596,7 @@ class Conv2DLSTMCell(_ConvLSTMCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_lstm_' + prefix : str, default ``'conv_lstm_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -673,7 +673,7 @@ class Conv3DLSTMCell(_ConvLSTMCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_lstm_' + prefix : str, default ``'conv_lstm_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -803,7 +803,7 @@ class Conv1DGRUCell(_ConvGRUCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_gru_' + prefix : str, default ``'conv_gru_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -875,7 +875,7 @@ class Conv2DGRUCell(_ConvGRUCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_gru_' + prefix : str, default ``'conv_gru_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -947,7 +947,7 @@ class Conv3DGRUCell(_ConvGRUCell): If argument type is string, it's equivalent to nn.Activation(act_type=str). See :func:`~mxnet.ndarray.Activation` for available choices. Alternatively, other activation blocks such as nn.LeakyReLU can be used. - prefix : str, default 'conv_gru_' + prefix : str, default ``'conv_gru_``' Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. diff --git a/python/mxnet/gluon/contrib/rnn/rnn_cell.py b/python/mxnet/gluon/contrib/rnn/rnn_cell.py index 1b9afee14..0cbc9eaac 100644 --- a/python/mxnet/gluon/contrib/rnn/rnn_cell.py +++ b/python/mxnet/gluon/contrib/rnn/rnn_cell.py @@ -26,7 +26,7 @@ class VariationalDropoutCell(ModifierCell): """ Applies Variational Dropout on base cell. - (https://arxiv.org/pdf/1512.05287.pdf, + (https://arxiv.org/pdf/1512.05287.pdf, \ https://www.stat.berkeley.edu/~tsmoon/files/Conference/asru2015.pdf). Variational dropout uses the same dropout mask across time-steps. It can be applied to RNN @@ -197,24 +197,29 @@ def unroll(self, length, inputs, begin_state=None, layout='NTC', merge_outputs=N class LSTMPCell(HybridRecurrentCell): r"""Long-Short Term Memory Projected (LSTMP) network cell. (https://arxiv.org/abs/1402.1128) + Each call computes the following function: + .. math:: \begin{array}{ll} i_t = sigmoid(W_{ii} x_t + b_{ii} + W_{ri} r_{(t-1)} + b_{ri}) \\ f_t = sigmoid(W_{if} x_t + b_{if} + W_{rf} r_{(t-1)} + b_{rf}) \\ - g_t = \tanh(W_{ig} x_t + b_{ig} + W_{rc} r_{(t-1)} + b_{rg}}) \\ + g_t = \tanh(W_{ig} x_t + b_{ig} + W_{rc} r_{(t-1)} + b_{rg}) \\ o_t = sigmoid(W_{io} x_t + b_{io} + W_{ro} r_{(t-1)} + b_{ro}) \\ c_t = f_t * c_{(t-1)} + i_t * g_t \\ h_t = o_t * \tanh(c_t) \\ r_t = W_{hr} h_t \end{array} + where :math:`r_t` is the projected recurrent activation at time `t`, - math:`h_t` is the hidden state at time `t`, :math:`c_t` is the + :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell state at time `t`, :math:`x_t` is the input at time `t`, and :math:`i_t`, :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, and out gates, respectively. + Parameters ---------- + hidden_size : int Number of units in cell state symbol. projection_size : int @@ -234,7 +239,7 @@ class LSTMPCell(HybridRecurrentCell): to zero. h2h_bias_initializer : str or Initializer Initializer for the bias vector. - prefix : str, default 'lstmp_' + prefix : str, default ``'lstmp_``' Prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None diff --git a/python/mxnet/gluon/data/dataloader.py b/python/mxnet/gluon/data/dataloader.py index 50e2ad9f7..9d762745a 100644 --- a/python/mxnet/gluon/data/dataloader.py +++ b/python/mxnet/gluon/data/dataloader.py @@ -26,6 +26,7 @@ import multiprocessing import multiprocessing.queues from multiprocessing.reduction import ForkingPickler +from multiprocessing.pool import ThreadPool import threading import numpy as np @@ -36,7 +37,6 @@ from . import sampler as _sampler from ... import nd, context -from ...recordio import MXRecordIO if sys.platform == 'darwin' or sys.platform == 'win32': def rebuild_ndarray(*args): @@ -159,29 +159,9 @@ def _as_in_context(data, ctx): return [_as_in_context(d, ctx) for d in data] return data -def _recursive_fork_recordio(obj, depth, max_depth=1000): - """Recursively find instance of MXRecordIO and reset file handler. - This is required for MXRecordIO which holds a C pointer to a opened file after fork. - """ - if depth >= max_depth: - return - if isinstance(obj, MXRecordIO): - obj.close() - obj.open() # re-obtain file hanlder in new process - elif (hasattr(obj, '__dict__')): - for _, v in obj.__dict__.items(): - _recursive_fork_recordio(v, depth + 1, max_depth) - -def worker_loop(dataset, key_queue, data_queue, batchify_fn): - """Worker loop for multiprocessing DataLoader.""" - # re-fork a new recordio handler in new process if applicable - # for a dataset with transform function, the depth of MXRecordIO is 1 - # for a lazy transformer, the depth is 2 - # for a user defined transformer, the depth is unknown, try a reasonable depth - limit = sys.getrecursionlimit() - max_recursion_depth = min(limit - 5, max(10, limit // 2)) - _recursive_fork_recordio(dataset, 0, max_recursion_depth) +def worker_loop_v1(dataset, key_queue, data_queue, batchify_fn): + """Worker loop for multiprocessing DataLoader.""" while True: idx, samples = key_queue.get() if idx is None: @@ -189,7 +169,7 @@ def worker_loop(dataset, key_queue, data_queue, batchify_fn): batch = batchify_fn([dataset[i] for i in samples]) data_queue.put((idx, batch)) -def fetcher_loop(data_queue, data_buffer, pin_memory=False): +def fetcher_loop_v1(data_queue, data_buffer, pin_memory=False, data_buffer_lock=None): """Fetcher loop for fetching data from queue and put in reorder dict.""" while True: idx, batch = data_queue.get() @@ -199,13 +179,17 @@ def fetcher_loop(data_queue, data_buffer, pin_memory=False): batch = _as_in_context(batch, context.cpu_pinned()) else: batch = _as_in_context(batch, context.cpu()) - data_buffer[idx] = batch + if data_buffer_lock is not None: + with data_buffer_lock: + data_buffer[idx] = batch + else: + data_buffer[idx] = batch -class _MultiWorkerIter(object): - """Interal multi-worker iterator for DataLoader.""" +class _MultiWorkerIterV1(object): + """Internal multi-worker iterator for DataLoader.""" def __init__(self, num_workers, dataset, batchify_fn, batch_sampler, pin_memory=False, - worker_fn=worker_loop): + worker_fn=worker_loop_v1): assert num_workers > 0, "_MultiWorkerIter is not for {} workers".format(num_workers) self._num_workers = num_workers self._dataset = dataset @@ -213,7 +197,10 @@ def __init__(self, num_workers, dataset, batchify_fn, batch_sampler, pin_memory= self._batch_sampler = batch_sampler self._key_queue = Queue() self._data_queue = Queue() if sys.version_info[0] <= 2 else SimpleQueue() + self._data_buffer = {} + self._data_buffer_lock = threading.Lock() + self._rcvd_idx = 0 self._sent_idx = 0 self._iter = iter(self._batch_sampler) @@ -227,10 +214,11 @@ def __init__(self, num_workers, dataset, batchify_fn, batch_sampler, pin_memory= worker.daemon = True worker.start() workers.append(worker) + self._workers = workers self._fetcher = threading.Thread( - target=fetcher_loop, - args=(self._data_queue, self._data_buffer, pin_memory)) + target=fetcher_loop_v1, + args=(self._data_queue, self._data_buffer, pin_memory, self._data_buffer_lock)) self._fetcher.daemon = True self._fetcher.start() @@ -261,7 +249,8 @@ def __next__(self): while True: if self._rcvd_idx in self._data_buffer: - batch = self._data_buffer.pop(self._rcvd_idx) + with self._data_buffer_lock: + batch = self._data_buffer.pop(self._rcvd_idx) self._rcvd_idx += 1 self._push_next() return batch @@ -275,13 +264,22 @@ def __iter__(self): def shutdown(self): """Shutdown internal workers by pushing terminate signals.""" if not self._shutdown: + # send shutdown signal to the fetcher and join data queue first + # Remark: loop_fetcher need to be joined prior to the workers. + # otherwise, the the fetcher may fail at getting data + self._data_queue.put((None, None)) + self._fetcher.join() + # send shutdown signal to all worker processes for _ in range(self._num_workers): self._key_queue.put((None, None)) - self._data_queue.put((None, None)) + # force shut down any alive worker processes + for w in self._workers: + if w.is_alive(): + w.terminate() self._shutdown = True -class DataLoader(object): +class DataLoaderV1(object): """Loads data from a dataset and returns mini-batches of data. Parameters @@ -372,8 +370,215 @@ def same_process_iter(): return same_process_iter() # multi-worker - return _MultiWorkerIter(self._num_workers, self._dataset, - self._batchify_fn, self._batch_sampler, self._pin_memory) + return _MultiWorkerIterV1(self._num_workers, self._dataset, + self._batchify_fn, self._batch_sampler, self._pin_memory) def __len__(self): return len(self._batch_sampler) + +_worker_dataset = None +def _worker_initializer(dataset): + """Initialier for processing pool.""" + # global dataset is per-process based and only available in worker processes + # this is only necessary to handle MXIndexedRecordIO because otherwise dataset + # can be passed as argument + global _worker_dataset + _worker_dataset = dataset + +def _worker_fn(samples, batchify_fn, dataset=None): + """Function for processing data in worker process.""" + # pylint: disable=unused-argument + # it is required that each worker process has to fork a new MXIndexedRecordIO handle + # preserving dataset as global variable can save tons of overhead and is safe in new process + global _worker_dataset + batch = batchify_fn([_worker_dataset[i] for i in samples]) + buf = io.BytesIO() + ForkingPickler(buf, pickle.HIGHEST_PROTOCOL).dump(batch) + return buf.getvalue() + +def _thread_worker_fn(samples, batchify_fn, dataset): + """Threadpool worker function for processing data.""" + return batchify_fn([dataset[i] for i in samples]) + +class _MultiWorkerIter(object): + """Internal multi-worker iterator for DataLoader.""" + def __init__(self, worker_pool, batchify_fn, batch_sampler, pin_memory=False, + worker_fn=_worker_fn, prefetch=0, dataset=None): + self._worker_pool = worker_pool + self._batchify_fn = batchify_fn + self._batch_sampler = batch_sampler + self._data_buffer = {} + self._rcvd_idx = 0 + self._sent_idx = 0 + self._iter = iter(self._batch_sampler) + self._worker_fn = worker_fn + self._pin_memory = pin_memory + self._dataset = dataset + # pre-fetch + for _ in range(prefetch): + self._push_next() + + def __len__(self): + return len(self._batch_sampler) + + def _push_next(self): + """Assign next batch workload to workers.""" + r = next(self._iter, None) + if r is None: + return + async_ret = self._worker_pool.apply_async( + self._worker_fn, (r, self._batchify_fn, self._dataset)) + self._data_buffer[self._sent_idx] = async_ret + self._sent_idx += 1 + + def __next__(self): + self._push_next() + if self._rcvd_idx == self._sent_idx: + assert not self._data_buffer, "Data buffer should be empty at this moment" + raise StopIteration + + assert self._rcvd_idx < self._sent_idx, "rcvd_idx must be smaller than sent_idx" + assert self._rcvd_idx in self._data_buffer, "fatal error with _push_next, rcvd_idx missing" + ret = self._data_buffer.pop(self._rcvd_idx) + batch = pickle.loads(ret.get()) if self._dataset is None else ret.get() + if self._pin_memory: + batch = _as_in_context(batch, context.cpu_pinned()) + batch = batch[0] if len(batch) == 1 else batch + self._rcvd_idx += 1 + return batch + + def next(self): + return self.__next__() + + def __iter__(self): + return self + + +class DataLoader(object): + """Loads data from a dataset and returns mini-batches of data. + + Parameters + ---------- + dataset : Dataset + Source dataset. Note that numpy and mxnet arrays can be directly used + as a Dataset. + batch_size : int + Size of mini-batch. + shuffle : bool + Whether to shuffle the samples. + sampler : Sampler + The sampler to use. Either specify sampler or shuffle, not both. + last_batch : {'keep', 'discard', 'rollover'} + How to handle the last batch if batch_size does not evenly divide + `len(dataset)`. + + keep - A batch with less samples than previous batches is returned. + discard - The last batch is discarded if its incomplete. + rollover - The remaining samples are rolled over to the next epoch. + batch_sampler : Sampler + A sampler that returns mini-batches. Do not specify batch_size, + shuffle, sampler, and last_batch if batch_sampler is specified. + batchify_fn : callable + Callback function to allow users to specify how to merge samples + into a batch. Defaults to `default_batchify_fn`:: + + def default_batchify_fn(data): + if isinstance(data[0], nd.NDArray): + return nd.stack(*data) + elif isinstance(data[0], tuple): + data = zip(*data) + return [default_batchify_fn(i) for i in data] + else: + data = np.asarray(data) + return nd.array(data, dtype=data.dtype) + + num_workers : int, default 0 + The number of multiprocessing workers to use for data preprocessing. + pin_memory : boolean, default False + If ``True``, the dataloader will copy NDArrays into pinned memory + before returning them. Copying from CPU pinned memory to GPU is faster + than from normal CPU memory. + prefetch : int, default is `num_workers * 2` + The number of prefetching batches only works if `num_workers` > 0. + If `prefetch` > 0, it allow worker process to prefetch certain batches before + acquiring data from iterators. + Note that using large prefetching batch will provide smoother bootstrapping performance, + but will consume more shared_memory. Using smaller number may forfeit the purpose of using + multiple worker processes, try reduce `num_workers` in this case. + By default it defaults to `num_workers * 2`. + thread_pool : bool, default False + If ``True``, use threading pool instead of multiprocessing pool. Using threadpool + can avoid shared memory usage. If `DataLoader` is more IO bounded or GIL is not a killing + problem, threadpool version may achieve better performance than multiprocessing. + + """ + def __init__(self, dataset, batch_size=None, shuffle=False, sampler=None, + last_batch=None, batch_sampler=None, batchify_fn=None, + num_workers=0, pin_memory=False, prefetch=None, thread_pool=False): + self._dataset = dataset + self._pin_memory = pin_memory + self._thread_pool = thread_pool + + if batch_sampler is None: + if batch_size is None: + raise ValueError("batch_size must be specified unless " \ + "batch_sampler is specified") + if sampler is None: + if shuffle: + sampler = _sampler.RandomSampler(len(dataset)) + else: + sampler = _sampler.SequentialSampler(len(dataset)) + elif shuffle: + raise ValueError("shuffle must not be specified if sampler is specified") + + batch_sampler = _sampler.BatchSampler( + sampler, batch_size, last_batch if last_batch else 'keep') + elif batch_size is not None or shuffle or sampler is not None or \ + last_batch is not None: + raise ValueError("batch_size, shuffle, sampler and last_batch must " \ + "not be specified if batch_sampler is specified.") + + self._batch_sampler = batch_sampler + self._num_workers = num_workers if num_workers >= 0 else 0 + self._worker_pool = None + self._prefetch = max(0, int(prefetch) if prefetch is not None else 2 * self._num_workers) + if self._num_workers > 0: + if self._thread_pool: + self._worker_pool = ThreadPool(self._num_workers) + else: + self._worker_pool = multiprocessing.Pool( + self._num_workers, initializer=_worker_initializer, initargs=[self._dataset]) + if batchify_fn is None: + if num_workers > 0: + self._batchify_fn = default_mp_batchify_fn + else: + self._batchify_fn = default_batchify_fn + else: + self._batchify_fn = batchify_fn + + def __iter__(self): + if self._num_workers == 0: + def same_process_iter(): + for batch in self._batch_sampler: + ret = self._batchify_fn([self._dataset[idx] for idx in batch]) + if self._pin_memory: + ret = _as_in_context(ret, context.cpu_pinned()) + yield ret + return same_process_iter() + + # multi-worker + return _MultiWorkerIter(self._worker_pool, self._batchify_fn, self._batch_sampler, + pin_memory=self._pin_memory, + worker_fn=_thread_worker_fn if self._thread_pool else _worker_fn, + prefetch=self._prefetch, + dataset=self._dataset if self._thread_pool else None) + + def __len__(self): + return len(self._batch_sampler) + + def __del__(self): + if self._worker_pool: + # manually terminate due to a bug that pool is not automatically terminated + # https://bugs.python.org/issue34172 + assert isinstance(self._worker_pool, multiprocessing.pool.Pool) + self._worker_pool.terminate() diff --git a/python/mxnet/gluon/data/dataset.py b/python/mxnet/gluon/data/dataset.py index c93a4b1cd..28d19c9fe 100644 --- a/python/mxnet/gluon/data/dataset.py +++ b/python/mxnet/gluon/data/dataset.py @@ -88,11 +88,7 @@ def transform_first(self, fn, lazy=True): Dataset The transformed dataset. """ - def base_fn(x, *args): - if args: - return (fn(x),) + args - return fn(x) - return self.transform(base_fn, lazy) + return self.transform(_TransformFirstClosure(fn), lazy) class SimpleDataset(Dataset): @@ -129,6 +125,16 @@ def __getitem__(self, idx): return self._fn(item) +class _TransformFirstClosure(object): + """Use callable object instead of nested function, it can be pickled.""" + def __init__(self, fn): + self._fn = fn + + def __call__(self, x, *args): + if args: + return (self._fn(x),) + args + return self._fn(x) + class ArrayDataset(Dataset): """A dataset that combines multiple dataset-like objects, e.g. Datasets, lists, arrays, etc. diff --git a/python/mxnet/gluon/data/vision/datasets.py b/python/mxnet/gluon/data/vision/datasets.py index 2c9800038..12ef7e16e 100644 --- a/python/mxnet/gluon/data/vision/datasets.py +++ b/python/mxnet/gluon/data/vision/datasets.py @@ -45,10 +45,9 @@ class MNIST(dataset._DownloadedDataset): train : bool, default True Whether to load the training or testing set. transform : function, default None - A user defined callback that transforms each sample. For example: - :: + A user defined callback that transforms each sample. For example:: - transform=lambda data, label: (data.astype(np.float32)/255, label) + transform=lambda data, label: (data.astype(np.float32)/255, label) """ def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'mnist'), @@ -106,10 +105,9 @@ class FashionMNIST(MNIST): train : bool, default True Whether to load the training or testing set. transform : function, default None - A user defined callback that transforms each sample. For example: - :: + A user defined callback that transforms each sample. For example:: - transform=lambda data, label: (data.astype(np.float32)/255, label) + transform=lambda data, label: (data.astype(np.float32)/255, label) """ def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'fashion-mnist'), @@ -130,7 +128,7 @@ def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'fashion-mnist class CIFAR10(dataset._DownloadedDataset): """CIFAR10 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html - Each sample is an image (in 3D NDArray) with shape (32, 32, 1). + Each sample is an image (in 3D NDArray) with shape (32, 32, 3). Parameters ---------- @@ -139,10 +137,9 @@ class CIFAR10(dataset._DownloadedDataset): train : bool, default True Whether to load the training or testing set. transform : function, default None - A user defined callback that transforms each sample. For example: - :: + A user defined callback that transforms each sample. For example:: - transform=lambda data, label: (data.astype(np.float32)/255, label) + transform=lambda data, label: (data.astype(np.float32)/255, label) """ def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar10'), @@ -193,7 +190,7 @@ def _get_data(self): class CIFAR100(CIFAR10): """CIFAR100 image classification dataset from https://www.cs.toronto.edu/~kriz/cifar.html - Each sample is an image (in 3D NDArray) with shape (32, 32, 1). + Each sample is an image (in 3D NDArray) with shape (32, 32, 3). Parameters ---------- @@ -204,10 +201,9 @@ class CIFAR100(CIFAR10): train : bool, default True Whether to load the training or testing set. transform : function, default None - A user defined callback that transforms each sample. For example: - :: + A user defined callback that transforms each sample. For example:: - transform=lambda data, label: (data.astype(np.float32)/255, label) + transform=lambda data, label: (data.astype(np.float32)/255, label) """ def __init__(self, root=os.path.join(base.data_dir(), 'datasets', 'cifar100'), @@ -238,14 +234,12 @@ class ImageRecordDataset(dataset.RecordFileDataset): filename : str Path to rec file. flag : {0, 1}, default 1 - If 0, always convert images to greyscale. - + If 0, always convert images to greyscale. \ If 1, always convert images to colored (RGB). transform : function, default None - A user defined callback that transforms each sample. For example: - :: + A user defined callback that transforms each sample. For example:: - transform=lambda data, label: (data.astype(np.float32)/255, label) + transform=lambda data, label: (data.astype(np.float32)/255, label) """ def __init__(self, filename, flag=1, transform=None): @@ -262,7 +256,9 @@ def __getitem__(self, idx): class ImageFolderDataset(dataset.Dataset): - """A dataset for loading image files stored in a folder structure like:: + """A dataset for loading image files stored in a folder structure. + + like:: root/car/0001.jpg root/car/xxxa.jpg @@ -279,10 +275,9 @@ class ImageFolderDataset(dataset.Dataset): If 0, always convert loaded images to greyscale (1 channel). If 1, always convert loaded images to colored (3 channels). transform : callable, default None - A function that takes data and label and transforms them: - :: + A function that takes data and label and transforms them:: - transform = lambda data, label: (data.astype(np.float32)/255, label) + transform = lambda data, label: (data.astype(np.float32)/255, label) Attributes ---------- diff --git a/python/mxnet/gluon/data/vision/transforms.py b/python/mxnet/gluon/data/vision/transforms.py index 3523be4d0..175076925 100644 --- a/python/mxnet/gluon/data/vision/transforms.py +++ b/python/mxnet/gluon/data/vision/transforms.py @@ -82,10 +82,10 @@ class Cast(HybridBlock): Inputs: - - **data**: input tensor with arbitrary shape. + - **data**: input tensor with arbitrary shape and dtype. Outputs: - - **out**: output tensor with the same shape as `data`. + - **out**: output tensor with the same shape as `data` and data type as dtype. """ def __init__(self, dtype='float32'): super(Cast, self).__init__() diff --git a/python/mxnet/gluon/loss.py b/python/mxnet/gluon/loss.py index 5d3ebb7ca..7b5832e1a 100644 --- a/python/mxnet/gluon/loss.py +++ b/python/mxnet/gluon/loss.py @@ -23,7 +23,7 @@ 'SigmoidBinaryCrossEntropyLoss', 'SigmoidBCELoss', 'SoftmaxCrossEntropyLoss', 'SoftmaxCELoss', 'KLDivLoss', 'CTCLoss', 'HuberLoss', 'HingeLoss', - 'SquaredHingeLoss', 'LogisticLoss', 'TripletLoss', 'PoissonNLLLoss'] + 'SquaredHingeLoss', 'LogisticLoss', 'TripletLoss', 'PoissonNLLLoss', 'CosineEmbeddingLoss'] import numpy as np from .. import ndarray @@ -621,7 +621,7 @@ class LogisticLoss(Loss): where `pred` is the classifier prediction and `label` is the target tensor containing values -1 or 1 (0 or 1 if `label_format` is binary). - `pred` and `label` can have arbitrary shape as long as they have the same number of elements. + `pred` and `label` can have arbitrary shape as long as they have the same number of elements. Parameters ---------- @@ -767,3 +767,71 @@ def hybrid_forward(self, F, pred, target, sample_weight=None, epsilon=1e-08): loss += stirling_factor loss = _apply_weighting(F, loss, self._weight, sample_weight) return F.mean(loss) + + +class CosineEmbeddingLoss(Loss): + r"""For a target label 1 or -1, vectors input1 and input2, the function computes the cosine distance + between the vectors. This can be interpreted as how similar/dissimilar two input vectors are. + + .. math:: + + L = \sum_i \begin{cases} 1 - {cos\_sim({input1}_i, {input2}_i)} & \text{ if } {label}_i = 1\\ + {cos\_sim({input1}_i, {input2}_i)} & \text{ if } {label}_i = -1 \end{cases}\\ + cos\_sim(input1, input2) = \frac{{input1}_i.{input2}_i}{||{input1}_i||.||{input2}_i||} + + `input1`, `input2` can have arbitrary shape as long as they have the same number of elements. + + Parameters + ---------- + weight : float or None + Global scalar weight for loss. + batch_axis : int, default 0 + The axis that represents mini-batch. + margin : float + Margin of separation between correct and incorrect pair. + + + Inputs: + - **input1**: a tensor with arbitrary shape + - **input2**: another tensor with same shape as pred to which input1 is + compared for similarity and loss calculation + - **label**: A 1-D tensor indicating for each pair input1 and input2, target label is 1 or -1 + - **sample_weight**: element-wise weighting tensor. Must be broadcastable + to the same shape as input1. For example, if input1 has shape (64, 10) + and you want to weigh each sample in the batch separately, + sample_weight should have shape (64, 1). + + Outputs: + - **loss**: The loss tensor with shape (batch_size,). + """ + def __init__(self, weight=None, batch_axis=0, margin=0, **kwargs): + super(CosineEmbeddingLoss, self).__init__(weight, batch_axis, **kwargs) + self._margin = margin + + def hybrid_forward(self, F, input1, input2, label, sample_weight=None): + input1 = _reshape_like(F, input1, input2) + label = label.reshape((-1, 1)) + cos_sim = self._cosine_similarity(F, input1, input2) + y_1 = label == 1 + y_minus_1 = label == -1 + cos_sim_a = (1 - cos_sim) * y_1 + + if F is ndarray: + z_array = F.array([0]) + else: + z_array = F.zeros((1, 1)) + cos_sim_b = F.broadcast_maximum(z_array, y_minus_1 * (cos_sim - self._margin), axis=1) + loss = cos_sim_a + cos_sim_b + loss = _apply_weighting(F, loss, self._weight, sample_weight) + return loss + + def _cosine_similarity(self, F, x, y, axis=-1): + # Calculates the cosine similarity between 2 vectors + x_norm = F.norm(x, axis=axis).reshape(-1, 1) + y_norm = F.norm(y, axis=axis).reshape(-1, 1) + x_dot_y = F.sum(x*y, axis=axis).reshape(-1, 1) + if F is ndarray: + eps_arr = F.array([1e-12]) + else: + eps_arr = F.full((1, 1), 1e-12) + return (x_dot_y / F.broadcast_maximum(x_norm * y_norm, eps_arr)) diff --git a/python/mxnet/gluon/model_zoo/vision/mobilenet.py b/python/mxnet/gluon/model_zoo/vision/mobilenet.py index 1a84e05af..886105712 100644 --- a/python/mxnet/gluon/model_zoo/vision/mobilenet.py +++ b/python/mxnet/gluon/model_zoo/vision/mobilenet.py @@ -62,7 +62,7 @@ def _add_conv_dw(out, dw_channels, channels, stride, relu6=False): class LinearBottleneck(nn.HybridBlock): r"""LinearBottleneck used in MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -138,7 +138,7 @@ def hybrid_forward(self, F, x): class MobileNetV2(nn.HybridBlock): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -223,7 +223,7 @@ def get_mobilenet_v2(multiplier, pretrained=False, ctx=cpu(), root=os.path.join(base.data_dir(), 'models'), **kwargs): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -269,7 +269,7 @@ def mobilenet1_0(**kwargs): def mobilenet_v2_1_0(**kwargs): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -300,7 +300,7 @@ def mobilenet0_75(**kwargs): def mobilenet_v2_0_75(**kwargs): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -331,7 +331,7 @@ def mobilenet0_5(**kwargs): def mobilenet_v2_0_5(**kwargs): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters @@ -362,7 +362,7 @@ def mobilenet0_25(**kwargs): def mobilenet_v2_0_25(**kwargs): r"""MobileNetV2 model from the `"Inverted Residuals and Linear Bottlenecks: - Mobile Networks for Classification, Detection and Segmentation" + Mobile Networks for Classification, Detection and Segmentation" `_ paper. Parameters diff --git a/python/mxnet/gluon/nn/basic_layers.py b/python/mxnet/gluon/nn/basic_layers.py index d26841977..4d514c283 100644 --- a/python/mxnet/gluon/nn/basic_layers.py +++ b/python/mxnet/gluon/nn/basic_layers.py @@ -75,7 +75,7 @@ def __len__(self): return len(self._children) def hybridize(self, active=True, **kwargs): - """Activates or deactivates `HybridBlock`s recursively. Has no effect on + """Activates or deactivates `HybridBlock` s recursively. Has no effect on non-hybrid children. Parameters @@ -160,9 +160,9 @@ class Dense(HybridBlock): Activation function to use. See help on `Activation` layer. If you don't specify anything, no activation is applied (ie. "linear" activation: `a(x) = x`). - use_bias : bool + use_bias : bool, default True Whether the layer uses a bias vector. - flatten: bool + flatten: bool, default True Whether the input tensor should be flattened. If true, all but the first axis of input data are collapsed together. If false, all but the last axis of input data are kept the same, and the transformation @@ -537,7 +537,7 @@ class LayerNorm(HybridBlock): .. math:: - out = \frac{x - mean[data, axis]}{ \sqrt{Var[data, axis]} + \epsilon} * gamma + beta + out = \frac{x - mean[data, axis]}{ \sqrt{Var[data, axis] + \epsilon}} * gamma + beta Parameters ---------- @@ -625,7 +625,7 @@ class Lambda(Block): block = Lambda('tanh') - 2) a function that conforms to "def function(*args)". For example:: + 2) a function that conforms to ``def function(*args)``. For example:: block = Lambda(lambda x: nd.LeakyReLU(x, slope=0.1)) @@ -663,20 +663,21 @@ class HybridLambda(HybridBlock): ---------- function : str or function Function used in lambda must be one of the following: - 1) the name of an operator that is available in both symbol and ndarray. For example:: + 1) The name of an operator that is available in both symbol and ndarray. For example:: block = HybridLambda('tanh') - 2) a function that conforms to "def function(F, data, *args)". For example:: + 2) A function that conforms to ``def function(F, data, *args)``. For example:: block = HybridLambda(lambda F, x: F.LeakyReLU(x, slope=0.1)) Inputs: - - ** *args **: one or more input data. First argument must be symbol or ndarray. - Their shapes depend on the function. + - ** *args **: one or more input data. First argument must be symbol or ndarray. Their \ + shapes depend on the function. Output: - ** *outputs **: one or more output data. Their shapes depend on the function. + """ def __init__(self, function, prefix=None): super(HybridLambda, self).__init__(prefix=prefix) diff --git a/python/mxnet/gluon/parameter.py b/python/mxnet/gluon/parameter.py index f53eeb006..2e130d498 100644 --- a/python/mxnet/gluon/parameter.py +++ b/python/mxnet/gluon/parameter.py @@ -580,7 +580,7 @@ class Constant(Parameter): will not change during training. But you can still update their values manually with the `set_data` method. - `Constant`s can be created with either:: + `Constant` s can be created with either:: const = mx.gluon.Constant('const', [[1,2],[3,4]]) @@ -740,9 +740,9 @@ def get(self, name, **kwargs): return param def get_constant(self, name, value=None): - """Retrieves a :py:class:`Constant` with name ``self.prefix+name``. If not found, + """Retrieves a :py:class:`.Constant` with name ``self.prefix+name``. If not found, :py:func:`get` will first try to retrieve it from "shared" dict. If still not - found, :py:func:`get` will create a new :py:class:`Constant` with key-word + found, :py:func:`get` will create a new :py:class:`.Constant` with key-word arguments and insert it to self. Parameters @@ -755,8 +755,8 @@ def get_constant(self, name, value=None): Returns ------- - Constant - The created or retrieved :py:class:`Constant`. + :py:class:`.Constant` + The created or retrieved :py:class:`.Constant`. """ name = self.prefix + name param = self._get_impl(name) diff --git a/python/mxnet/gluon/rnn/rnn_cell.py b/python/mxnet/gluon/rnn/rnn_cell.py index b57dc935a..6ef3604eb 100644 --- a/python/mxnet/gluon/rnn/rnn_cell.py +++ b/python/mxnet/gluon/rnn/rnn_cell.py @@ -102,6 +102,23 @@ def _mask_sequence_variable_length(F, data, length, valid_length, time_axis, mer squeeze_axis=True)) return outputs +def _reverse_sequences(sequences, unroll_step, valid_length=None): + if isinstance(sequences[0], symbol.Symbol): + F = symbol + else: + F = ndarray + + if valid_length is None: + reversed_sequences = list(reversed(sequences)) + else: + reversed_sequences = F.SequenceReverse(F.stack(*sequences, axis=0), + sequence_length=valid_length, + use_sequence_length=True) + reversed_sequences = F.split(reversed_sequences, axis=0, num_outputs=unroll_step, squeeze_axis=True) + + return reversed_sequences + + class RecurrentCell(Block): """Abstract base class for RNN cells @@ -333,7 +350,7 @@ class RNNCell(HybridRecurrentCell): Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. - prefix : str, default 'rnn_' + prefix : str, default ``'rnn_'`` Prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None @@ -440,7 +457,7 @@ class LSTMCell(HybridRecurrentCell): Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. - prefix : str, default 'lstm_' + prefix : str, default ``'lstm_'`` Prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None, default None @@ -565,7 +582,7 @@ class GRUCell(HybridRecurrentCell): Initializer for the bias vector. h2h_bias_initializer : str or Initializer, default 'zeros' Initializer for the bias vector. - prefix : str, default 'gru_' + prefix : str, default ``'gru_'`` prefix for name of `Block`s (and name of weight if params is `None`). params : Parameter or None, default None @@ -1035,14 +1052,7 @@ def unroll(self, length, inputs, begin_state=None, layout='NTC', merge_outputs=N self.reset() inputs, axis, F, batch_size = _format_sequence(length, inputs, layout, False) - if valid_length is None: - reversed_inputs = list(reversed(inputs)) - else: - reversed_inputs = F.SequenceReverse(F.stack(*inputs, axis=0), - sequence_length=valid_length, - use_sequence_length=True) - reversed_inputs = _as_list(F.split(reversed_inputs, axis=0, num_outputs=length, - squeeze_axis=True)) + reversed_inputs = list(_reverse_sequences(inputs, length, valid_length)) begin_state = _get_begin_state(self, F, begin_state, inputs, batch_size) states = begin_state @@ -1056,15 +1066,8 @@ def unroll(self, length, inputs, begin_state=None, layout='NTC', merge_outputs=N begin_state=states[len(l_cell.state_info(batch_size)):], layout=layout, merge_outputs=False, valid_length=valid_length) - if valid_length is None: - reversed_r_outputs = list(reversed(r_outputs)) - else: - reversed_r_outputs = F.SequenceReverse(F.stack(*r_outputs, axis=0), - sequence_length=valid_length, - use_sequence_length=True, - axis=0) - reversed_r_outputs = _as_list(F.split(reversed_r_outputs, axis=0, num_outputs=length, - squeeze_axis=True)) + reversed_r_outputs = _reverse_sequences(r_outputs, length, valid_length) + if merge_outputs is None: merge_outputs = isinstance(l_outputs, tensor_types) l_outputs, _, _, _ = _format_sequence(None, l_outputs, layout, merge_outputs) diff --git a/python/mxnet/gluon/rnn/rnn_layer.py b/python/mxnet/gluon/rnn/rnn_layer.py index e44b3600f..c43dc8527 100644 --- a/python/mxnet/gluon/rnn/rnn_layer.py +++ b/python/mxnet/gluon/rnn/rnn_layer.py @@ -35,11 +35,14 @@ def __init__(self, hidden_size, num_layers, layout, dropout, bidirectional, input_size, i2h_weight_initializer, h2h_weight_initializer, i2h_bias_initializer, h2h_bias_initializer, - mode, **kwargs): + mode, projection_size, h2r_weight_initializer, + lstm_state_clip_min, lstm_state_clip_max, lstm_state_clip_nan, + **kwargs): super(_RNNLayer, self).__init__(**kwargs) assert layout in ('TNC', 'NTC'), \ "Invalid layout %s; must be one of ['TNC' or 'NTC']"%layout self._hidden_size = hidden_size + self._projection_size = projection_size if projection_size else None self._num_layers = num_layers self._mode = mode self._layout = layout @@ -50,25 +53,50 @@ def __init__(self, hidden_size, num_layers, layout, self._h2h_weight_initializer = h2h_weight_initializer self._i2h_bias_initializer = i2h_bias_initializer self._h2h_bias_initializer = h2h_bias_initializer + self._h2r_weight_initializer = h2r_weight_initializer + self._lstm_state_clip_min = lstm_state_clip_min + self._lstm_state_clip_max = lstm_state_clip_max + self._lstm_state_clip_nan = lstm_state_clip_nan self._gates = {'rnn_relu': 1, 'rnn_tanh': 1, 'lstm': 4, 'gru': 3}[mode] ng, ni, nh = self._gates, input_size, hidden_size - for i in range(num_layers): - for j in ['l', 'r'][:self._dir]: - self._register_param('{}{}_i2h_weight'.format(j, i), - shape=(ng*nh, ni), - init=i2h_weight_initializer) - self._register_param('{}{}_h2h_weight'.format(j, i), - shape=(ng*nh, nh), - init=h2h_weight_initializer) - self._register_param('{}{}_i2h_bias'.format(j, i), - shape=(ng*nh,), - init=i2h_bias_initializer) - self._register_param('{}{}_h2h_bias'.format(j, i), - shape=(ng*nh,), - init=h2h_bias_initializer) - ni = nh * self._dir + if not projection_size: + for i in range(num_layers): + for j in ['l', 'r'][:self._dir]: + self._register_param('{}{}_i2h_weight'.format(j, i), + shape=(ng*nh, ni), + init=i2h_weight_initializer) + self._register_param('{}{}_h2h_weight'.format(j, i), + shape=(ng*nh, nh), + init=h2h_weight_initializer) + self._register_param('{}{}_i2h_bias'.format(j, i), + shape=(ng*nh,), + init=i2h_bias_initializer) + self._register_param('{}{}_h2h_bias'.format(j, i), + shape=(ng*nh,), + init=h2h_bias_initializer) + ni = nh * self._dir + else: + np = self._projection_size + for i in range(num_layers): + for j in ['l', 'r'][:self._dir]: + self._register_param('{}{}_i2h_weight'.format(j, i), + shape=(ng*nh, ni), + init=i2h_weight_initializer) + self._register_param('{}{}_h2h_weight'.format(j, i), + shape=(ng*nh, np), + init=h2h_weight_initializer) + self._register_param('{}{}_i2h_bias'.format(j, i), + shape=(ng*nh,), + init=i2h_bias_initializer) + self._register_param('{}{}_h2h_bias'.format(j, i), + shape=(ng*nh,), + init=h2h_bias_initializer) + self._register_param('{}{}_h2r_weight'.format(j, i), + shape=(np, nh), + init=h2r_weight_initializer) + ni = np * self._dir def _register_param(self, name, shape, init): p = self.params.get(name, shape=shape, init=init, @@ -114,6 +142,9 @@ def state_info(self, batch_size=0): def _unfuse(self): """Unfuses the fused RNN in to a stack of rnn cells.""" + assert not self._projection_size, "_unfuse does not support projection layer yet!" + assert not self._lstm_state_clip_min and not self._lstm_state_clip_max, \ + "_unfuse does not support state clipping yet!" get_cell = {'rnn_relu': lambda **kwargs: rnn_cell.RNNCell(self._hidden_size, activation='relu', **kwargs), @@ -189,7 +220,7 @@ def hybrid_forward(self, F, inputs, states=None, **kwargs): skip_states = states is None if skip_states: if F is ndarray: - states = self.begin_state(batch_size, ctx=inputs.context) + states = self.begin_state(batch_size, ctx=inputs.context, dtype=inputs.dtype) else: states = self.begin_state(0, func=symbol.zeros) if isinstance(states, tensor_types): @@ -209,16 +240,29 @@ def _forward_kernel(self, F, inputs, states, **kwargs): """ forward using CUDNN or CPU kenrel""" if self._layout == 'NTC': inputs = F.swapaxes(inputs, dim1=0, dim2=1) - params = (kwargs['{}{}_{}_{}'.format(d, l, g, t)].reshape(-1) - for t in ['weight', 'bias'] - for l in range(self._num_layers) - for d in ['l', 'r'][:self._dir] - for g in ['i2h', 'h2h']) + if self._projection_size is None: + params = (kwargs['{}{}_{}_{}'.format(d, l, g, t)].reshape(-1) + for t in ['weight', 'bias'] + for l in range(self._num_layers) + for d in ['l', 'r'][:self._dir] + for g in ['i2h', 'h2h']) + else: + params = (kwargs['{}{}_{}_{}'.format(d, l, g, t)].reshape(-1) + for t in ['weight', 'bias'] + for l in range(self._num_layers) + for d in ['l', 'r'][:self._dir] + for g in ['i2h', 'h2h', 'h2r'] + if g != 'h2r' or t != 'bias') + params = F._internal._rnn_param_concat(*params, dim=0) rnn = F.RNN(inputs, params, *states, state_size=self._hidden_size, + projection_size=self._projection_size, num_layers=self._num_layers, bidirectional=self._dir == 2, - p=self._dropout, state_outputs=True, mode=self._mode) + p=self._dropout, state_outputs=True, mode=self._mode, + lstm_state_clip_min=self._lstm_state_clip_min, + lstm_state_clip_max=self._lstm_state_clip_max, + lstm_state_clip_nan=self._lstm_state_clip_nan) if self._mode == 'lstm': outputs, states = rnn[0], [rnn[1], rnn[2]] @@ -318,7 +362,8 @@ def __init__(self, hidden_size, num_layers=1, activation='relu', dropout, bidirectional, input_size, i2h_weight_initializer, h2h_weight_initializer, i2h_bias_initializer, h2h_bias_initializer, - 'rnn_'+activation, **kwargs) + 'rnn_'+activation, None, None, None, None, False, + **kwargs) def state_info(self, batch_size=0): return [{'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), @@ -373,6 +418,20 @@ class LSTM(_RNNLayer): to zero. h2h_bias_initializer : str or Initializer Initializer for the bias vector. + projection_size: int, default None + The number of features after projection. + h2r_weight_initializer : str or Initializer, default None + Initializer for the projected recurrent weights matrix, used for the linear + transformation of the recurrent state to the projected space. + state_clip_min : float or None, default None + Minimum clip value of LSTM states. This option must be used together with + state_clip_max. If None, clipping is not applied. + state_clip_max : float or None, default None + Maximum clip value of LSTM states. This option must be used together with + state_clip_min. If None, clipping is not applied. + state_clip_nan : boolean, default False + Whether to stop NaN from propagating in state by clipping it to min/max. + If the clipping range is not specified, this option is ignored. input_size: int, default 0 The number of expected features in the input x. If not specified, it will be inferred from input. @@ -416,18 +475,28 @@ def __init__(self, hidden_size, num_layers=1, layout='TNC', dropout=0, bidirectional=False, input_size=0, i2h_weight_initializer=None, h2h_weight_initializer=None, i2h_bias_initializer='zeros', h2h_bias_initializer='zeros', + projection_size=None, h2r_weight_initializer=None, + state_clip_min=None, state_clip_max=None, state_clip_nan=False, **kwargs): super(LSTM, self).__init__(hidden_size, num_layers, layout, dropout, bidirectional, input_size, i2h_weight_initializer, h2h_weight_initializer, i2h_bias_initializer, h2h_bias_initializer, - 'lstm', **kwargs) + 'lstm', projection_size, h2r_weight_initializer, + state_clip_min, state_clip_max, state_clip_nan, + **kwargs) def state_info(self, batch_size=0): - return [{'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), - '__layout__': 'LNC'}, - {'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), - '__layout__': 'LNC'}] + if self._projection_size is None: + return [{'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), + '__layout__': 'LNC'}, + {'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), + '__layout__': 'LNC'}] + else: + return [{'shape': (self._num_layers * self._dir, batch_size, self._projection_size), + '__layout__': 'LNC'}, + {'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), + '__layout__': 'LNC'}] class GRU(_RNNLayer): @@ -522,7 +591,8 @@ def __init__(self, hidden_size, num_layers=1, layout='TNC', dropout, bidirectional, input_size, i2h_weight_initializer, h2h_weight_initializer, i2h_bias_initializer, h2h_bias_initializer, - 'gru', **kwargs) + 'gru', None, None, None, None, False, + **kwargs) def state_info(self, batch_size=0): return [{'shape': (self._num_layers * self._dir, batch_size, self._hidden_size), diff --git a/python/mxnet/gluon/trainer.py b/python/mxnet/gluon/trainer.py index 028e66075..f6c0a31b5 100644 --- a/python/mxnet/gluon/trainer.py +++ b/python/mxnet/gluon/trainer.py @@ -28,6 +28,15 @@ class Trainer(object): """Applies an `Optimizer` on a set of Parameters. Trainer should be used together with `autograd`. + .. note:: + + For the following cases, updates will always happen on kvstore, + i.e., you cannot set update_on_kvstore=False. + + - dist kvstore with sparse weights or sparse gradients + - dist async kvstore + - `optimizer.lr_scheduler` is not None + Parameters ---------- params : ParameterDict @@ -115,11 +124,12 @@ def _init_optimizer(self, optimizer, optimizer_params): "optimizer_params must be None if optimizer is an instance of " \ "Optimizer instead of str" self._optimizer = optimizer + # param_dict must not be deep copied, so that if user mutate the lr_mult + # or wd_mult of some parameters, it takes effect. self._optimizer.param_dict = param_dict else: self._optimizer = opt.create(optimizer, param_dict=param_dict, **optimizer_params) - self._updaters = [opt.get_updater(self._optimizer) \ for _ in self._contexts] @@ -158,55 +168,82 @@ def _reset_kvstore(self): def _init_kvstore(self): """Create kvstore.""" config = self._kvstore_params - # if weight is sparse, the weight must be updated on KVStore. - # training loop contains: - # - row_sparse_pull(sparse_weight) - # - forward() - # - backward() - # - push(sparse_grad), push(dense_grad) - # - pull(dense_weight) + # configure kvstore, update_on_kvstore and self._distributed on three cases: if self._contains_sparse_weight: + # If weight is sparse, kvstore must be present and the weight must be updated on kvstore. + # The training loop is the following: + # - row_sparse_pull(sparse_weight) + # - forward() + # - backward() + # - push_and_update(grad) + # - pull(weight) kvstore, update_on_kvstore = _create_sparse_kvstore(config['kvstore']) - # raise Error if update_on_kvstore is set to False by the user + self._distributed = 'dist' in kvstore.type + # raise err if user provides unsupported configs if config['update_on_kvstore'] is False: - raise RuntimeError("Cannot set update_on_kvstore to False when sparse weights " - "are present.") - # if weight is dense and grad is sparse, the weight better not be updated on KVStore. - # training loop contains: - # - forward() - # - backward() - # - push(grad) - # - pull(grad) - # - update(grad, weight) + raise ValueError("Cannot set update_on_kvstore=False when sparse weights " + "are present.") + elif self._contains_sparse_grad: + # For single node training with dense weight and sparse grad, + # we prefer update_on_kvstore=False because this is usually faster. + # This means we push and pull sparse gradients, and we do not store weight in kvstore. + # The training loop is the following: + # - forward() + # - backward() + # - push(grad) + # - pull(grad) + # - update(grad, weight) + # + # For multi-node training with dense weight and sparse grad, + # only update_on_kvstore=True is supported, due to the fact that + # kv.row_sparse_pull(grad) is not implemented. + # Therefore, we push sparse gradients and pull dense weights. + # The training loop contains: + # - forward() + # - backward() + # - push_and_update(grad) + # - pull(weight) arg_arrays = {param.name: param.data(self._contexts[0]) for param in self._params} kvstore, _ = _create_kvstore(config['kvstore'], len(self._contexts), arg_arrays) - update_on_kvstore = False - # normal case + self._distributed = 'dist' in kvstore.type if kvstore else False + update_on_kvstore = self._distributed + # raise err if user provides unsupported configs + if config['update_on_kvstore'] is not None: + if config['update_on_kvstore'] is False and self._distributed: + raise ValueError("Cannot set update_on_kvstore=False on dist kvstore " + "when sparse gradients are present.") + update_on_kvstore = config['update_on_kvstore'] + else: + # Training with dense weight and dense gradients. + # The only unsupported mode is async with update_on_kvstore=False arg_arrays = {param.name: param.data(self._contexts[0]) for param in self._params} kvstore, update_on_kvstore = _create_kvstore(config['kvstore'], len(self._contexts), arg_arrays) - if kvstore and 'async' in kvstore.type and config['update_on_kvstore'] is not None\ - and not config['update_on_kvstore']: - raise ValueError("Please set update_on_kvstore to true " - "when training in async mode.") - + self._distributed = 'dist' in kvstore.type if kvstore else False + if self._distributed and 'async' in kvstore.type: + update_on_kvstore = True + # raise err if user provides unsupported configs + if config['update_on_kvstore'] is False: + raise ValueError("Please set update_on_kvstore=True " + "when training in async mode.") if config['update_on_kvstore'] is not None: update_on_kvstore = config['update_on_kvstore'] + + # set grad compression and optimizers if kvstore: if self._compression_params: kvstore.set_gradient_compression(self._compression_params) - self._distributed = 'dist' in kvstore.type - if self._distributed: - # kv.pull(row_sparse_grad) is not supported for dist kvstore - update_on_kvstore = self._contains_sparse_weight or self._contains_sparse_grad \ - or 'async' in kvstore.type if update_on_kvstore: # optimizer preferably needs to be set before init for multiprecision kvstore.set_optimizer(self._optimizer) self._kvstore = kvstore self._update_on_kvstore = update_on_kvstore + if self._optimizer.lr_scheduler and not self._update_on_kvstore: + raise ValueError("update_on_kvstore=False does not support " \ + "optimizer with LRScheduler. Please " \ + "consider setting learning rate manually.") else: self._kvstore = None self._update_on_kvstore = None @@ -251,6 +288,16 @@ def _row_sparse_pull(self, parameter, out, row_id, full_idx=False): else: self._kvstore.row_sparse_pull(idx, out=out, row_ids=row_id, priority=-idx) + def _check_and_rescale_grad(self, scale): + if self._update_on_kvstore and self._distributed and self._kv_initialized: + if self._optimizer.rescale_grad != scale: + raise UserWarning('Possible change in the `batch_size` from previous ' + '`step` detected. Optimizer gradient normalizing ' + 'factor will not change w.r.t new batch_size when ' + 'update_on_kvstore=True and when distributed kvstore ' + 'is used.') + self._optimizer.rescale_grad = scale + def step(self, batch_size, ignore_stale_grad=False): """Makes one step of parameter update. Should be called after `autograd.backward()` and outside of `record()` scope. @@ -269,13 +316,14 @@ def step(self, batch_size, ignore_stale_grad=False): If true, ignores Parameters with stale gradient (gradient that has not been updated by `backward` after last step) and skip update. """ + rescale_grad = self._scale / batch_size + self._check_and_rescale_grad(rescale_grad) + if not self._kv_initialized: self._init_kvstore() if self._params_to_init: self._init_params() - self._optimizer.rescale_grad = self._scale / batch_size - self._allreduce_grads() self._update(ignore_stale_grad) @@ -341,7 +389,7 @@ def update(self, batch_size, ignore_stale_grad=False): 'is not supported. Try setting `update_on_kvstore` ' \ 'to False when creating trainer.' - self._optimizer.rescale_grad = self._scale / batch_size + self._check_and_rescale_grad(self._scale / batch_size) self._update(ignore_stale_grad) def _update(self, ignore_stale_grad=False): @@ -376,10 +424,16 @@ def _update(self, ignore_stale_grad=False): def save_states(self, fname): """Saves trainer states (e.g. optimizer, momentum) to a file. + Parameters ---------- fname : str Path to output states file. + + Note + ---- + `optimizer.param_dict`, which contains Parameter information (such as + `lr_mult` and `wd_mult`) will not be saved. """ assert self._optimizer is not None @@ -403,6 +457,12 @@ def load_states(self, fname): ---------- fname : str Path to input states file. + + Note + ---- + `optimizer.param_dict`, which contains Parameter information (such as + `lr_mult` and `wd_mult`) will not be loaded from the file, but rather set + based on current Trainer's parameters. """ if not self._kv_initialized: self._init_kvstore() @@ -412,8 +472,6 @@ def load_states(self, fname): if self._update_on_kvstore: self._kvstore.load_optimizer_states(fname) self._optimizer = self._kvstore._updater.optimizer - param_dict = {i: param for i, param in enumerate(self._params)} - self._optimizer.param_dict = param_dict else: with open(fname, 'rb') as f: states = f.read() @@ -421,3 +479,5 @@ def load_states(self, fname): updater.set_states(states) updater.optimizer = self._updaters[0].optimizer self._optimizer = self._updaters[0].optimizer + param_dict = {i: param for i, param in enumerate(self._params)} + self._optimizer.param_dict = param_dict diff --git a/python/mxnet/image/detection.py b/python/mxnet/image/detection.py index 3b9f64e12..d5b5ecab5 100644 --- a/python/mxnet/image/detection.py +++ b/python/mxnet/image/detection.py @@ -658,19 +658,26 @@ class ImageDetIter(ImageIter): Data name for provided symbols. label_name : str Name for detection labels + last_batch_handle : str, optional + How to handle the last batch. + This parameter can be 'pad'(default), 'discard' or 'roll_over'. + If 'pad', the last batch will be padded with data starting from the begining + If 'discard', the last batch will be discarded + If 'roll_over', the remaining elements will be rolled over to the next iteration kwargs : ... More arguments for creating augmenter. See mx.image.CreateDetAugmenter. """ def __init__(self, batch_size, data_shape, path_imgrec=None, path_imglist=None, path_root=None, path_imgidx=None, shuffle=False, part_index=0, num_parts=1, aug_list=None, imglist=None, - data_name='data', label_name='label', **kwargs): + data_name='data', label_name='label', last_batch_handle='pad', **kwargs): super(ImageDetIter, self).__init__(batch_size=batch_size, data_shape=data_shape, path_imgrec=path_imgrec, path_imglist=path_imglist, path_root=path_root, path_imgidx=path_imgidx, shuffle=shuffle, part_index=part_index, num_parts=num_parts, aug_list=[], imglist=imglist, - data_name=data_name, label_name=label_name) + data_name=data_name, label_name=label_name, + last_batch_handle=last_batch_handle) if aug_list is None: self.auglist = CreateDetAugmenter(data_shape, **kwargs) @@ -745,18 +752,16 @@ def reshape(self, data_shape=None, label_shape=None): if data_shape is not None: self.check_data_shape(data_shape) self.provide_data = [(self.provide_data[0][0], (self.batch_size,) + data_shape)] + self.data_shape = data_shape if label_shape is not None: self.check_label_shape(label_shape) self.provide_label = [(self.provide_label[0][0], (self.batch_size,) + label_shape)] + self.label_shape = label_shape - def next(self): - """Override the function for returning next batch.""" + def _batchify(self, batch_data, batch_label, start=0): + """Override the helper function for batchifying data""" + i = start batch_size = self.batch_size - c, h, w = self.data_shape - batch_data = nd.zeros((batch_size, c, h, w)) - batch_label = nd.empty(self.provide_label[0][1]) - batch_label[:] = -1 - i = 0 try: while i < batch_size: label, s = self.next_sample() @@ -781,7 +786,48 @@ def next(self): if not i: raise StopIteration - return io.DataBatch([batch_data], [batch_label], batch_size - i) + return i + + def next(self): + """Override the function for returning next batch.""" + batch_size = self.batch_size + c, h, w = self.data_shape + # if last batch data is rolled over + if self._cache_data is not None: + # check both the data and label have values + assert self._cache_label is not None, "_cache_label didn't have values" + assert self._cache_idx is not None, "_cache_idx didn't have values" + batch_data = self._cache_data + batch_label = self._cache_label + i = self._cache_idx + else: + batch_data = nd.zeros((batch_size, c, h, w)) + batch_label = nd.empty(self.provide_label[0][1]) + batch_label[:] = -1 + i = self._batchify(batch_data, batch_label) + # calculate the padding + pad = batch_size - i + # handle padding for the last batch + if pad != 0: + if self.last_batch_handle == 'discard': + raise StopIteration + # if the option is 'roll_over', throw StopIteration and cache the data + elif self.last_batch_handle == 'roll_over' and \ + self._cache_data is None: + self._cache_data = batch_data + self._cache_label = batch_label + self._cache_idx = i + raise StopIteration + else: + _ = self._batchify(batch_data, batch_label, i) + if self.last_batch_handle == 'pad': + self._allow_read = False + else: + self._cache_data = None + self._cache_label = None + self._cache_idx = None + + return io.DataBatch([batch_data], [batch_label], pad=pad) def augmentation_transform(self, data, label): # pylint: disable=arguments-differ """Override Transforms input data with specified augmentations.""" diff --git a/python/mxnet/image/image.py b/python/mxnet/image/image.py index eee2ccf14..1dd665607 100644 --- a/python/mxnet/image/image.py +++ b/python/mxnet/image/image.py @@ -38,8 +38,6 @@ from ..base import numeric_types from .. import ndarray as nd from ..ndarray import _internal -from ..ndarray._internal import _cvimresize as imresize -from ..ndarray._internal import _cvcopyMakeBorder as copyMakeBorder from .. import io from .. import recordio @@ -47,8 +45,8 @@ def imread(filename, *args, **kwargs): """Read and decode an image to an NDArray. - Note: `imread` uses OpenCV (not the CV2 Python library). - MXNet must have been built with USE_OPENCV=1 for `imdecode` to work. + .. note:: `imread` uses OpenCV (not the CV2 Python library). + MXNet must have been built with USE_OPENCV=1 for `imdecode` to work. Parameters ---------- @@ -85,15 +83,72 @@ def imread(filename, *args, **kwargs): return _internal._cvimread(filename, *args, **kwargs) +def imresize(src, w, h, *args, **kwargs): + r"""Resize image with OpenCV. + + .. note:: `imresize` uses OpenCV (not the CV2 Python library). MXNet must have been built + with USE_OPENCV=1 for `imresize` to work. + + Parameters + ---------- + src : NDArray + source image + w : int, required + Width of resized image. + h : int, required + Height of resized image. + interp : int, optional, default=1 + Interpolation method (default=cv2.INTER_LINEAR). + Possible values: + 0: Nearest Neighbors Interpolation. + 1: Bilinear interpolation. + 2: Area-based (resampling using pixel area relation). It may be a + preferred method for image decimation, as it gives moire-free + results. But when the image is zoomed, it is similar to the Nearest + Neighbors method. (used by default). + 3: Bicubic interpolation over 4x4 pixel neighborhood. + 4: Lanczos interpolation over 8x8 pixel neighborhood. + 9: Cubic for enlarge, area for shrink, bilinear for others + 10: Random select from interpolation method metioned above. + Note: + When shrinking an image, it will generally look best with AREA-based + interpolation, whereas, when enlarging an image, it will generally look best + with Bicubic (slow) or Bilinear (faster but still looks OK). + More details can be found in the documentation of OpenCV, please refer to + http://docs.opencv.org/master/da/d54/group__imgproc__transform.html. + + out : NDArray, optional + The output NDArray to hold the result. + + Returns + ------- + out : NDArray or list of NDArrays + The output of this function. + + Example + ------- + >>> with open("flower.jpeg", 'rb') as fp: + ... str_image = fp.read() + ... + >>> image = mx.img.imdecode(str_image) + >>> image + + >>> new_image = mx.img.resize(image, 240, 360) + >>> new_image + + """ + return _internal._cvimresize(src, w, h, *args, **kwargs) + + def imdecode(buf, *args, **kwargs): """Decode an image to an NDArray. - Note: `imdecode` uses OpenCV (not the CV2 Python library). - MXNet must have been built with USE_OPENCV=1 for `imdecode` to work. + .. note:: `imdecode` uses OpenCV (not the CV2 Python library). + MXNet must have been built with USE_OPENCV=1 for `imdecode` to work. Parameters ---------- - buf : str/bytes or numpy.ndarray + buf : str/bytes/bytearray or numpy.ndarray Binary image data as string or numpy ndarray. flag : int, optional, default=1 1 for three channel color output. 0 for grayscale output. @@ -135,10 +190,11 @@ def imdecode(buf, *args, **kwargs): """ if not isinstance(buf, nd.NDArray): - if sys.version_info[0] == 3 and not isinstance(buf, (bytes, np.ndarray)): - raise ValueError('buf must be of type bytes or numpy.ndarray,' + if sys.version_info[0] == 3 and not isinstance(buf, (bytes, bytearray, np.ndarray)): + raise ValueError('buf must be of type bytes, bytearray or numpy.ndarray,' 'if you would like to input type str, please convert to bytes') buf = nd.array(np.frombuffer(buf, dtype=np.uint8), dtype=np.uint8) + return _internal._cvimdecode(buf, *args, **kwargs) @@ -177,6 +233,59 @@ def scale_down(src_size, size): return int(w), int(h) +def copyMakeBorder(src, top, bot, left, right, *args, **kwargs): + """Pad image border with OpenCV. + + Parameters + ---------- + src : NDArray + source image + top : int, required + Top margin. + bot : int, required + Bottom margin. + left : int, required + Left margin. + right : int, required + Right margin. + type : int, optional, default='0' + Filling type (default=cv2.BORDER_CONSTANT). + 0 - cv2.BORDER_CONSTANT - Adds a constant colored border. + 1 - cv2.BORDER_REFLECT - Border will be mirror reflection of the + border elements, like this : fedcba|abcdefgh|hgfedcb + 2 - cv2.BORDER_REFLECT_101 or cv.BORDER_DEFAULT - Same as above, + but with a slight change, like this : gfedcb|abcdefgh|gfedcba + 3 - cv2.BORDER_REPLICATE - Last element is replicated throughout, + like this: aaaaaa|abcdefgh|hhhhhhh + 4 - cv2.BORDER_WRAP - it will look like this : cdefgh|abcdefgh|abcdefg + value : double, optional, default=0 + (Deprecated! Use ``values`` instead.) Fill with single value. + values : tuple of , optional, default=[] + Fill with value(RGB[A] or gray), up to 4 channels. + + out : NDArray, optional + The output NDArray to hold the result. + + Returns + ------- + out : NDArray or list of NDArrays + The output of this function. + + Example + -------- + >>> with open("flower.jpeg", 'rb') as fp: + ... str_image = fp.read() + ... + >>> image = mx.img.imdecode(str_image) + >>> image + + >>> new_image = mx_border = mx.image.copyMakeBorder(mx_img, 1, 2, 3, 4, type=0) + >>> new_image + + """ + return _internal._cvcopyMakeBorder(src, top, bot, left, right, *args, **kwargs) + + def _get_interp_method(interp, sizes=()): """Get the interpolation method for resize functions. The major purpose of this function is to wrap a random interp method selection @@ -235,8 +344,8 @@ def _get_interp_method(interp, sizes=()): def resize_short(src, size, interp=2): """Resizes shorter edge to size. - Note: `resize_short` uses OpenCV (not the CV2 Python library). - MXNet must have been built with OpenCV for `resize_short` to work. + .. note:: `resize_short` uses OpenCV (not the CV2 Python library). + MXNet must have been built with OpenCV for `resize_short` to work. Resizes the original image by setting the shorter edge to size and setting the longer edge accordingly. @@ -1144,7 +1253,7 @@ def __init__(self, batch_size, data_shape, label_width=1, self.shuffle = shuffle if self.imgrec is None: self.seq = imgkeys - elif shuffle or num_parts > 1: + elif shuffle or num_parts > 1 or path_imgidx: assert self.imgidx is not None self.seq = self.imgidx else: @@ -1260,7 +1369,7 @@ def next(self): i = self._cache_idx # clear the cache data else: - batch_data = nd.empty((batch_size, c, h, w)) + batch_data = nd.zeros((batch_size, c, h, w)) batch_label = nd.empty(self.provide_label[0][1]) i = self._batchify(batch_data, batch_label) # calculate the padding @@ -1284,6 +1393,7 @@ def next(self): self._cache_data = None self._cache_label = None self._cache_idx = None + return io.DataBatch([batch_data], [batch_label], pad=pad) def check_data_shape(self, data_shape): @@ -1321,8 +1431,8 @@ def locate(): def read_image(self, fname): """Reads an input image `fname` and returns the decoded raw bytes. - Example usage: - ---------- + Examples + -------- >>> dataIter.read_image('Face.jpg') # returns decoded raw bytes. """ with open(os.path.join(self.path_root, fname), 'rb') as fin: diff --git a/python/mxnet/initializer.py b/python/mxnet/initializer.py index 357e75b3b..b67ab624a 100755 --- a/python/mxnet/initializer.py +++ b/python/mxnet/initializer.py @@ -32,10 +32,11 @@ # inherit str for backward compatibility class InitDesc(str): - """Descriptor for the initialization pattern. + """ + Descriptor for the initialization pattern. - Parameter - --------- + Parameters + ---------- name : str Name of variable. attrs : dict of str to str @@ -67,7 +68,7 @@ def set_verbosity(self, verbose=False, print_func=None): print_func : function A function that computes statistics of initialized arrays. Takes an `NDArray` and returns an `str`. Defaults to mean - absolute value str((|x|/size(x)).asscalar()). + absolute value str((abs(x)/size(x)).asscalar()). """ self._verbose = verbose if print_func is None: diff --git a/python/mxnet/io/io.py b/python/mxnet/io/io.py index 20da2ea7c..2bd1d6115 100644 --- a/python/mxnet/io/io.py +++ b/python/mxnet/io/io.py @@ -102,8 +102,8 @@ def get_list(shapes, types): Parameters ---------- - shapes : a tuple of (name, shape) - types : a tuple of (name, type) + shapes : a tuple of (name_, shape_) + types : a tuple of (name_, np.dtype) """ if types is not None: type_dict = dict(types) @@ -490,8 +490,8 @@ class NDArrayIter(DataIter): """Returns an iterator for ``mx.nd.NDArray``, ``numpy.ndarray``, ``h5py.Dataset`` ``mx.nd.sparse.CSRNDArray`` or ``scipy.sparse.csr_matrix``. - Example usage: - ---------- + Examples + -------- >>> data = np.arange(40).reshape((10,2,2)) >>> labels = np.ones([10, 1]) >>> dataiter = mx.io.NDArrayIter(data, labels, 3, True, last_batch_handle='discard') diff --git a/python/mxnet/libinfo.py b/python/mxnet/libinfo.py index bcff69b19..6cc21e435 100644 --- a/python/mxnet/libinfo.py +++ b/python/mxnet/libinfo.py @@ -145,6 +145,39 @@ def find_lib_path(): return lib_path +def find_include_path(): + """Find MXNet included header files. + + Returns + ------- + incl_path : string + Path to the header files. + """ + incl_from_env = os.environ.get('MXNET_INCLUDE_PATH') + if incl_from_env: + if os.path.isdir(incl_from_env): + if not os.path.isabs(incl_from_env): + logging.warning("MXNET_INCLUDE_PATH should be an absolute path, instead of: %s", + incl_from_env) + else: + return incl_from_env + else: + logging.warning("MXNET_INCLUDE_PATH '%s' doesn't exist", incl_from_env) + + curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) + # include path in pip package + pip_incl_path = os.path.join(curr_path, 'include/') + if os.path.isdir(pip_incl_path): + return pip_incl_path + else: + # include path if build from source + src_incl_path = os.path.join(curr_path, '../../include/') + if os.path.isdir(src_incl_path): + return src_incl_path + else: + raise RuntimeError('Cannot find the MXNet include path in either ' + pip_incl_path + + ' or ' + src_incl_path + '\n') + # current version -__version__ = "1.3.1" +__version__ = "1.5.0" diff --git a/python/mxnet/metric.py b/python/mxnet/metric.py index 80ea26b5f..ecb8e1c3b 100644 --- a/python/mxnet/metric.py +++ b/python/mxnet/metric.py @@ -90,6 +90,7 @@ def __init__(self, name, output_names=None, self.name = str(name) self.output_names = output_names self.label_names = label_names + self._has_global_stats = kwargs.pop("has_global_stats", False) self._kwargs = kwargs self.reset() @@ -98,7 +99,7 @@ def __str__(self): def get_config(self): """Save configurations of metric. Can be recreated - from configs with metric.create(**config) + from configs with metric.create(``**config``) """ config = self._kwargs.copy() config.update({ @@ -148,6 +149,14 @@ def reset(self): """Resets the internal evaluation result to initial state.""" self.num_inst = 0 self.sum_metric = 0.0 + self.global_num_inst = 0 + self.global_sum_metric = 0.0 + + def reset_local(self): + """Resets the local portion of the internal evaluation results + to initial state.""" + self.num_inst = 0 + self.sum_metric = 0.0 def get(self): """Gets the current evaluation result. @@ -164,6 +173,24 @@ def get(self): else: return (self.name, self.sum_metric / self.num_inst) + def get_global(self): + """Gets the current global evaluation result. + + Returns + ------- + names : list of str + Name of the metrics. + values : list of float + Value of the evaluations. + """ + if self._has_global_stats: + if self.global_num_inst == 0: + return (self.name, float('nan')) + else: + return (self.name, self.global_sum_metric / self.global_num_inst) + else: + return self.get() + def get_name_value(self): """Returns zipped name and value pairs. @@ -179,6 +206,24 @@ def get_name_value(self): value = [value] return list(zip(name, value)) + def get_global_name_value(self): + """Returns zipped name and value pairs for global results. + + Returns + ------- + list of tuples + A (name, value) tuple list. + """ + if self._has_global_stats: + name, value = self.get_global() + if not isinstance(name, list): + name = [name] + if not isinstance(value, list): + value = [value] + return list(zip(name, value)) + else: + return self.get_name_value() + # pylint: disable=invalid-name register = registry.get_register_func(EvalMetric, 'metric') alias = registry.get_alias_func(EvalMetric, 'metric') @@ -263,7 +308,8 @@ class CompositeEvalMetric(EvalMetric): def __init__(self, metrics=None, name='composite', output_names=None, label_names=None): super(CompositeEvalMetric, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) if metrics is None: metrics = [] self.metrics = [create(i) for i in metrics] @@ -325,6 +371,15 @@ def reset(self): except AttributeError: pass + def reset_local(self): + """Resets the local portion of the internal evaluation results + to initial state.""" + try: + for metric in self.metrics: + metric.reset_local() + except AttributeError: + pass + def get(self): """Returns the current evaluation result. @@ -347,6 +402,28 @@ def get(self): values.extend(value) return (names, values) + def get_global(self): + """Returns the current evaluation result. + + Returns + ------- + names : list of str + Name of the metrics. + values : list of float + Value of the evaluations. + """ + names = [] + values = [] + for metric in self.metrics: + name, value = metric.get_global() + if isinstance(name, string_types): + name = [name] + if isinstance(value, numeric_types): + value = [value] + names.extend(name) + values.extend(value) + return (names, values) + def get_config(self): config = super(CompositeEvalMetric, self).get_config() config.update({'metrics': [i.get_config() for i in self.metrics]}) @@ -395,7 +472,8 @@ def __init__(self, axis=1, name='accuracy', output_names=None, label_names=None): super(Accuracy, self).__init__( name, axis=axis, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self.axis = axis def update(self, labels, preds): @@ -423,8 +501,11 @@ def update(self, labels, preds): check_label_shapes(label, pred_label) - self.sum_metric += (pred_label == label).sum() + num_correct = (pred_label == label).sum() + self.sum_metric += num_correct + self.global_sum_metric += num_correct self.num_inst += len(pred_label) + self.global_num_inst += len(pred_label) @register @@ -467,7 +548,8 @@ def __init__(self, top_k=1, name='top_k_accuracy', output_names=None, label_names=None): super(TopKAccuracy, self).__init__( name, top_k=top_k, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self.top_k = top_k assert(self.top_k > 1), 'Please use Accuracy if top_k is no more than 1' self.name += '_%d' % self.top_k @@ -487,7 +569,11 @@ def update(self, labels, preds): for label, pred_label in zip(labels, preds): assert(len(pred_label.shape) <= 2), 'Predictions should be no more than 2 dims' - pred_label = numpy.argsort(pred_label.asnumpy().astype('float32'), axis=1) + # Using argpartition here instead of argsort is safe because + # we do not care about the order of top k elements. It is + # much faster, which is important since that computation is + # single-threaded due to Python GIL. + pred_label = numpy.argpartition(pred_label.asnumpy().astype('float32'), -self.top_k) label = label.asnumpy().astype('int32') check_label_shapes(label, pred_label) num_samples = pred_label.shape[0] @@ -498,8 +584,11 @@ def update(self, labels, preds): num_classes = pred_label.shape[1] top_k = min(num_classes, self.top_k) for j in range(top_k): - self.sum_metric += (pred_label[:, num_classes - 1 - j].flat == label.flat).sum() + num_correct = (pred_label[:, num_classes - 1 - j].flat == label.flat).sum() + self.sum_metric += num_correct + self.global_sum_metric += num_correct self.num_inst += num_samples + self.global_num_inst += num_samples class _BinaryClassificationMetrics(object): @@ -515,6 +604,10 @@ def __init__(self): self.false_negatives = 0 self.false_positives = 0 self.true_negatives = 0 + self.global_true_positives = 0 + self.global_false_negatives = 0 + self.global_false_positives = 0 + self.global_true_negatives = 0 def update_binary_stats(self, label, pred): """ @@ -542,10 +635,18 @@ def update_binary_stats(self, label, pred): label_true = (label == 1) label_false = 1 - label_true - self.true_positives += (pred_true * label_true).sum() - self.false_positives += (pred_true * label_false).sum() - self.false_negatives += (pred_false * label_true).sum() - self.true_negatives += (pred_false * label_false).sum() + true_pos = (pred_true * label_true).sum() + false_pos = (pred_true * label_false).sum() + false_neg = (pred_false * label_true).sum() + true_neg = (pred_false * label_false).sum() + self.true_positives += true_pos + self.global_true_positives += true_pos + self.false_positives += false_pos + self.global_false_positives += false_pos + self.false_negatives += false_neg + self.global_false_negatives += false_neg + self.true_negatives += true_neg + self.global_true_negatives += true_neg @property def precision(self): @@ -554,6 +655,13 @@ def precision(self): else: return 0. + @property + def global_precision(self): + if self.global_true_positives + self.global_false_positives > 0: + return float(self.global_true_positives) / (self.global_true_positives + self.global_false_positives) + else: + return 0. + @property def recall(self): if self.true_positives + self.false_negatives > 0: @@ -561,6 +669,13 @@ def recall(self): else: return 0. + @property + def global_recall(self): + if self.global_true_positives + self.global_false_negatives > 0: + return float(self.global_true_positives) / (self.global_true_positives + self.global_false_negatives) + else: + return 0. + @property def fscore(self): if self.precision + self.recall > 0: @@ -569,17 +684,33 @@ def fscore(self): return 0. @property - def matthewscc(self): + def global_fscore(self): + if self.global_precision + self.global_recall > 0: + return 2 * self.global_precision * self.global_recall / (self.global_precision + self.global_recall) + else: + return 0. + + def matthewscc(self, use_global=False): """ Calculate the Matthew's Correlation Coefficent """ - if not self.total_examples: - return 0. + if use_global: + if not self.global_total_examples: + return 0. + + true_pos = float(self.global_true_positives) + false_pos = float(self.global_false_positives) + false_neg = float(self.global_false_negatives) + true_neg = float(self.global_true_negatives) + else: + if not self.total_examples: + return 0. + + true_pos = float(self.true_positives) + false_pos = float(self.false_positives) + false_neg = float(self.false_negatives) + true_neg = float(self.true_negatives) - true_pos = float(self.true_positives) - false_pos = float(self.false_positives) - false_neg = float(self.false_negatives) - true_neg = float(self.true_negatives) terms = [(true_pos + false_pos), (true_pos + false_neg), (true_neg + false_pos), @@ -594,11 +725,26 @@ def total_examples(self): return self.false_negatives + self.false_positives + \ self.true_negatives + self.true_positives + @property + def global_total_examples(self): + return self.global_false_negatives + self.global_false_positives + \ + self.global_true_negatives + self.global_true_positives + + def local_reset_stats(self): + self.false_positives = 0 + self.false_negatives = 0 + self.true_positives = 0 + self.true_negatives = 0 + def reset_stats(self): self.false_positives = 0 self.false_negatives = 0 self.true_positives = 0 self.true_negatives = 0 + self.global_false_positives = 0 + self.global_false_negatives = 0 + self.global_true_positives = 0 + self.global_true_negatives = 0 @register @@ -649,7 +795,8 @@ def __init__(self, name='f1', self.average = average self.metrics = _BinaryClassificationMetrics() EvalMetric.__init__(self, name=name, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -669,18 +816,30 @@ def update(self, labels, preds): if self.average == "macro": self.sum_metric += self.metrics.fscore + self.global_sum_metric += self.metrics.global_fscore self.num_inst += 1 + self.global_num_inst += 1 self.metrics.reset_stats() else: self.sum_metric = self.metrics.fscore * self.metrics.total_examples + self.global_sum_metric = self.metrics.global_fscore * self.metrics.global_total_examples self.num_inst = self.metrics.total_examples + self.global_num_inst = self.metrics.global_total_examples def reset(self): """Resets the internal evaluation result to initial state.""" self.sum_metric = 0. - self.num_inst = 0. + self.num_inst = 0 + self.global_num_inst = 0 + self.global_sum_metric = 0.0 self.metrics.reset_stats() + def reset_local(self): + """Resets the internal evaluation result to initial state.""" + self.sum_metric = 0. + self.num_inst = 0 + self.metrics.local_reset_stats() + @register class MCC(EvalMetric): @@ -750,7 +909,8 @@ def __init__(self, name='mcc', self._average = average self._metrics = _BinaryClassificationMetrics() EvalMetric.__init__(self, name=name, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -769,19 +929,32 @@ def update(self, labels, preds): self._metrics.update_binary_stats(label, pred) if self._average == "macro": - self.sum_metric += self._metrics.matthewscc + self.sum_metric += self._metrics.matthewscc() + self.global_sum_metric += self._metrics.matthewscc(use_global=True) self.num_inst += 1 + self.global_num_inst += 1 self._metrics.reset_stats() else: - self.sum_metric = self._metrics.matthewscc * self._metrics.total_examples + self.sum_metric = self._metrics.matthewscc() * self._metrics.total_examples + self.global_sum_metric = self._metrics.matthewscc(use_global=True) * \ + self._metrics.global_total_examples self.num_inst = self._metrics.total_examples + self.global_num_inst = self._metrics.global_total_examples def reset(self): """Resets the internal evaluation result to initial state.""" self.sum_metric = 0. self.num_inst = 0. + self.global_sum_metric = 0. + self.global_num_inst = 0. self._metrics.reset_stats() + def reset_local(self): + """Resets the internal evaluation result to initial state.""" + self.sum_metric = 0. + self.num_inst = 0. + self._metrics.local_reset_stats() + @register class Perplexity(EvalMetric): @@ -841,7 +1014,8 @@ def __init__(self, ignore_label, axis=-1, name='perplexity', output_names=None, label_names=None): super(Perplexity, self).__init__( name, ignore_label=ignore_label, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self.ignore_label = ignore_label self.axis = axis @@ -871,7 +1045,9 @@ def update(self, labels, preds): loss -= ndarray.sum(ndarray.log(ndarray.maximum(1e-10, pred))).asscalar() num += pred.size self.sum_metric += loss + self.global_sum_metric += loss self.num_inst += num + self.global_num_inst += num def get(self): """Returns the current evaluation result. @@ -881,7 +1057,23 @@ def get(self): Tuple of (str, float) Representing name of the metric and evaluation result. """ - return (self.name, math.exp(self.sum_metric/self.num_inst)) + if self.num_inst == 0: + return (self.name, float('nan')) + else: + return (self.name, math.exp(self.sum_metric/self.num_inst)) + + def get_global(self): + """Returns the current global evaluation result. + + Returns + ------- + Tuple of (str, float) + Representing name of the metric and evaluation result. + """ + if self.global_num_inst == 0: + return (self.name, float('nan')) + else: + return (self.name, math.exp(self.global_sum_metric/self.global_num_inst)) #################### # REGRESSION METRICS @@ -921,7 +1113,8 @@ class MAE(EvalMetric): def __init__(self, name='mae', output_names=None, label_names=None): super(MAE, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -945,8 +1138,11 @@ def update(self, labels, preds): if len(pred.shape) == 1: pred = pred.reshape(pred.shape[0], 1) - self.sum_metric += numpy.abs(label - pred).mean() + mae = numpy.abs(label - pred).mean() + self.sum_metric += mae + self.global_sum_metric += mae self.num_inst += 1 # numpy.prod(label.shape) + self.global_num_inst += 1 # numpy.prod(label.shape) @register @@ -981,7 +1177,8 @@ class MSE(EvalMetric): def __init__(self, name='mse', output_names=None, label_names=None): super(MSE, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -1005,8 +1202,11 @@ def update(self, labels, preds): if len(pred.shape) == 1: pred = pred.reshape(pred.shape[0], 1) - self.sum_metric += ((label - pred)**2.0).mean() + mse = ((label - pred)**2.0).mean() + self.sum_metric += mse + self.global_sum_metric += mse self.num_inst += 1 # numpy.prod(label.shape) + self.global_num_inst += 1 # numpy.prod(label.shape) @register @@ -1041,7 +1241,8 @@ class RMSE(EvalMetric): def __init__(self, name='rmse', output_names=None, label_names=None): super(RMSE, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -1065,8 +1266,11 @@ def update(self, labels, preds): if len(pred.shape) == 1: pred = pred.reshape(pred.shape[0], 1) - self.sum_metric += numpy.sqrt(((label - pred)**2.0).mean()) + rmse = numpy.sqrt(((label - pred)**2.0).mean()) + self.sum_metric += rmse + self.global_sum_metric += rmse self.num_inst += 1 + self.global_num_inst += 1 @register @@ -1110,7 +1314,8 @@ def __init__(self, eps=1e-12, name='cross-entropy', output_names=None, label_names=None): super(CrossEntropy, self).__init__( name, eps=eps, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self.eps = eps def update(self, labels, preds): @@ -1134,8 +1339,11 @@ def update(self, labels, preds): assert label.shape[0] == pred.shape[0] prob = pred[numpy.arange(label.shape[0]), numpy.int64(label)] - self.sum_metric += (-numpy.log(prob + self.eps)).sum() + cross_entropy = (-numpy.log(prob + self.eps)).sum() + self.sum_metric += cross_entropy + self.global_sum_metric += cross_entropy self.num_inst += label.shape[0] + self.global_num_inst += label.shape[0] @register @alias('nll_loss') @@ -1178,7 +1386,8 @@ def __init__(self, eps=1e-12, name='nll-loss', output_names=None, label_names=None): super(NegativeLogLikelihood, self).__init__( name, eps=eps, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self.eps = eps def update(self, labels, preds): @@ -1202,8 +1411,11 @@ def update(self, labels, preds): num_examples = pred.shape[0] assert label.shape[0] == num_examples, (label.shape[0], num_examples) prob = pred[numpy.arange(num_examples, dtype=numpy.int64), numpy.int64(label)] - self.sum_metric += (-numpy.log(prob + self.eps)).sum() + nll = (-numpy.log(prob + self.eps)).sum() + self.sum_metric += nll + self.global_sum_metric += nll self.num_inst += num_examples + self.global_num_inst += num_examples @register @alias('pearsonr') @@ -1238,7 +1450,8 @@ class PearsonCorrelation(EvalMetric): def __init__(self, name='pearsonr', output_names=None, label_names=None): super(PearsonCorrelation, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, labels, preds): """Updates the internal evaluation result. @@ -1256,8 +1469,11 @@ def update(self, labels, preds): check_label_shapes(label, pred, False, True) label = label.asnumpy() pred = pred.asnumpy() - self.sum_metric += numpy.corrcoef(pred.ravel(), label.ravel())[0, 1] + pearson_corr = numpy.corrcoef(pred.ravel(), label.ravel())[0, 1] + self.sum_metric += pearson_corr + self.global_sum_metric += pearson_corr self.num_inst += 1 + self.global_num_inst += 1 @register @@ -1278,7 +1494,8 @@ class Loss(EvalMetric): def __init__(self, name='loss', output_names=None, label_names=None): super(Loss, self).__init__( - name, output_names=output_names, label_names=label_names) + name, output_names=output_names, label_names=label_names, + has_global_stats=True) def update(self, _, preds): @@ -1286,8 +1503,11 @@ def update(self, _, preds): preds = [preds] for pred in preds: - self.sum_metric += ndarray.sum(pred).asscalar() + loss = ndarray.sum(pred).asscalar() + self.sum_metric += loss + self.global_sum_metric += loss self.num_inst += pred.size + self.global_num_inst += pred.size @register @@ -1353,7 +1573,8 @@ def __init__(self, feval, name=None, allow_extra_outputs=False, super(CustomMetric, self).__init__( name, feval=feval, allow_extra_outputs=allow_extra_outputs, - output_names=output_names, label_names=label_names) + output_names=output_names, label_names=label_names, + has_global_stats=True) self._feval = feval self._allow_extra_outputs = allow_extra_outputs @@ -1379,10 +1600,14 @@ def update(self, labels, preds): if isinstance(reval, tuple): (sum_metric, num_inst) = reval self.sum_metric += sum_metric + self.global_sum_metric += sum_metric self.num_inst += num_inst + self.global_num_inst += num_inst else: self.sum_metric += reval + self.global_sum_metric += reval self.num_inst += 1 + self.global_num_inst += 1 def get_config(self): raise NotImplementedError("CustomMetric cannot be serialized") diff --git a/python/mxnet/model.py b/python/mxnet/model.py index 2666f8bbc..38fe73915 100644 --- a/python/mxnet/model.py +++ b/python/mxnet/model.py @@ -62,6 +62,11 @@ def _create_sparse_kvstore(kvstore): ---------- kvstore : KVStore or str The kvstore. + + Returns + ------- + kvstore : KVStore + update_on_kvstore : bool. Always True. """ # always update on kvstore update_on_kvstore = True diff --git a/python/mxnet/module/base_module.py b/python/mxnet/module/base_module.py index c534261ea..ca8463153 100644 --- a/python/mxnet/module/base_module.py +++ b/python/mxnet/module/base_module.py @@ -22,7 +22,6 @@ import time import logging import warnings -import copy import numpy as np from .. import metric @@ -279,8 +278,8 @@ def score(self, eval_data, eval_metric, num_batch=None, batch_end_callback=None, def iter_predict(self, eval_data, num_batch=None, reset=True, sparse_row_id_fn=None): """Iterates over predictions. - Example Usage: - ---------- + Examples + -------- >>> for pred, i_batch, batch in module.iter_predict(eval_data): ... # pred is a list of outputs from the module ... # i_batch is a integer @@ -508,7 +507,6 @@ def fit(self, train_data, eval_data=None, eval_metric='acc', validation_metric = eval_metric if not isinstance(eval_metric, metric.EvalMetric): eval_metric = metric.create(eval_metric) - epoch_eval_metric = copy.deepcopy(eval_metric) ################################################################################ # training loop @@ -516,7 +514,6 @@ def fit(self, train_data, eval_data=None, eval_metric='acc', for epoch in range(begin_epoch, num_epoch): tic = time.time() eval_metric.reset() - epoch_eval_metric.reset() nbatch = 0 data_iter = iter(train_data) end_of_batch = False @@ -532,12 +529,8 @@ def fit(self, train_data, eval_data=None, eval_metric='acc', self.update_metric(eval_metric, [db.label for db in data_batch], pre_sliced=True) - self.update_metric(epoch_eval_metric, - [db.label for db in data_batch], - pre_sliced=True) else: self.update_metric(eval_metric, data_batch.label) - self.update_metric(epoch_eval_metric, data_batch.label) try: # pre fetch next batch @@ -550,7 +543,7 @@ def fit(self, train_data, eval_data=None, eval_metric='acc', monitor.toc_print() if end_of_batch: - eval_name_vals = epoch_eval_metric.get_name_value() + eval_name_vals = eval_metric.get_global_name_value() if batch_end_callback is not None: batch_end_params = BatchEndParam(epoch=epoch, nbatch=nbatch, diff --git a/python/mxnet/module/sequential_module.py b/python/mxnet/module/sequential_module.py index 8d563a4de..507f3f730 100644 --- a/python/mxnet/module/sequential_module.py +++ b/python/mxnet/module/sequential_module.py @@ -50,34 +50,35 @@ def __init__(self, logger=logging): if x.startswith('META_')]) def add(self, module, **kwargs): - """Adds a module to the chain. + """Add a module to the chain. Parameters ---------- module : BaseModule The new module to add. - kwargs : **keywords + kwargs : ``**keywords`` All the keyword arguments are saved as meta information for the added module. The currently known meta includes - `take_labels`: indicating whether the module expect to - take labels when doing computation. Note any module in - the chain can take labels (not necessarily only the top - most one), and they all take the same labels passed - from the original data batch for the `SequentialModule`. + take labels when doing computation. Note any module in + the chain can take labels (not necessarily only the top + most one), and they all take the same labels passed + from the original data batch for the `SequentialModule`. + Returns ------- self This function returns `self` to allow us to easily chain a series of `add` calls. - Examples -------- >>> # An example of addinging two modules to a chain. >>> seq_mod = mx.mod.SequentialModule() >>> seq_mod.add(mod1) >>> seq_mod.add(mod2) + """ self._modules.append(module) diff --git a/python/mxnet/ndarray/contrib.py b/python/mxnet/ndarray/contrib.py index 67ee7e4d6..6bbee8a6f 100644 --- a/python/mxnet/ndarray/contrib.py +++ b/python/mxnet/ndarray/contrib.py @@ -19,6 +19,7 @@ # pylint: disable=wildcard-import, unused-wildcard-import,redefined-outer-name """Contrib NDArray API of MXNet.""" import math +import numpy as np from ..context import current_context from ..random import uniform from ..base import _as_list @@ -28,7 +29,7 @@ except ImportError: pass -__all__ = ["rand_zipfian", "foreach", "while_loop", "cond"] +__all__ = ["rand_zipfian", "foreach", "while_loop", "cond", "isinf", "isfinite", "isnan"] # pylint: disable=line-too-long def rand_zipfian(true_classes, num_sampled, range_max, ctx=None): @@ -139,9 +140,9 @@ def foreach(body, data, init_states): NDArrays. body takes two arguments as input and outputs a tuple of two elements, - as illustrated below: + as illustrated below:: - out, states = body(data1, states) + out, states = body(data1, states) data1 can be either an NDArray or a list of NDArrays. If data is an NDArray, data1 is an NDArray. Otherwise, data1 is a list of NDArrays and has the same @@ -151,15 +152,15 @@ def foreach(body, data, init_states): are the second output of foreach. The computation done by this operator is equivalent to the pseudo code below - when the input data is NDArray: + when the input data is NDArray:: - states = init_states - outs = [] - for i in data.shape[0]: - s = data[i] - out, states = body(s, states) - outs.append(out) - outs = stack(*outs) + states = init_states + outs = [] + for i in data.shape[0]: + s = data[i] + out, states = body(s, states) + outs.append(out) + outs = stack(*outs) Parameters @@ -436,8 +437,8 @@ def cond(pred, then_func, else_func): -------- >>> a, b = mx.nd.array([1]), mx.nd.array([2]) >>> pred = a * b < 5 - >>> then_func = lambda a, b: (a + 5) * (b + 5) - >>> else_func = lambda a, b: (a - 5) * (b - 5) + >>> then_func = lambda: (a + 5) * (b + 5) + >>> else_func = lambda: (a - 5) * (b - 5) >>> outputs = mx.nd.contrib.cond(pred, then_func, else_func) >>> outputs[0] [42.] @@ -460,3 +461,84 @@ def _to_python_scalar(inputs, type_, name): return then_func() else: return else_func() + +def isinf(data): + """Performs an element-wise check to determine if the NDArray contains an infinite element + or not. + + + Parameters + ---------- + input : NDArray + An N-D NDArray. + + Returns + ------- + output: NDArray + The output NDarray, with same shape as input, where 1 indicates the array element is + equal to positive or negative infinity and 0 otherwise. + + Examples + -------- + >>> data = mx.nd.array([np.inf, -np.inf, np.NINF, -1]) + >>> output = mx.nd.contrib.isinf(data) + >>> output + [1. 1. 1. 0.] + + """ + return data.abs() == np.inf + +def isfinite(data): + """Performs an element-wise check to determine if the NDArray contains an infinite element + or not. + + + Parameters + ---------- + input : NDArray + An N-D NDArray. + + Returns + ------- + output: NDArray + The output NDarray, with same shape as input, where 1 indicates the array element is + finite i.e. not equal to positive or negative infinity and 0 in places where it is + positive or negative infinity. + + Examples + -------- + >>> data = mx.nd.array([np.inf, -np.inf, np.NINF, -1]) + >>> output = mx.nd.contrib.isfinite(data) + >>> output + [0. 0. 0. 1.] + + """ + is_data_not_nan = data == data + is_data_not_infinite = data.abs() != np.inf + return ndarray.logical_and(is_data_not_infinite, is_data_not_nan) + +def isnan(data): + """Performs an element-wise check to determine if the NDArray contains a NaN element + or not. + + + Parameters + ---------- + input : NDArray + An N-D NDArray. + + Returns + ------- + output: NDArray + The output NDarray, with same shape as input, where 1 indicates the array element is + NaN i.e. Not a Number and 0 otherwise. + + Examples + -------- + >>> data = mx.nd.array([np.nan, -1]) + >>> output = mx.nd.contrib.isnan(data) + >>> output + [1. 0.] + + """ + return data != data diff --git a/python/mxnet/ndarray/ndarray.py b/python/mxnet/ndarray/ndarray.py index bf1140d20..9a62620da 100644 --- a/python/mxnet/ndarray/ndarray.py +++ b/python/mxnet/ndarray/ndarray.py @@ -157,6 +157,13 @@ def waitall(): """Wait for all async operations to finish in MXNet. This function is used for benchmarking only. + + .. warning:: + + If your code has exceptions, `waitall` can cause silent failures. + For this reason you should avoid `waitall` in your code. + Use it only if you are confident that your code is error free. + Then make sure you call `wait_to_read` on all outputs after `waitall`. """ check_call(_LIB.MXNDArrayWaitAll()) @@ -399,7 +406,7 @@ def __setitem__(self, key, value): Parameters ---------- - key : int, slice, list, np.ndarray, NDArray, or tuple of all previous types + key : int, mxnet.ndarray.slice, list, np.ndarray, NDArray, or tuple of all previous types The indexing key. value : scalar or array-like object that can be broadcast to the shape of self[key] The value to set. @@ -467,7 +474,7 @@ def __getitem__(self, key): Parameters ---------- - key : int, slice, list, np.ndarray, NDArray, or tuple of all previous types + key : int, mxnet.ndarray.slice, list, np.ndarray, NDArray, or tuple of all previous types Indexing key. Examples @@ -998,7 +1005,7 @@ def reshape(self, *shape, **kwargs): Example:: - - without reverse=1, for input shape = (10,5,4), shape = (-1,0), output shape would be + - without reverse=1, for input shape = (10,5,4), shape = (-1,0), output shape would be \ (40,5). - with reverse=1, output shape will be (50,4). @@ -2642,9 +2649,9 @@ def add(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be added. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be added. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -2704,9 +2711,9 @@ def subtract(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be subtracted. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be subtracted. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -2765,9 +2772,9 @@ def multiply(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be multiplied. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be multiplied. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -2826,9 +2833,9 @@ def divide(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array in division. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array in division. The arrays to be divided. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -2883,9 +2890,9 @@ def modulo(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array in modulo. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array in modulo. The arrays to be taken modulo. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3002,9 +3009,9 @@ def maximum(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3059,9 +3066,9 @@ def minimum(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3120,9 +3127,9 @@ def equal(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3184,9 +3191,9 @@ def not_equal(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3251,9 +3258,9 @@ def greater(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3315,9 +3322,9 @@ def greater_equal(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3379,9 +3386,9 @@ def lesser(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3443,9 +3450,9 @@ def lesser_equal(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First array to be compared. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second array to be compared. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3506,9 +3513,9 @@ def logical_and(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First input of the function. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second input of the function. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3566,9 +3573,9 @@ def logical_or(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First input of the function. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second input of the function. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -3626,9 +3633,9 @@ def logical_xor(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.array First input of the function. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.array Second input of the function. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. diff --git a/python/mxnet/ndarray/random.py b/python/mxnet/ndarray/random.py index 1e941f79a..78339a020 100644 --- a/python/mxnet/ndarray/random.py +++ b/python/mxnet/ndarray/random.py @@ -25,7 +25,7 @@ __all__ = ['uniform', 'normal', 'randn', 'poisson', 'exponential', 'gamma', 'multinomial', 'negative_binomial', 'generalized_negative_binomial', - 'shuffle'] + 'shuffle', 'randint'] def _random_helper(random, sampler, params, shape, dtype, ctx, out, kwargs): @@ -59,23 +59,23 @@ def uniform(low=0, high=1, shape=_Null, dtype=_Null, ctx=None, out=None, **kwarg Parameters ---------- - low : float or NDArray + low : float or NDArray, optional Lower boundary of the output interval. All values generated will be greater than or equal to low. The default value is 0. - high : float or NDArray + high : float or NDArray, optional Upper boundary of the output interval. All values generated will be less than high. The default value is 1.0. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `low` and `high` are scalars, output shape will be `(m, n)`. If `low` and `high` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[low, high)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `low.context` when `low` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -111,21 +111,21 @@ def normal(loc=0, scale=1, shape=_Null, dtype=_Null, ctx=None, out=None, **kwarg Parameters ---------- - loc : float or NDArray + loc : float or NDArray, optional Mean (centre) of the distribution. - scale : float or NDArray + scale : float or NDArray, optional Standard deviation (spread or width) of the distribution. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `loc` and `scale` are scalars, output shape will be `(m, n)`. If `loc` and `scale` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[loc, scale)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `loc.context` when `loc` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -170,7 +170,7 @@ def randn(*shape, **kwargs): `scale` are scalars, output shape will be `(m, n)`. If `loc` and `scale` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[loc, scale)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'} Data type of output samples. Default is 'float32' ctx : Context Device context of output. Default is current context. Overridden by @@ -212,19 +212,19 @@ def poisson(lam=1, shape=_Null, dtype=_Null, ctx=None, out=None, **kwargs): Parameters ---------- - lam : float or NDArray + lam : float or NDArray, optional Expectation of interval, should be >= 0. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `lam` is a scalar, output shape will be `(m, n)`. If `lam` is an NDArray with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each entry in `lam`. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `lam.context` when `lam` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -259,19 +259,19 @@ def exponential(scale=1, shape=_Null, dtype=_Null, ctx=None, out=None, **kwargs) Parameters ---------- - scale : float or NDArray + scale : float or NDArray, optional The scale parameter, \beta = 1/\lambda. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `scale` is a scalar, output shape will be `(m, n)`. If `scale` is an NDArray with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each entry in `scale`. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `scale.context` when `scale` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -302,22 +302,22 @@ def gamma(alpha=1, beta=1, shape=_Null, dtype=_Null, ctx=None, out=None, **kwarg Parameters ---------- - alpha : float or NDArray + alpha : float or NDArray, optional The shape of the gamma distribution. Should be greater than zero. - beta : float or NDArray + beta : float or NDArray, optional The scale of the gamma distribution. Should be greater than zero. Default is equal to 1. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `alpha` and `beta` are scalars, output shape will be `(m, n)`. If `alpha` and `beta` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[alpha, beta)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `alpha.context` when `alpha` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -352,21 +352,21 @@ def negative_binomial(k=1, p=1, shape=_Null, dtype=_Null, ctx=None, Parameters ---------- - k : float or NDArray + k : float or NDArray, optional Limit of unsuccessful experiments, > 0. - p : float or NDArray + p : float or NDArray, optional Failure probability in each experiment, >= 0 and <=1. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `k` and `p` are scalars, output shape will be `(m, n)`. If `k` and `p` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[k, p)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `k.context` when `k` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -403,21 +403,21 @@ def generalized_negative_binomial(mu=1, alpha=1, shape=_Null, dtype=_Null, ctx=N Parameters ---------- - mu : float or NDArray + mu : float or NDArray, optional Mean of the negative binomial distribution. - alpha : float or NDArray + alpha : float or NDArray, optional Alpha (dispersion) parameter of the negative binomial distribution. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `mu` and `alpha` are scalars, output shape will be `(m, n)`. If `mu` and `alpha` are NDArrays with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[mu, alpha)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' - ctx : Context + ctx : Context, optional Device context of output. Default is current context. Overridden by `mu.context` when `mu` is an NDArray. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. @@ -455,17 +455,17 @@ def multinomial(data, shape=_Null, get_prob=False, out=None, dtype='int32', **kw `k` is the number of possible outcomes of each multinomial distribution. For example, data with shape `(m, n, k)` specifies `m*n` multinomial distributions each with `k` possible outcomes. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw from each distribution. If shape is empty one sample will be drawn from each distribution. - get_prob : bool + get_prob : bool, optional If true, a second array containing log likelihood of the drawn samples will also be returned. This is usually used for reinforcement learning, where you can provide reward as head gradient w.r.t. this array to estimate gradient. - out : NDArray + out : NDArray, optional Store output to an existing NDArray. - dtype : str or numpy.dtype + dtype : str or numpy.dtype, optional Data type of the sample output array. The default is int32. Note that the data type of the log likelihood array is the same with that of `data`. @@ -500,7 +500,7 @@ def shuffle(data, **kwargs): ---------- data : NDArray Input data array. - out : NDArray + out : NDArray, optional Array to store the result. Examples @@ -518,3 +518,45 @@ def shuffle(data, **kwargs): """ return _internal._shuffle(data, **kwargs) + + +def randint(low, high, shape=_Null, dtype=_Null, ctx=None, out=None, **kwargs): + """Draw random samples from a discrete uniform distribution. + + Samples are uniformly distributed over the half-open interval *[low, high)* + (includes *low*, but excludes *high*). + + Parameters + ---------- + low : int, required + Lower boundary of the output interval. All values generated will be + greater than or equal to low. + high : int, required + Upper boundary of the output interval. All values generated will be + less than high. + shape : int or tuple of ints, optional + The number of samples to draw. If shape is, e.g., `(m, n)` and `low` and + `high` are scalars, output shape will be `(m, n)`. + dtype : {'int32', 'int64'}, optional + Data type of output samples. Default is 'int32' + ctx : Context, optional + Device context of output. Default is current context. Overridden by + `low.context` when `low` is an NDArray. + out : NDArray, optional + Store output to an existing NDArray. + + + Examples + -------- + >>> mx.nd.random.randint(5, 100) + [ 90] + >> mx.nd.random.randint(-10, 2, ctx=mx.gpu(0)) + [ -8] + + >>> mx.nd.random.randint(-10, 10, shape=(2,)) + [ -5 4] + + """ + return _random_helper(_internal._random_randint, None, + [low, high], shape, dtype, ctx, out, kwargs) diff --git a/python/mxnet/ndarray/sparse.py b/python/mxnet/ndarray/sparse.py index 7b4cc9064..928079749 100644 --- a/python/mxnet/ndarray/sparse.py +++ b/python/mxnet/ndarray/sparse.py @@ -195,7 +195,8 @@ def asnumpy(self): return self.tostype('default').asnumpy() def astype(self, dtype, copy=True): - """Returns a copy of the array after casting to a specified type. + """Return a copy of the array after casting to a specified type. + Parameters ---------- dtype : numpy.dtype or str @@ -205,6 +206,7 @@ def astype(self, dtype, copy=True): allocated ndarray on the same context. If this is set to `False`, and the dtype requested is the same as the ndarray's dtype, the ndarray is returned instead of a copy. + Examples -------- >>> x = mx.nd.sparse.zeros('row_sparse', (2,3), dtype='float32') @@ -339,7 +341,7 @@ def __getitem__(self, key): Parameters ---------- - key : int or slice + key : int or mxnet.ndarray.NDArray.slice Indexing key. Examples @@ -387,7 +389,7 @@ def __setitem__(self, key, value): Parameters ---------- - key : slice + key : mxnet.ndarray.NDArray.slice The indexing key. value : NDArray or CSRNDArray or numpy.ndarray The value to set. @@ -418,7 +420,7 @@ def __setitem__(self, key, value): if isinstance(key, py_slice): if key.step is not None or key.start is not None or key.stop is not None: raise ValueError('Assignment with slice for CSRNDArray is not ' \ - 'implmented yet.') + 'implemented yet.') if isinstance(value, NDArray): # avoid copying to itself if value.handle is not self.handle: @@ -624,7 +626,7 @@ def __getitem__(self, key): Parameters ---------- - key : slice + key : mxnet.ndarray.NDArray.slice Indexing key. Examples @@ -652,7 +654,7 @@ def __setitem__(self, key, value): Parameters ---------- - key : slice + key : mxnet.ndarray.NDArray.slice The indexing key. value : NDArray or numpy.ndarray The value to set. @@ -1023,28 +1025,28 @@ def row_sparse_array(arg1, shape=None, ctx=None, dtype=None): - row_sparse_array(D): to construct a RowSparseNDArray with a dense ndarray ``D`` - - **D** (*array_like*) - An object exposing the array interface, an object whose \ - `__array__` method returns an array, or any (nested) sequence. - - **ctx** (*Context, optional*) - Device context \ - (default is the current default context). - - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ - The default dtype is ``D.dtype`` if ``D`` is an NDArray or numpy.ndarray, \ - float32 otherwise. + - **D** (*array_like*) - An object exposing the array interface, an object whose \ + `__array__` method returns an array, or any (nested) sequence. + - **ctx** (*Context, optional*) - Device context \ + (default is the current default context). + - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ + The default dtype is ``D.dtype`` if ``D`` is an NDArray or numpy.ndarray, \ + float32 otherwise. - row_sparse_array(S) to construct a RowSparseNDArray with a sparse ndarray ``S`` - - **S** (*RowSparseNDArray*) - A sparse ndarray. - - **ctx** (*Context, optional*) - Device context \ - (default is the current default context). - - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ - The default dtype is ``S.dtype``. + - **S** (*RowSparseNDArray*) - A sparse ndarray. + - **ctx** (*Context, optional*) - Device context \ + (default is the current default context). + - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ + The default dtype is ``S.dtype``. - row_sparse_array((D0, D1 .. Dn)) to construct an empty RowSparseNDArray with shape ``(D0, D1, ... Dn)`` - - **D0, D1 .. Dn** (*int*) - The shape of the ndarray - - **ctx** (*Context, optional*) - Device context \ - (default is the current default context). - - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ + - **D0, D1 .. Dn** (*int*) - The shape of the ndarray + - **ctx** (*Context, optional*) - Device context \ + (default is the current default context). + - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ The default dtype is float32. - row_sparse_array((data, indices)) @@ -1055,35 +1057,35 @@ def row_sparse_array(arg1, shape=None, ctx=None, dtype=None): represented by RowSparseNDArray ``rsp`` has \ ``dense[rsp.indices[i], :, :, :, ...] = rsp.data[i, :, :, :, ...]`` The row indices for are expected to be **sorted in ascending order.** \ - - **data** (*array_like*) - An object exposing the array interface, which \ - holds all the non-zero row slices of the array. - - **indices** (*array_like*) - An object exposing the array interface, which \ - stores the row index for each row slice with non-zero elements. - - **shape** (*tuple of int, optional*) - The shape of the array. The default \ - shape is inferred from the indices and indptr arrays. - - **ctx** (*Context, optional*) - Device context \ - (default is the current default context). - - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ - The default dtype is float32. + - **data** (*array_like*) - An object exposing the array interface, which \ + holds all the non-zero row slices of the array. + - **indices** (*array_like*) - An object exposing the array interface, which \ + stores the row index for each row slice with non-zero elements. + - **shape** (*tuple of int, optional*) - The shape of the array. The default \ + shape is inferred from the indices and indptr arrays. + - **ctx** (*Context, optional*) - Device context \ + (default is the current default context). + - **dtype** (*str or numpy.dtype, optional*) - The data type of the output array. \ + The default dtype is float32. Parameters ---------- - arg1: NDArray, numpy.ndarray, RowSparseNDArray, tuple of int or tuple of array_like + arg1 : NDArray, numpy.ndarray, RowSparseNDArray, tuple of int or tuple of array_like The argument to help instantiate the row sparse ndarray. See above for further details. shape : tuple of int, optional - The shape of the row sparse ndarray. + The shape of the row sparse ndarray. (Default value = None) ctx : Context, optional Device context (default is the current default context). dtype : str or numpy.dtype, optional - The data type of the output array. + The data type of the output array. (Default value = None) Returns ------- RowSparseNDArray An `RowSparseNDArray` with the `row_sparse` storage representation. - Example - ------- + Examples + -------- >>> a = mx.nd.sparse.row_sparse_array(([[1, 2], [3, 4]], [1, 4]), shape=(6, 2)) >>> a.asnumpy() array([[ 0., 0.], @@ -1203,9 +1205,9 @@ def add(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.sparse.array First array to be added. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.sparse.array Second array to be added. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -1275,9 +1277,9 @@ def subtract(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.sparse.array First array to be subtracted. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.sparse.array Second array to be subtracted. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape.__spec__ @@ -1346,9 +1348,9 @@ def multiply(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.sparse.array First array to be multiplied. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.sparse.array Second array to be multiplied. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. @@ -1430,9 +1432,9 @@ def divide(lhs, rhs): Parameters ---------- - lhs : scalar or array + lhs : scalar or mxnet.ndarray.sparse.array First array in division. - rhs : scalar or array + rhs : scalar or mxnet.ndarray.sparse.array Second array in division. The arrays to be divided. If ``lhs.shape != rhs.shape``, they must be broadcastable to a common shape. diff --git a/python/mxnet/optimizer/optimizer.py b/python/mxnet/optimizer/optimizer.py index bc03497fc..d290a3f2f 100644 --- a/python/mxnet/optimizer/optimizer.py +++ b/python/mxnet/optimizer/optimizer.py @@ -43,38 +43,43 @@ class Optimizer(object): Parameters ---------- - rescale_grad : float, optional + rescale_grad : float, optional, default 1.0 Multiply the gradient with `rescale_grad` before updating. Often choose to be ``1.0/batch_size``. - param_idx2name : dict from int to string, optional + param_idx2name : dict from int to string, optional, default None A dictionary that maps int index to string name. - clip_gradient : float, optional + clip_gradient : float, optional, default None Clip the gradient by projecting onto the box ``[-clip_gradient, clip_gradient]``. - learning_rate : float, optional + learning_rate : float, optional, default 0.01 The initial learning rate. - lr_scheduler : LRScheduler, optional + lr_scheduler : LRScheduler, optional, default None The learning rate scheduler. - wd : float, optional + wd : float, optional, default 0.0 The weight decay (or L2 regularization) coefficient. Modifies objective by adding a penalty for having large weights. - sym: Symbol, optional + sym: Symbol, optional, default None The Symbol this optimizer is applying to. - begin_num_update : int, optional + begin_num_update : int, optional, default 0 The initial number of updates. - multi_precision : bool, optional - Flag to control the internal precision of the optimizer. - ``False`` results in using the same precision as the weights (default), - ``True`` makes internal 32-bit copy of the weights and applies gradients - in 32-bit precision even if actual weights used in the model have lower precision. - Turning this on can improve convergence and accuracy when training with float16. + multi_precision : bool, optional, default False + Flag to control the internal precision of the optimizer.:: + + False: results in using the same precision as the weights (default), + True: makes internal 32-bit copy of the weights and applies gradients + in 32-bit precision even if actual weights used in the model have lower precision. + Turning this on can improve convergence and accuracy when training with float16. + + param_dict : dict of int -> gluon.Parameter, default None + Dictionary of parameter index to gluon.Parameter, used to lookup parameter attributes + such as lr_mult, wd_mult, etc. param_dict shall not be deep copied. Properties ---------- @@ -481,16 +486,17 @@ class SGD(Optimizer): Parameters ---------- momentum : float, optional - The momentum value. + The momentum value. lazy_update : bool, optional - Default is True. If True, lazy updates are applied \ - if the storage types of weight and grad are both ``row_sparse``. + Default is True. If True, lazy updates are applied \ + if the storage types of weight and grad are both ``row_sparse``. multi_precision: bool, optional - Flag to control the internal precision of the optimizer. - ``False`` results in using the same precision as the weights (default), - ``True`` makes internal 32-bit copy of the weights and applies gradients \ - in 32-bit precision even if actual weights used in the model have lower precision.\ - Turning this on can improve convergence and accuracy when training with float16. + Flag to control the internal precision of the optimizer.:: + + False: results in using the same precision as the weights (default), + True: makes internal 32-bit copy of the weights and applies gradients + in 32-bit precision even if actual weights used in the model have lower precision. + Turning this on can improve convergence and accuracy when training with float16. """ def __init__(self, momentum=0.0, lazy_update=True, **kwargs): super(SGD, self).__init__(**kwargs) @@ -562,7 +568,8 @@ class Signum(Optimizer): state = momentum * state + (1-momentum)*rescaled_grad weight = (1 - lr * wd_lh) * weight - lr * sign(state) - Reference: + References + ---------- Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli & Anima Anandkumar. (2018). signSGD: Compressed Optimisation for Non-Convex Problems. In ICML'18. @@ -634,6 +641,8 @@ class FTML(Optimizer): z = beta1 * z + (1 - beta1) * rescaled_grad - (d_t - beta1 * d_(t-1)) * weight weight = - z / d_t + For details of the update algorithm, see :class:`~mxnet.ndarray.ftml_update`. + This optimizer accepts the following parameters in addition to those accepted by :class:`.Optimizer`. @@ -683,8 +692,11 @@ class LBSGD(Optimizer): state = momentum * state + lr * rescale_grad * clip(grad, clip_gradient) + wd * weight weight = weight - state - For details of the update algorithm see :class:`~mxnet.ndarray.lbsgd_update` and - :class:`~mxnet.ndarray.lbsgd_mom_update`. + For details of the update algorithm see :class:`~mxnet.ndarray.sgd_update` + and :class:`~mxnet.ndarray.sgd_mom_update`. + In addition to the SGD updates the LBSGD optimizer uses the LARS, Layer-wise + Adaptive Rate Scaling, algorithm to have a separate learning rate for each + layer of the network, which leads to better stability over large batch sizes. This optimizer accepts the following parameters in addition to those accepted by :class:`.Optimizer`. @@ -692,20 +704,21 @@ class LBSGD(Optimizer): Parameters ---------- momentum : float, optional - The momentum value. + The momentum value. multi_precision: bool, optional - Flag to control the internal precision of the optimizer. - ``False`` results in using the same precision as the weights (default), - ``True`` makes internal 32-bit copy of the weights and applies gradients - in 32-bit precision even if actual weights used in the model have lower precision.`< - Turning this on can improve convergence and accuracy when training with float16. + Flag to control the internal precision of the optimizer.:: + + False: results in using the same precision as the weights (default), + True: makes internal 32-bit copy of the weights and applies gradients + in 32-bit precision even if actual weights used in the model have lower precision. + Turning this on can improve convergence and accuracy when training with float16. + warmup_strategy: string ('linear', 'power2', 'sqrt'. , 'lars' default : 'linear') warmup_epochs: unsigned, default: 5 batch_scale: unsigned, default: 1 (same as batch size*numworkers) updates_per_epoch: updates_per_epoch (default: 32, Default might not reflect true number batches per epoch. Used for warmup.) begin_epoch: unsigned, default 0, starting epoch. """ - def __init__(self, momentum=0.0, multi_precision=False, warmup_strategy='linear', warmup_epochs=5, batch_scale=1, updates_per_epoch=32, begin_epoch=0, num_epochs=60, **kwargs): @@ -934,11 +947,12 @@ class NAG(Optimizer): momentum : float, optional The momentum value. multi_precision: bool, optional - Flag to control the internal precision of the optimizer. - ``False`` results in using the same precision as the weights (default), - ``True`` makes internal 32-bit copy of the weights and applies gradients \ - in 32-bit precision even if actual weights used in the model have lower precision.\ - Turning this on can improve convergence and accuracy when training with float16. + Flag to control the internal precision of the optimizer.:: + + False: results in using the same precision as the weights (default), + True: makes internal 32-bit copy of the weights and applies gradients + in 32-bit precision even if actual weights used in the model have lower precision. + Turning this on can improve convergence and accuracy when training with float16. """ def __init__(self, momentum=0.0, **kwargs): super(NAG, self).__init__(**kwargs) @@ -1016,13 +1030,14 @@ class Adam(Optimizer): Stochastic Optimization*, available at http://arxiv.org/abs/1412.6980. If the storage types of grad is ``row_sparse``, and ``lazy_update`` is True, \ - **lazy updates** are applied by:: + **lazy updates** at step t are applied by:: for row in grad.indices: rescaled_grad[row] = clip(grad[row] * rescale_grad + wd * weight[row], clip_gradient) m[row] = beta1 * m[row] + (1 - beta1) * rescaled_grad[row] v[row] = beta2 * v[row] + (1 - beta2) * (rescaled_grad[row]**2) - w[row] = w[row] - learning_rate * m[row] / (sqrt(v[row]) + epsilon) + lr = learning_rate * sqrt(1 - beta1**t) / (1 - beta2**t) + w[row] = w[row] - lr * m[row] / (sqrt(v[row]) + epsilon) The lazy update only updates the mean and var for the weights whose row_sparse gradient indices appear in the current batch, rather than updating it for all indices. @@ -1030,12 +1045,13 @@ class Adam(Optimizer): throughput for some applications. However, it provides slightly different semantics than the original update, and may lead to different empirical results. - Otherwise, **standard updates** are applied by:: + Otherwise, **standard updates** at step t are applied by:: rescaled_grad = clip(grad * rescale_grad + wd * weight, clip_gradient) m = beta1 * m + (1 - beta1) * rescaled_grad v = beta2 * v + (1 - beta2) * (rescaled_grad**2) - w = w - learning_rate * m / (sqrt(v) + epsilon) + lr = learning_rate * sqrt(1 - beta1**t) / (1 - beta2**t) + w = w - lr * m / (sqrt(v) + epsilon) This optimizer accepts the following parameters in addition to those accepted by :class:`.Optimizer`. @@ -1176,9 +1192,11 @@ class RMSProp(Optimizer): epsilon : float, optional Small value to avoid division by 0. centered : bool, optional - Flag to control which version of RMSProp to use. - ``True`` will use Graves's version of `RMSProp`, - ``False`` will use Tieleman & Hinton's version of `RMSProp`. + Flag to control which version of RMSProp to use.:: + + True: will use Graves's version of `RMSProp`, + False: will use Tieleman & Hinton's version of `RMSProp`. + clip_weights : float, optional Clips weights into range ``[-clip_weights, clip_weights]``. """ diff --git a/python/mxnet/recordio.py b/python/mxnet/recordio.py index 6fc4d8e7b..bdc63235d 100644 --- a/python/mxnet/recordio.py +++ b/python/mxnet/recordio.py @@ -18,6 +18,7 @@ """Read and write for the RecordIO data format.""" from __future__ import absolute_import from collections import namedtuple +from multiprocessing import current_process import ctypes import struct @@ -36,8 +37,8 @@ class MXRecordIO(object): """Reads/writes `RecordIO` data format, supporting sequential read and write. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXRecordIO('tmp.rec', 'w') >>> for i in range(5): @@ -65,6 +66,7 @@ def __init__(self, uri, flag): self.uri = c_str(uri) self.handle = RecordIOHandle() self.flag = flag + self.pid = None self.is_open = False self.open() @@ -78,6 +80,7 @@ def open(self): self.writable = False else: raise ValueError("Invalid flag %s"%self.flag) + self.pid = current_process().pid self.is_open = True def __del__(self): @@ -109,6 +112,14 @@ def __setstate__(self, d): if is_open: self.open() + def _check_pid(self, allow_reset=False): + """Check process id to ensure integrity, reset if in new process.""" + if not self.pid == current_process().pid: + if allow_reset: + self.reset() + else: + raise RuntimeError("Forbidden operation in multiple processes") + def close(self): """Closes the record file.""" if not self.is_open: @@ -118,14 +129,15 @@ def close(self): else: check_call(_LIB.MXRecordIOReaderFree(self.handle)) self.is_open = False + self.pid = None def reset(self): """Resets the pointer to first item. If the record is opened with 'w', this function will truncate the file to empty. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXRecordIO('tmp.rec', 'r') >>> for i in range(2): ... item = record.read() @@ -143,8 +155,8 @@ def reset(self): def write(self, buf): """Inserts a string buffer as a record. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXRecordIO('tmp.rec', 'w') >>> for i in range(5): ... record.write('record_%d'%i) @@ -156,6 +168,7 @@ def write(self, buf): Buffer to write. """ assert self.writable + self._check_pid(allow_reset=False) check_call(_LIB.MXRecordIOWriterWriteRecord(self.handle, ctypes.c_char_p(buf), ctypes.c_size_t(len(buf)))) @@ -163,8 +176,8 @@ def write(self, buf): def read(self): """Returns record as a string. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXRecordIO('tmp.rec', 'r') >>> for i in range(5): ... item = record.read() @@ -182,6 +195,9 @@ def read(self): Buffer read. """ assert not self.writable + # trying to implicitly read from multiple processes is forbidden, + # there's no elegant way to handle unless lock is introduced + self._check_pid(allow_reset=False) buf = ctypes.c_char_p() size = ctypes.c_size_t() check_call(_LIB.MXRecordIOReaderReadRecord(self.handle, @@ -196,8 +212,8 @@ def read(self): class MXIndexedRecordIO(MXRecordIO): """Reads/writes `RecordIO` data format, supporting random access. - Example usage: - ---------- + Examples + --------- >>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) >>> record.close() @@ -255,14 +271,15 @@ def seek(self, idx): This function is internally called by `read_idx(idx)` to find the current reader pointer position. It doesn't return anything.""" assert not self.writable + self._check_pid(allow_reset=True) pos = ctypes.c_size_t(self.idx[idx]) check_call(_LIB.MXRecordIOReaderSeek(self.handle, pos)) def tell(self): """Returns the current position of write head. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w') >>> print(record.tell()) 0 @@ -283,8 +300,8 @@ def tell(self): def read_idx(self, idx): """Returns the record at given index. - Example usage: - ---------- + Examples + --------- >>> record = mx.recordio.MXIndexedRecordIO('tmp.idx', 'tmp.rec', 'w') >>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) @@ -299,8 +316,8 @@ def read_idx(self, idx): def write_idx(self, idx, buf): """Inserts input record at given index. - Example usage: - ---------- + Examples + --------- >>> for i in range(5): ... record.write_idx(i, 'record_%d'%i) >>> record.close() diff --git a/python/mxnet/rnn/rnn.py b/python/mxnet/rnn/rnn.py index 47307c55b..0255c55db 100644 --- a/python/mxnet/rnn/rnn.py +++ b/python/mxnet/rnn/rnn.py @@ -35,7 +35,7 @@ def save_rnn_checkpoint(cells, prefix, epoch, symbol, arg_params, aux_params): Parameters ---------- - cells : RNNCell or list of RNNCells + cells : mxnet.rnn.RNNCell or list of RNNCells The RNN cells used by this symbol. prefix : str Prefix of model name. @@ -65,7 +65,7 @@ def load_rnn_checkpoint(cells, prefix, epoch): Parameters ---------- - cells : RNNCell or list of RNNCells + cells : mxnet.rnn.RNNCell or list of RNNCells The RNN cells used by this symbol. prefix : str Prefix of model name. @@ -100,7 +100,7 @@ def do_rnn_checkpoint(cells, prefix, period=1): Parameters ---------- - cells : RNNCell or list of RNNCells + cells : mxnet.rnn.RNNCell or list of RNNCells The RNN cells used by this symbol. prefix : str The file prefix to checkpoint to diff --git a/python/mxnet/rnn/rnn_cell.py b/python/mxnet/rnn/rnn_cell.py index 3301102ba..6738aa19e 100644 --- a/python/mxnet/rnn/rnn_cell.py +++ b/python/mxnet/rnn/rnn_cell.py @@ -368,7 +368,7 @@ class RNNCell(BaseRNNCell): Number of units in output symbol. activation : str or Symbol, default 'tanh' Type of activation function. Options are 'relu' and 'tanh'. - prefix : str, default 'rnn_' + prefix : str, default ``'rnn_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -412,7 +412,7 @@ class LSTMCell(BaseRNNCell): ---------- num_hidden : int Number of units in output symbol. - prefix : str, default 'lstm_' + prefix : str, default ``'lstm_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -475,7 +475,7 @@ class GRUCell(BaseRNNCell): ---------- num_hidden : int Number of units in output symbol. - prefix : str, default 'gru_' + prefix : str, default ``'gru_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -554,7 +554,7 @@ class FusedRNNCell(BaseRNNCell): Whether to return the states that can be used as starting states next time. forget_bias : bias added to forget gate, default 1.0. Jozefowicz et al. 2015 recommends setting this to 1.0 - prefix : str, default '$mode_' such as 'lstm_' + prefix : str, default ``'$mode_'`` such as ``'lstm_'`` Prefix for names of layers (this prefix is also used for names of weights if `params` is None i.e. if `params` are being created and not reused) @@ -716,7 +716,7 @@ def unfuse(self): Returns ------- - cell : SequentialRNNCell + cell : mxnet.rnn.SequentialRNNCell unfused cell that can be used for stepping, and can run on CPU. """ stack = SequentialRNNCell() @@ -832,7 +832,7 @@ class DropoutCell(BaseRNNCell): dropout : float Percentage of elements to drop out, which is 1 - percentage to retain. - prefix : str, default 'dropout_' + prefix : str, default ``'dropout_'`` Prefix for names of layers (this prefix is also used for names of weights if `params` is None i.e. if `params` are being created and not reused) @@ -1007,7 +1007,7 @@ class BidirectionalCell(BaseRNNCell): params : RNNParams, default None. Container for weight sharing between cells. A new RNNParams container is created if `params` is None. - output_prefix : str, default 'bi_' + output_prefix : str, default ``'bi_'`` prefix for name of output """ def __init__(self, l_cell, r_cell, params=None, output_prefix='bi_'): @@ -1207,7 +1207,7 @@ class ConvRNNCell(BaseConvRNNCell): activation : str or Symbol, default functools.partial(symbol.LeakyReLU, act_type='leaky', slope=0.2) Type of activation function. - prefix : str, default 'ConvRNN_' + prefix : str, default ``'ConvRNN_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -1253,7 +1253,8 @@ def state_info(self): class ConvLSTMCell(BaseConvRNNCell): """Convolutional LSTM network cell. - Reference: + References + ---------- Xingjian et al. NIPS2015 Parameters @@ -1287,7 +1288,7 @@ class ConvLSTMCell(BaseConvRNNCell): activation : str or Symbol default functools.partial(symbol.LeakyReLU, act_type='leaky', slope=0.2) Type of activation function. - prefix : str, default 'ConvLSTM_' + prefix : str, default ``'ConvLSTM_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. @@ -1379,7 +1380,7 @@ class ConvGRUCell(BaseConvRNNCell): activation : str or Symbol, default functools.partial(symbol.LeakyReLU, act_type='leaky', slope=0.2) Type of activation function. - prefix : str, default 'ConvGRU_' + prefix : str, default ``'ConvGRU_'`` Prefix for name of layers (and name of weight if params is None). params : RNNParams, default None Container for weight sharing between cells. Created if None. diff --git a/python/mxnet/symbol/contrib.py b/python/mxnet/symbol/contrib.py index b55142232..d54267fc0 100644 --- a/python/mxnet/symbol/contrib.py +++ b/python/mxnet/symbol/contrib.py @@ -236,15 +236,15 @@ def foreach(body, data, init_states, name="foreach"): output data, the body function can return (out, []). The computation done by this operator is equivalent to the pseudo code below - when the input data is NDArray: - - states = init_states - outs = [] - for i in data.shape[0]: - s = data[i] - out, states = body(s, states) - outs.append(out) - outs = stack(*outs) + when the input data is NDArray:: + + states = init_states + outs = [] + for i in data.shape[0]: + s = data[i] + out, states = body(s, states) + outs.append(out) + outs = stack(*outs) Parameters diff --git a/python/mxnet/symbol/random.py b/python/mxnet/symbol/random.py index e9abe9c4a..34663cddf 100644 --- a/python/mxnet/symbol/random.py +++ b/python/mxnet/symbol/random.py @@ -23,7 +23,7 @@ __all__ = ['uniform', 'normal', 'poisson', 'exponential', 'gamma', 'multinomial', - 'negative_binomial', 'generalized_negative_binomial', 'shuffle'] + 'negative_binomial', 'generalized_negative_binomial', 'shuffle', 'randint'] def _random_helper(random, sampler, params, shape, dtype, kwargs): @@ -53,18 +53,18 @@ def uniform(low=0, high=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - low : float or Symbol + low : float or Symbol, optional Lower boundary of the output interval. All values generated will be greater than or equal to low. The default value is 0. - high : float or Symbol + high : float or Symbol, optional Upper boundary of the output interval. All values generated will be less than high. The default value is 1.0. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `low` and `high` are scalars, output shape will be `(m, n)`. If `low` and `high` are Symbols with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[low, high)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_uniform, _internal._sample_uniform, @@ -80,16 +80,16 @@ def normal(loc=0, scale=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - loc : float or Symbol + loc : float or Symbol, optional Mean (centre) of the distribution. - scale : float or Symbol + scale : float or Symbol, optional Standard deviation (spread or width) of the distribution. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `loc` and `scale` are scalars, output shape will be `(m, n)`. If `loc` and `scale` are Symbols with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[loc, scale)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_normal, _internal._sample_normal, @@ -104,14 +104,14 @@ def poisson(lam=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - lam : float or Symbol + lam : float or Symbol, optional Expectation of interval, should be >= 0. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `lam` is a scalar, output shape will be `(m, n)`. If `lam` is an Symbol with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each entry in `lam`. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_poisson, _internal._sample_poisson, @@ -130,14 +130,14 @@ def exponential(scale=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - scale : float or Symbol + scale : float or Symbol, optional The scale parameter, \beta = 1/\lambda. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `scale` is a scalar, output shape will be `(m, n)`. If `scale` is an Symbol with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each entry in `scale`. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_exponential, _internal._sample_exponential, @@ -152,17 +152,17 @@ def gamma(alpha=1, beta=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - alpha : float or Symbol + alpha : float or Symbol, optional The shape of the gamma distribution. Should be greater than zero. - beta : float or Symbol + beta : float or Symbol, optional The scale of the gamma distribution. Should be greater than zero. Default is equal to 1. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `alpha` and `beta` are scalars, output shape will be `(m, n)`. If `alpha` and `beta` are Symbols with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[alpha, beta)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_gamma, _internal._sample_gamma, @@ -179,16 +179,16 @@ def negative_binomial(k=1, p=1, shape=_Null, dtype=_Null, **kwargs): Parameters ---------- - k : float or Symbol + k : float or Symbol, optional Limit of unsuccessful experiments, > 0. - p : float or Symbol + p : float or Symbol, optional Failure probability in each experiment, >= 0 and <=1. shape : int or tuple of ints The number of samples to draw. If shape is, e.g., `(m, n)` and `k` and `p` are scalars, output shape will be `(m, n)`. If `k` and `p` are Symbols with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[k, p)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_negative_binomial, @@ -207,16 +207,16 @@ def generalized_negative_binomial(mu=1, alpha=1, shape=_Null, dtype=_Null, **kwa Parameters ---------- - mu : float or Symbol + mu : float or Symbol, optional Mean of the negative binomial distribution. - alpha : float or Symbol + alpha : float or Symbol, optional Alpha (dispersion) parameter of the negative binomial distribution. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw. If shape is, e.g., `(m, n)` and `mu` and `alpha` are scalars, output shape will be `(m, n)`. If `mu` and `alpha` are Symbols with shape, e.g., `(x, y)`, then output will have shape `(x, y, m, n)`, where `m*n` samples are drawn for each `[mu, alpha)` pair. - dtype : {'float16','float32', 'float64'} + dtype : {'float16', 'float32', 'float64'}, optional Data type of output samples. Default is 'float32' """ return _random_helper(_internal._random_generalized_negative_binomial, @@ -237,15 +237,15 @@ def multinomial(data, shape=_Null, get_prob=True, dtype='int32', **kwargs): `k` is the number of possible outcomes of each multinomial distribution. For example, data with shape `(m, n, k)` specifies `m*n` multinomial distributions each with `k` possible outcomes. - shape : int or tuple of ints + shape : int or tuple of ints, optional The number of samples to draw from each distribution. If shape is empty one sample will be drawn from each distribution. - get_prob : bool + get_prob : bool, optional If true, a second array containing log likelihood of the drawn samples will also be returned. This is usually used for reinforcement learning, where you can provide reward as head gradient w.r.t. this array to estimate gradient. - dtype : str or numpy.dtype + dtype : str or numpy.dtype, optional Data type of the sample output array. The default is int32. Note that the data type of the log likelihood array is the same with that of `data`. """ @@ -281,3 +281,27 @@ def shuffle(data, **kwargs): """ return _internal._shuffle(data, **kwargs) + + +def randint(low, high, shape=_Null, dtype=_Null, **kwargs): + """Draw random samples from a discrete uniform distribution. + + Samples are uniformly distributed over the half-open interval *[low, high)* + (includes *low*, but excludes *high*). + + Parameters + ---------- + low : int, required + Lower boundary of the output interval. All values generated will be + greater than or equal to low. + high : int, required + Upper boundary of the output interval. All values generated will be + less than high. + shape : int or tuple of ints, optional + The number of samples to draw. If shape is, e.g., `(m, n)` and `low` and + `high` are scalars, output shape will be `(m, n)`. + dtype : {'int32', 'int64'}, optional + Data type of output samples. Default is 'int32' + """ + return _random_helper(_internal._random_randint, None, + [low, high], shape, dtype, kwargs) diff --git a/python/mxnet/symbol/symbol.py b/python/mxnet/symbol/symbol.py index c6575072c..530d72796 100644 --- a/python/mxnet/symbol/symbol.py +++ b/python/mxnet/symbol/symbol.py @@ -1347,7 +1347,7 @@ def simple_bind(self, ctx, grad_req='write', type_dict=None, stype_dict=None, shared_buffer : Dict of string to `NDArray` The dict mapping argument names to the `NDArray` that can be reused for initializing the current executor. This buffer will be checked for reuse if one argument name - of the current executor is not found in `shared_arg_names`. The `NDArray`s are + of the current executor is not found in `shared_arg_names`. The `NDArray` s are expected have default storage type. kwargs : Dict of str->shape diff --git a/python/mxnet/symbol_doc.py b/python/mxnet/symbol_doc.py index 3cb199758..e59437a3c 100644 --- a/python/mxnet/symbol_doc.py +++ b/python/mxnet/symbol_doc.py @@ -44,9 +44,6 @@ - *Examples*: simple and short code snippet showing how to use this operator. It should show typical calling examples and behaviors (e.g. maps an input of what shape to an output of what shape). -- *Regression Test*: longer test code for the operators. We normally do not - expect the users to read those, but they will be executed by `doctest` to - ensure the behavior of each operator does not change unintentionally. """ from __future__ import absolute_import as _abs import re as _re @@ -75,8 +72,6 @@ class ActivationDoc(SymbolDoc): >>> mlp - Regression Test - --------------- ReLU activation >>> test_suites = [ @@ -107,8 +102,6 @@ class DropoutDoc(SymbolDoc): >>> data = Variable('data') >>> data_dp = Dropout(data=data, p=0.2) - Regression Test - --------------- >>> shape = (100, 100) # take larger shapes to be more statistical stable >>> x = np.ones(shape) >>> op = Dropout(p=0.5, name='dp') @@ -141,8 +134,6 @@ class EmbeddingDoc(SymbolDoc): >>> SymbolDoc.get_output_shape(op, letters=(seq_len, batch_size)) {'embed_output': (10L, 64L, 16L)} - Regression Test - --------------- >>> vocab_size, embed_dim = (26, 16) >>> batch_size = 12 >>> word_vecs = test_utils.random_arrays((vocab_size, embed_dim)) @@ -167,8 +158,6 @@ class FlattenDoc(SymbolDoc): >>> SymbolDoc.get_output_shape(flatten, data=(2, 3, 4, 5)) {'flat_output': (2L, 60L)} - Regression Test - --------------- >>> test_dims = [(2, 3, 4, 5), (2, 3), (2,)] >>> op = Flatten(name='flat') >>> for dims in test_dims: @@ -208,8 +197,6 @@ class FullyConnectedDoc(SymbolDoc): >>> net - Regression Test - --------------- >>> dim_in, dim_out = (3, 4) >>> x, w, b = test_utils.random_arrays((10, dim_in), (dim_out, dim_in), (dim_out,)) >>> op = FullyConnected(num_hidden=dim_out, name='FC') diff --git a/python/mxnet/test_utils.py b/python/mxnet/test_utils.py index 0bb28a0ef..0a4d17dc2 100644 --- a/python/mxnet/test_utils.py +++ b/python/mxnet/test_utils.py @@ -261,10 +261,14 @@ def rand_sparse_ndarray(shape, stype, density=None, dtype=None, distribution=Non Parameters ---------- shape: list or tuple - stype: str, valid values: "csr" or "row_sparse" - density, optional: float, should be between 0 and 1 - distribution, optional: str, valid values: "uniform" or "powerlaw" - dtype, optional: numpy.dtype, default value is None + stype: str + valid values: "csr" or "row_sparse" + density: float, optional + should be between 0 and 1 + distribution: str, optional + valid values: "uniform" or "powerlaw" + dtype: numpy.dtype, optional + default value is None Returns ------- @@ -336,7 +340,7 @@ def rand_sparse_ndarray(shape, stype, density=None, dtype=None, distribution=Non assert(False), "unknown storage type" return False -def rand_ndarray(shape, stype, density=None, dtype=None, +def rand_ndarray(shape, stype='default', density=None, dtype=None, modifier_func=None, shuffle_csr_indices=False, distribution=None): if stype == 'default': arr = mx.nd.array(random_arrays(shape), dtype=dtype) @@ -801,10 +805,12 @@ def check_numeric_gradient(sym, location, aux_states=None, numeric_eps=1e-3, rto location : list or tuple or dict Argument values used as location to compute gradient - - if type is list of numpy.ndarray + - if type is list of numpy.ndarray, \ inner elements should have the same order as mxnet.sym.list_arguments(). - - if type is dict of str -> numpy.ndarray + + - if type is dict of str -> numpy.ndarray, \ maps the name of arguments to the corresponding numpy.ndarray. + *In either case, value of all the arguments must be provided.* aux_states : list or tuple or dict, optional The auxiliary states required when generating the executor for the symbol. @@ -825,7 +831,7 @@ def check_numeric_gradient(sym, location, aux_states=None, numeric_eps=1e-3, rto References --------- - ..[1] https://github.com/Theano/Theano/blob/master/theano/gradient.py + [1] https://github.com/Theano/Theano/blob/master/theano/gradient.py """ assert dtype in (np.float16, np.float32, np.float64) # cannot use finite differences with small eps without high precision @@ -1379,7 +1385,7 @@ def list_gpus(): for cmd in nvidia_smi: try: re = subprocess.check_output([cmd, "-L"], universal_newlines=True) - except OSError: + except (subprocess.CalledProcessError, OSError): pass return range(len([i for i in re.split('\n') if 'GPU' in i])) @@ -1840,21 +1846,23 @@ def var_check(generator, sigma, nsamples=1000000): def chi_square_check(generator, buckets, probs, nsamples=1000000): """Run the chi-square test for the generator. The generator can be both continuous and discrete. - If the generator is continuous, the buckets should contain tuples of (range_min, range_max) and - the probs should be the corresponding ideal probability within the specific ranges. - Otherwise, the buckets should be the possible output of the discrete distribution and the probs - should be groud-truth probability. + + If the generator is continuous, the buckets should contain tuples of (range_min, range_max) \ + and the probs should be the corresponding ideal probability within the specific ranges. \ + Otherwise, the buckets should contain all the possible values generated over the discrete distribution and the \ + probs should be groud-truth probability. Usually the user is required to specify the probs parameter. - After obtatining the p value, we could further use the standard p > 0.05 threshold to get - the final result. + After obtaining the p value, we could further use the standard p > 0.05 (alpha) threshold to get \ + the final result. Examples:: - buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.norm.ppf(x, 0, 1), 5) - generator = lambda x: np.random.normal(0, 1.0, size=x) - p = chi_square_check(generator=generator, buckets=buckets, probs=probs) - assert(p > 0.05) + + buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.norm.ppf(x, 0, 1), 5) + generator = lambda x: np.random.normal(0, 1.0, size=x) + p = chi_square_check(generator=generator, buckets=buckets, probs=probs) + assert(p > 0.05) Parameters ---------- @@ -1863,8 +1871,8 @@ def chi_square_check(generator, buckets, probs, nsamples=1000000): generator(N) should generate N random samples. buckets: list of tuple or list of number The buckets to run the chi-square the test. Make sure that the buckets cover - the whole range of the distribution. Also, the buckets must be in ascending order and have - no intersection + the whole range of the distribution. Also, the buckets must be in ascending order and have + no intersection probs: list or tuple The ground-truth probability of the random value fall in a specific bucket. nsamples:int @@ -1898,21 +1906,23 @@ def chi_square_check(generator, buckets, probs, nsamples=1000000): buckets_npy[i * 2 + 1] = buckets[i][1] else: continuous_dist = False - buckets_npy = np.array(buckets) expected_freq = (nsamples * np.array(probs, dtype=np.float32)).astype(np.int32) if continuous_dist: sample_bucket_ids = np.searchsorted(buckets_npy, samples, side='right') else: - sample_bucket_ids = samples + sample_bucket_ids = np.array(samples) if continuous_dist: sample_bucket_ids = sample_bucket_ids // 2 obs_freq = np.zeros(shape=len(buckets), dtype=np.int) - for i in range(len(buckets)): - obs_freq[i] = (sample_bucket_ids == i).sum() + for i, _ in enumerate(buckets): + if continuous_dist: + obs_freq[i] = (sample_bucket_ids == i).sum() + else: + obs_freq[i] = (sample_bucket_ids == buckets[i]).sum() _, p = ss.chisquare(f_obs=obs_freq, f_exp=expected_freq) return p, obs_freq, expected_freq -def verify_generator(generator, buckets, probs, nsamples=1000000, nrepeat=5, success_rate=0.15): +def verify_generator(generator, buckets, probs, nsamples=1000000, nrepeat=5, success_rate=0.25, alpha=0.05): """Verify whether the generator is correct using chi-square testing. The test is repeated for "nrepeat" times and we check if the success rate is @@ -1935,6 +1945,8 @@ def verify_generator(generator, buckets, probs, nsamples=1000000, nrepeat=5, suc The times to repeat the test success_rate: float The desired success rate + alpha: float + The desired threshold for type-I error i.e. when a true null hypothesis is rejected Returns ------- @@ -1950,7 +1962,7 @@ def verify_generator(generator, buckets, probs, nsamples=1000000, nrepeat=5, suc cs_ret_l.append(cs_ret) obs_freq_l.append(obs_freq) expected_freq_l.append(expected_freq) - success_num = (np.array(cs_ret_l) > 0.05).sum() + success_num = (np.array(cs_ret_l) > alpha).sum() if success_num < nrepeat * success_rate: raise AssertionError("Generator test fails, Chi-square p=%s, obs_freq=%s, expected_freq=%s." "\nbuckets=%s, probs=%s" @@ -1998,3 +2010,20 @@ def compare_optimizer(opt1, opt2, shape, dtype, w_stype='default', g_stype='defa if compare_states: compare_ndarray_tuple(state1, state2, rtol=rtol, atol=atol) assert_almost_equal(w1.asnumpy(), w2.asnumpy(), rtol=rtol, atol=atol) + +class EnvManager(object): + """Environment variable setter and unsetter via with idiom""" + def __init__(self, key, val): + self._key = key + self._next_val = val + self._prev_val = None + + def __enter__(self): + self._prev_val = os.environ.get(self._key) + os.environ[self._key] = self._next_val + + def __exit__(self, ptype, value, trace): + if self._prev_val: + os.environ[self._key] = self._prev_val + else: + del os.environ[self._key] diff --git a/python/mxnet/torch.py b/python/mxnet/torch.py index dde1eed08..fdf8654a4 100644 --- a/python/mxnet/torch.py +++ b/python/mxnet/torch.py @@ -85,7 +85,7 @@ def _make_torch_function(handle): 'Invoke with\n{res}= mxnet.th.{name}(Parameters)\nor\n'+ 'mxnet.th.{name}({res}, Parameters).\n\n' + '{param_str}\n' + - 'Reference: ' + + 'References: ' + 'https://github.com/torch/torch7/blob/master/doc/maths.md\n').format( name=func_name[4:], param_str=param_str, res=res)) diff --git a/python/mxnet/visualization.py b/python/mxnet/visualization.py index a0eb253cc..1ebdcb54f 100644 --- a/python/mxnet/visualization.py +++ b/python/mxnet/visualization.py @@ -57,9 +57,22 @@ def print_summary(symbol, shape=None, line_length=120, positions=[.44, .64, .74, Rotal length of printed lines positions: list Relative or absolute positions of log elements in each line. + Returns ------ None + + Notes + ----- + If ``mxnet`` is imported, the visualization module can be used in its short-form. + For example, if we ``import mxnet`` as follows:: + + import mxnet + + this method in visualization module can be used in its short-form as:: + + mxnet.viz.print_summary(...) + """ if not isinstance(symbol, Symbol): raise TypeError("symbol must be Symbol") @@ -213,12 +226,14 @@ def plot_network(symbol, title="plot", save_format='pdf', shape=None, node_attrs input symbol names (str) to the corresponding tensor shape (tuple). node_attrs: dict, optional Specifies the attributes for nodes in the generated visualization. `node_attrs` is - a dictionary of Graphviz attribute names and values. For example, - ``node_attrs={"shape":"oval","fixedsize":"false"}`` - will use oval shape for nodes and allow variable sized nodes in the visualization. + a dictionary of Graphviz attribute names and values. For example:: + + node_attrs={"shape":"oval","fixedsize":"false"} + + will use oval shape for nodes and allow variable sized nodes in the visualization. hide_weights: bool, optional - If True (default), then inputs with names of form *_weight (corresponding to weight - tensors) or *_bias (corresponding to bias vectors) will be hidden for a cleaner + If True (default), then inputs with names of form *_weight* (corresponding to weight + tensors) or *_bias* (corresponding to bias vectors) will be hidden for a cleaner visualization. Returns @@ -236,6 +251,18 @@ def plot_network(symbol, title="plot", save_format='pdf', shape=None, node_attrs >>> digraph = mx.viz.plot_network(net, shape={'data':(100,200)}, ... node_attrs={"fixedsize":"false"}) >>> digraph.view() + + Notes + ----- + If ``mxnet`` is imported, the visualization module can be used in its short-form. + For example, if we ``import mxnet`` as follows:: + + import mxnet + + this method in visualization module can be used in its short-form as:: + + mxnet.viz.plot_network(...) + """ # todo add shape support try: diff --git a/python/setup.py b/python/setup.py index d6e02b3ce..11dd440fb 100644 --- a/python/setup.py +++ b/python/setup.py @@ -32,7 +32,7 @@ else: from setuptools import setup from setuptools.extension import Extension - kwargs = {'install_requires': ['numpy<=1.15.2,>=1.8.2', 'requests<2.19.0,>=2.18.4', 'graphviz<0.9.0,>=0.8.1'], 'zip_safe': False} + kwargs = {'install_requires': ['numpy<=1.15.2,>=1.8.2', 'requests>=2.20.0,<3', 'graphviz<0.9.0,>=0.8.1'], 'zip_safe': False} with_cython = False if '--with-cython' in sys.argv: diff --git a/readthedocs.yml b/readthedocs.yml index 3787245bd..70e32ae4b 100644 --- a/readthedocs.yml +++ b/readthedocs.yml @@ -1,3 +1,20 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + formats: - none requirements_file: docs/requirements.txt diff --git a/scala-package/.gitignore b/scala-package/.gitignore index 0f860e628..22b12b31d 100644 --- a/scala-package/.gitignore +++ b/scala-package/.gitignore @@ -1,5 +1,10 @@ .flattened-pom.xml core/src/main/scala/org/apache/mxnet/NDArrayAPIBase.scala core/src/main/scala/org/apache/mxnet/NDArrayBase.scala +core/src/main/scala/org/apache/mxnet/NDArrayRandomAPIBase.scala +core/src/main/scala/org/apache/mxnet/javaapi/NDArrayBase.scala core/src/main/scala/org/apache/mxnet/SymbolAPIBase.scala core/src/main/scala/org/apache/mxnet/SymbolBase.scala +core/src/main/scala/org/apache/mxnet/SymbolRandomAPIBase.scala +examples/scripts/infer/images/ +examples/scripts/infer/models/ diff --git a/scala-package/README.md b/scala-package/README.md index 79caaf309..3859e5f32 100644 --- a/scala-package/README.md +++ b/scala-package/README.md @@ -1,191 +1,202 @@ - Deep Learning for Scala/Java +MXNet Package for Scala/Java ===== -[![Build Status](https://travis-ci.org/dmlc/mxnet.svg?branch=master)](https://travis-ci.org/dmlc/mxnet) -[![GitHub license](http://dmlc.github.io/img/apache2.svg)](./LICENSE) - -Here you find the MXNet Scala Package! -It brings flexible and efficient GPU/CPU computing and state-of-art deep learning to JVM. +The MXNet Scala/Java Package brings flexible and efficient GPU/CPU computing and state-of-art deep learning to JVM. - It enables you to write seamless tensor/matrix computation with multiple GPUs in Scala, Java and other languages built on JVM. - It also enables you to construct and customize the state-of-art deep learning models in JVM languages, and apply them to tasks such as image classification and data science challenges. +- The Scala/Java Inferece APIs provides an easy out of the box solution for loading pre-trained MXNet models and running inference on them. -Install ------------- - -Technically, all you need is the `mxnet-full_2.10-{arch}-{xpu}-0.1.1.jar` in your classpath. -It will automatically extract the native library to a tempfile and load it. +Pre-Built Maven Packages +------------------------ + +### Stable ### + +The MXNet Scala/Java packages can be easily included in your Maven managed project. +The stable jar files for the packages are available on the [MXNet Maven Package Repository](https://search.maven.org/search?q=g:org.apache.mxnet) +Currently we provide packages for Linux (Ubuntu 16.04) (CPU and GPU) and macOS (CPU only). Stable packages for Windows and CentOS will come soon. For now, if you have a CentOS machine, follow the ```Build From Source``` section below. -Currently we provide `linux-x86_64-gpu`, `linux-x86_64-cpu` and `osx-x86_64-cpu`. Support for Windows will come soon. -Use the following dependency in maven, change the artifactId according to your own architecture, e.g., `mxnet-full_2.10-osx-x86_64-cpu` for OSX (and cpu-only). +To add MXNet Scala/Java package to your project, add the dependency as shown below corresponding to your platform, under the ```dependencies``` tag in your project's ```pom.xml``` : + +**Linux GPU** + +maven badge ```HTML org.apache.mxnet - mxnet-full_2.10-linux-x86_64-gpu - 0.1.1 + mxnet-full_2.11-linux-x86_64-gpu + [1.3.1,) ``` -You can also use `mxnet-core_2.10-0.1.1.jar` and put the compiled native library somewhere in your load path. +**Linux CPU** + +maven badge + +```HTML + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-cpu + [1.3.1,) + +``` + +**macOS CPU** + +maven badge ```HTML org.apache.mxnet - mxnet-core_2.10 - 0.1.1 + mxnet-full_2.11-osx-x86_64-cpu + [1.3.1,) ``` -If you have some native libraries conflict with the ones in the provided 'full' jar (e.g., you use openblas instead of atlas), this is a recommended way. -Refer to the next section for how to build it from the very source. +**Note:** ```[1.3.1,)<\version>``` indicates that we will fetch packages with version 1.3.1 or higher. This will always ensure that the pom.xml is able to fetch the latest and greatest jar files from Maven. -Build ------------- +### Nightly ### -Checkout the [Installation Guide](http://mxnet.incubator.apache.org/install/index.html) contains instructions to install mxnet. -Then you can compile the Scala Package by +Apart from these, the nightly builds representing the bleeding edge development on Scala/Java packages are also available on the [MXNet Maven Nexus Package Repository](https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~~~). +Currently we provide nightly packages for Linux (CPU and GPU) and MacOS (CPU only). The Linux nightly jar files also work on CentOS. Nightly packages for Windows will come soon. + +Add the following ```repository``` to your project's ```pom.xml``` file : + +````html + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + +```` + +Also, add the dependency which corresponds to your platform to the ```dependencies``` tag : + +**Linux GPU** + +maven badge + +```HTML + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-gpu + [1.5.0,) + +``` + +**Linux CPU** + +maven badge + +```HTML + + org.apache.mxnet + mxnet-full_2.11-linux-x86_64-cpu + [1.5.0,) + +``` + +**macOS CPU** + +maven badge +```HTML + + org.apache.mxnet + mxnet-full_2.11-osx-x86_64-cpu + [1.5.0,) + +``` + +**Note:** ```[1.5.0,)<\version>``` indicates that we will fetch packages with version 1.5.0 or higher. This will always ensure that the pom.xml is able to fetch the latest and greatest jar files from Maven Snapshot repository. + +Build From Source +----------------- + +Checkout the [Installation Guide](http://mxnet.incubator.apache.org/install/index.html) contains instructions to install mxnet package and build it from source. Scala maven build assume you already have a ``lib/libmxnet.so`` file. +If you have built MXNet from source and are looking to setup Scala from that point, you may simply run the following from the MXNet source root, Scala build will detect your platform (OSX/Linux) and libmxnet.so flavor (CPU/GPU): ```bash -make scalapkg +cd scala-package +mvn install ``` -(Optional) run unit/integration tests by +You can also run the unit tests and integration tests on the Scala Package by : ```bash -make scalaunittest -make scalaintegrationtest +cd scala-package +mvn integration-test -DskipTests=false ``` -Or run a subset of unit tests by, e.g., +Or run a subset of unit tests, for e.g., ```bash -make SCALA_TEST_ARGS=-Dsuites=org.apache.mxnet.NDArraySuite scalaunittest +cd scala-package +mvn -Dsuites=org.apache.mxnet.NDArraySuite integration-test ``` If everything goes well, you will find jars for `assembly`, `core` and `example` modules. -Also it produces the native library in `native/{your-architecture}/target`, which you can use to cooperate with the `core` module. +Also it produces the native library in `native/target`, which you can use to cooperate with the `core` module. + +Deploy to repository +-------------------- -Once you've downloaded and unpacked MNIST dataset to `./data/`, run the training example by +By default, `maven deploy` will deploy artifacts to local file system, you can file then in: ``scala-package/deploy/target/repo`` folder. + +For nightly build in CI, a snapshot build will be uploaded to apache repository with follow command: ```bash -java -Xmx4G -cp \ - scala-package/assembly/{your-architecture}/target/*:scala-package/examples/target/*:scala-package/examples/target/classes/lib/* \ - org.apache.mxnet.examples.imclassification.TrainMnist \ - --data-dir=./data/ \ - --num-epochs=10 \ - --network=mlp \ - --cpus=0,1,2,3 +cd scala-package +mvn deploy -Pnightly ``` -If you've compiled with `USE_DIST_KVSTORE` enabled, the python tools in `mxnet/tracker` can be used to launch distributed training. -The following command runs the above example using 2 worker nodes (and 2 server nodes) in local. Refer to [Distributed Training](http://mxnet.incubator.apache.org/how_to/multi_devices.html) for more details. +Use following command to deploy release build (push artifacts to apache staging repository): ```bash -tracker/dmlc_local.py -n 2 -s 2 \ - java -Xmx4G -cp \ - scala-package/assembly/{your-architecture}/target/*:scala-package/examples/target/*:scala-package/examples/target/classes/lib/* \ - org.apache.mxnet.examples.imclassification.TrainMnist \ - --data-dir=./data/ \ - --num-epochs=10 \ - --network=mlp \ - --cpus=0 \ - --kv-store=dist_sync +cd scala-package +mvn deploy -Pstaging ``` -Change the arguments and have fun! +Examples & Usage +------- +- To set up the Scala Project using IntelliJ IDE on macOS follow the instructions [here](https://mxnet.incubator.apache.org/tutorials/scala/mxnet_scala_on_intellij.html). +- Several examples on using the Scala APIs are provided in the [Scala Examples Folder](https://github.com/apache/incubator-mxnet/tree/master/scala-package/examples/) -Usage +Scala Training APIs ------- -Here is a Scala example of what training a simple 3-layer multilayer perceptron on MNIST looks like. You can download the MNIST dataset using [get_mnist_data script](https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/scripts/get_mnist_data.sh). - -```scala -import org.apache.mxnet._ -import org.apache.mxnet.optimizer.SGD - -// model definition -val data = Symbol.Variable("data") -val fc1 = Symbol.FullyConnected(name = "fc1")()(Map("data" -> data, "num_hidden" -> 128)) -val act1 = Symbol.Activation(name = "relu1")()(Map("data" -> fc1, "act_type" -> "relu")) -val fc2 = Symbol.FullyConnected(name = "fc2")()(Map("data" -> act1, "num_hidden" -> 64)) -val act2 = Symbol.Activation(name = "relu2")()(Map("data" -> fc2, "act_type" -> "relu")) -val fc3 = Symbol.FullyConnected(name = "fc3")()(Map("data" -> act2, "num_hidden" -> 10)) -val mlp = Symbol.SoftmaxOutput(name = "sm")()(Map("data" -> fc3)) - -// load MNIST dataset -val trainDataIter = IO.MNISTIter(Map( - "image" -> "data/train-images-idx3-ubyte", - "label" -> "data/train-labels-idx1-ubyte", - "data_shape" -> "(1, 28, 28)", - "label_name" -> "sm_label", - "batch_size" -> "50", - "shuffle" -> "1", - "flat" -> "0", - "silent" -> "0", - "seed" -> "10")) - -val valDataIter = IO.MNISTIter(Map( - "image" -> "data/t10k-images-idx3-ubyte", - "label" -> "data/t10k-labels-idx1-ubyte", - "data_shape" -> "(1, 28, 28)", - "label_name" -> "sm_label", - "batch_size" -> "50", - "shuffle" -> "1", - "flat" -> "0", "silent" -> "0")) - -// setup model and fit the training data -val model = FeedForward.newBuilder(mlp) - .setContext(Context.cpu()) - .setNumEpoch(10) - .setOptimizer(new SGD(learningRate = 0.1f, momentum = 0.9f, wd = 0.0001f)) - .setTrainData(trainDataIter) - .setEvalData(valDataIter) - .build() -``` +- Module API : +[The Module API](https://mxnet.incubator.apache.org/api/scala/module.html) provides an intermediate and high-level interface for performing computation with neural networks in MXNet. Modules provide high-level APIs for training, predicting, and evaluating. -Predict using the model in the following way: - -```scala -val probArrays = model.predict(valDataIter) -// in this case, we do not have multiple outputs -require(probArrays.length == 1) -val prob = probArrays(0) - -// get real labels -import scala.collection.mutable.ListBuffer -valDataIter.reset() -val labels = ListBuffer.empty[NDArray] -while (valDataIter.hasNext) { - val evalData = valDataIter.next() - labels += evalData.label(0).copy() -} -val y = NDArray.concatenate(labels) - -// get predicted labels -val py = NDArray.argmax_channel(prob) -require(y.shape == py.shape) - -// calculate accuracy -var numCorrect = 0 -var numInst = 0 -for ((labelElem, predElem) <- y.toArray zip py.toArray) { - if (labelElem == predElem) { - numCorrect += 1 - } - numInst += 1 -} -val acc = numCorrect.toFloat / numInst -println(s"Final accuracy = $acc") -``` +- KVStore API : +To run training over multiple GPUs and multiple hosts, one can use the [KVStore API](https://mxnet.incubator.apache.org/api/scala/kvstore.html). + +- IO/Data Loading : +MXNet Scala provides APIs for preparing data to feed as an input to models. Check out [Data Loading API](https://mxnet.incubator.apache.org/api/scala/io.html) for more info. + +Other available Scala APIs for training can be found [here](https://mxnet.incubator.apache.org/api/scala/index.html). + + +Scala Inference APIs +------- +The [Scala Inference APIs](https://mxnet.incubator.apache.org/api/scala/infer.html) provide an easy, out of the box solution to load a pre-trained MXNet model and run inference on it. The Inference APIs are present in the [Infer Package](https://github.com/apache/incubator-mxnet/tree/master/scala-package/infer) under the MXNet Scala Package repository, while the documentation for the Infer API is available [here](https://mxnet.incubator.apache.org/api/scala/docs/index.html#org.apache.mxnet.infer.package). -Release +Java Inference APIs ------- -- Version 0.1.1, March 24, 2016. - - Bug fix for MAE & MSE metrics. -- Version 0.1.0, March 22, 2016. +The [Java Inference APIs](http://mxnet.incubator.apache.org/api/java/index.html) also provide an easy, out of the box solution to load a pre-trained MXNet model and run inference on it. The Inference APIs are present in the [Infer Package](https://github.com/apache/incubator-mxnet/tree/master/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi) under the MXNet Scala Package repository, while the documentation for the Infer API is available [here](https://mxnet.incubator.apache.org/api/java/docs/index.html#org.apache.mxnet.infer.package). +More APIs will be added to the Java Inference APIs soon. + +JVM Memory Management +------- +The Scala/Java APIs also provide an automated resource management system, thus making it easy to manage the native memory footprint without any degradation in performance. +More details about JVM Memory Management are available [here](https://github.com/apache/incubator-mxnet/blob/master/scala-package/memory-management.md). License ------- MXNet Scala Package is licensed under [Apache-2](https://github.com/apache/incubator-mxnet/blob/master/scala-package/LICENSE) license. + +MXNet uses some 3rd party softwares. Following 3rd party license files are bundled inside Scala jar file: +* cub/LICENSE.TXT +* mkldnn/external/mklml_mac_2019.0.1.20180928/license.txt diff --git a/scala-package/assembly/linux-x86_64-cpu/pom.xml b/scala-package/assembly/linux-x86_64-cpu/pom.xml deleted file mode 100644 index a6a7bbd94..000000000 --- a/scala-package/assembly/linux-x86_64-cpu/pom.xml +++ /dev/null @@ -1,127 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-full-parent_2.11 - 1.3.1-SNAPSHOT - ../pom.xml - - - mxnet-full_2.11-linux-x86_64-cpu - MXNet Scala Package - Full Linux-x86_64 CPU-only - jar - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - - - org.apache.mxnet - libmxnet-scala-linux-x86_64-cpu - 1.3.1-SNAPSHOT - so - - - org.apache.mxnet - mxnet-infer_${scala.binary.version} - 1.3.1-SNAPSHOT - - - - - - - org.codehaus.mojo - flatten-maven-plugin - - - - remove - - - - - - flatten - process-resources - - flatten - - - - - flatten.clean - clean - - clean - - - - - - org.apache.maven.plugins - maven-deploy-plugin - false - - false - - - - org.apache.maven.plugins - maven-install-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - - - binary-jar - package - - single - - - false - - src/main/assembly/assembly.xml - - - - - sources-jar - package - - single - - - true - sources - - ${project.parent.basedir}/src/source.xml - - - - - javadoc-jar - package - - single - - - true - javadoc - - ${project.parent.basedir}/src/javadoc.xml - - - - - - - - diff --git a/scala-package/assembly/linux-x86_64-cpu/src/main/assembly/assembly.xml b/scala-package/assembly/linux-x86_64-cpu/src/main/assembly/assembly.xml deleted file mode 100644 index a574f8af2..000000000 --- a/scala-package/assembly/linux-x86_64-cpu/src/main/assembly/assembly.xml +++ /dev/null @@ -1,28 +0,0 @@ - - full - - jar - - false - - - - *:*:jar - - / - true - true - runtime - - - lib/native - libmxnet-scala.so - false - false - false - - org.apache.mxnet:libmxnet-scala-linux-x86_64-cpu:so - - - - diff --git a/scala-package/assembly/linux-x86_64-gpu/pom.xml b/scala-package/assembly/linux-x86_64-gpu/pom.xml deleted file mode 100644 index 992348908..000000000 --- a/scala-package/assembly/linux-x86_64-gpu/pom.xml +++ /dev/null @@ -1,127 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-full-parent_2.11 - 1.3.1-SNAPSHOT - ../pom.xml - - - mxnet-full_2.11-linux-x86_64-gpu - MXNet Scala Package - Full Linux-x86_64 GPU - jar - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - - - org.apache.mxnet - libmxnet-scala-linux-x86_64-gpu - 1.3.1-SNAPSHOT - so - - - org.apache.mxnet - mxnet-infer_${scala.binary.version} - 1.3.1-SNAPSHOT - - - - - - - org.codehaus.mojo - flatten-maven-plugin - - - - remove - - - - - - flatten - process-resources - - flatten - - - - - flatten.clean - clean - - clean - - - - - - org.apache.maven.plugins - maven-deploy-plugin - false - - false - - - - org.apache.maven.plugins - maven-install-plugin - - - org.apache.maven.plugins - maven-assembly-plugin - - - binary-jar - package - - single - - - false - - src/main/assembly/assembly.xml - - - - - sources-jar - package - - single - - - true - sources - - ${project.parent.basedir}/src/source.xml - - - - - javadoc-jar - package - - single - - - true - javadoc - - ${project.parent.basedir}/src/javadoc.xml - - - - - - - - diff --git a/scala-package/assembly/linux-x86_64-gpu/src/main/assembly/assembly.xml b/scala-package/assembly/linux-x86_64-gpu/src/main/assembly/assembly.xml deleted file mode 100644 index 3a064bf9f..000000000 --- a/scala-package/assembly/linux-x86_64-gpu/src/main/assembly/assembly.xml +++ /dev/null @@ -1,28 +0,0 @@ - - full - - jar - - false - - - - *:*:jar - - / - true - true - runtime - - - lib/native - libmxnet-scala.so - false - false - false - - org.apache.mxnet:libmxnet-scala-linux-x86_64-gpu:so - - - - diff --git a/scala-package/assembly/osx-x86_64-cpu/main/assembly/assembly.xml b/scala-package/assembly/osx-x86_64-cpu/main/assembly/assembly.xml deleted file mode 100644 index fecafecad..000000000 --- a/scala-package/assembly/osx-x86_64-cpu/main/assembly/assembly.xml +++ /dev/null @@ -1,30 +0,0 @@ - - full - - jar - - false - - - - *:*:jar - - / - true - true - runtime - - - lib/native - ${artifact.artifactId}${dashClassifier?}.${artifact.extension} - false - false - false - - *:*:dll:* - *:*:so:* - *:*:jnilib:* - - - - diff --git a/scala-package/assembly/osx-x86_64-cpu/pom.xml b/scala-package/assembly/osx-x86_64-cpu/pom.xml deleted file mode 100644 index d8773334b..000000000 --- a/scala-package/assembly/osx-x86_64-cpu/pom.xml +++ /dev/null @@ -1,123 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-full-parent_2.11 - 1.3.1-SNAPSHOT - ../pom.xml - - - mxnet-full_2.11-osx-x86_64-cpu - MXNet Scala Package - Full OSX-x86_64 CPU-only - jar - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - - - org.apache.mxnet - libmxnet-scala-osx-x86_64-cpu - 1.3.1-SNAPSHOT - jnilib - - - org.apache.mxnet - mxnet-infer_${scala.binary.version} - 1.3.1-SNAPSHOT - - - - - - - org.codehaus.mojo - flatten-maven-plugin - - - - remove - - - - - - flatten - process-resources - - flatten - - - - - flatten.clean - clean - - clean - - - - - - org.apache.maven.plugins - maven-deploy-plugin - false - - false - - - - org.apache.maven.plugins - maven-assembly-plugin - - - binary-jar - package - - single - - - false - - src/main/assembly/assembly.xml - - - - - sources-jar - package - - single - - - true - sources - - ${project.parent.basedir}/src/source.xml - - - - - javadoc-jar - package - - single - - - true - javadoc - - ${project.parent.basedir}/src/javadoc.xml - - - - - - - - diff --git a/scala-package/assembly/osx-x86_64-cpu/src/main/assembly/assembly.xml b/scala-package/assembly/osx-x86_64-cpu/src/main/assembly/assembly.xml deleted file mode 100644 index bdbd09f17..000000000 --- a/scala-package/assembly/osx-x86_64-cpu/src/main/assembly/assembly.xml +++ /dev/null @@ -1,28 +0,0 @@ - - full - - jar - - false - - - - *:*:jar - - / - true - true - runtime - - - lib/native - libmxnet-scala.jnilib - false - false - false - - org.apache.mxnet:libmxnet-scala-osx-x86_64-cpu:jnilib - - - - diff --git a/scala-package/assembly/pom.xml b/scala-package/assembly/pom.xml index 127e69e07..00aa8682f 100644 --- a/scala-package/assembly/pom.xml +++ b/scala-package/assembly/pom.xml @@ -5,106 +5,109 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-full-parent_2.11 - MXNet Scala Package - Full Parent + mxnet-full_2.11 + Assembly Scala Package pom + + ${project.parent.basedir}/.. + + + + + org.apache.mxnet + mxnet-core + INTERNAL + + + org.apache.mxnet + libmxnet-scala + INTERNAL + ${libtype} + + + org.apache.mxnet + mxnet-infer + INTERNAL + + + - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - - release + staging - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-source-plugin - - - package - - jar-no-fork - - - true - - - - - - org.apache.maven.plugins - maven-javadoc-plugin - - - package - - jar - - - true - - commons-codec:* - org.scala-lang:* - log4j:* - org.slf4j:* - - - - - org.apache.maven.plugins maven-gpg-plugin + 1.6 sign-artifacts - verify + deploy sign - - org.sonatype.plugins - nexus-staging-maven-plugin - true - - ossrh - https://oss.sonatype.org/ - true - - + + + + + org.apache.maven.plugins + maven-assembly-plugin + + + binary-jar + package + + single + + + false + + src/main/assembly/assembly.xml + + + + + sources-jar + package + + single + + + true + + src/main/assembly/source.xml + + + + + javadoc-jar + package + + single + + + true + + src/main/assembly/javadoc.xml + + + + + + + diff --git a/scala-package/assembly/src/javadoc.xml b/scala-package/assembly/src/javadoc.xml deleted file mode 100644 index 9d0be80e7..000000000 --- a/scala-package/assembly/src/javadoc.xml +++ /dev/null @@ -1,24 +0,0 @@ - - bundle - - jar - - false - - - - - - - org.apache.mxnet:mxnet-core_${scala.binary.version} - - true - - false - javadoc - / - true - - - - \ No newline at end of file diff --git a/scala-package/assembly/src/main/assembly/assembly.xml b/scala-package/assembly/src/main/assembly/assembly.xml new file mode 100644 index 000000000..7525df883 --- /dev/null +++ b/scala-package/assembly/src/main/assembly/assembly.xml @@ -0,0 +1,57 @@ + + full + + jar + + false + + + + *:*:jar + + + org.scala-lang:* + org.scala-lang.modules:* + + / + true + true + runtime + + + lib/native + libmxnet-scala.${libtype} + false + false + false + + org.apache.mxnet:libmxnet-scala:${libtype} + + + + + + ${MXNET_DIR}/lib + + libmxnet.so + libgfortran.so.3 + libquadmath.so.0 + libiomp5.so + libiomp5.dylib + libmklml_intel.so + libmklml.dylib + libmkldnn.so.0 + libmkldnn.0.dylib + + lib/native + + + ${MXNET_DIR}/3rdparty + + cub/LICENSE.TXT + mkldnn/external/mklml_mac_2019.0.1.20180928/license.txt + + / + + + diff --git a/scala-package/assembly/src/main/assembly/javadoc.xml b/scala-package/assembly/src/main/assembly/javadoc.xml new file mode 100644 index 000000000..8f30a2618 --- /dev/null +++ b/scala-package/assembly/src/main/assembly/javadoc.xml @@ -0,0 +1,15 @@ + + bundle + + jar + + false + + + ${rootdir}/core/target/site/scaladocs + / + + + diff --git a/scala-package/assembly/src/main/assembly/source.xml b/scala-package/assembly/src/main/assembly/source.xml new file mode 100644 index 000000000..87fcde360 --- /dev/null +++ b/scala-package/assembly/src/main/assembly/source.xml @@ -0,0 +1,19 @@ + + src + + + jar + + false + + + ${rootdir}/core/src/main/scala + + **\/*.scala + + / + + + diff --git a/scala-package/assembly/src/source.xml b/scala-package/assembly/src/source.xml deleted file mode 100644 index 0cbdcadf8..000000000 --- a/scala-package/assembly/src/source.xml +++ /dev/null @@ -1,32 +0,0 @@ - - src - - - jar - - false - - - true - - - org.apache.mxnet:mxnet-core_${scala.binary.version} - - - false - - - - src/main/scala - - **\/*.scala - - / - - - - - - \ No newline at end of file diff --git a/scala-package/core/pom.xml b/scala-package/core/pom.xml index e93169f08..7264c39e8 100644 --- a/scala-package/core/pom.xml +++ b/scala-package/core/pom.xml @@ -5,56 +5,74 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-core_2.11 + mxnet-core MXNet Scala Package - Core - - - unittest - - false - - - - integrationtest - - true - - - - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - + + false + - org.apache.maven.plugins - maven-deploy-plugin - - true - + org.codehaus.mojo + native-maven-plugin + true + + + javah + verify + + default + ${project.build.directory}/custom-javah + ${basedir} + org_apache_mxnet_native_c_api.h + + org.apache.mxnet.LibInfo + + + + javah + + + + + + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + verify-javah + verify + + exec + + + diff + ${project.build.directory}/custom-javah/org_apache_mxnet_native_c_api.h ${project.parent.basedir}/native/src/main/native/org_apache_mxnet_native_c_api.h + + + + apidoc-generation + generate-sources + + exec + + + java + ${project.parent.basedir} + -classpath %classpath:${rootdir}/init/target/classes:${rootdir}/macros/target/classes -Djava.library.path=${rootdir}/native/target org.apache.mxnet.APIDocGenerator ${rootdir}/core/src/main/scala/org/apache/mxnet/ + + + + org.apache.maven.plugins maven-jar-plugin @@ -70,29 +88,37 @@ org.apache.maven.plugins maven-compiler-plugin + + org.apache.maven.plugins + maven-clean-plugin + 3.1.0 + + + + src/main/scala/org/apache/mxnet + + NDArrayAPIBase.scala + NDArrayBase.scala + NDArrayRandomAPIBase.scala + javaapi/NDArrayBase.scala + SymbolAPIBase.scala + SymbolRandomAPIBase.scala + + false + + + + org.scalatest scalatest-maven-plugin - ${skipTests} - -Djava.library.path=${project.parent.basedir}/native/${platform}/target \ + -Djava.library.path=${project.parent.basedir}/native/target \ -Dlog4j.configuration=file://${project.basedir}/src/test/resources/log4j.properties - - org.apache.maven.plugins - maven-surefire-plugin - 2.22.0 - - false - - - - org.scalastyle - scalastyle-maven-plugin - org.scalastyle scalastyle-maven-plugin @@ -102,14 +128,14 @@ org.apache.mxnet - mxnet-init_${scala.binary.version} - 1.3.1-SNAPSHOT + mxnet-macros + INTERNAL provided org.apache.mxnet - mxnet-macros_${scala.binary.version} - 1.3.1-SNAPSHOT + mxnet-scala-init + INTERNAL provided diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala index b2a53fd9f..001bd04d2 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Base.scala @@ -18,7 +18,9 @@ package org.apache.mxnet import org.apache.mxnet.util.NativeLibraryLoader -import org.slf4j.{LoggerFactory, Logger} +import org.slf4j.{Logger, LoggerFactory} + +import scala.Specializable.Group private[mxnet] object Base { private val logger: Logger = LoggerFactory.getLogger("MXNetJVM") @@ -57,6 +59,9 @@ private[mxnet] object Base { val MX_REAL_TYPE = DType.Float32 + // The primitives currently supported for NDArray operations + val MX_PRIMITIVES = new Group ((Double, Float)) + try { try { tryLoadLibraryOS("mxnet-scala") @@ -67,6 +72,8 @@ private[mxnet] object Base { "Consider installing the library somewhere in the path " + "(for Windows: PATH, for Linux: LD_LIBRARY_PATH), " + "or specifying by Java cmd option -Djava.library.path=[lib path].") + logger.warn("LD_LIBRARY_PATH=" + System.getenv("LD_LIBRARY_PATH")) + logger.warn("java.library.path=" + System.getProperty("java.library.path")) NativeLibraryLoader.loadLibrary("mxnet-scala") } } catch { @@ -87,35 +94,8 @@ private[mxnet] object Base { @throws(classOf[UnsatisfiedLinkError]) private def tryLoadLibraryOS(libname: String): Unit = { - try { - logger.info(s"Try loading $libname from native path.") - System.loadLibrary(libname) - } catch { - case e: UnsatisfiedLinkError => - val os = System.getProperty("os.name") - // ref: http://lopica.sourceforge.net/os.html - if (os.startsWith("Linux")) { - tryLoadLibraryXPU(libname, "linux-x86_64") - } else if (os.startsWith("Mac")) { - tryLoadLibraryXPU(libname, "osx-x86_64") - } else { - // TODO(yizhi) support windows later - throw new UnsatisfiedLinkError() - } - } - } - - @throws(classOf[UnsatisfiedLinkError]) - private def tryLoadLibraryXPU(libname: String, arch: String): Unit = { - try { - // try gpu first - logger.info(s"Try loading $libname-$arch-gpu from native path.") - System.loadLibrary(s"$libname-$arch-gpu") - } catch { - case e: UnsatisfiedLinkError => - logger.info(s"Try loading $libname-$arch-cpu from native path.") - System.loadLibrary(s"$libname-$arch-cpu") - } + logger.info(s"Try loading $libname from native path.") + System.loadLibrary(libname) } // helper function definitions @@ -153,3 +133,21 @@ private[mxnet] object Base { } class MXNetError(val err: String) extends Exception(err) + +// Some type-classes to ease the work in Symbol.random and NDArray.random modules + +class SymbolOrScalar[T](val isScalar: Boolean) +object SymbolOrScalar { + def apply[T](implicit ev: SymbolOrScalar[T]): SymbolOrScalar[T] = ev + implicit object FloatWitness extends SymbolOrScalar[Float](true) + implicit object IntWitness extends SymbolOrScalar[Int](true) + implicit object SymbolWitness extends SymbolOrScalar[Symbol](false) +} + +class NDArrayOrScalar[T](val isScalar: Boolean) +object NDArrayOrScalar { + def apply[T](implicit ev: NDArrayOrScalar[T]): NDArrayOrScalar[T] = ev + implicit object FloatWitness extends NDArrayOrScalar[Float](true) + implicit object IntWitness extends NDArrayOrScalar[Int](true) + implicit object NDArrayWitness extends NDArrayOrScalar[NDArray](false) +} diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Context.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Context.scala index beeb430f6..ab44f434b 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Context.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Context.scala @@ -17,6 +17,8 @@ package org.apache.mxnet +import scala.language.implicitConversions + object Context { val devtype2str = Map(1 -> "cpu", 2 -> "gpu", 3 -> "cpu_pinned") val devstr2type = Map("cpu" -> 1, "gpu" -> 2, "cpu_pinned" -> 3) diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Executor.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Executor.scala index 19fb6fe5c..85f45bc66 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Executor.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Executor.scala @@ -106,9 +106,9 @@ class Executor private[mxnet](private[mxnet] val handle: ExecutorHandle, "is more efficient than the reverse." + "If you really want to up size, set allowUpSizing = true " + "to enable allocation of new arrays.") - newArgDict = newArgDict + (name -> NDArray.empty(newShape, arr.context)) + newArgDict = newArgDict + (name -> NDArray.empty(newShape, arr.context, arr.dtype)) if (dArr != null) { - newGradDict = newGradDict + (name -> NDArray.empty(newShape, dArr.context)) + newGradDict = newGradDict + (name -> NDArray.empty(newShape, dArr.context, dArr.dtype)) } } else { newArgDict = newArgDict + (name -> arr.reshape(newShape.toArray)) @@ -224,7 +224,6 @@ class Executor private[mxnet](private[mxnet] val handle: ExecutorHandle, /** * Get dictionary representation of argument arrrays. * @return The dictionary that maps name of arguments to NDArrays. - * @throws IllegalArgumentException if there are duplicated names in the arguments. */ def argDict: Map[String, NDArray] = { if (_argDict == null) { @@ -236,7 +235,6 @@ class Executor private[mxnet](private[mxnet] val handle: ExecutorHandle, /** * Get dictionary representation of gradient arrays. * @return The dictionary that maps name of arguments to gradient arrays. - * @throws IllegalArgumentException if there are duplicated names in the grads. */ def gradDict: Map[String, NDArray] = { if (_gradDict == null) { @@ -248,7 +246,6 @@ class Executor private[mxnet](private[mxnet] val handle: ExecutorHandle, /** * Get dictionary representation of auxiliary states arrays. * @return The dictionary that maps name of auxiliary states to NDArrays. - * @throws IllegalArgumentException if there are duplicated names in the auxiliary states. */ def auxDict: Map[String, NDArray] = { if (_auxDict == null) { @@ -265,8 +262,6 @@ class Executor private[mxnet](private[mxnet] val handle: ExecutorHandle, * Whether allow extra parameters that are not needed by symbol * If this is True, no error will be thrown when arg_params or aux_params * contain extra parameters that is not needed by the executor. - * @throws IllegalArgumentException - * If there is additional parameters in the dict but allow_extra_params=False */ def copyParamsFrom(argParams: Map[String, NDArray], auxParams: Map[String, NDArray], diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala b/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala index e8351422c..b580ad10a 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala @@ -25,6 +25,7 @@ import org.slf4j.LoggerFactory import scala.annotation.varargs import scala.collection.immutable.ListMap import scala.collection.mutable.ListBuffer +import scala.language.implicitConversions /** * IO iterators for loading training & validation data */ @@ -340,11 +341,11 @@ abstract class DataIter extends Iterator[DataBatch] { def getIndex(): IndexedSeq[Long] // The name and shape of data provided by this iterator - @deprecated + @deprecated("Use provideDataDesc instead", "1.3.0") def provideData: ListMap[String, Shape] // The name and shape of label provided by this iterator - @deprecated + @deprecated("Use provideLabelDesc instead", "1.3.0") def provideLabel: ListMap[String, Shape] // Provide type:DataDesc of the data @@ -404,7 +405,7 @@ object DataDesc { } } - @deprecated + @deprecated("Please use DataDesc methods instead", "1.3.0") implicit def ListMap2Descs(shapes: ListMap[String, Shape]): IndexedSeq[DataDesc] = { if (shapes != null) { shapes.map { case (k, s) => new DataDesc(k, s) }.toIndexedSeq diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Image.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Image.scala new file mode 100644 index 000000000..77881ab94 --- /dev/null +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Image.scala @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet +// scalastyle:off +import java.awt.image.BufferedImage +// scalastyle:on +import java.io.InputStream + +import scala.collection.mutable +import scala.collection.mutable.{ArrayBuffer, ListBuffer} + +/** + * Image API of Scala package + * enable OpenCV feature + */ +object Image { + + /** + * Decode image with OpenCV. + * Note: return image in RGB by default, instead of OpenCV's default BGR. + * @param buf Buffer containing binary encoded image + * @param flag Convert decoded image to grayscale (0) or color (1). + * @param to_rgb Whether to convert decoded image + * to mxnet's default RGB format (instead of opencv's default BGR). + * @return NDArray in HWC format + */ + def imDecode(buf: Array[Byte], flag: Int, + to_rgb: Boolean, + out: Option[NDArray]): NDArray = { + val nd = NDArray.array(buf.map( x => (x & 0xFF).toFloat), Shape(buf.length)) + val byteND = NDArray.api.cast(nd, "uint8") + val args : ListBuffer[Any] = ListBuffer() + val map : mutable.Map[String, Any] = mutable.Map() + args += byteND + map("flag") = flag + map("to_rgb") = to_rgb + if (out.isDefined) map("out") = out.get + NDArray.genericNDArrayFunctionInvoke("_cvimdecode", args, map.toMap) + } + + /** + * Same imageDecode with InputStream + * @param inputStream the inputStream of the image + * @return NDArray in HWC format + */ + def imDecode(inputStream: InputStream, flag: Int = 1, + to_rgb: Boolean = true, + out: Option[NDArray] = None): NDArray = { + val buffer = new Array[Byte](2048) + val arrBuffer = ArrayBuffer[Byte]() + var length = 0 + while (length != -1) { + length = inputStream.read(buffer) + if (length != -1) arrBuffer ++= buffer.slice(0, length) + } + imDecode(arrBuffer.toArray, flag, to_rgb, out) + } + + /** + * Read and decode image with OpenCV. + * Note: return image in RGB by default, instead of OpenCV's default BGR. + * @param filename Name of the image file to be loaded. + * @param flag Convert decoded image to grayscale (0) or color (1). + * @param to_rgb Whether to convert decoded image to mxnet's default RGB format + * (instead of opencv's default BGR). + * @return org.apache.mxnet.NDArray in HWC format + */ + def imRead(filename: String, flag: Option[Int] = None, + to_rgb: Option[Boolean] = None, + out: Option[NDArray] = None): NDArray = { + val args : ListBuffer[Any] = ListBuffer() + val map : mutable.Map[String, Any] = mutable.Map() + map("filename") = filename + if (flag.isDefined) map("flag") = flag.get + if (to_rgb.isDefined) map("to_rgb") = to_rgb.get + if (out.isDefined) map("out") = out.get + NDArray.genericNDArrayFunctionInvoke("_cvimread", args, map.toMap) + } + + /** + * Resize image with OpenCV. + * @param src source image in NDArray + * @param w Width of resized image. + * @param h Height of resized image. + * @param interp Interpolation method (default=cv2.INTER_LINEAR). + * @return org.apache.mxnet.NDArray + */ + def imResize(src: org.apache.mxnet.NDArray, w: Int, h: Int, + interp: Option[Int] = None, + out: Option[NDArray] = None): NDArray = { + val args : ListBuffer[Any] = ListBuffer() + val map : mutable.Map[String, Any] = mutable.Map() + args += src + map("w") = w + map("h") = h + if (interp.isDefined) map("interp") = interp.get + if (out.isDefined) map("out") = out.get + NDArray.genericNDArrayFunctionInvoke("_cvimresize", args, map.toMap) + } + + /** + * Pad image border with OpenCV. + * @param src source image + * @param top Top margin. + * @param bot Bottom margin. + * @param left Left margin. + * @param right Right margin. + * @param typeOf Filling type (default=cv2.BORDER_CONSTANT). + * @param value (Deprecated! Use ``values`` instead.) Fill with single value. + * @param values Fill with value(RGB[A] or gray), up to 4 channels. + * @return org.apache.mxnet.NDArray + */ + def copyMakeBorder(src: org.apache.mxnet.NDArray, top: Int, bot: Int, + left: Int, right: Int, typeOf: Option[Int] = None, + value: Option[Double] = None, values: Option[Any] = None, + out: Option[NDArray] = None): NDArray = { + val args : ListBuffer[Any] = ListBuffer() + val map : mutable.Map[String, Any] = mutable.Map() + args += src + map("top") = top + map("bot") = bot + map("left") = left + map("right") = right + if (typeOf.isDefined) map("type") = typeOf.get + if (value.isDefined) map("value") = value.get + if (values.isDefined) map("values") = values.get + if (out.isDefined) map("out") = out.get + NDArray.genericNDArrayFunctionInvoke("_cvcopyMakeBorder", args, map.toMap) + } + + /** + * Do a fixed crop on the image + * @param src Src image in NDArray + * @param x0 starting x point + * @param y0 starting y point + * @param w width of the image + * @param h height of the image + * @return cropped NDArray + */ + def fixedCrop(src: NDArray, x0: Int, y0: Int, w: Int, h: Int): NDArray = { + NDArray.api.crop(src, Shape(y0, x0, 0), Shape(y0 + h, x0 + w, src.shape.get(2))) + } + + /** + * Convert a NDArray image to a real image + * The time cost will increase if the image resolution is big + * @param src Source image file in RGB + * @return Buffered Image + */ + def toImage(src: NDArray): BufferedImage = { + require(src.dtype == DType.UInt8, "The input NDArray must be bytes") + require(src.shape.length == 3, "The input should contains height, width and channel") + val height = src.shape.get(0) + val width = src.shape.get(1) + val img = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB) + (0 until height).par.foreach(r => { + (0 until width).par.foreach(c => { + val arr = src.at(r).at(c).toArray + // NDArray in RGB + val red = arr(0).toByte & 0xFF + val green = arr(1).toByte & 0xFF + val blue = arr(2).toByte & 0xFF + val rgb = (red << 16) | (green << 8) | blue + img.setRGB(c, r, rgb) + }) + }) + img + } + +} diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/KVStore.scala b/scala-package/core/src/main/scala/org/apache/mxnet/KVStore.scala index 45189a13a..b2d4349b4 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/KVStore.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/KVStore.scala @@ -286,7 +286,7 @@ class KVStore(private[mxnet] val handle: KVStoreHandle) extends NativeResource { case cachedStates: MXKVStoreCachedStates => val bis = new BufferedInputStream (new FileInputStream (fname) ) try { - val bArray = Stream.continually (bis.read).takeWhile (- 1 !=).map (_.toByte).toArray + val bArray = Stream.continually (bis.read).takeWhile (_ != -1).map (_.toByte).toArray cachedStates.deserializeState(bArray) } finally { bis.close () diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/LibInfo.scala b/scala-package/core/src/main/scala/org/apache/mxnet/LibInfo.scala index 0a5683aa7..20b6ed9fc 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/LibInfo.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/LibInfo.scala @@ -93,6 +93,9 @@ private[mxnet] class LibInfo { @native def mxNDArraySyncCopyFromCPU(handle: NDArrayHandle, source: Array[MXFloat], size: Int): Int + @native def mxFloat64NDArraySyncCopyFromCPU(handle: NDArrayHandle, + source: Array[Double], + size: Int): Int @native def mxNDArrayLoad(fname: String, outSize: MXUintRef, handles: ArrayBuffer[NDArrayHandle], diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/MX_PRIMITIVES.scala b/scala-package/core/src/main/scala/org/apache/mxnet/MX_PRIMITIVES.scala new file mode 100644 index 000000000..cb9788569 --- /dev/null +++ b/scala-package/core/src/main/scala/org/apache/mxnet/MX_PRIMITIVES.scala @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet + +object MX_PRIMITIVES { + + /** + * This defines the basic primitives we can use in Scala for mathematical + * computations in NDArrays.This gives us a flexibility to expand to + * more supported primitives in the future. Currently Float and Double + * are supported. The functions which accept MX_PRIMITIVE_TYPE as input can also accept + * plain old Float and Double data as inputs because of the underlying + * implicit conversion between primitives to MX_PRIMITIVE_TYPE. + */ + trait MX_PRIMITIVE_TYPE extends Ordered[MX_PRIMITIVE_TYPE]{ + + def toString: String + + def unary_- : MX_PRIMITIVE_TYPE + } + + trait MXPrimitiveOrdering extends Ordering[MX_PRIMITIVE_TYPE] { + + def compare(x: MX_PRIMITIVE_TYPE, y: MX_PRIMITIVE_TYPE): Int = x.compare(y) + + } + + implicit object MX_PRIMITIVE_TYPE extends MXPrimitiveOrdering + + /** + * Wrapper over Float in Scala. + * @param data + */ + class MX_FLOAT(val data: Float) extends MX_PRIMITIVE_TYPE { + + override def toString: String = data.toString + + override def unary_- : MX_PRIMITIVE_TYPE = new MX_FLOAT(data.unary_-) + + override def compare(that: MX_PRIMITIVE_TYPE): Int = { + this.data.compareTo(that.asInstanceOf[MX_FLOAT].data) + } + } + + implicit def FloatToMX_Float(d : Float): MX_FLOAT = new MX_FLOAT(d) + + implicit def MX_FloatToFloat(d: MX_FLOAT) : Float = d.data + + implicit def IntToMX_Float(d: Int): MX_FLOAT = new MX_FLOAT(d.toFloat) + + /** + * Wrapper over Double in Scala. + * @param data + */ + class MX_Double(val data: Double) extends MX_PRIMITIVE_TYPE { + + override def toString: String = data.toString + + override def unary_- : MX_PRIMITIVE_TYPE = new MX_Double(data.unary_-) + + override def compare(that: MX_PRIMITIVE_TYPE): Int = { + this.data.compareTo(that.asInstanceOf[MX_Double].data) + } + } + + implicit def DoubleToMX_Double(d : Double): MX_Double = new MX_Double(d) + + implicit def MX_DoubleToDouble(d: MX_Double) : Double = d.data + +} diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala b/scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala index f9f2dbe42..163ed2682 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/NDArray.scala @@ -21,10 +21,12 @@ import java.nio.{ByteBuffer, ByteOrder} import org.apache.mxnet.Base._ import org.apache.mxnet.DType.DType +import org.apache.mxnet.MX_PRIMITIVES.{MX_PRIMITIVE_TYPE} import org.slf4j.LoggerFactory import scala.collection.mutable import scala.collection.mutable.{ArrayBuffer, ListBuffer} +import scala.language.implicitConversions import scala.ref.WeakReference /** @@ -39,6 +41,7 @@ object NDArray extends NDArrayBase { private val functions: Map[String, NDArrayFunction] = initNDArrayModule() val api = NDArrayAPI + val random = NDArrayRandomAPI private def addDependency(froms: Array[NDArray], tos: Array[NDArray]): Unit = { froms.foreach { from => @@ -260,16 +263,46 @@ object NDArray extends NDArrayBase { arr } - // Perform power operator + def full(shape: Shape, value: Double, ctx: Context): NDArray = { + val arr = empty(shape, ctx, DType.Float64) + arr.set(value) + arr + } + + /** + * Create a new NDArray filled with given value, with specified shape. + * @param shape shape of the NDArray. + * @param value value to be filled with + */ + def full(shape: Shape, value: Double): NDArray = { + full(shape, value, null) + } + + + /** + * Perform power operation on NDArray. Returns result as NDArray + * @param lhs + * @param rhs + */ def power(lhs: NDArray, rhs: NDArray): NDArray = { NDArray.genericNDArrayFunctionInvoke("_power", Seq(lhs, rhs)) } - def power(lhs: NDArray, rhs: Float): NDArray = { + /** + * Perform scalar power operation on NDArray. Returns result as NDArray + * @param lhs NDArray on which to perform the operation on. + * @param rhs The scalar input. Can be of type Float/Double + */ + def power(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_power_scalar", Seq(lhs, rhs)) } - def power(lhs: Float, rhs: NDArray): NDArray = { + /** + * Perform scalar power operation on NDArray. Returns result as NDArray + * @param lhs The scalar input. Can be of type Float/Double + * @param rhs NDArray on which to perform the operation on. + */ + def power(lhs: MX_PRIMITIVE_TYPE, rhs: NDArray): NDArray = { NDArray.genericNDArrayFunctionInvoke("_rpower_scalar", Seq(lhs, rhs)) } @@ -278,11 +311,21 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("_maximum", Seq(lhs, rhs)) } - def maximum(lhs: NDArray, rhs: Float): NDArray = { + /** + * Perform the max operation on NDArray. Returns the result as NDArray. + * @param lhs NDArray on which to perform the operation on. + * @param rhs The scalar input. Can be of type Float/Double + */ + def maximum(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_maximum_scalar", Seq(lhs, rhs)) } - def maximum(lhs: Float, rhs: NDArray): NDArray = { + /** + * Perform the max operation on NDArray. Returns the result as NDArray. + * @param lhs The scalar input. Can be of type Float/Double + * @param rhs NDArray on which to perform the operation on. + */ + def maximum(lhs: MX_PRIMITIVE_TYPE, rhs: NDArray): NDArray = { NDArray.genericNDArrayFunctionInvoke("_maximum_scalar", Seq(lhs, rhs)) } @@ -291,11 +334,21 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("_minimum", Seq(lhs, rhs)) } - def minimum(lhs: NDArray, rhs: Float): NDArray = { + /** + * Perform the min operation on NDArray. Returns the result as NDArray. + * @param lhs NDArray on which to perform the operation on. + * @param rhs The scalar input. Can be of type Float/Double + */ + def minimum(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_minimum_scalar", Seq(lhs, rhs)) } - def minimum(lhs: Float, rhs: NDArray): NDArray = { + /** + * Perform the min operation on NDArray. Returns the result as NDArray. + * @param lhs The scalar input. Can be of type Float/Double + * @param rhs NDArray on which to perform the operation on. + */ + def minimum(lhs: MX_PRIMITIVE_TYPE, rhs: NDArray): NDArray = { NDArray.genericNDArrayFunctionInvoke("_minimum_scalar", Seq(lhs, rhs)) } @@ -308,7 +361,15 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_equal", Seq(lhs, rhs)) } - def equal(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **equal to** (==) comparison operation with broadcasting. + * For each element in input arrays, return 1(true) if corresponding elements are same, + * otherwise return 0(false). + * + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def equal(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_equal_scalar", Seq(lhs, rhs)) } @@ -322,7 +383,15 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_not_equal", Seq(lhs, rhs)) } - def notEqual(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **not equal to** (!=) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if corresponding elements are different, + * otherwise return 0(false). + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def notEqual(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_not_equal_scalar", Seq(lhs, rhs)) } @@ -336,7 +405,16 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_greater", Seq(lhs, rhs)) } - def greater(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **greater than** (>) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are greater than rhs, + * otherwise return 0(false). + * + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def greater(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_greater_scalar", Seq(lhs, rhs)) } @@ -350,7 +428,16 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_greater_equal", Seq(lhs, rhs)) } - def greaterEqual(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **greater than or equal to** (>=) comparison + * operation with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are greater than equal to + * rhs, otherwise return 0(false). + * + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def greaterEqual(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_greater_equal_scalar", Seq(lhs, rhs)) } @@ -364,7 +451,15 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_lesser", Seq(lhs, rhs)) } - def lesser(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **lesser than** (<) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are less than rhs, + * otherwise return 0(false). + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def lesser(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_lesser_scalar", Seq(lhs, rhs)) } @@ -378,7 +473,16 @@ object NDArray extends NDArrayBase { NDArray.genericNDArrayFunctionInvoke("broadcast_lesser_equal", Seq(lhs, rhs)) } - def lesserEqual(lhs: NDArray, rhs: Float): NDArray = { + /** + * Returns the result of element-wise **lesser than or equal to** (<=) comparison + * operation with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are + * lesser than equal to rhs, otherwise return 0(false). + * + * @param lhs NDArray + * @param rhs The scalar input. Can be of type Float/Double + */ + def lesserEqual(lhs: NDArray, rhs: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_lesser_equal_scalar", Seq(lhs, rhs)) } @@ -395,6 +499,16 @@ object NDArray extends NDArrayBase { arr } + def array(sourceArr: Array[Double], shape: Shape, ctx: Context): NDArray = { + val arr = empty(shape, ctx, dtype = DType.Float64) + arr.set(sourceArr) + arr + } + + def array(sourceArr: Array[Double], shape: Shape): NDArray = { + array(sourceArr, shape, null) + } + /** * Returns evenly spaced values within a given interval. * Values are generated within the half-open interval [`start`, `stop`). In other @@ -643,6 +757,12 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, checkCall(_LIB.mxNDArraySyncCopyFromCPU(handle, source, source.length)) } + private def syncCopyfrom(source: Array[Double]): Unit = { + require(source.length == size, + s"array size (${source.length}) do not match the size of NDArray ($size)") + checkCall(_LIB.mxFloat64NDArraySyncCopyFromCPU(handle, source, source.length)) + } + /** * Return a sliced NDArray that shares memory with current one. * NDArray only support continuous slicing on axis 0 @@ -757,7 +877,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, * @param value Value to set * @return Current NDArray */ - def set(value: Float): NDArray = { + def set(value: MX_PRIMITIVE_TYPE): NDArray = { require(writable, "trying to assign to a readonly NDArray") NDArray.genericNDArrayFunctionInvoke("_set_value", Seq(value), Map("out" -> this)) this @@ -774,11 +894,17 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } + def set(other: Array[Double]): NDArray = { + require(writable, "trying to assign to a readonly NDArray") + syncCopyfrom(other) + this + } + def +(other: NDArray): NDArray = { NDArray.genericNDArrayFunctionInvoke("_plus", Seq(this, other)) } - def +(other: Float): NDArray = { + def +(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_plus_scalar", Seq(this, other)) } @@ -790,7 +916,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } - def +=(other: Float): NDArray = { + def +=(other: MX_PRIMITIVE_TYPE): NDArray = { if (!writable) { throw new IllegalArgumentException("trying to add to a readonly NDArray") } @@ -802,7 +928,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.genericNDArrayFunctionInvoke("_minus", Seq(this, other)) } - def -(other: Float): NDArray = { + def -(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_minus_scalar", Seq(this, other)) } @@ -814,7 +940,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } - def -=(other: Float): NDArray = { + def -=(other: MX_PRIMITIVE_TYPE): NDArray = { if (!writable) { throw new IllegalArgumentException("trying to subtract from a readonly NDArray") } @@ -826,7 +952,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.genericNDArrayFunctionInvoke("_mul", Seq(this, other)) } - def *(other: Float): NDArray = { + def *(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_mul_scalar", Seq(this, other)) } @@ -842,7 +968,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } - def *=(other: Float): NDArray = { + def *=(other: MX_PRIMITIVE_TYPE): NDArray = { if (!writable) { throw new IllegalArgumentException("trying to multiply to a readonly NDArray") } @@ -854,7 +980,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.genericNDArrayFunctionInvoke("_div", Seq(this, other)) } - def /(other: Float): NDArray = { + def /(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_div_scalar", Seq(this, other)) } @@ -866,7 +992,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } - def /=(other: Float): NDArray = { + def /=(other: MX_PRIMITIVE_TYPE): NDArray = { if (!writable) { throw new IllegalArgumentException("trying to divide from a readonly NDArray") } @@ -878,7 +1004,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.power(this, other) } - def **(other: Float): NDArray = { + def **(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.power(this, other) } @@ -886,7 +1012,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.genericNDArrayFunctionInvoke("_power", Seq(this, other), Map("out" -> this)) } - def **=(other: Float): NDArray = { + def **=(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_power_scalar", Seq(this, other), Map("out" -> this)) } @@ -894,7 +1020,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.greater(this, other) } - def >(other: Float): NDArray = { + def >(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.greater(this, other) } @@ -902,7 +1028,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.greaterEqual(this, other) } - def >=(other: Float): NDArray = { + def >=(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.greaterEqual(this, other) } @@ -910,7 +1036,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.lesser(this, other) } - def <(other: Float): NDArray = { + def <(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.lesser(this, other) } @@ -918,7 +1044,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.lesserEqual(this, other) } - def <=(other: Float): NDArray = { + def <=(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.lesserEqual(this, other) } @@ -926,7 +1052,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, NDArray.genericNDArrayFunctionInvoke("_mod", Seq(this, other)) } - def %(other: Float): NDArray = { + def %(other: MX_PRIMITIVE_TYPE): NDArray = { NDArray.genericNDArrayFunctionInvoke("_mod_scalar", Seq(this, other)) } @@ -938,7 +1064,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this } - def %=(other: Float): NDArray = { + def %=(other: MX_PRIMITIVE_TYPE): NDArray = { if (!writable) { throw new IllegalArgumentException("trying to take modulo from a readonly NDArray") } @@ -954,6 +1080,14 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, internal.toFloatArray } + /** + * Return a copied flat java array of current array (row-major) with datatype as Float64/Double. + * @return A copy of array content. + */ + def toFloat64Array: Array[Double] = { + internal.toDoubleArray + } + def internal: NDArrayInternal = { val myType = dtype val arrLength = DType.numOfBytes(myType) * size @@ -973,6 +1107,11 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, this.toArray(0) } + def toFloat64Scalar: Double = { + require(shape == Shape(1), "The current array is not a scalar") + this.toFloat64Array(0) + } + /** * Copy the content of current array to other. * @@ -995,7 +1134,7 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, * @return The copy target NDArray */ def copyTo(ctx: Context): NDArray = { - val ret = new NDArray(NDArray.newAllocHandle(shape, ctx, delayAlloc = true)) + val ret = new NDArray(NDArray.newAllocHandle(shape, ctx, delayAlloc = true, dtype = dtype)) copyTo(ret) } @@ -1045,11 +1184,11 @@ class NDArray private[mxnet](private[mxnet] val handle: NDArrayHandle, private[mxnet] object NDArrayConversions { implicit def int2Scalar(x: Int): NDArrayConversions = new NDArrayConversions(x.toFloat) - implicit def double2Scalar(x: Double): NDArrayConversions = new NDArrayConversions(x.toFloat) + implicit def double2Scalar(x: Double): NDArrayConversions = new NDArrayConversions(x) implicit def float2Scalar(x: Float): NDArrayConversions = new NDArrayConversions(x) } -private[mxnet] class NDArrayConversions(val value: Float) { +private[mxnet] class NDArrayConversions(val value: MX_PRIMITIVE_TYPE) { def +(other: NDArray): NDArray = { other + value } @@ -1143,34 +1282,39 @@ private[mxnet] class NDArrayFuncReturn(private[mxnet] val arr: Array[NDArray]) { def waitToRead(): Unit = head.waitToRead() def context: Context = head.context def set(value: Float): NDArray = head.set(value) + def set(value: Double): NDArray = head.set(value) def set(other: NDArray): NDArray = head.set(other) def set(other: Array[Float]): NDArray = head.set(other) + def set(other: Array[Double]): NDArray = head.set(other) def +(other: NDArray): NDArray = head + other - def +(other: Float): NDArray = head + other + def +(other: MX_PRIMITIVE_TYPE): NDArray = head + other def +=(other: NDArray): NDArray = head += other - def +=(other: Float): NDArray = head += other + def +=(other: MX_PRIMITIVE_TYPE): NDArray = head += other def -(other: NDArray): NDArray = head - other - def -(other: Float): NDArray = head - other + def -(other: MX_PRIMITIVE_TYPE): NDArray = head - other def -=(other: NDArray): NDArray = head -= other - def -=(other: Float): NDArray = head -= other + def -=(other: MX_PRIMITIVE_TYPE): NDArray = head -= other def *(other: NDArray): NDArray = head * other - def *(other: Float): NDArray = head * other + def *(other: MX_PRIMITIVE_TYPE): NDArray = head * other def unary_-(): NDArray = -head def *=(other: NDArray): NDArray = head *= other - def *=(other: Float): NDArray = head *= other + def *=(other: MX_PRIMITIVE_TYPE): NDArray = head *= other def /(other: NDArray): NDArray = head / other + def /(other: MX_PRIMITIVE_TYPE): NDArray = head / other def **(other: NDArray): NDArray = head ** other - def **(other: Float): NDArray = head ** other + def **(other: MX_PRIMITIVE_TYPE): NDArray = head ** other def >(other: NDArray): NDArray = head > other - def >(other: Float): NDArray = head > other + def >(other: MX_PRIMITIVE_TYPE): NDArray = head > other def >=(other: NDArray): NDArray = head >= other - def >=(other: Float): NDArray = head >= other + def >=(other: MX_PRIMITIVE_TYPE): NDArray = head >= other def <(other: NDArray): NDArray = head < other - def <(other: Float): NDArray = head < other + def <(other: MX_PRIMITIVE_TYPE): NDArray = head < other def <=(other: NDArray): NDArray = head <= other - def <=(other: Float): NDArray = head <= other + def <=(other: MX_PRIMITIVE_TYPE): NDArray = head <= other def toArray: Array[Float] = head.toArray + def toFloat64Array: Array[Double] = head.toFloat64Array def toScalar: Float = head.toScalar + def toFloat64Scalar: Double = head.toFloat64Scalar def copyTo(other: NDArray): NDArray = head.copyTo(other) def copyTo(ctx: Context): NDArray = head.copyTo(ctx) def copy(): NDArray = head.copy() diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/NDArrayAPI.scala b/scala-package/core/src/main/scala/org/apache/mxnet/NDArrayAPI.scala index 1d8551c1b..024fed1c4 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/NDArrayAPI.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/NDArrayAPI.scala @@ -15,11 +15,22 @@ * limitations under the License. */ package org.apache.mxnet -@AddNDArrayAPIs(false) + /** * typesafe NDArray API: NDArray.api._ * Main code will be generated during compile time through Macros */ +@AddNDArrayAPIs(false) object NDArrayAPI extends NDArrayAPIBase { // TODO: Implement CustomOp for NDArray } + +/** + * typesafe NDArray random module: NDArray.random._ + * Main code will be generated during compile time through Macros + */ +@AddNDArrayRandomAPIs(false) +object NDArrayRandomAPI extends NDArrayRandomAPIBase { + +} + diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Optimizer.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Optimizer.scala index c3f8aaec6..1fb634ceb 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Optimizer.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Optimizer.scala @@ -144,7 +144,7 @@ abstract class Optimizer extends Serializable { def deserializeState(bytes: Array[Byte]): AnyRef // Set individual learning rate scale for parameters - @deprecated("Use setLrMult instead.") + @deprecated("Use setLrMult instead.", "0.10.0") def setLrScale(lrScale: Map[Int, Float]): Unit = { val argsLrScale: Map[Either[Int, String], Float] = lrScale.map { case (k, v) => Left(k) -> v } setLrMult(argsLrScale) diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala b/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala index 30fe1473a..bb363c0c3 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala @@ -27,7 +27,7 @@ import scala.util.Try import scala.util.control.{ControlThrowable, NonFatal} /** - * This class manages automatically releasing of [[NativeResource]]s + * This class manages automatically releasing of `org.apache.mxnet.NativeResource`s */ class ResourceScope extends AutoCloseable { @@ -43,8 +43,8 @@ class ResourceScope extends AutoCloseable { ResourceScope.addToThreadLocal(this) /** - * Releases all the [[NativeResource]] by calling - * the associated [[NativeResource.close()]] method + * Releases all the `org.apache.mxnet.NativeResource` by calling + * the associated`'org.apache.mxnet.NativeResource.close()` method */ override def close(): Unit = { ResourceScope.removeFromThreadLocal(this) diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala index 4472a8426..29885fc72 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Symbol.scala @@ -22,6 +22,7 @@ import org.apache.mxnet.DType.DType import org.slf4j.{Logger, LoggerFactory} import scala.collection.mutable.{ArrayBuffer, ListBuffer} +import scala.language.implicitConversions /** * Symbolic configuration API of mxnet.
      @@ -841,6 +842,7 @@ object Symbol extends SymbolBase { private val bindReqMap = Map("null" -> 0, "write" -> 1, "add" -> 3) val api = SymbolAPI + val random = SymbolRandomAPI def pow(sym1: Symbol, sym2: Symbol): Symbol = { Symbol.createFromListedSymbols("_Power")(Array(sym1, sym2)) diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/SymbolAPI.scala b/scala-package/core/src/main/scala/org/apache/mxnet/SymbolAPI.scala index 1bfb0559c..f166de11e 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/SymbolAPI.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/SymbolAPI.scala @@ -19,11 +19,11 @@ package org.apache.mxnet import scala.collection.mutable -@AddSymbolAPIs(false) /** * typesafe Symbol API: Symbol.api._ * Main code will be generated during compile time through Macros */ +@AddSymbolAPIs(false) object SymbolAPI extends SymbolAPIBase { def Custom (op_type : String, kwargs : mutable.Map[String, Any], name : String = null, attr : Map[String, String] = null) : Symbol = { @@ -32,3 +32,13 @@ object SymbolAPI extends SymbolAPIBase { Symbol.createSymbolGeneral("Custom", name, attr, Seq(), map.toMap) } } + +/** + * typesafe Symbol random module: Symbol.random._ + * Main code will be generated during compile time through Macros + */ +@AddSymbolRandomAPIs(false) +object SymbolRandomAPI extends SymbolRandomAPIBase { + +} + diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/Visualization.scala b/scala-package/core/src/main/scala/org/apache/mxnet/Visualization.scala index 2a7b7a8b3..b990137b5 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/Visualization.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/Visualization.scala @@ -21,6 +21,7 @@ import scala.util.parsing.json._ import java.io.File import java.io.PrintWriter import scala.collection.mutable.ArrayBuffer +import scala.language.postfixOps object Visualization { diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/io/MXDataIter.scala b/scala-package/core/src/main/scala/org/apache/mxnet/io/MXDataIter.scala index 998017750..e30098c30 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/io/MXDataIter.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/io/MXDataIter.scala @@ -53,9 +53,9 @@ private[mxnet] class MXDataIter(private[mxnet] val handle: DataIterHandle, val label = currentBatch.label(0) // properties val res = ( - // TODO: need to allow user to specify DType and Layout - IndexedSeq(new DataDesc(dataName, data.shape, DType.Float32, Layout.UNDEFINED)), - IndexedSeq(new DataDesc(labelName, label.shape, DType.Float32, Layout.UNDEFINED)), + // TODO: need to allow user to specify Layout + IndexedSeq(new DataDesc(dataName, data.shape, data.dtype, Layout.UNDEFINED)), + IndexedSeq(new DataDesc(labelName, label.shape, label.dtype, Layout.UNDEFINED)), ListMap(dataName -> data.shape), ListMap(labelName -> label.shape), data.shape(0)) @@ -158,11 +158,11 @@ private[mxnet] class MXDataIter(private[mxnet] val handle: DataIterHandle, } // The name and shape of data provided by this iterator - @deprecated + @deprecated("Please use provideDataDesc instead", "1.3.0") override def provideData: ListMap[String, Shape] = _provideData // The name and shape of label provided by this iterator - @deprecated + @deprecated("Please use provideLabelDesc instead", "1.3.0") override def provideLabel: ListMap[String, Shape] = _provideLabel // Provide type:DataDesc of the data diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala b/scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala index e6be0ad02..e690abba0 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/io/NDArrayIter.scala @@ -61,7 +61,8 @@ class NDArrayIter(data: IndexedSeq[(DataDesc, NDArray)], dataBatchSize: Int = 1, shuffle: Boolean = false, lastBatchHandle: String = "pad", dataName: String = "data", labelName: String = "label") { - this(IO.initDataDesc(data, allowEmpty = false, dataName, MX_REAL_TYPE, Layout.UNDEFINED), + this(IO.initDataDesc(data, allowEmpty = false, dataName, + if (data == null || data.isEmpty) MX_REAL_TYPE else data(0).dtype, Layout.UNDEFINED), IO.initDataDesc(label, allowEmpty = true, labelName, MX_REAL_TYPE, Layout.UNDEFINED), dataBatchSize, shuffle, lastBatchHandle) } @@ -237,11 +238,11 @@ class NDArrayIter(data: IndexedSeq[(DataDesc, NDArray)], // The name and shape of data provided by this iterator - @deprecated + @deprecated("Please use provideDataDesc instead", "1.3.0") override def provideData: ListMap[String, Shape] = _provideData // The name and shape of label provided by this iterator - @deprecated + @deprecated("Please use provideLabelDesc instead", "1.3.0") override def provideLabel: ListMap[String, Shape] = _provideLabel // Provide type:DataDesc of the data @@ -272,7 +273,7 @@ object NDArrayIter { */ def addData(name: String, data: NDArray): Builder = { this.data = this.data ++ IndexedSeq((new DataDesc(name, - data.shape, DType.Float32, Layout.UNDEFINED), data)) + data.shape, data.dtype, Layout.UNDEFINED), data)) this } @@ -284,7 +285,7 @@ object NDArrayIter { */ def addLabel(name: String, label: NDArray): Builder = { this.label = this.label ++ IndexedSeq((new DataDesc(name, - label.shape, DType.Float32, Layout.UNDEFINED), label)) + label.shape, label.dtype, Layout.UNDEFINED), label)) this } diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/io/PrefetchingIter.scala b/scala-package/core/src/main/scala/org/apache/mxnet/io/PrefetchingIter.scala index e59e37063..d277351b1 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/io/PrefetchingIter.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/io/PrefetchingIter.scala @@ -178,11 +178,11 @@ class PrefetchingIter( override def getPad(): Int = this.currentBatch.pad // The name and shape of label provided by this iterator - @deprecated + @deprecated("Please use provideDataDesc instead", "1.3.0") override def provideLabel: ListMap[String, Shape] = this._provideLabel // The name and shape of data provided by this iterator - @deprecated + @deprecated("Please use provideLabelDesc instead", "1.3.0") override def provideData: ListMap[String, Shape] = this._provideData // Provide type:DataDesc of the data diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/io/ResizeIter.scala b/scala-package/core/src/main/scala/org/apache/mxnet/io/ResizeIter.scala index e840af939..9bc042a7b 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/io/ResizeIter.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/io/ResizeIter.scala @@ -134,13 +134,13 @@ class ResizeIter( } // The name and shape of data provided by this iterator - @deprecated + @deprecated("Please use provideDataDesc instead", "1.3.0") override def provideData: ListMap[String, Shape] = { dataIter.provideData } // The name and shape of label provided by this iterator - @deprecated + @deprecated("Please use provideLabelDesc instead", "1.3.0") override def provideLabel: ListMap[String, Shape] = { dataIter.provideLabel } diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Context.scala b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Context.scala index 5f0caedcc..3d397e3fc 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Context.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Context.scala @@ -17,8 +17,16 @@ package org.apache.mxnet.javaapi import collection.JavaConverters._ +import scala.language.implicitConversions -class Context(val context: org.apache.mxnet.Context) { +/** + * Constructing a context which is used to specify the device and device type that will + * be utilized by the engine. + * + * @param deviceTypeName {'cpu', 'gpu'} String representing the device type + * @param deviceId The device id of the device, needed for GPU + */ +class Context private[mxnet] (val context: org.apache.mxnet.Context) { val deviceTypeid: Int = context.deviceTypeid @@ -26,6 +34,11 @@ class Context(val context: org.apache.mxnet.Context) { = this(new org.apache.mxnet.Context(deviceTypeName, deviceId)) def withScope[T](body: => T): T = context.withScope(body) + + /** + * Return device type of current context. + * @return device_type + */ def deviceType: String = context.deviceType override def toString: String = context.toString @@ -42,6 +55,5 @@ object Context { val gpu: Context = org.apache.mxnet.Context.gpu() val devtype2str = org.apache.mxnet.Context.devstr2type.asJava val devstr2type = org.apache.mxnet.Context.devstr2type.asJava - def defaultCtx: Context = org.apache.mxnet.Context.defaultCtx } diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/IO.scala b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/IO.scala index 47b1c367c..d0e10815a 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/IO.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/IO.scala @@ -16,8 +16,9 @@ */ package org.apache.mxnet.javaapi +import scala.language.implicitConversions -class DataDesc(val dataDesc: org.apache.mxnet.DataDesc) { +class DataDesc private[mxnet] (val dataDesc: org.apache.mxnet.DataDesc) { def this(name: String, shape: Shape, dType: DType.DType, layout: String) = this(new org.apache.mxnet.DataDesc(name, shape, dType, layout)) @@ -30,5 +31,13 @@ object DataDesc{ implicit def toDataDesc(dataDesc: DataDesc): org.apache.mxnet.DataDesc = dataDesc.dataDesc + /** + * Get the dimension that corresponds to the batch size. + * @param layout layout string. For example, "NCHW". + * @return An axis indicating the batch_size dimension. When data-parallelism is used, + * the data will be automatically split and concatenate along the batch_size dimension. + * Axis can be -1, which means the whole array will be copied + * for each data-parallelism device. + */ def getBatchAxis(layout: String): Int = org.apache.mxnet.DataDesc.getBatchAxis(Some(layout)) } diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/NDArray.scala b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/NDArray.scala new file mode 100644 index 000000000..67809c158 --- /dev/null +++ b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/NDArray.scala @@ -0,0 +1,463 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.javaapi + +import org.apache.mxnet.javaapi.DType.DType + +import collection.JavaConverters._ +import scala.language.implicitConversions + +@AddJNDArrayAPIs(false) +object NDArray extends NDArrayBase { + implicit def fromNDArray(nd: org.apache.mxnet.NDArray): NDArray = new NDArray(nd) + + implicit def toNDArray(jnd: NDArray): org.apache.mxnet.NDArray = jnd.nd + + def waitall(): Unit = org.apache.mxnet.NDArray.waitall() + + /** + * One hot encoding indices into matrix out. + * @param indices An NDArray containing indices of the categorical features. + * @param out The result holder of the encoding. + * @return Same as out. + */ + def onehotEncode(indices: NDArray, out: NDArray): NDArray + = org.apache.mxnet.NDArray.onehotEncode(indices, out) + + /** + * Create an empty uninitialized new NDArray, with specified shape. + * + * @param shape shape of the NDArray. + * @param ctx The context of the NDArray. + * + * @return The created NDArray. + */ + def empty(shape: Shape, ctx: Context, dtype: DType.DType): NDArray + = org.apache.mxnet.NDArray.empty(shape, ctx, dtype) + def empty(ctx: Context, shape: Array[Int]): NDArray + = org.apache.mxnet.NDArray.empty(new Shape(shape), ctx) + def empty(ctx: Context, shape: java.util.List[java.lang.Integer]): NDArray + = org.apache.mxnet.NDArray.empty(new Shape(shape), ctx) + + /** + * Create a new NDArray filled with 0, with specified shape. + * + * @param shape shape of the NDArray. + * @param ctx The context of the NDArray. + * + * @return The created NDArray. + */ + def zeros(shape: Shape, ctx: Context, dtype: DType.DType): NDArray + = org.apache.mxnet.NDArray.zeros(shape, ctx, dtype) + def zeros(ctx: Context, shape: Array[Int]): NDArray + = org.apache.mxnet.NDArray.zeros(new Shape(shape), ctx) + def zeros(ctx: Context, shape: java.util.List[java.lang.Integer]): NDArray + = org.apache.mxnet.NDArray.zeros(new Shape(shape), ctx) + + /** + * Create a new NDArray filled with 1, with specified shape. + * @param shape shape of the NDArray. + * @param ctx The context of the NDArray. + * @return The created NDArray. + */ + def ones(shape: Shape, ctx: Context, dtype: DType.DType): NDArray + = org.apache.mxnet.NDArray.ones(shape, ctx, dtype) + def ones(ctx: Context, shape: Array[Int]): NDArray + = org.apache.mxnet.NDArray.ones(new Shape(shape), ctx) + def ones(ctx: Context, shape: java.util.List[java.lang.Integer]): NDArray + = org.apache.mxnet.NDArray.ones(new Shape(shape), ctx) + + /** + * Create a new NDArray filled with given value, with specified shape. + * @param shape shape of the NDArray. + * @param value value to be filled with + * @param ctx The context of the NDArray + */ + def full(shape: Shape, value: Float, ctx: Context): NDArray + = org.apache.mxnet.NDArray.full(shape, value, ctx) + + def full(shape: Shape, value: Double, ctx: Context): NDArray + = org.apache.mxnet.NDArray.full(shape, value, ctx) + + def power(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.power(lhs, rhs) + def power(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.power(lhs, rhs) + def power(lhs: Float, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.power(lhs, rhs) + def power(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.power(lhs, rhs) + def power(lhs: Double, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.power(lhs, rhs) + + def maximum(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.maximum(lhs, rhs) + def maximum(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.maximum(lhs, rhs) + def maximum(lhs: Float, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.maximum(lhs, rhs) + def maximum(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.maximum(lhs, rhs) + def maximum(lhs: Double, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.maximum(lhs, rhs) + + def minimum(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.minimum(lhs, rhs) + def minimum(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.minimum(lhs, rhs) + def minimum(lhs: Float, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.minimum(lhs, rhs) + def minimum(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.minimum(lhs, rhs) + def minimum(lhs: Double, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.minimum(lhs, rhs) + + + /** + * Returns the result of element-wise **equal to** (==) comparison operation with broadcasting. + * For each element in input arrays, return 1(true) if corresponding elements are same, + * otherwise return 0(false). + */ + def equal(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.equal(lhs, rhs) + def equal(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.equal(lhs, rhs) + def equal(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.equal(lhs, rhs) + + /** + * Returns the result of element-wise **not equal to** (!=) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if corresponding elements are different, + * otherwise return 0(false). + */ + def notEqual(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.notEqual(lhs, rhs) + def notEqual(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.notEqual(lhs, rhs) + def notEqual(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.notEqual(lhs, rhs) + + /** + * Returns the result of element-wise **greater than** (>) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are greater than rhs, + * otherwise return 0(false). + */ + def greater(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.greater(lhs, rhs) + def greater(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.greater(lhs, rhs) + def greater(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.greater(lhs, rhs) + + /** + * Returns the result of element-wise **greater than or equal to** (>=) comparison + * operation with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are greater than equal to rhs + * otherwise return 0(false). + */ + def greaterEqual(lhs: NDArray, rhs: NDArray): NDArray + = org.apache.mxnet.NDArray.greaterEqual(lhs, rhs) + def greaterEqual(lhs: NDArray, rhs: Float): NDArray + = org.apache.mxnet.NDArray.greaterEqual(lhs, rhs) + def greaterEqual(lhs: NDArray, rhs: Double): NDArray + = org.apache.mxnet.NDArray.greaterEqual(lhs, rhs) + + /** + * Returns the result of element-wise **lesser than** (<) comparison operation + * with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are less than rhs, + * otherwise return 0(false). + */ + def lesser(lhs: NDArray, rhs: NDArray): NDArray = org.apache.mxnet.NDArray.lesser(lhs, rhs) + def lesser(lhs: NDArray, rhs: Float): NDArray = org.apache.mxnet.NDArray.lesser(lhs, rhs) + def lesser(lhs: NDArray, rhs: Double): NDArray = org.apache.mxnet.NDArray.lesser(lhs, rhs) + + /** + * Returns the result of element-wise **lesser than or equal to** (<=) comparison + * operation with broadcasting. + * For each element in input arrays, return 1(true) if lhs elements are + * lesser than equal to rhs, otherwise return 0(false). + */ + def lesserEqual(lhs: NDArray, rhs: NDArray): NDArray + = org.apache.mxnet.NDArray.lesserEqual(lhs, rhs) + def lesserEqual(lhs: NDArray, rhs: Float): NDArray + = org.apache.mxnet.NDArray.lesserEqual(lhs, rhs) + def lesserEqual(lhs: NDArray, rhs: Double): NDArray + = org.apache.mxnet.NDArray.lesserEqual(lhs, rhs) + + /** + * Create a new NDArray that copies content from source_array. + * @param sourceArr Source data to create NDArray from. + * @param shape shape of the NDArray + * @param ctx The context of the NDArray, default to current default context. + * @return The created NDArray. + */ + def array(sourceArr: java.util.List[java.lang.Float], shape: Shape, ctx: Context = null): NDArray + = org.apache.mxnet.NDArray.array( + sourceArr.asScala.map(ele => Float.unbox(ele)).toArray, shape, ctx) + + /** + * Create a new NDArray that copies content from source_array. + * @param sourceArr Source data (list of Doubles) to create NDArray from. + * @param shape shape of the NDArray + * @param ctx The context of the NDArray, default to current default context. + * @return The created NDArray. + */ + def arrayWithDouble(sourceArr: java.util.List[java.lang.Double], shape: Shape, + ctx: Context = null): NDArray + = org.apache.mxnet.NDArray.array( + sourceArr.asScala.map(ele => Double.unbox(ele)).toArray, shape) + + /** + * Returns evenly spaced values within a given interval. + * Values are generated within the half-open interval [`start`, `stop`). In other + * words, the interval includes `start` but excludes `stop`. + * @param start Start of interval. + * @param stop End of interval. + * @param step Spacing between values. + * @param repeat Number of times to repeat each element. + * @param ctx Device context. + * @param dType The data type of the `NDArray`. + * @return NDArray of evenly spaced values in the specified range. + */ + def arange(start: Float, stop: Float, step: Float, repeat: Int, + ctx: Context, dType: DType.DType): NDArray = + org.apache.mxnet.NDArray.arange(start, Some(stop), step, repeat, ctx, dType) +} + +/** + * NDArray object in mxnet. + * NDArray is basic ndarray/Tensor like data structure in mxnet.
      + * + * NOTE: NDArray is stored in native memory. Use NDArray in a try-with-resources() construct + * or a [[org.apache.mxnet.ResourceScope]] in a try-with-resource to have them + * automatically disposed. You can explicitly control the lifetime of NDArray + * by calling dispose manually. Failure to do this will result in leaking native memory. + * + */ +class NDArray private[mxnet] (val nd: org.apache.mxnet.NDArray ) { + + def this(arr: Array[Float], shape: Shape, ctx: Context) = { + this(org.apache.mxnet.NDArray.array(arr, shape, ctx)) + } + + def this(arr: Array[Double], shape: Shape, ctx: Context) = { + this(org.apache.mxnet.NDArray.array(arr, shape, ctx)) + } + + def this(arr: java.util.List[java.lang.Float], shape: Shape, ctx: Context) = { + this(NDArray.array(arr, shape, ctx)) + } + + def serialize(): Array[Byte] = nd.serialize() + + /** + * Release the native memory.
      + * The NDArrays it depends on will NOT be disposed.
      + * The object shall never be used after it is disposed. + */ + def dispose(): Unit = nd.dispose() + + /** + * Dispose all NDArrays who help to construct this array.
      + * e.g. (a * b + c).disposeDeps() will dispose a, b, c (including their deps) and a * b + * @return this array + */ + def disposeDeps(): NDArray = nd.disposeDepsExcept() + + /** + * Dispose all NDArrays who help to construct this array, excepts those in the arguments.
      + * e.g. (a * b + c).disposeDepsExcept(a, b) + * will dispose c and a * b. + * Note that a, b's dependencies will not be disposed either. + * @param arr the Array of NDArray not to dispose + * @return this array + */ + def disposeDepsExcept(arr: Array[NDArray]): NDArray = + nd.disposeDepsExcept(arr.map(NDArray.toNDArray): _*) + + /** + * Return a sliced NDArray that shares memory with current one. + * NDArray only support continuous slicing on axis 0 + * + * @param start Starting index of slice. + * @param stop Finishing index of slice. + * + * @return a sliced NDArray that shares memory with current one. + */ + def slice(start: Int, stop: Int): NDArray = nd.slice(start, stop) + + /** + * Return a sliced NDArray at the ith position of axis0 + * @param i + * @return a sliced NDArray that shares memory with current one. + */ + def slice (i: Int): NDArray = nd.slice(i) + + /** + * Return a sub NDArray that shares memory with current one. + * the first axis will be rolled up, which causes its shape different from slice(i, i+1) + * @param idx index of sub array. + */ + def at(idx: Int): NDArray = nd.at(idx) + + def T: NDArray = nd.T + + /** + * Get data type of current NDArray. + * @return class representing type of current ndarray + */ + def dtype: DType = nd.dtype + + /** + * Return a copied numpy array of current array with specified type. + * @param dtype Desired type of result array. + * @return A copy of array content. + */ + def asType(dtype: DType): NDArray = nd.asType(dtype) + + /** + * Return a reshaped NDArray that shares memory with current one. + * @param dims New shape. + * + * @return a reshaped NDArray that shares memory with current one. + */ + def reshape(dims: Array[Int]): NDArray = nd.reshape(dims) + + /** + * Block until all pending writes operations on current NDArray are finished. + * This function will return when all the pending writes to the current + * NDArray finishes. There can still be pending read going on when the + * function returns. + */ + def waitToRead(): Unit = nd.waitToRead() + + /** + * Get context of current NDArray. + * @return The context of current NDArray. + */ + def context: Context = nd.context + + /** + * Set the values of the NDArray + * @param value Value to set + * @return Current NDArray + */ + def set(value: Float): NDArray = nd.set(value) + def set(value: Double): NDArray = nd.set(value) + def set(other: NDArray): NDArray = nd.set(other) + def set(other: Array[Float]): NDArray = nd.set(other) + def set(other: Array[Double]): NDArray = nd.set(other) + + def add(other: NDArray): NDArray = this.nd + other.nd + def add(other: Float): NDArray = this.nd + other + def add(other: Double): NDArray = this.nd + other + def addInplace(other: NDArray): NDArray = this.nd += other + def addInplace(other: Float): NDArray = this.nd += other + def addInplace(other: Double): NDArray = this.nd += other + def subtract(other: NDArray): NDArray = this.nd - other + def subtract(other: Float): NDArray = this.nd - other + def subtract(other: Double): NDArray = this.nd - other + def subtractInplace(other: NDArray): NDArray = this.nd -= other + def subtractInplace(other: Float): NDArray = this.nd -= other + def subtractInplace(other: Double): NDArray = this.nd -= other + def multiply(other: NDArray): NDArray = this.nd * other + def multiply(other: Float): NDArray = this.nd * other + def multiply(other: Double): NDArray = this.nd * other + def multiplyInplace(other: NDArray): NDArray = this.nd *= other + def multiplyInplace(other: Float): NDArray = this.nd *= other + def multiplyInplace(other: Double): NDArray = this.nd *= other + def div(other: NDArray): NDArray = this.nd / other + def div(other: Float): NDArray = this.nd / other + def div(other: Double): NDArray = this.nd / other + def divInplace(other: NDArray): NDArray = this.nd /= other + def divInplace(other: Float): NDArray = this.nd /= other + def divInplace(other: Double): NDArray = this.nd /= other + def pow(other: NDArray): NDArray = this.nd ** other + def pow(other: Float): NDArray = this.nd ** other + def pow(other: Double): NDArray = this.nd ** other + def powInplace(other: NDArray): NDArray = this.nd **= other + def powInplace(other: Float): NDArray = this.nd **= other + def powInplace(other: Double): NDArray = this.nd **= other + def mod(other: NDArray): NDArray = this.nd % other + def mod(other: Float): NDArray = this.nd % other + def mod(other: Double): NDArray = this.nd % other + def modInplace(other: NDArray): NDArray = this.nd %= other + def modInplace(other: Float): NDArray = this.nd %= other + def modInplace(other: Double): NDArray = this.nd %= other + def greater(other: NDArray): NDArray = this.nd > other + def greater(other: Float): NDArray = this.nd > other + def greater(other: Double): NDArray = this.nd > other + def greaterEqual(other: NDArray): NDArray = this.nd >= other + def greaterEqual(other: Float): NDArray = this.nd >= other + def greaterEqual(other: Double): NDArray = this.nd >= other + def lesser(other: NDArray): NDArray = this.nd < other + def lesser(other: Float): NDArray = this.nd < other + def lesser(other: Double): NDArray = this.nd < other + def lesserEqual(other: NDArray): NDArray = this.nd <= other + def lesserEqual(other: Float): NDArray = this.nd <= other + def lesserEqual(other: Double): NDArray = this.nd <= other + + /** + * Return a copied flat java array of current array (row-major). + * @return A copy of array content. + */ + def toArray: Array[Float] = nd.toArray + + /** + * Return a copied flat java array of current array (row-major). + * @return A copy of array content. + */ + def toFloat64Array: Array[Double] = nd.toFloat64Array + + /** + * Return a CPU scalar(float) of current ndarray. + * This ndarray must have shape (1,) + * + * @return The scalar representation of the ndarray. + */ + def toScalar: Float = nd.toScalar + + /** + * Return a CPU scalar(float) of current ndarray. + * This ndarray must have shape (1,) + * + * @return The scalar representation of the ndarray. + */ + def toFloat64Scalar: Double = nd.toFloat64Scalar + + /** + * Copy the content of current array to other. + * + * @param other Target NDArray or context we want to copy data to. + * @return The copy target NDArray + */ + def copyTo(other: NDArray): NDArray = nd.copyTo(other) + + /** + * Copy the content of current array to a new NDArray in the context. + * + * @param ctx Target context we want to copy data to. + * @return The copy target NDArray + */ + def copyTo(ctx: Context): NDArray = nd.copyTo(ctx) + + /** + * Clone the current array + * @return the copied NDArray in the same context + */ + def copy(): NDArray = copyTo(this.context) + + /** + * Get shape of current NDArray. + * @return an array representing shape of current ndarray + */ + def shape: Shape = nd.shape + + + def size: Int = shape.product + + /** + * Return an `NDArray` that lives in the target context. If the array + * is already in that context, `self` is returned. Otherwise, a copy is made. + * @param context The target context we want the return value to live in. + * @return A copy or `self` as an `NDArray` that lives in the target context. + */ + def asInContext(context: Context): NDArray = nd.asInContext(context) + + override def equals(obj: Any): Boolean = nd.equals(obj) + override def hashCode(): Int = nd.hashCode +} diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Shape.scala b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Shape.scala index 594e3a605..b795fe31f 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Shape.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/javaapi/Shape.scala @@ -18,12 +18,13 @@ package org.apache.mxnet.javaapi import collection.JavaConverters._ +import scala.language.implicitConversions /** * Shape of [[NDArray]] or other data */ -class Shape(val shape: org.apache.mxnet.Shape) { +class Shape private[mxnet] (val shape: org.apache.mxnet.Shape) { def this(dims: java.util.List[java.lang.Integer]) = this(new org.apache.mxnet.Shape(dims.asScala.map(Int.unbox))) def this(dims: Array[Int]) = this(new org.apache.mxnet.Shape(dims)) diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/module/BaseModule.scala b/scala-package/core/src/main/scala/org/apache/mxnet/module/BaseModule.scala index 30e57c57f..b73f4ad4b 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/module/BaseModule.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/module/BaseModule.scala @@ -210,7 +210,7 @@ abstract class BaseModule { * @param reset Default is `True`, indicating whether we should reset the data iter before start * doing prediction. * @return The return value will be a nested list like - * `[[out1_batch1, out2_batch1, ...], [out1_batch2, out2_batch2, ...]]` + * `[ [out1_batch1, out2_batch1, ...], [out1_batch2, out2_batch2, ...] ]` * This mode is useful because in some cases (e.g. bucketing), * the module does not necessarily produce the same number of outputs. */ @@ -501,7 +501,7 @@ abstract class BaseModule { * Get outputs of the previous forward computation. * @return In the case when data-parallelism is used, * the outputs will be collected from multiple devices. - * The results will look like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`, + * The results will look like `[ [out1_dev1, out1_dev2], [out2_dev1, out2_dev2] ]`, * those `NDArray` might live on different devices. */ def getOutputs(): IndexedSeq[IndexedSeq[NDArray]] @@ -519,7 +519,7 @@ abstract class BaseModule { * Get the gradients to the inputs, computed in the previous backward computation. * @return In the case when data-parallelism is used, * the grads will be collected from multiple devices. - * The results will look like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`, + * The results will look like `[ [grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2] ]`, * those `NDArray` might live on different devices. */ def getInputGrads(): IndexedSeq[IndexedSeq[NDArray]] diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/module/BucketingModule.scala b/scala-package/core/src/main/scala/org/apache/mxnet/module/BucketingModule.scala index 2262f5c6a..1ac798e1b 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/module/BucketingModule.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/module/BucketingModule.scala @@ -339,7 +339,7 @@ class BucketingModule(symGen: AnyRef => (Symbol, IndexedSeq[String], IndexedSeq[ * Get outputs of the previous forward computation. * @return In the case when data-parallelism is used, * the outputs will be collected from multiple devices. - * The results will look like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`, + * The results will look like `[ [out1_dev1, out1_dev2], [out2_dev1, out2_dev2] ]`, * those `NDArray` might live on different devices. */ override def getOutputs(): IndexedSeq[IndexedSeq[NDArray]] = { @@ -363,7 +363,7 @@ class BucketingModule(symGen: AnyRef => (Symbol, IndexedSeq[String], IndexedSeq[ * Get the gradients to the inputs, computed in the previous backward computation. * @return In the case when data-parallelism is used, * the grads will be collected from multiple devices. - * The results will look like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`, + * The results will look like `[ [grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2] ]`, * those `NDArray` might live on different devices. */ override def getInputGrads(): IndexedSeq[IndexedSeq[NDArray]] = { diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/module/DataParallelExecutorGroup.scala b/scala-package/core/src/main/scala/org/apache/mxnet/module/DataParallelExecutorGroup.scala index 5c567fe5d..df66ea772 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/module/DataParallelExecutorGroup.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/module/DataParallelExecutorGroup.scala @@ -517,7 +517,7 @@ class DataParallelExecutorGroup private[module]( * Get outputs of the previous forward computation. * @return In the case when data-parallelism is used, * the outputs will be collected from multiple devices. - * The results will look like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`, + * The results will look like `[ [out1_dev1, out1_dev2], [out2_dev1, out2_dev2] ]`, * those `NDArray` might live on different devices. */ def getOutputs(): IndexedSeq[IndexedSeq[NDArray]] = { @@ -539,7 +539,7 @@ class DataParallelExecutorGroup private[module]( * Get the gradients to the inputs, computed in the previous backward computation. * @return In the case when data-parallelism is used, * the grads will be collected from multiple devices. - * The results will look like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`, + * The results will look like `[ [grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2] ]`, * those `NDArray` might live on different devices. */ def getInputGrads(): IndexedSeq[IndexedSeq[NDArray]] = { diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/module/Module.scala b/scala-package/core/src/main/scala/org/apache/mxnet/module/Module.scala index fec1ba0dc..97df3dcb3 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/module/Module.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/module/Module.scala @@ -486,7 +486,7 @@ class Module(symbolVar: Symbol, * Get outputs of the previous forward computation. * @return In the case when data-parallelism is used, * the outputs will be collected from multiple devices. - * The results will look like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`, + * The results will look like `[ [out1_dev1, out1_dev2], [out2_dev1, out2_dev2] ]`, * those `NDArray` might live on different devices. */ def getOutputs(): IndexedSeq[IndexedSeq[NDArray]] = { @@ -510,7 +510,7 @@ class Module(symbolVar: Symbol, * Get the gradients to the inputs, computed in the previous backward computation. * @return In the case when data-parallelism is used, * the grads will be collected from multiple devices. - * The results will look like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`, + * The results will look like `[ [grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2] ]`, * those `NDArray` might live on different devices. */ def getInputGrads(): IndexedSeq[IndexedSeq[NDArray]] = { diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/module/SequentialModule.scala b/scala-package/core/src/main/scala/org/apache/mxnet/module/SequentialModule.scala index e75550a1d..2e506c08e 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/module/SequentialModule.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/module/SequentialModule.scala @@ -346,7 +346,7 @@ class SequentialModule extends BaseModule { * Get outputs of the previous forward computation. * @return In the case when data-parallelism is used, * the outputs will be collected from multiple devices. - * The results will look like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`, + * The results will look like `[ [out1_dev1, out1_dev2], [out2_dev1, out2_dev2] ]`, * those `NDArray` might live on different devices. */ def getOutputs(): IndexedSeq[IndexedSeq[NDArray]] = { @@ -370,7 +370,7 @@ class SequentialModule extends BaseModule { * Get the gradients to the inputs, computed in the previous backward computation. * @return In the case when data-parallelism is used, * the grads will be collected from multiple devices. - * The results will look like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`, + * The results will look like `[ [grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2] ]`, * those `NDArray` might live on different devices. */ def getInputGrads(): IndexedSeq[IndexedSeq[NDArray]] = { diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/util/NativeLibraryLoader.scala b/scala-package/core/src/main/scala/org/apache/mxnet/util/NativeLibraryLoader.scala index e94d32039..1e6d9c4f9 100644 --- a/scala-package/core/src/main/scala/org/apache/mxnet/util/NativeLibraryLoader.scala +++ b/scala-package/core/src/main/scala/org/apache/mxnet/util/NativeLibraryLoader.scala @@ -85,12 +85,18 @@ private[mxnet] object NativeLibraryLoader { } logger.debug(s"Attempting to load $loadLibname") val libFileInJar = libPathInJar + loadLibname - val is: InputStream = getClass.getResourceAsStream(libFileInJar) - if (is == null) { - throw new UnsatisfiedLinkError(s"Couldn't find the resource $loadLibname") - } - logger.info(s"Loading $loadLibname from $libPathInJar copying to $libname") - loadLibraryFromStream(libname, is) + saveLibraryToTemp("libmxnet.so", "/lib/native/libmxnet.so", true) + saveLibraryToTemp("libgfortran.so.3", "/lib/native/libgfortran.so.3", false) + saveLibraryToTemp("libquadmath.so.0", "/lib/native/libquadmath.so.0", false) + saveLibraryToTemp("libiomp5.so", "/lib/native/libiomp5.so", false) + saveLibraryToTemp("libiomp5.dylib", "/lib/native/libiomp5.dylib", false) + saveLibraryToTemp("libmklml_intel.so", "/lib/native/libmklml_intel.so", false) + saveLibraryToTemp("libmklml.dylib", "/lib/native/libmklml.dylib", false) + saveLibraryToTemp("libmkldnn.so.0", "/lib/native/libmkldnn.so.0", false) + saveLibraryToTemp("libmkldnn.0.dylib", "/lib/native/libmkldnn.0.dylib", false) + val tempfile: File = saveLibraryToTemp(libname, libFileInJar, true) + + loadLibraryFromFile(libname, tempfile) } /** @@ -109,7 +115,7 @@ private[mxnet] object NativeLibraryLoader { @throws(classOf[IOException]) private def createTempFile(name: String): File = { - new File(_tempDir + File.separator + name) + new File(_tempDir, name) } /** @@ -117,34 +123,56 @@ private[mxnet] object NativeLibraryLoader { * and loads from there. * * @param libname name of the library (just used in constructing the library name) - * @param is InputStream pointing to the library + * @param tempfile File pointing to the library */ - private def loadLibraryFromStream(libname: String, is: InputStream) { + private def loadLibraryFromFile(libname: String, tempfile: File) { try { - val tempfile: File = createTempFile(libname) - val os: OutputStream = new FileOutputStream(tempfile) - logger.debug("tempfile.getPath() = {}", tempfile.getPath) - val savedTime: Long = System.currentTimeMillis - val buf: Array[Byte] = new Array[Byte](8192) - var len: Int = is.read(buf) - while (len > 0) { - os.write(buf, 0, len) - len = is.read(buf) - } - os.flush() - val lock: InputStream = new FileInputStream(tempfile) - os.close() - val seconds: Double = (System.currentTimeMillis - savedTime).toDouble / 1e3 - logger.debug(s"Copying took $seconds seconds.") logger.debug("Loading library from {}", tempfile.getPath) System.load(tempfile.getPath) - lock.close() } catch { - case io: IOException => - logger.error("Could not create the temp file: {}", io.toString) case ule: UnsatisfiedLinkError => logger.error("Couldn't load copied link file: {}", ule.toString) throw ule } } + + /** + * Load a system library from a stream. Copies the library to a temp file + * and loads from there. + * + * @param libname name of the library (just used in constructing the library name) + * @param resource String resource path in the jar file + * @param required true if library is required + */ + private def saveLibraryToTemp(libname: String, resource: String, required: Boolean): File = { + try { + val is: InputStream = getClass.getResourceAsStream(resource) + if (is == null) { + if (required) { + throw new UnsatisfiedLinkError(s"Couldn't find the resource $resource") + } else { + null + } + } else { + val tempfile: File = new File(_tempDir, libname) + val os: OutputStream = new FileOutputStream(tempfile) + logger.debug("tempfile.getPath() = {}", tempfile.getPath) + val savedTime: Long = System.currentTimeMillis + val buf: Array[Byte] = new Array[Byte](8192) + var len: Int = is.read(buf) + while (len > 0) { + os.write(buf, 0, len) + len = is.read(buf) + } + os.close() + is.close() + val seconds: Double = (System.currentTimeMillis - savedTime).toDouble / 1e3 + logger.debug(s"Copying $libname took $seconds seconds.") + tempfile + } + } catch { + case io: IOException => + throw new UnsatisfiedLinkError(s"Could not create temp file for $libname") + } + } } diff --git a/scala-package/core/src/main/scala/org/apache/mxnet/util/OptionConversion.scala b/scala-package/core/src/main/scala/org/apache/mxnet/util/OptionConversion.scala new file mode 100644 index 000000000..2cf453ac3 --- /dev/null +++ b/scala-package/core/src/main/scala/org/apache/mxnet/util/OptionConversion.scala @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.util + +object OptionConversion { + implicit def someWrapper[A](noSome : A) : Option[A] = Option(noSome) +} diff --git a/scala-package/core/src/test/java/org/apache/mxnet/javaapi/NDArrayTest.java b/scala-package/core/src/test/java/org/apache/mxnet/javaapi/NDArrayTest.java new file mode 100644 index 000000000..86c7eb29d --- /dev/null +++ b/scala-package/core/src/test/java/org/apache/mxnet/javaapi/NDArrayTest.java @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.javaapi; + +import org.junit.Test; + +import java.util.Arrays; +import java.util.List; +import org.apache.mxnet.javaapi.NDArrayBase.*; + +import static org.junit.Assert.assertTrue; + +public class NDArrayTest { + @Test + public void testCreateNDArray() { + NDArray nd = new NDArray(new float[]{1.0f, 2.0f, 3.0f}, + new Shape(new int[]{1, 3}), + new Context("cpu", 0)); + int[] arr = new int[]{1, 3}; + assertTrue(Arrays.equals(nd.shape().toArray(), arr)); + assertTrue(nd.at(0).at(0).toArray()[0] == 1.0f); + List list = Arrays.asList(1.0f, 2.0f, 3.0f); + // Second way creating NDArray + nd = NDArray.array(list, + new Shape(new int[]{1, 3}), + new Context("cpu", 0)); + assertTrue(Arrays.equals(nd.shape().toArray(), arr)); + + List list2 = Arrays.asList(1d, 1d, 1d); + nd = NDArray.arrayWithDouble(list2, + new Shape(new int[]{1, 3}), + new Context("cpu", 0)); + + // Float64 assertion + assertTrue(nd.dtype() == DType.Float64()); + + } + + @Test + public void testZeroOneEmpty(){ + NDArray ones = NDArray.ones(new Context("cpu", 0), new int[]{100, 100}); + NDArray zeros = NDArray.zeros(new Context("cpu", 0), new int[]{100, 100}); + NDArray empty = NDArray.empty(new Context("cpu", 0), new int[]{100, 100}); + int[] arr = new int[]{100, 100}; + assertTrue(Arrays.equals(ones.shape().toArray(), arr)); + assertTrue(Arrays.equals(zeros.shape().toArray(), arr)); + assertTrue(Arrays.equals(empty.shape().toArray(), arr)); + } + + @Test + public void testComparison(){ + NDArray nd = new NDArray(new float[]{1.0f, 2.0f, 3.0f}, new Shape(new int[]{3}), new Context("cpu", 0)); + NDArray nd2 = new NDArray(new float[]{3.0f, 4.0f, 5.0f}, new Shape(new int[]{3}), new Context("cpu", 0)); + nd = nd.add(nd2); + float[] greater = new float[]{1, 1, 1}; + assertTrue(Arrays.equals(nd.greater(nd2).toArray(), greater)); + nd = nd.subtract(nd2); + nd = nd.subtract(nd2); + float[] lesser = new float[]{0, 0, 0}; + assertTrue(Arrays.equals(nd.greater(nd2).toArray(), lesser)); + + NDArray nd3 = new NDArray(new double[]{1.0, 2.0, 3.0}, new Shape(new int[]{3}), new Context("cpu", 0)); + nd3 = nd3.add(1.0); + double[] smaller = new double[] {2, 3, 4}; + assertTrue(Arrays.equals(smaller, nd3.toFloat64Array())); + + } + + @Test + public void testGenerated(){ + NDArray$ NDArray = NDArray$.MODULE$; + float[] arr = new float[]{1.0f, 2.0f, 3.0f}; + NDArray nd = new NDArray(arr, new Shape(new int[]{3}), new Context("cpu", 0)); + float result = NDArray.norm(NDArray.new normParam(nd))[0].toArray()[0]; + float cal = 0.0f; + for (float ele : arr) { + cal += ele * ele; + } + cal = (float) Math.sqrt(cal); + assertTrue(Math.abs(result - cal) < 1e-5); + NDArray dotResult = new NDArray(new float[]{0}, new Shape(new int[]{1}), new Context("cpu", 0)); + NDArray.dot(NDArray.new dotParam(nd, nd).setOut(dotResult)); + assertTrue(Arrays.equals(dotResult.toArray(), new float[]{14.0f})); + } +} diff --git a/scala-package/core/src/test/java/org/apache/mxnet/javaapi/ResourceScopeTestSuite.java b/scala-package/core/src/test/java/org/apache/mxnet/javaapi/ResourceScopeTestSuite.java new file mode 100644 index 000000000..1c246d870 --- /dev/null +++ b/scala-package/core/src/test/java/org/apache/mxnet/javaapi/ResourceScopeTestSuite.java @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package org.apache.mxnet.javaapi; + +import org.apache.mxnet.NativeResourceRef; +import org.apache.mxnet.ResourceScope; +import org.junit.Test; + +import java.util.*; +import java.util.concurrent.Callable; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class ResourceScopeTestSuite { + + /** + * This is a placeholder class to test out whether NDArray References get collected or not when using + * try-with-resources in Java. + * + */ + class TestNDArray { + NDArray selfArray; + + public TestNDArray(Context context, int[] shape) { + this.selfArray = NDArray.ones(context, shape); + } + + public boolean verifyIsDisposed() { + return this.selfArray.nd().isDisposed(); + } + + public NativeResourceRef getNDArrayReference() { + return this.selfArray.nd().ref(); + } + } + + @Test + public void testNDArrayAutoRelease() { + TestNDArray test = null; + + try (ResourceScope scope = new ResourceScope()) { + test = new TestNDArray(Context.cpu(), new int[]{100, 100}); + } + + assertTrue(test.verifyIsDisposed()); + } + + @Test + public void testObjectReleaseFromList() { + List list = new ArrayList<>(); + + try (ResourceScope scope = new ResourceScope()) { + for (int i = 0;i < 10; i++) { + list.add(new TestNDArray(Context.cpu(), new int[] {100, 100})); + } + } + + assertEquals(list.size() , 10); + for (TestNDArray item : list) { + assertTrue(item.verifyIsDisposed()); + } + } + + @Test + public void testObjectReleaseFromMap() { + Map stringToNDArrayMap = new HashMap<>(); + + try (ResourceScope scope = new ResourceScope()) { + for (int i = 0;i < 10; i++) { + stringToNDArrayMap.put(String.valueOf(i),new TestNDArray(Context.cpu(), new int[] {i, i})); + } + } + + assertEquals(stringToNDArrayMap.size(), 10); + for (Map.Entry entry : stringToNDArrayMap.entrySet()) { + assertTrue(entry.getValue().verifyIsDisposed()); + } + + Map ndArrayToStringMap = new HashMap<>(); + + try (ResourceScope scope = new ResourceScope()) { + for (int i = 0;i < 10; i++) { + ndArrayToStringMap.put(new TestNDArray(Context.cpu(), new int[] {i, i}), String.valueOf(i)); + } + } + + assertEquals(ndArrayToStringMap.size(), 10); + for (Map.Entry entry : ndArrayToStringMap.entrySet()) { + assertTrue(entry.getKey().verifyIsDisposed()); + } + + } +} diff --git a/scala-package/core/src/test/scala/org/apache/mxnet/IOSuite.scala b/scala-package/core/src/test/scala/org/apache/mxnet/IOSuite.scala index 2ec6f668d..d3969b0ce 100644 --- a/scala-package/core/src/test/scala/org/apache/mxnet/IOSuite.scala +++ b/scala-package/core/src/test/scala/org/apache/mxnet/IOSuite.scala @@ -303,5 +303,32 @@ class IOSuite extends FunSuite with BeforeAndAfterAll { assert(dataDesc(0).layout == Layout.NTC) assert(labelDesc(0).dtype == DType.Int32) assert(labelDesc(0).layout == Layout.NT) + + + // Test with passing Float64 hardcoded as Dtype of data + val dataIter4 = new NDArrayIter( + IO.initDataDesc(data, false, "data", DType.Float64, Layout.NTC), + IO.initDataDesc(label, false, "label", DType.Int32, Layout.NT), + 128, false, "pad") + val dataDesc4 = dataIter4.provideDataDesc + val labelDesc4 = dataIter4.provideLabelDesc + assert(dataDesc4(0).dtype == DType.Float64) + assert(dataDesc4(0).layout == Layout.NTC) + assert(labelDesc4(0).dtype == DType.Int32) + assert(labelDesc4(0).layout == Layout.NT) + + // Test with Float64 coming from the data itself + val dataF64 = IndexedSeq(NDArray.ones(shape0, dtype = DType.Float64), + NDArray.zeros(shape0, dtype = DType.Float64)) + + val dataIter5 = new NDArrayIter( + IO.initDataDesc(dataF64, false, "data", DType.Float64, Layout.NTC), + IO.initDataDesc(label, false, "label", DType.Int32, Layout.NT), + 128, false, "pad") + val dataDesc5 = dataIter5.provideDataDesc + assert(dataDesc5(0).dtype == DType.Float64) + assert(dataDesc5(0).dtype != DType.Float32) + assert(dataDesc5(0).layout == Layout.NTC) + } } diff --git a/scala-package/core/src/test/scala/org/apache/mxnet/ImageSuite.scala b/scala-package/core/src/test/scala/org/apache/mxnet/ImageSuite.scala new file mode 100644 index 000000000..67815ad6c --- /dev/null +++ b/scala-package/core/src/test/scala/org/apache/mxnet/ImageSuite.scala @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet + +import java.io.File +import java.net.URL + +import javax.imageio.ImageIO +import org.apache.commons.io.FileUtils +import org.scalatest.{BeforeAndAfterAll, FunSuite} +import org.slf4j.LoggerFactory + +class ImageSuite extends FunSuite with BeforeAndAfterAll { + private var imLocation = "" + private val logger = LoggerFactory.getLogger(classOf[ImageSuite]) + + private def downloadUrl(url: String, filePath: String, maxRetry: Option[Int] = None) : Unit = { + val tmpFile = new File(filePath) + var retry = maxRetry.getOrElse(3) + var success = false + if (!tmpFile.exists()) { + while (retry > 0 && !success) { + try { + FileUtils.copyURLToFile(new URL(url), tmpFile) + success = true + } catch { + case e: Exception => retry -= 1 + } + } + } else { + success = true + } + if (!success) throw new Exception(s"$url Download failed!") + } + + override def beforeAll(): Unit = { + val tempDirPath = System.getProperty("java.io.tmpdir") + imLocation = tempDirPath + "/inputImages/Pug-Cookie.jpg" + downloadUrl("https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg", + imLocation) + } + + test("Test load image") { + val nd = Image.imRead(imLocation) + logger.info(s"OpenCV load image with shape: ${nd.shape}") + require(nd.shape == Shape(576, 1024, 3), "image shape not Match!") + } + + test("Test load image from Socket") { + val url = new URL("https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg") + val inputStream = url.openStream + val nd = Image.imDecode(inputStream) + logger.info(s"OpenCV load image with shape: ${nd.shape}") + require(nd.shape == Shape(576, 1024, 3), "image shape not Match!") + } + + test("Test resize image") { + val nd = Image.imRead(imLocation) + val resizeIm = Image.imResize(nd, 224, 224) + logger.info(s"OpenCV resize image with shape: ${resizeIm.shape}") + require(resizeIm.shape == Shape(224, 224, 3), "image shape not Match!") + } + + test("Test crop image") { + val nd = Image.imRead(imLocation) + val nd2 = Image.fixedCrop(nd, 0, 0, 224, 224) + require(nd2.shape == Shape(224, 224, 3), "image shape not Match!") + } + + test("Test apply border") { + val nd = Image.imRead(imLocation) + val nd2 = Image.copyMakeBorder(nd, 1, 1, 1, 1) + require(nd2.shape == Shape(578, 1026, 3), s"image shape not Match!") + } + + test("Test convert to Image") { + val nd = Image.imRead(imLocation) + val resizeIm = Image.imResize(nd, 224, 224) + val tempDirPath = System.getProperty("java.io.tmpdir") + val img = Image.toImage(resizeIm) + ImageIO.write(img, "png", new File(tempDirPath + "/inputImages/out.png")) + logger.info(s"converted image stored in ${tempDirPath + "/inputImages/out.png"}") + } + +} diff --git a/scala-package/core/src/test/scala/org/apache/mxnet/NDArraySuite.scala b/scala-package/core/src/test/scala/org/apache/mxnet/NDArraySuite.scala index 5d88bb39e..bc7a0a026 100644 --- a/scala-package/core/src/test/scala/org/apache/mxnet/NDArraySuite.scala +++ b/scala-package/core/src/test/scala/org/apache/mxnet/NDArraySuite.scala @@ -21,7 +21,7 @@ import java.io.File import java.util.concurrent.atomic.AtomicInteger import org.apache.mxnet.NDArrayConversions._ -import org.scalatest.{Matchers, BeforeAndAfterAll, FunSuite} +import org.scalatest.{BeforeAndAfterAll, FunSuite, Matchers} class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { private val sequence: AtomicInteger = new AtomicInteger(0) @@ -29,6 +29,9 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { test("to java array") { val ndarray = NDArray.zeros(2, 2) assert(ndarray.toArray === Array(0f, 0f, 0f, 0f)) + + val float64Array = NDArray.zeros(Shape(2, 2), dtype = DType.Float64) + assert(float64Array.toFloat64Array === Array(0d, 0d, 0d, 0d)) } test("to scalar") { @@ -38,8 +41,17 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { assert(ndones.toScalar === 1f) } + test("to float 64 scalar") { + val ndzeros = NDArray.zeros(Shape(1), dtype = DType.Float64) + assert(ndzeros.toFloat64Scalar === 0d) + val ndones = NDArray.ones(Shape(1), dtype = DType.Float64) + assert(ndones.toFloat64Scalar === 1d) + } + test ("call toScalar on an ndarray which is not a scalar") { intercept[Exception] { NDArray.zeros(1, 1).toScalar } + intercept[Exception] { NDArray.zeros(shape = Shape (1, 1), + dtype = DType.Float64).toFloat64Scalar } } test("size and shape") { @@ -51,12 +63,20 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { test("dtype") { val arr = NDArray.zeros(3, 2) assert(arr.dtype === DType.Float32) + + val float64Array = NDArray.zeros(shape = Shape(3, 2), dtype = DType.Float64) + assert(float64Array.dtype === DType.Float64) } test("set scalar value") { val ndarray = NDArray.empty(2, 1) ndarray.set(10f) assert(ndarray.toArray === Array(10f, 10f)) + + val float64array = NDArray.empty(shape = Shape(2, 1), dtype = DType.Float64) + float64array.set(10d) + assert(float64array.toFloat64Array === Array(10d, 10d)) + } test("copy from java array") { @@ -66,19 +86,29 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { } test("plus") { - val ndzeros = NDArray.zeros(2, 1) - val ndones = ndzeros + 1f + var ndzeros = NDArray.zeros(2, 1) + var ndones = ndzeros + 1f assert(ndones.toArray === Array(1f, 1f)) assert((ndones + ndzeros).toArray === Array(1f, 1f)) assert((1 + ndones).toArray === Array(2f, 2f)) // in-place ndones += ndones assert(ndones.toArray === Array(2f, 2f)) + + // Float64 method test + ndzeros = NDArray.zeros(shape = Shape(2, 1), dtype = DType.Float64) + ndones = ndzeros + 1d + assert(ndones.toFloat64Array === Array(1d, 1d)) + assert((ndones + ndzeros).toFloat64Array === Array(1d, 1d)) + assert((1d + ndones).toArray === Array(2d, 2d)) + // in-place + ndones += ndones + assert(ndones.toFloat64Array === Array(2d, 2d)) } test("minus") { - val ndones = NDArray.ones(2, 1) - val ndzeros = ndones - 1f + var ndones = NDArray.ones(2, 1) + var ndzeros = ndones - 1f assert(ndzeros.toArray === Array(0f, 0f)) assert((ndones - ndzeros).toArray === Array(1f, 1f)) assert((ndzeros - ndones).toArray === Array(-1f, -1f)) @@ -86,23 +116,46 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { // in-place ndones -= ndones assert(ndones.toArray === Array(0f, 0f)) + + // Float64 methods test + ndones = NDArray.ones(shape = Shape(2, 1)) + ndzeros = ndones - 1d + assert(ndzeros.toFloat64Array === Array(0d, 0d)) + assert((ndones - ndzeros).toFloat64Array === Array(1d , 1d)) + assert((ndzeros - ndones).toFloat64Array === Array(-1d , -1d)) + assert((ndones - 1).toFloat64Array === Array(0d, 0d)) + // in-place + ndones -= ndones + assert(ndones.toArray === Array(0d, 0d)) + } test("multiplication") { - val ndones = NDArray.ones(2, 1) - val ndtwos = ndones * 2 + var ndones = NDArray.ones(2, 1) + var ndtwos = ndones * 2 assert(ndtwos.toArray === Array(2f, 2f)) assert((ndones * ndones).toArray === Array(1f, 1f)) assert((ndtwos * ndtwos).toArray === Array(4f, 4f)) ndtwos *= ndtwos // in-place assert(ndtwos.toArray === Array(4f, 4f)) + + // Float64 methods test + ndones = NDArray.ones(shape = Shape(2, 1), dtype = DType.Float64) + ndtwos = ndones * 2d + assert(ndtwos.toFloat64Array === Array(2d, 2d)) + assert((ndones * ndones).toFloat64Array === Array(1d, 1d)) + assert((ndtwos * ndtwos).toFloat64Array === Array(4d, 4d)) + ndtwos *= ndtwos + // in-place + assert(ndtwos.toFloat64Array === Array(4d, 4d)) + } test("division") { - val ndones = NDArray.ones(2, 1) - val ndzeros = ndones - 1f - val ndhalves = ndones / 2 + var ndones = NDArray.ones(2, 1) + var ndzeros = ndones - 1f + var ndhalves = ndones / 2 assert(ndhalves.toArray === Array(0.5f, 0.5f)) assert((ndhalves / ndhalves).toArray === Array(1f, 1f)) assert((ndones / ndones).toArray === Array(1f, 1f)) @@ -110,37 +163,75 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { ndhalves /= ndhalves // in-place assert(ndhalves.toArray === Array(1f, 1f)) + + // Float64 methods test + ndones = NDArray.ones(shape = Shape (2, 1), dtype = DType.Float64) + ndzeros = ndones - 1d + ndhalves = ndones / 2d + assert(ndhalves.toFloat64Array === Array(0.5d, 0.5d)) + assert((ndhalves / ndhalves).toFloat64Array === Array(1d, 1d)) + assert((ndones / ndones).toFloat64Array === Array(1d, 1d)) + assert((ndzeros / ndones).toFloat64Array === Array(0d, 0d)) + ndhalves /= ndhalves + // in-place + assert(ndhalves.toFloat64Array === Array(1d, 1d)) } test("full") { - val arr = NDArray.full(Shape(1, 2), 3f) + var arr = NDArray.full(Shape(1, 2), 3f) assert(arr.shape === Shape(1, 2)) assert(arr.toArray === Array(3f, 3f)) + + // Float64 methods test + arr = NDArray.full(Shape(1, 2), value = 5d, Context.cpu()) + assert(arr.toFloat64Array === Array (5d, 5d)) } test("clip") { - val ndarray = NDArray.empty(3, 2) + var ndarray = NDArray.empty(3, 2) ndarray.set(Array(1f, 2f, 3f, 4f, 5f, 6f)) assert(NDArray.clip(ndarray, 2f, 5f).toArray === Array(2f, 2f, 3f, 4f, 5f, 5f)) + + // Float64 methods test + ndarray = NDArray.empty(shape = Shape(3, 2), dtype = DType.Float64) + ndarray.set(Array(1d, 2d, 3d, 4d, 5d, 6d)) + assert(NDArray.clip(ndarray, 2d, 5d).toFloat64Array === Array(2d, 2d, 3d, 4d, 5d, 5d)) } test("sqrt") { - val ndarray = NDArray.empty(4, 1) + var ndarray = NDArray.empty(4, 1) ndarray.set(Array(0f, 1f, 4f, 9f)) assert(NDArray.sqrt(ndarray).toArray === Array(0f, 1f, 2f, 3f)) + + // Float64 methods test + ndarray = NDArray.empty(shape = Shape(4, 1), dtype = DType.Float64) + ndarray.set(Array(0d, 1d, 4d, 9d)) + assert(NDArray.sqrt(ndarray).toFloat64Array === Array(0d, 1d, 2d, 3d)) } test("rsqrt") { - val ndarray = NDArray.array(Array(1f, 4f), shape = Shape(2, 1)) + var ndarray = NDArray.array(Array(1f, 4f), shape = Shape(2, 1)) assert(NDArray.rsqrt(ndarray).toArray === Array(1f, 0.5f)) + + // Float64 methods test + ndarray = NDArray.array(Array(1d, 4d, 25d), shape = Shape(3, 1), Context.cpu()) + assert(NDArray.rsqrt(ndarray).toFloat64Array === Array(1d, 0.5d, 0.2d)) } test("norm") { - val ndarray = NDArray.empty(3, 1) + var ndarray = NDArray.empty(3, 1) ndarray.set(Array(1f, 2f, 3f)) - val normed = NDArray.norm(ndarray) + var normed = NDArray.norm(ndarray) assert(normed.shape === Shape(1)) assert(normed.toScalar === math.sqrt(14.0).toFloat +- 1e-3f) + + // Float64 methods test + ndarray = NDArray.empty(shape = Shape(3, 1), dtype = DType.Float64) + ndarray.set(Array(1d, 2d, 3d)) + normed = NDArray.norm(ndarray) + assert(normed.get.dtype === DType.Float64) + assert(normed.shape === Shape(1)) + assert(normed.toFloat64Scalar === math.sqrt(14.0) +- 1e-3d) } test("one hot encode") { @@ -176,25 +267,26 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { } test("power") { - val arr = NDArray.array(Array(3f, 5f), shape = Shape(2, 1)) + var arr = NDArray.array(Array(3f, 5f), shape = Shape(2, 1)) - val arrPower1 = NDArray.power(2f, arr) + var arrPower1 = NDArray.power(2f, arr) assert(arrPower1.shape === Shape(2, 1)) assert(arrPower1.toArray === Array(8f, 32f)) - val arrPower2 = NDArray.power(arr, 2f) + var arrPower2 = NDArray.power(arr, 2f) assert(arrPower2.shape === Shape(2, 1)) assert(arrPower2.toArray === Array(9f, 25f)) - val arrPower3 = NDArray.power(arr, arr) + var arrPower3 = NDArray.power(arr, arr) assert(arrPower3.shape === Shape(2, 1)) assert(arrPower3.toArray === Array(27f, 3125f)) - val arrPower4 = arr ** 2f + var arrPower4 = arr ** 2f + assert(arrPower4.shape === Shape(2, 1)) assert(arrPower4.toArray === Array(9f, 25f)) - val arrPower5 = arr ** arr + var arrPower5 = arr ** arr assert(arrPower5.shape === Shape(2, 1)) assert(arrPower5.toArray === Array(27f, 3125f)) @@ -206,84 +298,211 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { arr **= arr assert(arr.shape === Shape(2, 1)) assert(arr.toArray === Array(27f, 3125f)) + + // Float64 tests + arr = NDArray.array(Array(3d, 5d), shape = Shape(2, 1)) + + arrPower1 = NDArray.power(2d, arr) + assert(arrPower1.shape === Shape(2, 1)) + assert(arrPower1.dtype === DType.Float64) + assert(arrPower1.toFloat64Array === Array(8d, 32d)) + + arrPower2 = NDArray.power(arr, 2d) + assert(arrPower2.shape === Shape(2, 1)) + assert(arrPower2.dtype === DType.Float64) + assert(arrPower2.toFloat64Array === Array(9d, 25d)) + + arrPower3 = NDArray.power(arr, arr) + assert(arrPower3.shape === Shape(2, 1)) + assert(arrPower3.dtype === DType.Float64) + assert(arrPower3.toFloat64Array === Array(27d, 3125d)) + + arrPower4 = arr ** 2f + assert(arrPower4.shape === Shape(2, 1)) + assert(arrPower4.dtype === DType.Float64) + assert(arrPower4.toFloat64Array === Array(9d, 25d)) + + arrPower5 = arr ** arr + assert(arrPower5.shape === Shape(2, 1)) + assert(arrPower5.dtype === DType.Float64) + assert(arrPower5.toFloat64Array === Array(27d, 3125d)) + + arr **= 2d + assert(arr.shape === Shape(2, 1)) + assert(arr.dtype === DType.Float64) + assert(arr.toFloat64Array === Array(9d, 25d)) + + arr.set(Array(3d, 5d)) + arr **= arr + assert(arr.shape === Shape(2, 1)) + assert(arr.dtype === DType.Float64) + assert(arr.toFloat64Array === Array(27d, 3125d)) } test("equal") { - val arr1 = NDArray.array(Array(1f, 2f, 3f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 3f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = NDArray.equal(arr1, arr2) + var arrEqual1 = NDArray.equal(arr1, arr2) assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(1f, 0f, 1f, 0f)) - val arrEqual2 = NDArray.equal(arr1, 3f) + var arrEqual2 = NDArray.equal(arr1, 3f) assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(0f, 0f, 1f, 0f)) + + + // Float64 methods test + arr1 = NDArray.array(Array(1d, 2d, 3d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = NDArray.equal(arr1, arr2) + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(1d, 0d, 1d, 0d)) + + arrEqual2 = NDArray.equal(arr1, 3d) + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(0d, 0d, 1d, 0d)) } test("not_equal") { - val arr1 = NDArray.array(Array(1f, 2f, 3f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 3f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = NDArray.notEqual(arr1, arr2) + var arrEqual1 = NDArray.notEqual(arr1, arr2) assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(0f, 1f, 0f, 1f)) - val arrEqual2 = NDArray.notEqual(arr1, 3f) + var arrEqual2 = NDArray.notEqual(arr1, 3f) assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(1f, 1f, 0f, 1f)) + + // Float64 methods test + + arr1 = NDArray.array(Array(1d, 2d, 3d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = NDArray.notEqual(arr1, arr2) + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(0d, 1d, 0d, 1d)) + + arrEqual2 = NDArray.notEqual(arr1, 3d) + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(1d, 1d, 0d, 1d)) + } test("greater") { - val arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = arr1 > arr2 + var arrEqual1 = arr1 > arr2 assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(0f, 0f, 1f, 0f)) - val arrEqual2 = arr1 > 2f + var arrEqual2 = arr1 > 2f assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(0f, 0f, 1f, 1f)) + + // Float64 methods test + arr1 = NDArray.array(Array(1d, 2d, 4d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = arr1 > arr2 + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(0d, 0d, 1d, 0d)) + + arrEqual2 = arr1 > 2d + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(0d, 0d, 1d, 1d)) } test("greater_equal") { - val arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = arr1 >= arr2 + var arrEqual1 = arr1 >= arr2 assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(1f, 0f, 1f, 0f)) - val arrEqual2 = arr1 >= 2f + var arrEqual2 = arr1 >= 2f assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(0f, 1f, 1f, 1f)) + + // Float64 methods test + arr1 = NDArray.array(Array(1d, 2d, 4d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = arr1 >= arr2 + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(1d, 0d, 1d, 0d)) + + arrEqual2 = arr1 >= 2d + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(0d, 1d, 1d, 1d)) } test("lesser") { - val arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = arr1 < arr2 + var arrEqual1 = arr1 < arr2 assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(0f, 1f, 0f, 1f)) - val arrEqual2 = arr1 < 2f + var arrEqual2 = arr1 < 2f assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(1f, 0f, 0f, 0f)) + + // Float64 methods test + arr1 = NDArray.array(Array(1d, 2d, 4d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = arr1 < arr2 + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(0d, 1d, 0d, 1d)) + + arrEqual2 = arr1 < 2d + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(1d, 0d, 0d, 0d)) + } test("lesser_equal") { - val arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) - val arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) + var arr1 = NDArray.array(Array(1f, 2f, 4f, 5f), shape = Shape(2, 2)) + var arr2 = NDArray.array(Array(1f, 4f, 3f, 6f), shape = Shape(2, 2)) - val arrEqual1 = arr1 <= arr2 + var arrEqual1 = arr1 <= arr2 assert(arrEqual1.shape === Shape(2, 2)) assert(arrEqual1.toArray === Array(1f, 1f, 0f, 1f)) - val arrEqual2 = arr1 <= 2f + var arrEqual2 = arr1 <= 2f assert(arrEqual2.shape === Shape(2, 2)) assert(arrEqual2.toArray === Array(1f, 1f, 0f, 0f)) + + // Float64 methods test + arr1 = NDArray.array(Array(1d, 2d, 4d, 5d), shape = Shape(2, 2)) + arr2 = NDArray.array(Array(1d, 4d, 3d, 6d), shape = Shape(2, 2)) + + arrEqual1 = arr1 <= arr2 + assert(arrEqual1.shape === Shape(2, 2)) + assert(arrEqual1.dtype === DType.Float64) + assert(arrEqual1.toFloat64Array === Array(1d, 1d, 0d, 1d)) + + arrEqual2 = arr1 <= 2d + assert(arrEqual2.shape === Shape(2, 2)) + assert(arrEqual2.dtype === DType.Float64) + assert(arrEqual2.toFloat64Array === Array(1d, 1d, 0d, 0d)) } test("choose_element_0index") { @@ -294,11 +513,18 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { } test("copy to") { - val source = NDArray.array(Array(1f, 2f, 3f), shape = Shape(1, 3)) - val dest = NDArray.empty(1, 3) + var source = NDArray.array(Array(1f, 2f, 3f), shape = Shape(1, 3)) + var dest = NDArray.empty(1, 3) source.copyTo(dest) assert(dest.shape === Shape(1, 3)) assert(dest.toArray === Array(1f, 2f, 3f)) + + // Float64 methods test + source = NDArray.array(Array(1d, 2d, 3d), shape = Shape(1, 3)) + dest = NDArray.empty(shape = Shape(1, 3), dtype = DType.Float64) + source.copyTo(dest) + assert(dest.dtype === DType.Float64) + assert(dest.toFloat64Array === Array(1d, 2d, 3d)) } test("abs") { @@ -365,6 +591,12 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val arr = NDArray.maximum(arr1, arr2) assert(arr.shape === Shape(3, 1)) assert(arr.toArray === Array(4f, 2.1f, 3.7f)) + + // Float64 methods test + val arr3 = NDArray.array(Array(1d, 2d, 3d), shape = Shape(3, 1)) + val maxArr = NDArray.maximum(arr3, 10d) + assert(maxArr.shape === Shape(3, 1)) + assert(maxArr.toArray === Array(10d, 10d, 10d)) } test("min") { @@ -378,11 +610,18 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val arr = NDArray.minimum(arr1, arr2) assert(arr.shape === Shape(3, 1)) assert(arr.toArray === Array(1.5f, 1f, 3.5f)) + + // Float64 methods test + val arr3 = NDArray.array(Array(4d, 5d, 6d), shape = Shape(3, 1)) + val minArr = NDArray.minimum(arr3, 5d) + assert(minArr.shape === Shape(3, 1)) + assert(minArr.toFloat64Array === Array(4d, 5d, 5d)) } test("sum") { - val arr = NDArray.array(Array(1f, 2f, 3f, 4f), shape = Shape(2, 2)) + var arr = NDArray.array(Array(1f, 2f, 3f, 4f), shape = Shape(2, 2)) assert(NDArray.sum(arr).toScalar === 10f +- 1e-3f) + } test("argmaxChannel") { @@ -398,6 +637,12 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val arr = NDArray.concatenate(arr1, arr2) assert(arr.shape === Shape(3, 3)) assert(arr.toArray === Array(1f, 2f, 4f, 3f, 3f, 3f, 8f, 7f, 6f)) + + // Try concatenating float32 arr with float64 arr. Should get exception + intercept[Exception] { + val arr3 = NDArray.array(Array (5d, 6d, 7d), shape = Shape(1, 3)) + NDArray.concatenate(Array(arr1, arr3)) + } } test("concatenate axis-1") { @@ -406,6 +651,12 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val arr = NDArray.concatenate(Array(arr1, arr2), axis = 1) assert(arr.shape === Shape(2, 3)) assert(arr.toArray === Array(1f, 2f, 5f, 3f, 4f, 6f)) + + // Try concatenating float32 arr with float64 arr. Should get exception + intercept[Exception] { + val arr3 = NDArray.array(Array (5d, 6d), shape = Shape(2, 1)) + NDArray.concatenate(Array(arr1, arr3), axis = 1) + } } test("transpose") { @@ -428,6 +679,24 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val loadedArray = arrays(0) assert(loadedArray.shape === Shape(3, 1)) assert(loadedArray.toArray === Array(1f, 2f, 3f)) + assert(loadedArray.dtype === DType.Float32) + } finally { + val file = new File(filename) + file.delete() + } + + // Try the same for Float64 array + try { + val ndarray = NDArray.array(Array(1d, 2d, 3d), shape = Shape(3, 1), ctx = Context.cpu()) + NDArray.save(filename, Map("local" -> ndarray)) + val (keys, arrays) = NDArray.load(filename) + assert(keys.length === 1) + assert(keys(0) === "local") + assert(arrays.length === 1) + val loadedArray = arrays(0) + assert(loadedArray.shape === Shape(3, 1)) + assert(loadedArray.toArray === Array(1d, 2d, 3d)) + assert(loadedArray.dtype === DType.Float64) } finally { val file = new File(filename) file.delete() @@ -446,6 +715,24 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val loadedArray = arrays(0) assert(loadedArray.shape === Shape(3, 1)) assert(loadedArray.toArray === Array(1f, 2f, 3f)) + assert(loadedArray.dtype === DType.Float32) + } finally { + val file = new File(filename) + file.delete() + } + + // Try the same thing for Float64 array : + + try { + val ndarray = NDArray.array(Array(1d, 2d, 3d), shape = Shape(3, 1), ctx = Context.cpu()) + NDArray.save(filename, Array(ndarray)) + val (keys, arrays) = NDArray.load(filename) + assert(keys.length === 0) + assert(arrays.length === 1) + val loadedArray = arrays(0) + assert(loadedArray.shape === Shape(3, 1)) + assert(loadedArray.toArray === Array(1d, 2d, 3d)) + assert(loadedArray.dtype === DType.Float64) } finally { val file = new File(filename) file.delete() @@ -464,9 +751,11 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val ndarray2 = NDArray.array(Array(1f, 2f, 3f), shape = Shape(3, 1)) val ndarray3 = NDArray.array(Array(1f, 2f, 3f), shape = Shape(1, 3)) val ndarray4 = NDArray.array(Array(3f, 2f, 3f), shape = Shape(3, 1)) + val ndarray5 = NDArray.array(Array(3d, 2d, 3d), shape = Shape(3, 1), ctx = Context.cpu()) ndarray1 shouldEqual ndarray2 ndarray1 shouldNot equal(ndarray3) ndarray1 shouldNot equal(ndarray4) + ndarray5 shouldNot equal(ndarray3) } test("slice") { @@ -545,6 +834,7 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { val bytes = arr.serialize() val arrCopy = NDArray.deserialize(bytes) assert(arr === arrCopy) + assert(arrCopy.dtype === DType.Float32) } test("dtype int32") { @@ -576,4 +866,38 @@ class NDArraySuite extends FunSuite with BeforeAndAfterAll with Matchers { assert(arr.internal.toDoubleArray === Array(2d, 2d)) assert(arr.internal.toByteArray === Array(2.toByte, 2.toByte)) } + + test("NDArray random module is generated properly") { + val lam = NDArray.ones(1, 2) + val rnd = NDArray.random.poisson(lam = Some(lam), shape = Some(Shape(3, 4))) + val rnd2 = NDArray.random.poisson(lam = Some(1f), shape = Some(Shape(3, 4)), + dtype = Some("float64")) + assert(rnd.shape === Shape(1, 2, 3, 4)) + assert(rnd2.shape === Shape(3, 4)) + assert(rnd2.head.dtype === DType.Float64) + } + + test("NDArray random module is generated properly - special case of 'normal'") { + val mu = NDArray.ones(1, 2) + val sigma = NDArray.ones(1, 2) * 2 + val rnd = NDArray.random.normal(mu = Some(mu), sigma = Some(sigma), shape = Some(Shape(3, 4))) + val rnd2 = NDArray.random.normal(mu = Some(1f), sigma = Some(2f), shape = Some(Shape(3, 4)), + dtype = Some("float64")) + assert(rnd.shape === Shape(1, 2, 3, 4)) + assert(rnd2.shape === Shape(3, 4)) + assert(rnd2.head.dtype === DType.Float64) + } + + test("Generated api") { + // Without SomeConversion + val arr3 = NDArray.ones(Shape(1, 2), dtype = DType.Float64) + val arr4 = NDArray.ones(Shape(1), dtype = DType.Float64) + val arr5 = NDArray.api.norm(arr3, ord = Some(1), out = Some(arr4)) + // With SomeConversion + import org.apache.mxnet.util.OptionConversion._ + val arr = NDArray.ones(Shape(1, 2), dtype = DType.Float64) + val arr2 = NDArray.ones(Shape(1), dtype = DType.Float64) + NDArray.api.norm(arr, ord = 1, out = arr2) + val result = NDArray.api.dot(arr2, arr2) + } } diff --git a/scala-package/core/src/test/scala/org/apache/mxnet/SymbolSuite.scala b/scala-package/core/src/test/scala/org/apache/mxnet/SymbolSuite.scala index ebb61d7d4..d134c83ff 100644 --- a/scala-package/core/src/test/scala/org/apache/mxnet/SymbolSuite.scala +++ b/scala-package/core/src/test/scala/org/apache/mxnet/SymbolSuite.scala @@ -20,6 +20,7 @@ package org.apache.mxnet import org.scalatest.{BeforeAndAfterAll, FunSuite} class SymbolSuite extends FunSuite with BeforeAndAfterAll { + test("symbol compose") { val data = Symbol.Variable("data") @@ -71,4 +72,25 @@ class SymbolSuite extends FunSuite with BeforeAndAfterAll { val data2 = data.clone() assert(data.toJson === data2.toJson) } + + test("Symbol random module is generated properly") { + val lam = Symbol.Variable("lam") + val rnd = Symbol.random.poisson(lam = Some(lam), shape = Some(Shape(2, 2))) + val rnd2 = Symbol.random.poisson(lam = Some(1f), shape = Some(Shape(2, 2))) + // scalastyle:off println + println(s"Symbol.random.poisson debug info: ${rnd.debugStr}") + println(s"Symbol.random.poisson debug info: ${rnd2.debugStr}") + // scalastyle:on println + } + + test("Symbol random module is generated properly - special case of 'normal'") { + val loc = Symbol.Variable("loc") + val scale = Symbol.Variable("scale") + val rnd = Symbol.random.normal(mu = Some(loc), sigma = Some(scale), shape = Some(Shape(2, 2))) + val rnd2 = Symbol.random.normal(mu = Some(1f), sigma = Some(2f), shape = Some(Shape(2, 2))) + // scalastyle:off println + println(s"Symbol.random.sample_normal debug info: ${rnd.debugStr}") + println(s"Symbol.random.random_normal debug info: ${rnd2.debugStr}") + // scalastyle:on println + } } diff --git a/scala-package/deploy/pom.xml b/scala-package/deploy/pom.xml new file mode 100644 index 000000000..c51aa9a92 --- /dev/null +++ b/scala-package/deploy/pom.xml @@ -0,0 +1,140 @@ + + + 4.0.0 + + org.apache.mxnet + mxnet-parent + INTERNAL + ../pom.xml + + + mxnet-deployment + ${revision} + MXNet Scala Package - Full ${platform}-only + pom + + Scala Package for Apache MXNet (Incubating) - flexible and efficient library for deep learning. + + + + mxnet-full_2.11-${platform}-${flavor} + 1.5.0-SNAPSHOT + apache.snapshots.https + file://${project.build.directory}/repo + + + + + + + + org.apache.mxnet + mxnet-full_2.11 + INTERNAL + + + + + + staging + + 1.5.0 + apache.releases.https + https://repository.apache.org/content/repositories/staging + jar.asc,asc,asc + ,sources.jar,javadoc.jar + + ../assembly/target/mxnet-full_2.11-INTERNAL.jar.asc,../assembly/target/mxnet-full_2.11-INTERNAL-src.jar.asc,../assembly/target/mxnet-full_2.11-INTERNAL-bundle.jar.asc + + + + + nightly + + apache.snapshots.https + https://repository.apache.org/content/repositories/snapshots + + + + + + + + org.apache.maven.plugins + maven-assembly-plugin + + false + + + + + com.google.code.maven-replacer-plugin + replacer + 1.5.3 + + + deploy + + replace + + + + + ${basedir}/src/main/deploy/deploy.xml + ${project.build.directory}/deploy.xml + + + DESCRIPTION + ${project.description} + + + ARTIFACT_ID + ${ARTIFACT_ID} + + + PROJECT_VERSION + ${project.version} + + + SCALA_VERSION + ${scala.version} + + + + + + + org.apache.maven.plugins + maven-deploy-plugin + + + deploy-file + deploy + + deploy-file + + + ${project.description} + ${repositoryId} + ${repo_url} + ${project.groupId} + ${ARTIFACT_ID} + ${project.version} + jar + ${project.build.directory}/deploy.xml + ${rootdir}/assembly/target/mxnet-full_2.11-INTERNAL.jar + ${rootdir}/assembly/target/mxnet-full_2.11-INTERNAL-src.jar + ${rootdir}/assembly/target/mxnet-full_2.11-INTERNAL-bundle.jar + ${deploy_asc_types} + ${deploy_asc_classifers} + ${deploy_asc_files} + + + + + + + + diff --git a/scala-package/deploy/src/main/deploy/deploy.xml b/scala-package/deploy/src/main/deploy/deploy.xml new file mode 100644 index 000000000..7bfd20737 --- /dev/null +++ b/scala-package/deploy/src/main/deploy/deploy.xml @@ -0,0 +1,33 @@ + + + 4.0.0 + org.apache.mxnet + ARTIFACT_ID + PROJECT_VERSION + DESCRIPTION + + + org.scala-lang + scala-library + SCALA_VERSION + + + org.scala-lang + scala-reflect + SCALA_VERSION + + + org.scala-lang.modules + scala-parser-combinators_2.11 + 1.0.4 + + + org.scala-lang + scala-compiler + SCALA_VERSION + + + diff --git a/scala-package/dev/compile-mxnet-backend.sh b/scala-package/dev/compile-mxnet-backend.sh index b065e01af..114bf0766 100755 --- a/scala-package/dev/compile-mxnet-backend.sh +++ b/scala-package/dev/compile-mxnet-backend.sh @@ -33,7 +33,7 @@ MXNETDIR=$2 # below routine shamelessly copied from # https://github.com/apache/incubator-mxnet/blob/master/setup-utils/install-mxnet-osx-python.sh -# This routine executes a command, +# This routine executes a command, # prints error message on the console on non-zero exit codes and # returns the exit code to the caller. chkret() { @@ -51,7 +51,10 @@ chkret() { UNAME=`uname -s` chkret pushd $MXNETDIR -chkret git submodule update --init --recursive + +set +e +git submodule update --init --recursive +set -e # don't want to overwrite an existing config file cp make/config.mk ./config.mk diff --git a/scala-package/examples/pom.xml b/scala-package/examples/pom.xml index 436f29927..564102a9f 100644 --- a/scala-package/examples/pom.xml +++ b/scala-package/examples/pom.xml @@ -5,74 +5,17 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-examples_2.11 + mxnet-examples MXNet Scala Package - Examples - - - unittest - - true - - - - integrationtest - - false - - - - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - - release - - - - org.apache.maven.plugins - maven-source-plugin - - true - - - - org.apache.maven.plugins - maven-gpg-plugin - - true - - - - org.sonatype.plugins - nexus-staging-maven-plugin - - true - - - - - - + + true + @@ -136,7 +79,7 @@ ${skipTests} - -Djava.library.path=${project.parent.basedir}/native/${platform}/target \ + -Djava.library.path=${project.parent.basedir}/native/target \ -Dlog4j.configuration=file://${project.basedir}/src/test/resources/log4j.properties @@ -150,14 +93,14 @@ org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT + mxnet-core + INTERNAL provided org.apache.mxnet - mxnet-infer_${scala.binary.version} - 1.3.1-SNAPSHOT + mxnet-infer + INTERNAL provided diff --git a/scala-package/examples/scripts/benchmark/run_image_inference_bm.sh b/scala-package/examples/scripts/benchmark/run_image_inference_bm.sh index b647bdf8c..7bbf86ea1 100755 --- a/scala-package/examples/scripts/benchmark/run_image_inference_bm.sh +++ b/scala-package/examples/scripts/benchmark/run_image_inference_bm.sh @@ -19,23 +19,8 @@ set -e -echo $OSTYPE - -hw_type=cpu -if [[ $1 = gpu ]] -then - hw_type=gpu -fi - -platform=linux-x86_64 - -if [[ $OSTYPE = [darwin]* ]] -then - platform=osx-x86_64 -fi - MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/$platform-$hw_type/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* MODEL_NAME=$2 diff --git a/scala-package/examples/scripts/benchmark/run_java_inference_bm.sh b/scala-package/examples/scripts/benchmark/run_java_inference_bm.sh new file mode 100644 index 000000000..f426ddaaa --- /dev/null +++ b/scala-package/examples/scripts/benchmark/run_java_inference_bm.sh @@ -0,0 +1,27 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +set -e + +MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/* + +java -Xmx8G -Dmxnet.traceLeakedObjects=true -cp $CLASS_PATH \ + org.apache.mxnetexamples.javaapi.benchmark.JavaBenchmark $@ + diff --git a/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh b/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh index bf6b7b314..6b7c10758 100755 --- a/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh +++ b/scala-package/examples/scripts/benchmark/run_text_charrnn_bm.sh @@ -19,24 +19,8 @@ set -e -echo $OSTYPE - -hw_type=cpu -if [[ $1 = gpu ]] -then - hw_type=gpu -fi - -platform=linux-x86_64 - -if [[ $OSTYPE = [darwin]* ]] -then - platform=osx-x86_64 -fi - - MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/$platform-$hw_type/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* MODEL_NAME=$2 diff --git a/scala-package/examples/scripts/customop/run_customop.sh b/scala-package/examples/scripts/customop/run_customop.sh index 049df5874..1b5caf5e2 100755 --- a/scala-package/examples/scripts/customop/run_customop.sh +++ b/scala-package/examples/scripts/customop/run_customop.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use, -1 means cpu GPU=$1 diff --git a/scala-package/examples/scripts/customop/run_customopwithrtc.sh b/scala-package/examples/scripts/customop/run_customopwithrtc.sh index d44949d61..432eff562 100755 --- a/scala-package/examples/scripts/customop/run_customopwithrtc.sh +++ b/scala-package/examples/scripts/customop/run_customopwithrtc.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use GPU=0 diff --git a/scala-package/examples/scripts/infer/imageclassifier/get_resnet_18_data.sh b/scala-package/examples/scripts/infer/imageclassifier/get_resnet_18_data.sh index 1ce996e5c..b12993210 100755 --- a/scala-package/examples/scripts/infer/imageclassifier/get_resnet_18_data.sh +++ b/scala-package/examples/scripts/infer/imageclassifier/get_resnet_18_data.sh @@ -34,8 +34,8 @@ if [ ! -d "$image_path" ]; then fi if [ ! -f "$data_path" ]; then - wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -P $data_path - wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -P $data_path - wget https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -P $data_path - wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path + curl https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-symbol.json -o $data_path/resnet-18-symbol.json + curl https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/resnet-18-0000.params -o $data_path/resnet-18-0000.params + curl https://s3.us-east-2.amazonaws.com/scala-infer-models/resnet-18/synset.txt -o $data_path/synset.txt + curl https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -o $image_path/kitten.jpg fi diff --git a/scala-package/examples/scripts/infer/imageclassifier/get_resnet_data.sh b/scala-package/examples/scripts/infer/imageclassifier/get_resnet_data.sh index 6fd85e4f4..dc37bdbd7 100755 --- a/scala-package/examples/scripts/infer/imageclassifier/get_resnet_data.sh +++ b/scala-package/examples/scripts/infer/imageclassifier/get_resnet_data.sh @@ -34,8 +34,8 @@ if [ ! -d "$image_path" ]; then fi if [ ! -f "$data_path" ]; then - wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-0000.params -P $data_path - wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-symbol.json -P $data_path - wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/synset.txt -P $data_path - wget https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -P $image_path + curl https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-0000.params -o $data_path/resnet-152-0000.params + curl https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/resnet-152-symbol.json -o $data_path/resnet-152-symbol.json + curl https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/synset.txt -o $data_path/synset.txt + curl https://s3.us-east-2.amazonaws.com/mxnet-scala/scala-example-ci/resnet152/kitten.jpg -o $image_path/kitten.jpg fi diff --git a/scala-package/examples/scripts/infer/imageclassifier/run_classifier_example.sh b/scala-package/examples/scripts/infer/imageclassifier/run_classifier_example.sh index bd6509a82..aa021478b 100755 --- a/scala-package/examples/scripts/infer/imageclassifier/run_classifier_example.sh +++ b/scala-package/examples/scripts/infer/imageclassifier/run_classifier_example.sh @@ -20,7 +20,7 @@ set -e MXNET_ROOT=$(cd "$(dirname $0)/../../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-cpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* # model dir MODEL_PATH_PREFIX=$1 diff --git a/scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh b/scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh index 8787d6382..e901fc807 100755 --- a/scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh +++ b/scala-package/examples/scripts/infer/objectdetector/get_ssd_data.sh @@ -35,12 +35,11 @@ if [ ! -d "$image_path" ]; then fi if [ ! -f "$data_path" ]; then - wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -P $data_path - wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -P $data_path - wget https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -P $data_path - cd $image_path - wget https://cloud.githubusercontent.com/assets/3307514/20012566/cbb53c76-a27d-11e6-9aaa-91939c9a1cd5.jpg -O 000001.jpg - wget https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -O dog.jpg - wget https://cloud.githubusercontent.com/assets/3307514/20012563/cbb41382-a27d-11e6-92a9-18dab4fd1ad3.jpg -O person.jpg + curl https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json -o $data_path/resnet50_ssd_model-symbol.json + curl https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params -o $data_path/resnet50_ssd_model-0000.params + curl https://s3.amazonaws.com/model-server/models/resnet50_ssd/synset.txt -o $data_path/synset.txt + curl https://cloud.githubusercontent.com/assets/3307514/20012566/cbb53c76-a27d-11e6-9aaa-91939c9a1cd5.jpg -o $image_path/000001.jpg + curl https://cloud.githubusercontent.com/assets/3307514/20012567/cbb60336-a27d-11e6-93ff-cbc3f09f5c9e.jpg -o $image_path/dog.jpg + curl https://cloud.githubusercontent.com/assets/3307514/20012563/cbb41382-a27d-11e6-92a9-18dab4fd1ad3.jpg -o $image_path/person.jpg fi diff --git a/scala-package/examples/scripts/infer/objectdetector/run_ssd_example.sh b/scala-package/examples/scripts/infer/objectdetector/run_ssd_example.sh index 8cea892b5..9f290416b 100755 --- a/scala-package/examples/scripts/infer/objectdetector/run_ssd_example.sh +++ b/scala-package/examples/scripts/infer/objectdetector/run_ssd_example.sh @@ -17,9 +17,10 @@ # specific language governing permissions and limitations # under the License. +set -x MXNET_ROOT=$(cd "$(dirname $0)/../../../../../"; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-cpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # model dir and prefix MODEL_DIR=$1 diff --git a/scala-package/examples/scripts/infer/objectdetector/run_ssd_java_example.sh b/scala-package/examples/scripts/infer/objectdetector/run_ssd_java_example.sh new file mode 100755 index 000000000..07328b50d --- /dev/null +++ b/scala-package/examples/scripts/infer/objectdetector/run_ssd_java_example.sh @@ -0,0 +1,34 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +MXNET_ROOT=$(cd "$(dirname $0)/../../../../../"; pwd) +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/*:$MXNET_ROOT/scala-package/examples/src/main/scala/org/apache/mxnetexamples/api/java/infer/imageclassifier/* + +# model dir and prefix +MODEL_DIR=$1 +# input image +INPUT_IMG=$2 +# which input image dir +INPUT_DIR=$3 + +java -Xmx8G -cp $CLASS_PATH \ + org.apache.mxnetexamples.javaapi.infer.objectdetector.SSDClassifierExample \ + --model-path-prefix $MODEL_DIR \ + --input-image $INPUT_IMG \ + --input-dir $INPUT_DIR diff --git a/scala-package/examples/scripts/infer/predictor/run_predictor_java_example.sh b/scala-package/examples/scripts/infer/predictor/run_predictor_java_example.sh new file mode 100755 index 000000000..8e8752ee8 --- /dev/null +++ b/scala-package/examples/scripts/infer/predictor/run_predictor_java_example.sh @@ -0,0 +1,31 @@ +#!/bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +MXNET_ROOT=$(cd "$(dirname $0)/../../../../../"; pwd) +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* + +# model dir and prefix +MODEL_DIR=$1 +# input image +INPUT_IMG=$2 + +java -Xmx8G -cp $CLASS_PATH \ + org.apache.mxnetexamples.javaapi.infer.predictor.PredictorExample \ + --model-path-prefix $MODEL_DIR \ + --input-image $INPUT_IMG diff --git a/scala-package/examples/scripts/module/mnist_mlp.sh b/scala-package/examples/scripts/module/mnist_mlp.sh index 29306706d..907552a45 100755 --- a/scala-package/examples/scripts/module/mnist_mlp.sh +++ b/scala-package/examples/scripts/module/mnist_mlp.sh @@ -18,7 +18,7 @@ # under the License. ROOT_DIR=$(cd `dirname $0`/../../..; pwd) -CLASSPATH=$ROOT_DIR/assembly/osx-x86_64-cpu/target/*:$ROOT_DIR/examples/target/*:$ROOT_DIR/examples/target/classes/lib/* +CLASSPATH=$ROOT_DIR/assembly/target/*:$ROOT_DIR/examples/target/*:$ROOT_DIR/examples/target/classes/lib/* mkdir -p model java -Xmx4G -cp $CLASSPATH \ diff --git a/scala-package/examples/scripts/module/run_sequential_module.sh b/scala-package/examples/scripts/module/run_sequential_module.sh index 9e1f30e36..c7507850d 100755 --- a/scala-package/examples/scripts/module/run_sequential_module.sh +++ b/scala-package/examples/scripts/module/run_sequential_module.sh @@ -18,7 +18,7 @@ # under the License. ROOT_DIR=$(cd `dirname $0`/../../..; pwd) -CLASSPATH=$ROOT_DIR/assembly/linux-x86_64-cpu/target/*:$ROOT_DIR/examples/target/*:$ROOT_DIR/examples/target/classes/lib/* +CLASSPATH=$ROOT_DIR/assembly/target/*:$ROOT_DIR/examples/target/*:$ROOT_DIR/examples/target/classes/lib/* DATA_DIR=$ROOT_DIR/core/data diff --git a/scala-package/examples/scripts/neuralstyle_end2end/run_test_end2end.sh b/scala-package/examples/scripts/neuralstyle_end2end/run_test_end2end.sh index 0305ba47a..a040ba7ca 100755 --- a/scala-package/examples/scripts/neuralstyle_end2end/run_test_end2end.sh +++ b/scala-package/examples/scripts/neuralstyle_end2end/run_test_end2end.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* INPUT_IMG=$1 MODEL_DIR=$2 diff --git a/scala-package/examples/scripts/neuralstyle_end2end/run_train_end2end.sh b/scala-package/examples/scripts/neuralstyle_end2end/run_train_end2end.sh index cde290f7e..d63be1d40 100755 --- a/scala-package/examples/scripts/neuralstyle_end2end/run_train_end2end.sh +++ b/scala-package/examples/scripts/neuralstyle_end2end/run_train_end2end.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # more details please refer to # https://github.com/Ldpe2G/mxnet/blob/develop/example/neural-style/end_to_end/README.md diff --git a/scala-package/examples/scripts/profiler/run_profiler_matmul.sh b/scala-package/examples/scripts/profiler/run_profiler_matmul.sh index 824578b96..2f7828573 100755 --- a/scala-package/examples/scripts/profiler/run_profiler_matmul.sh +++ b/scala-package/examples/scripts/profiler/run_profiler_matmul.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use, -1 means cpu GPU=0 diff --git a/scala-package/examples/scripts/profiler/run_profiler_ndarray.sh b/scala-package/examples/scripts/profiler/run_profiler_ndarray.sh index 18f290e52..3b8cf22a5 100755 --- a/scala-package/examples/scripts/profiler/run_profiler_ndarray.sh +++ b/scala-package/examples/scripts/profiler/run_profiler_ndarray.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* MODE="all" diff --git a/scala-package/examples/scripts/rnn/run_lstm_bucketing.sh b/scala-package/examples/scripts/rnn/run_lstm_bucketing.sh index 33dfce4cc..508dcac80 100755 --- a/scala-package/examples/scripts/rnn/run_lstm_bucketing.sh +++ b/scala-package/examples/scripts/rnn/run_lstm_bucketing.sh @@ -19,12 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -OS=$(uname) -if [ "$OS" = "Darwin" ]; then - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -else - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -fi +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* DATA_TRAIN=$1 DATA_VAL=$2 diff --git a/scala-package/examples/scripts/rnn/run_test_charrnn.sh b/scala-package/examples/scripts/rnn/run_test_charrnn.sh index bf02a1d60..df1fac366 100755 --- a/scala-package/examples/scripts/rnn/run_test_charrnn.sh +++ b/scala-package/examples/scripts/rnn/run_test_charrnn.sh @@ -19,15 +19,10 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -OS=$(uname) -if [ "$OS" = "Darwin" ]; then - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -else - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -fi +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # you can get the training data file using the following command -# wget http://data.mxnet.io/data/char_lstm.zip +# curl -O http://data.mxnet.io/data/char_lstm.zip # unzip -o char_lstm.zip # for example ./datas/obama.txt DATA_PATH=$1 diff --git a/scala-package/examples/scripts/rnn/run_train_charrnn.sh b/scala-package/examples/scripts/rnn/run_train_charrnn.sh index 4877f5785..f8c9a3534 100755 --- a/scala-package/examples/scripts/rnn/run_train_charrnn.sh +++ b/scala-package/examples/scripts/rnn/run_train_charrnn.sh @@ -19,16 +19,12 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../../.."; pwd) -OS=$(uname) -if [ "$OS" = "Darwin" ]; then - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/osx-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -else - CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* -fi +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* + # which gpu card to use, -1 means cpu GPU=$1 # you can get the training data file using the following command -# wget http://data.mxnet.io/data/char_lstm.zip +# curl -O http://data.mxnet.io/data/char_lstm.zip # unzip -o char_lstm.zip # for example ./datas/obama.txt DATA_PATH=$2 diff --git a/scala-package/examples/scripts/run_cnntextclassification.sh b/scala-package/examples/scripts/run_cnntextclassification.sh index 88fae7cf5..1e9b46540 100755 --- a/scala-package/examples/scripts/run_cnntextclassification.sh +++ b/scala-package/examples/scripts/run_cnntextclassification.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use, -1 means cpu GPU=$1 diff --git a/scala-package/examples/scripts/run_gan_mnist.sh b/scala-package/examples/scripts/run_gan_mnist.sh index f7c26a79a..539adfdc9 100755 --- a/scala-package/examples/scripts/run_gan_mnist.sh +++ b/scala-package/examples/scripts/run_gan_mnist.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use, -1 means cpu GPU=$1 diff --git a/scala-package/examples/scripts/run_multitask.sh b/scala-package/examples/scripts/run_multitask.sh index 62047528b..abf272fbe 100755 --- a/scala-package/examples/scripts/run_multitask.sh +++ b/scala-package/examples/scripts/run_multitask.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # which gpu card to use, -1 means cpu GPU=$1 diff --git a/scala-package/examples/scripts/run_neuralstyle.sh b/scala-package/examples/scripts/run_neuralstyle.sh index 238c37127..fb4824376 100755 --- a/scala-package/examples/scripts/run_neuralstyle.sh +++ b/scala-package/examples/scripts/run_neuralstyle.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-gpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* INPUT_IMG=$1 STYLE_IMG=$2 MODEL_PATH=$MXNET_ROOT/example/neural-style/model/vgg19.params diff --git a/scala-package/examples/scripts/run_train_mnist.sh b/scala-package/examples/scripts/run_train_mnist.sh index ea53c1ade..418065e77 100755 --- a/scala-package/examples/scripts/run_train_mnist.sh +++ b/scala-package/examples/scripts/run_train_mnist.sh @@ -21,13 +21,15 @@ set -e MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) echo $MXNET_ROOT -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-cpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/*:$MXNET_ROOT/scala-package/infer/target/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # model dir DATA_PATH=$2 -java -XX:+PrintGC -Xms256M -Xmx512M -Dmxnet.traceLeakedObjects=false -cp $CLASS_PATH \ - org.apache.mxnetexamples.imclassification.TrainMnist \ - --data-dir /home/ubuntu/mxnet_scala/scala-package/examples/mnist/ \ +java -XX:+PrintGC -Dmxnet.traceLeakedObjects=false -cp $CLASS_PATH \ + org.apache.mxnetexamples.imclassification.TrainModel \ + --data-dir $MXNET_ROOT/scala-package/examples/mnist/ \ + --network mlp \ + --num-layers 50 \ --num-epochs 10000000 \ --batch-size 1024 \ No newline at end of file diff --git a/scala-package/examples/scripts/run_visualization.sh b/scala-package/examples/scripts/run_visualization.sh index 3e8f5cd78..7f5b94fbe 100755 --- a/scala-package/examples/scripts/run_visualization.sh +++ b/scala-package/examples/scripts/run_visualization.sh @@ -19,7 +19,7 @@ MXNET_ROOT=$(cd "$(dirname $0)/../../.."; pwd) -CLASS_PATH=$MXNET_ROOT/scala-package/assembly/linux-x86_64-cpu/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* +CLASS_PATH=$MXNET_ROOT/scala-package/assembly/assembly/target/*:$MXNET_ROOT/scala-package/examples/target/*:$MXNET_ROOT/scala-package/examples/target/classes/lib/* # please install the Graphviz library # if you are using ubuntu, use the following command: diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/InferBase.java b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/InferBase.java new file mode 100644 index 000000000..fdcde6b41 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/InferBase.java @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.mxnetexamples.javaapi.benchmark; + +import org.apache.mxnet.javaapi.Context; +import org.kohsuke.args4j.Option; + +import java.util.List; + +abstract class InferBase { + @Option(name = "--num-runs", usage = "Number of runs") + public int numRun = 1; + @Option(name = "--model-name", usage = "Name of the model") + public String modelName = ""; + @Option(name = "--batchsize", usage = "Size of the batch") + public int batchSize = 1; + + public abstract void preProcessModel(List context); + public abstract void runSingleInference(); + public abstract void runBatchInference(); +} diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/JavaBenchmark.java b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/JavaBenchmark.java new file mode 100644 index 000000000..4a6bb2dd3 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/JavaBenchmark.java @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.javaapi.benchmark; + +import org.apache.mxnet.javaapi.Context; +import org.kohsuke.args4j.CmdLineParser; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +public class JavaBenchmark { + + private static boolean runBatch = false; + + private static void parse(Object inst, String[] args) { + CmdLineParser parser = new CmdLineParser(inst); + try { + parser.parseArgument(args); + } catch (Exception e) { + System.err.println(e.getMessage() + e); + parser.printUsage(System.err); + System.exit(1); + } + } + + private static long percentile(int p, long[] seq) { + Arrays.sort(seq); + int k = (int) Math.ceil((seq.length - 1) * (p / 100.0)); + return seq[k]; + } + + private static void printStatistics(long[] inferenceTimesRaw, String metricsPrefix) { + long[] inferenceTimes = inferenceTimesRaw; + // remove head and tail + if (inferenceTimes.length > 2) { + inferenceTimes = Arrays.copyOfRange(inferenceTimesRaw, + 1, inferenceTimesRaw.length - 1); + } + double p50 = percentile(50, inferenceTimes) / 1.0e6; + double p99 = percentile(99, inferenceTimes) / 1.0e6; + double p90 = percentile(90, inferenceTimes) / 1.0e6; + long sum = 0; + for (long time: inferenceTimes) sum += time; + double average = sum / (inferenceTimes.length * 1.0e6); + + System.out.println( + String.format("\n%s_p99 %fms\n%s_p90 %fms\n%s_p50 %fms\n%s_average %1.2fms", + metricsPrefix, p99, metricsPrefix, p90, + metricsPrefix, p50, metricsPrefix, average) + ); + + } + + private static List bindToDevice() { + List context = new ArrayList(); + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + Integer.valueOf(System.getenv("SCALA_TEST_ON_GPU")) == 1) { + context.add(Context.gpu()); + } else { + context.add(Context.cpu()); + } + return context; + } + + public static void main(String[] args) { + if (args.length < 2) { + StringBuilder sb = new StringBuilder(); + sb.append("Please follow the format:"); + sb.append("\n --model-name "); + sb.append("\n --num-runs "); + sb.append("\n --batchsize "); + System.out.println(sb.toString()); + return; + } + String modelName = args[1]; + InferBase model = null; + switch(modelName) { + case "ObjectDetection": + runBatch = true; + ObjectDetectionBenchmark inst = new ObjectDetectionBenchmark(); + parse(inst, args); + model = inst; + break; + default: + System.err.println("Model name not found! " + modelName); + System.exit(1); + } + List context = bindToDevice(); + long[] result = new long[model.numRun]; + model.preProcessModel(context); + if (runBatch) { + for (int i =0;i < model.numRun; i++) { + long currTime = System.nanoTime(); + model.runBatchInference(); + result[i] = System.nanoTime() - currTime; + } + System.out.println("Batchsize: " + model.batchSize); + System.out.println("Num of runs: " + model.numRun); + printStatistics(result, modelName +"batch_inference"); + } + + model.batchSize = 1; + model.preProcessModel(context); + result = new long[model.numRun]; + for (int i = 0; i < model.numRun; i++) { + long currTime = System.nanoTime(); + model.runSingleInference(); + result[i] = System.nanoTime() - currTime; + } + System.out.println("Num of runs: " + model.numRun); + printStatistics(result, modelName + "single_inference"); + } +} diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/ObjectDetectionBenchmark.java b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/ObjectDetectionBenchmark.java new file mode 100644 index 000000000..257ea3241 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/benchmark/ObjectDetectionBenchmark.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.javaapi.benchmark; + +import org.apache.mxnet.infer.javaapi.ObjectDetector; +import org.apache.mxnet.javaapi.*; +import org.kohsuke.args4j.Option; + +import java.util.ArrayList; +import java.util.List; + +class ObjectDetectionBenchmark extends InferBase { + @Option(name = "--model-path-prefix", usage = "input model directory and prefix of the model") + public String modelPathPrefix = "/model/ssd_resnet50_512"; + @Option(name = "--input-image", usage = "the input image") + public String inputImagePath = "/images/dog.jpg"; + + private ObjectDetector objDet; + private NDArray img; + private NDArray$ NDArray = NDArray$.MODULE$; + + public void preProcessModel(List context) { + Shape inputShape = new Shape(new int[] {this.batchSize, 3, 512, 512}); + List inputDescriptors = new ArrayList<>(); + inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); + img = ObjectDetector.bufferedImageToPixels( + ObjectDetector.reshapeImage( + ObjectDetector.loadImageFromFile(inputImagePath), 512, 512 + ), + new Shape(new int[] {1, 3, 512, 512}) + ); + } + + public void runSingleInference() { + List nd = new ArrayList<>(); + nd.add(img); + objDet.objectDetectWithNDArray(nd, 3); + } + + public void runBatchInference() { + List nd = new ArrayList<>(); + NDArray[] temp = new NDArray[batchSize]; + for (int i = 0; i < batchSize; i++) temp[i] = img.copy(); + NDArray batched = NDArray.concat(temp, batchSize, 0, null)[0]; + nd.add(batched); + objDet.objectDetectWithNDArray(nd, 3); + } +} diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/README.md b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/README.md new file mode 100644 index 000000000..55741024d --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/README.md @@ -0,0 +1,97 @@ +# Single Shot Multi Object Detection using Java Inference API + +In this example, you will learn how to use Java Inference API to run Inference on pre-trained Single Shot Multi Object Detection (SSD) MXNet model. + +The model is trained on the [Pascal VOC 2012 dataset](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html). The network is a SSD model built on Resnet50 as base network to extract image features. The model is trained to detect the following entities (classes): ['aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']. For more details about the model, you can refer to the [MXNet SSD example](https://github.com/apache/incubator-mxnet/tree/master/example/ssd). + + +## Contents + +1. [Prerequisites](#prerequisites) +2. [Download artifacts](#download-artifacts) +3. [Setup datapath and parameters](#setup-datapath-and-parameters) +4. [Run the image inference example](#run-the-image-inference-example) +5. [Infer APIs](#infer-api-details) +6. [Next steps](#next-steps) + + +## Prerequisites + +1. [Build MXNet](http://mxnet.incubator.apache.org/install/scala_setup.html) +2. [Build MXNet Scala/Java Package](http://mxnet.incubator.apache.org/install/scala_setup.html) +3. [IntelliJ IDE (or alternative IDE) project setup](http://mxnet.incubator.apache.org/tutorials/java/mxnet_java_on_intellij.html) with the MXNet Scala/Java Package +4. wget + + +## Setup Guide + +### Download Artifacts +#### Step 1 +You can download the files using the script `get_ssd_data.sh`. It will download and place the model files in a `model` folder and the test image files in a `image` folder in the current directory. +From the `scala-package/examples/scripts/infer/objectdetector/` folder run: + +```bash +./get_ssd_data.sh +``` + +**Note**: You may need to run `chmod +x get_ssd_data.sh` before running this script. + +In the pre-trained model, the `input_name` is `data` and shape is `(1, 3, 512, 512)`. +This shape translates to: a batch of `1` image, the image has color and uses `3` channels (RGB), and the image has the dimensions of `512` pixels in height by `512` pixels in width. + +`image/jpeg` is the expected input type, since this example's image pre-processor only supports the handling of binary JPEG images. + +The output shape is `(1, 6132, 6)`. As with the input, the `1` is the number of images. `6132` is the number of prediction results, and `6` is for the size of each prediction. Each prediction contains the following components: +- `Class` +- `Accuracy` +- `Xmin` +- `Ymin` +- `Xmax` +- `Ymax` + + +### Setup Datapath and Parameters +#### Step 2 +The followings is the parameters defined for this example, you can find more information in the `class SSDClassifierExample`. + +| Argument | Comments | +| ----------------------------- | ---------------------------------------- | +| `model-path-prefix` | Folder path with prefix to the model (including json, params, and any synset file). | +| `input-image` | The image to run inference on. | +| `input-dir` | The directory of images to run inference on. | + + +## How to Run Inference +After the previous steps, you should be able to run the code using the following script that will pass all of the required parameters to the Infer API. + +From the `scala-package/examples/scripts/infer/objectdetector/` folder run: + +```bash +./run_ssd_example.sh ../models/resnet50_ssd/resnet50_ssd_model ../images/dog.jpg ../images +``` + +**Notes**: +* These are relative paths to this script. +* You may need to run `chmod +x run_ssd_example.sh` before running this script. + +The example should give expected output as shown below: +``` +Class: car +Probabilties: 0.99847263 +(Coord:,312.21335,72.0291,456.01443,150.66176) +Class: bicycle +Probabilties: 0.90473825 +(Coord:,155.95807,149.96362,383.8369,418.94513) +Class: dog +Probabilties: 0.8226818 +(Coord:,83.82353,179.13998,206.63783,476.7875) +``` +the outputs come from the the input image, with top3 predictions picked. + + +## Infer API Details +This example uses ObjectDetector class provided by MXNet's Java Infer APIs. It provides methods to load the images, create NDArray out of Java BufferedImage and run prediction using Classifier and Predictor APIs. + + +## References +This documentation used the model and inference setup guide from the [MXNet Model Server SSD example](https://github.com/awslabs/mxnet-model-server/blob/master/examples/ssd/README.md). diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/SSDClassifierExample.java b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/SSDClassifierExample.java new file mode 100644 index 000000000..a9c00f7f1 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/objectdetector/SSDClassifierExample.java @@ -0,0 +1,199 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.javaapi.infer.objectdetector; + +import org.apache.mxnet.infer.javaapi.ObjectDetectorOutput; +import org.kohsuke.args4j.CmdLineParser; +import org.kohsuke.args4j.Option; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import org.apache.mxnet.javaapi.*; +import org.apache.mxnet.infer.javaapi.ObjectDetector; + +// scalastyle:off +import java.awt.image.BufferedImage; +// scalastyle:on + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +import java.io.File; + +public class SSDClassifierExample { + @Option(name = "--model-path-prefix", usage = "input model directory and prefix of the model") + private String modelPathPrefix = "/model/ssd_resnet50_512"; + @Option(name = "--input-image", usage = "the input image") + private String inputImagePath = "/images/dog.jpg"; + @Option(name = "--input-dir", usage = "the input batch of images directory") + private String inputImageDir = "/images/"; + + final static Logger logger = LoggerFactory.getLogger(SSDClassifierExample.class); + + static List> + runObjectDetectionSingle(String modelPathPrefix, String inputImagePath, List context) { + Shape inputShape = new Shape(new int[]{1, 3, 512, 512}); + List inputDescriptors = new ArrayList(); + inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + BufferedImage img = ObjectDetector.loadImageFromFile(inputImagePath); + ObjectDetector objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); + return objDet.imageObjectDetect(img, 3); + } + + static List>> + runObjectDetectionBatch(String modelPathPrefix, String inputImageDir, List context) { + Shape inputShape = new Shape(new int[]{1, 3, 512, 512}); + List inputDescriptors = new ArrayList(); + inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + ObjectDetector objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); + + // Loading batch of images from the directory path + List> batchFiles = generateBatches(inputImageDir, 20); + List>> outputList + = new ArrayList>>(); + + for (List batchFile : batchFiles) { + List imgList = ObjectDetector.loadInputBatch(batchFile); + // Running inference on batch of images loaded in previous step + List> tmp + = objDet.imageBatchObjectDetect(imgList, 5); + outputList.add(tmp); + } + return outputList; + } + + static List> generateBatches(String inputImageDirPath, int batchSize) { + File dir = new File(inputImageDirPath); + + List> output = new ArrayList>(); + List batch = new ArrayList(); + for (File imgFile : dir.listFiles()) { + batch.add(imgFile.getPath()); + if (batch.size() == batchSize) { + output.add(batch); + batch = new ArrayList(); + } + } + if (batch.size() > 0) { + output.add(batch); + } + return output; + } + + public static void main(String[] args) { + SSDClassifierExample inst = new SSDClassifierExample(); + CmdLineParser parser = new CmdLineParser(inst); + try { + parser.parseArgument(args); + } catch (Exception e) { + logger.error(e.getMessage(), e); + parser.printUsage(System.err); + System.exit(1); + } + + String mdprefixDir = inst.modelPathPrefix; + String imgPath = inst.inputImagePath; + String imgDir = inst.inputImageDir; + + if (!checkExist(Arrays.asList(mdprefixDir + "-symbol.json", imgDir, imgPath))) { + logger.error("Model or input image path does not exist"); + System.exit(1); + } + + List context = new ArrayList(); + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + Integer.valueOf(System.getenv("SCALA_TEST_ON_GPU")) == 1) { + context.add(Context.gpu()); + } else { + context.add(Context.cpu()); + } + + try { + Shape inputShape = new Shape(new int[]{1, 3, 512, 512}); + Shape outputShape = new Shape(new int[]{1, 6132, 6}); + + + int width = inputShape.get(2); + int height = inputShape.get(3); + StringBuilder outputStr = new StringBuilder().append("\n"); + + List> output + = runObjectDetectionSingle(mdprefixDir, imgPath, context); + + for (List ele : output) { + for (ObjectDetectorOutput i : ele) { + outputStr.append("Class: " + i.getClassName() + "\n"); + outputStr.append("Probabilties: " + i.getProbability() + "\n"); + + List coord = Arrays.asList(i.getXMin() * width, + i.getXMax() * height, i.getYMin() * width, i.getYMax() * height); + StringBuilder sb = new StringBuilder(); + for (float c : coord) { + sb.append(", ").append(c); + } + outputStr.append("Coord:" + sb.substring(2) + "\n"); + } + } + logger.info(outputStr.toString()); + + List>> outputList = + runObjectDetectionBatch(mdprefixDir, imgDir, context); + + outputStr = new StringBuilder().append("\n"); + int index = 0; + for (List> i : outputList) { + for (List j : i) { + outputStr.append("*** Image " + (index + 1) + "***" + "\n"); + for (ObjectDetectorOutput k : j) { + outputStr.append("Class: " + k.getClassName() + "\n"); + outputStr.append("Probabilties: " + k.getProbability() + "\n"); + List coord = Arrays.asList(k.getXMin() * width, + k.getXMax() * height, k.getYMin() * width, k.getYMax() * height); + + StringBuilder sb = new StringBuilder(); + for (float c : coord) { + sb.append(", ").append(c); + } + outputStr.append("Coord:" + sb.substring(2) + "\n"); + } + index++; + } + } + logger.info(outputStr.toString()); + + } catch (Exception e) { + logger.error(e.getMessage(), e); + parser.printUsage(System.err); + System.exit(1); + } + System.exit(0); + } + + static Boolean checkExist(List arr) { + Boolean exist = true; + for (String item : arr) { + if (!(new File(item).exists())) { + logger.error("Cannot find: " + item); + exist = false; + } + } + return exist; + } +} diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/PredictorExample.java b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/PredictorExample.java new file mode 100644 index 000000000..c9b4426f5 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/PredictorExample.java @@ -0,0 +1,200 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.javaapi.infer.predictor; + +import org.apache.mxnet.infer.javaapi.Predictor; +import org.apache.mxnet.javaapi.*; +import org.kohsuke.args4j.CmdLineParser; +import org.kohsuke.args4j.Option; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.imageio.ImageIO; +import java.awt.Graphics2D; +import java.awt.image.BufferedImage; +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +/** + * This Class is a demo to show how users can use Predictor APIs to do + * Image Classification with all hand-crafted Pre-processing. + * All helper functions for image pre-processing are + * currently available in ObjectDetector class. + */ +public class PredictorExample { + @Option(name = "--model-path-prefix", usage = "input model directory and prefix of the model") + private String modelPathPrefix = "/model/ssd_resnet50_512"; + @Option(name = "--input-image", usage = "the input image") + private String inputImagePath = "/images/dog.jpg"; + + final static Logger logger = LoggerFactory.getLogger(PredictorExample.class); + + /** + * Load the image from file to buffered image + * It can be replaced by loadImageFromFile from ObjectDetector + * @param inputImagePath input image Path in String + * @return Buffered image + */ + private static BufferedImage loadIamgeFromFile(String inputImagePath) { + BufferedImage buf = null; + try { + buf = ImageIO.read(new File(inputImagePath)); + } catch (IOException e) { + System.err.println(e); + } + return buf; + } + + /** + * Reshape the current image using ImageIO and Graph2D + * It can be replaced by reshapeImage from ObjectDetector + * @param buf Buffered image + * @param newWidth desired width + * @param newHeight desired height + * @return a reshaped bufferedImage + */ + private static BufferedImage reshapeImage(BufferedImage buf, int newWidth, int newHeight) { + BufferedImage resizedImage = new BufferedImage(newWidth, newHeight, BufferedImage.TYPE_INT_RGB); + Graphics2D g = resizedImage.createGraphics(); + g.drawImage(buf, 0, 0, newWidth, newHeight, null); + g.dispose(); + return resizedImage; + } + + /** + * Convert an image from a buffered image into pixels float array + * It can be replaced by bufferedImageToPixels from ObjectDetector + * @param buf buffered image + * @return Float array + */ + private static float[] imagePreprocess(BufferedImage buf) { + // Get height and width of the image + int w = buf.getWidth(); + int h = buf.getHeight(); + + // get an array of integer pixels in the default RGB color mode + int[] pixels = buf.getRGB(0, 0, w, h, null, 0, w); + + // 3 times height and width for R,G,B channels + float[] result = new float[3 * h * w]; + + int row = 0; + // copy pixels to array vertically + while (row < h) { + int col = 0; + // copy pixels to array horizontally + while (col < w) { + int rgb = pixels[row * w + col]; + // getting red color + result[0 * h * w + row * w + col] = (rgb >> 16) & 0xFF; + // getting green color + result[1 * h * w + row * w + col] = (rgb >> 8) & 0xFF; + // getting blue color + result[2 * h * w + row * w + col] = rgb & 0xFF; + col += 1; + } + row += 1; + } + buf.flush(); + return result; + } + + /** + * Helper class to print the maximum prediction result + * @param probabilities The float array of probability + * @param modelPathPrefix model Path needs to load the synset.txt + */ + private static String printMaximumClass(float[] probabilities, + String modelPathPrefix) throws IOException { + String synsetFilePath = modelPathPrefix.substring(0, + 1 + modelPathPrefix.lastIndexOf(File.separator)) + "/synset.txt"; + BufferedReader reader = new BufferedReader(new FileReader(synsetFilePath)); + ArrayList list = new ArrayList<>(); + String line = reader.readLine(); + + while (line != null){ + list.add(line); + line = reader.readLine(); + } + reader.close(); + + int maxIdx = 0; + for (int i = 1;i probabilities[maxIdx]) { + maxIdx = i; + } + } + + return "Probability : " + probabilities[maxIdx] + " Class : " + list.get(maxIdx) ; + } + + public static void main(String[] args) { + PredictorExample inst = new PredictorExample(); + CmdLineParser parser = new CmdLineParser(inst); + try { + parser.parseArgument(args); + } catch (Exception e) { + logger.error(e.getMessage(), e); + parser.printUsage(System.err); + System.exit(1); + } + // Prepare the model + List context = new ArrayList(); + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + Integer.valueOf(System.getenv("SCALA_TEST_ON_GPU")) == 1) { + context.add(Context.gpu()); + } else { + context.add(Context.cpu()); + } + List inputDesc = new ArrayList<>(); + Shape inputShape = new Shape(new int[]{1, 3, 224, 224}); + inputDesc.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + Predictor predictor = new Predictor(inst.modelPathPrefix, inputDesc, context,0); + // Prepare data + BufferedImage img = loadIamgeFromFile(inst.inputImagePath); + + img = reshapeImage(img, 224, 224); + // predict + float[][] result = predictor.predict(new float[][]{imagePreprocess(img)}); + try { + System.out.println("Predict with Float input"); + System.out.println(printMaximumClass(result[0], inst.modelPathPrefix)); + } catch (IOException e) { + System.err.println(e); + } + // predict with NDArray + NDArray nd = new NDArray( + imagePreprocess(img), + new Shape(new int[]{1, 3, 224, 224}), + Context.cpu()); + List ndList = new ArrayList<>(); + ndList.add(nd); + List ndResult = predictor.predictWithNDArray(ndList); + try { + System.out.println("Predict with NDArray"); + System.out.println(printMaximumClass(ndResult.get(0).toArray(), inst.modelPathPrefix)); + } catch (IOException e) { + System.err.println(e); + } + } + +} diff --git a/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/README.md b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/README.md new file mode 100644 index 000000000..1f2c9e0e8 --- /dev/null +++ b/scala-package/examples/src/main/java/org/apache/mxnetexamples/javaapi/infer/predictor/README.md @@ -0,0 +1,61 @@ +# Image Classification using Java Predictor + +In this example, you will learn how to use Java Inference API to +build and run pre-trained Resnet 18 model. + +## Contents + +1. [Prerequisites](#prerequisites) +2. [Download artifacts](#download-artifacts) +3. [Setup datapath and parameters](#setup-datapath-and-parameters) +4. [Run the image classifier example](#run-the-image-inference-example) + +## Prerequisites + +1. Build from source with [MXNet](https://mxnet.incubator.apache.org/install/index.html) +2. [IntelliJ IDE (or alternative IDE) project setup](https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/java/mxnet_java_on_intellij.md) with the MXNet Java Package +3. wget + +## Download Artifacts + +For this tutorial, you can get the model and sample input image by running following bash file. This script will use `wget` to download these artifacts from AWS S3. + +From the `scala-package/examples/scripts/infer/imageclassifier/` folder run: + +```bash +./get_resnet_18_data.sh +``` + +**Note**: You may need to run `chmod +x get_resnet_18_data.sh` before running this script. + +### Setup Datapath and Parameters + +The available arguments are as follows: + +| Argument | Comments | +| ----------------------------- | ---------------------------------------- | +| `model-dir`                   | Folder path with prefix to the model (including json, params, and any synset file). | +| `input-image` | The image to run inference on. | + +## Run the image classifier example + +After the previous steps, you should be able to run the code using the following script that will pass all of the required parameters to the Predictor API. + +From the `scala-package/examples/scripts/infer/predictor/` folder run: + +```bash +bash run_predictor_java_example.sh ../models/resnet-18/resnet-18 ../images/kitten.jpg +``` + +**Notes**: +* These are relative paths to this script. +* You may need to run `chmod +x run_predictor_java_example.sh` before running this script. + +The example should give an output similar to the one shown below: +``` +Predict with Float input +Probability : 0.30337515 Class : n02123159 tiger cat +Predict with NDArray +Probability : 0.30337515 Class : n02123159 tiger cat +``` +the outputs come from the the input image, with top1 predictions picked. \ No newline at end of file diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/TrainModel.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/TrainModel.scala index f6c283c3d..9f0430eaa 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/TrainModel.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/TrainModel.scala @@ -19,6 +19,7 @@ package org.apache.mxnetexamples.imclassification import java.util.concurrent._ +import org.apache.mxnet.DType.DType import org.apache.mxnetexamples.imclassification.models._ import org.apache.mxnetexamples.imclassification.util.Trainer import org.apache.mxnet._ @@ -42,12 +43,13 @@ object TrainModel { * @return The final validation accuracy */ def test(model: String, dataPath: String, numExamples: Int = 60000, - numEpochs: Int = 10, benchmark: Boolean = false): Float = { + numEpochs: Int = 10, benchmark: Boolean = false, + dtype: DType = DType.Float32): Float = { ResourceScope.using() { val devs = Array(Context.cpu(0)) val envs: mutable.Map[String, String] = mutable.HashMap.empty[String, String] val (dataLoader, net) = dataLoaderAndModel("mnist", model, dataPath, - numExamples = numExamples, benchmark = benchmark) + numExamples = numExamples, benchmark = benchmark, dtype = dtype) val Acc = Trainer.fit(batchSize = 128, numExamples, devs = devs, network = net, dataLoader = dataLoader, kvStore = "local", numEpochs = numEpochs) @@ -69,7 +71,7 @@ object TrainModel { */ def dataLoaderAndModel(dataset: String, model: String, dataDir: String = "", numLayers: Int = 50, numExamples: Int = 60000, - benchmark: Boolean = false + benchmark: Boolean = false, dtype: DType = DType.Float32 ): ((Int, KVStore) => (DataIter, DataIter), Symbol) = { val (imageShape, numClasses) = dataset match { case "mnist" => (List(1, 28, 28), 10) @@ -80,16 +82,17 @@ object TrainModel { val List(channels, height, width) = imageShape val dataSize: Int = channels * height * width val (datumShape, net) = model match { - case "mlp" => (List(dataSize), MultiLayerPerceptron.getSymbol(numClasses)) - case "lenet" => (List(channels, height, width), Lenet.getSymbol(numClasses)) + case "mlp" => (List(dataSize), MultiLayerPerceptron.getSymbol(numClasses, dtype = dtype)) + case "lenet" => (List(channels, height, width), Lenet.getSymbol(numClasses, dtype = dtype)) case "resnet" => (List(channels, height, width), Resnet.getSymbol(numClasses, - numLayers, imageShape)) + numLayers, imageShape, dtype = dtype)) case _ => throw new Exception("Invalid model name") } val dataLoader: (Int, KVStore) => (DataIter, DataIter) = if (benchmark) { (batchSize: Int, kv: KVStore) => { - val iter = new SyntheticDataIter(numClasses, batchSize, datumShape, List(), numExamples) + val iter = new SyntheticDataIter(numClasses, batchSize, datumShape, List(), numExamples, + dtype) (iter, iter) } } else { @@ -116,8 +119,10 @@ object TrainModel { val dataPath = if (inst.dataDir == null) System.getenv("MXNET_HOME") else inst.dataDir + val dtype = DType.withName(inst.dType) + val (dataLoader, net) = dataLoaderAndModel(inst.dataset, inst.network, dataPath, - inst.numLayers, inst.numExamples, inst.benchmark) + inst.numLayers, inst.numExamples, inst.benchmark, dtype) val devs = if (inst.gpus != null) inst.gpus.split(',').map(id => Context.gpu(id.trim.toInt)) @@ -210,5 +215,8 @@ class TrainModel { private val numWorker: Int = 1 @Option(name = "--num-server", usage = "# of servers") private val numServer: Int = 1 + @Option(name = "--dtype", usage = "data type of the model to train. " + + "Can be float32/float64. Works only with synthetic data currently") + private val dType: String = "float32" } diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/datasets/SyntheticDataIter.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/datasets/SyntheticDataIter.scala index 9421f1021..e4d3b2ae7 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/datasets/SyntheticDataIter.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/datasets/SyntheticDataIter.scala @@ -24,7 +24,7 @@ import scala.collection.immutable.ListMap import scala.util.Random class SyntheticDataIter(numClasses: Int, val batchSize: Int, datumShape: List[Int], - labelShape: List[Int], maxIter: Int, dtype: DType = DType.Float32 + labelShape: List[Int], maxIter: Int, dType: DType = DType.Float32 ) extends DataIter { var curIter = 0 val random = new Random() @@ -35,12 +35,12 @@ class SyntheticDataIter(numClasses: Int, val batchSize: Int, datumShape: List[In var label: IndexedSeq[NDArray] = IndexedSeq( NDArray.api.random_uniform(Some(0f), Some(maxLabel), shape = Some(batchLabelShape))) var data: IndexedSeq[NDArray] = IndexedSeq( - NDArray.api.random_uniform(shape = Some(shape))) + NDArray.api.random_uniform(shape = Some(shape), dtype = Some(dType.toString))) val provideDataDesc: IndexedSeq[DataDesc] = IndexedSeq( - new DataDesc("data", shape, dtype, Layout.UNDEFINED)) + new DataDesc("data", shape, data(0).dtype, Layout.UNDEFINED)) val provideLabelDesc: IndexedSeq[DataDesc] = IndexedSeq( - new DataDesc("softmax_label", batchLabelShape, dtype, Layout.UNDEFINED)) + new DataDesc("softmax_label", batchLabelShape, label(0).dtype, Layout.UNDEFINED)) val getPad: Int = 0 override def getData(): IndexedSeq[NDArray] = data diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Lenet.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Lenet.scala index 76fb7bb66..6f8b138d5 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Lenet.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Lenet.scala @@ -17,6 +17,7 @@ package org.apache.mxnetexamples.imclassification.models +import org.apache.mxnet.DType.DType import org.apache.mxnet._ object Lenet { @@ -26,8 +27,8 @@ object Lenet { * @param numClasses Number of classes to classify into * @return model symbol */ - def getSymbol(numClasses: Int): Symbol = { - val data = Symbol.Variable("data") + def getSymbol(numClasses: Int, dtype: DType = DType.Float32): Symbol = { + val data = Symbol.Variable("data", dType = dtype) // first conv val conv1 = Symbol.api.Convolution(data = Some(data), kernel = Shape(5, 5), num_filter = 20) val tanh1 = Symbol.api.tanh(data = Some(conv1)) diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/MultiLayerPerceptron.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/MultiLayerPerceptron.scala index 5d880bbe0..089b65f24 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/MultiLayerPerceptron.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/MultiLayerPerceptron.scala @@ -17,6 +17,7 @@ package org.apache.mxnetexamples.imclassification.models +import org.apache.mxnet.DType.DType import org.apache.mxnet._ object MultiLayerPerceptron { @@ -26,8 +27,8 @@ object MultiLayerPerceptron { * @param numClasses Number of classes to classify into * @return model symbol */ - def getSymbol(numClasses: Int): Symbol = { - val data = Symbol.Variable("data") + def getSymbol(numClasses: Int, dtype: DType = DType.Float32): Symbol = { + val data = Symbol.Variable("data", dType = dtype) val fc1 = Symbol.api.FullyConnected(data = Some(data), num_hidden = 128, name = "fc1") val act1 = Symbol.api.Activation(data = Some(fc1), "relu", name = "relu") diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Resnet.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Resnet.scala index c3f43d97e..e5f597680 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Resnet.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/imclassification/models/Resnet.scala @@ -17,6 +17,7 @@ package org.apache.mxnetexamples.imclassification.models +import org.apache.mxnet.DType.DType import org.apache.mxnet._ object Resnet { @@ -77,13 +78,14 @@ object Resnet { */ def resnet(units: List[Int], numStages: Int, filterList: List[Int], numClasses: Int, imageShape: List[Int], bottleNeck: Boolean = true, bnMom: Float = 0.9f, - workspace: Int = 256, dtype: String = "float32", memonger: Boolean = false): Symbol = { + workspace: Int = 256, dtype: DType = DType.Float32, + memonger: Boolean = false): Symbol = { assert(units.size == numStages) var data = Symbol.Variable("data", shape = Shape(List(4) ::: imageShape), dType = DType.Float32) - if (dtype == "float32") { + if (dtype == DType.Float32) { data = Symbol.api.identity(Some(data), "id") - } else if (dtype == "float16") { - data = Symbol.api.cast(Some(data), "float16") + } else if (dtype == DType.Float16) { + data = Symbol.api.cast(Some(data), DType.Float16.toString) } data = Symbol.api.BatchNorm(Some(data), fix_gamma = Some(true), eps = Some(2e-5), momentum = Some(bnMom), name = "bn_data") @@ -118,8 +120,8 @@ object Resnet { kernel = Some(Shape(7, 7)), pool_type = Some("avg"), name = "pool1") val flat = Symbol.api.Flatten(Some(pool1)) var fc1 = Symbol.api.FullyConnected(Some(flat), num_hidden = numClasses, name = "fc1") - if (dtype == "float16") { - fc1 = Symbol.api.cast(Some(fc1), "float32") + if (dtype == DType.Float16) { + fc1 = Symbol.api.cast(Some(fc1), DType.Float32.toString) } Symbol.api.SoftmaxOutput(Some(fc1), name = "softmax") } @@ -134,7 +136,7 @@ object Resnet { * @return Model symbol */ def getSymbol(numClasses: Int, numLayers: Int, imageShape: List[Int], convWorkspace: Int = 256, - dtype: String = "float32"): Symbol = { + dtype: DType = DType.Float32): Symbol = { val List(channels, height, width) = imageShape val (numStages, units, filterList, bottleNeck): (Int, List[Int], List[Int], Boolean) = if (height <= 28) { diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/README.md b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/README.md index 69328a44b..77aec7bb5 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/README.md +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/README.md @@ -28,18 +28,13 @@ The model is trained on the [Pascal VOC 2012 dataset](http://host.robots.ox.ac.u ### Download Artifacts #### Step 1 You can download the files using the script `get_ssd_data.sh`. It will download and place the model files in a `model` folder and the test image files in a `image` folder in the current directory. -From the `scala-package/examples/scripts/infer/imageclassifier/` folder run: +From the `scala-package/examples/scripts/infer/objectdetector/` folder run: ```bash -./get_resnet_data.sh +./get_ssd_data.sh ``` -**Note**: You may need to run `chmod +x get_resnet_data.sh` before running this script. - -Alternatively use the following links to download the Symbol and Params files via your browser: -- [resnet50_ssd_model-symbol.json](https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-symbol.json) -- [resnet50_ssd_model-0000.params](https://s3.amazonaws.com/model-server/models/resnet50_ssd/resnet50_ssd_model-0000.params) -- [synset.txt](https://github.com/awslabs/mxnet-model-server/blob/master/examples/ssd/synset.txt) +**Note**: You may need to run `chmod +x get_ssd_data.sh` before running this script. In the pre-trained model, the `input_name` is `data` and shape is `(1, 3, 512, 512)`. This shape translates to: a batch of `1` image, the image has color and uses `3` channels (RGB), and the image has the dimensions of `512` pixels in height by `512` pixels in width. @@ -57,13 +52,6 @@ The output shape is `(1, 6132, 6)`. As with the input, the `1` is the number of ### Setup Datapath and Parameters #### Step 2 -The code `Line 31: val baseDir = System.getProperty("user.dir")` in the example will automatically searches the work directory you have defined. Please put the files in your [work directory](https://stackoverflow.com/questions/16239130/java-user-dir-property-what-exactly-does-it-mean). - -Alternatively, if you would like to use your own path, please change line 31 into your own path -```scala -val baseDir = -``` - The followings is the parameters defined for this example, you can find more information in the `class SSDClassifierExample`. | Argument | Comments | @@ -79,7 +67,7 @@ After the previous steps, you should be able to run the code using the following From the `scala-package/examples/scripts/inferexample/objectdetector/` folder run: ```bash -./run_ssd_example.sh ../models/resnet50_ssd_model ../images/dog.jpg ../images +./run_ssd_example.sh ../models/resnet50_ssd/resnet50_ssd_model ../images/dog.jpg ../images ``` **Notes**: diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala index f752ef6da..07d1cc82e 100644 --- a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/objectdetector/SSDClassifierExample.scala @@ -182,9 +182,9 @@ object SSDClassifierExample { def checkExist(arr : Array[String]) : Boolean = { var exist : Boolean = true for (item <- arr) { - exist = Files.exists(Paths.get(item)) && exist - if (!exist) { + if (!(Files.exists(Paths.get(item)))) { logger.error("Cannot find: " + item) + exist = false } } exist diff --git a/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/predictor/PredictorExample.scala b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/predictor/PredictorExample.scala new file mode 100644 index 000000000..be90936ce --- /dev/null +++ b/scala-package/examples/src/main/scala/org/apache/mxnetexamples/infer/predictor/PredictorExample.scala @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.infer.predictor + +import java.io.File + +import scala.io +import org.apache.mxnet._ +import org.apache.mxnet.infer.Predictor +import org.apache.mxnetexamples.benchmark.CLIParserBase +import org.kohsuke.args4j.{CmdLineParser, Option} + +import scala.collection.JavaConverters._ + +object PredictorExample { + + def loadModel(modelPathPrefix : String, inputDesc : IndexedSeq[DataDesc], + context : Context, epoch : Int): Predictor = { + new Predictor(modelPathPrefix, inputDesc, context, Some(epoch)) + } + + def doInference(predictor : Predictor, imageND : NDArray): IndexedSeq[NDArray] = { + predictor.predictWithNDArray(IndexedSeq(imageND)) + } + + def preProcess(imagePath: String, h: Int, w: Int) : NDArray = { + var img = Image.imRead(imagePath) + img = Image.imResize(img, h, w) + // HWC -> CHW + img = NDArray.api.transpose(img, Some(Shape(2, 0, 1))) + img = NDArray.api.expand_dims(img, 0) + img.asType(DType.Float32) + } + + def postProcess(modelPathPrefix : String, result : Array[Float]) : String = { + val dirPath = modelPathPrefix.substring(0, 1 + modelPathPrefix.lastIndexOf(File.separator)) + val d = new File(dirPath) + require(d.exists && d.isDirectory, s"directory: $dirPath not found") + val f = io.Source.fromFile(dirPath + "synset.txt") + val s = f.getLines().toIndexedSeq + val maxIdx = result.zipWithIndex.maxBy(_._1)._2 + printf(s"Predict Result ${s(maxIdx)} with prob ${result(maxIdx)}\n") + s(maxIdx) + } + + def main(args : Array[String]): Unit = { + val inst = new CLIParser + val parser: CmdLineParser = new CmdLineParser(inst) + + parser.parseArgument(args.toList.asJava) + + var context = Context.cpu() + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + System.getenv("SCALA_TEST_ON_GPU").toInt == 1) { + context = Context.gpu() + } + + val imgWidth = 224 + val imgHeight = 224 + + val inputDesc = IndexedSeq(new DataDesc("data", Shape(1, 3, imgHeight, imgWidth), + DType.Float32, Layout.NCHW)) + + val predictor = loadModel(inst.modelPathPrefix, inputDesc, context, 0) + val img = preProcess(inst.inputImagePath, imgHeight, imgWidth) + val result = doInference(predictor, img)(0).toArray + postProcess(inst.modelPathPrefix, result) + } + +} + +class CLIParser extends CLIParserBase{ + @Option(name = "--model-path-prefix", usage = "the input model directory") + val modelPathPrefix: String = "/resnet-152/resnet-152" + @Option(name = "--input-image", usage = "the input image") + val inputImagePath: String = "/images/kitten.jpg" +} diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/cnntextclassification/CNNClassifierExampleSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/cnntextclassification/CNNClassifierExampleSuite.scala index 44025c045..a7e36dfc3 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/cnntextclassification/CNNClassifierExampleSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/cnntextclassification/CNNClassifierExampleSuite.scala @@ -29,8 +29,7 @@ import org.slf4j.LoggerFactory import scala.sys.process.Process /** - * Integration test for imageClassifier example. - * This will run as a part of "make scalatest" + * Integration test for CNN example. */ class CNNClassifierExampleSuite extends FunSuite with BeforeAndAfterAll { private val logger = LoggerFactory.getLogger(classOf[CNNClassifierExampleSuite]) diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/imclassification/IMClassificationExampleSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/imclassification/IMClassificationExampleSuite.scala index 6e9667abe..0daba5a97 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/imclassification/IMClassificationExampleSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/imclassification/IMClassificationExampleSuite.scala @@ -19,7 +19,7 @@ package org.apache.mxnetexamples.imclassification import java.io.File -import org.apache.mxnet.Context +import org.apache.mxnet.{Context, DType} import org.apache.mxnetexamples.Util import org.scalatest.{BeforeAndAfterAll, FunSuite} import org.slf4j.LoggerFactory @@ -55,9 +55,15 @@ class IMClassificationExampleSuite extends FunSuite with BeforeAndAfterAll { for(model <- List("mlp", "lenet", "resnet")) { test(s"Example CI: Test Image Classification Model ${model}") { - var context = Context.cpu() val valAccuracy = TrainModel.test(model, "", 10, 1, benchmark = true) } } + for(model <- List("mlp", "lenet", "resnet")) { + test(s"Example CI: Test Image Classification Model ${model} with Float64 input") { + val valAccuracy = TrainModel.test(model, "", 10, 1, benchmark = true, + dtype = DType.Float64) + } + } + } diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/imageclassifier/ImageClassifierExampleSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/imageclassifier/ImageClassifierExampleSuite.scala index d8631df54..c5308ac37 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/imageclassifier/ImageClassifierExampleSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/imageclassifier/ImageClassifierExampleSuite.scala @@ -20,17 +20,13 @@ package org.apache.mxnetexamples.infer.imageclassifier import org.scalatest.{BeforeAndAfterAll, FunSuite} import org.slf4j.LoggerFactory import java.io.File -import java.net.URL - -import org.apache.commons.io.FileUtils -import org.apache.mxnet.{Context, NDArrayCollector} +import org.apache.mxnet.Context import org.apache.mxnetexamples.Util import sys.process.Process /** * Integration test for imageClassifier example. - * This will run as a part of "make scalatest" */ class ImageClassifierExampleSuite extends FunSuite with BeforeAndAfterAll { private val logger = LoggerFactory.getLogger(classOf[ImageClassifierExampleSuite]) diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/objectdetector/ObjectDetectorExampleSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/objectdetector/ObjectDetectorExampleSuite.scala index addc837e6..bd960bdde 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/objectdetector/ObjectDetectorExampleSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/objectdetector/ObjectDetectorExampleSuite.scala @@ -18,10 +18,7 @@ package org.apache.mxnetexamples.infer.objectdetector import java.io.File -import java.net.URL - -import org.apache.commons.io.FileUtils -import org.apache.mxnet.{Context, NDArrayCollector} +import org.apache.mxnet.Context import org.apache.mxnetexamples.Util import org.scalatest.{BeforeAndAfterAll, FunSuite} import org.slf4j.LoggerFactory diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/predictor/PredictorExampleSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/predictor/PredictorExampleSuite.scala new file mode 100644 index 000000000..97ca33e71 --- /dev/null +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/infer/predictor/PredictorExampleSuite.scala @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnetexamples.infer.predictor + +import java.io.File + +import org.apache.mxnet._ +import org.apache.mxnetexamples.Util +import org.scalatest.{BeforeAndAfterAll, FunSuite} +import org.slf4j.LoggerFactory + +class PredictorExampleSuite extends FunSuite with BeforeAndAfterAll { + private val logger = LoggerFactory.getLogger(classOf[PredictorExampleSuite]) + private var modelDirPrefix = "" + private var inputImagePath = "" + private var context = Context.cpu() + + override def beforeAll(): Unit = { + logger.info("Downloading resnet-18 model") + + val tempDirPath = System.getProperty("java.io.tmpdir") + logger.info("tempDirPath: %s".format(tempDirPath)) + + val baseUrl = "https://s3.us-east-2.amazonaws.com/scala-infer-models" + + Util.downloadUrl(baseUrl + "/resnet-18/resnet-18-symbol.json", + tempDirPath + "/resnet18/resnet-18-symbol.json") + Util.downloadUrl(baseUrl + "/resnet-18/resnet-18-0000.params", + tempDirPath + "/resnet18/resnet-18-0000.params") + Util.downloadUrl(baseUrl + "/resnet-18/synset.txt", + tempDirPath + "/resnet18/synset.txt") + Util.downloadUrl("https://s3.amazonaws.com/model-server/inputs/Pug-Cookie.jpg", + tempDirPath + "/inputImages/resnet18/Pug-Cookie.jpg") + + modelDirPrefix = tempDirPath + File.separator + "resnet18/resnet-18" + inputImagePath = tempDirPath + File.separator + + "inputImages/resnet18/Pug-Cookie.jpg" + + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + System.getenv("SCALA_TEST_ON_GPU").toInt == 1) { + context = Context.gpu() + } + val props = System.getProperties + props.setProperty("mxnet.disableShapeCheck", "true") + } + + override def afterAll(): Unit = { + val props = System.getProperties + props.setProperty("mxnet.disableShapeCheck", "false") + } + + test("test Predictor With Fixed Shape and random shape") { + val inputDesc = IndexedSeq(new DataDesc("data", Shape(1, 3, 224, 224), + DType.Float32, Layout.NCHW)) + val predictor = PredictorExample.loadModel(modelDirPrefix, inputDesc, context, 0) + // fix size + var img = PredictorExample.preProcess(inputImagePath, 224, 224) + var result = PredictorExample.doInference(predictor, img)(0) + var top1 = PredictorExample.postProcess(modelDirPrefix, result.toArray) + assert(top1 === "n02110958 pug, pug-dog") + // random size 512 + img = PredictorExample.preProcess(inputImagePath, 512, 512) + result = PredictorExample.doInference(predictor, img)(0) + top1 = PredictorExample.postProcess(modelDirPrefix, result.toArray) + assert(top1 === "n02110958 pug, pug-dog") + // original size + img = PredictorExample.preProcess(inputImagePath, 1024, 576) + result = PredictorExample.doInference(predictor, img)(0) + top1 = PredictorExample.postProcess(modelDirPrefix, result.toArray) + assert(top1 === "n02110958 pug, pug-dog") + } +} diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/multitask/MultiTaskSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/multitask/MultiTaskSuite.scala index 983978dba..65902c7ad 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/multitask/MultiTaskSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/multitask/MultiTaskSuite.scala @@ -25,8 +25,7 @@ import org.scalatest.FunSuite /** - * Integration test for imageClassifier example. - * This will run as a part of "make scalatest" + * Integration test for Multi-task example. */ class MultiTaskSuite extends FunSuite { test("Multitask Test") { diff --git a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/profiler/ProfilerSuite.scala b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/profiler/ProfilerSuite.scala index 859a8c064..67638b8fa 100644 --- a/scala-package/examples/src/test/scala/org/apache/mxnetexamples/profiler/ProfilerSuite.scala +++ b/scala-package/examples/src/test/scala/org/apache/mxnetexamples/profiler/ProfilerSuite.scala @@ -25,8 +25,7 @@ import org.apache.mxnet.Profiler import org.apache.mxnet.Context /** - * Integration test for imageClassifier example. - * This will run as a part of "make scalatest" + * Integration test for profiler example. */ class ProfilerSuite extends FunSuite with BeforeAndAfterAll { private val logger = LoggerFactory.getLogger(classOf[ProfilerSuite]) diff --git a/scala-package/infer/pom.xml b/scala-package/infer/pom.xml index e50100169..13ceebb83 100644 --- a/scala-package/infer/pom.xml +++ b/scala-package/infer/pom.xml @@ -2,104 +2,75 @@ - 4.0.0 - - mxnet-parent_2.11 - org.apache.mxnet - 1.3.1-SNAPSHOT - ../pom.xml - + 4.0.0 + + mxnet-parent + org.apache.mxnet + INTERNAL + ../pom.xml + - mxnet-infer_2.11 - MXNet Scala Package - Inference + + false + - - - unittest - - false - - - - integrationtest - - true - - - - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - + mxnet-infer + MXNet Scala Package - Inference - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-jar-plugin - - - META-INF/*.SF - META-INF/*.DSA - META-INF/*.RSA - - - - - org.apache.maven.plugins - maven-compiler-plugin - - - org.scalatest - scalatest-maven-plugin - - ${skipTests} - - -Djava.library.path=${project.parent.basedir}/native/${platform}/target \ - -Dlog4j.configuration=file://${project.basedir}/src/test/resources/log4j.properties - - - - - org.scalastyle - scalastyle-maven-plugin - - - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - provided - - - - org.mockito - mockito-all - 1.10.19 - test - - + + + + org.apache.maven.plugins + maven-jar-plugin + + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + org.apache.maven.plugins + maven-compiler-plugin + + + org.scalatest + scalatest-maven-plugin + + + -Djava.library.path=${project.parent.basedir}/native/target \ + -Dlog4j.configuration=file://${project.basedir}/src/test/resources/log4j.properties + + + + + org.scalastyle + scalastyle-maven-plugin + + + + + + org.apache.mxnet + mxnet-core + INTERNAL + provided + + + + org.mockito + mockito-all + 1.10.19 + test + + + + junit + junit + 4.11 + test + + + diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Classifier.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Classifier.scala index adeb33d34..bf6581588 100644 --- a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Classifier.scala +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Classifier.scala @@ -17,25 +17,27 @@ package org.apache.mxnet.infer -import org.apache.mxnet.{Context, DataDesc, NDArray} +import org.apache.mxnet._ import java.io.File +import org.apache.mxnet.MX_PRIMITIVES.MX_PRIMITIVE_TYPE import org.slf4j.LoggerFactory import scala.io import scala.collection.mutable.ListBuffer +import scala.collection.parallel.mutable.ParArray trait ClassifierBase { /** * Takes an array of floats and returns corresponding (Label, Score) tuples - * @param input Indexed sequence one-dimensional array of floats + * @param input Indexed sequence one-dimensional array of floats/doubles * @param topK (Optional) How many result (sorting based on the last axis) * elements to return. Default returns unsorted output. * @return Indexed sequence of (Label, Score) tuples */ - def classify(input: IndexedSeq[Array[Float]], - topK: Option[Int] = None): IndexedSeq[(String, Float)] + def classify[@specialized (Base.MX_PRIMITIVES) T](input: IndexedSeq[Array[T]], + topK: Option[Int] = None): IndexedSeq[(String, T)] /** * Takes a sequence of NDArrays and returns (Label, Score) tuples @@ -77,17 +79,35 @@ class Classifier(modelPathPrefix: String, /** * Takes flat arrays as input and returns (Label, Score) tuples. - * @param input Indexed sequence one-dimensional array of floats + * @param input Indexed sequence one-dimensional array of floats/doubles * @param topK (Optional) How many result (sorting based on the last axis) * elements to return. Default returns unsorted output. * @return Indexed sequence of (Label, Score) tuples */ - override def classify(input: IndexedSeq[Array[Float]], - topK: Option[Int] = None): IndexedSeq[(String, Float)] = { + override def classify[@specialized (Base.MX_PRIMITIVES) T](input: IndexedSeq[Array[T]], + topK: Option[Int] = None): IndexedSeq[(String, T)] = { + + // considering only the first output + val result = input(0)(0) match { + case d: Double => { + classifyImpl(input.asInstanceOf[IndexedSeq[Array[Double]]], topK) + } + case _ => { + classifyImpl(input.asInstanceOf[IndexedSeq[Array[Float]]], topK) + } + } + + result.asInstanceOf[IndexedSeq[(String, T)]] + } + + private def classifyImpl[B, A <: MX_PRIMITIVE_TYPE] + (input: IndexedSeq[Array[B]], topK: Option[Int] = None)(implicit ev: B => A) + : IndexedSeq[(String, B)] = { // considering only the first output val predictResult = predictor.predict(input)(0) - var result: IndexedSeq[(String, Float)] = IndexedSeq.empty + + var result: IndexedSeq[(String, B)] = IndexedSeq.empty if (topK.isDefined) { val sortedIndex = predictResult.zipWithIndex.sortBy(-_._1).map(_._2).take(topK.get) @@ -104,23 +124,27 @@ class Classifier(modelPathPrefix: String, * @param input Indexed sequence of NDArrays * @param topK (Optional) How many result (sorting based on the last axis) * elements to return. Default returns unsorted output. - * @return Traversable sequence of (Label, Score) tuples + * @return Traversable sequence of (Label, Score) tuples. */ override def classifyWithNDArray(input: IndexedSeq[NDArray], topK: Option[Int] = None) : IndexedSeq[IndexedSeq[(String, Float)]] = { // considering only the first output - val predictResultND: NDArray = predictor.predictWithNDArray(input)(0) - - val predictResult: ListBuffer[Array[Float]] = ListBuffer[Array[Float]]() + // Copy NDArray to CPU to avoid frequent GPU to CPU copying + val predictResultND: NDArray = + predictor.predictWithNDArray(input)(0).asInContext(Context.cpu()) + // Parallel Execution with ParArray for better performance + val predictResultPar: ParArray[Array[Float]] = + new ParArray[Array[Float]](predictResultND.shape(0)) // iterating over the individual items(batch size is in axis 0) - for (i <- 0 until predictResultND.shape(0)) { + (0 until predictResultND.shape(0)).toVector.par.foreach( i => { val r = predictResultND.at(i) - predictResult += r.toArray + predictResultPar(i) = r.toArray r.dispose() - } + }) + val predictResult = predictResultPar.toArray var result: ListBuffer[IndexedSeq[(String, Float)]] = ListBuffer.empty[IndexedSeq[(String, Float)]] diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ImageClassifier.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ImageClassifier.scala index 96be12179..3c80f9226 100644 --- a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ImageClassifier.scala +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ImageClassifier.scala @@ -17,7 +17,8 @@ package org.apache.mxnet.infer -import org.apache.mxnet.{Context, DataDesc, NDArray, Shape} +import org.apache.mxnet.DType.DType +import org.apache.mxnet._ import scala.collection.mutable.ListBuffer @@ -70,14 +71,18 @@ class ImageClassifier(modelPathPrefix: String, * * @param inputImage Path prefix of the input image * @param topK Number of result elements to return, sorted by probability + * @param dType The precision at which to run the inference. + * specify the DType as DType.Float64 for Double precision. + * Defaults to DType.Float32 * @return List of list of tuples of (Label, Probability) */ - def classifyImage(inputImage: BufferedImage, - topK: Option[Int] = None): IndexedSeq[IndexedSeq[(String, Float)]] = { + def classifyImage + (inputImage: BufferedImage, topK: Option[Int] = None, dType: DType = DType.Float32): + IndexedSeq[IndexedSeq[(String, Float)]] = { val scaledImage = ImageClassifier.reshapeImage(inputImage, width, height) val imageShape = inputShape.drop(1) - val pixelsNDArray = ImageClassifier.bufferedImageToPixels(scaledImage, imageShape) + val pixelsNDArray = ImageClassifier.bufferedImageToPixels(scaledImage, imageShape, dType) val imgWithBatchNum = NDArray.api.expand_dims(pixelsNDArray, 0) inputImage.flush() scaledImage.flush() @@ -95,16 +100,19 @@ class ImageClassifier(modelPathPrefix: String, * * @param inputBatch Input array of buffered images * @param topK Number of result elements to return, sorted by probability + * @param dType The precision at which to run the inference. + * specify the DType as DType.Float64 for Double precision. + * Defaults to DType.Float32 * @return List of list of tuples of (Label, Probability) */ - def classifyImageBatch(inputBatch: Traversable[BufferedImage], topK: Option[Int] = None): - IndexedSeq[IndexedSeq[(String, Float)]] = { + def classifyImageBatch(inputBatch: Traversable[BufferedImage], topK: Option[Int] = None, + dType: DType = DType.Float32): IndexedSeq[IndexedSeq[(String, Float)]] = { val inputBatchSeq = inputBatch.toIndexedSeq val imageBatch = inputBatchSeq.indices.par.map(idx => { val scaledImage = ImageClassifier.reshapeImage(inputBatchSeq(idx), width, height) val imageShape = inputShape.drop(1) - val imgND = ImageClassifier.bufferedImageToPixels(scaledImage, imageShape) + val imgND = ImageClassifier.bufferedImageToPixels(scaledImage, imageShape, dType) val imgWithBatch = NDArray.api.expand_dims(imgND, 0).get handler.execute(imgND.dispose()) imgWithBatch @@ -152,11 +160,29 @@ object ImageClassifier { * returned by this method after the use. *

      * @param resizedImage BufferedImage to get pixels from + * * @param inputImageShape Input shape; for example for resnet it is (3,224,224). Should be same as inputDescriptor shape. + * @param dType The DataType of the NDArray created from the image + * that should be returned. + * Currently it defaults to Dtype.Float32 * @return NDArray pixels array with shape (3, 224, 224) in CHW format */ - def bufferedImageToPixels(resizedImage: BufferedImage, inputImageShape: Shape): NDArray = { + def bufferedImageToPixels(resizedImage: BufferedImage, inputImageShape: Shape, + dType : DType = DType.Float32): NDArray = { + + if (dType == DType.Float64) { + val result = getFloatPixelsArray(resizedImage) + NDArray.array(result.map(_.toDouble), shape = inputImageShape) + } + else { + val result = getFloatPixelsArray(resizedImage) + NDArray.array(result, shape = inputImageShape) + } + } + + private def getFloatPixelsArray(resizedImage: BufferedImage): Array[Float] = { + // Get height and width of the image val w = resizedImage.getWidth() val h = resizedImage.getHeight() @@ -166,7 +192,6 @@ object ImageClassifier { // 3 times height and width for R,G,B channels val result = new Array[Float](3 * h * w) - var row = 0 // copy pixels to array vertically while (row < h) { @@ -184,11 +209,10 @@ object ImageClassifier { } row += 1 } + resizedImage.flush() - // creating NDArray according to the input shape - val pixelsArray = NDArray.array(result, shape = inputImageShape) - pixelsArray + result } /** diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ObjectDetector.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ObjectDetector.scala index a9b21f8c1..78b237a4a 100644 --- a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ObjectDetector.scala +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/ObjectDetector.scala @@ -19,6 +19,8 @@ package org.apache.mxnet.infer // scalastyle:off import java.awt.image.BufferedImage + +import scala.collection.parallel.mutable.ParArray // scalastyle:on import org.apache.mxnet.NDArray import org.apache.mxnet.DataDesc @@ -94,39 +96,39 @@ class ObjectDetector(modelPathPrefix: String, def objectDetectWithNDArray(input: IndexedSeq[NDArray], topK: Option[Int]) : IndexedSeq[IndexedSeq[(String, Array[Float])]] = { - val predictResult = predictor.predictWithNDArray(input)(0) - var batchResult = ListBuffer[IndexedSeq[(String, Array[Float])]]() - for (i <- 0 until predictResult.shape(0)) { + // Copy NDArray to CPU to avoid frequent GPU to CPU copying + val predictResult = predictor.predictWithNDArray(input)(0).asInContext(Context.cpu()) + // Parallel Execution with ParArray for better performance + var batchResult = new ParArray[IndexedSeq[(String, Array[Float])]](predictResult.shape(0)) + (0 until predictResult.shape(0)).toArray.par.foreach( i => { val r = predictResult.at(i) - batchResult += sortAndReformat(r, topK) + batchResult(i) = sortAndReformat(r, topK) handler.execute(r.dispose()) - } + }) handler.execute(predictResult.dispose()) batchResult.toIndexedSeq } private[infer] def sortAndReformat(predictResultND: NDArray, topK: Option[Int]) : IndexedSeq[(String, Array[Float])] = { - val predictResult: ListBuffer[Array[Float]] = ListBuffer[Array[Float]]() - val accuracy: ListBuffer[Float] = ListBuffer[Float]() - // iterating over the all the predictions val length = predictResultND.shape(0) - for (i <- 0 until length) { + val predictResult = (0 until length).toArray.par.flatMap( i => { val r = predictResultND.at(i) val tempArr = r.toArray - if (tempArr(0) != -1.0) { - predictResult += tempArr - accuracy += tempArr(1) + val res = if (tempArr(0) != -1.0) { + Array[Array[Float]](tempArr) } else { // Ignore the minus 1 part + Array[Array[Float]]() } handler.execute(r.dispose()) - } + res + }).toArray var result = IndexedSeq[(String, Array[Float])]() if (topK.isDefined) { - var sortedIndices = accuracy.zipWithIndex.sortBy(-_._1).map(_._2) + var sortedIndices = predictResult.zipWithIndex.sortBy(-_._1(1)).map(_._2) sortedIndices = sortedIndices.take(topK.get) // takeRight(5) would provide the output as Array[Accuracy, Xmin, Ymin, Xmax, Ymax result = sortedIndices.map(idx @@ -136,7 +138,6 @@ class ObjectDetector(modelPathPrefix: String, result = predictResult.map(ele => (synset(ele(0).toInt), ele.takeRight(5))).toIndexedSeq } - result } diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Predictor.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Predictor.scala index e2a0e7ca2..67692a316 100644 --- a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Predictor.scala +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/Predictor.scala @@ -17,13 +17,16 @@ package org.apache.mxnet.infer +import org.apache.mxnet.MX_PRIMITIVES.MX_PRIMITIVE_TYPE import org.apache.mxnet.io.NDArrayIter -import org.apache.mxnet.{Context, DataDesc, NDArray, Shape} +import org.apache.mxnet._ import org.apache.mxnet.module.Module import scala.collection.mutable.ListBuffer +import scala.util.Try import org.slf4j.LoggerFactory + /** * Base Trait for MXNet Predictor classes. */ @@ -34,11 +37,13 @@ private[infer] trait PredictBase { *

      * This method will take input as IndexedSeq one dimensional arrays and creates the * NDArray needed for inference. The array will be reshaped based on the input descriptors. - * @param input: An IndexedSequence of a one-dimensional array. + * @param input: An Indexed Sequence of a one-dimensional array of datatype + * Float or Double An IndexedSequence is needed when the model has more than one input. * @return Indexed sequence array of outputs */ - def predict(input: IndexedSeq[Array[Float]]): IndexedSeq[Array[Float]] + def predict[@specialized (Base.MX_PRIMITIVES) T](input: IndexedSeq[Array[T]]) + : IndexedSeq[Array[T]] /** * Predict using NDArray as input. @@ -76,6 +81,21 @@ class Predictor(modelPathPrefix: String, private val logger = LoggerFactory.getLogger(classOf[Predictor]) + /* + By setting -Dmxnet.disableShapeCheck=true would disable the data Shape + Check of the predictor. Some model may allow different lens of the data + such as Seq2Seq, however there maybe risk of crashes if the lens beyond + the acceptable range of the model + */ + private val traceProperty = "mxnet.disableShapeCheck" + private lazy val shapeCheckDisabled = { + val value = Try(System.getProperty(traceProperty).toBoolean).getOrElse(false) + if (value) { + logger.warn("Shape check is disabled (property {} is set)", traceProperty) + } + value + } + require(inputDescriptors.head.layout.size != 0, "layout size should not be zero") protected[infer] var batchIndex = inputDescriptors(0).layout.indexOf('N') @@ -106,13 +126,13 @@ class Predictor(modelPathPrefix: String, * Takes input as IndexedSeq one dimensional arrays and creates the NDArray needed for inference * The array will be reshaped based on the input descriptors. * - * @param input: An IndexedSequence of a one-dimensional array. + * @param input: An IndexedSequence of a one-dimensional array + * of data type Float or Double. An IndexedSequence is needed when the model has more than one input. * @return Indexed sequence array of outputs */ - override def predict(input: IndexedSeq[Array[Float]]) - : IndexedSeq[Array[Float]] = { - + override def predict[@specialized (Base.MX_PRIMITIVES) T](input: IndexedSeq[Array[T]]) + : IndexedSeq[Array[T]] = { require(input.length == inputDescriptors.length, s"number of inputs provided: ${input.length} does not match number of inputs " + s"in inputDescriptors: ${inputDescriptors.length}") @@ -122,12 +142,30 @@ class Predictor(modelPathPrefix: String, s"number of elements:${i.length} in the input does not match the shape:" + s"${d.shape.toString()}") } + + // Infer the dtype of input and call relevant method + val result = input(0)(0) match { + case d: Double => predictImpl(input.asInstanceOf[IndexedSeq[Array[Double]]]) + case _ => predictImpl(input.asInstanceOf[IndexedSeq[Array[Float]]]) + } + + result.asInstanceOf[IndexedSeq[Array[T]]] + } + + private def predictImpl[B, A <: MX_PRIMITIVE_TYPE] + (input: IndexedSeq[Array[B]])(implicit ev: B => A) + : IndexedSeq[Array[B]] = { + var inputND: ListBuffer[NDArray] = ListBuffer.empty[NDArray] for((i, d) <- input.zip(inputDescriptors)) { val shape = d.shape.toVector.patch(from = batchIndex, patch = Vector(1), replaced = 1) - - inputND += mxNetHandler.execute(NDArray.array(i, Shape(shape))) + if (d.dtype == DType.Float64) { + inputND += mxNetHandler.execute(NDArray.array(i.asInstanceOf[Array[Double]], Shape(shape))) + } + else { + inputND += mxNetHandler.execute(NDArray.array(i.asInstanceOf[Array[Float]], Shape(shape))) + } } // rebind with batchsize 1 @@ -141,7 +179,8 @@ class Predictor(modelPathPrefix: String, val resultND = mxNetHandler.execute(mod.predict(new NDArrayIter( inputND.toIndexedSeq, dataBatchSize = 1))) - val result = resultND.map((f : NDArray) => f.toArray) + val result = + resultND.map((f : NDArray) => if (f.dtype == DType.Float64) f.toFloat64Array else f.toArray) mxNetHandler.execute(inputND.foreach(_.dispose)) mxNetHandler.execute(resultND.foreach(_.dispose)) @@ -151,9 +190,11 @@ class Predictor(modelPathPrefix: String, mxNetHandler.execute(mod.bind(inputDescriptors, forTraining = false, forceRebind = true)) } - result + result.asInstanceOf[IndexedSeq[Array[B]]] } + + /** * Predict using NDArray as input * This method is useful when the input is a batch of data @@ -172,9 +213,11 @@ class Predictor(modelPathPrefix: String, for((i, d) <- inputBatch.zip(iDescriptors)) { require(inputBatch(0).shape(batchIndex) == i.shape(batchIndex), "All inputs should be of same batch size") - require(i.shape.drop(batchIndex + 1) == d.shape.drop(batchIndex + 1), - s"Input Data Shape: ${i.shape} should match the inputDescriptor " + - s"shape: ${d.shape} except batchSize") + if (!shapeCheckDisabled) { + require(i.shape.drop(batchIndex + 1) == d.shape.drop(batchIndex + 1), + s"Input Data Shape: ${i.shape} should match the inputDescriptor " + + s"shape: ${d.shape} except batchSize") + } } val inputBatchSize = inputBatch(0).shape(batchIndex) @@ -182,8 +225,8 @@ class Predictor(modelPathPrefix: String, // rebind with the new batchSize if (batchSize != inputBatchSize) { logger.info(s"Latency increased due to batchSize mismatch $batchSize vs $inputBatchSize") - val desc = iDescriptors.map((f : DataDesc) => new DataDesc(f.name, - Shape(f.shape.toVector.patch(batchIndex, Vector(inputBatchSize), 1)), f.dtype, f.layout) ) + val desc = inputBatch.zip(iDescriptors).map(f => new DataDesc(f._2.name, + f._1.shape, f._2.dtype, f._2.layout)) mxNetHandler.execute(mod.bind(desc, forceRebind = true, forTraining = false)) } @@ -200,7 +243,7 @@ class Predictor(modelPathPrefix: String, private[infer] def loadModule(): Module = { val mod = mxNetHandler.execute(Module.loadCheckpoint(modelPathPrefix, epoch.get, - contexts = contexts)) + contexts = contexts, dataNames = inputDescriptors.map(desc => desc.name))) mxNetHandler.execute(mod.bind(inputDescriptors, forTraining = false)) mod } diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetector.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetector.scala new file mode 100644 index 000000000..3014f8d97 --- /dev/null +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetector.scala @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi + +// scalastyle:off +import java.awt.image.BufferedImage +// scalastyle:on + +import org.apache.mxnet.javaapi.{Context, DataDesc, NDArray, Shape} + +import scala.collection.JavaConverters +import scala.collection.JavaConverters._ +import scala.language.implicitConversions + +/** + * The ObjectDetector class helps to run ObjectDetection tasks where the goal + * is to find bounding boxes and corresponding labels for objects in a image. + * + * @param modelPathPrefix Path prefix from where to load the model artifacts. + * These include the symbol, parameters, and synset.txt. + * Example: file://model-dir/ssd_resnet50_512 (containing + * ssd_resnet50_512-symbol.json, ssd_resnet50_512-0000.params, + * and synset.txt) + * @param inputDescriptors Descriptors defining the input node names, shape, + * layout and type parameters + * @param contexts Device contexts on which you want to run inference. + * Defaults to CPU. + * @param epoch Model epoch to load; defaults to 0 + */ +class ObjectDetector private[mxnet] (val objDetector: org.apache.mxnet.infer.ObjectDetector){ + + def this(modelPathPrefix: String, inputDescriptors: java.util.List[DataDesc], contexts: + java.util.List[Context], epoch: Int) + = this { + val informationDesc = JavaConverters.asScalaIteratorConverter(inputDescriptors.iterator) + .asScala.toIndexedSeq map {a => a: org.apache.mxnet.DataDesc} + val inContexts = (contexts.asScala.toList map {a => a: org.apache.mxnet.Context}).toArray + // scalastyle:off + new org.apache.mxnet.infer.ObjectDetector(modelPathPrefix, informationDesc, inContexts, Some(epoch)) + // scalastyle:on + } + + /** + * Detects objects and returns bounding boxes with corresponding class/label + * + * @param inputImage Path prefix of the input image + * @param topK Number of result elements to return, sorted by probability + * @return List of list of tuples of + * (class, [probability, xmin, ymin, xmax, ymax]) + */ + def imageObjectDetect(inputImage: BufferedImage, topK: Int): + java.util.List[java.util.List[ObjectDetectorOutput]] = { + val ret = objDetector.imageObjectDetect(inputImage, Some(topK)) + (ret map {a => (a map {e => new ObjectDetectorOutput(e._1, e._2)}).asJava}).asJava + } + + /** + * Takes input images as NDArrays. Useful when you want to perform multiple operations on + * the input array, or when you want to pass a batch of input images. + * + * @param input Indexed Sequence of NDArrays + * @param topK (Optional) How many top_k (sorting will be based on the last axis) + * elements to return. If not passed, returns all unsorted output. + * @return List of list of tuples of + * (class, [probability, xmin, ymin, xmax, ymax]) + */ + def objectDetectWithNDArray(input: java.util.List[NDArray], topK: Int): + java.util.List[java.util.List[ObjectDetectorOutput]] = { + val ret = objDetector.objectDetectWithNDArray(convert(input.asScala.toIndexedSeq), Some(topK)) + (ret map {a => (a map {e => new ObjectDetectorOutput(e._1, e._2)}).asJava}).asJava + } + + /** + * To classify batch of input images according to the provided model + * + * @param inputBatch Input array of buffered images + * @param topK Number of result elements to return, sorted by probability + * @return List of list of tuples of (class, probability) + */ + def imageBatchObjectDetect(inputBatch: java.util.List[BufferedImage], topK: Int): + java.util.List[java.util.List[ObjectDetectorOutput]] = { + val ret = objDetector.imageBatchObjectDetect(inputBatch.asScala, Some(topK)) + (ret map {a => (a map {e => new ObjectDetectorOutput(e._1, e._2)}).asJava}).asJava + } + + def convert[B, A <% B](l: IndexedSeq[A]): IndexedSeq[B] = l map { a => a: B } + +} + + +object ObjectDetector { + implicit def fromObjectDetector(OD: org.apache.mxnet.infer.ObjectDetector): + ObjectDetector = new ObjectDetector(OD) + + implicit def toObjectDetector(jOD: ObjectDetector): + org.apache.mxnet.infer.ObjectDetector = jOD.objDetector + + def loadImageFromFile(inputImagePath: String): BufferedImage = { + org.apache.mxnet.infer.ImageClassifier.loadImageFromFile(inputImagePath) + } + + def reshapeImage(img : BufferedImage, newWidth: Int, newHeight: Int): BufferedImage = { + org.apache.mxnet.infer.ImageClassifier.reshapeImage(img, newWidth, newHeight) + } + + def bufferedImageToPixels(resizedImage: BufferedImage, inputImageShape: Shape): NDArray = { + org.apache.mxnet.infer.ImageClassifier.bufferedImageToPixels(resizedImage, inputImageShape) + } + + def loadInputBatch(inputImagePaths: java.util.List[String]): java.util.List[BufferedImage] = { + org.apache.mxnet.infer.ImageClassifier + .loadInputBatch(inputImagePaths.asScala.toList).toList.asJava + } +} diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetectorOutput.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetectorOutput.scala new file mode 100644 index 000000000..5a6ac7599 --- /dev/null +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/ObjectDetectorOutput.scala @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi + +/** + * The ObjectDetectorOutput class is a simple POJO helper class that is used to simplify + * the interactions with ObjectDetector predict results. The class stores the bounding box + * coordinates, name of preicted class, and the probability. + */ + + +class ObjectDetectorOutput (className: String, args: Array[Float]){ + + /** + * Gets the predicted class's name. + * + * @return String representing the name of the predicted class + */ + def getClassName: String = className + + /** + * Gets the probability of the predicted class. + * + * @return Float representing the probability of predicted class + */ + def getProbability: Float = args(0) + + /** + * Gets the minimum X coordinate for the bounding box containing the predicted object. + * + * @return Float of the min X coordinate for the object bounding box + */ + def getXMin: Float = args(1) + + /** + * Gets the maximum X coordinate for the bounding box containing the predicted object. + * + * @return Float of the max X coordinate for the object bounding box + */ + def getXMax: Float = args(2) + + /** + * Gets the minimum Y coordinate for the bounding box containing the predicted object. + * + * @return Float of the min Y coordinate for the object bounding box + */ + def getYMin: Float = args(3) + + /** + * Gets the maximum Y coordinate for the bounding box containing the predicted object. + * + * @return Float of the max Y coordinate for the object bounding box + */ + def getYMax: Float = args(4) + +} diff --git a/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/Predictor.scala b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/Predictor.scala new file mode 100644 index 000000000..146fe9310 --- /dev/null +++ b/scala-package/infer/src/main/scala/org/apache/mxnet/infer/javaapi/Predictor.scala @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi + +import org.apache.mxnet.javaapi.{Context, DataDesc, NDArray} + +import scala.collection.JavaConverters +import scala.collection.JavaConverters._ + +/** + * Implementation of prediction routines. + * + * @param modelPathPrefix Path prefix from where to load the model artifacts. + * These include the symbol, parameters, and synset.txt + * Example: file://model-dir/resnet-152 (containing + * resnet-152-symbol.json, resnet-152-0000.params, and synset.txt). + * @param inputDescriptors Descriptors defining the input node names, shape, + * layout and type parameters + *

      Note: If the input Descriptors is missing batchSize + * ('N' in layout), a batchSize of 1 is assumed for the model. + * @param contexts Device contexts on which you want to run inference; defaults to CPU + * @param epoch Model epoch to load; defaults to 0 + + */ + +// JavaDoc description of class to be updated in https://issues.apache.org/jira/browse/MXNET-1178 +class Predictor private[mxnet] (val predictor: org.apache.mxnet.infer.Predictor){ + def this(modelPathPrefix: String, inputDescriptors: java.util.List[DataDesc], + contexts: java.util.List[Context], epoch: Int) + = this { + val informationDesc = JavaConverters.asScalaIteratorConverter(inputDescriptors.iterator) + .asScala.toIndexedSeq map {a => a: org.apache.mxnet.DataDesc} + val inContexts = (contexts.asScala.toList map {a => a: org.apache.mxnet.Context}).toArray + new org.apache.mxnet.infer.Predictor(modelPathPrefix, informationDesc, inContexts, Some(epoch)) + } + + /** + * Takes input as Array of one dimensional arrays and creates the NDArray needed for inference + * The array will be reshaped based on the input descriptors. Example of calling in Java: + * + * 

      +    * {@code
+    * float tmp[][] = new float[1][224];
+    * for (int x = 0; x < 1; x++)
+    *   for (int y = 0; y < 224; y++)
+    *     tmp[x][y] = (int)(Math.random()*10);
+    * predictor.predict(tmp);
+    * }
+    *
      + * + * @param input: An Array of a one-dimensional array. + An extra Array is needed for when the model has more than one input. + * @return Indexed sequence array of outputs + */ + def predict(input: Array[Array[Float]]): + Array[Array[Float]] = { + predictor.predict(input).toArray + } + + /** + * Takes input as Array of one dimensional arrays and creates the NDArray needed for inference + * The array will be reshaped based on the input descriptors. Example of calling in Java: + * + * 
      +    * {@code
+    * double tmp[][] = new double[1][224];
+    * for (int x = 0; x < 1; x++)
+    *   for (int y = 0; y < 224; y++)
+    *     tmp[x][y] = (int)(Math.random()*10);
+    * predictor.predict(tmp);
+    * }
+    *
      + * + * @param input: An Array of a one-dimensional array. + An extra Array is needed for when the model has more than one input. + * @return Indexed sequence array of outputs + */ + + def predict(input: Array[Array[Double]]): + Array[Array[Double]] = { + predictor.predict(input).toArray + } + + /** + * Takes input as List of one dimensional arrays and creates the NDArray needed for inference + * The array will be reshaped based on the input descriptors. + * + * @param input: A List of a one-dimensional array. + An extra List is needed for when the model has more than one input. + * @return Indexed sequence array of outputs + */ + def predict(input: java.util.List[java.util.List[java.lang.Float]]): + java.util.List[java.util.List[java.lang.Float]] = { + val in = JavaConverters.asScalaIteratorConverter(input.iterator).asScala.toIndexedSeq + (predictor.predict(in map {a => a.asScala.map(Float2float).toArray}) + map {b => b.map(float2Float).toList.asJava}).asJava + } + + + + /** + * Predict using NDArray as input + * This method is useful when the input is a batch of data + * Note: User is responsible for managing allocation/deallocation of input/output NDArrays. + * + * @param input List of NDArrays + * @return Output of predictions as NDArrays + */ + def predictWithNDArray(input: java.util.List[NDArray]): + java.util.List[NDArray] = { + val ret = predictor.predictWithNDArray(convert(JavaConverters + .asScalaIteratorConverter(input.iterator).asScala.toIndexedSeq)) + // TODO: For some reason the implicit wasn't working here when trying to use convert. + // So did it this way. Needs to be figured out + (ret map {a => new NDArray(a)}).asJava + } + + private def convert[B, A <% B](l: IndexedSeq[A]): IndexedSeq[B] = l map { a => a: B } +} diff --git a/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorOutputTest.java b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorOutputTest.java new file mode 100644 index 000000000..04041fcda --- /dev/null +++ b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorOutputTest.java @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi; + +import org.junit.Assert; +import org.junit.Test; + +public class ObjectDetectorOutputTest { + + private String predictedClassName = "lion"; + + private float delta = 0.00001f; + + @Test + public void testConstructor() { + + float[] arr = new float[]{0f, 1f, 2f, 3f, 4f}; + + ObjectDetectorOutput odOutput = new ObjectDetectorOutput(predictedClassName, arr); + + Assert.assertEquals(odOutput.getClassName(), predictedClassName); + Assert.assertEquals("Threshold not matching", odOutput.getProbability(), 0f, delta); + Assert.assertEquals("Threshold not matching", odOutput.getXMin(), 1f, delta); + Assert.assertEquals("Threshold not matching", odOutput.getXMax(), 2f, delta); + Assert.assertEquals("Threshold not matching", odOutput.getYMin(), 3f, delta); + Assert.assertEquals("Threshold not matching", odOutput.getYMax(), 4f, delta); + + } + + @Test (expected = ArrayIndexOutOfBoundsException.class) + public void testIncompleteArgsConstructor() { + + float[] arr = new float[]{0f, 1f}; + + ObjectDetectorOutput odOutput = new ObjectDetectorOutput(predictedClassName, arr); + + Assert.assertEquals(odOutput.getClassName(), predictedClassName); + Assert.assertEquals("Threshold not matching", odOutput.getProbability(), 0f, delta); + Assert.assertEquals("Threshold not matching", odOutput.getXMin(), 1f, delta); + + // This is where exception will be thrown + odOutput.getXMax(); + } +} diff --git a/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorTest.java b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorTest.java new file mode 100644 index 000000000..a5e64911d --- /dev/null +++ b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/ObjectDetectorTest.java @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi; + +import org.apache.mxnet.Layout; +import org.apache.mxnet.javaapi.DType; +import org.apache.mxnet.javaapi.DataDesc; +import org.apache.mxnet.javaapi.NDArray; +import org.apache.mxnet.javaapi.Shape; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +import java.awt.image.BufferedImage; +import java.util.ArrayList; +import java.util.List; + +public class ObjectDetectorTest { + + private List inputDesc; + private BufferedImage inputImage; + + private List> expectedResult; + + private ObjectDetector objectDetector; + + private int batchSize = 1; + + private int channels = 3; + + private int imageHeight = 512; + + private int imageWidth = 512; + + private String dataName = "data"; + + private int topK = 5; + + private String predictedClassName = "lion"; // Random string + + private Shape getTestShape() { + + return new Shape(new int[] {batchSize, channels, imageHeight, imageWidth}); + } + + @Before + public void setUp() { + + inputDesc = new ArrayList<>(); + inputDesc.add(new DataDesc(dataName, getTestShape(), DType.Float32(), Layout.NCHW())); + inputImage = new BufferedImage(imageWidth, imageHeight, BufferedImage.TYPE_INT_RGB); + objectDetector = Mockito.mock(ObjectDetector.class); + expectedResult = new ArrayList<>(); + expectedResult.add(new ArrayList()); + expectedResult.get(0).add(new ObjectDetectorOutput(predictedClassName, new float[]{})); + } + + @Test + public void testObjectDetectorWithInputImage() { + + Mockito.when(objectDetector.imageObjectDetect(inputImage, topK)).thenReturn(expectedResult); + List> actualResult = objectDetector.imageObjectDetect(inputImage, topK); + Mockito.verify(objectDetector, Mockito.times(1)).imageObjectDetect(inputImage, topK); + Assert.assertEquals(expectedResult, actualResult); + } + + + @Test + public void testObjectDetectorWithBatchImage() { + + List batchImage = new ArrayList<>(); + batchImage.add(inputImage); + Mockito.when(objectDetector.imageBatchObjectDetect(batchImage, topK)).thenReturn(expectedResult); + List> actualResult = objectDetector.imageBatchObjectDetect(batchImage, topK); + Mockito.verify(objectDetector, Mockito.times(1)).imageBatchObjectDetect(batchImage, topK); + Assert.assertEquals(expectedResult, actualResult); + } + + @Test + public void testObjectDetectorWithNDArrayInput() { + + NDArray inputArr = ObjectDetector.bufferedImageToPixels(inputImage, getTestShape()); + List inputL = new ArrayList<>(); + inputL.add(inputArr); + Mockito.when(objectDetector.objectDetectWithNDArray(inputL, 5)).thenReturn(expectedResult); + List> actualResult = objectDetector.objectDetectWithNDArray(inputL, topK); + Mockito.verify(objectDetector, Mockito.times(1)).objectDetectWithNDArray(inputL, topK); + Assert.assertEquals(expectedResult, actualResult); + } +} diff --git a/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/PredictorTest.java b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/PredictorTest.java new file mode 100644 index 000000000..e7a6c9652 --- /dev/null +++ b/scala-package/infer/src/test/java/org/apache/mxnet/infer/javaapi/PredictorTest.java @@ -0,0 +1,100 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.infer.javaapi; + +import org.apache.mxnet.javaapi.Context; +import org.apache.mxnet.javaapi.NDArray; +import org.apache.mxnet.javaapi.Shape; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +public class PredictorTest { + + Predictor mockPredictor; + + @Before + public void setUp() { + mockPredictor = Mockito.mock(Predictor.class); + } + + @Test + public void testPredictWithFloatArray() { + + float tmp[][] = new float[1][224]; + for (int x = 0; x < 1; x++) { + for (int y = 0; y < 224; y++) + tmp[x][y] = (int) (Math.random() * 10); + } + + float [][] expectedResult = new float[][] {{1f, 2f}}; + Mockito.when(mockPredictor.predict(tmp)).thenReturn(expectedResult); + float[][] actualResult = mockPredictor.predict(tmp); + + Mockito.verify(mockPredictor, Mockito.times(1)).predict(tmp); + Assert.assertArrayEquals(expectedResult, actualResult); + } + + @Test + public void testPredictWithNDArray() { + + float[] tmpArr = new float[224]; + for (int y = 0; y < 224; y++) + tmpArr[y] = (int) (Math.random() * 10); + + NDArray arr = new org.apache.mxnet.javaapi.NDArray(tmpArr, new Shape(new int[] {1, 1, 1, 224}), new Context("cpu", 0)); + + List inputList = new ArrayList<>(); + inputList.add(arr); + + NDArray expected = new NDArray(tmpArr, new Shape(new int[] {1, 1, 1, 224}), new Context("cpu", 0)); + List expectedResult = new ArrayList<>(); + expectedResult.add(expected); + + Mockito.when(mockPredictor.predictWithNDArray(inputList)).thenReturn(expectedResult); + + List actualOutput = mockPredictor.predictWithNDArray(inputList); + + Mockito.verify(mockPredictor, Mockito.times(1)).predictWithNDArray(inputList); + + Assert.assertEquals(expectedResult, actualOutput); + } + + @Test + public void testPredictWithListOfFloatsAsInput() { + List> input = new ArrayList<>(); + + input.add(Arrays.asList(new Float[] {1f, 2f})); + + List> expectedOutput = new ArrayList<>(input); + + Mockito.when(mockPredictor.predict(input)).thenReturn(expectedOutput); + + List> actualOutput = mockPredictor.predict(input); + + Mockito.verify(mockPredictor, Mockito.times(1)).predict(input); + + Assert.assertEquals(expectedOutput, actualOutput); + + } +} \ No newline at end of file diff --git a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ClassifierSuite.scala b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ClassifierSuite.scala index b28aeba1d..d9ccec468 100644 --- a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ClassifierSuite.scala +++ b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ClassifierSuite.scala @@ -22,7 +22,7 @@ import java.nio.file.{Files, Paths} import java.util import org.apache.mxnet.module.Module -import org.apache.mxnet.{Context, DataDesc, NDArray, Shape} +import org.apache.mxnet.{Context, DType, DataDesc, NDArray, Shape} import org.mockito.Matchers._ import org.mockito.Mockito import org.scalatest.{BeforeAndAfterAll, FunSuite} @@ -127,6 +127,29 @@ class ClassifierSuite extends FunSuite with BeforeAndAfterAll { } + test("ClassifierSuite-flatFloat64Array-topK") { + val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2))) + val inputData = Array.fill[Double](12)(1d) + + val predictResult : IndexedSeq[Array[Double]] = + IndexedSeq[Array[Double]](Array(.98d, 0.97d, 0.96d, 0.99d)) + + val testClassifier = new MyClassifier(modelPath, inputDescriptor) + + Mockito.doReturn(predictResult).when(testClassifier.predictor) + .predict(any(classOf[IndexedSeq[Array[Double]]])) + + val result: IndexedSeq[(String, Double)] = testClassifier. + classify(IndexedSeq(inputData), topK = Some(10)) + + assert((result(0)_2).getClass == 1d.getClass) + + assertResult(predictResult(0).sortBy(-_)) { + result.map(_._2).toArray + } + + } + test("ClassifierSuite-flatArrayInput") { val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2))) val inputData = Array.fill[Float](12)(1) @@ -147,6 +170,28 @@ class ClassifierSuite extends FunSuite with BeforeAndAfterAll { } } + test("ClassifierSuite-flatArrayFloat64Input") { + val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2))) + val inputData = Array.fill[Double](12)(1d) + + val predictResult : IndexedSeq[Array[Double]] = + IndexedSeq[Array[Double]](Array(.98d, 0.97d, 0.96d, 0.99d)) + + val testClassifier = new MyClassifier(modelPath, inputDescriptor) + + Mockito.doReturn(predictResult).when(testClassifier.predictor) + .predict(any(classOf[IndexedSeq[Array[Double]]])) + + val result: IndexedSeq[(String, Double)] = testClassifier. + classify(IndexedSeq(inputData)) + + assert((result(0)_2).getClass == 1d.getClass) + + assertResult(predictResult(0)) { + result.map(_._2).toArray + } + } + test("ClassifierSuite-NDArray1InputWithoutTopK") { val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2))) val inputDataShape = Shape(1, 3, 2, 2) diff --git a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ImageClassifierSuite.scala b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ImageClassifierSuite.scala index 1c291e1e7..5198c4a1f 100644 --- a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ImageClassifierSuite.scala +++ b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/ImageClassifierSuite.scala @@ -68,6 +68,10 @@ class ImageClassifierSuite extends ClassifierSuite with BeforeAndAfterAll { val result = ImageClassifier.bufferedImageToPixels(image2, Shape(3, 2, 2)) assert(result.shape == inputDescriptor(0).shape.drop(1)) + assert(result.dtype == DType.Float32) + + val resultFloat64 = ImageClassifier.bufferedImageToPixels(image2, Shape(3, 2, 2), DType.Float64) + assert(resultFloat64.dtype == DType.Float64) } test("ImageClassifierSuite-testWithInputImage") { @@ -106,8 +110,10 @@ class ImageClassifierSuite extends ClassifierSuite with BeforeAndAfterAll { predictResult(i).map(_._2).toArray } } + } + test("ImageClassifierSuite-testWithInputBatchImage") { val dType = DType.Float32 val inputDescriptor = IndexedSeq[DataDesc](new DataDesc(modelPath, Shape(1, 3, 512, 512), @@ -152,4 +158,5 @@ class ImageClassifierSuite extends ClassifierSuite with BeforeAndAfterAll { } } } + } diff --git a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/PredictorSuite.scala b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/PredictorSuite.scala index 509ffb35d..9afbc9b3d 100644 --- a/scala-package/infer/src/test/scala/org/apache/mxnet/infer/PredictorSuite.scala +++ b/scala-package/infer/src/test/scala/org/apache/mxnet/infer/PredictorSuite.scala @@ -19,7 +19,7 @@ package org.apache.mxnet.infer import org.apache.mxnet.io.NDArrayIter import org.apache.mxnet.module.{BaseModule, Module} -import org.apache.mxnet.{DataDesc, Layout, NDArray, Shape} +import org.apache.mxnet._ import org.mockito.Matchers._ import org.mockito.Mockito import org.scalatest.{BeforeAndAfterAll, FunSuite} @@ -91,6 +91,36 @@ class PredictorSuite extends FunSuite with BeforeAndAfterAll { , any[Option[BaseModule]], any[String]) } + test("PredictorSuite-testWithFlatFloat64Arrays") { + + val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2), + layout = Layout.NCHW, dtype = DType.Float64)) + val inputData = Array.fill[Double](12)(1d) + + // this will disposed at the end of the predict call on Predictor. + val predictResult = IndexedSeq(NDArray.ones(Shape(1, 3, 2, 2), dtype = DType.Float64)) + + val testPredictor = new MyPredictor("xyz", inputDescriptor) + + Mockito.doReturn(predictResult).when(testPredictor.mockModule) + .predict(any(classOf[NDArrayIter]), any[Int], any[Boolean]) + + val testFun = testPredictor.predict(IndexedSeq(inputData)) + + assert(testFun.size == 1, "output size should be 1 ") + + assert(testFun(0)(0).getClass == 1d.getClass) + + assert(Array.fill[Double](12)(1d).mkString == testFun(0).mkString) + + // Verify that the module was bound with batch size 1 and rebound back to the original + // input descriptor. the number of times is twice here because loadModule overrides the + // initial bind. + Mockito.verify(testPredictor.mockModule, Mockito.times(2)).bind(any[IndexedSeq[DataDesc]], + any[Option[IndexedSeq[DataDesc]]], any[Boolean], any[Boolean], any[Boolean] + , any[Option[BaseModule]], any[String]) + } + test("PredictorSuite-testWithNDArray") { val inputDescriptor = IndexedSeq[DataDesc](new DataDesc("data", Shape(2, 3, 2, 2), layout = Layout.NCHW)) diff --git a/scala-package/init-native/linux-x86_64/pom.xml b/scala-package/init-native/linux-x86_64/pom.xml deleted file mode 100644 index 811308e9d..000000000 --- a/scala-package/init-native/linux-x86_64/pom.xml +++ /dev/null @@ -1,106 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-scala-init-native-parent - 1.3.1-SNAPSHOT - ../pom.xml - - - libmxnet-init-scala-linux-x86_64 - MXNet Scala Package - Initializer Native Linux-x86_64 - http://maven.apache.org - - so - - - - org.apache.mxnet - mxnet-init_${scala.binary.version} - 1.3.1-SNAPSHOT - jar - compile - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-compiler-plugin - - - - org.codehaus.mojo - native-maven-plugin - true - - - linux - generic-classic - ${cxx} - ${cxx} - - - ../src/main/native - - org_apache_mxnet_init_native_c_api.cc - - - - - -std=c++0x - - - -I${project.basedir}/../../../include - ${all_includes} - ${cflags} - - - -shared - - - ${all_ldpaths} - -Wl,--whole-archive - ${lddeps} - -Wl,--no-whole-archive - - - ${ldflags} - -fopenmp - - - - - - javah - generate-sources - - linux - default - ${project.build.directory}/custom-javah - ${basedir} - org_apache_mxnet_init_native_c_api.h - - org.apache.mxnet.init.LibInfo - - - - javah - - - - - - - diff --git a/scala-package/init-native/osx-x86_64/pom.xml b/scala-package/init-native/osx-x86_64/pom.xml deleted file mode 100644 index 1ab17ea0e..000000000 --- a/scala-package/init-native/osx-x86_64/pom.xml +++ /dev/null @@ -1,105 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-scala-init-native-parent - 1.3.1-SNAPSHOT - ../pom.xml - - - libmxnet-init-scala-osx-x86_64 - MXNet Scala Package - Initializer Native OSX-x86_64 - http://maven.apache.org - - jnilib - - - - org.apache.mxnet - mxnet-init_${scala.binary.version} - 1.3.1-SNAPSHOT - jar - compile - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-compiler-plugin - - - - org.codehaus.mojo - native-maven-plugin - true - - - darwin - generic-classic - ${cxx} - ${cxx} - - - ../src/main/native - - org_apache_mxnet_init_native_c_api.cc - - - - - -std=c++0x - - - -I${project.basedir}/../../../include - ${cflags} - - - -shared - - - -framework JavaVM - -Wl,-exported_symbol,_Java_* - -Wl,-x - ${lddeps} - -force_load ${project.basedir}/../../../lib/libmxnet.a - - - ${ldflags} - - - - - - javah - generate-sources - - darwin - default - ${project.build.directory}/custom-javah - ${basedir} - org_apache_mxnet_init_native_c_api.h - - org.apache.mxnet.init.LibInfo - - - - javah - - - - - - - diff --git a/scala-package/init-native/pom.xml b/scala-package/init-native/pom.xml index 834a69946..1721f8cbd 100644 --- a/scala-package/init-native/pom.xml +++ b/scala-package/init-native/pom.xml @@ -5,46 +5,170 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-scala-init-native-parent - MXNet Scala Package - Initializer Native Parent - pom + libmxnet-init-scala + MXNet Scala Package - Initializer Native + + + ${project.parent.basedir}/.. + + + ${libtype} - osx-x86_64-cpu - - osx-x86_64 - - - - linux-x86_64-cpu - - linux-x86_64 - + osx-x86_64 + + mac + + + + + org.codehaus.mojo + native-maven-plugin + true + + darwin + generic-classic + ${cxx} + ${cxx} + + + src/main/native + + org_apache_mxnet_init_native_c_api.cc + + + + + -std=c++0x + + + -I${MXNET_DIR}/include + -I${MXNET_DIR}/3rdparty/dmlc-core/include + -I${MXNET_DIR}/3rdparty/mshadow + -I${MXNET_DIR}/3rdparty/dlpack/include + -I${MXNET_DIR}/3rdparty/tvm/nnvm/include + -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0 + -g -O0 -fPIC -msse3 -mf16c + -Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs + + + -shared + + + -framework JavaVM + -Wl,-exported_symbol,_Java_* + -Wl,-x + + + -lmxnet -L${MXNET_DIR}/lib + + + + + + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + post-native-build + package + + exec + + + install_name_tool + -add_rpath @loader_path ${project.build.directory}/${project.artifactId}.jnilib + + + + + + - linux-x86_64-gpu - - linux-x86_64 - + linux-x86_64 + + + unix + Linux + + + + + + org.codehaus.mojo + native-maven-plugin + true + + linux + generic-classic + ${cxx} + ${cxx} + + + src/main/native + + org_apache_mxnet_init_native_c_api.cc + + + + + -std=c++0x + + + -I${MXNET_DIR}/include + -I${MXNET_DIR}/3rdparty/dmlc-core/include + -I${MXNET_DIR}/3rdparty/mshadow + -I${MXNET_DIR}/3rdparty/dlpack/include + -I${MXNET_DIR}/3rdparty/tvm/nnvm/include + -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0 + -O3 -DNDEBUG=1 -fPIC -msse3 -mf16c + -Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs + + + -shared + + + -Wl,--whole-archive + -Wl,--no-whole-archive -pthread -lm -fopenmp -lrt + + + -Wl,-rpath=${dollar}ORIGIN -lmxnet -L${MXNET_DIR}/lib + + + + + - org.apache.maven.plugins - maven-deploy-plugin - - true - + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + link-native-lib + generate-resources + + exec + + + bash + -c 'ln -sf ${MXNET_DIR}/lib/* ${project.build.directory}/' + + + - diff --git a/scala-package/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h b/scala-package/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h new file mode 100644 index 000000000..6ff6ae6a1 --- /dev/null +++ b/scala-package/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h @@ -0,0 +1,45 @@ +/* DO NOT EDIT THIS FILE - it is machine generated */ +#include +/* Header for class org_apache_mxnet_init_LibInfo */ + +#ifndef _Included_org_apache_mxnet_init_LibInfo +#define _Included_org_apache_mxnet_init_LibInfo +#ifdef __cplusplus +extern "C" { +#endif +/* + * Class: org_apache_mxnet_init_LibInfo + * Method: mxSymbolListAtomicSymbolCreators + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_init_LibInfo_mxSymbolListAtomicSymbolCreators + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_init_LibInfo + * Method: mxSymbolGetAtomicSymbolInfo + * Signature: (JLorg/apache/mxnet/init/Base/RefString;Lorg/apache/mxnet/init/Base/RefString;Lorg/apache/mxnet/init/Base/RefInt;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lorg/apache/mxnet/init/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_init_LibInfo_mxSymbolGetAtomicSymbolInfo + (JNIEnv *, jobject, jlong, jobject, jobject, jobject, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_init_LibInfo + * Method: mxListAllOpNames + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_init_LibInfo_mxListAllOpNames + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_init_LibInfo + * Method: nnGetOpHandle + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/init/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_init_LibInfo_nnGetOpHandle + (JNIEnv *, jobject, jstring, jobject); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/scala-package/init/pom.xml b/scala-package/init/pom.xml index 80f883d7f..a0bb6be38 100644 --- a/scala-package/init/pom.xml +++ b/scala-package/init/pom.xml @@ -5,65 +5,62 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT - + mxnet-parent + INTERNAL + ../pom.xml - mxnet-init_2.11 + mxnet-scala-init MXNet Scala Package - Initializer - - - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - - apache-release - - - - - org.codehaus.mojo - exec-maven-plugin - 1.6.0 - - - compile-mxnet-backend - compile - - exec - - - bash - ${project.parent.basedir}/dev/compile-mxnet-backend.sh ${build.platform} ${project.parent.basedir}/../ - - - - - - org.apache.maven.plugins - maven-deploy-plugin + + + + org.codehaus.mojo + native-maven-plugin + true + + + javah + verify - true + default + ${project.build.directory}/custom-javah + ${basedir} + org_apache_mxnet_init_native_c_api.h + + org.apache.mxnet.init.LibInfo + - - - - - + + javah + + + +       + + + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + verify-javah + verify + + exec + + + diff + + ${project.build.directory}/custom-javah/org_apache_mxnet_init_native_c_api.h + ${project.parent.basedir}/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h + + + + + +       +       + diff --git a/scala-package/init/src/main/scala/org/apache/mxnet/init/Base.scala b/scala-package/init/src/main/scala/org/apache/mxnet/init/Base.scala index 7402dbd3b..b5a6286af 100644 --- a/scala-package/init/src/main/scala/org/apache/mxnet/init/Base.scala +++ b/scala-package/init/src/main/scala/org/apache/mxnet/init/Base.scala @@ -17,6 +17,8 @@ package org.apache.mxnet.init +import java.io.File + object Base { tryLoadInitLibrary() val _LIB = new LibInfo @@ -37,18 +39,22 @@ object Base { @throws(classOf[UnsatisfiedLinkError]) private def tryLoadInitLibrary(): Unit = { - var baseDir = System.getProperty("user.dir") + "/init-native" - // TODO(lanKing520) Update this to use relative path to the MXNet director. - // TODO(lanking520) baseDir = sys.env("MXNET_BASEDIR") + "/scala-package/init-native" - if (System.getenv().containsKey("MXNET_BASEDIR")) { - baseDir = sys.env("MXNET_BASEDIR") + var userDir : File = new File(System.getProperty("user.dir")) + var nativeDir : File = new File(userDir, "init-native") + if (!nativeDir.exists()) { + nativeDir = new File(userDir.getParent, "init-native") + if (!nativeDir.exists()) { + throw new IllegalStateException("scala-init should be executed inside scala-package folder") + } } + val baseDir = nativeDir.getAbsolutePath + val os = System.getProperty("os.name") // ref: http://lopica.sourceforge.net/os.html if (os.startsWith("Linux")) { - System.load(s"$baseDir/linux-x86_64/target/libmxnet-init-scala-linux-x86_64.so") + System.load(s"$baseDir/target/libmxnet-init-scala.so") } else if (os.startsWith("Mac")) { - System.load(s"$baseDir/osx-x86_64/target/libmxnet-init-scala-osx-x86_64.jnilib") + System.load(s"$baseDir/target/libmxnet-init-scala.jnilib") } else { // TODO(yizhi) support windows later throw new UnsatisfiedLinkError() diff --git a/scala-package/macros/pom.xml b/scala-package/macros/pom.xml index ae4c60a1e..52dfde181 100644 --- a/scala-package/macros/pom.xml +++ b/scala-package/macros/pom.xml @@ -5,63 +5,20 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-macros_2.11 + mxnet-macros MXNet Scala Package - Macros - - - unittest - - false - - - - integrationtest - - true - - - - osx-x86_64-cpu - - osx-x86_64 - jnilib - - - - linux-x86_64-cpu - - linux-x86_64 - so - - - - linux-x86_64-gpu - - linux-x86_64 - so - - - - org.apache.mxnet - mxnet-init_${scala.binary.version} - 1.3.1-SNAPSHOT - provided - - - org.apache.mxnet - libmxnet-init-scala-${platform} - 1.3.1-SNAPSHOT + mxnet-scala-init + INTERNAL provided - ${libtype} commons-io @@ -70,16 +27,8 @@ - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - org.apache.maven.plugins maven-jar-plugin @@ -95,39 +44,15 @@ org.apache.maven.plugins maven-compiler-plugin - - org.codehaus.mojo - exec-maven-plugin - 1.6.0 - - - apidoc-generation - package - - java - - - - - - ${project.parent.basedir}/init/target/classes - - - ${project.parent.basedir}/core/src/main/scala/org/apache/mxnet/ - - org.apache.mxnet.APIDocGenerator - - org.scalatest scalatest-maven-plugin - ${skiptest} ${project.parent.basedir}/init-native - -Djava.library.path=${project.parent.basedir}/native/${platform}/target \ + -Djava.library.path=${project.parent.basedir}/native/target \ -Dlog4j.configuration=file://${project.basedir}/src/test/resources/log4j.properties @@ -138,5 +63,4 @@ - diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala index b4efa6594..ede16f73d 100644 --- a/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/APIDocGenerator.scala @@ -17,178 +17,274 @@ package org.apache.mxnet -import org.apache.mxnet.init.Base._ -import org.apache.mxnet.utils.CToScalaUtils import java.io._ import java.security.MessageDigest -import scala.collection.mutable.{ArrayBuffer, ListBuffer} +import scala.collection.mutable.ListBuffer /** * This object will generate the Scala documentation of the new Scala API * Two file namely: SymbolAPIBase.scala and NDArrayAPIBase.scala * The code will be executed during Macros stage and file live in Core stage */ -private[mxnet] object APIDocGenerator{ - case class absClassArg(argName : String, argType : String, argDesc : String, isOptional : Boolean) - case class absClassFunction(name : String, desc : String, - listOfArgs: List[absClassArg], returnType : String) +private[mxnet] object APIDocGenerator extends GeneratorBase with RandomHelpers { - - def main(args: Array[String]) : Unit = { + def main(args: Array[String]): Unit = { val FILE_PATH = args(0) val hashCollector = ListBuffer[String]() - hashCollector += absClassGen(FILE_PATH, true) - hashCollector += absClassGen(FILE_PATH, false) + hashCollector += typeSafeClassGen(FILE_PATH, true) + hashCollector += typeSafeClassGen(FILE_PATH, false) + hashCollector += typeSafeRandomClassGen(FILE_PATH, true) + hashCollector += typeSafeRandomClassGen(FILE_PATH, false) hashCollector += nonTypeSafeClassGen(FILE_PATH, true) hashCollector += nonTypeSafeClassGen(FILE_PATH, false) + hashCollector += javaClassGen(FILE_PATH) val finalHash = hashCollector.mkString("\n") } - def MD5Generator(input : String) : String = { + def MD5Generator(input: String): String = { val md = MessageDigest.getInstance("MD5") md.update(input.getBytes("UTF-8")) val digest = md.digest() org.apache.commons.codec.binary.Base64.encodeBase64URLSafeString(digest) } - def absClassGen(FILE_PATH : String, isSymbol : Boolean) : String = { - // scalastyle:off - val absClassFunctions = getSymbolNDArrayMethods(isSymbol) - // Defines Operators that should not generated - val notGenerated = Set("Custom") - // TODO: Add Filter to the same location in case of refactor - val absFuncs = absClassFunctions.filterNot(_.name.startsWith("_")) - .filterNot(ele => notGenerated.contains(ele.name)) - .map(absClassFunction => { - val scalaDoc = generateAPIDocFromBackend(absClassFunction) - val defBody = generateAPISignature(absClassFunction, isSymbol) - s"$scalaDoc\n$defBody" - }) - val packageName = if (isSymbol) "SymbolAPIBase" else "NDArrayAPIBase" - val apacheLicence = "/*\n* Licensed to the Apache Software Foundation (ASF) under one or more\n* contributor license agreements. See the NOTICE file distributed with\n* this work for additional information regarding copyright ownership.\n* The ASF licenses this file to You under the Apache License, Version 2.0\n* (the \"License\"); you may not use this file except in compliance with\n* the License. You may obtain a copy of the License at\n*\n* http://www.apache.org/licenses/LICENSE-2.0\n*\n* Unless required by applicable law or agreed to in writing, software\n* distributed under the License is distributed on an \"AS IS\" BASIS,\n* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n* See the License for the specific language governing permissions and\n* limitations under the License.\n*/\n" - val scalaStyle = "// scalastyle:off" - val packageDef = "package org.apache.mxnet" - val imports = "import org.apache.mxnet.annotation.Experimental" - val absClassDef = s"abstract class $packageName" - val finalStr = s"$apacheLicence\n$scalaStyle\n$packageDef\n$imports\n$absClassDef {\n${absFuncs.mkString("\n")}\n}" - val pw = new PrintWriter(new File(FILE_PATH + s"$packageName.scala")) - pw.write(finalStr) - pw.close() - MD5Generator(finalStr) + def typeSafeClassGen(FILE_PATH: String, isSymbol: Boolean): String = { + val generated = typeSafeFunctionsToGenerate(isSymbol, isContrib = false) + .map { func => + val scalaDoc = generateAPIDocFromBackend(func) + val decl = generateAPISignature(func, isSymbol) + s"$scalaDoc\n$decl" + } + + writeFile( + FILE_PATH, + "package org.apache.mxnet", + if (isSymbol) "SymbolAPIBase" else "NDArrayAPIBase", + "import org.apache.mxnet.annotation.Experimental", + generated) } - def nonTypeSafeClassGen(FILE_PATH : String, isSymbol : Boolean) : String = { - // scalastyle:off - val absClassFunctions = getSymbolNDArrayMethods(isSymbol) - val absFuncs = absClassFunctions.map(absClassFunction => { - val scalaDoc = generateAPIDocFromBackend(absClassFunction, false) - if (isSymbol) { - val defBody = s"def ${absClassFunction.name}(name : String = null, attr : Map[String, String] = null)(args : org.apache.mxnet.Symbol*)(kwargs : Map[String, Any] = null): org.apache.mxnet.Symbol" - s"$scalaDoc\n$defBody" - } else { - val defBodyWithKwargs = s"def ${absClassFunction.name}(kwargs: Map[String, Any] = null)(args: Any*) : org.apache.mxnet.NDArrayFuncReturn" - val defBody = s"def ${absClassFunction.name}(args: Any*) : org.apache.mxnet.NDArrayFuncReturn" - s"$scalaDoc\n$defBodyWithKwargs\n$scalaDoc\n$defBody" + def typeSafeRandomClassGen(FILE_PATH: String, isSymbol: Boolean): String = { + val generated = typeSafeRandomFunctionsToGenerate(isSymbol) + .map { func => + val scalaDoc = generateAPIDocFromBackend(func) + val typeParameter = randomGenericTypeSpec(isSymbol, false) + val decl = generateAPISignature(func, isSymbol, typeParameter) + s"$scalaDoc\n$decl" } - }) - val packageName = if (isSymbol) "SymbolBase" else "NDArrayBase" - val apacheLicence = "/*\n* Licensed to the Apache Software Foundation (ASF) under one or more\n* contributor license agreements. See the NOTICE file distributed with\n* this work for additional information regarding copyright ownership.\n* The ASF licenses this file to You under the Apache License, Version 2.0\n* (the \"License\"); you may not use this file except in compliance with\n* the License. You may obtain a copy of the License at\n*\n* http://www.apache.org/licenses/LICENSE-2.0\n*\n* Unless required by applicable law or agreed to in writing, software\n* distributed under the License is distributed on an \"AS IS\" BASIS,\n* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n* See the License for the specific language governing permissions and\n* limitations under the License.\n*/\n" - val scalaStyle = "// scalastyle:off" - val packageDef = "package org.apache.mxnet" - val imports = "import org.apache.mxnet.annotation.Experimental" - val absClassDef = s"abstract class $packageName" - val finalStr = s"$apacheLicence\n$scalaStyle\n$packageDef\n$imports\n$absClassDef {\n${absFuncs.mkString("\n")}\n}" - import java.io._ - val pw = new PrintWriter(new File(FILE_PATH + s"$packageName.scala")) - pw.write(finalStr) - pw.close() - MD5Generator(finalStr) + + writeFile( + FILE_PATH, + "package org.apache.mxnet", + if (isSymbol) "SymbolRandomAPIBase" else "NDArrayRandomAPIBase", + """import org.apache.mxnet.annotation.Experimental + |import scala.reflect.ClassTag""".stripMargin, + generated) } - // Generate ScalaDoc type - def generateAPIDocFromBackend(func : absClassFunction, withParam : Boolean = true) : String = { - val desc = ArrayBuffer[String]() - desc += " * 
      "
-      func.desc.split("\n").foreach({ currStr =>
-      desc += s"  * $currStr"
-    })
-    desc += "  *
      " - val params = func.listOfArgs.map({ absClassArg => - val currArgName = absClassArg.argName match { - case "var" => "vari" - case "type" => "typeOf" - case _ => absClassArg.argName + def nonTypeSafeClassGen(FILE_PATH: String, isSymbol: Boolean): String = { + val absFuncs = functionsToGenerate(isSymbol, isContrib = false) + .map { func => + val scalaDoc = generateAPIDocFromBackend(func, false) + if (isSymbol) { + s"""$scalaDoc + |def ${func.name}(name : String = null, attr : Map[String, String] = null) + | (args : org.apache.mxnet.Symbol*)(kwargs : Map[String, Any] = null): + | org.apache.mxnet.Symbol + """.stripMargin + } else { + s"""$scalaDoc + |def ${func.name}(kwargs: Map[String, Any] = null) + | (args: Any*): org.apache.mxnet.NDArrayFuncReturn + | + |$scalaDoc + |def ${func.name}(args: Any*): org.apache.mxnet.NDArrayFuncReturn + """.stripMargin + } } - s" * @param $currArgName\t\t${absClassArg.argDesc}" - }) + + writeFile( + FILE_PATH, + "package org.apache.mxnet", + if (isSymbol) "SymbolBase" else "NDArrayBase", + "import org.apache.mxnet.annotation.Experimental", + absFuncs) + } + + def javaClassGen(filePath : String) : String = { + val notGenerated = Set("Custom") + val absClassFunctions = functionsToGenerate(false, false, true) + val absFuncs = absClassFunctions.filterNot(ele => notGenerated.contains(ele.name)) + .groupBy(_.name.toLowerCase).map(ele => { + /* Pattern matching for not generating deprecated method + * Group all method name in lowercase + * Kill the capital lettered method such as Cast vs cast + * As it defined by default it deprecated + */ + if (ele._2.length == 1) ele._2.head + else { + if (ele._2.head.name.head.isLower) ele._2.head + else ele._2.last + } + }).map(absClassFunction => { + generateJavaAPISignature(absClassFunction) + }).toSeq + val packageName = "NDArrayBase" + val packageDef = "package org.apache.mxnet.javaapi" + writeFile( + filePath + "javaapi/", + packageDef, + packageName, + "import org.apache.mxnet.annotation.Experimental", + absFuncs) + } + + def generateAPIDocFromBackend(func: Func, withParam: Boolean = true): String = { + def fixDesc(desc: String): String = { + var curDesc = desc + var prevDesc = "" + while ( curDesc != prevDesc ) { + prevDesc = curDesc + curDesc = curDesc.replace("[[", "`[ [").replace("]]", "] ]") + } + curDesc + } + val desc = fixDesc(func.desc).split("\n") + .mkString(" *\n * {{{\n *\n * ", "\n * ", "\n * }}}\n * ") + + val params = func.listOfArgs.map { absClassArg => + s" * @param ${absClassArg.safeArgName}\t\t${fixDesc(absClassArg.argDesc)}" + } + val returnType = s" * @return ${func.returnType}" + if (withParam) { - s" /**\n${desc.mkString("\n")}\n${params.mkString("\n")}\n$returnType\n */" + s""" /** + |$desc + |${params.mkString("\n")} + |$returnType + | */""".stripMargin + } else { + s""" /** + |$desc + |$returnType + | */""".stripMargin + } + } + + def generateAPISignature(func: Func, isSymbol: Boolean, typeParameter: String = ""): String = { + val argDef = ListBuffer[String]() + + argDef ++= typedFunctionCommonArgDef(func) + + if (isSymbol) { + argDef += "name : String = null" + argDef += "attr : Map[String, String] = null" } else { - s" /**\n${desc.mkString("\n")}\n$returnType\n */" + argDef += "out : Option[NDArray] = None" + } + + val returnType = func.returnType + + s"""@Experimental + |def ${func.name}$typeParameter (${argDef.mkString(", ")}): $returnType""".stripMargin } - def generateAPISignature(func : absClassFunction, isSymbol : Boolean) : String = { + def generateJavaAPISignature(func : Func) : String = { + val useParamObject = func.listOfArgs.count(arg => arg.isOptional) >= 2 var argDef = ListBuffer[String]() + var classDef = ListBuffer[String]() + var requiredParam = ListBuffer[String]() func.listOfArgs.foreach(absClassArg => { - val currArgName = absClassArg.argName match { - case "var" => "vari" - case "type" => "typeOf" - case _ => absClassArg.argName - } - if (absClassArg.isOptional) { - argDef += s"$currArgName : Option[${absClassArg.argType}] = None" + val currArgName = absClassArg.safeArgName + // scalastyle:off + if (absClassArg.isOptional && useParamObject) { + classDef += + s"""private var $currArgName: ${absClassArg.argType} = null + |/** + | * @param $currArgName\t\t${absClassArg.argDesc} + | */ + |def set${currArgName.capitalize}($currArgName : ${absClassArg.argType}): ${func.name}Param = { + | this.$currArgName = $currArgName + | this + | }""".stripMargin } else { + requiredParam += s" * @param $currArgName\t\t${absClassArg.argDesc}" argDef += s"$currArgName : ${absClassArg.argType}" } + classDef += s"def get${currArgName.capitalize}() = this.$currArgName" + // scalastyle:on }) - var returnType = func.returnType - if (isSymbol) { - argDef += "name : String = null" - argDef += "attr : Map[String, String] = null" + val experimentalTag = "@Experimental" + val returnType = "Array[NDArray]" + val scalaDoc = generateAPIDocFromBackend(func) + val scalaDocNoParam = generateAPIDocFromBackend(func, false) + if(useParamObject) { + classDef += + s"""private var out : org.apache.mxnet.NDArray = null + |def setOut(out : NDArray) : ${func.name}Param = { + | this.out = out + | this + | } + | def getOut() = this.out + | """.stripMargin + s"""$scalaDocNoParam + | $experimentalTag + | def ${func.name}(po: ${func.name}Param) : $returnType + | /** + | * This Param Object is specifically used for ${func.name} + | ${requiredParam.mkString("\n")} + | */ + | class ${func.name}Param(${argDef.mkString(",")}) { + | ${classDef.mkString("\n ")} + | }""".stripMargin } else { - argDef += "out : Option[NDArray] = None" - returnType = "org.apache.mxnet.NDArrayFuncReturn" + argDef += "out : NDArray" + s"""$scalaDoc + |$experimentalTag + | def ${func.name}(${argDef.mkString(", ")}) : $returnType + | """.stripMargin } - val experimentalTag = "@Experimental" - s"$experimentalTag\ndef ${func.name} (${argDef.mkString(", ")}) : $returnType" } + def writeFile(FILE_PATH: String, packageDef: String, className: String, + imports: String, absFuncs: Seq[String]): String = { - // List and add all the atomic symbol functions to current module. - private def getSymbolNDArrayMethods(isSymbol : Boolean): List[absClassFunction] = { - val opNames = ListBuffer.empty[String] - val returnType = if (isSymbol) "Symbol" else "NDArray" - _LIB.mxListAllOpNames(opNames) - // TODO: Add '_linalg_', '_sparse_', '_image_' support - // TODO: Add Filter to the same location in case of refactor - opNames.map(opName => { - val opHandle = new RefLong - _LIB.nnGetOpHandle(opName, opHandle) - makeAtomicSymbolFunction(opHandle.value, opName, "org.apache.mxnet." + returnType) - }).toList.filterNot(_.name.startsWith("_")) - } + val finalStr = + s"""/* + | * Licensed to the Apache Software Foundation (ASF) under one or more + | * contributor license agreements. See the NOTICE file distributed with + | * this work for additional information regarding copyright ownership. + | * The ASF licenses this file to You under the Apache License, Version 2.0 + | * (the "License"); you may not use this file except in compliance with + | * the License. You may obtain a copy of the License at + | * + | * http://www.apache.org/licenses/LICENSE-2.0 + | * + | * Unless required by applicable law or agreed to in writing, software + | * distributed under the License is distributed on an "AS IS" BASIS, + | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + | * See the License for the specific language governing permissions and + | * limitations under the License. + | */ + | + |$packageDef + | + |$imports + | + |// scalastyle:off + |abstract class $className { + |${absFuncs.mkString("\n")} + |}""".stripMargin - // Create an atomic symbol function by handle and function name. - private def makeAtomicSymbolFunction(handle: SymbolHandle, aliasName: String, returnType : String) - : absClassFunction = { - val name = new RefString - val desc = new RefString - val keyVarNumArgs = new RefString - val numArgs = new RefInt - val argNames = ListBuffer.empty[String] - val argTypes = ListBuffer.empty[String] - val argDescs = ListBuffer.empty[String] - - _LIB.mxSymbolGetAtomicSymbolInfo( - handle, name, desc, numArgs, argNames, argTypes, argDescs, keyVarNumArgs) - val argList = argNames zip argTypes zip argDescs map { case ((argName, argType), argDesc) => - val typeAndOption = CToScalaUtils.argumentCleaner(argName, argType, returnType) - new absClassArg(argName, typeAndOption._1, argDesc, typeAndOption._2) - } - new absClassFunction(aliasName, desc.value, argList.toList, returnType) + + val pw = new PrintWriter(new File(FILE_PATH + s"$className.scala")) + pw.write(finalStr) + pw.close() + MD5Generator(finalStr) } + } diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/GeneratorBase.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/GeneratorBase.scala new file mode 100644 index 000000000..498c4e943 --- /dev/null +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/GeneratorBase.scala @@ -0,0 +1,232 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet + +import org.apache.mxnet.init.Base.{RefInt, RefLong, RefString, _LIB} +import org.apache.mxnet.utils.{CToScalaUtils, OperatorBuildUtils} + +import scala.collection.mutable.ListBuffer +import scala.reflect.macros.blackbox + +private[mxnet] abstract class GeneratorBase { + type Handle = Long + + case class Arg(argName: String, argType: String, argDesc: String, isOptional: Boolean) { + def safeArgName: String = argName match { + case "var" => "vari" + case "type" => "typeOf" + case _ => argName + } + } + + case class Func(name: String, desc: String, listOfArgs: List[Arg], returnType: String) + + def functionsToGenerate(isSymbol: Boolean, isContrib: Boolean, + isJava: Boolean = false): List[Func] = { + val l = getBackEndFunctions(isSymbol, isJava) + if (isContrib) { + l.filter(func => func.name.startsWith("_contrib_") || !func.name.startsWith("_")) + } else { + l.filterNot(_.name.startsWith("_")) + } + } + + // filter the operators to generate in the type-safe Symbol.api and NDArray.api + protected def typeSafeFunctionsToGenerate(isSymbol: Boolean, isContrib: Boolean): List[Func] = { + // Operators that should not be generated + val notGenerated = Set("Custom") + + val l = getBackEndFunctions(isSymbol) + val res = if (isContrib) { + l.filter(func => func.name.startsWith("_contrib_") || !func.name.startsWith("_")) + } else { + l.filterNot(_.name.startsWith("_")) + } + res.filterNot(ele => notGenerated.contains(ele.name)) + } + + protected def getBackEndFunctions(isSymbol: Boolean, isJava: Boolean = false): List[Func] = { + val opNames = ListBuffer.empty[String] + _LIB.mxListAllOpNames(opNames) + opNames.map(opName => { + val opHandle = new RefLong + _LIB.nnGetOpHandle(opName, opHandle) + makeAtomicFunction(opHandle.value, opName, isSymbol, isJava) + }).toList + } + + private def makeAtomicFunction(handle: Handle, aliasName: String, + isSymbol: Boolean, isJava: Boolean): Func = { + val name = new RefString + val desc = new RefString + val keyVarNumArgs = new RefString + val numArgs = new RefInt + val argNames = ListBuffer.empty[String] + val argTypes = ListBuffer.empty[String] + val argDescs = ListBuffer.empty[String] + + _LIB.mxSymbolGetAtomicSymbolInfo( + handle, name, desc, numArgs, argNames, argTypes, argDescs, keyVarNumArgs) + val paramStr = OperatorBuildUtils.ctypes2docstring(argNames, argTypes, argDescs) + val extraDoc: String = if (keyVarNumArgs.value != null && keyVarNumArgs.value.length > 0) { + s"This function support variable length of positional input (${keyVarNumArgs.value})." + } else { + "" + } + val realName = if (aliasName == name.value) "" else s"(a.k.a., ${name.value})" + val docStr = s"$aliasName $realName\n${desc.value}\n\n$paramStr\n$extraDoc\n" + + val argList = argNames zip argTypes zip argDescs map { case ((argName, argType), argDesc) => + val family = if (isJava) "org.apache.mxnet.javaapi.NDArray" + else if (isSymbol) "org.apache.mxnet.Symbol" + else "org.apache.mxnet.NDArray" + val typeAndOption = + CToScalaUtils.argumentCleaner(argName, argType, family, isJava) + Arg(argName, typeAndOption._1, argDesc, typeAndOption._2) + } + val returnType = + if (isJava) "Array[org.apache.mxnet.javaapi.NDArray]" + else if (isSymbol) "org.apache.mxnet.Symbol" + else "org.apache.mxnet.NDArrayFuncReturn" + Func(aliasName, desc.value, argList.toList, returnType) + } + + /** + * Generate class structure for all function APIs + * + * @param c + * @param funcDef DefDef type of function definitions + * @param annottees + * @return + */ + protected def structGeneration(c: blackbox.Context) + (funcDef: List[c.universe.DefDef], annottees: c.Expr[Any]*) + : c.Expr[Any] = { + import c.universe._ + val inputs = annottees.map(_.tree).toList + // pattern match on the inputs + val modDefs = inputs map { + case ClassDef(mods, name, something, template) => + val q = template match { + case Template(superMaybe, emptyValDef, defs) => + Template(superMaybe, emptyValDef, defs ++ funcDef) + case ex => + throw new IllegalArgumentException(s"Invalid template: $ex") + } + ClassDef(mods, name, something, q) + case ModuleDef(mods, name, template) => + val q = template match { + case Template(superMaybe, emptyValDef, defs) => + Template(superMaybe, emptyValDef, defs ++ funcDef) + case ex => + throw new IllegalArgumentException(s"Invalid template: $ex") + } + ModuleDef(mods, name, q) + case ex => + throw new IllegalArgumentException(s"Invalid macro input: $ex") + } + // wrap the result up in an Expr, and return it + val result = c.Expr(Block(modDefs, Literal(Constant()))) + result + } + + // build function argument definition, with optionality, and safe names + protected def typedFunctionCommonArgDef(func: Func): List[String] = { + func.listOfArgs.map(arg => + if (arg.isOptional) { + // let's avoid a stupid Option[Array[...]] + if (arg.argType.startsWith("Array[")) { + s"${arg.safeArgName} : ${arg.argType} = Array.empty" + } else { + s"${arg.safeArgName} : Option[${arg.argType}] = None" + } + } + else { + s"${arg.safeArgName} : ${arg.argType}" + } + ) + } +} + +// a mixin to ease generating the Random module +private[mxnet] trait RandomHelpers { + self: GeneratorBase => + + // a generic type spec used in Symbol.random and NDArray.random modules + protected def randomGenericTypeSpec(isSymbol: Boolean, fullPackageSpec: Boolean): String = { + val classTag = if (fullPackageSpec) "scala.reflect.ClassTag" else "ClassTag" + if (isSymbol) s"[T: SymbolOrScalar : $classTag]" + else s"[T: NDArrayOrScalar : $classTag]" + } + + // filter the operators to generate in the type-safe Symbol.random and NDArray.random + protected def typeSafeRandomFunctionsToGenerate(isSymbol: Boolean): List[Func] = { + getBackEndFunctions(isSymbol) + .filter(f => f.name.startsWith("_sample_") || f.name.startsWith("_random_")) + .map(f => f.copy(name = f.name.stripPrefix("_"))) + // unify _random and _sample + .map(f => unifyRandom(f, isSymbol)) + // deduplicate + .groupBy(_.name) + .mapValues(_.head) + .values + .toList + } + + // unify call targets (random_xyz and sample_xyz) and unify their argument types + private def unifyRandom(func: Func, isSymbol: Boolean): Func = { + var typeConv = Set("org.apache.mxnet.NDArray", "org.apache.mxnet.Symbol", + "Float", "Int") + + func.copy( + name = func.name.replaceAll("(random|sample)_", ""), + listOfArgs = func.listOfArgs + .map(hackNormalFunc) + .map(arg => + if (typeConv(arg.argType)) arg.copy(argType = "T") + else arg + ) + // TODO: some functions are non consistent in random_ vs sample_ regarding optionality + // we may try to unify that as well here. + ) + } + + // hacks to manage the fact that random_normal and sample_normal have + // non-consistent parameter naming in the back-end + // this first one, merge loc/scale and mu/sigma + protected def hackNormalFunc(arg: Arg): Arg = { + if (arg.argName == "loc") arg.copy(argName = "mu") + else if (arg.argName == "scale") arg.copy(argName = "sigma") + else arg + } + + // this second one reverts this merge prior to back-end call + protected def unhackNormalFunc(func: Func): String = { + if (func.name.equals("normal")) { + s"""if(target.equals("random_normal")) { + | if(map.contains("mu")) { map("loc") = map("mu"); map.remove("mu") } + | if(map.contains("sigma")) { map("scale") = map("sigma"); map.remove("sigma") } + |} + """.stripMargin + } else { + "" + } + + } + +} diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala index 2d3a1c7ec..c18694b59 100644 --- a/scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/NDArrayMacro.scala @@ -17,11 +17,7 @@ package org.apache.mxnet -import org.apache.mxnet.init.Base._ -import org.apache.mxnet.utils.{CToScalaUtils, OperatorBuildUtils} - import scala.annotation.StaticAnnotation -import scala.collection.mutable.{ArrayBuffer, ListBuffer} import scala.language.experimental.macros import scala.reflect.macros.blackbox @@ -30,207 +26,209 @@ private[mxnet] class AddNDArrayFunctions(isContrib: Boolean) extends StaticAnnot } private[mxnet] class AddNDArrayAPIs(isContrib: Boolean) extends StaticAnnotation { - private[mxnet] def macroTransform(annottees: Any*) = macro NDArrayMacro.typeSafeAPIDefs + private[mxnet] def macroTransform(annottees: Any*) = macro TypedNDArrayAPIMacro.typeSafeAPIDefs } -private[mxnet] object NDArrayMacro { - case class NDArrayArg(argName: String, argType: String, isOptional : Boolean) - case class NDArrayFunction(name: String, listOfArgs: List[NDArrayArg]) - - // scalastyle:off havetype - def addDefs(c: blackbox.Context)(annottees: c.Expr[Any]*) = { - impl(c)(annottees: _*) - } - def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*) = { - typeSafeAPIImpl(c)(annottees: _*) - } - // scalastyle:off havetype +private[mxnet] class AddNDArrayRandomAPIs(isContrib: Boolean) extends StaticAnnotation { + private[mxnet] def macroTransform(annottees: Any*) = + macro TypedNDArrayRandomAPIMacro.typeSafeAPIDefs +} - private val ndarrayFunctions: List[NDArrayFunction] = initNDArrayModule() +/** + * For non-typed NDArray API + */ +private[mxnet] object NDArrayMacro extends GeneratorBase { - private def impl(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + def addDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { import c.universe._ - val isContrib: Boolean = c.prefix.tree match { case q"new AddNDArrayFunctions($b)" => c.eval[Boolean](c.Expr(b)) } - val newNDArrayFunctions = { - if (isContrib) ndarrayFunctions.filter(_.name.startsWith("_contrib_")) - else ndarrayFunctions.filterNot(_.name.startsWith("_")) - } + impl(c)(isContrib, annottees: _*) + } - val functionDefs = newNDArrayFunctions flatMap { NDArrayfunction => - val funcName = NDArrayfunction.name - val termName = TermName(funcName) - Seq( - // scalastyle:off - // (yizhi) We are investigating a way to make these functions type-safe - // and waiting to see the new approach is stable enough. - // Thus these functions may be deprecated in the future. - // e.g def transpose(kwargs: Map[String, Any] = null)(args: Any*) - q"def $termName(kwargs: Map[String, Any] = null)(args: Any*) = {genericNDArrayFunctionInvoke($funcName, args, kwargs)}".asInstanceOf[DefDef], - // e.g def transpose(args: Any*) - q"def $termName(args: Any*) = {genericNDArrayFunctionInvoke($funcName, args, null)}".asInstanceOf[DefDef] - // scalastyle:on - ) - } + private def impl(c: blackbox.Context) + (isContrib: Boolean, annottees: c.Expr[Any]*): c.Expr[Any] = { + import c.universe._ + + val functions = functionsToGenerate(isSymbol = false, isContrib) + + val functionDefs = functions.flatMap { NDArrayfunction => + val funcName = NDArrayfunction.name + val termName = TermName(funcName) + Seq( + // e.g def transpose(kwargs: Map[String, Any] = null)(args: Any*) + q""" + def $termName(kwargs: Map[String, Any] = null)(args: Any*) = { + genericNDArrayFunctionInvoke($funcName, args, kwargs) + } + """.asInstanceOf[DefDef], + // e.g def transpose(args: Any*) + q""" + def $termName(args: Any*) = { + genericNDArrayFunctionInvoke($funcName, args, null) + } + """.asInstanceOf[DefDef] + ) + } - structGeneration(c)(functionDefs, annottees : _*) + structGeneration(c)(functionDefs, annottees: _*) } +} - private def typeSafeAPIImpl(c: blackbox.Context)(annottees: c.Expr[Any]*) : c.Expr[Any] = { - import c.universe._ +/** + * NDArray.api code generation + */ +private[mxnet] object TypedNDArrayAPIMacro extends GeneratorBase { + def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + import c.universe._ val isContrib: Boolean = c.prefix.tree match { case q"new AddNDArrayAPIs($b)" => c.eval[Boolean](c.Expr(b)) } - // Defines Operators that should not generated - val notGenerated = Set("Custom") - - val newNDArrayFunctions = { - if (isContrib) ndarrayFunctions.filter( - func => func.name.startsWith("_contrib_") || !func.name.startsWith("_")) - else ndarrayFunctions.filterNot(_.name.startsWith("_")) - }.filterNot(ele => notGenerated.contains(ele.name)) - - val functionDefs = newNDArrayFunctions.map { ndarrayfunction => - - // Construct argument field - var argDef = ListBuffer[String]() - // Construct Implementation field - var impl = ListBuffer[String]() - impl += "val map = scala.collection.mutable.Map[String, Any]()" - impl += "val args = scala.collection.mutable.ArrayBuffer.empty[NDArray]" - ndarrayfunction.listOfArgs.foreach({ ndarrayarg => - // var is a special word used to define variable in Scala, - // need to changed to something else in order to make it work - val currArgName = ndarrayarg.argName match { - case "var" => "vari" - case "type" => "typeOf" - case default => ndarrayarg.argName - } - if (ndarrayarg.isOptional) { - argDef += s"${currArgName} : Option[${ndarrayarg.argType}] = None" - } - else { - argDef += s"${currArgName} : ${ndarrayarg.argType}" - } - // NDArray arg implementation - val returnType = "org.apache.mxnet.NDArray" - - // TODO: Currently we do not add place holder for NDArray - // Example: an NDArray operator like the following format - // nd.foo(arg1: NDArray(required), arg2: NDArray(Optional), arg3: NDArray(Optional) - // If we place nd.foo(arg1, arg3 = arg3), do we need to add place holder for arg2? - // What it should be? - val base = - if (ndarrayarg.argType.equals(returnType)) { - s"args += $currArgName" - } else if (ndarrayarg.argType.equals(s"Array[$returnType]")){ - s"args ++= $currArgName" - } else { - "map(\"" + ndarrayarg.argName + "\") = " + currArgName - } - impl.append( - if (ndarrayarg.isOptional) s"if (!$currArgName.isEmpty) $base.get" - else base - ) - }) - // add default out parameter - argDef += "out : Option[NDArray] = None" - impl += "if (!out.isEmpty) map(\"out\") = out.get" - // scalastyle:off - impl += "org.apache.mxnet.NDArray.genericNDArrayFunctionInvoke(\"" + ndarrayfunction.name + "\", args.toSeq, map.toMap)" - // scalastyle:on - // Combine and build the function string - val returnType = "org.apache.mxnet.NDArrayFuncReturn" - var finalStr = s"def ${ndarrayfunction.name}" - finalStr += s" (${argDef.mkString(",")}) : $returnType" - finalStr += s" = {${impl.mkString("\n")}}" - c.parse(finalStr).asInstanceOf[DefDef] - } - structGeneration(c)(functionDefs, annottees : _*) + val functionDefs = typeSafeFunctionsToGenerate(isSymbol = false, isContrib) + .map(f => buildTypedFunction(c)(f)) + + structGeneration(c)(functionDefs, annottees: _*) } - private def structGeneration(c: blackbox.Context) - (funcDef : List[c.universe.DefDef], annottees: c.Expr[Any]*) - : c.Expr[Any] = { + protected def buildTypedFunction(c: blackbox.Context) + (function: Func): c.universe.DefDef = { import c.universe._ - val inputs = annottees.map(_.tree).toList - // pattern match on the inputs - val modDefs = inputs map { - case ClassDef(mods, name, something, template) => - val q = template match { - case Template(superMaybe, emptyValDef, defs) => - Template(superMaybe, emptyValDef, defs ++ funcDef) - case ex => - throw new IllegalArgumentException(s"Invalid template: $ex") - } - ClassDef(mods, name, something, q) - case ModuleDef(mods, name, template) => - val q = template match { - case Template(superMaybe, emptyValDef, defs) => - Template(superMaybe, emptyValDef, defs ++ funcDef) - case ex => - throw new IllegalArgumentException(s"Invalid template: $ex") + + val returnType = "org.apache.mxnet.NDArrayFuncReturn" + + // Construct API arguments declaration + val argDecl = super.typedFunctionCommonArgDef(function) :+ "out : Option[NDArray] = None" + + // Map API input args to backend args + val backendArgsMapping = + function.listOfArgs.map { arg => + // ndarrays go to args, other types go to kwargs + if (arg.argType.equals(s"Array[org.apache.mxnet.NDArray]")) { + s"args ++= ${arg.safeArgName}.toSeq" + } else { + val base = if (arg.argType.equals("org.apache.mxnet.NDArray")) { + s"args += ${arg.safeArgName}" + } else { + s"""map("${arg.argName}") = ${arg.safeArgName}""" + } + if (arg.isOptional) s"if (!${arg.safeArgName}.isEmpty) $base.get" + else base } - ModuleDef(mods, name, q) - case ex => - throw new IllegalArgumentException(s"Invalid macro input: $ex") - } - // wrap the result up in an Expr, and return it - val result = c.Expr(Block(modDefs, Literal(Constant()))) - result + } + + val impl = + s""" + |def ${function.name} + | (${argDecl.mkString(",")}): $returnType = { + | + | val map = scala.collection.mutable.Map[String, Any]() + | val args = scala.collection.mutable.ArrayBuffer.empty[org.apache.mxnet.NDArray] + | + | if (!out.isEmpty) map("out") = out.get + | + | ${backendArgsMapping.mkString("\n")} + | + | org.apache.mxnet.NDArray.genericNDArrayFunctionInvoke( + | "${function.name}", args.toSeq, map.toMap) + |} + """.stripMargin + + c.parse(impl).asInstanceOf[DefDef] } +} + +/** + * NDArray.random code generation + */ +private[mxnet] object TypedNDArrayRandomAPIMacro extends GeneratorBase + with RandomHelpers { + def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + // Note: no contrib managed in this module + val functionDefs = typeSafeRandomFunctionsToGenerate(isSymbol = false) + .map(f => buildTypedFunction(c)(f)) - // List and add all the atomic symbol functions to current module. - private def initNDArrayModule(): List[NDArrayFunction] = { - val opNames = ListBuffer.empty[String] - _LIB.mxListAllOpNames(opNames) - opNames.map(opName => { - val opHandle = new RefLong - _LIB.nnGetOpHandle(opName, opHandle) - makeNDArrayFunction(opHandle.value, opName) - }).toList + structGeneration(c)(functionDefs, annottees: _*) } - // Create an atomic symbol function by handle and function name. - private def makeNDArrayFunction(handle: NDArrayHandle, aliasName: String) - : NDArrayFunction = { - val name = new RefString - val desc = new RefString - val keyVarNumArgs = new RefString - val numArgs = new RefInt - val argNames = ListBuffer.empty[String] - val argTypes = ListBuffer.empty[String] - val argDescs = ListBuffer.empty[String] - - _LIB.mxSymbolGetAtomicSymbolInfo( - handle, name, desc, numArgs, argNames, argTypes, argDescs, keyVarNumArgs) - val paramStr = OperatorBuildUtils.ctypes2docstring(argNames, argTypes, argDescs) - val extraDoc: String = if (keyVarNumArgs.value != null && keyVarNumArgs.value.length > 0) { - s"This function support variable length of positional input (${keyVarNumArgs.value})." - } else { - "" - } - val realName = if (aliasName == name.value) "" else s"(a.k.a., ${name.value})" - val docStr = s"$aliasName $realName\n${desc.value}\n\n$paramStr\n$extraDoc\n" - // scalastyle:off println - if (System.getenv("MXNET4J_PRINT_OP_DEF") != null - && System.getenv("MXNET4J_PRINT_OP_DEF").toLowerCase == "true") { - println("NDArray function definition:\n" + docStr) - } - // scalastyle:on println - val argList = argNames zip argTypes map { case (argName, argType) => - val typeAndOption = - CToScalaUtils.argumentCleaner(argName, argType, "org.apache.mxnet.NDArray") - new NDArrayArg(argName, typeAndOption._1, typeAndOption._2) - } - new NDArrayFunction(aliasName, argList.toList) + protected def buildTypedFunction(c: blackbox.Context) + (function: Func): c.universe.DefDef = { + import c.universe._ + + val returnType = "org.apache.mxnet.NDArrayFuncReturn" + + // Construct API arguments declaration + val argDecl = super.typedFunctionCommonArgDef(function) :+ "out : Option[NDArray] = None" + + // Map API input args to backend args + val backendArgsMapping = + function.listOfArgs.map { arg => + // ndarrays go to args, other types go to kwargs + if (arg.argType.equals("Array[org.apache.mxnet.NDArray]")) { + s"args ++= ${arg.safeArgName}.toSeq" + } else { + if (arg.argType.equals("T")) { + if (arg.isOptional) { + s"""if(${arg.safeArgName}.isDefined) { + | if(isScalar) { + | map("${arg.argName}") = ${arg.safeArgName}.get + | } else { + | args += ${arg.safeArgName}.get.asInstanceOf[org.apache.mxnet.NDArray] + | } + |} + """.stripMargin + } else { + s"""if(isScalar) { + | map("${arg.argName}") = ${arg.safeArgName} + |} else { + | args += ${arg.safeArgName}.asInstanceOf[org.apache.mxnet.NDArray] + |} + """.stripMargin + } + } else { + if (arg.isOptional) { + s"""if (${arg.safeArgName}.isDefined) map("${arg.argName}")=${arg.safeArgName}.get""" + } else { + s"""map("${arg.argName}") = ${arg.safeArgName}""" + } + } + } + } + + val impl = + s""" + |def ${function.name}${randomGenericTypeSpec(false, true)} + | (${argDecl.mkString(",")}): $returnType = { + | + | val map = scala.collection.mutable.Map[String, Any]() + | val args = scala.collection.mutable.ArrayBuffer.empty[org.apache.mxnet.NDArray] + | val isScalar = NDArrayOrScalar[T].isScalar + | + | if(out.isDefined) map("out") = out.get + | + | ${backendArgsMapping.mkString("\n")} + | + | val target = if(isScalar) { + | "random_${function.name}" + | } else { + | "sample_${function.name}" + | } + | + | ${unhackNormalFunc(function)} + | + | org.apache.mxnet.NDArray.genericNDArrayFunctionInvoke( + | target, args.toSeq, map.toMap) + |} + """.stripMargin + + c.parse(impl).asInstanceOf[DefDef] } + + } diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/SymbolMacro.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/SymbolMacro.scala index 42aa11781..7ec80b9c0 100644 --- a/scala-package/macros/src/main/scala/org/apache/mxnet/SymbolMacro.scala +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/SymbolMacro.scala @@ -17,226 +17,187 @@ package org.apache.mxnet + import scala.annotation.StaticAnnotation -import scala.collection.mutable.ListBuffer import scala.language.experimental.macros import scala.reflect.macros.blackbox -import org.apache.mxnet.init.Base._ -import org.apache.mxnet.utils.{CToScalaUtils, OperatorBuildUtils} private[mxnet] class AddSymbolFunctions(isContrib: Boolean) extends StaticAnnotation { - private[mxnet] def macroTransform(annottees: Any*) = macro SymbolImplMacros.addDefs + private[mxnet] def macroTransform(annottees: Any*) = macro SymbolMacro.addDefs } private[mxnet] class AddSymbolAPIs(isContrib: Boolean) extends StaticAnnotation { - private[mxnet] def macroTransform(annottees: Any*) = macro SymbolImplMacros.typeSafeAPIDefs + private[mxnet] def macroTransform(annottees: Any*) = macro TypedSymbolAPIMacro.typeSafeAPIDefs } -private[mxnet] object SymbolImplMacros { - case class SymbolArg(argName: String, argType: String, isOptional : Boolean) - case class SymbolFunction(name: String, listOfArgs: List[SymbolArg]) - - // scalastyle:off havetype - def addDefs(c: blackbox.Context)(annottees: c.Expr[Any]*) = { - impl(c)(annottees: _*) - } - def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*) = { - typedAPIImpl(c)(annottees: _*) - } - // scalastyle:on havetype +private[mxnet] class AddSymbolRandomAPIs(isContrib: Boolean) extends StaticAnnotation { + private[mxnet] def macroTransform(annottees: Any*) = + macro TypedSymbolRandomAPIMacro.typeSafeAPIDefs +} - private val symbolFunctions: List[SymbolFunction] = initSymbolModule() +/** + * For non-typed Symbol API + */ +private[mxnet] object SymbolMacro extends GeneratorBase { - /** - * Implementation for fixed input API structure - */ - private def impl(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + def addDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { import c.universe._ - val isContrib: Boolean = c.prefix.tree match { case q"new AddSymbolFunctions($b)" => c.eval[Boolean](c.Expr(b)) } - val newSymbolFunctions = { - if (isContrib) symbolFunctions.filter( - func => func.name.startsWith("_contrib_") || !func.name.startsWith("_")) - else symbolFunctions.filter(!_.name.startsWith("_")) - } + impl(c)(isContrib, annottees: _*) + } + + private def impl(c: blackbox.Context) + (isContrib: Boolean, annottees: c.Expr[Any]*): c.Expr[Any] = { + import c.universe._ + val functions = functionsToGenerate(isSymbol = false, isContrib) - val functionDefs = newSymbolFunctions map { symbolfunction => - val funcName = symbolfunction.name - val tName = TermName(funcName) - q""" + val functionDefs = functions.map { symbolfunction => + val funcName = symbolfunction.name + val tName = TermName(funcName) + q""" def $tName(name : String = null, attr : Map[String, String] = null) - (args : org.apache.mxnet.Symbol*)(kwargs : Map[String, Any] = null) - : org.apache.mxnet.Symbol = { - createSymbolGeneral($funcName,name,attr,args,kwargs) - } + (args : org.apache.mxnet.Symbol*)(kwargs : Map[String, Any] = null) + : org.apache.mxnet.Symbol = { + createSymbolGeneral($funcName,name,attr,args,kwargs) + } """.asInstanceOf[DefDef] - } + } - structGeneration(c)(functionDefs, annottees : _*) + structGeneration(c)(functionDefs, annottees: _*) } +} - /** - * Implementation for Dynamic typed API Symbol.api. - */ - private def typedAPIImpl(c: blackbox.Context)(annottees: c.Expr[Any]*) : c.Expr[Any] = { - import c.universe._ +/** + * Symbol.api code generation + */ +private[mxnet] object TypedSymbolAPIMacro extends GeneratorBase { + def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + import c.universe._ val isContrib: Boolean = c.prefix.tree match { case q"new AddSymbolAPIs($b)" => c.eval[Boolean](c.Expr(b)) } - // Defines Operators that should not generated - val notGenerated = Set("Custom") - - // TODO: Put Symbol.api.foo --> Stable APIs - // Symbol.contrib.bar--> Contrib APIs - val newSymbolFunctions = { - if (isContrib) symbolFunctions.filter( - func => func.name.startsWith("_contrib_") || !func.name.startsWith("_")) - else symbolFunctions.filter(!_.name.startsWith("_")) - }.filterNot(ele => notGenerated.contains(ele.name)) - - val functionDefs = newSymbolFunctions map { symbolfunction => - - // Construct argument field - var argDef = ListBuffer[String]() - // Construct Implementation field - var impl = ListBuffer[String]() - impl += "val map = scala.collection.mutable.Map[String, Any]()" - impl += "var args = Seq[org.apache.mxnet.Symbol]()" - symbolfunction.listOfArgs.foreach({ symbolarg => - // var is a special word used to define variable in Scala, - // need to changed to something else in order to make it work - val currArgName = symbolarg.argName match { - case "var" => "vari" - case "type" => "typeOf" - case default => symbolarg.argName - } - if (symbolarg.isOptional) { - argDef += s"${currArgName} : Option[${symbolarg.argType}] = None" - } - else { - argDef += s"${currArgName} : ${symbolarg.argType}" - } - // Symbol arg implementation - val returnType = "org.apache.mxnet.Symbol" - val base = - if (symbolarg.argType.equals(s"Array[$returnType]")) { - if (symbolarg.isOptional) s"if (!$currArgName.isEmpty) args = $currArgName.get.toSeq" - else s"args = $currArgName.toSeq" - } else { - if (symbolarg.isOptional) { - // scalastyle:off - s"if (!$currArgName.isEmpty) map(" + "\"" + symbolarg.argName + "\"" + s") = $currArgName.get" - // scalastyle:on - } - else "map(\"" + symbolarg.argName + "\"" + s") = $currArgName" - } + val functionDefs = typeSafeFunctionsToGenerate(isSymbol = true, isContrib) + .map(f => buildTypedFunction(c)(f)) - impl += base - }) - argDef += "name : String = null" - argDef += "attr : Map[String, String] = null" - // scalastyle:off - // TODO: Seq() here allows user to place Symbols rather than normal arguments to run, need to fix if old API deprecated - impl += "org.apache.mxnet.Symbol.createSymbolGeneral(\"" + symbolfunction.name + "\", name, attr, args, map.toMap)" - // scalastyle:on - // Combine and build the function string - val returnType = "org.apache.mxnet.Symbol" - var finalStr = s"def ${symbolfunction.name}" - finalStr += s" (${argDef.mkString(",")}) : $returnType" - finalStr += s" = {${impl.mkString("\n")}}" - c.parse(finalStr).asInstanceOf[DefDef] - } - structGeneration(c)(functionDefs, annottees : _*) + structGeneration(c)(functionDefs, annottees: _*) } - /** - * Generate class structure for all function APIs - * @param c - * @param funcDef DefDef type of function definitions - * @param annottees - * @return - */ - private def structGeneration(c: blackbox.Context) - (funcDef : List[c.universe.DefDef], annottees: c.Expr[Any]*) - : c.Expr[Any] = { + protected def buildTypedFunction(c: blackbox.Context) + (function: Func): c.universe.DefDef = { import c.universe._ - val inputs = annottees.map(_.tree).toList - // pattern match on the inputs - val modDefs = inputs map { - case ClassDef(mods, name, something, template) => - val q = template match { - case Template(superMaybe, emptyValDef, defs) => - Template(superMaybe, emptyValDef, defs ++ funcDef) - case ex => - throw new IllegalArgumentException(s"Invalid template: $ex") - } - ClassDef(mods, name, something, q) - case ModuleDef(mods, name, template) => - val q = template match { - case Template(superMaybe, emptyValDef, defs) => - Template(superMaybe, emptyValDef, defs ++ funcDef) - case ex => - throw new IllegalArgumentException(s"Invalid template: $ex") + + val returnType = "org.apache.mxnet.Symbol" + + // Construct API arguments declaration + val argDecl = super.typedFunctionCommonArgDef(function) :+ + "name : String = null" :+ + "attr : Map[String, String] = null" + + // Map API input args to backend args + val backendArgsMapping = + function.listOfArgs.map { arg => + if (arg.argType.equals(s"Array[org.apache.mxnet.Symbol]")) { + s"args = ${arg.safeArgName}.toSeq" + } else { + // all go in kwargs + if (arg.isOptional) { + s"""if (!${arg.safeArgName}.isEmpty) map("${arg.argName}") = ${arg.safeArgName}.get""" + } else { + s"""map("${arg.argName}") = ${arg.safeArgName}""" + } } - ModuleDef(mods, name, q) - case ex => - throw new IllegalArgumentException(s"Invalid macro input: $ex") - } - // wrap the result up in an Expr, and return it - val result = c.Expr(Block(modDefs, Literal(Constant()))) - result + } + + val impl = + s""" + |def ${function.name} + | (${argDecl.mkString(",")}): $returnType = { + | + | val map = scala.collection.mutable.Map[String, Any]() + | var args = scala.collection.Seq[org.apache.mxnet.Symbol]() + | + | ${backendArgsMapping.mkString("\n")} + | + | org.apache.mxnet.Symbol.createSymbolGeneral( + | "${function.name}", name, attr, args, map.toMap) + |} + """.stripMargin + + c.parse(impl).asInstanceOf[DefDef] } +} + + +/** + * Symbol.random code generation + */ +private[mxnet] object TypedSymbolRandomAPIMacro extends GeneratorBase + with RandomHelpers { + + def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*): c.Expr[Any] = { + val functionDefs = typeSafeRandomFunctionsToGenerate(isSymbol = true) + .map(f => buildTypedFunction(c)(f)) - // List and add all the atomic symbol functions to current module. - private def initSymbolModule(): List[SymbolFunction] = { - val opNames = ListBuffer.empty[String] - _LIB.mxListAllOpNames(opNames) - // TODO: Add '_linalg_', '_sparse_', '_image_' support - opNames.map(opName => { - val opHandle = new RefLong - _LIB.nnGetOpHandle(opName, opHandle) - makeAtomicSymbolFunction(opHandle.value, opName) - }).toList + structGeneration(c)(functionDefs, annottees: _*) } - // Create an atomic symbol function by handle and function name. - private def makeAtomicSymbolFunction(handle: SymbolHandle, aliasName: String) - : SymbolFunction = { - val name = new RefString - val desc = new RefString - val keyVarNumArgs = new RefString - val numArgs = new RefInt - val argNames = ListBuffer.empty[String] - val argTypes = ListBuffer.empty[String] - val argDescs = ListBuffer.empty[String] - - _LIB.mxSymbolGetAtomicSymbolInfo( - handle, name, desc, numArgs, argNames, argTypes, argDescs, keyVarNumArgs) - val paramStr = OperatorBuildUtils.ctypes2docstring(argNames, argTypes, argDescs) - val extraDoc: String = if (keyVarNumArgs.value != null && keyVarNumArgs.value.length > 0) { - s"This function support variable length of positional input (${keyVarNumArgs.value})." - } else { - "" + protected def buildTypedFunction(c: blackbox.Context) + (function: Func): c.universe.DefDef = { + import c.universe._ + + val returnType = "org.apache.mxnet.Symbol" + + // Construct API arguments declaration + val argDecl = super.typedFunctionCommonArgDef(function) :+ + "name : String = null" :+ + "attr : Map[String, String] = null" + + // Map API input args to backend args + val backendArgsMapping = + function.listOfArgs.map { arg => + if (arg.argType.equals(s"Array[org.apache.mxnet.Symbol]")) { + s"args = ${arg.safeArgName}.toSeq" + } else { + // all go in kwargs + if (arg.isOptional) { + s"""if (${arg.safeArgName}.isDefined) map("${arg.argName}") = ${arg.safeArgName}.get""" + } else { + s"""map("${arg.argName}") = ${arg.safeArgName}""" + } + } } - val realName = if (aliasName == name.value) "" else s"(a.k.a., ${name.value})" - val docStr = s"$aliasName $realName\n${desc.value}\n\n$paramStr\n$extraDoc\n" - // scalastyle:off println - if (System.getenv("MXNET4J_PRINT_OP_DEF") != null - && System.getenv("MXNET4J_PRINT_OP_DEF").toLowerCase == "true") { - println("Symbol function definition:\n" + docStr) - } - // scalastyle:on println - val argList = argNames zip argTypes map { case (argName, argType) => - val typeAndOption = - CToScalaUtils.argumentCleaner(argName, argType, "org.apache.mxnet.Symbol") - new SymbolArg(argName, typeAndOption._1, typeAndOption._2) - } - new SymbolFunction(aliasName, argList.toList) + + val impl = + s""" + |def ${function.name}${randomGenericTypeSpec(true, true)} + | (${argDecl.mkString(",")}): $returnType = { + | + | val map = scala.collection.mutable.Map[String, Any]() + | var args = scala.collection.Seq[org.apache.mxnet.Symbol]() + | val isScalar = SymbolOrScalar[T].isScalar + | + | ${backendArgsMapping.mkString("\n")} + | + | val target = if(isScalar) { + | "random_${function.name}" + | } else { + | "sample_${function.name}" + | } + | + | ${unhackNormalFunc(function)} + | + | org.apache.mxnet.Symbol.createSymbolGeneral( + | target, name, attr, args, map.toMap) + |} + """.stripMargin + + c.parse(impl).asInstanceOf[DefDef] } } + diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/javaapi/JavaNDArrayMacro.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/javaapi/JavaNDArrayMacro.scala new file mode 100644 index 000000000..4dfd6eb04 --- /dev/null +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/javaapi/JavaNDArrayMacro.scala @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.mxnet.javaapi + +import org.apache.mxnet.GeneratorBase + +import scala.annotation.StaticAnnotation +import scala.collection.mutable.ListBuffer +import scala.language.experimental.macros +import scala.reflect.macros.blackbox + +private[mxnet] class AddJNDArrayAPIs(isContrib: Boolean) extends StaticAnnotation { + private[mxnet] def macroTransform(annottees: Any*) = macro JavaNDArrayMacro.typeSafeAPIDefs +} + +private[mxnet] object JavaNDArrayMacro extends GeneratorBase { + + // scalastyle:off havetype + def typeSafeAPIDefs(c: blackbox.Context)(annottees: c.Expr[Any]*) = { + typeSafeAPIImpl(c)(annottees: _*) + } + // scalastyle:off havetype + + private def typeSafeAPIImpl(c: blackbox.Context)(annottees: c.Expr[Any]*) : c.Expr[Any] = { + import c.universe._ + + val isContrib: Boolean = c.prefix.tree match { + case q"new AddJNDArrayAPIs($b)" => c.eval[Boolean](c.Expr(b)) + } + // Defines Operators that should not generated + val notGenerated = Set("Custom") + + val newNDArrayFunctions = functionsToGenerate(false, false, true) + .filterNot(ele => notGenerated.contains(ele.name)).groupBy(_.name.toLowerCase).map(ele => { + /* Pattern matching for not generating deprecated method + * Group all method name in lowercase + * Kill the capital lettered method such as Cast vs cast + * As it defined by default it deprecated + */ + if (ele._2.length == 1) ele._2.head + else { + if (ele._2.head.name.head.isLower) ele._2.head + else ele._2.last + } + }) + + val functionDefs = ListBuffer[DefDef]() + val classDefs = ListBuffer[ClassDef]() + + newNDArrayFunctions.foreach { ndarrayfunction => + + val useParamObject = ndarrayfunction.listOfArgs.count(arg => arg.isOptional) >= 2 + // Construct argument field with all required args + var argDef = ListBuffer[String]() + // Construct function Implementation field (e.g norm) + var impl = ListBuffer[String]() + impl += "val map = scala.collection.mutable.Map[String, Any]()" + impl += + "val args= scala.collection.mutable.ArrayBuffer.empty[org.apache.mxnet.NDArray]" + ndarrayfunction.listOfArgs.foreach({ ndarrayArg => + // var is a special word used to define variable in Scala, + // need to changed to something else in order to make it work + var currArgName = ndarrayArg.safeArgName + if (useParamObject) currArgName = s"po.get${currArgName.capitalize}()" + argDef += s"$currArgName : ${ndarrayArg.argType}" + // NDArray arg implementation + val returnType = "org.apache.mxnet.javaapi.NDArray" + val base = + if (ndarrayArg.argType.equals(returnType)) { + s"args += $currArgName" + } else if (ndarrayArg.argType.equals(s"Array[$returnType]")){ + s"$currArgName.foreach(args+=_)" + } else { + "map(\"" + ndarrayArg.argName + "\") = " + currArgName + } + impl.append( + if (ndarrayArg.isOptional) s"if ($currArgName != null) $base" + else base + ) + }) + // add default out parameter + argDef += s"out: org.apache.mxnet.javaapi.NDArray" + if (useParamObject) { + impl += "if (po.getOut() != null) map(\"out\") = po.getOut()" + } else { + impl += "if (out != null) map(\"out\") = out" + } + val returnType = "Array[org.apache.mxnet.javaapi.NDArray]" + // scalastyle:off + // Combine and build the function string + impl += "val finalArr = org.apache.mxnet.NDArray.genericNDArrayFunctionInvoke(\"" + + ndarrayfunction.name + "\", args.toSeq, map.toMap).arr" + impl += "finalArr.map(ele => new NDArray(ele))" + if (useParamObject) { + val funcDef = + s"""def ${ndarrayfunction.name}(po: ${ndarrayfunction.name}Param): $returnType = { + | ${impl.mkString("\n")} + | }""".stripMargin + functionDefs += c.parse(funcDef).asInstanceOf[DefDef] + } else { + val funcDef = + s"""def ${ndarrayfunction.name}(${argDef.mkString(",")}): $returnType = { + | ${impl.mkString("\n")} + | }""".stripMargin + functionDefs += c.parse(funcDef).asInstanceOf[DefDef] + } + } + structGeneration(c)(functionDefs.toList, annottees : _*) + } +} diff --git a/scala-package/macros/src/main/scala/org/apache/mxnet/utils/CToScalaUtils.scala b/scala-package/macros/src/main/scala/org/apache/mxnet/utils/CToScalaUtils.scala index d0ebe5b1d..57c4cfba1 100644 --- a/scala-package/macros/src/main/scala/org/apache/mxnet/utils/CToScalaUtils.scala +++ b/scala-package/macros/src/main/scala/org/apache/mxnet/utils/CToScalaUtils.scala @@ -18,22 +18,35 @@ package org.apache.mxnet.utils private[mxnet] object CToScalaUtils { - + private val javaType = Map( + "float" -> "java.lang.Float", + "int" -> "java.lang.Integer", + "long" -> "java.lang.Long", + "double" -> "java.lang.Double", + "bool" -> "java.lang.Boolean") + private val scalaType = Map( + "float" -> "Float", + "int" -> "Int", + "long" -> "Long", + "double" -> "Double", + "bool" -> "Boolean") // Convert C++ Types to Scala Types - def typeConversion(in : String, argType : String = "", - argName : String, returnType : String) : String = { + def typeConversion(in : String, argType : String = "", argName : String, + returnType : String, isJava : Boolean) : String = { + val header = returnType.split("\\.").dropRight(1) + val types = if (isJava) javaType else scalaType in match { - case "Shape(tuple)" | "ShapeorNone" => "org.apache.mxnet.Shape" + case "Shape(tuple)" | "ShapeorNone" => s"${header.mkString(".")}.Shape" case "Symbol" | "NDArray" | "NDArray-or-Symbol" => returnType case "Symbol[]" | "NDArray[]" | "NDArray-or-Symbol[]" | "SymbolorSymbol[]" => s"Array[$returnType]" - case "float" | "real_t" | "floatorNone" => "org.apache.mxnet.Base.MXFloat" - case "int" | "intorNone" | "int(non-negative)" => "Int" - case "long" | "long(non-negative)" => "Long" - case "double" | "doubleorNone" => "Double" + case "float" | "real_t" | "floatorNone" => types("float") + case "int" | "intorNone" | "int(non-negative)" => types("int") + case "long" | "long(non-negative)" => types("long") + case "double" | "doubleorNone" => types("double") case "string" => "String" - case "boolean" | "booleanorNone" => "Boolean" + case "boolean" | "booleanorNone" => types("bool") case "tupleof" | "tupleof" | "tupleof<>" | "ptr" | "" => "Any" case default => throw new IllegalArgumentException( s"Invalid type for args: $default\nString argType: $argType\nargName: $argName") @@ -52,8 +65,8 @@ private[mxnet] object CToScalaUtils { * @param argType Raw arguement Type description * @return (Scala_Type, isOptional) */ - def argumentCleaner(argName: String, - argType : String, returnType : String) : (String, Boolean) = { + def argumentCleaner(argName: String, argType : String, + returnType : String, isJava : Boolean) : (String, Boolean) = { val spaceRemoved = argType.replaceAll("\\s+", "") var commaRemoved : Array[String] = new Array[String](0) // Deal with the case e.g: stype : {'csr', 'default', 'row_sparse'} @@ -71,9 +84,9 @@ private[mxnet] object CToScalaUtils { s"""expected "optional" got ${commaRemoved(1)}""") require(commaRemoved(2).startsWith("default="), s"""expected "default=..." got ${commaRemoved(2)}""") - (typeConversion(commaRemoved(0), argType, argName, returnType), true) + (typeConversion(commaRemoved(0), argType, argName, returnType, isJava), true) } else if (commaRemoved.length == 2 || commaRemoved.length == 1) { - val tempType = typeConversion(commaRemoved(0), argType, argName, returnType) + val tempType = typeConversion(commaRemoved(0), argType, argName, returnType, isJava) val tempOptional = tempType.equals("org.apache.mxnet.Symbol") (tempType, tempOptional) } else { diff --git a/scala-package/macros/src/test/scala/org/apache/mxnet/MacrosSuite.scala b/scala-package/macros/src/test/scala/org/apache/mxnet/MacrosSuite.scala index c3a7c58c1..4069bba25 100644 --- a/scala-package/macros/src/test/scala/org/apache/mxnet/MacrosSuite.scala +++ b/scala-package/macros/src/test/scala/org/apache/mxnet/MacrosSuite.scala @@ -43,7 +43,8 @@ class MacrosSuite extends FunSuite with BeforeAndAfterAll { ) for (idx <- input.indices) { - val result = CToScalaUtils.argumentCleaner("Sample", input(idx), "org.apache.mxnet.Symbol") + val result = CToScalaUtils.argumentCleaner("Sample", input(idx), + "org.apache.mxnet.Symbol", false) assert(result._1 === output(idx)._1 && result._2 === output(idx)._2) } } diff --git a/scala-package/memory-management.md b/scala-package/memory-management.md new file mode 100644 index 000000000..33c36b6e6 --- /dev/null +++ b/scala-package/memory-management.md @@ -0,0 +1,118 @@ +# JVM Memory Management +The Scala and Java bindings of Apache MXNet use native memory (memory from the C++ heap in either RAM or GPU memory) for most of the MXNet objects such as NDArray, Symbol, Executor, KVStore, Data Iterators, etc. +The associated Scala classes act only as wrappers. The operations done on these wrapper objects are then directed to the high performance MXNet C++ backend via the Java Native Interface (JNI). Therefore, the bytes are stored in the C++ native heap which allows for fast access. + +However, the JVM Garbage Collector only manages objects allocated in the JVM Heap and is not aware of the memory footprint of these objects in the native memory. Hence, the allocation/deallocation of native memory must be managed by MXNet Scala. +Allocating native memory is straight forward and is done during the construction of the object by calling the associated C++ API through JNI. However, since JVM languages do not have destructors, the deallocation of these objects must be done explicitly. +MXNet Scala provides a few easy modes of operation which are explained in detail below. + +## Memory Management in Scala +### 1. [ResourceScope.using](https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala#L106) (Recommended) +`ResourceScope.using` provides the familiar Java try-with-resources primitive in Scala and will automatically manage the memory of all the MXNet objects created in the associated code block (`body`). It works by tracking the allocations performed inside the code block deallocating when exiting the block. +Passing MXNet objects out of a using block can be easily accomplished by simply returning an object or an iterable containing multiple MXNet objects. If you have nested using blocks, then the returned objects will be moved into the parent scope as well. + +**Usage** +```scala +ResourceScope.using() { + ResourceScope.using() { + val r1 = NDArray.ones(Shape(2, 2)) + val r2 = NDArray.ones(Shape(3, 4)) + val r3 = NDArray.ones(Shape(5, 6)) + val r4 = NDArray.ones(Shape(7, 8)) + (r3, r4) + } + r4 +} +``` +In the example above, we have two ResourceScopes stacked together. In the inner scope, 4 NDArrays `(r1, r2, r3, r4)` are created and the NDArrays +`(r3, r4)` are returned. The inner ResourceScope recognizes that it should not deallocate these objects and automatically moves `r3` and `r4` to the outer scope. When the outer scope +returns `r4` from its code-block, it will only deallocate `r3` and will remove `r4` from its list of objects to be deallocated. All other objects are automatically released by calling the C++ backend to free the native memory. + +**Note:** +You should consider nesting ResourceScopes when you have layers of functionality in your application code or create a lot of MXNet objects such as NDArrays. +For example, holding onto all the memory that is created for an entire training loop can result in running out of memory, especially when training on GPUs which might only have 8 to 16 GB. +It is recommended not to use a single ResourceScope block which spans the entire training code. You should instead nest multiple scopes: an innermost scope where you run forward-backward passes on each batch, a middle scope for each epoch, and an outer scope that runs the entire training script. This is demonstrated in the example below: + +```scala +ResourceScope.using() { + val m = Module() + m.bind() + val k = KVStore(...) + ResourceScope.using() { + val itr = MXIterator(..) + val num_epochs: Int = 100 + //... + for (i <- 0 until num_epoch) { + ResourceScope.using() { + val dataBatch = itr.next() + while(itr.next()) { + m.forward(dataBatch) + m.backward(dataBatch) + m.update() + } + } + } + } +} + +``` + +### 2. Using Phantom References (Recommended for some use cases) + +Apache MXNet uses [Phantom References](https://docs.oracle.com/javase/8/docs/api/java/lang/ref/PhantomReference.html) to track all MXNet Objects that have native memory associated with it. +When the Garbage Collector runs, it identifies unreachable Scala/Java objects in the JVM Heap and finalizes them. +It then enqueues objects which are ready to be reclaimed into a reference queue. We take advantage of this and do a +pre-mortem cleanup on these wrapper objects by freeing the corresponding native memory as well. + +This approach is automatic and does not require any special code to clean up the native memory. However, the Garbage Collector is not aware of the potentially large amount of native memory used and therefore may not free up memory often enough with it's standard behavior. +You can control the frequency of garbage collection by calling System.gc() at strategic points such as the end of an epoch or the end of a mini-batch. + +This approach could be suitable for some use cases such as inference on CPUs where you have a large amount of Memory (RAM) on your system. + +**Note:** +Calling GC too frequently can also cause your application to perform poorly. This approach might not be suitable +for use cases which quickly allocate a large number of large NDArrays such as when training a GAN model. + +### 3. Using dispose Pattern (least Recommended) + +There might be situations where you want to manually manage the lifecycle of Apache MXNet objects. For such use-cases, we have provided the `dispose()` method which will manually deallocate the associated native memory when called. We have also +made all MXNet objects [AutoCloseable](https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html). If you are using Java8 and above you can use it with try-with-resources or call close() in the finally block. + +**Note:** +We recommend you avoid manually managing MXNet objects and instead use `ResourceScope.using`. This creates less readable code and could leak memory if you miss calling dispose (until it is cleaned up by the Garbage Collector through the Phantom References). + +```scala +def showDispose(): Unit = { + val r = NDArray.ones(Shape (2, 2)) + r.dispose() +} +``` + +## Memory Management in Java +Memory Management in MXNet Java is similar to Scala. We recommend you use [ResourceScope](https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/ResourceScope.scala#L32) in a `try-with-resources` block or in a `try-finally` block. +The [try-with-resource](https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html) tracks the resources declared in the try block and automatically closes them upon exiting (supported from Java 7 onwards). +The ResourceScope discussed above implements AutoCloseable and tracks all MXNet Objects created at a Thread Local scope level. + +```java +try(ResourceScope scope = new ResourceScope()) { + NDArray test = NDArray.ones((Shape (2,2)) +} +``` +or +```java +try { + ResourceScope scope = new ResourceScope() + NDArray test = NDArray.ones((Shape(2,2)) +} finally { + scope.close() +} +``` + +**Note:** +A ResourceScope within a try block tracks all MXNet Native Object Allocations (NDArray, Symbol, Executor, etc.,) and deallocates them at +the end of the try block. This is also true of the objects that are returned e.g. in the example above, the native memory associated with `test` would be deallocated even if it were to be returned. +If you use the object outside of the try block, the process might crash due to illegal memory access. + +To retain certain objects created within try blocks, you should explicitly remove them from the scope by calling `scope.moveToOuterScope`. +It is highly recommended to nest multiple try-with-resource ResourceScopes so you do not have to explicitly manage the lifecycle of the Native objects. + diff --git a/scala-package/mxnet-demo/java-demo/Makefile b/scala-package/mxnet-demo/java-demo/Makefile new file mode 100644 index 000000000..3811d75d2 --- /dev/null +++ b/scala-package/mxnet-demo/java-demo/Makefile @@ -0,0 +1,52 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +SCALA_VERSION_PROFILE := 2.11 + +ifeq ($(OS),Windows_NT) + UNAME_S := Windows +else + UNAME_S := $(shell uname -s) +endif + +ifeq ($(UNAME_S), Windows) + # TODO: currently scala package does not support windows + SCALA_PKG_PROFILE := windows +else + ifeq ($(UNAME_S), Darwin) + SCALA_PKG_PROFILE := osx-x86_64-cpu + else + SCALA_PKG_PROFILE := linux-x86_64 + ifeq ($(USE_CUDA), 1) + SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-gpu + else + SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-cpu + endif + endif +endif + +javademo: + (mvn install -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE)) + +java_ci_demo: + (mvn -Pci-nightly install -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE)) + +javaclean: + (mvn clean -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE)) diff --git a/scala-package/mxnet-demo/java-demo/README.md b/scala-package/mxnet-demo/java-demo/README.md new file mode 100644 index 000000000..73e9284a9 --- /dev/null +++ b/scala-package/mxnet-demo/java-demo/README.md @@ -0,0 +1,90 @@ +# MXNet Java Sample Project +This is an project created to use Maven-published Scala/Java package with two Java examples. +## Setup +You can use the `Makefile` to make the Java package. Simply do the following: +```Bash +make javademo +``` +This will load the default parameter for all the environment variable. +If you want to run with GPU on Linux, just simply add `USE_CUDA=1` when you run the make file + +You can use the following instruction as an alternative to achieve the same result: +User are required to use `mvn package` to build the package, + which are shown below: +```Bash +export SCALA_VERSION_PROFILE=2.11 +export SCALA_PKG_PROFILE= +mvn package -Dmxnet.profile=$SCALA_PKG_PROFILE \ + -Dmxnet.scalaprofile=$SCALA_VERSION_PROFILE +``` +These environment variable (`SCALA_PKG_PROFILE`, `SCALA_VERSION_PROFILE`) +should be set before executing the line above. +The `SCALA_PKG_PROFILE` should be chosen from `osx-x86_64-cpu`, `linux-x86_64-cpu` or `linux-x86_64-gpu`. + + +## Run +### Hello World +The Scala file is being executed using Java. You can execute the helloWorld example as follows: +```Bash +bash bin/java_sample.sh +``` +You can also run the following command manually: +```Bash +java -cp $CLASSPATH sample.HelloWorld +``` +However, you have to define the Classpath before you run the demo code. More information can be found in the `java_sample.sh`. +The `CLASSPATH` should point to the jar file you have downloaded. + +It will load the library automatically and run the example + +In order to use the `Param Object`. We requires user to place this line in the front: +``` +static NDArray$ NDArray = NDArray$.MODULE$; +``` +It would help to have the NDArray companion object static and accessable from the outside. + +### Object Detection using Inference API +We also provide an example to do object detection, which downloads a ImageNet trained resnet50 model and runs inference on an image to return the classification result as +```Bash +Class: car +Probabilties: 0.99847263 +Coord:312.21335, 72.02908, 456.01443, 150.66176 +Class: bicycle +Probabilties: 0.9047381 +Coord:155.9581, 149.96365, 383.83694, 418.94516 +Class: dog +Probabilties: 0.82268167 +Coord:83.82356, 179.14001, 206.63783, 476.78754 +``` + +you can run using the command shown below: +```Bash +bash bin/run_od.sh +``` +or the command below as an alternative +```Bash +java -cp $CLASSPATH sample.ObjectDetection +``` + +If you want to test run on GPU, you can set a environment variable as follows: +```Bash +export SCALA_TEST_ON_GPU=1 +``` +## Clean up +Clean up for Maven package is simple, you can run the pre-configed `Makefile` as: +```Bash +make javaclean +``` + +## Q & A +If you are facing opencv issue on Ubuntu, please try as follows to install opencv 3.4 (required by 1.2.0 package and above) +```Bash +sudo add-apt-repository ppa:timsc/opencv-3.4 +sudo apt-get update +sudo apt install libopencv-imgcodecs3.4 +``` + +Is there any other version available? + +You can find nightly release version from [here](https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.5.0-SNAPSHOT~~). +Please keep the same version in the Makefile or [above version](https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~~~) to run this demo. diff --git a/tools/pip_package/MANIFEST.in b/scala-package/mxnet-demo/java-demo/bin/java_sample.sh old mode 100644 new mode 100755 similarity index 81% rename from tools/pip_package/MANIFEST.in rename to scala-package/mxnet-demo/java-demo/bin/java_sample.sh index 5c6a72377..4fb724aca --- a/tools/pip_package/MANIFEST.in +++ b/scala-package/mxnet-demo/java-demo/bin/java_sample.sh @@ -14,7 +14,7 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. - -include README -recursive-include * *.py -recursive-include * *.so +#!/bin/bash +CURR_DIR=$(cd $(dirname $0)/../; pwd) +CLASSPATH=$CLASSPATH:$CURR_DIR/target/*:$CLASSPATH:$CURR_DIR/target/dependency/* +java -Xmx8G -cp $CLASSPATH mxnet.HelloWorld \ No newline at end of file diff --git a/ci/docker/qemu/ansible.cfg b/scala-package/mxnet-demo/java-demo/bin/run_od.sh old mode 100644 new mode 100755 similarity index 81% rename from ci/docker/qemu/ansible.cfg rename to scala-package/mxnet-demo/java-demo/bin/run_od.sh index 24e2ec87b..abd0bf5b1 --- a/ci/docker/qemu/ansible.cfg +++ b/scala-package/mxnet-demo/java-demo/bin/run_od.sh @@ -14,7 +14,7 @@ # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. - -[defaults] -host_key_checking = False -ansible_python_interpreter = /usr/bin/python3 +#!/bin/bash +CURR_DIR=$(cd $(dirname $0)/../; pwd) +CLASSPATH=$CLASSPATH:$CURR_DIR/target/*:$CLASSPATH:$CURR_DIR/target/dependency/* +java -Xmx8G -cp $CLASSPATH mxnet.ObjectDetection \ No newline at end of file diff --git a/scala-package/mxnet-demo/java-demo/pom.xml b/scala-package/mxnet-demo/java-demo/pom.xml new file mode 100644 index 000000000..b7502a66c --- /dev/null +++ b/scala-package/mxnet-demo/java-demo/pom.xml @@ -0,0 +1,71 @@ + + + 4.0.0 + Demo + mxnet-java-demo + 1.0-SNAPSHOT + MXNet Java Demo + + + + ci-nightly + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + + + [1.5.0-SNAPSHOT, ) + + + + + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + + + + 1.8 + 1.8 + [1.5.0-SNAPSHOT, ) + + + + + org.apache.mxnet + mxnet-full_${mxnet.scalaprofile}-${mxnet.profile} + ${mxnet.version} + + + commons-io + commons-io + 2.4 + + + + + + + org.apache.maven.plugins + maven-dependency-plugin + 2.9 + + + copy-dependencies + package + + copy-dependencies + + + + + + + + \ No newline at end of file diff --git a/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/HelloWorld.java b/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/HelloWorld.java new file mode 100644 index 000000000..71981e269 --- /dev/null +++ b/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/HelloWorld.java @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package mxnet; + +import org.apache.mxnet.javaapi.*; +import java.util.Arrays; + +public class HelloWorld { + static NDArray$ NDArray = NDArray$.MODULE$; + + public static void main(String[] args) { + System.out.println("Hello World!"); + NDArray nd = new NDArray(new float[]{2.0f, 3.0f}, new Shape(new int[]{1, 2}), Context.cpu()); + System.out.println(nd.shape()); + NDArray nd2 = NDArray.dot(NDArray.new dotParam(nd, nd.T()))[0]; + System.out.println(Arrays.toString(nd2.toArray())); + } +} diff --git a/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/ObjectDetection.java b/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/ObjectDetection.java new file mode 100644 index 000000000..cfe9b66c4 --- /dev/null +++ b/scala-package/mxnet-demo/java-demo/src/main/java/mxnet/ObjectDetection.java @@ -0,0 +1,101 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package mxnet; +import org.apache.mxnet.infer.javaapi.ObjectDetectorOutput; +import org.apache.mxnet.javaapi.*; +import org.apache.mxnet.infer.javaapi.ObjectDetector; +import org.apache.commons.io.FileUtils; +import java.io.File; +import java.net.URL; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +public class ObjectDetection { + private static String modelPath; + private static String imagePath; + + private static void downloadUrl(String url, String filePath) { + File tmpFile = new File(filePath); + if (!tmpFile.exists()) { + try { + FileUtils.copyURLToFile(new URL(url), tmpFile); + } catch (Exception exception) { + System.err.println(exception); + } + } + } + + public static void downloadModelImage() { + String tempDirPath = System.getProperty("java.io.tmpdir"); + System.out.println("tempDirPath: %s".format(tempDirPath)); + imagePath = tempDirPath + "/inputImages/resnetssd/dog-ssd.jpg"; + String imgURL = "https://s3.amazonaws.com/model-server/inputs/dog-ssd.jpg"; + downloadUrl(imgURL, imagePath); + modelPath = tempDirPath + "/resnetssd/resnet50_ssd_model"; + System.out.println("Download model files, this can take a while..."); + String modelURL = "https://s3.amazonaws.com/model-server/models/resnet50_ssd/"; + downloadUrl(modelURL + "resnet50_ssd_model-symbol.json", + tempDirPath + "/resnetssd/resnet50_ssd_model-symbol.json"); + downloadUrl(modelURL + "resnet50_ssd_model-0000.params", + tempDirPath + "/resnetssd/resnet50_ssd_model-0000.params"); + downloadUrl(modelURL + "synset.txt", + tempDirPath + "/resnetssd/synset.txt"); + } + + static List> + runObjectDetectionSingle(String modelPathPrefix, String inputImagePath, List context) { + Shape inputShape = new Shape(new int[] {1, 3, 512, 512}); + List inputDescriptors = new ArrayList(); + inputDescriptors.add(new DataDesc("data", inputShape, DType.Float32(), "NCHW")); + ObjectDetector objDet = new ObjectDetector(modelPathPrefix, inputDescriptors, context, 0); + return objDet.imageObjectDetect(ObjectDetector.loadImageFromFile(inputImagePath), 3); + } + + public static void main(String[] args) { + List context = new ArrayList(); + if (System.getenv().containsKey("SCALA_TEST_ON_GPU") && + Integer.valueOf(System.getenv("SCALA_TEST_ON_GPU")) == 1) { + context.add(Context.gpu()); + } else { + context.add(Context.cpu()); + } + downloadModelImage(); + Shape inputShape = new Shape(new int[] {1, 3, 512, 512}); + Shape outputShape = new Shape(new int[] {1, 6132, 6}); + int width = inputShape.get(2); + int height = inputShape.get(3); + List> output + = runObjectDetectionSingle(modelPath, imagePath, context); + String outputStr = "\n"; + for (List ele : output) { + for (ObjectDetectorOutput i : ele) { + outputStr += "Class: " + i.getClassName() + "\n"; + outputStr += "Probabilties: " + i.getProbability() + "\n"; + + List coord = Arrays.asList(i.getXMin() * width, + i.getXMax() * height, i.getYMin() * width, i.getYMax() * height); + StringBuilder sb = new StringBuilder(); + for (float c: coord) { + sb.append(", ").append(c); + } + outputStr += "Coord:" + sb.substring(2)+ "\n"; + } + } + System.out.println(outputStr); + } +} \ No newline at end of file diff --git a/scala-package/mxnet-demo/Makefile b/scala-package/mxnet-demo/scala-demo/Makefile similarity index 90% rename from scala-package/mxnet-demo/Makefile rename to scala-package/mxnet-demo/scala-demo/Makefile index 227697ba2..9f7f2840e 100644 --- a/scala-package/mxnet-demo/Makefile +++ b/scala-package/mxnet-demo/scala-demo/Makefile @@ -17,7 +17,6 @@ SCALA_VERSION_PROFILE := 2.11 SCALA_VERSION := 2.11.8 -MXNET_VERSION := 1.2.0 ifeq ($(OS),Windows_NT) UNAME_S := Windows @@ -44,11 +43,14 @@ endif scalademo: (mvn package -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ - -Dmxnet.version=$(MXNET_VERSION) \ + -Dscala.version=$(SCALA_VERSION)) + +scala_ci_demo: + (mvn -Pci-nightly package -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ -Dscala.version=$(SCALA_VERSION)) scalaclean: (mvn clean -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ - -Dmxnet.version=$(MXNET_VERSION) \ -Dscala.version=$(SCALA_VERSION)) \ No newline at end of file diff --git a/scala-package/mxnet-demo/README.md b/scala-package/mxnet-demo/scala-demo/README.md similarity index 85% rename from scala-package/mxnet-demo/README.md rename to scala-package/mxnet-demo/scala-demo/README.md index e30a61a2f..1cc5a6d42 100644 --- a/scala-package/mxnet-demo/README.md +++ b/scala-package/mxnet-demo/scala-demo/README.md @@ -4,15 +4,16 @@ This is an project created to use Maven-published Scala package with two Scala e User are required to use `mvn package` to build the package, which are shown below: ```Bash -export SCALA_VERSION_PROFILE=2.11 SCALA_VERSION=2.11.8 MXNET_VERSION=1.2.0 +export SCALA_VERSION_PROFILE=2.11 SCALA_VERSION=2.11.8 export SCALA_PKG_PROFILE= mvn package -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ - -Dmxnet.version=$(MXNET_VERSION) \ -Dscala.version=$(SCALA_VERSION) ``` -These environment variable (`SCALA_PKG_PROFILE`, `SCALA_VERSION_PROFILE`, `MXNET_VERSION`, `SCALA_VERSION`) -should be set before executing the line above. +These environment variable (`SCALA_PKG_PROFILE`, `SCALA_VERSION_PROFILE`, `SCALA_VERSION`) +should be set before executing the line above. + +To obtain the most recent MXNet version, please click [here](https://mvnrepository.com/search?q=org.apache.mxnet) You can also use the `Makefile` as an alternative to do the same thing. Simply do the following: ```Bash @@ -25,7 +26,7 @@ This will load the default parameter for all the environment variable. ### Hello World The Scala file is being executed using Java. You can execute the helloWorld example as follows: ```Bash -java -Xmx8G -cp $CLASSPATH sample.HelloWorld +java -cp $CLASSPATH sample.HelloWorld ``` However, you have to define the Classpath before you run the demo code. More information can be found in the `demo.sh` And you can run the bash script as follows: ```Bash @@ -41,7 +42,7 @@ You can review the complete example [here](https://github.com/apache/incubator-m you can run using the command shown below: ```Bash -java -Xmx8G -cp $CLASSPATH sample.ImageClassificationExample +java -cp $CLASSPATH sample.ImageClassificationExample ``` or script as follows: ```Bash @@ -59,7 +60,7 @@ make scalaclean ``` ## Q & A -If you are facing opencv issue on Ubuntu, please try as follows to install opencv 3.4 (required by 1.2.0 package) +If you are facing opencv issue on Ubuntu, please try as follows to install opencv 3.4 (required by 1.2.0 package and above) ```Bash sudo add-apt-repository ppa:timsc/opencv-3.4 sudo apt-get update diff --git a/scala-package/mxnet-demo/bin/demo.sh b/scala-package/mxnet-demo/scala-demo/bin/demo.sh similarity index 100% rename from scala-package/mxnet-demo/bin/demo.sh rename to scala-package/mxnet-demo/scala-demo/bin/demo.sh diff --git a/scala-package/mxnet-demo/bin/run_im.sh b/scala-package/mxnet-demo/scala-demo/bin/run_im.sh similarity index 100% rename from scala-package/mxnet-demo/bin/run_im.sh rename to scala-package/mxnet-demo/scala-demo/bin/run_im.sh diff --git a/scala-package/mxnet-demo/pom.xml b/scala-package/mxnet-demo/scala-demo/pom.xml similarity index 87% rename from scala-package/mxnet-demo/pom.xml rename to scala-package/mxnet-demo/scala-demo/pom.xml index 8fc30e78c..1d89f9765 100644 --- a/scala-package/mxnet-demo/pom.xml +++ b/scala-package/mxnet-demo/scala-demo/pom.xml @@ -4,11 +4,30 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 Demo - mxnet-scala-demo_2.11 + mxnet-scala-demo 1.0-SNAPSHOT MXNet Scala Demo pom + + + ci-nightly + + + Apache Snapshot + https://repository.apache.org/content/groups/snapshots + + + + [1.5.0-SNAPSHOT, ) + + + + + + [1.3.1, ) + + org.apache.mxnet diff --git a/scala-package/mxnet-demo/src/main/scala/sample/HelloWorld.scala b/scala-package/mxnet-demo/scala-demo/src/main/scala/sample/HelloWorld.scala similarity index 100% rename from scala-package/mxnet-demo/src/main/scala/sample/HelloWorld.scala rename to scala-package/mxnet-demo/scala-demo/src/main/scala/sample/HelloWorld.scala diff --git a/scala-package/mxnet-demo/src/main/scala/sample/ImageClassificationExample.scala b/scala-package/mxnet-demo/scala-demo/src/main/scala/sample/ImageClassificationExample.scala similarity index 100% rename from scala-package/mxnet-demo/src/main/scala/sample/ImageClassificationExample.scala rename to scala-package/mxnet-demo/scala-demo/src/main/scala/sample/ImageClassificationExample.scala diff --git a/scala-package/native/README.md b/scala-package/native/README.md new file mode 100644 index 000000000..c87b064ff --- /dev/null +++ b/scala-package/native/README.md @@ -0,0 +1,67 @@ +# MXNet Scala JNI + +MXNet Scala JNI is a thin wrapper layer of underlying libmxnet.so. + +## javah +JNI native code requires a header file that matches the java/scala interface, +this file is usually generated with javah. + +In our case, org_apache_mxnet_native_c.h is generated and will be used to compile native code. + +To improve build performance, we check in generated org_apache_mxnet_native_c.h file. +And we added a check to detect mismatch with Scala code and generated header. The checker will +make sure we won't forget to update org_apache_mxnet_native_c.h file. + + +## Linker options + +Scala JNI (libmxnet-scala.so/libmxnet-scala.jnilib) is dynamically linked to libmxnet.so. +MXNet Scala will trying to load libmxnet.so from system LD_LIBRARY_PATH first. +If it failed, the try to resolve libmxnet.so in the same location as libmxnet-scala.so file. + +### Linux +``` +-Wl,-rpath=$ORIGIN -lmxnet +``` +Above option will tell system to looking for libmxnet.so from the same location. + + +### Mac OSX +On Mac, we have to execute install_name_tool command to change library loading path: +```bash +install_name_tool -change lib/libmxnet.so @loader_path/libmxnet.so libmxnet-scala.jnilib +``` + +Other linker options: +* -shared : link as shared library +* -Wl,-install_name,libmxnet-scala.jnilib : avoid use build machine's absolute path +* -framework JavaVM : Stand jni options for mac +* -Wl,-exported_symbol,_Java_* : Stand jni options for mac +* -Wl,-x : Do not put non-global symbols in the output file's symbol table. + + +## Compiler flags + +Scala JNI code technically doesn't need on any of MXNet make flags, +however c_api.h header links to many other dependencies header file, +which requires us to add DMSHADOW_USE_MKL and DMSHADOW_USE_CUDA to compile the JNI code. +These flags are not actually used by JNI and won't impact Scala's behavior. + + +### Linux + +``` +-DMSHADOW_USE_MKL=0 +-DMSHADOW_USE_CUDA=0 +-O3 -DNDEBUG=1 -fPIC -msse3 -mf16c +-Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs +``` + +### Mac OSX + +``` +-DMSHADOW_USE_MKL=0 +-DMSHADOW_USE_CUDA=0 +-g -O0 -fPIC -msse3 -mf16c +-Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs +``` diff --git a/scala-package/native/linux-x86_64-cpu/pom.xml b/scala-package/native/linux-x86_64-cpu/pom.xml deleted file mode 100644 index 7603c0307..000000000 --- a/scala-package/native/linux-x86_64-cpu/pom.xml +++ /dev/null @@ -1,106 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-scala-native-parent - 1.3.1-SNAPSHOT - ../pom.xml - - - libmxnet-scala-linux-x86_64-cpu - MXNet Scala Package - Native Linux-x86_64 CPU-only - http://maven.apache.org - - so - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - jar - compile - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-compiler-plugin - - - - org.codehaus.mojo - native-maven-plugin - true - - - linux - generic-classic - ${cxx} - ${cxx} - - - ../src/main/native - - org_apache_mxnet_native_c_api.cc - - - - - -std=c++0x - - - -I${project.basedir}/../../../include - ${all_includes} - ${cflags} - - - -shared - - - ${all_ldpaths} - -Wl,--whole-archive - ${lddeps} - -Wl,--no-whole-archive - - - ${ldflags} - -fopenmp - - - - - - javah - generate-sources - - linux - default - ${project.build.directory}/custom-javah - ${basedir} - org_apache_mxnet_native_c_api.h - - org.apache.mxnet.LibInfo - - - - javah - - - - - - - diff --git a/scala-package/native/linux-x86_64-gpu/pom.xml b/scala-package/native/linux-x86_64-gpu/pom.xml deleted file mode 100644 index df2ec912e..000000000 --- a/scala-package/native/linux-x86_64-gpu/pom.xml +++ /dev/null @@ -1,106 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-scala-native-parent - 1.3.1-SNAPSHOT - ../pom.xml - - - libmxnet-scala-linux-x86_64-gpu - MXNet Scala Package - Native Linux-x86_64 GPU - http://maven.apache.org - - so - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - jar - compile - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-compiler-plugin - - - - org.codehaus.mojo - native-maven-plugin - true - - - linux - generic-classic - ${cxx} - ${cxx} - - - ../src/main/native - - org_apache_mxnet_native_c_api.cc - - - - - -std=c++0x - - - -I${project.basedir}/../../../include - ${all_includes} - ${cflags} - - - -shared - - - ${all_ldpaths} - -Wl,--whole-archive - ${lddeps} - -Wl,--no-whole-archive - - - ${ldflags} - -fopenmp - - - - - - javah - generate-sources - - linux - default - ${project.build.directory}/custom-javah - ${basedir} - org_apache_mxnet_native_c_api.h - - org.apache.mxnet.LibInfo - - - - javah - - - - - - - diff --git a/scala-package/native/osx-x86_64-cpu/pom.xml b/scala-package/native/osx-x86_64-cpu/pom.xml deleted file mode 100644 index cff0b14a6..000000000 --- a/scala-package/native/osx-x86_64-cpu/pom.xml +++ /dev/null @@ -1,106 +0,0 @@ - - - 4.0.0 - - org.apache.mxnet - mxnet-scala-native-parent - 1.3.1-SNAPSHOT - ../pom.xml - - - libmxnet-scala-osx-x86_64-cpu - MXNet Scala Package - Native OSX-x86_64 CPU-only - http://maven.apache.org - - jnilib - - - - org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT - jar - compile - - - - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - - - - org.apache.maven.plugins - maven-compiler-plugin - - - - org.codehaus.mojo - native-maven-plugin - true - - - darwin - generic-classic - ${cxx} - ${cxx} - - - ../src/main/native - - org_apache_mxnet_native_c_api.cc - - - - - -std=c++0x - - - -I../../../include - ${cflags} - - - -shared - - - -framework JavaVM - -Wl,-exported_symbol,_Java_* - -Wl,-x - ${lddeps} - -force_load ${project.basedir}/../../../lib/libmxnet.a - -force_load ${project.basedir}/../../../3rdparty/tvm/nnvm/lib/libnnvm.a - - - ${ldflags} - - - - - - javah - generate-sources - - darwin - default - ${project.build.directory}/custom-javah - ${basedir} - org_apache_mxnet_native_c_api.h - - org.apache.mxnet.LibInfo - - - - javah - - - - - - - diff --git a/scala-package/native/pom.xml b/scala-package/native/pom.xml index 49922a9ad..7b776d5b5 100644 --- a/scala-package/native/pom.xml +++ b/scala-package/native/pom.xml @@ -5,46 +5,165 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-scala-native-parent - MXNet Scala Package - Native Parent - pom + libmxnet-scala + MXNet Scala Package - Native + ${libtype} + + + ${project.parent.basedir}/.. + - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - + osx-x86_64 + + mac + + + + + org.codehaus.mojo + native-maven-plugin + true + + darwin + generic-classic + ${cxx} + ${cxx} + + + src/main/native + + org_apache_mxnet_native_c_api.cc + + + + + -std=c++0x + + + -I${MXNET_DIR}/include + -I${MXNET_DIR}/3rdparty/dmlc-core/include + -I${MXNET_DIR}/3rdparty/mshadow + -I${MXNET_DIR}/3rdparty/dlpack/include + -I${MXNET_DIR}/3rdparty/tvm/nnvm/include + -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0 + -g -O0 -fPIC -msse3 -mf16c + -Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs + + + -shared + + + -framework JavaVM + -Wl,-exported_symbol,_Java_* + -Wl,-x + + + -Wl,-install_name,libmxnet-scala.jnilib -lmxnet -L${MXNET_DIR}/lib + + + + + + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + post-native-build + package + + exec + + + install_name_tool + -add_rpath @loader_path ${project.build.directory}/${project.artifactId}.jnilib + + + + + + - linux-x86_64-gpu - - linux-x86_64-gpu - + linux-x86_64 + + + unix + Linux + + + + + + org.codehaus.mojo + native-maven-plugin + true + + linux + generic-classic + ${cxx} + ${cxx} + + + src/main/native + + org_apache_mxnet_native_c_api.cc + + + + + -std=c++0x + + + -I${MXNET_DIR}/include + -I${MXNET_DIR}/3rdparty/dmlc-core/include + -I${MXNET_DIR}/3rdparty/mshadow + -I${MXNET_DIR}/3rdparty/dlpack/include + -I${MXNET_DIR}/3rdparty/tvm/nnvm/include + -DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0 + -O3 -DNDEBUG=1 -fPIC -msse3 -mf16c + -Wall -Wsign-compare -Wno-unused-parameter -Wno-unknown-pragmas -Wno-unused-local-typedefs + + + -shared + + + -Wl,-rpath=${dollar}ORIGIN -lmxnet -L${MXNET_DIR}/lib + + + + + - org.apache.maven.plugins - maven-deploy-plugin - - true - + org.codehaus.mojo + exec-maven-plugin + 1.6.0 + + + link-native-lib + generate-resources + + exec + + + bash + -c 'ln -sf ${MXNET_DIR}/lib/* ${project.build.directory}/' + + + - diff --git a/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.cc b/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.cc index 17d166eac..ea6e9c8f5 100644 --- a/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.cc +++ b/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.cc @@ -33,6 +33,7 @@ #include #include #include +#include #include "jni_helper_func.h" JavaVM *_jvm; @@ -423,6 +424,15 @@ JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySyncCopyFromCPU return ret; } +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFloat64NDArraySyncCopyFromCPU + (JNIEnv *env, jobject obj, jlong arrayPtr, jdoubleArray sourceArr, jint arrSize) { + jdouble *sourcePtr = env->GetDoubleArrayElements(sourceArr, NULL); + int ret = MXNDArraySyncCopyFromCPU(reinterpret_cast(arrayPtr), + static_cast(sourcePtr), arrSize); + env->ReleaseDoubleArrayElements(sourceArr, sourcePtr, 0); + return ret; +} + JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayGetContext (JNIEnv *env, jobject obj, jlong arrayPtr, jobject devTypeId, jobject devId) { int outDevType; diff --git a/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h b/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h new file mode 100644 index 000000000..7e8e03de9 --- /dev/null +++ b/scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h @@ -0,0 +1,861 @@ +/* DO NOT EDIT THIS FILE - it is machine generated */ +#include +/* Header for class org_apache_mxnet_LibInfo */ + +#ifndef _Included_org_apache_mxnet_LibInfo +#define _Included_org_apache_mxnet_LibInfo +#ifdef __cplusplus +extern "C" { +#endif +/* + * Class: org_apache_mxnet_LibInfo + * Method: nativeLibInit + * Signature: ()I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_nativeLibInit + (JNIEnv *, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxGetLastError + * Signature: ()Ljava/lang/String; + */ +JNIEXPORT jstring JNICALL Java_org_apache_mxnet_LibInfo_mxGetLastError + (JNIEnv *, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxListAllOpNames + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxListAllOpNames + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: nnGetOpHandle + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_nnGetOpHandle + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxImperativeInvoke + * Signature: (J[J[JLscala/collection/mutable/ArrayBuffer;I[Ljava/lang/String;[Ljava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxImperativeInvoke + (JNIEnv *, jobject, jlong, jlongArray, jlongArray, jobject, jint, jobjectArray, jobjectArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayCreateNone + * Signature: (Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayCreateNone + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayCreateEx + * Signature: ([IIIIIILorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayCreateEx + (JNIEnv *, jobject, jintArray, jint, jint, jint, jint, jint, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayWaitAll + * Signature: ()I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayWaitAll + (JNIEnv *, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayWaitToRead + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayWaitToRead + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxListFunctions + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxListFunctions + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxFuncDescribe + * Signature: (JLorg/apache/mxnet/Base/RefInt;Lorg/apache/mxnet/Base/RefInt;Lorg/apache/mxnet/Base/RefInt;Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFuncDescribe + (JNIEnv *, jobject, jlong, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxFuncGetInfo + * Signature: (JLorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFuncGetInfo + (JNIEnv *, jobject, jlong, jobject, jobject, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxFuncInvoke + * Signature: (J[J[F[J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFuncInvoke + (JNIEnv *, jobject, jlong, jlongArray, jfloatArray, jlongArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxFuncInvokeEx + * Signature: (J[J[F[JI[[B[[B)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFuncInvokeEx + (JNIEnv *, jobject, jlong, jlongArray, jfloatArray, jlongArray, jint, jobjectArray, jobjectArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayGetShape + * Signature: (JLorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayGetShape + (JNIEnv *, jobject, jlong, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArraySyncCopyToCPU + * Signature: (J[BI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySyncCopyToCPU + (JNIEnv *, jobject, jlong, jbyteArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArraySlice + * Signature: (JIILorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySlice + (JNIEnv *, jobject, jlong, jint, jint, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayAt + * Signature: (JILorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayAt + (JNIEnv *, jobject, jlong, jint, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayReshape + * Signature: (JI[ILorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayReshape + (JNIEnv *, jobject, jlong, jint, jintArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArraySyncCopyFromCPU + * Signature: (J[FI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySyncCopyFromCPU + (JNIEnv *, jobject, jlong, jfloatArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxFloat64NDArraySyncCopyFromCPU + * Signature: (J[DI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxFloat64NDArraySyncCopyFromCPU + (JNIEnv *, jobject, jlong, jdoubleArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayLoad + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ArrayBuffer;Lorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayLoad + (JNIEnv *, jobject, jstring, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArraySave + * Signature: (Ljava/lang/String;[J[Ljava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySave + (JNIEnv *, jobject, jstring, jlongArray, jobjectArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayGetContext + * Signature: (JLorg/apache/mxnet/Base/RefInt;Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayGetContext + (JNIEnv *, jobject, jlong, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArraySaveRawBytes + * Signature: (JLscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArraySaveRawBytes + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayLoadFromRawBytes + * Signature: ([BLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayLoadFromRawBytes + (JNIEnv *, jobject, jbyteArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNDArrayGetDType + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNDArrayGetDType + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxInitPSEnv + * Signature: ([Ljava/lang/String;[Ljava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxInitPSEnv + (JNIEnv *, jobject, jobjectArray, jobjectArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreRunServer + * Signature: (JLorg/apache/mxnet/KVServerControllerCallback;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreRunServer + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreGetNumDeadNode + * Signature: (JILorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreGetNumDeadNode + (JNIEnv *, jobject, jlong, jint, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreCreate + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreCreate + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreInit + * Signature: (JI[I[J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreInit + (JNIEnv *, jobject, jlong, jint, jintArray, jlongArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreInitEx + * Signature: (JI[Ljava/lang/String;[J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreInitEx + (JNIEnv *, jobject, jlong, jint, jobjectArray, jlongArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStorePush + * Signature: (JI[I[JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStorePush + (JNIEnv *, jobject, jlong, jint, jintArray, jlongArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStorePushEx + * Signature: (JI[Ljava/lang/String;[JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStorePushEx + (JNIEnv *, jobject, jlong, jint, jobjectArray, jlongArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStorePull + * Signature: (JI[I[JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStorePull + (JNIEnv *, jobject, jlong, jint, jintArray, jlongArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStorePullEx + * Signature: (JI[Ljava/lang/String;[JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStorePullEx + (JNIEnv *, jobject, jlong, jint, jobjectArray, jlongArray, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreSetUpdater + * Signature: (JLorg/apache/mxnet/MXKVStoreUpdater;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreSetUpdater + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreIsWorkerNode + * Signature: (Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreIsWorkerNode + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreGetType + * Signature: (JLorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreGetType + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreSendCommmandToServers + * Signature: (JILjava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreSendCommmandToServers + (JNIEnv *, jobject, jlong, jint, jstring); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreBarrier + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreBarrier + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreGetGroupSize + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreGetGroupSize + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreGetRank + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreGetRank + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreSetBarrierBeforeExit + * Signature: (JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreSetBarrierBeforeExit + (JNIEnv *, jobject, jlong, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxKVStoreFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxKVStoreFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxListDataIters + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxListDataIters + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterCreateIter + * Signature: (J[Ljava/lang/String;[Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterCreateIter + (JNIEnv *, jobject, jlong, jobjectArray, jobjectArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterGetIterInfo + * Signature: (JLorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefString;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterGetIterInfo + (JNIEnv *, jobject, jlong, jobject, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterBeforeFirst + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterBeforeFirst + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterNext + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterNext + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterGetLabel + * Signature: (JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterGetLabel + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterGetData + * Signature: (JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterGetData + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterGetIndex + * Signature: (JLscala/collection/mutable/ListBuffer;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterGetIndex + (JNIEnv *, jobject, jlong, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDataIterGetPadNum + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDataIterGetPadNum + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorOutputs + * Signature: (JLscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorOutputs + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorForward + * Signature: (JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorForward + (JNIEnv *, jobject, jlong, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorBackward + * Signature: (J[J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorBackward + (JNIEnv *, jobject, jlong, jlongArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorPrint + * Signature: (JLorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorPrint + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorSetMonitorCallback + * Signature: (JLorg/apache/mxnet/MXMonitorCallback;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorSetMonitorCallback + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListAtomicSymbolCreators + * Signature: (Lscala/collection/mutable/ListBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListAtomicSymbolCreators + (JNIEnv *, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolGetAtomicSymbolInfo + * Signature: (JLorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolGetAtomicSymbolInfo + (JNIEnv *, jobject, jlong, jobject, jobject, jobject, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCreateAtomicSymbol + * Signature: (J[Ljava/lang/String;[Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCreateAtomicSymbol + (JNIEnv *, jobject, jlong, jobjectArray, jobjectArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolSetAttr + * Signature: (JLjava/lang/String;Ljava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolSetAttr + (JNIEnv *, jobject, jlong, jstring, jstring); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListAttrShallow + * Signature: (JLorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListAttrShallow + (JNIEnv *, jobject, jlong, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListAttr + * Signature: (JLorg/apache/mxnet/Base/RefInt;Lscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListAttr + (JNIEnv *, jobject, jlong, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCompose + * Signature: (JLjava/lang/String;[Ljava/lang/String;[J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCompose + (JNIEnv *, jobject, jlong, jstring, jobjectArray, jlongArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCreateVariable + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCreateVariable + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolGetAttr + * Signature: (JLjava/lang/String;Lorg/apache/mxnet/Base/RefString;Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolGetAttr + (JNIEnv *, jobject, jlong, jstring, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListArguments + * Signature: (JLscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListArguments + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCopy + * Signature: (JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCopy + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListAuxiliaryStates + * Signature: (JLscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListAuxiliaryStates + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolListOutputs + * Signature: (JLscala/collection/mutable/ArrayBuffer;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolListOutputs + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCreateGroup + * Signature: ([JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCreateGroup + (JNIEnv *, jobject, jlongArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolPrint + * Signature: (JLorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolPrint + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolGetInternals + * Signature: (JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolGetInternals + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolInferType + * Signature: (J[Ljava/lang/String;[ILscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolInferType + (JNIEnv *, jobject, jlong, jobjectArray, jintArray, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolInferShape + * Signature: (JI[Ljava/lang/String;[I[ILscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lscala/collection/mutable/ListBuffer;Lorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolInferShape + (JNIEnv *, jobject, jlong, jint, jobjectArray, jintArray, jintArray, jobject, jobject, jobject, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolGetOutput + * Signature: (JILorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolGetOutput + (JNIEnv *, jobject, jlong, jint, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolSaveToJSON + * Signature: (JLorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolSaveToJSON + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCreateFromJSON + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCreateFromJSON + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorBindX + * Signature: (JIII[Ljava/lang/String;[I[II[J[J[I[JLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorBindX + (JNIEnv *, jobject, jlong, jint, jint, jint, jobjectArray, jintArray, jintArray, jint, jlongArray, jlongArray, jintArray, jlongArray, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxExecutorBindEX + * Signature: (JIII[Ljava/lang/String;[I[II[J[J[I[JJLorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxExecutorBindEX + (JNIEnv *, jobject, jlong, jint, jint, jint, jobjectArray, jintArray, jintArray, jint, jlongArray, jlongArray, jintArray, jlongArray, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolSaveToFile + * Signature: (JLjava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolSaveToFile + (JNIEnv *, jobject, jlong, jstring); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolCreateFromFile + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolCreateFromFile + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSymbolFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSymbolFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRandomSeed + * Signature: (I)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRandomSeed + (JNIEnv *, jobject, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxNotifyShutdown + * Signature: ()I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxNotifyShutdown + (JNIEnv *, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOWriterCreate + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOWriterCreate + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOReaderCreate + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOReaderCreate + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOWriterFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOWriterFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOReaderFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOReaderFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOWriterWriteRecord + * Signature: (JLjava/lang/String;I)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOWriterWriteRecord + (JNIEnv *, jobject, jlong, jstring, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOReaderReadRecord + * Signature: (JLorg/apache/mxnet/Base/RefString;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOReaderReadRecord + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOWriterTell + * Signature: (JLorg/apache/mxnet/Base/RefInt;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOWriterTell + (JNIEnv *, jobject, jlong, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRecordIOReaderSeek + * Signature: (JI)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRecordIOReaderSeek + (JNIEnv *, jobject, jlong, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRtcCreate + * Signature: (Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;[J[JLjava/lang/String;Lorg/apache/mxnet/Base/RefLong;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRtcCreate + (JNIEnv *, jobject, jstring, jobjectArray, jobjectArray, jlongArray, jlongArray, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRtcPush + * Signature: (J[J[JIIIIII)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRtcPush + (JNIEnv *, jobject, jlong, jlongArray, jlongArray, jint, jint, jint, jint, jint, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxRtcFree + * Signature: (J)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxRtcFree + (JNIEnv *, jobject, jlong); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxCustomOpRegister + * Signature: (Ljava/lang/String;Lorg/apache/mxnet/CustomOpProp;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxCustomOpRegister + (JNIEnv *, jobject, jstring, jobject); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSetProfilerConfig + * Signature: ([Ljava/lang/String;[Ljava/lang/String;)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSetProfilerConfig + (JNIEnv *, jobject, jobjectArray, jobjectArray); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxSetProfilerState + * Signature: (I)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxSetProfilerState + (JNIEnv *, jobject, jint); + +/* + * Class: org_apache_mxnet_LibInfo + * Method: mxDumpProfile + * Signature: (I)I + */ +JNIEXPORT jint JNICALL Java_org_apache_mxnet_LibInfo_mxDumpProfile + (JNIEnv *, jobject, jint); + +#ifdef __cplusplus +} +#endif +#endif diff --git a/scala-package/packageTest/Makefile b/scala-package/packageTest/Makefile new file mode 100644 index 000000000..8c12c1d04 --- /dev/null +++ b/scala-package/packageTest/Makefile @@ -0,0 +1,91 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +SCALA_VERSION_PROFILE := 2.11 +SCALA_VERSION := 2.11.8 +MXNET_VERSION := "[1.3.0-SNAPSHOT,)" + +MXNET_REPO = https://repository.apache.org/content/repositories/snapshots + +ifeq ($(OS),Windows_NT) + UNAME_S := Windows +else + UNAME_S := $(shell uname -s) +endif + +ifeq ($(UNAME_S), Windows) + # TODO: currently scala package does not support windows + SCALA_PKG_PROFILE := windows +else + ifeq ($(UNAME_S), Darwin) + SCALA_PKG_PROFILE := osx-x86_64-cpu + else + SCALA_PKG_PROFILE := linux-x86_64 + ifeq ($(USE_CUDA), 1) + SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-gpu + else + SCALA_PKG_PROFILE := $(SCALA_PKG_PROFILE)-cpu + endif + endif +endif + +ifeq ($(CI), 1) + MAVEN_ARGS := -B +endif + +PROFILES := -Ptest +ifeq ($(UNIT), 1) + PROFILES := "$(PROFILES),unittest" +endif +ifeq ($(INTEGRATION), 1) + PROFILES := "$(PROFILES),integrationtest" +endif + +ifneq ($(UNIT), 1) + ifneq ($(INTEGRATION), 1) + PROFILES := "$(PROFILES),unittest,integrationtest" + endif +endif + + +clean: + (mvn $(MAVEN_ARGS) clean -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ + -Dmxnet.version=$(MXNET_VERSION) \ + -Dscala.version=$(SCALA_VERSION)) + +testinstall: + (mvn $(MAVEN_ARGS) integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + $(PROFILES) \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ + -Dmxnet.version=$(MXNET_VERSION) \ + -Dscala.version=$(SCALA_VERSION)) + +testlocal: + (mvn $(MAVEN_ARGS) integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + $(PROFILES),fromLocal \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ + -Dmxnet.version=$(MXNET_VERSION) \ + -Dscala.version=$(SCALA_VERSION)) + +testsnapshot: + (mvn $(MAVEN_ARGS) integration-test -Dmxnet.profile=$(SCALA_PKG_PROFILE) \ + $(PROFILES),fromSnapshots \ + -Dmxnet.scalaprofile=$(SCALA_VERSION_PROFILE) \ + -Dmxnet.repo=$(MXNET_REPO) \ + -Dmxnet.version=$(MXNET_VERSION) \ + -Dscala.version=$(SCALA_VERSION)) diff --git a/scala-package/packageTest/README.md b/scala-package/packageTest/README.md new file mode 100644 index 000000000..3f1eeb842 --- /dev/null +++ b/scala-package/packageTest/README.md @@ -0,0 +1,72 @@ +# MXNet Scala Package Test + +This is an project created to run the test suite on a fully packaged mxnet jar. The test suite is found locally but mxnet is from the target jarfile. + +## General Setup + +To setup the packageTest, you must first build your tests. To build the tests, follow these steps from the mxnet main directory: + +1. Build MXNet and the scala package from source following the directions [here](https://mxnet.incubator.apache.org/install/scala_setup.html#source) +2. Build the tests by running `make scalatestcompile`. +3. Follow setup instructions below for your testing goal + +## Running + +There are three different modes of operation for testing based on the location of the jar and where it is coming from: + +### Test Installed Jars + +If you have a jar file, you can install it to your maven cache repository(`~/.m2/repository`). This might be useful if you acquire the .jar file from elsewhere. To install, it is easiest to use `mvn install:install-file -Dfile= -DpomFile=`. If the pom file is not available, you can also run `mvn install:install-file -Dfile= -DgroupId= -DartifactId= -Dversion= -Dpackaging=`. With the full mxnet jar, this might look like `mvn install:install-file -Dfile= -DgroupId=org.apache.mxnet -DartifactId=mxnet-full_2.11-linux-x86_64-cpu -Dversion=1.3.0 -Dpackaging=jar`. + +You can also run `make scalainstall` to install from a local build. + +After installing, run `make testinstall` in the package test directory to run the tests. Note that unless you also install an additional mxnetexamples jar, you can only run the unit tests. + +### Test Local Deployment + +To test the jars that would be produced by a deployment, you can run `make scaladeploylocal` from the main mxnet directory. This produces a local snapshot located at `scala-package/local-snapshot`. To test this local snapshot, run `make testlocal`. + +### Remote Repository Snapshot + +This mode is to test a jar located in a remote repository. The default repository is the apache snapshot repisotory located at `https://repository.apache.org/content/repositories/snapshots`. Note that the actual jar in a repisotory should be located at `$repoUrl/org/apache/mxnet/mxnet-full_$scalaVersion-$osMode/$version/*.jar`. + +Test the snapshot repo using `make testsnapshot` or a different repo using `make testsnapshot MXNET_REPO=$NEW_REPO_URL`. + +### Options + +You are able to run unit tests, integration tests, or both using this utility. To run the unit tests, add the flag `UNIT=1` to make (e.g. `make testsnapshot UNIT=1`). Use `INTEGRATION=1` for integration tests. The default behavior is to run both the unit and integration tests. However, the integration tests require that the mxnet examples be installed in addition to the full mxnet package (see test mode instructions above). + +An additional option, you can specify the mxnet version with `MXNET_VERSION=1.3.1-SNAPSHOT`. + +## Cleaning Up + +You can clean temporary files and target artifacts by running `make scalaclean`. + +## Troubleshooting + +### Missing Examples + +If you fail with the following error +``` +[ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project mxnet-scala-packagetest-examples_2.11: There are test failures -> [Help 1] +[ERROR] +[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. +[ERROR] Re-run Maven using the -X switch to enable full debug logging. +[ERROR] +[ERROR] For more information about the errors and possible solutions, please read the following articles: +[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException +[ERROR] +[ERROR] After correcting the problems, you can resume the build with the command +[ERROR] mvn -rf :mxnet-scala-packagetest-examples_2.11 +Makefile:57: recipe for target 'scalaintegrationtest' failed +make: *** [scalaintegrationtest] Error 1 +``` + +and stacktrace begins with the following, + +``` +*** RUN ABORTED *** + java.lang.NoClassDefFoundError: org/apache/mxnetexamples/Util$ +``` + +you are missing the mxnetexamples package. See your test mode installation section for details. diff --git a/scala-package/packageTest/core/pom.xml b/scala-package/packageTest/core/pom.xml new file mode 100644 index 000000000..bdcd7662f --- /dev/null +++ b/scala-package/packageTest/core/pom.xml @@ -0,0 +1,39 @@ + + + 4.0.0 + + PackageTest + mxnet-scala-packagetest_2.11 + 1.0-SNAPSHOT + ../pom.xml + + + mxnet-scala-packagetest-core_2.11 + MXNet Scala Package Test + pom + + + + unittest + + false + + + + + + + + org.scalatest + scalatest-maven-plugin + 1.0 + + ${project.build.outputDirectory},${project.build.testOutputDirectory},../../core/target/test-classes + + + + + + diff --git a/scala-package/packageTest/core/scripts b/scala-package/packageTest/core/scripts new file mode 120000 index 000000000..f806668aa --- /dev/null +++ b/scala-package/packageTest/core/scripts @@ -0,0 +1 @@ +../../core/scripts \ No newline at end of file diff --git a/scala-package/packageTest/examples/pom.xml b/scala-package/packageTest/examples/pom.xml new file mode 100644 index 000000000..e11be657e --- /dev/null +++ b/scala-package/packageTest/examples/pom.xml @@ -0,0 +1,48 @@ + + + 4.0.0 + + PackageTest + mxnet-scala-packagetest_2.11 + 1.0-SNAPSHOT + ../pom.xml + + + mxnet-scala-packagetest-examples_2.11 + MXNet Scala Package Test + pom + + + + integrationtest + + false + + + + + + + + org.scalatest + scalatest-maven-plugin + 1.0 + + ${project.build.outputDirectory},${project.build.testOutputDirectory},../../examples/target/test-classes + + + + + + + + org.apache.mxnet + mxnet-examples_${mxnet.scalaprofile} + ${mxnet.version} + test + + + + diff --git a/scala-package/packageTest/examples/scripts b/scala-package/packageTest/examples/scripts new file mode 120000 index 000000000..2bba4eeec --- /dev/null +++ b/scala-package/packageTest/examples/scripts @@ -0,0 +1 @@ +../../examples/scripts \ No newline at end of file diff --git a/scala-package/packageTest/infer/pom.xml b/scala-package/packageTest/infer/pom.xml new file mode 100644 index 000000000..7c5a096d6 --- /dev/null +++ b/scala-package/packageTest/infer/pom.xml @@ -0,0 +1,38 @@ + + + 4.0.0 + + PackageTest + mxnet-scala-packagetest_2.11 + 1.0-SNAPSHOT + ../pom.xml + + + mxnet-scala-packagetest-infer_2.11 + MXNet Scala Package Test + pom + + + + unittest + + false + + + + + + + + org.scalatest + scalatest-maven-plugin + 1.0 + + ${project.build.outputDirectory},${project.build.testOutputDirectory},../../infer/target/test-classes + + + + + diff --git a/scala-package/packageTest/pom.xml b/scala-package/packageTest/pom.xml new file mode 100644 index 000000000..f7d9e3b18 --- /dev/null +++ b/scala-package/packageTest/pom.xml @@ -0,0 +1,196 @@ + + + 4.0.0 + PackageTest + mxnet-scala-packagetest_2.11 + 1.0-SNAPSHOT + MXNet Scala Package Test + pom + + + core + infer + + + + + test + + + integrationtest + + examples + + + + fromSnapshots + + + apache-snapshots + ${mxnet.repo} + default + + true + + + + + + fromLocal + + + local-snapshot + file://${basedir}/../deploy/target/repo + + true + + + + + + + + true + + + + + org.apache.mxnet + mxnet-full_${mxnet.scalaprofile}-${mxnet.profile} + ${mxnet.version} + + + org.scala-lang + scala-library + ${scala.version} + + + commons-io + commons-io + 2.4 + + + org.scalatest + scalatest_${mxnet.scalaprofile} + 3.0.4 + test + + + org.scalacheck + scalacheck_${mxnet.scalaprofile} + 1.13.5 + test + + + org.mockito + mockito-all + 1.10.19 + test + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 3.3 + + 1.6 + 1.6 + UTF-8 + + + + maven-resources-plugin + 2.7 + + + org.apache.maven.plugins + maven-dependency-plugin + 2.9 + + + copy-dependencies + package + + copy-dependencies + + + ${project.build.outputDirectory}/lib + runtime + test,provided + false + false + true + + + + + + org.apache.maven.plugins + maven-jar-plugin + 2.5 + + + package + + jar + + + + **/* + + + + + + + net.alchim31.maven + scala-maven-plugin + 3.2.2 + + + compile + + compile + + compile + + + + + org.apache.maven.plugins + maven-surefire-plugin + 2.19 + + true + + + + org.scalatest + scalatest-maven-plugin + 1.0 + + ${skipTests} + ${project.build.directory}/surefire-reports + . + F + WDF TestSuite.txt + + + + test + integration-test + + test + + + + + + + + diff --git a/scala-package/pom.xml b/scala-package/pom.xml index fe78a629e..6665e953d 100644 --- a/scala-package/pom.xml +++ b/scala-package/pom.xml @@ -6,11 +6,12 @@ org.apache apache - 19 + 19 + org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL MXNet Scala Package - Parent https://github.com/apache/incubator-mxnet/tree/master/scala-package @@ -37,8 +38,11 @@ 2.11.8 - 2.11 - + + g++ + $ + ${project.basedir}/.. + true pom @@ -46,46 +50,18 @@ init init-native macros - core native + core infer examples spark assembly + deploy - - release - - - - org.codehaus.mojo - build-helper-maven-plugin - - - generate-sources - - add-source - - - - ${project.build.directory}/genjavadoc - - - - - - - - - scala-2.11 - - 2.11.8 - 2.11 - @@ -115,31 +91,66 @@ - scala-2.12 + osx-x86_64 + + + mac + + + + osx-x86_64 + jnilib + cpu + + + + linux-x86_64 + + + unix + Linux + + - 2.12.4 - 2.12 + linux-x86_64 + so + - org.apache.maven.plugins - maven-enforcer-plugin + org.codehaus.mojo + exec-maven-plugin + 1.6.0 - enforce-versions + init-build-flavor + initialize - enforce + exec - - - - *:*_2.11 - *:*_2.10 - - - + bash + -c 'mkdir -p ${project.build.directory}; if [[ $(ldd ${MXNET_DIR}/lib/libmxnet.so | grep libcuda.so | wc -l) == "0" ]]; then echo flavor=cpu > ${project.build.directory}/flavor.properties; else echo flavor=gpu > ${project.build.directory}/flavor.properties; fi' + + + + + + org.codehaus.mojo + properties-maven-plugin + 1.0.0 + + + read-properties + initialize + + read-project-properties + + + + ${project.build.directory}/flavor.properties + @@ -152,19 +163,25 @@ - org.apache.maven.plugins - maven-release-plugin - - true - false - - - - org.apache.maven.plugins - maven-deploy-plugin - - true - + org.commonjava.maven.plugins + directory-maven-plugin + 0.1 + + + directories + + directory-of + + initialize + + rootdir + + org.apache.mxnet + mxnet-parent + + + + org.apache.maven.plugins @@ -190,8 +207,8 @@ maven-compiler-plugin 3.3 - 1.6 - 1.6 + 1.7 + 1.7 UTF-8 @@ -207,14 +224,18 @@ org.apache.maven.plugins maven-assembly-plugin - 2.5.5 + 3.1.0 org.apache.maven.plugins maven-surefire-plugin - 2.19 + 2.22.0 - true + ${skipJavaTests} + + -Djava.library.path=${project.parent.basedir}/native/target + + false @@ -228,7 +249,6 @@ scalatest-maven-plugin 1.0 - ${skipTests} ${project.build.directory}/surefire-reports . F @@ -253,7 +273,7 @@ org.scalastyle scalastyle-maven-plugin - 0.8.0 + 1.0.0 false true @@ -261,7 +281,7 @@ false ${basedir}/src/main/scala ${basedir}/src/test/scala - scalastyle-config.xml + ${rootdir}/scalastyle-config.xml ${basedir}/target/scalastyle-output.xml UTF-8 @@ -312,28 +332,26 @@ + + + org.apache.maven.plugins + maven-deploy-plugin + + true + + - - org.scala-lang - scala-library - ${scala.version} - - - org.scala-lang - scala-reflect - ${scala.version} - commons-codec commons-codec 1.10 - log4j - log4j - 1.2.17 + org.apache.logging.log4j + log4j-core + 2.11.1 provided @@ -349,7 +367,7 @@ org.scalatest - scalatest_${scala.binary.version} + scalatest_2.11 3.0.4 test @@ -360,13 +378,25 @@ org.scalacheck - scalacheck_${scala.binary.version} + scalacheck_2.11 1.13.5 test + + + + org.scala-lang + scala-library + ${scala.version} + + + org.scala-lang + scala-reflect + ${scala.version} + org.scala-lang.modules - scala-parser-combinators_${scala.binary.version} + scala-parser-combinators_2.11 1.0.4 diff --git a/scala-package/spark/README.md b/scala-package/spark/README.md index 06106648c..503c27903 100644 --- a/scala-package/spark/README.md +++ b/scala-package/spark/README.md @@ -16,7 +16,8 @@ Checkout the [Installation Guide](http://mxnet.io/get_started/setup.html) contai Compile the Scala Package by ```bash -make scalapkg +cd scala-package +mvn package ``` This will automatically build the `spark` submodule. Now you can submit Spark job with these built jars. diff --git a/scala-package/spark/bin/run-mnist-example.sh b/scala-package/spark/bin/run-mnist-example.sh index 4ebd6c61d..4f747f2c9 100755 --- a/scala-package/spark/bin/run-mnist-example.sh +++ b/scala-package/spark/bin/run-mnist-example.sh @@ -27,9 +27,9 @@ OS="" if [ "$(uname)" == "Darwin" ]; then # Do something under Mac OS X platform - OS='osx-x86_64-cpu' + OS='osx-x86_64' elif [ "$(expr substr $(uname -s) 1 5)" == "Linux" ]; then - OS='linux-x86_64-cpu' + OS='linux-x86_64' fi LIB_DIR=${SPARK_MODULE_DIR}/target/classes/lib diff --git a/scala-package/spark/pom.xml b/scala-package/spark/pom.xml index f26ba19ac..f2737e933 100644 --- a/scala-package/spark/pom.xml +++ b/scala-package/spark/pom.xml @@ -5,47 +5,28 @@ 4.0.0 org.apache.mxnet - mxnet-parent_2.11 - 1.3.1-SNAPSHOT + mxnet-parent + INTERNAL ../pom.xml - mxnet-spark_2.11 + mxnet-spark MXNet Scala Package - Spark ML 1.6.3 - - - osx-x86_64-cpu - - osx-x86_64-cpu - - - - linux-x86_64-cpu - - linux-x86_64-cpu - - - - linux-x86_64-gpu - - linux-x86_64-gpu - - - + org.apache.mxnet - mxnet-core_${scala.binary.version} - 1.3.1-SNAPSHOT + mxnet-core + INTERNAL provided org.apache.spark - spark-mllib_${scala.binary.version} + spark-mllib_2.11 ${spark.version} diff --git a/snapcraft.yaml b/snapcraft.yaml index ba5497250..9791cd86f 100644 --- a/snapcraft.yaml +++ b/snapcraft.yaml @@ -1,5 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + name: mxnet -version: '1.3.1' +version: '1.5.0' summary: MXNet is a deep learning framework designed for efficiency and flexibility. description: | MXNet is a deep learning framework designed for both efficiency and diff --git a/src/c_api/c_api.cc b/src/c_api/c_api.cc index 56e318097..80bd60538 100644 --- a/src/c_api/c_api.cc +++ b/src/c_api/c_api.cc @@ -122,7 +122,18 @@ int MXGetGPUCount(int* out) { API_END(); } +// Deprecated: use MXGetGPUMemoryInformation64() instead. int MXGetGPUMemoryInformation(int dev, int *free_mem, int *total_mem) { + API_BEGIN(); + uint64_t free_mem64 = 0UL; + uint64_t total_mem64 = 0UL; + Context::GetGPUMemoryInformation(dev, &free_mem64, &total_mem64); + *free_mem = static_cast(free_mem64); + *total_mem = static_cast(total_mem64); + API_END(); +} + +int MXGetGPUMemoryInformation64(int dev, uint64_t *free_mem, uint64_t *total_mem) { API_BEGIN(); Context::GetGPUMemoryInformation(dev, free_mem, total_mem); API_END(); diff --git a/src/c_api/c_api_common.h b/src/c_api/c_api_common.h index 079b587e9..ecb05bc78 100644 --- a/src/c_api/c_api_common.h +++ b/src/c_api/c_api_common.h @@ -29,37 +29,29 @@ #include #include #include +#include #include #include #include #include -/*! \brief macro to guard beginning and end section of all functions */ -#define API_BEGIN() try { on_enter_api(__FUNCTION__); -/*! \brief every function starts with API_BEGIN(); - and finishes with API_END() or API_END_HANDLE_ERROR */ -#define API_END() } catch(dmlc::Error &_except_) { on_exit_api(); return MXAPIHandleException(_except_); } on_exit_api(); return 0; // NOLINT(*) /*! - * \brief every function starts with API_BEGIN(); - * and finishes with API_END() or API_END_HANDLE_ERROR - * The finally clause contains procedure to cleanup states when an error happens. + * \brief Macros to guard beginning and end section of all functions + * every function starts with API_BEGIN() + * and finishes with API_END() or API_END_HANDLE_ERROR() + * The finally clause contains procedure to cleanup states when an error happens. */ -#define API_END_HANDLE_ERROR(Finalize) } catch(dmlc::Error &_except_) { Finalize; on_exit_api(); return MXAPIHandleException(_except_); } on_exit_api(); return 0; // NOLINT(*) +#ifndef API_BEGIN +#define API_BEGIN MX_API_BEGIN +#endif -/*! - * \brief Set the last error message needed by C API - * \param msg The error message to set. - */ -void MXAPISetLastError(const char* msg); -/*! - * \brief handle exception throwed out - * \param e the exception - * \return the return value of API after exception is handled - */ -inline int MXAPIHandleException(const dmlc::Error &e) { - MXAPISetLastError(e.what()); - return -1; -} +#ifndef API_END +#define API_END MX_API_END +#endif + +#ifndef API_END_HANDLE_ERROR +#define API_END_HANDLE_ERROR MX_API_END_HANDLE_ERROR +#endif using namespace mxnet; @@ -137,10 +129,6 @@ inline void CopyAttr(const nnvm::IndexedGraph& idx, // stores keys that will be converted to __key__ extern const std::vector kHiddenKeys; - -extern void on_enter_api(const char *function); -extern void on_exit_api(); - } // namespace mxnet #endif // MXNET_C_API_C_API_COMMON_H_ diff --git a/src/c_api/c_api_executor.cc b/src/c_api/c_api_executor.cc index 85e9efafe..e2e53c726 100644 --- a/src/c_api/c_api_executor.cc +++ b/src/c_api/c_api_executor.cc @@ -148,8 +148,6 @@ int MXExecutorBindEX(SymbolHandle symbol_handle, NDArrayHandle *aux_states, ExecutorHandle shared_exec, ExecutorHandle *out) { - Executor* exec = nullptr; - API_BEGIN(); nnvm::Symbol *symb = static_cast(symbol_handle); Context ctx = Context::Create(static_cast(dev_type), dev_id); @@ -181,7 +179,7 @@ int MXExecutorBindEX(SymbolHandle symbol_handle, *out = Executor::Bind(*symb, ctx, ctx_map, in_args_vec, arg_grad_vec, grad_req_vec, aux_states_vec, reinterpret_cast(shared_exec)); - API_END_HANDLE_ERROR(delete exec); + API_END(); } /*! @@ -558,8 +556,11 @@ int MXExecutorReshape(int partial_shaping, NDArrayHandle** aux_states, ExecutorHandle shared_exec, ExecutorHandle *out) { + Executor* new_exec = nullptr; + MXAPIThreadLocalEntry *ret = MXAPIThreadLocalStore::Get(); API_BEGIN(); + *out = nullptr; // ensure we can know whether to free executor on early abort // create shape map for in_args and aux_states std::unordered_map kwargs(num_provided_arg_shapes); for (mx_uint i = 0; i < num_provided_arg_shapes; ++i) { @@ -581,8 +582,9 @@ int MXExecutorReshape(int partial_shaping, std::vector aux_state_vec; Executor* exec = static_cast(shared_exec); - *out = exec->Reshape(partial_shaping, allow_up_sizing, ctx, ctx_map, kwargs, + new_exec = exec->Reshape(partial_shaping, allow_up_sizing, ctx, ctx_map, kwargs, &in_arg_vec, &arg_grad_vec, &aux_state_vec); + *out = new_exec; ret->ret_handles.clear(); ret->ret_handles.reserve(in_arg_vec.size()+arg_grad_vec.size()+aux_state_vec.size()); @@ -623,7 +625,7 @@ int MXExecutorReshape(int partial_shaping, *aux_states = &(ret->ret_handles[nd_idx]); nd_idx = ret->ret_handles.size(); } - API_END_HANDLE_ERROR(delete (Executor*)(*out)); + API_END_HANDLE_ERROR(delete new_exec); } int MXExecutorGetOptimizedSymbol(ExecutorHandle handle, diff --git a/src/c_api/c_api_symbolic.cc b/src/c_api/c_api_symbolic.cc index 6cb9ec728..9dc6b50ae 100644 --- a/src/c_api/c_api_symbolic.cc +++ b/src/c_api/c_api_symbolic.cc @@ -276,8 +276,8 @@ int MXSymbolListAttr(SymbolHandle symbol, } *out_size = attr_list.size()/2; ret->ret_vec_charp.clear(); - for (size_t i = 0; i < attr_list.size(); ++i) { - ret->ret_vec_charp.push_back(attr_list[i].c_str()); + for (const auto& attr : attr_list) { + ret->ret_vec_charp.push_back(attr.c_str()); } *out = dmlc::BeginPtr(ret->ret_vec_charp); API_END(); @@ -305,8 +305,8 @@ int MXSymbolListAttrShallow(SymbolHandle symbol, } *out_size = attr_list.size()/2; ret->ret_vec_charp.clear(); - for (size_t i = 0; i < attr_list.size(); ++i) { - ret->ret_vec_charp.push_back(attr_list[i].c_str()); + for (auto &attr : attr_list) { + ret->ret_vec_charp.push_back(attr.c_str()); } *out = dmlc::BeginPtr(ret->ret_vec_charp); API_END(); diff --git a/src/c_api/c_predict_api.cc b/src/c_api/c_predict_api.cc index cae5c68aa..bd599e0b6 100644 --- a/src/c_api/c_predict_api.cc +++ b/src/c_api/c_predict_api.cc @@ -139,11 +139,11 @@ int _CreatePartialOut(const char* symbol_json_str, std::unordered_set arg_names, aux_names; std::vector arg_names_vec = sym.ListInputNames(Symbol::kReadOnlyArgs); std::vector aux_names_vec = sym.ListInputNames(Symbol::kAuxiliaryStates); - for (size_t i = 0; i < arg_names_vec.size(); ++i) { - arg_names.insert(arg_names_vec[i]); + for (const auto &arg_name : arg_names_vec) { + arg_names.insert(arg_name); } - for (size_t i = 0; i < aux_names_vec.size(); ++i) { - aux_names.insert(aux_names_vec[i]); + for (const auto &aux_name : aux_names_vec) { + aux_names.insert(aux_name); } std::vector data; std::vector names; @@ -508,13 +508,13 @@ int MXNDListCreate(const char* nd_file_bytes, ret->keys.resize(arrays.size()); } ret->indptr.push_back(0); - for (size_t i = 0; i < arrays.size(); ++i) { - TShape shape = arrays[i].shape(); + for (auto &array : arrays) { + TShape shape = array.shape(); size_t begin = ret->data.size(); size_t size = shape.Size(); ret->shapes.push_back(shape); ret->data.resize(begin + size); - arrays[i].SyncCopyToCPU(dmlc::BeginPtr(ret->data) + begin, size); + array.SyncCopyToCPU(dmlc::BeginPtr(ret->data) + begin, size); ret->indptr.push_back(begin + size); } *out = ret; diff --git a/src/common/cuda_utils.h b/src/common/cuda_utils.h index 047edde88..0dd9d2db3 100644 --- a/src/common/cuda_utils.h +++ b/src/common/cuda_utils.h @@ -286,22 +286,35 @@ inline DType __device__ CudaMin(DType a, DType b) { class DeviceStore { public: /*! \brief default constructor- only optionally restores previous device */ - explicit DeviceStore(bool restore = true) : restore_(restore) { + explicit DeviceStore(int requested_device = -1, bool restore = true) : + restore_device_(-1), + current_device_(requested_device), + restore_(restore) { if (restore_) CUDA_CALL(cudaGetDevice(&restore_device_)); + if (requested_device != restore_device_) { + SetDevice(requested_device); + } } ~DeviceStore() { - if (restore_) + if (restore_ && + current_device_ != restore_device_ && + current_device_ != -1 && + restore_device_ != -1) CUDA_CALL(cudaSetDevice(restore_device_)); } void SetDevice(int device) { - CUDA_CALL(cudaSetDevice(device)); + if (device != -1) { + CUDA_CALL(cudaSetDevice(device)); + current_device_ = device; + } } private: int restore_device_; + int current_device_; bool restore_; }; diff --git a/src/common/exec_utils.h b/src/common/exec_utils.h index fbe544221..8de6f6525 100644 --- a/src/common/exec_utils.h +++ b/src/common/exec_utils.h @@ -591,14 +591,14 @@ inline nnvm::Graph AssignContext(nnvm::Graph g, g.attrs["device"] = std::make_shared(std::move(device)); g = nnvm::pass::PlaceDevice(g, "__ctx_group__", device_map, "_CrossDeviceCopy"); - const auto& assigned_device = g.GetAttr("device"); + const auto& assigned_devices = g.GetAttr("device"); exec::ContextVector vcontext; - for (size_t i = 0; i < assigned_device.size(); ++i) { - if (assigned_device[i] == -1) { + for (auto context : assigned_devices) { + if (context == -1) { vcontext.push_back(default_ctx); } else { - vcontext.push_back(ctx_list[assigned_device[i]]); + vcontext.push_back(ctx_list[context]); } } diff --git a/src/common/utils.h b/src/common/utils.h index 26889792e..b902b38b9 100644 --- a/src/common/utils.h +++ b/src/common/utils.h @@ -474,6 +474,10 @@ inline void LogStorageFallback(const nnvm::NodeAttrs& attrs, #if MXNET_USE_MKLDNN == 1 if (!MKLDNNEnvSet()) common::LogOnce("MXNET_MKLDNN_ENABLED flag is off. " "You can re-enable by setting MXNET_MKLDNN_ENABLED=1"); + if (GetMKLDNNCacheSize() != -1) common::LogOnce("MXNET_MKLDNN_CACHE_NUM is set." + "Should only be set if " + "your model has variable input shapes, " + "as cache size may grow unbounded"); #endif } @@ -713,6 +717,23 @@ inline void EmplaceBackZeros(const NDArrayStorageType stype, const TShape &shape } } + +/*! + * \brief parallelize copy by OpenMP. + */ +template +inline void ParallelCopy(DType* dst, const DType* src, index_t size) { + static index_t copy_block_size = dmlc::GetEnv("MXNET_CPU_PARALLEL_COPY_SIZE", 200000); + if (size >= copy_block_size) { + #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) + for (index_t i = 0; i < size; ++i) { + dst[i] = src[i]; + } + } else { + std::memcpy(dst, src, sizeof(DType) * size); + } +} + } // namespace common } // namespace mxnet #endif // MXNET_COMMON_UTILS_H_ diff --git a/src/engine/engine.cc b/src/engine/engine.cc index 1c72f33d2..a33f0b2c1 100644 --- a/src/engine/engine.cc +++ b/src/engine/engine.cc @@ -48,7 +48,7 @@ inline Engine* CreateEngine() { ret = CreateNaiveEngine(); #endif - if (ret ==nullptr) { + if (ret == nullptr) { LOG(FATAL) << "Cannot find Engine " << type; } if (!default_engine) { diff --git a/src/engine/stream_manager.h b/src/engine/stream_manager.h index d4ac042ff..516e04bf5 100644 --- a/src/engine/stream_manager.h +++ b/src/engine/stream_manager.h @@ -65,9 +65,6 @@ template RunContext StreamManager::GetRunContext( Context const& ctx) { RunContext ret; -#if MXNET_USE_CUDA - mxnet::common::cuda::DeviceStore device_store; -#endif switch (ctx.dev_mask()) { case cpu::kDevMask: ret = RunContext{ctx, nullptr}; @@ -75,11 +72,11 @@ RunContext StreamManager::GetRunContext( case gpu::kDevMask: { #if MXNET_USE_CUDA std::size_t use_counter; - device_store.SetDevice(ctx.dev_id); { std::lock_guard lock{mutex_}; auto&& counter = gpu_cnt_.at(ctx.dev_id); if (counter == -1) { + mxnet::common::cuda::DeviceStore device_store(ctx.dev_id); for (auto&& i : gpu_streams_.at(ctx.dev_id)) { i = mshadow::NewStream(true, MXNET_USE_CUDNN != 0, ctx.dev_id); } @@ -104,19 +101,16 @@ template RunContext StreamManager::GetIORunContext( Context const& ctx) { RunContext ret; -#if MXNET_USE_CUDA - mxnet::common::cuda::DeviceStore device_store; -#endif switch (ctx.dev_mask()) { case cpu::kDevMask: ret = RunContext{ctx, nullptr}; break; case gpu::kDevMask: { #if MXNET_USE_CUDA - device_store.SetDevice(ctx.dev_id); { std::lock_guard lock{mutex_}; if (gpu_io_streams_.at(ctx.dev_id) == nullptr) { + mxnet::common::cuda::DeviceStore device_store(ctx.dev_id); gpu_io_streams_.at(ctx.dev_id) = mshadow::NewStream(false, false, ctx.dev_id); } } diff --git a/src/engine/threaded_engine.h b/src/engine/threaded_engine.h index a2c1a2b94..ccfd09d64 100644 --- a/src/engine/threaded_engine.h +++ b/src/engine/threaded_engine.h @@ -182,7 +182,7 @@ class ThreadedVar final private: // TODO(hotpxl) change this to spinlock for faster runtime // TODO(hotpxl) consider rename head - /*! \brief inetrnal mutex of the ThreadedVar */ + /*! \brief internal mutex of the ThreadedVar */ std::mutex mutex_; /*! * \brief number of pending reads operation in the variable. diff --git a/src/engine/threaded_engine_pooled.cc b/src/engine/threaded_engine_pooled.cc index 1abb82fd6..c6eb99508 100644 --- a/src/engine/threaded_engine_pooled.cc +++ b/src/engine/threaded_engine_pooled.cc @@ -31,6 +31,9 @@ #include "./threaded_engine.h" #include "./thread_pool.h" #include "./stream_manager.h" +#if MXNET_USE_CUDA +#include "../common/cuda_utils.h" +#endif namespace mxnet { namespace engine { @@ -130,10 +133,13 @@ class ThreadedEnginePooled : public ThreadedEngine { * \param opr_block The operator block. */ void DoExecute(OprBlock* opr_block) { +#if MXNET_USE_CUDA + mxnet::common::cuda::DeviceStore device_store(-1, false); +#endif assert(opr_block->wait.load() == 0); if (opr_block->ctx.dev_mask() == gpu::kDevMask) { #if MXNET_USE_CUDA - CUDA_CALL(cudaSetDevice(opr_block->ctx.dev_id)); + device_store.SetDevice(opr_block->ctx.dev_id); #else // MXNET_USE_CUDA LOG(FATAL) << "Please compile with CUDA enabled"; #endif // MXNET_USE_CUDA diff --git a/src/executor/attach_op_execs_pass.cc b/src/executor/attach_op_execs_pass.cc index a0176fab0..fe3a6bd3d 100644 --- a/src/executor/attach_op_execs_pass.cc +++ b/src/executor/attach_op_execs_pass.cc @@ -58,12 +58,10 @@ class StorageFallbackOpExecutor : public OpExecutor { if (!init_) { pre_temp_buf_.clear(); post_temp_buf_.clear(); - for (size_t i = 0; i < in_array.size(); i++) { - auto &nd = in_array[i]; + for (const auto& nd : in_array) { pre_temp_buf_.emplace_back(nd.shape(), nd.ctx(), true, nd.dtype()); } - for (size_t i = 0; i < out_array.size(); i++) { - auto &nd = out_array[i]; + for (const auto& nd : out_array) { post_temp_buf_.emplace_back(nd.shape(), nd.ctx(), true, nd.dtype()); } init_ = true; diff --git a/src/executor/graph_executor.cc b/src/executor/graph_executor.cc index cc8994767..03413d03b 100644 --- a/src/executor/graph_executor.cc +++ b/src/executor/graph_executor.cc @@ -1145,14 +1145,13 @@ void GraphExecutor::InitCachedOps() { // the variables std::vector use_vars, mutate_vars; - for (size_t i = 0; i < exec->in_array.size(); ++i) { - auto& nd = exec->in_array[i]; + for (const auto& nd : exec->in_array) { use_vars.push_back(nd.var()); } - for (auto& r : exec->op_ctx.requested) { + for (const auto& r : exec->op_ctx.requested) { mutate_vars.push_back(r.var); } - for (auto& nd : exec->out_array) { + for (const auto& nd : exec->out_array) { mutate_vars.push_back(nd.var()); } if (exec->var() != nullptr) { @@ -1581,8 +1580,8 @@ static nnvm::Symbol PartitionGraph(const nnvm::Symbol& src, std::vector aux_state_ctxes(aux_states.size()); size_t i1 = 0, i2 = 0; - for (size_t i = 0; i < input_names.size(); ++i) { - if (i2 < aux_names.size() && aux_names[i2] == input_names[i]) { + for (const auto& input_name : input_names) { + if (i2 < aux_names.size() && aux_names[i2] == input_name) { arg_shapes.push_back(aux_states[i2].shape()); arg_dtypes.push_back(aux_states[i2].dtype()); arg_stypes.push_back(aux_states[i2].storage_type()); @@ -1590,7 +1589,7 @@ static nnvm::Symbol PartitionGraph(const nnvm::Symbol& src, ++i2; } else { CHECK(i1 < arg_names.size()); - CHECK_EQ(arg_names[i1], input_names[i]); + CHECK_EQ(arg_names[i1], input_name); arg_shapes.push_back(in_args->at(i1).shape()); arg_dtypes.push_back(in_args->at(i1).dtype()); arg_stypes.push_back(in_args->at(i1).storage_type()); diff --git a/src/executor/tensorrt_pass.cc b/src/executor/tensorrt_pass.cc index b5fc8d15f..d26704c35 100644 --- a/src/executor/tensorrt_pass.cc +++ b/src/executor/tensorrt_pass.cc @@ -324,10 +324,10 @@ nnvm::NodePtr ConvertNnvmGraphToOnnx(const nnvm::Graph &g, std::unordered_map* const params_map) { auto p = nnvm::Node::Create(); p->attrs.op = nnvm::Op::Get("_trt_op"); - op::TRTParam trt_param = op::nnvm_to_onnx::ConvertNnvmGraphToOnnx(g, params_map); - p->attrs.dict["serialized_output_map"] = trt_param.serialized_output_map; - p->attrs.dict["serialized_input_map"] = trt_param.serialized_input_map; - p->attrs.dict["serialized_onnx_graph"] = trt_param.serialized_onnx_graph; + op::ONNXParam onnx_param = op::nnvm_to_onnx::ConvertNnvmGraphToOnnx(g, params_map); + p->attrs.dict["serialized_output_map"] = onnx_param.serialized_output_map; + p->attrs.dict["serialized_input_map"] = onnx_param.serialized_input_map; + p->attrs.dict["serialized_onnx_graph"] = onnx_param.serialized_onnx_graph; if (p->op()->attr_parser != nullptr) { p->op()->attr_parser(&(p->attrs)); } diff --git a/src/executor/trt_graph_executor.cc b/src/executor/trt_graph_executor.cc index 65dbb2979..ec35fee98 100644 --- a/src/executor/trt_graph_executor.cc +++ b/src/executor/trt_graph_executor.cc @@ -407,14 +407,7 @@ nnvm::Symbol TrtGraphExecutor::GetOptimizedSymbol() { Symbol ret; ret.outputs = std::vector(graph_.outputs.begin(), graph_.outputs.begin() + num_forward_outputs_); - ret = ret.Copy(); - static const Op* trt_op = Op::Get("_trt_op"); - DFSVisit(ret.outputs, [](const nnvm::NodePtr n) { - if (n->op() == trt_op) { - n->attrs.dict.clear(); - } - }); - return ret; + return ret.Copy(); } Executor *TrtGraphExecutor::TensorRTBind(nnvm::Symbol symbol, diff --git a/src/imperative/cached_op.cc b/src/imperative/cached_op.cc index 1f115cd64..f4047d1bd 100644 --- a/src/imperative/cached_op.cc +++ b/src/imperative/cached_op.cc @@ -278,10 +278,10 @@ bool CachedOp::SetForwardGraph( shape_inputs.reserve(inputs.size()); dtype_inputs.reserve(inputs.size()); storage_type_inputs.reserve(inputs.size()); - for (uint32_t i = 0; i < inputs.size(); ++i) { - shape_inputs.emplace_back(inputs[i]->shape()); - dtype_inputs.emplace_back(inputs[i]->dtype()); - storage_type_inputs.emplace_back(inputs[i]->storage_type()); + for (auto input : inputs) { + shape_inputs.emplace_back(input->shape()); + dtype_inputs.emplace_back(input->dtype()); + storage_type_inputs.emplace_back(input->storage_type()); } bool match = true; @@ -658,9 +658,9 @@ void CachedOp::StaticRunOps( arg_dtypes.clear(); arg_shapes.reserve(ndinputs.size()); arg_dtypes.reserve(ndinputs.size()); - for (size_t i = 0; i < ndinputs.size(); ++i) { - arg_shapes.emplace_back(ndinputs[i]->shape()); - arg_dtypes.emplace_back(ndinputs[i]->dtype()); + for (auto& ndinput : ndinputs) { + arg_shapes.emplace_back(ndinput->shape()); + arg_dtypes.emplace_back(ndinput->dtype()); } state.op_states[i] = createop[node.source->op()]( node.source->attrs, default_ctx, arg_shapes, arg_dtypes); @@ -784,7 +784,9 @@ OpStatePtr CachedOp::DynamicForward( states.reserve(idx.num_nodes()); std::vector arrays; arrays.reserve(buff.size()); - for (size_t i = 0; i < buff.size(); ++i) arrays.push_back(&buff[i]); + for (auto& buffered_array : buff) { + arrays.push_back(&buffered_array); + } for (size_t i = 0; i < num_inputs; ++i) { arrays[idx.entry_id(idx.input_nodes()[i], 0)] = inputs[i]; } @@ -855,10 +857,15 @@ OpStatePtr CachedOp::Forward( int prev_bulk_size = Engine::Get()->set_bulk_size(config_.forward_bulk_size); OpStatePtr op_state; - if (config_.static_alloc) { - op_state = StaticForward(default_ctx, inputs, outputs); - } else { - op_state = DynamicForward(default_ctx, inputs, outputs); + try { + if (config_.static_alloc) { + op_state = StaticForward(default_ctx, inputs, outputs); + } else { + op_state = DynamicForward(default_ctx, inputs, outputs); + } + } catch (const dmlc::Error& e) { + Engine::Get()->set_bulk_size(prev_bulk_size); + throw e; } Engine::Get()->set_bulk_size(prev_bulk_size); @@ -907,7 +914,9 @@ void CachedOp::DynamicBackward( buff.resize(idx.num_node_entries()); std::vector arrays; arrays.reserve(buff.size()); - for (size_t i = 0; i < buff.size(); ++i) arrays.push_back(&buff[i]); + for (auto& buffered_array : buff) { + arrays.push_back(&buffered_array); + } for (size_t i = 0; i < inputs.size(); ++i) { if (runtime.info.bwd_input_eid[i] == kEidNotExist) { continue; @@ -1058,10 +1067,15 @@ void CachedOp::Backward( int prev_bulk_size = Engine::Get()->set_bulk_size(config_.backward_bulk_size); - if (config_.static_alloc) { - StaticBackward(retain_graph, state, inputs, reqs, outputs); - } else { - DynamicBackward(retain_graph, state, inputs, reqs, outputs); + try { + if (config_.static_alloc) { + StaticBackward(retain_graph, state, inputs, reqs, outputs); + } else { + DynamicBackward(retain_graph, state, inputs, reqs, outputs); + } + } catch (const dmlc::Error& e) { + Engine::Get()->set_bulk_size(prev_bulk_size); + throw e; } Engine::Get()->set_bulk_size(prev_bulk_size); @@ -1177,8 +1191,9 @@ void CachedOpBackward(const OpStatePtr& state_ptr, in_ptrs.push_back(&(*it)); } CHECK_EQ(in_ptrs.size(), s.op->num_backward_inputs()); - for (size_t i = 0; i < out_bufs.size(); i++) - out_ptrs.push_back(&out_bufs[i]); + for (auto& out_buf : out_bufs) { + out_ptrs.push_back(&out_buf); + } CHECK_EQ(out_ptrs.size(), s.op->num_backward_outputs()); // Set is_training correct for the imperative executor. bool orig_is_train; @@ -1283,8 +1298,8 @@ void CachedOpParamParser(nnvm::NodeAttrs* attrs) { nnvm::Symbol sym; sym.outputs = g.outputs; std::vector > flags; - for (auto it = attrs->dict.begin(); it != attrs->dict.end(); it++) - flags.emplace_back(it->first, it->second); + for (const auto& attr : attrs->dict) + flags.emplace_back(attr.first, attr.second); attrs->parsed = CachedOpPtr(new CachedOp(sym, flags)); } } diff --git a/src/imperative/imperative.cc b/src/imperative/imperative.cc index 0c5ff8417..a381b2384 100644 --- a/src/imperative/imperative.cc +++ b/src/imperative/imperative.cc @@ -106,8 +106,16 @@ OpStatePtr Imperative::Invoke( SetShapeType(ctx, attrs, inputs, outputs, &dispatch_mode); std::vector req; SetWriteInplaceReq(inputs, outputs, &req); - - return InvokeOp(ctx, attrs, inputs, outputs, req, dispatch_mode); + OpStatePtr ret = InvokeOp(ctx, attrs, inputs, outputs, req, dispatch_mode); + // the followinng loop is used for finding out the correct shape when some shapes are dynamic + for (size_t i = 0; i < outputs.size(); i++) { + if (outputs[i]->shape().ndim() == 0) { + // the WaitToRead overhead here does not seem to be avoidable + outputs[i]->WaitToRead(); + outputs[i]->SetShapeFromChunk(); + } + } + return ret; } void Imperative::MarkVariables( @@ -189,8 +197,8 @@ void Imperative::RecordOp( std::vector* p_save_outputs) { MXAPIThreadLocalEntry *local_buff = MXAPIThreadLocalStore::Get(); - for (uint32_t i = 0; i < outputs.size(); ++i) { - CHECK(AGInfo::IsNone(*(outputs[i]))) + for (auto output : outputs) { + CHECK(AGInfo::IsNone(*output)) << "Assigning to NDArrays that are already in a computational graph " << "will cause undefined behavior when evaluating gradients. " << "Please call backward first to clear the graph or do this out side of " @@ -247,8 +255,8 @@ void Imperative::RecordOp( node->inputs[i] = inputs[i]->entry_; } - for (uint32_t i = 0; i < outputs.size(); ++i) { - CHECK(AGInfo::IsNone(*(outputs[i]))) + for (auto output : outputs) { + CHECK(AGInfo::IsNone(*output)) << "Inplace operations (+=, -=, x[:]=, etc) are not supported when " << "recording with autograd."; } @@ -348,7 +356,7 @@ std::vector Imperative::Backward( exec::AggregateGradient, nullptr, nullptr, zero_ops, "_copy"); CHECK_EQ(g_graph.outputs.size(), xs.size()); - for (const auto &e : g_graph.outputs) { + for (const auto& e : g_graph.outputs) { if (e.node->op() == nullptr) { auto node = Node::Create(); node->attrs.op = copy_op; @@ -375,7 +383,9 @@ std::vector Imperative::Backward( std::vector states; std::vector arrays; arrays.reserve(buff.size()); - for (size_t i = 0; i < buff.size(); ++i) arrays.push_back(&buff[i]); + for (auto& buffered_array : buff) { + arrays.push_back(&buffered_array); + } if (create_graph) { states.resize(num_forward_nodes); nnvm::DFSVisit(sym.outputs, [&](const nnvm::NodePtr& n) { @@ -390,12 +400,12 @@ std::vector Imperative::Backward( ref_count[eid] = 1; } }); - for (size_t i = 0; i < ograd_entries.size(); ++i) { - AGInfo& info = AGInfo::Get(ograd_entries[i].node); - if (!idx.exist(ograd_entries[i].node.get())) continue; - size_t eid = idx.entry_id(ograd_entries[i]); + for (auto& ograd_entry : ograd_entries) { + AGInfo& info = AGInfo::Get(ograd_entry.node); + if (!idx.exist(ograd_entry.node.get())) continue; + size_t eid = idx.entry_id(ograd_entry); buff[eid] = info.outputs[0]; - buff[eid].entry_ = ograd_entries[i]; + buff[eid].entry_ = ograd_entry; } } else { states.reserve(num_forward_nodes); @@ -409,10 +419,10 @@ std::vector Imperative::Backward( if (retain_graph || info.grad_req != kNullOp) ref_count[eid] = 1; } } - for (size_t i = 0; i < ograd_entries.size(); ++i) { - if (!idx.exist(ograd_entries[i].node.get())) continue; - AGInfo& info = AGInfo::Get(ograd_entries[i].node); - arrays[idx.entry_id(ograd_entries[i])] = &info.outputs[0]; + for (auto& ograd_entry : ograd_entries) { + if (!idx.exist(ograd_entry.node.get())) continue; + AGInfo& info = AGInfo::Get(ograd_entry.node); + arrays[idx.entry_id(ograd_entry)] = &info.outputs[0]; } } for (size_t i = num_forward_outputs; i < graph.outputs.size(); ++i) { @@ -494,9 +504,16 @@ std::vector Imperative::Backward( bool prev_training = set_is_training(is_train); int prev_bulk_size = Engine::Get()->set_bulk_size(backward_bulk_size_); - RunGraph(retain_graph, idx, arrays, num_forward_nodes, idx.num_nodes(), - std::move(array_reqs), std::move(ref_count), &states, dispatch_modes, - is_recording()); + try { + RunGraph(retain_graph, idx, arrays, num_forward_nodes, idx.num_nodes(), + std::move(array_reqs), std::move(ref_count), &states, dispatch_modes, + is_recording()); + } catch (const dmlc::Error& e) { + Engine::Get()->set_bulk_size(prev_bulk_size); + set_is_recording(prev_recording); + set_is_training(prev_training); + throw e; + } Engine::Get()->set_bulk_size(prev_bulk_size); set_is_recording(prev_recording); diff --git a/src/imperative/imperative_utils.cc b/src/imperative/imperative_utils.cc index c84a3b9be..1a676e620 100644 --- a/src/imperative/imperative_utils.cc +++ b/src/imperative/imperative_utils.cc @@ -79,9 +79,9 @@ void RunGraph( arg_dtypes.clear(); arg_shapes.reserve(ndinputs.size()); arg_dtypes.reserve(ndinputs.size()); - for (size_t i = 0; i < ndinputs.size(); ++i) { - arg_shapes.emplace_back(ndinputs[i]->shape()); - arg_dtypes.emplace_back(ndinputs[i]->dtype()); + for (auto& ndinput : ndinputs) { + arg_shapes.emplace_back(ndinput->shape()); + arg_dtypes.emplace_back(ndinput->dtype()); } states[i] = createop[node.source->op()]( node.source->attrs, ctx, arg_shapes, arg_dtypes); diff --git a/src/imperative/imperative_utils.h b/src/imperative/imperative_utils.h index 69fe2c560..dbfb2b581 100644 --- a/src/imperative/imperative_utils.h +++ b/src/imperative/imperative_utils.h @@ -117,11 +117,13 @@ inline void SetShapeType(const Context& ctx, for (auto& i : outputs) { out_shapes.push_back(i->shape()); } - CHECK(infershape.count(attrs.op)) - << "Operator " << attrs.op->name << " is missing FInferShape attribute"; - CHECK(infershape[attrs.op](attrs, &in_shapes, &out_shapes)); - CHECK_EQ(out_shapes.size(), outputs.size()); - + bool is_dynamic_shape_existing = false; + if (!infershape.count(attrs.op)) { + is_dynamic_shape_existing = true; + } else { + CHECK(infershape[attrs.op](attrs, &in_shapes, &out_shapes)); + CHECK_EQ(out_shapes.size(), outputs.size()); + } // infer type std::vector& in_types = ret->arg_types; in_types.clear(); @@ -178,7 +180,10 @@ inline void SetShapeType(const Context& ctx, for (size_t i = 0; i < outputs.size(); ++i) { NDArrayStorageType storage_type = static_cast(out_storage_types[i]); if (outputs[i]->is_none()) { - if (storage_type == kDefaultStorage) { + if (is_dynamic_shape_existing) { + // once there is dynamic shape somewhere, we could not pre-determine the shape. + *outputs[i] = NDArray(ctx, out_types[i]); + } else if (storage_type == kDefaultStorage) { *outputs[i] = NDArray(out_shapes[i], ctx, true, out_types[i]); } else { *outputs[i] = NDArray(storage_type, out_shapes[i], ctx, true, out_types[i]); diff --git a/src/initialize.cc b/src/initialize.cc index ddda3f18a..de7edd1b1 100644 --- a/src/initialize.cc +++ b/src/initialize.cc @@ -57,11 +57,13 @@ class LibraryInitializer { Engine::Get()->Start(); }, []() { - // Make children single threaded since they are typically workers - dmlc::SetEnv("MXNET_CPU_WORKER_NTHREADS", 1); + // Conservative thread management for multiprocess workers + const size_t mp_worker_threads = dmlc::GetEnv("MXNET_MP_WORKER_NTHREADS", 1); + dmlc::SetEnv("MXNET_CPU_WORKER_NTHREADS", mp_worker_threads); dmlc::SetEnv("OMP_NUM_THREADS", 1); #if MXNET_USE_OPENCV && !__APPLE__ - cv::setNumThreads(0); // disable opencv threading + const size_t mp_cv_num_threads = dmlc::GetEnv("MXNET_MP_OPENCV_NUM_THREADS", 0); + cv::setNumThreads(mp_cv_num_threads); // disable opencv threading #endif // MXNET_USE_OPENCV engine::OpenMP::Get()->set_enabled(false); Engine::Get()->Start(); diff --git a/src/io/image_aug_default.cc b/src/io/image_aug_default.cc index bea2e2c07..f31664709 100644 --- a/src/io/image_aug_default.cc +++ b/src/io/image_aug_default.cc @@ -209,15 +209,14 @@ class DefaultImageAugmenter : public ImageAugmenter { public: // contructor DefaultImageAugmenter() { - rotateM_ = cv::Mat(2, 3, CV_32F); seed_init_state = false; } void Init(const std::vector >& kwargs) override { std::vector > kwargs_left; kwargs_left = param_.InitAllowUnknown(kwargs); - for (size_t i = 0; i < kwargs_left.size(); i++) { - if (!strcmp(kwargs_left[i].first.c_str(), "rotate_list")) { - const char* val = kwargs_left[i].second.c_str(); + for (auto& kwarg : kwargs_left) { + if (!strcmp(kwarg.first.c_str(), "rotate_list")) { + const char* val = kwarg.second.c_str(); const char *end = val + strlen(val); char buf[128]; while (val < end) { @@ -473,18 +472,18 @@ class DefaultImageAugmenter : public ImageAugmenter { param_.saturation)(*prnd); int rand_order[3] = {0, 1, 2}; std::random_shuffle(std::begin(rand_order), std::end(rand_order)); - for (int i = 0; i < 3; ++i) { - if (rand_order[i] == 0) { + for (int i : rand_order) { + if (i == 0) { // brightness res.convertTo(res, -1, alpha_b, 0); } - if (rand_order[i] == 1) { + if (i == 1) { // contrast cvtColor(res, temp_, CV_RGB2GRAY); float gray_mean = cv::mean(temp_)[0]; res.convertTo(res, -1, alpha_c, (1 - alpha_c) * gray_mean); } - if (rand_order[i] == 2) { + if (i == 2) { // saturation cvtColor(res, temp_, CV_RGB2GRAY); cvtColor(temp_, temp_, CV_GRAY2BGR); @@ -550,8 +549,6 @@ class DefaultImageAugmenter : public ImageAugmenter { private: // temporal space cv::Mat temp_; - // rotation param - cv::Mat rotateM_; // eigval and eigvec for adding pca noise // store eigval * eigvec as eigvec float eigvec[3][3] = { { 55.46f * -0.5675f, 4.794f * 0.7192f, 1.148f * 0.4009f }, diff --git a/src/io/image_det_aug_default.cc b/src/io/image_det_aug_default.cc index 15169d8f7..afe5174b7 100644 --- a/src/io/image_det_aug_default.cc +++ b/src/io/image_det_aug_default.cc @@ -349,19 +349,19 @@ class ImageDetLabel { if (!valid) return false; // transform ground-truth labels std::vector new_objects; - for (auto iter = objects_.begin(); iter != objects_.end(); ++iter) { + for (auto& object : objects_) { if (image_det_aug_default_enum::kCenter == crop_emit_mode) { - float center_x = (iter->left + iter->right) * 0.5f; - float center_y = (iter->top + iter->bottom) * 0.5f; + float center_x = (object.left + object.right) * 0.5f; + float center_y = (object.top + object.bottom) * 0.5f; if (!crop_box.contains(cv::Point2f(center_x, center_y))) { continue; } - new_objects.push_back(iter->Project(crop_box)); + new_objects.push_back(object.Project(crop_box)); } else if (image_det_aug_default_enum::kOverlap == crop_emit_mode) { - Rect gt_box = iter->ToRect(); + Rect gt_box = object.ToRect(); float overlap = (crop_box & gt_box).area() / gt_box.area(); if (overlap > emit_overlap_thresh) { - new_objects.push_back(iter->Project(crop_box)); + new_objects.push_back(object.Project(crop_box)); } } } @@ -375,8 +375,8 @@ class ImageDetLabel { */ bool TryPad(const Rect pad_box) { // update all objects inplace - for (auto it = objects_.begin(); it != objects_.end(); ++it) { - *it = it->Project(pad_box); + for (auto& object : objects_) { + object = object.Project(pad_box); } return true; } @@ -384,8 +384,8 @@ class ImageDetLabel { /*! \brief flip image and object coordinates horizontally */ bool TryMirror() { // flip all objects horizontally - for (auto it = objects_.begin(); it != objects_.end(); ++it) { - *it = it->HorizontalFlip(); + for (auto& object : objects_) { + object = object.HorizontalFlip(); } return true; } diff --git a/src/io/image_io.cc b/src/io/image_io.cc index a996a2208..b3f7c40b2 100644 --- a/src/io/image_io.cc +++ b/src/io/image_io.cc @@ -143,11 +143,8 @@ void ImdecodeImpl(int flag, bool to_rgb, void* data, size_t size, cv::Mat dst; if (out->is_none()) { cv::Mat res = cv::imdecode(buf, flag); - if (res.empty()) { - LOG(INFO) << "Decoding failed. Invalid image file."; - *out = NDArray(); - return; - } + CHECK(!res.empty()) << "Decoding failed. Invalid image file."; + *out = NDArray(mshadow::Shape3(res.rows, res.cols, flag == 0 ? 1 : 3), Context::CPU(), false, mshadow::kUint8); dst = cv::Mat(out->shape()[0], out->shape()[1], flag == 0 ? CV_8U : CV_8UC3, @@ -189,6 +186,8 @@ void Imdecode(const nnvm::NodeAttrs& attrs, uint8_t* str_img = inputs[0].data().dptr(); size_t len = inputs[0].shape().Size(); + CHECK(len > 0) << "Input cannot be an empty buffer"; + TShape oshape(3); oshape[2] = param.flag == 0 ? 1 : 3; if (get_jpeg_size(str_img, len, &oshape[1], &oshape[0])) { diff --git a/src/io/iter_image_recordio_2.cc b/src/io/iter_image_recordio_2.cc index f41483955..b567c7297 100644 --- a/src/io/iter_image_recordio_2.cc +++ b/src/io/iter_image_recordio_2.cc @@ -285,9 +285,9 @@ inline bool ImageRecordIOParser2::ParseNext(DataBatch *out) { shape_vec.push_back(param_.label_width); TShape label_shape(shape_vec.begin(), shape_vec.end()); - out->data.at(0) = NDArray(data_shape, Context::CPUPinned(0), false, + out->data.at(0) = NDArray(data_shape, Context::CPU(0), false, mshadow::DataType::kFlag); - out->data.at(1) = NDArray(label_shape, Context::CPUPinned(0), false, + out->data.at(1) = NDArray(label_shape, Context::CPU(0), false, mshadow::DataType::kFlag); unit_size_[0] = param_.data_shape.Size(); unit_size_[1] = param_.label_width; diff --git a/src/kvstore/comm.h b/src/kvstore/comm.h index 581ef81cc..08f6155cb 100644 --- a/src/kvstore/comm.h +++ b/src/kvstore/comm.h @@ -231,9 +231,9 @@ class CommCPU : public Comm { << "BroadcastRowSparse expects row-sparse src NDArray"; CHECK_EQ(src.ctx().dev_mask(), Context::kCPU) << "BroadcastRowSparse with src on gpu context not supported"; - for (size_t i = 0; i < dst.size(); ++i) { - NDArray* out = dst[i].first; - NDArray row_id = dst[i].second; + for (const auto& dst_kv : dst) { + NDArray* out = dst_kv.first; + NDArray row_id = dst_kv.second; CHECK_EQ(out->storage_type(), kRowSparseStorage) << "BroadcastRowSparse expects row_sparse dst NDArray"; CHECK_EQ(row_id.ctx().dev_mask(), Context::kCPU) @@ -621,9 +621,9 @@ class CommDevice : public Comm { CHECK_EQ(src.storage_type(), kRowSparseStorage) << "BroadcastRowSparse expects row-sparse src NDArray"; - for (size_t i = 0; i < dst.size(); ++i) { - NDArray* out = dst[i].first; - NDArray row_id = dst[i].second; + for (const auto& dst_kv : dst) { + NDArray* out = dst_kv.first; + NDArray row_id = dst_kv.second; CHECK_EQ(out->storage_type(), kRowSparseStorage) << "BroadcastRowSparse expects row_sparse dst NDArray"; CHECK_EQ(row_id.ctx(), src.ctx()) @@ -686,17 +686,17 @@ class CommDevice : public Comm { ctx_info[d.dev_id] = std::make_pair(d, 0); } - for (size_t i = 0; i < sorted_key_attrs_.size(); ++i) { - const int key = std::get<0>(sorted_key_attrs_[i]); - const TShape& shape = std::get<1>(sorted_key_attrs_[i]); - const int type = std::get<2>(sorted_key_attrs_[i]); + for (auto& sorted_key_attr : sorted_key_attrs_) { + const int key = std::get<0>(sorted_key_attr); + const TShape& shape = std::get<1>(sorted_key_attr); + const int type = std::get<2>(sorted_key_attr); auto& buf = merge_buf_[key]; Context ctx; size_t min_size = std::numeric_limits::max(); - for (auto it = ctx_info.begin(); it != ctx_info.end(); ++it) { - size_t size = it->second.second; + for (auto& ctx_info_kv : ctx_info) { + size_t size = ctx_info_kv.second.second; if (size <= min_size) { - ctx = it->second.first; + ctx = ctx_info_kv.second.first; min_size = size; } } @@ -724,10 +724,9 @@ class CommDevice : public Comm { int enabled = 0; std::vector p2p(n*n); - // Restores active device to what it was before EnableP2P - mxnet::common::cuda::DeviceStore device_store; for (int i = 0; i < n; ++i) { - device_store.SetDevice(gpus[i]); + // Restores active device to what it was before EnableP2P + mxnet::common::cuda::DeviceStore device_store(gpus[i]); for (int j = 0; j < n; j++) { int access; cudaDeviceCanAccessPeer(&access, gpus[i], gpus[j]); diff --git a/src/kvstore/comm_tree.h b/src/kvstore/comm_tree.h index 8d36803ec..11d99c021 100644 --- a/src/kvstore/comm_tree.h +++ b/src/kvstore/comm_tree.h @@ -77,9 +77,6 @@ class CommDeviceTree : public CommDevice { // BroadcastRowSparse InitMergeBuffer(devs_); InitMergeBufferTree(); - if (dmlc::GetEnv("MXNET_ENABLE_GPU_P2P", 1)) { - EnableP2P(); - } } } @@ -102,7 +99,7 @@ class CommDeviceTree : public CommDevice { if (stype == kDefaultStorage) { // Copy everything into buf.merged for each gpu - for (size_t i = 0; i < src.size(); ++i) { + for (const auto& src_gpu_value : src) { int start = scan_[root][depth_]; int end = scan_[root][depth_+1]; @@ -110,8 +107,8 @@ class CommDeviceTree : public CommDevice { int topo_id = topology[j]; TreeBufferEntry& buf = tree_merge_buf_[topo_id][key]; - if (devs_[topo_id] == src[i].ctx()) { - CopyFromTo(src[i], &(buf.merged[merged_row]), priority); + if (devs_[topo_id] == src_gpu_value.ctx()) { + CopyFromTo(src_gpu_value, &(buf.merged[merged_row]), priority); } } } @@ -328,7 +325,7 @@ class CommDeviceTree : public CommDevice { } private: - void EnableP2P() { + void EnableP2P(std::vector* p2p) { #if MXNET_USE_CUDA std::vector gpus; for (const auto& d : devs_) { @@ -338,10 +335,10 @@ class CommDeviceTree : public CommDevice { } int n = static_cast(gpus.size()); int enabled = 0; - std::vector p2p(n*n); - mxnet::common::cuda::DeviceStore device_store; + p2p->clear(); + p2p->resize(n*n, 0); for (int i = 0; i < n; ++i) { - device_store.SetDevice(gpus[i]); + mxnet::common::cuda::DeviceStore device_store(gpus[i]); for (int j = 0; j < n; j++) { int access; cudaDeviceCanAccessPeer(&access, gpus[i], gpus[j]); @@ -349,7 +346,7 @@ class CommDeviceTree : public CommDevice { cudaError_t e = cudaDeviceEnablePeerAccess(gpus[j], 0); if (e == cudaSuccess || e == cudaErrorPeerAccessAlreadyEnabled) { ++enabled; - p2p[i*n+j] = 1; + (*p2p)[i*n+j] = 1; } } } @@ -363,7 +360,7 @@ class CommDeviceTree : public CommDevice { std::string access(n, '.'); for (int i = 0; i < n; ++i) { for (int j = 0; j < n; ++j) { - access[j] = p2p[i*n+j] ? 'v' : '.'; + access[j] = (*p2p)[i*n+j] ? 'v' : '.'; } LOG(WARNING) << access; } @@ -374,7 +371,9 @@ class CommDeviceTree : public CommDevice { void QueryTopology() { #if MXNET_USE_CUDA std::vector link_matrix(devs_.size()*devs_.size()); - GetP2PWeight(devs_, &link_matrix); + std::vector p2p_matrix(devs_.size()*devs_.size()); + EnableP2P(&p2p_matrix); + GetP2PWeight(devs_, p2p_matrix, &link_matrix); if (backtrack_) LOG(INFO) << "Using Backtracking to generate trees"; else @@ -401,10 +400,10 @@ class CommDeviceTree : public CommDevice { bool delay_alloc = true; std::map key_dist; - for (size_t i = 0; i < tree_sorted_key_attrs_.size(); ++i) { - const int key = std::get<0>(tree_sorted_key_attrs_[i]); - const TShape& shape = std::get<1>(tree_sorted_key_attrs_[i]); - const int type = std::get<2>(tree_sorted_key_attrs_[i]); + for (auto& tree_sorted_key_attr : tree_sorted_key_attrs_) { + const int key = std::get<0>(tree_sorted_key_attr); + const TShape& shape = std::get<1>(tree_sorted_key_attr); + const int type = std::get<2>(tree_sorted_key_attr); if (key_dist.find(shape.Size()) == key_dist.end()) key_dist[shape.Size()] = 1; @@ -485,8 +484,8 @@ class CommDeviceTree : public CommDevice { } } - for (auto it = key_dist.begin(); it != key_dist.end(); ++it) { - LOG(INFO) << "Size " << it->first << " occurs " << it->second << " times"; + for (auto& kv : key_dist) { + LOG(INFO) << "Size " << kv.first << " occurs " << kv.second << " times"; } inited_ = true; } diff --git a/src/kvstore/gpu_topology.h b/src/kvstore/gpu_topology.h index a8801499c..777fb47f9 100644 --- a/src/kvstore/gpu_topology.h +++ b/src/kvstore/gpu_topology.h @@ -123,6 +123,9 @@ inline bool IsConnected(const std::vector& matrix, int num_gpus) { /** * \brief Generate adjacency matrix with row/col numbering from 0, 1, ..., n_gpu * \param devs is a vector of GPU contexts + * \param p2p_matrix is adjacency matrix of P2P connections where + * 0: no P2P connection + * 1: P2P connection * \param matrix is adjacency matrix of link topology graph * where edge weight represents relative performance of NVIDIA GPUs * 0: Self-connection @@ -131,7 +134,9 @@ inline bool IsConnected(const std::vector& matrix, int num_gpus) { * 3: 2 NVLink connections */ template -inline void GetP2PWeight(const std::vector& devs, std::vector* matrix) { +inline void GetP2PWeight(const std::vector& devs, + const std::vector& p2p_matrix, + std::vector* matrix) { int num_gpus = devs.size(); int count = 0; std::vector zero_dev_id(num_gpus, -1); @@ -161,11 +166,61 @@ inline void GetP2PWeight(const std::vector& devs, std::vector* matri } } - // Check that all GPUs have at least 1 NVLink connection - int max_value = 0; - for (unsigned int i = 0; i < max.size(); ++i) { - if (max[i] > max_value) - max_value = max[i]; + // Check that all P2P connections are detected by GetP2PAttribute + // If yes, then continue as before + // If not, then treat fallback to using p2p_matrix (from EnableP2P) + // + // We have observed that with CUDA 9.0 p3.16xlarge: + // + // 0 2 2 3 3 1 1 1 . v v v v . . . + // 2 0 3 2 1 3 1 1 v . v v . v . . + // 2 3 0 3 1 1 2 1 v v . v . . v . + // 3 2 3 0 1 1 1 2 v v v . . . . v + // 3 1 1 1 0 2 2 3 v . . . . v v v + // 1 3 1 1 2 0 3 2 . v . . v . v v + // 1 1 2 1 2 3 0 3 . . v . v v . v + // 1 1 1 2 3 2 3 0 . . . v v v v . + // + // matrix p2p_matrix + // + // Here, they are correctly detected, because the 2s and 3s correspond to + // links that have P2P connections between them. However for CUDA 9.2 p3.16xlarge: + // + // 0 2 2 1 1 1 1 1 . v v v v . . . + // 2 0 1 2 1 1 1 1 v . v v . v . . + // 2 1 0 1 1 1 2 1 v v . v . . v . + // 1 2 1 0 1 1 1 2 v v v . . . . v + // 1 1 1 1 0 2 2 1 v . . . . v v v + // 1 1 1 1 2 0 1 2 . v . . v . v v + // 1 1 2 1 2 1 0 1 . . v . v v . v + // 1 1 1 2 1 2 1 0 . . . v v v v . + // + // matrix p2p_matrix + // + // The fastest connections (3 - double NVLink) are not recognized as being any + if (kLogTree) { + PrintMatrix("matrix", *matrix, num_gpus, num_gpus); + PrintMatrix("p2p_matrix", p2p_matrix, num_gpus, num_gpus); + } + + // different from (1 - non-P2P PCI-E). This is why we fallback to p2p_matrix. + bool matrix_correct = true; + for (unsigned i = 0; i < p2p_matrix.size(); ++i) { + if (p2p_matrix[i] > 0 && (*matrix)[i] == 1) { + matrix_correct = false; + break; + } + } + + if (!matrix_correct) { + LOG(WARNING) << "cudaDeviceGetP2PAttribute incorrect. " + << "Falling back to cudaDeviceEnablePeerAccess for topology detection"; + for (unsigned i = 0; i < p2p_matrix.size(); ++i) { + if (p2p_matrix[i] > 0) + (*matrix)[i] = 2; + else + (*matrix)[i] = 1; + } } // If all GPUs are connected by NVLink, then we can use NVLink only @@ -188,6 +243,8 @@ inline void GetP2PWeight(const std::vector& devs, std::vector* matri matrix_value = (matrix_value == 1) ? 1./num_gpus : matrix_value; } } + if (kLogTree) + PrintMatrix("Weight", *matrix, num_gpus, num_gpus); #else LOG(WARNING) << "GPU required for link topology"; @@ -278,8 +335,8 @@ inline bool KernighanLin(const std::vector& W, std::vector* P, // 0) For every partition, determine if it can be partitioned further. // To do this, we must do a histogram of each partition: - for (unsigned i=0; i < P->size(); ++i) { - histogram[(*P)[i]]++; + for (int partition : *P) { + histogram[partition]++; } bool stop = true; @@ -315,13 +372,13 @@ inline bool KernighanLin(const std::vector& W, std::vector* P, // 1b) Shuffle using random generator std::shuffle(cluster_list.begin(), cluster_list.end(), *gen); - for (unsigned i = 0; i < cluster_list.size(); ++i) { + for (int cluster : cluster_list) { if (first_partition < target_partition) { - int dest = cluster_list[i]; + int dest = cluster; P_temp[dest] = 1; first_partition++; } else { - int dest = cluster_list[i]; + int dest = cluster; P_temp[dest] = -1; } } @@ -758,8 +815,8 @@ inline bool FormTopology(const std::vector& result, std::vector* topo_row, std::vector* scan_row, int depth) { - for (unsigned i = 0; i < result.size(); ++i) - if (result[i] == -1) + for (int result_value : result) + if (result_value == -1) return false; scan_row->push_back(topo_row->size()); @@ -1027,7 +1084,7 @@ inline void ComputeTreesFromRoot(std::vector* W, bool success = true; if (reset == 1) { - // LOG(INFO) << "No valid binary tree found from root " << root << ", try backtracking"; + LOG(INFO) << "No valid binary tree found from root " << root << ", try backtracking"; success = BacktrackGenerateBinaryTree(W, num_elements, root, topo, scan); } else { *topo = topo_temp; @@ -1078,8 +1135,8 @@ inline void ComputeTrees(const std::vector& W, int from = std::min((*topo)[row][col], (*topo)[row][col+1]); int dest = std::max((*topo)[row][col], (*topo)[row][col+1]); if (from != dest) { - adj[from*num_elements+dest] += 1; - adj[dest*num_elements+from] += 1; + adj.at(from*num_elements+dest) += 1; + adj.at(dest*num_elements+from) += 1; } } } diff --git a/src/ndarray/ndarray.cc b/src/ndarray/ndarray.cc index 4ccd3380d..6c4ce9c75 100644 --- a/src/ndarray/ndarray.cc +++ b/src/ndarray/ndarray.cc @@ -124,8 +124,8 @@ NDArray::Chunk::~Chunk() { } #endif if (mem.h.size > 0) Storage::Get()->Free(mem.h); - for (size_t i = 0; i < mem.aux_h.size(); i++) { - if (mem.aux_h[i].size > 0) Storage::Get()->Free(mem.aux_h[i]); + for (const auto& aux : mem.aux_h) { + if (aux.size > 0) Storage::Get()->Free(aux); } } }, shandle.ctx, var); @@ -332,11 +332,10 @@ struct NDArrayDLManager { }; DLManagedTensor* NDArray::ToDLPack() const { + CHECK(!is_none()) << "NDArray is not initialized"; NDArrayDLManager* dlmanager(new NDArrayDLManager); dlmanager->handle = *this; - if (!is_none()) { - dlmanager->tensor.dl_tensor = data().dltensor(); - } + dlmanager->tensor.dl_tensor = dlmanager->handle.data().dltensor(); dlmanager->tensor.manager_ctx = dlmanager; dlmanager->tensor.deleter = [](DLManagedTensor* dlmanager){ delete static_cast(dlmanager->manager_ctx); @@ -456,17 +455,10 @@ void NDArray::Chunk::SetMKLMem(const TShape &shape, int dtype) { mkldnn::memory::dims dims; // These are shapes supprted by MKLDNN. - if (shape.ndim() == 1 || shape.ndim() == 2 || shape.ndim() == 4 - || shape.ndim() == 5) { + if (shape.ndim() >= 1 && shape.ndim() <= 5) { dims.resize(shape.ndim()); for (size_t i = 0; i < dims.size(); i++) dims[i] = shape[i]; - } else if (shape.ndim() == 3) { - // If there are 3 dimensions, we'll force it to 4 dimensions. - dims.resize(shape.ndim() + 1); - dims[0] = 1; - for (size_t i = 0; i < shape.ndim(); i++) - dims[i + 1] = shape[i]; } else { LOG(FATAL) << "MKLDNN doesn't support " << shape.ndim() << " dimensions"; } @@ -474,6 +466,7 @@ void NDArray::Chunk::SetMKLMem(const TShape &shape, int dtype) { switch (dims.size()) { case 1: layout = mkldnn::memory::format::x; break; case 2: layout = mkldnn::memory::format::nc; break; + case 3: layout = mkldnn::memory::format::ncw; break; case 4: layout = mkldnn::memory::format::nchw; break; // This isn't the right layout when the data has 5 dimensions in MXNet. // MXNet interprets 5 dimensions as ncdhw, but MKLDNN doesn't have @@ -1282,17 +1275,17 @@ void CopyFromTo(const NDArray& from, const NDArray *to, int priority) { void ElementwiseSum(const std::vector &source, NDArray *out, int priority) { std::vector const_vars; const_vars.reserve(source.size()); - for (size_t i = 0; i < source.size(); ++i) { - if (source[i].var() != out->var()) { - const_vars.push_back(source[i].var()); + for (const auto& source_array : source) { + if (source_array.var() != out->var()) { + const_vars.push_back(source_array.var()); } - CHECK_EQ(source[i].shape() , out->shape()) + CHECK_EQ(source_array.shape() , out->shape()) << "operands shape mismatch"; if (out->ctx().dev_mask() == Context::kCPU) { - CHECK_EQ(source[i].ctx().dev_mask(), Context::kCPU) + CHECK_EQ(source_array.ctx().dev_mask(), Context::kCPU) << "operands context mismatch"; } else { - CHECK_EQ(source[i].ctx(), out->ctx()) + CHECK_EQ(source_array.ctx(), out->ctx()) << "operands context mismatch"; } } diff --git a/src/ndarray/ndarray_function.cc b/src/ndarray/ndarray_function.cc index 43295d6e1..a613d5a3d 100644 --- a/src/ndarray/ndarray_function.cc +++ b/src/ndarray/ndarray_function.cc @@ -38,13 +38,15 @@ void Copy(const TBlob &from, TBlob *to, RunContext ctx) { MSHADOW_TYPE_SWITCH(to->type_flag_, DType, { if (to->type_flag_ == from.type_flag_) { - mshadow::Copy(to->FlatTo1D(), - from.FlatTo1D()); + const index_t size = from.Size(); + CHECK_EQ(size, to->Size()) << "copying size mismatch, from: " << size * sizeof(DType) + << " bytes, to: " << to->Size() * sizeof(DType) << " bytes."; + common::ParallelCopy(to->dptr(), from.dptr(), size); } else { - MSHADOW_TYPE_SWITCH(from.type_flag_, SrcDType, { - to->FlatTo1D() = - mshadow::expr::tcast(from.FlatTo1D()); - }) + MSHADOW_TYPE_SWITCH(from.type_flag_, SrcDType, { + to->FlatTo1D() = + mshadow::expr::tcast(from.FlatTo1D()); + }) } }) } diff --git a/src/nnvm/graph_editor.cc b/src/nnvm/graph_editor.cc index 1dee3c14e..e3ff3f723 100644 --- a/src/nnvm/graph_editor.cc +++ b/src/nnvm/graph_editor.cc @@ -76,22 +76,21 @@ bool CutGraphInputs(const std::vector &input_entries, std::vector var_nodes; orig_entries->clear(); orig_entries->reserve(input_entries.size()); - for (size_t i = 0; i < input_entries.size(); i++) { - nnvm::NodeEntry *e = input_entries[i]; + for (auto input_entry : input_entries) { // If the node is a variable itself, we may want to skip the node. - if (e->node->is_variable() && skip_var) + if (input_entry->node->is_variable() && skip_var) continue; auto it = std::find_if(orig_entries->begin(), orig_entries->end(), - pred_entry(*e)); + pred_entry(*input_entry)); bool exist = (it != orig_entries->end()); - orig_entries->push_back(*e); + orig_entries->push_back(*input_entry); nnvm::NodePtr n; // If we haven't seen the entry before, we need to create a new var node // for the node entry. if (!exist) { nnvm::Symbol sym; - sym.outputs.push_back(*e); + sym.outputs.push_back(*input_entry); n = nnvm::CreateVariableNode(sym.ListOutputNames()[0]); } else { // Otherwise, we use the var node created before. @@ -100,7 +99,7 @@ bool CutGraphInputs(const std::vector &input_entries, n = var_nodes[idx]; } var_nodes.push_back(n); - *e = nnvm::NodeEntry{n, 0, 0}; + *input_entry = nnvm::NodeEntry{n, 0, 0}; } return true; } diff --git a/src/nnvm/legacy_json_util.cc b/src/nnvm/legacy_json_util.cc index 0697aebbd..a2d14c213 100644 --- a/src/nnvm/legacy_json_util.cc +++ b/src/nnvm/legacy_json_util.cc @@ -76,9 +76,9 @@ Graph UpgradeJSON_FixParsing(Graph g) { n->op()->attr_parser(&(n->attrs)); // add back removed hidden keys - for (const auto &kv : hidden_keys) { + for (const auto& kv : hidden_keys) { bool flag = false; - for (const auto &key : kHiddenKeys) { + for (const auto& key : kHiddenKeys) { size_t pos = kv.first.rfind(key); if (pos == 0 && key.length() == kv.first.length()) { n->attrs.dict["__"+key+"__"] = kv.second; @@ -211,8 +211,8 @@ Graph LoadLegacyJSONPass(Graph g) { << ". Attempting to upgrade..."; upgrading = true; } - for (auto it = upgrader_list.begin(); it != upgrader_list.end(); ++it) { - if (it->first > version) load = it->second(load); + for (auto& upgrader : upgrader_list) { + if (upgrader.first > version) load = upgrader.second(load); } if (upgrading) LOG(INFO) << "Symbol successfully upgraded!"; return load; diff --git a/src/nnvm/legacy_op_util.cc b/src/nnvm/legacy_op_util.cc index 63be619be..4ab777b6a 100644 --- a/src/nnvm/legacy_op_util.cc +++ b/src/nnvm/legacy_op_util.cc @@ -388,13 +388,13 @@ std::vector > OpBackInplaceOption(const NodeAttrs& attrs) { std::vector out_data_index(prop.outputs.size()); int counter = 0; - for (size_t i = 0; i < in_data_index.size(); ++i) { + for (const int& i : in_data_index) { in_data_index[i] = counter++; } - for (size_t i = 0; i < out_grad_index.size(); ++i) { + for (const int& i : out_grad_index) { out_grad_index[i] = counter++; } - for (size_t i = 0; i < out_data_index.size(); ++i) { + for (const int& i : out_data_index) { out_data_index[i] = counter++; } diff --git a/src/operator/bilinear_sampler-inl.h b/src/operator/bilinear_sampler-inl.h index 499d23396..49a5b5e5d 100644 --- a/src/operator/bilinear_sampler-inl.h +++ b/src/operator/bilinear_sampler-inl.h @@ -176,12 +176,12 @@ class BilinearSamplerProp : public OperatorProperty { std::vector *out_type, std::vector *aux_type) const override { int dtype = -1; - for (size_t i = 0; i < in_type->size(); ++i) { + for (int type : *in_type) { if (dtype == -1) { - dtype = in_type->at(i); + dtype = type; } else { - CHECK(in_type->at(i) == dtype || - in_type->at(i) == -1) << + CHECK(type == dtype || + type == -1) << "Non-uniform data type in BilinearSampler"; } } diff --git a/src/operator/c_lapack_api.cc b/src/operator/c_lapack_api.cc new file mode 100644 index 000000000..c6293bf8f --- /dev/null +++ b/src/operator/c_lapack_api.cc @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include "c_lapack_api.h" + +#if (MSHADOW_USE_MKL && MXNET_USE_LAPACK) +#elif MXNET_USE_LAPACK +#else + // use pragma message instead of warning + #pragma message("Warning: lapack usage not enabled, linalg-operators will not be available." \ + " Ensure that lapack library is installed and build with USE_LAPACK=1 to get lapack" \ + " functionalities.") + + // Define compilable stubs. + #define MXNET_LAPACK_CWRAPPER1(func, dtype) \ + int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype* a, int lda) { \ + LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ + return 1; \ + } + + #define MXNET_LAPACK_CWRAPPER2(func, dtype) \ + int MXNET_LAPACK_##func(int matrix_layout, int m, int n, dtype* a, \ + int lda, dtype* tau, dtype* work, int lwork) { \ + LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ + return 1; \ + } + + #define MXNET_LAPACK_CWRAPPER3(func, dtype) \ + int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype *a, \ + int lda, dtype *w, dtype *work, int lwork, \ + int *iwork, int liwork) { \ + LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ + return 1; \ + } + + #define MXNET_LAPACK_UNAVAILABLE(func) \ + int mxnet_lapack_##func(...) { \ + LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ + return 1; \ + } + MXNET_LAPACK_CWRAPPER1(spotrf, float) + MXNET_LAPACK_CWRAPPER1(dpotrf, double) + MXNET_LAPACK_CWRAPPER1(spotri, float) + MXNET_LAPACK_CWRAPPER1(dpotri, double) + + MXNET_LAPACK_UNAVAILABLE(sposv) + MXNET_LAPACK_UNAVAILABLE(dposv) + + MXNET_LAPACK_CWRAPPER2(sgelqf, float) + MXNET_LAPACK_CWRAPPER2(dgelqf, double) + MXNET_LAPACK_CWRAPPER2(sorglq, float) + MXNET_LAPACK_CWRAPPER2(dorglq, double) + + MXNET_LAPACK_CWRAPPER3(ssyevd, float) + MXNET_LAPACK_CWRAPPER3(dsyevd, double) +#endif // MSHADOW_USE_MKL == 0 diff --git a/src/operator/c_lapack_api.h b/src/operator/c_lapack_api.h index 46c8b963f..cd6977554 100644 --- a/src/operator/c_lapack_api.h +++ b/src/operator/c_lapack_api.h @@ -324,42 +324,26 @@ inline void flip(int m, int n, DType *b, int ldb, DType *a, int lda) { #else - // use pragma message instead of warning - #pragma message("Warning: lapack usage not enabled, linalg-operators will not be available." \ - " Ensure that lapack library is installed and build with USE_LAPACK=1 to get lapack" \ - " functionalities.") + #define MXNET_LAPACK_ROW_MAJOR 101 #define MXNET_LAPACK_COL_MAJOR 102 // Define compilable stubs. #define MXNET_LAPACK_CWRAPPER1(func, dtype) \ - inline int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype* a, int lda) { \ - LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ - return 1; \ - } + int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype* a, int lda); #define MXNET_LAPACK_CWRAPPER2(func, dtype) \ - inline int MXNET_LAPACK_##func(int matrix_layout, int m, int n, dtype* a, \ - int lda, dtype* tau, dtype* work, int lwork) { \ - LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ - return 1; \ - } + int MXNET_LAPACK_##func(int matrix_layout, int m, int n, dtype* a, \ + int lda, dtype* tau, dtype* work, int lwork); #define MXNET_LAPACK_CWRAPPER3(func, dtype) \ - inline int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype *a, \ + int MXNET_LAPACK_##func(int matrix_layout, char uplo, int n, dtype *a, \ int lda, dtype *w, dtype *work, int lwork, \ - int *iwork, int liwork) { \ - LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ - return 1; \ - } + int *iwork, int liwork); #define MXNET_LAPACK_UNAVAILABLE(func) \ - inline int mxnet_lapack_##func(...) { \ - LOG(FATAL) << "MXNet build without lapack. Function " << #func << " is not available."; \ - return 1; \ - } - + int mxnet_lapack_##func(...); MXNET_LAPACK_CWRAPPER1(spotrf, float) MXNET_LAPACK_CWRAPPER1(dpotrf, double) MXNET_LAPACK_CWRAPPER1(spotri, float) @@ -375,7 +359,10 @@ inline void flip(int m, int n, DType *b, int ldb, DType *a, int lda) { MXNET_LAPACK_CWRAPPER3(ssyevd, float) MXNET_LAPACK_CWRAPPER3(dsyevd, double) - + #undef MXNET_LAPACK_CWRAPPER1 + #undef MXNET_LAPACK_CWRAPPER2 + #undef MXNET_LAPACK_CWRAPPER3 + #undef MXNET_LAPACK_UNAVAILABLE #endif template diff --git a/src/operator/contrib/adamw-inl.h b/src/operator/contrib/adamw-inl.h new file mode 100644 index 000000000..3d76b33ae --- /dev/null +++ b/src/operator/contrib/adamw-inl.h @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2016 by Contributors + * \file optimizer_op-inl.h + * \brief Optimizer operators + * \author Haibin Lin + */ +#ifndef MXNET_OPERATOR_CONTRIB_ADAMW_INL_H_ +#define MXNET_OPERATOR_CONTRIB_ADAMW_INL_H_ +#include +#include +#include +#include +#include +#include +#include +#include +#include "../operator_common.h" +#include "../mshadow_op.h" +#include "../elemwise_op_common.h" +#include "../mxnet_op.h" + +namespace mxnet { +namespace op { + +struct AdamWParam : public dmlc::Parameter { + float lr; + float beta1; + float beta2; + float epsilon; + float wd; + float eta; + float rescale_grad; + float clip_gradient; + DMLC_DECLARE_PARAMETER(AdamWParam) { + DMLC_DECLARE_FIELD(lr) + .describe("Learning rate"); + DMLC_DECLARE_FIELD(beta1) + .set_default(0.9f) + .describe("The decay rate for the 1st moment estimates."); + DMLC_DECLARE_FIELD(beta2) + .set_default(0.999f) + .describe("The decay rate for the 2nd moment estimates."); + DMLC_DECLARE_FIELD(epsilon) + .set_default(1e-8f) + .describe("A small constant for numerical stability."); + DMLC_DECLARE_FIELD(wd) + .set_default(0.0f) + .describe("Weight decay augments the objective function with a " + "regularization term that penalizes large weights. " + "The penalty scales with the square of the magnitude of each weight."); + DMLC_DECLARE_FIELD(eta) + .describe("Learning rate schedule multiplier"); + DMLC_DECLARE_FIELD(rescale_grad) + .set_default(1.0f) + .describe("Rescale gradient to grad = rescale_grad*grad."); + DMLC_DECLARE_FIELD(clip_gradient) + .set_default(-1.0f) + .describe("Clip gradient to the range of [-clip_gradient, clip_gradient] " + "If clip_gradient <= 0, gradient clipping is turned off. " + "grad = max(min(grad, clip_gradient), -clip_gradient)."); + } +}; + +/* + * \brief adam_w update. + */ +template +inline void AdamWUpdate(const nnvm::NodeAttrs& attrs, + const OpContext &ctx, + const std::vector &inputs, + const std::vector &req, + const std::vector &outputs) { + using namespace mshadow; + using namespace mshadow::expr; + using namespace mshadow_op; + const AdamWParam& param = nnvm::get(attrs.parsed); + Stream* s = ctx.get_stream(); + MSHADOW_REAL_TYPE_SWITCH(inputs[0].type_flag_, DType, { + Tensor weight = inputs[0].FlatTo2D(s); + Tensor grad = inputs[1].FlatTo2D(s); + Tensor mean = inputs[2].FlatTo2D(s); + Tensor var = inputs[3].FlatTo2D(s); + Tensor out = outputs[0].FlatTo2D(s); + + grad = scalar(param.rescale_grad) * grad; + if (param.clip_gradient >= 0.0f) { + mean = scalar(param.beta1)*mean + scalar(1.f-param.beta1) * + F(grad, DType(param.clip_gradient)); + var = scalar(param.beta2)*var + scalar(1.f-param.beta2)*F( + F(grad, DType(param.clip_gradient))); + } else { + mean = scalar(param.beta1)*mean + scalar(1.f-param.beta1) * grad; + var = scalar(param.beta2)*var + scalar(1.f-param.beta2) * F(grad); + } + Assign(out, req[0], + weight - + scalar(param.eta) * (scalar(param.lr) * + mean / (F(var) + scalar(param.epsilon)) + + (scalar(param.wd) * weight))); + }); +} + +} // namespace op +} // namespace mxnet + +#endif // MXNET_OPERATOR_CONTRIB_ADAMW_INL_H_ diff --git a/src/operator/contrib/adamw.cc b/src/operator/contrib/adamw.cc new file mode 100644 index 000000000..94623fe08 --- /dev/null +++ b/src/operator/contrib/adamw.cc @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2016 by Contributors + * \file optimizer_op.cc + * \brief Optimizer operators + * \author Haibin Lin + */ +#include "./adamw-inl.h" + +namespace mxnet { +namespace op { + +DMLC_REGISTER_PARAMETER(AdamWParam); + +NNVM_REGISTER_OP(_contrib_adamw_update) +.describe(R"code(Update function for AdamW optimizer. AdamW is seen as a modification of +Adam by decoupling the weight decay from the optimization steps taken w.r.t. the loss function. + +Adam update consists of the following steps, where g represents gradient and m, v +are 1st and 2nd order moment estimates (mean and variance). + +.. math:: + + g_t = \nabla J(W_{t-1})\\ + m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t\\ + v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2\\ + W_t = W_{t-1} - \eta_t (\alpha \frac{ m_t }{ \sqrt{ v_t } + \epsilon } + wd W_{t-1}) + +It updates the weights using:: + + m = beta1*m + (1-beta1)*grad + v = beta2*v + (1-beta2)*(grad**2) + w -= eta * (learning_rate * m / (sqrt(v) + epsilon) + w * wd) + +)code" ADD_FILELINE) +.set_num_inputs(4) +.set_num_outputs(1) +.set_attr_parser(ParamParser) +.set_attr("FInferShape", ElemwiseShape<4, 1>) +.set_attr("FInferType", ElemwiseType<4, 1>) +.set_attr("FMutateInputs", + [](const nnvm::NodeAttrs& attrs) { + return std::vector{2, 3}; + }) +.set_attr("FCompute", AdamWUpdate) +.add_argument("weight", "NDArray-or-Symbol", "Weight") +.add_argument("grad", "NDArray-or-Symbol", "Gradient") +.add_argument("mean", "NDArray-or-Symbol", "Moving mean") +.add_argument("var", "NDArray-or-Symbol", "Moving variance") +.add_arguments(AdamWParam::__FIELDS__()); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/adamw.cu b/src/operator/contrib/adamw.cu new file mode 100644 index 000000000..b7452f861 --- /dev/null +++ b/src/operator/contrib/adamw.cu @@ -0,0 +1,35 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2018 by Contributors + * \file adamw.cu + * \brief Optimizer operators + * \author Haibin Lin + */ +#include "./adamw-inl.h" + +namespace mxnet { +namespace op { + +NNVM_REGISTER_OP(_contrib_adamw_update) +.set_attr("FCompute", AdamWUpdate); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/adaptive_avg_pooling.cc b/src/operator/contrib/adaptive_avg_pooling.cc index a65f5fe8d..720cf9844 100644 --- a/src/operator/contrib/adaptive_avg_pooling.cc +++ b/src/operator/contrib/adaptive_avg_pooling.cc @@ -205,11 +205,11 @@ NNVM_REGISTER_OP(_contrib_AdaptiveAvgPooling2D) Applies a 2D adaptive average pooling over a 4D input with the shape of (NCHW). The pooling kernel and stride sizes are automatically chosen for desired output sizes. -- If a single integer is provided for output_size, the output size is -(N x C x output_size x output_size) for any input (NCHW). +- If a single integer is provided for output_size, the output size is \ + (N x C x output_size x output_size) for any input (NCHW). -- If a tuple of integers (height, width) are provided for output_size, the output size is -(N x C x height x width) for any input (NCHW). +- If a tuple of integers (height, width) are provided for output_size, the output size is \ + (N x C x height x width) for any input (NCHW). )code" ADD_FILELINE) .set_attr_parser(ParamParser) diff --git a/src/operator/contrib/boolean_mask-inl.h b/src/operator/contrib/boolean_mask-inl.h new file mode 100644 index 000000000..ac0681ba9 --- /dev/null +++ b/src/operator/contrib/boolean_mask-inl.h @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/*! + * Copyright (c) 2018 by Contributors + * \file boolean_mask-inl.h +*/ + +#ifndef MXNET_OPERATOR_CONTRIB_BOOLEAN_MASK_INL_H_ +#define MXNET_OPERATOR_CONTRIB_BOOLEAN_MASK_INL_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../operator_common.h" +#include "../mxnet_op.h" +#include "../tensor/init_op.h" +#include "../mshadow_op.h" +#include "../elemwise_op_common.h" + +namespace mxnet { +namespace op { + +struct BooleanMaskParam : public dmlc::Parameter { + int axis; + DMLC_DECLARE_PARAMETER(BooleanMaskParam) { + DMLC_DECLARE_FIELD(axis).set_default(0) + .describe("An integer that represents the axis in NDArray to mask from."); + } +}; + +template +inline void BooleanMaskForward(const nnvm::NodeAttrs& attrs, + const OpContext &ctx, + const std::vector &inputs, + const std::vector &req, + const std::vector &outputs) { + // TODO(@junrushao1994): This implementation is a proof-of-concept, + // hence very slow actually. Performance should be improved in the future. + CHECK_EQ(inputs.size(), 2U); + CHECK_EQ(outputs.size(), 1U); + const BooleanMaskParam& param = nnvm::get(attrs.parsed); + const int axis = param.axis; + const NDArray &data = inputs[0]; + const NDArray &idx = inputs[1]; + const NDArray &out = outputs[0]; + CHECK_EQ(axis, 0) << "Not supported yet"; + CHECK_EQ(data.shape()[axis], idx.shape()[0]); + CHECK_EQ(idx.shape().ndim(), 1U); + // count the number of 1s in `idx`, so that we could know the output dimension + size_t valid_num = 0; + MSHADOW_TYPE_SWITCH(idx.dtype(), DType, { + DType* idx_dptr = idx.data().dptr(); + int length = idx.shape()[0]; + for (int i = 0; i < length; i++) { + if (idx_dptr[i]) { + ++valid_num; + } + } + }); + // set the output shape forcefully + TShape s = data.shape(); + s[axis] = valid_num; + const_cast(out).Init(s); + // do the copy + MSHADOW_TYPE_SWITCH(idx.dtype(), DType, { + DType* idx_dptr = idx.data().dptr(); + int length = idx.shape()[0]; + mshadow::Stream *stream = ctx.get_stream(); + for (int i = 0, j = 0; i < length; ++i) { + if (idx_dptr[i]) { + NDArray src = data.At(i); + NDArray dst = out.At(j++); + CHECK(src.shape() == dst.shape()); + mxnet_op::copy(stream, dst.data(), src.data()); + } + } + }); +} + +template +inline void BooleanMaskBackward(const nnvm::NodeAttrs& attrs, + const OpContext &ctx, + const std::vector &inputs, + const std::vector &req, + const std::vector &outputs) { + CHECK_EQ(inputs.size(), 3U); + CHECK_EQ(outputs.size(), 2U); + // inputs: {ograd, data, idx} + // outputs: {igrad_data, igrad_idx} + const NDArray& ograd = inputs[0]; + const NDArray& idx = inputs[2]; + const NDArray& igrad_data = outputs[0]; + MSHADOW_TYPE_SWITCH(idx.dtype(), DType, { + DType* idx_dptr = idx.data().dptr(); + int length = idx.shape()[0]; + mshadow::Stream *stream = ctx.get_stream(); + Fill(stream, igrad_data.data(), req[0], 0); + for (int i = 0, j = 0; i < length; ++i) { + if (idx_dptr[i]) { + NDArray src = ograd.At(j++); + NDArray dst = igrad_data.At(i); + CHECK(src.shape() == dst.shape()); + mxnet_op::copy(stream, dst.data(), src.data()); + } + } + }); +} + +} // namespace op +} // namespace mxnet + +#endif // MXNET_OPERATOR_CONTRIB_BOOLEAN_MASK_INL_H_ diff --git a/src/operator/contrib/boolean_mask.cc b/src/operator/contrib/boolean_mask.cc new file mode 100644 index 000000000..2dcafb6b9 --- /dev/null +++ b/src/operator/contrib/boolean_mask.cc @@ -0,0 +1,114 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +/*! + * Copyright (c) 2018 by Contributors + * \file boolean_mask.cc +*/ + +#include "./boolean_mask-inl.h" + +namespace mxnet { +namespace op { + +DMLC_REGISTER_PARAMETER(BooleanMaskParam); + + +bool BooleanMaskType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 2); + CHECK_EQ(out_attrs->size(), 1); + TYPE_ASSIGN_CHECK(*out_attrs, 0, in_attrs->at(0)); + TYPE_ASSIGN_CHECK(*in_attrs, 0, out_attrs->at(0)); + return out_attrs->at(0) != -1; +} + +bool BooleanMaskStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 2); + CHECK_EQ(out_attrs->size(), 1); + for (int &attr : *in_attrs) { + CHECK_EQ(attr, kDefaultStorage) << "Only default storage is supported"; + } + for (int &attr : *out_attrs) { + attr = kDefaultStorage; + } + *dispatch_mode = DispatchMode::kFComputeEx; + return true; +} + +bool BooleanMaskBackStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 3); + CHECK_EQ(out_attrs->size(), 2); + for (int &attr : *in_attrs) { + CHECK_EQ(attr, kDefaultStorage) << "Only default storage is supported"; + } + for (int &attr : *out_attrs) { + attr = kDefaultStorage; + } + for (size_t i = 0; i < out_attrs->size(); i++) + out_attrs->at(i) = kDefaultStorage; + *dispatch_mode = DispatchMode::kFComputeEx; + return true; +} + +NNVM_REGISTER_OP(_contrib_boolean_mask) +.describe(R"code( +Experimental CPU-only support for boolean masking. +Given an n-d NDArray data, and a 1-d NDArray index, +the operator produces an un-predeterminable shaped n-d NDArray out, +which stands for the rows in x where the corresonding element in index is non-zero. + +>>> data = mx.nd.array([[1, 2, 3],[4, 5, 6],[7, 8, 9]]) +>>> index = mx.nd.array([0, 1, 0]) +>>> out = mx.nd.contrib.boolean_mask(data, index) +>>> out + +[[4. 5. 6.]] + + +)code" ADD_FILELINE) +.set_attr_parser(ParamParser) +.set_num_inputs(2) +.set_num_outputs(1) +.set_attr("FInferType", BooleanMaskType) +.set_attr("FComputeEx", BooleanMaskForward) +.set_attr("FInferStorageType", BooleanMaskStorageType) +.set_attr("FGradient", ElemwiseGradUseIn{"_backward_contrib_boolean_mask"}) +.add_argument("data", "NDArray-or-Symbol", "Data") +.add_argument("index", "NDArray-or-Symbol", "Mask") +.add_arguments(BooleanMaskParam::__FIELDS__()); + +NNVM_REGISTER_OP(_backward_contrib_boolean_mask) +.set_num_inputs(3) +.set_num_outputs(2) +.set_attr("TIsBackward", true) +.set_attr("FInferStorageType", BooleanMaskBackStorageType) +.set_attr("FComputeEx", BooleanMaskBackward) +.add_arguments(BooleanMaskParam::__FIELDS__()); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/bounding_box-inl.h b/src/operator/contrib/bounding_box-inl.h index 8e963461e..031dd952d 100644 --- a/src/operator/contrib/bounding_box-inl.h +++ b/src/operator/contrib/bounding_box-inl.h @@ -785,7 +785,7 @@ void BipartiteMatchingForward(const nnvm::NodeAttrs& attrs, .get_with_shape(Shape2(batch_size, col), s); Shape<1> sort_index_shape = Shape1(dshape.Size()); index_t workspace_size = sort_index_shape.Size(); - workspace_size += ((sort_index_shape.Size() * sizeof(int32_t) - 1) / sizeof(DType)) * 2; + workspace_size += (sort_index_shape.Size() * 2 * sizeof(int32_t) - 1) / sizeof(DType) + 1; Tensor workspace = ctx.requested[0] .get_space_typed(Shape1(workspace_size), s); Tensor scores_copy(workspace.dptr_, diff --git a/src/operator/contrib/bounding_box.cc b/src/operator/contrib/bounding_box.cc index 53052ad5a..56925f94d 100644 --- a/src/operator/contrib/bounding_box.cc +++ b/src/operator/contrib/bounding_box.cc @@ -47,22 +47,29 @@ the in_grad will be assigned 0. In summary, gradients are sticked to its boxes, will either be moved or discarded according to its original index in input. -Input requirements: -1. Input tensor have at least 2 dimensions, (n, k), any higher dims will be regarded -as batch, e.g. (a, b, c, d, n, k) == (a*b*c*d, n, k) -2. n is the number of boxes in each batch -3. k is the width of each box item. +Input requirements:: + + 1. Input tensor have at least 2 dimensions, (n, k), any higher dims will be regarded + as batch, e.g. (a, b, c, d, n, k) == (a*b*c*d, n, k) + 2. n is the number of boxes in each batch + 3. k is the width of each box item. By default, a box is [id, score, xmin, ymin, xmax, ymax, ...], additional elements are allowed. + - `id_index`: optional, use -1 to ignore, useful if `force_suppress=False`, which means -we will skip highly overlapped boxes if one is `apple` while the other is `car`. + we will skip highly overlapped boxes if one is `apple` while the other is `car`. + - `coord_start`: required, default=2, the starting index of the 4 coordinates. -Two formats are supported: - `corner`: [xmin, ymin, xmax, ymax] - `center`: [x, y, width, height] + Two formats are supported: + + - `corner`: [xmin, ymin, xmax, ymax] + + - `center`: [x, y, width, height] + - `score_index`: required, default=1, box score/confidence. -When two boxes overlap IOU > `overlap_thresh`, the one with smaller score will be suppressed. + When two boxes overlap IOU > `overlap_thresh`, the one with smaller score will be suppressed. + - `in_format` and `out_format`: default='corner', specify in/out box formats. Examples:: diff --git a/src/operator/contrib/dgl_graph-inl.h b/src/operator/contrib/dgl_graph-inl.h new file mode 100644 index 000000000..f31071b89 --- /dev/null +++ b/src/operator/contrib/dgl_graph-inl.h @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#ifndef MXNET_OPERATOR_CONTRIB_DGL_GRAPH_INL_H_ +#define MXNET_OPERATOR_CONTRIB_DGL_GRAPH_INL_H_ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../operator_common.h" +#include "../mxnet_op.h" +#include "../mshadow_op.h" +#include "../tensor/init_op.h" + +namespace mxnet { +namespace op { + +template +void DGLAdjacencyForwardEx(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + CHECK_EQ(inputs.size(), 1U); + CHECK_EQ(outputs.size(), 1U); + CHECK_EQ(req.size(), 1U); + CHECK_EQ(inputs[0].storage_type(), kCSRStorage); + CHECK_EQ(outputs[0].storage_type(), kCSRStorage); + CHECK_EQ(req[0], kWriteTo); + const TBlob &in_idx = inputs[0].aux_data(csr::kIdx); + const TBlob &in_indptr = inputs[0].aux_data(csr::kIndPtr); + + outputs[0].CheckAndAllocData(in_idx.shape_); + outputs[0].CheckAndAllocAuxData(csr::kIdx, in_idx.shape_); + outputs[0].CheckAndAllocAuxData(csr::kIndPtr, in_indptr.shape_); + + mshadow::Stream *s = ctx.get_stream(); + Fill(s, outputs[0].data(), req[0], 1.0); + mxnet_op::copy(s, outputs[0].aux_data(csr::kIdx), in_idx); + mxnet_op::copy(s, outputs[0].aux_data(csr::kIndPtr), in_indptr); +} + +} // namespace op +} // namespace mxnet + +#endif // MXNET_OPERATOR_CONTRIB_DGL_GRAPH_INL_H_ diff --git a/src/operator/contrib/dgl_graph.cc b/src/operator/contrib/dgl_graph.cc new file mode 100644 index 000000000..a03cbef0b --- /dev/null +++ b/src/operator/contrib/dgl_graph.cc @@ -0,0 +1,1631 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include +#include +#include +#include +#include +#include +#include +#include "../elemwise_op_common.h" +#include "../../imperative/imperative_utils.h" +#include "../subgraph_op_common.h" +#include "./dgl_graph-inl.h" + +namespace mxnet { +namespace op { + +typedef int64_t dgl_id_t; + +////////////////////////////// Graph Sampling /////////////////////////////// + +/* + * ArrayHeap is used to sample elements from vector + */ +class ArrayHeap { + public: + explicit ArrayHeap(const std::vector& prob) { + vec_size_ = prob.size(); + bit_len_ = ceil(log2(vec_size_)); + limit_ = 1 << bit_len_; + // allocate twice the size + heap_.resize(limit_ << 1, 0); + // allocate the leaves + for (int i = limit_; i < vec_size_+limit_; ++i) { + heap_[i] = prob[i-limit_]; + } + // iterate up the tree (this is O(m)) + for (int i = bit_len_-1; i >= 0; --i) { + for (int j = (1 << i); j < (1 << (i + 1)); ++j) { + heap_[j] = heap_[j << 1] + heap_[(j << 1) + 1]; + } + } + } + ~ArrayHeap() {} + + /* + * Remove term from index (this costs O(log m) steps) + */ + void Delete(size_t index) { + size_t i = index + limit_; + float w = heap_[i]; + for (int j = bit_len_; j >= 0; --j) { + heap_[i] -= w; + i = i >> 1; + } + } + + /* + * Add value w to index (this costs O(log m) steps) + */ + void Add(size_t index, float w) { + size_t i = index + limit_; + for (int j = bit_len_; j >= 0; --j) { + heap_[i] += w; + i = i >> 1; + } + } + + /* + * Sample from arrayHeap + */ + size_t Sample(unsigned int* seed) { + float xi = heap_[1] * (rand_r(seed)%100/101.0); + int i = 1; + while (i < limit_) { + i = i << 1; + if (xi >= heap_[i]) { + xi -= heap_[i]; + i += 1; + } + } + return i - limit_; + } + + /* + * Sample a vector by given the size n + */ + void SampleWithoutReplacement(size_t n, std::vector* samples, unsigned int* seed) { + // sample n elements + for (size_t i = 0; i < n; ++i) { + samples->at(i) = this->Sample(seed); + this->Delete(samples->at(i)); + } + } + + private: + int vec_size_; // sample size + int bit_len_; // bit size + int limit_; + std::vector heap_; +}; + +struct NeighborSampleParam : public dmlc::Parameter { + int num_args; + dgl_id_t num_hops; + dgl_id_t num_neighbor; + dgl_id_t max_num_vertices; + DMLC_DECLARE_PARAMETER(NeighborSampleParam) { + DMLC_DECLARE_FIELD(num_args).set_lower_bound(2) + .describe("Number of input NDArray."); + DMLC_DECLARE_FIELD(num_hops) + .set_default(1) + .describe("Number of hops."); + DMLC_DECLARE_FIELD(num_neighbor) + .set_default(2) + .describe("Number of neighbor."); + DMLC_DECLARE_FIELD(max_num_vertices) + .set_default(100) + .describe("Max number of vertices."); + } +}; + +DMLC_REGISTER_PARAMETER(NeighborSampleParam); + +/* + * Check uniform Storage Type + */ +static bool CSRNeighborUniformSampleStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 1; + CHECK_EQ(out_attrs->size(), 3 * num_subgraphs); + + // input[0] is csr_graph + CHECK_EQ(in_attrs->at(0), mxnet::kCSRStorage); + // the rest input ndarray is seed_vector + for (size_t i = 0; i < num_subgraphs; i++) + CHECK_EQ(in_attrs->at(1 + i), mxnet::kDefaultStorage); + + bool success = true; + // sample_id + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i], mxnet::kDefaultStorage)) { + success = false; + } + } + // sub_graph + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i + num_subgraphs], mxnet::kCSRStorage)) { + success = false; + } + } + // sub_layer + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i + 2*num_subgraphs], mxnet::kDefaultStorage)) { + success = false; + } + } + + *dispatch_mode = DispatchMode::kFComputeEx; + + return success; +} + +/* + * Check non-uniform Storage Type + */ +static bool CSRNeighborNonUniformSampleStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 2; + CHECK_EQ(out_attrs->size(), 4 * num_subgraphs); + + // input[0] is csr_graph + CHECK_EQ(in_attrs->at(0), mxnet::kCSRStorage); + // input[1] is probability + CHECK_EQ(in_attrs->at(1), mxnet::kDefaultStorage); + + // the rest input ndarray is seed_vector + for (size_t i = 0; i < num_subgraphs; i++) + CHECK_EQ(in_attrs->at(2 + i), mxnet::kDefaultStorage); + + bool success = true; + // sample_id + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i], mxnet::kDefaultStorage)) { + success = false; + } + } + // sub_graph + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i + num_subgraphs], mxnet::kCSRStorage)) { + success = false; + } + } + // sub_probability + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i + 2*num_subgraphs], mxnet::kDefaultStorage)) { + success = false; + } + } + // sub_layer + for (size_t i = 0; i < num_subgraphs; i++) { + if (!type_assign(&(*out_attrs)[i + 3*num_subgraphs], mxnet::kDefaultStorage)) { + success = false; + } + } + + *dispatch_mode = DispatchMode::kFComputeEx; + + return success; +} + +/* + * Check uniform Shape + */ +static bool CSRNeighborUniformSampleShape(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 1; + CHECK_EQ(out_attrs->size(), 3 * num_subgraphs); + // input[0] is csr graph + CHECK_EQ(in_attrs->at(0).ndim(), 2U); + CHECK_EQ(in_attrs->at(0)[0], in_attrs->at(0)[1]); + + // the rest input ndarray is seed vector + for (size_t i = 0; i < num_subgraphs; i++) { + CHECK_EQ(in_attrs->at(1 + i).ndim(), 1U); + } + + // Output + bool success = true; + TShape out_shape(1); + // We use the last element to store the actual + // number of vertices in the subgraph. + out_shape[0] = params.max_num_vertices + 1; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i, out_shape); + success = success && + out_attrs->at(i).ndim() != 0U && + out_attrs->at(i).Size() != 0U; + } + // sub_csr + TShape out_csr_shape(2); + out_csr_shape[0] = params.max_num_vertices; + out_csr_shape[1] = in_attrs->at(0)[1]; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i + num_subgraphs, out_csr_shape); + success = success && + out_attrs->at(i + num_subgraphs).ndim() != 0U && + out_attrs->at(i + num_subgraphs).Size() != 0U; + } + // sub_layer + TShape out_layer_shape(1); + out_layer_shape[0] = params.max_num_vertices; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i + 2*num_subgraphs, out_layer_shape); + success = success && + out_attrs->at(i + 2*num_subgraphs).ndim() != 0U && + out_attrs->at(i + 2*num_subgraphs).Size() != 0U; + } + + return success; +} + +/* + * Check non-uniform Shape + */ +static bool CSRNeighborNonUniformSampleShape(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 2; + CHECK_EQ(out_attrs->size(), 4 * num_subgraphs); + // input[0] is csr graph + CHECK_EQ(in_attrs->at(0).ndim(), 2U); + CHECK_EQ(in_attrs->at(0)[0], in_attrs->at(0)[1]); + + // input[1] is probability + CHECK_EQ(in_attrs->at(1).ndim(), 1U); + + // the rest ndarray is seed vector + for (size_t i = 0; i < num_subgraphs; i++) { + CHECK_EQ(in_attrs->at(2 + i).ndim(), 1U); + } + + // Output + bool success = true; + TShape out_shape(1); + // We use the last element to store the actual + // number of vertices in the subgraph. + out_shape[0] = params.max_num_vertices + 1; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i, out_shape); + success = success && + out_attrs->at(i).ndim() != 0U && + out_attrs->at(i).Size() != 0U; + } + // sub_csr + TShape out_csr_shape(2); + out_csr_shape[0] = params.max_num_vertices; + out_csr_shape[1] = in_attrs->at(0)[1]; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i + num_subgraphs, out_csr_shape); + success = success && + out_attrs->at(i + num_subgraphs).ndim() != 0U && + out_attrs->at(i + num_subgraphs).Size() != 0U; + } + // sub_probability + TShape out_prob_shape(1); + out_prob_shape[0] = params.max_num_vertices; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i + 2*num_subgraphs, out_prob_shape); + success = success && + out_attrs->at(i + 2*num_subgraphs).ndim() != 0U && + out_attrs->at(i + 2*num_subgraphs).Size() != 0U; + } + // sub_layer + TShape out_layer_shape(1); + out_layer_shape[0] = params.max_num_vertices; + for (size_t i = 0; i < num_subgraphs; i++) { + SHAPE_ASSIGN_CHECK(*out_attrs, i + 3*num_subgraphs, out_prob_shape); + success = success && + out_attrs->at(i + 3*num_subgraphs).ndim() != 0U && + out_attrs->at(i + 3*num_subgraphs).Size() != 0U; + } + + return success; +} + +/* + * Check uniform Type + */ +static bool CSRNeighborUniformSampleType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 1; + CHECK_EQ(out_attrs->size(), 3 * num_subgraphs); + + bool success = true; + for (size_t i = 0; i < num_subgraphs; i++) { + TYPE_ASSIGN_CHECK(*out_attrs, i, in_attrs->at(1)); + TYPE_ASSIGN_CHECK(*out_attrs, i + num_subgraphs, in_attrs->at(0)); + TYPE_ASSIGN_CHECK(*out_attrs, i + 2*num_subgraphs, in_attrs->at(1)); + success = success && + out_attrs->at(i) != -1 && + out_attrs->at(i + num_subgraphs) != -1 && + out_attrs->at(i + 2*num_subgraphs) != -1; + } + + return success; +} + +/* + * Check non-uniform Type + */ +static bool CSRNeighborNonUniformSampleType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + size_t num_subgraphs = params.num_args - 2; + CHECK_EQ(out_attrs->size(), 4 * num_subgraphs); + + bool success = true; + for (size_t i = 0; i < num_subgraphs; i++) { + TYPE_ASSIGN_CHECK(*out_attrs, i, in_attrs->at(2)); + TYPE_ASSIGN_CHECK(*out_attrs, i + num_subgraphs, in_attrs->at(0)); + TYPE_ASSIGN_CHECK(*out_attrs, i + 2*num_subgraphs, in_attrs->at(1)); + TYPE_ASSIGN_CHECK(*out_attrs, i + 3*num_subgraphs, in_attrs->at(2)); + success = success && + out_attrs->at(i) != -1 && + out_attrs->at(i + num_subgraphs) != -1 && + out_attrs->at(i + 2*num_subgraphs) != -1 && + out_attrs->at(i + 3*num_subgraphs) != -1; + } + + return success; +} + +static void RandomSample(size_t set_size, + size_t num, + std::vector* out, + unsigned int* seed) { + std::unordered_set sampled_idxs; + while (sampled_idxs.size() < num) { + sampled_idxs.insert(rand_r(seed) % set_size); + } + out->clear(); + for (auto it = sampled_idxs.begin(); it != sampled_idxs.end(); it++) { + out->push_back(*it); + } +} + +static void NegateSet(const std::vector &idxs, + size_t set_size, + std::vector* out) { + // idxs must have been sorted. + auto it = idxs.begin(); + size_t i = 0; + CHECK_GT(set_size, idxs.back()); + for (; i < set_size && it != idxs.end(); i++) { + if (*it == i) { + it++; + continue; + } + out->push_back(i); + } + for (; i < set_size; i++) { + out->push_back(i); + } +} + +/* + * Uniform sample + */ +static void GetUniformSample(const dgl_id_t* val_list, + const dgl_id_t* col_list, + const size_t ver_len, + const size_t max_num_neighbor, + std::vector* out_ver, + std::vector* out_edge, + unsigned int* seed) { + // Copy ver_list to output + if (ver_len <= max_num_neighbor) { + for (size_t i = 0; i < ver_len; ++i) { + out_ver->push_back(col_list[i]); + out_edge->push_back(val_list[i]); + } + return; + } + // If we just sample a small number of elements from a large neighbor list. + std::vector sorted_idxs; + if (ver_len > max_num_neighbor * 2) { + sorted_idxs.reserve(max_num_neighbor); + RandomSample(ver_len, max_num_neighbor, &sorted_idxs, seed); + std::sort(sorted_idxs.begin(), sorted_idxs.end()); + } else { + std::vector negate; + negate.reserve(ver_len - max_num_neighbor); + RandomSample(ver_len, ver_len - max_num_neighbor, + &negate, seed); + std::sort(negate.begin(), negate.end()); + NegateSet(negate, ver_len, &sorted_idxs); + } + // verify the result. + CHECK_EQ(sorted_idxs.size(), max_num_neighbor); + for (size_t i = 1; i < sorted_idxs.size(); i++) { + CHECK_GT(sorted_idxs[i], sorted_idxs[i - 1]); + } + for (auto idx : sorted_idxs) { + out_ver->push_back(col_list[idx]); + out_edge->push_back(val_list[idx]); + } +} + +/* + * Non-uniform sample via ArrayHeap + */ +static void GetNonUniformSample(const float* probability, + const dgl_id_t* val_list, + const dgl_id_t* col_list, + const size_t ver_len, + const size_t max_num_neighbor, + std::vector* out_ver, + std::vector* out_edge, + unsigned int* seed) { + // Copy ver_list to output + if (ver_len <= max_num_neighbor) { + for (size_t i = 0; i < ver_len; ++i) { + out_ver->push_back(col_list[i]); + out_edge->push_back(val_list[i]); + } + return; + } + // Make sample + std::vector sp_index(max_num_neighbor); + std::vector sp_prob(ver_len); + for (size_t i = 0; i < ver_len; ++i) { + sp_prob[i] = probability[col_list[i]]; + } + ArrayHeap arrayHeap(sp_prob); + arrayHeap.SampleWithoutReplacement(max_num_neighbor, &sp_index, seed); + out_ver->resize(max_num_neighbor); + out_edge->resize(max_num_neighbor); + for (size_t i = 0; i < max_num_neighbor; ++i) { + size_t idx = sp_index[i]; + out_ver->at(i) = col_list[idx]; + out_edge->at(i) = val_list[idx]; + } + sort(out_ver->begin(), out_ver->end()); + sort(out_edge->begin(), out_edge->end()); +} + +/* + * Used for subgraph sampling + */ +struct neigh_list { + std::vector neighs; + std::vector edges; + neigh_list(const std::vector &_neighs, + const std::vector &_edges) + : neighs(_neighs), edges(_edges) {} +}; + +/* + * Sample sub-graph from csr graph + */ +static void SampleSubgraph(const NDArray &csr, + const NDArray &seed_arr, + const NDArray &sampled_ids, + const NDArray &sub_csr, + float* sub_prob, + const NDArray &sub_layer, + const float* probability, + int num_hops, + size_t num_neighbor, + size_t max_num_vertices) { + unsigned int time_seed = time(nullptr); + size_t num_seeds = seed_arr.shape().Size(); + CHECK_GE(max_num_vertices, num_seeds); + + const dgl_id_t* val_list = csr.data().dptr(); + const dgl_id_t* col_list = csr.aux_data(csr::kIdx).dptr(); + const dgl_id_t* indptr = csr.aux_data(csr::kIndPtr).dptr(); + const dgl_id_t* seed = seed_arr.data().dptr(); + dgl_id_t* out = sampled_ids.data().dptr(); + dgl_id_t* out_layer = sub_layer.data().dptr(); + + // BFS traverse the graph and sample vertices + // + std::unordered_set sub_ver_mp; + std::vector > sub_vers; + sub_vers.reserve(num_seeds * 10); + // add seed vertices + for (size_t i = 0; i < num_seeds; ++i) { + auto ret = sub_ver_mp.insert(seed[i]); + // If the vertex is inserted successfully. + if (ret.second) { + sub_vers.emplace_back(seed[i], 0); + } + } + std::vector tmp_sampled_src_list; + std::vector tmp_sampled_edge_list; + // ver_id, position + std::vector > neigh_pos; + neigh_pos.reserve(num_seeds); + std::vector neighbor_list; + size_t num_edges = 0; + + // sub_vers is used both as a node collection and a queue. + // In the while loop, we iterate over sub_vers and new nodes are added to the vector. + // A vertex in the vector only needs to be accessed once. If there is a vertex behind idx + // isn't in the last level, we will sample its neighbors. If not, the while loop terminates. + size_t idx = 0; + while (idx < sub_vers.size() && + sub_ver_mp.size() < max_num_vertices) { + dgl_id_t dst_id = sub_vers[idx].first; + int cur_node_level = sub_vers[idx].second; + idx++; + // If the node is in the last level, we don't need to sample neighbors + // from this node. + if (cur_node_level >= num_hops) + continue; + + tmp_sampled_src_list.clear(); + tmp_sampled_edge_list.clear(); + dgl_id_t ver_len = *(indptr+dst_id+1) - *(indptr+dst_id); + if (probability == nullptr) { // uniform-sample + GetUniformSample(val_list + *(indptr + dst_id), + col_list + *(indptr + dst_id), + ver_len, + num_neighbor, + &tmp_sampled_src_list, + &tmp_sampled_edge_list, + &time_seed); + } else { // non-uniform-sample + GetNonUniformSample(probability, + val_list + *(indptr + dst_id), + col_list + *(indptr + dst_id), + ver_len, + num_neighbor, + &tmp_sampled_src_list, + &tmp_sampled_edge_list, + &time_seed); + } + CHECK_EQ(tmp_sampled_src_list.size(), tmp_sampled_edge_list.size()); + size_t pos = neighbor_list.size(); + neigh_pos.emplace_back(dst_id, pos); + // First we push the size of neighbor vector + neighbor_list.push_back(tmp_sampled_edge_list.size()); + // Then push the vertices + for (size_t i = 0; i < tmp_sampled_src_list.size(); ++i) { + neighbor_list.push_back(tmp_sampled_src_list[i]); + } + // Finally we push the edge list + for (size_t i = 0; i < tmp_sampled_edge_list.size(); ++i) { + neighbor_list.push_back(tmp_sampled_edge_list[i]); + } + num_edges += tmp_sampled_src_list.size(); + for (size_t i = 0; i < tmp_sampled_src_list.size(); ++i) { + // If we have sampled the max number of vertices, we have to stop. + if (sub_ver_mp.size() >= max_num_vertices) + break; + // We need to add the neighbor in the hashtable here. This ensures that + // the vertex in the queue is unique. If we see a vertex before, we don't + // need to add it to the queue again. + auto ret = sub_ver_mp.insert(tmp_sampled_src_list[i]); + // If the sampled neighbor is inserted to the map successfully. + if (ret.second) + sub_vers.emplace_back(tmp_sampled_src_list[i], cur_node_level + 1); + } + } + // Let's check if there is a vertex that we haven't sampled its neighbors. + for (; idx < sub_vers.size(); idx++) { + if (sub_vers[idx].second < num_hops) { + LOG(WARNING) + << "The sampling is truncated because we have reached the max number of vertices\n" + << "Please use a smaller number of seeds or a small neighborhood"; + break; + } + } + + // Copy sub_ver_mp to output[0] + // Copy layer + size_t num_vertices = sub_ver_mp.size(); + std::sort(sub_vers.begin(), sub_vers.end(), + [](const std::pair &a1, const std::pair &a2) { + return a1.first < a2.first; + }); + for (size_t i = 0; i < sub_vers.size(); i++) { + out[i] = sub_vers[i].first; + out_layer[i] = sub_vers[i].second; + } + // The last element stores the actual + // number of vertices in the subgraph. + out[max_num_vertices] = sub_ver_mp.size(); + + // Copy sub_probability + if (sub_prob != nullptr) { + for (size_t i = 0; i < sub_ver_mp.size(); ++i) { + dgl_id_t idx = out[i]; + sub_prob[i] = probability[idx]; + } + } + // Construct sub_csr_graph + TShape shape_1(1); + TShape shape_2(1); + shape_1[0] = num_edges; + shape_2[0] = max_num_vertices+1; + sub_csr.CheckAndAllocData(shape_1); + sub_csr.CheckAndAllocAuxData(csr::kIdx, shape_1); + sub_csr.CheckAndAllocAuxData(csr::kIndPtr, shape_2); + dgl_id_t* val_list_out = sub_csr.data().dptr(); + dgl_id_t* col_list_out = sub_csr.aux_data(1).dptr(); + dgl_id_t* indptr_out = sub_csr.aux_data(0).dptr(); + indptr_out[0] = 0; + size_t collected_nedges = 0; + + // Both the out array and neigh_pos are sorted. By scanning the two arrays, we can see + // which vertices have neighbors and which don't. + std::sort(neigh_pos.begin(), neigh_pos.end(), + [](const std::pair &a1, const std::pair &a2) { + return a1.first < a2.first; + }); + size_t idx_with_neigh = 0; + for (size_t i = 0; i < num_vertices; i++) { + dgl_id_t dst_id = *(out + i); + // If a vertex is in sub_ver_mp but not in neigh_pos, this vertex must not + // have edges. + size_t edge_size = 0; + if (idx_with_neigh < neigh_pos.size() && dst_id == neigh_pos[idx_with_neigh].first) { + size_t pos = neigh_pos[idx_with_neigh].second; + CHECK_LT(pos, neighbor_list.size()); + edge_size = neighbor_list[pos]; + CHECK_LE(pos + edge_size * 2 + 1, neighbor_list.size()); + + std::copy_n(neighbor_list.begin() + pos + 1, + edge_size, + col_list_out + collected_nedges); + std::copy_n(neighbor_list.begin() + pos + edge_size + 1, + edge_size, + val_list_out + collected_nedges); + collected_nedges += edge_size; + idx_with_neigh++; + } + indptr_out[i+1] = indptr_out[i] + edge_size; + } + for (size_t i = num_vertices+1; i <= max_num_vertices; ++i) { + indptr_out[i] = indptr_out[i-1]; + } +} + +/* + * Operator: contrib_csr_neighbor_uniform_sample + */ +static void CSRNeighborUniformSampleComputeExCPU(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + int num_subgraphs = inputs.size() - 1; + CHECK_EQ(outputs.size(), 3 * num_subgraphs); + +#pragma omp parallel for + for (int i = 0; i < num_subgraphs; i++) { + SampleSubgraph(inputs[0], // graph_csr + inputs[i + 1], // seed vector + outputs[i], // sample_id + outputs[i + 1*num_subgraphs], // sub_csr + nullptr, // sample_id_probability + outputs[i + 2*num_subgraphs], // sample_id_layer + nullptr, // probability + params.num_hops, + params.num_neighbor, + params.max_num_vertices); + } +} + +NNVM_REGISTER_OP(_contrib_dgl_csr_neighbor_uniform_sample) +.describe(R"code(This operator samples sub-graphs from a csr graph via an +uniform probability. The operator is designed for DGL. + +The operator outputs three sets of NDArrays to represent the sampled results +(the number of NDArrays in each set is the same as the number of seed NDArrays): +1) a set of 1D NDArrays containing the sampled vertices, 2) a set of CSRNDArrays representing +the sampled edges, 3) a set of 1D NDArrays indicating the layer where a vertex is sampled. +The first set of 1D NDArrays have a length of max_num_vertices+1. The last element in an NDArray +indicate the acutal number of vertices in a subgraph. The third set of NDArrays have a length +of max_num_vertices, and the valid number of vertices is the same as the ones in the first set. + +Example: + + .. code:: python + + shape = (5, 5) + data_np = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], dtype=np.int64) + indices_np = np.array([1,2,3,4,0,2,3,4,0,1,3,4,0,1,2,4,0,1,2,3], dtype=np.int64) + indptr_np = np.array([0,4,8,12,16,20], dtype=np.int64) + a = mx.nd.sparse.csr_matrix((data_np, indices_np, indptr_np), shape=shape) + a.asnumpy() + seed = mx.nd.array([0,1,2,3,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=2, max_num_vertices=5) + + out[0] + [0 1 2 3 4 5] + + + out[1].asnumpy() + array([[ 0, 1, 0, 3, 0], + [ 5, 0, 0, 7, 0], + [ 9, 0, 0, 11, 0], + [13, 0, 15, 0, 0], + [17, 0, 19, 0, 0]]) + + out[2] + [0 0 0 0 0] + + +)code" ADD_FILELINE) +.set_attr_parser(ParamParser) +.set_num_inputs([](const NodeAttrs& attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + return params.num_args; +}) +.set_num_outputs([](const NodeAttrs& attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + size_t num_subgraphs = params.num_args - 1; + return num_subgraphs * 3; +}) +.set_attr("FInferStorageType", CSRNeighborUniformSampleStorageType) +.set_attr("FInferShape", CSRNeighborUniformSampleShape) +.set_attr("FInferType", CSRNeighborUniformSampleType) +.set_attr("FComputeEx", CSRNeighborUniformSampleComputeExCPU) +.add_argument("csr_matrix", "NDArray-or-Symbol", "csr matrix") +.add_argument("seed_arrays", "NDArray-or-Symbol[]", "seed vertices") +.set_attr("key_var_num_args", "num_args") +.add_arguments(NeighborSampleParam::__FIELDS__()); + +/* + * Operator: contrib_csr_neighbor_non_uniform_sample + */ +static void CSRNeighborNonUniformSampleComputeExCPU(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + + int num_subgraphs = inputs.size() - 2; + CHECK_EQ(outputs.size(), 4 * num_subgraphs); + + const float* probability = inputs[1].data().dptr(); + +#pragma omp parallel for + for (int i = 0; i < num_subgraphs; i++) { + float* sub_prob = outputs[i+2*num_subgraphs].data().dptr(); + SampleSubgraph(inputs[0], // graph_csr + inputs[i + 2], // seed vector + outputs[i], // sample_id + outputs[i + 1*num_subgraphs], // sub_csr + sub_prob, // sample_id_probability + outputs[i + 3*num_subgraphs], // sample_id_layer + probability, + params.num_hops, + params.num_neighbor, + params.max_num_vertices); + } +} + +NNVM_REGISTER_OP(_contrib_dgl_csr_neighbor_non_uniform_sample) +.describe(R"code(This operator samples sub-graph from a csr graph via an +non-uniform probability. The operator is designed for DGL. + +The operator outputs four sets of NDArrays to represent the sampled results +(the number of NDArrays in each set is the same as the number of seed NDArrays): +1) a set of 1D NDArrays containing the sampled vertices, 2) a set of CSRNDArrays representing +the sampled edges, 3) a set of 1D NDArrays with the probability that vertices are sampled, +4) a set of 1D NDArrays indicating the layer where a vertex is sampled. +The first set of 1D NDArrays have a length of max_num_vertices+1. The last element in an NDArray +indicate the acutal number of vertices in a subgraph. The third and fourth set of NDArrays have a length +of max_num_vertices, and the valid number of vertices is the same as the ones in the first set. + +Example: + + .. code:: python + + shape = (5, 5) + prob = mx.nd.array([0.9, 0.8, 0.2, 0.4, 0.1], dtype=np.float32) + data_np = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], dtype=np.int64) + indices_np = np.array([1,2,3,4,0,2,3,4,0,1,3,4,0,1,2,4,0,1,2,3], dtype=np.int64) + indptr_np = np.array([0,4,8,12,16,20], dtype=np.int64) + a = mx.nd.sparse.csr_matrix((data_np, indices_np, indptr_np), shape=shape) + seed = mx.nd.array([0,1,2,3,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=2, max_num_vertices=5) + + out[0] + [0 1 2 3 4 5] + + + out[1].asnumpy() + array([[ 0, 1, 2, 0, 0], + [ 5, 0, 6, 0, 0], + [ 9, 10, 0, 0, 0], + [13, 14, 0, 0, 0], + [ 0, 18, 19, 0, 0]]) + + out[2] + [0.9 0.8 0.2 0.4 0.1] + + + out[3] + [0 0 0 0 0] + + +)code" ADD_FILELINE) +.set_attr_parser(ParamParser) +.set_num_inputs([](const NodeAttrs& attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + return params.num_args; +}) +.set_num_outputs([](const NodeAttrs& attrs) { + const NeighborSampleParam& params = + nnvm::get(attrs.parsed); + size_t num_subgraphs = params.num_args - 2; + return num_subgraphs * 4; +}) +.set_attr("FInferStorageType", CSRNeighborNonUniformSampleStorageType) +.set_attr("FInferShape", CSRNeighborNonUniformSampleShape) +.set_attr("FInferType", CSRNeighborNonUniformSampleType) +.set_attr("FComputeEx", CSRNeighborNonUniformSampleComputeExCPU) +.add_argument("csr_matrix", "NDArray-or-Symbol", "csr matrix") +.add_argument("probability", "NDArray-or-Symbol", "probability vector") +.add_argument("seed_arrays", "NDArray-or-Symbol[]", "seed vertices") +.set_attr("key_var_num_args", "num_args") +.add_arguments(NeighborSampleParam::__FIELDS__()); + +///////////////////////// Create induced subgraph /////////////////////////// + +struct DGLSubgraphParam : public dmlc::Parameter { + int num_args; + bool return_mapping; + DMLC_DECLARE_PARAMETER(DGLSubgraphParam) { + DMLC_DECLARE_FIELD(num_args).set_lower_bound(2) + .describe("Number of input arguments, including all symbol inputs."); + DMLC_DECLARE_FIELD(return_mapping) + .describe("Return mapping of vid and eid between the subgraph and the parent graph."); + } +}; // struct DGLSubgraphParam + +DMLC_REGISTER_PARAMETER(DGLSubgraphParam); + +static bool DGLSubgraphStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->at(0), kCSRStorage); + for (size_t i = 1; i < in_attrs->size(); i++) + CHECK_EQ(in_attrs->at(i), kDefaultStorage); + + bool success = true; + *dispatch_mode = DispatchMode::kFComputeEx; + for (size_t i = 0; i < out_attrs->size(); i++) { + if (!type_assign(&(*out_attrs)[i], mxnet::kCSRStorage)) + success = false; + } + return success; +} + +static bool DGLSubgraphShape(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + CHECK_EQ(in_attrs->at(0).ndim(), 2U); + for (size_t i = 1; i < in_attrs->size(); i++) + CHECK_EQ(in_attrs->at(i).ndim(), 1U); + + size_t num_g = params.num_args - 1; + for (size_t i = 0; i < num_g; i++) { + TShape gshape(2); + gshape[0] = in_attrs->at(i + 1)[0]; + gshape[1] = in_attrs->at(i + 1)[0]; + out_attrs->at(i) = gshape; + } + for (size_t i = num_g; i < out_attrs->size(); i++) { + TShape gshape(2); + gshape[0] = in_attrs->at(i - num_g + 1)[0]; + gshape[1] = in_attrs->at(i - num_g + 1)[0]; + out_attrs->at(i) = gshape; + } + return true; +} + +static bool DGLSubgraphType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + size_t num_g = params.num_args - 1; + for (size_t i = 0; i < num_g; i++) { + CHECK_EQ(in_attrs->at(i + 1), mshadow::kInt64); + } + for (size_t i = 0; i < out_attrs->size(); i++) { + out_attrs->at(i) = in_attrs->at(0); + } + return true; +} + +class Bitmap { + const size_t size = 1024 * 1024 * 4; + const size_t mask = size - 1; + std::vector map; + + size_t hash(dgl_id_t id) const { + return id & mask; + } + public: + Bitmap(const dgl_id_t *vid_data, int64_t len): map(size) { + for (int64_t i = 0; i < len; ++i) { + map[hash(vid_data[i])] = 1; + } + } + + bool test(dgl_id_t id) const { + return map[hash(id)]; + } +}; + +/* + * This uses a hashtable to check if a node is in the given node list. + */ +class HashTableChecker { + std::unordered_map oldv2newv; + Bitmap map; + + public: + HashTableChecker(const dgl_id_t *vid_data, int64_t len): map(vid_data, len) { + oldv2newv.reserve(len); + for (int64_t i = 0; i < len; ++i) { + oldv2newv[vid_data[i]] = i; + } + } + + void CollectOnRow(const dgl_id_t col_idx[], const dgl_id_t eids[], size_t row_len, + std::vector *new_col_idx, + std::vector *orig_eids) { + // TODO(zhengda) I need to make sure the column index in each row is sorted. + for (size_t j = 0; j < row_len; ++j) { + const dgl_id_t oldsucc = col_idx[j]; + const dgl_id_t eid = eids[j]; + Collect(oldsucc, eid, new_col_idx, orig_eids); + } + } + + void Collect(const dgl_id_t old_id, const dgl_id_t old_eid, + std::vector *col_idx, + std::vector *orig_eids) { + if (!map.test(old_id)) + return; + + auto it = oldv2newv.find(old_id); + if (it != oldv2newv.end()) { + const dgl_id_t new_id = it->second; + col_idx->push_back(new_id); + if (orig_eids) + orig_eids->push_back(old_eid); + } + } +}; + +static void GetSubgraph(const NDArray &csr_arr, const NDArray &varr, + const NDArray &sub_csr, const NDArray *old_eids) { + const TBlob &data = varr.data(); + int64_t num_vertices = csr_arr.shape()[0]; + const size_t len = varr.shape()[0]; + const dgl_id_t *vid_data = data.dptr(); + HashTableChecker def_check(vid_data, len); + // check if varr is sorted. + CHECK(std::is_sorted(vid_data, vid_data + len)) << "The input vertex list has to be sorted"; + + // Collect the non-zero entries in from the original graph. + std::vector row_idx(len + 1); + std::vector col_idx; + std::vector orig_eids; + col_idx.reserve(len * 50); + orig_eids.reserve(len * 50); + const dgl_id_t *eids = csr_arr.data().dptr(); + const dgl_id_t *indptr = csr_arr.aux_data(csr::kIndPtr).dptr(); + const dgl_id_t *indices = csr_arr.aux_data(csr::kIdx).dptr(); + for (size_t i = 0; i < len; ++i) { + const dgl_id_t oldvid = vid_data[i]; + CHECK_LT(oldvid, num_vertices) << "Vertex Id " << oldvid << " isn't in a graph of " + << num_vertices << " vertices"; + size_t row_start = indptr[oldvid]; + size_t row_len = indptr[oldvid + 1] - indptr[oldvid]; + def_check.CollectOnRow(indices + row_start, eids + row_start, row_len, + &col_idx, old_eids == nullptr ? nullptr : &orig_eids); + + row_idx[i + 1] = col_idx.size(); + } + + TShape nz_shape(1); + nz_shape[0] = col_idx.size(); + TShape indptr_shape(1); + indptr_shape[0] = row_idx.size(); + + // Store the non-zeros in a subgraph with edge attributes of new edge ids. + sub_csr.CheckAndAllocData(nz_shape); + sub_csr.CheckAndAllocAuxData(csr::kIdx, nz_shape); + sub_csr.CheckAndAllocAuxData(csr::kIndPtr, indptr_shape); + dgl_id_t *indices_out = sub_csr.aux_data(csr::kIdx).dptr(); + dgl_id_t *indptr_out = sub_csr.aux_data(csr::kIndPtr).dptr(); + std::copy(col_idx.begin(), col_idx.end(), indices_out); + std::copy(row_idx.begin(), row_idx.end(), indptr_out); + dgl_id_t *sub_eids = sub_csr.data().dptr(); + for (int64_t i = 0; i < nz_shape[0]; i++) + sub_eids[i] = i; + + // Store the non-zeros in a subgraph with edge attributes of old edge ids. + if (old_eids) { + old_eids->CheckAndAllocData(nz_shape); + old_eids->CheckAndAllocAuxData(csr::kIdx, nz_shape); + old_eids->CheckAndAllocAuxData(csr::kIndPtr, indptr_shape); + dgl_id_t *indices_out = old_eids->aux_data(csr::kIdx).dptr(); + dgl_id_t *indptr_out = old_eids->aux_data(csr::kIndPtr).dptr(); + dgl_id_t *sub_eids = old_eids->data().dptr(); + std::copy(col_idx.begin(), col_idx.end(), indices_out); + std::copy(row_idx.begin(), row_idx.end(), indptr_out); + std::copy(orig_eids.begin(), orig_eids.end(), sub_eids); + } +} + +static void DGLSubgraphComputeExCPU(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + int num_g = params.num_args - 1; +#pragma omp parallel for + for (int i = 0; i < num_g; i++) { + const NDArray *old_eids = params.return_mapping ? &outputs[i + num_g] : nullptr; + GetSubgraph(inputs[0], inputs[i + 1], outputs[i], old_eids); + } +} + +NNVM_REGISTER_OP(_contrib_dgl_subgraph) +.describe(R"code(This operator constructs an induced subgraph for +a given set of vertices from a graph. The operator accepts multiple +sets of vertices as input. For each set of vertices, it returns a pair +of CSR matrices if return_mapping is True: the first matrix contains edges +with new edge Ids, the second matrix contains edges with the original +edge Ids. + +Example: + + .. code:: python + + x=[[1, 0, 0, 2], + [3, 0, 4, 0], + [0, 5, 0, 0], + [0, 6, 7, 0]] + v = [0, 1, 2] + dgl_subgraph(x, v, return_mapping=True) = + [[1, 0, 0], + [2, 0, 3], + [0, 4, 0]], + [[1, 0, 0], + [3, 0, 4], + [0, 5, 0]] + +)code" ADD_FILELINE) +.set_attr_parser(ParamParser) +.set_num_inputs([](const NodeAttrs& attrs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + return params.num_args; +}) +.set_num_outputs([](const NodeAttrs& attrs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + int num_varray = params.num_args - 1; + if (params.return_mapping) + return num_varray * 2; + else + return num_varray; +}) +.set_attr("FListInputNames", + [](const NodeAttrs& attrs) { + const DGLSubgraphParam& params = nnvm::get(attrs.parsed); + std::vector names; + names.reserve(params.num_args); + names.emplace_back("graph"); + for (int i = 1; i < params.num_args; ++i) + names.push_back("varray" + std::to_string(i - 1)); + return names; +}) +.set_attr("FInferStorageType", DGLSubgraphStorageType) +.set_attr("FInferShape", DGLSubgraphShape) +.set_attr("FInferType", DGLSubgraphType) +.set_attr("FComputeEx", DGLSubgraphComputeExCPU) +.set_attr("key_var_num_args", "num_args") +.add_argument("graph", "NDArray-or-Symbol", "Input graph where we sample vertices.") +.add_argument("data", "NDArray-or-Symbol[]", + "The input arrays that include data arrays and states.") +.add_arguments(DGLSubgraphParam::__FIELDS__()); + +///////////////////////// Edge Id /////////////////////////// + +inline bool EdgeIDShape(const nnvm::NodeAttrs& attrs, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 3U); + CHECK_EQ(out_attrs->size(), 1U); + CHECK_EQ(in_attrs->at(1).ndim(), 1U); + CHECK_EQ(in_attrs->at(2).ndim(), 1U); + CHECK_EQ(in_attrs->at(1)[0], in_attrs->at(2)[0]); + + SHAPE_ASSIGN_CHECK(*out_attrs, 0, in_attrs->at(1)); + SHAPE_ASSIGN_CHECK(*in_attrs, 1, out_attrs->at(0)); + SHAPE_ASSIGN_CHECK(*in_attrs, 2, out_attrs->at(0)); + return out_attrs->at(0).ndim() != 0U && out_attrs->at(0).Size() != 0U; +} + +inline bool EdgeIDType(const nnvm::NodeAttrs& attrs, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 3U); + CHECK_EQ(out_attrs->size(), 1U); + + TYPE_ASSIGN_CHECK(*out_attrs, 0, in_attrs->at(0)); + TYPE_ASSIGN_CHECK(*in_attrs, 0, out_attrs->at(0)); + return out_attrs->at(0) != -1; +} + +inline bool EdgeIDStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 3U) << "Only works for 2d arrays"; + CHECK_EQ(out_attrs->size(), 1U); + int& in_stype = in_attrs->at(0); + int& out_stype = out_attrs->at(0); + bool dispatched = false; + if (!dispatched && in_stype == kCSRStorage) { + // csr -> dns + dispatched = storage_type_assign(&out_stype, kDefaultStorage, + dispatch_mode, DispatchMode::kFComputeEx); + } + if (!dispatched) { + LOG(ERROR) << "Cannot dispatch edge_id storage type, only works for csr matrices"; + } + return dispatched; +} + +struct edge_id_csr_forward { + template + MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, + const IType* in_indices, const IType* in_indptr, + const CType* u, const CType* v) { + const int64_t target_row_id = static_cast(u[i]); + const IType target_col_id = static_cast(v[i]); + auto ptr = std::find(in_indices + in_indptr[target_row_id], + in_indices + in_indptr[target_row_id + 1], target_col_id); + if (ptr == in_indices + in_indptr[target_row_id + 1]) { + // does not exist in the range + out_data[i] = DType(-1); + } else { + out_data[i] = *(in_data + (ptr - in_indices)); + } + } +}; + +template +void EdgeIDForwardCsrImpl(const OpContext& ctx, + const std::vector& inputs, + const OpReqType req, + const NDArray& output) { + using namespace mshadow; + using namespace mxnet_op; + using namespace csr; + if (req == kNullOp) return; + CHECK_EQ(inputs.size(), 3U); + CHECK_EQ(req, kWriteTo) << "EdgeID with CSR only supports kWriteTo"; + Stream *s = ctx.get_stream(); + const NDArray& u = inputs[1]; + const nnvm::dim_t out_elems = u.shape().Size(); + if (!inputs[0].storage_initialized()) { + MSHADOW_TYPE_SWITCH(output.dtype(), DType, { + Kernel, xpu>::Launch( + s, out_elems, output.data().dptr(), DType(-1)); + }); + return; + } + const NDArray& data = inputs[0]; + const TBlob& in_data = data.data(); + const TBlob& in_indices = data.aux_data(kIdx); + const TBlob& in_indptr = data.aux_data(kIndPtr); + const NDArray& v = inputs[2]; + + CHECK_EQ(data.aux_type(kIdx), data.aux_type(kIndPtr)) + << "The dtypes of indices and indptr don't match"; + MSHADOW_TYPE_SWITCH(data.dtype(), DType, { + MSHADOW_IDX_TYPE_SWITCH(data.aux_type(kIdx), IType, { + MSHADOW_TYPE_SWITCH(u.dtype(), CType, { + Kernel::Launch( + s, out_elems, output.data().dptr(), in_data.dptr(), + in_indices.dptr(), in_indptr.dptr(), + u.data().dptr(), v.data().dptr()); + }); + }); + }); +} + +template +void EdgeIDForwardEx(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + CHECK_EQ(inputs.size(), 3U); + CHECK_EQ(outputs.size(), 1U); + CHECK_EQ(req.size(), 1U); + const auto in_stype = inputs[0].storage_type(); + const auto out_stype = outputs[0].storage_type(); + if (in_stype == kCSRStorage && out_stype == kDefaultStorage) { + EdgeIDForwardCsrImpl(ctx, inputs, req[0], outputs[0]); + } else { + LogUnimplementedOp(attrs, ctx, inputs, req, outputs); + } +} + +NNVM_REGISTER_OP(_contrib_edge_id) +.describe(R"code(This operator implements the edge_id function for a graph +stored in a CSR matrix (the value of the CSR stores the edge Id of the graph). +output[i] = input[u[i], v[i]] if there is an edge between u[i] and v[i]], +otherwise output[i] will be -1. Both u and v should be 1D vectors. + +Example: + + .. code:: python + + x = [[ 1, 0, 0 ], + [ 0, 2, 0 ], + [ 0, 0, 3 ]] + u = [ 0, 0, 1, 1, 2, 2 ] + v = [ 0, 1, 1, 2, 0, 2 ] + edge_id(x, u, v) = [ 1, -1, 2, -1, -1, 3 ] + +The storage type of ``edge_id`` output depends on storage types of inputs + - edge_id(csr, default, default) = default + - default and rsp inputs are not supported + +)code" ADD_FILELINE) +.set_num_inputs(3) +.set_num_outputs(1) +.set_attr("FListInputNames", + [](const NodeAttrs& attrs) { + return std::vector{"data", "u", "v"}; + }) +.set_attr("FInferShape", EdgeIDShape) +.set_attr("FInferType", EdgeIDType) +.set_attr("FInferStorageType", EdgeIDStorageType) +.set_attr("FComputeEx", EdgeIDForwardEx) +.add_argument("data", "NDArray-or-Symbol", "Input ndarray") +.add_argument("u", "NDArray-or-Symbol", "u ndarray") +.add_argument("v", "NDArray-or-Symbol", "v ndarray"); + +///////////////////////// DGL Adjacency /////////////////////////// + +inline bool DGLAdjacencyShape(const nnvm::NodeAttrs& attrs, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 1U); + CHECK_EQ(out_attrs->size(), 1U); + + SHAPE_ASSIGN_CHECK(*out_attrs, 0, in_attrs->at(0)); + SHAPE_ASSIGN_CHECK(*in_attrs, 0, out_attrs->at(0)); + return out_attrs->at(0).ndim() != 0U && out_attrs->at(0).Size() != 0U; +} + +inline bool DGLAdjacencyType(const nnvm::NodeAttrs& attrs, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 1U); + CHECK_EQ(out_attrs->size(), 1U); + CHECK_EQ(in_attrs->at(0), mshadow::kInt64); + TYPE_ASSIGN_CHECK(*out_attrs, 0, mshadow::kFloat32); + return out_attrs->at(0) != -1; +} + +inline bool DGLAdjacencyStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 1U) << "Only works for 2d arrays"; + CHECK_EQ(out_attrs->size(), 1U); + int& out_stype = out_attrs->at(0); + bool dispatched = storage_type_assign(&out_stype, kCSRStorage, + dispatch_mode, DispatchMode::kFComputeEx); + if (!dispatched) { + LOG(ERROR) << "Cannot dispatch output storage type: " << common::stype_string(out_stype) + << ". dgl_adjacency only works for csr matrices"; + } + return dispatched; +} + +NNVM_REGISTER_OP(_contrib_dgl_adjacency) +.describe(R"code(This operator converts a CSR matrix whose values are edge Ids +to an adjacency matrix whose values are ones. The output CSR matrix always has +the data value of float32. + +Example: + + .. code:: python + + x = [[ 1, 0, 0 ], + [ 0, 2, 0 ], + [ 0, 0, 3 ]] + dgl_adjacency(x) = + [[ 1, 0, 0 ], + [ 0, 1, 0 ], + [ 0, 0, 1 ]] + +)code" ADD_FILELINE) +.set_num_inputs(1) +.set_num_outputs(1) +.set_attr("FListInputNames", + [](const NodeAttrs& attrs) { + return std::vector{"data"}; + }) +.set_attr("FInferShape", DGLAdjacencyShape) +.set_attr("FInferType", DGLAdjacencyType) +.set_attr("FInferStorageType", DGLAdjacencyStorageType) +.set_attr("FComputeEx", DGLAdjacencyForwardEx) +.add_argument("data", "NDArray-or-Symbol", "Input ndarray"); + +///////////////////////// Compact subgraphs /////////////////////////// + +struct SubgraphCompactParam : public dmlc::Parameter { + int num_args; + bool return_mapping; + nnvm::Tuple graph_sizes; + DMLC_DECLARE_PARAMETER(SubgraphCompactParam) { + DMLC_DECLARE_FIELD(num_args).set_lower_bound(2) + .describe("Number of input arguments."); + DMLC_DECLARE_FIELD(return_mapping) + .describe("Return mapping of vid and eid between the subgraph and the parent graph."); + DMLC_DECLARE_FIELD(graph_sizes) + .describe("the number of vertices in each graph."); + } +}; // struct SubgraphCompactParam + +DMLC_REGISTER_PARAMETER(SubgraphCompactParam); + +static inline size_t get_num_graphs(const SubgraphCompactParam ¶ms) { + // Each CSR needs a 1D array to store the original vertex Id for each row. + return params.num_args / 2; +} + +static void CompactSubgraph(const NDArray &csr, const NDArray &vids, + const NDArray &out_csr, size_t graph_size) { + TBlob in_idx_data = csr.aux_data(csr::kIdx); + TBlob in_ptr_data = csr.aux_data(csr::kIndPtr); + const dgl_id_t *indices_in = in_idx_data.dptr(); + const dgl_id_t *indptr_in = in_ptr_data.dptr(); + const dgl_id_t *row_ids = vids.data().dptr(); + size_t num_elems = csr.aux_data(csr::kIdx).shape_.Size(); + // The last element in vids is the actual number of vertices in the subgraph. + CHECK_EQ(vids.shape()[0], in_ptr_data.shape_[0]); + CHECK_EQ(static_cast(row_ids[vids.shape()[0] - 1]), graph_size); + + // Prepare the Id map from the original graph to the subgraph. + std::unordered_map id_map; + id_map.reserve(graph_size); + for (size_t i = 0; i < graph_size; i++) { + id_map.insert(std::pair(row_ids[i], i)); + CHECK_NE(row_ids[i], -1); + } + + TShape nz_shape(1); + nz_shape[0] = num_elems; + TShape indptr_shape(1); + CHECK_EQ(out_csr.shape()[0], graph_size); + indptr_shape[0] = graph_size + 1; + CHECK_GE(in_ptr_data.shape_[0], indptr_shape[0]); + + out_csr.CheckAndAllocData(nz_shape); + out_csr.CheckAndAllocAuxData(csr::kIdx, nz_shape); + out_csr.CheckAndAllocAuxData(csr::kIndPtr, indptr_shape); + + dgl_id_t *indices_out = out_csr.aux_data(csr::kIdx).dptr(); + dgl_id_t *indptr_out = out_csr.aux_data(csr::kIndPtr).dptr(); + dgl_id_t *sub_eids = out_csr.data().dptr(); + std::copy(indptr_in, indptr_in + indptr_shape[0], indptr_out); + for (int64_t i = 0; i < nz_shape[0]; i++) { + dgl_id_t old_id = indices_in[i]; + auto it = id_map.find(old_id); + CHECK(it != id_map.end()); + indices_out[i] = it->second; + sub_eids[i] = i; + } +} + +static void SubgraphCompactComputeExCPU(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + int num_g = get_num_graphs(params); +#pragma omp parallel for + for (int i = 0; i < num_g; i++) { + CompactSubgraph(inputs[i], inputs[i + num_g], outputs[i], params.graph_sizes[i]); + } +} + +static bool SubgraphCompactStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + size_t num_g = get_num_graphs(params); + CHECK_EQ(num_g * 2, in_attrs->size()); + // These are the input subgraphs. + for (size_t i = 0; i < num_g; i++) + CHECK_EQ(in_attrs->at(i), kCSRStorage); + // These are the vertex Ids in the original graph. + for (size_t i = 0; i < num_g; i++) + CHECK_EQ(in_attrs->at(i + num_g), kDefaultStorage); + + bool success = true; + *dispatch_mode = DispatchMode::kFComputeEx; + for (size_t i = 0; i < out_attrs->size(); i++) { + if (!type_assign(&(*out_attrs)[i], mxnet::kCSRStorage)) + success = false; + } + return success; +} + +static bool SubgraphCompactShape(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + size_t num_g = get_num_graphs(params); + CHECK_EQ(num_g * 2, in_attrs->size()); + // These are the input subgraphs. + for (size_t i = 0; i < num_g; i++) { + CHECK_EQ(in_attrs->at(i).ndim(), 2U); + CHECK_GE(in_attrs->at(i)[0], params.graph_sizes[i]); + CHECK_GE(in_attrs->at(i)[1], params.graph_sizes[i]); + } + // These are the vertex Ids in the original graph. + for (size_t i = 0; i < num_g; i++) { + CHECK_EQ(in_attrs->at(i + num_g).ndim(), 1U); + CHECK_GE(in_attrs->at(i + num_g)[0], params.graph_sizes[i]); + } + + for (size_t i = 0; i < num_g; i++) { + TShape gshape(2); + gshape[0] = params.graph_sizes[i]; + gshape[1] = params.graph_sizes[i]; + out_attrs->at(i) = gshape; + if (params.return_mapping) + out_attrs->at(i + num_g) = gshape; + } + return true; +} + +static bool SubgraphCompactType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + for (size_t i = 0; i < in_attrs->size(); i++) { + CHECK_EQ(in_attrs->at(i), mshadow::kInt64); + } + for (size_t i = 0; i < out_attrs->size(); i++) { + out_attrs->at(i) = mshadow::kInt64; + } + return true; +} + +NNVM_REGISTER_OP(_contrib_dgl_graph_compact) +.describe(R"code(This operator compacts a CSR matrix generated by +dgl_csr_neighbor_uniform_sample and dgl_csr_neighbor_non_uniform_sample. +The CSR matrices generated by these two operators may have many empty +rows at the end and many empty columns. This operator removes these +empty rows and empty columns. + +Example: + + .. code:: python + + shape = (5, 5) + data_np = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], dtype=np.int64) + indices_np = np.array([1,2,3,4,0,2,3,4,0,1,3,4,0,1,2,4,0,1,2,3], dtype=np.int64) + indptr_np = np.array([0,4,8,12,16,20], dtype=np.int64) + a = mx.nd.sparse.csr_matrix((data_np, indices_np, indptr_np), shape=shape) + seed = mx.nd.array([0,1,2,3,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, + num_neighbor=2, max_num_vertices=6) + subg_v = out[0] + subg = out[1] + compact = mx.nd.contrib.dgl_graph_compact(subg, subg_v, + graph_sizes=(subg_v[-1].asnumpy()[0]), return_mapping=False) + + compact.asnumpy() + array([[0, 0, 0, 1, 0], + [2, 0, 3, 0, 0], + [0, 4, 0, 0, 5], + [0, 6, 0, 0, 7], + [8, 9, 0, 0, 0]]) + +)code" ADD_FILELINE) +.set_attr_parser(ParamParser) +.set_num_inputs([](const NodeAttrs& attrs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + return params.num_args; +}) +.set_num_outputs([](const NodeAttrs& attrs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + int num_varray = get_num_graphs(params); + if (params.return_mapping) + return num_varray * 2; + else + return num_varray; +}) +.set_attr("FListInputNames", + [](const NodeAttrs& attrs) { + const SubgraphCompactParam& params = nnvm::get(attrs.parsed); + std::vector names; + names.reserve(params.num_args); + size_t num_graphs = get_num_graphs(params); + for (size_t i = 0; i < num_graphs; i++) + names.push_back("graph" + std::to_string(i)); + for (size_t i = 0; i < num_graphs; ++i) + names.push_back("varray" + std::to_string(i)); + return names; +}) +.set_attr("FInferStorageType", SubgraphCompactStorageType) +.set_attr("FInferShape", SubgraphCompactShape) +.set_attr("FInferType", SubgraphCompactType) +.set_attr("FComputeEx", SubgraphCompactComputeExCPU) +.set_attr("key_var_num_args", "num_args") +.add_argument("graph_data", "NDArray-or-Symbol[]", "Input graphs and input vertex Ids.") +.add_arguments(SubgraphCompactParam::__FIELDS__()); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/dgl_graph.cu b/src/operator/contrib/dgl_graph.cu new file mode 100644 index 000000000..336e9d4f8 --- /dev/null +++ b/src/operator/contrib/dgl_graph.cu @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#include "dgl_graph-inl.h" + +namespace mxnet { +namespace op { + +NNVM_REGISTER_OP(_contrib_dgl_adjacency) +.set_attr("FComputeEx", DGLAdjacencyForwardEx); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/index_copy-inl.h b/src/operator/contrib/index_copy-inl.h index b97138a88..923fb0f4f 100644 --- a/src/operator/contrib/index_copy-inl.h +++ b/src/operator/contrib/index_copy-inl.h @@ -32,6 +32,7 @@ #include "../elemwise_op_common.h" #include "../mshadow_op.h" #include "../mxnet_op.h" +#include "../tensor/init_op.h" namespace mxnet { namespace op { @@ -83,12 +84,12 @@ void IndexCopyForward(const nnvm::NodeAttrs& attrs, }); } -template struct index_copy_backward { template MSHADOW_XINLINE static void Map(int i, int dim, int index_size, + int req1, int req2, DType* out_grad, IType* index, DType* in_grad_1, @@ -98,12 +99,12 @@ struct index_copy_backward { int idx = static_cast(index[p]); if (i >= idx*dim && i < (idx+1)*dim) { int offset = i - idx*dim; - KERNEL_ASSIGN(in_grad_2[p*dim+offset], req, out_grad[i]); + KERNEL_ASSIGN(in_grad_2[p*dim+offset], req2, out_grad[i]); return; } } // Copy to in_grad_1 - KERNEL_ASSIGN(in_grad_1[i], req, out_grad[i]); + KERNEL_ASSIGN(in_grad_1[i], req1, out_grad[i]); } }; @@ -122,18 +123,19 @@ void IndexCopyBackward(const nnvm::NodeAttrs& attrs, const TBlob& in_grad_2 = outputs[2]; int dim = inputs[3].Size() / inputs[2].Size(); int index_size = inputs[2].Size(); + Fill(s, outputs[0], req[0], 0); + Fill(s, outputs[2], req[2], 0); // index_copy_backward MSHADOW_TYPE_SWITCH(out_grad.type_flag_, DType, { MSHADOW_TYPE_SWITCH(index.type_flag_, IType, { - MXNET_ASSIGN_REQ_SWITCH(req[0], req_type, { - mxnet_op::Kernel, xpu>::Launch(s, + mxnet_op::Kernel::Launch(s, out_grad.Size(), dim, index_size, + req[0], req[2], out_grad.dptr(), index.dptr(), in_grad_1.dptr(), in_grad_2.dptr()); - }); }); }); } diff --git a/src/operator/contrib/index_copy.cc b/src/operator/contrib/index_copy.cc index 316c8a718..70a32a1b2 100644 --- a/src/operator/contrib/index_copy.cc +++ b/src/operator/contrib/index_copy.cc @@ -37,30 +37,32 @@ static bool IndexCopyType(const nnvm::NodeAttrs& attrs, } NNVM_REGISTER_OP(_contrib_index_copy) -.describe(R"code(Copies the elements of a `new_tensor` into the `old_tensor` by -selecting the indices in the order given in `index`. The output will be a new tensor -contains the rest elements of old tensor and the copied elements of new tensor. -For example, if `index[i] == j`, then the `i`th row of `new_tensor` is copied to the -`j`th row of output. +.describe(R"code(Copies the elements of a `new_tensor` into the `old_tensor`. -The `index` must be a vector and it must have the same size with the `0`th dimimention of -`new_tensor`. Also, the `0`th dimimention of old_tensor must `>=` the `0`th dimimention of +This operator copies the elements by selecting the indices in the order given in `index`. +The output will be a new tensor containing the rest elements of old tensor and +the copied elements of new tensor. +For example, if `index[i] == j`, then the `i` th row of `new_tensor` is copied to the +`j` th row of output. + +The `index` must be a vector and it must have the same size with the `0` th dimension of +`new_tensor`. Also, the `0` th dimension of old_tensor must `>=` the `0` th dimension of `new_tensor`, or an error will be raised. Examples:: -x = mx.nd.zeros((5,3)) -t = mx.nd.array([[1,2,3],[4,5,6],[7,8,9]]) -index = mx.nd.array([0,4,2]) + x = mx.nd.zeros((5,3)) + t = mx.nd.array([[1,2,3],[4,5,6],[7,8,9]]) + index = mx.nd.array([0,4,2]) -mx.nd.contrib.index_copy(x, index, t) + mx.nd.contrib.index_copy(x, index, t) -[[1. 2. 3.] - [0. 0. 0.] - [7. 8. 9.] - [0. 0. 0.] - [4. 5. 6.]] - + [[1. 2. 3.] + [0. 0. 0.] + [7. 8. 9.] + [0. 0. 0.] + [4. 5. 6.]] + )code" ADD_FILELINE) .set_num_inputs(3) diff --git a/src/operator/contrib/multibox_detection.cc b/src/operator/contrib/multibox_detection.cc index c005dfa06..b4f66d8fc 100644 --- a/src/operator/contrib/multibox_detection.cc +++ b/src/operator/contrib/multibox_detection.cc @@ -174,7 +174,6 @@ inline void MultiBoxDetectionForward(const Tensor &out, } // apply nms -#pragma omp parallel for num_threads(omp_threads) for (int i = 0; i < nkeep; ++i) { int offset_i = i * 6; if (p_out[offset_i] < 0) continue; // skip eliminated diff --git a/src/operator/contrib/nnvm_to_onnx-inl.h b/src/operator/contrib/nnvm_to_onnx-inl.h index 58f88b051..011ffe6b7 100644 --- a/src/operator/contrib/nnvm_to_onnx-inl.h +++ b/src/operator/contrib/nnvm_to_onnx-inl.h @@ -37,7 +37,6 @@ #include #include -#include #include #include @@ -49,13 +48,48 @@ #include #include -#include "./tensorrt-inl.h" #include "../operator_common.h" #include "../../common/utils.h" #include "../../common/serialization.h" namespace mxnet { namespace op { + +namespace nnvm_to_onnx { + enum class TypeIO { Inputs = 0, Outputs = 1 }; + using NameToIdx_t = std::map; + using InferenceTuple_t = std::tuple; + using InferenceMap_t = std::map; +} // namespace nnvm_to_onnx + +struct ONNXParam : public dmlc::Parameter { + std::string serialized_onnx_graph; + std::string serialized_input_map; + std::string serialized_output_map; + nnvm_to_onnx::NameToIdx_t input_map; + nnvm_to_onnx::InferenceMap_t output_map; + ::onnx::ModelProto onnx_pb_graph; + + ONNXParam() {} + + ONNXParam(const ::onnx::ModelProto& onnx_graph, + const nnvm_to_onnx::InferenceMap_t& input_map, + const nnvm_to_onnx::NameToIdx_t& output_map) { + common::Serialize(input_map, &serialized_input_map); + common::Serialize(output_map, &serialized_output_map); + onnx_graph.SerializeToString(&serialized_onnx_graph); + } + +DMLC_DECLARE_PARAMETER(ONNXParam) { + DMLC_DECLARE_FIELD(serialized_onnx_graph) + .describe("Serialized ONNX graph"); + DMLC_DECLARE_FIELD(serialized_input_map) + .describe("Map from inputs to topological order as input."); + DMLC_DECLARE_FIELD(serialized_output_map) + .describe("Map from outputs to order in g.outputs."); + } +}; + namespace nnvm_to_onnx { using namespace nnvm; @@ -76,7 +110,7 @@ void ConvertConstant(GraphProto* const graph_proto, const std::string& node_name, std::unordered_map* const shared_buffer); -void ConvertOutput(op::tensorrt::InferenceMap_t* const trt_output_map, +void ConvertOutput(op::nnvm_to_onnx::InferenceMap_t* const trt_output_map, GraphProto* const graph_proto, const std::unordered_map::iterator& out_iter, const std::string& node_name, @@ -133,7 +167,7 @@ void ConvertElementwiseAdd(NodeProto *node_proto, const nnvm::IndexedGraph &ig, const array_view &inputs); -TRTParam ConvertNnvmGraphToOnnx( +ONNXParam ConvertNnvmGraphToOnnx( const nnvm::Graph &g, std::unordered_map *const shared_buffer); diff --git a/src/operator/contrib/nnvm_to_onnx.cc b/src/operator/contrib/nnvm_to_onnx.cc index 902466614..784384e94 100644 --- a/src/operator/contrib/nnvm_to_onnx.cc +++ b/src/operator/contrib/nnvm_to_onnx.cc @@ -47,7 +47,6 @@ #include "../../operator/nn/fully_connected-inl.h" #include "../../operator/nn/pooling-inl.h" #include "../../operator/softmax_output-inl.h" -#include "./tensorrt-inl.h" #if MXNET_USE_TENSORRT_ONNX_CHECKER #include @@ -55,14 +54,17 @@ namespace mxnet { namespace op { + +DMLC_REGISTER_PARAMETER(ONNXParam); + namespace nnvm_to_onnx { -op::TRTParam ConvertNnvmGraphToOnnx( +op::ONNXParam ConvertNnvmGraphToOnnx( const nnvm::Graph& g, std::unordered_map* const shared_buffer) { - op::TRTParam trt_param; - op::tensorrt::NameToIdx_t trt_input_map; - op::tensorrt::InferenceMap_t trt_output_map; + op::ONNXParam onnx_param; + op::nnvm_to_onnx::NameToIdx_t onnx_input_map; + op::nnvm_to_onnx::InferenceMap_t onnx_output_map; const nnvm::IndexedGraph& ig = g.indexed_graph(); const auto& storage_types = g.GetAttr("storage_type"); @@ -105,7 +107,7 @@ op::TRTParam ConvertNnvmGraphToOnnx( current_input++; continue; } - trt_input_map.emplace(node_name, current_input++); + onnx_input_map.emplace(node_name, current_input++); ConvertPlaceholder(node_name, placeholder_shapes, graph_proto); } else { // If it's not a placeholder, then by exclusion it's a constant. @@ -140,23 +142,23 @@ op::TRTParam ConvertNnvmGraphToOnnx( auto out_iter = output_lookup.find(node_name); // We found an output if (out_iter != output_lookup.end()) { - ConvertOutput(&trt_output_map, graph_proto, out_iter, node_name, g, + ConvertOutput(&onnx_output_map, graph_proto, out_iter, node_name, g, storage_types, dtypes); } // output found } // conversion function exists } // loop over i from 0 to num_nodes - model_proto.SerializeToString(&trt_param.serialized_onnx_graph); - common::Serialize(trt_input_map, - &trt_param.serialized_input_map); - common::Serialize(trt_output_map, - &trt_param.serialized_output_map); + model_proto.SerializeToString(&onnx_param.serialized_onnx_graph); + common::Serialize(onnx_input_map, + &onnx_param.serialized_input_map); + common::Serialize(onnx_output_map, + &onnx_param.serialized_output_map); #if MXNET_USE_TENSORRT_ONNX_CHECKER onnx::checker::check_model(model_proto); #endif // MXNET_USE_TENSORRT_ONNX_CHECKER - return trt_param; + return onnx_param; } void ConvertConvolution(NodeProto* node_proto, const NodeAttrs& attrs, @@ -489,7 +491,7 @@ void ConvertConstant( } void ConvertOutput( - op::tensorrt::InferenceMap_t* const trt_output_map, + op::nnvm_to_onnx::InferenceMap_t* const output_map, GraphProto* const graph_proto, const std::unordered_map::iterator& out_iter, const std::string& node_name, const nnvm::Graph& g, @@ -501,10 +503,10 @@ void ConvertOutput( int dtype = dtypes[out_idx]; // This should work with fp16 as well - op::tensorrt::InferenceTuple_t out_tuple{out_iter->second, out_shape, storage_type, + op::nnvm_to_onnx::InferenceTuple_t out_tuple{out_iter->second, out_shape, storage_type, dtype}; - trt_output_map->emplace(node_name, out_tuple); + output_map->emplace(node_name, out_tuple); auto graph_out = graph_proto->add_output(); auto tensor_type = graph_out->mutable_type()->mutable_tensor_type(); diff --git a/src/operator/contrib/nnz.cc b/src/operator/contrib/nnz.cc new file mode 100644 index 000000000..a94f52fda --- /dev/null +++ b/src/operator/contrib/nnz.cc @@ -0,0 +1,189 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2018 by Contributors + * \file nnz.cc + * \brief CPU Implementation of nnz operator + */ +#include +#include +#include +#include +#include "../elemwise_op_common.h" +#include "../tensor/init_op.h" +#include "../mshadow_op.h" +#include "../mxnet_op.h" + +namespace mxnet { +namespace op { + +struct NNZParam : public dmlc::Parameter { + dmlc::optional axis; + DMLC_DECLARE_PARAMETER(NNZParam) { + DMLC_DECLARE_FIELD(axis) + .set_default(dmlc::optional()) + .describe("Select between the number of values across the whole matrix, " + "in each column, or in each row."); + } +}; + +static bool NNZType(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 1U); + CHECK_EQ(out_attrs->size(), 1U); + // infer int64 for count + TYPE_ASSIGN_CHECK(*out_attrs, 0, mshadow::kInt64); + return true; +} + +inline bool NNZShape(const nnvm::NodeAttrs& attrs, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 1U); + CHECK_EQ(out_attrs->size(), 1U); + // csr_matrix is 2-D + CHECK_EQ(in_attrs->at(0).ndim(), 2); + const NNZParam& param = nnvm::get(attrs.parsed); + // whole matrix + if (!param.axis.has_value()) { + SHAPE_ASSIGN_CHECK(*out_attrs, 0, mshadow::Shape1(1)); + } else if (param.axis.value() == 0) { + // columns + SHAPE_ASSIGN_CHECK(*out_attrs, 0, mshadow::Shape1(in_attrs->at(0)[1])); + } else if (param.axis.value() == 1) { + // rows + SHAPE_ASSIGN_CHECK(*out_attrs, 0, mshadow::Shape1(in_attrs->at(0)[0])); + } else { + LOG(FATAL) << "Unexpected value for axis(" << param.axis.value() + << "). Candidates are None, 0, and 1"; + } + return true; +} + +template +void NNZComputeCsrImpl(const NNZParam& param, + const OpContext& ctx, + const NDArray& input, + const OpReqType req, + const TBlob& output); + +struct CsrNNZRowKernel { + /*! + * \brief Map function for general case of take grad + * \param tid global thread id + * \param out ptr to output + * \param indptr ptr to source csr indptr + */ + template + MSHADOW_XINLINE static void Map(int tid, DType* out, const IType* indptr) { + out[tid] = static_cast(indptr[tid + 1] - indptr[tid]); + } +}; + +template<> +void NNZComputeCsrImpl(const NNZParam& param, + const OpContext& ctx, + const NDArray& input, + const OpReqType req, + const TBlob& output) { + using namespace csr; + CHECK_EQ(req, kWriteTo); + mshadow::Stream *s = ctx.get_stream(); + if (!input.storage_initialized()) { + Fill(s, output, kWriteTo, 0); + return; + } + MSHADOW_TYPE_SWITCH(output.type_flag_, DType, { + MSHADOW_IDX_TYPE_SWITCH(input.aux_type(kIndPtr), IType, { + DType* out_ptr = output.dptr(); + const IType* indptr = input.aux_data(kIndPtr).dptr(); + const nnvm::dim_t num_rows = input.shape()[0]; + if (!param.axis.has_value()) { + // whole matrix + out_ptr[0] = indptr[num_rows]; + } else if (param.axis.value() == 0) { + // column + LOG(FATAL) << "getnnz with axis = 0 is not supported yet"; + } else if (param.axis.value() == 1) { + // row + mxnet_op::Kernel::Launch(s, num_rows, out_ptr, indptr); + } + }); + }); +} + +template +void NNZComputeEx(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + CHECK_EQ(inputs.size(), 1U); + CHECK_EQ(outputs.size(), 1U); + CHECK_EQ(req.size(), 1U); + const auto in_stype = inputs[0].storage_type(); + const auto out_stype = outputs[0].storage_type(); + const NNZParam& param = nnvm::get(attrs.parsed); + if (in_stype == kCSRStorage && out_stype == kDefaultStorage) { + NNZComputeCsrImpl(param, ctx, inputs[0], req[0], outputs[0].data()); + } else { + LogUnimplementedOp(attrs, ctx, inputs, req, outputs); + } +} + +bool NNZStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + CHECK_EQ(in_attrs->size(), 1); + CHECK_EQ(out_attrs->size(), 1); + bool dispatched = false; + const auto in_stype = in_attrs->at(0); + auto& out_stype = out_attrs->at(0); + // only support csr for now + if (!dispatched && in_stype == kCSRStorage) { + dispatched = storage_type_assign(&out_stype, kDefaultStorage, + dispatch_mode, DispatchMode::kFComputeEx); + } + return dispatched; +} + +DMLC_REGISTER_PARAMETER(NNZParam); + +NNVM_REGISTER_OP(_contrib_getnnz) +.describe(R"code(Number of stored values for a sparse tensor, including explicit zeros. + +This operator only supports CSR matrix on CPU. + +)code" ADD_FILELINE) +.set_num_inputs(1) +.set_num_outputs(1) +.set_attr_parser(ParamParser) +.set_attr("FInferShape", NNZShape) +.set_attr("FInferType", NNZType) +.set_attr("FInferStorageType", NNZStorageType) +.set_attr("FComputeEx", NNZComputeEx) +.add_argument("data", "NDArray-or-Symbol", "Input") +.add_arguments(NNZParam::__FIELDS__()); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/contrib/quadratic_op-inl.h b/src/operator/contrib/quadratic_op-inl.h index 71cb76a7b..a6fa260f1 100644 --- a/src/operator/contrib/quadratic_op-inl.h +++ b/src/operator/contrib/quadratic_op-inl.h @@ -20,7 +20,7 @@ /*! * \file quad_function-inl.h * \brief Operator implementing quadratic function. - * For using as an exmaple in the tutorial of adding operators + * For using as an example in the tutorial of adding operators * in MXNet backend. */ #ifndef MXNET_OPERATOR_CONTRIB_QUADRATIC_OP_INL_H_ diff --git a/src/operator/contrib/quadratic_op.cc b/src/operator/contrib/quadratic_op.cc index d8b2d785c..043379e44 100644 --- a/src/operator/contrib/quadratic_op.cc +++ b/src/operator/contrib/quadratic_op.cc @@ -29,12 +29,16 @@ namespace op { DMLC_REGISTER_PARAMETER(QuadraticParam); NNVM_REGISTER_OP(_contrib_quadratic) -.describe(R"code(This operators implements the quadratic function: +.describe(R"code(This operators implements the quadratic function. + .. math:: f(x) = ax^2+bx+c + where :math:`x` is an input tensor and all operations in the function are element-wise. + Example:: + x = [[1, 2], [3, 4]] y = quadratic(data=x, a=1, b=2, c=3) y = [[6, 11], [18, 27]] diff --git a/src/operator/contrib/roi_align-inl.h b/src/operator/contrib/roi_align-inl.h index 263f72a6a..9f4d7ce48 100644 --- a/src/operator/contrib/roi_align-inl.h +++ b/src/operator/contrib/roi_align-inl.h @@ -20,7 +20,7 @@ * Copyright (c) 2018 by Contributors * \file roi_align-inl.h * \brief roi align operator and symbol - * \author Hang Zhang + * \author Hang Zhang, Shesung * modified from Caffe2 */ #ifndef MXNET_OPERATOR_CONTRIB_ROI_ALIGN_INL_H_ @@ -35,7 +35,6 @@ namespace mxnet { namespace op { - // Declare enumeration of input order to make code more intuitive. // These enums are only visible within this header namespace roialign { @@ -48,6 +47,7 @@ struct ROIAlignParam : public dmlc::Parameter { TShape pooled_size; float spatial_scale; int sample_ratio; + bool position_sensitive; DMLC_DECLARE_PARAMETER(ROIAlignParam) { DMLC_DECLARE_FIELD(pooled_size) .set_expect_ndim(2).enforce_nonzero() @@ -57,6 +57,10 @@ struct ROIAlignParam : public dmlc::Parameter { "Equals the reciprocal of total stride in convolutional layers"); DMLC_DECLARE_FIELD(sample_ratio).set_default(-1) .describe("Optional sampling ratio of ROI align, using adaptive size by default."); + DMLC_DECLARE_FIELD(position_sensitive).set_default(false) + .describe("Whether to perform position-sensitive RoI pooling. PSRoIPooling is " + "first proposaled by R-FCN and it can reduce the input channels by ph*pw times, " + "where (ph, pw) is the pooled_size"); } }; diff --git a/src/operator/contrib/roi_align.cc b/src/operator/contrib/roi_align.cc index 3ce2213d7..e584ea303 100644 --- a/src/operator/contrib/roi_align.cc +++ b/src/operator/contrib/roi_align.cc @@ -20,7 +20,7 @@ * Copyright (c) 2018 by Contributors * \file roi_align.cc * \brief roi align operator - * \author Hang Zhang + * \author Hang Zhang, Shesung * Adapted from Caffe2 */ #include "./roi_align-inl.h" @@ -142,6 +142,7 @@ void ROIAlignForward( const int nthreads, const T* bottom_data, const T& spatial_scale, + const bool position_sensitive, const int channels, const int height, const int width, @@ -156,6 +157,8 @@ void ROIAlignForward( int n_rois = nthreads / channels / pooled_width / pooled_height; // (n, c, ph, pw) is an element in the pooled output // can be parallelized using omp +#pragma omp parallel for \ +num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) for (int n = 0; n < n_rois; n++) { int index_n = n * channels * pooled_width * pooled_height; @@ -208,19 +211,23 @@ void ROIAlignForward( roi_bin_grid_w, &pre_calc); - int c; -#pragma omp parallel for private(c) \ -num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) - for (c = 0; c < channels; c++) { + for (int c = 0; c < channels; c++) { int index_n_c = index_n + c * pooled_width * pooled_height; - const T* offset_bottom_data = - bottom_data + (roi_batch_ind * channels + c) * height * width; int pre_calc_index = 0; for (int ph = 0; ph < pooled_height; ph++) { for (int pw = 0; pw < pooled_width; pw++) { int index = index_n_c + ph * pooled_width + pw; + int c_unpooled = c; + int channels_unpooled = channels; + if (position_sensitive) { + c_unpooled = c * pooled_height * pooled_width + ph * pooled_width + pw; + channels_unpooled = channels * pooled_height * pooled_width; + } + const T* offset_bottom_data = + bottom_data + (roi_batch_ind * channels_unpooled + c_unpooled) + * height * width; T output_val = 0.; for (int iy = 0; iy < roi_bin_grid_h; iy++) { for (int ix = 0; ix < roi_bin_grid_w; ix++) { @@ -310,6 +317,7 @@ void ROIAlignBackward( const T* top_diff, const int /*num_rois*/, const T& spatial_scale, + const bool position_sensitive, const int channels, const int height, const int width, @@ -347,8 +355,15 @@ void ROIAlignBackward( T bin_size_h = static_cast(roi_height) / static_cast(pooled_height); T bin_size_w = static_cast(roi_width) / static_cast(pooled_width); + int c_unpooled = c; + int channels_unpooled = channels; + if (position_sensitive) { + c_unpooled = c * pooled_height * pooled_width + ph * pooled_width + pw; + channels_unpooled = channels * pooled_height * pooled_width; + } T* offset_bottom_diff = - bottom_diff + (roi_batch_ind * channels + c) * height * width; + bottom_diff + (roi_batch_ind * channels_unpooled + c_unpooled) + * height * width; int top_offset = (n * channels + c) * pooled_height * pooled_width; const T* offset_top_diff = top_diff + top_offset; @@ -426,7 +441,7 @@ void ROIAlignForwardCompute(const nnvm::NodeAttrs& attrs, const int count = out_data[roialign::kOut].Size(); // const int num_rois = in_data[roialign::kBox].size(0); - const int channels = in_data[roialign::kData].size(1); + const int channels = out_data[roialign::kOut].size(1); // channels of pooled output const int height = in_data[roialign::kData].size(2); const int width = in_data[roialign::kData].size(3); const int pooled_height = out_data[roialign::kOut].size(2); @@ -439,9 +454,9 @@ void ROIAlignForwardCompute(const nnvm::NodeAttrs& attrs, const DType *bottom_rois = in_data[roialign::kBox].dptr(); DType *top_data = out_data[roialign::kOut].dptr(); - ROIAlignForward(count, bottom_data, param.spatial_scale, channels, - height, width, pooled_height, pooled_width, param.sample_ratio, - bottom_rois, rois_cols, top_data); + ROIAlignForward(count, bottom_data, param.spatial_scale, param.position_sensitive, + channels, height, width, pooled_height, pooled_width, + param.sample_ratio, bottom_rois, rois_cols, top_data); }) } @@ -470,7 +485,7 @@ void ROIAlignBackwardCompute(const nnvm::NodeAttrs& attrs, const int count = out_grad[0].Size(); const int num_rois = in_data[0].size(0); - const int channels = outputs[0].size(1); + const int channels = out_grad[0].size(1); // channels of pooled output const int height = outputs[0].size(2); const int width = outputs[0].size(3); const int pooled_height = out_grad[0].size(2); @@ -489,8 +504,9 @@ void ROIAlignBackwardCompute(const nnvm::NodeAttrs& attrs, Fill(s, outputs[0], kWriteTo, static_cast(0)); } ROIAlignBackward(count, top_diff, num_rois, param.spatial_scale, - channels, height, width, pooled_height, pooled_width, - param.sample_ratio, grad_in, bottom_rois, rois_cols); + param.position_sensitive, channels, height, width, + pooled_height, pooled_width, param.sample_ratio, grad_in, + bottom_rois, rois_cols); } if (kWriteTo == req[roialign::kBox]) { Fill(s, outputs[1], kWriteTo, static_cast(0)); @@ -515,8 +531,8 @@ input features at four regularly sampled locations in each RoI bin. Then the feature map can be aggregated by avgpooling. -Reference ---------- +References +---------- He, Kaiming, et al. "Mask R-CNN." ICCV, 2017 )code" ADD_FILELINE) @@ -545,8 +561,17 @@ He, Kaiming, et al. "Mask R-CNN." ICCV, 2017 CHECK_EQ(bshape[1], 5) << "bbox should be a 2D tensor of shape [batch, 5]"; // out: [num_rois, c, pooled_h, pooled_w] out_shape->clear(); - out_shape->push_back( - Shape4(bshape[0], dshape[1], param.pooled_size[0], param.pooled_size[1])); + if (param.position_sensitive) { + CHECK_EQ(dshape[1] % (param.pooled_size[0]*param.pooled_size[1]), 0) << + "Input channels should be divided by pooled_size[0]*pooled_size[1]" + "when position_sensitive is true."; + out_shape->push_back( + Shape4(bshape[0], dshape[1]/param.pooled_size[0]/param.pooled_size[1], + param.pooled_size[0], param.pooled_size[1])); + } else { + out_shape->push_back( + Shape4(bshape[0], dshape[1], param.pooled_size[0], param.pooled_size[1])); + } return true; }) .set_attr("FInferType", [](const nnvm::NodeAttrs& attrs, diff --git a/src/operator/contrib/roi_align.cu b/src/operator/contrib/roi_align.cu index d3db70b73..38b461d5f 100644 --- a/src/operator/contrib/roi_align.cu +++ b/src/operator/contrib/roi_align.cu @@ -20,7 +20,7 @@ * Copyright (c) 2018 by Contributors * \file roi_align.cu * \brief roi align operator - * \author Hang Zhang + * \author Hang Zhang, Shesung * Adapted from Caffe2 */ #include "./roi_align-inl.h" @@ -111,6 +111,7 @@ __global__ void RoIAlignForwardKernel( const int nthreads, const T* bottom_data, const T spatial_scale, + const bool position_sensitive, const int channels, const int height, const int width, @@ -145,8 +146,15 @@ __global__ void RoIAlignForwardKernel( T bin_size_h = static_cast(roi_height) / static_cast(pooled_height); T bin_size_w = static_cast(roi_width) / static_cast(pooled_width); + int c_unpooled = c; + int channels_unpooled = channels; + if (position_sensitive) { + c_unpooled = c * pooled_height * pooled_width + ph * pooled_width + pw; + channels_unpooled = channels * pooled_height * pooled_width; + } const T* offset_bottom_data = - bottom_data + (roi_batch_ind * channels + c) * height * width; + bottom_data + (roi_batch_ind * channels_unpooled + c_unpooled) + * height * width; // We use roi_bin_grid to sample the grid and mimic integral int roi_bin_grid_h = (sampling_ratio > 0) @@ -242,6 +250,7 @@ __global__ void RoIAlignBackwardKernel( const T* top_diff, const int num_rois, const T spatial_scale, + const bool position_sensitive, const int channels, const int height, const int width, @@ -276,8 +285,15 @@ __global__ void RoIAlignBackwardKernel( T bin_size_h = static_cast(roi_height) / static_cast(pooled_height); T bin_size_w = static_cast(roi_width) / static_cast(pooled_width); + int c_unpooled = c; + int channels_unpooled = channels; + if (position_sensitive) { + c_unpooled = c * pooled_height * pooled_width + ph * pooled_width + pw; + channels_unpooled = channels * pooled_height * pooled_width; + } T* offset_bottom_diff = - bottom_diff + (roi_batch_ind * channels + c) * height * width; + bottom_diff + (roi_batch_ind * channels_unpooled + c_unpooled) + * height * width; int top_offset = (n * channels + c) * pooled_height * pooled_width; const T* offset_top_diff = top_diff + top_offset; @@ -357,7 +373,7 @@ void ROIAlignForwardCompute(const nnvm::NodeAttrs& attrs, const int count = out_data[roialign::kOut].Size(); const int num_rois = in_data[roialign::kBox].size(0); - const int channels = in_data[roialign::kData].size(1); + const int channels = out_data[roialign::kOut].size(1); // channels of pooled output const int height = in_data[roialign::kData].size(2); const int width = in_data[roialign::kData].size(3); const int pooled_height = out_data[roialign::kOut].size(2); @@ -377,6 +393,7 @@ void ROIAlignForwardCompute(const nnvm::NodeAttrs& attrs, count, bottom_data, param.spatial_scale, + param.position_sensitive, channels, height, width, @@ -414,7 +431,7 @@ void ROIAlignBackwardCompute(const nnvm::NodeAttrs& attrs, const int count = out_grad[0].Size(); const int num_rois = in_data[0].size(0); - const int channels = outputs[0].size(1); + const int channels = out_grad[0].size(1); // channels of pooled output const int height = outputs[0].size(2); const int width = outputs[0].size(3); const int pooled_height = out_grad[0].size(2); @@ -445,6 +462,7 @@ void ROIAlignBackwardCompute(const nnvm::NodeAttrs& attrs, top_diff, num_rois, param.spatial_scale, + param.position_sensitive, channels, height, width, diff --git a/src/operator/contrib/tensorrt-inl.h b/src/operator/contrib/tensorrt-inl.h index be335ab12..062d22e35 100644 --- a/src/operator/contrib/tensorrt-inl.h +++ b/src/operator/contrib/tensorrt-inl.h @@ -38,7 +38,6 @@ #include #include -#include #include #include @@ -49,6 +48,7 @@ #include #include +#include "nnvm_to_onnx-inl.h" #include "../operator_common.h" #include "../../common/utils.h" #include "../../common/serialization.h" @@ -60,49 +60,15 @@ namespace mxnet { namespace op { using namespace nnvm; -using namespace ::onnx; using int64 = ::google::protobuf::int64; -namespace tensorrt { - enum class TypeIO { Inputs = 0, Outputs = 1 }; - using NameToIdx_t = std::map; - using InferenceTuple_t = std::tuple; - using InferenceMap_t = std::map; -} // namespace tensorrt using trt_name_to_idx = std::map; -struct TRTParam : public dmlc::Parameter { - std::string serialized_onnx_graph; - std::string serialized_input_map; - std::string serialized_output_map; - tensorrt::NameToIdx_t input_map; - tensorrt::InferenceMap_t output_map; - ::onnx::ModelProto onnx_pb_graph; - - TRTParam() {} - - TRTParam(const ::onnx::ModelProto& onnx_graph, - const tensorrt::InferenceMap_t& input_map, - const tensorrt::NameToIdx_t& output_map) { - common::Serialize(input_map, &serialized_input_map); - common::Serialize(output_map, &serialized_output_map); - onnx_graph.SerializeToString(&serialized_onnx_graph); - } - -DMLC_DECLARE_PARAMETER(TRTParam) { - DMLC_DECLARE_FIELD(serialized_onnx_graph) - .describe("Serialized ONNX graph"); - DMLC_DECLARE_FIELD(serialized_input_map) - .describe("Map from inputs to topological order as input."); - DMLC_DECLARE_FIELD(serialized_output_map) - .describe("Map from outputs to order in g.outputs."); - } -}; struct TRTEngineParam { nvinfer1::IExecutionContext* trt_executor; - std::vector > binding_map; + std::vector > binding_map; }; } // namespace op diff --git a/src/operator/contrib/tensorrt.cc b/src/operator/contrib/tensorrt.cc index 619fe1e2b..88a65fba3 100644 --- a/src/operator/contrib/tensorrt.cc +++ b/src/operator/contrib/tensorrt.cc @@ -44,20 +44,18 @@ namespace mxnet { namespace op { -DMLC_REGISTER_PARAMETER(TRTParam); - OpStatePtr GetPtrMapping(nvinfer1::ICudaEngine* trt_engine, - tensorrt::NameToIdx_t input_map, - tensorrt::NameToIdx_t output_map) { + nnvm_to_onnx::NameToIdx_t input_map, + nnvm_to_onnx::NameToIdx_t output_map) { TRTEngineParam param; for (int b = 0; b < trt_engine->getNbBindings(); ++b) { const std::string& binding_name = trt_engine->getBindingName(b); if (trt_engine->bindingIsInput(b)) { param.binding_map.emplace_back(input_map[binding_name], - tensorrt::TypeIO::Inputs); + nnvm_to_onnx::TypeIO::Inputs); } else { param.binding_map.emplace_back(output_map[binding_name], - tensorrt::TypeIO::Outputs); + nnvm_to_onnx::TypeIO::Outputs); } } param.trt_executor = trt_engine->createExecutionContext(); @@ -67,7 +65,7 @@ OpStatePtr GetPtrMapping(nvinfer1::ICudaEngine* trt_engine, OpStatePtr TRTCreateState(const nnvm::NodeAttrs& attrs, Context /*ctx*/, const std::vector& /*ishape*/, const std::vector& /*itype*/) { - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); ::onnx::ModelProto model_proto; bool success = model_proto.ParseFromString(node_param.serialized_onnx_graph); @@ -82,7 +80,7 @@ OpStatePtr TRTCreateState(const nnvm::NodeAttrs& attrs, Context /*ctx*/, nvinfer1::ICudaEngine* const trt_engine = ::onnx_to_tensorrt::onnxToTrtCtx( node_param.serialized_onnx_graph, batch_size, 1 << 30); - tensorrt::NameToIdx_t output_map; + nnvm_to_onnx::NameToIdx_t output_map; for (auto& el : node_param.output_map) { output_map[el.first] = std::get<0>(el.second); } @@ -90,7 +88,7 @@ OpStatePtr TRTCreateState(const nnvm::NodeAttrs& attrs, Context /*ctx*/, } void TRTParamParser(nnvm::NodeAttrs* attrs) { - TRTParam param_; + ONNXParam param_; try { param_.Init(attrs->dict); @@ -114,7 +112,7 @@ void TRTParamParser(nnvm::NodeAttrs* attrs) { inline bool TRTInferShape(const NodeAttrs& attrs, std::vector* /*in_shape*/, std::vector* out_shape) { - const auto &node_param = nnvm::get(attrs.parsed); + const auto &node_param = nnvm::get(attrs.parsed); for (auto& el : node_param.output_map) { (*out_shape)[std::get<0>(el.second)] = std::get<1>(el.second); } @@ -131,7 +129,7 @@ inline bool TRTInferStorageType(const NodeAttrs& /*attrs*/, const int /*dev_mask inline bool TRTInferType(const NodeAttrs& attrs, std::vector* /*in_dtype*/, std::vector* out_dtype) { - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); for (auto& el : node_param.output_map) { (*out_dtype)[std::get<0>(el.second)] = std::get<3>(el.second); } @@ -140,7 +138,7 @@ inline bool TRTInferType(const NodeAttrs& attrs, std::vector* /*in_dtype*/, inline std::vector TRTListInputNames(const NodeAttrs& attrs) { std::vector output; - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); output.resize(node_param.input_map.size()); for (auto& el : node_param.input_map) { output[el.second] = el.first; @@ -150,7 +148,7 @@ inline std::vector TRTListInputNames(const NodeAttrs& attrs) { inline std::vector TRTListOutputNames(const NodeAttrs& attrs) { std::vector output; - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); output.resize(node_param.output_map.size()); for (auto& el : node_param.output_map) { output[std::get<0>(el.second)] = el.first; @@ -162,11 +160,11 @@ NNVM_REGISTER_OP(_trt_op) .describe(R"code(TRT operation (one engine) )code" ADD_FILELINE) .set_num_inputs([](const NodeAttrs& attrs) { - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); return node_param.input_map.size(); }) .set_num_outputs([](const NodeAttrs& attrs) { - const auto& node_param = nnvm::get(attrs.parsed); + const auto& node_param = nnvm::get(attrs.parsed); return node_param.output_map.size(); }) .set_attr_parser(TRTParamParser) diff --git a/src/operator/contrib/tensorrt.cu b/src/operator/contrib/tensorrt.cu index 2fe8727b7..9a9c3c024 100644 --- a/src/operator/contrib/tensorrt.cu +++ b/src/operator/contrib/tensorrt.cu @@ -52,7 +52,7 @@ void TRTCompute(const OpStatePtr& state, const OpContext& ctx, std::vector bindings; bindings.reserve(param.binding_map.size()); for (auto& p : param.binding_map) { - if (p.second == tensorrt::TypeIO::Inputs) { + if (p.second == nnvm_to_onnx::TypeIO::Inputs) { bindings.emplace_back(inputs[p.first].dptr_); } else { bindings.emplace_back(outputs[p.first].dptr_); diff --git a/src/operator/cudnn_rnn-inl.h b/src/operator/cudnn_rnn-inl.h index 077428f54..7c450b77c 100644 --- a/src/operator/cudnn_rnn-inl.h +++ b/src/operator/cudnn_rnn-inl.h @@ -26,6 +26,8 @@ #ifndef MXNET_OPERATOR_CUDNN_RNN_INL_H_ #define MXNET_OPERATOR_CUDNN_RNN_INL_H_ +#define USE_CUDNN_LSTM_PROJ MXNET_USE_CUDNN == 1 && CUDNN_VERSION >= 7200 + #include #include #include @@ -38,7 +40,7 @@ namespace mxnet { namespace op { #if defined(__CUDACC__) && MXNET_USE_CUDNN == 1 && CUDNN_MAJOR >= 5 template -class CuDNNRNNOp : public Operator{ +class CuDNNRNNOp : public Operator { public: explicit CuDNNRNNOp(RNNParam param) { this->param_ = param; @@ -69,6 +71,32 @@ class CuDNNRNNOp : public Operator{ default: LOG(FATAL) << "Not implmented"; } +#if USE_CUDNN_LSTM_PROJ + if (param_.projection_size.has_value()) { + CHECK_EQ(param_.mode, rnn_enum::kLstm) + << "Projection is only supported for LSTM."; + CHECK_GE(param_.state_size, param_.projection_size.value()) + << "State size must be larger than projection size."; + } +#else + CHECK(!param_.projection_size.has_value()) + << "Projection is only supported for LSTM with CuDNN version later than 7.1.1."; +#endif +#if USE_CUDNN_LSTM_PROJ + if (param_.lstm_state_clip_min.has_value() + || param_.lstm_state_clip_max.has_value()) { + CHECK_EQ(param_.mode, rnn_enum::kLstm) + << "State clipping is only supported for LSTM."; + CHECK(param_.lstm_state_clip_min.has_value() && param_.lstm_state_clip_max.has_value()) + << "lstm_state_clip_min and lstm_state_clip_max must be specified together."; + CHECK_GE(param_.lstm_state_clip_max.value(), param_.lstm_state_clip_min.value()) + << "lstm_state_clip_max must be greater or equal to lstm_state_clip_min"; + } +#else + CHECK(!param_.lstm_state_clip_min.has_value() + && !param_.lstm_state_clip_max.has_value()) + << "State clipping is only supported for LSTM with CuDNN version later than 7.2.1."; +#endif // RNN Direction direction_ = param_.bidirectional ? CUDNN_BIDIRECTIONAL : CUDNN_UNIDIRECTIONAL; // Other @@ -92,6 +120,13 @@ class CuDNNRNNOp : public Operator{ CUDNN_CALL(cudnnCreateRNNDescriptor(&rnn_desc_)); CUDNN_CALL(cudnnCreateDropoutDescriptor(&dropout_desc_)); + + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnCreateRNNDataDescriptor(&x_data_desc_)); + CUDNN_CALL(cudnnCreateRNNDataDescriptor(&y_data_desc_)); + CUDNN_CALL(cudnnCreateRNNDataDescriptor(&dx_data_desc_)); + CUDNN_CALL(cudnnCreateRNNDataDescriptor(&dy_data_desc_)); + #endif } ~CuDNNRNNOp() { @@ -123,6 +158,12 @@ class CuDNNRNNOp : public Operator{ Storage::Get()->Free(dropout_states_); } } + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnDestroyRNNDataDescriptor(x_data_desc_)); + CUDNN_CALL(cudnnDestroyRNNDataDescriptor(y_data_desc_)); + CUDNN_CALL(cudnnDestroyRNNDataDescriptor(dx_data_desc_)); + CUDNN_CALL(cudnnDestroyRNNDataDescriptor(dy_data_desc_)); + #endif } virtual void Forward(const OpContext &ctx, const std::vector &in_data, @@ -169,7 +210,89 @@ class CuDNNRNNOp : public Operator{ Tensor temp_space = ctx.requested[rnn_enum::kTempSpace].get_space_typed( mshadow::Shape1(temp_size), s); + #if USE_CUDNN_LSTM_PROJ + std::vector seqLengthArray(param_.batch_size_, param_.seq_length_); + CUDNN_CALL(cudnnSetRNNDataDescriptor(x_data_desc_, + dtype_, + CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_PACKED, + param_.seq_length_, + param_.batch_size_, + param_.input_size_, + seqLengthArray.data(), + nullptr)); + int out_size = + (param_.projection_size.has_value()) ? param_.projection_size.value() : param_.state_size; + out_size = (param_.bidirectional) ? (out_size * 2) : out_size; + CUDNN_CALL(cudnnSetRNNDataDescriptor(y_data_desc_, + dtype_, + CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_PACKED, + param_.seq_length_, + param_.batch_size_, + out_size, + seqLengthArray.data(), + nullptr)); if (ctx.is_train) { + CUDNN_CALL(cudnnSetRNNDataDescriptor(dx_data_desc_, + dtype_, + CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_PACKED, + param_.seq_length_, + param_.batch_size_, + param_.input_size_, + seqLengthArray.data(), + nullptr)); + CUDNN_CALL(cudnnSetRNNDataDescriptor(dy_data_desc_, + dtype_, + CUDNN_RNN_DATA_LAYOUT_SEQ_MAJOR_PACKED, + param_.seq_length_, + param_.batch_size_, + out_size, + seqLengthArray.data(), + nullptr)); + } + #endif + + #if USE_CUDNN_LSTM_PROJ + bool clip_state = param_.lstm_state_clip_min.has_value(); + bool clip_nan = param_.lstm_state_clip_nan; + CUDNN_CALL(cudnnRNNSetClip(s->dnn_handle_, + rnn_desc_, + clip_state ? CUDNN_RNN_CLIP_MINMAX : CUDNN_RNN_CLIP_NONE, + clip_nan ? CUDNN_NOT_PROPAGATE_NAN : CUDNN_PROPAGATE_NAN, + clip_state ? param_.lstm_state_clip_min.value() : 0.0, + clip_state ? param_.lstm_state_clip_max.value() : 0.0)); + #endif + + if (ctx.is_train) { + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnRNNForwardTrainingEx(s->dnn_handle_, + rnn_desc_, + x_data_desc_, + x.dptr_, + hx_desc_, + hx.dptr_, + cx_desc_, + cx_ptr, + w_desc_, + w.dptr_, + y_data_desc_, + y.dptr_, + hy_desc_, + hy_ptr, + cy_desc_, + cy_ptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + temp_space.dptr_, + workspace_byte_, + reserve_space_.dptr, + reserve_space_byte_)); + #else CUDNN_CALL(cudnnRNNForwardTraining(s->dnn_handle_, rnn_desc_, param_.seq_length_, @@ -191,8 +314,36 @@ class CuDNNRNNOp : public Operator{ workspace_byte_, reserve_space_.dptr, reserve_space_byte_)); + #endif } else { - // inference mode + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnRNNForwardInferenceEx(s->dnn_handle_, + rnn_desc_, + x_data_desc_, + x.dptr_, + hx_desc_, + hx.dptr_, + cx_desc_, + cx_ptr, + w_desc_, + w.dptr_, + y_data_desc_, + y.dptr_, + hy_desc_, + hy_ptr, + cy_desc_, + cy_ptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + nullptr, + temp_space.dptr_, + workspace_byte_)); + #else CUDNN_CALL(cudnnRNNForwardInference(s->dnn_handle_, rnn_desc_, param_.seq_length_, @@ -212,6 +363,7 @@ class CuDNNRNNOp : public Operator{ cy_ptr, temp_space.dptr_, workspace_byte_)); + #endif } } @@ -283,6 +435,52 @@ class CuDNNRNNOp : public Operator{ Tensor temp_space = ctx.requested[rnn_enum::kTempSpace].get_space_typed( mshadow::Shape1(temp_size), s); + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnRNNBackwardDataEx(s->dnn_handle_, + rnn_desc_, + y_data_desc_, + y.dptr_, + dy_data_desc_, + dy.dptr_, + nullptr, + nullptr, + dhy_desc_, + dhy_ptr, + dcy_desc_, + dcy_ptr, + w_desc_, + w.dptr_, + hx_desc_, + hx.dptr_, + cx_desc_, + cx_ptr, + dx_data_desc_, + dx.dptr_, + dhx_desc_, + dhx.dptr_, + dcx_desc_, + dcx_ptr, + nullptr, + nullptr, + temp_space.dptr_, + workspace_byte_, + reserve_space_.dptr, + reserve_space_byte_)); + CUDNN_CALL(cudnnRNNBackwardWeightsEx(s->dnn_handle_, + rnn_desc_, + x_data_desc_, + x.dptr_, + hx_desc_, + hx.dptr_, + y_data_desc_, + y.dptr_, + temp_space.dptr_, + workspace_byte_, + dw_desc_, + dw.dptr_, + reserve_space_.dptr, + reserve_space_byte_)); + #else CUDNN_CALL(cudnnRNNBackwardData(s->dnn_handle_, rnn_desc_, param_.seq_length_, @@ -325,6 +523,7 @@ class CuDNNRNNOp : public Operator{ dw.dptr_, reserve_space_.dptr, reserve_space_byte_)); + #endif } private: @@ -367,8 +566,6 @@ class CuDNNRNNOp : public Operator{ dimA[0] = param_.batch_size_; dimA[1] = param_.input_size_; dimA[2] = 1; - dimA[0] = param_.batch_size_; - dimA[1] = param_.input_size_; strideA[0] = dimA[2] * dimA[1]; strideA[1] = dimA[2]; strideA[2] = 1; @@ -391,10 +588,10 @@ class CuDNNRNNOp : public Operator{ strideA[2] = 1; CUDNN_CALL(cudnnSetTensorNdDescriptor(y_vec[i], - dtype_, - 3, - dimA, - strideA)); + dtype_, + 3, + dimA, + strideA)); CUDNN_CALL(cudnnSetTensorNdDescriptor(dy_vec[i], dtype_, 3, @@ -413,42 +610,85 @@ class CuDNNRNNOp : public Operator{ strideA[0] = dimA[2] * dimA[1]; strideA[1] = dimA[2]; strideA[2] = 1; + #if USE_CUDNN_LSTM_PROJ + int dimB[3]; + int strideB[3]; + dimB[0] = param_.num_layers * (param_.bidirectional ? 2 : 1); + dimB[1] = param_.batch_size_; + dimB[2] = param_.projection_size.has_value() ? + param_.projection_size.value() : param_.state_size; + strideB[0] = dimB[2] * dimB[1]; + strideB[1] = dimB[2]; + strideB[2] = 1; + #endif + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnSetTensorNdDescriptor(hx_desc_, + dtype_, + 3, + dimB, + strideB)); + #else CUDNN_CALL(cudnnSetTensorNdDescriptor(hx_desc_, dtype_, 3, dimA, strideA)); + #endif CUDNN_CALL(cudnnSetTensorNdDescriptor(cx_desc_, dtype_, 3, dimA, strideA)); + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnSetTensorNdDescriptor(hy_desc_, + dtype_, + 3, + dimB, + strideB)); + #else CUDNN_CALL(cudnnSetTensorNdDescriptor(hy_desc_, dtype_, 3, dimA, strideA)); + #endif CUDNN_CALL(cudnnSetTensorNdDescriptor(cy_desc_, dtype_, 3, dimA, strideA)); + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnSetTensorNdDescriptor(dhx_desc_, + dtype_, + 3, + dimB, + strideB)); + #else CUDNN_CALL(cudnnSetTensorNdDescriptor(dhx_desc_, dtype_, 3, dimA, strideA)); + #endif CUDNN_CALL(cudnnSetTensorNdDescriptor(dcx_desc_, dtype_, 3, dimA, strideA)); + #if USE_CUDNN_LSTM_PROJ + CUDNN_CALL(cudnnSetTensorNdDescriptor(dhy_desc_, + dtype_, + 3, + dimB, + strideB)); + #else CUDNN_CALL(cudnnSetTensorNdDescriptor(dhy_desc_, dtype_, 3, dimA, strideA)); + #endif CUDNN_CALL(cudnnSetTensorNdDescriptor(dcy_desc_, dtype_, 3, @@ -470,26 +710,26 @@ class CuDNNRNNOp : public Operator{ seed_)); // RNN descriptors #if CUDNN_MAJOR >= 6 - cudnnRNNAlgo_t rnn_algo = CUDNN_RNN_ALGO_STANDARD; - CUDNN_CALL(cudnnSetRNNDescriptor_v6(s->dnn_handle_, - rnn_desc_, - param_.state_size, - param_.num_layers, - dropout_desc_, - input_mode_, - direction_, - mode_, - rnn_algo, - dtype_)); + cudnnRNNAlgo_t rnn_algo = CUDNN_RNN_ALGO_STANDARD; + CUDNN_CALL(cudnnSetRNNDescriptor_v6(s->dnn_handle_, + rnn_desc_, + param_.state_size, + param_.num_layers, + dropout_desc_, + input_mode_, + direction_, + mode_, + rnn_algo, + dtype_)); #else - CUDNN_CALL(cudnnSetRNNDescriptor(rnn_desc_, - param_.state_size, - param_.num_layers, - dropout_desc_, - input_mode_, - direction_, - mode_, - dtype_)); + CUDNN_CALL(cudnnSetRNNDescriptor(rnn_desc_, + param_.state_size, + param_.num_layers, + dropout_desc_, + input_mode_, + direction_, + mode_, + dtype_)); #endif #if CUDNN_MAJOR >= 7 cudnnMathType_t math_type = CUDNN_DEFAULT_MATH; @@ -503,6 +743,14 @@ class CuDNNRNNOp : public Operator{ #endif CUDNN_CALL(cudnnSetRNNMatrixMathType(rnn_desc_, math_type)); #endif + #if USE_CUDNN_LSTM_PROJ + if (param_.projection_size.has_value()) { + CUDNN_CALL(cudnnSetRNNProjectionLayers(s->dnn_handle_, + rnn_desc_, + param_.projection_size.value(), + 0)); + } + #endif // Get temp space sizes CUDNN_CALL(cudnnGetRNNWorkspaceSize(s->dnn_handle_, rnn_desc_, @@ -591,6 +839,9 @@ class CuDNNRNNOp : public Operator{ size_t workspace_byte_, reserve_space_byte_, dropout_byte_; int workspace_size_, dropout_size_; std::vector x_desc_vec_, y_desc_vec_, dx_desc_vec_, dy_desc_vec_; + #if USE_CUDNN_LSTM_PROJ + cudnnRNNDataDescriptor_t x_data_desc_, y_data_desc_, dx_data_desc_, dy_data_desc_; + #endif cudnnTensorDescriptor_t hx_desc_, cx_desc_; cudnnTensorDescriptor_t hy_desc_, cy_desc_; cudnnTensorDescriptor_t dhx_desc_, dcx_desc_; diff --git a/src/operator/custom/custom.cc b/src/operator/custom/custom.cc index 1bcbbbcf3..2643abbe9 100644 --- a/src/operator/custom/custom.cc +++ b/src/operator/custom/custom.cc @@ -91,7 +91,7 @@ void AttrParser(NodeAttrs* attrs) { CustomParam& params = nnvm::get(attrs->parsed); std::vector keys, vals; - for (auto &p : attrs->dict) { + for (auto& p : attrs->dict) { if (p.first == "op_type") { params.op_type = p.second; } else { @@ -185,7 +185,7 @@ bool InferType(const NodeAttrs& attrs, for (size_t i = 0; i < params.num_args; ++i) { types.push_back((*in_type)[i]); } - for (const auto &i : *out_type) { + for (const auto& i : *out_type) { types.push_back(i); } for (size_t i = 0; i < params.num_auxs; ++i) { @@ -387,11 +387,11 @@ void BackwardEx(const OpStatePtr& state, const OpContext& ctx, cpys.push_back(*nd); ptrs[params.bwd_idx[i]] = reinterpret_cast(nd); } - for (size_t i = 0; i < ptrs.size(); ++i) { + for (auto& ptr : ptrs) { NDArray* nd; - if (ptrs[i] == nullptr) { + if (ptr == nullptr) { nd = new NDArray(); - ptrs[i] = reinterpret_cast(nd); + ptr = reinterpret_cast(nd); } } for (size_t i = 0; i < outputs.size(); ++i) { @@ -457,8 +457,8 @@ inline bool BackwardInferStorageType(const nnvm::NodeAttrs& attrs, stypes[i] = (*iattr)[i]; } - for (size_t i = 0; i < oattr->size(); i++) { - stypes.push_back((*oattr)[i]); + for (int i : *oattr) { + stypes.push_back(i); tags.push_back(2); } diff --git a/src/operator/custom/native_op-inl.h b/src/operator/custom/native_op-inl.h index 05ad124ad..8da04abc0 100644 --- a/src/operator/custom/native_op-inl.h +++ b/src/operator/custom/native_op-inl.h @@ -206,9 +206,9 @@ class NativeOpProp : public OperatorProperty { void Init(const std::vector >& kwargs) override { param_.Init(kwargs); - for (auto iter = kwargs.begin(); iter != kwargs.end(); ++iter) { - if (iter->first == "info") { - sscanf(iter->second.c_str(), "%p", ¶m_.pinfo); + for (const auto& kwarg : kwargs) { + if (kwarg.first == "info") { + sscanf(kwarg.second.c_str(), "%p", ¶m_.pinfo); } } param_.num_inputs_ = ListArguments().size(); @@ -229,10 +229,10 @@ class NativeOpProp : public OperatorProperty { for (const auto& s : *in_shape) size += s.ndim(); std::vector shapes_buffer(size); uint32_t *ptr = shapes_buffer.data(); - for (auto iter = in_shape->begin(); iter != in_shape->end(); ++iter) { + for (const auto& shape : *in_shape) { shapes.push_back(ptr); - ndims.push_back(iter->ndim()); - ptr = nnvm::ShapeTypeCast(iter->begin(), iter->end(), ptr); + ndims.push_back(shape.ndim()); + ptr = nnvm::ShapeTypeCast(shape.begin(), shape.end(), ptr); } shapes.resize(param_.num_inputs_+param_.num_outputs_); ndims.resize(param_.num_inputs_+param_.num_outputs_); diff --git a/src/operator/custom/ndarray_op-inl.h b/src/operator/custom/ndarray_op-inl.h index 02c5630b6..5490747d7 100644 --- a/src/operator/custom/ndarray_op-inl.h +++ b/src/operator/custom/ndarray_op-inl.h @@ -131,10 +131,10 @@ class NDArrayOpProp : public OperatorProperty { for (const auto& s : *in_shape) size += s.ndim(); std::vector shapes_buffer(size); uint32_t *ptr = shapes_buffer.data(); - for (auto iter = in_shape->begin(); iter != in_shape->end(); ++iter) { + for (const auto& shape : *in_shape) { shapes.push_back(ptr); - ndims.push_back(iter->ndim()); - ptr = nnvm::ShapeTypeCast(iter->begin(), iter->end(), ptr); + ndims.push_back(shape.ndim()); + ptr = nnvm::ShapeTypeCast(shape.begin(), shape.end(), ptr); } shapes.resize(param_.num_inputs_+param_.num_outputs_); ndims.resize(param_.num_inputs_+param_.num_outputs_); diff --git a/src/operator/elemwise_op_common.h b/src/operator/elemwise_op_common.h index cf44da699..e622ce216 100644 --- a/src/operator/elemwise_op_common.h +++ b/src/operator/elemwise_op_common.h @@ -100,7 +100,7 @@ inline bool ElemwiseStorageAttr(const nnvm::NodeAttrs& attrs, * \tparam rsp whether row sparse stype is supported * \tparam rsp whether csr stype is supported */ -template +template inline bool ElemwiseStorageType(const nnvm::NodeAttrs& attrs, const int dev_mask, DispatchMode* dispatch_mode, @@ -115,7 +115,7 @@ inline bool ElemwiseStorageType(const nnvm::NodeAttrs& attrs, template + index_t n_in = -1, index_t n_out = -1> inline bool ElemwiseAttr(const nnvm::NodeAttrs& attrs, std::vector *in_attrs, std::vector *out_attrs, @@ -128,33 +128,37 @@ inline bool ElemwiseAttr(const nnvm::NodeAttrs& attrs, if (n_out != -1) out_size = static_cast(n_out); - auto deduce = [&](std::vector *vec, size_t size, const char *name) { + CHECK_LE(in_size, in_attrs->size()); + CHECK_LE(out_size, out_attrs->size()); + auto deduce = [&](const std::vector& vec, size_t size, const char *name) { for (size_t i = 0; i < size; ++i) { - CHECK(assign(&dattr, (*vec)[i])) + CHECK(assign(&dattr, vec.at(i))) << "Incompatible attr in node " << attrs.name << " at " << i << "-th " << name << ": " << "expected " << attr_string(dattr) - << ", got " << attr_string((*vec)[i]); + << ", got " << attr_string(vec.at(i)); } }; - deduce(in_attrs, in_size, "input"); - if (reverse_infer) deduce(out_attrs, out_size, "output"); + deduce(*in_attrs, in_size, "input"); + if (reverse_infer) + deduce(*out_attrs, out_size, "output"); auto write = [&](std::vector *vec, size_t size, const char *name) { for (size_t i = 0; i < size; ++i) { - CHECK(assign(&(*vec)[i], dattr)) + CHECK(assign(&(vec->at(i)), dattr)) << "Incompatible attr in node " << attrs.name << " at " << i << "-th " << name << ": " << "expected " << attr_string(dattr) - << ", got " << attr_string((*vec)[i]); + << ", got " << attr_string(vec->at(i)); } }; write(in_attrs, in_size, "input"); write(out_attrs, out_size, "output"); - if (is_none(dattr)) return false; + if (is_none(dattr)) + return false; return true; } -template +template inline bool ElemwiseShape(const nnvm::NodeAttrs& attrs, std::vector *in_attrs, std::vector *out_attrs) { @@ -168,7 +172,7 @@ inline bool ElemwiseShape(const nnvm::NodeAttrs& attrs, attrs, in_attrs, out_attrs, TShape()); } -template +template inline bool ElemwiseType(const nnvm::NodeAttrs& attrs, std::vector *in_attrs, std::vector *out_attrs) { diff --git a/src/operator/grid_generator-inl.h b/src/operator/grid_generator-inl.h index 105630cfc..258ec9ae9 100644 --- a/src/operator/grid_generator-inl.h +++ b/src/operator/grid_generator-inl.h @@ -248,12 +248,12 @@ class GridGeneratorProp : public OperatorProperty { std::vector *out_type, std::vector *aux_type) const override { int dtype = -1; - for (size_t i = 0; i < in_type->size(); ++i) { + for (int type : *in_type) { if (dtype == -1) { - dtype = in_type->at(i); + dtype = type; } else { - CHECK(in_type->at(i) == dtype || - in_type->at(i) == -1) << + CHECK(type == dtype || + type == -1) << "Non-uniform data type in GridGenerator"; } } diff --git a/src/operator/l2_normalization-inl.h b/src/operator/l2_normalization-inl.h index d53e0c5ca..c7e71424a 100644 --- a/src/operator/l2_normalization-inl.h +++ b/src/operator/l2_normalization-inl.h @@ -216,7 +216,7 @@ class L2NormalizationOp : public Operator { } } - private: + protected: L2NormalizationParam param_; }; // class L2NormalizationOp diff --git a/src/operator/l2_normalization.cc b/src/operator/l2_normalization.cc index f2f485ae6..6801a0a20 100644 --- a/src/operator/l2_normalization.cc +++ b/src/operator/l2_normalization.cc @@ -23,13 +23,111 @@ * \brief l2 normalization operator */ #include "./l2_normalization-inl.h" + +/* VisualStudio only supports openmp 2.0 */ +#ifdef _MSC_VER +#define collapse(x) +#endif + namespace mxnet { namespace op { + +template +class L2NormalizationOpCPU : public L2NormalizationOp { + public: + explicit L2NormalizationOpCPU(L2NormalizationParam p) + : L2NormalizationOp(p) {} + void Forward(const OpContext &ctx, const std::vector &in_data, + const std::vector &req, + const std::vector &out_data, + const std::vector &aux_args) override { + using namespace mshadow; + using namespace mshadow::expr; + if (req[l2_normalization::kOut] == kNullOp) return; + CHECK_EQ(req[l2_normalization::kOut], kWriteTo); + CHECK_EQ(in_data.size(), 1U); + CHECK_EQ(out_data.size(), 2U); + Stream *s = ctx.get_stream(); + TShape orig_shape = in_data[l2_normalization::kData].shape_; + auto omp_threads = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); + if (this->param_.mode == l2_normalization::kInstance) { + Shape<2> dshape = Shape2(orig_shape[0], + orig_shape.ProdShape(1, orig_shape.ndim())); + Tensor data = in_data[l2_normalization::kData] + .get_with_shape(dshape, s); + Tensor out = out_data[l2_normalization::kOut] + .get_with_shape(dshape, s); + Tensor norm = out_data[l2_normalization::kNorm].get(s); +#pragma omp parallel for num_threads(omp_threads) + for (int shape0 = 0; shape0 < static_cast(dshape[0]); shape0++) { + norm[shape0] = DType(this->param_.eps); + for (int shape1 = 0; shape1 < static_cast(dshape[1]); shape1++) { + norm[shape0] += data[shape0][shape1] * data[shape0][shape1]; + } + norm[shape0] = std::sqrt(norm[shape0]); + for (int shape1 = 0; shape1 < static_cast(dshape[1]); shape1++) { + out[shape0][shape1] = data[shape0][shape1] / norm[shape0]; + } + } + } else if (this->param_.mode == l2_normalization::kChannel) { + CHECK_GE(orig_shape.ndim(), 3U); + Shape<3> dshape = Shape3(orig_shape[0], orig_shape[1], + orig_shape.ProdShape(2, orig_shape.ndim())); + Tensor data = in_data[l2_normalization::kData] + .get_with_shape(dshape, s); + Tensor out = out_data[l2_normalization::kOut] + .get_with_shape(dshape, s); + Shape<2> norm_shape = Shape2(dshape[0], dshape[2]); + Tensor norm = out_data[l2_normalization::kNorm] + .get_with_shape(norm_shape, s); +#pragma omp parallel for num_threads(omp_threads) collapse(2) + for (int shape0 = 0; shape0 < static_cast(dshape[0]); shape0++) { + for (int shape2 = 0; shape2 < static_cast(dshape[2]); shape2++) { + norm[shape0][shape2] = DType(this->param_.eps); + for (int shape1 = 0; shape1 < static_cast(dshape[1]); shape1++) { + norm[shape0][shape2] += data[shape0][shape1][shape2] * data[shape0][shape1][shape2]; + } + norm[shape0][shape2] = std::sqrt(norm[shape0][shape2]); + for (int shape1 = 0; shape1 < static_cast(dshape[1]); shape1++) { + out[shape0][shape1][shape2] = data[shape0][shape1][shape2] / norm[shape0][shape2]; + } + } + } + } else if (this->param_.mode == l2_normalization::kSpatial) { + CHECK_GE(orig_shape.ndim(), 3U); + Shape<3> dshape = Shape3(orig_shape[0], orig_shape[1], + orig_shape.ProdShape(2, orig_shape.ndim())); + Tensor data = in_data[l2_normalization::kData] + .get_with_shape(dshape, s); + Tensor out = out_data[l2_normalization::kOut] + .get_with_shape(dshape, s); + Shape<2> norm_shape = Shape2(dshape[0], dshape[1]); + Tensor norm = out_data[l2_normalization::kNorm] + .get_with_shape(norm_shape, s); +#pragma omp parallel for num_threads(omp_threads) collapse(2) + for (int shape0 = 0; shape0 < static_cast(dshape[0]); shape0++) { + for (int shape1 = 0; shape1 < static_cast(dshape[1]); shape1++) { + norm[shape0][shape1] = DType(this->param_.eps); + for (int shape2 = 0; shape2 < static_cast(dshape[2]); shape2++) { + norm[shape0][shape1] += data[shape0][shape1][shape2] * data[shape0][shape1][shape2]; + } + norm[shape0][shape1] = std::sqrt(norm[shape0][shape1]); + for (int shape2 = 0; shape2 < static_cast(dshape[2]); shape2++) { + out[shape0][shape1][shape2] = data[shape0][shape1][shape2] / norm[shape0][shape1]; + } + } + } + } else { + LOG(FATAL) << "Unexpected mode in l2 normalization"; + } + } +}; + template<> Operator* CreateOp(L2NormalizationParam param, int dtype) { Operator* op = nullptr; MSHADOW_REAL_TYPE_SWITCH(dtype, DType, { - op = new L2NormalizationOp(param); + op = new L2NormalizationOpCPU(param); }); return op; } @@ -37,7 +135,7 @@ Operator* CreateOp(L2NormalizationParam param, int dtype) { // DO_BIND_DISPATCH comes from static_operator_common.h Operator* L2NormalizationProp::CreateOperatorEx(Context ctx, std::vector *in_shape, std::vector *in_type) const { - DO_BIND_DISPATCH(CreateOp, param_, in_type->at(0)); + DO_BIND_DISPATCH(CreateOp, this->param_, in_type->at(0)); } DMLC_REGISTER_PARAMETER(L2NormalizationParam); diff --git a/src/operator/linalg_impl.h b/src/operator/linalg_impl.h index c0ae97ad3..4e63f61f1 100644 --- a/src/operator/linalg_impl.h +++ b/src/operator/linalg_impl.h @@ -18,7 +18,7 @@ */ /*! - * \file linalg.h + * \file linalg_impl.h * \brief Implementation of unified tensor interface for advanced linear algebra functions * (specifically BLAS3/LAPACK) from within mxnet. */ @@ -315,9 +315,57 @@ void linalg_gemm(const Tensor + inline void linalg_batch_gemm(const Tensor& A, + const Tensor& B, + const Tensor& C, + float alpha, float beta, bool tA, + bool tB, Stream* s) { + using namespace mxnet; + using mshadow::gpu; + CHECK_NOTNULL(s); + linalg_check_batch_size(A.size(0), B.size(0), C.size(0)); + check_gemm(A[0], B[0], C[0], alpha, beta, tA, tB); + auto blas_handle = Stream::GetBlasHandle(s); + bool use_tensor_ops = + GetEnvAllowTensorCore() && GetEnvAllowTensorCoreConversion(); + + using namespace mshadow::cuda; + auto cublas_math_mode = + use_tensor_ops ? CUBLAS_TENSOR_OP_MATH : CUBLAS_DEFAULT_MATH; + auto previous_math_mode = SetCublasMathMode(blas_handle, cublas_math_mode); + + // cublasGemmStridedBatchedEx is only supported for GPU with architecture + // capabilities equal or greater than 5.0. Fall back to + // cublasSgemmStridedBatched, which doesn't support implicit conversion + // to half-precision to use TensorCores + auto cc_major = (s->prop).major; + if ((cc_major >= 5) && use_tensor_ops) { + CUBLAS_CALL(cublasGemmStridedBatchedEx( + blas_handle, (tB ? CUBLAS_OP_T : CUBLAS_OP_N), + (tA ? CUBLAS_OP_T : CUBLAS_OP_N), C.size(2), C.size(1), + (tB ? B.size(2) : B.size(1)), &alpha, B.dptr_, CUDA_R_32F, + B.stride_, B.size(1) * B.stride_, A.dptr_, CUDA_R_32F, A.stride_, + A.size(1) * A.stride_, &beta, C.dptr_, CUDA_R_32F, C.stride_, + C.size(1) * C.stride_, A.size(0), CUDA_R_32F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } else { + CUBLAS_CALL(cublasSgemmStridedBatched( + blas_handle, (tB ? CUBLAS_OP_T : CUBLAS_OP_N), + (tA ? CUBLAS_OP_T : CUBLAS_OP_N), C.size(2), C.size(1), + (tB ? B.size(2) : B.size(1)), &alpha, B.dptr_, B.stride_, + B.size(1) * B.stride_, A.dptr_, A.stride_, A.size(1) * A.stride_, + &beta, C.dptr_, C.stride_, C.size(1) * C.stride_, A.size(0))); + } + SetCublasMathMode(blas_handle, previous_math_mode); + } + #endif // CUDA_VERSION < 9010 + // Version where matrix rows are given by second axis. #define LINALG_GPU_BATCH_GEMM_AXIS(fname, DType) \ template<> inline \ diff --git a/src/operator/math_functions-inl.h b/src/operator/math_functions-inl.h index a5b83ea3c..be5bbe20d 100644 --- a/src/operator/math_functions-inl.h +++ b/src/operator/math_functions-inl.h @@ -60,6 +60,8 @@ double name(double a, double b) { \ return ::name(a, b); \ } +MXNET_UNARY_MATH_FUNC(erf) + MXNET_UNARY_MATH_FUNC(exp) MXNET_UNARY_MATH_FUNC(expm1) diff --git a/src/operator/mshadow_op.h b/src/operator/mshadow_op.h index 06a223dda..0b20a0263 100644 --- a/src/operator/mshadow_op.h +++ b/src/operator/mshadow_op.h @@ -169,6 +169,10 @@ struct softrelu : public mxnet_op::tunable { MXNET_UNARY_MATH_OP(softrelu_grad, -math::expm1(-a)); +MXNET_UNARY_MATH_OP(erf_grad, 2.0 / math::sqrt(PI) * math::exp(-(a * a))); + +MXNET_SIMPLE_UNARY_MATH_OP(erf); + MXNET_SIMPLE_UNARY_MATH_OP(exp); MXNET_SIMPLE_UNARY_MATH_OP(expm1); diff --git a/src/operator/mxnet_op.h b/src/operator/mxnet_op.h index e77569671..6cab19908 100644 --- a/src/operator/mxnet_op.h +++ b/src/operator/mxnet_op.h @@ -289,8 +289,8 @@ inline int get_num_threads(const int N) { /* \brief Compute flattened index given coordinates and shape. */ template -MSHADOW_XINLINE int ravel(const Shape& coord, const Shape& shape) { - int ret = 0; +MSHADOW_XINLINE index_t ravel(const Shape& coord, const Shape& shape) { + index_t ret = 0; #pragma unroll for (int i = 0; i < ndim; ++i) { ret = ret * shape[i] + (shape[i] > coord[i]) * coord[i]; @@ -301,11 +301,11 @@ MSHADOW_XINLINE int ravel(const Shape& coord, const Shape& shape) { /* Compute coordinates from flattened index given shape */ template -MSHADOW_XINLINE Shape unravel(const int idx, const Shape& shape) { +MSHADOW_XINLINE Shape unravel(const index_t idx, const Shape& shape) { Shape ret; #pragma unroll - for (int i = ndim-1, j = idx; i >=0; --i) { - int tmp = j / shape[i]; + for (index_t i = ndim-1, j = idx; i >=0; --i) { + auto tmp = j / shape[i]; ret[i] = j - tmp*shape[i]; j = tmp; } @@ -315,8 +315,8 @@ MSHADOW_XINLINE Shape unravel(const int idx, const Shape& shape) { /* Compute dot product of two vector */ template -MSHADOW_XINLINE int dot(const Shape& coord, const Shape& stride) { - int ret = 0; +MSHADOW_XINLINE index_t dot(const Shape& coord, const Shape& stride) { + index_t ret = 0; #pragma unroll for (int i = 0; i < ndim; ++i) { ret += coord[i] * stride[i]; @@ -327,12 +327,12 @@ MSHADOW_XINLINE int dot(const Shape& coord, const Shape& stride) { /* Combining unravel and dot */ template -MSHADOW_XINLINE int unravel_dot(const int idx, const Shape& shape, +MSHADOW_XINLINE index_t unravel_dot(const index_t idx, const Shape& shape, const Shape& stride) { - int ret = 0; + index_t ret = 0; #pragma unroll - for (int i = ndim-1, j = idx; i >=0; --i) { - int tmp = j / shape[i]; + for (index_t i = ndim-1, j = idx; i >=0; --i) { + auto tmp = j / shape[i]; ret += (j - tmp*shape[i])*stride[i]; j = tmp; } @@ -433,51 +433,51 @@ struct op_with_req { /*! \brief input is one tensor */ template - MSHADOW_XINLINE static void Map(int i, DType *out, const DType *in) { + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *in) { KERNEL_ASSIGN(out[i], req, OP::Map(in[i])); } /*! \brief inputs are two tensors */ template - MSHADOW_XINLINE static void Map(int i, DType *out, const DType *lhs, const DType *rhs) { + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *lhs, const DType *rhs) { KERNEL_ASSIGN(out[i], req, OP::Map(lhs[i], rhs[i])); } /*! \brief input is tensor and a scalar value */ template - MSHADOW_XINLINE static void Map(int i, DType *out, const DType *in, const DType value) { + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *in, const DType value) { KERNEL_ASSIGN(out[i], req, OP::Map(in[i], value)); } /*! \brief input is tensor and two scalar value */ template - MSHADOW_XINLINE static void Map(int i, DType *out, const DType *in, + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *in, const DType value_1, const DType value_2) { KERNEL_ASSIGN(out[i], req, OP::Map(in[i], value_1, value_2)); } /*! \brief No inputs (ie fill to constant value) */ template - MSHADOW_XINLINE static void Map(int i, DType *out) { + MSHADOW_XINLINE static void Map(index_t i, DType *out) { KERNEL_ASSIGN(out[i], req, OP::Map()); } /*! \brief input is single scalar value */ template - MSHADOW_XINLINE static void Map(int i, DType *out, const DType value) { + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType value) { KERNEL_ASSIGN(out[i], req, OP::Map(value)); } /*! \brief inputs are two tensors and a scalar value */ template - MSHADOW_XINLINE static void Map(int i, DType *out, + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *input_1, const DType *input_2, const DType value) { KERNEL_ASSIGN(out[i], req, OP::Map(input_1[i], input_2[i], value)); } /*! \brief inputs are three tensors (ie backward grad with binary grad function) */ template - MSHADOW_XINLINE static void Map(int i, DType *out, + MSHADOW_XINLINE static void Map(index_t i, DType *out, const DType *input_1, const DType *input_2, const DType *input_3) { @@ -498,26 +498,57 @@ struct Kernel { * \brief Launch a generic CPU kernel. * When using this for a new kernel op, add declaration and tuning objects to * operator_tune.cc - * \tparam Args Varargs type to eventually pass to the OP::Map() functoion + * \tparam Args Varargs type to eventually pass to the OP::Map() function * \param N Number of iterations - * \param args Varargs to eventually pass to the OP::Map() functoion + * \param args Varargs to eventually pass to the OP::Map() function */ template - inline static bool Launch(mshadow::Stream *, const int N, Args... args) { + inline static bool Launch(mshadow::Stream *, const size_t N, Args... args) { #ifdef _OPENMP const int omp_threads = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); if (omp_threads < 2) { - for (int i = 0; i < N; ++i) { + for (size_t i = 0; i < N; ++i) { OP::Map(i, args...); } } else { #pragma omp parallel for num_threads(omp_threads) - for (int i = 0; i < N; ++i) { + for (index_t i = 0; i < static_cast(N); ++i) { OP::Map(i, args...); } } #else - for (int i = 0; i < N; ++i) { + for (size_t i = 0; i < N; ++i) { + OP::Map(i, args...); + } +#endif + return true; + } + + /*! + * \brief Launch a generic CPU kernel with dynamic schedule. This is recommended + * for irregular workloads such as spmv. + * When using this for a new kernel op, add declaration and tuning objects to + * operator_tune.cc + * \tparam Args Varargs type to eventually pass to the OP::Map() function + * \param N Number of iterations + * \param args Varargs to eventually pass to the OP::Map() function + */ + template + inline static bool LaunchDynamic(mshadow::Stream *, const int64_t N, Args... args) { +#ifdef _OPENMP + const int omp_threads = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(false); + if (omp_threads < 2) { + for (int64_t i = 0; i < N; ++i) { + OP::Map(i, args...); + } + } else { + #pragma omp parallel for num_threads(omp_threads) schedule(dynamic) + for (int64_t i = 0; i < N; ++i) { + OP::Map(i, args...); + } + } +#else + for (int64_t i = 0; i < N; ++i) { OP::Map(i, args...); } #endif @@ -530,28 +561,28 @@ struct Kernel { * operator_tune.cc * \tparam PRIMITIVE_OP The primitive operation to use for tuning * \tparam DType Data type - * \tparam Args Varargs type to eventually pass to the OP::Map() functoion + * \tparam Args Varargs type to eventually pass to the OP::Map() function * \param N Number of iterations * \param dest Destination pointer (used to infer DType) - * \param args Varargs to eventually pass to the OP::Map() functoion + * \param args Varargs to eventually pass to the OP::Map() function */ template - static void LaunchTuned(mshadow::Stream *, const int N, Args... args) { + static void LaunchTuned(mshadow::Stream *, const size_t N, Args... args) { #ifdef _OPENMP const int omp_threads = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); if (omp_threads < 2 || !tuned_op::UseOMP( - static_cast(N), static_cast(omp_threads))) { - for (int i = 0; i < N; ++i) { + N, static_cast(omp_threads))) { + for (size_t i = 0; i < N; ++i) { OP::Map(i, args...); } } else { #pragma omp parallel for num_threads(omp_threads) - for (int i = 0; i < N; ++i) { + for (index_t i = 0; i < static_cast(N); ++i) { OP::Map(i, args...); } } #else - for (int i = 0; i < N; ++i) { + for (size_t i = 0; i < N; ++i) { OP::Map(i, args...); } #endif @@ -560,20 +591,20 @@ struct Kernel { /*! * \brief Launch custom-tuned kernel where each thread is set to * operate on a contiguous partition - * \tparam Args Varargs type to eventually pass to the OP::Map() functoion + * \tparam Args Varargs type to eventually pass to the OP::Map() function * \param N Number of iterations * \param args Varargs to eventually pass to the UseOMP() and OP::Map() functions */ template - inline static void LaunchEx(mshadow::Stream *s, const int N, Args... args) { + inline static void LaunchEx(mshadow::Stream *s, const size_t N, Args... args) { #ifdef _OPENMP const int omp_threads = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); if (omp_threads < 2) { OP::Map(0, N, args...); } else { - const int length = (N + omp_threads - 1) / omp_threads; + const auto length = (N + omp_threads - 1) / omp_threads; #pragma omp parallel for num_threads(omp_threads) - for (int i = 0; i < N; i += length) { + for (index_t i = 0; i < static_cast(N); i += length) { OP::Map(i, i + length > N ? N - i : length, args...); } } @@ -586,16 +617,16 @@ struct Kernel { * \brief Launch a tunable OP with implicitly-supplied data type * \tparam DType Data type * \tparam T OP type - * \tparam Args Varargs type to eventually pass to the OP::Map() functoion + * \tparam Args Varargs type to eventually pass to the OP::Map() function * \param s Stream (usually null for CPU) * \param N Number of iterations - * \param args Varargs to eventually pass to the OP::Map() functoion + * \param args Varargs to eventually pass to the OP::Map() function * \return Always true */ template static MSHADOW_CINLINE typename std::enable_if::value, bool>::type - Launch(mshadow::Stream *s, const int N, DType *dest, Args... args) { + Launch(mshadow::Stream *s, const size_t N, DType *dest, Args... args) { LaunchTuned(s, N, dest, args...); return true; } @@ -604,16 +635,16 @@ struct Kernel { * \brief Launch a tunable OP wrapper with explicitly-supplied data type (ie op_with_req) * \tparam DType Data type * \tparam T Wrapper type - * \tparam Args Varargs type to eventually pass to the OP::Map() functoion + * \tparam Args Varargs type to eventually pass to the OP::Map() function * \param s Stream (usually null for CPU) * \param N Number of iterations - * \param args Varargs to eventually pass to the OP::Map() functoion + * \param args Varargs to eventually pass to the OP::Map() function * \return Always true */ template static MSHADOW_CINLINE typename std::enable_if::value, bool>::type - Launch(mshadow::Stream *s, const int N, DType *dest, Args... args) { + Launch(mshadow::Stream *s, const size_t N, DType *dest, Args... args) { LaunchTuned(s, N, dest, args...); return true; } @@ -669,7 +700,7 @@ template struct set_to_int : public tunable { // mxnet_op version (when used directly with Kernel<>::Launch()) */ template - MSHADOW_XINLINE static void Map(int i, DType *out) { + MSHADOW_XINLINE static void Map(index_t i, DType *out) { out[i] = DType(val); } // mshadow_op version (when used with op_with_req<>) diff --git a/src/operator/nn/activation-inl.h b/src/operator/nn/activation-inl.h index 2705177f9..1d8e4c2b6 100644 --- a/src/operator/nn/activation-inl.h +++ b/src/operator/nn/activation-inl.h @@ -48,6 +48,9 @@ enum ActivationOpInputs {kData}; enum ActivationOpOutputs {kOut}; enum ActivationOpResource {kTempSpace}; enum ActivationOpType {kReLU, kSigmoid, kTanh, kSoftReLU, kSoftSign}; + +// Get the number of inputs to the gradient depending on the activation type +int GradNumInputs(int act_type); } // activation struct ActivationParam : public dmlc::Parameter { @@ -199,13 +202,8 @@ void ActivationGradCompute(const nnvm::NodeAttrs& attrs, const std::vector& req, const std::vector& outputs) { const ActivationParam& param = nnvm::get(attrs.parsed); -#if (MXNET_USE_CUDNN == 1 || MXNET_USE_MKLDNN == 1) - bool relu = param.act_type == activation::kReLU; - CHECK_EQ(inputs.size(), relu ? 2U : 3U); -#else - bool softsign = param.act_type == activation::kSoftSign; - CHECK_EQ(inputs.size(), softsign ? 3U : 2U); -#endif + const int act_type = param.act_type; + CHECK_EQ(inputs.size(), activation::GradNumInputs(act_type)); CHECK_EQ(outputs.size(), 1U); CHECK_EQ(req.size(), 1U); ActivationGradComputeImpl(attrs, ctx, inputs, req, outputs); diff --git a/src/operator/nn/activation.cc b/src/operator/nn/activation.cc index ba44ebd4e..fb920c31c 100644 --- a/src/operator/nn/activation.cc +++ b/src/operator/nn/activation.cc @@ -30,13 +30,34 @@ #if MXNET_USE_MKLDNN == 1 #include "./mkldnn/mkldnn_base-inl.h" #include "./mkldnn/mkldnn_ops-inl.h" -#endif // MXNET_USE_MKLDNN +#endif // MXNET_USE_MKLDNN == 1 #include "../operator_common.h" #include "../../common/utils.h" namespace mxnet { namespace op { +namespace activation { + +int GradNumInputs(int act_type) { + // check activation.cu \sa ActivationGradCompute + switch (act_type) { + case kReLU: + return 2; + case kSoftReLU: + case kSoftSign: + case kTanh: + case kSigmoid: + return 3; + default: + CHECK(false) << "missing activation type"; + } + // unreachable + return -1; +} + +} // namespace activation + DMLC_REGISTER_PARAMETER(ActivationParam); // This will determine the order of the inputs for backward computation. @@ -44,24 +65,28 @@ struct ActivationGrad { const char *op_name; std::vector operator()(const nnvm::NodePtr& n, const std::vector& ograds) const { + // ograds, output... std::vector heads(ograds.begin(), ograds.end()); heads.emplace_back(nnvm::NodeEntry{n, activation::kOut, 0}); const NodeAttrs& attrs = n->attrs; + using namespace activation; int act_type = dmlc::get(attrs.parsed).act_type; - if (act_type == activation::kSoftSign) { - // for softsign need the inputs to compute the activation. - heads.push_back(n->inputs[activation::kData]); - } - -#if (MXNET_USE_CUDNN == 1 || MXNET_USE_MKLDNN == 1) // for ReLU, no need to pass input data. This enables inplace optimization during the // forward pass. - if (act_type != activation::kReLU && - act_type != activation::kSoftSign) { - heads.push_back(n->inputs[activation::kData]); + // check activation.cu \sa ActivationGradCompute + switch (act_type) { + case kReLU: + break; + case kSoftReLU: + case kSoftSign: + case kTanh: + case kSigmoid: + heads.push_back(n->inputs[activation::kData]); + break; + default: + CHECK(false) << "missing activation type"; } -#endif return MakeGradNode(op_name, n, heads, n->attrs.dict); } }; @@ -72,9 +97,10 @@ static void ActivationComputeExCPU(const nnvm::NodeAttrs& attrs, const std::vector& inputs, const std::vector& req, const std::vector& outputs) { + const ActivationParam& param = nnvm::get(attrs.parsed); CHECK_EQ(inputs.size(), 1U); CHECK_EQ(outputs.size(), 1U); - if (SupportMKLDNN(inputs[0])) { + if (SupportMKLDNNAct(param, inputs[0])) { MKLDNN_OPCHECK_INIT(false, outputs.size(), inputs, outputs); MKLDNNActivationForward(attrs, ctx, inputs[0], req[0], outputs[0]); MKLDNN_OPCHECK_RUN(ActivationCompute, attrs, ctx, inputs, req, outputs); @@ -89,21 +115,19 @@ void ActivationGradComputeExCPU(const nnvm::NodeAttrs& attrs, const std::vector& req, const std::vector& outputs) { const ActivationParam& param = nnvm::get(attrs.parsed); - bool relu = param.act_type == activation::kReLU; - CHECK_EQ(inputs.size(), relu ? 2U : 3U); - if (SupportMKLDNN(inputs[0])) { + CHECK_EQ(inputs.size(), activation::GradNumInputs(param.act_type)); + if (SupportMKLDNNAct(param, inputs[0])) { MKLDNN_OPCHECK_INIT(true, outputs.size(), inputs, outputs); // XXX: for y = relu(x), y is passed as "in_data" to Backward() - MKLDNNActivationBackward(attrs, ctx, inputs[0], relu ? inputs[1] : inputs[2], req[0], + const bool relu = param.act_type == activation::kReLU; + MKLDNNActivationBackward(attrs, ctx, inputs.at(0), relu ? inputs.at(1) : inputs.at(2), req[0], outputs[0]); - MKLDNN_OPCHECK_RUN(ActivationGradCompute, attrs, ctx, inputs, req, outputs); + MKLDNN_OPCHECK_RUN(ActivationGradCompute, attrs, ctx, inputs, req, outputs); return; } FallBackCompute(ActivationGradComputeImpl, attrs, ctx, inputs, req, outputs); } -#endif -#if MXNET_USE_MKLDNN == 1 inline static bool ActivationStorageType(const nnvm::NodeAttrs& attrs, const int dev_mask, DispatchMode* dispatch_mode, @@ -122,16 +146,12 @@ inline static bool BackwardActStorageType(const nnvm::NodeAttrs& attrs, std::vector *in_attrs, std::vector *out_attrs) { const ActivationParam& param = nnvm::get(attrs.parsed); - if (param.act_type != activation::kReLU) { - CHECK_EQ(in_attrs->size(), 3U); - } else { - // for ReLU activation, the backward pass only needs ograd and output - CHECK_EQ(in_attrs->size(), 2U); - } + CHECK_EQ(in_attrs->size(), activation::GradNumInputs(param.act_type)); return MKLDNNStorageType(attrs, dev_mask, SupportMKLDNNAct(param), dispatch_mode, in_attrs, out_attrs); } -#endif +#endif // MXNET_USE_MKLDNN == 1 + MXNET_OPERATOR_REGISTER_UNARY(Activation) .describe(R"code(Applies an activation function element-wise to the input. @@ -163,18 +183,16 @@ The following activation functions are supported: NNVM_REGISTER_OP(_backward_Activation) .set_num_inputs([](const nnvm::NodeAttrs& attrs) { - int act_type = dmlc::get(attrs.parsed).act_type; - // for ReLU activation, the backward pass only needs ograd and output - if (act_type == activation::kReLU) return 2; - return 3; - }) + const int act_type = dmlc::get(attrs.parsed).act_type; + return activation::GradNumInputs(act_type); +}) .set_num_outputs(1) .set_attr("TIsBackward", true) #if MXNET_USE_MKLDNN == 1 .set_attr("FInferStorageType", BackwardActStorageType) #endif -.set_attr("FInferShape", ElemwiseShape<3, 1>) -.set_attr("FInferType", ElemwiseType<3, 1>) +.set_attr("FInferShape", ElemwiseShape<-1, 1>) +.set_attr("FInferType", ElemwiseType<-1, 1>) .set_attr("FInplaceOption", [](const NodeAttrs& attrs){ return std::vector >{{0, 0}}; }) diff --git a/src/operator/nn/activation.cu b/src/operator/nn/activation.cu index 8892cc34f..ec7db844b 100644 --- a/src/operator/nn/activation.cu +++ b/src/operator/nn/activation.cu @@ -54,12 +54,13 @@ void ActivationCompute(const nnvm::NodeAttrs& attrs, CHECK_EQ(inputs.size(), 1U); CHECK_EQ(outputs.size(), 1U); const ActivationParam& param = nnvm::get(attrs.parsed); + const int act_type = param.act_type; // SoftReLU and kSoftSign are both not supported by CUDNN yet - if (param.act_type == activation::kSoftReLU) { + if (act_type == activation::kSoftReLU) { ActivationForward(ctx, inputs[0], req[0], outputs[0]); - } else if (param.act_type == activation::kSoftSign) { + } else if (act_type == activation::kSoftSign) { ActivationForward(ctx, inputs[0], req[0], outputs[0]); } else { @@ -76,23 +77,28 @@ void ActivationGradCompute(const nnvm::NodeAttrs& attrs, const std::vector& req, const std::vector& outputs) { const ActivationParam& param = nnvm::get(attrs.parsed); - bool relu = param.act_type == activation::kReLU; - CHECK_EQ(inputs.size(), relu ? 2U : 3U); + const int act_type = param.act_type; + CHECK_EQ(inputs.size(), activation::GradNumInputs(act_type)); CHECK_EQ(outputs.size(), 1U); CHECK_EQ(req.size(), 1U); // both SoftReLU and SoftSign not supported by CUDNN yet - if (param.act_type == activation::kSoftReLU) { + if (act_type == activation::kSoftReLU) { ActivationBackward( - ctx, inputs[0], inputs[1], req[0], outputs[0]); - } else if (param.act_type == activation::kSoftSign) { + ctx, inputs.at(0), inputs.at(1), req[0], outputs[0]); + } else if (act_type == activation::kSoftSign) { ActivationBackward( - ctx, inputs[0], inputs[2], req[0], outputs[0]); - } else { - MSHADOW_REAL_TYPE_SWITCH(inputs[0].type_flag_, DType, { + ctx, inputs.at(0), inputs.at(2), req[0], outputs[0]); + } else if (act_type == activation::kReLU) { + MSHADOW_REAL_TYPE_SWITCH(inputs.at(0).type_flag_, DType, { // XXX: for y = relu(x), y is passed as "in_data" to Backward() - get_cudnn_op(param).Backward(ctx, inputs[0], relu ? inputs[1] : inputs[2], - inputs[1], req[0], outputs[0]); + get_cudnn_op(param).Backward(ctx, inputs.at(0), inputs.at(1), + inputs.at(1), req[0], outputs[0]); + }); + } else { + MSHADOW_REAL_TYPE_SWITCH(inputs.at(0).type_flag_, DType, { + get_cudnn_op(param).Backward(ctx, inputs.at(0), inputs.at(2), + inputs.at(1), req[0], outputs[0]); }); } } diff --git a/src/operator/nn/batch_norm.cc b/src/operator/nn/batch_norm.cc index be542ba5b..6254a1e18 100644 --- a/src/operator/nn/batch_norm.cc +++ b/src/operator/nn/batch_norm.cc @@ -567,10 +567,9 @@ axis to be the last item in the input shape. Both ``gamma`` and ``beta`` are learnable parameters. But if ``fix_gamma`` is true, then set ``gamma`` to 1 and its gradient to 0. -Note:: - -When fix_gamma is set to True, no sparse support is provided. If fix_gamma is set to False, -the sparse tensors will fallback. +.. Note:: + When ``fix_gamma`` is set to True, no sparse support is provided. If ``fix_gamma is`` set to False, + the sparse tensors will fallback. )code" ADD_FILELINE) .set_num_inputs(5) diff --git a/src/operator/nn/concat.cc b/src/operator/nn/concat.cc index ac8a814ce..711fe9c49 100644 --- a/src/operator/nn/concat.cc +++ b/src/operator/nn/concat.cc @@ -86,14 +86,17 @@ static bool RNNParamConcatShape(const nnvm::NodeAttrs& attrs, CHECK_EQ(in_shape->size(), static_cast(param_.num_args)); TShape dshape; index_t size = 0; - int num_zero = 0; + std::vector zero_indices; int axis = -1; for (int i = 0; i < param_.num_args; ++i) { TShape tmp = (*in_shape)[i]; if (tmp.ndim()) { axis = CheckAxis(param_.dim, tmp.ndim()); - num_zero += tmp[axis] == 0; - size += tmp[axis]; + if (tmp[axis] == 0) { + zero_indices.emplace_back(i); + } else { + size += tmp[axis]; + } tmp[axis] = 0; shape_assign(&dshape, tmp); } @@ -113,18 +116,18 @@ static bool RNNParamConcatShape(const nnvm::NodeAttrs& attrs, << "Incompatible input shape: expected " << dshape << ", got " << (*in_shape)[i]; } - if (!num_zero) dshape[axis] = size; + if (zero_indices.empty()) dshape[axis] = size; CHECK(shape_assign(&(*out_shape)[0], dshape)) << "Incompatible output shape: expected " << dshape << ", got " << (*out_shape)[0]; - if ((*out_shape)[0][axis] != 0 && num_zero) { + if ((*out_shape)[0][axis] != 0 && !zero_indices.empty()) { int residual = (*out_shape)[0][axis] - size; CHECK_GE(residual, 0) << "Input size already exceeds output size. Residual: " << residual; - CHECK(num_zero <= 2 && num_zero >= 0) - << "Expecting 1 or 2 inputs that need shape inference. Got: " << num_zero; + CHECK(zero_indices.size() <= 2 && zero_indices.size() >= 0) + << "Expecting 1 or 2 inputs that need shape inference. Got: " << zero_indices.size(); bool need_infer = !(*out_shape)[0].Size(); - for (int i = 0; i < num_zero; i++) { - (*in_shape)[i*2][axis] = residual / num_zero; + for (int i : zero_indices) { + (*in_shape)[i][axis] = residual / zero_indices.size(); need_infer = need_infer || !(*in_shape)[i].Size(); } return !need_infer; @@ -139,12 +142,12 @@ static bool ConcatType(const nnvm::NodeAttrs& attrs, const ConcatParam& param_ = nnvm::get(attrs.parsed); int dtype = -1; - for (size_t i = 0; i < in_type->size(); ++i) { + for (int i : *in_type) { if (dtype == -1) { - dtype = in_type->at(i); + dtype = i; } else { - CHECK(in_type->at(i) == dtype || - in_type->at(i) == -1) << + CHECK(i == dtype || + i == -1) << "Non-uniform data type in Concat"; } } diff --git a/src/operator/nn/ctc_loss.cc b/src/operator/nn/ctc_loss.cc index c381677b3..d9c7606f2 100644 --- a/src/operator/nn/ctc_loss.cc +++ b/src/operator/nn/ctc_loss.cc @@ -53,6 +53,9 @@ NNVM_REGISTER_OP(CTCLoss) .add_alias("_contrib_CTCLoss") .add_alias("_contrib_ctc_loss") .describe(R"code(Connectionist Temporal Classification Loss. + +.. note:: The existing alias ``contrib_CTCLoss`` is deprecated. + The shapes of the inputs and outputs: - **data**: `(sequence_length, batch_size, alphabet_size)` diff --git a/src/operator/nn/cudnn/cudnn_algoreg-inl.h b/src/operator/nn/cudnn/cudnn_algoreg-inl.h index 3b59fd1c3..21d3a30ba 100644 --- a/src/operator/nn/cudnn/cudnn_algoreg-inl.h +++ b/src/operator/nn/cudnn/cudnn_algoreg-inl.h @@ -30,6 +30,8 @@ #include #include #include +#include +#include #include "../../../common/cuda_utils.h" #include "../convolution-inl.h" #include "../deconvolution-inl.h" @@ -65,7 +67,11 @@ class CuDNNAlgo { template class CuDNNAlgoReg { public: - bool Find(const ParamType ¶m, + using AlgoSetter_t = std::function *, + CuDNNAlgo *, + CuDNNAlgo *)>; + + void FindOrElseRegister(const ParamType ¶m, const std::vector &in_shape, const std::vector &out_shape, cudnnDataType_t cudnn_data_type, @@ -75,7 +81,8 @@ class CuDNNAlgoReg { bool add_to_weight, CuDNNAlgo *fwd, CuDNNAlgo *bwd, - CuDNNAlgo *flt) { + CuDNNAlgo *flt, + const AlgoSetter_t &algo_setter) { CHECK(in_shape.size() == 2 || in_shape.size() == 3); ParamKey key{param, in_shape[0], in_shape[1], out_shape[0], cudnn_data_type, cudnn_forward_compute_type, cudnn_backward_compute_type, sm_arch, add_to_weight}; @@ -85,45 +92,28 @@ class CuDNNAlgoReg { *fwd = i->second.fwd; *bwd = i->second.bwd; *flt = i->second.flt; - return true; - } - return false; - } - - void Register(const ParamType ¶m, - const std::vector &in_shape, - const std::vector &out_shape, - cudnnDataType_t cudnn_data_type, - cudnnDataType_t cudnn_forward_compute_type, - cudnnDataType_t cudnn_backward_compute_type, - int sm_arch, - bool add_to_weight, - const CuDNNAlgo &fwd, - const CuDNNAlgo &bwd, - const CuDNNAlgo &flt) { - CHECK(in_shape.size() == 2 || in_shape.size() == 3); - ParamKey key{param, in_shape[0], in_shape[1], out_shape[0], cudnn_data_type, - cudnn_forward_compute_type, cudnn_backward_compute_type, sm_arch, add_to_weight}; - std::lock_guard guard(lock_); - if (param.cudnn_tune.value() && reg_.size() % 50 == 0) { - LOG(INFO) << "Running performance tests to find the best convolution " - "algorithm, " - "this can take a while... (setting env variable " - "MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)"; - if (reg_.size() >= 1000) { - // Many people are very concerned about this warning, so change the warning once. - if (!is_warning_autotune_) { - LOG(INFO) - << "If you see this message in the middle of training, you are " - "probably using bucketing. Consider setting env variable " - "MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable cudnn tuning."; - is_warning_autotune_ = true; + } else { + if (param.cudnn_tune.value() && reg_.size() % 50 == 0) { + LOG(INFO) << "Running performance tests to find the best convolution " + "algorithm, " + "this can take a while... (setting env variable " + "MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)"; + if (reg_.size() >= 1000) { + // Many people are very concerned about this warning, so change the warning once. + if (!is_warning_autotune_) { + LOG(INFO) + << "If you see this message in the middle of training, you are " + "probably using bucketing. Consider setting env variable " + "MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable cudnn tuning."; + is_warning_autotune_ = true; + } } } + // Call provided function to determine the algos- likely uses cudnnFind() or cudnnGet() + algo_setter(fwd, bwd, flt); + // Save result so future lookups hit in this registry + reg_.insert(std::pair(key, CudnnAlgorithms{*fwd, *bwd, *flt})); } - reg_[key].fwd = fwd; - reg_[key].bwd = bwd; - reg_[key].flt = flt; } static CuDNNAlgoReg *Get(); diff --git a/src/operator/nn/cudnn/cudnn_convolution-inl.h b/src/operator/nn/cudnn/cudnn_convolution-inl.h index 53bd76c9c..3bd6c5a38 100644 --- a/src/operator/nn/cudnn/cudnn_convolution-inl.h +++ b/src/operator/nn/cudnn/cudnn_convolution-inl.h @@ -26,6 +26,7 @@ #ifndef MXNET_OPERATOR_NN_CUDNN_CUDNN_CONVOLUTION_INL_H_ #define MXNET_OPERATOR_NN_CUDNN_CUDNN_CONVOLUTION_INL_H_ +#include #include #include #include @@ -611,236 +612,265 @@ class CuDNNConvolutionOp { } } - void SelectAlgo(const RunContext& rctx, + void CuDNNAlgoSetter(const RunContext& rctx, const std::vector& in_shape, const std::vector& out_shape, cudnnDataType_t cudnn_forward_compute_type, - cudnnDataType_t cudnn_backward_compute_type) { - if (!CuDNNConvAlgoReg::Get()->Find(param_, in_shape, out_shape, dtype_, - cudnn_forward_compute_type, cudnn_backward_compute_type, - SMArch(rctx.ctx.dev_id), add_to_weight_, - &forward_algo_, &back_algo_, &back_algo_w_)) { - mshadow::Stream *s = rctx.get_stream(); - CHECK_EQ(s->dnn_handle_ownership_, mshadow::Stream::OwnHandle); - size_t workspace_byte = static_cast(param_.workspace * sizeof(DType)); - #if CUDNN_MAJOR >= 7 - // Starting with cuDNNv7, the algo number returned by *Get*() is not the entire - // story: the notion of whether the algo ran in Tensor Core mode is not known. - // Since we want to report the Tensor Core mode in the verbose output, we switch - // to using the new *Get*_v7() call. Since the function signature of *Get*_v7() matches - // that of *Find*(), we can unify the find-vs-get logic by using function pointers. - - // Forward Algorithm Find/Get() v7 - std::vector fwd_results(MaxForwardAlgos(s->dnn_handle_)); - int actual_fwd_algos = 0; - auto fwd_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionForwardAlgorithm_v7 - : cudnnFindConvolutionForwardAlgorithm; - CUDNN_CALL((*fwd_algo_discoverer)(s->dnn_handle_, - in_desc_, - filter_desc_, - forward_conv_desc_, - out_desc_, - fwd_results.size(), - &actual_fwd_algos, - fwd_results.data())); - fwd_results.resize(actual_fwd_algos); - AlgoFinalSelect(fwd_results, "forward", - workspace_byte, &forward_algo_); - - // Backprop-to-Filter Algorithm Find/Get() v7 - auto max_bwd_filt_algos = MaxBackwardFilterAlgos(s->dnn_handle_); - std::vector bwd_filt_results(max_bwd_filt_algos); - int actual_bwd_filter_algos = 0; - // In cudnn v7.1.4, find() returned wgrad algos that could fail for large c if we - // were summing into the output (i.e. beta != 0). Get() returned OK algos though. - auto bwd_filter_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardFilterAlgorithm_v7 - : cudnnFindConvolutionBackwardFilterAlgorithm; - CUDNN_CALL((*bwd_filter_algo_discoverer)(s->dnn_handle_, + cudnnDataType_t cudnn_backward_compute_type, + CuDNNAlgo *fwd, + CuDNNAlgo *bwd, + CuDNNAlgo *flt) { + // Not in algo registry, must determine via *Get*() or *Find*() + mshadow::Stream *s = rctx.get_stream(); + CHECK_EQ(s->dnn_handle_ownership_, mshadow::Stream::OwnHandle); + size_t workspace_byte = static_cast(param_.workspace * sizeof(DType)); +#if CUDNN_MAJOR >= 7 + // Starting with cuDNNv7, the algo number returned by *Get*() is not the entire + // story: the notion of whether the algo ran in Tensor Core mode is not known. + // Since we want to report the Tensor Core mode in the verbose output, we switch + // to using the new *Get*_v7() call. Since the function signature of *Get*_v7() matches + // that of *Find*(), we can unify the find-vs-get logic by using function pointers. + + // Forward Algorithm Find/Get() v7 + std::vector fwd_results(MaxForwardAlgos(s->dnn_handle_)); + int actual_fwd_algos = 0; + auto fwd_algo_discoverer = + param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionForwardAlgorithm_v7 + : cudnnFindConvolutionForwardAlgorithm; + CUDNN_CALL((*fwd_algo_discoverer)(s->dnn_handle_, + in_desc_, + filter_desc_, + forward_conv_desc_, + out_desc_, + fwd_results.size(), + &actual_fwd_algos, + fwd_results.data())); + fwd_results.resize(actual_fwd_algos); + AlgoFinalSelect(fwd_results, "forward", + workspace_byte, fwd); + + // Backprop-to-Filter Algorithm Find/Get() v7 + auto max_bwd_filt_algos = MaxBackwardFilterAlgos(s->dnn_handle_); + std::vector bwd_filt_results(max_bwd_filt_algos); + int actual_bwd_filter_algos = 0; + // In cudnn v7.1.4, find() returned wgrad algos that could fail for large c if we + // were summing into the output (i.e. beta != 0). Get() returned OK algos though. + auto bwd_filter_algo_discoverer = + param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardFilterAlgorithm_v7 + : cudnnFindConvolutionBackwardFilterAlgorithm; + CUDNN_CALL((*bwd_filter_algo_discoverer)(s->dnn_handle_, + in_desc_, + out_desc_, + back_conv_desc_w_, + filter_desc_, + bwd_filt_results.size(), + &actual_bwd_filter_algos, + bwd_filt_results.data())); + bwd_filt_results.resize(actual_bwd_filter_algos); + AlgoFinalSelect(bwd_filt_results, "backprop-to-filter", + workspace_byte, flt); + + // Backprop-to-Data Algorithm Find/Get() v7 + auto max_bwd_data_algos = MaxBackwardDataAlgos(s->dnn_handle_); + std::vector bwd_data_results(max_bwd_data_algos); + int actual_bwd_data_algos = 0; + auto bwd_data_algo_discoverer = + param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardDataAlgorithm_v7 + : cudnnFindConvolutionBackwardDataAlgorithm; + CUDNN_CALL((*bwd_data_algo_discoverer)(s->dnn_handle_, + filter_desc_, + out_desc_, + back_conv_desc_, + in_desc_, + bwd_data_results.size(), + &actual_bwd_data_algos, + bwd_data_results.data())); + bwd_data_results.resize(actual_bwd_data_algos); + AlgoFinalSelect(bwd_data_results, "backprop-to-data", + workspace_byte, bwd); +#else + // CUDNN_MAJOR < 7 + const int kMaxAlgos = 10; + int nalgo = kMaxAlgos; + int i = 0; + size_t min_memory_needs = 0; + // Forward Algorithm Find/Get, v6 and earlier + if (CUDNN_MAJOR == 6 && param_.layout.value() == mshadow::kNHWC) { + // In cuDNNv6, for kNHWC, only CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM is + // supported. Hard-coded this since the algo find() or get() throws an FPE. + fwd->Set(CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM, false); + } else if (!param_.cudnn_tune.value()) { + cudnnConvolutionFwdAlgo_t fastest_fwd_algo; + CUDNN_CALL(cudnnGetConvolutionForwardAlgorithm(s->dnn_handle_, in_desc_, - out_desc_, - back_conv_desc_w_, filter_desc_, - bwd_filt_results.size(), - &actual_bwd_filter_algos, - bwd_filt_results.data())); - bwd_filt_results.resize(actual_bwd_filter_algos); - AlgoFinalSelect(bwd_filt_results, "backprop-to-filter", - workspace_byte, &back_algo_w_); - - // Backprop-to-Data Algorithm Find/Get() v7 - auto max_bwd_data_algos = MaxBackwardDataAlgos(s->dnn_handle_); - std::vector bwd_data_results(max_bwd_data_algos); - int actual_bwd_data_algos = 0; - auto bwd_data_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardDataAlgorithm_v7 - : cudnnFindConvolutionBackwardDataAlgorithm; - CUDNN_CALL((*bwd_data_algo_discoverer)(s->dnn_handle_, - filter_desc_, - out_desc_, - back_conv_desc_, - in_desc_, - bwd_data_results.size(), - &actual_bwd_data_algos, - bwd_data_results.data())); - bwd_data_results.resize(actual_bwd_data_algos); - AlgoFinalSelect(bwd_data_results, "backprop-to-data", - workspace_byte, &back_algo_); - #else - // CUDNN_MAJOR < 7 - const int kMaxAlgos = 10; - int nalgo = kMaxAlgos; - int i = 0; - size_t min_memory_needs = 0; - // Forward Algorithm Find/Get, v6 and earlier - if (CUDNN_MAJOR == 6 && param_.layout.value() == mshadow::kNHWC) { - // In cuDNNv6, for kNHWC, only CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM is - // supported. Hard-coded this since the algo find() or get() throws an FPE. - forward_algo_.Set(CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM, false); - } else if (!param_.cudnn_tune.value()) { - cudnnConvolutionFwdAlgo_t fastest_fwd_algo; - CUDNN_CALL(cudnnGetConvolutionForwardAlgorithm(s->dnn_handle_, - in_desc_, - filter_desc_, - forward_conv_desc_, - out_desc_, - CUDNN_CONVOLUTION_FWD_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_fwd_algo)); - forward_algo_.Set(fastest_fwd_algo, false); - } else { - cudnnConvolutionFwdAlgoPerf_t fwd_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionForwardAlgorithm(s->dnn_handle_, - in_desc_, - filter_desc_, - forward_conv_desc_, - out_desc_, - kMaxAlgos, - &nalgo, - fwd_algo)); - i = 0; - while (i < nalgo - && (fwd_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == conv::kLimited - && fwd_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = - (i == 0) ? fwd_algo[i].memory : std::min(min_memory_needs, fwd_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " forward algorithms with minimum memory requirement " - << min_memory_needs << " bytes have been tried. Workspace size is set to " - << workspace_byte << " bytes, please consider reducing the batch/model size, " - << "or increasing workspace size."; - } else { - forward_algo_.Set(fwd_algo[i].algo, false); - } + forward_conv_desc_, + out_desc_, + CUDNN_CONVOLUTION_FWD_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_fwd_algo)); + fwd->Set(fastest_fwd_algo, false); + } else { + cudnnConvolutionFwdAlgoPerf_t fwd_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionForwardAlgorithm(s->dnn_handle_, + in_desc_, + filter_desc_, + forward_conv_desc_, + out_desc_, + kMaxAlgos, + &nalgo, + fwd_algo)); + i = 0; + while (i < nalgo + && (fwd_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == conv::kLimited + && fwd_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = + (i == 0) ? fwd_algo[i].memory : std::min(min_memory_needs, fwd_algo[i].memory); } - // Backprop-to-Filter Algorithm Find/Get, v6 and earlier - if (!param_.cudnn_tune.value()) { - cudnnConvolutionBwdFilterAlgo_t fastest_bwd_filt_algo; - CUDNN_CALL(cudnnGetConvolutionBackwardFilterAlgorithm(s->dnn_handle_, - in_desc_, - out_desc_, - back_conv_desc_w_, - filter_desc_, - CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_bwd_filt_algo)); - back_algo_w_.Set(fastest_bwd_filt_algo, false); + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, "forward"); } else { - cudnnConvolutionBwdFilterAlgoPerf_t bwd_filter_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionBackwardFilterAlgorithm(s->dnn_handle_, - in_desc_, - out_desc_, - back_conv_desc_w_, - filter_desc_, - kMaxAlgos, - &nalgo, - bwd_filter_algo)); - i = 0; - while (i < nalgo - && (bwd_filter_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == conv::kLimited - && bwd_filter_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = (i == 0) ? - bwd_filter_algo[i].memory : - std::min(min_memory_needs, bwd_filter_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " backward filter algorithms with minimum memory requirement " - << min_memory_needs << " bytes have been tried. Workspace size is set to " - << workspace_byte << " bytes, please consider reducing the batch/model size, " - << "or increasing workspace size."; - } else { - back_algo_w_.Set(bwd_filter_algo[i].algo, false); - } + fwd->Set(fwd_algo[i].algo, false); } - // Backprop-to-Data Algorithm Get(), v6 and earlier - if (!param_.cudnn_tune.value()) { - cudnnConvolutionBwdDataAlgo_t fastest_bwd_data_algo; - CUDNN_CALL(cudnnGetConvolutionBackwardDataAlgorithm(s->dnn_handle_, - filter_desc_, - out_desc_, - back_conv_desc_, - in_desc_, - CUDNN_CONVOLUTION_BWD_DATA_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_bwd_data_algo)); - back_algo_.Set(fastest_bwd_data_algo, false); - } else { - cudnnConvolutionBwdDataAlgoPerf_t bwd_data_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionBackwardDataAlgorithm(s->dnn_handle_, - filter_desc_, - out_desc_, - back_conv_desc_, + } + // Backprop-to-Filter Algorithm Find/Get, v6 and earlier + if (!param_.cudnn_tune.value()) { + cudnnConvolutionBwdFilterAlgo_t fastest_bwd_filt_algo; + CUDNN_CALL(cudnnGetConvolutionBackwardFilterAlgorithm(s->dnn_handle_, + in_desc_, + out_desc_, + back_conv_desc_w_, + filter_desc_, + CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_bwd_filt_algo)); + flt->Set(fastest_bwd_filt_algo, false); + } else { + cudnnConvolutionBwdFilterAlgoPerf_t bwd_filter_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionBackwardFilterAlgorithm(s->dnn_handle_, in_desc_, + out_desc_, + back_conv_desc_w_, + filter_desc_, kMaxAlgos, &nalgo, - bwd_data_algo)); - i = 0; - while (i < nalgo - && (bwd_data_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == conv::kLimited - && bwd_data_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = (i == 0) ? - bwd_data_algo[i].memory : - std::min(min_memory_needs, bwd_data_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " backward data algorithms with minimum memory requirement " - << min_memory_needs << " bytes have been tried. Workspace size is set to " - << workspace_byte << " bytes, please consider reducing the batch/model size, " - << "or increasing workspace size."; - } else { - back_algo_.Set(bwd_data_algo[i].algo, false); - } + bwd_filter_algo)); + i = 0; + while (i < nalgo + && (bwd_filter_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == conv::kLimited + && bwd_filter_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = (i == 0) ? + bwd_filter_algo[i].memory : + std::min(min_memory_needs, bwd_filter_algo[i].memory); } - #endif // CUDNN_MAJOR < 7 - - // Fix for issue #11241 - int cudnn_find_issue_max_features = 64 * 1024; - if (add_to_weight_ && Features(in_shape[conv::kData]) >= cudnn_find_issue_max_features) { - this->back_algo_w_.Set(CUDNN_CONVOLUTION_BWD_FILTER_ALGO_1, true); + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, "backward filter"); + } else { + flt->Set(bwd_filter_algo[i].algo, false); } + } + // Backprop-to-Data Algorithm Get(), v6 and earlier + if (!param_.cudnn_tune.value()) { + cudnnConvolutionBwdDataAlgo_t fastest_bwd_data_algo; + CUDNN_CALL(cudnnGetConvolutionBackwardDataAlgorithm(s->dnn_handle_, + filter_desc_, + out_desc_, + back_conv_desc_, + in_desc_, + CUDNN_CONVOLUTION_BWD_DATA_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_bwd_data_algo)); + bwd->Set(fastest_bwd_data_algo, false); + } else { + cudnnConvolutionBwdDataAlgoPerf_t bwd_data_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionBackwardDataAlgorithm(s->dnn_handle_, + filter_desc_, + out_desc_, + back_conv_desc_, + in_desc_, + kMaxAlgos, + &nalgo, + bwd_data_algo)); + i = 0; + while (i < nalgo + && (bwd_data_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == conv::kLimited + && bwd_data_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = (i == 0) ? + bwd_data_algo[i].memory : + std::min(min_memory_needs, bwd_data_algo[i].memory); + } + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, "backward data"); + } else { + bwd->Set(bwd_data_algo[i].algo, false); + } + } +#endif // CUDNN_MAJOR < 7 - // An algo specification by the user may be cached here, but another - // convolution will match only if identically specified. - // We're caching results of *Get* as well as *Find*, but these records - // will be held distinctly because param_.cudnn_tune is part of the key. - CuDNNConvAlgoReg::Get()->Register(param_, in_shape, out_shape, dtype_, - cudnn_forward_compute_type, - cudnn_backward_compute_type, - SMArch(rctx.ctx.dev_id), this->add_to_weight_, - this->forward_algo_, - this->back_algo_, this->back_algo_w_); + // Fix for issue #11241 + int cudnn_find_issue_max_features = 64 * 1024; + if (add_to_weight_ && Features(in_shape[conv::kData]) >= cudnn_find_issue_max_features) { + flt->Set(CUDNN_CONVOLUTION_BWD_FILTER_ALGO_1, true); } + } + + void SelectAlgo(const RunContext& rctx, + const std::vector& in_shape, + const std::vector& out_shape, + cudnnDataType_t cudnn_forward_compute_type, + cudnnDataType_t cudnn_backward_compute_type) { + auto algo_setter = [&](CuDNNAlgo *fwd, + CuDNNAlgo *bwd, + CuDNNAlgo *flt) { + if (param_.cudnn_tune.value() == conv::kOff) { + // The routine will only be calling cudnnGet, so no need to grab the Storage lock. + this->CuDNNAlgoSetter(rctx, in_shape, out_shape, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + fwd, bwd, flt); + } else { + // One potential problem is that cudnnFind() uses cudaMalloc() to directly allocate + // I/O and workspace areas, and these allocations may result in an out-of-memory + // error even though the StorageMangager free pool is not empty. Ideally, cudnnFind + // would use MXNet's storage allocator for its I/O and workspace areas, instead of using + // the area carved out by MXNET_GPU_MEM_POOL_RESERVE. + // To get somewhat the same effect as this, we can pre-allocate the areas needed for the + // I/Os (possibly triggering a desirable StorageManager::ReleaseAll()), followed by a + // DirectFree(), which makes these areas available for cudnn's subsequent cudaMalloc(). + + // Allocate for x (or dx), w (or dw) and y (or dy). + ReserveElements({in_shape[conv::kData].Size(), + in_shape[conv::kWeight].Size(), + out_shape[conv::kOut].Size()}); + + // We're about to call cudnnFind so we need to quiet the system by grabbing + // the Storage lock. Concurrent cudaMalloc's can disrupt the accurate timing + // measurements of the algos, and can prevent the cuda driver's proper freeing + // of cudnnFind's internal temporary allocations. Grabbing the lock might also + // impede other threads from launching work on the GPU. + std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); + this->CuDNNAlgoSetter(rctx, in_shape, out_shape, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + fwd, bwd, flt); + } + }; + + CuDNNConvAlgoReg::Get()->FindOrElseRegister(param_, in_shape, out_shape, dtype_, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + SMArch(rctx.ctx.dev_id), add_to_weight_, + &forward_algo_, &back_algo_, &back_algo_w_, algo_setter); + // If we're allowing Tensor Core variants of the algos to be considered in // *Find*() or *Get*(), but a non-Tensor-Core algo variant is the fastest, // we must change the descriptor to preclude Tensor Core. Simplest is to @@ -859,6 +889,7 @@ class CuDNNConvolutionOp { size_t workspace_byte, CuDNNAlgo *algo) { // Determine the fastest acceptable algo that matches the algo_preference (-1 = any), // regardless of mathType. + bool enforce_determinism = dmlc::GetEnv("MXNET_ENFORCE_DETERMINISM", false); for (decltype(perf_results.size()) i = 0; i != perf_results.size(); ++i) { const auto &result = perf_results[i]; bool algo_is_tensor_core = false; @@ -866,6 +897,9 @@ class CuDNNConvolutionOp { algo_is_tensor_core = result.mathType == CUDNN_TENSOR_OP_MATH; #endif if (result.status == CUDNN_STATUS_SUCCESS && + #if CUDNN_MAJOR >= 7 + (!enforce_determinism || result.determinism == cudnnDeterminism_t::CUDNN_DETERMINISTIC) && + #endif (param_.cudnn_tune.value() != conv::kLimited || result.memory <= workspace_byte)) { algo->Set(result.algo, algo_is_tensor_core); return; @@ -877,6 +911,7 @@ class CuDNNConvolutionOp { << " please consider reducing batch/model size or increasing the workspace size"; } + void GetTempSize(const OpContext& ctx) { mshadow::Stream *s = ctx.get_stream(); size_t back_size = 0, back_size_w = 0; @@ -975,6 +1010,25 @@ class CuDNNConvolutionOp { return c; } + // Make a number of allocations and directly free them, ensuring room for an equivalent set of + // cudaMalloc() calls by (say) cudnnFind(). `elements` spec the alloc size in DTypes, not bytes. + void ReserveElements(const std::vector &elements) { + std::vector handles; + for (size_t alloc_element : elements) + handles.push_back(Storage::Get()->Alloc(alloc_element * sizeof(DType), Context::GPU())); + for (auto &handle : handles) + Storage::Get()->DirectFree(handle); + } + + // Log that no suitable algo was found that met the workspace constraints, then exit. + void LogNoSuitableAlgoAndExit(int num_algos_tried, size_t min_memory_needs, + size_t workspace_byte, std::string algo_kind) { + LOG(FATAL) << num_algos_tried << " " << algo_kind << " with minimum memory requirement " + << min_memory_needs << " bytes have been tried. Workspace size is set to " + << workspace_byte << " bytes, please consider reducing the batch/model size, " + << "or increasing workspace size."; + } + std::vector param_stride_; std::vector param_dilate_; std::vector param_pad_; diff --git a/src/operator/nn/cudnn/cudnn_deconvolution-inl.h b/src/operator/nn/cudnn/cudnn_deconvolution-inl.h index 041bea66f..72ba2c95f 100644 --- a/src/operator/nn/cudnn/cudnn_deconvolution-inl.h +++ b/src/operator/nn/cudnn/cudnn_deconvolution-inl.h @@ -26,6 +26,7 @@ #ifndef MXNET_OPERATOR_NN_CUDNN_CUDNN_DECONVOLUTION_INL_H_ #define MXNET_OPERATOR_NN_CUDNN_CUDNN_DECONVOLUTION_INL_H_ +#include #include #include #include @@ -538,245 +539,273 @@ class CuDNNDeconvolutionOp { } } - void SelectAlgo(const RunContext& rctx, - const std::vector& in_shape, - const std::vector& out_shape, - cudnnDataType_t cudnn_forward_compute_type, - cudnnDataType_t cudnn_backward_compute_type) { - if (!CuDNNDeconvAlgoReg::Get()->Find(param_, in_shape, out_shape, dtype_, - cudnn_forward_compute_type, - cudnn_backward_compute_type, - SMArch(rctx.ctx.dev_id), add_to_weight_, - &forward_algo_, &back_algo_, &back_algo_w_)) { - mshadow::Stream *s = rctx.get_stream(); - CHECK_EQ(s->dnn_handle_ownership_, mshadow::Stream::OwnHandle); - size_t workspace_byte = static_cast(param_.workspace * sizeof(DType)); - #if CUDNN_MAJOR >= 7 - // Starting with cuDNNv7, the algo number returned by *Get*() is not the entire - // story: the notion of whether the algo ran in Tensor Core mode is not known. - // Since we want to report the Tensor Core mode in the verbose output, we switch - // to using the new *Get*_v7() call. Since the function signature of *Get*_v7() matches - // that of *Find*(), we can unify the find-vs-get logic by using function pointers. - - // Forward Algorithm Find/Get() v7 - std::vector fwd_results(MaxForwardAlgos(s->dnn_handle_)); - int actual_fwd_algos = 0; - auto fwd_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionForwardAlgorithm_v7 + void CuDNNAlgoSetter(const RunContext& rctx, + const std::vector& in_shape, + const std::vector& out_shape, + cudnnDataType_t cudnn_forward_compute_type, + cudnnDataType_t cudnn_backward_compute_type, + CuDNNAlgo *fwd, + CuDNNAlgo *bwd, + CuDNNAlgo *flt) { + // Not in algo registry, must determine via *Get*() or *Find*() + mshadow::Stream *s = rctx.get_stream(); + CHECK_EQ(s->dnn_handle_ownership_, mshadow::Stream::OwnHandle); + size_t workspace_byte = static_cast(param_.workspace * sizeof(DType)); +#if CUDNN_MAJOR >= 7 + // Starting with cuDNNv7, the algo number returned by *Get*() is not the entire + // story: the notion of whether the algo ran in Tensor Core mode is not known. + // Since we want to report the Tensor Core mode in the verbose output, we switch + // to using the new *Get*_v7() call. Since the function signature of *Get*_v7() matches + // that of *Find*(), we can unify the find-vs-get logic by using function pointers. + + // Forward Algorithm Find/Get() v7 + std::vector fwd_results(MaxForwardAlgos(s->dnn_handle_)); + int actual_fwd_algos = 0; + auto fwd_algo_discoverer = + param_.cudnn_tune.value() == deconv::kOff ? cudnnGetConvolutionForwardAlgorithm_v7 : cudnnFindConvolutionForwardAlgorithm; - CUDNN_CALL((*fwd_algo_discoverer)(s->dnn_handle_, - out_desc_, - filter_desc_, - back_conv_desc_, // fwd algo used to backprop-to-data - in_desc_, - fwd_results.size(), - &actual_fwd_algos, - fwd_results.data())); - fwd_results.resize(actual_fwd_algos); - AlgoFinalSelect(fwd_results, "forward", - workspace_byte, &forward_algo_); - - // Backprop-to-Filter Algorithm Find/Get() v7 - auto max_bwd_filt_algos = MaxBackwardFilterAlgos(s->dnn_handle_); - std::vector bwd_filt_results(max_bwd_filt_algos); - int actual_bwd_filter_algos = 0; - // In cudnn v7.1.4, find() returned wgrad algos that could fail for large c if we - // were summing into the output (i.e. beta != 0). Get() returned OK algos though. - auto bwd_filter_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardFilterAlgorithm_v7 - : cudnnFindConvolutionBackwardFilterAlgorithm; - CUDNN_CALL((*bwd_filter_algo_discoverer)(s->dnn_handle_, - out_desc_, - in_desc_, - back_conv_desc_, - filter_desc_, - bwd_filt_results.size(), - &actual_bwd_filter_algos, - bwd_filt_results.data())); - bwd_filt_results.resize(actual_bwd_filter_algos); - AlgoFinalSelect(bwd_filt_results, "backprop-to-filter", - workspace_byte, &back_algo_w_); - - // Backprop-to-Data Algorithm Find/Get() v7 - auto max_bwd_data_algos = MaxBackwardDataAlgos(s->dnn_handle_); - std::vector bwd_data_results(max_bwd_data_algos); - int actual_bwd_data_algos = 0; - auto bwd_data_algo_discoverer = - param_.cudnn_tune.value() == conv::kOff ? cudnnGetConvolutionBackwardDataAlgorithm_v7 + CUDNN_CALL((*fwd_algo_discoverer)(s->dnn_handle_, + out_desc_, + filter_desc_, + back_conv_desc_, // fwd algo used to backprop-to-data + in_desc_, + fwd_results.size(), + &actual_fwd_algos, + fwd_results.data())); + fwd_results.resize(actual_fwd_algos); + AlgoFinalSelect(fwd_results, "forward", + workspace_byte, fwd); + + // Backprop-to-Filter Algorithm Find/Get() v7 + auto max_bwd_filt_algos = MaxBackwardFilterAlgos(s->dnn_handle_); + std::vector bwd_filt_results(max_bwd_filt_algos); + int actual_bwd_filter_algos = 0; + // In cudnn v7.1.4, find() returned wgrad algos that could fail for large c if we + // were summing into the output (i.e. beta != 0). Get() returned OK algos though. + auto bwd_filter_algo_discoverer = + param_.cudnn_tune.value() == deconv::kOff ? cudnnGetConvolutionBackwardFilterAlgorithm_v7 + : cudnnFindConvolutionBackwardFilterAlgorithm; + CUDNN_CALL((*bwd_filter_algo_discoverer)(s->dnn_handle_, + out_desc_, + in_desc_, + back_conv_desc_, + filter_desc_, + bwd_filt_results.size(), + &actual_bwd_filter_algos, + bwd_filt_results.data())); + bwd_filt_results.resize(actual_bwd_filter_algos); + AlgoFinalSelect(bwd_filt_results, "backprop-to-filter", + workspace_byte, flt); + // Backprop-to-Data Algorithm Find/Get() v7 + auto max_bwd_data_algos = MaxBackwardDataAlgos(s->dnn_handle_); + std::vector bwd_data_results(max_bwd_data_algos); + int actual_bwd_data_algos = 0; + auto bwd_data_algo_discoverer = + param_.cudnn_tune.value() == deconv::kOff ? cudnnGetConvolutionBackwardDataAlgorithm_v7 : cudnnFindConvolutionBackwardDataAlgorithm; - CUDNN_CALL((*bwd_data_algo_discoverer)(s->dnn_handle_, + CUDNN_CALL((*bwd_data_algo_discoverer)(s->dnn_handle_, + filter_desc_, + in_desc_, + forward_conv_desc_, // bwd algo used in inference + out_desc_, + bwd_data_results.size(), + &actual_bwd_data_algos, + bwd_data_results.data())); + bwd_data_results.resize(actual_bwd_data_algos); + AlgoFinalSelect(bwd_data_results, "backprop-to-data", + workspace_byte, bwd); +#else + // CUDNN_MAJOR < 7 + const int kMaxAlgos = 10; + int nalgo = kMaxAlgos; + int i = 0; + size_t min_memory_needs = 0; + // Forward Algorithm Find/Get, v6 and earlier + if (CUDNN_MAJOR == 6 && param_.layout.value() == mshadow::kNHWC) { + // In cuDNNv6, for kNHWC, only CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM is + // supported. Hard-coded this since the algo find() or get() throws an FPE. + fwd->Set(CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM, false); + } else if (!param_.cudnn_tune.value()) { + cudnnConvolutionFwdAlgo_t fastest_fwd_algo; + CUDNN_CALL(cudnnGetConvolutionForwardAlgorithm(s->dnn_handle_, + out_desc_, + filter_desc_, + back_conv_desc_, // fwd algo used in dgrad + in_desc_, + CUDNN_CONVOLUTION_FWD_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_fwd_algo)); + fwd->Set(fastest_fwd_algo, false); + } else { + cudnnConvolutionFwdAlgoPerf_t fwd_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionForwardAlgorithm(s->dnn_handle_, + out_desc_, + filter_desc_, + back_conv_desc_, // fwd algo used in dgrad + in_desc_, + kMaxAlgos, + &nalgo, + fwd_algo)); + i = 0; + while (i < nalgo + && (fwd_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == deconv::kLimited + && fwd_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = (i == 0) ? + fwd_algo[i].memory : + std::min(min_memory_needs, fwd_algo[i].memory); + } + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, + "forward algos (for use in deconv op backprop-to-data)"); + } else { + fwd->Set(fwd_algo[i].algo, false); + } + } + // Backprop-to-Filter Algorithm Find/Get, v6 and earlier + if (!param_.cudnn_tune.value()) { + cudnnConvolutionBwdFilterAlgo_t fastest_bwd_filt_algo; + CUDNN_CALL(cudnnGetConvolutionBackwardFilterAlgorithm(s->dnn_handle_, + out_desc_, + in_desc_, + back_conv_desc_, + filter_desc_, + CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_bwd_filt_algo)); + flt->Set(fastest_bwd_filt_algo, false); + } else { + cudnnConvolutionBwdFilterAlgoPerf_t bwd_filter_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionBackwardFilterAlgorithm(s->dnn_handle_, + out_desc_, + in_desc_, + back_conv_desc_, + filter_desc_, + kMaxAlgos, + &nalgo, + bwd_filter_algo)); + i = 0; + while (i < nalgo + && (bwd_filter_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == deconv::kLimited + && bwd_filter_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = (i == 0) ? + bwd_filter_algo[i].memory : + std::min(min_memory_needs, bwd_filter_algo[i].memory); + } + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, + "backward filter algos (for use in deconv op backprop-to-filter)"); + } else { + flt->Set(bwd_filter_algo[i].algo, false); + } + } + // Backprop-to-Data Algorithm Get(), v6 and earlier + if (!param_.cudnn_tune.value()) { + cudnnConvolutionBwdDataAlgo_t fastest_bwd_data_algo; + CUDNN_CALL(cudnnGetConvolutionBackwardDataAlgorithm(s->dnn_handle_, + filter_desc_, + in_desc_, + forward_conv_desc_, // bwd algo used for inference + out_desc_, + CUDNN_CONVOLUTION_BWD_DATA_SPECIFY_WORKSPACE_LIMIT, + workspace_byte, + &fastest_bwd_data_algo)); + bwd->Set(fastest_bwd_data_algo, false); + } else { + cudnnConvolutionBwdDataAlgoPerf_t bwd_data_algo[kMaxAlgos]; + CUDNN_CALL(cudnnFindConvolutionBackwardDataAlgorithm(s->dnn_handle_, filter_desc_, in_desc_, forward_conv_desc_, // bwd algo used in inference out_desc_, - bwd_data_results.size(), - &actual_bwd_data_algos, - bwd_data_results.data())); - bwd_data_results.resize(actual_bwd_data_algos); - AlgoFinalSelect(bwd_data_results, "backprop-to-data", - workspace_byte, &back_algo_); - #else - // CUDNN_MAJOR < 7 - const int kMaxAlgos = 10; - int nalgo = kMaxAlgos; - int i = 0; - size_t min_memory_needs = 0; - // Forward Algorithm Find/Get, v6 and earlier - if (CUDNN_MAJOR == 6 && param_.layout.value() == mshadow::kNHWC) { - // In cuDNNv6, for kNHWC, only CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM is - // supported. Hard-coded this since the algo find() or get() throws an FPE. - forward_algo_.Set(CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM, false); - } else if (!param_.cudnn_tune.value()) { - cudnnConvolutionFwdAlgo_t fastest_fwd_algo; - CUDNN_CALL(cudnnGetConvolutionForwardAlgorithm(s->dnn_handle_, - out_desc_, - filter_desc_, - back_conv_desc_, // fwd algo used in dgrad - in_desc_, - CUDNN_CONVOLUTION_FWD_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_fwd_algo)); - forward_algo_.Set(fastest_fwd_algo, false); - } else { - cudnnConvolutionFwdAlgoPerf_t fwd_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionForwardAlgorithm(s->dnn_handle_, - out_desc_, - filter_desc_, - back_conv_desc_, // fwd algo used in dgrad - in_desc_, - kMaxAlgos, - &nalgo, - fwd_algo)); - i = 0; - while (i < nalgo - && (fwd_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == deconv::kLimited - && fwd_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = (i == 0) ? - fwd_algo[i].memory : - std::min(min_memory_needs, fwd_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " forward algorithms" - << " (for use in deconvolution operator backprop-to-data)" - << " with minimum memory requirement " << min_memory_needs - << " bytes have been tried. Workspace size is set to " << workspace_byte - << " bytes, please consider reducing the batch/model size," - << " or increasing workspace size."; - } else { - forward_algo_.Set(fwd_algo[i].algo, false); - } + kMaxAlgos, + &nalgo, + bwd_data_algo)); + i = 0; + while (i < nalgo + && (bwd_data_algo[i].status != CUDNN_STATUS_SUCCESS + || (param_.cudnn_tune.value() == deconv::kLimited + && bwd_data_algo[i].memory > workspace_byte))) { + ++i; + min_memory_needs = (i == 0) ? + bwd_data_algo[i].memory : + std::min(min_memory_needs, bwd_data_algo[i].memory); } - // Backprop-to-Filter Algorithm Find/Get, v6 and earlier - if (!param_.cudnn_tune.value()) { - cudnnConvolutionBwdFilterAlgo_t fastest_bwd_filt_algo; - CUDNN_CALL(cudnnGetConvolutionBackwardFilterAlgorithm(s->dnn_handle_, - out_desc_, - in_desc_, - back_conv_desc_, - filter_desc_, - CUDNN_CONVOLUTION_BWD_FILTER_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_bwd_filt_algo)); - back_algo_w_.Set(fastest_bwd_filt_algo, false); + if (i == nalgo) { + LogNoSuitableAlgoAndExit(nalgo, min_memory_needs, workspace_byte, + "backward data algos (for use in deconv op forward inference)"); } else { - cudnnConvolutionBwdFilterAlgoPerf_t bwd_filter_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionBackwardFilterAlgorithm(s->dnn_handle_, - out_desc_, - in_desc_, - back_conv_desc_, - filter_desc_, - kMaxAlgos, - &nalgo, - bwd_filter_algo)); - i = 0; - while (i < nalgo - && (bwd_filter_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == deconv::kLimited - && bwd_filter_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = (i == 0) ? - bwd_filter_algo[i].memory : - std::min(min_memory_needs, bwd_filter_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " backward filter algorithms" - << " (for use in deconvolution operator backprop-to-filter)" - << " with minimum memory requirement " << min_memory_needs - << " bytes have been tried. Workspace size is set to " << workspace_byte - << " bytes, please consider reducing the batch/model size," - << " or increasing workspace size."; - } else { - back_algo_w_.Set(bwd_filter_algo[i].algo, false); - } + bwd->Set(bwd_data_algo[i].algo, false); } - // Backprop-to-Data Algorithm Get(), v6 and earlier - if (!param_.cudnn_tune.value()) { - cudnnConvolutionBwdDataAlgo_t fastest_bwd_data_algo; - CUDNN_CALL(cudnnGetConvolutionBackwardDataAlgorithm(s->dnn_handle_, - filter_desc_, - in_desc_, - forward_conv_desc_, // bwd algo used for inference - out_desc_, - CUDNN_CONVOLUTION_BWD_DATA_SPECIFY_WORKSPACE_LIMIT, - workspace_byte, - &fastest_bwd_data_algo)); - back_algo_.Set(fastest_bwd_data_algo, false); + } +#endif // CUDNN_MAJOR < 7 + + // Fix for issue #11241 + int cudnn_find_issue_max_features = 64 * 1024; + // With deconvolution, the algo sensitivity is to a large number of output features + if (add_to_weight_ && Features(out_shape[deconv::kOut]) >= cudnn_find_issue_max_features) { + flt->Set(CUDNN_CONVOLUTION_BWD_FILTER_ALGO_1, true); + } + } + + void SelectAlgo(const RunContext& rctx, + const std::vector& in_shape, + const std::vector& out_shape, + cudnnDataType_t cudnn_forward_compute_type, + cudnnDataType_t cudnn_backward_compute_type) { + auto algo_setter = [&](CuDNNAlgo *fwd, + CuDNNAlgo *bwd, + CuDNNAlgo *flt) { + if (param_.cudnn_tune.value() == deconv::kOff) { + // The routine will only be calling cudnnGet, so no need to grab the Storage lock. + this->CuDNNAlgoSetter(rctx, in_shape, out_shape, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + fwd, bwd, flt); } else { - cudnnConvolutionBwdDataAlgoPerf_t bwd_data_algo[kMaxAlgos]; - CUDNN_CALL(cudnnFindConvolutionBackwardDataAlgorithm(s->dnn_handle_, - filter_desc_, - in_desc_, - forward_conv_desc_, // bwd algo used in inference - out_desc_, - kMaxAlgos, - &nalgo, - bwd_data_algo)); - i = 0; - while (i < nalgo - && (bwd_data_algo[i].status != CUDNN_STATUS_SUCCESS - || (param_.cudnn_tune.value() == deconv::kLimited - && bwd_data_algo[i].memory > workspace_byte))) { - ++i; - min_memory_needs = (i == 0) ? - bwd_data_algo[i].memory : - std::min(min_memory_needs, bwd_data_algo[i].memory); - } - if (i == nalgo) { - LOG(FATAL) << nalgo << " backward data algorithms" - << " (for use in deconvolution operator forward inference) with" - << " minimum memory requirement " << min_memory_needs - << " bytes have been tried. Workspace size is set to " << workspace_byte - << " bytes, please consider reducing the batch/model size," - << " or increasing workspace size."; - } else { - back_algo_.Set(bwd_data_algo[i].algo, false); - } - } - #endif // CUDNN_MAJOR < 7 + // One potential problem is that cudnnFind() uses cudaMalloc() to directly allocate + // I/O and workspace areas, and these allocations may result in an out-of-memory + // error even though the StorageMangager free pool is not empty. Ideally, cudnnFind + // would use MXNet's storage allocator for its I/O and workspace areas, instead of using + // the area carved out by MXNET_GPU_MEM_POOL_RESERVE. + // To get somewhat the same effect as this, we can pre-allocate the areas needed for the + // I/Os (possibly triggering a desirable StorageManager::ReleaseAll()), followed by a + // DirectFree(), which makes these areas available for cudnn's subsequent cudaMalloc(). + + // Allocate for x (or dx), w (or dw) and y (or dy). + ReserveElements({in_shape[conv::kData].Size(), + in_shape[conv::kWeight].Size(), + out_shape[conv::kOut].Size()}); - // Fix for issue #11241 - int cudnn_find_issue_max_features = 64 * 1024; - // With deconvolution, the algo sensitivity is to a large number of output features - if (add_to_weight_ && Features(out_shape[deconv::kOut]) >= cudnn_find_issue_max_features) { - this->back_algo_w_.Set(CUDNN_CONVOLUTION_BWD_FILTER_ALGO_1, true); + // We're about to call cudnnFind so we need to quiet the system by grabbing + // the Storage lock. Concurrent cudaMalloc's can disrupt the accurate timing + // measurements of the algos, and can prevent the cuda driver's proper freeing + // of cudnnFind's internal temporary allocations. Grabbing the lock might also + // impede other threads from launching work on the GPU. + std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); + this->CuDNNAlgoSetter(rctx, in_shape, out_shape, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + fwd, bwd, flt); } + }; + + // An algo specification by the user may be cached here, but another + // convolution will match only if identically specified. + // We're caching results of *Get* as well as *Find*, but these records + // will be held distinctly because param_.cudnn_tune is part of the key. + CuDNNDeconvAlgoReg::Get()->FindOrElseRegister(param_, in_shape, out_shape, dtype_, + cudnn_forward_compute_type, + cudnn_backward_compute_type, + SMArch(rctx.ctx.dev_id), add_to_weight_, + &forward_algo_, &back_algo_, &back_algo_w_, algo_setter); - // An algo specification by the user may be cached here, but another - // convolution will match only if identically specified. - // We're caching results of *Get* as well as *Find*, but these records - // will be held distinctly because param_.cudnn_tune is part of the key. - CuDNNDeconvAlgoReg::Get()->Register(param_, in_shape, out_shape, dtype_, - cudnn_forward_compute_type, - cudnn_backward_compute_type, - SMArch(rctx.ctx.dev_id), this->add_to_weight_, - this->forward_algo_, - this->back_algo_, this->back_algo_w_); - } // If we're allowing Tensor Core variants of the algos to be considered in // *Find*() or *Get*(), but a non-Tensor-Core algo variant is the fastest, // we must change the descriptor to preclude Tensor Core. Simplest is to @@ -800,6 +829,7 @@ class CuDNNDeconvolutionOp { void AlgoFinalSelect(const std::vector &perf_results, std::string kernel_name, size_t workspace_byte, CuDNNAlgo *algo) { // Determine the fastest acceptable algo regardless of mathType. + bool enforce_determinism = dmlc::GetEnv("MXNET_ENFORCE_DETERMINISM", false); for (decltype(perf_results.size()) i = 0; i != perf_results.size(); ++i) { const auto &result = perf_results[i]; bool algo_is_tensor_core = false; @@ -807,6 +837,9 @@ class CuDNNDeconvolutionOp { algo_is_tensor_core = result.mathType == CUDNN_TENSOR_OP_MATH; #endif if (result.status == CUDNN_STATUS_SUCCESS && + #if CUDNN_MAJOR >= 7 + (!enforce_determinism || result.determinism == cudnnDeterminism_t::CUDNN_DETERMINISTIC) && + #endif (param_.cudnn_tune.value() != conv::kLimited || result.memory <= workspace_byte)) { algo->Set(result.algo, algo_is_tensor_core); return; @@ -818,6 +851,7 @@ class CuDNNDeconvolutionOp { << " please consider reducing batch/model size or increasing the workspace size"; } + void GetTempSize(const OpContext& ctx) { mshadow::Stream *s = ctx.get_stream(); size_t back_data_algo_workspace_size = 0; @@ -921,6 +955,26 @@ class CuDNNDeconvolutionOp { return c; } + // Make a number of allocations and directly free them, ensuring room for an equivalent set of + // cudaMalloc() calls by (say) cudnnFind(). `elements` spec the alloc size in DTypes, not bytes. + void ReserveElements(const std::vector &elements) { + std::vector handles; + for (size_t alloc_element : elements) + handles.push_back(Storage::Get()->Alloc(alloc_element * sizeof(DType), Context::GPU())); + for (auto &handle : handles) + Storage::Get()->DirectFree(handle); + } + + + // Log that no suitable algo was found that met the workspace constraints, then exit. + void LogNoSuitableAlgoAndExit(int num_algos_tried, size_t min_memory_needs, + size_t workspace_byte, std::string algo_kind) { + LOG(FATAL) << num_algos_tried << " " << algo_kind << " with minimum memory requirement " + << min_memory_needs << " bytes have been tried. Workspace size is set to " + << workspace_byte << " bytes, please consider reducing the batch/model size, " + << "or increasing workspace size."; + } + std::vector param_stride_; std::vector param_dilate_; diff --git a/src/operator/nn/cudnn/cudnn_pooling-inl.h b/src/operator/nn/cudnn/cudnn_pooling-inl.h index bc3ee3660..89fa73ef5 100644 --- a/src/operator/nn/cudnn/cudnn_pooling-inl.h +++ b/src/operator/nn/cudnn/cudnn_pooling-inl.h @@ -48,7 +48,12 @@ class CuDNNPoolingOp { param_ = p; switch (param_.pool_type) { case pool_enum::kMaxPooling: + #if CUDNN_MAJOR >= 7 + mode_ = dmlc::GetEnv("MXNET_ENFORCE_DETERMINISM", false) ? + CUDNN_POOLING_MAX_DETERMINISTIC : CUDNN_POOLING_MAX; + #else mode_ = CUDNN_POOLING_MAX; + #endif break; case pool_enum::kAvgPooling: if (param_.count_include_pad.has_value() && !param_.count_include_pad.value()) { diff --git a/src/operator/nn/dropout-inl.h b/src/operator/nn/dropout-inl.h index b7c40fbdf..9668c227b 100644 --- a/src/operator/nn/dropout-inl.h +++ b/src/operator/nn/dropout-inl.h @@ -82,7 +82,8 @@ class DropoutOp { static void BernoulliGenerate(common::random::RandGenerator gen, int n, double p, int* r) { typename RandGenerator::Impl genImpl(&gen, 1); - const int seed = 17 + genImpl.rand() % 4096; // NOLINT(runtime/threadsafe_fn) + const int seed = 17 + abs(genImpl.rand() % 4096); + CHECK_GE(seed, 0); const int nthr = engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); #pragma omp parallel num_threads(nthr) { @@ -92,7 +93,7 @@ class DropoutOp { const int my_amount = std::min(my_offset + avg_amount, n) - my_offset; if (my_amount > 0) { VSLStreamStatePtr stream; - vslNewStream(&stream, VSL_BRNG_MCG31, seed + my_offset); + vslNewStream(&stream, VSL_BRNG_MCG31, seed); vslSkipAheadStream(stream, my_offset); viRngBernoulli(VSL_RNG_METHOD_BERNOULLI_ICDF, stream, my_amount, r + my_offset, p); vslDeleteStream(&stream); diff --git a/src/operator/nn/mkldnn/mkldnn_act.cc b/src/operator/nn/mkldnn/mkldnn_act.cc index c914b38b5..8c64888b4 100644 --- a/src/operator/nn/mkldnn/mkldnn_act.cc +++ b/src/operator/nn/mkldnn/mkldnn_act.cc @@ -49,6 +49,15 @@ bool SupportMKLDNNAct(const ActivationParam& param) { || param.act_type == activation::kTanh; } +bool SupportMKLDNNAct(const ActivationParam& param, const NDArray &input) { + // MKL-DNN Activation supports 1d, 2d, 3d, 4d data layout + if ((input.shape().ndim() < 1) || + (input.shape().ndim() > 4) || + (input.dtype() != mshadow::kFloat32)) + return false; + return SupportMKLDNNAct(param); +} + static inline mkldnn::algorithm GetMKLDNNActAlgo(const ActivationParam& param) { switch (param.act_type) { case activation::kReLU: @@ -147,10 +156,7 @@ static MKLDNNActForward &GetActForward(const ActivationParam& param, auto it = fwds.find(key); if (it == fwds.end()) { MKLDNNActForward fwd(param, ctx.is_train, in_data, in_mem); - auto ins_ret = fwds.insert(std::pair( - key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&fwds, key, fwd); } return it->second; } @@ -261,10 +267,7 @@ static inline MKLDNNActBackward &GetActBackward(const ActivationParam ¶m, auto it = bwds.find(key); if (it == bwds.end()) { MKLDNNActBackward bwd(param, in_data, in_mem, *out_grad.GetMKLDNNData()); - auto ins_ret = - bwds.insert(std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&bwds, key, bwd); } return it->second; } diff --git a/src/operator/nn/mkldnn/mkldnn_base-inl.h b/src/operator/nn/mkldnn/mkldnn_base-inl.h index 8a2f4a3e5..18ef3f3e7 100644 --- a/src/operator/nn/mkldnn/mkldnn_base-inl.h +++ b/src/operator/nn/mkldnn/mkldnn_base-inl.h @@ -60,7 +60,7 @@ #include "mxnet/op_attr_types.h" using namespace mkldnn; namespace mxnet { -extern bool EnableMkldnnWarnGenerated(); + // ===== CpuEngine ======================================= // cpu_engine singleton class CpuEngine { @@ -147,6 +147,23 @@ static inline bool MKLDNNEnvSet() { return is_mkldnn_enabled; } +static inline int GetMKLDNNCacheSize() { + static int mkldnn_cache_size = dmlc::GetEnv("MXNET_MKLDNN_CACHE_NUM", -1); + return mkldnn_cache_size; +} + +// TODO(alex): (MXNET-1075) Will remove env variable and calculate cache size during runtime +template +static typename std::unordered_map::iterator AddToCache( + std::unordered_map* cache, const S &key, const I &item) { + int mkldnn_cache_size = GetMKLDNNCacheSize(); + if (mkldnn_cache_size != -1 && static_cast(cache->size()) > mkldnn_cache_size) + cache->erase(cache->begin()); + auto ins_return = cache->insert(std::pair(key, item)); + CHECK(ins_return.second); + return ins_return.first; +} + /* * This is to align address to a certain alignment. */ @@ -158,10 +175,11 @@ struct ConvolutionParam; struct DeconvolutionParam; struct SoftmaxParam; bool SupportMKLDNNAct(const ActivationParam& param); +bool SupportMKLDNNAct(const ActivationParam& param, const NDArray &input); bool SupportMKLDNNConv(const ConvolutionParam& params, const NDArray &input); bool SupportMKLDNNDeconv(const DeconvolutionParam& params, const NDArray &input); bool SupportMKLDNNSoftmax(const SoftmaxParam& param); -} +} // namespace op static int GetTypeSize(int dtype) { int size = -1; @@ -208,6 +226,18 @@ static inline int get_mxnet_type(mkldnn_data_type_t dtype) { } } +static inline size_t GetMemDescSize(const mkldnn::memory::desc &md) { + if (md.data.ndims == 0) return 0; + + size_t ret = 1; + for (int i = 0; i < md.data.ndims; i++) { + ret *= md.data.dims[i]; + } + + ret *= mshadow::mshadow_sizeof(get_mxnet_type(md.data.data_type)); + return ret; +} + inline static mkldnn::memory::desc GetMemDesc(const NDArray &arr, int ndim) { mkldnn::memory::dims dims(ndim); for (size_t i = 0; i < dims.size(); i++) dims[i] = arr.shape()[i]; @@ -221,15 +251,24 @@ inline static mkldnn::memory::desc GetMemDesc(const NDArray &arr) { inline static mkldnn::memory::desc GetWeightDesc(const NDArray &arr, int num_groups) { + auto ndim = arr.shape().ndim(); + mkldnn::memory::dims tz = mkldnn::memory::dims{0}; if (num_groups == 1) { return GetMemDesc(arr); } else { - CHECK_EQ(arr.shape().ndim(), 4U); - mkldnn::memory::dims tz = mkldnn::memory::dims{ num_groups, - static_cast(arr.shape()[0] / num_groups), - static_cast(arr.shape()[1]), - static_cast(arr.shape()[2]), - static_cast(arr.shape()[3])}; + CHECK((ndim == 3) || (ndim == 4)) + << "MKL-DNN weight currectly supports 3d and 4d layout"; + const int N = 0, H = 2, W = 3, C = 1; + if (ndim == 3) { + tz = mkldnn::memory::dims{ + num_groups, static_cast(arr.shape()[N] / num_groups), + static_cast(arr.shape()[C]), static_cast(arr.shape()[H])}; + } else { + tz = mkldnn::memory::dims{ + num_groups, static_cast(arr.shape()[N] / num_groups), + static_cast(arr.shape()[C]), static_cast(arr.shape()[H]), + static_cast(arr.shape()[W])}; + } return mkldnn::memory::desc{tz, get_mkldnn_type(arr.dtype()), mkldnn::memory::format::any}; } diff --git a/src/operator/nn/mkldnn/mkldnn_base.cc b/src/operator/nn/mkldnn/mkldnn_base.cc index a60d6555c..ccb9d7ec0 100644 --- a/src/operator/nn/mkldnn/mkldnn_base.cc +++ b/src/operator/nn/mkldnn/mkldnn_base.cc @@ -70,9 +70,10 @@ mkldnn::memory *TmpMemMgr::Alloc(const mkldnn::memory::primitive_desc &pd) { } else { // If curr_mem has been initialized and we still reach here. It means // the current allocated memory isn't enough. - if (this->curr_mem) + if (this->curr_mem && dmlc::GetEnv("MXNET_MKLDNN_DEBUG", false)) { LOG(WARNING) << "Allocate " << pd.get_size() << " bytes with malloc directly"; + } mkldnn_mem_ptr ret(new mkldnn::memory(pd)); MKLDNNStream::Get()->RegisterMem(ret); return ret.get(); @@ -238,39 +239,49 @@ const mkldnn::memory *GetWeights(const NDArray &arr, return mem; mkldnn::memory::data_type type = get_mkldnn_type(arr.dtype()); + mkldnn::memory::dims tz = mkldnn::memory::dims{0}; + mkldnn::memory::format format = mkldnn::memory::format::format_undef; auto engine = CpuEngine::Get()->get_engine(); + const int O = 0, I = 1, H = 2, W = 3; if (arr.shape().ndim() == 2) { - mkldnn::memory::dims tz = mkldnn::memory::dims{ - static_cast(arr.shape()[0]), static_cast(arr.shape()[1])}; - mkldnn::memory::desc md = - mkldnn::memory::desc{tz, type, mkldnn::memory::format::oi}; - mkldnn::memory::primitive_desc pd = - mkldnn::memory::primitive_desc{md, engine}; - mem = arr.GetMKLDNNData(pd); - } else if (arr.shape().ndim() == 4 && num_groups == 1) { - mkldnn::memory::dims tz = mkldnn::memory::dims{ - static_cast(arr.shape()[0]), static_cast(arr.shape()[1]), - static_cast(arr.shape()[2]), static_cast(arr.shape()[3])}; - mkldnn::memory::desc md = - mkldnn::memory::desc{tz, type, mkldnn::memory::format::oihw}; - mkldnn::memory::primitive_desc pd = - mkldnn::memory::primitive_desc{md, engine}; - mem = arr.GetMKLDNNData(pd); + tz = mkldnn::memory::dims{static_cast(arr.shape()[O]), + static_cast(arr.shape()[I])}; + format = mkldnn::memory::format::oi; + } else if (arr.shape().ndim() == 3) { + tz = num_groups > 1 + ? mkldnn::memory::dims{num_groups, + static_cast(arr.shape()[O] / + num_groups), + static_cast(arr.shape()[I]), + static_cast(arr.shape()[H])} + : mkldnn::memory::dims{static_cast(arr.shape()[O]), + static_cast(arr.shape()[I]), + static_cast(arr.shape()[H])}; + format = num_groups > 1 ? mkldnn::memory::format::goiw + : mkldnn::memory::format::oiw; } else if (arr.shape().ndim() == 4) { - mkldnn::memory::dims tz = mkldnn::memory::dims{ num_groups, - static_cast(arr.shape()[0] / num_groups), - static_cast(arr.shape()[1]), - static_cast(arr.shape()[2]), - static_cast(arr.shape()[3])}; - mkldnn::memory::desc md = - mkldnn::memory::desc{tz, type, mkldnn::memory::format::goihw}; - mkldnn::memory::primitive_desc pd = - mkldnn::memory::primitive_desc{md, engine}; - mem = arr.GetMKLDNNData(pd); + tz = num_groups > 1 + ? mkldnn::memory::dims{num_groups, + static_cast(arr.shape()[O] / + num_groups), + static_cast(arr.shape()[I]), + static_cast(arr.shape()[H]), + static_cast(arr.shape()[W])} + : mkldnn::memory::dims{static_cast(arr.shape()[O]), + static_cast(arr.shape()[I]), + static_cast(arr.shape()[H]), + static_cast(arr.shape()[W])}; + format = num_groups > 1 ? mkldnn::memory::format::goihw + : mkldnn::memory::format::oihw; } else { LOG(FATAL) << "The weight array has an unsupported number of dimensions"; return nullptr; } + mkldnn::memory::desc md = + mkldnn::memory::desc{tz, type, format}; + mkldnn::memory::primitive_desc pd = + mkldnn::memory::primitive_desc{md, engine}; + mem = arr.GetMKLDNNData(pd); if (mem == nullptr) mem = arr.GetMKLDNNDataReorder(target_pd); if (mem->get_primitive_desc() == target_pd) return mem; @@ -284,6 +295,7 @@ mkldnn_memory_format_t GetDefaultFormat(int num_dims) { switch (num_dims) { case 1: return mkldnn_x; case 2: return mkldnn_nc; + case 3: return mkldnn_ncw; case 4: return mkldnn_nchw; case 5: return mkldnn_goihw; default: @@ -300,6 +312,30 @@ mkldnn_memory_format_t GetDefaultFormat(const mkldnn::memory::desc &desc) { return mkldnn_oi; else return desc.data.format; + } else if (desc.data.ndims == 3) { + switch (desc.data.format) { + case mkldnn_ncw: + case mkldnn_nwc: + case mkldnn_nCw8c: + case mkldnn_nCw16c: + return mkldnn_ncw; + case mkldnn_oiw: + case mkldnn_wio: + case mkldnn_Owi8o: + case mkldnn_OIw8i8o: + case mkldnn_OIw8o8i: + case mkldnn_OIw16i16o: + case mkldnn_OIw16o16i: + case mkldnn_Oiw16o: + case mkldnn_Owi16o: + case mkldnn_OIw8i16o2i: + case mkldnn_OIw8o16i2o: + case mkldnn_IOw16o16i: + return mkldnn_oiw; + default: + LOG(FATAL) << "Unknown MKLDNN format for 3 dimensions: " << desc.data.format; + return mkldnn_format_undef; + } } else if (desc.data.ndims == 4) { switch (desc.data.format) { case mkldnn_nchw: @@ -311,9 +347,12 @@ mkldnn_memory_format_t GetDefaultFormat(const mkldnn::memory::desc &desc) { case mkldnn_oihw: case mkldnn_ihwo: case mkldnn_hwio: + case mkldnn_oIhw8i: + case mkldnn_oIhw16i: case mkldnn_OIhw8i8o: case mkldnn_OIhw16i16o: case mkldnn_OIhw4i16o4i: + case mkldnn_OIhw4i16o4i_s8s8: case mkldnn_OIhw8i16o2i: case mkldnn_OIhw8o16i2o: case mkldnn_OIhw8o8i: @@ -325,6 +364,18 @@ mkldnn_memory_format_t GetDefaultFormat(const mkldnn::memory::desc &desc) { case mkldnn_Ohwi16o: case mkldnn_OhIw16o4i: return mkldnn_oihw; + case mkldnn_goiw: + case mkldnn_gOwi8o: + case mkldnn_gOIw8o8i: + case mkldnn_gOIw8i8o: + case mkldnn_gOIw16i16o: + case mkldnn_gOIw16o16i: + case mkldnn_gOiw16o: + case mkldnn_gOwi16o: + case mkldnn_gOIw8i16o2i: + case mkldnn_gOIw8o16i2o: + case mkldnn_gIOw16o16i: + return mkldnn_goiw; default: LOG(FATAL) << "Unknown MKLDNN format for 4 dimensions: " << desc.data.format; return mkldnn_format_undef; diff --git a/src/operator/nn/mkldnn/mkldnn_batch_norm-inl.h b/src/operator/nn/mkldnn/mkldnn_batch_norm-inl.h index e605c9bb1..403baaa94 100644 --- a/src/operator/nn/mkldnn/mkldnn_batch_norm-inl.h +++ b/src/operator/nn/mkldnn/mkldnn_batch_norm-inl.h @@ -198,10 +198,7 @@ static MKLDNNBNForward &GetBNForward(const BatchNormParam& param, auto fwd_pd = _GetFwd(*in_data.GetMKLDNNData(), ctx.is_train, (DType) param.eps, flags); MKLDNNBNForward fwd(fwd_pd, ctx.is_train); - auto ins_ret = fwds.insert(std::pair( - key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&fwds, key, fwd); } return it->second; } @@ -360,10 +357,7 @@ static MKLDNNBNBackward &GetBNBackward( if (it == bwds.end()) { auto bwd_pd = _GetBwd(in_mem, diff_mem, param.eps, flags); MKLDNNBNBackward bwd(bwd_pd); - auto ins_ret = - bwds.insert(std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&bwds, key, bwd); } return it->second; } diff --git a/src/operator/nn/mkldnn/mkldnn_concat-inl.h b/src/operator/nn/mkldnn/mkldnn_concat-inl.h new file mode 100644 index 000000000..d3866cc3d --- /dev/null +++ b/src/operator/nn/mkldnn/mkldnn_concat-inl.h @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file mkldnn_concat-inl.h + * \brief + * \author Wenting Jiang +*/ +#ifndef MXNET_OPERATOR_NN_MKLDNN_MKLDNN_CONCAT_INL_H_ +#define MXNET_OPERATOR_NN_MKLDNN_MKLDNN_CONCAT_INL_H_ + + +#if MXNET_USE_MKLDNN == 1 +#include +#include +#include "../concat-inl.h" +#include "./mkldnn_ops-inl.h" +#include "./mkldnn_base-inl.h" + +namespace mxnet { +namespace op { + +class MKLDNNConcatFwd { + public: + mkldnn::concat::primitive_desc fwd_pd; + + MKLDNNConcatFwd(int concat_dim, const std::vector &data_md) + : fwd_pd(concat_dim, data_md) { + data.resize(data_md.size()); + } + + void SetNewMem(const std::vector &in_data, const mkldnn::memory &output); + + const mkldnn::concat &GetFwd() const; + + private: + std::shared_ptr fwd; + std::vector> data; + std::vector data_mem; + std::shared_ptr out; +}; + +static MKLDNNConcatFwd &GetConcatForward( + int concat_dim, const std::vector &in_data, + const std::vector &data_md) { +#if DMLC_CXX11_THREAD_LOCAL + static thread_local std::unordered_map fwds; +#else + static MX_THREAD_LOCAL std::unordered_map fwds; +#endif + OpSignature key; + key.AddSign(concat_dim); + key.AddSign(in_data); + + auto it = fwds.find(key); + if (it == fwds.end()) { + MKLDNNConcatFwd fwd(concat_dim, data_md); + it = AddToCache(&fwds, key, fwd); + } + return it->second; +} + +} // namespace op +} // namespace mxnet + +#endif // MXNET_USE_MKLDNN == 1 +#endif // MXNET_OPERATOR_NN_MKLDNN_MKLDNN_CONCAT_INL_H_ diff --git a/src/operator/nn/mkldnn/mkldnn_concat.cc b/src/operator/nn/mkldnn/mkldnn_concat.cc index af81e1fe3..8e2b57781 100644 --- a/src/operator/nn/mkldnn/mkldnn_concat.cc +++ b/src/operator/nn/mkldnn/mkldnn_concat.cc @@ -22,79 +22,36 @@ * \brief * \author Wenting Jiang */ -#include "../concat-inl.h" -#include "./mkldnn_ops-inl.h" -#include "./mkldnn_base-inl.h" #if MXNET_USE_MKLDNN == 1 +#include "mkldnn_concat-inl.h" + namespace mxnet { namespace op { -class MKLDNNConcatFwd { - std::shared_ptr fwd; - std::vector> data; - std::vector data_mem; - std::shared_ptr out; - - public: - mkldnn::concat::primitive_desc fwd_pd; - - MKLDNNConcatFwd( - int concat_dim, - const std::vector &data_md): fwd_pd(concat_dim, data_md) { - data.resize(data_md.size()); - } - - void SetNewMem(const std::vector &in_data, - const mkldnn::memory &output) { - CHECK_EQ(in_data.size(), data.size()); - for (size_t i = 0; i < data.size(); i++) { - if (this->data[i] == nullptr) { - this->data[i] = std::shared_ptr(new mkldnn::memory( - in_data[i]->get_primitive_desc(), in_data[i]->get_data_handle())); - this->data_mem.push_back(*this->data[i]); - } else { - this->data[i]->set_data_handle(in_data[i]->get_data_handle()); - } +void MKLDNNConcatFwd::SetNewMem(const std::vector &in_data, + const mkldnn::memory &output) { + CHECK_EQ(in_data.size(), data.size()); + for (size_t i = 0; i < data.size(); i++) { + if (this->data[i] == nullptr) { + this->data[i] = std::shared_ptr( + new mkldnn::memory(in_data[i]->get_primitive_desc(), in_data[i]->get_data_handle())); + this->data_mem.push_back(*this->data[i]); + } else { + this->data[i]->set_data_handle(in_data[i]->get_data_handle()); } - if (this->out == nullptr) - this->out = std::shared_ptr(new mkldnn::memory( - fwd_pd.dst_primitive_desc(), output.get_data_handle())); - else - this->out->set_data_handle(output.get_data_handle()); - - if (this->fwd == nullptr) - fwd.reset(new mkldnn::concat(fwd_pd, data_mem, *out)); } + if (this->out == nullptr) + this->out = std::shared_ptr( + new mkldnn::memory(fwd_pd.dst_primitive_desc(), output.get_data_handle())); + else + this->out->set_data_handle(output.get_data_handle()); - const mkldnn::concat &GetFwd() const { - return *fwd; - } -}; - -static MKLDNNConcatFwd &GetConcatForward( - int concat_dim, const std::vector &in_data, - const std::vector &data_md) { -#if DMLC_CXX11_THREAD_LOCAL - static thread_local std::unordered_map fwds; -#else - static MX_THREAD_LOCAL std::unordered_map fwds; -#endif - OpSignature key; - key.AddSign(concat_dim); - key.AddSign(in_data); - - auto it = fwds.find(key); - if (it == fwds.end()) { - MKLDNNConcatFwd fwd(concat_dim, data_md); - auto ins_ret = fwds.insert(std::pair( - key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; - } - return it->second; + if (this->fwd == nullptr) fwd.reset(new mkldnn::concat(fwd_pd, data_mem, *out)); } +const mkldnn::concat &MKLDNNConcatFwd::GetFwd() const { return *fwd; } + void MKLDNNConcatForward(const nnvm::NodeAttrs& attrs, const OpContext &ctx, const std::vector &in_data, const std::vector &req, diff --git a/src/operator/nn/mkldnn/mkldnn_convolution.cc b/src/operator/nn/mkldnn/mkldnn_convolution.cc index 6a70ae40a..7f423ce45 100644 --- a/src/operator/nn/mkldnn/mkldnn_convolution.cc +++ b/src/operator/nn/mkldnn/mkldnn_convolution.cc @@ -37,9 +37,12 @@ namespace op { DMLC_REGISTER_PARAMETER(MKLDNNConvParam); bool SupportMKLDNNConv(const ConvolutionParam& params, const NDArray &input) { - if (params.kernel.ndim() != 2) + if ((params.kernel.ndim() != 1) && + (params.kernel.ndim() != 2)) return false; - return SupportMKLDNNQuantize(input.dtype()) && input.shape().ndim() == 4; + return SupportMKLDNNQuantize(input.dtype()) && + ((input.shape().ndim() == 3) || + (input.shape().ndim() == 4)); } mkldnn::convolution_forward::primitive_desc GetConvFwdImpl( @@ -51,15 +54,26 @@ mkldnn::convolution_forward::primitive_desc GetConvFwdImpl( auto weight_md = GetWeightDesc(weights, param.conv_param.num_group); auto out_md = GetMemDesc(output); auto engine = CpuEngine::Get()->get_engine(); - CHECK_GE(param.conv_param.stride.ndim(), 2U); - CHECK_GE(param.conv_param.pad.ndim(), 2U); - CHECK_GE(param.conv_param.dilate.ndim(), 2U); - mkldnn::memory::dims strides{0, 0}; - strides[0] = param.conv_param.stride[0]; - strides[1] = param.conv_param.stride[1]; - mkldnn::memory::dims padding{0, 0}; - padding[0] = param.conv_param.pad[0]; - padding[1] = param.conv_param.pad[1]; + mkldnn::memory::dims strides(param.conv_param.kernel.ndim()); + mkldnn::memory::dims padding(param.conv_param.kernel.ndim()); + if (param.conv_param.kernel.ndim() == 1) { + CHECK_GE(param.conv_param.stride.ndim(), 1U); + CHECK_GE(param.conv_param.pad.ndim(), 1U); + CHECK_GE(param.conv_param.dilate.ndim(), 1U); + strides[0] = param.conv_param.stride[0]; + padding[0] = param.conv_param.pad[0]; + } else if (param.conv_param.kernel.ndim() == 2) { + CHECK_GE(param.conv_param.stride.ndim(), 2U); + CHECK_GE(param.conv_param.pad.ndim(), 2U); + CHECK_GE(param.conv_param.dilate.ndim(), 2U); + strides[0] = param.conv_param.stride[0]; + strides[1] = param.conv_param.stride[1]; + padding[0] = param.conv_param.pad[0]; + padding[1] = param.conv_param.pad[1]; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv kernel size " + << param.conv_param.kernel.ndim() << ", supporting only 1 or 2."; + } mkldnn::primitive_attr attr; mkldnn::post_ops ops; if (param.mkldnn_param.with_relu) { @@ -85,32 +99,69 @@ mkldnn::convolution_forward::primitive_desc GetConvFwdImpl( attr.set_int_output_round_mode(round_nearest); } + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. if (param.conv_param.dilate.ndim() == 0 && bias == nullptr) { mkldnn::convolution_forward::desc desc(prop, mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + auto conv_pd = mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + while (conv_pd.dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else if (param.conv_param.dilate.ndim() == 0) { auto bias_md = GetMemDesc(*bias); mkldnn::convolution_forward::desc desc(prop, mkldnn::algorithm::convolution_direct, data_md, weight_md, bias_md, out_md, strides, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + auto conv_pd = mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + while (conv_pd.dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else { - mkldnn::memory::dims dilates{0, 0}; - dilates[0] = param.conv_param.dilate[0] - 1; - dilates[1] = param.conv_param.dilate[1] - 1; + mkldnn::memory::dims dilates(param.conv_param.kernel.ndim()); + if (param.conv_param.dilate.ndim() == 1) { + dilates[0] = param.conv_param.dilate[0] - 1; + } else if (param.conv_param.dilate.ndim() == 2) { + dilates[0] = param.conv_param.dilate[0] - 1; + dilates[1] = param.conv_param.dilate[1] - 1; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv dilate size " + << param.conv_param.dilate.ndim() + << ", supporting only 1 or 2."; + } if (bias == nullptr) { mkldnn::convolution_forward::desc desc(prop, mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + auto conv_pd = mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + while (conv_pd.dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else { auto bias_md = GetMemDesc(*bias); mkldnn::convolution_forward::desc desc(prop, mkldnn::algorithm::convolution_direct, data_md, weight_md, bias_md, out_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + auto conv_pd = mkldnn::convolution_forward::primitive_desc(desc, attr, engine); + while (conv_pd.dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } } } @@ -122,27 +173,63 @@ static mkldnn::convolution_backward_data::primitive_desc GetConvBwdData( auto weight_md = GetWeightDesc(weights, param.num_group); auto out_md = GetMemDesc(output); auto engine = CpuEngine::Get()->get_engine(); - CHECK_GE(param.stride.ndim(), 2U); - CHECK_GE(param.pad.ndim(), 2U); - CHECK_GE(param.dilate.ndim(), 2U); - mkldnn::memory::dims strides{0, 0}; - strides[0] = param.stride[0]; - strides[1] = param.stride[1]; - mkldnn::memory::dims padding{0, 0}; - padding[0] = param.pad[0]; - padding[1] = param.pad[1]; + mkldnn::memory::dims strides(param.kernel.ndim()); + mkldnn::memory::dims padding(param.kernel.ndim()); + if (param.kernel.ndim() == 1) { + CHECK_GE(param.stride.ndim(), 1U); + CHECK_GE(param.pad.ndim(), 1U); + CHECK_GE(param.dilate.ndim(), 1U); + strides[0] = param.stride[0]; + padding[0] = param.pad[0]; + } else if (param.kernel.ndim() == 2) { + CHECK_GE(param.stride.ndim(), 2U); + CHECK_GE(param.pad.ndim(), 2U); + CHECK_GE(param.dilate.ndim(), 2U); + strides[0] = param.stride[0]; + strides[1] = param.stride[1]; + padding[0] = param.pad[0]; + padding[1] = param.pad[1]; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv kernel size " << param.kernel.ndim() + << ", supporting only 1 or 2."; + } + + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. if (param.dilate.ndim() == 0) { mkldnn::convolution_backward_data::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_data::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_data::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.diff_src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else { - mkldnn::memory::dims dilates{0, 0}; - dilates[0] = param.dilate[0] - 1; - dilates[1] = param.dilate[1] - 1; + mkldnn::memory::dims dilates(param.kernel.ndim()); + if (param.dilate.ndim() == 1) { + dilates[0] = param.dilate[0] - 1; + } else if (param.dilate.ndim() == 2) { + dilates[0] = param.dilate[0] - 1; + dilates[1] = param.dilate[1] - 1; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv dilate size " + << param.dilate.ndim() << ", supporting only 1 or 2."; + } mkldnn::convolution_backward_data::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_data::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_data::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.diff_src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } } @@ -154,41 +241,89 @@ static mkldnn::convolution_backward_weights::primitive_desc GetConvBwdWeights( auto weight_md = GetWeightDesc(weights, param.num_group); auto out_md = GetMemDesc(output); auto engine = CpuEngine::Get()->get_engine(); - CHECK_GE(param.stride.ndim(), 2U); - CHECK_GE(param.pad.ndim(), 2U); - CHECK_GE(param.dilate.ndim(), 2U); - mkldnn::memory::dims strides{0, 0}; - strides[0] = param.stride[0]; - strides[1] = param.stride[1]; - mkldnn::memory::dims padding{0, 0}; - padding[0] = param.pad[0]; - padding[1] = param.pad[1]; + mkldnn::memory::dims strides(param.kernel.ndim()); + mkldnn::memory::dims padding(param.kernel.ndim()); + if (param.kernel.ndim() == 1) { + CHECK_GE(param.stride.ndim(), 1U); + CHECK_GE(param.pad.ndim(), 1U); + CHECK_GE(param.dilate.ndim(), 1U); + strides[0] = param.stride[0]; + padding[0] = param.pad[0]; + } else if (param.kernel.ndim() == 2) { + CHECK_GE(param.stride.ndim(), 2U); + CHECK_GE(param.pad.ndim(), 2U); + CHECK_GE(param.dilate.ndim(), 2U); + strides[0] = param.stride[0]; + strides[1] = param.stride[1]; + padding[0] = param.pad[0]; + padding[1] = param.pad[1]; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv kernel size " << param.kernel.ndim() + << ", supporting only 1 or 2."; + } + + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. if (param.dilate.ndim() == 0 && bias == nullptr) { mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.diff_weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else if (param.dilate.ndim() == 0) { auto bias_md = GetMemDesc(*bias); mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, bias_md, out_md, strides, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.diff_weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else { - mkldnn::memory::dims dilates{0, 0}; - dilates[0] = param.dilate[0] - 1; - dilates[1] = param.dilate[1] - 1; + mkldnn::memory::dims dilates(param.kernel.ndim()); + if (param.dilate.ndim() == 1) { + dilates[0] = param.dilate[0] - 1; + } else if (param.dilate.ndim() == 2) { + dilates[0] = param.dilate[0] - 1; + dilates[1] = param.dilate[1] - 1; + } else { + LOG(FATAL) << "Unexpected MKL-DNN Conv dilate size " + << param.dilate.ndim() << ", supporting only 1 or 2."; + } if (bias == nullptr) { mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, out_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.diff_weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } else { auto bias_md = GetMemDesc(*bias); mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, data_md, weight_md, bias_md, out_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto conv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (conv_pd.diff_dst_primitive_desc().get_size() != GetArraySize(output) || + conv_pd.src_primitive_desc().get_size() != GetArraySize(data) || + conv_pd.diff_weights_primitive_desc().get_size() != GetArraySize(weights)) { + CHECK(conv_pd.next_impl()) << "No implementation"; + } + return conv_pd; } } } @@ -261,10 +396,7 @@ MKLDNNConvForward &GetConvFwd(const ConvolutionParam ¶m, full_param.conv_param = param; full_param.mkldnn_param.Init(std::unordered_map()); MKLDNNConvForward fwd(full_param, is_train, data, weights, bias, output); - auto ins_ret = fwds.insert( - std::pair(key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&fwds, key, fwd); } return it->second; } @@ -276,9 +408,18 @@ void MKLDNNConvolutionForwardFullFeature(const MKLDNNConvFullParam ¶m, const std::vector &req, const std::vector &out_data) { TmpMemMgr::Get()->Init(ctx.requested[conv::kTempSpace]); - NDArray weight = in_data[conv::kWeight]; + + auto data = in_data[conv::kData]; + if (data.IsView() && data.IsMKLDNNData()) + data = data.Reorder2Default(); + + auto weight = in_data[conv::kWeight]; + if (weight.IsView() && weight.IsMKLDNNData()) + weight = weight.Reorder2Default(); + bool no_bias = param.conv_param.no_bias && !param.mkldnn_param.with_bn; - auto data_mem = in_data[conv::kData].GetMKLDNNDataReorder( + + auto data_mem = data.GetMKLDNNDataReorder( fwd->fwd_pd.src_primitive_desc()); const mkldnn::memory *weight_mem; if (ctx.is_train) { @@ -480,13 +621,11 @@ static inline MKLDNNConvBackward &GetConvBwd( if (bias) key.AddSign(*bias); + auto it = bwds.find(key); if (it == bwds.end()) { MKLDNNConvBackward bwd(param, data, weights, bias, output, fwd_pd); - auto ins_ret = bwds.insert( - std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&bwds, key, bwd); } return it->second; } @@ -500,19 +639,32 @@ void MKLDNNConvolutionBackward(const nnvm::NodeAttrs& attrs, const OpContext &ct MKLDNNConvFullParam full_param; full_param.conv_param = nnvm::get(attrs.parsed); full_param.mkldnn_param.Init(std::unordered_map()); + + auto data = inputs[conv::kData + 1]; + if (data.IsView() && data.IsMKLDNNData()) + data = data.Reorder2Default(); + + auto weight = inputs[conv::kWeight + 1]; + if (weight.IsView() && weight.IsMKLDNNData()) + weight = weight.Reorder2Default(); + + const NDArray* bias = full_param.conv_param.no_bias ? nullptr : &inputs[conv::kBias + 1]; + + auto out_grad = inputs[conv::kOut]; + if (out_grad.IsView() && out_grad.IsMKLDNNData()) + out_grad = out_grad.Reorder2Default(); + mkldnn::convolution_forward::primitive_desc fwd_pd = GetConvFwdImpl( - full_param, ctx.is_train, inputs[conv::kData + 1], inputs[conv::kWeight + 1], - full_param.conv_param.no_bias ? nullptr : &inputs[conv::kBias + 1], - inputs[conv::kOut]); + full_param, ctx.is_train, data, weight, bias, out_grad); const ConvolutionParam ¶m = full_param.conv_param; CHECK_NE(req[conv::kWeight], kWriteInplace) << "cannot write weight inplace"; - MKLDNNConvBackward &convBwd = GetConvBwd(attrs, inputs[conv::kData + 1], - inputs[conv::kWeight + 1], nullptr, inputs[conv::kOut], fwd_pd); - auto out_grad_mem = inputs[conv::kOut].GetMKLDNNDataReorder( + MKLDNNConvBackward &convBwd = GetConvBwd(attrs, data, + weight, bias, out_grad, fwd_pd); + auto out_grad_mem = out_grad.GetMKLDNNDataReorder( convBwd.bwdData_pd.diff_dst_primitive_desc()); if (req[conv::kData]) { - auto weight_mem = GetWeights(inputs[conv::kWeight + 1], + auto weight_mem = GetWeights(weight, convBwd.bwdData_pd.weights_primitive_desc(), param.num_group); auto in_grad_mem = CreateMKLDNNMem(in_grad[conv::kData], convBwd.bwdData_pd.diff_src_primitive_desc(), req[conv::kData]); @@ -521,14 +673,13 @@ void MKLDNNConvolutionBackward(const nnvm::NodeAttrs& attrs, const OpContext &ct CommitOutput(in_grad[conv::kData], in_grad_mem); } if (req[conv::kWeight]) { - MKLDNNConvBackward &convBwdWeight = GetConvBwd(attrs, inputs[conv::kData + 1], - inputs[conv::kWeight + 1], param.no_bias ? nullptr : &inputs[conv::kBias + 1], - inputs[conv::kOut], fwd_pd); + MKLDNNConvBackward &convBwdWeight = GetConvBwd(attrs, data, + weight, bias, out_grad, fwd_pd); if (convBwdWeight.bwdData_pd.diff_dst_primitive_desc() != convBwdWeight.bwdWeights_pd.diff_dst_primitive_desc()) - out_grad_mem = inputs[conv::kOut].GetMKLDNNDataReorder( + out_grad_mem = out_grad.GetMKLDNNDataReorder( convBwdWeight.bwdWeights_pd.diff_dst_primitive_desc()); - auto data_mem = inputs[conv::kData + 1].GetMKLDNNDataReorder( + auto data_mem = data.GetMKLDNNDataReorder( convBwdWeight.bwdWeights_pd.src_primitive_desc()); auto in_grad_weight = CreateMKLDNNWeightGrad( in_grad[conv::kWeight], diff --git a/src/operator/nn/mkldnn/mkldnn_deconvolution.cc b/src/operator/nn/mkldnn/mkldnn_deconvolution.cc index 54d4f6708..4da48fa3f 100644 --- a/src/operator/nn/mkldnn/mkldnn_deconvolution.cc +++ b/src/operator/nn/mkldnn/mkldnn_deconvolution.cc @@ -52,17 +52,34 @@ static mkldnn::convolution_forward::primitive_desc GetDeconvBwd_( bool has_bias, const mkldnn::memory::desc &out_md, const mkldnn::engine &engine, const mkldnn::memory::dims &strides, const mkldnn::memory::dims &padding, const mkldnn::memory::dims &dilates) { + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. if (!has_bias) { mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, mkldnn::algorithm::convolution_direct, out_md, weights_md, data_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, engine); + auto deconv_pd = mkldnn::convolution_forward::primitive_desc(desc, engine); + while (deconv_pd.dst_primitive_desc().get_size() != GetMemDescSize(data_md) || + deconv_pd.src_primitive_desc().get_size() != GetMemDescSize(out_md) || + deconv_pd.weights_primitive_desc().get_size() != GetMemDescSize(weights_md)) { + CHECK(deconv_pd.next_impl()) << "No implementation"; + } + return deconv_pd; } else { auto bias_md = GetBiasDesc(data_md); mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, mkldnn::algorithm::convolution_direct, out_md, weights_md, bias_md, data_md, strides, dilates, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_forward::primitive_desc(desc, engine); + auto deconv_pd = mkldnn::convolution_forward::primitive_desc(desc, engine); + while (deconv_pd.dst_primitive_desc().get_size() != GetMemDescSize(data_md) || + deconv_pd.src_primitive_desc().get_size() != GetMemDescSize(out_md) || + deconv_pd.weights_primitive_desc().get_size() != GetMemDescSize(weights_md)) { + CHECK(deconv_pd.next_impl()) << "No implementation"; + } + return deconv_pd; } } @@ -90,7 +107,18 @@ static mkldnn::convolution_backward_data::primitive_desc GetDeconvFwdImpl( mkldnn::convolution_backward_data::desc desc(mkldnn::algorithm::convolution_direct, out_md, weight_md, data_md, strides, dilate, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_data::primitive_desc(desc, engine, bwd_pd); + auto deconv_pd = mkldnn::convolution_backward_data::primitive_desc(desc, engine, bwd_pd); + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. + while (deconv_pd.diff_dst_primitive_desc().get_size() != GetMemDescSize(data_md) || + deconv_pd.diff_src_primitive_desc().get_size() != GetMemDescSize(out_md) || + deconv_pd.weights_primitive_desc().get_size() != GetMemDescSize(weight_md)) { + CHECK(deconv_pd.next_impl()) << "No implementation"; + } + return deconv_pd; } static mkldnn::convolution_forward::primitive_desc GetDeconvBwdDataImpl( @@ -137,16 +165,34 @@ GetDeconvBwdWeightsImpl( mkldnn::memory::dims dilate{0, 0}; dilate[0] = param.dilate[0] - 1; dilate[1] = param.dilate[1] - 1; + + // MKL-DNN introduced padded formats since 0.15 which require more memory + // for computation compared with the actual tensor size. Currently, MKL-DNN + // operators are still reusing those memory from memory planning and the + // memory size may smaller than what MKL-DNN kernels require. So here we need + // select suboptimal kernel for computation according to tensor sizes. if (!has_bias) { mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, out_md, weight_md, data_md, strides, dilate, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto deconv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (deconv_pd.diff_dst_primitive_desc().get_size() != GetMemDescSize(data_md) || + deconv_pd.src_primitive_desc().get_size() != GetMemDescSize(out_md) || + deconv_pd.diff_weights_primitive_desc().get_size() != GetMemDescSize(weight_md)) { + CHECK(deconv_pd.next_impl()) << "No implementation"; + } + return deconv_pd; } else { auto bias_md = GetBiasDesc(data_md); mkldnn::convolution_backward_weights::desc desc(mkldnn::algorithm::convolution_direct, out_md, weight_md, bias_md, data_md, strides, dilate, padding, padding, mkldnn::padding_kind::zero); - return mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + auto deconv_pd = mkldnn::convolution_backward_weights::primitive_desc(desc, engine, fwd_pd); + while (deconv_pd.diff_dst_primitive_desc().get_size() != GetMemDescSize(data_md) || + deconv_pd.src_primitive_desc().get_size() != GetMemDescSize(out_md) || + deconv_pd.diff_weights_primitive_desc().get_size() != GetMemDescSize(weight_md)) { + CHECK(deconv_pd.next_impl()) << "No implementation"; + } + return deconv_pd; } } @@ -166,7 +212,8 @@ class MKLDNNDeconvForward { const NDArray &output); void SetDataHandle(const DeconvolutionParam& param, const OpContext &ctx, - const std::vector &in_data, + const NDArray &in_data, + const NDArray &weight, const std::vector &req, const std::vector &out_data); @@ -197,19 +244,19 @@ MKLDNNDeconvForward::MKLDNNDeconvForward(const DeconvolutionParam& param, void MKLDNNDeconvForward::SetDataHandle(const DeconvolutionParam& param, const OpContext &ctx, - const std::vector &in_data, + const NDArray &in_data, + const NDArray &weight, const std::vector &req, const std::vector &out_data) { - auto data_mem = in_data[deconv::kData].GetMKLDNNDataReorder( + auto data_mem = in_data.GetMKLDNNDataReorder( fwd_pd.diff_dst_primitive_desc()); - NDArray weight = in_data[deconv::kWeight]; const mkldnn::memory *weight_mem; if (ctx.is_train) { // TODO(zhengda) kvstore doesn't handle MKLDNN correctly. Let's reorder it // to the default format for now. if (weight.IsMKLDNNData()) // This asks the engine to reorder data after the weight array is used. - weight.Reorder2DefaultAsync(); + const_cast(weight).Reorder2DefaultAsync(); weight_mem = GetWeights(weight, fwd_pd.weights_primitive_desc(), param.num_group); } else { // For inference, we want to reorder the weight array so we don't need to @@ -218,7 +265,7 @@ void MKLDNNDeconvForward::SetDataHandle(const DeconvolutionParam& param, weight_mem = GetWeights(weight, fwd_pd.weights_primitive_desc(), param.num_group); // We also need to modify the layout on the original weight array. The // data conversion happens after the weight array is used. - weight.MKLDNNDataReorderAsync(fwd_pd.weights_primitive_desc()); + const_cast(weight).MKLDNNDataReorderAsync(fwd_pd.weights_primitive_desc()); } else { weight_mem = weight.GetMKLDNNData(); CHECK(weight_mem->get_primitive_desc() == fwd_pd.weights_primitive_desc()); @@ -241,19 +288,19 @@ void MKLDNNDeconvForward::Execute(const std::vector &out_data) { static void MKLDNNDeconvFwdBiasPostProcess(const DeconvolutionParam& param, const OpContext &ctx, - const std::vector &in_data, + const NDArray &bias, const std::vector &out_data) { // add bias, broadcast bias to dim 1: channel if (!param.no_bias) { // MKLDNN only supports float right now. typedef float DType; Stream *s = ctx.get_stream(); - Tensor bias = in_data[deconv::kBias].data().get(s); + Tensor b = bias.data().get(s); // If the output data is stored in a special MKLDNN format, data() // automatically converts its format to the default format. // Unfortunately, MKLDNN doesn't support broadcast. Tensor out_cpu = out_data[deconv::kOut].data().get(s); - out_cpu += mshadow::expr::broadcast<1>(bias, out_cpu.shape_); + out_cpu += mshadow::expr::broadcast<1>(b, out_cpu.shape_); } } @@ -283,10 +330,7 @@ static inline MKLDNNDeconvForward &GetDeconvFwd( if (it == fwds.end()) { bool has_bias = (bias != nullptr); MKLDNNDeconvForward fwd(param, data, weights, has_bias, output); - auto ins_ret = fwds.insert( - std::pair(key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&fwds, key, fwd); } return it->second; } @@ -298,15 +342,24 @@ void MKLDNNDeconvolutionForward(const nnvm::NodeAttrs& attrs, const OpContext &c TmpMemMgr::Get()->Init(ctx.requested[deconv::kTempSpace]); const DeconvolutionParam& param = nnvm::get(attrs.parsed); + auto data = in_data[deconv::kData]; + if (data.IsView() && data.IsMKLDNNData()) + data = data.Reorder2Default(); + + auto weight = in_data[deconv::kWeight]; + if (weight.IsView() && weight.IsMKLDNNData()) + weight = weight.Reorder2Default(); + + const NDArray* bias = param.no_bias ? nullptr : &in_data[deconv::kBias]; + MKLDNNDeconvForward &deconvFwd = GetDeconvFwd( - attrs, in_data[deconv::kData], in_data[deconv::kWeight], - param.no_bias ? nullptr : &in_data[deconv::kBias], out_data[deconv::kOut]); + attrs, data, weight, bias, out_data[deconv::kOut]); - deconvFwd.SetDataHandle(param, ctx, in_data, req, out_data); + deconvFwd.SetDataHandle(param, ctx, data, weight, req, out_data); deconvFwd.Execute(out_data); - MKLDNNDeconvFwdBiasPostProcess(param, ctx, in_data, out_data); + MKLDNNDeconvFwdBiasPostProcess(param, ctx, *bias, out_data); } class MKLDNNDeconvBackwardData { @@ -371,10 +424,7 @@ static inline MKLDNNDeconvBackwardData &GetDeconvBwdData( auto it = bwds.find(key); if (it == bwds.end()) { MKLDNNDeconvBackwardData bwd(param, data, weights, output); - auto ins_ret = bwds.insert( - std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&bwds, key, bwd); } return it->second; } @@ -460,17 +510,24 @@ void MKLDNNDeconvolutionBackward(const nnvm::NodeAttrs &attrs, TmpMemMgr::Get()->Init(ctx.requested[deconv::kTempSpace]); const std::vector &in_grad = outputs; const DeconvolutionParam ¶m = nnvm::get(attrs.parsed); + + auto data = inputs[deconv::kData + 1]; + if (data.IsView() && data.IsMKLDNNData()) + data = data.Reorder2Default(); + + auto weight = inputs[deconv::kWeight + 1]; + if (weight.IsView() && weight.IsMKLDNNData()) + weight = weight.Reorder2Default(); + CHECK_NE(req[deconv::kWeight], kWriteInplace) << "cannot write weight inplace"; MKLDNNDeconvBackwardData &bwd_data = - GetDeconvBwdData(param, inputs[deconv::kData + 1], - inputs[deconv::kWeight + 1], inputs[deconv::kOut]); + GetDeconvBwdData(param, data, weight, inputs[deconv::kOut]); auto out_grad_mem = inputs[deconv::kOut].GetMKLDNNDataReorder( bwd_data.pd.src_primitive_desc()); if (req[deconv::kData]) { auto weight_mem = - GetWeights(inputs[deconv::kWeight + 1], - bwd_data.pd.weights_primitive_desc(), param.num_group); + GetWeights(weight, bwd_data.pd.weights_primitive_desc(), param.num_group); auto in_grad_mem = CreateMKLDNNMem(in_grad[deconv::kData], bwd_data.pd.dst_primitive_desc(), req[deconv::kData]); @@ -480,12 +537,12 @@ void MKLDNNDeconvolutionBackward(const nnvm::NodeAttrs &attrs, } if (req[deconv::kWeight]) { MKLDNNDeconvBackwardWeights &bwd_weights = GetDeconvBwdWeights( - param, inputs[deconv::kData + 1], inputs[deconv::kWeight + 1], + param, data, weight, inputs[deconv::kOut], bwd_data.pd); if (bwd_data.pd.src_primitive_desc() != bwd_weights.pd.src_primitive_desc()) out_grad_mem = inputs[deconv::kOut].GetMKLDNNDataReorder( bwd_weights.pd.src_primitive_desc()); - auto data_mem = inputs[deconv::kData + 1].GetMKLDNNDataReorder( + auto data_mem = data.GetMKLDNNDataReorder( bwd_weights.pd.diff_dst_primitive_desc()); auto in_grad_weight = CreateMKLDNNWeightGrad( in_grad[deconv::kWeight], bwd_weights.pd.diff_weights_primitive_desc(), diff --git a/src/operator/nn/mkldnn/mkldnn_lrn-inl.h b/src/operator/nn/mkldnn/mkldnn_lrn-inl.h index bc386bedd..31b293a14 100644 --- a/src/operator/nn/mkldnn/mkldnn_lrn-inl.h +++ b/src/operator/nn/mkldnn/mkldnn_lrn-inl.h @@ -189,10 +189,7 @@ static MKLDNNLRNFwd &GetLRNFwd(const LRNParam& param, auto it = lrn_fwds.find(key); if (it == lrn_fwds.end()) { MKLDNNLRNFwd fwd(param, ctx.is_train, in_data); - auto ins_ret = lrn_fwds.insert(std::pair - (key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&lrn_fwds, key, fwd); } return it->second; } @@ -284,10 +281,7 @@ static MKLDNNLRNBwd &GetLRNBwd(const LRNParam ¶m, const NDArray &in_data, const mkldnn::memory::desc diff_md = out_grad.GetMKLDNNData()->get_primitive_desc().desc(); MKLDNNLRNBwd bwd(param, in_data_md, diff_md); - auto ins_ret = - lrn_bwds.insert(std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&lrn_bwds, key, bwd); } return it->second; } diff --git a/src/operator/nn/mkldnn/mkldnn_pooling-inl.h b/src/operator/nn/mkldnn/mkldnn_pooling-inl.h index 66679613d..f548778c7 100644 --- a/src/operator/nn/mkldnn/mkldnn_pooling-inl.h +++ b/src/operator/nn/mkldnn/mkldnn_pooling-inl.h @@ -113,20 +113,12 @@ inline bool SupportMKLDNNPooling(const PoolingParam ¶m, if (!ret) return false; - if (param.pooling_convention == pool_enum::kValid) + if (param.pooling_convention == pool_enum::kValid) { return true; - else - return false; - -// need to support pooling convention full -// https://issues.apache.org/jira/browse/MXNET-33 -#if 0 - if (((dshape[2] + 2 * param.pad[0] - param.kernel[0]) % param.stride[0] == 0) && - ((dshape[3] + 2 * param.pad[1] - param.kernel[1]) % param.stride[1] == 0)) - return true; - else - return false; -#endif + } else { + // currently, only max-pooling is supported for full convention + return param.pool_type == pool_enum::kMaxPooling; + } } inline bool MKLDNNRequireWorkspace(const PoolingParam ¶m) { diff --git a/src/operator/nn/mkldnn/mkldnn_pooling.cc b/src/operator/nn/mkldnn/mkldnn_pooling.cc index 161094430..f4d681ded 100644 --- a/src/operator/nn/mkldnn/mkldnn_pooling.cc +++ b/src/operator/nn/mkldnn/mkldnn_pooling.cc @@ -134,6 +134,14 @@ mkldnn::algorithm GetMKLDNNPoolAlgo(const PoolingParam ¶m) { } } +static inline int GetPaddingSizeFull(int x, int padl, int padr, int k, int s) { + if ((x + padl + padr - k) % s != 0) { + return (padr + s - ((x + padl + padr - k) % s)); + } else { + return padr; + } +} + mkldnn::pooling_forward::primitive_desc GetPoolingFwdPdesc( const PoolingParam ¶m, const bool is_train, const memory::desc &data_md, const memory::desc &out_md) { @@ -154,11 +162,17 @@ mkldnn::pooling_forward::primitive_desc GetPoolingFwdPdesc( int pad_l_ = param.pad[1], pad_r_ = param.pad[1]; int stride_h_ = param.stride[0], stride_w_ = param.stride[1]; + if (param.pooling_convention == pool_enum::kFull) { + pad_b_ = GetPaddingSizeFull(data_md.data.dims[2], pad_t_, pad_b_, kernel_h_, stride_h_); + pad_r_ = GetPaddingSizeFull(data_md.data.dims[3], pad_l_, pad_r_, kernel_w_, stride_w_); + } + const mkldnn::engine engine = CpuEngine::Get()->get_engine(); if (param.global_pool) { pad_t_ = pad_b_ = pad_l_ = pad_r_ = 0; stride_h_ = stride_w_ = 1; } + if (pad_t_ != 0 || pad_l_ != 0) { CHECK(param.pool_type == pool_enum::kAvgPooling || param.pool_type == pool_enum::kMaxPooling) @@ -167,7 +181,6 @@ mkldnn::pooling_forward::primitive_desc GetPoolingFwdPdesc( CHECK_LT(pad_t_, kernel_h_); } - const mkldnn::algorithm alg = GetMKLDNNPoolAlgo(param); mkldnn::prop_kind kind = mkldnn::prop_kind::forward_scoring; if (is_train && alg != algorithm::pooling_avg) { @@ -227,26 +240,28 @@ MKLDNNPoolingFwd &GetPoolingFwd(const PoolingParam ¶m, int pad_l_ = param.pad[1], pad_r_ = param.pad[1]; int stride_h_ = param.stride[0], stride_w_ = param.stride[1]; + if (param.pooling_convention == pool_enum::kFull) { + pad_b_ = GetPaddingSizeFull(data_md.data.dims[2], pad_t_, pad_b_, kernel_h_, stride_h_); + pad_r_ = GetPaddingSizeFull(data_md.data.dims[3], pad_l_, pad_r_, kernel_w_, stride_w_); + } + if (param.global_pool) { - pad_t_ = pad_b_ = pad_l_ = pad_r_ = 0; - stride_h_ = stride_w_ = 1; + pad_t_ = pad_b_ = pad_l_ = pad_r_ = 0; + stride_h_ = stride_w_ = 1; } if (pad_t_ != 0 || pad_l_ != 0) { - CHECK(param.pool_type == pool_enum::kAvgPooling || - param.pool_type == pool_enum::kMaxPooling) - << "Padding implemented only for average and max pooling."; - CHECK_LT(pad_l_, kernel_w_); - CHECK_LT(pad_t_, kernel_h_); + CHECK(param.pool_type == pool_enum::kAvgPooling || + param.pool_type == pool_enum::kMaxPooling) + << "Padding implemented only for average and max pooling."; + CHECK_LT(pad_l_, kernel_w_); + CHECK_LT(pad_t_, kernel_h_); } const mkldnn::algorithm alg = GetMKLDNNPoolAlgo(param); MKLDNNPoolingFwd fwd(data, output, kernel_h_, kernel_w_, stride_h_, stride_w_, pad_t_, pad_b_, pad_l_, pad_r_, alg, with_workspace, is_train); - auto ins_ret = pooling_fwds.insert( - std::pair(key, fwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&pooling_fwds, key, fwd); } return it->second; } @@ -353,6 +368,12 @@ MKLDNNPoolingBwd &GetPoolingBwd(const PoolingParam ¶m, int pad_t_ = param.pad[0], pad_b_ = param.pad[0]; int pad_l_ = param.pad[1], pad_r_ = param.pad[1]; int stride_h_ = param.stride[0], stride_w_ = param.stride[1]; + + if (param.pooling_convention == pool_enum::kFull) { + pad_b_ = GetPaddingSizeFull(data_md.data.dims[2], pad_t_, pad_b_, kernel_h_, stride_h_); + pad_r_ = GetPaddingSizeFull(data_md.data.dims[3], pad_l_, pad_r_, kernel_w_, stride_w_); + } + if (param.global_pool) { pad_t_ = pad_b_ = pad_l_ = pad_r_ = 0; stride_h_ = stride_w_ = 1; @@ -364,10 +385,7 @@ MKLDNNPoolingBwd &GetPoolingBwd(const PoolingParam ¶m, mkldnn::padding_kind::zero); const auto pdesc = pooling_backward::primitive_desc(desc, cpu_engine, fwd_pd); MKLDNNPoolingBwd bwd(pdesc, with_workspace); - auto ins_ret = pooling_bwds.insert( - std::pair(key, bwd)); - CHECK(ins_ret.second); - it = ins_ret.first; + it = AddToCache(&pooling_bwds, key, bwd); } return it->second; } diff --git a/src/operator/operator_common.h b/src/operator/operator_common.h index d7c141724..b1822647c 100644 --- a/src/operator/operator_common.h +++ b/src/operator/operator_common.h @@ -337,8 +337,8 @@ inline bool storage_type_assign(StorageTypeVector* stypes, const DispatchMode target_dispatch) { CHECK_GT(stypes->size(), 0); bool success = true; - for (size_t i = 0; i < stypes->size(); i++) { - if (!type_assign(&(*stypes)[i], target_stype)) { + for (int& stype : *stypes) { + if (!type_assign(&stype, target_stype)) { success = false; } } diff --git a/src/operator/operator_tune-inl.h b/src/operator/operator_tune-inl.h index 127691bcc..1dbcf4298 100644 --- a/src/operator/operator_tune-inl.h +++ b/src/operator/operator_tune-inl.h @@ -56,7 +56,7 @@ namespace op { #endif #endif // MXNET_NO_INLINE -#define OUTSIDE_COUNT_SHIFT 9 +#define OUTSIDE_COUNT_SHIFT 3 namespace tune { @@ -356,7 +356,8 @@ class OperatorTune : public OperatorTuneByType { static duration_t GetOMPLoopOverhead() { // It was found empirically that OMP times was not heavily tied to number of cores, // so take an average across all core counts - const auto max_cores = static_cast(omp_get_num_procs()) >> 1; + const auto max_cores_default = static_cast(omp_get_num_procs()) >> 1; + const auto max_cores = dmlc::GetEnv("MXNET_USE_NUM_CORES_OPERATOR_TUNING", max_cores_default); if (max_cores >= 2) { std::vector core_times; // Take care of any OMP lazy-init with a throwaway call diff --git a/src/operator/operator_tune.cc b/src/operator/operator_tune.cc index cf5412f98..2018e80cb 100644 --- a/src/operator/operator_tune.cc +++ b/src/operator/operator_tune.cc @@ -235,6 +235,8 @@ IMPLEMENT_UNARY_WORKLOAD_BWD(mxnet::op::mshadow_op::log2_grad); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_FWD(mxnet::op::mshadow_op::log10); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_BWD(mxnet::op::mshadow_op::log10_grad); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_FWD(mxnet::op::mshadow_op::sin); // NOLINT() +IMPLEMENT_UNARY_WORKLOAD_FWD(mxnet::op::mshadow_op::erf); // NOLINT() +IMPLEMENT_UNARY_WORKLOAD_BWD(mxnet::op::mshadow_op::erf_grad); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_BWD(mxnet::op::mshadow_op::sin_grad); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_FWD(mxnet::op::mshadow_op::sinh); // NOLINT() IMPLEMENT_UNARY_WORKLOAD_BWD(mxnet::op::mshadow_op::sinh_grad); // NOLINT() diff --git a/src/operator/operator_util.cc b/src/operator/operator_util.cc index 326a1ca38..0c6f176a0 100644 --- a/src/operator/operator_util.cc +++ b/src/operator/operator_util.cc @@ -407,13 +407,13 @@ class SimpleOpPropBase : public OperatorProperty { CHECK_LE(in_type->size(), this->ListArguments().size()); int dtype = -1; // reduce dtype to a common one. - for (unsigned i = 0; i < in_type->size(); ++i) { + for (int i : *in_type) { if (dtype == -1) { - dtype = in_type->at(i); + dtype = i; } else { - CHECK(in_type->at(i) == -1 || - in_type->at(i) == dtype) << - "Non-uniform input data type. Expected " << dtype << "got " << in_type->at(i); + CHECK(i == -1 || + i == dtype) << + "Non-uniform input data type. Expected " << dtype << "got " << i; } } diff --git a/src/operator/quantization/mkldnn/mkldnn_quantized_concat.cc b/src/operator/quantization/mkldnn/mkldnn_quantized_concat.cc new file mode 100644 index 000000000..d9e884e82 --- /dev/null +++ b/src/operator/quantization/mkldnn/mkldnn_quantized_concat.cc @@ -0,0 +1,119 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2018 by Contributors + * \file quantized_concat.cc + * \brief + */ + +#if MXNET_USE_MKLDNN == 1 +#include "../../nn/mkldnn/mkldnn_concat-inl.h" +#include "../quantization_utils.h" + +namespace mxnet { +namespace op { + +namespace quantized_concat_enum { +enum QuantizedConcatOutputs { kOut, kMin, kMax }; +} + +static float GetScale(const NDArray& data, float min, float max) { + auto data_range = (data.dtype() == mshadow::kInt8) ? kInt8Range : kUint8Range; + return data_range / MaxAbs(min, max); +} + +static void MKLDNNQuantizedConcatForward(const nnvm::NodeAttrs& attrs, const OpContext& ctx, + const std::vector& in_data, + const std::vector& req, + const std::vector& out_data) { + const ConcatParam& param_ = nnvm::get(attrs.parsed); + CHECK_EQ(in_data.size(), static_cast(param_.num_args * 3)); + CHECK_EQ(out_data.size(), 3U); + // Collect data min/max and output_neg_min, output_pos_max + std::vector data_min(param_.num_args); + std::vector data_max(param_.num_args); + float output_neg_min = 0.f; // 0.f is the maximum for output_neg_min + float output_pos_max = 0.f; // 0.f is the minimum for output_pos_max + for (int i = 0; i < param_.num_args; ++i) { + data_min[i] = in_data[param_.num_args + 2 * i].data().dptr()[0]; + if (data_min[i] < output_neg_min) output_neg_min = data_min[i]; + data_max[i] = in_data[param_.num_args + 2 * i + 1].data().dptr()[0]; + if (data_max[i] > output_pos_max) output_pos_max = data_max[i]; + } + out_data[quantized_concat_enum::kMin].data().dptr()[0] = output_neg_min; + out_data[quantized_concat_enum::kMax].data().dptr()[0] = output_pos_max; + auto out_scale = GetScale(out_data[quantized_concat_enum::kOut], output_neg_min, output_pos_max); + std::vector data_md; + std::vector data_mem; + // new_data_mem is for auto-free new created mkldnn memory + std::vector> new_data_mem; + for (int i = 0; i < param_.num_args; ++i) { + auto i_scale = GetScale(in_data[i], data_min[i], data_max[i]); + if (i_scale == out_scale) { + auto mem = in_data[i].GetMKLDNNData(); + data_mem.push_back(mem); + data_md.push_back(mem->get_primitive_desc()); + } else { + auto mem = in_data[i].GetMKLDNNData(); + auto pd = mem->get_primitive_desc(); + const auto rescaled_mem = std::make_shared(pd); + new_data_mem.push_back(rescaled_mem); + std::vector reorder_scale = {out_scale / i_scale}; + primitive_attr reorder_attr; + reorder_attr.set_int_output_round_mode(round_mode::round_nearest); + reorder_attr.set_output_scales(0, reorder_scale); + const auto reorder_pd = mkldnn::reorder::primitive_desc(pd, pd, reorder_attr); + MKLDNNStream::Get()->RegisterPrim(mkldnn::reorder(reorder_pd, *mem, *rescaled_mem)); + data_mem.push_back(rescaled_mem.get()); + data_md.push_back(pd); + } + } + MKLDNNConcatFwd& fwd = GetConcatForward(param_.dim, in_data, data_md); + mxnet::mkldnn_output_t out_mem = + CreateMKLDNNMem(out_data[quantized_concat_enum::kOut], fwd.fwd_pd.dst_primitive_desc(), + req[concat_enum::kOut]); + fwd.SetNewMem(data_mem, *out_mem.second); + MKLDNNStream::Get()->RegisterPrim(fwd.GetFwd()); + CommitOutput(out_data[concat_enum::kOut], out_mem); + MKLDNNStream::Get()->Submit(); +} + +inline static bool ConcatStorageType(const nnvm::NodeAttrs& attrs, const int dev_mask, + DispatchMode* dispatch_mode, std::vector* in_attrs, + std::vector* out_attrs) { + const ConcatParam& param_ = nnvm::get(attrs.parsed); + CHECK_EQ(in_attrs->size(), static_cast(param_.num_args * 3)); + CHECK_EQ(out_attrs->size(), 3U); + + return MKLDNNStorageType(attrs, dev_mask, true, dispatch_mode, in_attrs, out_attrs); +} + +NNVM_REGISTER_OP(_contrib_quantized_concat) +.set_attr("FInferStorageType", ConcatStorageType) +.set_attr("FComputeEx", MKLDNNQuantizedConcatForward) +.set_attr("FResourceRequest", [](const NodeAttrs& n) { + return std::vector{ResourceRequest::kTempSpace}; +}) +.set_attr("TIsMKLDNN", true); + +} // namespace op +} // namespace mxnet + +#endif // MXNET_USE_MKLDNN == 1 diff --git a/src/operator/quantization/quantization_utils.h b/src/operator/quantization/quantization_utils.h index 5b096ac00..ee7112205 100644 --- a/src/operator/quantization/quantization_utils.h +++ b/src/operator/quantization/quantization_utils.h @@ -31,6 +31,8 @@ namespace mxnet { namespace op { +static const size_t kUint8Range = 255; +static const size_t kInt8Range = 127; template MSHADOW_XINLINE int Sign(T val) { diff --git a/src/operator/quantization/quantize_graph_pass.cc b/src/operator/quantization/quantize_graph_pass.cc index 2fa790dc8..fcd0fb421 100644 --- a/src/operator/quantization/quantize_graph_pass.cc +++ b/src/operator/quantization/quantize_graph_pass.cc @@ -68,8 +68,7 @@ std::vector OfflineParams(std::vector&& outputs, std::unordered_map mirror_map; nnvm::NodeEntryMap entry_var; auto need_offline = [&](NodePtr n) { - return n->op() && - (n->op()->name == "_contrib_quantize") && + return (n->op() == Op::Get("_contrib_quantize")) && n->inputs[0].node->is_variable() && offline_params.count(n->inputs[0].node->attrs.name); }; @@ -117,9 +116,10 @@ inline bool NeedQuantize(NodePtr node, const std::unordered_set& ex } Graph QuantizeGraph(Graph &&src) { - static auto& quantized_op_map = Op::GetAttr("FQuantizedOp"); - static auto& need_requantize_map = Op::GetAttr("FNeedRequantize"); - static auto& avoid_quantize_input_map = + static const auto& flist_outputs = nnvm::Op::GetAttr("FListOutputNames"); + static const auto& quantized_op_map = Op::GetAttr("FQuantizedOp"); + static const auto& need_requantize_map = Op::GetAttr("FNeedRequantize"); + static const auto& avoid_quantize_input_map = Op::GetAttr("FAvoidQuantizeInput"); auto offline_params = src.GetAttr>("offline_params"); auto excluded_nodes = src.GetAttr>("excluded_nodes"); @@ -130,6 +130,7 @@ Graph QuantizeGraph(Graph &&src) { // graph. Key is the currently visited graph's node pointer, and value is a copied node of the key // node. The existing key's value may be updated with the newly created quantize/dequantize op. std::unordered_map mirror_map; + nnvm::NodeEntryMap mirror_entry_map; DFSVisit(src.outputs, [&](const NodePtr& node) { NodePtr new_node = Node::Create(); // If the currently visited node needs quantization, insert a quantize op node before the @@ -154,30 +155,46 @@ Graph QuantizeGraph(Graph &&src) { if (avoid_quantize_input_map.count(node->op()) && avoid_quantize_input_map[node->op()](node->attrs, i)) { new_node->inputs.emplace_back(mirror_entry); - } else if (!NeedQuantize(e.node, excluded_nodes) && - (mirror_node->op() == nullptr || - mirror_node->op()->name != "_contrib_quantize")) { - NodePtr quantize_node = InsertNode("_contrib_quantize", - e.node->attrs.name + "_quantize", new_node, mirror_entry); - quantize_node->attrs.dict["out_type"] = quantized_dtype; - quantize_node->op()->attr_parser(&(quantize_node->attrs)); - if (calib_quantize) { - NodePtr min_var = CreateNode("nullptr", e.node->attrs.name + "_min"); - quantize_node->inputs.emplace_back(NodeEntry{min_var, 0, 0}); - NodePtr max_var = CreateNode("nullptr", e.node->attrs.name + "_max"); - quantize_node->inputs.emplace_back(NodeEntry{max_var, 0, 0}); + } else if (!NeedQuantize(e.node, excluded_nodes)) { + if (mirror_entry_map.count(e)) { + new_node->inputs.emplace_back(mirror_entry_map[e]); } else { - NodePtr min_node = InsertNode("min", - e.node->attrs.name + "_min", quantize_node, mirror_entry); - min_node->op()->attr_parser(&(min_node->attrs)); + // When there're multiple entrys outgoing from a single node, need to add entry + // index (or output name) into quantize/min/max node to distinguish them. + // Or the output name is not ending with 'output', just put the output name here + // to better align with calibration phase. No need to change name to weights/bias. + std::string suffix = ""; + if (mirror_node->op() != nullptr) { + auto list_output_names_func = flist_outputs.get(e.node->op(), nullptr); + if (list_output_names_func != nullptr) { + std::vector names = list_output_names_func(e.node->attrs); + suffix = "_" + names[e.index]; + } else { + suffix = "_" + std::to_string(e.index); + } + } + + NodePtr quantize_node = InsertNode("_contrib_quantize", + e.node->attrs.name + suffix + "_quantize", new_node, mirror_entry); + quantize_node->attrs.dict["out_type"] = quantized_dtype; + quantize_node->op()->attr_parser(&(quantize_node->attrs)); + if (calib_quantize) { + NodePtr min_var = CreateNode("nullptr", e.node->attrs.name + suffix + "_min"); + quantize_node->inputs.emplace_back(NodeEntry{min_var, 0, 0}); + NodePtr max_var = CreateNode("nullptr", e.node->attrs.name + suffix + "_max"); + quantize_node->inputs.emplace_back(NodeEntry{max_var, 0, 0}); + } else { + NodePtr min_node = InsertNode("min", + e.node->attrs.name + suffix + "_min", quantize_node, mirror_entry); + min_node->op()->attr_parser(&(min_node->attrs)); - NodePtr max_node = InsertNode("max", - e.node->attrs.name + "_max", quantize_node, mirror_entry); - max_node->op()->attr_parser(&(max_node->attrs)); + NodePtr max_node = InsertNode("max", + e.node->attrs.name + suffix + "_max", quantize_node, mirror_entry); + max_node->op()->attr_parser(&(max_node->attrs)); + } + mirror_entry_map[e] = NodeEntry{quantize_node, 0, e.version}; } - mirror_map[e.node.get()] = std::move(quantize_node); - } else if (mirror_node->op() != nullptr - && mirror_node->op()->name == "_contrib_dequantize") { + } else if (mirror_node->op() == Op::Get("_contrib_dequantize")) { new_node->inputs.emplace_back(NodeEntry{mirror_node->inputs[0].node, e.index, e.version}); } else { // If the entry e's node needs quantization, or mirror_entry is from a quantize op, @@ -192,8 +209,7 @@ Graph QuantizeGraph(Graph &&src) { for (size_t i = 0; i < node->inputs.size(); ++i) { const auto& e = node->inputs[i]; NodePtr mirror_node = mirror_map.at(e.node.get()); - if (mirror_node->op() != nullptr - && mirror_node->op()->name == "_contrib_dequantize") { + if (mirror_node->op() == Op::Get("_contrib_dequantize")) { mirror_node = mirror_node->inputs[0].node; } NodeEntry mirror_entry = NodeEntry{ @@ -206,7 +222,7 @@ Graph QuantizeGraph(Graph &&src) { // skip non-quantized input continue; } - if (quantized_op_map.count(e.node->op())) { + if (NeedQuantize(e.node, excluded_nodes)) { // here we calculate the output number (exclude min/max, in order to // calculate min/max index from mirror node) based on assumption that // there is only 1min and 1max output from mirror node (which is @@ -215,12 +231,17 @@ Graph QuantizeGraph(Graph &&src) { min_index = num_outputs + 2 * e.index; max_index = num_outputs + 2 * e.index + 1; } else { - CHECK(mirror_node->op() != nullptr && - mirror_node->op()->name == "_contrib_quantize") + CHECK(mirror_entry_map.count(e)) << "The input is not quantize or quantized_op"; } - new_node->inputs.emplace_back(NodeEntry{mirror_node, min_index, 0}); - new_node->inputs.emplace_back(NodeEntry{mirror_node, max_index, 0}); + if (mirror_entry_map.count(e)) { + auto quantize_entry = mirror_entry_map[e]; + new_node->inputs.emplace_back(NodeEntry{quantize_entry.node, min_index, 0}); + new_node->inputs.emplace_back(NodeEntry{quantize_entry.node, max_index, 0}); + } else { + new_node->inputs.emplace_back(NodeEntry{mirror_node, min_index, 0}); + new_node->inputs.emplace_back(NodeEntry{mirror_node, max_index, 0}); + } } // If the new_node op registered attr FNeedRequantize, insert requantize node after it. @@ -261,8 +282,7 @@ Graph QuantizeGraph(Graph &&src) { mirror_node, e.index, e.version}; // if input node is quantized operator, add dequantize node if (NeedQuantize(e.node, excluded_nodes) && - (mirror_node->op() == nullptr || - mirror_node->op()->name != "_contrib_dequantize")) { + (mirror_node->op() != Op::Get("_contrib_dequantize"))) { // here we calculate the output number (exclude min/max, in order to // calculate min/max index from mirror node) based on assumption that // there is only 1min and 1max output from mirror node (which is @@ -279,10 +299,9 @@ Graph QuantizeGraph(Graph &&src) { new_node->inputs.emplace_back(NodeEntry{dequantize_node, 0, 0}); mirror_map[e.node.get()] = std::move(dequantize_node); - } else if (mirror_node->op() != nullptr - && mirror_node->op()->name == "_contrib_quantize") { + } else if (mirror_entry_map.count(e)) { new_node->inputs.emplace_back( - NodeEntry{mirror_node->inputs[0].node, e.index, e.version}); + NodeEntry{mirror_entry_map[e].node->inputs[0].node, e.index, e.version}); } else { new_node->inputs.emplace_back( NodeEntry{mirror_node, e.index, e.version}); @@ -295,7 +314,8 @@ Graph QuantizeGraph(Graph &&src) { std::vector outputs; for (const auto& e : src.outputs) { - if (quantized_op_map.count(e.node->op())) { + if (NeedQuantize(e.node, excluded_nodes)) { + // Only insert dequantize for those Ops supports quantize and not excluded. NodePtr mirror_node = mirror_map.at(e.node.get()); NodeEntry mirror_entry = NodeEntry{mirror_node, e.index, e.version}; size_t num_inputs = e.node->num_inputs(); @@ -333,7 +353,7 @@ Graph SetCalibTableToQuantizedGraph(Graph&& g) { // If the current op is requantize // find the thresholds from the calibration table with the key equal // to the current op's input node name, e.g. a quantized_conv2d node. - if (node->op() != nullptr && node->op()->name == "_contrib_requantize") { + if (node->op() == Op::Get("_contrib_requantize")) { NodePtr quantized_op_node = node->inputs[0].node; CHECK(quantized_op_node->op() != nullptr) << quantized_op_node->attrs.name << " must be an quantized op node"; diff --git a/src/operator/quantization/quantized_concat.cc b/src/operator/quantization/quantized_concat.cc new file mode 100644 index 000000000..3504df82d --- /dev/null +++ b/src/operator/quantization/quantized_concat.cc @@ -0,0 +1,149 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * Copyright (c) 2018 by Contributors + * \file quantized_concat.cc + * \brief +*/ + +#include "../nn/concat-inl.h" + +namespace mxnet { +namespace op { + +static bool ConcatShape(const nnvm::NodeAttrs& attrs, std::vector* in_shape, + std::vector* out_shape) { + const ConcatParam& param_ = nnvm::get(attrs.parsed); + CHECK_EQ(in_shape->size(), static_cast(param_.num_args * 3)); + CHECK_EQ(out_shape->size(), 3U); + TShape dshape; + index_t size = 0; + bool has_zero = false; + int axis = -1; + for (int i = 0; i < param_.num_args; ++i) { + TShape tmp = (*in_shape)[i]; + if (tmp.ndim()) { + axis = CheckAxis(param_.dim, tmp.ndim()); + has_zero = tmp[axis] == 0 || has_zero; + size += tmp[axis]; + tmp[axis] = 0; + shape_assign(&dshape, tmp); + } + } + + TShape tmp = (*out_shape)[0]; + if (tmp.ndim()) { + axis = CheckAxis(param_.dim, tmp.ndim()); + tmp[axis] = 0; + shape_assign(&dshape, tmp); + } + + if (dshape.ndim() == 0) return false; + + for (int i = 0; i < param_.num_args; ++i) { + CHECK(shape_assign(&(*in_shape)[i], dshape)) + << "Incompatible input shape: expected " << dshape << ", got " << (*in_shape)[i]; + } + + if (!has_zero) dshape[axis] = size; + CHECK(shape_assign(&(*out_shape)[0], dshape)) + << "Incompatible output shape: expected " << dshape << ", got " << (*out_shape)[0]; + + for (int i = param_.num_args; i < param_.num_args * 3; ++i) { + SHAPE_ASSIGN_CHECK(*in_shape, i, TShape{1}); + } + SHAPE_ASSIGN_CHECK(*out_shape, 1, TShape{1}); + SHAPE_ASSIGN_CHECK(*out_shape, 2, TShape{1}); + return dshape.Size() != 0; +} + +static bool ConcatType(const nnvm::NodeAttrs& attrs, std::vector* in_type, + std::vector* out_type) { + const ConcatParam& param_ = nnvm::get(attrs.parsed); + CHECK_EQ(in_type->size(), static_cast(param_.num_args * 3)); + CHECK_EQ(out_type->size(), 3U); + int dtype = mshadow::kUint8; + + for (int i = 0; i < param_.num_args; ++i) { + if (in_type->at(i) == mshadow::kInt8) { + dtype = mshadow::kInt8; + } else { + TYPE_ASSIGN_CHECK(*in_type, i, mshadow::kUint8); + } + } + TYPE_ASSIGN_CHECK(*out_type, 0, dtype); + TYPE_ASSIGN_CHECK(*out_type, 1, mshadow::kFloat32); + TYPE_ASSIGN_CHECK(*out_type, 2, mshadow::kFloat32); + + return true; +} + +NNVM_REGISTER_OP(_contrib_quantized_concat) +.describe(R"code(Joins input arrays along a given axis. + +The dimensions of the input arrays should be the same except the axis along +which they will be concatenated. +The dimension of the output array along the concatenated axis will be equal +to the sum of the corresponding dimensions of the input arrays. +All inputs with different min/max will be rescaled by using largest [min, max] pairs. +If any input holds int8, then the output will be int8. Otherwise output will be uint8. + +)code" ADD_FILELINE) +.set_num_inputs([](const NodeAttrs& attrs) { + const ConcatParam& params = nnvm::get(attrs.parsed); + return params.num_args * 3; +}) +.set_num_outputs(3) +.set_attr_parser(ParamParser) +.set_attr("FListInputNames", [](const NodeAttrs& attrs) { + const ConcatParam& params = nnvm::get(attrs.parsed); + std::vector ret; + for (int i = 0; i < params.num_args; ++i) { + ret.push_back(std::string("arg") + std::to_string(i)); + } + for (int i = 0; i < params.num_args; ++i) { + ret.push_back(std::string("arg") + std::to_string(i) + "_min"); + ret.push_back(std::string("arg") + std::to_string(i) + "_max"); + } + return ret; +}) +.set_attr("FListOutputNames", [](const NodeAttrs& attrs) { + return std::vector{"output", "min_output", "max_output"}; +}) +.set_attr("FInferType", ConcatType) +.set_attr("FInferShape", ConcatShape) +.set_attr("key_var_num_args", "num_args") +.add_argument("data", "NDArray-or-Symbol[]", "List of arrays to concatenate") +.add_arguments(ConcatParam::__FIELDS__()); + +NNVM_REGISTER_OP(Concat) +.set_attr("FQuantizedOp", [](const NodeAttrs& attrs) { + nnvm::NodePtr node = nnvm::Node::Create(); + node->attrs.op = Op::Get("_contrib_quantized_concat"); + node->attrs.name = "quantized_" + attrs.name; + node->attrs.dict = attrs.dict; + if (node->op()->attr_parser != nullptr) { + node->op()->attr_parser(&(node->attrs)); + } + return node; +}); + +} // namespace op +} // namespace mxnet diff --git a/src/operator/quantization/quantized_fully_connected.cc b/src/operator/quantization/quantized_fully_connected.cc index e334fe7ec..64ce73ba1 100644 --- a/src/operator/quantization/quantized_fully_connected.cc +++ b/src/operator/quantization/quantized_fully_connected.cc @@ -23,11 +23,17 @@ * \brief * \author Ziheng Jiang, Jun Wu */ +#include +#include "quantization_utils.h" #include "../nn/fully_connected-inl.h" namespace mxnet { namespace op { +namespace quantized_fc { +enum QuantizedfcOpResource {kTempSpace}; +} + bool QuantizedFullyConnectedShape(const nnvm::NodeAttrs& attrs, std::vector *in_shape, std::vector *out_shape) { @@ -79,6 +85,151 @@ bool QuantizedFullyConnectedType(const nnvm::NodeAttrs& attrs, return true; } +bool QuantizedFullyConnectedStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector *in_attrs, + std::vector *out_attrs) { + *dispatch_mode = DispatchMode::kFCompute; + if (dev_mask == mshadow::cpu::kDevMask) { + *dispatch_mode = DispatchMode::kFComputeEx; + } + + for (auto &v : *out_attrs) { + v = kDefaultStorage; + if (common::stype_string(v).compare("unknown") == 0) { + return false; + } + } + + for (auto &v : *in_attrs) { + v = kDefaultStorage; + if (common::stype_string(v).compare("unknown") == 0) { + return false; + } + } + return true; +} + +struct QuantizedSumInitKernelWithBias { + // init sum data with bias for matrix b (n) + MSHADOW_XINLINE static void Map(int i, int32_t *out, + const int8_t *bias, const float *min_out, + const float *max_out, const float *min_bias, + const float *max_bias) { + typedef int32_t T1; + typedef int8_t T2; + using mshadow::red::limits::MinValue; + using mshadow::red::limits::MaxValue; + float float_for_one_out_quant = + MaxAbs(*min_out, *max_out) / static_cast(MaxValue()); + float float_for_one_bias_quant = + MaxAbs(*min_bias, *max_bias) / static_cast(MaxValue()); + if (float_for_one_out_quant != 0) { + out[i] = bias[i] * float_for_one_bias_quant / + float_for_one_out_quant; + } else { + LOG(INFO) << "float_for_one_out_quant is 0," + << " need to check the why MaxAbs(*min_out, *max_out) of out_data is 0!"; + out[i] = 0; + } + } +}; + + +template +void QuantizedFullyConnectedForward(const nnvm::NodeAttrs& attrs, + const OpContext &ctx, + const std::vector &in_data, + const std::vector &req, + const std::vector &out_data) { +#if MSHADOW_USE_MKL == 1 + const FullyConnectedParam& param = nnvm::get(attrs.parsed); + using namespace mshadow; + using namespace mxnet_op; + size_t num_inputs = param.no_bias ? 2 : 3; + CHECK_EQ(in_data.size(), num_inputs * 3); + CHECK_EQ(out_data.size(), 3U); + const NDArray& data = in_data[0]; + const NDArray& weight = in_data[1]; + const NDArray& out = out_data[0]; + TShape dshape = data.shape(); + TShape wshape = weight.shape(); + TShape oshape = out.shape(); + auto output_temp = out.data().dptr(); + auto weight_temp = weight.data().dptr(); + auto data_temp = data.data().dptr(); + const int omp_threads = mxnet::engine::OpenMP::Get()->GetRecommendedOMPThreadCount(); + const float alpha = 1.0f; + const float beta = 1.0f; + const CBLAS_OFFSET offsetc = CblasFixOffset; + const MKL_INT8 oa = 0; + const MKL_INT8 ob = 0; + MKL_INT32 oc = 0; + const int m = dshape[0], n = wshape[0], k = dshape.ProdShape(1, dshape.ndim()); + Stream *s = ctx.get_stream(); + // cblas_gemm_s8u8s32 required first matrix must be uint8 + // shift data from int8(from -128 to 127) to uint8 (from 0 to 255) + int shift = 128; + Tensor shiftdata = + ctx.requested[quantized_fc::kTempSpace].get_space_typed( + Shape1(m * k), s); + #pragma omp parallel for num_threads(omp_threads) + for (int i = 0; i < m * k; ++i) { + shiftdata.dptr_[i] = data_temp[i] + shift; + } + + Kernel::Launch(s, 1, + out_data[1].data().dptr(), out_data[2].data().dptr(), + in_data[num_inputs].data().dptr(), in_data[num_inputs+1].data().dptr(), + in_data[num_inputs+2].data().dptr(), in_data[num_inputs+3].data().dptr()); + if (!param.no_bias) { + const NDArray& bias = in_data[2]; + Kernel::Launch(s, n, out.data().dptr(), + bias.data().dptr(), out_data[1].data().dptr(), + out_data[2].data().dptr(), in_data[7].data().dptr(), + in_data[8].data().dptr()); + } else { + #pragma omp parallel for num_threads(omp_threads) + for (int i = 0; i < m * n; ++i) { + output_temp[i] = 0; + } + } + #pragma omp parallel for num_threads(omp_threads) + for (int i = 0; i < n; ++i) { + for (int j = 0; j < k; ++j) { + output_temp[i] -= shift * weight_temp[i * k + j]; + } + } + #pragma omp parallel for num_threads(omp_threads) + for (int i = n; i < m * n; ++i) { + output_temp[i] = output_temp[i % n]; + } + cblas_gemm_s8u8s32(CblasRowMajor, + CblasNoTrans, + CblasTrans, + offsetc, + m, + n, + k, + alpha, + shiftdata.dptr_, + k, + oa, + weight.data().dptr(), + k, + ob, + beta, + out.data().dptr(), + n, + &oc); +#else + LOG(FATAL) << "Quantized fully connected operator relies on cblas_gemm_s8u8s32" + << " which is only supported by MKL BLAS." + << " Please build MXNet with USE_BLAS=mkl to leverage this operator."; +#endif +} + NNVM_REGISTER_OP(_contrib_quantized_fully_connected) .describe(R"code(Fully Connected operator for input, weight and bias data type of int8, and accumulates in type int32 for the output. For each argument, two more arguments of type @@ -112,7 +263,14 @@ and max thresholds representing the threholds for quantizing the float32 output }) .set_attr("FInferShape", QuantizedFullyConnectedShape) .set_attr("FInferType", QuantizedFullyConnectedType) +.set_attr("FInferStorageType", QuantizedFullyConnectedStorageType) .set_attr("FNeedRequantize", [](const NodeAttrs& attrs) { return true; }) +.set_attr("FComputeEx", + QuantizedFullyConnectedForward) +.set_attr("FResourceRequest", + [](const NodeAttrs& attrs) { + return std::vector{ResourceRequest::kTempSpace}; + }) .add_argument("data", "NDArray-or-Symbol", "Input data.") .add_argument("weight", "NDArray-or-Symbol", "weight.") .add_argument("bias", "NDArray-or-Symbol", "bias.") @@ -135,6 +293,5 @@ NNVM_REGISTER_OP(FullyConnected) } return node; }); - } // namespace op } // namespace mxnet diff --git a/src/operator/quantization/quantized_pooling.cc b/src/operator/quantization/quantized_pooling.cc index 779e244c8..8b62db9c0 100644 --- a/src/operator/quantization/quantized_pooling.cc +++ b/src/operator/quantization/quantized_pooling.cc @@ -52,17 +52,30 @@ bool QuantizedPoolingShape(const nnvm::NodeAttrs& attrs, << "kernel size (" << param.kernel[1] << ") exceeds input (" << dshape[W] << " padded to " << (dshape[W] + 2*param.pad[1]) << ")"; - // only support valid convention + oshape[N] = dshape[N]; oshape[C] = dshape[C]; if (param.global_pool) { oshape[H] = 1; oshape[W] = 1; } else { - oshape[H] = 1 + (dshape[H] + 2 * param.pad[0] - param.kernel[0]) / - param.stride[0]; - oshape[W] = 1 + (dshape[W] + 2 * param.pad[1] - param.kernel[1]) / - param.stride[1]; + if (param.pooling_convention == pool_enum::kValid) { + oshape[H] = 1 + + (dshape[H] + 2 * param.pad[0] - param.kernel[0]) / + param.stride[0]; + oshape[W] = 1 + + (dshape[W] + 2 * param.pad[1] - param.kernel[1]) / + param.stride[1]; + } else { + oshape[H] = 1 + static_cast(std::ceil( + static_cast(dshape[H] + 2 * param.pad[0] - + param.kernel[0]) / + param.stride[0])); + oshape[W] = 1 + static_cast(std::ceil( + static_cast(dshape[W] + 2 * param.pad[1] - + param.kernel[1]) / + param.stride[1])); + } } SHAPE_ASSIGN_CHECK(*in_shape, 1, TShape{1}); @@ -114,8 +127,8 @@ inline static bool QuantizedPoolingStorageType(const nnvm::NodeAttrs &attrs, #else CHECK_EQ(out_attrs->size(), 3); #endif - for (size_t i = 0; i < out_attrs->size(); i++) - (*out_attrs)[i] = kDefaultStorage; + for (int& out_attr : *out_attrs) + out_attr = kDefaultStorage; return true; } diff --git a/src/operator/random/multisample_op.h b/src/operator/random/multisample_op.h index 360c100c6..abd4a2c6c 100644 --- a/src/operator/random/multisample_op.h +++ b/src/operator/random/multisample_op.h @@ -78,8 +78,8 @@ inline bool MultiSampleOpShape(const nnvm::NodeAttrs& attrs, tshape = TShape(tshape.begin(), tshape.begin()+(tshape.ndim()-sshape.ndim())); } // Shape assignemnt/checking for inputs. - for (size_t i = 0; i < in_attrs->size(); ++i) { - if ( !shape_assign(&tshape, (*in_attrs)[i])) return false; + for (const auto& in_attr : *in_attrs) { + if ( !shape_assign(&tshape, in_attr)) return false; } for (size_t i = 0; i < in_attrs->size(); ++i) { SHAPE_ASSIGN_CHECK(*in_attrs, i, tshape); @@ -105,8 +105,8 @@ inline bool MultiSampleOpType(const nnvm::NodeAttrs& attrs, // All inputs must have same type. int dtype = -1; - for (size_t i = 0; i < in_attrs->size(); ++i) { - if (!type_assign(&dtype, (*in_attrs)[i])) return false; + for (int in_attr : *in_attrs) { + if (!type_assign(&dtype, in_attr)) return false; } for (size_t i = 0; i < in_attrs->size(); ++i) { TYPE_ASSIGN_CHECK(*in_attrs, i, dtype); diff --git a/src/operator/random/sample_op.cc b/src/operator/random/sample_op.cc index a2b332456..b065615e1 100644 --- a/src/operator/random/sample_op.cc +++ b/src/operator/random/sample_op.cc @@ -36,16 +36,46 @@ DMLC_REGISTER_PARAMETER(SampleExponentialParam); DMLC_REGISTER_PARAMETER(SamplePoissonParam); DMLC_REGISTER_PARAMETER(SampleNegBinomialParam); DMLC_REGISTER_PARAMETER(SampleGenNegBinomialParam); - -#define MXNET_OPERATOR_REGISTER_SAMPLE(name, ParamType) \ - NNVM_REGISTER_OP(name) \ - .set_num_inputs(0) \ - .set_num_outputs(1) \ - .set_attr_parser(ParamParser) \ - .set_attr("FInferShape", InitShape) \ - .set_attr("FInferType", SampleOpType) \ - .set_attr("FResourceRequest", SampleResource) \ - .add_arguments(ParamType::__FIELDS__()) +DMLC_REGISTER_PARAMETER(SampleRandIntParam); + +DMLC_REGISTER_PARAMETER(SampleUniformLikeParam); +DMLC_REGISTER_PARAMETER(SampleNormalLikeParam); +DMLC_REGISTER_PARAMETER(SampleGammaLikeParam); +DMLC_REGISTER_PARAMETER(SampleExponentialLikeParam); +DMLC_REGISTER_PARAMETER(SamplePoissonLikeParam); +DMLC_REGISTER_PARAMETER(SampleNegBinomialLikeParam); +DMLC_REGISTER_PARAMETER(SampleGenNegBinomialLikeParam); + +#define MXNET_OPERATOR_REGISTER_SAMPLE(name, ParamType) \ + NNVM_REGISTER_OP(name) \ + .set_num_inputs(0) \ + .set_num_outputs(1) \ + .set_attr_parser(ParamParser) \ + .set_attr("FInferShape", InitShape) \ + .set_attr("FInferType", SampleOpType) \ + .set_attr("FResourceRequest", SampleResource) \ + .add_arguments(ParamType::__FIELDS__()) \ + .set_attr("FInferStorageType", InitStorageType) \ + .set_attr("FCompute", Sample_) \ + .set_attr("FComputeEx", SampleEx_) + +#define MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(name, ParamType) \ + NNVM_REGISTER_OP(name) \ + .set_num_inputs(1) \ + .set_num_outputs(1) \ + .set_attr_parser(ParamParser) \ + .set_attr("FInferShape", ElemwiseShape<1, 1>) \ + .set_attr("FInferType", ElemwiseType<1, 1>) \ + .set_attr("FResourceRequest", SampleResource) \ + .set_attr("FIgnoreInputs", \ + [](const NodeAttrs& attrs) { return std::vector(1, 0); }) \ + .set_attr("FGradient", MakeZeroGradNodes) \ + .add_arguments(ParamType::__FIELDS__()) \ + .add_argument("data", "NDArray-or-Symbol", "The input") \ + .set_attr("FInferStorageType", \ + ElemwiseStorageType<1, 1, false, true, false>) \ + .set_attr("FCompute", Sample_) \ + .set_attr("FComputeEx", SampleEx_) // Add "uniform" alias for backward compatibility MXNET_OPERATOR_REGISTER_SAMPLE(_random_uniform, SampleUniformParam) @@ -63,10 +93,7 @@ Example:: uniform(low=0, high=1, shape=(2,2)) = [[ 0.60276335, 0.85794562], [ 0.54488319, 0.84725171]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); // Add "normal" alias for backward compatibility MXNET_OPERATOR_REGISTER_SAMPLE(_random_normal, SampleNormalParam) @@ -76,16 +103,14 @@ MXNET_OPERATOR_REGISTER_SAMPLE(_random_normal, SampleNormalParam) .. note:: The existing alias ``normal`` is deprecated. -Samples are distributed according to a normal distribution parametrized by *loc* (mean) and *scale* (standard deviation). +Samples are distributed according to a normal distribution parametrized by *loc* (mean) and *scale* +(standard deviation). Example:: normal(loc=0, scale=1, shape=(2,2)) = [[ 1.89171135, -1.16881478], [-1.23474145, 1.55807114]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); MXNET_OPERATOR_REGISTER_SAMPLE(_random_gamma, SampleGammaParam) .add_alias("random_gamma") @@ -97,10 +122,7 @@ Example:: gamma(alpha=9, beta=0.5, shape=(2,2)) = [[ 7.10486984, 3.37695289], [ 3.91697288, 3.65933681]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); MXNET_OPERATOR_REGISTER_SAMPLE(_random_exponential, SampleExponentialParam) .add_alias("random_exponential") @@ -112,11 +134,7 @@ Example:: exponential(lam=4, shape=(2,2)) = [[ 0.0097189 , 0.08999364], [ 0.04146638, 0.31715935]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", - InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); MXNET_OPERATOR_REGISTER_SAMPLE(_random_poisson, SamplePoissonParam) .add_alias("random_poisson") @@ -129,10 +147,7 @@ Example:: poisson(lam=4, shape=(2,2)) = [[ 5., 2.], [ 4., 6.]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); MXNET_OPERATOR_REGISTER_SAMPLE(_random_negative_binomial, SampleNegBinomialParam) .add_alias("random_negative_binomial") @@ -146,11 +161,7 @@ Example:: negative_binomial(k=3, p=0.4, shape=(2,2)) = [[ 4., 7.], [ 2., 5.]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", - InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); MXNET_OPERATOR_REGISTER_SAMPLE(_random_generalized_negative_binomial, SampleGenNegBinomialParam) .add_alias("random_generalized_negative_binomial") @@ -165,11 +176,111 @@ Example:: generalized_negative_binomial(mu=2.0, alpha=0.3, shape=(2,2)) = [[ 2., 1.], [ 6., 4.]] -)code" ADD_FILELINE) -.set_attr("FInferStorageType", - InitStorageType) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE(_random_randint, SampleRandIntParam) +.add_alias("random_randint") +.describe(R"code(Draw random samples from a discrete uniform distribution. + +Samples are uniformly distributed over the half-open interval *[low, high)* +(includes *low*, but excludes *high*). + +Example:: + + randint(low=0, high=5, shape=(2,2)) = [[ 0, 2], + [ 3, 1]] + +)code" ADD_FILELINE); + +// *_like operators + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_uniform_like, SampleUniformLikeParam) +.describe(R"code(Draw random samples from a uniform distribution according to the input array shape. + +Samples are uniformly distributed over the half-open interval *[low, high)* +(includes *low*, but excludes *high*). + +Example:: + + uniform(low=0, high=1, data=ones(2,2)) = [[ 0.60276335, 0.85794562], + [ 0.54488319, 0.84725171]] + +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_normal_like, SampleNormalLikeParam) +.describe(R"code(Draw random samples from a normal (Gaussian) distribution according to the input array shape. + +Samples are distributed according to a normal distribution parametrized by *loc* (mean) and *scale* +(standard deviation). + +Example:: + + normal(loc=0, scale=1, data=ones(2,2)) = [[ 1.89171135, -1.16881478], + [-1.23474145, 1.55807114]] +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_gamma_like, SampleGammaLikeParam) +.describe(R"code(Draw random samples from a gamma distribution according to the input array shape. + +Samples are distributed according to a gamma distribution parametrized by *alpha* (shape) and *beta* (scale). + +Example:: + + gamma(alpha=9, beta=0.5, data=ones(2,2)) = [[ 7.10486984, 3.37695289], + [ 3.91697288, 3.65933681]] +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_exponential_like, SampleExponentialLikeParam) +.describe(R"code(Draw random samples from an exponential distribution according to the input array shape. + +Samples are distributed according to an exponential distribution parametrized by *lambda* (rate). + +Example:: + + exponential(lam=4, data=ones(2,2)) = [[ 0.0097189 , 0.08999364], + [ 0.04146638, 0.31715935]] +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_poisson_like, SamplePoissonLikeParam) +.describe(R"code(Draw random samples from a Poisson distribution according to the input array shape. + +Samples are distributed according to a Poisson distribution parametrized by *lambda* (rate). +Samples will always be returned as a floating point data type. + +Example:: + + poisson(lam=4, data=ones(2,2)) = [[ 5., 2.], + [ 4., 6.]] +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_negative_binomial_like, SampleNegBinomialLikeParam) +.describe(R"code(Draw random samples from a negative binomial distribution according to the input array shape. + +Samples are distributed according to a negative binomial distribution parametrized by +*k* (limit of unsuccessful experiments) and *p* (failure probability in each experiment). +Samples will always be returned as a floating point data type. + +Example:: + + negative_binomial(k=3, p=0.4, data=ones(2,2)) = [[ 4., 7.], + [ 2., 5.]] +)code" ADD_FILELINE); + +MXNET_OPERATOR_REGISTER_SAMPLE_LIKE(_random_generalized_negative_binomial_like, + SampleGenNegBinomialLikeParam) +.describe(R"code(Draw random samples from a generalized negative binomial distribution according to the +input array shape. + +Samples are distributed according to a generalized negative binomial distribution parametrized by +*mu* (mean) and *alpha* (dispersion). *alpha* is defined as *1/k* where *k* is the failure limit of the +number of unsuccessful experiments (generalized to real numbers). +Samples will always be returned as a floating point data type. + +Example:: + + generalized_negative_binomial(mu=2.0, alpha=0.3, data=ones(2,2)) = [[ 2., 1.], + [ 6., 4.]] +)code" ADD_FILELINE); } // namespace op } // namespace mxnet diff --git a/src/operator/random/sample_op.cu b/src/operator/random/sample_op.cu index 7a593d0d3..39ab55afc 100644 --- a/src/operator/random/sample_op.cu +++ b/src/operator/random/sample_op.cu @@ -27,33 +27,27 @@ namespace mxnet { namespace op { -NNVM_REGISTER_OP(_random_uniform) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_normal) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_gamma) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_exponential) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_poisson) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_negative_binomial) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); - -NNVM_REGISTER_OP(_random_generalized_negative_binomial) -.set_attr("FCompute", Sample_>) -.set_attr("FComputeEx", SampleEx_>); +#define MXNET_OPERATOR_REGISTER_SAMPLE_GPU(name, ParamType) \ + NNVM_REGISTER_OP(name) \ + .set_attr("FCompute", Sample_) \ + .set_attr("FComputeEx", SampleEx_); \ + +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_uniform, SampleUniformParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_normal, SampleNormalParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_gamma, SampleGammaParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_exponential, SampleExponentialParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_poisson, SamplePoissonParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_negative_binomial, SampleNegBinomialParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_generalized_negative_binomial, SampleGenNegBinomialParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_randint, SampleRandIntParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_uniform_like, SampleUniformLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_normal_like, SampleNormalLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_gamma_like, SampleGammaLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_exponential_like, SampleExponentialLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_poisson_like, SamplePoissonLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_negative_binomial_like, SampleNegBinomialLikeParam) +MXNET_OPERATOR_REGISTER_SAMPLE_GPU(_random_generalized_negative_binomial_like, + SampleGenNegBinomialLikeParam) } // namespace op } // namespace mxnet diff --git a/src/operator/random/sample_op.h b/src/operator/random/sample_op.h index a81b41a09..b12dfafbc 100644 --- a/src/operator/random/sample_op.h +++ b/src/operator/random/sample_op.h @@ -38,12 +38,53 @@ namespace mxnet { namespace op { -struct SampleUniformParam : public dmlc::Parameter { - float low; - float high; + +struct SampleOpParam { TShape shape; std::string ctx; int dtype; +}; + +struct UniformParam { + float low; + float high; +}; + +struct NormalParam { + float loc; + float scale; +}; + +struct GammaParam { + float alpha; + float beta; +}; + +struct ExponentialParam { + float lam; +}; + +struct PoissonParam { + float lam; +}; + +struct NegBinomialParam { + int k; + float p; +}; + +struct GenNegBinomialParam { + float mu; + float alpha; +}; + +struct RandIntParam { + int64_t low; + int64_t high; +}; + +struct SampleUniformParam : public dmlc::Parameter, + UniformParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleUniformParam) { DMLC_DECLARE_FIELD(low).set_default(0.0f) .describe("Lower bound of the distribution."); @@ -67,12 +108,8 @@ struct SampleUniformParam : public dmlc::Parameter { } }; -struct SampleNormalParam : public dmlc::Parameter { - float loc; - float scale; - TShape shape; - std::string ctx; - int dtype; +struct SampleNormalParam : public dmlc::Parameter, + NormalParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleNormalParam) { DMLC_DECLARE_FIELD(loc).set_default(0.0f) .describe("Mean of the distribution."); @@ -96,12 +133,8 @@ struct SampleNormalParam : public dmlc::Parameter { } }; -struct SampleGammaParam : public dmlc::Parameter { - float alpha; - float beta; - TShape shape; - std::string ctx; - int dtype; +struct SampleGammaParam : public dmlc::Parameter, + GammaParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleGammaParam) { DMLC_DECLARE_FIELD(alpha).set_default(1.0f) .describe("Alpha parameter (shape) of the gamma distribution."); @@ -125,11 +158,8 @@ struct SampleGammaParam : public dmlc::Parameter { } }; -struct SampleExponentialParam : public dmlc::Parameter { - float lam; - TShape shape; - std::string ctx; - int dtype; +struct SampleExponentialParam : public dmlc::Parameter, + ExponentialParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleExponentialParam) { DMLC_DECLARE_FIELD(lam).set_default(1.0f) .describe("Lambda parameter (rate) of the exponential distribution."); @@ -151,11 +181,8 @@ struct SampleExponentialParam : public dmlc::Parameter { } }; -struct SamplePoissonParam : public dmlc::Parameter { - float lam; - TShape shape; - std::string ctx; - int dtype; +struct SamplePoissonParam : public dmlc::Parameter, + PoissonParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SamplePoissonParam) { DMLC_DECLARE_FIELD(lam).set_default(1.0f) .describe("Lambda parameter (rate) of the Poisson distribution."); @@ -177,12 +204,8 @@ struct SamplePoissonParam : public dmlc::Parameter { } }; -struct SampleNegBinomialParam : public dmlc::Parameter { - int k; - float p; - TShape shape; - std::string ctx; - int dtype; +struct SampleNegBinomialParam : public dmlc::Parameter, + NegBinomialParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleNegBinomialParam) { DMLC_DECLARE_FIELD(k).set_default(1) .describe("Limit of unsuccessful experiments."); @@ -206,12 +229,8 @@ struct SampleNegBinomialParam : public dmlc::Parameter { } }; -struct SampleGenNegBinomialParam : public dmlc::Parameter { - float mu; - float alpha; - TShape shape; - std::string ctx; - int dtype; +struct SampleGenNegBinomialParam : public dmlc::Parameter, + GenNegBinomialParam, SampleOpParam { DMLC_DECLARE_PARAMETER(SampleGenNegBinomialParam) { DMLC_DECLARE_FIELD(mu).set_default(1.0f) .describe("Mean of the negative binomial distribution."); @@ -235,6 +254,96 @@ struct SampleGenNegBinomialParam : public dmlc::Parameter, + RandIntParam, SampleOpParam { + DMLC_DECLARE_PARAMETER(SampleRandIntParam) { + DMLC_DECLARE_FIELD(low) + .describe("Lower bound of the distribution."); + DMLC_DECLARE_FIELD(high) + .describe("Upper bound of the distribution."); + DMLC_DECLARE_FIELD(shape) + .set_default(TShape()) + .describe("Shape of the output."); + DMLC_DECLARE_FIELD(ctx) + .set_default("") + .describe("Context of output, in format [cpu|gpu|cpu_pinned](n)." + " Only used for imperative calls."); + DMLC_DECLARE_FIELD(dtype) + .add_enum("None", -1) + .add_enum("int32", kInt32) + .add_enum("int64", kInt64) + .set_default(-1) + .describe("DType of the output in case this can't be inferred. " + "Defaults to int32 if not defined (dtype=None)."); + } +}; + +struct SampleUniformLikeParam : public dmlc::Parameter, + UniformParam { + DMLC_DECLARE_PARAMETER(SampleUniformLikeParam) { + DMLC_DECLARE_FIELD(low).set_default(0.0f) + .describe("Lower bound of the distribution."); + DMLC_DECLARE_FIELD(high).set_default(1.0f) + .describe("Upper bound of the distribution."); + } +}; + +struct SampleNormalLikeParam : public dmlc::Parameter, + NormalParam { + DMLC_DECLARE_PARAMETER(SampleNormalLikeParam) { + DMLC_DECLARE_FIELD(loc).set_default(0.0f) + .describe("Mean of the distribution."); + DMLC_DECLARE_FIELD(scale).set_default(1.0f) + .describe("Standard deviation of the distribution."); + } +}; + +struct SampleGammaLikeParam : public dmlc::Parameter, + GammaParam { + DMLC_DECLARE_PARAMETER(SampleGammaLikeParam) { + DMLC_DECLARE_FIELD(alpha).set_default(1.0f) + .describe("Alpha parameter (shape) of the gamma distribution."); + DMLC_DECLARE_FIELD(beta).set_default(1.0f) + .describe("Beta parameter (scale) of the gamma distribution."); + } +}; + +struct SampleExponentialLikeParam : public dmlc::Parameter, + ExponentialParam { + DMLC_DECLARE_PARAMETER(SampleExponentialLikeParam) { + DMLC_DECLARE_FIELD(lam).set_default(1.0f) + .describe("Lambda parameter (rate) of the exponential distribution."); + } +}; + +struct SamplePoissonLikeParam : public dmlc::Parameter, + PoissonParam { + DMLC_DECLARE_PARAMETER(SamplePoissonLikeParam) { + DMLC_DECLARE_FIELD(lam).set_default(1.0f) + .describe("Lambda parameter (rate) of the Poisson distribution."); + } +}; + +struct SampleNegBinomialLikeParam : public dmlc::Parameter, + NegBinomialParam { + DMLC_DECLARE_PARAMETER(SampleNegBinomialLikeParam) { + DMLC_DECLARE_FIELD(k).set_default(1) + .describe("Limit of unsuccessful experiments."); + DMLC_DECLARE_FIELD(p).set_default(1.0f) + .describe("Failure probability in each experiment."); + } +}; + +struct SampleGenNegBinomialLikeParam : public dmlc::Parameter, + GenNegBinomialParam { + DMLC_DECLARE_PARAMETER(SampleGenNegBinomialLikeParam) { + DMLC_DECLARE_FIELD(mu).set_default(1.0f) + .describe("Mean of the negative binomial distribution."); + DMLC_DECLARE_FIELD(alpha).set_default(1.0f) + .describe("Alpha (dispersion) parameter of the negative binomial distribution."); + } +}; + using FSampleCompute = std::function(&p2, Shape1(1)), s); } -template -struct SampleMaster; +template +static inline void uniform_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const UniformParam& param = nnvm::get(attrs.parsed); + CHECK_GE(param.high, param.low) << "low must be less or equal to high in uniform distribution"; + Tensor low, high; + GetSamplingTempData(param.low, param.high, ctx, + &low, &high); + UniformSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(low, high, out, pgen, s); + }); +} -template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { +template +static inline void normal_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const NormalParam& param = nnvm::get(attrs.parsed); + CHECK_GT(param.scale, 0) << "scale parameter in gaussian has to be positive"; + Tensor loc, scale; + GetSamplingTempData(param.loc, param.scale, ctx, &loc, &scale); + NormalSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(loc, scale, out, pgen, s); + }); +} + +template +static inline void gamma_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const GammaParam& param = nnvm::get(attrs.parsed); + CHECK_GT(param.alpha, 0) << "alpha parameter in gamma distribution has to be positive"; + CHECK_GT(param.beta, 0) << "beta parameter in gamma distribution has to be positive"; + Tensor alpha, beta; + GetSamplingTempData(param.alpha, param.beta, ctx, &alpha, &beta); + GammaSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(alpha, beta, out, pgen, s); + }); +} + +template +static inline void exponential_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const ExponentialParam& param = nnvm::get(attrs.parsed); + CHECK_GT(param.lam, 0) << "lambda parameter in exponential distribution has to be positive"; + Tensor lam, dummy; + GetSamplingTempData(param.lam, 0, ctx, &lam, &dummy); + ExponentialSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(lam, out, pgen, s); + }); +} + +template +static inline void poisson_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const PoissonParam& param = nnvm::get(attrs.parsed); + CHECK_GE(param.lam, 0) << "lambda parameter in poisson distribution has to be non-negative"; + Tensor lam, dummy; + GetSamplingTempData(param.lam, 0, ctx, &lam, &dummy); + PoissonSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(lam, out, pgen, s); + }); +} + +template +static inline void neg_binomial_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const NegBinomialParam& param = nnvm::get(attrs.parsed); + CHECK_GE(param.k, 0) << "k parameter in negative binomial distribution has to be non-negative"; + CHECK_GE(param.p, 0) << "p parameter in negative binomial distribution has to be non-negative"; + Tensor k, p; + GetSamplingTempData(param.k, param.p, ctx, &k, &p); + NegativeBinomialSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(k, p, out, pgen, s); + }); +} + +template +static inline void gen_neg_binomial_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + Stream *s = ctx.get_stream(); + const GenNegBinomialParam& param = nnvm::get(attrs.parsed); + CHECK_GE(param.mu, 0) + << "mu parameter in generalized negative binomial distribution has to be non-negative"; + CHECK_GE(param.alpha, 0) + << "alpha parameter in generalized negative binomial distribution has to be non-negative"; + Tensor mu, alpha; + GetSamplingTempData(param.mu, param.alpha, ctx, &mu, &alpha); + GeneralizedNegativeBinomialSampler sampler; + MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandGenerator *pgen = ctx.requested[0].get_parallel_random(); + Tensor out = outputs->FlatTo1D(s); + sampler.Sample(mu, alpha, out, pgen, s); + }); +} + +template +static inline void rand_int_op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { Stream *s = ctx.get_stream(); - const SampleUniformParam& param = nnvm::get(attrs.parsed); + const SampleRandIntParam& param = nnvm::get(attrs.parsed); CHECK_GE(param.high, param.low) << "low must be less or equal to high in uniform distribution"; - Tensor low, high; - GetSamplingTempData(param.low, param.high, ctx, + Tensor low, high; + GetSamplingTempData(param.low, param.high, ctx, &low, &high); - UniformSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { + RandIntSampler sampler; + MSHADOW_TYPE_SWITCH(outputs[0].type_flag_, OType, { RandGenerator *pgen = ctx.requested[0].get_parallel_random(); Tensor out = outputs->FlatTo1D(s); sampler.Sample(low, high, out, pgen, s); }); +} + +template +struct SampleMaster; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + uniform_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SampleNormalParam& param = nnvm::get(attrs.parsed); - CHECK_GT(param.scale, 0) << "scale parameter in gaussian has to be positive"; - Tensor loc, scale; - GetSamplingTempData(param.loc, param.scale, ctx, &loc, &scale); - NormalSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(loc, scale, out, pgen, s); - }); +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + uniform_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SampleGammaParam& param = nnvm::get(attrs.parsed); - CHECK_GT(param.alpha, 0) << "alpha parameter in gamma distribution has to be positive"; - CHECK_GT(param.beta, 0) << "beta parameter in gamma distribution has to be positive"; - Tensor alpha, beta; - GetSamplingTempData(param.alpha, param.beta, ctx, &alpha, &beta); - GammaSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(alpha, beta, out, pgen, s); - }); +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + normal_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SampleExponentialParam& param = nnvm::get(attrs.parsed); - CHECK_GT(param.lam, 0) << "lambda parameter in exponential distribution has to be positive"; - Tensor lam, dummy; - GetSamplingTempData(param.lam, 0, ctx, &lam, &dummy); - ExponentialSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(lam, out, pgen, s); - }); +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + normal_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SamplePoissonParam& param = nnvm::get(attrs.parsed); - CHECK_GE(param.lam, 0) << "lambda parameter in poisson distribution has to be non-negative"; - Tensor lam, dummy; - GetSamplingTempData(param.lam, 0, ctx, &lam, &dummy); - PoissonSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(lam, out, pgen, s); - }); +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + gamma_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { - static void op(const nnvm::NodeAttrs& attrs, - const OpContext& ctx, - const OpReqType& req, - TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SampleNegBinomialParam& param = nnvm::get(attrs.parsed); - CHECK_GE(param.k, 0) << "k parameter in negative binomial distribution has to be non-negative"; - CHECK_GE(param.p, 0) << "p parameter in negative binomial distribution has to be non-negative"; - Tensor k, p; - GetSamplingTempData(param.k, param.p, ctx, &k, &p); - NegativeBinomialSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(k, p, out, pgen, s); - }); +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + gamma_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + exponential_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + exponential_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + poisson_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + poisson_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + neg_binomial_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + neg_binomial_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + gen_neg_binomial_op(attrs, ctx, req, outputs); } }; template -struct SampleMaster> { +struct SampleMaster { + static inline void op(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const OpReqType& req, + TBlob* outputs) { + gen_neg_binomial_op(attrs, ctx, req, outputs); + } +}; + +template +struct SampleMaster { static void op(const nnvm::NodeAttrs& attrs, const OpContext& ctx, const OpReqType& req, TBlob* outputs) { - Stream *s = ctx.get_stream(); - const SampleGenNegBinomialParam& param = nnvm::get(attrs.parsed); - CHECK_GE(param.mu, 0) - << "mu parameter in generalized negative binomial distribution has to be non-negative"; - CHECK_GE(param.alpha, 0) - << "alpha parameter in generalized negative binomial distribution has to be non-negative"; - Tensor mu, alpha; - GetSamplingTempData(param.mu, param.alpha, ctx, &mu, &alpha); - GeneralizedNegativeBinomialSampler sampler; - MSHADOW_REAL_TYPE_SWITCH(outputs[0].type_flag_, OType, { - RandGenerator *pgen = ctx.requested[0].get_parallel_random(); - Tensor out = outputs->FlatTo1D(s); - sampler.Sample(mu, alpha, out, pgen, s); - }); + rand_int_op(attrs, ctx, req, outputs); } }; -template +template void SampleComputeEx_(const nnvm::NodeAttrs& attrs, const OpContext& ctx, const std::vector& inputs, const std::vector& req, const std::vector& outputs, - SampleMaster sample_master) { + SampleMaster sample_master) { using namespace mxnet::op; NDArray output = outputs[0]; mshadow::Stream *s = ctx.get_stream(); @@ -438,24 +702,24 @@ void SampleComputeEx_(const nnvm::NodeAttrs& attrs, } } -template +template void Sample_(const nnvm::NodeAttrs& attrs, const OpContext& ctx, const std::vector& inputs, const std::vector& req, const std::vector& outputs) { TBlob out = outputs[0]; - SampleMaster::op(attrs, ctx, req[0], &out); + SampleMaster::op(attrs, ctx, req[0], &out); } -template +template void SampleEx_(const nnvm::NodeAttrs& attrs, const OpContext& ctx, const std::vector& inputs, const std::vector& req, const std::vector& outputs) { - SampleMaster sample_master; - SampleComputeEx_(attrs, ctx, inputs, req, outputs, sample_master); + SampleMaster sample_master; + SampleComputeEx_(attrs, ctx, inputs, req, outputs, sample_master); } template @@ -468,7 +732,7 @@ inline bool SampleOpType(const nnvm::NodeAttrs& attrs, int dtype = -1; int dtype_out = (*out_type)[0]; if (dtype_out != -1) { - // Output type can be inferred, use it and make sure it + // Output type can be inferred, use it and make sure it matches dtype = dtype_out; if (param.dtype != -1) { // dtype given in args, check that it matches the output type @@ -487,13 +751,47 @@ inline bool SampleOpType(const nnvm::NodeAttrs& attrs, } bool dtype_ok = (dtype == kFloat16) || (dtype == kFloat32) || (dtype == kFloat64); - CHECK_EQ(dtype_ok, true) << "Output type must be float16, float32, or float64: dtype is " + CHECK(dtype_ok) << "Output type must be float16, float32, float64: dtype is " << dtype_out << " vs " << kFloat16 << " or " << kFloat32 << " or " << kFloat64; TYPE_ASSIGN_CHECK(*out_type, 0, dtype); return true; } +template<> +inline bool SampleOpType(const nnvm::NodeAttrs& attrs, + std::vector *in_type, + std::vector *out_type) { + const SampleRandIntParam& param = nnvm::get(attrs.parsed); + CHECK_EQ(in_type->size(), 0); + CHECK_EQ(out_type->size(), 1); + int dtype = -1; + int dtype_out = (*out_type)[0]; + if (dtype_out != -1) { + // Output type can be inferred, use it and make sure it matches + dtype = dtype_out; + if (param.dtype != -1) { + // dtype given in args, check that it matches the output type + CHECK_EQ(dtype_out, param.dtype) << "Output type does not match requested type: " + << dtype_out << " vs " << param.dtype; + } + } else { + // Output type can't be inferred + if (param.dtype != -1) { + // Use dtype given in args + dtype = param.dtype; + } else { + // Use default + dtype = kInt32; + } + } + bool dtype_ok = (dtype == kInt32) || (dtype == kInt64); + CHECK(dtype_ok) << "Output type must be int32, int64: dtype is " + << dtype_out << " vs " << kInt32 << " or " << kInt64; + TYPE_ASSIGN_CHECK(*out_type, 0, dtype); + return true; +} + inline std::vector SampleResource(const NodeAttrs& attrs) { return { ResourceRequest::kParallelRandom, ResourceRequest::kTempSpace }; } diff --git a/src/operator/random/sampler.h b/src/operator/random/sampler.h index 44f80ab56..00963a678 100644 --- a/src/operator/random/sampler.h +++ b/src/operator/random/sampler.h @@ -43,32 +43,33 @@ namespace op { template inline static void LaunchRNG(mshadow::Stream *s, common::random::RandGenerator *gen, - const int N, Args... args) { + const index_t N, Args... args) { // minimal check to avoid division by zero, below. // if `N` is zero the map operation is a no-op in any case. if (N <= 0) { return; } - const int nloop = (N + RandGenerator::kMinNumRandomPerThread - 1) / + const index_t nloop = (N + RandGenerator::kMinNumRandomPerThread - 1) / RandGenerator::kMinNumRandomPerThread; - const int nthread = std::min(nloop, RandGenerator::kNumRandomStates); - const int step = (N + nthread - 1) / nthread; + const index_t nthread = std::min(nloop, + static_cast(RandGenerator::kNumRandomStates)); + const index_t step = (N + nthread - 1) / nthread; Kernel::Launch(s, nthread, *gen, N, step, args...); } #define RNG_KERNEL_LOOP(xpu, GType, thread_id, gen, N, step, ...) \ - const int start = thread_id * step; \ - const int end = start + step; \ + const index_t start = thread_id * step; \ + const index_t end = start + step; \ typename RandGenerator::Impl genImpl(&gen, thread_id); \ - for (int i = start; i < end && i < N; ++i) { \ + for (index_t i = start; i < end && i < N; ++i) { \ {__VA_ARGS__} \ } template struct SampleUniformKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *lower, const IType *upper, OType *out) { RNG_KERNEL_LOOP(xpu, OType, id, gen, N, step, { @@ -93,11 +94,43 @@ struct UniformSampler { }; template -struct SampleNormalKernel { +struct SampleRandIntKernel { template MSHADOW_XINLINE static void Map(int id, RandGenerator gen, const int N, const int step, index_t nParm, index_t nSample, + const IType *lower, const IType *upper, OType *out) { + RNG_KERNEL_LOOP(xpu, OType, id, gen, N, step, { + index_t nBatch(1 + (nSample - 1) / nParm); + if (sizeof(IType) == sizeof(int64_t)) + out[i] = OType(lower[i / nBatch] + + genImpl.rand_int64() % (upper[i / nBatch] - lower[i / nBatch])); + else + out[i] = OType(lower[i / nBatch] + + genImpl.rand() % (upper[i / nBatch] - lower[i / nBatch])); + }); + } +}; + +template +struct RandIntSampler { + template + MSHADOW_FORCE_INLINE void Sample(const Tensor& lower, + const Tensor& upper, + const Tensor& out, + RandGenerator *pgen, + Stream *s) { + LaunchRNG, xpu>(s, pgen, out.size(0), lower.size(0), out.size(0), + lower.dptr_, upper.dptr_, out.dptr_); + } +}; + +template +struct SampleNormalKernel { + template + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, + index_t nParm, index_t nSample, const IType *mean, const IType *std, OType *out) { RNG_KERNEL_LOOP(xpu, OType, id, gen, N, step, { index_t nBatch(1 + (nSample - 1) / nParm); @@ -122,8 +155,8 @@ struct NormalSampler { template struct SampleExponentialKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *lambda, OType *out) { RNG_KERNEL_LOOP(xpu, OType, id, gen, N, step, { @@ -170,8 +203,8 @@ MSHADOW_XINLINE OType SampleGamma(IType a, IType b, typename RandGenerator struct SampleGammaKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *alpha, const IType *beta, OType *out) { RNG_KERNEL_LOOP(xpu, FType, id, gen, N, step, { @@ -232,8 +265,8 @@ MSHADOW_XINLINE int SamplePoisson(float lambda, typename RandGenerator struct SamplePoissonKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *lambda, OType *out) { RNG_KERNEL_LOOP(xpu, float, id, gen, N, step, { @@ -259,8 +292,8 @@ struct PoissonSampler { template struct SampleNegativeBinomialKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *k, const IType *p, OType *out) { RNG_KERNEL_LOOP(xpu, float, id, gen, N, step, { @@ -291,8 +324,8 @@ struct NegativeBinomialSampler { template struct SampleGeneralizedNegativeBinomialKernel { template - MSHADOW_XINLINE static void Map(int id, RandGenerator gen, - const int N, const int step, + MSHADOW_XINLINE static void Map(index_t id, RandGenerator gen, + const index_t N, const index_t step, index_t nParm, index_t nSample, const IType *mu, const IType *alpha, OType *out) { RNG_KERNEL_LOOP(xpu, float, id, gen, N, step, { diff --git a/src/operator/rnn-inl.h b/src/operator/rnn-inl.h index 11ca066c7..545e31bd8 100644 --- a/src/operator/rnn-inl.h +++ b/src/operator/rnn-inl.h @@ -54,7 +54,8 @@ inline int GetRnnParamSize(int num_layer, int input_size, int state_size, int direction, - int mode) { + int mode, + const dmlc::optional& projection_size) { int size = state_size * direction; switch (mode) { case rnn_enum::kRnnRelu: @@ -69,7 +70,15 @@ inline int GetRnnParamSize(int num_layer, } int size1 = (input_size + state_size + 2) * size; // first layer size int size2 = (state_size * direction + state_size + 2) * size; // other layers size + if (projection_size.has_value()) { + int proj_size = projection_size.value(); + size1 = (input_size + proj_size + 2) * size; + size2 = (proj_size * direction + proj_size + 2) * size; + } int param_size = size1 + (num_layer - 1) * size2; + if (projection_size.has_value()) { + param_size += projection_size.value() * state_size * num_layer * direction; + } return param_size; } @@ -154,6 +163,9 @@ struct RNNParam : public dmlc::Parameter { float p, pkeep_; int seq_length_, batch_size_, input_size_; bool lstm_q_; // whether type is lstm + dmlc::optional projection_size; + dmlc::optional lstm_state_clip_min, lstm_state_clip_max; + bool lstm_state_clip_nan; DMLC_DECLARE_PARAMETER(RNNParam) { DMLC_DECLARE_FIELD(state_size) @@ -174,10 +186,29 @@ struct RNNParam : public dmlc::Parameter { DMLC_DECLARE_FIELD(p).set_default(0.) .set_range(0, 1) - .describe("Dropout probability, fraction of the input that gets dropped out at training time"); + .describe("drop rate of the dropout on the outputs of each RNN layer, except the last layer."); DMLC_DECLARE_FIELD(state_outputs).set_default(false) .describe("Whether to have the states as symbol outputs."); + + DMLC_DECLARE_FIELD(projection_size) + .set_default(dmlc::optional()) + .describe("size of project size"); + + DMLC_DECLARE_FIELD(lstm_state_clip_min) + .set_default(dmlc::optional()) + .describe("Minimum clip value of LSTM states. This option must be used together with " + "lstm_state_clip_max."); + + DMLC_DECLARE_FIELD(lstm_state_clip_max) + .set_default(dmlc::optional()) + .describe("Maximum clip value of LSTM states. This option must be used together with " + "lstm_state_clip_min."); + + DMLC_DECLARE_FIELD(lstm_state_clip_nan) + .set_default(false) + .describe("Whether to stop NaN from propagating in state by clipping it to min/max. " + "If clipping range is not specified, this option is ignored."); } }; @@ -349,8 +380,15 @@ template class RNNOp : public Operator{ public: explicit RNNOp(RNNParam p) - :param_(p), init_space_(false), reserve_space_size_(0) - {} + :param_(p), init_space_(false), reserve_space_size_(0) { + if (param_.projection_size.has_value()) { + LOG(FATAL) << "hidden layer projection is only supported for GPU with CuDNN later than 7.1.1"; + } + if (param_.lstm_state_clip_min.has_value() + || param_.lstm_state_clip_max.has_value()) { + LOG(FATAL) << "LSTM state clipping is only supported for GPU with CuDNN later than 7.2.1"; + } + } ~RNNOp() { if (init_space_) { @@ -646,26 +684,33 @@ class RNNProp : public OperatorProperty { int input_size = dshape[2]; int numDirections = param_.bidirectional ? 2 : 1; int total_layers = numDirections * param_.num_layers; // double for bidirectional + int layer_size = (param_.projection_size.has_value()) ? + param_.projection_size.value() : param_.state_size; SHAPE_ASSIGN_CHECK(*in_shape, rnn_enum::kState, - Shape3(total_layers, batch_size, param_.state_size)); + Shape3(total_layers, batch_size, layer_size)); if (param_.mode == rnn_enum::kLstm) SHAPE_ASSIGN_CHECK(*in_shape, - rnn_enum::kStateCell, - Shape3(total_layers, batch_size, param_.state_size)); + rnn_enum::kStateCell, + Shape3(total_layers, batch_size, param_.state_size)); // calculate parameter vector length int param_size = GetRnnParamSize(param_.num_layers, - input_size, - param_.state_size, - numDirections, - param_.mode); + input_size, + param_.state_size, + numDirections, + param_.mode, + param_.projection_size); SHAPE_ASSIGN_CHECK(*in_shape, rnn_enum::kParams, Shape1(param_size)); out_shape->clear(); // output: [sequence len, batch, output size] TShape oshape = dshape; - oshape[2] = numDirections * param_.state_size; + if (param_.projection_size.has_value()) { + oshape[2] = numDirections * param_.projection_size.value(); + } else { + oshape[2] = numDirections * param_.state_size; + } out_shape->push_back(oshape); if (!param_.state_outputs) { return true; @@ -674,11 +719,20 @@ class RNNProp : public OperatorProperty { TShape outStateShape = dshape; outStateShape[0] = total_layers; outStateShape[1] = batch_size; - outStateShape[2] = param_.state_size; + if (param_.projection_size.has_value()) { + outStateShape[2] = param_.projection_size.value(); + } else { + outStateShape[2] = param_.state_size; + } out_shape->push_back(outStateShape); // Deal with lstm cell state - if (param_.mode == rnn_enum::kLstm) - out_shape->push_back(outStateShape); + if (param_.mode == rnn_enum::kLstm) { + TShape cellStateShape = dshape; + cellStateShape[0] = total_layers; + cellStateShape[1] = batch_size; + cellStateShape[2] = param_.state_size; + out_shape->push_back(cellStateShape); + } return true; } } diff --git a/src/operator/rnn.cc b/src/operator/rnn.cc index 9ba764904..82b03c0fa 100644 --- a/src/operator/rnn.cc +++ b/src/operator/rnn.cc @@ -48,6 +48,11 @@ MXNET_REGISTER_OP_PROPERTY(RNN, RNNProp) .describe(R"code(Applies recurrent layers to input data. Currently, vanilla RNN, LSTM and GRU are implemented, with both multi-layer and bidirectional support. +When the input data is of type float32 and the environment variables MXNET_CUDA_ALLOW_TENSOR_CORE +and MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION are set to 1, this operator will try to use +pseudo-float16 precision (float32 math with float16 I/O) precision in order to use +Tensor Cores on suitable NVIDIA GPUs. This can sometimes give significant speedups. + **Vanilla RNN** Applies a single-gate recurrent layer to input X. Two kinds of activation function are supported: diff --git a/src/operator/rnn.cu b/src/operator/rnn.cu index 59517932b..402a8cf5f 100644 --- a/src/operator/rnn.cu +++ b/src/operator/rnn.cu @@ -40,7 +40,7 @@ Operator* CreateOp(RNNParam param, int dtype) { op = new CuDNNRNNOp(param); }) #else - LOG(FATAL) << "RNN is only available for cuDNN at the moment."; + LOG(FATAL) << "RNN on GPU is only available for cuDNN at the moment."; #endif // MXNET_USE_CUDNN && CUDNN_MAJOR return op; } diff --git a/src/operator/spatial_transformer-inl.h b/src/operator/spatial_transformer-inl.h index 3e863d877..a7ecdaecb 100644 --- a/src/operator/spatial_transformer-inl.h +++ b/src/operator/spatial_transformer-inl.h @@ -216,12 +216,12 @@ class SpatialTransformerProp : public OperatorProperty { std::vector *out_type, std::vector *aux_type) const override { int dtype = -1; - for (size_t i = 0; i < in_type->size(); ++i) { + for (int i_type : *in_type) { if (dtype == -1) { - dtype = in_type->at(i); + dtype = i_type; } else { - CHECK(in_type->at(i) == dtype || - in_type->at(i) == -1) << + CHECK(i_type == dtype || + i_type == -1) << "Non-uniform data type in SpatialTransformer"; } } diff --git a/src/operator/spatial_transformer.cu b/src/operator/spatial_transformer.cu index 33dbe3e7c..fd330bd4c 100644 --- a/src/operator/spatial_transformer.cu +++ b/src/operator/spatial_transformer.cu @@ -35,12 +35,23 @@ template __device__ bool between(DType value, int lowerBound, int upperBound) { return (value >= lowerBound && value <= upperBound); } + template -__global__ void BilinearSamplingForwardKernel(const int i_c, const int i_h, - const int i_w, const DType* data, - const DType* grid, const int o_n, - const int o_c, const int o_h, - const int o_w, DType* out) { +__global__ void +/* + * In order to not generate the code that uses too many + * registers (resulting in too many resources requested + * error) we need to tell the compiler that we will be + * launching this kernel with cuda::kMaxThreadsPerBlock + * threads per block. Setting __launch_bounds__ ensures + * that such configuration can always be launched. + */ +__launch_bounds__(cuda::kMaxThreadsPerBlock, 1) +BilinearSamplingForwardKernel(const int i_c, const int i_h, + const int i_w, const DType* data, + const DType* grid, const int o_n, + const int o_c, const int o_h, + const int o_w, DType* out) { for (int index = (blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x; index < o_n * o_c * o_h * o_w; index += blockDim.x * gridDim.x * gridDim.y) { @@ -77,13 +88,23 @@ __global__ void BilinearSamplingForwardKernel(const int i_c, const int i_h, } } +/* + * In order to not generate the code that uses too many + * registers (resulting in too many resources requested + * error) we need to tell the compiler that we will be + * launching this kernel with cuda::kMaxThreadsPerBlock + * threads per block. Setting __launch_bounds__ ensures + * that such configuration can always be launched. + */ template -__global__ void BilinearSamplingBackwardKernel(const int i_c, const int i_h, - const int i_w, const DType* grad, - const DType* data, const int o_n, - const int o_c, const int o_h, - const int o_w, DType* g_input, - DType* grid_src) { +__global__ void +__launch_bounds__(cuda::kMaxThreadsPerBlock, 1) +BilinearSamplingBackwardKernel(const int i_c, const int i_h, + const int i_w, const DType* grad, + const DType* data, const int o_n, + const int o_c, const int o_h, + const int o_w, DType* g_input, + DType* grid_src) { for (int index = (blockIdx.x + blockIdx.y * gridDim.x) * blockDim.x + threadIdx.x; index < o_n * o_h * o_w; index += blockDim.x * gridDim.x * gridDim.y) { diff --git a/src/operator/subgraph/mkldnn/mkldnn_conv-inl.h b/src/operator/subgraph/mkldnn/mkldnn_conv-inl.h index 8675446f5..b44f2fb0e 100644 --- a/src/operator/subgraph/mkldnn/mkldnn_conv-inl.h +++ b/src/operator/subgraph/mkldnn/mkldnn_conv-inl.h @@ -36,9 +36,6 @@ struct MKLDNNConvFusionParam { std::shared_ptr bn_param; }; -static const size_t uint8_range = 255; -static const size_t int8_range = 127; - enum MKLDNNConvOpOutputs { kOut, kMin, kMax }; } // namespace op diff --git a/src/operator/subgraph/mkldnn/mkldnn_conv.cc b/src/operator/subgraph/mkldnn/mkldnn_conv.cc index a1083d09b..65e0e5c4b 100644 --- a/src/operator/subgraph/mkldnn/mkldnn_conv.cc +++ b/src/operator/subgraph/mkldnn/mkldnn_conv.cc @@ -109,7 +109,7 @@ static void QuantizeConvWeightBias(NDArray *weight, NDArray *bias, #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) for (int c = 0; c < static_cast(channel); ++c) { DType weight_range = MaxAbs(weight_c_min[c], weight_c_max[c]); - weight_scales->at(c) = int8_range / weight_range; + weight_scales->at(c) = kInt8Range / weight_range; const DType *fp_ptr = weight_ptr + c * offset; int8_t *quan_ptr = quan_weight_ptr + c * offset; for (size_t k = 0; k < offset; ++k) { @@ -125,7 +125,7 @@ static void QuantizeConvWeightBias(NDArray *weight, NDArray *bias, } weight_scales->resize(1); DType weight_range = MaxAbs(total_min, total_max); - weight_scales->at(0) = int8_range / weight_range; + weight_scales->at(0) = kInt8Range / weight_range; #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) for (int c = 0; c < static_cast(channel); ++c) { const DType *fp_ptr = weight_ptr + c * offset; @@ -327,7 +327,7 @@ void SgMKLDNNConvOperator::Forward(const OpContext &ctx, // Quantize weight and bias. if (mkldnn_param.quantized) { CHECK(data.dtype() == mshadow::kInt8 || data.dtype() == mshadow::kUint8); - auto data_range = (data.dtype() == mshadow::kInt8) ? int8_range : uint8_range; + auto data_range = (data.dtype() == mshadow::kInt8) ? kInt8Range : kUint8Range; float data_scale = data_range / MaxAbs(cached_data_min_, cached_data_max_); MSHADOW_REAL_TYPE_SWITCH(cached_weight_.dtype(), DType, { QuantizeConvWeightBias(&cached_weight_, &cached_bias_, @@ -346,12 +346,12 @@ void SgMKLDNNConvOperator::Forward(const OpContext &ctx, LOG(FATAL) << "Can't handle negetive value for QuantizeData"; } if (mkldnn_param.with_sum) { - auto quantized_sum_range = cached_sum_min_ < 0 ? int8_range : uint8_range; + auto quantized_sum_range = cached_sum_min_ < 0 ? kInt8Range : kUint8Range; sum_in_scale = quantized_sum_range / MaxAbs(cached_sum_min_, cached_sum_max_); } if (post_requantize) { quantized_out_range = - IsOutputUInt8(mkldnn_param) ? uint8_range : int8_range; + IsOutputUInt8(mkldnn_param) ? kUint8Range : kInt8Range; out_range = MaxAbs(*out_min_ptr, *out_max_ptr); output_scale = quantized_out_range / out_range; full_conv_param.requantize_scales.resize(channel); @@ -626,8 +626,12 @@ std::vector> SgMKLDNNConvInplaceOption( } nnvm::NodePtr SgMKLDNNConvQuantizedOp(const NodeAttrs& attrs) { + auto const ¶m = nnvm::get(attrs.parsed); nnvm::NodePtr node = nnvm::Node::Create(); node->attrs.op = Op::Get("_sg_mkldnn_conv"); + CHECK_EQ(param.full_conv_param.conv_param.kernel.ndim(), 2U) + << "Quantized Convolution of MKL-DNN only supports 2D kernel currently." + << "Please exclude this layer from the quantized model."; node->attrs.name = "quantized_" + attrs.name; node->attrs.dict = attrs.dict; node->attrs.dict["quantized"] = "true"; diff --git a/src/operator/subgraph/partition_graph.cc b/src/operator/subgraph/partition_graph.cc index d1263201a..2605f9354 100644 --- a/src/operator/subgraph/partition_graph.cc +++ b/src/operator/subgraph/partition_graph.cc @@ -53,7 +53,6 @@ namespace sg { // sg stands for subgraph struct SimpleNode; using SimpleNodePtr = std::shared_ptr; - /*! * \brief Node of the undirected graph which replicates the network structures * of the computational graph. It is used to ease the graph traversal for finding @@ -244,8 +243,8 @@ bool LabelSubgraph(const Graph& g, s.push(descendant); size_t count = 0; while (!s.empty()) { - CHECK_LT(count, indexed_graph.num_nodes()) - << "Finding ancestor failed. There is probably a loop in the graph"; + CHECK_LT(count, indexed_graph.num_nodes()) << "Finding ancestor failed. There is probably" + " a loop in the graph"; ++count; const nnvm::Node* top = s.top(); s.pop(); @@ -509,7 +508,7 @@ void DeduplicateEntries(std::vector* entries) { */ void FindInputEntries( const Graph& g, const std::vector& simple_nodes, - const std::vector& subgraph_nodes, + const std::vector& subgraph_nodes, const std::unordered_map& entry_top_order_map, std::vector* input_entries, @@ -522,6 +521,7 @@ void FindInputEntries( } else { CHECK_EQ(subgraph_nodes[i]->label, label); } + auto& inputs = subgraph_nodes[i]->node->inputs; for (size_t j = 0; j < inputs.size(); ++j) { auto& e = inputs[j]; @@ -556,11 +556,11 @@ void FindInputEntries( * \param entry_top_order_map mapping entry pointer to its top sorted position * \param output_entries output entries of the subgraph */ -void FindOutputEntries( - Graph* g, const std::vector& simple_nodes, - const std::vector& subgraph_nodes, - const std::unordered_map& - entry_top_order_map, +void FindOutputEntries(Graph* g, + const std::vector& simple_nodes, + const std::vector& subgraph_nodes, + const std::unordered_map& + entry_top_order_map, std::vector* output_entries, nnvm::NodeEntryMap>* output_entry_map) { if (subgraph_nodes.empty()) return; @@ -631,7 +631,7 @@ void CutGraphInputs( const std::vector& input_entries, std::vector* orig_entries, const nnvm::NodeEntryMap>& input_entry_map, - const bool skip_var = false) { + const bool skip_var = false) { orig_entries->resize(input_entries.size()); // map for creating unique var nodes for deduplicating entries from the same node std::unordered_map name_count_map; @@ -643,7 +643,6 @@ void CutGraphInputs( } orig_entries->at(i) = *e; - nnvm::Symbol sym; sym.outputs.push_back(*e); const auto output_names = sym.ListOutputNames(); @@ -658,7 +657,7 @@ void CutGraphInputs( nnvm::NodePtr n = nnvm::CreateVariableNode(var_name + std::to_string(name_count_map[var_name])); for (auto* entry : input_entry_map.at(*e)) { *entry = nnvm::NodeEntry{n, 0, 0}; - } + } } } @@ -774,7 +773,6 @@ void CreateSubgraphNode(Graph* g, for (size_t i = 0; i < output_entries.size(); ++i) { sym.outputs[i] = *output_entries[i]; } - const SubgraphPropertyPtr& subg_prop = g->GetAttr("subgraph_property"); nnvm::NodePtr n; if (!subg_prop->NeedGraphAttrs()) { @@ -787,6 +785,7 @@ void CreateSubgraphNode(Graph* g, n = subg_prop->CreateSubgraphNode(subgraph, subgraph_id); } subgraphs->insert({n.get(), sym}); + // Connect the external nodes to the subgraph node. subg_prop->ConnectSubgraphOutputs(n, &output_entries, output_entry_map); subg_prop->ConnectSubgraphInputs(n, &input_entries, &orig_input_entries); @@ -795,7 +794,6 @@ void CreateSubgraphNode(Graph* g, for (size_t i = 0; i < n->inputs.size(); ++i) { auto& e = n->inputs[i]; // update entry_top_order_map with newly created orig_input_entries - auto it = entry_top_order_map->find(input_entries[i]); CHECK(it != entry_top_order_map->end()); entry_top_order_map->emplace(&e, it->second); diff --git a/src/operator/tensor/broadcast_reduce-inl.h b/src/operator/tensor/broadcast_reduce-inl.h index 39c68bd62..141d2fb83 100644 --- a/src/operator/tensor/broadcast_reduce-inl.h +++ b/src/operator/tensor/broadcast_reduce-inl.h @@ -19,8 +19,8 @@ /*! * Copyright (c) 2015-2017 by Contributors - * \file broadcast_reduce_kernel.h - * \brief Function definition of elementwise unary operators + * \file broadcast_reduce-inl.h + * \brief CPU-specific Function definition of broadcast and reduce operators */ #ifndef MXNET_OPERATOR_TENSOR_BROADCAST_REDUCE_INL_H_ #define MXNET_OPERATOR_TENSOR_BROADCAST_REDUCE_INL_H_ @@ -53,14 +53,14 @@ MSHADOW_XINLINE Shape calc_stride(const Shape& shape) { } template -MSHADOW_XINLINE void unravel_dot(const int idx, const Shape& shape, - const Shape& stridej, const Shape& stridek, int* j, int* k) { +MSHADOW_XINLINE void unravel_dot(const index_t idx, const Shape& shape, + const Shape& stridej, const Shape& stridek, index_t* j, index_t* k) { *j = 0; *k = 0; #pragma unroll - for (int i = ndim-1, idx_t = idx; i >=0; --i) { - const int tmp = idx_t / shape[i]; - const int coord = idx_t - tmp*shape[i]; + for (index_t i = ndim-1, idx_t = idx; i >=0; --i) { + const auto tmp = idx_t / shape[i]; + const auto coord = idx_t - tmp*shape[i]; *j += coord*stridej[i]; *k += coord*stridek[i]; idx_t = tmp; @@ -68,11 +68,11 @@ MSHADOW_XINLINE void unravel_dot(const int idx, const Shape& shape, } template -MSHADOW_XINLINE Shape unravel(const int idx, const Shape& shape) { +MSHADOW_XINLINE Shape unravel(const index_t idx, const Shape& shape) { Shape ret; #pragma unroll - for (int i = ndim-1, j = idx; i >=0; --i) { - int tmp = j / shape[i]; + for (index_t i = ndim-1, j = idx; i >=0; --i) { + auto tmp = j / shape[i]; ret[i] = j - tmp*shape[i]; j = tmp; } @@ -80,10 +80,10 @@ MSHADOW_XINLINE Shape unravel(const int idx, const Shape& shape) { } template -MSHADOW_XINLINE int ravel(const Shape& coord, const Shape& shape) { - int ret = 0; +MSHADOW_XINLINE index_t ravel(const Shape& coord, const Shape& shape) { + index_t ret = 0; #pragma unroll - for (int i = 0; i < ndim; ++i) { + for (index_t i = 0; i < ndim; ++i) { ret = ret * shape[i] + (shape[i] > 1) * coord[i]; } return ret; @@ -111,12 +111,12 @@ MSHADOW_XINLINE int diff(const Shape& small, const Shape& big, Shape } template -MSHADOW_XINLINE int unravel_dot(const int idx, const Shape& shape, +MSHADOW_XINLINE index_t unravel_dot(const index_t idx, const Shape& shape, const Shape& stride) { - int ret = 0; + index_t ret = 0; #pragma unroll - for (int i = ndim-1, j = idx; i >=0; --i) { - int tmp = j / shape[i]; + for (index_t i = ndim-1, j = idx; i >=0; --i) { + auto tmp = j / shape[i]; ret += (j - tmp*shape[i])*stride[i]; j = tmp; } @@ -124,8 +124,8 @@ MSHADOW_XINLINE int unravel_dot(const int idx, const Shape& shape, } template -MSHADOW_XINLINE int dot(const Shape& coord, const Shape& stride) { - int ret = 0; +MSHADOW_XINLINE index_t dot(const Shape& coord, const Shape& stride) { + index_t ret = 0; #pragma unroll for (int i = 0; i < ndim; ++i) ret += coord[i] * stride[i]; @@ -142,27 +142,27 @@ MSHADOW_XINLINE void assign(DType* dst, const bool addto, const DType src) { } template -MSHADOW_XINLINE void binary_broadcast_assign(const int idx, const bool addto, +MSHADOW_XINLINE void binary_broadcast_assign(const index_t idx, const bool addto, const DType* __restrict lhs, const DType* __restrict rhs, DType* out, const Shape& lshape, const Shape& rshape, const Shape& oshape) { const Shape coord = unravel(idx, oshape); - const int j = ravel(coord, lshape); - const int k = ravel(coord, rshape); + const index_t j = ravel(coord, lshape); + const index_t k = ravel(coord, rshape); assign(&out[idx], addto, OP::Map(lhs[j], rhs[k])); } template -MSHADOW_XINLINE void seq_reduce_assign(const int idx, const int M, const bool addto, +MSHADOW_XINLINE void seq_reduce_assign(const index_t idx, const size_t M, const bool addto, const DType* __restrict big, DType *small, const Shape& bshape, const Shape& sshape, const Shape& rshape, const Shape& rstride) { Shape coord = unravel(idx, sshape); - int j = ravel(coord, bshape); + index_t j = ravel(coord, bshape); DType val, residual; Reducer::SetInitValue(val, residual); - for (int k = 0; k < M; ++k) { + for (size_t k = 0; k < M; ++k) { coord = unravel(k, rshape); Reducer::Reduce(val, OP::Map(big[j + dot(coord, rstride)]), residual); } @@ -176,10 +176,10 @@ MSHADOW_XINLINE void seq_reduce_assign(const int idx, const int M, const bool ad #else template -void binary_broadcast_compute(const int N, const bool addto, const DType *lhs, +void binary_broadcast_compute(const size_t N, const bool addto, const DType *lhs, const DType *rhs, DType *out, const Shape lshape, const Shape rshape, const Shape oshape) { - for (int idx = 0; idx < N; ++idx) { + for (size_t idx = 0; idx < N; ++idx) { binary_broadcast_assign(idx, addto, lhs, rhs, out, lshape, rshape, oshape); } } @@ -188,26 +188,26 @@ template void BinaryBroadcastComputeImpl(Stream *s, const OpReqType req, const TBlob& lhs, const TBlob& rhs, const TBlob& out) { if (req == kNullOp) return; - int N = out.shape_.Size(); + size_t N = out.shape_.Size(); binary_broadcast_compute(N, req == kAddTo, lhs.dptr(), rhs.dptr(), out.dptr(), lhs.shape_.get(), rhs.shape_.get(), out.shape_.get()); } template -void seq_reduce_compute(const int N, const int M, const bool addto, +void seq_reduce_compute(const size_t N, const size_t M, const bool addto, const DType *big, DType *small, const Shape bshape, const Shape sshape, const Shape rshape, const Shape rstride) { #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) - for (int idx = 0; idx < N; ++idx) { + for (index_t idx = 0; idx < static_cast(N); ++idx) { seq_reduce_assign(idx, M, addto, big, small, bshape, sshape, rshape, rstride); } } template -void seq_reduce_compute_extra_mem(const int N, const int M, const bool addto, +void seq_reduce_compute_extra_mem(const size_t N, const size_t M, const bool addto, const DType* big, DType* small, const Shape bshape, const Shape sshape, @@ -215,12 +215,12 @@ void seq_reduce_compute_extra_mem(const int N, const int M, const bool addto, const Shape rstride, const index_t* ws_dptr) { #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) - for (int idx = 0; idx < N; ++idx) { + for (index_t idx = 0; idx < static_cast(N); ++idx) { Shape coord = unravel(idx, sshape); - int j = ravel(coord, bshape); + index_t j = ravel(coord, bshape); DType val, residual; Reducer::SetInitValue(val, residual); - for (int k = 0; k < M; ++k) { + for (size_t k = 0; k < M; ++k) { Reducer::Reduce(val, OP::Map(big[j + ws_dptr[k]]), residual); } assign(&small[idx], addto, val); @@ -233,7 +233,7 @@ void Reduce(Stream* s, const TBlob& small, const OpReqType req, if (req == kNullOp) return; Shape rshape, rstride; diff(small.shape_.get(), big.shape_.get(), &rshape, &rstride); - int N = small.shape_.Size(), M = rshape.Size(); + size_t N = small.shape_.Size(), M = rshape.Size(); seq_reduce_compute( N, M, req == kAddTo, big.dptr(), small.dptr(), big.shape_.get(), small.shape_.get(), rshape, rstride); @@ -247,9 +247,9 @@ void ReduceWithExtraMem(Stream* s, const TBlob& small, const OpReqType req, Shape rshape, rstride; diff(small.shape_.get(), big.shape_.get(), &rshape, &rstride); index_t* ws_dptr = reinterpret_cast(workspace.dptr_); - int N = small.shape_.Size(), M = rshape.Size(); + size_t N = small.shape_.Size(), M = rshape.Size(); #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) - for (int k = 0; k < M; k++) { + for (index_t k = 0; k < static_cast(M); k++) { Shape coord = unravel(k, rshape); ws_dptr[k] = dot(coord, rstride); } @@ -272,7 +272,7 @@ size_t ReduceWorkspaceSize(Stream *s, const TShape& small, const OpReqType } template -MSHADOW_XINLINE void seq_reduce_assign(const int idx, const int M, const bool addto, +MSHADOW_XINLINE void seq_reduce_assign(const index_t idx, const size_t M, const bool addto, const DType* __restrict big, const DType* __restrict lhs, const DType* __restrict rhs, DType *small, const Shape& big_shape, const Shape& lhs_shape0, @@ -282,20 +282,20 @@ MSHADOW_XINLINE void seq_reduce_assign(const int idx, const int M, const bool ad const Shape& rstride, const Shape& lhs_stride, const Shape& rhs_stride) { Shape coord = unravel(idx, small_shape); - const int idx_big0 = ravel(coord, big_shape); - const int idx_lhs0 = ravel(coord, lhs_shape0); - const int idx_rhs0 = ravel(coord, rhs_shape0); + const index_t idx_big0 = ravel(coord, big_shape); + const index_t idx_lhs0 = ravel(coord, lhs_shape0); + const index_t idx_rhs0 = ravel(coord, rhs_shape0); DType val, residual; Reducer::SetInitValue(val, residual); - for (int k = 0; k < M; ++k) { + for (size_t k = 0; k < M; ++k) { Shape coord_big = unravel(k, rshape); - int idx_big = idx_big0 + dot(coord_big, rstride); + index_t idx_big = idx_big0 + dot(coord_big, rstride); Shape coord_lhs = unravel(k, lhs_shape); - int idx_lhs = idx_lhs0 + dot(coord_lhs, lhs_stride); + index_t idx_lhs = idx_lhs0 + dot(coord_lhs, lhs_stride); Shape coord_rhs = unravel(k, rhs_shape); - int idx_rhs = idx_rhs0 + dot(coord_rhs, rhs_stride); + index_t idx_rhs = idx_rhs0 + dot(coord_rhs, rhs_stride); Reducer::Reduce(val, OP1::Map(big[idx_big], OP2::Map(lhs[idx_lhs], rhs[idx_rhs])), residual); } @@ -304,7 +304,7 @@ MSHADOW_XINLINE void seq_reduce_assign(const int idx, const int M, const bool ad } template -void seq_reduce_compute(const int N, const int M, const bool addto, +void seq_reduce_compute(const size_t N, const size_t M, const bool addto, const DType *big, const DType *lhs, const DType *rhs, DType *small, const Shape big_shape, const Shape small_shape, const Shape rshape, const Shape rstride, @@ -312,7 +312,7 @@ void seq_reduce_compute(const int N, const int M, const bool addto, const Shape rhs_shape, const Shape rhs_stride, const Shape& lhs_shape0, const Shape& rhs_shape0) { #pragma omp parallel for num_threads(engine::OpenMP::Get()->GetRecommendedOMPThreadCount()) - for (int idx = 0; idx < N; ++idx) { + for (index_t idx = 0; idx < static_cast(N); ++idx) { seq_reduce_assign(idx, M, addto, big, lhs, rhs, small, big_shape, lhs_shape0, rhs_shape0, small_shape, rshape, lhs_shape, rhs_shape, rstride, lhs_stride, rhs_stride); @@ -326,8 +326,8 @@ void Reduce(Stream *s, const TBlob& small, const OpReqType req, if (req == kNullOp) return; Shape rshape, rstride; diff(small.shape_.get(), big.shape_.get(), &rshape, &rstride); - int N = small.shape_.Size(); - int M = rshape.Size(); + size_t N = small.shape_.Size(); + size_t M = rshape.Size(); Shape lhs_shape, lhs_stride; diff(small.shape_.get(), lhs.shape_.get(), &lhs_shape, &lhs_stride); diff --git a/src/operator/tensor/broadcast_reduce_op.h b/src/operator/tensor/broadcast_reduce_op.h index 0944d255a..1edcb5a74 100644 --- a/src/operator/tensor/broadcast_reduce_op.h +++ b/src/operator/tensor/broadcast_reduce_op.h @@ -19,8 +19,8 @@ /*! * Copyright (c) 2015 by Contributors - * \file elementwise_unary_op-inl.h - * \brief Function definition of elementwise unary operators + * \file broadcast_reduce_op.h + * \brief Function definition of broadcast and reduce operators */ #ifndef MXNET_OPERATOR_TENSOR_BROADCAST_REDUCE_OP_H_ #define MXNET_OPERATOR_TENSOR_BROADCAST_REDUCE_OP_H_ diff --git a/src/operator/tensor/broadcast_reduce_op_index.cc b/src/operator/tensor/broadcast_reduce_op_index.cc index 969c23af9..c18a8bcf9 100644 --- a/src/operator/tensor/broadcast_reduce_op_index.cc +++ b/src/operator/tensor/broadcast_reduce_op_index.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2016 by Contributors * \file broadcast_reduce_op_index.cc - * \brief CPU Implementation of broadcast and reduce functions. + * \brief CPU Implementation of broadcast and reduce functions based on index. */ #include "./broadcast_reduce_op.h" diff --git a/src/operator/tensor/broadcast_reduce_op_index.cu b/src/operator/tensor/broadcast_reduce_op_index.cu index 0d7b29dfa..c8f2b0755 100644 --- a/src/operator/tensor/broadcast_reduce_op_index.cu +++ b/src/operator/tensor/broadcast_reduce_op_index.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file broadcast_reduce_op.cu - * \brief GPU Implementation of broadcast and reduce functions. + * \file broadcast_reduce_op_index.cu + * \brief GPU Implementation of broadcast and reduce functions based on index. */ #include "./broadcast_reduce_op.h" diff --git a/src/operator/tensor/broadcast_reduce_op_value.cc b/src/operator/tensor/broadcast_reduce_op_value.cc index c3bc9cfd3..1fdbb8920 100644 --- a/src/operator/tensor/broadcast_reduce_op_value.cc +++ b/src/operator/tensor/broadcast_reduce_op_value.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file broadcast_reduce_op.cc - * \brief CPU Implementation of broadcast and reduce functions. + * \file broadcast_reduce_op_value.cc + * \brief CPU Implementation of broadcast and reduce functions based on value. */ #include "./broadcast_reduce_op.h" diff --git a/src/operator/tensor/broadcast_reduce_op_value.cu b/src/operator/tensor/broadcast_reduce_op_value.cu index 12ddcdd94..881f52090 100644 --- a/src/operator/tensor/broadcast_reduce_op_value.cu +++ b/src/operator/tensor/broadcast_reduce_op_value.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file broadcast_reduce_op.cu - * \brief GPU Implementation of broadcast and reduce functions. + * \file broadcast_reduce_op_value.cu + * \brief GPU Implementation of broadcast and reduce functions based on value. */ #include "./broadcast_reduce_op.h" diff --git a/src/operator/tensor/cast_storage-inl.cuh b/src/operator/tensor/cast_storage-inl.cuh index a5b4df538..39e522664 100644 --- a/src/operator/tensor/cast_storage-inl.cuh +++ b/src/operator/tensor/cast_storage-inl.cuh @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file cast_storage-inl.cuh - * \brief implementation of cast_storage op on GPU + * \brief GPU implementation of cast_storage op */ #ifndef MXNET_OPERATOR_TENSOR_CAST_STORAGE_INL_CUH_ #define MXNET_OPERATOR_TENSOR_CAST_STORAGE_INL_CUH_ diff --git a/src/operator/tensor/cast_storage-inl.h b/src/operator/tensor/cast_storage-inl.h index f905bf8f7..cdb624631 100644 --- a/src/operator/tensor/cast_storage-inl.h +++ b/src/operator/tensor/cast_storage-inl.h @@ -19,7 +19,7 @@ /*! * \file cast_storage-inl.h - * \brief cast_storage implementation for dense and sparse tensors + * \brief CPU implementation of cast_storage operator for dense and sparse tensors */ #ifndef MXNET_OPERATOR_TENSOR_CAST_STORAGE_INL_H_ #define MXNET_OPERATOR_TENSOR_CAST_STORAGE_INL_H_ diff --git a/src/operator/tensor/control_flow_op.cu b/src/operator/tensor/control_flow_op.cu index 7c34a6536..6371b62d4 100644 --- a/src/operator/tensor/control_flow_op.cu +++ b/src/operator/tensor/control_flow_op.cu @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file control_flow_op.cu - * \brief + * \brief GPU implementation of control flow */ #include "./control_flow_op.h" diff --git a/src/operator/tensor/control_flow_op.h b/src/operator/tensor/control_flow_op.h index e9aa9f63f..9d0e8cf90 100644 --- a/src/operator/tensor/control_flow_op.h +++ b/src/operator/tensor/control_flow_op.h @@ -19,7 +19,7 @@ /*! * Copyright (c) 2017 by Contributors - * \file control_flow.h + * \file control_flow_op.h * \brief Function definitions of operators for controlling flow */ #ifndef MXNET_OPERATOR_TENSOR_CONTROL_FLOW_OP_H_ @@ -46,7 +46,7 @@ struct where { // DType is the output data type // CType is condition data type template - MSHADOW_XINLINE static void Map(int i, DType* out, const CType* cond, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const CType* cond, const DType* x, const DType* y) { KERNEL_ASSIGN(out[i], req, (0 != cond[i]? x[i] : y[i])); } @@ -64,7 +64,7 @@ struct where_csr { // CType is condition data type // i is for i-th row in the output template - MSHADOW_XINLINE static void Map(int i, DType* out, const IType* cond_idx, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const IType* cond_idx, const IType* cond_indptr, const CType* cond_data, const nnvm::dim_t num_cols, const DType* x) { using nnvm::dim_t; @@ -92,8 +92,8 @@ struct where_batch { // DType is the output data type // CType is the condition data type template - MSHADOW_XINLINE static void Map(int i, DType* out, const CType* cond, - const DType* x, const DType* y, int M) { + MSHADOW_XINLINE static void Map(index_t i, DType* out, const CType* cond, + const DType* x, const DType* y, index_t M) { KERNEL_ASSIGN(out[i], req, (0 != cond[i/M]? x[i] : y[i])); } }; @@ -109,7 +109,7 @@ struct where_backward { // DType is the output data type // CType is condition data type template - MSHADOW_XINLINE static void Map(int i, DType* grad_out, + MSHADOW_XINLINE static void Map(index_t i, DType* grad_out, const DType* grad_in, const CType* cond) { KERNEL_ASSIGN(grad_out[i], req, @@ -130,7 +130,7 @@ struct where_backward_csr { // CType is condition data type // IType is condition aux data type template - MSHADOW_XINLINE static void Map(int i, DType* grad_out, + MSHADOW_XINLINE static void Map(index_t i, DType* grad_out, const DType* grad_in, const CType* cond_data, const IType* cond_idx, @@ -161,9 +161,9 @@ struct where_batch_backward { // DType is the output data type // CType is condition data type template - MSHADOW_XINLINE static void Map(int i, DType* grad_out, + MSHADOW_XINLINE static void Map(index_t i, DType* grad_out, const DType* grad_in, - const CType* cond, int M) { + const CType* cond, index_t M) { KERNEL_ASSIGN(grad_out[i], req, ((0 == cond[i/M])^negate)? grad_in[i] : static_cast(0)); } diff --git a/src/operator/tensor/diag_op-inl.h b/src/operator/tensor/diag_op-inl.h index deab2569e..23123cfab 100644 --- a/src/operator/tensor/diag_op-inl.h +++ b/src/operator/tensor/diag_op-inl.h @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file diag_op-inl.h -* \brief CPU Implementation of the diag op +* \brief Function definition of the diag op * \author Istvan Fehervari, Zhijingcheng Yu */ diff --git a/src/operator/tensor/diag_op.cc b/src/operator/tensor/diag_op.cc index cd5be9d0f..9dcdb63d8 100644 --- a/src/operator/tensor/diag_op.cc +++ b/src/operator/tensor/diag_op.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file diag_op.cc -* \brief +* \brief CPU implementation of diag operator * \author Istvan Fehervari, Zhijingcheng Yu */ diff --git a/src/operator/tensor/diag_op.cu b/src/operator/tensor/diag_op.cu index a3928f763..c7db3d997 100644 --- a/src/operator/tensor/diag_op.cu +++ b/src/operator/tensor/diag_op.cu @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file diag_op.cu -* \brief GPU Implementation of the diag op +* \brief GPU Implementation of the diag operator * \author Istvan Fehervari */ diff --git a/src/operator/tensor/dot-inl.cuh b/src/operator/tensor/dot-inl.cuh index 6c94ad1ec..d6fed4af8 100644 --- a/src/operator/tensor/dot-inl.cuh +++ b/src/operator/tensor/dot-inl.cuh @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file dot-inl.cuh - * \brief implementation of matrix dot op on GPU + * \brief GPU specific function definition of matrix dot operator */ #ifndef MXNET_OPERATOR_TENSOR_DOT_INL_CUH_ #define MXNET_OPERATOR_TENSOR_DOT_INL_CUH_ diff --git a/src/operator/tensor/dot-inl.h b/src/operator/tensor/dot-inl.h index 5e469108e..69c35f85c 100644 --- a/src/operator/tensor/dot-inl.h +++ b/src/operator/tensor/dot-inl.h @@ -791,6 +791,14 @@ inline void DotCsrDnsDnsImpl(const OpContext& ctx, s, num_threads, data_out.dptr()); } num_threads = mxnet_op::get_num_threads(data_out.shape_[0]); + bool dynamic = false; + const dim_t large_matrix_threshold = 1024 * 10; + if (data_out.shape_[0] > large_matrix_threshold) { + dynamic = true; + // each unit of work processes at least 1024 elements in the output + const dim_t unit_work_per_thread = 1024; + num_threads = data_out.Size() / unit_work_per_thread; + } dim_t seg_len = (data_out.shape_[0] + num_threads - 1) / num_threads; if (trans_lhs) { mxnet_op::Kernel::Launch(s, num_threads, @@ -798,10 +806,17 @@ inline void DotCsrDnsDnsImpl(const OpContext& ctx, col_idx_l.dptr(), data_r.dptr(), seg_len, lhs.shape()[0], data_out.shape_[0], data_out.shape_[1]); } else { - mxnet_op::Kernel::Launch(s, num_threads, - data_out.dptr(), data_l.dptr(), indptr_l.dptr(), - col_idx_l.dptr(), data_r.dptr(), seg_len, - data_out.shape_[0], data_out.shape_[1]); + if (dynamic) { + mxnet_op::Kernel::LaunchDynamic(s, num_threads, + data_out.dptr(), data_l.dptr(), indptr_l.dptr(), + col_idx_l.dptr(), data_r.dptr(), seg_len, + data_out.shape_[0], data_out.shape_[1]); + } else { + mxnet_op::Kernel::Launch(s, num_threads, + data_out.dptr(), data_l.dptr(), indptr_l.dptr(), + col_idx_l.dptr(), data_r.dptr(), seg_len, + data_out.shape_[0], data_out.shape_[1]); + } } }); }); diff --git a/src/operator/tensor/elemwise_binary_broadcast_op-inl.cuh b/src/operator/tensor/elemwise_binary_broadcast_op-inl.cuh index 199857617..d02004d75 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op-inl.cuh +++ b/src/operator/tensor/elemwise_binary_broadcast_op-inl.cuh @@ -17,6 +17,10 @@ * under the License. */ +/*! + * \file elemwise_binary_broadcast_op-inl.cuh + * \brief CUDA specific Function definition of elementwise binary broadcast operators + */ #ifndef MXNET_OPERATOR_TENSOR_ELEMWISE_BINARY_BROADCAST_OP_CUH_ #define MXNET_OPERATOR_TENSOR_ELEMWISE_BINARY_BROADCAST_OP_CUH_ #include diff --git a/src/operator/tensor/elemwise_binary_broadcast_op.h b/src/operator/tensor/elemwise_binary_broadcast_op.h index 8fd2cb4c5..304422038 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op.h +++ b/src/operator/tensor/elemwise_binary_broadcast_op.h @@ -19,8 +19,8 @@ /*! * Copyright (c) 2015 by Contributors - * \file elementwise_binary_broadcast_op.h - * \brief Function definition of elementwise unary operators + * \file elemwise_binary_broadcast_op.h + * \brief Function definition of elementwise binary broadcast operators */ #ifndef MXNET_OPERATOR_TENSOR_ELEMWISE_BINARY_BROADCAST_OP_H_ #define MXNET_OPERATOR_TENSOR_ELEMWISE_BINARY_BROADCAST_OP_H_ @@ -190,7 +190,7 @@ namespace mxnet_op { template struct binary_broadcast_kernel { /*! \brief Map function for binary_broadcast_kernel */ - MSHADOW_XINLINE static void Map(int base, int length, OpReqType req, + MSHADOW_XINLINE static void Map(index_t base, index_t length, OpReqType req, const Shape &lstride, const Shape &rstride, const Shape &oshape, DType *lhs, DType *rhs, DType *out) { @@ -199,7 +199,7 @@ struct binary_broadcast_kernel { auto ridx = static_cast(dot(coord, rstride)); KERNEL_ASSIGN(out[base], req, OP::Map(lhs[lidx], rhs[ridx])); // starts from 1 to avoid extra inc at end of loop - for (int i = 1; i < length; ++i) { + for (index_t i = 1; i < length; ++i) { inc(&coord, oshape, &lidx, lstride, &ridx, rstride); // When tuning, don't actually run the op, since it's not going to be tuned against // the actual op we'll eventually be using @@ -208,7 +208,7 @@ struct binary_broadcast_kernel { } /*! \brief Map function for binary_broadcast_kernel */ - MSHADOW_XINLINE static void Map(int base, int length, OpReqType req, + MSHADOW_XINLINE static void Map(index_t base, index_t length, OpReqType req, const Shape &lstride, const Shape &rstride, const Shape &oshape, DType lhs, DType *rhs, DType *out) { @@ -217,7 +217,7 @@ struct binary_broadcast_kernel { auto ridx = static_cast(dot(coord, rstride)); KERNEL_ASSIGN(out[base], req, OP::Map(lhs, rhs[ridx])); // starts from 1 to avoid extra inc at end of loop - for (int i = 1; i < length; ++i) { + for (index_t i = 1; i < length; ++i) { inc(&coord, oshape, &lidx, lstride, &ridx, rstride); // When tuning, don't actually run the op, since it's not going to be tuned against // the actual op we'll eventually be using @@ -238,7 +238,7 @@ struct csr_dns_csr_broadcast_kernel { * \param out ptr to the data buffer of the result csr matrix */ template - MSHADOW_XINLINE static void Map(int row, const DType *csr_data, const CType *csr_indices, + MSHADOW_XINLINE static void Map(index_t row, const DType *csr_data, const CType *csr_indices, const RType *csr_indptr, const DType *dns, DType *out) { const nnvm::dim_t curr_row_i = csr_indptr[row]; const nnvm::dim_t next_row_i = csr_indptr[row + 1]; @@ -257,7 +257,7 @@ struct csr_dns_csr_broadcast_kernel { * \param nnz number of non-zero elements in input csr matrix */ template - MSHADOW_XINLINE static void Map(int i, const DType *csr_data, const DType* scalar_ptr, + MSHADOW_XINLINE static void Map(index_t i, const DType *csr_data, const DType* scalar_ptr, DType *out, const nnvm::dim_t nnz) { const DType scale = scalar_ptr[0]; if (i < nnz) { @@ -269,7 +269,7 @@ struct csr_dns_csr_broadcast_kernel { template struct csr_dns_map_kernel { template - MSHADOW_XINLINE static void Map(int row, const DType *csr_data, const CType *csr_indices, + MSHADOW_XINLINE static void Map(index_t row, const DType *csr_data, const CType *csr_indices, const RType *csr_indptr, DType *out, const nnvm::dim_t num_rows, const nnvm::dim_t num_cols) { if (row < num_rows) { diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_basic.cc b/src/operator/tensor/elemwise_binary_broadcast_op_basic.cc index 7301eced1..6d3ce933a 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_basic.cc +++ b/src/operator/tensor/elemwise_binary_broadcast_op_basic.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_broadcast_op_basic.cc + * \brief CPU Implementation of basic functions for elementwise binary broadcast operator. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_basic.cu b/src/operator/tensor/elemwise_binary_broadcast_op_basic.cu index a00330d24..aa0850ac5 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_basic.cu +++ b/src/operator/tensor/elemwise_binary_broadcast_op_basic.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_broadcast_op_basic.cu + * \brief GPU Implementation of basic functions for elementwise binary broadcast operator. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_extended.cc b/src/operator/tensor/elemwise_binary_broadcast_op_extended.cc index 8fc3c48b1..842007eb4 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_extended.cc +++ b/src/operator/tensor/elemwise_binary_broadcast_op_extended.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2016 by Contributors * \file elemwise_binary_broadcast_op_extended.cc - * \brief CPU Implementation of unary function. + * \brief CPU Implementation of extended functions for elementwise binary broadcast operator. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_extended.cu b/src/operator/tensor/elemwise_binary_broadcast_op_extended.cu index 27a764cd9..e8e79f726 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_extended.cu +++ b/src/operator/tensor/elemwise_binary_broadcast_op_extended.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_broadcast_op_extended.cu + * \brief GPU Implementation of extended functions for elementwise binary broadcast operator. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_logic.cc b/src/operator/tensor/elemwise_binary_broadcast_op_logic.cc index 3cb3ba33b..cd433e00a 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_logic.cc +++ b/src/operator/tensor/elemwise_binary_broadcast_op_logic.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2016 by Contributors * \file elemwise_binary_broadcast_op_logic.cc - * \brief CPU Implementation of unary function. + * \brief CPU Implementation of elementwise binary broadcast logical operators. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_broadcast_op_logic.cu b/src/operator/tensor/elemwise_binary_broadcast_op_logic.cu index d6b01aad7..4bec07b70 100644 --- a/src/operator/tensor/elemwise_binary_broadcast_op_logic.cu +++ b/src/operator/tensor/elemwise_binary_broadcast_op_logic.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_broadcast_op_logic.cu + * \brief GPU Implementation of elementwise binary broadcast logical operators. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_op-inl.h b/src/operator/tensor/elemwise_binary_op-inl.h index 72a02ff5f..42f907f71 100644 --- a/src/operator/tensor/elemwise_binary_op-inl.h +++ b/src/operator/tensor/elemwise_binary_op-inl.h @@ -18,7 +18,7 @@ */ /*! - * \file elemwise_binary_op.h + * \file elemwise_binary_op-inl.h * \brief Function definition of elementwise binary operators */ #ifndef MXNET_OPERATOR_TENSOR_ELEMWISE_BINARY_OP_INL_H_ diff --git a/src/operator/tensor/elemwise_binary_op.cc b/src/operator/tensor/elemwise_binary_op.cc index 9ccbacc2f..173d65fa0 100644 --- a/src/operator/tensor/elemwise_binary_op.cc +++ b/src/operator/tensor/elemwise_binary_op.cc @@ -16,6 +16,13 @@ * specific language governing permissions and limitations * under the License. */ + +/*! + * Copyright (c) 2016 by Contributors + * \file elemwise_binary_op.cc + * \brief CPU implementation of elementwise binary operators + */ + #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_op_basic.cc b/src/operator/tensor/elemwise_binary_op_basic.cc index 339290df8..f7599adcd 100644 --- a/src/operator/tensor/elemwise_binary_op_basic.cc +++ b/src/operator/tensor/elemwise_binary_op_basic.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_op_basic.cc + * \brief CPU Implementation of basic elementwise binary broadcast operators */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op-inl.h" diff --git a/src/operator/tensor/elemwise_binary_op_basic.cu b/src/operator/tensor/elemwise_binary_op_basic.cu index 981e7aba9..f88b8ebd6 100644 --- a/src/operator/tensor/elemwise_binary_op_basic.cu +++ b/src/operator/tensor/elemwise_binary_op_basic.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_op_basic.cu + * \brief GPU Implementation of basic elementwise binary broadcast operators */ #include #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_op_extended.cc b/src/operator/tensor/elemwise_binary_op_extended.cc index 26d429c53..53c0b5c05 100644 --- a/src/operator/tensor/elemwise_binary_op_extended.cc +++ b/src/operator/tensor/elemwise_binary_op_extended.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_op_extended.cc + * \brief CPU Implementation of extended functions for elementwise binary broadcast operator.. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op-inl.h" diff --git a/src/operator/tensor/elemwise_binary_op_extended.cu b/src/operator/tensor/elemwise_binary_op_extended.cu index 54eceb2f8..585d379b0 100644 --- a/src/operator/tensor/elemwise_binary_op_extended.cu +++ b/src/operator/tensor/elemwise_binary_op_extended.cu @@ -19,7 +19,7 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu + * \file elemwise_binary_op_extended.cu * \brief GPU Implementation of unary function. */ #include "./elemwise_unary_op.h" diff --git a/src/operator/tensor/elemwise_binary_op_logic.cc b/src/operator/tensor/elemwise_binary_op_logic.cc index 5c0b44299..a6f61c2f0 100644 --- a/src/operator/tensor/elemwise_binary_op_logic.cc +++ b/src/operator/tensor/elemwise_binary_op_logic.cc @@ -19,7 +19,7 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc + * \file elemwise_binary_op_logic.cc * \brief CPU Implementation of unary function. */ #include "./elemwise_unary_op.h" diff --git a/src/operator/tensor/elemwise_binary_op_logic.cu b/src/operator/tensor/elemwise_binary_op_logic.cu index 456622d75..e36e69711 100644 --- a/src/operator/tensor/elemwise_binary_op_logic.cu +++ b/src/operator/tensor/elemwise_binary_op_logic.cu @@ -19,7 +19,7 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu + * \file elemwise_binary_op_logic.cu * \brief GPU Implementation of unary function. */ #include "./elemwise_unary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc index 6118ddf19..1eb3da65e 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cc +++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_scalar_op_basic.cc + * \brief CPU Implementation of basic binary scalar functions. */ #include "../../common/utils.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu index 51b3866c8..3c8392056 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_basic.cu +++ b/src/operator/tensor/elemwise_binary_scalar_op_basic.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_scalar_op_basic.cu + * \brief CPU Implementation of basic binary scalar functions. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_extended.cc b/src/operator/tensor/elemwise_binary_scalar_op_extended.cc index a0c4149d5..dbe3c4f62 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_extended.cc +++ b/src/operator/tensor/elemwise_binary_scalar_op_extended.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_scalar_op_extended.cc + * \brief CPU Implementation of extended binary scalar functions. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_extended.cu b/src/operator/tensor/elemwise_binary_scalar_op_extended.cu index 96884724d..2bd52d7b9 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_extended.cu +++ b/src/operator/tensor/elemwise_binary_scalar_op_extended.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_scalar_op_extended.cu + * \brief GPU Implementation of extended binary scalar functions. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_logic.cc b/src/operator/tensor/elemwise_binary_scalar_op_logic.cc index 07473a154..17e76153e 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_logic.cc +++ b/src/operator/tensor/elemwise_binary_scalar_op_logic.cc @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cc - * \brief CPU Implementation of unary function. + * \file elemwise_binary_scalar_op_logic.cc + * \brief CPU Implementation of binary scalar logic functions. */ #include "./elemwise_unary_op.h" #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/elemwise_binary_scalar_op_logic.cu b/src/operator/tensor/elemwise_binary_scalar_op_logic.cu index 8ec8b7359..6c393e071 100644 --- a/src/operator/tensor/elemwise_binary_scalar_op_logic.cu +++ b/src/operator/tensor/elemwise_binary_scalar_op_logic.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2016 by Contributors - * \file elemwise_binary_scalar_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_binary_scalar_op_logic.cu + * \brief GPU Implementation of binary scalar logic functions. */ #include "elemwise_binary_scalar_op.h" diff --git a/src/operator/tensor/elemwise_scatter_op.cc b/src/operator/tensor/elemwise_scatter_op.cc index 959faf75b..dd6da0ce4 100644 --- a/src/operator/tensor/elemwise_scatter_op.cc +++ b/src/operator/tensor/elemwise_scatter_op.cc @@ -16,6 +16,11 @@ * specific language governing permissions and limitations * under the License. */ + +/*! + * \file elemwise_scatter_op.cc + * \brief CPU implementation of elementwise scatter operators + */ #include "./elemwise_binary_op-inl.h" #include "./elemwise_binary_scalar_op.h" #include "./elemwise_scatter_op.h" diff --git a/src/operator/tensor/elemwise_scatter_op.cu b/src/operator/tensor/elemwise_scatter_op.cu index acb7edf21..913aa9512 100644 --- a/src/operator/tensor/elemwise_scatter_op.cu +++ b/src/operator/tensor/elemwise_scatter_op.cu @@ -16,6 +16,11 @@ * specific language governing permissions and limitations * under the License. */ + + /*! + * \file elemwise_scatter_op.cu + * \brief GPU implementation of elementwise scatter operators + */ #include "./elemwise_binary_scalar_op.h" #include "./elemwise_scatter_op.h" diff --git a/src/operator/tensor/elemwise_scatter_op.h b/src/operator/tensor/elemwise_scatter_op.h index 33bc0da8a..0e52a86da 100644 --- a/src/operator/tensor/elemwise_scatter_op.h +++ b/src/operator/tensor/elemwise_scatter_op.h @@ -18,7 +18,7 @@ */ /*! - * \file elementwise_scatter_op.h + * \file elemwise_scatter_op.h * \brief Function definition of elementwise scatter operators */ #ifndef MXNET_OPERATOR_TENSOR_ELEMWISE_SCATTER_OP_H_ diff --git a/src/operator/tensor/elemwise_sum.cc b/src/operator/tensor/elemwise_sum.cc index 1666537e2..85b58b6e0 100644 --- a/src/operator/tensor/elemwise_sum.cc +++ b/src/operator/tensor/elemwise_sum.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file elemwise_sum.cc - * \brief elementwise sum operator + * \brief CPU implementation of elementwise sum operator */ #include "./elemwise_sum.h" #include "../../ndarray/ndarray_function.h" diff --git a/src/operator/tensor/elemwise_sum.cu b/src/operator/tensor/elemwise_sum.cu index e2b41e385..f9a248214 100644 --- a/src/operator/tensor/elemwise_sum.cu +++ b/src/operator/tensor/elemwise_sum.cu @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file elemwise_sum.cu - * \brief elementwise sum operator + * \brief GPU implementation of elementwise sum operator */ #include "./elemwise_sum.h" #include "../../ndarray/ndarray_function.h" diff --git a/src/operator/tensor/elemwise_sum.h b/src/operator/tensor/elemwise_sum.h index acf73e722..08f57d15f 100644 --- a/src/operator/tensor/elemwise_sum.h +++ b/src/operator/tensor/elemwise_sum.h @@ -20,7 +20,7 @@ /*! * Copyright (c) 2015 by Contributors * \file elemwise_sum.h - * \brief elementwise sum + * \brief Function definition of elementwise sum * \author Bing Xu */ #ifndef MXNET_OPERATOR_TENSOR_ELEMWISE_SUM_H_ diff --git a/src/operator/tensor/elemwise_unary_op.h b/src/operator/tensor/elemwise_unary_op.h index 83b86bf1d..8d5ad055b 100644 --- a/src/operator/tensor/elemwise_unary_op.h +++ b/src/operator/tensor/elemwise_unary_op.h @@ -29,11 +29,15 @@ #include #include #include +#include #include "./cast_storage-inl.h" #include "../mshadow_op.h" #include "../mxnet_op.h" #include "../elemwise_op_common.h" #include "../../ndarray/ndarray_function.h" +#if MSHADOW_USE_MKL == 1 +#include "mkl.h" +#endif namespace mxnet { namespace op { @@ -348,6 +352,43 @@ class UnaryOp : public OpBase { LogUnimplementedOp(attrs, ctx, inputs, req, outputs); } } + +#if MSHADOW_USE_MKL == 1 + static inline void MKLLog(MKL_INT size, const float* pIn, float* pOut) { + vsLn(size, pIn, pOut); + } + + static inline void MKLLog(MKL_INT size, const double* pIn, double* pOut) { + vdLn(size, pIn, pOut); + } +#endif + + template + static void LogCompute(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + if (req[0] == kNullOp) return; + // if defined MSHADOW_USE_MKL then call mkl log when req is KWriteTo, type_flag + // is mshadow::kFloat32 or mshadow::kFloat64 and data size less than or equal MKL_INT_MAX +#if MSHADOW_USE_MKL == 1 + auto type_flag = inputs[0].type_flag_; + const size_t MKL_INT_MAX = (sizeof(MKL_INT) == sizeof(int)) ? INT_MAX : LLONG_MAX; + size_t input_size = inputs[0].Size(); + if (req[0] == kWriteTo && + input_size <= MKL_INT_MAX && + (type_flag == mshadow::kFloat32 || type_flag == mshadow::kFloat64)) { + MSHADOW_SGL_DBL_TYPE_SWITCH(type_flag, DType, { + MKLLog(input_size, inputs[0].dptr(), outputs[0].dptr()); + }); + } else { + Compute(attrs, ctx, inputs, req, outputs); + } +#else + Compute(attrs, ctx, inputs, req, outputs); +#endif + } }; /*! \brief Map legacy unary_bwd to backward_grad */ diff --git a/src/operator/tensor/elemwise_unary_op_basic.cc b/src/operator/tensor/elemwise_unary_op_basic.cc index 49ae976cf..7f69395d1 100644 --- a/src/operator/tensor/elemwise_unary_op_basic.cc +++ b/src/operator/tensor/elemwise_unary_op_basic.cc @@ -19,7 +19,7 @@ /*! * \file elemwise_unary_op_basic.cc - * \brief CPU Implementation of unary function. + * \brief CPU Implementation of elementwise unary function. */ #include #include "elemwise_unary_op.h" @@ -236,6 +236,20 @@ NNVM_REGISTER_OP(_backward_copy) return std::vector{true}; }); +NNVM_REGISTER_OP(_backward_reshape) +.set_num_inputs(1) +.set_num_outputs(1) +.set_attr("TIsBackward", true) +.set_attr("FInplaceOption", + [](const NodeAttrs& attrs){ + return std::vector >{{0, 0}}; + }) +.set_attr("FCompute", UnaryOp::IdentityCompute) +.set_attr("FInplaceIdentity", + [](const NodeAttrs& attrs){ + return std::vector{true}; + }); + MXNET_OPERATOR_REGISTER_UNARY(BlockGrad) MXNET_ADD_SPARSE_OP_ALIAS(stop_gradient) .add_alias("stop_gradient") @@ -492,7 +506,8 @@ void ShapeComputeCPU(const nnvm::NodeAttrs& attrs, CHECK_EQ(req.size(), 1U); const TBlob& in_data = inputs[0]; const TBlob& out_data = outputs[0]; - memcpy(out_data.dptr_, in_data.shape_.data(), in_data.ndim() * sizeof(int64_t)); + size_t type_size = mshadow::mshadow_sizeof(out_data.type_flag_); + memcpy(out_data.dptr_, in_data.shape_.data(), in_data.ndim() * type_size); } NNVM_REGISTER_OP(shape_array) @@ -542,8 +557,9 @@ void SizeComputeCPU(const nnvm::NodeAttrs& attrs, CHECK_EQ(req.size(), 1U); const TBlob& in_data = inputs[0]; const TBlob& out_data = outputs[0]; + size_t type_size = mshadow::mshadow_sizeof(out_data.type_flag_); const index_t size_var = in_data.Size(); - memcpy(out_data.dptr_, &size_var, 1U * sizeof(int64_t)); + memcpy(out_data.dptr_, &size_var, type_size); } NNVM_REGISTER_OP(size_array) @@ -884,6 +900,22 @@ The storage type of ``cbrt`` output depends upon the input storage type: MXNET_OPERATOR_REGISTER_BINARY_WITH_SPARSE_CPU_DR(_backward_cbrt, unary_bwd); +// erf +MXNET_OPERATOR_REGISTER_UNARY(erf) +.describe(R"code(Returns element-wise gauss error function of the input. + +Example:: + + erf([0, -1., 10.]) = [0., -0.8427, 1.] + +)code" ADD_FILELINE) +.set_attr("FCompute", UnaryOp::Compute) +.set_attr("FGradient", ElemwiseGradUseIn{"_backward_erf"}); + +MXNET_OPERATOR_REGISTER_BINARY(_backward_erf) +.set_attr("FCompute", + ElemwiseBinaryOp::Compute>); + // rcbrt MXNET_OPERATOR_REGISTER_UNARY(rcbrt) .describe(R"code(Returns element-wise inverse cube-root value of the input. @@ -922,7 +954,7 @@ The storage type of ``exp`` output is always dense .set_attr("FGradient", ElemwiseGradUseOut{"_mul"}); // log -MXNET_OPERATOR_REGISTER_UNARY_WITH_SPARSE_DR(log, cpu, mshadow_op::log) +MXNET_OPERATOR_REGISTER_UNARY(log) MXNET_ADD_SPARSE_OP_ALIAS(log) .describe(R"code(Returns element-wise Natural logarithmic value of the input. @@ -931,6 +963,7 @@ The natural logarithm is logarithm in base *e*, so that ``log(exp(x)) = x`` The storage type of ``log`` output is always dense )code" ADD_FILELINE) +.set_attr("FCompute", UnaryOp::LogCompute) .set_attr("FGradient", ElemwiseGradUseIn{"_backward_log"}); // log10 diff --git a/src/operator/tensor/elemwise_unary_op_basic.cu b/src/operator/tensor/elemwise_unary_op_basic.cu index 8e2a0991b..14f2be02a 100644 --- a/src/operator/tensor/elemwise_unary_op_basic.cu +++ b/src/operator/tensor/elemwise_unary_op_basic.cu @@ -18,8 +18,8 @@ */ /*! - * \file elemwise_unary_op_trig.cu - * \brief GPU Implementation of unary trigometric functions. + * \file elemwise_unary_op_basic.cu + * \brief GPU Implementation of unary functions. */ #include "./elemwise_binary_op.h" @@ -54,12 +54,24 @@ NNVM_REGISTER_OP(_backward_softsign) .set_attr("FCompute", ElemwiseBinaryOp::Compute< gpu, unary_bwd>); +// erf +NNVM_REGISTER_OP(erf) +.set_attr("FCompute", UnaryOp::Compute); + +NNVM_REGISTER_OP(_backward_erf) +.set_attr("FCompute", + ElemwiseBinaryOp::Compute>); + // copy NNVM_REGISTER_OP(_copy) .set_attr("FCompute", UnaryOp::IdentityCompute) .set_attr("FComputeEx", UnaryOp::IdentityComputeEx); NNVM_REGISTER_OP(_backward_copy) +.set_attr("FCompute", UnaryOp::IdentityCompute) +.set_attr("FComputeEx", UnaryOp::IdentityComputeEx); + +NNVM_REGISTER_OP(_backward_reshape) .set_attr("FCompute", UnaryOp::IdentityCompute); NNVM_REGISTER_OP(BlockGrad) diff --git a/src/operator/tensor/elemwise_unary_op_trig.cc b/src/operator/tensor/elemwise_unary_op_trig.cc index 288719f48..28a11cc83 100644 --- a/src/operator/tensor/elemwise_unary_op_trig.cc +++ b/src/operator/tensor/elemwise_unary_op_trig.cc @@ -19,7 +19,7 @@ /*! * \file elemwise_unary_op_trig.cc - * \brief CPU Implementation of unary trigometric functions. + * \brief CPU Implementation of unary trigonometric functions. */ #include #include "elemwise_unary_op.h" diff --git a/src/operator/tensor/elemwise_unary_op_trig.cu b/src/operator/tensor/elemwise_unary_op_trig.cu index 90f248b3f..8e28b9c60 100644 --- a/src/operator/tensor/elemwise_unary_op_trig.cu +++ b/src/operator/tensor/elemwise_unary_op_trig.cu @@ -18,8 +18,8 @@ */ /*! - * \file elemwise_unary_op.cu - * \brief GPU Implementation of unary function. + * \file elemwise_unary_op_trig.cu + * \brief GPU Implementation of unary trigonometric function. */ #include "./elemwise_binary_op.h" diff --git a/src/operator/tensor/histogram-inl.h b/src/operator/tensor/histogram-inl.h index 08620e841..40acb55d1 100644 --- a/src/operator/tensor/histogram-inl.h +++ b/src/operator/tensor/histogram-inl.h @@ -17,6 +17,10 @@ * under the License. */ +/*! + * \file histogram-inl.h + * \brief Function definition of histogram operator +*/ #ifndef MXNET_OPERATOR_TENSOR_HISTOGRAM_INL_H_ #define MXNET_OPERATOR_TENSOR_HISTOGRAM_INL_H_ diff --git a/src/operator/tensor/histogram.cc b/src/operator/tensor/histogram.cc index ac28606f9..3c4eaa158 100644 --- a/src/operator/tensor/histogram.cc +++ b/src/operator/tensor/histogram.cc @@ -17,6 +17,10 @@ * under the License. */ +/*! + * \file histogram.cc + * \brief CPU implementation of histogram operator +*/ #include "./histogram-inl.h" namespace mxnet { diff --git a/src/operator/tensor/histogram.cu b/src/operator/tensor/histogram.cu index c3c836a4b..b811aba46 100644 --- a/src/operator/tensor/histogram.cu +++ b/src/operator/tensor/histogram.cu @@ -17,6 +17,10 @@ * under the License. */ +/*! + * \file histogram.cu + * \brief GPU implementation of histogram operator +*/ #include "./histogram-inl.h" #include "./util/tensor_util-inl.cuh" diff --git a/src/operator/tensor/indexing_op.cc b/src/operator/tensor/indexing_op.cc index 98e2536f4..c39418dbe 100644 --- a/src/operator/tensor/indexing_op.cc +++ b/src/operator/tensor/indexing_op.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file indexing_op.cc - * \brief + * \brief CPU implementation of indexing operator * \author Siyi Li, Chi Zhang */ @@ -28,6 +28,28 @@ namespace mxnet { namespace op { +template +struct TakeCPU { + // assume that idx have been flattened to a 1-D tensor (N,) + // assume that out_data and in_data have been flattened to 2-D tensors, (N, M) and (K, M) + // M is the number of columns of in_data and out_data + // K is the number of rows of in_data + // i is the index of out_data + template + MSHADOW_XINLINE static void Map(index_t i, DType* out_data, const DType* in_data, + const IType* idx, const size_t M, const int64_t K) { + int64_t j = static_cast(idx[i]); + if (clip) { + if (j <= 0) j = 0; + else if (j >= K) j = K - 1; + } else { + j = j % K; + j += (j < 0) ? K : 0; + } + std::memcpy(out_data + i * M, in_data + j * M, M * sizeof(DType)); + } +}; + /* * \brief returns true if all indices are between [min, max] * \param data_ptr the indices to check @@ -48,6 +70,29 @@ bool CheckIndexOutOfBound(const DType* data_ptr, size_t data_size, return is_valid; } +// Embedding forward implementation with dense weight +template<> +void EmbeddingOpForwardDnsImpl(mshadow::Stream* s, + const TBlob& data, + const TBlob& weight, + const OpReqType req, + const TBlob& output) { + using namespace mxnet_op; + const TShape& ishape = data.shape_; + const TShape& oshape = output.shape_; + + MSHADOW_TYPE_SWITCH(output.type_flag_, DType, { + MSHADOW_TYPE_SWITCH(data.type_flag_, IType, { + Tensor idx = data.get_with_shape( + Shape1(ishape.ProdShape(0, ishape.ndim())), s); + Tensor wmat = weight.get(s); + Tensor out = output.get_with_shape( + Shape2(oshape.ProdShape(0, oshape.ndim()-1), oshape[oshape.ndim()-1]), s); + Kernel, cpu>::Launch(s, oshape.Size() / wmat.shape_[1], out.dptr_, wmat.dptr_, + idx.dptr_, wmat.shape_[1], wmat.shape_[0]); + }); + }); +} template<> void SparseEmbeddingOpForwardRspImpl(const OpContext& ctx, @@ -137,7 +182,10 @@ struct CsrTakeRowCountKernel { MSHADOW_XINLINE static void Map(int tid, RType* out_indptr, const RType* src_indptr, const IType* idx_ptr, const nnvm::dim_t num_rows) { - if (tid == 0) out_indptr[0] = 0; + if (tid == 0) { + out_indptr[0] = 0; + return; + } nnvm::dim_t idx = static_cast(idx_ptr[tid - 1]); // clip mode if (clip) { @@ -181,7 +229,7 @@ void TakeOpForwardCsrImpl(const TakeParam& params, out.CheckAndAllocAuxData(kIndPtr, {Shape1(num_rows + 1)}); MSHADOW_TYPE_SWITCH(idx.type_flag_, IType, { - MSHADOW_SGL_DBL_TYPE_SWITCH(arr.dtype(), DType, { + MSHADOW_TYPE_SWITCH(arr.dtype(), DType, { MSHADOW_IDX_TYPE_SWITCH(out.aux_type(kIdx), RType, { RType* out_indptr = out.aux_data(kIndPtr).dptr(); const RType* src_indptr = arr.aux_data(kIndPtr).dptr(); @@ -224,6 +272,74 @@ void TakeOpForwardCsrImpl(const TakeParam& params, }); } +template<> +void TakeOpForward(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + using namespace mxnet_op; + if (req[take_::kOut] == kNullOp) return; + const TakeParam& param = nnvm::get(attrs.parsed); + CHECK_EQ(inputs.size(), 2U); + CHECK_EQ(outputs.size(), 1U); + + const TShape& idxshape = inputs[take_::kIdx].shape_; + const TShape& arrshape = inputs[take_::kArr].shape_; + const TShape& oshape = outputs[take_::kOut].shape_; + + Stream *s = ctx.get_stream(); + const int actual_axis = param.axis + ((param.axis < 0) ? arrshape.ndim() : 0); + + MSHADOW_TYPE_SWITCH(outputs[0].type_flag_, DType, { // output data type + MSHADOW_TYPE_SWITCH(inputs[1].type_flag_, IType, { // index data type + if (actual_axis == 0) { + if (param.mode == take_::kClip) { + Kernel, cpu>::Launch(s, idxshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + oshape.Size()/idxshape.Size(), arrshape[0]); + } else { + Kernel, cpu>::Launch(s, idxshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + oshape.Size()/idxshape.Size(), arrshape[0]); + } + } else { + mshadow::Shape<10> in_strides; + int stride = 1; + for (int i = arrshape.ndim() - 1; i >= 0; stride *= arrshape[i], --i) { + in_strides[i] = stride; + } + mshadow::Shape<10> out_strides; + stride = 1; + for (int i = oshape.ndim() - 1; i >= 0; stride *= oshape[i], --i) { + out_strides[i] = stride; + } + if (param.mode == take_::kClip) { + Kernel, cpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + in_strides, out_strides, arrshape.ndim(), + oshape.ndim(), idxshape.ndim(), + arrshape[actual_axis], actual_axis); + } else if (param.mode == take_::kWrap) { + Kernel, cpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + in_strides, out_strides, arrshape.ndim(), + oshape.ndim(), idxshape.ndim(), + arrshape[actual_axis], actual_axis); + } + } + }); + }); +} + template<> inline void SparseEmbeddingOpBackwardRspImpl(const bool deterministic, const OpContext& ctx, @@ -304,19 +420,19 @@ inline void SparseEmbeddingOpBackwardRspImpl(const bool deterministic, template inline typename std::enable_if<(!std::is_same::value), void>::type -GatherNDBackwardImpl(int N, int M, int K, +GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, const IType* indices, mshadow::Stream *s) { #pragma omp parallel for - for (int i = 0; i < N; i++) { - int offset = 0; - for (int j = 0; j < M; ++j) { - offset += strides[j] * static_cast(indices[j*N + i]); + for (index_t i = 0; i < N; i++) { + index_t offset = 0; + for (index_t j = 0; j < M; ++j) { + offset += strides[j] * static_cast(indices[j*N + i]); } - for (int j = 0; j < K; ++j) { + for (index_t j = 0; j < K; ++j) { #pragma omp atomic out[offset + j] += data[i * K + j]; } @@ -325,18 +441,18 @@ GatherNDBackwardImpl(int N, int M, int K, template inline typename std::enable_if::value, void>::type -GatherNDBackwardImpl(int N, int M, int K, +GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, const IType* indices, mshadow::Stream *s) { - for (int i = 0; i < N; i++) { - int offset = 0; - for (int j = 0; j < M; ++j) { - offset += strides[j] * static_cast(indices[j*N + i]); + for (index_t i = 0; i < N; i++) { + index_t offset = 0; + for (index_t j = 0; j < M; ++j) { + offset += strides[j] * static_cast(indices[j*N + i]); } - for (int j = 0; j < K; ++j) { + for (index_t j = 0; j < K; ++j) { out[offset + j] += data[i * K + j]; } } diff --git a/src/operator/tensor/indexing_op.cu b/src/operator/tensor/indexing_op.cu index bdc7f6e84..bad3e5a1a 100644 --- a/src/operator/tensor/indexing_op.cu +++ b/src/operator/tensor/indexing_op.cu @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file indexing_op.cu - * \brief + * \brief GPU implementation of indexing operator * \author Siyi Li, Chi Zhang */ @@ -116,6 +116,31 @@ struct AddTakeGradRspDeterministicKernel { } }; +/*! \brief name the struct Take instead of take + * to avoid conflict with the take function in mshadow + */ +template +struct TakeGPU { + // assume that idx have been flattened to a 1-D tensor (N,) + // assume that out_data and in_data have been flattened to 2-D tensors, (N, M) and (K, M) + // M is the number of columns of in_data and out_data + // K is the number of rows of in_data + // i is the index of out_data + template + MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, + const IType* idx, const int64_t M, const int64_t K) { + int64_t j = static_cast(idx[i/M]); + if (clip) { + if (j <= 0) j = 0; + else if (j >= K) j = K - 1; + } else { + j = j % K; + j += (j < 0) ? K : 0; + } + out_data[i] = in_data[j * M + i % M]; + } +}; + /* * \brief returns true if all indices are between [min, max] * \param s the stream @@ -137,6 +162,30 @@ bool CheckIndexOutOfBound(mshadow::Stream *s, const DType* data_ptr, size_t return is_valid == 0; } +// Embedding forward implementation with dense weight +template<> +void EmbeddingOpForwardDnsImpl(mshadow::Stream* s, + const TBlob& data, + const TBlob& weight, + const OpReqType req, + const TBlob& output) { + using namespace mxnet_op; + const TShape& ishape = data.shape_; + const TShape& oshape = output.shape_; + + MSHADOW_TYPE_SWITCH(output.type_flag_, DType, { + MSHADOW_TYPE_SWITCH(data.type_flag_, IType, { + Tensor idx = data.get_with_shape( + Shape1(ishape.ProdShape(0, ishape.ndim())), s); + Tensor wmat = weight.get(s); + Tensor out = output.get_with_shape( + Shape2(oshape.ProdShape(0, oshape.ndim()-1), oshape[oshape.ndim()-1]), s); + Kernel, gpu>::Launch(s, oshape.Size(), out.dptr_, wmat.dptr_, + idx.dptr_, wmat.shape_[1], wmat.shape_[0]); + }); + }); +} + template<> void SparseEmbeddingOpForwardRspImpl(const OpContext& ctx, const TBlob& data, @@ -390,22 +439,22 @@ inline void SparseEmbeddingOpBackwardRspImpl(const bool deterministic, struct backward_gather_nd_gpu { template - MSHADOW_XINLINE static void Map(int i, int N, int M, int K, + MSHADOW_XINLINE static void Map(index_t i, index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, const IType* indices) { - int offset = 0; - for (int j = 0; j < M; ++j) { + index_t offset = 0; + for (index_t j = 0; j < M; ++j) { offset += strides[j] * static_cast(indices[j*N + i]); } - for (int j = 0; j < K; ++j) { + for (index_t j = 0; j < K; ++j) { atomicAdd(out + (offset + j), data[i * K + j]); } } }; template -inline void GatherNDBackwardImpl(int N, int M, int K, +inline void GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, @@ -414,6 +463,72 @@ inline void GatherNDBackwardImpl(int N, int M, int K, mxnet_op::Kernel::Launch(s, N, N, M, K, strides, out, data, indices); } +template<> +void TakeOpForward(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + using namespace mxnet_op; + if (req[take_::kOut] == kNullOp) return; + const TakeParam& param = nnvm::get(attrs.parsed); + CHECK_EQ(inputs.size(), 2U); + CHECK_EQ(outputs.size(), 1U); + + const TShape& idxshape = inputs[take_::kIdx].shape_; + const TShape& arrshape = inputs[take_::kArr].shape_; + const TShape& oshape = outputs[take_::kOut].shape_; + + Stream *s = ctx.get_stream(); + const int actual_axis = param.axis + ((param.axis < 0) ? arrshape.ndim() : 0); + + MSHADOW_TYPE_SWITCH(outputs[0].type_flag_, DType, { // output data type + MSHADOW_TYPE_SWITCH(inputs[1].type_flag_, IType, { // index data type + if (actual_axis == 0) { + if (param.mode == take_::kClip) { + Kernel, gpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + oshape.Size()/idxshape.Size(), arrshape[0]); + } else { + Kernel, gpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + oshape.Size()/idxshape.Size(), arrshape[0]); + } + } else { + mshadow::Shape<10> in_strides; + int stride = 1; + for (int i = arrshape.ndim() - 1; i >= 0; stride *= arrshape[i], --i) { + in_strides[i] = stride; + } + mshadow::Shape<10> out_strides; + stride = 1; + for (int i = oshape.ndim() - 1; i >= 0; stride *= oshape[i], --i) { + out_strides[i] = stride; + } + if (param.mode == take_::kClip) { + Kernel, gpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + in_strides, out_strides, arrshape.ndim(), oshape.ndim(), + idxshape.ndim(), arrshape[actual_axis], actual_axis); + } else if (param.mode == take_::kWrap) { + Kernel, gpu>::Launch(s, oshape.Size(), + outputs[take_::kOut].dptr(), + inputs[take_::kArr].dptr(), + inputs[take_::kIdx].dptr(), + in_strides, out_strides, arrshape.ndim(), oshape.ndim(), + idxshape.ndim(), arrshape[actual_axis], actual_axis); + } + } + }); + }); +} + NNVM_REGISTER_OP(Embedding) .set_attr("FCompute", EmbeddingOpForward) .set_attr("FComputeEx", SparseEmbeddingOpForwardEx); diff --git a/src/operator/tensor/indexing_op.h b/src/operator/tensor/indexing_op.h index 5282a7ea9..fba331e25 100644 --- a/src/operator/tensor/indexing_op.h +++ b/src/operator/tensor/indexing_op.h @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file indexing_op.h - * \brief + * \brief Function definition of indexing operator * \author Bing Xu, Siyi Li, Chi Zhang, Haibin Lin */ #ifndef MXNET_OPERATOR_TENSOR_INDEXING_OP_H_ @@ -301,25 +301,6 @@ inline bool SparseEmbeddingOpBackwardStorageType(const nnvm::NodeAttrs& attrs, */ template struct Take { - // assume that idx have been flattened to a 1-D tensor (N,) - // assume that out_data and in_data have been flattened to 2-D tensors, (N, M) and (K, M) - // M is the number of columns of in_data and out_data - // K is the number of rows of in_data - // i is the index of out_data - template - MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, - const IType* idx, const int M, const int K) { - int j = static_cast(idx[i/M]); - if (clip) { - if (j <= 0) j = 0; - else if (j >= K) j = K - 1; - } else { - j = j % K; - j += (j < 0) ? K : 0; - } - out_data[i] = in_data[j * M + i % M]; - } - /*! * \brief Map function for take operator * \param i global thread id @@ -333,27 +314,28 @@ struct Take { * \param axis axis id */ template - MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, const IType* idx, + MSHADOW_XINLINE static void Map(index_t i, DType* out_data, const DType* in_data, + const IType* idx, const mshadow::Shape<10> in_stride, const mshadow::Shape<10> out_stride, const int in_ndims, const int out_ndims, const int idx_ndims, const int axis_dim, const int axis) { // i is the global flattened index in the output - const int out_head_index = (axis == 0) ? 0 : (i / out_stride[axis - 1]); - const int out_rest_index = (axis == 0) ? i : (i % out_stride[axis - 1]); - const int out_mid_index = out_rest_index / in_stride[axis]; - const int out_tail_index = (axis == in_ndims - 1) ? - 0 : (out_rest_index % in_stride[axis]); - int idx_index = static_cast(idx[out_mid_index]); + const int64_t out_head_index = (axis == 0) ? 0 : (i / out_stride[axis - 1]); + const int64_t out_rest_index = (axis == 0) ? i : (i % out_stride[axis - 1]); + const int64_t out_mid_index = out_rest_index / in_stride[axis]; + const int64_t out_tail_index = (axis == in_ndims - 1) ? + 0 : (out_rest_index % in_stride[axis]); + int64_t idx_index = static_cast(idx[out_mid_index]); if (clip) { idx_index = (idx_index < 0) ? 0 : idx_index; idx_index = (idx_index > axis_dim - 1) ? (axis_dim - 1) : idx_index; } idx_index %= axis_dim; idx_index += (idx_index < 0) ? axis_dim : 0; - const int in_tail_index = out_tail_index; - const int in_head_index = out_head_index; - int in_src_index = in_tail_index + idx_index * in_stride[axis]; + const int64_t in_tail_index = out_tail_index; + const int64_t in_head_index = out_head_index; + int64_t in_src_index = in_tail_index + idx_index * in_stride[axis]; in_src_index += (axis == 0) ? 0 : in_head_index * in_stride[axis - 1]; out_data[i] = in_data[in_src_index]; } @@ -365,24 +347,7 @@ void EmbeddingOpForwardDnsImpl(mshadow::Stream* s, const TBlob& data, const TBlob& weight, const OpReqType req, - const TBlob& output) { - using namespace mxnet_op; - const TShape& ishape = data.shape_; - const TShape& oshape = output.shape_; - - MSHADOW_TYPE_SWITCH(output.type_flag_, DType, { - MSHADOW_TYPE_SWITCH(data.type_flag_, IType, { - Tensor idx = data.get_with_shape( - Shape1(ishape.ProdShape(0, ishape.ndim())), s); - Tensor wmat = weight.get(s); - Tensor out = output.get_with_shape( - Shape2(oshape.ProdShape(0, oshape.ndim()-1), oshape[oshape.ndim()-1]), s); - Kernel, xpu>::Launch(s, oshape.Size(), out.dptr_, wmat.dptr_, - idx.dptr_, wmat.shape_[1], wmat.shape_[0]); - }); - }); -} - + const TBlob& output); template struct TakeRspKernel { @@ -397,7 +362,7 @@ struct TakeRspKernel { * \param nnr number of non-zero rows */ template - MSHADOW_XINLINE static void Map(int i, + MSHADOW_XINLINE static void Map(index_t i, const IType* data, DType* out, const RType* weight_idx, @@ -825,66 +790,7 @@ void TakeOpForward(const nnvm::NodeAttrs& attrs, const OpContext& ctx, const std::vector& inputs, const std::vector& req, - const std::vector& outputs) { - using namespace mxnet_op; - if (req[take_::kOut] == kNullOp) return; - const TakeParam& param = nnvm::get(attrs.parsed); - CHECK_EQ(inputs.size(), 2U); - CHECK_EQ(outputs.size(), 1U); - - const TShape& idxshape = inputs[take_::kIdx].shape_; - const TShape& arrshape = inputs[take_::kArr].shape_; - const TShape& oshape = outputs[take_::kOut].shape_; - - Stream *s = ctx.get_stream(); - const int actual_axis = param.axis + ((param.axis < 0) ? arrshape.ndim() : 0); - - MSHADOW_TYPE_SWITCH(outputs[0].type_flag_, DType, { // output data type - MSHADOW_TYPE_SWITCH(inputs[1].type_flag_, IType, { // index data type - if (actual_axis == 0) { - if (param.mode == take_::kClip) { - Kernel, xpu>::Launch(s, oshape.Size(), - outputs[take_::kOut].dptr(), - inputs[take_::kArr].dptr(), - inputs[take_::kIdx].dptr(), - oshape.Size()/idxshape.Size(), arrshape[0]); - } else { - Kernel, xpu>::Launch(s, oshape.Size(), - outputs[take_::kOut].dptr(), - inputs[take_::kArr].dptr(), - inputs[take_::kIdx].dptr(), - oshape.Size()/idxshape.Size(), arrshape[0]); - } - } else { - mshadow::Shape<10> in_strides; - int stride = 1; - for (int i = arrshape.ndim() - 1; i >= 0; stride *= arrshape[i], --i) { - in_strides[i] = stride; - } - mshadow::Shape<10> out_strides; - stride = 1; - for (int i = oshape.ndim() - 1; i >= 0; stride *= oshape[i], --i) { - out_strides[i] = stride; - } - if (param.mode == take_::kClip) { - Kernel, xpu>::Launch(s, oshape.Size(), - outputs[take_::kOut].dptr(), - inputs[take_::kArr].dptr(), - inputs[take_::kIdx].dptr(), - in_strides, out_strides, arrshape.ndim(), oshape.ndim(), - idxshape.ndim(), arrshape[actual_axis], actual_axis); - } else if (param.mode == take_::kWrap) { - Kernel, xpu>::Launch(s, oshape.Size(), - outputs[take_::kOut].dptr(), - inputs[take_::kArr].dptr(), - inputs[take_::kIdx].dptr(), - in_strides, out_strides, arrshape.ndim(), oshape.ndim(), - idxshape.ndim(), arrshape[actual_axis], actual_axis); - } - } - }); - }); -} + const std::vector& outputs); struct TakeGradGeneralKernel { /*! @@ -1490,15 +1396,15 @@ inline bool ScatterNDType(const nnvm::NodeAttrs& attrs, struct scatter_nd { template - MSHADOW_XINLINE static void Map(int i, OpReqType req, int N, int M, int K, + MSHADOW_XINLINE static void Map(index_t i, OpReqType req, index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, const IType* indices) { - int offset = 0; - for (int j = 0; j < M; ++j) { - offset += strides[j] * static_cast(indices[j*N + i]); + index_t offset = 0; + for (index_t j = 0; j < M; ++j) { + offset += strides[j] * static_cast(indices[j*N + i]); } - for (int j = 0; j < K; ++j) { + for (index_t j = 0; j < K; ++j) { KERNEL_ASSIGN(out[offset+j], req, data[i*K + j]); } } @@ -1511,17 +1417,18 @@ void ScatterNDForward(const nnvm::NodeAttrs& attrs, const std::vector& req, const std::vector& outputs) { using namespace mshadow; + using nnvm::dim_t; CHECK_EQ(inputs.size(), 2U); CHECK_EQ(outputs.size(), 1U); if (req[0] == kNullOp) return; mshadow::Stream *s = ctx.get_stream(); const TShape& oshape = outputs[0].shape_; const TShape& ishape = inputs[1].shape_; - int M = ishape[0]; - int N = ishape.Size() / M; - int K = oshape.ProdShape(M, oshape.ndim()); + dim_t M = ishape[0]; + dim_t N = ishape.Size() / M; + dim_t K = oshape.ProdShape(M, oshape.ndim()); mshadow::Shape<10> strides; - for (int i = M-1, stride = K; i >= 0; stride *= oshape[i], --i) strides[i] = stride; + for (dim_t i = M-1, stride = K; i >= 0; stride *= oshape[i], --i) strides[i] = stride; if (kWriteTo == req[0]) { Fill(s, outputs[0], req[0], 0); } @@ -1536,7 +1443,7 @@ void ScatterNDForward(const nnvm::NodeAttrs& attrs, template inline typename std::enable_if<(!std::is_same::value), void>::type -GatherNDBackwardImpl(int N, int M, int K, +GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, @@ -1545,7 +1452,7 @@ GatherNDBackwardImpl(int N, int M, int K, template inline typename std::enable_if::value, void>::type -GatherNDBackwardImpl(int N, int M, int K, +GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, @@ -1553,7 +1460,7 @@ GatherNDBackwardImpl(int N, int M, int K, mshadow::Stream *s); template -inline void GatherNDBackwardImpl(int N, int M, int K, +inline void GatherNDBackwardImpl(index_t N, index_t M, index_t K, const mshadow::Shape<10> strides, DType* out, const DType* data, @@ -1567,17 +1474,18 @@ void GatherNDBackward(const nnvm::NodeAttrs& attrs, const std::vector& req, const std::vector& outputs) { using namespace mshadow; + using nnvm::dim_t; CHECK_EQ(inputs.size(), 2U); CHECK_EQ(outputs.size(), 1U); if (req[0] == kNullOp) return; mshadow::Stream *s = ctx.get_stream(); const TShape& oshape = outputs[0].shape_; const TShape& ishape = inputs[1].shape_; - int M = ishape[0]; - int N = ishape.Size() / M; - int K = oshape.ProdShape(M, oshape.ndim()); + dim_t M = ishape[0]; + dim_t N = ishape.Size() / M; + dim_t K = oshape.ProdShape(M, oshape.ndim()); mshadow::Shape<10> strides; - for (int i = M-1, stride = K; i >= 0; stride *= oshape[i], --i) strides[i] = stride; + for (dim_t i = M-1, stride = K; i >= 0; stride *= oshape[i], --i) strides[i] = stride; if (kWriteTo == req[0]) { Fill(s, outputs[0], req[0], 0); } diff --git a/src/operator/tensor/init_op.h b/src/operator/tensor/init_op.h index 4e52b087f..e9e67cb1a 100644 --- a/src/operator/tensor/init_op.h +++ b/src/operator/tensor/init_op.h @@ -453,7 +453,7 @@ void EyeFill(const nnvm::NodeAttrs& attrs, struct range_fwd { template - MSHADOW_XINLINE static void Map(int i, int repeat, DType start, DType step, + MSHADOW_XINLINE static void Map(index_t i, int repeat, DType start, DType step, int req, DType* out) { KERNEL_ASSIGN(out[i], req, start + (i/repeat) * step); } @@ -471,8 +471,8 @@ void RangeCompute(const nnvm::NodeAttrs& attrs, MSHADOW_TYPE_SWITCH(outputs[0].type_flag_, DType, { // Force unsigned params to take two's complement form on ARM to ensure consistency with x86 // results. Casting negative floats to unsigned types is undefined in the CPP standard. - auto step = std::is_signed() ? param.step : static_cast(param.step); - auto start = std::is_signed() ? param.start : static_cast(param.start); + auto step = std::is_signed() ? param.step : static_cast(param.step); + auto start = std::is_signed() ? param.start : static_cast(param.start); Kernel::Launch(s, outputs[0].Size(), static_cast(param.repeat), diff --git a/src/operator/tensor/la_op_inline.h b/src/operator/tensor/la_op-inl.h similarity index 77% rename from src/operator/tensor/la_op_inline.h rename to src/operator/tensor/la_op-inl.h index b48310897..e89a0824a 100644 --- a/src/operator/tensor/la_op_inline.h +++ b/src/operator/tensor/la_op-inl.h @@ -19,11 +19,12 @@ /*! * Copyright (c) 2017 by Contributors - * \file la_op_inline.h + * \file la_op-inl.h * \brief Operators for advanced linear algebra. + * \note See https://arxiv.org/pdf/1710.08717.pdf for details of gradient computations. */ -#ifndef MXNET_OPERATOR_TENSOR_LA_OP_INLINE_H_ -#define MXNET_OPERATOR_TENSOR_LA_OP_INLINE_H_ +#ifndef MXNET_OPERATOR_TENSOR_LA_OP_INL_H_ +#define MXNET_OPERATOR_TENSOR_LA_OP_INL_H_ #include "../linalg.h" @@ -32,20 +33,29 @@ namespace op { using namespace mshadow; -// Helper functions. -struct CopyLowerToUpper { +// Copies lower/upper triangular part to upper/lower, i.e. to the opposite side. +struct CopyTriangularToOppositeSide { template - MSHADOW_XINLINE static void Map(int i, int matrix_size, int stride, DType* data) { + MSHADOW_XINLINE static void Map(int i, int matrix_size, int stride, DType* data, bool to_lower) { // Below computation works even when we are dealing with a batch of matrices. const int row((i % matrix_size) / stride), col(i % stride); - if ( row > col ) data[i + (col - row) * (stride - 1)] = data[i]; + if (row > col) { + if (to_lower) { + data[i] = data[i + (col - row) * (stride - 1)]; + } else { + data[i + (col - row) * (stride - 1)] = data[i]; + } + } } }; -struct ZeroUpper { + +// Zero's lower/upper triangular part of a matrix. +struct ZeroTriangular { template - MSHADOW_XINLINE static void Map(int i, int matrix_size, int stride, DType* data) { + MSHADOW_XINLINE static void Map(int i, int matrix_size, int stride, DType* data, + bool zero_lower) { const int row((i % matrix_size) / stride), col(i % stride); - if ( row < col ) data[i] = 0; + if ((!zero_lower && (row < col)) || (zero_lower && (row > col))) data[i] = 0; } }; struct Scale { @@ -103,87 +113,91 @@ struct gemm2 { } }; -// L = potrf(A). +// B = potrf(A). struct potrf { template - static void op(const Tensor& A, const Tensor& L, + static void op(const Tensor& A, const Tensor& B, Stream *s, const nnvm::NodeAttrs& attrs) { - if ( A.dptr_ != L.dptr_ ) Copy(L, A, s); - linalg_batch_potrf(L, true, s); + const LaCholeskyParam& param = nnvm::get(attrs.parsed); + if ( A.dptr_ != B.dptr_ ) Copy(B, A, s); + linalg_batch_potrf(B, param.lower, s); using namespace mxnet_op; - Kernel::Launch(s, L.MSize(), L.size(1)*L.stride_, L.stride_, L.dptr_); + Kernel::Launch(s, B.MSize(), B.size(1)*B.stride_, B.stride_, + B.dptr_, !param.lower); } template - static void op(const Tensor& A, const Tensor& L, + static void op(const Tensor& A, const Tensor& B, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(A, L, s, attrs); + op(A, B, s, attrs); } }; -// A = potri(L). +// A = potri(B). struct potri { template - static void op(const Tensor& L, const Tensor& A, + static void op(const Tensor& B, const Tensor& A, Stream *s, const nnvm::NodeAttrs& attrs) { - if ( A.dptr_ != L.dptr_ ) Copy(A, L, s); - linalg_batch_potri(A, true, s); + const LaCholeskyParam& param = nnvm::get(attrs.parsed); + if ( A.dptr_ != B.dptr_ ) Copy(A, B, s); + linalg_batch_potri(A, param.lower, s); using namespace mxnet_op; - Kernel::Launch(s, A.MSize(), A.size(1)*A.stride_, A.stride_, A.dptr_); + Kernel::Launch(s, A.MSize(), A.size(1)*A.stride_, A.stride_, + A.dptr_, !param.lower); } template - static void op(const Tensor& L, const Tensor& A, + static void op(const Tensor& B, const Tensor& A, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(L, A, s, attrs); + op(B, A, s, attrs); } }; -// B = trsm(L,A) +// C = trsm(A,B) struct trsm { template - static void op(const Tensor& L, const Tensor& B, - DType alpha, bool rightside, bool transpose, Stream *s) { - linalg_batch_trsm(L, B, alpha, rightside, true, transpose, s); + static void op(const Tensor& A, const Tensor& C, + DType alpha, bool rightside, bool lower, bool transpose, Stream *s) { + linalg_batch_trsm(A, C, alpha, rightside, lower, transpose, s); } template - static void op(const Tensor& L, const Tensor& A, - const Tensor& B, + static void op(const Tensor& A, const Tensor& B, + const Tensor& C, Stream *s, const nnvm::NodeAttrs& attrs) { - if ( A.dptr_ != B.dptr_ ) Copy(B, A, s); + if ( B.dptr_ != C.dptr_ ) Copy(C, B, s); const LaTriangMatrixMultParam& param = nnvm::get(attrs.parsed); - op(L, B, DType(param.alpha), param.rightside, param.transpose, s); + op(A, C, DType(param.alpha), param.rightside, param.lower, param.transpose, s); } template - static void op(const Tensor& L, const Tensor& A, - const Tensor& B, + static void op(const Tensor& A, const Tensor& B, + const Tensor& C, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(L, A, B, s, attrs); + op(A, B, C, s, attrs); } }; -// B = trmm(L,A) +// C = trmm(A,B) struct trmm { template - static void op(const Tensor& L, const Tensor& B, - DType alpha, bool rightside, bool transpose, Stream *s) { - linalg_batch_trmm(L, B, alpha, rightside, true, transpose, s); + static void op(const Tensor& A, const Tensor& C, + DType alpha, bool rightside, bool lower, bool transpose, Stream *s) { + linalg_batch_trmm(A, C, alpha, rightside, lower, transpose, s); } template - static void op(const Tensor& L, const Tensor& A, - const Tensor& B, Stream *s, + static void op(const Tensor& A, const Tensor& B, + const Tensor& C, Stream *s, const nnvm::NodeAttrs& attrs) { - if ( A.dptr_ != B.dptr_ ) Copy(B, A, s); + if ( B.dptr_ != C.dptr_ ) Copy(C, B, s); const LaTriangMatrixMultParam& param = nnvm::get(attrs.parsed); - op(L, B, DType(param.alpha), param.rightside, param.transpose, s); + op(A, C, DType(param.alpha), param.rightside, param.lower, param.transpose, s); } template - static void op(const Tensor& L, const Tensor& A, - const Tensor& B, const OpContext& ctx, + static void op(const Tensor& A, const Tensor& B, + const Tensor& C, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(L, A, B, s, attrs); + op(A, B, C, s, attrs); } }; @@ -223,8 +237,8 @@ struct syrk { linalg_batch_syrk(A, B, alpha, beta, tA, s); // Symmetric B is in lower triangle: Copy to upper using namespace mxnet_op; - Kernel::Launch(s, B.MSize(), B.size(1)*B.stride_, - B.stride_, B.dptr_); + Kernel::Launch(s, B.MSize(), B.size(1)*B.stride_, + B.stride_, B.dptr_, false); } template static void op(const Tensor& A, const Tensor& B, @@ -276,8 +290,8 @@ struct gelqf { Tensor QLeft(Qi.dptr_, Shape2(m, m), Qi.stride_, s); Copy(Li, QLeft, s); using namespace mxnet_op; - Kernel::Launch(s, Li.MSize(), m*Li.stride_, Li.stride_, - Li.dptr_); + Kernel::Launch(s, Li.MSize(), m*Li.stride_, Li.stride_, + Li.dptr_, false); // Call orglq: Input is Qi and part of work. Overwrites Qi by final Q // matrix (conversion from internal representation) linalg_orglq(Qi, work, s); @@ -395,117 +409,129 @@ struct gemm2_backward { struct potrf_backward { template - static void op(const Tensor& dL, const Tensor& L, + static void op(const Tensor& dB, const Tensor& B, const Tensor& dA, Stream* s, const nnvm::NodeAttrs& attrs) { - // Backward of L = potrf(A). - // dA = 0.5 * L**T * copyLTU(L**T * dL) * L**(-1) + // Backward of B = potrf(A). + // dA = 0.5 * B**T * copyLTU(B**T * dB) * B**(-1) // Here, copyLTU(M) creates a symmetric matrix from the square matrix M // by setting the upper triangle to be equal to the lower triangle, leaving // lower triangle and diagonal unchanged. - if ( dL.dptr_ != dA.dptr_ ) { - Copy(dA, dL, s); + // The function also handles the case when B is upper triangular by appropriate + // transpositions. + const LaCholeskyParam& param = nnvm::get(attrs.parsed); + if ( dB.dptr_ != dA.dptr_ ) { + Copy(dA, dB, s); } - trmm::op(L, dA, DType(1.0), false, true, s); + trmm::op(B, dA, DType(1.0), !param.lower, param.lower, true, s); using namespace mxnet_op; - Kernel::Launch - (s, dA.MSize(), dA.size(1)*dA.stride_, dA.stride_, dA.dptr_); - trsm::op(L, dA, DType(1.0), false, true, s); - trsm::op(L, dA, DType(0.5), true, false, s); + Kernel::Launch + (s, dA.MSize(), dA.size(1)*dA.stride_, dA.stride_, dA.dptr_, !param.lower); + trsm::op(B, dA, DType(1.0), false, param.lower, param.lower, s); + trsm::op(B, dA, DType(0.5), true, param.lower, !param.lower, s); } template - static void op(const Tensor& dL, const Tensor& L, + static void op(const Tensor& dB, const Tensor& B, const Tensor& dA, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(dL, L, dA, s, attrs); + op(dB, B, dA, s, attrs); } }; struct potri_backward { template - static void op(const Tensor& dA, const Tensor& L, - const Tensor& A, const Tensor& dL, + static void op(const Tensor& dA, const Tensor& B, + const Tensor& A, const Tensor& dB, Stream* s, const nnvm::NodeAttrs& attrs) { - // Backward of A = potri(L). - // dL = -tril( A * (dA + dA**T) * L**(-T)), where tril() extracts lower triangle + // Backward of A = potri(B). + // dB = -tril( A * (dA + dA**T) * B**(-T)), where tril() extracts lower triangle // and diagonal. We must not assume that dA is symmetric. + // The function also handles the case when B is upper triangular by appropriate + // transpositions. // Note: Calling gemm twice here is a bit wasteful, but otherwise the symmetrization // of dA would require temporary memory. - gemm::op(A, dA, dL, DType(1.), DType(0.), false, false, s); - gemm::op(A, dA, dL, DType(1.), DType(1.), false, true, s); - trsm::op(L, dL, DType(-1.), true, true, s); + const LaCholeskyParam& param = nnvm::get(attrs.parsed); + if (param.lower) { + gemm::op(A, dA, dB, DType(1.), DType(0.), false, false, s); + gemm::op(A, dA, dB, DType(1.), DType(1.), false, true, s); + } else { + gemm::op(dA, A, dB, DType(1.), DType(0.), false, false, s); + gemm::op(dA, A, dB, DType(1.), DType(1.), true, false, s); + } + trsm::op(B, dB, DType(-1.), param.lower, param.lower, true, s); using namespace mxnet_op; - Kernel::Launch(s, dL.MSize(), dL.size(1)*dL.stride_, dL.stride_, - dL.dptr_); + Kernel::Launch(s, dB.MSize(), dB.size(1)*dB.stride_, dB.stride_, + dB.dptr_, !param.lower); } template - static void op(const Tensor& dA, const Tensor& L, - const Tensor& A, const Tensor& dL, + static void op(const Tensor& dA, const Tensor& B, + const Tensor& A, const Tensor& dB, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(dA, L, A, dL, s, attrs); + op(dA, B, A, dB, s, attrs); } }; struct trsm_backward { template - static void op(const Tensor& dB, const Tensor& L, - const Tensor& A, const Tensor& B, - const Tensor& dL, const Tensor& dA, + static void op(const Tensor& dC, const Tensor& A, + const Tensor& B, const Tensor& C, + const Tensor& dA, const Tensor& dB, Stream* s, const nnvm::NodeAttrs& attrs) { - // Backward of B = trsm(L,A). + // Backward of C = trsm(A,B). const LaTriangMatrixMultParam& param = nnvm::get(attrs.parsed); + // Compute dB + if ( dB.dptr_ != dC.dptr_ ) Copy(dB, dC, s); + trsm::op(A, dB, DType(param.alpha), param.rightside, param.lower, !param.transpose, s); // Compute dA - if ( dA.dptr_ != dB.dptr_ ) Copy(dA, dB, s); - trsm::op(L, dA, DType(param.alpha), param.rightside, !param.transpose, s); - // Compute dL const bool da_left(param.rightside == param.transpose); DType scale(-1.0/param.alpha); - (da_left ? gemm::op(dA, B, dL, scale, DType(0), param.transpose, !param.transpose, s) - : gemm::op(B, dA, dL, scale, DType(0), !param.transpose, param.transpose, s)); + (da_left ? gemm::op(dB, C, dA, scale, DType(0), param.transpose, !param.transpose, s) + : gemm::op(C, dB, dA, scale, DType(0), !param.transpose, param.transpose, s)); using namespace mxnet_op; - Kernel::Launch(s, dL.MSize(), dL.size(1)*dL.stride_, dL.stride_, dL.dptr_); + Kernel::Launch(s, dA.MSize(), dA.size(1)*dA.stride_, dA.stride_, + dA.dptr_, !param.lower); } template - static void op(const Tensor& dB, const Tensor& L, - const Tensor& A, const Tensor& B, - const Tensor& dL, const Tensor& dA, + static void op(const Tensor& dC, const Tensor& A, + const Tensor& B, const Tensor& C, + const Tensor& dA, const Tensor& dB, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(dB, L, A, B, dL, dA, s, attrs); + op(dC, A, B, C, dA, dB, s, attrs); } }; struct trmm_backward { template - static void op(const Tensor& dB, const Tensor& L, - const Tensor& A, const Tensor& dL, - const Tensor& dA, Stream* s, + static void op(const Tensor& dC, const Tensor& A, + const Tensor& B, const Tensor& dA, + const Tensor& dB, Stream* s, const nnvm::NodeAttrs& attrs) { - // Backward of B = trmm(L,A). + // Backward of C = trmm(A,B). const LaTriangMatrixMultParam& param = nnvm::get(attrs.parsed); - // Compute dL + // Compute dA DType scale(param.alpha); if (param.rightside == param.transpose) { - gemm::op(dB, A, dL, scale, DType(0.), param.transpose, !param.transpose, s); + gemm::op(dC, B, dA, scale, DType(0.), param.transpose, !param.transpose, s); } else { - gemm::op(A, dB, dL, scale, DType(0.), !param.transpose, param.transpose, s); + gemm::op(B, dC, dA, scale, DType(0.), !param.transpose, param.transpose, s); } using namespace mxnet_op; - Kernel::Launch(s, dL.MSize(), dL.size(1)*dL.stride_, dL.stride_, - dL.dptr_); - // Compute dA - if (dA.dptr_ != dB.dptr_) Copy(dA, dB, s); - trmm::op(L, dA, scale, param.rightside, !param.transpose, s); + Kernel::Launch(s, dA.MSize(), dA.size(1)*dA.stride_, dA.stride_, + dA.dptr_, !param.lower); + // Compute dB + if (dB.dptr_ != dC.dptr_) Copy(dB, dC, s); + trmm::op(A, dB, scale, param.rightside, param.lower, !param.transpose, s); } template - static void op(const Tensor& dB, const Tensor& L, - const Tensor& A, const Tensor& dL, - const Tensor& dA, const OpContext& ctx, + static void op(const Tensor& dC, const Tensor& A, + const Tensor& B, const Tensor& dA, + const Tensor& dB, const OpContext& ctx, const nnvm::NodeAttrs& attrs) { Stream *s = ctx.get_stream(); - op(dB, L, A, dL, dA, s, attrs); + op(dC, A, B, dA, dB, s, attrs); } }; @@ -586,13 +612,13 @@ struct gelqf_backward { Tensor tempM = ctx.requested[0] .get_space_typed(dL.shape_, s); Copy(tempM, dL, s); - trmm::op(L, tempM, DType(1.0), false, true, s); + trmm::op(L, tempM, DType(1.0), false, true, true, s); gemm::op(dA, Q, tempM, DType(-1.0), DType(1.0), false, true, s); - Kernel::Launch + Kernel::Launch (s, tempM.MSize(), tempM.size(1)*tempM.stride_, tempM.stride_, - tempM.dptr_); + tempM.dptr_, false); gemm::op(tempM, Q, dA, DType(1.0), DType(1.0), false, false, s); - trsm::op(L, dA, DType(1.0), false, true, s); + trsm::op(L, dA, DType(1.0), false, true, true, s); } }; @@ -672,4 +698,4 @@ struct syevd_backward { } // namespace op } // namespace mxnet -#endif // MXNET_OPERATOR_TENSOR_LA_OP_INLINE_H_ +#endif // MXNET_OPERATOR_TENSOR_LA_OP_INL_H_ diff --git a/src/operator/tensor/la_op.cc b/src/operator/tensor/la_op.cc index b17716502..0f3c2954a 100644 --- a/src/operator/tensor/la_op.cc +++ b/src/operator/tensor/la_op.cc @@ -20,16 +20,17 @@ /*! * Copyright (c) 2017 by Contributors * \file la_op.cc - * \brief CPU-Operators for advanced linear algebra. + * \brief CPU implementation of Operators for advanced linear algebra. */ #include "./la_op.h" -#include "./la_op_inline.h" +#include "./la_op-inl.h" namespace mxnet { namespace op { DMLC_REGISTER_PARAMETER(LaMatrixMacParam); DMLC_REGISTER_PARAMETER(LaMatrixMultParam); +DMLC_REGISTER_PARAMETER(LaCholeskyParam); DMLC_REGISTER_PARAMETER(LaTriangMatrixMultParam); DMLC_REGISTER_PARAMETER(LaSyrkParam); @@ -61,6 +62,11 @@ calls. For example let *A*, *B*, *C* be 5 dimensional tensors. Then gemm(*A*, *B without the overhead of the additional swapaxis operations. +When the input data is of type float32 and the environment variables MXNET_CUDA_ALLOW_TENSOR_CORE +and MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION are set to 1, this operator will try to use +pseudo-float16 precision (float32 math with float16 I/O) precision in order to use +Tensor Cores on suitable NVIDIA GPUs. This can sometimes give significant speedups. + .. note:: The operator supports float32 and float64 data types only. Examples:: @@ -133,6 +139,11 @@ calls. For example let *A*, *B* be 5 dimensional tensors. Then gemm(*A*, *B*, ax without the overhead of the additional swapaxis operations. +When the input data is of type float32 and the environment variables MXNET_CUDA_ALLOW_TENSOR_CORE +and MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION are set to 1, this operator will try to use +pseudo-float16 precision (float32 math with float16 I/O) precision in order to use +Tensor Cores on suitable NVIDIA GPUs. This can sometimes give significant speedups. + .. note:: The operator supports float32 and float64 data types only. Examples:: @@ -178,11 +189,12 @@ NNVM_REGISTER_OP(_linalg_potrf) .describe(R"code(Performs Cholesky factorization of a symmetric positive-definite matrix. Input is a tensor *A* of dimension *n >= 2*. -If *n=2*, the Cholesky factor *L* of the symmetric, positive definite matrix *A* is -computed. *L* is lower triangular (entries of upper triangle are all zero), has +If *n=2*, the Cholesky factor *B* of the symmetric, positive definite matrix *A* is +computed. *B* is triangular (entries of upper or lower triangle are all zero), has positive diagonal entries, and: - *A* = *L* \* *L*\ :sup:`T` + *A* = *B* \* *B*\ :sup:`T` if *lower* = *true* + *A* = *B*\ :sup:`T` \* *B* if *lower* = *false* If *n>2*, *potrf* is performed separately on the trailing two dimensions for all inputs (batch mode). @@ -201,6 +213,7 @@ Examples:: )code" ADD_FILELINE) .set_num_inputs(1) .set_num_outputs(1) +.set_attr_parser(ParamParser) .set_attr("FListInputNames", [](const NodeAttrs& attrs) { return std::vector{"A"}; } ) .set_attr("FInferShape", ElemwiseShape<1, 1>) @@ -214,6 +227,7 @@ Examples:: NNVM_REGISTER_OP(_backward_linalg_potrf) .set_num_inputs(2) .set_num_outputs(1) +.set_attr_parser(ParamParser) .set_attr("FInplaceOption", [](const NodeAttrs& attrs) { return std::vector >{{0, 0}}; }) .set_attr("FResourceRequest", [](const NodeAttrs& attrs) @@ -227,10 +241,11 @@ NNVM_REGISTER_OP(_linalg_potri) .describe(R"code(Performs matrix inversion from a Cholesky factorization. Input is a tensor *A* of dimension *n >= 2*. -If *n=2*, *A* is a lower triangular matrix (entries of upper triangle are all zero) +If *n=2*, *A* is a triangular matrix (entries of upper or lower triangle are all zero) with positive diagonal. We compute: - *out* = *A*\ :sup:`-T` \* *A*\ :sup:`-1` + *out* = *A*\ :sup:`-T` \* *A*\ :sup:`-1` if *lower* = *true* + *out* = *A*\ :sup:`-1` \* *A*\ :sup:`-T` if *lower* = *false* In other words, if *A* is the Cholesky factor of a symmetric positive definite matrix *B* (obtained by *potrf*), then @@ -259,6 +274,7 @@ Examples:: )code" ADD_FILELINE) .set_num_inputs(1) .set_num_outputs(1) +.set_attr_parser(ParamParser) .set_attr("FListInputNames", [](const NodeAttrs& attrs) { return std::vector{"A"}; } ) .set_attr("FInferShape", ElemwiseShape<1, 1>) @@ -272,6 +288,7 @@ Examples:: NNVM_REGISTER_OP(_backward_linalg_potri) .set_num_inputs(3) .set_num_outputs(1) +.set_attr_parser(ParamParser) .set_attr("FResourceRequest", [](const NodeAttrs& attrs) { return std::vector{ResourceRequest::kTempSpace}; }) .set_attr("TIsBackward", true) @@ -283,7 +300,7 @@ NNVM_REGISTER_OP(_linalg_trmm) Input are tensors *A*, *B*, each of dimension *n >= 2* and having the same shape on the leading *n-2* dimensions. -If *n=2*, *A* must be lower triangular. The operator performs the BLAS3 function +If *n=2*, *A* must be triangular. The operator performs the BLAS3 function *trmm*: *out* = *alpha* \* *op*\ (*A*) \* *B* @@ -346,7 +363,7 @@ NNVM_REGISTER_OP(_linalg_trsm) Input are tensors *A*, *B*, each of dimension *n >= 2* and having the same shape on the leading *n-2* dimensions. -If *n=2*, *A* must be lower triangular. The operator performs the BLAS3 function +If *n=2*, *A* must be triangular. The operator performs the BLAS3 function *trsm*, solving for *out* in: *op*\ (*A*) \* *out* = *alpha* \* *B* diff --git a/src/operator/tensor/la_op.cu b/src/operator/tensor/la_op.cu index d736845df..29a484663 100644 --- a/src/operator/tensor/la_op.cu +++ b/src/operator/tensor/la_op.cu @@ -19,10 +19,10 @@ /*! * \file la_op.cu - * \brief GPU-Operators for advanced linear algebra. + * \brief GPU implementation of Operators for advanced linear algebra. */ #include "./la_op.h" -#include "./la_op_inline.h" +#include "./la_op-inl.h" namespace mxnet { namespace op { diff --git a/src/operator/tensor/la_op.h b/src/operator/tensor/la_op.h index 8e2acd747..0327dd19b 100644 --- a/src/operator/tensor/la_op.h +++ b/src/operator/tensor/la_op.h @@ -20,7 +20,7 @@ /*! * Copyright (c) 2017 by Contributors * \file la_op.h - * \brief Operators for advanced linear algebra. + * \brief Function definition of Operators for advanced linear algebra. */ #ifndef MXNET_OPERATOR_TENSOR_LA_OP_H_ #define MXNET_OPERATOR_TENSOR_LA_OP_H_ @@ -81,10 +81,22 @@ struct LaMatrixMultParam : public dmlc::Parameter { } }; +// Parameters for Cholesky factorization and matrix inversion +struct LaCholeskyParam : public dmlc::Parameter { + bool lower; + DMLC_DECLARE_PARAMETER(LaCholeskyParam) { + DMLC_DECLARE_FIELD(lower) + .set_default(true) + .describe + ("True if the triangular matrix is lower triangular, false if it is upper triangular."); + } +}; + // Parameters for matrix-matrix multiplication where one is a triangular matrix. struct LaTriangMatrixMultParam : public dmlc::Parameter { bool transpose; bool rightside; + bool lower; double alpha; DMLC_DECLARE_PARAMETER(LaTriangMatrixMultParam) { DMLC_DECLARE_FIELD(transpose) @@ -93,6 +105,10 @@ struct LaTriangMatrixMultParam : public dmlc::Parameter DMLC_DECLARE_FIELD(rightside) .set_default(false) .describe("Multiply triangular matrix from the right to non-triangular one."); + DMLC_DECLARE_FIELD(lower) + .set_default(true) + .describe + ("True if the triangular matrix is lower triangular, false if it is upper triangular."); DMLC_DECLARE_FIELD(alpha) .set_default(1.0) .describe("Scalar factor to be applied to the result."); diff --git a/src/operator/tensor/matrix_op-inl.h b/src/operator/tensor/matrix_op-inl.h index bfee083ca..3b229cf38 100644 --- a/src/operator/tensor/matrix_op-inl.h +++ b/src/operator/tensor/matrix_op-inl.h @@ -626,9 +626,9 @@ inline void GetIndexRange(const TShape& dshape, const nnvm::Tuple>& param_begin, const nnvm::Tuple>& param_end, const nnvm::Tuple>& param_step, - common::StaticArray* begin, - common::StaticArray* end, - common::StaticArray* step) { + common::StaticArray* begin, + common::StaticArray* end, + common::StaticArray* step) { CHECK_NE(dshape.ndim(), 0U); CHECK_LE(param_begin.ndim(), dshape.ndim()) << "Slicing axis exceeds data dimensions"; @@ -646,8 +646,8 @@ inline void GetIndexRange(const TShape& dshape, } for (index_t i = 0; i < param_begin.ndim(); ++i) { - int b = 0, e = dshape[i], s = 1; - const int len = dshape[i]; + index_t b = 0, e = dshape[i], s = 1; + const index_t len = dshape[i]; if (param_step.ndim() != 0U) { const auto& opt_step_val = param_step[i]; if (opt_step_val.has_value()) { @@ -724,7 +724,7 @@ inline bool SliceOpShape(const nnvm::NodeAttrs& attrs, TShape oshape = dshape; MXNET_NDIM_SWITCH(dshape.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(dshape, param.begin, param.end, param.step, &begin, &end, &step); for (index_t i = 0; i < param.begin.ndim(); ++i) { const int b = begin[i], e = end[i], s = step[i]; @@ -743,19 +743,19 @@ template struct slice_forward { // i is the i-th row after flattening out into 2D tensor template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* data, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* data, const mshadow::Shape dshape, const mshadow::Shape oshape, - const common::StaticArray begin, - const common::StaticArray step) { - const int data_last_dim_size = dshape[ndim-1]; - const int out_last_dim_size = oshape[ndim-1]; - const int step_last_dim = step[ndim-1]; - const int begin_last_dim = begin[ndim-1]; - const int j = i % out_last_dim_size; - int irow = 0; // row id of flattend 2D data - int stride = 1; - int idx = i / out_last_dim_size; + const common::StaticArray begin, + const common::StaticArray step) { + const index_t data_last_dim_size = dshape[ndim-1]; + const index_t out_last_dim_size = oshape[ndim-1]; + const index_t step_last_dim = step[ndim-1]; + const index_t begin_last_dim = begin[ndim-1]; + const index_t j = i % out_last_dim_size; + index_t irow = 0; // row id of flattend 2D data + index_t stride = 1; + index_t idx = i / out_last_dim_size; #pragma unroll for (int k = ndim - 2; k >= 0; --k) { irow += stride * ((idx % oshape[k]) * step[k] + begin[k]); @@ -771,20 +771,20 @@ template struct slice_forward { // i is the i-th row after flattening out into 2D tensor template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* data, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* data, const mshadow::Shape dshape, const mshadow::Shape oshape, - const common::StaticArray begin, - const common::StaticArray step) { - const int data_last_dim_size = dshape[ndim-1]; - const int out_last_dim_size = oshape[ndim-1]; - const int step_last_dim = step[ndim-1]; - const int begin_last_dim = begin[ndim-1]; - int out_offset = i * out_last_dim_size; - for (int j = 0; j < out_last_dim_size; ++j) { - int irow = 0; // row id of flattend 2D data - int stride = 1; - int idx = i; + const common::StaticArray begin, + const common::StaticArray step) { + const index_t data_last_dim_size = dshape[ndim-1]; + const index_t out_last_dim_size = oshape[ndim-1]; + const index_t step_last_dim = step[ndim-1]; + const index_t begin_last_dim = begin[ndim-1]; + index_t out_offset = i * out_last_dim_size; + for (index_t j = 0; j < out_last_dim_size; ++j) { + index_t irow = 0; // row id of flattend 2D data + index_t stride = 1; + index_t idx = i; #pragma unroll for (int k = ndim - 2; k >= 0; --k) { irow += stride * ((idx % oshape[k]) * step[k] + begin[k]); @@ -813,11 +813,11 @@ void SliceOpForward(const nnvm::NodeAttrs& attrs, const TBlob& out = outputs[0]; const SliceParam& param = nnvm::get(attrs.parsed); MXNET_NDIM_SWITCH(data.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(data.shape_, param.begin, param.end, param.step, &begin, &end, &step); MSHADOW_TYPE_SWITCH(out.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], Req, { - int num_threads = out.shape_.FlatTo2D()[0]; + size_t num_threads = out.shape_.FlatTo2D()[0]; if (std::is_same::value) { num_threads *= out.shape_.get()[ndim - 1]; } @@ -836,20 +836,20 @@ template struct slice_assign { // i is the i-th row after flattening out into 2D tensor template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* val, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* val, const mshadow::Shape oshape, const mshadow::Shape vshape, - const common::StaticArray begin, - const common::StaticArray step) { - const int data_last_dim_size = oshape[ndim-1]; - const int out_last_dim_size = vshape[ndim-1]; - const int step_last_dim = step[ndim-1]; - const int begin_last_dim = begin[ndim-1]; - int offset = i * out_last_dim_size; - for (int j = 0; j < out_last_dim_size; ++j) { - int irow = 0; // row id of flattend 2D out - int stride = 1; - int idx = i; + const common::StaticArray begin, + const common::StaticArray step) { + const index_t data_last_dim_size = oshape[ndim-1]; + const index_t out_last_dim_size = vshape[ndim-1]; + const index_t step_last_dim = step[ndim-1]; + const index_t begin_last_dim = begin[ndim-1]; + index_t offset = i * out_last_dim_size; + for (index_t j = 0; j < out_last_dim_size; ++j) { + index_t irow = 0; // row id of flattend 2D out + index_t stride = 1; + index_t idx = i; #pragma unroll for (int k = ndim - 2; k >= 0; --k) { irow += stride * ((idx % vshape[k]) * step[k] + begin[k]); @@ -866,19 +866,19 @@ template struct slice_assign { // i is the i-th row after flattening out into 2D tensor template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* val, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* val, const mshadow::Shape oshape, const mshadow::Shape vshape, - const common::StaticArray begin, - const common::StaticArray step) { - const int data_last_dim_size = oshape[ndim-1]; - const int out_last_dim_size = vshape[ndim-1]; - const int step_last_dim = step[ndim-1]; - const int begin_last_dim = begin[ndim-1]; - const int j = i % out_last_dim_size; - int irow = 0; // row id of flattend 2D out - int stride = 1; - int idx = i / out_last_dim_size; + const common::StaticArray begin, + const common::StaticArray step) { + const index_t data_last_dim_size = oshape[ndim-1]; + const index_t out_last_dim_size = vshape[ndim-1]; + const index_t step_last_dim = step[ndim-1]; + const index_t begin_last_dim = begin[ndim-1]; + const index_t j = i % out_last_dim_size; + index_t irow = 0; // row id of flattend 2D out + index_t stride = 1; + index_t idx = i / out_last_dim_size; #pragma unroll for (int k = ndim - 2; k >= 0; --k) { irow += stride * ((idx % vshape[k]) * step[k] + begin[k]); @@ -911,7 +911,7 @@ void SliceOpBackward(const nnvm::NodeAttrs& attrs, LOG(FATAL) << "_slice_backward does not support kWriteInplace"; } MXNET_NDIM_SWITCH(ograd.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(igrad.shape_, param.begin, param.end, param.step, &begin, &end, &step); MSHADOW_TYPE_SWITCH(ograd.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], Req, { @@ -937,7 +937,7 @@ inline bool SliceAssignOpShape(const nnvm::NodeAttrs& attrs, TShape vshape = dshape; // vshape is the value shape on the right hand side const SliceParam& param = nnvm::get(attrs.parsed); MXNET_NDIM_SWITCH(dshape.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(dshape, param.begin, param.end, param.step, &begin, &end, &step); for (index_t i = 0; i < param.begin.ndim(); ++i) { const int b = begin[i], e = end[i], s = step[i]; @@ -975,7 +975,7 @@ void SliceAssignOpForward(const nnvm::NodeAttrs& attrs, const SliceParam& param = nnvm::get(attrs.parsed); MXNET_NDIM_SWITCH(data.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(data.shape_, param.begin, param.end, param.step, &begin, &end, &step); MSHADOW_TYPE_SWITCH(out.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], Req, { @@ -992,7 +992,7 @@ void SliceAssignOpForward(const nnvm::NodeAttrs& attrs, } struct SliceAssignScalarParam : public dmlc::Parameter { - real_t scalar; + double scalar; nnvm::Tuple> begin, end; nnvm::Tuple> step; DMLC_DECLARE_PARAMETER(SliceAssignScalarParam) { @@ -1024,20 +1024,20 @@ template struct slice_assign_scalar { // i is the i-th row after flattening out into 2D tensor template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType val, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType val, const OpReqType req, const mshadow::Shape oshape, const mshadow::Shape vshape, - const common::StaticArray begin, - const common::StaticArray step) { - const int data_last_dim_size = oshape[ndim-1]; - const int out_last_dim_size = vshape[ndim-1]; - const int step_last_dim = step[ndim-1]; - const int begin_last_dim = begin[ndim-1]; - for (int j = 0; j < out_last_dim_size; ++j) { - int irow = 0; // row id of flattend 2D out - int stride = 1; - int idx = i; + const common::StaticArray begin, + const common::StaticArray step) { + const index_t data_last_dim_size = oshape[ndim-1]; + const index_t out_last_dim_size = vshape[ndim-1]; + const index_t step_last_dim = step[ndim-1]; + const index_t begin_last_dim = begin[ndim-1]; + for (index_t j = 0; j < out_last_dim_size; ++j) { + index_t irow = 0; // row id of flattend 2D out + index_t stride = 1; + index_t idx = i; #pragma unroll for (int k = ndim - 2; k >= 0; --k) { irow += stride * ((idx % vshape[k]) * step[k] + begin[k]); @@ -1076,7 +1076,7 @@ void SliceAssignScalarOpForward(const nnvm::NodeAttrs& attrs, TShape vshape = data.shape_; const SliceAssignScalarParam& param = nnvm::get(attrs.parsed); MXNET_NDIM_SWITCH(data.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(data.shape_, param.begin, param.end, param.step, &begin, &end, &step); for (index_t i = 0; i < param.begin.ndim(); ++i) { const int b = begin[i], e = end[i], s = step[i]; @@ -1107,7 +1107,7 @@ struct SliceAxisParam : public dmlc::Parameter { }; inline void GetSliceAxisParams(const SliceAxisParam& param, const TShape& ishape, - int* axis, int* begin, int* end) { + int* axis, index_t* begin, index_t* end) { *axis = param.axis; if (*axis < 0) { *axis += static_cast(ishape.ndim()); @@ -1115,7 +1115,7 @@ inline void GetSliceAxisParams(const SliceAxisParam& param, const TShape& ishape CHECK(*axis < static_cast(ishape.ndim()) && *axis >= 0) << "Transformed axis must be smaller than the source ndim and larger than zero! Recieved axis=" << param.axis << ", src_ndim=" << ishape.ndim() << ", transformed axis=" << *axis; - int axis_size = static_cast(ishape[*axis]); + index_t axis_size = static_cast(ishape[*axis]); *begin = param.begin; *end = -1; if (*begin < 0) { @@ -1149,7 +1149,8 @@ inline bool SliceAxisShape(const nnvm::NodeAttrs& attrs, CHECK_EQ(in_attrs->size(), 1U); CHECK_EQ(out_attrs->size(), 1U); TShape& ishape = (*in_attrs)[0]; - int axis, begin, end; + int axis; + index_t begin, end; GetSliceAxisParams(param, ishape, &axis, &begin, &end); TShape shape(ishape.ndim()); for (index_t i = 0; i < ishape.ndim(); ++i) { @@ -1173,7 +1174,8 @@ void SliceAxis(const nnvm::NodeAttrs& attrs, using namespace mshadow::expr; const SliceAxisParam& param = nnvm::get(attrs.parsed); mshadow::Stream *s = ctx.get_stream(); - int axis, begin, end; + int axis; + index_t begin, end; GetSliceAxisParams(param, inputs[0].shape_, &axis, &begin, &end); int ndim = static_cast(outputs[0].ndim()); @@ -1207,7 +1209,8 @@ void SliceAxisGrad_(const nnvm::NodeAttrs& attrs, using namespace mshadow::op; using namespace mshadow::expr; mshadow::Stream *s = ctx.get_stream(); - int axis, begin, end; + int axis; + index_t begin, end; GetSliceAxisParams(param, outputs[0].shape_, &axis, &begin, &end); int ndim = static_cast(outputs[0].shape_.ndim()); @@ -1354,7 +1357,7 @@ void SliceLikeForward(const nnvm::NodeAttrs& attrs, SliceLikeInferRanges(ishape, from_shape, param.axes, ¶m_begin, ¶m_end, ¶m_step); MXNET_NDIM_SWITCH(data.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(data.shape_, param_begin, param_end, param_step, &begin, &end, &step); MSHADOW_TYPE_SWITCH(out.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], Req, { @@ -1400,7 +1403,7 @@ void SliceLikeBackward(const nnvm::NodeAttrs& attrs, SliceLikeInferRanges(ishape, from_shape, param.axes, ¶m_begin, ¶m_end, ¶m_step); MXNET_NDIM_SWITCH(ograd.ndim(), ndim, { - common::StaticArray begin, end, step; + common::StaticArray begin, end, step; GetIndexRange(ograd.shape_, param_begin, param_end, param_step, &begin, &end, &step); MSHADOW_TYPE_SWITCH(ograd.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], Req, { @@ -1429,7 +1432,7 @@ struct ClipParam : public dmlc::Parameter { struct clip { template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* datas, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* datas, DType a_min, DType a_max) { DType data = datas[i]; if (data > a_max) { @@ -1445,7 +1448,7 @@ struct clip { struct clip_grad { template - MSHADOW_XINLINE static void Map(int i, DType* out, const DType* grad, const DType* datas, + MSHADOW_XINLINE static void Map(index_t i, DType* out, const DType* grad, const DType* datas, DType a_min, DType a_max) { DType data = datas[i]; if (data > a_max) { @@ -1934,7 +1937,7 @@ struct reverse { } #ifdef __CUDACC__ template - __device__ static void Map(int index, index_t nreversedim, const DType *src, DType *dst, + __device__ static void Map(index_t index, index_t nreversedim, const DType *src, DType *dst, const index_t * stride_, const index_t * trailing_) { __shared__ index_t stride_share[REVERSE_MAX_DIM]; @@ -1949,7 +1952,7 @@ struct reverse { } #else template - MSHADOW_XINLINE static void Map(int index, index_t nreversedim, const DType *src, DType *dst, + MSHADOW_XINLINE static void Map(index_t index, index_t nreversedim, const DType *src, DType *dst, const index_t * stride_, const index_t * trailing_) { index_t new_idx = ReverseIndex(index, nreversedim, stride_, trailing_); @@ -1977,11 +1980,11 @@ void ReverseOpForward(const nnvm::NodeAttrs& attrs, std::vector stride_(param.axis.ndim()); std::vector trailing_(param.axis.ndim()); index_t reverse_index = 0; - for (auto axis_iter = param.axis.begin() ; axis_iter!= param.axis.end(); ++axis_iter) { - CHECK_LT(*axis_iter, static_cast(ishape.ndim())); - stride_[reverse_index] = ishape[*axis_iter]; + for (int axis : param.axis) { + CHECK_LT(axis, static_cast(ishape.ndim())); + stride_[reverse_index] = ishape[axis]; trailing_[reverse_index] = 1; - for (index_t i2 = *axis_iter + 1; i2 < ishape.ndim(); ++i2) { + for (index_t i2 = axis + 1; i2 < ishape.ndim(); ++i2) { trailing_[reverse_index] *= ishape[i2]; } reverse_index++; @@ -2141,10 +2144,10 @@ struct SqueezeParam : public dmlc::Parameter { // move all the zeros to the last of the shape array // and keep the relative order of the non-zero values. // Returns the new shape size after moving all zeros to the end. -inline uint32_t SqueezeShapeHelper(TShape* shape) { +inline size_t SqueezeShapeHelper(TShape* shape) { CHECK(shape != nullptr); - uint32_t count = 0; - for (uint32_t i = 0; i < shape->ndim(); ++i) { + size_t count = 0; + for (size_t i = 0; i < shape->ndim(); ++i) { if ((*shape)[i] == 0) { ++count; } else { @@ -2167,7 +2170,7 @@ inline bool SqueezeShape(const nnvm::NodeAttrs& attrs, if (param.axis.has_value()) { // preprocess axis TShape axes = param.axis.value(); - for (uint32_t i = 0; i < axes.ndim(); ++i) { + for (size_t i = 0; i < axes.ndim(); ++i) { if (axes[i] < 0) { axes[i] += dndim; CHECK_GE(axes[i], 0) @@ -2182,11 +2185,11 @@ inline bool SqueezeShape(const nnvm::NodeAttrs& attrs, oshape[axes[i]] = 0; } } else { - for (uint32_t i = 0; i < oshape.ndim(); ++i) { + for (size_t i = 0; i < oshape.ndim(); ++i) { if (oshape[i] == 1) oshape[i] = 0; } } - uint32_t oshape_size = SqueezeShapeHelper(&oshape); + size_t oshape_size = SqueezeShapeHelper(&oshape); if (oshape_size == 0) { // corner case when dshape is (1, 1, 1, 1) oshape[0] = 1; oshape_size = 1; @@ -2229,7 +2232,7 @@ inline bool DepthToSpaceOpShape(const nnvm::NodeAttrs& attrs, expected_out[0] = in_shape[0]; expected_out[1] = in_shape[1] / (block * block); - uint32_t i = 2; + size_t i = 2; while (i < expected_out.ndim()) { expected_out[i] = in_shape[i] * block; ++i; @@ -2259,9 +2262,9 @@ inline bool DepthToSpaceOpType(const nnvm::NodeAttrs& attrs, * \param inp_index index within input tensor from where value is retrieved * \param offset_arr array containing the linear offset of input tensor */ -MSHADOW_XINLINE void update_index(int index_position, int dim_size, int *idx, - int *inp_index, const int* offset_arr) { - int next_idx_val = *idx / dim_size; +MSHADOW_XINLINE void update_index(index_t index_position, index_t dim_size, index_t *idx, + index_t *inp_index, const index_t* offset_arr) { + index_t next_idx_val = *idx / dim_size; *inp_index += (*idx - next_idx_val * dim_size) * offset_arr[index_position]; *idx = next_idx_val; } @@ -2280,9 +2283,9 @@ MSHADOW_XINLINE void update_index(int index_position, int dim_size, int *idx, template struct depth_to_space_forward { template - MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, - const int block, const int* size, const int* offset_arr) { - int inp_index = 0, idx = i, dim_size; + MSHADOW_XINLINE static void Map(index_t i, DType* out_data, const DType* in_data, + const int block, const index_t* size, const index_t* offset_arr) { + index_t inp_index = 0, idx = i, dim_size; dim_size = block; update_index(2, dim_size, &idx, &inp_index, offset_arr); dim_size = size[3]; @@ -2315,9 +2318,9 @@ struct depth_to_space_forward { template struct compute_offset_for_depth_to_space { template - MSHADOW_XINLINE static void Map(int i, DType* offset_arr, DType* size, const int block, - const int32_t size0, const int32_t size1, const int32_t size2, - const int32_t size3) { + MSHADOW_XINLINE static void Map(index_t i, DType* offset_arr, DType* size, const int block, + const index_t size0, const index_t size1, const index_t size2, + const index_t size3) { size[0] = size0; size[1] = size1; size[2] = size2; @@ -2349,10 +2352,10 @@ void DepthToSpaceOpForward(const nnvm::NodeAttrs& attrs, int block = param.block_size; mshadow::Tensor workspace = - ctx.requested[0].get_space_typed(mshadow::Shape1(sizeof(int32_t) * 10), s); + ctx.requested[0].get_space_typed(mshadow::Shape1(sizeof(index_t) * 10), s); char* workspace_curr_ptr = workspace.dptr_; - int32_t* offset_arr = reinterpret_cast(workspace_curr_ptr); - int32_t* size = reinterpret_cast(workspace_curr_ptr + sizeof(int32_t) * 6); + index_t* offset_arr = reinterpret_cast(workspace_curr_ptr); + index_t* size = reinterpret_cast(workspace_curr_ptr + sizeof(index_t) * 6); MSHADOW_TYPE_SWITCH(out_data.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], req_type, { @@ -2431,9 +2434,9 @@ inline bool SpaceToDepthOpType(const nnvm::NodeAttrs& attrs, template struct space_to_depth_forward { template - MSHADOW_XINLINE static void Map(int i, DType* out_data, const DType* in_data, const int block, - const int* size, const int* offset_arr) { - int inp_index = 0, idx = i, dim_size; + MSHADOW_XINLINE static void Map(index_t i, DType* out_data, const DType* in_data, const int block, + const index_t* size, const index_t* offset_arr) { + index_t inp_index = 0, idx = i, dim_size; dim_size = size[3] / block; update_index(4, dim_size, &idx, &inp_index, offset_arr); dim_size = size[2] / block; @@ -2466,9 +2469,9 @@ struct space_to_depth_forward { template struct compute_offset_for_space_to_depth { template - MSHADOW_XINLINE static void Map(int i, DType* offset_arr, DType* size, const int block, - const int32_t size0, const int32_t size1, - const int32_t size2, const int32_t size3) { + MSHADOW_XINLINE static void Map(index_t i, DType* offset_arr, DType* size, const int block, + const index_t size0, const index_t size1, + const index_t size2, const index_t size3) { size[0] = size0; size[1] = size1; size[2] = size2; @@ -2500,10 +2503,10 @@ void SpaceToDepthOpForward(const nnvm::NodeAttrs& attrs, int block = param.block_size; mshadow::Tensor workspace = - ctx.requested[0].get_space_typed(mshadow::Shape1(sizeof(int32_t) * 10), s); + ctx.requested[0].get_space_typed(mshadow::Shape1(sizeof(index_t) * 10), s); char* workspace_curr_ptr = workspace.dptr_; - int32_t* offset_arr = reinterpret_cast(workspace_curr_ptr); - int32_t* size = reinterpret_cast(workspace_curr_ptr + sizeof(int32_t) * 6); + index_t* offset_arr = reinterpret_cast(workspace_curr_ptr); + index_t* size = reinterpret_cast(workspace_curr_ptr + sizeof(index_t) * 6); MSHADOW_TYPE_SWITCH(out_data.type_flag_, DType, { MXNET_ASSIGN_REQ_SWITCH(req[0], req_type, { diff --git a/src/operator/tensor/matrix_op.cc b/src/operator/tensor/matrix_op.cc index 77d9bf06e..db8efa454 100644 --- a/src/operator/tensor/matrix_op.cc +++ b/src/operator/tensor/matrix_op.cc @@ -103,6 +103,57 @@ DMLC_REGISTER_PARAMETER(StackParam); DMLC_REGISTER_PARAMETER(SqueezeParam); DMLC_REGISTER_PARAMETER(DepthToSpaceParam); +#if MXNET_USE_MKLDNN == 1 +void MKLDNNReshape(const NDArray &in_data, const NDArray &out_data) { + MSHADOW_TYPE_SWITCH(in_data.dtype(), DType, { + auto this_mem = in_data.GetMKLDNNData(); + auto out_dptr = out_data.data().dptr(); + mkldnn::memory::primitive_desc this_pd = this_mem->get_primitive_desc(); + mkldnn::memory::desc this_desc = this_pd.desc(); + mkldnn::memory::dims dims(this_desc.data.dims, + this_desc.data.dims + this_desc.data.ndims); + auto this_dtype = static_cast(this_desc.data.data_type); + auto this_format = static_cast(GetDefaultFormat(this_desc)); + mkldnn::memory::desc data_md(dims, this_dtype, this_format); + mkldnn::memory::primitive_desc pd(data_md, this_pd.get_engine()); + auto temp_mem = mkldnn::memory(pd, out_dptr); + MKLDNNStream::Get()->RegisterPrim(mkldnn::reorder(*this_mem, temp_mem)); + MKLDNNStream::Get()->Submit(); + + // Removing out_data mkl_mem_ and store data in the default format + const_cast(out_data).InvalidateMKLDNNData(); + }); +} + +static void ReshapeComputeExCPU(const nnvm::NodeAttrs& attrs, + const OpContext& ctx, + const std::vector& inputs, + const std::vector& req, + const std::vector& outputs) { + CHECK_EQ(inputs.size(), 1U); + CHECK_EQ(outputs.size(), 1U); + // If inputs are supposed to be in MKLDNN format and + // MKLDNNsupport the data type or the shape. Then convert + // it to the output format and shape + if (SupportMKLDNNArray(inputs[0].dtype(), inputs[0].shape()) && req[0] != kAddTo) { + MKLDNNReshape(inputs[0], outputs[0]); + return; + } + FallBackCompute(UnaryOp::IdentityCompute, attrs, ctx, inputs, req, outputs); +} + +inline static bool ReshapeStorageType(const nnvm::NodeAttrs& attrs, + const int dev_mask, + DispatchMode* dispatch_mode, + std::vector* in_attrs, + std::vector* out_attrs) { + CHECK_EQ(in_attrs->size(), 1U); + CHECK_EQ(out_attrs->size(), 1U); + return MKLDNNStorageType(attrs, dev_mask, true, dispatch_mode, in_attrs, + out_attrs); +} +#endif + NNVM_REGISTER_OP(Reshape) .add_alias("reshape") .describe(R"code(Reshapes the input array. @@ -172,8 +223,16 @@ If the argument `reverse` is set to 1, then the special values are inferred from .set_attr_parser(ParamParser) .set_attr("FInferShape", ReshapeShape) .set_attr("FInferType", ElemwiseType<1, 1>) -.set_attr("FGradient", ElemwiseGradUseNone{"_backward_copy"}) +.set_attr("FGradient", ElemwiseGradUseNone{"_backward_reshape"}) .set_attr("FCompute", UnaryOp::IdentityCompute) +#if MXNET_USE_MKLDNN == 1 +.set_attr("TIsMKLDNN", true) +.set_attr("FComputeEx", ReshapeComputeExCPU) +.set_attr("FInferStorageType", ReshapeStorageType) +.set_attr("FResourceRequest", [](const NodeAttrs& n) { + return std::vector{ResourceRequest::kTempSpace}; +}) +#else .set_attr("FInplaceOption", [](const NodeAttrs& attrs) { return std::vector >{{0, 0}}; @@ -182,6 +241,7 @@ If the argument `reverse` is set to 1, then the special values are inferred from [](const NodeAttrs& attrs){ return std::vector{true}; }) +#endif .add_argument("data", "NDArray-or-Symbol", "Input data to reshape.") .add_arguments(ReshapeParam::__FIELDS__()); @@ -210,6 +270,7 @@ static void FlattenEx(const nnvm::NodeAttrs& attrs, #endif } +#if MXNET_USE_MKLDNN == 1 static inline bool FlattenStorageType(const nnvm::NodeAttrs& attrs, const int dev_mask, DispatchMode* dispatch_mode, @@ -217,17 +278,10 @@ static inline bool FlattenStorageType(const nnvm::NodeAttrs& attrs, std::vector *out_attrs) { CHECK_EQ(in_attrs->size(), 1); CHECK_EQ(out_attrs->size(), 1); - bool ret = ElemwiseStorageType<1, 1, false, false, false>(attrs, dev_mask, dispatch_mode, - in_attrs, out_attrs); -#if MXNET_USE_MKLDNN == 1 - if (dev_mask == mshadow::cpu::kDevMask - && in_attrs->at(0) == kDefaultStorage - && out_attrs->at(0) == kDefaultStorage) { - *dispatch_mode = DispatchMode::kFComputeEx; - } -#endif - return ret; + return MKLDNNStorageType(attrs, dev_mask, true, dispatch_mode, in_attrs, + out_attrs); } +#endif NNVM_REGISTER_OP(Flatten) .add_alias("flatten") @@ -261,7 +315,9 @@ Example:: .set_num_outputs(1) .set_attr("FInferShape", FlattenShape) .set_attr("FInferType", ElemwiseType<1, 1>) +#if MXNET_USE_MKLDNN == 1 .set_attr("FInferStorageType", FlattenStorageType) +#endif .set_attr("FGradient", ElemwiseGradUseNone{ "_backward_copy" }) .set_attr("FCompute", UnaryOp::IdentityCompute) .set_attr("FComputeEx", FlattenEx) @@ -359,7 +415,7 @@ will return a new array with shape ``(2,1,3,4)``. [](const NodeAttrs& attrs){ return std::vector{true}; }) -.set_attr("FGradient", ElemwiseGradUseNone{"_backward_copy"}) +.set_attr("FGradient", ElemwiseGradUseNone{"_backward_reshape"}) .set_attr("FCompute", UnaryOp::IdentityCompute) .add_argument("data", "NDArray-or-Symbol", "Source input") .add_arguments(ExpandDimParam::__FIELDS__()); @@ -396,9 +452,9 @@ The storage type of ``slice`` output depends on storage types of inputs - otherwise, ``slice`` generates output with default storage .. note:: When input data storage type is csr, it only supports -step=(), or step=(None,), or step=(1,) to generate a csr output. -For other step parameter values, it falls back to slicing -a dense tensor. + step=(), or step=(None,), or step=(1,) to generate a csr output. + For other step parameter values, it falls back to slicing + a dense tensor. Example:: diff --git a/src/operator/tensor/ordering_op-inl.h b/src/operator/tensor/ordering_op-inl.h index c076f90d7..1847a533d 100644 --- a/src/operator/tensor/ordering_op-inl.h +++ b/src/operator/tensor/ordering_op-inl.h @@ -20,7 +20,7 @@ /*! * Copyright (c) 2016 by Contributors * \file ordering_op-inl.h - * \brief Function definition of matrix related operators + * \brief Function definition of ordering operators */ #ifndef MXNET_OPERATOR_TENSOR_ORDERING_OP_INL_H_ #define MXNET_OPERATOR_TENSOR_ORDERING_OP_INL_H_ diff --git a/src/operator/tensor/ordering_op.cc b/src/operator/tensor/ordering_op.cc index 189ea19fa..fb0029959 100644 --- a/src/operator/tensor/ordering_op.cc +++ b/src/operator/tensor/ordering_op.cc @@ -19,7 +19,7 @@ /*! * Copyright (c) 2016 by Contributors - * \file ordering.cc + * \file ordering_op.cc * \brief CPU Implementation of the ordering operations */ // this will be invoked by gcc and compile CPU version diff --git a/src/operator/tensor/ordering_op.cu b/src/operator/tensor/ordering_op.cu index d78361448..db47913e9 100644 --- a/src/operator/tensor/ordering_op.cu +++ b/src/operator/tensor/ordering_op.cu @@ -19,8 +19,8 @@ /*! * Copyright (c) 2015 by Contributors - * \file matrix_op.cu - * \brief GPU Implementation of matrix operations + * \file ordering_op.cu + * \brief GPU Implementation of ordering operations */ // this will be invoked by gcc and compile GPU version #include "./ordering_op-inl.h" diff --git a/src/operator/tensor/ravel.cc b/src/operator/tensor/ravel.cc index 94e389484..7bbfac5d5 100644 --- a/src/operator/tensor/ravel.cc +++ b/src/operator/tensor/ravel.cc @@ -20,7 +20,7 @@ /*! * Copyright (c) 2018 by Contributors * \file ravel.cc - * \brief CPU-operators for ravel/unravel. + * \brief CPU implementation of operators for ravel/unravel. */ #include "./ravel.h" diff --git a/src/operator/tensor/ravel.cu b/src/operator/tensor/ravel.cu index cae504820..860fd2378 100644 --- a/src/operator/tensor/ravel.cu +++ b/src/operator/tensor/ravel.cu @@ -19,7 +19,7 @@ /*! * \file ravel.cu - * \brief GPU-Operators for ravel/unravel. + * \brief GPU implementation of Operators for ravel/unravel. */ #include "./ravel.h" diff --git a/src/operator/tensor/sparse_retain-inl.h b/src/operator/tensor/sparse_retain-inl.h index 5a8aee868..52401beba 100644 --- a/src/operator/tensor/sparse_retain-inl.h +++ b/src/operator/tensor/sparse_retain-inl.h @@ -19,7 +19,7 @@ /*! * \file sparse_retain-inl.h - * \brief + * \brief Function definition of sparse_retain operator */ #ifndef MXNET_OPERATOR_TENSOR_SPARSE_RETAIN_INL_H_ #define MXNET_OPERATOR_TENSOR_SPARSE_RETAIN_INL_H_ diff --git a/src/operator/tensor/sparse_retain.cc b/src/operator/tensor/sparse_retain.cc index f8fc325c0..8630457e6 100644 --- a/src/operator/tensor/sparse_retain.cc +++ b/src/operator/tensor/sparse_retain.cc @@ -19,7 +19,7 @@ /*! * \file sparse_retain.cc - * \brief + * \brief CPU implementation of sparse_retain operator */ #include "./sparse_retain-inl.h" diff --git a/src/operator/tensor/sparse_retain.cu b/src/operator/tensor/sparse_retain.cu index 6b4ac1bdf..bff8b8b12 100644 --- a/src/operator/tensor/sparse_retain.cu +++ b/src/operator/tensor/sparse_retain.cu @@ -19,7 +19,7 @@ /*! * \file sparse_retain.cu - * \brief + * \brief GPU implementation of sparse_retain operator */ #include "./sparse_retain-inl.h" diff --git a/src/profiler/aggregate_stats.cc b/src/profiler/aggregate_stats.cc index 66f833e69..4c135beaf 100644 --- a/src/profiler/aggregate_stats.cc +++ b/src/profiler/aggregate_stats.cc @@ -51,10 +51,9 @@ void AggregateStats::Dump(std::ostream& os, bool clear) { << "\tNote that counter items are counter values and not time units." << std::endl; std::unique_lock lk(m_); - for (auto type_iter = stats_.begin(), type_e_iter = stats_.end(); - type_iter != type_e_iter; ++type_iter) { - const std::string& type = type_iter->first; - const std::unordered_map& mm = type_iter->second; + for (const auto& stat : stats_) { + const std::string& type = stat.first; + const std::unordered_map& mm = stat.second; if (!mm.empty()) { os << type << std::endl << "=================" << std::endl; os << std::setw(25) << std::left << "Name" @@ -87,10 +86,10 @@ void AggregateStats::Dump(std::ostream& os, bool clear) { << std::setw(16) << std::right << "-------------" << std::endl; - for (auto iter = mm.begin(), e_iter = mm.end(); iter != e_iter; ++iter) { - const StatData &data = iter->second; + for (const auto& iter : mm) { + const StatData &data = iter.second; if (data.type_ == StatData::kDuration || data.type_ == StatData::kCounter) { - const std::string &name = iter->first; + const std::string &name = iter.first; os << std::setw(25) << std::left << name << std::setw(16) << std::right << data.total_count_; os << " " diff --git a/src/storage/cpu_device_storage.h b/src/storage/cpu_device_storage.h index a00a78b43..e2797ad01 100644 --- a/src/storage/cpu_device_storage.h +++ b/src/storage/cpu_device_storage.h @@ -43,12 +43,12 @@ class CPUDeviceStorage { * \param size Size to allocate. * \return Pointer to the storage. */ - inline static void* Alloc(size_t size); + inline static void* Alloc(Storage::Handle* handle); /*! * \brief Deallocation. * \param ptr Pointer to deallocate. */ - inline static void Free(void* ptr); + inline static void Free(Storage::Handle handle); private: /*! @@ -66,7 +66,8 @@ class CPUDeviceStorage { #endif }; // class CPUDeviceStorage -inline void* CPUDeviceStorage::Alloc(size_t size) { +inline void* CPUDeviceStorage::Alloc(Storage::Handle* handle) { + const size_t size = handle->size; void* ptr; #if _MSC_VER ptr = _aligned_malloc(size, alignment_); @@ -78,7 +79,8 @@ inline void* CPUDeviceStorage::Alloc(size_t size) { return ptr; } -inline void CPUDeviceStorage::Free(void* ptr) { +inline void CPUDeviceStorage::Free(Storage::Handle handle) { + void * ptr = handle.dptr; #if _MSC_VER _aligned_free(ptr); #else diff --git a/src/storage/gpu_device_storage.h b/src/storage/gpu_device_storage.h index 435c7e81d..562badb87 100644 --- a/src/storage/gpu_device_storage.h +++ b/src/storage/gpu_device_storage.h @@ -46,17 +46,19 @@ class GPUDeviceStorage { * \param size Size to allocate. * \return Pointer to the storage. */ - inline static void* Alloc(size_t size); + inline static void* Alloc(Storage::Handle* handle); /*! * \brief Deallocation. * \param ptr Pointer to deallocate. */ - inline static void Free(void* ptr); + inline static void Free(Storage::Handle handle); }; // class GPUDeviceStorage -inline void* GPUDeviceStorage::Alloc(size_t size) { +inline void* GPUDeviceStorage::Alloc(Storage::Handle* handle) { + const size_t size = handle->size; void* ret = nullptr; #if MXNET_USE_CUDA + mxnet::common::cuda::DeviceStore device_store(handle->ctx.real_dev_id(), true); #if MXNET_USE_NCCL std::lock_guard l(Storage::Get()->GetMutex(Context::kGPU)); #endif // MXNET_USE_NCCL @@ -69,8 +71,10 @@ inline void* GPUDeviceStorage::Alloc(size_t size) { return ret; } -inline void GPUDeviceStorage::Free(void* ptr) { +inline void GPUDeviceStorage::Free(Storage::Handle handle) { #if MXNET_USE_CUDA + void * ptr = handle.dptr; + mxnet::common::cuda::DeviceStore device_store(handle.ctx.real_dev_id(), true); #if MXNET_USE_NCCL std::lock_guard l(Storage::Get()->GetMutex(Context::kGPU)); #endif // MXNET_USE_NCCL diff --git a/src/storage/naive_storage_manager.h b/src/storage/naive_storage_manager.h index b05b242a7..55112b5a8 100644 --- a/src/storage/naive_storage_manager.h +++ b/src/storage/naive_storage_manager.h @@ -49,7 +49,7 @@ class NaiveStorageManager final : public StorageManager { void Free(Storage::Handle handle) override; void DirectFree(Storage::Handle handle) override { - DeviceStorage::Free(handle.dptr); + DeviceStorage::Free(handle); } private: @@ -58,12 +58,12 @@ class NaiveStorageManager final : public StorageManager { template void NaiveStorageManager::Alloc(Storage::Handle* handle) { - handle->dptr = DeviceStorage::Alloc(handle->size); + handle->dptr = DeviceStorage::Alloc(handle); } template void NaiveStorageManager::Free(Storage::Handle handle) { - DeviceStorage::Free(handle.dptr); + DeviceStorage::Free(handle); } } // namespace storage diff --git a/src/storage/pinned_memory_storage.h b/src/storage/pinned_memory_storage.h index e3fec2f4a..c4ababbdc 100644 --- a/src/storage/pinned_memory_storage.h +++ b/src/storage/pinned_memory_storage.h @@ -41,29 +41,33 @@ class PinnedMemoryStorage { * \param size Size to allocate. * \return Pointer to the storage. */ - inline static void* Alloc(size_t size); + inline static void* Alloc(Storage::Handle* handle); /*! * \brief Deallocation. * \param ptr Pointer to deallocate. */ - inline static void Free(void* ptr); + inline static void Free(Storage::Handle handle); }; -inline void* PinnedMemoryStorage::Alloc(size_t size) { +inline void* PinnedMemoryStorage::Alloc(Storage::Handle* handle) { void* ret = nullptr; + const size_t size = handle->size; #if MXNET_USE_NCCL std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); #endif + mxnet::common::cuda::DeviceStore device_store(handle->ctx.real_dev_id(), true); // make the memory available across all devices CUDA_CALL(cudaHostAlloc(&ret, size, cudaHostAllocPortable)); return ret; } -inline void PinnedMemoryStorage::Free(void* ptr) { +inline void PinnedMemoryStorage::Free(Storage::Handle handle) { + void * ptr = handle.dptr; #if MXNET_USE_NCCL std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); #endif + mxnet::common::cuda::DeviceStore device_store(handle.ctx.real_dev_id(), true); cudaError_t err = cudaFreeHost(ptr); // ignore unloading error, as memory has already been recycled if (err != cudaSuccess && err != cudaErrorCudartUnloading) { diff --git a/src/storage/pooled_storage_manager.h b/src/storage/pooled_storage_manager.h index f3a9b16cd..c407a9f00 100644 --- a/src/storage/pooled_storage_manager.h +++ b/src/storage/pooled_storage_manager.h @@ -57,6 +57,11 @@ class GPUPooledStorageManager final : public StorageManager { GPUPooledStorageManager() { reserve_ = dmlc::GetEnv("MXNET_GPU_MEM_POOL_RESERVE", 5); page_size_ = dmlc::GetEnv("MXNET_GPU_MEM_POOL_PAGE_SIZE", 4096); + large_alloc_round_size_ = dmlc::GetEnv("MXNET_GPU_MEM_LARGE_ALLOC_ROUND_SIZE", 2 * 1024 * 1024); + if (large_alloc_round_size_ <= 0) { + LOG(FATAL) << "MXNET_GPU_MEM_LARGE_ALLOC_ROUND_SIZE cannot be set to a value <= 0, found: " + << large_alloc_round_size_; + } if (page_size_ < NDEV) { LOG(FATAL) << "MXNET_GPU_MEM_POOL_PAGE_SIZE cannot be set to a value smaller than " << NDEV \ << ". Got " << page_size_ << "."; @@ -79,8 +84,9 @@ class GPUPooledStorageManager final : public StorageManager { private: void DirectFreeNoLock(Storage::Handle handle) { + mxnet::common::cuda::DeviceStore device_store(handle.ctx.real_dev_id(), true); cudaError_t err = cudaFree(handle.dptr); - size_t size = std::max(handle.size, page_size_); + size_t size = RoundAllocSize(handle.size); // ignore unloading error, as memory has already been recycled if (err != cudaSuccess && err != cudaErrorCudartUnloading) { LOG(FATAL) << "CUDA: " << cudaGetErrorString(err); @@ -88,12 +94,31 @@ class GPUPooledStorageManager final : public StorageManager { used_memory_ -= size; } + // Round a value 'x' up to the next multiple of 'multiple' + size_t RoundToMultiple(size_t x, size_t multiple) { + size_t retVal = ((x + multiple - 1) / multiple) * multiple; + return retVal; + } + + size_t RoundAllocSize(size_t size) { + // Round up small allocs to the page_size_ to consolidate the pool lookups + size = std::max(size, page_size_); + // To ensure proper freeing under some driver variants, make sure + // large allocs entirely occupy their slabs, which cannot then be + // locked by smaller permanent allocations sharing the slab. + if (size > large_alloc_round_size_) + size = RoundToMultiple(size, large_alloc_round_size_); + return size; + } + private: void ReleaseAll(); // used memory size_t used_memory_ = 0; // page size size_t page_size_; + // size that large allocations should be rounded to, for proper freeing. + size_t large_alloc_round_size_; // percentage of reserved memory int reserve_; // number of devices @@ -105,9 +130,10 @@ class GPUPooledStorageManager final : public StorageManager { void GPUPooledStorageManager::Alloc(Storage::Handle* handle) { std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); - size_t size = std::max(handle->size, page_size_); + size_t size = RoundAllocSize(handle->size); auto&& reuse_it = memory_pool_.find(size); if (reuse_it == memory_pool_.end() || reuse_it->second.size() == 0) { + mxnet::common::cuda::DeviceStore device_store(handle->ctx.real_dev_id(), true); size_t free, total; cudaMemGetInfo(&free, &total); if (free <= total * reserve_ / 100 || size > free - total * reserve_ / 100) @@ -130,7 +156,7 @@ void GPUPooledStorageManager::Alloc(Storage::Handle* handle) { void GPUPooledStorageManager::Free(Storage::Handle handle) { std::lock_guard lock(Storage::Get()->GetMutex(Context::kGPU)); - size_t size = std::max(handle.size, page_size_); + size_t size = RoundAllocSize(handle.size); auto&& reuse_pool = memory_pool_[size]; reuse_pool.push_back(handle.dptr); } @@ -228,6 +254,7 @@ class GPUPooledRoundedStorageManager final : public StorageManager { } void DirectFreeNoLock(Storage::Handle handle) { + mxnet::common::cuda::DeviceStore device_store(handle.ctx.real_dev_id(), true); cudaError_t err = cudaFree(handle.dptr); size_t size = get_size(get_bucket(handle.size)); // ignore unloading error, as memory has already been recycled @@ -264,6 +291,7 @@ void GPUPooledRoundedStorageManager::Alloc(Storage::Handle* handle) { size_t size = get_size(bucket); auto&& reuse_pool = memory_pool_[bucket]; if (reuse_pool.size() == 0) { + mxnet::common::cuda::DeviceStore device_store(handle->ctx.real_dev_id(), true); size_t free, total; cudaMemGetInfo(&free, &total); if (free <= total * reserve_ / 100 || size > free - total * reserve_ / 100) diff --git a/src/storage/storage.cc b/src/storage/storage.cc index 97277e8d6..a0275dd0a 100644 --- a/src/storage/storage.cc +++ b/src/storage/storage.cc @@ -48,35 +48,6 @@ class StorageImpl : public Storage { static int num_gpu_device; #endif // MXNET_USE_CUDA - static void ActivateDevice(Context ctx) { - switch (ctx.dev_type) { - case Context::kCPU: - break; - case Context::kCPUPinned: -#if MXNET_USE_CUDA - if (num_gpu_device > 0) { - CUDA_CALL(cudaSetDevice(ctx.real_dev_id())); - } -#endif // MXNET_USE_CUDA - break; - case Context::kCPUShared: { -#if defined(ANDROID) || defined(__ANDROID__) - LOG(FATAL) << "Unimplemented device"; -#endif // defined(ANDROID) || defined(__ANDROID__) - } - break; - case Context::kGPU: { -#if MXNET_USE_CUDA - if (num_gpu_device > 0) { - CUDA_CALL(cudaSetDevice(ctx.real_dev_id())); - } -#endif // MXNET_USE_CUDA - break; - } - default: - LOG(FATAL) << "Unimplemented device"; - } - } // internal storage managers std::array, kMaxNumberOfDevices> storage_managers_; @@ -100,6 +71,8 @@ void StorageImpl::Alloc(Storage::Handle* handle) { case Context::kCPUShared: { #if !defined(ANDROID) && !defined(__ANDROID__) ptr = new storage::CPUSharedStorageManager(); +#else + LOG(FATAL) << "Unimplemented device"; #endif // !defined(ANDROID) && !defined(__ANDROID__) break; } @@ -153,13 +126,6 @@ void StorageImpl::Alloc(Storage::Handle* handle) { return ptr; }); -#if MXNET_USE_CUDA - // Will restore gpu device to before ActivateDevice if necessary - bool restore = handle->ctx.dev_type == Context::kCPUPinned || - handle->ctx.dev_type == Context::kGPU; - mxnet::common::cuda::DeviceStore device_store(restore); -#endif - this->ActivateDevice(handle->ctx); manager->Alloc(handle); profiler_.OnAlloc(*handle); } @@ -173,12 +139,6 @@ void StorageImpl::Free(Storage::Handle handle) { return nullptr; }); -#if MXNET_USE_CUDA - // Will restore gpu device to before ActivateDevice if necessary - bool restore = ctx.dev_type == Context::kCPUPinned || ctx.dev_type == Context::kGPU; - mxnet::common::cuda::DeviceStore device_store(restore); -#endif - this->ActivateDevice(ctx); manager->Free(handle); profiler_.OnFree(handle); } @@ -192,12 +152,6 @@ void StorageImpl::DirectFree(Storage::Handle handle) { return nullptr; }); -#if MXNET_USE_CUDA - // Will restore gpu device to before ActivateDevice if necessary - bool restore = ctx.dev_type == Context::kCPUPinned || ctx.dev_type == Context::kGPU; - mxnet::common::cuda::DeviceStore device_store(restore); -#endif - this->ActivateDevice(ctx); manager->DirectFree(handle); profiler_.OnFree(handle); } diff --git a/tests/cpp/include/test_mkldnn.h b/tests/cpp/include/test_mkldnn.h new file mode 100644 index 000000000..c705a6004 --- /dev/null +++ b/tests/cpp/include/test_mkldnn.h @@ -0,0 +1,625 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file test_mkldnn.h + * \brief helper functions to test mkldnn. + * \author Alex Zai + */ + +#ifndef TEST_MKLDNN_H_ +#define TEST_MKLDNN_H_ + +#if MXNET_USE_MKLDNN == 1 + +#include +#include +#include +#include "../../../3rdparty/mkldnn/include/mkldnn_types.h" +#include "../../../3rdparty/googletest/googletest/include/gtest/gtest.h" +#include "../../../src/operator/nn/mkldnn/mkldnn_base-inl.h" + +using namespace mxnet; + +inline static mkldnn::memory::primitive_desc GetMemPD(const TShape s, int dtype, + mkldnn::memory::format format) { + mkldnn::memory::dims dims(s.ndim()); + for (size_t i = 0; i < dims.size(); i++) + dims[i] = s[i]; + mkldnn::memory::desc desc{dims, get_mkldnn_type(dtype), format}; + return mkldnn::memory::primitive_desc(desc, CpuEngine::Get()->get_engine()); +} + +inline static mkldnn::memory::primitive_desc GetExpandedMemPD( + mkldnn::memory::primitive_desc pd, float scale, int dim = 0) { + CHECK(dim < pd.desc().data.ndims) << "dimension cannot be larger than total dimensions of input"; + nnvm::TShape s(pd.desc().data.ndims); + for (size_t i = 0; i < pd.desc().data.ndims; i++) + s[i] = pd.desc().data.dims[i]; + s[dim] = static_cast(s[dim] * scale); + return GetMemPD(s, mshadow::DataType::kFlag, + static_cast(pd.desc().data.format)); +} + +struct TestArrayShapes { + std::vector shapes; + std::vector pds; +}; + +// Init arrays with the default layout. +inline static void InitDefaultArray(NDArray *arr, bool is_rand = false) { + const TBlob &blob = arr->data(); + mshadow::default_real_t *data = blob.dptr(); + int size = blob.Size(); + + for (int i = 0; i < size; i++) + if (is_rand) { + data[i] = (std::rand() % 100) - 50; + } else { + data[i] = i % 100 - 50; + } +} + + +// Init arrays with the specified layout. +inline static void InitMKLDNNArray(NDArray *arr, const mkldnn::memory::primitive_desc &pd, + bool is_rand = false) { + InitDefaultArray(arr, is_rand); + arr->MKLDNNDataReorderAsync(pd); + arr->WaitToRead(); +} + +inline static bool IsSameShape(mkldnn::memory::primitive_desc pd, TShape shape) { + if (pd.desc().data.ndims != shape.ndim()) return false; + for (size_t i = 0; i < shape.ndim(); i++) + if (pd.desc().data.dims[i] != shape[i]) return false; + return true; +} + +// This function gets special MKLDNN formats without knowing the specific +// hardware configuration. Certainly, it potentially misses some format if +// it's specific for certain array shapes. It covers at least one special format +// for each of the formats: nchw, oihw, goihw. +// To test the logic of the code in NDArray, these formats should be enough. +inline static std::vector GetMKLDNNFormat(size_t num_dims, int dtype) { + if (num_dims == 4) { + mkldnn::memory::dims data_dims{1, 3, 224, 224}; + mkldnn::memory::desc data_md{data_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims weight_dims{96, 3, 11, 11}; + mkldnn::memory::desc weight_md{weight_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims output_dims{1, 96, 54, 54}; + mkldnn::memory::desc out_md{output_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims strides{4, 4}; + mkldnn::memory::dims padding{0, 0}; + + mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, + mkldnn::algorithm::convolution_direct, + data_md, weight_md, out_md, strides, + padding, padding, mkldnn::padding_kind::zero); + mkldnn::convolution_forward::primitive_desc pd(desc, CpuEngine::Get()->get_engine()); + while (pd.dst_primitive_desc().get_size() != GetMemDescSize(out_md) || + pd.src_primitive_desc().get_size() != GetMemDescSize(data_md) || + pd.weights_primitive_desc().get_size() != GetMemDescSize(weight_md)) { + CHECK(pd.next_impl()) << "No implementation"; + } + + std::vector ret(1); + ret[0] = static_cast(pd.dst_primitive_desc().desc().data.format); + printf("format: %d \n", ret[0]); + return ret; + } else if (num_dims == 5) { + mkldnn::memory::dims data_dims{1, 32, 112, 112}; + mkldnn::memory::desc data_md{data_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims weight_dims{32, 1, 1, 3, 3}; + mkldnn::memory::desc weight_md{weight_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims output_dims{1, 32, 112, 112}; + mkldnn::memory::desc out_md{output_dims, get_mkldnn_type(dtype), + mkldnn::memory::format::any}; + mkldnn::memory::dims strides{1, 1}; + mkldnn::memory::dims padding{1, 1}; + + mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, + mkldnn::algorithm::convolution_direct, + data_md, weight_md, out_md, strides, + padding, padding, mkldnn::padding_kind::zero); + mkldnn::convolution_forward::primitive_desc pd(desc, CpuEngine::Get()->get_engine()); + while (pd.dst_primitive_desc().get_size() != GetMemDescSize(out_md) || + pd.src_primitive_desc().get_size() != GetMemDescSize(data_md) || + pd.weights_primitive_desc().get_size() != GetMemDescSize(weight_md)) { + CHECK(pd.next_impl()) << "No implementation"; + } + + std::vector ret(1); + ret[0] = static_cast(pd.weights_primitive_desc().desc().data.format); + printf("format: %d\n", ret[0]); + return ret; + } else { + return std::vector(); + } +} + +inline static TestArrayShapes GetTestArrayShapes(bool spatial_data_format = false) { + int dtype = mshadow::DataType::kFlag; + std::vector shapes; + std::vector pds; + { + // 1D + TShape s(1); + s[0] = 279936; + shapes.push_back(s); + pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::x)); + s[0] = 34848; + shapes.push_back(s); + pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::x)); + } + { + // 2D + TShape s(2); + s[0] = 96; + s[1] = 2916; + shapes.push_back(s); + pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::nc)); + s[0] = 96; + s[1] = 363; + shapes.push_back(s); + pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::nc)); + } + { + // 4D + TShape s1(4); + s1[0] = 10; s1[1] = 96; s1[2] = 54; s1[3] = 54; + shapes.push_back(s1); + pds.push_back(GetMemPD(s1, dtype, mkldnn::memory::format::nchw)); + + TShape s2(4); + s2[0] = 96; s2[1] = 3; s2[2] = 11; s2[3] = 11; + shapes.push_back(s2); + pds.push_back(GetMemPD(s2, dtype, mkldnn::memory::format::oihw)); + + std::vector formats = GetMKLDNNFormat(4, dtype); + if (!spatial_data_format) { + pds.push_back(GetMemPD(s1, dtype, formats[0])); + } + } + { + // 5D + TShape s(5); + s[0] = 96; s[1] = 1; s[2] = 3; s[3] = 11; s[4] = 11; + shapes.push_back(s); + pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::goihw)); + + std::vector formats = GetMKLDNNFormat(5, dtype); + if (!spatial_data_format) { + pds.push_back(GetMemPD(s, dtype, formats[0])); + } + } + + TestArrayShapes ret; + ret.shapes = shapes; + ret.pds = pds; + return ret; +} + +struct NDArrayAttrs { + NDArray arr; + std::string desc; + NDArrayAttrs(NDArray arr, std::string desc) : arr(arr), desc(desc) {} +}; + +struct OpAttrs { + nnvm::NodeAttrs attrs; + std::vector dispatches; + std::set requests; + std::unordered_set accept_dims; + int num_inputs; + int num_outputs; + int input_types; + int output_types; +}; + +enum ArrayTypes { + Normal = 1, + MKLDNN = 2, + MKLDNNDiffShape = 4, + MKLDNNDiffDim = 8, + NormalReshaped = 16, + MKLDNNReshaped = 32, + MKLDNNReshapedDiffShape = 64, + MKLDNNReshapedDiffDim = 128, + NormalReused = 256, + MKLDNNReused = 512, + MKLDNNReusedDiffDim = 1024, + NormalReshapedReused = 2048, + NormalReusedDiffDtype = 4096, + All = 8191, +}; + + +inline NDArray CreateKernelNDArray(TShape kernel, int num_filters, TShape input, + bool is_deconv = false) { + CHECK_EQ(kernel.ndim(), 2) << "mkldnn only supports 2d filters on 4d inputs"; + TShape target_shape(4); + target_shape[0] = is_deconv ? input[1] : num_filters; + target_shape[1] = is_deconv ? num_filters : input[1]; + target_shape[2] = kernel[0]; + target_shape[3] = kernel[1]; + int dtype = mshadow::DataType::kFlag; + NDArray arr(target_shape, Context()); + auto pd = GetMemPD(target_shape, dtype, mkldnn::memory::format::nchw); + InitMKLDNNArray(&arr, pd); + return arr; +} + +inline NDArray CreateBiasNDArray(TShape target_shape) { + int dtype = mshadow::DataType::kFlag; + NDArray arr(target_shape, Context()); + auto pd = GetMemPD(target_shape, dtype, mkldnn::memory::format::x); + InitMKLDNNArray(&arr, pd); + return arr; +} + +inline int CalculateWidthConvOutput(int width, int kernel, int padding, int stride) { + return (width - kernel + 2 * padding) / stride + 1; +} + +inline int CalculateWidthDeconvOutput(int width, int kernel, int padding, int stride) { + return stride * (width - 1) + kernel - 2 * padding; +} + +inline std::string CreateShapeString(int value, int dim) { + std::stringstream ss; + ss << "("; + for (int i = 0; i < dim; i++) { + ss << value; + if (i != dim - 1) ss << ","; + } + ss << ")"; + return ss.str(); +} + +inline void PrintVerifyMsg(const NDArrayAttrs &arr1, const NDArrayAttrs &arr2) { + TShape t1 = arr1.arr.shape(); + TShape t2 = arr2.arr.shape(); + std::stringstream ss; + std::cout << "Verifying: " << arr1.desc.c_str() << " " << + t1 << " with " << arr2.desc.c_str() << " " << t2 << "\n"; +} + +/* + * We want to get a few types of NDArrays for testing: + * 1. Normal NDArray + * 2. Normal NDArray with MKLDNN layout (output from an MKLDNN operator) + * 3. Normal NDArray with MKLDNN layout whose MKLDNN memory may have different + * dimensions from the NDArray (result of MKLDNNDataReorderAsync). However, this + * type of NDArrays only exists for weight arrays. I don't think we should + * pass them to all operators. + * In the inference mode, the MKLDNN memory in the weight array will be + * reordered to 5 dimensions. + * 4. Reshaped/sliced NDArray + * 5. Reshaped/sliced NDArray with MKLDNN layout (reshape/slice from Normal NDArray + * with MKLDNN layout) + * 6. Reshaped/sliced NDArray with MKLDNN layout whose MKLDNN memory may have + * different dimensions from the NDArray (result of MKLDNNDataReorderAsync). + * However, this type of NDArrays only exists for weight arrays. I don't think + * we should pass them to all operators. + * In the inference mode, the MKLDNN memory in the weight array will be + * reordered to 5 dimensions. + * + * num_inputs / dim arguments used to scale shape (used for concat backwards to enlarge input shapes) + */ +inline std::vector GetTestInputArrays( + int types = ArrayTypes::All, bool rand = false, + std::vector scale = {1}, bool spatial_data_format = false) { + TestArrayShapes tas = GetTestArrayShapes(spatial_data_format); + std::vector shapes = tas.shapes; + std::vector pds = tas.pds; + + std::vector in_arrs; + std::string desc; + + int slice_amount = scale[0]; + for (auto shape : shapes) { + if (scale.size() > shape.ndim()) + continue; + + for (size_t dim = 0; dim < scale.size(); ++dim) + shape[dim] = static_cast(round(shape[dim] * scale[dim])); + + // Type 1. + NDArray arr(shape, Context()); + if (types & ArrayTypes::Normal) { + InitDefaultArray(&arr, rand); + in_arrs.emplace_back(arr, "Normal NDArray"); + } + + // Type 4 + arr = NDArray(shape, Context()); + if (types & ArrayTypes::NormalReshaped) { + InitDefaultArray(&arr, rand); + in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), + "Reshaped Normal NDArray"); + } + + + for (auto pd : pds) { + for (size_t dim = 0; dim < scale.size(); ++dim) { + // preserve if matching layout else just expand on 0 dim + if (shape.ndim() == pd.desc().data.ndims) + pd = GetExpandedMemPD(pd, scale[dim], dim); + else + pd = GetExpandedMemPD(pd, scale[dim]); + } + + if (shape.Size() != pd.get_size() / sizeof(mshadow::default_real_t)) + continue; + + // Type 2, 3. + arr = NDArray(shape, Context()); + if (shape.ndim() == pd.desc().data.ndims && IsSameShape(pd, shape) + && types & ArrayTypes::MKLDNN) { + desc = "MKLDNN NDArray"; + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr, desc); + } else if (shape.ndim() == pd.desc().data.ndims && !IsSameShape(pd, shape) + && types & ArrayTypes::MKLDNNDiffShape) { + desc = "MKLDNN NDArray with different shape"; + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr, desc); + } else if (shape.ndim() != pd.desc().data.ndims && types & ArrayTypes::MKLDNNDiffDim) { + std::stringstream ss; + ss << "MKLDNN NDArray with different dim " << + shape.ndim() << "/" << pd.desc().data.ndims; + desc = ss.str(); + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr, desc); + } + + + // Type 5, 6. + arr = NDArray(shape, Context()); + if (shape.ndim() == pd.desc().data.ndims && IsSameShape(pd, shape) + && types & ArrayTypes::MKLDNNReshaped) { + desc = "Reshaped MKLDNN NDArray"; + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); + } else if (shape.ndim() == pd.desc().data.ndims && !IsSameShape(pd, shape) + && types & ArrayTypes::MKLDNNReshapedDiffShape) { + desc = "Reshaped MKLDNN NDArray with different shape"; + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); + } else if (shape.ndim() != pd.desc().data.ndims + && types & ArrayTypes::MKLDNNReshapedDiffDim) { + std::stringstream ss; + ss << "MKLDNN NDArray with different dim " << + shape.ndim() << "/" << pd.desc().data.ndims; + desc = ss.str(); + InitMKLDNNArray(&arr, pd, rand); + in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); + } + } + } + return in_arrs; +} + +/* + * We want to get a few types of NDArrays for testing: + * 1. Normal NDArray + * 2. Normal NDArray with MKLDNN layout (output from an MKLDNN operator) + * 3. Normal NDArray with MKLDNN layout whose MKLDNN memory may have different + * dimensions from the NDArray (result of MKLDNNDataReorderAsync). However, this + * type of NDArrays only exists for weight arrays. I don't think we should + * pass them to all operators. + * In the inference mode, the MKLDNN memory in the weight array will be + * reordered to 5 dimensions. + * 4. Reshaped/sliced NDArray + * 5. Reused NDArray (this is created by the MXNet executor). This type of + * NDArrays can only be used as output arrays. + * 6. Reused NDArray converted from an array with a different data type. + * 7. Reused reshaped/sliced NDArray. + * 8. Reused NDArray with MKLDNN layout. + * 9. Reused NDArray with MKLDNN layout of different dimensions. + * + * Optional num_inputs / dim args can be passed to modify input shape (used for Concat test) + */ +inline std::vector GetTestOutputArrays( + const TShape &shp, + const std::vector &pds, + std::vectorscale = {1}, bool rand = true, int types = ArrayTypes::All) { + TShape shape = shp; + + for (int dim = 0; dim < scale.size(); dim++) + shape[dim] = static_cast(shape[dim] * scale[dim]); + + std::vector in_arrs; + std::string desc; + // Type 1. + NDArray arr(shape, Context()); + + if (types & ArrayTypes::Normal) { + in_arrs.emplace_back(arr, "Normal NDArray"); + InitDefaultArray(&in_arrs.back().arr, rand); + } + + TShape tmp_shape = shape; + if (types & ArrayTypes::NormalReshaped) { + // Type 4. + tmp_shape[0] = shape[0] * 2; + NDArray arr0(tmp_shape, Context()); + InitDefaultArray(&arr0, rand); + in_arrs.emplace_back(arr0.Slice(1, shape[0] + 1), "Reshaped NDArray"); + } + + nnvm::TShape s(1); + if (types & ArrayTypes::NormalReused) { + // Type 5. + // Get a reused version. + s[0] = shape.Size(); + NDArray arr1(s, Context()); + arr1 = arr1.AsArray(shape, arr1.dtype()); + InitDefaultArray(&arr1, rand); + in_arrs.emplace_back(arr1, "Reused NDArray"); + } + + if (types & ArrayTypes::NormalReusedDiffDtype) { + // Type 6. + s[0] = shape.Size() * GetTypeSize(mshadow::default_type_flag); + NDArray arr2(s, Context(), true, mshadow::kUint8); + arr2 = arr2.AsArray(shape, mshadow::default_type_flag); + InitDefaultArray(&arr2, rand); + in_arrs.emplace_back(arr2, "Reused NDArray with diff data type"); + } + + if (types & ArrayTypes::NormalReshapedReused) { + // Type 7 + s[0] = shape.Size() * GetTypeSize(mshadow::default_type_flag) * 2; + NDArray arr3(s, Context(), true, mshadow::kUint8); + tmp_shape[0] = shape[0] * 2; + arr3 = arr3.AsArray(tmp_shape, mshadow::default_type_flag); + InitDefaultArray(&arr3, rand); + in_arrs.emplace_back(arr3.Slice(1, shape[0] + 1), "Reused+Reshaped NDArray"); + } + + for (auto pd : pds) { + if (shape.Size() != pd.get_size() / sizeof(mshadow::default_real_t)) + continue; + + if (scale.size() > pd.desc().data.ndims) + continue; + + for (int dim = 0; dim < scale.size(); dim++) + pd = GetExpandedMemPD(pd, scale[dim]); + + // Type 2, 3. + arr = NDArray(shape, Context()); + desc = "MKLDNN NDArray"; + if (shape.ndim() != pd.desc().data.ndims) { + std::stringstream ss; + ss << "MKLDNN NDArray with different memory layout " + << shape.ndim() << "/" << pd.desc().data.ndims; + desc = ss.str(); + } + + if ((types & ArrayTypes::MKLDNN && shape.ndim() == pd.desc().data.ndims) || + (types & ArrayTypes::MKLDNNDiffDim && shape.ndim() != pd.desc().data.ndims)) { + in_arrs.emplace_back(arr, desc); + InitMKLDNNArray(&in_arrs.back().arr, pd, rand); + } + + // Type 8, 9. + // Get a reused version. + nnvm::TShape s(1); + s[0] = shape.Size(); + NDArray arr = NDArray(s, Context()); + arr = arr.AsArray(shape, arr.dtype()); + InitMKLDNNArray(&arr, pd, rand); + desc = "Reused MKLDNN NDArray"; + if (shape.ndim() != pd.desc().data.ndims) { + std::stringstream ss; + ss << "Reused MKLDNN NDArray with different memory layout " + << shape.ndim() << "/" << pd.desc().data.ndims; + desc = ss.str(); + } + + if ((types & ArrayTypes::MKLDNNReused && shape.ndim() == pd.desc().data.ndims) || + (types & ArrayTypes::MKLDNNReusedDiffDim && shape.ndim() != pd.desc().data.ndims)) { + in_arrs.emplace_back(arr, desc); + } + } + return in_arrs; +} + +/* + * Determines axis ndarrays are concatenated by + * Used to verify concat/concat backwards operator + */ +inline int GetDim(TShape input_shape, TShape output_shape) { + CHECK(input_shape.Size() != output_shape.Size()); + for (size_t i = 0; i < input_shape.ndim(); i++) { + if (input_shape[i] != output_shape[i]) + return i; + } + return -1; +} + +/* + * Calculates the size of continuous block of array inside larger concatenated array + * Used to verify concat/concat backwards operator + */ +inline int GetBlockSize(TShape shape, int dim) { + int block_size = 1; + for (int i = shape.ndim() - 1; i >= dim; i--) + block_size *= shape[i]; + return block_size; +} + +inline int CalculateWidthPoolOutput(int width, int kernel, int padding, int stride) { + return (width - kernel + 2 * padding) / stride + 1; +} + +using VerifyFunc = std::function &in_arrs, + const std::vector &out_arrs)>; + +inline void VerifyAddRequest(const std::vector &in_arrs, + const std::vector &original_outputs, + const std::vector &new_outputs, + VerifyFunc verify_fn) { + CHECK(original_outputs.size() == new_outputs.size()); + std::vector tmp_outputs; + NDArray tmp; + for (size_t i = 0; i < new_outputs.size(); i++) { + tmp = new_outputs[i]->Reorder2Default() - original_outputs[i]->Reorder2Default(); + tmp_outputs.push_back(&tmp); + } + Engine::Get()->WaitForAll(); + verify_fn(in_arrs, tmp_outputs); +} + +inline void VerifyCopyResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + NDArray tmp1 = in_arrs[0]->Reorder2Default(); + NDArray tmp2 = out_arrs[0]->Reorder2Default(); + EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); + TBlob d1 = tmp1.data(); + TBlob d2 = tmp2.data(); + EXPECT_EQ(memcmp(d1.dptr_, d2.dptr_, + tmp1.shape().Size() * sizeof(mshadow::default_real_t)), 0); +} + +inline void VerifySumResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + NDArray in1 = in_arrs[0]->Reorder2Default(); + NDArray in2 = in_arrs[1]->Reorder2Default(); + NDArray out = out_arrs[0]->Reorder2Default(); + EXPECT_EQ(in1.shape().Size(), in2.shape().Size()); + EXPECT_EQ(in1.shape().Size(), out.shape().Size()); + + mshadow::default_real_t *d1 = in1.data().dptr(); + mshadow::default_real_t *d2 = in2.data().dptr(); + mshadow::default_real_t *o = out.data().dptr(); + for (size_t i = 0; i < in1.shape().Size(); i++) + ASSERT_EQ(d1[i] + d2[i], o[i]); +} + +#endif // MXNET_USE_MKLDNN +#endif // TEST_MKLDNN_H_ diff --git a/tests/cpp/kvstore/gpu_topology_test.cc b/tests/cpp/kvstore/gpu_topology_test.cc index 0f6d5f11c..29afb16bd 100644 --- a/tests/cpp/kvstore/gpu_topology_test.cc +++ b/tests/cpp/kvstore/gpu_topology_test.cc @@ -28,24 +28,20 @@ #include #include "../src/kvstore/gpu_topology.h" -void GenerateMatrix(std::vector* W, int num_gpus, float k, - std::mt19937* gen) { +void GenerateMatrix(std::vector* W, int num_gpus, std::mt19937* gen) { std::uniform_real_distribution<> dis(0., 1.); for (int row = 0; row < num_gpus; ++row) { for (int col = row+1; col < num_gpus; ++col) { - float sample = dis(*gen); - if (sample < k) - continue; - sample = dis(*gen); - if (sample < 0.33f) { - (*W)[row*num_gpus+col] = 1.f; - (*W)[col*num_gpus+row] = 1.f; + double sample = dis(*gen); + if (sample < 0.33) { + (*W)[row*num_gpus+col] = 1.; + (*W)[col*num_gpus+row] = 1.; } else if (sample < 0.66f) { - (*W)[row*num_gpus+col] = 2.f; - (*W)[col*num_gpus+row] = 2.f; + (*W)[row*num_gpus+col] = 2.; + (*W)[col*num_gpus+row] = 2.; } else { - (*W)[row*num_gpus+col] = 3.f; - (*W)[col*num_gpus+row] = 3.f; + (*W)[row*num_gpus+col] = 3.; + (*W)[col*num_gpus+row] = 3.; } } } @@ -71,11 +67,12 @@ void TestComputeTreesRandomized(int num_gpus, float alpha, int backtrack, bool satisfied = false; std::vector W(num_gpus*num_gpus, 0.f); int depth = mxnet::kvstore::ComputeDepth(num_gpus); - while (!satisfied) { - float k = dis(*gen); - std::fill(W.begin(), W.end(), 0.f); - GenerateMatrix(&W, num_gpus, k, gen); - satisfied = IsSatisfactory(W, num_gpus, depth); + GenerateMatrix(&W, num_gpus, gen); + satisfied = IsSatisfactory(W, num_gpus, depth); + if (mxnet::kvstore::kLogTree && !satisfied) { + LOG(ERROR) << " topology connectivity not satisfied " + "(out edges per node less than tree depth)"; + mxnet::kvstore::PrintMatrix("W", W, num_gpus, num_gpus); } std::vector> topo; @@ -561,8 +558,7 @@ TEST(GpuTopology, TestComputeTrees1) { std::mt19937 gen(1); float alpha = 0.7; bool backtrack = true; - // Do 5 randomized tests per GPU count from 2 to 16 - for (int num_gpus = 2; num_gpus <= 16; ++num_gpus) { + for (int num_gpus = 2; num_gpus <= 8; ++num_gpus) { LOG(INFO) << "Testing " << num_gpus << " x " << num_gpus; for (int i = 0; i < 5; ++i) { TestComputeTreesRandomized(num_gpus, alpha, backtrack, &gen); diff --git a/tests/cpp/operator/activation_perf.cc b/tests/cpp/operator/activation_perf.cc index 1bd8ca89c..bba8a3ec5 100644 --- a/tests/cpp/operator/activation_perf.cc +++ b/tests/cpp/operator/activation_perf.cc @@ -38,13 +38,27 @@ const kwargs_t basic_activation_args = { }; * \brief Generic bidirectional sanity test */ TEST(ACTIVATION_PERF, ExecuteBidirectional) { + using namespace std; TShape shape({5, 5}); - kwargs_t kwargs = basic_activation_args; - kwargs.push_back({"act_type", "tanh"}); - - test::op::CoreOperatorRunner runner; - runner.RunBidirectional(false, { shape }, test::op::CoreOpExecutor::ArgsWithOpName( - kwargs, "Activation", "_backward_Activation"), 1); + vector activations = { + "relu", + "sigmoid", + "tanh", + "softrelu", + "softsign" + }; + for (const string& activation : activations) { + kwargs_t activation_args = {{"act_type", activation}}; + test::op::CoreOperatorRunner runner; + runner.RunBidirectional(false, { shape }, test::op::CoreOpExecutor::ArgsWithOpName( + activation_args, "Activation", "_backward_Activation"), 1); + } + for (const string& activation : activations) { + kwargs_t activation_args = {{"act_type", activation}}; + test::op::CoreOperatorRunner runner; + runner.RunBidirectional(true, { shape }, test::op::CoreOpExecutor::ArgsWithOpName( + activation_args, "Activation", "_backward_Activation"), 1); + } } /*! diff --git a/tests/cpp/operator/krprod_test.cc b/tests/cpp/operator/krprod_test.cc index 26c2661bc..66ddddd77 100644 --- a/tests/cpp/operator/krprod_test.cc +++ b/tests/cpp/operator/krprod_test.cc @@ -250,6 +250,8 @@ TEST(row_wise_kronecker, FourInputMatrices) { FreeSpace(&result); } + +#if MXNET_USE_LAPACK == 1 TEST(khatri_rao, OneInputMatrix) { // Input matrices of shape (2, 4) which is also the expected result DType mat[8] {1, 2, 3, 4, 5, 6, 7, 8}; @@ -444,5 +446,6 @@ TEST(inv_khatri_rao, ThreeInputMatricesTranposed) { FreeSpace(&kr_t); FreeSpace(&actual_dot); } +#endif // MXNET_USE_LAPACK == 1 } // namespace op } // namespace mxnet diff --git a/tests/cpp/operator/mkldnn.cc b/tests/cpp/operator/mkldnn.cc deleted file mode 100644 index 14578bec5..000000000 --- a/tests/cpp/operator/mkldnn.cc +++ /dev/null @@ -1,1646 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ - -/*! - * \file mkldnn.cc - * \brief test functions in mkldnn. - * \author Da Zheng - */ - -#if MXNET_USE_MKLDNN == 1 - -#include -#include -#include -#include -#include "gtest/gtest.h" -#include "mxnet/imperative.h" -#include "../../src/operator/nn/mkldnn/mkldnn_base-inl.h" -#include "../../src/operator/nn/mkldnn/mkldnn_ops-inl.h" -#include "../../src/operator/nn/mkldnn/mkldnn_pooling-inl.h" -#include "../../src/operator/nn/pooling-inl.h" - -using namespace mxnet; - -#if __GNUC__ >= 5 -bool test_mem_align(void *mem, size_t size, size_t alignment, size_t space) { - void *ret1, *ret2; - size_t space1, space2; - space1 = space; - space2 = space; - ret1 = mxnet::AlignMem(mem, size, alignment, &space1); - ret2 = std::align(alignment, size, mem, space2); - EXPECT_EQ(ret1, ret2); - EXPECT_EQ(space1, space2); - return ret1 == ret2; -} -#endif - -TEST(MKLDNN_UTIL_FUNC, AlignMem) { -#if __GNUC__ >= 5 - size_t alignment = 4096; - void *mem; - size_t size, space; - // When mem has been aligned. - mem = reinterpret_cast(0x10000); - size = 1000; - space = 10000; - test_mem_align(mem, size, alignment, space); - - // When mem isn't aligned and we have enough space for alignment. - mem = reinterpret_cast(0x10010); - size = 1000; - space = 10000; - test_mem_align(mem, size, alignment, space); - - // When mem isn't aligned and we don't have enough memory for alignment - mem = reinterpret_cast(0x10010); - size = 1000; - space = 1001; - test_mem_align(mem, size, alignment, space); - - for (size_t i = 0; i < 10000; i++) { - mem = reinterpret_cast(random()); - size = random() % 2000; - space = random() % 2000; - test_mem_align(mem, size, alignment, space); - } -#else - // std::align is not supported in GCC < 5.0, this test case will be checked - // with newer version - LOG(INFO) << "Skipped for GCC " << __GNUC__ << "." << __GNUC_MINOR__; -#endif -} - -TEST(MKLDNN_UTIL_FUNC, MemFormat) { - // Check whether the number of format is correct. - CHECK_EQ(mkldnn_format_last, 67); - CHECK_EQ(mkldnn_nchw, 5); - CHECK_EQ(mkldnn_oihw, 15); -} - -// Init arrays with the default layout. -static void InitDefaultArray(NDArray *arr, bool is_rand = false) { - const TBlob &blob = arr->data(); - mshadow::default_real_t *data = blob.dptr(); - int size = blob.Size(); - - for (int i = 0; i < size; i++) - if (is_rand) { - data[i] = (std::rand() % 100) - 50; - } else { - data[i] = i % 100 - 50; - } -} - -using VerifyFunc = std::function &in_arrs, - const std::vector &out_arrs)>; - -// Init arrays with the specified layout. -static void InitMKLDNNArray(NDArray *arr, const mkldnn::memory::primitive_desc &pd, - bool is_rand = false) { - InitDefaultArray(arr, is_rand); - arr->MKLDNNDataReorderAsync(pd); - arr->WaitToRead(); -} - -static void VerifyDefMem(const mkldnn::memory &mem) { - mkldnn::memory::primitive_desc pd = mem.get_primitive_desc(); - mshadow::default_real_t *data - = static_cast(mem.get_data_handle()); - size_t size = pd.get_size() / sizeof(mshadow::default_real_t); - size_t num_same = 0; - for (int i = 0; i < size; i++) - num_same += data[i] == static_cast(i % 100 - 50); - EXPECT_EQ(num_same, size); -} - -static void VerifyMem(const mkldnn::memory &mem) { - mkldnn::memory::primitive_desc pd = mem.get_primitive_desc(); - - if (pd.desc().data.format == GetDefaultFormat(pd.desc())) { - VerifyDefMem(mem); - } else { - mkldnn::memory::dims dims(pd.desc().data.ndims); - for (size_t i = 0; i < dims.size(); i++) - dims[i] = pd.desc().data.dims[i]; - mkldnn::memory::desc desc{dims, - static_cast(pd.desc().data.data_type), - static_cast(GetDefaultFormat(pd.desc()))}; - mkldnn::memory::primitive_desc new_pd(desc, CpuEngine::Get()->get_engine()); - mkldnn::memory new_mem(new_pd); - - std::vector net; - net.push_back(mkldnn::reorder(mem, new_mem)); - mkldnn::stream(mkldnn::stream::kind::eager).submit(net).wait(); - VerifyDefMem(new_mem); - } -} - -static bool IsSameShape(mkldnn::memory::primitive_desc pd, TShape shape) { - if (pd.desc().data.ndims != shape.ndim()) return false; - for (size_t i = 0; i < shape.ndim(); i++) - if (pd.desc().data.dims[i] != shape[i]) return false; - return true; -} - -static mkldnn::memory::primitive_desc GetMemPD(const TShape s, int dtype, - mkldnn::memory::format format) { - mkldnn::memory::dims dims(s.ndim()); - for (size_t i = 0; i < dims.size(); i++) - dims[i] = s[i]; - mkldnn::memory::desc desc{dims, get_mkldnn_type(dtype), format}; - return mkldnn::memory::primitive_desc(desc, CpuEngine::Get()->get_engine()); -} - -static mkldnn::memory::primitive_desc GetExpandedMemPD( - mkldnn::memory::primitive_desc pd, float scale, int dim = 0) { - CHECK(dim < pd.desc().data.ndims) << "dimension cannot be larger than total dimensions of input"; - nnvm::TShape s(pd.desc().data.ndims); - for (size_t i = 0; i < pd.desc().data.ndims; i++) - s[i] = pd.desc().data.dims[i]; - s[dim] = static_cast(s[dim] * scale); - return GetMemPD(s, mshadow::DataType::kFlag, - static_cast(pd.desc().data.format)); -} - -// This function gets special MKLDNN formats without knowing the specific -// hardware configuration. Certainly, it potentially misses some format if -// it's specific for certain array shapes. It covers at least one special format -// for each of the formats: nchw, oihw, goihw. -// To test the logic of the code in NDArray, these formats should be enough. -static std::vector GetMKLDNNFormat(size_t num_dims, int dtype) { - if (num_dims == 4) { - mkldnn::memory::dims data_dims{1, 3, 224, 224}; - mkldnn::memory::desc data_md{data_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims weight_dims{96, 3, 11, 11}; - mkldnn::memory::desc weight_md{weight_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims output_dims{1, 96, 54, 54}; - mkldnn::memory::desc out_md{output_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims strides{4, 4}; - mkldnn::memory::dims padding{0, 0}; - - mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, - mkldnn::algorithm::convolution_direct, - data_md, weight_md, out_md, strides, - padding, padding, mkldnn::padding_kind::zero); - mkldnn::convolution_forward::primitive_desc pd(desc, CpuEngine::Get()->get_engine()); - std::vector ret(2); - ret[0] = static_cast(pd.dst_primitive_desc().desc().data.format); - ret[1] = static_cast(pd.weights_primitive_desc().desc().data.format); - printf("format: %d, %d\n", ret[0], ret[1]); - return ret; - } else if (num_dims == 5) { - mkldnn::memory::dims data_dims{1, 32, 112, 112}; - mkldnn::memory::desc data_md{data_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims weight_dims{32, 1, 1, 3, 3}; - mkldnn::memory::desc weight_md{weight_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims output_dims{1, 32, 112, 112}; - mkldnn::memory::desc out_md{output_dims, get_mkldnn_type(dtype), - mkldnn::memory::format::any}; - mkldnn::memory::dims strides{1, 1}; - mkldnn::memory::dims padding{1, 1}; - - mkldnn::convolution_forward::desc desc(mkldnn::prop_kind::forward_training, - mkldnn::algorithm::convolution_direct, - data_md, weight_md, out_md, strides, - padding, padding, mkldnn::padding_kind::zero); - mkldnn::convolution_forward::primitive_desc pd(desc, CpuEngine::Get()->get_engine()); - std::vector ret(1); - ret[0] = static_cast(pd.weights_primitive_desc().desc().data.format); - printf("format: %d\n", ret[0]); - return ret; - } else { - return std::vector(); - } -} - -struct TestArrayShapes { - std::vector shapes; - std::vector pds; -}; - -static TestArrayShapes GetTestArrayShapes() { - int dtype = mshadow::DataType::kFlag; - std::vector shapes; - std::vector pds; - { - // 1D - TShape s(1); - s[0] = 279936; - shapes.push_back(s); - pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::x)); - s[0] = 34848; - shapes.push_back(s); - pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::x)); - } - { - // 2D - TShape s(2); - s[0] = 96; - s[1] = 2916; - shapes.push_back(s); - pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::nc)); - s[0] = 96; - s[1] = 363; - shapes.push_back(s); - pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::nc)); - } - { - // 4D - TShape s1(4); - s1[0] = 10; s1[1] = 96; s1[2] = 54; s1[3] = 54; - shapes.push_back(s1); - pds.push_back(GetMemPD(s1, dtype, mkldnn::memory::format::nchw)); - - TShape s2(4); - s2[0] = 96; s2[1] = 3; s2[2] = 11; s2[3] = 11; - shapes.push_back(s2); - pds.push_back(GetMemPD(s2, dtype, mkldnn::memory::format::oihw)); - - std::vector formats = GetMKLDNNFormat(4, dtype); - pds.push_back(GetMemPD(s1, dtype, formats[0])); - pds.push_back(GetMemPD(s2, dtype, formats[1])); - } - { - // 5D - TShape s(5); - s[0] = 96; s[1] = 1; s[2] = 3; s[3] = 11; s[4] = 11; - shapes.push_back(s); - pds.push_back(GetMemPD(s, dtype, mkldnn::memory::format::goihw)); - - std::vector formats = GetMKLDNNFormat(5, dtype); - pds.push_back(GetMemPD(s, dtype, formats[0])); - } - - TestArrayShapes ret; - ret.shapes = shapes; - ret.pds = pds; - return ret; -} - -TEST(MKLDNN_NDArray, GetDataReorder) { - TestArrayShapes tas = GetTestArrayShapes(); - std::vector shapes = tas.shapes; - std::vector pds = tas.pds; - - - // Reorder from the default to any other layout. - for (auto s : shapes) { - NDArray arr(s, Context()); - InitDefaultArray(&arr); - for (auto pd : pds) { - if (s.Size() == pd.get_size() / sizeof(mshadow::default_real_t)) { - const mkldnn::memory *mem = arr.GetMKLDNNDataReorder(pd); - printf("reorder from ("); - for (size_t i = 0; i < s.ndim(); i++) - printf("%ld, ", s[i]); - printf(") to ("); - for (int i = 0; i < pd.desc().data.ndims; i++) - printf("%d, ", pd.desc().data.dims[i]); - printf("), format: %d\n", pd.desc().data.format); - MKLDNNStream::Get()->Submit(false); - VerifyMem(*mem); - MKLDNNStream::Get()->Cleanup(); - } - } - } - - // Reorder from a special layout to another layout. - for (auto s : shapes) { - for (auto from_pd : pds) { - if (from_pd.get_size() / sizeof(mshadow::default_real_t) == s.Size()) { - NDArray arr(s, Context()); - // There is possibility that the dimensions of an NDArray doesn't match - // with the MKLDNN memory inside. - printf("Init array ("); - for (size_t i = 0; i < s.ndim(); i++) - printf("%ld, ", s[i]); - printf(") with MKLDNN memory ("); - for (int i = 0; i < from_pd.desc().data.ndims; i++) - printf("%d, ", from_pd.desc().data.dims[i]); - printf("), format: %d\n", from_pd.desc().data.format); - InitMKLDNNArray(&arr, from_pd); - for (auto to_pd : pds) { - if (to_pd.get_size() / sizeof(mshadow::default_real_t) == s.Size()) { - const mkldnn::memory *mem = arr.GetMKLDNNDataReorder(to_pd); - printf("reorder from ("); - for (size_t i = 0; i < s.ndim(); i++) - printf("%ld, ", s[i]); - printf("), format: %d to (", - arr.GetMKLDNNData()->get_primitive_desc().desc().data.format); - for (int i = 0; i < to_pd.desc().data.ndims; i++) - printf("%d, ", to_pd.desc().data.dims[i]); - printf("), format: %d\n", to_pd.desc().data.format); - MKLDNNStream::Get()->Submit(false); - VerifyMem(*mem); - MKLDNNStream::Get()->Cleanup(); - } - } - } - } - } -} - -struct NDArrayAttrs { - NDArray arr; - std::string desc; - NDArrayAttrs(NDArray arr, std::string desc) : arr(arr), desc(desc) {} -}; - -struct OpAttrs { - nnvm::NodeAttrs attrs; - std::vector dispatches; - std::set requests; - int num_inputs; - int num_outputs; - int input_types; - int output_types; -}; - -enum ArrayTypes { - Normal = 1, - MKLDNN = 2, - MKLDNNDiffShape = 4, - MKLDNNDiffDim = 8, - NormalReshaped = 16, - MKLDNNReshaped = 32, - MKLDNNReshapedDiffShape = 64, - MKLDNNReshapedDiffDim = 128, - NormalReused = 256, - MKLDNNReused = 512, - MKLDNNReusedDiffDim = 1024, - NormalReshapedReused = 2048, - NormalReusedDiffDtype = 4096, - All = 8191, -}; - -OpAttrs GetCopyOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_copy"); - attrs.num_inputs = 1; - attrs.num_outputs = 1; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetCopyBackwardsOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_copy"); - attrs.num_inputs = 1; - attrs.num_outputs = 1; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetReluOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("Activation"); - attrs.attrs.dict.insert({"act_type", "relu"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.num_inputs = 1; - attrs.num_outputs = 1; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetReluBackwardsOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_Activation"); - attrs.attrs.dict.insert({"act_type", "relu"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.num_inputs = 2; - attrs.num_outputs = 1; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetSumOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("elemwise_add"); - attrs.num_inputs = 2; - attrs.num_outputs = 1; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetSumBackwardsOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_add"); - attrs.num_inputs = 1; - attrs.num_outputs = 2; - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - attrs.requests.insert(OpReqType::kWriteTo); - attrs.requests.insert(OpReqType::kWriteInplace); - attrs.requests.insert(OpReqType::kAddTo); - return attrs; -} - -OpAttrs GetConcatOp(int num_args, int dim) { - OpAttrs attrs; - attrs.attrs.op = Op::Get("concat"); - attrs.num_inputs = num_args; - attrs.num_outputs = 1; - attrs.attrs.dict.insert({"num_args" , std::to_string(num_args)}); - attrs.attrs.dict.insert({"dim" , std::to_string(dim)}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - return attrs; -} - -OpAttrs GetConcatBackwardsOp(int num_args, int dim) { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_Concat"); - attrs.num_inputs = 2; - attrs.num_outputs = num_args; - attrs.attrs.dict.insert({"num_args" , std::to_string(num_args)}); - attrs.attrs.dict.insert({"dim" , std::to_string(dim)}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.dispatches.resize(2); - attrs.dispatches[0] = DispatchMode::kFCompute; - attrs.dispatches[1] = DispatchMode::kFComputeEx; - return attrs; -} - -std::string CreateShapeString(int value, int dim) { - std::stringstream ss; - ss << "("; - for (int i = 0; i < dim; i++) { - ss << value; - if (i != dim - 1) ss << ","; - } - ss << ")"; - return ss.str(); -} - - -OpAttrs GetPoolingOp(int kernel, int dim, int stride, int pad) { - OpAttrs attrs; - attrs.attrs.op = Op::Get("Pooling"); - attrs.num_inputs = 1; - attrs.num_outputs = dim == 2 ? 2 : 1; - attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); - attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); - attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); - attrs.attrs.dict.insert({"pool_type" , "max"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - return attrs; -} - -OpAttrs GetPoolingBackwardsOp(int kernel, int dim, int stride, int pad) { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_Pooling"); - attrs.num_inputs = dim == 2 ? 5 : 3; - attrs.num_outputs = 1; - attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); - attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); - attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); - attrs.attrs.dict.insert({"pool_type" , "max"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - return attrs; -} - - -void PrintVerifyMsg(const NDArrayAttrs &arr1, const NDArrayAttrs &arr2) { - TShape t1 = arr1.arr.shape(); - TShape t2 = arr2.arr.shape(); - std::stringstream ss; - std::cout << "Verifying: " << arr1.desc.c_str() << " " << - t1 << " with " << arr2.desc.c_str() << " " << t2 << "\n"; -} - -OpAttrs GetLRNOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("LRN"); - attrs.num_inputs = 1; - attrs.num_outputs = 2; - attrs.attrs.dict.insert({"nsize" , "3"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.dispatches.resize(2); - attrs.requests.insert(OpReqType::kWriteTo); - attrs.input_types = ArrayTypes::Normal | - ArrayTypes::MKLDNN | - ArrayTypes::NormalReshaped | - ArrayTypes::MKLDNNReshaped; - attrs.output_types = ArrayTypes::Normal | - ArrayTypes::MKLDNN | - ArrayTypes::NormalReshaped | - ArrayTypes::MKLDNNReshaped; - return attrs; -} - -OpAttrs GetLRNBackwardsOp() { - OpAttrs attrs; - attrs.attrs.op = Op::Get("_backward_LRN"); - attrs.num_inputs = 3; - attrs.num_outputs = 1; - attrs.attrs.dict.insert({"nsize" , "3"}); - attrs.attrs.op->attr_parser(&attrs.attrs); - attrs.dispatches.resize(2); - attrs.requests.insert(OpReqType::kWriteTo); - return attrs; -} - -/* - * We want to get a few types of NDArrays for testing: - * 1. Normal NDArray - * 2. Normal NDArray with MKLDNN layout (output from an MKLDNN operator) - * 3. Normal NDArray with MKLDNN layout whose MKLDNN memory may have different - * dimensions from the NDArray (result of MKLDNNDataReorderAsync). However, this - * type of NDArrays only exists for weight arrays. I don't think we should - * pass them to all operators. - * In the inference mode, the MKLDNN memory in the weight array will be - * reordered to 5 dimensions. - * 4. Reshaped/sliced NDArray - * 5. Reshaped/sliced NDArray with MKLDNN layout (reshape/slice from Normal NDArray - * with MKLDNN layout) - * 6. Reshaped/sliced NDArray with MKLDNN layout whose MKLDNN memory may have - * different dimensions from the NDArray (result of MKLDNNDataReorderAsync). - * However, this type of NDArrays only exists for weight arrays. I don't think - * we should pass them to all operators. - * In the inference mode, the MKLDNN memory in the weight array will be - * reordered to 5 dimensions. - * - * num_inputs / dim arguments used to scale shape (used for concat backwards to enlarge input shapes) - */ -std::vector GetTestInputArrays( - int types = ArrayTypes::All, bool rand = false, - int num_inputs = 1, int dim = 0) { - TestArrayShapes tas = GetTestArrayShapes(); - std::vector shapes = tas.shapes; - std::vector pds = tas.pds; - - std::vector in_arrs; - std::string desc; - - int slice_amount = 1; - if (dim == 0) - slice_amount = num_inputs; - for (auto shape : shapes) { - if (dim >= shape.ndim()) - continue; - shape[dim] = shape[dim] * num_inputs; - - // Type 1. - NDArray arr(shape, Context()); - if (types & ArrayTypes::Normal) { - InitDefaultArray(&arr, rand); - in_arrs.emplace_back(arr, "Normal NDArray"); - } - - // Type 4 - arr = NDArray(shape, Context()); - if (types & ArrayTypes::NormalReshaped) { - InitDefaultArray(&arr, rand); - in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), - "Reshaped Normal NDArray"); - } - - - for (auto pd : pds) { - if (num_inputs > 1) { - // preserve if matching layout else just expand on 0 dim - if (shape.ndim() == pd.desc().data.ndims) - pd = GetExpandedMemPD(pd, num_inputs, dim); - else - pd = GetExpandedMemPD(pd, num_inputs); - } - - if (shape.Size() != pd.get_size() / sizeof(mshadow::default_real_t)) - continue; - - // Type 2, 3. - arr = NDArray(shape, Context()); - if (shape.ndim() == pd.desc().data.ndims && IsSameShape(pd, shape) - && types & ArrayTypes::MKLDNN) { - desc = "MKLDNN NDArray"; - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr, desc); - } else if (shape.ndim() == pd.desc().data.ndims && !IsSameShape(pd, shape) - && types & ArrayTypes::MKLDNNDiffShape) { - desc = "MKLDNN NDArray with different shape"; - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr, desc); - } else if (shape.ndim() != pd.desc().data.ndims && types & ArrayTypes::MKLDNNDiffDim) { - std::stringstream ss; - ss << "MKLDNN NDArray with different dim " << - shape.ndim() << "/" << pd.desc().data.ndims; - desc = ss.str(); - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr, desc); - } - - - // Type 5, 6. - arr = NDArray(shape, Context()); - if (shape.ndim() == pd.desc().data.ndims && IsSameShape(pd, shape) - && types & ArrayTypes::MKLDNNReshaped) { - desc = "Reshaped MKLDNN NDArray"; - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); - } else if (shape.ndim() == pd.desc().data.ndims && !IsSameShape(pd, shape) - && types & ArrayTypes::MKLDNNReshapedDiffShape) { - desc = "Reshaped MKLDNN NDArray with different shape"; - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); - } else if (shape.ndim() != pd.desc().data.ndims - && types & ArrayTypes::MKLDNNReshapedDiffDim) { - std::stringstream ss; - ss << "MKLDNN NDArray with different dim " << - shape.ndim() << "/" << pd.desc().data.ndims; - desc = ss.str(); - InitMKLDNNArray(&arr, pd, rand); - in_arrs.emplace_back(arr.Slice(slice_amount, arr.shape()[0] - slice_amount), desc); - } - } - } - return in_arrs; -} - -/* - * We want to get a few types of NDArrays for testing: - * 1. Normal NDArray - * 2. Normal NDArray with MKLDNN layout (output from an MKLDNN operator) - * 3. Normal NDArray with MKLDNN layout whose MKLDNN memory may have different - * dimensions from the NDArray (result of MKLDNNDataReorderAsync). However, this - * type of NDArrays only exists for weight arrays. I don't think we should - * pass them to all operators. - * In the inference mode, the MKLDNN memory in the weight array will be - * reordered to 5 dimensions. - * 4. Reshaped/sliced NDArray - * 5. Reused NDArray (this is created by the MXNet executor). This type of - * NDArrays can only be used as output arrays. - * 6. Reused NDArray converted from an array with a different data type. - * 7. Reused reshaped/sliced NDArray. - * 8. Reused NDArray with MKLDNN layout. - * 9. Reused NDArray with MKLDNN layout of different dimensions. - * - * Optional num_inputs / dim args can be passed to modify input shape (used for Concat test) - */ -std::vector GetTestOutputArrays( - const TShape &shp, - const std::vector &pds, - std::vectorscale = {1}, bool rand = true, int types = ArrayTypes::All) { - TShape shape = shp; - - for (int dim = 0; dim < scale.size(); dim++) - shape[dim] = static_cast(shape[dim] * scale[dim]); - - std::vector in_arrs; - std::string desc; - // Type 1. - NDArray arr(shape, Context()); - - if (types & ArrayTypes::Normal) { - in_arrs.emplace_back(arr, "Normal NDArray"); - InitDefaultArray(&in_arrs.back().arr, rand); - } - - TShape tmp_shape = shape; - if (types & ArrayTypes::NormalReshaped) { - // Type 4. - tmp_shape[0] = shape[0] * 2; - NDArray arr0(tmp_shape, Context()); - InitDefaultArray(&arr0, rand); - in_arrs.emplace_back(arr0.Slice(1, shape[0] + 1), "Reshaped NDArray"); - } - - nnvm::TShape s(1); - if (types & ArrayTypes::NormalReused) { - // Type 5. - // Get a reused version. - s[0] = shape.Size(); - NDArray arr1(s, Context()); - arr1 = arr1.AsArray(shape, arr1.dtype()); - InitDefaultArray(&arr1, rand); - in_arrs.emplace_back(arr1, "Reused NDArray"); - } - - if (types & ArrayTypes::NormalReusedDiffDtype) { - // Type 6. - s[0] = shape.Size() * GetTypeSize(mshadow::default_type_flag); - NDArray arr2(s, Context(), true, mshadow::kUint8); - arr2 = arr2.AsArray(shape, mshadow::default_type_flag); - InitDefaultArray(&arr2, rand); - in_arrs.emplace_back(arr2, "Reused NDArray with diff data type"); - } - - if (types & ArrayTypes::NormalReshapedReused) { - // Type 7 - s[0] = shape.Size() * GetTypeSize(mshadow::default_type_flag) * 2; - NDArray arr3(s, Context(), true, mshadow::kUint8); - tmp_shape[0] = shape[0] * 2; - arr3 = arr3.AsArray(tmp_shape, mshadow::default_type_flag); - InitDefaultArray(&arr3, rand); - in_arrs.emplace_back(arr3.Slice(1, shape[0] + 1), "Reused+Reshaped NDArray"); - } - - for (auto pd : pds) { - if (shape.Size() != pd.get_size() / sizeof(mshadow::default_real_t)) - continue; - - if (scale.size() > pd.desc().data.ndims) - continue; - - for (int dim = 0; dim < scale.size(); dim++) - pd = GetExpandedMemPD(pd, scale[dim]); - - // Type 2, 3. - arr = NDArray(shape, Context()); - desc = "MKLDNN NDArray"; - if (shape.ndim() != pd.desc().data.ndims) { - std::stringstream ss; - ss << "MKLDNN NDArray with different memory layout " - << shape.ndim() << "/" << pd.desc().data.ndims; - desc = ss.str(); - } - - if ((types & ArrayTypes::MKLDNN && shape.ndim() == pd.desc().data.ndims) || - (types & ArrayTypes::MKLDNNDiffDim && shape.ndim() != pd.desc().data.ndims)) { - in_arrs.emplace_back(arr, desc); - InitMKLDNNArray(&in_arrs.back().arr, pd, rand); - } - - // Type 8, 9. - // Get a reused version. - nnvm::TShape s(1); - s[0] = shape.Size(); - NDArray arr = NDArray(s, Context()); - arr = arr.AsArray(shape, arr.dtype()); - InitMKLDNNArray(&arr, pd, rand); - desc = "Reused MKLDNN NDArray"; - if (shape.ndim() != pd.desc().data.ndims) { - std::stringstream ss; - ss << "Reused MKLDNN NDArray with different memory layout " - << shape.ndim() << "/" << pd.desc().data.ndims; - desc = ss.str(); - } - - if ((types & ArrayTypes::MKLDNNReused && shape.ndim() == pd.desc().data.ndims) || - (types & ArrayTypes::MKLDNNReusedDiffDim && shape.ndim() != pd.desc().data.ndims)) { - in_arrs.emplace_back(arr, desc); - } - } - return in_arrs; -} - -TEST(MKLDNN_NDArray, GetTestInputArraysConcat) { - auto in_arrs = GetTestInputArrays(); - for (int dim = 0; dim < 5; dim++) { - for (int num_inputs = 2; num_inputs < 5; num_inputs++) { - std::vector expanded_arrs = GetTestInputArrays( - ArrayTypes::All, false, num_inputs, dim); - int i = 0; - for (auto &arr : in_arrs) { - if (dim >= arr.arr.shape().ndim()) - continue; - auto ex_arr = expanded_arrs[i]; - PrintVerifyMsg(arr, ex_arr); - EXPECT_EQ(arr.arr.shape().Size() * num_inputs, ex_arr.arr.shape().Size()); - EXPECT_EQ(arr.arr.shape()[dim] * num_inputs, ex_arr.arr.shape()[dim]); - i++; - } - } - } -} - -TEST(MKLDNN_NDArray, GetTestOutputArraysConcat) { - auto shapes_pds = GetTestArrayShapes(); - std::vector shapes; shapes = shapes_pds.shapes; - std::vector pds = shapes_pds.pds; - for (auto &shape : shapes) { - for (int dim = 0; dim < 5; dim++) { - for (int num_inputs = 2; num_inputs < 5; num_inputs++) { - if (shape.ndim() <= dim) - continue; - std::cout << "Extending " << shape << " dim " << - dim << " and " << num_inputs << "num_inputs\n"; - std::vector scale_vector(shape.ndim()); - for (int i = 0; i < shape.ndim(); i++) - scale_vector[i] = 1; - scale_vector[dim] = num_inputs; - auto output_arrs = GetTestOutputArrays(shape, pds, scale_vector); - for (auto &out_arr : output_arrs) { - auto out_shape = out_arr.arr.shape(); - EXPECT_EQ(shape.Size() * num_inputs, out_arr.arr.shape().Size()); - EXPECT_EQ(shape[dim] * num_inputs, out_arr.arr.shape()[dim]); - } - } - } - } -} - -void VerifyCopyResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray tmp1 = in_arrs[0]->Reorder2Default(); - NDArray tmp2 = out_arrs[0]->Reorder2Default(); - EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); - TBlob d1 = tmp1.data(); - TBlob d2 = tmp2.data(); - EXPECT_EQ(memcmp(d1.dptr_, d2.dptr_, - tmp1.shape().Size() * sizeof(mshadow::default_real_t)), 0); -} - -void AssertEqual(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray tmp1 = in_arrs[0]->Reorder2Default(); - NDArray tmp2 = out_arrs[0]->Reorder2Default(); - EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); - TBlob blob1 = tmp1.data(); - TBlob blob2 = tmp2.data(); - mshadow::default_real_t *d1 = static_cast(blob1.dptr_); - mshadow::default_real_t *d2 = static_cast(blob2.dptr_); - for (int i = 0; i < tmp1.shape().Size(); i++) - ASSERT_FLOAT_EQ(d1[i], d2[i]); -} - -void VerifyActResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray tmp1 = in_arrs[0]->Reorder2Default(); - NDArray tmp2 = out_arrs[0]->Reorder2Default(); - TBlob blob1 = tmp1.data(); - TBlob blob2 = tmp2.data(); - mshadow::default_real_t *d1 = static_cast(blob1.dptr_); - mshadow::default_real_t *d2 = static_cast(blob2.dptr_); - EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); - for (size_t i = 0; i < tmp1.shape().Size(); i++) { - EXPECT_EQ(std::fmax(d1[i], 0), d2[i]); - } -} - -void VerifySumResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray in1 = in_arrs[0]->Reorder2Default(); - NDArray in2 = in_arrs[1]->Reorder2Default(); - NDArray out = out_arrs[0]->Reorder2Default(); - EXPECT_EQ(in1.shape().Size(), in2.shape().Size()); - EXPECT_EQ(in1.shape().Size(), out.shape().Size()); - - mshadow::default_real_t *d1 = in1.data().dptr(); - mshadow::default_real_t *d2 = in2.data().dptr(); - mshadow::default_real_t *o = out.data().dptr(); - for (size_t i = 0; i < in1.shape().Size(); i++) - ASSERT_EQ(d1[i] + d2[i], o[i]); -} - -void VerifyActBackwardsResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray tmp1 = in_arrs[0]->Reorder2Default(); // out grads - NDArray tmp2 = in_arrs[1]->Reorder2Default(); // input - NDArray tmp3 = out_arrs[0]->Reorder2Default(); // input grads - TBlob blob1 = tmp1.data(); - TBlob blob2 = tmp2.data(); - TBlob blob3 = tmp3.data(); - mshadow::default_real_t *d1 = static_cast(blob1.dptr_); - mshadow::default_real_t *d2 = static_cast(blob2.dptr_); - mshadow::default_real_t *d3 = static_cast(blob3.dptr_); - EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); - for (size_t i = 0; i < tmp1.shape().Size(); i++) { - ASSERT_EQ(d2[i] > 0 ? d1[i] : 0, d3[i]); - } -} - -void VerifySumBackwardsResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - NDArray out_grads = in_arrs[0]->Reorder2Default(); // out grads - NDArray input_grads1 = out_arrs[0]->Reorder2Default(); // input grads - NDArray input_grads2 = out_arrs[1]->Reorder2Default(); // input grads - mshadow::default_real_t *og = out_grads.data().dptr(); - mshadow::default_real_t *ig1 = input_grads1.data().dptr(); - mshadow::default_real_t *ig2 = input_grads2.data().dptr(); - for (size_t i = 0; i < out_grads.shape().Size(); i++) { - ASSERT_EQ(og[i], ig1[i]); - ASSERT_EQ(og[i], ig2[i]); - } -} - -/* - * Determines axis ndarrays are concatenated by - * Used to verify concat/concat backwards operator - */ -int GetDim(TShape input_shape, TShape output_shape) { - CHECK(input_shape.Size() != output_shape.Size()); - for (size_t i = 0; i < input_shape.ndim(); i++) { - if (input_shape[i] != output_shape[i]) - return i; - } - return -1; -} - -/* - * Calculates the size of continuous block of array inside larger concatenated array - * Used to verify concat/concat backwards operator - */ -int GetBlockSize(TShape shape, int dim) { - int block_size = 1; - for (int i = shape.ndim() - 1; i >= dim; i--) - block_size *= shape[i]; - return block_size; -} - -void VerifyConcatResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - int num_inputs = in_arrs.size(); - int input_size = in_arrs[0]->shape().Size(); - TShape input_shape = in_arrs[0]->shape(); - NDArray output = out_arrs[0]->Reorder2Default(); - size_t total_size = output.shape().Size(); - EXPECT_EQ(input_size * num_inputs, total_size); - mshadow::default_real_t *out_data = output.data().dptr(); - - int dim = GetDim(input_shape, output.shape()); - int block_size = GetBlockSize(input_shape, dim); - int num_blocks = input_size / block_size; - for (size_t input_num = 0; input_num < num_inputs; input_num++) { - NDArray tmp = in_arrs[input_num]->Reorder2Default(); - mshadow::default_real_t* data = tmp.data().dptr(); - for (size_t block_num = 0; block_num < num_blocks; block_num++) { - for (size_t i = 0; i < block_size; i++) - ASSERT_EQ(data[block_num * block_size + i], - out_data[(block_num * num_inputs + input_num) * block_size + i]); - } - } -} - -void VerifyAddRequest(const std::vector &in_arrs, - const std::vector &original_outputs, - const std::vector &new_outputs, - VerifyFunc verify_fn) { - CHECK(original_outputs.size() == new_outputs.size()); - std::vector tmp_outputs; - NDArray tmp; - for (size_t i = 0; i < new_outputs.size(); i++) { - tmp = new_outputs[i]->Reorder2Default() - original_outputs[i]->Reorder2Default(); - tmp_outputs.push_back(&tmp); - } - Engine::Get()->WaitForAll(); - verify_fn(in_arrs, tmp_outputs); -} - -void VerifyConcatBackwardsResult(const std::vector &in_arrs, - const std::vector &out_arrs) { - // in_arrs is larger array, out_arr is ammler - int num_inputs = out_arrs.size(); - int input_size = out_arrs[0]->shape().Size(); - TShape input_shape = out_arrs[0]->shape(); - NDArray output = in_arrs[0]->Reorder2Default(); - size_t total_size = output.shape().Size(); - EXPECT_EQ(input_size * num_inputs, total_size); - mshadow::default_real_t *out_data = output.data().dptr(); - - int dim = GetDim(input_shape, output.shape()); - int block_size = GetBlockSize(input_shape, dim); - int num_blocks = input_size / block_size; - for (size_t input_num = 0; input_num < num_inputs; input_num++) { - NDArray tmp = out_arrs[input_num]->Reorder2Default(); - mshadow::default_real_t* data = tmp.data().dptr(); - for (size_t block_num = 0; block_num < num_blocks; block_num++) { - for (size_t i = 0; i < block_size; i++) - ASSERT_EQ(data[block_num * block_size + i], - out_data[(block_num * num_inputs + input_num) * block_size + i]); - } - } -} - -TEST(MKLDNN_NDArray, CopyFrom) { - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - std::vector in_arrs = GetTestInputArrays(); - for (auto &in_arr : in_arrs) { - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) - continue; - std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (auto &out_arr : out_arrs) { - const mkldnn::memory *mem = in_arr.arr.GetMKLDNNData(); - out_arr.arr.CopyFrom(*mem); - MKLDNNStream::Get()->Submit(); - std::vector inputs(1); - inputs[0] = &in_arr.arr; - VerifyCopyResult(inputs, {&out_arr.arr}); - } - } -} - -void TestOp(const OpAttrs &attrs, VerifyFunc verify_fn) { - std::vector inputs(attrs.num_inputs); - std::vector outputs(attrs.num_outputs); - std::vector req(attrs.num_outputs); - std::vector in_arrs; - std::vector> out_arrs(attrs.num_outputs); - std::vector dispatches = attrs.dispatches; - - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - if (attrs.requests.find(OpReqType::kWriteTo) != attrs.requests.end()) { - std::vector in_arrs = GetTestInputArrays(); - for (auto &in_arr : in_arrs) { - for (auto &dispatch : dispatches) { - std::vector> out_arrs(attrs.num_outputs); - for (int i = 0; i < attrs.num_outputs; i++) - out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (int i = 0; i < attrs.num_inputs; i++) - inputs[i] = &in_arr.arr; - for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { - for (int i = 0; i < attrs.num_outputs; i++) { - req[i] = kWriteTo; - outputs[i] = &out_arrs[i][output_i].arr; - } - PrintVerifyMsg(in_arr, out_arrs[0][output_i]); - Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, - outputs, req, dispatch, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - verify_fn(inputs, outputs); - } - } - } - } - - if (attrs.requests.find(OpReqType::kWriteInplace) != attrs.requests.end()) { - for (auto &dispatch : dispatches) { - in_arrs = GetTestInputArrays(); - for (auto &arr : in_arrs) { - // If the array is a view, we shouldn't write data to it. - if (arr.arr.IsView()) - continue; - NDArrayAttrs orig(arr.arr.Copy(arr.arr.ctx()), "InPlace Copy"); - for (int i = 0; i < attrs.num_inputs; i++) - inputs[i] = &arr.arr; - for (int i = 0; i < attrs.num_outputs; i++) { - req[i] = kWriteInplace; - outputs[i] = &arr.arr; - } - PrintVerifyMsg(orig, arr); - Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, outputs, req, - dispatch, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - std::vector orig_inputs(attrs.num_inputs); - for (int i = 0; i < attrs.num_inputs; i++) - orig_inputs[i] = &orig.arr; - verify_fn(orig_inputs, outputs); - } - } - } - - if (attrs.requests.find(OpReqType::kAddTo) != attrs.requests.end()) { - std::vector original_outputs(attrs.num_outputs); - in_arrs = GetTestInputArrays(); - for (auto &in_arr : in_arrs) { - for (auto &dispatch : dispatches) { - for (int i = 0; i < attrs.num_outputs; i++) - out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (size_t i = 0; i < attrs.num_inputs; i++) - inputs[i] = &in_arr.arr; - for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { - NDArray tmp; - for (size_t i = 0; i < attrs.num_outputs; i++) { - auto out_arr = out_arrs[i][output_i]; - tmp = out_arr.arr.Copy(out_arr.arr.ctx()); - original_outputs[i] = &tmp; - outputs[i] = &out_arrs[i][output_i].arr; - req[i] = kAddTo; - } - PrintVerifyMsg(in_arr, out_arrs[0][output_i]); - Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, - outputs, req, dispatch, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - VerifyAddRequest(inputs, original_outputs, outputs, verify_fn); - } - } - } - } -} - -void TestConcatOp(const OpAttrs &attrs, VerifyFunc verify_fn, - bool backwards = false) { - std::vector inputs(attrs.num_inputs); - std::vector outputs(attrs.num_outputs); - std::vector req(attrs.num_outputs); - std::vector dispatches = attrs.dispatches; - - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - std::vector in_arrs = GetTestInputArrays(); - - // concat backwards uses scaled up inputs - if (backwards) { - std::string str_dim = const_cast(attrs).attrs.dict["dim"]; - int dim = std::stoi(str_dim); - in_arrs = GetTestInputArrays(ArrayTypes::All, false, attrs.num_outputs, dim); - } - - for (auto &in_arr : in_arrs) { - for (auto &dispatch : dispatches) { - std::vector> out_arrs(attrs.num_outputs); - - std::string str_dim = const_cast(attrs).attrs.dict["dim"]; - int dim = std::stoi(str_dim); - if (dim >= in_arr.arr.shape().ndim()) - continue; - float scale = backwards ? 1 / static_cast(attrs.num_outputs) : - static_cast(attrs.num_inputs); - - std::vector scale_vector(in_arr.arr.shape().ndim()); - for (int i = 0; i < in_arr.arr.shape().ndim(); i++) - scale_vector[i] = 1; - scale_vector[dim] = scale; - for (int i = 0; i < attrs.num_outputs; i++) - out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); - - for (int i = 0; i < attrs.num_inputs; i++) - inputs[i] = &in_arr.arr; - - for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { - for (int i = 0; i < attrs.num_outputs; i++) { - req[i] = kWriteTo; - outputs[i] = &out_arrs[i][output_i].arr; - } - PrintVerifyMsg(in_arr, out_arrs[0][output_i]); - Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, - outputs, req, dispatch, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - verify_fn(inputs, outputs); - } - } - } -} - -// compares output of fcompute with fcomputex -void TestOpEx(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { - std::vector inputs(forward_attrs.num_inputs); - std::vector outputs(forward_attrs.num_outputs); - std::vector ex_outputs(forward_attrs.num_outputs); - - std::vector backwards_input(backwards_attrs.num_inputs); - std::vector backwards_outputs(backwards_attrs.num_outputs); - std::vector backwards_ex_outputs(backwards_attrs.num_outputs); - - - std::vector req(forward_attrs.num_outputs); - std::vector back_req(backwards_attrs.num_outputs); - - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - std::vector in_arrs = GetTestInputArrays(forward_attrs.input_types, true); - std::vector> out_arrs(forward_attrs.num_outputs); - std::vector> ex_out_arrs(forward_attrs.num_outputs); - - if (forward_attrs.requests.find(OpReqType::kWriteTo) != forward_attrs.requests.end()) { - for (int i1 = 0; i1 < in_arrs.size(); i1++) { - auto in_arr = in_arrs[i1]; - - // TODO(alex): (MXNET-845) Remove when MKLDNN supports other dims - if (in_arr.arr.shape().ndim() != 4) - continue; - - for (int i = 0; i < forward_attrs.num_outputs; i++) { - out_arrs[i] = - GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, forward_attrs.output_types); - ex_out_arrs[i] = - GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, forward_attrs.output_types); - } - - for (int i = 0; i < forward_attrs.num_inputs; i++) - inputs[i] = &in_arr.arr; - - for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { - if (out_arrs[0][output_i].arr.IsMKLDNNData()) - continue; - - for (int i = 0; i < forward_attrs.num_outputs; i++) { - req[i] = kWriteTo; - outputs[i] = &out_arrs[i][output_i].arr; - ex_outputs[i] = &ex_out_arrs[i][output_i].arr; - } - Imperative::Get()->set_is_training(true); - - PrintVerifyMsg(in_arr, out_arrs[0][output_i]); - Imperative::Get()->InvokeOp( - Context(), forward_attrs.attrs, inputs, outputs, req, - DispatchMode::kFCompute, mxnet::OpStatePtr()); - Imperative::Get()->InvokeOp( - Context(), forward_attrs.attrs, inputs, ex_outputs, req, - DispatchMode::kFComputeEx, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - AssertEqual(outputs, ex_outputs); - - // backwards test performed same time since output needed - backwards_input[0] = outputs[0]; // output grad - backwards_input[1] = inputs[0]; // input - backwards_input[2] = outputs[1]; // out norm - - auto tmp_output = GetTestInputArrays(forward_attrs.input_types, true)[i1]; - backwards_outputs[0] = &tmp_output.arr; - - auto tmp_output2 = GetTestInputArrays(forward_attrs.input_types, true)[i1]; - backwards_ex_outputs[0] = &tmp_output2.arr; - - for (int i = 0; i < backwards_attrs.num_outputs; i++) - back_req[i] = kWriteTo; - - std::cout << "Backwards: "; - PrintVerifyMsg(out_arrs[0][output_i], tmp_output); - Imperative::Get()->InvokeOp( - Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, - back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); - Imperative::Get()->InvokeOp( - Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, - back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - AssertEqual(backwards_outputs, backwards_ex_outputs); - } - } - } -} - -int CalculateWidthPoolOutput(int width, int kernel, int padding, int stride) { - return (width - kernel + 2 * padding) / stride + 1; -} - -void TestPoolingOp(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { - std::vector inputs(forward_attrs.num_inputs); - std::vector outputs(forward_attrs.num_outputs); - std::vector ex_outputs(forward_attrs.num_outputs); - - std::vector backwards_input(backwards_attrs.num_inputs); - std::vector backwards_outputs(backwards_attrs.num_outputs); - std::vector backwards_ex_outputs(backwards_attrs.num_outputs); - - - std::vector req(forward_attrs.num_outputs); - std::vector back_req(backwards_attrs.num_outputs); - std::vector dispatches = forward_attrs.dispatches; - - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - mxnet::op::PoolingParam param; - param.Init(forward_attrs.attrs.dict); - TShape kernel = param.kernel; - TShape padding = param.pad; - TShape stride = param.stride; - - std::vector in_arrs = GetTestInputArrays(); - std::vector> out_arrs(forward_attrs.num_outputs); - std::vector> ex_out_arrs(forward_attrs.num_outputs); - - for (int i1 = 0; i1 < in_arrs.size(); i1++) { - auto in_arr = in_arrs[i1]; - - // can only pool only 3D and 4D inputs - TShape input_shape = in_arr.arr.shape(); - if (input_shape.ndim() != kernel.ndim() + 2) - continue; - // cannot pool if ndarray and mkldnn memory have different ndim - if (in_arr.arr.IsView() || in_arr.arr.GetMKLDNNData()->get_primitive_desc().desc().data.ndims - != in_arr.arr.shape().ndim()) - continue; - std::vector scale_vector(in_arr.arr.shape().ndim()); - for (int i = 0; i < in_arr.arr.shape().ndim(); i++) { - if (i < 2) - scale_vector[i] = 1; - else - scale_vector[i] = CalculateWidthPoolOutput( - input_shape[i], kernel[i-2], padding[i-2], stride[i-2]) / - static_cast(input_shape[i]); - } - for (int i = 0; i < forward_attrs.num_outputs; i++) { - out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); - ex_out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); - } - - for (int i = 0; i < forward_attrs.num_inputs; i++) - inputs[i] = &in_arr.arr; - - for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { - for (int i = 0; i < forward_attrs.num_outputs; i++) { - req[i] = kWriteTo; - outputs[i] = &out_arrs[i][output_i].arr; - ex_outputs[i] = &ex_out_arrs[i][output_i].arr; - } - Imperative::Get()->set_is_training(true); - - PrintVerifyMsg(in_arr, out_arrs[0][output_i]); - Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, - outputs, req, DispatchMode::kFCompute, mxnet::OpStatePtr()); - Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, - ex_outputs, req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - VerifyCopyResult(outputs, ex_outputs); - - - // backwards test performed same time since output needed - if (backwards_attrs.num_inputs == 3) { - backwards_input[0] = outputs[0]; // output grad - backwards_input[1] = inputs[0]; // input - backwards_input[2] = outputs[0]; // output - } else if (backwards_attrs.num_inputs == 5) { - backwards_input[0] = outputs[0]; // output grad - backwards_input[1] = outputs[0]; // workspace grad - backwards_input[2] = inputs[0]; // input - backwards_input[3] = outputs[0]; // output - backwards_input[4] = ex_outputs[1]; // workspace - } - - // needs copies of inputs since they be reused in next iteration - // cannot use Copy method since we need to maintain MKLDNN format - auto tmp_output = GetTestInputArrays()[i1]; - auto tmp_output2 = GetTestInputArrays()[i1]; - backwards_outputs[0] = &tmp_output.arr; - backwards_ex_outputs[0] = &tmp_output2.arr; - back_req[0] = kWriteTo; - std::cout << "Backwards: "; - PrintVerifyMsg(out_arrs[0][output_i], tmp_output); - Imperative::Get()->InvokeOp( - Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, - back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); - Imperative::Get()->InvokeOp( - Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, - back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); - Engine::Get()->WaitForAll(); - VerifyCopyResult(backwards_outputs, backwards_ex_outputs); - } - } -} - -TEST(IMPERATIVE, CopyOp) { - OpAttrs attrs = GetCopyOp(); - TestOp(attrs, VerifyCopyResult); -} - -TEST(IMPERATIVE, CopyBackwardsOp) { - OpAttrs attrs = GetCopyBackwardsOp(); - TestOp(attrs, VerifyCopyResult); -} - -TEST(IMPERATIVE, ActOp) { - OpAttrs attrs = GetReluOp(); - TestOp(attrs, VerifyActResult); -} - -TEST(IMPERATIVE, ActBackwardsOp) { - OpAttrs attrs = GetReluBackwardsOp(); - TestOp(attrs, VerifyActBackwardsResult); -} - -TEST(IMPERATIVE, SumOp) { - OpAttrs attrs = GetSumOp(); - TestOp(attrs, VerifySumResult); -} - -TEST(IMPERATIVE, SumBackwardsOp) { - OpAttrs attrs = GetSumBackwardsOp(); - TestOp(attrs, VerifySumBackwardsResult); -} - -TEST(IMPERATIVE, ConcatOp) { - for (int num_inputs = 2; num_inputs < 4; num_inputs++) { - for (int dim = 0; dim < 5; dim++) { - OpAttrs attrs = GetConcatOp(num_inputs, dim); - TestConcatOp(attrs, VerifyConcatResult); - } - } -} - -TEST(IMPERATIVE, ConcatBackwardsOp) { - for (int num_inputs = 2; num_inputs < 4; num_inputs++) { - for (int dim = 0; dim < 5; dim++) { - OpAttrs attrs = GetConcatBackwardsOp(num_inputs, dim); - TestConcatOp(attrs, VerifyConcatBackwardsResult, true); - } - } -} - -TEST(IMPERATIVE, LRNOp) { - OpAttrs forward_attrs = GetLRNOp(); - OpAttrs backwards_attrs = GetLRNBackwardsOp(); - TestOpEx(forward_attrs, backwards_attrs); -} - -TEST(IMPERATIVE, PoolingOp) { - for (int dim = 2; dim < 4; dim++) { - for (int kernel = 1; kernel < 4; kernel++) { - for (int stride = 1; stride < 3; stride++) { - for (int pad = 0; pad < 2; pad++) { - if (kernel / 2. < pad) - continue; - OpAttrs forward_attrs = GetPoolingOp(kernel, dim, stride, pad); - OpAttrs backwards_attrs = GetPoolingBackwardsOp(kernel, dim, stride, pad); - TestPoolingOp(forward_attrs, backwards_attrs); - } - } - } - } -} - -TEST(MKLDNN_BASE, MKLDNNSum) { - std::vector in_arrs = GetTestInputArrays(); - std::vector in_arrs2 = GetTestInputArrays(ArrayTypes::All, true); - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (auto &out_arr : out_arrs) { - auto in_mem1 = in_arr.arr.GetMKLDNNData(); - auto in_mem2 = in_arr2.arr.GetMKLDNNData(); - if (out_arr.arr.IsView()) - continue; - auto out_mem = out_arr.arr.GetMKLDNNData(); - PrintVerifyMsg(in_arr, in_arr); - op::MKLDNNSum(*in_mem1, *in_mem2, *out_mem); - MKLDNNStream::Get()->Submit(); - VerifySumResult({&in_arr.arr, &in_arr2.arr}, {&out_arr.arr}); - } - } - - // in place - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - auto input_mem = in_arr.arr.GetMKLDNNData(); - auto input_mem2 = in_arr2.arr.GetMKLDNNData(); - NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); - orig_arr.arr.WaitToRead(); - PrintVerifyMsg(orig_arr, in_arr); - InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); - orig_arr.arr.CopyFrom(*input_mem); - op::MKLDNNSum(*input_mem, *input_mem2, *input_mem); - MKLDNNStream::Get()->Submit(); - VerifySumResult({&orig_arr.arr, &in_arr2.arr}, {&in_arr.arr}); - } -} - -TEST(MKLDNN_BASE, CreateMKLDNNMem) { - std::vector in_arrs = GetTestInputArrays(); - std::vector in_arrs2 = GetTestInputArrays(ArrayTypes::All, true); - TestArrayShapes tas = GetTestArrayShapes(); - std::vector pds = tas.pds; - MKLDNNStream *stream = MKLDNNStream::Get(); - - // kWriteTo - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (auto &out_arr : out_arrs) { - auto in_mem = in_arr.arr.GetMKLDNNData(); - auto in_mem2 = in_arr2.arr.GetMKLDNNData(); - NDArray orig_output = out_arr.arr.Copy(out_arr.arr.ctx()); - orig_output.WaitToRead(); - PrintVerifyMsg(in_arr, out_arr); - auto out_mem = out_arr.arr.GetMKLDNNData(); - auto output_mem_t = CreateMKLDNNMem(out_arr.arr, out_mem->get_primitive_desc(), kWriteTo); - op::MKLDNNSum(*in_mem, *in_mem2, *output_mem_t.second); - CommitOutput(out_arr.arr, output_mem_t); - stream->Submit(); - VerifySumResult({&in_arr.arr, &in_arr2.arr}, {&out_arr.arr}); - } - } - - // kWriteInPlace - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - auto input_mem = in_arr.arr.GetMKLDNNData(); - auto input_mem2 = in_arr2.arr.GetMKLDNNData(); - NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); - orig_arr.arr.WaitToRead(); - PrintVerifyMsg(orig_arr, in_arr); - InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); - orig_arr.arr.CopyFrom(*input_mem); - auto output_mem_t = CreateMKLDNNMem(in_arr.arr, - input_mem->get_primitive_desc(), kWriteInplace, &in_arr.arr); - op::MKLDNNSum(*input_mem, *input_mem2, *output_mem_t.second); - CommitOutput(in_arr.arr, output_mem_t); - stream->Submit(); - VerifySumResult({&orig_arr.arr, &in_arr2.arr}, {&in_arr.arr}); - } - - // kAddTo - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); - for (auto &out_arr : out_arrs) { - auto in_mem = in_arr.arr.GetMKLDNNData(); - auto in_mem2 = in_arr2.arr.GetMKLDNNData(); - NDArray orig_output = out_arr.arr.Copy(out_arr.arr.ctx()); - orig_output.WaitToRead(); - PrintVerifyMsg(in_arr, out_arr); - auto out_mem = out_arr.arr.GetMKLDNNData(); - auto output_mem_t = CreateMKLDNNMem(out_arr.arr, out_mem->get_primitive_desc(), kAddTo); - op::MKLDNNSum(*in_mem, *in_mem2, *output_mem_t.second); - CommitOutput(out_arr.arr, output_mem_t); - stream->Submit(); - VerifyAddRequest( - {&in_arr.arr, &in_arr2.arr}, {&orig_output}, {&out_arr.arr}, VerifySumResult); - } - } - - // kNullOp - for (int i = 0; i < in_arrs.size(); i++) { - auto in_arr = in_arrs[i]; - auto in_arr2 = in_arrs2[i]; - if (!SupportMKLDNN(in_arr.arr)) - continue; - if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { - continue; - } - auto input_mem = in_arr.arr.GetMKLDNNData(); - auto input_mem2 = in_arr2.arr.GetMKLDNNData(); - NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); - orig_arr.arr.WaitToRead(); - PrintVerifyMsg(orig_arr, in_arr); - InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); - orig_arr.arr.CopyFrom(*input_mem); - auto output_mem_t = CreateMKLDNNMem(in_arr.arr, input_mem->get_primitive_desc(), kNullOp); - op::MKLDNNSum(*input_mem, *input_mem2, *output_mem_t.second); - CommitOutput(in_arr.arr, output_mem_t); - stream->Submit(); - // original and input should be the same since noop - VerifyCopyResult({&orig_arr.arr}, {&in_arr.arr}); - } -} - -#endif diff --git a/tests/cpp/operator/mkldnn_operator_test.cc b/tests/cpp/operator/mkldnn_operator_test.cc new file mode 100644 index 000000000..3bf3228a4 --- /dev/null +++ b/tests/cpp/operator/mkldnn_operator_test.cc @@ -0,0 +1,1356 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file mkldnn_test.cc + * \brief test functions for mkldnn operators. + * \author Alex Zai + */ + +#if MXNET_USE_MKLDNN == 1 + +#include +#include +#include +#include +#include "gtest/gtest.h" +#include "mxnet/imperative.h" +#include "../../src/operator/nn/mkldnn/mkldnn_base-inl.h" +#include "../../src/operator/nn/mkldnn/mkldnn_ops-inl.h" +#include "../../src/operator/nn/mkldnn/mkldnn_pooling-inl.h" +#include "../../src/operator/nn/pooling-inl.h" +#include "../../src/operator/nn/convolution-inl.h" +#include "../../src/operator/nn/deconvolution-inl.h" +#include "../include/test_mkldnn.h" + +using namespace mxnet; + +OpAttrs GetCopyOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_copy"); + attrs.num_inputs = 1; + attrs.num_outputs = 1; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetCopyBackwardsOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_copy"); + attrs.num_inputs = 1; + attrs.num_outputs = 1; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetReluOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("Activation"); + attrs.attrs.dict.insert({"act_type", "relu"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.num_inputs = 1; + attrs.num_outputs = 1; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetReluBackwardsOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_Activation"); + attrs.attrs.dict.insert({"act_type", "relu"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.num_inputs = 2; + attrs.num_outputs = 1; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetSumOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("elemwise_add"); + attrs.num_inputs = 2; + attrs.num_outputs = 1; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetSumBackwardsOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_add"); + attrs.num_inputs = 1; + attrs.num_outputs = 2; + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.requests.insert(OpReqType::kAddTo); + return attrs; +} + +OpAttrs GetConcatOp(int num_args, int dim) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("concat"); + attrs.num_inputs = num_args; + attrs.num_outputs = 1; + attrs.attrs.dict.insert({"num_args" , std::to_string(num_args)}); + attrs.attrs.dict.insert({"dim" , std::to_string(dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + return attrs; +} + +OpAttrs GetConcatBackwardsOp(int num_args, int dim) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_Concat"); + attrs.num_inputs = 2; + attrs.num_outputs = num_args; + attrs.attrs.dict.insert({"num_args" , std::to_string(num_args)}); + attrs.attrs.dict.insert({"dim" , std::to_string(dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.dispatches.resize(2); + attrs.dispatches[0] = DispatchMode::kFCompute; + attrs.dispatches[1] = DispatchMode::kFComputeEx; + return attrs; +} + +OpAttrs GetPoolingOp(int kernel, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("Pooling"); + attrs.num_inputs = 1; + attrs.num_outputs = dim == 2 ? 2 : 1; + attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); + attrs.attrs.dict.insert({"pool_type" , "max"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + return attrs; +} + +OpAttrs GetPoolingBackwardsOp(int kernel, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_Pooling"); + attrs.num_inputs = dim == 2 ? 5 : 3; + attrs.num_outputs = 1; + attrs.attrs.dict.insert({"kernel", CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"stride", CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad", CreateShapeString(pad, dim)}); + attrs.attrs.dict.insert({"pool_type", "max"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + return attrs; +} + +OpAttrs GetLRNOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("LRN"); + attrs.num_inputs = 1; + attrs.num_outputs = 2; + attrs.attrs.dict.insert({"nsize" , "3"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.accept_dims.insert(4); + attrs.requests.insert(OpReqType::kWriteTo); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + return attrs; +} + +OpAttrs GetLRNBackwardsOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_LRN"); + attrs.num_inputs = 3; + attrs.num_outputs = 1; + attrs.attrs.dict.insert({"nsize" , "3"}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.dispatches.resize(2); + attrs.requests.insert(OpReqType::kWriteTo); + return attrs; +} + +OpAttrs GetSoftmaxOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("softmax"); + attrs.num_inputs = 1; + attrs.num_outputs = 1; + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.accept_dims.insert({1, 2, 3, 4, 5}); + attrs.requests.insert(OpReqType::kWriteTo); + attrs.requests.insert(OpReqType::kWriteInplace); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + return attrs; +} + +OpAttrs GetFullyConnectedOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("FullyConnected"); + attrs.attrs.dict.insert({"num_hidden" , "20"}); + attrs.num_inputs = 3; + attrs.num_outputs = 1; + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.requests.insert(OpReqType::kWriteTo); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped; + return attrs; +} + +OpAttrs GetFullyConnectedBackwardsOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_FullyConnected"); + attrs.attrs.dict.insert({"num_hidden" , "20"}); + attrs.num_inputs = 3; + attrs.num_outputs = 3; + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.requests.insert(OpReqType::kWriteTo); + return attrs; +} + +OpAttrs GetConvOp(int kernel, int num_filters, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("Convolution"); + attrs.num_inputs = 3; + attrs.num_outputs = 1; + attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"num_filter" , std::to_string(num_filters)}); + attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped | + ArrayTypes::NormalReused | + ArrayTypes::MKLDNNReused | + ArrayTypes::NormalReshapedReused; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped | + ArrayTypes::NormalReused | + ArrayTypes::MKLDNNReused | + ArrayTypes::NormalReshapedReused | + ArrayTypes::NormalReusedDiffDtype; + return attrs; +} + +OpAttrs GetConvBackwardOp(int kernel, int num_filters, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_Convolution"); + attrs.num_inputs = 4; + attrs.num_outputs = 3; + attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"num_filter" , std::to_string(num_filters)}); + attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + return attrs; +} + +OpAttrs GetDeconvOp(int kernel, int num_filters, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("Deconvolution"); + attrs.num_inputs = 2; + attrs.num_outputs = 1; + attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"num_filter" , std::to_string(num_filters)}); + attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped | + ArrayTypes::NormalReused | + ArrayTypes::MKLDNNReused | + ArrayTypes::NormalReshapedReused; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN | + ArrayTypes::NormalReshaped | + ArrayTypes::MKLDNNReshaped | + ArrayTypes::NormalReused | + ArrayTypes::MKLDNNReused | + ArrayTypes::NormalReshapedReused | + ArrayTypes::NormalReusedDiffDtype; + return attrs; +} + +OpAttrs GetDeconvBackwardOp(int kernel, int num_filters, int dim, int stride, int pad) { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_Deconvolution"); + attrs.num_inputs = 3; + attrs.num_outputs = 2; + attrs.attrs.dict.insert({"kernel" , CreateShapeString(kernel, dim)}); + attrs.attrs.dict.insert({"num_filter" , std::to_string(num_filters)}); + attrs.attrs.dict.insert({"stride" , CreateShapeString(stride, dim)}); + attrs.attrs.dict.insert({"pad" , CreateShapeString(pad, dim)}); + attrs.attrs.op->attr_parser(&attrs.attrs); + return attrs; +} + +OpAttrs GetBNOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("BatchNorm"); + attrs.num_inputs = 5; + attrs.num_outputs = 3; + attrs.accept_dims.insert(4); + attrs.requests.insert(OpReqType::kWriteTo); + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.input_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN; + attrs.output_types = ArrayTypes::Normal | + ArrayTypes::MKLDNN; + return attrs; +} + +OpAttrs GetBNBackwardOp() { + OpAttrs attrs; + attrs.attrs.op = Op::Get("_backward_BatchNorm"); + attrs.num_inputs = 8; + attrs.num_outputs = 3; + attrs.attrs.op->attr_parser(&attrs.attrs); + attrs.requests.insert(OpReqType::kWriteTo); + return attrs; +} + +void AssertEqual(const std::vector &in_arrs, + const std::vector &out_arrs, + float rtol = 1e-5, float atol = 1e-8) { + NDArray tmp1 = in_arrs[0]->Reorder2Default(); + NDArray tmp2 = out_arrs[0]->Reorder2Default(); + EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); + TBlob blob1 = tmp1.data(); + TBlob blob2 = tmp2.data(); + mshadow::default_real_t *d1 = static_cast(blob1.dptr_); + mshadow::default_real_t *d2 = static_cast(blob2.dptr_); + for (int i = 0; i < tmp1.shape().Size(); i++) { + float abs_err = fabs((d1[i]) - (d2[i])); + ASSERT_LE(abs_err, (atol + rtol * fabs(d2[i]))); + } +} + +void VerifyActResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + NDArray tmp1 = in_arrs[0]->Reorder2Default(); + NDArray tmp2 = out_arrs[0]->Reorder2Default(); + TBlob blob1 = tmp1.data(); + TBlob blob2 = tmp2.data(); + mshadow::default_real_t *d1 = static_cast(blob1.dptr_); + mshadow::default_real_t *d2 = static_cast(blob2.dptr_); + EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); + for (size_t i = 0; i < tmp1.shape().Size(); i++) { + EXPECT_EQ(std::fmax(d1[i], 0), d2[i]); + } +} + +void VerifyActBackwardsResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + NDArray tmp1 = in_arrs[0]->Reorder2Default(); // out grads + NDArray tmp2 = in_arrs[1]->Reorder2Default(); // input + NDArray tmp3 = out_arrs[0]->Reorder2Default(); // input grads + TBlob blob1 = tmp1.data(); + TBlob blob2 = tmp2.data(); + TBlob blob3 = tmp3.data(); + mshadow::default_real_t *d1 = static_cast(blob1.dptr_); + mshadow::default_real_t *d2 = static_cast(blob2.dptr_); + mshadow::default_real_t *d3 = static_cast(blob3.dptr_); + EXPECT_EQ(tmp1.shape().Size(), tmp2.shape().Size()); + for (size_t i = 0; i < tmp1.shape().Size(); i++) { + ASSERT_EQ(d2[i] > 0 ? d1[i] : 0, d3[i]); + } +} + +void VerifySumBackwardsResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + NDArray out_grads = in_arrs[0]->Reorder2Default(); // out grads + NDArray input_grads1 = out_arrs[0]->Reorder2Default(); // input grads + NDArray input_grads2 = out_arrs[1]->Reorder2Default(); // input grads + mshadow::default_real_t *og = out_grads.data().dptr(); + mshadow::default_real_t *ig1 = input_grads1.data().dptr(); + mshadow::default_real_t *ig2 = input_grads2.data().dptr(); + for (size_t i = 0; i < out_grads.shape().Size(); i++) { + ASSERT_EQ(og[i], ig1[i]); + ASSERT_EQ(og[i], ig2[i]); + } +} + +void VerifyConcatResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + int num_inputs = in_arrs.size(); + int input_size = in_arrs[0]->shape().Size(); + TShape input_shape = in_arrs[0]->shape(); + NDArray output = out_arrs[0]->Reorder2Default(); + size_t total_size = output.shape().Size(); + EXPECT_EQ(input_size * num_inputs, total_size); + mshadow::default_real_t *out_data = output.data().dptr(); + + int dim = GetDim(input_shape, output.shape()); + int block_size = GetBlockSize(input_shape, dim); + int num_blocks = input_size / block_size; + for (size_t input_num = 0; input_num < num_inputs; input_num++) { + NDArray tmp = in_arrs[input_num]->Reorder2Default(); + mshadow::default_real_t* data = tmp.data().dptr(); + for (size_t block_num = 0; block_num < num_blocks; block_num++) { + for (size_t i = 0; i < block_size; i++) + ASSERT_EQ(data[block_num * block_size + i], + out_data[(block_num * num_inputs + input_num) * block_size + i]); + } + } +} + +void VerifyConcatBackwardsResult(const std::vector &in_arrs, + const std::vector &out_arrs) { + // in_arrs is larger array, out_arr is ammler + int num_inputs = out_arrs.size(); + int input_size = out_arrs[0]->shape().Size(); + TShape input_shape = out_arrs[0]->shape(); + NDArray output = in_arrs[0]->Reorder2Default(); + size_t total_size = output.shape().Size(); + EXPECT_EQ(input_size * num_inputs, total_size); + mshadow::default_real_t *out_data = output.data().dptr(); + + int dim = GetDim(input_shape, output.shape()); + int block_size = GetBlockSize(input_shape, dim); + int num_blocks = input_size / block_size; + for (size_t input_num = 0; input_num < num_inputs; input_num++) { + NDArray tmp = out_arrs[input_num]->Reorder2Default(); + mshadow::default_real_t* data = tmp.data().dptr(); + for (size_t block_num = 0; block_num < num_blocks; block_num++) { + for (size_t i = 0; i < block_size; i++) + ASSERT_EQ(data[block_num * block_size + i], + out_data[(block_num * num_inputs + input_num) * block_size + i]); + } + } +} + +void TestOp(const OpAttrs &attrs, VerifyFunc verify_fn) { + std::vector inputs(attrs.num_inputs); + std::vector outputs(attrs.num_outputs); + std::vector req(attrs.num_outputs); + std::vector in_arrs; + std::vector> out_arrs(attrs.num_outputs); + std::vector dispatches = attrs.dispatches; + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + if (attrs.requests.find(OpReqType::kWriteTo) != attrs.requests.end()) { + std::vector in_arrs = GetTestInputArrays(); + for (auto &in_arr : in_arrs) { + for (auto &dispatch : dispatches) { + std::vector> out_arrs(attrs.num_outputs); + for (int i = 0; i < attrs.num_outputs; i++) + out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (int i = 0; i < attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (int i = 0; i < attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + } + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, + outputs, req, dispatch, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + verify_fn(inputs, outputs); + } + } + } + } + + if (attrs.requests.find(OpReqType::kWriteInplace) != attrs.requests.end()) { + for (auto &dispatch : dispatches) { + in_arrs = GetTestInputArrays(); + for (auto &arr : in_arrs) { + // If the array is a view, we shouldn't write data to it. + if (arr.arr.IsView()) + continue; + NDArrayAttrs orig(arr.arr.Copy(arr.arr.ctx()), "InPlace Copy"); + for (int i = 0; i < attrs.num_inputs; i++) + inputs[i] = &arr.arr; + for (int i = 0; i < attrs.num_outputs; i++) { + req[i] = kWriteInplace; + outputs[i] = &arr.arr; + } + PrintVerifyMsg(orig, arr); + Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, outputs, req, + dispatch, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + std::vector orig_inputs(attrs.num_inputs); + for (int i = 0; i < attrs.num_inputs; i++) + orig_inputs[i] = &orig.arr; + verify_fn(orig_inputs, outputs); + } + } + } + + if (attrs.requests.find(OpReqType::kAddTo) != attrs.requests.end()) { + std::vector original_outputs(attrs.num_outputs); + in_arrs = GetTestInputArrays(); + for (auto &in_arr : in_arrs) { + for (auto &dispatch : dispatches) { + for (int i = 0; i < attrs.num_outputs; i++) + out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (size_t i = 0; i < attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + NDArray tmp; + for (size_t i = 0; i < attrs.num_outputs; i++) { + auto out_arr = out_arrs[i][output_i]; + tmp = out_arr.arr.Copy(out_arr.arr.ctx()); + original_outputs[i] = &tmp; + outputs[i] = &out_arrs[i][output_i].arr; + req[i] = kAddTo; + } + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, + outputs, req, dispatch, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + VerifyAddRequest(inputs, original_outputs, outputs, verify_fn); + } + } + } + } +} + +void TestConcatOp(const OpAttrs &attrs, VerifyFunc verify_fn, + bool backwards = false) { + std::vector inputs(attrs.num_inputs); + std::vector outputs(attrs.num_outputs); + std::vector req(attrs.num_outputs); + std::vector dispatches = attrs.dispatches; + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + std::vector in_arrs = GetTestInputArrays(); + + // concat backwards uses scaled up inputs + if (backwards) { + std::string str_dim = const_cast(attrs).attrs.dict["dim"]; + int dim = std::stoi(str_dim); + std::vector scale_vector(dim+1); + for (size_t i = 0; i < dim+1; ++i) + scale_vector[i] = 1; + scale_vector[dim] = attrs.num_outputs; + in_arrs = GetTestInputArrays(ArrayTypes::All, false, scale_vector); + } + + for (auto &in_arr : in_arrs) { + for (auto &dispatch : dispatches) { + std::vector> out_arrs(attrs.num_outputs); + + std::string str_dim = const_cast(attrs).attrs.dict["dim"]; + int dim = std::stoi(str_dim); + if (dim >= in_arr.arr.shape().ndim()) + continue; + float scale = backwards ? 1 / static_cast(attrs.num_outputs) : + static_cast(attrs.num_inputs); + + std::vector scale_vector(in_arr.arr.shape().ndim()); + for (int i = 0; i < in_arr.arr.shape().ndim(); i++) + scale_vector[i] = 1; + scale_vector[dim] = scale; + for (int i = 0; i < attrs.num_outputs; i++) + out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); + + for (int i = 0; i < attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (int i = 0; i < attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + } + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp(Context(), attrs.attrs, inputs, + outputs, req, dispatch, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + verify_fn(inputs, outputs); + } + } + } +} + +void TestOpExBackward(const OpAttrs &forward_attrs, + const OpAttrs &backwards_attrs, + const OpReqType &req, + const std::vector &inputs, + const std::vector &outputs, + const NDArrayAttrs &in_arr, + const NDArrayAttrs &out_arr) { + std::vector backwards_input(backwards_attrs.num_inputs); + std::vector backwards_outputs(backwards_attrs.num_outputs); + std::vector backwards_ex_outputs(backwards_attrs.num_outputs); + std::vector back_req(backwards_attrs.num_outputs); + + if (req == kWriteTo) { + // backwards test performed same time since output needed + backwards_input[0] = outputs[0]; // output grad + backwards_input[1] = inputs[0]; // input + backwards_input[2] = outputs[1]; // out norm + + auto tmp_output = in_arr.arr; + backwards_outputs[0] = &tmp_output; + backwards_ex_outputs[0] = &tmp_output; + + for (int i = 0; i < backwards_attrs.num_outputs; i++) + back_req[i] = kWriteTo; + + std::cout << "Backwards: "; + PrintVerifyMsg(out_arr, in_arr); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, + back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, + back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(backwards_outputs, backwards_ex_outputs); + } +} + + +// compares output of fcompute with fcomputex +void TestOpEx(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { + std::vector inputs(forward_attrs.num_inputs); + std::vector outputs(forward_attrs.num_outputs); + std::vector ex_outputs(forward_attrs.num_outputs); + std::vector req(forward_attrs.num_outputs); + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + std::vector in_arrs = GetTestInputArrays(forward_attrs.input_types, true); + std::vector> out_arrs(forward_attrs.num_outputs); + std::vector> ex_out_arrs(forward_attrs.num_outputs); + + if (forward_attrs.requests.find(OpReqType::kWriteTo) != forward_attrs.requests.end()) { + for (int i1 = 0; i1 < in_arrs.size(); i1++) { + auto in_arr = in_arrs[i1]; + + CHECK_NE(forward_attrs.accept_dims.size(), 0); + if (forward_attrs.accept_dims.find(in_arr.arr.shape().ndim()) == + forward_attrs.accept_dims.end()) + continue; + + for (int i = 0; i < forward_attrs.num_outputs; i++) { + out_arrs[i] = + GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, forward_attrs.output_types); + ex_out_arrs[i] = + GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, forward_attrs.output_types); + } + + for (int i = 0; i < forward_attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (int i = 0; i < forward_attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + ex_outputs[i] = &ex_out_arrs[i][output_i].arr; + } + Imperative::Get()->set_is_training(true); + + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, outputs, req, + DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, ex_outputs, req, + DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(outputs, ex_outputs); + + if (!backwards_attrs.requests.empty()) { + TestOpExBackward(forward_attrs, backwards_attrs, OpReqType::kWriteTo, + inputs, outputs, in_arr, out_arrs[0][output_i]); + } + } + } + } + + if (forward_attrs.requests.find(OpReqType::kWriteInplace) != forward_attrs.requests.end()) { + for (int i1 = 0; i1 < in_arrs.size(); i1++) { + auto in_arr = in_arrs[i1]; + + // If the array is a view, we shouldn't write data to it. + if (in_arr.arr.IsView()) + continue; + + NDArrayAttrs orig(in_arr.arr.Copy(in_arr.arr.ctx()), "InPlace Copy"); + for (int i = 0; i < forward_attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + + for (int i = 0; i < forward_attrs.num_outputs; i++) { + req[i] = kWriteInplace; + outputs[i] = &in_arr.arr; + ex_outputs[i] = &in_arr.arr; + } + Imperative::Get()->set_is_training(true); + PrintVerifyMsg(orig, in_arr); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, outputs, req, + DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, ex_outputs, req, + DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(outputs, ex_outputs); + } + } +} + + +void TestOpExBNBackward(const OpAttrs &forward_attrs, + const OpAttrs &backwards_attrs, + const OpReqType &req, + const std::vector &inputs, + const std::vector &outputs, + const NDArrayAttrs &in_arr, + NDArrayAttrs* out_arr) { + std::vector backwards_input(backwards_attrs.num_inputs); + + std::vector backwards_buffer(backwards_attrs.num_outputs); + std::vector backwards_buffer2(backwards_attrs.num_outputs); + + std::vector backwards_outputs(backwards_attrs.num_outputs); + std::vector backwards_ex_outputs(backwards_attrs.num_outputs); + std::vector backwards_req(backwards_attrs.num_outputs); + + if (req == kWriteTo) { + backwards_input[0] = &(out_arr->arr); // output grad + backwards_input[1] = outputs[1]; // mean + backwards_input[2] = outputs[2]; // var + backwards_input[3] = inputs[0]; // data + backwards_input[4] = inputs[1]; // gamma + backwards_input[5] = inputs[2]; // beta + backwards_input[6] = inputs[3]; // moving mean + backwards_input[7] = inputs[4]; // moving var + + for (size_t i = 0; i < backwards_attrs.num_outputs; i++) { + auto tmp_output = in_arr.arr; + backwards_buffer.emplace_back(tmp_output.Copy(Context())); + backwards_buffer2.emplace_back(tmp_output.Copy(Context())); + backwards_outputs[i] = &backwards_buffer.back(); + backwards_ex_outputs[i] = &backwards_buffer2.back(); + Engine::Get()->WaitForAll(); + backwards_req[i] = kWriteTo; + } + + std::cout << "Backwards: "; + PrintVerifyMsg(*out_arr, in_arr); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, + backwards_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, + backwards_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(backwards_outputs, backwards_ex_outputs); + } +} + +// compares output of fcompute with fcomputex +void TestOpExBN(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { + std::vector inputs(forward_attrs.num_inputs); + std::vector inputs2(forward_attrs.num_inputs); + std::vector inputs_buffer(forward_attrs.num_inputs); + std::vector inputs2_buffer(forward_attrs.num_inputs); + std::vector outputs(forward_attrs.num_outputs); + std::vector ex_outputs(forward_attrs.num_outputs); + std::vector req(forward_attrs.num_outputs); + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + std::vector in_arrs = GetTestInputArrays(forward_attrs.input_types, false); + std::vector> out_arrs(forward_attrs.num_outputs); + std::vector> ex_out_arrs(forward_attrs.num_outputs); + + if (forward_attrs.requests.find(OpReqType::kWriteTo) != forward_attrs.requests.end()) { + for (int i1 = 0; i1 < in_arrs.size(); i1++) { + auto in_arr = in_arrs[i1]; + + CHECK_NE(forward_attrs.accept_dims.size(), 0); + if (forward_attrs.accept_dims.find(in_arr.arr.shape().ndim()) == + forward_attrs.accept_dims.end()) + continue; + for (int i = 0; i < forward_attrs.num_outputs; i++) { + out_arrs[i] = + GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, true, forward_attrs.output_types); + ex_out_arrs[i] = + GetTestOutputArrays(in_arr.arr.shape(), pds, {1}, true, forward_attrs.output_types); + } + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + inputs_buffer.clear(); + inputs2_buffer.clear(); + + for (int i = 0; i < forward_attrs.num_inputs; i++) { + inputs_buffer.emplace_back(in_arr.arr.Copy(Context())); + inputs2_buffer.emplace_back(in_arr.arr.Copy(Context())); + Engine::Get()->WaitForAll(); + inputs[i] = &inputs_buffer.back(); + inputs2[i] = &inputs2_buffer.back(); + } + for (int i = 0; i < forward_attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + ex_outputs[i] = &ex_out_arrs[i][output_i].arr; + } + Imperative::Get()->set_is_training(true); + + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, outputs, req, + DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs2, ex_outputs, req, + DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(outputs, ex_outputs); + + if (!backwards_attrs.requests.empty()) { + TestOpExBNBackward(forward_attrs, backwards_attrs, OpReqType::kWriteTo, + inputs, outputs, in_arr, &out_arrs[0][output_i]); + } + } + } + } +} + +// Computes second dimension of FC weight matrix based on input shape +uint32_t GetFCWeightDim2(const nnvm::TShape arr) { + uint32_t dim = 1; + for (int i = 1; i < arr.ndim(); i++) { + dim *= arr[i]; + } + return dim; +} + +void TestFullyConnectedOp(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { + std::vector inputs(forward_attrs.num_inputs); + std::vector outputs(forward_attrs.num_outputs); + std::vector ex_outputs(forward_attrs.num_outputs); + + std::vector backwards_input(backwards_attrs.num_inputs); + std::vector backwards_outputs(backwards_attrs.num_outputs); + std::vector backwards_ex_outputs(backwards_attrs.num_outputs); + + std::vector req(forward_attrs.num_outputs); + std::vector back_req(backwards_attrs.num_outputs); + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + std::vector in_arrs = GetTestInputArrays(forward_attrs.input_types, true); + std::vector> out_arrs(forward_attrs.num_outputs); + std::vector> ex_out_arrs(forward_attrs.num_outputs); + + std::string str_hid = const_cast(forward_attrs).attrs.dict["num_hidden"]; + int num_hid = std::stoi(str_hid); + + if (forward_attrs.requests.find(OpReqType::kWriteTo) != forward_attrs.requests.end()) { + for (int i1 = 0; i1 < in_arrs.size(); i1++) { + auto in_arr = in_arrs[i1]; + auto in_shape = in_arr.arr.shape(); + if (in_shape.ndim() < 2) + continue; + + nnvm::TShape wt_shape(2); + wt_shape[0] = num_hid; + wt_shape[1] = GetFCWeightDim2(in_shape); + NDArray weights(wt_shape, Context()); + InitDefaultArray(&weights, false); + + nnvm::TShape bias_shape(1); + bias_shape[0] = num_hid; + NDArray bias(bias_shape, Context()); + InitDefaultArray(&bias, false); + + inputs[0] = &in_arr.arr; + inputs[1] = &weights; + inputs[2] = &bias; + + nnvm::TShape out_shape(2); + out_shape[0] = in_shape[0]; + out_shape[1] = num_hid; + + for (int i = 0; i < forward_attrs.num_outputs; i++) { + out_arrs[i] = + GetTestOutputArrays(out_shape, pds, {1}, forward_attrs.output_types); + ex_out_arrs[i] = + GetTestOutputArrays(out_shape, pds, {1}, forward_attrs.output_types); + } + + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (int i = 0; i < forward_attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + ex_outputs[i] = &ex_out_arrs[i][output_i].arr; + } + Imperative::Get()->set_is_training(true); + + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, outputs, req, + DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), forward_attrs.attrs, inputs, ex_outputs, req, + DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(outputs, ex_outputs); + + // backwards test performed same time since output needed + backwards_input[0] = outputs[0]; // output grad + backwards_input[1] = inputs[0]; // input + backwards_input[2] = inputs[1]; // weights + + auto tmp_output = GetTestInputArrays(forward_attrs.input_types, true)[i1]; + NDArray back_weights(wt_shape, Context()); + NDArray back_bias(bias_shape, Context()); + backwards_outputs[0] = &tmp_output.arr; + backwards_outputs[1] = &back_weights; + backwards_outputs[2] = &back_bias; + + auto tmp_output2 = GetTestInputArrays(forward_attrs.input_types, true)[i1]; + NDArray back_ex_weights(wt_shape, Context()); + NDArray back_ex_bias(bias_shape, Context()); + backwards_ex_outputs[0] = &tmp_output2.arr; + backwards_ex_outputs[1] = &back_ex_weights; + backwards_ex_outputs[2] = &back_ex_bias; + + for (int i = 0; i < backwards_attrs.num_outputs; i++) + back_req[i] = kWriteTo; + + std::cout << "Backwards: "; + PrintVerifyMsg(out_arrs[0][output_i], tmp_output); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, + back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, + back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + AssertEqual(backwards_outputs, backwards_ex_outputs); + } + } + } +} + +template +void TestConvOp(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs, + bool is_deconv = false) { + std::vector inputs(forward_attrs.num_inputs); + std::vector outputs(forward_attrs.num_outputs); + std::vector ex_outputs(forward_attrs.num_outputs); + + std::vector backwards_input(backwards_attrs.num_inputs); + std::vector backwards_outputs(backwards_attrs.num_outputs); + std::vector backwards_ex_outputs(backwards_attrs.num_outputs); + + + std::vector req(forward_attrs.num_outputs); + std::vector back_req(backwards_attrs.num_outputs); + std::vector dispatches = forward_attrs.dispatches; + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + P param; + param.Init(forward_attrs.attrs.dict); + TShape kernel = param.kernel; + TShape padding = param.pad; + TShape stride = param.stride; + int num_filter = param.num_filter; + + std::vector in_arrs = GetTestInputArrays( + forward_attrs.input_types, true, {1}, true); + std::vector> out_arrs(forward_attrs.num_outputs); + std::vector> ex_out_arrs(forward_attrs.num_outputs); + + for (size_t i1 = 0; i1 < in_arrs.size(); ++i1) { + auto in_arr = in_arrs[i1]; + + // can only conv only 4D inputs + TShape input_shape = in_arr.arr.shape(); + if (input_shape.ndim() != kernel.ndim() + 2) + continue; + + float scale = CalculateWidthConvOutput(input_shape[2], kernel[0], padding[0], stride[0]) + / static_cast(input_shape[2]); + + if (is_deconv) { + scale = CalculateWidthDeconvOutput(input_shape[2], kernel[0], padding[0], stride[0]) + / static_cast(input_shape[2]); + } + std::vector scale_vector(in_arr.arr.shape().ndim()); + scale_vector[0] = 1; + scale_vector[1] = static_cast(num_filter) / input_shape[1]; + scale_vector[2] = scale; + scale_vector[3] = scale; + + for (size_t i = 0; i < forward_attrs.num_outputs; ++i) { + out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, + scale_vector, true, forward_attrs.output_types); + ex_out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, + scale_vector, true, forward_attrs.output_types); + } + NDArray ndkernel = CreateKernelNDArray(kernel, num_filter, in_arr.arr.shape(), is_deconv); + TShape bias_shape = {num_filter}; + NDArray ndbias = CreateBiasNDArray(bias_shape); + inputs[0] = &in_arr.arr; + inputs[1] = &ndkernel; + + if (!param.no_bias) { + inputs[2] = &ndbias; + } + + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (size_t i = 0; i < forward_attrs.num_outputs; ++i) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + ex_outputs[i] = &ex_out_arrs[i][output_i].arr; + } + Imperative::Get()->set_is_training(true); + + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, + outputs, req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, + ex_outputs, req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + VerifyCopyResult(outputs, ex_outputs); + + // backwards test performed same time since output needed + backwards_input[0] = outputs[0]; // output grad + backwards_input[1] = inputs[0]; // input + backwards_input[2] = inputs[1]; // kernel + + if (!param.no_bias) { + backwards_input[3] = inputs[2]; // bias + } + + auto tmp_output = GetTestInputArrays(forward_attrs.input_types, true, {1}, true)[i1]; + NDArray tmp_kernel = CreateKernelNDArray(kernel, num_filter, in_arr.arr.shape(), is_deconv); + NDArray tmp_bias = CreateBiasNDArray(bias_shape); + backwards_outputs[0] = &tmp_output.arr; + backwards_outputs[1] = &tmp_kernel; + if (!param.no_bias) { + backwards_outputs[2] = &tmp_bias; + } + + auto tmp_output2 = GetTestInputArrays(forward_attrs.input_types, true, {1}, true)[i1]; + NDArray tmp_kernel2 = CreateKernelNDArray(kernel, num_filter, in_arr.arr.shape(), is_deconv); + NDArray tmp_bias2 = CreateBiasNDArray(bias_shape); + backwards_ex_outputs[0] = &tmp_output2.arr; + backwards_ex_outputs[1] = &tmp_kernel2; + if (!param.no_bias) { + backwards_ex_outputs[2] = &tmp_bias2; + } + + for (size_t i = 0; i < backwards_attrs.num_outputs; ++i) + back_req[i] = kWriteTo; + + std::cout << "Backwards: "; + PrintVerifyMsg(out_arrs[0][output_i], tmp_output); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, + back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, + back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + VerifyCopyResult(backwards_outputs, backwards_ex_outputs); + } + } +} + +void TestPoolingOp(const OpAttrs &forward_attrs, const OpAttrs &backwards_attrs) { + std::vector inputs(forward_attrs.num_inputs); + std::vector outputs(forward_attrs.num_outputs); + std::vector ex_outputs(forward_attrs.num_outputs); + + std::vector backwards_input(backwards_attrs.num_inputs); + std::vector backwards_outputs(backwards_attrs.num_outputs); + std::vector backwards_ex_outputs(backwards_attrs.num_outputs); + + + std::vector req(forward_attrs.num_outputs); + std::vector back_req(backwards_attrs.num_outputs); + std::vector dispatches = forward_attrs.dispatches; + + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + mxnet::op::PoolingParam param; + param.Init(forward_attrs.attrs.dict); + TShape kernel = param.kernel; + TShape padding = param.pad; + TShape stride = param.stride; + + std::vector in_arrs = GetTestInputArrays(); + std::vector> out_arrs(forward_attrs.num_outputs); + std::vector> ex_out_arrs(forward_attrs.num_outputs); + + for (int i1 = 0; i1 < in_arrs.size(); i1++) { + auto in_arr = in_arrs[i1]; + + // can only pool only 3D and 4D inputs + TShape input_shape = in_arr.arr.shape(); + if (input_shape.ndim() != kernel.ndim() + 2) + continue; + // cannot pool if ndarray and mkldnn memory have different ndim + if (in_arr.arr.IsView() || in_arr.arr.GetMKLDNNData()->get_primitive_desc().desc().data.ndims + != in_arr.arr.shape().ndim()) + continue; + std::vector scale_vector(in_arr.arr.shape().ndim()); + for (int i = 0; i < in_arr.arr.shape().ndim(); i++) { + if (i < 2) + scale_vector[i] = 1; + else + scale_vector[i] = CalculateWidthPoolOutput( + input_shape[i], kernel[i-2], padding[i-2], stride[i-2]) / + static_cast(input_shape[i]); + } + for (int i = 0; i < forward_attrs.num_outputs; i++) { + out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); + ex_out_arrs[i] = GetTestOutputArrays(in_arr.arr.shape(), pds, scale_vector); + } + + for (int i = 0; i < forward_attrs.num_inputs; i++) + inputs[i] = &in_arr.arr; + + for (size_t output_i = 0; output_i < out_arrs[0].size(); output_i++) { + for (int i = 0; i < forward_attrs.num_outputs; i++) { + req[i] = kWriteTo; + outputs[i] = &out_arrs[i][output_i].arr; + ex_outputs[i] = &ex_out_arrs[i][output_i].arr; + } + Imperative::Get()->set_is_training(true); + + PrintVerifyMsg(in_arr, out_arrs[0][output_i]); + Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, + outputs, req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp(Context(), forward_attrs.attrs, inputs, + ex_outputs, req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + VerifyCopyResult(outputs, ex_outputs); + + + // backwards test performed same time since output needed + if (backwards_attrs.num_inputs == 3) { + backwards_input[0] = outputs[0]; // output grad + backwards_input[1] = inputs[0]; // input + backwards_input[2] = outputs[0]; // output + } else if (backwards_attrs.num_inputs == 5) { + backwards_input[0] = outputs[0]; // output grad + backwards_input[1] = outputs[0]; // workspace grad + backwards_input[2] = inputs[0]; // input + backwards_input[3] = outputs[0]; // output + backwards_input[4] = ex_outputs[1]; // workspace + } + + // needs copies of inputs since they be reused in next iteration + // cannot use Copy method since we need to maintain MKLDNN format + auto tmp_output = GetTestInputArrays()[i1]; + auto tmp_output2 = GetTestInputArrays()[i1]; + backwards_outputs[0] = &tmp_output.arr; + backwards_ex_outputs[0] = &tmp_output2.arr; + back_req[0] = kWriteTo; + std::cout << "Backwards: "; + PrintVerifyMsg(out_arrs[0][output_i], tmp_output); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_outputs, + back_req, DispatchMode::kFCompute, mxnet::OpStatePtr()); + Imperative::Get()->InvokeOp( + Context(), backwards_attrs.attrs, backwards_input, backwards_ex_outputs, + back_req, DispatchMode::kFComputeEx, mxnet::OpStatePtr()); + Engine::Get()->WaitForAll(); + VerifyCopyResult(backwards_outputs, backwards_ex_outputs); + } + } +} + +TEST(IMPERATIVE, CopyOp) { + OpAttrs attrs = GetCopyOp(); + TestOp(attrs, VerifyCopyResult); +} + +TEST(IMPERATIVE, CopyBackwardsOp) { + OpAttrs attrs = GetCopyBackwardsOp(); + TestOp(attrs, VerifyCopyResult); +} + +TEST(IMPERATIVE, ActOp) { + OpAttrs attrs = GetReluOp(); + TestOp(attrs, VerifyActResult); +} + +TEST(IMPERATIVE, ActBackwardsOp) { + OpAttrs attrs = GetReluBackwardsOp(); + TestOp(attrs, VerifyActBackwardsResult); +} + +TEST(IMPERATIVE, SumOp) { + OpAttrs attrs = GetSumOp(); + TestOp(attrs, VerifySumResult); +} + +TEST(IMPERATIVE, SumBackwardsOp) { + OpAttrs attrs = GetSumBackwardsOp(); + TestOp(attrs, VerifySumBackwardsResult); +} + +TEST(IMPERATIVE, ConcatOp) { + for (int num_inputs = 2; num_inputs < 4; num_inputs++) { + for (int dim = 0; dim < 5; dim++) { + OpAttrs attrs = GetConcatOp(num_inputs, dim); + TestConcatOp(attrs, VerifyConcatResult); + } + } +} + +TEST(IMPERATIVE, ConcatBackwardsOp) { + for (int num_inputs = 2; num_inputs < 4; num_inputs++) { + for (int dim = 0; dim < 5; dim++) { + OpAttrs attrs = GetConcatBackwardsOp(num_inputs, dim); + TestConcatOp(attrs, VerifyConcatBackwardsResult, true); + } + } +} + +TEST(IMPERATIVE, LRNOp) { + OpAttrs forward_attrs = GetLRNOp(); + OpAttrs backwards_attrs = GetLRNBackwardsOp(); + TestOpEx(forward_attrs, backwards_attrs); +} + +TEST(IMPERATIVE, SoftmaxOp) { + OpAttrs forward_attrs = GetSoftmaxOp(); + OpAttrs backwards_attrs; + TestOpEx(forward_attrs, backwards_attrs); +} + +TEST(IMPERATIVE, FullyConnectedOp) { + OpAttrs forward_attrs = GetFullyConnectedOp(); + OpAttrs backwards_attrs = GetFullyConnectedBackwardsOp(); + TestFullyConnectedOp(forward_attrs, backwards_attrs); +} + +TEST(IMPERATIVE, PoolingOp) { + for (int dim = 2; dim < 4; dim++) { + for (int kernel = 1; kernel < 4; kernel++) { + for (int stride = 1; stride < 3; stride++) { + for (int pad = 0; pad < 2; pad++) { + if (kernel / 2. < pad) + continue; + OpAttrs forward_attrs = GetPoolingOp(kernel, dim, stride, pad); + OpAttrs backwards_attrs = GetPoolingBackwardsOp(kernel, dim, stride, pad); + TestPoolingOp(forward_attrs, backwards_attrs); + } + } + } + } +} + +TEST(IMPERATIVE, ConvOp) { + int dim = 2; // MKLDNN conv only supports 2d kernels + for (size_t num_filters = 2; num_filters < 3; ++num_filters) { + for (size_t kernel = 1; kernel < 4; ++kernel) { + for (size_t stride = 1; stride < 3; ++stride) { + for (size_t pad = 0; pad < 2; ++pad) { + if (kernel / 2. < pad) + continue; + OpAttrs forward_attrs = GetConvOp(kernel, num_filters, dim, stride, pad); + OpAttrs backwards_attrs = GetConvBackwardOp(kernel, num_filters, dim, stride, pad); + TestConvOp(forward_attrs, backwards_attrs); + } + } + } + } +} + +TEST(IMPERATIVE, DeconvOp) { + int dim = 2; // MKLDNN deconv only supports 2d kernels + for (size_t num_filters = 2; num_filters < 3; ++num_filters) { + for (size_t kernel = 1; kernel < 3; ++kernel) { + for (size_t stride = 1; stride < 3; ++stride) { + for (size_t pad = 0; pad < 2; ++pad) { + if (kernel / 2. < pad) + continue; + OpAttrs forward_attrs = GetDeconvOp(kernel, num_filters, dim, stride, pad); + OpAttrs backwards_attrs = GetDeconvBackwardOp(kernel, num_filters, dim, stride, pad); + TestConvOp(forward_attrs, backwards_attrs, true); + } + } + } + } +} + +TEST(IMPERATIVE, BNOp) { + OpAttrs forward_attrs = GetBNOp(); + OpAttrs backwards_attrs = GetBNBackwardOp(); + TestOpExBN(forward_attrs, backwards_attrs); +} + +#endif diff --git a/tests/cpp/operator/mkldnn_test.cc b/tests/cpp/operator/mkldnn_test.cc new file mode 100644 index 000000000..31e762f21 --- /dev/null +++ b/tests/cpp/operator/mkldnn_test.cc @@ -0,0 +1,420 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/*! + * \file mkldnn_test.cc + * \brief test functions in mkldnn. + * \author Da Zheng + */ + +#if MXNET_USE_MKLDNN == 1 + +#include +#include +#include +#include +#include "gtest/gtest.h" +#include "mxnet/imperative.h" +#include "../../src/operator/nn/mkldnn/mkldnn_ops-inl.h" +#include "../../src/operator/nn/mkldnn/mkldnn_base-inl.h" +#include "../include/test_mkldnn.h" + +using namespace mxnet; + +#if __GNUC__ >= 5 +bool test_mem_align(void *mem, size_t size, size_t alignment, size_t space) { + void *ret1, *ret2; + size_t space1, space2; + space1 = space; + space2 = space; + ret1 = mxnet::AlignMem(mem, size, alignment, &space1); + ret2 = std::align(alignment, size, mem, space2); + EXPECT_EQ(ret1, ret2); + EXPECT_EQ(space1, space2); + return ret1 == ret2; +} +#endif + +TEST(MKLDNN_UTIL_FUNC, AlignMem) { +#if __GNUC__ >= 5 + size_t alignment = 4096; + void *mem; + size_t size, space; + // When mem has been aligned. + mem = reinterpret_cast(0x10000); + size = 1000; + space = 10000; + test_mem_align(mem, size, alignment, space); + + // When mem isn't aligned and we have enough space for alignment. + mem = reinterpret_cast(0x10010); + size = 1000; + space = 10000; + test_mem_align(mem, size, alignment, space); + + // When mem isn't aligned and we don't have enough memory for alignment + mem = reinterpret_cast(0x10010); + size = 1000; + space = 1001; + test_mem_align(mem, size, alignment, space); + + for (size_t i = 0; i < 10000; i++) { + mem = reinterpret_cast(random()); + size = random() % 2000; + space = random() % 2000; + test_mem_align(mem, size, alignment, space); + } +#else + // std::align is not supported in GCC < 5.0, this test case will be checked + // with newer version + LOG(INFO) << "Skipped for GCC " << __GNUC__ << "." << __GNUC_MINOR__; +#endif +} + +static void VerifyDefMem(const mkldnn::memory &mem) { + mkldnn::memory::primitive_desc pd = mem.get_primitive_desc(); + mshadow::default_real_t *data + = static_cast(mem.get_data_handle()); + size_t size = pd.get_size() / sizeof(mshadow::default_real_t); + size_t num_same = 0; + for (int i = 0; i < size; i++) + num_same += data[i] == static_cast(i % 100 - 50); + EXPECT_EQ(num_same, size); +} + +TEST(MKLDNN_UTIL_FUNC, MemFormat) { + // Check whether the number of format is correct. + CHECK_EQ(mkldnn_format_last, 112); + CHECK_EQ(mkldnn_nchw, 7); + CHECK_EQ(mkldnn_oihw, 16); +} + +static void VerifyMem(const mkldnn::memory &mem) { + mkldnn::memory::primitive_desc pd = mem.get_primitive_desc(); + + if (pd.desc().data.format == GetDefaultFormat(pd.desc())) { + VerifyDefMem(mem); + } else { + mkldnn::memory::dims dims(pd.desc().data.ndims); + for (size_t i = 0; i < dims.size(); i++) + dims[i] = pd.desc().data.dims[i]; + mkldnn::memory::desc desc{dims, + static_cast(pd.desc().data.data_type), + static_cast(GetDefaultFormat(pd.desc()))}; + mkldnn::memory::primitive_desc new_pd(desc, CpuEngine::Get()->get_engine()); + mkldnn::memory new_mem(new_pd); + + std::vector net; + net.push_back(mkldnn::reorder(mem, new_mem)); + mkldnn::stream(mkldnn::stream::kind::eager).submit(net).wait(); + VerifyDefMem(new_mem); + } +} + +TEST(MKLDNN_NDArray, GetDataReorder) { + TestArrayShapes tas = GetTestArrayShapes(); + std::vector shapes = tas.shapes; + std::vector pds = tas.pds; + + + // Reorder from the default to any other layout. + for (auto s : shapes) { + NDArray arr(s, Context()); + InitDefaultArray(&arr); + for (auto pd : pds) { + if (s.Size() == pd.get_size() / sizeof(mshadow::default_real_t)) { + const mkldnn::memory *mem = arr.GetMKLDNNDataReorder(pd); + printf("reorder from ("); + for (size_t i = 0; i < s.ndim(); i++) + printf("%ld, ", s[i]); + printf(") to ("); + for (int i = 0; i < pd.desc().data.ndims; i++) + printf("%d, ", pd.desc().data.dims[i]); + printf("), format: %d\n", pd.desc().data.format); + MKLDNNStream::Get()->Submit(false); + VerifyMem(*mem); + MKLDNNStream::Get()->Cleanup(); + } + } + } + + // Reorder from a special layout to another layout. + for (auto s : shapes) { + for (auto from_pd : pds) { + if (from_pd.get_size() / sizeof(mshadow::default_real_t) == s.Size()) { + NDArray arr(s, Context()); + // There is possibility that the dimensions of an NDArray doesn't match + // with the MKLDNN memory inside. + printf("Init array ("); + for (size_t i = 0; i < s.ndim(); i++) + printf("%ld, ", s[i]); + printf(") with MKLDNN memory ("); + for (int i = 0; i < from_pd.desc().data.ndims; i++) + printf("%d, ", from_pd.desc().data.dims[i]); + printf("), format: %d\n", from_pd.desc().data.format); + InitMKLDNNArray(&arr, from_pd); + for (auto to_pd : pds) { + if (to_pd.get_size() / sizeof(mshadow::default_real_t) == s.Size()) { + const mkldnn::memory *mem = arr.GetMKLDNNDataReorder(to_pd); + printf("reorder from ("); + for (size_t i = 0; i < s.ndim(); i++) + printf("%ld, ", s[i]); + printf("), format: %d to (", + arr.GetMKLDNNData()->get_primitive_desc().desc().data.format); + for (int i = 0; i < to_pd.desc().data.ndims; i++) + printf("%d, ", to_pd.desc().data.dims[i]); + printf("), format: %d\n", to_pd.desc().data.format); + MKLDNNStream::Get()->Submit(false); + VerifyMem(*mem); + MKLDNNStream::Get()->Cleanup(); + } + } + } + } + } +} + +TEST(MKLDNN_BASE, MKLDNNSum) { + std::vector in_arrs = GetTestInputArrays(); + std::vector in_arrs2 = GetTestInputArrays(ArrayTypes::All, true); + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (auto &out_arr : out_arrs) { + auto in_mem1 = in_arr.arr.GetMKLDNNData(); + auto in_mem2 = in_arr2.arr.GetMKLDNNData(); + if (out_arr.arr.IsView()) + continue; + auto out_mem = out_arr.arr.GetMKLDNNData(); + PrintVerifyMsg(in_arr, in_arr); + op::MKLDNNSum(*in_mem1, *in_mem2, *out_mem); + MKLDNNStream::Get()->Submit(); + VerifySumResult({&in_arr.arr, &in_arr2.arr}, {&out_arr.arr}); + } + } + + // in place + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + auto input_mem = in_arr.arr.GetMKLDNNData(); + auto input_mem2 = in_arr2.arr.GetMKLDNNData(); + NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); + orig_arr.arr.WaitToRead(); + PrintVerifyMsg(orig_arr, in_arr); + InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); + orig_arr.arr.CopyFrom(*input_mem); + op::MKLDNNSum(*input_mem, *input_mem2, *input_mem); + MKLDNNStream::Get()->Submit(); + VerifySumResult({&orig_arr.arr, &in_arr2.arr}, {&in_arr.arr}); + } +} + +TEST(MKLDNN_BASE, CreateMKLDNNMem) { + std::vector in_arrs = GetTestInputArrays(); + std::vector in_arrs2 = GetTestInputArrays(ArrayTypes::All, true); + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + MKLDNNStream *stream = MKLDNNStream::Get(); + + // kWriteTo + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (auto &out_arr : out_arrs) { + auto in_mem = in_arr.arr.GetMKLDNNData(); + auto in_mem2 = in_arr2.arr.GetMKLDNNData(); + NDArray orig_output = out_arr.arr.Copy(out_arr.arr.ctx()); + orig_output.WaitToRead(); + PrintVerifyMsg(in_arr, out_arr); + auto out_mem = out_arr.arr.GetMKLDNNData(); + auto output_mem_t = CreateMKLDNNMem(out_arr.arr, out_mem->get_primitive_desc(), kWriteTo); + op::MKLDNNSum(*in_mem, *in_mem2, *output_mem_t.second); + CommitOutput(out_arr.arr, output_mem_t); + stream->Submit(); + VerifySumResult({&in_arr.arr, &in_arr2.arr}, {&out_arr.arr}); + } + } + + // kWriteInPlace + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + auto input_mem = in_arr.arr.GetMKLDNNData(); + auto input_mem2 = in_arr2.arr.GetMKLDNNData(); + NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); + orig_arr.arr.WaitToRead(); + PrintVerifyMsg(orig_arr, in_arr); + InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); + orig_arr.arr.CopyFrom(*input_mem); + auto output_mem_t = CreateMKLDNNMem(in_arr.arr, + input_mem->get_primitive_desc(), kWriteInplace, &in_arr.arr); + op::MKLDNNSum(*input_mem, *input_mem2, *output_mem_t.second); + CommitOutput(in_arr.arr, output_mem_t); + stream->Submit(); + VerifySumResult({&orig_arr.arr, &in_arr2.arr}, {&in_arr.arr}); + } + + // kAddTo + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (auto &out_arr : out_arrs) { + auto in_mem = in_arr.arr.GetMKLDNNData(); + auto in_mem2 = in_arr2.arr.GetMKLDNNData(); + NDArray orig_output = out_arr.arr.Copy(out_arr.arr.ctx()); + orig_output.WaitToRead(); + PrintVerifyMsg(in_arr, out_arr); + auto out_mem = out_arr.arr.GetMKLDNNData(); + auto output_mem_t = CreateMKLDNNMem(out_arr.arr, out_mem->get_primitive_desc(), kAddTo); + op::MKLDNNSum(*in_mem, *in_mem2, *output_mem_t.second); + CommitOutput(out_arr.arr, output_mem_t); + stream->Submit(); + VerifyAddRequest( + {&in_arr.arr, &in_arr2.arr}, {&orig_output}, {&out_arr.arr}, VerifySumResult); + } + } + + // kNullOp + for (int i = 0; i < in_arrs.size(); i++) { + auto in_arr = in_arrs[i]; + auto in_arr2 = in_arrs2[i]; + if (!SupportMKLDNN(in_arr.arr)) + continue; + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) { + continue; + } + auto input_mem = in_arr.arr.GetMKLDNNData(); + auto input_mem2 = in_arr2.arr.GetMKLDNNData(); + NDArrayAttrs orig_arr(in_arr.arr.Copy(in_arr.arr.ctx()), "In Place Copy"); + orig_arr.arr.WaitToRead(); + PrintVerifyMsg(orig_arr, in_arr); + InitMKLDNNArray(&orig_arr.arr, input_mem->get_primitive_desc()); + orig_arr.arr.CopyFrom(*input_mem); + auto output_mem_t = CreateMKLDNNMem(in_arr.arr, input_mem->get_primitive_desc(), kNullOp); + op::MKLDNNSum(*input_mem, *input_mem2, *output_mem_t.second); + CommitOutput(in_arr.arr, output_mem_t); + stream->Submit(); + // original and input should be the same since noop + VerifyCopyResult({&orig_arr.arr}, {&in_arr.arr}); + } +} + +TEST(MKLDNN_NDArray, GetTestInputArraysConcat) { + auto in_arrs = GetTestInputArrays(); + for (int dim = 0; dim < 5; dim++) { + for (int num_inputs = 2; num_inputs < 5; num_inputs++) { + std::vector scale_vector(dim + 1); + for (size_t i = 0; i < dim + 1; ++i) + scale_vector[i] = 1; + scale_vector[dim] = num_inputs; + std::vector expanded_arrs = GetTestInputArrays( + ArrayTypes::All, false, scale_vector); + int i = 0; + for (auto &arr : in_arrs) { + if (dim >= arr.arr.shape().ndim()) + continue; + auto ex_arr = expanded_arrs[i]; + PrintVerifyMsg(arr, ex_arr); + EXPECT_EQ(arr.arr.shape().Size() * num_inputs, ex_arr.arr.shape().Size()); + EXPECT_EQ(arr.arr.shape()[dim] * num_inputs, ex_arr.arr.shape()[dim]); + i++; + } + } + } +} + +TEST(MKLDNN_NDArray, GetTestOutputArraysConcat) { + auto shapes_pds = GetTestArrayShapes(); + std::vector shapes; shapes = shapes_pds.shapes; + std::vector pds = shapes_pds.pds; + for (auto &shape : shapes) { + for (int dim = 0; dim < 5; dim++) { + for (int num_inputs = 2; num_inputs < 5; num_inputs++) { + if (shape.ndim() <= dim) + continue; + std::cout << "Extending " << shape << " dim " << + dim << " and " << num_inputs << "num_inputs\n"; + std::vector scale_vector(shape.ndim()); + for (int i = 0; i < shape.ndim(); i++) + scale_vector[i] = 1; + scale_vector[dim] = num_inputs; + auto output_arrs = GetTestOutputArrays(shape, pds, scale_vector); + for (auto &out_arr : output_arrs) { + auto out_shape = out_arr.arr.shape(); + EXPECT_EQ(shape.Size() * num_inputs, out_arr.arr.shape().Size()); + EXPECT_EQ(shape[dim] * num_inputs, out_arr.arr.shape()[dim]); + } + } + } + } +} + +TEST(MKLDNN_NDArray, CopyFrom) { + TestArrayShapes tas = GetTestArrayShapes(); + std::vector pds = tas.pds; + + std::vector in_arrs = GetTestInputArrays(); + for (auto &in_arr : in_arrs) { + if (in_arr.arr.IsMKLDNNData() && in_arr.arr.IsView()) + continue; + std::vector out_arrs = GetTestOutputArrays(in_arr.arr.shape(), pds); + for (auto &out_arr : out_arrs) { + const mkldnn::memory *mem = in_arr.arr.GetMKLDNNData(); + out_arr.arr.CopyFrom(*mem); + MKLDNNStream::Get()->Submit(); + std::vector inputs(1); + inputs[0] = &in_arr.arr; + VerifyCopyResult(inputs, {&out_arr.arr}); + } + } +} + +#endif diff --git a/tests/jenkins/run_test.sh b/tests/jenkins/run_test.sh index bc69ca1d7..5ef8c1ac0 100755 --- a/tests/jenkins/run_test.sh +++ b/tests/jenkins/run_test.sh @@ -56,8 +56,8 @@ nosetests3 --verbose tests/python/train || exit -1 echo "BUILD scala_test" export PATH=$PATH:/opt/apache-maven/bin -make scalapkg || exit -1 -make scalatest || exit -1 +cd scala-package +mvn install || exit -1 # echo "BUILD julia_test" # export MXNET_HOME="${PWD}" diff --git a/tests/jenkins/run_test_amzn_linux_gpu.sh b/tests/jenkins/run_test_amzn_linux_gpu.sh index ecfb5211b..57d9c7884 100755 --- a/tests/jenkins/run_test_amzn_linux_gpu.sh +++ b/tests/jenkins/run_test_amzn_linux_gpu.sh @@ -65,5 +65,5 @@ nosetests3 --verbose tests/python/train #julia -e 'try Pkg.clone("MXNet"); catch end; Pkg.checkout("MXNet"); Pkg.build("MXNet"); Pkg.test("MXNet")' || exit -1 echo "BUILD scala_test" -make scalapkg -make scalatest +cd scala-package +mvn integration-test diff --git a/tests/jenkins/run_test_ubuntu.sh b/tests/jenkins/run_test_ubuntu.sh index cdddd2865..28e00331b 100755 --- a/tests/jenkins/run_test_ubuntu.sh +++ b/tests/jenkins/run_test_ubuntu.sh @@ -68,6 +68,6 @@ nosetests3 --verbose tests/python/train || exit 1 echo "BUILD scala_test" export PATH=$PATH:/opt/apache-maven/bin -make scalapkg || exit 1 -make scalatest || exit 1 +cd scala-package +mvn integration-test || exit 1 diff --git a/tests/nightly/Jenkinsfile b/tests/nightly/Jenkinsfile index de550a071..758c86476 100755 --- a/tests/nightly/Jenkinsfile +++ b/tests/nightly/Jenkinsfile @@ -20,12 +20,12 @@ mx_lib = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/nnvm/lib/libnnvm.a' -node('mxnetlinux-cpu') { +node('utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') utils.main_wrapper( core_logic: { @@ -113,6 +113,22 @@ core_logic: { } } }, + 'Java Demo: CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/java-demo') { + utils.init_git() + utils.docker_run('ubuntu_nightly_cpu', 'nightly_java_demo_test_cpu', false) + } + } + }, + 'Scala Demo: CPU': { + node(NODE_LINUX_CPU) { + ws('workspace/scala-demo') { + utils.init_git() + utils.docker_run('ubuntu_nightly_cpu', 'nightly_scala_demo_test_cpu', false) + } + } + }, 'MXNetJS: CPU': { node(NODE_LINUX_CPU) { ws('workspace/nt-mxnetjs') { diff --git a/tests/nightly/JenkinsfileForBinaries b/tests/nightly/JenkinsfileForBinaries index 39dc8796e..eb003097f 100755 --- a/tests/nightly/JenkinsfileForBinaries +++ b/tests/nightly/JenkinsfileForBinaries @@ -20,12 +20,12 @@ mx_lib = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a' -node('mxnetlinux-cpu') { +node('utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') utils.main_wrapper( core_logic: { @@ -90,6 +90,22 @@ core_logic: { utils.docker_run('ubuntu_nightly_gpu', 'nightly_straight_dope_python3_multi_gpu_tests', true) } } + }, + 'Tutorial: Python2': { + node(NODE_LINUX_GPU) { + ws('workspace/tutorial-test-python2') { + utils.unpack_and_init('gpu', mx_lib) + utils.docker_run('ubuntu_nightly_gpu', 'nightly_tutorial_test_ubuntu_python2_gpu', true) + } + } + }, + 'Tutorial: Python3': { + node(NODE_LINUX_GPU) { + ws('workspace/tutorial-test-python3') { + utils.unpack_and_init('gpu', mx_lib) + utils.docker_run('ubuntu_nightly_gpu', 'nightly_tutorial_test_ubuntu_python3_gpu', true) + } + } } } } diff --git a/tests/nightly/apache_rat_license_check/rat-excludes b/tests/nightly/apache_rat_license_check/rat-excludes index 0c305f498..5969f01a3 100755 --- a/tests/nightly/apache_rat_license_check/rat-excludes +++ b/tests/nightly/apache_rat_license_check/rat-excludes @@ -5,19 +5,15 @@ .*ipynb .*html .*json -.*js .*txt .*md 3rdparty/* R-package/* -src/operator/mkl/* trunk/* docker/* -docker_multiarch/* .*\\.m .*\\.mk .*\\.R -Dockerfile* .*svg .*cfg .*config @@ -27,21 +23,15 @@ build/* .*\\.t MANIFEST Changes -META.yml .*csv .*names CODEOWNERS -prepare_mkl.sh -readthedocs.yml snap.python -snapcraft.yaml -image-classification-predict.cc bbox.pyx cpu_nms.pyx gpu_nms.pyx nms_kernel.cu _mask.pyx -unicodemap_en_baidu.csv coco.py base.pyi special_functions-inl.h @@ -50,7 +40,6 @@ im2col.h pool.h README.rst dataset.cPickle -rcnn/* image-classification/* rat-excludes apache-rat-tasks/* @@ -58,3 +47,6 @@ moderngpu/* deformable_im2col.cuh deformable_im2col.h REQUIRE +include/* +.*.iml +.*.json.ref \ No newline at end of file diff --git a/tests/nightly/broken_link_checker_test/JenkinsfileForBLC b/tests/nightly/broken_link_checker_test/JenkinsfileForBLC index a65074439..8b6454393 100755 --- a/tests/nightly/broken_link_checker_test/JenkinsfileForBLC +++ b/tests/nightly/broken_link_checker_test/JenkinsfileForBLC @@ -18,12 +18,12 @@ // //This is a Jenkinsfile for the broken link checker test. -node('mxnetlinux-cpu') { +node('utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'mxnetlinux-cpu', linux_gpu: 'mxnetlinux-gpu', linux_gpu_p3: 'mxnetlinux-gpu-p3', windows_cpu: 'mxnetwindows-cpu', windows_gpu: 'mxnetwindows-gpu') +utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu') utils.main_wrapper( core_logic: { diff --git a/tests/nightly/dist_async_kvstore.py b/tests/nightly/dist_async_kvstore.py index 3e400eafa..b990b6b3f 100644 --- a/tests/nightly/dist_async_kvstore.py +++ b/tests/nightly/dist_async_kvstore.py @@ -27,22 +27,26 @@ nworker = kv.num_workers def test_gluon_trainer_type(): - def check_trainer_kv_update(update_on_kv): + def check_trainer_kv_update(weight_stype, update_on_kv): params = mx.gluon.ParameterDict() - x = params.get('x', shape=(10,1), lr_mult=1.0) + x = params.get('x', shape=(10,1), lr_mult=1.0, stype=weight_stype) params.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') try: - trainer = mx.gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, kvstore=kv, update_on_kvstore=update_on_kv) + trainer = mx.gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, + kvstore=kv, update_on_kvstore=update_on_kv) trainer._init_kvstore() assert trainer._kv_initialized assert trainer._update_on_kvstore is True except ValueError: assert update_on_kv is False - check_trainer_kv_update(False) - check_trainer_kv_update(True) - check_trainer_kv_update(None) + check_trainer_kv_update('default', False) + check_trainer_kv_update('default', True) + check_trainer_kv_update('default', None) + check_trainer_kv_update('row_sparse', False) + check_trainer_kv_update('row_sparse', True) + check_trainer_kv_update('row_sparse', None) print('worker ' + str(my_rank) + ' passed test_gluon_trainer_type') if __name__ == "__main__": - test_gluon_trainer_type() \ No newline at end of file + test_gluon_trainer_type() diff --git a/tests/nightly/dist_device_sync_kvstore.py b/tests/nightly/dist_device_sync_kvstore.py index 75b48f42c..7fd0333ae 100644 --- a/tests/nightly/dist_device_sync_kvstore.py +++ b/tests/nightly/dist_device_sync_kvstore.py @@ -90,6 +90,25 @@ def check_init(kv, cur_keys, cur_shape, device=False): my_rank = kv.rank print('worker ' + str(my_rank) + ' is initialized') +def test_gluon_trainer_type(): + def check_trainer_kv_update(update_on_kv): + params = mx.gluon.ParameterDict() + x = params.get('x', shape=(10,1), lr_mult=1.0) + params.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') + try: + trainer = mx.gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, kvstore=kv, update_on_kvstore=update_on_kv) + trainer._init_kvstore() + assert trainer._kv_initialized + assert trainer._update_on_kvstore is True + except ValueError: + assert update_on_kv is False + + check_trainer_kv_update(False) + check_trainer_kv_update(True) + check_trainer_kv_update(None) + my_rank = kv.rank + print('worker ' + str(my_rank) + ' passed test_gluon_trainer_type') + if __name__ == "__main__": test_sync_init() test_sync_push_pull() diff --git a/tests/nightly/dist_sync_kvstore.py b/tests/nightly/dist_sync_kvstore.py index 861b85913..4523a361c 100644 --- a/tests/nightly/dist_sync_kvstore.py +++ b/tests/nightly/dist_sync_kvstore.py @@ -376,18 +376,26 @@ def check_invalid_pull(): check_invalid_pull() def test_gluon_trainer_type(): - def check_trainer_kv_type(stype, grad_stype, update_on_kv): + def check_trainer_kv_type(stype, grad_stype, update_on_kv, expected): params = mx.gluon.ParameterDict() x = params.get('x', shape=(10,1), lr_mult=1.0, stype=stype, grad_stype=grad_stype) params.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') - trainer = mx.gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, kvstore=kv) - trainer._init_kvstore() - assert trainer._kv_initialized - assert trainer._update_on_kvstore is update_on_kv - - check_trainer_kv_type('default', 'default', False) - check_trainer_kv_type('default', 'row_sparse', True) - check_trainer_kv_type('row_sparse', 'row_sparse', True) + trainer = mx.gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, + kvstore=kv, update_on_kvstore=update_on_kv) + try: + trainer._init_kvstore() + assert trainer._kv_initialized + assert trainer._update_on_kvstore is expected + except Exception as err: + assert isinstance(err, expected) + + check_trainer_kv_type('default', 'default', None, True) + check_trainer_kv_type('default', 'default', True, True) + check_trainer_kv_type('default', 'default', False, False) + check_trainer_kv_type('default', 'row_sparse', None, True) + check_trainer_kv_type('default', 'row_sparse', False, ValueError) + check_trainer_kv_type('row_sparse', 'row_sparse', None, True) + check_trainer_kv_type('row_sparse', 'row_sparse', False, ValueError) print('worker ' + str(my_rank) + ' passed test_gluon_trainer_type') def test_gluon_trainer_step(): diff --git a/tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC b/tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC index cb35161f5..08f562189 100644 --- a/tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC +++ b/tests/nightly/model_backwards_compatibility_check/JenkinsfileForMBCC @@ -20,12 +20,12 @@ mx_lib = 'lib/libmxnet.so, lib/libmxnet.a, 3rdparty/dmlc-core/libdmlc.a, 3rdparty/tvm/nnvm/lib/libnnvm.a' -node('restricted-mxnetlinux-cpu') { +node('restricted-utility') { // Loading the utilities requires a node context unfortunately checkout scm utils = load('ci/Jenkinsfile_utils.groovy') } -utils.assign_node_labels(linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu') +utils.assign_node_labels(utility: 'restricted-utility', linux_cpu: 'restricted-mxnetlinux-cpu') utils.main_wrapper( core_logic: { diff --git a/tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh b/tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh index 336c61df2..02d480d9d 100755 --- a/tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh +++ b/tests/nightly/model_backwards_compatibility_check/train_mxnet_legacy_models.sh @@ -61,8 +61,8 @@ echo `pwd` ## This list is sorted in descending order chronologically. ## Sample output for the below git tag command is : 1.2.0 utils 1.1.0 1.0.0 0.12.1 ## so from this sample, we will pick up all the versions matching with the current latest version -## Now while performing inference the latest version could be 1.3.0, which will help in validating models trained -## on 1.1.0 and 1.2.0 by loading them on the latest version (1.3.0) +## Now while performing inference the latest version could be 1.4.0, which will help in validating models trained +## on 1.1.0 and 1.2.0 by loading them on the latest version (1.4.0) ## Over a period of time, the model repository will grow since with every new release we ## upload models trained on newer versions as well through this script previous_versions=($(git tag --sort=-creatordate | grep --invert-match rc)) diff --git a/tests/nightly/test_large_array.py b/tests/nightly/test_large_array.py index 121acc174..696fdb1d4 100644 --- a/tests/nightly/test_large_array.py +++ b/tests/nightly/test_large_array.py @@ -15,20 +15,137 @@ # specific language governing permissions and limitations # under the License. -import unittest import mxnet as mx +import numpy as np from mxnet import gluon, nd +# dimension constants +MEDIUM_X = 10000 +LARGE_X = 100000000 +LARGE_Y = 50000000 +SMALL_Y = 50 +LARGE_SIZE = LARGE_X * SMALL_Y + +def test_gluon_embedding(): + m = gluon.nn.Embedding(SMALL_Y, MEDIUM_X) + m.initialize() + a = nd.zeros((MEDIUM_X, SMALL_Y)) + b = m(a) + assert b.shape == (MEDIUM_X, SMALL_Y, MEDIUM_X) + assert b.asnumpy().size == LARGE_SIZE + +def test_ndarray_zeros(): + a = nd.zeros(shape=(LARGE_X, SMALL_Y)) + assert a[-1][0] == 0 + assert a.shape == (LARGE_X, SMALL_Y) + assert a.size == LARGE_SIZE + +def test_ndarray_ones(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + assert a[-1][0] == 1 + assert nd.sum(a).asnumpy() == LARGE_SIZE + +def test_ndarray_random_uniform(): + a = nd.random.uniform(shape=(LARGE_X, SMALL_Y)) + assert a[-1][0] != 0 + +def test_ndarray_empty(): + a = nd.empty((LARGE_X, SMALL_Y)) + assert a.shape == (LARGE_X, SMALL_Y) + +def test_elementwise(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.ones(shape=(LARGE_X, SMALL_Y)) + res = a + b + assert np.sum(res[-1].asnumpy() == 2) == a.shape[1] + res = a + 1 + assert np.sum(res[-1].asnumpy() == 2) == a.shape[1] + res = nd.sqrt(a + 3) + assert np.sum(res[-1].asnumpy() == 2) == a.shape[1] + +def test_reduce(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + assert nd.sum(a).asnumpy() == a.shape[0] * a.shape[1] + +def test_dot(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.ones(shape=(SMALL_Y, SMALL_Y)) + res = nd.dot(a, b) + assert np.sum(res[-1].asnumpy() == SMALL_Y) == b.shape[1] + +def test_FullyConnected(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.ones(shape=(SMALL_Y, SMALL_Y)) + res = nd.FullyConnected(a, b, num_hidden=b.shape[1], no_bias=True) + assert np.sum(res[-1].asnumpy() == SMALL_Y) == b.shape[1] + +def test_broadcast(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.arange(0, LARGE_X).reshape(LARGE_X, 1) + res = nd.broadcast_to(b, shape=(b.shape[0], SMALL_Y)) + assert np.sum(res[-1].asnumpy() == LARGE_X) == res.shape[1] + res = mx.nd.broadcast_like(b, a) + assert np.sum(res[-1].asnumpy() == LARGE_X) == a.shape[1] + +def test_clip(): + a = nd.arange(0, LARGE_X).reshape(LARGE_X, 1) + b = nd.broadcast_to(a, shape=(a.shape[0], SMALL_Y)) + res = nd.clip(b, a_min=100, a_max=1000) + assert np.sum(res[-1].asnumpy() == 1000) == b.shape[1] + +def test_take(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + idx = nd.arange(LARGE_X-1000, LARGE_X) + res = nd.take(a, idx) + assert np.sum(res[-1].asnumpy() == 1) == res.shape[1] + +def test_slice(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + res = nd.slice(a, begin=(LARGE_X-1000, 1), end=(LARGE_X, SMALL_Y)) + assert np.sum(res[-1].asnumpy() == 1) == res.shape[1] + +def test_slice_assign(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + a[LARGE_X-1:LARGE_X] = 1000 + assert np.sum(a[-1].asnumpy() == 1000) == a.shape[1] + +def test_expand_dims(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + res = nd.expand_dims(a, axis=1) + assert res.shape == (a.shape[0], 1, a.shape[1]) + +def test_squeeze(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + data = nd.expand_dims(a, axis=1) + res = nd.squeeze(data) + assert res.shape == a.shape + +def test_broadcast_div(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.ones(shape=(LARGE_X, 1)) * 2 + res = a / b + assert np.sum(res[-1].asnumpy() == 0.5) == a.shape[1] + +def test_Dense(ctx=mx.cpu(0)): + data = mx.nd.ones(shape=(50*1000*1000, 100)) + linear = gluon.nn.Dense(100) + linear.initialize(ctx=ctx) + res = linear(data) + res.wait_to_read() + assert res.shape == (50000000, 100) + +def test_where(): + a = nd.ones(shape=(LARGE_X, SMALL_Y)) + b = nd.arange(0, LARGE_X).reshape(LARGE_X, 1) + b = nd.broadcast_to(b, shape=(b.shape[0], SMALL_Y)) + res = nd.where(b > 100, a, b) + assert np.sum(res[-1].asnumpy() == 1) == b.shape[1] + + csr_cond = nd.sparse.cast_storage(b < 10, 'csr') + res = nd.sparse.where(csr_cond, a, b) + assert np.sum(res[0].asnumpy() == 1) == b.shape[1] -class TestLargeArray(unittest.TestCase): - def test_ndarray2numpy(self): - m = gluon.nn.Embedding(14000, 128) - m.initialize() - ind = nd.zeros((700000, 128)) - x = m(ind) - x.shape - test = x.asnumpy() - assert (x.shape == test.shape) if __name__ == '__main__': - unittest.main() + import nose + nose.runmodule() diff --git a/tests/python-pytest/onnx/README.md b/tests/python-pytest/onnx/README.md new file mode 100644 index 000000000..d8f58cba3 --- /dev/null +++ b/tests/python-pytest/onnx/README.md @@ -0,0 +1,33 @@ +# ONNX tests + +## Directory structure: + +```bash +. +├── README.md +├── backend.py +├── backend_rep.py +├── backend_test.py +├── gluon_backend_test.py +├── mxnet_backend_test.py +├── mxnet_export_test.py +├── test_cases.py +├── test_models.py +└── test_node.py +``` + +* `backend.py` - MXNetBackend. This file contains prepare(). \ +This class can be used for both, MXNet and Gluon backend. +* `backend_rep.py` - MXNetBackendRep and GluonBackendRep for running inference +* `backend_test.py` - prepare tests by including tests from `test_cases.py` +* `gluon_backend_test.py` - Set backend as gluon and execute ONNX tests for ONNX->Gluon import. +* `mxnet_backend_test.py` - Set backend as gluon and add tests for ONNX->MXNet import/export. +Since MXNetBackend for export, tests both import and export, the test list in this file is +a union of tests that execute for import and export, export alone, and import alone. +* `mxnet_export_test.py` - Execute unit tests for testing MXNet export code - this is not specific to +any operator. +* `test_cases.py` - list of test cases for operators/models that are supported +for "both", import and export, "import" alone, or "export" alone. +* `test_models.py` - custom tests for models +* `test_node.py` - custom tests for operators. These tests are written independent of ONNX tests, in case +ONNX doesn't have tests yet or for MXNet specific operators. \ No newline at end of file diff --git a/tests/python-pytest/onnx/export/backend.py b/tests/python-pytest/onnx/backend.py similarity index 57% rename from tests/python-pytest/onnx/export/backend.py rename to tests/python-pytest/onnx/backend.py index e23cc0149..2f9e2470d 100644 --- a/tests/python-pytest/onnx/export/backend.py +++ b/tests/python-pytest/onnx/backend.py @@ -16,47 +16,57 @@ # under the License. # coding: utf-8 -"""backend wrapper for onnx test infrastructure""" -import numpy as np +"""MXNet/Gluon backend wrapper for onnx test infrastructure""" + from mxnet.contrib.onnx.onnx2mx.import_onnx import GraphProto from mxnet.contrib.onnx.mx2onnx.export_onnx import MXNetGraph +import mxnet as mx +import numpy as np + try: from onnx import helper, TensorProto, mapping from onnx.backend.base import Backend except ImportError: - raise ImportError("Onnx and protobuf need to be installed") -from backend_rep import MXNetBackendRep + raise ImportError("Onnx and protobuf need to be installed. Instructions to" + + " install - https://github.com/onnx/onnx#installation") +from backend_rep import MXNetBackendRep, GluonBackendRep + -# Using these functions for onnx test infrastructure. -# Implemented by following onnx docs guide: -# https://github.com/onnx/onnx/blob/master/docs/Implementing%20an%20ONNX%20backend.md # MXNetBackend class will take an ONNX model with inputs, perform a computation, # and then return the output. +# Implemented by following onnx docs guide: +# https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md class MXNetBackend(Backend): - """MXNet backend for ONNX""" + """MXNet/Gluon backend for ONNX""" + + backend = 'mxnet' + operation = 'import' + + @classmethod + def set_params(cls, backend, operation): + cls.backend = backend + cls.operation = operation @staticmethod - def perform_import_export(graph_proto, input_shape): + def perform_import_export(sym, arg_params, aux_params, input_shape): """ Import ONNX model to mxnet model and then export to ONNX model and then import it back to mxnet for verifying the result""" graph = GraphProto() - sym, arg_params, aux_params = graph.from_onnx(graph_proto) - params = {} params.update(arg_params) params.update(aux_params) # exporting to onnx graph proto format converter = MXNetGraph() - graph_proto = converter.create_onnx_graph_proto(sym, params, in_shape=input_shape, in_type=mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype('float32')]) + graph_proto = converter.create_onnx_graph_proto(sym, params, in_shape=input_shape, + in_type=mapping.NP_TYPE_TO_TENSOR_TYPE[np.dtype('float32')]) # importing back to MXNET for verifying result. sym, arg_params, aux_params = graph.from_onnx(graph_proto) return sym, arg_params, aux_params - @classmethod def prepare(cls, model, device='CPU', **kwargs): """For running end to end model(used for onnx test backend) @@ -76,13 +86,31 @@ def prepare(cls, model, device='CPU', **kwargs): Returns object of MXNetBackendRep class which will be in turn used to run inference on the input model and return the result for comparison. """ + backend = kwargs.get('backend', cls.backend) + operation = kwargs.get('operation', cls.operation) graph = GraphProto() - metadata = graph.get_graph_metadata(model.graph) - input_data = metadata['input_tensor_data'] - input_shape = [data[1] for data in input_data] - sym, arg_params, aux_params = MXNetBackend.perform_import_export(model.graph, input_shape) - return MXNetBackendRep(sym, arg_params, aux_params, device) + if device == 'CPU': + ctx = mx.cpu() + else: + raise NotImplementedError("ONNX tests are run only for CPU context.") + + if backend == 'mxnet': + sym, arg_params, aux_params = graph.from_onnx(model.graph) + if operation == 'export': + metadata = graph.get_graph_metadata(model.graph) + input_data = metadata['input_tensor_data'] + input_shape = [data[1] for data in input_data] + sym, arg_params, aux_params = MXNetBackend.perform_import_export(sym, arg_params, aux_params, + input_shape) + + return MXNetBackendRep(sym, arg_params, aux_params, device) + elif backend == 'gluon': + if operation == 'import': + net = graph.graph_to_gluon(model.graph, ctx) + return GluonBackendRep(net, device) + elif operation == 'export': + raise NotImplementedError("Gluon->ONNX export not implemented.") @classmethod def supports_device(cls, device): @@ -92,6 +120,4 @@ def supports_device(cls, device): prepare = MXNetBackend.prepare -run_node = MXNetBackend.run_node - supports_device = MXNetBackend.supports_device diff --git a/tests/python-pytest/onnx/import/mxnet_backend_rep.py b/tests/python-pytest/onnx/backend_rep.py similarity index 56% rename from tests/python-pytest/onnx/import/mxnet_backend_rep.py rename to tests/python-pytest/onnx/backend_rep.py index 938f25d38..be6bc88d9 100644 --- a/tests/python-pytest/onnx/import/mxnet_backend_rep.py +++ b/tests/python-pytest/onnx/backend_rep.py @@ -22,7 +22,9 @@ except ImportError: raise ImportError("Onnx and protobuf need to be installed. Instructions to" + " install - https://github.com/onnx/onnx#installation") +import numpy as np import mxnet as mx +from mxnet import nd # Using these functions for onnx test infrastructure. # Implemented by following onnx docs guide: @@ -55,9 +57,6 @@ def run(self, inputs, **kwargs): params : numpy array result obtained after running the inference on mxnet """ - data_forward = [] - for val in inputs: - data_forward.append(mx.nd.array(val)) # create module, passing cpu context if self.device == 'CPU': ctx = mx.cpu() @@ -69,30 +68,70 @@ def run(self, inputs, **kwargs): data_names = [graph_input for graph_input in self.symbol.list_inputs() if graph_input not in self.arg_params and graph_input not in self.aux_params] - data_shapes = [] + data_forward = [] for idx, input_name in enumerate(data_names): - data_shapes.append((input_name, inputs[idx].shape)) - - # module bind method requires all data to have same batch size, - # using module if all data have same batch size - if len(set([data_shape[1][0] for data_shape in data_shapes])) == 1: - mod = mx.mod.Module(symbol=self.symbol, data_names=data_names, context=ctx, - label_names=None) - mod.bind(for_training=False, data_shapes=data_shapes, - label_shapes=None) - mod.set_params(arg_params=self.arg_params, aux_params=self.aux_params) - - # run inference - mod.forward(mx.io.DataBatch(data_forward)) - result = mod.get_outputs()[0].asnumpy() - # split operator inference returns 1 less dimension - if self.symbol.name.startswith('split'): - return [i.asnumpy() for i in mod.get_outputs()] - return [result] - # using symbol bind method if data have different batch size + val = inputs[idx] + data_forward.append(mx.nd.array(val)) + + if self.arg_params: + for idx, input_name in enumerate(self.arg_params): + val = self.arg_params[input_name] + data_names.append(input_name) + data_forward.append(mx.nd.array(val)) + + args = dict(zip(data_names, data_forward)) + exe = self.symbol.bind(ctx, args=args, aux_states=self.aux_params) + exe.forward(is_train=False) + result = [] + for output in exe.outputs: + result.append(output.asnumpy()) + return result + + +# GluonBackendRep object will be returned by GluonBackend's prepare method which is used to +# execute a model repeatedly. +# Inputs will be passed to the run method of MXNetBackendRep class, it will perform computation and +# retrieve the corresponding results for comparison to the onnx backend. +# https://github.com/onnx/onnx/blob/master/onnx/backend/test/runner/__init__.py. +# Implemented by following onnx docs guide: +# https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md + +class GluonBackendRep(BackendRep): + """Running model inference on gluon backend and return the result + to onnx test infrastructure for comparison.""" + def __init__(self, net, device): + self.net = net + self.device = device + + def run(self, inputs, **kwargs): + """Run model inference and return the result + + Parameters + ---------- + inputs : numpy array + input to run a layer on + + Returns + ------- + params : numpy array + result obtained after running the inference on mxnet + """ + # create module, passing cpu context + if self.device == 'CPU': + ctx = mx.cpu() + else: + raise NotImplementedError("ONNX tests are run only for CPU context.") + + # run inference + net_inputs = [nd.array(input_data, ctx=ctx) for input_data in inputs] + net_outputs = self.net(*net_inputs) + results = [] + if isinstance(net_outputs, list): + for output in net_outputs: + results.append(output.asnumpy()) + result = results else: - exec1 = self.symbol.bind(ctx, args=dict(zip(data_names, data_forward))) - exec1.forward(is_train=False) - result = exec1.outputs[0].asnumpy() - return [result] + results.extend([o for o in net_outputs.asnumpy()]) + result = [np.array(results)] + return result diff --git a/tests/python-pytest/onnx/backend_test.py b/tests/python-pytest/onnx/backend_test.py new file mode 100644 index 000000000..8eaa303a6 --- /dev/null +++ b/tests/python-pytest/onnx/backend_test.py @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""ONNX test backend wrapper""" +try: + import onnx.backend.test +except ImportError: + raise ImportError("Onnx and protobuf need to be installed") + +import test_cases +import unittest +import backend as mxnet_backend +import logging + +operations = ['import', 'export'] +backends = ['mxnet', 'gluon'] +# This is a pytest magic variable to load extra plugins +pytest_plugins = "onnx.backend.test.report", + + +def test_suite(backend_tests): # type: () -> unittest.TestSuite + ''' + TestSuite that can be run by TestRunner + This has been borrowed from onnx/onnx/backend/test/runner/__init__.py, + since Python3 cannot sort objects of type 'Type' as Runner.test_suite() + expects. + ''' + suite = unittest.TestSuite() + for case in backend_tests.test_cases.values(): + suite.addTests(unittest.defaultTestLoader.loadTestsFromTestCase(case)) + return suite + + +def prepare_tests(backend, oper): + """ + Prepare the test list + :param backend: mxnet/gluon backend + :param oper: str. export or import + :return: backend test list + """ + BACKEND_TESTS = onnx.backend.test.BackendTest(backend, __name__) + implemented_ops = test_cases.IMPLEMENTED_OPERATORS_TEST.get('both', []) + \ + test_cases.IMPLEMENTED_OPERATORS_TEST.get(oper, []) + + for op_test in implemented_ops: + BACKEND_TESTS.include(op_test) + + basic_models = test_cases.BASIC_MODEL_TESTS.get('both', []) + \ + test_cases.BASIC_MODEL_TESTS.get(oper, []) + + for basic_model_test in basic_models: + BACKEND_TESTS.include(basic_model_test) + + std_models = test_cases.STANDARD_MODEL.get('both', []) + \ + test_cases.STANDARD_MODEL.get(oper, []) + + for std_model_test in std_models: + BACKEND_TESTS.include(std_model_test) + + # Tests for scalar ops are in test_node.py + BACKEND_TESTS.exclude('.*scalar.*') + + return BACKEND_TESTS + + +for bkend in backends: + for operation in operations: + log = logging.getLogger(bkend + operation) + if bkend == 'gluon' and operation == 'export': + log.warning('Gluon->ONNX export not implemented. Skipping tests...') + continue + log.info('Executing tests for ' + bkend + ' backend: ' + operation) + mxnet_backend.MXNetBackend.set_params(bkend, operation) + BACKEND_TESTS = prepare_tests(mxnet_backend, operation) + unittest.TextTestRunner().run(test_suite(BACKEND_TESTS.enable_report())) diff --git a/tests/python-pytest/onnx/export/backend_rep.py b/tests/python-pytest/onnx/export/backend_rep.py deleted file mode 100644 index 8729eafea..000000000 --- a/tests/python-pytest/onnx/export/backend_rep.py +++ /dev/null @@ -1,84 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# coding: utf-8 -"""backend rep for onnx test infrastructure""" -try: - from onnx.backend.base import BackendRep -except ImportError: - raise ImportError("Onnx and protobuf need to be installed") -import mxnet as mx - -# Using these functions for onnx test infrastructure. -# Implemented by following onnx docs guide: -# https://github.com/onnx/onnx/blob/master/docs/Implementing%20an%20ONNX%20backend.md -# MXNetBackendRep object will be returned by MXNetBackend's prepare method which is used to -# execute a model repeatedly. -# Inputs will be passed to the run method of MXNetBackendRep class, it will perform computation and -# retrieve the corresponding results for comparison to the onnx backend. -# https://github.com/onnx/onnx/blob/master/onnx/backend/test/runner/__init__.py. - -class MXNetBackendRep(BackendRep): - """Running model inference on mxnet engine and return the result - to onnx test infrastructure for comparison.""" - def __init__(self, symbol, arg_params, aux_params, device): - self.symbol = symbol - self.arg_params = arg_params - self.aux_params = aux_params - self.device = device - - def run(self, inputs, **kwargs): - """Run model inference and return the result - - Parameters - ---------- - inputs : numpy array - input to run a layer on - - Returns - ------- - params : numpy array - result obtained after running the inference on mxnet - """ - data_forward = [] - for val in inputs: - data_forward.append(mx.nd.array(val)) - # create module, passing cpu context - if self.device == 'CPU': - ctx = mx.cpu() - else: - raise NotImplementedError("ONNX tests are run only for CPU context.") - - # To fetch the data names of the input to the model we list the inputs of the symbol graph - # and exclude the argument and auxiliary parameters from the list - data_names = [graph_input for graph_input in self.symbol.list_inputs() - if graph_input not in self.arg_params and graph_input not in self.aux_params] - - data_shapes = [] - for idx, input_name in enumerate(data_names): - data_shapes.append((input_name, inputs[idx].shape)) - - mod = mx.mod.Module(symbol=self.symbol, data_names=data_names, context=ctx, - label_names=None) - mod.bind(for_training=False, data_shapes=data_shapes, - label_shapes=None) - mod.set_params(arg_params=self.arg_params, aux_params=self.aux_params) - - # run inference - mod.forward(mx.io.DataBatch(data_forward)) - result = mod.get_outputs()[0].asnumpy() - return [result] diff --git a/tests/python-pytest/onnx/export/mxnet_export_test.py b/tests/python-pytest/onnx/export/mxnet_export_test.py deleted file mode 100644 index 9f91369d6..000000000 --- a/tests/python-pytest/onnx/export/mxnet_export_test.py +++ /dev/null @@ -1,251 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -""" -Tests for individual operators -This module contains operator tests which currently do not exist on -ONNX backend test framework. Once we have PRs on the ONNX repo and get -those PRs merged, this file will get EOL'ed. -""" -# pylint: disable=too-many-locals,wrong-import-position,import-error -from __future__ import absolute_import -import sys -import os -import unittest -import logging -import tarfile -from collections import namedtuple -import numpy as np -import numpy.testing as npt -from onnx import numpy_helper, helper -from onnx import TensorProto -from mxnet.test_utils import download -from mxnet.contrib import onnx as onnx_mxnet -import mxnet as mx -CURR_PATH = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) -sys.path.insert(0, os.path.join(CURR_PATH, '../../../python/unittest')) -import backend -from common import with_seed - -logger = logging.getLogger() -logger.setLevel(logging.DEBUG) -URLS = { - 'bvlc_googlenet': - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/bvlc_googlenet.tar.gz', - 'bvlc_reference_caffenet': - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/bvlc_reference_caffenet.tar.gz', - 'bvlc_reference_rcnn_ilsvrc13': - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/bvlc_reference_rcnn_ilsvrc13.tar.gz', - 'inception_v1': - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/inception_v1.tar.gz', - 'inception_v2': - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/inception_v2.tar.gz' -} - -def get_test_files(name): - """Extract tar file and returns model path and input, output data""" - tar_name = download(URLS.get(name), dirname=CURR_PATH.__str__()) - # extract tar file - tar_path = os.path.join(CURR_PATH, tar_name) - tar = tarfile.open(tar_path.__str__(), "r:*") - tar.extractall(path=CURR_PATH.__str__()) - tar.close() - data_dir = os.path.join(CURR_PATH, name) - model_path = os.path.join(data_dir, 'model.onnx') - - inputs = [] - outputs = [] - # get test files - for test_file in os.listdir(data_dir): - case_dir = os.path.join(data_dir, test_file) - # skip the non-dir files - if not os.path.isdir(case_dir): - continue - input_file = os.path.join(case_dir, 'input_0.pb') - input_tensor = TensorProto() - with open(input_file, 'rb') as proto_file: - input_tensor.ParseFromString(proto_file.read()) - inputs.append(numpy_helper.to_array(input_tensor)) - - output_tensor = TensorProto() - output_file = os.path.join(case_dir, 'output_0.pb') - with open(output_file, 'rb') as proto_file: - output_tensor.ParseFromString(proto_file.read()) - outputs.append(numpy_helper.to_array(output_tensor)) - - return model_path, inputs, outputs - - -def forward_pass(sym, arg, aux, data_names, input_data): - """ Perform forward pass on given data""" - # create module - mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) - mod.bind(for_training=False, data_shapes=[(data_names[0], input_data.shape)], label_shapes=None) - mod.set_params(arg_params=arg, aux_params=aux, - allow_missing=True, allow_extra=True) - # run inference - batch = namedtuple('Batch', ['data']) - mod.forward(batch([mx.nd.array(input_data)]), is_train=False) - - return mod.get_outputs()[0].asnumpy() - - -def test_models(model_name, input_shape, output_shape): - """ Tests Googlenet model for both onnx import and export""" - model_path, inputs, outputs = get_test_files(model_name) - logging.info("Translating model from ONNX model zoo to Mxnet") - sym, arg_params, aux_params = onnx_mxnet.import_model(model_path) - params = {} - params.update(arg_params) - params.update(aux_params) - - dir_path = os.path.dirname(model_path) - new_model_name = "exported_" + model_name + ".onnx" - onnx_file = os.path.join(dir_path, new_model_name) - - logging.info("Translating converted model from mxnet to ONNX") - converted_model_path = onnx_mxnet.export_model(sym, params, [input_shape], np.float32, onnx_file) - - sym, arg_params, aux_params = onnx_mxnet.import_model(converted_model_path) - - metadata = onnx_mxnet.get_model_metadata(converted_model_path) - assert len(metadata) == 2 - assert metadata.get('input_tensor_data') - assert metadata.get('input_tensor_data')[0][1] == input_shape - assert metadata.get('output_tensor_data') - assert metadata.get('output_tensor_data')[0][1] == output_shape - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - logging.info("Running inference on onnx re-import model in mxnet") - # run test for each test file - for input_data, output_data in zip(inputs, outputs): - result = forward_pass(sym, arg_params, aux_params, data_names, input_data) - - # verify the results - npt.assert_equal(result.shape, output_data.shape) - npt.assert_almost_equal(output_data, result, decimal=3) - logging.info(model_name + " conversion successful") - - -def test_model_accuracy(model_name, input_shape): - """ Imports ONNX model, runs inference, exports and imports back - run inference, compare result with the previous inference result""" - model_path, inputs, outputs = get_test_files(model_name) - logging.info("Translating model from ONNX model zoo to Mxnet") - sym, arg_params, aux_params = onnx_mxnet.import_model(model_path) - - metadata = onnx_mxnet.get_model_metadata(model_path) - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - expected_result= [] - for input_data, output_data in zip(inputs, outputs): - result = forward_pass(sym, arg_params, aux_params, data_names, input_data) - expected_result.append(result) - - params = {} - params.update(arg_params) - params.update(aux_params) - - dir_path = os.path.dirname(model_path) - new_model_name = "exported_" + model_name + ".onnx" - onnx_file = os.path.join(dir_path, new_model_name) - - logging.info("Translating converted model from mxnet to ONNX") - converted_model_path = onnx_mxnet.export_model(sym, params, [input_shape], np.float32, - onnx_file) - - sym, arg_params, aux_params = onnx_mxnet.import_model(converted_model_path) - - metadata = onnx_mxnet.get_model_metadata(converted_model_path) - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - actual_result = [] - for input_data, output_data in zip(inputs, outputs): - result = forward_pass(sym, arg_params, aux_params, data_names, input_data) - actual_result.append(result) - - # verify the results - for expected, actual in zip(expected_result, actual_result): - npt.assert_equal(expected.shape, actual.shape) - npt.assert_almost_equal(expected, actual, decimal=3) - -@with_seed() -def test_spacetodepth(): - n, c, h, w = shape = (1, 1, 4, 6) - input1 = np.random.rand(n, c, h, w).astype("float32") - blocksize = 2 - inputs = [helper.make_tensor_value_info("input1", TensorProto.FLOAT, shape=shape)] - - outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=(1, 4, 2, 3))] - - nodes = [helper.make_node("SpaceToDepth", ["input1"], ["output"], block_size=blocksize)] - - graph = helper.make_graph(nodes, - "spacetodepth_test", - inputs, - outputs) - - spacetodepth_model = helper.make_model(graph) - - bkd_rep = backend.prepare(spacetodepth_model) - output = bkd_rep.run([input1]) - - tmp = np.reshape(input1, [n, c, - h // blocksize, blocksize, - w // blocksize, blocksize]) - tmp = np.transpose(tmp, [0, 3, 5, 1, 2, 4]) - numpy_op = np.reshape(tmp, [n, c * (blocksize**2), - h // blocksize, - w // blocksize]) - - npt.assert_almost_equal(output[0], numpy_op) - -@with_seed() -def test_square(): - input1 = np.random.randint(1, 10, (2, 3)).astype("float32") - - ipsym = mx.sym.Variable("input1") - square = mx.sym.square(data=ipsym) - model = mx.mod.Module(symbol=square, data_names=['input1'], label_names=None) - model.bind(for_training=False, data_shapes=[('input1', np.shape(input1))], label_shapes=None) - model.init_params() - - args, auxs = model.get_params() - params = {} - params.update(args) - params.update(auxs) - - converted_model = onnx_mxnet.export_model(square, params, [np.shape(input1)], np.float32, "square.onnx") - - sym, arg_params, aux_params = onnx_mxnet.import_model(converted_model) - result = forward_pass(sym, arg_params, aux_params, ['input1'], input1) - - numpy_op = np.square(input1) - - npt.assert_almost_equal(result, numpy_op) - -if __name__ == '__main__': - test_models("bvlc_googlenet", (1, 3, 224, 224), (1, 1000)) - test_models("bvlc_reference_caffenet", (1, 3, 224, 224), (1, 1000)) - test_models("bvlc_reference_rcnn_ilsvrc13", (1, 3, 224, 224), (1, 200)) - - # Comparing MXNet inference result, since MXNet results don't match - # ONNX expected results due to AveragePool issue github issue(#10194) - test_model_accuracy("inception_v1", (1, 3, 224, 224)) - test_model_accuracy("inception_v2", (1, 3, 224, 224)) - - unittest.main() diff --git a/tests/python-pytest/onnx/export/onnx_backend_test.py b/tests/python-pytest/onnx/export/onnx_backend_test.py deleted file mode 100644 index 678435d92..000000000 --- a/tests/python-pytest/onnx/export/onnx_backend_test.py +++ /dev/null @@ -1,147 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""ONNX test backend wrapper""" -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function -from __future__ import unicode_literals - -import unittest -try: - import onnx.backend.test -except ImportError: - raise ImportError("Onnx and protobuf need to be installed") - -import backend as mxnet_backend - -# This is a pytest magic variable to load extra plugins -pytest_plugins = "onnx.backend.test.report", - -BACKEND_TESTS = onnx.backend.test.BackendTest(mxnet_backend, __name__) - -IMPLEMENTED_OPERATORS_TEST = [ - 'test_random_uniform', - 'test_random_normal', - 'test_add', - 'test_sub', - 'test_mul', - 'test_div', - 'test_neg', - 'test_abs', - 'test_sum', - 'test_tanh', - 'test_cos', - 'test_sin', - 'test_tan', - 'test_acos', - 'test_asin', - 'test_atan' - 'test_ceil', - 'test_floor', - 'test_concat', - 'test_identity', - 'test_sigmoid', - 'test_relu', - 'test_constant_pad', - 'test_edge_pad', - 'test_reflect_pad', - 'test_reduce_min', - 'test_reduce_max', - 'test_reduce_mean', - 'test_reduce_prod', - 'test_reduce_sum_d', - 'test_reduce_sum_keepdims_random', - 'test_squeeze', - 'test_softmax_example', - 'test_softmax_large_number', - 'test_softmax_axis_2', - 'test_transpose', - 'test_globalmaxpool', - 'test_globalaveragepool', - # enabling partial test cases for matmul - 'test_matmul_3d', - 'test_matmul_4d', - 'test_slice_cpu', - 'test_slice_neg', - 'test_squeeze_', - 'test_reciprocal', - 'test_sqrt', - 'test_pow', - 'test_exp_', - 'test_argmax', - 'test_argmin', - 'test_min', - 'test_max' - #pytorch operator tests - 'test_operator_exp', - 'test_operator_maxpool', - 'test_operator_params', - 'test_operator_permute2', - 'test_clip' - 'test_cast', - 'test_depthtospace' - ] - -BASIC_MODEL_TESTS = [ - 'test_AvgPool2D', - 'test_BatchNorm', - 'test_ConstantPad2d', - 'test_Conv2d', - 'test_ELU', - 'test_LeakyReLU', - 'test_MaxPool', - 'test_PReLU', - 'test_ReLU', - 'test_selu_default' - 'test_Sigmoid', - 'test_Softmax', - 'test_softmax_functional', - 'test_softmax_lastdim', - 'test_Tanh' - ] - -STANDARD_MODEL = [ - 'test_bvlc_alexnet', - 'test_densenet121', - # 'test_inception_v1', - # 'test_inception_v2', - 'test_resnet50', - # 'test_shufflenet', - 'test_squeezenet', - 'test_vgg16', - 'test_vgg19' - ] - -for op_test in IMPLEMENTED_OPERATORS_TEST: - BACKEND_TESTS.include(op_test) - -for basic_model_test in BASIC_MODEL_TESTS: - BACKEND_TESTS.include(basic_model_test) - -for std_model_test in STANDARD_MODEL: - BACKEND_TESTS.include(std_model_test) - -BACKEND_TESTS.exclude('.*broadcast.*') -BACKEND_TESTS.exclude('.*bcast.*') - - -# import all test cases at global scope to make them visible to python.unittest -globals().update(BACKEND_TESTS.enable_report().test_cases) - -if __name__ == '__main__': - unittest.main() diff --git a/tests/python-pytest/onnx/import/gluon_backend.py b/tests/python-pytest/onnx/import/gluon_backend.py deleted file mode 100644 index 25be60b57..000000000 --- a/tests/python-pytest/onnx/import/gluon_backend.py +++ /dev/null @@ -1,75 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# coding: utf-8 -"""Gluon backend wrapper for onnx test infrastructure""" -from mxnet.contrib.onnx.onnx2mx.import_onnx import GraphProto -import mxnet as mx - -try: - from onnx import helper, TensorProto - from onnx.backend.base import Backend -except ImportError: - raise ImportError("Onnx and protobuf need to be installed. Instructions to" - + " install - https://github.com/onnx/onnx#installation") -from gluon_backend_rep import GluonBackendRep - -# GluonBackend class will take an ONNX model with inputs, perform a computation, -# and then return the output. -# Implemented by following onnx docs guide: -# https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md - -class GluonBackend(Backend): - """Gluon backend for ONNX""" - - @classmethod - def prepare(cls, model, device='CPU', **kwargs): - """For running end to end model(used for onnx test backend) - - Parameters - ---------- - model : onnx ModelProto object - loaded onnx graph - device : 'CPU' - specifying device to run test on - kwargs : - other arguments - - Returns - ------- - GluonBackendRep : object - Returns object of GluonBackendRep class which will be in turn - used to run inference on the input model and return the result for comparison. - """ - graph = GraphProto() - if device == 'CPU': - ctx = mx.cpu() - else: - raise NotImplementedError("ONNX tests are run only for CPU context.") - - net = graph.graph_to_gluon(model.graph, ctx) - return GluonBackendRep(net, device) - - @classmethod - def supports_device(cls, device): - """Supports only CPU for testing""" - return device == 'CPU' - - -prepare = GluonBackend.prepare - -supports_device = GluonBackend.supports_device diff --git a/tests/python-pytest/onnx/import/gluon_backend_rep.py b/tests/python-pytest/onnx/import/gluon_backend_rep.py deleted file mode 100644 index 04c6ddde6..000000000 --- a/tests/python-pytest/onnx/import/gluon_backend_rep.py +++ /dev/null @@ -1,71 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# coding: utf-8 -"""gluon backend rep for onnx test infrastructure""" -import numpy as np -try: - from onnx.backend.base import BackendRep -except ImportError: - raise ImportError("Onnx and protobuf need to be installed. Instructions to" - + " install - https://github.com/onnx/onnx#installation") -import mxnet as mx -from mxnet import nd - -# GluonBackendRep object will be returned by GluonBackend's prepare method which is used to -# execute a model repeatedly. -# Inputs will be passed to the run method of MXNetBackendRep class, it will perform computation and -# retrieve the corresponding results for comparison to the onnx backend. -# https://github.com/onnx/onnx/blob/master/onnx/backend/test/runner/__init__.py. -# Implemented by following onnx docs guide: -# https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md - - -class GluonBackendRep(BackendRep): - """Running model inference on gluon backend and return the result - to onnx test infrastructure for comparison.""" - def __init__(self, net, device): - self.net = net - self.device = device - - def run(self, inputs, **kwargs): - """Run model inference and return the result - - Parameters - ---------- - inputs : numpy array - input to run a layer on - - Returns - ------- - params : numpy array - result obtained after running the inference on mxnet - """ - # create module, passing cpu context - if self.device == 'CPU': - ctx = mx.cpu() - else: - raise NotImplementedError("ONNX tests are run only for CPU context.") - - # run inference - net_inputs = [nd.array(input_data, ctx=ctx) for input_data in inputs] - net_outputs = self.net(*net_inputs) - results = [] - results.extend([o for o in net_outputs.asnumpy()]) - result = np.array(results) - - return [result] diff --git a/tests/python-pytest/onnx/import/gluon_backend_test.py b/tests/python-pytest/onnx/import/gluon_backend_test.py deleted file mode 100644 index 6dd5f8a07..000000000 --- a/tests/python-pytest/onnx/import/gluon_backend_test.py +++ /dev/null @@ -1,55 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""ONNX test backend wrapper""" -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function -from __future__ import unicode_literals - -import unittest -try: - import onnx.backend.test -except ImportError: - raise ImportError("Onnx and protobuf need to be installed. Instructions to" - + " install - https://github.com/onnx/onnx#installation") - -import gluon_backend -import test_cases - -# This is a pytest magic variable to load extra plugins -pytest_plugins = "onnx.backend.test.report", - -BACKEND_TESTS = onnx.backend.test.BackendTest(gluon_backend, __name__) - -for op_tests in test_cases.IMPLEMENTED_OPERATORS_TEST: - BACKEND_TESTS.include(op_tests) - -for std_model_test in test_cases.STANDARD_MODEL: - BACKEND_TESTS.include(std_model_test) - -for basic_model_test in test_cases.BASIC_MODEL_TESTS: - BACKEND_TESTS.include(basic_model_test) - -BACKEND_TESTS.exclude('.*broadcast.*') -BACKEND_TESTS.exclude('.*bcast.*') - -# import all test cases at global scope to make them visible to python.unittest -globals().update(BACKEND_TESTS.enable_report().test_cases) - -if __name__ == '__main__': - unittest.main() diff --git a/tests/python-pytest/onnx/import/mxnet_backend.py b/tests/python-pytest/onnx/import/mxnet_backend.py deleted file mode 100644 index 10f89ecbb..000000000 --- a/tests/python-pytest/onnx/import/mxnet_backend.py +++ /dev/null @@ -1,67 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# coding: utf-8 -"""MXNet backend wrapper for onnx test infrastructure""" -from mxnet.contrib.onnx.onnx2mx.import_onnx import GraphProto -try: - from onnx import helper, TensorProto - from onnx.backend.base import Backend -except ImportError: - raise ImportError("Onnx and protobuf need to be installed. Instructions to" - + " install - https://github.com/onnx/onnx#installation") -from mxnet_backend_rep import MXNetBackendRep - -# MXNetBackend class will take an ONNX model with inputs, perform a computation, -# and then return the output. -# Implemented by following onnx docs guide: -# https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md - -class MXNetBackend(Backend): - """MXNet backend for ONNX""" - - @classmethod - def prepare(cls, model, device='CPU', **kwargs): - """For running end to end model(used for onnx test backend) - - Parameters - ---------- - model : onnx ModelProto object - loaded onnx graph - device : 'CPU' - specifying device to run test on - kwargs : - other arguments - - Returns - ------- - MXNetBackendRep : object - Returns object of MXNetBackendRep class which will be in turn - used to run inference on the input model and return the result for comparison. - """ - graph = GraphProto() - sym, arg_params, aux_params = graph.from_onnx(model.graph) - return MXNetBackendRep(sym, arg_params, aux_params, device) - - @classmethod - def supports_device(cls, device): - """Supports only CPU for testing""" - return device == 'CPU' - -prepare = MXNetBackend.prepare - -supports_device = MXNetBackend.supports_device diff --git a/tests/python-pytest/onnx/import/mxnet_backend_test.py b/tests/python-pytest/onnx/import/mxnet_backend_test.py deleted file mode 100644 index d9e4dccae..000000000 --- a/tests/python-pytest/onnx/import/mxnet_backend_test.py +++ /dev/null @@ -1,55 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""ONNX test backend wrapper""" -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function -from __future__ import unicode_literals - -import unittest -try: - import onnx.backend.test -except ImportError: - raise ImportError("Onnx and protobuf need to be installed. Instructions to" - + " install - https://github.com/onnx/onnx#installation") - -import mxnet_backend -import test_cases - -# This is a pytest magic variable to load extra plugins -pytest_plugins = "onnx.backend.test.report", - -BACKEND_TESTS = onnx.backend.test.BackendTest(mxnet_backend, __name__) - -for op_tests in test_cases.IMPLEMENTED_OPERATORS_TEST: - BACKEND_TESTS.include(op_tests) - -for basic_model_test in test_cases.BASIC_MODEL_TESTS: - BACKEND_TESTS.include(basic_model_test) - -for std_model_test in test_cases.STANDARD_MODEL: - BACKEND_TESTS.include(std_model_test) - -BACKEND_TESTS.exclude('.*broadcast.*') -BACKEND_TESTS.exclude('.*bcast.*') - -# import all test cases at global scope to make them visible to python.unittest -globals().update(BACKEND_TESTS.enable_report().test_cases) - -if __name__ == '__main__': - unittest.main() diff --git a/tests/python-pytest/onnx/import/onnx_import_test.py b/tests/python-pytest/onnx/import/onnx_import_test.py deleted file mode 100644 index 573dd74a4..000000000 --- a/tests/python-pytest/onnx/import/onnx_import_test.py +++ /dev/null @@ -1,290 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -""" -Tests for individual operators -This module contains operator tests which currently do not exist on -ONNX backend test framework. Once we have PRs on the ONNX repo and get -those PRs merged, this file will get EOL'ed. -""" -# pylint: disable=too-many-locals,wrong-import-position,import-error -from __future__ import absolute_import -import sys -import os -import unittest -import logging -import hashlib -import tarfile -from collections import namedtuple -import numpy as np -import numpy.testing as npt -from onnx import helper -from onnx import numpy_helper -from onnx import TensorProto -from mxnet.test_utils import download -from mxnet.contrib import onnx as onnx_mxnet -import mxnet as mx -CURR_PATH = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) -sys.path.insert(0, os.path.join(CURR_PATH, '../../../python/unittest')) -from common import with_seed -import mxnet_backend - - -URLS = { - 'bvlc_googlenet' : - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/opset7/bvlc_googlenet.tar.gz', - 'bvlc_reference_caffenet' : - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/opset7/bvlc_reference_caffenet.tar.gz', - 'bvlc_reference_rcnn_ilsvrc13' : - 'https://s3.amazonaws.com/onnx-mxnet/model-zoo/opset7/bvlc_reference_rcnn_ilsvrc13.tar.gz', -} - -@with_seed() -def test_broadcast(): - """Test for broadcasting in onnx operators.""" - input1 = np.random.rand(1, 3, 4, 5).astype("float32") - input2 = np.random.rand(1, 5).astype("float32") - inputs = [helper.make_tensor_value_info("input1", TensorProto.FLOAT, shape=(1, 3, 4, 5)), - helper.make_tensor_value_info("input2", TensorProto.FLOAT, shape=(1, 5))] - - outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=(1, 3, 4, 5))] - - nodes = [helper.make_node("Add", ["input1", "input2"], ["output"])] - - graph = helper.make_graph(nodes, - "bcast_test", - inputs, - outputs) - - bcast_model = helper.make_model(graph) - - bkd_rep = mxnet_backend.prepare(bcast_model) - numpy_op = input1 + input2 - output = bkd_rep.run([input1, input2]) - npt.assert_almost_equal(output[0], numpy_op) - -@with_seed() -def test_greater(): - """Test for logical greater in onnx operators.""" - input1 = np.random.rand(1, 3, 4, 5).astype("float32") - input2 = np.random.rand(1, 5).astype("float32") - inputs = [helper.make_tensor_value_info("input1", TensorProto.FLOAT, shape=(1, 3, 4, 5)), - helper.make_tensor_value_info("input2", TensorProto.FLOAT, shape=(1, 5))] - - outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=(1, 3, 4, 5))] - - nodes = [helper.make_node("Greater", ["input1", "input2"], ["output"])] - - graph = helper.make_graph(nodes, - "greater_test", - inputs, - outputs) - - greater_model = helper.make_model(graph) - - bkd_rep = mxnet_backend.prepare(greater_model) - numpy_op = np.greater(input1, input2).astype(np.float32) - output = bkd_rep.run([input1, input2]) - npt.assert_almost_equal(output[0], numpy_op) - -@with_seed() -def test_lesser(): - """Test for logical greater in onnx operators.""" - input1 = np.random.rand(1, 3, 4, 5).astype("float32") - input2 = np.random.rand(1, 5).astype("float32") - inputs = [helper.make_tensor_value_info("input1", TensorProto.FLOAT, shape=(1, 3, 4, 5)), - helper.make_tensor_value_info("input2", TensorProto.FLOAT, shape=(1, 5))] - - outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=(1, 3, 4, 5))] - - nodes = [helper.make_node("Less", ["input1", "input2"], ["output"])] - - graph = helper.make_graph(nodes, - "lesser_test", - inputs, - outputs) - - greater_model = helper.make_model(graph) - - bkd_rep = mxnet_backend.prepare(greater_model) - numpy_op = np.less(input1, input2).astype(np.float32) - output = bkd_rep.run([input1, input2]) - npt.assert_almost_equal(output[0], numpy_op) - -@with_seed() -def test_equal(): - """Test for logical greater in onnx operators.""" - input1 = np.random.rand(1, 3, 4, 5).astype("float32") - input2 = np.random.rand(1, 5).astype("float32") - inputs = [helper.make_tensor_value_info("input1", TensorProto.FLOAT, shape=(1, 3, 4, 5)), - helper.make_tensor_value_info("input2", TensorProto.FLOAT, shape=(1, 5))] - - outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=(1, 3, 4, 5))] - - nodes = [helper.make_node("Equal", ["input1", "input2"], ["output"])] - - graph = helper.make_graph(nodes, - "equal_test", - inputs, - outputs) - - greater_model = helper.make_model(graph) - - bkd_rep = mxnet_backend.prepare(greater_model) - numpy_op = np.equal(input1, input2).astype(np.float32) - output = bkd_rep.run([input1, input2]) - npt.assert_almost_equal(output[0], numpy_op) - -def test_super_resolution_example(): - """Test the super resolution example in the example/onnx folder""" - sys.path.insert(0, os.path.join(CURR_PATH, '../../../../example/onnx/')) - import super_resolution - - sym, arg_params, aux_params = super_resolution.import_onnx() - - logging.info("Asserted the result of the onnx model conversion") - output_img_dim = 672 - input_image, img_cb, img_cr = super_resolution.get_test_image() - result_img = super_resolution.perform_inference(sym, arg_params, aux_params, - input_image, img_cb, img_cr) - - assert hashlib.md5(result_img.tobytes()).hexdigest() == '0d98393a49b1d9942106a2ed89d1e854' - assert result_img.size == (output_img_dim, output_img_dim) - -def get_test_files(name): - """Extract tar file and returns model path and input, output data""" - tar_name = download(URLS.get(name), dirname=CURR_PATH.__str__()) - # extract tar file - tar_path = os.path.join(CURR_PATH, tar_name) - tar = tarfile.open(tar_path.__str__(), "r:*") - tar.extractall(path=CURR_PATH.__str__()) - tar.close() - data_dir = os.path.join(CURR_PATH, name) - model_path = os.path.join(data_dir, 'model.onnx') - - inputs = [] - outputs = [] - # get test files - for test_file in os.listdir(data_dir): - case_dir = os.path.join(data_dir, test_file) - # skip the non-dir files - if not os.path.isdir(case_dir): - continue - input_file = os.path.join(case_dir, 'input_0.pb') - input_tensor = TensorProto() - with open(input_file, 'rb') as proto_file: - input_tensor.ParseFromString(proto_file.read()) - inputs.append(numpy_helper.to_array(input_tensor)) - - output_tensor = TensorProto() - output_file = os.path.join(case_dir, 'output_0.pb') - with open(output_file, 'rb') as proto_file: - output_tensor.ParseFromString(proto_file.read()) - outputs.append(numpy_helper.to_array(output_tensor)) - - return model_path, inputs, outputs - -def test_bvlc_googlenet(): - """ Tests Googlenet model""" - model_path, inputs, outputs = get_test_files('bvlc_googlenet') - logging.info("Translating Googlenet model from ONNX to Mxnet") - sym, arg_params, aux_params = onnx_mxnet.import_model(model_path) - metadata = onnx_mxnet.get_model_metadata(model_path) - assert len(metadata) == 2 - assert metadata.get('input_tensor_data') - assert metadata.get('input_tensor_data') == [(u'data_0', (1, 3, 224, 224))] - assert metadata.get('output_tensor_data') - assert metadata.get('output_tensor_data') == [(u'prob_1', (1, 1000))] - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - # run test for each test file - for input_data, output_data in zip(inputs, outputs): - # create module - mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) - mod.bind(for_training=False, data_shapes=[(data_names[0], input_data.shape)], label_shapes=None) - mod.set_params(arg_params=arg_params, aux_params=aux_params, - allow_missing=True, allow_extra=True) - # run inference - batch = namedtuple('Batch', ['data']) - mod.forward(batch([mx.nd.array(input_data)]), is_train=False) - - # verify the results - npt.assert_equal(mod.get_outputs()[0].shape, output_data.shape) - npt.assert_almost_equal(output_data, mod.get_outputs()[0].asnumpy(), decimal=3) - logging.info("Googlenet model conversion Successful") - -def test_bvlc_reference_caffenet(): - """Tests the bvlc cafenet model""" - model_path, inputs, outputs = get_test_files('bvlc_reference_caffenet') - logging.info("Translating Caffenet model from ONNX to Mxnet") - sym, arg_params, aux_params = onnx_mxnet.import_model(model_path) - metadata = onnx_mxnet.get_model_metadata(model_path) - assert len(metadata) == 2 - assert metadata.get('input_tensor_data') - assert metadata.get('input_tensor_data') == [(u'data_0', (1, 3, 224, 224))] - assert metadata.get('output_tensor_data') - assert metadata.get('output_tensor_data') == [(u'prob_1', (1, 1000))] - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - # run test for each test file - for input_data, output_data in zip(inputs, outputs): - # create module - mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) - mod.bind(for_training=False, data_shapes=[(data_names[0], input_data.shape)], label_shapes=None) - mod.set_params(arg_params=arg_params, aux_params=aux_params, - allow_missing=True, allow_extra=True) - # run inference - batch = namedtuple('Batch', ['data']) - mod.forward(batch([mx.nd.array(input_data)]), is_train=False) - - # verify the results - npt.assert_equal(mod.get_outputs()[0].shape, output_data.shape) - npt.assert_almost_equal(output_data, mod.get_outputs()[0].asnumpy(), decimal=3) - logging.info("Caffenet model conversion Successful") - -def test_bvlc_rcnn_ilsvrc13(): - """Tests the bvlc rcnn model""" - model_path, inputs, outputs = get_test_files('bvlc_reference_rcnn_ilsvrc13') - logging.info("Translating rcnn_ilsvrc13 model from ONNX to Mxnet") - sym, arg_params, aux_params = onnx_mxnet.import_model(model_path) - metadata = onnx_mxnet.get_model_metadata(model_path) - assert len(metadata) == 2 - assert metadata.get('input_tensor_data') - assert metadata.get('input_tensor_data') == [(u'data_0', (1, 3, 224, 224))] - assert metadata.get('output_tensor_data') - assert metadata.get('output_tensor_data') == [(u'fc-rcnn_1', (1, 200))] - data_names = [input_name[0] for input_name in metadata.get('input_tensor_data')] - - # run test for each test file - for input_data, output_data in zip(inputs, outputs): - # create module - mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) - mod.bind(for_training=False, data_shapes=[(data_names[0], input_data.shape)], label_shapes=None) - mod.set_params(arg_params=arg_params, aux_params=aux_params, - allow_missing=True, allow_extra=True) - # run inference - batch = namedtuple('Batch', ['data']) - mod.forward(batch([mx.nd.array(input_data)]), is_train=False) - - # verify the results - npt.assert_equal(mod.get_outputs()[0].shape, output_data.shape) - npt.assert_almost_equal(output_data, mod.get_outputs()[0].asnumpy(), decimal=3) - logging.info("rcnn_ilsvrc13 model conversion Successful") - - -if __name__ == '__main__': - unittest.main() diff --git a/tests/python-pytest/onnx/import/test_cases.py b/tests/python-pytest/onnx/import/test_cases.py deleted file mode 100644 index 7f34247c9..000000000 --- a/tests/python-pytest/onnx/import/test_cases.py +++ /dev/null @@ -1,120 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -"""Test Cases to be run for the import module""" - -IMPLEMENTED_OPERATORS_TEST = [ - 'test_split_equal' - 'test_random_', - 'test_add', - 'test_sub', - 'test_mul', - 'test_div', - 'test_neg', - 'test_abs', - 'test_sum', - 'test_tanh', - 'test_ceil', - 'test_floor', - 'test_concat', - 'test_identity', - 'test_sigmoid', - 'test_relu', - 'test_constant_pad', - 'test_edge_pad', - 'test_reflect_pad', - 'test_squeeze_', - 'test_unsqueeze', - 'test_softmax_example', - 'test_softmax_large_number', - 'test_softmax_axis_2', - 'test_transpose', - 'test_globalmaxpool', - 'test_globalaveragepool', - 'test_global_lppooling', - 'test_slice_cpu', - 'test_slice_neg', - 'test_reciprocal', - 'test_sqrt', - 'test_pow', - 'test_exp_', - 'test_argmax', - 'test_argmin', - 'test_min', - 'test_logical_', - # enabling partial test cases for matmul - 'test_matmul_3d', - 'test_matmul_4d', - 'test_clip', - 'test_softsign', - 'test_reduce_', - 'test_softplus', - 'test_mean', - 'test_acos', - 'test_asin', - 'test_atan', - 'test_cos', - 'test_sin', - 'test_tan', - 'test_shape', - 'test_hardsigmoid_', - 'test_averagepool_1d', - 'test_averagepool_2d_pads_count_include_pad', - 'test_averagepool_2d_precomputed_pads_count_include_pad', - 'test_averagepool_2d_precomputed_strides', - 'test_averagepool_2d_strides', - 'test_averagepool_3d', - 'test_LpPool_', - 'test_cast', - 'test_instancenorm', - #pytorch operator tests - 'test_operator_exp', - 'test_operator_maxpool', - 'test_operator_params', - 'test_operator_permute2', - 'test_depthtospace' - ] - -BASIC_MODEL_TESTS = [ - 'test_AvgPool2D', - 'test_BatchNorm', - 'test_ConstantPad2d' - 'test_Conv2d', - 'test_ELU', - 'test_LeakyReLU', - 'test_MaxPool', - 'test_PReLU', - 'test_ReLU', - 'test_selu_default', - 'test_Sigmoid', - 'test_Softmax', - 'test_softmax_functional', - 'test_softmax_lastdim', - 'test_Tanh' - ] - -STANDARD_MODEL = [ - 'test_bvlc_alexnet', - 'test_densenet121', - #'test_inception_v1', - #'test_inception_v2', - 'test_resnet50', - #'test_shufflenet', - 'test_squeezenet', - 'test_zfnet512', - 'test_vgg19' - ] diff --git a/tests/python-pytest/onnx/mxnet_export_test.py b/tests/python-pytest/onnx/mxnet_export_test.py new file mode 100644 index 000000000..6c81198a8 --- /dev/null +++ b/tests/python-pytest/onnx/mxnet_export_test.py @@ -0,0 +1,121 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +# pylint: disable=too-many-locals,wrong-import-position,import-error +from __future__ import absolute_import +import os +import unittest +import logging +import tempfile +from mxnet import nd, sym +from mxnet.gluon import nn +from mxnet.contrib import onnx as onnx_mxnet +import mxnet as mx + +logger = logging.getLogger() +logger.setLevel(logging.DEBUG) + + +def _assert_sym_equal(lhs, rhs): + assert lhs.list_inputs() == rhs.list_inputs() # input names must be identical + assert len(lhs.list_outputs()) == len(rhs.list_outputs()) # number of outputs must be identical + + +def _force_list(output): + if isinstance(output, nd.NDArray): + return [output] + return list(output) + + +def _optional_group(symbols, group=False): + if group: + return sym.Group(symbols) + else: + return symbols + + +def _check_onnx_export(net, group_outputs=False, shape_type=tuple, extra_params={}): + net.initialize() + data = nd.random.uniform(0, 1, (1, 1024)) + output = _force_list(net(data)) # initialize weights + net_sym = _optional_group(net(sym.Variable('data')), group_outputs) + net_params = {name: param._reduce() for name, param in net.collect_params().items()} + net_params.update(extra_params) + with tempfile.TemporaryDirectory() as tmpdirname: + onnx_file_path = os.path.join(tmpdirname, 'net.onnx') + export_path = onnx_mxnet.export_model( + sym=net_sym, + params=net_params, + input_shape=[shape_type(data.shape)], + onnx_file_path=onnx_file_path) + assert export_path == onnx_file_path + # Try importing the model to symbol + _assert_sym_equal(net_sym, onnx_mxnet.import_model(export_path)[0]) + + # Try importing the model to gluon + imported_net = onnx_mxnet.import_to_gluon(export_path, ctx=None) + _assert_sym_equal(net_sym, _optional_group(imported_net(sym.Variable('data')), group_outputs)) + + # Confirm network outputs are the same + imported_net_output = _force_list(imported_net(data)) + for out, imp_out in zip(output, imported_net_output): + mx.test_utils.assert_almost_equal(out.asnumpy(), imp_out.asnumpy()) + + +class TestExport(unittest.TestCase): + """ Tests ONNX export. + """ + + def test_onnx_export_single_output(self): + net = nn.HybridSequential(prefix='single_output_net') + with net.name_scope(): + net.add(nn.Dense(100, activation='relu'), nn.Dense(10)) + _check_onnx_export(net) + + def test_onnx_export_multi_output(self): + class MultiOutputBlock(nn.HybridBlock): + def __init__(self): + super(MultiOutputBlock, self).__init__() + with self.name_scope(): + self.net = nn.HybridSequential() + for i in range(10): + self.net.add(nn.Dense(100 + i * 10, activation='relu')) + + def hybrid_forward(self, F, x): + out = tuple(block(x) for block in self.net._children.values()) + return out + + net = MultiOutputBlock() + assert len(sym.Group(net(sym.Variable('data'))).list_outputs()) == 10 + _check_onnx_export(net, group_outputs=True) + + def test_onnx_export_list_shape(self): + net = nn.HybridSequential(prefix='list_shape_net') + with net.name_scope(): + net.add(nn.Dense(100, activation='relu'), nn.Dense(10)) + _check_onnx_export(net, shape_type=list) + + def test_onnx_export_extra_params(self): + net = nn.HybridSequential(prefix='extra_params_net') + with net.name_scope(): + net.add(nn.Dense(100, activation='relu'), nn.Dense(10)) + _check_onnx_export(net, extra_params={'extra_param': nd.array([1, 2])}) + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/python-pytest/onnx/test_cases.py b/tests/python-pytest/onnx/test_cases.py new file mode 100644 index 000000000..6ec37092d --- /dev/null +++ b/tests/python-pytest/onnx/test_cases.py @@ -0,0 +1,131 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +IMPLEMENTED_OPERATORS_TEST = { + 'both': ['test_add', + 'test_sub', + 'test_mul', + 'test_div', + 'test_neg', + 'test_abs', + 'test_sum', + 'test_tanh', + 'test_ceil', + 'test_floor', + 'test_concat', + 'test_identity', + 'test_sigmoid', + 'test_relu', + 'test_constant_pad', + 'test_edge_pad', + 'test_reflect_pad', + 'test_softmax_example', + 'test_softmax_large_number', + 'test_softmax_axis_2', + 'test_transpose', + 'test_globalmaxpool', + 'test_globalaveragepool', + 'test_slice_cpu', + 'test_slice_neg', + 'test_reciprocal', + 'test_sqrt', + 'test_pow', + 'test_exp_', + 'test_argmax', + 'test_argmin', + 'test_min', + # pytorch operator tests + 'test_exp_', + 'test_operator_maxpool', + 'test_operator_params', + 'test_operator_permute2', + 'test_cos', + 'test_sin', + 'test_tan', + 'test_acos', + 'test_asin', + 'test_atan', + 'test_squeeze', + 'test_matmul', + 'test_depthtospace', + 'test_hardsigmoid', + 'test_instancenorm', + 'test_shape', + 'test_cast', + 'test_clip', + 'test_size', + 'test_dropout', + 'test_unsqueeze', + 'test_log_', + 'test_flatten_default_axis', + 'test_leakyrelu', + 'test_selu_default', + 'test_elu', + 'test_max_', + 'test_softplus', + 'test_reduce_' + ], + 'import': ['test_gather', + 'test_softsign', + 'test_mean', + 'test_averagepool_1d', + 'test_averagepool_2d_pads_count_include_pad', + 'test_averagepool_2d_precomputed_pads_count_include_pad', + 'test_averagepool_2d_precomputed_strides', + 'test_averagepool_2d_strides', + 'test_averagepool_3d', + 'test_split_equal', + 'test_hardmax' + ], + 'export': ['test_random_uniform', + 'test_random_normal', + 'test_reduce_min', + 'test_reduce_max', + 'test_reduce_mean', + 'test_reduce_prod', + 'test_reduce_sum_d', + 'test_reduce_sum_keepdims_random', + 'test_lrn' + ] +} + +BASIC_MODEL_TESTS = { + 'both': ['test_AvgPool2D', + 'test_BatchNorm', + 'test_ConstantPad2d' + 'test_Conv2d', + 'test_MaxPool', + 'test_PReLU', + 'test_Softmax', + 'test_softmax_functional', + 'test_softmax_lastdim', + ] +} + +STANDARD_MODEL = { + 'both': ['test_bvlc_alexnet', + 'test_densenet121', + # 'test_inception_v1', + # 'test_inception_v2', + 'test_resnet50', + # 'test_shufflenet', + 'test_squeezenet', + 'test_vgg19' + ], + 'import': ['test_zfnet512'], + 'export': ['test_vgg16'] +} diff --git a/tests/python-pytest/onnx/test_models.py b/tests/python-pytest/onnx/test_models.py new file mode 100644 index 000000000..f85786141 --- /dev/null +++ b/tests/python-pytest/onnx/test_models.py @@ -0,0 +1,178 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + + +# pylint: disable=too-many-locals,wrong-import-position,import-error +from __future__ import absolute_import +import sys +import os +import unittest +import logging +import tarfile +from collections import namedtuple +import numpy as np +import numpy.testing as npt +from onnx import numpy_helper +from onnx import TensorProto +from mxnet.test_utils import download +from mxnet.contrib import onnx as onnx_mxnet +import mxnet as mx + +CURR_PATH = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) +sys.path.insert(0, os.path.join(CURR_PATH, '../../python/unittest')) + + +logger = logging.getLogger() +logger.setLevel(logging.DEBUG) +URLS = { + 'bvlc_googlenet': + 'https://s3.amazonaws.com/download.onnx/models/opset_8/bvlc_googlenet.tar.gz', + 'bvlc_reference_caffenet': + 'https://s3.amazonaws.com/download.onnx/models/opset_8/bvlc_reference_caffenet.tar.gz', + 'bvlc_reference_rcnn_ilsvrc13': + 'https://s3.amazonaws.com/download.onnx/models/opset_8/bvlc_reference_rcnn_ilsvrc13.tar.gz', + 'inception_v1': + 'https://s3.amazonaws.com/download.onnx/models/opset_8/inception_v1.tar.gz', + 'inception_v2': + 'https://s3.amazonaws.com/download.onnx/models/opset_8/inception_v2.tar.gz' +} + +test_model_path = "https://s3.amazonaws.com/onnx-mxnet/test_model.onnx" + +def get_test_files(name): + """Extract tar file and returns model path and input, output data""" + tar_name = download(URLS.get(name), dirname=CURR_PATH.__str__()) + # extract tar file + tar_path = os.path.join(CURR_PATH, tar_name) + tar = tarfile.open(tar_path.__str__(), "r:*") + tar.extractall(path=CURR_PATH.__str__()) + tar.close() + data_dir = os.path.join(CURR_PATH, name) + model_path = os.path.join(data_dir, 'model.onnx') + + inputs = [] + outputs = [] + # get test files + for test_file in os.listdir(data_dir): + case_dir = os.path.join(data_dir, test_file) + # skip the non-dir files + if not os.path.isdir(case_dir): + continue + input_file = os.path.join(case_dir, 'input_0.pb') + input_tensor = TensorProto() + with open(input_file, 'rb') as proto_file: + input_tensor.ParseFromString(proto_file.read()) + inputs.append(numpy_helper.to_array(input_tensor)) + + output_tensor = TensorProto() + output_file = os.path.join(case_dir, 'output_0.pb') + with open(output_file, 'rb') as proto_file: + output_tensor.ParseFromString(proto_file.read()) + outputs.append(numpy_helper.to_array(output_tensor)) + + return model_path, inputs, outputs + + +def forward_pass(sym, arg, aux, data_names, input_data): + """ Perform forward pass on given data""" + # create module + mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) + mod.bind(for_training=False, data_shapes=[(data_names[0], input_data.shape)], label_shapes=None) + mod.set_params(arg_params=arg, aux_params=aux, + allow_missing=True, allow_extra=True) + # run inference + batch = namedtuple('Batch', ['data']) + mod.forward(batch([mx.nd.array(input_data)]), is_train=False) + + return mod.get_outputs()[0].asnumpy() + + +class TestModel(unittest.TestCase): + """ Tests for models. + Tests are dynamically added. + Therefore edit test_models to add more tests. + """ + def test_import_export(self): + def get_model_results(modelpath): + symbol, args, aux = onnx_mxnet.import_model(modelpath) + + data = onnx_mxnet.get_model_metadata(modelpath) + data_names = [input_name[0] for input_name in data.get('input_tensor_data')] + + result = [] + for input_data, output_data in zip(inputs, outputs): + output = forward_pass(symbol, args, aux, data_names, input_data) + result.append(output) + return symbol, args, aux, result, data + + for test in test_cases: + model_name, input_shape, output_shape = test + with self.subTest(model_name): + model_path, inputs, outputs = get_test_files(model_name) + logging.info("Translating " + model_name + " from ONNX model zoo to MXNet") + + sym, arg_params, aux_params, expected_result, _ = get_model_results(model_path) + + params = {} + params.update(arg_params) + params.update(aux_params) + + dir_path = os.path.dirname(model_path) + new_model_name = "exported_" + model_name + ".onnx" + onnx_file = os.path.join(dir_path, new_model_name) + + logging.info("Translating converted model from mxnet to ONNX") + converted_model_path = onnx_mxnet.export_model(sym, params, [input_shape], np.float32, onnx_file) + + sym, arg_params, aux_params, actual_result, metadata = get_model_results(converted_model_path) + + assert len(metadata) == 2 + assert metadata.get('input_tensor_data') + assert metadata.get('input_tensor_data')[0][1] == input_shape + assert metadata.get('output_tensor_data') + assert metadata.get('output_tensor_data')[0][1] == output_shape + + # verify the results + for expected, actual in zip(expected_result, actual_result): + npt.assert_equal(expected.shape, actual.shape) + npt.assert_almost_equal(expected, actual, decimal=3) + + logging.info(model_name + " conversion successful") + + def test_nodims_import(self): + # Download test model without dims mentioned in params + test_model = download(test_model_path, dirname=CURR_PATH.__str__()) + input_data = np.array([0.2, 0.5]) + nd_data = mx.nd.array(input_data).expand_dims(0) + sym, arg_params, aux_params = onnx_mxnet.import_model(test_model) + model_metadata = onnx_mxnet.get_model_metadata(test_model) + input_names = [inputs[0] for inputs in model_metadata.get('input_tensor_data')] + output_data = forward_pass(sym, arg_params, aux_params, input_names, nd_data) + assert(output_data.shape == (1,1)) + +# test_case = ("model name", input shape, output shape) +test_cases = [ + ("bvlc_googlenet", (1, 3, 224, 224), (1, 1000)), + ("bvlc_reference_caffenet", (1, 3, 224, 224), (1, 1000)), + ("bvlc_reference_rcnn_ilsvrc13", (1, 3, 224, 224), (1, 200)), + ("inception_v1", (1, 3, 224, 224), (1, 1000)), + ("inception_v2", (1, 3, 224, 224), (1, 1000)) +] + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/python-pytest/onnx/test_node.py b/tests/python-pytest/onnx/test_node.py new file mode 100644 index 000000000..25fe9c9f9 --- /dev/null +++ b/tests/python-pytest/onnx/test_node.py @@ -0,0 +1,278 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Tests for individual operators +This module contains operator tests which currently do not exist on +ONNX backend test framework. Once we have PRs on the ONNX repo and get +those PRs merged, this file will get EOL'ed. +""" +# pylint: disable=too-many-locals,wrong-import-position,import-error +from __future__ import absolute_import +import sys +import os +import unittest +import logging +import tarfile +from collections import namedtuple +import numpy as np +import numpy.testing as npt +from onnx import numpy_helper, helper, load_model +from onnx import TensorProto +from mxnet.test_utils import download +from mxnet.contrib import onnx as onnx_mxnet +import mxnet as mx +import backend + +CURR_PATH = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) +sys.path.insert(0, os.path.join(CURR_PATH, '../../python/unittest')) + +logger = logging.getLogger() +logger.setLevel(logging.DEBUG) + + +def get_rnd(shape, low=-1.0, high=1.0, dtype=np.float32): + if dtype == np.float32: + return (np.random.uniform(low, high, + np.prod(shape)).reshape(shape).astype(np.float32)) + elif dtype == np.int32: + return (np.random.randint(low, high, + np.prod(shape)).reshape(shape).astype(np.float32)) + elif dtype == np.bool_: + return np.random.choice(a=[False, True], size=shape).astype(np.float32) + + +def _fix_attributes(attrs, attribute_mapping): + new_attrs = attrs + attr_modify = attribute_mapping.get('modify', {}) + for k, v in attr_modify.items(): + new_attrs[v] = new_attrs.pop(k, None) + + attr_add = attribute_mapping.get('add', {}) + for k, v in attr_add.items(): + new_attrs[k] = v + + attr_remove = attribute_mapping.get('remove', []) + for k in attr_remove: + if k in new_attrs: + del new_attrs[k] + + return new_attrs + + +def forward_pass(sym, arg, aux, data_names, input_data): + """ Perform forward pass on given data + :param sym: Symbol + :param arg: Arg params + :param aux: Aux params + :param data_names: Input names (list) + :param input_data: Input data (list). If there is only one input, + pass it as a list. For example, if input is [1, 2], + pass input_data=[[1, 2]] + :return: result of forward pass + """ + data_shapes = [] + data_forward = [] + for idx in range(len(data_names)): + val = input_data[idx] + data_shapes.append((data_names[idx], np.shape(val))) + data_forward.append(mx.nd.array(val)) + # create module + mod = mx.mod.Module(symbol=sym, data_names=data_names, context=mx.cpu(), label_names=None) + mod.bind(for_training=False, data_shapes=data_shapes, label_shapes=None) + if not arg and not aux: + mod.init_params() + else: + mod.set_params(arg_params=arg, aux_params=aux, + allow_missing=True, allow_extra=True) + # run inference + batch = namedtuple('Batch', ['data']) + mod.forward(batch(data_forward), is_train=False) + + return mod.get_outputs()[0].asnumpy() + + +def get_input_tensors(input_data): + input_tensor = [] + input_names = [] + input_sym = [] + for idx, ip in enumerate(input_data): + name = "input" + str(idx + 1) + input_sym.append(mx.sym.Variable(name)) + input_names.append(name) + input_tensor.append(helper.make_tensor_value_info(name, + TensorProto.FLOAT, shape=np.shape(ip))) + return input_names, input_tensor, input_sym + + +def get_onnx_graph(testname, input_names, inputs, output_name, output_shape, attr): + outputs = [helper.make_tensor_value_info("output", TensorProto.FLOAT, shape=output_shape)] + + nodes = [helper.make_node(output_name, input_names, ["output"], **attr)] + + graph = helper.make_graph(nodes, testname, inputs, outputs) + + model = helper.make_model(graph) + return model + +class TestNode(unittest.TestCase): + """ Tests for models. + Tests are dynamically added. + Therefore edit test_models to add more tests. + """ + def test_import_export(self): + for test in test_cases: + test_name, mxnet_op, onnx_name, inputs, attrs, mxnet_specific, fix_attrs, check_value, check_shape = test + with self.subTest(test_name): + names, input_tensors, inputsym = get_input_tensors(inputs) + if inputs: + test_op = mxnet_op(*inputsym, **attrs) + mxnet_output = forward_pass(test_op, None, None, names, inputs) + outputshape = np.shape(mxnet_output) + else: + test_op = mxnet_op(**attrs) + shape = attrs.get('shape', (1,)) + x = mx.nd.zeros(shape, dtype='float32') + xgrad = mx.nd.zeros(shape, dtype='float32') + exe = test_op.bind(ctx=mx.cpu(), args={'x': x}, args_grad={'x': xgrad}) + mxnet_output = exe.forward(is_train=False)[0].asnumpy() + outputshape = np.shape(mxnet_output) + + if mxnet_specific: + onnxmodelfile = onnx_mxnet.export_model(test_op, {}, [np.shape(ip) for ip in inputs], + np.float32, + onnx_name + ".onnx") + onnxmodel = load_model(onnxmodelfile) + else: + onnx_attrs = _fix_attributes(attrs, fix_attrs) + onnxmodel = get_onnx_graph(test_name, names, input_tensors, onnx_name, outputshape, onnx_attrs) + + bkd_rep = backend.prepare(onnxmodel, operation='export') + output = bkd_rep.run(inputs) + + if check_value: + npt.assert_almost_equal(output[0], mxnet_output) + + if check_shape: + npt.assert_equal(output[0].shape, outputshape) + + input1 = get_rnd((1, 10, 2, 3)) + ipsym = mx.sym.Variable("input1") + for test in test_scalar_ops: + if test == 'Add': + outsym = 2 + ipsym + if test == "Sub": + outsym = ipsym - 2 + if test == "rSub": + outsym = ipsym.__rsub__(2) + if test == "Mul": + outsym = 2 * ipsym + if test == "Div": + outsym = ipsym / 2 + if test == "Pow": + outsym = ipsym ** 2 + forward_op = forward_pass(outsym, None, None, ['input1'], input1) + converted_model = onnx_mxnet.export_model(outsym, {}, [np.shape(input1)], np.float32, + onnx_file_path=outsym.name + ".onnx") + + sym, arg_params, aux_params = onnx_mxnet.import_model(converted_model) + result = forward_pass(sym, arg_params, aux_params, ['input1'], input1) + + npt.assert_almost_equal(result, forward_op) + + def test_imports(self): + for test in import_test_cases: + test_name, onnx_name, inputs, np_op, attrs = test + with self.subTest(test_name): + names, input_tensors, inputsym = get_input_tensors(inputs) + np_out = [np_op(*inputs, **attrs)] + output_shape = np.shape(np_out) + onnx_model = get_onnx_graph(test_name, names, input_tensors, onnx_name, output_shape, attrs) + bkd_rep = backend.prepare(onnx_model, operation='import') + mxnet_out = bkd_rep.run(inputs) + npt.assert_almost_equal(np_out, mxnet_out, decimal=4) + +# test_case = ("test_case_name", mxnet op, "ONNX_op_name", [input_list], attribute map, MXNet_specific=True/False, +# fix_attributes = {'modify': {mxnet_attr_name: onnx_attr_name}, +# 'remove': [attr_name], +# 'add': {attr_name: value}, +# check_value=True/False, check_shape=True/False) +test_cases = [ + ("test_equal", mx.sym.broadcast_equal, "Equal", [get_rnd((1, 3, 4, 5)), get_rnd((1, 5))], {}, False, {}, True, + False), + ("test_greater", mx.sym.broadcast_greater, "Greater", [get_rnd((1, 3, 4, 5)), get_rnd((1, 5))], {}, False, {}, True, + False), + ("test_less", mx.sym.broadcast_lesser, "Less", [get_rnd((1, 3, 4, 5)), get_rnd((1, 5))], {}, False, {}, True, + False), + ("test_and", mx.sym.broadcast_logical_and, "And", + [get_rnd((3, 4, 5), dtype=np.bool_), get_rnd((3, 4, 5), dtype=np.bool_)], {}, False, {}, True, False), + ("test_xor", mx.sym.broadcast_logical_xor, "Xor", + [get_rnd((3, 4, 5), dtype=np.bool_), get_rnd((3, 4, 5), dtype=np.bool_)], {}, False, {}, True, False), + ("test_or", mx.sym.broadcast_logical_or, "Or", + [get_rnd((3, 4, 5), dtype=np.bool_), get_rnd((3, 4, 5), dtype=np.bool_)], {}, False, {}, True, False), + ("test_not", mx.sym.logical_not, "Not", [get_rnd((3, 4, 5), dtype=np.bool_)], {}, False, {}, True, False), + ("test_square", mx.sym.square, "Pow", [get_rnd((2, 3), dtype=np.int32)], {}, True, {}, True, False), + ("test_spacetodepth", mx.sym.space_to_depth, "SpaceToDepth", [get_rnd((1, 1, 4, 6))], + {'block_size': 2}, False, {}, True, False), + ("test_softmax", mx.sym.SoftmaxOutput, "Softmax", [get_rnd((1000, 1000)), get_rnd(1000)], + {'ignore_label': 0, 'use_ignore': False}, True, {}, True, False), + ("test_logistic_regression", mx.sym.LogisticRegressionOutput, "Sigmoid", + [get_rnd((1000, 1000)), get_rnd((1000, 1000))], {}, True, {}, True, False), + ("test_fullyconnected", mx.sym.FullyConnected, "Gemm", [get_rnd((4, 3)), get_rnd((4, 3)), get_rnd(4)], + {'num_hidden': 4, 'name': 'FC'}, True, {}, True, False), + ("test_lppool1", mx.sym.Pooling, "LpPool", [get_rnd((2, 3, 20, 20))], + {'kernel': (4, 5), 'pad': (0, 0), 'stride': (1, 1), 'p_value': 1, 'pool_type': 'lp'}, False, + {'modify': {'kernel': 'kernel_shape', 'pad': 'pads', 'stride': 'strides', 'p_value': 'p'}, + 'remove': ['pool_type']}, True, False), + ("test_lppool2", mx.sym.Pooling, "LpPool", [get_rnd((2, 3, 20, 20))], + {'kernel': (4, 5), 'pad': (0, 0), 'stride': (1, 1), 'p_value': 2, 'pool_type': 'lp'}, False, + {'modify': {'kernel': 'kernel_shape', 'pad': 'pads', 'stride': 'strides', 'p_value': 'p'}, + 'remove': ['pool_type']}, True, False), + ("test_globallppool1", mx.sym.Pooling, "GlobalLpPool", [get_rnd((2, 3, 20, 20))], + {'kernel': (4, 5), 'pad': (0, 0), 'stride': (1, 1), 'p_value': 1, 'pool_type': 'lp', 'global_pool': True}, False, + {'modify': {'p_value': 'p'}, + 'remove': ['pool_type', 'kernel', 'pad', 'stride', 'global_pool']}, True, False), + ("test_globallppool2", mx.sym.Pooling, "GlobalLpPool", [get_rnd((2, 3, 20, 20))], + {'kernel': (4, 5), 'pad': (0, 0), 'stride': (1, 1), 'p_value': 2, 'pool_type': 'lp', 'global_pool': True}, False, + {'modify': {'p_value': 'p'}, + 'remove': ['pool_type', 'kernel', 'pad', 'stride', 'global_pool']}, True, False), + ("test_roipool", mx.sym.ROIPooling, "MaxRoiPool", + [[[get_rnd(shape=(8, 6), low=1, high=100, dtype=np.int32)]], [[0, 0, 0, 4, 4]]], + {'pooled_size': (2, 2), 'spatial_scale': 0.7}, False, + {'modify': {'pooled_size': 'pooled_shape'}}, True, False), + + # since results would be random, checking for shape alone + ("test_multinomial", mx.sym.sample_multinomial, "Multinomial", + [np.array([0, 0.1, 0.2, 0.3, 0.4]).astype("float32")], + {'shape': (10,)}, False, {'modify': {'shape': 'sample_size'}}, False, True), + ("test_random_normal", mx.sym.random_normal, "RandomNormal", [], + {'shape': (2, 2), 'loc': 0, 'scale': 1}, False, {'modify': {'loc': 'mean'}}, False, True), + ("test_random_uniform", mx.sym.random_uniform, "RandomUniform", [], + {'shape': (2, 2), 'low': 0.5, 'high': 1.0}, False, {}, False, True) +] + +test_scalar_ops = ['Add', 'Sub', 'rSub' 'Mul', 'Div', 'Pow'] + +# test_case = ("test_case_name", "ONNX_op_name", [input_list], np_op, attribute map) +import_test_cases = [ + ("test_lpnormalization_default", "LpNormalization", [get_rnd([5, 3, 3, 2])], np.linalg.norm, {'ord':2, 'axis':-1}), + ("test_lpnormalization_ord1", "LpNormalization", [get_rnd([5, 3, 3, 2])], np.linalg.norm, {'ord':1, 'axis':-1}), + ("test_lpnormalization_ord2", "LpNormalization", [get_rnd([5, 3, 3, 2])], np.linalg.norm, {'ord':2, 'axis':1}) +] + +if __name__ == '__main__': + unittest.main() diff --git a/tests/python/gpu/test_device.py b/tests/python/gpu/test_device.py index 66772dc86..cd8145c3d 100644 --- a/tests/python/gpu/test_device.py +++ b/tests/python/gpu/test_device.py @@ -19,35 +19,23 @@ import numpy as np import unittest import os +import logging + +from mxnet.test_utils import EnvManager shapes = [(10), (100), (1000), (10000), (100000), (2,2), (2,3,4,5,6,7,8)] keys = [1,2,3,4,5,6,7] -num_gpus = len(mx.test_utils.list_gpus()) +num_gpus = mx.context.num_gpus() if num_gpus > 8 : - print("The machine has {} gpus. We will run the test on 8 gpus.".format(num_gpus)) - print("There is a limit for all PCI-E hardware on creating number of P2P peers. The limit is 8.") + logging.warn("The machine has {} gpus. We will run the test on 8 gpus.".format(num_gpus)) + logging.warn("There is a limit for all PCI-E hardware on creating number of P2P peers. The limit is 8.") num_gpus = 8; gpus = range(1, 1+num_gpus) -class EnvManager: - def __init__(self, key, val): - self._key = key - self._next_val = val - self._prev_val = None - - def __enter__(self): - try: - self._prev_val = os.environ[self._key] - except KeyError: - self._prev_val = '' - os.environ[self._key] = self._next_val - - def __exit__(self, ptype, value, trace): - os.environ[self._key] = self._prev_val - +@unittest.skipIf(mx.context.num_gpus() < 1, "test_device_pushpull needs at least 1 GPU") def test_device_pushpull(): def check_dense_pushpull(kv_type): for shape, key in zip(shapes, keys): @@ -63,20 +51,16 @@ def check_dense_pushpull(kv_type): for x in range(n_gpus): assert(np.sum(np.abs((res[x]-n_gpus).asnumpy()))==0) - envs1 = '1' - key1 = 'MXNET_KVSTORE_TREE_ARRAY_BOUND' - envs2 = ['','1'] - key2 = 'MXNET_KVSTORE_USETREE' - for i in range(2): - for val2 in envs2: - with EnvManager(key2, val2): + kvstore_tree_array_bound = 'MXNET_KVSTORE_TREE_ARRAY_BOUND' + kvstore_usetree_values = ['','1'] + kvstore_usetree = 'MXNET_KVSTORE_USETREE' + for _ in range(2): + for x in kvstore_usetree_values: + with EnvManager(kvstore_usetree, x): check_dense_pushpull('local') check_dense_pushpull('device') - - os.environ[key1] = envs1 - os.environ[key1] = '' - - print ("Passed") + os.environ[kvstore_tree_array_bound] = '1' + del os.environ[kvstore_tree_array_bound] if __name__ == '__main__': test_device_pushpull() diff --git a/tests/python/gpu/test_gluon_gpu.py b/tests/python/gpu/test_gluon_gpu.py index 8394276c8..54bfcee47 100644 --- a/tests/python/gpu/test_gluon_gpu.py +++ b/tests/python/gpu/test_gluon_gpu.py @@ -22,18 +22,22 @@ import time import multiprocessing as mp import unittest +import random import mxnet as mx import numpy as np import unittest +import math from nose.tools import assert_raises from mxnet.test_utils import check_consistency, set_default_context, assert_almost_equal from mxnet.base import MXNetError from mxnet import autograd from numpy.testing import assert_allclose +from mxnet.test_utils import rand_ndarray + curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) sys.path.insert(0, os.path.join(curr_path, '../unittest')) -from common import setup_module, with_seed, teardown, assert_raises_cudnn_disabled +from common import setup_module, with_seed, teardown, assert_raises_cudnn_not_satisfied from test_gluon import * from test_loss import * from test_gluon_rnn import * @@ -57,7 +61,7 @@ def check_rnn_layer(layer): for g, c in zip(gs, cs): assert_almost_equal(g.asnumpy(), c.asnumpy(), rtol=1e-2, atol=1e-6) - +@with_seed() def check_rnn_layer_w_rand_inputs(layer): layer.collect_params().initialize(ctx=[mx.cpu(0), mx.gpu(0)]) x = mx.nd.uniform(shape=(10, 16, 30)) @@ -77,7 +81,81 @@ def check_rnn_layer_w_rand_inputs(layer): @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='7.2.1') +def test_lstmp(): + hidden_size, projection_size = 3, 2 + rtol, atol = 1e-2, 1e-2 + batch_size, seq_len = 7, 11 + input_size = 5 + lstm_input = mx.nd.uniform(shape=(seq_len, batch_size, input_size), ctx=mx.gpu(0)) + shapes = {'i2h_weight': (hidden_size*4, input_size), + 'h2h_weight': (hidden_size*4, projection_size), + 'i2h_bias': (hidden_size*4,), + 'h2h_bias': (hidden_size*4,), + 'h2r_weight': (projection_size, hidden_size)} + weights = {k: rand_ndarray(v) for k, v in shapes.items()} + lstm_layer = gluon.rnn.LSTM(hidden_size, projection_size=projection_size, + input_size=input_size, prefix='lstm0_') + lstm_cell = gluon.contrib.rnn.LSTMPCell(hidden_size=hidden_size, + projection_size=projection_size, + input_size=input_size, + prefix='lstm0_l0_') + lstm_layer.initialize(ctx=mx.gpu(0)) + lstm_cell.initialize(ctx=mx.gpu(0)) + layer_params = lstm_layer.collect_params() + cell_params = lstm_cell.collect_params() + for k, v in weights.items(): + layer_params['lstm0_l0_'+k].set_data(v.copy()) + cell_params['lstm0_l0_'+k].set_data(v.copy()) + with autograd.record(): + layer_output = lstm_layer(lstm_input.copy()) + cell_output = lstm_cell.unroll(seq_len, lstm_input.copy(), layout='TNC', + merge_outputs=True)[0] + assert_almost_equal(layer_output.asnumpy(), cell_output.asnumpy(), rtol=rtol, atol=atol) + layer_output.backward() + cell_output.backward() + for k, v in weights.items(): + layer_grad = layer_params['lstm0_l0_'+k].grad() + cell_grad = cell_params['lstm0_l0_'+k].grad() + print('checking gradient for {}'.format('lstm0_l0_'+k)) + assert_almost_equal(layer_grad.asnumpy(), cell_grad.asnumpy(), + rtol=rtol, atol=atol) + check_rnn_layer_forward(gluon.rnn.LSTM(10, 2, projection_size=5), mx.nd.ones((8, 3, 20))) + check_rnn_layer_forward(gluon.rnn.LSTM(10, 2, projection_size=5, bidirectional=True), mx.nd.ones((8, 3, 20)), [mx.nd.ones((4, 3, 5)), mx.nd.ones((4, 3, 10))]) + + check_rnn_layer_forward(gluon.rnn.LSTM(10, 2, dropout=0.5, projection_size=5), mx.nd.ones((8, 3, 20)), + run_only=True) + check_rnn_layer_forward(gluon.rnn.LSTM(10, 2, bidirectional=True, dropout=0.5, projection_size=5), + mx.nd.ones((8, 3, 20)), + [mx.nd.ones((4, 3, 5)), mx.nd.ones((4, 3, 10))], run_only=True) + + +@with_seed() +@assert_raises_cudnn_not_satisfied(min_version='7.2.1') +def test_lstm_clip(): + hidden_size, projection_size = 4096, 2048 + batch_size, seq_len = 32, 80 + input_size = 50 + clip_min, clip_max, clip_nan = -5, 5, True + lstm_input = mx.nd.uniform(shape=(seq_len, batch_size, input_size), ctx=mx.gpu(0)) + lstm_states = [mx.nd.uniform(shape=(2, batch_size, projection_size), ctx=mx.gpu(0)), + mx.nd.uniform(shape=(2, batch_size, hidden_size), ctx=mx.gpu(0))] + lstm_layer = gluon.rnn.LSTM(hidden_size, projection_size=projection_size, + input_size=input_size, prefix='lstm0_', + bidirectional=True, + state_clip_min=clip_min, + state_clip_max=clip_max, + state_clip_nan=clip_nan) + lstm_layer.initialize(ctx=mx.gpu(0)) + with autograd.record(): + _, layer_output_states = lstm_layer(lstm_input, lstm_states) + cell_states = layer_output_states[0].asnumpy() + assert (cell_states >= clip_min).all() and (cell_states <= clip_max).all() + assert not np.isnan(cell_states).any() + + +@with_seed() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnn_layer(): check_rnn_layer(gluon.rnn.RNN(100, num_layers=3)) check_rnn_layer(gluon.rnn.RNN(100, activation='tanh', num_layers=3)) @@ -88,7 +166,65 @@ def test_rnn_layer(): check_rnn_layer_w_rand_inputs(gluon.rnn.LSTM(100, num_layers=3, bidirectional=True)) +def check_layer_bidirectional(size, in_size, proj_size): + class RefBiLSTM(gluon.Block): + def __init__(self, size, proj_size, **kwargs): + super(RefBiLSTM, self).__init__(**kwargs) + with self.name_scope(): + self._lstm_fwd = gluon.rnn.LSTM(size, projection_size=proj_size, bidirectional=False, prefix='l0') + self._lstm_bwd = gluon.rnn.LSTM(size, projection_size=proj_size, bidirectional=False, prefix='r0') + + def forward(self, inpt): + fwd = self._lstm_fwd(inpt) + bwd_inpt = nd.flip(inpt, 0) + bwd = self._lstm_bwd(bwd_inpt) + bwd = nd.flip(bwd, 0) + return nd.concat(fwd, bwd, dim=2) + weights = {} + for d in ['l', 'r']: + weights['lstm_{}0_i2h_weight'.format(d)] = mx.random.uniform(shape=(size*4, in_size)) + if proj_size: + weights['lstm_{}0_h2h_weight'.format(d)] = mx.random.uniform(shape=(size*4, proj_size)) + weights['lstm_{}0_h2r_weight'.format(d)] = mx.random.uniform(shape=(proj_size, size)) + else: + weights['lstm_{}0_h2h_weight'.format(d)] = mx.random.uniform(shape=(size*4, size)) + weights['lstm_{}0_i2h_bias'.format(d)] = mx.random.uniform(shape=(size*4,)) + weights['lstm_{}0_h2h_bias'.format(d)] = mx.random.uniform(shape=(size*4,)) + + net = gluon.rnn.LSTM(size, projection_size=proj_size, bidirectional=True, prefix='lstm_') + ref_net = RefBiLSTM(size, proj_size, prefix='lstm_') + net.initialize() + ref_net.initialize() + net_params = net.collect_params() + ref_net_params = ref_net.collect_params() + for k in weights: + net_params[k].set_data(weights[k]) + ref_net_params[k.replace('l0', 'l0l0').replace('r0', 'r0l0')].set_data(weights[k]) + + data = mx.random.uniform(shape=(11, 10, in_size)) + assert_allclose(net(data).asnumpy(), ref_net(data).asnumpy()) + +@with_seed() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') +def test_layer_bidirectional(): + check_layer_bidirectional(7, 5, 0) + @with_seed() +@assert_raises_cudnn_not_satisfied(min_version='7.2.1') +def test_layer_bidirectional_proj(): + check_layer_bidirectional(7, 5, 3) + + +@with_seed() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') +def test_rnn_layer_begin_state_type(): + fake_data = nd.random.uniform(shape=(3, 5, 7), dtype='float16') + modeling_layer = gluon.rnn.LSTM(hidden_size=11, num_layers=2, dropout=0.2, bidirectional=True) + modeling_layer.cast('float16') + modeling_layer.initialize() + modeling_layer(fake_data) + + def test_gluon_ctc_consistency(): loss = mx.gluon.loss.CTCLoss() data = mx.nd.arange(0, 4, repeat=40, ctx=mx.gpu(0)).reshape((2,20,4)).flip(axis=0) @@ -186,7 +322,7 @@ def _syncParameters(bn1, bn2, ctx): input2grad = mx.nd.concat(*[output.grad.as_in_context(input.context) for output in inputs2], dim=0) assert_almost_equal(input1.grad.asnumpy(), input2grad.asnumpy(), atol=1e-3, rtol=1e-3) - +@with_seed() def test_sync_batchnorm(): def get_num_devices(): for i in range(100): @@ -203,6 +339,7 @@ def get_num_devices(): _check_batchnorm_result(mx.nd.random.uniform(shape=(4, 1, 4, 4)), num_devices=ndev, cuda=True) + @with_seed() def test_symbol_block_fp16(): # Test case to verify if initializing the SymbolBlock from a model with params @@ -233,6 +370,45 @@ def test_symbol_block_fp16(): break assert np.dtype(net_fp16.params[param_name].dtype) == np.dtype(np.float16) + +@with_seed() +def test_large_models(): + ctx = default_context() + # Create model + net = gluon.nn.HybridSequential() + + largest_num_features = 256 + with net.name_scope(): + net.add(nn.Conv2D(largest_num_features, 3)) + + net.hybridize() + net.initialize(mx.init.Normal(sigma=0.01), ctx=ctx) + + # Compute the height (=width) of the square tensor of the given size in bytes + def tensor_size(big_tensor_bytes): + bytes_per_float = 4 + sz = int(math.sqrt(big_tensor_bytes / largest_num_features / bytes_per_float)) + return (sz // 100) * 100 + + # The idea is to create models with large tensors of (say) 20% of the total memory. + # This in the past has given cudnnFind() trouble when it needed to allocate similar I/O's + # from the area carved out by the MXNET_GPU_MEM_POOL_RESERVE setting (by default 5%). + (free_mem_bytes, total_mem_bytes) = mx.context.gpu_memory_info(ctx.device_id) + start_size = tensor_size(0.20 * total_mem_bytes) + num_trials = 10 + sys.stderr.write(' testing global memory of size {} ... '.format(total_mem_bytes)) + sys.stderr.flush() + for i in range(num_trials): + sz = start_size - 10 * i + (height, width) = (sz,sz) + sys.stderr.write(" {}x{} ".format(height,width)) + sys.stderr.flush() + data_in = nd.random_uniform(low=0, high=255, shape=(1, 3, height, width), + ctx=ctx, dtype="float32") + # Evaluate model + net(data_in).asnumpy() + + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/gpu/test_kvstore_gpu.py b/tests/python/gpu/test_kvstore_gpu.py index 4232a590a..8ff8752f5 100644 --- a/tests/python/gpu/test_kvstore_gpu.py +++ b/tests/python/gpu/test_kvstore_gpu.py @@ -21,7 +21,7 @@ import mxnet as mx import numpy as np import unittest -from mxnet.test_utils import assert_almost_equal, default_context +from mxnet.test_utils import assert_almost_equal, default_context, EnvManager curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) sys.path.insert(0, os.path.join(curr_path, '../unittest')) from common import setup_module, with_seed, teardown @@ -30,22 +30,6 @@ keys = [5, 7, 11] str_keys = ['b', 'c', 'd'] -class EnvManager: - def __init__(self, key, val): - self._key = key - self._next_val = val - self._prev_val = None - - def __enter__(self): - try: - self._prev_val = os.environ[self._key] - except KeyError: - self._prev_val = '' - os.environ[self._key] = self._next_val - - def __exit__(self, ptype, value, trace): - os.environ[self._key] = self._prev_val - def init_kv_with_str(stype='default', kv_type='local'): """init kv """ kv = mx.kv.create(kv_type) diff --git a/tests/python/gpu/test_operator_gpu.py b/tests/python/gpu/test_operator_gpu.py index 02895cd92..7a7c6f69d 100644 --- a/tests/python/gpu/test_operator_gpu.py +++ b/tests/python/gpu/test_operator_gpu.py @@ -32,7 +32,7 @@ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__))) sys.path.insert(0, os.path.join(curr_path, '../unittest')) -from common import setup_module, with_seed, teardown, assert_raises_cudnn_disabled +from common import setup_module, with_seed, teardown, assert_raises_cudnn_not_satisfied from test_operator import * from test_optimizer import * from test_random import * @@ -411,7 +411,7 @@ def test_3d_batchnorm(fix_gamma, use_global_stats): @with_seed(1234) -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_convolution_with_type(): sym1 = mx.sym.Convolution(num_filter=3, kernel=(3,3), name='conv') @@ -683,11 +683,11 @@ def test_deconvolution_options(): check_consistency_NxM([sym, sym_no_cudnn], ctx_list) # 2D deconvolution - ctx_list = [{'ctx': mx.gpu(0), 'deconv_data': (2, 2, 10, 10), 'type_dict': {'deconv_data': np.float64}}, - {'ctx': mx.gpu(0), 'deconv_data': (2, 2, 10, 10), 'type_dict': {'deconv_data': np.float32}}, - {'ctx': mx.gpu(0), 'deconv_data': (2, 2, 10, 10), 'type_dict': {'deconv_data': np.float16}}, - {'ctx': mx.cpu(0), 'deconv_data': (2, 2, 10, 10), 'type_dict': {'deconv_data': np.float64}}, - {'ctx': mx.cpu(0), 'deconv_data': (2, 2, 10, 10), 'type_dict': {'deconv_data': np.float32}}] + ctx_list = [{'ctx': mx.gpu(0), 'deconv_data': (2, 8, 10, 10), 'type_dict': {'deconv_data': np.float64}}, + {'ctx': mx.gpu(0), 'deconv_data': (2, 8, 10, 10), 'type_dict': {'deconv_data': np.float32}}, + {'ctx': mx.gpu(0), 'deconv_data': (2, 8, 10, 10), 'type_dict': {'deconv_data': np.float16}}, + {'ctx': mx.cpu(0), 'deconv_data': (2, 8, 10, 10), 'type_dict': {'deconv_data': np.float64}}, + {'ctx': mx.cpu(0), 'deconv_data': (2, 8, 10, 10), 'type_dict': {'deconv_data': np.float32}}] # Pad > 0 sym = mx.sym.Deconvolution(num_filter=2, kernel=(3,3), pad=(1,1), name='deconv') sym_no_cudnn = mx.sym.Deconvolution(num_filter=2, kernel=(3,3), pad=(1,1), cudnn_off=True, name='deconv') @@ -996,6 +996,35 @@ def test_3d_pooling(pool_type, p_value=2, count_include_pad=True): test_3d_pooling('lp', p_value=3) +@with_seed() +def test_pooling_full_2d(): + def test_pooling_full_2d_type(pool_type): + data = (2, 2, 10, 10) + kernel = (4, 5) + pad = (1, 2) + stride = (3, 4) + + convention = 'full' + ctx_list = [] + sym_list = [] + + # o_h = ceil((10 + 1 + 1 - 4) / 3) + 1 = 4 + # o_w = ceil((10 + 2 + 2 - 5) / 4) + 1 = 4 + ctx_list.append({'ctx': mx.cpu(0), 'pool_data': data, 'type_dict': {'pool_data': np.float32}}) + sym_list.append(mx.sym.Pooling(kernel=kernel, pad=pad, stride=stride, pool_type=pool_type, + pooling_convention=convention, global_pool=False, name='pool')) + + ctx_list.append({'ctx': mx.gpu(0), 'pool_data': data, 'type_dict': {'pool_data': np.float32}}) + sym_list.append(mx.sym.Pooling(kernel=kernel, pad=pad, stride=stride, pool_type=pool_type, + pooling_convention=convention, global_pool=False, name='pool')) + + check_consistency(sym_list, ctx_list) + + test_pooling_full_2d_type('max') + test_pooling_full_2d_type('avg') + test_pooling_full_2d_type('sum') + + @with_seed() def test_global_pooling(): def test_1d_pooling(pool_type, p_value=2): @@ -1364,7 +1393,7 @@ def check_rnn_consistency(cell1, cell2): assert_allclose(mod1.get_outputs()[0].asnumpy(), mod2.get_outputs()[0].asnumpy(), rtol=1e-2, atol=1e-4) @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnn(): fused = mx.rnn.FusedRNNCell(100, num_layers=2, mode='rnn_relu', prefix='') @@ -1376,7 +1405,7 @@ def test_rnn(): check_rnn_consistency(stack, fused) @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_lstm_forget_bias(): forget_bias = 2.0 fused = mx.rnn.FusedRNNCell(10, forget_bias=forget_bias, num_layers=2, mode='lstm', prefix='') @@ -1398,7 +1427,7 @@ def test_lstm_forget_bias(): assert_allclose(args[bias_name].asnumpy(), expected_bias) @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_gru(): fused = mx.rnn.FusedRNNCell(100, num_layers=2, mode='gru', prefix='') @@ -1410,7 +1439,7 @@ def test_gru(): check_rnn_consistency(stack, fused) @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_bidirectional(): fused = mx.rnn.FusedRNNCell(100, num_layers=2, mode='gru', prefix='', bidirectional=True) @@ -1429,7 +1458,7 @@ def test_bidirectional(): check_rnn_consistency(stack, fused) @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_unfuse(): for mode in ['rnn_tanh', 'rnn_relu', 'lstm', 'gru']: fused = mx.rnn.FusedRNNCell( @@ -1605,7 +1634,7 @@ def test_deformable_convolution_options(): name='deformable_conv') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_residual_fused(): cell = mx.rnn.ResidualCell( mx.rnn.FusedRNNCell(50, num_layers=3, mode='lstm', diff --git a/tests/python/mkl/test_mkldnn.py b/tests/python/mkl/test_mkldnn.py index e5490096f..01ba03cab 100644 --- a/tests/python/mkl/test_mkldnn.py +++ b/tests/python/mkl/test_mkldnn.py @@ -98,6 +98,37 @@ def __getitem__(self, key): assert_almost_equal(y[0, 0, 0, 0], 0.016711406) break +@with_seed() +def test_mkldnn_reshape(): + def test_reshape_after_conv(dst_shape): + shape = (1,1,4,4) + data = mx.symbol.Variable('data') + conv = mx.symbol.Convolution(data=data, num_filter=16, kernel=(1, 1), pad=(0, 0), stride=(1, 1)) + res = mx.symbol.reshape(data=conv, shape=dst_shape) + exe = res.simple_bind(mx.cpu(), data=shape, grad_req='null') + + val1 = np.random.uniform(-1, 1, (4, 4)) + val2 = np.random.uniform(-1, 1, (1, 1, 1, 1)) + val3 = np.random.uniform(-1 ,1, (1)) + + exe.arg_arrays[0][:] = val1 + exe.arg_arrays[1][:] = val2 + exe.arg_arrays[2][:] = val3 + outputs = exe.forward(is_train=False)[0].asnumpy() + + conv_exe = conv.simple_bind(mx.cpu(), data=shape, grad_req='null') + conv_exe.arg_arrays[0][:] = val1 + conv_exe.arg_arrays[1][:] = val2 + conv_exe.arg_arrays[2][:] = val3 + data_npy = conv_exe.forward(is_train=False)[0].asnumpy() + assert_almost_equal(outputs, data_npy.reshape(dst_shape)) + + + # Test mkldnn reshape (Using shape) + test_cases = [(256), (16, 16), (4, 4, 16), (4, 4, 4, 4)] + for test_case in test_cases: + test_reshape_after_conv(test_case) + @with_seed() def test_reshape_before_conv(): @@ -383,6 +414,21 @@ def check_fullyconnected_training(stype): for stype in stypes: check_fullyconnected_training(stype) +def test_softmax_with_large_inputs(): + def softmax_forward(input_data, true_output): + data = mx.sym.Variable('data') + out1 = data.softmax(axis=1) + exec1 = out1.bind(mx.cpu(), args={'data': input_data}) + exec1.forward()[0].wait_to_read() + ndarr = exec1.outputs[0][0][0][0] + nparr = ndarr.asnumpy() + assert_almost_equal(nparr, true_output, rtol=1e-5, atol=1e-5) + + softmax_forward(mx.nd.array([[[[-1e30,-1e30]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[1e30,1e30]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[-3.4e38,-3.4e38]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[3.4e38,3.4e38]]]]), np.array([1.0,1.0])) + @with_seed() def test_non_mkldnn_fcomputeex(): # test special case where MKLDNN formatted NDArray feeds into non-mkldnn fcomputeex operator diff --git a/tests/python/mkl/test_subgraph.py b/tests/python/mkl/test_subgraph.py index 71784dcd3..be6feaeb9 100644 --- a/tests/python/mkl/test_subgraph.py +++ b/tests/python/mkl/test_subgraph.py @@ -66,7 +66,7 @@ def check_qsym_dummy_forward(qsym, batch, data_shape, label_shape): output.wait_to_read() return mod.get_outputs() -def check_quantize(sym, data_shape): +def check_quantize(sym, data_shape, check_conv=True): fc = mx.sym.FullyConnected(data=sym, num_hidden=10, flatten=True, name='fc') sym = mx.sym.SoftmaxOutput(data=fc, name='softmax') sym_sg = sym.get_backend_symbol("MKLDNN") @@ -106,7 +106,8 @@ def check_quantize(sym, data_shape): calib_quantize_op=True, num_calib_examples=5) qsym = qsym.get_backend_symbol("MKLDNN_POST_QUANTIZE") - check_qsym_calibrated(qsym) + if check_conv: + check_qsym_calibrated(qsym) quantized_out = check_qsym_forward(qsym, qarg_params, qaux_params, batch, data_shape, label_shape) for i in range(len(ref_out)): assert_almost_equal(ref_out[i].asnumpy(), quantized_out[i].asnumpy(), atol = 1) @@ -229,6 +230,15 @@ def conv_bn_sum_relu(no_bias, data_shape): relu = mx.symbol.Activation(data=sum1, name='relu', act_type="relu") return relu, conv_bn_add_relu_attr +# single concat case +def single_concat(data_shape, input_num, dim): + data, weight = head_symbol(data_shape) + inputs = [] + for i in range(input_num): + inputs.append(data) + concat = mx.symbol.Concat(*inputs, name="concat", dim=dim) + return concat + def tail_neg_symbol(sym1, sym2): fc1 = mx.sym.FullyConnected(data=sym1, num_hidden=10, flatten=True, name='fc1') fc2 = mx.sym.FullyConnected(data=sym2, num_hidden=10, flatten=True, name='fc2') @@ -463,6 +473,15 @@ def test_pos_conv_bn_sum_relu(): net, attrs = conv_bn_sum_relu(True, data_shape) check_fusion(net, data_shape, attrs) +def test_pos_single_concat(): + for data_shape in DATA_SHAPE: + net = single_concat(data_shape, 2, 1) + check_quantize(net, data_shape, False) + net = single_concat(data_shape, 4, 2) + check_quantize(net, data_shape, False) + net = single_concat(data_shape, 4, 3) + check_quantize(net, data_shape, False) + @with_seed() def test_neg_conv_bn(): for data_shape in DATA_SHAPE: diff --git a/tests/python/quantization/test_quantization.py b/tests/python/quantization/test_quantization.py index 5ae2c6c39..3ff4b6930 100644 --- a/tests/python/quantization/test_quantization.py +++ b/tests/python/quantization/test_quantization.py @@ -26,6 +26,7 @@ from mxnet.module import Module from mxnet.io import NDArrayIter import unittest +import operator def is_test_for_gpu(): return mx.current_context().device_type == 'gpu' @@ -214,7 +215,7 @@ def check_quantized_conv(data_shape, kernel, num_filter, pad, stride, no_bias, q @with_seed() def test_quantized_pooling(): - def check_quantized_pooling(data_shape, kernel, pool_type, pad, stride, global_pool, qdtype): + def check_quantized_pooling(data_shape, kernel, pool_type, pad, stride, global_pool, qdtype, convention='valid'): if is_test_for_native_cpu(): print('skipped testing quantized_pooling for native cpu since it is not supported yet') return @@ -224,7 +225,8 @@ def check_quantized_pooling(data_shape, kernel, pool_type, pad, stride, global_p data = mx.sym.Variable(name='data', shape=data_shape, dtype='float32') pooling_fp32 = mx.sym.Pooling(data=data, kernel=kernel, pad=pad, stride=stride, - pool_type=pool_type, global_pool=global_pool, cudnn_off=False) + pool_type=pool_type, global_pool=global_pool, cudnn_off=False, + pooling_convention=convention) arg_shapes, _, _ = pooling_fp32.infer_shape(data=data_shape) arg_names = pooling_fp32.list_arguments() pooling_fp32_exe = pooling_fp32.simple_bind(ctx=mx.current_context(), grad_req='null') @@ -242,9 +244,10 @@ def check_quantized_pooling(data_shape, kernel, pool_type, pad, stride, global_p min_data = mx.sym.Variable(name='min_data') max_data = mx.sym.Variable(name='max_data') quantized_pooling = mx.sym.contrib.quantized_pooling(data=qdata, min_data=min_data, - max_data=max_data, kernel=kernel, - pad=pad, stride=stride, pool_type=pool_type, - global_pool=global_pool) + max_data=max_data, kernel=kernel, + pad=pad, stride=stride, pool_type=pool_type, + global_pool=global_pool, + pooling_convention=convention) pooling_int8_exe = quantized_pooling.simple_bind(ctx=mx.current_context(), grad_req='null') qarg_names = quantized_pooling.list_arguments() pooling_int8_exe.arg_dict[qarg_names[0]][:] = pooling_fp32_exe.arg_dict[arg_names[0]].astype(qdtype) @@ -266,12 +269,25 @@ def check_quantized_pooling(data_shape, kernel, pool_type, pad, stride, global_p check_quantized_pooling((3, 512, 7, 7), (7, 7), 'avg', (0, 0), (1, 1), False, qdtype) check_quantized_pooling((3, 512, 7, 7), (7, 7), 'avg', (0, 0), (1, 1), True, qdtype) + check_quantized_pooling((3, 4, 56, 56), (3, 3), 'max', (0, 0), (2, 2), False, qdtype, 'full') + check_quantized_pooling((3, 4, 56, 56), (3, 3), 'max', (0, 0), (2, 2), True, qdtype, 'full') + check_quantized_pooling((3, 512, 7, 7), (7, 7), 'avg', (0, 0), (1, 1), False, qdtype, 'full') + check_quantized_pooling((3, 512, 7, 7), (7, 7), 'avg', (0, 0), (1, 1), True, qdtype, 'full') + + @with_seed() def test_quantized_fc(): def check_quantized_fc(data_shape, num_hidden, no_bias, qdtype, flatten=True): if mx.current_context().device_type != 'gpu': - print('skipped testing quantized_fc on cpu since it is not supported yet') - return + hasMKL = False; + for key in os.environ.keys(): + if operator.eq(key, "BUILD_TAG"): + if os.environ['BUILD_TAG'].find("MKL") != -1: + hasMKL = True + break + if hasMKL == False: + print('skipped testing quantized_fc on cpu since s8u8s32 is only supported by MKL BLAS library') + return elif qdtype == 'uint8' and is_test_for_gpu(): print('skipped testing quantized_fc for gpu uint8 since it is not supported yet') return @@ -283,16 +299,16 @@ def check_quantized_fc(data_shape, num_hidden, no_bias, qdtype, flatten=True): fc_fp32_exe = fc_fp32.simple_bind(ctx=mx.current_context(), grad_req='null') if qdtype == 'uint8': data_low = 0.0 - data_high = 127.0 + data_high = 63.0 else: - data_low = -127.0 - data_high = 127.0 + data_low = -63.0 + data_high = 63.0 fc_fp32_exe.arg_dict[arg_names[0]][:] = mx.nd.random.uniform(low=data_low, high=data_high, shape=data_shape).astype('int32') - fc_fp32_exe.arg_dict[arg_names[1]][:] = mx.nd.random.uniform(low=-127.0, high=127.0, + fc_fp32_exe.arg_dict[arg_names[1]][:] = mx.nd.random.uniform(low=data_low, high=data_high, shape=arg_shapes[1]).astype('int32') if not no_bias: - fc_fp32_exe.arg_dict[arg_names[2]][:] = mx.nd.random.uniform(low=-127.0, high=127.0, + fc_fp32_exe.arg_dict[arg_names[2]][:] = mx.nd.random.uniform(low=data_low, high=data_high, shape=arg_shapes[2]).astype('int32') output = fc_fp32_exe.forward()[0] @@ -335,6 +351,10 @@ def check_quantized_fc(data_shape, num_hidden, no_bias, qdtype, flatten=True): check_quantized_fc((32, 111, 2, 2), 100, True, qdtype) check_quantized_fc((32, 512, 2, 2), 100, False, qdtype) check_quantized_fc((32, 111, 2, 2), 100, False, qdtype) + check_quantized_fc((256, 2048, 2, 2), 800, False, qdtype) + check_quantized_fc((256, 111, 2, 2), 800, False, qdtype) + check_quantized_fc((256, 2048, 2, 2), 800, True, qdtype) + check_quantized_fc((256, 111, 2, 2), 800, True, qdtype) @with_seed() def test_quantized_flatten(): @@ -398,16 +418,35 @@ def get_fp32_sym(): def get_fp32_residual(): data = mx.sym.Variable('data') - conv = mx.sym.Convolution(data=data, num_filter=4, kernel=(1,1), pad=(0,0), - no_bias=True, name='conv') - bn = mx.sym.BatchNorm(data=conv, fix_gamma=False, eps=2e-5, momentum=0.9, name='bn') - act = mx.sym.Activation(data=bn + data, act_type='relu', name='relu') - pool = mx.sym.Pooling(act, kernel=(4, 4), pool_type='avg', name='pool') - fc = mx.sym.FullyConnected(pool, num_hidden=10, flatten=True, name='fc') + conv0 = mx.sym.Convolution(data=data, num_filter=4, kernel=(1,1), pad=(0,0), + no_bias=True, name='conv0') + bn = mx.sym.BatchNorm(data=conv0, fix_gamma=False, eps=2e-5, momentum=0.9, name='bn') + act0 = mx.sym.Activation(data=bn + data, act_type='relu', name='relu0') + pool0 = mx.sym.Pooling(act0, kernel=(4, 4), pool_type='avg', name='pool0') + conv1 = mx.sym.Convolution(data=pool0, num_filter=4, kernel=(1,1), pad=(0,0), + no_bias=False, name='conv1') + act1 = mx.sym.Activation(data=conv1, act_type='relu', name='relu1') + pool1 = mx.sym.Pooling(act1, kernel=(4, 4), pool_type='avg', name='pool1') + fc = mx.sym.FullyConnected(pool1, num_hidden=10, flatten=True, name='fc') sym = mx.sym.SoftmaxOutput(fc, grad_scale=1, ignore_label=-1, multi_output=False, out_grad=False, preserve_shape=False, use_ignore=False, name='softmax') return sym +def get_fp32_sym_with_multiple_outputs(length=1): + data = mx.sym.Variable('data') + inputs = list(mx.sym.split(data, axis=0, num_outputs=length, squeeze_axis=1, name='split')) + + _conv_outs = [] + for i in range(length): + _conv_outs.append(mx.sym.Convolution(data=inputs[i], kernel=(1, 1), num_filter=16, name='conv_{0}'.format(i))) + conv_out = [mx.sym.expand_dims(i, axis=0) for i in _conv_outs] + conv_out = mx.sym.Concat(*conv_out, dim=0, name='concat') + reshape_out = mx.sym.reshape(data=conv_out, shape=((length, -1)), name='reshape') + fc_out = mx.sym.FullyConnected(reshape_out, num_hidden=10, flatten=True, name='fc') + sym= mx.sym.SoftmaxOutput(fc_out, grad_scale=1, ignore_label=-1, multi_output=False, + out_grad=False, preserve_shape=False, use_ignore=False, name='softmax') + return sym + @with_seed() def test_quantize_model(): def check_quantize_model(qdtype): @@ -439,52 +478,60 @@ def check_qsym_qdtype(qsym, qdtype): assert v['out_type'] == qdtype sym = get_fp32_sym() - mod = Module(symbol=sym) batch_size = 4 - data_shape = (batch_size, 4, 10, 10) label_shape = (batch_size, 10) - mod.bind(data_shapes=[('data', data_shape)], label_shapes=[('softmax_label', label_shape)]) - mod.init_params() - arg_params, aux_params = mod.get_params() - qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=sym, - arg_params=arg_params, - aux_params=aux_params, - ctx=mx.current_context(), - quantized_dtype=qdtype, - calib_mode='none') - check_params(arg_params, qarg_params, qsym) - check_params(aux_params, qaux_params) - - calib_data = mx.nd.random.uniform(shape=data_shape) - calib_data = NDArrayIter(data=calib_data) - calib_data = DummyIter(calib_data) - qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=sym, - arg_params=arg_params, - aux_params=aux_params, - ctx=mx.current_context(), - quantized_dtype=qdtype, - calib_mode='naive', - calib_data=calib_data, - num_calib_examples=20) - check_params(arg_params, qarg_params, qsym) - check_params(aux_params, qaux_params) - check_qsym_calibrated(qsym) - check_qsym_qdtype(qsym, qdtype) + data_shape = (batch_size, 4, 10, 10) + + length = batch_size # specify num of outputs from split op + msym = get_fp32_sym_with_multiple_outputs(length) + msym_label_shape = (length, 10) + msym_data_shape = (length, 4, 4, 10, 10) + + for s, dshape, lshape in zip((sym, msym), (data_shape, msym_data_shape), + (label_shape, msym_label_shape)): + mod = Module(symbol=s) + mod.bind(data_shapes=[('data', dshape)], label_shapes=[('softmax_label', lshape)]) + mod.init_params() + arg_params, aux_params = mod.get_params() + qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=s, + arg_params=arg_params, + aux_params=aux_params, + ctx=mx.current_context(), + quantized_dtype=qdtype, + calib_mode='none') + check_params(arg_params, qarg_params, qsym) + check_params(aux_params, qaux_params) + + calib_data = mx.nd.random.uniform(shape=dshape) + calib_data = NDArrayIter(data=calib_data, batch_size=batch_size) + calib_data = DummyIter(calib_data) + qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=s, + arg_params=arg_params, + aux_params=aux_params, + ctx=mx.current_context(), + quantized_dtype=qdtype, + calib_mode='naive', + calib_data=calib_data, + num_calib_examples=20) + check_params(arg_params, qarg_params, qsym) + check_params(aux_params, qaux_params) + check_qsym_calibrated(qsym) + check_qsym_qdtype(qsym, qdtype) for qdtype in ['int8', 'uint8']: check_quantize_model(qdtype) @with_seed() -def test_quantize_residual_unit(): +def test_quantize_model_with_forward(): def check_quantize_model(qdtype): if is_test_for_native_cpu(): - print('skipped testing quantized_residual_unit for native cpu since it is not supported yet') + print('skipped testing test_quantize_model_with_forward for native cpu since it is not supported yet') return elif qdtype == 'int8' and is_test_for_mkldnn(): - print('skipped testing quantized_residual_unit for mkldnn cpu int8 since it is not supported yet') + print('skipped testing test_quantize_model_with_forward for mkldnn cpu int8 since it is not supported yet') return elif qdtype == 'uint8' and is_test_for_gpu(): - print('skipped testing quantized_residual_unit for gpu uint8 since it is not supported yet') + print('skipped testing test_quantize_model_with_forward for gpu uint8 since it is not supported yet') return def check_params(params, qparams, qsym=None): @@ -526,46 +573,64 @@ def check_qsym_forward(qsym, qarg_params, qaux_params, data_shape, label_shape): for output in mod.get_outputs(): output.wait_to_read() - sym = get_fp32_residual() - mod = Module(symbol=sym) batch_size = 4 data_shape = (batch_size, 4, 10, 10) label_shape = (batch_size, 10) - mod.bind(data_shapes=[('data', data_shape)], label_shapes=[('softmax_label', label_shape)]) - mod.init_params() - arg_params, aux_params = mod.get_params() - excluded_sym_names = [] - if mx.current_context() == mx.cpu(): - excluded_sym_names += ['fc'] - qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=sym, - arg_params=arg_params, - aux_params=aux_params, - excluded_sym_names=excluded_sym_names, - ctx=mx.current_context(), - quantized_dtype=qdtype, - calib_mode='none') - check_params(arg_params, qarg_params, qsym) - check_params(aux_params, qaux_params) - check_qsym_forward(qsym, qarg_params, qaux_params, data_shape, label_shape) - - calib_data = mx.nd.random.uniform(shape=data_shape) - calib_data = NDArrayIter(data=calib_data) - calib_data = DummyIter(calib_data) - qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=sym, - arg_params=arg_params, - aux_params=aux_params, - excluded_sym_names=excluded_sym_names, - ctx=mx.current_context(), - quantized_dtype=qdtype, - calib_mode='naive', - calib_data=calib_data, - num_calib_examples=20) - check_params(arg_params, qarg_params, qsym) - check_params(aux_params, qaux_params) - check_qsym_calibrated(qsym) - check_qsym_qdtype(qsym, qdtype) - check_qsym_forward(qsym, qarg_params, qaux_params, data_shape, label_shape) + + length = batch_size # specify num of outputs from split op + msym = get_fp32_sym_with_multiple_outputs(length) + msym_label_shape = (length, 10) + msym_data_shape = (length, 4, 4, 10, 10) + + for s, dshape, lshape in zip((sym, msym), (data_shape, msym_data_shape), + (label_shape, msym_label_shape)): + mod = Module(symbol=s) + mod.bind(data_shapes=[('data', dshape)], label_shapes=[('softmax_label', lshape)]) + + mod.init_params() + arg_params, aux_params = mod.get_params() + excluded_names = [] + if mx.current_context() == mx.cpu(): + excluded_names += ['fc'] + excluded_names += ['concat'] + + optional_names = ['pool0'] + for skip_optional_names in [False, True]: + exclude_sym_names = [] + if skip_optional_names: + excluded_sym_names = excluded_names + else: + excluded_sym_names = excluded_names + optional_names + + qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=s, + arg_params=arg_params, + aux_params=aux_params, + excluded_sym_names=excluded_sym_names, + ctx=mx.current_context(), + quantized_dtype=qdtype, + calib_mode='none') + check_params(arg_params, qarg_params, qsym) + check_params(aux_params, qaux_params) + check_qsym_forward(qsym, qarg_params, qaux_params, dshape, lshape) + + calib_data = mx.nd.random.uniform(shape=dshape) + calib_data = NDArrayIter(data=calib_data, batch_size=batch_size) + calib_data = DummyIter(calib_data) + qsym, qarg_params, qaux_params = mx.contrib.quant.quantize_model(sym=s, + arg_params=arg_params, + aux_params=aux_params, + excluded_sym_names=excluded_sym_names, + ctx=mx.current_context(), + quantized_dtype=qdtype, + calib_mode='naive', + calib_data=calib_data, + num_calib_examples=20) + check_params(arg_params, qarg_params, qsym) + check_params(aux_params, qaux_params) + check_qsym_calibrated(qsym) + check_qsym_qdtype(qsym, qdtype) + check_qsym_forward(qsym, qarg_params, qaux_params, dshape, lshape) for qdtype in ['int8', 'uint8']: check_quantize_model(qdtype) diff --git a/tests/python/unittest/common.py b/tests/python/unittest/common.py index f98bb79df..abfba73ab 100644 --- a/tests/python/unittest/common.py +++ b/tests/python/unittest/common.py @@ -95,16 +95,17 @@ def random_seed(seed=None): random.seed(next_seed) -def assert_raises_cudnn_disabled(): +def assert_raises_cudnn_not_satisfied(min_version): def test_helper(orig_test): @make_decorator(orig_test) def test_new(*args, **kwargs): - cudnn_disabled = (os.getenv('CUDNN_OFF_TEST_ONLY') == "true") - if not cudnn_disabled or mx.context.current_context().device_type == 'cpu': + cudnn_off = os.getenv('CUDNN_OFF_TEST_ONLY') == 'true' + cudnn_env_version = os.getenv('CUDNN_VERSION', None if cudnn_off else '7.3.1') + cudnn_test_disabled = cudnn_off or cudnn_env_version < min_version + if not cudnn_test_disabled or mx.context.current_context().device_type == 'cpu': orig_test(*args, **kwargs) else: - errors = (MXNetError, RuntimeError) - assert_raises(errors, orig_test, *args, **kwargs) + assert_raises((MXNetError, RuntimeError), orig_test, *args, **kwargs) return test_new return test_helper diff --git a/tests/python/unittest/test_autograd.py b/tests/python/unittest/test_autograd.py index f4815ec49..68aa804cb 100644 --- a/tests/python/unittest/test_autograd.py +++ b/tests/python/unittest/test_autograd.py @@ -21,6 +21,7 @@ from mxnet.autograd import * from mxnet.test_utils import * from common import setup_module, with_seed, teardown +from mxnet.test_utils import EnvManager def grad_and_loss(func, argnum=None): @@ -120,8 +121,9 @@ def check_unary_func(x): autograd_assert(x, func=f_square, grad_func=f_square_grad) uniform = nd.uniform(shape=(4, 5)) stypes = ['default', 'row_sparse', 'csr'] - for stype in stypes: - check_unary_func(uniform.tostype(stype)) + with EnvManager('MXNET_STORAGE_FALLBACK_LOG_VERBOSE', '0'): + for stype in stypes: + check_unary_func(uniform.tostype(stype)) @with_seed() def test_binary_func(): @@ -138,11 +140,12 @@ def check_binary_func(x, y): uniform_x = nd.uniform(shape=(4, 5)) uniform_y = nd.uniform(shape=(4, 5)) stypes = ['default', 'row_sparse', 'csr'] - for stype_x in stypes: - for stype_y in stypes: - x = uniform_x.tostype(stype_x) - y = uniform_y.tostype(stype_y) - check_binary_func(x, y) + with EnvManager('MXNET_STORAGE_FALLBACK_LOG_VERBOSE', '0'): + for stype_x in stypes: + for stype_y in stypes: + x = uniform_x.tostype(stype_x) + y = uniform_y.tostype(stype_y) + check_binary_func(x, y) @with_seed() diff --git a/tests/python/unittest/test_contrib_operator.py b/tests/python/unittest/test_contrib_operator.py index 58728d8da..43d3db648 100644 --- a/tests/python/unittest/test_contrib_operator.py +++ b/tests/python/unittest/test_contrib_operator.py @@ -261,6 +261,7 @@ def test_multibox_target_op(): assert_array_equal(loc_mask.asnumpy(), expected_loc_mask) assert_array_equal(cls_target.asnumpy(), expected_cls_target) + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_dgl_graph.py b/tests/python/unittest/test_dgl_graph.py new file mode 100644 index 000000000..e24cf4deb --- /dev/null +++ b/tests/python/unittest/test_dgl_graph.py @@ -0,0 +1,245 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# pylint: skip-file +from __future__ import print_function +import numpy as np +import scipy as sp +import mxnet as mx +import random +import itertools +from numpy.testing import assert_allclose, assert_array_equal +from mxnet.test_utils import * +import unittest + +def check_uniform(out, num_hops, max_num_vertices): + sample_id = out[0] + sub_csr = out[1] + layer = out[2] + # check sample_id + assert (len(sample_id) == max_num_vertices+1) + num_vertices = sample_id[-1].asnumpy()[0] + # check sub_csr + sub_csr.check_format(full_check=True) + assert np.all((sub_csr.indptr[num_vertices:] == sub_csr.indptr[num_vertices]).asnumpy()) + # check layer + for data in layer[:num_vertices]: + assert(data <= num_hops) + +def check_non_uniform(out, num_hops, max_num_vertices): + sample_id = out[0] + sub_csr = out[1] + prob = out[2] + layer = out[3] + # check sample_id + assert (len(sample_id) == max_num_vertices+1) + num_vertices = sample_id[-1].asnumpy()[0] + # check sub_csr + sub_csr.check_format(full_check=True) + assert np.all((sub_csr.indptr[num_vertices:] == sub_csr.indptr[num_vertices]).asnumpy()) + # check prob + assert (len(prob) == max_num_vertices) + # check layer + for data in layer[:num_vertices]: + assert(data <= num_hops) + +def check_compact(csr, id_arr, num_nodes): + compact = mx.nd.contrib.dgl_graph_compact(csr, id_arr, graph_sizes=num_nodes, return_mapping=False) + assert compact.shape[0] == num_nodes + assert compact.shape[1] == num_nodes + assert mx.nd.sum(compact.indptr == csr.indptr[0:(num_nodes + 1)]).asnumpy() == num_nodes + 1 + sub_indices = compact.indices.asnumpy() + indices = csr.indices.asnumpy() + id_arr = id_arr.asnumpy() + for i in range(len(sub_indices)): + sub_id = sub_indices[i] + assert id_arr[sub_id] == indices[i] + +def test_uniform_sample(): + shape = (5, 5) + data_np = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], dtype=np.int64) + indices_np = np.array([1,2,3,4,0,2,3,4,0,1,3,4,0,1,2,4,0,1,2,3], dtype=np.int64) + indptr_np = np.array([0,4,8,12,16,20], dtype=np.int64) + a = mx.nd.sparse.csr_matrix((data_np, indices_np, indptr_np), shape=shape) + + seed = mx.nd.array([0,1,2,3,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 3) + check_uniform(out, num_hops=1, max_num_vertices=5) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=1, max_num_vertices=4) + assert (len(out) == 3) + check_uniform(out, num_hops=1, max_num_vertices=4) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=2, num_neighbor=1, max_num_vertices=3) + assert (len(out) == 3) + check_uniform(out, num_hops=2, max_num_vertices=3) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0,2,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 3) + check_uniform(out, num_hops=1, max_num_vertices=5) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 3) + check_uniform(out, num_hops=1, max_num_vertices=5) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=2, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 3) + check_uniform(out, num_hops=2, max_num_vertices=5) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_uniform_sample(a, seed, num_args=2, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 3) + check_uniform(out, num_hops=1, max_num_vertices=5) + num_nodes = out[0][-1].asnumpy() + assert num_nodes > 0 + assert num_nodes < len(out[0]) + check_compact(out[1], out[0], num_nodes) + +def test_non_uniform_sample(): + shape = (5, 5) + prob = mx.nd.array([0.9, 0.8, 0.2, 0.4, 0.1], dtype=np.float32) + data_np = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20], dtype=np.int64) + indices_np = np.array([1,2,3,4,0,2,3,4,0,1,3,4,0,1,2,4,0,1,2,3], dtype=np.int64) + indptr_np = np.array([0,4,8,12,16,20], dtype=np.int64) + a = mx.nd.sparse.csr_matrix((data_np, indices_np, indptr_np), shape=shape) + + seed = mx.nd.array([0,1,2,3,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 4) + check_non_uniform(out, num_hops=1, max_num_vertices=5) + + seed = mx.nd.array([0], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=1, max_num_vertices=4) + assert (len(out) == 4) + check_non_uniform(out, num_hops=1, max_num_vertices=4) + + seed = mx.nd.array([0], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=2, num_neighbor=1, max_num_vertices=4) + assert (len(out) == 4) + check_non_uniform(out, num_hops=2, max_num_vertices=4) + + seed = mx.nd.array([0,2,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 4) + check_non_uniform(out, num_hops=1, max_num_vertices=5) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 4) + check_non_uniform(out, num_hops=1, max_num_vertices=5) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=2, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 4) + check_non_uniform(out, num_hops=2, max_num_vertices=5) + + seed = mx.nd.array([0,4], dtype=np.int64) + out = mx.nd.contrib.dgl_csr_neighbor_non_uniform_sample(a, prob, seed, num_args=3, num_hops=1, num_neighbor=2, max_num_vertices=5) + assert (len(out) == 4) + check_non_uniform(out, num_hops=1, max_num_vertices=5) + +def test_edge_id(): + shape = rand_shape_2d() + data = rand_ndarray(shape, stype='csr', density=0.4) + ground_truth = np.zeros(shape, dtype=np.float32) + ground_truth -= 1.0 + indptr_np = data.indptr.asnumpy() + data_np = data.data.asnumpy() + indices_np = data.indices.asnumpy() + for i in range(shape[0]): + for j in range(indptr_np[i], indptr_np[i+1]): + idx = indices_np[j] + ground_truth[i, idx] = data_np[j] + + np_u = np.random.randint(0, shape[0], size=(5, )) + np_v = np.random.randint(0, shape[1], size=(5, )) + mx_u = mx.nd.array(np_u) + mx_v = mx.nd.array(np_v) + assert_almost_equal(mx.nd.contrib.edge_id(data, mx_u, mx_v).asnumpy(), + ground_truth[np_u, np_v], rtol=1e-5, atol=1e-6) + +def generate_graph(n): + arr = sp.sparse.random(n, n, density=0.2, format='coo') + arr.data = np.arange(0, len(arr.row), dtype=np.float32) + return arr.tocsr(), mx.nd.sparse.csr_matrix(arr.tocsr()).astype(np.int64) + +def test_subgraph(): + sp_g, g = generate_graph(100) + vertices = np.unique(np.random.randint(0, 100, size=(20))) + subgs = mx.nd.contrib.dgl_subgraph(g, mx.nd.array(vertices, dtype=np.int64), + return_mapping=True) + subgs[0].check_format() + subgs[1].check_format() + assert_array_equal(subgs[0].indptr, subgs[1].indptr) + assert_array_equal(subgs[0].indices, subgs[1].indices) + sp_subg = subgs[1].asscipy() + for i in range(len(subgs[0].indptr) - 1): + subv1 = i + v1 = vertices[subv1] + row_start = int(subgs[0].indptr[subv1].asnumpy()[0]) + row_end = int(subgs[0].indptr[subv1 + 1].asnumpy()[0]) + if row_start >= len(subgs[0].indices): + remain = subgs[0].indptr[subv1:].asnumpy() + assert np.sum(remain == row_start) == len(remain) + break + row = subgs[0].indices[row_start:row_end] + for j, subv2 in enumerate(row.asnumpy()): + v2 = vertices[subv2] + assert sp_g[v1, v2] == sp_subg[subv1, subv2] + +def test_adjacency(): + sp_g, g = generate_graph(100) + adj = mx.nd.contrib.dgl_adjacency(g) + assert adj.dtype == np.float32 + assert adj.shape == g.shape + assert_array_equal(adj.indptr, g.indptr) + assert_array_equal(adj.indices, g.indices) + assert_array_equal(adj.data, mx.nd.ones(shape=g.indices.shape)) + +if __name__ == "__main__": + import nose + nose.runmodule() diff --git a/tests/python/unittest/test_gluon.py b/tests/python/unittest/test_gluon.py index 84c3f245e..b9f02178c 100644 --- a/tests/python/unittest/test_gluon.py +++ b/tests/python/unittest/test_gluon.py @@ -23,7 +23,8 @@ from mxnet.gluon import nn from mxnet.test_utils import assert_almost_equal from mxnet.ndarray.ndarray import _STORAGE_TYPE_STR_TO_ID -from common import setup_module, with_seed, assertRaises, teardown, assert_raises_cudnn_disabled +from common import (setup_module, with_seed, assertRaises, teardown, + assert_raises_cudnn_not_satisfied) import numpy as np from numpy.testing import assert_array_equal from nose.tools import raises, assert_raises @@ -339,7 +340,7 @@ def hybrid_forward(self, F, x): net.hybridize() assert isinstance(net(mx.nd.zeros((16, 10))), mx.nd.NDArray) - # Test case to verify if initializing the SymbolBlock from a model with params + # Test case to verify if initializing the SymbolBlock from a model with params # other than fp32 param dtype. # 1. Load a resnet model, cast it to fp64 and export @@ -1322,7 +1323,7 @@ def record_name(block): @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_summary(): net = gluon.model_zoo.vision.resnet50_v1() net.initialize() @@ -1933,7 +1934,6 @@ def hybrid_forward(self, F, x): @with_seed() -@unittest.skip('Test failing, tracked by https://github.com/apache/incubator-mxnet/issues/12715') def test_slice_batchnorm(): class Net(gluon.HybridBlock): def __init__(self, slice, **kwargs): @@ -2009,7 +2009,6 @@ def hybrid_forward(self, F, x): @with_seed() -@unittest.skip('Flaky test: https://github.com/apache/incubator-mxnet/issues/12767') def test_slice_batchnorm_reshape_batchnorm(): class Net(gluon.HybridBlock): def __init__(self, shape, slice, **kwargs): @@ -2414,7 +2413,7 @@ def hybrid_forward(self, F, x): x_reshape = x.reshape(self.reshape) out = self.act(x_reshape) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for act in acts: x = mx.nd.random.uniform(-1, 1, shape=(4, 16, 32, 32)) shape = (4, 32, 32, -1) @@ -2436,7 +2435,7 @@ def hybrid_forward(self, F, x): out = self.act(x_slice) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for act in acts: x = mx.nd.random.uniform(-1, 1, shape=(8, 32, 64, 64)) slice = [(0, 16, 32, 32), (4, 32, 64, 64)] @@ -2460,7 +2459,7 @@ def hybrid_forward(self, F, x): y_reshape = y.reshape(self.reshape[1]) out = self.act1(y_reshape) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for idx0, act0 in enumerate(acts): for idx1, act1 in enumerate(acts): if idx1 == idx0: @@ -2487,7 +2486,7 @@ def hybrid_forward(self, F, x): y_slice = y.slice(begin=self.slice[1][0], end=self.slice[1][1]) out = self.act1(y_slice) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for idx0, act0 in enumerate(acts): for idx1, act1 in enumerate(acts): if idx1 == idx0: @@ -2515,7 +2514,7 @@ def hybrid_forward(self, F, x): y_slice = y.slice(begin=self.slice[0], end=self.slice[1]) out = self.act1(y_slice) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for idx0, act0 in enumerate(acts): for idx1, act1 in enumerate(acts): if idx1 == idx0: @@ -2544,7 +2543,7 @@ def hybrid_forward(self, F, x): y_reshape = y.reshape(self.reshape) out = self.act1(y_reshape) return out - acts = ["relu", "sigmoid", "tanh", "softrelu"] + acts = ["relu", "sigmoid", "tanh", "softrelu", "softsign"] for idx0, act0 in enumerate(acts): for idx1, act1 in enumerate(acts): if idx1 == idx0: diff --git a/tests/python/unittest/test_gluon_data.py b/tests/python/unittest/test_gluon_data.py index e4206095f..353a819dd 100644 --- a/tests/python/unittest/test_gluon_data.py +++ b/tests/python/unittest/test_gluon_data.py @@ -77,6 +77,10 @@ def _dataset_transform_fn(x, y): """Named transform function since lambda function cannot be pickled.""" return x, y +def _dataset_transform_first_fn(x): + """Named transform function since lambda function cannot be pickled.""" + return x + @with_seed() def test_recordimage_dataset_with_data_loader_multiworker(): recfile = prepare_record() @@ -95,17 +99,13 @@ def test_recordimage_dataset_with_data_loader_multiworker(): assert x.shape[0] == 1 and x.shape[3] == 3 assert y.asscalar() == i - # try limit recursion depth - import sys - old_limit = sys.getrecursionlimit() - sys.setrecursionlimit(500) # this should be smaller than any default value used in python - dataset = gluon.data.vision.ImageRecordDataset(recfile).transform(_dataset_transform_fn) + # with transform_first + dataset = gluon.data.vision.ImageRecordDataset(recfile).transform_first(_dataset_transform_first_fn) loader = gluon.data.DataLoader(dataset, 1, num_workers=5) for i, (x, y) in enumerate(loader): assert x.shape[0] == 1 and x.shape[3] == 3 assert y.asscalar() == i - sys.setrecursionlimit(old_limit) @with_seed() def test_sampler(): @@ -156,9 +156,10 @@ def __getitem__(self, key): @with_seed() def test_multi_worker(): data = Dataset() - loader = gluon.data.DataLoader(data, batch_size=1, num_workers=5) - for i, batch in enumerate(loader): - assert (batch.asnumpy() == i).all() + for thread_pool in [True, False]: + loader = gluon.data.DataLoader(data, batch_size=1, num_workers=5, thread_pool=thread_pool) + for i, batch in enumerate(loader): + assert (batch.asnumpy() == i).all() class _Dummy(Dataset): """Dummy dataset for randomized shape arrays.""" @@ -244,6 +245,17 @@ def test_multi_worker_forked_data_loader(): for i, data in enumerate(loader): pass +@with_seed() +def test_multi_worker_dataloader_release_pool(): + # will trigger too many open file if pool is not released properly + for _ in range(100): + A = np.random.rand(999, 2000) + D = mx.gluon.data.DataLoader(A, batch_size=8, num_workers=8) + the_iter = iter(D) + next(the_iter) + del the_iter + del D + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_gluon_rnn.py b/tests/python/unittest/test_gluon_rnn.py index c1d5f6a59..edc43d21b 100644 --- a/tests/python/unittest/test_gluon_rnn.py +++ b/tests/python/unittest/test_gluon_rnn.py @@ -22,7 +22,7 @@ from numpy.testing import assert_allclose import unittest from mxnet.test_utils import almost_equal, assert_almost_equal -from common import assert_raises_cudnn_disabled +from common import assert_raises_cudnn_not_satisfied def test_rnn(): @@ -71,7 +71,7 @@ def test_lstm_forget_bias(): assert_allclose(mod.get_params()[0][bias_argument].asnumpy(), expected_bias) -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_lstm_cpu_inference(): # should behave the same as lstm cell EXPECTED_LSTM_OUTPUT = np.array([[[0.72045636, 0.72045636, 0.95215213, 0.95215213], @@ -243,7 +243,7 @@ def test_bidirectional(): assert outs == [(10, 200), (10, 200), (10, 200)] -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_layer_bidirectional(): class RefBiLSTM(gluon.Block): def __init__(self, size, **kwargs): @@ -278,7 +278,7 @@ def forward(self, inpt): net_params[k].set_data(weights[k]) ref_net_params[k.replace('l0', 'l0l0').replace('r0', 'r0l0')].set_data(weights[k]) - data = mx.random.uniform(shape=(3, 10, in_size)) + data = mx.random.uniform(shape=(11, 10, in_size)) assert_allclose(net(data).asnumpy(), ref_net(data).asnumpy()) @@ -467,7 +467,7 @@ def check_rnn_layer_forward(layer, inputs, states=None, run_only=False): mx.test_utils.assert_almost_equal(np_dx, inputs.grad.asnumpy(), rtol=1e-3, atol=1e-5) -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnn_layers(): check_rnn_layer_forward(gluon.rnn.RNN(10, 2), mx.nd.ones((8, 3, 20))) check_rnn_layer_forward(gluon.rnn.RNN(10, 2, bidirectional=True), mx.nd.ones((8, 3, 20)), mx.nd.ones((4, 3, 10))) @@ -490,12 +490,11 @@ def test_rnn_layers(): check_rnn_layer_forward(gluon.rnn.GRU(10, 2, bidirectional=True, dropout=0.5), mx.nd.ones((8, 3, 20)), mx.nd.ones((4, 3, 10)), run_only=True) - net = gluon.nn.HybridSequential() + net = gluon.nn.Sequential() net.add(gluon.rnn.LSTM(10, bidirectional=True)) net.add(gluon.nn.BatchNorm(axis=2)) net.add(gluon.nn.Flatten()) net.add(gluon.nn.Dense(3, activation='relu')) - net.hybridize() net.collect_params().initialize() with mx.autograd.record(): net(mx.nd.ones((2, 3, 10))).backward() @@ -592,14 +591,43 @@ def test_cell_fill_shape(): assert cell.i2h_weight.shape[1] == 7, cell.i2h_weight.shape[1] -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_layer_fill_shape(): layer = gluon.rnn.LSTM(10) + layer.hybridize() check_rnn_layer_forward(layer, mx.nd.ones((3, 2, 7))) print(layer) assert layer.l0_i2h_weight.shape[1] == 7, layer.l0_i2h_weight.shape[1] +def test_bidirectional_unroll_valid_length(): + # Test BidirectionalCell. + # In 1.3.1 version, after hybridize( ), BidirectionalCell would failed when pass valid_length to unroll( ). + class BiLSTM(gluon.nn.HybridBlock): + def __init__(self, rnn_size, time_step, **kwargs): + super(BiLSTM, self).__init__(**kwargs) + self.time_step = time_step + with self.name_scope(): + self.bi_lstm = gluon.rnn.BidirectionalCell( + gluon.rnn.LSTMCell(rnn_size, prefix='rnn_l0_'), + gluon.rnn.LSTMCell(rnn_size, prefix='rnn_r0_'), + output_prefix='lstm_bi_') + + def hybrid_forward(self, F, inputs, valid_len): + outputs, states = self.bi_lstm.unroll(self.time_step, inputs, valid_length=valid_len, + layout='NTC', merge_outputs=True) + return outputs, states + + rnn_size, time_step = 100, 3 + net = BiLSTM(rnn_size, time_step) + net.initialize() + net.hybridize() + inputs_data = mx.nd.random.uniform(shape=(10, 3, 50)) + valid_len = mx.nd.array([1]*10) + outputs, _ = net(inputs_data, valid_len) + assert outputs.shape == (10, 3, 200) + + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_gluon_trainer.py b/tests/python/unittest/test_gluon_trainer.py index b513016a6..39ee57e6b 100644 --- a/tests/python/unittest/test_gluon_trainer.py +++ b/tests/python/unittest/test_gluon_trainer.py @@ -55,16 +55,15 @@ def dict_equ(a, b): y.backward() trainer.step(1) + assert trainer._optimizer.param_dict == trainer._optimizer.param_dict assert (x.data(mx.cpu(1)).asnumpy() == -2).all() x.lr_mult = 0.5 - with mx.autograd.record(): for w in x.list_data(): y = w + 1 y.backward() trainer.step(1) - assert (x.data(mx.cpu(1)).asnumpy() == -4).all() trainer.save_states('test_trainer.states') @@ -74,12 +73,13 @@ def dict_equ(a, b): if trainer._update_on_kvstore: dict_equ(trainer._kvstore._updater.states, states) assert trainer._optimizer == trainer._kvstore._updater.optimizer + # invalid usage of update and allreduce_grads if update_on_kvstore + assert_raises(AssertionError, trainer.update, 1) + assert_raises(AssertionError, trainer.allreduce_grads) else: for updater in trainer._updaters: dict_equ(updater.states, states) assert trainer._optimizer == trainer._updaters[0].optimizer - assert_raises(AssertionError, trainer.update, 1) - assert_raises(AssertionError, trainer.allreduce_grads) x = gluon.Parameter('x', shape=(10,)) x.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') @@ -211,8 +211,10 @@ def check_trainer_reset_kv(kv): # load would reset kvstore mx.nd.waitall() params.load('test_trainer_reset_kv.params') - assert trainer._kvstore is None - assert trainer._kv_initialized is False + if trainer._update_on_kvstore: + # drop kvstore state if new parameters are loaded + assert trainer._kvstore is None + assert trainer._kv_initialized is False with mx.autograd.record(): for w in x.list_data(): y = w + 1 @@ -227,28 +229,74 @@ def check_trainer_reset_kv(kv): @with_seed() def test_trainer_sparse_kv(): - def check_trainer_sparse_kv(kv, stype, grad_stype, update_on_kv): + def check_trainer_sparse_kv(kv, stype, grad_stype, update_on_kv, expected): params = gluon.ParameterDict() x = params.get('x', shape=(10,1), lr_mult=1.0, stype=stype, grad_stype=grad_stype) params.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') - trainer = gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, kvstore=kv) + trainer = gluon.Trainer(params, 'sgd', {'learning_rate': 0.1}, + kvstore=kv, update_on_kvstore=update_on_kv) all_rows = mx.nd.arange(0, 10, ctx=mx.cpu(0)) - ws = x.list_data() if stype == 'default' else x.list_row_sparse_data(all_rows) + try: + ws = x.list_data() if stype == 'default' else x.list_row_sparse_data(all_rows) + with mx.autograd.record(): + for w in ws: + y = w + 1 + y.backward() + trainer.step(1) + assert trainer._kvstore.type == kv + assert trainer._kv_initialized + assert trainer._update_on_kvstore is expected + # the updated parameter should be based on the loaded checkpoint + mx.nd.waitall() + updated_w = x.data(mx.cpu(0)) if stype == 'default' else x.row_sparse_data(all_rows) + assert (updated_w == -0.2).asnumpy().all() + except Exception as err: + assert isinstance(err, expected) + + kvs = ['local', 'device'] + for kv in kvs: + check_trainer_sparse_kv(kv, 'default', 'default', True, True) + check_trainer_sparse_kv(kv, 'default', 'default', False, False) + check_trainer_sparse_kv(kv, 'default', 'default', None, True) + check_trainer_sparse_kv(kv, 'default', 'row_sparse', None, False) + check_trainer_sparse_kv(kv, 'default', 'row_sparse', True, True) + check_trainer_sparse_kv(kv, 'default', 'row_sparse', False, False) + check_trainer_sparse_kv(kv, 'row_sparse', 'row_sparse', None, True) + check_trainer_sparse_kv(kv, 'row_sparse', 'row_sparse', False, ValueError) + +@with_seed() +def test_trainer_lr_sched(): + x = gluon.Parameter('x', shape=(10,)) + x.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') + freq = 2 + factor = 0.1 + lr = 1 + lr_sched = mx.lr_scheduler.FactorScheduler(freq, factor=factor, base_lr=lr) + trainer = gluon.Trainer([x], 'sgd', {'learning_rate': lr, 'lr_scheduler': lr_sched}) + for i in range(10): with mx.autograd.record(): - for w in ws: + for w in x.list_data(): y = w + 1 y.backward() trainer.step(1) - assert trainer._kvstore.type == kv - assert trainer._kv_initialized - assert trainer._update_on_kvstore is update_on_kv - # the updated parameter should be based on the loaded checkpoint - mx.nd.waitall() - updated_w = x.data(mx.cpu(0)) if stype == 'default' else x.row_sparse_data(all_rows) - assert (updated_w == -0.2).asnumpy().all() + if i % freq == 0: + assert trainer.learning_rate == lr, (lr, trainer.learning_rate, i) + lr *= factor + mx.nd.waitall() - kvs = ['local', 'device'] - for kv in kvs: - check_trainer_sparse_kv(kv, 'default', 'default', True) - check_trainer_sparse_kv(kv, 'default', 'row_sparse', False) - check_trainer_sparse_kv(kv, 'row_sparse', 'row_sparse', True) +@with_seed() +def test_trainer_invalid_lr_sched(): + x = gluon.Parameter('x', shape=(10,)) + x.initialize(ctx=[mx.cpu(0), mx.cpu(1)], init='zeros') + freq = 2 + factor = 0.1 + lr = 1 + lr_sched = mx.lr_scheduler.FactorScheduler(freq, factor=factor, base_lr=lr) + invalid_trainer = gluon.Trainer([x], 'sgd', {'learning_rate': lr, 'lr_scheduler': lr_sched}, + update_on_kvstore=False) + with mx.autograd.record(): + for w in x.list_data(): + y = w + 1 + y.backward() + assert_raises(ValueError, invalid_trainer.step, 1) + mx.nd.waitall() diff --git a/tests/python/unittest/test_image.py b/tests/python/unittest/test_image.py index b4ca4af0e..c8ff53f35 100644 --- a/tests/python/unittest/test_image.py +++ b/tests/python/unittest/test_image.py @@ -25,6 +25,7 @@ from nose.tools import raises + def _get_data(url, dirname): import os, tarfile download(url, dirname=dirname, overwrite=False) @@ -50,6 +51,62 @@ def _generate_objects(): label = np.hstack((cid[:, np.newaxis], boxes)).ravel().tolist() return [2, 5] + label +def _test_imageiter_last_batch(imageiter_list, assert_data_shape): + test_iter = imageiter_list[0] + # test batch data shape + for _ in range(3): + for batch in test_iter: + assert batch.data[0].shape == assert_data_shape + test_iter.reset() + # test last batch handle(discard) + test_iter = imageiter_list[1] + i = 0 + for batch in test_iter: + i += 1 + assert i == 5 + # test last_batch_handle(pad) + test_iter = imageiter_list[2] + i = 0 + for batch in test_iter: + if i == 0: + first_three_data = batch.data[0][:2] + if i == 5: + last_three_data = batch.data[0][1:] + i += 1 + assert i == 6 + assert np.array_equal(first_three_data.asnumpy(), last_three_data.asnumpy()) + # test last_batch_handle(roll_over) + test_iter = imageiter_list[3] + i = 0 + for batch in test_iter: + if i == 0: + first_image = batch.data[0][0] + i += 1 + assert i == 5 + test_iter.reset() + first_batch_roll_over = test_iter.next() + assert np.array_equal( + first_batch_roll_over.data[0][1].asnumpy(), first_image.asnumpy()) + assert first_batch_roll_over.pad == 2 + # test iteratopr work properly after calling reset several times when last_batch_handle is roll_over + for _ in test_iter: + pass + test_iter.reset() + first_batch_roll_over_twice = test_iter.next() + assert np.array_equal( + first_batch_roll_over_twice.data[0][2].asnumpy(), first_image.asnumpy()) + assert first_batch_roll_over_twice.pad == 1 + # we've called next once + i = 1 + for _ in test_iter: + i += 1 + # test the third epoch with size 6 + assert i == 6 + # test shuffle option for sanity test + test_iter = imageiter_list[4] + for _ in test_iter: + pass + class TestImage(unittest.TestCase): IMAGES_URL = "http://data.mxnet.io/data/test_images.tar.gz" @@ -84,7 +141,7 @@ def test_imdecode(self): try: import cv2 except ImportError: - return + raise unittest.SkipTest("Unable to import cv2.") for img in TestImage.IMAGES: with open(img, 'rb') as fp: str_image = fp.read() @@ -92,16 +149,38 @@ def test_imdecode(self): cv_image = cv2.imread(img) assert_almost_equal(image.asnumpy(), cv_image) + def test_imdecode_bytearray(self): + try: + import cv2 + except ImportError: + return + for img in TestImage.IMAGES: + with open(img, 'rb') as fp: + str_image = bytearray(fp.read()) + image = mx.image.imdecode(str_image, to_rgb=0) + cv_image = cv2.imread(img) + assert_almost_equal(image.asnumpy(), cv_image) + + @raises(mx.base.MXNetError) + def test_imdecode_empty_buffer(self): + mx.image.imdecode(b'', to_rgb=0) + + @raises(mx.base.MXNetError) + def test_imdecode_invalid_image(self): + image = mx.image.imdecode(b'clearly not image content') + assert_equal(image, None) + def test_scale_down(self): assert mx.image.scale_down((640, 480), (720, 120)) == (640, 106) assert mx.image.scale_down((360, 1000), (480, 500)) == (360, 375) assert mx.image.scale_down((300, 400), (0, 0)) == (0, 0) + @with_seed() def test_resize_short(self): try: import cv2 except ImportError: - return + raise unittest.SkipTest("Unable to import cv2") for img in TestImage.IMAGES: cv_img = cv2.imread(img) mx_img = mx.nd.array(cv_img[:, :, (2, 1, 0)]) @@ -118,6 +197,25 @@ def test_resize_short(self): mx_resized = mx.image.resize_short(mx_img, new_size, interp) assert_almost_equal(mx_resized.asnumpy()[:, :, (2, 1, 0)], cv_resized, atol=3) + @with_seed() + def test_imresize(self): + try: + import cv2 + except ImportError: + raise unittest.SkipTest("Unable to import cv2") + for img in TestImage.IMAGES: + cv_img = cv2.imread(img) + mx_img = mx.nd.array(cv_img[:, :, (2, 1, 0)]) + new_h = np.random.randint(1, 1000) + new_w = np.random.randint(1, 1000) + for interp_val in range(0, 2): + cv_resized = cv2.resize(cv_img, (new_w, new_h), interpolation=interp_val) + mx_resized = mx.image.imresize(mx_img, new_w, new_h, interp=interp_val) + assert_almost_equal(mx_resized.asnumpy()[:, :, (2, 1, 0)], cv_resized, atol=3) + out_img = mx.nd.zeros((new_h, new_w, 3), dtype=mx_img.dtype) + mx.image.imresize(mx_img, new_w, new_h, interp=interp_val, out=out_img) + assert_almost_equal(out_img.asnumpy()[:, :, (2, 1, 0)], cv_resized, atol=3) + def test_color_normalize(self): for _ in range(10): mean = np.random.rand(3) * 255 @@ -131,86 +229,57 @@ def test_color_normalize(self): @unittest.skip("The test fail with python errors. Temporarily disabled till it gets fixed") def test_imageiter(self): - def check_imageiter(dtype='float32'): - im_list = [[np.random.randint(0, 5), x] for x in TestImage.IMAGES] - fname = './data/test_imageiter.lst' - file_list = ['\t'.join([str(k), str(np.random.randint(0, 5)), x]) - for k, x in enumerate(TestImage.IMAGES)] - with open(fname, 'w') as f: - for line in file_list: - f.write(line + '\n') - - test_list = ['imglist', 'path_imglist'] + im_list = [[np.random.randint(0, 5), x] for x in TestImage.IMAGES] + fname = './data/test_imageiter.lst' + file_list = ['\t'.join([str(k), str(np.random.randint(0, 5)), x]) + for k, x in enumerate(TestImage.IMAGES)] + with open(fname, 'w') as f: + for line in file_list: + f.write(line + '\n') + test_list = ['imglist', 'path_imglist'] + for dtype in ['int32', 'float32', 'int64', 'float64']: for test in test_list: imglist = im_list if test == 'imglist' else None path_imglist = fname if test == 'path_imglist' else None - - test_iter = mx.image.ImageIter(2, (3, 224, 224), label_width=1, imglist=imglist, - path_imglist=path_imglist, path_root='', dtype=dtype) - # test batch data shape - for _ in range(3): - for batch in test_iter: - assert batch.data[0].shape == (2, 3, 224, 224) - test_iter.reset() - # test last batch handle(discard) - test_iter = mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, - path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='discard') - i = 0 - for batch in test_iter: - i += 1 - assert i == 5 - # test last_batch_handle(pad) - test_iter = mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, - path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='pad') - i = 0 - for batch in test_iter: - if i == 0: - first_three_data = batch.data[0][:2] - if i == 5: - last_three_data = batch.data[0][1:] - i += 1 - assert i == 6 - assert np.array_equal(first_three_data.asnumpy(), last_three_data.asnumpy()) - # test last_batch_handle(roll_over) - test_iter = mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, - path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='roll_over') - i = 0 - for batch in test_iter: - if i == 0: - first_image = batch.data[0][0] - i += 1 - assert i == 5 - test_iter.reset() - first_batch_roll_over = test_iter.next() - assert np.array_equal( - first_batch_roll_over.data[0][1].asnumpy(), first_image.asnumpy()) - assert first_batch_roll_over.pad == 2 - # test iteratopr work properly after calling reset several times when last_batch_handle is roll_over - for _ in test_iter: - pass - test_iter.reset() - first_batch_roll_over_twice = test_iter.next() - assert np.array_equal( - first_batch_roll_over_twice.data[0][2].asnumpy(), first_image.asnumpy()) - assert first_batch_roll_over_twice.pad == 1 - # we've called next once - i = 1 - for _ in test_iter: - i += 1 - # test the third epoch with size 6 - assert i == 6 - # test shuffle option for sanity test - test_iter = mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, shuffle=True, - path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='pad') - for _ in test_iter: - pass - - for dtype in ['int32', 'float32', 'int64', 'float64']: - check_imageiter(dtype) + imageiter_list = [ + mx.image.ImageIter(2, (3, 224, 224), label_width=1, imglist=imglist, + path_imglist=path_imglist, path_root='', dtype=dtype), + mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, + path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='discard'), + mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, + path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='pad'), + mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, + path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='roll_over'), + mx.image.ImageIter(3, (3, 224, 224), label_width=1, imglist=imglist, shuffle=True, + path_imglist=path_imglist, path_root='', dtype=dtype, last_batch_handle='pad') + ] + _test_imageiter_last_batch(imageiter_list, (2, 3, 224, 224)) - # test with default dtype - check_imageiter() + @with_seed() + def test_copyMakeBorder(self): + try: + import cv2 + except ImportError: + raise unittest.SkipTest("Unable to import cv2") + for img in TestImage.IMAGES: + cv_img = cv2.imread(img) + mx_img = mx.nd.array(cv_img) + top = np.random.randint(1, 10) + bot = np.random.randint(1, 10) + left = np.random.randint(1, 10) + right = np.random.randint(1, 10) + new_h, new_w, _ = mx_img.shape + new_h += top + bot + new_w += left + right + val = [np.random.randint(1, 255)] * 3 + for type_val in range(0, 5): + cv_border = cv2.copyMakeBorder(cv_img, top, bot, left, right, borderType=type_val, value=val) + mx_border = mx.image.copyMakeBorder(mx_img, top, bot, left, right, type=type_val, values=val) + assert_almost_equal(mx_border.asnumpy(), cv_border) + out_img = mx.nd.zeros((new_h , new_w, 3), dtype=mx_img.dtype) + mx.image.copyMakeBorder(mx_img, top, bot, left, right, type=type_val, values=val, out=out_img) + assert_almost_equal(out_img.asnumpy(), cv_border) @with_seed() def test_augmenters(self): @@ -240,14 +309,20 @@ def test_image_detiter(self): im_list = [_generate_objects() + [x] for x in TestImage.IMAGES] det_iter = mx.image.ImageDetIter(2, (3, 300, 300), imglist=im_list, path_root='') for _ in range(3): - for batch in det_iter: + for _ in det_iter: pass - det_iter.reset() - + det_iter.reset() val_iter = mx.image.ImageDetIter(2, (3, 300, 300), imglist=im_list, path_root='') det_iter = val_iter.sync_label_shape(det_iter) + assert det_iter.data_shape == val_iter.data_shape + assert det_iter.label_shape == val_iter.label_shape + + # test batch_size is not divisible by number of images + det_iter = mx.image.ImageDetIter(4, (3, 300, 300), imglist=im_list, path_root='') + for _ in det_iter: + pass - # test file list + # test file list with last batch handle fname = './data/test_imagedetiter.lst' im_list = [[k] + _generate_objects() + [x] for k, x in enumerate(TestImage.IMAGES)] with open(fname, 'w') as f: @@ -255,10 +330,19 @@ def test_image_detiter(self): line = '\t'.join([str(k) for k in line]) f.write(line + '\n') - det_iter = mx.image.ImageDetIter(2, (3, 400, 400), path_imglist=fname, - path_root='') - for batch in det_iter: - pass + imageiter_list = [ + mx.image.ImageDetIter(2, (3, 400, 400), + path_imglist=fname, path_root=''), + mx.image.ImageDetIter(3, (3, 400, 400), + path_imglist=fname, path_root='', last_batch_handle='discard'), + mx.image.ImageDetIter(3, (3, 400, 400), + path_imglist=fname, path_root='', last_batch_handle='pad'), + mx.image.ImageDetIter(3, (3, 400, 400), + path_imglist=fname, path_root='', last_batch_handle='roll_over'), + mx.image.ImageDetIter(3, (3, 400, 400), shuffle=True, + path_imglist=fname, path_root='', last_batch_handle='pad') + ] + _test_imageiter_last_batch(imageiter_list, (2, 3, 400, 400)) @unittest.skip("The test fail with python errors. Temporarily disabled till it gets fixed") def test_det_augmenters(self): diff --git a/julia/deps/cpcblas.sh b/tests/python/unittest/test_libinfo.py old mode 100755 new mode 100644 similarity index 75% rename from julia/deps/cpcblas.sh rename to tests/python/unittest/test_libinfo.py index 99342897a..66bf03111 --- a/julia/deps/cpcblas.sh +++ b/tests/python/unittest/test_libinfo.py @@ -1,5 +1,3 @@ -#!/bin/sh - # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -17,9 +15,16 @@ # specific language governing permissions and limitations # under the License. +import os +import mxnet as mx +from mxnet import libinfo -# be invoked from build.jl +def test_include_path(): + incl_path = libinfo.find_include_path() + assert os.path.exists(incl_path) + assert os.path.isdir(incl_path) -set -e -diff ../../cblas.h include/cblas.h || cp -v ../../cblas.h include/cblas.h +if __name__ == '__main__': + import nose + nose.runmodule() diff --git a/tests/python/unittest/test_loss.py b/tests/python/unittest/test_loss.py index 2b062fba5..18d1ebf8f 100644 --- a/tests/python/unittest/test_loss.py +++ b/tests/python/unittest/test_loss.py @@ -349,6 +349,23 @@ def test_triplet_loss(): assert mod.score(data_iter, eval_metric=mx.metric.Loss())[0][1] < 0.05 @with_seed() +def test_cosine_loss(): + #Generating samples + input1 = mx.nd.random.randn(3, 2) + input2 = mx.nd.random.randn(3, 2) + label = mx.nd.sign(mx.nd.random.randn(input1.shape[0])) + #Calculating loss from cosine embedding loss function in Gluon + Loss = gluon.loss.CosineEmbeddingLoss() + loss = Loss(input1, input2, label) + + # Calculating the loss Numpy way + numerator = mx.nd.sum(input1 * input2, keepdims=True, axis=1) + denominator = mx.nd.sqrt(mx.nd.sum(input1**2, axis=1, keepdims=True)) \ + * mx.nd.sqrt(mx.nd.sum(input2**2, axis=1, keepdims=True)) + numpy_loss = mx.nd.where(label == 1, 1-numerator/denominator, \ + mx.nd.broadcast_maximum(mx.nd.array([0]), numerator/denominator, axis=1)) + assert_almost_equal(loss.asnumpy(), numpy_loss.asnumpy(), rtol=1e-3, atol=1e-5) + def test_poisson_nllloss(): pred = mx.nd.random.normal(shape=(3, 4)) min_pred = mx.nd.min(pred) @@ -404,6 +421,7 @@ def test_poisson_nllloss_mod(): optimizer='adam') assert mod.score(data_iter, eval_metric=mx.metric.Loss())[0][1] < 0.05 + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_metric.py b/tests/python/unittest/test_metric.py index 26277d2ac..2821c4bba 100644 --- a/tests/python/unittest/test_metric.py +++ b/tests/python/unittest/test_metric.py @@ -18,6 +18,8 @@ import mxnet as mx import numpy as np import json +from common import with_seed +from copy import deepcopy def check_metric(metric, *args, **kwargs): metric = mx.metric.create(metric, *args, **kwargs) @@ -37,6 +39,67 @@ def test_metrics(): composite = mx.metric.create(['acc', 'f1']) check_metric(composite) +def _check_global_metric(metric, *args, **kwargs): + def _create_pred_label(): + if use_same_shape: + pred = mx.nd.random.uniform(0, 1, shape=shape) + label = mx.nd.random.uniform(0, 1, shape=shape) + else: + # Make a random prediction + idx = np.random.rand(*shape).argsort(1) + pred = mx.nd.array(1 - 0.1 * idx) + # Label is half 1 and half 0 + # Setting all 0s or all 1s would make either + # MCC or F1 metrics always produce 0 + label = mx.nd.ones(shape[0]) + label[:shape[0] // 2] = 0 + return pred, label + + shape = kwargs.pop('shape', (10,10)) + use_same_shape = kwargs.pop('use_same_shape', False) + m1 = mx.metric.create(metric, *args, **kwargs) + m2 = deepcopy(m1) + # check that global stats are not reset when calling + # reset_local() + for i in range(10): + pred, label = _create_pred_label() + m1.update([label], [pred]) + m1.reset_local() + m2.update([label], [pred]) + assert m1.get_global() == m2.get() + + # check that reset_local() properly resets the local state + m1.reset_local() + m2.reset() + pred, label = _create_pred_label() + m1.update([label], [pred]) + m1.reset_local() + pred, label = _create_pred_label() + m1.update([label], [pred]) + m2.update([label], [pred]) + assert m1.get() == m2.get() + +@with_seed() +def test_global_metric(): + _check_global_metric('acc') + _check_global_metric('TopKAccuracy', top_k=3) + _check_global_metric('f1', shape=(10,2)) + _check_global_metric('f1', shape=(10,2), average='micro') + _check_global_metric('mcc', shape=(10,2)) + _check_global_metric('mcc', shape=(10,2), average='micro') + _check_global_metric('perplexity', -1) + _check_global_metric('pearsonr', use_same_shape=True) + _check_global_metric('nll_loss') + _check_global_metric('loss') + _check_global_metric('ce') + _check_global_metric('mae', use_same_shape=True) + _check_global_metric('mse', use_same_shape=True) + _check_global_metric('rmse', use_same_shape=True) + def custom_metric(label, pred): + return np.mean(np.abs(label-pred)) + _check_global_metric(custom_metric, use_same_shape=True) + _check_global_metric(['acc', 'f1'], shape=(10,2)) + def test_nll_loss(): metric = mx.metric.create('nll_loss') pred = mx.nd.array([[0.2, 0.3, 0.5], [0.6, 0.1, 0.3]]) diff --git a/tests/python/unittest/test_module.py b/tests/python/unittest/test_module.py index 9ff13a7da..adb31745a 100644 --- a/tests/python/unittest/test_module.py +++ b/tests/python/unittest/test_module.py @@ -813,6 +813,65 @@ def test_forward_types(): assert mod.predict(data1).shape == (1, 10) +def test_reference_single_batch_during_fit(): + """ + When using C++-based iterators, it's important that only a single batch is referenced at a time. Because C++ + iterators are exposed to the Python code through a C API, there is no concept of reference counting. Hence, + typically C++ iterators will deallocate a batch when next() is called on them. So, we need to make sure the Python + code only references a single batch at a time, otherwise the Python code will attempt to access freed memory, + resulting in either (a) garbage accuracy or (b) a segmentation fault. + """ + current_batch_i = None + + class MockBatch(object): + def __init__(self, i): + self.i = i + + @property + def label(self): + global current_batch_i + assert self.i == current_batch_i + + class MockTrainData(object): + def __init__(self, batches): + self._i = 0 + self._batches = batches + self.provide_data = None + self.provide_label = None + self.reset = lambda: None + + def __iter__(self): + self._i = 0 + return self + + def __next__(self): + global current_batch_i + + if self._i < self._batches: + current_batch_i = self._i + self._i += 1 + return MockBatch(current_batch_i) + raise StopIteration + + def next(self): + return self.__next__() + + mod = mx.mod.BaseModule() + + def empty_fn(*args, **kwargs): + pass + mod.bind = empty_fn + mod.init_params = empty_fn + mod.init_optimizer = empty_fn + mod.forward = empty_fn + mod.backward = empty_fn + mod.update = empty_fn + mod.update_metric = empty_fn + mod.get_params = lambda: (None, None) + + train_data = MockTrainData(batches=2) + mod.fit(train_data, num_epoch=1) + if __name__ == '__main__': import nose diff --git a/tests/python/unittest/test_ndarray.py b/tests/python/unittest/test_ndarray.py index 0044ae38f..0aa485539 100644 --- a/tests/python/unittest/test_ndarray.py +++ b/tests/python/unittest/test_ndarray.py @@ -1355,6 +1355,17 @@ def test_assign_float_value_to_ndarray(): b[0] = a[0] assert same(a, b.asnumpy()) +def test_assign_large_int_to_ndarray(): + """Test case from https://github.com/apache/incubator-mxnet/issues/11639""" + a = mx.nd.zeros((4, 1), dtype=np.int32) + a[1,0] = int(16800001) + a[2,0] = int(16800002) + b = a.asnumpy() + assert same(b[1,0], 16800001) + a = a-1 + b = a.asnumpy() + assert same(b[1,0], 16800000) + @with_seed() def test_assign_a_row_to_ndarray(): """Test case from https://github.com/apache/incubator-mxnet/issues/9976""" @@ -1495,6 +1506,48 @@ def test_dlpack(): mx.test_utils.assert_almost_equal(a_np, d_np) mx.test_utils.assert_almost_equal(a_np, e_np) +@with_seed() +def test_ndarray_is_inf(): + random_dimensions = np.random.randint(2, 5) + random_shape = [np.random.randint(2, 5) for i in range(random_dimensions)] + data = mxnet.test_utils.rand_ndarray(random_shape,'default') + data[0][0] = np.inf + data[0][1] = -np.inf + data[1][0] = np.nan + data[1][1] = 5 + output = mx.nd.contrib.isinf(data) + expected_output = np.isinf(data.asnumpy()) + np.testing.assert_equal(output.asnumpy(), expected_output.astype(int)) + # astype since numpy functions default return type is boolean array instead of int + +@with_seed() +def test_ndarray_is_finite(): + random_dimensions = np.random.randint(2, 5) + random_shape = [np.random.randint(2, 5) for i in range(random_dimensions)] + data = mxnet.test_utils.rand_ndarray(random_shape,'default') + data[0][0] = np.inf + data[0][1] = -np.inf + data[1][0] = np.nan + data[1][1] = 5 + output = mx.nd.contrib.isfinite(data) + expected_output = np.isfinite(data.asnumpy()) + np.testing.assert_equal(output.asnumpy(), expected_output.astype(int)) + # astype since numpy functions default return type is boolean array instead of int + +@with_seed() +def test_ndarray_is_nan(): + random_dimensions = np.random.randint(2, 5) + random_shape = [np.random.randint(2, 5) for i in range(random_dimensions)] + data = mxnet.test_utils.rand_ndarray(random_shape,'default') + data[0][0] = np.inf + data[0][1] = -np.inf + data[1][0] = np.nan + data[1][1] = 5 + output = mx.nd.contrib.isnan(data) + expected_output = np.isnan(data.asnumpy()) + np.testing.assert_equal(output.asnumpy(), expected_output.astype(int)) + # astype since numpy functions default return type is boolean array instead of int + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_operator.py b/tests/python/unittest/test_operator.py index 1041aefce..20d3eb8f4 100644 --- a/tests/python/unittest/test_operator.py +++ b/tests/python/unittest/test_operator.py @@ -17,6 +17,7 @@ # pylint: skip-file from __future__ import print_function +from __future__ import division import numpy as np import mxnet as mx import copy @@ -27,8 +28,9 @@ from numpy.testing import assert_allclose, assert_array_equal from mxnet.test_utils import * from mxnet.base import py_str, MXNetError, _as_list -from common import setup_module, with_seed, teardown, assert_raises_cudnn_disabled, assertRaises +from common import setup_module, with_seed, teardown, assert_raises_cudnn_not_satisfied, assertRaises import unittest +import os def check_rnn_consistency(cell1, cell2, T, N, I, H, grad_req, rtol=1e-2, atol=1e-4): dshape = (N, T, I) @@ -72,7 +74,7 @@ def check_rnn_consistency(cell1, cell2, T, N, I, H, grad_req, rtol=1e-2, atol=1e @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_lstm_sym(): T, N, I, H = 5, 32, 800, 800 fused = mx.rnn.FusedRNNCell(H, num_layers=3, mode='lstm', get_next_state=True, prefix='') @@ -86,7 +88,7 @@ def test_lstm_sym(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_lstm_bidirectional(): T, N, I, H = 5, 20, 800, 800 fused = mx.rnn.FusedRNNCell(H, num_layers=2, mode='lstm', @@ -107,7 +109,7 @@ def test_lstm_bidirectional(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_gru_sym(): T, N, I, H = 5, 32, 800, 800 fused = mx.rnn.FusedRNNCell(H, num_layers=3, mode='gru', get_next_state=True, prefix='') @@ -121,7 +123,7 @@ def test_gru_sym(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_gru_bidirectional(): T, N, I, H = 5, 20, 800, 800 @@ -144,7 +146,7 @@ def test_gru_bidirectional(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnntanh_sym(): T, N, I, H = 5, 32, 800, 800 @@ -159,7 +161,7 @@ def test_rnntanh_sym(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnntanh_bidirectional(): T, N, I, H = 5, 20, 800, 800 @@ -181,7 +183,7 @@ def test_rnntanh_bidirectional(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnnrelu_sym(): T, N, I, H = 5, 32, 200, 200 @@ -196,7 +198,7 @@ def test_rnnrelu_sym(): check_rnn_consistency(fused, stack, T, N, I, H, 'null') @with_seed() -@assert_raises_cudnn_disabled() +@assert_raises_cudnn_not_satisfied(min_version='5.1.10') def test_rnnrelu_bidirectional(): T, N, I, H = 5, 20, 200, 200 @@ -768,6 +770,7 @@ def fleaky_relu_grad(grad, x, y, act_type, slope=0.25): # NOTE(haojin2): Skipping the numeric check tests for float16 data type due to precision issues, # the analytical checks are still performed on each and every data type to verify the correctness. @with_seed() +@unittest.skip("Flaky test tracked by https://github.com/apache/incubator-mxnet/issues/12885") def test_prelu(): def fprelu(x, gamma): pos_indices = x > 0 @@ -1601,33 +1604,33 @@ def check_batchnorm_training(stype): def test_convolution_grouping(): for dim in [1, 2, 3]: num_filter = 4 - num_group = 2 - kernel = (3,) * dim - shape = (1, 4) + (9,) * dim - - x = mx.sym.Variable('x') - w = mx.sym.Variable('w') - b = mx.sym.Variable('b') - y1 = mx.sym.Convolution(data=x, weight=w, bias=b, num_filter=num_filter, num_group=num_group, kernel=kernel) - xslice = mx.sym.SliceChannel(data=x, num_outputs=num_group, axis=1) - wslice = mx.sym.SliceChannel(data=w, num_outputs=num_group, axis=0) - bslice = mx.sym.SliceChannel(data=b, num_outputs=num_group, axis=0) - y2 = mx.sym.Concat(*[mx.sym.Convolution(data=xslice[i], weight=wslice[i], bias=bslice[i], - num_filter=num_filter//num_group, kernel=kernel) - for i in range(num_group)]) - - exe1 = y1.simple_bind(default_context(), x=shape) - exe2 = y2.simple_bind(default_context(), x=shape, w=(num_filter, shape[1]//num_group) + kernel, b=(num_filter,)) - for arr1, arr2 in zip(exe1.arg_arrays, exe2.arg_arrays): - arr1[:] = np.float32(np.random.normal(size=arr1.shape)) - arr2[:] = arr1 - exe1.forward(is_train=True) - exe1.backward(exe1.outputs[0]) - exe2.forward(is_train=True) - exe2.backward(exe2.outputs[0]) - - for arr1, arr2 in zip(exe1.outputs + exe1.grad_arrays, exe2.outputs + exe2.grad_arrays): - np.testing.assert_allclose(arr1.asnumpy(), arr2.asnumpy(), rtol=1e-3, atol=1e-3) + for num_group in [1, 2]: + kernel = (3,) * dim + shape = (1, 4) + (9,) * dim + + x = mx.sym.Variable('x') + w = mx.sym.Variable('w') + b = mx.sym.Variable('b') + y1 = mx.sym.Convolution(data=x, weight=w, bias=b, num_filter=num_filter, num_group=num_group, kernel=kernel) + xslice = mx.sym.SliceChannel(data=x, num_outputs=num_group, axis=1) + wslice = mx.sym.SliceChannel(data=w, num_outputs=num_group, axis=0) + bslice = mx.sym.SliceChannel(data=b, num_outputs=num_group, axis=0) + y2 = mx.sym.Concat(*[mx.sym.Convolution(data=xslice[i], weight=wslice[i], bias=bslice[i], + num_filter=num_filter//num_group, kernel=kernel) + for i in range(num_group)]) + + exe1 = y1.simple_bind(default_context(), x=shape) + exe2 = y2.simple_bind(default_context(), x=shape, w=(num_filter, shape[1]//num_group) + kernel, b=(num_filter,)) + for arr1, arr2 in zip(exe1.arg_arrays, exe2.arg_arrays): + arr1[:] = np.float32(np.random.normal(size=arr1.shape)) + arr2[:] = arr1 + exe1.forward(is_train=True) + exe1.backward(exe1.outputs[0]) + exe2.forward(is_train=True) + exe2.backward(exe2.outputs[0]) + + for arr1, arr2 in zip(exe1.outputs + exe1.grad_arrays, exe2.outputs + exe2.grad_arrays): + np.testing.assert_allclose(arr1.asnumpy(), arr2.asnumpy(), rtol=1e-3, atol=1e-3) @unittest.skip("Flaky test https://github.com/apache/incubator-mxnet/issues/12203") @@ -3496,6 +3499,10 @@ def test_special_functions_using_scipy(): mathematical_core("gammaln", lambda x: mx.sym.gammaln(x), lambda x: scipy_special.gammaln(x), lambda x: scipy_special.psi(x), 0.5, 0.5) + # erf + mathematical_core("erf", lambda x: mx.sym.erf(x), lambda x: scipy_special.erf(x), + lambda x: 2.0 / math.sqrt(math.pi) * math.exp(-(x ** 2)), 0.5, 0.5) + def rounding(name, forward_mxnet_call, forward_numpy_call, data_init=5., grad_init=2.): data = mx.symbol.Variable('data') @@ -3618,6 +3625,7 @@ def test_special_functions_using_scipy(): @with_seed() +@unittest.skip("Flaky test, tracked at https://github.com/apache/incubator-mxnet/issues/12901") def test_clip(): data = mx.symbol.Variable('data') shape = (30, 30) @@ -4384,6 +4392,7 @@ def test_1d_cond(): test_1d_cond() +@unittest.skip("Flaky test. Tracked in https://github.com/apache/incubator-mxnet/issues/13600") @with_seed() def test_softmin(): for ndim in range(1, 5): @@ -4444,6 +4453,20 @@ def test_log_softmax(): check_symbolic_forward(sym, [data], [np.log(np_softmax(data, axis=axis)+1e-20)]) check_numeric_gradient(sym, [data], rtol=0.05, atol=1e-3) +def test_softmax_with_large_inputs(): + def softmax_forward(input_data, true_output): + data = mx.sym.Variable('data') + out1 = data.softmax(axis=1) + exec1 = out1.bind(default_context(), args={'data': input_data}) + exec1.forward()[0].wait_to_read() + ndarr = exec1.outputs[0][0][0][0] + nparr = ndarr.asnumpy() + assert_almost_equal(nparr, true_output, rtol=1e-5, atol=1e-5) + + softmax_forward(mx.nd.array([[[[-1e30,-1e30]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[1e30,1e30]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[-3.4e38,-3.4e38]]]]), np.array([1.0,1.0])) + softmax_forward(mx.nd.array([[[[3.4e38,3.4e38]]]]), np.array([1.0,1.0])) @with_seed() def test_pick(): @@ -4768,24 +4791,43 @@ def test_index_copy(): x = mx.nd.zeros((5,3)) t = mx.nd.array([[1,2,3],[4,5,6],[7,8,9]]) index = mx.nd.array([0,4,2], dtype=np.int64) + tensor = mx.nd.array([[1,2,3],[0,0,0],[7,8,9],[0,0,0],[4,5,6]]) + x_grad = mx.nd.array([[0,0,0],[1,1,1],[0,0,0],[1,1,1],[0,0,0]]) + t_grad = mx.nd.array([[1,1,1],[1,1,1],[1,1,1]]) - x.attach_grad() t.attach_grad() - index.attach_grad() - with mx.autograd.record(): out = mx.nd.contrib.index_copy(x, index, t) - out.backward() - - tensor = mx.nd.array([[1,2,3],[0,0,0],[7,8,9],[0,0,0],[4,5,6]]) - x_grad = mx.nd.array([[0,0,0],[1,1,1],[0,0,0],[1,1,1],[0,0,0]]) - t_grad = mx.nd.array([[1,1,1],[1,1,1],[1,1,1]]) - index_grad = mx.nd.array([0,0,0]) + out.backward() + assert same(out.asnumpy(), tensor.asnumpy()) + assert same(t.grad.asnumpy(), t_grad.asnumpy()) + x.attach_grad() + t.attach_grad() + with mx.autograd.record(): + out = mx.nd.contrib.index_copy(x, index, t) + out.backward() assert same(out.asnumpy(), tensor.asnumpy()) assert same(x.grad.asnumpy(), x_grad.asnumpy()) assert same(t.grad.asnumpy(), t_grad.asnumpy()) - assert same(index.grad.asnumpy(), index_grad.asnumpy()) + + +@with_seed() +def test_boolean_mask(): + if default_context().device_type != 'cpu': + return + data = mx.nd.array([[1, 2, 3],[4, 5, 6],[7, 8, 9]]) + index = mx.nd.array([0, 1, 0]) + data.attach_grad() + with mx.autograd.record(): + out = mx.nd.contrib.boolean_mask(data, index) + out.backward() + data.grad.wait_to_read() + expected = np.array([[4, 5, 6]]) + expected_grad = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]]) + assert same(out.asnumpy(), expected) + assert same(data.grad.asnumpy(), expected_grad) + @with_seed() def test_div_sqrt_dim(): @@ -5218,43 +5260,10 @@ def test_deformable_psroipooling(): grad_nodes=grad_nodes, ctx=mx.gpu(0)) -# Helper functions for test_laop - -def _make_symm_symbol(a, ndims): - assert ndims >= 2 - tr_shape = list(range(ndims)) - tr_shape[-1] = ndims-2 - tr_shape[-2] = ndims-1 - tr_shape = tuple(tr_shape) - return 0.5 * (a + mx.sym.transpose(a, axes=tr_shape)) - -def _make_lower_triangle_symm(a, ndims, m, dtype=np.float32): - assert ndims >= 2 - # The last two dimensions must both be m - # Create mask for lower triangle and diagonal - index = mx.sym.arange(start=0, stop=m, step=1, dtype=np.int32) - lt_mask = mx.sym.one_hot(index, depth=m, dtype=dtype) - for j in range(1, m): - part1 = mx.sym.zeros(shape=(j, m), dtype=dtype) - index = mx.sym.arange(start=0, stop=m-j, step=1, dtype=np.int32) - part2 = mx.sym.one_hot(index, depth=m, dtype=dtype) - lt_mask = lt_mask + mx.sym.concat(*[part1, part2], dim=0) - shp = tuple([1]*(ndims-2) + [m, m]) - lt_mask = mx.sym.reshape(lt_mask, shape=shp) - return mx.sym.broadcast_mul(a, lt_mask) - -# @ankkhedia: Getting rid of fixed seed as flakiness could not be reproduced -# tracked at https://github.com/apache/incubator-mxnet/issues/11718 -@with_seed() -def test_laop(): - dtype = np.float64 - rtol_fw = 1e-7 - atol_fw = 1e-9 +def _gemm_test_helper(dtype, grad_check, rtol_fw = 1e-7, atol_fw = 1e-9): num_eps = 1e-6 rtol_bw = 1e-5 atol_bw = 1e-6 - # enable numerical checking of gradients - grad_check = 1 data1 = mx.symbol.Variable('data1') data2 = mx.symbol.Variable('data2') @@ -5269,15 +5278,14 @@ def test_laop(): rep_3x = lambda a, m, n :\ np.reshape(np.tile(np.array(a).flatten(), 3), (3, 1, m, n)) - # Test gemm separately from other la-operators. shape1 = (2, 3) shape2 = (3, 2) shape3 = (3, 3) shape4 = (2, 2) - data_in1 = np.random.uniform(1, 10, shape1) - data_in2 = np.random.uniform(1, 10, shape2) - data_in3 = np.random.uniform(1, 10, shape3) - data_in4 = np.random.uniform(1, 10, shape4) + data_in1 = np.random.uniform(1, 10, shape1).astype(dtype) + data_in2 = np.random.uniform(1, 10, shape2).astype(dtype) + data_in3 = np.random.uniform(1, 10, shape3).astype(dtype) + data_in4 = np.random.uniform(1, 10, shape4).astype(dtype) # Check all transpositions of gemm operator. data_in1_t = np.transpose(data_in1) data_in2_t = np.transpose(data_in2) @@ -5379,142 +5387,212 @@ def test_laop(): if grad_check == 1: check_grad(test_gemm, [a2, b2]) - # Now test all the other operators. +# Test gemm separately from other la-operators. +@with_seed() +def test_gemm(): + _gemm_test_helper(np.float64, True) + os.environ["MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION"] = "0" + _gemm_test_helper(np.float32, False, rtol_fw = 1e-5, atol_fw = 1e-7) + if default_context().device_type == 'gpu': + os.environ["MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION"] = "1" + _gemm_test_helper(np.float32, False, rtol_fw = 2e-5, atol_fw = 2e-7) + os.environ["MXNET_CUDA_TENSOR_OP_MATH_ALLOW_CONVERSION"] = "0" - # Tests with trivial 1x1 matrices. - shape = (4, 4, 1, 1) - data_in = np.random.uniform(1, 10, shape) - # test potrf - # Note: Have to symmetrize input, for gradient test to work - res_potrf = np.sqrt(data_in) - test_potrf = mx.sym.linalg.potrf(data1) - check_fw(test_potrf, [data_in], [res_potrf]) - if grad_check == 1: - check_grad(test_potrf, [data_in]) - # test potri - ones = mx.nd.ones(shape).asnumpy() - res_potri = np.divide(ones, data_in * data_in) - test_potri = mx.sym.linalg.potri(data1) - check_fw(test_potri, [data_in], [res_potri]) - if grad_check == 1: - check_grad(test_potri, [data_in]) - # test trsm - trian_in = data_in * 7. - test_trsm = mx.sym.linalg.trsm(data1, data2, alpha=7.) - check_fw(test_trsm, [trian_in, data_in], [ones]) - if grad_check == 1: - check_grad(test_trsm, [trian_in,data_in]) - # test trmm - trian_in = np.divide(ones, trian_in) - test_trmm = mx.sym.linalg.trmm(data1, data2, alpha=7., transpose=True, - rightside=True) - check_fw(test_trmm, [trian_in, data_in], [ones]) - if grad_check == 1: - check_grad(test_trmm, [trian_in, data_in]) - # test sumlogdiag - res_sumlogdiag = np.reshape(np.log(data_in), (4, 4)) - test_sumlogdiag = mx.sym.linalg.sumlogdiag(data1) - check_fw(test_sumlogdiag, [data_in], [res_sumlogdiag]) - if grad_check == 1: - check_grad(test_sumlogdiag, [data_in]) - - # more elaborate example of Cholesky factorization - matrix = np.array([[9., 3., -6., 12.], - [3., 26., -7., -11.], - [-6., -7., 9., 7.], - [12., -11., 7., 65.]]) - trian = np.array([[3., 0., 0., 0.], - [1., 5., 0., 0.], - [-2., -1., 2., 0.], - [4., -3., 6., 2.]]) - pow = np.array([[2., 1., 1., 1.], - [1., 4., 1., 1.], - [1., 1., 8., 1.], - [1., 1., 1., 16.]]) - inv = np.array([[8.95/3., 0.05/3., 2.65, -2.5/3.], - [0.05/3., 0.05, 0.05, 0.], - [2.65, 0.05, 2.5, -0.75], - [-2.5/3., 0., -0.75, 0.25]]) - ident = np.eye(4) - - # test potrf - test_potrf = mx.sym.linalg.potrf(_make_symm_symbol(data1, ndims=4)) - a = rep_3x(matrix, 4, 4) - r = rep_3x(trian, 4, 4) - check_fw(test_potrf, [a], [r]) - if grad_check == 1: - check_grad(test_potrf, [a]) - - #test potri - data1_ltri = _make_lower_triangle_symm( - data1, ndims=4, m=4, dtype=dtype) - test_potri = mx.sym.linalg.potri(data1_ltri) - a = rep_3x(trian, 4, 4) - r = rep_3x(inv, 4, 4) - check_fw(test_potri, [a], [r]) - if grad_check == 1: - check_grad(test_potri, [a]) - - # test trsm - test_trsm = mx.sym.linalg.trsm(data1_ltri, data2, alpha=7.) - a = rep_3x(trian, 4, 4) - b = rep_3x(matrix, 4, 4) - r = rep_3x(7. * np.transpose(trian), 4, 4) - check_fw(test_trsm, [a, b], [r]) - if grad_check == 1: - check_grad(test_trsm, [a, b]) +# Helper functions for test_laop - test_trsm2 = mx.sym.linalg.trsm( - data1_ltri, data2, alpha=-2., rightside=True, transpose=True) - r = rep_3x(-2. * trian, 4, 4) - check_fw(test_trsm2, [a, b], [r]) - if grad_check == 1: - check_grad(test_trsm2, [a, b]) +def _make_symm_symbol(a, ndims): + assert ndims >= 2 + tr_shape = list(range(ndims)) + tr_shape[-1] = ndims-2 + tr_shape[-2] = ndims-1 + tr_shape = tuple(tr_shape) + return 0.5 * (a + mx.sym.transpose(a, axes=tr_shape)) - test_trsm3 = mx.sym.linalg.trsm( - data1_ltri, data2, alpha=0.5, transpose=True) - b = rep_3x(np.transpose(trian), 4, 4) - r = rep_3x(0.5 * ident, 4, 4) - check_fw(test_trsm3, [a, b], [r]) - if grad_check == 1: - check_grad(test_trsm3, [a, b]) +def _make_triangle_symm(a, ndims, m, lower, dtype=np.float32): + assert ndims >= 2 + # The last two dimensions must both be m + # Create mask for lower triangle and diagonal + index = mx.sym.arange(start=0, stop=m, step=1, dtype=np.int32) + lt_mask = mx.sym.one_hot(index, depth=m, dtype=dtype) + for j in range(1, m): + part1 = mx.sym.zeros(shape=(j, m), dtype=dtype) + index = mx.sym.arange(start=0, stop=m-j, step=1, dtype=np.int32) + part2 = mx.sym.one_hot(index, depth=m, dtype=dtype) + lt_mask = lt_mask + mx.sym.concat(*[part1, part2], dim=0) + if not lower: + lt_mask = mx.sym.reshape(lt_mask, shape=(m, m)) + lt_mask = mx.sym.transpose(lt_mask, axes=(1, 0)) + shp = tuple([1]*(ndims-2) + [m, m]) + lt_mask = mx.sym.reshape(lt_mask, shape=shp) + return mx.sym.broadcast_mul(a, lt_mask) - test_trsm4 = mx.sym.linalg.trsm( - data1_ltri, data2, alpha=-0.5, rightside=True) - b = rep_3x(trian, 4, 4) - r = rep_3x(-0.5 * ident, 4, 4) - check_fw(test_trsm4, [a, b], [r]) - if grad_check == 1: - check_grad(test_trsm4, [a, b]) - - # test trmm - test_trmm = mx.sym.linalg.trmm( - data1_ltri, data2, alpha=7., transpose=True, rightside=True) - a = rep_3x(trian, 4, 4) - b = rep_3x(matrix, 4, 4) - r = rep_3x(7. * np.dot(matrix, trian.T), 4, 4) - check_fw(test_trmm, [a, b], [r]) - if grad_check == 1: - check_grad(test_trmm, [a, b]) +# @ankkhedia: Getting rid of fixed seed as flakiness could not be reproduced +# tracked at https://github.com/apache/incubator-mxnet/issues/11718 +@with_seed() +def test_laop(): + dtype = np.float64 + rtol_fw = 1e-7 + atol_fw = 1e-9 + num_eps = 1e-6 + rtol_bw = 1e-5 + atol_bw = 1e-6 + # enable numerical checking of gradients + grad_check = 1 - test_trmm2 = mx.sym.linalg.trmm(data1_ltri, data2, alpha=-2.) - r = rep_3x(-2. * np.dot(trian, matrix), 4, 4) - check_fw(test_trmm2, [a, b], [r]) - if grad_check == 1: - check_grad(test_trmm2, [a, b]) + data1 = mx.symbol.Variable('data1') + data2 = mx.symbol.Variable('data2') + data3 = mx.symbol.Variable('data3') - test_trmm3 = mx.sym.linalg.trmm(data1_ltri, data2, rightside=True) - r = rep_3x(np.dot(matrix, trian), 4, 4) - check_fw(test_trmm3, [a, b], [r]) - if grad_check == 1: - check_grad(test_trmm3, [a, b]) + check_fw = lambda sym, location, expected :\ + check_symbolic_forward(sym, location, expected, rtol=rtol_fw, + atol=atol_fw, dtype=dtype) + check_grad = lambda sym, location:\ + check_numeric_gradient(sym, location, numeric_eps=num_eps, rtol=rtol_bw, + atol=atol_bw, dtype=dtype) + rep_3x = lambda a, m, n :\ + np.reshape(np.tile(np.array(a).flatten(), 3), (3, 1, m, n)) - test_trmm4 = mx.sym.linalg.trmm( - data1_ltri, data2, alpha=1.2, transpose=True) - r = rep_3x(1.2 * np.dot(trian.T, matrix), 4, 4) - check_fw(test_trmm4, [a, b], [r]) - if grad_check == 1: - check_grad(test_trmm4, [a, b]) + for lower in [True, False]: + upper = not lower + + # Tests with trivial 1x1 matrices. + shape = (4, 4, 1, 1) + data_in = np.random.uniform(1, 10, shape) + # test potrf + # Note: Have to symmetrize input, for gradient test to work + res_potrf = np.sqrt(data_in) + test_potrf = mx.sym.linalg.potrf(data1, lower=lower) + check_fw(test_potrf, [data_in], [res_potrf]) + if grad_check == 1: + check_grad(test_potrf, [data_in]) + # test potri + ones = mx.nd.ones(shape).asnumpy() + res_potri = np.divide(ones, data_in * data_in) + test_potri = mx.sym.linalg.potri(data1, lower=lower) + check_fw(test_potri, [data_in], [res_potri]) + if grad_check == 1: + check_grad(test_potri, [data_in]) + # test trsm + trian_in = data_in * 7. + test_trsm = mx.sym.linalg.trsm(data1, data2, alpha=7., lower=lower) + check_fw(test_trsm, [trian_in, data_in], [ones]) + if grad_check == 1: + check_grad(test_trsm, [trian_in,data_in]) + # test trmm + trian_in = np.divide(ones, trian_in) + test_trmm = mx.sym.linalg.trmm(data1, data2, alpha=7., transpose=True, + rightside=True, lower=lower) + check_fw(test_trmm, [trian_in, data_in], [ones]) + if grad_check == 1: + check_grad(test_trmm, [trian_in, data_in]) + # test sumlogdiag + res_sumlogdiag = np.reshape(np.log(data_in), (4, 4)) + test_sumlogdiag = mx.sym.linalg.sumlogdiag(data1) + check_fw(test_sumlogdiag, [data_in], [res_sumlogdiag]) + if grad_check == 1: + check_grad(test_sumlogdiag, [data_in]) + + # more elaborate example of Cholesky factorization + matrix = np.array([[9., 3., -6., 12.], + [3., 26., -7., -11.], + [-6., -7., 9., 7.], + [12., -11., 7., 65.]]) + trian = np.array([[3., 0., 0., 0.], + [1., 5., 0., 0.], + [-2., -1., 2., 0.], + [4., -3., 6., 2.]]) + pow = np.array([[2., 1., 1., 1.], + [1., 4., 1., 1.], + [1., 1., 8., 1.], + [1., 1., 1., 16.]]) + inv = np.array([[8.95/3., 0.05/3., 2.65, -2.5/3.], + [0.05/3., 0.05, 0.05, 0.], + [2.65, 0.05, 2.5, -0.75], + [-2.5/3., 0., -0.75, 0.25]]) + ident = np.eye(4) + + low_trian = trian + if not lower: + trian = np.transpose(trian) + + # test potrf + test_potrf = mx.sym.linalg.potrf(_make_symm_symbol(data1, ndims=4), lower=lower) + a = rep_3x(matrix, 4, 4) + r = rep_3x(trian, 4, 4) + check_fw(test_potrf, [a], [r]) + if grad_check == 1: + check_grad(test_potrf, [a]) + + #test potri + data1_ltri = _make_triangle_symm( + data1, ndims=4, m=4, lower=lower, dtype=dtype) + test_potri = mx.sym.linalg.potri(data1_ltri, lower=lower) + a = rep_3x(trian, 4, 4) + r = rep_3x(inv, 4, 4) + check_fw(test_potri, [a], [r]) + if grad_check == 1: + check_grad(test_potri, [a]) + + # test trsm + test_trsm = mx.sym.linalg.trsm(data1_ltri, data2, alpha=7., transpose=upper, lower=lower) + a = rep_3x(trian, 4, 4) + b = rep_3x(matrix, 4, 4) + r = rep_3x(7. * np.transpose(low_trian), 4, 4) + check_fw(test_trsm, [a, b], [r]) + if grad_check == 1: + check_grad(test_trsm, [a, b]) + + test_trsm2 = mx.sym.linalg.trsm( + data1_ltri, data2, alpha=-2., rightside=True, transpose=lower, lower=lower) + r = rep_3x(-2. * low_trian, 4, 4) + check_fw(test_trsm2, [a, b], [r]) + if grad_check == 1: + check_grad(test_trsm2, [a, b]) + + test_trsm3 = mx.sym.linalg.trsm( + data1_ltri, data2, alpha=0.5, transpose=lower, lower=lower) + b = rep_3x(np.transpose(low_trian), 4, 4) + r = rep_3x(0.5 * ident, 4, 4) + check_fw(test_trsm3, [a, b], [r]) + if grad_check == 1: + check_grad(test_trsm3, [a, b]) + + test_trsm4 = mx.sym.linalg.trsm( + data1_ltri, data2, alpha=-0.5, rightside=True, transpose=upper, lower=lower) + b = rep_3x(low_trian, 4, 4) + r = rep_3x(-0.5 * ident, 4, 4) + check_fw(test_trsm4, [a, b], [r]) + if grad_check == 1: + check_grad(test_trsm4, [a, b]) + + # test trmm + test_trmm = mx.sym.linalg.trmm( + data1_ltri, data2, alpha=7., transpose=True, rightside=True, lower=lower) + a = rep_3x(trian, 4, 4) + b = rep_3x(matrix, 4, 4) + r = rep_3x(7. * np.dot(matrix, trian.T), 4, 4) + check_fw(test_trmm, [a, b], [r]) + if grad_check == 1: + check_grad(test_trmm, [a, b]) + + test_trmm2 = mx.sym.linalg.trmm(data1_ltri, data2, alpha=-2., lower=lower) + r = rep_3x(-2. * np.dot(trian, matrix), 4, 4) + check_fw(test_trmm2, [a, b], [r]) + if grad_check == 1: + check_grad(test_trmm2, [a, b]) + + test_trmm3 = mx.sym.linalg.trmm(data1_ltri, data2, rightside=True, lower=lower) + r = rep_3x(np.dot(matrix, trian), 4, 4) + check_fw(test_trmm3, [a, b], [r]) + if grad_check == 1: + check_grad(test_trmm3, [a, b]) + + test_trmm4 = mx.sym.linalg.trmm( + data1_ltri, data2, alpha=1.2, transpose=True, lower=lower) + r = rep_3x(1.2 * np.dot(trian.T, matrix), 4, 4) + check_fw(test_trmm4, [a, b], [r]) + if grad_check == 1: + check_grad(test_trmm4, [a, b]) # test sumlogdiag a = rep_3x(pow, 4, 4) @@ -6698,7 +6776,7 @@ def get_output_names_callback(name, arr): @with_seed() def test_activation(): - shape=(9, 10) + shapes = [(9,), (9, 10), (9, 10, 10), (1, 9, 10, 10)] dtype_l = [np.float64, np.float32, np.float16] rtol_l = [1e-7, 1e-6, 1e-2] atol_l = [1e-7, 1e-6, 1e-2] @@ -6729,17 +6807,19 @@ def test_activation(): } # Loop over operators for name, op in unary_ops.items(): - # Loop over dtype's - for ind in range(len(dtype_l)): - dtype = dtype_l[ind] - rtol = rtol_l[ind] - atol = atol_l[ind] - compare_forw_backw_unary_op( - name, op[0], op[1], op[2], shape, op[3], op[4], rtol, atol, - dtype) - # Finite difference testing - finite_diff_unary_op( - name, op[0], shape, op[3], op[4], rtol_fd, atol_fd, num_eps) + # Loop over shapes + for shape in shapes: + # Loop over dtype's + for ind in range(len(dtype_l)): + dtype = dtype_l[ind] + rtol = rtol_l[ind] + atol = atol_l[ind] + compare_forw_backw_unary_op( + name, op[0], op[1], op[2], shape, op[3], op[4], rtol, atol, + dtype) + # Finite difference testing + finite_diff_unary_op( + name, op[0], shape, op[3], op[4], rtol_fd, atol_fd, num_eps) @with_seed() def test_ravel(): @@ -6823,14 +6903,16 @@ def bilinear_interpolate(bottom, height, width, y, x): ] return val, grad - def roialign_forward_backward(data, rois, pooled_size, spatial_scale, sampling_ratio, dy): + def roialign_forward_backward(data, rois, pooled_size, spatial_scale, sampling_ratio, + position_sensitive, dy): N, C, H, W = data.shape R = rois.shape[0] PH, PW = pooled_size assert len(rois.shape) == 2 assert rois.shape[1] == 5 - out = np.zeros((R, C, PH, PW)) + C_out = C // PH // PW if position_sensitive else C + out = np.zeros((R, C_out, PH, PW)) dx = np.zeros_like(data) drois = np.zeros_like(rois) @@ -6848,24 +6930,25 @@ def roialign_forward_backward(data, rois, pooled_size, spatial_scale, sampling_r roi_bin_grid_h = int(np.ceil(roi_h * 1.0 / PH)) roi_bin_grid_w = int(np.ceil(roi_w * 1.0 / PW)) count = roi_bin_grid_h * roi_bin_grid_w - for c in range(C): + for c in range(C_out): for ph in range(PH): for pw in range(PW): val = 0.0 + c_in = c * PH * PW + ph * PW + pw if position_sensitive else c for iy in range(roi_bin_grid_h): y = sh + ph * bin_h + (iy + 0.5) * bin_h / roi_bin_grid_h for ix in range(roi_bin_grid_w): x = sw + pw * bin_w + (ix + 0.5) * bin_w / roi_bin_grid_w - v, g = bilinear_interpolate(bdata[c], H, W, y, x) + v, g = bilinear_interpolate(bdata[c_in], H, W, y, x) val += v # compute grad for qy, qx, qw in g: - dx[batch_ind, c, qy, qx] += dy[r, c, ph, pw] * qw * 1.0 / count + dx[batch_ind, c_in, qy, qx] += dy[r, c, ph, pw] * qw * 1.0 / count out[r, c, ph, pw] = val * 1.0 / count return out, [dx, drois] - def test_roi_align_value(sampling_ratio=0): + def test_roi_align_value(sampling_ratio=0, position_sensitive=False): ctx=default_context() dtype = np.float32 @@ -6874,6 +6957,7 @@ def test_roi_align_value(sampling_ratio=0): assert H == W R = 7 pooled_size = (3, 4) + C = C * pooled_size[0] * pooled_size[1] if position_sensitive else C spatial_scale = H * 1.0 / dlen data = mx.nd.array(np.arange(N*C*W*H).reshape((N,C,H,W)), ctx=ctx, dtype = dtype) @@ -6888,11 +6972,14 @@ def test_roi_align_value(sampling_ratio=0): rois.attach_grad() with mx.autograd.record(): output = mx.nd.contrib.ROIAlign(data, rois, pooled_size=pooled_size, - spatial_scale=spatial_scale, sample_ratio=sampling_ratio) - dy = mx.nd.random.uniform(-1, 1, (R, C) + pooled_size, ctx=ctx, dtype = dtype) + spatial_scale=spatial_scale, sample_ratio=sampling_ratio, + position_sensitive=position_sensitive) + C_out = C // pooled_size[0] // pooled_size[1] if position_sensitive else C + dy = mx.nd.random.uniform(-1, 1, (R, C_out) + pooled_size, ctx=ctx, dtype = dtype) output.backward(dy) real_output, [dx, drois] = roialign_forward_backward(data.asnumpy(), rois.asnumpy(), pooled_size, - spatial_scale, sampling_ratio, dy.asnumpy()) + spatial_scale, sampling_ratio, + position_sensitive, dy.asnumpy()) assert np.allclose(output.asnumpy(), real_output) # It seems that the precision between Cfloat and Pyfloat is different. assert np.allclose(data.grad.asnumpy(), dx, atol = 1e-5), np.abs(data.grad.asnumpy() - dx).max() @@ -6918,7 +7005,8 @@ def test_roi_align_autograd(sampling_ratio=0): numeric_eps=1e-4, rtol=1e-1, atol=1e-4, ctx=ctx) test_roi_align_value() - test_roi_align_value(2) + test_roi_align_value(sampling_ratio=2) + test_roi_align_value(position_sensitive=True) test_roi_align_autograd() @with_seed() diff --git a/tests/python/unittest/test_optimizer.py b/tests/python/unittest/test_optimizer.py index 334b7d4c0..eb33f9b52 100644 --- a/tests/python/unittest/test_optimizer.py +++ b/tests/python/unittest/test_optimizer.py @@ -15,7 +15,9 @@ # specific language governing permissions and limitations # under the License. +import itertools import numpy as np +import itertools import mxnet as mx import mxnet.lr_scheduler as lr_scheduler from mxnet import gluon @@ -501,16 +503,14 @@ def test_ftml(): # ADAM - class PyAdam(mx.optimizer.Optimizer): """python reference implemenation of adam""" def __init__(self, learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8, - decay_factor=(1 - 1e-8), lazy_update=True, **kwargs): + lazy_update=True, **kwargs): super(PyAdam, self).__init__(learning_rate=learning_rate, **kwargs) self.beta1 = beta1 self.beta2 = beta2 self.epsilon = epsilon - self.decay_factor = decay_factor self.lazy_update = lazy_update def create_state(self, index, weight): @@ -613,6 +613,79 @@ def test_adam(): dtype, w_stype='default', g_stype='row_sparse', rtol=1e-4, atol=2e-5) +# AdaMax +class PyAdamax(mx.optimizer.Optimizer): + """The python reference of AdaMax optimizer. + + This class implements the AdaMax optimizer, one variant of Adam based on the infinity norm, + available at http://arxiv.org/abs/1412.6980 Section 7. + + The optimizer updates the weight by:: + grad = clip(grad * rescale_grad + wd * weight, clip_gradient) + m = beta1 * m_t + (1 - beta1) * grad + u = maximum(beta2 * u, abs(grad)) + weight -= lr / (1 - beta1**t) * m / u + + This optimizer accepts the following parameters in addition to those accepted + by :class:`.Optimizer`. + + Parameters + ---------- + beta1 : float, optional + Exponential decay rate for the first moment estimates. + beta2 : float, optional + Exponential decay rate for the second moment estimates. + """ + def __init__(self, learning_rate=0.002, beta1=0.9, beta2=0.999, **kwargs): + super(PyAdamax, self).__init__(learning_rate=learning_rate, **kwargs) + self.beta1 = beta1 + self.beta2 = beta2 + + def create_state(self, index, weight): + return (mx.nd.zeros(weight.shape, weight.context, dtype=weight.dtype), # mean + mx.nd.zeros(weight.shape, weight.context, dtype=weight.dtype)) # variance + + def update(self, index, weight, grad, state): + self._update_count(index) + lr = self._get_lr(index) + wd = self._get_wd(index) + + t = self._index_update_count[index] + lr /= (1. - self.beta1**t) + + # preprocess grad + grad = grad * self.rescale_grad + wd * weight + if self.clip_gradient is not None: + grad = mx.nd.clip(grad, -self.clip_gradient, self.clip_gradient) + + # update m_t and u_t + m_t, u_t = state + m_t[:] = self.beta1 * m_t + (1. - self.beta1) * grad + u_t[:] = mx.nd.maximum(self.beta2 * u_t, mx.nd.abs(grad)) + + # update weight + weight[:] -= lr * m_t / u_t + + +@with_seed() +def test_adamax(): + opt1 = PyAdamax + opt2 = mx.optimizer.Adamax + shape = (3, 4, 5) + cg_options = [{}, {'clip_gradient': 0.4}, {'clip_gradient': 0.5}] + rg_options = [{}, {'rescale_grad': 0.14}, {'rescale_grad': 0.8}] + wd_options = [{}, {'wd': 0.03}, {'wd': 0.05}, {'wd': 0.07}] + mp_options = [{}, {'multi_precision': False}, {'multi_precision': True}] + for dtype in [np.float16, np.float32, np.float64]: + for params in itertools.product(cg_options, rg_options, wd_options, mp_options): + kwarg = {k: v for param in params for k, v in param.items()} + if (dtype == np.float16 and + ('multi_precision' not in kwarg or + not kwarg['multi_precision'])): + continue + compare_optimizer(opt1(**kwarg), opt2(**kwarg), shape, dtype) + + # Signum class PySignum(mx.optimizer.Optimizer): """The python reference of Signum optimizer. @@ -976,8 +1049,8 @@ def update(self, index, weight, grad, state): div = grad / mx.nd.sqrt(history + self.float_stable_eps) weight[:] += (div + weight * wd) * -lr +@with_seed() def test_adagrad(): - mx.random.seed(0) opt1 = PyAdaGrad opt2 = mx.optimizer.AdaGrad shape = (3, 4, 5) @@ -1002,6 +1075,76 @@ def test_adagrad(): compare_optimizer(opt1(**kwarg), opt2(**kwarg), shape, dtype, g_stype='row_sparse') +# AdaDelta +class PyAdaDelta(mx.optimizer.Optimizer): + """The python reference of AdaDelta optimizer. + + This class implements AdaDelta, an optimizer described in *ADADELTA: An adaptive + learning rate method*, available at https://arxiv.org/abs/1212.5701. + + This optimizer updates each weight by:: + + grad = clip(grad * rescale_grad + wd * weight, clip_gradient) + acc_grad = rho * acc_grad + (1. - rho) * grad ** 2 + cur_delta = sqrt(acc_delta + epsilon) / sqrt(acc_grad + epsilon) * grad + acc_delta = rho * acc_delta + (1. - rho) * cur_delta ** 2 + weight -= (cur_delta + wd * weight) + + This optimizer accepts the following parameters in addition to those accepted + by :class:`.Optimizer`. + + Parameters + ---------- + rho: float + Decay rate for both squared gradients and delta. + epsilon : float + Small value to avoid division by 0. + """ + def __init__(self, rho=0.90, epsilon=1e-5, **kwargs): + super(PyAdaDelta, self).__init__(**kwargs) + self.rho = rho + self.epsilon = epsilon + + def create_state(self, index, weight): + return (mx.nd.zeros(weight.shape, weight.context), + mx.nd.zeros(weight.shape, weight.context)) + + def update(self, index, weight, grad, state): + self._update_count(index) + wd = self._get_wd(index) + + grad *= self.rescale_grad + if self.clip_gradient is not None: + grad = mx.nd.clip(grad, -self.clip_gradient, self.clip_gradient) + + acc_grad, acc_delta = state + + acc_grad[:] = self.rho * acc_grad + (1. - self.rho) * grad ** 2 + current_delta = (mx.nd.sqrt(acc_delta + self.epsilon) / + mx.nd.sqrt(acc_grad + self.epsilon)) * grad + acc_delta[:] = self.rho * acc_delta + (1. - self.rho) * current_delta ** 2 + + # update weight + weight[:] -= current_delta + wd * weight + +@with_seed() +def test_adadelta(): + opt1 = PyAdaDelta + opt2 = mx.optimizer.AdaDelta + shape = (3, 4, 5) + rho_options = [{'rho': 0.9}] + eps_options = [{}, {'epsilon': 1e-8}] + cg_options = [{}, {'clip_gradient': 0.4}, {'clip_gradient': 0.5}] + rg_options = [{}, {'rescale_grad': 0.14}, {'rescale_grad': 0.8}] + wd_options = [{}, {'wd': 0.0}] + for dtype in [np.float16, np.float32]: + for params in itertools.product(rho_options, eps_options, cg_options, + rg_options, wd_options): + kwarg = {k: v for param in params for k, v in param.items()} + if dtype is np.float16: + kwarg.update({'multi_precision': True}) + compare_optimizer(opt1(**kwarg), opt2(**kwarg), shape, dtype) + def test_factor_scheduler(): base_lr = 1 diff --git a/tests/python/unittest/test_random.py b/tests/python/unittest/test_random.py index 4310658ae..405602f07 100644 --- a/tests/python/unittest/test_random.py +++ b/tests/python/unittest/test_random.py @@ -45,6 +45,16 @@ def check_with_device(device, dtype): ('std', lambda x, params: np.std(x.astype(np.float64)) - params['scale'], tol) ] }, + { + 'name': 'normal_like', + 'symbol': mx.sym.random.normal_like, + 'ndop': mx.nd.random.normal_like, + 'params': { 'loc': 10.0, 'scale': 0.5 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64) - params['loc']), tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - params['scale'], tol) + ] + }, { 'name': 'randn', 'ndop': mx.nd.random.randn, @@ -66,62 +76,122 @@ def check_with_device(device, dtype): ] }, { - 'name': 'gamma', - 'symbol': mx.sym.random.gamma, - 'ndop': mx.nd.random.gamma, - 'params': { 'alpha': 9.0, 'beta': 0.5 }, - 'inputs': [ ('alpha', [ [ 0.0, 2.5 ], [ 9.75, 11.0 ] ]) , ('beta', [ [ 1.0, 0.7 ], [ 0.5, 0.3 ] ]) ], - 'checks': [ - ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['alpha'] * params['beta'], tol), - ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['alpha'] * params['beta'] ** 2), tol) - ] - }, - { - 'name': 'exponential', - 'symbol': mx.sym.random.exponential, - 'ndop': mx.nd.random.exponential, - 'params': { 'scale': 1.0/4.0 }, - 'inputs': [ ('scale', [ [ 1.0/1.0, 1.0/8.5 ], [ 1.0/2.7 , 1.0/0.5 ] ]) ], - 'checks': [ - ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['scale'], tol), - ('std', lambda x, params: np.std(x.astype(np.float64)) - params['scale'], tol) - ] - }, - { - 'name': 'poisson', - 'symbol': mx.sym.random.poisson, - 'ndop': mx.nd.random.poisson, - 'params': { 'lam': 4.0 }, - 'inputs': [ ('lam', [ [ 25.0, 8.5 ], [ 2.7 , 0.5 ] ]) ], - 'checks': [ - ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['lam'], tol), - ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['lam']), tol) - ] - }, - { - 'name': 'neg-binomial', - 'symbol': mx.sym.random.negative_binomial, - 'ndop': mx.nd.random.negative_binomial, - 'params': { 'k': 3, 'p': 0.4 }, - 'inputs': [ ('k', [ [ 3, 4 ], [ 5 , 6 ] ]) , ('p', [ [ 0.4 , 0.77 ], [ 0.5, 0.84 ] ]) ], - 'checks': [ - ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['k'] * (1.0 - params['p']) / params['p'], tol), - ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['k'] * (1.0 - params['p']))/params['p'], tol) - ] - }, - { - 'name': 'gen-neg-binomial', - 'symbol': mx.sym.random.generalized_negative_binomial, - 'ndop': mx.nd.random.generalized_negative_binomial, - 'params': { 'mu': 2.0, 'alpha': 0.3 }, - 'inputs': [ ('mu', [ [ 2.0, 2.5 ], [ 1.3, 1.9 ] ]) , ('alpha', [ [ 1.0, 0.1 ], [ 0.2, 0.5 ] ]) ], - 'checks': [ - ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['mu'], tol), - ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['mu'] + params['alpha'] * params['mu'] ** 2 ), tol) - ] - } + 'name': 'uniform_like', + 'symbol': mx.sym.random.uniform_like, + 'ndop': mx.nd.random.uniform_like, + 'params': { 'low': -1.5, 'high': 3.0 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - (params['low'] + params['high']) / 2.0, tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(1.0 / 12.0) * (params['high'] - params['low']), tol) + ] + }, + { + 'name': 'gamma', + 'symbol': mx.sym.random.gamma, + 'ndop': mx.nd.random.gamma, + 'params': { 'alpha': 9.0, 'beta': 0.5 }, + 'inputs': [ ('alpha', [ [ 0.0, 2.5 ], [ 9.75, 11.0 ] ]) , ('beta', [ [ 1.0, 0.7 ], [ 0.5, 0.3 ] ]) ], + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['alpha'] * params['beta'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['alpha'] * params['beta'] ** 2), tol) + ] + }, + { + 'name': 'gamma_like', + 'symbol': mx.sym.random.gamma_like, + 'ndop': mx.nd.random.gamma_like, + 'params': { 'alpha': 9.0, 'beta': 0.5 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['alpha'] * params['beta'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['alpha'] * params['beta'] ** 2), tol) + ] + }, + { + 'name': 'exponential', + 'symbol': mx.sym.random.exponential, + 'ndop': mx.nd.random.exponential, + 'params': { 'scale': 1.0/4.0 }, + 'inputs': [ ('scale', [ [ 1.0/1.0, 1.0/8.5 ], [ 1.0/2.7 , 1.0/0.5 ] ]) ], + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['scale'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - params['scale'], tol) + ] + }, + { + 'name': 'exponential_like', + 'symbol': mx.sym.random.exponential_like, + 'ndop': mx.nd.random.exponential_like, + 'params': { 'lam': 4.0 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - 1.0/params['lam'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - 1.0/params['lam'], tol) + ] + }, + { + 'name': 'poisson', + 'symbol': mx.sym.random.poisson, + 'ndop': mx.nd.random.poisson, + 'params': { 'lam': 4.0 }, + 'inputs': [ ('lam', [ [ 25.0, 8.5 ], [ 2.7 , 0.5 ] ]) ], + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['lam'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['lam']), tol) + ] + }, + { + 'name': 'poisson_like', + 'symbol': mx.sym.random.poisson_like, + 'ndop': mx.nd.random.poisson_like, + 'params': { 'lam': 4.0 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['lam'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['lam']), tol) + ] + }, + { + 'name': 'neg_binomial', + 'symbol': mx.sym.random.negative_binomial, + 'ndop': mx.nd.random.negative_binomial, + 'params': { 'k': 3, 'p': 0.4 }, + 'inputs': [ ('k', [ [ 3, 4 ], [ 5 , 6 ] ]) , ('p', [ [ 0.4 , 0.77 ], [ 0.5, 0.84 ] ]) ], + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['k'] * (1.0 - params['p']) / params['p'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['k'] * (1.0 - params['p']))/params['p'], tol) + ] + }, + { + 'name': 'neg_binomial_like', + 'symbol': mx.sym.random.negative_binomial_like, + 'ndop': mx.nd.random.negative_binomial_like, + 'params': { 'k': 3, 'p': 0.4 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['k'] * (1.0 - params['p']) / params['p'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['k'] * (1.0 - params['p']))/params['p'], tol) + ] + }, + { + 'name': 'gen_neg_binomial', + 'symbol': mx.sym.random.generalized_negative_binomial, + 'ndop': mx.nd.random.generalized_negative_binomial, + 'params': { 'mu': 2.0, 'alpha': 0.3 }, + 'inputs': [ ('mu', [ [ 2.0, 2.5 ], [ 1.3, 1.9 ] ]) , ('alpha', [ [ 1.0, 0.1 ], [ 0.2, 0.5 ] ]) ], + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['mu'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['mu'] + params['alpha'] * params['mu'] ** 2 ), tol) + ] + }, + { + 'name': 'gen_neg_binomial_like', + 'symbol': mx.sym.random.generalized_negative_binomial_like, + 'ndop': mx.nd.random.generalized_negative_binomial_like, + 'params': { 'mu': 2.0, 'alpha': 0.3 }, + 'checks': [ + ('mean', lambda x, params: np.mean(x.astype(np.float64)) - params['mu'], tol), + ('std', lambda x, params: np.std(x.astype(np.float64)) - np.sqrt(params['mu'] + params['alpha'] * params['mu'] ** 2 ), tol) + ] + }, - ] + ] # Create enough samples such that we get a meaningful distribution. shape = (500, 500) @@ -136,6 +206,10 @@ def check_with_device(device, dtype): if name == 'randn': params.pop('shape') # randn does not accept shape param args = shape + if name.endswith('_like'): + params['data'] = mx.nd.ones(params.pop('shape'), + dtype=params.pop('dtype'), + ctx=params.pop('ctx')) mx.random.seed(128) ret1 = ndop(*args, **params).asnumpy() mx.random.seed(128) @@ -171,6 +245,8 @@ def check_with_device(device, dtype): X = mx.sym.Variable("X") params = symbdic['params'].copy() params.update(shape=shape, dtype=dtype) + if name.endswith('_like'): + params['data'] = mx.sym.ones(params.pop('shape')) Y = symbol(**params) + X x = mx.nd.zeros(shape, dtype=dtype, ctx=device) xgrad = mx.nd.zeros(shape, dtype=dtype, ctx=device) @@ -189,6 +265,7 @@ def check_with_device(device, dtype): ret1 = un1.asnumpy() for check_name, check_func, tol in symbdic['checks']: assert np.abs(check_func(ret1, params)) < tol, "symbolic test: %s check for `%s` did not pass" % (check_name, name) + if name.endswith('_like'): continue # check multi-distribution sampling symbol = symbdic['symbol'] @@ -428,7 +505,6 @@ def test_normal_generator(): num_buckets = 5 for dtype in ['float16', 'float32', 'float64']: for mu, sigma in [(0.0, 1.0), (1.0, 5.0)]: - print("ctx=%s, dtype=%s, Mu=%g, Sigma=%g:" % (ctx, dtype, mu, sigma)) buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.norm.ppf(x, mu, sigma), num_buckets) # Quantize bucket boundaries to reflect the actual dtype and adjust probs accordingly buckets = np.array(buckets, dtype=dtype).tolist() @@ -449,7 +525,6 @@ def test_uniform_generator(): ctx = mx.context.current_context() for dtype in ['float16', 'float32', 'float64']: for low, high in [(-1.0, 1.0), (1.0, 3.0)]: - print("ctx=%s, dtype=%s, Low=%g, High=%g:" % (ctx, dtype, low, high)) scale = high - low buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.uniform.ppf(x, loc=low, scale=scale), 5) # Quantize bucket boundaries to reflect the actual dtype and adjust probs accordingly @@ -469,7 +544,6 @@ def test_gamma_generator(): ctx = mx.context.current_context() for dtype in ['float16', 'float32', 'float64']: for kappa, theta in [(0.5, 1.0), (1.0, 5.0)]: - print("ctx=%s, dtype=%s, Shape=%g, Scale=%g:" % (ctx, dtype, kappa, theta)) buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.gamma.ppf(x, a=kappa, loc=0, scale=theta), 5) generator_mx = lambda x: mx.nd.random.gamma(kappa, theta, shape=x, ctx=ctx, dtype=dtype).asnumpy() verify_generator(generator=generator_mx, buckets=buckets, probs=probs, success_rate=success_rate) @@ -484,7 +558,6 @@ def test_exponential_generator(): ctx = mx.context.current_context() for dtype in ['float16', 'float32', 'float64']: for scale in [0.1, 1.0]: - print("ctx=%s, dtype=%s, Scale=%g:" % (ctx, dtype, scale)) buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.expon.ppf(x, loc=0, scale=scale), 5) generator_mx = lambda x: mx.nd.random.exponential(scale, shape=x, ctx=ctx, dtype=dtype).asnumpy() verify_generator(generator=generator_mx, buckets=buckets, probs=probs) @@ -499,7 +572,6 @@ def test_poisson_generator(): ctx = mx.context.current_context() for dtype in ['float16', 'float32', 'float64']: for lam in [1, 10]: - print("ctx=%s, dtype=%s, Lambda=%d:" % (ctx, dtype, lam)) buckets = [(-1.0, lam - 0.5), (lam - 0.5, 2 * lam + 0.5), (2 * lam + 0.5, np.inf)] probs = [ss.poisson.cdf(bucket[1], lam) - ss.poisson.cdf(bucket[0], lam) for bucket in buckets] generator_mx = lambda x: mx.nd.random.poisson(lam, shape=x, ctx=ctx, dtype=dtype).asnumpy() @@ -516,7 +588,6 @@ def test_negative_binomial_generator(): for dtype in ['float16', 'float32', 'float64']: success_num = 2 success_prob = 0.2 - print("ctx=%s, dtype=%s, Success Num=%d:, Success Prob=%g" % (ctx, dtype, success_num, success_prob)) buckets = [(-1.0, 2.5), (2.5, 5.5), (5.5, 8.5), (8.5, np.inf)] probs = [ss.nbinom.cdf(bucket[1], success_num, success_prob) - ss.nbinom.cdf(bucket[0], success_num, success_prob) for bucket in buckets] @@ -529,7 +600,6 @@ def test_negative_binomial_generator(): for _ in range(10)]) verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs) # Also test the Gamm-Poisson Mixture - print('Gamm-Poisson Mixture Test:') alpha = 1.0 / success_num mu = (1.0 - success_prob) / success_prob / alpha generator_mx = lambda x: mx.nd.random.generalized_negative_binomial(mu, alpha, @@ -566,7 +636,6 @@ def quantize_probs(probs, dtype): trials = 5 buckets = list(range(6)) for dtype in ['float16', 'float32', 'float64']: - print("ctx=%s, dtype=%s" %(ctx, dtype)) quantized_probs = quantize_probs(probs, dtype) generator_mx = lambda x: mx.nd.random.multinomial(data=mx.nd.array(quantized_probs, ctx=ctx, dtype=dtype), shape=x).asnumpy() @@ -767,6 +836,54 @@ def testLarge(data, repeat): testLarge(mx.nd.arange(0, 100000).reshape((10, 10000)), 10) testLarge(mx.nd.arange(0, 100000).reshape((10000, 10)), 10) +@with_seed() +def test_randint(): + dtypes = ['int32', 'int64'] + for dtype in dtypes: + params = { + 'low': -1, + 'high': 3, + 'shape' : (500, 500), + 'dtype' : dtype, + 'ctx' : mx.context.current_context() + } + mx.random.seed(128) + ret1 = mx.nd.random.randint(**params).asnumpy() + mx.random.seed(128) + ret2 = mx.nd.random.randint(**params).asnumpy() + assert same(ret1, ret2), \ + "ndarray test: `%s` should give the same result with the same seed" + +@with_seed() +def test_randint_extremes(): + a = mx.nd.random.randint(dtype='int64', low=50000000, high=50000010, ctx=mx.context.current_context()) + assert a>=50000000 and a<=50000010 + +@with_seed() +def test_randint_generator(): + ctx = mx.context.current_context() + for dtype in ['int32', 'int64']: + for low, high in [(50000000, 50001000),(-50000100,-50000000),(-500,199)]: + scale = high - low + buckets, probs = gen_buckets_probs_with_ppf(lambda x: ss.uniform.ppf(x, loc=low, scale=scale), 5) + # Quantize bucket boundaries to reflect the actual dtype and adjust probs accordingly + buckets = np.array(buckets, dtype=dtype).tolist() + probs = [(buckets[i][1] - buckets[i][0]) / float(scale) for i in range(5)] + generator_mx = lambda x: mx.nd.random.randint(low, high, shape=x, ctx=ctx, dtype=dtype).asnumpy() + verify_generator(generator=generator_mx, buckets=buckets, probs=probs, nrepeat=100) + # Scipy uses alpha = 0.01 for testing discrete distribution generator but we are using default alpha=0.05 (higher threshold ensures robustness) + # Refer - https://github.com/scipy/scipy/blob/9f12af697763fb5f9767d5cb1280ce62456a3974/scipy/stats/tests/test_discrete_basic.py#L45 + generator_mx_same_seed = \ + lambda x: np.concatenate( + [mx.nd.random.randint(low, high, shape=x // 10, ctx=ctx, dtype=dtype).asnumpy() + for _ in range(10)]) + verify_generator(generator=generator_mx_same_seed, buckets=buckets, probs=probs, nrepeat=100) + +@with_seed() +def test_randint_without_dtype(): + a = mx.nd.random.randint(low=50000000, high=50000010, ctx=mx.context.current_context()) + assert(a.dtype, 'int32') + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_sparse_ndarray.py b/tests/python/unittest/test_sparse_ndarray.py index 8dd250cf9..7600ea944 100644 --- a/tests/python/unittest/test_sparse_ndarray.py +++ b/tests/python/unittest/test_sparse_ndarray.py @@ -1018,19 +1018,38 @@ def test_sparse_take(): def check_sparse_take(density, mode): data_shape = rand_shape_2d() idx_shape = (np.random.randint(low=1, high=10),) - data = rand_ndarray(data_shape, 'csr', density=density) + data = rand_ndarray(data_shape, 'csr', density=density).astype('int32') idx = mx.nd.array(np.random.randint(low=-5, high=15, size=idx_shape)) - result = mx.nd.take(data, idx, mode=mode) data_np = data.asnumpy() idx_np = idx.asnumpy().astype('int32') expected_result = np.take(data_np, idx_np, mode=mode, axis=0) + result = mx.nd.take(data, idx, mode=mode) assert_almost_equal(result.asnumpy(), expected_result) + assert result.indptr[0].asscalar() == 0 densities = [0, 0.5, 1] modes = ['clip', 'wrap'] for d in densities: for m in modes: check_sparse_take(d, m) +@with_seed() +def test_sparse_getnnz(): + if default_context().device_type is 'gpu': + return + def check_sparse_getnnz(density, axis): + shape = rand_shape_2d() + data = rand_ndarray(shape, 'csr', density=density) + data_sp = data.asscipy() + result = mx.nd.contrib.getnnz(data, axis=axis) + expected_result = data_sp.getnnz(axis=axis) + assert_almost_equal(result.asnumpy(), expected_result) + + densities = [0, 0.5, 1] + axis = [1, None] + for d in densities: + for a in axis: + check_sparse_getnnz(d, a) + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/python/unittest/test_sparse_operator.py b/tests/python/unittest/test_sparse_operator.py index 8d5efeb0d..dac2c1101 100644 --- a/tests/python/unittest/test_sparse_operator.py +++ b/tests/python/unittest/test_sparse_operator.py @@ -2306,6 +2306,48 @@ def check_sparse_quadratic_function(a, b, c, expected_stype): check_sparse_quadratic_function(a, b, 0.0, 'csr') check_sparse_quadratic_function(a, b, 1.0, 'default') +def test_reshape_backward_fallback(): + """ + out + | \ + w_x x + / + w + in which x is a sparse tensor. + Due to sparse gradient optimization in sym.dot, grad(w_x) is sparse. + Though sym.reshape itself does not have sparse version, + if we somehow make grad(w) sparse as well, e.g., + - by setting args_grad in symbol.bind + - or, we can have out_y = sym.dot(sparse_y, w), then grad(w) will be inferred as sparse + reshape backward (from w_x to w) needs to understand how to handle sparse inputs. + """ + ctx = default_context() + w_shape = (12, 4) + w_x_shape = (1, 48) + x_nd = rand_ndarray((4, 1), 'csr') + + w_nd = rand_ndarray(w_shape) + + w_x_nd = w_nd.reshape(w_x_shape) + out_x_nd = mx.nd.dot(x_nd, w_x_nd) + + w_x_backward_grad = mx.nd.dot(x_nd, out_x_nd, transpose_a=True).asnumpy() + expected_grad_nd = w_x_backward_grad.reshape(w_shape) + + x = mx.sym.Variable('x', stype='csr') + w = mx.sym.Variable('w') + + w_x = mx.sym.reshape(w, w_x_shape, name="w_x") + out = mx.sym.sparse.dot(x, w_x, name='out_x') + + grad_w_nd = rand_ndarray(w_shape, 'row_sparse') + executor = out.bind(ctx=ctx, args={"x": x_nd, "w": w_nd}, + args_grad={"w": grad_w_nd}) + executor.forward(is_train=True) + executor.backward(out_x_nd) + + assert_almost_equal(grad_w_nd.asnumpy(), expected_grad_nd) + if __name__ == '__main__': import nose nose.runmodule() diff --git a/tests/requirements.txt b/tests/requirements.txt index 3ca696b28..f64f7ffb6 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -2,3 +2,4 @@ mock nose nose-timer +ipython diff --git a/tests/tutorials/test_sanity_tutorials.py b/tests/tutorials/test_sanity_tutorials.py index cd3f6bfcb..644a61183 100644 --- a/tests/tutorials/test_sanity_tutorials.py +++ b/tests/tutorials/test_sanity_tutorials.py @@ -27,6 +27,7 @@ whitelist = ['basic/index.md', 'c++/basics.md', 'c++/index.md', + 'c++/subgraphAPI.md', 'control_flow/index.md', 'embedded/index.md', 'embedded/wine_detector.md', @@ -49,12 +50,16 @@ 'scala/mnist.md', 'scala/index.md', 'scala/mxnet_scala_on_intellij.md', + 'scala/mxnet_java_install_and_run_examples.md', 'sparse/index.md', 'speech_recognition/index.md', 'unsupervised_learning/index.md', 'vision/index.md', 'tensorrt/index.md', - 'tensorrt/inference_with_trt.md'] + 'tensorrt/inference_with_trt.md', + 'java/index.md', + 'java/mxnet_java_on_intellij.md', + 'java/ssd_inference.md'] whitelist_set = set(whitelist) def test_tutorial_downloadable(): diff --git a/tests/tutorials/test_tutorials.py b/tests/tutorials/test_tutorials.py index 5b8e2152b..8d8ef398d 100644 --- a/tests/tutorials/test_tutorials.py +++ b/tests/tutorials/test_tutorials.py @@ -73,6 +73,9 @@ def test_basic_module(): def test_basic_data(): assert _test_tutorial_nb('basic/data') +def test_basic_reshape_transpose(): + assert _test_tutorial_nb('basic/reshape_transpose') + def test_gluon_customop(): assert _test_tutorial_nb('gluon/customop') @@ -100,7 +103,7 @@ def test_gluon_autograd(): def test_gluon_gluon(): assert _test_tutorial_nb('gluon/gluon') -def test_gluon_save_load_model(): +def test_gluon_save_load_params(): assert _test_tutorial_nb('gluon/save_load_params') def test_gluon_hybrid(): @@ -117,7 +120,10 @@ def test_gluon_learning_rate_schedules(): def test_gluon_learning_rate_schedules_advanced(): assert _test_tutorial_nb('gluon/learning_rate_schedules_advanced') - + +def test_gluon_info_gan(): + assert _test_tutorial_nb('gluon/info_gan') + def test_nlp_cnn(): assert _test_tutorial_nb('nlp/cnn') @@ -189,6 +195,4 @@ def test_vision_cnn_visualization(): def test_control_flow(): assert _test_tutorial_nb('control_flow/ControlFlowTutorial') - -def test_subgraphapi(): - assert _test_tutorial_nb('c++/subgraphAPI') + diff --git a/tools/build/build_lib.sh b/tools/build/build_lib.sh new file mode 100755 index 000000000..032fcb920 --- /dev/null +++ b/tools/build/build_lib.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the libraries of mxnet. +make_config=config/pip_${PLATFORM}_${VARIANT}.mk +if [[ ! -f $make_config ]]; then + >&2 echo "Couldn't find make config $make_config for the current settings." + exit 1 +fi + +git clone --recursive https://github.com/apache/incubator-mxnet mxnet-build + +>&2 echo "Now building mxnet modules..." +cp $make_config mxnet-build/config.mk + +cd mxnet-build + +make DEPS_PATH=$DEPS_PATH DMLCCORE +make DEPS_PATH=$DEPS_PATH $PWD/3rdparty/tvm/nnvm/lib/libnnvm.a +make DEPS_PATH=$DEPS_PATH PSLITE + +if [[ $VARIANT == *mkl ]]; then + MKLDNN_LICENSE='license.txt' + if [[ $PLATFORM == 'linux' ]]; then + IOMP_LIBFILE='libiomp5.so' + MKLML_LIBFILE='libmklml_intel.so' + MKLDNN_LIBFILE='libmkldnn.so.0' + else + IOMP_LIBFILE='libiomp5.dylib' + MKLML_LIBFILE='libmklml.dylib' + MKLDNN_LIBFILE='libmkldnn.0.dylib' + fi + make DEPS_PATH=$DEPS_PATH mkldnn + cp 3rdparty/mkldnn/LICENSE ./MKLML_LICENSE +fi + +if [[ $VARIANT == *mkl ]]; then + >&2 echo "Copying MKL license." + rm lib/libmkldnn.{so,dylib} + rm lib/libmkldnn.0.*.dylib + rm lib/libmkldnn.so.0.* +fi + +>&2 echo "Now building mxnet..." +make DEPS_PATH=$DEPS_PATH || exit 1; + +if [[ $PLATFORM == 'linux' ]]; then + cp -L /usr/lib/gcc/x86_64-linux-gnu/4.8/libgfortran.so lib/libgfortran.so.3 + cp -L /usr/lib/x86_64-linux-gnu/libquadmath.so.0 lib/libquadmath.so.0 +fi + +# Print the linked objects on libmxnet.so +>&2 echo "Checking linked objects on libmxnet.so..." +if [[ ! -z $(command -v readelf) ]]; then + readelf -d lib/libmxnet.so + strip --strip-unneeded lib/libmxnet.so +elif [[ ! -z $(command -v otool) ]]; then + otool -L lib/libmxnet.so + strip -u -r -x lib/libmxnet.so +else + >&2 echo "Not available" +fi + +cd ../ diff --git a/tools/build/build_wheel.sh b/tools/build/build_wheel.sh new file mode 100755 index 000000000..a79634117 --- /dev/null +++ b/tools/build/build_wheel.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the wheel for binary distribution and performs sanity check. + +cd mxnet-build +echo $(git rev-parse HEAD) >> python/mxnet/COMMIT_HASH +cd - + +# Make wheel for testing +python setup.py bdist_wheel + +wheel_name=$(ls -t dist | head -n 1) +pip install -U --user --force-reinstall dist/$wheel_name +python sanity_test.py diff --git a/tools/dependencies/README.md b/tools/dependencies/README.md new file mode 100644 index 000000000..cfe3d6c75 --- /dev/null +++ b/tools/dependencies/README.md @@ -0,0 +1,14 @@ +# Overview + +This folder contains scripts for building the dependencies from source. The static libraries from +the build artifacts can be used to create self-contained shared object for mxnet through static +linking. + +# Settings + +The scripts use the following environment variables for setting behavior: + +`DEPS_PATH`: the location in which the libraries are downloaded, built, and installed. +`PLATFORM`: name of the OS in lower case. Supported options are 'linux' and 'darwin'. + +It also expects the following build tools in path: make, cmake, tar, unzip, autoconf, nasm diff --git a/tools/dependencies/cityhash.sh b/tools/dependencies/cityhash.sh new file mode 100755 index 000000000..81cc9cbaa --- /dev/null +++ b/tools/dependencies/cityhash.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of cityhash that can be used as dependency of mxnet. +CITYHASH_VERSION=1.1.1 +if [[ ! -f $DEPS_PATH/lib/libcityhash.a ]]; then + # Download and build cityhash + >&2 echo "Building cityhash..." + git clone https://github.com/google/cityhash $DEPS_PATH/cityhash-$CITYHASH_VERSION + cd $DEPS_PATH/cityhash-$CITYHASH_VERSION + git reset --hard 8af9b8c2b889d80c22d6bc26ba0df1afb79a30db + ./configure -prefix=$DEPS_PATH --enable-sse4.2 + make CXXFLAGS="-g -O3 -msse4.2" + make install + cd - +fi diff --git a/tools/dependencies/curl.sh b/tools/dependencies/curl.sh new file mode 100755 index 000000000..9633edb78 --- /dev/null +++ b/tools/dependencies/curl.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of libcurl that can be used as dependency of mxnet. +LIBCURL_VERSION=7.61.0 +if [[ ! -f $DEPS_PATH/lib/libcurl.a ]]; then + # download and build libcurl + >&2 echo "Building libcurl..." + curl -s -L https://curl.haxx.se/download/curl-$LIBCURL_VERSION.zip -o $DEPS_PATH/libcurl.zip + unzip -q $DEPS_PATH/libcurl.zip -d $DEPS_PATH + cd $DEPS_PATH/curl-$LIBCURL_VERSION + if [[ $PLATFORM == 'linux' ]]; then + CONFIG_FLAG="" + elif [[ $PLATFORM == 'darwin' ]]; then + CONFIG_FLAG="--with-darwinssl" + fi + ./configure $CONFIG_FLAG \ + --with-zlib \ + --with-nghttps2 \ + --without-zsh-functions-dir \ + --without-librtmp \ + --without-libssh2 \ + --disable-debug \ + --disable-curldebug \ + --enable-symbol-hiding=yes \ + --enable-optimize=yes \ + --enable-shared=no \ + --enable-http=yes \ + --enable-ipv6=yes \ + --disable-ftp \ + --disable-ldap \ + --disable-ldaps \ + --disable-rtsp \ + --disable-proxy \ + --disable-dict \ + --disable-telnet \ + --disable-tftp \ + --disable-pop3 \ + --disable-imap \ + --disable-smb \ + --disable-smtp \ + --disable-gopher \ + --disable-manual \ + --prefix=$DEPS_PATH + make + make install + cd - +fi diff --git a/tools/dependencies/eigen.sh b/tools/dependencies/eigen.sh new file mode 100755 index 000000000..ac2f75a03 --- /dev/null +++ b/tools/dependencies/eigen.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script imports the headers from eigen3 that can be used to in opencv. +EIGEN_VERSION=3.3.4 +if [[ ! -d $DEPS_PATH/include/eigen3 ]]; then + # download eigen + >&2 echo "Loading eigen..." + curl -s -L https://github.com/eigenteam/eigen-git-mirror/archive/$EIGEN_VERSION.zip -o $DEPS_PATH/eigen.zip + unzip -q $DEPS_PATH/eigen.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/eigen-git-mirror-$EIGEN_VERSION/build + cd $DEPS_PATH/eigen-git-mirror-$EIGEN_VERSION/build + cmake \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH .. + make install + cd - +fi diff --git a/tools/dependencies/libpng.sh b/tools/dependencies/libpng.sh new file mode 100755 index 000000000..d1523c654 --- /dev/null +++ b/tools/dependencies/libpng.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of libpng that can be used as dependency of mxnet/opencv. +PNG_VERSION=1.6.34 +if [[ ! -f $DEPS_PATH/lib/libpng.a ]]; then + # download and build libpng + >&2 echo "Building libpng..." + curl -s -L https://github.com/glennrp/libpng/archive/v$PNG_VERSION.zip -o $DEPS_PATH/libpng.zip + unzip -q $DEPS_PATH/libpng.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/libpng-$PNG_VERSION/build + cd $DEPS_PATH/libpng-$PNG_VERSION/build + cmake \ + -D PNG_SHARED=OFF \ + -D PNG_STATIC=ON \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ + -D CMAKE_C_FLAGS=-fPIC .. + make + make install + mkdir -p $DEPS_PATH/include/libpng + ln -s $DEPS_PATH/include/png.h $DEPS_PATH/include/libpng/png.h + cd - +fi diff --git a/tools/dependencies/libtiff.sh b/tools/dependencies/libtiff.sh new file mode 100755 index 000000000..14dcb2d7b --- /dev/null +++ b/tools/dependencies/libtiff.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of libtiff that can be used as dependency of mxnet/opencv. +TIFF_VERSION="4-0-9" +if [[ ! -f $DEPS_PATH/lib/libtiff.a ]]; then + # download and build libtiff + >&2 echo "Building libtiff..." + curl -s -L https://gitlab.com/libtiff/libtiff/-/archive/Release-v$TIFF_VERSION/libtiff-Release-v$TIFF_VERSION.zip -o $DEPS_PATH/libtiff.zip + unzip -q $DEPS_PATH/libtiff.zip -d $DEPS_PATH + cd $DEPS_PATH/libtiff-Release-v$TIFF_VERSION + ./configure --quiet --disable-shared --disable-jpeg --disable-zlib --disable-jbig --disable-lzma --prefix=$DEPS_PATH + make + make install + cd - +fi diff --git a/tools/dependencies/libturbojpeg.sh b/tools/dependencies/libturbojpeg.sh new file mode 100755 index 000000000..4991906f8 --- /dev/null +++ b/tools/dependencies/libturbojpeg.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of libturbojpeg that can be used as dependency of +# mxnet/opencv. +TURBO_JPEG_VERSION=1.5.90 +if [[ $PLATFORM == 'darwin' ]]; then + JPEG_NASM_OPTION="-D CMAKE_ASM_NASM_COMPILER=/usr/local/bin/nasm" +fi + +if [[ ! -f $DEPS_PATH/lib/libjpeg.a ]] || [[ ! -f $DEPS_PATH/lib/libturbojpeg.a ]]; then + # download and build libjpeg + >&2 echo "Building libjpeg-turbo..." + curl -s -L https://github.com/libjpeg-turbo/libjpeg-turbo/archive/$TURBO_JPEG_VERSION.zip -o $DEPS_PATH/libjpeg.zip + unzip -q $DEPS_PATH/libjpeg.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/libjpeg-turbo-$TURBO_JPEG_VERSION/build + cd $DEPS_PATH/libjpeg-turbo-$TURBO_JPEG_VERSION/build + cmake \ + -G"Unix Makefiles" \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ + -D CMAKE_C_FLAGS=-fPIC \ + -D WITH_JAVA=FALSE \ + -D WITH_JPEG7=TRUE \ + -D WITH_JPEG8=TRUE \ + $JPEG_NASM_OPTION \ + -D ENABLE_SHARED=FALSE .. + make + make install + cd - +fi diff --git a/ci/docker/qemu/playbook.yml b/tools/dependencies/libz.sh old mode 100644 new mode 100755 similarity index 55% rename from ci/docker/qemu/playbook.yml rename to tools/dependencies/libz.sh index 3b9e7c52b..927f1de82 --- a/ci/docker/qemu/playbook.yml +++ b/tools/dependencies/libz.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash + # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -15,34 +17,20 @@ # specific language governing permissions and limitations # under the License. ---- -- name: provision QEMU VM - hosts: all - gather_facts: no - become: true - become_user: root - tasks: - - name: Wait until ssh is available - wait_for_connection: - delay: 0 - sleep: 3 - timeout: 400 - - command: hostname - register: h - - debug: msg="{{ h.stdout }}" - - - name: copy mxnet artifacts - copy: - src: "{{ item }}" - dest: mxnet_dist/ - with_fileglob: "/work/mxnet/build/*.whl" - - - name: copy runtime_functions.py - copy: - src: "/work/runtime_functions.py" - dest: . - - file: - path: runtime_functions.py - mode: 0755 - - +# This script builds the static library of libz that can be used as dependency of mxnet. +ZLIB_VERSION=1.2.6 +if [[ ! -f $DEPS_PATH/lib/libz.a ]]; then + # Download and build zlib + >&2 echo "Building zlib..." + curl -s -L https://github.com/LuaDist/zlib/archive/$ZLIB_VERSION.zip -o $DEPS_PATH/zlib.zip + unzip -q $DEPS_PATH/zlib.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/zlib-$ZLIB_VERSION/build + cd $DEPS_PATH/zlib-$ZLIB_VERSION/build + cmake \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ + -D BUILD_SHARED_LIBS=OFF .. + make + make install + cd - +fi diff --git a/tools/dependencies/lz4.sh b/tools/dependencies/lz4.sh new file mode 100755 index 000000000..a4269bf29 --- /dev/null +++ b/tools/dependencies/lz4.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of lz4 that can be used as dependency of mxnet. +LZ4_VERSION=r130 +if [[ ! -f $DEPS_PATH/lib/liblz4.a ]]; then + # Download and build lz4 + >&2 echo "Building lz4..." + curl -s -L https://github.com/lz4/lz4/archive/$LZ4_VERSION.zip -o $DEPS_PATH/lz4.zip + unzip -q $DEPS_PATH/lz4.zip -d $DEPS_PATH + cd $DEPS_PATH/lz4-$LZ4_VERSION + make + make PREFIX=$DEPS_PATH install + cd - +fi diff --git a/example/python-howto/debug_conv.py b/tools/dependencies/make_shared_dependencies.sh old mode 100644 new mode 100755 similarity index 58% rename from example/python-howto/debug_conv.py rename to tools/dependencies/make_shared_dependencies.sh index 9de421d8e..d678fddcc --- a/example/python-howto/debug_conv.py +++ b/tools/dependencies/make_shared_dependencies.sh @@ -1,3 +1,5 @@ +#!/usr/bin/env bash + # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information @@ -15,25 +17,24 @@ # specific language governing permissions and limitations # under the License. -import mxnet as mx - -data_shape = (1,3,5,5) -class SimpleData(object): - - def __init__(self, data): - self.data = data - -data = mx.sym.Variable('data') -conv = mx.sym.Convolution(data=data, kernel=(3,3), pad=(1,1), stride=(1,1), num_filter=1) -mon = mx.mon.Monitor(1) +# This is a convenience script for calling the build scripts of all dependency libraries. +# Environment variables should be set beforehand. +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" -mod = mx.mod.Module(conv) -mod.bind(data_shapes=[('data', data_shape)]) -mod._exec_group.install_monitor(mon) -mod.init_params() -input_data = mx.nd.ones(data_shape) -mod.forward(data_batch=SimpleData([input_data])) -res = mod.get_outputs()[0].asnumpy() -print(res) +if [[ ! $PLATFORM == 'darwin' ]]; then + source $DIR/openblas.sh +fi +source $DIR/libz.sh +source $DIR/libturbojpeg.sh +source $DIR/libpng.sh +source $DIR/libtiff.sh +source $DIR/openssl.sh +source $DIR/curl.sh +source $DIR/eigen.sh +source $DIR/opencv.sh +source $DIR/protobuf.sh +source $DIR/cityhash.sh +source $DIR/zmq.sh +source $DIR/lz4.sh diff --git a/tools/dependencies/openblas.sh b/tools/dependencies/openblas.sh new file mode 100755 index 000000000..9463e3325 --- /dev/null +++ b/tools/dependencies/openblas.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of openblas that can be used as dependency of mxnet. +OPENBLAS_VERSION=0.3.3 +if [[ ! -e $DEPS_PATH/lib/libopenblas.a ]]; then + # download and build openblas + >&2 echo "Building openblas..." + + curl -s -L https://github.com/xianyi/OpenBLAS/archive/v$OPENBLAS_VERSION.zip -o $DEPS_PATH/openblas.zip + unzip -q $DEPS_PATH/openblas.zip -d $DEPS_PATH + cd $DEPS_PATH/OpenBLAS-$OPENBLAS_VERSION + + make DYNAMIC_ARCH=1 NO_SHARED=1 USE_OPENMP=1 + make PREFIX=$DEPS_PATH install + cd - + ln -s libopenblas.a $DEPS_PATH/lib/libcblas.a + ln -s libopenblas.a $DEPS_PATH/lib/liblapack.a +fi diff --git a/tools/dependencies/opencv.sh b/tools/dependencies/opencv.sh new file mode 100755 index 000000000..99d0ecb71 --- /dev/null +++ b/tools/dependencies/opencv.sh @@ -0,0 +1,192 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of opencv that can be used as dependency of mxnet. +# It expects openblas, libjpeg, libpng, libtiff, eigen, etc., to be in $DEPS_PATH. +OPENCV_VERSION=3.4.2 +DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )" +if [[ $PLATFORM == 'linux' ]]; then + OPENCV_LAPACK_OPTIONS=" \ + -D OpenBLAS_HOME=$DEPS_PATH \ + -D OpenBLAS_INCLUDE_DIR=$DEPS_PATH/include \ + -D OpenBLAS_LIB=$DEPS_PATH/lib/libopenblas.a \ + -D LAPACK_INCLUDE_DIR=$DEPS_PATH/include \ + -D LAPACK_LINK_LIBRARIES=$DEPS_PATH/lib/ \ + -D LAPACK_LIBRARIES=$DEPS_PATH/lib/libopenblas.a \ + -D LAPACK_CBLAS_H='cblas.h' \ + -D LAPACK_LAPACKE_H='lapacke.h' \ + -D LAPACK_IMPL='OpenBLAS' \ + -D HAVE_LAPACK=1" +fi + +if [[ ! -f $DEPS_PATH/lib/libopencv_core.a ]] || [[ ! -f $DEPS_PATH/lib/libopencv_imgcodecs.a ]] || [[ ! -f $DEPS_PATH/lib/libopencv_imgproc.a ]]; then + # download and build opencv since we need the static library + >&2 echo "Building opencv..." + curl -s -L https://github.com/opencv/opencv/archive/$OPENCV_VERSION.zip -o $DEPS_PATH/opencv.zip + unzip -q $DEPS_PATH/opencv.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/opencv-$OPENCV_VERSION/build + cd $DEPS_PATH/opencv-$OPENCV_VERSION/build + cmake \ + -D OPENCV_ENABLE_NONFREE=OFF \ + -D WITH_1394=OFF \ + -D WITH_ARAVIS=OFF \ + -D WITH_AVFOUNDATION=OFF \ + -D WITH_CAROTENE=OFF \ + -D WITH_CLP=OFF \ + -D WITH_CSTRIPES=OFF \ + -D WITH_CPUFEATURES=OFF \ + -D WITH_CUBLAS=OFF \ + -D WITH_CUDA=OFF \ + -D WITH_CUFFT=OFF \ + -D WITH_DIRECTX=OFF \ + -D WITH_DSHOW=OFF \ + -D WITH_EIGEN=ON \ + -D WITH_FFMPEG=OFF \ + -D WITH_GDAL=OFF \ + -D WITH_GDCM=OFF \ + -D WITH_GIGEAPI=OFF \ + -D WITH_GPHOTO2=OFF \ + -D WITH_GSTREAMER=OFF \ + -D WITH_GSTREAMER_0_10=OFF \ + -D WITH_GTK=OFF \ + -D WITH_GTK_2_X=OFF \ + -D WITH_HALIDE=OFF \ + -D WITH_IMAGEIO=OFF \ + -D WITH_IMGCODEC_HDR=OFF \ + -D WITH_IMGCODEC_PXM=OFF \ + -D WITH_IMGCODEC_SUNRASTER=OFF \ + -D WITH_INF_ENGINE=OFF \ + -D WITH_INTELPERC=OFF \ + -D WITH_IPP=OFF \ + -D WITH_IPP_A=OFF \ + -D WITH_ITT=OFF \ + -D WITH_JASPER=OFF \ + -D WITH_JPEG=ON \ + -D WITH_LAPACK=ON \ + -D WITH_LIBREALSENSE=OFF \ + -D WITH_LIBV4L=OFF \ + -D WITH_MATLAB=OFF \ + -D WITH_MFX=OFF \ + -D WITH_MSMF=OFF \ + -D WITH_NVCUVID=OFF \ + -D WITH_OPENCL=OFF \ + -D WITH_OPENCLAMDBLAS=OFF \ + -D WITH_OPENCLAMDFFT=OFF \ + -D WITH_OPENCL_SVM=OFF \ + -D WITH_OPENEXR=OFF \ + -D WITH_OPENGL=OFF \ + -D WITH_OPENMP=OFF \ + -D WITH_OPENNI=OFF \ + -D WITH_OPENNI2=OFF \ + -D WITH_OPENVX=OFF \ + -D WITH_PNG=ON \ + -D WITH_PROTOBUF=OFF \ + -D WITH_PTHREADS_PF=ON \ + -D WITH_PVAPI=OFF \ + -D WITH_QT=OFF \ + -D WITH_QTKIT=OFF \ + -D WITH_QUICKTIME=OFF \ + -D WITH_TBB=OFF \ + -D WITH_TIFF=ON \ + -D WITH_UNICAP=OFF \ + -D WITH_V4L=OFF \ + -D WITH_VA=OFF \ + -D WITH_VA_INTEL=OFF \ + -D WITH_VFW=OFF \ + -D WITH_VTK=OFF \ + -D WITH_WEBP=OFF \ + -D WITH_WIN32UI=OFF \ + -D WITH_XIMEA=OFF \ + -D WITH_XINE=OFF \ + -D BUILD_ANDROID_EXAMPLES=OFF \ + -D BUILD_ANDROID_PROJECTS=OFF \ + -D BUILD_ANDROID_SERVICE=OFF \ + -D BUILD_CUDA_STUBS=OFF \ + -D BUILD_DOCS=OFF \ + -D BUILD_EXAMPLES=OFF \ + -D BUILD_FAT_JAVA_LIB=OFF \ + -D BUILD_IPP_IW=OFF \ + -D BUILD_ITT_IW=OFF \ + -D BUILD_JAVA=OFF \ + -D BUILD_JASPER=OFF \ + -D BUILD_JPEG=OFF \ + -D BUILD_OPENEXR=OFF \ + -D BUILD_PACKAGE=OFF \ + -D BUILD_PERF_TESTS=OFF \ + -D BUILD_PNG=OFF \ + -D BUILD_SHARED_LIBS=OFF \ + -D BUILD_TBB=OFF \ + -D BUILD_TESTS=OFF \ + -D BUILD_TIFF=OFF \ + -D BUILD_WEBP=OFF \ + -D BUILD_WITH_DEBUG_INFO=OFF \ + -D BUILD_WITH_DYNAMIC_IPP=OFF \ + -D BUILD_WITH_STATIC_CRT=OFF \ + -D BUILD_ZLIB=OFF \ + -D BUILD_opencv_apps=OFF \ + -D BUILD_opencv_aruco=OFF \ + -D BUILD_opencv_calib3d=OFF \ + -D BUILD_opencv_contrib=OFF \ + -D BUILD_opencv_dnn=OFF \ + -D BUILD_opencv_features2d=OFF \ + -D BUILD_opencv_flann=OFF \ + -D BUILD_opencv_gpu=OFF \ + -D BUILD_opencv_gpuarithm=OFF \ + -D BUILD_opencv_gpubgsegm=OFF \ + -D BUILD_opencv_gpucodec=OFF \ + -D BUILD_opencv_gpufeatures2d=OFF \ + -D BUILD_opencv_gpufilters=OFF \ + -D BUILD_opencv_gpuimgproc=OFF \ + -D BUILD_opencv_gpulegacy=OFF \ + -D BUILD_opencv_gpuoptflow=OFF \ + -D BUILD_opencv_gpustereo=OFF \ + -D BUILD_opencv_gpuwarping=OFF \ + -D BUILD_opencv_highgui=OFF \ + -D BUILD_opencv_java=OFF \ + -D BUILD_opencv_js=OFF \ + -D BUILD_opencv_ml=OFF \ + -D BUILD_opencv_ml=OFF \ + -D BUILD_opencv_nonfree=OFF \ + -D BUILD_opencv_objdetect=OFF \ + -D BUILD_opencv_photo=OFF \ + -D BUILD_opencv_python=OFF \ + -D BUILD_opencv_python2=OFF \ + -D BUILD_opencv_python3=OFF \ + -D BUILD_opencv_superres=OFF \ + -D BUILD_opencv_video=OFF \ + -D BUILD_opencv_videoio=OFF \ + -D BUILD_opencv_videostab=OFF \ + -D BUILD_opencv_viz=OFF \ + -D BUILD_opencv_world=OFF \ + $OPENCV_LAPACK_OPTIONS \ + -D OPENCV_LIB_INSTALL_PATH=lib \ + -D OPENCV_INCLUDE_INSTALL_PATH=include \ + -D CMAKE_LIBRARY_PATH=$DEPS_PATH/lib \ + -D CMAKE_INCLUDE_PATH=$DEPS_PATH/include \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH .. + if [[ $PLATFORM == 'linux' ]]; then + cp $DIR/patch/opencv_lapack.h ./ + fi + make + make install + cd - + # @szha: compatibility header + cat $DEPS_PATH/include/opencv2/imgcodecs/imgcodecs_c.h >> $DEPS_PATH/include/opencv2/imgcodecs.hpp +fi diff --git a/tools/dependencies/openssl.sh b/tools/dependencies/openssl.sh new file mode 100755 index 000000000..b7e4317d4 --- /dev/null +++ b/tools/dependencies/openssl.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of openssl that can be used as dependency of mxnet. +OPENSSL_VERSION=1.0.2l +if [[ ! -f $DEPS_PATH/lib/libssl.a ]] || [[ ! -f $DEPS_PATH/lib/libcrypto.a ]]; then + # download and build openssl + >&2 echo "Building openssl..." + OPENSSL_VERSION=$(echo $OPENSSL_VERSION | sed 's/\./_/g') + curl -s -L https://github.com/openssl/openssl/archive/OpenSSL_$OPENSSL_VERSION.zip -o $DEPS_PATH/openssl.zip + unzip -q $DEPS_PATH/openssl.zip -d $DEPS_PATH + cd $DEPS_PATH/openssl-OpenSSL_$OPENSSL_VERSION + if [[ $PLATFORM == 'linux' ]]; then + TARGET=linux-x86_64 + elif [[ $PLATFORM == 'darwin' ]]; then + TARGET=darwin64-x86_64-cc + fi + ./Configure no-shared no-zlib --prefix=$DEPS_PATH --openssldir=$DEPS_PATH/ssl $TARGET + make + make install + cd - +fi diff --git a/tools/dependencies/patch/opencv_lapack.h b/tools/dependencies/patch/opencv_lapack.h new file mode 100644 index 000000000..97af9d67e --- /dev/null +++ b/tools/dependencies/patch/opencv_lapack.h @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +extern "C" { +#include "cblas.h" +#include "lapacke.h" +} diff --git a/tools/dependencies/protobuf.sh b/tools/dependencies/protobuf.sh new file mode 100755 index 000000000..156470104 --- /dev/null +++ b/tools/dependencies/protobuf.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of protobuf along with protoc, that can be used as dependency of mxnet. +PROTOBUF_VERSION=3.5.1 +if [[ $PLATFORM == 'darwin' ]]; then + DY_EXT="dylib" +else + DY_EXT="so" +fi + +LIBPROTOBUF="$DEPS_PATH/lib/libprotobuf.$DY_EXT" +LIBPROTOC="$DEPS_PATH/lib/libprotoc.$DY_EXT" +if [[ ! -e $LIBPROTOBUF ]] || [[ ! -e $LIBPROTOC ]]; then + # Download and build protobuf + >&2 echo "Building protobuf..." + curl -s -L https://github.com/google/protobuf/archive/v$PROTOBUF_VERSION.zip -o $DEPS_PATH/protobuf.zip + unzip -q $DEPS_PATH/protobuf.zip -d $DEPS_PATH + cd $DEPS_PATH/protobuf-$PROTOBUF_VERSION + ./autogen.sh + ./configure -prefix=$DEPS_PATH + make + make install + cd - +fi + +export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$(dirname $(find $DEPS_PATH -type f -name 'libprotoc*' | grep protobuf | head -n 1)):$DEPS_PATH/lib diff --git a/tools/dependencies/zmq.sh b/tools/dependencies/zmq.sh new file mode 100755 index 000000000..55e17798c --- /dev/null +++ b/tools/dependencies/zmq.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script builds the static library of zeroMQ that can be used as dependency of mxnet. +ZEROMQ_VERSION=4.2.2 +if [[ ! -f $DEPS_PATH/lib/libzmq.a ]]; then + # Download and build zmq + >&2 echo "Building zmq..." + curl -s -L https://github.com/zeromq/libzmq/archive/v$ZEROMQ_VERSION.zip -o $DEPS_PATH/zeromq.zip + unzip -q $DEPS_PATH/zeromq.zip -d $DEPS_PATH + mkdir -p $DEPS_PATH/libzmq-$ZEROMQ_VERSION/build + cd $DEPS_PATH/libzmq-$ZEROMQ_VERSION/build + cmake \ + -D CMAKE_BUILD_TYPE=RELEASE \ + -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ + -D WITH_LIBSODIUM=OFF \ + -D BUILD_SHARED_LIBS=OFF .. + make + make install + cp $DEPS_PATH/lib/x86_64-linux-gnu/libzmq.a $DEPS_PATH/lib/libzmq.a + cd - +fi diff --git a/tools/license_header.py b/tools/license_header.py index 2f12a1479..199d56c7e 100755 --- a/tools/license_header.py +++ b/tools/license_header.py @@ -24,11 +24,11 @@ - add the default license header to source files that do not contain a valid license: - python license_header.py add + license_header.py add - check if every files has a license header - python license_header.py check + license_header.py check """ import re @@ -37,6 +37,7 @@ from itertools import chain import logging import sys +import subprocess # the default apache license _LICENSE = """Licensed to the Apache Software Foundation (ASF) under one @@ -60,24 +61,35 @@ _LICENSE_PATTERNS = ['Licensed to the Apache Software Foundation'] # the folders or files that will be ignored -_WHITE_LIST = ['R-package/', +_WHITE_LIST = [ + # Licensed under docker/Dockerfiles/License.md 'docker/Dockerfiles', + + # Git submodules under different licenses '3rdparty', - 'src/operator/mkl/', + + # Code shared with project by author - see file for details 'src/operator/special_functions-inl.h', + + # Code generated by scala-package, checked in, and verified + 'scala-package/init-native/src/main/native/org_apache_mxnet_init_native_c_api.h', + 'scala-package/native/src/main/native/org_apache_mxnet_native_c_api.h', + + # Licensed under Caffe header 'src/operator/nn/pool.h', 'src/operator/contrib/psroi_pooling-inl.h', 'src/operator/contrib/nn/deformable_im2col.h', 'src/operator/contrib/nn/deformable_im2col.cuh', 'src/operator/nn/im2col.h', 'src/operator/nn/im2col.cuh', + + # Licenses in headers + 'docs/_static/searchtools_custom.js', + 'docs/_static/js/clipboard.js', + 'docs/_static/js/clipboard.min.js', + + # Licensed under 2-Clause BSD in header 'example/ssd/dataset/pycocotools/coco.py', - 'example/rcnn/rcnn/cython/setup.py', - 'example/rcnn/rcnn/cython/nms_kernel.cu', - 'prepare_mkl.sh', - 'example/image-classification/predict-cpp/image-classification-predict.cc', - 'src/operator/contrib/ctc_include/', - 'julia/REQUIRE' ] # language extensions and the according commment mark @@ -85,14 +97,28 @@ '.pm':'#', '.scala':'*', '.cc':'*', '.sh':'#', '.cmake':'#', '.java':'*', '.sh':'#', '.cpp':'*', '.hpp':'*', '.c':'*', '.bat':'rem', '.pl':'#', '.m':'%', '.R':'#', '.mk':'#', '.cfg':'#', - '.t':'#', '.ps1':'#', '.jl':'#'} + '.t':'#', '.ps1':'#', '.jl':'#', '.clj':';;', '.pyx':'#', '.js':'*'} # Previous license header, which will be removed _OLD_LICENSE = re.compile('.*Copyright.*by Contributors') -def _has_license(lines): + +def get_mxnet_root(): + curpath = os.path.abspath(os.path.dirname(__file__)) + def is_mxnet_root(path: str) -> bool: + return os.path.exists(os.path.join(path, ".mxnet_root")) + while not is_mxnet_root(curpath): + parent = os.path.abspath(os.path.join(curpath, os.pardir)) + if parent == curpath: + raise RuntimeError("Got to the root and couldn't find a parent folder with .mxnet_root") + curpath = parent + return curpath + + +def _lines_have_license(lines): return any([any([p in l for p in _LICENSE_PATTERNS]) for l in lines]) + def _get_license(comment_mark): if comment_mark == '*': body = '/*\n' @@ -111,29 +137,41 @@ def _get_license(comment_mark): body += '\n' return body -def _valid_file(fname, verbose=False): + +def should_have_license(fname): if any([l in fname for l in _WHITE_LIST]): - if verbose: - logging.info('skip ' + fname + ', it matches the white list') + logging.debug('skip ' + fname + ', it matches the white list') return False _, ext = os.path.splitext(fname) if ext not in _LANGS: - if verbose: - logging.info('skip ' + fname + ', unknown file extension') + logging.debug('skip ' + fname + ', unknown file extension') return False return True -def process_file(fname, action, verbose=True): - if not _valid_file(fname, verbose): + +def file_has_license(fname): + if not should_have_license(fname): + return True + try: + with open(fname, 'r', encoding="utf-8") as f: + lines = f.readlines() + if not lines or _lines_have_license(lines): + return True + else: + logging.error("File %s doesn't have a license", fname) + return False + except UnicodeError: return True + return True + + +def file_add_license(fname): + if not should_have_license(fname): + return with open(fname, 'r', encoding="utf-8") as f: lines = f.readlines() - if not lines: - return True - if _has_license(lines): - return True - elif action == 'check': - return False + if _lines_have_license(lines): + return _, ext = os.path.splitext(fname) with open(fname, 'w', encoding="utf-8") as f: # shebang line @@ -144,29 +182,43 @@ def process_file(fname, action, verbose=True): for l in lines: f.write(l.rstrip()+'\n') logging.info('added license header to ' + fname) - return True + return -def process_folder(root, action): - excepts = [] - for root, _, files in os.walk(root): - for f in files: - fname = os.path.normpath(os.path.join(root, f)) - if not process_file(fname, action): - excepts.append(fname) - if action == 'check' and excepts: - logging.warning('The following files do not contain a valid license, '+ - 'you can use `python tools/license_header.py add [file]` to add'+ - 'them automatically: ') - for x in excepts: - logging.warning(x) - return False - return True -if __name__ == '__main__': - logging.getLogger().setLevel(logging.INFO) - logging.basicConfig(format='%(asctime)-15s %(message)s') +def under_git(): + return subprocess.run(['git', 'rev-parse', 'HEAD'], + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode == 0 + + +def git_files(): + return list(map(os.fsdecode, + subprocess.check_output('git ls-tree -r HEAD --name-only -z'.split()).split(b'\0'))) + + +def file_generator(path: str): + for (dirpath, dirnames, files) in os.walk(path): + for file in files: + yield os.path.abspath(os.path.join(dirpath, file)) + + +def foreach(fn, iterable): + for x in iterable: + fn(x) + + +def script_name(): + """:returns: script name with leading paths removed""" + return os.path.split(sys.argv[0])[1] + + +def main(): + logging.basicConfig( + format='{}: %(levelname)s %(message)s'.format(script_name()), + level=os.environ.get("LOGLEVEL", "INFO")) + parser = argparse.ArgumentParser( description='Add or check source license header') + parser.add_argument( 'action', nargs=1, type=str, choices=['add', 'check'], default='add', @@ -177,19 +229,26 @@ def process_folder(root, action): help='Files to add license header to') args = parser.parse_args() - files = list(chain(*args.file)) action = args.action[0] - has_license = True - if len(files) > 0: - for file in files: - has_license = process_file(file, action) - if action == 'check' and not has_license: - logging.warn("{} doesn't have a license".format(file)) - has_license = False - else: - has_license = process_folder(os.path.join(os.path.dirname(__file__), '..'), action) - if not has_license: - sys.exit(1) + files = list(chain(*args.file)) + if not files and action =='check': + if under_git(): + logging.info("Git detected: Using files under version control") + files = git_files() + else: + logging.info("Using files under mxnet sources root") + files = file_generator(get_mxnet_root()) + + if action == 'check': + if not all(map(file_has_license, files)): + return 1 + else: + logging.info("All known and whitelisted files have license") + return 0 else: - sys.exit(0) + assert action == 'add' + foreach(file_add_license, files) + return 0 +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/pip/MANIFEST.in b/tools/pip/MANIFEST.in new file mode 100644 index 000000000..8037b6a10 --- /dev/null +++ b/tools/pip/MANIFEST.in @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +include README +include mxnet/COMMIT_HASH +recursive-include mxnet/tools * +recursive-include mxnet *.py +recursive-include mxnet *.so +recursive-include mxnet *.so.* +recursive-include mxnet *.dylib +recursive-include mxnet *_LICENSE +recursive-include mxnet *.h +recursive-include mxnet *.cuh +recursive-include dmlc_tracker *.py diff --git a/tools/pip/sanity_test.py b/tools/pip/sanity_test.py new file mode 100644 index 000000000..dc51e4799 --- /dev/null +++ b/tools/pip/sanity_test.py @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# coding: utf-8 +"""Sanity test.""" +from __future__ import print_function +import sys +from base64 import b64decode + +try: + import mxnet as mx + mx.img.imdecode(b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==')).asnumpy() + print('Test succeeded') +except: + import traceback + print('Test failed') + traceback.print_exc() + sys.exit(1) diff --git a/tools/pip/setup.py b/tools/pip/setup.py new file mode 100644 index 000000000..d5db6d87f --- /dev/null +++ b/tools/pip/setup.py @@ -0,0 +1,195 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# coding: utf-8 +# pylint: disable=invalid-name, exec-used +"""Setup mxnet package for pip.""" +from __future__ import absolute_import +from datetime import datetime +import os +import sys +import shutil +import platform + +if platform.system() == 'Linux': + sys.argv.append('--universal') + sys.argv.append('--plat-name=manylinux1_x86_64') + +from setuptools import setup, find_packages +from setuptools.dist import Distribution + +# We can not import `mxnet.info.py` in setup.py directly since mxnet/__init__.py +# Will be invoked which introduces dependences +CURRENT_DIR = os.path.dirname(__file__) +libinfo_py = os.path.join(CURRENT_DIR, 'mxnet-build/python/mxnet/libinfo.py') +libinfo = {'__file__': libinfo_py} +exec(compile(open(libinfo_py, "rb").read(), libinfo_py, 'exec'), libinfo, libinfo) + +LIB_PATH = libinfo['find_lib_path']() +__version__ = libinfo['__version__'] +if 'TRAVIS_TAG' not in os.environ or not os.environ['TRAVIS_TAG'].strip(): + __version__ += 'b{0}'.format(datetime.today().strftime('%Y%m%d')) +elif 'TRAVIS_TAG' in os.environ and os.environ['TRAVIS_TAG'].startswith('patch-'): + __version__ = os.environ['TRAVIS_TAG'].split('-')[1] + +class BinaryDistribution(Distribution): + def has_ext_modules(self): + return platform.system() == 'Darwin' + + +DEPENDENCIES = [ + 'numpy<1.15.0,>=1.8.2', + 'requests>=2.20.0', + 'graphviz<0.9.0,>=0.8.1' +] + +shutil.rmtree(os.path.join(CURRENT_DIR, 'mxnet'), ignore_errors=True) +shutil.rmtree(os.path.join(CURRENT_DIR, 'dmlc_tracker'), ignore_errors=True) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/python/mxnet'), + os.path.join(CURRENT_DIR, 'mxnet')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/dmlc-core/tracker/dmlc_tracker'), + os.path.join(CURRENT_DIR, 'dmlc_tracker')) +shutil.copy(LIB_PATH[0], os.path.join(CURRENT_DIR, 'mxnet')) + +# copy tools to mxnet package +shutil.rmtree(os.path.join(CURRENT_DIR, 'mxnet/tools'), ignore_errors=True) +os.mkdir(os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copy(os.path.join(CURRENT_DIR, 'mxnet-build/tools/launch.py'), os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copy(os.path.join(CURRENT_DIR, 'mxnet-build/tools/im2rec.py'), os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copy(os.path.join(CURRENT_DIR, 'mxnet-build/tools/kill-mxnet.py'), os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copy(os.path.join(CURRENT_DIR, 'mxnet-build/tools/parse_log.py'), os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copy(os.path.join(CURRENT_DIR, 'mxnet-build/tools/diagnose.py'), os.path.join(CURRENT_DIR, 'mxnet/tools')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/tools/caffe_converter'), os.path.join(CURRENT_DIR, 'mxnet/tools/caffe_converter')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/tools/bandwidth'), os.path.join(CURRENT_DIR, 'mxnet/tools/bandwidth')) + +# copy headers to mxnet package +shutil.rmtree(os.path.join(CURRENT_DIR, 'mxnet/include'), ignore_errors=True) +os.mkdir(os.path.join(CURRENT_DIR, 'mxnet/include')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/include/mxnet'), + os.path.join(CURRENT_DIR, 'mxnet/include/mxnet')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/dlpack/include/dlpack'), + os.path.join(CURRENT_DIR, 'mxnet/include/dlpack')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/dmlc-core/include/dmlc'), + os.path.join(CURRENT_DIR, 'mxnet/include/dmlc')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/mshadow/mshadow'), + os.path.join(CURRENT_DIR, 'mxnet/include/mshadow')) +shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/tvm/nnvm/include/nnvm'), + os.path.join(CURRENT_DIR, 'mxnet/include/nnvm')) + +package_name = 'mxnet' + +variant = os.environ['mxnet_variant'].upper() +if variant != 'CPU': + package_name = 'mxnet_{0}'.format(variant.lower()) + +with open('doc/PYPI_README.md') as readme_file: + long_description = readme_file.read() + +with open('doc/{0}_ADDITIONAL.md'.format(variant)) as variant_doc: + long_description = long_description + variant_doc.read() + +# pypi only supports rst, so use pandoc to convert +import pypandoc +if platform.system() == 'Darwin': + pypandoc.download_pandoc() +long_description = pypandoc.convert_text(long_description, 'rst', 'md') +short_description = 'MXNet is an ultra-scalable deep learning framework.' +libraries = [] +if variant == 'CPU': + libraries.append('openblas') +else: + if variant.startswith('CU92'): + libraries.append('CUDA-9.2') + elif variant.startswith('CU91'): + libraries.append('CUDA-9.1') + elif variant.startswith('CU90'): + libraries.append('CUDA-9.0') + elif variant.startswith('CU80'): + libraries.append('CUDA-8.0') + elif variant.startswith('CU75'): + libraries.append('CUDA-7.5') + if variant.endswith('MKL'): + libraries.append('MKLDNN') + +short_description += ' This version uses {0}.'.format(' and '.join(libraries)) + +package_data = {'mxnet': [os.path.join('mxnet', os.path.basename(LIB_PATH[0]))], + 'dmlc_tracker': []} +if variant.endswith('MKL'): + if platform.system() == 'Darwin': + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libmklml.dylib'), os.path.join(CURRENT_DIR, 'mxnet')) + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libiomp5.dylib'), os.path.join(CURRENT_DIR, 'mxnet')) + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libmkldnn.0.dylib'), os.path.join(CURRENT_DIR, 'mxnet')) + package_data['mxnet'].append('mxnet/libmklml.dylib') + package_data['mxnet'].append('mxnet/libiomp5.dylib') + package_data['mxnet'].append('mxnet/libmkldnn.0.dylib') + else: + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libmklml_intel.so'), os.path.join(CURRENT_DIR, 'mxnet')) + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libiomp5.so'), os.path.join(CURRENT_DIR, 'mxnet')) + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libmkldnn.so.0'), os.path.join(CURRENT_DIR, 'mxnet')) + package_data['mxnet'].append('mxnet/libmklml_intel.so') + package_data['mxnet'].append('mxnet/libiomp5.so') + package_data['mxnet'].append('mxnet/libmkldnn.so.0') + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), '../MKLML_LICENSE'), os.path.join(CURRENT_DIR, 'mxnet')) + package_data['mxnet'].append('mxnet/MKLML_LICENSE') +if platform.system() == 'Linux': + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libgfortran.so.3'), os.path.join(CURRENT_DIR, 'mxnet')) + package_data['mxnet'].append('mxnet/libgfortran.so.3') + shutil.copy(os.path.join(os.path.dirname(LIB_PATH[0]), 'libquadmath.so.0'), os.path.join(CURRENT_DIR, 'mxnet')) + package_data['mxnet'].append('mxnet/libquadmath.so.0') + +from mxnet.base import _generate_op_module_signature +from mxnet.ndarray.register import _generate_ndarray_function_code +from mxnet.symbol.register import _generate_symbol_function_code +_generate_op_module_signature('mxnet', 'symbol', _generate_symbol_function_code) +_generate_op_module_signature('mxnet', 'ndarray', _generate_ndarray_function_code) + +setup(name=package_name, + version=__version__, + long_description=long_description, + description=short_description, + zip_safe=False, + packages=find_packages(), + package_data=package_data, + include_package_data=True, + install_requires=DEPENDENCIES, + distclass=BinaryDistribution, + license='Apache 2.0', + classifiers=[ # https://pypi.org/pypi?%3Aaction=list_classifiers + 'Development Status :: 5 - Production/Stable', + 'Intended Audience :: Developers', + 'Intended Audience :: Education', + 'Intended Audience :: Science/Research', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: C++', + 'Programming Language :: Cython', + 'Programming Language :: Other', # R, Scala + 'Programming Language :: Perl', + 'Programming Language :: Python', + 'Programming Language :: Python :: 2.7', + 'Programming Language :: Python :: 3.4', + 'Programming Language :: Python :: 3.5', + 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: Implementation :: CPython', + 'Topic :: Scientific/Engineering', + 'Topic :: Scientific/Engineering :: Artificial Intelligence', + 'Topic :: Scientific/Engineering :: Mathematics', + 'Topic :: Software Development', + 'Topic :: Software Development :: Libraries', + 'Topic :: Software Development :: Libraries :: Python Modules', + ], + url='https://github.com/apache/incubator-mxnet') diff --git a/tools/pip_package/README.md b/tools/pip_package/README.md deleted file mode 100644 index f289c98b7..000000000 --- a/tools/pip_package/README.md +++ /dev/null @@ -1,9 +0,0 @@ -MXNet Python Package -==================== -MXNet is a deep learning framework designed for both *efficiency* and *flexibility*. -It allows you to mix the flavours of deep learning programs together to maximize the efficiency and your productivity. - - -Installation ------------- -To install, check [Build Instruction](http://mxnet.io/get_started/setup.html) diff --git a/tools/pip_package/make_pip_package.sh b/tools/pip_package/make_pip_package.sh deleted file mode 100755 index 46b4938b0..000000000 --- a/tools/pip_package/make_pip_package.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/usr/bin/env bash - -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - - -# Assuming the script is run at mxnet/tools/pip_package -# This script builds from scratch the dependencies of mxnet into static -# librareis and statically links them to produce a (mostly) standalone -# libmxnet.so, then packages it into the python wheel. -# It assumes the build environment to be a sandbox that doesn't have the .so -# objects for the dependencies, i.e. zlib, openblas, libjpeg, libpng, libtiff -# and opencv. - -# Install necessary build tools -if [ -n "$(command -v apt-get)" ]; then - sudo apt-get update; - sudo apt-get install -y build-essential git python-pip zip pkg-config cmake -elif [ -n "$(command -v yum)" ]; then - sudo yum install -y cmake - sudo yum groupinstall -y "Development Tools" - sudo yum install -y python27 python27-setuptools python27-tools python-pip -else - echo "Need a package manager to install build tools, e.g. apt/yum" - exit 1 -fi -sudo pip install -U pip setuptools wheel - -# Set up path as temporary working directory -DEPS_PATH=$PWD/../../deps -mkdir $DEPS_PATH - -# Dependencies can be updated here. Be sure to verify the download link before -# changing. The dependencies are: -ZLIB_VERSION=1.2.6 -OPENBLAS_VERSION=0.2.19 -JPEG_VERSION=8.4.0 -PNG_VERSION=1.5.10 -TIFF_VERSION=3.8.2 -OPENCV_VERSION=2.4.13 - -# Setup path to dependencies -export PKG_CONFIG_PATH=$DEPS_PATH/lib/pkgconfig:$DEPS_PATH/lib64/pkgconfig:$PKG_CONFIG_PATH -export CPATH=$DEPS_PATH/include:$CPATH - -# Position Independent code must be turned on for statically linking .a -export CC="gcc -fPIC" -export CXX="g++ -fPIC" - -# Download and build zlib -curl -L https://github.com/LuaDist/zlib/archive/$ZLIB_VERSION.zip -o $DEPS_PATH/zlib.zip -unzip $DEPS_PATH/zlib.zip -d $DEPS_PATH -mkdir $DEPS_PATH/zlib-$ZLIB_VERSION/build -cd $DEPS_PATH/zlib-$ZLIB_VERSION/build -cmake -D CMAKE_BUILD_TYPE=RELEASE \ - -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ - -D BUILD_SHARED_LIBS=OFF .. -make -j$(nproc) -make install -cd - - -# download and build openblas -curl -L https://github.com/xianyi/OpenBLAS/archive/v$OPENBLAS_VERSION.zip -o $DEPS_PATH/openblas.zip -unzip $DEPS_PATH/openblas.zip -d $DEPS_PATH -cd $DEPS_PATH/OpenBLAS-$OPENBLAS_VERSION -make FC=gfortran -j $(($(nproc) + 1)) -make PREFIX=$DEPS_PATH install -cd - -ln -s $DEPS_PATH/lib/libopenblas_haswellp-r0.2.19.a $DEPS_PATH/lib/libcblas.a - -# download and build libjpeg -curl -L https://github.com/LuaDist/libjpeg/archive/$JPEG_VERSION.zip -o $DEPS_PATH/libjpeg.zip -unzip $DEPS_PATH/libjpeg.zip -d $DEPS_PATH -cd $DEPS_PATH/libjpeg-$JPEG_VERSION -./configure --disable-shared --prefix=$DEPS_PATH -make -j$(nproc) -make test -make install -cd - - -# download and build libpng -curl -L https://github.com/LuaDist/libpng/archive/$PNG_VERSION.zip -o $DEPS_PATH/libpng.zip -unzip $DEPS_PATH/libpng.zip -d $DEPS_PATH -mkdir $DEPS_PATH/libpng-$PNG_VERSION/build -cd $DEPS_PATH/libpng-$PNG_VERSION/build -cmake -D CMAKE_BUILD_TYPE=RELEASE \ - -D CMAKE_INSTALL_PREFIX=$DEPS_PATH \ - -D PNG_CONFIGURE_LIBPNG=-fPIC \ - -D BUILD_SHARED_LIBS=OFF .. -make -j$(nproc) -make install -cd - - -# download and build libtiff -curl -L https://github.com/LuaDist/libtiff/archive/$TIFF_VERSION.zip -o $DEPS_PATH/libtiff.zip -unzip $DEPS_PATH/libtiff.zip -d $DEPS_PATH -cd $DEPS_PATH/libtiff-$TIFF_VERSION -./configure --disable-shared --prefix=$DEPS_PATH -make -j$(nproc) -make install -cd - - -# download and build opencv since we need the static library -curl -L https://github.com/Itseez/opencv/archive/$OPENCV_VERSION.zip -o $DEPS_PATH/opencv.zip -unzip $DEPS_PATH/opencv.zip -d $DEPS_PATH -mkdir $DEPS_PATH/opencv-$OPENCV_VERSION/build -cd $DEPS_PATH/opencv-$OPENCV_VERSION/build -cmake -D WITH_1394=OFF \ - -D WITH_AVFOUNDATION=OFF \ - -D WITH_CUDA=OFF \ - -D WITH_VTK=OFF \ - -D WITH_CUFFT=OFF \ - -D WITH_CUBLAS=OFF \ - -D WITH_NVCUVID=OFF \ - -D WITH_EIGEN=ON \ - -D WITH_VFW=OFF \ - -D WITH_FFMPEG=OFF \ - -D WITH_GSTREAMER=OFF \ - -D WITH_GTK=OFF \ - -D WITH_JASPER=OFF \ - -D WITH_JPEG=ON \ - -D WITH_PNG=ON \ - -D WITH_QUICKTIME=OFF \ - -D WITH_TBB=ON \ - -D WITH_TIFF=OFF \ - -D WITH_V4L=OFF \ - -D WITH_LIBV4L=OFF \ - -D WITH_DSHOW=OFF \ - -D WITH_MSMF=OFF \ - -D WITH_OPENCL=OFF \ - -D WITH_OPENCLAMDFFT=OFF \ - -D WITH_OPENCLAMDBLAS=OFF \ - -D BUILD_SHARED_LIBS=OFF \ - -D BUILD_opencv_apps=OFF \ - -D BUILD_opencv_gpu=OFF \ - -D BUILD_opencv_video=OFF \ - -D BUILD_opencv_contrib=OFF \ - -D BUILD_opencv_nonfree=OFF \ - -D BUILD_opencv_flann=OFF \ - -D BUILD_opencv_features2d=OFF \ - -D BUILD_opencv_calib3d=OFF \ - -D BUILD_opencv_objdetect=OFF \ - -D BUILD_opencv_ml=OFF \ - -D BUILD_opencv_photo=OFF \ - -D BUILD_DOCS=OFF \ - -D BUILD_PACKAGE=OFF \ - -D CMAKE_BUILD_TYPE=RELEASE \ - -D CMAKE_INSTALL_PREFIX=$DEPS_PATH .. -make -j $(nproc) -make install # user will always have access to home, so no sudo needed -cd - - -# Although .so building is explicitly turned off for most libraries, sometimes -# they still get created. So, remove them just to make sure they don't -# interfere, or otherwise we might get libmxnet.so that is not self-contained. -rm $DEPS_PATH/{lib,lib64}/*.{so,so.0} - -# Go to the parent path and build mxnet -cd ../../ -cp make/pip_$(uname | tr '[:upper:]' '[:lower:]')_cpu.mk config.mk -make -j $(nproc) - -# Generate wheel. The output is in the mxnet/tools/pip_package/dist path. -cd tools/pip_package -python setup.py bdist_wheel diff --git a/tools/pip_package/setup.py b/tools/pip_package/setup.py deleted file mode 100644 index e4bf48236..000000000 --- a/tools/pip_package/setup.py +++ /dev/null @@ -1,60 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -# pylint: disable=invalid-name, exec-used -"""Setup mxnet package.""" -from __future__ import absolute_import -import os -import shutil - -from setuptools import setup, find_packages -from setuptools.dist import Distribution - -# We can not import `mxnet.info.py` in setup.py directly since mxnet/__init__.py -# Will be invoked which introduces dependences -CURRENT_DIR = os.path.dirname(__file__) -libinfo_py = os.path.join(CURRENT_DIR, '../../python/mxnet/libinfo.py') -libinfo = {'__file__': libinfo_py} -exec(compile(open(libinfo_py, "rb").read(), libinfo_py, 'exec'), libinfo, libinfo) - -LIB_PATH = libinfo['find_lib_path']() -__version__ = libinfo['__version__'] - -class BinaryDistribution(Distribution): - def has_ext_modules(self): - return True - - -DEPENDENCIES = [ - 'numpy', -] - -shutil.rmtree(os.path.join(CURRENT_DIR, 'mxnet'), ignore_errors=True) -shutil.copytree(os.path.join(CURRENT_DIR, '../../python/mxnet'), - os.path.join(CURRENT_DIR, 'mxnet')) -shutil.copy(LIB_PATH[0], os.path.join(CURRENT_DIR, 'mxnet')) - -setup(name='mxnet', - version=__version__, - description=open(os.path.join(CURRENT_DIR, 'README.md')).read(), - zip_safe=False, - packages=find_packages(), - package_data={'mxnet': [os.path.join('mxnet', os.path.basename(LIB_PATH[0]))]}, - include_package_data=True, - install_requires=DEPENDENCIES, - distclass=BinaryDistribution, - url='https://github.com/dmlc/mxnet') diff --git a/tools/setup_gpu_build_tools.sh b/tools/setup_gpu_build_tools.sh new file mode 100755 index 000000000..44b44c574 --- /dev/null +++ b/tools/setup_gpu_build_tools.sh @@ -0,0 +1,254 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# This script installs the tools and libraries for CUDA GPU on Ubuntu. +# Usage: VARIANT=cu92mkl; DEPS_PATH=$HOME; setup_gpu_build_tools.sh $VARIANT $DEPS_PATH; +# It installs the tools into DEPS_PATH as specified by the second argument, and will set +# the following environment variables: +# PATH, CPLUS_INCLUDE_PATH, C_INCLUDE_PATH, LIBRARY_PATH, LD_LIBRARY_PATH, NVCC + +VARIANT=$1 +DEPS_PATH=$2 + +>&2 echo "Setting CUDA versions for $VARIANT" +if [[ $VARIANT == cu100* ]]; then + CUDA_VERSION='10.0.130-1' + CUDA_PATCH_VERSION='10.0.130-1' + LIBCUDA_VERSION='410.48-0ubuntu1' + LIBCUDNN_VERSION='7.3.1.20-1+cuda10.0' + LIBNCCL_VERSION='2.3.4-1+cuda9.2' +elif [[ $VARIANT == cu92* ]]; then + CUDA_VERSION='9.2.148-1' + CUDA_PATCH_VERSION='9.2.148.1-1' + LIBCUDA_VERSION='396.44-0ubuntu1' + LIBCUDNN_VERSION='7.3.1.20-1+cuda9.2' + LIBNCCL_VERSION='2.3.4-1+cuda9.2' +elif [[ $VARIANT == cu91* ]]; then + CUDA_VERSION='9.1.85-1' + CUDA_PATCH_VERSION='9.1.85.3-1' + LIBCUDA_VERSION='396.44-0ubuntu1' + LIBCUDNN_VERSION='7.1.3.16-1+cuda9.1' + LIBNCCL_VERSION='2.2.12-1+cuda9.1' +elif [[ $VARIANT == cu90* ]]; then + CUDA_VERSION='9.0.176-1' + CUDA_PATCH_VERSION='9.0.176.3-1' + LIBCUDA_VERSION='384.145-0ubuntu1' + LIBCUDNN_VERSION='7.3.1.20-1+cuda9.0' + LIBNCCL_VERSION='2.3.4-1+cuda9.0' +elif [[ $VARIANT == cu80* ]]; then + CUDA_VERSION='8.0.61-1' + CUDA_PATCH_VERSION='8.0.61.2-1' + LIBCUDA_VERSION='375.88-0ubuntu1' + LIBCUDNN_VERSION='7.2.1.38-1+cuda8.0' + LIBNCCL_VERSION='2.3.4-1+cuda8.0' +elif [[ $VARIANT == cu75* ]]; then + CUDA_VERSION='7.5-18' + CUDA_PATCH_VERSION='7.5-18' + LIBCUDA_VERSION='375.88-0ubuntu1' + LIBCUDNN_VERSION='6.0.21-1+cuda7.5' + LIBNCCL_VERSION='' +fi +if [[ $VARIANT == cu* ]]; then + CUDA_MAJOR_VERSION=$(echo $CUDA_VERSION | tr '-' '.' | cut -d. -f1,2) + CUDA_MAJOR_DASH=$(echo $CUDA_VERSION | tr '-' '.' | cut -d. -f1,2 | tr '.' '-') + NVIDIA_MAJOR_VERSION=$(echo $LIBCUDA_VERSION | cut -d. -f1) + LIBCUDA_MAJOR=$(echo $LIBCUDA_VERSION | cut -d. -f1) + LIBCUDNN_MAJOR=$(echo $LIBCUDNN_VERSION | cut -d. -f1) + os_name=$(cat /etc/*release | grep '^ID=' | sed 's/^.*=//g') + os_version=$(cat /etc/*release | grep VERSION_ID | sed 's/^.*"\([0-9]*\)\.\([0-9]*\)"/\1\2/g') + os_id="${os_name}${os_version}" + if [[ $CUDA_MAJOR_DASH == 9-* ]] || [[ $CUDA_MAJOR_DASH == 10-* ]]; then + os_id="ubuntu1604" + fi + export PATH=/usr/lib/binutils-2.26/bin/:${PATH}:$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/bin + export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/include + export C_INCLUDE_PATH=${C_INCLUDE_PATH}:$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/include + export LIBRARY_PATH=${LIBRARY_PATH}:$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/lib64:$DEPS_PATH/usr/lib/x86_64-linux-gnu:$DEPS_PATH/usr/lib/nvidia-$NVIDIA_MAJOR_VERSION + export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/lib64:$DEPS_PATH/usr/lib/x86_64-linux-gnu:$DEPS_PATH/usr/lib/nvidia-$NVIDIA_MAJOR_VERSION + export NVCC=$DEPS_PATH/usr/local/cuda-$CUDA_MAJOR_VERSION/bin/nvcc +fi + +# list of debs to download from nvidia +if [[ $VARIANT == cu100* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvcc-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + "libnccl-dev_${LIBNCCL_VERSION}_amd64.deb" \ + ) +elif [[ $VARIANT == cu92* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvcc-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + "libnccl-dev_${LIBNCCL_VERSION}_amd64.deb" \ + ) +elif [[ $VARIANT == cu91* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvcc-${CUDA_MAJOR_DASH}_9.1.85.2-1_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + "libnccl-dev_${LIBNCCL_VERSION}_amd64.deb" \ + ) +elif [[ $VARIANT == cu90* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_9.0.176.4-1_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_9.0.176.4-1_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + "libnccl-dev_${LIBNCCL_VERSION}_amd64.deb" \ + ) +elif [[ $VARIANT == cu80* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + "libnccl-dev_${LIBNCCL_VERSION}_amd64.deb" \ + ) +elif [[ $VARIANT == cu75* ]]; then + cuda_files=( \ + "cuda-core-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cublas-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cublas-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-cudart-dev-${CUDA_MAJOR_DASH}_${CUDA_PATCH_VERSION}_amd64.deb" \ + "cuda-curand-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-curand-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cufft-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-nvrtc-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-cusolver-dev-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "cuda-misc-headers-${CUDA_MAJOR_DASH}_${CUDA_VERSION}_amd64.deb" \ + "libcuda1-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + "nvidia-${LIBCUDA_MAJOR}_${LIBCUDA_VERSION}_amd64.deb" \ + ) + ml_files=( \ + "libcudnn${LIBCUDNN_MAJOR}-dev_${LIBCUDNN_VERSION}_amd64.deb" \ + ) +fi + + +if [[ ! -d $DEPS_PATH/usr/local/cuda-${CUDA_MAJOR_VERSION} ]]; then + prefix=$DEPS_PATH + + for item in ${cuda_files[*]} + do + echo "Installing $item" + curl -sL "http://developer.download.nvidia.com/compute/cuda/repos/${os_id}/x86_64/${item}" -o package.deb + dpkg -X package.deb ${prefix} + rm package.deb + done + for item in ${ml_files[*]} + do + echo "Installing $item" + curl -sL "http://developer.download.nvidia.com/compute/machine-learning/repos/${os_id}/x86_64/${item}" -o package.deb + dpkg -X package.deb ${prefix} + rm package.deb + done + + cp ${prefix}/usr/include/x86_64-linux-gnu/cudnn_v${LIBCUDNN_MAJOR}.h ${prefix}/include/cudnn.h + ln -s libcudnn_static_v${LIBCUDNN_MAJOR}.a ${prefix}/usr/lib/x86_64-linux-gnu/libcudnn.a + cp ${prefix}/usr/local/cuda-${CUDA_MAJOR_VERSION}/lib64/*.a ${prefix}/lib/ + cp ${prefix}/usr/include/nccl.h ${prefix}/include/nccl.h + ln -s libnccl_static.a ${prefix}/usr/lib/x86_64-linux-gnu/libnccl.a +fi