pt-br-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Brazilian or European Portuguese?\n",
    "\n",
    "This simple classifier verifies which Portuguese norm prevails in a given text.\n",
    "\n",
    "We start by defining the problem - what question do we want to answer?<br>\n",
    "In this case - is the text written in <a href=\"https://en.wikipedia.org/wiki/Portuguese_dialects\">European or Brazilian Portuguese</a>?\n",
    "\n",
    "In order to do it, I:\n",
    "- gather data \n",
    "- prepare them for the ML algorithm\n",
    "- train it and verify its accuracy\n",
    "- export the trained algorithm to use it in the future.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gathering data\n",
    "\n",
    "That is - what data do I need, where to gather them and how. In this case I chose TED.com as my source of data. It's a reasonably large database of lectures with a clear division between both Portuguese dialects. I scraped the lecture transcriptions using a Scrapy spider. The code for the two spiders is <a href=\"https://github.com/MarekCichy/pt-br-classifier/blob/master/PTCrawler.py\">here</a> and <a href=\"https://github.com/MarekCichy/pt-br-classifier/blob/master/BRCrawler.py\">here</a>.\n",
    "\n",
    "# Data wrangling\n",
    "\n",
    "I scraped about 2500 TED talks for each version of Portuguese, some 12-18 thousand characters per talk. After cleaning formatting (leaving only pure text), I mark the transcriptions \"0\" for PT-PT (European Portuguese) and \"1\" for PT-BR, then transform it with scikit-learn's CountVectorizer. \n",
    "\n",
    "CountVectorizer basically counts the number of occurences of all the words in each document. I do not use any other preprocessing. I do not stem the words, as I want to maintain the grammatical differences (e.g. verb conjugation differences) between both dialects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import json\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer \n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import metrics\n",
    "\n",
    "from nltk import sent_tokenize\n",
    "\n",
    "from joblib import dump, load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing scraped texts\n",
    "pt_talks = pd.read_excel('talks_pt.xlsx').dropna()\n",
    "br_talks = pd.read_excel('talks_br.xlsx').dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Lindy Lou Isonhood</th>\n",
       "      <td>\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tFoi numa quarta-feira,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jonathan Wilker</th>\n",
       "      <td>\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tGostava que viessem co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Michael C. Bush</th>\n",
       "      <td>\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tInterrogamos CEOs, pol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rene Laufer</th>\n",
       "      <td>\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tA mais de seis mil ano...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Kym Worthy</th>\n",
       "      <td>\\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tEm Agosto de 2009, a m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    0\n",
       "Lindy Lou Isonhood  \\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tFoi numa quarta-feira,...\n",
       "Jonathan Wilker     \\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tGostava que viessem co...\n",
       "Michael C. Bush     \\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tInterrogamos CEOs, pol...\n",
       "Rene Laufer         \\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tA mais de seis mil ano...\n",
       "Kym Worthy          \\n\\t\\t\\t\\t\\t\\t\\t\\t\\t\\t\\tEm Agosto de 2009, a m..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pt_talks.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2409 entries, Lindy Lou Isonhood to Stefana Broadbent\n",
      "Data columns (total 1 columns):\n",
      "0    2409 non-null object\n",
      "dtypes: object(1)\n",
      "memory usage: 37.6+ KB\n"
     ]
    }
   ],
   "source": [
    "pt_talks.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Lindy Lou Isonhood</th>\n",
       "      <td>Foi numa quarta-feira,            ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jonathan Wilker</th>\n",
       "      <td>Gostava que viessem comigo numa vi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Michael C. Bush</th>\n",
       "      <td>Interrogamos CEOs, polícias, camio...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rene Laufer</th>\n",
       "      <td>A mais de seis mil anos-luz       ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Kym Worthy</th>\n",
       "      <td>Em Agosto de 2009, a minha procura...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                 text  label\n",
       "Lindy Lou Isonhood              Foi numa quarta-feira,            ...      0\n",
       "Jonathan Wilker                 Gostava que viessem comigo numa vi...      0\n",
       "Michael C. Bush                 Interrogamos CEOs, polícias, camio...      0\n",
       "Rene Laufer                     A mais de seis mil anos-luz       ...      0\n",
       "Kym Worthy                      Em Agosto de 2009, a minha procura...      0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#cleaning up\n",
    "pt_talks.columns = ['text']\n",
    "pt_talks.text = pt_talks.text.apply(lambda x:x.replace('\\n',' '))\n",
    "pt_talks.text = pt_talks.text.apply(lambda x:x.replace('\\t',' '))\n",
    "pt_talks['label'] = 0\n",
    "pt_talks.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Stacie Bosley</th>\n",
       "      <td>Em 2004,            uma nova empre...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>César Hidalgo</th>\n",
       "      <td>Sou só eu            ou há outras ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sarah Barmak</th>\n",
       "      <td>Em nossa cultura, tendemos a ver o...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shohini Ghose</th>\n",
       "      <td>Se quisesse dar apenas uma olhadin...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Balsher Singh Sidhu</th>\n",
       "      <td>Olhando do espaço, nosso planeta p...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                  text  label\n",
       "Stacie Bosley                    Em 2004,            uma nova empre...      1\n",
       "César Hidalgo                    Sou só eu            ou há outras ...      1\n",
       "Sarah Barmak                     Em nossa cultura, tendemos a ver o...      1\n",
       "Shohini Ghose                    Se quisesse dar apenas uma olhadin...      1\n",
       "Balsher Singh Sidhu              Olhando do espaço, nosso planeta p...      1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "br_talks.columns = ['text']\n",
    "br_talks.text = br_talks.text.apply(lambda x:x.replace('\\t',' '))\n",
    "br_talks.text = br_talks.text.apply(lambda x:x.replace('\\n',' '))\n",
    "br_talks['label'] = 1\n",
    "br_talks.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2897 entries, Stacie Bosley to José Antonio Abreu\n",
      "Data columns (total 2 columns):\n",
      "text     2897 non-null object\n",
      "label    2897 non-null int64\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 67.9+ KB\n"
     ]
    }
   ],
   "source": [
    "br_talks.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 5306 entries, Stacie Bosley to Stefana Broadbent\n",
      "Data columns (total 2 columns):\n",
      "text     5306 non-null object\n",
      "label    5306 non-null int64\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 124.4+ KB\n"
     ]
    }
   ],
   "source": [
    "#making a joint DF\n",
    "joint_df = pd.concat([br_talks, pt_talks])\n",
    "joint_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Stacie Bosley</th>\n",
       "      <td>Em 2004,            uma nova empre...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>César Hidalgo</th>\n",
       "      <td>Sou só eu            ou há outras ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sarah Barmak</th>\n",
       "      <td>Em nossa cultura, tendemos a ver o...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shohini Ghose</th>\n",
       "      <td>Se quisesse dar apenas uma olhadin...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Balsher Singh Sidhu</th>\n",
       "      <td>Olhando do espaço, nosso planeta p...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                  text  label\n",
       "Stacie Bosley                    Em 2004,            uma nova empre...      1\n",
       "César Hidalgo                    Sou só eu            ou há outras ...      1\n",
       "Sarah Barmak                     Em nossa cultura, tendemos a ver o...      1\n",
       "Shohini Ghose                    Se quisesse dar apenas uma olhadin...      1\n",
       "Balsher Singh Sidhu              Olhando do espaço, nosso planeta p...      1"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joint_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    2897\n",
       "0    2409\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joint_df.label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(min_df=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Thomas Dolby                      Eu tenho vindo a tocar no TED há q...\n",
       "Mike Gil                          Quem aqui é fascinado pela vida no...\n",
       "Shimon Steinberg                  Sou um amante de insetos,         ...\n",
       "Chiki Sarkar                      Olhem ao seu redor.            Sej...\n",
       "Elizabeth Lev                     Imaginem que estão em Roma        ...\n",
       "Rachel Pike                       Quero falar a vocês hoje sobre a m...\n",
       "Tiana Epps-johnson                Ok, eu quero tirar um momento para...\n",
       "DeAndrea Salvador                 Quando eu criança,            pass...\n",
       "Stefano Mancuso                   As vezes folheio            revist...\n",
       "Vinay Venkatraman                 Frugal Digital é basicamente um pe...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = joint_df.label\n",
    "X_train, X_test, y_train, y_test = train_test_split(joint_df['text'], y, test_size=0.33, random_state=53)\n",
    "X_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    3555.000000\n",
       "mean        0.536428\n",
       "std         0.498741\n",
       "min         0.000000\n",
       "25%         0.000000\n",
       "50%         1.000000\n",
       "75%         1.000000\n",
       "max         1.000000\n",
       "Name: label, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1751.000000\n",
       "mean        0.565391\n",
       "std         0.495847\n",
       "min         0.000000\n",
       "25%         0.000000\n",
       "50%         1.000000\n",
       "75%         1.000000\n",
       "max         1.000000\n",
       "Name: label, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = vectorizer.fit_transform(X_train.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'eu': 23892,\n",
       " 'tenho': 54360,\n",
       " 'vindo': 57598,\n",
       " 'tocar': 54918,\n",
       " 'no': 39520,\n",
       " 'ted': 54128,\n",
       " 'há': 29681,\n",
       " 'quase': 46240,\n",
       " 'uma': 56460,\n",
       " 'década': 19979,\n",
       " 'raramente': 46764,\n",
       " 'toquei': 55070,\n",
       " 'músicas': 38800,\n",
       " 'novas': 39705,\n",
       " 'da': 15630,\n",
       " 'minha': 37502,\n",
       " 'autoria': 6774,\n",
       " 'isso': 32894,\n",
       " 'aconteceu': 1651,\n",
       " 'em': 20550,\n",
       " 'grande': 28194,\n",
       " 'parte': 41585,\n",
       " 'porque': 43853,\n",
       " 'não': 39836,\n",
       " 'havia': 28850,\n",
       " 'nenhuma': 39234,\n",
       " 'risos': 49645,\n",
       " 'estado': 23300,\n",
       " 'ocupado': 40188,\n",
       " 'com': 12018,\n",
       " 'alguns': 3101,\n",
       " 'projetos': 45245,\n",
       " 'um': 56459,\n",
       " 'deles': 16317,\n",
       " 'era': 22178,\n",
       " 'este': 23391,\n",
       " 'barco': 7391,\n",
       " 'salva': 50442,\n",
       " 'vida': 57483,\n",
       " 'dos': 19674,\n",
       " 'anos': 4063,\n",
       " '30': 488,\n",
       " 'que': 46246,\n",
       " 'andado': 3846,\n",
       " 'restaurar': 49028,\n",
       " 'jardim': 33031,\n",
       " 'casa': 10057,\n",
       " 'de': 15812,\n",
       " 'praia': 44150,\n",
       " 'inglaterra': 31421,\n",
       " 'portanto': 43877,\n",
       " 'agora': 2583,\n",
       " 'quando': 46195,\n",
       " 'as': 5635,\n",
       " 'calotas': 9343,\n",
       " 'polares': 43588,\n",
       " 'meu': 37205,\n",
       " 'estúdio': 23812,\n",
       " 'gravação': 28262,\n",
       " 'erguer': 22199,\n",
       " 'se': 50781,\n",
       " 'como': 12257,\n",
       " 'arca': 5094,\n",
       " 'vou': 58123,\n",
       " 'flutuar': 26061,\n",
       " 'mundo': 38582,\n",
       " 'afogado': 2433,\n",
       " 'personagem': 42731,\n",
       " 'romance': 49827,\n",
       " 'ballard': 7240,\n",
       " 'durante': 19903,\n",
       " 'dia': 18366,\n",
       " 'recolhe': 47255,\n",
       " 'energia': 21298,\n",
       " 'partir': 41667,\n",
       " 'painéis': 41167,\n",
       " 'solares': 52318,\n",
       " 'teto': 54629,\n",
       " 'sala': 50352,\n",
       " 'navegação': 39060,\n",
       " 'turbina': 56200,\n",
       " '450': 597,\n",
       " 'watts': 58294,\n",
       " 'topo': 55058,\n",
       " 'do': 19399,\n",
       " 'mastro': 36352,\n",
       " 'escurece': 22609,\n",
       " 'muita': 38471,\n",
       " 'posso': 43970,\n",
       " 'iluminar': 29967,\n",
       " 'farol': 25174,\n",
       " 'fico': 25625,\n",
       " 'lá': 35372,\n",
       " 'até': 6522,\n",
       " 'manhã': 35882,\n",
       " 'cedo': 10408,\n",
       " 'trabalho': 55288,\n",
       " 'canções': 9653,\n",
       " 'gostaria': 28029,\n",
       " 'para': 41370,\n",
       " 'vocês': 57980,\n",
       " 'estiverem': 23548,\n",
       " 'dispostos': 18995,\n",
       " 'ser': 51323,\n",
       " 'os': 40871,\n",
       " 'primeiros': 44837,\n",
       " 'ouvir': 40994,\n",
       " 'aplausos': 4595,\n",
       " 'sobre': 52115,\n",
       " 'billie': 7983,\n",
       " 'holiday': 29335,\n",
       " 'parece': 41503,\n",
       " 'numa': 39769,\n",
       " 'certa': 10613,\n",
       " 'noite': 39540,\n",
       " '1947': 282,\n",
       " 'ela': 20304,\n",
       " 'deixou': 16289,\n",
       " 'seu': 51445,\n",
       " 'espaço': 22857,\n",
       " 'físico': 27058,\n",
       " 'desapareceu': 16858,\n",
       " 'toda': 54938,\n",
       " 'reaparecer': 47000,\n",
       " 'mas': 36282,\n",
       " 'sei': 50974,\n",
       " 'onde': 40474,\n",
       " 'estava': 23384,\n",
       " 'comigo': 12238,\n",
       " 'vidas': 57484,\n",
       " 'quente': 46331,\n",
       " 'música': 38799,\n",
       " 'esgueirou': 22739,\n",
       " 'sorrateiramente': 52559,\n",
       " 'entre': 21802,\n",
       " 'meus': 37207,\n",
       " 'braços': 8651,\n",
       " 'despertos': 17855,\n",
       " 'gole': 27941,\n",
       " 'papa': 41334,\n",
       " 'azeda': 7021,\n",
       " 'estranha': 23613,\n",
       " 'fértil': 27043,\n",
       " 'doce': 19452,\n",
       " 'pedaço': 42025,\n",
       " 'lixo': 35011,\n",
       " 'pânico': 46049,\n",
       " 'na': 38808,\n",
       " 'porta': 43865,\n",
       " 'palco': 41203,\n",
       " 'carnegie': 9909,\n",
       " 'hall': 28716,\n",
       " 'famosa': 25101,\n",
       " 'cantora': 9635,\n",
       " 'jazz': 33068,\n",
       " 'sem': 51045,\n",
       " 'permissão': 42621,\n",
       " 'deve': 18267,\n",
       " 'ter': 54438,\n",
       " 'deixado': 16256,\n",
       " 'prédio': 45707,\n",
       " 'corpo': 14461,\n",
       " 'alma': 3234,\n",
       " 'num': 39768,\n",
       " 'banco': 7285,\n",
       " 'piano': 43026,\n",
       " 'ranger': 46715,\n",
       " 'esta': 23244,\n",
       " 'lua': 35230,\n",
       " 'única': 59049,\n",
       " 'testemunha': 54601,\n",
       " 'respirar': 48889,\n",
       " 'ouvido': 40987,\n",
       " 'desta': 17947,\n",
       " 'vez': 57401,\n",
       " 'amor': 3647,\n",
       " 'pistola': 43275,\n",
       " 'carregada': 9948,\n",
       " 'ao': 4306,\n",
       " 'amanhecer': 3447,\n",
       " 'foi': 26155,\n",
       " 'embora': 20643,\n",
       " 'rio': 49607,\n",
       " 'congelado': 13241,\n",
       " 'johnny': 33228,\n",
       " 'walker': 58252,\n",
       " 'vimos': 57582,\n",
       " 'nova': 39701,\n",
       " 'sozinhos': 52646,\n",
       " 'fica': 25595,\n",
       " 'novamente': 39703,\n",
       " 'tempo': 54298,\n",
       " 'enganador': 21438,\n",
       " 'astuto': 6029,\n",
       " 'mesmo': 37054,\n",
       " 'assim': 5800,\n",
       " 'eco': 20062,\n",
       " 'coração': 14409,\n",
       " 'diz': 19354,\n",
       " 'quem': 46326,\n",
       " 'aqui': 5038,\n",
       " 'fascinado': 25198,\n",
       " 'pela': 42141,\n",
       " 'fundo': 26943,\n",
       " 'mar': 36066,\n",
       " 'fantástico': 25132,\n",
       " 'acabamos': 1253,\n",
       " 'fazer': 25298,\n",
       " 'vamos': 56896,\n",
       " 'examinar': 24118,\n",
       " 'por': 43817,\n",
       " 'instante': 31780,\n",
       " 'simples': 51754,\n",
       " 'ato': 6304,\n",
       " 'pessoa': 42928,\n",
       " 'levantar': 34475,\n",
       " 'mão': 38731,\n",
       " 'levou': 34529,\n",
       " 'muitas': 38472,\n",
       " 'outras': 40972,\n",
       " 'fazerem': 25299,\n",
       " 'verdade': 57240,\n",
       " 'pessoas': 42933,\n",
       " 'rede': 47583,\n",
       " 'social': 52215,\n",
       " 'têm': 56327,\n",
       " 'prioridades': 44877,\n",
       " 'comum': 12687,\n",
       " 'vezes': 57402,\n",
       " 'benéfico': 7811,\n",
       " 'imitar': 30130,\n",
       " 'outro': 40974,\n",
       " 'lembrem': 34341,\n",
       " 'escola': 22434,\n",
       " 'vestir': 57374,\n",
       " 'alguém': 3103,\n",
       " 'estiloso': 23490,\n",
       " 'algo': 3086,\n",
       " 'bem': 7740,\n",
       " 'legal': 34235,\n",
       " 'comportamento': 12497,\n",
       " 'também': 53922,\n",
       " 'animais': 4000,\n",
       " 'selvagens': 51043,\n",
       " 'exemplo': 24271,\n",
       " 'algumas': 3100,\n",
       " 'aves': 6945,\n",
       " 'imitam': 30127,\n",
       " 'sinais': 51815,\n",
       " 'alarme': 2881,\n",
       " 'espalhar': 22796,\n",
       " 'informação': 31365,\n",
       " 'predadores': 44299,\n",
       " 'aproximando': 4952,\n",
       " 'será': 51421,\n",
       " 'fato': 25239,\n",
       " 'imitarem': 30131,\n",
       " 'poderia': 43528,\n",
       " 'afetar': 2350,\n",
       " 'ecossistemas': 20120,\n",
       " 'inteiros': 31934,\n",
       " 'nós': 39864,\n",
       " 'seres': 51339,\n",
       " 'humanos': 29619,\n",
       " 'dependemos': 16596,\n",
       " 'fui': 26826,\n",
       " 'levado': 34458,\n",
       " 'essa': 23232,\n",
       " 'questão': 46398,\n",
       " 'enquanto': 21578,\n",
       " 'estudava': 23733,\n",
       " 'recifes': 47218,\n",
       " 'corais': 14398,\n",
       " 'sustentam': 53637,\n",
       " 'milhões': 37413,\n",
       " 'meio': 36740,\n",
       " 'pesca': 42867,\n",
       " 'turismo': 56218,\n",
       " 'áfrica': 58833,\n",
       " 'todo': 54944,\n",
       " 'dependem': 16595,\n",
       " 'peixes': 42134,\n",
       " 'desempenham': 17263,\n",
       " 'papel': 41343,\n",
       " 'fundamental': 26914,\n",
       " 'alimentando': 3154,\n",
       " 'algas': 3077,\n",
       " 'houver': 29554,\n",
       " 'controle': 14112,\n",
       " 'essas': 23234,\n",
       " 'podem': 43519,\n",
       " 'matar': 36368,\n",
       " 'dominar': 19564,\n",
       " 'mudança': 38433,\n",
       " 'desastrosa': 16902,\n",
       " 'difícil': 18523,\n",
       " 'ou': 40940,\n",
       " 'impossível': 30391,\n",
       " 'reverter': 49375,\n",
       " 'entender': 21703,\n",
       " 'evitar': 23998,\n",
       " 'estão': 23792,\n",
       " 'comendo': 12119,\n",
       " 'pode': 43518,\n",
       " 'eles': 20379,\n",
       " 'partes': 41590,\n",
       " 'abertas': 980,\n",
       " 'recife': 47217,\n",
       " 'expostas': 24622,\n",
       " 'quais': 46171,\n",
       " 'raras': 46765,\n",
       " 'ocasiões': 40106,\n",
       " 'parecem': 41504,\n",
       " 'perceber': 42348,\n",
       " 'estou': 23574,\n",
       " 'observando': 39990,\n",
       " 'então': 21903,\n",
       " 'muito': 38473,\n",
       " 'claramente': 11401,\n",
       " 'comer': 12140,\n",
       " 'fora': 26221,\n",
       " 'assustador': 5965,\n",
       " 'queria': 46351,\n",
       " 'faziam': 25304,\n",
       " 'situações': 52001,\n",
       " 'arriscadas': 5498,\n",
       " 'colegas': 11822,\n",
       " 'colocamos': 11919,\n",
       " 'suportes': 53433,\n",
       " 'enormes': 21562,\n",
       " 'câmeras': 15521,\n",
       " 'vídeo': 58196,\n",
       " 'coral': 14405,\n",
       " 'monitorar': 37936,\n",
       " 'remotamente': 48320,\n",
       " 'áreas': 58862,\n",
       " 'inteiras': 31932,\n",
       " 'alimentação': 3166,\n",
       " 'produzem': 45057,\n",
       " 'perspectiva': 42752,\n",
       " 'cima': 11203,\n",
       " 'mostra': 38230,\n",
       " 'alimentar': 3155,\n",
       " 'movimentos': 38381,\n",
       " 'precisos': 44276,\n",
       " 'muitos': 38474,\n",
       " 'diferentes': 18480,\n",
       " 'mostrados': 38236,\n",
       " 'pontos': 43767,\n",
       " 'coloridos': 11986,\n",
       " 'analisando': 3773,\n",
       " 'milhares': 37409,\n",
       " 'nas': 38965,\n",
       " 'descobrimos': 17039,\n",
       " 'padrão': 41115,\n",
       " 'esses': 23240,\n",
       " 'apesar': 4547,\n",
       " 'serem': 51328,\n",
       " 'espécies': 23130,\n",
       " 'nadarem': 38833,\n",
       " 'cardumes': 9844,\n",
       " 'estavam': 23385,\n",
       " 'imitando': 30129,\n",
       " 'outros': 40976,\n",
       " 'modo': 37800,\n",
       " 'peixe': 42133,\n",
       " 'entrar': 21784,\n",
       " 'nessas': 39274,\n",
       " 'perigosas': 42573,\n",
       " 'levar': 34493,\n",
       " 'permaneciam': 42605,\n",
       " 'mais': 35666,\n",
       " 'comiam': 12229,\n",
       " 'eram': 22180,\n",
       " 'rodeados': 49759,\n",
       " 'estar': 23350,\n",
       " 'acontecendo': 1640,\n",
       " 'querer': 46343,\n",
       " 'passar': 41732,\n",
       " 'vital': 57834,\n",
       " 'apenas': 4483,\n",
       " 'veja': 57066,\n",
       " 'predador': 44298,\n",
       " 'fuja': 26827,\n",
       " 'poderá': 43540,\n",
       " 'alertar': 3036,\n",
       " 'perigo': 42569,\n",
       " 'entrando': 21779,\n",
       " 'segurança': 50954,\n",
       " 'indica': 30985,\n",
       " 'acontece': 1638,\n",
       " 'são': 53731,\n",
       " 'conectados': 13018,\n",
       " 'dentro': 16548,\n",
       " 'redes': 47598,\n",
       " 'sociais': 52214,\n",
       " 'fornecer': 26338,\n",
       " 'seguro': 50967,\n",
       " 'nossas': 39626,\n",
       " 'análises': 4285,\n",
       " 'indicam': 30994,\n",
       " 'poderiam': 43529,\n",
       " 'representar': 48606,\n",
       " '60': 677,\n",
       " 'das': 15773,\n",
       " 'consumidas': 13704,\n",
       " 'comunidade': 12718,\n",
       " 'fluxo': 26072,\n",
       " 'recursos': 47546,\n",
       " 'descobertas': 17027,\n",
       " 'sugerem': 53182,\n",
       " 'predatória': 44301,\n",
       " 'problema': 44945,\n",
       " 'nos': 39622,\n",
       " 'remove': 48326,\n",
       " 'romper': 49845,\n",
       " 'remanescentes': 48273,\n",
       " 'esconder': 22479,\n",
       " 'menos': 36879,\n",
       " 'devido': 18298,\n",
       " 'falta': 25051,\n",
       " 'informações': 31366,\n",
       " 'importantes': 30357,\n",
       " 'tornaria': 55109,\n",
       " 'vulneráveis': 58156,\n",
       " 'prevemos': 44768,\n",
       " 'hoje': 29325,\n",
       " 'notavelmente': 39653,\n",
       " 'permitem': 42626,\n",
       " 'ações': 7052,\n",
       " 'sejam': 50985,\n",
       " 'divulgadas': 19332,\n",
       " 'alimentam': 3151,\n",
       " 'economia': 20084,\n",
       " 'global': 27874,\n",
       " 'todos': 54946,\n",
       " 'nossa': 39625,\n",
       " 'descoberta': 17026,\n",
       " 'melhores': 36788,\n",
       " 'caminhos': 9432,\n",
       " 'gerir': 27667,\n",
       " 'forma': 26255,\n",
       " 'sustentável': 53653,\n",
       " 'somos': 52465,\n",
       " 'afetados': 2346,\n",
       " 'ação': 7051,\n",
       " 'podemos': 43521,\n",
       " 'pelas': 42143,\n",
       " 'único': 59051,\n",
       " 'distante': 19101,\n",
       " 'imitação': 30135,\n",
       " 'dele': 16306,\n",
       " 'obrigado': 39941,\n",
       " 'sou': 52604,\n",
       " 'amante': 3449,\n",
       " 'insetos': 31667,\n",
       " 'aliás': 3221,\n",
       " 'desde': 17208,\n",
       " 'criança': 14921,\n",
       " 'pouco': 44057,\n",
       " 'tarde': 53989,\n",
       " 'fiz': 25923,\n",
       " 'bacharelato': 7095,\n",
       " 'formei': 26295,\n",
       " 'me': 36586,\n",
       " 'zoologia': 58778,\n",
       " 'universidade': 56546,\n",
       " 'tel': 54168,\n",
       " 'aviv': 6979,\n",
       " 'apaixonei': 4353,\n",
       " 'depois': 16616,\n",
       " 'tirei': 54840,\n",
       " 'curso': 15398,\n",
       " 'disciplina': 18808,\n",
       " 'ciência': 11375,\n",
       " 'pensei': 42272,\n",
       " 'poderei': 43524,\n",
       " 'praticar': 44179,\n",
       " 'contribuir': 14071,\n",
       " 'passei': 41764,\n",
       " 'proteção': 45553,\n",
       " 'plantas': 43391,\n",
       " 'contra': 13943,\n",
       " 'prejudiciais': 44437,\n",
       " 'cheguei': 10890,\n",
       " 'controlo': 14117,\n",
       " 'biológico': 8045,\n",
       " 'pragas': 44142,\n",
       " 'definimos': 16157,\n",
       " 'uso': 56664,\n",
       " 'organismos': 40703,\n",
       " 'vivos': 57917,\n",
       " 'reduzir': 47666,\n",
       " 'populações': 43809,\n",
       " 'às': 58825,\n",
       " 'visa': 57737,\n",
       " 'redução': 47677,\n",
       " 'químicos': 46508,\n",
       " 'propósito': 45443,\n",
       " 'bons': 8453,\n",
       " 'estamos': 23334,\n",
       " 'falar': 24954,\n",
       " 'existem': 24355,\n",
       " 'só': 53779,\n",
       " 'últimos': 59044,\n",
       " '120': 57,\n",
       " 'começaram': 12202,\n",
       " 'souberam': 52608,\n",
       " 'cada': 9154,\n",
       " 'explorar': 24556,\n",
       " 'usar': 56635,\n",
       " 'fenómeno': 25447,\n",
       " 'seja': 50984,\n",
       " 'natural': 39027,\n",
       " 'segundo': 50943,\n",
       " 'suas': 52843,\n",
       " 'necessidades': 39109,\n",
       " 'vê': 58185,\n",
       " 'lo': 35026,\n",
       " 'nossos': 39628,\n",
       " 'quintais': 46449,\n",
       " 'basta': 7499,\n",
       " 'arranjar': 5372,\n",
       " 'lupa': 35305,\n",
       " 'veem': 57040,\n",
       " 'lente': 34376,\n",
       " 'aumentar': 6591,\n",
       " '10': 23,\n",
       " 'sim': 51717,\n",
       " 'abri': 1118,\n",
       " 'la': 33876,\n",
       " 'virem': 57697,\n",
       " 'folha': 26172,\n",
       " 'verão': 57343,\n",
       " 'novo': 39720,\n",
       " 'minúsculos': 37567,\n",
       " 'pequenas': 42313,\n",
       " 'aranhas': 5072,\n",
       " 'milímetro': 37459,\n",
       " 'dois': 19518,\n",
       " 'milímetros': 37460,\n",
       " 'comprimento': 12598,\n",
       " 'distinguir': 19111,\n",
       " 'maus': 36480,\n",
       " 'existe': 24354,\n",
       " 'literalmente': 34944,\n",
       " 'frente': 26672,\n",
       " 'deste': 17967,\n",
       " 'edifício': 20142,\n",
       " 'seguramente': 50951,\n",
       " 'olhar': 40384,\n",
       " 'está': 23773,\n",
       " 'precisamos': 44250,\n",
       " 'saber': 50181,\n",
       " 'explorá': 24578,\n",
       " 'percorrer': 42406,\n",
       " 'exemplos': 24272,\n",
       " 'praga': 44141,\n",
       " 'prejuízos': 44442,\n",
       " 'inflige': 31297,\n",
       " 'planta': 43378,\n",
       " 'qual': 46174,\n",
       " 'inimigo': 31495,\n",
       " 'agente': 2531,\n",
       " 'controlado': 14096,\n",
       " 'biologicamente': 8036,\n",
       " 'inseto': 31666,\n",
       " 'bom': 8381,\n",
       " 'genericamente': 27479,\n",
       " 'ácaros': 58828,\n",
       " 'chamemos': 10728,\n",
       " 'lhes': 34561,\n",
       " 'seis': 50978,\n",
       " 'pernas': 42656,\n",
       " 'oito': 40341,\n",
       " 'observá': 40012,\n",
       " 'los': 35170,\n",
       " 'devastadora': 18257,\n",
       " 'ácaro': 58827,\n",
       " 'aranha': 5071,\n",
       " 'faz': 25287,\n",
       " 'teia': 54157,\n",
       " 'parecida': 41521,\n",
       " 'vemos': 57106,\n",
       " 'mãe': 38728,\n",
       " 'provavelmente': 45599,\n",
       " 'filhos': 25692,\n",
       " 'esquerda': 23180,\n",
       " 'direita': 18746,\n",
       " 'ovo': 41029,\n",
       " 'lado': 33916,\n",
       " 'direito': 18749,\n",
       " 'tipo': 54800,\n",
       " 'estragos': 23597,\n",
       " 'infligir': 31300,\n",
       " 'ver': 57228,\n",
       " 'pepino': 42304,\n",
       " 'algodão': 3087,\n",
       " 'tomate': 55003,\n",
       " 'verdes': 57250,\n",
       " 'brancas': 8606,\n",
       " 'causa': 10263,\n",
       " 'bocas': 8295,\n",
       " 'destas': 17966,\n",
       " 'chega': 10860,\n",
       " 'natureza': 39034,\n",
       " 'oferece': 40266,\n",
       " 'boa': 8270,\n",
       " 'resto': 49046,\n",
       " 'tão': 56305,\n",
       " 'pequeno': 42321,\n",
       " 'nada': 38823,\n",
       " 'correr': 14512,\n",
       " 'rapidamente': 46743,\n",
       " 'caça': 10338,\n",
       " 'perseguindo': 42696,\n",
       " 'dama': 15665,\n",
       " 'esquerdo': 23185,\n",
       " 'perfurar': 42520,\n",
       " 'sugar': 53179,\n",
       " 'fluidos': 26044,\n",
       " 'corporais': 14462,\n",
       " 'fim': 25755,\n",
       " 'cinco': 11212,\n",
       " 'minutos': 37559,\n",
       " 'isto': 32898,\n",
       " 'cadáver': 9185,\n",
       " 'vulgar': 58149,\n",
       " 'murcho': 38608,\n",
       " 'sugado': 53175,\n",
       " 'junto': 33409,\n",
       " 'indivíduos': 31074,\n",
       " 'saciados': 50241,\n",
       " 'ninfa': 39478,\n",
       " 'jovem': 33287,\n",
       " 'refeição': 47725,\n",
       " '24': 433,\n",
       " 'horas': 29455,\n",
       " 'cerca': 10572,\n",
       " '15': 104,\n",
       " '20': 349,\n",
       " 'ovos': 41030,\n",
       " 'sempre': 51093,\n",
       " 'fome': 26187,\n",
       " 'israel': 32886,\n",
       " 'primavera': 44828,\n",
       " 'temperatura': 54284,\n",
       " 'sobe': 52096,\n",
       " 'bruscamente': 8837,\n",
       " 'estas': 23365,\n",
       " 'estes': 23449,\n",
       " 'todas': 54940,\n",
       " 'folhagem': 26173,\n",
       " 'fresca': 26703,\n",
       " 'chamada': 10698,\n",
       " 'floração': 25996,\n",
       " 'primaveril': 44830,\n",
       " 'temos': 54274,\n",
       " 'fêmeas': 27050,\n",
       " 'amazonas': 3505,\n",
       " 'dão': 19975,\n",
       " 'origem': 40795,\n",
       " 'machos': 35479,\n",
       " 'chama': 10697,\n",
       " 'vivem': 57873,\n",
       " 'felizes': 25418,\n",
       " 'segregam': 50906,\n",
       " 'líquido': 35422,\n",
       " 'pegajoso': 42092,\n",
       " 'açucarado': 7049,\n",
       " 'chamado': 10700,\n",
       " 'envolve': 22008,\n",
       " 'superiores': 53356,\n",
       " 'passou': 41786,\n",
       " 'verde': 57248,\n",
       " 'preto': 44751,\n",
       " 'fungo': 26954,\n",
       " 'negro': 39208,\n",
       " 'cobre': 11619,\n",
       " 'aparece': 4396,\n",
       " 'salvação': 50466,\n",
       " 'através': 6399,\n",
       " 'vespa': 57353,\n",
       " 'parasita': 41473,\n",
       " 'duas': 19829,\n",
       " 'claro': 11417,\n",
       " 'elegante': 20347,\n",
       " 'voador': 57946,\n",
       " 'rápido': 50131,\n",
       " 'aguçado': 2724,\n",
       " 'manobra': 35942,\n",
       " 'coloca': 11912,\n",
       " 'face': 24876,\n",
       " 'vítima': 58207,\n",
       " 'inclinando': 30725,\n",
       " 'abdómen': 961,\n",
       " 'inserindo': 31657,\n",
       " 'tenta': 54375,\n",
       " 'fugir': 26819,\n",
       " 'morde': 38107,\n",
       " 'segrega': 50901,\n",
       " 'diversos': 19250,\n",
       " 'líquidos': 35423,\n",
       " 'realidade': 46934,\n",
       " 'introduzido': 32341,\n",
       " 'uns': 56560,\n",
       " 'dias': 18423,\n",
       " 'consoante': 13534,\n",
       " 'larva': 34095,\n",
       " 'tudo': 56152,\n",
       " 'ficção': 25635,\n",
       " 'disso': 19064,\n",
       " 'nosso': 39627,\n",
       " 'quintal': 46450,\n",
       " 'resultado': 49072,\n",
       " 'final': 25759,\n",
       " 'múmias': 38794,\n",
       " 'visual': 57811,\n",
       " 'dum': 19849,\n",
       " 'morto': 38193,\n",
       " 'contém': 14147,\n",
       " 'desenvolvimento': 17384,\n",
       " 'nascimento': 38995,\n",
       " 'completo': 12460,\n",
       " 'filmes': 25720,\n",
       " 'etc': 23842,\n",
       " 'bastam': 7500,\n",
       " 'fêmea': 27049,\n",
       " 'acasala': 1324,\n",
       " 'imediatamente': 30089,\n",
       " 'macho': 35478,\n",
       " 'vai': 56809,\n",
       " 'curto': 15417,\n",
       " 'vive': 57870,\n",
       " 'três': 56112,\n",
       " 'quatro': 46242,\n",
       " 'precisa': 44246,\n",
       " 'dar': 15742,\n",
       " '400': 567,\n",
       " 'significa': 51648,\n",
       " 'tem': 54250,\n",
       " 'pôr': 46110,\n",
       " 'seus': 51449,\n",
       " 'história': 29286,\n",
       " 'riqueza': 49619,\n",
       " 'inimigos': 31496,\n",
       " 'naturais': 39026,\n",
       " 'último': 59043,\n",
       " 'começar': 12200,\n",
       " 'nomes': 39570,\n",
       " 'esquisitos': 23214,\n",
       " 'vos': 58092,\n",
       " 'massacrei': 36320,\n",
       " 'latinos': 34127,\n",
       " 'criaturas': 14946,\n",
       " 'ok': 40349,\n",
       " 'comuns': 12731,\n",
       " 'bonita': 8438,\n",
       " 'má': 38703,\n",
       " 'vejam': 57067,\n",
       " 'pimentos': 43113,\n",
       " 'doces': 19456,\n",
       " 'pimento': 43112,\n",
       " 'exótico': 24829,\n",
       " 'ornamental': 40826,\n",
       " 'comestível': 12169,\n",
       " 'sofre': 52272,\n",
       " 'duma': 19850,\n",
       " 'doença': 19496,\n",
       " 'viral': 57681,\n",
       " 'transmitida': 55570,\n",
       " 'pelos': 42151,\n",
       " 'adultos': 2226,\n",
       " 'vem': 57104,\n",
       " 'minúsculo': 37566,\n",
       " 'pirata': 43217,\n",
       " 'adulto': 2225,\n",
       " 'jovens': 33288,\n",
       " 'perfura': 42511,\n",
       " 'sugando': 53178,\n",
       " 'passando': 41729,\n",
       " 'outra': 40971,\n",
       " 'presa': 44557,\n",
       " 'continuando': 13897,\n",
       " 'local': 35035,\n",
       " 'espalharmos': 22800,\n",
       " 'piratas': 43220,\n",
       " 'sobem': 52097,\n",
       " 'flores': 26003,\n",
       " 'flor': 25992,\n",
       " 'inundada': 32391,\n",
       " 'eliminarem': 20467,\n",
       " 'situação': 52000,\n",
       " 'positiva': 43932,\n",
       " 'prejuízo': 44441,\n",
       " 'fruto': 26791,\n",
       " 'nenhum': 39233,\n",
       " 'nestas': 39278,\n",
       " 'circunstâncias': 11300,\n",
       " 'vemo': 57105,\n",
       " 'base': 7458,\n",
       " 'fazemos': 25292,\n",
       " 'nordeste': 39584,\n",
       " 'área': 58861,\n",
       " 'kibbutz': 33682,\n",
       " 'complexo': 12469,\n",
       " 'produz': 45054,\n",
       " 'massa': 36310,\n",
       " 'palavras': 41195,\n",
       " 'ali': 3114,\n",
       " 'amplificamos': 3702,\n",
       " '000': 1,\n",
       " 'metros': 37187,\n",
       " 'quadrados': 46150,\n",
       " 'estufas': 23748,\n",
       " 'produzir': 45067,\n",
       " 'predatórios': 44304,\n",
       " 'vespas': 57354,\n",
       " 'parasitas': 41475,\n",
       " 'paisagem': 41173,\n",
       " 'montanhas': 38007,\n",
       " 'jordão': 33255,\n",
       " 'vale': 56824,\n",
       " 'inverno': 32490,\n",
       " 'ameno': 3594,\n",
       " 'agradável': 2617,\n",
       " 'condição': 12970,\n",
       " 'excelente': 24163,\n",
       " 'produção': 45083,\n",
       " 'manipulação': 35928,\n",
       " 'genética': 27533,\n",
       " 'ogms': 40326,\n",
       " 'geneticamente': 27487,\n",
       " 'modificados': 37782,\n",
       " 'nenhuns': 39236,\n",
       " 'buscá': 8993,\n",
       " 'coisa': 11741,\n",
       " 'damos': 15673,\n",
       " 'condições': 12971,\n",
       " 'ótimas': 59028,\n",
       " 'salas': 50364,\n",
       " 'proliferarem': 45260,\n",
       " 'multiplicarem': 38546,\n",
       " 'reproduzirem': 48663,\n",
       " 'obtemos': 40054,\n",
       " 'microscópio': 37336,\n",
       " 'canto': 9632,\n",
       " 'superior': 53355,\n",
       " 'predatório': 44303,\n",
       " 'grupo': 28407,\n",
       " 'ampola': 3714,\n",
       " 'grama': 28166,\n",
       " 'destes': 17973,\n",
       " '80': 758,\n",
       " 'suficientes': 53153,\n",
       " 'controlar': 14103,\n",
       " '4000': 568,\n",
       " 'campo': 9473,\n",
       " 'morangos': 38091,\n",
       " 'estação': 23389,\n",
       " 'ano': 4048,\n",
       " 'disto': 19123,\n",
       " 'acreditem': 1739,\n",
       " 'mim': 37461,\n",
       " 'várias': 58171,\n",
       " 'dezenas': 18354,\n",
       " 'quilogramas': 46416,\n",
       " 'chamo': 10731,\n",
       " 'amplificação': 3705,\n",
       " 'prejudicamos': 44429,\n",
       " 'equilíbrio': 22137,\n",
       " 'pelo': 42149,\n",
       " 'contrário': 14135,\n",
       " 'levamo': 34461,\n",
       " 'campos': 9477,\n",
       " 'cultura': 15283,\n",
       " 'já': 33494,\n",
       " 'prejudicado': 44426,\n",
       " 'entramos': 21778,\n",
       " 'inverter': 32506,\n",
       " 'roda': 49741,\n",
       " 'proporcionar': 45395,\n",
       " 'agrícola': 2682,\n",
       " 'reduzindo': 47665,\n",
       " 'ideia': 29780,\n",
       " 'impacto': 30176,\n",
       " 'neste': 39279,\n",
       " 'quadro': 46165,\n",
       " 'sucedido': 53110,\n",
       " 'utilizamos': 56708,\n",
       " '1000': 25,\n",
       " 'hectares': 28902,\n",
       " 'termos': 54517,\n",
       " 'israelitas': 32890,\n",
       " 'sob': 52095,\n",
       " 'reduziram': 47667,\n",
       " 'efeito': 20213,\n",
       " '75': 738,\n",
       " 'pesticidas': 42941,\n",
       " 'ainda': 2751,\n",
       " 'especialmente': 22883,\n",
       " 'visam': 57740,\n",
       " 'forte': 26391,\n",
       " 'impõe': 30523,\n",
       " 'pergunta': 42528,\n",
       " 'perguntarmos': 42543,\n",
       " 'aos': 4311,\n",
       " 'produtores': 45052,\n",
       " 'agricultores': 2659,\n",
       " 'porquê': 43855,\n",
       " 'número': 39871,\n",
       " 'respostas': 48958,\n",
       " 'recebemos': 47119,\n",
       " 'igual': 29895,\n",
       " 'perguntamos': 42535,\n",
       " 'formos': 26306,\n",
       " 'sudeste': 53140,\n",
       " 'rift': 49570,\n",
       " 'situa': 51988,\n",
       " 'pérola': 46063,\n",
       " 'agricultura': 2660,\n",
       " 'israelita': 32889,\n",
       " 'alto': 3350,\n",
       " 'nível': 39855,\n",
       " 'protegidas': 45523,\n",
       " 'diretos': 18764,\n",
       " 'deserto': 17402,\n",
       " 'aumentarmos': 6597,\n",
       " 'observar': 39991,\n",
       " 'avós': 6988,\n",
       " 'netos': 39289,\n",
       " 'distribuindo': 19169,\n",
       " 'fatos': 25243,\n",
       " 'especiais': 22864,\n",
       " 'máscaras': 38722,\n",
       " 'gás': 28561,\n",
       " 'aplicar': 4607,\n",
       " 'refere': 47727,\n",
       " 'aplicação': 4616,\n",
       " 'resposta': 48957,\n",
       " 'cultivadores': 15263,\n",
       " 'ficam': 25597,\n",
       " 'facto': 24907,\n",
       " 'resistência': 48806,\n",
       " 'tornem': 55128,\n",
       " 'resistentes': 48793,\n",
       " 'caso': 10111,\n",
       " 'bactérias': 7114,\n",
       " 'antibióticos': 4148,\n",
       " 'mesma': 37051,\n",
       " 'acontecer': 1641,\n",
       " 'felizmente': 25419,\n",
       " 'quer': 46336,\n",
       " 'extremamente': 24793,\n",
       " 'rara': 46763,\n",
       " 'dificilmente': 18491,\n",
       " 'evolução': 24049,\n",
       " 'relação': 48216,\n",
       " 'exigência': 24342,\n",
       " 'pública': 46118,\n",
       " 'quanto': 46219,\n",
       " 'público': 46120,\n",
       " 'exige': 24321,\n",
       " 'maior': 35659,\n",
       " 'toma': 54979,\n",
       " 'consciência': 13382,\n",
       " 'devem': 18270,\n",
       " 'possam': 43945,\n",
       " 'possível': 43992,\n",
       " 'substituir': 53034,\n",
       " 'químico': 46507,\n",
       " 'produtora': 45050,\n",
       " 'interessada': 32052,\n",
       " 'usando': 56634,\n",
       " 'cabeça': 9108,\n",
       " 'sua': 52835,\n",
       " 'quero': 46359,\n",
       " 'chegar': 10868,\n",
       " 'visão': 57828,\n",
       " 'sonho': 52502,\n",
       " 'sabem': 50177,\n",
       " 'reparem': 48453,\n",
       " 'fosso': 26437,\n",
       " 'tomarmos': 54995,\n",
       " 'consideração': 13503,\n",
       " 'geral': 27612,\n",
       " 'indústria': 31118,\n",
       " 'mundial': 38579,\n",
       " '250': 443,\n",
       " 'dólares': 20012,\n",
       " 'culturas': 15287,\n",
       " 'penso': 42277,\n",
       " '100': 24,\n",
       " 'vinte': 57620,\n",
       " 'mil': 37390,\n",
       " 'pois': 43575,\n",
       " 'enorme': 21560,\n",
       " 'preencher': 44349,\n",
       " 'eliminar': 20465,\n",
       " 'longo': 35143,\n",
       " ...}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.vocabulary_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 0)\t7\n",
      "  (1, 0)\t3\n",
      "  (2, 0)\t2\n",
      "  (3, 0)\t17\n",
      "  (4, 0)\t1\n",
      "  (5, 0)\t4\n",
      "  (6, 0)\t7\n",
      "  (7, 0)\t5\n",
      "  (8, 0)\t2\n",
      "  (9, 0)\t8\n"
     ]
    }
   ],
   "source": [
    "print(train[:10,23892])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Thomas Dolby                    Eu tenho vindo a tocar no TED há q...\n",
       "Mike Gil                        Quem aqui é fascinado pela vida no...\n",
       "Shimon Steinberg                Sou um amante de insetos,         ...\n",
       "Chiki Sarkar                    Olhem ao seu redor.            Sej...\n",
       "Elizabeth Lev                   Imaginem que estão em Roma        ...\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.iloc[:5]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3555x59061 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 2211580 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "Unsurprisingly, I split the texts into a training and testing sets and train a MultinomialNaiveBayes model on these. The model basically checks the frequency of a given word in the BR and PT sets and thus determines if the word suggests a more \"Brazilian\" or \"Portuguese\" word. All of the words in a phrase are weighed and the final probability is given.\n",
    "\n",
    "I transform the test set - I want to be able to classify short texts, of, say, 4 sentences. Thus, I split the test set texts into smaller ones. Let's see how it reflects on the classifier's precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_classifier = MultinomialNB()\n",
    "nb_classifier.fit(train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "test_tokenized = X_test.apply(lambda x: sent_tokenize(x))\n",
    "spread = test_tokenized.apply(pd.Series)\n",
    "spread = spread.fillna('')\n",
    "for y in range(0,340,4):\n",
    "    spread['grouped'+str(y/4)] = spread.iloc[:,y:y+4].apply(lambda x: ' '.join(x), axis=1)\n",
    "spread = spread.loc[:,'grouped0.0':]\n",
    "spread['label'] = y_test\n",
    "test_df = spread.melt(id_vars='label')\n",
    "test_df['len'] = test_df.value.apply(lambda x: len(x))\n",
    "test_df = test_df[test_df.len > 30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = vectorizer.transform(test_df.value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8654743849835536"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred = nb_classifier.predict(test)\n",
    "proba = nb_classifier.predict_proba(test)\n",
    "score = metrics.accuracy_score(test_df.label, pred)\n",
    "score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[18393,  3450],\n",
       "       [ 3380, 25548]], dtype=int64)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = metrics.confusion_matrix(test_df.label, pred, labels = [0,1])\n",
    "cm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Result\n",
    "\n",
    "Not bad! A simple method and a dataset smaller than the whole Wikipedia gave 86,47% accuracy for short texts, 4 sentences long. These are, naturally, samples from the same source as the test set. In the future I want to check the classifier's performance in other genres.\n",
    "\n",
    "For the time being, let's print out some incorrect classification examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([8.67390449e-01, 9.99985685e-01, 9.99998752e-01, ...,\n",
       "       9.04493681e-04, 9.99999825e-01, 4.64320545e-02])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proba_br = proba[:,1]\n",
    "proba_br"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[378, 1, 'grouped0.0',\n",
       "        '            Nós vemos com os olhos. Mas também vemos com o cérebro. E ver com o cérebro é geralmente chamado de imaginação. E estamos familiarizados com os cenários de nossa própria imaginação,            nossos cenários internos.',\n",
       "        230, 0, 0.19057855616759076],\n",
       "       [386, 1, 'grouped0.0',\n",
       "        '            Em março de 2017,            o prefeito da Cidade do Cabo declarou calamidade pública,            porque a cidade tinha menos de quatro meses de água potável. Cada habitante ficou restrito a 100 litros de água por dia. Mas o que significa isso na prática? Com 100 litros de água por dia, podemos tomar um banho de cinco minutos,            lavar o rosto duas vezes            e provavelmente dar descarga cerca de cinco vezes.',\n",
       "        438, 0, 0.08104028835696907],\n",
       "       [402, 1, 'grouped0.0',\n",
       "        '            Como cientista e como ser humano,            tenho tentado tornar-me            sensível à maravilha. Acho que, na noite passada,  Jason Webley chamou isso            de \"conspirar para fazer parte da magia.\" É uma sorte que  a minha carreira como bióloga            me permita mergulhar profundamente na vida            de criaturas verdadeiramente maravilhosas            que compartilham o nosso planeta:            os vaga-lumes. Sei que, para muitos de nós,            os vaga-lumes podem trazer muito boas recordações:            a infância, o verão,            até outras palestras TED.',\n",
       "        605, 0, 0.09637414523143342],\n",
       "       [408, 1, 'grouped0.0',\n",
       "        '            Pela primeira vez na história contemporânea dos EUA            temos cinco gerações interagindo no trabalho. Os veteranos, nascidos entre 1922 e 1943,            são conhecidos como a Geração Grandiosa, os maduros, os silenciosos. São conhecidos pelo sacrifício pessoal,            respeito pelas autoridades e trabalho como recompensa. Os Baby Boomers vieram logo depois, nascidos entre 1944 e 1960.',\n",
       "        412, 0, 0.12309728624189954],\n",
       "       [417, 0, 'grouped0.0',\n",
       "        '            Na década passada,            estudei grupos armados não estatais:            organizações armadas, como terroristas, insurgentes ou milícias. Eu documento o que esses grupos fazem quando não estão em guerra. O meu objetivo é entender melhor  estes atores da violência,            e estudar formas de encorajar  a transição das práticas violentas            para confrontos não violentos. Eu trabalho no terreno,  no mundo da política e na biblioteca.',\n",
       "        463, 1, 0.8447439954476356],\n",
       "       [418, 1, 'grouped0.0',\n",
       "        '            Gostaria de introduzi-lo em uma área emergente da ciência,            uma que ainda é especulativa mas extremamente emocionante,            e que certamente está  crescendo rapidamente. Biologia quântica faz uma pergunta muito simples:            será que a mecânica quântica,            aquela estranha, maravilhosa e poderosa            teoria do mundo subatômico  dos átomos e moléculas,            que apoia tanto a física quanto a química moderna,            também desempenha um papel  dentro da célula viva? Em outras palavras: existem processos, mecanismos, fenômenos            em organismos vivos que só podem ser explicados            com uma mão amiga da mecânica quântica? A Biologia quântica não é nova;            é conhecida desde o início dos anos 1930.',\n",
       "        782, 0, 0.31420770601419],\n",
       "       [430, 0, 'grouped0.0',\n",
       "        '            Quando era jovem, orgulhava-me de não ser conformista            no conservador estado americano  em que vivo, o Kansas. Não costumava seguir o rebanho. Não tinha medo de experimentar  roupas ou penteados estranhos. Não tinha medo de dar a minha opinião e era extremamente social.',\n",
       "        292, 1, 0.8159982274811204],\n",
       "       [431, 1, 'grouped0.0',\n",
       "        '            \"Estes são tempos\",            disse Thomas Paine,            \"que desafiam as almas dos homens\". E estão nos desafiando agora. Este é um momento crítico da história do Ocidente. Nós vimos eleições e sociedades divididas.',\n",
       "        233, 0, 0.20474666890313745],\n",
       "       [440, 1, 'grouped0.0',\n",
       "        '            Hoje, tenho apenas um pedido. Por favor, não digam que sou normal. Agora, gostaria de apresentar-lhes meus dois irmãos. Remi tem 22 anos,            é alto e muito bonito.',\n",
       "        183, 0, 0.3058471138129017],\n",
       "       [445, 1, 'grouped0.0',\n",
       "        '            Junho de 2010. (Risos)            Aterrissei, pela primeira vez, em Roma, na Itália. Não estava lá a passeio. Estava lá para resolver o problema da fome no mundo.',\n",
       "        174, 0, 0.4240743645040658]], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_falses = test_df.reset_index()\n",
    "check_falses['pred'] = pred\n",
    "check_falses['proba_br'] = proba_br\n",
    "\n",
    "falses = check_falses[check_falses['label'] != check_falses['pred']]\n",
    "falses.values[50:60]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>label</th>\n",
       "      <th>len</th>\n",
       "      <th>pred</th>\n",
       "      <th>proba_br</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>6830.000000</td>\n",
       "      <td>6830.000000</td>\n",
       "      <td>6830.000000</td>\n",
       "      <td>6830.000000</td>\n",
       "      <td>6.830000e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>33214.630015</td>\n",
       "      <td>0.494876</td>\n",
       "      <td>358.703953</td>\n",
       "      <td>0.505124</td>\n",
       "      <td>5.069281e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>25269.979102</td>\n",
       "      <td>0.500010</td>\n",
       "      <td>189.998301</td>\n",
       "      <td>0.500010</td>\n",
       "      <td>3.391841e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.111707e-08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5%</th>\n",
       "      <td>2711.600000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>111.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.857553e-03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>27982.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>334.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5.054829e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95%</th>\n",
       "      <td>83040.550000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>683.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>9.936389e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>145241.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>5094.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               index        label          len         pred      proba_br\n",
       "count    6830.000000  6830.000000  6830.000000  6830.000000  6.830000e+03\n",
       "mean    33214.630015     0.494876   358.703953     0.505124  5.069281e-01\n",
       "std     25269.979102     0.500010   189.998301     0.500010  3.391841e-01\n",
       "min         0.000000     0.000000    32.000000     0.000000  1.111707e-08\n",
       "5%       2711.600000     0.000000   111.000000     0.000000  7.857553e-03\n",
       "50%     27982.500000     0.000000   334.000000     1.000000  5.054829e-01\n",
       "95%     83040.550000     1.000000   683.000000     1.000000  9.936389e-01\n",
       "max    145241.000000     1.000000  5094.000000     1.000000  1.000000e+00"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "falses.describe(percentiles=[.05, .5, .95])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>label</th>\n",
       "      <th>len</th>\n",
       "      <th>pred</th>\n",
       "      <th>proba_br</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>50771.000000</td>\n",
       "      <td>50771.000000</td>\n",
       "      <td>50771.000000</td>\n",
       "      <td>50771.000000</td>\n",
       "      <td>5.077100e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>32809.481121</td>\n",
       "      <td>0.569774</td>\n",
       "      <td>409.724823</td>\n",
       "      <td>0.571153</td>\n",
       "      <td>5.680843e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>24935.362826</td>\n",
       "      <td>0.495113</td>\n",
       "      <td>203.416989</td>\n",
       "      <td>0.494916</td>\n",
       "      <td>4.421417e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.318441e-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5%</th>\n",
       "      <td>2543.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.344424e-07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>27671.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>382.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>7.861301e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95%</th>\n",
       "      <td>81126.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>760.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>148356.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>10736.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               index         label           len          pred      proba_br\n",
       "count   50771.000000  50771.000000  50771.000000  50771.000000  5.077100e+04\n",
       "mean    32809.481121      0.569774    409.724823      0.571153  5.680843e-01\n",
       "std     24935.362826      0.495113    203.416989      0.494916  4.421417e-01\n",
       "min         0.000000      0.000000     31.000000      0.000000  5.318441e-29\n",
       "5%       2543.500000      0.000000    150.000000      0.000000  5.344424e-07\n",
       "50%     27671.000000      1.000000    382.000000      1.000000  7.861301e-01\n",
       "95%     81126.000000      1.000000    760.000000      1.000000  1.000000e+00\n",
       "max    148356.000000      1.000000  10736.000000      1.000000  1.000000e+00"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_falses.describe(percentiles=[.05, .5, .95])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifier comparison\n",
    "\n",
    "How would logistic regression fare on the same problem? Were my tutors right in recommending NaiveBayes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.99915611814346"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr = LogisticRegression()\n",
    "lr.fit(train, y_train)\n",
    "lr.score(train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7444998128853085"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr.score(test, test_df.label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4VGX2wPHvyaSTQknoPbSEqiJIBxGxYF8EZVE0gIogYndVVH4WRLAgIEUURBTFBRUXZS1rQ1Gp0ntJaCmkl0lm5v39cScREUICmUzK+TzPPMmduXPvSQj3zH3LecUYg1JKKQXg4+0AlFJKlR+aFJRSShXSpKCUUqqQJgWllFKFNCkopZQqpElBKaVUIU0KSimlCmlSUJWKiBwQkRwRyRSRYyKyQERCTtmnu4h8KyIZIpImIitEJOaUfcJE5DUROeQ+1h73dsQZzisicp+IbBGRLBGJF5GlItLekz+vUqVNk4KqjK4xxoQAnYALgMcLXhCRbsB/gU+B+kAzYBOwWkSau/fxB74B2gJXAGFAdyAZ6HKGc74OjAfuA2oCrYBPgKtLGryI+Jb0PUqVFtEZzaoyEZEDwEhjzNfu7SlAW2PM1e7tH4HNxpgxp7zvCyDRGHObiIwEngeijDGZxThnS2AH0M0Y89sZ9vkOeM8Y85Z7e4Q7zp7ubQOMBe4HfIFVQKYx5qGTjvEp8L0x5hURqQ+8AfQGMoFXjTHTi/ErUqpIeqegKi0RaQhcCexxbwdjfeJfeprdPwIGuL+/DPiyOAnBrT8Qf6aEUALXA12BGOB9YIiICICI1AAuB5aIiA+wAusOp4H7/PeLyMDzPL9SmhRUpfSJiGQAcUAC8LT7+ZpYf/NHT/Oeo0BBf0GtM+xzJiXd/0xeNMacMMbkAD8CBujlfu0fwC/GmCPAxUCkMWaSMSbPGLMPmAcMLYUYVBWnSUFVRtcbY0KBvkAb/rzYpwAuoN5p3lMPSHJ/n3yGfc6kpPufSVzBN8Zq110C3OJ+6lZgsfv7JkB9EUkteAD/AuqUQgyqitOkoCotY8z3wAJgqns7C/gFGHya3W/G6lwG+BoYKCLVinmqb4CGItK5iH2ygOCTtuueLuRTtj8A/iEiTbCalf7tfj4O2G+MqX7SI9QYc1Ux41XqjDQpqMruNWCAiHRybz8G3O4ePhoqIjVE5DmgG/Cse59FWBfef4tIGxHxEZFaIvIvEfnbhdcYsxuYBXwgIn1FxF9EAkVkqIg85t5tI3CjiASLSAsg9myBG2M2AInAW8AqY0yq+6XfgHQReVREgkTEJiLtROTic/kFKXUyTQqqUjPGJALvAk+5t38CBgI3YvUDHMQattrTfXHHGGPH6mzeAXwFpGNdiCOAX89wqvuAGcBMIBXYC9yA1SEM8CqQBxwHFvJnU9DZfOCO5f2TfiYncA3WkNv9WM1ebwHhxTymUmekQ1KVUkoV0jsFpZRShTQpKKWUKqRJQSmlVCFNCkoppQpVuMJbERERpmnTpt4OQymlKpR169YlGWMiz7ZfhUsKTZs2Ze3atd4OQymlKhQROVic/bT5SCmlVCFNCkoppQppUlBKKVVIk4JSSqlCmhSUUkoV8lhSEJG3RSRBRLac4XURkenuBdH/EJELPRWLUkqp4vHkncICrEXPz+RKoKX7MRp404OxKKWUKgaPzVMwxvwgIk2L2OU64F33ClNrRKS6iNQzxpTGsoZKKQ8xxuAy4HC5cDgNeQ4XTmNwuQwOl8HpMriM9dXpMuTmu/DxAWPAZUzhV5cBsL66XAbDn6//uY+1ne90YU46d+GxCp8rOK71PCedw2BIzLBTI9i/MH7rzAU/D6ds/71y9J/7mDO858/XxTixOXKxufLwc+Xg43JgM/mIKw+by4F/fjpGbIhx4GOciMuBj3FQLS+RPFswPi4nPsaBGOtraF4CmRKCMz+X+t2G0OrCPqXzD3kG3py81oCTlh8E4t3P/S0piMhorLsJGjduXCbBKVWWXC5DrsNJTp6TLLuTfJeLPIeLfKeL1Ox8DHA8Pbdw33yXId/hIi0nn3ynC18fwe50kZhhBwN+Nh+cJ12YCx7JWXZEhABfH5wnXcQLHodOZBMR4o/DZawLvvuYgX4+GEPheyojwUUIudSQDELJIVSyCSOLauQSJtkEY6eRHCeLIIKxEyx2QskmhFzCJYuWEo8DG0GS57EY1+xpCpU4KchpnjvtX5sxZi4wF6Bz586V8y9SVRgulyElO4+kzDwy7Q7Sc/LJsDuIT8km0NeG3eHC7nCSmesgO9+60OfkObE7nCRn5ZGd58RHYG9iFqGBvtjzXeTkO887rgBfH/xsPmTaHUSE+BPga8PHB3x9fPAR91cfITnTTt3wQAJ8fbD5CH5+Pth8fPD1EeqEBZKYaadprWD8bD7uh5CcmUfDGkHYfARfm7Wvzf3wt/ngaxN8RPD1EXx8BJtI4XNZdgfVg/2x+Qg+AiIgYr0mgI9YzyMF3/91P8FKciIgCD4+FL5XCve1vvoAkpeJzZGJb34mPnmZ+GYdA6cdv4Q/8HHlI7np+OQk45O8E3wDkZwUJD8bceQU+3dtgiMxfkEQGI7xC4OgxhB4Cf752TjCG4FvECakNvgGIPk5mJC6YPNHfP3A5UT8QzC+AYjNH3xsiM0PxAfxDwabH+LjR2pmFk889TRvvb2AJs2imPfWfPr08WxCAO8mhXig0UnbDYEjXopFVUHGGDLsDk5k5pGclUdaTh7pOQ52HMsgz+Ei055PcmYeqTn51oU/10FGbj5ZecW7gAf52agWYCPQz0Y1f18C/HwI8PXB4TTUCQugdd0wHE4XjWoGu/exEeRvI8/holaIP4G+Nuui7Gtd1GtVC8Df14dgfxu+toKLsQ/+7gu3yOk+Z1VgLhfkpEBWgvU14yhkHIfE7ZCZADZ/yMuyXrNnQPJu8PEDV37Rxw1rANUioGYzcOZB897gHwJ+wRAQAoHVrYwU3ggCw8A/FALDrdd8A6EgIXnwR3c6nXS/uBc7d+7koYce4ZlnniEoKMiDZ/yTN5PCZ8BYEVmCtSh5mvYnqNKQ53CRmp3HsfRcjqXlsjshk5w8J+m51kU+MdNOUoadhAw7mXbHGY8TGRpArWr+1Aj2p0XtEEICfAkP8iM4wJcawX4E+NqIDA2gZjV/wgJ9CfK3ERLgS6CfDX+b9alcncKRB+mHISsJshIh4whkn7AeuamQcQwSd1gJAOEMjQfWxd83ACJbWxfsGk2gUVdwOaBOWwgIdT/CwC8QqkVaySAwrCx/2hJLTk6mZs2a2Gw2nn/+eRo1akTnzp3LNAaPJQUR+QDoC0SISDzwNOAHYIyZDawErgL2ANnAHZ6KRVUemXYHx9JyOJCUzYHkLOJOZJPnNBxNyyEp086h5GzSc09/oQ8L9CXCfaFvXTeU3q0iqRMWSESIPxGhAVQP8iMsyI/I0ABCA3wr3ydvTzLmz0/rSXsgLwPSDluf8tOPWBf7tHiwp5/+/f6hEFQDgsKhRjOof6G1Xbed9dUvGGpFQUhda9unck2xMsawePFixo8fz+TJkxk1ahQ33HCDV2Lx5OijW87yugHu9dT5VcXkdBmOp+dyMDmb5Cw7B5Oz2X40nT/i0zienovd4Trt+2LqhVE7LIALGtWgTlgA4cH+RIb4Uy88iLrhgdSs5o+frXJdSMpcXjac2Asn9lkX+MzjcHQT5KTC0Y1/319s7k/o9a0LfZMe1nZ4AwipA0E1rdeqRYDNr+x/nnIiLi6Ou+++m5UrV3LJJZfQo0cPr8ZT4Upnq8ohO8/BoRPZHE7J4UByNtuOpLPtaDp7EzPJO+XC36B6EM0jq9G+QThtG4RRNyyQphHVqO1uugn21z/jUuHMh5QD1iP1IKQfhcPrID/Hei7z2F/3t/lDtdrgHwydY8EvCBpcaD1Xo4nVXONj88IPUnF88MEH3HXXXTidTl577TXGjh2Lzebd35n+b1IelWV3sPN4Br/vP8H6QykcSc3lSGoOyVl/HbZXI9iP9g2r07NFLZpGVKNeeCDhQf60qhNCaGDV/RRZ6vJzrU/5CdusT/opByBpt3UHkHLAapMvID4QWs9qu4+6FGo2tzpna7WA6o2tZhxtYjsvNWrUoGvXrsydO5dmzZp5OxwA5HQTNcqzzp07G11kp/wxxnA4NYcNh1I5lpbL5sNp7DyWwZ7ETJwnjWtvVDOI7s0jaFQziCa1qlG/ehCNagYRGRKgbfilyZ5pXeiTdsOxzZC4Ew78CHmZf93PNxBqRkGt5tbXyNZWU0+NJlYTj37SL1UOh4NXX32VvLw8nnjiCcD6v1MWf/siss4Yc9Zea71TUCWW73Sx+3gmexIzWb07iX1Jmew8lvGXDt764YG0qhvKwLZ1aN+wOq3rhNLAPc5dlaKsJDj2Bxzfao3aSYu3EkDGSQP5fHytT/dR/SA4AhpcZHXghtSxOm4rWadtebVp0yZiY2NZt24dN998c2EyKG8fhjQpqLPKyM3n573JbIxLZd2BFP44nEpuvtXu72/zoX71QK7uUJ/oeqG0rhNK04hq1AkL9HLUlVDKQYj7FY5stDp2D67+6+vVIq2x9c16Q0Qrq7knohVEtLSagJRX2O12nnvuOSZPnkzNmjVZunQpN910U7lLBgU0Kai/sTucbD2Szq/7TvDLvmTW7Esmz2GVUmhTL5QhnRtxQeMatKwTQovaIQT4ahNDqUs/YnXy7vrSGs6ZsAPS463XbAHWJ/1O/4TQOtCsD0S2sb5X5c7u3bt56aWXuPXWW3nllVeoVauWt0MqkiYFBcD2o+n8sjeZX/cn8/2uxMI7gajIagzr2pgBMXW4sHENAv00AZQqp8Nq+z+22XokbIcj662JXQXCGlgTsxrcA837Wu3+VXgIZ0WQmZnJp59+yrBhw2jXrh07duygefPm3g6rWDQpVGEHkrJYueUoq7YcY1N8GmDN4r2uYwP6tI6kc9Ma1A7VZqBS48iDw2vh6B/W1/i11tBP4x6C6+NnNfdE9Yd6HaFhZ6jbwZqRqyqMr776itGjR3Pw4EEuvPBCoqOjK0xCAE0KVc7h1By+2X6c99YcZNdxayRKh4bhPDCgFTdc0IBGNYO9HGElYs+Aw+vhwE9w6BeI/x0cVqVTgiOsT//tbrKGedbraCUEbfuvsFJSUnjooYd4++23adWqFd9//z3R0dHeDqvENClUAflOFz/tSeLtn/bz4+4kAFrWDuGpQTFcHlNHE0FpcditBHDgJ9j7rdUcZNzF82q3hQtvs9r/G1zkHu6po34qC6fTSY8ePdi1axePP/44EydOJDCwYt7haVKoxLYfTWf+T/v5cssxMu0Oqgf7MbZfC67pWJ9WdULK7eiHCiM3HXb/17oD2Ps/SNlvVd0UGzS8GHpOgEZdrO+Da3o7WuUBSUlJhQXsXnjhBRo3bsyFF1bslYU1KVQyDqeL1XuTmf/Tfn7YlUignw9Xta/HgOg6XBpdW0cKnQ9jrAv/rlWw8wvrjqDgTqDhxdD6Kmj/D6szOCDUm5EqDzPGsGjRIu6//34mT57M6NGjuf76670dVqnQpFBJJGfa+WhtPIt+OcCRtFyq+duI7dmMu/o0187i8+HMt2YCb/rQmheQ5l4ssGYUdLsXWvS3+gb8yqbWvfK+gwcPctddd7Fq1Sq6d+9O7969vR1SqdKkUMGlZefz2je7WPTLQRwuw0VNavD4VdEMiKmjw0fPRcoBq3M4Ybs1U/jAaqsMtG8gtLgMut9nzQyOaOntSJUXvPfee9xzzz0YY3jjjTcYM2YMPpWsb0iTQgWVnpvPtFU7+eD3OPIcLv5xUUNGdG9Kuwbh3g6tYnG5rDUA9nwNW5db/QNgFYOrGQXtb7I6h1sO0CYhRWRkJD169GDOnDk0adLE2+F4hBbEq2DiU7J568f9/HtdPBl2B0M6N2J4tyaaDEoi5SBs+xQOrbGahHJTredrx0C7G615ArWjtUlIkZ+fz7Rp08jPz+epp54Cyq6AXWnTgniVzLG0XN78bg+L1hzEAANj6jKqdzMuaqKjWoolcZdVMmLrcmvGMFi1gdpcDY0vsRaAqRXl3RhVubJhwwZiY2PZsGEDQ4cOLbcF7EqbJoVyzuUyzPtxH698tQu7u5novktb0riWzi0okjFW2eity6xho4fXWc/XjoEB/2clA00C6jRyc3OZNGkSU6ZMISIign//+9/ceOON3g6rzGhSKMe2HUnn8eWb2RSXSveoWrxwQ3uaRlTzdljlW2YCbPrAGi2UsNV6rv4FcNmzEHOdNXtYqSLs2bOHqVOncttttzFt2jRq1Kjh7ZDKlCaFcuin3Um89dM+vtuZSEiAL8/f0I5buzSu9Let58yeaZWRWPu21URkXFCnPVw5BdoMstYEVqoImZmZLF++nOHDh9OuXTt27txZblZCK2uaFMqRPIeL2d/v5ZWvdlGzmj/jLm3B8EuaUFvXJvg7Zz788SFsWmJ1FhuXtTxk9/ug4y1Qu423I1QVxKpVqxg9ejRxcXF07tyZ6OjoKpsQQJNCufH1tuM88clmjqfb6dc6kteGXkB4kJZH/oucVKuTeOsnsO0TyE2DgHCrnETjbtbiMlpQThVTcnIyDzzwAO+++y5t2rThxx9/rJAF7EqbJgUvy7Q7mPjJFpZtOEzzyGrMv709l7aprU1FBdKPwuaPYPvn7jkExppI1uZqaD8YWg7UwnKqxAoK2O3Zs4cnnniCJ598ssIWsCttmhS8xBjD4l8P8fKqnaTl5HNXn+bc378VQf46CxmXC7Z8DOvfteoLYSAyGno9aJWUaNIdAkK8HaWqgBITE6lVqxY2m42XXnqJJk2a0KlTJ2+HVa5oUvCCQ8nZTPp8K19vT+CCxtV58uoYLmpStUY4nFZmAmz5N/w+35plHNbQSgQdbrZWG1PqHBljWLBgAQ888ACTJ0/mrrvu4rrrrvN2WOWSJoUy9u2O44xfshGXy/DAgFaM6RuFr60KN38YYxWc2/wxbF4K+dlQpx1cNws6DAGb/omq83PgwAFGjx7NV199Ra9evejXr5+3QyrX9H9cGTHGmoT24hc7iIoMYe7wi2geWUWbQIxxr0S21koEx/4Av2BoNRB6PQR12oL2qahSsGjRIu655x5EhFmzZnHXXXdVugJ2pU2TQhnItDt46KNNfLn1mDWyaMgFhAdXwZFFednw62xYv9CqRgrWDOOrp0HHW8FfZ2mr0lWnTh169+7N7Nmzady4sbfDqRC0IJ6HJWfaGTz7F/YlZTG+f0vu698Sm08V+xScegh+nQMb34ecE9bw0XY3WXcG1fU/qio9+fn5TJkyBafTycSJE70dTrmiBfHKgTyHi1HvriUuJZu3R3Tm0jZ1vB1S2Tq8Hn542VqlTARaDIDu46BZL29Hpiqh9evXc+edd7Jp0yZuvfXWClvN1Ns0KXhIcqade95bz/pDqTw8sHXVSgh7voE1b8Ker8CvGvQYDxfdblUlVaqU5eTk8OyzzzJ16lQiIyNZvnx5pVka0xs8mhRE5ArgdcAGvGWMmXzK642BhUB19z6PGWNWejKmspCSlcfNc34hLiWHaYM7ctNFDb0dkuc582HnSlg9HQ6vhZC61nDS7uOs8hNKeci+fft45ZVXGDFiBC+//HKVK2BX2jyWFETEBswEBgDxwO8i8pkxZttJuz0JfGSMeVNEYoCVQFNPxVQW8hwuRr67loPJ2bw94mJ6t4r0dkie5bBbfQU/vWL1HYQ3hismw0V3gJ/OEFWekZ6ezrJlyxgxYgRt27Zl9+7dlXYltLLmyTuFLsAeY8w+ABFZAlwHnJwUDBDm/j4cOOLBeMrEjG93s+5gCi/c0L5yJ4T8HPhlJvw2FzKPQ2QbGLzQqkqqcwuUB61cuZK7776bw4cP07VrV6KjozUhlCJP/u9tAMSdtB0PdD1ln2eA/4rIOKAacNnpDiQio4HRQLkeVvbO6v1M/3YPfVtHcmvX8hvneclNs2Yc/zYPMo5Awy5wzXRrJJF26ikPSkpKYsKECbz33nvExMSwevVqLWDnAZ5MCqe7Qpw6/vUWYIExZpqIdAMWiUg7Y4zrL28yZi4wF6whqR6J9jwtWnOQZ1ds47Lo2sy49UJvh1P68rJgzSxYMxuyk6BpL7huBkRdqslAeVxBAbt9+/YxceJE/vWvfxEQoBVxPcGTSSEeaHTSdkP+3jwUC1wBYIz5RUQCgQggwYNxlbrPNh3hqU+20Ld1JDNuvZBAv0pU1M5htxav+d8LYE+3FrXveb9VplopDzt+/DiRkZHYbDamTp1KkyZN6NChg7fDqtQ8Od/7d6CliDQTEX9gKPDZKfscAvoDiEg0EAgkejCmUve/nQk8+NFGOjWqzqxhlSwh7P8BZnSGLx+zSk/cvgKGL9OEoDzOGMP8+fNp3bo1c+fOBeCaa67RhFAGPHanYIxxiMhYYBXWcNO3jTFbRWQSsNYY8xnwIDBPRCZgNS2NMBVoivXqPUmMXLiWJjWDeev2zgT7V5IO1iMb4ZtnYe+3ENYAhr4Pra/SZiJVJvbt28eoUaP49ttv6dOnD5dddtquRuUhHr2KueccrDzluYknfb8N6OHJGDwlNTuP8Us20rhmMEvuuoSIkErQvunIg3XvwFcTwRYAlz4JXe6CwLCzv1epUrBw4ULGjBmDzWZj9uzZjBo1SgvYlbFK8tG27E1asY3kLDszb72A2qGVYDz+rlWw8mFIPQhNesJN8yCsvrejUlVM/fr1ufTSS3nzzTdp2LAKTPoshzQpnIP31hxk2YbDjOrVjK7Na3k7nPNjz4Svn4bf34LqTWDIYmupS20qUmUgLy+PyZMn43K5eOaZZxgwYAADBgzwdlhVmiaFEkrNzuOFldvp2SKCR69o4+1wzp0xVlmKVU9Ayn64eCRc/rzOQlZl5vfff+fOO+9ky5YtDB8+XAvYlRPaWFcCxhjuW7KR3Hwnj13ZpuKumJa4ExbdAEtuBZcDhi+31jTQhKDKQHZ2Ng899BCXXHIJKSkpfPbZZ7z77ruaEMoJvVMogTk/7OOHXYk8PLA17RqEezucksvLtmoU/TgNfIOg/9NWwTpbFVzwR3nN/v37eeONNxg1ahQvvfQS4eEV8P9SJaZJoZgOJWfzyle76NgwnDF9o7wdTsnFr4WPboP0w9D2BqupKLyBt6NSVURaWhrLli3jjjvuoG3btuzZs4dGjRqd/Y2qzFXQ9o+yN+nzbbhchmk3d6x4t7mH1sC711uzk2/7FP7xjiYEVWb+85//0LZtW0aOHMmOHTsANCGUY5oUiuH7XYl8vf04d/RoSovaod4Op/jysuCbSfD2FdaaBrH/heZ9dWSRKhOJiYkMGzaMQYMGUaNGDX755RfatKnAgzOqCG0+KobJX+wgMjSABwa09nYoxXdsi9VcdGIvdBgCV78CASHejkpVEU6nk549e7J//36effZZHnvsMfz9/b0dliqGYiUFd+2ixsaYPR6Op9z5344Eth9N5+lrYgjyryB1jX6ZBf990koCwz6GljruW5WNY8eOUbt2bWw2G9OmTaNp06a0a9fO22GpEjhr85GIXA1sBr5yb3cSkeWeDqw8yMjN5/mV26lVzb9irI+QlQQfDodVj1vNROM2aEJQZcLlcjFnzhxatWrFnDlzABg0aJAmhAqoOHcKk7AWx/kfgDFmo4i08GhU5cTnfxxlT0Imc4ZfRIBvOb5LcOTBLzNg9WvWIjhd7oKBL+gKaKpM7Nmzh1GjRvHdd99x6aWXMnDgQG+HpM5Dca4a+caY1FNG3FSYSqbnyhjDy6t2Ujs0gAHRdbwdzpnF/Q6fjYXEHdbCN5c/B/U7eTsqVUW88847jBkzBn9/f+bNm0dsbGzFG52n/qI4SWG7iNwM+IhIM2A8sMazYXnfz3uTOZGVx5NXR+PjUw7/yO2ZVnnr3+ZBaF0tb628onHjxgwcOJCZM2fSoIEOc64MipMUxgITARewDGt9hMc9GVR5sODnAwT6+TCsazlcEPzEPqtMRcoB6DAUrngRgmt6OypVBdjtdl588UVcLheTJk2if//+9O/f39thqVJUnKQw0BjzKPBowRMiciNWgqiUDqfm8NW249zZo1n5G3GUuBMWXG31Iwz/BKL6eTsiVUX8+uuvxMbGsnXrVm6//XYtYFdJFWfy2pOnee6J0g6kPPlkw2GA8jfi6MBqWHgN5OfCiM81IagykZWVxQMPPEC3bt1IS0vj888/Z8GCBZoQKqkz3imIyEDgCqCBiLxy0kthWE1JlZLLZfhobRydGlWnRe1yNNlr0xJYMR6qRcLIT6F2tLcjUlXEwYMHmTVrFnfffTeTJ08mLExX4qvMimo+SgC2ALnA1pOezwAe82RQ3rR6bxIHk7O5q3c5KXrncsEPL8N3L0CjS2DwO7oimvK41NRUPv74Y0aOHElMTAx79uzRldCqiDMmBWPMBmCDiCw2xuSWYUxe9eWWYwT4+nDDBeVgJEV+Diy9A3Z9ATHXww2zwS/I21GpSu7TTz/lnnvuISEhgZ49e9KmTRtNCFVIcfoUGojIEhH5Q0R2FTw8HpkXGGP4eW8ynZvW8H4Hsz0DPvynlRD6T7Qqm2pCUB6UkJDA0KFDuf7664mMjGTNmjVawK4KKs7oowXAc8BU4ErgDippn8J3uxLZn5TFyF7NvBtIahy8dyMk7YYrJsMl93g3HlXpOZ1OevTowaFDh3juued45JFH8PPTxZeqouIkhWBjzCoRmWqM2Qs8KSI/ejowb/hq23GC/GzcdKEXb5WTdsPCayEnBYYt1dpFyqOOHDlC3bp1sdlsvP766zRt2pSYmBhvh6W8qDjNR3axxp7tFZG7ReQaoLaH4/KKTXGpXNSkBoF+Xmo6StxlLYaTnwW3r9CEoDzG5XLx5ptv0qZNG2bPng3AVVddpQlBFSspTABCgPuAHsAo4E5PBuUNyZl2th1Np31DL60Xu/d/1qS0vEwY9m9odLF34lCV3q5du+jXrx9jxoyha9euXHnlld4OSZUjZ20+Msb86v42AxgOICKVbijC7wdOYAz0ahnhhZPPh5UPQ60o+MfbULd92cegqoT58+czduxYAgMDefvttxkxYoROQlOQNZJdAAAgAElEQVR/UeSdgohcLCLXi0iEe7utiLxLJSyI9/X2BAAubFyjbE+89m34zwPQpLu1XKYmBOVBTZs25corr2Tbtm3ccccdmhDU35wxKYjIi8BiYBjwpYg8gbWmwiagVdmEV3bWH0qhYY2gsu1P+PEV+HyCVfJ62FJrHWWlSpHdbufJJ5/kySetajX9+/dn2bJl1KtXz8uRqfKqqOaj64COxpgcEakJHHFv7yyb0MqO3eEkId1O12ZlWGl0/SKr9HWrK2DwAp2DoErdzz//TGxsLDt27ODOO+/UAnaqWIpqPso1xuQAGGNOADsqY0IAWHcghUy7g5suKqOukm2fwor7oP4FMHihJgRVqjIzMxk/fjw9e/YkOzubL7/8kvnz52tCUMVSVFJoLiLL3I/lQNOTtotVNltErhCRnSKyR0ROWy9JRG4WkW0islVE3j+XH+J8fbX9OEDZ3CnEr4OP77T6Dkb8B/wCPX9OVaUcOnSIOXPmcO+997JlyxZdHlOVSFHNRzedsj2jJAcWERswExgAxAO/i8hnxphtJ+3TEmvBnh7GmBQRKfP5D8YYftiVSJdmNakVEuDZk+WkwL9jITAcbvsU/Kt59nyqykhJSWHp0qWMHj2amJgY9u3bR/36WjhRlVxRBfG+Oc9jdwH2GGP2AYjIEqx+im0n7TMKmGmMSXGfM+E8z1li6TkO9iZmcW1HDxfAcznhg1sg9RAMX66dyqrULF++nDFjxpCYmEifPn1o3bq1JgR1zoozee1cNQDiTtqOdz93slZAKxFZLSJrROSK0x1IREaLyFoRWZuYmFiqQR5OzQGgaURwqR73bz4dC4d+gaunQvM+nj2XqhKOHTvG4MGDufHGG6lbty6//fYbrVu39nZYqoIrTu2jc3W6Xi1zmvO3BPoCDYEfRaSdMSb1L28yZi4wF6Bz586nHuO8bD+aDkCdMA+27R9aA5veh463wkV3eO48qspwOp306tWLuLg4XnjhBR566CEtYKdKRbGTgogEGGPsJTh2PNDopO2GWMNaT91njTEmH9gvIjuxksTvJTjPedl6JB0/m3BREw8159gzrLkI1SLhqimgI0DUeYiPj6d+/frYbDamT59Os2bNtLy1KlVnbT4SkS4ishnY7d7uKCJvFOPYvwMtRaSZiPgDQ4HPTtnnE6Cf+7gRWM1J+0oQ/3nbGJdCTP1w/GweaElzOqx+hIRtMOg1CAgt/XOoKsHlcvHGG2/Qpk0b3nzzTQCuvPJKTQiq1BXnSjgdGAQkAxhjNuG+kBfFGOMAxgKrgO3AR8aYrSIySUSude+2CkgWkW1Ys6UfNsYkl/zHODc5eU42xafR2VN3Cd9PhgM/woBJED3IM+dQld6OHTvo3bs39913Hz179mTQIP1bUp5TnOYjH2PMwVMmvjiLc3BjzEpg5SnPTTzpewM84H6UuZ/2JOF0GS5pXqv0D779c2tt5faDocf40j++qhLeeustxo4dS3BwMAsXLmT48OE6CU15VHGSQpyIdAGMe+7BOKBSLMd5OCUbgLb1w0r3wKlxsPxuqB0D15ZoeodSfxEVFcU111zDjBkzqFOnjrfDUVVAcZLCPVhNSI2B48DX7ucqvDX7ThDo50O98FIceZSfA4v/AS4HDHlPZyyrEsnNzWXSpEkAvPDCC/Tr149+/c7aWqtUqSlOUnAYY4Z6PBIvOJaeS8MawaV7O/7LTEjcYRW5qxVVesdVld7q1auJjY1l586djBw5UgvYKa8oTkfz7yKyUkRuF5FKM3zG5TJsjEulW2n2JyTuhO9fgtZXQ9sbSu+4qlLLyMhg3Lhx9OrVC7vdzqpVq5g3b54mBOUVZ00Kxpgo4DngImCziHwiIhX+zuFYei5A6Q1FdebD8rvAFgBXvVw6x1RVQnx8PG+99Rbjxo1j8+bNXH755d4OSVVhxboiGmN+NsbcB1wIpGMtvlOhncjKA6Bz01IajrrmTTiyAQY8C+EerqOkKrzk5OTC+QbR0dHs27eP119/nZCQEC9Hpqq64kxeCxGRYSKyAvgNSAS6ezwyDzvuvlOoWxqdzA47/PQKNOoKF8ee//FUpWWM4eOPPyYmJob77ruPnTutJUp0JTRVXhTnTmELcAkwxRjTwhjzoDHmVw/H5XF/xKcBlM7Io9XTrbLYfR49/2OpSuvo0aPcdNNNDB48mEaNGrF27VotYKfKneKMPmpujHF5PJIylpaTD0Cd0PNMCsl7rbuE1ldBi/6lEJmqjAoK2B0+fJgpU6YwYcIEfH09WY9SqXNzxr9KEZlmjHkQ+LeI/K0yqTHmRo9G5mEb4lJpUTsEH5/zGOHhdMCy0YDAFZNLLTZVecTFxdGgQQNsNhszZ86kWbNmtGrVytthKXVGRX1U+dD9tVJOyc22O/A9n4QA8ONUOLwWbpwHNZqUTmCqUnA6ncycOZPHH3+cKVOmcO+99+qymKpCOGOfgjHmN/e30caYb05+ANFlE57n7E7IpFOj6ud+gB0r4bsXrWaj9oNLLzBV4W3fvp1evXoxfvx4+vTpwzXXXOPtkJQqtuJ0NN95mucq9BAbu8Oq5xcScI5tutknYMV9ENEKbpijaySoQnPnzqVTp07s2rWLRYsW8Z///IfGjRt7Oyyliq2oPoUhWGsgNBORZSe9FAqknv5dFUNypjVHoUmtc1yC88dpkJUIt34IgaVcTE9VaC1btuSGG25g+vTp1K5d29vhKFViRX1U/g1rDYWGwMyTns8ANngyKE87dMKqjtqgRlDJ35yfC+sWQsuB0OCiUo5MVTQ5OTk888wziAiTJ0/WAnaqwjtjUjDG7Af2Y1VFrVTi3EmhecQ5zB5dOx/yMuDikaUclapofvjhB0aOHMnu3bu5++67tYCdqhTO2KcgIt+7v6aIyImTHikicqLsQix9BXMUqgeXcKHztHj4+hloeDG0HFD6gakKIT09nTFjxtCnTx+cTifffPMNb775piYEVSkU1XxUcA8cURaBlKWEDDv+Nh/Cg0qYFL5+FozLWjhHLwBV1pEjR1iwYAEPPPAAkyZNolq1at4OSalSU9SQ1IJZzI0AmzHGCXQD7gIq9P+C4+m5RIYGlOyTXfJe2PJvuPA2qK2LpVc1SUlJzJo1C4A2bdqwf/9+pk2bpglBVTrFGZL6CdZSnFHAu1hzFN73aFQetv1oOrXDAkr2pk/uAfGBbmM9E5Qql4wxfPjhh8TExHD//feza5e1Eq0ujakqq+IkBZcxJh+4EXjNGDMOqNC1oTNzHdhKcpdwdBPE/Qp9H9XV1KqQI0eOcP311zN06FCaNGnCunXrtESFqvSKtRyniAwGhgPXu58rYWN8+ZLrcNGyTgkWkVv/Lvj4QecKPWdPlYDT6aR3794cPnyYqVOnMn78eC1gp6qE4vyV3wmMwSqdvU9EmgEfeDYsz3G5DGk5+dQo7sijzATY+L61vGZwTc8Gp7zu4MGDNGzYEJvNxqxZs2jevDktWrTwdlhKlZniLMe5BbgPWCsibYA4Y8zzHo/MQzJyHThdpvjDUb+bDPk50PshzwamvMrpdPLKK68QHR1duCLa5ZdfrglBVTlnvVMQkV7AIuAwIEBdERlujFnt6eA84UhaDgC+PsXoTkk/AhsWwQXDIFIXQ6mstmzZQmxsLL/99huDBg3i+uuvP/ublKqkitN89CpwlTFmG4CIRGMlic6eDMxTMu0OAOqEFWNxndWvg8sBvR70cFTKW2bPns19991HeHg477//PkOHDtVJaKpKK87oI/+ChABgjNkO+HsuJM9Kd89mPuvazC4X7FwJzftCzeYej0uVLWOsdaOio6MZPHgw27Zt45ZbbtGEoKq84twprBeROVh3BwDDqMAF8QpKXNSsdpa8dngtpB7Su4RKJjs7m4kTJ2Kz2XjppZfo06cPffr08XZYSpUbxblTuBvYCzwCPArsw5rVXCHl5lsTtYP8bEXvuPYda7Ja9LVlEJUqC9999x0dOnRg2rRpZGZmFt4tKKX+VOSdgoi0B6KA5caYKWUTkmfFpVgVUgP9isiHDjtsXwGNLtFhqJVAWloajzzyCHPnziUqKopvv/1Wy1srdQZFVUn9F1aJi2HAVyJyuhXYKpxg9x1CtaJWXTu42iqP3WVUGUWlPOno0aO89957PPTQQ/zxxx+aEJQqQlHNR8OADsaYwcDFwD0lPbiIXCEiO0Vkj4g8VsR+/xARIyIeH9GUk+/Ezyb42c7woxsD370EQTWh5eWeDkd5SGJiIm+88QZgFbA7cOAAL7/8MsHB57janlJVRFFJwW6MyQIwxiSeZd+/EREb1optVwIxwC0iEnOa/UKxJsf9WpLjn6sTWXmEBhYxce3gaohbA/3+BQHnsAiP8ipjDO+//z7R0dE8+OCDhQXsIiMjvRyZUhVDURf65iKyzP1YDkSdtL2siPcV6ALsMcbsM8bkAUuA606z3/8BU4DcEkd/DnYnZBbdyfzjK+BXDdoPLotwVCmKi4vjmmuuYdiwYbRo0YINGzZoATulSqiojuabTtmeUcJjNwDiTtqOB7qevIOIXAA0MsZ8LiJnrCMhIqOB0QCNGzcuYRh/Fexv44yTmVMOwt5voOcECKp+XudRZcvhcNC3b1+OHTvGq6++yrhx47DZzjLCTCn1N0Wt0fzNeR77dLOACscAiogP1mzpEWc7kDFmLjAXoHPnzuc1jvBEVh71w4NO/+LmpdbXDkPO5xSqDB04cIBGjRrh6+vLnDlzaN68Oc2b62RDpc5VifoJSigea9W2Ag2BIydthwLtgO9E5ABwCfCZpzub03LyyXe6Tv/i5qVQ/wKoHe3JEFQpcDgcTJ06lejo6MIV0S677DJNCEqdJ08mhd+BliLSTET8gaHAZwUvGmPSjDERxpimxpimwBrgWmPMWg/GxNE0aynOv8k4Dok7oOVAT55elYI//viDbt268fDDDzNw4EBuuunUlk6l1LkqdlIQkRKtX2mMcQBjgVXAduAjY8xWEZkkIl6bJhzsb+O0Nwqb3EtERA8q03hUycyaNYuLLrqIgwcP8uGHH7J8+XLq16/v7bCUqjSKUzq7CzAfCAcai0hHYKR7Wc4iGWNWAitPeW7iGfbtW5yAz5fTZWha6zRj1fd+C7VaQN32ZRGGKiFjDCJCu3btGDp0KK+++ioRERHeDkupSqc4dwrTgUFAMoAxZhNQYaeE2h0u/H1P+bHtGXDwZ4i61DtBqTPKyspiwoQJPPLIIwD07t2bRYsWaUJQykOKkxR8jDEHT3nO6YlgPM3pMjhdpxm8tGMluPKhjTYdlSfffPMN7du357XXXsNut2sBO6XKQHGSQpy7CcmIiE1E7gd2eTguj8hzWJ0JzlMvLvG/W18bVsh1gyqd1NRURo4cyWWXXYavry8//PAD06dP17UOlCoDxUkK9wAPAI2B41hDR0tcB6k8yHP3MNcOPWmBHWNg9yqr6ci/mpciUyc7fvw4S5Ys4dFHH2XTpk306tXL2yEpVWWctaPZGJOANZy0wnO4k4Kf7aRPnEc3WYvpdL/PS1Ep+DMRjB8/ntatW3PgwAHtN1DKC4oz+mgeJ81ELmCMGe2RiDzI4e5P+EuF1IKmo8aXeCEiZYxh8eLFjB8/nszMTK666ipatmypCUEpLylO89HXwDfux2qgNmD3ZFCekml3AJDl/grA3v9BWEOo085LUVVdhw4d4uqrr2b48OG0bt2ajRs30rJlS2+HpVSVVpzmow9P3haRRcBXHovIgwrKW9Sv7q59ZAwc2QCNu4J2YpapggJ2CQkJTJ8+nTFjxmgBO6XKgbMmhdNoBjQp7UDKQmp2PnDSUpxJuyDjCDTp4cWoqpZ9+/bRpEkTfH19mTdvHlFRUTRt2tTbYSml3M7afCQiKSJywv1IxbpL+JfnQyt9BXMUCkekHvrF+tq4m3cCqkIcDgcvvfQSMTExzJw5E4D+/ftrQlCqnCnyTkGsgeEdgcPup1ymAs8gsjusOXcRIe4yTvG/g28g1Gnrxagqv40bNxIbG8v69eu54YYbGDxYFzBSqrwq8k7BnQCWG2Oc7keFTQgAx9Ks/vGAguajhB1Q/0LtT/CgGTNmcPHFF3P48GE+/vhjli1bRr169bwdllLqDIoz+ug3EbnQ45GUgYL5Cf4FQ1JP7IPI1l6MqPIq+PzQoUMHhg0bxrZt27TEtVIVwBmbj0TE113+uicwSkT2AllYK6oZY0yFSxQFM5pDAnyt9RNyTkD1Rmd5lyqJzMxMnnjiCfz8/Jg6dSq9e/emd+/e3g5LKVVMRfUp/AZcCFxfRrF4nD3fSgoBvjbYt9p6sqmWUCgt//3vfxk9ejSHDh1i3LhxheWulVIVR1FJQQCMMXvLKBaPy86zJq0F+dtg63LwC4Z6Hb0cVcWXkpLCAw88wIIFC2jdujU//PADPXv29HZYSqlzUFRSiBSRB870ojHmFQ/E41F5Dhci7r6FpN1Qszn4lmhBOXUaCQkJfPzxxzz++ONMnDiRwMDAs79JKVUuFZUUbEAI7juGyiDfZfDz8UHycyBxO3Qb6+2QKqxjx47xwQcfMGHChMICdrVq1fJ2WEqp81RUUjhqjJlUZpGUgXyHC1+bQNJO64na0d4NqAIyxvDuu+8yYcIEsrOzGTRoEC1bttSEoFQlUdSQ1Epzh1Dg0IlsbCJweJ31RPXG3g2ogjlw4ABXXHEFI0aMICYmRgvYKVUJFXWn0L/MoigjNYL9ybA7rPUTAGrrTObicjgc9OvXj6SkJGbOnMndd9+Nj09xprkopSqSMyYFY8yJsgykLOQ7XTSsEQT5udYTwTW9G1AFsGfPHpo1a4avry9vv/02zZs3p0mTClkPUSlVDFXqo16e04W/rw8c3wIRrbW8RRHy8/N54YUXaNu2bWEBu379+mlCUKqSO5fS2RXWgeQs/Hx8ICsR/IK8HU65tX79emJjY9m4cSODBw9myJAh3g5JKVVGqtSdQkRIAEmZdmsdhbrtvR1OuTR9+nS6dOnCsWPHWLZsGR999BF16tTxdlhKqTJSpZKC02WIruEu9BpY3bvBlDMFBewuuOACbrvtNrZt28YNN9zg5aiUUmWtSjUfOZyGhibR2ojQoZQAGRkZPP744wQEBDBt2jR69epFr15aD0qpqqrK3Sk0dMVbG3U7eDeYcuDLL7+kXbt2zJo1C2MMFXy5DKVUKahSScHhclHb5b5TqNncu8F4UXJyMrfffjtXXnkl1apVY/Xq1bzyyita0VQpVbWSgtNlaOCIA5s/BIZ7OxyvSU5OZvny5Tz11FNs2LCBbt10jWqllMWjSUFErhCRnSKyR0QeO83rD4jINhH5Q0S+ERGPDoJ3uAzhrlTwDapycxSOHj3K1KlTMcbQqlUrDh48yKRJkwgI0CqxSqk/eSwpiIgNmAlcCcQAt4hIzCm7bQA6G2M6AB8DUzwVD1h3CnUch6FG1ZmAZYzh7bffJjo6mqeeeoo9e/YAUKNGDS9HppQqjzx5p9AF2GOM2WeMyQOWANedvIMx5n/GmGz35hqgoQfjweEy5PkEgsvpydOUG/v37+fyyy8nNjaWjh07smnTJi1gp5QqkieHpDYA4k7ajge6FrF/LPDF6V4QkdHAaIDGjc+9sqnTZQhxpkGdvud8jIrC4XBw6aWXkpyczJtvvsno0aO1gJ1S6qw8mRRO12h/2jGPIvJPoDPQ53SvG2PmAnMBOnfufM7jJh0uFzUdCZW6EN7u3btp3rw5vr6+vPPOO0RFRdGoUSNvh6WUqiA8+dExHjj5atQQOHLqTiJyGfAEcK0xxu7BeMBhrdGMM9+jp/GG/Px8nnvuOdq1a8eMGTMA6Nu3ryYEpVSJePJO4XegpYg0Aw4DQ4FbT95BRC4A5gBXGGMSPBgLAH6uHOubGk09faoytXbtWmJjY/njjz8YOnQot9xyi7dDUkpVUB67UzDGOICxwCpgO/CRMWariEwSkWvdu72MtQ70UhHZKCKfeSoegBBXhvVNUOWpe/T666/TtWtXkpKS+PTTT/nggw+oXbu2t8NSSlVQHq19ZIxZCaw85bmJJ31/mSfPf6o6ruPWN9Uq/kXTGIOI0LlzZ2JjY5kyZQrVq1eeZKeU8o4qVRCvljlhdX8HhHg7lHOWnp7Oo48+SmBgIK+++io9evSgR48e3g5LKVVJVKkxikEFUyLCPTodwmNWrlxJ27ZtmTt3Lr6+vlrATilV6qpUUvAzedY3QRVrNm9SUhL//Oc/ufrqqwkPD+fnn3/m5Zdf1gJ2SqlSV6WSQpjJxCDgX7Gaj1JSUlixYgVPP/0069evp2vXouYAKqXUuasyfQoulyGYXPJs1QjwsXk7nLM6fPgwixcv5uGHH6Zly5YcPHhQO5KVUh5XZe4UnMZQU9Kx+5bvuwRjDPPmzSMmJoZnnnmGvXv3AmhCUEqViSpzp+B0GULIIdevBmHeDuYM9u7dy6hRo/jf//5H3759mTdvHi1atPB2WEqdk/z8fOLj48nNzfV2KFVKYGAgDRs2xM/P75zeX2WSgsNlaCFHcNrqezuU03I4HPTv358TJ04wZ84cRo4cqQXsVIUWHx9PaGgoTZs21UERZcQYQ3JyMvHx8TRr1uycjlFlkoLTacgkiDBnprdD+YudO3cSFRWFr68vCxcuJCoqioYNK+aQWaVOlpubqwmhjIkItWrVIjEx8ZyPUWU+ijpcLlrKYdJCysfazHl5eTz77LO0b9+emTNnAtCnTx9NCKpS0YRQ9s73d1517hRchiwC8HPmeDsUfvvtN2JjY9myZQu33norw4YN83ZISikFVKU7BaeTUHLICPNux+1rr71Gt27dCuceLF68mIiICK/GpFRlJSI8+OCDhdtTp07lmWeeKfI9n332GZMnTz6n8z3zzDM0aNCATp060aZNG+655x5cLtc5HctbqkxScOXl4CdOHH7eGXtUUJKiS5cujBo1iq1btzJo0CCvxKJUVREQEMCyZctISkoq9nuuvfZaHnvssXM+54QJE9i4cSPbtm1j8+bNfP/99+d8LG+oMs1HDqe1wI7Yzm2Y1rlKS0vjkUceISgoiNdee43u3bvTvXv3Mo1BKW97dsVWth1JL9VjxtQP4+lr2ha5j6+vL6NHj+bVV1/l+eef/8trK1as4LnnniMvL49atWqxePFi6tSpw4IFC1i7di3PP/88HTt2ZN++ffj4+JCdnU3r1q3Zt28fhw4d4t577yUxMZHg4GDmzZtHmzZt/nL8vLw8cnNzqVHDKqszb9485s6dS15eHi1atGDRokU4nU46dOjArl278PPzIz09nQ4dOrB79+4znmPp0qU8++yz2Gw2wsPD+eGHH0r191p17hQc7rpHZZgUVqxYQUxMDG+99RYBAQFawE4pL7j33ntZvHgxaWlpf3m+Z8+erFmzhg0bNjB06FCmTJnyl9fDw8Pp2LFj4Sf9FStWMHDgQPz8/Bg9ejRvvPEG69atY+rUqYwZM6bwfa+++iqdOnWiXr16tGrVik6dOgFw44038vvvv7Np0yaio6OZP38+oaGh9O3bl//85z8ALFmyhJtuuqnIc0yaNIlVq1axadMmPvus9JegqTJ3Cs6COwUfz//IiYmJjB8/ng8++ID27dvzySefcPHFF3v8vEqVV2f7RO9JYWFh3HbbbUyfPp2goKDC5+Pj4xkyZAhHjx4lLy/vtOP6hwwZwocffki/fv1YsmQJY8aMITMzk59//pnBgwcX7me3/7mS8IQJE3jooYfIz8/nH//4B0uWLGHo0KFs2bKFJ598ktTUVDIzMxk4cCAAI0eOZMqUKVx//fW88847zJs3r8hz9OjRgxEjRnDzzTdz4403lvrvq8rcKTjz3esy2zyfFNLS0li5ciXPPvssa9eu1YSglJfdf//9zJ8/n6ysrMLnxo0bx9ixY9m8eTNz5sw57czra6+9li+++IITJ06wbt06Lr30UlwuF9WrV2fjxo2Fj+3bt//tvX5+flxxxRWFzTsjRoxgxowZbN68maeffrrwfD169ODAgQN8//33OJ1O2rVrV+Q5Zs+ezXPPPUdcXBydOnUiOTm5VH9XVSYpmHzrH8DmoWHTcXFxvPjiixhjaNGiBQcPHmTixIn4+/t75oRKqWKrWbMmN998M/Pnzy98Li0tjQYNGgCwcOHC074vJCSELl26MH78eAYNGoTNZiMsLIxmzZqxdOlSwBpEsmnTpr+91xjDzz//TFRUFAAZGRnUq1eP/Px8Fi9e/Jd9b7vtNm655RbuuOMOgCLPsXfvXrp27cqkSZOIiIggLi7ufH41f1NlkkJBn4JPKScFl8vF7Nmzadu2Lc8991xhAbvw8PDSPZFS6rw8+OCDfxmF9MwzzzB48GB69epV5LDwIUOG8N577zFkyJDC5xYvXsz8+fPp2LEjbdu25dNPPy18raBPoV27djgcjsK+gP/7v/+ja9euDBgw4G+d0sOGDSMlJYVbbrnlrOd4+OGHad++Pe3ataN379507Njx/H4xpzLGVKjHRRddZM7FH2t/NObpMLPtm/fO6f2ns2vXLtOnTx8DmP79+5u9e/eW2rGVqui2bdvm7RAqjKVLl5p//vOfpXa80/3ugbWmGNfYKtPRbBzujiBb6TTnOBwOBgwYQGpqKvPnz+eOO+7QKf1KqRIbN24cX3zxBStXrvR2KEAVGn3kk3MCANt5th9t376dli1b4uvry6JFi4iKiqJ+/fJZeVUpVf698cYb3g7hL6pMn4Itx2pLdAWfW0kJu93O008/TYcOHZgxYwYAvXr10oSglKpUqsydgstYdwjiX63E712zZg2xsbFs27aN4cOHM3z48NIOTymlyoUqc6fgMlZRKvEPOsuefzVt2jS6d+9ORkYGK1eu5N1336VWrVqeCFEppbyuyiQF465UaCvmjOaCyobdunXj7rvvZsuWLVx55ZUei08ppcqDKpMUXO4yFxCWe3sAAA01SURBVD62on/k1NRUYmNjGT9+PADdu3dn1qxZhIWV15WdlVJnEhISct7HOHDgAEFBQXTq1ImYmBhuu+028gsqJFRCVSYpGHfzkW8RZS4++eQTYmJiWLhwIaGhoVrATikFQFRUFBs3bmTz5s3Ex8fz0UcfeTskj6k6Hc1OJwA+NtvfXktISGDs2LEsXbqUTp068fnnn3Ph/7d3/9FV1/cdx58vEI0Oiq3aQYkYemLd+JFSlkW0uI7RIVgnrkYRA8iQSR2ZtI55nM6zrpuRtTZVrEgRiJbTFipQx+nqQU75IboEgmsNhFUJP6RxTFmmuIIIJO/98f1we0kCuUnuveHe+36ck3Pu/d7P937f73uT+7nfz+eb92fUqHSH6Fz2evEB+O8dyX3OASNgYucXw3nrrbeYOXMmhw4d4rLLLqOqqorBgwezZ88eysrKaG5uZuLEiVRWVvKb35y+pnvv3r0pKSnh7bffBqC5uZkHHniATZs28dFHHzFnzhxmz55NS0sL5eXlbN68mSFDhtDS0sLMmTMpLS1NSuqplDNnCrREncJ57QwfffDBB6xfv55HHnmEbdu2eYfgXBYrLy9n+vTp1NXVUVZWxr333gvA3LlzmTt3LrW1tWe81PzYsWNs3bqVCRMmALB06VL69+9PbW0ttbW1PPPMM+zbt481a9awf/9+duzYwZIlS6iurk5bft2VO2cKYeK4Vxg+OnDgAMuXL+fBBx+ksLCQAwcO0K9fv54M0bns1YVv9KlSXV3NmjVrAJg2bRr3339/bPsLL7wAwB133MG8efNi++zZs4eRI0eye/duSktLKSoqAuCll16irq6OVatWAVGRvd27d/PKK69w66230qtXLwYMGMDYsWPTmWK3pPRMQdIESW9IapDUZn07SRdIWhke3yqpIFWxnJpT6KVeLFy4kGHDhlFRURErYOcdgnO5KZHyNKfmFBoaGqipqYktbmNmPPnkk7Hy1vv27WP8+PEZPR+Zsk5BUm/gKWAiMBSYImloq2Z3Ae+ZWSHwHeBfUhWPheGjKVOmMGfOHK655hrq6+spLCxM1SGdc+ega6+9lhUrVgBRJdIxY8YAMHr0aFavXg0Qe7y1gQMHMn/+fB599FEArr/+ep5++unY1UhvvvkmR44cYcyYMaxevZqWlhbeeecdNm3alOKskieVZwolQIOZ7TWz48AKYFKrNpOAU4XMVwHjlKKqcqcuSa2vr6eqqop169ZRUFCQikM5584RR48eJT8/P/ZTWVnJggULqKqqoqioiOXLl/PEE08A8Pjjj1NZWUlJSQkHDx48Y/n7m2++maNHj7JlyxZmzZrF0KFDGTVqFMOHD2f27NmcPHmSW265hfz8/Ni2q6++OmPK6adyTmEQEL/6QyNw9ZnamNlJSYeBS4D/iW8k6W7gboDBgwd3KZi8AVdR/evR/Hv1Iq4oaLvsnnMu+5yaS2xtw4YNbbYNGjSImpoaJLFixQqKi4sBKCgoYOfOnbF2kk5bVKeiooKKioo2z/fYY4/Rt29fmpqaKCkpYcSIEd1NJy1S2Sm0942/9UBbIm0ws8XAYoDi4uIuDdZ9bvxUGD+1K7s653LAa6+9Rnl5OWbGxRdfzLJly7r1fDfeeCPvv/8+x48f5+GHH2bAgAFJijS1UtkpNAKXx93PB/7rDG0aJZ0H9Af+N4UxOedcu6677rp2l9XsqkyaR4iXyjmFWuBKSUMknQ/cDqxt1WYtcGe4XQpssEyetnfOncb/nNOvu695yjoFMzsJlAPrgP8Efmxm9ZK+Iemm0GwpcImkBuA+oM1lq865zJSXl0dTU5N3DGlkZjQ1NZGXl9fl51CmvWHFxcW2ffv2ng7DOdeBEydO0NjYyLFjx3o6lJySl5dHfn4+ffr0OW27pNfMrLij/XPmP5qdc+nVp08fhgzxK/0yTe7UPnLOOdch7xScc87FeKfgnHMuJuMmmiUdAt7q4u6X0uq/pXOA55wbPOfc0J2crzCzyzpqlHGdQndI2p7I7Hs28Zxzg+ecG9KRsw8fOeeci/FOwTnnXEyudQqLezqAHuA55wbPOTekPOecmlNwzjl3drl2puCcc+4svFNwzjkXk5WdgqQJkt6Q1CCpTeVVSRdIWhke3yqpIP1RJlcCOd8naZekOkk/l3RFT8SZTB3lHNeuVJJJyvjLFxPJWdJt4b2ul/TDdMeYbAn8bg+WtFHSL8Lv9w09EWeySFom6V1JO8/wuCQtCK9HnaRRSQ3AzLLqB+gN7AE+DZwPvA4MbdXmr4BF4fbtwMqejjsNOY8FLgq378mFnEO7fsDLQA1Q3NNxp+F9vhL4BfDxcP+TPR13GnJeDNwTbg8F9vd03N3M+Y+AUcDOMzx+A/Ai0cqVo4GtyTx+Np4plAANZrbXzI4DK4BJrdpMAp4Lt1cB4yS1tzRopugwZzPbaGZHw90aopXwMlki7zPAPwHfBLKhfnMiOf8l8JSZvQdgZu+mOcZkSyRnAz4Wbven7QqPGcXMXubsK1BOAr5vkRrgYkkDk3X8bOwUBgG/jrvfGLa128aixYAOA5ekJbrUSCTneHcRfdPIZB3mLOlzwOVm9tN0BpZCibzPnwE+I+lVSTWSJqQtutRIJOevA1MlNQI/A/46PaH1mM7+vXdKNq6n0N43/tbX3SbSJpMknI+kqUAx8IWURpR6Z81ZUi/gO8CMdAWUBom8z+cRDSH9MdHZ4BZJw83s/RTHliqJ5DwFeNbMvi3pGmB5yLkl9eH1iJR+fmXjmUIjcHnc/Xzank7G2kg6j+iU82yna+e6RHJG0heBh4CbzOyjNMWWKh3l3A8YDmyStJ9o7HVthk82J/q7/a9mdsLM9gFvEHUSmSqRnO8CfgxgZtVAHlHhuGyV0N97V2Vjp1ALXClpiKTziSaS17Zqsxa4M9wuBTZYmMHJUB3mHIZSvkfUIWT6ODN0kLOZHTazS82swMwKiOZRbjKzTF7LNZHf7ReILipA0qVEw0l70xplciWS8wFgHICk3yfqFA6lNcr0WgtMD1chjQYOm9nBZD151g0fmdlJSeXAOqIrF5aZWb2kbwDbzWwtsJToFLOB6Azh9p6LuPsSzPlbQF/g+TCnfsDMbuqxoLspwZyzSoI5rwPGS9oFNAN/a2ZNPRd19ySY898Az0j6GtEwyoxM/pIn6UdEw3+XhnmSfwD6AJjZIqJ5kxuABuAo8BdJPX4Gv3bOOeeSLBuHj5xzznWRdwrOOedivFNwzjkX452Cc865GO8UnHPOxXin4M45kpol/TLup+AsbQvOVE2yk8fcFCpxvh5KRFzVhef4iqTp4fYMSZ+Ke2yJpKFJjrNW0sgE9vmqpIu6e2yXG7xTcOeiD81sZNzP/jQdt8zMPktULPFbnd3ZzBaZ2ffD3RnAp+Iem2Vmu5IS5W/jXEhicX4V8E7BJcQ7BZcRwhnBFkn/EX6ubafNMEnbwtlFnaQrw/apcdu/J6l3B4d7GSgM+44Ldfp3hDr3F4Tt8/Xb9SkeC9u+LmmepFKi+lI/CMe8MHzDL5Z0j6RvxsU8Q9KTXYyzmrhCaJKelrRd0ToK/xi23UvUOW2UtDFsGy+pOryOz0vq28FxXA7xTsGdiy6MGzr6Sdj2LvCnZjYKmAwsaGe/rwBPmNlIog/lxlD2YDLw+bC9GSjr4Ph/BuyQlAc8C0w2sxFEFQDukfQJ4M+BYWZWBPxz/M5mtgrYTvSNfqSZfRj38Crgy3H3JwMruxjnBKKyFqc8ZGbFQBHwBUlFZraAqC7OWDMbG0pf/D3wxfBabgfu6+A4LodkXZkLlxU+DB+M8foA3w1j6M1ENX1aqwYekpQPrDGz3ZLGAX8A1IbyHhcSdTDt+YGkD4H9ROWXrwL2mdmb4fHngDnAd4nWZ1gi6d+AhEtzm9khSXtDzZrd4RivhuftTJy/Q1T2IX7Vrdsk3U30dz2QaMGZulb7jg7bXw3HOZ/odXMO8E7BZY6vAe8AnyU6w22zaI6Z/VDSVuBLwDpJs4jKDD9nZn+XwDHK4gvmSWp3jY1Qj6eEqAjb7UA58CedyGUlcBvwK+AnZmaKPqETjpNoBbL5wFPAlyUNAeYBf2hm70l6lqgwXGsC1pvZlE7E63KIDx+5TNEfOBhq5E8j+pZ8GkmfBvaGIZO1RMMoPwdKJX0ytPmEEl+f+ldAgaTCcH8asDmMwfc3s58RTeK2dwXQ/xGV727PGuBmonUAVoZtnYrTzE4QDQONDkNPHwOOAIcl/S4w8Qyx1ACfP5WTpIsktXfW5XKUdwouUywE7pRUQzR0dKSdNpOBnZJ+Cfwe0ZKFu4g+PF+SVAesJxpa6ZCZHSOqQPm8pB1AC7CI6AP2p+H5NhOdxbT2LLDo1ERzq+d9D9gFXGFm28K2TscZ5iq+Dcwzs9eJ1mauB5YRDUmdshh4UdJGMztEdGXUj8JxaoheK+cAr5LqnHMujp8pOOeci/FOwTnnXIx3Cs4552K8U3DOORfjnYJzzrkY7xScc87FeKfgnHMu5v8BMf3BD+Ne/WkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_nb_pred_prob = nb_classifier.predict_proba(test)[:,1]\n",
    "y_lr_pred_prob = lr.predict_proba(test)[:,1]\n",
    "\n",
    "nb_fpr, nb_tpr, nb_thresholds = metrics.roc_curve(test_df.label, y_nb_pred_prob)\n",
    "lr_fpr, lr_tpr, lr_thresholds = metrics.roc_curve(test_df.label, y_lr_pred_prob)\n",
    "\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(nb_fpr, nb_tpr, label='NaiveBayes')\n",
    "plt.plot(lr_fpr, lr_tpr, label='LogReg')\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('ROC Curve')\n",
    "plt.legend(loc='lower right') \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export\n",
    "\n",
    "The original method is clearly superior to logistic regression. \n",
    "In the last step, I export the data (word counts and NaiveBayes weights) into a JSON format and the model itself into a Joblib format in order to use it externally, e.g. on a <a href=\"http://marekcichy.alwaysdata.net\">Web page</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_df = pd.DataFrame([vectorizer.inverse_transform(nb_classifier.coef_)[0], \n",
    "                        nb_classifier.feature_count_[0], nb_classifier.feature_count_[1], \n",
    "                        nb_classifier.feature_log_prob_[0], nb_classifier.feature_log_prob_[1]])\n",
    "\n",
    "json_df = json_df.T\n",
    "\n",
    "json_df.columns = ['word', 'pt_count', 'br_count', 'pt_log_prob', 'br_log_prob']\n",
    "json_df = json_df.set_index('word')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pt_count         14460\n",
       "br_count         29596\n",
       "pt_log_prob   -5.16163\n",
       "br_log_prob   -4.63149\n",
       "Name: eu, dtype: object"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_df.loc['eu']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_dict = json_df.to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data.json', 'w', encoding='utf-8') as f:\n",
    "    json.dump(json_dict, f, ensure_ascii=False, indent=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ptclassifier.joblib']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline = Pipeline([('vectorizer',vectorizer),('classifier',nb_classifier)])\n",
    "\n",
    "dump(pipeline, 'ptclassifier.joblib', protocol = 2) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}