Materials/Session_5/EDGAR_walkthrough.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KSLo3otRtfNt"
   },
   "source": [
    "# Real world example: Full EDGAR walkthrough"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HrXei8w_tfNu"
   },
   "source": [
    "**Author:** Ties de Kok ([Personal Website](http://www.tiesdekok.com))  \n",
    "**Last updated:** 7 July 2018  \n",
    "**Python version:** Python 3.6  \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6JCMdd-dtfNv"
   },
   "source": [
    "## *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tRrfNQNntfNw"
   },
   "source": [
    "A usual project that involves EDGAR data consists of 5 steps:\n",
    "\n",
    "1. Determine URLs   \n",
    "2. Download   \n",
    "3. Prepare   \n",
    "4. Extract relevant sections    \n",
    "5. Perform textual analysis to generate metrics    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6JCMdd-dtfNv"
   },
   "source": [
    "## *Demonstration*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal for today is to perform a \"real\" task where we combine everything that we learned in the previous 4 sessions.  \n",
    "\n",
    "### Task: \n",
    "\n",
    "> Generate some NLP metrics for the MD&A sections of 10-K filings by TESLA.   \n",
    "\n",
    "Identifiers for Tesla:  \n",
    "* **CIK:** `0001318605`   \n",
    "* **GVKEY:** `184996`     \n",
    "\n",
    "Link to EDGAR page:  \n",
    "https://www.sec.gov/cgi-bin/browse-edgar?CIK=0001318605&action=getcompany\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3PsiOafatfNy"
   },
   "source": [
    "### *Relevant notebooks*\n",
    "\n",
    "1. [`0_python_basics.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)\n",
    "1. [`1_opening_files.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)\n",
    "1. [`2_handling data.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)\n",
    "1. [`3_web_scraping.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)\n",
    "1. [`4_NLP_notebook.ipynb`](https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/0_python_basics.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sFJZfP8RtfN3"
   },
   "source": [
    "## Step 0: preamble <br> ------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AxrZ8EKDtfNy"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "UMX6iPhbtfNz"
   },
   "outputs": [],
   "source": [
    "## Standard\n",
    "import os, re, sys, time\n",
    "\n",
    "## Data handling\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "## \"Web scraping\" and HTML\n",
    "import requests\n",
    "import lxml.html\n",
    "\n",
    "## Convenience functions\n",
    "from os.path import join\n",
    "from tqdm import tqdm_notebook as tqdm\n",
    "\n",
    "## NLP (Spacy)\n",
    "import en_core_web_sm\n",
    "nlp = en_core_web_sm.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Packages not installed in `uw-python-camp` environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import unidecode\n",
    "import wrds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install these packages by opening up the Anaconda command line (terminal) and running:   \n",
    "```\n",
    "activate uw-python-camp\n",
    "pip install unidecode\n",
    "pip install wrds\n",
    "```\n",
    "\n",
    "**Note:** you will need to use `source active uw-python-camp` if on Mac OS or Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sFJZfP8RtfN3"
   },
   "source": [
    "## Step 1: get URL to EDGAR filings <br> -----------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KDRklqyVtfN4"
   },
   "source": [
    "Before we can download our EDGAR filings we first need to:\n",
    "\n",
    "1. Identify the filings that we are interested in  \n",
    "2. Collect the URLs that we can use to download these filings  \n",
    "\n",
    "We can do this in three ways:  \n",
    "\n",
    "1. Use WRDS SEC Analytics Suite   \n",
    "2. Download index files  \n",
    "3. Scrape EDGAR directly   \n",
    "\n",
    "**We will use WRDS SEC Analyics Suite (because we can)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wyGwRsE7tfN6"
   },
   "source": [
    "### 1) Get all 10-K filings using the WRDS SEC Analytics Suite   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Approach 1: use the WRDS website   \n",
    "> WRDS SEC Analytics Suite --> WRDS SEC Filings Index Data   \n",
    "\n",
    "(Data already downloaded in `data/tesla_10-k_data.dta`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cik</th>\n",
       "      <th>fdate</th>\n",
       "      <th>form</th>\n",
       "      <th>fname</th>\n",
       "      <th>iname</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2012-02-27</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2013-03-07</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2014-02-26</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2015-02-26</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2016-02-24</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2017-03-01</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0001318605</td>\n",
       "      <td>2018-02-23</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956-index.htm</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          cik      fdate  form                                        fname  \\\n",
       "0  0001318605 2011-03-03  10-K  edgar/data/1318605/0001193125-11-054847.txt   \n",
       "1  0001318605 2012-02-27  10-K  edgar/data/1318605/0001193125-12-081990.txt   \n",
       "2  0001318605 2013-03-07  10-K  edgar/data/1318605/0001193125-13-096241.txt   \n",
       "3  0001318605 2014-02-26  10-K  edgar/data/1318605/0001193125-14-069681.txt   \n",
       "4  0001318605 2015-02-26  10-K  edgar/data/1318605/0001564590-15-001031.txt   \n",
       "5  0001318605 2016-02-24  10-K  edgar/data/1318605/0001564590-16-013195.txt   \n",
       "6  0001318605 2017-03-01  10-K  edgar/data/1318605/0001564590-17-003118.txt   \n",
       "7  0001318605 2018-02-23  10-K  edgar/data/1318605/0001564590-18-002956.txt   \n",
       "\n",
       "                                               iname  \n",
       "0  edgar/data/1318605/0001193125-11-054847-index.htm  \n",
       "1  edgar/data/1318605/0001193125-12-081990-index.htm  \n",
       "2  edgar/data/1318605/0001193125-13-096241-index.htm  \n",
       "3  edgar/data/1318605/0001193125-14-069681-index.htm  \n",
       "4  edgar/data/1318605/0001564590-15-001031-index.htm  \n",
       "5  edgar/data/1318605/0001564590-16-013195-index.htm  \n",
       "6  edgar/data/1318605/0001564590-17-003118-index.htm  \n",
       "7  edgar/data/1318605/0001564590-18-002956-index.htm  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tesla_10k_df = pd.read_stata(join('data', 'tesla_10-k_data.dta'))\n",
    "tesla_10k_df[['cik', 'fdate', 'form', 'fname', 'iname']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Approach 2: use the `WRDS` Python package   \n",
    "For an introduction to this approach see my ARC blog post:   \n",
    "https://arc.eaa-online.org/blog/retrieving-data-wrds-directly-using-python-r-and-stata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your WRDS username [kokti]:tcjkok\n",
      "Enter your password:········\n",
      "WRDS recommends setting up a .pgpass file.\n",
      "You can find more info here:\n",
      "https://www.postgresql.org/docs/9.5/static/libpq-pgpass.html.\n",
      "Loading library list...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "db = wrds.Connection()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cik_tesla = '0001318605'\n",
    "\n",
    "sql_query = \"\"\"\n",
    "SELECT gvkey,\n",
    "       cik,\n",
    "       fdate,\n",
    "       form,\n",
    "       fname,\n",
    "       iname\n",
    "FROM wrdssec.forms \n",
    "WHERE form = '10-K' \n",
    "AND fdate >= '01-01-2009'\n",
    "AND cik = '{}'\n",
    "\"\"\".format(cik_tesla)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "SELECT gvkey,\n",
      "       cik,\n",
      "       fdate,\n",
      "       form,\n",
      "       fname,\n",
      "       iname\n",
      "FROM wrdssec.forms \n",
      "WHERE form = '10-K' \n",
      "AND fdate >= '01-01-2009'\n",
      "AND cik = '0001318605'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(sql_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gvkey</th>\n",
       "      <th>cik</th>\n",
       "      <th>fdate</th>\n",
       "      <th>form</th>\n",
       "      <th>fname</th>\n",
       "      <th>iname</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2012-02-27</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2013-03-07</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2014-02-26</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2015-02-26</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2016-02-24</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2017-03-01</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118-index.htm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2018-02-23</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956-index.htm</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    gvkey         cik       fdate  form  \\\n",
       "0  184996  0001318605  2011-03-03  10-K   \n",
       "1  184996  0001318605  2012-02-27  10-K   \n",
       "2  184996  0001318605  2013-03-07  10-K   \n",
       "3  184996  0001318605  2014-02-26  10-K   \n",
       "4  184996  0001318605  2015-02-26  10-K   \n",
       "5  184996  0001318605  2016-02-24  10-K   \n",
       "6  184996  0001318605  2017-03-01  10-K   \n",
       "7  184996  0001318605  2018-02-23  10-K   \n",
       "\n",
       "                                         fname  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt   \n",
       "1  edgar/data/1318605/0001193125-12-081990.txt   \n",
       "2  edgar/data/1318605/0001193125-13-096241.txt   \n",
       "3  edgar/data/1318605/0001193125-14-069681.txt   \n",
       "4  edgar/data/1318605/0001564590-15-001031.txt   \n",
       "5  edgar/data/1318605/0001564590-16-013195.txt   \n",
       "6  edgar/data/1318605/0001564590-17-003118.txt   \n",
       "7  edgar/data/1318605/0001564590-18-002956.txt   \n",
       "\n",
       "                                               iname  \n",
       "0  edgar/data/1318605/0001193125-11-054847-index.htm  \n",
       "1  edgar/data/1318605/0001193125-12-081990-index.htm  \n",
       "2  edgar/data/1318605/0001193125-13-096241-index.htm  \n",
       "3  edgar/data/1318605/0001193125-14-069681-index.htm  \n",
       "4  edgar/data/1318605/0001564590-15-001031-index.htm  \n",
       "5  edgar/data/1318605/0001564590-16-013195-index.htm  \n",
       "6  edgar/data/1318605/0001564590-17-003118-index.htm  \n",
       "7  edgar/data/1318605/0001564590-18-002956-index.htm  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tesla_10k_df = db.raw_sql(sql_query)\n",
    "tesla_10k_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HqOcJWiLtfOe"
   },
   "source": [
    "## Step 2: download EDGAR filings <br> ----------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zi8vrVA6tfOe"
   },
   "source": [
    "Each EDGAR filing consists of several documents. I recommend to download the aggregate TXT file called: `Complete submission text file`   \n",
    "\n",
    "See for example: https://www.sec.gov/Archives/edgar/data/1318605/0001193125-11-054847-index.htm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2a) Understand EDGAR files and urls   \n",
    "You need to add \"https://www.sec.gov/Archives/\" to create the full URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "edgar_url = \"https://www.sec.gov/Archives/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://www.sec.gov/Archives/edgar/data/1318605/0001193125-11-054847-index.htm\n",
      "https://www.sec.gov/Archives/edgar/data/1318605/0001193125-11-054847.txt\n"
     ]
    }
   ],
   "source": [
    "print(edgar_url + tesla_10k_df.iloc[0]['iname'])\n",
    "print(edgar_url + tesla_10k_df.iloc[0]['fname'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "tesla_10k_df['download_url'] = tesla_10k_df.fname.apply(lambda file_url: edgar_url + file_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2b) Create basic downloader that can deal with failure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://www.sec.gov/Archives/edgar/data/1318605/0001193125-11-054847.txt'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_url = tesla_10k_df.iloc[0]['download_url']\n",
    "test_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_file(url, max_tries=4, sleep_time = 1):\n",
    "    failed_attempts = 0\n",
    "    while True: \n",
    "        res = requests.get(url)\n",
    "        \n",
    "        if res.status_code == 200:\n",
    "            return True, res.text\n",
    "        else:\n",
    "            if failed_attempts < max_tries:\n",
    "                failed_attempts += 1\n",
    "                time.sleep(sleep_time)\n",
    "            else:\n",
    "                return False, 'Could not download'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "test_res = download_file(test_url)\n",
    "print(test_res[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2c) Download all the fillings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think carefully about the data structure! I usually recommend to use dictionaries for things that you want to keep associated with an identifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0427f8e3acbf440eb58ea8603ad7c9a1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tesla_10k_dict = {}\n",
    "status_dict = {}\n",
    "for index, row in tqdm(tesla_10k_df.iterrows(), total=len(tesla_10k_df.index)):\n",
    "    download_res = download_file(row['download_url'])\n",
    "    \n",
    "    status_dict[index] = download_res[0]\n",
    "    \n",
    "    if download_res[0]:\n",
    "        tesla_10k_dict[row['fname']] = download_res[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "tesla_10k_df['download_status'] = pd.Series(status_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gvkey</th>\n",
       "      <th>cik</th>\n",
       "      <th>fdate</th>\n",
       "      <th>form</th>\n",
       "      <th>fname</th>\n",
       "      <th>iname</th>\n",
       "      <th>download_url</th>\n",
       "      <th>download_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847-index.htm</td>\n",
       "      <td>https://www.sec.gov/Archives/edgar/data/131860...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>184996</td>\n",
       "      <td>0001318605</td>\n",
       "      <td>2012-02-27</td>\n",
       "      <td>10-K</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990-index.htm</td>\n",
       "      <td>https://www.sec.gov/Archives/edgar/data/131860...</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    gvkey         cik       fdate  form  \\\n",
       "0  184996  0001318605  2011-03-03  10-K   \n",
       "1  184996  0001318605  2012-02-27  10-K   \n",
       "\n",
       "                                         fname  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt   \n",
       "1  edgar/data/1318605/0001193125-12-081990.txt   \n",
       "\n",
       "                                               iname  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847-index.htm   \n",
       "1  edgar/data/1318605/0001193125-12-081990-index.htm   \n",
       "\n",
       "                                        download_url  download_status  \n",
       "0  https://www.sec.gov/Archives/edgar/data/131860...             True  \n",
       "1  https://www.sec.gov/Archives/edgar/data/131860...             True  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tesla_10k_df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SEC-DOCUMENT>0001193125-11-054847.txt : 20110303\n",
      "<SEC-HEADER>0001193125-11-054847.hdr.sgml : 20110303\n",
      "<ACCEPTANCE-DATETIME>20110303144736\n",
      "ACCESSION NUMBER:\t\t0001193125-11-054847\n",
      "CONFORMED SUBMISSION TYPE:\t10-K\n",
      "PUBLIC DOCUMENT COUNT:\t\t8\n",
      "CONFORMED PERIOD OF REPORT:\t20101231\n",
      "FILED AS OF DATE:\t\t20110303\n",
      "DATE AS OF CHANGE:\t\t20110303\n",
      "\n",
      "FILER:\n",
      "\n",
      "\tCOMPANY DATA:\t\n",
      "\t\tCOMPANY CONFORMED NAME:\t\t\tTESLA MOTORS INC\n",
      "\t\tCENTRAL INDEX KEY:\t\t\t0001318605\n",
      "\t\tSTANDARD INDUSTRIAL CLASSIFICATION:\tMOTOR VEHICLES & PASSENGER CAR BODIES [3711]\n",
      "\t\tIRS NUMBER:\t\t\t\t912197729\n",
      "\t\tSTATE OF INCORPORATION:\t\t\tDE\n",
      "\n",
      "\tFILING VALUES:\n",
      "\t\tFORM TYPE:\t\t10-K\n",
      "\t\tSEC ACT:\t\t1934 Act\n",
      "\t\tSEC FILE NUMBER:\t001-34756\n",
      "\t\tFILM NUMBER:\t\t11659731\n",
      "\n",
      "\tBUSINESS ADDRESS:\t\n",
      "\t\tSTREET 1:\t\t3500 DEER CREEK RD\n",
      "\t\tCITY:\t\t\tPALO ALTO\n",
      "\t\tSTATE:\t\t\tCA\n",
      "\t\tZIP:\t\t\t94070\n",
      "\t\tBUSINESS PHONE:\t\t650-681-5000\n",
      "\n",
      "\tMAIL ADDRESS:\t\n",
      "\t\tSTREET 1:\t\t3500 DEER CREEK RD\n",
      "\t\tCITY:\t\t\tPALO ALTO\n",
      "\t\tSTATE:\t\t\tCA\n",
      "\t\tZIP:\t\t\t94070\n",
      "</SEC-HEADER>\n",
      "<DOCUMENT>\n",
      "<TYPE>10-K\n",
      "<SEQUENCE>1\n",
      "<FILENAME>d10k.htm\n",
      "<DESCRIPTION>F\n"
     ]
    }
   ],
   "source": [
    "print(tesla_10k_dict['edgar/data/1318605/0001193125-11-054847.txt'][:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xDR9oNdatfOs"
   },
   "source": [
    "## Step 3: prepare EDGAR filing <br> -----------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9LxbIdo1tfOt"
   },
   "source": [
    "Before we can start working with the EDGAR filings we need to process the file we retrieved, in this case: \n",
    "1. Split the documents up  \n",
    "2. Keep the 10-K document\n",
    "3. Parse raw text into HTML + extract text   \n",
    "4. Clean the text up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3a) Create a test document save it for manual inspection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_doc = tesla_10k_dict['edgar/data/1318605/0001193125-11-054847.txt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('test_doc.txt', 'w') as f:\n",
    "    f.write(test_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3b) Create logic to split the documents up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do we do this? Regular Expressions! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define regular expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "gFdJyB4htfOu"
   },
   "outputs": [],
   "source": [
    "pattern_dict = {\n",
    "    'documents' : re.compile(r\"<document>(.*?)</document>\", re.IGNORECASE | re.DOTALL),\n",
    "    'metadata' : {\n",
    "        'type' : re.compile(r\"<type>(.*?)\\n\", re.IGNORECASE | re.DOTALL),\n",
    "        'sequence' : re.compile(r\"<sequence>(.*?)\\n\", re.IGNORECASE | re.DOTALL),\n",
    "        'filename' : re.compile(r\"<filename>(.*?)\\n\", re.IGNORECASE | re.DOTALL),\n",
    "        'description' : re.compile(r\"<description>(.*?)\\n\", re.IGNORECASE | re.DOTALL)\n",
    "    },\n",
    "    'text' : re.compile(r\"<text>(.*?)</text>\", re.IGNORECASE | re.DOTALL)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['d10k.htm',\n",
       " 'dex1047.htm',\n",
       " 'dex211.htm',\n",
       " 'dex231.htm',\n",
       " 'dex311.htm',\n",
       " 'dex312.htm',\n",
       " 'dex321.htm',\n",
       " 'g151489g81s77.jpg']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern_dict['metadata']['filename'].findall(test_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### *Alternative: use `LXML` to parse the SGML structure*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "sgml_parsed = lxml.html.fromstring(test_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "d10k.htm\n",
      "dex1047.htm\n",
      "dex211.htm\n",
      "dex231.htm\n",
      "dex311.htm\n",
      "dex312.htm\n",
      "dex321.htm\n",
      "g151489g81s77.jpg\n"
     ]
    }
   ],
   "source": [
    "for document in sgml_parsed.cssselect('document'):\n",
    "    print(document.cssselect('filename')[0].text.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split up into documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    }
   ],
   "source": [
    "docs_split = pattern_dict['documents'].findall(test_doc)\n",
    "print(len(docs_split))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create function to extract information from one document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_metadata(doc, pattern_dict=pattern_dict):\n",
    "    data_dict = {}\n",
    "    \n",
    "    data_dict['metadata'] = {}\n",
    "    for key, pattern in pattern_dict['metadata'].items():\n",
    "        matches = pattern.findall(doc)\n",
    "        if matches:\n",
    "            data_dict['metadata'][key] = matches[0]\n",
    "        else:\n",
    "            data_dict['metadata'][key] = np.nan\n",
    "            \n",
    "    text_match = pattern_dict['text'].findall(doc)\n",
    "    if text_match:\n",
    "        data_dict['text'] = text_match[0]\n",
    "    else:\n",
    "        data_dict['text'] = np.nan\n",
    "        \n",
    "    return data_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = extract_metadata(docs_split[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': 'FOR THE FISCAL YEAR ENDED DECEMBER 31, 2010',\n",
       " 'filename': 'd10k.htm',\n",
       " 'sequence': '1',\n",
       " 'type': '10-K'}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test['metadata']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Apply extraction function to all filings to retrieve the 10-K part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "47956f686113415a8f3e5c84401ede79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "data_10k_dict = {}\n",
    "for filename, data in tqdm(tesla_10k_dict.items()):\n",
    "    docs_split = pattern_dict['documents'].findall(data)\n",
    "        \n",
    "    for doc in docs_split:\n",
    "        doc_data = extract_metadata(doc)\n",
    "        \n",
    "        ## Only keep 10-K document\n",
    "        if doc_data['metadata']['type'] == '10-K':\n",
    "            data_10k_dict[filename] = doc_data['text']\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_10k_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "<HTML><HEAD>\n",
      "<TITLE>For the fiscal year ended December 31, 2010</TITLE>\n",
      "</HEAD>\n",
      " <BODY BGCOLOR=\"WHITE\">\n",
      "<h5 align=\"left\"><a href=\"#toc\">Table of Contents</a></h5>\n",
      "\n",
      " <P STYLE=\"line-height:0px;margin-top:0px;margin-bottom:0px;border-bottom:0.5pt solid #000000\">&nbsp;</P>\n",
      "<P STYLE=\"line-height:3px;margin-top:0px;margin-bottom:2px;border-bottom:0.5pt solid #000000\">&nbsp;</P> <P STYLE=\"margin-top:4px;margin-bottom:0px\" ALIGN=\"center\"><FONT STYLE=\"font-family:Times New Roman\" SIZE=\"5\"><B>UNITED STATES </B></FONT></P>\n",
      "<P STYLE=\"margin-top:0px;margin-bottom:0px\" ALIGN=\"center\"><FONT STYLE=\"font-family:Times New Roman\" SIZE=\"5\"><B>SECURITIES AND EXCHANGE COMMISSION </B></FONT></P> <P STYLE=\"margin-top:0px;margin-bottom:0px\" ALIGN=\"center\"><FONT\n",
      "STYLE=\"font-family:Times New Roman\" SIZE=\"2\"><B>Washington, D.C. 20549 </B></FONT></P> <P STYLE=\"font-size:4px;margin-top:0px;margin-bottom:0px\">&nbsp;</P><center>\n",
      "<P STYLE=\"line-height:6px;margin-top:0px;margin-bottom:2px;border-bottom:1pt solid #0000\n"
     ]
    }
   ],
   "source": [
    "print(data_10k_dict['edgar/data/1318605/0001193125-11-054847.txt'][:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3c) Parse raw 10-K data as HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are only interested in the text we can use `.text_content()` on the html object to extract only the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'For the fiscal year ended December 31, 2010'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = lxml.html.fromstring(data_10k_dict['edgar/data/1318605/0001193125-11-054847.txt'])\n",
    "\n",
    "test.cssselect('TITLE')[0].text_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "86bd856ace28453d8ae465b5680c1937",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "html_10k_dict = {}\n",
    "text_10k_dict = {}\n",
    "for filename, raw_text in tqdm(data_10k_dict.items()):\n",
    "    html = lxml.html.fromstring(raw_text)\n",
    "    html_10k_dict[filename] = html\n",
    "    text_10k_dict[filename] = html.text_content()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' gallon city and 25 miles per gallon highway. Furthermore, we expect our electric vehicles will have lower relative\\n\\n \\xa0 5 \\n\\n\\n\\n\\nTable of Contents\\n\\n\\n\\n\\n\\xa0\\n \\nmaintenance costs than hybrid, plug-in hybrid, or internal combustion engine vehicles due to fewer moving parts and the absence of certain components, including oil, oil filters, spark plugs and\\nengine valves. Additionally, government incentives that are currently available can reduce the cost of ownership even further.  \\xa0\\n\\n\\n\\xa0\\n\\x95\\n\\xa0\\n High-Performance Without Compromised Design or Functionality. We believe we have been able to successfully overcome the design and performance\\ntradeoff issues that encumbered most early electric vehicle designs. We believe the Tesla Roadster delivers an unparalleled driving experience with instantaneous and sustained acceleration through an extended range of speed. In addition, our planned\\nModel\\xa0S is being designed to seat five adults, provide best in class storage in the trunk and hood while offering design'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_10k_dict['edgar/data/1318605/0001193125-11-054847.txt'][20000:21000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3d) Clean up the text   \n",
    "I usually do at least two things:  \n",
    "\n",
    "1. I try to fix encoding errors to avoid getting too many weird characters  \n",
    "2. Replace newline characters with a whitespace\n",
    "3. Remove as many unnescessary whitespaces as possible   \n",
    "4. Remove \"table of contents\" text and page number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3002984f9d1f4c6d94234fae9f8743a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "cleantext_10k_dict = {}\n",
    "for filename, text in tqdm(text_10k_dict.items()):\n",
    "    ## Fix encoding\n",
    "    clean_text = unidecode.unidecode(text)\n",
    "    \n",
    "    ## Replace newline characters with space\n",
    "    clean_text = re.sub('\\s', ' ', clean_text)\n",
    "    \n",
    "    ## Remove duplicate whitespaces\n",
    "    clean_text = ' '.join([word for word in clean_text.split(' ') if word])\n",
    "    \n",
    "    ## Replace \"Page number + Table of Contents footer\"\n",
    "    clean_text = re.sub(' \\d+ Table of Contents ', ' ', clean_text)\n",
    "    \n",
    "    cleantext_10k_dict[filename] = clean_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'nformation Administration of the U.S. Department of Energy, the cost per mile to fuel the Tesla Roadster is approximately 70% less than the cost to fuel the 2009 Porsche 911 Carrera, which has an EPA mileage rating of 18 miles per gallon city and 25 miles per gallon highway. Furthermore, we expect our electric vehicles will have lower relative maintenance costs than hybrid, plug-in hybrid, or internal combustion engine vehicles due to fewer moving parts and the absence of certain components, including oil, oil filters, spark plugs and engine valves. Additionally, government incentives that are currently available can reduce the cost of ownership even further. High-Performance Without Compromised Design or Functionality. We believe we have been able to successfully overcome the design and performance tradeoff issues that encumbered most early electric vehicle designs. We believe the Tesla Roadster delivers an unparalleled driving experience with instantaneous and sustained acceleration through an extended range of speed. In addition, our planned Model S is being designed to seat five adults, provide best in class storage in the trunk and hood while offering design and performance comparable to, or better than, other premium sedans. Our Vehicles and Products We currently design, manufacture and sell the Tesla Roadster, our first production vehicle. We are designing our second vehicle, the Model S, and currently plan to begin production of the Model S in mid-2012. We intend to design the Model S with an adaptable platform architecture and common electric powertrain so that we can use the platform of the Model S to create future electric vehicles targeting additional segments of the passenger vehicle market. The Tesla Roadster Our first vehicle, the Tesla Roadster, is the first high-performance electric sports car and the only highway-capable electric vehicle available in the United States today. The two-seat, convertible Tesla Roadster has a combination of range, style'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cleantext_10k_dict['edgar/data/1318605/0001193125-11-054847.txt'][19000:21000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BB7q7YAstfPI"
   },
   "source": [
    "## Step 4: extract section <br> --------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4a) Create a function that can extract the MD&A part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y78bZpDJtfPI"
   },
   "source": [
    "The MD&A section starts with something along the lines of:\n",
    "\n",
    "> Item 7, Management's Discussion and Analysis of Financial Condition and Results of Operations\n",
    "\n",
    "And ends with the next section that starts with something along the lines of:\n",
    "\n",
    "> Item 7A, Quantitative and Qualitative Disclosures About Market Risk; \n",
    "\n",
    "So the trick is to identify where the MD&A sections starts and where it ends, once we know that we can extract the MD&A part.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "JlDTY1vFtfPJ"
   },
   "outputs": [],
   "source": [
    "def extract_MDA_from_10K(text, section_number_range = 10, keyword_range = 200, \n",
    "                         reference_range = 10, title_range = 100, min_length = 1000):\n",
    "    \"\"\"Extract the MD&A part out of a 10-K filing. \"\"\"\n",
    "    \n",
    "    ## Split the document based on the word \"item\"\n",
    "    parts = re.split('item|Item|ITEM', text)\n",
    "    \n",
    "    # ------------------------------------------\n",
    "    # Step 1: try to find where the MD&A begins\n",
    "    # ------------------------------------------\n",
    "\n",
    "    ## Loop over each part and check whether it starts with 7 that is not followed by an \"a\"\n",
    "    valid_start_parts = []\n",
    "    for i, text in enumerate(parts):\n",
    "        if '7' in text.strip()[:section_number_range] and not '7a' in text.strip().lower()[:section_number_range]:\n",
    "            valid_start_parts.append(i)\n",
    "    \n",
    "    # ----------------------------------------\n",
    "    # Step 2: try to find where the MD&A ends\n",
    "    # ----------------------------------------\n",
    "\n",
    "    results = []\n",
    "    reference = 0\n",
    "\n",
    "    ## Loop over all elements that start with 7 (excluding those starting with 7a)\n",
    "    for i in valid_start_parts:\n",
    "        mda_text = parts[i]\n",
    "        endPoint = i+1\n",
    "\n",
    "        isValid = 0\n",
    "        reference = 0\n",
    "\n",
    "        ## Loop over the elements that follow the current element and check whether it matches item 7a\n",
    "        while endPoint < len(parts):\n",
    "            end_part_text = parts[endPoint]\n",
    "            ## Check if potential candidate for ending part\n",
    "            if '7a' in end_part_text.strip().lower()[:section_number_range]:\n",
    "                \n",
    "                ## Check for keywords (quan and qual) that indicate start section 7a (and not just 7a mentioned in the text)\n",
    "                if end_part_text[:keyword_range].lower().find('quan') >= 0 and end_part_text[:keyword_range].lower().find('qual') >= 0:\n",
    "                    \n",
    "                    ## Final check to make sure it is not a reference, if not then consider valid and break loop\n",
    "                    if mda_text[-reference_range:].lower().find(' see ') <= 0 and mda_text[-reference_range:].lower().find(' in ') <= 0:\n",
    "                        isValid = 1\n",
    "                        break\n",
    "            \n",
    "            ## If not the end then add the part of the text but keep searching by going to the next one\n",
    "            mda_text = ' '.join((mda_text, parts[endPoint]))\n",
    "            endPoint += 1\n",
    "            \n",
    "        # ----------------------------------------\n",
    "        # Step 3: double check it is a valid text\n",
    "        # ----------------------------------------\n",
    "        \n",
    "        ## Short hits are either invalid or imply incorporated by reference\n",
    "        if (len(mda_text) < min_length):\n",
    "            if mda_text.lower().find('incorporated') and mda_text.lower().find('reference'):\n",
    "                reference = 1\n",
    "            else:\n",
    "                isValid = 0\n",
    "\n",
    "        ## Check whether it contains the title, otherwise consider it an invalid hit\n",
    "        if mda_text[:title_range].lower().find('discussion') < 0 and mda_text[:title_range].lower().find('analysis') < 0:\n",
    "            isValid = 0\n",
    "\n",
    "        ## Append valid hits that do not indicate incorporated by reference\n",
    "        if isValid:\n",
    "            if not reference:\n",
    "                results.append(mda_text)\n",
    "\n",
    "    ## Return the shortest possible valid mda_text:\n",
    "    if len(results) > 0:\n",
    "        return {'status' : 'Good', \n",
    "                'mda_text' : sorted(results, key = len)[0]}\n",
    "    else:\n",
    "        if reference:\n",
    "            return {'status' : 'Reference', \n",
    "                    'mda_text' : None}\n",
    "        else:\n",
    "            return {'status' : 'Not found', \n",
    "                    'mda_text' : None}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4b) Apply MD&A function on the text of each 10-K filing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7bde6889b20f4f25b8d1d09e3da19c62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "mda_text_dict = {}\n",
    "mda_status_list = []\n",
    "for filename, text in tqdm(cleantext_10k_dict.items()):\n",
    "    mda_result = extract_MDA_from_10K(text)\n",
    "    \n",
    "    mda_status_list.append([filename, mda_result['status']])\n",
    "    \n",
    "    if mda_result['status'] == 'Good':\n",
    "        mda_text_dict[filename] = mda_result['mda_text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>Good</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename status\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt   Good\n",
       "1  edgar/data/1318605/0001193125-12-081990.txt   Good\n",
       "2  edgar/data/1318605/0001193125-13-096241.txt   Good\n",
       "3  edgar/data/1318605/0001193125-14-069681.txt   Good\n",
       "4  edgar/data/1318605/0001564590-15-001031.txt   Good\n",
       "5  edgar/data/1318605/0001564590-16-013195.txt   Good\n",
       "6  edgar/data/1318605/0001564590-17-003118.txt   Good\n",
       "7  edgar/data/1318605/0001564590-18-002956.txt   Good"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(mda_status_list, columns=['filename', 'status'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' 7. MANAGEMENTS DISCUSSION AND ANALYSIS OF FINANCIAL CONDITION AND RESULTS OF OPERATIONS The following discussion and analysis should be read in conjunction with our consolidated financial statements and the related notes that appear elsewhere in this Annual Report on Form 10-K. These discussions contain forward-looking statements reflecting our current expectations that involve risks and uncertainties. These forward-looking statements include, but are not limited to, statements concerning our strategy, future operations, future financial position, future revenues, projected costs, expectations regarding demand and acceptance for our technologies, growth opportunities and trends in the market in which we operate, prospects and plans and objectives of management. The words anticipates, believes, estimates, expects, intends, may, plans, projects, will, would and similar expressions are intended to identify forward-looking statements, although not all forward-looking statements contain these identifying words. We may not actually achieve the plans, intentions or expectations disclosed in our forward-looking statements and you should not place undue reliance on our forward-looking statements. Actual results or events could differ materially from the plans, intentions and expectations disclosed in the forward-looking statements that we make. These forward-looking statements involve risks and uncertainties that could cause our actual results to differ materially from those in the forward-looking statements, including, without limitation, the risks set forth in Part I,   1A, Risk Factors in this Annual Report on Form 10-K and in our other filings with the Securities and Exchange Commission. We do not assume any obligation to update any forward-looking statements. Overview and 2010 Highlights We design, develop, manufacture and sell high-performance fully electric vehicles and advanced electric vehicle powertrain components. We own our sales and service network, and market and sell our vehicles directly to consumers via the phone and internet, in-person at our corporate events and through our network of Tesla stores. We were incorporated in Delaware in 2003, opened our first store in Los Angeles, California in May 2008, and introduced our first vehicle, the Tesla Roadster, in early 2008. In July 2009, we introduced a new Roadster model, the Tesla Roadster 2, and its higher performance option package Roadster Sport, as well as launched the Tesla Roadster in Europe. On July 1, 2010, we introduced the Roadster 2.5, with new styling and an upgraded interior. During the year ended December 31, 2010, we experienced solid performance from sales of the Tesla Roadster, especially considering that the automotive sales recognized in 2009 were driven primarily from the fulfillment of a significant reservations list that had accumulated prior to and through 2009. Our automotive sales revenues in 2010 were primarily driven by retail store expansion globally and higher'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_text_dict['edgar/data/1318605/0001193125-11-054847.txt'][:3000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'etermine with certainty the potential costs to remediate any pre-existing contamination. Based on managements best estimate, we estimated the fair value of the environmental liabilities that we assumed to be $5.3 million, which is not reflected in the table above as the timing of any potential payments cannot be reasonably determined at this time. As NUMMI continues with its decommissioning activities and as we continue with our planned construction and operating activities, it is reasonably possible that our estimate of environmental liabilities may change materially. We have reached an agreement with NUMMI under which, over a ten year period, we will pay the first $15.0 million of any costs of any governmentally-required remediation activities for contamination that existed prior to the completion of the facility and land purchase for any known or unknown environmental conditions, and NUMMI has agreed to pay the next $15.0 million for such remediation activities. Our agreement provides, in part, that NUMMI will pay up to the first $15.0 million on our behalf if such expenses are incurred in the first four years of our agreement, subject to our reimbursement of such costs on the fourth anniversary date of the closing. On the ten-year anniversary of the closing or whenever $30.0 million has been spent on the remediation activities, whichever comes first, NUMMIs liability to us with respect to remediation activities ceases, and we are responsible for any and all environmental conditions at the Fremont site. At that point in time, we have agreed to indemnify, defend, and hold harmless NUMMI from all liability and we have released NUMMI for any known or unknown claims except for NUMMIs obligations for representations and warranties under the agreement. As of December 31, 2010 and 2009, we held reservation payments of $30.8 million and $26.0 million from potential customers, respectively, which are not reflected in the table above. As of December 31, 2010, we held reservation payments for undelivered Tesla Roadsters in an aggregate amount of $2.5 million and reservation payments for Model S sedans in an aggregate amount of $28.3 million. As of December 31, 2009, we held reservation payments for undelivered Tesla Roadsters in an aggregate amount of $8.2 million and reservation payments for Model S sedans in an aggregate amount of $17.9 million. In order to convert the reservation payments into revenue, we will need to sell vehicles to these customers. All reservation payments for the Model S are fully refundable until such time that a customer enters into a purchase agreement. Off-Balance Sheet Arrangements During the periods presented, we did not have relationships with unconsolidated entities or financial partnerships, such as entities often referred to as structured finance or special purpose entities, which would have been established for the purpose of facilitating off-balance sheet arrangements or other contractually narrow or limited purposes. '"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_text_dict['edgar/data/1318605/0001193125-11-054847.txt'][-3000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5Ck-3kf7tfPM"
   },
   "source": [
    "## Step 5: calculate metrics <br> ------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5a) Use spacy to split text up into sentence and words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse each text using Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2eb323f3c8db4ffeb1979d0b6d4f6820",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "spacy_mda_dict = {}\n",
    "for filename, mda_text in tqdm(mda_text_dict.items()):\n",
    "    spacy_mda_dict[filename] = nlp(mda_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split up into sentences and tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad070eda6e434a8282ea9207cee81236",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "mda_sen_and_token_dict = {}\n",
    "for filename, spacy_obj in tqdm(spacy_mda_dict.items()):\n",
    "    sentences = []\n",
    "    for sentence in spacy_obj.sents:\n",
    "        sentences.append([token.text.lower() for token in sentence])\n",
    "    mda_sen_and_token_dict[filename] = sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the',\n",
       " 'following',\n",
       " 'discussion',\n",
       " 'and',\n",
       " 'analysis',\n",
       " 'should',\n",
       " 'be',\n",
       " 'read',\n",
       " 'in',\n",
       " 'conjunction',\n",
       " 'with',\n",
       " 'our',\n",
       " 'consolidated',\n",
       " 'financial',\n",
       " 'statements',\n",
       " 'and',\n",
       " 'the',\n",
       " 'related',\n",
       " 'notes',\n",
       " 'that',\n",
       " 'appear',\n",
       " 'elsewhere',\n",
       " 'in',\n",
       " 'this',\n",
       " 'annual',\n",
       " 'report',\n",
       " 'on',\n",
       " 'form',\n",
       " '10-k.']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_sen_and_token_dict['edgar/data/1318605/0001193125-11-054847.txt'][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b) Calculate tone based on Loughran and McDonald dictionary at the MD&A level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the sake of simplicity, assume this is the tone score we want to calculate: $\\frac{(Num\\ Positive\\ Words - Num\\ Negative\\ Words)}{Total\\ Number\\ of\\ Words}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "cqD5i0VbtfPM"
   },
   "outputs": [],
   "source": [
    "lm_df = pd.read_excel(os.path.join('data', 'LoughranMcDonald_MasterDictionary_2014.xlsx'))\n",
    "negative_words = [str(x).lower() for x in lm_df[lm_df.Negative != 0].Word.values]\n",
    "positive_words = [str(x).lower() for x in lm_df[lm_df.Positive != 0].Word.values]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bb507d7f1014637b3243deca96c2e39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tone_data = []\n",
    "for filename, sentences in tqdm(mda_sen_and_token_dict.items()):\n",
    "    pos_words, neg_words, num_words = 0, 0, 0\n",
    "    \n",
    "    for sentence in sentences:\n",
    "        num_words += len(sentence)\n",
    "        \n",
    "        for neg_word in negative_words:\n",
    "            neg_words += sentence.count(neg_word)\n",
    "            \n",
    "        for pos_word in positive_words:\n",
    "            pos_words += sentence.count(pos_word)\n",
    "            \n",
    "    tone_data.append([filename, pos_words, neg_words, num_words])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>pos_words</th>\n",
       "      <th>neg_words</th>\n",
       "      <th>num_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>78</td>\n",
       "      <td>128</td>\n",
       "      <td>19838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>57</td>\n",
       "      <td>107</td>\n",
       "      <td>19417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>71</td>\n",
       "      <td>114</td>\n",
       "      <td>17851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>74</td>\n",
       "      <td>94</td>\n",
       "      <td>16658</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>61</td>\n",
       "      <td>67</td>\n",
       "      <td>10962</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  pos_words  neg_words  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt         78        128   \n",
       "1  edgar/data/1318605/0001193125-12-081990.txt         57        107   \n",
       "2  edgar/data/1318605/0001193125-13-096241.txt         71        114   \n",
       "3  edgar/data/1318605/0001193125-14-069681.txt         74         94   \n",
       "4  edgar/data/1318605/0001564590-15-001031.txt         61         67   \n",
       "\n",
       "   num_words  \n",
       "0      19838  \n",
       "1      19417  \n",
       "2      17851  \n",
       "3      16658  \n",
       "4      10962  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_tone_df = pd.DataFrame(tone_data, columns = ['filename', 'pos_words', 'neg_words', 'num_words'])\n",
    "mda_tone_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>pos_words</th>\n",
       "      <th>neg_words</th>\n",
       "      <th>num_words</th>\n",
       "      <th>tone_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>78</td>\n",
       "      <td>128</td>\n",
       "      <td>19838</td>\n",
       "      <td>-0.002520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>57</td>\n",
       "      <td>107</td>\n",
       "      <td>19417</td>\n",
       "      <td>-0.002575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>71</td>\n",
       "      <td>114</td>\n",
       "      <td>17851</td>\n",
       "      <td>-0.002409</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>74</td>\n",
       "      <td>94</td>\n",
       "      <td>16658</td>\n",
       "      <td>-0.001201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>61</td>\n",
       "      <td>67</td>\n",
       "      <td>10962</td>\n",
       "      <td>-0.000547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>52</td>\n",
       "      <td>66</td>\n",
       "      <td>10589</td>\n",
       "      <td>-0.001322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>65</td>\n",
       "      <td>83</td>\n",
       "      <td>14361</td>\n",
       "      <td>-0.001253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>83</td>\n",
       "      <td>71</td>\n",
       "      <td>14310</td>\n",
       "      <td>0.000839</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  pos_words  neg_words  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt         78        128   \n",
       "1  edgar/data/1318605/0001193125-12-081990.txt         57        107   \n",
       "2  edgar/data/1318605/0001193125-13-096241.txt         71        114   \n",
       "3  edgar/data/1318605/0001193125-14-069681.txt         74         94   \n",
       "4  edgar/data/1318605/0001564590-15-001031.txt         61         67   \n",
       "5  edgar/data/1318605/0001564590-16-013195.txt         52         66   \n",
       "6  edgar/data/1318605/0001564590-17-003118.txt         65         83   \n",
       "7  edgar/data/1318605/0001564590-18-002956.txt         83         71   \n",
       "\n",
       "   num_words  tone_score  \n",
       "0      19838   -0.002520  \n",
       "1      19417   -0.002575  \n",
       "2      17851   -0.002409  \n",
       "3      16658   -0.001201  \n",
       "4      10962   -0.000547  \n",
       "5      10589   -0.001322  \n",
       "6      14361   -0.001253  \n",
       "7      14310    0.000839  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_tone_df['tone_score'] = (mda_tone_df['pos_words'] - mda_tone_df['neg_words']) / mda_tone_df['num_words']\n",
    "mda_tone_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5b) Calculate tone based on Loughran and McDonald dictionary at the sentence level"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate word counts at the sentence level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9e25ac22491d4374bf474468b872e8b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, max=8), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "tone_data_sen = []\n",
    "for filename, sentences in tqdm(mda_sen_and_token_dict.items()):\n",
    "    pos_words, neg_words, num_words = 0, 0, 0\n",
    "    \n",
    "    for i, sentence in enumerate(sentences):\n",
    "        pos_words, neg_words, num_words = 0, 0, len(sentence)\n",
    "        \n",
    "        for neg_word in negative_words:\n",
    "            neg_words += sentence.count(neg_word)\n",
    "            \n",
    "        for pos_word in positive_words:\n",
    "            pos_words += sentence.count(pos_word)\n",
    "            \n",
    "        tone_data_sen.append([filename, i, pos_words, neg_words, num_words])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert to Pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>sentence</th>\n",
       "      <th>pos_words</th>\n",
       "      <th>neg_words</th>\n",
       "      <th>num_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  sentence  pos_words  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt         0          0   \n",
       "1  edgar/data/1318605/0001193125-11-054847.txt         1          0   \n",
       "2  edgar/data/1318605/0001193125-11-054847.txt         2          0   \n",
       "3  edgar/data/1318605/0001193125-11-054847.txt         3          0   \n",
       "4  edgar/data/1318605/0001193125-11-054847.txt         4          1   \n",
       "\n",
       "   neg_words  num_words  \n",
       "0          0          3  \n",
       "1          0         11  \n",
       "2          0         29  \n",
       "3          0         17  \n",
       "4          0         60  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_sen_tone_df = pd.DataFrame(tone_data_sen, columns = ['filename', 'sentence', 'pos_words', 'neg_words', 'num_words'])\n",
    "mda_sen_tone_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate sentence level tone score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>sentence</th>\n",
       "      <th>pos_words</th>\n",
       "      <th>neg_words</th>\n",
       "      <th>num_words</th>\n",
       "      <th>tone_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>29</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0.016667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  sentence  pos_words  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt         0          0   \n",
       "1  edgar/data/1318605/0001193125-11-054847.txt         1          0   \n",
       "2  edgar/data/1318605/0001193125-11-054847.txt         2          0   \n",
       "3  edgar/data/1318605/0001193125-11-054847.txt         3          0   \n",
       "4  edgar/data/1318605/0001193125-11-054847.txt         4          1   \n",
       "\n",
       "   neg_words  num_words  tone_score  \n",
       "0          0          3    0.000000  \n",
       "1          0         11    0.000000  \n",
       "2          0         29    0.000000  \n",
       "3          0         17    0.000000  \n",
       "4          0         60    0.016667  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_sen_tone_df['tone_score'] = (mda_sen_tone_df['pos_words'] - mda_sen_tone_df['neg_words']) / mda_sen_tone_df['num_words']\n",
    "mda_sen_tone_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create indicator whether sentence is positive, negative, or neutral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify_sentence(tone_score):\n",
    "    if tone_score > 0:\n",
    "        return 'pos'\n",
    "    elif tone_score < 0:\n",
    "        return 'neg'\n",
    "    else:\n",
    "        return 'neut'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>sentence</th>\n",
       "      <th>pos_words</th>\n",
       "      <th>neg_words</th>\n",
       "      <th>num_words</th>\n",
       "      <th>tone_score</th>\n",
       "      <th>tone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3086</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>280</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2059</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>416</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>162</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>41</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1179</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2250</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>136</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>-0.100000</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3064</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>258</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1979</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>336</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>39</td>\n",
       "      <td>0.025641</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1337</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>197</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1553</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>413</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2181</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>67</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>neut</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         filename  sentence  pos_words  \\\n",
       "3086  edgar/data/1318605/0001564590-17-003118.txt       280          0   \n",
       "2059  edgar/data/1318605/0001193125-14-069681.txt       416          0   \n",
       "162   edgar/data/1318605/0001193125-11-054847.txt       162          0   \n",
       "1179  edgar/data/1318605/0001193125-13-096241.txt        39          0   \n",
       "2250  edgar/data/1318605/0001564590-15-001031.txt       136          0   \n",
       "3064  edgar/data/1318605/0001564590-17-003118.txt       258          0   \n",
       "1979  edgar/data/1318605/0001193125-14-069681.txt       336          1   \n",
       "1337  edgar/data/1318605/0001193125-13-096241.txt       197          4   \n",
       "1553  edgar/data/1318605/0001193125-13-096241.txt       413          0   \n",
       "2181  edgar/data/1318605/0001564590-15-001031.txt        67          0   \n",
       "\n",
       "      neg_words  num_words  tone_score  tone  \n",
       "3086          0         24    0.000000  neut  \n",
       "2059          0         13    0.000000  neut  \n",
       "162           0         41    0.000000  neut  \n",
       "1179          0         24    0.000000  neut  \n",
       "2250          2         20   -0.100000   neg  \n",
       "3064          0         11    0.000000  neut  \n",
       "1979          0         39    0.025641   pos  \n",
       "1337          0        100    0.040000   pos  \n",
       "1553          0          2    0.000000  neut  \n",
       "2181          0         32    0.000000  neut  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_sen_tone_df['tone'] = mda_sen_tone_df.tone_score.apply(classify_sentence)\n",
    "mda_sen_tone_df.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate per MD&A how many positive, negative, and neutral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>tone</th>\n",
       "      <th>filename</th>\n",
       "      <th>neg</th>\n",
       "      <th>neut</th>\n",
       "      <th>pos</th>\n",
       "      <th>sen_tone_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>82</td>\n",
       "      <td>461</td>\n",
       "      <td>46</td>\n",
       "      <td>-0.078091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>76</td>\n",
       "      <td>435</td>\n",
       "      <td>40</td>\n",
       "      <td>-0.082759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>79</td>\n",
       "      <td>378</td>\n",
       "      <td>46</td>\n",
       "      <td>-0.087302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>63</td>\n",
       "      <td>358</td>\n",
       "      <td>50</td>\n",
       "      <td>-0.036313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>42</td>\n",
       "      <td>274</td>\n",
       "      <td>35</td>\n",
       "      <td>-0.025547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>42</td>\n",
       "      <td>273</td>\n",
       "      <td>26</td>\n",
       "      <td>-0.058608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>52</td>\n",
       "      <td>369</td>\n",
       "      <td>39</td>\n",
       "      <td>-0.035230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>46</td>\n",
       "      <td>382</td>\n",
       "      <td>48</td>\n",
       "      <td>0.005236</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "tone                                     filename  neg  neut  pos  \\\n",
       "0     edgar/data/1318605/0001193125-11-054847.txt   82   461   46   \n",
       "1     edgar/data/1318605/0001193125-12-081990.txt   76   435   40   \n",
       "2     edgar/data/1318605/0001193125-13-096241.txt   79   378   46   \n",
       "3     edgar/data/1318605/0001193125-14-069681.txt   63   358   50   \n",
       "4     edgar/data/1318605/0001564590-15-001031.txt   42   274   35   \n",
       "5     edgar/data/1318605/0001564590-16-013195.txt   42   273   26   \n",
       "6     edgar/data/1318605/0001564590-17-003118.txt   52   369   39   \n",
       "7     edgar/data/1318605/0001564590-18-002956.txt   46   382   48   \n",
       "\n",
       "tone  sen_tone_score  \n",
       "0          -0.078091  \n",
       "1          -0.082759  \n",
       "2          -0.087302  \n",
       "3          -0.036313  \n",
       "4          -0.025547  \n",
       "5          -0.058608  \n",
       "6          -0.035230  \n",
       "7           0.005236  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_tone_counts = pd.pivot_table(mda_sen_tone_df, columns='tone', index='filename', aggfunc='count')['tone_score'].reset_index()\n",
    "mda_tone_counts['sen_tone_score'] = (mda_tone_counts['pos'] - mda_tone_counts['neg']) / mda_tone_counts['neut']\n",
    "mda_tone_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5c) Graph both tone scores over time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine both scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>tone_score</th>\n",
       "      <th>sen_tone_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>-0.002520</td>\n",
       "      <td>-0.078091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>-0.002575</td>\n",
       "      <td>-0.082759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>-0.002409</td>\n",
       "      <td>-0.087302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  tone_score  sen_tone_score\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt   -0.002520       -0.078091\n",
       "1  edgar/data/1318605/0001193125-12-081990.txt   -0.002575       -0.082759\n",
       "2  edgar/data/1318605/0001193125-13-096241.txt   -0.002409       -0.087302"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_scores_df = pd.merge(mda_tone_df[['filename', 'tone_score']], mda_tone_counts[['filename', 'sen_tone_score']], how='left', on='filename')\n",
    "mda_scores_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add year of each filing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "mda_scores_df = pd.merge(mda_scores_df, tesla_10k_df[['fname', 'fdate']].rename(columns={'fname' : 'filename'}), how='left', on='filename')\n",
    "mda_scores_df['fdate'] = pd.to_datetime(mda_scores_df['fdate'])\n",
    "mda_scores_df['year'] = mda_scores_df['fdate'].dt.year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>filename</th>\n",
       "      <th>tone_score</th>\n",
       "      <th>sen_tone_score</th>\n",
       "      <th>fdate</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>edgar/data/1318605/0001193125-11-054847.txt</td>\n",
       "      <td>-0.002520</td>\n",
       "      <td>-0.078091</td>\n",
       "      <td>2011-03-03</td>\n",
       "      <td>2011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>edgar/data/1318605/0001193125-12-081990.txt</td>\n",
       "      <td>-0.002575</td>\n",
       "      <td>-0.082759</td>\n",
       "      <td>2012-02-27</td>\n",
       "      <td>2012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>edgar/data/1318605/0001193125-13-096241.txt</td>\n",
       "      <td>-0.002409</td>\n",
       "      <td>-0.087302</td>\n",
       "      <td>2013-03-07</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>edgar/data/1318605/0001193125-14-069681.txt</td>\n",
       "      <td>-0.001201</td>\n",
       "      <td>-0.036313</td>\n",
       "      <td>2014-02-26</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>edgar/data/1318605/0001564590-15-001031.txt</td>\n",
       "      <td>-0.000547</td>\n",
       "      <td>-0.025547</td>\n",
       "      <td>2015-02-26</td>\n",
       "      <td>2015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>edgar/data/1318605/0001564590-16-013195.txt</td>\n",
       "      <td>-0.001322</td>\n",
       "      <td>-0.058608</td>\n",
       "      <td>2016-02-24</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>edgar/data/1318605/0001564590-17-003118.txt</td>\n",
       "      <td>-0.001253</td>\n",
       "      <td>-0.035230</td>\n",
       "      <td>2017-03-01</td>\n",
       "      <td>2017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>edgar/data/1318605/0001564590-18-002956.txt</td>\n",
       "      <td>0.000839</td>\n",
       "      <td>0.005236</td>\n",
       "      <td>2018-02-23</td>\n",
       "      <td>2018</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      filename  tone_score  sen_tone_score  \\\n",
       "0  edgar/data/1318605/0001193125-11-054847.txt   -0.002520       -0.078091   \n",
       "1  edgar/data/1318605/0001193125-12-081990.txt   -0.002575       -0.082759   \n",
       "2  edgar/data/1318605/0001193125-13-096241.txt   -0.002409       -0.087302   \n",
       "3  edgar/data/1318605/0001193125-14-069681.txt   -0.001201       -0.036313   \n",
       "4  edgar/data/1318605/0001564590-15-001031.txt   -0.000547       -0.025547   \n",
       "5  edgar/data/1318605/0001564590-16-013195.txt   -0.001322       -0.058608   \n",
       "6  edgar/data/1318605/0001564590-17-003118.txt   -0.001253       -0.035230   \n",
       "7  edgar/data/1318605/0001564590-18-002956.txt    0.000839        0.005236   \n",
       "\n",
       "       fdate  year  \n",
       "0 2011-03-03  2011  \n",
       "1 2012-02-27  2012  \n",
       "2 2013-03-07  2013  \n",
       "3 2014-02-26  2014  \n",
       "4 2015-02-26  2015  \n",
       "5 2016-02-24  2016  \n",
       "6 2017-03-01  2017  \n",
       "7 2018-02-23  2018  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mda_scores_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the `tone_score` to make it more comparable to `sen_tone_score`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "mda_scores_df['tone_score_scaled'] = mda_scores_df['tone_score'] * 30"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1d817a47d30>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmUAAAFcCAYAAACJEsnfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XucVXW9//HXR0BI8YKAipKCx0sKIiZCPewgpuCFTmrWMdOfZBc7ZZrHU6nHfoK3Qh+ebr+sDqcoTxaiVmppklrEyWMKGCp4SUxMvHIR8q4Mn98fe0HjsAaG9sheM/N6Ph77MXut9d1rfeazZ+A937X23pGZSJIkqbE2a3QBkiRJMpRJkiRVgqFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiS1WURcFRGTGl1HZxURW0TETRGxMiKmNbqeMhExNCJebXQdUmdkKJM6sIj4cURMbbHu4IhYFhEDGlVXSxFxWEQsanQdHcDxwHZA38w8oZ4dRcSEiHixuL0SEaubLa9on3IltSdDmdSxnQEcFRFjASKiF/BfwL9l5tPteaCI6Nae++ssIqJ7O+5uV+DhzFxVbx2ZeWVm9s7M3sA/AX9Zs5yZ27ZTvZLakaFM6sAycxlwOjAlIrYEJgKPZuYPASJis4j494h4NCKWRsTVEdGn2bbrIuKZiFgRETMjYu81+y5OVV4REbdExEvAPzY/dkQ8FBFHNlvuGRHPR8TQFuO2AX4B7NJspmb7iOgVEd+MiKcj4smI+GpEbF485rCIWBQRX4yIJRHxVESc3GyfvYrxT0TEsxHx7SKQriMi9oyIWcUpwaUR8ZNm2/aNiNsiYnnRhy822/+Gavv3iHiGWggmIt4fEfcWvfx98z4UY5+KiL8WfRtTUuclwL8DJxY9mlA8R+dHxOMR8VxE/DAiti7G7x4RGRGnRMRfgF+X/pCsR0TsGhG/KPryaER8stm2f4yIeUXNT0fExa3s4zMR8XBEvBARjzR/niRtHEOZ1MFl5rXAXGAacCrwqWabzwLGA6OBgcBLwDebbf8lsAewIzAf+FGL3X8EuADYCrizxbb/Bk5qtvw+YFFmzm9R30rWnal5DjgfGAEMA/YHDgLObfbQgcDbgJ2AfwG+syaQAJcDg4vH7gEMAs5bpzk1lwA3AX2KfV4Ba8PibdQC4wBgT2Bm8Zi21NYb2AX4TEQcSC2cfQLoC0wFboiIzSNiCLXn5J2ZuTVwJPCXlkVm5nnAZcCPix5dWezvJGAM8A/F9/CNFg8dDbyD2vPcZsXM2q+K73lA8fiJEfGeYsi3gQuKmvcCbmxlV08ChwNbA58F/jMi3rExtUgqZKY3b946+A3YAXgR+FyL9Y8ABzdbfjvwGrBZyT76AQlsWSxfBUxtMeYqYFKzff0V6F0sXw+c1Up9h1ELbM3XPQ6Ma7Y8HljYbPyLQLdm25dTC0qbAa8Cuzbb9o/AI60c+yfAd4CdW6z/P8CcVh6zodpeBTZvtv2/gIkt9vEotTC3F/AscCjQfQPP48XAD5st/w44tdnykDXPH7B78Xzt0oafj7L+Hwo82GLdJcD/K+7fA5wNbNdizFDg1fUc6zbg443+nfDmrSPenCmTOoHMfBZYCixosWkX4BfFKbUVwP3U/iPfPiK6RcRlEfHniPgrsLB4TL9mj39iPcd8ArgbODYitgPGUQtAbTWAWvhZ43Fg52bLSzOzqdnyy9Rmp3YEegL3Nvu+fgls38px/g3oAcyJiPsjYkKx/u387Xve2NqezczXmy3vCpy9pp6ipgHUguDDRQ0XAs9FxLSI2LGV47a0U0kdmwP9m61r9TnagF2B3VvUfAa1/kIttI4AHomIP0Rx3WJLEXFsRMwuTgGvoDZz169srKT1a88LVCVVz2LgI5l5V8sNEXEKcBTwXmr/2fcFlgDRbFhuYP9XUju91huYlZnPtDKubD9PU1zYXizvQu1U2IY8C7wO7FWE0fXK2gsePgEQEaOBWyNiFrUwc2wrD9tQbS2/nyeoneq7tJUargKuKk6Z/hfwFeCUDdUOPFXUscYu1L73JdROKZOZG3qOWvMEMD8z92+l5gXAh6L2Ao+TgJ9FcT3iGsXp5OnA0cCtmbkqIm7jzT9DktrImTKpc/su8OWI2AWguMD+/cW2raidClsGbEHt1NXG+hkwitq1RP+9nnHPAv0iYqtm66YB50dEv4joD/xfaqdH16uYPfse8PWI6B81AyNiXNn4iPjniFgzy7WCWqBqonaN1C4R8dni2q+tI2Lk31nbFOC0iDiwqKd3RPxTRGwZEXtHxCER0RN4pbg1rWdfzU0DzoqIQUXvLgGmZebqNj5+fX4H9Cy+/54R0T0i9ouI4QARcXJEbFf0eyWwmnXD6Nuo/XG/BFgdEcfS4gUhktrOUCZ1bl8FbgFuj4gXgP8FDiy2/YDaTMxT1E57/u/G7jwzX6J2LdkuxdfWxs0HfgosKk6VbU/tBQT3Ujuleh9wF7UZpLb4N2qze3dTCwy/pnbBf5lRwOyovYL0Z8BpmfmXrL0AYSxwHPAc8Cfg4OIxG1VbMRP5aWrXrj1f7GvNiyB6UruAfynwDLWL9b/Uxu/zv6jNRP0P8GfgBeBzbXzsehWnX4+k9iKCv1DrwRXUZj2hNvv1p+Ln5gLg+Bank9ecNj+b2gsGllG79u6W9qhP6ori75/5liSIiAupXWz+0UbXIkkdmdeUSfq7RURfatdGHd/oWiSpo/P0paS/S0R8mtpprxsyc6NPfUqS3szTl5IkSRXgTJkkSVIFGMokSZIqoENe6N+vX78cNGhQo8uQJEnaoLlz5y7NzP4bGtchQ9mgQYOYM2dOo8uQJEnaoIh4fMOjPH0pSZJUCYYySZKkCjCUSZIkVUCHvKaszBtvvMHixYt59dVXG12KNqFevXoxcOBAevTo0ehSJEmqS6cJZYsXL2arrbZi0KBBRESjy9EmkJksW7aMxYsXM3jw4EaXI0lSXTrN6ctXX32Vvn37Gsi6kIigb9++zo5KkjqFThPKAANZF+RzLknqLDpVKJMkSeqoOs01ZS0NOuemdt3fosnj17t9xYoV/OQnP+Ezn/lMux63sxozZgyXX345I0aMaNP4mTNncvnll/PLX/7yLa5MkqTGcKasnaxYsYJvf/vbjS7j79LU1NToEiRJ6vI67UzZpnbOOefw6KOPMnz4cMaOHQvAr371KyKCL33pSxx//PHMnDmTSZMm0a9fP+bPn88BBxzAVVddRUQwd+5czjrrLF588UX69evHD3/4QwYMGFB6rG9+85t897vfpXv37uyzzz5cffXVvPjii5x++unMmTOHiGDixIkcd9xxTJs2jS9/+ctkJuPHj+fSSy8FoHfv3px11lnMmDGD//iP/+Btb3vbW3L8T3/608yePZtXXnmFD37wg1xwwQXr7O/Xv/41EydO5LXXXuMf/uEf+MEPfkDv3r255ZZbOPPMM+nXrx/vfOc72+mZkiRVWXud6drQGa4qMpS1k8mTJzN//nzmzZvHT3/6U7773e9y7733snTpUg488EBGjx4NwB//+EcWLFjATjvtxEEHHcQdd9zBqFGjOP3007nhhhvo378/06dP57zzzmPq1KmtHuuxxx6jZ8+erFixAoCLLrqIbbbZhvvvvx+A559/nqeeeoqzzz6buXPn0qdPH8aNG8f111/PMcccw0svvcTQoUO58MILeeONNzj44IPb/fgAl1xyCdtttx1NTU0ceuih3HfffQwbNmztvpYuXcrFF1/MbbfdxpZbbsmll17KV7/6Vb74xS/yyU9+kt/85jfsvvvuHH/88e3wLEmSVF2GsrfA73//e0444QS6devGDjvswMEHH8zs2bPZeuutGTlyJAMHDgRg+PDhLFq0iG233Zb58+evnWFrampqdZYKYNiwYZx44okcc8wxHHPMMQDcdtttXH311WvH9OnTh1mzZjFmzBj69699MP2JJ57IrFmzOOaYY+jWrRvHHXccAA8//PBbcnyAa665hilTprBq1SqefvppHnjggTeFsj/84Q888MADHHTQQQC8/vrrvPvd7+ahhx5i8ODB7LHHHgCcdNJJTJkyZYO9lySpozKUvQUys9VtPXv2XHu/W7durFq1isxkyJAh3HnnnW3a/0033cSsWbO48cYbueiii1iwYAGZuc7bQ6yvjl69etGtW7e1496K4z/22GNcfvnlzJ49mz59+vDRj350nfcUy0zGjh3LtGnT3rR+3rx5vt2FJKlL8UL/drLVVlvxwgsvADB69GimT59OU1MTS5YsYdasWYwcObLVx+61114sWbJkbSh64403WLBgQenY1atX88QTT3DIIYdw2WWXsWLFCl588UXGjRvHt771rbXjnn/+eUaNGsXvfvc7li5dSlNTE9OmTePggw/eZMf/61//ypZbbsk222zDs88+y69+9at19veud72LO+64g4ULFwLw8ssv86c//Yl3vOMdPPbYYzz66KMA64Q2SZI6m047U7apL/Dr27cvBx10EEOHDuXII49k2LBh7LfffkQEl112GTvuuCMPPfRQ6WM333xzrrvuOs444wxWrlzJqlWrOPPMMxkyZMg6Y5uamjjppJNYuXIlmcm//uu/su222/KlL32J0047jaFDh9KtWzcmTpzIBz7wAb7yla9wyCGHkJkcddRRHH300Zv0+Pvvvz9Dhgxht912W3uKsrn+/fvzwx/+kBNOOIHXXnsNgIsvvpg999yTKVOmMH78ePr168d73vMe5s+fv7FPiyRJHUas7xRXVY0YMSLnzJnzpnUPPvgge++9d4MqUiP53EtS59EZX30ZEXMzc4NvzOnpS0mSpApol1AWEUdExMMRsTAizinZ3jMiphfb74qIQcX6sRExNyLuL76+tz3q6SxOO+00hg8f/qbbD37wgy5zfEmSupK6rymLiG7AFcBYYDEwOyJuzMwHmg37OPB8Zu4eER8GLgWOB5YC/5SZT0XEUGAGsHO9NXUWV1xxRZc+viRJXUl7zJSNBBZm5p8z83XgaqDl1eRHA1cW968DDo2IyMw/ZuZTxfoFQK+I6IkkSVIX0x6hbGfgiWbLi1l3tmvtmMxcBawE+rYYcxzwx8x8rewgEXFqRMyJiDlLlixph7IlSZKqoz1CWdk7fLZ8Sed6x0TEEGqnND/V2kEyc0pmjsjMEWveoV6SJKmzaI9Qthh4e7PlgcBTrY2JiO7ANsDyYnkg8HPg5Mx8tB3q6VTmzZvHzTff3OgyJEnSW6w93jx2NrBHRAwGngQ+DHykxZgbgQnAncAHgd9kZkbEtsBNwLmZeUc71PI3k7Zp190xaWX77q+N5s2bx5w5czjqqKMacvx6NDU1rf0oJ0mStH51z5QV14h9ltorJx8ErsnMBRFxYUS8vxj2faBvRCwEzgLWvG3GZ4Hdgf8bEfOK2/b11tQIL730EuPHj2e//fZj6NChTJ8+nblz53LwwQdzwAEHcPjhh/P0008DMGbMGM4++2xGjhzJnnvuyf/8z/+U7vP111/n/PPPZ/r06QwfPpzp06ezfPlyjjnmGIYNG8a73vUu7rvvPgAmTZrExz72McaMGcNuu+3GN7/5zbX7ueqqqxg5ciTDhw/nU5/6FE1NTaXHa2pq4qMf/ShDhw5l33335Wtf+xoACxcu5LDDDmO//fbjne98J48++iiZyRe+8IW1Y6dPnw7AzJkzOeSQQ/jIRz7Cvvvuu1HHlySpK2uXj1nKzJuBm1usO7/Z/VeBD5U87mLg4vaoodFuueUWdtppJ266qfZOxCtXruTII4/khhtuoH///kyfPp3zzjuPqVOnArBq1Sruvvtubr75Zi644AJuu+22dfa5+eabc+GFFzJnzpy1nyt5+umns//++3P99dfzm9/8hpNPPpl58+YB8NBDD/Hb3/6WF154gb322otPf/rTLFy4kOnTp3PHHXfQo0cPPvOZz/DjH/+Yk08+eZ3jzZs3jyeffHLtxxmtWLECgBNPPJFzzjmHY489lldffZXVq1fzs5/9jHnz5nHvvfeydOlSDjzwQEaPHg3A3Xffzfz58xk8eDAPPvhgm48vSVJX1mk/+3JT23ffffn85z/P2Wefzfve9z769OnD/PnzGTt2LFCbhRowYMDa8R/4wAcAOOCAA1i0aFGbj/P73/+en/70pwC8973vZdmyZaxcWTu1On78eHr27EnPnj3ZfvvtefbZZ7n99tuZO3cuBx54IACvvPIK229fPhm522678ec//5nTTz+d8ePHM27cOF544QWefPJJjj32WAB69eq1to4TTjiBbt26scMOO3DwwQcze/Zstt56a0aOHMngwYMBNur4kiR1ZYaydrLnnnsyd+5cbr75Zs4991zGjh3LkCFDuPPOO0vH9+xZezu2bt26sWrVqjYfp+yzSiPiTftsvt/MZMKECXzlK1/Z4L779OnDvffey4wZM7jiiiu45ppr+PrXv97mOtbYcsst3zSurceXJKkr87Mv28lTTz3FFltswUknncTnP/957rrrLpYsWbI2lL3xxhssWLBgo/e71VZb8cILL6xdHj16ND/+8Y+B2vVb/fr1Y+utt2718YceeijXXXcdzz33HADLly/n8ccfLx27dOlSVq9ezXHHHcdFF13EPffcw9Zbb83AgQO5/vrrAXjttdd4+eWXGT16NNOnT6epqYklS5Ywa9YsRo4cWdfxJUnqypwpayf3338/X/jCF9hss83o0aMH3/nOd+jevTtnnHEGK1euZNWqVZx55pkMGTJko/Z7yCGHMHnyZIYPH865557LpEmTOOWUUxg2bBhbbLEFV1555Xofv88++3DxxRczbtw4Vq9eTY8ePbjiiivYdddd1xn75JNPcsopp7B69WqAtbNbP/rRj/jUpz7F+eefT48ePbj22ms59thjufPOO9lvv/2ICC677DJ23HFHHnroob/7+JIkdWWxvtNQVTVixIicM2fOm9Y9+OCD7L333g2qSI3kcy9Jncegc25ql/0smjy+XfbTHiJibmaO2NA4T19KkiRVgKcvK2LGjBmcffbZb1o3ePBgfv7zn78lxxs1ahSvvfbmjxn90Y9+tPa9xSRJ0qZlKKuIww8/nMMPP3yTHe+uu+7aZMeSJEkb1qlOX3bE6+NUH59zSVJn0WlCWa9evVi2bJn/SXchmcmyZcvWvqGtJEkdWac5fTlw4EAWL17MkiVLGl2KNqFevXoxcODARpchSVLdOk0o69Gjx9qP9pEkSepoOs3pS0mSpI7MUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqoDujS5AkiSp3U3aph32sbL+fWwEZ8okSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaqAdgllEXFERDwcEQsj4pyS7T0jYnqx/a6IGNRs27nF+ocj4vD2qEeSJKmjqTuURUQ34ArgSGAf4ISI2KfFsI8Dz2fm7sDXgEuLx+4DfBgYAhwBfLvYnyRJUpfSHjNlI4GFmfnnzHwduBo4usWYo4Eri/vXAYdGRBTrr87M1zLzMWBhsT9JkqQupT0+Zmln4Ilmy4uBUa2NycxVEbES6Fus/0OLx+5cdpCIOBU4FWCXXXapu+hB59xU9z4AFvX6SP072cQf47A+7dGXdukJ2JfWVKQv/g6Vsy/l/B0qZ1/WtWjy+HbaUzW+n43RHjNlUbIu2zimLY+trcyckpkjMnNE//79N7JESZKkamuPULYYeHuz5YHAU62NiYjuwDbA8jY+VpIkqdNrj1A2G9gjIgZHxObULty/scWYG4EJxf0PAr/JzCzWf7h4deZgYA/g7naoSZIkqUOp+5qy4hqxzwIzgG7A1MxcEBEXAnMy80bg+8CPImIhtRmyDxePXRAR1wAPAKuA0zKzqd6aJEmSOpr2uNCfzLwZuLnFuvOb3X8V+FArj70EuKQ96pAkSeqofEd/SZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFdG90AZLUmSyaPL59djSpfXYjqeNwpkySJKkCDGWSJEkVYCiTJEmqAEOZJElSBdQVyiJiu4i4NSIeKb72aWXchGLMIxExoVi3RUTcFBEPRcSCiJhcTy2SJEkdWb0zZecAt2fmHsDtxfKbRMR2wERgFDASmNgsvF2eme8A9gcOiogj66xHkiSpQ6o3lB0NXFncvxI4pmTM4cCtmbk8M58HbgWOyMyXM/O3AJn5OnAPMLDOeiRJkjqkekPZDpn5NEDxdfuSMTsDTzRbXlysWysitgX+idpsW6mIODUi5kTEnCVLltRZtiRJUrVs8M1jI+I2YMeSTee18RhRsi6b7b87MA34Zmb+ubWdZOYUYArAiBEjsrVxkiRJHdEGQ1lmHtbatoh4NiIGZObTETEAeK5k2GJgTLPlgcDMZstTgEcy8+ttqliSJKkTqvf05Y3AhOL+BOCGkjEzgHER0ae4wH9csY6IuBjYBjizzjokSZI6tHpD2WRgbEQ8AowtlomIERHxPYDMXA5cBMwubhdm5vKIGEjtFOg+wD0RMS8iPlFnPZIkSR1SXR9InpnLgENL1s8BPtFseSowtcWYxZRfbyZJktTl+I7+kiRJFWAokyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKqOstMSRJb5FJKxtdgaRNzJkySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaqA7o0uQJIk1WHSykZXoHZiKJP0d1k0eXz77GhS++xGkjo6Q5mkxvKvfEkCvKZMkiSpEgxlkiRJFeDpS0mSGqTdrs1Up+BMmSRJUgXUFcoiYruIuDUiHim+9mll3IRizCMRMaFk+40RMb+eWiRJkjqyemfKzgFuz8w9gNuL5TeJiO2AicAoYCQwsXl4i4gPAC/WWYckSVKHVu81ZUcDY4r7VwIzgbNbjDkcuDUzlwNExK3AEcC0iOgNnAWcClxTZy2SpIpql2unJtW/C6nK6p0p2yEznwYovm5fMmZn4Ilmy4uLdQAXAf8BvLyhA0XEqRExJyLmLFmypL6qJUmSKmaDM2URcRuwY8mm89p4jChZlxExHNg9M/81IgZtaCeZOQWYAjBixIhs47ElSZI6hA2Gssw8rLVtEfFsRAzIzKcjYgDwXMmwxfztFCfAQGqnOd8NHBARi4o6to+ImZk5BkmSpC6m3tOXNwJrXk05AbihZMwMYFxE9Cku8B8HzMjM72TmTpk5CHgP8CcDmSRJ6qrqDWWTgbER8QgwtlgmIkZExPcAigv8LwJmF7cL11z0L0mSpJq6Xn2ZmcuAQ0vWzwE+0Wx5KjB1PftZBAytpxZJkqSOzHf0lyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgUYyiRJkiqgez0PjojtgOnAIGAR8M+Z+XzJuAnAl4rFizPzymL95sC3gDHAauC8zPxpPTVJkjqpSSsbXYH0lqp3puwc4PbM3AO4vVh+kyK4TQRGASOBiRHRp9h8HvBcZu4J7AP8rs56JEmSOqR6Q9nRwJXF/SuBY0rGHA7cmpnLi1m0W4Ejim0fA74CkJmrM3NpnfVIkiR1SPWGsh0y82mA4uv2JWN2Bp5otrwY2Dkiti2WL4qIeyLi2ojYobUDRcSpETEnIuYsWbKkzrIlSZKqZYOhLCJui4j5Jbej23iMKFmX1K5nGwjckZnvBO4ELm9tJ5k5JTNHZOaI/v37t/HQkiRJHcMGL/TPzMNa2xYRz0bEgMx8OiIGAM+VDFtM7UL+NQYCM4FlwMvAz4v11wIfb1vZkiRJnUu9py9vBCYU9ycAN5SMmQGMi4g+xQX+44AZmZnAL/hbYDsUeKDOeiRJkjqkekPZZGBsRDwCjC2WiYgREfE9gMxcDlwEzC5uFxbrAM4GJkXEfcD/Af6tznokSZI6pLrepywzl1Gb4Wq5fg7wiWbLU4GpJeMeB0bXU4MkSVJn4Dv6S5IkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFGMokSZIqwFAmSZJUAYYySZKkCjCUSZIkVYChTJIkqQIMZZIkSRVgKJMkSaoAQ5kkSVIFdG90AVJHsGjy+Pp3Mqn+XUiSOi9nyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFdBl3xKjXd7iAHybA0mS1C6cKZMkSaoAQ5kkSVIFGMokSZIqoMteU6ZyfpyQJEmN4UyZJElSBThTVq9JKxtdgSRJ6gTqmimLiO0i4taIeKT42qeVcROKMY9ExIRm60+IiPsj4r6IuCUi+tVTjyRJUkdV7+nLc4DbM3MP4PZi+U0iYjtgIjAKGAlMjIg+EdEd+AZwSGaMeP7bAAAKQ0lEQVQOA+4DPltnPZIkSR1SvaHsaODK4v6VwDElYw4Hbs3M5Zn5PHArcAQQxW3LiAhga+CpOuuRJEnqkOoNZTtk5tMAxdftS8bsDDzRbHkxsHNmvgF8GrifWhjbB/h+aweKiFMjYk5EzFmyZEmdZUuSJFXLBkNZRNwWEfNLbke38RhRsi4joge1ULY/sBO105fntraTzJySmSMyc0T//v3beGhJkqSOYYOvvszMw1rbFhHPRsSAzHw6IgYAz5UMWwyMabY8EJgJDC/2/2ixr2souSZNkiSpK6j39OWNwJpXU04AbigZMwMYV1zc3wcYV6x7EtgnItZMe40FHqyzHkmSpA6p3vcpmwxcExEfB/4CfAggIkYA/5KZn8jM5RFxETC7eMyFmbm8GHcBMCsi3gAeBz5aZz2SJEkdUl2hLDOXAYeWrJ8DfKLZ8lRgasm47wLfracGSZKkzsB39Ff781MOJEnaaH72pSRJUgUYyiRJkirAUCZJklQBhjJJkqQKMJRJkiRVgKFMkiSpAgxlkiRJFWAokyRJqgBDmSRJUgX4jv7SpuInHUiS1sOZMkmSpAowlEmSJFWAoUySJKkCDGWSJEkVYCiTJEmqAEOZJElSBRjKJEmSKsBQJkmSVAGGMkmSpAowlEmSJFWAoUySJKkCIjMbXcNGi4glwOONrqPQD1ja6CIqxp6Usy/l7Es5+7Iue1LOvpSrUl92zcz+GxrUIUNZlUTEnMwc0eg6qsSelLMv5exLOfuyLntSzr6U64h98fSlJElSBRjKJEmSKsBQVr8pjS6gguxJOftSzr6Usy/rsifl7Eu5DtcXrymTJEmqAGfKJEmSKsBQJkmSVAGGMkmSpAowlEmSJFWAoawdRMQpja6hkSLiHRFxaET0brH+iEbVVAURMTIiDizu7xMRZ0XEUY2uq2oi4r8bXUOVRMR7ip+VcY2upZEiYlREbF3cf1tEXBARv4iISyNim0bX1ygRcUZEvL3RdVRNRGweESdHxGHF8kci4lsRcVpE9Gh0fW3lqy/bQUT8JTN3aXQdjRARZwCnAQ8Cw4HPZeYNxbZ7MvOdjayvUSJiInAk0B24FRgFzAQOA2Zk5iWNq65xIuLGlquAQ4DfAGTm+zd5UQ0WEXdn5sji/iep/T79HBgH/CIzJzeyvkaJiAXAfpm5KiKmAC8D1wGHFus/0NACGyQiVgIvAY8C04BrM3NJY6tqvIj4MbV/b7cAVgC9gZ9R+3mJzJzQwPLazFDWRhFxX2ubgD0zs+emrKcqIuJ+4N2Z+WJEDKL2j+aPMvMbEfHHzNy/oQU2SNGX4UBP4BlgYGb+NSLeBtyVmcMaWmCDRMQ9wAPA94Ck9vszDfgwQGb+rnHVNUbz35OImA0clZlLImJL4A+ZuW9jK2yMiHgwM/cu7r/pD7yImJeZwxtXXeNExB+BA6j9gXc88H5gLrXfo59l5gsNLK9hIuK+zBwWEd2BJ4GdMrMpIgK4t6P8m9u90QV0IDsAhwPPt1gfwP9u+nIqo1tmvgiQmYsiYgxwXUTsSq03XdWqzGwCXo6IRzPzrwCZ+UpErG5wbY00AvgccB7whcycFxGvdMUw1sxmEdGH2uUksWbWIzNfiohVjS2toeZHxCmZ+QPg3ogYkZlzImJP4I1GF9dAmZmrgV8Dvy5OzR0JnABcDmzwQ687qc0iYnNgS2qzZdsAy6n9YdxhTl8aytrul0DvzJzXckNEzNz05VTGMxExfE1fihmz9wFTgS75F37h9YjYIjNfpvZXLQDFtTBdNpQV/5l8LSKuLb4+i/8ObUNtpiOAjIgdM/OZ4hrNrvyHzSeAb0TEl4ClwJ0R8QTwRLGtq3rTz0RmvgHcCNxYzMR3Vd8HHgK6Ufuj79qI+DPwLuDqRha2MTx9qbpExEBqs0LPlGw7KDPvaEBZDRcRPTPztZL1/YABmXl/A8qqnIgYDxyUmf/e6FqqJiK2AHbIzMcaXUsjRcRWwG7UwvvizHy2wSU1VETsmZl/anQdVRQROwFk5lMRsS21U7x/ycy7G1tZ2xnK2kFE9F5zCk9/Y1/K2Zdy9mVd9qScfSlnX8p1pL74lhjt44FGF1BR9qWcfSlnX9ZlT8rZl3L2pVyH6UtXv5ajzSLirNY2UXvpbZdkX8rZl3L2ZV32pJx9KWdfynWWvjhT1nZfBvoAW7W49aZr99G+lLMv5ezLuuxJOftSzr6U6xR9caas7e4Brs/MuS03RERXfiWQfSlnX8rZl3XZk3L2pZx9Kdcp+uKF/m0UEXsBy8veOTkiduiqrwiyL+XsSzn7si57Us6+lLMv5TpLXwxlkiRJFdBhzrM2WkRsExGTI+KhiFhW3B4s1m3b6Poaxb6Usy/l7Mu67Ek5+1LOvpTrLH0xlLXdNdQ+YmlMZvbNzL7UPkj5eeDahlbWWPalnH0pZ1/WZU/K2Zdy9qVcp+iLpy/bKCIezsy9NnZbZ2dfytmXcvZlXfaknH0pZ1/KdZa+OFPWdo9HxBcjYoc1KyJih4g4m9pnsXVV9qWcfSlnX9ZlT8rZl3L2pVyn6IuhrO2OB/oCv4uI5yNiOTAT2A7450YW1mD2pZx9KWdf1mVPytmXcvalXKfoi6cvN0JEvAMYCPyh+edoRcQRmXlL4yprLPtSzr6Usy/rsifl7Es5+1KuM/TFmbI2iogzgBuAzwLzI+LoZpu/3JiqGs++lLMv5ezLuuxJOftSzr6U6yx98R392+6TwAGZ+WJEDAKui4hBmfkNap+t1VXZl3L2pZx9WZc9KWdfytmXcp2iL4aytuu2Zjo0MxdFxBhqT/qudKAn/C1gX8rZl3L2ZV32pJx9KWdfynWKvnj6su2eiYjhaxaKJ/99QD9g34ZV1Xj2pZx9KWdf1mVPytmXcvalXKfoixf6t1FEDARWZeYzJdsOysw7GlBWw9mXcvalnH1Zlz0pZ1/K2ZdynaUvhjJJkqQK8PSlJElSBRjKJEmSKsBQJkmSVAGGMknaSBHRrdE1SOp8DGWSOrWIuCgiPtds+ZKIOCMivhARsyPivoi4oNn26yNibkQsiIhTm61/MSIujIi7gHdv4m9DUhdgKJPU2X0fmAAQEZsBHwaeBfYARgLDgQMiYnQx/mOZeQAwAjgjIvoW67cE5mfmqMz8/ab8BiR1Db6jv6ROrXh372URsT+wA/BH4EBgXHEfoDe1kDaLWhA7tlj/9mL9MqAJ+OmmrF1S12Iok9QVfA/4KLAjMBU4FPhKZv5n80HFR7McBrw7M1+OiJlAr2Lzq5nZtKkKltT1ePpSUlfwc+AIajNkM4rbxyKiN0BE7BwR2wPbAM8XgewdwLsaVbCkrseZMkmdXma+HhG/BVYUs12/joi9gTsjAuBF4CTgFuBfIuI+4GHgD42qWVLX48csSer0igv87wE+lJmPNLoeSSrj6UtJnVpE7AMsBG43kEmqMmfKJEmSKsCZMkmSpAowlEmSJFWAoUySJKkCDGWSJEkVYCiTJEmqAEOZJElSBfx/QBQcrJCPdNwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d817a47710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mda_scores_df.set_index('year')[['tone_score_scaled', 'sen_tone_score']].plot(kind='bar', \n",
    "                                                                              figsize=(10,5),\n",
    "                                                                              title='Yearly tone scores for Tesla')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot changes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "mda_scores_df['sen_tone_score_lagged'] = mda_scores_df.sen_tone_score.shift(1)\n",
    "mda_scores_df['sen_tone_score_delta'] = mda_scores_df['sen_tone_score'] - mda_scores_df['sen_tone_score_lagged']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1d815072588>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmUAAAFcCAYAAACJEsnfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAHftJREFUeJzt3X+8ZXVd7/HXm5kAZQR1GBAYBiwxoatRjpCRZoIIZpJXTbBuaCLXLlztoZVmKSP+SPtF9tAyrmJkJSr5A41Q09CyRAZEfgjEQCgjP+K3DogEfu4faw1uzqwzc8ZzYH3POa/n47Efs9da373WZ3/2OXPe57vW3idVhSRJksa1zdgFSJIkyVAmSZLUBEOZJElSAwxlkiRJDTCUSZIkNcBQJkmS1ABDmbSAJPmbJGvGrmOhSvLmJH81dh0/qHT+OsltSf5t7HqGJFmWpJI8auxapAeboUx6ACX52ySnTFn3s0luTrLbWHVNleSQJFePXYcecE8DfhbYvap+ejY7SvK0JBv62x19kNowcdt9TiqWFhFDmfTAegXwrCTPAEiyPfD/gFdX1XVzeaAkS+ZyfwtFkqVj19CQvYD/rKo7t/aBU/tYVWdX1bKqWgb8eL9u2cTt2rkpWVo8DGXSA6iqbgb+L3Bykh2AE4Arq+qvAJJsk+R1Sa5MclOS05I8YmLb6Umu7083nZ1k34377k9VvivJWUnuAJ4yeewklyU5fGJ5uyS3JvkfU8btBHwCWDUxy7FLku2T/FmS65J8M8mfJNm2f8whSa5O8ttJbkxybZJfndjn9v34a5LckOTP+0C6iSSPTfKFJLf3Pfi7iW2PT/JPSW7p+/DbE/vfUm2vS3I9XQgmyXOSfLXv5b9O7cOUmgaP29uu7/23k1yc5CcnHvd7Sa7qt12S5DkT245J8vkkJ/U1XJXk0IntP9LX9e0kn07yF5OnSpMclORL/WMvSPLUiW0v7Z/zt/v9HjnwnI4F3g08pX+NX9+vf3mSdelmbz+WfgY3ydJ+9uv/JFkHXDZdvzbTx+V9r65P8o2+P+m3/ViSL/av+41J3jfNPp6X5ML+uX09yWu2tg5p3qgqb968PcA34HTgDOBmYNXE+t8EvgjsAWwPvBd4f79tG+DFwMP6be8E1k489m+AW4En92O369et6be/DvjbifHPA74yTX2HAFdPWfdW4N+AFcAuwDnACRPj76ELmT8EPAe4A9ix3/5O4KPAI4AdgTOBN01z7A8Dr+mfw/bAQf36nYAbgFf2z21H4ICtqO2twLbAQ4An9ft6ErAE+DXgSmDbgXo2d9w3A98Bntnv5w+Bf5147C8Bu/XP5UXABmDXftsxwH/3x15CF9avmXjsl4G39zU/Ffg28Ff9tj37r51n9vs+DLgJWN7XdzuwTz92N2C/aXp9DHD2xPKhwH8B+/e9/3Pgc/22pUABZ/Wv40M28/X9GKAG1n8GOKl/DXYHLgR+ud/2CeA3gPTbN77uy/rjPqpffgawb/+8V9N9zR8y9ve0N28PxG30Arx5Www3YNf+B/Qrp6y/AvjZieU9ge8C2wzsY+f+h9UO/fLfAKdMGTMZyvYEvgUs65c/BrxqmvqGQtnXgUMnln8eWDcxfgOwZGL7Lf0PzW2Au4C9JrY9BbhimmP/HfAXwB5T1v8vJkLoVtZ2FxOBi2627IQp+7hyYxDYiuO+GThrYvkJwIbNvO4XAz/f3z8GuGxi247967kz8MP96/6Qie2n8f1Q9rvA+6bs+7PAL/f7uQ14LrD9Fr4Op4ayU4G3TqnpXmAl3w9lT53B1/cmoQz4kf7rb+nEupcBn+jvfwR4B334mhhzv1A2cKz3ME3A9+Ztvt88fSk9CKrqBrqZjUumbFoFfKI/JXUbcBHdD6RdkixJ8gf96ahvAev6x+w88fhrNnPMa+hmX56b5JF0syJ/N934AbvRhZ+Nvk43o7fRTVV178TynXQ/UB9FN8P01Ynn9Um6Ga0hr6abbVub5KIkR/fr9+T7z3lra7uhqu6eWN4LeM3GevqadpvymI02d1yA6yfu3wnssHEhyYsnTpHeBjyO+79eUx8LXc92B26uqu9MbJ98bfcCjppS/0/RXbD/LeAo4Djg+iSfTPLYzdQ/aXcm+tjv61bu35dpv8a2YC+63tw0UfMf0/2CAt1M5I7ABX3PjhraSZKn9qe3b0pyO/Ar3L+n0oLhBbDSuNYDL6qqc6ZuSPIS4FnA0+l+cC4HbqQ73bNRbWH/p9L9EFsGfKGqrp9m3NB+rqP7wXp5v7wK+OYWjgfdqb+7gR/tw+hmVfeGh2Og+wEMfCbJF+jCwHOnediWapv6fK4B3lhVb59B/Zs77rSS/DDdjN/BwDlVdW+Si7n/6zWd64DlSbavqrv6dXvSzfhtrOl9VfXrQw+uqn8E/jHJQ4DfB/4S+LkZHPdauj5ufA4PoztVubleztQ1dKdVl1fVJvvof2l4SX+N2dOBs5J8nm52bdKHgDcAp1bVd5O8h5n1VJp3nCmTxvVu4K1JVgGku8B+48XhD6M7pXUz8FDgLT/A/j8CHAgcD/z1ZsbdAOzc/1De6APAG5LsnGQF8Hq606Ob1c+evQf40yQr0lk5eVH7pCS/lGTjzMxtdCHgXrpr8FYlOT7Jtkl2THLAD1jbycBxSZ7U17MsyS+ke/PFVJs77uZsPO12Y/e0cgzdTNkWVdWVdLOkJ/TH/Bm6U7IbvZ9uxvMZ/Qzq9kl+LsnuSXbrn8tD6cLwHXT9m4kPAC9N8oQk29EFun+pqvUzfPzmntMVwPl0X9/L0r1x5bFJDgJIcmSS3frANvm63yfJNnR9vRm4u+/L82Zbm9QqQ5k0rj+hu5D6s0m+TXfx+pP6be+jm8m4lu6051Z/2GdV3UF3Ldmq/t/pxl0M/D1wdX+qaRfgjcBX6cLChXQX0//+DA/9arrZvS/TzZZ8GthnmrEHAuemewfpR4DjquobVXU73UXez6O7GP0/6D5ji62trZ+J/HW6maxb+339yjRjN3fcaVXVhcCf9c/5OrpAtskM6GYcRXeB/810b6D4IF0op6quppu9ez1d6PsGXY+3oXvTwG/1x7wZ+Gm6EL5FVXUWcCLdmzKuo/s6+eWtqHlLXkh3uvJyumsOP0D35gyAg4Dzk2zo17906sxqVX0PeDndtWe3A6+ie9OMtCBlYFZZ0gKS5ES6d3y+eOxaNHNJ/h64oKreNHYtkh4czpRJC1iS5cBL6E7fqWFJDkjy6P4037OAZwMfH7suSQ8eQ5m0QCX5dbrTXB+vqib/zqHuZ3fgC3SfT3YS8LL+lKikRcLTl5IkSQ1wpkySJKkBhjJJkqQGzMsPj915551r7733HrsMSZKkLTrvvPNuqqoVWxo3L0PZ3nvvzdq1a8cuQ5IkaYuSfH3Lozx9KUmS1ARDmSRJUgMMZZIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSA+bln1mSNL5LH7fv2CXcZ9/LLh27BEmaNWfKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAbMSShLcliSy5OsS/Lage3bJflgv/2cJHtP2b4qyYYkvzkX9UiSJM03sw5lSZYA7wIOB/YDjkqy35RhLwVurarHACcBb5+y/STgH2dbiyRJ0ny1dA72cQCwrqquAkhyGnAE8LWJMUcAa/r7pwPvTJKqqiS/CFwF3DEHtUiSpHlszZo1Y5dwnwe7lrk4fbkHcM3E8vp+3eCYqroHuB1YnmQH4DXAG7d0kCTHJlmbZO2NN944B2VLkiS1Yy5CWQbW1QzHvBE4qao2bOkgVXVyVa2uqtUrVqz4AcqUJElq11ycvlwP7DmxvBK4dpox65MsBXYCbgEOBJ6f5A+AhwPfS3JXVb1zDuqSJEmaN+YilJ0L7JPk0cA3gSOBF00ZcwZwNPDvwPOBz1VVAU/ZOCDJGmCDgUySJC1Gsw5lVXVPkuOBTwFLgFOq6pIkJwJrq+oM4L3A+5Oso5shO3K2x5UkSVpI5mKmjKo6Ezhzyro3TNy/C3jBFvaxZi5qkSRJmo/8RH9JkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJasDSsQuQJGmxWv/afxm7hPusfNtTxi5h0XOmTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGzEkoS3JYksuTrEvy2oHt2yX5YL/9nCR79+ufkeS8JBf1/z59LuqRJEmab2YdypIsAd4FHA7sBxyVZL8pw14K3FpVjwFOAt7er78J+IWqejxwNPD+2dYjSZI0H83FTNkBwLqquqqq7gZOA46YMuYI4NT+/unAwUlSVV+pqmv79ZcA2yfZbg5qkiRJmlfmIpTtAVwzsby+Xzc4pqruAW4Hlk8Z8zzgK1X13aGDJDk2ydoka2+88cY5KFuSJKkdcxHKMrCutmZMkh+jO6X5v6c7SFWdXFWrq2r1ihUrfqBCJUmSWjUXoWw9sOfE8krg2unGJFkK7ATc0i+vBD4K/GpVXTkH9UiSJM07cxHKzgX2SfLoJNsCRwJnTBlzBt2F/ADPBz5XVZXk4cA/AL9TVV+cg1okSZLmpVmHsv4aseOBTwGXAh+qqkuSnJjkOf2w9wLLk6wDXgVs/NiM44HHAK9PckF/22W2NUmSJM03S+diJ1V1JnDmlHVvmLh/F/CCgce9GXjzXNQgSZI0n/mJ/pIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDTCUSZIkNcBQJkmS1ABDmSRJUgMMZZIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDVg6dgGStJC86+WfG7uE+xz37qePXYKkreBMmSRJUgMMZZIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDTCUSZIkNcBQJkmS1ABDmSRJUgMMZZIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDTCUSZIkNcBQJkmS1ABDmSRJUgMMZZIkSQ0wlEmSJDVgTkJZksOSXJ5kXZLXDmzfLskH++3nJNl7Ytvv9OsvT/LMuahHkiRpvpl1KEuyBHgXcDiwH3BUkv2mDHspcGtVPQY4CXh7/9j9gCOBHwMOA/68358kSdKiMhczZQcA66rqqqq6GzgNOGLKmCOAU/v7pwMHJ0m//rSq+m5V/Sewrt+fJEnSojIXoWwP4JqJ5fX9usExVXUPcDuwfIaPlSRJWvCWzsE+MrCuZjhmJo/tdpAcCxwLsGrVqq2pT1tjzU5jV/B9a24fu4L7PP7Ux49dwn0uOvqisUsAYN/LLh27hCYd9+6nj11Ck/74hc8eu4T7vPqDnxy7hPusfNtTxi6hOWvWrBm7hNHMxUzZemDPieWVwLXTjUmyFNgJuGWGjwWgqk6uqtVVtXrFihVzULYkSVI75iKUnQvsk+TRSbalu3D/jCljzgCO7u8/H/hcVVW//sj+3ZmPBvYBvjwHNUmSJM0rsz59WVX3JDke+BSwBDilqi5JciKwtqrOAN4LvD/JOroZsiP7x16S5EPA14B7gOOq6t7Z1iRJkjTfzMU1ZVTVmcCZU9a9YeL+XcALpnnsW4C3zEUdkiRJ85Wf6C9JktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDTCUSZIkNWBOPjxWWuha+SPgkqSFy1Cm+1tz+9gVSJK0KHn6UpIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWrArEJZkkcm+UySK/p/HzHNuKP7MVckObpf99Ak/5DksiSXJHnbbGqRJEmaz2Y7U/Za4LNVtQ/w2X75fpI8EjgBOBA4ADhhIrz9UVU9DvgJ4KAkh8+yHkmSpHlptqHsCODU/v6pwC8OjHkm8JmquqWqbgU+AxxWVXdW1T8DVNXdwPnAylnWI0mSNC/NNpTtWlXXAfT/7jIwZg/gmonl9f26+yR5OPALdLNtkiRJi87SLQ1I8k/AowY2/e4Mj5GBdTWx/6XAB4A/q6qrNlPHscCxAKtWrZrhoSVJkuaHLYayqjpkum1JbkiyW1Vdl2Q34L8Ghq0HnjaxvBI4e2L5ZOCKqvrTLdRxcj+W1atX1+bGSpIkzTezPX15BnB0f/9o4OMDYz4FHJrkEf0F/of260jyZmAn4DdmWYckSdK8NttQ9jbgGUmuAJ7RL5NkdZL3AFTVLcCbgHP724lVdUuSlXSnQPcDzk9yQZJjZlmPJEnSvLTF05ebU1U3AwcPrF8LHDOxfApwypQx6xm+3kySJGnR8RP9JUmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGMkmSpAYYyiRJkhpgKJMkSWqAoUySJKkBhjJJkqQGGMokSZIasHTsAiRJC9+rP/jJsUuQmudMmSRJUgMMZZIkSQ0wlEmSJDXAUCZJktQAQ5kkSVIDDGWSJEkNMJRJkiQ1wFAmSZLUAEOZJElSAwxlkiRJDTCUSZIkNSBVNXYNWy3JjcDXx66jtzNw09hFNMaeDLMvw+zLMPuyKXsyzL4Ma6kve1XVii0NmpehrCVJ1lbV6rHraIk9GWZfhtmXYfZlU/ZkmH0ZNh/74ulLSZKkBhjKJEmSGmAom72Txy6gQfZkmH0ZZl+G2ZdN2ZNh9mXYvOuL15RJkiQ1wJkySZKkBhjKJEmSGmAokyRJaoChTJIkqQGGsjmQ5CVj1zCmJI9LcnCSZVPWHzZWTS1IckCSJ/X390vyqiTPGruu1iT567FraEmSn+m/Vg4du5YxJTkwyY79/YckeWOSTyR5e5Kdxq5vLElekWTPsetoTZJtk/xqkkP65RcleWeS45L80Nj1zZTvvpwDSb5RVavGrmMMSV4BHAdcCuwPvLKqPt5vO7+qfnLM+saS5ATgcGAp8BngQOBs4BDgU1X1lvGqG0+SM6auAn4O+BxAVT3nQS9qZEm+XFUH9PdfRvf99FHgUOATVfW2MesbS5JLgB+vqnuSnAzcCZwOHNyv/5+jFjiSJLcDdwBXAh8APlxVN45b1fiS/C3d/7cPBW4DlgEfoft6SVUdPWJ5M2Yom6EkF063CXhsVW33YNbTiiQXAU+uqg1J9qb7T/P9VfWOJF+pqp8YtcCR9H3ZH9gOuB5YWVXfSvIQ4JyqesKoBY4kyfnA14D3AEX3/fMB4EiAqvr8eNWNY/L7JMm5wLOq6sYkOwBfqqrHj1vhOJJcWlX79vfv9wtekguqav/xqhtPkq8AT6T7Be+FwHOA8+i+jz5SVd8esbzRJLmwqp6QZCnwTWD3qro3SYCvzpf/c5eOXcA8sivwTODWKesD/NuDX04zllTVBoCqujrJ04DTk+xF15vF6p6quhe4M8mVVfUtgKr6TpLvjVzbmFYDrwR+F/itqrogyXcWYxibsE2SR9BdTpKNsx5VdUeSe8YtbVQXJ3lJVb0P+GqS1VW1Nsljgf8eu7gRVVV9D/g08On+1NzhwFHAHwFb/KPXC9Q2SbYFdqCbLdsJuIXuF+N5c/rSUDZznwSWVdUFUzckOfvBL6cZ1yfZf2Nf+hmzZwOnAIvyN/ze3UkeWlV30v1WC0B/LcyiDWX9D5OTkny4//cG/H9oJ7qZjgCV5FFVdX1/jeZi/sXmGOAdSX4PuAn49yTXANf02xar+31NVNV/A2cAZ/Qz8YvVe4HLgCV0v/R9OMlVwE8Bp41Z2Nbw9KVmJclKulmh6we2HVRVXxyhrNEl2a6qvjuwfmdgt6q6aISympPk54GDqup1Y9fSmiQPBXatqv8cu5YxJXkY8MN04X19Vd0wckmjSvLYqvqPsetoUZLdAarq2iQPpzvF+42q+vK4lc2coWwOJFm28RSevs++DLMvw+zLpuzJMPsyzL4Mm0998SMx5sbXxi6gUfZlmH0ZZl82ZU+G2Zdh9mXYvOnLYr+WY8aSvGq6TXRvvV2U7Msw+zLMvmzKngyzL8Psy7CF0hdnymburcAjgIdNuS1jcffRvgyzL8Psy6bsyTD7Msy+DFsQfXGmbObOBz5WVedN3ZBkMb8TyL4Msy/D7Mum7Mkw+zLMvgxbEH3xQv8ZSvKjwC1Dn5ycZNfF+o4g+zLMvgyzL5uyJ8PsyzD7Mmyh9MVQJkmS1IB5c551bEl2SvK2JJclubm/Xdqve/jY9Y3FvgyzL8Psy6bsyTD7Msy+DFsofTGUzdyH6P7E0tOqanlVLaf7Q8q3Ah8etbJx2Zdh9mWYfdmUPRlmX4bZl2ELoi+evpyhJJdX1Y9u7baFzr4Msy/D7Mum7Mkw+zLMvgxbKH1xpmzmvp7kt5PsunFFkl2TvIbub7EtVvZlmH0ZZl82ZU+G2Zdh9mXYguiLoWzmXggsBz6f5NYktwBnA48EfmnMwkZmX4bZl2H2ZVP2ZJh9GWZfhi2Ivnj6ciskeRywEvjS5N/RSnJYVZ01XmXjsi/D7Msw+7IpezLMvgyzL8MWQl+cKZuhJK8APg4cD1yc5IiJzW8dp6rx2Zdh9mWYfdmUPRlmX4bZl2ELpS9+ov/MvQx4YlVtSLI3cHqSvavqHXR/W2uxsi/D7Msw+7IpezLMvgyzL8MWRF8MZTO3ZON0aFVdneRpdC/6XsyjF/wBYF+G2Zdh9mVT9mSYfRlmX4YtiL54+nLmrk+y/8aF/sV/NrAz8PjRqhqffRlmX4bZl03Zk2H2ZZh9GbYg+uKF/jOUZCVwT1VdP7DtoKr64ghljc6+DLMvw+zLpuzJMPsyzL4MWyh9MZRJkiQ1wNOXkiRJDTCUSZIkNcBQJkmS1ABDmSRtpSRLxq5B0sJjKJO0oCV5U5JXTiy/JckrkvxWknOTXJjkjRPbP5bkvCSXJDl2Yv2GJCcmOQd48oP8NCQtAoYySQvde4GjAZJsAxwJ3ADsAxwA7A88MclT+/G/VlVPBFYDr0iyvF+/A3BxVR1YVf/6YD4BSYuDn+gvaUHrP9375iQ/AewKfAV4EnBofx9gGV1I+wJdEHtuv37Pfv3NwL3A3z+YtUtaXAxlkhaD9wAvBh4FnAIcDPx+Vf3l5KD+T7McAjy5qu5Mcjawfb/5rqq698EqWNLi4+lLSYvBR4HD6GbIPtXffi3JMoAkeyTZBdgJuLUPZI8DfmqsgiUtPs6USVrwquruJP8M3NbPdn06yb7AvycB2AD8CnAW8PIkFwKXA18aq2ZJi49/ZknSgtdf4H8+8IKqumLseiRpiKcvJS1oSfYD1gGfNZBJapkzZZIkSQ1wpkySJKkBhjJJkqQGGMokSZIaYCiTJElqgKFMkiSpAYYySZKkBvx/paDN7cXoSIgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1d816479358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mda_scores_df.set_index('year')['sen_tone_score_delta'].plot(kind='bar', \n",
    "                                                             figsize=(10,5),\n",
    "                                                             title='Yearly tone score changes for Tesla')"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "name": "EDGAR_walkthrough.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}