AI for Medical Treatment/Week 2/C3M2_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ANfBloGyVq50"
   },
   "source": [
    "# Natural Language Entity Extraction\n",
    "Extracting Ground Truth Labels from Radiology Reports\n",
    "<hr/>\n",
    "\n",
    "<img src=\"example-report.png\" width=\"30%\" align=\"right\" style=\"padding-left:20px\">\n",
    "\n",
    "In this assignment you'll learn to extract information from unstructured medical text. In particular, you'll learn about the following topics:\n",
    "\n",
    "- Extracting disease labels from clinical reports\n",
    "    - Text matching\n",
    "    - Evaluating a labeler\n",
    "    - Negation detection\n",
    "    - Dependency parsing\n",
    "- Question Answering with BERT\n",
    "    - Preprocessing text for input\n",
    "    - Extracting answers from model output \n",
    "\n",
    "This assignment is inspired by the [work](https://arxiv.org/abs/1901.07031) done by Irvin et al and you should be well positioned to read and understand this manuscript by the end of the assignment. \n",
    "\n",
    "\n",
    "<a href=\"https://ieeexplore.ieee.org/document/7780643\">Image Credit</a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This assignment covers the folowing topics:\n",
    "\n",
    "\n",
    "- [1. Extracting Labels](#1)\n",
    "  - [1.1 Text Matching](#1-1)\n",
    "      - [Exercise 1](#ex-01)\n",
    "  - [1.2 Evaluating The Performance](#1-2)      \n",
    "  - [1.3 Cleanup](#1-3)\n",
    "  - [1.4 Finding Negative Mentions](#1-4)\n",
    "      - [Exercise 2](#ex-02)\n",
    "  - [1.5 Dependency Parsing](#1-5)\n",
    "- [2. Question Answering Using BERT](#2)\n",
    "  - [2.1 Roadmap](#2-1)\n",
    "  - [2.2 Preparing The Input](#2-2)\n",
    "      - [Exercise 3](#ex-03)\n",
    "  - [2.3 Getting Answer From Model Output](#2-3)\n",
    "      - [Exercise 4](#ex-04)\n",
    "      - [Exercise 5](#ex-05)\n",
    "  - [2.4 Putting It All Together](#2-4)\n",
    "  - [2.5 Try It Out](#2-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the following libraries for this assignment.\n",
    "\n",
    "- `matplotlib.pyplot` - standard plotting library\n",
    "- `nltk` - an NLP package \n",
    "- `pandas` - we'll use this to keep track of data\n",
    "- `tensorflow` - standard deep learning library\n",
    "- `transformers` - convenient access to pretrained natural language models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import nltk \n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from transformers import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, load the helper `util` library that we have prepared for this assignment in order to abstract some of the details. \n",
    "- We do encourage you to review the `util.py` file to learn more about the utility functions behind the scene."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5ruouERuVq53"
   },
   "outputs": [],
   "source": [
    "import util\n",
    "from util import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9E1tLUwQVq57"
   },
   "source": [
    "<a name=\"1\"></a>\n",
    "## 1 Extracting Labels\n",
    "\n",
    "In this part of the assignment, you'll extract disease labels for patients from unstructured clinical reports.  Unlike most assignments, instead of \"learning\" from the dataset, you will primarily build different \"rules\" that help us extract knowledge from natural language. \n",
    "- Because there is less risk of overfitting when using a \"rules-based\" approach, you will just use one dataset which will also be the test set.\n",
    "\n",
    "The test set consists of 1,000 X-ray reports that have been manually labeled by a board certified radiologist for the presence or lack of presence of different pathologies. \n",
    "- You also have access to the extracted \"Impression\" section of each report which is the overall summary of the radiologists for each X-ray. \n",
    "\n",
    "Have a look at the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0h94kdGkVq57",
    "outputId": "c45d8b0d-ac9c-4b9e-ac49-91fdfd3cd400"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test_df size: (1000, 12)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Report Impression</th>\n",
       "      <th>Cardiomegaly</th>\n",
       "      <th>Lung Lesion</th>\n",
       "      <th>Airspace Opacity</th>\n",
       "      <th>Edema</th>\n",
       "      <th>Consolidation</th>\n",
       "      <th>Pneumonia</th>\n",
       "      <th>Atelectasis</th>\n",
       "      <th>Pneumothorax</th>\n",
       "      <th>Pleural Effusion</th>\n",
       "      <th>Pleural Other</th>\n",
       "      <th>Fracture</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\\n \\n1.mild pulmonary edema, and cardiomegaly....</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\\n \\n1.unremarkable cardiomediastinal silhouet...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\\n1. lines and tubes are unchanged in position...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\n1. postoperative portable film with a right-...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>\\n \\n1.single frontal view of the chest demons...</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   Report Impression  Cardiomegaly  \\\n",
       "0  \\n \\n1.mild pulmonary edema, and cardiomegaly....          True   \n",
       "1  \\n \\n1.unremarkable cardiomediastinal silhouet...         False   \n",
       "2  \\n1. lines and tubes are unchanged in position...         False   \n",
       "3  \\n1. postoperative portable film with a right-...         False   \n",
       "4  \\n \\n1.single frontal view of the chest demons...         False   \n",
       "\n",
       "   Lung Lesion  Airspace Opacity  Edema  Consolidation  Pneumonia  \\\n",
       "0        False             False   True          False      False   \n",
       "1        False              True  False          False      False   \n",
       "2        False              True  False          False      False   \n",
       "3        False              True   True          False      False   \n",
       "4        False              True  False          False      False   \n",
       "\n",
       "   Atelectasis  Pneumothorax  Pleural Effusion  Pleural Other  Fracture  \n",
       "0         True         False              True          False     False  \n",
       "1        False         False             False          False      True  \n",
       "2        False         False             False          False     False  \n",
       "3         True          True             False          False     False  \n",
       "4        False         False              True          False     False  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"test_df size: {}\".format(test_df.shape))\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are a few example impressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Report Impression 0:\n",
      "\n",
      " \n",
      "1.mild pulmonary edema, and cardiomegaly.  trace pleural fluid \n",
      "effusions.\n",
      " \n",
      "2.low lung volumes with minimal basilar atelectasis.\n",
      " \n",
      "3.no new focal consolidation. \n",
      " \n",
      "4.interval placement of defibrillation pads.  \n",
      " \n",
      "\n",
      "\n",
      "Report Impression 1:\n",
      "\n",
      " \n",
      "1.unremarkable cardiomediastinal silhouette\n",
      " \n",
      "2.diffuse reticular pattern, which can be seen with an atypical \n",
      "infection or chronic fibrotic change.  no focal consolidation.\n",
      " \n",
      "3.no pleural effusion or pneumothorax\n",
      " \n",
      "4.mild degenerative changes in the lumbar spine and old right rib \n",
      "fractures. \n",
      " \n",
      "\n",
      "\n",
      "Report Impression 2:\n",
      "\n",
      "1. lines and tubes are unchanged in position.\n",
      "2. increasing retrocardiac opacity and left midlung zone opacity.\n",
      "3. there is a deep left costophrenic sulcus which is increased when\n",
      "compared with prior films. no definite evidence of left\n",
      "pneumothorax. clinical correlation is recommended. if clinically\n",
      "indicated, consider film in expiration or decubitus views.\n",
      "4. the icu team was informed of these results at 10 am on 05_02_2005.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(3):\n",
    "    print(f'\\nReport Impression {i}:')\n",
    "    print(test_df.loc[i, 'Report Impression'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that these reports are fairly unstructured, which makes information extraction challenging. Your goal will be to extract the presence or absence of different abnormalities from the raw text. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, see the distribution of abnormalities in the dataset. We have placed all the names of these abnormalities in a list named `CATEGORIES` and you'll use it as labels the graph below. You will also be using this list in the next coding exercises."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nPAAsvcbVq5-",
    "outputId": "d5f008ea-462f-4204-9838-05b791b43a7e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwcAAAEvCAYAAAAQOL09AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3de5SdVZ3n//fHAAkYiCKIZVBLJa1yDRBo7yI6jIoKaKaRRgF1Oj97vDuo6XakAbuFbm3NoCMaFUFlWuRmI7hEmpt4AxJyRS62Em350dC2Grkr4Tt/nF1wKCtJVVJV55C8X2tl1X72s5+9v+f4LDzfs/d+TqoKSZIkSXpMrwOQJEmS1B9MDiRJkiQBJgeSJEmSGpMDSZIkSYDJgSRJkqTG5ECSJEkSAFv0OgA9bIcddqjBwcFehyFJkqRN2OLFi39VVTuOdM7koI8MDg6yaNGiXochSZKkTViSn6/tnMuKJEmSJAEmB5IkSZIakwNJkiRJgMmBJEmSpMbkQJIkSRJgciBJkiSpMTmQJEmSBJgcSJIkSWpMDiRJkiQBJgeSJEmSmi16HYAetuLW1QzOv6jXYUy6VScf3OsQJEmShDMHkiRJkhqTA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKnZ5J5WlGQNsKKr6tCqWrUR/c0GnlxV39rY2CRJkqR+tsklB8C9VTV7pBNJAqSqHhxDf7OBOcCok4MNHEeSJEnqqU1+WVGSwSQ3JfkysBJ4SpJTkyxKcn2SE7ra7pfkB0mWJbkmyQzgRODwJEuTHJ7k+CTHdl2zso0x0jjvT3JtkuXd40iSJEn9aFNMDrZuH+SXJjm/1c0CPlNVu1XVz4EPVdUcYE/gJUn2TLIVcBbw7qraC3g5cDdwHHBWVc2uqrPWM/ZD4wDPasf705l92DfJi8f7xUqSJEnjZZNfVpRkEPh5Vf2oq82fJZlH5/UPALsCBdxWVdcCVNXv2vVjGbt7nIPavyXteDqdZOG73Re0OOYBTNlux7GMJUmSJI2rTTE5GMndQ4UkTweOBfarqt8kOR2YNoa+HuCRMy7d197dVQ5wUlV9bl2dVdVCYCHA1IFZNYY4JEmSpHG1KS4rWp/t6HyIX51kJ+CVrf4mYCDJfgBJtk2yBXAnsG3X9auAfVqbfYCnr2Wci4G3JJne2s5M8sRxfi2SJEnSuNlcZg4eUlXLkiwBbgT+Dfh+q/99ksOBTyXZGriXzr6Dy4H5SZYCJwHnAkcluR64Grh5LeN8J8lzgB+2pUl3AW8E7pjI1ydJkiRtqFS5kqVfTB2YVQNHL+h1GJNu1ckH9zoESZKkzUaSxe3hPH9kc1xWJEmSJGkEJgeSJEmSAJMDSZIkSY3JgSRJkiRgM3xaUT/bY+YMFrk5V5IkST3izIEkSZIkwORAkiRJUmNyIEmSJAlwz0FfWXHragbnX9TrMNbLHy2TJEnaNDlzIEmSJAkwOZAkSZLUmBxIkiRJAkwOJEmSJDUmB5IkSZKASU4OkqxJsjTJyiRnJ9mm1d81CWMfk+TTazl3aJLlSW5IsiLJocOue3LX8aokO0x0vJIkSdJkm+yZg3uranZV7Q78HnjbeHaeZMyPZk2yF/Bx4JCqeg7wWuDjSfZsTY4BnryWyyc8PkmSJGmy9HJZ0VXALsMrk7w/ybXtm/wTWt1gkpVdbY5NcnwrX5FkQZJFwLuTvCbJ1UmWJPmXJDutJ45jgY9W1S0A7e9JwPuTzAXmAGe2GY+t2zXvTHJdm2V4dovjsUlOS3JNG/uQVn9MkguSXAZcusHvliRJkjTBepIctG/QXwmsGFZ/EDAL2B+YDeyb5MWj6HKrqppTVf8IfA94blXtDXwN+MB6rt0NWDysbhGwW1Wd08pHthmPe9v5X1XVPsCpdJILgA8Bl1XV/sBLgY8leWw7tw8wt6peMnzwJPOSLEqyaM09q0fxUiVJkqSJMdnLXLZOsrSVrwK+OOz8Qe3fknY8nU6y8Iv19HtWV3ln4KwkA8BWwC0bFfHIzmt/FwOva+WDgNcmGUoWpgFPbeVLqurXI3VUVQuBhQBTB2bVBMQqSZIkjcpkJwf3VtXsdZwPcFJVfe4RlcnOPHKWY9qw6+7uKn8K+ERVXZDkAOD49cT0Y2BfYFlX3b7A9eu45v72dw0Pv4cBXl9VNw2L/U+HxSdJkiT1pX57lOnFwFuSTAdIMjPJE4HbgScmeUKSqcCr19HHDODWVj56FGN+HPirJINtzEHgr4F/bOfvBLYdZezvTJLWz96juEaSJEnqG3319Jyq+k6S5wA/bJ+x7wLeWFV3JDkRuIbOB/8b19HN8cDZSX4DXAY8fT1jLk3yQeCbSbYE/gB8oKqGlj+dDnw2yb3A89bR1UeABcDyJI+hs5xpXUmMJEmS1FdS5TL3fjF1YFYNHL2g12Gs16qTD+51CJIkSdpASRZX1ZyRzvXbsiJJkiRJPWJyIEmSJAkwOZAkSZLUmBxIkiRJAvrsaUWbuz1mzmCRm30lSZLUI84cSJIkSQJMDiRJkiQ1JgeSJEmSAPcc9JUVt65mcP5FvQ5DkqQJ5Y9pSv3LmQNJkiRJgMmBJEmSpMbkQJIkSRJgciBJkiSp2ajkIMmaJEuTrExydpJtWv1d4xPeOsc+Jsmn11L/Hy2uoX+7tnMfS3J9+7tjkquTLEnyojGOPSfJKeP1WiRJkqR+sLFPK7q3qmYDJDkTeBvwiY2OqkmyRVU9sAGXnlVV7xihfh6wfVWtSfIGYEVV/fexdl5Vi4BFGxCXJEmS1LfGc1nRVcAuwyuTvD/JtUmWJzmh1Q0mWdnV5tgkx7fyFUkWJFkEvDvJa7q+4f+XJDttSHBJLgCmA4uTfBD4B+CQNrOwdfdsR5K5SU5v5f/WZkaWJfluqzsgyYWtvH2Sb7TX96Mke7b645Oc1l7Pz5K8a0PiliRJkibLuPzOQZItgFcC3x5WfxAwC9gfCHBBkhcDv1hPl1tV1ZzWx+OB51ZVJfnvwAeA/7me6w9P8sKu4+dV1WuT3NU103E7MGdohiHJ2vo6DvivVXVrkseNcP4EYElVHZrkQODLwOx27tnAS4FtgZuSnFpVf1hP7JIkSVJPbGxysHWSpa18FfDFYecPav+WtOPpdJKF9SUHZ3WVdwbOSjIAbAXcMoq41rasaEN8Hzg9ydeB80Y4/0Lg9QBVdVmSJyTZrp27qKruB+5PcgewE/DL7ouTzKOz3Ikp2+04TiFLkiRJYzduew7WIsBJVfW5R1QmO/PIJU3Thl13d1f5U8AnquqCJAcAx294uOtUI8VTVW9L8qfAwXSWJO07hj7v7yqvYYT3u6oWAgsBpg7MquHnJUmSpMky0Y8yvRh4S5LpAElmJnkicDvwxPYt+1Tg1evoYwZwaysfPYGx3p7kOUkeAxw2VJnkmVV1dVUdB/wH8JRh110FHNnaHgD8qqp+N4FxSpIkSRNiXPYcrE1VfSfJc4AftjX9dwFvrKo7kpwIXEPng/+N6+jmeODsJL8BLgOePoqhh+85+B9V9YP1XDMfuJBOArCIzhIogI8lmUVnFuRSYBnwkmHxnZZkOXAPE5vASJIkSRMmVa5k6RdTB2bVwNELeh2GJEkTatXJB/c6BGmzlmTx0MN/hvMXkiVJkiQBJgeSJEmSGpMDSZIkSYDJgSRJkqRmQp9WpLHZY+YMFrlJS5IkST3izIEkSZIkwORAkiRJUmNyIEmSJAkwOZAkSZLUuCG5j6y4dTWD8y/qdRh9xV/RlCRJmjzOHEiSJEkCTA4kSZIkNSYHkiRJkgCTA0mSJEnNuCcHSdYkWZpkZZKzk2wz3mNsjCTHJHly1/GqJDv0MiZJkiSpH0zEzMG9VTW7qnYHfg+8bQLG2BjHAE9eX6PRSOLTniRJkrTJmOhlRVcBuyQ5IMkVSc5JcmOSM5MEIMm+Sa5MsjjJxUkGWv0VSea08g5JVrXyMUm+keSS9q3/O5K8L8mSJD9Ksn1rN7sdL09yfpLHJ5kLzAHObLMbW7c435nkuiQrkjy7Xb99G2d562fPVn98kq8k+T7wlSSDSa5q11+X5Pmt3WFJLk3HQJKbkzxpgt9vSZIkaYNNWHLQvlV/JbCiVe0NvAfYFXgG8IIkWwKfAuZW1b7AacDfjaL73YHXAfu19vdU1d7AD4GjWpsvAx+sqj1bDH9TVecAi4Aj2+zGva3tr6pqH+BU4NhWdwKwpF3/162/IbsCL6+qI4A7gP/Srj8cOAWgqs4HbgPeDny+jf/vo3htkiRJUk9MxLKYrZMsbeWrgC8CzweuqapfArTzg8Bv6XzQv6RNJEyh84F6fS6vqjuBO5OsBr7Z6lcAeyaZATyuqq5s9WcAZ6+jv/Pa38V0kg6AFwKvB6iqy5I8Icl27dwFXYnFlsCnk8wG1gB/0tXvO4GVwI+q6p9GGjjJPGAewJTtdlzPy5YkSZImzkQkB/dW1ezuivbB//6uqjVt7ADXV9XzRujnAR6e2Zg27Fx3Xw92HT/Ihr2moeuH4lqfu7vK7wVuB/aiE+99Xed2bjHtlOQxVfXg8I6qaiGwEGDqwKwae+iSJEnS+Oj1o0xvAnZM8jyAJFsm2a2dWwXs28pzx9JpVa0GfpPkRa3qTcDQLMKdwLaj6OYq4MgW1wF0lh79boR2M4Db2gf/N9GZ/RhaVnUacARwA/C+sbwGSZIkabL19Gk7VfX7tkn4lLYUaAtgAXA98HHg623ZzUUb0P3RwGfbo1R/Bry51Z/e6u8FRpqxGHI8cFqS5cA9rb+RfAY4N8lRwLd5eFbhr4Grqup7SZYB1ya5qKpu2IDXIkmSJE24VLmSpV9MHZhVA0cv6HUYfWXVyQf3OgRJkqRNSpLFVTVnpHO9XlYkSZIkqU+YHEiSJEkCTA4kSZIkNSYHkiRJkoAeP61Ij7THzBkscgOuJEmSesSZA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKlxQ3IfWXHragbnX9TrMKRNgr+uLUnS2DlzIEmSJAkwOZAkSZLUmBxIkiRJAkwOJEmSJDV9nRwkOTRJJXl2O56d5FWjuO6AJBdu4JjvSbLNBl77tiRHbci1kiRJUq/1dXIAHAF8r/0FmA2sNznYSO8BNig5qKrPVtWXxzkeSZIkaVL0bXKQZDrwQuCtwBuSbAWcCByeZGmSw5M8NslpSa5JsiTJISP0M2KbJFOSfDzJyiTLk7wzybuAJwOXJ7m8tTs1yaIk1yc5oavfk5P8uF378VZ3fJJjW/ldXee/NsFvlyRJkrTR+vl3Dg4Bvl1VNyf5T2AP4DhgTlW9AyDJR4HLquotSR4HXJPkX4b186G1tDkKGARmV9UDSbavql8neR/w0qr61dD1rX4KcGmSPYFbgcOAZ1dVtX6Hmw88varuX8t5SZIkqa/07cwBnaVEQ9+4f42HlxZ1OwiYn2QpcAUwDXjqKNu8HPhcVT0AUFW/Xkscf5bkOmAJsBuwK7AauA/4YpLXAfeMcN1y4MwkbwQeWNuLTDKvzUwsWnPP6rU1kyRJkiZcX84cJNkeOBDYI0kBU4ACrh/eFHh9Vd007PqdRtFmNHE8HTgW2K+qfpPkdGBam2nYH3gZMBd4R4u328HAi4HXAB9KssdQItKtqhYCCwGmDsyq9QYlSZIkTZB+nTmYC3ylqp5WVYNV9RTgFjrf+G/b1e5i4J1pn/ST7D1CX2trcwnw/yXZotVv3+rv7BpjO+BuYHVLOF7Z2k4HZlTVt4D3Ant1D5jkMcBTqupy4IPADGD6Br0TkiRJ0iTp1+TgCOD8YXXnAk8Cdh3akAx8BNgSWJ7k+nY83NrafAH4RatfBvx5q18IfDvJ5VW1jM5yohuB/wt8v7XZFrgwyXI6T1N637AxpwBfTbKiXX9KVf12rG+CJEmSNJlS5UqWfjF1YFYNHL2g12FIm4RVJx/c6xAkSepLSRZX1ZyRzvXrzIEkSZKkSWZyIEmSJAkwOZAkSZLUmBxIkiRJAvr0dw42V3vMnMEiN1FKkiSpR5w5kCRJkgSYHEiSJElqTA4kSZIkASYHkiRJkho3JPeRFbeuZnD+Rb0OY7385VlJkqRNkzMHkiRJkgCTA0mSJEmNyYEkSZIkwORAkiRJUtOXyUGSNUmWJlmZ5Owk2/Q6ptFIMifJKb2OQ5IkSdoQfZkcAPdW1eyq2h34PfC2Xgc0GlW1qKre1es4JEmSpA3Rr8lBt6uAXZIckOSKJOckuTHJmUkCkGTfJFcmWZzk4iQDrf6KJHNaeYckq1r5mCTfSHJJklVJ3pHkfUmWJPlRku1bu9nteHmS85M8vqvfv09yTZKbk7yo1R+Q5MJW3j/JD1ufP0jyrMl+4yRJkqSx6OvkIMkWwCuBFa1qb+A9wK7AM4AXJNkS+BQwt6r2BU4D/m4U3e8OvA7Yr7W/p6r2Bn4IHNXafBn4YFXt2WL4m67rt6iq/Vs83fVDbgRe1Po8DvjoqF60JEmS1CP9+iNoWydZ2spXAV8Eng9cU1W/BGjnB4Hf0vmgf0mbSJgC3DaKMS6vqjuBO5OsBr7Z6lcAeyaZATyuqq5s9WcAZ3ddf177u7jFMdwM4Iwks4ACthwpiCTzgHkAU7bbcRRhS5IkSROjX5ODe6tqdndF++B/f1fVGjrxB7i+qp43Qj8P8PDsyLRh57r7erDr+EFG974MtR+KY7iP0ElADksyCFwxUidVtRBYCDB1YFaNYlxJkiRpQvT1sqJRugnYMcnzAJJsmWS3dm4VsG8rzx1Lp1W1GvjN0H4C4E3Aleu4ZLgZwK2tfMxYxpYkSZJ64VGfHFTV7+l88P/7JMuApXSWIAF8HPjLJEuAHTag+6OBjyVZDswGThzDtf8AnNTG7tcZGkmSJOkhqXIlS7+YOjCrBo5e0Osw1mvVyQf3OgRJkiRtoCSLq2rOSOce9TMHkiRJksaHyYEkSZIkwORAkiRJUmNyIEmSJAnwKTp9ZY+ZM1jkZl9JkiT1iDMHkiRJkgCTA0mSJEmNyYEkSZIkwORAkiRJUuOG5D6y4tbVDM6/qNdhbDL8JWdJkqSxceZAkiRJEmByIEmSJKkxOZAkSZIEmBxIkiRJaiYlOUjypCRfS/LTJIuTfCvJn0zwmMcnObaVT0zy8hHaHJDkwvX0MzvJq7qOX5tk/vhHLEmSJPXWhD+tKEmA84EzquoNrW4vYCfg5okeH6CqjtuIy2cDc4Bvtb4uAC4Yj7gkSZKkfjIZMwcvBf5QVZ8dqqiqZcD3knwsycokK5IcDg99m39FknOS3JjkzJZgkOTkJD9OsjzJx1vdYJLLWt2lSZ46PIAkpyeZ28qvaP1eB7yuq83+SX6YZEmSHyR5VpKtgBOBw5MsTXJ4kmOSfHpdY7fxTmn9/GxobEmSJKmfTUZysDuweIT619H5Vn4v4OXAx5IMtHN7A+8BdgWeAbwgyROAw4DdqmpP4G9b20/RmZXYEzgTOGVtgSSZBnweeA2wL/CkrtM3Ai+qqr2B44CPVtXvW/msqppdVWcN63JdYw8ALwReDZy8tpgkSZKkftHLDckvBP6pqtZU1e3AlcB+7dw1VfXLqnoQWAoMAquB+4AvJnkdcE9r+zzg/7byV1q/a/Ns4Jaq+klVFfDVrnMzgLOTrAQ+Cew2itewrrG/UVUPVtWP6SyhGlGSeUkWJVm05p7VoxhSkiRJmhiTkRxcT+db+rG4v6u8Btiiqh4A9gfOofNt/LfHJ7yHfAS4vKp2pzOzMG0j++t+DVlbo6paWFVzqmrOlG1mbOSQkiRJ0oabjOTgMmBqknlDFUn2BH5LZy3/lCQ7Ai8GrllbJ0mmAzOq6lvAe+ksRwL4AfCGVj4SuGodsdwIDCZ5Zjs+ouvcDODWVj6mq/5OYNu19DeWsSVJkqS+NuHJQVu+cxjw8vYo0+uBk+gsx1kOLKOTQHygqv59HV1tC1yYZDnwPeB9rf6dwJtb/ZuAd68jlvuAecBFbUPyHV2n/wE4KckSHvkUp8uBXYc2JA/rctRjS5IkSf0unc/u6gdTB2bVwNELeh3GJmPVyQf3OgRJkqS+k2RxVc0Z6Zy/kCxJkiQJMDmQJEmS1JgcSJIkSQJMDiRJkiQ1W6y/iSbLHjNnsMhNtJIkSeoRZw4kSZIkASYHkiRJkhqTA0mSJEmAyYEkSZKkxg3JfWTFrasZnH9Rr8OQtBnwF8QlSSNx5kCSJEkSYHIgSZIkqTE5kCRJkgSYHEiSJElqNpvkIMmaJEu7/s0foc0BSS7sRXySJElSr21OTyu6t6pm9zoISZIkqV9tNjMHa5PkFUluTHId8Lqu+scmOS3JNUmWJDmk1R+T5BtJLkmyKsk7kryvtflRku1bu79Icm2SZUnOTbJNj16iJEmSNCqbU3Kw9bBlRYcnmQZ8HngNsC/wpK72HwIuq6r9gZcCH0vy2HZudzqJxH7A3wH3VNXewA+Bo1qb86pqv6raC7gBeOtEv0BJkiRpY2zWy4qSzAZuqaqftOOvAvPa6YOA1yY5th1PA57aypdX1Z3AnUlWA99s9SuAPVt59yR/CzwOmA5cPFJQSeYNjTllux037hVKkiRJG2FzSg7GKsDrq+qmR1Qmfwrc31X1YNfxgzz8np4OHFpVy5IcAxww0iBVtRBYCDB1YFaNU+ySJEnSmG1Oy4pGciMwmOSZ7fiIrnMXA+9MEoAke4+x722B25JsCRy50ZFKkiRJE2xzSg6G7zk4uaruo7Ok56K2IfmOrvYfAbYElie5vh2PxYeBq4Hv00lCJEmSpL6WKley9IupA7Nq4OgFvQ5D0mZg1ckH9zoESVKPJFlcVXNGOrc5zRxIkiRJWgeTA0mSJEmAyYEkSZKkxuRAkiRJEuDvHPSVPWbOYJGbBCVJktQjzhxIkiRJAkwOJEmSJDUmB5IkSZIA9xz0lRW3rmZw/kW9DkOSJEkTqJ9/iNKZA0mSJEmAyYEkSZKkxuRAkiRJEmByIEmSJKkxOZAkSZIEjDI5SHJokkry7K66Jyc5Z+JC2zhJtkqyIMm/JvlJkn9OsvM4j/HQe5BkdpJXjWf/kiRJ0mQa7czBEcD32l8Aqur/r6q5wxsm6ZfHo34U2BZ4VlXNAr4BnJck4zXAsPdgNmByIEmSpEet9SYHSaYDLwTeCryhq34wycpWPibJBUkuAy5NMpDku0mWJlmZ5EWt3V1JPpnk+iSXJtmx1f9FkmuTLEtybpJtWv1OSc5v9cuSPL/VvzHJNa3/zyWZMizmbYA3A++tqjUAVfUl4H7gwBb7jUnOTHJDknO6xjyuxbIyycKhZCLJLkn+pcVxXZJnDr0HSbYCTgQObzEd3mYrhl7fY9oMxo4b+j+UJEmSNNFGM3NwCPDtqroZ+M8k+66l3T7A3Kp6CfDnwMVVNRvYC1ja2jwWWFRVuwFXAn/T6s+rqv2qai/gBjqJCMApwJWtfh/g+iTPAQ4HXtD6XwMcOSyWXYBfVNXvhtUvAnZr5WcBn6mq5wC/A/5Hq/90i2V3YGvg1a3+TOD/tFieD9w21GlV/R44DjirqmZX1VnAV7viejmwrKr+Yy3vnSRJktRzo0kOjgC+1spfo2tp0TCXVNWvW/la4M1Jjgf2qKo7W/2DwFmt/FU6MxIAuye5KskKOh+ohz7AHwicClBVa6pqNfAyYF/g2iRL2/EzRvE6hvu3qvr+CLG8NMnVLZYDgd2SbAvMrKrzWyz3VdU96+n/NOCoVn4L8KWRGiWZl2RRkkVr7lm9AS9DkiRJGh/r3B+QZHs6H5D3SFLAFKCSvH+E5ncPFarqu0leDBwMnJ7kE1X15RGuqfb3dODQqlqW5BjggHWFBZxRVX+1jjY/BZ6aZNuuxAQ6ScWFw8Z+KJYk04DPAHOq6t9acjNtHeOsVbv+9iQHAvvzx7MbQ+0WAgsBpg7MGh6TJEmSNGnWN3MwF/hKVT2tqgar6inALcCL1nVRkqcBt1fV54Ev0FkSNDTe0AbeP6ezyRk6G4dvS7Ilj/wQfSnwl63PKUlmtLq5SZ7Y6rdv4z2kqu4GzgA+MbQfIclRwDbAZa3ZU5M8b1gsQ4nAr9pei7mtvzuBXyY5tPU1dWiPQpc72+vo9gU6sxJnD+19kCRJkvrV+pKDI4Dzh9Wdy9qXFg05AFiWZAmd/QH/u9XfDezfNjIfSGcTL8CHgauB7wM3dvXzbjrLfFYAi4Fdq+rHwP8CvpNkOXAJMDBCDH8F3AfcnOQnwH8DDquqoW/nbwLenuQG4PHAqVX1W+DzwErgYjrLo4a8CXhXG/MHwJOGjXc5sOvQhuRWdwEwnbUsKZIkSZL6SR7+rDwJgyV3VdX0SRtw7XEMAhe2TccTOc4c4JNVtc6ZliFTB2bVwNELJjIkSZIk9diqkw/u6fhJFlfVnJHO9ctvEmxyksynsyRqxL0GkiRJUr8Z7Y+gjYt+mDUAqKpVEz1rUFUnt70a31t/a0mSJKn3JjU5kCRJktS/TA4kSZIkAe456Ct7zJzBoh5vUJEkSdLmy5kDSZIkSYDJgSRJkqTG5ECSJEkS4J6DvrLi1tUMzr+o12GsV69/uEOSJEkTw5kDSZIkSYDJgSRJkqTG5ECSJEkSYHIgSZIkqTE5kCRJkgT0SXKQ5K5JHOuYJJ/eyD5OTPLy8YpJkiRJ6gc+ynQDVNVxvY5BkiRJGm99MXMwkiRXJJnTyjskWdXKxyQ5L8m3k/wkyT90XfPWJDcnuSbJ58cyQ5Dkje26pUk+l2RK+3d6kpVJViR5b2t7epK5rfyyJEva+dOSTG31q5KckOS6du7Z4/j2SJIkSeOub5OD9ZgNHA7sARye5ClJngx8GHgu8AJg1B/Gkzyn9feCqpoNrAGObOPMrKrdq2oP4EvDrpsGnA4c3s5vAfxlV5NfVdU+wKnAsWsZe16SRUkWrbln9WhDliRJksbdozU5uLSqVlfVfcCPgacB+wNXVtWvq+oPwNlj6O9lwL7AtUmWtuNnAD8DnpHkU0leAfxu2HXPAm6pqpvb8RnAi7vOn9f+LgYGRxq4qhZW1ZyqmjNlmxljCFmSJEkaX/285+ABHk5epg07d39XeQ0b/zoCnFFVf/VHJ5K9gP8KvA34M+AtY+h3KM7xiFGSJEmaUP08c7CKzrf5AHNH0f5a4CVJHk0TU50AAAksSURBVJ9kC+D1YxjrUmBukicCJNk+ydOS7AA8pqrOBf4XsM+w624CBpPs0o7fBFw5hnElSZKkvtEv32Zvk+SXXcefAD4OfD3JPOCi9XVQVbcm+ShwDfBr4EZgbYv4j0lyaNfxc+l8+P9OkscAfwDeDtwLfKnVATxiZqGq7kvyZuDslpBcC3x2fbFKkiRJ/ShV1esYxk2S6VV1V/ugfj5wWlWd3+u4RmvqwKwaOHpBr8NYr1UnH9zrECRJkrSBkiyuqjkjnevnZUUb4vi2oXglcAvwjR7HI0mSJD1q9MuyonFRVSM+LlSSJEnS+m1qMweSJEmSNpDJgSRJkiRgE1tW9Gi3x8wZLHKzryRJknrEmQNJkiRJgMmBJEmSpMbkQJIkSRLgnoO+suLW1QzOX++PQUt9wR/DkyRp0+PMgSRJkiTA5ECSJElSY3IgSZIkCTA5kCRJktRMeHKQ5ElJvpbkp0kWJ/lWkj/ZwL6OT3JsK5+Y5OXjG+3G6Y5PkiRJerSZ0KcVJQlwPnBGVb2h1e0F7ATcPIprU1UPjnS+qo4b53AlSZKkzdpEzxy8FPhDVX12qKKqlgFLklya5LokK5IcApBkMMlNSb4MrASekuRDSW5O8j3gWUP9JDk9ydxWflmSJa2v05JMbfWrkpyUZGmSRUn2SXJxm8V4W1df709ybZLlSU7oqv9wi+d7Sf6pa9biL1r7ZUnOTbJN94tO8swk13Udz+o+liRJkvrRRCcHuwOLR6i/Dzisqvahk0D8Y5spAJgFfKaqdgN2AN4AzAZeBew3vKMk04DTgcOrag86syF/2dXkF1U1G7iqtZsLPBc4oV1/UBtz/zbOvklenGQ/4PXAXsArgTldfZ5XVftV1V7ADcBbu2Oqqp8Cq5PMblVvBr609rdJkiRJ6r1e/QhagI8meTHwIDCTzlIjgJ9X1Y9a+UXA+VV1D0CSC0bo61nALVU1tEzpDODtwIJ2PHTNCmB6Vd0J3Jnk/iSPAw5q/5a0dtPpJAvbAv9cVfcB9yX5ZteYuyf5W+Bxrf3FI8T1BeDNSd4HHE4n+fjjNyKZB8wDmLLdjiM1kSRJkibFRM8cXA/sO0L9kcCOwL7tW/3bgWnt3N3jHMP97e+DXeWh4y3oJConVdXs9m+Xqvrievo8HXhHm6k4gYdj73YunRmHVwOLq+o/R+qoqhZW1ZyqmjNlmxmjflGSJEnSeJvo5OAyYGr7dhyAJHsCTwPuqKo/JHlpOx7Jd4FDk2ydZFvgNSO0uQkYTLJLO34TcOUYYrwYeEuS6S2+mUmeCHwfeE2Sae3cq7uu2Ra4LcmWdBKdP9JmHC4GTsUlRZIkSXoUmNBlRVVVSQ4DFiT5IJ29BquA44FTkqwAFgE3ruX665KcBSwD7gCuHaHNfUneDJydZIvW5rPD260jxu8keQ7ww7bt4S7gjVV1bVvGtJzOzMYKYHW77MPA1cB/tL/brqX7M4HDgO+MNh5JkiSpV1JVvY6hbyWZXlV3tacRfReYV1WjfupQe7rRjKr68GjaTx2YVQNHL1h/Q6kPrDr54F6HIEmSNkCSxVU1Z6RzvdqQ/GixMMmudPYUnDHGxOB84JnAgRMVnCRJkjSeTA7Woar+fCOuPWw8Y5EkSZIm2kRvSJYkSZL0KGFyIEmSJAlwWVFf2WPmDBa5yVOSJEk94syBJEmSJMDkQJIkSVJjciBJkiQJMDmQJEmS1JgcSJIkSQJMDiRJkiQ1JgeSJEmSAJMDSZIkSY3JgSRJkiTA5ECSJElSY3IgSZIkCTA5kCRJktSkqnodg5okdwI39ToObRJ2AH7V6yC0SfBe0njyftJ48V7aOE+rqh1HOrHFZEeidbqpqub0Ogg9+iVZ5L2k8eC9pPHk/aTx4r00cVxWJEmSJAkwOZAkSZLUmBz0l4W9DkCbDO8ljRfvJY0n7yeNF++lCeKGZEmSJEmAMweSJEmSGpODPpDkFUluSvKvSeb3Oh71vySnJbkjycquuu2TXJLkJ+3v41t9kpzS7q/lSfbpXeTqN0mekuTyJD9Ocn2Sd7d67yeNSZJpSa5JsqzdSye0+qcnubrdM2cl2arVT23H/9rOD/YyfvWfJFOSLElyYTv2XpoEJgc9lmQK8H+AVwK7Akck2bW3UelR4HTgFcPq5gOXVtUs4NJ2DJ17a1b7Nw84dZJi1KPDA8D/rKpdgecCb2//DfJ+0ljdDxxYVXsBs4FXJHku8PfAJ6tqF+A3wFtb+7cCv2n1n2ztpG7vBm7oOvZemgQmB723P/CvVfWzqvo98DXgkB7HpD5XVd8Ffj2s+hDgjFY+Azi0q/7L1fEj4HFJBiYnUvW7qrqtqq5r5Tvp/B/xTLyfNEbtnrirHW7Z/hVwIHBOqx9+Lw3dY+cAL0uSSQpXfS7JzsDBwBfacfBemhQmB703E/i3ruNftjpprHaqqtta+d+BnVrZe0yj0qbi9wauxvtJG6AtA1kK3AFcAvwU+G1VPdCadN8vD91L7fxq4AmTG7H62ALgA8CD7fgJeC9NCpMDaRNUnceQ+SgyjVqS6cC5wHuq6nfd57yfNFpVtaaqZgM705kZf3aPQ9KjUJJXA3dU1eJex7I5MjnovVuBp3Qd79zqpLG6fWh5R/t7R6v3HtM6JdmSTmJwZlWd16q9n7TBquq3wOXA8+gsPduineq+Xx66l9r5GcB/TnKo6k8vAF6bZBWd5dYHAv8b76VJYXLQe9cCs9oO/K2ANwAX9DgmPTpdABzdykcD/9xVf1R7ysxzgdVdy0W0mWvrcr8I3FBVn+g65f2kMUmyY5LHtfLWwH+hs4flcmBuazb8Xhq6x+YCl5U/viSgqv6qqnauqkE6n4suq6oj8V6aFP4IWh9I8io6a+umAKdV1d/1OCT1uST/BBwA7ADcDvwN8A3g68BTgZ8Df1ZVv24f/j5N5+lG9wBvrqpFvYhb/SfJC4GrgBU8vLb3r+nsO/B+0qgl2ZPOptApdL58/HpVnZjkGXS+/d0eWAK8saruTzIN+AqdfS6/Bt5QVT/rTfTqV0kOAI6tqld7L00OkwNJkiRJgMuKJEmSJDUmB5IkSZIAkwNJkiRJjcmBJEmSJMDkQJIkSVJjciBJkiQJMDmQJEmS1JgcSJIkSQLg/wHoUm95s3ANtAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,5))\n",
    "plt.barh(y=CATEGORIES, width=test_df[CATEGORIES].sum(axis=0))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that pathologies like Airspace Opacity, Pleural Effusion, and Edema are present in many of the reports while Lung Lesion and Pneumonia are not as common in this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "E76U0d7AVq6A"
   },
   "source": [
    "<a name=\"1-1\"></a>\n",
    "### 1.1 Text Matching\n",
    "One of the easiest and surprisingly effective ways to label our dataset is to search for presence of different keywords in the impression text. \n",
    "\n",
    "We have prepared a list of relevant keywords for each pathology for you to use in detecting the presence of each label. \n",
    "- You can access these keywords for each label by calling the `get_mention_keywords(observation)` function.\n",
    "\n",
    "Here's an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wiRnTs6BVq6B",
    "outputId": "794c7926-1d70-4473-cec1-a4844e74de43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Related keywords for Airspace Opacity are:\n",
      "opaci, decreased translucency, increased density, airspace disease, air-space disease, air space disease, infiltrate, infiltration, interstitial marking, interstitial pattern, interstitial lung, reticular pattern, reticular marking, reticulation, parenchymal scarring, peribronchial thickening, wall thickening, scar\n"
     ]
    }
   ],
   "source": [
    "cat = CATEGORIES[2]\n",
    "related_keywords = get_mention_keywords(cat)\n",
    "print(\"Related keywords for {} are:\\n{}\".format(cat, ', '.join(related_keywords)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Y4spu0NVq6D"
   },
   "source": [
    "<a name='ex-01'></a>\n",
    "### Exercise 1: Get Labels\n",
    "\n",
    "You can use this simple approach to start constructing labels for each report. Fill in the `get_labels()` function below.\n",
    "- It takes in a report (as an array of sentences)\n",
    "- It returns a dictionary that maps each category to a boolean value, which indicates the presence or absence of the abnormality.\n",
    "\n",
    "Note that in Python, the `in` keyword can be used on a string to find substrings.  For instance:\n",
    "\n",
    "```CPP\n",
    "s = 'hello how are you? I am fine.'\n",
    "if \"are you\" in s:\n",
    "    print(True)\n",
    "else:\n",
    "    print(False)\n",
    "```\n",
    "This outputs `True`, because the `in` keyword is able to find substrings.\n",
    "\n",
    "Also note that whitespace and punctuation matter.\n",
    "\n",
    "```CPP\n",
    "s = 'hello how are you? I am fine.'\n",
    "if \"areyou\" in s:\n",
    "    print(True)\n",
    "else:\n",
    "    print(False)\n",
    "```\n",
    "This outputs `False`, because there only `are you` exists in the string and not `areyou`.\n",
    "\n",
    "Also, note that the string matching is case sensitive\n",
    "```CPP\n",
    "s = 'hello how are you? I am fine.'\n",
    "if \"Hello\" in s:\n",
    "    print(True)\n",
    "else:\n",
    "    print(False)\n",
    "```\n",
    "This returns `False` because in the string, 'hello' has a lowercase 'h', so `Hello` does not exist in the string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use the get_mention_keywords() function</li>\n",
    "    <li>Make sure to make the sentence as well as the phrase all lowercase before looking for the phrase inside the sentence.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_DKhgVrPVq6D"
   },
   "outputs": [],
   "source": [
    "# UNQ_C1 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def get_labels(sentence_l):\n",
    "    \"\"\"\n",
    "    Returns a dictionary that indicates presence of each category (from CATEGORIES) \n",
    "    in the given sentences.\n",
    "    hint: loop over the sentences array and use get_mention_keywords() function.\n",
    "    \n",
    "    Args: \n",
    "        sentence_l (array of strings): array of strings representing impression section\n",
    "    Returns:\n",
    "        observation_d (dict): dictionary mapping observation from CATEGORIES array to boolean value\n",
    "    \"\"\"\n",
    "    observation_d = {}\n",
    "    ### START CODE HERE ###\n",
    "    # loop through each category\n",
    "    for cat in CATEGORIES:\n",
    "        \n",
    "        # Initialize the observations for all categories to be False\n",
    "        observation_d[cat] = False\n",
    "\n",
    "    # For each sentence in the list:\n",
    "    for s in sentence_l: # complete this line\n",
    "        \n",
    "        # Set the characters to all lowercase, for consistent string matching\n",
    "        s = s.lower()\n",
    "        \n",
    "        # for each category\n",
    "        for cat in CATEGORIES: # complete this line\n",
    "            \n",
    "            # for each phrase that is related to the keyword (use the given function)\n",
    "            for phrase in get_mention_keywords(cat): # complete this line\n",
    "            \n",
    "                # make the phrase all lowercase for consistent string matching\n",
    "                phrase = phrase.lower()\n",
    "                \n",
    "                # check if the phrase appears in the sentence\n",
    "                if phrase in s: # complete this line\n",
    "                    observation_d[cat] = True\n",
    "\n",
    "    \n",
    "    ### END CODE HERE ###\n",
    "    return observation_d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case\n",
      "\n",
      "Test Sentences:\n",
      "\n",
      "Diffuse Reticular Pattern, which can be seen with an atypical infection or chronic fibrotic change.\n",
      "no Focal Consolidation.\n",
      "\n",
      "Retrieved labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : True\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Expected labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : True\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Labels match for Cardiomegaly!\n",
      "Labels match for Lung Lesion!\n",
      "Labels match for Airspace Opacity!\n",
      "Labels match for Edema!\n",
      "Labels match for Consolidation!\n",
      "Labels match for Pneumonia!\n",
      "Labels match for Atelectasis!\n",
      "Labels match for Pneumothorax!\n",
      "Labels match for Pleural Effusion!\n",
      "Labels match for Pleural Other!\n",
      "Labels match for Fracture!\n"
     ]
    }
   ],
   "source": [
    "print(\"Test Case\")\n",
    "\n",
    "test_sentences = [\"Diffuse Reticular Pattern, which can be seen with an atypical infection or chronic fibrotic change.\", \n",
    "                  \"no Focal Consolidation.\"]\n",
    "print(\"\\nTest Sentences:\\n\")\n",
    "for s in test_sentences:\n",
    "    print(s)\n",
    "\n",
    "print()\n",
    "retrieved_labels = get_labels(test_sentences)\n",
    "print(\"Retrieved labels: \")\n",
    "\n",
    "for key, value in sorted(retrieved_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Expected labels: \")\n",
    "expected_labels = {'Cardiomegaly': False, 'Lung Lesion': False, 'Airspace Opacity': True, 'Edema': False, 'Consolidation': True, 'Pneumonia': True, 'Atelectasis': False, 'Pneumothorax': False, 'Pleural Effusion': False, 'Pleural Other': False, 'Fracture': False}\n",
    "for key, value in sorted(expected_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "for category in CATEGORIES:\n",
    "    if category not in retrieved_labels:\n",
    "        print(f'Category {category} not found in retrieved labels. Please check code.')\n",
    "    \n",
    "    elif retrieved_labels[category] == expected_labels[category]:\n",
    "        print(f'Labels match for {category}!')\n",
    "    \n",
    "    else:\n",
    "        print(f'Labels mismatch for {category}. Please check code.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Note\n",
    "\n",
    "You may have noticed that the second sentence is \"no Focal Consolidation\", and that the consolidation label is set to True.  We'll come back to this shortly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mceW_rpsVq6J"
   },
   "source": [
    "<a name=\"1-2\"></a>\n",
    "### 1.2 Evaluating The Performance\n",
    "In order to evaluate the performance of your labeler, you will use a metric called the [F1 score](https://en.wikipedia.org/wiki/F1_score). \n",
    "- The F1 score is the harmonic mean of precision and recall. \n",
    "- This score is a common metric that is used in information retrieval problems. \n",
    "\n",
    "The reason that we care both about precision and recall is that only a small subset of the labels for each report are positive and the rest are negative. \n",
    "- So a traditional metric such as accuracy could be very high if you just report every label as False. \n",
    "- Precision and recall (summarized in the F1 score) help you measure your classification's performance for both positive cases as well as negative cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$F_1 = \\left( \\frac{2}{ \\frac{1}{recall} + \\frac{1}{precision} }\\right) = 2 \\times \\frac{precision \\times recall}{precision + recall}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "a9ZgyZxpVq6J"
   },
   "source": [
    "We've implemented a utility function for you called `get_f1_table()` below to calculate your function's performance on the whole test set. \n",
    "- This function will take advantage of modules from the `bioc` and `negbio` python packages to intelligently split your paragraph to sentences and then apply your function on each sentence. \n",
    "- It then returns a table with the calculated F1 score for each category. \n",
    "- You can see how this function is implemented in `util.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Vs6cU5i_Vq6K",
    "outputId": "e2a2ed94-c302-41de-d5b1-39d47d16ce70"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.597</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.718\n",
       "1        Lung Lesion  0.641\n",
       "2   Airspace Opacity  0.923\n",
       "3              Edema  0.708\n",
       "4      Consolidation  0.270\n",
       "5          Pneumonia  0.369\n",
       "6        Atelectasis  0.646\n",
       "7       Pneumothorax  0.218\n",
       "8   Pleural Effusion  0.722\n",
       "9      Pleural Other  0.667\n",
       "10          Fracture  0.689\n",
       "11           Average  0.597"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels, test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YkHxb06BVq6N"
   },
   "source": [
    "You can see that Airspace Opacity has the highest F1 score at 0.923 while Pneumothorax has the lowest at 0.218. \n",
    "\n",
    "These numbers are actually not too bad given the simplicity of this rules-based implemention, but we can do a lot better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"1-3\"></a>\n",
    "### 1.3 Cleanup\n",
    "\n",
    "First of all, the text that you use as input is very messy. \n",
    "- You should be able to do better by doing some simple pre-processing. \n",
    "- We have already implemented the `clean` function for you, which does basic text cleanup. \n",
    "- Among other things, it converts pattern such as \"and/or\" to \"or\", replaces repeated whitespace with just one space, and removes redundant punctuation, etc. \n",
    "\n",
    "Run the following example to see how cleanup changes the input. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "raw text: \n",
      "\n",
      "\n",
      " \n",
      "1.  bibasilar opacities, without significant change from the prior \n",
      "study, likely representing bilateral pleural effusions with \n",
      "atelectasis and/or pneumonia.\n",
      " \n",
      "2.  interval decrease in focal opacification within the right upper \n",
      "lung.\n",
      " \n",
      "3.  lines and tubes are unchanged. the distal tip of enteric tube \n",
      "remains coiled within the stomach.\n",
      " \n",
      " \n",
      "\n",
      "cleaned text: \n",
      "\n",
      "1. bibasilar opacities, without significant change from the prior study, likely representing bilateral pleural effusions with atelectasis or pneumonia. 2. interval decrease in focal opacification within the right upper lung. 3. lines and tubes are unchanged. the distal tip of enteric tube remains coiled within the stomach.\n"
     ]
    }
   ],
   "source": [
    "raw_text = test_df.loc[28, 'Report Impression']\n",
    "print(\"raw text: \\n\\n\" + raw_text)\n",
    "print(\"cleaned text: \\n\\n\" + clean(raw_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the \"and/or\" in observation 1 was transformed into \"or\", and the standardization of the whitespace. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have implemented a function for this which you can add to the pipeline, by passing the `cleanup=True` flag to our `get_f1_table()` function. Let's see if cleaning up the text can improve performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EvLEUqT5Vq6N",
    "outputId": "30bee7a4-759c-4847-d785-d72bccaf0831"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.707</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.725\n",
       "1        Lung Lesion  0.641\n",
       "2   Airspace Opacity  0.922\n",
       "3              Edema  0.707\n",
       "4      Consolidation  0.270\n",
       "5          Pneumonia  0.369\n",
       "6        Atelectasis  0.646\n",
       "7       Pneumothorax  0.218\n",
       "8   Pleural Effusion  0.722\n",
       "9      Pleural Other  0.687\n",
       "10          Fracture  0.689\n",
       "11           Average  0.600"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels, test_df, cleanup=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see a very modest improvement in Cardiomegaly and Pleural Other, but overall the impact is fairly low. In the next section you'll make a change which has a much bigger impact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GrKV6fcSVq6P"
   },
   "source": [
    "<a name=\"1-4\"></a>\n",
    "### 1.4 Finding Negative Mentions\n",
    "\n",
    "<a name='ex-02'></a>\n",
    "### Exercise 2\n",
    "So far you have just been treating the presence of a keyword in the impression section as a signal for the presence of that condition in the report. This approach currently has a big problem: \n",
    "- This is ignoring negations! \n",
    "- For example, consider the following report: \"No sign of consolidation\". Currently, our matching approach calls this sentence a positive case of consolidation!\n",
    "\n",
    "Implement your `get_labels()` function one more time.\n",
    "- Use a boolean \"flag\" to indicate whether a negation like \"no\" or \"not\" appears in a sentence.\n",
    "- Only set a label to `True` if the word \"not\" or \"no\" did not appear in the sentence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use the get_mention_keywords() function.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kHHsbWolVq6Q"
   },
   "outputs": [],
   "source": [
    "# UNQ_C2 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def get_labels_negative_aware(sentence_l):\n",
    "    \"\"\"\n",
    "    Returns a dictionary that indicates presence of categories in\n",
    "    sentences within the impression section of the report.\n",
    "    Only set a label to True if no 'negative words' appeared in the sentence.\n",
    "    hint: loop over the sentences array and use get_mention_keywords() function.\n",
    "    \n",
    "    Args: \n",
    "        sentence_l (array of strings): array of strings representing impression section\n",
    "    Returns:\n",
    "        observation_d (dict): dictionary mapping observation from CATEGORIES array to boolean value\n",
    "    \"\"\"\n",
    "    # Notice that all of the negative words are written in lowercase\n",
    "    negative_word_l = [\"no\", \"not\", \"doesn't\", \"does not\", \"have not\", \"can not\", \"can't\", \"n't\"]\n",
    "    observation_d = {}\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    # Initialize the observation dictionary \n",
    "    # so that all categories are not marked present.\n",
    "    for cat in CATEGORIES: # complete this line\n",
    "        \n",
    "        # Initialize category to not present.\n",
    "        observation_d[cat] = False\n",
    "\n",
    "    # Loop through each sentence in the list of sentences\n",
    "    for s in sentence_l: # complete this line\n",
    "        \n",
    "        # make the sentence all lowercase\n",
    "        s = s.lower()\n",
    "        \n",
    "        # Initialize the flag to indicate no negative mentions (yet)\n",
    "        negative_flag = False\n",
    "        \n",
    "        # Go through all the negative words in the list\n",
    "        for neg in negative_word_l: # complete this line\n",
    "            \n",
    "            # Check if the word is a substring in the sentence\n",
    "            if neg in s: # complete this line\n",
    "                # set the flag to indicate a negative mention\n",
    "                negative_flag = True\n",
    "                \n",
    "                # Once a single negative mention is found,\n",
    "                # you can stop checking the remaining negative words\n",
    "                None # complete this line\n",
    "\n",
    "        # When a negative word was not found in the sentence,\n",
    "        # check for the presence of the diseases\n",
    "        if not negative_flag: # complete this line\n",
    "            \n",
    "            # Loop through the categories list\n",
    "            for cat in CATEGORIES: # complete this line\n",
    "                \n",
    "                # Loop through each phrase that indicates this category\n",
    "                for phrase in get_mention_keywords(cat): # complete this line\n",
    "\n",
    "                        # make the phrase all lowercase\n",
    "                        phrase = phrase.lower()\n",
    "                        \n",
    "                        # Check if the phrase is a substring in the sentence\n",
    "                        if phrase in s: # complete this line\n",
    "                            \n",
    "                            # Set the observation dictionary\n",
    "                            # to indicate the presence of this category\n",
    "                            observation_d[cat] = True\n",
    "\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return observation_d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case\n",
      "\n",
      "Test Sentences:\n",
      "\n",
      "Diffuse Reticular pattern, which can be seen with an atypical infection or chronic fibrotic change.\n",
      "No Focal Consolidation.\n",
      "\n",
      "Retrieved labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : False\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Expected labels: \n",
      "Airspace Opacity : True\n",
      "Atelectasis : False\n",
      "Cardiomegaly : False\n",
      "Consolidation : False\n",
      "Edema : False\n",
      "Fracture : False\n",
      "Lung Lesion : False\n",
      "Pleural Effusion : False\n",
      "Pleural Other : False\n",
      "Pneumonia : True\n",
      "Pneumothorax : False\n",
      "\n",
      "Test Results:\n",
      "Labels match for Cardiomegaly!\n",
      "Labels match for Lung Lesion!\n",
      "Labels match for Airspace Opacity!\n",
      "Labels match for Edema!\n",
      "Labels match for Consolidation!\n",
      "Labels match for Pneumonia!\n",
      "Labels match for Atelectasis!\n",
      "Labels match for Pneumothorax!\n",
      "Labels match for Pleural Effusion!\n",
      "Labels match for Pleural Other!\n",
      "Labels match for Fracture!\n"
     ]
    }
   ],
   "source": [
    "print(\"Test Case\")\n",
    "\n",
    "test_sentences = [\"Diffuse Reticular pattern, which can be seen with an atypical infection or chronic fibrotic change.\", \n",
    "                  \"No Focal Consolidation.\"]\n",
    "print(\"\\nTest Sentences:\\n\")\n",
    "for s in test_sentences:\n",
    "    print(s)\n",
    "\n",
    "print()\n",
    "retrieved_labels = get_labels_negative_aware(test_sentences)\n",
    "print(\"Retrieved labels: \")\n",
    "\n",
    "for key, value in sorted(retrieved_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Expected labels: \")\n",
    "expected_labels = {'Cardiomegaly': False, 'Lung Lesion': False, 'Airspace Opacity': True, 'Edema': False, 'Consolidation': False, 'Pneumonia': True, 'Atelectasis': False, 'Pneumothorax': False, 'Pleural Effusion': False, 'Pleural Other': False, 'Fracture': False}\n",
    "for key, value in sorted(expected_labels.items(), key=lambda x: x[0]): \n",
    "    print(\"{} : {}\".format(key, value))\n",
    "print()\n",
    "\n",
    "print(\"Test Results:\")\n",
    "for category in CATEGORIES:\n",
    "    if category not in retrieved_labels:\n",
    "        print(f'Category {category} not found in retrieved labels. Please check code.')\n",
    "    \n",
    "    elif retrieved_labels[category] == expected_labels[category]:\n",
    "        print(f'Labels match for {category}!')\n",
    "    \n",
    "    else:\n",
    "        print(f'Labels mismatch for {category}. Please check code.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you implemented this correctly, you should have Consolidation set to **False**, because the test sentence contains \"No Focal Consolidation.  \n",
    "\n",
    "With the basic labeling method `get_labels()`, this set Consolidation to True, because it didn't look for 'negative' words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r72mTZ4TVq6w"
   },
   "source": [
    "Check how this changes your aggregate performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZFUKz7ObVq6x",
    "outputId": "4e192adc-028a-4b53-a000-d51e84e75ec6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.463</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.682</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.807\n",
       "1        Lung Lesion  0.465\n",
       "2   Airspace Opacity  0.905\n",
       "3              Edema  0.795\n",
       "4      Consolidation  0.424\n",
       "5          Pneumonia  0.463\n",
       "6        Atelectasis  0.642\n",
       "7       Pneumothorax  0.656\n",
       "8   Pleural Effusion  0.859\n",
       "9      Pleural Other  0.621\n",
       "10          Fracture  0.864\n",
       "11           Average  0.682"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_f1_table(get_labels_negative_aware, test_df, cleanup=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7JoYejfYVq6z"
   },
   "source": [
    "You should see a generally significant boost in the F1 score across the board. \n",
    "- With the `get_labels()` method, Pneumothorax has an F1 score of 0.218.\n",
    "- With the `get_labels_negative_aware()` method, Pneumothorax has an F1 score of 0.656."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fun Exercise\n",
    "- Try to print some examples for which our rule does not perform well. \n",
    "- Can you add other negative words to the list of negative words in order to make the average F1 score better?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section, you'll see how to use some more sophisticated NLP tools to do even better!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FLNojinBVq61"
   },
   "source": [
    "<a name=\"1-5\"></a>\n",
    "### 1.5 Dependency Parsing\n",
    "Our heuristic for detecting negation is still relatively simple and you can think of some examples that would fool it. To improve performance even further, you'll leverage a more sophisticated approach using a [dependency parser](https://nlp.stanford.edu/software/nndep.html). \n",
    "\n",
    "<img src=\"nndep-example.png\">\n",
    "\n",
    "[Image Credit](https://nlp.stanford.edu/software/nndep.html)\n",
    "\n",
    "A dependency parser extracts the underlying structure of the text, vizualized in the tree diagram above. \n",
    "- Using a dependency parser, you can detect the relationship between different words in the sentence and understand which word is affected by negative phrases. \n",
    "\n",
    "Implementations of dependency parsers are very complex, but luckily there are some great off-the-shelf tools to do this. \n",
    "- One example is [NegBio](https://github.com/ncbi-nlp/NegBio), a package specifically designed for finding negative and uncertain mentions in X-ray radiology reports. \n",
    "- In addition to detecting negations, `negbio` can be configured to use a dependency parser that has been specifically trained for biomedical text. \n",
    "- This increases our performance given that biomedical text is full of acronyms, nomenclature and medical jargon.\n",
    "\n",
    "We've configured `negbio` to:\n",
    "1. Parse sentences using the Bllip parser trained using [David McClosky’s biomedical model](https://nlp.stanford.edu/~mcclosky/biomedical.html)\n",
    "2. Compute the universal dependency graph of each sentence using Stanford [CoreNLP](https://stanfordnlp.github.io/CoreNLP/)\n",
    "3. Use a collection of dependency graph rules to detect the negative or positive presence of a phrase based on configurable patterns. It also comes packaged with the negation and uncertainty rules from the CheXpert project.\n",
    "\n",
    "The catch with this more involved method is that it takes a while (~1.5h on a fast laptop) to run the detection pipeline on the 1000-sample dataset. \n",
    "- Hence, you can do your comparison on a smaller subset of the data (200 samples). \n",
    "\n",
    "Run the following cells to get predictions using the `negbio` engine \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vjyK4jdHVq61"
   },
   "outputs": [],
   "source": [
    "sampled_test = test_df.sample(200,random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code for applying the dependency is implemented for you with the function `get_negbio_preds`. \n",
    "- If you're interested, you can take a look at the implementation in the `utils` file. \n",
    "\n",
    "Run the next cell to extract predictions from the sampled test set. \n",
    "\n",
    "### Note \n",
    "- This should take about **5 minutes** to run.\n",
    "- If you want to run it again, you may need to restart the kernel (close and re-open the assignment). \n",
    "    - If you run it a second time without restarting the kernel you'll get an error message like\n",
    "    \n",
    "    ```RuntimeError: Parser is already loaded and can only be loaded once.```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XRQ9PvZOVq7p",
    "outputId": "8d34d299-d641-4f60-a497-17115efaad89"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 200/200 [05:28<00:00,  1.64s/it]\n"
     ]
    }
   ],
   "source": [
    "negbio_preds = get_negbio_preds(sampled_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's calculate the new F1 scores to see the dependency parser does. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JcSCLZqFVq7r"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.696</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label     F1\n",
       "0       Cardiomegaly  0.877\n",
       "1        Lung Lesion  0.889\n",
       "2   Airspace Opacity  0.952\n",
       "3              Edema  0.808\n",
       "4      Consolidation  0.370\n",
       "5          Pneumonia  0.467\n",
       "6        Atelectasis  0.725\n",
       "7       Pneumothorax  0.326\n",
       "8   Pleural Effusion  0.875\n",
       "9      Pleural Other  0.588\n",
       "10          Fracture  0.783\n",
       "11           Average  0.696"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_f1(sampled_test, negbio_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QKSLuA68Vq7t"
   },
   "source": [
    "And finally, let's compare all methods side by side!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9rnihcg8Vq7u"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>F1 Basic</th>\n",
       "      <th>F1 Cleaned</th>\n",
       "      <th>F1 Negative Basic</th>\n",
       "      <th>F1 Negbio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Cardiomegaly</td>\n",
       "      <td>0.758</td>\n",
       "      <td>0.771</td>\n",
       "      <td>0.815</td>\n",
       "      <td>0.877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lung Lesion</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.333</td>\n",
       "      <td>0.889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airspace Opacity</td>\n",
       "      <td>0.948</td>\n",
       "      <td>0.948</td>\n",
       "      <td>0.922</td>\n",
       "      <td>0.952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Edema</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.759</td>\n",
       "      <td>0.771</td>\n",
       "      <td>0.808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Consolidation</td>\n",
       "      <td>0.260</td>\n",
       "      <td>0.260</td>\n",
       "      <td>0.409</td>\n",
       "      <td>0.370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pneumonia</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.359</td>\n",
       "      <td>0.467</td>\n",
       "      <td>0.467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Atelectasis</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.579</td>\n",
       "      <td>0.725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pneumothorax</td>\n",
       "      <td>0.167</td>\n",
       "      <td>0.167</td>\n",
       "      <td>0.600</td>\n",
       "      <td>0.326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Pleural Effusion</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.730</td>\n",
       "      <td>0.839</td>\n",
       "      <td>0.875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Pleural Other</td>\n",
       "      <td>0.588</td>\n",
       "      <td>0.556</td>\n",
       "      <td>0.533</td>\n",
       "      <td>0.588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Fracture</td>\n",
       "      <td>0.741</td>\n",
       "      <td>0.741</td>\n",
       "      <td>0.889</td>\n",
       "      <td>0.783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.607</td>\n",
       "      <td>0.605</td>\n",
       "      <td>0.651</td>\n",
       "      <td>0.696</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Label  F1 Basic  F1 Cleaned  F1 Negative Basic  F1 Negbio\n",
       "0       Cardiomegaly     0.758       0.771              0.815      0.877\n",
       "1        Lung Lesion     0.762       0.762              0.333      0.889\n",
       "2   Airspace Opacity     0.948       0.948              0.922      0.952\n",
       "3              Edema     0.759       0.759              0.771      0.808\n",
       "4      Consolidation     0.260       0.260              0.409      0.370\n",
       "5          Pneumonia     0.359       0.359              0.467      0.467\n",
       "6        Atelectasis     0.605       0.605              0.579      0.725\n",
       "7       Pneumothorax     0.167       0.167              0.600      0.326\n",
       "8   Pleural Effusion     0.730       0.730              0.839      0.875\n",
       "9      Pleural Other     0.588       0.556              0.533      0.588\n",
       "10          Fracture     0.741       0.741              0.889      0.783\n",
       "11           Average     0.607       0.605              0.651      0.696"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "basic = get_f1_table(get_labels, sampled_test).rename(columns={\"F1\": \"F1 Basic\"})\n",
    "clean_basic = get_f1_table(get_labels, sampled_test, cleanup=True).rename(columns={\"F1\": \"F1 Cleaned\"})\n",
    "negated_basic = get_f1_table(get_labels_negative_aware, sampled_test, cleanup=True).rename(columns={\"F1\": \"F1 Negative Basic\"})\n",
    "negated_negbio = calculate_f1(sampled_test, negbio_preds).rename(columns={\"F1\": \"F1 Negbio\"})\n",
    "\n",
    "joined_preds = basic.merge(clean_basic, on=\"Label\")\n",
    "joined_preds = joined_preds.merge(negated_basic, on=\"Label\")\n",
    "joined_preds = joined_preds.merge(negated_negbio,  on=\"Label\")\n",
    "\n",
    "joined_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see an improvement using the heavier NLP machinery. The F1 for categories, such as Airspace Opacity, Cardiomegaly, Edema, and Pleural Effusion are already fairly high, while others leave something to be desired. To see how you can extend this to improve performance even more, you can take a look at the CheXpert Labeller paper [here]( https://arxiv.org/pdf/1901.07031.pdf). \n",
    "\n",
    "When you're ready, move on to the next section to explore AI techniques for querying raw text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u5zj8Lc7V3P9"
   },
   "source": [
    "<a name=\"2\"></a>\n",
    "## 2 Question Answering Using BERT "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous section, you looked at extracting disease labels from radiology reports. That could use a rule-based system because you asked a limited kind of question: \"Is the disease present?\". \n",
    "\n",
    "What if you want to support any question a physician might want to ask? To do this, you'll have to use more recent artificial intelligence techniques and large datasets. \n",
    "- In this section, you'll walk you through the pre- and post-processing involved in applying [BERT](https://github.com/google-research/bert) to the problem of question answering. \n",
    "- After developing this infrastructure, you'll use the model to answer questions from clinical notes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing question answering can take a few steps, even using pretrained models. \n",
    "- First retrieve our model and tokenizer. Recall from the lessons that the tokenizer prepares the input, mapping each word to a unique element in the vocabulary and inserting special tokens. \n",
    "- Then, the model processes these tokenized inputs to create valuable embeddings and performs tasks such as question answering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load The Tokenizer\n",
    "\n",
    "You will use the tokenizer in the next function that you implement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"./models\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hhAd2kkcbDHa"
   },
   "source": [
    "<a name=\"2-1\"></a>\n",
    "### 2.1 Roadmap\n",
    "\n",
    "The following function takes in a question and a passage, as well as some hyperparameters, then outputs the answer (according to our model).  You will first implemenet helper functions that will be called by `get_model_answer` and then complete this function using the helper functions.\n",
    "\n",
    "```CPP\n",
    "def get_model_answer(model, question, passage, tokenizer, max_seq_length=384):\n",
    "    \"\"\"\n",
    "    # prepare input\n",
    "    ...\n",
    "        \n",
    "    # get scores for start of answer and end of answer\n",
    "    ...\n",
    "    # using scores, get most likely answer\n",
    "    ...\n",
    "    \n",
    "    # using span start and end, construct answer as string\n",
    "    ...\n",
    "    return answer\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-Ius9KDaXXZk"
   },
   "source": [
    "<a name='2-2'></a>\n",
    "### 2.2 Preparing The Input\n",
    "\n",
    "<a name='ex-03'></a>\n",
    "### Exercise 3: Prepare BERT Input\n",
    "\n",
    "Your first task will be the prepare the raw passage and question for input into the model. \n",
    "\n",
    "Given the strings `p` and `q`, you want to turn them into an input of the following form: \n",
    "\n",
    "`[CLS]` `[q_token1]`, `[q_token2]`, ..., `[SEP]` `[p_token1]`, `[p_token2]`, ...\n",
    "\n",
    "Here, the special characters `[CLS]` and `[SEP]` let the model know which part of the input is the question and which is the answer. \n",
    "- The question appears between `[CLS]` and `[SEP]`.\n",
    "- The answer appears after `[SEP]`\n",
    "\n",
    "You'll also pad the input to the max input length, since BERT takes in a fixed-length input.\n",
    "\n",
    "Fill in the function below to prepare input to BERT. You'll return three items. \n",
    "- First is `input_ids`, which holds the numerical ids of each token. \n",
    "- Second, you'll output the `input_mask`, which has 1's in parts of the input tensor representing input tokens, and 0's where there is padding. \n",
    "- Finally, you'll output `tokens`, the output of the tokenizer (including the `[CLS]` and `[SEP]` tokens). You can see exactly what is expected in the test case below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>The format of the tokens will be <i>[CLS]&lt;question_tokens&gt;[SEP]&lt;answer_tokens&gt;</i></li>\n",
    "    <li>To generate a list that repeats an item, such as 'a', you can use ['a'] * 4 to get ['a', 'a', 'a', 'a'].</li>\n",
    "    <li>To create padding, generate a list of zeros: [0,0,...0] and add it to the list that you are padding.</li>\n",
    "    <li>The number of zeros to pad is the max sequence length minus the length of the list before padding.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a2ivYwfIXPHF"
   },
   "outputs": [],
   "source": [
    "# UNQ_C3 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def prepare_bert_input(question, passage, tokenizer, max_seq_length=384):\n",
    "    \"\"\"\n",
    "    Prepare question and passage for input to BERT. \n",
    "\n",
    "    Args:\n",
    "        question (string): question string\n",
    "        passage (string): passage string where answer should lie\n",
    "        tokenizer (Tokenizer): used for transforming raw string input\n",
    "        max_seq_length (int): length of BERT input\n",
    "    \n",
    "    Returns:\n",
    "        input_ids (tf.Tensor): tensor of size (1, max_seq_length) which holds\n",
    "                               ids of tokens in input\n",
    "        input_mask (list): list of length max_seq_length of 1s and 0s with 1s\n",
    "                           in indices corresponding to input tokens, 0s in\n",
    "                           indices corresponding to padding\n",
    "        tokens (list): list of length of actual string tokens corresponding to input_ids\n",
    "    \"\"\"\n",
    "    # tokenize question\n",
    "    question_tokens = tokenizer.tokenize(question)\n",
    "    \n",
    "    # tokenize passage\n",
    "    passage_token = tokenizer.tokenize(passage)\n",
    "\n",
    "    # get special tokens \n",
    "    CLS = tokenizer.cls_token\n",
    "    SEP = tokenizer.sep_token\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    # manipulate tokens to get input in correct form (not adding padding yet)\n",
    "    # CLS {question_tokens} SEP {answer_tokens} \n",
    "    # This should be a list of tokens\n",
    "    tokens = [CLS] + question_tokens + [SEP] + passage_token\n",
    "\n",
    "    \n",
    "    # Convert tokens into integer IDs.\n",
    "    input_ids = tokenizer.convert_tokens_to_ids(tokens)\n",
    "    \n",
    "    # Create an input mask which has integer 1 for each token in the 'tokens' list\n",
    "    input_mask = [1] * len(input_ids)\n",
    "\n",
    "    # pad input_ids with 0s until it is the max_seq_length\n",
    "    # Create padding for input_ids by creating a list of zeros [0,0,...0]\n",
    "    # Add the padding to input_ids so that its length equals max_seq_length\n",
    "    input_ids = input_ids + [0] * (max_seq_length - len(input_ids))\n",
    "    \n",
    "    # Do the same to pad the input_mask so its length is max_seq_length\n",
    "    input_mask = input_mask + [0] * (max_seq_length - len(input_mask))\n",
    "\n",
    "    # END CODE HERE\n",
    "\n",
    "    return tf.expand_dims(tf.convert_to_tensor(input_ids), 0), input_mask, tokens  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "p_MTOZnDaH8S"
   },
   "source": [
    "You can test by running it on your sample input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 411
    },
    "colab_type": "code",
    "id": "ZaM_uXl6Z9Gl",
    "outputId": "e6a6a48b-f5fb-45b4-ef77-e9b4540b19ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Case:\n",
      "\n",
      "Passage: My name is Bob.\n",
      "Question: What is my name?\n",
      "\n",
      "Tokens:\n",
      "['[CLS]', 'What', 'is', 'my', 'name', '?', '[SEP]', 'My', 'name', 'is', 'Bob', '.']\n",
      "\n",
      "Corresponding input IDs:\n",
      "tf.Tensor(\n",
      "[[ 101 1327 1110 1139 1271  136  102 1422 1271 1110 3162  119    0    0\n",
      "     0    0    0    0    0    0]], shape=(1, 20), dtype=int32)\n",
      "\n",
      "Mask:\n",
      "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "passage = \"My name is Bob.\"\n",
    "\n",
    "question = \"What is my name?\"\n",
    "\n",
    "input_ids, input_mask, tokens = prepare_bert_input(question, passage, tokenizer, 20)\n",
    "print(\"Test Case:\\n\")\n",
    "print(\"Passage: {}\".format(passage))\n",
    "print(\"Question: {}\".format(question))\n",
    "print()\n",
    "print(\"Tokens:\")\n",
    "print(tokens)\n",
    "print(\"\\nCorresponding input IDs:\")\n",
    "print(input_ids)\n",
    "print(\"\\nMask:\")\n",
    "print(input_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "```CPP\n",
    "Passage: My name is Bob.\n",
    "Question: What is my name?\n",
    "Tokens:\n",
    "['[CLS]', 'What', 'is', 'my', 'name', '?', '[SEP]', 'My', 'name', 'is', 'Bob', '.']\n",
    "\n",
    "Corresponding input IDs:\n",
    "tf.Tensor(\n",
    "[[ 101 1327 1110 1139 1271  136  102 1422 1271 1110 3162  119    0    0\n",
    "     0    0    0    0    0    0]], shape=(1, 20), dtype=int32)\n",
    "\n",
    "Mask:\n",
    "[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8g_vHX3Ra0Ge"
   },
   "source": [
    "<a name='2-3'></a>\n",
    "### 2.3 Getting Answer From Model Output\n",
    "\n",
    "<a name='ex-04'></a>\n",
    "### Exercise 4: Get Span From Scores\n",
    "\n",
    "After taking in the tokenized input, the model outputs two vectors. \n",
    "- The first vector contains the scores (more formally, logits) for the starting index of the answer. \n",
    "    - A higher score means that index is more likely to be the start of the answer span in the passage. \n",
    "- The second vector contains the score for the end index of the answer. \n",
    "\n",
    "You want to output the span that maximizes the start score and end score. \n",
    "- To be valid, the start index has to occur before the end index. Formally, you want to find:\n",
    "\n",
    "$$\\arg\\max_{i <= j, mask_i=1, mask_j = 1} start\\_scores[i] + end\\_scores[j]$$\n",
    "- In words, this formulas is saying, calculate the sum and start scores of start position 'i' and end position 'j', given the constraint that the start 'i' is either before or at the end position 'j'; then find the positions 'i' and 'j' where this sum is the highest.\n",
    "- Furthermore, you want to make sure that $i$ and $j$ are in the relevant parts of the input (i.e. where `input_mask` equals 1.)\n",
    "\n",
    "Fill in the following function to get the best start and end locations given the start and end scores and the input mask. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    If you're more comfortable manipulating data with numpy, then here are some methods you might find useful:\n",
    "    <li><i>{tensor}.numpy()</i> - converts a tf.Tensor object to a numpy array</li>\n",
    "    <li><i>np.argmax()</i> - finds the index of the maximum value in a numpy array</li>\n",
    "    \n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d66oFFx1aXIf"
   },
   "outputs": [],
   "source": [
    "# UNQ_C4 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def get_span_from_scores(start_scores, end_scores, input_mask, verbose=False):\n",
    "    \"\"\"\n",
    "    Find start and end indices that maximize sum of start score\n",
    "    and end score, subject to the constraint that start is before end\n",
    "    and both are valid according to input_mask.\n",
    "\n",
    "    Args:\n",
    "        start_scores (list): contains scores for start positions, shape (1, n)\n",
    "        end_scores (list): constains scores for end positions, shape (1, n)\n",
    "        input_mask (list): 1 for valid positions and 0 otherwise\n",
    "    \"\"\"\n",
    "    n = len(start_scores)\n",
    "    max_start_i = -1\n",
    "    max_end_j = -1\n",
    "    max_start_score = -np.inf\n",
    "    max_end_score = -np.inf\n",
    "    max_sum = -np.inf\n",
    "    \n",
    "    # Find i and j that maximizes start_scores[i] + end_scores[j]\n",
    "    # so that i <= j and input_mask[i] == input_mask[j] == 1\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    # set the range for i\n",
    "    for i in range(n): # complete this line\n",
    "        \n",
    "        # set the range for j\n",
    "        for j in range(i, n): #complete this line\n",
    "\n",
    "            # both input masks should be 1\n",
    "            if (input_mask[i] == 1) & (input_mask[j] == 1): # complete this line\n",
    "                \n",
    "                # check if the sum of the start and end scores is greater than the previous max sum\n",
    "                if start_scores[i] + end_scores[j] > max_sum: # complete this line\n",
    "\n",
    "                    # calculate the new max sum\n",
    "                    max_sum = start_scores[i] + end_scores[j]\n",
    "        \n",
    "                    # save the index of the max start score\n",
    "                    max_start_i = i\n",
    "                \n",
    "                    # save the index for the max end score\n",
    "                    max_end_j = j\n",
    "                    \n",
    "                    # save the value of the max start score\n",
    "                    max_start_val = start_scores[i]\n",
    "                    \n",
    "                    # save the value of the max end score\n",
    "                    max_end_val = end_scores[j]\n",
    "                                        \n",
    "    ### END CODE HERE ###\n",
    "    if verbose:\n",
    "        print(f\"max start is at index i={max_start_i} and score {max_start_val}\")\n",
    "        print(f\"max end is at index i={max_end_j} and score {max_end_val}\")\n",
    "        print(f\"max start + max end sum of scores is {max_sum}\")\n",
    "    return max_start_i, max_end_j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xoa678SU5sLQ"
   },
   "source": [
    "We can test this out on the following sample start scores and end scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "FUQGHgir5wwl",
    "outputId": "2c540701-7b55-47dd-db58-15d601e83c8b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max start is at index i=1 and score 2.0\n",
      "max end is at index i=4 and score 4.0\n",
      "max start + max end sum of scores is 6.0\n",
      "Expected: (1, 4) \n",
      "Returned: (1, 4)\n"
     ]
    }
   ],
   "source": [
    "start_scores = tf.convert_to_tensor([-1, 2, 0.4, -0.3, 0, 8, 10, 12], dtype=float)\n",
    "end_scores = tf.convert_to_tensor([5, 1, 1, 3, 4, 10, 10, 10], dtype=float)\n",
    "input_mask = [1, 1, 1, 1, 1, 0, 0, 0]\n",
    "\n",
    "start, end = get_span_from_scores(start_scores, end_scores, input_mask, verbose=True)\n",
    "\n",
    "print(\"Expected: (1, 4) \\nReturned: ({}, {})\".format(start, end))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "\n",
    "```CPP\n",
    "max start is at index i=1 and score 2.0\n",
    "max end is at index i=4 and score 4.0\n",
    "max start + max end sum of scores is 6.0\n",
    "Expected: (1, 4) \n",
    "Returned: (1, 4)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max start is at index i=1 and score 2.0\n",
      "max end is at index i=1 and score 5.0\n",
      "max start + max end sum of scores is 7.0\n",
      "Expected: (1, 1) \n",
      "Returned: (1, 1)\n"
     ]
    }
   ],
   "source": [
    "# Test 2\n",
    "\n",
    "start_scores = tf.convert_to_tensor([0, 2, -1, 0.4, -0.3, 0, 8, 10, 12], dtype=float)\n",
    "end_scores = tf.convert_to_tensor([0, 5, 1, 1, 3, 4, 10, 10, 10], dtype=float)\n",
    "input_mask = [1, 1, 1, 1, 1, 0, 0, 0, 0 ]\n",
    "\n",
    "start, end = get_span_from_scores(start_scores, end_scores, input_mask, verbose=True)\n",
    "\n",
    "print(\"Expected: (1, 1) \\nReturned: ({}, {})\".format(start, end))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "```CPP\n",
    "max start is at index i=1 and score 2.0\n",
    "max end is at index i=1 and score 5.0\n",
    "max start + max end sum of scores is 7.0\n",
    "Expected: (1, 1) \n",
    "Returned: (1, 1)\n",
    "```\n",
    "If your expected output differes in this second test, please check how you set the range of your for loops."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ILR_O0xh6zoJ"
   },
   "source": [
    "<a name='ex-05'></a>\n",
    "### Exercise 05: Post-Processing\n",
    "\n",
    "Finally, we'll add some post-processing to get the final string. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C5 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)\n",
    "def construct_answer(tokens):\n",
    "    \"\"\"\n",
    "    Combine tokens into a string, remove some hash symbols, and leading/trailing whitespace.\n",
    "    Args:\n",
    "        tokens: a list of tokens (strings)\n",
    "    \n",
    "    Returns:\n",
    "        out_string: the processed string.\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE (REPLACE INSTANCES OF 'None' with your code) ###\n",
    "    \n",
    "    # join the tokens together with whitespace\n",
    "    out_string = ' '.join(tokens)\n",
    "    \n",
    "    # replace ' ##' with empty string\n",
    "    out_string = out_string.replace(' ##', '')\n",
    "    \n",
    "    # remove leading and trailing whitespace\n",
    "    out_string = out_string.strip()\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    # if there is an '@' symbol in the tokens, remove all whitespace\n",
    "    if '@' in tokens:\n",
    "        out_string = out_string.replace(' ', '')\n",
    "\n",
    "    return out_string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmp_out_string_1: hello how  are  you?, length 20\n",
      "tmp_out_string_2: @hellohowareyou?, length 16\n"
     ]
    }
   ],
   "source": [
    "# Test\n",
    "\n",
    "tmp_tokens_1 = [' ## hello', 'how ', 'are ', 'you?      ']\n",
    "tmp_out_string_1 = construct_answer(tmp_tokens_1)\n",
    "\n",
    "print(f\"tmp_out_string_1: {tmp_out_string_1}, length {len(tmp_out_string_1)}\")\n",
    "\n",
    "\n",
    "tmp_tokens_2 = ['@',' ## hello', 'how ', 'are ', 'you?      ']\n",
    "tmp_out_string_2 = construct_answer(tmp_tokens_2)\n",
    "print(f\"tmp_out_string_2: {tmp_out_string_2}, length {len(tmp_out_string_2)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Expected output\n",
    "```CPP\n",
    "tmp_out_string_1: hello how  are  you?, length 20\n",
    "tmp_out_string_2: @hellohowareyou?, length 16\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2-4'></a>\n",
    "### 2.4 Putting It All Together\n",
    "The `get_model_answer` function takes all the functions that you've implemented and performs question-answering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First load the pre-trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = TFAutoModelForQuestionAnswering.from_pretrained(\"./models\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Model Answer\n",
    "\n",
    "Using the helper functions that you implemented, fill out the get_model_answer function to create your question-answering system.\n",
    "- This function has been completed for you, but we recommend reviewing the code to understand what it's doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_model_answer(model, question, passage, tokenizer, max_seq_length=384):\n",
    "    \"\"\"\n",
    "    Identify answer in passage for a given question using BERT. \n",
    "\n",
    "    Args:\n",
    "        model (Model): pretrained Bert model which we'll use to answer questions\n",
    "        question (string): question string\n",
    "        passage (string): passage string\n",
    "        tokenizer (Tokenizer): used for preprocessing of input\n",
    "        max_seq_length (int): length of input for model\n",
    "        \n",
    "    Returns:\n",
    "        answer (string): answer to input question according to model\n",
    "    \"\"\" \n",
    "    # prepare input: use the function prepare_bert_input\n",
    "    input_ids, input_mask, tokens = prepare_bert_input(question, passage, tokenizer, max_seq_length)\n",
    "    \n",
    "    # get scores for start of answer and end of answer\n",
    "    # use the model returned by TFAutoModelForQuestionAnswering.from_pretrained(\"./models\")\n",
    "    # pass in in the input ids that are returned by prepare_bert_input\n",
    "    start_scores, end_scores = model(input_ids)\n",
    "    \n",
    "    # start_scores and end_scores will be tensors of shape [1,max_seq_length]\n",
    "    # To pass these into get_span_from_scores function, \n",
    "    # take the value at index 0 to get a tensor of shape [max_seq_length]\n",
    "    start_scores = start_scores[0]\n",
    "    end_scores = end_scores[0]\n",
    "    \n",
    "    # using scores, get most likely answer\n",
    "    # use the get_span_from_scores function\n",
    "    span_start, span_end = get_span_from_scores(start_scores, end_scores, input_mask)\n",
    "    \n",
    "    # Using array indexing to get the tokens from the span start to span end (including the span_end)\n",
    "    answer_tokens = tokens[span_start:span_end+1]\n",
    "    \n",
    "    # Combine the tokens into a single string and perform post-processing\n",
    "    # use construct_answer\n",
    "    answer = construct_answer(answer_tokens)\n",
    "    \n",
    "    return answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W2nXup8d68y1"
   },
   "source": [
    "<a name='2-5'></a>\n",
    "### 2.5 Try It Out\n",
    "\n",
    "Now that we've prepared all the pieces, let's try an example from the SQuAD dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output: Computational complexity theory\n",
      "Expected: Computational complexity theory\n"
     ]
    }
   ],
   "source": [
    "passage = \"Computational complexity theory is a branch of the theory \\\n",
    "           of computation in theoretical computer science that focuses \\\n",
    "           on classifying computational problems according to their inherent \\\n",
    "           difficulty, and relating those classes to each other. A computational \\\n",
    "           problem is understood to be a task that is in principle amenable to \\\n",
    "           being solved by a computer, which is equivalent to stating that the \\\n",
    "           problem may be solved by mechanical application of mathematical steps, \\\n",
    "           such as an algorithm.\"\n",
    "\n",
    "question = \"What branch of theoretical computer science deals with broadly \\\n",
    "            classifying computational problems by difficulty and class of relationship?\"\n",
    "\n",
    "print(\"Output: {}\".format(get_model_answer(model, question, passage, tokenizer)))\n",
    "print(\"Expected: Computational complexity theory\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output: pharma\n",
      "Expected: pharma\n"
     ]
    }
   ],
   "source": [
    "passage = \"The word pharmacy is derived from its root word pharma which was a term used since \\\n",
    "           the 15th–17th centuries. However, the original Greek roots from pharmakos imply sorcery \\\n",
    "           or even poison. In addition to pharma responsibilities, the pharma offered general medical \\\n",
    "           advice and a range of services that are now performed solely by other specialist practitioners, \\\n",
    "           such as surgery and midwifery. The pharma (as it was referred to) often operated through a \\\n",
    "           retail shop which, in addition to ingredients for medicines, sold tobacco and patent medicines. \\\n",
    "           Often the place that did this was called an apothecary and several languages have this as the \\\n",
    "           dominant term, though their practices are more akin to a modern pharmacy, in English the term \\\n",
    "           apothecary would today be seen as outdated or only approproriate if herbal remedies were on offer \\\n",
    "           to a large extent. The pharmas also used many other herbs not listed. The Greek word Pharmakeia \\\n",
    "           (Greek: φαρμακεία) derives from pharmakon (φάρμακον), meaning 'drug', 'medicine' (or 'poison').\"\n",
    "\n",
    "question = \"What word is the word pharmacy taken from?\"\n",
    "\n",
    "print(\"Output: {}\".format(get_model_answer(model, question, passage, tokenizer)))\n",
    "print(\"Expected: pharma\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try it on clinical notes. Below we have an excerpt of a doctor's notes for a patient with an abnormal echocardiogram (this sample is taken from [here](https://www.mtsamples.com/site/pages/sample.asp?Type=6-Cardiovascular%20/%20Pulmonary&Sample=1597-Abnormal%20Echocardiogram))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question 1: How old is the patient?\n",
      "\n",
      "Answer: 86\n",
      "\n",
      "\n",
      "Question 2: Does the patient have any complaints?\n",
      "\n",
      "Answer: The patient complains of shortness of breath\n",
      "\n",
      "\n",
      "Question 3: What is the reason for this consultation?\n",
      "\n",
      "Answer: further evaluation\n",
      "\n",
      "\n",
      "Question 4: What does her echocardiogram show?\n",
      "\n",
      "Answer: severe mitral regurgitation and also large pleural effusion\n",
      "\n",
      "\n",
      "Question 5: What other symptoms does the patient have?\n",
      "\n",
      "Answer: colitis and also diverticulitis\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "passage = \"Abnormal echocardiogram findings and followup. Shortness of breath, congestive heart failure, \\\n",
    "           and valvular insufficiency. The patient complains of shortness of breath, which is worsening. \\\n",
    "           The patient underwent an echocardiogram, which shows severe mitral regurgitation and also large \\\n",
    "           pleural effusion. The patient is an 86-year-old female admitted for evaluation of abdominal pain \\\n",
    "           and bloody stools. The patient has colitis and also diverticulitis, undergoing treatment. \\\n",
    "           During the hospitalization, the patient complains of shortness of breath, which is worsening. \\\n",
    "           The patient underwent an echocardiogram, which shows severe mitral regurgitation and also large \\\n",
    "           pleural effusion. This consultation is for further evaluation in this regard. As per the patient, \\\n",
    "           she is an 86-year-old female, has limited activity level. She has been having shortness of breath \\\n",
    "           for many years. She also was told that she has a heart murmur, which was not followed through \\\n",
    "           on a regular basis.\"\n",
    "\n",
    "q1 = \"How old is the patient?\"\n",
    "q2 = \"Does the patient have any complaints?\"\n",
    "q3 = \"What is the reason for this consultation?\"\n",
    "q4 = \"What does her echocardiogram show?\"\n",
    "q5 = \"What other symptoms does the patient have?\"\n",
    "\n",
    "\n",
    "questions = [q1, q2, q3, q4, q5]\n",
    "\n",
    "for i, q in enumerate(questions):\n",
    "    print(\"Question {}: {}\".format(i+1, q))\n",
    "    print()\n",
    "    print(\"Answer: {}\".format(get_model_answer(model, q, passage, tokenizer)))\n",
    "    print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even without fine-tuning, the model is able to reasonably answer most of the questions! Of course, it isn't perfect. For example, it doesn't give much detail for Question 3. To improve performance, you would ideally collect a medical QA dataset and fine tune the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations, you've finished the assignment for Week 2 of Course 3! Well done!"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "C3M3_Assignment.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "coursera": {
   "schema_names": [
    "AI4MC3-2"
   ]
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}