5-UN-speeches-spacy-processing-topic-modeling-v2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eff095e0",
   "metadata": {},
   "source": [
    "## Further Text Processing & Re-Topic Modeling\n",
    "* Earlier topic models had the issue of multiple derivatives of the same word (meet, meetings, meeting) etc. \n",
    "    * This diluted our topic models accuracy\n",
    "* Resolve this by further processing our text with `SpaCy` and `textacy` to extract more specific parts of the text\n",
    "    * E.G. lemma forms of these words, named-entities etc.\n",
    "* Then try re-implementing `sklearn.decomposition NMF` and see if we can improve our results from notebooks (3) and (4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "395f8a59",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "import math\n",
    "import regex as re  \n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import NMF\n",
    "from tqdm.auto import tqdm\n",
    "import textacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3f00d2ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# spacy components \n",
    "import spacy\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from spacy.util import compile_prefix_regex, compile_infix_regex, compile_suffix_regex\n",
    "\n",
    "# * download spacy medium English model\n",
    "# !python -m spacy download en_core_web_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a37c0a76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the language object, labelled \"nlp\" from here on by convention \n",
    "## disable parser as it won't be needed for now\n",
    "nlp = spacy.load('en_core_web_md', disable=[\"parser\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4098e07",
   "metadata": {},
   "source": [
    "* Load DF pickle from notebook (1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "d661e42b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read pickled DF from notebook #1 (initial exploratory data visuals)\n",
    "df = pd.read_pickle('df_un_general_debates.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45bccc85",
   "metadata": {},
   "source": [
    "### The tokenisation problem from earlier notebook: \n",
    "* E.G. problems like: co-operation is split into co and operation etc.\n",
    "* Need to utilise lemmatisation to avoid getting functionally-identical stems of the same word (meet, meets, meetings) etc.\n",
    "\n",
    "* Below are functions to overcome this problem & handle other issues like noun phrase extraction etc.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e094b6cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "785c0bf9",
   "metadata": {},
   "source": [
    "# FURTHER TEXT PROCESSING WITH SPACY\n",
    "\n",
    "\n",
    "\n",
    "## Customising Re-tokenisation \n",
    "* Spacy/NLTK by default splits on: hash signs, hyphens and underscores\n",
    "    * Hence our problem with \"co-operation\" \n",
    "* Spacy's tokeniser is rule based:\n",
    "    * splits text on white space\n",
    "    * then uses prefix, suffix, and infix spliting rules defined by REGEX to further split remaining tokens\n",
    "    * exceptions are applied for english specific oddities, like \"can't\" (should be split into lemmas: can and not) \n",
    "    \n",
    "    \n",
    "* Thus, re-apply logic to create new tokens "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "daa9e749",
   "metadata": {},
   "outputs": [],
   "source": [
    "def custom_tokenizer(nlp):\n",
    "    \n",
    "    # use default patterns except the ones matched by re.search\n",
    "    prefixes = [pattern for pattern in nlp.Defaults.prefixes \n",
    "                if pattern not in ['-', '_', '#']]\n",
    "    \n",
    "    suffixes = [pattern for pattern in nlp.Defaults.suffixes\n",
    "                if pattern not in ['_']]\n",
    "    \n",
    "    infixes  = [pattern for pattern in nlp.Defaults.infixes\n",
    "                if not re.search(pattern, 'xx-xx')]\n",
    "\n",
    "    # these are the default params from spacy's \"Tokenizer.__init__\"\n",
    "    return Tokenizer(vocab          = nlp.vocab, \n",
    "                     rules          = nlp.Defaults.tokenizer_exceptions,\n",
    "                     prefix_search  = compile_prefix_regex(prefixes).search,\n",
    "                     suffix_search  = compile_suffix_regex(suffixes).search,\n",
    "                     infix_finditer = compile_infix_regex(infixes).finditer,\n",
    "                     token_match    = nlp.Defaults.token_match)\n",
    "                     #url_match [optional]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b13f632b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# recall: nlp = spacy.load('en_core_web_md', disable=[\"parser\"])\n",
    "## thus, alter default behaviour \n",
    "nlp.tokenizer = custom_tokenizer(nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "df07bb94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print stopwords \n",
    "# print(nlp.Defaults.stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7c3e70b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add our own custom corpus specific stopwors again \n",
    "\n",
    "additional_stopwords = {'dear','regards','also','would','must'}\n",
    "# exclude_stopwords = {''}\n",
    "\n",
    "nlp.Defaults.stop_words |= additional_stopwords\n",
    "# nlp.Defaults.stop_words -= exclude_stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0140b185",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ca23673f",
   "metadata": {},
   "source": [
    "## LEMMATISATION\n",
    "* The mapping words to their uninflected roots\n",
    "* Requires a lookup dictionary + knowledge of the part of speech\n",
    "    * So it can know the lemma of the (noun) meeting = meeting and the lemma of the verb \"meeting\" is meet\n",
    "    * Spacy can do this automatically for English with automatic part-of-speech dependency\n",
    "    \n",
    "    \n",
    "* The `pos_` attribute contains the simplified tag of the universal part-of-speech tagset- whcih remains stable across different models\n",
    "\n",
    "\n",
    "* The general logic that follows is that: \"Nouns, Verbs, Adjectives, Adverbs\" are content words (much of the sentence meaning depends on them)\n",
    "    * And that function words (pronouns, prepositions, conjunctions, determiners) create grammatical relationships within sentences - not always highly relevant to a lot of NLP applications\n",
    "    \n",
    "    \n",
    "* NOTE: lemmatisation could affect sentiment analysis results\n",
    "    * As in the example below, (best) becomes (good) which may be misleading for sentiment analysis purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "5bbbaa43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ryan | need | sit | talk | good | exciting | plan | Pete\n"
     ]
    }
   ],
   "source": [
    "#E.G. filtering on stopwords / punctuation\n",
    "sample_text = \"dear Ryan, we need to sit down and talk about our best and most exciting plans. regards, Pete\"\n",
    "sample_doc = nlp(sample_text)\n",
    "\n",
    "print(*[t.lemma_ for t in sample_doc if not t.is_stop and not t.is_punct], sep=\" | \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c49fd1c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dear | Ryan | plan | regard | Pete\n"
     ]
    }
   ],
   "source": [
    "#E.G. filtering on POS tag\n",
    "sample_text = \"dear Ryan, we need to sit down and talk about our best and most exciting plans. regards, Pete\"\n",
    "sample_doc = nlp(sample_text)\n",
    "\n",
    "print(*[t.lemma_ for t in sample_doc if t.pos_ in ['NOUN','PROPN']], sep=\" | \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107505d9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d27f224b",
   "metadata": {},
   "source": [
    "## Textacy's `extract.words` function\n",
    "* Utilising POS tags + additional token properties like: \n",
    "    * `is_punct` or `is_stop` \n",
    "    \n",
    "    \n",
    "* For more information, see docs:\n",
    "    * https://textacy.readthedocs.io/en/0.11.0/api_reference/extract.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "d61b8b52",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_text = \"dear Ryan, we need to sit down and talk about our best and most exciting plans. regards, Pete\"\n",
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7f67e01d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract tokens for ['adjectives','nouns'] from passed in nlp(\"doc\") object \n",
    "tokens = textacy.extract.words(doc, \n",
    "            filter_stops = True,           # default True, no stopwords\n",
    "            filter_punct = True,           # default True, no punctuation\n",
    "            filter_nums = True,            # default False, no numbers\n",
    "            include_pos = ['ADJ', 'NOUN'], # default None = include all\n",
    "            exclude_pos = None,            # default None = exclude none\n",
    "            min_freq = 1)                  # minimum frequency of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "b187358f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best|exciting|plans\n"
     ]
    }
   ],
   "source": [
    "print(*[t for t in tokens], sep='|')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56e28fa4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "38182c22",
   "metadata": {},
   "source": [
    "* Then create a wrapper around `textacy` by forwarding keyword arguments from (**kwargs)\n",
    "    * This function then accepts the same parameters as textacy's `extract.words` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e27d1837",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_lemmas(doc, **kwargs):\n",
    "    return [t.lemma_ for t in textacy.extract.words(doc, **kwargs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "e3f48c5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "good|exciting|plan\n"
     ]
    }
   ],
   "source": [
    "lemmas = extract_lemmas(doc, include_pos=['ADJ', 'NOUN'])\n",
    "print(*lemmas, sep='|')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "703c0249",
   "metadata": {},
   "source": [
    "* \"best exciting plans\" becomes lemmatised as \"good exciting plan\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b05d87a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cc6ff75",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "778f3934",
   "metadata": {},
   "source": [
    "### Noun phrase extraction \n",
    "* based on POS patterns\n",
    "    * function takes:\n",
    "        * Doc\n",
    "        * list of POS tags\n",
    "        * A seprator character to join words of the noun phrase\n",
    "        \n",
    "\n",
    "* constructed pattern searches for sequences of nouns, that are preceded by a token with one of the pre-specified POS tags\n",
    "    * Returns lemmas\n",
    "    \n",
    "    \n",
    "* Will extract all phrases consisting of an adjective or a noun followed by a sequence of nouns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "619e286e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.11.0\n"
     ]
    }
   ],
   "source": [
    "# slight syntax difference dependign on textacy version\n",
    "# if version >= 0.11, then add \"token_matches\" after the textacy.extract.matches\n",
    "print(textacy.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "94b8d52b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_noun_phrases(doc, preceding_pos=['NOUN'], sep='_'):\n",
    "    patterns = []\n",
    "    for pos in preceding_pos:\n",
    "        patterns.append(f\"POS:{pos} POS:NOUN:+\")\n",
    "\n",
    "    spans = textacy.extract.matches.token_matches(doc, patterns=patterns)\n",
    "\n",
    "    return [sep.join([t.lemma_ for t in s]) for s in spans]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9f69d6ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# paste exampe again\n",
    "sample_text = \"dear Ryan, we need to sit down and talk about our best and most exciting plans,\\\n",
    "as well as our fanciest adventures. regards, Pete\"\n",
    "\n",
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b726d02c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "exciting_plan | fancy_adventure\n"
     ]
    }
   ],
   "source": [
    "print(*extract_noun_phrases(doc, ['ADJ', 'NOUN']), sep=' | ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d75a800",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1495c13b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "18b8a70e",
   "metadata": {},
   "source": [
    "## Extracting Named Entities\n",
    "* makes use of SpaCy's `displacy` function for visualisations "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "e66aab33",
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy import displacy \n",
    "# from pathlib import Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "36c0faaf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>country_name</th>\n",
       "      <th>speaker</th>\n",
       "      <th>position</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "      <th>tokens</th>\n",
       "      <th>len_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7322</th>\n",
       "      <td>70</td>\n",
       "      <td>2015</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Ms. Julie Bishop</td>\n",
       "      <td>Minister for Foreign Affairs</td>\n",
       "      <td>We meet this day at an important time for the ...</td>\n",
       "      <td>12832</td>\n",
       "      <td>[meet, day, important, time, united, nations, ...</td>\n",
       "      <td>1095</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      session  year country country_name           speaker  \\\n",
       "7322       70  2015     AUS    Australia  Ms. Julie Bishop   \n",
       "\n",
       "                          position  \\\n",
       "7322  Minister for Foreign Affairs   \n",
       "\n",
       "                                                   text  length  \\\n",
       "7322  We meet this day at an important time for the ...   12832   \n",
       "\n",
       "                                                 tokens  len_tokens  \n",
       "7322  [meet, day, important, time, united, nations, ...        1095  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.query('country==\"AUS\" and year==2015')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fd525b4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pick an example text, E.G. from Australia in 2015\n",
    "sample_aus_2015 = df.iloc[7322]['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8053b3a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# put into spacy // convert object dtype to string\n",
    "sample_aus_2015_doc = nlp(str(sample_aus_2015))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "fba6336a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: pass in jupyter=False to avoid premature rendering in notebook\n",
    "html = displacy.render(sample_aus_2015_doc, style=\"ent\", jupyter=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "efe5333b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# write and visualise \n",
    "with open(\"sample_aus_2015_vis.html\", \"w+\") as f:\n",
    "    f.write(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f6ea2f2",
   "metadata": {},
   "source": [
    "* see REPO for the example without needing to run code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38e92532",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "df43b8d1",
   "metadata": {},
   "source": [
    "## Extracting named entities of certain types using `textacy.extract.entities`\n",
    "* documentation:\n",
    "    * https://textacy.readthedocs.io/en/0.11.0/api_reference/extract.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e3d6185e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_entities(doc, include_types=None, sep='_'):\n",
    "\n",
    "    ents = textacy.extract.entities(doc,\n",
    "             include_types=include_types, # pass in with function call \n",
    "             exclude_types=None,\n",
    "             drop_determiners=True,  #\"the\" etc.\n",
    "             min_freq=1) # remove entities which occur fewer than min_freq() times\n",
    "\n",
    "    # return the lemma + a \" / \" seperatign the lemma from its NER tag (E.G. GPE, person, nominal etc.)\n",
    "    return [sep.join([t.lemma_ for t in e])+' / '+e.label_ for e in ents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "92fd75db",
   "metadata": {},
   "outputs": [],
   "source": [
    "# random fake string\n",
    "sample_text = \"Henry Kissinger, ex-chairman of Microsoft, now lives in San Francisco, USA.\"\n",
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c83e0462",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Henry_Kissinger / PERSON',\n",
       " 'Microsoft / ORG',\n",
       " 'San_Francisco / GPE',\n",
       " 'USA / GPE']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extract_entities(doc, ['PERSON', 'GPE', 'ORG', 'LOC'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e93612d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17f44185",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "13db8dfb",
   "metadata": {},
   "source": [
    "## Combined function-call with all sub-elements \n",
    "* combines:\n",
    "    * `extract_lemmas`\n",
    "    * `extract_noun_phrases`\n",
    "    * `extract_entities` \n",
    "    \n",
    "* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "32376459",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_lemmas(doc, **kwargs):\n",
    "    return [t.lemma_ for t in textacy.extract.words(doc, **kwargs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a5d6bd21",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_noun_phrases(doc, preceding_pos=['NOUN'], sep='_'):\n",
    "    patterns = []\n",
    "    for pos in preceding_pos:\n",
    "        patterns.append(f\"POS:{pos} POS:NOUN:+\")\n",
    "\n",
    "    spans = textacy.extract.matches.token_matches(doc, patterns=patterns)\n",
    "\n",
    "    return [sep.join([t.lemma_ for t in s]) for s in spans]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "52c2f9bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_entities(doc, include_types=None, sep='_'):\n",
    "\n",
    "    ents = textacy.extract.entities(doc,\n",
    "             include_types=include_types, # pass in with function call \n",
    "             exclude_types=None,\n",
    "             drop_determiners=True,  #\"the\" etc.\n",
    "             min_freq=1) # remove entities which occur fewer than min_freq() times\n",
    "\n",
    "    # return the lemma + a \" / \" seperatign the lemma from its NER tag (E.G. GPE, person, nominal etc.)\n",
    "    return [sep.join([t.lemma_ for t in e])+' / '+e.label_ for e in ents]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "1bd9ed2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_nlp(doc):\n",
    "    return {\n",
    "    'lemmas'          : extract_lemmas(doc, \n",
    "                                     exclude_pos = ['PART', 'PUNCT', \n",
    "                                        'DET', 'PRON', 'SYM', 'SPACE'],\n",
    "                                     filter_stops = True),\n",
    "    'adjs_verbs'      : extract_lemmas(doc, include_pos = ['ADJ', 'VERB']),\n",
    "    'nouns'           : extract_lemmas(doc, include_pos = ['NOUN', 'PROPN']),\n",
    "    'noun_phrases'    : extract_noun_phrases(doc, ['NOUN']),\n",
    "    'adj_noun_phrases': extract_noun_phrases(doc, ['ADJ']),\n",
    "    'entities'        : extract_entities(doc, ['PERSON', 'ORG', 'GPE', 'LOC'])\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "90908ef3",
   "metadata": {},
   "outputs": [],
   "source": [
    "additional_stopwords = {'dear','regards','also','would','must'}\n",
    "# exclude_stopwords = {''}\n",
    "\n",
    "nlp.Defaults.stop_words |= additional_stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5595e7e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5ba3d28e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lemmas: ['good', 'friend', 'Ryan', 'Peters', 'like', 'fancy', 'adventure', 'game']\n",
      "adjs_verbs: ['good', 'like', 'fancy']\n",
      "nouns: ['friend', 'Ryan', 'Peters', 'adventure', 'game']\n",
      "noun_phrases: ['adventure_game']\n",
      "adj_noun_phrases: ['good_friend', 'fancy_adventure', 'fancy_adventure_game']\n",
      "entities: ['Ryan_Peters / PERSON']\n"
     ]
    }
   ],
   "source": [
    "#E.G. \n",
    "text = \"My best friend Ryan Peters also likes fancy adventure games.\"\n",
    "doc = nlp(text)\n",
    "for col, values in extract_nlp(doc).items():\n",
    "    print(f\"{col}: {values}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "82221c05",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['lemmas', 'adjs_verbs', 'nouns', 'noun_phrases', 'adj_noun_phrases', 'entities']\n"
     ]
    }
   ],
   "source": [
    "# view the additional grammatical elements to be added to the DF\n",
    "nlp_columns = list(extract_nlp(nlp.make_doc('')).keys())\n",
    "print(nlp_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "472932c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise empty variables to over-write with data\n",
    "for col in nlp_columns:\n",
    "    df[col] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8d7c853",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "8cc4912b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defined at start of notebook but again here for clarity \n",
    "nlp.tokenizer = custom_tokenizer(nlp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "87265f04",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 50\n",
    "batches = math.ceil(len(df) / batch_size) ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ea5bab38",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "151"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ff2559b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# WARNING # 1H runtime total\n",
    "for i in tqdm(range(0, len(df), batch_size), total=batches):\n",
    "    docs = nlp.pipe(df['text'][i:i+batch_size])\n",
    "    \n",
    "    for j, doc in enumerate(docs):\n",
    "        for col, values in extract_nlp(doc).items():\n",
    "            df[col].iloc[i+j] = values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8d023b1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "cf3f2abe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>session</th>\n",
       "      <th>year</th>\n",
       "      <th>country</th>\n",
       "      <th>country_name</th>\n",
       "      <th>speaker</th>\n",
       "      <th>position</th>\n",
       "      <th>text</th>\n",
       "      <th>length</th>\n",
       "      <th>tokens</th>\n",
       "      <th>len_tokens</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>adjs_verbs</th>\n",
       "      <th>nouns</th>\n",
       "      <th>noun_phrases</th>\n",
       "      <th>adj_noun_phrases</th>\n",
       "      <th>entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>ALB</td>\n",
       "      <td>Albania</td>\n",
       "      <td>Mr. NAS</td>\n",
       "      <td>NaN</td>\n",
       "      <td>33: May I first convey to our President the co...</td>\n",
       "      <td>51419</td>\n",
       "      <td>[may, first, convey, president, congratulation...</td>\n",
       "      <td>4125</td>\n",
       "      <td>[33, convey, President, congratulation, albani...</td>\n",
       "      <td>[convey, albanian, twenty-fifth, take, twenty-...</td>\n",
       "      <td>[President, congratulation, delegation, electi...</td>\n",
       "      <td>[balance_sheet, world_security, world_arena, l...</td>\n",
       "      <td>[albanian_delegation, twenty-fifth_session, fi...</td>\n",
       "      <td>[General_Assembly / ORG, General_Assembly / OR...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>ARG</td>\n",
       "      <td>Argentina</td>\n",
       "      <td>Mr. DE PABLO PARDO</td>\n",
       "      <td>NaN</td>\n",
       "      <td>177.\\t : It is a fortunate coincidence that pr...</td>\n",
       "      <td>29286</td>\n",
       "      <td>[fortunate, coincidence, precisely, time, unit...</td>\n",
       "      <td>2327</td>\n",
       "      <td>[177, fortunate, coincidence, precisely, time,...</td>\n",
       "      <td>[fortunate, celebrate, twenty-five, eminent, l...</td>\n",
       "      <td>[coincidence, time, United, Nations, year, exi...</td>\n",
       "      <td>[state_limit, extent_idea, case_discouragement...</td>\n",
       "      <td>[fortunate_coincidence, twenty-five_year, emin...</td>\n",
       "      <td>[United_Nations / ORG, Organization / ORG, Gen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>1970</td>\n",
       "      <td>AUS</td>\n",
       "      <td>Australia</td>\n",
       "      <td>Mr. McMAHON</td>\n",
       "      <td>NaN</td>\n",
       "      <td>100.\\t  It is a pleasure for me to extend to y...</td>\n",
       "      <td>31839</td>\n",
       "      <td>[pleasure, extend, mr, president, warmest, con...</td>\n",
       "      <td>2545</td>\n",
       "      <td>[100, pleasure, extend, Mr., President, warm, ...</td>\n",
       "      <td>[extend, warm, distinguished, play, authoritat...</td>\n",
       "      <td>[pleasure, Mr., President, congratulation, Aus...</td>\n",
       "      <td>[anniversary_session, war_wnidi, world_peace, ...</td>\n",
       "      <td>[warm_congratulation, distinguished_part, auth...</td>\n",
       "      <td>[Australia_Government / ORG, General_Assembly ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   session  year country country_name             speaker position  \\\n",
       "0       25  1970     ALB      Albania             Mr. NAS      NaN   \n",
       "1       25  1970     ARG    Argentina  Mr. DE PABLO PARDO      NaN   \n",
       "2       25  1970     AUS    Australia         Mr. McMAHON      NaN   \n",
       "\n",
       "                                                text  length  \\\n",
       "0  33: May I first convey to our President the co...   51419   \n",
       "1  177.\\t : It is a fortunate coincidence that pr...   29286   \n",
       "2  100.\\t  It is a pleasure for me to extend to y...   31839   \n",
       "\n",
       "                                              tokens  len_tokens  \\\n",
       "0  [may, first, convey, president, congratulation...        4125   \n",
       "1  [fortunate, coincidence, precisely, time, unit...        2327   \n",
       "2  [pleasure, extend, mr, president, warmest, con...        2545   \n",
       "\n",
       "                                              lemmas  \\\n",
       "0  [33, convey, President, congratulation, albani...   \n",
       "1  [177, fortunate, coincidence, precisely, time,...   \n",
       "2  [100, pleasure, extend, Mr., President, warm, ...   \n",
       "\n",
       "                                          adjs_verbs  \\\n",
       "0  [convey, albanian, twenty-fifth, take, twenty-...   \n",
       "1  [fortunate, celebrate, twenty-five, eminent, l...   \n",
       "2  [extend, warm, distinguished, play, authoritat...   \n",
       "\n",
       "                                               nouns  \\\n",
       "0  [President, congratulation, delegation, electi...   \n",
       "1  [coincidence, time, United, Nations, year, exi...   \n",
       "2  [pleasure, Mr., President, congratulation, Aus...   \n",
       "\n",
       "                                        noun_phrases  \\\n",
       "0  [balance_sheet, world_security, world_arena, l...   \n",
       "1  [state_limit, extent_idea, case_discouragement...   \n",
       "2  [anniversary_session, war_wnidi, world_peace, ...   \n",
       "\n",
       "                                    adj_noun_phrases  \\\n",
       "0  [albanian_delegation, twenty-fifth_session, fi...   \n",
       "1  [fortunate_coincidence, twenty-five_year, emin...   \n",
       "2  [warm_congratulation, distinguished_part, auth...   \n",
       "\n",
       "                                            entities  \n",
       "0  [General_Assembly / ORG, General_Assembly / OR...  \n",
       "1  [United_Nations / ORG, Organization / ORG, Gen...  \n",
       "2  [Australia_Government / ORG, General_Assembly ...  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#E.G. \n",
    "## some further analysis can be done on these other grammatical structures after \n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2312ee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96bb26fe",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "22a63ea6",
   "metadata": {},
   "source": [
    "## Lemmatisation & Topic Modeling\n",
    "* Just restrict out function to lemmaisation (reduce training time and more relevant to our topic model) \n",
    "* SpaCy medium/large model is more accurate than NLTK\n",
    "* Use these lemmas to try re- topic modeling with SKlearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "7dc67212",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_lemmas(doc, **kwargs):\n",
    "    return [t.lemma_ for t in textacy.extract.words(doc, **kwargs)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "4b001a89",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_nlp_simp(doc):\n",
    "    return {\n",
    "    'lemmas'          : extract_lemmas(doc, \n",
    "                                     include_pos = ['ADJ', 'NOUN'],\n",
    "                                     filter_stops = True)\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "068048eb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "7ae9fc6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the number of paragraphs of the text, splitting on the punctuation and ignoring the whitespace after them\n",
    "df[\"paragraphs\"] = df[\"text\"].map(lambda text: re.split('[.?!]\\s*\\n', text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "9ada0212",
   "metadata": {},
   "outputs": [],
   "source": [
    "paragraph_df = pd.DataFrame([{ \"text\": paragraph, \"year\": year } \n",
    "                               for paragraphs, year in zip(df[\"paragraphs\"], df[\"year\"]) \n",
    "                                    for paragraph in paragraphs if paragraph])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "a3da6f6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# just keep text \n",
    "paragraph_df = pd.DataFrame([{ \"text\": paragraph} \n",
    "                               for paragraphs in (df[\"paragraphs\"]) \n",
    "                                    for paragraph in paragraphs if paragraph])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "3f7b73ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33: May I first convey to our President the co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34.\\tIn taking up the work on the agenda of th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35.\\tThe utilization of the United Nations to ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text\n",
       "0  33: May I first convey to our President the co...\n",
       "1  34.\\tIn taking up the work on the agenda of th...\n",
       "2  35.\\tThe utilization of the United Nations to ..."
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#E.G.\n",
    "paragraph_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a90eb952",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "47f573c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1000 sents a batch, 283 batches to map whole DF\n",
    "batch_size = 1000\n",
    "batches = math.ceil(len(paragraph_df) / batch_size) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "44ce4ceb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise empty values to iterate over in loop through nlp.doc objects \n",
    "paragraph_df['noun_adj_lemmas'] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "259fdae0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d61ed8c23a941cf8b1605844931c84a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/283 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#loop over paragraph df \"text\" and apply the function by updating the None values in \"noun_adj_lemmas\" with their lemmas\n",
    "for i in tqdm(range(0, len(paragraph_df), batch_size), total=batches):\n",
    "    docs = nlp.pipe(paragraph_df['text'][i:i+batch_size])\n",
    "    \n",
    "    for j, doc in enumerate(docs):\n",
    "        for col, values in extract_nlp_simp(doc).items(): # remove col? \n",
    "            paragraph_df['noun_adj_lemmas'].iloc[i+j] = values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "9100ce94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>noun_adj_lemmas</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33: May I first convey to our President the co...</td>\n",
       "      <td>[congratulation, albanian, delegation, electio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34.\\tIn taking up the work on the agenda of th...</td>\n",
       "      <td>[work, agenda, twenty-, fifth, session, eve, t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35.\\tThe utilization of the United Nations to ...</td>\n",
       "      <td>[utilization, policy, able, hand, aggression, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36.\\tThe whole of progressive mankind recalls ...</td>\n",
       "      <td>[progressive, mankind, admiration, heroic, str...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37.\\tAll this has had well known consequences ...</td>\n",
       "      <td>[consequence, damaging, authority, ability, in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282205</th>\n",
       "      <td>For some months now, we have watched heartbrea...</td>\n",
       "      <td>[month, heartbreaking, harrowing, scene, despe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282206</th>\n",
       "      <td>This tragic situation could have been avoided ...</td>\n",
       "      <td>[tragic, situation, respect, independence, cou...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282207</th>\n",
       "      <td>My country, Zimbabwe, is committed to a fair, ...</td>\n",
       "      <td>[country, fair, effective, multilateralism, in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282208</th>\n",
       "      <td>We invite other countries with which we may ha...</td>\n",
       "      <td>[country, difference, nature, threat, pressure...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>282209</th>\n",
       "      <td>The seventieth anniversary of our Organization...</td>\n",
       "      <td>[seventieth, anniversary, adoption, far-reachi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>282210 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     text  \\\n",
       "0       33: May I first convey to our President the co...   \n",
       "1       34.\\tIn taking up the work on the agenda of th...   \n",
       "2       35.\\tThe utilization of the United Nations to ...   \n",
       "3       36.\\tThe whole of progressive mankind recalls ...   \n",
       "4       37.\\tAll this has had well known consequences ...   \n",
       "...                                                   ...   \n",
       "282205  For some months now, we have watched heartbrea...   \n",
       "282206  This tragic situation could have been avoided ...   \n",
       "282207  My country, Zimbabwe, is committed to a fair, ...   \n",
       "282208  We invite other countries with which we may ha...   \n",
       "282209  The seventieth anniversary of our Organization...   \n",
       "\n",
       "                                          noun_adj_lemmas  \n",
       "0       [congratulation, albanian, delegation, electio...  \n",
       "1       [work, agenda, twenty-, fifth, session, eve, t...  \n",
       "2       [utilization, policy, able, hand, aggression, ...  \n",
       "3       [progressive, mankind, admiration, heroic, str...  \n",
       "4       [consequence, damaging, authority, ability, in...  \n",
       "...                                                   ...  \n",
       "282205  [month, heartbreaking, harrowing, scene, despe...  \n",
       "282206  [tragic, situation, respect, independence, cou...  \n",
       "282207  [country, fair, effective, multilateralism, in...  \n",
       "282208  [country, difference, nature, threat, pressure...  \n",
       "282209  [seventieth, anniversary, adoption, far-reachi...  \n",
       "\n",
       "[282210 rows x 2 columns]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#E.G. \n",
    "paragraph_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "462fad60",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "5a96037a",
   "metadata": {},
   "outputs": [],
   "source": [
    "## update pickle \n",
    "# paragraph_df.to_pickle('paragraph_df_lemmas.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "61d80e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read pickle \n",
    "paragraph_df = pd.read_pickle('paragraph_df_lemmas.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b1f931a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# add some additional corpus specific stopwords \n",
    "additional_stopwords = {'dear','regards','also','would','must', 'congratulation',\n",
    "                       'world','new','year', 'nation', 'today', 'time', 'challenge', 'great', 'people',\n",
    "                       'session', 'delegation', 'presidency', 'behalf', 'country', 'co', 'international', 'operation'} \n",
    "# exclude_stopwords = {''}\n",
    "\n",
    "nlp.Defaults.stop_words |= additional_stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fa974ced",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "9ca3a5d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "## tf-idf vectoriser for speech's text: ## from notebook #3 for sklearn \n",
    "\n",
    "## start with just unigrams \n",
    "tfidf_para_vectorizer_lemma = TfidfVectorizer(stop_words=nlp.Defaults.stop_words, \n",
    "                                                 min_df=6, \n",
    "                                                 max_df=0.7, \n",
    "                                                 smooth_idf = True, \n",
    "                                                 ngram_range=(1,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2f80dae8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tfidf vectoriser requires strings, not list (quickly adjust the output) \n",
    "paragraph_df['noun_adj_lemma_str'] = paragraph_df['noun_adj_lemmas'].apply(lambda x: ' '.join(map(str, x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c4eaafce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>noun_adj_lemmas</th>\n",
       "      <th>noun_adj_lemma_str</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33: May I first convey to our President the co...</td>\n",
       "      <td>[congratulation, albanian, delegation, electio...</td>\n",
       "      <td>congratulation albanian delegation election tw...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34.\\tIn taking up the work on the agenda of th...</td>\n",
       "      <td>[work, agenda, twenty-, fifth, session, eve, t...</td>\n",
       "      <td>work agenda twenty- fifth session eve twenty-f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35.\\tThe utilization of the United Nations to ...</td>\n",
       "      <td>[utilization, policy, able, hand, aggression, ...</td>\n",
       "      <td>utilization policy able hand aggression part w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36.\\tThe whole of progressive mankind recalls ...</td>\n",
       "      <td>[progressive, mankind, admiration, heroic, str...</td>\n",
       "      <td>progressive mankind admiration heroic struggle...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37.\\tAll this has had well known consequences ...</td>\n",
       "      <td>[consequence, damaging, authority, ability, in...</td>\n",
       "      <td>consequence damaging authority ability incumbe...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0  33: May I first convey to our President the co...   \n",
       "1  34.\\tIn taking up the work on the agenda of th...   \n",
       "2  35.\\tThe utilization of the United Nations to ...   \n",
       "3  36.\\tThe whole of progressive mankind recalls ...   \n",
       "4  37.\\tAll this has had well known consequences ...   \n",
       "\n",
       "                                     noun_adj_lemmas  \\\n",
       "0  [congratulation, albanian, delegation, electio...   \n",
       "1  [work, agenda, twenty-, fifth, session, eve, t...   \n",
       "2  [utilization, policy, able, hand, aggression, ...   \n",
       "3  [progressive, mankind, admiration, heroic, str...   \n",
       "4  [consequence, damaging, authority, ability, in...   \n",
       "\n",
       "                                  noun_adj_lemma_str  \n",
       "0  congratulation albanian delegation election tw...  \n",
       "1  work agenda twenty- fifth session eve twenty-f...  \n",
       "2  utilization policy able hand aggression part w...  \n",
       "3  progressive mankind admiration heroic struggle...  \n",
       "4  consequence damaging authority ability incumbe...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paragraph_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bc55d6b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "72c79f37",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jason\\anaconda3\\lib\\site-packages\\sklearn\\feature_extraction\\text.py:388: UserWarning: Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens ['ll', 've'] not in stop_words.\n",
      "  warnings.warn('Your stop_words may be inconsistent with '\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 5.19 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "tfidf_para_lemma_vectors = tfidf_para_vectorizer_lemma.fit_transform(paragraph_df['noun_adj_lemma_str'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "1f2d5636",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(282210, 11224)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_para_lemma_vectors.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3226213",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "84db4401",
   "metadata": {},
   "source": [
    "## Re-import display topics function from notebook (3)\n",
    "* Or imported as separate py function in github repo for simplicity "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d2dfdd24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function defined in notebook #3 as separate py file \n",
    "from display_topics_func import display_topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "1daba2bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from: from sklearn.decomposition import NMF\n",
    "# repeat steps from initial model in notebook #3\n",
    "# try with 10 topics first \n",
    "nmf_para_model = NMF(n_components=10, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "78cb47f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jason\\anaconda3\\lib\\site-packages\\sklearn\\decomposition\\_nmf.py:312: FutureWarning: The 'init' value, when 'init=None' and n_components is less than n_samples and n_features, will be changed from 'nndsvd' to 'nndsvda' in 1.1 (renaming of 0.26).\n",
      "  warnings.warn((\"The 'init' value, when 'init=None' and \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 26.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# define matrixes from para model + the afore-defined para vectors variable \n",
    "W_para_matrix = nmf_para_model.fit_transform(tfidf_para_lemma_vectors)\n",
    "H_para_matrix = nmf_para_model.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7189d995",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a19578c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Topic 00\n",
      "  problem (11.88)\n",
      "  solution (6.64)\n",
      "  question (1.42)\n",
      "  debt (0.83)\n",
      "  search (0.77)\n",
      "  peaceful (0.74)\n",
      "  effort (0.71)\n",
      "  concern (0.71)\n",
      "\n",
      "Topic 01\n",
      "  economic (4.54)\n",
      "  development (4.54)\n",
      "  social (1.92)\n",
      "  economy (1.09)\n",
      "  growth (1.03)\n",
      "  resource (0.90)\n",
      "  political (0.88)\n",
      "  sustainable (0.86)\n",
      "\n",
      "Topic 02\n",
      "  right (10.51)\n",
      "  human (8.26)\n",
      "  freedom (1.68)\n",
      "  respect (1.49)\n",
      "  fundamental (1.43)\n",
      "  violation (1.11)\n",
      "  protection (1.10)\n",
      "  democracy (1.08)\n",
      "\n",
      "Topic 03\n",
      "  nuclear (7.87)\n",
      "  weapon (6.12)\n",
      "  disarmament (3.03)\n",
      "  arm (2.05)\n",
      "  proliferation (1.56)\n",
      "  destruction (1.46)\n",
      "  mass (1.23)\n",
      "  race (1.15)\n",
      "\n",
      "Topic 04\n",
      "  peace (14.89)\n",
      "  security (7.20)\n",
      "  stability (2.25)\n",
      "  region (2.16)\n",
      "  maintenance (1.28)\n",
      "  process (1.24)\n",
      "  keeping (1.18)\n",
      "  justice (1.14)\n",
      "\n",
      "Topic 05\n",
      "  election (4.22)\n",
      "  general (3.19)\n",
      "  work (3.11)\n",
      "  secretary (3.09)\n",
      "  predecessor (1.66)\n",
      "  success (1.64)\n",
      "  appreciation (1.58)\n",
      "  tribute (1.49)\n",
      "\n",
      "Topic 06\n",
      "  principle (2.17)\n",
      "  independence (1.57)\n",
      "  relation (1.31)\n",
      "  policy (1.23)\n",
      "  state (1.16)\n",
      "  force (1.13)\n",
      "  sovereignty (1.12)\n",
      "  foreign (0.94)\n",
      "\n",
      "Topic 07\n",
      "  resolution (7.29)\n",
      "  palestinian (4.19)\n",
      "  territory (2.53)\n",
      "  arab (2.52)\n",
      "  israeli (1.73)\n",
      "  settlement (1.64)\n",
      "  question (1.47)\n",
      "  implementation (1.46)\n",
      "\n",
      "Topic 08\n",
      "  community (1.38)\n",
      "  global (1.10)\n",
      "  reform (0.96)\n",
      "  effort (0.88)\n",
      "  issue (0.85)\n",
      "  action (0.83)\n",
      "  member (0.81)\n",
      "  change (0.80)\n",
      "\n",
      "Topic 09\n",
      "  conflict (7.82)\n",
      "  situation (2.72)\n",
      "  war (2.54)\n",
      "  region (2.12)\n",
      "  settlement (1.42)\n",
      "  party (1.30)\n",
      "  effort (1.29)\n",
      "  peaceful (1.28)\n"
     ]
    }
   ],
   "source": [
    "# display topics for paragraph model, set top words = 8 \n",
    "# change to para vectorizer \n",
    "display_topics(nmf_para_model, tfidf_para_vectorizer_lemma.get_feature_names(),\n",
    "              top_words=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86b67271",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8f62c4ae",
   "metadata": {},
   "source": [
    "* Definitely visible that the bulk of the topics are more well defined than in notebook (3) without lemmas - seems to be a definite improvement\n",
    "    * Demonstrated by the quick drop off in values for contribution by word to topic \n",
    "    * With the exception of topic 8 (which has the highest % of documents assigned to it (see below) generally because the topics are quite broad compared to the others "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "5fda175c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 0 = 5.14%\n",
      "Topic 1 = 13.23%\n",
      "Topic 2 = 8.60%\n",
      "Topic 3 = 6.02%\n",
      "Topic 4 = 10.92%\n",
      "Topic 5 = 9.15%\n",
      "Topic 6 = 13.27%\n",
      "Topic 7 = 7.41%\n",
      "Topic 8 = 17.74%\n",
      "Topic 9 = 8.52%\n"
     ]
    }
   ],
   "source": [
    "# visualise what % of documents could be assigned mainly to each topic \n",
    "for i, val in enumerate(W_para_matrix.sum(axis=0) / W_para_matrix.sum() * 100.0 ):\n",
    "    print(\"Topic {} = {:.2f}%\".format(i, val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fda2e74b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9b71ee3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ffeb7200",
   "metadata": {},
   "source": [
    "## Future Scope of works\n",
    "\n",
    "\n",
    "* try re-iterate with NER? To see mentions of organisations by year etc.? \n",
    "* Use other grammatical structures to draw some insights\n",
    "* Try text summarisation techniques \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}