MST698S_Session4_NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Tools and Techniques\n",
    "\n",
    "# Natural Language Processing Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform the tasks below to demonstrate your understanding of NLP.  You may reference the TextBlob documentation here:\n",
    "\n",
    "https://textblob.readthedocs.io/en/dev/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Natural Language Processing is a very versatile tool in data science.  Among other things, it can be used to quantitate the affect, or emotional content, of text.  This technique is called sentiment analysis, and is often used in industry to measure how customers feel about a particular company or product.\n",
    "\n",
    "Sputnik is a Russian media outlet.  In this exercise, you will measure the sentiment of an English-language Sputnik article that mentions a U.S. political candidates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the space below, specify the location of the news_text.txt file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file_path = r''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the following function can be used to import the news article as a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#open text\n",
    "def open_text(file_path):\n",
    "    with open(file_path, 'r') as document:\n",
    "        text = document.read()\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the space below, use the above function to open the news_text.txt content, and print the resulting string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in the space below, import TextBlob using the following:\n",
    "\n",
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to instantiate a TextBlob object from a string of text.  This can be done as follows:\n",
    "\n",
    "blob = TextBlob(text)\n",
    "\n",
    "Instantiate the TextBlob object from the imported text in the space below, and print the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have instatiated your text as a TextBlob object, you can perform text manipulations.  First, let's generate a list of all sentences in your TextBlob object:\n",
    "\n",
    "blob_sentences = blob.sentences\n",
    "\n",
    "Do this in the space below, and print the result (you should see a list of sentences):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to pick out sentences that specificially mention the U.S. political candidates, since we are only interested in the affect surrounding those individuals.  The candidates in question are former Vice President Joe Biden, and Senator Bernie Sanders.  First, let's filter the article sentences so that we are only analyzing sentences that mention Mr. Biden:\n",
    "\n",
    "filtered_sentences = []\n",
    "for sentence in blob_sentences:\n",
    "    if 'Biden' in sentence:\n",
    "        filtered_sentences.append(sentence)\n",
    "\n",
    "Do this in the space below, and print the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in the space below, do the same for Mr. Sanders:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to measure the sentiment of each sentence, and then calculate an average sentiment for each candidate.  Sentiment has two components - objectivity and polarity.  Objectivity is the measure of how objective verses subjective a sentence is, and ranges from 0 for completely objective, to 1 for completely subjective.  Polarity is the measure of affect - how postive or negative the \"feeling\" is of the that particular sentence, and ranges from -1 (\"bad\" feelings) to +1 (\"good\" feelings).\n",
    "\n",
    "A good rule of thumb when measuring sentiment is to place a higher weight on sentences that are more subjective.  Thus, we will define our sentiment as the product of objectivity and polarity in order to weight our polarity measurements:\n",
    "\n",
    "sentiment = objectivity*polarity\n",
    "\n",
    "In TextBlob, the sentiment method will return a tuple, where the first value is the polarity, and the second value is the subjectivity:\n",
    "\n",
    "blob.sentiment\n",
    ">> ('polarity value','subjectivty value')\n",
    "\n",
    "In the space below, measure and print the sentiment of the second sentence of the \"Sanders_filtered_sentences\" list using the following command:\n",
    "\n",
    "Sanders_filtered_sentences[1].sentiment\n",
    "\n",
    "Now print the polarity:\n",
    "\n",
    "Sanders_filtered_sentences[1].sentiment[0]\n",
    "\n",
    "Now print the subjectivity:\n",
    "\n",
    "Sanders_filtered_sentences[1].sentiment[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to loop through the sentences in each list and append the sentiment (product of polarity and subjectivity) to each list.  We can do this as follows:\n",
    "\n",
    "Sanders_sentiment = []\n",
    "for sentence in Sanders_filtered_sentences:\n",
    "    sentiment = sentence.sentiment[0]*sentence.sentiment[1]\n",
    "    Sanders_sentiment.append(sentiment)\n",
    "\n",
    "Do this in the space below, and print the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in the space below, repeat this for former Vice President Biden:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, calculate the average sentiment of Senator Sanders.  Note that you will have to import the Numpy library to do this:\n",
    "\n",
    "import numpy as np\n",
    "Sanders_ave_sentiment = np.average(Sanders_sentiment)\n",
    "\n",
    "Calculate and print this in the space below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in the space below, do this for Vice President Biden:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, this exercise is based on only one article, so we can't derive a statistically robust inference.  However, based on this cursory analysis, do you think this article favors Mr. Sanders or Mr. Biden (the higher the number, the better the feeling)?  Why?\n",
    "\n",
    "How might this method be useful for analyzing large bodies of news text?  What could we do to get a more robust inference?\n",
    "\n",
    "Now, read the article.  Do you think these numbers are reasonable?  Why or why not?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}