SparkML - 04 Text_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Statement: IMDB Comment Sentiment Classifier\n",
    "\n",
    "Dataset: For this exercise we will use a dataset hosted at http://ai.stanford.edu/~amaas/data/sentiment/\n",
    "\n",
    "**Problem Statement**: \n",
    "This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. Raw text and already processed bag of words formats are provided. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch a spark session, verify the spark session UI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://198.166.1.126:4040'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spark.sparkContext.uiWebUrl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IMDB comments dataset has been stored in the following location "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   50000 data/imdb-comments.json\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l data/imdb-comments.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 50000 lines in the file. Let's the first line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 66M\tdata/imdb-comments.json\r\n"
     ]
    }
   ],
   "source": [
    "!du -sh data/imdb-comments.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total size of the file is 66MB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"label\":\"test\",\"sentiment\":\"pos\",\"name\":\"0_10.txt\",\"content\":\"I went and saw this movie last night after being coaxed to by a few friends of mine. I'll admit that I was reluctant to see it because from what I knew of Ashton Kutcher he was only able to do comedy. I was wrong. Kutcher played the character of Jake Fischer very well, and Kevin Costner played Ben Randall with such professionalism. The sign of a good movie is that it can toy with our emotions. This one did exactly that. The entire theater (which was sold out) was overcome by laughter during the first half of the movie, and were moved to tears during the second half. While exiting the theater I not only saw many women in tears, but many full grown men as well, trying desperately not to let anyone see them crying. This movie was great, and I suggest that you go see it before you judge.\"}\r\n"
     ]
    }
   ],
   "source": [
    "!head -n 1 data/imdb-comments.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each line is a self contained json doc. Load the dataset using spark reader specifying the file format as json. As we see above size of the file is 66 MB, we should at least 2 partitons, since I am using dual core system, I will repartition the data to 4. Also will cache the data after repartitioning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "imdb = spark.read.format(\"json\").load(\"data/imdb-comments.json\").repartition(4).cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find total number of records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print Schema and view the field types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- content: string (nullable = true)\n",
      " |-- label: string (nullable = true)\n",
      " |-- name: string (nullable = true)\n",
      " |-- sentiment: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "imdb.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at a few sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+------------+---------+\n",
      "|             content|label|        name|sentiment|\n",
      "+--------------------+-----+------------+---------+\n",
      "|Actor turned dire...| test| 10000_7.txt|      pos|\n",
      "|I saw this film o...| test| 10004_9.txt|      pos|\n",
      "|\"Quitting\" may be...| test| 10008_8.txt|      pos|\n",
      "|I must say, every...| test| 10011_9.txt|      pos|\n",
      "|Even if you're a ...| test| 10015_8.txt|      pos|\n",
      "|Since this cartoo...| test| 10019_8.txt|      pos|\n",
      "|This short is one...| test|10022_10.txt|      pos|\n",
      "|Warner Brothers t...| test|10026_10.txt|      pos|\n",
      "|I was fortunate t...| test|  1002_9.txt|      pos|\n",
      "|The historical in...| test| 10033_8.txt|      pos|\n",
      "|This movie is goo...| test| 10037_7.txt|      pos|\n",
      "|Although \"They Di...| test|10040_10.txt|      pos|\n",
      "|George Armstrong ...| test| 10044_8.txt|      pos|\n",
      "|Although this fil...| test| 10048_8.txt|      pos|\n",
      "|All in all, an ex...| test| 10051_8.txt|      pos|\n",
      "|i have to rate th...| test|10055_10.txt|      pos|\n",
      "|Why does everyone...| test|10059_10.txt|      pos|\n",
      "|For late-80s chee...| test| 10062_9.txt|      pos|\n",
      "|Madonna gets into...| test| 10066_7.txt|      pos|\n",
      "|[CONTAINS SPOILER...| test|  1006_8.txt|      pos|\n",
      "+--------------------+-----+------------+---------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "imdb.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*label* - column indicate whethet the data belong to training or test bucket.\n",
    "*sentiment* - column indicates whether the comment carries positive or negative sentiment. This column has been manually curated.\n",
    "\n",
    "Find out for each combination of label and sentimnet how many records are there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import *\n",
    "from pyspark.sql.types import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+-----+-----+\n",
      "|sentiment| test|train|\n",
      "+---------+-----+-----+\n",
      "|      pos|12500|12500|\n",
      "|      neg|12500|12500|\n",
      "+---------+-----+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "imdb.groupBy(\"sentiment\").pivot(\"label\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at a sample comment value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i have to rate this movie at a 10. i\\'m sorry but i think it\\'s classic comedy. then, if you\\'re rating it to other Madonna movies...well, what? you wanna tell me it wasn\\'t her best movie ever? didn\\'t Mira Sorvino win an Oscar for almost the same performance not ten years later? please, this movie deserves much more credit than it gets. plus, i like to think of it as an A+ sociological study into the lifestyles of the 80\\'s. remember when you could shoplift from Sam Goody and Cartier in the same day? remember when women wore bushy eyebrows proudly? so it was no \"Last Emperor\", it was still good. there are certain movies i\\'d be willing to watch everyday. three, actually, that pep up my day and make me smile. if you like \"Who\\'s That Girl?\" then i\\'d also recommend \"Party Girl\" and \"Romy and Michelle\\'s High School Reunion\".'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "content = imdb.sample(False, 0.001, 1).first().content\n",
    "content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Register a UDF function to clean the comment from the html tags. If BeautifulSoup is not installed, you can install it using pip \n",
    "\n",
    "```(shell command)\n",
    "$ pip install BeautifulSoup4\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "from pyspark.sql.types import * \n",
    "import re\n",
    "def remove_html_tags(text):\n",
    "    text = BeautifulSoup(text, \"html5lib\").text.lower() #removed html tags\n",
    "    text = re.sub(\"[\\W]+\", \" \", text)\n",
    "    return text\n",
    "\n",
    "spark.udf.register(\"remove_html_tags\", remove_html_tags, StringType())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the remove_html_tags function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i have to rate this movie at a 10 i m sorry but i think it s classic comedy then if you re rating it to other madonna movies well what you wanna tell me it wasn t her best movie ever didn t mira sorvino win an oscar for almost the same performance not ten years later please this movie deserves much more credit than it gets plus i like to think of it as an a sociological study into the lifestyles of the 80 s remember when you could shoplift from sam goody and cartier in the same day remember when women wore bushy eyebrows proudly so it was no last emperor it was still good there are certain movies i d be willing to watch everyday three actually that pep up my day and make me smile if you like who s that girl then i d also recommend party girl and romy and michelle s high school reunion '"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "remove_html_tags(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the the udf on the imdb dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'i have to rate this movie at a 10 i m sorry but i think it s classic comedy then if you re rating it to other madonna movies well what you wanna tell me it wasn t her best movie ever didn t mira sorvino win an oscar for almost the same performance not ten years later please this movie deserves much more credit than it gets plus i like to think of it as an a sociological study into the lifestyles of the 80 s remember when you could shoplift from sam goody and cartier in the same day remember when women wore bushy eyebrows proudly so it was no last emperor it was still good there are certain movies i d be willing to watch everyday three actually that pep up my day and make me smile if you like who s that girl then i d also recommend party girl and romy and michelle s high school reunion '"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_clean = imdb.withColumn(\"content\", expr(\"remove_html_tags(content)\")).cache()\n",
    "imdb_clean.sample(False, 0.001, 1).first().content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use Tokenizer to split the string into terms. Then use StopWordsRemover to remove stop words like prepositions, apply CountVectorizer to find all distinct terms and found of each term per document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import HashingTF, IDF, Tokenizer, CountVectorizer, StopWordsRemover"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(inputCol=\"content\", outputCol=\"terms\")\n",
    "terms_data = tokenizer.transform(imdb_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'have', 'to', 'rate', 'this', 'movie', 'at', 'a', '10', 'i', 'm', 'sorry', 'but', 'i', 'think', 'it', 's', 'classic', 'comedy', 'then', 'if', 'you', 're', 'rating', 'it', 'to', 'other', 'madonna', 'movies', 'well', 'what', 'you', 'wanna', 'tell', 'me', 'it', 'wasn', 't', 'her', 'best', 'movie', 'ever', 'didn', 't', 'mira', 'sorvino', 'win', 'an', 'oscar', 'for', 'almost', 'the', 'same', 'performance', 'not', 'ten', 'years', 'later', 'please', 'this', 'movie', 'deserves', 'much', 'more', 'credit', 'than', 'it', 'gets', 'plus', 'i', 'like', 'to', 'think', 'of', 'it', 'as', 'an', 'a', 'sociological', 'study', 'into', 'the', 'lifestyles', 'of', 'the', '80', 's', 'remember', 'when', 'you', 'could', 'shoplift', 'from', 'sam', 'goody', 'and', 'cartier', 'in', 'the', 'same', 'day', 'remember', 'when', 'women', 'wore', 'bushy', 'eyebrows', 'proudly', 'so', 'it', 'was', 'no', 'last', 'emperor', 'it', 'was', 'still', 'good', 'there', 'are', 'certain', 'movies', 'i', 'd', 'be', 'willing', 'to', 'watch', 'everyday', 'three', 'actually', 'that', 'pep', 'up', 'my', 'day', 'and', 'make', 'me', 'smile', 'if', 'you', 'like', 'who', 's', 'that', 'girl', 'then', 'i', 'd', 'also', 'recommend', 'party', 'girl', 'and', 'romy', 'and', 'michelle', 's', 'high', 'school', 'reunion']\n"
     ]
    }
   ],
   "source": [
    "print(terms_data.sample(False, 0.001, 1).first().terms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "remover = StopWordsRemover(inputCol=\"terms\", outputCol=\"filtered\")\n",
    "terms_stop_removed = remover.transform(terms_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['rate', 'movie', '10', 'm', 'sorry', 'think', 'classic', 'comedy', 're', 'rating', 'madonna', 'movies', 'well', 'wanna', 'tell', 'wasn', 'best', 'movie', 'ever', 'didn', 'mira', 'sorvino', 'win', 'oscar', 'almost', 'performance', 'ten', 'years', 'later', 'please', 'movie', 'deserves', 'much', 'credit', 'gets', 'plus', 'like', 'think', 'sociological', 'study', 'lifestyles', '80', 'remember', 'shoplift', 'sam', 'goody', 'cartier', 'day', 'remember', 'women', 'wore', 'bushy', 'eyebrows', 'proudly', 'last', 'emperor', 'still', 'good', 'certain', 'movies', 'd', 'willing', 'watch', 'everyday', 'three', 'actually', 'pep', 'day', 'make', 'smile', 'like', 'girl', 'd', 'also', 'recommend', 'party', 'girl', 'romy', 'michelle', 'high', 'school', 'reunion']\n"
     ]
    }
   ],
   "source": [
    "print(terms_stop_removed.sample(False, 0.001, 1).first().filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SparseVector(103999, {0: 3.0, 3: 2.0, 4: 1.0, 10: 1.0, 11: 1.0, 16: 1.0, 19: 1.0, 21: 2.0, 23: 2.0, 24: 1.0, 36: 1.0, 39: 1.0, 44: 1.0, 49: 1.0, 56: 1.0, 59: 1.0, 60: 1.0, 66: 1.0, 67: 1.0, 98: 1.0, 113: 1.0, 114: 1.0, 122: 2.0, 134: 1.0, 136: 1.0, 139: 2.0, 148: 2.0, 170: 1.0, 185: 1.0, 191: 1.0, 197: 1.0, 249: 1.0, 252: 1.0, 255: 1.0, 266: 1.0, 267: 1.0, 270: 2.0, 474: 1.0, 548: 1.0, 589: 1.0, 639: 1.0, 641: 1.0, 670: 1.0, 783: 1.0, 807: 1.0, 838: 1.0, 867: 1.0, 874: 1.0, 953: 1.0, 1113: 1.0, 1115: 1.0, 1556: 1.0, 1740: 1.0, 1904: 1.0, 2691: 1.0, 2745: 1.0, 2803: 1.0, 3481: 1.0, 3710: 1.0, 3979: 1.0, 4204: 1.0, 9709: 1.0, 9775: 1.0, 10080: 1.0, 11397: 1.0, 11497: 1.0, 13556: 1.0, 14823: 1.0, 17291: 1.0, 23274: 1.0, 23458: 1.0, 46723: 1.0, 96230: 1.0})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vectorizer = CountVectorizer(inputCol=\"filtered\", outputCol=\"count_vectors\")\n",
    "count_vectorizer_model = count_vectorizer.fit(terms_stop_removed)\n",
    "count_vectorized = count_vectorizer_model.transform(terms_stop_removed)\n",
    "count_vectorized.sample(False, 0.001, 1).first().count_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "count_vectorized Dataframe contains a column count_vectors that is a SparseVector representing which term appears and how many times. The key is the index of all unique terms. You can find list of terms count_vectorizer_model.vocabulary. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['movie', 'film', 'one', 'like', 'good', 'time', 'even', 'story', 'really', 'see', 'well', 'much', 'bad', 'get', 'people', 'great', 'also', 'first', 'made', 'make', 'way', 'movies', 'characters', 'think', 'watch', 'character', 'films', 'two', 'many', 'seen', 'love', 'never', 'plot', 'life', 'acting', 'show', 'best', 'know', 'little', 'ever', 'man', 'better', 'end', 'scene', 'still', 'say', 'scenes', 've', 'something', 'm', 'go', 'back', 'real', 'thing', 'watching', 'actors', 're', 'doesn', 'director', 'didn', 'years', 'funny', 'though', 'old', 'another', 'work', '10', 'actually', 'nothing', 'makes', 'look', 'find', 'going', 'new', 'lot', 'every', 'part', 'world', 'cast', 'us', 'things', 'want', 'quite', 'pretty', 'horror', 'around', 'seems', 'young', 'take', 'big', 'however', 'got', 'thought', 'fact', 'enough', 'long', 'give', 'may', 'comedy', 'series'] \n",
      "\n",
      "Total no of terms 103999\n"
     ]
    }
   ],
   "source": [
    "print(count_vectorizer_model.vocabulary[:100], \"\\n\\nTotal no of terms\", len(count_vectorizer_model.vocabulary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+-----+------------+---------+--------------------+--------------------+--------------------+\n",
      "|             content|label|        name|sentiment|               terms|            filtered|       count_vectors|\n",
      "+--------------------+-----+------------+---------+--------------------+--------------------+--------------------+\n",
      "|actor turned dire...| test| 10000_7.txt|      pos|[actor, turned, d...|[actor, turned, d...|(103999,[1,4,7,10...|\n",
      "|i saw this film o...| test| 10004_9.txt|      pos|[i, saw, this, fi...|[saw, film, septe...|(103999,[0,1,2,5,...|\n",
      "| quitting may be ...| test| 10008_8.txt|      pos|[, quitting, may,...|[, quitting, may,...|(103999,[0,2,3,4,...|\n",
      "|i must say every ...| test| 10011_9.txt|      pos|[i, must, say, ev...|[must, say, every...|(103999,[0,2,5,7,...|\n",
      "|even if you re a ...| test| 10015_8.txt|      pos|[even, if, you, r...|[even, re, fan, j...|(103999,[0,1,2,6,...|\n",
      "|since this cartoo...| test| 10019_8.txt|      pos|[since, this, car...|[since, cartoon, ...|(103999,[0,4,10,1...|\n",
      "|this short is one...| test|10022_10.txt|      pos|[this, short, is,...|[short, one, best...|(103999,[2,3,5,10...|\n",
      "|warner brothers t...| test|10026_10.txt|      pos|[warner, brothers...|[warner, brothers...|(103999,[1,2,5,9,...|\n",
      "|i was fortunate t...| test|  1002_9.txt|      pos|[i, was, fortunat...|[fortunate, atten...|(103999,[1,12,15,...|\n",
      "|the historical in...| test| 10033_8.txt|      pos|[the, historical,...|[historical, inac...|(103999,[1,2,10,1...|\n",
      "|this movie is goo...| test| 10037_7.txt|      pos|[this, movie, is,...|[movie, good, ent...|(103999,[0,1,3,4,...|\n",
      "|although they die...| test|10040_10.txt|      pos|[although, they, ...|[although, died, ...|(103999,[0,10,22,...|\n",
      "|george armstrong ...| test| 10044_8.txt|      pos|[george, armstron...|[george, armstron...|(103999,[0,3,9,10...|\n",
      "|although this fil...| test| 10048_8.txt|      pos|[although, this, ...|[although, film, ...|(103999,[1,3,12,1...|\n",
      "|all in all an exc...| test| 10051_8.txt|      pos|[all, in, all, an...|[excellent, movie...|(103999,[0,1,2,3,...|\n",
      "|i have to rate th...| test|10055_10.txt|      pos|[i, have, to, rat...|[rate, movie, 10,...|(103999,[0,3,4,10...|\n",
      "|why does everyone...| test|10059_10.txt|      pos|[why, does, every...|[everyone, feel, ...|(103999,[0,10,25,...|\n",
      "|for late 80s chee...| test| 10062_9.txt|      pos|[for, late, 80s, ...|[late, 80s, chees...|(103999,[0,1,2,8,...|\n",
      "|madonna gets into...| test| 10066_7.txt|      pos|[madonna, gets, i...|[madonna, gets, a...|(103999,[0,2,3,10...|\n",
      "| contains spoiler...| test|  1006_8.txt|      pos|[, contains, spoi...|[, contains, spoi...|(103999,[2,3,4,5,...|\n",
      "+--------------------+-----+------------+---------+--------------------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "count_vectorized.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SparkVector represents a vector of 103999, that means in the dataset (corpus) there are 103999 unique terms. Per document, only a few will be present. Find density of each count_vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|             density|\n",
      "+--------------------+\n",
      "|0.001615400148078...|\n",
      "|0.001375013221280...|\n",
      "|0.003884652737045...|\n",
      "|6.730833950326445E-4|\n",
      "|0.003326955066875643|\n",
      "|8.269310281829633E-4|\n",
      "|  6.0577505552938E-4|\n",
      "|0.003951961076548813|\n",
      "|5.769286243136953E-4|\n",
      "|5.865441013855903E-4|\n",
      "|0.001163472725699...|\n",
      "|3.365416975163222...|\n",
      "|0.002115404955816...|\n",
      "|5.096202848104309E-4|\n",
      "|0.002144251387032...|\n",
      "|7.019298262483293E-4|\n",
      "|3.557726516601121E-4|\n",
      "|6.538524408888547E-4|\n",
      "|6.826988721045394E-4|\n",
      "|0.001923095414378...|\n",
      "+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "vocab_len = len(count_vectorizer_model.vocabulary)\n",
    "spark.udf.register(\"density\", lambda r: r.numNonzeros() / vocab_len, DoubleType())\n",
    "count_vectorized.select(expr(\"density(count_vectors) density\")).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Density report shows, the count_vectors has very low density which illustrate the benefit of the choice of DenseVector for this column. \n",
    "\n",
    "Now, calculate tfidf for the document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "idf = IDF(inputCol=\"count_vectors\", outputCol=\"features\")\n",
    "idf_model = idf.fit(count_vectorized)\n",
    "idf_data = idf_model.transform(count_vectorized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SparseVector(103999, {0: 1.4768, 3: 1.5391, 4: 0.9676, 10: 1.2039, 11: 1.2672, 16: 1.3666, 19: 1.4081, 21: 3.0759, 23: 3.0707, 24: 1.5228, 36: 1.643, 39: 1.6425, 44: 1.7577, 49: 1.8738, 56: 2.0108, 59: 1.9746, 60: 1.9529, 66: 2.0092, 67: 1.986, 98: 2.3585, 113: 2.2417, 114: 2.2703, 122: 4.723, 134: 2.3357, 136: 2.4248, 139: 5.0252, 148: 4.8888, 170: 2.6037, 185: 2.5749, 191: 2.6059, 197: 2.6278, 249: 2.7842, 252: 2.983, 255: 2.9508, 266: 2.7288, 267: 2.8055, 270: 5.7584, 474: 3.3226, 548: 3.4364, 589: 3.5006, 639: 3.5247, 641: 3.5432, 670: 3.7058, 783: 3.7602, 807: 3.8728, 838: 3.7732, 867: 3.9644, 874: 3.7943, 953: 3.9302, 1113: 4.4063, 1115: 4.1352, 1556: 4.4346, 1740: 4.5357, 1904: 4.6649, 2691: 5.0452, 2745: 5.2668, 2803: 5.1673, 3481: 6.024, 3710: 5.7764, 3979: 5.6781, 4204: 5.5994, 9709: 6.6767, 9775: 6.6927, 10080: 7.1062, 11397: 7.1062, 11497: 7.1062, 13556: 7.2089, 14823: 7.9294, 17291: 7.6009, 23274: 8.3349, 23458: 8.1117, 46723: 9.4335, 96230: 10.1267})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "idf_data.sample(False, 0.001, 1).first().features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- content: string (nullable = true)\n",
      " |-- label: string (nullable = true)\n",
      " |-- name: string (nullable = true)\n",
      " |-- sentiment: string (nullable = true)\n",
      " |-- terms: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- filtered: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- count_vectors: vector (nullable = true)\n",
      " |-- features: vector (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "idf_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply StringIndexer to conver the sentiment column from String type to number type - this is prerequisit to apply the LogisticRegression algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import StringIndexer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "string_indexer = StringIndexer(inputCol=\"sentiment\", outputCol=\"sentiment_idx\")\n",
    "string_indexer_model = string_indexer.fit(idf_data)\n",
    "label_encoded = string_indexer_model.transform(idf_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+-------------+\n",
      "|sentiment|sentiment_idx|\n",
      "+---------+-------------+\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "|      pos|          0.0|\n",
      "+---------+-------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "label_encoded.select(\"sentiment\", \"sentiment_idx\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the data into traininf and testing groups with 70/30 ratio. Cache the dataframe so that training runs faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DataFrame[content: string, label: string, name: string, sentiment: string, terms: array<string>, filtered: array<string>, count_vectors: vector, features: vector, sentiment_idx: double]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training, testing = label_encoded.randomSplit(weights=[0.7, 0.3], seed=1)\n",
    "training.cache()\n",
    "testing.cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the StringIndex has done the expected job and training and testing data maintain the ratio of positive and negative records as in the whole dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+---------+-----+\n",
      "|sentiment_idx|sentiment|count|\n",
      "+-------------+---------+-----+\n",
      "|          1.0|      neg|17588|\n",
      "|          0.0|      pos|17493|\n",
      "+-------------+---------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "training.groupBy(\"sentiment_idx\", \"sentiment\").count().show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+---------+-----+\n",
      "|sentiment_idx|sentiment|count|\n",
      "+-------------+---------+-----+\n",
      "|          1.0|      neg| 7412|\n",
      "|          0.0|      pos| 7507|\n",
      "+-------------+---------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "testing.groupBy(\"sentiment_idx\", \"sentiment\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply LogisticRegression classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.classification import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr = LogisticRegression(maxIter=10000, regParam=0.1, elasticNetParam=0.0, \n",
    "                        featuresCol=\"features\", labelCol=\"sentiment_idx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the parameters that the LogisticRegression classifier takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aggregationDepth: suggested depth for treeAggregate (>= 2). (default: 2)\n",
      "elasticNetParam: the ElasticNet mixing parameter, in range [0, 1]. For alpha = 0, the penalty is an L2 penalty. For alpha = 1, it is an L1 penalty. (default: 0.0, current: 0.0)\n",
      "family: The name of family which is a description of the label distribution to be used in the model. Supported options: auto, binomial, multinomial (default: auto)\n",
      "featuresCol: features column name. (default: features, current: features)\n",
      "fitIntercept: whether to fit an intercept term. (default: True)\n",
      "labelCol: label column name. (default: label, current: sentiment_idx)\n",
      "maxIter: max number of iterations (>= 0). (default: 100, current: 10000)\n",
      "predictionCol: prediction column name. (default: prediction)\n",
      "probabilityCol: Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities. (default: probability)\n",
      "rawPredictionCol: raw prediction (a.k.a. confidence) column name. (default: rawPrediction)\n",
      "regParam: regularization parameter (>= 0). (default: 0.0, current: 0.1)\n",
      "standardization: whether to standardize the training features before fitting the model. (default: True)\n",
      "threshold: Threshold in binary classification prediction, in range [0, 1]. If threshold and thresholds are both set, they must match.e.g. if threshold is p, then thresholds must be equal to [1-p, p]. (default: 0.5)\n",
      "thresholds: Thresholds in multi-class classification to adjust the probability of predicting each class. Array must have length equal to the number of classes, with values > 0, excepting that at most one value may be 0. The class with largest value p/t is predicted, where p is the original probability of that class and t is the class's threshold. (undefined)\n",
      "tol: the convergence tolerance for iterative algorithms (>= 0). (default: 1e-06)\n",
      "weightCol: weight column name. If this is not set or empty, we treat all instance weights as 1.0. (undefined)\n"
     ]
    }
   ],
   "source": [
    "print(lr.explainParams())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lr_model = lr.fit(training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2.51503716e-02,  -1.04120923e-02,  -4.21445079e-03,\n",
       "         1.18781794e-02,  -3.33666996e-02,  -7.94887342e-03,\n",
       "         5.33751771e-02,  -1.53108258e-02,   6.94185422e-04,\n",
       "        -4.11386001e-02,  -6.25835201e-02,   1.83998400e-02,\n",
       "         1.02248816e-01,   4.10455223e-03,  -1.28143344e-02,\n",
       "        -1.18196410e-01,  -3.71174518e-02,  -1.77866941e-02,\n",
       "         1.95544804e-02,   3.13396651e-02,   1.32105214e-04,\n",
       "        -7.52385438e-04,   9.64050948e-03,  -2.51652295e-02,\n",
       "        -1.25783662e-02,   6.92955948e-03,  -9.04940252e-03,\n",
       "        -2.11651342e-03,  -1.24592921e-02,  -2.08417761e-02,\n",
       "        -4.71834211e-02,   5.46507832e-03,   5.07141996e-02,\n",
       "        -2.66495203e-02,   5.37636201e-02,  -5.69553007e-03,\n",
       "        -7.21254668e-02,  -2.63271603e-03,  -9.25606328e-03,\n",
       "         1.24997605e-02,  -1.06348790e-02,   4.13027466e-02,\n",
       "        -3.81248996e-03,   3.17777254e-03,  -4.69227273e-02,\n",
       "         1.32144342e-03,   1.13002078e-02,   2.00829754e-03,\n",
       "         1.85775716e-02,   2.12438504e-02,  -5.32934064e-03,\n",
       "        -1.18813821e-02,  -1.68511771e-02,   4.11861584e-02,\n",
       "         2.57282250e-02,   2.41306731e-02,   1.35062263e-02,\n",
       "         2.26315297e-02,   2.65258022e-02,   3.14338915e-02,\n",
       "        -2.88148333e-02,  -7.98606670e-03,  -1.35765102e-02,\n",
       "         5.34864698e-03,   1.31468506e-02,   2.16080556e-04,\n",
       "        -2.01497040e-02,   2.20020455e-02,   7.86800262e-02,\n",
       "        -3.00290661e-02,   4.17093967e-03,  -1.79565040e-02,\n",
       "         8.66556925e-03,  -7.38618594e-03,  -3.04300759e-02,\n",
       "        -8.58851892e-03,   1.85936984e-03,  -2.56014534e-02,\n",
       "        -1.23312956e-02,  -1.43582423e-02,  -1.29405231e-02,\n",
       "         1.07360208e-02,  -2.38923353e-02,   1.36855590e-02,\n",
       "         5.28180281e-03,   6.87347247e-04,   2.01543382e-02,\n",
       "        -1.37910872e-02,  -7.48063131e-03,   8.95914615e-05,\n",
       "         1.84630623e-03,  -3.04174861e-03,  -8.16288249e-03,\n",
       "         7.34015148e-03,   2.34041705e-02,  -2.61610589e-03,\n",
       "         9.46615713e-03,  -1.55757717e-02,  -6.77275095e-03,\n",
       "        -1.54926610e-02])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_model.coefficients[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the training summary find out the cost decay of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_summary = lr_model.summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x10deeac18>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGoxJREFUeJzt3X2UHXWd5/H3p+9Nd+juhKc0SUwCAQ0gYMCZBoQFgR3R\ngGhEZ3gQx/E4ayZzFscZ50GYOTuHPXs8OsOuy87ImmWUdcZ1zM4MAlmIBFAUFJUkEEIeCIQokECS\nJiBJJ+Shu7/7x62+XNp+uJ2uSnXdfF7n9Mm9VdV1v6mT7k9+v1/V76eIwMzMDKAp7wLMzGz8cCiY\nmVmVQ8HMzKocCmZmVuVQMDOzKoeCmZlVORTMzKzKoWBmZlUOBTMzqyrnXcBoTZkyJWbPnp13GWZm\nhbJy5cpXIqJjpOMKFwqzZ89mxYoVeZdhZlYokp6v5zh3H5mZWZVDwczMqjINBUnzJG2QtFHSDYPs\n/3NJq5KvNZJ6JR2TZU1mZja0zEJBUgm4FbgMOA24VtJptcdExM0RcVZEnAXcCPwoIl7NqiYzMxte\nli2Fc4CNEbEpIvYDi4H5wxx/LfCdDOsxM7MRZBkKM4AXa95vTrb9GkmtwDzgjgzrMTOzEYyXgeYP\nAT8ZqutI0gJJKySt6OrqOsSlmZkdPrIMhS3ArJr3M5Ntg7mGYbqOIuK2iOiMiM6YODnFEs3MrFaW\nobAcmCPpREnNVH7xLxl4kKQjgYuAu+s56fade1Mt0szM3pTZE80R0SPpemAZUAJuj4i1khYm+xcl\nh14J3B8Ru+s6L9DT20e5NF56vszMGociIu8aRqVl+pzYvmkdRx4xIe9SzMwKQ9LKiOgc6bhC/nd7\nz/6evEswM2tIhQyF3fscCmZmWShoKPTmXYKZWUMqaCi4pWBmloVihsJ+txTMzLJQzFBwS8HMLBPF\nDAXffWRmlolihoJbCmZmmShoKHhMwcwsC4ULhSbJLQUzs4wUMBR895GZWVYKGApuKZiZZaV4odAk\nz31kZpaRwoVCSdDtloKZWSYKFwpNEns8pmBmlonihUKT3FIwM8tI8UJBYo+fUzAzy0QBQ8FPNJuZ\nZaVwoVBqErv391C0ZUTNzIqgcKHQJNEXsPdAX96lmJk1nOKFQpMAz5RqZpaF4oVCJRM8rmBmloHC\nhUJJSUvBdyCZmaWucKHQJHcfmZllpXih0D+m4O4jM7PUFS8UqmMK7j4yM0tbAUPB3UdmZlkpXii4\n+8jMLDOFC4X+u488U6qZWfoKFwoSlD1TqplZJjINBUnzJG2QtFHSDUMcc7GkVZLWSvpRPedtaymz\nx6FgZpa6clYnllQCbgUuBTYDyyUtiYh1NcccBfxPYF5EvCDpuHrO3dZcott3H5mZpS7LlsI5wMaI\n2BQR+4HFwPwBx3wc+G5EvAAQEdvrOXFbS9nrNJuZZSDLUJgBvFjzfnOyrdbJwNGSfihppaRPDnYi\nSQskrZC0oquri9aWsscUzMwykPdAcxn4TeCDwAeA/yTp5IEHRcRtEdEZEZ0dHR20t5R895GZWQYy\nG1MAtgCzat7PTLbV2gzsiIjdwG5JDwNnAs8Md+LW5jI7uvekWauZmZFtS2E5MEfSiZKagWuAJQOO\nuRu4QFJZUitwLrB+pBO3t5T9RLOZWQYyaylERI+k64FlQAm4PSLWSlqY7F8UEesl3QesBvqAr0fE\nmpHO3dpc8txHZmYZyLL7iIhYCiwdsG3RgPc3AzeP5rztLWVPc2FmloG8B5oPSmtzmX09ffT0ep1m\nM7M0FTIU2lpKAOz2HUhmZqkqaChUer3chWRmlq5Ch4KfajYzS1cxQ6G50n3k+Y/MzNJVzFDobym4\n+8jMLFXFDIXmSih4/iMzs3QVMxSSu488/5GZWboKGgpuKZiZZaHQoeC7j8zM0lXIUGid4LuPzMyy\nUMhQaGoSrc0l331kZpayQoYCVOY/8vTZZmbpKmwotLd4+mwzs7QVNhRamz19tplZ2gobCl59zcws\nfYUNhVZ3H5mZpa6wodDmloKZWeqKGwrNJY8pmJmlrLih0FJmj7uPzMxSVdxQSJ5TiIi8SzEzaxjF\nDYWWMn0Bew/05V2KmVnDKHAo9M9/5HEFM7O0FDcUmj1TqplZ2oobCm4pmJmlrsCh0N9S8B1IZmZp\nKWwotHqdZjOz1BU2FNr7Wwp+VsHMLDWFDYXW5sqYgp9qNjNLT2FDob+l4PmPzMzSk2koSJonaYOk\njZJuGGT/xZJel7Qq+frres/d2uKWgplZ2spZnVhSCbgVuBTYDCyXtCQi1g049JGIuGK0528pl5hQ\nErt995GZWWqybCmcA2yMiE0RsR9YDMxP8wO8+pqZWbqyDIUZwIs17zcn2wY6X9JqSd+TdPpgJ5K0\nQNIKSSu6urqq29tbyl5ox8wsRXkPND8OHB8Rc4G/B+4a7KCIuC0iOiOis6Ojo7q91WsqmJmlKstQ\n2ALMqnk/M9lWFRE7I6I7eb0UmCBpSr0f4NXXzMzSlWUoLAfmSDpRUjNwDbCk9gBJ0yQpeX1OUs+O\nej+grcUtBTOzNGV291FE9Ei6HlgGlIDbI2KtpIXJ/kXAbwN/KKkHeAO4Jkaxak5bc5kd3fszqN7M\n7PCUWShAtUto6YBti2pefxX46sGev62l7LmPzMxSlPdA85i0Npc8S6qZWYoKHQrtbimYmaWq0KHQ\n2lxmf08fB3q9TrOZWRoKHQr9q695+mwzs3QUPBQ8U6qZWZoaIxQ8rmBmlopih0L/Qju+A8nMLBXF\nDgW3FMzMUlXsUGh2KJiZpamuUJD0rXq2HWrVu4/cfWRmlop6WwpvWecgWVXtN9MvZ3T6u4/8AJuZ\nWTqGDQVJN0raBcyVtDP52gVsB+4+JBUOoz8U9viWVDOzVAwbChHxpYiYBNwcEZOTr0kRcWxE3HiI\nahxS64RK91G3H14zM0tFvd1H90hqA5D0CUlfkXRChnXVpalJlUnx3H1kZpaKekPha8AeSWcCfwo8\nB/xTZlWNQmuzV18zM0tLvaHQkyx+Mx/4akTcCkzKrqz6tbeU2O3uIzOzVNS7yM4uSTcCvwtcKKkJ\nmJBdWfVrbS77OQUzs5TU21K4GtgHfDoitgIzgZszq2oU2lvcfWRmlpa6QiEJgm8DR0q6AtgbEeNj\nTMHdR2Zmqan3ieargMeA3wGuAn4u6bezLKxebW4pmJmlpt4xhb8Czo6I7QCSOoAHgX/LqrB6tTWX\nPKZgZpaSescUmvoDIbFjFN+bqbaWsldeMzNLSb0thfskLQO+k7y/GliaTUmj05Y8pxARSMq7HDOz\nQhs2FCS9A5gaEX8u6aPABcmun1IZeM5dW0uZvoC9B/o4Ill0x8zMDs5IXUC3ADsBIuK7EfH5iPg8\ncGeyL3f902d7plQzs7EbKRSmRsRTAzcm22ZnUtEo9S+045lSzczGbqRQOGqYfUekWcjBckvBzCw9\nI4XCCkmfGbhR0n8AVmZT0ui8uaaC70AyMxurke4++mPgTknX8WYIdALNwJVZFlav1mavvmZmlpZh\nQyEitgHnS7oEOCPZfG9E/CDzyurU3t9S8LMKZmZjVu/cRw9FxN8nX3UHgqR5kjZI2ijphmGOO1tS\nz8FMndGa3Ibqp5rNzMYus6eSJZWAW4HLgNOAayWdNsRxfwPcfzCf099S8PxHZmZjl+VUFecAGyNi\nU0TsBxZTWaRnoM8CdwDbB9k3otYWtxTMzNKSZSjMAF6seb852VYlaQaVAeuvDXciSQskrZC0oqur\n6y37WsolJpTEbt99ZGY2ZnlPancL8IWI6BvuoIi4LSI6I6Kzo6Pj1/Z79TUzs3TUOyHewdgCzKp5\nPzPZVqsTWJxMZDcFuFxST0TcNZoPam8pe6EdM7MUZBkKy4E5kk6kEgbXAB+vPSAiTux/LembwD2j\nDQSo3IHkloKZ2dhlFgoR0SPpemAZUAJuj4i1khYm+xel9Vlefc3MLB1ZthSIiKUMWHdhqDCIiE8d\n7Oe0tbilYGaWhrwHmlPR1lz23EdmZilojFBoKXvuIzOzFDRIKJTcUjAzS0FjhEKzWwpmZmlojFBo\nKbO/p48DvcM+A2dmZiNoiFDonynV02ebmY1NQ4SCZ0o1M0tHQ4RCa38oeFzBzGxMGiIU2vunz/Yd\nSGZmY9IQodC/TrNbCmZmY9MQodDu7iMzs1Q0RChU12n2QLOZ2Zg0RCi82VLwmIKZ2Vg0RCj47iMz\ns3Q0RihM8N1HZmZpaIhQaGqSV18zM0tBQ4QCVOY/2uOBZjOzMWmcUGgu0e2BZjOzMWmcUGgps8fd\nR2ZmY9I4oeA1FczMxqxxQsGrr5mZjVnDhEJrS9l3H5mZjVHDhEJ7c9nTXJiZjVHDhEJrS8nTXJiZ\njVHDhEJ7S6WlEBF5l2JmVlgNEwqtzWUi4I0Dbi2YmR2shgmF6upr7kIyMztoDRMKXn3NzGzsGiYU\n2vqnz/YdSGZmB62BQsHdR2ZmY5VpKEiaJ2mDpI2Sbhhk/3xJqyWtkrRC0gUH+1luKZiZjV05qxNL\nKgG3ApcCm4HlkpZExLqaw74PLImIkDQX+Bfg1IP5vDaPKZiZjVmWLYVzgI0RsSki9gOLgfm1B0RE\nd7z5YEEbcNAPGfR3H+1x95GZ2UHLMhRmAC/WvN+cbHsLSVdKehq4F/j0YCeStCDpXlrR1dU16If1\ntxQ8U6qZ2cHLfaA5Iu6MiFOBjwD/ZYhjbouIzojo7OjoGPQ8/WMKXn3NzOzgZRkKW4BZNe9nJtsG\nFREPAydJmnIwH9ZcbmJCSV59zcxsDLIMheXAHEknSmoGrgGW1B4g6R2SlLz+DaAF2HGwHzjr6FbW\nbHl9DCWbmR3eMguFiOgBrgeWAeuBf4mItZIWSlqYHPYxYI2kVVTuVLo6xjCj3WXvmsajz73CK937\nxlq+mdlhKdMxhYhYGhEnR8TbI+KLybZFEbEoef03EXF6RJwVEedFxI/H8nlXzH0bfQHfW7M1jfLN\nzA47uQ80p+nUaZM4qaONe1e/lHcpZmaF1FChIIkr5r6Nn//iVbbv3Jt3OWZmhdNQoQDwobnTiYCl\nT72cdylmZoXTcKEwZ+okTpk6iXsdCmZmo9ZwoQDwwbnTWf7L13j59TfyLsXMrFAaMhSumDsdgHtX\nu7VgZjYaDRkKJ3W0c9r0ye5CMjMbpYYMBah0IT3xwq/Y/NqevEsxMyuMhg2FD819G+AuJDOz0WjY\nUDj+2FbmzjySexwKZmZ1a9hQgMqA81NbXuf5HbvzLsXMrBAaOhQuf1flLiS3FszM6tPQoTDz6Fbe\nffxRDgUzszo1dChAZebU9S/v5Lmu7rxLMTMb9xo+FC5/1zTAdyGZmdWj4UNh+pFHcPbso7nH02mb\nmY2o4UMBKl1Iz2zr5pltu/IuxcxsXDssQuGyd01D8l1IZmYjOSxC4bhJEzn3xGO4Z/VLjGEJaDOz\nhndYhALAh8+cwaau3fzpvz7Jju59eZdjZjYuHTahcFXnTK6/5B38vydf4re+8iP+7/IX6Otzq8HM\nrNZhEwrlUhN/9oFTWPpHF3LycZP4wh1Pcc1tP+NZDz6bmVUdNqHQb87USSxe8B7+9mNzeWb7Li7/\nu0e4ednT7D3Qm3dpZma5O+xCAaCpSVx19iy+//mL+NCZb+PWh57j/f/9YR55tivv0szMcnVYhkK/\nY9tb+MpVZ/HPnzmXcpP43W88xo3ffYrufT15l2ZmlovDOhT6nf/2KSz93IUseO9JLF7+AvNueZif\nPrcj77LMzA45h0Ji4oQSf3n5O/nXPziPcpO49h9+xk1L1vLGfo81mNnhw6EwQOfsY1j6uQv51Pmz\n+eajv+Tyv3uElc+/mndZZmaHhENhEK3NZW768On882fOZX9PH7+z6Kd8ael6Xtu9P+/SzMwylWko\nSJonaYOkjZJuGGT/dZJWS3pK0qOSzsyyntE6/+1TWPYn7+Xqs2fxvx7eROcXH+Ta237G//7JL9j8\n2p68yzMzS52ymgtIUgl4BrgU2AwsB66NiHU1x5wPrI+I1yRdBtwUEecOd97Ozs5YsWJFJjUPZ82W\n17lvzVaWrd3Ks9srC/acMWMy7z9tGh84fRonT21H0iGvy8ysHpJWRkTniMdlGArnUfkl/4Hk/Y0A\nEfGlIY4/GlgTETOGO29eoVBrU1c396/bxv1rt/L4C78C4IRjW3nfO6fyvndOpXP20UwouWfOzMaP\nekOhnGENM4AXa95vBoZrBfw+8L0M60nNSR3tLLyonYUXvZ3tO/fywPpt3L92G9/66fN848e/YPLE\nMpecehzve+dULjqlg8kTJ+RdsplZXbIMhbpJuoRKKFwwxP4FwAKA448//hBWNrLjJk/kunNP4Lpz\nT2D3vh4eefYVHly/jR88vZ27V71EuUm856RjmXfGNK6YO52jWpvzLtnMbEi5dx9JmgvcCVwWEc+M\ndN7x0H1Uj96+4IkXXuOB9dt4YN02NnXtZkJJXHLKcVz57hlccupxTJxQyrtMMztMjIcxhTKVgebf\nArZQGWj+eESsrTnmeOAHwCcj4tF6zluUUKgVEax9aSd3PbGFu598ia5d+5g8scwH507nI2fN4OzZ\nx9DU5EFqM8tO7qGQFHE5cAtQAm6PiC9KWggQEYskfR34GPB88i09IxVdxFCo1dPbx6PP7eCuJ7Zw\n39qt7Nnfy/QjJzJ35pGcOm0yp06bxCnTJnHCsW2UHBRmlpJxEQpZKHoo1Nqzv4cH1lUGqde/vJNf\n7thN/7o/Eyc0cfLUSZwydRLHH9PKpIll2lrKTJpYpr1lAm0tpeq2lnKJckk0l5ooN4lSk3x7rJm9\nxXi4+8hG0NpcZv5ZM5h/VuUu3Df29/Ls9l08vXUXG7bu4umtO3low3Ze6R7dk9QSTGhqYkKpEhBN\nTUKAJCqNDyFBk6Cy583vq+ytPddbw6V6TPXYocNnLLk00reOJfRG/M5hDsiyrpE45u1QcCiMI0c0\nl5g78yjmzjzqLdv39fSye18vu/f1sGtvD937eiqv9/XQvbeHA719yVdwoLePnt4+9vcGPb199PQF\nEUFfQBBEkLRGgr6+Nz8jqDRRahuO/S/7t/UfM+CPQY2lBTrSd46lcTvyuYc+YsSPzbDRHVme3A4L\nD9Z5nEOhAFrKJVrKJY5p8+2sZnZwvvaJ+o7zY7dmZlblUDAzsyqHgpmZVTkUzMysyqFgZmZVDgUz\nM6tyKJiZWZVDwczMqgo395GkXcCGvOsYxBTglbyLGITrGh3XNTrjtS4Yv7XlVdcJEdEx0kFFfKJ5\nQz2TOh1qkla4rvq5rtFxXaM3Xmsbr3X1c/eRmZlVORTMzKyqiKFwW94FDMF1jY7rGh3XNXrjtbbx\nWhdQwIFmMzPLThFbCmZmlpFChYKkeZI2SNoo6Ya86+kn6ZeSnpK0SlJua4VKul3SdklrarYdI+kB\nSc8mfx49Tuq6SdKW5JqtStbzPtR1zZL0kKR1ktZK+lyyPddrNkxduV4zSRMlPSbpyaSu/5xsz/t6\nDVVX7v/GkjpKkp6QdE/yPvefyeEUpvtIUgl4BrgU2AwsB66NiHW5FkYlFIDOiMj1nmhJ7wW6gX+K\niDOSbX8LvBoRX06C9OiI+MI4qOsmoDsi/uuhrGVAXdOB6RHxuKRJwErgI8CnyPGaDVPXVeR4zVRZ\na7QtIrolTQB+DHwO+Cj5Xq+h6ppHzv/Gkvo+D3QCkyPiivHwMzmcIrUUzgE2RsSmiNgPLAbm51zT\nuBIRDwOvDtg8H/jH5PU/UvnlckgNUVfuIuLliHg8eb0LWA/MIOdrNkxduYqK7uTthOQryP96DVVX\n7iTNBD4IfL1mc+4/k8MpUijMAF6seb+ZcfCDkgjgQUkrJS3Iu5gBpkbEy8nrrcDUPIsZ4LOSVifd\nS7k2oSXNBt4N/JxxdM0G1AU5X7OkK2QVsB14ICLGxfUaoi7I/9/YLcBfADUroud/vYZTpFAYzy6I\niLOAy4D/mHSXjDtR6SscF/+DAr4GnAScBbwM/Le8CpHUDtwB/HFE7Kzdl+c1G6Su3K9ZRPQm/9Zn\nAudIOmPA/lyu1xB15Xq9JF0BbI+IlUMdM85+JoFihcIWYFbN+5nJttxFxJbkz+3AnVS6usaLbUkf\ndX9f9fac6wEgIrYlP8h9wD+Q0zVL+qDvAL4dEd9NNud+zQara7xcs6SWXwEPUem3z/16DVbXOLhe\n/w74cDLmuBj495L+D+Poeg2mSKGwHJgj6URJzcA1wJKca0JSWzIYiKQ24P3AmuG/65BaAvxe8vr3\ngLtzrKWq/4cicSU5XLNkgPIbwPqI+ErNrlyv2VB15X3NJHVIOip5fQSVmz6eJv/rNWhdeV+viLgx\nImZGxGwqv69+EBGfYJz+TFZFRGG+gMup3IH0HPBXedeT1HQS8GTytTbPuoDvUGkmH6Ay5vL7wLHA\n94FngQeBY8ZJXd8CngJWU/khmZ5DXRdQabqvBlYlX5fnfc2GqSvXawbMBZ5IPn8N8NfJ9ryv11B1\n5f5vrKbGi4F7xsP1GumrMLekmplZ9orUfWRmZhlzKJiZWZVDwczMqhwKZmZW5VAwM7Mqh4IdtiR1\nJ3/OlvTxlM/9lwPeP5rm+c2y4lAwg9nAqEJBUnmEQ94SChFx/ihrMsuFQ8EMvgxcmMy5/yfJ5Go3\nS1qeTKb2BwCSLpb0iKQlwLpk213JRIhr+ydDlPRl4IjkfN9OtvW3SpSce40qa3BcXXPuH0r6N0lP\nS/p28mSz2SE10v92zA4HNwB/FhFXACS/3F+PiLMltQA/kXR/cuxvAGdExC+S95+OiFeT6RWWS7oj\nIm6QdH1UJmgb6KNUJmg7E5iSfM/Dyb53A6cDLwE/oTJ3zo/T/+uaDc0tBbNf937gk8lUzD+nMi3B\nnGTfYzWBAPBHkp4EfkZlwsY5DO8C4DtRmahtG/Aj4Oyac2+OygRuq6h0a5kdUm4pmP06AZ+NiGVv\n2ShdDOwe8P59wHkRsUfSD4GJY/jcfTWve/HPp+XALQUz2AVMqnm/DPjDZPpqJJ2czIA70JHAa0kg\nnAq8p2bfgf7vH+AR4Opk3KIDeC/wWCp/C7MU+H8iZpVZNHuTbqBvAv+DStfN48lgbxeDL5l4H7BQ\n0npgA5UupH63AaslPR4R19VsvxM4j8qsugH8RURsTULFLHeeJdXMzKrcfWRmZlUOBTMzq3IomJlZ\nlUPBzMyqHApmZlblUDAzsyqHgpmZVTkUzMys6v8DOfMjTsUnmYIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10de4a048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.Series(training_summary.objectiveHistory).plot()\n",
    "plt.xlabel(\"Iteration\")\n",
    "plt.ylabel(\"Cost\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find area under the curve. Closer to 1 is better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9998771122211515"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_summary.areaUnderROC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = lr_model.transform(testing).withColumn(\"match\", expr(\"prediction == sentiment_idx\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+-------------+---------+-----+\n",
      "|prediction|sentiment_idx|sentiment|match|\n",
      "+----------+-------------+---------+-----+\n",
      "|       1.0|          1.0|      neg| true|\n",
      "|       1.0|          1.0|      neg| true|\n",
      "|       1.0|          1.0|      neg| true|\n",
      "|       1.0|          1.0|      neg| true|\n",
      "|       0.0|          0.0|      pos| true|\n",
      "|       1.0|          1.0|      neg| true|\n",
      "|       0.0|          0.0|      pos| true|\n",
      "|       0.0|          0.0|      pos| true|\n",
      "|       0.0|          0.0|      pos| true|\n",
      "|       0.0|          0.0|      pos| true|\n",
      "+----------+-------------+---------+-----+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.select(\"prediction\", \"sentiment_idx\", \"sentiment\", \"match\").sample(False, 0.01).show(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+----+----+\n",
      "|sentiment_idx| 0.0| 1.0|\n",
      "+-------------+----+----+\n",
      "|          0.0|6779| 728|\n",
      "|          1.0| 890|6522|\n",
      "+-------------+----+----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.groupBy(\"sentiment_idx\").pivot(\"prediction\").count().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the accuracy of the prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8915476908639989"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy = predictions.select(expr(\"sum(cast(match as int))\")).first()[0] / predictions.count()\n",
    "accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}