IMDB sentiment analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB Movie Review sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training set contains [pos/, neg/] directories for the reviews with binary labels positive and negative. Within these directories, reviews are stored in text files named following the convention [[id]_[rating].txt] where [id] is a unique id and [rating] is the star rating for that review on a 1-10 scale. For example, the file [train/pos/200_8.txt] is the text for a positive-labeled train set example with unique id 200 and star rating 8/10 from IMDb. \n",
    "\n",
    "The test set contains 11000 files without labels. Here our task is to label these files with 0 or 1, which is negative or positive. Using the training set, you should get the score for each file. If the score is <= 4, then the label is 0. Or if the score is >=7, then the label is 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the original dataset I used for this project. \n",
    "\n",
    "http://ai.stanford.edu/~amaas/data/sentiment/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "onlyfiles = [f for f in listdir(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/pos\") if isfile(join(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/pos\", f))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12500"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(onlyfiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "12500 movies reviews in training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "name =[]\n",
    "for path in onlyfiles:\n",
    "    os.path.splitext(path)\n",
    "    name.append(os.path.splitext(path)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12500"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We need to first split movie id and rating from given  file name(eg: \"1221_8.txt\" means Id=1221 rating=8)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "movieId=[]\n",
    "rating=[]\n",
    "for feat in name:\n",
    "    movieId.append(feat.rsplit( \"_\", 1 )[ 0 ])\n",
    "    rating.append(feat.rsplit( \"_\", 1 )[ 1 ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a list of reviews and append all reviews by reading each file in a pos (positive review dir) folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "review =[]\n",
    "for path in onlyfiles:\n",
    "    txt = open(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/pos/\"+path)\n",
    "    review.append(txt.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Pandas Dataframe of reviews and given sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# First create a dictionary of review and target(sentiment column)\n",
    "movie_dict={\"review\":review,\"target\":1} \n",
    "movie_df=pd.DataFrame(data=movie_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As all reviews from \"pos\" folder are positive review, assign 1 to the target column for positive sentiment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Hitchcock displays his already developed under...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A drifter looking for a job is mistaken for a ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>This movie resonated with me on two levels. As...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I liked Timothy Dalton very much even though h...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The movie is just plain fun....maybe more fun ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  target\n",
       "0  Hitchcock displays his already developed under...       1\n",
       "1  A drifter looking for a job is mistaken for a ...       1\n",
       "2  This movie resonated with me on two levels. As...       1\n",
       "3  I liked Timothy Dalton very much even though h...       1\n",
       "4  The movie is just plain fun....maybe more fun ...       1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeat the above process for \"neg\" folder which contains negative sentiment review files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "onlyfilesNeg = [f for f in listdir(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/neg\") if isfile(join(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/neg\", f))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "name =[]\n",
    "for path in onlyfilesNeg:\n",
    "    os.path.splitext(path)\n",
    "    name.append(os.path.splitext(path)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "movieId=[]\n",
    "rating=[]\n",
    "for feat in name:\n",
    "    movieId.append(feat.rsplit( \"_\", 1 )[ 0 ])\n",
    "    rating.append(feat.rsplit( \"_\", 1 )[ 1 ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "review =[]\n",
    "for path in onlyfilesNeg:\n",
    "    txt = open(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/train/neg/\"+path)\n",
    "    review.append(txt.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "movie_dict_neg={\"review\":review,\"target\":0}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Feeding The Masses was just another movie tryi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The infamous Ed Wood \"classic\" Plan 9 From Out...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Holy cow, what a piece of sh*t this movie is. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Kevin Kline and Meg Ryan are among that class ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>In the words of Charles Dance's character in t...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  target\n",
       "0  Feeding The Masses was just another movie tryi...       0\n",
       "1  The infamous Ed Wood \"classic\" Plan 9 From Out...       0\n",
       "2  Holy cow, what a piece of sh*t this movie is. ...       0\n",
       "3  Kevin Kline and Meg Ryan are among that class ...       0\n",
       "4  In the words of Charles Dance's character in t...       0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie_df1=pd.DataFrame(data=movie_dict_neg)\n",
    "movie_df1.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine both positive and negative sentiment dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "movieDF=movie_df1.append(movie_df,ignore_index = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Feeding The Masses was just another movie tryi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The infamous Ed Wood \"classic\" Plan 9 From Out...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Holy cow, what a piece of sh*t this movie is. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Kevin Kline and Meg Ryan are among that class ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>In the words of Charles Dance's character in t...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              review  target\n",
       "0  Feeding The Masses was just another movie tryi...       0\n",
       "1  The infamous Ed Wood \"classic\" Plan 9 From Out...       0\n",
       "2  Holy cow, what a piece of sh*t this movie is. ...       0\n",
       "3  Kevin Kline and Meg Ryan are among that class ...       0\n",
       "4  In the words of Charles Dance's character in t...       0"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movieDF.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24995</th>\n",
       "      <td>Director Otto Preminger reunites with his Laur...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24996</th>\n",
       "      <td>I was brought up on Doc Savage,and was petrifi...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24997</th>\n",
       "      <td>God Bless 80's slasher films. This is a fun, f...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24998</th>\n",
       "      <td>I love ghost stories in general, but I PARTICU...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24999</th>\n",
       "      <td>This movie was very good. If you are one who l...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  review  target\n",
       "24995  Director Otto Preminger reunites with his Laur...       1\n",
       "24996  I was brought up on Doc Savage,and was petrifi...       1\n",
       "24997  God Bless 80's slasher films. This is a fun, f...       1\n",
       "24998  I love ghost stories in general, but I PARTICU...       1\n",
       "24999  This movie was very good. If you are one who l...       1"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movieDF.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "target=movieDF.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now time for some Natural Language Processing stuff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import all necessary packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cross_validation import train_test_split\n",
    "from sklearn import naive_bayes\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TFIDF Vectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopset = set(stopwords.words('english'))\n",
    "vectorizer = TfidfVectorizer(use_idf=True, lowercase=True, strip_accents='ascii', stop_words=stopset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert movieDF.review from text to features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X= vectorizer.fit_transform(movieDF.review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000,)\n",
      "(25000, 74681)\n"
     ]
    }
   ],
   "source": [
    "print (target.shape)\n",
    "print (X.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Train Split as usual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test,y_train, y_test = train_test_split(X, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will train a Naive Bayes classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = naive_bayes.MultinomialNB()\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.93903005610950296"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We can test our model's accuracy like this:\n",
    "roc_auc_score(y_test, clf.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let test our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "movie_reviews_array=np.array([\"Jupiter Ascending was a disapointing and terrible movie\"])\n",
    "\n",
    "movie_review_vector = vectorizer.transform(movie_reviews_array)\n",
    "\n",
    "print (clf.predict(movie_review_vector))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\n"
     ]
    }
   ],
   "source": [
    "movie_reviews_array=np.array([\"Jupiter Ascending was a not disapointing and awesome movie\"])\n",
    "\n",
    "movie_review_vector = vectorizer.transform(movie_reviews_array)\n",
    "\n",
    "print (clf.predict(movie_review_vector))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#we will train a naive_bayes classifier\n",
    "model = naive_bayes.MultinomialNB()\n",
    "model.fit(X, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-process test data like train data as above :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import glob, os\n",
    "review =[]\n",
    "file_names=[]\n",
    "os.chdir(\"/home/rajat/Dev/Python/Kaggle/Kaggle Data/IMDB sentiment Analysis/test\")\n",
    "for file in glob.glob(\"*.txt\"):\n",
    "    file_names.append(file)\n",
    "    txt = open(file)\n",
    "    review.append(txt.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11000"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11000"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(file_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "name =[]\n",
    "for path in file_names:\n",
    "    os.path.splitext(path)\n",
    "    name.append(os.path.splitext(path)[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "testDict={\"id\":name,\"review\":review}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1337</td>\n",
       "      <td>This movie was awful, plain and simple! The an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8398</td>\n",
       "      <td>This movie has everything typical horror movie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1604</td>\n",
       "      <td>Jim Henson's The Muppet Movie is a charming, f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>67</td>\n",
       "      <td>I am an avid movie watcher and I enjoy a wide ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9699</td>\n",
       "      <td>The reason why this movie sucks, have these pe...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     id                                             review\n",
       "0  1337  This movie was awful, plain and simple! The an...\n",
       "1  8398  This movie has everything typical horror movie...\n",
       "2  1604  Jim Henson's The Muppet Movie is a charming, f...\n",
       "3    67  I am an avid movie watcher and I enjoy a wide ...\n",
       "4  9699  The reason why this movie sucks, have these pe..."
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df=pd.DataFrame(data=testDict)\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rajat/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:1: FutureWarning: convert_objects is deprecated.  Use the data-type specific converters pd.to_datetime, pd.to_timedelta and pd.to_numeric.\n",
      "  if __name__ == '__main__':\n"
     ]
    }
   ],
   "source": [
    "test_df['id'] = test_df['id'].convert_objects(convert_numeric=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id         int64\n",
       "review    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform to lower-case, Remove the punctuations, Remove the stopwrods, Tokenize the remaining string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## For more info, see \n",
    "\n",
    "stemmer = nltk.stem.porter.PorterStemmer()\n",
    "\n",
    "def get_tokens(inp_txt):\n",
    "    \n",
    "    ## Lower case: ABC -> abc\n",
    "    txt_lower = inp_txt.lower()\n",
    "  \n",
    "    ## Remove punctuations (!, ', \", ., :, ;, )\n",
    "    #txt_lower_nopunct = txt_lower.translate(string.maketrans(\"\",\"\"), string.punctuation)\n",
    "    #print(txt_lower_nopunct)\n",
    "    \n",
    "    \n",
    "    ## Tokenize:\n",
    "    tokens = nltk.word_tokenize(txt_lower) #_nopunct)\n",
    "    #tokens = nltk.wordpunct_tokenize(txt_lower)\n",
    "    \n",
    "    ## remove stop-words:\n",
    "    tokens_filtered = [w for w in tokens if not w in stopwords.words('english')]\n",
    "    \n",
    "    ## stemming:\n",
    "    stems = [stemmer.stem(t) for t in tokens_filtered]\n",
    "    stems_nopunct = [s for s in stems if re.match('^[a-zA-Z]+$', s) is not None]\n",
    "    return (stems_nopunct)clf.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF Feature Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import scipy\n",
    "import sklearn\n",
    "tfidf = sklearn.feature_extraction.text.TfidfVectorizer(\n",
    "    encoding = 'utf-8',\n",
    "    decode_error = 'replace',\n",
    "    strip_accents = 'ascii',\n",
    "    analyzer = 'word',\n",
    "    smooth_idf = True,\n",
    "    tokenizer = get_tokens\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorizing the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of samples N= 25000,  Number of features d= 74536\n",
      "Number of Test Documents: 11000,  Number of features: 74536\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(sublinear_tf=True, max_df=0.5,\n",
    "                             stop_words='english')\n",
    "X_train = vectorizer.fit_transform(movieDF.review)\n",
    "\n",
    "print(\"Number of samples N= %d,  Number of features d= %d\" % X_train.shape)\n",
    "\n",
    "\n",
    "### Transforming the test dataset:\n",
    "X_test = vectorizer.transform(test_df.review)\n",
    "print(\"Number of Test Documents: %d,  Number of features: %d\" %X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We will train our model using Naive Bayes classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = naive_bayes.MultinomialNB()\n",
    "model.fit(X_train, target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "preds=model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, ..., 1, 0, 0])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "moviesid=test_df.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11000"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rajat/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:5: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)\n"
     ]
    }
   ],
   "source": [
    "submission = pd.DataFrame()\n",
    "submission[\"id\"] = test_df[\"id\"]\n",
    "submission[\"labels\"] = preds\n",
    "\n",
    "submission=submission.sort(\"id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4370</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9261</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2444</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8185</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5201</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  labels\n",
       "4370   0       0\n",
       "9261   1       0\n",
       "2444   2       0\n",
       "8185   3       0\n",
       "5201   4       0"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10856</th>\n",
       "      <td>10995</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10463</th>\n",
       "      <td>10996</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1364</th>\n",
       "      <td>10997</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4747</th>\n",
       "      <td>10998</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5215</th>\n",
       "      <td>10999</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  labels\n",
       "10856  10995       1\n",
       "10463  10996       1\n",
       "1364   10997       0\n",
       "4747   10998       0\n",
       "5215   10999       0"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submission.to_csv('/home/rajat/Dev/Python/solution.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets try xgboost to build our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import xgboost as xgb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "xgtrain = xgb.DMatrix(X_train,target)\n",
    "xgtest = xgb.DMatrix(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xgboost_params = { \"objective\":\"binary:logistic\",    # binary classification \n",
    "              \"eval_metric\" : \"error\",    # evaluation metric \n",
    "              \"nthread\" : 4,   # number of threads to be used \n",
    "              \"max_depth\": 5,    # maximum depth of tree \n",
    "              \"eta\" : 0.02\n",
    "                  }\n",
    "num_round = 10\n",
    "bst = xgb.train(xgboost_params, xgtrain, num_round);\n",
    "\n",
    "# get prediction\n",
    "pred = bst.predict(xgtest);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.43140259  0.51734471  0.54881501 ...,  0.51734471  0.54881501\n",
      "  0.4520092 ]\n"
     ]
    }
   ],
   "source": [
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "final=(pred+preds)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 1 ..., 1 0 0]\n"
     ]
    }
   ],
   "source": [
    "print(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.2157013   0.75867236  0.77440751 ...,  0.75867236  0.27440751\n",
      "  0.2260046 ]\n"
     ]
    }
   ],
   "source": [
    "print(final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.46200319817391311"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25867235660552979"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.median(final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rajat/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:5: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)\n"
     ]
    }
   ],
   "source": [
    "submission1 = pd.DataFrame()\n",
    "submission1[\"id\"] = test_df[\"id\"]\n",
    "submission1[\"labels\"] = pred\n",
    "\n",
    "submission1=submission1.sort(\"id\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4370</th>\n",
       "      <td>0</td>\n",
       "      <td>0.421748</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9261</th>\n",
       "      <td>1</td>\n",
       "      <td>0.511580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2444</th>\n",
       "      <td>2</td>\n",
       "      <td>0.438821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8185</th>\n",
       "      <td>3</td>\n",
       "      <td>0.438821</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5201</th>\n",
       "      <td>4</td>\n",
       "      <td>0.517345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id    labels\n",
       "4370   0  0.421748\n",
       "9261   1  0.511580\n",
       "2444   2  0.438821\n",
       "8185   3  0.438821\n",
       "5201   4  0.517345"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10856</th>\n",
       "      <td>10995</td>\n",
       "      <td>0.548815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10463</th>\n",
       "      <td>10996</td>\n",
       "      <td>0.548815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1364</th>\n",
       "      <td>10997</td>\n",
       "      <td>0.442438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4747</th>\n",
       "      <td>10998</td>\n",
       "      <td>0.517345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5215</th>\n",
       "      <td>10999</td>\n",
       "      <td>0.517345</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id    labels\n",
       "10856  10995  0.548815\n",
       "10463  10996  0.548815\n",
       "1364   10997  0.442438\n",
       "4747   10998  0.517345\n",
       "5215   10999  0.517345"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission1.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rajat/anaconda3/lib/python3.5/site-packages/pandas/core/indexing.py:128: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "submission1[\"labels\"].loc[submission1[\"labels\"]>=0.5]=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/rajat/anaconda3/lib/python3.5/site-packages/pandas/core/indexing.py:128: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  self._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "submission1[\"labels\"].loc[submission1[\"labels\"]<0.5]=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10856</th>\n",
       "      <td>10995</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10463</th>\n",
       "      <td>10996</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1364</th>\n",
       "      <td>10997</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4747</th>\n",
       "      <td>10998</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5215</th>\n",
       "      <td>10999</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  labels\n",
       "10856  10995     1.0\n",
       "10463  10996     1.0\n",
       "1364   10997     0.0\n",
       "4747   10998     1.0\n",
       "5215   10999     1.0"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission1.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4370</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9261</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2444</th>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8185</th>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5201</th>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id  labels\n",
       "4370   0     0.0\n",
       "9261   1     1.0\n",
       "2444   2     0.0\n",
       "8185   3     0.0\n",
       "5201   4     1.0"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "submission1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "submission1.to_csv('/home/rajat/Dev/Python/solution1.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}