text_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![DSL_logo](dsl_logo.png)\n",
    "\n",
    "\n",
    "# Introduction to Text Analysis with Python\n",
    "\n",
    "Welcome to the Digital Scholarship Lab introduction to Text Analysis with Python class. In this class we'll learn the basics of text analysis:\n",
    "\n",
    "- parsing text\n",
    "- analyzing the text\n",
    "\n",
    "We'll use our own home made analysis tool first, then we'll use a python library called `TextBlob` to use some built in analysis tools.\n",
    "\n",
    "This workshop assumes you've completed the DSL 'Introduction to Python Workshop'\n",
    "\n",
    "We'll use the Zoom's chat feature to interact.\n",
    "\n",
    "Be sure to enable line numbers by looking for the 'gear' icon and checking the box in the 'Editor' panel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Text Analysis\n",
    "\n",
    "Let's go through a basic text analysis example! It'll be broken down into 3 steps:\n",
    "\n",
    "1. Load the text from a file\n",
    "1. Break the text into words\n",
    "1. Apply analysis to the words\n",
    "1. Display our results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EG. Scrabble!\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/5/5d/Scrabble_game_in_progress.jpg\" width =500x>\n",
    "\n",
    "We are going to use all of our Python powers to create a Scrabble score calculator. It will encorporate all of the following ideas:\n",
    "\n",
    "- Functions\n",
    "- Loops\n",
    "- Dictionaries\n",
    "- Variables\n",
    "\n",
    "We are going to use this function to do some analysis on the poem [_\"That is Just to Say\"_](https://poets.org/poem/just-say)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Q1** Add some code to line `40` and `41` to complete this function\n",
    "- Test it in the cell that follows the function definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function will return the Scrabble score of a word\n",
    "\n",
    "def scrabble_score(word):\n",
    "    \n",
    "    #Dictionary of our scrabble scores\n",
    "    score_lookup = {\n",
    "        \"a\": 1,\n",
    "        \"b\": 3,\n",
    "        \"c\": 3,\n",
    "        \"d\": 2,\n",
    "        \"e\": 1,\n",
    "        \"f\": 4,\n",
    "        \"g\": 2,\n",
    "        \"h\": 4,\n",
    "        \"i\": 1,\n",
    "        \"j\": 8,\n",
    "        \"k\": 5,\n",
    "        \"l\": 1,\n",
    "        \"m\": 3,\n",
    "        \"n\": 1,\n",
    "        \"o\": 1,\n",
    "        \"p\": 3,\n",
    "        \"q\": 10,\n",
    "        \"r\": 1,\n",
    "        \"s\": 1,\n",
    "        \"t\": 1,\n",
    "        \"u\": 1,\n",
    "        \"v\": 4,\n",
    "        \"w\": 4,\n",
    "        \"x\": 8,\n",
    "        \"y\": 4,\n",
    "        \"z\": 10,\n",
    "        \"\\n\": 0\n",
    "        \n",
    "    }\n",
    "    \n",
    "    score = 0\n",
    "    \n",
    "    # You'll need a for loop\n",
    "    for  in word:\n",
    "        score = \n",
    "    \n",
    "    return score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Verify that the code works by calculating the score of this absurdly long word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(scrabble_score(\"supercalifragilisticexpialidocious\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Finding the highest scored  word\n",
    "\n",
    "What is the highest scored word in the poem \"That is Just to Say\" in the file [plums](plums.txt)\n",
    "\n",
    "- First we open the text file and display it to the screen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poem_text = open(\"plums.txt\",\"r\").read()\n",
    "print(poem_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We split up the text of the document into separate words using the `split` function\n",
    "- `split` takes what character you'd like to break up the string with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "poem_words = poem_text.split(\" \")\n",
    "print(poem_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Now we'll write a loop that goes through all of those words\n",
    "- If the score for the word being considered is higher than the higest scored word, this new word is the highest score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "highest_score = 0;\n",
    "for word in poem_words:aa\n",
    "    if scrabble_score(word) > highest_score:\n",
    "        highest_score = scrabble_score(word)\n",
    "\n",
    "print(highest_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Q2** Can you change the following code to calculate the **total scrabble score** of the poem instead of just the top word score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_score = 0;\n",
    "for word in poem_words:\n",
    "    total_score\n",
    "\n",
    "print(total_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bringing it all together\n",
    "\n",
    "- let's put everything together into a function that finds the highest scored word in a text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function will open up text_file \n",
    "# and find the word with the hightest score\n",
    "#\n",
    "def top_word_score(text_file):\n",
    "    \n",
    "    #to keep track of what the highest scored word is\n",
    "    highest_score = 0\n",
    "    \n",
    "    #opens our poem file an splits it into words\n",
    "    file_text = open(text_file,\"r\").read()\n",
    "    file_words = file_text.split(\" \")\n",
    "    \n",
    "    #go through all of our words from the poem and find the highest score\n",
    "    for word in file_words:\n",
    "        if scrabble_score(word) > highest_score:\n",
    "            highest_score = scrabble_score(word)\n",
    "    return highest_score\n",
    "    \n",
    "print(top_word_score(\"plums.txt\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Q3** Can you write a function that computes the total score of all the words in the variable `text_file`? You can reuse some of the code from `top_word_score`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function will open up text_file \n",
    "# and find the word with the hightest score\n",
    "#\n",
    "def total_word_score(text_file):\n",
    "    total_score = 0\n",
    "    \n",
    "    return total_score\n",
    "\n",
    "print(total_word_score(\"plums.txt\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So far so good\n",
    "\n",
    "- What happens if we use it on capital letters?\n",
    "- What kind of strategy should we try?\n",
    "\n",
    "_HINT:_ We can force letters to be uppercase or lower case... Try to Google what function that might be"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "scrabble_score(\"I\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Q4** Try to re-write your code so that you scrabble score machine will work with capitlized letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## So far so good\n",
    "\n",
    "Our score counter is sort of working...\n",
    "- What happens if we try to use it on a file that has punctuation, like this one: [plums_punctuation.txt](plums_punctuation.txt)\n",
    "- Try to change the code in the cells above to use the file `plums_punctuation.txt` and find out what happens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Q5** Can you write some code that like we did above to figure that out?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scrabble Score for more things!\n",
    "\n",
    "- Can you find another piece of text you'd like to apply the Scrabble Score Text analysis on?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Congrats!\n",
    "\n",
    "You now have a fully functional Scrabble scoring machine"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}