MST698S_Session3_Beautiful_Soup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Tools and Techniques\n",
    "\n",
    "# Beautiful Soup Exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform the tasks below to demonstrate your understanding of web scraping.  For reference, you may need to refer to the Beautiful Soup documentation:\n",
    "\n",
    "https://www.crummy.com/software/BeautifulSoup/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the space below, import Beautiful Soup and Requests using the following:\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function will create a Beautiful Soup object from an HTML source:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#get HTML page\n",
    "url = 'https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States'\n",
    "page = requests.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#get soup\n",
    "def get_soup(page):\n",
    "    soup = BeautifulSoup(page.content, 'html.parser')\n",
    "    return soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the space below, create a Beautiful Soup object from the imported HTML document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, open the HTML document in a text viewer.  Before you pull tag content, understand the tag structure of the document."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Now that you have instantiated a Beautiful Soup object from your HTML text, you can access the content using Beautiful Soup methods.  For example, to find all anchors ('a' tags) and save them as a list:\n",
    "\n",
    "anchors = soup.find_all('a')\n",
    "\n",
    "In the space below, find all anchors in your Beautiful Soup object, and print the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Anchors are used in HTML documents for a lot of different things.  One of the primary uses of anchors is to embed hyperlinks.  In the 'a' tag, there is an attribute called 'href' that specifies the hyperlink reference.\n",
    "\n",
    "The above markup will display a hyperlink that points 'hyperlink text' the href value.\n",
    "\n",
    "Beautiful soup can be used to pull specific inforamation from HTML tags.  For example, if you want all hyperlink references from links, you could use the following:\n",
    "\n",
    "links = []\n",
    "for item in anchors:\n",
    "    link = item.get('href')\n",
    "    links.append(link)\n",
    "\n",
    "You've already generated a list of all anchors.  Now, iterate over that list and collect all hyperlinks in the document in the space below, and print the list of hyperlinks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_hyperlinks = []\n",
    "for item in anchors:\n",
    "    link = item.get('href')\n",
    "    all_hyperlinks.append(link)\n",
    "print(all_hyperlinks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for something more challenging.  You have to pay close attention to the HTML structure of your target website.  You will have to examine the hierarchy to determine where your items of interest are.  One way to do that is to use the find_all method to collect all content with a particular attribute.  For example, consider the following:\n",
    "\n",
    "article_body = soup.find_all('div', {'class': 'article-p-wrapper'})\n",
    "\n",
    "This will generate a list of all the content within the soup object that falls within a \"div\" tag with \"class='article-p-wrapper'.  Note that the attribute name and value are passed to the find_all method as a dictionary, with the attribute name as the key, and the target attribute value as the dictionary value.\n",
    "\n",
    "Now, examine the your html document in a text editor.  Using the above example, find the headline of the news article and print it in the space below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, do the same for the article body in the space below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}