UCSFProfilesJSON.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Request JSON data from an API endpoint\n",
    "\n",
    "In this section, we will use the **requests** module to obtain JSON data from an api endpoint supplied by the UCSF Profiles application.  Because python can easily retrieve JSON data as a dictionary structure, we will be able to use techniques for python dictionaries to parse and analyze the information. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note on this data...\n",
    "\n",
    "The service we will use in this section is an api provided by the UCSF Profiles application.\n",
    "\n",
    "http://profiles.ucsf.edu/about/ForDevelopers.aspx\n",
    "\n",
    "You might want to take a few minutes to visit this site, follow the \"learn more and get started\" link, and read about the service, the api, and how different apps are using it.  \n",
    "\n",
    "For this tutorial, we'll use the sample JSON link provided by the documentation:\n",
    "\n",
    "http://api.profiles.ucsf.edu/json/v2/?ProfilesURLName=kirsten.bibbins-domingo&source=YourURLGoesHere&publications=full\n",
    "\n",
    "First, let's start by importing the requests module to retrieve data from the web.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use on a small part of this library - you may want to look through the tutorial and documentation at http://docs.python-requests.org/en/master/.  \n",
    "\n",
    "You can pass request parameters to the web service through the URL itself or through a payload.  Both will work, though the payload approach can be useful when the amount of data you need to send to the URL becomes unwieldy.  Both approaches are shown below, though the direct inclusion in the URL method is commented out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#url = 'http://api.profiles.ucsf.edu/json/v2/?ProfilesURLName=kirsten.bibbins-domingo&source=YourURLGoesHere&publications=full'\n",
    "url = 'http://api.profiles.ucsf.edu/json/v2/'\n",
    "payload = {'ProfilesURLName': 'kirsten.bibbins-domingo', 'source':'YourURLGoesHere', 'publications':'full'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# r = requests.get(url)\n",
    "r = requests.get(url, payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The requests module provides a method to view the resulting url from a payload - this can also be useful if you'd like to look at the page directly in your browser.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://api.profiles.ucsf.edu/json/v2/?source=YourURLGoesHere&publications=full&ProfilesURLName=kirsten.bibbins-domingo'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And convert it to JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = r.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#uncomment to see the raw JSON response\n",
    "#data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the type to verify that the data we retrieved is now stored in python as a dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can investigate the keys by visually inspecting the document, but JSON responses can be long, nested, and complex.  An easier way is to list the keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['api_notes', 'Profiles'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the information data in the 'Profiles' section of this dictionary.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "profiles = data['Profiles']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, a dictionary can contain either primitives or objects, including other data structures such as lists or other dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#commented out for length\n",
    "#profiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at this data, we can see the [] denoting a list.  Let's check the type and length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(type(profiles))\n",
    "print(len(profiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is only one element to this list, so let's grab it and take a look..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "profile = profiles[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And in this case, we're back to dealing with a dictionary.  So far, we have a list containing dictionaries nested in a list.  Let's check the keys again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['NIHGrants_beta', 'FreetextKeywords', 'PublicationCount', 'Name', 'Videos', 'AwardOrHonors', 'Email', 'School', 'PhotoURL', 'Keywords', 'FirstName', 'Title', 'SlideShare_beta', 'Titles', 'ProfilesURL', 'GlobalHealth_beta', 'Twitter_beta', 'Narrative', 'Publications', 'Education_Training', 'WebLinks_beta', 'ResearchActivitiesAndFunding', 'MediaLinks_beta', 'Address', 'Department', 'LastName'])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profile.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're back to a list, again, of length one (feel free to check as an exercise).  Let's grab the first (and only) element here..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "profileData = profiles[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and take a look at the keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['NIHGrants_beta', 'FreetextKeywords', 'PublicationCount', 'Name', 'Videos', 'AwardOrHonors', 'Email', 'School', 'PhotoURL', 'Keywords', 'FirstName', 'Title', 'SlideShare_beta', 'Titles', 'ProfilesURL', 'GlobalHealth_beta', 'Twitter_beta', 'Narrative', 'Publications', 'Education_Training', 'WebLinks_beta', 'ResearchActivitiesAndFunding', 'MediaLinks_beta', 'Address', 'Department', 'LastName'])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profileData.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you might want to try looking into different elemets of this list.  Some are data points, containing no further nested data, others contain lists or dictionaries.  we'll take a closer look at **ResearchActivitiesAndFunding**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "researchAndFunding = profileData['ResearchActivitiesAndFunding']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This key maps to a list, this time with a larger (and, it turns out, variable) number of entries.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(researchAndFunding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "researchAndFundingData = researchAndFunding[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'EndDate': '2022-08-31',\n",
       " 'Role': 'Principal Investigator',\n",
       " 'Sponsor': 'NIH/NHLBI',\n",
       " 'SponsorAwardID': 'K12HL138046',\n",
       " 'StartDate': '2017-09-01',\n",
       " 'Title': 'UCSF Career Development Program in Implementation Research in Heart and Lung Diseases'}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "researchAndFundingData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see, from the key and value pairs, that this dictionary does not contain any further nested data, each key maps to a string value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this final node in the JSON tree, we can get at the metadata for a particular publication, including the publication Title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'UCSF Career Development Program in Implementation Research in Heart and Lung Diseases'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "researchAndFundingData['Title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "***Exercise*** \n",
    "\n",
    "Try writing a loop to get just the research and funding data titles for a researcher"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}