example-use-cases/archive-wide-queries/fetch_initial_release_date.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cbcd5604",
   "metadata": {},
   "source": [
    "Open and run this in Google Colab: <a href=\"https://colab.research.google.com/github/rcsb/rcsb-training-resources/blob/master/example-use-cases/archive-wide-queries/fetch_initial_release_date.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "608705e1-ae70-478f-9701-b4c4248acfbe",
   "metadata": {},
   "source": [
    "# Fetch Initial Release Date for All PDB IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6796f66-3cae-45fe-9ae0-e1385fee55cd",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "The code in this notebook is designed to perform the following tasks:\n",
    "\n",
    "1. Use Data API to get a list of all PDB IDs currently released\n",
    "2. Use a Data API query to retrieve initial release data\n",
    "\n",
    "### Purpose\n",
    "\n",
    "This notebook is designed to help you fetch initial release dates for all PDB IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8608c310-ce0f-4cd5-8f38-eeee3d724207",
   "metadata": {},
   "source": [
    "## Libraries\n",
    "\n",
    "These libraries will be called in the coding cells in this notebook. \n",
    "\n",
    "| Library |Contents | Source |\n",
    "| :-----: | :------- | :----- |\n",
    "| requests | simple HTTP library for Python | [documentation](https://requests.readthedocs.io/en/latest/) |\n",
    "| dateutil | extensions to the standard datetime module | [documentation](https://dateutil.readthedocs.io/en/stable/) |\n",
    "| rcsb-api | python interface for the API services at [RCSB Protein Data Bank](https://www.rcsb.org)| [py-rcsb-api on GitHub](https://github.com/rcsb/py-rcsb-api) |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d08a9f3-ec48-4be0-9a2f-4dec1ff26c93",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "To use this notebook, you will need to have the following libraries installed in your computing environment: json, rcsbsearchapi, python_graphql_client. To install from the command line on your computer, use this command:\n",
    "\n",
    "`pip install python-dateutil`\\\n",
    "`pip install requests`\\\n",
    "`pip install rcsb-api`\n",
    "\n",
    "To install from within a Jupyter notebook or CoLab notebook, you need to type the same command in a coding cell, preceded by %:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bee74152-3ac4-4257-be2a-f2de3bbea49c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this coding cell to install necessary libraries if they are not already in your system or environment\n",
    "%pip install python-dateutil\n",
    "%pip install requests\n",
    "%pip install rcsb-api"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0edae8a4-a3bb-4f47-a506-9a68a5a7f910",
   "metadata": {},
   "source": [
    "## Running the Notebook\n",
    "\n",
    "Then coding cell below contains all of the raw code for this example. **Experienced coders** should use this as you see fit.\n",
    "\n",
    "For **novice coders**, the code is broken up into smaller chunks in the subsequent coding cells, with stepwise inputs and outputs to better explain how this code can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7b9186-c308-41b8-a303-8f24775b526d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from dateutil import parser\n",
    "from rcsbapi.data import DataQuery as Query\n",
    "\n",
    "# Step 1: Retrieve all PDB IDs from Data API\n",
    "url = 'https://data.rcsb.org/rest/v1/holdings/current/entry_ids'\n",
    "response = requests.get(url)\n",
    "ids = eval(response.text)\n",
    "\n",
    "# Step 2: Split full list of IDs into batches\n",
    "batchSize = 5_000\n",
    "idBatches = [ids[i:i+batchSize] for i in range(0, len(ids), batchSize)]\n",
    "\n",
    "#Step 3: Query release date\n",
    "release_dates = []\n",
    "for batch in idBatches:\n",
    "    query = Query(\n",
    "        input_type=\"entries\",\n",
    "        input_ids=batch,\n",
    "        return_data_list=[\"rcsb_accession_info.initial_release_date\"]\n",
    "    )\n",
    "    data = query.exec()\n",
    "    for d in data['data']['entries']:\n",
    "        entry_id = d['rcsb_id']\n",
    "        isodate = d[\"rcsb_accession_info\"][\"initial_release_date\"]\n",
    "        date = parser.parse(isodate).strftime('%Y-%m-%d')\n",
    "        release_dates.append({\n",
    "            \"pdb_id\": entry_id,\n",
    "            \"release_date\": date\n",
    "        })\n",
    "print(release_dates[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ebb4b3",
   "metadata": {},
   "source": [
    "## Code Breakdown"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99c39144-6d81-43e6-b6ca-c4bcf76dc2e1",
   "metadata": {},
   "source": [
    "### Importing Libraries\n",
    "\n",
    "The following simply imports the required libraries that contain the methods that are called in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccde6203",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from dateutil import parser\n",
    "from rcsbapi.data import DataQuery as Query"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9f910e-9b92-4fc0-823d-ad545b725056",
   "metadata": {},
   "source": [
    "### Step 1 \n",
    "\n",
    "The [Repository Holdings Service REST API](https://data.rcsb.org/redoc/index.html#tag/Repository-Holdings-Service) current entries endpoint provides a full list of current PDB IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "490ee066-4eda-45d7-b7dc-554925c756ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: Retrieve all PDB IDs from Data API\n",
    "\n",
    "url = 'https://data.rcsb.org/rest/v1/holdings/current/entry_ids'\n",
    "response = requests.get(url)\n",
    "ids = eval(response.text)\n",
    "\n",
    "print(f\"There are {len(ids)} released structures in the PDB archive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ced9dcb5",
   "metadata": {},
   "source": [
    "### Step 2\n",
    "Requesting a large number of objects at a time is resource intensive and not recommended. Making requests in periodic batches, instead of a single request for a large number of objects, is more effective"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef4d858",
   "metadata": {},
   "outputs": [],
   "source": [
    "batchSize = 5_000\n",
    "idBatches = [ids[i:i+batchSize] for i in range(0, len(ids), batchSize)]\n",
    "\n",
    "print(f\"Split IDs into {len(idBatches)} batches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1076ea3-8ba7-458a-8038-cebb407727a2",
   "metadata": {},
   "source": [
    "### Step 3\n",
    "\n",
    "In this step, we create a query object that points to the data API on the RCSB PDB website. This query extracts data about the initial release date for a given list of IDs. Example query below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fbbe1d9-bd7f-4cf4-b568-28f47dee91b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3: Run data API query to retrieve initial release date for the first ID\n",
    "query = Query(\n",
    "    input_type=\"entries\",\n",
    "    input_ids=[\"4HHB\"],\n",
    "    return_data_list=[\"rcsb_accession_info.initial_release_date\"]\n",
    ")\n",
    "data = query.exec()\n",
    "print(data['data']['entries'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f74db77-1910-4377-ba13-aeac6914b6d8",
   "metadata": {},
   "source": [
    "### Step 4\n",
    "\n",
    "The final step is to produce the results we want to see and store for future use. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1f822a-8636-4a06-8292-40f805ce979d",
   "metadata": {},
   "outputs": [],
   "source": [
    "release_dates = []\n",
    "for batch in idBatches:\n",
    "    query = Query(\n",
    "        input_type=\"entries\",\n",
    "        input_ids=batch,\n",
    "        return_data_list=[\"rcsb_accession_info.initial_release_date\"]\n",
    "    )\n",
    "    data = query.exec()\n",
    "    for d in data['data']['entries']:\n",
    "        entry_id = d['rcsb_id']\n",
    "        isodate = d[\"rcsb_accession_info\"][\"initial_release_date\"]\n",
    "        date = parser.parse(isodate).strftime('%Y-%m-%d')\n",
    "        release_dates.append({\n",
    "            \"pdb_id\": entry_id,\n",
    "            \"release_date\": date\n",
    "        })\n",
    "print(release_dates[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By modifying \"return_data_list\" parameter of the query object, any available data attributes can be queried for all currently released PDB entries. Available fields can be explored [here](https://data.rcsb.org/data-attributes.html) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}