1.7_snapshot_update.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "487b4777-9981-4de5-a440-916ee4af0513",
   "metadata": {},
   "source": [
    "# Snapshot Update\n",
    "\n",
    "This notebook shows how to run a Snapshot Extraction Update operation with the minimal steps.\n",
    "\n",
    "In this notebook...\n",
    "* [Dependencies and Initialisation](#Dependencies-and-Initialisation)\n",
    "* [Snapshot instance](#Snapshot-Instance)\n",
    "* [Running the Update Operation](#Running-the-Update-Operation)\n",
    "* [Load the downloaded AVRO files to a Pandas DataFrame](#Load-the-downloaded-AVRO-files-to-a-Pandas-DataFrame)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd55c3bc-ffc6-4634-923f-aa8aee26d7d0",
   "metadata": {},
   "source": [
    "## Dependencies and Initialisation\n",
    "Import statements and environment initialisation using the package `dotenv`. More details in the [Configuration notebook](0.2_configuration.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7535aef1-faff-4eb1-b798-4d453468d498",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n"
     ]
    }
   ],
   "source": [
    "from factiva.news import Snapshot\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ecaa78-e0cd-4d37-a872-436686f316c1",
   "metadata": {},
   "source": [
    "## Snapshot Instance\n",
    "\n",
    "Creates a Snapshot object based on a short Snapshot ID:\n",
    "\n",
    "* **`snapshot_id`**: _Optional_. Short ID of the Snapshot to update. e.g. `'ztj2gkbldt'` Necessary for further update jobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9de04d8a-43d8-4ae9-8c72-63e40f76904f",
   "metadata": {},
   "outputs": [],
   "source": [
    "s = Snapshot(snapshot_id='ztj2gkbldt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bf7ac0-0a75-411c-9809-781bdca9aa9b",
   "metadata": {},
   "source": [
    "## Running the Update Operation\n",
    "\n",
    "This operation builds a collection of files containing the delta from the previously executed operation (either Snapshot or Update).\n",
    "\n",
    "The `<Snapshot>.process_update()` function directly submits, monitors the job and download the content. If a more manual process is required (send job, monitor job, get results), please see the [detailed package documentation](https://factiva-news-python.readthedocs.io/).\n",
    "\n",
    "* **`update_type`**: _Optional_. Default: `additions`. Possible values are: `additions`, `replacements` or `deletes` \n",
    "* **`download_path`**: _Optional_. Location to download the content. If not provided, a new folder named as the `update_id` will be created in the same folder as this notebook.\n",
    "\n",
    "> **Note:** This job is likely to fail if the same combination of `snapshot_id` and `update_type` is sent **within 24 hours**. This update mechanism is designed for low-frequency scenarios. If a higher frequency is needed (hourly or near-real-time), please consider using the Streams service.\n",
    "\n",
    "To review the **Snapshot History**, please see the notebook [User Statistics](1.1_user_statistics.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a495fac7-5096-43d6-9d0b-4d083cf02409",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done!\n",
      "CPU times: user 340 ms, sys: 42.4 ms, total: 382 ms\n",
      "Wall time: 1min 59s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "s.process_update(update_type='deletes')\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24c726d7-4f38-4507-bec0-7aac4c3e21bb",
   "metadata": {},
   "source": [
    "## Load the downloaded AVRO files to a Pandas DataFrame\n",
    "Restuls are stored in the folder named as the Job ID property (`<Snapshot>.last_update_job.id`). A custom tool allows to load its contents to a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8179fc2b-7d2d-4c23-8f96-4a11c9b97af7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from factiva.news import SnapshotFiles\n",
    "sf = SnapshotFiles()\n",
    "articles = sf.read_folder(s.last_update_job.job_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b3b45fce-12f6-4a2b-a21d-4de74c88e237",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['copyright', 'subject_codes', 'modification_datetime', 'body',\n",
       "       'company_codes_occur_ticker_exchange', 'company_codes_occur',\n",
       "       'company_codes_about', 'company_codes_lineage',\n",
       "       'company_codes_ticker_exchange', 'snippet',\n",
       "       'company_codes_relevance_ticker_exchange', 'market_index_codes',\n",
       "       'section', 'company_codes_association_ticker_exchange',\n",
       "       'currency_codes', 'company_codes_about_ticker_exchange',\n",
       "       'region_of_origin', 'company_codes_lineage_ticker_exchange',\n",
       "       'ingestion_datetime', 'modification_date', 'source_name',\n",
       "       'language_code', 'region_codes', 'company_codes_association',\n",
       "       'person_codes', 'byline', 'dateline', 'company_codes_relevance',\n",
       "       'source_code', 'an', 'word_count', 'company_codes', 'industry_codes',\n",
       "       'title', 'publication_datetime', 'publisher_name', 'action'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1c91f9d0-941b-41d6-b0b3-0675973778a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>an</th>\n",
       "      <th>publication_datetime</th>\n",
       "      <th>title</th>\n",
       "      <th>industry_codes</th>\n",
       "      <th>language_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>OFBOAR0020210118eh1f00060</td>\n",
       "      <td>2021-01-15</td>\n",
       "      <td>GS Yuasa - The organizational chart displays i...</td>\n",
       "      <td></td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>OFBOAR0020210125eh1k0010g</td>\n",
       "      <td>2021-01-20</td>\n",
       "      <td>Exide Industries - The organizational chart di...</td>\n",
       "      <td></td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>OFBOAR0020210208eh21002mk</td>\n",
       "      <td>2021-02-01</td>\n",
       "      <td>Sion Power - The organizational chart displays...</td>\n",
       "      <td></td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>OFBOAR0020201221egcg001ze</td>\n",
       "      <td>2020-12-16</td>\n",
       "      <td>NEC Energy Solutions - The organizational char...</td>\n",
       "      <td></td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>JKPOST0020210729eh7t00004</td>\n",
       "      <td>2021-07-29</td>\n",
       "      <td>Indonesian government inks MoU with Korean fir...</td>\n",
       "      <td>,i3432,i35104,i351,iaut,iindele,iindstrls,itec...</td>\n",
       "      <td>en</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          an publication_datetime  \\\n",
       "0  OFBOAR0020210118eh1f00060           2021-01-15   \n",
       "1  OFBOAR0020210125eh1k0010g           2021-01-20   \n",
       "2  OFBOAR0020210208eh21002mk           2021-02-01   \n",
       "3  OFBOAR0020201221egcg001ze           2020-12-16   \n",
       "4  JKPOST0020210729eh7t00004           2021-07-29   \n",
       "\n",
       "                                               title  \\\n",
       "0  GS Yuasa - The organizational chart displays i...   \n",
       "1  Exide Industries - The organizational chart di...   \n",
       "2  Sion Power - The organizational chart displays...   \n",
       "3  NEC Energy Solutions - The organizational char...   \n",
       "4  Indonesian government inks MoU with Korean fir...   \n",
       "\n",
       "                                      industry_codes language_code  \n",
       "0                                                               en  \n",
       "1                                                               en  \n",
       "2                                                               en  \n",
       "3                                                               en  \n",
       "4  ,i3432,i35104,i351,iaut,iindele,iindstrls,itec...            en  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles[['an', 'publication_datetime', 'title', 'industry_codes', 'language_code']].head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}