notebooks/bmi_nwis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/gantian127/bmi_nwis/blob/master/docs/source/_static/bmi_nwis_logo.png?raw=true\" width='450' align='center'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quick Start Tutorial "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will help you get started using the bmi_nwis package to download the National Water Information System (NWIS) datasets. \n",
    "\n",
    "This tutorial includes the following sections:\n",
    "\n",
    "1. [Brief Introduction](#section1)\n",
    "\n",
    "   This section provides basic information about bmi_nwis package. \n",
    "   <br>\n",
    "   \n",
    "2. [Start with Examples](#section2)\n",
    "   \n",
    "   This section provides two examples to demonstrate how to use the dataretrieval package and bmi_nwis package to download the same NWIS dataset for visualization.\n",
    "   \n",
    "   \n",
    "**Suggested citation**: Gan, T. (2023). Jupyter Notebooks for the NWIS Data Component, HydroShare, https://www.hydroshare.org/resource/1473ac1368b44808b57cca2848923779/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section1'></a>\n",
    "## 1. Brief Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "bmi_nwis package is an implementation of the Basic Model Interface ([BMI](https://bmi-spec.readthedocs.io/en/latest/)) for the [USGS NWIS dataset](https://waterdata.usgs.gov/nwis). \n",
    "This package uses the [dataretrieval](https://github.com/USGS-python/dataretrieval) package \n",
    "to download the NWIS dataset and wraps the dataset with BMI for data control and query. This package is not implemented for people to use but is the key element to convert the NWIS dataset into a data component ([pymt_nwis](https://pymt-nwis.readthedocs.io)) for \n",
    "the [PyMT](https://pymt.readthedocs.io/en/latest/?badge=latest) modeling framework developed by Community Surface Dynamics Modeling System ([CSDMS](https://csdms.colorado.edu/wiki/Main_Page)).\n",
    "\n",
    "Please note that the current bmi_nwis implementation only supports to download time series data \n",
    "for instantaneous values and daily mean values ('iv' or 'dv' service option in the dataretrieval package).\n",
    "\n",
    "To install bmi_nwis package, please follow the instructions [here](https://github.com/gantian127/bmi_nwis#install-package)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section2'></a>\n",
    "## 2. Start with Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataretrieval package is a Python alternative to USGS-R's dataRetrieval package for obtaining research datasets from web services. \n",
    "\n",
    "In bmi_nwis package, BmiNwis class is designed to download the USGS NWIS dataset using dataretrieval package and wrap the dataset with BMI. Please note that the current bmi_nwis implementation only supports to download time series data for instantaneous values and daily mean values ('iv' or 'dv' service option in the dataretrieval package). \n",
    "\n",
    "The following examples demonstrate how to download the same dataset using the dataretrieval package and BmiNwis for data visualization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: use the dataretrieval package to download data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, get_record( ) function from the dataretrieval package is used to download the instantaneous datasets at the USGS Gage [03339000](https://waterdata.usgs.gov/nwis/uv?site_no=03339000) from Jan 1st to 3rd, 2022. Time series plots of stream flow and gage height are created. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import dataretrieval.nwis as nwis\n",
    "\n",
    "# get data from NWIS\n",
    "dataset = nwis.get_record(\n",
    "    sites=\"03339000\", service=\"iv\", start=\"2022-01-01\", end=\"2022-01-03\"\n",
    ")\n",
    "\n",
    "# plot data\n",
    "ax = dataset.plot(\n",
    "    y=[\"00060\", \"00065\"],\n",
    "    subplots=True,\n",
    "    figsize=(10, 8),\n",
    "    xlabel=\"Time\",\n",
    "    title=\"Time Series Data at USGS Gage 03339000\",\n",
    ")\n",
    "ax[0].set_ylabel(\"Stream flow (ft3/s)\")\n",
    "ax[1].set_ylabel(\"Gage height (ft)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: use BmiNwis class to download data (Demonstration of how to use BMI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import BmiNwis class and instantiate it. A configuration file (yaml file) is required to provide the [parameter settings](https://bmi_nwis.readthedocs.io/en/latest/#parameter-settings) for data download. An example configure_file.yaml file is provided in the same folder with this Jupyter Notebook file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cftime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from bmi_nwis import BmiNwis\n",
    "\n",
    "# initiate a data component\n",
    "data_comp = BmiNwis()\n",
    "data_comp.initialize(\"config_file.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use variable related methods from BmiNwis class to check the variable information of the NWIS dataset. This data component stores multiple variables, including discharge, gage height, and water temperature. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get variable info\n",
    "for var_name in data_comp.get_output_var_names():\n",
    "    var_unit = data_comp.get_var_units(var_name)\n",
    "    var_location = data_comp.get_var_location(var_name)\n",
    "    var_type = data_comp.get_var_type(var_name)\n",
    "    var_grid = data_comp.get_var_grid(var_name)\n",
    "    var_itemsize = data_comp.get_var_itemsize(var_name)\n",
    "    var_nbytes = data_comp.get_var_nbytes(var_name)\n",
    "\n",
    "    print(f\"{var_name=}\")\n",
    "    print(f\"{var_unit=}\")\n",
    "    print(f\"{var_location=}\")\n",
    "    print(f\"{var_type=}\")\n",
    "    print(f\"{var_grid=}\")\n",
    "    print(f\"{var_itemsize=}\")\n",
    "    print(f\"{var_nbytes=}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use time related methods of BmiNwis class to check the time information of the NWIS dataset. The time values are stored in a format which follows [CF convention](http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.pdf). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time = data_comp.get_start_time()\n",
    "end_time = data_comp.get_end_time()\n",
    "time_step = data_comp.get_time_step()\n",
    "time_unit = data_comp.get_time_units()\n",
    "time_steps = int((end_time - start_time) / time_step) + 1\n",
    "\n",
    "print(f\"{start_time=}\")\n",
    "print(f\"{end_time=}\")\n",
    "print(f\"{time_step=}\")\n",
    "print(f\"{time_unit=}\")\n",
    "print(f\"{time_steps=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use grid related methods to check the grid information of the NWIS dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get variable grid info\n",
    "grid_type = data_comp.get_grid_type(var_grid)\n",
    "grid_rank = data_comp.get_grid_rank(var_grid)\n",
    "grid_node_count = data_comp.get_grid_node_count(var_grid)\n",
    "\n",
    "site_lon = np.empty(grid_node_count)\n",
    "data_comp.get_grid_x(var_grid, site_lon)\n",
    "\n",
    "site_lat = np.empty(grid_node_count)\n",
    "data_comp.get_grid_y(var_grid, site_lat)\n",
    "\n",
    "print(f\"{grid_type=}\")\n",
    "print(f\"{grid_rank=}\")\n",
    "print(f\"{grid_node_count=}\")\n",
    "print(f\"{site_lon[0]=}\")\n",
    "print(f\"{site_lat[0]=}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initiate a dataframe to store the time series data. Loop through each time step to get the stream flow, gage height and time values. get_value( ) method returns the stream flow and gage height values at each time step. update( ) method updates the current time step of the data component. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initiate dataframe to store data\n",
    "dataset = pd.DataFrame(columns=[\"00060\", \"00065\", \"time\"])\n",
    "\n",
    "for i in range(0, time_steps):\n",
    "    # get stream flow data\n",
    "    stream_flow = np.empty(1)\n",
    "    data_comp.get_value(\"Stream flow\", stream_flow)\n",
    "\n",
    "    # get gage height data\n",
    "    gage_height = np.empty(1)\n",
    "    data_comp.get_value(\"Height\", gage_height)\n",
    "\n",
    "    # get time data\n",
    "    cftime_value = data_comp.get_current_time()\n",
    "    time = cftime.num2pydate(cftime_value, time_unit)\n",
    "\n",
    "    # add new row to dataframe\n",
    "    dataset.loc[len(dataset)] = [stream_flow[0], gage_height[0], time]\n",
    "\n",
    "    # update to next time step\n",
    "    data_comp.update()\n",
    "\n",
    "# convert time to local time\n",
    "dataset = dataset.set_index(\"time\").tz_localize(tz=\"UTC\").tz_convert(tz=\"US/Central\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot data\n",
    "ax = dataset.plot(\n",
    "    y=[\"00060\", \"00065\"],\n",
    "    subplots=True,\n",
    "    figsize=(10, 8),\n",
    "    xlabel=\"Time\",\n",
    "    title=\"Time Series Data at USGS Gage 03339000\",\n",
    ")\n",
    "ax[0].set_ylabel(\"Stream flow (ft3/s)\")\n",
    "ax[1].set_ylabel(\"Gage height (ft)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the example by finalizing the component. finalize( ) method performs tasks that take place after using the data component, such as deallocating memory and closing files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_comp.finalize()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}