01-connect_and_validate_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Alt text](images/banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9501e8a8-d435-4451-a8ae-513e984aafe9"
   },
   "source": [
    "## Connection and Data Validation Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "062a2e71-ddbf-4c77-aaeb-d5aa7ac8d265"
   },
   "source": [
    "### Load the Credentials\n",
    "\n",
    "These environment variables are automatically set in WS Pipelines and are needed to access various services. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e875ff36-3235-43fb-8008-4bfb334c1325",
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "TOKEN = os.getenv(\"USER_ACCESS_TOKEN\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5dc1650f-d3fd-49f3-820d-dbce4ab98d04"
   },
   "outputs": [],
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "fe00539c-a0ab-4769-ba7b-805adea59cf8",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-10-23 07:24:33.933101: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2023-10-23 07:24:33.933161: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2023-10-23 07:24:33.933204: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2023-10-23 07:24:35.469552: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "from botocore.client import Config\n",
    "from sklearn.model_selection import train_test_split\n",
    "from dataclasses import dataclass\n",
    "import tensorflow_data_validation as tfdv\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from ibm_watson_studio_pipelines import WSPipelines\n",
    "import warnings\n",
    "\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading Variables and Utils from common python file\n",
    "\n",
    "In this section we load the variables and functions from the common python file. This file contains the variables and functions that are common to all the notebooks in this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import vars_and_utils as vars_and_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d44e72ca-25cf-44c2-a8f4-4ad6eff50e4a"
   },
   "source": [
    "## Load the Training Data \n",
    "\n",
    "this will check if the training data exists within a defined db2 table. If it does not exist, it will load the data from the web and store it in the project space as a .csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "0487c397-3d75-4292-ae6f-8f2cd4bb8f19",
    "tags": []
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'training_file_path' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[14], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m gcr_df \u001b[38;5;241m=\u001b[39m load_data_from_project(\u001b[43mtraining_file_path\u001b[49m)\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m## Encode for ease of use with OpenScale\u001b[39;00m\n\u001b[1;32m      4\u001b[0m gcr_df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mRisk\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m gcr_df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mRisk\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mmap({\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mRisk\u001b[39m\u001b[38;5;124m'\u001b[39m:\u001b[38;5;241m1\u001b[39m,\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mNo Risk\u001b[39m\u001b[38;5;124m'\u001b[39m:\u001b[38;5;241m0\u001b[39m})\n",
      "\u001b[0;31mNameError\u001b[0m: name 'training_file_path' is not defined"
     ]
    }
   ],
   "source": [
    "gcr_df = vars_and_utils.load_german_credit_risk_data()\n",
    "\n",
    "## Encode for ease of use with OpenScale\n",
    "gcr_df['Risk'] = gcr_df['Risk'].map({'Risk':1,'No Risk':0})\n",
    "gcr_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e38d5859-3df7-4174-8f8c-a047c0dcdb3c"
   },
   "source": [
    "## Data Validation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "37d30747-1f1d-42ad-ab50-6001740df627",
    "tags": []
   },
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class Datavalidation:\n",
    "    \"\"\"\n",
    "    \n",
    "    Data Validation Class\n",
    "    \n",
    "    \"\"\"\n",
    "    dataframe : pd.DataFrame\n",
    "    mask_per :int\n",
    "    \n",
    "    \n",
    "    def split_data(self,seed=32):\n",
    "        \"\"\"\n",
    "        Split Data into Train and Test Splits\n",
    "        \n",
    "        \"\"\"\n",
    "        np.random.seed(seed)\n",
    "        mask = np.random.rand(len(self.dataframe)) <= self.mask_per\n",
    "        training_data = gcr_df[mask]\n",
    "        testing_data = gcr_df[~mask]\n",
    "\n",
    "        print(f\"No. of training examples: {training_data.shape[0]}\")\n",
    "        print(f\"No. of testing examples: {testing_data.shape[0]}\")\n",
    "        \n",
    "        return training_data, testing_data\n",
    "    \n",
    "    # TODO: Replace with Db2/fileystem\n",
    "    def save_data_in_filesystem(self,df,filename):\n",
    "        \"\"\"\n",
    "        Save Data in Filesystem\n",
    "\n",
    "        Passed filename should involve path\n",
    "\n",
    "        \"\"\"\n",
    "        try:\n",
    "            df.to_csv(filename,index=False)\n",
    "            print(f\"File {filename} persisted successfully\")\n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "            print(f\"File serialization for {filename} failed\")\n",
    "    \n",
    "    def generate_statistics(self,df):\n",
    "        \"\"\"\n",
    "        \n",
    "        Generate Statistics on a given Dataframe\n",
    "        \n",
    "        \"\"\"\n",
    "        train_stats = tfdv.generate_statistics_from_dataframe(df)\n",
    "        tfdv.visualize_statistics(train_stats)\n",
    "        return train_stats\n",
    "    \n",
    "    def inferSchema(self,stats):\n",
    "        \n",
    "        \"\"\"\n",
    "        InferSchema on a given Dataframe\n",
    "        \n",
    "        \"\"\"\n",
    "        schema = tfdv.infer_schema(statistics=stats)\n",
    "        tfdv.display_schema(schema=schema)\n",
    "        return schema\n",
    "    \n",
    "    def compare_statistics(self,lhs,rhs):\n",
    "        \"\"\"\n",
    "        \n",
    "        Compare Statistics between a test dataframe and reference Schema\n",
    "        \n",
    "        \"\"\"\n",
    "        # Compare evaluation data with training data\n",
    "        tfdv.visualize_statistics(lhs_statistics=lhs, rhs_statistics=rhs,\n",
    "                                  lhs_name='TEST_DATASET', rhs_name='TRAIN_DATASET')\n",
    "        \n",
    "        \n",
    "    def check_for_anomalies(self,testable_stats,ref_schema):\n",
    "        \"\"\"\n",
    "        \n",
    "        Check for any anomalies based on statistics and schema and values\n",
    "        \n",
    "        \"\"\"\n",
    "        anomalies = tfdv.validate_statistics(statistics=testable_stats, schema=ref_schema)\n",
    "        tfdv.display_anomalies(anomalies)\n",
    "        if len(anomalies.anomaly_info.items()) > 0:\n",
    "            logger.error(\"Anomalies found in dataset...\")\n",
    "            logger.error(str(self.anomalies.anomaly_info.items()))\n",
    "            return True\n",
    "        else:\n",
    "            return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fff2bb05-abd0-4e0a-9727-d1127a617f57"
   },
   "source": [
    "###  Split Data into Train and Eval Splits to Check for Consistency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "65163ff9-3193-4837-ab55-d66de3a5076f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "classvalidate = Datavalidation(dataframe=gcr_df,mask_per=0.8) \n",
    "\n",
    "training_data, testing_data = classvalidate.split_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "75029b2c-6341-4a59-957c-cbb2d33e3e39"
   },
   "source": [
    "## Generate Training Stats on both Splits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "36a97f3f-3cd7-493d-8a4b-c3c87ae0710f",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_stats = classvalidate.generate_statistics(training_data)\n",
    "test_stats = classvalidate.generate_statistics(testing_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ed4d3f56-e852-4269-a3dd-8426d71bed8e"
   },
   "source": [
    "## Infer Data Schemas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "13f7be93-3fb1-49ca-9238-ebb1fcc1af28",
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_schema = classvalidate.inferSchema(train_stats)\n",
    "test_schema = classvalidate.inferSchema(test_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ea9f07ac-ac4b-4ecd-8840-a5ab07bfb7f8"
   },
   "source": [
    "## Compare Eval and Train Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f5ccfb5d-05bd-4b3b-9f4d-8082252457c3",
    "tags": []
   },
   "outputs": [],
   "source": [
    "classvalidate.compare_statistics(lhs=test_stats,rhs=train_stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "915f6a20-aab3-47b5-86e9-93bde2f548ff"
   },
   "source": [
    "## Check For Data Anomalies "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eae32b5f-46b3-4e5d-919e-68c206a47b8e"
   },
   "source": [
    "### Check eval data for errors by validating the eval data stats using the previously inferred schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f2927d8b-fa0f-420f-9181-a080cdfcb748",
    "tags": []
   },
   "outputs": [],
   "source": [
    "anomaly_status = classvalidate.check_for_anomalies(test_stats,train_schema)\n",
    "anomaly_status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c914eade-25d9-497a-960e-ec5a57282def"
   },
   "source": [
    "## Save Train and Test Data for Data Preparation Stage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0f1d08cc-e4f7-4010-8b4e-7414529e874a",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# TODO: Replace with Db2/fileystem\n",
    "if not anomaly_status:\n",
    "    classvalidate.save_data_in_filesystem(df=training_data,filename=vars_and_utils.train_data_path)\n",
    "    classvalidate.save_data_in_filesystem(df=testing_data,filename=vars_and_utils.test_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "23d79a03-46e7-49de-a549-65797ba3d46c"
   },
   "source": [
    "## Check if the validation steps were successful\n",
    "This checks if anomalies were found and if the data was successfully split into train and eval splits and stored as files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def validation_successfull(train_data_path, test_data_path):\n",
    "    if anomaly_status: # no anomalies\n",
    "        return False\n",
    "    elif not os.path.exists(train_data_path): # train data file exists\n",
    "        return False\n",
    "    elif not os.path.exists(test_data_path): # test data file exists\n",
    "        return False\n",
    "    else:\n",
    "        print (\"validation of the data successfull\")\n",
    "        return True\n",
    "    \n",
    "validation_successfull(vars_and_utils.train_data_path, vars_and_utils.test_data_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b42568a9-36f8-407e-be7a-f8bbbf93444a"
   },
   "source": [
    "## Register the output variables for the next pipeine stage\n",
    "every notebook outputs a \"was_successful\" boolean variable. The logic behind this is different for every notebook and can be altered to fit the needs of the project.\n",
    "If needed additional variables can be created here but they also need to registered as output variables in the Watson Pipelines UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9f210efd-7358-41a4-9ab5-37a856f3ab47",
    "tags": []
   },
   "outputs": [],
   "source": [
    "validation_params = {}\n",
    "validation_params['was_succesfull'] = validation_successfull(vars_and_utils.train_data_path, vars_and_utils.test_data_path)\n",
    "\n",
    "pipelines_client = WSPipelines.from_token(TOKEN)\n",
    "pipelines_client.store_results(validation_params)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "bd385fe162c5ca0c84973b7dd5c518456272446b2b64e67c2a69f949ca7a1754"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}