Notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d21d3ef0-db4e-4944-aebe-ec8a4b406599",
   "metadata": {},
   "source": [
    "# CC-Scanner"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04044800",
   "metadata": {},
   "source": [
    "## Enhanced Credit Card Fraud Detection using Graph Neural Networks\n",
    "\n",
    "---\n",
    "\n",
    "## Introduction\n",
    "Learn to Boost fraud detection accuracy and developer efficiency through this program's end-to-end, no-code, graph-neural-networks-boosted and multi-node distributed workflows.\n",
    "## Solution Technical Overview\n",
    "Fraud detection has traditionally been tackled with classical machine learning algorithms such as gradient boosted machines. However, such supervised machine learning algorithms can lead to unsatisfactory precision and recall due to a few reasons:\n",
    "- Severe class imbalance: ratio of fraud to non-fraud transactions is extremely imbalanced with typical values less than 1% \n",
    "- Complex fraudster behavior which evolves with time: it is quite difficult to capture user behavior using traditional ML techniques \n",
    "- Scale of data: credit card transaction datasets can have billions of transactions which require distributed preprocessing and training \n",
    "- Latency of fraud detection: it is important to detect fraud quickly in order to minimize losses, thus highlighting the need for distributed inference <br />\n",
    "\n",
    "In this Enhanced Fraud Detection reference kit, we employ Graph Neural Networks (GNN) popular for their ability to capture complex behavioral patterns (e.g., fraudsters performing multiple small transactions from different cards to not get caught). We also demonstrate a boost in accuracy by using GNN-boosted features over a baseline trained on traditional ML-only features. \n",
    "\n",
    "## Validated Hardware Details\n",
    "There are workflow-specific hardware and software setup requirements depending on\n",
    "how the workflow is run. Bare metal development system and Docker\\* image running\n",
    "locally have the same system requirements.\n",
    "\n",
    "| Supported Hardware           | Precision  |\n",
    "| ---------------------------- | ---------- |\n",
    "| Intel® 1st, 2nd, 3rd, and 4th Gen Xeon® Scalable Performance processors | FP32 |\n",
    "|Memory|>200GB|\n",
    "|Storage|>50GB|\n",
    "## How It Works\n",
    "The high-level architecture of the reference use case is shown in the diagram below. We use a credit card transaction dataset open-sourced by IBM (commonly known as the tabformer dataset) in this reference use case to demonstrate the capabilities outlined in the Solution Technical Overview section. \n",
    "\n",
    "![folder-structure](assets/architecture.png)\n",
    "\n",
    "\n",
    "### Task 1: Feature Engineering (Edge Featurization)\n",
    "The feature engineering stage ingests the raw data, encodes each column into features using the logic defined in the feature engineering config yaml file and saves processed data.\n",
    "### Task 2: GNN Training (Node Featurization)\n",
    "The GNN training stage creates homogenous graphs by consuming the processed data generated by Task 1 and trains a GraphSage model in a self-supervised link prediction task setting to learn the latent representations of the nodes (cards and merchants).  Once the GNN model is trained, the GNN workflow will concatenate the card and merchant features generated by the model to the corresponding transaction features and save the GNN-boosted features to a CSV file.\n",
    "### Task 3: XGBoost Training (Fraud Classification)\n",
    "The XGBoost training stage trains a binary classification model using the data splitting, model parameters and runtime parameters set in the XGB training config yaml file. AUCPR (Area Under the Precision-Recall Curve) is used as the evaluation metric due to its robustness in evaluating highly imbalanced datasets. Data splitting is based on temporal sequence to simulate real-life scenario. The model performance on the tabformer dataset can be found in the table from results section on README.md."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10dc9b64",
   "metadata": {},
   "source": [
    "## Run Using Docker\n",
    "Follow these instructions to set up and run a single node pipeline with our provided Docker image.\n",
    "For running distributed pipeline on bare metal, see the bare metal instructions on README.md file. \n",
    "\n",
    "Before running the next cell, refer to ```Getting Started``` section on README.md and follow the instructions. Once user has completed the steps and has\n",
    "declared the ENVVARs in the same terminal that will run Jupyter Lab, continue with the execution of this notebook. ENVVARs declared on terminal will be present when running this notebook and assigned to variables for Python script when running the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ec85d0a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-06T23:37:21.158158Z",
     "start_time": "2023-06-06T23:37:21.148447Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "WORKSPACE = os.environ['WORKSPACE']\n",
    "DATASET_DIR = os.environ['DATASET_DIR']\n",
    "print(\"Work dir: {}\".format(WORKSPACE))\n",
    "print(\"Dataset dir: {}\".format(DATASET_DIR))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5cfc404",
   "metadata": {},
   "source": [
    "#### Set Up Docker Engine\n",
    "You'll need to install Docker Engine on your development system.\n",
    "Note that while **Docker Engine** is free to use, **Docker Desktop** may require\n",
    "you to purchase a license.  See the [Docker Engine Server installation\n",
    "instructions](https://docs.docker.com/engine/install/#server) for details.\n",
    "\n",
    "#### Setup Docker Compose\n",
    "Ensure you have Docker Compose installed on your machine. If you don't have this tool installed, consult the official [Docker Compose installation documentation](https://docs.docker.com/compose/install/linux/#install-the-plugin-manually)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4d60849",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "DOCKER_CONFIG=os.getenv('DOCKER_CONFIG', '')\n",
    "DOCKER_CONFIG+=f\"{os.getenv('HOME','')}/.docker\" if DOCKER_CONFIG == '' else f\":{os.getenv('HOME','')}/.docker\"\n",
    "!mkdir -p $DOCKER_CONFIG/cli-plugins\n",
    "!curl -SL https://github.com/docker/compose/releases/download/v2.7.0/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose\n",
    "!chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose\n",
    "!docker compose version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de610008",
   "metadata": {},
   "source": [
    "### Set Up Docker Image\n",
    "Build or pull the provided docker image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c784a819",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose build"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6487716",
   "metadata": {},
   "source": [
    "OR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fa90859",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-02T21:02:17.717918Z",
     "start_time": "2023-06-02T21:02:17.507810Z"
    },
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker pull SelimWaly/ai-workflows:pa-fraud-detection-classical-ml\n",
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker pull SelimWaly/ai-workflows:pa-fraud-detection-gnn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f236a1",
   "metadata": {},
   "source": [
    "### Run Pipeline with Docker Compose\n",
    "#### Run feature engineering to get edge features\n",
    "The `preprocess` workflow will ingest the raw data in the ```$DATASET_DIR/raw_data/``` directory, generate a preprocessed CSV file, and save it in the ```$OUTPUT_DIR/data/edge_data/``` directory.\n",
    "\n",
    "Run the `preprocess` workflow with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eff64835",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose run preprocess 2>&1 | tee preprocess.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c5ea25b",
   "metadata": {},
   "source": [
    "The table below shows some of the environment variables you can control according to your needs.\n",
    "\n",
    "| Environment Variable Name | Default Value | Description |\n",
    "| --- | --- | --- |\n",
    "| CONFIG_DIR | `${WORKSPACE}/credit-card-fraud-detection/configs`       | Configurations directory |\n",
    "| OUTPUT_DIR | `${WORKSPACE}/credit-card-fraud-detection/docker/output` | Logfile and Checkpoint output |\n",
    "\n",
    "##### Train and evaluate XGBoost model with edge features only\n",
    "\n",
    "The `preprocess` workflow must complete successfully before running the `baseline-training`.\n",
    "\n",
    "The `baseline-training` workflow will consume the CSV file generated from `preprocess` workflow above, and run a training of a XGBoost model. It will also print out AUCPR (Area Under the Precision-Recall Curve) results to the console.\n",
    "\n",
    "Run the `baseline-training` workflow with the command below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2fc9d45",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose run baseline-training 2>&1 | tee baseline-training.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6f19308",
   "metadata": {},
   "source": [
    "The table below shows some of the environment variables you can control according to your needs.\n",
    "\n",
    "| Environment Variable Name | Default Value | Description |\n",
    "| --- | --- | --- |\n",
    "| CONFIG_DIR | `${WORKSPACE}/credit-card-fraud-detection/configs`       | Configurations directory |\n",
    "| OUTPUT_DIR | `${WORKSPACE}/credit-card-fraud-detection/docker/output` | Logfile and Checkpoint output |\n",
    "\n",
    "#### Train and Evaluate XGBoost model with both edge features and GNN generated node features\n",
    "\n",
    "To see the improvement over the baseline training you can run the `xgb-training` workflow. Before running the `xgb-training`, the `preprocess` workflow must complete successfully.\n",
    "\n",
    "The `xgb-training` workflow consumes the CSV file generated from `preprocess` above, runs the `gnn-analytics` pipeline to generate optimized features, and runs a training of a XGBoost model using these features. It will also print out AUCPR (Area Under the Precision-Recall Curve) results to the console.\n",
    "\n",
    "Note: as this step runs GNN training first, we don't expect to see output for a while. Once GNN training finishes, we will start seeing output from XGBoost training. You can also check GNN log in parallel while running `xgb-training` by using the following command from terminal inside $WORKSPACE/credit-card-fraud-detection/docker:\n",
    "```bash\n",
    "docker compose logs gnn-analytics -f\n",
    "```\n",
    "After running the command from terminal, skip the next cell and run `xgb-training`. Alternatively, you can check GNN log from this notebook by running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbff1e94",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#To run this cell, first run the cell bellow and once it has finished run this cell.\n",
    "#The process will keep running until interrupted by stopping it with \"Interrupt the kernel\".\n",
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose logs gnn-analytics -f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e2b226c",
   "metadata": {},
   "source": [
    "Run the `xgb-training` container with the command below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10dbb6a9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose run xgb-training 2>&1 | tee xgb-training.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af036523",
   "metadata": {},
   "source": [
    "This command runs the `gnn-analytics` workflow implicitly to generate the node features first and then uses edge features generated from [Step 2](#train-and-evaluate-xgboost-model-with-edge-features-only) to train the XGBoost model and will print out AUCPR (Area Under the Precision-Recall Curve) results to the console.\n",
    "\n",
    "Note: This steps runs the GNN training first which can take several hours to finish. \n",
    "\n",
    "The table below shows some of the environment variables you can control according to your needs.\n",
    "\n",
    "| Environment Variable Name | Default Value | Description |\n",
    "| --- | --- | --- |\n",
    "| CONFIG_DIR | `${WORKSPACE}/credit-card-fraud-detection/configs`       | Configurations directory |\n",
    "| OUTPUT_DIR | `${WORKSPACE}/credit-card-fraud-detection/docker/output` | Logfile and Checkpoint output |\n",
    "\n",
    "#### View Logs\n",
    "Run these commands to check the `preprocess`, `baseline-training`, and `xgb-training` logs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a5a446e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!echo $'========================Preprocess========================'\n",
    "!cat $WORKSPACE/cc-scanner/docker/preprocess.log\n",
    "!echo $'\\n====================Baseline Training====================='\n",
    "!cat $WORKSPACE/cc-scanner/docker/baseline-training.log\n",
    "!echo $'\\n=======================XGB Training======================='\n",
    "!cat $WORKSPACE/cc-scanner/docker/xgb-training.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6d6725e",
   "metadata": {},
   "source": [
    "You can also check GNN log using the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4cc935b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The process will keep running until interrupted by stopping it with \"Interrupt the kernel\".\n",
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose logs gnn-analytics -f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e09bf2",
   "metadata": {},
   "source": [
    "#### Reproducing results\n",
    "After running hyperparameter optimization (HPO), we found the best params for baseline and final models. You can use them to recreate our results from README.md by running the following cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e05804ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "#The code comments `hpo_spec` section from the config files and uncomments `model_spec` section.\n",
    "#It can also return the config files to their original state by selecting `hpo_spec` as input parameter.\n",
    "def change_case(case):\n",
    "    file_list = [\"baseline-xgb-training.yaml\", \"xgb-training.yaml\"]\n",
    "    config_path = \"/cc-scanner/configs/single-node/\"\n",
    "    compose_file = WORKSPACE+\"/cc-scanner/docker/docker-compose.yml\"\n",
    "    \n",
    "    with open(compose_file, \"r\") as f:\n",
    "        content_list = f.readlines()\n",
    "\n",
    "    #74:77 is the range of lines to comment or uncomment from the docker-compose.yml file.\n",
    "    if case == \"model_spec\":\n",
    "        content_list[74:77] = [\"#\"+elem if elem[0] !=\"#\" else elem for elem in content_list[74:77]]\n",
    "    elif case == \"hpo_spec\":\n",
    "        content_list[74:77] = [elem[1:] if elem[0] ==\"#\" else elem for elem in content_list[74:77]]\n",
    "\n",
    "    with open(compose_file, \"w\") as f:\n",
    "        f.writelines(content_list)\n",
    "    \n",
    "    for file in file_list:\n",
    "        path = WORKSPACE+config_path+file\n",
    "        yml_info = []\n",
    "        with open(path, 'r') as f:\n",
    "            for line in f:\n",
    "                if \"data_spec:\" in line or \"hpo_spec:\" in line or \"model_spec:\" in line:\n",
    "                    yml_info.append([])\n",
    "\n",
    "                if \"#\" == line[0]:\n",
    "                    yml_info[-1].append(line[2:])\n",
    "                else:\n",
    "                    yml_info[-1].append(line)\n",
    "\n",
    "        with open(path, 'w') as f:            \n",
    "            for config in yml_info:\n",
    "                if  \"data_spec:\" in config[0] or case in config[0]:\n",
    "                    for elem in config:\n",
    "                        f.write(elem)\n",
    "                else:\n",
    "                    for elem in config:\n",
    "                        f.write(\"# \"+elem)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0d744a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "change_case(\"model_spec\") #Activates model_spec\n",
    "\n",
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose run baseline-training 2>&1 | tee baseline-training-model_spec.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f1b6c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose run xgb-training 2>&1 | tee xgb-training-model_spec.log\n",
    "\n",
    "change_case(\"hpo_spec\") #Returns to default (hpo_spec) configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "469dc5a2",
   "metadata": {},
   "source": [
    "---\n",
    "We consider the AUCPR calculation from the last line (starts with [999]) as our final result. Previous lines are intermediate evaluations which can be used to track progress of the model. We expect final results to match closely to our reported numbers although intermediate evaluations could be different.\n",
    "\n",
    "You can also check logs of the results running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7465d30",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!echo $'\\n====================Baseline Training====================='\n",
    "!cat $WORKSPACE/cc-scanner/docker/baseline-training-model_spec.log\n",
    "!echo $'\\n=======================XGB Training======================='\n",
    "!cat $WORKSPACE/cc-scanner/docker/xgb-training-model_spec.log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "221b24b0",
   "metadata": {},
   "source": [
    "\n",
    "Run the following command to stop all services and containers created by docker compose and remove them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d81aa71",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $WORKSPACE/cc-scanner/docker && \\\n",
    "docker compose down"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}