data_centric_evaluation/Lab - Data-Centric Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9ffd9348",
   "metadata": {},
   "source": [
    "# Lab - Data-Centric Evaluation of ML Models\n",
    "\n",
    "This lab is intended to accompany the lecture on Data-Centric Evaluation of ML Models. In this lab, you are given a training dataset and your goal is to improve the dataset in order to boost the accuracy of a fixed classification model on a given (fixed) test dataset. Reviewing the lecture notes will be helpful to get some ideas.\n",
    "\n",
    "You can get all the dependencies you need for this notebook by running:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "158f9d1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install cleanlab matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59acb88e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import sklearn\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.metrics import balanced_accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e608a19",
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 123\n",
    "np.random.seed(SEED)  # for reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b58f3d8a",
   "metadata": {},
   "source": [
    "# Load dataset\n",
    "\n",
    "The data come from a 3-class classification task where the goal is to predict `y` based on features `x1`, ..., `x5`.\n",
    "We convert the data to a numpy array format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf3eb22",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"train.csv\")\n",
    "df_test = pd.read_csv(\"test.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "922ccebe",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a22b105b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb082792",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df[\"y\"].values\n",
    "df.drop(columns=[\"y\"], inplace=True)\n",
    "X = df.values\n",
    "\n",
    "y_test = df_test[\"y\"].values\n",
    "df_test.drop(columns=[\"y\"], inplace=True)\n",
    "X_test = df_test.values\n",
    "\n",
    "print(X)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93dae00e",
   "metadata": {},
   "source": [
    "Here's a simple plot of the first two features, colored by class. Visualizing the dataset may be helpful to you in this assignment, but can be tricky with 6-dimensional features!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee220714",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(X[:,0], X[:,1], c=list(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e22b2088",
   "metadata": {},
   "source": [
    "# Improve ML Model via data-centric techniques"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f90b04e",
   "metadata": {},
   "source": [
    "We train a simple neural network model (Multi-Layer Perceptron classifier) and evaluate it on the given test set. \n",
    "Note that the evaluation metric used is `balanced_accuracy`. This is the accuracy amongst all examples in a given class, averaged over all classes, which better represents performance on minority classes.\n",
    "\n",
    "You should never change the test data labels `y_test` in this lab, nor the neural network model. You may change the training data `X`, `y` as you see fit, and update `X_test` accordingly so your model is able to make predictions on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e44ea5c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_evaluate_model(X, y, X_test, y_test):\n",
    "    model = MLPClassifier(early_stopping=True, random_state=SEED)\n",
    "    model.fit(X, y)\n",
    "    predictions = model.predict(X_test)\n",
    "    acc = balanced_accuracy_score(y_true=y_test, y_pred=predictions)\n",
    "    print(f\"Balanced accuracy = {acc}\")\n",
    "    return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d967913e",
   "metadata": {},
   "source": [
    "A baseline version of the model is trained on the original given data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bda77e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = train_evaluate_model(X, y, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10c576fb",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "Your goal is to produce a version of this same model that has much better test accuracy! \n",
    "You should accomplish this by changing `X` and `y` as you see fit. Consider various ideas presented in the lecture and look through the data yourself to see if other ideas come to mind! Greater than 80% test accuracy is achievable without modifying the model at all. If you choose to drop/add features to `X` or renormalize some values, then you'll want to apply these operations to `X_test` as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56097167",
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: find my_X, my_y such that train_evaluate_model(my_X, my_y, X_test, y_test)\n",
    "## reports better (higher) performance.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dcaiclass",
   "language": "python",
   "name": "dcaiclass"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}