Part I - Leveraging Intel Optimizations with Hugging Face for Enhanced Model Performance.ipynb

{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Fine-Tuning Hugging Face DistilBERT for Emotion Classification\n",
    "\n",
    "<img src=\"https://www.developer-tech.com/wp-content/uploads/sites/3/2023/08/intel-pytorch-foundation-ai-development-artificial-intelligence-coding-programming-machine-learning.jpg\" alt=\"Alt Text\" style=\"width: 400px;\"/>\n",
    "\n",
    "This notebook demonstrates how to fine-tune the `distilbert-base-uncased` model for multi-class emotion classification using Hugging Face's Transformers library. It covers data preprocessing, training, evaluation, and inference.\n",
    "\n",
    "---\n",
    "\n",
    "## Why This is Important\n",
    "\n",
    "Fine-tuning pre-trained models for specific tasks such as emotion classification enables developers to build NLP applications effectively. This hands-on guide will help you:\n",
    "\n",
    "- Load and preprocess datasets from the Hugging Face library.\n",
    "- Fine-tune a model for a specific classification task.\n",
    "- Evaluate and test the model's performance on new data.\n",
    "\n",
    "---\n"
   ],
   "id": "b5757744abb2d0c2"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Environment Setup and Dependencies Installation\n",
    "\n",
    "This cell ensures the required libraries are installed. We use `transformers`, `torch`, and other essential libraries to fine-tune the model.\n"
   ],
   "id": "2cd497ad3a2cbc7f"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:05:32.847500Z",
     "start_time": "2024-11-23T08:05:32.844438Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# !pip install transformers==4.35.2\n",
    "# !pip install torch==2.1.0\n",
    "# !pip install datasets==2.16.1\n",
    "# !pip install accelerate==0.26.0\n"
   ],
   "id": "7d813f0b2486107f",
   "outputs": [],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Loading Libraries and Packages\n",
    "\n",
    "Import the core libraries required for the task. These include Hugging Face's Transformers for the model and tokenizer, the Datasets library for data handling, and utilities like Numpy and Scikit-learn for performance evaluation.\n"
   ],
   "id": "3deb62df2081fbc7"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:05:35.656046Z",
     "start_time": "2024-11-23T08:05:32.864864Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments\n",
    "from datasets import load_dataset\n",
    "import numpy as np\n",
    "from sklearn.metrics import accuracy_score\n",
    "import torch\n"
   ],
   "id": "cf266263890e1153",
   "outputs": [],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Dataset Loading\n",
    "\n",
    "Load the `emotion` dataset from Hugging Face's Datasets library. This dataset is used for training and evaluation.\n"
   ],
   "id": "491a5e9b7417c14e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:05:49.399747Z",
     "start_time": "2024-11-23T08:05:35.662283Z"
    }
   },
   "cell_type": "code",
   "source": "dataset = load_dataset(\"emotion\")\n",
   "id": "b3f72c8ed941d156",
   "outputs": [],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Model and Tokenizer Initialization\n",
    "\n",
    "Initialize the pre-trained `distilbert-base-uncased` model and tokenizer. The model is configured for sequence classification with six labels.\n"
   ],
   "id": "5f6bfe53bd9a8f7d"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:05:50.634113Z",
     "start_time": "2024-11-23T08:05:49.406373Z"
    }
   },
   "cell_type": "code",
   "source": [
    "model_name = \"distilbert-base-uncased\"\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=6)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n"
   ],
   "id": "c7e69d031c38f734",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/aswathshakthi/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:797: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.bias', 'pre_classifier.bias', 'pre_classifier.weight', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Data Preprocessing\n",
    "\n",
    "Preprocess the dataset by tokenizing text inputs to make them compatible with the model.\n"
   ],
   "id": "59a9b7650dec3df4"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:05:50.657798Z",
     "start_time": "2024-11-23T08:05:50.642321Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def preprocess_function(examples):\n",
    "    return tokenizer(examples['text'], padding='max_length', truncation=True)\n",
    "\n",
    "# Apply preprocessing\n",
    "encoded_dataset = dataset.map(preprocess_function, batched=True)\n"
   ],
   "id": "81f496db82bae99b",
   "outputs": [],
   "execution_count": 6
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Training the Model\n",
    "\n",
    "Define training arguments and set up the Hugging Face `Trainer` for fine-tuning the model.\n"
   ],
   "id": "b160046205f2c792"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:28:26.103963Z",
     "start_time": "2024-11-23T08:05:50.665137Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Define training arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=\"./results\",\n",
    "    learning_rate=2e-5,\n",
    "    per_device_train_batch_size=16,\n",
    "    per_device_eval_batch_size=16,\n",
    "    num_train_epochs=1,\n",
    "    weight_decay=0.01,\n",
    "    evaluation_strategy=\"epoch\",\n",
    ")\n",
    "\n",
    "# Define the compute metrics function\n",
    "def compute_metrics(pred):\n",
    "    labels = pred.label_ids\n",
    "    preds = np.argmax(pred.predictions, axis=-1)\n",
    "    return {'accuracy': accuracy_score(labels, preds)}\n",
    "\n",
    "# Define the trainer\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    train_dataset=encoded_dataset[\"train\"],\n",
    "    eval_dataset=encoded_dataset[\"validation\"],\n",
    "    compute_metrics=compute_metrics,\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "trainer.train()\n"
   ],
   "id": "a089f86ec5eb9c9d",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ],
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='1000' max='1000' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [1000/1000 22:32, Epoch 1/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.276600</td>\n",
       "      <td>0.212747</td>\n",
       "      <td>0.921000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=1000, training_loss=0.5033166809082031, metrics={'train_runtime': 1354.9642, 'train_samples_per_second': 11.808, 'train_steps_per_second': 0.738, 'total_flos': 2119629570048000.0, 'train_loss': 0.5033166809082031, 'epoch': 1.0})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 7
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Model Evaluation\n",
    "\n",
    "Evaluate the model's performance on the validation dataset to check its accuracy in emotion classification.\n"
   ],
   "id": "a3169c8187983295"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:29:16.010142Z",
     "start_time": "2024-11-23T08:28:26.173035Z"
    }
   },
   "cell_type": "code",
   "source": "trainer.evaluate()\n",
   "id": "22714c1605650ae",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ],
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='125' max='125' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [125/125 00:49]\n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.21274743974208832,\n",
       " 'eval_accuracy': 0.921,\n",
       " 'eval_runtime': 49.8103,\n",
       " 'eval_samples_per_second': 40.152,\n",
       " 'eval_steps_per_second': 2.51,\n",
       " 'epoch': 1.0}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "#### Inference and Testing\n",
    "\n",
    "Test the fine-tuned model on new sentences to predict their associated emotions.\n"
   ],
   "id": "5c593dcd34af2e9c"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-23T08:48:18.643954Z",
     "start_time": "2024-11-23T08:48:18.372258Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Define test sentences\n",
    "test_sentences = [\n",
    "    \"I am feeling incredibly happy and joyful today!\",\n",
    "    \"I am so sad and down.\",\n",
    "    \"I have mixed feelings about this.\",\n",
    "    \"This is absolutely terrifying!\",\n",
    "]\n",
    "\n",
    "# Preprocess the test sentences\n",
    "encoded_input = tokenizer(test_sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')\n",
    "\n",
    "# Predict using the fine-tuned model\n",
    "with torch.no_grad():\n",
    "    predictions = model(**encoded_input)\n",
    "\n",
    "# Convert predictions to human-readable labels\n",
    "predicted_labels = np.argmax(predictions.logits.numpy(), axis=1)\n",
    "\n",
    "# Mapping for the 'emotion' dataset labels\n",
    "label_map = {0: \"sadness\", 1: \"joy\", 2: \"love\", 3: \"anger\", 4: \"fear\", 5: \"surprise\"}\n",
    "\n",
    "# Print predictions\n",
    "for sentence, label_id in zip(test_sentences, predicted_labels):\n",
    "    print(f\"Sentence: '{sentence}' - Emotion Prediction: {label_map[label_id]}\")\n"
   ],
   "id": "730b9169d4a79a32",
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Placeholder storage has not been allocated on MPS device!",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mRuntimeError\u001B[0m                              Traceback (most recent call last)",
      "Cell \u001B[0;32mIn[10], line 14\u001B[0m\n\u001B[1;32m     12\u001B[0m \u001B[38;5;66;03m# Predict using the fine-tuned model\u001B[39;00m\n\u001B[1;32m     13\u001B[0m \u001B[38;5;28;01mwith\u001B[39;00m torch\u001B[38;5;241m.\u001B[39mno_grad():\n\u001B[0;32m---> 14\u001B[0m     predictions \u001B[38;5;241m=\u001B[39m \u001B[43mmodel\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mencoded_input\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m     16\u001B[0m \u001B[38;5;66;03m# Convert predictions to human-readable labels\u001B[39;00m\n\u001B[1;32m     17\u001B[0m predicted_labels \u001B[38;5;241m=\u001B[39m np\u001B[38;5;241m.\u001B[39margmax(predictions\u001B[38;5;241m.\u001B[39mlogits\u001B[38;5;241m.\u001B[39mnumpy(), axis\u001B[38;5;241m=\u001B[39m\u001B[38;5;241m1\u001B[39m)\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1518\u001B[0m, in \u001B[0;36mModule._wrapped_call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1516\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_compiled_call_impl(\u001B[38;5;241m*\u001B[39margs, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mkwargs)  \u001B[38;5;66;03m# type: ignore[misc]\u001B[39;00m\n\u001B[1;32m   1517\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 1518\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_call_impl\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1527\u001B[0m, in \u001B[0;36mModule._call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1522\u001B[0m \u001B[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001B[39;00m\n\u001B[1;32m   1523\u001B[0m \u001B[38;5;66;03m# this function, and just call forward.\u001B[39;00m\n\u001B[1;32m   1524\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m (\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_pre_hooks\n\u001B[1;32m   1525\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_backward_hooks\n\u001B[1;32m   1526\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_forward_pre_hooks):\n\u001B[0;32m-> 1527\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mforward_call\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   1529\u001B[0m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[1;32m   1530\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py:779\u001B[0m, in \u001B[0;36mDistilBertForSequenceClassification.forward\u001B[0;34m(self, input_ids, attention_mask, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)\u001B[0m\n\u001B[1;32m    771\u001B[0m \u001B[38;5;250m\u001B[39m\u001B[38;5;124mr\u001B[39m\u001B[38;5;124;03m\"\"\"\u001B[39;00m\n\u001B[1;32m    772\u001B[0m \u001B[38;5;124;03mlabels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\u001B[39;00m\n\u001B[1;32m    773\u001B[0m \u001B[38;5;124;03m    Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,\u001B[39;00m\n\u001B[1;32m    774\u001B[0m \u001B[38;5;124;03m    config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If\u001B[39;00m\n\u001B[1;32m    775\u001B[0m \u001B[38;5;124;03m    `config.num_labels > 1` a classification loss is computed (Cross-Entropy).\u001B[39;00m\n\u001B[1;32m    776\u001B[0m \u001B[38;5;124;03m\"\"\"\u001B[39;00m\n\u001B[1;32m    777\u001B[0m return_dict \u001B[38;5;241m=\u001B[39m return_dict \u001B[38;5;28;01mif\u001B[39;00m return_dict \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m \u001B[38;5;28;01melse\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mconfig\u001B[38;5;241m.\u001B[39muse_return_dict\n\u001B[0;32m--> 779\u001B[0m distilbert_output \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mdistilbert\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m    780\u001B[0m \u001B[43m    \u001B[49m\u001B[43minput_ids\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43minput_ids\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    781\u001B[0m \u001B[43m    \u001B[49m\u001B[43mattention_mask\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mattention_mask\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    782\u001B[0m \u001B[43m    \u001B[49m\u001B[43mhead_mask\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mhead_mask\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    783\u001B[0m \u001B[43m    \u001B[49m\u001B[43minputs_embeds\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43minputs_embeds\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    784\u001B[0m \u001B[43m    \u001B[49m\u001B[43moutput_attentions\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43moutput_attentions\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    785\u001B[0m \u001B[43m    \u001B[49m\u001B[43moutput_hidden_states\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43moutput_hidden_states\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    786\u001B[0m \u001B[43m    \u001B[49m\u001B[43mreturn_dict\u001B[49m\u001B[38;5;241;43m=\u001B[39;49m\u001B[43mreturn_dict\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    787\u001B[0m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m    788\u001B[0m hidden_state \u001B[38;5;241m=\u001B[39m distilbert_output[\u001B[38;5;241m0\u001B[39m]  \u001B[38;5;66;03m# (bs, seq_len, dim)\u001B[39;00m\n\u001B[1;32m    789\u001B[0m pooled_output \u001B[38;5;241m=\u001B[39m hidden_state[:, \u001B[38;5;241m0\u001B[39m]  \u001B[38;5;66;03m# (bs, dim)\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1518\u001B[0m, in \u001B[0;36mModule._wrapped_call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1516\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_compiled_call_impl(\u001B[38;5;241m*\u001B[39margs, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mkwargs)  \u001B[38;5;66;03m# type: ignore[misc]\u001B[39;00m\n\u001B[1;32m   1517\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 1518\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_call_impl\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1527\u001B[0m, in \u001B[0;36mModule._call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1522\u001B[0m \u001B[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001B[39;00m\n\u001B[1;32m   1523\u001B[0m \u001B[38;5;66;03m# this function, and just call forward.\u001B[39;00m\n\u001B[1;32m   1524\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m (\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_pre_hooks\n\u001B[1;32m   1525\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_backward_hooks\n\u001B[1;32m   1526\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_forward_pre_hooks):\n\u001B[0;32m-> 1527\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mforward_call\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   1529\u001B[0m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[1;32m   1530\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py:597\u001B[0m, in \u001B[0;36mDistilBertModel.forward\u001B[0;34m(self, input_ids, attention_mask, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict)\u001B[0m\n\u001B[1;32m    594\u001B[0m \u001B[38;5;66;03m# Prepare head mask if needed\u001B[39;00m\n\u001B[1;32m    595\u001B[0m head_mask \u001B[38;5;241m=\u001B[39m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mget_head_mask(head_mask, \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mconfig\u001B[38;5;241m.\u001B[39mnum_hidden_layers)\n\u001B[0;32m--> 597\u001B[0m embeddings \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43membeddings\u001B[49m\u001B[43m(\u001B[49m\u001B[43minput_ids\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43minputs_embeds\u001B[49m\u001B[43m)\u001B[49m  \u001B[38;5;66;03m# (bs, seq_length, dim)\u001B[39;00m\n\u001B[1;32m    599\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mtransformer(\n\u001B[1;32m    600\u001B[0m     x\u001B[38;5;241m=\u001B[39membeddings,\n\u001B[1;32m    601\u001B[0m     attn_mask\u001B[38;5;241m=\u001B[39mattention_mask,\n\u001B[0;32m   (...)\u001B[0m\n\u001B[1;32m    605\u001B[0m     return_dict\u001B[38;5;241m=\u001B[39mreturn_dict,\n\u001B[1;32m    606\u001B[0m )\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1518\u001B[0m, in \u001B[0;36mModule._wrapped_call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1516\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_compiled_call_impl(\u001B[38;5;241m*\u001B[39margs, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mkwargs)  \u001B[38;5;66;03m# type: ignore[misc]\u001B[39;00m\n\u001B[1;32m   1517\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 1518\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_call_impl\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1527\u001B[0m, in \u001B[0;36mModule._call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1522\u001B[0m \u001B[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001B[39;00m\n\u001B[1;32m   1523\u001B[0m \u001B[38;5;66;03m# this function, and just call forward.\u001B[39;00m\n\u001B[1;32m   1524\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m (\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_pre_hooks\n\u001B[1;32m   1525\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_backward_hooks\n\u001B[1;32m   1526\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_forward_pre_hooks):\n\u001B[0;32m-> 1527\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mforward_call\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   1529\u001B[0m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[1;32m   1530\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py:120\u001B[0m, in \u001B[0;36mEmbeddings.forward\u001B[0;34m(self, input_ids, input_embeds)\u001B[0m\n\u001B[1;32m    108\u001B[0m \u001B[38;5;250m\u001B[39m\u001B[38;5;124;03m\"\"\"\u001B[39;00m\n\u001B[1;32m    109\u001B[0m \u001B[38;5;124;03mParameters:\u001B[39;00m\n\u001B[1;32m    110\u001B[0m \u001B[38;5;124;03m    input_ids (torch.Tensor):\u001B[39;00m\n\u001B[0;32m   (...)\u001B[0m\n\u001B[1;32m    117\u001B[0m \u001B[38;5;124;03membeddings)\u001B[39;00m\n\u001B[1;32m    118\u001B[0m \u001B[38;5;124;03m\"\"\"\u001B[39;00m\n\u001B[1;32m    119\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m input_ids \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m:\n\u001B[0;32m--> 120\u001B[0m     input_embeds \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mword_embeddings\u001B[49m\u001B[43m(\u001B[49m\u001B[43minput_ids\u001B[49m\u001B[43m)\u001B[49m  \u001B[38;5;66;03m# (bs, max_seq_length, dim)\u001B[39;00m\n\u001B[1;32m    122\u001B[0m seq_length \u001B[38;5;241m=\u001B[39m input_embeds\u001B[38;5;241m.\u001B[39msize(\u001B[38;5;241m1\u001B[39m)\n\u001B[1;32m    124\u001B[0m \u001B[38;5;66;03m# Setting the position-ids to the registered buffer in constructor, it helps\u001B[39;00m\n\u001B[1;32m    125\u001B[0m \u001B[38;5;66;03m# when tracing the model without passing position-ids, solves\u001B[39;00m\n\u001B[1;32m    126\u001B[0m \u001B[38;5;66;03m# isues similar to issue #5664\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1518\u001B[0m, in \u001B[0;36mModule._wrapped_call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1516\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_compiled_call_impl(\u001B[38;5;241m*\u001B[39margs, \u001B[38;5;241m*\u001B[39m\u001B[38;5;241m*\u001B[39mkwargs)  \u001B[38;5;66;03m# type: ignore[misc]\u001B[39;00m\n\u001B[1;32m   1517\u001B[0m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[0;32m-> 1518\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43m_call_impl\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1527\u001B[0m, in \u001B[0;36mModule._call_impl\u001B[0;34m(self, *args, **kwargs)\u001B[0m\n\u001B[1;32m   1522\u001B[0m \u001B[38;5;66;03m# If we don't have any hooks, we want to skip the rest of the logic in\u001B[39;00m\n\u001B[1;32m   1523\u001B[0m \u001B[38;5;66;03m# this function, and just call forward.\u001B[39;00m\n\u001B[1;32m   1524\u001B[0m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m (\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m \u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39m_forward_pre_hooks\n\u001B[1;32m   1525\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_backward_pre_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_backward_hooks\n\u001B[1;32m   1526\u001B[0m         \u001B[38;5;129;01mor\u001B[39;00m _global_forward_hooks \u001B[38;5;129;01mor\u001B[39;00m _global_forward_pre_hooks):\n\u001B[0;32m-> 1527\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mforward_call\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43margs\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[38;5;241;43m*\u001B[39;49m\u001B[43mkwargs\u001B[49m\u001B[43m)\u001B[49m\n\u001B[1;32m   1529\u001B[0m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[1;32m   1530\u001B[0m     result \u001B[38;5;241m=\u001B[39m \u001B[38;5;28;01mNone\u001B[39;00m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/modules/sparse.py:162\u001B[0m, in \u001B[0;36mEmbedding.forward\u001B[0;34m(self, input)\u001B[0m\n\u001B[1;32m    161\u001B[0m \u001B[38;5;28;01mdef\u001B[39;00m \u001B[38;5;21mforward\u001B[39m(\u001B[38;5;28mself\u001B[39m, \u001B[38;5;28minput\u001B[39m: Tensor) \u001B[38;5;241m-\u001B[39m\u001B[38;5;241m>\u001B[39m Tensor:\n\u001B[0;32m--> 162\u001B[0m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mF\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43membedding\u001B[49m\u001B[43m(\u001B[49m\n\u001B[1;32m    163\u001B[0m \u001B[43m        \u001B[49m\u001B[38;5;28;43minput\u001B[39;49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mweight\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mpadding_idx\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mmax_norm\u001B[49m\u001B[43m,\u001B[49m\n\u001B[1;32m    164\u001B[0m \u001B[43m        \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mnorm_type\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mscale_grad_by_freq\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43msparse\u001B[49m\u001B[43m)\u001B[49m\n",
      "File \u001B[0;32m~/PycharmProjects/MLOps/madewithml/venv/lib/python3.10/site-packages/torch/nn/functional.py:2233\u001B[0m, in \u001B[0;36membedding\u001B[0;34m(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)\u001B[0m\n\u001B[1;32m   2227\u001B[0m     \u001B[38;5;66;03m# Note [embedding_renorm set_grad_enabled]\u001B[39;00m\n\u001B[1;32m   2228\u001B[0m     \u001B[38;5;66;03m# XXX: equivalent to\u001B[39;00m\n\u001B[1;32m   2229\u001B[0m     \u001B[38;5;66;03m# with torch.no_grad():\u001B[39;00m\n\u001B[1;32m   2230\u001B[0m     \u001B[38;5;66;03m#   torch.embedding_renorm_\u001B[39;00m\n\u001B[1;32m   2231\u001B[0m     \u001B[38;5;66;03m# remove once script supports set_grad_enabled\u001B[39;00m\n\u001B[1;32m   2232\u001B[0m     _no_grad_embedding_renorm_(weight, \u001B[38;5;28minput\u001B[39m, max_norm, norm_type)\n\u001B[0;32m-> 2233\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mtorch\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43membedding\u001B[49m\u001B[43m(\u001B[49m\u001B[43mweight\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[38;5;28;43minput\u001B[39;49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mpadding_idx\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mscale_grad_by_freq\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43msparse\u001B[49m\u001B[43m)\u001B[49m\n",
      "\u001B[0;31mRuntimeError\u001B[0m: Placeholder storage has not been allocated on MPS device!"
     ]
    }
   ],
   "execution_count": 10
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# Conclusion and Discussion\n",
    "\n",
    "### Conclusion\n",
    "\n",
    "This notebook demonstrated how to fine-tune a pre-trained Hugging Face model for emotion classification. By walking through data preprocessing, model training, evaluation, and testing, we successfully utilized `distilbert-base-uncased` to classify emotions.\n",
    "\n",
    "### Discussion\n",
    "\n",
    "The ease of integration provided by Hugging Face's tools allows developers to focus on the task rather than complex configurations. Fine-tuning models like DistilBERT provides accurate results for various NLP tasks, paving the way for further advancements in AI and natural language understanding.\n"
   ],
   "id": "f6cb688c1c729eac"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}