models/model_train/image-segmentation_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "00024d19",
   "metadata": {},
   "source": [
    "<td>\n",
    "   <a target=\"_blank\" href=\"https://www.clarifai.com/\" ><img src=\"https://upload.wikimedia.org/wikipedia/commons/b/bc/Clarifai_Logo_FC_Web.png\" width=256/></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da69603",
   "metadata": {},
   "source": [
    "<td>\n",
    "<a href=\"https://colab.research.google.com/github/Clarifai/examples/blob/main/models/model_train/image-segmentation_training.ipynb\" target=\"_blank\"><img\n",
    "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Colab\"></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2482f50",
   "metadata": {},
   "source": [
    "# Models\n",
    "\n",
    "\n",
    "\n",
    "Clarifai offers a range of powerful model types, each designed to generate meaningful outputs based on user specific inputs and AI tasks.\n",
    "\n",
    "There are wide variety of models that can be used as standalone solutions, or as building blocks for your own custom business solutions.\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db356341",
   "metadata": {},
   "source": [
    "Clarifai Models are the recommended starting points for many users because they offer incredibly fast training times when you customize them using the \"embedding-classifier\" (Transfer Learning Classifier) model type.\n",
    "\n",
    "But there are many cases where accuracy and the ability to carefully target solutions take priority over speed and ease of use. Additionally, you may need a model to learn new features, not recognized by existing Clarifai Models. For these cases, it is possible to \"deep fine-tune\" your custom models and integrate them directly within your workflows.\n",
    "\n",
    "You might consider deep training if you have:\n",
    "\n",
    "- A custom tailored dataset\n",
    "- Accurate labels\n",
    "- Expertise and time to fine-tune models\n",
    "\n",
    "_______\n",
    "On the [Clarifai Community](https://clarifai.com/explore) explore page, you can click the [Models](https://clarifai.com/explore/models) tab to search and access the models available for everyone to use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cde3b72",
   "metadata": {},
   "source": [
    "This notebook contains Model Train demo for **visual-segmenter** Model Type with **MMDetection** Template."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "800de27a",
   "metadata": {},
   "source": [
    "# Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fd7f15",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfb70f5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install clarifai'[all]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ed766dc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ[\"CLARIFAI_PAT\"] = \"PAT\" # replace with your own PAT key here"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49de2e2b",
   "metadata": {},
   "source": [
    "*Note: Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46eaf1e6",
   "metadata": {},
   "source": [
    "### For Colab\n",
    "To access data files from Clarifai examples repo, you can clone the repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f3c537b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/Clarifai/examples.git\n",
    "%cd /content/examples/models/model_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f30d26ea",
   "metadata": {},
   "source": [
    "### For Local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c70785ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "git clone https://github.com/Clarifai/examples.git\n",
    "cd examples/models/model_train"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "407c786d",
   "metadata": {},
   "source": [
    "## VISUAL-SEGMENTER"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "289ca9ef",
   "metadata": {},
   "source": [
    "**Input: Images and videos**\n",
    "\n",
    "**Output: Regions**\n",
    "\n",
    "Visual Segmenter, also known as Semantic Segmentation, is a type of deep fine-tuned model used in image analysis and understanding tasks.\n",
    "\n",
    "It aims to achieve a fine-grained understanding of the content within an image by associating each pixel with a particular class label. This is more detailed than traditional object detection, which typically identifies bounding boxes around objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259ecd1a",
   "metadata": {},
   "source": [
    "### Creating an App"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5eaf6277",
   "metadata": {},
   "outputs": [],
   "source": [
    "from clarifai.client.user import User\n",
    "#replace your \"user_id\"\n",
    "client = User(user_id=\"user_id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "28056328",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">2024-03-20 15:11:55 </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> clarifai.client.user:                                                      <a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/user.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">user.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/user.py#127\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">127</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         App created                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">           </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         code: SUCCESS                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">           </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         description: <span style=\"color: #008000; text-decoration-color: #008000\">\"Ok\"</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">           </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         req_id: <span style=\"color: #008000; text-decoration-color: #008000\">\"69762f1acc47b1b9f14cca43a67ff741\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">           </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">           </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m2024-03-20 15:11:55\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m clarifai.client.user:                                                      \u001b]8;id=977135;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/user.py\u001b\\\u001b[2muser.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=812141;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/user.py#127\u001b\\\u001b[2m127\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         App created                                                                \u001b[2m           \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         code: SUCCESS                                                              \u001b[2m           \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         description: \u001b[32m\"Ok\"\u001b[0m                                                          \u001b[2m           \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         req_id: \u001b[32m\"69762f1acc47b1b9f14cca43a67ff741\"\u001b[0m                                 \u001b[2m           \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                                                                                    \u001b[2m           \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "app = client.create_app(app_id=\"app_id\", base_workflow=\"Universal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdd607e6",
   "metadata": {},
   "source": [
    "### Uploading Image Segmentation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d5eb9227",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/adithyansukumar/work/ml_training_error/examples/datasets/upload/image_segmentation/coco'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "module_path = os.path.join(os.getcwd().split('/models/model_train')[0],'datasets/upload/image_segmentation/coco')\n",
    "module_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a57643bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing load_module_dataloader for calling the dataloader object in dataset.py in the local data folder\n",
    "from clarifai.datasets.upload.utils import load_module_dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b85f1bcc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading annotations into memory...\n",
      "Done (t=0.00s)\n",
      "creating index...\n",
      "index created!\n"
     ]
    }
   ],
   "source": [
    "coco_dataloader = load_module_dataloader(module_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "24264654",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">2024-03-20 15:12:01 </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> clarifai.client.app:                                                        <a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">app.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py#310\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">310</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Dataset created                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         code: SUCCESS                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         description: <span style=\"color: #008000; text-decoration-color: #008000\">\"Ok\"</span>                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         req_id: <span style=\"color: #008000; text-decoration-color: #008000\">\"d44173ad561e47bb435f92d564b70d5e\"</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m2024-03-20 15:12:01\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m clarifai.client.app:                                                        \u001b]8;id=480330;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py\u001b\\\u001b[2mapp.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=177524;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py#310\u001b\\\u001b[2m310\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         Dataset created                                                             \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         code: SUCCESS                                                               \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         description: \u001b[32m\"Ok\"\u001b[0m                                                           \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         req_id: \u001b[32m\"d44173ad561e47bb435f92d564b70d5e\"\u001b[0m                                  \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                                                                                     \u001b[2m          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Uploading Dataset: 100%|██████████| 1/1 [01:04<00:00, 64.69s/it]\n"
     ]
    }
   ],
   "source": [
    "dataset = app.create_dataset(dataset_id=\"segmentation_dataset\")\n",
    "dataset.upload_dataset(dataloader=coco_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a40825c",
   "metadata": {},
   "source": [
    "### List Trainable Model Types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "71254639",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['visual-classifier',\n",
       " 'visual-detector',\n",
       " 'visual-segmenter',\n",
       " 'visual-embedder',\n",
       " 'clusterer',\n",
       " 'text-classifier',\n",
       " 'embedding-classifier',\n",
       " 'text-to-text']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "app.list_trainable_model_types()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e02f87b2",
   "metadata": {},
   "source": [
    "### Create a Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "467498cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">2024-03-20 15:13:06 </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> clarifai.client.app:                                                        <a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">app.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py#334\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">334</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Model created                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         code: SUCCESS                                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         description: <span style=\"color: #008000; text-decoration-color: #008000\">\"Ok\"</span>                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         req_id: <span style=\"color: #008000; text-decoration-color: #008000\">\"38ca9570013f8135372a318da9e4f28e\"</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">          </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m2024-03-20 15:13:06\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m clarifai.client.app:                                                        \u001b]8;id=762214;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py\u001b\\\u001b[2mapp.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=379523;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/app.py#334\u001b\\\u001b[2m334\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         Model created                                                               \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         code: SUCCESS                                                               \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         description: \u001b[32m\"Ok\"\u001b[0m                                                           \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         req_id: \u001b[32m\"38ca9570013f8135372a318da9e4f28e\"\u001b[0m                                  \u001b[2m          \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                                                                                     \u001b[2m          \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "MODEL_ID = \"segmenter\"\n",
    "MODEL_TYPE_ID = \"visual-segmenter\"\n",
    "model = app.create_model(model_id=MODEL_ID, model_type_id=MODEL_TYPE_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc16c4d8",
   "metadata": {},
   "source": [
    "### List Templates for the Model Type\n",
    "\n",
    "Templates give you the control to choose the specific architecture used by your neural network, as well as define a set of hyperparameters you can use to fine-tune the way your model learns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9f2d3b9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['MMSegmentation_AdvancedConfig', 'MMSegmentation_SegFormer']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.list_training_templates()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f3ca868",
   "metadata": {},
   "source": [
    "### Save params\n",
    "Save the parameters for the specific model template in a YAML file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "aa53dc65",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_params = model.get_params(template='MMSegmentation_SegFormer')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8c87de00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dataset_id': '',\n",
       " 'dataset_version_id': '',\n",
       " 'concepts': [],\n",
       " 'train_params': {'invalid_data_tolerance_percent': 5.0,\n",
       "  'template': 'MMSegmentation_SegFormer',\n",
       "  'image_size': [520.0],\n",
       "  'batch_size': 2.0,\n",
       "  'num_epochs': 1.0,\n",
       "  'per_item_lrate': 7.5e-06,\n",
       "  'pretrained_weights': 'ade20k'}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import yaml\n",
    "# Preview YAML content\n",
    "YAML_FILE_PATH=\"/Users/adithyansukumar/work/ml_training_error/examples/models/model_train/params.yaml\"\n",
    "file = open(YAML_FILE_PATH)\n",
    "data = yaml.safe_load(file)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcec6009",
   "metadata": {},
   "source": [
    "#### Get param info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2199cf92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'fieldType': 'ENUM', 'defaultValue': 'ade20k', 'description': 'whether to init with pretrained weights.', 'modelTypeEnumOptions': [{'id': 'None'}, {'id': 'ade20k'}], 'param': 'pretrained_weights'}\n"
     ]
    }
   ],
   "source": [
    "print(model.get_param_info(param = 'pretrained_weights'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf22475",
   "metadata": {},
   "source": [
    "### Update params\n",
    "Note: User can edit the params in the YAML file or directly update the params using model.update_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9e785b5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['id-elephant',\n",
       " 'id-laptop',\n",
       " 'id-diningtable',\n",
       " 'id-cellphone',\n",
       " 'id-cat',\n",
       " 'id-keyboard',\n",
       " 'id-mouse',\n",
       " 'id-toilet',\n",
       " 'id-oven',\n",
       " 'id-pizza',\n",
       " 'id-clock',\n",
       " 'id-chair',\n",
       " 'id-cup',\n",
       " 'id-kite',\n",
       " 'id-couch',\n",
       " 'id-baseballbat',\n",
       " 'id-handbag',\n",
       " 'id-person']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#list concepts in the app\n",
    "concepts = [concept.id for concept in app.list_concepts()]\n",
    "concepts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "77055b0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">2024-03-20 15:15:30 </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> clarifai.client.dataset:                                                <a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/dataset.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">dataset.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/dataset.py#101\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">101</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Dataset Version created                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         code: SUCCESS                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         description: <span style=\"color: #008000; text-decoration-color: #008000\">\"Ok\"</span>                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         req_id: <span style=\"color: #008000; text-decoration-color: #008000\">\"6f611476ff874a0889fc6f0b974b50ac\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m2024-03-20 15:15:30\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m clarifai.client.dataset:                                                \u001b]8;id=455037;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/dataset.py\u001b\\\u001b[2mdataset.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=608924;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/dataset.py#101\u001b\\\u001b[2m101\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         Dataset Version created                                                 \u001b[2m              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         code: SUCCESS                                                           \u001b[2m              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         description: \u001b[32m\"Ok\"\u001b[0m                                                       \u001b[2m              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         req_id: \u001b[32m\"6f611476ff874a0889fc6f0b974b50ac\"\u001b[0m                              \u001b[2m              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                                                                                 \u001b[2m              \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'d9b62cac13a2480c819b8341d73794ca'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#creating dataset version\n",
    "dataset_version = dataset.create_version()\n",
    "dataset_version_id = dataset_version.version.id\n",
    "dataset_version_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a1adb4dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#update params\n",
    "model.update_params(dataset_id = 'segmentation_dataset', dataset_version_id=dataset_version_id,\n",
    "                    concepts = concepts, num_epochs = 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f9f252b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dataset_id': 'segmentation_dataset',\n",
       " 'dataset_version_id': 'd9b62cac13a2480c819b8341d73794ca',\n",
       " 'concepts': ['id-elephant',\n",
       "  'id-laptop',\n",
       "  'id-diningtable',\n",
       "  'id-cellphone',\n",
       "  'id-cat',\n",
       "  'id-keyboard',\n",
       "  'id-mouse',\n",
       "  'id-toilet',\n",
       "  'id-oven',\n",
       "  'id-pizza',\n",
       "  'id-clock',\n",
       "  'id-chair',\n",
       "  'id-cup',\n",
       "  'id-kite',\n",
       "  'id-couch',\n",
       "  'id-baseballbat',\n",
       "  'id-handbag',\n",
       "  'id-person'],\n",
       " 'train_params': {'invalid_data_tolerance_percent': 5.0,\n",
       "  'template': 'MMSegmentation_SegFormer',\n",
       "  'image_size': [520.0],\n",
       "  'batch_size': 2.0,\n",
       "  'num_epochs': 5,\n",
       "  'per_item_lrate': 7.5e-06,\n",
       "  'pretrained_weights': 'ade20k'}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.training_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92dcf007",
   "metadata": {},
   "source": [
    "### Start Model Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0b5383b0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">2024-03-20 15:15:37 </span><span style=\"color: #000080; text-decoration-color: #000080\">INFO    </span> clarifai.client.model:                                                    <a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/model.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">model.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/model.py#241\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">241</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         Model Training Started                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         code: SUCCESS                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         description: <span style=\"color: #008000; text-decoration-color: #008000\">\"Ok\"</span>                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         req_id: <span style=\"color: #008000; text-decoration-color: #008000\">\"0db699c6f5b7bd9890f82def15bd78dc\"</span>                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                                                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m2024-03-20 15:15:37\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO    \u001b[0m clarifai.client.model:                                                    \u001b]8;id=734384;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/model.py\u001b\\\u001b[2mmodel.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=75903;file:///Users/adithyansukumar/.pyenv/versions/3.9.10/lib/python3.9/site-packages/clarifai/client/model.py#241\u001b\\\u001b[2m241\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         Model Training Started                                                    \u001b[2m            \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         code: SUCCESS                                                             \u001b[2m            \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         description: \u001b[32m\"Ok\"\u001b[0m                                                         \u001b[2m            \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         req_id: \u001b[32m\"0db699c6f5b7bd9890f82def15bd78dc\"\u001b[0m                                \u001b[2m            \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                                                                                   \u001b[2m            \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_version_id = model.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f48cab5",
   "metadata": {},
   "source": [
    "### Check Model Training Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6830219e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current Status: code: MODEL_QUEUED_FOR_TRAINING\n",
      "description: \"Model is currently in queue for training.\"\n",
      "\n",
      "Waiting---\n",
      "Current Status: code: MODEL_TRAINING\n",
      "description: \"Training stage in progress: 0/1 complete.\"\n",
      "\n",
      "Waiting---\n",
      "Current Status: code: MODEL_TRAINING\n",
      "description: \"Deployment stage in progress: 3/4 complete.\"\n",
      "\n",
      "Waiting---\n",
      "Current Status: code: MODEL_TRAINING\n",
      "description: \"Deployment stage in progress: 3/4 complete.\"\n",
      "\n",
      "Waiting---\n",
      "code: MODEL_TRAINED\n",
      "description: \"Model is trained and ready\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "while True:\n",
    "    status = model.training_status(version_id=model_version_id,training_logs=False)\n",
    "    if status.code == 21106: #MODEL_TRAINING_FAILED\n",
    "        print(status)\n",
    "        break\n",
    "    elif status.code == 21100: #MODEL_TRAINED\n",
    "        print(status)\n",
    "        break\n",
    "    else:\n",
    "        print(\"Current Status:\",status)\n",
    "        print(\"Waiting---\")\n",
    "        time.sleep(120)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdd71761",
   "metadata": {},
   "source": [
    "### Model Prediction\n",
    "Predicting with the Trained Model.\n",
    "Note: Refer this [notebook](https://github.com/Clarifai/examples/blob/main/models/model_predict.ipynb) for more info on Model Predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c5fdb80c",
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_PATH = os.path.join(os.getcwd().split('/models')[0],'datasets/upload/image_segmentation/coco/images/000000166563.jpg')\n",
    "\n",
    "prediction_response = model.predict_by_filepath(IMAGE_PATH, input_type=\"image\")\n",
    "\n",
    "# Get the output\n",
    "regions = prediction_response.outputs[0].data.regions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "622adab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the predicted masks\n",
    "import cv2\n",
    "from urllib.request import urlopen\n",
    "import numpy as np\n",
    "from PIL import Image as PILImage\n",
    "from io import BytesIO\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.patches as mpatches\n",
    "\n",
    "img = plt.imread(IMAGE_PATH)\n",
    "masks = []\n",
    "concepts = []\n",
    "for region in regions:\n",
    "    if region.data.concepts[0].value > 0.05:\n",
    "        masks.append(np.array(PILImage.open(BytesIO(region.region_info.mask.image.base64))))\n",
    "        concepts.append(region.data.concepts[0].name)\n",
    "\n",
    "\n",
    "# Generate random colors\n",
    "colors = []\n",
    "for i in range(len(masks)):\n",
    "    r = random.randint(0,255)\n",
    "    g = random.randint(0,255)\n",
    "    b = random.randint(0,255)\n",
    "    colors.append((b,g,r))\n",
    "\n",
    "# Map masks to overlays\n",
    "overlays = []\n",
    "for i in range(len(masks)):\n",
    "    mask = masks[i]\n",
    "    color = colors[i]\n",
    "\n",
    "    overlay = np.zeros_like(img)\n",
    "    overlay[mask > 0] = color\n",
    "    overlays.append(overlay)\n",
    "\n",
    "# Overlay masks on original image\n",
    "overlayed = np.copy(img)\n",
    "\n",
    "for overlay in overlays:\n",
    "  # Apply alpha blending\n",
    "  cv2.addWeighted(overlay, 0.15, overlayed, 0.85, 0, overlayed)\n",
    "\n",
    "overlayed = cv2.convertScaleAbs(overlayed, alpha=1.5, beta=50)\n",
    "\n",
    "\n",
    "# Display overlayed image\n",
    "plt.imshow(overlayed) \n",
    "\n",
    "# Create legend with colors and concepts\n",
    "legend_items = []\n",
    "for i in range(len(overlays)):\n",
    "    color = [c/255 for c in colors[i]]\n",
    "    concept = concepts[i]\n",
    "    legend_items.append(mpatches.Patch(color=color, label=concept))\n",
    "\n",
    "plt.legend(handles=legend_items, loc='lower left', bbox_to_anchor=(1.05, 0))\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62be4db",
   "metadata": {},
   "source": [
    "## Note\n",
    "\n",
    "- This Notebook is a demo to get started with Model Training in Clarifai Platform with Python SDK.\n",
    "- For better accuracy of the Model, Choose your own data and different Templates and Hyperparams."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66f7c3f9",
   "metadata": {},
   "source": [
    "## Clarifai Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83d01c6",
   "metadata": {},
   "source": [
    "**Website**: [https://www.clarifai.com](https://www.clarifai.com/)\n",
    "\n",
    "**Demo**: [https://clarifai.com/demo](https://clarifai.com/demo)\n",
    "\n",
    "**Sign up for a free Account**: [https://clarifai.com/signup](https://clarifai.com/signup)\n",
    "\n",
    "**Developer Guide**: [https://docs.clarifai.com](https://docs.clarifai.com/)\n",
    "\n",
    "**Clarifai Community**: [https://clarifai.com/explore](https://clarifai.com/explore)\n",
    "\n",
    "**Python SDK Docs**: [https://docs.clarifai.com/python-sdk/api-reference](https://docs.clarifai.com/python-sdk/api-reference)\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}