monailabel/monailabel_endoscopy_cvat_tooltracking.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License. \n",
    "\n",
    "# MONAI Label Endoscopy App - Surgical Tool Segmentation with CVAT\n",
    "\n",
    "***The computer-assisted intervention (CAI) workflow with MONAI Label***\n",
    "\n",
    "In this notebook, we provide the endoscopy app example with surgical tool segmentation. The tooltracking bundle is used for the demonstration, CVAT+Nuclio as the client viewer and function deployment platform, we show how MONAI Label can provide interactive labeling support for endoscopic applications.\n",
    "\n",
    "\n",
    "**Surgical Tool Segmentation with the tooltracking bundle:**\n",
    "\n",
    "This model is trained using a flexible UNet structure with a UNet architecture decoder. Datasets use private samples from Activ Surgical.\n",
    "\n",
    "In this tutorial, we provide publicly available sample data for demonstration. We highlight the active learning and continuous learning process with MONAI Label + Endoscopic integration.\n",
    "\n",
    "![cvat](./figures/monailabel_endoscopy_cvat/cvat.png)\n",
    "\n",
    "## 1. Prepare MONAI Label\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "\n",
    "### Prerequisites\n",
    "- **Install MONAI Label** weekly preview release: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install monailabel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download endoscopy app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel apps --download --name endoscopy --output apps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download sample data\n",
    "Several sample data are provided for this application, the sample images are from [EndoVis17](https://endovis.grand-challenge.org/) data, which is public. Users can create their own dataset format similar to the sample data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir datasets\n",
    "!wget \"https://github.com/Project-MONAI/MONAILabel/releases/download/data/endoscopy_frames.zip\" -O datasets/endoscopy_frames.zip\n",
    "!unzip datasets/endoscopy_frames.zip -d datasets/endoscopy_frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Create a validation set and modify bundle config files\n",
    "\n",
    "Users need to provide a validation set so that the training process can evaluate model performance and save model checkpoints with the best metric. Active learning process requires the validation set.\n",
    "\n",
    "Create the validation set following the bundle config files, the path and filenames need to be consistent between file system and the train.json.\n",
    "\n",
    "Modify the train.json in the bundle config file, the ```dataset_dir```, ```val_images```, and ```val_labels``` are required to be the same with file system.\n",
    "\n",
    "![data1](./figures/monailabel_endoscopy_cvat/data1.png)\n",
    "\n",
    "![bundle](./figures/monailabel_endoscopy_cvat/bundle_config.png)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.  Setup CVAT and Nuclio\n",
    "\n",
    "\n",
    "CVAT and Nuclio are not maintained by MONAI Label team.  We provide minimized setup steps for using CVAT, more details about CVAT preparation can be found in:\n",
    "\n",
    "- [CVAT Viewer Configuration](https://opencv.github.io/cvat/docs/administration/basics/installation/)\n",
    "- [Semi-automatic and Automatic Annotation](https://opencv.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/)\n",
    "\n",
    "For following step, setup in the local machine (e.g., Ubuntu), and execute commands.\n",
    "\n",
    "\n",
    "### 2.1 Step 1: Get CVAT from GitHub and checkout to version 2.1.0\n",
    "\n",
    "Open a **Terminal** in local machine and execute:\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/opencv/cvat\n",
    "cd cvat\n",
    "# Checkout to required stable version to v2.1.0\n",
    "git checkout v2.1.0\n",
    "```\n",
    "\n",
    "### 2.2 Step 2: Set the Host and CVAT version, configure CVAT\n",
    "\n",
    "\n",
    "```bash\n",
    "# use real-ip instead of localhost to make the CVAT projects sharable\n",
    "export CVAT_HOST=127.0.0.1\n",
    "export CVAT_VERSION=v2.1.0\n",
    "# Start CVAT from docker-compose, make sure the IP and port are available.\n",
    "python3 -m pip install setuptools docker-compose # this is optional if docker-compose is already installed.\n",
    "docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d\n",
    "# Create CVAT username and password\n",
    "docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser'\n",
    "```\n",
    "\n",
    "The setup process takes port 8070, 8080, 8090, if alternative ports are preferred, refer to [CVAT Guide](https://opencv.github.io/cvat/docs/administration/basics/installation/)\n",
    "\n",
    "CVAT should be on after this step, refer to **http://127.0.0.1:8080** in Chrome to open CVAT, login with the superuser username and password:\n",
    "\n",
    "![cvat1](./figures/monailabel_endoscopy_cvat/cvat1.png)\n",
    "\n",
    "\n",
    "### 2.3 Step 3: Setup Nuclio container platform for function container management\n",
    "\n",
    "In the local host machine, start a **Terminal** to open a command line interface.\n",
    "\n",
    "```bash\n",
    "# Get Nuclio dashboard \n",
    "wget https://github.com/nuclio/nuclio/releases/download/1.5.16/nuctl-1.5.16-linux-amd64\n",
    "chmod +x nuctl-1.5.16-linux-amd64\n",
    "ln -sf $(pwd)/nuctl-1.5.16-linux-amd64 /usr/local/bin/nuctl\n",
    "\n",
    "```\n",
    "\n",
    "### 2.4 Step 4: Endoscopic Models Deployment with MONAI Label\n",
    "\n",
    "This step is to deploy MONAI Label plugin with endoscopic models using Nuclio tool.\n",
    "\n",
    "```bash\n",
    "# make a working folder in local machine, name workspace\n",
    "mkdir workspace\n",
    "\n",
    "# 1: If in Toolkit, users can get CVAT plugin folder from the toolkit docker\n",
    "docker cp $(docker container ls | grep 'nvcr.io/nvidia/clara/monai-toolkit' | awk '{print $1}'):/usr/local/monailabel/plugins/cvat workspace\n",
    "# Deploy all endoscopy models\n",
    "./workspace/cvat/deploy.sh endoscopy\n",
    "# Or to deploy specific function and model, e.g., tooltracking\n",
    "./workspace/cvat/deploy.sh endoscopy tooltracking\n",
    "\n",
    "# 2 : If MONAI Label is used in local and acquired from Github, get latest CVAT plugin folder from the Github\n",
    "git clone https://github.com/Project-MONAI/MONAILabel.git\n",
    "# Deploy all endoscopy models\n",
    "./MONAILabel/plugins/cvat/deploy.sh endoscopy\n",
    "# Or to deploy specific function and model, e.g., tooltracking\n",
    "./MONAILabel/plugins/cvat/deploy.sh endoscopy tooltracking\n",
    "```\n",
    "\n",
    "After model deployment, users can see the model names in the **Models** tag of CVAT\n",
    "\n",
    "![cvat2](./figures/monailabel_endoscopy_cvat/cvat2.png)\n",
    "\n",
    "To check or monitor the status of deployed function container, users can open Nuclio platform at http://127.0.0.1:8070 by default and see whether deployed models are running.\n",
    "\n",
    "![cvat3](./figures/monailabel_endoscopy_cvat/cvat3.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Usecase 1: Inference with CVAT+MONAI Label Function \n",
    "\n",
    "MONAI Label deployed models can be directly used in the CVAT without a monailabel start_server command, as the deployed functions are running inside the containers. Users can check the status in the deployed function container by logging docker image, e.g, tooltracking container: ```docker container logs nuclio-nuclio-monailabel.endoscopy.tooltracking``` in JupyterLab terminal.\n",
    "\n",
    "In the following steps, users can download same sample data by:\n",
    "\n",
    "```bash\n",
    "mkdir datasets\n",
    "wget \"https://github.com/Project-MONAI/MONAILabel/releases/download/data/endoscopy_frames.zip\" -O datasets/endoscopy_frames.zip\n",
    "unzip datasets/endoscopy_frames.zip -d datasets/endoscopy_frames\n",
    "```\n",
    "\n",
    "Select frames to upload to CVAT.\n",
    "\n",
    "### 3.1 Create task and upload images to CVAT with UI\n",
    "\n",
    "Click the \"+\" icon in the **Tasks** tag, create a name and add a label, then click \"Done\". Select and upload an image. Then click **Submit** to create a task.\n",
    "\n",
    "![cvat4](./figures/monailabel_endoscopy_cvat/cvat4.png)\n",
    "\n",
    "A \"job\" will be shown in the CVAT, click the \"job\" to open the labeling task:\n",
    "\n",
    "![cvat5](./figures/monailabel_endoscopy_cvat/cvat5.png)\n",
    "\n",
    "\n",
    "### 3.2 Inference with MONAI Label model\n",
    "\n",
    "On the labeling task page, click the **AI Tools** on the left panel. Select **Detectors**, choose **tooltracking**\n",
    "in the dropdown scroll. Pick label and finally click **Annotate** button. \n",
    "\n",
    "![cvat6](./figures/monailabel_endoscopy_cvat/cvat6.png)\n",
    "\n",
    "The predicted mask will be shown as polygon in the CVAT, users can edit the mask and click **Save** icon on the top. Once finished annotation, click \"Menu\", change the job state to \"complete\", and \"Finish the job\".  \n",
    "\n",
    "![cvat7](./figures/monailabel_endoscopy_cvat/cvat7.png)\n",
    "\n",
    "Users can export annoated masks using CVAT UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Usecase 2: Active Learning Workflow with MONAI Label + CVAT\n",
    "\n",
    "The advanced feature of the endoscopic application is to actively select and fine-tune models. \n",
    "\n",
    "Below is a step-by-step tutorial on using MONAI Label+CVAT for auto segmentation, active learning and fine-tuning models using surgical tool segmentation bundle. Pre-trained model is provided by the bundle and loaded by MONAI Label server.\n",
    "\n",
    "### 4.1 Start MONAI Label server for Active Learning\n",
    "\n",
    "Connect the MONAI Label server with cvat datastore and URL, set USERNAME and PASSWORD same as login credentials for CVAT.\n",
    "\n",
    "```bash\n",
    "# cd to the folder where have the downloaded apps and data\n",
    "cd /opt/tutorials/monaiLabel\n",
    "\n",
    "export MONAI_LABEL_DATASTORE=cvat\n",
    "export MONAI_LABEL_DATASTORE_URL=http://127.0.0.1:8080\n",
    "# the USERNAME and PASSWARD are the authentications created with command:\n",
    "# \"docker exec -it cvat bash -ic 'python3 ~/manage.py createsuperuser\"\n",
    "export MONAI_LABEL_DATASTORE_USERNAME=<USERNAME>\n",
    "export MONAI_LABEL_DATASTORE_PASSWORD=<PASSWORD>\n",
    "\n",
    "# Specify the path of the downloaded endoscopy app (--app <APPPATH>) and the sample dataset (--studies <DATAPATH>)\n",
    "monailabel start_server \\\n",
    "  --app apps/endoscopy \\\n",
    "  --studies datasets/endoscopy_frames \\\n",
    "  --conf models tooltracking \\\n",
    "  --conf epistemic_enabled true \\\n",
    "  --conf epistemic_top_k 2\n",
    "```\n",
    "\n",
    "Then MONAI Label server will start scoring inference for each unlabeled image using epistemic scoring method. Users can set the number of image to label in CVAT for each iteration, the ```--conf epistemic_top_k 10``` specifies the workload for each active learning loop.\n",
    "\n",
    "MONAI Label server will automatically upload selected images to CVAT.\n",
    "Note: if images have been uploaded or the prior iteration's task is not complete, MONAI Label will prevent uploading more.\n",
    "\n",
    "The automatically created project and tasks can be seen here:\n",
    "![cvat8](./figures/monailabel_endoscopy_cvat/cvat8.png)\n",
    "\n",
    "![cvat9](./figures/monailabel_endoscopy_cvat/cvat9.png)\n",
    "\n",
    "### 4.2 Labeling with CVAT and MONAI Label AI Tool\n",
    "\n",
    "The created tasks will run inference automatically, users can see predicted mask in the job page, edit them and follow same annotating process in Section 3. Once finished, click save, change job state and finish the job.\n",
    "\n",
    "![cvat10](./figures/monailabel_endoscopy_cvat/cvat10.png)\n",
    "\n",
    "### 4.3 Trigger Model Fine-Tuning.\n",
    "\n",
    "Continue to annotate until all created jobs are completed. MONAI Label will monitor the datastore status to check whether all jobs are in \"completed\" status.\n",
    "\n",
    "Once all jobs are completed, MONAI Label server will automatically trigger model fine-tuning and train the model with newly generated labels.\n",
    "\n",
    "![cvat11](./figures/monailabel_endoscopy_cvat/cvat11.png)\n",
    "\n",
    "\n",
    "### 4.4 Scoring and Create Next iteration for CVAT\n",
    "\n",
    "Users will get a fine-tuned model if users provided the validation set.\n",
    "\n",
    "![cvat12](./figures/monailabel_endoscopy_cvat/cvat12.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion\n",
    "\n",
    "This tutorial demonstrates MONAI Label usage with endoscopy app, MONAI Label provides an advanced active learning workflow for endoscopic applications. Best with CVAT and Nuclio deployment, the MONAI Label has the leading functionality of computer assisted intervention tasks. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}