monailabel/monailabel_monaibundle_3dslicer_multiorgan_seg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) MONAI Consortium  \n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");  \n",
    "you may not use this file except in compliance with the License.  \n",
    "You may obtain a copy of the License at  \n",
    "&nbsp;&nbsp;&nbsp;&nbsp;http://www.apache.org/licenses/LICENSE-2.0  \n",
    "Unless required by applicable law or agreed to in writing, software  \n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,  \n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  \n",
    "See the License for the specific language governing permissions and  \n",
    "limitations under the License. \n",
    "\n",
    "# MONAI Label MONAIBUNDLE App - Multi-Organ Segmentation\n",
    "\n",
    "***The Robust Workflow with Customized Models in MONAI Label***\n",
    "\n",
    "In this notebook, we provide a monaibundle app usecase example with multi-organ segmentation. Using 13 abdominal organs segmentation with Swin UNETR network (bundle) as the demonstration, 3D Slicer as the client viewer, we show how MONAI Label workflow serves as interactive AI-Assisted tool for labeling multiple abdominal organs with bundle config files. \n",
    "\n",
    "\n",
    "**Multi-Organ Segmentation with Swin UNETR bundle:**\n",
    "\n",
    "This tutorial uses a Swin UNETR bundle for the task of multi-organ segmentation task. The architecture of Swin UNETR is demonstrated as below\n",
    "![workflow](./figures/monailabel_monaibundle_3dslicer/swinunetr.png)\n",
    "\n",
    "**Target:** 13 abdominal organs including\n",
    "- Spleen\n",
    "- Right Kidney\n",
    "- Left Kidney\n",
    "- Gallbladder\n",
    "- Esophagus\n",
    "- Liver\n",
    "- Stomach\n",
    "- Aorta\n",
    "- IVC\n",
    "- Portal and Splenic Veins\n",
    "- Pancreas\n",
    "- Right adrenal gland\n",
    "- Left adrenal gland\n",
    "\n",
    "**Modality:** CT\n",
    "\n",
    "The multi-organ segmentation with MONAI Label shown in 3D Slicer\n",
    "\n",
    "![workflow](./figures/monailabel_monaibundle_3dslicer/multi_organ_seg_slicer.png)\n",
    "\n",
    "## 1. Prepare MONAI Label\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup environment\n",
    "\n",
    "### Prerequisites\n",
    "- **Install MONAI Label** weekly preview release: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install monailabel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Download monaibundle app\n",
    "Get and copy the monaibundle app using monailabel API "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel apps --download --name monaibundle --output apps"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download sample data\n",
    "The multi-organ segmentation task takes CT images as input. The bundle model Swin UNETR pre-trained weights are trained with Beyond The Cranial Vault (BTCV). In this tutorial, some sample CT images from MSD are used (for the purpose of app demonstration on Slicer 3D). Download MSD Task09 dataset as the sample dataset using monailabel API. \n",
    "\n",
    "**Note**: In this tutorial, we use some abdomen CT data for MONAI Label demostrations. If users want to use the BTCV dataset with ground truth annotations, please refer to [SwinUNETR 3D Segmentation](../3d_segmentation/swin_unetr_btcv_segmentation_3d.ipynb) for the dataset details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel datasets --download --name Task09_Spleen --output datasets"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Starting MONAI Label Server\n",
    "\n",
    "Specify the bundle name in `--conf models <BUNDLENAME>` argument. For example: swin_unetr_btcv_segmentation\n",
    "\n",
    "**Note:** If you are running a MONAI Label server in this notebook and plan to close the notebook, ensure that you terminate the kernel first. Failing to do so might result in an issue where the port remains occupied from the previous session when you try to restart the server later. To prevent this, always terminate the kernel before closing the notebook.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!monailabel start_server --app apps/monaibundle --studies datasets/Task09_Spleen/imagesTr --conf models swin_unetr_btcv_segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 (Optional) File Structure for monaibundle app\n",
    "\n",
    "To double check and confirm the correct monaibundle app and bundle files structure.\n",
    "\n",
    "The directory tree structure should be:\n",
    "\n",
    "```\n",
    "└── monaibundle\n",
    "    ├── __init__.py\n",
    "    ├── lib\n",
    "    │   ├── activelearning\n",
    "    │   │   └── __init__.py\n",
    "    │   ├── infers\n",
    "    │   │   └── __init__.py\n",
    "    │   ├── __init__.py\n",
    "    │   └── trainers\n",
    "    │       └── __init__.py\n",
    "    ├── main.py\n",
    "    ├── model\n",
    "    │   └── swin_unetr_btcv_segmentation_v0.3.9\n",
    "    │       ├── configs\n",
    "    │       │   ├── evaluate.json\n",
    "    │       │   ├── inference.json\n",
    "    │       │   ├── logging.conf\n",
    "    │       │   ├── metadata.json\n",
    "    │       │   ├── multi_gpu_train.json\n",
    "    │       │   └── train.json\n",
    "    │       ├── docs\n",
    "    │       │   ├── data_license.txt\n",
    "    │       │   └── README.md\n",
    "    │       ├── eval\n",
    "    │       ├── LICENSE\n",
    "    │       └── models\n",
    "    │           └── model.pt\n",
    "    ├── README.md\n",
    "    └── requirements.txt\n",
    "```\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Auto Segmentation with 3D Slicer\n",
    "\n",
    "Below is a step-by-step tutorial on using 3D Slicer for auto multi-organ segmentation with Swin UNETR bundle. Pre-trained model is provided by the bundle and loaded by MONAI Label server.\n",
    "\n",
    "\n",
    "**Start 3D Slicer and MONAI Label Plugin**: the tutorial starts with monaibundle app in the 3D Slicer, please refer to the [\"Hello, World!\" tutorial](monailabel_HelloWorld_radiology_3dslicer.ipynb) for detailed instruction on installing 3D Slicer and MONAI Label plugin in Section 2.\n",
    "\n",
    "\n",
    "\n",
    "### 2.1 Select Model and Fetch Data\n",
    "\n",
    "- Select the bundle model loaded with MONAI Label server, \"**swin_unetr_btcv_segmentation**\" in this tutorial. **Note:** the model name can be different based on the downloaded bundle and can be changed and specified in the file system.\n",
    "\n",
    "click **next sample** to load image data from datastore.\n",
    "\n",
    "![slicer1](./figures/monailabel_monaibundle_3dslicer/swin_slicer1.png)\n",
    "\n",
    "\n",
    "### 2.2 Run Auto Segmentation\n",
    "\n",
    "- MONAI Label has loaded the pre-trained weights within the bundle for multi-organ segmentation task, click **run** to do inference on the currently loaded data. \n",
    "\n",
    "- Note: the pre-trained models are in the bundle's \"**models**\" folder. For instance, in this usecase, a downloaded pre-trained model is saved at \"**apps/monaibundle/model/{BUNDLENAME}/models/model.pt**\".\n",
    "\n",
    "- Note: Swin UNETR is a large model, by default it will take ~10G memory for inference, modify the bundle config file if out of memory.\n",
    "\n",
    "- Users can monitor the logs in the MONAI Label server terminal. \n",
    "- Users can move the axis and slices to view the correct location of segmentation.\n",
    "\n",
    "![slicer1](./figures/monailabel_monaibundle_3dslicer/swin_slicer2.png)\n",
    "\n",
    "\n",
    "### 2.3 Edit Annotations and Submit Label\n",
    "\n",
    "- Users can edit the inference labels manually using Segment Editor, the tools for adjusting annotation. Once done annotation, click **Submit Label** to save the ground truth label to the file system.\n",
    "\n",
    "- Note: the final annotations will be saved to \"**labels/final**\" folder in the study dataset, for example, in this usecase, the ground truth label will be saved to \"**datasets/Task09_Spleen/imagesTr/labels/final**\"\n",
    "\n",
    "#### Corrections with Segment Editor:\n",
    "\n",
    "![slicer1](./figures/monailabel_monaibundle_3dslicer/swin_slicer3.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Active Learning with 3D Slicer\n",
    "\n",
    "Active learning and interactive fine-tuning models have highlighted features in MONAI Label, users can train their models anytime when new annotated labels are saved. Click **Train** button, MONAI Label server will fetch saved final ground truth label and fine-tune the prior model. \n",
    "\n",
    "Users can monitor the logs in the MONAI Label server terminal\n",
    "\n",
    "![slicer1](./figures/monailabel_monaibundle_3dslicer/swin_slicer4.png)\n",
    "\n",
    "Same to the Radiology app, users can do auto segmentation using the latest fine-tuned model. \n",
    "The active learning process will select unlabeled batch data.\n",
    "\n",
    "The next image is selected, already trained images will be marked as labeled data, then won't be selected for the next labeling batch. Active learning strategies such as \"first/random\" will be used for selecting which unlabeled data to fetch. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Repeat Until All Data are Annotated and Trained \n",
    "\n",
    "Repeated fetch data and active learning process to section 2.2, until all unlabeled data are annotated and trained. \n",
    "\n",
    "For each training loop, the new best metric model will be saved in \"**model/model.pt**\", in this usecase, \"\"**apps/monaibundle/{BUNDLENAME}/model/model.pt**\" is saved."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion\n",
    "\n",
    "This tutorial demonstrates MONAI Label usage with monaibundle app, the multi-organ segmentation with Swin UNETR bundle is introduced. Active learning process, auto segmentation, segmentation editor, submit labels, and save models are included with the bundle. The monaibundle app highlights the robust model deployment for MONAI Label."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}