datasets/export/dataset_export.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<td>\n",
    "   <a target=\"_blank\" href=\"https://www.clarifai.com/\" ><img src=\"https://upload.wikimedia.org/wikipedia/commons/b/bc/Clarifai_Logo_FC_Web.png\" width=256/></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<td>\n",
    "<a href=\"https://colab.research.google.com/github/Clarifai/examples/blob/main/datasets/export/dataset_export.ipynb\" target=\"_blank\"><img\n",
    "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Colab\"></a>\n",
    "</td>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Export\n",
    "\n",
    "A dataset is a collection of data examples you can use to annotate and train, test, evaluate your machine learning models. With Clarifai datasets, you can manage the datasets you want to use for visual search, training, and evaluation.\n",
    "\n",
    "This notebook contains\n",
    "- Examples of how to export your datasets from clarifai app using features from `Dataset`. Converting that dataset to other annotation formats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* [Requirements](#requirements)\n",
    "\n",
    "* [Upload Dataset](#upload_dataset)\n",
    "\n",
    "* [Dataset Export](#dataset_export)\n",
    "\n",
    "* [Export Clarifai to Other Formats](#export_annotation)\n",
    "    * [COCO](#coco)\n",
    "    * [YOLO](#yolo)\n",
    "\n",
    "* [Supported Formats](#formats)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements <a id=\"requirements\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U clarifai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "#Replace your PAT\n",
    "os.environ['CLARIFAI_PAT'] = \"PAT\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: Guide to get your [PAT](https://docs.clarifai.com/clarifai-basics/authentication/personal-access-tokens)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### For Colab\n",
    "\n",
    "Note: To use the data from the example repo itself"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/Clarifai/examples.git\n",
    "%cd examples/datasets/upload/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from clarifai.client.dataset import Dataset\n",
    "#replace your \"user_id\", \"app_id\", \"dataset_id\".\n",
    "dataset = Dataset(user_id=\"user_id\", app_id=\"app_id\", dataset_id=\"dataset_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload Dataset  <a id=\"upload_dataset\"></a>\n",
    "\n",
    "- Examples of how to upload your local directory datasets into Clarifai App using `module_dir` feature from `Dataset`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object Detection - VOC - 2012"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing load_module_dataloader for calling the dataloader object in dataset.py in the local data folder\n",
    "from clarifai.datasets.upload.utils import load_module_dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Uploading Dataset: 100%|██████████████████████████████████████████████| 1/1 [00:08<00:00,  8.24s/it]\n"
     ]
    }
   ],
   "source": [
    "voc_dataloader = load_module_dataloader('../upload/image_detection/voc')\n",
    "dataset.upload_dataset(dataloader=voc_dataloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a dataset version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_demo_version = dataset.create_version()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Export    <a id=\"dataset_export\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exporting the Dataset to your local path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_demo_version.export(save_path='output_demo.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extract the zip file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "def extract_zip_file(file_path, extract_path='.'):\n",
    "    with zipfile.ZipFile(file_path, 'r') as zip_ref:\n",
    "        zip_ref.extractall(extract_path)\n",
    "\n",
    "# Usage\n",
    "extract_zip_file('output_demo.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preview of the exported folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all/\n",
      "    annotations/\n",
      "        example-notebook-2009_004382.json\n",
      "        example-notebook-2008_003182.json\n",
      "        example-notebook-2008_000853.json\n",
      "        example-notebook-2008_008526.json\n",
      "        example-notebook-2012_000690.json\n",
      "        example-notebook-2007_000464.json\n",
      "        example-notebook-2011_006412.json\n",
      "        example-notebook-2011_000430.json\n",
      "        example-notebook-2011_001610.json\n",
      "        example-notebook-2009_004315.json\n",
      "    inputs/\n",
      "        example-notebook-2009_004315.png\n",
      "        example-notebook-2011_001610.png\n",
      "        example-notebook-2011_000430.png\n",
      "        example-notebook-2008_008526.png\n",
      "        example-notebook-2009_004382.png\n",
      "        example-notebook-2007_000464.png\n",
      "        example-notebook-2012_000690.png\n",
      "        example-notebook-2008_000853.png\n",
      "        example-notebook-2011_006412.png\n",
      "        example-notebook-2008_003182.png\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "def list_files(startpath):\n",
    "    for root, dirs, files in os.walk(startpath):\n",
    "        level = root.replace(startpath, '').count(os.sep)\n",
    "        indent = ' ' * 4 * (level)\n",
    "        print('{}{}/'.format(indent, os.path.basename(root)))\n",
    "        subindent = ' ' * 4 * (level + 1)\n",
    "        for f in files:\n",
    "            print('{}{}'.format(subindent, f))\n",
    "\n",
    "list_files('./all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Export Clarifai to Other Formats  <a id=\"export_annotation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements <a id=\"requirements\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing clarifai-datautils  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U clarifai-datautils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Refer to the clarifai-datautils repo for more info: https://github.com/Clarifai/clarifai-python-datautils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Clarifai Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjay/virtualenv/v1/lib/python3.9/site-packages/pandas/compat/__init__.py:124: UserWarning: Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.\n",
      "  warnings.warn(msg)\n",
      "/Users/sanjay/virtualenv/v1/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset\n",
      "\tsize=10\n",
      "\tsource_path=None\n",
      "\tannotated_items_count=10\n",
      "\tannotations_count=18\n",
      "subsets\n",
      "\tdefault: # of items=10, # of annotated items=10, # of annotations=18, annotation types=['bbox']\n",
      "infos\n",
      "\tcategories\n",
      "\tlabel: ['bird', 'person', 'dog', 'sofa', 'bottle', 'cat', 'cow', 'horse']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from clarifai_datautils import ImageAnnotations\n",
    "clarifai_dataset = ImageAnnotations.import_from(path='./all',format= 'clarifai')\n",
    "print(clarifai_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export to COCO Format    <a id=\"coco\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**save_images** param will save the images too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "clarifai_dataset.export_to(path='./clarifai_to_coco',format='coco_detection',save_images=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preview of the converted COCO Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clarifai_to_coco/\n",
      "    images/\n",
      "        default/\n",
      "            example-notebook-2009_004315.png\n",
      "            example-notebook-2011_001610.png\n",
      "            example-notebook-2011_000430.png\n",
      "            example-notebook-2008_008526.png\n",
      "            example-notebook-2009_004382.png\n",
      "            example-notebook-2007_000464.png\n",
      "            example-notebook-2012_000690.png\n",
      "            example-notebook-2008_000853.png\n",
      "            example-notebook-2011_006412.png\n",
      "            example-notebook-2008_003182.png\n",
      "    annotations/\n",
      "        instances_default.json\n"
     ]
    }
   ],
   "source": [
    "list_files('./clarifai_to_coco')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export to YOLO Format    <a id=\"yolo\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**save_images** param will save the images too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "clarifai_dataset.export_to(path='./clarifai_to_yolo',format='yolo',save_images=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preview of the converted YOLO Format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clarifai_to_yolo/\n",
      "    obj.data\n",
      "    train.txt\n",
      "    obj.names\n",
      "    obj_train_data/\n",
      "        example-notebook-2009_004315.png\n",
      "        example-notebook-2011_000430.txt\n",
      "        example-notebook-2011_001610.png\n",
      "        example-notebook-2008_008526.txt\n",
      "        example-notebook-2009_004315.txt\n",
      "        example-notebook-2011_000430.png\n",
      "        example-notebook-2011_001610.txt\n",
      "        example-notebook-2008_008526.png\n",
      "        example-notebook-2009_004382.png\n",
      "        example-notebook-2007_000464.png\n",
      "        example-notebook-2012_000690.txt\n",
      "        example-notebook-2011_006412.txt\n",
      "        example-notebook-2008_000853.txt\n",
      "        example-notebook-2008_003182.txt\n",
      "        example-notebook-2009_004382.txt\n",
      "        example-notebook-2007_000464.txt\n",
      "        example-notebook-2012_000690.png\n",
      "        example-notebook-2008_000853.png\n",
      "        example-notebook-2011_006412.png\n",
      "        example-notebook-2008_003182.png\n"
     ]
    }
   ],
   "source": [
    "list_files('./clarifai_to_yolo')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note: Here is the list of [Supported Formats](https://github.com/Clarifai/examples/tree/main/Data_Utils#supported-formats)    <a id=\"formats\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clarifai Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Website**: [https://www.clarifai.com](https://www.clarifai.com/)\n",
    "\n",
    "**Demo**: [https://clarifai.com/demo](https://clarifai.com/demo)\n",
    "\n",
    "**Sign up for a free Account**: [https://clarifai.com/signup](https://clarifai.com/signup)\n",
    "\n",
    "**Developer Guide**: [https://docs.clarifai.com](https://docs.clarifai.com/)\n",
    "\n",
    "**Clarifai Community**: [https://clarifai.com/explore](https://clarifai.com/explore)\n",
    "\n",
    "**Python SDK Docs**: [https://docs.clarifai.com/python-sdk/api-reference](https://docs.clarifai.com/python-sdk/api-reference)\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}