notebooks/generate_saliencybox_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Idea\n",
    "Based on the saliency maps, bounding boxes can be automatically generated. The\n",
    "bounding box generation and selection process is explained in [the original paper]\n",
    "(https://arxiv.org/abs/2204.11535).\n",
    "\n",
    "## File structure\n",
    "\n",
    "First, download the scene-specific parameters from [Google Drive](https://drive.google.com/drive/folders/1jWjJDcpYJ6cNECV1USC0dg0xRwMTSRWb?usp=sharing).\n",
    "\n",
    "In your PVDN dataset (day cycle) directory, the downloaded parameter config files have to be stored like this:\n",
    "```\n",
    "/path/to/PVDN/day/<split>/labels/kpbms_params\n",
    "├── S*****.json\n",
    "├── S*****.json\n",
    "├── ....\n",
    "├── S*****.json\n",
    "```\n",
    "\n",
    "For example, for the **val** split, this would look like this:\n",
    "```\n",
    "/path/to/PVDN/day/val/labels/kpbms_params\n",
    "├── S00071.json\n",
    "├── S00092.json\n",
    "├── S00100.json\n",
    "├── S00101.json\n",
    "├── S00121.json\n",
    "├── S00123.json\n",
    "├── S00126.json\n",
    "├── S00132.json\n",
    "├── S00135.json\n",
    "├── S00164.json\n",
    "├── S00168.json\n",
    "├── S00192.json\n",
    "├── S00195.json\n",
    "├── S00260.json\n",
    "├── S00284.json\n",
    "├── S00294.json\n",
    "├── S00309.json\n",
    "├── S00355.json\n",
    "├── S00370.json\n",
    "└── S00372.json\n",
    "```\n",
    "\n",
    "A scene config file looks like this (example: S00071):\n",
    "```\n",
    "{\n",
    "    \"kernel_size\": 300,\n",
    "    \"lower_direct\": 0.7000000000000001,\n",
    "    \"lower_indirect\": 0.9,\n",
    "    \"n\": 6,\n",
    "    \"selem_size\": 13,\n",
    "    \"sigma\": 0,\n",
    "    \"upper_direct\": 1.0,\n",
    "    \"upper_indirect\": 1.0\n",
    "}\n",
    "```\n",
    "\n",
    "## Generate the dataset\n",
    "The following code will generate the dataset. For each image, a `.json` file\n",
    "containing the bounding box annotations as well as the labels will be stored. Note\n",
    "that as soon as a bounding box contains at least one direct keypoint annotation, it\n",
    "is labeled as direct.\n",
    "\n",
    "Regarding the file structure of the annotations, in your __labels/__ directory of\n",
    "your dataset a new directory __kpbms_boxes/__ will be generated. Within this\n",
    "directory, all image specific annotation files are stored. An annotation file looks\n",
    "something like this:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"bounding_boxes\": [\n",
    "        [567, 537, 573, 541],\n",
    "        [592, 537, 599, 542],\n",
    "        [565, 554, 573, 560],\n",
    "        [589, 554, 599, 561],\n",
    "        [695, 544, 727, 555]],\n",
    "    \"labels\": [1, 1, 2, 2, 2]\n",
    "}\n",
    "```\n",
    "\n",
    "### Run the code\n",
    "\n",
    "**Note:** Make sure to adjust the dataset paths in the second cell accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install pip --upgrade\n",
    "!pip install -e ..  # install our package\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "\n",
    "from kpsaliency.datasets import SaliencyBoxDataset\n",
    "from kpsaliency.generators import KPBMSBoxGenerator, KPBMSGenerator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "## setup all paths\n",
    "\n",
    "dataset_dir = \"/path/to/PVDN/day\"   # adapt this to your\n",
    "                                    # specific path\n",
    "splits = (\"train\", \"val\", \"test\")\n",
    "\n",
    "assert os.path.exists(dataset_dir)\n",
    "for split in splits:\n",
    "    assert os.path.exists(os.path.join(dataset_dir, split))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "is_executing": true,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "## run the dataset generation process\n",
    "\n",
    "for split in splits:\n",
    "    # setup and check paths\n",
    "    split_dir = os.path.join(dataset_dir, split)\n",
    "    params_dir = os.path.join(split_dir, \"labels/kpbms_params\")\n",
    "    if not os.path.isdir(params_dir):\n",
    "        raise NotADirectoryError(f\"{params_dir} not found. Please check that you set\"\n",
    "                                 f\" up the parameter directories properly.\")\n",
    "\n",
    "    # setup dataset\n",
    "    dataset = SaliencyBoxDataset(split_dir)\n",
    "\n",
    "    # setup scene-specific generators\n",
    "    generators = {\n",
    "        s.split(\".\")[0]: KPBMSBoxGenerator(KPBMSGenerator.from_json(os.path.join(params_dir, s)))\n",
    "        for s in os.listdir(params_dir)\n",
    "    }\n",
    "\n",
    "    # run generation\n",
    "    print(split)\n",
    "    with np.errstate(invalid='ignore'):\n",
    "        dataset.generate_dataset(box_generator=generators, n_workers=4, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also generate a dataset just from a single parameter description file. In this case, the same parameter setting is used for each scene:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    # adjust those variables to your specific case\n",
    "    split = \"train\"\n",
    "    param_path = \"/path/to/params.json\"\n",
    "\n",
    "    # setup dataset\n",
    "    split_dir = os.path.join(dataset_dir, split)\n",
    "    dataset = SaliencyBoxDataset(split_dir)\n",
    "\n",
    "    # setup generator\n",
    "    generator = KPBMSBoxGenerator(KPBMSGenerator.from_json(param_path))\n",
    "\n",
    "    # run generation\n",
    "    print(split)\n",
    "    with np.errstate(invalid='ignore'):\n",
    "        dataset.generate_dataset(box_generator=generator, n_workers=8, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the quality of the generated dataset\n",
    "\n",
    "As soon as you generated the dataset you can assess its quality based on the metrics described in the paper. A script version for evaluating your dataset quality is also provided in [scripts/evaluate_kpbms_groundtruth.py](../scripts/evaluate_kpbms_groundtruth.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import argparse\n",
    "import json\n",
    "import os\n",
    "\n",
    "from kpsaliency.datasets import SaliencyBoxDataset\n",
    "from kpsaliency.metrics.bboxes import DatasetEvaluator\n",
    "\n",
    "\n",
    "for split in (\"train\", \"val\", \"test\"):\n",
    "    print(\"Split:\", split)\n",
    "    split_dir = os.path.join(dataset_dir, split)\n",
    "    box_dir = os.path.join(\n",
    "        split_dir, \"labels/kpbms_boxes\"\n",
    "    )\n",
    "    dataset = SaliencyBoxDataset(path=split_dir, load_images=False,\n",
    "                                bbox_path=box_dir)\n",
    "    evaluator = DatasetEvaluator()\n",
    "    prec, rec, fsc, kp_quality, kp_quality_std, box_quality, box_quality_std, combined = evaluator.evaluate_dataset(dataset, verbose=True)\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "91fe56c5e2124e00678905795c10e31eb68b12b8b365f9643410209af76e629f"
  },
  "kernelspec": {
   "display_name": "Python 3.8.10 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}