notebooks/benchmarking_a_method.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmarking a method\n",
    "\n",
    "When developing a new method, you might want to see how other methods perform\n",
    "on the dataset you are evaluating. Photoholmes includes the implementation for several\n",
    "methods from the literature, covering a diverse array of approaches to forgery detection\n",
    "to compare to.\n",
    "\n",
    "It also includes a Benchmark object to easily evaluate the performance of a method over\n",
    "a dataset. This ensures a fair and reproducible comparison between methods.\n",
    "This notebook is a short tutorial on how to use Benchmark object, using as an example \n",
    "the included method DQ and a custom method we will define on the Columbia dataset.\n",
    "\n",
    "1. [ Running an included method ](#running-an-included-method)\n",
    "    - [Method selection](#method-selection)\n",
    "    - [Loading the dataset](#loading-a-dataset-for-evaluation)\n",
    "    - [Selecting the metrics](#selecting-the-metrics)\n",
    "    - [Benchmark evaluation](#benchmark-evaluation)\n",
    "    \n",
    "2. [ Running a custom method ](#running-a-custom-method)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running an included method\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Method selection\n",
    "\n",
    "To run an included method, we first import and instantiate the method. You can do this by using the **method factory**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.methods import MethodFactory, MethodRegistry\n",
    "\n",
    "\n",
    "dq, dq_preprocessing = MethodFactory.load(MethodRegistry.DQ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or by **importing** the method **directly**. In this case, it's important to also import the corresponding **preprocessing** pipeline, as it is a required argument for the Dataset. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.methods.dq import DQ, dq_preprocessing\n",
    "\n",
    "dq = DQ()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on about the methods and how to use them, you can check the [methods documentation](\n",
    "    ../SRC/photoholmes/methods/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading a Dataset for evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To continue, we must first download the [Columbia Uncompressed Image Splicing Detection](https://www.ee.columbia.edu/ln/dvmm/downloads/authsplcuncmp/) dataset. Keep in mind this dataset is under a [ research-only use License ](https://www.ee.columbia.edu/ln/dvmm/downloads/authsplcuncmp/dlform.html). You can download the dataset [ here ](https://www.dropbox.com/sh/786qv3yhvc7s9ki/AACbEEzGPrD3_y38bpWHzgdqa?e=1&dl=0).\n",
    "\n",
    "Once downloaded, unzip the files and update the following variable with the path to the dataset folder.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columbia_dataset_path: str = (\n",
    "    \"../data/Benchmark/Columbia Uncompressed Image Splicing Detection\"  # UPDATE WITH THE PATH ON YOUR COMPUTER\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as with the methods, we can load a dataset by **direct import**:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.datasets.columbia import ColumbiaDataset\n",
    "\n",
    "dataset = ColumbiaDataset(\n",
    "    dataset_path=columbia_dataset_path,\n",
    "    load=[\"image\", \"dct_coefficients\"],\n",
    "    preprocessing_pipeline=dq_preprocessing,\n",
    ")\n",
    "print(\"Total images: \", len(dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or using the **factory**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.datasets import DatasetFactory, DatasetRegistry\n",
    "\n",
    "\n",
    "dataset = DatasetFactory.load(\n",
    "    DatasetRegistry.COLUMBIA,\n",
    "    dataset_path=columbia_dataset_path,\n",
    "    load=[\"image\", \"dct_coefficients\"],\n",
    "    preprocessing_pipeline=dq_preprocessing,\n",
    ")\n",
    "print(\"Total images: \", len(dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on about the datasets and how to use them, you can check the [datasets documentation](\n",
    "    ../src/photoholmes/datasets/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting the metrics\n",
    "\n",
    "Lastly, we need to select the metrics to evaluate. We will load the Auroc, IoU and F1 using the MetricFactory which allows us to easily select the metrics we want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.metrics.factory import MetricFactory\n",
    "\n",
    "metrics = MetricFactory.load([\"auroc\", \"f1\", \"iou\"])\n",
    "print(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on about the metrics and how to use them, you can check the [metrics documentation](\n",
    "    ../src/photoholmes/metrics/README.md)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark evaluation\n",
    "\n",
    "Now, we are ready to run the Benchmark. First, we create a Benchmark Object. The constructor allows to tune the following parameters:\n",
    "\n",
    "- **save_method_outputs:** Whether to save the method outputs.\n",
    "- **save_extra_outputs:** Whether to save extra outputs.\n",
    "- **save_metrics:** Whether to save metrics.\n",
    "- **output_folder:** Path to the output folder.\n",
    "- **device:** torch Device for computation.\n",
    "- **use_existing_output:** Whether to use existing saved outputs.\n",
    "- **verbose:** Verbosity level.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.benchmark import Benchmark\n",
    "\n",
    "benchmark = Benchmark(\n",
    "    save_method_outputs=True,\n",
    "    save_extra_outputs=False,\n",
    "    save_metrics=True,\n",
    "    output_folder=\"example_output\",\n",
    "    device=\"cpu\",\n",
    "    use_existing_output=False,\n",
    "    verbose=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to go! The following cell will run the evaluation. It should take around two minutes to continue.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dq_results = benchmark.run(\n",
    "    method=dq,\n",
    "    dataset=dataset,\n",
    "    metrics=metrics,\n",
    ")\n",
    "print(dq_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that a folder example_output has been created. There, the benchmark will create the following folder structure:\n",
    "\n",
    "```terminal\n",
    "example_output/\n",
    "└── dq\n",
    "    └── columbiadataset\n",
    "        ├── metrics\n",
    "        │   └── <timestamp>_tampered_and_pristine\n",
    "        │       ├── heatmap_report.json\n",
    "        │       └── heatmap_state.pt\n",
    "        └── outputs\n",
    "            ├── canong3_02_sub_01\n",
    "            │   └── output.npz\n",
    "            ├── canong3_02_sub_02\n",
    "            ...\n",
    "```\n",
    "\n",
    "Inside the _\\<method\\>_\\/_\\<dataset\\>_ folder, in this case _dq/colmubiadataset_, you will find two folders: _output/_ and _metrics_. Inside the ouputs\n",
    "folder you will find the saved model outputs, so they can be reused and save compute time. There are three types of output:\n",
    "\n",
    "1. _output.npz_: saves the benchmark outputs (heatmap, mask and/or detection)\n",
    "2. _output_extra.npz_: saves any extra output arrays that were included in the benchmark output. This will be included only if the benchmark has the\n",
    "   _save_extra_output=True_.\n",
    "3. _output_extra_json.json_: save any extra output that isn't an array.\n",
    "\n",
    "On the metrics folder, you will find the benchmarking results. Every time you run a benchmark, a folder with the name _\\<timestamp\\>_ _ _\\<dataset_type\\>_ is created (the latter being _tampered only_ or _tampered and pristine_). Inside the folder, you will find metric files for each type of output your method outputs.\n",
    "Photoholmes divides method outputs into three types:\n",
    "\n",
    "1. **heatmap:** a probability map.\n",
    "2. **mask:** a binary mask.\n",
    "3. **detection:** a score for detection.\n",
    "\n",
    "Your method can predict only one output per type. Inside the metrics folder you will find a report for each type your method outputed, in this case\n",
    "only _heatmap_report.json_.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congrats! You have benchmarked DQ on the columbia dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a custom method\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's implemented a basic method to use our benchmark on. We won't do anything fancy, simply predict a random array in the same shape\n",
    "as the image.\n",
    "\n",
    "To do so, we should create a child class from _BaseMethod_ and overide the _\\_\\_init\\_\\__, _predict_ and _benchmark_ to the desired behaviour. In our notation, predict is the class method that performs the prediction of the heatmap (mask, detection or others), and benchmark method returns these outputs in the form of a dictionary with standardized keys. For more details, see the documentation. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from typing import Tuple\n",
    "from photoholmes.methods.base import BaseMethod, BenchmarkOutput\n",
    "import numpy as np\n",
    "from numpy.typing import NDArray\n",
    "import torch\n",
    "from photoholmes.preprocessing.image import ToNumpy\n",
    "\n",
    "from photoholmes.preprocessing.pipeline import PreProcessingPipeline\n",
    "\n",
    "\n",
    "class RandomMethod(BaseMethod):\n",
    "\n",
    "    def __init__(self, threshold=0.5):\n",
    "        super().__init__()\n",
    "        self.threshold = threshold\n",
    "\n",
    "    def predict(self, image: NDArray) -> Tuple[NDArray, NDArray, float]:\n",
    "        heatmap = np.random.random(size=image.shape[:2])\n",
    "        detection = random.random()\n",
    "        mask = heatmap>self.threshold\n",
    "        return heatmap, mask, detection\n",
    "\n",
    "    def benchmark(self, image: NDArray) -> BenchmarkOutput:\n",
    "        heatmap, mask, detection = self.predict(image)\n",
    "        return {\n",
    "            \"heatmap\": torch.from_numpy(heatmap),\n",
    "            \"mask\": torch.from_numpy(mask),\n",
    "            \"detection\": torch.tensor([detection]),\n",
    "        }\n",
    "\n",
    "\n",
    "method = RandomMethod()\n",
    "random_method_preprocessing = PreProcessingPipeline(\n",
    "    [ToNumpy()],\n",
    "    inputs=[\"image\"],\n",
    "    outputs_keys=[\"image\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test it out in an image of Paul McCartney drinking \"fernet\". The image is downloaded from [Taringa](https://media.taringa.net/knn/fit:550/Z3M6Ly9rbjMvdGFyaW5nYS9ELzIvNy9GLzQvRi9veWVjb21vdmFhLzQzNy5qcGc)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! mkdir -p data\n",
    "! curl https://media.taringa.net/knn/fit:550/Z3M6Ly9rbjMvdGFyaW5nYS9ELzIvNy9GLzQvRi9veWVjb21vdmFhLzQzNy5qcGc -o data/paul.webp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.utils.image import read_image, plot\n",
    "\n",
    "img = read_image(\"data/paul.webp\")\n",
    "plot(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.utils.image import plot_multiple\n",
    "\n",
    "input_preprocessed = random_method_preprocessing(image=img)\n",
    "heatmap, mask, score = method.predict(**input_preprocessed)\n",
    "print(\"Detection score:\", score)\n",
    "plot_multiple([heatmap, mask], titles=[\"Heatmap\", \"Mask\"], title=\"Random Method\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our method clearly isn't good, but it serves as a good enough example to test the benchmark functionality.\n",
    "\n",
    "Let's load our datasets and metrics again and run the benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.datasets.factory import DatasetFactory, DatasetRegistry\n",
    "from photoholmes.metrics.factory import MetricFactory, MetricRegistry\n",
    "\n",
    "\n",
    "columbia_dataset_path: str = (\n",
    "    \"data/Columbia Uncompressed Image Splicing Detection\"  # UPDATE WITH THE PATH ON YOUR COMPUTER\n",
    ")\n",
    "\n",
    "dataset = DatasetFactory.load(\n",
    "    dataset_name=DatasetRegistry.COLUMBIA,\n",
    "    dataset_path=columbia_dataset_path,\n",
    "    load=[\"image\", \"dct_coefficients\"],\n",
    "    preprocessing_pipeline=random_method_preprocessing,\n",
    ")\n",
    "print(\"Total images: \", len(dataset))\n",
    "\n",
    "metrics = MetricFactory.load([MetricRegistry.AUROC, MetricRegistry.F1, MetricRegistry.IoU])\n",
    "print(metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from photoholmes.benchmark import Benchmark\n",
    "\n",
    "benchmark = Benchmark(output_folder=\"example_output\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_method_results = benchmark.run(method=method, dataset=dataset, metrics=metrics)\n",
    "print(random_method_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you check the output folder, you will notice a folder _randommethod/columbiadataset_ with the evaluation results."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}