convert.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Converting YOLOv8 to OpenVINO Optimized Model"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the inital YOLO model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Need specific version of ultralytics to use certain functions\n",
    "!pip install -q 'ultralytics==8.0.43'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING ⚠️ Ultralytics settings reset to defaults. This is normal and may be due to a recent ultralytics package update, but may have overwritten previous settings. \n",
      "View and update settings with 'yolo settings' or at '/home/qed42/.config/Ultralytics/settings.yaml'\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "from ultralytics import YOLO\n",
    "\n",
    "models_dir = Path('./models')\n",
    "models_dir.mkdir(exist_ok=True)\n",
    "\n",
    "DET_MODEL_NAME = \"yolov8n\"\n",
    "\n",
    "det_model = YOLO(models_dir / f'{DET_MODEL_NAME}.pt')\n",
    "label_map = det_model.model.names\n",
    "\n",
    "import json\n",
    "with open(models_dir / f'{DET_MODEL_NAME}_labels.json', 'w') as f:\n",
    "    json.dump(label_map, f)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert to OpenVINO IR\n",
    "\n",
    "The term \"OpenVINO IR model\" refers to the Intermediate Representation (IR) model that OpenVINO uses. An IR is a model format that has been converted from the original training framework (like TensorFlow, PyTorch, ONNX, etc.) into a format that OpenVINO can efficiently work with.\n",
    "\n",
    "IR is a pair of files:\n",
    "\n",
    "- .xml: Describes the network topology.\n",
    "- .bin: Contains the weights and biases from the trained model.\n",
    "\n",
    "By converting a trained model into an IR model, it can then be optimized by the Model Optimizer, a component of OpenVINO, for better execution on various Intel hardware (CPU, integrated GPU, VPU, FPGA, etc.). This enables high performance inference and makes OpenVINO a powerful toolkit for deploying AI models across a variety of Intel platforms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "det_model_path = models_dir / f\"{DET_MODEL_NAME}_openvino_model/{DET_MODEL_NAME}.xml\"\n",
    "if not det_model_path.exists():\n",
    "    det_model.export(format=\"openvino\", dynamic=True, half=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimize model using NNCF Post-training Quantization API\n",
    "\n",
    "[NNCF](https://github.com/openvinotoolkit/nncf) provides a suite of advanced algorithms for Neural Networks inference optimization in OpenVINO with minimal accuracy drop.\n",
    "We will use 8-bit quantization in post-training mode (without the fine-tuning pipeline) to optimize YOLOv8.\n",
    "\n",
    "The optimization process contains the following steps:\n",
    "\n",
    "1. Create a Dataset for quantization.\n",
    "2. Run `nncf.quantize` for getting an optimized model.\n",
    "3. Serialize OpenVINO IR model, using the `openvino.runtime.serialize` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino\n",
      "'datasets/val2017.zip' already exists.\n",
      "'datasets/coco2017labels-segments.zip' already exists.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b96d06f0925b4f42959aaae0863cc4c7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "datasets/coco.yaml:   0%|          | 0.00/1.25k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mval: \u001b[0mScanning datasets/coco/labels/val2017.cache... 4952 images, 48 backgrounds, 0 corrupt: 100%|██████████| 5000/5000 [00:00<?, ?it/s]\n"
     ]
    }
   ],
   "source": [
    "# Creating the dataset for quantization\n",
    "\n",
    "DATA_URL = \"http://images.cocodataset.org/zips/val2017.zip\"\n",
    "LABELS_URL = \"https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels-segments.zip\"\n",
    "CFG_URL = \"https://raw.githubusercontent.com/ultralytics/ultralytics/main/ultralytics/datasets/coco.yaml\"\n",
    "\n",
    "OUT_DIR = Path('./datasets')\n",
    "\n",
    "DATA_PATH = OUT_DIR / \"val2017.zip\"\n",
    "LABELS_PATH = OUT_DIR / \"coco2017labels-segments.zip\"\n",
    "CFG_PATH = OUT_DIR / \"coco.yaml\"\n",
    "\n",
    "import nncf\n",
    "from utils import download_file\n",
    "from ultralytics.yolo.data.utils import check_det_dataset\n",
    "from zipfile import ZipFile\n",
    "from ultralytics.yolo.cfg import get_cfg\n",
    "from ultralytics.yolo.utils import DEFAULT_CFG\n",
    "from ultralytics.yolo.utils import ops\n",
    "from typing import Dict\n",
    "\n",
    "download_file(DATA_URL, DATA_PATH.name, DATA_PATH.parent) # type: ignore\n",
    "download_file(LABELS_URL, LABELS_PATH.name, LABELS_PATH.parent) # type: ignore\n",
    "download_file(CFG_URL, CFG_PATH.name, CFG_PATH.parent) # type: ignore\n",
    "\n",
    "if not (OUT_DIR / \"coco/labels\").exists():\n",
    "    with ZipFile(LABELS_PATH , \"r\") as zip_ref:\n",
    "        zip_ref.extractall(OUT_DIR)\n",
    "    with ZipFile(DATA_PATH , \"r\") as zip_ref:\n",
    "        zip_ref.extractall(OUT_DIR / 'coco/images')\n",
    "\n",
    "args = get_cfg(cfg=DEFAULT_CFG)\n",
    "args.data = str(CFG_PATH)\n",
    "\n",
    "det_validator = det_model.ValidatorClass(args=args)\n",
    "\n",
    "det_validator.data = check_det_dataset(args.data)\n",
    "det_data_loader = det_validator.get_dataloader(\"datasets/coco\", 1)\n",
    "\n",
    "det_validator.is_coco = True\n",
    "det_validator.class_map = ops.coco80_to_coco91_class()\n",
    "det_validator.names = det_model.model.names # type: ignore\n",
    "det_validator.metrics.names = det_validator.names\n",
    "det_validator.nc = det_model.model.model[-1].nc # type: ignore\n",
    "\n",
    "def transform_fn(data_item:Dict):\n",
    "    \"\"\"\n",
    "    Quantization transform function. Extracts and preprocess input data from dataloader item for quantization.\n",
    "    Parameters:\n",
    "       data_item: Dict with data item produced by DataLoader during iteration\n",
    "    Returns:\n",
    "        input_tensor: Input data for quantization\n",
    "    \"\"\"\n",
    "    input_tensor = det_validator.preprocess(data_item)['img'].numpy()\n",
    "    return input_tensor\n",
    "\n",
    "\n",
    "quantization_dataset = nncf.Dataset(det_data_loader, transform_fn)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nncf.quantize` function provides an interface for model quantization. It requires an instance of the OpenVINO Model and quantization dataset. \n",
    "Optionally, some additional parameters for the configuration quantization process (number of samples for quantization, preset, ignored scope, etc.) can be provided. YOLOv8 model contains non-ReLU activation functions, which require asymmetric quantization of activations. To achieve a better result, we will use a `mixed` quantization preset. It provides symmetric quantization of weights and asymmetric quantization of activations. For more accurate results, we should keep the operation in the postprocessing subgraph in floating point precision, using the `ignored_scope` parameter.\n",
    "\n",
    ">**Note**: Model post-training quantization is time-consuming process. Be patient, it can take several minutes depending on your hardware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:12 ignored nodes was found by name in the NNCFGraph\n",
      "INFO:nncf:9 ignored nodes was found by types in the NNCFGraph\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 128 /model.22/Sigmoid\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 156 /model.22/dfl/conv/Conv\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 178 /model.22/Sub\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 179 /model.22/Add_10\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 205 /model.22/Div_1\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 193 /model.22/Sub_1\n",
      "INFO:nncf:Not adding activation input quantizer for operation: 218 /model.22/Mul_5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Statistics collection: 100%|██████████| 300/300 [01:06<00:00,  4.50it/s]\n",
      "Biases correction: 100%|██████████| 63/63 [00:04<00:00, 13.57it/s]\n"
     ]
    }
   ],
   "source": [
    "ignored_scope = nncf.IgnoredScope(\n",
    "    types=[\"Multiply\", \"Subtract\", \"Sigmoid\"],  # ignore operations\n",
    "    names=[\n",
    "        \"/model.22/dfl/conv/Conv\",           # in the post-processing subgraph\n",
    "        \"/model.22/Add\",\n",
    "        \"/model.22/Add_1\",\n",
    "        \"/model.22/Add_2\",\n",
    "        \"/model.22/Add_3\",\n",
    "        \"/model.22/Add_4\",   \n",
    "        \"/model.22/Add_5\",\n",
    "        \"/model.22/Add_6\",\n",
    "        \"/model.22/Add_7\",\n",
    "        \"/model.22/Add_8\",\n",
    "        \"/model.22/Add_9\",\n",
    "        \"/model.22/Add_10\"\n",
    "    ]\n",
    ")\n",
    "\n",
    "from openvino.runtime import Core\n",
    "\n",
    "core = Core()\n",
    "det_ov_model = core.read_model(det_model_path)\n",
    "\n",
    "# Detection model\n",
    "quantized_det_model = nncf.quantize(\n",
    "    det_ov_model,\n",
    "    quantization_dataset,\n",
    "    preset=nncf.QuantizationPreset.MIXED,\n",
    "    ignored_scope=ignored_scope\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Quantized detection model will be saved to models/yolov8n_openvino_int8_model/yolov8n.xml\n"
     ]
    }
   ],
   "source": [
    "from openvino.runtime import serialize\n",
    "\n",
    "# saveing model\n",
    "int8_model_det_path = models_dir / f'{DET_MODEL_NAME}_openvino_int8_model/{DET_MODEL_NAME}.xml'\n",
    "print(f\"Quantized detection model will be saved to {int8_model_det_path}\")\n",
    "serialize(quantized_det_model, str(int8_model_det_path))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare Performance of the Original and Quantized Models\n",
    "Finally, use the OpenVINO [Benchmark Tool](https://docs.openvino.ai/2023.0/openvino_inference_engine_tools_benchmark_tool_README.html) to measure the inference performance of the `FP32` and `INT8` models.\n",
    "\n",
    "> **Note**: For more accurate performance, it is recommended to run `benchmark_app` in a terminal/command prompt after closing other applications. Run `benchmark_app -m <model_path> -d CPU -shape \"<input_shape>\"` to benchmark async inference on CPU on specific input data shape for one minute. Change `CPU` to `GPU` to benchmark on GPU. Run `benchmark_app --help` to see an overview of all command-line options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"CPU\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] CPU\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 25.05 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : f32 / [...] / [?,3,?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [?,84,?]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'images': [1,3,640,640]\n",
      "[ INFO ] Reshape model took 19.90 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : u8 / [N,C,H,W] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [1,84,8400]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 218.22 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: torch_jit\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4\n",
      "[ INFO ]   NUM_STREAMS: 4\n",
      "[ INFO ]   AFFINITY: Affinity.CORE\n",
      "[ INFO ]   INFERENCE_NUM_THREADS: 8\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]   ENABLE_CPU_PINNING: True\n",
      "[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'images' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 60000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 86.37 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            1096 iterations\n",
      "[ INFO ] Duration:         60199.14 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        210.48 ms\n",
      "[ INFO ]    Average:       219.39 ms\n",
      "[ INFO ]    Min:           132.87 ms\n",
      "[ INFO ]    Max:           479.32 ms\n",
      "[ INFO ] Throughput:   18.21 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference FP32 model (OpenVINO IR)\n",
    "!benchmark_app -m $det_model_path -d $device -api async -shape \"[1,3,640,640]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] CPU\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 36.88 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : f32 / [...] / [?,3,?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [?,84,?]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'images': [1,3,640,640]\n",
      "[ INFO ] Reshape model took 25.15 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : u8 / [N,C,H,W] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [1,84,8400]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 471.62 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: torch_jit\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4\n",
      "[ INFO ]   NUM_STREAMS: 4\n",
      "[ INFO ]   AFFINITY: Affinity.CORE\n",
      "[ INFO ]   INFERENCE_NUM_THREADS: 8\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]   ENABLE_CPU_PINNING: True\n",
      "[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'images' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 15000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 59.95 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            444 iterations\n",
      "[ INFO ] Duration:         15142.26 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        132.24 ms\n",
      "[ INFO ]    Average:       135.97 ms\n",
      "[ INFO ]    Min:           94.65 ms\n",
      "[ INFO ]    Max:           214.89 ms\n",
      "[ INFO ] Throughput:   29.32 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "!benchmark_app -m $int8_model_det_path -d $device -api async -shape \"[1,3,640,640]\" -t 15"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Integrating preprocessing to model\n",
    "\n",
    "Preprocessing API enables making preprocessing a part of the model reducing application code and dependency on additional image processing libraries. \n",
    "The main advantage of Preprocessing API is that preprocessing steps will be integrated into the execution graph and will be performed on a selected device (CPU/GPU etc.) rather than always being executed on CPU as part of an application. This will improve selected device utilization.\n",
    "\n",
    "For more information, refer to the overview of [Preprocessing API](https://docs.openvino.ai/2023.0/openvino_docs_OV_Runtime_UG_Preprocessing_Overview.html).\n",
    "\n",
    "For example, we can integrate converting input data layout and normalization defined in `image_to_tensor` function.\n",
    "\n",
    "The integration process consists of the following steps:\n",
    "1. Initialize a PrePostProcessing object.\n",
    "2. Define the input data format.\n",
    "3. Describe preprocessing steps.\n",
    "4. Integrating Steps into a Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input \"images\":\n",
      "    User's input tensor: [1,640,640,3], [N,H,W,C], u8\n",
      "    Model's expected tensor: [1,640,640,3], [N,H,W,C], u8\n",
      "    Pre-processing steps (3):\n",
      "      convert type (f32): ([1,640,640,3], [N,H,W,C], u8) -> ([1,640,640,3], [N,H,W,C], f32)\n",
      "      convert layout [N,C,H,W]: ([1,640,640,3], [N,H,W,C], f32) -> ([1,3,640,640], [N,C,H,W], f32)\n",
      "      scale (255,255,255): ([1,3,640,640], [N,C,H,W], f32) -> ([1,3,640,640], [N,C,H,W], f32)\n",
      "    Implicit pre-processing steps (2):\n",
      "      convert type (u8): ([1,3,640,640], [N,C,H,W], f32) -> ([1,3,640,640], [N,C,H,W], u8)\n",
      "      convert layout [N,H,W,C]: ([1,3,640,640], [N,C,H,W], u8) -> ([1,640,640,3], [N,H,W,C], u8)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from openvino.preprocess import PrePostProcessor\n",
    "from openvino.runtime import Type, Layout\n",
    "\n",
    "ppp = PrePostProcessor(quantized_det_model)\n",
    "ppp.input(0).tensor().set_shape([1, 640, 640, 3]).set_element_type(Type.u8).set_layout(Layout('NHWC'))\n",
    "ppp.input(0).preprocess().convert_element_type(Type.f32).convert_layout(Layout('NCHW')).scale([255., 255., 255.])\n",
    "print(ppp)\n",
    "\n",
    "quantized_model_with_preprocess = ppp.build()\n",
    "quantized_model_with_preprocess_path = str(int8_model_det_path.with_name(f\"{DET_MODEL_NAME}_with_preprocess.xml\"))\n",
    "serialize(quantized_model_with_preprocess, quantized_model_with_preprocess_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Re run the benchmark tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] CPU\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 36.21 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : f32 / [...] / [?,3,?,?]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [?,84,?]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'images': [1,3,640,640]\n",
      "[ INFO ] Reshape model took 25.53 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : u8 / [N,C,H,W] / [1,3,640,640]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [1,84,8400]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 447.54 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: torch_jit\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4\n",
      "[ INFO ]   NUM_STREAMS: 4\n",
      "[ INFO ]   AFFINITY: Affinity.CORE\n",
      "[ INFO ]   INFERENCE_NUM_THREADS: 8\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]   ENABLE_CPU_PINNING: True\n",
      "[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'images' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 15000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 48.76 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            476 iterations\n",
      "[ INFO ] Duration:         15181.16 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        125.62 ms\n",
      "[ INFO ]    Average:       127.30 ms\n",
      "[ INFO ]    Min:           96.14 ms\n",
      "[ INFO ]    Max:           209.64 ms\n",
      "[ INFO ] Throughput:   31.35 FPS\n"
     ]
    }
   ],
   "source": [
    "# Inference INT8 model (OpenVINO IR)\n",
    "!benchmark_app -m $int8_model_det_path -d $device -api async -shape \"[1,3,640,640]\" -t 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Step 1/11] Parsing and validating input arguments\n",
      "[ INFO ] Parsing input parameters\n",
      "[Step 2/11] Loading OpenVINO Runtime\n",
      "[ INFO ] OpenVINO:\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] Device info:\n",
      "[ INFO ] CPU\n",
      "[ INFO ] Build ................................. 2023.0.0-10926-b4452d56304-releases/2023/0\n",
      "[ INFO ] \n",
      "[ INFO ] \n",
      "[Step 3/11] Setting device configuration\n",
      "[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.\n",
      "[Step 4/11] Reading model files\n",
      "[ INFO ] Loading model files\n",
      "[ INFO ] Read model took 26.03 ms\n",
      "[ INFO ] Original model I/O parameters:\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : u8 / [N,H,W,C] / [1,640,640,3]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [1,84,8400]\n",
      "[Step 5/11] Resizing model to match image sizes and given batch\n",
      "[ INFO ] Model batch size: 1\n",
      "[ INFO ] Reshaping model: 'images': [1,640,640,3]\n",
      "[ INFO ] Reshape model took 0.08 ms\n",
      "[Step 6/11] Configuring input of the model\n",
      "[ INFO ] Model inputs:\n",
      "[ INFO ]     images (node: images) : u8 / [N,H,W,C] / [1,640,640,3]\n",
      "[ INFO ] Model outputs:\n",
      "[ INFO ]     output0 (node: output0) : f32 / [...] / [1,84,8400]\n",
      "[Step 7/11] Loading the model to the device\n",
      "[ INFO ] Compile model took 369.02 ms\n",
      "[Step 8/11] Querying optimal runtime parameters\n",
      "[ INFO ] Model:\n",
      "[ INFO ]   NETWORK_NAME: torch_jit\n",
      "[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4\n",
      "[ INFO ]   NUM_STREAMS: 4\n",
      "[ INFO ]   AFFINITY: Affinity.CORE\n",
      "[ INFO ]   INFERENCE_NUM_THREADS: 8\n",
      "[ INFO ]   PERF_COUNT: False\n",
      "[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>\n",
      "[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT\n",
      "[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE\n",
      "[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0\n",
      "[ INFO ]   ENABLE_CPU_PINNING: True\n",
      "[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE\n",
      "[ INFO ]   ENABLE_HYPER_THREADING: True\n",
      "[ INFO ]   EXECUTION_DEVICES: ['CPU']\n",
      "[Step 9/11] Creating infer requests and preparing input tensors\n",
      "[ WARNING ] No input files were given for input 'images'!. This input will be filled with random values!\n",
      "[ INFO ] Fill input 'images' with random values \n",
      "[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 15000 ms duration)\n",
      "[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).\n",
      "[ INFO ] First inference took 37.39 ms\n",
      "[Step 11/11] Dumping statistics report\n",
      "[ INFO ] Execution Devices:['CPU']\n",
      "[ INFO ] Count:            664 iterations\n",
      "[ INFO ] Duration:         15143.50 ms\n",
      "[ INFO ] Latency:\n",
      "[ INFO ]    Median:        88.95 ms\n",
      "[ INFO ]    Average:       90.97 ms\n",
      "[ INFO ]    Min:           73.13 ms\n",
      "[ INFO ]    Max:           144.16 ms\n",
      "[ INFO ] Throughput:   43.85 FPS\n"
     ]
    }
   ],
   "source": [
    "!benchmark_app -m $quantized_model_with_preprocess_path -d $device -api async -shape \"[1,640,640,3]\" -t 15"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DONE, now you can either run demo.py or demo.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openvino-test-1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}