inc_sample_for_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e37ea7a6",
   "metadata": {},
   "source": [
    "# Intel® Neural Compressor Sample for Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "799b2d67",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This is a demo to show an End-To-End pipeline to speed up AI model by Intel® Neural Compressor.\n",
    "\n",
    "1. Train a CNN AlexNet model by Keras and Intel Optimization for Tensorflow based on dataset MNIST.\n",
    "\n",
    "2. Quantize the frozen PB model file by Intel® Neural Compressor to INT8 model.\n",
    "\n",
    "3. Test and compare the performance of FP32 and INT8 model by same script.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e538cd95-f291-41aa-9b48-00956855aec1",
   "metadata": {},
   "source": [
    "## Code\n",
    "Please refer to [README.md](README.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f79a98-2704-43b6-97e1-d4eabcf10e20",
   "metadata": {},
   "source": [
    "## Edit script (Optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da204585-fcec-419f-877d-b78c59e160dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile run_inc_ft_mnist_sample.sh\n",
    "#!/bin/bash\n",
    "\n",
    "echo \"Enable Conda Env.\"\n",
    "source /glob/development-tools/versions/oneapi/2022.1.1/oneapi/intelpython/python3.9/etc/profile.d/conda.sh\n",
    "#conda activate user_tensorflow\n",
    "conda activate /data/oneapi_workshop/INC\n",
    "\n",
    "echo \"Train Model by Keras/Tensorflow with MNIST\"\n",
    "python keras_tf_train_mnist.py\n",
    "\n",
    "FP32_FILE=\"fp32_frozen.pb\"\n",
    "if [ ! -f $FP32_FILE ]; then\n",
    "    echo \"$FP32_FILE not exists.\"\n",
    "    echo \"Train AlexNet model is fault, exit!\"\n",
    "    exit 1\n",
    "else\n",
    "    echo \"Training is finished\"\n",
    "fi\n",
    "\n",
    "echo \"Enable Intel Optimization for Tensorflow by exporting TF_ENABLE_MKL_NATIVE_FORMAT=0\"\n",
    "echo \"Intel Optimized TensorFlow 2.5.0 and later require to set environment variable TF_ENABLE_MKL_NATIVE_FORMAT=0 before running Intel® Neural Compressor quantize Fp32 model or deploying the quantized model.\"\n",
    "\n",
    "export TF_ENABLE_MKL_NATIVE_FORMAT=0\n",
    "\n",
    "echo \"Quantize Model by Intel Neural Compressor\"\n",
    "python inc_quantize_model.py\n",
    "\n",
    "INT8_FILE=\"alexnet_int8_model.pb\"\n",
    "if [ ! -f $INT8_FILE ]; then\n",
    "    echo \"$INT8_FILE not exists.\"\n",
    "    echo \"Quantize FP32 model is fault, exit!\"\n",
    "    exit 1\n",
    "else\n",
    "    echo \"Quantization is finished\"\n",
    "fi\n",
    "\n",
    "echo \"Execute the profiling_inc.py with FP32 model file\"\n",
    "python profiling_inc.py --input-graph=./fp32_frozen.pb --omp-num-threads=4 --num-inter-threads=1 --num-intra-threads=4 --index=32\n",
    "echo \"FP32 performance test is finished\"\n",
    "\n",
    "echo \"Execute the profiling_inc.py with INT8 model file\"\n",
    "python profiling_inc.py --input-graph=./alexnet_int8_model.pb --omp-num-threads=4 --num-inter-threads=1 --num-intra-threads=4 --index=8\n",
    "echo \"INT8 performance test is finished\"\n",
    "\n",
    "echo \"Compare the Performance of FP32 and INT8 Models\"\n",
    "python compare_perf.py\n",
    "echo \"Please check the PNG files to see the performance!\"\n",
    "\n",
    "if [[ $? -eq 0 ]]\n",
    "then\n",
    "  echo \"This demo is finished successfully!\"\n",
    "else\n",
    "  echo \"This demo is fault!\"\n",
    "fi\n",
    "\n",
    "echo \"Thank you!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd11036b-2535-425b-8bd4-2125263c64ec",
   "metadata": {},
   "source": [
    "## Check Script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4278f0e-83a1-4bf0-bf04-93dcbfa4c5f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat run_inc_ft_mnist_sample.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71858ff2-c7b6-425e-a7c4-eff227cc481e",
   "metadata": {},
   "source": [
    "## Prepare Running Environment\n",
    "\n",
    "Please refer to [README.md](README.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "735982ec-2398-479b-a927-01d7e9f30ea1",
   "metadata": {},
   "source": [
    "### Remove all old output files (Optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c7ca46e-0fc8-4818-ac57-d354414ee6d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf run_inc_ft_mnist_sample.sh.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82f09276",
   "metadata": {},
   "source": [
    "## Run in Intel® DevCloud\n",
    "\n",
    "Job submit to compute node with the property 'clx' or 'icx' or 'spr' which support Intel® Deep Learning Boost (avx512_vnni)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f51bc091",
   "metadata": {},
   "outputs": [],
   "source": [
    "!qsub run_inc_ft_mnist_sample.sh -d `pwd` -l nodes=1:icx:ppn=2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0d7cab-1b60-4689-b153-506e5818b811",
   "metadata": {},
   "source": [
    "Check job status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7199754d-e7e4-4e52-868d-0a1ca79cb064",
   "metadata": {},
   "outputs": [],
   "source": [
    "!qstat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc5b605-47d1-485f-bfb1-cd7ab9f3f83c",
   "metadata": {},
   "source": [
    "### Check Result\n",
    "\n",
    "#### Check Result in Log File\n",
    "Check the latest created log file with prefix: **run_inc_ft_mnist_sample.sh.o**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6b36c9c-f612-4517-914c-d5ca6ee92d27",
   "metadata": {},
   "outputs": [],
   "source": [
    "!tail -23 `ls -lAtr run_inc_ft_mnist_sample.sh.o* |  tail -1 | awk '{print $9}'`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9822cad-22e8-415d-b33b-fd12065c5163",
   "metadata": {},
   "source": [
    "Check any existed log file, for example **run_inc_ft_mnist_sample.sh.o1842343**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d49e69a-ccfe-42f0-982d-38395ed4a808",
   "metadata": {},
   "outputs": [],
   "source": [
    "!tail -23 run_inc_ft_mnist_sample.sh.o1842343"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a80934c4-8ddd-48c3-acc5-63dc0bb1372a",
   "metadata": {},
   "source": [
    "#### Check Result in PNG file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44c31db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "listOfImageNames = ['fp32_int8_aboslute.png',\n",
    "                    'fp32_int8_times.png']\n",
    "\n",
    "for imageName in listOfImageNames:\n",
    "    display(Image(filename=imageName))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b4cded5-3723-42e5-aec1-8ec514ccd49e",
   "metadata": {},
   "source": [
    "## Run in Customer Server or Cloud\n",
    "\n",
    "Note, it's recommended to use 2nd Generation Intel® Xeon® Scalable Processors or newer to get better performance improvement.\n",
    "\n",
    "### Run in Jupyter Notebook\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "741279c7-f788-47f1-ab9a-8f0628a79d16",
   "metadata": {},
   "outputs": [],
   "source": [
    "run_inc_ft_mnist_sample.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3cb8011-31c4-4a7c-be00-775d2ec940f4",
   "metadata": {},
   "source": [
    "### Check Result\n",
    "\n",
    "#### Check Result in Screen Output\n",
    "\n",
    "```\n",
    "...\n",
    "\n",
    "Compare the Performance of FP32 and INT8 Models\n",
    "Model            FP32                     INT8                    \n",
    "throughput(fps)  572.4982883964987        3218.52236638019        \n",
    "latency(ms)      2.8339174329018104       1.9863116497896156      \n",
    "accuracy(%)      0.9799                   0.9796                  \n",
    "\n",
    "Save to fp32_int8_aboslute.png\n",
    "\n",
    "Model            FP32                     INT8                    \n",
    "throughput_times 1                        5.621889936815179       \n",
    "latency_times    1                        0.7009066766478504      \n",
    "accuracy_diff(%) 0                        -0.029999999999986926   \n",
    "\n",
    "Save to fp32_int8_times.png\n",
    "Please check the PNG files to see the performance!\n",
    "This demo is finished successfully!\n",
    "Thank you!\n",
    "\n",
    "########################################################################\n",
    "# End of output for job 1842253.v-qsvr-1.aidevcloud\n",
    "# Date: Thu 27 Jan 2022 07:05:52 PM PST\n",
    "########################################################################\n",
    "\n",
    "...\n",
    "\n",
    "```\n",
    "#### Check Result in PNG file\n",
    "\n",
    "The demo creates figure files: fp32_int8_aboslute.png, fp32_int8_times.png to show performance bar. They could be used in report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82c4f0b7-2451-41db-bd84-0fc26e74aab2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "listOfImageNames = ['fp32_int8_aboslute.png',\n",
    "                    'fp32_int8_times.png']\n",
    "\n",
    "for imageName in listOfImageNames:\n",
    "    display(Image(filename=imageName))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f882b021-190d-438e-9cc8-f76b501c6be5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Tensorflow (AI kit)",
   "language": "python",
   "name": "c009-intel_distribution_of_python_3_oneapi-beta05-tf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "nbTranslate": {
   "displayLangs": [
    "*"
   ],
   "hotkey": "alt-t",
   "langInMainMenu": true,
   "sourceLang": "en",
   "targetLang": "fr",
   "useGoogleTranslate": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}