demo/serving/bentoml/cloud-native-model-serving-with-bentoml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "erfOlc-T8kY3"
   },
   "source": [
    "# **BentoML Example: Image Segmentation with PaddleHub**\n",
    "**BentoML makes moving trained ML models to production easy:**\n",
    "\n",
    "\n",
    "\n",
    "*   Package models trained with any ML framework and reproduce them for model serving in production\n",
    "* **Deploy anywhere** for online API serving or offline batch serving\n",
    "* High-Performance API model server with adaptive micro-batching support\n",
    "* Central hub for managing models and deployment process via Web UI and APIs\n",
    "* Modular and flexible design making it adaptable to your infrastrcuture\n",
    "\n",
    "BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.\n",
    "\n",
    "Before reading this example project, be sure to check out the [Getting started guide](https://github.com/bentoml/BentoML/blob/master/guides/quick-start/bentoml-quick-start-guide.ipynb) to learn about the basic concepts in BentoML.\n",
    "\n",
    "This notebook demonstrates how to use BentoML to turn a Paddlehub module into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.\n",
    "\n",
    "This example notebook is based on the [Python quick guide from PaddleHub](https://github.com/PaddlePaddle/PaddleHub/blob/release/v2.0/docs/docs_en/quick_experience/python_use_hub_en.md).\n"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "54jFhiru8NWO"
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "XHOPuMGm-Nl2"
   },
   "outputs": [],
   "source": [
    "!pip3 install -q bentoml paddlepaddle paddlehub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KXz5IFU94P9D"
   },
   "outputs": [],
   "source": [
    "!hub install deeplabv3p_xception65_humanseg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bWx5VF_LLTef"
   },
   "source": [
    "## Prepare Input Data"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yayroXhE-sos"
   },
   "outputs": [],
   "source": [
    "!wget https://paddlehub.bj.bcebos.com/resources/test_image.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zcrHdbJxAHh0"
   },
   "source": [
    "## Create BentoService with PaddleHub Module Instantiation"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "s_T8YQRjALqg"
   },
   "outputs": [],
   "source": [
    "%%writefile paddlehub_service.py\n",
    "import paddlehub as hub\n",
    "import bentoml\n",
    "from bentoml import env, artifacts, api, BentoService\n",
    "import imageio\n",
    "from bentoml.adapters import ImageInput\n",
    "\n",
    "\n",
    "@env(infer_pip_packages=True)\n",
    "class PaddleHubService(bentoml.BentoService):\n",
    "    def __init__(self):\n",
    "      super(PaddleHubService, self).__init__()\n",
    "      self.module = hub.Module(name=\"deeplabv3p_xception65_humanseg\")\n",
    "\n",
    "    @api(input=ImageInput(), batch=True)\n",
    "    def predict(self, images):\n",
    "        results = self.module.segmentation(images=images, visualization=True)\n",
    "        return [result['data'] for result in results]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ESc4D_muCWNx"
   },
   "outputs": [],
   "source": [
    "# Import the custom BentoService defined above\n",
    "from paddlehub_service import PaddleHubService\n",
    "import numpy as np\n",
    "import cv2\n",
    "\n",
    "# Pack it with required artifacts\n",
    "bento_svc = PaddleHubService()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ondQXpNCy_TV"
   },
   "outputs": [],
   "source": [
    "# Predict with the initialized module\n",
    "image = cv2.imread(\"test_image.jpg\")\n",
    "images = [image]\n",
    "segmentation_results = bento_svc.predict(images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f-61QUPd6w9h"
   },
   "source": [
    "### Visualizing the result"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jNnyhPQt59ey"
   },
   "outputs": [],
   "source": [
    "# View the segmentation mask layer\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "for result in segmentation_results:\n",
    "    plt.imshow(cv2.cvtColor(result, cv2.COLOR_BGR2RGB))\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kmJkYFPNRnmA"
   },
   "outputs": [],
   "source": [
    "# Get the segmented image of the original image\n",
    "for result, original in zip(segmentation_results, images):\n",
    "    result = cv2.cvtColor(result, cv2.COLOR_GRAY2RGB)\n",
    "    original_mod = cv2.cvtColor(original, cv2.COLOR_RGB2RGBA)\n",
    "    mask = result / 255\n",
    "    *_, alpha = cv2.split(mask)\n",
    "    mask = cv2.merge((mask, alpha))\n",
    "    segmented_image = (original_mod * mask).clip(0, 255).astype(np.uint8)\n",
    "    \n",
    "    plt.imshow(cv2.cvtColor(segmented_image, cv2.COLOR_BGRA2RGBA))\n",
    "    plt.axis('off')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "07YXA0ne7ZBc"
   },
   "source": [
    "### Start dev server for testing"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pUM64JEKaRWt"
   },
   "outputs": [],
   "source": [
    "# Start a dev model server\n",
    "bento_svc.start_dev_server()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3valpr2oa_OM"
   },
   "source": [
    "!curl -i \\\n",
    "  -F image=@test_image.jpg \\\n",
    "  localhost:5000/predict"
   ],
   "outputs": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oCW5xuPebByD"
   },
   "outputs": [],
   "source": [
    "# Stop the dev model server\n",
    "bento_svc.stop_dev_server()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qwSpmZ1u7gez"
   },
   "source": [
    "### Save the BentoService for deployment"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kCHUw-_Hy6tH"
   },
   "outputs": [],
   "source": [
    "saved_path = bento_svc.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IvUU0k0JCxYk"
   },
   "source": [
    "## REST API Model Serving"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CeJEIDyj_xGK"
   },
   "outputs": [],
   "source": [
    "!bentoml serve PaddleHubService:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FPoKbR6cCq8_"
   },
   "source": [
    "If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RodE8ooiCqRw"
   },
   "outputs": [],
   "source": [
    "!bentoml serve PaddleHubService:latest --run-with-ngrok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FMCrkYb5DDHB"
   },
   "source": [
    "## Make request to the REST server\n",
    "\n",
    "*After navigating to the location of this notebook, copy and paste the following code to your terminal and run it to make request*"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fMyLXOIUDXSn"
   },
   "outputs": [],
   "source": [
    "curl -i \\\n",
    "  --header \"Content-Type: image/jpeg\" \\\n",
    "  --request POST \\\n",
    "  --data-binary @test_image.jpg \\\n",
    "  localhost:5000/predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DlGTKeMnEEyE"
   },
   "source": [
    "## Launch inference job from CLI"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CBqvdN9-iyQu"
   },
   "outputs": [],
   "source": [
    "!bentoml run PaddleHubService:latest predict --input-file test_image.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6RA0JpPjDMt8"
   },
   "source": [
    "## Containerize model server with Docker\n",
    "\n",
    "One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.\n",
    "\n",
    "Note that docker is **not available in Google Colab**. You will need to download and run this notebook locally to try out this containerization with docker feature.\n",
    "\n",
    "If you already have docker configured, simply run the follow command to product a docker container serving the PaddeHub prediction service created above:"
   ],
   "outputs": [],
   "execution_count": null
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JKUGBMNWDJnr"
   },
   "outputs": [],
   "source": [
    "!bentoml containerize PaddleHubService:latest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0nyRChqMDwv4"
   },
   "outputs": [],
   "source": [
    "!docker run --rm -p 5000:5000 PaddleHubService:latest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Jb-srm9RENeh"
   },
   "source": [
    "# **Deployment Options**\n",
    "\n",
    "If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:\n",
    "\n",
    "* [AWS Lambda Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_lambda.html)\n",
    "* [AWS SageMaker Deployment Guide](https://docs.bentoml.org/en/latest/deployment/aws_sagemaker.html)\n",
    "* [Azure Functions Deployment Guide](https://docs.bentoml.org/en/latest/deployment/azure_functions.html)\n",
    "\n",
    "If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:\n",
    "\n",
    "* [AWS ECS Deployment](https://docs.bentoml.org/en/latest/deployment/aws_ecs.html)\n",
    "* [Google Cloud Run Deployment](https://docs.bentoml.org/en/latest/deployment/google_cloud_run.html)\n",
    "* [Azure container instance Deployment](https://docs.bentoml.org/en/latest/deployment/azure_container_instance.html)\n",
    "* [Heroku Deployment](https://docs.bentoml.org/en/latest/deployment/heroku.html)\n",
    "\n",
    "Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:\n",
    "\n",
    "* [Kubernetes Deployment](https://docs.bentoml.org/en/latest/deployment/kubernetes.html)\n",
    "* [Knative Deployment](https://docs.bentoml.org/en/latest/deployment/knative.html)\n",
    "* [Kubeflow Deployment](https://docs.bentoml.org/en/latest/deployment/kubeflow.html)\n",
    "* [KFServing Deployment](https://docs.bentoml.org/en/latest/deployment/kfserving.html)\n",
    "* [Clipper.ai Deployment Guide](https://docs.bentoml.org/en/latest/deployment/clipper.html)"
   ],
   "outputs": [],
   "execution_count": null
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "PaddleHub_deeplabv3p_xception65_humanseg.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}