01c - Object Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Object Detection\n",
    "\n",
    "*Object detection* is a form of computer vision in which a machine learning model is trained to classify individual instances of objects in an image, and indicate a *bounding box* that marks its location. Youi can think of this as a progression from *image classification* (in which the model answers the question \"what is this an image of?\") to building solutions where we can ask the model \"what objects are in this image, and where are they?\".\n",
    "\n",
    "<p style='text-align:center'><img src='./images/object-detection.jpg' alt='A robot identifying fruit'/></p>\n",
    "\n",
    "For example, a grocery store might use an object detection model to implement an automated checkout system that scans a conveyor belt using a camera, and can identify specific items without the need to place each item on the belt and scan them individually.\n",
    "\n",
    "The **Custom Vision** cognitive service in Microsoft Azure provides a cloud-based solution for creating and publishing custom object detection models.\n",
    "\n",
    "## Create a Custom Vision resource\n",
    "\n",
    "To use the Custom Vision service, you need an Azure resource that you can use to train a model, and a resource with which you can publish it for applications to use. You can use the same resource for each of these tasks, or you can use different resources for each to allocate costs separately provided both resources are created in the same region. The resource for either (or both) tasks can be a general **Cognitive Services** resource, or a specific **Custom Vision** resource. Use the following instructions to create a new **Custom Vision** resource (or you can use an existing resource if you have one).\n",
    "\n",
    "1. In a new browser tab, open the Azure portal at [https://portal.azure.com](https://portal.azure.com), and sign in using the Microsoft account associated with your Azure subscription.\n",
    "2. Select the **&#65291;Create a resource** button, search for *custom vision*, and create a **Custom Vision** resource with the following settings:\n",
    "    - **Create options**: Both\n",
    "    - **Subscription**: *Your Azure subscription*\n",
    "    - **Resource group**: *Create a new resource group with a unique name*\n",
    "    - **Name**: *Enter a unique name*\n",
    "    - **Training location**: *Choose any available region*\n",
    "    - **Training pricing tier**: F0\n",
    "    - **Prediction location**: *The same as the training location*\n",
    "    - **Prediction pricing tier**: F0\n",
    "\n",
    "    > **Note**: If you already have an F0 custom vision service in your subscription, select **S0** for this one.\n",
    "\n",
    "3. Wait for the resource to be created.\n",
    "\n",
    "## Create a Custom Vision project\n",
    "\n",
    "To train an object detection model, you need to create a Custom Vision project based on your trainign resource. To do this, you'll use the Custom Vision portal.\n",
    "\n",
    "1. In a new browser tab, open the Custom Vision portal at [https://customvision.ai](https://customvision.ai), and sign in using the Microsoft account associated with your Azure subscription.\n",
    "2. Create a new project with the following settings:\n",
    "    - **Name**: Grocery Detection\n",
    "    - **Description**: Object detection for groceries.\n",
    "    - **Resource**: *The Custom Vision resource you created previously*\n",
    "    - **Project Types**: Object Detection\n",
    "    - **Domains**: General\n",
    "3. Wait for the project to be created and opened in the browser.\n",
    "\n",
    "## Add and tag images\n",
    "\n",
    "To train an object detection model, you need to upload images that contain the classes you want the model to identify, and tag them to indicate bounding boxes for each object instance.\n",
    "\n",
    "1. Download and extract the training images from https://aka.ms/fruit-objects. The extracted folder contains a collection of images of fruit.\n",
    "2. In the Custom Vision portal, in your object detection project, select **Add images** and upload all of the images in the extracted folder.\n",
    "3. After the images have been uploaded, select the first one to open it.\n",
    "4. Hold the mouse over any object in the image until an automatically detected region is displayed like the image below. Then select the object, and if necessary resize the region to surround it.\n",
    "\n",
    "    <p style='text-align:center'><img src='./images/object-region.jpg' alt='The default region for an object'/></p>\n",
    "\n",
    "    Alternatively, you can simply drag around the object to create a region.\n",
    "\n",
    "5. When the region surrounds the object, add a new tag with the appropriate object type (*apple*, *banana*, or *orange*) as shown here:\n",
    "\n",
    "    <p style='text-align:center'><img src='./images/object-tag.jpg' alt='A tagged object in an image'/></p>\n",
    "\n",
    "6. Select and tag each other object in the image, resizing the regions and adding new tags as required.\n",
    "\n",
    "    <p style='text-align:center'><img src='./images/object-tags.jpg' alt='Two tagged objects in an image'/></p>\n",
    "\n",
    "7. Use the **>** link on the right to go to the next image, and tag its objects. Then just keep working through the entire image collection, tagging each apple, banana, and orange.\n",
    "\n",
    "8. When you have finished tagging the last image, close the **Image Detail** editor and on the **Training Images** page, under **Tags**, select **Tagged** to see all of your tagged images:\n",
    "\n",
    "    <p style='text-align:center'><img src='./images/tagged-images.jpg' alt='Tagged images in a project'/></p>\n",
    "\n",
    "## Train and test a model\n",
    "\n",
    "Now that you've tagged the images in your project, you're ready to train a model.\n",
    "\n",
    "1. In the Custom Vision project, click **Train** to train an object detection model using the tagged images. Select the **Quick Training** option.\n",
    "2. Wait for training to complete (it might take ten minutes or so), and then review the *Precision*, *Recall*, and *mAP* performance metrics - these measure the prediction accuracy of the classification model, and should all be high.\n",
    "3. At the top right of the page, click **Quick Test**, and then in the **Image URL** box, enter `https://aka.ms/apple-orange` and view the prediction that is generated. Then close the **Quick Test** window.\n",
    "\n",
    "## Publish and consume the object detection model\n",
    "\n",
    "Now you're ready to publish your trained model and use it from a client application.\n",
    "\n",
    "1. At the top left of the **Performance** page, click **&#128504; Publish** to publish the trained model with the following settings:\n",
    "    - **Model name**: detect-produce\n",
    "    - **Prediction Resource**: *Your custom vision **prediction** resource*.\n",
    "2. After publishing, click the *settings* (&#9881;) icon at the top right of the **Performance** page to view the project settings. Then, under **General** (on the left), copy the **Project Id** and paste it into the code cell below replacing **YOUR_PROJECT_ID**.\n",
    "\n",
    "> (*if you used a **Cognitive Services** resource instead of creating a **Custom Vision** resource at the beginning of this exercise, you can copy its key and endpoint from the right side of the project settings, paste it into the code cell below, and run it to see the results. Otherwise, continue completing the steps below to get the key and endpoint for your Custom Vision prediction resource*).\n",
    "\n",
    "3. At the top left of the **Project Settings** page, click the *Projects Gallery* (&#128065;) icon to return to the Custom Vision portal home page, where your project is now listed.\n",
    "4. On the Custom Vision portal home page, at the top right, click the *settings* (&#9881;) icon to view the settings for your Custom Vision service. Then, under **Resources**, expand your *prediction* resource (<u>not</u> the training resource) and copy its **Key** and **Endpoint** values to the code cell below, replacing **YOUR_KEY** and **YOUR_ENDPOINT**.\n",
    "5. Run the code cell below by clicking its green <span style=\"color:green\">&#9655</span> button (at the top left of the cell) to set the variables to your project ID, key, and endpoint values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_id = 'YOUR_PROJECT_ID' # Replace with your project ID\n",
    "cv_key = 'YOUR_KEY' # Replace with your prediction resource primary key\n",
    "cv_endpoint = 'YOUR_ENDPOINT' # Replace with your prediction resource endpoint\n",
    "\n",
    "model_name = 'detect-produce' # this must match the model name you set when publishing your model iteration exactly (including case)!\n",
    "print('Ready to predict using model {} in project {}'.format(model_name, project_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Client applications can use the details above to connect to and your custom vision object detection model.\n",
    "\n",
    "Run the following code cell, which uses your model to detect individual produce items in an image.\n",
    "\n",
    "> **Note**: Don't worry too much about the details of the code. It uses the Python SDK for the Custom Vision service to submit an image to your model and retrieve predictions for detected objects. Each prediction consists of a class name (*apple*, *banana*, or *orange*) and *bounding box* coordinates that indicate where in the image the predicted object has been detected. The code then uses this information to draw a labelled box around each object on the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient\n",
    "from msrest.authentication import ApiKeyCredentials\n",
    "from matplotlib import pyplot as plt\n",
    "from PIL import Image, ImageDraw, ImageFont\n",
    "import numpy as np\n",
    "import os\n",
    "%matplotlib inline\n",
    "\n",
    "# Load a test image and get its dimensions\n",
    "test_img_file = os.path.join('data', 'object-detection', 'produce.jpg')\n",
    "test_img = Image.open(test_img_file)\n",
    "test_img_h, test_img_w, test_img_ch = np.array(test_img).shape\n",
    "\n",
    "# Get a prediction client for the object detection model\n",
    "credentials = ApiKeyCredentials(in_headers={\"Prediction-key\": cv_key})\n",
    "predictor = CustomVisionPredictionClient(endpoint=cv_endpoint, credentials=credentials)\n",
    "\n",
    "print('Detecting objects in {} using model {} in project {}...'.format(test_img_file, model_name, project_id))\n",
    "\n",
    "# Detect objects in the test image\n",
    "with open(test_img_file, mode=\"rb\") as test_data:\n",
    "    results = predictor.detect_image(project_id, model_name, test_data)\n",
    "\n",
    "# Create a figure to display the results\n",
    "fig = plt.figure(figsize=(8, 8))\n",
    "plt.axis('off')\n",
    "\n",
    "# Display the image with boxes around each detected object\n",
    "draw = ImageDraw.Draw(test_img)\n",
    "lineWidth = int(np.array(test_img).shape[1]/100)\n",
    "object_colors = {\n",
    "    \"apple\": \"lightgreen\",\n",
    "    \"banana\": \"yellow\",\n",
    "    \"orange\": \"orange\"\n",
    "}\n",
    "for prediction in results.predictions:\n",
    "    color = 'white' # default for 'other' object tags\n",
    "    if (prediction.probability*100) > 50:\n",
    "        if prediction.tag_name in object_colors:\n",
    "            color = object_colors[prediction.tag_name]\n",
    "        left = prediction.bounding_box.left * test_img_w \n",
    "        top = prediction.bounding_box.top * test_img_h \n",
    "        height = prediction.bounding_box.height * test_img_h\n",
    "        width =  prediction.bounding_box.width * test_img_w\n",
    "        points = ((left,top), (left+width,top), (left+width,top+height), (left,top+height),(left,top))\n",
    "        draw.line(points, fill=color, width=lineWidth)\n",
    "        plt.annotate(prediction.tag_name + \": {0:.2f}%\".format(prediction.probability * 100),(left,top), backgroundcolor=color)\n",
    "plt.imshow(test_img)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View the resulting predictions, which show the objects detected and the probability for each prediction."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "version": "3.5.3-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}