TensorFlow image_processing component

source/_components/image_processing.tensorflow.markdown

@@ -0,0 +1,163 @@
+---
+layout: page
+title: "TensorFlow"
+description: "Detect and recognize objects with TensorFlow."
+date: 2018-10-24 00:00
+sidebar: true
+comments: false
+sharing: true
+footer: true
+logo: tensorflow.png
+ha_category: Image Processing
+ha_iot_class: "Local Polling"
+ha_release: 0.82
+---
+
+The `tensorflow` image processing platform allows you to detect and recognize objects in a camera image using [TensorFlow](https://www.tensorflow.org/). The state of the entity is the number of objects detected, and recognized objects are listed in the `summary` attribute along with quantity. The `matches` attribute provides the confidence `score` for recognition and the bounding `box` of the object for each detection category.
+
+<p class='note warning'>
+  The following packages must be installed on Hassbian after following the setup for the component to work:
+  `$ sudo apt-get install libatlas-base-dev libopenjp2-7 libtiff5`
+</p>
+
+## {% linkable_title Setup %}
+
+You need to install the `tensorflow` Python packages with: `$ pip3 install tensorflow`. The wheel is not available for all platforms. See [the official install guide](https://www.tensorflow.org/install/) for other options. Hass.io has this package pre-installed.
+
+This component requires files to be downloaded, compiled on your computer, and added to the Home Assistant configuration directory. These steps can be performed using the sample script at [this gist](https://gist.github.com/hunterjm/6f9332f92b60c3d5e448ad936d7353c3). Alternatively, if you wish to perform the process manually, the process is as follows:
+
+- Clone [tensorflow/models](https://github.com/tensorflow/models/tree/master/research/object_detection)
+- Compile protobuf models located in `research/object_detection/protos` with `protoc`
+- Create the following directory structure inside your config directory:
+
+```bash
+  |- {config_dir}
+    | - tensorflow/
+      |- object_detection/
+        |- __init__.py
+```
+
+- Copy required object_detection dependancies to the `object_detection` folder inside of the `tensorflow` folder:
+
+  - `research/object_detection/data`
+  - `research/object_detection/utils`
+  - `research/object_detection/protos`
+
+## {% linkable_title Model Selection %}
+
+Lastly, it is time to pick a model. It is recommended to start with one of the COCO models available in the [Model Detection Zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md).
+
+The trade-off between the different models is accuracy vs speed.  Users with a decent CPU should start with the `faster_rcnn_inception_v2_coco` model. If you are running on an ARM device like a Raspberry Pi, start with the `ssd_mobilenet_v2_coco` model.
+
+Whichever model you choose, download it and place the `frozen_inference_graph.pb` file in the `tensorflow` folder in your configuration directory.
+
+## {% linkable_title Configuration %}
+
+To enable this platform in your installation, add the following to your `configuration.yaml` file:
+
+```yaml
+# Example configuration.yaml entry
+image_processing:
+  - platform: tensorflow
+    source:
+      - entity_id: camera.local_file
+    model:
+      graph: /home/homeassistant/.homeassistant/tensorflow/frozen_inference_graph.pb
+```
+
+{% configuration %}
+source:
+  description: The list of image sources.
+  required: true
+  type: map
+  keys:
+    entity_id:
+      description: A camera entity id to get picture from.
+      required: true
+      type: string
+    name:
+      description: This parameter allows you to override the name of your `image_processing` entity.
+      required: false
+      type: string
+file_out:
+    description: A [template](/docs/configuration/templating/#processing-incoming-data) for the component to save processed images including bounding boxes. `camera_entity` is available as the `entity_id` string of the triggered source camera.
+    required: false
+    type: list
+model:
+    description: Information about the TensorFlow model.
+    required: true
+    type: map
+    keys:
+      graph:
+        description: Full path to `frozen_inference_graph.pb`.
+        required: true
+        type: string
+      labels:
+       description: Full path to a `*label_map.pbtext`.
+       required: false
+       type: string
+       default: tensorflow/object_detection/data/mscoco_label_map.pbtxt
+      model_dir:
+        description: Full path to tensorflow models directory.
+        required: false
+        type: string
+        default: /tensorflow inside config
+      area:
+        description: Custom detection area. Only objects fully in this box will be reported. Top of image is 0, bottom is 1.  Same left to right.
+        required: false
+        type: map
+        keys:
+          top:
+            description: Top line defined as % from top of image.
+            required: false
+            type: float
+            default: 0
+          left:
+            description: Left line defined as % from left of image.
+            required: false
+            type: float
+            default: 0
+          bottom:
+            description: Bottom line defined as % from top of image.
+            required: false
+            type: float
+            default: 1
+          right:
+            description: Right line defined as % from left of image.
+            required: false
+            type: float
+            default: 1
+      categories:
+        description: List of categories to include in object detection. Can be seen in the file provided to `labels`.
+        type: list
+        required: false
+{% endconfiguration %}
+
+`categories` can also be defined as dictionary providing an `area` for each category as seen in the advanced configuration below:
+
+```yaml
+# Example advanced configuration.yaml entry
+image_processing:
+  - platform: tensorflow
+    source:
+      - entity_id: camera.driveway
+      - entity_id: camera.backyard
+    file_out:
+      - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}{% endraw %}_latest.jpg"
+      - "/tmp/{% raw %}{{ camera_entity.split('.')[1] }}_{{ now().strftime('%Y%m%d_%H%M%S') }}{% endraw %}.jpg"
+    model:
+      graph: /home/homeassistant/.homeassistant/tensorflow/frozen_inference_graph.pb
+      categories:
+        - category: person
+          area:
+            # Exclude top 10% of image
+            top: 0.1
+            # Exclude right 15% of image
+            right: 0.85
+        - car
+        - truck
+```
+
+## {% linkable_title Optimising resources %}
+
+[Image processing components](https://www.home-assistant.io/components/image_processing/) process the image from a camera at a fixed period given by the `scan_interval`. This leads to excessive processing if the image on the camera hasn't changed, as the default `scan_interval` is 10 seconds. You can override this by adding to your config `scan_interval: 10000` (setting the interval to 10,000 seconds), and then call the `image_processing.scan` service when you actually want to perform processing.