Snips ASR and NLU component

[michaelfester]

Description:

Snips on-device ASR and NLU custom component.

For installation and setup of the Snips SDK, check out the Snips Installation Guide.

The Snips SDK must be installed and running locally, and Home Assistant should use the Snips SDK as an MQTT broker for simplicity. For instance, if the Snips platform is running locally on port 9898, the following should be added to configuration.yaml:

mqtt:
  broker: 127.0.0.1
  port: 9898

Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.io#2892

Example entry for configuration.yaml:

Just like Alexa and API.ai, Snips transforms voice and text into intents. Intents are created via the Snips Console, and can then be handled in Home Assistant by adding the intents, and accompanying actions, to the snips entry in configuration.yaml:

snips:
  intents:
    Lights:
      action:
        - service_template: >
            {%- if light_state == "on" -%}
              light.turn_on
            {%- else -%}
              light.turn_off
            {%- endif -%}
          data_template:
            entity_id: light.living_room

[homeassistant]

Hi @michaelfester,

It seems you haven't yet signed a CLA. Please do so here.

Once you do that we will be able to review and accept this pull request.

Thanks!

[houndci-bot]

no newline at end of file

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

blank line contains whitespace

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

multiple statements on one line (colon)

[houndci-bot]

line too long (110 > 79 characters)

[houndci-bot]

local variable 'a' is assigned to but never used

[houndci-bot]

continuation line under-indented for visual indent

[houndci-bot]

expected 2 blank lines, found 1

[houndci-bot]

expected 2 blank lines, found 1

[houndci-bot]

'asyncio' imported but unused

[balloob]

Very cool! Looking forward to Snips support.

This will also require a test to be written. Could be something like this (untested):

import asyncio

from homeassistant.bootstrap import async_setup_component
from tests.common import async_fire_mqtt_message, mock_service

EXAMPLE_MSG = """
{
  // snips message from hermes/nlu/intentParsed for intent Lights
}
"""

@asyncio.coroutine
def test_snips_call_action(hass, mqtt_mock):
    """Test calling action via Snips."""
    calls = mock_service(hass, 'test', 'service')

    result = yield from async_setup_component(hass, 'snips', {
        'snips': {
            'intents': {
                'Lights': {
                    'action': {
                        'service': 'test.service'
                    }
                }
            }
        }
    })
    assert result

    async_fire_mqtt_message(hass, 'hermes/nlu/intentParsed',
                            EXAMPLE_MSG)
    yield from hass.async_block_till_done()

    assert len(calls) == 1

[balloob]

Off to a good start, will need some more work. There is also a bunch of lint issues that need to be addressed https://travis-ci.org/home-assistant/home-assistant/jobs/245793758#L282

[balloob]

Since message_received only cares about messages from hermes/nlu/intentParsed, better to just listen to that topic instead and not allow it to be configured?

[balloob]

Please change this to debug.

[balloob]

Please remove this.

[balloob]

Please remove this.

[balloob]

This can raise a TypeError if invalid JSON.

[balloob]

It's probably the easiest to write a voluptuous schema to validate the schema that Snips is sending. That way you can get rid of all other checks if some piece of data is available.

[balloob]

Catch KeyError. Bare exceptions are not allowed.

[balloob]

This is not possible. Python will raise if a key does not exists. Unless the JSON value of slots can be set to null ? Again, probably easier to write a voluptuous schema to validate this info.

[balloob]

Errrrrr, what is happening here? 🤔

[houndci-bot]

indentation contains mixed spaces and tabs

[houndci-bot]

expected 2 blank lines, found 1

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[houndci-bot]

indentation contains tabs

[bastshoes]

Hello.
In most cases HASS users already have MQTT installed in their setup. Currently HASS do not support multiple MQTT brokers.
As I understand from documentation Snips Dockers image includes MQTT broker and it available on port 9898.
Is it possible to use existing broker with Snips component?

[michaelfester]

Hi @bastshoes! Currently this is not possible but we are fixing this :)

[balloob]

Component never worked because the name extraction code was wrong. It referenced intentName instead of intent_name (as per Snips docs). Slot parsing had the same problem. You referenced slotName instead of slot_name (as per Snips docs).

Another issue was that mock_service was no longer allowed to be called in an async context. Rebased the branch and imported async_mock_service instead.

Converted it to async because there was no reason to use threads.

I've added some schema validation to validate incoming data instead of having to check each value individually.

Now all tests should pass and this is good to go.

[balloob]

Cherry-picked for 0.48.

[michaelfester]

Excellent stuff. Thanks @balloob!

[Mika56]

Are you certain about the intentName and slotName keys?
I had multiple errors (Intent has invalid schema: required key not provided @ data['slots'][0]['slot_name'], Intent has invalid schema: required key not provided @ data['text'], ...)

I had to build a MQTT bridge that adapts the output from Snips:

var mqtt = require('mqtt');

var snipsClient = mqtt.connect('mqtt://localhost:9898');
var hassClient = mqtt.connect('mqtt://192.168.1.247:1884');

snipsClient.on('connect', function() { 
	console.log('Connected to Snips broker'); 
});

hassClient.on('connect', function() {
	console.log('Connected to HASS broker');
});

snipsClient.on('message', function(topic, message) {
	console.log(' < '+topic);
	message = message.toString();
	if(message) {
		message = JSON.parse(message);
		//console.log(message);
		if(message.hasOwnProperty('intent')) {
			message.intent.intent_name = message.intent.intentName;
		}
		if(message.hasOwnProperty('slots')) {
			for(var i=0;i<message.slots.length;i++) {
				message.slots[i].slot_name = message.slots[i].slotName;
			}
		}
		if(message.hasOwnProperty('input')) {
			message.text = message.input;
		}
		//console.log(message);
		hassClient.publish(topic, JSON.stringify(message));
	}
	else {
		hassClient.publish(topic, message);
	}
});

snipsClient.subscribe('#');

[balloob]

@Mika56 that's weird. I see intent_name and slot_name used all over the docs https://github.com/snipsco/snips-platform-documentation/wiki/5.-Learn-more:-Key-Concepts#3-natural-language-understanding and https://medium.com/snips-ai/integrating-snips-with-home-assistant-314723645c77#6bac

I do admit that I didn't test it with an actual Snips instance.

@michaelfester can you chime in what the right variable names are ?

• text or input
• slot_name or slotName
• intent_name or intentName

[michaelfester]

Yes, change of API. Update throughout :)