Integrate with Local Home SDK

[kubawolanin]

Recently Google announced they'll allow (finally!) to control smart home devices through local network. That would mean essentially that e.g Google Home could directly send requests to our openHAB instance that's living in the same network.
This is a big deal mostly from the performance standpoint.

They provided a sample:
https://github.com/actions-on-google/smart-home-local

which I would like to explore in order to see if we can easily integrate that with our current Assistant add-on.

[kubawolanin]

Looks like we should incorporate UPnP or mDNS announcement from OH instance in order to make it work:

The SDK introduces two new intents, IDENTIFY and REACHABLE_DEVICES. The local home platform scans the user's home network via mDNS, UDP, or UPnP to discover any smart devices connected to the Assistant, and triggers IDENTIFY to verify that the device IDs match those returned from the familiar Smart Home API SYNC intent. If the detected device is a hub or bridge, REACHABLE_DEVICES is triggered and treats the hub as the proxy device for communicating locally. Once the local execution path from Google Home to a device is established, the device properties are updated in Home Graph.

Source: https://developers.googleblog.com/2019/07/developer-preview-of-local-home-sdk.html

[marziman]

Thanks!
Just thinking if we will need the cloud for local connection. I will digg deeper ..

[alackmann]

Is there any progress being made on this one? @marziman @kubawolanin - it'd be great to be triggering OH directly rather than via an internet hop.

[marziman]

@kubawolanin
I took a closer look and we need to understand, that this means a new binding on OH local level with support for Java Script. Other than that we need to rebuild some functionalities in Java if I am not wrong. So there is some local work which needs to be done.

Let me add new features to the current version, and I will try to draw a picture on how we could tackle it possibly.

BR Mehmet

[Xeiotos]

@marziman I'm looking into this (haven't gotten super-deep into the Local SDK yet), why would you need a new binding on OH local level with support for Javascript? As far as I've seen the Local SDK runs the app in JavaScript directly on the Google Home/Google Nest device. There's support for Hub-like devices (OpenHAB) through which the Google Assistant device can reach the local devices (lights, ...).

As far as I understand, provided you can run this project on the Google Home itself to translate Google Assistant requests to the OH Rest API, no work on OH level would be needed, right?

The only thing I can't get a handle on (right now) is the mDNS/UDP/UPnP discovery of OpenHAB by the Google Assistant device; but I'm fairly sure it would be trivial to someone more familiar with OpenHAB services.

Please correct me if my assumptions are wrong; I'm not overly familiar with this topic.

[gytisgreitai]

You don't. Yes, the Local SDK runs on the devices. Work on OH would be needed to implement some sort of binding/service that would respond to UDP/mDNS request from Google Devices. As far as I've looked you cannot do this with current bindings (though you could probably hardcode your google device ips and constantly send them predefined UDP packets).

But for now there is no need to hurry, cause:

1. Though Local SDK was announced In more than half a year ago, support for hub devices (that is things like OpenHAB which have a lot of devices behind them) landed only in the middle of December
2. In order to use hub devices you must switch your Google Devices to Preview mode ... because changes are not yet in latest firmware
3. This whole thing is still in Preview mode - you cannot make a production release to it.

I've implemented this local SDK and tested on my Google Home Mini, but so far I am not impressed:

1. No "visual" speed difference when issuing voice commands
2. It's kind of buggy. It always toggles the state correctly on openHAB but I would say 50% of the time I get a response "Sorry, the XXX is currently unavailable"

[radokristof]

@gytisgreitai the processing of the spoken voice is still done on remote (Google) servers, right? So that's why I think there is no really difference is speed, because it still needs to use "remote" servers, even if devices are local... Also this means that this still not work if there is no internet.
First I thought that this will be a game changer and should be implemented fairly fast, but now after realizing these, I don't know if it makes any difference.
One thing though: Maybe if this will get implemented, you wouldn't need to use myopenhab at all?

[gytisgreitai]

@radokristof for Mini gen1 - yes. As For Google Nest Mini (aka. gen 2) the internet says Nest Mini will be able to learn the most commonly-used commands, and then process them locally so that the Assistant’s response time is much swifter. Afaik gen1 does not have the hardware power to do it (sadly, I have only gen 1 to test)

myopenhab or any other form of remote access will be needed, because of initial sync. You must tell google cloud that device X is available for local execution.

[radokristof]

Ah thanks! I think I will buy more Assistant device, I will try to get newer Nest Mini.
Anyway I hoped that somehow you could use mdns also to announce and add local devices directly to Google Home...

[Tihmann]

Obviously Yeelight just released local SDK support https://stadt-bremerhaven.de/yeelight-setzt-nun-auf-lokale-steuerung-durch-den-google-home/

[Tihmann]

Maybe it is worth to have a look at that solution. They don't mention necessary Preview mode

[gytisgreitai]

Though I don't speak german but AFAIK Yeelight is not a hub device, these are separate devices with separate IP addresses which can be directly controlled. Regarding the preview mode, here is the comment from Google employees google-home/smart-home-local#13 (comment)

[robnielsen]

Local Home SDK has been released: https://developers.googleblog.com/2020/04/local-home-sdk-ready-for-actions.html

[robnielsen]

Documentation: https://developers.google.com/assistant/smarthome/concepts/local

[chris400]

@gytisgreitai the processing of the spoken voice is still done on remote (Google) servers, right? So that's why I think there is no really difference is speed, because it still needs to use "remote" servers, even if devices are local... Also this means that this still not work if there is no internet.

You can control devices on the iOS and Android GH app and on the screen of Nest Hub. No voice is involved.

[robertcsakany]

I think it can be a very nice feature. Its a big help if google home app can control the device without internet by screen. As a matter of fact the OpenHAB android app is slow as hell in local network too, because it does not use offline mode. Another reason Home app is more easy to use for example for my family, because assistant can be used easly. I have now myopenhab instance running in my own server, but if I can remove it from the chain it can be a big help.

[voruti]

Are there any updates on this?

[michikrug]

Not from my side.

[JohannesPtaszyk]

I think this would still be a good thing to implement, since fulfilment is done locally instead of using myopenhab, if I understand correctly.
So wouldn't this also lower the load on myopenhab servers?

[cholawo]

I was reading this morning that Matter was added to google nest devices, I see there's some conversations about it in openhab. My understanding is that if openhab exposed items via Matter to nest devices in the home, then you'd get local fulfilment right away, and also we might not even need openhab-google-assistant?

[michikrug]

Well, I guess openhab then would need to act as something like a "Matter bridge" for the devices connected.
As I am not at all into the topic, I can not tell if this would work at all.

[barneysspeedshop]

It looks like this might be integrated into HomeAssistant... Maybe we could take a look at how they did it and apply some of that here?
https://www.home-assistant.io/integrations/google_assistant_sdk/