Arduino version

[createcandle]

I've tried to create an Arduino version of this project. That may allow users of the WebThings Gateway to create new satellites by simply plugging an Atom Echo into their Raspberry Pi and click on a "make a new satellite" button.

https://github.com/createcandle/voco-mini-satellite

The goal is to make it compatible with Voco. Voco still relies on Snips for now, so I'm curious: would it be hard to make the code compatible with Snips again? Have there been big changes in this regard?

Is this correct for the Atom Echo?

{
  "mqtt_host": "192.168.2.167",
  "mqtt_port": 1883,
  "mqtt_user": "",
  "mqtt_pass": "",
  "mqtt_valid": true,
  "mute_input": false,
  "mute_output": false,
  "amp_output": 0,
  "brightness": 30,
  "hotword_brightness": 100,
  "hotword_detection": 1,
  "volume": 100,
  "gain": 5
}

Also, I seem to run into the same error where is reboots every few seconds. I've added some extra debug outputs in the log below to try and figure out at what point things to wrong.

/config.json
{
  "mqtt_host": "192.168.2.167",
  "mqtt_port": 1883,
  "mqtt_user": "",
  "mqtt_pass": "",
  "mqtt_valid": true,
  "mute_input": false,
  "mute_output": false,
  "amp_output": 0,
  "brightness": 30,
  "hotword_brightness": 100,
  "hotword_detection": 1,
  "volume": 100,
  "gain": 5
}
config file loaded
no audiogroup so xEventGroupCreate
Enter WifiDisconnected
Total heap: 276128
Free heap: 203396
Enter WifiConnected
Connected to Wifi with IP: 192.168.2.137
end of setupEnter MQTTDisconnected
MQTT was not initialized.
past setClientId
192.168.2.167
1883
clientID = atomechoAudio
past asyncclient.connectend of mqttdisconnectedEntry complete
Enter MQTTConnected
Connected as atomecho
Enter Idle
Still Idle
start streaming
past stream
end of idle
Enter MQTTDisconnected
Connect failed, retry
Enter MQTTDisconnected
past setClientIdpast setClientId

192.192168.168..2.167
2.167
1883
1883
clientID = atomechoAudio
clientID = past asyncclient.connectatomechoAudio
past asyncclient.connect
Backtrace: 0x4008c728:0x3ffbe170 0x4008c959:0x3ffbe190 0x400e8aaf:0x3ffbe1b0 0x40084789:0x3ffbe1d0 0x40081a7b:0x3ffdecc0 0x400826bf:0x3ffdece0 0x4008126f:0x3ffded00 0x400ddfe1:0x3ffded20 0x400de05a:0x3ffded60 0x400d2082:0x3ffded90 0x400d20c3:0x3ffdee20 0x400d21cf:0x3ffdee40 0x400d2a50:0x3ffdee60 0x40088ce5:0x3ffdf2e0

Rebooting...

It's almost as if it's running twice.

[Romkabouter]

I do not think an arduino version is going to allow you upload the sketch to the candle addon manager.

I am investigating this reboot issue, I suspect an issue with a broker.
What broker are you using?
It makes two connections, 1 for asynch and 1 for sync.
When the Idle state is entered, all is well. But then the connection is dropped which leads to the MQTTDisconnected state.
This state tries to reconnect every 5 seconds.

I remember an issue with asyncClient.disconnect() causing crashes, I will see if I can find it.

[Romkabouter]

With regards to Snips, it should be relatively easy. There are a couple of topics you need to recreate. If you check version 6.0, the code is still there.
Also, there is an Arduino version working for the Matrix Voice in there. Might be a good starter

[createcandle]

What broker are you using?

1613744295: mosquitto version 1.5.7 starting
1613744295: Using default config.

This is on Raspbian (WebThings 1.0 image), on a Pi4.

I have indeed been looking at the version 6.0 of your code, although I think you mentioned somewhere that it won't work on the Atom Echo.

The version of the code I just uploaded is stable, in the sense that it doesn't crash. However, it does this by continuously sending out x instead of actual audio data.

            //audioServer.publish(audioFrameTopic.c_str(),(uint8_t *)payload, sizeof(payload));
            audioServer.publish(audioFrameTopic.c_str(),"x", 1);

I do not think an arduino version is going to allow you upload the sketch to the candle addon manager.

I'm the creator of the addon, so I hope it will work. Does the Arduino CLI not support uploading to ESP32? I would probably have to remove loading the config from SPIFFS to keep things simpler, but that seems doable. What issues do you see?

[Romkabouter]

I have indeed been looking at the version 6.0 of your code, although I think you mentioned somewhere that it won't work on the Atom Echo.

Indeed, but it might give you a good start if you need it :)
If you already have this code into an ino file that's great.

If your version works, there is some issue with the actual audio. Can you verify the MQTT_MAX_PACKET_SIZE in your load_settings.py is set to 2000?
This is needed because the default is too small to hold the wave data

I do not know all the details of the addon, does the addon upload a sketch to the esp32? And how does this connect to Voco?
Maybe you can change you addon so that it can load bin files as well?

[createcandle]

Candle Manager allows users to customise normal arduino code through a UI. It also automatically manages Arduino libraries. In the end, all users have to do is use a UI. There is a walkthrough here:

https://www.candlesmarthome.com/candle-manager-tutorial

Currently it's geared torwards the Arduino Nano, since Candle devices use that as their basis.

I started looking into creating a privacy focussed smart doorbell, and that's what lead me to your project, and the discovery of the Atom Echo. Privacy friendly voice control benefits from having cheap small satellites. Voco already supports using Raspberry Pi (Zero) devices as additional satellites. But having tiny plug-and-play satellites is a very enticing proposition, so I had to look into it :-)

[Romkabouter]

So back here ;)

I'd like to test this, I have a spare Pi. I can install the image you mentioned, and then I have to install Candle Manager?

[Romkabouter]

I got WebThings running with Voco addon, but now I am stuck.
Where can I set a remote device? I can only select attached device or build in.

Can you help me configure? Where did you run the Snips Watch command?

edit: I am running the esp32 version now in the Echo and I am connected with MQTT to the ip of webthings.
I do not see my satellite on Voco, I have enabled ssh in the developer section.

[Romkabouter]

Not there yet, but the snips detects the audio stream.
I am now checking on the ping/pong system of voco, I think that is used by the addon to detect satellites.
Uploading a sketch might also become a challenge, because when I plugged in the Atom in the Pi, it was not detected by the Candle Manager. But that is not the first step, the first step is to be able to use the Atom as a Voco satellite :)

[Romkabouter]

After doing some study, I do not think voco support satellites like this.
The main issue is that voco does not follow the hermes protocol, to be more specific, it does not publish to the playBytes topic but rather does some local play.
Therefore, no finished message is published and you get the timeout on the dialogue manager.
I was able to trigger the hotword like you did (in the other issue), but also got the timeout and did a scan on all topics of the broker.

[Romkabouter]

Hi @createcandle, I have checking the code for Voco addon.
This software does not follow the hermes protocol at all.
I could trigger the hotword, and the snips engine started listening. But the addon expects a satellite to be a voco addon as well. It send a publish to hermes/voco/play with the name of the file, which is then played on the voco satellite (which has that file on the system as wel)
This in incorrect, because it should be sending the wave data over MQTT to playBytes.

Due to this design limitations, it take a great deal of effort to get is working. Effort which, in my opinion, could be better spent by implementing the hermes protocol correctly and switching over to Rhasspy. Or maybe even keep snips, but implement the protocol correctly and completely

I see you already have a repo open, so I wish you the best of luck with it but I will have to close this issue for the above stated reason.

[createcandle]

Sorry I missed all this, I wasn't logged into this account.

I do not see my satellite on Voco

Correct, it won't show up because it doesn't have the extra voco commands. But it could in theory still work with snips. It just won't show up in the UI.

I am now checking on the ping/pong system of voco, I think that is used by the addon to detect satellites.

Yeah all that won't be necessary. It just means that they are not shown in the UI.

Not there yet, but the snips detects the audio stream.

Yes that's how far I had gotten. It just didn't seem the ASR could understand the audio , likely being because of the encoding of the wav?

Can you help me configure? Where did you run the Snips Watch command?

I hope I was able to explain that from the other account a bit. It's placed in ~/.webthings/addons/voco/snips/.

Uploading a sketch might also become a challenge, because when I plugged in the Atom in the Pi, it was not detected by the Candle Manager.

Yes, I would have to add ESP32 support.

The main issue is that voco does not follow the hermes protocol, to be more specific, it does not publish to the playBytes topic but rather does some local play.

I know :-) I'm the creator of both Voco and Candle Manger. I would have to add that in, and make it follow the Hermes protocol more. This would have some advantages. For now I would be happy just being able to launch a command. The return audio channel would be a step after that.

Therefore, no finished message is published and you get the timeout on the dialogue manager.

I know. But in theory that shouldn't matter, since feedback sounds can be disabled. I believe the suggestion was to disable these feedback sounds.

But the addon expects a satellite to be a voco addon as well.

At the moment, yes. And only to play sounds. The extra Voco stuff is built on top of a normal Snips installation that doesn't output it's own sounds. So the Voco stuff should theoretically not matter?

This in incorrect, because it should be sending the wave data over MQTT to playBytes.

I wouldn't call it incorrect. It's just that the return channel has been disabled.

Due to this design limitations, it take a great deal of effort to get is working. Effort which, in my opinion, could be better spent by implementing the hermes protocol correctly and switching over to Rhasspy. Or maybe even keep snips, but implement the protocol correctly and completely

Indeed. Implementing the protocol with a MQTT return channel for audio could be instated. Doing so would be worth it if there was an indication that these Atom Echo satellites could work well. But for that I wanted to see if I could get Snips to recognise a command first. It felt like a logical first step.

From my tests it seems that it shouldn;t matter that Voco currently doesn't have a return channel. Snips was able to detect the hotword in some rare situations. From looking at the messages, it seems that the lack of a return channel wasn't the issue. The issue was that when the ASR was listening to the stream, it didn't recognise anything. This could be because of the structure of the WAV data.

So these were my intended steps:

1. Get hotword detected. If that works:
2. Get voice command recognised. If that works:
3. Implement audio return channel. If that works:
4. Implement ability to upload via Candle Manager.

Voco may at some point in the future switch over the Rhasspy. I wasn hoping you could help me with creating a version that has the Snips wav implementation. But I guess another option is to leave this project for now, and get back to it if/when Rhasspy has been implemented.

Either option is fine. Thank you for looking into it, I already appreciate that.

[Romkabouter]

Good steps, I will post updates in the open issue for this :)
I will try version 6.0 on my Matrix Voice with Voco, since that was working with Snips.

If that works, I need to take some steps back to see what went incompatably along the way. I will keep you posted, good to hear you are the developer of both packages :)

[createcandle]

Thanks, that's a good idea. If that works, then that may narrow it down.