how to enable two way audio? #174

Maxmudjon · 2018-04-02T17:11:48Z

No description provided.

DanielWeeber · 2018-07-17T07:31:18Z

+1
//another edit: seems like I cannot test it with my camera. please test is with yours, see my fork https://github.com/DanielWeeber/homebridge-camera-ffmpeg

DanielWeeber · 2018-07-19T15:33:27Z

Unfortunately this does not seem to work. I bought a new camera to test this. Via the app of the manufacturer its working, not via homebridge.

I found something interesting here:

https://github.com/thoukydides/homebridge-skybell/issues/9

llemtt · 2018-07-23T15:51:42Z

How do you managed to instruct the homebridge client that camera has a speaker and then start stream to the camera?

DanielWeeber · 2018-07-23T19:23:20Z

If you mean try it yourself and use it for your camera:
Just set "audio" to "true" in your config.
Check README for that.

https://github.com/KhaosT/homebridge-camera-ffmpeg/blob/master/README.md
{
"platform": "Camera-ffmpeg",
"cameras": [
{
"name": "Camera Name",
"videoConfig": {
"source": "-re -i rtsp://myfancy_rtsp_stream",
"stillImageSource": "-i http://faster_still_image_grab_url/this_is_optional.jpg",
"maxStreams": 2,
"maxWidth": 1280,
"maxHeight": 720,
"maxFPS": 30,
"maxBitrate": 200,
"vcodec": "h264_omx",
"audio": true,
"packetSize": 188,
"debug": true
}
}
]
}

If you mean from a programming perspective:
For "Camera has a microphone":
https://github.com/KhaosT/homebridge-camera-ffmpeg/blob/master/ffmpeg.js#L382

For "Camera has a microphone and speaker":
https://github.com/DanielWeeber/homebridge-camera-ffmpeg/blob/master/ffmpeg.js#L382

llemtt · 2018-07-24T10:26:24Z

Thanks Daniel

now I have audio input active, do you mean that I have to use your version to try to send audio to webcam also?

I see that the speaker service is created only in your version.

DanielWeeber · 2018-07-26T16:35:40Z

That is correct. If you use my version the icon will show up, but it will not work. That is exactly what we are discussing here.. ;)
We need to utilize a third ffmpeg stream or a RTSP backchannel. I'm not sure.

llemtt · 2018-07-30T14:11:48Z

Current version of ffmpeg won't let you bind to the same udp port, so first of all we have to put in place an udp proxy to act as a gateway between the port used to talk with the homekit client and two local ports to be used with ffmpeg.

Maybe I'm going to try that next days.

DanielWeeber · 2018-08-01T09:09:17Z

Would be nice. I'm not an expert with ffmpeg at all.

llemtt · 2018-08-01T19:39:06Z

Ok now I have a "proof of concept" working!

I implemented the udp proxy part and managed to receive the "speaker" stream with a separate ffmpeg instance and save to a file...

Tomorrow I'll try to find time to little polish the code and attach here.

llemtt · 2018-08-06T20:17:14Z

Here is my working POC

ffmpeg.js.txt

Set config["audio"] = "2way", it create the proxy and a "speaker.sdp" file in working directory that let's you play the audio sent from iOS.

I saved mine to a file like this:

ffmpeg -v trace -protocol_whitelist "file,udp,rtp" -i speaker.sdp speaker.mp4

DanielWeeber · 2018-08-07T07:53:39Z

Nice work. As soon as this is finished I am happy to test it!

brownad · 2018-08-08T11:02:39Z

Is the idea to handle separate input and output audio streams from the video? That’s exactly what I’d like to achieve - then you’d have 2way audio working @llmett

brownad · 2018-08-08T11:04:27Z

If so I’d like to bring the ideas across to get the intercom working fully. That would be a win 🏖

Maxmudjon · 2018-08-25T10:37:02Z

@llemtt give me your image of raspberry

llemtt · 2018-08-28T14:49:36Z

Attached second version of my working POC!

ffmpeg.js.txt

It uses an additional ffmpeg instance to manage the speaker, set config["audio"] = "2way "+{your speaker ffmpeg output} (I've tested mine writing to a file with "2way -y speaker.mp4").

Requires audio streaming to ios device to start in less than 10 seconds otherwise the "speaker" ffmpeg instance will timeout.

On raspberry tested working also with "2way -f alsa default".

brownad · 2018-08-28T15:45:08Z

Is the expected end state to pass it a stream to write it to rather than a file?

llemtt · 2018-08-29T07:56:44Z

Yes, at the moment I don't have a way to try it out other than sending to a file, alsa, or a DLINK camera having a complicated proprietary format to solve first 😒 (btw I'm looking for a "cheaper than doorbird" doorbell with a decently working streaming api...)

It would have been better to use a single ffmpeg instance but it's a nightmare because without a steady streaming to speaker (which is not because it's on only when the microphone button is active) it blocks every other input waiting. This could be a nice improvement in the "not so near" future.

Maxmudjon · 2018-11-10T22:52:52Z

ERROR: Speaker FFmpeg exited with code 1

{
"platform": "Camera-ffmpeg",
"cameras": [
{
"name": "Home 2way",
"videoConfig": {
"source": "-rtsp_transport tcp -y -i rtsp://192.168.1.17/unicast",
"maxBitrate": 1600,
"maxStreams": 2,
"maxWidth": 1280,
"maxHeight": 720,
"vcodec": "h264_omx",
"audio": "2way -f alsa default"
}
}
]
}

llemtt · 2018-11-12T09:22:14Z

Can you add "debug": true to your config and get the ffmpeg output into the log?

BTW you have to configure alsa to work under the user running homebridge.

Maxmudjon · 2018-11-12T12:48:26Z

[11/12/2018, 12:48:00 PM] [Camera-ffmpeg-2way] { sessionID: <Buffer 3b a9 fc 5c 74 36 43 c4 b9 f8 8f f0 89 24 f9 44>,
type: 'start',
video:
{ profile: 2,
level: 2,
width: 1280,
height: 720,
fps: 10,
ssrc: 434345868,
pt: 99,
max_bit_rate: 299,
rtcp_interval: 1056964608,
mtu: 1378 },
audio:
{ codec: 'AAC-eld',
channel: 1,
bit_rate: 0,
sample_rate: 16,
packet_time: 30,
pt: 110,
ssrc: 4201289906,
max_bit_rate: 24,
rtcp_interval: 1084227584,
comfort_pt: 13 } }
[11/12/2018, 12:48:00 PM] [Camera-ffmpeg-2way] Start streaming video from Home 2way with 1280x720@299kBit
ffmpeg -nostats -nostdin -rtsp_transport tcp -y -i rtsp://192.168.1.17/unicast -map 0:0 -vcodec h264_omx -pix_fmt yuv420p -r 10 -f rawvideo -tune zerolatency -vf scale=1280:720 -b:v 299k -bufsize 299k -maxrate 299k -payload_type 99 -ssrc 5825497 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params ZIZnVdVJjKkFLBDvrf8VGtD85/p2MhV2tgNH/ydF srtp://192.168.1.10:60331?rtcpport=60331&localrtcpport=60331&pkt_size=1316 -map 0:1 -acodec libfdk_aac -profile:a aac_eld -flags +global_header -f null -ar 16k -b:a 24k -bufsize 24k -ac 1 -payload_type 110 -ssrc 14017304 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params pifjylvtto/qoTs5uefgoxKtpzk88Zd2t1De1gNz srtp://127.0.0.1:61436?rtcpport=61436&localrtcpport=9998&pkt_size=188
CONFIGDAN: 2way -y alsa default ***** -y alsa default
[Speaker] ffmpeg -v error -nostats -nostdin -max_ts_probe 0 -protocol_whitelist file,udp,rtp -i Home_2way_speaker.sdp -y alsa default
proxy listening 0.0.0.0:61436
ffmpeg version N-92398-g10bc4c3a7d Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516
configuration: --prefix=/usr/local --arch=armel --target-os=linux --enable-omx-rpi --enable-nonfree --enable-gpl --enable-libfdk-aac --enable-mmal --enable-libx264 --enable-decoder=h264 --enable-network --enable-protocol=tcp --enable-demuxer=rtsp
libavutil 56. 23.101 / 56. 23.101
libavcodec 58. 39.100 / 58. 39.100
libavformat 58. 22.100 / 58. 22.100
libavdevice 58. 6.100 / 58. 6.100
libavfilter 7. 43.100 / 7. 43.100
libswscale 5. 4.100 / 5. 4.100
libswresample 3. 4.100 / 3. 4.100
libpostproc 55. 4.100 / 55. 4.100

[Speaker] [NULL @ 0x25a9df0] Unable to find a suitable output format for 'alsa'
alsa: Invalid argument

[11/12/2018, 12:48:00 PM] [Camera-ffmpeg-2way] ERROR: Speaker FFmpeg exited with code 1
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, rtsp, from 'rtsp://192.168.1.17/unicast':

Metadata:
title : LIVE555 Streaming Media v2014.07.04
comment : LIVE555 Streaming Media v2014.07.04
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 1280x720, 40 tbr, 90k tbn, 180k tbc
Stream #0:1: Audio: pcm_alaw, 8000 Hz, mono, s16, 64 kb/s

Codec AVOption tune (Tune the encoding params (cf. x264 --fullhelp)) specified for output file #0 (srtp://192.168.1.10:60331?rtcpport=60331&localrtcpport=60331&pkt_size=1316) has not been used for any stream. The most likely reason is either wrong type (e.g. a video option with no video streams) or that it is a private option of some encoder which was not actually used for any stream.

Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_omx))
Stream #0:1 -> #1:0 (pcm_alaw (native) -> aac (libfdk_aac))

Output #1, rtp, to 'srtp://127.0.0.1:61436?rtcpport=61436&localrtcpport=9998&pkt_size=188':
Metadata:
title : LIVE555 Streaming Media v2014.07.04
comment : LIVE555 Streaming Media v2014.07.04
encoder :
Lavf58.22.100
Stream #1:0: Audio: aac (libfdk_aac) (ELD), 16000 Hz, mono, s16, 24 kb/s
Metadata:
encoder : Lavc58.39.100 libfdk_aac