Set OpenSSL MTU to avoid Edge "DTLS handshake failed"

[michaelfig]

Quick-and-dirty way to get Edge to be able to negotiate DTLS successfully. Otherwise it fails with:

ORTC18615: ORTC RTCDtlsTransport: DTLS handshake failed. hr=c004e00f.

See: https://mta.openssl.org/pipermail/openssl-users/2015-June/001506.html for OpenSSL implementation information. We could implement a BIO filter, but can't crib the code from Janus because it's GPLv3.

I still don't have Edge working in my project, but at least it doesn't give an error anymore.

Thanks,
Michael.

[michaelfig]

Correction: I can stream from Edge's getUserMedia to other browsers, but not with video.captureStream() (which is what I'd expect... Edge doesn't support it). Still can't get Edge to display a stream from itself or other browsers, even with the DTLS issue solved.

[ibc]

Thanks. I cannot yet test Edge (no Windows machine), however I've been able to test Edge using an online service and indeed I get DTLS handshake failed. hr=c004e00f.

Some questions:

• Have you took a look to DTLS handshake fragmentation #4? Well, it's basically the same solution, am I right?
  • Note that mediasoup uses OpenSSL 1.1.0 so the issue in Crash due OPENSSL_assert(s->d1->mtu >= dtls1_min_mtu(s)); openssl/openssl#346 is fixed.
• Why MTU 1200? Isn't it too small? I do know that Chrome/libwebrtc internally assumes MTU 1200 for WebRTC media, but wouldn't 1350 work as well? Have you tested it?

[ibc]

Regarding the Edge issue related to received media (Consumer), please check #192 and comment there. I think it may be 100% related.

[michaelfig]

• Have you took a look to DTLS handshake fragmentation #4? Well, it's basically the same solution, am I right?

Yes, you are right. I don't know why I didn't search the issues (I thought I had)... was looking in the source code for mtu, but didn't find it.

• Note that mediasoup uses OpenSSL 1.1.0 so the issue in openssl/openssl#346 is fixed.

Okay, great.

• Why MTU 1200? Isn't it too small? I do know that Chrome/libwebrtc internally assumes MTU 1200 for WebRTC media, but wouldn't 1350 work as well? Have you tested it?

I just used what Janus did for the magic number. 1350 works fine, in my testing.

[ibc]

mediasoup 2.2.4 released