[BUG] HSv5 accepts HSv4 connection with bad passphrase

[nhaleft]

Describe the bug
Versions of SRT that use HSv5 blindly accept incoming connections using HSv4. This behavior is contrary to what is described in SRT's documentation.
This is relevant because the version of SRT used in VLC appears to use HSv4.
Thus, connecting to an encrypted SRT socket from VLC with the wrong passphrase results in the connection being accepted, and VLC receiving encrypted packets that it cannot decrypt. Additionally, this blocks other incoming connections that may have the correct passphrase.

To Reproduce
Steps to reproduce the behavior:

1. Run srt-live-transmit w/ HSv5 (v1.4.2 is what we used) in listener mode with a passphrase and enforcedencryption
2. With VLC (or another srt-live-transmit process w/ HSv4), connect to the SRT process made in step 1 with an incorrect passphrase.

Expected behavior
I expect the SRT socket using HSv5 to reach the SRT_KM_S_BADSECRET state and reject the incoming HSv4 connection.

Desktop (please provide the following information):

• OS: listener=CentOS 7.6 and VLC caller on macOS Catalina Version 10.15.5
• SRT Version / commit ID: listener=v1.4.2, caller<=1.2.3 (whichever SRT version VLC v3.0.10 supports)
• VLC Version: 3.0.10

Additional context
If this is an irreconcilable difference between HSv4 and HSv5, then please add this case to the documentation so I can point to it when I make a request to VLC to support newer versions of SRT.
If a newer version of VLC has already provided support for newer versions of SRT (couldn't find any info on this in VLC's docs), then please let me know.

Attachments
I have a pcap from the listening SRT process, but it's larger than 10MB when gzipped so I can't attach it here. If you think it would be useful I can trim it or upload it elsewhere.
I have attached a debug log from the listener as well.
debug_log.tar.gz

[maxsharabayko]

HSv4 does not have the information about the key to be used (Key Material message as HS extension). Instead, a separate control packet with the key is expected to arrive later on. Therefore, there is no way to determine if the peer has the correct key at the stage of handshaking to reject the connection.

It could be determined later, waiting for several seconds for the follow-up KM message and breaking the connection if none arrives. But it would complicate the logic and bring some unclear heuristics in the process.

A corresponding note on this limitation needs to be added to the docs.

[ethouris]

The SRTO_ENFORCEDENCRYPTION option is a new option introduced likely after 1.3.2 version and only since this version it is default. Earlier versions simply didn't have it, however they behaved as if this option was off.

Also in HSv4 supporting versions there wasn't any possibility to reject a connection at all. Therefore, the listener will not reject the connection no matter what, and this cannot be fixed now with the old version.

This applies also to newer versions, unfortunately, because the connection mechanism with HSv4 is different than with HSv5 and if at least one side doesn't support HSv5, the connection will be done in the frames of HSv4, and this procedure doesn't support rejection as well.

There possibly could be made something to change it in a newer version, but the problem is that the HSv4 handshake does only a "bare" handskaking - doesn't exchange any data beside the basic ones needed for connection, which means that the encryption state is not even known until the connection is complete and therefore there is no reason to reject a connection whatsoever. Worseover, at the moment when this parameter is known, the terms of "caller" and "listener" no longer exist, only "sender" and "receiver".

Theoretically it could be possible that in HSv5 agent (no matter if listener or caller) that has been connected using HSv4 due to requirements of the peer, the RECEIVER that is receiving packets, they are encrypted, the encryption phase has been finished,
and the agent is unable to decrypt them, could immediately break the connection. Possible, but due to the following reasons this is a bad idea:

• The SENDER if it is a HSv5-capable version (be it listener or caller) will be unaware of the inability of the receiver to decrypt the packets and there's no way to make it aware anyhow because the peer receiver is the old version that didn't predict that case
• In case when password is wrong, but has the right length, the decryption attempt doesn't fail, just declares successful decryption and returns gibberish. Therefore relying on decryption failure isn't reliable.

I personally think that the only thing that could be done to make things better is that SRTO_ENFORCEDENCRYPTION should be updated with a description as to how things work for versions before 1.3.0 and also against versions between 1.3.0 and the version when this option was introduced.

[jeandube]

The SRT sender knows an HSv4 receiver does not have the right passphrase from the KMrsp message. It cannot reject the connection in the handshake but can close the established connection in that case, limiting the period of packets transmitted to RTT, and freeing the port to permit a legit peer to connect and receive stream.

[ethouris]

Ah, ok right - except the fact that UMSG_EXT messages are not granted delivery, so it can't be done 100% reliably. And still the case of a password of correct length is true.

Actually the idea behind having SRTO_ENFORCEDENCRYPTION true by default was to not allow a listener that is password-protected receive an unencrypted stream. But in order to enforce that protection in full length the application should as well set the minimum version (SRTO_MINVERSION) to 1.3.0, in which case 1.2.0 version SRT will be rejected always. This is the only way how it can be fully protected because if you allow connections from 1.2.0 callers, you must allow a "bare" handshake that is unknown as to whether it will ever send KMX messages (which can be also send AFTER some data packets). At best it could be done possibly something like that it should wait up to 1 second since the first data packet and:

• if no KMX message was received in this time, break the connection
• if KMX processing failed, break the connection
• At the first data packet received without encryption flags set, break the connection

Not sure if this is a good idea, but maybe better than it is now.