Unitytls integration #784

AndreasReich · 2018-01-08T12:11:07Z

Provides an alternative TLS binding that will allow us to use TLS functionality on all unity platforms.

Use/Activation
Binding must be set from outside via native call (mono_unity_install_unitytls_interface). If this is done before Mono registers its TLS providers, unitytls will be the only available provider. Note that this registration mechanism provides only unitytls functionality that is actually used.
The unity side of this binding can be found in the unity branch platform/foundation/tls-module-monointegration

Support
Binding itself is fairly straight-forward and should provide support for everything that the two other main tls implementions (btls, appletls) support as well. However, there are most likely differences in cipher support, error reporting and robustness of loading data.

Testing
Black-box tests via the user facing SslStream are part of the change on the unity sided binding since currently that is the only place where unitytls is available. These tests show that with our TLS implementation, we have at least better TLS support than mono's current AppleTls implementation (in terms of TLS support and behavior being more similar to .Net)
Earlier versions that relied on a different mechanism for native calls, passed Mono's very shallow SslStreamTest.
There are currently no tests against Mono.Security which should work as well but has low priority for us.

Changed namespaces from UnityTls to Unity, change casing from 'TLS' to 'Tls'

…tr to Int8=byte ptr since char has a different meaning in C# than in C

…esult as well

…pdated UnityTls interface with trace methods

…s for UnityTlsContext

CheckAndThrow for verify results forwards now to the one without verify result.

…eeded.

…rors

…precaution

Development up until this point happened on unity-2017-02-staging

… test (only activated for unity)

Note that previous versions of mono would directly pass on whatever exception the implementation throws. Now, not only is TlsException wrapped with the correct AuthentificationException but - since it is run form a task - it is passed on as an AggregateException

(non-breaking change)

…ificates

…removing the need for unitytls_x509list_get_size

…ead of function call

Adds code complexity with little gain. With native interface being a property this also feels more natural now.

…nd simplified branch-flow for UnityTls support

All additional tests are now done from unity to ensure that unitytls inkl binding to unity is working properly

AndreasReich · 2018-01-08T16:50:07Z

mcs/class/System/Mono.UnityTls/UnityTlsContext.cs

+		public override void FinishHandshake ()
+		{
+			// Query some data. Ignore errors on the way since failure is not crucial.
+			var cipherSuite = UnityTls.NativeInterface.unitytls_tlsctx_get_ciphersuite(m_TlsContext, null);


passing null for the errorstate is soon no longer supported. need to change this

…retrieving information about the connection from tlsctx. Passing null for the errorstate is no longer supported in an upcoming unitytls version.

bitter · 2018-01-08T17:50:19Z

mcs/class/System/Mono.UnityTls/UnityTls.cs

+
+    unsafe internal static partial class UnityTls
+    {
+        private const string DLLNAME = "MacStandalonePlayer_TLSModule_Dynamic.dylib";


bitter · 2018-01-08T19:00:00Z

mcs/class/System/Mono.UnityTls/UnityTlsContext.cs

+
+		public override void StartHandshake ()
+		{
+			// TODO: Client->Server authentification is not supported by UnityTls as of writing


should we fix this before pushing to master?

Is this feature fully functional and tested in UnityTls by now. If so, clearly I should add the necessary wiring and another integration test on the unity side of this PR!

Together with SNI. I wonder how SNI is supported in Mono..

Done and tested.

bitter · 2018-01-08T19:05:39Z

mcs/class/System/Mono.UnityTls/UnityTlsContext.cs

+
+			// .Net implementation gives the server a verification callback (with null cert) even if AskForClientCertificate is false.
+			// We stick to this behavior here.
+			if (IsServer && !AskForClientCertificate) {


I don't think we have decided or even confirmed what the behavior is for unitytls in that regard. Feels like we at least need a conformance test to establish behavior.

I can look into that while wiring Client->Server auth to mono

bitter · 2018-01-08T19:09:49Z

mcs/class/System/Mono.UnityTls/UnityTlsContext.cs

+				}
+
+				return bufferLen;
+			} catch { // handle all exceptions since we don't want to let them go through native code.


I wonder if this is really okay... I imagine this is a fairly common error. Like dropping a network connection or some other io failure. In which case we will hide the real reason for the failure.

Don't think it is great either. But if we jump out via exception here, the unitytls impl ends up in an invalid state. I'd call that worse. I guess we could maybe always "cache" the last exception and throw it when we're outside of the read/write/verify block (whenever cached exception is non zero and we registered an error in handshake/read/write). What do you think?

How do the other TLS implementations handle this?

Turns out AppleTls does exactly what I described. So I better get on with it :)

Done. Also added tests for user exception forwarding

joncham · 2018-01-09T04:36:48Z

mcs/class/System/Mono.UnityTls/UnityTls.cs

+namespace Mono.Unity
+{
+    using UInt8 = Byte;
+    using Int8 = Byte;


You like want SByte here.

I intentionally put Byte here since what we get on the C# side from encoding and marshaling functions are always byte[]
Unitytls uses UInt8 to denote raw buffers and Int8/char for strings. Since we need to push all in and outgoing strings through Encoding it is easier to pass byte* along. The aliases here are just there to keep the semantic in the interface and make it more similar to the c original.

I guess I should add a comment explaining that!

joncham · 2018-01-09T04:37:08Z

mcs/class/System/Mono.UnityTls/UnityTls.cs

+    using Int8 = Byte;
+
+    [StructLayout (LayoutKind.Sequential)]
+    internal struct size_t


Not blocking, but does this size_t provide enough value over just using IntPtr?

It's mostly about the implicit cast to and from int. Since size_t is used for buffer sizes / array length, it would require quite a few explicit casts

joncham · 2018-01-09T04:41:16Z

mcs/class/System/Mono.UnityTls/UnityTlsContext.cs

+			try {
+				if (m_WriteBuffer == null || m_WriteBuffer.Length < bufferLen)
+					m_WriteBuffer = new byte[bufferLen];
+				Marshal.Copy ((IntPtr)data, m_WriteBuffer, 0, bufferLen);


Does this need to be thread safe?

There will be always only a single WriteCallback per context. So this should be safe :)

…n properly.

Fixed RemoteCertificate property always being null

AndreasReich · 2018-01-11T09:50:40Z

Quite happy with the latest improvements yesterday. I ramped up test coverage on the Unity side (in platform/foundation/tls-module-monointegration) and I'm quite happy how this turns out.
unity-tls api is stabilizing as well, so if none of you has concerns we're good to go.
Until the unity sided call is in place, nothing can break anyways ;)

joncham · 2018-01-11T17:41:52Z

mcs/class/System/Mono.UnityTls/UnityTls.cs

+        public static bool IsSupported()
+        {
+            try {
+                return NativeInterface != null;


It seems wasteful to try/catch in here, although I guess this is only called once. Any reason this can't be a little cleaner? Perhaps NativeInterface can check result of mono_unity_get_unitytls_interface() and return null if that is null rather than always doing the Marshal.PtrToStructure call and raising exception?

My idea here was that I would also catch any exception thrown from Marshal.PtrToStructure. Looking at its possible exceptions again now I realize however that this should never happen so I agree with your suggestion. Will change accordingly!

joncham · 2018-01-11T17:42:33Z

mcs/class/System/Mono.UnityTls/UnityTls.cs

+        }
+
+        [DllImport("__Internal")]
+        private static extern IntPtr mono_unity_get_unitytls_interface();


This should likely be an internal call and connected via the normal mono icall mechanism for class libraries.

As is, we need to be sure to export this symbol across all platforms which is not always easy.

Switched it to icall: 65af3e7
Completely fresh build(.zip) from katana still passes my tests so I guess I did everything right there? :)

…g but on a simple null check

…e_struct Change due the fact that the interface is soon to be used in other places.

AndreasReich added 30 commits December 15, 2017 16:28

[unitytls] Added UnityTLS interface defintion

d773c76

[unitytls] Added UnityTlsProvider stub + registration.

90d29d7

Changed namespaces from UnityTls to Unity, change casing from 'TLS' to 'Tls'

[unitytls] Added stubs for UnityTlsContext/Stream

5ce647e

[unitytls] Activate SslStreamTest on Mac if we're building for unity

b1e0d68

[unitytls] size_t is now a struct instead of an alias. Change char* p…

4d49d8b

…tr to Int8=byte ptr since char has a different meaning in C# than in C

[unitytls] Implemented UnityTlsProvider.ValidateCertificate

8f1c80b

[unitytls] fixed access modifies for some structs in UnityTls interface

5b0f3c4

[unitytls] Added debug helper for handling unitytls_errorstate

1b25fe8

[unitytls] Implemented UnityTlsContext init

598e0d4

[unitytls] added checking method that evaluates unitytls_x509verify_r…

98e940d

…esult as well

[unitytls] size_t is now IntPtr and can convert to int implicitely. U…

0b84cf2

…pdated UnityTls interface with trace methods

[unitytls] Implemented tracing, basic handshake and read/write method…

74d5592

…s for UnityTlsContext

[unitytls] Cleaned up CheckAndThrow methods.

f7dac9e

CheckAndThrow for verify results forwards now to the one without verify result.

[unitytls] Implemented verify callback

ad45770

[unitytls] Use of CertHelper in TlsProvider

f77b0c5

[unitytls] Mono coding style compliance fixes

f6c2b75

[unitytls] Removed full mapping of ciphersuite enum since it is not n…

cfa1e04

…eeded.

[sslstreamtest] Added check for correct certificates and no policy er…

445e584

…rors

[sslstreamtest] a few renamings for clarification

594e2ac

[unitytls] Added exception handling for native callbacks as a safety …

ca57238

…precaution

[unitytls] implemented data read/write

af91c35

[unitytls] Filling out MonoTlsConnectionInfo now

5bb3eb6

[unitytls] moved conversion functions into separate file

9ea74ae

[unitytls] Updated interfaces to newer mono version.

22b6ae9

Development up until this point happened on unity-2017-02-staging

[sslstreamtest] Added separate test suite for BadSsl authentification…

6ecb133

… test (only activated for unity)

[unitytls] unitytls api update - changed callback names

af58b51

(non-breaking change)

[unitytls] removed unused & unnecessary handling of intermediate cert…

4a0611c

…ificates

[unitytls] Fixed path casing for a couple of files

d282ba9

[unitytls] use der instead of pem in NativeChainToManagedCollection

01d462a

AndreasReich added 9 commits January 5, 2018 18:13

[unitytls] mono_unity_unitytls_interface exposes now invalid_handle, …

b990700

…removing the need for unitytls_x509list_get_size

[unitytls] Removed calls to UnityTls.GetInterface

a47260d

[unitytls] Making use of Settings.EnabledCiphers now

b1a8eca

[unitytls] Native interface is retrieved via static property now inst…

589f130

…ead of function call

[unitytls] Removed local caching of unitytls native interface again.

ca2058f

Adds code complexity with little gain. With native interface being a property this also feels more natural now.

Merge branch 'unity-master' into unity-master-staging-unitytls

0a1e777

[unitytls] Removed commented debug output in MonoTlsProviderFactory a…

d90ff2f

…nd simplified branch-flow for UnityTls support

[unitytls] Reverted remaining changes in system tests

f9e79da

All additional tests are now done from unity to ensure that unitytls inkl binding to unity is working properly

[unitytls] added comment explaining use of mono_unity_unitytls_interface

9889bec

AndreasReich assigned bitter and joncham Jan 8, 2018

AndreasReich mentioned this pull request Jan 8, 2018

UnityTls install dummy function for old mono #785

Merged

AndreasReich commented Jan 8, 2018

View reviewed changes

[unitytls] FinishHandshake no longer passes null for errorState when …

a5df54c

…retrieving information about the connection from tlsctx. Passing null for the errorstate is no longer supported in an upcoming unitytls version.

bitter approved these changes Jan 8, 2018

View reviewed changes

joncham reviewed Jan 9, 2018

View reviewed changes

AndreasReich added 6 commits January 9, 2018 10:29

[unitytls] Removed leftover from previous p/invoke based integration.

fe8bf68

[unitytls] User exceptions from read/write/handshake are now passed o…

1165e17

…n properly.

[unitytls] Commented on UInt8/Int8 aliases

8f7503b

[unitytls] Implemented client->server authentification

02aa179

Fixed RemoteCertificate property always being null

[unitytls] Update of error code to newest version

3843d8e

[unitytls] Mono convention compliance fixes

601111c

joncham reviewed Jan 11, 2018

View reviewed changes

AndreasReich added 3 commits January 12, 2018 11:05

[unitytls] UnityTls.IsSupported no longer relies on exception handlin…

52e347b

…g but on a simple null check

[unitytls] mono_unity_get_unitytls_interface is now called via icall

65af3e7

[unitytls] Renamed mono_unity_unitytls_interface to unitytls_interfac…

5a28379

…e_struct Change due the fact that the interface is soon to be used in other places.

joncham approved these changes Jan 16, 2018

View reviewed changes

AndreasReich merged commit cdaebd8 into unity-master Jan 17, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unitytls integration #784

Unitytls integration #784

AndreasReich commented Jan 8, 2018

AndreasReich Jan 8, 2018

AndreasReich Jan 8, 2018

bitter Jan 8, 2018

bitter Jan 8, 2018

AndreasReich Jan 9, 2018

bitter Jan 9, 2018

bitter Jan 9, 2018

AndreasReich Jan 10, 2018

bitter Jan 8, 2018

AndreasReich Jan 9, 2018

AndreasReich Jan 10, 2018

bitter Jan 8, 2018

AndreasReich Jan 9, 2018

bitter Jan 9, 2018

AndreasReich Jan 9, 2018

AndreasReich Jan 10, 2018

AndreasReich Jan 10, 2018

joncham Jan 9, 2018

AndreasReich Jan 9, 2018

joncham Jan 9, 2018

AndreasReich Jan 9, 2018

joncham Jan 9, 2018

AndreasReich Jan 9, 2018

AndreasReich commented Jan 11, 2018

joncham Jan 11, 2018

AndreasReich Jan 12, 2018

AndreasReich Jan 12, 2018

joncham Jan 11, 2018

joncham Jan 11, 2018

AndreasReich Jan 12, 2018

Unitytls integration #784

Unitytls integration #784

Conversation

AndreasReich commented Jan 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreasReich commented Jan 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment