[Mono.Android] NTLM support in AndroidMessageHandler

[simonrozsival]

This PR contributes to dotnet/runtime#62264

I attempted to implement the authentication as AuthModules similarly to the Basic and Digest modules first, but it wasn't very suitable for the challenge/response auth schemes. I tried replicating SocketHttpHandler's AuthenticationHelper.NtAuth.cs in the new NegotiateAuthenticationHelper static class. NTLM/Negotiate support is available only for .NET 7+ because it uses its new managed implementation of NTLM/Negotiate and it uses a new public class NegotiateAuthentication.

This feature has dependencies on several dlls that aren't used by most apps so the code is behind a feature flag $(AndroidUseNegotiateAuthentication)/Xamarin.Android.Net.UseNegotiateAuthentication. If this flag isn't set to true, the helper class and all its dependencies are linked away.

Any suggestions for the implementation itself and for the feature flag documentation are welcome. I'm aware that the rudimentary mention of the feature flag in the docs isn't sufficient and I'll try to improve it.

[simonrozsival]

This feature increases app sizes noticeably so it shouldn't be included if the customers don't need it. Unfortunately, I ran into problems while I was trying to figure out trimming for this PR and I need to consult it with you before I spend more time on it.

First, here's how customers will use NT authentication in their apps:

var cache = new CredentialCache ();
cache.Add (uri, "Negotiate", new NetworkCredential(username, password, domain));
cache.Add (uri, "NTLM", new NetworkCredential(username, password, domain));

var handler = new AndroidMessageHandler { Credentials = cache };
var client = new HttpClient (handler);

var response = await client.GetAsync (uri);
// -> 200 OK

I have two ideas how we could go about linking out the feature when it's not used but neither of them is great IMO.

1. Crude variant
   • NT auth code depends on the Credentials and Proxy setters
   • if either of the setters is present, we don't trim the NT auth code
   • pros:
     • since Mam remapping opt #7059 was merged it works for basic apps
     • the normal C# code works just fine, no need to opt-in
   • cons:
     • we will not trim it if the customer uses:
       • Basic or Digest pre-authentication
       • XmlReader anywhere in the project
       • possibly under other circumstances
     • it wasn't reliable when I tested it (in some projects it wasn't trimmed even when it wasn't used, there were no usages of the creds or proxy setters shown in ILSpy)
2. Explicit opt-in
   • if the customer wants to use NT authentication, they need to include a property in their csproj file:

   <UseNTAuthentication>true</UseNTAuthentication>

   • when the property isn't set to true the NT auth code will be trimmed out
   • pros:
     • we won't include the code + its dependencies in any app that doesn't use the feature
     • we don't change the public API of AndroidMessageHandler and HttpClientHandler
   • cons:
     • customers who want to use the feature but are unaware of the property will run into problems when they try to implement it
   • solutions:
     • we would have to document that feature and make sure it's easy to find (update the docs of AndroidMessageHandler; mention it in a Note in https://docs.microsoft.com/en-us/dotnet/framework/network-programming/ntlm-and-kerberos-authentication and possibly in some other articles)
     • add a checkbox to the VS project settings UI (I have no idea how that works, but we have that for the UseNativeHttpHandler feature switch, don't we?)
     • we could throw an exception (or at least write to the warning log) informing the user about the property when the requests fails with 401 and the response contains WWW-Authenticate headers with just the NTLM and Negotiate values

Is there some other options that I haven't considered yet? I think that the second option is better but I don't know if the poor DX is acceptable. As far as I understand it NTLM is quite a niche feature that most apps don't need but it could be a big deal for some customers.

[filipnavara]

I'd lean lightly towards <UseNTAuthentication>true</UseNTAuthentication> (with slighly different name) and bind it to the new NegotiateAuthentication API in dotnet/runtime being trimmed.

I have a couple of PRs on dotnet/runtime that improve the testing infrastructure but once that's processed I'll proceed with submitting the NegotiateAuthentication implementation. I'd certainly welcome any help with getting the trimming right.

[jonathanpeppers]

@simonrozsival given that this feature wouldn't be used in a lot of apps, can you use a linker "feature switch"?

https://github.com/dotnet/runtime/blob/main/docs/workflow/trimming/feature-switches.md

The name could be $(UseNTAuthentication) if you implement the same in xamarin-macios, otherwise I'd probably use $(AndroidUseNTAuthentication).

We can document the flag in a couple places, like this file is synced to the ms-docs website:

• https://github.com/xamarin/xamarin-android/blob/main/Documentation/guides/building-apps/build-process.md

[filipnavara]

The name could be $(UseNTAuthentication) if you implement the same in xamarin-macios, otherwise I'd probably use $(AndroidUseNTAuthentication).

xamarin-macios uses native NSUrlSession APIs which already support NTLM. Hence there's nothing to link out. I would prefer the switch to live in the dotnet/runtime repo and guard the new NegotiateAuthentication API (public version of the NTAuthentication API).

[simonrozsival]

@filipnavara the NegotiateAuthentication class won't be used anywhere in the dotnet runtime, will it? If so then I believe we don't need a feature switch in the runtime itself and keep it here. When the negotiate auth is trimmed from an android app, the NegotiateAuthentication class will be trimmed with it. We can call the switch AndroidUseNegotiateAuthentication/Xamarin.Android.Net.UseNegotiateAuthentication to align it with the new NegotiateAuthentication API and put it into Microsoft.Android.Sdk.ILLink.targets.

[filipnavara]

@filipnavara the NegotiateAuthentication class won't be used anywhere in the dotnet runtime, will it?

It will be used in SocketsHttpHandler and I would expect the switch to affect both handlers.

[simonrozsival]

@filipnavara I see. Will it make sense to have a feature switch for negotiate authentication in SocketsHttpHandler though? 🤔 We can postpone the decision for later, I won't push this PR forward without the new public API anyway.

[filipnavara]

Will it make sense to have a feature switch for negotiate authentication in SocketsHttpHandler though?

It's the exact same dependency chain that gets pulled in to the final app, so I would say it makes sense to enable the same level of trimming.

[jonpryor]

Is there a "complete" NTLM test suite somewhere? Something that checks for "invalid" hostnames, "valid but dodgy-looking" hostnames, escape characters, ../../../ path escape fragments, etc.? Something to help verify that there are no latent security-related or -adjacent issues in this NTLM authentication?

"Obviously" running tests against a "fake NTLM server" is not a complete "real-world" unit test…

[jonpryor]

Draft (and woefully short) commit message:

[Mono.Android] Optional NTLM support in AndroidMessageHandler (#6999)

Context: https://github.com/dotnet/runtime/issues/62264
Context? https://github.com/wfurt/Ntlm

Update `Xamarin.Android.Net.AndroidMessageHandler` to *optionally*
support NTLM authentication.

NTLM authentication can be enabled by setting the
`$(AndroidUseNegotiateAuthentication)` MSBuild property to True.
If this property is False or isn't set, then NTLM support is linked
away during the package build.

[jonpryor]

…and while trying a quick search for NTLM & security concerns, ran across:

• https://www.crowdstrike.com/cybersecurity-101/ntlm-windows-new-technology-lan-manager/
• https://medium.com/cyber-security-resources/hacking-and-cracking-ntlm-hash-to-get-windows-admin-password-f44819b01db5

Sounds like Kerberos-based auth should be preferred in some environments?

Then there's Microsoft documentation: https://docs.microsoft.com/en-us/windows-server/security/kerberos/ntlm-overview#BKMK_APP

Kerberos version 5 authentication is the preferred authentication method for Active Directory environments…

Then there's this beauty: https://docs.microsoft.com/en-us/archive/blogs/authentication/ntlms-time-has-passed

There’s really very little reason not to migrate to the Negotiate SSP. If you haven’t considered migrating, you should do so sooner rather than later. The chief benefit of Negotiate is that it will use Kerberos if possible, and can securely downgrade to NTLM if it becomes necessary.

🤔

Also, is this PR adding NTLM or NTLMv2? (That feels kinda important?)

[wfurt]

It is NTLMv2. NTLM is not great and it should be considered as compat feature for legacy apps @jonpryor
Getting Kerberos working is whole another beast .... that can be probably done if needed.
Either by building and bundling MIT Kerberos libs or by using c# Kerberos.NET.

As far as testing, runtime (and default handlers) use container-based approach via special enterprise-test pipeline. I'm not sure if that is practical for this repo. @filipnavara is also working on extending test coverage using Kerberos.NET. That should hopefully land before 7.0 ships.

[filipnavara]

The NegotiateAuthentication API is opaque for the Android side but in general it implements Negotiate, NTLM and Kerberos. We try to test the scenarios on the runtime side but it's tricky. There's a full server-side NTLM implementation as part of the unit tests that checks what the client-side would send. For Kerberos we are working on better tests and some of that work is in the PRs already.

Specifically for Android the dotnet/runtime implementation of NegotiateAuthentication supports only NTLM and Negotiate (wrapping just the NTLM). The NTLM implementation is quite strict NTLMv2 with integrity checks that are always enabled. Kerberos can be added in future on the dotnet/runtime side but it would either have to be opt-in (due to size) or lot of code would have to be reshuffled to make it smaller (eg. it requires DNS SRV lookups that have no public API yet, large part of the crypto stack, etc.).

Testing all this stuff is hard... there were latent bugs for many years in code that had no test coverage or insufficient coverage on some platforms. That said, I think we should focus on the actual authentication exchange being tested in the dotnet/runtime side and on the Android side it's sufficient to test the HTTP flow and Authorization headers.

[jonpryor]

@simonrozsival : thank you for this change.

[jonpryor]

@simonrozsival, @filipnavara : would appreciate review of the draft commit message.

Is there anything that should be mentioned wrt usage? What's supported? What isn't supported? (NTLM v1 should probably be called out as explicitly not supported, presumably?) When & why you would want to use this option, or what stack trace/error message people would encounter which is a sign that they should enable this feature? Links to other sources of documentation?

[simonrozsival]

When & why you would want to use this option, or what stack trace/error message people would encounter which is a sign that they should enable this feature?

@jonpryor The only reason why you would want to use this option is IMO that you need to use it. I think we should mention that we recommend using NTLM only for authentication against legacy servers.

We don't throw any exceptions if it's not enabled, the requests will all just result in 401 responses. Some customers could be confused when they specify the credentials, yet the user isn't authenticated. The commit message you suggested explicitly mentions that the NTLM support is linked away when the feature isn't enabled though.

I can't find any reasonable documentation we could link to (@filipnavara, @wfurt suggestions for existing docs are welcome). I would add a short example usage instead.

I suggest changing the commit message to something like:

Update `Xamarin.Android.Net.AndroidMessageHandler` to *optionally*
support NTLMv2 authentication in .NET 7+. This authentication method is
recommended only for legacy services that do not provide any more
secure options.

NTLM authentication can be enabled by setting the
`$(AndroidUseNegotiateAuthentication)` MSBuild property to True.
If this property is False or isn't set, then NTLM support is linked
away during the package build.

Example usage:

    // .csproj
    <PropertyGroup>
        <AndroidUseNegotiateAuthentication>true</AndroidUseNegotiateAuthentication>
    </PropertyGroup>

    // C#
    var cache = new CredentialCache ();
    cache.Add (serverUri, "Negotiate", new NetworkCredential(username, password, domain));

    var handler = new AndroidMessageHandler { Credentials = cache };
    var client = new HttpClient (handler);

    var response = await client.GetAsync (requestUri); // 200 OK

[jonpryor]

@simonrozsival : thank you, and sorry for the delay. Hopefully final "squash-and-merge" commit message:

[Mono.Android] Optional NTLM support in AndroidMessageHandler (#6999)

Context: https://github.com/dotnet/runtime/issues/62264
Context? https://github.com/wfurt/Ntlm

Update `Xamarin.Android.Net.AndroidMessageHandler` to *optionally*
support NTLMv2 authentication in .NET 7+.  This authentication method
is recommended only for legacy services that do not provide any more
secure options.

If an endpoint requires NTLMv2 authentication and NTMLv2 is not
enabled, then the endpoint will return HTTP-401 errors.

NTLMv2 authentication can be enabled by setting the
`$(AndroidUseNegotiateAuthentication)` MSBuild property to True.
If this property is False or isn't set, then NTLMv2 support is linked
away during the package build.

Example `.csproj` changes to enable NTLMv2 support:

	<PropertyGroup>
	  <AndroidUseNegotiateAuthentication>true</AndroidUseNegotiateAuthentication>
	</PropertyGroup>

Example C# `HttpClient` usage using NTLMv2 authentication:

	var cache = new CredentialCache ();
	cache.Add (serverUri, "Negotiate", new NetworkCredential(username, password, domain));

	var handler = new AndroidMessageHandler {
	    Credentials = cache,
	};
	var client = new HttpClient (handler);
	var response = await client.GetAsync (requestUri);      // 200 OK; 401 is NTLMv2 isn't enabled