.NET 5 RC1 breaks with "Address family not supported by protocol" in AWS Lambda but not in preview 8 and earlier

[normj]

Description

I apologize for not having all of the data yet but given the closeness to the GA release I wanted to get the conversation started as soon as possible. I'll add more information as I continue my research.

AWS Lambda, which is the serverless offering at AWS, allows users to deploy Lambda functions as a self contained publish bundle for .NET developers. It has been possible to use .NET 5 preview versions in Lambda up till the latest RC1. Now when users deploy their functions it fails right away access the Lambda API through the HttpClient with the following error messages.

Unhandled exception. System.Net.Http.HttpRequestException: Address family not supported by protocol (127.0.0.1:9001)
 ---> System.Net.Sockets.SocketException (97): Address family not supported by protocol
   at System.Net.Sockets.Socket..ctor(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType)
   at System.Net.Connections.SocketsConnectionFactory.CreateSocket(AddressFamily addressFamily, SocketType socketType, ProtocolType protocolType, EndPoint endPoint, IConnectionProperties options)
   at System.Net.Connections.SocketsConnectionFactory.ConnectAsync(EndPoint endPoint, IConnectionProperties options, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.ConnectAsync(ConnectionFactory factory, DnsEndPoint endPoint, IConnectionProperties options, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(ConnectionFactory factory, DnsEndPoint endPoint, IConnectionProperties options, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
   at Amazon.Lambda.RuntimeSupport.InternalRuntimeApiClient.NextAsync(CancellationToken cancellationToken)
   at Amazon.Lambda.RuntimeSupport.RuntimeApiClient.GetNextInvocationAsync(CancellationToken cancellationToken)
   at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.InvokeOnceAsync(CancellationToken cancellationToken)
   at Amazon.Lambda.RuntimeSupport.LambdaBootstrap.RunAsync(CancellationToken cancellationToken)
   at Lambda5Test.Function.Main(String[] args) in C:\Temp\Lambda5Test\src\Lambda5Test\Function.cs:line 21
   at Lambda5Test.Function.<Main>(String[] args)

The address it is connecting to is Lambda's runtime API what all functions have to interact with.

My theory is the HttpClient is attempting to use IPv6 which Lambda does not support and the recent changes to the SocketsConnectionFactory are triggering HttpClient to use IPv6. In particular the following constructor that is defaulting the AddressFamily enum value for IPv6

runtime/src/libraries/System.Net.Connections/src/System/Net/Connections/Sockets/SocketsConnectionFactory.cs

Line 49 in 1341c35

: this(AddressFamily.InterNetworkV6, socketType, protocolType)

Configuration

Running in AWS under Amazon Linux 2 in the Lambda environment with no IPv6 support.

Regression?

This has worked up through .NET 5 Preview 8.

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[geoffkizer]

Pretty sure this is fixed in RC2 by #42075. Can you grab an RC2 build and confirm?

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[normj]

That is great if already fixed. I have tried to with the nightly builds of RC2 but since I have to do a self-contained published I get the following errors:

... publish: /home/ec2-user/NET5LambdaTest/src/NET5LambdaTest/NET5LambdaTest.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Runtime.linux-x64 with version (= 5.0.0-rc.2.20462.5)
... publish: /home/ec2-user/NET5LambdaTest/src/NET5LambdaTest/NET5LambdaTest.csproj : error NU1102:   - Found 29 version(s) in nuget.org [ Nearest version: 5.0.0-rc.1.20451.14 ]
... publish: /home/ec2-user/NET5LambdaTest/src/NET5LambdaTest/NET5LambdaTest.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-x64 with version (= 5.0.0-rc.2.20464.16)
... publish: /home/ec2-user/NET5LambdaTest/src/NET5LambdaTest/NET5LambdaTest.csproj : error NU1102:   - Found 31 version(s) in nuget.org [ Nearest version: 5.0.0-rc.1.20451.17 ]

Any suggestions on how to do self contained publishes using a nightly build?

[richlander]

Try this nuget.config -> https://github.com/dotnet/core/blob/master/samples/nuget.config

Same as the one @ https://github.com/dotnet/installer#build-status

[normj]

Whoops, should a look a little harder and I would have seen the nuget.config.

The good news is I did confirmed with rc.2.20464.14 the issue is fixed and I was successfully able to run a self contained .NET 5 RC2 Lambda function. Thanks for getting back with me so quickly, just had a bit of freak out of .NET 5 going out with a breaking issue for us 😅

[danmoseley]

@geoffkizer just curious I thought #42075 added new api and a code change would have been required to be affected by that change?

@normj i guess we can close this? Thanks so much for reporting it!

[geoffkizer]

@danmosemsft Yeah, it's a little complicated...

This bug was introduced when we did the original System.Net.Connections work in #39524. That was mostly reverted in #41648, but we kept some of the changes from #39524 to try to minimize code churn. Unfortunately, that included this bug.

When reviewing #42075, @stephentoub noticed the issue here and so we fixed it as part of that PR -- meaning we reverted back to the original pre-#39524 behavior.

I wish we had a test for this, but it would require an environment where IPv6 is not supported. Ideally, we'd add that to CI but it's probably far from trivial.

[Gnbrkm41]

Run the test in a docker container with --sysctl net.ipv6.conf.all.disable_ipv6=1? 😄

[normj]

@danmosemsft yes I'm good closing this now that it has been confirmed to be fixed in RC2.

[danmoseley]

@geoffkizer ah..