Kestrel: Segfault when specifying default certificate on Ubuntu 20.04

[VMelnalksnis]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

After upgrading an ASP.NET Core project from .NET 6 to .NET 7, the application fails to start with a segmentation fault when specifying a default certificate for Kestrel.

Expected Behavior

Application works with a certificate on .NET 7 same as before on .NET 6.

Steps To Reproduce

1. Start the application without specifying a certificate
2. Add the following in appsettings.json:

{
  "Kestrel": {
    "Certificates": {
      "Default": {
        "Path": "/path/to/cert.p12",
        "Password": "password"
      }
    }
  }
}

3. After the configuration has been reloaded, see Main process exited, code=killed, status=11/SEGV

Alternatively, can just specify the certificate from the start and see that the application crashes before starting.

Exceptions (if any)

I've seen two types of segfaults in the hypervisor syslog:
after updating appsettings.json

kernel: .NET ThreadPool[3421307]: segfault at 0 ip 00007f32507860be sp 00007f3226bb58c0 error 4 in libcrypto.so.1.1[7f3250674000+19b000]
kernel: Code: 00 00 4c 89 f1 4c 89 fa e8 bf 24 07 00 85 c0 0f 84 bf 00 00 00 8b 54 24 1c 49 8d 7c 24 10 4c 89 fe e8 56 2b f0 ff 85 c0 74 52 <48> 63 75 00 48 8b 7d 08 45 31 c9 49 89 d8 4c 89 f1 4c 89 fa e8 49

and starting the application with default certificates:

kernel: Gnomeshade.WebA[3397350]: segfault at 0 ip 00007f705b6200be sp 00007fff3b39f640 error 4 in libcrypto.so.1.1[7f705b50e000+19b000]
kernel: Code: 00 00 4c 89 f1 4c 89 fa e8 bf 24 07 00 85 c0 0f 84 bf 00 00 00 8b 54 24 1c 49 8d 7c 24 10 4c 89 fe e8 56 2b f0 ff 85 c0 74 52 <48> 63 75 00 48 8b 7d 08 45 31 c9 49 89 d8 4c 89 f1 4c 89 fa e8 49

.NET Version

7.0.102

Anything else?

Running on Ubuntu 20.04.5 LTS LXC container on Proxmox 7.3-4.
libbssl version:

~$ sudo apt show libssl1.1
Package: libssl1.1
Version: 1.1.1f-1ubuntu2.16

[amcasey]

@VMelnalksnis Can you tell us more about your project? I followed your steps with a simple React app (dotnet new react) on Ubuntu 20.04.5 (in WLS2, since I don't have proxmox set up) and I was able to connect to the page and see my specified cert.

Have you tried your steps with a toy project (i.e. not your real one) to see whether it's somehow related to your project (vs proxmox, ubuntu, libssl, etc)?

[VMelnalksnis]

@amcasey I tried to run dotnet new web with the same configuration on the same container, and it also failed with a segmentation fault:

kernel: TestApplication[1956636]: segfault at 0 ip 00007f7f211e00be sp 00007fff92d9b940 error 4 in libcrypto.so.1.1[7f7f210ce000+19b000]
kernel: Code: 00 00 4c 89 f1 4c 89 fa e8 bf 24 07 00 85 c0 0f 84 bf 00 00 00 8b 54 24 1c 49 8d 7c 24 10 4c 89 fe e8 56 2b f0 ff 85 c0 74 52 <48> 63 75 00 48 8b 7d 08 45 31 c9 49 89 d8 4c 89 f1 4c 89 fa e8 49

After changing the target framework to net6.0 it worked. I also tried to publish with --no-self-contained and install dotnet-sdk-7.0 on the container, and it also failed with a segmentation fault.

The project is this one https://github.com/VMelnalksnis/Gnomeshade/tree/master/source/Gnomeshade.WebApi. The only things changed for Kestrel were content root, web root and cipher suites policy for TLS, but I also tried without it.

[amcasey]

@VMelnalksnis Thanks! It's very helpful to know that it repros in a new project. I'll play around with dotnet new web to see if I can trigger the crash.

[amcasey]

@VMelnalksnis Sorry, I'm still not seeing it. Is the appsettings JSON in the description just part of your file or the entire file?

If you can repro the issue consistently, can you collect and share a dump?

Hi @VMelnalksnis. We have added the "Needs: Author Feedback" label to this issue, which indicates that we have an open question for you before we can take further action. This issue will be closed automatically in 7 days if we do not hear back from you by then - please feel free to re-open it if you come back to this issue after that time.

[VMelnalksnis]

@amcasey No, that was only the part that caused the crash. I have other sections for other libraries as well, such as ElasticApm and ConnectionStrings, and I can post those as well if they're relevant. Here's the full Kestrel section:

"Kestrel": {
    "EndpointDefaults": {
        "Protocols": "Http1AndHttp2"
    },
    "Endpoints": {
        "Https": {
            "Url": "https://some.fully.qualified.name.com:443",

            "CheckCertificateRevocation": true,
            "ClientCertificateMode": "Optional",
            "SslProtocols": ["Tls13"],
        }
    },
    "Certificates": {
        "Default": {
            "Path": "/path/to/host/certificate.p12",
            "Password": "password"
        }
   }
}

I tried the same configuration by changing each value separately as well: CheckCertificateRevocation, SslProtocols to TLS12, https port to 8443/443.

As for the dump, I'm not entirely sure where to look for it. The common answers for Ubuntu were not correct, I'm guessing because it's an LXC and/or on Proxmox; I'll keep looking.

[amcasey]

@VMelnalksnis Thanks! Since you have a repro with the dotnet new web template, do you have the option of testing it out on an Ubuntu machine/VM outside proxmox?

[amcasey]

We have some docs here about how to collect dotnet dumps, but this is a native crash, so I'm not sure they'll work in this case. I'll see if I can figure out how to collect a core dump on Ubuntu 20.04.

[amcasey]

Well, that was harder than it needed to be...

I managed to cobble together these commands to get core files to appear in /var/crash.

ulimit -c unlimited
sudo sysctl -w kernel.core_pattern=/var/crash/core.%e.%p.%h.%t

I think you can get back to normal with

ulimit -c 0
systemctl restart apport

Mostly from here. No warranty, express or implied. 😛

[VMelnalksnis]

@amcasey Thanks for the info - when running non-self-contained, setting DOTNET_DbgEnableMiniDump and ulimit -c unlimited was enough, and path was controlled by DOTNET_DbgMiniDumpName.
Here's the dump - coredump.zip

[amcasey]

I'm seeing

libcoreclr!sigsegv_handler+0x260
libpthread_2_31+0x14420
libcrypto_so_1+0x18a0be
libSystem_Security_Cryptography_Native_OpenSsl!MakeCertId+0x1e
libSystem_Security_Cryptography_Native_OpenSsl!BuildOcspRequest+0x1e
libSystem_Security_Cryptography_Native_OpenSsl!CryptoNative_X509BuildOcspRequest+0x37
Interop+Crypto.<X509BuildOcspRequest>g____PInvoke|66_0(IntPtr, IntPtr)+0x4e
System_Net_Security_7f663ec70000!Interop+Crypto.X509BuildOcspRequest(IntPtr, IntPtr)+0x4f
System_Net_Security_7f663ec70000!System.Net.Security.SslStreamCertificateContext+<FetchOcspAsync>d__27.MoveNext()+0xdc
System_Private_CoreLib_7f663bad0000!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl+<StartAsync>d__30`1[[System.__Canon, System.Private.CoreLib]], Microsoft.AspNetCore.Server.Kestrel.Core]](<StartAsync>d__30`1<System.__Canon> ByRef)+0xac4d
System_Private_CoreLib_7f663bad0000!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib]].Start[[System.Net.Security.SslStreamCertificateContext+<FetchOcspAsync>d__27, System.Net.Security]](<FetchOcspAsync>d__27 ByRef)+0x20
System_Net_Security_7f663ec70000!System.Net.Security.SslStreamCertificateContext.FetchOcspAsync()+0x69
System_Net_Security_7f663ec70000!System.Net.Security.SslStreamCertificateContext.DownloadOcspAsync()+0x2ea
System_Net_Security_7f663ec70000!System.Net.Security.SslStreamCertificateContext.AddRootCertificate(System.Security.Cryptography.X509Certificates.X509Certificate2)+0x59
System_Net_Security_7f663ec70000!System.Net.Security.SslStreamCertificateContext.Create(System.Security.Cryptography.X509Certificates.X509Certificate2, System.Security.Cryptography.X509Certificates.X509Certificate2Collection, Boolean, System.Net.Security.SslCertificateTrust, Boolean)+0x438
Microsoft_AspNetCore_Server_Kestrel_Core_7f663dd50000!Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionMiddleware..ctor(Microsoft.AspNetCore.Connections.ConnectionDelegate, Microsoft.AspNetCore.Server.Kestrel.Https.HttpsConnectionAdapterOptions, Microsoft.Extensions.Logging.ILoggerFactory)+0x1c0

[amcasey]

Based on the stack, I'm guessing dotnet/runtime will want to take a look. Probably @bartonjs?