Azure SQL Private Link: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.

[Nico-VanHaaster]

Although this issue may be similar to #1402 we have the following environments and scenarios.

Client
Microsoft.Data.SqlClient - 4.0.1 & 4.1.0 (tried both versions with same results)
Dockerized application running on Kubernetes. Image: mcr.microsoft.com/dotnet/runtime-deps:latest

Server
Azure SQL Database with Private Link enabled

Scenario 1
When executing the application from outside of the private link (using the public IP Address) we are able to connect the SQL Server Database with no errors.

Scenario 2
When executing the same application and within the Private LINK networks. Using the standard .database.windows.net (which resolves to .privatelink.database.windows.net and the internal private IP Address). We receive the error.

Unhandled exception. Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: TCP Provider, error: 35 - An internal exception was caught)
 ---> System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback

Scenario 3
When executing the same application and using private link (private IP Address) however adding "TrustServerCertificate=true". We are able to connect to the SQL Server.

Ideally we do not want to set TrustServerCertificate=true as from our understanding the could put us at risk of a man-in-the-middle attack.

Has anyone come across this issue in the past or have a workaround that keeps our connections secure?

Cheers.

[JRahnama]

@Nico-VanHaaster is domain of the private link same as public one? If not certificate wont be working correctly. the domain of the certificate needs to be same as AD user. See issue #1437

[JRahnama]

@Nico-VanHaaster on some other notes, are you able to get access using SSMS when encryption is requested by private link?

[gopinathrimc]

Having same issue when trying to connect to AWS RDS SQL 2019 instance from a lambda running dotnet 6 and using Microsoft.Data.SqlClient 4.0.1 .
Some approaches I have seen is implementing RemoteCertificateValidationCallback and using the certificate to validate the server's certificate. But not sure how to go about integrating that approach while using Microsoft.Data.SqlClient .

[JRahnama]

@Nico-VanHaaster two questions to narrow down the issue:

1. how is the SSL/TLS certificate setup is it SNI or IP based SSL?
2. Is there any IPV preference setup ( such as IPV6 or IPV4 as a preferred one)?

[Nico-VanHaaster]

@JRahnama

These are private endpoints to Azure Sql Databases. We don't manage the certificates, SNI etc as it's an Azure PaaS services

It is very easy to duplicate the issue following my OP.

[quinten-miller]

@Nico-VanHaaster we had this same issue within our environment, which is as follows:
Azure function app. .Net Core 6
EF 6 which has a dependency on Microsoft.Data.SqlClient 4.x
Linux App Service Plan
Private endpoints
Connecting to SQL Azure
... and a firewall within the enterprise.

It turns out the the issue was due to not being able to reach the Certificate Revocation List. In version 4 of the SqlClient library there was a change in behavior that cause this to drop the connection - the exception that you are seeing.

there is quite a long thread on a similar issue, scrolling to the bottom will show a fix merged into the SqlClient library that changes this behavior for V5

For us, the firewall was blocking access to *.digicert.com, in our case AppInsights had the failing requests (failures), and we could see the 403 in there.

This was testable by jumping on the server (in our case both non-prod and prod, given non-prod actually worked) and running:
curl crl3.digicert.com/Omniroot2025.crl --verbose --output Omniroot2025.crl

In non-prod, we got the CRL, in prod we got a 403.

So while there as a change in behavior, in the end the fix on our side is to update the firewall with appropriate rules.

An alternative was to add TrustServerCertificate=true, however this was not as desirable as punching holes in the firewall.

Hope this helps

[JRahnama]

@Nico-VanHaaster can you test with the latest release (v5.0) of M.D.S to check if the issue still exists?

[JRahnama]

Closing due to inactivity. Feel free to comment if the issue still exists.

[GilShvarzman]

This still exists. Trying to connect to a private endpoint of an Azure Database Server using a VPN results in this exception.
Setting TrustServerCertificate=true results in another exception - Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot open server "<private_ip_address>" requested by the login. The login failed.
Also, there is no firewall set up.

[JRahnama]

@GilShvarzman the error message is very different in your case. the server has rejected the login.

[ecoonradt]

For anyone finding this thread I would like to report that updating the Microsoft.Data.SqlClient package to the latest version does solve this problem. EF 7 brings the latest version with it as well.

[danielleiszen]

I experienced the same error when trying to run code first migrations on a GKE deployed SQL Server
image: mssql: mcr.microsoft.com/mssql/server:2019-latest
from a service based on
image: mcr.microsoft.com/dotnet/aspnet:7.0
using nuget package Microsoft.EntityFrameworkCore.SqlServer 7.0.0

when I rolled back to Microsoft.EntityFrameworkCore.SqlServer 6.0.11 the connection succeeded and the migration went through.

I think this still may be an issue. So, I would suggest reopening this issue.

The full stack trace:

System.Security.Authentication.AuthenticationException: The remote certificate was rejected by the provided RemoteCertificateValidationCallback.
at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken)
at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
at Microsoft.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)
at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.EnableSsl(UInt32 info, SqlConnectionEncryptOption encrypt, Boolean integratedSecurity)
at Microsoft.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(SqlConnectionEncryptOption encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean& marsCapable, Boolean& fedAuthRequired, Boolean tlsFirst)
at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, SqlConnectionString connectionOptions, Boolean withFailover)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)
at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)
at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)
at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()

[joe-plumb]

Stumbled upon this issue when attempting to connect Azure Function to Azure SQL using Managed Identity Auth as I was receiving the same error. My issue was DNS had not been correctly configured in my environment and so I was trying to use the Private Endpoint IP of Azure SQL instead of relying on DNS which was causing this error.

Setting up Private DNS Zone in Azure to resolve the FQDN to the Private Endpoint has resolved the RemoteCertificateValidationCallback error.