Connection close with TLSv1.3 after updating to Elixir 1.16 and OTP 26

[isavita]

We are experiencing an issue related to SSL and receiving a connection closed when using TLSv1.3. We are upgrading our application from erlang 24.3.4.8 with elixir 1.14.3 to erlang 26.2.2 with elixir 1.16.1. On our older version the service responds fine. When using TLSv1.2 on the newer upgrade that is also fine. It only seems to be a problem with TLSv1.3.

We are using the below which shows the connection close response. We receive this for every request to the service. Other services which also use certificates are fine. This particular service was fine with connecting to it before the upgrade. It also fine with TLSv1.2 with our upgrade.

{:ok, p} = Finch.start_link(name: MyFinch, pools: %{default: [conn_opts: [transport_opts: [{:verify, :verify_peer}, {:cacertfile, "/path/to/ca.crt"}, {:certfile, "/path/to/cert.crt"}, {:keyfile, "/path/to/key.key"}, {:keep_secrets, true}, {:versions, [:"tlsv1.3"]}, {:log_level, :debug}], ssl_key_log_file: "/var/log/ssl.log"]]}) 
 
Finch.build(:get, "https://some-service.com", [], nil) |> Finch.request(MyFinch) 
{:error, %Mint.TransportError{reason: :closed}}

When enabling low level logs, we can see a second handshake which has an empty certificate list.

>>> Handshake, Certificate 
[{certificate_request_context,<<>>},{certificate_list,[]}]

We had a handshake before this which had certificates. The older version has certificates for both of the handshakes.

<<< Handshake, Certificate 
[{certificate_request_context,<<>>}, 
  {certificate_list, 
    [{certificate_entry, 
      <<... 

We have spent a lot of time considering the service and certificates but we do not have any feedback this is the problem. It was also fine before our upgrade using TLSv1.3 so very puzzling for us. Any help on how we can debug or solve this is much appreciated.

[ericmj]

Can you verify that you can open connections using the underlying Erlang SSL APIs: https://www.erlang.org/doc/man/ssl

You may have to change some options slightly but it should all be documented on the above page.

[isavita]

@ericmj Thank you very much. I tried to compare Elixir 1.14.3 and OTP 24 with both 1) Elixir 1.16.1 and OTP 26, and 2) Elixir 1.16.1 and OTP 25.

I used the following code to fetch some data with :ssl:

defmodule SSLDebug do
  def run do
    ssl_opts = [
      {:verify, :verify_peer},
      {:cacertfile, "/path/to/ca.crt"},
      {:certfile, "/path/to/client.crt"},
      {:keyfile, "/path/to/client.key"},
      {:keep_secrets, true},
      {:versions, [:"tlsv1.3"]},
      {:log_level, :debug},
      {:log_alert, true}
    ]

    host = 'mywebsite.com'
    port = 443

    case :ssl.connect(host, port, ssl_opts) do
      {:ok, socket} ->
        IO.puts("SSL connection established")

        path = "/myendpoint"
        request = "GET #{path} HTTP/1.1\r\nHost: #{host}\r\n\r\n"

        case :ssl.send(socket, request) do
          :ok ->
            IO.puts("Request sent")
          {:error, reason} ->
            IO.puts("Error sending request: #{inspect(reason)}")
        end

        response = receive_response(socket, [], 3_000)

        :ok = :ssl.close(socket)

        case response do
          {:ok, data} ->
            IO.puts("Response:")
            IO.puts(data)
          {:error, reason} ->
            IO.puts("Error receiving response: #{inspect(reason)}")
        end

      {:error, reason} ->
        IO.puts("SSL connection failed: #{inspect(reason)}")
    end
  end

  defp receive_response(socket, acc, timeout) do
    receive do
      {:ssl, ^socket, data} ->
        IO.puts("Received data: #{inspect(data)}")
        receive_response(socket, [acc, data], timeout)
      {:ssl_closed, ^socket} ->
        IO.puts("Socket closed")
        {:ok, :erlang.iolist_to_binary(acc)}
      {:ssl_error, ^socket, reason} ->
        IO.puts("SSL error: #{inspect(reason)}")
        {:error, reason}
    after
      timeout ->
        IO.puts("Receive timeout after #{timeout} milliseconds")
        {:ok, :erlang.iolist_to_binary(acc)}
    end
  end
end

SSLDebug.run()

Everything worked perfectly on Elixir 1.14.3 and OTP 24, but it failed on both 1) Elixir 1.16.1 and OTP 26, and 2) Elixir 1.16.1 and OTP 25.

The issue seems to be that it's not submitting the following:

0000 - 14 03 03 00 01 01                                   ......
>>> Handshake, Certificate
[{certificate_request_context,<<>>},{certificate_list,[]}]
writing (30 bytes) TLS 1.2 Record Protocol, application_data

And it ends with:

Socket closed
Response:

It appears to be a problem with the :ssl version.

On OTP 26, I can see:

iex(1)> :ssl.versions()
[
  ssl_app: ~c"11.1.1",
  supported: [:"tlsv1.3", :"tlsv1.2"],
  supported_dtls: [:"dtlsv1.2"],
  available: [:"tlsv1.3", :"tlsv1.2", :"tlsv1.1", :tlsv1],
  available_dtls: [:"dtlsv1.2", :dtlsv1],
  implemented: [:"tlsv1.3", :"tlsv1.2", :"tlsv1.1", :tlsv1],
  implemented_dtls: [:"dtlsv1.2", :dtlsv1]
]

Compared to OTP 24:

iex(1)> :ssl.versions()
[
  ssl_app: '10.7.3.5',
  supported: [:"tlsv1.3", :"tlsv1.2"],
  supported_dtls: [:"dtlsv1.2"],
  available: [:"tlsv1.3", :"tlsv1.2", :"tlsv1.1", :tlsv1],
  available_dtls: [:"dtlsv1.2", :dtlsv1],
  implemented: [:"tlsv1.3", :"tlsv1.2", :"tlsv1.1", :tlsv1],
  implemented_dtls: [:"dtlsv1.2", :dtlsv1]
]

The problem seems to be with the ssl_app version. It changed from ssl_app: '10.7.3.5' in OTP 24 to ssl_app: ~c"11.1.1" in OTP 26.

[ericmj]

Since this is being tracked on the OTP repo I will close this for now. Thanks for opening an issue there.

If it turns out to be a Mint issue we will of course reopen.