ssl_client1 fails on TLS 1.3

[duduita]

Summary

If I attempt to make a request to https://api.sunrisesunset.io/json?lat=47.333&lng=13.333 using Postman, or a browser, it works perfectly.

However, when making an HTTP request using the mbedTLS sample client ssl_client1.c with these parameters:

#define SERVER_PORT "443"
#define SERVER_NAME "api.sunrisesunset.io"
#define GET_REQUEST "GET /json?lat=47.333&lng=13.333 HTTP/1.1\r\nHost: api.sunrisesunset.io\r\n\r\n"

I got the following error:

ssl_client.c:0042: client hello, adding server name extension: api.sunrisesunset.io
ssl_tls13_client.c:0057: client hello, adding supported versions extension
ssl_tls13_client.c:0080: supported version: [3:4]
ssl_tls13_client.c:0086: supported version: [3:3]
ssl_tls13_client.c:0572: no cookie to send; skip extension
ssl_tls13_client.c:0285: client hello: adding key share extension
ssl_tls13_generic.c:1651: Perform PSA-based ECDH/FFDH computation.
ssl_tls13_generic.c:1689: psa_generate_key() returned -27648 (-0x6c00)
ssl_client.c:1012: <= write client hello
ssl_tls.c:4617: <= handshake
 failed
  ! mbedtls_ssl_handshake returned -0x6c00

Last error was: -27648 - SSL - Internal error (eg, unexpected failure in lower-level module)

What is interesting is that when I force the mbedTLS to use TLS 1.2, instead TLS 1.3 by adding the following:

mbedtls_ssl_conf_max_version(&conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3); // Force TLS 1.2

The request works perfectly! So, it indicates to me that there might be a problem with the TLS 1.3, somehow the server is not properly working with mbedTLS TLS 1.3.

System information

Mbed TLS version (number or commit id): 2ca6c285a0dd3f33982dd57299012dacab1ff206
Operating system and version: macOS 13.2.1 (22D68)
Configuration (if not default, please attach mbedtls_config.h): default

Expected behavior

The request should return a successful result.

Actual behavior

The request is failing during the handshake as previously mentioned:

ssl_tls.c:4617: <= handshake
 failed
  ! mbedtls_ssl_handshake returned -0x6c00

Last error was: -27648 - SSL - Internal error (eg, unexpected failure in lower-level module)

Steps to reproduce

Just change these configs on ssl_client1.c sample:

#define SERVER_PORT "443"
#define SERVER_NAME "api.sunrisesunset.io"
#define GET_REQUEST "GET /json?lat=47.333&lng=13.333 HTTP/1.1\r\nHost: api.sunrisesunset.io\r\n\r\n"

Then, rebuild and run that.

Additional information

• It's working normally when calling from a browser, so it's not a server issue.
• It seems to be solved by forcing the usage of TLS 1.2, as previously mentioned:
  mbedtls_ssl_conf_max_version(&conf, MBEDTLS_SSL_MAJOR_VERSION_3, MBEDTLS_SSL_MINOR_VERSION_3);

[gilles-peskine-arm]

Internal error where the TLS layer calls psa_generate_key is a sign that psa_crypto_init hasn't been called.

ssl_client1 is only calling psa_crypto_init when MBEDTLS_USE_PSA_CRYPTO is enabled. That's correct for TLS 1.2, but psa_crypto_init also needs to be called before making a TLS 1.3 connection regardless of MBEDTLS_USE_PSA_CRYPTO. So that's a bug in ssl_client1, and probably many other sample programs. This bug applies to Mbed TLS 3.6 LTS, not to 2.28 (no functional TLS 1.3) and not to 4.0 (PSA will become always on).

If you're concerned about that specific site, please try with ssl_client2, or in a configuration with MBEDTLS_USE_PSA_CRYPTO. In your own code, please make sure you call psa_crypto_init before making a potentially-1.3 TLS connection.

[duduita]

Thanks for helping me @gilles-peskine-arm.

After enabling psa_crypto_init I'm not seeing anymore the error -27648 - SSL - Internal error, but I'm seeing -9984 - X509 - Certificate verification failed, e.g. CRL, CA or signature check failed, as can be seen in the debug logs:

ssl_tls13_generic.c:0709: x509_verify_cert() returned -9984 (-0x2700)
ssl_tls13_generic.c:0786: ! Certificate verification flags 00000008
ssl_tls13_generic.c:0832: <= parse certificate
ssl_msg.c:5168: => send alert message
ssl_msg.c:5169: send alert level=2 message=48
ssl_msg.c:2943: => write record
ssl_msg.c:3030: output record: msgtype = 21, version = [3:3], msglen = 2
ssl_msg.c:3033: dumping 'output record sent to network' (7 bytes)
ssl_msg.c:3033: 0000:  15 03 03 00 02 02 30                             ......0
ssl_msg.c:2353: => flush output
ssl_msg.c:2369: message length: 7, out_left: 7
ssl_msg.c:2374: ssl->f_send() returned 7 (-0xfffffff9)
ssl_msg.c:2401: <= flush output
ssl_msg.c:3080: <= write record
ssl_msg.c:5180: <= send alert message
ssl_tls.c:4617: <= handshake
 failed
  ! mbedtls_ssl_handshake returned -0x2700

Last error was: -9984 - X509 - Certificate verification failed, e.g. CRL, CA or signature check failed

ssl_tls.c:5521: => free
ssl_tls.c:5583: <= free

Since this server works normally on other clients, but the certificate verification is failing on mbedTLS, does it indicate that the root CA certificates chain mbedtls_test_cas_pem should be adjusted to be able to verify this server certificate?

[gilles-peskine-arm]

Mbed TLS doesn't ship with any integration to the OS CA list. ssl_client1 is a demo program intended to pair with an ad hoc server, not intended as a curl equivalent. In your own code, call mbedtls_x509_crt_parse_file or mbedtls_x509_crt_parse_path to load certificate(s) to pass to mbedtls_ssl_conf_ca_chain.

[duduita]

Mbed TLS doesn't ship with any integration to the OS CA list.

I didn't mean that, of course, mbedTLS doesn't use the OS CA list, instead it uses mbedtls_test_cas_pem as I mentioned.

Just to make me clearer, I just expect to see the certificate validation working properly when using psa_crypto_init as you suggested, since when forcing the TLS 1.2 usage it worked perfectly, as previously mentioned, without any change in the ssl_client1 sample, i.e., using the same CA chain list.

To sum up, using psa_crypto_init solved my previous issue, but now suddenly the certificate verification started to fail, even when making requests to the same server, and using the same CA chain list, I just would like to understand why it started to happen to be able to fix it.

[gilles-peskine-arm]

Oh, this turns out to be another deficiency in ssl_client1:

    /* OPTIONAL is not optimal for security,
     * but makes interop easier in this simplified example */
    mbedtls_ssl_conf_authmode(&conf, MBEDTLS_SSL_VERIFY_OPTIONAL);
…
    mbedtls_printf("  . Verifying peer X.509 certificate...");

    /* In real life, we probably want to bail out when ret != 0 */
    if ((flags = mbedtls_ssl_get_verify_result(&ssl)) != 0) {
        mbedtls_printf(" failed\n");
    }

It's dubious practice in the first place, and in TLS 1.3 we don't allow bypassing server authentication so it doesn't work at all. I've filed #9079 to improve the sample program.

So, my previous reply here was wrong for TLS 1.2, but correct for TLS 1.3: you do need to have a matching root CA, not just for real-world code, but also for ssl_client1.

[sfan5]

It would have been nice to document the need to call psa_crypto_init in the 3.6.0 release notes.
Applications who only used mbedTLS for "minimal" TLS connections probably don't already happen to have a call to this function.

[gilles-peskine-arm]

Mbed TLS 3.6.1

When a TLS connection reaches a point where it might negotiate TLS 1.3, the TLS layer will call psa_crypto_init. Done in #9501.

Mbed TLS 4.0

In Mbed TLS 4.0, all TLS connections will require psa_crypto_init anyway, which makes this issue moot.

Testing

We will add an interoperability test between ssl_client1 and OpenSSL to our CI. Work in progress for 3.6: #9505

[RytoEX]

Mbed TLS 3.6.1

When a TLS connection reaches a point where it might negotiate TLS 1.3, the TLS layer will call psa_crypto_init. Done in #9281.

Mbed TLS 4.0

@gilles-peskine-arm Did you mean #9501? #9281 doesn't seem to be related to psa_crypto_init().