User getting invalid peer certificate: UnknownIssuer

[eaddingtonwhite]

We had a customer report sign up failing with following error from his Windows machine.

[2022-09-08T16:46:31Z INFO  momento::commands::account] Signing up for Momento...
[2022-09-08T16:46:31Z DEBUG reqwest::connect] starting new connection: https://signup.registry.prod.a.momentohq.com/
[2022-09-08T16:46:31Z DEBUG hyper::client::connect::dns] resolving host="[signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/)"
[2022-09-08T16:46:31Z DEBUG hyper::client::connect::http] connecting to 44.240.4.123:443
[2022-09-08T16:46:31Z DEBUG hyper::client::connect::http] connected to 44.240.4.123:443
[2022-09-08T16:46:31Z DEBUG rustls::client::hs] No cached session for DnsName(DnsName(DnsName("[signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/)")))
[2022-09-08T16:46:31Z DEBUG rustls::client::hs] Not resuming any session
[2022-09-08T16:46:32Z DEBUG rustls::client::hs] Using ciphersuite Tls13(Tls13CipherSuite { suite: TLS13_AES_256_GCM_SHA384, bulk: Aes256Gcm })
[2022-09-08T16:46:32Z DEBUG rustls::client::tls13] Not resuming
[2022-09-08T16:46:32Z DEBUG rustls::client::tls13] TLS1.3 encrypted extensions: []
[2022-09-08T16:46:32Z DEBUG rustls::client::hs] ALPN protocol is None
[2022-09-08T16:46:32Z WARN  rustls::conn] Sending fatal alert BadCertificate
ERROR: error sending request for url (https://signup.registry.prod.a.momentohq.com/token/create): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

Need to investigate more but one thing that stood out was that his CLI was using tls 1.3 under the hood vs 1.2 on my mac. Wondering if maybe is a general issue with AWS APIGW, Custom Domains, and TLS1.3

There were few issues open on rustls and maybe webpki package around tls1.3 and and name constraint checks that maybe related.
rustls/rustls#731
briansmith/webpki#226

Also upgraded to latest version of open_ssl on my mac and was able to get a similar-looking error by forcing tls1.3

$ openssl s_client -tls1_3  -connect signup.registry.prod.a.momentohq.com:443
CONNECTED(00000006)
00B6D21301000000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1584:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 270 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

One idea was to maybe try and always force only tls1.2 and lower in rust sdk for account signup endpoint for now as workaround but need to explore/discuss this more.

https://docs.rs/rustls/latest/rustls/struct.ConfigBuilder.html#method.with_cipher_suites

We were able to verify on another team member's windows laptop as well and it worked out of the box. Here is the log output from his run. Important to note looks like its using TLS1.2 under the hood instead of TLS1.3

[2022-09-09T02:19:32Z INFO  momento::commands::account] Signing up for Momento...
[2022-09-09T02:19:32Z DEBUG reqwest::connect] starting new connection: https://signup.registry.prod.a.momentohq.com/
[2022-09-09T02:19:32Z DEBUG hyper::client::connect::dns] resolving host="[signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/)"
[2022-09-09T02:19:32Z DEBUG hyper::client::connect::http] connecting to 52.26.252.202:443
[2022-09-09T02:19:32Z DEBUG hyper::client::connect::http] connected to 52.26.252.202:443
[2022-09-09T02:19:32Z DEBUG rustls::client::hs] No cached session for DnsName(DnsName(DnsName("[signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/)")))
[2022-09-09T02:19:32Z DEBUG rustls::client::hs] Not resuming any session
[2022-09-09T02:19:32Z DEBUG rustls::client::hs] ALPN protocol is Some(b"h2")
[2022-09-09T02:19:32Z DEBUG rustls::client::hs] Using ciphersuite Tls12(Tls12CipherSuite { suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, bulk: Aes128Gcm })
[2022-09-09T02:19:32Z DEBUG rustls::client::tls12] ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }
[2022-09-09T02:19:32Z DEBUG rustls::client::tls12] Server DNS name is DnsName(DnsName(DnsName("[signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/)")))
[2022-09-09T02:19:32Z DEBUG rustls::client::tls12] Session not saved: server didn't allocate id or ticket
[2022-09-09T02:19:32Z DEBUG h2::client] binding client connection
[2022-09-09T02:19:32Z DEBUG h2::client] client connection bound
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_write] send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
[2022-09-09T02:19:32Z DEBUG h2::proto::connection] Connection; peer=Client
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_read] received frame=Settings { flags: (0x0), max_concurrent_streams: 128, initial_window_size: 65536, max_frame_size: 16777215 }
[2022-09-09T02:19:32Z DEBUG hyper::client::pool] pooling idle connection for ("https", [signup.registry.prod.a.momentohq.com](http://signup.registry.prod.a.momentohq.com/))
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_write] send frame=Settings { flags: (0x1: ACK) }
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_read] received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 2147418112 }
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_write] send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_write] send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_write] send frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
[2022-09-09T02:19:32Z DEBUG h2::codec::framed_read] received frame=Settings { flags: (0x1: ACK) }
[2022-09-09T02:19:32Z DEBUG h2::proto::settings] received settings ACK; applying Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
[2022-09-09T02:19:40Z DEBUG h2::codec::framed_read] received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
[2022-09-09T02:19:40Z DEBUG h2::codec::framed_read] received frame=Data { stream_id: StreamId(1) }
[2022-09-09T02:19:40Z DEBUG h2::codec::framed_read] received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
[2022-09-09T02:19:40Z DEBUG reqwest::async_impl::client] response '201 Created' for https://signup.registry.prod.a.momentohq.com/token/create
[2022-09-09T02:19:40Z INFO  momento::commands::account] Success! Your access token will be emailed to you shortly.
[2022-09-09T02:19:40Z DEBUG h2::codec::framed_write] send frame=GoAway { error_code: NO_ERROR, last_stream_id: StreamId(0) }
[2022-09-09T02:19:40Z DEBUG h2::proto::connection] Connection::poll; connection error error=GoAway(b"", NO_ERROR, Library)
[2022-09-09T02:19:40Z DEBUG rustls::conn] Sending warning alert CloseNotify

[kvcache]

it does say "unknown issuer" and the certificate appears to be issued by :

openssl s_client -showcerts -connect signup.registry.prod.a.momentohq.com:443 | grep issuer
depth=4 C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
verify return:1
depth=3 C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
verify return:1
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = *.execute-api.us-west-2.amazonaws.com
verify return:1
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon

[eaddingtonwhite]

Looking more at AWS API GW docs looking at now they dont claim to support tls1.3 yet for regional endpoints it seems just edge optimized ones which were not using yet.
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-custom-domain-tls-version.html#apigateway-custom-domain-tls-version-regional-and-websocket

NLB's are supposed to support now though so as we move to the new identity endpoints we should be able to add TLs1.3 since its hosted behind NLB on our control plane svc now vs lambda and AWS API GW.
https://aws.amazon.com/about-aws/whats-new/2021/10/aws-network-load-balancer-supports-tls-1-3/