test_asyncio: test_create_connection_ssl_failed_certificate() failed on ARM64 macOS 3.x buildbot

[vstinner]

ARM64 macOS 3.x:
https://buildbot.python.org/all/#/builders/725/builds/5088

The test started to fail when OpenSSL was upgrade from 3.0.0 to 3.1.1 at July 13.

cc @pablogsal @ned-deily @ambv

• Last successful build, build 4995 (July 13 at 5:01PM): https://buildbot.python.org/all/#/builders/725/builds/4995
  • test.pythoninfo: ssl.OPENSSL_VERSION: OpenSSL 3.0.0 7 sep 2021
• Fist failure, build 4999 (July 13 at 5:23PM): https://buildbot.python.org/all/#/builders/725/builds/4999
  • test.pythoninfo: ssl.OPENSSL_VERSION: OpenSSL 3.1.1 30 May 2023
• Between build 4995 and 4999, build failed with an exception

Error:

ERROR: test_create_connection_ssl_failed_certificate (test.test_asyncio.test_ssl.TestSSL.test_create_connection_ssl_failed_certificate)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_asyncio/test_ssl.py", line 454, in test_create_connection_ssl_failed_certificate
    self.loop.run_until_complete(client(srv.addr))
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_asyncio/test_ssl.py", line 441, in client
    reader, writer = await asyncio.open_connection(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/streams.py", line 47, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 1126, in create_connection
    transport, protocol = await self._create_connection_transport(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 1159, in _create_connection_transport
    await waiter
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/sslproto.py", line 575, in _on_handshake_complete
    raise handshake_exc
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/sslproto.py", line 557, in _do_handshake
    self._sslobj.do_handshake()
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/ssl.py", line 917, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1024)

test.pythoninfo:

os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-mini', release='22.5.0', version='Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103', machine='arm64')
platform.architecture: 64bit
platform.platform: macOS-13.4.1-arm64-arm-64bit
platform.python_implementation: CPython

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 3.1.1 30 May 2023
ssl.OPENSSL_VERSION_INFO: (3, 1, 0, 1, 0)
ssl.OP_ALL: 0x80000050
ssl.OP_NO_TLSv1_1: 0x10000000
ssl.SSLContext.maximum_version: -1
ssl.SSLContext.minimum_version: 771
ssl.SSLContext.options: 2186412112
ssl.SSLContext.protocol: 16
ssl.SSLContext.verify_mode: 2

ssl.default_https_context.maximum_version: -1
ssl.default_https_context.minimum_version: 771
ssl.default_https_context.options: 2186412112
ssl.default_https_context.protocol: 16
ssl.default_https_context.verify_mode: 2

ssl.stdlib_context.maximum_version: -1
ssl.stdlib_context.minimum_version: 771
ssl.stdlib_context.options: 2186412112
ssl.stdlib_context.protocol: 16
ssl.stdlib_context.verify_mode: 0

Linked PRs

• gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL #107586
• [3.12] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) #107587
• [3.11] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) #107588
• [3.10] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) #107589
• [3.9] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) #107590
• [3.8] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) #107591
• [3.10] Revert "[3.10] gh-107077: Raise SSLCertVerificationError even if the error is set via SSL_ERROR_SYSCALL (GH-107586) (#107589)" #107602

[vstinner]

The test expects ssl.SSLCertVerificationError exception:

with self.assertRaises(ssl.SSLCertVerificationError):

But a ssl.SSLError error is raised instead:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1024)

According to the error message:

• ERR_GET_LIB(e) is ERR_LIB_SSL: SSL: is the error message
• ERR_GET_REASON(e) is SSL_R_CERTIFICATE_VERIFY_FAILED: CERTIFICATE_VERIFY_FAILED is the error message

I'm not sure why PySSL_SetError() of Modules/_ssl.c doesn't go to the type = state->PySSLCertVerificationErrorObject; code path. I suppose that the difference between OpenSSL 3.0 and OpenSSL 3.1 is the err.ssl (sslsock->err.ssl) value. open OpenSSL 3.0, it's SSL_ERROR_SSL. Maybe on OpenSSL 3.1, it's different?

[vstinner]

@pablogsal: Would it be possible that I get a SSH access to this buildbot machine?

[ned-deily]

I believe @gpshead was the most recent person to update the error code handling for the various OpenSSL versions.

[josephshen]

this error location can be narrowed between OpenSSL 3.1.1 and 3.09

[gpshead]

It smells like some error reporting changes within OpenSSL that we just need to be aware of?

Building against openssl 3.1.1 on Linux I cannot reproduce this. Odd. This shouldn't be platform specific AFAICT.

You can trigger a PR test run on a subset of our stable bots via adding the !buildbot regex-of-bot-name text as comment to a PR.

[josephshen]

this error was triggered by OpenSSL commit 00ce831152403589f7f0e558d5291de12bfedc5f

diff --git a/crypto/store/store_lib.c b/crypto/store/store_lib.c
index 5ff9278629..5ac7ff593e 100644
--- a/crypto/store/store_lib.c
+++ b/crypto/store/store_lib.c
@@ -114,13 +114,17 @@ OSSL_STORE_open_ex(const char *uri, OSSL_LIB_CTX *libctx, const char *propq,
         scheme = schemes[i];
         OSSL_TRACE1(STORE, "Looking up scheme %s\n", scheme);
 #ifndef OPENSSL_NO_DEPRECATED_3_0
+        ERR_set_mark();
         if ((loader = ossl_store_get0_loader_int(scheme)) != NULL) {
+            ERR_clear_last_mark();
             no_loader_found = 0;
             if (loader->open_ex != NULL)
                 loader_ctx = loader->open_ex(loader, uri, libctx, propq,
                                              ui_method, ui_data);
             else
                 loader_ctx = loader->open(loader, uri, ui_method, ui_data);
+        } else {
+            ERR_pop_to_mark();
         }
 #endif
         if (loader == NULL

[pablogsal]

@pablogsal: Would it be possible that I get a SSH access to this buildbot machine?

I can try to set https://tmate.io/ because I don't have a stable IP infortunately.

[vstinner]

Thanks for investigating and fixing this ssl issue @pablogsal!

[serhiy-storchaka]

Backports to 3.8 and 3.9 are still not merged.

[ambv]

They are merged now.