-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Click house/openssl 3.0.10 #7
Conversation
Two key 3DES only sets two keys and the random generation errors out if fewer than three keys are required. It shouldn't. Fixes openssl#20212 Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Matt Caswell <[email protected]> (Merged from openssl#20224) (cherry picked from commit 587e040)
BIO_set_accept_name() can return error values -1 and 0 according to my analysis tool and the documentation. Documentation says a value of 1 indicates success. Currently, only an error value != 0 is checked which erroneously interprets a -1 error return value as success. Fix it by changing the check condition. CLA: trivial Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20206) (cherry picked from commit a811b63) (cherry picked from commit 510e493)
Allocate memory for a new SSL session. If any of these steps fail, free the key memory and the tmpsess object before returning 0 to prevent a memory leak. Fixes: openssl#20110 CLA: trivial Reviewed-by: Paul Yang <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20213) (cherry picked from commit 8e2552b) (cherry picked from commit 4cfae92)
Documentation is necessary as static and dynamic linking cause SIGSEGV during atexit() processing on the platform. Fixes: 19951 Signed-off-by: Randall S. Becker <[email protected]> Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#19952) (cherry picked from commit e80518d)
BIO_set_md() can return an error value <= 0 according to my analysis tool and the documentation. But only an error value == 0 is currently checked. Fix it by changing the check condition. CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20195) (cherry picked from commit abf6546) (cherry picked from commit 9b77bd9)
default_check() can return a zero value to indicate an internal error in one condition for the PRE_CTRL_STR_TO_PARAMS state. This state can be reached from the default_fixup_args() function which does not check for a zero value. All other callers of default_check() in that file do check for a zero return value. Fix it by changing the check to <= 0. CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#20175) (cherry picked from commit 650f047)
CMS_SharedInfo_encode() can also return a negative error value, but this is not checked in the current check, only the zero error return value is covered. A previous PR [1] fixed the other caller's check of CMS_SharedInfo_encode in this file, but it seems like this place was missed. Fix it by changing the check to <= 0. [1] openssl@a752fc4 CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#20181) (cherry picked from commit ba06181)
This function returns an errorcode <= 0, but only < 0 is checked. Other callers that check the return value perform this check correctly. Fix it by changing the check to <= 0. CLA: trivial Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#20186) (cherry picked from commit 8263749)
BIO_dup_state() returns an error code <= 0 according to my analysis tool and the documentation. Currently only == 0 is checked. Fix it by changing the check condition. CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20194) (cherry picked from commit 89601c7) (cherry picked from commit cf3cf2b)
_umul128() is x86_64 (x64) only, while __umulh() works everywhere, but doesn't generate optimal code on x64 Reviewed-by: Dmitry Belyavskiy <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#20244) (cherry picked from commit 075652f)
Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20228) (cherry picked from commit 1472127)
Also add corresponding tests and to this end update credentials Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: David von Oheimb <[email protected]> (Merged from openssl#20160) (cherry picked from commit 6b58f49)
Conditioning it on $disabled{shared} isn't right, it will still end up in the static variant of the library. It's better to use SHARED_SOURCE for these sorts of things. Fixes openssl#20238 Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Ben Kaduk <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20240) (cherry picked from commit cd870db)
The test tries to use DES but that may not be available. But for the purpose of regression testing CVE-2023-0215 the cipher is not relevant, so we use AES-128 instead. Fixes openssl#20249 Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20276) (cherry picked from commit c400a1f)
BN_priv_rand_range_ex() and BN_add() both return a 0 on failure and a 1 on success. In case of failure, the algorithm should fail. However, the branch that it goes through on failure is "goto end", not "goto err". Therefore, the algorithm will return 1 which indicates success instead of 0 for failure, leading to potential problems for the callers. Fix it by changing the goto to "goto err" instead of "goto end". CLA: trivial Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Shane Lontis <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#20279) (cherry picked from commit 835b90a) (cherry picked from commit d1e1a8f)
CLA: trivial Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Kurt Roeckx <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20327) (cherry picked from commit 7e55051)
CLA: trivial Reviewed-by: Todd Short <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20310)
There is no point in calling OPENSSL_init_crypto() unless we are actually going to be using the default libctx. Fixes openssl#20315 Reviewed-by: Dmitry Belyavskiy <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20341) (cherry picked from commit 7a6a0ba) (cherry picked from commit ba8e207)
Reviewed-by: Dmitry Belyavskiy <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20341) (cherry picked from commit 0aa7d7f) (cherry picked from commit ed8d2c9)
CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20294) (cherry picked from commit ab5a172)
CLA: trivial Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20351) (cherry picked from commit 7fed519)
Reviewed-by: Richard Levitte <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20312) (cherry picked from commit 1dc35d4)
TLS pipelining provides the ability for libssl to read or write multiple records in parallel. It requires special ciphers to do this, and there are currently no built-in ciphers that provide this capability. However, the dasync engine does have such a cipher, so we add a test for this capability using that engine. Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20208) (cherry picked from commit 24c7d36)
…lled The pipeline input/output buf arrays must remain accessible to the EVP_CIPHER_CTX until EVP_Cipher is subsequently called. This fixes an asan error discovered by the newly added pipeline test. Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20208) (cherry picked from commit df9c7ce)
During read pipelining we must ensure that the buffer is sufficiently large to read enough data to fill our pipelines. We also remove some code that moved data to the start of the packet if we can. This was unnecessary because of later code which would end up moving it anyway. The earlier move was also incorrect in the case that |clearold| was 0. This would cause the read pipelining code to fail with sufficiently large records. Fixes openssl#20197 Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20208) (cherry picked from commit 1d06598)
Document the effect on the internal read buffer when using pipelining. Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20208) (cherry picked from commit 89ed544)
We shouldn't be putting more data into a pipeline than the value of split_send_fragment. This is a backport of a fix which was included in a much larger commit in master (c618679) related to moving the pipelining code into the new record layer that exists there. Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Paul Dale <[email protected]> (Merged from openssl#20208) (cherry picked from commit 2c4b1c7)
Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: David von Oheimb <[email protected]> (Merged from openssl#20275)
… improvements Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: David von Oheimb <[email protected]> (Merged from openssl#20275)
Also document CMS_decrypt_set1_password() and fix CMS_EnvelopedData_create.pod. Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: David von Oheimb <[email protected]> (Merged from openssl#20209)
Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#21515) (cherry picked from commit bc5d9cc)
The pre-existing error cases where DH_check returned zero are not related to the dh params in any way, but are only triggered by out-of-memory errors, therefore having *ret set to zero feels right, but since the new error case is triggered by too large p values that is something different. On the other hand some callers of this function might not be prepared to handle the return value correctly but only rely on *ret. Therefore we set some error bits in *ret as additional safety measure. Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#21524) (cherry picked from commit 81d10e6)
If |q| >= |p| then the q value is obviously wrong as q is supposed to be a prime divisor of p-1. We check if p is overly large so this added test implies that q is not large either when performing subsequent tests using that q value. Otherwise if it is too large these additional checks of the q value such as the primality test can then trigger DoS by doing overly long computations. Fixes CVE-2023-3817 Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#21550) (cherry picked from commit 1c16253) (cherry picked from commit 6a1eb62)
This must fail with DH_CHECK_INVALID_Q_VALUE and with DH_CHECK_Q_NOT_PRIME unset. Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#21550) (cherry picked from commit ad5d355) (cherry picked from commit 1478ffa)
Reviewed-by: Matt Caswell <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#21550) (cherry picked from commit 4b29762) (cherry picked from commit fb54f41)
Apply normal sentence case to db update message CLA: trivial Reviewed-by: Hugo Landau <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Todd Short <[email protected]> (Merged from openssl#21535) (cherry picked from commit ccb2f30) (cherry picked from commit 4c1fc2c)
Also fixes a similar regression in X509_VERIFY_PARAM_add0_table(). Commit 38ebfc3 introduced a regression in 3.0.6 that changed the return value of the two functions above from 1 on success to the number of entries in the stack. If there are more than one entry then this is a change in behaviour which should not have been introduced into a stable release. This reverts the behaviour back to what it was prior to the change. The code is slightly different to the original code in that we also handle a possible -1 return value from the stack push function. This should never happen in reality because we never pass a NULL stack as a parameter - but for the sake of robustness we handle it anyway. Note that the changed behaviour exists in all versions of 3.1 (it never had the original version). But 3.1 should be fully backwards compatible with 3.0 so we should change it there too. Fixes openssl#21570 Reviewed-by: Paul Dale <[email protected]> Reviewed-by: Tom Cosgrove <[email protected]> (Merged from openssl#21576) (cherry picked from commit e3d897d)
Reviewed-by: Tomas Mraz <[email protected]> Release: yes
Reviewed-by: Tomas Mraz <[email protected]> Release: yes
Reviewed-by: Tomas Mraz <[email protected]> Release: yes
|
----- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E openssl#9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E openssl#10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E openssl#11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E openssl#12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E openssl#16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E openssl#17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E openssl#18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E openssl#20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E openssl#21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E openssl#23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E openssl#24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E openssl#25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E openssl#26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E openssl#27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E openssl#29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E openssl#30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E openssl#10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E openssl#12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E openssl#15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E openssl#17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E openssl#18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e)
Sometimes the error handling returns an ASN1_STRING object in *out although that was not passed in by the caller, and sometimes the error handling deletes the ASN1_STRING but forgets to clear the *out parameter. Therefore the caller has no chance to know, if the leaked object in *out shall be deleted or not. This may cause a use-after-free error e.g. in asn1_str2type: ==63312==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000073280 at pc 0x7f2652e93b08 bp 0x7ffe0e1951c0 sp 0x7ffe0e1951b0 READ of size 8 at 0x603000073280 thread T0 #0 0x7f2652e93b07 in asn1_string_embed_free crypto/asn1/asn1_lib.c:354 #1 0x7f2652eb521a in asn1_primitive_free crypto/asn1/tasn_fre.c:204 #2 0x7f2652eb50a9 in asn1_primitive_free crypto/asn1/tasn_fre.c:199 #3 0x7f2652eb5b67 in ASN1_item_free crypto/asn1/tasn_fre.c:20 #4 0x7f2652e8e13b in asn1_str2type crypto/asn1/asn1_gen.c:740 #5 0x7f2652e8e13b in generate_v3 crypto/asn1/asn1_gen.c:137 #6 0x7f2652e9166c in ASN1_generate_v3 crypto/asn1/asn1_gen.c:92 #7 0x7f2653307b9b in do_othername crypto/x509v3/v3_alt.c:577 #8 0x7f2653307b9b in a2i_GENERAL_NAME crypto/x509v3/v3_alt.c:492 openssl#9 0x7f26533087c2 in v2i_subject_alt crypto/x509v3/v3_alt.c:327 openssl#10 0x7f26533107fc in do_ext_nconf crypto/x509v3/v3_conf.c:100 openssl#11 0x7f2653310f33 in X509V3_EXT_nconf crypto/x509v3/v3_conf.c:45 openssl#12 0x7f2653311426 in X509V3_EXT_add_nconf_sk crypto/x509v3/v3_conf.c:312 openssl#13 0x7f265331170c in X509V3_EXT_REQ_add_nconf crypto/x509v3/v3_conf.c:360 openssl#14 0x564ed19d5f25 in req_main apps/req.c:806 openssl#15 0x564ed19b8de0 in do_cmd apps/openssl.c:564 openssl#16 0x564ed1985165 in main apps/openssl.c:183 openssl#17 0x7f2651c4a082 in __libc_start_main ../csu/libc-start.c:308 openssl#18 0x564ed1985acd in _start (/home/ed/OPCToolboxV5/Source/Core/OpenSSL/openssl/apps/openssl+0x139acd) 0x603000073280 is located 16 bytes inside of 24-byte region [0x603000073270,0x603000073288) freed by thread T0 here: #0 0x7f265413440f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:122 #1 0x7f265315a429 in CRYPTO_free crypto/mem.c:311 #2 0x7f265315a429 in CRYPTO_free crypto/mem.c:300 #3 0x7f2652e757b9 in ASN1_mbstring_ncopy crypto/asn1/a_mbstr.c:191 #4 0x7f2652e75ec5 in ASN1_mbstring_copy crypto/asn1/a_mbstr.c:38 #5 0x7f2652e8e227 in asn1_str2type crypto/asn1/asn1_gen.c:681 #6 0x7f2652e8e227 in generate_v3 crypto/asn1/asn1_gen.c:137 #7 0x7f2652e9166c in ASN1_generate_v3 crypto/asn1/asn1_gen.c:92 #8 0x7f2653307b9b in do_othername crypto/x509v3/v3_alt.c:577 openssl#9 0x7f2653307b9b in a2i_GENERAL_NAME crypto/x509v3/v3_alt.c:492 openssl#10 0x7f26533087c2 in v2i_subject_alt crypto/x509v3/v3_alt.c:327 openssl#11 0x7f26533107fc in do_ext_nconf crypto/x509v3/v3_conf.c:100 openssl#12 0x7f2653310f33 in X509V3_EXT_nconf crypto/x509v3/v3_conf.c:45 openssl#13 0x7f2653311426 in X509V3_EXT_add_nconf_sk crypto/x509v3/v3_conf.c:312 openssl#14 0x7f265331170c in X509V3_EXT_REQ_add_nconf crypto/x509v3/v3_conf.c:360 openssl#15 0x564ed19d5f25 in req_main apps/req.c:806 openssl#16 0x564ed19b8de0 in do_cmd apps/openssl.c:564 openssl#17 0x564ed1985165 in main apps/openssl.c:183 openssl#18 0x7f2651c4a082 in __libc_start_main ../csu/libc-start.c:308 previously allocated by thread T0 here: #0 0x7f2654134808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x7f265315a4fd in CRYPTO_malloc crypto/mem.c:221 #2 0x7f265315a4fd in CRYPTO_malloc crypto/mem.c:198 #3 0x7f265315a945 in CRYPTO_zalloc crypto/mem.c:236 #4 0x7f2652e939a4 in ASN1_STRING_type_new crypto/asn1/asn1_lib.c:341 #5 0x7f2652e74e51 in ASN1_mbstring_ncopy crypto/asn1/a_mbstr.c:150 #6 0x7f2652e75ec5 in ASN1_mbstring_copy crypto/asn1/a_mbstr.c:38 #7 0x7f2652e8e227 in asn1_str2type crypto/asn1/asn1_gen.c:681 #8 0x7f2652e8e227 in generate_v3 crypto/asn1/asn1_gen.c:137 openssl#9 0x7f2652e9166c in ASN1_generate_v3 crypto/asn1/asn1_gen.c:92 openssl#10 0x7f2653307b9b in do_othername crypto/x509v3/v3_alt.c:577 openssl#11 0x7f2653307b9b in a2i_GENERAL_NAME crypto/x509v3/v3_alt.c:492 openssl#12 0x7f26533087c2 in v2i_subject_alt crypto/x509v3/v3_alt.c:327 openssl#13 0x7f26533107fc in do_ext_nconf crypto/x509v3/v3_conf.c:100 openssl#14 0x7f2653310f33 in X509V3_EXT_nconf crypto/x509v3/v3_conf.c:45 openssl#15 0x7f2653311426 in X509V3_EXT_add_nconf_sk crypto/x509v3/v3_conf.c:312 openssl#16 0x7f265331170c in X509V3_EXT_REQ_add_nconf crypto/x509v3/v3_conf.c:360 openssl#17 0x564ed19d5f25 in req_main apps/req.c:806 openssl#18 0x564ed19b8de0 in do_cmd apps/openssl.c:564 openssl#19 0x564ed1985165 in main apps/openssl.c:183 openssl#20 0x7f2651c4a082 in __libc_start_main ../csu/libc-start.c:308 Reviewed-by: Paul Yang <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#23138)
if the private key is output to stdout using the HARNESS_OSSL_PREFIX, out is a stack of BIOs and must therefore free'd using BIO_free_all. Steps to reproduce: $ HARNESS_OSSL_PREFIX=x OPENSSL_CONF=apps/openssl.cnf util/shlib_wrap.sh apps/openssl req -new -keyout - -passout pass: </dev/null [...] Direct leak of 128 byte(s) in 1 object(s) allocated from: #0 0x7f6f692b89cf in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69 #1 0x7f6f686eda00 in CRYPTO_malloc crypto/mem.c:202 #2 0x7f6f686edba0 in CRYPTO_zalloc crypto/mem.c:222 #3 0x7f6f68471bdf in BIO_new_ex crypto/bio/bio_lib.c:83 #4 0x7f6f68491a8f in BIO_new_fp crypto/bio/bss_file.c:95 #5 0x555c5f58b378 in dup_bio_out apps/lib/apps.c:3014 #6 0x555c5f58f9ac in bio_open_default_ apps/lib/apps.c:3175 #7 0x555c5f58f9ac in bio_open_default apps/lib/apps.c:3203 #8 0x555c5f528537 in req_main apps/req.c:683 openssl#9 0x555c5f50e315 in do_cmd apps/openssl.c:426 openssl#10 0x555c5f4c5575 in main apps/openssl.c:307 openssl#11 0x7f6f680461c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 SUMMARY: AddressSanitizer: 128 byte(s) leaked in 1 allocation(s). Reviewed-by: Tom Cosgrove <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#23365)
The following issue was found in automatic tests with thread sanitizer builds in ClickHouse (which uses OpenSSL 3.2.1): The first stack does proper locking (function 'x509_store_add', x509_lu.c) but in the second stack, function 'get_cert_by_subject_ex' (by_dir.b) forgets to lock when calling 'sk_X509_OBJECT_is_sorted'. ``` WARNING: ThreadSanitizer: data race (pid=1870) Write of size 4 at 0x7b08003d6810 by thread T552 (mutexes: write M0, write M1, write M2, write M3): #0 OPENSSL_sk_insert build_docker/./contrib/openssl/crypto/stack/stack.c:280:16 (clickhouse+0x203ad7e4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 OPENSSL_sk_push build_docker/./contrib/openssl/crypto/stack/stack.c:401:12 (clickhouse+0x203ad7e4) #2 x509_store_add build_docker/./contrib/openssl/crypto/x509/x509_lu.c:419:17 (clickhouse+0x203d4a52) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 X509_STORE_add_cert build_docker/./contrib/openssl/crypto/x509/x509_lu.c:432:10 (clickhouse+0x203d48a2) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #4 X509_load_cert_file_ex build_docker/./contrib/openssl/crypto/x509/by_file.c:127:18 (clickhouse+0x203b74e6) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:333:22 (clickhouse+0x203b684c) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #7 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #8 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) openssl#11 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#16 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#20 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#21 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#22 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#23 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) ``` ``` Previous read of size 4 at 0x7b08003d6810 by thread T553 (mutexes: write M4, write M5, write M6): #0 OPENSSL_sk_is_sorted build_docker/./contrib/openssl/crypto/stack/stack.c:490:33 (clickhouse+0x203adcff) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:423:10 (clickhouse+0x203b6d8f) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #2 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #4 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) #7 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #8 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#11 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#16 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) ```
The following issue was found in automatic tests with thread sanitizer builds in ClickHouse (which uses OpenSSL 3.2.1) [0]. The first stack [1] does proper locking (function 'x509_store_add', x509_lu.c) but in the second stack [2], function 'get_cert_by_subject_ex' (by_dir.b) forgets to lock when calling 'sk_X509_OBJECT_is_sorted'. [0] ClickHouse/ClickHouse#63049 [1] WARNING: ThreadSanitizer: data race (pid=1870) Write of size 4 at 0x7b08003d6810 by thread T552 (mutexes: write M0, write M1, write M2, write M3): #0 OPENSSL_sk_insert build_docker/./contrib/openssl/crypto/stack/stack.c:280:16 (clickhouse+0x203ad7e4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 OPENSSL_sk_push build_docker/./contrib/openssl/crypto/stack/stack.c:401:12 (clickhouse+0x203ad7e4) #2 x509_store_add build_docker/./contrib/openssl/crypto/x509/x509_lu.c:419:17 (clickhouse+0x203d4a52) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 X509_STORE_add_cert build_docker/./contrib/openssl/crypto/x509/x509_lu.c:432:10 (clickhouse+0x203d48a2) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #4 X509_load_cert_file_ex build_docker/./contrib/openssl/crypto/x509/by_file.c:127:18 (clickhouse+0x203b74e6) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:333:22 (clickhouse+0x203b684c) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #7 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #8 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) openssl#11 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#16 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#20 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#21 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#22 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#23 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) [2] Previous read of size 4 at 0x7b08003d6810 by thread T553 (mutexes: write M4, write M5, write M6): #0 OPENSSL_sk_is_sorted build_docker/./contrib/openssl/crypto/stack/stack.c:490:33 (clickhouse+0x203adcff) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:423:10 (clickhouse+0x203b6d8f) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #2 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #4 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) #7 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #8 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#11 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#16 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) CLA: trivial
During my testing (*), I got below stack about leaked memory. - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_public) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. (*) ClickHouse/clickhouse-private#10107 (comment) ``` ================================================================= ==2757181==ERROR: LeakSanitizer: detected memory leaks Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x5672f4a48083 in malloc (/data/ch4/build_asan/programs/clickhouse+0xa2b7083) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) #1 0x567321d05c7e in CRYPTO_malloc build_asan/./contrib/openssl/crypto/mem.c:202:11 #2 0x567321d05c7e in CRYPTO_zalloc build_asan/./contrib/openssl/crypto/mem.c:222:11 #3 0x567321d5f9cf in EVP_RAND_CTX_new build_asan/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x567321d627db in rand_new_drbg build_asan/./contrib/openssl/crypto/rand/rand_lib.c:658:11 #5 0x567321d61e7b in RAND_get0_public build_asan/./contrib/openssl/crypto/rand/rand_lib.c:777:16 #6 0x567321d61d4f in RAND_bytes_ex build_asan/./contrib/openssl/crypto/rand/rand_lib.c:378:12 #7 0x56731cfe9267 in ossl_random openssl.c #8 0x56731cf534ae in Curl_rand (/data/ch4/build_asan/programs/clickhouse+0x327c24ae) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#9 0x56731cf757dc in Curl_socketpair (/data/ch4/build_asan/programs/clickhouse+0x327e47dc) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#10 0x56731cf40bfe in Curl_multi_handle (/data/ch4/build_asan/programs/clickhouse+0x327afbfe) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#11 0x56731cecb260 in curl_easy_perform (/data/ch4/build_asan/programs/clickhouse+0x3273a260) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#12 0x56731cd4a351 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#13 0x56731cd3176b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::H ttp::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#14 0x56731cd3176b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#15 0x56731cd2f891 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#16 0x56731cdd66db in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#17 0x56731cd7c75b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#18 0x56731cdd0218 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#19 0x56731cec5f18 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#20 0x56731cec6d0b in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#21 0x56731cdcd019 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#22 0x56731cdd6e17 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#23 0x56731cdb0ef5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#24 0x56731cdfc53e in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#25 0x56731ce4bff1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#26 0x56731ce4bff1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#27 0x56731ce0bdd8 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 openssl#28 0x56731ce396dc in Azure::Storage::Blobs::BlobServiceClient::CreateBlobContainer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_service_client.cpp:264:41 openssl#29 0x56731018fcff in DB::getAzureBlobContainerClient(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./src/Disks/ObjectStorages/AzureBlobStorage/AzureBlobStorageAuth.cpp:236:75 openssl#30 0x5673115a7352 in DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:315:13 openssl#31 0x5673115a7352 in decltype(std::declval<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const> const&>(), std::declval<bool>() )) std::__1::__invoke[abi:v15000]<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#32 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IObjectStorage>, false>::__call<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_pt r<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#33 0x5673115a7352 in std::__1::__function::__default_alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v 15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 openssl#34 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__function::__policy_invoker<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::__call_impl<std::__1::__function::__default _alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>>(std::__1::__function::__policy_storage const*, std::__1::basic_string<cha r, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 openssl#35 0x5673115a0780 in std::__1::__function::__policy_func<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::_ _1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 openssl#36 0x5673115a0780 in std::__1::function<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 openssl#37 0x5673115a0780 in DB::ObjectStorageFactory::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:135:12 openssl#38 0x56731159ef45 in DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1: :shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool) const build_asan/./src/Disks/ObjectStorages/RegisterDiskObjectStorage.cpp:27:64 openssl#39 0x56731159ef45 in decltype(std::declval<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const>>(), std::declval<std::__1::map<std: :__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&>(), std::declval<bool>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::regis terDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::_ _1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_ string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#40 0x56731159ef45 in std::__1::shared_ptr<DB::IDisk> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IDisk>, false>::__call<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0 &, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<ch ar>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#41 0x56731159ef45 in std::__1::__function::__default_alloc_func<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0, std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, s td::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator <char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::b asic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 ```
The leak was reported in ClickHouse/ClickHouse#63792 - a SSL session is established - somewhere down in OpenSSL, the per-thread error queue is clear'ed - when no error queue exists (e.g. because the thread was just spawned), a new error queue is allocated - the error queue is stored in TLS, and a free handler is registered which deletes it when the thread ends - Leak sanitizer does not understand ^^ (false positive) ``` Direct leak of 9944 byte(s) in 11 object(s) allocated from: #0 0x5571bf7e04cf in malloc (/usr/bin/clickhouse+0xa39a4cf) (BuildId: 06c4931100f632dde9d0ecd3cf6a67776742e29b) #1 0x5571ecbc06be in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x5571ecbc06be in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x5571ecb03c6a in ossl_err_get_state_int build_docker/./contrib/openssl/crypto/err/err.c:691:17 #4 0x5571ecb037b5 in ERR_clear_error build_docker/./contrib/openssl/crypto/err/err.c:339:10 #5 0x5571ec9cf0d0 in state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:366:5 #6 0x5571ec90c7d4 in SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 #7 0x5571e70cc501 in Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 #8 0x5571e70cce6b in Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:149:2 openssl#9 0x5571e70d81f7 in Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:87:8 openssl#10 0x5571e0ca708a in DB::Connection::connect(DB::ConnectionTimeouts const&) build_docker/./src/Client/Connection.cpp:158:29 ```
ClickHouse issue ClickHouse/clickhouse-private#10107 (comment) reports this memory leak: (note how it is very similar to ClickHouse/clickhouse-private#10107 (comment)) ``` Direct leak of 528 byte(s) in 11 object(s) allocated from: #0 0x561369af24cf in malloc (/usr/bin/clickhouse+0xa6cf4cf) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) #1 0x561397b86a7e in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x561397b86a7e in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x561397be07cf in EVP_RAND_CTX_new build_docker/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x561397be35fb in rand_new_drbg build_docker/./contrib/openssl/crypto/rand/rand_lib.c:665:11 #5 0x561397be2a9b in RAND_get0_private build_docker/./contrib/openssl/crypto/rand/rand_lib.c:827:16 #6 0x561397be296f in RAND_priv_bytes_ex build_docker/./contrib/openssl/crypto/rand/rand_lib.c:356:12 #7 0x5613978d7cd8 in SSL_CTX_new_ex build_docker/./contrib/openssl/ssl/ssl_lib.c:4016:13 #8 0x561392d1eabd in ossl_connect_common openssl.c openssl#9 0x561392d11e23 in ssl_cf_connect vtls.c openssl#10 0x561392c13a1f in cf_setup_connect connect.c openssl#11 0x561392c1b0e5 in cf_hc_connect cf-https-connect.c openssl#12 0x561392c0a332 in Curl_conn_connect (/usr/bin/clickhouse+0x337e7332) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#13 0x561392c7a28a in multi_runsingle multi.c openssl#14 0x561392c78f6d in curl_multi_perform (/usr/bin/clickhouse+0x33855f6d) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#15 0x561392bfd53e in curl_easy_perform (/usr/bin/clickhouse+0x337da53e) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#16 0x561392a7c411 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#17 0x561392a6382b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#18 0x561392a6382b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#19 0x561392a61951 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#20 0x561392b0879b in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#21 0x561392aae81b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#22 0x561392b022d8 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#23 0x561392b2e02a in Azure::Storage::_internal::SharedKeyPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/shared_key_policy.hpp:36:25 openssl#24 0x561392bf7fd8 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#25 0x561392bf8dcb in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#26 0x561392aff0d9 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#27 0x561392b08ed7 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#28 0x561392ae2fb5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#29 0x561392b2e5fe in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#30 0x561392b7e0b1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#31 0x561392b7e0b1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#32 0x561392b3de98 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 ``` - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_private) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. As a result, the fix is similar to earlier fix 5c4b034
ClickHouse issue ClickHouse/clickhouse-private#10107 (comment) reports this memory leak: (note how it is very similar to ClickHouse/clickhouse-private#10107 (comment)) ``` Direct leak of 528 byte(s) in 11 object(s) allocated from: #0 0x561369af24cf in malloc (/usr/bin/clickhouse+0xa6cf4cf) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) #1 0x561397b86a7e in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x561397b86a7e in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x561397be07cf in EVP_RAND_CTX_new build_docker/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x561397be35fb in rand_new_drbg build_docker/./contrib/openssl/crypto/rand/rand_lib.c:665:11 #5 0x561397be2a9b in RAND_get0_private build_docker/./contrib/openssl/crypto/rand/rand_lib.c:827:16 #6 0x561397be296f in RAND_priv_bytes_ex build_docker/./contrib/openssl/crypto/rand/rand_lib.c:356:12 #7 0x5613978d7cd8 in SSL_CTX_new_ex build_docker/./contrib/openssl/ssl/ssl_lib.c:4016:13 #8 0x561392d1eabd in ossl_connect_common openssl.c openssl#9 0x561392d11e23 in ssl_cf_connect vtls.c openssl#10 0x561392c13a1f in cf_setup_connect connect.c openssl#11 0x561392c1b0e5 in cf_hc_connect cf-https-connect.c openssl#12 0x561392c0a332 in Curl_conn_connect (/usr/bin/clickhouse+0x337e7332) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#13 0x561392c7a28a in multi_runsingle multi.c openssl#14 0x561392c78f6d in curl_multi_perform (/usr/bin/clickhouse+0x33855f6d) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#15 0x561392bfd53e in curl_easy_perform (/usr/bin/clickhouse+0x337da53e) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#16 0x561392a7c411 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#17 0x561392a6382b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#18 0x561392a6382b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#19 0x561392a61951 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#20 0x561392b0879b in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#21 0x561392aae81b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#22 0x561392b022d8 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#23 0x561392b2e02a in Azure::Storage::_internal::SharedKeyPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/shared_key_policy.hpp:36:25 openssl#24 0x561392bf7fd8 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#25 0x561392bf8dcb in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#26 0x561392aff0d9 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#27 0x561392b08ed7 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#28 0x561392ae2fb5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#29 0x561392b2e5fe in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#30 0x561392b7e0b1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#31 0x561392b7e0b1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#32 0x561392b3de98 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 ``` - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_private) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. As a result, the fix is similar to earlier fix 5c4b034
The following issue was found in automatic tests with thread sanitizer builds in ClickHouse (which uses OpenSSL 3.2.1) [0]. The first stack [1] does proper locking (function 'x509_store_add', x509_lu.c) but in the second stack [2], function 'get_cert_by_subject_ex' (by_dir.b) forgets to lock when calling 'sk_X509_OBJECT_is_sorted'. [0] ClickHouse/ClickHouse#63049 [1] WARNING: ThreadSanitizer: data race (pid=1870) Write of size 4 at 0x7b08003d6810 by thread T552 (mutexes: write M0, write M1, write M2, write M3): #0 OPENSSL_sk_insert build_docker/./contrib/openssl/crypto/stack/stack.c:280:16 (clickhouse+0x203ad7e4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 OPENSSL_sk_push build_docker/./contrib/openssl/crypto/stack/stack.c:401:12 (clickhouse+0x203ad7e4) #2 x509_store_add build_docker/./contrib/openssl/crypto/x509/x509_lu.c:419:17 (clickhouse+0x203d4a52) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 X509_STORE_add_cert build_docker/./contrib/openssl/crypto/x509/x509_lu.c:432:10 (clickhouse+0x203d48a2) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #4 X509_load_cert_file_ex build_docker/./contrib/openssl/crypto/x509/by_file.c:127:18 (clickhouse+0x203b74e6) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:333:22 (clickhouse+0x203b684c) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #7 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #8 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) openssl#11 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#16 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#20 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#21 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#22 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#23 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) [2] Previous read of size 4 at 0x7b08003d6810 by thread T553 (mutexes: write M4, write M5, write M6): #0 OPENSSL_sk_is_sorted build_docker/./contrib/openssl/crypto/stack/stack.c:490:33 (clickhouse+0x203adcff) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #1 get_cert_by_subject_ex build_docker/./contrib/openssl/crypto/x509/by_dir.c:423:10 (clickhouse+0x203b6d8f) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #2 X509_LOOKUP_by_subject_ex build_docker/./contrib/openssl/crypto/x509/x509_lu.c:105:16 (clickhouse+0x203d46ec) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #3 ossl_x509_store_ctx_get_by_subject build_docker/./contrib/openssl/crypto/x509/x509_lu.c:360:17 (clickhouse+0x203d46ec) #4 X509_STORE_CTX_get1_issuer build_docker/./contrib/openssl/crypto/x509/x509_lu.c:782:10 (clickhouse+0x203d56cb) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #5 get1_trusted_issuer build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3194:10 (clickhouse+0x203db4a9) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #6 build_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:3324:40 (clickhouse+0x203db4a9) #7 verify_chain build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:240:15 (clickhouse+0x203dbe27) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) #8 x509_verify_x509 build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:358 (clickhouse+0x203d7fd8) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#9 X509_verify_cert build_docker/./contrib/openssl/crypto/x509/x509_vfy.c:293:56 (clickhouse+0x203d8215) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#10 ssl_verify_internal build_docker/./contrib/openssl/ssl/ssl_cert.c:496:13 (clickhouse+0x2019a2a4) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#11 ssl_verify_cert_chain build_docker/./contrib/openssl/ssl/ssl_cert.c:543:12 (clickhouse+0x2019a402) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#12 tls_post_process_server_certificate build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:2072:9 (clickhouse+0x20227658) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#13 ossl_statem_client_post_process_message build_docker/./contrib/openssl/ssl/statem/statem_clnt.c:1159:16 (clickhouse+0x202272ee) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#14 read_state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:712:35 (clickhouse+0x2021e96d) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#15 state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:478:21 (clickhouse+0x2021e96d) openssl#16 ossl_statem_connect build_docker/./contrib/openssl/ssl/statem/statem.c:297:12 (clickhouse+0x2021ddce) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#17 SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 (clickhouse+0x201a5781) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#18 SSL_connect build_docker/./contrib/openssl/ssl/ssl_lib.c:2208:12 (clickhouse+0x201a5893) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) openssl#19 Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 (clickhouse+0x1d179567) (BuildId: 3ceefd39df36d762f06bf9aab19cfc3467e4558b) CLA: trivial Reviewed-by: Todd Short <[email protected]> Reviewed-by: Tomas Mraz <[email protected]> (Merged from openssl#24295) (cherry picked from commit af75373)
----- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E openssl#9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E openssl#10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E openssl#11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E openssl#12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E openssl#16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E openssl#17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E openssl#18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E openssl#20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E openssl#21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E openssl#23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E openssl#24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E openssl#25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E openssl#26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E openssl#27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E openssl#29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E openssl#30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E openssl#10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E openssl#12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E openssl#15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E openssl#17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E openssl#18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e)
During my testing (*), I got below stack about leaked memory. - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_public) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. (*) ClickHouse/clickhouse-private#10107 (comment) ``` ================================================================= ==2757181==ERROR: LeakSanitizer: detected memory leaks Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x5672f4a48083 in malloc (/data/ch4/build_asan/programs/clickhouse+0xa2b7083) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) #1 0x567321d05c7e in CRYPTO_malloc build_asan/./contrib/openssl/crypto/mem.c:202:11 #2 0x567321d05c7e in CRYPTO_zalloc build_asan/./contrib/openssl/crypto/mem.c:222:11 #3 0x567321d5f9cf in EVP_RAND_CTX_new build_asan/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x567321d627db in rand_new_drbg build_asan/./contrib/openssl/crypto/rand/rand_lib.c:658:11 #5 0x567321d61e7b in RAND_get0_public build_asan/./contrib/openssl/crypto/rand/rand_lib.c:777:16 #6 0x567321d61d4f in RAND_bytes_ex build_asan/./contrib/openssl/crypto/rand/rand_lib.c:378:12 #7 0x56731cfe9267 in ossl_random openssl.c #8 0x56731cf534ae in Curl_rand (/data/ch4/build_asan/programs/clickhouse+0x327c24ae) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#9 0x56731cf757dc in Curl_socketpair (/data/ch4/build_asan/programs/clickhouse+0x327e47dc) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#10 0x56731cf40bfe in Curl_multi_handle (/data/ch4/build_asan/programs/clickhouse+0x327afbfe) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#11 0x56731cecb260 in curl_easy_perform (/data/ch4/build_asan/programs/clickhouse+0x3273a260) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#12 0x56731cd4a351 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#13 0x56731cd3176b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::H ttp::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#14 0x56731cd3176b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#15 0x56731cd2f891 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#16 0x56731cdd66db in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#17 0x56731cd7c75b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#18 0x56731cdd0218 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#19 0x56731cec5f18 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#20 0x56731cec6d0b in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#21 0x56731cdcd019 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#22 0x56731cdd6e17 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#23 0x56731cdb0ef5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#24 0x56731cdfc53e in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#25 0x56731ce4bff1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#26 0x56731ce4bff1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#27 0x56731ce0bdd8 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 openssl#28 0x56731ce396dc in Azure::Storage::Blobs::BlobServiceClient::CreateBlobContainer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_service_client.cpp:264:41 openssl#29 0x56731018fcff in DB::getAzureBlobContainerClient(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./src/Disks/ObjectStorages/AzureBlobStorage/AzureBlobStorageAuth.cpp:236:75 openssl#30 0x5673115a7352 in DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:315:13 openssl#31 0x5673115a7352 in decltype(std::declval<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const> const&>(), std::declval<bool>() )) std::__1::__invoke[abi:v15000]<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#32 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IObjectStorage>, false>::__call<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_pt r<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#33 0x5673115a7352 in std::__1::__function::__default_alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v 15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 openssl#34 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__function::__policy_invoker<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::__call_impl<std::__1::__function::__default _alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>>(std::__1::__function::__policy_storage const*, std::__1::basic_string<cha r, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 openssl#35 0x5673115a0780 in std::__1::__function::__policy_func<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::_ _1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 openssl#36 0x5673115a0780 in std::__1::function<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 openssl#37 0x5673115a0780 in DB::ObjectStorageFactory::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:135:12 openssl#38 0x56731159ef45 in DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1: :shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool) const build_asan/./src/Disks/ObjectStorages/RegisterDiskObjectStorage.cpp:27:64 openssl#39 0x56731159ef45 in decltype(std::declval<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const>>(), std::declval<std::__1::map<std: :__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&>(), std::declval<bool>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::regis terDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::_ _1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_ string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#40 0x56731159ef45 in std::__1::shared_ptr<DB::IDisk> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IDisk>, false>::__call<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0 &, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<ch ar>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#41 0x56731159ef45 in std::__1::__function::__default_alloc_func<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0, std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, s td::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator <char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::b asic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 ```
The leak was reported in ClickHouse/ClickHouse#63792 - a SSL session is established - somewhere down in OpenSSL, the per-thread error queue is clear'ed - when no error queue exists (e.g. because the thread was just spawned), a new error queue is allocated - the error queue is stored in TLS, and a free handler is registered which deletes it when the thread ends - Leak sanitizer does not understand ^^ (false positive) ``` Direct leak of 9944 byte(s) in 11 object(s) allocated from: #0 0x5571bf7e04cf in malloc (/usr/bin/clickhouse+0xa39a4cf) (BuildId: 06c4931100f632dde9d0ecd3cf6a67776742e29b) #1 0x5571ecbc06be in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x5571ecbc06be in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x5571ecb03c6a in ossl_err_get_state_int build_docker/./contrib/openssl/crypto/err/err.c:691:17 #4 0x5571ecb037b5 in ERR_clear_error build_docker/./contrib/openssl/crypto/err/err.c:339:10 #5 0x5571ec9cf0d0 in state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:366:5 #6 0x5571ec90c7d4 in SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 #7 0x5571e70cc501 in Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 #8 0x5571e70cce6b in Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:149:2 openssl#9 0x5571e70d81f7 in Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:87:8 openssl#10 0x5571e0ca708a in DB::Connection::connect(DB::ConnectionTimeouts const&) build_docker/./src/Client/Connection.cpp:158:29 ```
ClickHouse issue ClickHouse/clickhouse-private#10107 (comment) reports this memory leak: (note how it is very similar to ClickHouse/clickhouse-private#10107 (comment)) ``` Direct leak of 528 byte(s) in 11 object(s) allocated from: #0 0x561369af24cf in malloc (/usr/bin/clickhouse+0xa6cf4cf) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) #1 0x561397b86a7e in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x561397b86a7e in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x561397be07cf in EVP_RAND_CTX_new build_docker/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x561397be35fb in rand_new_drbg build_docker/./contrib/openssl/crypto/rand/rand_lib.c:665:11 #5 0x561397be2a9b in RAND_get0_private build_docker/./contrib/openssl/crypto/rand/rand_lib.c:827:16 #6 0x561397be296f in RAND_priv_bytes_ex build_docker/./contrib/openssl/crypto/rand/rand_lib.c:356:12 #7 0x5613978d7cd8 in SSL_CTX_new_ex build_docker/./contrib/openssl/ssl/ssl_lib.c:4016:13 #8 0x561392d1eabd in ossl_connect_common openssl.c openssl#9 0x561392d11e23 in ssl_cf_connect vtls.c openssl#10 0x561392c13a1f in cf_setup_connect connect.c openssl#11 0x561392c1b0e5 in cf_hc_connect cf-https-connect.c openssl#12 0x561392c0a332 in Curl_conn_connect (/usr/bin/clickhouse+0x337e7332) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#13 0x561392c7a28a in multi_runsingle multi.c openssl#14 0x561392c78f6d in curl_multi_perform (/usr/bin/clickhouse+0x33855f6d) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#15 0x561392bfd53e in curl_easy_perform (/usr/bin/clickhouse+0x337da53e) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#16 0x561392a7c411 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#17 0x561392a6382b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#18 0x561392a6382b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#19 0x561392a61951 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#20 0x561392b0879b in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#21 0x561392aae81b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#22 0x561392b022d8 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#23 0x561392b2e02a in Azure::Storage::_internal::SharedKeyPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/shared_key_policy.hpp:36:25 openssl#24 0x561392bf7fd8 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#25 0x561392bf8dcb in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#26 0x561392aff0d9 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#27 0x561392b08ed7 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#28 0x561392ae2fb5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#29 0x561392b2e5fe in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#30 0x561392b7e0b1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#31 0x561392b7e0b1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#32 0x561392b3de98 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 ``` - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_private) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. As a result, the fix is similar to earlier fix 5c4b034
----- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E openssl#9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E openssl#10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E openssl#11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E openssl#12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E openssl#16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E openssl#17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E openssl#18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E openssl#20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E openssl#21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E openssl#23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E openssl#24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E openssl#25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E openssl#26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E openssl#27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E openssl#29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E openssl#30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E openssl#10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E openssl#12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E openssl#15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E openssl#17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E openssl#18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E openssl#20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e)
During my testing (*), I got below stack about leaked memory. - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_public) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. (*) ClickHouse/clickhouse-private#10107 (comment) ``` ================================================================= ==2757181==ERROR: LeakSanitizer: detected memory leaks Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x5672f4a48083 in malloc (/data/ch4/build_asan/programs/clickhouse+0xa2b7083) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) #1 0x567321d05c7e in CRYPTO_malloc build_asan/./contrib/openssl/crypto/mem.c:202:11 #2 0x567321d05c7e in CRYPTO_zalloc build_asan/./contrib/openssl/crypto/mem.c:222:11 #3 0x567321d5f9cf in EVP_RAND_CTX_new build_asan/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x567321d627db in rand_new_drbg build_asan/./contrib/openssl/crypto/rand/rand_lib.c:658:11 #5 0x567321d61e7b in RAND_get0_public build_asan/./contrib/openssl/crypto/rand/rand_lib.c:777:16 #6 0x567321d61d4f in RAND_bytes_ex build_asan/./contrib/openssl/crypto/rand/rand_lib.c:378:12 #7 0x56731cfe9267 in ossl_random openssl.c #8 0x56731cf534ae in Curl_rand (/data/ch4/build_asan/programs/clickhouse+0x327c24ae) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#9 0x56731cf757dc in Curl_socketpair (/data/ch4/build_asan/programs/clickhouse+0x327e47dc) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#10 0x56731cf40bfe in Curl_multi_handle (/data/ch4/build_asan/programs/clickhouse+0x327afbfe) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#11 0x56731cecb260 in curl_easy_perform (/data/ch4/build_asan/programs/clickhouse+0x3273a260) (BuildId: 4d868624feaa8b5598d9d0b327749c877bfb6cfd) openssl#12 0x56731cd4a351 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#13 0x56731cd3176b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::H ttp::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#14 0x56731cd3176b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#15 0x56731cd2f891 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#16 0x56731cdd66db in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#17 0x56731cd7c75b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#18 0x56731cdd0218 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#19 0x56731cec5f18 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#20 0x56731cec6d0b in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#21 0x56731cdcd019 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#22 0x56731cdd6e17 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#23 0x56731cdb0ef5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#24 0x56731cdfc53e in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#25 0x56731ce4bff1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#26 0x56731ce4bff1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#27 0x56731ce0bdd8 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 openssl#28 0x56731ce396dc in Azure::Storage::Blobs::BlobServiceClient::CreateBlobContainer(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_asan/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_service_client.cpp:264:41 openssl#29 0x56731018fcff in DB::getAzureBlobContainerClient(Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_asan/./src/Disks/ObjectStorages/AzureBlobStorage/AzureBlobStorageAuth.cpp:236:75 openssl#30 0x5673115a7352 in DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:315:13 openssl#31 0x5673115a7352 in decltype(std::declval<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const> const&>(), std::declval<bool>() )) std::__1::__invoke[abi:v15000]<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#32 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IObjectStorage>, false>::__call<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_pt r<DB::Context const> const&, bool>(DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#33 0x5673115a7352 in std::__1::__function::__default_alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v 15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 openssl#34 0x5673115a7352 in std::__1::shared_ptr<DB::IObjectStorage> std::__1::__function::__policy_invoker<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::__call_impl<std::__1::__function::__default _alloc_func<DB::registerAzureObjectStorage(DB::ObjectStorageFactory&)::$_0, std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>>(std::__1::__function::__policy_storage const*, std::__1::basic_string<cha r, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 openssl#35 0x5673115a0780 in std::__1::__function::__policy_func<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::_ _1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool&&) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 openssl#36 0x5673115a0780 in std::__1::function<std::__1::shared_ptr<DB::IObjectStorage> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 openssl#37 0x5673115a0780 in DB::ObjectStorageFactory::create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const> const&, bool) const build_asan/./src/Disks/ObjectStorages/ObjectStorageFactory.cpp:135:12 openssl#38 0x56731159ef45 in DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1: :shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool) const build_asan/./src/Disks/ObjectStorages/RegisterDiskObjectStorage.cpp:27:64 openssl#39 0x56731159ef45 in decltype(std::declval<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&>()(std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<Poco::Util::AbstractConfiguration const&>(), std::declval<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(), std::declval<std::__1::shared_ptr<DB::Context const>>(), std::declval<std::__1::map<std: :__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&>(), std::declval<bool>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::regis terDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::_ _1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_ string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 openssl#40 0x56731159ef45 in std::__1::shared_ptr<DB::IDisk> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<DB::IDisk>, false>::__call<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool>(DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0 &, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<ch ar>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/invoke.h:470:16 openssl#41 0x56731159ef45 in std::__1::__function::__default_alloc_func<DB::registerDiskObjectStorage(DB::DiskFactory&, bool)::$_0, std::__1::shared_ptr<DB::IDisk> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>, std::__1::map<std::__1::basic_string<char, s td::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool, bool)>::operator()[abi:v15000](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator <char>> const&, Poco::Util::AbstractConfiguration const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context const>&&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::shared_ptr<DB::IDisk>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::b asic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::shared_ptr<DB::IDisk>>>> const&, bool&&, bool&&) build_asan/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 ```
The leak was reported in ClickHouse/ClickHouse#63792 - a SSL session is established - somewhere down in OpenSSL, the per-thread error queue is clear'ed - when no error queue exists (e.g. because the thread was just spawned), a new error queue is allocated - the error queue is stored in TLS, and a free handler is registered which deletes it when the thread ends - Leak sanitizer does not understand ^^ (false positive) ``` Direct leak of 9944 byte(s) in 11 object(s) allocated from: #0 0x5571bf7e04cf in malloc (/usr/bin/clickhouse+0xa39a4cf) (BuildId: 06c4931100f632dde9d0ecd3cf6a67776742e29b) #1 0x5571ecbc06be in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x5571ecbc06be in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x5571ecb03c6a in ossl_err_get_state_int build_docker/./contrib/openssl/crypto/err/err.c:691:17 #4 0x5571ecb037b5 in ERR_clear_error build_docker/./contrib/openssl/crypto/err/err.c:339:10 #5 0x5571ec9cf0d0 in state_machine build_docker/./contrib/openssl/ssl/statem/statem.c:366:5 #6 0x5571ec90c7d4 in SSL_do_handshake build_docker/./contrib/openssl/ssl/ssl_lib.c:4746:19 #7 0x5571e70cc501 in Poco::Net::SecureSocketImpl::connectSSL(bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:206:11 #8 0x5571e70cce6b in Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:149:2 openssl#9 0x5571e70d81f7 in Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&) build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:87:8 openssl#10 0x5571e0ca708a in DB::Connection::connect(DB::ConnectionTimeouts const&) build_docker/./src/Client/Connection.cpp:158:29 ```
ClickHouse issue ClickHouse/clickhouse-private#10107 (comment) reports this memory leak: (note how it is very similar to ClickHouse/clickhouse-private#10107 (comment)) ``` Direct leak of 528 byte(s) in 11 object(s) allocated from: #0 0x561369af24cf in malloc (/usr/bin/clickhouse+0xa6cf4cf) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) #1 0x561397b86a7e in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 #2 0x561397b86a7e in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 #3 0x561397be07cf in EVP_RAND_CTX_new build_docker/./contrib/openssl/crypto/evp/evp_rand.c:353:11 #4 0x561397be35fb in rand_new_drbg build_docker/./contrib/openssl/crypto/rand/rand_lib.c:665:11 #5 0x561397be2a9b in RAND_get0_private build_docker/./contrib/openssl/crypto/rand/rand_lib.c:827:16 #6 0x561397be296f in RAND_priv_bytes_ex build_docker/./contrib/openssl/crypto/rand/rand_lib.c:356:12 #7 0x5613978d7cd8 in SSL_CTX_new_ex build_docker/./contrib/openssl/ssl/ssl_lib.c:4016:13 #8 0x561392d1eabd in ossl_connect_common openssl.c openssl#9 0x561392d11e23 in ssl_cf_connect vtls.c openssl#10 0x561392c13a1f in cf_setup_connect connect.c openssl#11 0x561392c1b0e5 in cf_hc_connect cf-https-connect.c openssl#12 0x561392c0a332 in Curl_conn_connect (/usr/bin/clickhouse+0x337e7332) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#13 0x561392c7a28a in multi_runsingle multi.c openssl#14 0x561392c78f6d in curl_multi_perform (/usr/bin/clickhouse+0x33855f6d) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#15 0x561392bfd53e in curl_easy_perform (/usr/bin/clickhouse+0x337da53e) (BuildId: 22880fad595a96b17eb9add20e7a01f8ded54c49) openssl#16 0x561392a7c411 in Azure::Core::Http::CurlConnection::CurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2441:24 openssl#17 0x561392a6382b in std::__1::__unique_if<Azure::Core::Http::CurlConnection>::__unique_single std::__1::make_unique[abi:v15000]<Azure::Core::Http::CurlConnection, Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 openssl#18 0x561392a6382b in Azure::Core::Http::_detail::CurlConnectionPool::ExtractOrCreateCurlConnection(Azure::Core::Http::Request&, Azure::Core::Http::CurlTransportOptions const&, bool) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:2126:10 openssl#19 0x561392a61951 in Azure::Core::Http::CurlTransport::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/core/azure-core/src/http/curl/curl.cpp:351:48 openssl#20 0x561392b0879b in Azure::Core::Http::Policies::_internal::TransportPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/transport_policy.cpp:121:40 openssl#21 0x561392aae81b in Azure::Core::Http::Policies::_internal::LogPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/log_policy.cpp:114:23 openssl#22 0x561392b022d8 in Azure::Core::Http::Policies::_internal::RequestActivityPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/request_activity_policy.cpp:110:23 openssl#23 0x561392b2e02a in Azure::Storage::_internal::SharedKeyPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/shared_key_policy.hpp:36:25 openssl#24 0x561392bf7fd8 in Azure::Storage::_internal::StoragePerRetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_per_retry_policy.cpp:57:23 openssl#25 0x561392bf8dcb in Azure::Storage::_internal::StorageSwitchToSecondaryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/src/storage_switch_to_secondary_policy.cpp:36:32 openssl#26 0x561392aff0d9 in Azure::Core::Http::Policies::_internal::RetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/retry_policy.cpp:146:34 openssl#27 0x561392b08ed7 in Azure::Core::Http::Policies::_internal::TelemetryPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/src/http/telemetry_policy.cpp:23:21 openssl#28 0x561392ae2fb5 in Azure::Core::Http::Policies::_internal::RequestIdPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp:453:27 openssl#29 0x561392b2e5fe in Azure::Storage::_internal::StorageServiceVersionPolicy::Send(Azure::Core::Http::Request&, Azure::Core::Http::Policies::NextHttpPolicy, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-common/inc/azure/storage/common/internal/storage_service_version_policy.hpp:34:25 openssl#30 0x561392b7e0b1 in Azure::Core::Http::_internal::HttpPipeline::Send(Azure::Core::Http::Request&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/core/azure-core/inc/azure/core/internal/http/pipeline.hpp:230:29 openssl#31 0x561392b7e0b1 in Azure::Storage::Blobs::_detail::BlobContainerClient::Create(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Blobs::_detail::BlobContainerClient::CreateBlobContainerOptions const&, Azure::Core::Context const&) build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/rest_client.cpp:1415:36 openssl#32 0x561392b3de98 in Azure::Storage::Blobs::BlobContainerClient::Create(Azure::Storage::Blobs::CreateBlobContainerOptions const&, Azure::Core::Context const&) const build_docker/./contrib/azure/sdk/storage/azure-storage-blobs/src/blob_container_client.cpp:258:12 ``` - at database startup, Azure is registered as an object storage (--> registerAzureObjectStorage) - this calls into Azure, then into curl, and then into OpenSSL - curl asks OpenSSL for a bunch of random numbers (--> function 'ossl_random' in curl's OpenSSL wrapper) - OpenSSL initializes the random number generator and stores it in some random number generator context object (--> *RAND_get0_private) - this object is registered via pthread_key_create and pthread_setspecific registered in TLS - if registerAzureObjectStorage was the only place which initializes the RNG, we could argue that the leaked memory does not matter anyways as it is released after shutdown - RAND_get0_public also registers a free handler (rand_delete_thread_state) that runs in TLS, so the memory is released also if registration is called from arbitrary other threads. In sum: this is a false positive. As a result, the fix is similar to earlier fix 5c4b034
No description provided.