possible memory leak in soter_rsa_gen_key() [OpenSSL]

[outspace]

in https://github.com/cossacklabs/themis/blob/master/src/soter/openssl/soter_rsa_common.c, in

if (1 > EVP_PKEY_CTX_ctrl(pkey_ctx, -1, -1, EVP_PKEY_CTRL_RSA_KEYGEN_BITS, SOTER_RSA_KEY_LENGTH, NULL)) {
        return SOTER_FAIL;
    }

I think that BN_free(pub_exp) need to be called in this plase. Please check it out.

[Lagovas]

yes, agree. we will fix it

[Lagovas]

but read documentation and think we should not change anything. from documentation of openssl, EVP_PKEY_CTX_ctrl and EVP_PKEY_CTRL_RSA_KEYGEN_BITS the macros for EVP_PKEY_CTX_set_rsa_keygen_pubexp function call - https://docs.huihoo.com/doxygen/openssl/1.0.1c/crypto_2rsa_2rsa_8h_source.html#l00232
And docs tell next:

The EVP_PKEY_CTX_set_rsa_keygen_pubexp() macro sets the public exponent value for RSA key generation to pubexp. Currently it should be an odd integer. The pubexp pointer is used internally by this function so it should not be modified or freed after the call. If not specified 65537 is used.

So we should not free after call

[outspace]

Ok, then why you call it in https://github.com/cossacklabs/themis/blob/master/src/soter/openssl/soter_rsa_key_pair_gen.c,
https://github.com/cossacklabs/themis/blob/master/src/soter/boringssl/soter_rsa_common.c?

[Lagovas]

hm, you right. not so easy as I thought) thanks very match, I will re-check more carefully usage of these functions and unify across our backends

[ilammy]

I have perused OpenSSL's documentation and source code and it seems that the current code in OpenSSL backend is correct:

themis/src/soter/openssl/soter_rsa_common.c

Lines 62 to 64 in da8b52c

	if (1 > EVP_PKEY_CTX_ctrl(pkey_ctx, -1, -1, EVP_PKEY_CTRL_RSA_KEYGEN_BITS, SOTER_RSA_KEY_LENGTH, NULL)) {
	return SOTER_FAIL;
	}

We must not free pub_exp after a successful call to EVP_PKEY_CTX_set_rsa_keygen_pubexp() because it transfers ownership over the bignum to EVP_PKEY_CTX:

https://github.com/openssl/openssl/blob/70b0b977f73cd70e17538af3095d18e0cf59132e/crypto/rsa/rsa_pmeth.c#L472-L479

We should fix other places instead.

[ilammy]

Superseded by #517