Summary:
There are some comments on subclasses in EncryptedEnv module which are
duplicate to their parent classes, it would be nice to remove the
duplication and keep the consistency if the comments on parent classes
updated in someday.

Pull Request resolved: facebook#11549

Reviewed By: akankshamahajan15

Differential Revision: D47007061

Pulled By: ajkr

fbshipit-source-id: 8bfdaf9f2418a24ca951c30bb88e90ac861d9016

apache/incubator-pegasus#1575

Cherry-pick from
tikv@113b363

Summary:
Introduce `KeyManagedEncryptedEnv` which wraps around `EncryptedEnv` but
provides an `KeyManager` API to enable key management per file. Also
implements `AESBlockCipher` with OpenSSL.

Test Plan:
not tested yet. will update.

Signed-off-by: Yi Wu <[email protected]>
Signed-off-by: tabokie <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@3d44a33

Summary:
Instead of using openssl's raw `AES_encrypt` and `AES_decrypt` API,
which is a low level call to encrypt or decrypt exact one block (16
bytes), we change to use the `EVP_*` API. The former is deprecated, and
will use the default C implementation without AES-NI support. Also the
EVP API is capable of handing CTR mode on its own.

Test Plan:
will add tests

Signed-off-by: Yi Wu <[email protected]>

---------

Signed-off-by: Yi Wu <[email protected]>
Co-authored-by: yiwu-arbug <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@2360562

Summary:
Fix NewRandomRWFile and ReuseWritableFile misuse of `GetFile()` and
`NewFile()`. See inline comments.

Test Plan:
manual test with tikv

Signed-off-by: Yi Wu <[email protected]>

Co-authored-by: yiwu-arbug <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@93e89a5

fix bug: tikv/tikv#9115

Summary: we need to update encryption metadata via
encryption::DataKeyManager, which cannot combine with the actual file
operation into one atomic operation. In RenameFile, when the src_file
has been removed, power is off, then we may lost the file info of
src_file next restart.

Signed-off-by: Xintao [[email protected]](mailto:[email protected])

Signed-off-by: Xintao <[email protected]>
Co-authored-by: Xintao <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@bbd27cf

used LinkFile instead of RenameFile api of key manager. But LinkFile
needs check the dst file information, in RenameFile logic, we don't care
about that. So just skip encryption for current file.

Signed-off-by: Xintao [[email protected]](mailto:[email protected])

…cebook#9)

apache/incubator-pegasus#1575

Cherry-pick from
tikv@1868d12

Signed-off-by: Xintao <[email protected]>
Signed-off-by: tabokie <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@4cebfc1

* Add SM4-CTR encryption algorithm
* Adjust block size for sm4 encryption
* Add UT for SM4 encryption
* Adjust macros indentation for sm4
* Fix format for adding sm4

Signed-off-by: Jarvis Zheng <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@9464766

In some env, user installed openssl by yum install, and the openssl
software may compiled with OPENSSL_NO_SM4 flag, so although the version
is >= 1.1.1, but we still could not use sm4 in that situation.

Signed-off-by: Jarvis Zheng <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@acc624f

* hook delete dir in encrypted env
* add a comment

Signed-off-by: tabokie <[email protected]>
Co-authored-by: Xinye Tao <[email protected]>

apache/incubator-pegasus#1575

Cherry-pick from
tikv@14f36f8
(without compaction related code)

* fix renaming encrypted directory

Signed-off-by: tabokie <[email protected]>