Modify ldb Tool to be Encryption Aware #3455
Assignees
Labels
Comments
rahuldesirazu
added
kind/improve-ux
rahuldesirazu
added a commit
that referenced
this issue
Feb 26, 2020
Summary: Add `--key_file` option to ldb to pass in a `<key_id>:</path/to/key/file>` and an option `--only_verify_checksums` flag to `scan` to not print out entries and only do checksum validation. Also provide a yb-admin `write_universe_key_to_file` command to retrieve the universe key and write it to a local file. Test Plan: Jenkins: build platform: mac Jenkins: compile only Reviewers: bogdan, mikhail, timur Reviewed By: timur Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7876
rahuldesirazu
added a commit
that referenced
this issue
Mar 25, 2020
Summary: Add `--key_file` option to ldb to pass in a `<key_id>:</path/to/key/file>` and an option `--only_verify_checksums` flag to `scan` to not print out entries and only do checksum validation. Also provide a yb-admin `write_universe_key_to_file` command to retrieve the universe key and write it to a local file. Test Plan: Jenkins: build platform: mac Jenkins: compile only Reviewers: bogdan, mikhail, timur Reviewed By: timur Subscribers: ybase Differential Revision: https://phabricator.dev.yugabyte.com/D7876
rahuldesirazu
added a commit
that referenced
this issue
Mar 25, 2020
Summary: Support the following yb-admin commands for encryption at rest: ``` add_universe_key_to_all_masters(String keyid, String key_path) all_masters_have_universe_key_in_memory(String keyId) rotate_universe_key_in_memory(String keyId) disable_encryption_in_memory() ``` [#3455] Modify ldb Tool to be Encryption Aware Add `--key_file` option to ldb to pass in a `<key_id>:</path/to/key/file>` and an option `--only_verify_checksums` flag to `scan` to not print out entries and only do checksum validation. Also provide a yb-admin `write_universe_key_to_file` command to retrieve the universe key and write it to a local file. [#3454] Make OpenSSL usage thread-safe https://www.openssl.org/docs/man1.0.2/man3/CRYPTO_set_locking_callback.html Set a locking and thread id callback to enable thread-safe usage for openssl. Move the openssl init process to a once per-process initialization in RpcServerBase. [#3707] A workaround for encryption counter overflow This is a workaround for the encrypted file corruption issue described at #3707. When reading a block from an encrypted SSTable and getting a checksum mismatch, we now try to increment byte 11 of the initialization vector (carrying over into earlier bytes when encountering 0xff) and decrypt and verify checksum once again. Similarly, when reading an SSTable footer, we use magic number comparison instead of checksum verification. This workaround is turned on by default and is controlled using the new `--encryption_counter_overflow_read_path_workaround` flag. This diff is not fixing the underlying ignored overflow issue yet. While it would be easy to do so, the old read path would not be able to read files written with the new write path. Instead, we will deploy this workaround, disable encryption, perform major compactions on all data, and then deploy the real fix and re-enable encryption. Also we restrict the range of the unsigned 32-bit randomly-generated initial counter value to [0, 0x7fffffff] by default. This reduces the effective key size by 1 bit but eliminates the overflow issue for all files up to 32 GiB in size. This range could be customized using the new flags: `--encryption_counter_min` and `--encryption_counter_max` (both of these bounds are inclusive). Also avoid storing a reference to a shared pointer in TableReaderOptions. This fixes an ASAN issue in the new test. #3974 Enable checksum verification for meta blocks of encrypted files Enable checksum verification for meta blocks of encrypted files. When I tried to enable checksum verification for meta blocks unconditionally, a lot of RocksDB unit tests fail, so I am leaving checksum verification for meta blocks of unencrypted files for a future diff. Also enable an encrypted SSTable test that is fixed by this change and fails without this change. [#3976] Fix encryption format for newly created files For newly created encrypted files, overflow the counter into the rest of the initialization vector to match OpenSSL behavior. To do this add a proto field to encrypted files to indicate whether it is of the old or new format. For old block-based files, do the checksum retry read path (first try non-overflow, then overflow key). For new block-based files, use the old read path that only tries to decrypt using overflowed keys. Test Plan: manual testing with yb-admin Jenkins: build platform: mac Jenkins: compile only ybd --cxx-test ctr_cipher_stream-test --gtest_filter TestCipherStream.ConcurrentEncryption Jenkins New test, encrypted_sstable-test, that fails without the fix and passes with the fix. Jenkins Unit tests at the stream level with counter set right before overflow. Integration cluster tests that test series of flushes and compactions. Subscribers: kannan, bogdan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D8183
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Modify rocksdb ldb to take in an optional key parameter for encrypted databases.
The text was updated successfully, but these errors were encountered: