Add support for cloud data store #517

de-sh · 2020-06-23T12:15:00Z

Summary

Integration with rockset/rocksdb-cloud to add CloudEnv features for operating with cloud services like AWS S3 as a datastore.

Created a sub-crate librocksdb_cloud_sys which contains the rocksdb-cloud sub-module containing code for managing cloud interactions, extracted from rockset/rocksdb-cloud and currently residing in tikv/rocksdb~6.4.cloud
Made necessary changes to tikv/rocksdb~6.4.tikv that is housed in the sub-module rocksdb within the parent crate.
Wrote CMakeLists.txt for use in the build process of rocksdb_cloud as a static library, usable for C-ABI based FFI calls. Added the above code to the build process of parent crate librocksdb_sys along with C-ABI binding code within crocksdb
Created 'cloud.rs' and added new_aws_env() method to Env type of rocksdb.rs that merely makes a call to the CloudEnv::NewAwsEnv() via FFI for use as library features upstream. A test for the same was also added here.
Used feature based conditional compile macros to all cloud interaction code.
Made requisite changes to CI config for handling AWS SDK compilation.
Added a sequence of steps to update submodule within Makefile

Solves #514

Checkpoints

Added rocksdb-cloud as submodule in crate librocksdb_cloud_sys.
Write CMakeLists.txt to build only cloud code.
Generate C-bindings to interface with NewAwsEnv and CloudEnvOptions
Create Env::new_aws_env as an interface in rust

.gitmodules

Cargo.toml

librocksdb_cloud_sys/Cargo.toml

librocksdb_cloud_sys/CMakeLists.txt

.travis.yml

.gitmodules

librocksdb_sys/Cargo.toml

Cargo.toml

librocksdb_sys/librocksdb_cloud_sys/Cargo.toml

librocksdb_sys/librocksdb_cloud_sys/CMakeLists.txt

src/rocksdb.rs

src/cloud.rs

Related to tikv/rust-rocksdb#517 and #182 ### Summary Adds code used in compiling with [rocksdb~6.4.tikv](https://github.com/tikv/rocksdb/tree/6.4.tikv) along with [rocksdb~6.4.cloud](https://github.com/tikv/rocksdb/tree/6.4.cloud) ### Changes - Makes public the struct LogReporter so that it can be accessed by [rocksdb~6.4.cloud/cloud/manifest_reader.cc#L114](https://github.com/tikv/rocksdb/blob/6a7c514453e3e3e1c3bb57c18d8e0280089ccdcd/cloud/manifest_reader.cc#L114) Signed-off-by: Devdutt Shenoi <[email protected]>

librocksdb_sys/crocksdb/c.cc

yiwu-arbug · 2020-08-01T02:46:38Z

librocksdb_sys/crocksdb/c.cc

+  std::unique_ptr<CloudEnv> cloud_env;
+
+  CloudEnv* cenv;
+  if (SaveError(errptr, CloudEnv::NewAwsEnv(base_env->rep, cloud_options->rep,


the list of params are much different to in the cloud code https://github.com/tikv/rocksdb/blob/6.4.cloud/include/rocksdb/cloud/cloud_env_options.h#L291 does the code really being built?

yiwu-arbug

For all the new files, add the following header

 // Copyright 2020 TiKV Project Authors. Licensed under Apache-2.0.

Signed-off-by: Devdutt Shenoi <[email protected]>

librocksdb_sys/crocksdb/c.cc

yiwu-arbug

Can you update Makefile to include a update_rocksdb_cloud target, similar to update_rocksdb and update_titan?

Used in PR tikv/rust-rocksdb#517 and #181 Adds code used in compiling with [rocksdb~6.4.tikv](https://github.com/tikv/rocksdb/tree/6.4.tikv) along with [rocksdb~6.4.cloud](https://github.com/tikv/rocksdb/tree/6.4.cloud) Signed-off-by: Devdutt Shenoi <[email protected]>

Signed-off-by: Devdutt Shenoi <[email protected]>

yiwu-arbug

Looks like the travis cache works now.

yiwu-arbug

LGTM

Related to tikv/rust-rocksdb#517 and tikv#182 Adds code used in compiling with [rocksdb~6.4.tikv](https://github.com/tikv/rocksdb/tree/6.4.tikv) along with [rocksdb~6.4.cloud](https://github.com/tikv/rocksdb/tree/6.4.cloud) - Makes public the struct LogReporter so that it can be accessed by [rocksdb~6.4.cloud/cloud/manifest_reader.cc#L114](https://github.com/tikv/rocksdb/blob/6a7c514453e3e3e1c3bb57c18d8e0280089ccdcd/cloud/manifest_reader.cc#L114) Signed-off-by: Devdutt Shenoi <[email protected]> Signed-off-by: tabokie <[email protected]>

Related to tikv/rust-rocksdb#517 and #182 Adds code used in compiling with [rocksdb~6.4.tikv](https://github.com/tikv/rocksdb/tree/6.4.tikv) along with [rocksdb~6.4.cloud](https://github.com/tikv/rocksdb/tree/6.4.cloud) - Makes public the struct LogReporter so that it can be accessed by [rocksdb~6.4.cloud/cloud/manifest_reader.cc#L114](https://github.com/tikv/rocksdb/blob/6a7c514453e3e3e1c3bb57c18d8e0280089ccdcd/cloud/manifest_reader.cc#L114) Signed-off-by: Devdutt Shenoi <[email protected]> Signed-off-by: tabokie <[email protected]>

yiwu-arbug reviewed Jun 23, 2020

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

de-sh changed the title ~~submodules: add rockset/rocksdb-cloud~~ Add support for cloud data store Jun 27, 2020

yiwu-arbug reviewed Jun 27, 2020

View reviewed changes

.gitmodules Outdated Show resolved Hide resolved

yiwu-arbug reviewed Jul 8, 2020

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

yiwu-arbug added the status/WIP Status: Working in progress label Jul 9, 2020

yiwu-arbug reviewed Jul 9, 2020

View reviewed changes

librocksdb_cloud_sys/Cargo.toml Outdated Show resolved Hide resolved

yiwu-arbug reviewed Jul 9, 2020

View reviewed changes

librocksdb_cloud_sys/Cargo.toml Outdated Show resolved Hide resolved

de-sh changed the base branch from master to tikv-3.x July 10, 2020 20:33

yiwu-arbug reviewed Jul 13, 2020

View reviewed changes

librocksdb_cloud_sys/CMakeLists.txt Outdated Show resolved Hide resolved

yiwu-arbug changed the base branch from tikv-3.x to master July 15, 2020 18:14

yiwu-arbug changed the base branch from master to tikv-3.x July 15, 2020 18:14

yiwu-arbug reviewed Jul 19, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

de-sh mentioned this pull request Jul 21, 2020

Code changes to plugin rocksdb_cloud tikv/rocksdb#181

Merged

de-sh changed the base branch from tikv-3.x to master July 21, 2020 20:52

yiwu-arbug reviewed Jul 24, 2020

View reviewed changes

de-sh mentioned this pull request Jul 25, 2020

Compile cloud features with 6.4.tikv branch tikv/rocksdb#182

Merged

yiwu-arbug mentioned this pull request Jul 30, 2020

Demo TiKV performance on S3 tikv/tikv#8367

Open

4 tasks

yiwu-arbug added sig/engine SIG: Engine and removed status/WIP Status: Working in progress labels Aug 1, 2020

yiwu-arbug reviewed Aug 1, 2020

View reviewed changes

librocksdb_sys/crocksdb/c.cc Outdated Show resolved Hide resolved

yiwu-arbug reviewed Aug 1, 2020

View reviewed changes

librocksdb_sys/crocksdb/c.cc Show resolved Hide resolved

yiwu-arbug reviewed Aug 1, 2020

View reviewed changes

de-sh added 6 commits August 1, 2020 21:51

rocksdb-cloud: add submodule

fa71c46

Signed-off-by: Devdutt Shenoi <[email protected]>

initiate crate librocksdb_cloud_sys

fdb9942

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: add code to build

1326e49

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: move out CMakeLists.txt

05af468

Signed-off-by: Devdutt Shenoi <[email protected]>

CMake: update build process

b3f8422

Signed-off-by: Devdutt Shenoi <[email protected]>

Complete sub-crate addition librocksdb_sys/librocksdb_cloud_sys

06edfc4

Signed-off-by: Devdutt Shenoi <[email protected]>

de-sh added 10 commits August 1, 2020 21:51

submodules update

ed42f17

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: rename link library to rocksdb_cloud

0c4766b

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: use fork tikv/rocksdb/6.4.cloud

06a5e35

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: restructure binding code, add test build with add aws sdk

0919d3d

Signed-off-by: Devdutt Shenoi <[email protected]>

CI: Conditionally build AWS-CPP-SDK for cloud test only

0f33e1c

Signed-off-by: Devdutt Shenoi <[email protected]>

CI: remove comments and outer os def

8622093

Signed-off-by: Devdutt Shenoi <[email protected]>

*: add License note

8de085e

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: refactor signature used with NewAwsEnv

a742516

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb: refactor test according to example, clang-fmt

5e9a51b

Signed-off-by: Devdutt Shenoi <[email protected]>

rocksdb-cloud: update submod to v5.18.3+edits

b3bda19

Signed-off-by: Devdutt Shenoi <[email protected]>

yiwu-arbug reviewed Aug 1, 2020

View reviewed changes

librocksdb_sys/crocksdb/c.cc Outdated Show resolved Hide resolved

yiwu-arbug reviewed Aug 1, 2020

View reviewed changes

de-sh added 2 commits August 2, 2020 11:14

crocksdb: use static_cast and solve test warn

65d8e95

Signed-off-by: Devdutt Shenoi <[email protected]>

make: add update_rocksdb_cloud sequence to Makefile

7266339

Signed-off-by: Devdutt Shenoi <[email protected]>

de-sh mentioned this pull request Aug 2, 2020

storage-engine: Add S3 as a storage location with rocksdb-cloud tikv/tikv#8383

Closed

4 tasks

de-sh and others added 4 commits August 3, 2020 13:06

rocksdb/tikv: update submodule

93f7d77

Signed-off-by: Devdutt Shenoi <[email protected]>

cmake: downgrade version of rocksdb used

87a4d2e

Signed-off-by: Devdutt Shenoi <[email protected]>

CI: cache /tmp/aws-sdk-cpp-x.x.x

5211b88

Signed-off-by: Devdutt Shenoi <[email protected]>

Merge branch 'master' into master

38c489c

yiwu-arbug reviewed Aug 5, 2020

View reviewed changes

yiwu-arbug approved these changes Aug 5, 2020

View reviewed changes

ti-srebot added the status/LGT1 label Aug 5, 2020

yiwu-arbug merged commit 6ad6e05 into tikv:master Aug 5, 2020

yiwu-arbug mentioned this pull request Aug 5, 2020

Support rocksdb-cloud #514

Closed

3 tasks

leoncamel added a commit to leoncamel/tikv.rust-rocksdb that referenced this pull request Jun 11, 2021

Cherry-pick tikv#517

d09b0e9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for cloud data store #517

Add support for cloud data store #517

de-sh commented Jun 23, 2020 •

edited

Loading

yiwu-arbug Aug 1, 2020

yiwu-arbug left a comment

yiwu-arbug left a comment

yiwu-arbug left a comment

yiwu-arbug left a comment

Add support for cloud data store #517

Add support for cloud data store #517

Conversation

de-sh commented Jun 23, 2020 • edited Loading

Summary

Checkpoints

yiwu-arbug Aug 1, 2020

Choose a reason for hiding this comment

yiwu-arbug left a comment

Choose a reason for hiding this comment

yiwu-arbug left a comment

Choose a reason for hiding this comment

yiwu-arbug left a comment

Choose a reason for hiding this comment

yiwu-arbug left a comment

Choose a reason for hiding this comment

de-sh commented Jun 23, 2020 •

edited

Loading