Skip to content

Conversation

@LuciferYang
Copy link
Contributor

What changes were proposed in this pull request?

This pr aims to upgrade rocksdbjni from 7.6.0 to 7.7.3.

Why are the changes needed?

This version bring the performance of DeleteRange() behavior and fixes some bugs, the release notes as follows:

Does this PR introduce any user-facing change?

No

How was this patch tested?

  • Pass GitHub Actions
  • Manual test RocksDBBenchmark:

7.6.0

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.374   0.286   0.591   0.591
dbCreation                                      4       67.738  3.709   259.012 259.012
naturalIndexCreateIterator                      1024    0.006   0.002   1.595   0.007
naturalIndexDescendingCreateIterator            1024    0.007   0.006   0.068   0.008
naturalIndexDescendingIteration                 1024    0.006   0.004   0.060   0.008
naturalIndexIteration                           1024    0.006   0.004   0.119   0.009
randomDeleteIndexed                             1024    0.028   0.021   0.295   0.038
randomDeletesNoIndex                            1024    0.015   0.013   0.043   0.018
randomUpdatesIndexed                            1024    0.082   0.032   31.250  0.086
randomUpdatesNoIndex                            1024    0.033   0.030   0.661   0.036
randomWritesIndexed                             1024    0.119   0.034   52.561  0.118
randomWritesNoIndex                             1024    0.039   0.033   1.894   0.043
refIndexCreateIterator                          1024    0.005   0.005   0.020   0.007
refIndexDescendingCreateIterator                1024    0.003   0.003   0.027   0.005
refIndexDescendingIteration                     1024    0.007   0.005   0.046   0.008
refIndexIteration                               1024    0.007   0.005   0.321   0.010
sequentialDeleteIndexed                         1024    0.021   0.017   0.099   0.026
sequentialDeleteNoIndex                         1024    0.015   0.012   0.043   0.019
sequentialUpdatesIndexed                        1024    0.042   0.036   0.899   0.049
sequentialUpdatesNoIndex                        1024    0.039   0.031   0.945   0.047
sequentialWritesIndexed                         1024    0.048   0.041   2.022   0.056
sequentialWritesNoIndex                         1024    0.039   0.031   2.419   0.041

7.7.3

[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.365   0.272   0.531   0.531
dbCreation                                      4       66.947  3.915   258.255 258.255
naturalIndexCreateIterator                      1024    0.005   0.002   1.378   0.006
naturalIndexDescendingCreateIterator            1024    0.005   0.005   0.061   0.007
naturalIndexDescendingIteration                 1024    0.005   0.004   0.033   0.008
naturalIndexIteration                           1024    0.006   0.004   0.048   0.009
randomDeleteIndexed                             1024    0.026   0.018   0.271   0.034
randomDeletesNoIndex                            1024    0.015   0.012   0.044   0.018
randomUpdatesIndexed                            1024    0.079   0.032   29.256  0.083
randomUpdatesNoIndex                            1024    0.035   0.032   0.521   0.039
randomWritesIndexed                             1024    0.115   0.033   49.182  0.118
randomWritesNoIndex                             1024    0.040   0.034   1.876   0.045
refIndexCreateIterator                          1024    0.004   0.004   0.015   0.005
refIndexDescendingCreateIterator                1024    0.003   0.002   0.025   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.053   0.008
refIndexIteration                               1024    0.007   0.005   0.259   0.010
sequentialDeleteIndexed                         1024    0.021   0.017   0.118   0.027
sequentialDeleteNoIndex                         1024    0.015   0.012   0.043   0.018
sequentialUpdatesIndexed                        1024    0.044   0.037   0.902   0.053
sequentialUpdatesNoIndex                        1024    0.041   0.030   1.155   0.050
sequentialWritesIndexed                         1024    0.049   0.038   2.132   0.059
sequentialWritesNoIndex                         1024    0.039   0.030   2.527   0.042

@LuciferYang LuciferYang marked this pull request as draft October 25, 2022 03:53
@github-actions github-actions bot added the BUILD label Oct 25, 2022
@LuciferYang LuciferYang marked this pull request as ready for review October 25, 2022 04:54
Copy link
Contributor

@itholic itholic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merged to master.

@LuciferYang
Copy link
Contributor Author

Thanks @HyukjinKwon @itholic @dongjoon-hyun

SandishKumarHN pushed a commit to SandishKumarHN/spark that referenced this pull request Dec 12, 2022
### What changes were proposed in this pull request?
This pr aims to upgrade rocksdbjni from 7.6.0 to 7.7.3.

### Why are the changes needed?
This version bring the performance of `DeleteRange()` behavior and fixes some bugs, the release notes as follows:

- https://github.com/facebook/rocksdb/releases/tag/v7.7.2
- https://github.com/facebook/rocksdb/releases/tag/v7.7.3

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass GitHub Actions
- Manual test `RocksDBBenchmark`:

**7.6.0**

```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.374   0.286   0.591   0.591
dbCreation                                      4       67.738  3.709   259.012 259.012
naturalIndexCreateIterator                      1024    0.006   0.002   1.595   0.007
naturalIndexDescendingCreateIterator            1024    0.007   0.006   0.068   0.008
naturalIndexDescendingIteration                 1024    0.006   0.004   0.060   0.008
naturalIndexIteration                           1024    0.006   0.004   0.119   0.009
randomDeleteIndexed                             1024    0.028   0.021   0.295   0.038
randomDeletesNoIndex                            1024    0.015   0.013   0.043   0.018
randomUpdatesIndexed                            1024    0.082   0.032   31.250  0.086
randomUpdatesNoIndex                            1024    0.033   0.030   0.661   0.036
randomWritesIndexed                             1024    0.119   0.034   52.561  0.118
randomWritesNoIndex                             1024    0.039   0.033   1.894   0.043
refIndexCreateIterator                          1024    0.005   0.005   0.020   0.007
refIndexDescendingCreateIterator                1024    0.003   0.003   0.027   0.005
refIndexDescendingIteration                     1024    0.007   0.005   0.046   0.008
refIndexIteration                               1024    0.007   0.005   0.321   0.010
sequentialDeleteIndexed                         1024    0.021   0.017   0.099   0.026
sequentialDeleteNoIndex                         1024    0.015   0.012   0.043   0.019
sequentialUpdatesIndexed                        1024    0.042   0.036   0.899   0.049
sequentialUpdatesNoIndex                        1024    0.039   0.031   0.945   0.047
sequentialWritesIndexed                         1024    0.048   0.041   2.022   0.056
sequentialWritesNoIndex                         1024    0.039   0.031   2.419   0.041
```

**7.7.3**

```
[INFO] Running org.apache.spark.util.kvstore.RocksDBBenchmark
                                                count   mean    min     max     95th
dbClose                                         4       0.365   0.272   0.531   0.531
dbCreation                                      4       66.947  3.915   258.255 258.255
naturalIndexCreateIterator                      1024    0.005   0.002   1.378   0.006
naturalIndexDescendingCreateIterator            1024    0.005   0.005   0.061   0.007
naturalIndexDescendingIteration                 1024    0.005   0.004   0.033   0.008
naturalIndexIteration                           1024    0.006   0.004   0.048   0.009
randomDeleteIndexed                             1024    0.026   0.018   0.271   0.034
randomDeletesNoIndex                            1024    0.015   0.012   0.044   0.018
randomUpdatesIndexed                            1024    0.079   0.032   29.256  0.083
randomUpdatesNoIndex                            1024    0.035   0.032   0.521   0.039
randomWritesIndexed                             1024    0.115   0.033   49.182  0.118
randomWritesNoIndex                             1024    0.040   0.034   1.876   0.045
refIndexCreateIterator                          1024    0.004   0.004   0.015   0.005
refIndexDescendingCreateIterator                1024    0.003   0.002   0.025   0.004
refIndexDescendingIteration                     1024    0.006   0.005   0.053   0.008
refIndexIteration                               1024    0.007   0.005   0.259   0.010
sequentialDeleteIndexed                         1024    0.021   0.017   0.118   0.027
sequentialDeleteNoIndex                         1024    0.015   0.012   0.043   0.018
sequentialUpdatesIndexed                        1024    0.044   0.037   0.902   0.053
sequentialUpdatesNoIndex                        1024    0.041   0.030   1.155   0.050
sequentialWritesIndexed                         1024    0.049   0.038   2.132   0.059
sequentialWritesNoIndex                         1024    0.039   0.030   2.527   0.042
```

Closes apache#38382 from LuciferYang/SPARK-40905.

Authored-by: yangjie01 <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants