HDDS-8028. JNI for RocksDB SST Dump tool

[swamirishi]

What changes were proposed in this pull request?

JNI for RocksDB SST Dump tool for reading tombstone entries from SST file for smart computation of SnapDiff.
rocks-native-build can be triggered by passing -Drocks_tools_native while maven build.

mvn clean install -DskipTests -Drocks_tools_native -pl :hdds-rocks-native

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8028

How was this patch tested?

Manual Testing.

[kerneltime]

Open questions

• How can we avoid building the JNI as part of each build? Can we separate out the JNI build as an independent artifact that gets pulled in as a JNI dependency?
• Can we build the JNI and update the docker images used via CI/CD to avoid the hit per build

[neils-dev]

Thanks @swamirishi . If would be a great help if you can let me know if there are any special build parameters you have in mind for building this like the native library compile for hadoop?
ie. $ mvn package -Pdist,native -DskipTests -Dtar

Also, what would be a good way that I can manually test the jni build?

[swamirishi]

n let me know if there are any special build parameters you have in mind for building this like the native library compile for hadoop?
There are no special build parameters. Currently the module is built by default for all runs.

[prashantpogde]

Open questions

• How can we avoid building the JNI as part of each build? Can we separate out the JNI build as an independent artifact that gets pulled in as a JNI dependency?
• Can we build the JNI and update the docker images used via CI/CD to avoid the hit per build

We are planning to have a seprate profile to enable efficient snapdiff build. The default build will just include regular snapdiff path.

[prashantpogde]

Changes look good to me once these pending comments are taken care of. We should write some integration test but I guess we can make it end to end driven by snapdiff.

[smengcl]

lgtm apart from a few minor comments inline.

[prashantpogde]

LGTM

[smengcl]

Verified that build time is unchanged with normal dev workflow (does not compile RocksDB Tools without -Drocks_tools_native).

Current master branch b1695e3 done in 1m58s with skip tests/javadoc/shade:

$ mvn clean install -DskipTests -e -Dmaven.javadoc.skip=true -DskipShade
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:58 min
[INFO] Finished at: 2023-03-10T12:30:59-08:00

Compilation passed locally in 1m59s on db3300d with the same maven args:

$ mvn clean install -DskipTests -e -Dmaven.javadoc.skip=true -DskipShade
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:59 min
[INFO] Finished at: 2023-03-10T12:25:36-08:00

With -Drocks_tools_native, compilation passed locally in 3m26s on db3300d with skip tests/javadoc/shade:

$ mvn clean install -DskipTests -e -Dmaven.javadoc.skip=true -DskipShade -Drocks_tools_native
...
[INFO] Apache Ozone HDDS RocksDB Tools .................... SUCCESS [01:27 min]
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:26 min
[INFO] Finished at: 2023-03-10T14:05:34-08:00

comments addressed

[smengcl]

Thanks @swamirishi for the patch. It is a huge effort to put this all together.

Thanks @adoroszlai @prashantpogde for the code review.

if there are any special build parameters you have in mind for building this like the native library compile for hadoop? ie. $ mvn package -Pdist,native -DskipTests -Dtar

Hi @neils-dev , the extra maven parameter has been included in the PR description now, namely -Drocks_tools_native.

@kerneltime w.r.t. to the build time concern, as tested above, by default it will not impact the build time. Adding -Drocks_tools_native to trigger the RocksDB tool compilation adds another ~1.5 min to the total build time (M1 Pro MBP), which is not required for most other dev work.