Skip to content

HDDS-10149. New JNI layer for RawSSTFileReader & RawSSTFileReaderIterator #6182

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swamirishi merged 40 commits into from
Feb 22, 2024
Merged

HDDS-10149. New JNI layer for RawSSTFileReader & RawSSTFileReaderIterator #6182

Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
4d60dc7
HDDS-10149. Fix native lib to not include rocksdb library as static l…
swamirishi Feb 6, 2024
2dc7c0b
HDDS-10149. Fix checkstyle
swamirishi Feb 7, 2024
824678d
HDDS-10149. Test
swamirishi Feb 7, 2024
f2a7869
HDDS-10149. New JNI layer containing new rawSstFileReader on rocksdb …
swamirishi Feb 14, 2024
f25a570
HDDS-10149. Remove usage of managed sst dump tool
swamirishi Feb 14, 2024
a11705a
HDDS-10149. Fix findbugs & rat check
swamirishi Feb 14, 2024
c80234d
HDDS-10149. Fix findbugs
swamirishi Feb 14, 2024
406db5b
HDDS-10149. Fix checkstyle
swamirishi Feb 14, 2024
41d557d
HDDS-10149. Fix cpp shadow error
swamirishi Feb 14, 2024
f9dcb2e
HDDS-10149. Fix native
swamirishi Feb 14, 2024
194c8af
HDDS-10149. Fix native
swamirishi Feb 14, 2024
09e640c
HDDS-10149. Fix native
swamirishi Feb 14, 2024
bbb15ad
HDDS-10149. Debug native failure
swamirishi Feb 14, 2024
dd78c1e
HDDS-10149. Debug native failure
swamirishi Feb 14, 2024
2d2c8f6
HDDS-10149. Fix test cases
swamirishi Feb 14, 2024
be7faa3
HDDS-10149. Remove debug flag
swamirishi Feb 14, 2024
5a64230
HDDS-10149. Adding log
swamirishi Feb 14, 2024
f4aa7ec
HDDS-10149. Fix bug
swamirishi Feb 14, 2024
0e1f50d
HDDS-10149. Add log
swamirishi Feb 14, 2024
209c258
HDDS-10149. Add ssh session
swamirishi Feb 15, 2024
2e08516
Revert "HDDS-10149. Fix bug"
swamirishi Feb 15, 2024
1a8ac2b
Revert "HDDS-10149. Add ssh session"
swamirishi Feb 15, 2024
b3d101e
Revert "HDDS-10149. Add log"
swamirishi Feb 15, 2024
5cd071c
HDDS-10149. Fix native library compile issue for gcc>=5
swamirishi Feb 15, 2024
ef4b147
HDDS-10149. Mark integration test healthy
swamirishi Feb 15, 2024
f324d32
HDDS-10149. Fix checkstyle
swamirishi Feb 15, 2024
76a5df6
HDDS-10149. Fix SnapshotDiffManager
swamirishi Feb 15, 2024
b5d4646
HDDS-10149. Fix TestSnapshotDiffManager::testGetSnapshotDiffReportJob
swamirishi Feb 15, 2024
66921e9
HDDS-10149. Unmark unhealthy snapshots native tests
swamirishi Feb 15, 2024
8cc3df4
Merge remote-tracking branch 'apache/master' into HEAD
swamirishi Feb 15, 2024
c0752e1
HDDS-10149. Fix checkstyle
swamirishi Feb 15, 2024
da80339
HDDS-10149. Fix testcase
swamirishi Feb 15, 2024
03f22f9
HDDS-10149. Address review comments
swamirishi Feb 22, 2024
2d70537
HDDS-10149. Fix checkstyle within patch file
swamirishi Feb 22, 2024
4c3890b
HDDS-10149. Remove unwanted classes and tools from rocks tools
swamirishi Feb 22, 2024
8a1b0bb
HDDS-10149. Remove unwanted classes and tools from rocks tools
swamirishi Feb 22, 2024
1ab1bc6
HDDS-10149. Change Function definitions & imports for raw_sst_file_it…
swamirishi Feb 22, 2024
a36487b
HDDS-10149. Address review comments
swamirishi Feb 22, 2024
9b48779
HDDS-10149. Address review comments
swamirishi Feb 22, 2024
c4ee8c8
HDDS-10149. Address review comments
swamirishi Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 0 additions & 18 deletions hadoop-hdds/common/src/main/resources/ozone-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4273,15 +4273,6 @@
</description>
</property>

<property>
<name>ozone.om.snapshot.sst_dumptool.pool.size</name>
<value>1</value>
<tag>OZONE, OM</tag>
<description>
Threadpool size for SST Dumptool which would be used for computing snapdiff when native library is enabled.
</description>
</property>

<property>
<name>ozone.om.snapshot.load.native.lib</name>
<value>true</value>
Expand All @@ -4291,15 +4282,6 @@
</description>
</property>

<property>
<name>ozone.om.snapshot.sst_dumptool.buffer.size</name>
<value>8KB</value>
<tag>OZONE, OM</tag>
<description>
Buffer size for SST Dumptool Pipe which would be used for computing snapdiff when native library is enabled.
</description>
</property>

<property>
<name>ozone.om.snapshot.diff.max.allowed.keys.changed.per.job</name>
<value>10000000</value>
Expand Down
48 changes: 48 additions & 0 deletions ...cksdb/src/main/java/org/apache/hadoop/hdds/utils/db/managed/JniLibNamePropertyWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* <p>
* http://www.apache.org/licenses/LICENSE-2.0
* <p>
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hdds.utils.db.managed;

import org.rocksdb.util.Environment;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.Writer;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

/**
* Class to write the rocksdb lib name to a file.
* This would be used to build native ozone_rocksdb_tools library.
*/
public final class JniLibNamePropertyWriter {

private JniLibNamePropertyWriter() {
}

public static void main(String[] args) {
String filePath = args[0];
try (Writer writer = new OutputStreamWriter(
Files.newOutputStream(Paths.get(filePath)), StandardCharsets.UTF_8)) {
writer.write("rocksdbLibName=" +
Environment.getJniLibraryFileName("rocksdb"));
} catch (IOException e) {
e.printStackTrace();
}
}
}
203 changes: 58 additions & 145 deletions hadoop-hdds/rocks-native/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,6 @@

<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<bzip2.url>https://sourceware.org/pub/bzip2/bzip2-${bzip2.version}.tar.gz</bzip2.url>
<zlib.url>https://zlib.net/fossils/zlib-${zlib.version}.tar.gz</zlib.url>
</properties>

<build>
Expand Down Expand Up @@ -113,79 +111,80 @@
<build>
<plugins>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>rocksdb source download</id>
<phase>generate-sources</phase>
<goals>
<goal>wget</goal>
</goals>
<configuration>
<url>https://github.com/facebook/rocksdb/archive/refs/tags/v${rocksdb.version}.tar.gz</url>
<outputFileName>rocksdb-v${rocksdb.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/rocksdb</outputDirectory>
</configuration>
</execution>
<execution>
<id>zlib source download</id>
<phase>generate-sources</phase>
<goals>
<goal>wget</goal>
</goals>
<configuration>
<url>${zlib.url}</url>
<outputFileName>zlib-${zlib.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/zlib</outputDirectory>
</configuration>
</execution>
<execution>
<id>bzip2 source download</id>
<phase>generate-sources</phase>
<id>set-property</id>
<phase>initialize</phase>
<goals>
<goal>wget</goal>
<goal>java</goal>
</goals>
<configuration>
<url>${bzip2.url}</url>
<outputFileName>bzip2-v${bzip2.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/bzip2</outputDirectory>
<mainClass>org.apache.hadoop.hdds.utils.db.managed.JniLibNamePropertyWriter</mainClass>
<arguments>
<argument>${project.build.directory}/propertyFile.txt</argument>
</arguments>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>properties-maven-plugin</artifactId>
<executions>
<execution>
<id>lz4 source download</id>
<phase>generate-sources</phase>
<id>read-property-from-file</id>
<phase>initialize</phase> <!-- Choose the appropriate phase -->
Comment thread
hemantk-12 marked this conversation as resolved.
Outdated
<goals>
<goal>wget</goal>
<goal>read-project-properties</goal>
</goals>
<configuration>
<url>https://github.com/lz4/lz4/archive/refs/tags/v${lz4.version}.tar.gz</url>
<outputFileName>lz4-v${lz4.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/lz4</outputDirectory>
<files>
<file>${project.build.directory}/propertyFile.txt</file>
</files>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>snappy source download</id>
<phase>generate-sources</phase>
<id>unpack-dependency</id>
<phase>initialize</phase> <!-- Choose the appropriate phase -->
<goals>
<goal>wget</goal>
<goal>unpack</goal>
</goals>
<configuration>
<url>https://github.com/google/snappy/archive/refs/tags/${snappy.version}.tar.gz</url>
<outputFileName>snappy-v${snappy.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/snappy</outputDirectory>
<artifactItems>
<artifactItem>
<groupId>org.rocksdb</groupId>
<artifactId>rocksdbjni</artifactId>
<type>jar</type>
<overWrite>false</overWrite>
<outputDirectory>${project.build.directory}/rocksdbjni</outputDirectory>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<executions>
<execution>
<id>zstd source download</id>
<id>rocksdb source download</id>
<phase>generate-sources</phase>
<goals>
<goal>wget</goal>
</goals>
<configuration>
<url>https://github.com/facebook/zstd/archive/refs/tags/v${zstd.version}.tar.gz</url>
<outputFileName>zstd-v${zstd.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/zstd</outputDirectory>
<url>https://github.com/facebook/rocksdb/archive/refs/tags/v${rocksdb.version}.tar.gz</url>
<outputFileName>rocksdb-v${rocksdb.version}.tar.gz</outputFileName>
<outputDirectory>${project.build.directory}/rocksdb</outputDirectory>
</configuration>
</execution>
</executions>
Expand Down Expand Up @@ -219,89 +218,6 @@
<configuration>
<target>
<untar src="${project.build.directory}/rocksdb/rocksdb-v${rocksdb.version}.tar.gz" compression="gzip" dest="${project.build.directory}/rocksdb/" />
<untar src="${project.build.directory}/zlib/zlib-${zlib.version}.tar.gz" compression="gzip" dest="${project.build.directory}/zlib/" />
<untar src="${project.build.directory}/bzip2/bzip2-v${bzip2.version}.tar.gz" compression="gzip" dest="${project.build.directory}/bzip2/" />
<move file="${project.build.directory}/bzip2/bzip2-bzip2-${bzip2.version}" tofile="${project.build.directory}/bzip2/bzip2-${bzip2.version}" failonerror="false" />
<untar src="${project.build.directory}/lz4/lz4-v${lz4.version}.tar.gz" compression="gzip" dest="${project.build.directory}/lz4/" />
<untar src="${project.build.directory}/snappy/snappy-v${snappy.version}.tar.gz" compression="gzip" dest="${project.build.directory}/snappy/" />
<untar src="${project.build.directory}/zstd/zstd-v${zstd.version}.tar.gz" compression="gzip" dest="${project.build.directory}/zstd/" />
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>build-zlib</id>
<phase>process-sources</phase>
<configuration>
<target>
<chmod file="${project.build.directory}/zlib/zlib-${zlib.version}/configure" perm="775" />
<exec executable="./configure" dir="${project.build.directory}/zlib/zlib-${zlib.version}" failonerror="true">
<arg line="--static"/>
<env key="CFLAGS" value="-fPIC"/>
</exec>
<exec executable="make" dir="${project.build.directory}/zlib/zlib-${zlib.version}" failonerror="true"/>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>build-bzip2</id>
<phase>process-sources</phase>
<configuration>
<target>
<exec executable="make" dir="${project.build.directory}/bzip2/bzip2-${bzip2.version}" failonerror="true">
<arg line="CFLAGS='-fPIC'"/>
</exec>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>build-lz4</id>
<phase>process-sources</phase>
<configuration>
<target>
<exec executable="make" dir="${project.build.directory}/lz4/lz4-${lz4.version}" failonerror="true">
<arg line="CFLAGS='-fPIC'"/>
</exec>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>build-zstd</id>
<phase>process-sources</phase>
<configuration>
<target>
<exec executable="make" dir="${project.build.directory}/zstd/zstd-${zstd.version}" failonerror="true">
<arg line="CFLAGS='-fPIC'"/>
</exec>
</target>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
<execution>
<id>build-snappy</id>
<phase>process-sources</phase>
<configuration>
<target>
<mkdir dir="${project.build.directory}/snappy/lib"/>
<exec executable="cmake" failonerror="true" dir="${project.build.directory}/snappy/lib">
<arg line="${project.build.directory}/snappy/snappy-${snappy.version}"/>
<env key="CFLAGS" value="-fPIC"/>
<env key="CXXFLAGS" value="-fPIC"/>
</exec>
<exec executable="make" dir="${project.build.directory}/snappy/lib" failonerror="true"/>
</target>
</configuration>
<goals>
Expand All @@ -320,8 +236,7 @@
</exec>
<exec executable="make" dir="${project.build.directory}/rocksdb/rocksdb-${rocksdb.version}" failonerror="true">
<arg line="DEBUG_LEVEL=0"/>
<arg line="EXTRA_CXXFLAGS='-fPIC -I${project.build.directory}/snappy/lib -I${project.build.directory}/snappy/snappy-${snappy.version} -I${project.build.directory}/lz4/lz4-${lz4.version}/lib -I${project.build.directory}/zstd/zstd-${zstd.version}/lib -I${project.build.directory}/zstd/zstd-${zstd.version}/lib/dictBuilder -I${project.build.directory}/bzip2/bzip2-${bzip2.version} -I${project.build.directory}/zlib/zlib-${zlib.version}'"/>
<arg line="EXTRA_LDFLAGS='-L${project.build.directory}/snappy/lib -L${project.build.directory}/lz4/lz4-${lz4.version}/lib -L${project.build.directory}/zstd/zstd-${zstd.version}/lib -L${project.build.directory}/bzip2/bzip2-${bzip2.version} -L${project.build.directory}/zlib/zlib-${zlib.version}'"/>
<arg line="EXTRA_CXXFLAGS='-fPIC -D_GLIBCXX_USE_CXX11_ABI=0'"/>
<arg line="-j${system.numCores}"/>
<arg line="tools"/>
</exec>
Expand All @@ -346,14 +261,12 @@
<arg line="-DSST_DUMP_INCLUDE=${sstDump.include}"/>
<arg line="-DCMAKE_STANDARDS=${cmake.standards}"/>
<arg line="-DROCKSDB_HEADERS=${project.build.directory}/rocksdb/rocksdb-${rocksdb.version}/include"/>
<arg line="-DROCKSDB_LIB=${project.build.directory}/rocksdb/rocksdb-${rocksdb.version}"/>
<arg line="-DZLIB_LIB=${project.build.directory}/zlib/zlib-${zlib.version}"/>
<arg line="-DBZIP2_LIB=${project.build.directory}/bzip2/bzip2-${bzip2.version}"/>
<arg line="-DLZ4_LIB=${project.build.directory}/lz4/lz4-${lz4.version}/lib"/>
<arg line="-DSNAPPY_LIB=${project.build.directory}/snappy/lib"/>
<arg line="-DZSTD_LIB=${project.build.directory}/zstd/zstd-${zstd.version}/lib"/>
<arg line="-DROCKSDB_TOOLS_LIB=${project.build.directory}/rocksdb/rocksdb-${rocksdb.version}"/>
<arg line="-DROCKSDB_LIB=${project.build.directory}/rocksdbjni/${rocksdbLibName}"/>
</exec>
<exec executable="make" dir="${project.build.directory}/native/rocksdb" failonerror="true"/>
<delete dir="${project.build.directory}/rocksdb/rocksdb-${rocksdb.version}"
includeemptydirs="true"/>
</target>
</configuration>
<goals>
Expand Down Expand Up @@ -423,8 +336,8 @@
<configuration>
<javahPath>${env.JAVA_HOME}/bin/javah</javahPath>
<javahClassNames>
<javahClassName>org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool</javahClassName>
<javahClassName>org.apache.hadoop.hdds.utils.db.managed.PipeInputStream</javahClassName>
<javahClassName>org.apache.hadoop.hdds.utils.db.managed.ManagedRawSSTFileReader</javahClassName>
<javahClassName>org.apache.hadoop.hdds.utils.db.managed.ManagedRawSSTFileReaderIterator</javahClassName>
</javahClassNames>
<javahOutputDirectory>${project.build.directory}/native/javah</javahOutputDirectory>
</configuration>
Expand Down Expand Up @@ -479,8 +392,8 @@
<argument>${project.build.outputDirectory}:${project.build.directory}/dependency/*</argument>
<argument>-h</argument>
<argument>${project.build.directory}/native/javah</argument>
<argument>${project.basedir}/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedSSTDumpTool.java</argument>
<argument>${project.basedir}/src/main/java/org/apache/hadoop/hdds/utils/db/managed/PipeInputStream.java</argument>
<argument>${project.basedir}/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRawSSTFileReader.java</argument>
<argument>${project.basedir}/src/main/java/org/apache/hadoop/hdds/utils/db/managed/ManagedRawSSTFileReaderIterator.java</argument>
</arguments>
</configuration>
</execution>
Expand Down
Loading