Skip to content

HDDS-12821. Update Build from Source user doc. #8262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jojochuang merged 2 commits into from
Apr 14, 2025
Merged

HDDS-12821. Update Build from Source user doc. #8262

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7c60aaa
HDDS-12821. Update Build from Source user doc.
jojochuang Apr 11, 2025
c782f40
Add heading
jojochuang Apr 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 80 additions & 11 deletions hadoop-hdds/docs/content/start/FromSource.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,15 @@ weight: 30
-->

{{< requirements >}}
* Java 1.8
* Maven
* x86 or ARM64
* Linux or MacOS
* JDK 1.8 or higher
* [Maven 3.6 or later](https://maven.apache.org/download.cgi)
* Internet connection for first build (to fetch all Maven and Ozone dependencies)

You will also need to install the following dependencies to build native code (optional):
* gcc
* cmake
{{< /requirements >}}

<div class="alert alert-info" role="alert">
Expand All @@ -32,29 +39,91 @@ planning to build sources yourself, you can safely skip this page.

</div>

If you are a Hadoop ninja, and wise in the ways of Apache, you already know
that a real Apache release is a source release.

If you want to build from sources, Please untar the source tarball (or clone the latest code
from the [git repository](https://github.com/apache/ozone)) and run the ozone build command. This instruction assumes that you have all the
dependencies to build Hadoop on your build machine. If you need instructions
on how to build Hadoop, please look at the Apache Hadoop Website.
from the [git repository](https://github.com/apache/ozone)).

## ARM-based Linux
If you are using an ARM-based Linux, patch protobuf 2.5.0 and build it from source.

```bash
mvn clean package -DskipTests=true
sudo yum install -y make cmake gcc g++ patch
# or for Ubuntu and Debian:
# sudo apt-get install -y make cmake gcc g++ patch

# Download protobuf 2.5.0 tarball
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz | tar zx
cd protobuf-2.5.0

# patch protobuf 2.5.0
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
patch -p1 < protobuf-2.5.0-arm64.patch
# build protobuf
./configure --disable-shared
make
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x
cp $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-linux-aarch_64.exe
```

## ARM-based Apple Silicon (Apple M1 ... etc)

```bash
PROTOBUF_VERSION="3.7.1"
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-all-${PROTOBUF_VERSION}.tar.gz | tar zx
cd protobuf-${PROTOBUF_VERSION}
./configure --disable-shared
make -j
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe

cd ..
# Download protobuf 2.5.0 tarball
PROTOBUF_VERSION="2.5.0"
curl -sSL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOBUF_VERSION}/protobuf-${PROTOBUF_VERSION}.tar.gz | tar zx
cd protobuf-${PROTOBUF_VERSION}

Copy link
Contributor

@devmadhuu devmadhuu Apr 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we don't need to add below lines in src/google/protobuf/stubs/platform_macros.h ?

#elif defined(__arm64__)
#define GOOGLE_PROTOBUF_ARCH_ARM 1
#define GOOGLE_PROTOBUF_ARCH_64_BIT 1

Copy link
Contributor Author

@jojochuang jojochuang Apr 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we do. The next line downloads a patch files and builds it. It essentially does the same thing.

# patch protobuf 2.5.0
curl -L -O https://gist.githubusercontent.com/liusheng/64aee1b27de037f8b9ccf1873b82c413/raw/118c2fce733a9a62a03281753572a45b6efb8639/protobuf-2.5.0-arm64.patch
patch -p1 < protobuf-2.5.0-arm64.patch
# build protobuf
./configure --disable-shared
make
# install protoc to the local Maven repository
mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=${PROTOBUF_VERSION} -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=src/protoc
# workaround for Maven 3.9.x. Not needed for 3.8.x or earlier
cp $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64 $HOME/.m2/repository/com/google/protobuf/protoc/${PROTOBUF_VERSION}/protoc-${PROTOBUF_VERSION}-osx-aarch_64.exe
```

## Build Ozone
Run the ozone build command.

```bash
mvn clean package -DskipTests=true
```
This will build an `ozone-\<version\>` directory in your `hadoop-ozone/dist/target` directory.

You can copy this tarball and use this instead of binary artifacts that are
provided along with the official release.

To create tar file distribution, use the `-Pdist` profile:
Depending on the network speed, the build can take anywhere from 10 minutes to 20 minutes.

To create tarball file distribution, use the `-Pdist` profile:

```bash
mvn clean package -DskipTests=true -Pdist
```

## Other useful Maven build options

* Use `-DskipShade` to skip shaded Ozone FS jar file creation. Saves time, but you can't test integration with other software that uses Ozone as a Hadoop-compatible file system.
* Use `-DskipRecon` to skip building Recon Web UI. It saves about 2 minutes.
* Use `-Dmaven.javadoc.skip=true` to skip building javadocs.
* Use `-Drocks_tools_native` to build the RocksDB native code for the Ozone Snapshot feature. This is optional and not required for building Ozone. It is only needed if you want to build the RocksDB native code for Ozone.


## How to run Ozone from build

When you have the new distribution, you can start a local cluster [with docker-compose]({{< ref "start/RunningViaDocker.md">}}).
Expand All @@ -67,4 +136,4 @@ docker-compose up -d

## How to test the build

`compose` subfolder contains multiple type of example setup (secure, non-secure, HA, Yarn). They can be tested with the help of [robotframework](http://robotframework.org/) with executing `test.sh` in any of the directories.
`compose` subfolder contains multiple type of example setup (secure, non-secure, HA, Yarn). They can be tested with the help of [robotframework](http://robotframework.org/) with executing `test.sh` in any of the directories.