Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/rc.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this build the docs for the JNI components too? Or do we need to enable the JNI build to get the Javadocs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this build the docs for the JNI components too?

No.

Or do we need to enable the JNI build to get the Javadocs?

Yes. But it's easy because we already have JNI enabled jobs. I'll do it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we can use docker compose run vcpkg-jni but it doesn't create JAR. So I can't use it for mvn site.
I used ci/scripts/jni_full_build.sh instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, vcpkg-jni only builds the shared object and not the full package

Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,49 @@ jobs:
name: release-source
path: |
apache-arrow-java-*
docs:
name: Docs
runs-on: ubuntu-latest
timeout-minutes: 30
needs:
- source
env:
DOCKER_VOLUME_PREFIX: .docker/
permissions:
contents: read
packages: write
steps:
- name: Download source archive
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
name: release-source
- name: Extract source archive
run: |
tar -xf apache-arrow-java-*.tar.gz --strip-components=1
- uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Cache
uses: actions/cache@1bd1e32a3bdc45362d1e726936510720a7c30a57 # v4.2.0
with:
path: .docker
key: docs-${{ hashFiles('compose.yaml', '**/pom.xml', '**/*.java') }}
restore-keys: docs-
- name: Build
run: |
docker compose run \
--volume "${PWD}/build/:/build/" \
docs
- name: Compress into single artifact to keep directory structure
run: |
tar -cvzf docs.tar.gz -C build/java docs
- name: Upload artifacts
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: release-docs-${{ matrix.platform.arch }}
path: docs.tar.gz
jni-linux:
name: JNI ${{ matrix.platform.runs_on }} ${{ matrix.platform.arch }}
runs-on: ${{ matrix.platform.runs_on }}
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
uses: actions/cache@6849a6489940f00c2f30c0fb92c6274307ccb58a # v4.1.2
with:
path: .docker
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('**/compose.yaml', '**/pom.xml') }}
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('compose.yaml', '**/pom.xml', '**/*.java') }}
restore-keys: maven-${{ matrix.jdk }}-${{ matrix.maven }}-
- name: Execute Docker Build
env:
Expand Down Expand Up @@ -108,12 +108,12 @@ jobs:
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh $(pwd) $(pwd)/build
run: ci/scripts/build.sh . build jni
- name: Test
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh $(pwd) $(pwd)/build
run: ci/scripts/test.sh . build jni

windows:
name: AMD64 Windows Server 2022 Java JDK ${{ matrix.jdk }}
Expand All @@ -139,12 +139,12 @@ jobs:
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh $(pwd) $(pwd)/build
run: ci/scripts/build.sh . build jni
- name: Test
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh $(pwd) $(pwd)/build
run: ci/scripts/test.sh . build jni

integration:
name: AMD64 integration
Expand Down
37 changes: 23 additions & 14 deletions ci/scripts/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,19 @@
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
set -euo pipefail

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
exit
fi

source_dir=${1}
build_dir=${2}
build_dir=${2}/java
java_jni_dist_dir=${3}

: "${BUILD_DOCS_JAVA:=OFF}"

mvn="mvn -B -DskipTests -Drat.skip=true -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"

if [ "$ARROW_JAVA_SKIP_GIT_PLUGIN" ]; then
if [ "${ARROW_JAVA_SKIP_GIT_PLUGIN:-OFF}" = "ON" ]; then
mvn="${mvn} -Dmaven.gitcommitid.skip=true"
fi

Expand Down Expand Up @@ -61,27 +59,38 @@ done

pushd "${build_dir}"

if [ "${ARROW_JAVA_SHADE_FLATBUFFERS}" == "ON" ]; then
# TODO: ARROW_JAVA_SHADE_FLATBUFFERS isn't used for our artifacts. Do
# we need this?
# See also: https://github.com/apache/arrow/issues/22021
if [ "${ARROW_JAVA_SHADE_FLATBUFFERS:-OFF}" == "ON" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to revisit this sometime because of #67 (and it appears Google has closed the issue asking them to fix it upstream).

AFAIK the discussion in the issue you found is weird...We shouldn't need to change imports ourselves, the shading should take care of that. But that's not relevant here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Can we use #67 for this topic? Or should we create a separated issue for this topic?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can use #67!

mvn="${mvn} -Pshade-flatbuffers"
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
if [ "${ARROW_JAVA_CDATA:-OFF}" = "ON" ]; then
mvn="${mvn} -Darrow.c.jni.dist.dir=${java_jni_dist_dir} -Parrow-c-data"
fi

if [ "${ARROW_JAVA_JNI}" = "ON" ]; then
if [ "${ARROW_JAVA_JNI:-OFF}" = "ON" ]; then
mvn="${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni"
fi

# Use `2 * ncores` threads
${mvn} -T 2C clean install

if [ "${BUILD_DOCS_JAVA}" == "ON" ]; then
# HTTP pooling is turned of to avoid download issues https://issues.apache.org/jira/browse/ARROW-11633
# GH-43378: Maven site plugins not compatible with multithreading
mkdir -p "${build_dir}"/docs/java/reference
${mvn} -Dcheckstyle.skip=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false clean install site
rsync -a target/site/apidocs/ "${build_dir}"/docs/java/reference
if [ "${ARROW_JAVA_BUILD_DOCS:-OFF}" == "ON" ]; then
# HTTP pooling is turned off to avoid download issues:
# https://github.com/apache/arrow/issues/27496
#
# Maven site plugins not compatible with multithreading:
# https://github.com/apache/arrow/issues/43378
${mvn} \
-Dcheckstyle.skip=true \
-Dhttp.keepAlive=false \
-Dmaven.wagon.http.pool=false \
site
rm -rf docs/reference
mkdir -p docs
cp -a target/site/apidocs/ docs/reference
fi

popd
2 changes: 1 addition & 1 deletion ci/scripts/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
fi

source_dir=${1}
build_dir=${2}
build_dir=${2}/java
java_jni_dist_dir=${3}

mvn="mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
Expand Down
28 changes: 25 additions & 3 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,30 @@ services:
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
command:
/bin/bash -c "
/arrow-java/ci/scripts/build.sh /arrow-java /build &&
/arrow-java/ci/scripts/test.sh /arrow-java /build"
/arrow-java/ci/scripts/build.sh /arrow-java /build /jni &&
/arrow-java/ci/scripts/test.sh /arrow-java /build /jni"

docs:
# Build docs.
#
# Usage:
# docker compose build docs
# docker compose run docs
# Parameters:
# MAVEN: 3.9.9
# JDK: 11, 17, 21
image: ${ARCH}/maven:${MAVEN}-eclipse-temurin-${JDK}
volumes:
- .:/arrow-java:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
environment:
ARROW_JAVA_BUILD_DOCS: "ON"
ARROW_JAVA_SKIP_GIT_PLUGIN: "ON"
command:
- /arrow-java/ci/scripts/build.sh
- /arrow-java
- /build
- /jni

conda-jni-cdata:
# Builds and tests just the C Data Interface JNI library and JARs.
Expand Down Expand Up @@ -103,7 +125,7 @@ services:
volumes:
- .:/arrow-java:delegated
- ${ARROW_REPO_ROOT}:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}ccache:/ccache:delegated
- ${DOCKER_VOLUME_PREFIX}ccache-cache:/ccache:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
environment:
ARROW_JAVA_CDATA: "ON"
Expand Down
Loading