Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion .github/workflows/rc.yml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this build the docs for the JNI components too? Or do we need to enable the JNI build to get the Javadocs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this build the docs for the JNI components too?

No.

Or do we need to enable the JNI build to get the Javadocs?

Yes. But it's easy because we already have JNI enabled jobs. I'll do it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we can use docker compose run vcpkg-jni but it doesn't create JAR. So I can't use it for mvn site.
I used ci/scripts/jni_full_build.sh instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, vcpkg-jni only builds the shared object and not the full package

Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,14 @@ jobs:
contents: read
packages: write
steps:
- name: Prepare
run: |
if [ ${{ matrix.platform.arch }} = "x86_64" ]; then
ARROW_JAVA_BUILD_DOCS=ON
else
ARROW_JAVA_BUILD_DOCS=ON
fi
echo "ARROW_JAVA_BUILD_DOCS=${ARROW_JAVA_BUILD_DOCS}" >> ${GIHUTB_ENV}
- name: Download source archive
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
Expand Down Expand Up @@ -133,7 +141,10 @@ jobs:
restore-keys: jni-linux-${{ matrix.platform.arch }}-
- name: Build
run: |
docker compose run vcpkg-jni
docker compose run \
--env ARROW_JAVA_BUILD_DOCS=${ARROW_JAVA_BUILD_DOCS} \
--volume "${PWD}/build/:/build/" \
vcpkg-jni
- name: Push Docker image
if: success() && github.event_name == 'push' && github.repository == 'apache/arrow-java' && github.ref_name == 'main'
run: |
Expand All @@ -145,6 +156,15 @@ jobs:
with:
name: jni-linux-${{ matrix.platform.arch }}
path: jni-linux-${{ matrix.platform.arch }}.tar.gz
- name: Compress docs
if: env.ARROW_JAVA_BUILD_DOCS == 'docs'
run: tar -cvzf docs.tar.gz -C build/java docs
- name: Upload artifacts
if: env.ARROW_JAVA_BUILD_DOCS == 'docs'
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: release-docs-${{ matrix.platform.arch }}
path: docs.tar.gz
jni-macos:
name: JNI ${{ matrix.platform.runs_on }} ${{ matrix.platform.arch }}
runs-on: ${{ matrix.platform.runs_on }}
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
uses: actions/cache@6849a6489940f00c2f30c0fb92c6274307ccb58a # v4.1.2
with:
path: .docker
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('**/compose.yaml', '**/pom.xml') }}
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('compose.yaml', '**/pom.xml', '**/*.java') }}
restore-keys: maven-${{ matrix.jdk }}-${{ matrix.maven }}-
- name: Execute Docker Build
env:
Expand Down Expand Up @@ -108,12 +108,12 @@ jobs:
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh $(pwd) $(pwd)/build
run: ci/scripts/build.sh . build jni
- name: Test
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh $(pwd) $(pwd)/build
run: ci/scripts/test.sh . build jni

windows:
name: AMD64 Windows Server 2022 Java JDK ${{ matrix.jdk }}
Expand All @@ -139,12 +139,12 @@ jobs:
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh $(pwd) $(pwd)/build
run: ci/scripts/build.sh . build jni
- name: Test
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh $(pwd) $(pwd)/build
run: ci/scripts/test.sh . build jni

integration:
name: AMD64 integration
Expand Down
37 changes: 23 additions & 14 deletions ci/scripts/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,21 +17,19 @@
# specific language governing permissions and limitations
# under the License.

set -eo pipefail
set -euo pipefail

if [[ "${ARROW_JAVA_BUILD:-ON}" != "ON" ]]; then
exit
fi

source_dir=${1}
build_dir=${2}
build_dir=${2}/java
java_jni_dist_dir=${3}

: "${BUILD_DOCS_JAVA:=OFF}"

mvn="mvn -B -DskipTests -Drat.skip=true -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"

if [ "$ARROW_JAVA_SKIP_GIT_PLUGIN" ]; then
if [ "${ARROW_JAVA_SKIP_GIT_PLUGIN:-OFF}" = "ON" ]; then
mvn="${mvn} -Dmaven.gitcommitid.skip=true"
fi

Expand Down Expand Up @@ -61,27 +59,38 @@ done

pushd "${build_dir}"

if [ "${ARROW_JAVA_SHADE_FLATBUFFERS}" == "ON" ]; then
# TODO: ARROW_JAVA_SHADE_FLATBUFFERS isn't used for our artifacts. Do
# we need this?
# See also: https://github.com/apache/arrow/issues/22021
if [ "${ARROW_JAVA_SHADE_FLATBUFFERS:-OFF}" == "ON" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to revisit this sometime because of #67 (and it appears Google has closed the issue asking them to fix it upstream).

AFAIK the discussion in the issue you found is weird...We shouldn't need to change imports ourselves, the shading should take care of that. But that's not relevant here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Can we use #67 for this topic? Or should we create a separated issue for this topic?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can use #67!

mvn="${mvn} -Pshade-flatbuffers"
fi

if [ "${ARROW_JAVA_CDATA}" = "ON" ]; then
if [ "${ARROW_JAVA_CDATA:-OFF}" = "ON" ]; then
mvn="${mvn} -Darrow.c.jni.dist.dir=${java_jni_dist_dir} -Parrow-c-data"
fi

if [ "${ARROW_JAVA_JNI}" = "ON" ]; then
if [ "${ARROW_JAVA_JNI:-OFF}" = "ON" ]; then
mvn="${mvn} -Darrow.cpp.build.dir=${java_jni_dist_dir} -Parrow-jni"
fi

# Use `2 * ncores` threads
${mvn} -T 2C clean install

if [ "${BUILD_DOCS_JAVA}" == "ON" ]; then
# HTTP pooling is turned of to avoid download issues https://issues.apache.org/jira/browse/ARROW-11633
# GH-43378: Maven site plugins not compatible with multithreading
mkdir -p "${build_dir}"/docs/java/reference
${mvn} -Dcheckstyle.skip=true -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false clean install site
rsync -a target/site/apidocs/ "${build_dir}"/docs/java/reference
if [ "${ARROW_JAVA_BUILD_DOCS:-OFF}" == "ON" ]; then
# HTTP pooling is turned off to avoid download issues:
# https://github.com/apache/arrow/issues/27496
#
# Maven site plugins not compatible with multithreading:
# https://github.com/apache/arrow/issues/43378
${mvn} \
-Dcheckstyle.skip=true \
-Dhttp.keepAlive=false \
-Dmaven.wagon.http.pool=false \
site
rm -rf docs/reference
mkdir -p docs
cp -a target/site/apidocs/ docs/reference
fi

popd
8 changes: 6 additions & 2 deletions ci/scripts/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,14 @@ if [[ "${ARROW_JAVA_TEST:-ON}" != "ON" ]]; then
exit
fi

source_dir=${1}
build_dir=${2}
source_dir="$(cd "${1}" && pwd)"
build_dir=${2}/java
java_jni_dist_dir=${3}

if [ -d "${java_jni_dist_dir}" ]; then
java_jni_dist_dir="$(cd "${java_jni_dist_dir}" && pwd)"
fi

mvn="mvn -B -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"
# Use `2 * ncores` threads
mvn="${mvn} -T 2C"
Expand Down
6 changes: 3 additions & 3 deletions compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ services:
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
command:
/bin/bash -c "
/arrow-java/ci/scripts/build.sh /arrow-java /build &&
/arrow-java/ci/scripts/test.sh /arrow-java /build"
/arrow-java/ci/scripts/build.sh /arrow-java /build /jni &&
/arrow-java/ci/scripts/test.sh /arrow-java /build /jni"

conda-jni-cdata:
# Builds and tests just the C Data Interface JNI library and JARs.
Expand Down Expand Up @@ -103,7 +103,7 @@ services:
volumes:
- .:/arrow-java:delegated
- ${ARROW_REPO_ROOT}:/arrow:delegated
- ${DOCKER_VOLUME_PREFIX}ccache:/ccache:delegated
- ${DOCKER_VOLUME_PREFIX}ccache-cache:/ccache:delegated
- ${DOCKER_VOLUME_PREFIX}maven-cache:/root/.m2:delegated
environment:
ARROW_JAVA_CDATA: "ON"
Expand Down
Loading