Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -543,8 +543,15 @@ jobs:
java-gather:
needs: [java-build]
runs-on: linux-amd64-cpu4
# The container image is only used to make rapids-is-release-build
# available for the release-flag check below. The artifact assembly
# itself does not depend on any image-specific tooling.
container:
image: "rapidsai/ci-wheel:26.10-latest"
permissions:
contents: read
outputs:
is_release: ${{ steps.release-check.outputs.is_release }}
steps:
- name: Checkout code repo
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
Expand All @@ -568,3 +575,33 @@ jobs:
name: cudf_java_maven_repo
path: ${{ runner.temp }}/maven-repo
if-no-files-found: error
- name: Determine release-build status
id: release-check
run: |
if rapids-is-release-build; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking maybe this could be done in ./java/ci/assemble_maven_repo.sh? If you give an id: to that "Assemble Maven repository layout" step, you can reference its output directly and avoid having an extra step.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that, but placing it there felt out of place (the script is release-agnostic), and that is why I added it as an explicit step here.

echo "is_release=true" >> "${GITHUB_OUTPUT}"
else
echo "is_release=false" >> "${GITHUB_OUTPUT}"
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.

# Publish tagged release candidates to Maven Central via the Sonatype
# Central Publisher Portal. Release path only (vYY.MM.PP tags). Does not
# publish nightlies.
# TODO: add nightly Sonatype snapshot publishing.
java-publish:
needs: [java-gather]
if: ${{ needs.java-gather.outputs.is_release == 'true' && (inputs.build_type || 'branch') == 'branch' }}
permissions:
actions: read
contents: read
uses: rapidsai/shared-workflows/.github/workflows/maven-publish.yaml@main
secrets:
GPG_PRIVATE_KEY: ${{ secrets.GPG_PRIVATE_KEY }}
GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
MAVEN_DEPLOY_TOKEN: ${{ secrets.MAVEN_DEPLOY_TOKEN }}
with:
publication-type: 'rc'
artifact-name: cudf_java_maven_repo
source-git-sha: ${{ inputs.sha || github.sha }}
# false = validate + drop (safe). true = stage PENDING for manual publish.
stage-for-maven-central-publish: true
3 changes: 3 additions & 0 deletions ci/build_java.sh
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ if [[ -z ${RAPIDS_CUDA_VERSION:-} ]]; then
exit 1
fi

export HOST_UID="${HOST_UID:-$(id -u)}"
export HOST_GID="${HOST_GID:-$(id -g)}"

RAPIDS_CUDA_VERSION="$(cudf_java_normalize_cuda_version "${RAPIDS_CUDA_VERSION}")"
export RAPIDS_CUDA_VERSION
CLASSIFIER="$(cudf_java_maven_classifier "${RAPIDS_CUDA_VERSION}")"
Expand Down
63 changes: 46 additions & 17 deletions java/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,15 @@ so plain `rm -rf` works.
--cuda-version 12.9
```

This compiles the JNI layer against the static libcudf from Step 1 and emits a
single classifier JAR (e.g. `cudf-26.10.0-SNAPSHOT-cuda12.jar`) plus its POM
into a classifier-named subdirectory under `--output-dir`:
Optional `GITHUB_REF` selects release tag vs SNAPSHOT versioning. Unset means
SNAPSHOT. See the versioning section below.

```
This compiles the JNI layer against the static libcudf from Step 1 and emits
the classifier JAR (e.g. `cudf-26.10.0-SNAPSHOT-cuda12.jar`), a
classifier-independent sources jar and javadoc jar, and the POM into a
classifier-named subdirectory under `--output-dir`:
Comment thread
coderabbitai[bot] marked this conversation as resolved.

```text
/tmp/jars/cuda12/
cudf-26.10.0-SNAPSHOT-cuda12.jar
cudf-26.10.0-SNAPSHOT.pom
Expand All @@ -55,9 +59,10 @@ The classifier is derived from `--cuda-version` (major) + host arch (`uname
`aarch64`. Producing the ARM classifiers requires a real `aarch64` host.
Repeat Step 2 for each classifier, pointing `--libcudf-dir` at the matching
static libcudf tree and using the same `--output-dir` (each classifier lands
in its own subdirectory). Concurrent invocations for different classifiers
are safe because each nests its own bind-mount over `/repo/java/target`
inside the container.
in its own subdirectory). Concurrent SNAPSHOT invocations for different
classifiers are safe because each nests its own bind-mount over
`/repo/java/target` inside the container. Release builds rewrite the shared
`java/pom.xml` and must not overlap.

### Step 3 - Assemble the Maven repository layout

Expand All @@ -68,21 +73,45 @@ inside the container.
```

This walks every subdirectory of `--jars-dir` (each subdir name IS the
classifier), gathers the per-classifier JAR and shared POM, derives the
artifact version from the JAR filenames (requiring a single unique version
across subdirs), and lays them out as:

```
/tmp/maven-repo/ai/rapids/cudf/26.10.0-SNAPSHOT/
cudf-26.10.0-SNAPSHOT-cuda12.jar
cudf-26.10.0-SNAPSHOT-cuda13.jar
cudf-26.10.0-SNAPSHOT.pom
classifier), gathers the per-classifier JAR, one shared sources jar, one
shared javadoc jar, the shared POM, and seeds an unclassified primary JAR
as a copy of the `cuda12` classifier. Derives the artifact version from
the JAR filenames (requiring a single unique version across subdirs) and
lays them out as:

```text
/tmp/maven-repo/ai/rapids/cudf/<CUDF_VERSION>-SNAPSHOT/
cudf-<CUDF_VERSION>-SNAPSHOT.jar
cudf-<CUDF_VERSION>-SNAPSHOT-cuda12.jar
cudf-<CUDF_VERSION>-SNAPSHOT-cuda13.jar
cudf-<CUDF_VERSION>-SNAPSHOT-sources.jar
cudf-<CUDF_VERSION>-SNAPSHOT-javadoc.jar
cudf-<CUDF_VERSION>-SNAPSHOT.pom
```

The set of classifiers is whatever subdirectories are present under
`--jars-dir`. For a local `x86_64`-only run, populate `/tmp/jars/cuda12/`
and `/tmp/jars/cuda13/`. For the full four-way release build, add
`/tmp/jars/cuda12-arm64/` and `/tmp/jars/cuda13-arm64/`.
`/tmp/jars/cuda12-arm64/` and `/tmp/jars/cuda13-arm64/`. The `cuda12`
subdirectory is required because the unclassified primary JAR is copied from
it, so an `aarch64`-only set of subdirectories is not a valid gather input.

### Release Tag vs SNAPSHOT Versioning

Release tag CI runs (`GITHUB_REF=refs/tags/vYY.MM.PP`) produce release-versioned
JARs (`cudf-<CUDF_VERSION>-*.jar`). All other runs produce `-SNAPSHOT`. Gated by
[`rapids-is-release-build`](https://github.com/rapidsai/gha-tools/blob/main/tools/rapids-is-release-build).
`GITHUB_REF` is optional. Unset or non-tag values stay SNAPSHOT.

To rehearse the release path locally:

```bash
GITHUB_REF=refs/tags/vYY.MM.PP ./java/ci/test_java_build_local.sh
```

Rewrites `java/pom.xml` in place for packaging, then restores it on exit.

### GitHub Actions

In GitHub Actions (`.github/workflows/build.yaml`), the `java-build` matrix job
runs Steps 1-2 per (CUDA x arch) entry and uploads each classifier subdir as a
Expand Down
14 changes: 13 additions & 1 deletion java/ci/assemble_maven_repo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -166,12 +166,24 @@ if [[ -z "${FIRST_VERSION}" ]]; then
exit 1
fi

DEST_DIR="${OUTPUT_DIR}/${GROUP_PATH}/${ARTIFACT_ID}/${FIRST_VERSION}"

# Seed the unclassified primary from cuda12. Maven Central serves this to
# consumers depending on ai.rapids:cudf without a <classifier>.
PRIMARY_SOURCE="${DEST_DIR}/cudf-${FIRST_VERSION}-cuda12.jar"
if [[ ! -f "${PRIMARY_SOURCE}" ]]; then
echo "Error: ${PRIMARY_SOURCE} missing." >&2
exit 1
fi
UNCLASSIFIED="${DEST_DIR}/cudf-${FIRST_VERSION}.jar"
cp -f "${PRIMARY_SOURCE}" "${UNCLASSIFIED}"
echo " + cudf-${FIRST_VERSION}.jar (unclassified primary, copy of cuda12)"

# Sources and javadoc jars are classifier-independent (pure Java, no arch or
# cuda variation). Every classifier subdir produces byte-equivalent copies;
# pick the lexicographically first subdir's copy as canonical. Fail fast if
# any subdir is missing either file - that indicates -Prelease or
# -Pjavadoc-jdk17 did not activate for that classifier's build.
DEST_DIR="${OUTPUT_DIR}/${GROUP_PATH}/${ARTIFACT_ID}/${FIRST_VERSION}"
FIRST_CLASSIFIER_SUBDIR=""
for subdir in "${JARS_DIR}"/*/; do
if [[ -z ${FIRST_CLASSIFIER_SUBDIR} ]]; then
Expand Down
4 changes: 4 additions & 0 deletions java/ci/build_cudf_java_jar.sh
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,10 @@ DOCKER_ARGS=(
--env REPO_ROOT=/repo
)

if [[ -n ${GITHUB_REF:-} ]]; then
DOCKER_ARGS+=(--env GITHUB_REF="${GITHUB_REF}")
fi

if [[ -n ${CMAKE_CUDA_ARCHITECTURES} ]]; then
DOCKER_ARGS+=(--env CMAKE_CUDA_ARCHITECTURES="${CMAKE_CUDA_ARCHITECTURES}")
fi
Expand Down
35 changes: 30 additions & 5 deletions java/ci/build_cudf_java_jar_in_container.sh
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,25 @@ if [[ -z ${RAPIDS_CUDA_VERSION:-} ]]; then
exit 1
fi

_chown_outputs_on_exit() {
if [[ -n ${HOST_UID:-} && -n ${HOST_GID:-} ]]; then
chown -R "${HOST_UID}:${HOST_GID}" "${OUTPUT_DIR}" "${REPO_ROOT}/java/target" 2>/dev/null || true
if [[ -z ${HOST_UID} || -z ${HOST_GID} ]]; then
echo "Error: HOST_UID and HOST_GID must both be set" >&2
exit 1
fi

POM_WAS_REWRITTEN=0
_cleanup_on_exit() {
local prior_status=$?
if [[ ${POM_WAS_REWRITTEN} -eq 1 ]]; then
if ! mv -f "${REPO_ROOT}/java/pom.xml.backup" "${REPO_ROOT}/java/pom.xml"; then
echo "Warning: failed to restore ${REPO_ROOT}/java/pom.xml from pom.xml.backup" >&2
fi
fi
if ! chown -R "${HOST_UID}:${HOST_GID}" "${OUTPUT_DIR}" "${REPO_ROOT}/java/target"; then
echo "Warning: chown -R ${HOST_UID}:${HOST_GID} on ${OUTPUT_DIR} + ${REPO_ROOT}/java/target failed. Outputs may remain owned by root." >&2
fi
return "${prior_status}"
Comment thread
coderabbitai[bot] marked this conversation as resolved.
}
trap _chown_outputs_on_exit EXIT
trap _cleanup_on_exit EXIT

BUILD_ARG=(
-B
Expand Down Expand Up @@ -83,7 +96,18 @@ BUILD_ARG+=("-Dcmake.ccache.opts=${CMAKE_CCACHE_OPTS[*]}")

cd "${REPO_ROOT}/java"

CUDF_VERSION="$(cudf_java_scl mvn help:evaluate -Dexpression=project.version -q -DforceStdout "${BUILD_ARG[@]}")"
CUDF_VERSION="$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout "${BUILD_ARG[@]}")"

# Release tag builds strip -SNAPSHOT and rewrite the POM so packaged artifacts
# carry the release version. Non-release builds keep -SNAPSHOT. The EXIT trap
# restores java/pom.xml after packaging (the rewritten POM is copied to OUTPUT_DIR).
if rapids-is-release-build; then
CUDF_VERSION="${CUDF_VERSION%-SNAPSHOT}"
cp -p "${REPO_ROOT}/java/pom.xml" "${REPO_ROOT}/java/pom.xml.backup"
POM_WAS_REWRITTEN=1
mvn versions:set -DnewVersion="${CUDF_VERSION}" -DgenerateBackupPoms=false "${BUILD_ARG[@]}"
fi

Comment thread
NvTimLiu marked this conversation as resolved.
rapids-logger "Packaging cuDF Java JAR ${CUDF_VERSION}"

# Omit the `clean` goal: java/target may be a bind-mount point, so `mvn clean`
Expand Down Expand Up @@ -136,6 +160,7 @@ done

cp -f "${MAIN_JAR}" "${OUTPUT_DIR}/"
cp -f pom.xml "${OUTPUT_DIR}/cudf-${CUDF_VERSION}.pom"

rapids-logger "Emitted artifacts to ${OUTPUT_DIR}"
if command -v sccache >/dev/null 2>&1; then
sccache --show-adv-stats || true
Expand Down
19 changes: 15 additions & 4 deletions java/ci/build_static_libcudf_in_container.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,20 @@ if [[ -z ${RAPIDS_CUDA_VERSION:-} ]]; then
exit 1
fi

if [[ -z ${HOST_UID} || -z ${HOST_GID} ]]; then
echo "Error: HOST_UID and HOST_GID must both be set" >&2
exit 1
fi

_cleanup_on_exit() {
local prior_status=$?
if ! chown -R "${HOST_UID}:${HOST_GID}" "${INSTALL_PREFIX}"; then
echo "Warning: chown -R ${HOST_UID}:${HOST_GID} on ${INSTALL_PREFIX} failed. Outputs may remain owned by root." >&2
fi
return "${prior_status}"
}
trap _cleanup_on_exit EXIT
Comment thread
coderabbitai[bot] marked this conversation as resolved.

CMAKE_ARGS=(
-S "${REPO_ROOT}/cpp"
-B "${BUILD_DIR}"
Expand Down Expand Up @@ -75,10 +89,7 @@ cudf_java_scl cmake "${CMAKE_ARGS[@]}"
cmake --build "${BUILD_DIR}" --parallel "${PARALLEL_LEVEL}"
cmake --install "${BUILD_DIR}"

# Hand the install tree back to the host user (host wrapper passes HOST_UID/GID).
if [[ -n ${HOST_UID:-} && -n ${HOST_GID:-} ]]; then
chown -R "${HOST_UID}:${HOST_GID}" "${INSTALL_PREFIX}"
fi
rapids-logger "Emitted static libcudf install tree to ${INSTALL_PREFIX}"
if command -v sccache >/dev/null 2>&1; then
sccache --show-adv-stats || true
fi
Loading