-
Notifications
You must be signed in to change notification settings - Fork 220
Portable standalone tarball build with Docker #2443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 17 commits
0342601
a936a82
6e1d8b6
e86938b
3b6132c
f06426b
eac4049
0743a95
05fbbfe
472089c
266bd65
04c2e2b
a0e7f83
fabdd4f
30041f5
c048d87
4430d5e
8a7b7fc
b45f475
fb6d382
c8e26d4
2443799
bbbe993
5419933
3ca9566
279e6cd
143a10e
1ba4f83
f812e92
6f1cf27
d073064
de7e247
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Environment and runner for the standalone C build (see ci/standalone_c/build_standalone_c.sh). | ||
| # Build args: CUDA_VERSION (default 13.0), PYTHON_VERSION (default 3.11). | ||
| # | ||
| # End-user instructions: https://docs.nvidia.com/cuvs/installation#build-the-standalone-c-library-with-docker | ||
| # Quick run: docker run -v $(pwd):/workspace -v $(pwd)/build:/build <image> [--tarball-build-tests] | ||
|
|
||
| ARG CUDA_VERSION=13.0 | ||
| ARG PYTHON_VERSION=3.11 | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
| FROM rapidsai/ci-wheel:26.10-cuda${CUDA_VERSION}-rockylinux8-py${PYTHON_VERSION} | ||
|
|
||
| ARG NINJA_VERSION=v1.13.1 | ||
|
|
||
| # System packages required by ci/standalone_c/build_standalone_c.sh | ||
| RUN dnf install -y \ | ||
| patch \ | ||
| tar \ | ||
| unzip \ | ||
| wget \ | ||
| && dnf clean all | ||
|
|
||
| # Install ninja (architecture-aware) | ||
| RUN set -euo pipefail; \ | ||
| if ! command -V ninja >/dev/null 2>&1; then \ | ||
| case "$(uname -m)" in \ | ||
| x86_64) \ | ||
| wget --no-hsts -q -O /tmp/ninja-linux.zip "https://github.com/ninja-build/ninja/releases/download/${NINJA_VERSION}/ninja-linux.zip"; \ | ||
| ;; \ | ||
| aarch64) \ | ||
| wget --no-hsts -q -O /tmp/ninja-linux.zip "https://github.com/ninja-build/ninja/releases/download/${NINJA_VERSION}/ninja-linux-aarch64.zip"; \ | ||
| ;; \ | ||
| *) \ | ||
| echo "Unrecognized platform '$(uname -m)'" >&2; exit 1; \ | ||
| ;; \ | ||
| esac; \ | ||
| unzip -d /usr/bin /tmp/ninja-linux.zip; \ | ||
| chmod +x /usr/bin/ninja; \ | ||
| rm /tmp/ninja-linux.zip; \ | ||
| fi | ||
|
|
||
| # CMake; sccache/date are configured at runtime by ci/standalone_c/build_standalone_c.sh | ||
| RUN rapids-pip-retry install cmake \ | ||
| && pyenv rehash | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
|
|
||
| # Output directory for the standalone archive. Bind-mount a host folder here | ||
| # (e.g. -v $(pwd)/build:/build) so libcuvs_c.tar.gz is written to the host. | ||
| ENV BUILD_OUTPUT_DIR=/build | ||
| ENV CUVS_TARBALL_IN_CONTAINER=1 | ||
|
|
||
| # Run from repo root; the repo is expected to be bind-mounted at /workspace. | ||
| WORKDIR /workspace | ||
|
|
||
| # Run the build script, which writes the standalone tarball to the mounted output dir. | ||
| # Example: docker run -v $(pwd):/workspace -v $(pwd)/build:/build <image> [--tarball-build-tests] | ||
| ENTRYPOINT ["/bin/bash", "-c", "ci/standalone_c/build_standalone_c.sh \"$@\"", "--"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,7 +19,7 @@ ARGS=$* | |
| # scripts, and that this script resides in the repo dir! | ||
| REPODIR=$(cd "$(dirname "$0")"; pwd) | ||
|
|
||
| VALIDARGS="clean libcuvs python rust go java docs tests bench-ann examples --uninstall -v -g -n --allgpuarch --no-mg --mnmg-tests --no-cpu --cpu-only --no-shared-libs --no-nvtx --show_depr_warn --incl-cache-stats --time -h --run-java-tests" | ||
| VALIDARGS="clean libcuvs python rust go java docs tests bench-ann examples tarball --tarball-build-tests --uninstall -v -g -n --allgpuarch --no-mg --mnmg-tests --no-cpu --cpu-only --no-shared-libs --no-nvtx --show_depr_warn --incl-cache-stats --time -h --run-java-tests" | ||
| HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<tool>] [--limit-tests=<targets>] [--limit-bench-ann=<targets>] [--build-metrics=<filename>] | ||
| where <target> is: | ||
| clean - remove all existing build artifacts and configuration (start over) | ||
|
|
@@ -33,6 +33,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to | |
| tests - build the tests | ||
| bench-ann - build end-to-end ann benchmarks | ||
| examples - build the examples | ||
| tarball - build the standalone C library tarball with Docker | ||
|
|
||
| and <flag> is: | ||
| -v - verbose build mode | ||
|
|
@@ -56,6 +57,7 @@ HELP="$0 [<target> ...] [<flag> ...] [--cmake-args=\"<args>\"] [--cache-tool=<to | |
| --no-shared-libs - build without shared libraries | ||
| --show_depr_warn - show cmake deprecation warnings | ||
| --run-java-tests - run Java tests after building | ||
| --tarball-build-tests - include the C library tests in the standalone tarball | ||
| --build-metrics - filename for generating build metrics report for libcuvs | ||
| --incl-cache-stats - include cache statistics in build metrics report | ||
| --cmake-args=\\\"<args>\\\" - pass arbitrary list of CMake configuration options (escape all quotes in argument) | ||
|
|
@@ -574,3 +576,61 @@ if hasArg examples; then | |
| ./build.sh | ||
| popd | ||
| fi | ||
|
|
||
| ################################################################################ | ||
| # Build the standalone C library tarball (if requested) | ||
|
|
||
| if hasArg tarball; then | ||
| if [[ "${CUVS_TARBALL_IN_CONTAINER:-0}" == "1" ]]; then | ||
| BUILD_OUTPUT_DIR="${BUILD_OUTPUT_DIR:-${REPODIR}}" | ||
| tar czf "${BUILD_OUTPUT_DIR}/libcuvs_c.tar.gz" -C "${REPODIR}/c/build/install" . | ||
| ls -lh "${BUILD_OUTPUT_DIR}/libcuvs_c.tar.gz" | ||
| else | ||
| CUDA_VERSION="${CUDA_VERSION:-13.0}" | ||
| PYTHON_VERSION="${PYTHON_VERSION:-3.11}" | ||
| BUILD_OUTPUT_DIR="${BUILD_OUTPUT_DIR:-${REPODIR}/build}" | ||
| IMAGE_NAME="${IMAGE_NAME:-cuvs-standalone-c}" | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see anything in this PR / repo that's overriding this value... recommend just hard-coding it and removing the indirection. In general I think this PR would benefit from a pass for conciseness / strictness. There's a lot of stuff here that appears to be more flexible that we really need. If you do really want to keep the ability to overwrite these things, then the environment variable names should be made more specific, e.g. |
||
|
|
||
| BUILD_ARGS=() | ||
| if hasArg --tarball-build-tests; then | ||
| BUILD_ARGS+=(--tarball-build-tests) | ||
| fi | ||
|
|
||
| mkdir -p "${BUILD_OUTPUT_DIR}" | ||
| BUILD_OUTPUT_DIR_ABS=$(cd "${BUILD_OUTPUT_DIR}"; pwd) | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
|
|
||
| echo "Building Docker image ${IMAGE_NAME} (CUDA ${CUDA_VERSION}, Python ${PYTHON_VERSION})..." | ||
| docker build -f "${REPODIR}/Dockerfile.standalone" \ | ||
| --build-arg CUDA_VERSION="${CUDA_VERSION}" \ | ||
| --build-arg PYTHON_VERSION="${PYTHON_VERSION}" \ | ||
| -t "${IMAGE_NAME}" \ | ||
| "${REPODIR}" | ||
|
|
||
| DOCKER_ENV=() | ||
| for var in ${CUVS_TARBALL_DOCKER_ENV_VARS:-}; do | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
| if [[ ! "${var}" =~ ^[a-zA-Z_][a-zA-Z0-9_]*$ ]]; then | ||
| echo "Invalid Docker environment variable name: ${var}" >&2 | ||
| exit 1 | ||
| fi | ||
| if [[ -n "${!var:-}" ]]; then | ||
| DOCKER_ENV+=(-e "${var}=${!var}") | ||
| fi | ||
| done | ||
|
|
||
| echo "Running standalone C build in container..." | ||
| docker run --rm \ | ||
| -v "${REPODIR}:/workspace" \ | ||
| -v "${BUILD_OUTPUT_DIR_ABS}:/build" \ | ||
| "${DOCKER_ENV[@]}" \ | ||
| "${IMAGE_NAME}" \ | ||
| "${BUILD_ARGS[@]}" | ||
|
|
||
| if [[ ! -f "${BUILD_OUTPUT_DIR_ABS}/libcuvs_c.tar.gz" ]]; then | ||
| echo "Expected tarball not found at ${BUILD_OUTPUT_DIR_ABS}/libcuvs_c.tar.gz" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| cp -v "${BUILD_OUTPUT_DIR_ABS}/libcuvs_c.tar.gz" "${REPODIR}/libcuvs_c.tar.gz" | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
| echo "Copied libcuvs_c.tar.gz to ${REPODIR}/libcuvs_c.tar.gz" | ||
| fi | ||
| fi | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| #!/bin/bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2025-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # | ||
| # Pass CI-specific environment variables to the standalone tarball build. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" | ||
|
|
||
| export CUVS_TARBALL_DOCKER_ENV_VARS="AWS_REGION AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN RAPIDS_AUX_SECRET_1 RAPIDS_ARTIFACTS_DIR RAPIDS_BUILD_TYPE RAPIDS_DATETIME_STRING RAPIDS_REPOSITORY RAPIDS_SHA RAPIDS_REF_NAME RAPIDS_NIGHTLY_DATE" | ||
|
cjnolet marked this conversation as resolved.
Outdated
|
||
|
|
||
| exec "${REPO_ROOT}/build.sh" tarball "$@" | ||
Uh oh!
There was an error while loading. Please reload this page.