-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Define pandas and polars compatibility testing versions in dependencies.yaml #22189
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8f2e2b8
81b0f70
8443e4b
fb4de7a
461f6ad
f40253e
8052c36
5f7106e
dc2d077
9812afb
6565365
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,29 +15,11 @@ CUDF_POLARS_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="cudf_polars_${RAPIDS_PY_CUDA_SUFF | |
| LIBCUDF_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcudf_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp) | ||
| PYLIBCUDF_WHEELHOUSE=$(rapids-download-from-github "$(rapids-package-name "wheel_python" pylibcudf --stable --cuda "$RAPIDS_CUDA_VERSION")") | ||
|
|
||
| rapids-logger "Installing cudf_polars and its dependencies" | ||
|
|
||
| # generate constraints (possibly pinning to oldest support versions of dependencies) | ||
| rapids-generate-pip-constraints py_test_cudf_polars "${PIP_CONSTRAINT}" | ||
|
|
||
| # notes: | ||
| # | ||
| # * echo to expand wildcard before adding `[test]` requires for pip | ||
| # * just providing --constraint="${PIP_CONSTRAINT}" to be explicit, and because | ||
| # that environment variable is ignored if any other --constraint are passed via the CLI | ||
| # | ||
| rapids-pip-retry install \ | ||
| -v \ | ||
| --prefer-binary \ | ||
| --constraint "${PIP_CONSTRAINT}" \ | ||
| "$(echo "${CUDF_POLARS_WHEELHOUSE}"/cudf_polars_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)[test,dask,ray]" \ | ||
| "$(echo "${LIBCUDF_WHEELHOUSE}"/libcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \ | ||
| "$(echo "${PYLIBCUDF_WHEELHOUSE}"/pylibcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" | ||
|
|
||
| rapids-logger "Run cudf_polars tests" | ||
|
|
||
| available_polars_versions=$(python -m pip index versions polars --json | jq '.versions') | ||
| POLARS_VERSIONS=$(python ci/utils/filter_package_versions.py dependencies.yaml run_cudf_polars polars "$available_polars_versions") | ||
| read -r -a VERSIONS <<< "$(python ci/utils/get_matrix_values.py dependencies.yaml test_cudf_polars_compat polars_compat_version)" | ||
| LATEST_VERSION="${VERSIONS[-1]}" | ||
|
|
||
| # shellcheck disable=SC2317 | ||
| function set_exitcode() | ||
|
|
@@ -51,14 +33,41 @@ set +e | |
| PASSED=() | ||
| FAILED=() | ||
|
|
||
| read -r -a VERSIONS <<< "${POLARS_VERSIONS}" | ||
| LATEST_VERSION="${VERSIONS[-1]}" | ||
|
|
||
| for version in "${VERSIONS[@]}"; do | ||
| rapids-logger "Installing polars==${version}" | ||
| rapids-pip-retry install -U "polars==${version}" | ||
|
|
||
| rapids-logger "Running tests for polars==${version}" | ||
| rapids-logger "Testing cudf_polars with polars ${version}.*" | ||
|
|
||
| # Generate requirements for this polars compat version. | ||
| polars_requirements_txt="polars-compat-${version}-requirements.txt" | ||
| rapids-dependency-file-generator \ | ||
| --config dependencies.yaml \ | ||
| --file-key test_cudf_polars_compat \ | ||
| --output requirements \ | ||
| --matrix "polars_compat_version=${version}" \ | ||
| > "${polars_requirements_txt}" | ||
|
|
||
| env_name="venv_polars_${version}" | ||
| python -m venv --clear "${env_name}" | ||
| # shellcheck disable=SC1090 | ||
| source "${env_name}/bin/activate" | ||
|
|
||
| rapids-logger "Installing cudf_polars and its dependencies for polars ${version}.*" | ||
|
|
||
| # notes: | ||
| # | ||
| # * echo to expand wildcard before adding `[test]` requires for pip | ||
| # * just providing --constraint="${PIP_CONSTRAINT}" to be explicit, and because | ||
| # that environment variable is ignored if any other --constraint are passed via the CLI | ||
| # | ||
| rapids-pip-retry install \ | ||
| -v \ | ||
| --prefer-binary \ | ||
| --constraint "${PIP_CONSTRAINT}" \ | ||
| "$(echo "${CUDF_POLARS_WHEELHOUSE}"/cudf_polars_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)[test,dask,ray]" \ | ||
| "$(echo "${LIBCUDF_WHEELHOUSE}"/libcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \ | ||
| "$(echo "${PYLIBCUDF_WHEELHOUSE}"/pylibcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \ | ||
| -r "polars-compat-${version}-requirements.txt" | ||
|
|
||
|
Comment on lines
+48
to
+69
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Guard venv setup/install failures to avoid running tests in a bad environment. With Suggested patch- python -m venv --clear "${env_name}"
+ if ! python -m venv --clear "${env_name}"; then
+ EXITCODE=1
+ FAILED+=("${version}")
+ rapids-logger "Failed to create venv for polars ${version}.*"
+ continue
+ fi
# shellcheck disable=SC1090
- source "${env_name}/bin/activate"
+ if ! source "${env_name}/bin/activate"; then
+ EXITCODE=1
+ FAILED+=("${version}")
+ rapids-logger "Failed to activate venv for polars ${version}.*"
+ rm -rf "${env_name}" "${polars_requirements_txt}"
+ continue
+ fi
...
- rapids-pip-retry install \
+ if ! rapids-pip-retry install \
-v \
--prefer-binary \
--constraint "${PIP_CONSTRAINT}" \
"$(echo "${CUDF_POLARS_WHEELHOUSE}"/cudf_polars_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)[test,dask,ray]" \
"$(echo "${LIBCUDF_WHEELHOUSE}"/libcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \
"$(echo "${PYLIBCUDF_WHEELHOUSE}"/pylibcudf_"${RAPIDS_PY_CUDA_SUFFIX}"*.whl)" \
- -r "polars-compat-${version}-requirements.txt"
+ -r "polars-compat-${version}-requirements.txt"; then
+ EXITCODE=1
+ FAILED+=("${version}")
+ rapids-logger "Dependency install failed for polars ${version}.*"
+ deactivate
+ rm -rf "${env_name}" "${polars_requirements_txt}"
+ continue
+ fiAlso applies to: 91-103 🧰 Tools🪛 Shellcheck (0.11.0)[info] 51-51: Not following: ./bin/activate was not specified as input (see shellcheck -x). (SC1091) 🤖 Prompt for AI Agents |
||
| rapids-logger "Running tests for polars ${version}.*" | ||
|
|
||
| if [ "${version}" == "${LATEST_VERSION}" ]; then | ||
| COVERAGE_ARGS=( | ||
|
|
@@ -79,13 +88,17 @@ for version in "${VERSIONS[@]}"; do | |
| -ra \ | ||
| --junitxml="${RAPIDS_TESTS_DIR}/junit-cudf-polars-${version}.xml" | ||
|
|
||
| if [ $? -ne 0 ]; then | ||
| test_exitcode=$? | ||
| deactivate | ||
| rm -rf "${env_name}" "${polars_requirements_txt}" | ||
|
|
||
| if [ ${test_exitcode} -ne 0 ]; then | ||
| EXITCODE=1 | ||
| FAILED+=("${version}") | ||
| rapids-logger "Tests failed for polars==${version}" | ||
| rapids-logger "Tests failed for polars ${version}.*" | ||
| else | ||
| PASSED+=("${version}") | ||
| rapids-logger "Tests passed for polars==${version}" | ||
| rapids-logger "Tests passed for polars ${version}.*" | ||
| fi | ||
| done | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """Extract matrix dimension values for a file key from dependencies.yaml.""" | ||
|
|
||
| import argparse | ||
|
|
||
| import yaml | ||
|
|
||
|
|
||
| def get_matrix_values( | ||
| deps_yaml_path: str, file_key: str, matrix_var: str | ||
| ) -> list[str]: | ||
| with open(deps_yaml_path) as f: | ||
| deps = yaml.safe_load(f) | ||
|
|
||
| file_cfg = deps.get("files", {}).get(file_key) | ||
| if file_cfg is None: | ||
| raise RuntimeError( | ||
| f"File key '{file_key}' not found in {deps_yaml_path}" | ||
| ) | ||
|
|
||
| matrix = file_cfg.get("matrix", {}) | ||
| values = matrix.get(matrix_var) | ||
| if values is None: | ||
| raise RuntimeError( | ||
| f"Matrix variable '{matrix_var}' not found in file key '{file_key}'" | ||
| ) | ||
|
|
||
| return [str(v) for v in values] | ||
|
Comment on lines
+24
to
+30
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Validate matrix value type before iterating. On Line 30, a scalar YAML value would be iterated as characters and silently produce invalid output. Fail fast unless the matrix value is a sequence. Suggested patch- values = matrix.get(matrix_var)
+ values = matrix.get(matrix_var)
if values is None:
raise RuntimeError(
f"Matrix variable '{matrix_var}' not found in file key '{file_key}'"
)
+ if not isinstance(values, (list, tuple)):
+ raise RuntimeError(
+ f"Matrix variable '{matrix_var}' in file key '{file_key}' must be a list"
+ )
return [str(v) for v in values]🤖 Prompt for AI Agents |
||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser( | ||
| description="Extract matrix dimension values for a file key in dependencies.yaml" | ||
| ) | ||
| parser.add_argument("deps_yaml", help="Path to dependencies.yaml") | ||
| parser.add_argument("file_key", help="File key in the 'files' section") | ||
| parser.add_argument("matrix_var", help="Matrix variable name to extract") | ||
| args = parser.parse_args() | ||
|
|
||
| values = get_matrix_values(args.deps_yaml, args.file_key, args.matrix_var) | ||
| print(" ".join(values)) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add an explicit GPU preflight check before running compat tests.
This loop assumes CUDA/GPU availability but never verifies it before invoking pytest. Add a fail-fast guard (e.g.,
nvidia-smi -L) to produce a clear CI error.Suggested patch
if version_lte "${RAPIDS_PY_VERSION}" "3.13"; then + if ! nvidia-smi -L >/dev/null 2>&1; then + rapids-logger "GPU not detected; cannot run cudf.pandas compatibility tests" + exit 1 + fi for version in "${versions[@]}"; do📝 Committable suggestion
🧰 Tools
🪛 Shellcheck (0.11.0)
[info] 108-108: Not following: ./bin/activate was not specified as input (see shellcheck -x).
(SC1091)
🤖 Prompt for AI Agents