Skip to content

Add environment file & documentation for GPU tests #633

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
charlesbluca merged 3 commits into from
Jul 25, 2022
Merged

Add environment file & documentation for GPU tests #633

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
af10637
Add gpuCI environment file
charlesbluca Jul 16, 2022
a714f1e
Add documentation for GPU tests / environment
charlesbluca Jul 16, 2022
a4a98b1
Add GPU testing to docs page
charlesbluca Jul 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,19 @@ You can run the tests (after installation) with

pytest tests

GPU-specific tests require additional dependencies specified in `continuous_integration/gpuci/environment.yaml`.
These can be added to the development environment by running

```
conda env update -n dask-sql -f continuous_integration/gpuci/environment.yaml
```

And GPU-specific tests can be run with

```
pytest tests -m gpu --rungpu
```

## SQL Server

`dask-sql` comes with a small test implementation for a SQL server.
Expand Down
17 changes: 17 additions & 0 deletions continuous_integration/gpuci/environment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: gpuci
channels:
- rapidsai
- rapidsai-nightly
- nvidia
dependencies:
- rust>=1.60.0
Copy link
Collaborator

@ayushdg ayushdg Jul 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we target these changes for the datafusion branch?

Copy link
Collaborator Author

@charlesbluca charlesbluca Jul 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would make sense, though on gpuCI's end we would need to grab this file from one of the branches to pull in during image builds (example of this being done with the environment file), and it just seemed to make more sense to target main for this.

If we want to have different versions of this file in main and the datafusion branch, that might motivate using separate images for gpuCI running on datafusion PRs vs. PRs into main, though not sure if that's worth the extra builds since the images only differ by a handful of added packages

Copy link
Collaborator

@ayushdg ayushdg Jul 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense w.r.t CI. It might also make sense to include this file (in a separate PR) in the datafusion branch even if we don't use it for gpuCI since one of the motivations for the file was for users working on that branch to be able to setup gpu environments to run tests locally

- setuptools-rust>=1.2.0
- cudatoolkit=11.5
- cudf=22.08
- cuml=22.08
- dask-cudf=22.08
- dask-cuda=22.08
- numpy>=1.20.0
- ucx-proc=*=gpu
- ucx-py=0.27
- xgboost=*=cuda_*
13 changes: 13 additions & 0 deletions docs/source/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,19 @@ You can run the tests (after installation) with

pytest tests

GPU-specific tests require additional dependencies specified in `continuous_integration/gpuci/environment.yaml`.
These can be added to the development environment by running

.. code-block:: bash

conda env update -n dask-sql -f continuous_integration/gpuci/environment.yaml

And GPU-specific tests can be run with

.. code-block:: bash

pytest tests -m gpu --rungpu

This repository uses pre-commit hooks. To install them, call

.. code-block:: bash
Expand Down