Repository navigation
Update README.md generalizing all cuDF components #20357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
f04968b
Update README.md generalizing all cuDF components
mroeschke 453c3f3
Apply suggestions from code review
mroeschke 4384ed6
Add examples section
mroeschke 1d9c104
Misspelled License
mroeschke 763e80a
Missing quotes
mroeschke 97b9dac
Merge remote-tracking branch 'upstream/main' into ref/cudf/readme
mroeschke 4f800b3
Add more detailed installation per library
mroeschke 401fe9f
Specify nightly from conda
mroeschke 3b77153
Grammar
mroeschke dc5a666
Merge remote-tracking branch 'upstream/main' into ref/cudf/readme
mroeschke e98dd41
Include nightly wheels location
mroeschke File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,99 +1,139 @@ | ||
| # <div align="left"><img src="img/rapids_logo.png" width="90px"/> cuDF - GPU DataFrames</div> | ||
| # <div align="left"><img src="img/rapids_logo.png" width="90px"/> cuDF - A GPU-accelerated DataFrame library for tabular data processing</div> | ||
|
|
||
| ## 📢 cuDF can now be used as a no-code-change accelerator for pandas! To learn more, see [here](https://rapids.ai/cudf-pandas/)! | ||
| cuDF (pronounced "KOO-dee-eff") is an [Apache 2.0 licensed](LICENSE), GPU-accelerated DataFrame library | ||
| for tabular data processing. The cuDF library is one part of the [RAPIDS](https://rapids.ai/) GPU | ||
| Accelerated Data Science suite of libraries. | ||
|
|
||
| cuDF (pronounced "KOO-dee-eff") is a GPU DataFrame library | ||
| for loading, joining, aggregating, filtering, and otherwise | ||
| manipulating data. cuDF leverages | ||
| [libcudf](https://docs.rapids.ai/api/libcudf/stable/), a | ||
| blazing-fast C++/CUDA dataframe library and the [Apache | ||
| Arrow](https://arrow.apache.org/) columnar format to provide a | ||
| GPU-accelerated pandas API. | ||
| ## About | ||
|
|
||
| You can import `cudf` directly and use it like `pandas`: | ||
| cuDF is composed of multiple libraries including: | ||
|
|
||
| ```python | ||
| import cudf | ||
| * [libcudf](https://docs.rapids.ai/api/cudf/stable/libcudf_docs/): A CUDA C++ library with [Apache Arrow](https://arrow.apache.org/) compliant | ||
| data structures and fundamental algorithms for tabular data. | ||
| * [pylibcudf](https://docs.rapids.ai/api/cudf/stable/pylibcudf/): A Python library providing [Cython](https://cython.org/) bindings for libcudf. | ||
| * [cudf](https://docs.rapids.ai/api/cudf/stable/user_guide/): A Python library providing | ||
| - A DataFrame library mirroring the [pandas](https://pandas.pydata.org/) API | ||
| - A zero-code change accelerator, [cudf.pandas](https://docs.rapids.ai/api/cudf/stable/cudf_pandas/), for existing pandas code. | ||
| * [cudf-polars](https://docs.rapids.ai/api/cudf/stable/cudf_polars/): A Python library providing a GPU engine for [Polars](https://pola.rs/) | ||
| * [dask-cudf](https://docs.rapids.ai/api/dask-cudf/stable/): A Python library providing a GPU backend for [Dask](https://www.dask.org/) DataFrames | ||
|
|
||
| tips_df = cudf.read_csv("https://github.com/plotly/datasets/raw/master/tips.csv") | ||
| tips_df["tip_percentage"] = tips_df["tip"] / tips_df["total_bill"] * 100 | ||
| Notable projects that use cuDF include: | ||
|
|
||
| # display average tip by dining party size | ||
| print(tips_df.groupby("size").tip_percentage.mean()) | ||
| ``` | ||
| * [Spark RAPIDS](https://github.com/NVIDIA/spark-rapids): A GPU accelerator plugin for [Apache Spark](https://spark.apache.org/) | ||
| * [Velox-cuDF](https://github.com/facebookincubator/velox/blob/main/velox/experimental/cudf/README.md): A [Velox](https://velox-lib.io/) | ||
| extension module to execute Velox plans on the GPU | ||
| * [Sirius](https://www.sirius-db.com/): A GPU-native SQL engine providing extensions for libraries like [DuckDB](https://duckdb.org/) | ||
|
|
||
| Or, you can use cuDF as a no-code-change accelerator for pandas, using | ||
| [`cudf.pandas`](https://docs.rapids.ai/api/cudf/stable/cudf_pandas). | ||
| `cudf.pandas` supports 100% of the pandas API, utilizing cuDF for | ||
| supported operations and falling back to pandas when needed: | ||
| ## Installation | ||
|
|
||
| ```python | ||
| %load_ext cudf.pandas # pandas operations now use the GPU! | ||
| ### System Requirements | ||
|
|
||
| import pandas as pd | ||
| Operating System, GPU driver, and supported CUDA version information can be found at the [RAPIDS Installation Guide](https://docs.rapids.ai/install/#system-req) | ||
|
|
||
| tips_df = pd.read_csv("https://github.com/plotly/datasets/raw/master/tips.csv") | ||
| tips_df["tip_percentage"] = tips_df["tip"] / tips_df["total_bill"] * 100 | ||
| ### pip | ||
|
|
||
| Each cudf library is available on PyPI. You will need to match the major version number of your installed CUDA version with a `-cu##` suffix | ||
| when installing from PyPI. | ||
|
|
||
| ```bash | ||
| # CUDA 13 | ||
| pip install libcudf-cu13 | ||
| pip install pylibcudf-cu13 | ||
| pip install cudf-cu13 | ||
| pip install cudf-polars-cu13 | ||
| pip install dask-cudf-cu13 | ||
|
|
||
| # display average tip by dining party size | ||
| print(tips_df.groupby("size").tip_percentage.mean()) | ||
| # CUDA 12 | ||
| pip install libcudf-cu12 | ||
| pip install pylibcudf-cu12 | ||
| pip install cudf-cu12 | ||
| pip install cudf-polars-cu12 | ||
| pip install dask-cudf-cu12 | ||
| ``` | ||
|
|
||
| ## Resources | ||
| ### conda | ||
|
|
||
| - [Try cudf.pandas now](https://nvda.ws/rapids-cudf): Explore `cudf.pandas` on a free GPU enabled instance on Google Colab! | ||
| - [Install](https://docs.rapids.ai/install): Instructions for installing cuDF and other [RAPIDS](https://rapids.ai) libraries. | ||
| - [cudf (Python) documentation](https://docs.rapids.ai/api/cudf/stable/) | ||
| - [libcudf (C++/CUDA) documentation](https://docs.rapids.ai/api/libcudf/stable/) | ||
| - [RAPIDS Community](https://rapids.ai/learn-more/#get-involved): Get help, contribute, and collaborate. | ||
| Each cudf library is available to be installed with the conda package manager by specifying the `-c rapidsai` channel. | ||
|
|
||
| See the [RAPIDS install page](https://docs.rapids.ai/install) for | ||
| the most up-to-date information and commands for installing cuDF | ||
| and other RAPIDS packages. | ||
| ```bash | ||
| conda install -c rapidsai libcudf | ||
| conda install -c rapidsai pylibcudf | ||
| conda install -c rapidsai cudf | ||
| conda install -c rapidsai cudf-polars | ||
| conda install -c rapidsai dask-cudf | ||
| ``` | ||
|
|
||
| ## Installation | ||
| A development version of each library is available as a nightly release by specifying the `-c rapidsai-nightly` channel instead. | ||
|
|
||
| ### CUDA/GPU requirements | ||
| ### source | ||
|
|
||
| * CUDA 12.0+ with a compatible NVIDIA driver | ||
| * Volta architecture or better (Compute Capability >=7.0) | ||
| To install cuDF from source, please follow [the contribution guide](CONTRIBUTING.md#setting-up-your-build-environment) detailing | ||
| how to setup the build environment. | ||
|
|
||
| ### Pip | ||
| ## Examples | ||
|
|
||
| cuDF can be installed via `pip` from the NVIDIA Python Package Index. | ||
| Be sure to select the appropriate cuDF package depending | ||
| on the major version of CUDA available in your environment: | ||
| The following examples showcase reading a parquet file, dropping missing rows with a null value, | ||
| and performing a groupby aggregation on the data. | ||
|
|
||
| ```bash | ||
| # CUDA 13 | ||
| pip install cudf-cu13 | ||
| ### cudf | ||
|
|
||
| # CUDA 12 | ||
| pip install cudf-cu12 | ||
| `import cudf` and the APIs are largely similar to pandas. | ||
|
|
||
| ```python | ||
| import cudf | ||
|
|
||
| df = cudf.read_parquet("data.parquet") | ||
| df.dropna().groupby(["A", "B"]).mean() | ||
| ``` | ||
|
|
||
| ### Conda | ||
| ### cudf.pandas | ||
|
|
||
| cuDF can be installed with conda (via [miniforge](https://github.com/conda-forge/miniforge)) from the `rapidsai` channel: | ||
| With a Python file containing pandas code: | ||
|
|
||
| ```python | ||
| import pandas as pd | ||
|
|
||
| df = cudf.read_parquet("data.parquet") | ||
| df.dropna().groupby(["A", "B"]).mean() | ||
| ``` | ||
|
|
||
| Use cudf.pandas by invoking `python` with `-m cudf.pandas` | ||
|
|
||
| ```bash | ||
| # CUDA 13 | ||
| conda install -c rapidsai -c conda-forge cudf=25.12 cuda-version=13.0 | ||
| $ python -m cudf.pandas script.py | ||
| ``` | ||
|
|
||
| # CUDA 12 | ||
| conda install -c rapidsai -c conda-forge cudf=25.12 cuda-version=12.9 | ||
| If running the pandas code in an interactive Jupyter environment, call `%load_ext cudf.pandas` before | ||
| importing pandas. | ||
|
|
||
| ```python | ||
| In [1]: %load_ext cudf.pandas | ||
|
|
||
| In [2]: import pandas as pd | ||
|
|
||
| In [3]: df = cudf.read_parquet("data.parquet") | ||
|
|
||
| In [4]: df.dropna().groupby(["A", "B"]).mean() | ||
| ``` | ||
|
|
||
| We also provide [nightly Conda packages](https://anaconda.org/rapidsai-nightly) built from the HEAD | ||
| of our latest development branch. | ||
| ### cudf-polars | ||
|
|
||
| Using Polars' [lazy API](https://docs.pola.rs/user-guide/lazy/), call `collect` with `engine="gpu"` to run | ||
| the operation on the GPU | ||
|
|
||
| ```python | ||
| import polars as pl | ||
|
|
||
| lf = pl.scan_parquet("data.parquet") | ||
| lf.drop_nulls().group_by(["A", "B"]).mean().collect(engine="gpu") | ||
| ``` | ||
|
|
||
| Note: cuDF is supported only on Linux, and with Python versions 3.10 and later. | ||
| ## Questions and Discussion | ||
|
|
||
| See the [RAPIDS installation guide](https://docs.rapids.ai/install) for more OS and version info. | ||
| For bug reports or feature requests, please [file an issue](https://github.com/rapidsai/cudf/issues/new/choose) on the GitHub issue tracker. | ||
|
|
||
| ## Build/Install from Source | ||
| See build [instructions](CONTRIBUTING.md#setting-up-your-build-environment). | ||
| For questions or discussion about cuDF and GPU data processing, feel free to post in the [RAPIDS Slack](https://rapids.ai/slack-invite) workspace. | ||
|
|
||
| ## Contributing | ||
|
|
||
| Please see our [guide for contributing to cuDF](CONTRIBUTING.md). | ||
| cuDF is open to contributions from the community! Please see our [guide for contributing to cuDF](CONTRIBUTING.md) for more information. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're mentioning development versions, we should also mention the nightly wheels https://anaconda.org/rapidsai-wheels-nightly/repo?type=pypi&label=main