Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions docs/cudf/source/cudf_polars/benchmarks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Benchmarks

<!-- TODO: add PDS-DS (TPC-DS variant) section, mirroring the PDS-H instructions below -->

## PDS-H (TPC-H variant)

The steps below reproduce the PDS-H benchmark results using the Polars GPU engine.

### Setup

Create and activate a fresh environment ([conda](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html), [mamba](https://mamba.readthedocs.io/en/latest/user_guide/mamba.html), [Python venv](https://docs.python.org/3/library/venv.html), or [uv](https://docs.astral.sh/uv/)), then install the dependencies:

```bash
CUDA_MAJOR=$(nvidia-smi | grep -oP 'CUDA Version: \K[0-9]+')
pip install tpchgen-cli
Comment thread
mroeschke marked this conversation as resolved.
Outdated
pip install --extra-index-url https://pypi.anaconda.org/rapidsai-wheels-nightly/simple "cudf-polars-cu${CUDA_MAJOR}>=0.0.0a0"
Comment thread
Matt711 marked this conversation as resolved.
Outdated
```

This installs:

- **`tpchgen-cli`**: a Rust-based TPC-H data generator used to produce the benchmark dataset as Parquet files.
- **`cudf-polars`**: the Polars GPU engine, along with its dependencies including `polars`. Because the GPU engine pins to a tested range of Polars versions, the nightly wheel will install the highest Polars version that the GPU engine currently supports, which may not be the latest Polars release.
Comment thread
Matt711 marked this conversation as resolved.
Outdated

### Generate data

The following generates SF1000 (scale factor 1000, roughly 1TB of data):

```bash
tpchgen-cli --output-dir="data/tables/scale-1000.0" --format=parquet -s 1000.0
Comment thread
Matt711 marked this conversation as resolved.
Outdated
```

### Run

Set these environment variables before running to match the configuration used for the published results:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes me nervous, since it'll inevitably go out of date. Maybe that's the right tradeoff, but could we at least link to those published benchmarks from here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed them from here. And added defaults for POLARS_MAX_THREADS=1 and OMP_NUM_THREADS=1 for the GPU benchmarks in the actual runner.

Also did the same for RAPIDSMPF_NUM_STREAMING_THREADS


```bash
export POLARS_MAX_THREADS=1
export OMP_NUM_THREADS=1
export LIBCUDF_NUM_HOST_WORKERS=4
export KVIKIO_NTHREADS=8
export RAPIDSMPF_num_streaming_threads=8
```

**Single GPU** (`--frontend spmd`, single-process streaming executor, equivalent to `collect(engine="gpu")`):

```bash
python -m cudf_polars.streaming.benchmarks.pdsh all \
--frontend spmd \
--path data/tables/scale-1000.0
```

**Multi GPU** (`--frontend ray`, Ray-managed distributed streaming executor):

If running inside a Docker container, increase `/dev/shm` by passing `--shm-size=16g` to
`docker run`. All multi-GPU frontends use UCX for intra-node communication, which relies on
POSIX shared memory (`/dev/shm`) for GPU-to-GPU transfers. Docker's default `/dev/shm` is
64MB, which is far too small and will cause failures on any non-trivial workload.

By default all visible GPUs are used. To select specific devices, set `CUDA_VISIBLE_DEVICES`.
To limit the number of GPUs, use `--num-gpus`:

```bash
# All visible GPUs
python -m cudf_polars.streaming.benchmarks.pdsh all \
--frontend ray \
--path data/tables/scale-1000.0

# Specific devices
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m cudf_polars.streaming.benchmarks.pdsh all \
--frontend ray \
--path data/tables/scale-1000.0

# Limit to N GPUs
python -m cudf_polars.streaming.benchmarks.pdsh all \
--frontend ray \
--num-gpus 4 \
--path data/tables/scale-1000.0
```

The `--path` value must match the `--output-dir` used during data generation.
Update both consistently when changing scale factors (e.g. `scale-100.0`).

### Tuning

The commands above use default settings, which gives a realistic baseline without manual tuning. The most impactful options to adjust are:

| Option | Description |
|--------|-------------|
| `--target-partition-size` | Target IO chunk size in bytes fed to the GPU. The most impactful lever; tune this first if query performance is below expectations. Default: auto. |
| `--broadcast-limit` | Maximum table size in bytes for broadcast joins instead of shuffle. Increasing this can significantly speed up join-heavy queries. Default: auto. |
| `--spill-device-limit` | GPU memory usage percentage before spilling to host. Lower this if hitting out-of-memory errors. Default: 80%. |
| `--pinned-memory` / `--pinned-max-pool-size` | Enable and size a pinned host memory pool for faster CPU to GPU transfers. |
4 changes: 3 additions & 1 deletion docs/cudf/source/cudf_polars/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@ On a single GPU, you can run TB-scale workloads with significant speedups compar
PDS-DS (SF3K)
```

For more information on the benchmarks being run, see the PDS-DS queries in the [cuDF GitHub repository](https://github.com/rapidsai/cudf/tree/release/26.06/python/cudf_polars/cudf_polars/streaming/benchmarks).
<!-- TODO: replace this link with {doc}`benchmarks` once the published results are reproducible using those instructions -->
For more information on the benchmarks being run, see the PDS queries in the [cuDF GitHub repository](https://github.com/rapidsai/cudf/tree/main/python/cudf_polars/cudf_polars/streaming/benchmarks).

## Learn More

Expand All @@ -109,6 +110,7 @@ options
profiling
other_engines
memory_errors
benchmarks
api
developer_docs
```
Expand Down
Loading