CHANGELOG.md

dask-cuda 24.10.00 (9 Oct 2024)

🚨 Breaking Changes

• Replace cuDF (de)serializer with cuDF spill-aware (de)serializer (#1369) @pentschev

📖 Documentation

• Fix typo in spilling documentation (#1384) @rjzamora
• Add notes on cudf spilling to docs (#1383) @rjzamora

🚀 New Features

• [Benchmark] Add parquet read benchmark (#1371) @rjzamora
• Replace cuDF (de)serializer with cuDF spill-aware (de)serializer (#1369) @pentschev

🛠️ Improvements

• Update update-version.sh to use packaging lib (#1387) @AyodeAwe
• Use CI workflow branch 'branch-24.10' again (#1386) @jameslamb
• Update to flake8 7.1.1. (#1385) @bdice
• enable Python 3.12 tests on PRs (#1382) @jameslamb
• Add support for Python 3.12 (#1380) @jameslamb
• Update rapidsai/pre-commit-hooks (#1379) @KyleFromNVIDIA
• Drop Python 3.9 support (#1377) @jameslamb
• Remove NumPy <2 pin (#1375) @seberg
• Update pre-commit hooks (#1373) @KyleFromNVIDIA
• Merge branch-24.08 into branch-24.10 (#1368) @jameslamb

dask-cuda 24.08.00 (7 Aug 2024)

🐛 Bug Fixes

• Fix partitioning in explicit-comms shuffle (#1356) @rjzamora
• Update cuDF's assert_eq import (#1353) @pentschev

🚀 New Features

• Add arguments to enable cuDF spilling and set statistics (#1362) @pentschev
• Allow disabling RMM in benchmarks (#1352) @pentschev

🛠️ Improvements

• consolidate cuda_suffixed=false blocks in dependencies.yaml, fix update-version.sh (#1367) @jameslamb
• split up CUDA-suffixed dependencies in dependencies.yaml (#1364) @jameslamb
• Use verify-alpha-spec hook (#1360) @KyleFromNVIDIA
• Use workflow branch 24.08 again (#1359) @KyleFromNVIDIA
• Build and test with CUDA 12.5.1 (#1357) @KyleFromNVIDIA
• Drop setup.py (#1354) @jakirkham
• remove .gitattributes (#1350) @jameslamb
• make conda recipe data-loading stricter (#1349) @jameslamb
• Adopt CI/packaging codeowners (#1347) @bdice
• Remove text builds of documentation (#1346) @vyasr
• use rapids-build-backend (#1343) @jameslamb

dask-cuda 24.06.00 (5 Jun 2024)

🐛 Bug Fixes

• Fix license name (#1337) @raydouglass
• Skip TCP-only DGX tests with UCX 1.16 (#1331) @pentschev
• Update explicit-comms for dask-expr support (#1323) @rjzamora
• Skip explicit-comms tests when dask-expr is active (#1322) @rjzamora
• Relax type-check in test_proxy.py (#1321) @rjzamora

📖 Documentation

• Fix broken links in docs (#1329) @pentschev

🛠️ Improvements

• remove 'tomli' dependency (#1338) @jameslamb
• Trap CI test errors with their original exit codes (#1330) @pentschev
• Prevent path conflict in builds (#1325) @AyodeAwe

dask-cuda 24.04.00 (10 Apr 2024)

🐛 Bug Fixes

• handle more RAPIDS version formats in update-version.sh (#1307) @jameslamb

🚀 New Features

• Allow using pandas 2 (#1308) @vyasr
• Support CUDA 12.2 (#1302) @jameslamb

🛠️ Improvements

• Use conda env create --yes instead of --force (#1326) @bdice
• Add upper bound to prevent usage of NumPy 2 (#1320) @bdice
• Generalize GHA selectors for pure Python testing (#1318) @jakirkham
• Requre NumPy 1.23+ (#1316) @jakirkham
• Add support for Python 3.11 (#1315) @jameslamb
• target branch-24.04 for GitHub Actions workflows (#1314) @jameslamb
• Filter dd deprecation (#1312) @rjzamora
• Update ops-bot.yaml (#1310) @AyodeAwe

dask-cuda 24.02.00 (12 Feb 2024)

🚨 Breaking Changes

• Publish nightly wheels to NVIDIA index instead of PyPI (#1294) @pentschev

🐛 Bug Fixes

• Fix get_device_memory_ids (#1305) @wence-
• Prevent double UCX initialization in test_dgx (#1301) @pentschev
• Update to Dask's shuffle_method kwarg (#1300) @pentschev
• Add timeout to test_dask_use_explicit_comms (#1298) @pentschev
• Publish nightly wheels to NVIDIA index instead of PyPI (#1294) @pentschev
• Make versions PEP440 compliant (#1279) @vyasr
• Generate pyproject.toml with dfg (#1276) @vyasr
• Fix rapids dask dependency version (#1275) @vyasr

🛠️ Improvements

• Remove usages of rapids-env-update (#1304) @KyleFromNVIDIA
• refactor CUDA versions in dependencies.yaml (#1303) @jameslamb
• Start generating conda test environments (#1291) @charlesbluca
• Branch 24.02 merge branch 23.12 (#1286) @vyasr

dask-cuda 23.12.00 (6 Dec 2023)

🐛 Bug Fixes

• Update actions/labeler to v4 (#1292) @raydouglass
• Increase Nanny close timeout for test_spilling_local_cuda_cluster (#1289) @pentschev
• Fix path (#1277) @vyasr
• Add missing alpha spec (#1273) @vyasr
• Set minimum click to 8.1 (#1272) @jacobtomlinson
• Reenable tests that were segfaulting (#1266) @pentschev
• Increase close timeout of Nanny in LocalCUDACluster (#1260) @pentschev
• Small reorganization and fixes for test_spill (#1255) @pentschev
• Update plugins to inherit from WorkerPlugin (#1230) @jrbourbeau

🚀 New Features

• Add support for UCXX (#1268) @pentschev

🛠️ Improvements

• Fix license (#1285) @vyasr
• Build concurrency for nightly and merge triggers (#1282) @bdice
• Use new rapids-dask-dependency metapackage for managing dask versions (#1270) @galipremsagar
• Remove ucp.reset() requirement from test_dgx (#1269) @pentschev
• Generate proper, consistent nightly versions for pip and conda packages (#1267) @galipremsagar
• Unpin dask and distributed for 23.12 development (#1264) @galipremsagar
• Move some dask_cuda.utils pieces to their own modules (#1263) @pentschev
• Update shared-action-workflows references (#1261) @AyodeAwe
• Use branch-23.12 workflows. (#1259) @bdice
• dask-cuda: Build CUDA 12.0 ARM conda packages. (#1238) @bdice

dask-cuda 23.10.00 (11 Oct 2023)

🐛 Bug Fixes

• Monkeypatch protocol.loads ala dask/distributed#8216 (#1247) @wence-
• Explicit-comms: preserve partition IDs (#1240) @madsbk
• Increase test timeouts further to reduce CI failures (#1234) @pentschev
• Use conda mambabuild not mamba mambabuild (#1231) @bdice
• Increate timeouts of tests that frequently timeout in CI (#1228) @pentschev
• Adapt to non-string task keys in distributed (#1225) @wence-
• Update test_worker_timeout (#1223) @pentschev
• Avoid importing loads_function from distributed (#1220) @rjzamora

🚀 New Features

• Enable maximum pool size for RMM async allocator (#1221) @pentschev

🛠️ Improvements

• Pin dask and distributed for 23.10 release (#1251) @galipremsagar
• Update test_spill.py to avoid FutureWarnings (#1243) @pentschev
• Remove obsolete pytest filterwarnings (#1241) @pentschev
• Update image names (#1233) @AyodeAwe
• Use copy-pr-bot (#1227) @ajschmidt8
• Unpin dask and distributed for 23.10 development (#1222) @galipremsagar

dask-cuda 23.08.00 (9 Aug 2023)

🐛 Bug Fixes

• Ensure plugin config can be passed from worker to client (#1212) @wence-
• Adjust to new get_default_shuffle_method name (#1200) @pentschev
• Increase minimum timeout to wait for workers in CI (#1192) @pentschev

📖 Documentation

• Remove RTD configuration and references to RTD page (#1211) @charlesbluca
• Clarify memory_limit docs (#1207) @pentschev

🚀 New Features

• Remove versioneer (#1204) @pentschev
• Remove code for Distributed<2023.5.1 compatibility (#1191) @pentschev
• Specify disk spill compression based on Dask config (#1190) @pentschev

🛠️ Improvements

• Pin dask and distributed for 23.08 release (#1214) @galipremsagar
• Revert CUDA 12.0 CI workflows to branch-23.08. (#1210) @bdice
• Use minimal Numba dependencies for CUDA 12 (#1209) @jakirkham
• Aggregate reads & writes in disk_io (#1205) @jakirkham
• CUDA 12 Support (#1201) @quasiben
• Remove explicit UCX config from tests (#1199) @pentschev
• use rapids-upload-docs script (#1194) @AyodeAwe
• Unpin dask and distributed for development (#1189) @galipremsagar
• Remove documentation build scripts for Jenkins (#1187) @ajschmidt8
• Use KvikIO in Dask-CUDA (#925) @jakirkham

dask-cuda 23.06.00 (7 Jun 2023)

🚨 Breaking Changes

• Update minimum Python version to Python 3.9 (#1164) @shwina

🐛 Bug Fixes

• Increase pytest CI timeout (#1196) @pentschev
• Increase minimum timeout to wait for workers in CI (#1193) @pentschev
• Disable np.bool deprecation warning (#1182) @pentschev
• Always upload on branch/nightly builds (#1177) @raydouglass
• Workaround for DeviceHostFile tests with CuPy>=12.0.0 (#1175) @pentschev
• Temporarily relax Python constraint (#1166) @vyasr

📖 Documentation

• [doc] Add document about main guard. (#1157) @trivialfis

🚀 New Features

• Require Numba 0.57.0+ (#1185) @jakirkham
• Revert "Temporarily relax Python constraint" (#1171) @vyasr
• Update to zict 3.0 (#1160) @pentschev

🛠️ Improvements

• Add __main__ entrypoint to dask-cuda-worker CLI (#1181) @hmacdope
• run docs nightly too (#1176) @AyodeAwe
• Fix GHAs Workflows (#1172) @ajschmidt8
• Remove matrix_filter from workflows (#1168) @charlesbluca
• Revert to branch-23.06 for shared-action-workflows (#1167) @shwina
• Update minimum Python version to Python 3.9 (#1164) @shwina
• Remove usage of rapids-get-rapids-version-from-git (#1163) @jjacobelli
• Use ARC V2 self-hosted runners for GPU jobs (#1159) @jjacobelli

dask-cuda 23.04.00 (6 Apr 2023)

🚨 Breaking Changes

• Pin dask and distributed for release (#1153) @galipremsagar
• Update minimum pandas and numpy pinnings (#1139) @galipremsagar

🐛 Bug Fixes

• Rectify dask-core pinning in pip requirements (#1155) @galipremsagar
• Monkey patching all locations of get_default_shuffle_algorithm (#1142) @madsbk
• Update usage of get_worker() in tests (#1141) @pentschev
• Update rmm_cupy_allocator usage (#1138) @jakirkham
• Serialize of ProxyObject to pickle fixed attributes (#1137) @madsbk
• Explicit-comms: update monkey patching of Dask (#1135) @madsbk
• Fix for bytes/str discrepancy after PyNVML update (#1118) @pentschev

🚀 New Features

• Allow specifying dashboard address in benchmarks (#1147) @pentschev
• Add argument to enable RMM alloaction tracking in benchmarks (#1145) @pentschev
• Reinstate --death-timeout CLI option (#1140) @charlesbluca
• Extend RMM async allocation support (#1116) @pentschev
• Allow using stream-ordered and managed RMM allocators in benchmarks (#1012) @pentschev

🛠️ Improvements

• Pin dask and distributed for release (#1153) @galipremsagar
• Update minimum pandas and numpy pinnings (#1139) @galipremsagar
• Drop Python 3.7 handling for pickle protocol 4 (#1132) @jakirkham
• Adapt to rapidsai/rmm#1221 which moves allocator callbacks (#1129) @wence-
• Merge branch-23.02 into branch-23.04 (#1128) @ajschmidt8
• Template Conda recipe's about metadata (#1121) @jakirkham
• Fix GHA build workflow (#1120) @AjayThorve
• Reduce error handling verbosity in CI tests scripts (#1113) @AjayThorve
• Update shared workflow branches (#1112) @ajschmidt8
• Remove gpuCI scripts. (#1111) @bdice
• Unpin dask and distributed for development (#1110) @galipremsagar
• Move date to build string in conda recipe (#1103) @ajschmidt8

dask-cuda 23.02.00 (9 Feb 2023)

🚨 Breaking Changes

• Pin dask and distributed for release (#1106) @galipremsagar

🐛 Bug Fixes

• pre-commit: Update isort version to 5.12.0 (#1098) @wence-
• explicit-comms: don't mix - and _ in config (#1096) @madsbk
• Update cudf.Buffer pointer access method (#1094) @pentschev
• Update tests for Python 3.10 (#1086) @pentschev
• Use pkgutil.iter_modules to get un-imported module for test_pre_import (#1085) @charlesbluca
• Make proxy tests with LocalCUDACluster asynchronous (#1084) @pentschev
• Ensure consistent results from safe_sizeof() in test (#1071) @madsbk
• Pass missing argument to groupby benchmark compute (#1069) @mattf
• Reorder channel priority. (#1067) @bdice
• Fix owner check when the owner is a cupy array (#1061) @wence-

🛠️ Improvements

• Pin dask and distributed for release (#1106) @galipremsagar
• Update shared workflow branches (#1105) @ajschmidt8
• Proxify: make duplicate check optional (#1101) @madsbk
• Fix whitespace & add URLs in pyproject.toml (#1092) @jakirkham
• pre-commit: spell, whitespace, and mypy check (#1091) @madsbk
• shuffle: use cuDF's partition_by_hash() when available (#1090) @madsbk
• add initial docs build (#1089) @AjayThorve
• Remove --get-cluster-configuration option, check for scheduler in dask cuda config (#1088) @charlesbluca
• Add timeout to pytest command (#1082) @ajschmidt8
• shuffle-benchmark: add --partition-distribution (#1081) @madsbk
• Ensure tests run for Python 3.10 (#1080) @ajschmidt8
• Use TrackingResourceAdaptor to get better debug info (#1079) @madsbk
• Improve shuffle-benchmark (#1074) @madsbk
• Update builds for CUDA 11.8 and Python 310 (#1072) @ajschmidt8
• Shuffle by partition to reduce memory usage significantly (#1068) @madsbk
• Enable copy_prs. (#1063) @bdice
• Add GitHub Actions Workflows (#1062) @bdice
• Unpin dask and distributed for development (#1060) @galipremsagar
• Switch to the new dask CLI (#981) @jacobtomlinson

dask-cuda 22.12.00 (8 Dec 2022)

🚨 Breaking Changes

• Make local_directory a required argument for spilling impls (#1023) @wence-

🐛 Bug Fixes

• Fix parse_memory_limit function call (#1055) @galipremsagar
• Work around Jupyter errors in CI (#1041) @pentschev
• Fix version constraint (#1036) @wence-
• Support the new Buffer in cudf (#1033) @madsbk
• Install Dask nightly last in CI (#1029) @pentschev
• Fix recorded time in merge benchmark (#1028) @wence-
• Switch pre-import not found test to sync definition (#1026) @pentschev
• Make local_directory a required argument for spilling impls (#1023) @wence-
• Fixes for handling MIG devices (#950) @pentschev

📖 Documentation

• Merge 22.10 into 22.12 (#1016) @pentschev
• Merge 22.08 into 22.10 (#1010) @pentschev

🚀 New Features

• Allow specifying fractions as RMM pool initial/maximum size (#1021) @pentschev
• Add feature to get cluster configuration (#1006) @quasiben
• Add benchmark option to use dask-noop (#994) @wence-

🛠️ Improvements

• Ensure linting checks for whole repo in CI (#1053) @pentschev
• Pin dask and distributed for release (#1046) @galipremsagar
• Remove pytest-asyncio dependency (#1045) @pentschev
• Migrate as much as possible to pyproject.toml (#1035) @jakirkham
• Re-implement shuffle using staging (#1030) @madsbk
• Explicit-comms-shuffle: fine control of task scheduling (#1025) @madsbk
• Remove stale labeler (#1024) @raydouglass
• Unpin dask and distributed for development (#1005) @galipremsagar
• Support cuDF's built-in spilling (#984) @madsbk

dask-cuda 22.10.00 (12 Oct 2022)

🐛 Bug Fixes

• Revert "Update rearrange_by_column patch for explicit comms" (#1001) @rjzamora
• Address CI failures caused by upstream distributed and cupy changes (#993) @rjzamora
• DeviceSerialized.reduce_ex: convert frame to numpy arrays (#977) @madsbk

📖 Documentation

• Remove line-break that's breaking link (#982) @ntabris
• Dask-cuda best practices (#976) @quasiben

🚀 New Features

• Add Groupby benchmark (#979) @rjzamora

🛠️ Improvements

• Pin dask and distributed for release (#1003) @galipremsagar
• Update rearrange_by_column patch for explicit comms (#992) @rjzamora
• benchmarks: Add option to suppress output of point to point data (#985) @wence-
• Unpin dask and distributed for development (#971) @galipremsagar

dask-cuda 22.08.00 (17 Aug 2022)

🚨 Breaking Changes

• Fix useless property (#944) @wence-

🐛 Bug Fixes

• Fix distributed error related to loop_in_thread (#963) @galipremsagar
• Add __rmatmul__ to ProxyObject (#960) @jakirkham
• Always use versioneer command classes in setup.py (#948) @wence-
• Do not dispatch removed cudf.Frame._index object (#947) @pentschev
• Fix useless property (#944) @wence-
• LocalCUDACluster's memory limit: None means no limit (#943) @madsbk
• ProxyManager: support memory_limit=None (#941) @madsbk
• Remove deprecated loop kwarg to Nanny in CUDAWorker (#934) @pentschev
• Import cleanup fixture in test_dask_cuda_worker.py (#924) @pentschev

📖 Documentation

• Switch docs to use common js & css code (#967) @galipremsagar
• Switch language from None to "en" in docs build (#939) @galipremsagar

🚀 New Features

• Add communications bandwidth to benchmarks (#938) @pentschev

🛠️ Improvements

• Pin dask & distributed for release (#965) @galipremsagar
• Test memory_limit=None for CUDAWorker (#946) @wence-
• benchmarks: Record total number of workers in dataframe (#945) @wence-
• Benchmark refactoring: tidy data and multi-node capability via --scheduler-file (#940) @wence-
• Add util functions to simplify printing benchmarks results (#937) @pentschev
• Add --multiprocessing-method option to benchmarks (#933) @wence-
• Remove click pinning (#932) @charlesbluca
• Remove compiler variables (#929) @ajschmidt8
• Unpin dask & distributed for development (#927) @galipremsagar

dask-cuda 22.06.00 (7 Jun 2022)

🚨 Breaking Changes

• Upgrade numba pinning to be in-line with rest of rapids (#912) @galipremsagar

🐛 Bug Fixes

• Reduce test_cudf_cluster_device_spill test and speed it up (#918) @pentschev
• Update ImportError tests with --pre-import (#914) @pentschev
• Add xfail mark to test_pre_import_not_found (#908) @pentschev
• Increase spill tests timeout to 30 seconds (#901) @pentschev
• Fix errors related with distributed.worker.memory.terminate (#900) @pentschev
• Skip tests on import error for some optional packages (#899) @pentschev
• Update auto host_memory computation when threads per worker > 1 (#896) @ayushdg
• Update black to 22.3.0 (#889) @charlesbluca
• Remove legacy check_python_3 (#886) @pentschev

📖 Documentation

• Add documentation for RAPIDS_NO_INITIALIZE (#898) @charlesbluca
• Use upstream warning functions for CUDA initialization (#894) @charlesbluca

🛠️ Improvements

• Pin dask and distributed for release (#922) @galipremsagar
• Pin dask & distributed for release (#916) @galipremsagar
• Upgrade numba pinning to be in-line with rest of rapids (#912) @galipremsagar
• Removing test of cudf.merge_sorted() (#905) @madsbk
• Disable include-ignored coverage warnings (#903) @pentschev
• Fix ci/local script (#902) @Ethyling
• Use conda to build python packages during GPU tests (#897) @Ethyling
• Pull requirements.txt into Conda recipe (#893) @jakirkham
• Unpin dask & distributed for development (#892) @galipremsagar
• Build packages using mambabuild (#846) @Ethyling

dask-cuda 22.04.00 (6 Apr 2022)

🚨 Breaking Changes

• Introduce incompatible-types and enables spilling of CuPy arrays (#856) @madsbk

🐛 Bug Fixes

• Resolve build issues / consistency with conda-forge packages (#883) @charlesbluca
• Increase test_worker_force_spill_to_disk timeout (#857) @pentschev

📖 Documentation

• Remove description from non-existing --nprocs CLI argument (#852) @pentschev

🚀 New Features

• Add --pre-import/pre_import argument (#854) @pentschev
• Remove support for UCX < 1.11.1 (#830) @pentschev

🛠️ Improvements

• Raise ImportError when platform is not Linux (#885) @pentschev
• Temporarily disable new ops-bot functionality (#880) @ajschmidt8
• Pin dask & distributed (#878) @galipremsagar
• Upgrade min dask & distributed versions (#872) @galipremsagar
• Add .github/ops-bot.yaml config file (#871) @ajschmidt8
• Make Dask CUDA work with the new WorkerMemoryManager abstraction (#870) @shwina
• Implement ProxifyHostFile.evict() (#862) @madsbk
• Introduce incompatible-types and enables spilling of CuPy arrays (#856) @madsbk
• Spill to disk clean up (#853) @madsbk
• ProxyObject to support matrix multiplication (#849) @madsbk
• Unpin max dask and distributed (#847) @galipremsagar
• test_gds: skip if GDS is not available (#845) @madsbk
• ProxyObject implement array_function (#843) @madsbk
• Add option to track RMM allocations (#842) @shwina

dask-cuda 22.02.00 (2 Feb 2022)

🐛 Bug Fixes

• Ignore DeprecationWarning from distutils.Version classes (#823) @pentschev
• Handle explicitly disabled UCX transports (#820) @pentschev
• Fix regex pattern to match to in test_on_demand_debug_info (#819) @pentschev
• Fix skipping GDS test if cucim is not installed (#813) @pentschev
• Unpin Dask and Distributed versions (#810) @pentschev
• Update to UCX-Py 0.24 (#805) @pentschev

📖 Documentation

• Fix Dask-CUDA version to 22.02 (#835) @jakirkham
• Merge branch-21.12 into branch-22.02 (#829) @pentschev
• Clarify LocalCUDACluster's n_workers docstrings (#812) @pentschev

🚀 New Features

• Pin dask & distributed versions (#832) @galipremsagar
• Expose rmm-maximum_pool_size argument (#827) @VibhuJawa
• Simplify UCX configs, permitting UCX_TLS=all (#792) @pentschev

🛠️ Improvements

• Add avg and std calculation for time and throughput (#828) @quasiben
• sizeof test: increase tolerance (#825) @madsbk
• Query UCX-Py from gpuCI versioning service (#818) @pentschev
• Standardize Distributed config separator in get_ucx_config (#806) @pentschev
• Fixed ProxyObject.__del__ to use the new Disk IO API from #791 (#802) @madsbk
• GPUDirect Storage (GDS) support for spilling (#793) @madsbk
• Disk IO interface (#791) @madsbk

dask-cuda 21.12.00 (9 Dec 2021)

🐛 Bug Fixes

• Remove automatic doc labeler (#807) @pentschev
• Add create_cuda_context UCX config from Distributed (#801) @pentschev
• Ignore deprecation warnings from pkg_resources (#784) @pentschev
• Fix parsing of device by UUID (#780) @pentschev
• Avoid creating CUDA context in LocalCUDACluster parent process (#765) @pentschev
• Remove gen_cluster spill tests (#758) @pentschev
• Update memory_pause_fraction in test_spill (#757) @pentschev

📖 Documentation

• Add troubleshooting page with PCI Bus ID issue description (#777) @pentschev

🚀 New Features

• Handle UCX-Py FutureWarning on UCX < 1.11.1 deprecation (#799) @pentschev
• Pin max dask & distributed versions (#794) @galipremsagar
• Update to UCX-Py 0.23 (#752) @pentschev

🛠️ Improvements

• Fix spill-to-disk triggered by Dask explicitly (#800) @madsbk
• Fix Changelog Merge Conflicts for branch-21.12 (#797) @ajschmidt8
• Use unittest.mock.patch for all os.environ tests (#787) @pentschev
• Logging when RMM allocation fails (#782) @madsbk
• Tally IDs instead of device buffers directly (#779) @madsbk
• Avoid proxy object aliasing (#775) @madsbk
• Test of sizeof proxy object (#774) @madsbk
• gc.collect when spilling on demand (#771) @madsbk
• Reenable explicit comms tests (#770) @madsbk
• Simplify JIT-unspill and writing docs (#768) @madsbk
• Increase CUDAWorker close timeout (#764) @pentschev
• Ignore known but expected test warnings (#759) @pentschev
• Spilling on demand (#756) @madsbk
• Revert "Temporarily skipping some tests because of a bug in Dask (#753)" (#754" (#754)) @madsbk
• Temporarily skipping some tests because of a bug in Dask (#753) @madsbk
• Removing the FrameProxyObject workaround (#751) @madsbk
• Use cuDF Frame instead of Table (#748) @madsbk
• Remove proxy object locks (#747) @madsbk
• Unpin dask & distributed in CI (#742) @galipremsagar
• Update SSHCluster usage in benchmarks with new CUDAWorker (#326) @pentschev

dask-cuda 21.10.00 (7 Oct 2021)

🐛 Bug Fixes

• Drop test setting UCX global options via Dask config (#738) @pentschev
• Prevent CUDA context errors when testing on single-GPU (#737) @pentschev
• Handle ucp import error during initialize() (#729) @pentschev
• Check if CUDA context was created in distributed.comm.ucx (#722) @pentschev
• Fix registering correct dispatches for cudf.Index (#718) @galipremsagar
• Register percentile_lookup for FrameProxyObject (#716) @galipremsagar
• Leave interface unset when ucx_net_devices unset in LocalCUDACluster (#711) @pentschev
• Update to UCX-Py 0.22 (#710) @pentschev
• Missing fixes to Distributed config namespace refactoring (#703) @pentschev
• Reset UCX-Py after rdmacm tests run (#702) @pentschev
• Skip DGX InfiniBand tests when "rc" transport is unavailable (#701) @pentschev
• Update UCX config namespace (#695) @pentschev
• Bump isort hook version (#682) @charlesbluca

📖 Documentation

• Update more docs for UCX 1.11+ (#720) @pentschev
• Forward-merge branch-21.08 to branch-21.10 (#707) @jakirkham

🚀 New Features

• Warn if CUDA context is created on incorrect device with LocalCUDACluster (#719) @pentschev
• Add --benchmark-json option to all benchmarks (#700) @charlesbluca
• Remove Distributed tests from CI (#699) @pentschev
• Add device memory limit argument to benchmarks (#683) @charlesbluca
• Support for LocalCUDACluster with MIG (#674) @akaanirban

🛠️ Improvements

• Pin max dask and distributed versions to 2021.09.1 (#735) @galipremsagar
• Implements a ProxyManagerDummy for convenience (#733) @madsbk
• Add __array_ufunc__ support for ProxyObject (#731) @galipremsagar
• Use has_cuda_context from Distributed (#723) @pentschev
• Fix deadlock and simplify proxy tracking (#712) @madsbk
• JIT-unspill: support spilling to/from disk (#708) @madsbk
• Tests: replacing the obsolete cudf.testing._utils.assert_eq calls (#706) @madsbk
• JIT-unspill: warn when spill to disk triggers (#705) @madsbk
• Remove max version pin for dask & distributed on development branch (#693) @galipremsagar
• ENH Replace gpuci_conda_retry with gpuci_mamba_retry (#675) @dillon-cullinan

dask-cuda 21.08.00 (4 Aug 2021)

🐛 Bug Fixes

• Use aliases to check for installed UCX version (#692) @pentschev
• Don't install Dask main branch in CI for 21.08 release (#687) @pentschev
• Skip test_get_ucx_net_devices_raises on UCX >= 1.11.0 (#685) @pentschev
• Fix NVML index usage in CUDAWorker/LocalCUDACluster (#671) @pentschev
• Add --protocol flag to dask-cuda-worker (#670) @jacobtomlinson
• Fix assert_eq related imports (#663) @galipremsagar
• Small tweaks to make compatible with dask-mpi (#656) @jacobtomlinson
• Remove Dask version pin (#647) @pentschev
• Fix CUDA_VISIBLE_DEVICES tests (#638) @pentschev
• Add make_meta_dispatch handling (#637) @galipremsagar
• Update UCX-Py version in CI to 0.21.* (#636) @pentschev

📖 Documentation

• Deprecation warning for ucx_net_devices='auto' on UCX 1.11+ (#681) @pentschev
• Update documentation on InfiniBand with UCX >= 1.11 (#669) @pentschev
• Merge branch-21.06 (#622) @pentschev

🚀 New Features

• Treat Deprecation/Future warnings as errors (#672) @pentschev
• Update parse_bytes imports to resolve deprecation warnings (#662) @pentschev

🛠️ Improvements

• Pin max dask & distributed versions (#686) @galipremsagar
• Fix DGX tests warnings on RMM pool size and file not closed (#673) @pentschev
• Remove dot calling style for pytest (#661) @quasiben
• get_device_memory_objects(): dispatch on cudf.core.frame.Frame (#658) @madsbk
• Fix 21.08 forward-merge conflicts (#655) @ajschmidt8
• Fix conflicts in 643 (#644) @ajschmidt8

dask-cuda 21.06.00 (9 Jun 2021)

🐛 Bug Fixes

• Handle importing relocated dispatch functions (#623) @jakirkham
• Fix DGX tests for UCX 1.9 (#619) @pentschev
• Add PROJECTS var (#614) @ajschmidt8

📖 Documentation

• Bump docs copyright year (#616) @charlesbluca
• Update RTD site to redirect to RAPIDS docs (#615) @charlesbluca
• Document DASK_JIT_UNSPILL (#604) @madsbk

🚀 New Features

• Disable reuse endpoints with UCX >= 1.11 (#620) @pentschev

🛠️ Improvements

• Adding profiling to dask shuffle (#625) @arunraman
• Update CHANGELOG.md links for calver (#618) @ajschmidt8
• Fixing Dataframe merge benchmark (#617) @madsbk
• Fix DGX tests for UCX 1.10+ (#613) @pentschev
• Update docs build script (#612) @ajschmidt8

dask-cuda 0.19.0 (21 Apr 2021)

🐛 Bug Fixes

• Pin Dask and Distributed <=2021.04.0 (#585) @pentschev
• Unblock CI by xfailing test_dataframe_merge_empty_partitions (#581) @pentschev
• Install Dask + Distributed from main (#546) @jakirkham
• Replace compute() calls on CuPy benchmarks by persist() (#537) @pentschev

📖 Documentation

• Add standalone examples of UCX usage (#551) @charlesbluca
• Improve UCX documentation and examples (#545) @charlesbluca
• Auto-merge branch-0.18 to branch-0.19 (#538) @GPUtester

🚀 New Features

• Add option to enable RMM logging (#542) @charlesbluca
• Add capability to log spilling (#442) @pentschev

🛠️ Improvements

• Fix UCX examples for InfiniBand (#556) @charlesbluca
• Fix list to tuple conversion (#555) @madsbk
• Add column masking operation for CuPy benchmarking (#553) @jakirkham
• Update Changelog Link (#550) @ajschmidt8
• cuDF-style operations & NVTX annotations for local CuPy benchmark (#548) @charlesbluca
• Prepare Changelog for Automation (#543) @ajschmidt8
• Add --enable-rdmacm flag to benchmarks utils (#539) @pentschev
• ProxifyHostFile: tracking of external objects (#527) @madsbk
• Test broadcast merge in local_cudf_merge benchmark (#507) @rjzamora

dask-cuda 0.18.0 (24 Feb 2021)

Breaking Changes 🚨

• Explicit-comms house cleaning (#515) @madsbk

Bug Fixes 🐛

• Fix device synchronization in local_cupy benchmark (#518) @pentschev
• Proxify register lazy (#492) @madsbk
• Work on deadlock issue 431 (#490) @madsbk
• Fix usage of --dashboard-address in dask-cuda-worker (#487) @pentschev
• Fail if scheduler starts with '-' in dask-cuda-worker (#485) @pentschev

Documentation 📖

• Add device synchonization for local CuPy benchmarks with Dask profiling (#533) @charlesbluca

New Features 🚀

• Shuffle benchmark (#496) @madsbk

Improvements 🛠️

• Update stale GHA with exemptions & new labels (#531) @mike-wendt
• Add GHA to mark issues/prs as stale/rotten (#528) @Ethyling
• Add operations/arguments to local CuPy array benchmark (#524) @charlesbluca
• Explicit-comms house cleaning (#515) @madsbk
• Fixing fixed-attribute-proxy-object-test (#511) @madsbk
• Prepare Changelog for Automation (#509) @ajschmidt8
• remove conditional check to start conda uploads (#504) @jolorunyomi
• ProxyObject: ignore initial fixed attribute errors (#503) @madsbk
• JIT-unspill: fix potential deadlock (#501) @madsbk
• Hostfile: register the removal of an existing key (#500) @madsbk
• proxy_object: cleanup type dispatching (#497) @madsbk
• Redesign and implementation of dataframe shuffle (#494) @madsbk
• Add --threads-per-worker option to benchmarks (#489) @pentschev
• Extend CuPy benchmark with more operations (#488) @pentschev
• Auto-label PRs based on their content (#480) @jolorunyomi
• CI: cleanup style check (#477) @madsbk
• Individual CUDA object spilling (#451) @madsbk
• FIX Move codecov upload to gpu build script (#450) @dillon-cullinan
• Add support for connecting a CUDAWorker to a cluster object (#428) @jacobtomlinson

0.17.0

• Fix benchmark output when scheduler address is specified (#414) @quasiben
• Fix typo in benchmark utils (#416) @quasiben
• More RMM options in benchmarks (#419) @quasiben
• Add utility function to establish all-to-all connectivity upon request (#420) @quasiben
• Filter rmm_pool_size warnings in benchmarks (#422) @pentschev
• Add functionality to plot cuDF benchmarks (#423) @quasiben
• Decrease data size to shorten spilling tests time (#422) @pentschev
• Temporarily xfail explicit-comms tests (#432) @pentschev
• Add codecov.yml and ignore uncovered files (#433) @pentschev
• Do not skip DGX/TCP tests when ucp is not installed (#436) @pentschev
• Support UUID in CUDA_VISIBLE_DEVICES (#437) @pentschev
• Unify device_memory_limit parsing and set default to 0.8 (#439) @pentschev
• Update and clean gpuCI scripts (#440) @msadang
• Add notes on controlling number of workers to docs (#441) @quasiben
• Add CPU support to CuPy transpose sum benchmark (#444) @pentschev
• Update builddocs dependency requirements (#447) @quasiben
• Fix versioneer (#448) @jakirkham
• Cleanup conda recipe (#449) @jakirkham
• Fix pip install issues with new resolver (#454) @jakirkham
• Make threads per worker consistent (#456) @pentschev
• Support for ProxyObject binary operations (#458) @madsbk
• Support for ProxyObject pickling (#459) @madsbk
• Clarify RMM pool is a per-worker attribute on docs (#462) @pentschev
• Fix typo on specializations docs (#463) @vfdev-5

0.16.0

• Parse pool size only when set (#396) @quasiben
• Improve CUDAWorker scheduler-address parsing and init (#397) @necaris
• Add benchmark for da.map_overlap (#399) @jakirkham
• Explicit-comms: dataframe shuffle (#401) @madsbk
• Use new NVTX module (#406) @pentschev
• Run Dask's NVML tests (#408) @quasiben
• Skip tests that require cuDF/UCX-Py, when not installed (#411) @pentschev

0.15.0

• Fix-up versioneer (#305) @jakirkham
• Require Distributed 2.15.0+ (#306) @jakirkham
• Rely on Dask's ability to serialize collections (#307) @jakirkham
• Ensure CI installs GPU build of UCX (#308) @pentschev
• Skip 2nd serialization pass of DeviceSerialized (#309) @jakirkham
• Fix tests related to latest RMM changes (#310) @pentschev
• Fix dask-cuda-worker's interface argument (#314) @pentschev
• Check only for memory type during test_get_device_total_memory (#315) @pentschev
• Fix and improve DGX tests (#316) @pentschev
• Install dependencies via meta package (#317) @raydouglass
• Fix errors when TLS files are not specified (#320) @pentschev
• Refactor dask-cuda-worker into CUDAWorker class (#324) @jacobtomlinson
• Add missing init.py to dask_cuda/cli (#327) @pentschev
• Add Dask distributed GPU tests to CI (#329) @quasiben
• Fix rmm_pool_size argument name in docstrings (#329) @quasiben
• Add CPU support to benchmarks (#338) @quasiben
• Fix isort configuration (#339) @madsbk
• Explicit-comms: cleanup and bug fix (#340) @madsbk
• Add support for RMM managed memory (#343) @pentschev
• Update docker image in local build script (#345) @sean-frye
• Support pickle protocol 5 based spilling (#349) @jakirkham
• Use get_n_gpus for RMM test with dask-cuda-worker (#356) @pentschev
• Update RMM tests based on deprecated CNMeM (#359) @jakirkham
• Fix a black error in explicit comms (#360) @jakirkham
• Fix an isort error (#360) @jakirkham
• Set RMM_NO_INITIALIZE environment variable (#363) @quasiben
• Fix bash lines in docs (#369) @quasiben
• Replace RMM_NO_INITIALIZE with RAPIDS_NO_INITIALIZE (#371) @jakirkham
• Fixes for docs and RTD updates (#373) @quasiben
• Confirm DGX tests are running baremetal (#376) @pentschev
• Set RAPIDS_NO_INITIALIZE at the top of CUDAWorker/LocalCUDACluster (#379) @pentschev
• Change pytest's basetemp in CI build script (#380) @pentschev
• Pin Numba version to exclude 0.51.0 (#385) @quasiben

0.14.0

• Publish branch-0.14 to conda (#262) @trxcllnt
• Fix behavior for memory_limit=0 (#269) @pentschev
• Raise serialization errors when spilling (#272) @jakirkham
• Fix dask-cuda-worker memory_limit (#279) @pentschev
• Add NVTX annotations for spilling (#282) @pentschev
• Skip existing on conda uploads (#284) @raydouglass
• Local gpuCI build script (#285) @efajardo-nv
• Remove deprecated DGX class (#286) @pentschev
• Add RDMACM support (#287) @pentschev
• Read the Docs Setup (#290) @quasiben
• Raise ValueError when ucx_net_devices="auto" and IB is disabled (#291) @pentschev
• Multi-node benchmarks (#293) @pentschev
• Add docs for UCX (#294) @pentschev
• Add --runs argument to CuPy benchmark (#295) @pentschev
• Fixing LocalCUDACluster example. Adding README links to docs (#297) @randerzander
• Add nfinal argument to shuffle_group, required in Dask >= 2.17 (#299) @pentschev
• Initialize parent process' UCX configuration (#301) @pentschev
• Add Read the Docs link (#302) @jakirkham

0.13.0

• Use RMM's DeviceBuffer directly (#235) @jakirkham
• Add RMM pool support from dask-cuda-worker/LocalCUDACluster (#236) @pentschev
• Restrict CuPy to <7.2 (#239) @quasiben
• Fix UCX configurations (#246) @pentschev
• Respect temporary-directory config for spilling (#247) @jakirkham
• Relax CuPy pin (#248) @jakirkham
• Added ignore_index argument to partition_by_hash() (#253) @madsbk
• Use "dask" serialization to move to/from host (#256) @jakirkham
• Drop Numba DeviceNDArray code for sizeof (#257) @jakirkham
• Support spilling of device objects in dictionaries (#260) @madsbk

0.12.0

• Add ucx-py dependency to CI (#212) @raydouglass
• Follow-up revision of local_cudf_merge benchmark (#213) @rjzamora
• Add codeowners file (#217) @raydouglass
• Add pypi upload script (#218) @raydouglass
• Skip existing on PyPi uploads (#219) @raydouglass
• Make benchmarks use rmm_cupy_allocator (#220) @madsbk
• cudf-merge-benchmark now reports throughput (#222) @madsbk
• Fix dask-cuda-worker --interface/--net-devices docs (#223) @pentschev
• Use RMM for serialization when available (#227) @pentschev

0.11.0

• Use UCX-Py initialization API (#152) @pentschev
• Remove all CUDA labels (#160) @mike-wendt
• Setting UCX options through dask global config (#168) @madsbk
• Make test_cudf_device_spill xfail (#170) @pentschev
• Updated CI, cleanup tests and reformat Python files (#171) @madsbk
• Fix GPU dependency versions (#173) @dillon-cullinan
• Set LocalCUDACluster n_workers equal to the length of CUDA_VISIBLE_DEVICES (#174) @mrocklin
• Simplify dask-cuda code (#175) @madsbk
• DGX inherit from LocalCUDACluster (#177) @madsbk
• Single-node CUDA benchmarks (#179) @madsbk
• Set TCP for UCX tests (#180) @pentschev
• Single-node cuDF merge benchmarks (#183) @madsbk
• Add black and isort checks in CI (#185) @pentschev
• Remove outdated xfail/importorskip test entries (#188) @pentschev
• Use UCX-Py's TopologicalDistance to determine IB interfaces in DGX (#189) @pentschev
• Dask Performance Report (#192) @madsbk
• Allow test_cupy_device_spill to xfail (#195) @pentschev
• Use ucx-py from rapidsai-nightly in CI (#196) @pentschev
• LocalCUDACluster sets closest network device (#200) @madsbk
• Expand cudf-merge benchmark (#201) @rjzamora
• Added --runs to merge benchmark (#202) @madsbk
• Move UCX code to LocalCUDACluster and deprecate DGX (#205) @pentschev
• Add markdown output option to cuDF merge benchmark (#208) @quasiben

0.10.0

• Change the updated new_worker_spec API for upstream (#128) @mrocklin
• Update TOTAL_MEMORY to match new distributed MEMORY_LIMIT (#131) @pentschev
• Bum Dask requirement to 2.4 (#133) @mrocklin
• Use YYMMDD tag in nightly build (#134) @mluukkainen
• Automatically determine CPU affinity (#138) @pentschev
• Fix full memory use check testcase (#139) @ksangeek
• Use pynvml to get memory info without creating CUDA context (#140) @pentschev
• Pass missing local_directory to Nanny from dask-cuda-worker (#141) @pentschev
• New worker_spec function for worker recipes (#147) @pentschev
• Add new Scheduler class supporting environment variables (#149) @pentschev
• Support for TCP over UCX (#152) @pentschev

0.9.0

• Fix serialization of collections and bump dask to 2.3.0 (#118) @pentschev
• Add versioneer (#88) @matthieubulte
• Python CodeCov Integration (#91) @dillon-cullinan
• Update cudf, dask, dask-cudf, distributed version requirements (#97) @pentschev
• Improve device memory spilling performance (#98) @pentschev
• Update dask-cuda for dask 2.2 (#101) @mrocklin
• Streamline CUDA_REL environment variable (#102) @okoskinen
• Replace ncores= parameter with nthreads= (#101) @mrocklin
• Fix remove CodeCov upload from build script (#115) @dillon-cullinan
• Remove CodeCov upload (#116) @dillon-cullinan

0.8.0

• Add device memory spill support (LRU-based only) (#51) @pentschev
• Update CI dependency to CuPy 6.0.0 (#53) @pentschev
• Add a hard-coded DGX configuration (#46) (#70) @mrocklin
• Fix LocalCUDACluster data spilling and its test (#67) @pentschev
• Add test skipping functionality to build.sh (#71) @dillon-cullinan
• Replace use of ncores= keywords with nthreads= (#75) @mrocklin
• Fix device memory spilling with cuDF (#65) @pentschev
• LocalCUDACluster calls _correct_state() to ensure workers started (#78) @pentschev
• Delay some of spilling test assertions (#80) @pentschev