Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploying RAPIDS #39

Closed
quasiben opened this issue Aug 8, 2020 · 53 comments
Closed

Deploying RAPIDS #39

quasiben opened this issue Aug 8, 2020 · 53 comments

Comments

@quasiben
Copy link

quasiben commented Aug 8, 2020

When deploying to GPUs I'd like the option of using the RAPIDS stack. This can be deployed either with a set of conda packages (with optional nightlies):

conda install -c rapidsai -c nvidia -c conda-forge rapids=0.14 cudatoolkit=$CUDA_VERSION

or official rapids images on dockerhub. In either case, deploying Dask+RAPIDS typically means using dask-cuda-workers instead of the dask-worker. dask-cuda-workers are opinionated in configuring GPUs for dask (1 worker/1 thread per GPU).

I'm not sure what is the best for users and coiled here. Some up front knowledge may be needed to be exposed to the user about the GPUs, specifically the CUDA VERSION on the machine: 10.1/10.2/11. We'd also want the option of using a different worker (maybe this is generalized to any executable ? though that's probably a big can of worms).

The RAPIDS nightly docker images is quite nice as it combines RAPIDS/DASK/XGBoost/etc and necessary dependencies. However, the conda env is store /opt/conda/rapids and I don't think coiled exposes a custom path/conda option for docker images at the moments

The next thing after RAPIDS is working nicely is getting BlazingSQL which can also be installed with conda/docker:

conda install -c blazingsql/label/cuda10.2 -c blazingsql -c rapidsai -c nvidia -c conda-forge -c defaults blazingsql python=3.7

From BSQL install docs.
@mrocklin
Copy link
Member

mrocklin commented Aug 8, 2020

In either case, deploying Dask+RAPIDS typically means using dask-cuda-workers instead of the dask-worker. dask-cuda-workers are opinionated in configuring GPUs for dask (1 worker/1 thread per GPU).

I've raised an issue internally with a design for supporting CUDAWorker (and other things along the way) I think that it will take about a day, but it might not be the first thing we start working on on Monday.

I'm curious if we can, short term, ask for tasks that have a single CPU core and if that would sufficiently mimic dask-cuda-worker for short-term use.

I'm not sure what is the best for users and coiled here. Some up front knowledge may be needed to be exposed to the user about the GPUs, specifically the CUDA VERSION on the machine: 10.1/10.2/11.

Yeah, I'm not sure what is best here yet. Let's think on this.

The RAPIDS nightly docker images is quite nice as it combines RAPIDS/DASK/XGBoost/etc and necessary dependencies. However, the conda env is store /opt/conda/rapids and I don't think coiled exposes a custom path/conda option for docker images at the moments

I don't suppose that the python command on the path points to the right executable, does it?

@mrocklin
Copy link
Member

mrocklin commented Aug 8, 2020

Hrm, I tried building a software environment, but I ran into the space-on-device issue again (which we're working to resolve)

coiled.create_software_environment(
    name='rapids', 
    conda={
        "channels": ["blazingsql/label/cuda10.2", "blazingsql", "rapidsai", "nvidia", "conda-forge", "defaults"], 
        "dependencies": ["blazingsql", "rapids=0.14", "cudatoolkit=10.2"]
    }
)
Traceback 
~/workspace/coiled/cloud/python-api/coiled/core.py in create_software_environment(name, conda, pip, container, log_output, post_build)
    796             container=container,
    797             post_build=post_build,
--> 798             log_output=log_output,
    799         )
    800 

~/workspace/coiled/cloud/python-api/coiled/core.py in create_software_environment(self, name, conda, pip, container, post_build, log_output)
    437             container=container,
    438             post_build=post_build,
--> 439             log_output=log_output,
    440         )
    441 

~/workspace/coiled/cloud/python-api/coiled/core.py in _sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    276         else:
    277             return sync(
--> 278                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    279             )
    280 

~/workspace/distributed/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    337     if error[0]:
    338         typ, exc, tb = error[0]
--> 339         raise exc.with_traceback(tb)
    340     else:
    341         return result[0]

~/workspace/distributed/distributed/utils.py in f()
    321             if callback_timeout is not None:
    322                 future = asyncio.wait_for(future, callback_timeout)
--> 323             result[0] = yield future
    324         except Exception as exc:
    325             error[0] = sys.exc_info()

~/miniconda/lib/python3.7/site-packages/tornado/gen.py in run(self)
    733 
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

~/workspace/coiled/cloud/python-api/coiled/core.py in _create_software_environment(self, name, conda, pip, container, post_build, log_output)
    520         error_details = await self._websocket_stream(ws, log_output, use_spinner=False)
    521         if error_details:
--> 522             raise ValueError(f"Unable to update Environment: {error_details}")
    523 
    524     async def _list_software_environments(self, account=None):

ValueError: Unable to update Environment: Docker build failed:
STEP 1: FROM continuumio/miniconda3:4.8.2

STEP 2: COPY environment.yml environment.yml

STEP 3: RUN conda env update -n base -f environment.yml && conda clean --all -y

Collecting package metadata (repodata.json): ...working... done

Solving environment: ...working... done



Downloading and Extracting Packages

cudf-0.14.0          | 25.7 MB   | ########## | 100% 

pandas-0.25.3        | 11.4 MB   | ########## | 100% 

kiwisolver-1.2.0     | 87 KB     | ########## | 100% 

parso-0.7.1          | 70 KB     | ########## | 100% 

numba-0.49.1         | 3.1 MB    | ########## | 100% 

zstd-1.4.4           | 982 KB    | ########## | 100% 

defusedxml-0.6.0     | 22 KB     | ########## | 100% 

matplotlib-base-3.3. | 6.7 MB    | ########## | 100% 

expat-2.2.9          | 191 KB    | ########## | 100% 

libdap4-3.20.4       | 18.5 MB   | ########## | 100% 

distributed-2.22.0   | 1.0 MB    | ########## | 100% 

cycler-0.10.0        | 9 KB      | ########## | 100% 

dask-xgboost-0.2.0.d | 14 KB     | ########## | 100% 

libgomp-9.3.0        | 374 KB    | ########## | 100% 

libevent-2.1.10      | 1.3 MB    | ########## | 100% 

pandocfilters-1.4.2  | 9 KB      | ########## | 100% 

ipython_genutils-0.2 | 21 KB     | ########## | 100% 

libgcc-ng-9.3.0      | 7.8 MB    | ########## | 100% 

webencodings-0.5.1   | 12 KB     | ########## | 100% 

c-ares-1.16.1        | 108 KB    | ########## | 100% 

xorg-libxau-1.0.9    | 13 KB     | ########## | 100% 

libprotobuf-3.8.0    | 4.7 MB    | ########## | 100% 

ipykernel-5.3.4      | 165 KB    | ########## | 100% 

pcre-8.44            | 261 KB    | ########## | 100% 

libgpg-error-1.36    | 259 KB    | ########## | 100% 

libhwloc-2.1.0       | 2.7 MB    | ########## | 100% 

requests-2.24.0      | 47 KB     | ########## | 100% 

arrow-cpp-0.15.0     | 18.1 MB   | ########## | 100% 

cugraph-0.14.0       | 6.9 MB    | ########## | 100% 

fsspec-0.8.0         | 61 KB     | ########## | 100% 

proj-7.0.0           | 3.7 MB    | ########## | 100% 

pyviz_comms-0.7.6    | 13 KB     | ########## | 100% 

joblib-0.16.0        | 203 KB    | ########## | 100% 

libgdal-3.0.4        | 18.7 MB   | ########## | 100% 

pyrsistent-0.16.0    | 89 KB     | ########## | 100% 

munch-2.5.0          | 12 KB     | ########## | 100% 

xgboost-1.1.0dev.rap | 12 KB     | ########## | 100% 

librmm-0.14.0        | 189 KB    | ########## | 100% 

xorg-renderproto-0.1 | 8 KB      | ########## | 100% 

pyproj-2.6.1.post1   | 438 KB    | ########## | 100% 

zlib-1.2.11          | 105 KB    | ########## | 100% 

xorg-libxdmcp-1.1.3  | 18 KB     | ########## | 100% 

jupyter-server-proxy | 17 KB     | ########## | 100% 

cyrus-sasl-2.1.27    | 232 KB    | ########## | 100% 

libedit-3.1.20191231 | 122 KB    | ########## | 100% 

libcumlprims-0.14.1  | 6.0 MB    | ########## | 100% 

openssl-1.1.1g       | 2.1 MB    | ########## | 100% 

pthread-stubs-0.4    | 5 KB      | ########## | 100% 

libuv-1.34.0         | 922 KB    | ########## | 100% 

bsql-rapids-thirdpar | 1.2 MB    | ########## | 100% 

libllvm8-8.0.1       | 23.2 MB   | ########## | 100% 

poppler-0.67.0       | 8.9 MB    | ########## | 100% 

wcwidth-0.2.5        | 33 KB     | ########## | 100% 

dask-core-2.22.0     | 624 KB    | ########## | 100% 

pygments-2.6.1       | 683 KB    | ########## | 100% 

panel-0.6.4          | 332 KB    | ########## | 100% 

pyppeteer-0.0.25     | 59 KB     | ########## | 100% 

libspatialite-4.3.0a | 3.1 MB    | ########## | 100% 

libwebp-1.1.0        | 85 KB     | ########## | 100% 

importlib_metadata-1 | 3 KB      | ########## | 100% 

libcuspatial-0.14.0  | 3.4 MB    | ########## | 100% 

tblib-1.6.0          | 14 KB     | ########## | 100% 

cairo-1.16.0         | 1.5 MB    | ########## | 100% 

cuml-0.14.0          | 9.6 MB    | ########## | 100% 

_openmp_mutex-4.5    | 22 KB     | ########## | 100% 

xorg-libsm-1.2.3     | 25 KB     | ########## | 100% 

libkml-1.3.0         | 643 KB    | ########## | 100% 

ucx-py-0.14.0+gf6ec8 | 137 KB    | ########## | 100% 

toolz-0.10.0         | 46 KB     | ########## | 100% 

ipython-7.17.0       | 1.1 MB    | ########## | 100% 

libwebp-base-1.1.0   | 845 KB    | ########## | 100% 

libtiff-4.1.0        | 668 KB    | ########## | 100% 

_libgcc_mutex-0.1    | 3 KB      | ########## | 100% 

typing_extensions-3. | 25 KB     | ########## | 100% 

pyzmq-19.0.2         | 479 KB    | ########## | 100% 

thrift-cpp-0.12.0    | 2.4 MB    | ########## | 100% 

zeromq-4.3.2         | 326 KB    | ########## | 100% 

cligj-0.5.0          | 8 KB      | ########## | 100% 

attrs-19.3.0         | 35 KB     | ########## | 100% 

xarray-0.16.0        | 514 KB    | ########## | 100% 

boost-1.70.0         | 337 KB    | ########## | 100% 

cuxfilter-0.14.0     | 101 KB    | ########## | 100% 

krb5-1.16.4          | 1.4 MB    | ########## | 100% 

backports.functools_ | 8 KB      | ########## | 100% 

libxcb-1.13          | 396 KB    | ########## | 100% 

snappy-1.1.8         | 32 KB     | ########## | 100% 

markupsafe-1.1.1     | 26 KB     | ########## | 100% 

hdf5-1.10.5          | 3.1 MB    | ########## | 100% 

parquet-cpp-1.5.1    | 3 KB      | ########## | 100% 

bleach-3.1.5         | 112 KB    | ########## | 100% 

fastavro-0.24.0      | 388 KB    | ########## | 100% 

rapids-xgboost-0.14. | 4 KB      | ########## | 100% 

ncurses-6.2          | 993 KB    | ########## | 100% 

libnetcdf-4.7.4      | 1.3 MB    | ########## | 100% 

gettext-0.19.8.1     | 3.6 MB    | ########## | 100% 

ucx-1.8.0+gf6ec8d4   | 8.9 MB    | ########## | 100% 

freexl-1.0.5         | 46 KB     | ########## | 100% 

pickleshare-0.7.5    | 13 KB     | ########## | 100% 

libgfortran-ng-7.5.0 | 1.3 MB    | ########## | 100% 

xz-5.2.5             | 343 KB    | ########## | 100% 

pyarrow-0.15.0       | 3.2 MB    | ########## | 100% 

poppler-data-0.4.9   | 3.4 MB    | ########## | 100% 

bsql-toolchain-gcp-c | 2.5 MB    | ########## | 100% 

libntlm-1.4          | 32 KB     | ########## | 100% 

imagecodecs-lite-201 | 188 KB    | ########## | 100% 

libblas-3.8.0        | 11 KB     | ########## | 100% 

xerces-c-3.2.2       | 1.7 MB    | ########## | 100% 

geotiff-1.5.1        | 279 KB    | ########## | 100% 

cloudpickle-1.5.0    | 22 KB     | ########## | 100% 

bsql-toolchain-aws-c | 1.7 MB    | ########## | 100% 

jsonschema-3.2.0     | 89 KB     | ########## | 100% 

pandoc-2.10.1        | 19.3 MB   | ########## | 100% 

rtree-0.9.4          | 43 KB     | ########## | 100% 

ld_impl_linux-64-2.3 | 612 KB    | ########## | 100% 

sqlalchemy-1.3.18    | 1.8 MB    | ########## | 100% 

datashape-0.5.4      | 49 KB     | ########## | 100% 

dask-cudf-0.14.0     | 82 KB     | ########## | 100% 

geopandas-0.8.1      | 925 KB    | ########## | 100% 

notebook-6.1.1       | 6.2 MB    | ########## | 100% 

libxgboost-1.1.0dev. | 31.5 MB   | ########## | 100% 

prometheus_client-0. | 44 KB     | ########## | 100% 

olefile-0.46         | 31 KB     | ########## | 100% 

pixman-0.38.0        | 594 KB    | ########## | 100% 

chardet-3.0.4        | 169 KB    | ########## | 100% 

packaging-20.4       | 32 KB     | ########## | 100% 

cryptography-3.0     | 633 KB    | ########## | 100% 

pyct-0.4.6           | 3 KB      | ########## | 100% 

dlpack-0.3           | 13 KB     | ########## | 100% 

pyct-core-0.4.6      | 13 KB     | ########## | 100% 

uriparser-0.9.3      | 49 KB     | ########## | 100% 

libgcrypt-1.8.4      | 614 KB    | ########## | 100% 

pyyaml-5.3.1         | 185 KB    | ########## | 100% 

rapids-0.14.1        | 4 KB      | ########## | 100% 

zict-2.0.0           | 10 KB     | ########## | 100% 

bsql-toolchain-0.14. | 6 KB      | ########## | 100% 

xorg-xextproto-7.3.0 | 27 KB     | ########## | 100% 

xorg-xproto-7.0.31   | 72 KB     | ########## | 100% 

xorg-libxrender-0.9. | 31 KB     | ########## | 100% 

tqdm-4.48.2          | 53 KB     | ########## | 100% 

openjpeg-2.3.1       | 475 KB    | ########## | 100% 

fastrlock-0.5        | 31 KB     | ########## | 100% 

pexpect-4.8.0        | 79 KB     | ########## | 100% 

cudatoolkit-10.2.89  | 539.9 MB  | ########## | 100% 

zipp-3.1.0           | 10 KB     | ########## | 100% 

numpy-1.19.1         | 5.2 MB    | ########## | 100% 

click-7.1.2          | 64 KB     | ########## | 100% 

tifffile-2020.6.3    | 109 KB    | ########## | 100% 

blazingsql-0.14      | 133.4 MB  | ########## | 100% 

libssh2-1.9.0        | 225 KB    | ########## | 100% 

psutil-5.7.2         | 335 KB    | ########## | 100% 

libffi-3.2.1         | 47 KB     | ########## | 100% 

wheel-0.34.2         | 24 KB     | ########## | 100% 

appdirs-1.4.3        | 11 KB     | ########## | 100% 

nodejs-13.13.0       | 15.8 MB   | ########## | 100% 

imageio-2.9.0        | 3.1 MB    | ########## | 100% 

entrypoints-0.3      | 12 KB     | ########## | 100% 

scikit-image-0.17.2  | 10.8 MB   | ########## | 100% 

fiona-1.8.13         | 955 KB    | ########## | 100% 

sasl-0.2.1           | 56 KB     | ########## | 100% 

pip-20.2.1           | 1.1 MB    | ########## | 100% 

spdlog-1.7.0         | 410 KB    | ########## | 100% 

re2-2020.04.01       | 438 KB    | ########## | 100% 

xorg-libxext-1.3.4   | 51 KB     | ########## | 100% 

xorg-libice-1.0.10   | 57 KB     | ########## | 100% 

param-1.9.3          | 60 KB     | ########## | 100% 

simpervisor-0.3      | 8 KB      | ########## | 100% 

python-dateutil-2.8. | 220 KB    | ########## | 100% 

networkx-2.4         | 1.2 MB    | ########## | 100% 

nbformat-5.0.7       | 99 KB     | ########## | 100% 

freetype-2.10.2      | 905 KB    | ########## | 100% 

tzcode-2020a         | 425 KB    | ########## | 100% 

lz4-c-1.8.3          | 187 KB    | ########## | 100% 

cudnn-7.6.5          | 184.3 MB  | ########## | 100% 

giflib-5.2.1         | 80 KB     | ########## | 100% 

readline-8.0         | 281 KB    | ########## | 100% 

partd-1.1.0          | 17 KB     | ########## | 100% 

locket-0.2.0         | 6 KB      | ########## | 100% 

brotli-1.0.7         | 386 KB    | ########## | 100% 

grpc-cpp-1.23.0      | 4.5 MB    | ########## | 100% 

websockets-8.1       | 90 KB     | ########## | 100% 

pyparsing-2.4.7      | 60 KB     | ########## | 100% 

pyee-7.0.2           | 13 KB     | ########## | 100% 

libpng-1.6.37        | 308 KB    | ########## | 100% 

dask-2.22.0          | 4 KB      | ########## | 100% 

libcugraph-0.14.0    | 14.3 MB   | ########## | 100% 

libspatialindex-1.9. | 3.1 MB    | ########## | 100% 

pynvml-8.0.4         | 31 KB     | ########## | 100% 

pytz-2020.1          | 227 KB    | ########## | 100% 

multidict-4.7.5      | 69 KB     | ########## | 100% 

jedi-0.17.2          | 949 KB    | ########## | 100% 

shapely-1.7.0        | 435 KB    | ########## | 100% 

boost-cpp-1.70.0     | 21.1 MB   | ########## | 100% 

postgresql-12.2      | 5.0 MB    | ########## | 100% 

libcblas-3.8.0       | 11 KB     | ########## | 100% 

cusignal-0.14.0      | 76 KB     | ########## | 100% 

colorcet-2.0.1       | 1.5 MB    | ########## | 100% 

cupy-7.7.0           | 20.5 MB   | ########## | 100% 

libpq-12.2           | 2.6 MB    | ########## | 100% 

jpeg-9d              | 266 KB    | ########## | 100% 

rmm-0.14.0           | 689 KB    | ########## | 100% 

libiconv-1.15        | 2.0 MB    | ########## | 100% 

python-3.7.8         | 53.1 MB   | ########## | 100% 

netifaces-0.10.9     | 17 KB     | ########## | 100% 

libcurl-7.68.0       | 564 KB    | ########## | 100% 

thrift-0.13.0        | 120 KB    | ########## | 100% 

gdal-3.0.4           | 1.4 MB    | ########## | 100% 

nbconvert-5.6.1      | 487 KB    | ########## | 100% 

argon2-cffi-20.1.0   | 47 KB     | ########## | 100% 

libcuml-0.14.0       | 42.3 MB   | ########## | 100% 

openjdk-8.0.192      | 69.6 MB   | ########## | 100% 

threadpoolctl-2.1.0  | 15 KB     | ########## | 100% 

tk-8.6.10            | 3.2 MB    | ########## | 100% 

pycparser-2.20       | 94 KB     | ########## | 100% 

libsodium-1.0.18     | 339 KB    | ########## | 100% 

jinja2-2.11.2        | 93 KB     | ########## | 100% 

click-plugins-1.1.1  | 9 KB      | ########## | 100% 

libopenblas-0.3.10   | 7.8 MB    | ########## | 100% 

json-c-0.13.1        | 76 KB     | ########## | 100% 

datashader-0.10.0    | 14.0 MB   | ########## | 100% 

lcms2-2.11           | 431 KB    | ########## | 100% 

backports-1.0        | 4 KB      | ########## | 100% 

prompt-toolkit-3.0.5 | 233 KB    | ########## | 100% 

urllib3-1.25.10      | 92 KB     | ########## | 100% 

multipledispatch-0.6 | 12 KB     | ########## | 100% 

setuptools-49.2.1    | 929 KB    | ########## | 100% 

gtest-1.10.0         | 250 KB    | ########## | 100% 

libcudf-0.14.0       | 102.8 MB  | ########## | 100% 

libstdcxx-ng-9.3.0   | 4.0 MB    | ########## | 100% 

idna-2.10            | 52 KB     | ########## | 100% 

kealib-1.4.13        | 172 KB    | ########## | 100% 

six-1.15.0           | 14 KB     | ########## | 100% 

future-0.18.2        | 712 KB    | ########## | 100% 

bokeh-1.4.0          | 13.5 MB   | ########## | 100% 

double-conversion-3. | 85 KB     | ########## | 100% 

jupyter_client-6.1.6 | 75 KB     | ########## | 100% 

terminado-0.8.3      | 23 KB     | ########## | 100% 

geos-3.8.1           | 1.0 MB    | ########## | 100% 

hdf4-4.2.13          | 964 KB    | ########## | 100% 

libuuid-2.32.1       | 26 KB     | ########## | 100% 

cppzmq-4.6.0         | 20 KB     | ########## | 100% 

sortedcontainers-2.2 | 25 KB     | ########## | 100% 

jupyter_core-4.6.3   | 71 KB     | ########## | 100% 

pyopenssl-19.1.0     | 47 KB     | ########## | 100% 

sqlite-3.32.3        | 1.4 MB    | ########## | 100% 

pyhive-0.6.2         | 37 KB     | ########## | 100% 

importlib-metadata-1 | 44 KB     | ########## | 100% 

libhdfs3-2.3         | 8.9 MB    | ########## | 100% 

pysocks-1.7.1        | 27 KB     | ########## | 100% 

yarl-1.3.0           | 133 KB    | ########## | 100% 

liblapack-3.8.0      | 11 KB     | ########## | 100% 

xorg-kbproto-1.0.7   | 26 KB     | ########## | 100% 

cfitsio-3.470        | 1.4 MB    | ########## | 100% 

tbb-2018.0.5         | 1.1 MB    | ########## | 100% 

yaml-0.2.5           | 82 KB     | ########## | 100% 

libgsasl-1.8.0       | 132 KB    | ########## | 100% 

tiledb-1.7.7         | 2.0 MB    | ########## | 100% 

ca-certificates-2020 | 145 KB    | ########## | 100% 

brotlipy-0.7.0       | 346 KB    | ########## | 100% 

cffi-1.14.1          | 223 KB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    | ########## | 100% 

llvmlite-0.32.1      | 316 KB    | ########## | 100% 

libxml2-2.9.10       | 1.3 MB    | ########## | 100% 

libnvstrings-0.14.0  | 30.2 MB   | ########## | 100% 

nvstrings-0.14.0     | 129 KB    | ########## | 100% 

tornado-6.0.4        | 639 KB    | ########## | 100% 

msgpack-python-1.0.0 | 90 KB     | ########## | 100% 

jpype1-1.0.2         | 3.0 MB    | ########## | 100% 

traitlets-4.3.3      | 133 KB    | ########## | 100% 

curl-7.68.0          | 137 KB    | ########## | 100% 

heapdict-1.0.1       | 7 KB      | ########## | 100% 

markdown-3.2.2       | 61 KB     | ########## | 100% 

gflags-2.2.2         | 114 KB    | ########## | 100% 

scikit-learn-0.23.2  | 6.8 MB    | ########## | 100% 

decorator-4.4.2      | 11 KB     | ########## | 100% 

send2trash-1.5.0     | 12 KB     | ########## | 100% 

cuspatial-0.14.0     | 3.6 MB    | ########## | 100% 

icu-64.2             | 12.6 MB   | ########## | 100% 

backcall-0.2.0       | 13 KB     | ########## | 100% 

xorg-libx11-1.6.11   | 920 KB    | ########## | 100% 

ptyprocess-0.6.0     | 15 KB     | ########## | 100% 

cytoolz-0.10.1       | 432 KB    | ########## | 100% 

pywavelets-1.1.1     | 4.4 MB    | ########## | 100% 

scipy-1.5.2          | 18.5 MB   | ########## | 100% 

pillow-7.2.0         | 675 KB    | ########## | 100% 

bzip2-1.0.8          | 396 KB    | ########## | 100% 

testpath-0.4.4       | 85 KB     | ########## | 100% 

python_abi-3.7       | 4 KB      | ########## | 100% 

fontconfig-2.13.1    | 340 KB    | ########## | 100% 

mistune-0.8.4        | 53 KB     | ########## | 100% 

thrift_sasl-0.4.2    | 14 KB     | ########## | 100% 

async-timeout-3.0.1  | 11 KB     | ########## | 100% 

dask-cuda-0.14.1     | 71 KB     | ########## | 100% 

glib-2.65.0          | 3.5 MB    | ########## | 100% 

nccl-2.5.7.1         | 104.1 MB  | ########## | 100% 

ruamel_yaml-0.15.80  | 267 KB    | ########## | 100% 

py-xgboost-1.1.0dev. | 106 KB    | ########## | 100% 

glog-0.4.0           | 104 KB    | ########## | 100% 

aiohttp-3.6.2        | 634 KB    | ########## | 100% 

Preparing transaction: ...working... done

Verifying transaction: ...working... done

Executing transaction: ...working... done

#

# To activate this environment, use

#

#     $ conda activate base

#

# To deactivate an active environment, use

#

#     $ conda deactivate



Cache location: /opt/conda/pkgs

Will remove the following tarballs:



/opt/conda/pkgs

---------------

lz4-c-1.8.3-he1b5a44_1001.tar.bz2            187 KB

poppler-data-0.4.9-1.tar.bz2                 3.4 MB

libxgboost-1.1.0dev.rapidsai0.14-cuda10.2_0.tar.bz2    31.5 MB

numpy-1.19.1-py37h8960a57_0.tar.bz2          5.2 MB

async-timeout-3.0.1-py_1000.tar.bz2           11 KB

libcuml-0.14.0-cuda10.2_0.tar.bz2           42.3 MB

markdown-3.2.2-py_0.tar.bz2                   61 KB

bsql-toolchain-0.14.0-0.tar.bz2                6 KB

mistune-0.8.4-py37h8f50634_1001.tar.bz2       53 KB

fsspec-0.8.0-py_0.tar.bz2                     61 KB

typing_extensions-3.7.4.2-py_0.tar.bz2        25 KB

gettext-0.19.8.1-hc5be6a0_1002.tar.bz2       3.6 MB

geos-3.8.1-he1b5a44_0.tar.bz2                1.0 MB

jsonschema-3.2.0-py37hc8dfbb8_1.tar.bz2       89 KB

pyproj-2.6.1.post1-py37h34dd122_0.tar.bz2     438 KB

cairo-1.16.0-hcf35c78_1003.tar.bz2           1.5 MB

libevent-2.1.10-hcdb4288_1.tar.bz2           1.3 MB

pandoc-2.10.1-h516909a_0.tar.bz2            19.3 MB

hdf4-4.2.13-hf30be14_1003.tar.bz2            964 KB

arrow-cpp-0.15.0-py37h090bef1_2.tar.bz2     18.1 MB

bsql-toolchain-aws-cpp-0.14.0-0.tar.bz2      1.7 MB

tbb-2018.0.5-h2d50403_0.tar.bz2              1.1 MB

bsql-toolchain-gcp-cpp-0.14.0-0.tar.bz2      2.5 MB

libcblas-3.8.0-17_openblas.tar.bz2            11 KB

python-dateutil-2.8.1-py_0.tar.bz2           220 KB

bsql-rapids-thirdparty-0.14.0-0.tar.bz2      1.2 MB

dask-cuda-0.14.1-py37_0.tar.bz2               71 KB

kiwisolver-1.2.0-py37h99015e2_0.tar.bz2       87 KB

urllib3-1.25.10-py_0.tar.bz2                  92 KB

poppler-0.67.0-h14e79db_8.tar.bz2            8.9 MB

libgomp-9.3.0-h24d8f2e_14.tar.bz2            374 KB

boost-1.70.0-py37h9de70de_1.tar.bz2          337 KB

requests-2.24.0-pyh9f0ad1d_0.tar.bz2          47 KB

tornado-6.0.4-py37h8f50634_1.tar.bz2         639 KB

argon2-cffi-20.1.0-py37h8f50634_1.tar.bz2      47 KB

libpng-1.6.37-hed695b0_1.tar.bz2             308 KB

libwebp-1.1.0-h56121f0_4.tar.bz2              85 KB

boost-cpp-1.70.0-h8e57a91_2.tar.bz2         21.1 MB

snappy-1.1.8-he1b5a44_3.tar.bz2               32 KB

xorg-renderproto-0.11.1-h14c3975_1002.tar.bz2       8 KB

sortedcontainers-2.2.2-pyh9f0ad1d_0.tar.bz2      25 KB

markupsafe-1.1.1-py37h8f50634_1.tar.bz2       26 KB

pillow-7.2.0-py37h718be6c_1.tar.bz2          675 KB

nodejs-13.13.0-hf5d1a2b_0.tar.bz2           15.8 MB

libgpg-error-1.36-he1b5a44_0.tar.bz2         259 KB

libiconv-1.15-h516909a_1006.tar.bz2          2.0 MB

brotlipy-0.7.0-py37h8f50634_1000.tar.bz2     346 KB

websockets-8.1-py37h8f50634_1.tar.bz2         90 KB

yarl-1.3.0-py37h516909a_1000.tar.bz2         133 KB

nccl-2.5.7.1-hc6a2c23_0.tar.bz2            104.1 MB

jpeg-9d-h516909a_0.tar.bz2                   266 KB

notebook-6.1.1-py37hc8dfbb8_0.tar.bz2        6.2 MB

simpervisor-0.3-py_1.tar.bz2                   8 KB

xorg-kbproto-1.0.7-h14c3975_1002.tar.bz2      26 KB

libwebp-base-1.1.0-h516909a_3.tar.bz2        845 KB

numba-0.49.1-py37h0573a6f_0.conda            3.1 MB

libcurl-7.68.0-hda55be3_0.tar.bz2            564 KB

certifi-2020.6.20-py37hc8dfbb8_0.tar.bz2     151 KB

cryptography-3.0-py37hb09aad4_0.tar.bz2      633 KB

cloudpickle-1.5.0-py_0.tar.bz2                22 KB

libgsasl-1.8.0-h19a2143_1004.tar.bz2         132 KB

pyee-7.0.2-pyh9f0ad1d_0.tar.bz2               13 KB

ipython-7.17.0-py37hc6149b9_0.tar.bz2        1.1 MB

cytoolz-0.10.1-py37h516909a_0.tar.bz2        432 KB

idna-2.10-pyh9f0ad1d_0.tar.bz2                52 KB

glib-2.65.0-h6f030ca_0.tar.bz2               3.5 MB

ruamel_yaml-0.15.80-py37h8f50634_1001.tar.bz2     267 KB

sasl-0.2.1-py37h3340039_1002.tar.bz2          56 KB

ucx-py-0.14.0+gf6ec8d4-py37_0.tar.bz2        137 KB

libblas-3.8.0-17_openblas.tar.bz2             11 KB

llvmlite-0.32.1-py37h5202443_0.tar.bz2       316 KB

jedi-0.17.2-py37hc8dfbb8_0.tar.bz2           949 KB

nbconvert-5.6.1-py37hc8dfbb8_1.tar.bz2       487 KB

yaml-0.2.5-h516909a_0.tar.bz2                 82 KB

libkml-1.3.0-h4fcabce_1010.tar.bz2           643 KB

libopenblas-0.3.10-pthreads_hb3c22a3_4.tar.bz2     7.8 MB

munch-2.5.0-py_0.tar.bz2                      12 KB

chardet-3.0.4-py37hc8dfbb8_1006.tar.bz2      169 KB

dask-core-2.22.0-py_0.tar.bz2                624 KB

proj-7.0.0-h5a2d94f_2.tar.bz2                3.7 MB

openjpeg-2.3.1-h981e76c_3.tar.bz2            475 KB

libxcb-1.13-h14c3975_1002.tar.bz2            396 KB

geopandas-0.8.1-py_0.tar.bz2                 925 KB

pip-20.2.1-py_0.tar.bz2                      1.1 MB

dask-2.22.0-py_0.tar.bz2                       4 KB

pcre-8.44-he1b5a44_0.tar.bz2                 261 KB

networkx-2.4-py_1.tar.bz2                    1.2 MB

thrift_sasl-0.4.2-py37h8f50634_0.tar.bz2      14 KB

bleach-3.1.5-pyh9f0ad1d_0.tar.bz2            112 KB

dlpack-0.3-he1b5a44_1.tar.bz2                 13 KB

heapdict-1.0.1-py_0.tar.bz2                    7 KB

glog-0.4.0-h49b9bf7_3.tar.bz2                104 KB

cfitsio-3.470-hb60a0a2_2.tar.bz2             1.4 MB

multidict-4.7.5-py37h8f50634_1.tar.bz2        69 KB

threadpoolctl-2.1.0-pyh5ca1d4c_0.tar.bz2      15 KB

bokeh-1.4.0-py37hc8dfbb8_1.tar.bz2          13.5 MB

giflib-5.2.1-h516909a_2.tar.bz2               80 KB

ptyprocess-0.6.0-py_1001.tar.bz2              15 KB

traitlets-4.3.3-py37hc8dfbb8_1.tar.bz2       133 KB

bzip2-1.0.8-h516909a_2.tar.bz2               396 KB

xorg-xproto-7.0.31-h14c3975_1007.tar.bz2      72 KB

tk-8.6.10-hed695b0_0.tar.bz2                 3.2 MB

backcall-0.2.0-pyh9f0ad1d_0.tar.bz2           13 KB

openssl-1.1.1g-h516909a_1.tar.bz2            2.1 MB

ld_impl_linux-64-2.34-hc38a660_9.tar.bz2     612 KB

ipython_genutils-0.2.0-py_1.tar.bz2           21 KB

rtree-0.9.4-py37h8526d28_1.tar.bz2            43 KB

blazingsql-0.14-cuda10.2_py37_949.tar.bz2   133.4 MB

pickleshare-0.7.5-py37hc8dfbb8_1001.tar.bz2      13 KB

rmm-0.14.0-py37_0.tar.bz2                    689 KB

c-ares-1.16.1-h516909a_0.tar.bz2             108 KB

libxml2-2.9.10-hee79883_0.tar.bz2            1.3 MB

libprotobuf-3.8.0-h8b12597_0.tar.bz2         4.7 MB

msgpack-python-1.0.0-py37h99015e2_1.tar.bz2      90 KB

packaging-20.4-pyh9f0ad1d_0.tar.bz2           32 KB

dask-xgboost-0.2.0.dev28-cuda10.2py37_0.tar.bz2      14 KB

libnvstrings-0.14.0-cuda10.2_0.tar.bz2      30.2 MB

ncurses-6.2-he1b5a44_1.tar.bz2               993 KB

xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0.tar.bz2      12 KB

pygments-2.6.1-py_0.tar.bz2                  683 KB

sqlalchemy-1.3.18-py37h8f50634_0.tar.bz2     1.8 MB

pyct-core-0.4.6-py_0.tar.bz2                  13 KB

freexl-1.0.5-h516909a_1002.tar.bz2            46 KB

librmm-0.14.0-cuda10.2_0.tar.bz2             189 KB

ipykernel-5.3.4-py37h43977f1_0.tar.bz2       165 KB

postgresql-12.2-hf1211e9_0.tar.bz2           5.0 MB

appdirs-1.4.3-py_1.tar.bz2                    11 KB

cuml-0.14.0-cuda10.2_py37_0.tar.bz2          9.6 MB

expat-2.2.9-he1b5a44_2.tar.bz2               191 KB

colorcet-2.0.1-py_0.tar.bz2                  1.5 MB

pandocfilters-1.4.2-py_1.tar.bz2               9 KB

parso-0.7.1-pyh9f0ad1d_0.tar.bz2              70 KB

libsodium-1.0.18-h516909a_0.tar.bz2          339 KB

_openmp_mutex-4.5-1_gnu.tar.bz2               22 KB

gtest-1.10.0-hc9558a2_4.tar.bz2              250 KB

thrift-cpp-0.12.0-hf3afdfd_1004.tar.bz2      2.4 MB

distributed-2.22.0-py37hc8dfbb8_0.tar.bz2     1.0 MB

pyarrow-0.15.0-py37h8b68381_1.tar.bz2        3.2 MB

brotli-1.0.7-he1b5a44_1004.tar.bz2           386 KB

libgdal-3.0.4-h3dfc09a_6.tar.bz2            18.7 MB

wheel-0.34.2-py_1.tar.bz2                     24 KB

fastrlock-0.5-py37h3340039_0.tar.bz2          31 KB

cupy-7.7.0-py37h940342b_0.tar.bz2           20.5 MB

libgcc-ng-9.3.0-h24d8f2e_14.tar.bz2          7.8 MB

param-1.9.3-py_0.tar.bz2                      60 KB

libuuid-2.32.1-h14c3975_1000.tar.bz2          26 KB

nvstrings-0.14.0-py37_0.tar.bz2              129 KB

libcumlprims-0.14.1-cuda10.2_0.tar.bz2       6.0 MB

six-1.15.0-pyh9f0ad1d_0.tar.bz2               14 KB

python-3.7.8-h6f2ec95_1_cpython.tar.bz2     53.1 MB

rapids-0.14.1-cuda10.2_py37_0.tar.bz2          4 KB

cudnn-7.6.5-cuda10.2_0.conda               184.3 MB

imagecodecs-lite-2019.12.3-py37h03ebfcd_1.tar.bz2     188 KB

imageio-2.9.0-py_0.tar.bz2                   3.1 MB

libedit-3.1.20191231-h46ee950_1.tar.bz2      122 KB

xorg-libsm-1.2.3-h84519dc_1000.tar.bz2        25 KB

cffi-1.14.1-py37h2b28604_0.tar.bz2           223 KB

openjdk-8.0.192-h516909a_1005.tar.bz2       69.6 MB

libcugraph-0.14.0-cuda10.2_0.tar.bz2        14.3 MB

nbformat-5.0.7-py_0.tar.bz2                   99 KB

parquet-cpp-1.5.1-2.tar.bz2                    3 KB

libntlm-1.4-h516909a_1002.tar.bz2             32 KB

libspatialindex-1.9.3-he1b5a44_3.tar.bz2     3.1 MB

libcuspatial-0.14.0-cuda10.2_0.tar.bz2       3.4 MB

psutil-5.7.2-py37h8f50634_0.tar.bz2          335 KB

jupyter-server-proxy-1.5.0-py_0.tar.bz2       17 KB

xorg-libx11-1.6.11-h516909a_0.tar.bz2        920 KB

pywavelets-1.1.1-py37h03ebfcd_1.tar.bz2      4.4 MB

jinja2-2.11.2-pyh9f0ad1d_0.tar.bz2            93 KB

decorator-4.4.2-py_0.tar.bz2                  11 KB

scikit-learn-0.23.2-py37h6785257_0.tar.bz2     6.8 MB

pyviz_comms-0.7.6-pyh9f0ad1d_0.tar.bz2        13 KB

cugraph-0.14.0-py37_0.tar.bz2                6.9 MB

datashape-0.5.4-py_1.tar.bz2                  49 KB

partd-1.1.0-py_0.tar.bz2                      17 KB

tifffile-2020.6.3-py_0.tar.bz2               109 KB

xorg-libxext-1.3.4-h516909a_0.tar.bz2         51 KB

dask-cudf-0.14.0-py37_0.tar.bz2               82 KB

gflags-2.2.2-he1b5a44_1004.tar.bz2           114 KB

locket-0.2.0-py_2.tar.bz2                      6 KB

fiona-1.8.13-py37h0492a4a_1.tar.bz2          955 KB

libssh2-1.9.0-hab1572f_5.tar.bz2             225 KB

libhdfs3-2.3-h311b756_1006.tar.bz2           8.9 MB

tblib-1.6.0-py_0.tar.bz2                      14 KB

py-xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0.tar.bz2     106 KB

gdal-3.0.4-py37h4b180d9_6.tar.bz2            1.4 MB

xz-5.2.5-h516909a_1.tar.bz2                  343 KB

jupyter_client-6.1.6-py_0.tar.bz2             75 KB

krb5-1.16.4-h2fd8d38_0.tar.bz2               1.4 MB

matplotlib-base-3.3.0-py37hd478181_1.tar.bz2     6.7 MB

scikit-image-0.17.2-py37h0da4684_1.tar.bz2    10.8 MB

libffi-3.2.1-he1b5a44_1007.tar.bz2            47 KB

shapely-1.7.0-py37hc88ce51_3.tar.bz2         435 KB

pyyaml-5.3.1-py37h8f50634_0.tar.bz2          185 KB

xarray-0.16.0-py_0.tar.bz2                   514 KB

fastavro-0.24.0-py37h8f50634_0.tar.bz2       388 KB

zipp-3.1.0-py_0.tar.bz2                       10 KB

tqdm-4.48.2-pyh9f0ad1d_0.tar.bz2              53 KB

attrs-19.3.0-py_0.tar.bz2                     35 KB

re2-2020.04.01-he1b5a44_0.tar.bz2            438 KB

toolz-0.10.0-py_0.tar.bz2                     46 KB

pycparser-2.20-pyh9f0ad1d_2.tar.bz2           94 KB

kealib-1.4.13-hec59c27_0.tar.bz2             172 KB

defusedxml-0.6.0-py_0.tar.bz2                 22 KB

cyrus-sasl-2.1.27-he38ecfd_0.tar.bz2         232 KB

rapids-xgboost-0.14.1-cuda10.2_py37_0.tar.bz2       4 KB

readline-8.0-he28a2e2_2.tar.bz2              281 KB

olefile-0.46-py_0.tar.bz2                     31 KB

panel-0.6.4-0.tar.bz2                        332 KB

libgfortran-ng-7.5.0-hdf63c60_14.tar.bz2     1.3 MB

fontconfig-2.13.1-h86ecdb6_1001.tar.bz2      340 KB

pyhive-0.6.2-pyh9f0ad1d_0.tar.bz2             37 KB

ucx-1.8.0+gf6ec8d4-cuda10.2_20.tar.bz2       8.9 MB

pixman-0.38.0-h516909a_1003.tar.bz2          594 KB

liblapack-3.8.0-17_openblas.tar.bz2           11 KB

entrypoints-0.3-py37hc8dfbb8_1001.tar.bz2      12 KB

sqlite-3.32.3-hcee41ef_1.tar.bz2             1.4 MB

pyparsing-2.4.7-pyh9f0ad1d_0.tar.bz2          60 KB

libdap4-3.20.4-hd3bb157_0.tar.bz2           18.5 MB

importlib-metadata-1.7.0-py37hc8dfbb8_0.tar.bz2      44 KB

xorg-libxdmcp-1.1.3-h516909a_0.tar.bz2        18 KB

xorg-libice-1.0.10-h516909a_0.tar.bz2         57 KB

setuptools-49.2.1-py37hc8dfbb8_0.tar.bz2     929 KB

pandas-0.25.3-py37hb3f55d8_0.tar.bz2        11.4 MB

future-0.18.2-py37hc8dfbb8_1.tar.bz2         712 KB

cuspatial-0.14.0-py37_0.tar.bz2              3.6 MB

cycler-0.10.0-py_2.tar.bz2                     9 KB

hdf5-1.10.5-nompi_h3c11f04_1104.tar.bz2      3.1 MB

libhwloc-2.1.0-h3c4fd83_0.tar.bz2            2.7 MB

scipy-1.5.2-py37hb14ef9d_0.tar.bz2          18.5 MB

json-c-0.13.1-hbfbb72e_1002.tar.bz2           76 KB

spdlog-1.7.0-hc9558a2_2.tar.bz2              410 KB

xorg-libxau-1.0.9-h14c3975_0.tar.bz2          13 KB

pynvml-8.0.4-py_1.tar.bz2                     31 KB

aiohttp-3.6.2-py37h516909a_0.tar.bz2         634 KB

zict-2.0.0-py_0.tar.bz2                       10 KB

curl-7.68.0-hf8cf82a_0.tar.bz2               137 KB

lcms2-2.11-hbd6801e_0.tar.bz2                431 KB

libstdcxx-ng-9.3.0-hdf63c60_14.tar.bz2       4.0 MB

zstd-1.4.4-h3b9ef0a_2.tar.bz2                982 KB

xerces-c-3.2.2-h8412b87_1004.tar.bz2         1.7 MB

cppzmq-4.6.0-hc9558a2_0.tar.bz2               20 KB

click-7.1.2-pyh9f0ad1d_0.tar.bz2              64 KB

pyrsistent-0.16.0-py37h8f50634_0.tar.bz2      89 KB

libpq-12.2-hae5116b_0.tar.bz2                2.6 MB

cusignal-0.14.0-py37_0.tar.bz2                76 KB

ca-certificates-2020.6.20-hecda079_0.tar.bz2     145 KB

pyct-0.4.6-py_0.tar.bz2                        3 KB

wcwidth-0.2.5-pyh9f0ad1d_1.tar.bz2            33 KB

joblib-0.16.0-py_0.tar.bz2                   203 KB

thrift-0.13.0-py37h3340039_1.tar.bz2         120 KB

libtiff-4.1.0-hc7e4089_6.tar.bz2             668 KB

pytz-2020.1-pyh9f0ad1d_0.tar.bz2             227 KB

terminado-0.8.3-py37hc8dfbb8_1.tar.bz2        23 KB

libllvm8-8.0.1-hc9558a2_0.tar.bz2           23.2 MB

uriparser-0.9.3-he1b5a44_1.tar.bz2            49 KB

backports.functools_lru_cache-1.6.1-py_0.tar.bz2       8 KB

netifaces-0.10.9-py37h8f50634_1002.tar.bz2      17 KB

pthread-stubs-0.4-h14c3975_1001.tar.bz2        5 KB

libuv-1.34.0-h516909a_0.tar.bz2              922 KB

multipledispatch-0.6.0-py_0.tar.bz2           12 KB

libnetcdf-4.7.4-nompi_h9f9fd6a_101.tar.bz2     1.3 MB

zeromq-4.3.2-he1b5a44_3.tar.bz2              326 KB

geotiff-1.5.1-h05acad5_10.tar.bz2            279 KB

prometheus_client-0.8.0-pyh9f0ad1d_0.tar.bz2      44 KB

pyppeteer-0.0.25-py_1.tar.bz2                 59 KB

datashader-0.10.0-py_0.tar.bz2              14.0 MB

send2trash-1.5.0-py_0.tar.bz2                 12 KB

tzcode-2020a-h516909a_0.tar.bz2              425 KB

pysocks-1.7.1-py37hc8dfbb8_1.tar.bz2          27 KB

libcudf-0.14.0-cuda10.2_0.tar.bz2          102.8 MB

xorg-xextproto-7.3.0-h14c3975_1002.tar.bz2      27 KB

tiledb-1.7.7-hcde45ca_0.tar.bz2              2.0 MB

libspatialite-4.3.0a-h2482549_1038.tar.bz2     3.1 MB

python_abi-3.7-1_cp37m.tar.bz2                 4 KB

testpath-0.4.4-py_0.tar.bz2                   85 KB

click-plugins-1.1.1-py_0.tar.bz2               9 KB

pyopenssl-19.1.0-py_1.tar.bz2                 47 KB

cudf-0.14.0-py37_0.tar.bz2                  25.7 MB

icu-64.2-he1b5a44_1.tar.bz2                 12.6 MB

libgcrypt-1.8.4-hf484d3e_1000.tar.bz2        614 KB

prompt-toolkit-3.0.5-py_1.tar.bz2            233 KB

cudatoolkit-10.2.89-h6bb024c_0.tar.bz2     539.9 MB

grpc-cpp-1.23.0-h18db393_0.tar.bz2           4.5 MB

webencodings-0.5.1-py_1.tar.bz2               12 KB

xorg-libxrender-0.9.10-h516909a_1002.tar.bz2      31 KB

cligj-0.5.0-py_0.tar.bz2                       8 KB

pyzmq-19.0.2-py37hac76be4_0.tar.bz2          479 KB

pexpect-4.8.0-py37hc8dfbb8_1.tar.bz2          79 KB

jupyter_core-4.6.3-py37hc8dfbb8_1.tar.bz2      71 KB

freetype-2.10.2-he06d7ca_0.tar.bz2           905 KB

cuxfilter-0.14.0-py37_0.tar.bz2              101 KB

jpype1-1.0.2-py37h99015e2_0.tar.bz2          3.0 MB

zlib-1.2.11-h516909a_1006.tar.bz2            105 KB

importlib_metadata-1.7.0-0.tar.bz2             3 KB

backports-1.0-py_2.tar.bz2                     4 KB

_libgcc_mutex-0.1-conda_forge.tar.bz2          3 KB

double-conversion-3.1.5-he1b5a44_2.tar.bz2      85 KB



---------------------------------------------------

Total:                                      1.77 GB



Removed lz4-c-1.8.3-he1b5a44_1001.tar.bz2

Removed poppler-data-0.4.9-1.tar.bz2

Removed libxgboost-1.1.0dev.rapidsai0.14-cuda10.2_0.tar.bz2

Removed numpy-1.19.1-py37h8960a57_0.tar.bz2

Removed async-timeout-3.0.1-py_1000.tar.bz2

Removed libcuml-0.14.0-cuda10.2_0.tar.bz2

Removed markdown-3.2.2-py_0.tar.bz2

Removed bsql-toolchain-0.14.0-0.tar.bz2

Removed mistune-0.8.4-py37h8f50634_1001.tar.bz2

Removed fsspec-0.8.0-py_0.tar.bz2

Removed typing_extensions-3.7.4.2-py_0.tar.bz2

Removed gettext-0.19.8.1-hc5be6a0_1002.tar.bz2

Removed geos-3.8.1-he1b5a44_0.tar.bz2

Removed jsonschema-3.2.0-py37hc8dfbb8_1.tar.bz2

Removed pyproj-2.6.1.post1-py37h34dd122_0.tar.bz2

Removed cairo-1.16.0-hcf35c78_1003.tar.bz2

Removed libevent-2.1.10-hcdb4288_1.tar.bz2

Removed pandoc-2.10.1-h516909a_0.tar.bz2

Removed hdf4-4.2.13-hf30be14_1003.tar.bz2

Removed arrow-cpp-0.15.0-py37h090bef1_2.tar.bz2

Removed bsql-toolchain-aws-cpp-0.14.0-0.tar.bz2

Removed tbb-2018.0.5-h2d50403_0.tar.bz2

Removed bsql-toolchain-gcp-cpp-0.14.0-0.tar.bz2

Removed libcblas-3.8.0-17_openblas.tar.bz2

Removed python-dateutil-2.8.1-py_0.tar.bz2

Removed bsql-rapids-thirdparty-0.14.0-0.tar.bz2

Removed dask-cuda-0.14.1-py37_0.tar.bz2

Removed kiwisolver-1.2.0-py37h99015e2_0.tar.bz2

Removed urllib3-1.25.10-py_0.tar.bz2

Removed poppler-0.67.0-h14e79db_8.tar.bz2

Removed libgomp-9.3.0-h24d8f2e_14.tar.bz2

Removed boost-1.70.0-py37h9de70de_1.tar.bz2

Removed requests-2.24.0-pyh9f0ad1d_0.tar.bz2

Removed tornado-6.0.4-py37h8f50634_1.tar.bz2

Removed argon2-cffi-20.1.0-py37h8f50634_1.tar.bz2

Removed libpng-1.6.37-hed695b0_1.tar.bz2

Removed libwebp-1.1.0-h56121f0_4.tar.bz2

Removed boost-cpp-1.70.0-h8e57a91_2.tar.bz2

Removed snappy-1.1.8-he1b5a44_3.tar.bz2

Removed xorg-renderproto-0.11.1-h14c3975_1002.tar.bz2

Removed sortedcontainers-2.2.2-pyh9f0ad1d_0.tar.bz2

Removed markupsafe-1.1.1-py37h8f50634_1.tar.bz2

Removed pillow-7.2.0-py37h718be6c_1.tar.bz2

Removed nodejs-13.13.0-hf5d1a2b_0.tar.bz2

Removed libgpg-error-1.36-he1b5a44_0.tar.bz2

Removed libiconv-1.15-h516909a_1006.tar.bz2

Removed brotlipy-0.7.0-py37h8f50634_1000.tar.bz2

Removed websockets-8.1-py37h8f50634_1.tar.bz2

Removed yarl-1.3.0-py37h516909a_1000.tar.bz2

Removed nccl-2.5.7.1-hc6a2c23_0.tar.bz2

Removed jpeg-9d-h516909a_0.tar.bz2

Removed notebook-6.1.1-py37hc8dfbb8_0.tar.bz2

Removed simpervisor-0.3-py_1.tar.bz2

Removed xorg-kbproto-1.0.7-h14c3975_1002.tar.bz2

Removed libwebp-base-1.1.0-h516909a_3.tar.bz2

Removed numba-0.49.1-py37h0573a6f_0.conda

Removed libcurl-7.68.0-hda55be3_0.tar.bz2

Removed certifi-2020.6.20-py37hc8dfbb8_0.tar.bz2

Removed cryptography-3.0-py37hb09aad4_0.tar.bz2

Removed cloudpickle-1.5.0-py_0.tar.bz2

Removed libgsasl-1.8.0-h19a2143_1004.tar.bz2

Removed pyee-7.0.2-pyh9f0ad1d_0.tar.bz2

Removed ipython-7.17.0-py37hc6149b9_0.tar.bz2

Removed cytoolz-0.10.1-py37h516909a_0.tar.bz2

Removed idna-2.10-pyh9f0ad1d_0.tar.bz2

Removed glib-2.65.0-h6f030ca_0.tar.bz2

Removed ruamel_yaml-0.15.80-py37h8f50634_1001.tar.bz2

Removed sasl-0.2.1-py37h3340039_1002.tar.bz2

Removed ucx-py-0.14.0+gf6ec8d4-py37_0.tar.bz2

Removed libblas-3.8.0-17_openblas.tar.bz2

Removed llvmlite-0.32.1-py37h5202443_0.tar.bz2

Removed jedi-0.17.2-py37hc8dfbb8_0.tar.bz2

Removed nbconvert-5.6.1-py37hc8dfbb8_1.tar.bz2

Removed yaml-0.2.5-h516909a_0.tar.bz2

Removed libkml-1.3.0-h4fcabce_1010.tar.bz2

Removed libopenblas-0.3.10-pthreads_hb3c22a3_4.tar.bz2

Removed munch-2.5.0-py_0.tar.bz2

Removed chardet-3.0.4-py37hc8dfbb8_1006.tar.bz2

Removed dask-core-2.22.0-py_0.tar.bz2

Removed proj-7.0.0-h5a2d94f_2.tar.bz2

Removed openjpeg-2.3.1-h981e76c_3.tar.bz2

Removed libxcb-1.13-h14c3975_1002.tar.bz2

Removed geopandas-0.8.1-py_0.tar.bz2

Removed pip-20.2.1-py_0.tar.bz2

Removed dask-2.22.0-py_0.tar.bz2

Removed pcre-8.44-he1b5a44_0.tar.bz2

Removed networkx-2.4-py_1.tar.bz2

Removed thrift_sasl-0.4.2-py37h8f50634_0.tar.bz2

Removed bleach-3.1.5-pyh9f0ad1d_0.tar.bz2

Removed dlpack-0.3-he1b5a44_1.tar.bz2

Removed heapdict-1.0.1-py_0.tar.bz2

Removed glog-0.4.0-h49b9bf7_3.tar.bz2

Removed cfitsio-3.470-hb60a0a2_2.tar.bz2

Removed multidict-4.7.5-py37h8f50634_1.tar.bz2

Removed threadpoolctl-2.1.0-pyh5ca1d4c_0.tar.bz2

Removed bokeh-1.4.0-py37hc8dfbb8_1.tar.bz2

Removed giflib-5.2.1-h516909a_2.tar.bz2

Removed ptyprocess-0.6.0-py_1001.tar.bz2

Removed traitlets-4.3.3-py37hc8dfbb8_1.tar.bz2

Removed bzip2-1.0.8-h516909a_2.tar.bz2

Removed xorg-xproto-7.0.31-h14c3975_1007.tar.bz2

Removed tk-8.6.10-hed695b0_0.tar.bz2

Removed backcall-0.2.0-pyh9f0ad1d_0.tar.bz2

Removed openssl-1.1.1g-h516909a_1.tar.bz2

Removed ld_impl_linux-64-2.34-hc38a660_9.tar.bz2

Removed ipython_genutils-0.2.0-py_1.tar.bz2

Removed rtree-0.9.4-py37h8526d28_1.tar.bz2

Removed blazingsql-0.14-cuda10.2_py37_949.tar.bz2

Removed pickleshare-0.7.5-py37hc8dfbb8_1001.tar.bz2

Removed rmm-0.14.0-py37_0.tar.bz2

Removed c-ares-1.16.1-h516909a_0.tar.bz2

Removed libxml2-2.9.10-hee79883_0.tar.bz2

Removed libprotobuf-3.8.0-h8b12597_0.tar.bz2

Removed msgpack-python-1.0.0-py37h99015e2_1.tar.bz2

Removed packaging-20.4-pyh9f0ad1d_0.tar.bz2

Removed dask-xgboost-0.2.0.dev28-cuda10.2py37_0.tar.bz2

Removed libnvstrings-0.14.0-cuda10.2_0.tar.bz2

Removed ncurses-6.2-he1b5a44_1.tar.bz2

Removed xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0.tar.bz2

Removed pygments-2.6.1-py_0.tar.bz2

Removed sqlalchemy-1.3.18-py37h8f50634_0.tar.bz2

Removed pyct-core-0.4.6-py_0.tar.bz2

Removed freexl-1.0.5-h516909a_1002.tar.bz2

Removed librmm-0.14.0-cuda10.2_0.tar.bz2

Removed ipykernel-5.3.4-py37h43977f1_0.tar.bz2

Removed postgresql-12.2-hf1211e9_0.tar.bz2

Removed appdirs-1.4.3-py_1.tar.bz2

Removed cuml-0.14.0-cuda10.2_py37_0.tar.bz2

Removed expat-2.2.9-he1b5a44_2.tar.bz2

Removed colorcet-2.0.1-py_0.tar.bz2

Removed pandocfilters-1.4.2-py_1.tar.bz2

Removed parso-0.7.1-pyh9f0ad1d_0.tar.bz2

Removed libsodium-1.0.18-h516909a_0.tar.bz2

Removed _openmp_mutex-4.5-1_gnu.tar.bz2

Removed gtest-1.10.0-hc9558a2_4.tar.bz2

Removed thrift-cpp-0.12.0-hf3afdfd_1004.tar.bz2

Removed distributed-2.22.0-py37hc8dfbb8_0.tar.bz2

Removed pyarrow-0.15.0-py37h8b68381_1.tar.bz2

Removed brotli-1.0.7-he1b5a44_1004.tar.bz2

Removed libgdal-3.0.4-h3dfc09a_6.tar.bz2

Removed wheel-0.34.2-py_1.tar.bz2

Removed fastrlock-0.5-py37h3340039_0.tar.bz2

Removed cupy-7.7.0-py37h940342b_0.tar.bz2

Removed libgcc-ng-9.3.0-h24d8f2e_14.tar.bz2

Removed param-1.9.3-py_0.tar.bz2

Removed libuuid-2.32.1-h14c3975_1000.tar.bz2

Removed nvstrings-0.14.0-py37_0.tar.bz2

Removed libcumlprims-0.14.1-cuda10.2_0.tar.bz2

Removed six-1.15.0-pyh9f0ad1d_0.tar.bz2

Removed python-3.7.8-h6f2ec95_1_cpython.tar.bz2

Removed rapids-0.14.1-cuda10.2_py37_0.tar.bz2

Removed cudnn-7.6.5-cuda10.2_0.conda

Removed imagecodecs-lite-2019.12.3-py37h03ebfcd_1.tar.bz2

Removed imageio-2.9.0-py_0.tar.bz2

Removed libedit-3.1.20191231-h46ee950_1.tar.bz2

Removed xorg-libsm-1.2.3-h84519dc_1000.tar.bz2

Removed cffi-1.14.1-py37h2b28604_0.tar.bz2

Removed openjdk-8.0.192-h516909a_1005.tar.bz2

Removed libcugraph-0.14.0-cuda10.2_0.tar.bz2

Removed nbformat-5.0.7-py_0.tar.bz2

Removed parquet-cpp-1.5.1-2.tar.bz2

Removed libntlm-1.4-h516909a_1002.tar.bz2

Removed libspatialindex-1.9.3-he1b5a44_3.tar.bz2

Removed libcuspatial-0.14.0-cuda10.2_0.tar.bz2

Removed psutil-5.7.2-py37h8f50634_0.tar.bz2

Removed jupyter-server-proxy-1.5.0-py_0.tar.bz2

Removed xorg-libx11-1.6.11-h516909a_0.tar.bz2

Removed pywavelets-1.1.1-py37h03ebfcd_1.tar.bz2

Removed jinja2-2.11.2-pyh9f0ad1d_0.tar.bz2

Removed decorator-4.4.2-py_0.tar.bz2

Removed scikit-learn-0.23.2-py37h6785257_0.tar.bz2

Removed pyviz_comms-0.7.6-pyh9f0ad1d_0.tar.bz2

Removed cugraph-0.14.0-py37_0.tar.bz2

Removed datashape-0.5.4-py_1.tar.bz2

Removed partd-1.1.0-py_0.tar.bz2

Removed tifffile-2020.6.3-py_0.tar.bz2

Removed xorg-libxext-1.3.4-h516909a_0.tar.bz2

Removed dask-cudf-0.14.0-py37_0.tar.bz2

Removed gflags-2.2.2-he1b5a44_1004.tar.bz2

Removed locket-0.2.0-py_2.tar.bz2

Removed fiona-1.8.13-py37h0492a4a_1.tar.bz2

Removed libssh2-1.9.0-hab1572f_5.tar.bz2

Removed libhdfs3-2.3-h311b756_1006.tar.bz2

Removed tblib-1.6.0-py_0.tar.bz2

Removed py-xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0.tar.bz2

Removed gdal-3.0.4-py37h4b180d9_6.tar.bz2

Removed xz-5.2.5-h516909a_1.tar.bz2

Removed jupyter_client-6.1.6-py_0.tar.bz2

Removed krb5-1.16.4-h2fd8d38_0.tar.bz2

Removed matplotlib-base-3.3.0-py37hd478181_1.tar.bz2

Removed scikit-image-0.17.2-py37h0da4684_1.tar.bz2

Removed libffi-3.2.1-he1b5a44_1007.tar.bz2

Removed shapely-1.7.0-py37hc88ce51_3.tar.bz2

Removed pyyaml-5.3.1-py37h8f50634_0.tar.bz2

Removed xarray-0.16.0-py_0.tar.bz2

Removed fastavro-0.24.0-py37h8f50634_0.tar.bz2

Removed zipp-3.1.0-py_0.tar.bz2

Removed tqdm-4.48.2-pyh9f0ad1d_0.tar.bz2

Removed attrs-19.3.0-py_0.tar.bz2

Removed re2-2020.04.01-he1b5a44_0.tar.bz2

Removed toolz-0.10.0-py_0.tar.bz2

Removed pycparser-2.20-pyh9f0ad1d_2.tar.bz2

Removed kealib-1.4.13-hec59c27_0.tar.bz2

Removed defusedxml-0.6.0-py_0.tar.bz2

Removed cyrus-sasl-2.1.27-he38ecfd_0.tar.bz2

Removed rapids-xgboost-0.14.1-cuda10.2_py37_0.tar.bz2

Removed readline-8.0-he28a2e2_2.tar.bz2

Removed olefile-0.46-py_0.tar.bz2

Removed panel-0.6.4-0.tar.bz2

Removed libgfortran-ng-7.5.0-hdf63c60_14.tar.bz2

Removed fontconfig-2.13.1-h86ecdb6_1001.tar.bz2

Removed pyhive-0.6.2-pyh9f0ad1d_0.tar.bz2

Removed ucx-1.8.0+gf6ec8d4-cuda10.2_20.tar.bz2

Removed pixman-0.38.0-h516909a_1003.tar.bz2

Removed liblapack-3.8.0-17_openblas.tar.bz2

Removed entrypoints-0.3-py37hc8dfbb8_1001.tar.bz2

Removed sqlite-3.32.3-hcee41ef_1.tar.bz2

Removed pyparsing-2.4.7-pyh9f0ad1d_0.tar.bz2

Removed libdap4-3.20.4-hd3bb157_0.tar.bz2

Removed importlib-metadata-1.7.0-py37hc8dfbb8_0.tar.bz2

Removed xorg-libxdmcp-1.1.3-h516909a_0.tar.bz2

Removed xorg-libice-1.0.10-h516909a_0.tar.bz2

Removed setuptools-49.2.1-py37hc8dfbb8_0.tar.bz2

Removed pandas-0.25.3-py37hb3f55d8_0.tar.bz2

Removed future-0.18.2-py37hc8dfbb8_1.tar.bz2

Removed cuspatial-0.14.0-py37_0.tar.bz2

Removed cycler-0.10.0-py_2.tar.bz2

Removed hdf5-1.10.5-nompi_h3c11f04_1104.tar.bz2

Removed libhwloc-2.1.0-h3c4fd83_0.tar.bz2

Removed scipy-1.5.2-py37hb14ef9d_0.tar.bz2

Removed json-c-0.13.1-hbfbb72e_1002.tar.bz2

Removed spdlog-1.7.0-hc9558a2_2.tar.bz2

Removed xorg-libxau-1.0.9-h14c3975_0.tar.bz2

Removed pynvml-8.0.4-py_1.tar.bz2

Removed aiohttp-3.6.2-py37h516909a_0.tar.bz2

Removed zict-2.0.0-py_0.tar.bz2

Removed curl-7.68.0-hf8cf82a_0.tar.bz2

Removed lcms2-2.11-hbd6801e_0.tar.bz2

Removed libstdcxx-ng-9.3.0-hdf63c60_14.tar.bz2

Removed zstd-1.4.4-h3b9ef0a_2.tar.bz2

Removed xerces-c-3.2.2-h8412b87_1004.tar.bz2

Removed cppzmq-4.6.0-hc9558a2_0.tar.bz2

Removed click-7.1.2-pyh9f0ad1d_0.tar.bz2

Removed pyrsistent-0.16.0-py37h8f50634_0.tar.bz2

Removed libpq-12.2-hae5116b_0.tar.bz2

Removed cusignal-0.14.0-py37_0.tar.bz2

Removed ca-certificates-2020.6.20-hecda079_0.tar.bz2

Removed pyct-0.4.6-py_0.tar.bz2

Removed wcwidth-0.2.5-pyh9f0ad1d_1.tar.bz2

Removed joblib-0.16.0-py_0.tar.bz2

Removed thrift-0.13.0-py37h3340039_1.tar.bz2

Removed libtiff-4.1.0-hc7e4089_6.tar.bz2

Removed pytz-2020.1-pyh9f0ad1d_0.tar.bz2

Removed terminado-0.8.3-py37hc8dfbb8_1.tar.bz2

Removed libllvm8-8.0.1-hc9558a2_0.tar.bz2

Removed uriparser-0.9.3-he1b5a44_1.tar.bz2

Removed backports.functools_lru_cache-1.6.1-py_0.tar.bz2

Removed netifaces-0.10.9-py37h8f50634_1002.tar.bz2

Removed pthread-stubs-0.4-h14c3975_1001.tar.bz2

Removed libuv-1.34.0-h516909a_0.tar.bz2

Removed multipledispatch-0.6.0-py_0.tar.bz2

Removed libnetcdf-4.7.4-nompi_h9f9fd6a_101.tar.bz2

Removed zeromq-4.3.2-he1b5a44_3.tar.bz2

Removed geotiff-1.5.1-h05acad5_10.tar.bz2

Removed prometheus_client-0.8.0-pyh9f0ad1d_0.tar.bz2

Removed pyppeteer-0.0.25-py_1.tar.bz2

Removed datashader-0.10.0-py_0.tar.bz2

Removed send2trash-1.5.0-py_0.tar.bz2

Removed tzcode-2020a-h516909a_0.tar.bz2

Removed pysocks-1.7.1-py37hc8dfbb8_1.tar.bz2

Removed libcudf-0.14.0-cuda10.2_0.tar.bz2

Removed xorg-xextproto-7.3.0-h14c3975_1002.tar.bz2

Removed tiledb-1.7.7-hcde45ca_0.tar.bz2

Removed libspatialite-4.3.0a-h2482549_1038.tar.bz2

Removed python_abi-3.7-1_cp37m.tar.bz2

Removed testpath-0.4.4-py_0.tar.bz2

Removed click-plugins-1.1.1-py_0.tar.bz2

Removed pyopenssl-19.1.0-py_1.tar.bz2

Removed cudf-0.14.0-py37_0.tar.bz2

Removed icu-64.2-he1b5a44_1.tar.bz2

Removed libgcrypt-1.8.4-hf484d3e_1000.tar.bz2

Removed prompt-toolkit-3.0.5-py_1.tar.bz2

Removed cudatoolkit-10.2.89-h6bb024c_0.tar.bz2

Removed grpc-cpp-1.23.0-h18db393_0.tar.bz2

Removed webencodings-0.5.1-py_1.tar.bz2

Removed xorg-libxrender-0.9.10-h516909a_1002.tar.bz2

Removed cligj-0.5.0-py_0.tar.bz2

Removed pyzmq-19.0.2-py37hac76be4_0.tar.bz2

Removed pexpect-4.8.0-py37hc8dfbb8_1.tar.bz2

Removed jupyter_core-4.6.3-py37hc8dfbb8_1.tar.bz2

Removed freetype-2.10.2-he06d7ca_0.tar.bz2

Removed cuxfilter-0.14.0-py37_0.tar.bz2

Removed jpype1-1.0.2-py37h99015e2_0.tar.bz2

Removed zlib-1.2.11-h516909a_1006.tar.bz2

Removed importlib_metadata-1.7.0-0.tar.bz2

Removed backports-1.0-py_2.tar.bz2

Removed _libgcc_mutex-0.1-conda_forge.tar.bz2

Removed double-conversion-3.1.5-he1b5a44_2.tar.bz2

WARNING: /root/.conda/pkgs does not exist

Cache location: /opt/conda/pkgs

Will remove the following packages:

/opt/conda/pkgs

---------------



rapids-0.14.1-cuda10.2_py37_0                 12 KB

python_abi-3.7-1_cp37m                        10 KB

importlib_metadata-1.7.0-0                     8 KB

_libgcc_mutex-0.1-conda_forge                  6 KB

parquet-cpp-1.5.1-2                            8 KB

rapids-xgboost-0.14.1-cuda10.2_py37_0          9 KB

bsql-toolchain-0.14.0-0                       17 KB

libcblas-3.8.0-17_openblas                    38 KB

dask-2.22.0-py_0                              12 KB

libblas-3.8.0-17_openblas                     39 KB

liblapack-3.8.0-17_openblas                   38 KB

xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0      37 KB

pyct-0.4.6-py_0                                7 KB

pthread-stubs-0.4-h14c3975_1001               12 KB

_openmp_mutex-4.5-1_gnu                       93 KB



---------------------------------------------------

Total:                                       347 KB



removing rapids-0.14.1-cuda10.2_py37_0

removing python_abi-3.7-1_cp37m

removing importlib_metadata-1.7.0-0

removing _libgcc_mutex-0.1-conda_forge

removing parquet-cpp-1.5.1-2

removing rapids-xgboost-0.14.1-cuda10.2_py37_0

removing bsql-toolchain-0.14.0-0

removing libcblas-3.8.0-17_openblas

removing dask-2.22.0-py_0

removing libblas-3.8.0-17_openblas

removing liblapack-3.8.0-17_openblas

removing xgboost-1.1.0dev.rapidsai0.14-cuda10.2py37_0

removing pyct-0.4.6-py_0

removing pthread-stubs-0.4-h14c3975_1001

removing _openmp_mutex-4.5-1_gnu

STEP 4: COMMIT e49ee83ce0





==> WARNING: A newer version of conda exists. <==

  current version: 4.8.2

  latest version: 4.8.3



Please update conda by running



    $ conda update -n base -c defaults conda





Getting image source signatures

Copying blob sha256:f2cb0ecef392f2a630fa1205b874ab2e2aedf96de04d0b8838e4e728e28142da

Copying blob sha256:875120aa853cf59c6c5bc24af9f448a55f9b64db0bab58c9ee18f8a92ed8ac33

Copying blob sha256:fcd8d39597dd39d0c68670479e4d240fa9dba04a1246587384df9e1aa31b17d4

Copying blob sha256:6cd19a4f6e7a004acd117342e7f1972ded4cb29be18b8624f00eb3ba07cc5a81

Copying config sha256:bff516e990195910efb4983a7390d6fedf180718eaff26b77f2d9b67ebb0b212

Writing manifest to image destination

Storing signatures

level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: write /opt/conda/bin/psql: no space left on device"

error committing container for step {Env:[PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 LC_ALL=C.UTF-8] Command:run Args:[conda env update -n base -f environment.yml && conda clean --all -y] Flags:[] Attrs:map[] Message:RUN conda env update -n base -f environment.yml && conda clean --all -y Original:RUN conda env update -n base -f environment.yml && conda clean --all -y}: error copying layers and metadata for container "18f1241f314ec9557ec8c8e3554bb97c9984967030e726d0537af6c4ed1ce3c7": Error committing the finished image: error adding layer with blob "sha256:6cd19a4f6e7a004acd117342e7f1972ded4cb29be18b8624f00eb3ba07cc5a81": ApplyLayer exit status 1 stdout:  stderr: write /opt/conda/bin/psql: no space left on device

@quasiben
Copy link
Author

I don't suppose that the python command on the path points to the right executable, does it?

Yes, I think it is though the docker image, by default starts a jupyter lab session as well:
https://github.com/rapidsai/docker/blob/branch-0.15/context/.run_in_rapids.sh

@jacobtomlinson and i were looking to. change some of the behaviors around Jupyterlab moving forward: rapidsai/docker#133

Any progress on the space issue ?

@jacobtomlinson
Copy link

The container currently activates the correct conda environment as part of the entrypoint. So if you run the container with a custom command then python should be the correct one.

@mrocklin
Copy link
Member

@jrbourbeau is working this afternoon on generalizing our use of dask-worker to make this customizable. We should be dask-cuda-worker friendly within a day or two.

I'm curious, have folks been able to get things running today with dask-worker? @quasiben I think that you had this up at some point. If you have a cluster configuration that I could start up I would welcome that.

cc also @sheer-coiled who might be interested in this topic.

@mrocklin
Copy link
Member

Also, timing wise, I have a goal of having things up and running smoothly by next Tuesday, when I'm on the hook to give a multi-gpu RAPIDS talk and mini-tutorial with RAPIDS-Academy (we bumped it from tomorrow to next week).

As a stretch goal, I think that @TomAugspurger is participating in a Coiled live stream with Hugo on Thursday on machine learning. I'm not sure how much he's planning to talk about GPU stuff, but if we made this easy for him we might tempt him into it :)

@quasiben
Copy link
Author

I tried creating an env this morning and I'm still seeing no space left on device. However, the env was created and the docker image moved onto cleanup after the install step

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020

@quasiben just confirming this was on beta.coiledhq.com?

@quasiben
Copy link
Author

quasiben commented Aug 12, 2020 via email

@quasiben
Copy link
Author

On another coiled endpoint I was successful in building an env there cc @mrocklin

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020

@quasiben I've pushed an update to the infrastructure so the no space left on device issue is hopefully vanquished.

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020 •
edited
Loading

I have a cluster configuration necaris/gputest on beta.coiledhq.com that will consistently launch. However, it's not able to run GPU code:

ImportError: CuPy is not correctly installed.

If you are using wheel distribution (cupy-cudaXX), make sure that the version of CuPy you installed matches with the version of CUDA on your host.
Also, confirm that only one CuPy package is installed:
  $ pip freeze

If you are building CuPy from source, please check your environment, uninstall CuPy and reinstall it with:
  $ pip install cupy --no-cache-dir -vvvv

Check the Installation Guide for details:
  https://docs.cupy.dev/en/latest/install.html

original error: libcuda.so.1: cannot open shared object file: No such file or directory

Software environment details:

{
  "channels": [
    "rapidsai",
    "nvidia",
    "conda-forge"
  ],
  "dependencies": [
    "cudatoolkit=10.1",
    "cudf",
    "cupy",
    "dask",
    "distributed",
    "pandas",
    "python=3.7"
  ]
}

Output of client.run(lambda: subprocess.check_output(['nvidia-smi'])):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   39C    P0    25W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Output of docker inspect on the container (with the TLS details snipped out):

[
    {
        "Id": "aad0062158911ed3f244ad55b9d36d87e5b3357960c120ea2dadee20c3ac6c39",
        "Created": "2020-08-12T16:09:32.709328463Z",
        "Path": "python",
        "Args": [
            "-m",
            "distributed.cli.dask_spec",
            "--spec",
            "{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"death_timeout\": \"60 seconds\", \"security\": {\"tls_ca_file\": \" \", \"tls_worker_key\": \" \", \"tls_worker_cert\": \" \"}}}"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-08-12T16:09:55.100729755Z",
            "FinishedAt": "2020-08-12T16:14:53.793583622Z"
        },
        "Image": "sha256:30aa34e35c5e71194a6fa784c8f288b219b9f805ba9bc14320c0426f7ad423a1",
        "ResolvConfPath": "/var/lib/docker/containers/d37d21ed714415973db591b6e3d5e5321df3bd5b04db2ef382a5eae432f999d7/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/d37d21ed714415973db591b6e3d5e5321df3bd5b04db2ef382a5eae432f999d7/hostname",
        "HostsPath": "/var/lib/docker/containers/d37d21ed714415973db591b6e3d5e5321df3bd5b04db2ef382a5eae432f999d7/hosts",
        "LogPath": "",
        "Name": "/ecs-necaris-necaris-ba0c99ae-4-worker-1-dask-worker-9cdbf4f7efd0e29ee901",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "awslogs",
                "Config": {
                    "awslogs-create-group": "true",
                    "awslogs-credentials-endpoint": "/v2/credentials/b11ce8a5-25f7-47fa-aa8f-ef584a388e7e",
                    "awslogs-group": "necaris",
                    "awslogs-region": "us-east-2",
                    "awslogs-stream": "necaris-necaris-ba0c99ae-4-worker/dask-worker/f1f4ffb3bea341239dceb669a8848eee"
                }
            },
            "NetworkMode": "container:d37d21ed714415973db591b6e3d5e5321df3bd5b04db2ef382a5eae432f999d7",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": [],
            "CapDrop": [],
            "Capabilities": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "nvidia",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 8192,
            "Memory": 62277025792,
            "NanoCpus": 0,
            "CgroupParent": "/ecs/f1f4ffb3bea341239dceb669a8848eee",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 62277025792,
            "MemorySwap": 124554051584,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": [
                {
                    "Name": "nofile",
                    "Hard": 4096,
                    "Soft": 1024
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/4d16b73ba2f4dc30dc203975e3891b8fad671aa685d720a2e60abb4ba4882437-init/diff:/var/lib/docker/overlay2/3babeafe89f5396e24c1daa22b1a47cefdb694b1abead0e46a574b74f170b3a3/diff:/var/lib/docker/overlay2/4a224778196efd94a05012f35dbf61d8940581cf787c01bc6cf36361a89de4ed/diff:/var/lib/docker/overlay2/b742cdc99149eaf9a47b7ac7524b07a6e3d24dd7e91da8063fc26e517144f602/diff:/var/lib/docker/overlay2/daec24de05f77aa921c5781d1bbf5be1b6c15eec270aee51596c0ce7a9771c08/diff",
                "MergedDir": "/var/lib/docker/overlay2/4d16b73ba2f4dc30dc203975e3891b8fad671aa685d720a2e60abb4ba4882437/merged",
                "UpperDir": "/var/lib/docker/overlay2/4d16b73ba2f4dc30dc203975e3891b8fad671aa685d720a2e60abb4ba4882437/diff",
                "WorkDir": "/var/lib/docker/overlay2/4d16b73ba2f4dc30dc203975e3891b8fad671aa685d720a2e60abb4ba4882437/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "ip-10-1-11-29.us-east-2.compute.internal",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NVIDIA_VISIBLE_DEVICES=GPU-3c4538b4-cf76-93eb-f2a4-c177171049b0",
                "ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/16eb5756-1194-41dd-9d3c-7f04d9d4618a",
                "ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v4/16eb5756-1194-41dd-9d3c-7f04d9d4618a",
                "AWS_EXECUTION_ENV=AWS_ECS_EC2",
                "DASK_SCHEDULER_ADDRESS=tls://ip-10-1-11-50.us-east-2.compute.internal:8786",
                "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/256db1f9-d7b3-419e-b629-a671e74da494",
                "PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LANG=C.UTF-8",
                "LC_ALL=C.UTF-8"
            ],
            "Cmd": [
                "python",
                "-m",
                "distributed.cli.dask_spec",
                "--spec",
                "{\"cls\": \"dask.distributed.Nanny\", \"opts\": {\"death_timeout\": \"60 seconds\", \"security\": {\"tls_ca_file\": \"\", \"tls_worker_key\": \"\", \"tls_worker_cert\": \"\"}}}"
            ],
            "Image": "769926636128.dkr.ecr.us-east-2.amazonaws.com/dev/necaris-gputest:3562f15cc0",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.amazonaws.ecs.cluster": "dev",
                "com.amazonaws.ecs.container-name": "dask-worker",
                "com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-east-2:769926636128:task/dev/f1f4ffb3bea341239dceb669a8848eee",
                "com.amazonaws.ecs.task-definition-family": "necaris-necaris-ba0c99ae-4-worker",
                "com.amazonaws.ecs.task-definition-version": "1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]

Output of nvidia-smi on the host:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   36C    P0    24W / 300W |      0MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Host OS, etc:

❯ aws ec2 describe-images --image-ids ami-0733efab97426efd7
{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2020-08-05T20:35:00.000Z",
            "ImageId": "ami-0733efab97426efd7",
            "ImageLocation": "amazon/amzn2-ami-ecs-gpu-hvm-2.0.20200805-x86_64-ebs",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "591542846629",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-07a4855bd8bf8b4af",
                        "VolumeSize": 30,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                }
            ],
            "Description": "Amazon Linux AMI 2.0.20200805 x86_64 ECS HVM GP2",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "amzn2-ami-ecs-gpu-hvm-2.0.20200805-x86_64-ebs",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm"
        }
    ]
}

$ uname -a
Linux ip-10-1-11-216.us-east-2.compute.internal 4.14.186-146.268.amzn2.x86_64 #1 SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

@mrocklin
Copy link
Member 

mrocklin@carbon-7:~$ coiled env inspect necaris/gputest
container:
None


conda:
{'channels': ['rapidsai', 'nvidia', 'conda-forge'],
 'dependencies': ['cudatoolkit=10.1',
                  'cudf',
                  'cupy',
                  'dask',
                  'distributed',
                  'pandas',
                  'python=3.7']}

It looks like you're using cudatoolkit 10.1 in the conda environment, but that maybe the CUDA drivers aren't accessible from the docker image?

Output of client.run(lambda: subprocess.check_output(['nvidia-smi'])):

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: N/A      |

I wonder if we might need a different docker image base?

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020

@mrocklin that seems like a possibility! I've created another cluster configuration necaris/gputest-rapids using a container-based software environment (necaris/rapids-docker using the rapidsai/rapidsai:0.14-cuda10.1-base-ubuntu18.04-py3.7 image) and it's doing basic GPU things:

In [7]: future = client.submit(cupy.ones, 1000)

In [8]: future = client.submit(cupy.sum, future)

In [9]: future.result()
Out[9]: array(1000.)

@selshowk
Copy link

@necaris I guess you could just switch all images to be built of the nvidia cuda base image (is the current base image based off ubuntu or the python docker env)? I'll probably hit into this soon with the stuff I'm doing (its pretty much my next step) so let me know if you'd like me to help in any way.

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020

@sheer-coiled the current base image is based on continuumio/miniconda which I believe is based on Debian -- after talking to @quasiben it sounds like looking into nvidia/cuda is a logical next step for this too.

@mrocklin
Copy link
Member

mrocklin commented Aug 12, 2020 via email

@quasiben
Copy link
Author

I just tried this an coiled indeed does install into base and I want to install into rapids

In [28]: coiled.create_software_environment(name="rapidsai/blazingsql",container="rapidsai/rapidsai:0.14-cuda10.2-runtime-ubuntu18.04-p
    ...: y3.7",
    ...:     conda={
    ...:         "channels": ["blazingsql/label/cuda10.2", "blazingsql",
    ...:         "rapidsai", "nvidia", "conda-forge", "defaults"],
    ...:         "dependencies": ["blazingsql", "s3fs"],
    ...:     },
    ...: )
Updating software environment...
Solving conda environment...
Conda environment solved!
Building Docker image
(this takes a few minutes)
STEP 1: FROM rapidsai/rapidsai:0.14-cuda10.2-runtime-ubuntu18.04-py3.7
STEP 2: COPY environment.yml environment.yml
STEP 3: RUN conda env update -n base -f environment.yml && conda clean --all -y
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

@jrbourbeau
Copy link
Member

I just tried this an coiled indeed does install into base and I want to install into rapids

Thanks @quasiben, we'll work to expose the name of the conda environment which conda env update uses

@necaris
Copy link
Contributor

necaris commented Aug 12, 2020

@quasiben would it make sense to create a "base" software environment on beta.coiledhq.com based on nvidia/cuda that we know works for the Coiled environment right now? Rather than having to build a custom Docker image and use that?

@mrocklin
Copy link
Member

mrocklin commented Aug 12, 2020 via email

@jacobtomlinson
Copy link

Thanks for the cc. Anything I can do in particular to help here?

@mrocklin
Copy link
Member

Is there a combination docker image + conda/pip install + postbuild command that would get us to close to the RAPIDS docker image, but starting from a more barebones miniconda install? If so, constructing a software environment on Coiled around that would be useful.

coiled.create_software_environment(
    name="rapidsai/default",
    container="...",
    conda={"channels": [...], "dependencies": [...]},
    post_build=[...]
)

@jrbourbeau is the PR that would allow for use of CUDAWorker in a released version yet for folks to play with?

@mrocklin
Copy link
Member

For context, starting from the rapids base image is awkward because

  1. Folks locally won't be able to reproduce it on their machines without docker
  2. It uses a different conda environment, which we aren't currently set up for

And the miniconda base image won't work because

  1. It doesn't have cuda magic.

So @necaris 's question was "Hey, could we start from nvidia/cuda and go from there with conda?

@jacobtomlinson
Copy link

jacobtomlinson commented Aug 13, 2020 •
edited
Loading

You could definitely start from nvidia/cuda or the debian based continuumio/miniconda and install the missing pieces. It may help to trace all the image layers back to see what gets installed where. I think the RAPIDS image is built with the following layers:

I imagine to build all images from nvidia/cuda onwards you would need to be running NVIDIA Docker on a GPU powered build machine. Which is one of the main reasons why we recommend folks use the RAPIDS images.

@necaris
Copy link
Contributor

necaris commented Aug 13, 2020

Also, to be clear, I'm happy to dig into this and try and build that image, but I suspect that with your deeper insight into the cuda magic it'll take you much less time to come up with something workable.

@jacobtomlinson
Copy link

@necaris I suspect the main blocker here will be running the NVIDIA Docker runtime in order to build these images.

I expect the process will be to add the NVIDIA apt repo, install the drivers with apt, install miniconda with the shell script, install RAPIDS with conda.

@mrocklin
Copy link
Member

mrocklin commented Aug 13, 2020 via email

@necaris
Copy link
Contributor

necaris commented Aug 13, 2020

@jacobtomlinson from your breakdown of the layers it seems like we could maybe start from gpuci/miniconda-cuda and go from there. Is that image available somewhere, or do we have to duplicate it?

@jrbourbeau
Copy link
Member

is the PR that would allow for use of CUDAWorker in a released version yet for folks to play with?

It's not yet in a released version. I was planning to release tomorrow (Friday) after the distributed release, but we're in a release-able state now if it'd be useful to push something out for others use today

@jacobtomlinson
Copy link

jacobtomlinson commented Aug 13, 2020 •
edited
Loading

@mrocklin I think you might be right. As long as you start from an image where all the GPU stuff is done you should be fine. Just note that tools like nvidia-smi are not available in any upstream image unless run with the NVIDIA Docker runtime. @quasiben got stuck in that hole recently trying to debug the image when it turned out to be the runtime.

@necaris AFAIK that image should be available on Docker Hub, so you can build directly from it.

FROM gpuci/miniconda-cuda

RUN ...

@mrocklin
Copy link
Member

mrocklin commented Aug 13, 2020 via email

@necaris
Copy link
Contributor

necaris commented Aug 13, 2020

@jacobtomlinson we're reasonably sure we've ironed out the issues with ensuring the container is run with the right runtime 😄 so I'm looking into gpuci/miniconda-cuda now...

@jrbourbeau
Copy link
Member

Yes, this will be a coiled release + re-deploy to beta

@jacobtomlinson
Copy link

Yeah I'm sure you're doing the right thing when it comes to running the container. The issue Ben and I had was when debugging something from a different machine/environment and thinking "oh I think the image is broken" when actually our debugging environment wasn't representative. Having binaries not exist when run with a different runtime can feel weird because usually everything is contained in the image. I just wanted to flag the trap more than anything.

@necaris
Copy link
Contributor

necaris commented Aug 13, 2020

@jacobtomlinson thanks for the extra warning! I banged my head against it a few times recently so I'm still cautious around the runtime (and the question of how exactly AWS passes the environment and arguments to Docker, too).

@jrbourbeau
Copy link
Member

Just wanted to check in on this issue, do have a software environment for deploying rapids today?

@necaris
Copy link
Contributor

necaris commented Aug 20, 2020

@jrbourbeau on beta, dev/gpu-play deploys cupy and cudf, but I don't know about the rest of the RAPIDS stack.

@mrocklin
Copy link
Member

mrocklin commented Aug 20, 2020 via email

@quasiben
Copy link
Author

quasiben commented Sep 2, 2020

Should we try another round of testing ?

@necaris
Copy link
Contributor

necaris commented Sep 2, 2020

@quasiben I think we're in a good place for it :-)

@quasiben
Copy link
Author

quasiben commented Sep 3, 2020

I tried on beta on get the following error:

In [7]: cluster = coiled.Cluster(n_workers=1, configuration="quasiben/rapids-015-test")
Creating Cluster. This takes about a minute .../Checking environment images
Valid environment image found
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-776e99d26c6a> in <module>
----> 1 cluster = coiled.Cluster(n_workers=1, configuration="quasiben/rapids-015-test")

~/miniconda3/envs/coiled/lib/python3.7/site-packages/coiled/cluster.py in __init__(self, n_workers, configuration, name, asynchronous, cloud, account, shutdown_on_close, region)
     76
     77         if not self.asynchronous:
---> 78             self.sync(self._start)
     79
     80     @property

~/miniconda3/envs/coiled/lib/python3.7/site-packages/distributed/deploy/cluster.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    172             return future
    173         else:
--> 174             return sync(self.loop, func, *args, **kwargs)
    175
    176     def _log(self, log):

~/miniconda3/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    337     if error[0]:
    338         typ, exc, tb = error[0]
--> 339         raise exc.with_traceback(tb)
    340     else:
    341         return result[0]

~/miniconda3/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in f()
    321             if callback_timeout is not None:
    322                 future = asyncio.wait_for(future, callback_timeout)
--> 323             result[0] = yield future
    324         except Exception as exc:
    325             error[0] = sys.exc_info()

~/miniconda3/envs/coiled/lib/python3.7/site-packages/tornado/gen.py in run(self)
    733
    734                     try:
--> 735                         value = future.result()
    736                     except Exception:
    737                         exc_info = sys.exc_info()

~/miniconda3/envs/coiled/lib/python3.7/site-packages/coiled/cluster.py in _start(self)
    128
    129         self.security, info = await self.cloud.security(
--> 130             cluster_id=self.cluster_id, account=self.account  # type: ignore
    131         )
    132

~/miniconda3/envs/coiled/lib/python3.7/site-packages/coiled/core.py in _security(self, cluster_id, account)
    432                 # cluster is stopped, probably very shortly after starting
    433                 raise ValueError(
--> 434                     "Unable to get security info, cluster status is unexpectedly STOPPED"
    435                 )
    436             if data["status"] != "pending" and data["public_address"]:

ValueError: Unable to get security info, cluster status is unexpectedly STOPPED

@mrocklin
Copy link
Member

mrocklin commented Sep 3, 2020 via email

@quasiben
Copy link
Author

quasiben commented Sep 3, 2020

I can't do cluster.get_logs() because the cluster did not get created.

Happy to spend some time with Rami. Let's coordinate over email

@quasiben
Copy link
Author

@necaris and I were able to get things working now with Rapids Docker Images. We needed to fix a small issue in dask-cuda (thanks @necaris and @jrbourbeau ) and we also need to add a set a few things create_cluster_configuration (shown below). Two things which would help ergonomics:

  1. Running code remotely: Run local code remotely on a worker dask/distributed#4003
  2. Better output support (return/prints are little awkward if you don't have a GPU locally)
import coiled

coiled.create_software_environment(name="rapids-docker-016-nightly",
                                   container="rapidsai/rapidsai-nightly:0.16-cuda10.2-base-ubuntu18.04")

coiled.create_cluster_configuration(name='rapids-docker-example',
                                    worker_memory='15GiB',
                                    software='quasiben/rapids-docker-016-nightly',
                                    worker_options={'rmm_pool_size': '12GiB'},
                                    worker_class='dask_cuda.CUDAWorker',
                                    worker_gpu=1)

cluster = coiled.Cluster(n_workers=1, configuration="quasiben/rapids-docker-example")

from dask.distributed import Client
import dask.dataframe as dd

coiled.create_cluster_configuration(name='rapids-docker-example',
                                    worker_memory='15GiB',
                                    software='quasiben/rapids-docker-016-nightly',
                                    worker_options={'rmm_pool_size': '12GiB'},
                                    worker_class='dask_cuda.CUDAWorker',
                                    worker_gpu=1)
cluster = coiled.Cluster(n_workers=1, configuration="quasiben/rapids-docker-example")


# nvidia-smi from cli
def nvida_smi():
    import subprocess
    return subprocess.check_output(['nvidia-smi'])

for w, res in client.run(nvida_smi).items():
    print(w, ":")
    print(res.decode('utf8'))

#nvidia-smi from pynvml
def gpu_mem():
    from pynvml.smi import nvidia_smi
    nvsmi = nvidia_smi.getInstance()
    return nvsmi.DeviceQuery('memory.free, memory.total')

for w, res in client.run(gpu_mem).items():
    print(w, ":")
    print(res)

def tips():
    import cudf, io, requests
    from io import StringIO
    url = "https://github.com/plotly/datasets/raw/master/tips.csv"
    content = requests.get(url).content.decode('utf-8')
    tips_df = cudf.read_csv(StringIO(content))
    tips_df['tip_percentage'] = tips_df['tip'] / tips_df['total_bill'] * 100
    # display average tip by dining party size
    return str(tips_df.groupby('size').tip_percentage.mean())

for w, res in client.run(tips).items():
    print(w, ":")
    print(res)

def dask_cudf():
    import cudf
    cdf = cudf.datasets.timeseries()
    ddf = dd.from_pandas(cdf, npartitions=5)
    return ddf.groupby('name').agg(['mean', 'max', 'min']).compute().to_pandas()

for w, res in client.run(dask_cudf).items():
    print(w, ":")
    print(res)

I think we are now functional and now we need to document/experiment -- I'm +1 on closing. Thought?

@mrocklin
Copy link
Member

This is very exciting! Thanks for putting this together @quasiben . I look forward to playing with it. Some thoughts:

  1. Any thoughts on claiming the rapidsai/nightly spot on Coiled for the RAPIDS nightly container?
  2. We might consider also using this with jobs or notebooks (cc @jrbourbeau) due to the lack of a local GPU

I'm inclined to keep this open until we have a doc page with an easy onramp for new users

@necaris
Copy link
Contributor

necaris commented Sep 14, 2020

I'm inclined to keep this open until we have a doc page with an easy onramp for new users

👍 from me

@mrocklin
Copy link
Member

I've put a rapidsai/nightly software environment and cluster configuration in the rapidsai account. So anyone should be able to do

import coiled
cluster = coiled.Cluster(configuration="rapidsai/nightly", account=...)

(where ... is some account with GPU access).

I started setting up an example to use this with Optuna (I was trying to play around earlier cc @jrbourbeau ), but needed to add a couple of packages. I did that with the following:

coiled.create_software_environment(
    name="mrocklin/dask-optuna", 
    container="rapidsai/rapidsai-nightly:0.16-cuda10.2-base-ubuntu18.04", 
    pip=["dask-optuna"], 
    conda={"channels": ["rapidsai-nightly", "nvidia", "conda-forge", "defaults"], "dependencies": ["optuna"]}, 
    conda_env_name="rapids"
)

However this appears to have overwritten a large number of conda packages, so I may have screwed something up.

@mrocklin
Copy link
Member

@quasiben if I wanted to have all of the wonder GPU setup of the rapids image, but use my own conda environment, is there some sort of rapids-base docker image that would work?

@necaris
Copy link
Contributor

necaris commented Sep 15, 2020

@mrocklin gpuci/miniconda-cuda:10.1-runtime-ubuntu18.04 might fit your needs? I've used it to test a couple of software environments, e.g. see necaris/gpuci-223

@mrocklin
Copy link
Member

mrocklin commented Sep 15, 2020 via email

@quasiben
Copy link
Author

quasiben commented Sep 15, 2020 •
edited
Loading

+1 for @necaris's suggestion. gpuci/rapidsai:0.16-cuda10.2-runtime-ubuntu18.04-py3.8 is what the rapids images pull from, though I don't know the difference

@shughes-uk
Copy link

shughes-uk commented Mar 10, 2023 •
edited
Loading

Time to close this, we have some functional stuff going. More improvements on the horizon though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mrocklin @necaris @shughes-uk @quasiben @jacobtomlinson @jrbourbeau @selshowk