PySpark Notebook with DeltaLake for production

This repo tries to replicate databricks runtime, plus feature-rich jupyter/docker-stacks.

Base image: rapidsai/rapidsai:22.02-cuda11.5-runtime-ubuntu20.04-py3.8

(Almost) Everything in the jupyter/all-spark-notebook and (eventually) jupyter/r-notebook images, and their ancestor images. (Inheritance tree)
delta-lake and delta-spark

All additional packages from jupyter/r-notebook.
All additional packages that are on top of Databricks runtime dependencies tree (10.3 ML GPU runtime)
xgboost and Spark distribution of xgboost (Waiting for this PR)
hyperopt

Check .env.template for environment variables template, or modify and copy these lines

echo "JUPYTER_PATH=<path-to-notebook-directory>" > .env
echo "NB_UID=`id -u`" >> .env
echo "NB_GID=`id -g`" >> .env

Get path-to-notebook-directory using pwd in the notebook directory

docker-compose up -d