diff --git a/tests/README.md b/tests/README.md index 893df94c2..94d526bab 100644 --- a/tests/README.md +++ b/tests/README.md @@ -63,9 +63,26 @@ GitHub workflows `azureml-unit-tests.yml`, `azureml-cpu-nightly.yml`, `azureml-g There are three scripts used with each workflow, all of them are located in [ci/azureml_tests](./ci/azureml_tests/): -* `submit_groupwise_azureml_pytest.py`: this script uses parameters in the workflow yml to set up the AzureML environment for testing using the AzureML SDK. -* `run_groupwise_pytest.py`: this script uses pytest to run the tests of the libraries and notebooks. This script runs in an AzureML workspace with the environment created by the script above. -* `test_groups.py`: this script defines the groups of tests. If the tests are part of the unit tests, the total compute time of each group should be less than 15min. If the tests are part of the nightly builds, the total time of each group should be less than 35min. +* [`submit_groupwise_azureml_pytest.py`](./ci/azureml_tests/submit_groupwise_azureml_pytest.py): + this script uses parameters in the workflow yml to set up the + AzureML environment for testing using the AzureML SDK. +* [`run_groupwise_pytest.py`](./ci/azureml_tests/run_groupwise_pytest.pyy): + this script uses pytest to run the tests of the libraries and + notebooks. This script runs in an AzureML workspace with the + environment created by the script above. +* [`aml_utils.py`](./ci/azureml_tests/aml_utils.py): this script + defines several utility functions using + [the AzureML Python SDK v2](https://learn.microsoft.com/en-us/azure/machine-learning/concept-v2?view=azureml-api-2). + These fuctions are used by scripts above to set up the compute and + the environment for the tests on AzureML. For example, the + environment with all dependencies of Recommenders is created by the + function `get_or_create_environment` via the [Dockerfile](../tools/docker/Dockerfile). + More details on Docker support can be found at [tools/docker/README.md](../tools/docker/README.md). +* [`test_groups.py`](./ci/azureml_tests/test_groups.py): this script + defines the groups of tests. If the tests are part of the unit + tests, the total compute time of each group should be less than + 15min. If the tests are part of the nightly builds, the total time + of each group should be less than 35min. ## How to contribute tests to the repository diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index edcac0fb7..926d91f28 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -89,7 +89,7 @@ RUN ${CONDA_PREFIX}/bin/conda create -n Recommenders -c conda-forge -y \ FROM deps AS final ARG RECO_LOCAL_DIR="./" -ARG RECO_DIR="/tmp/recommenders" +ARG RECO_DIR="/root/Recommenders" ARG RECO_GIT_URL="git+https://github.com/recommenders-team/recommenders.git" # Copy Recommenders into the image @@ -97,12 +97,14 @@ COPY ${RECO_LOCAL_DIR} ${RECO_DIR} # Install Recommenders and its dependencies RUN source ${CONDA_PREFIX}/bin/activate && \ - conda activate ${ENV_NAME} && \ + conda activate Recommenders && \ if [ -z "${GIT_REF}" ]; then \ pip install ${RECO_DIR}${EXTRAS}; \ else \ pip install recommenders${EXTRAS}@${RECO_GIT_URL}@${GIT_REF}; \ fi && \ + jupyter notebook --generate-config && \ + echo "c.MultiKernelManager.default_kernel_name = 'Recommenders'" >> /root/.jupyter/jupyter_notebook_config.py && \ python -m ipykernel install --user --name Recommenders --display-name "Python (Recommenders)" # Activate Recommenders Conda environment @@ -113,3 +115,6 @@ ENV JAVA_LD_LIBRARY_PATH="${JAVA_HOME}/lib/server" ENV PATH="${CONDA_PREFIX}/envs/Recommenders/bin:${CONDA_PREFIX}/condabin:${PATH}" ENV CONDA_PREFIX="${CONDA_PREFIX}/envs/Recommenders" ENV PS1='(Recommenders) \[\]\[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$ \[\]' + +EXPOSE 8888 +CMD ["jupyter", "notebook", "--ip=0.0.0.0", "--port=8888", "--no-browser", "--allow-root", "--ServerApp.allow_origin='*'", "--IdentityProvider.token=''"] \ No newline at end of file diff --git a/tools/docker/README.md b/tools/docker/README.md index adf5997a7..4c513c7c1 100644 --- a/tools/docker/README.md +++ b/tools/docker/README.md @@ -1,92 +1,115 @@ Docker Support ============== -The Dockerfile in this directory will build Docker images with all the dependencies and code needed to run example notebooks or unit tests included in this repository. +The Dockerfile in this directory will build Docker images with all +the dependencies and code needed to run example notebooks or unit +tests included in this repository. It is also used by +* [.devcontainer/devcontainer.json](../../.devcontainer/devcontainer.json) + to build + [VS Code Dev Contianers](https://code.visualstudio.com/docs/devcontainers/containers) + that can facilitate the development of Recommenders, +* and [tests/ci/azureml_tests/aml_utils.py](../../tests/ci/azureml_tests/aml_utils.py) + to create the environment in [the testing workflows of Recommenders](../../.github/workflows/). -Multiple environments are supported by using [multistage builds](https://docs.docker.com/develop/develop-images/multistage-build/). In order to efficiently build the Docker images in this way, [Docker BuildKit](https://docs.docker.com/develop/develop-images/build_enhancements/) is necessary. -The following examples show how to build and run the Docker image for CPU, PySpark, and GPU environments. +Multiple environments are supported by using +[multistage builds](https://docs.docker.com/build/building/multi-stage/). +The following examples show how to build and run the Docker image for +CPU, PySpark, and GPU environments. -Note: On some platforms, one needs to manually specify the environment variable for `DOCKER_BUILDKIT`to make sure the build runs well. For example, on a Windows machine, this can be done by the powershell command as below, before building the image -``` -$env:DOCKER_BUILDKIT=1 -``` +Once the container is running you can access Jupyter notebooks at +http://localhost:8888. -Warning: On some platforms using Docker Buildkit interferes with Anaconda environment installation. If you find that the docker build is hanging during Anaconda environment setup stage try building the container without Buildkit enabled. - -Once the container is running you can access Jupyter notebooks at http://localhost:8888. Building and Running with Docker -------------------------------- -See examples below for the case of conda. If you use venv or virtualenv instead, replace `--build-arg VIRTUAL_ENV=conda` with `--build-arg VIRTUAL_ENV=venv` or `--build-arg VIRTUAL_ENV=virtualenv`, respectively. -
-CPU environment - -``` -DOCKER_BUILDKIT=1 docker build -t recommenders:cpu --build-arg ENV=cpu --build-arg VIRTUAL_ENV=conda . -docker run -p 8888:8888 -d recommenders:cpu -``` +* **CPU environment** -
+ ```bash + docker build -t recommenders:cpu . + docker run -v ../../examples:/root/examples -p 8888:8888 -d recommenders:cpu + ``` -
-PySpark environment -``` -DOCKER_BUILDKIT=1 docker build -t recommenders:pyspark --build-arg ENV=pyspark --build-arg VIRTUAL_ENV=conda . -docker run -p 8888:8888 -d recommenders:pyspark -``` +* **PySpark environment** -
+ ```bash + docker build -t recommenders:pyspark --build-arg EXTRAS=[spark] . + docker run -v ../../examples:/root/examples -p 8888:8888 -d recommenders:pyspark + ``` -
-GPU environment +* **GPU environment** -``` -DOCKER_BUILDKIT=1 docker build -t recommenders:gpu --build-arg ENV=gpu --build-arg VIRTUAL_ENV=conda . -docker run --runtime=nvidia -p 8888:8888 -d recommenders:gpu -``` + ```bash + docker build -t recommenders:gpu --build-arg COMPUTE=gpu . + docker run --runtime=nvidia -v ../../examples:/root/examples -p 8888:8888 -d recommenders:gpu + ``` -
-
-GPU + PySpark environment +* **GPU + PySpark environment** -``` -DOCKER_BUILDKIT=1 docker build -t recommenders:full --build-arg ENV=full --build-arg VIRTUAL_ENV=conda . -docker run --runtime=nvidia -p 8888:8888 -d recommenders:full -``` + ```bash + docker build -t recommenders:gpu-pyspark --build-arg COMPUTE=gpu --build-arg EXTRAS=[gpu,spark] . + docker run --runtime=nvidia -v ../../examples:/root/examples -p 8888:8888 -d recommenders:gpu-pyspark + ``` -
Build Arguments --------------- -There are several build arguments which can change how the image is built. Similar to the `ENV` build argument these are specified during the docker build command. +There are several build arguments which can change how the image is +built. Similar to the `ENV` build argument these are specified during +the docker build command. Build Arg|Description| ---------|-----------| -ENV|Environment to use, options: cpu, pyspark, gpu, full (defaults to cpu)| -VIRTUAL_ENV|Virtual environment to use; mandatory argument, must be one of "conda", "venv", "virtualenv"| -ANACONDA|Anaconda installation script (defaults to miniconda3 4.6.14)| +`COMPUTE`|Compute to use, options: `cpu`, `gpu` (defaults to `cpu`)| +`EXTRAS`|Extra dependencies to use, options: `dev`, `gpu`, `spark` (defaults to none ("")); For example, `[gpu,spark]`| +`GIT_REF`|Git ref of Recommenders to install, options: `main`, `staging`, etc (defaults to `main`); Empty value means editable installation of current clone| +`JDK_VERSION`|OpenJDK version to use (defaults to `21`)| +`PYTHON_VERSION`|Python version to use (defaults to `3.11`)| +`RECO_DIR`|Path to the copy of Recommenders in the container when `GIT_REF` is empty (defaults to `/root/Recommenders`)| + +Examples: +* Install Python 3.10 and the Recommenders package from the staging branch. + + ```bash + docker build -t recommenders:staging --build-arg GIT_REF=staging --build-arg PYTHON_VERSION=3.10 . + ``` + +* Install the current local clone of Recommenders and its extra 'dev' dependencies. -Example: + ```bash + # Go to the root directory of Recommenders to copy the local clone into the Docker image + cd ../../ + docker build -t recommenders:dev --build-arg GIT_REF= --build-arg EXTRAS=[dev] -f tools/docker/Dockerfile . + ``` -``` -DOCKER_BUILDKIT=1 docker build -t recommenders:cpu --build-arg ENV=cpu --build-arg VIRTUAL_ENV=conda . -``` +In order to see detailed progress you can provide a flag during the +build command: ```--progress=plain``` -In order to see detailed progress with BuildKit you can provide a flag during the build command: ```--progress=plain``` -Running tests with docker +Running tests with Docker ------------------------- -To run the tests using e.g. the CPU image, do the following: -``` -docker run -it recommenders:cpu bash -c 'pip install pytest; \ -pip install pytest-cov; \ -pip install pytest-mock; \ -apt-get install -y git; \ -git clone https://github.com/recommenders-team/recommenders.git; \ -cd recommenders; \ -pytest tests/unit -m "not spark and not gpu and not notebooks and not experimental"' -``` \ No newline at end of file +* Run the tests using the `recommenders:cpu` image built above. + NOTE: The `recommender:cpu` image only installs the Recommenders + package under [../../recommenders/](../../recommenders/). + + ```bash + docker run -it recommenders:cpu bash -c 'pip install pytest; \ + pip install pytest-cov; \ + pip install pytest-mock; \ + apt-get install -y git; \ + git clone https://github.com/recommenders-team/recommenders.git; \ + cd Recommenders; \ + pytest tests/unit -m "not spark and not gpu and not notebooks and not experimental"' + ``` + +* Run the tests using the `recommenders:dev` image built above. + NOTE: The `recommenders:dev` image has a full copy of your local + Recommenders repository. + + ```bash + docker run -it recommenders:dev bash -c 'cd Recommenders; \ + pytest tests/unit -m "not spark and not gpu and not notebooks and not experimental"' + ``` \ No newline at end of file