Fix Surprise and Python 3.7

NEWS.md

@@ -6,7 +6,7 @@ We have a new release [Recommenders 0.7.0](https://github.com/microsoft/recommen
 
 In this, we have changed the names of the folders which contain the source code, so that they are more informative. This implies that you will need to change any import statements that reference the recommenders package. Specifically, the folder `reco_utils` has been renamed to `recommenders` and its subfolders have been renamed according to [issue 1390](https://github.com/microsoft/recommenders/issues/1390).  
 
-The recommenders package now supports three types of environments: [venv](https://docs.python.org/3/library/venv.html) and [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#) with Python 3.6, [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) with Python versions 3.6 and 3.7.
+The recommenders package now supports three types of environments: [venv](https://docs.python.org/3/library/venv.html), [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#) and [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) with Python versions 3.6 and 3.7.
 
 We have also added new evaluation metrics: _novelty, serendipity, diversity and coverage_ (see the [evalution notebooks](examples/03_evaluate/README.md)).
 

README.md

@@ -10,7 +10,7 @@ We have changed the names of the folders which contain the source code, so that
 
 The previous release ([0.6.0](https://github.com/microsoft/recommenders/releases/tag/0.6.0)) is compatible with the old style of naming of modules. 
 
-The recommenders package now supports three types of environments: [venv](https://docs.python.org/3/library/venv.html) and [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#) with Python 3.6, [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) with Python versions 3.6 and 3.7.
+The recommenders package now supports three types of environments: [venv](https://docs.python.org/3/library/venv.html), [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#) and [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) with Python versions 3.6 and 3.7.
 
 We have also added new evaluation metrics: _novelty, serendipity, diversity and coverage_ (see the [evalution notebooks](examples/03_evaluate/README.md)).
 
@@ -42,8 +42,7 @@ For a more detailed overview of the repository, please see the documents on the
 Please see the [setup guide](SETUP.md) for more details on setting up your machine locally, on a [data science virtual machine (DSVM)](https://azure.microsoft.com/en-gb/services/virtual-machines/data-science-virtual-machines/) or on [Azure Databricks](SETUP.md#setup-guide-for-azure-databricks).
 
 The installation of the recommenders package has been tested with 
-- Python version 3.6 and [venv](https://docs.python.org/3/library/venv.html) or [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#)
-- Python versions 3.6, 3.7 and [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) 
+- Python versions 3.6, 3.7 and [venv](https://docs.python.org/3/library/venv.html), [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#) or [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment) 
 
 and currently does not support version 3.8 and above. It is recommended to install the package and its dependencies inside a clean environment (such as [conda](https://docs.conda.io/projects/conda/en/latest/glossary.html?highlight=environment#conda-environment), [venv](https://docs.python.org/3/library/venv.html) or [virtualenv](https://virtualenv.pypa.io/en/latest/index.html#)).
 
@@ -68,11 +67,6 @@ pip install --upgrade pip
 pip install --upgrade setuptools
 pip install recommenders[examples]
 ```
-In the case of conda, you also need to 
-```bash
-conda install numpy-base
-```
-after the pip installation.
 
 4. Register your (conda or virtual) environment with Jupyter:
 

SETUP.md

@@ -151,6 +151,7 @@ create the file `%RECO_ENV%\etc\conda\deactivate.d\env_vars.bat` and add:
 
 It is straightforward to install the recommenders package within a [virtual environment](https://docs.python.org/3/library/venv.html). However, setting up CUDA for use with a GPU can be cumbersome. We thus
 recommend setting up [Nvidia docker](https://github.com/NVIDIA/nvidia-docker) and running the virtual environment within a container, as the most convenient way to do this.  
+In the following `3.6` should be replaced with the Python version you are using. 
 
     # Start docker daemon if not running
     sudo dockerd &

conda.md

@@ -1,6 +1,6 @@
 One possible way to use the repository is to run all the recommender utilities directly from a local copy of the source code (without building the package). This requires installing all the necessary dependencies from Anaconda and PyPI.
 
-To this end we provide a script, [generate_conda_file.py](tools/generate_conda_file.py), to generate a conda-environment yaml file which you can use to create the target environment using Python 3.6 with all the correct dependencies.
+To this end we provide a script, [generate_conda_file.py](tools/generate_conda_file.py), to generate a conda-environment yaml file which you can use to create the target environment using Python 3.6 or 3.7 with all the correct dependencies.
 
 Assuming the repo is cloned as `Recommenders` in the local system, to install **a default (Python CPU) environment**:
 

recommenders/README.md

@@ -22,11 +22,6 @@ To install core utilities, CPU-based algorithms, and dependencies
 pip install --upgrade pip
 pip install recommenders
 ```
-In the case of conda, you also need to 
-```bash
-conda install numpy-base
-```
-after the pip installation.
 
 ## Optional Dependencies
 

setup.py

@@ -22,7 +22,7 @@
     version += ".post" + str(int(time.time()))
 
 install_requires = [
-    "numpy>=1.14",
+    "numpy>=1.19",
     "pandas>1.0.3,<2",
     "scipy>=1.0.0,<2",
     "tqdm>=4.31.1,<5",
@@ -43,7 +43,8 @@
     "pyyaml>=5.4.1,<6",
     "requests>=2.0.0,<3",
     "cornac>=1.1.2,<2",
-    "scikit-surprise>=0.19.1,<=1.1.1",
+    # For Surprise, specify the tarball in order to avoid incompatibilities of compiled .pyx files with numpy versions < 1.20 
+    "scikit-surprise@https://files.pythonhosted.org/packages/97/37/5d334adaf5ddd65da99fc65f6507e0e4599d092ba048f4302fe8775619e8/scikit-surprise-1.1.1.tar.gz",
     "retrying>=1.3.3",
 ]
 

tools/docker/Dockerfile

@@ -21,7 +21,7 @@ RUN if [ "${VIRTUAL_ENV}" != "conda" ] && [ "${VIRTUAL_ENV}" != "venv" ] && [ "$
 RUN apt-get update && \
     apt-get install -y curl build-essential cmake
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then apt-get install -y libpython3.7; fi
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then apt-get install -y libpython3.6; fi
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then apt-get install -y libpython3.7; fi
 
 # Install Anaconda
 ARG ANACONDA="https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
@@ -38,19 +38,19 @@ SHELL ["/bin/bash", "--login", "-c"]
 
 # Python version supported by recommenders
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then conda install python=3.7; fi
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then apt-get -y install python3.6; \
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then apt-get -y install python3.7; \
     apt-get -y install python3-pip; \
-    apt-get -y install python3.6-venv; fi
-RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then apt-get -y install python3.6; \
+    apt-get -y install python3.7-venv; fi
+RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then apt-get -y install python3.7; \
     apt-get -y install python3-pip; \
-    python3.6 -m pip install --user virtualenv; fi
+    python3.7 -m pip install --user virtualenv; fi
 
 # Activate the virtual environment
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then python3.6 -m venv --system-site-packages $HOME/${VIRTUAL_ENV}; \
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then python3.7 -m venv --system-site-packages $HOME/${VIRTUAL_ENV}; \
     source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install --upgrade pip; \
     pip install --upgrade setuptools; fi
-RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then python3.6 -m virtualenv $HOME/${VIRTUAL_ENV}; \
+RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then python3.7 -m virtualenv $HOME/${VIRTUAL_ENV}; \
     source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install --upgrade pip; \
     pip install --upgrade setuptools; fi
@@ -62,10 +62,7 @@ FROM base AS cpu
 
 RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install recommenders[xlearn,examples]; fi
-# Note that "conda install numpy-base" needs to be called after the pip installation
-# so that numpy works smoothly with other dependencies in python 3.7
-RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then pip install recommenders[xlearn,examples]; \
-    conda install -n base numpy-base; fi
+RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then pip install recommenders[xlearn,examples]; fi
 
 
 ###############
@@ -84,8 +81,7 @@ ENV JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64" \
 # Install dependencies in virtual environment
 RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install recommenders[spark,xlearn,examples]; fi
-RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then pip install recommenders[spark,xlearn,examples]; \
-    conda install -n base numpy-base; fi
+RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then pip install recommenders[spark,xlearn,examples]; fi
 
 
 ###########
@@ -105,7 +101,7 @@ RUN if [ "${VIRTUAL_ENV}" != "conda" ] && [ "${VIRTUAL_ENV}" != "venv" ] && [ "$
 RUN apt-get update && \
     apt-get install -y curl build-essential cmake
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then apt-get install -y libpython3.7; fi
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then apt-get install -y libpython3.6; fi
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; then apt-get install -y libpython3.7; fi
 
 # Install Anaconda
 ARG ANACONDA="https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"
@@ -121,28 +117,27 @@ SHELL ["/bin/bash", "--login", "-c"]
 
 # Python version supported by recommenders
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then conda install python=3.7; fi
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then apt-get -y install python3.6; \
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then apt-get -y install python3.7; \
     apt-get -y install python3-pip; \
-    apt-get -y install python3.6-venv; fi
-RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then apt-get -y install python3.6; \
+    apt-get -y install python3.7-venv; fi
+RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then apt-get -y install python3.7; \
     apt-get -y install python3-pip; \
-    python3.6 -m pip install --user virtualenv; fi
+    python3.7 -m pip install --user virtualenv; fi
 
 # Activate the virtual environment
-RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then python3.6 -m venv --system-site-packages $HOME/${VIRTUAL_ENV}; \
+RUN if [ "${VIRTUAL_ENV}" = "venv" ] ; then python3.7 -m venv --system-site-packages $HOME/${VIRTUAL_ENV}; \
     source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install --upgrade pip; \
     pip install --upgrade setuptools; \
     pip install recommenders[gpu,xlearn,examples]; fi
-RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then python3.6 -m virtualenv $HOME/${VIRTUAL_ENV}; \
+RUN if [ "${VIRTUAL_ENV}" = "virtualenv" ] ; then python3.7 -m virtualenv $HOME/${VIRTUAL_ENV}; \
     source $HOME/${VIRTUAL_ENV}/bin/activate; \
     pip install --upgrade pip; \
     pip install --upgrade setuptools; \
     pip install recommenders[gpu,xlearn,examples]; fi
 
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then \
-    pip install recommenders[gpu,xlearn,examples] -f https://download.pytorch.org/whl/cu100/torch_stable.html && \
-    conda install -n base numpy-base; fi
+    pip install recommenders[gpu,xlearn,examples] -f https://download.pytorch.org/whl/cu100/torch_stable.html ; fi
 
 
 ############
@@ -168,10 +163,6 @@ RUN if [ "${VIRTUAL_ENV}" = "venv" ] || [ "${VIRTUAL_ENV}" = "virtualenv" ]; the
     pip install recommenders[all]; fi
 RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then pip install recommenders[all]; fi
 
-# It is not clear why but force-reinstall is needed at this place, 
-# otherwise numpy-base does not appear in conda list  
-RUN if [ "${VIRTUAL_ENV}" = "conda" ] ; then conda update --force-reinstall -y -n base numpy-base; fi
-
 
 #############
 # Final Stage