Installation Notes

This page is intended for Tutorial participants who would like to setup an own environment with Radical-Pilot, SAGA-Hadoop and the Hadoop/Spark tools.

Anaconda (with Python2)

• https://www.continuum.io/downloads

Jupyterhub:

Web page: https://github.com/jupyter/jupyterhub Installation:

Install node: https://nodejs.org/en/download/

/opt/anaconda/bin/conda create --name python2 python=2
/opt/anaconda/bin/conda create --name python3 python=3

source /opt/anaconda/bin/activate python3
pip install jupyterhub
mkdir $HOME/jupyterhub
cd $HOME/jupyterhub
jupyterhub --generate_config


su root
cd /root/jupyterhub
source activate python3
nohup jupyterhub -f jupyterhub_config.py &

PAM issue:

https://github.com/jupyter/jupyterhub/issues/323

Configurations:

(Jupyterhub Config)[./configs/jupyterhub_config.py]

SAGA-Hadoop

pip install --upgrade saga-hadoop

Hadoop:

• Recommended to use Ambari-based automatic installation: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/index.html

Spark

• Download binary version of Spark 1.5.1 and install next to HDP: http://spark.apache.org

Radial-Pilot

The Hadoop integration is not part of the current Radical-Pilot 0.37 release and needs to be installed from source (Status 11/13/2015). To setup an Anaconda environment use the following steps:

conda create -y -p python2 python=2.7 
source activate python2

Checkout Radical-Pilot and dependencies:

mkdir radical
cd radical
git clone https://github.com/radical-cybertools/radical.pilot.git 
git clone https://github.com/radical-cybertools/saga-python.git 
git clone https://github.com/radical-cybertools/radical.utils.git 

cd radical.pilot
git checkout feature/anaconda_setup
pip install . --upgrade

cd ../saga-python
pip install . --upgrade

cd ../radical-utils
pip install . --upgrade