Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script for setting up an AMI #11

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
17 changes: 17 additions & 0 deletions scripts/ramp_environment.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: ramp
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this file used for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still had in mind we should separate the "backend" requirements for the moment.

Since we install ramp-backend and ramp-workflow via pip, all of their dependencies are also installed via pip by default. If we specify their dependencies in this file, we then can use conda for most of these dependencies.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, missed it in the script that you do the conda env update twice, with this one and the one of the kit

channels:
- conda-forge
dependencies:
- numpy
- pyyaml
- psycopg2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will not really work as desired I think since you defined psycopg2-binary as the dependency in setup.py (of ramp-backend). But I think it is actually better to change setup.py to use just psycopg2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching the mismatch.

I switched from psycopg2 to psycopg2-binary because it was advised in a Warning message. I have to check if both exist within conda though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conda already provides binary packages, so there should not exist such a thing (It's rather strange practice of the psycopg2 devs to do this I think)

- sqlalchemy
- boto3
- psutil
- cloudpickle
- gitpython
- pip:
- memory_profiler
- https://api.github.com/repos/paris-saclay-cds/ramp-backend/zipball/master
- https://api.github.com/repos/pythonprofilers/memory_profiler/zipball/master

123 changes: 123 additions & 0 deletions scripts/ubuntu_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
#!/bin/bash
#
# Setup a RAMP backend architecture for a given RAMP project

set -e

# Script variables definition
project_name="$1"
ramp_dependencies="$2"

RAMPKIT_DIR="$HOME/ramp-kits"
kit_url="https://github.com/ramp-kits/$project_name"
kit_dir="$RAMPKIT_DIR/$project_name"

ami_environment="$kit_dir/ami_environment.yml"
data_dir="$kit_dir/data"

# Text management
bold=`tput bold`
normal=`tput sgr0`
underline=`tput smul`

##############
# Script help
##############
usage()
{
echo "
${bold}Setup a RAMP backend architecture for a given RAMP project${normal}

Usage:

${bold}./ubuntu_setup.sh <ramp_kit_name> <ramp_environment_file>${normal}

where <ramp_kit_name> stands for the name of the RAMP kit on GitHub
${underline}https://github.com/ramp-kits/<ramp_kit_name>${normal}

where <ramp_environment_file> is the path to the `ramp_environment.yml`
provided with this script
"
}

#######################################################
# Batch install of latest Miniconda in $HOME directory
#######################################################
miniconda_install() {
LATEST_MINICONDA="http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh"
wget -q $LATEST_MINICONDA -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda3

echo 'export PATH="${HOME}/miniconda3/bin:$PATH"' >> ~/.profile
source ~/.profile

conda update --yes --quiet conda
pip install --upgrade pip
}

#################################################################
# Install Python dependencies in the base conda environment
#
# Argument:
# path to a conda environment file (usually 'environment.yml')
#################################################################
update_conda_env() {
environment_file=$1
conda env update --name base --file $environment_file
}

#############################################################
# Upgrade the kernal and install conda and RAMP dependencies
#############################################################
system_setup() {
sudo apt-get update
sudo apt-get upgrade --yes
miniconda_install
update_conda_env $ramp_dependencies
}

#################################################################
# Clone the ramp-kit, install dependencies and download the data
#################################################################
project_setup() {
mkdir $RAMPKIT_DIR
git clone $kit_url $kit_dir
update_conda_env $ami_environment
rm -rf $data_dir && mkdir $data_dir
echo
echo "${bold}==> Last step is to put the backend data into '$data_dir/'${normal}"
echo
}


start_prompt() {
echo
echo "---------------------------------------------------------------"
echo " Starting the setup of the RAMP AMI for $project_name project"
echo "---------------------------------------------------------------"
echo
}

end_prompt() {
echo
echo "---------------------------------------------------------------"
echo " Ending the setup of the RAMP AMI for $project_name project"
echo "---------------------------------------------------------------"
echo
}

#######
# Main
#######
main() {
if [[ "$project_name" == "" || "$project_name" == "help" ]]; then
usage
else
start_prompt
system_setup
project_setup
end_prompt
fi
}

main
2 changes: 1 addition & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
'numpy',
'pyyaml',
'sqlalchemy',
'psycopg2-binary'],
'psycopg2'],
platforms='any',
packages=find_packages(),
entry_points={
Expand Down