galaxy-hackathon-2014

To demonstrate creating a new tool for galaxy that uses docker for installing dependencies a Dockerfile and galaxy tool config files were generated for BAYSIC. BAYSIC uses bayesian statistics to calculate the posterior probabilities for a set of variant calls given multiple independent variant calls sets for the same sample or dataset. The this tool was developed on a galaxy server running from a EC2 instance. I will first covered installing galaxy on an EC2 instance then installing the BAYSIC tool.

Personal Galaxy EC2 Server

EC2 instance

• start with a base Ubuntu 14.04 64 bit image [ubuntu-trusty-14.04-amd64-server-20140416.1 (ami-1d8c9574)]
• add a rule to the security group with for port 8080 (or whatever port you use for galaxy), custom tcp protocol, and source 0.0.0.0/0 (can change to specific IPs for increased security)

Galaxy Install

• clone galaxy repository pull-request with modified code for using tools with Docker dependencies

$ hg clone https://bitbucket.org/galaxy/galaxy-central/pull-request/401/allow-tools-and-deployers-to-specify/diff
$ sh run.sh

• Uncomment host=0.0.0.0 (can change to specific IPs for increased security)

# The address on which to listen.  By default, only listen to localhost (Galaxy
# will not be accessible over the network).  Use '0.0.0.0' to listen on all
# available network interfaces.
host = 0.0.0.0

• Add docker runner to sudoers (sudo visudo) file (replace galaxy with your username you are running galaxy under):stolen from apetkau https://github.com/apetkau/galaxy-hackathon-2014

galaxy  ALL = (root) NOPASSWD: SETENV: /usr/bin/docker

• run sh run.sh to check galaxy installation. Go to instance public DNS port 8080, e.g. http://ec2-54-91-222-102.compute-1.amazonaws.com:8080.

Install Docker

• Following steps from docker documentation for installing docker on ubuntu http://docs.docker.com/installation/ubuntulinux/

curl -s https://get.docker.io/ubuntu/ | sudo sh

• Testing docker install

sudo docker run -i -t ubuntu /bin/bash

Setup BAYSIC Tool

• Create a directory for docker tools and move the tool config to the docker directory and the Dockerfile to a tools specific subdirectory.
  • Note: Dockerfiles must be named "Dockerfile", inorder to prevent conflicts place Dockerfiles in independent directories.

bash
$ mkdir tools/docker tools/docker/BAYSIC
$ cp baysicDocker.xml tools/docker/
$ cp Dockerfile tools/docker/BAYSIC 

• In tool_conf.xml add

xml
  <section>
    <tool file="docker/baysicDocker.xml"/>
  </section>

Create Docker image with BAYSIC dependencies

• Build docker image using Dockerfile
  • The Dockerfile runs the BAYSIC test script, make sure tests are okay before proceeding.

bash
cd tools/docker/BAYSIC
sudo docker build .

• Save docker image
  • get docker container id bash sudo docker ps -l
  • commit new image from container generated by the Dockerfile
    • need to name the image ubuntu:baysic for the tools config script

bash sudo docker commit CONTAINER ubuntu:baysic

Create job_conf.xml and add

cp job_conf.xml.sample_basic job_conf.xml

Add in job_conf.xml:

    <destinations default="docker_local">
       <destination id="local" runner="local"/>
       <destination id="docker_local" runner="local">
         <param id="docker_enabled">true</param>
       </destination>
    </destinations>

Test Installation

• Run Galaxy

$ sh run.sh

• Run tool Combine variant call sets with BAYSIC, will need to upload vcf files to run.

• Check log file. If tool ran you should see entries like:

galaxy.jobs DEBUG 2014-06-29 20:16:26,010 (50) Working directory for job is: /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50
galaxy.jobs.handler DEBUG 2014-06-29 20:16:26,025 (50) Dispatching to local runner
128.220.159.7 - - [29/Jun/2014:20:16:26 +0000] "GET /api/histories/df7a1f0c02a5b08e/contents HTTP/1.1" 200 - "http://ec2-54-91-222-102.compute-1.amazonaws.com:8080/root/index" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
galaxy.jobs DEBUG 2014-06-29 20:16:26,367 (50) Persisting job destination (destination id: docker_local)
galaxy.jobs.handler INFO 2014-06-29 20:16:26,438 (50) Job dispatched
galaxy.jobs.runners DEBUG 2014-06-29 20:16:27,088 (50) command is: sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /home/ubuntu/galaxy-central/galaxy-central:/home/ubuntu/galaxy-central/galaxy-central:ro -v /home/ubuntu/galaxy-central/galaxy-central/tools/docker:/home/ubuntu/galaxy-central/galaxy-central/tools/docker:ro -v /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50:/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50:rw -v /home/ubuntu/galaxy-central/galaxy-central/database/files:/home/ubuntu/galaxy-central/galaxy-central/database/files:rw -w /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50 --net none ubuntu:baysic /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/container.sh
galaxy.jobs.runners.local DEBUG 2014-06-29 20:16:27,088 (50) executing job script: /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/galaxy_50.sh
galaxy.jobs DEBUG 2014-06-29 20:16:27,188 (50) Persisting job destination (destination id: docker_local)
galaxy.jobs.runners.local DEBUG 2014-06-29 20:17:43,407 execution finished: /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/galaxy_50.sh
galaxy.jobs.runners DEBUG 2014-06-29 20:17:43,739 executing external set_meta script for job 50: /home/ubuntu/galaxy-central/galaxy-central/set_metadata.sh ./database/files /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50 . /home/ubuntu/galaxy-central/galaxy-central/universe_wsgi.ini /home/ubuntu/galaxy-central/galaxy-central/database/tmp/tmpkRxoWs /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/galaxy.json /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_in_HistoryDatasetAssociation_52_slhdrI,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_kwds_HistoryDatasetAssociation_52_2KgeAN,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_out_HistoryDatasetAssociation_52_QU7JY5,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_results_HistoryDatasetAssociation_52_IGnf7T,,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_override_HistoryDatasetAssociation_52_ZMzQN0 /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_in_HistoryDatasetAssociation_53_YMeewb,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_kwds_HistoryDatasetAssociation_53_Y7NwNb,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_out_HistoryDatasetAssociation_53_NkL9_q,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_results_HistoryDatasetAssociation_53_8ofrTE,,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_override_HistoryDatasetAssociation_53_T19q89 /home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_in_HistoryDatasetAssociation_54_nqhKWl,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_kwds_HistoryDatasetAssociation_54_rgd0Ee,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_out_HistoryDatasetAssociation_54_Pthxrz,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_results_HistoryDatasetAssociation_54_yJKBgy,,/home/ubuntu/galaxy-central/galaxy-central/database/job_working_directory/000/50/metadata_override_HistoryDatasetAssociation_54_MHRLrg