Jupyter notebook preview service
This project is part of the Terra platform of service APIs. It is primarily intended to provide preview versions of notebooks and R markdown files for Terra-based web applications, such as the Terra workbench research environment.
Your web-app needs to display read-only versions of Jupyter notebooks and R markdown files.
Under the hood, Jupyter .ipynb files are JSON documents with a particular format. Jupyter provides a commandline tool / library nbconvert for interacting with these documents and converting them to various formats, in particular HTML.
Similarly, R Markdown .Rmd files are documents with a particular format, and the rmarkdown package provides mechanisms for converting R Markdown documents to HTML.
This project is essentially nbconvert and rmarkdown as a REST service. It takes an input notebook or R Markdown document and returns HTML that can be embedded in your application.
A swagger-ui page is available at /swagger-ui/ on any running instance. For existing instances, those are:
- dev: https://calhoun.dsde-dev.broadinstitute.org/swagger-ui/
- alpha: https://calhoun.dsde-alpha.broadinstitute.org/swagger-ui/
- perf: https://calhoun.dsde-perf.broadinstitute.org/swagger-ui/
- staging: https://calhoun.dsde-staging.broadinstitute.org/swagger-ui/
- prod: https://calhoun.dsde-prod.broadinstitute.org/swagger-ui/
This project uses the Flask Python web framework.
We use Poetry to manage our dependencies. From their website:
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you. Poetry offers a lockfile to ensure repeatable installs, and can build your project for distribution.
Install Poetry
curl -sSL https://install.python-poetry.org | python3 -
If you need to change any dependency versions:
- update the pyproject.toml file
- run the following to update the lock file
poetry lock
To install dependencies
poetry install
To update dependencies
poetry update
Run a local containerized server:
docker image build . -t calhoun-test:0
docker kill t1
docker run -e FLASK_DEBUG=1 --rm -itd --name t1 -p 8080:8080 calhoun-test:0
This will start a Calhoun server at localhost:8080.
Access the application locally:
You can skip the container and run a local app with Flask:
python3 -m venv env
source env/bin/activate
pip install Flask
export FLASK_DEBUG=1
Install Pandoc and R
brew install pandoc
brew install R
Install R packages
R
> install.packages(c("rmarkdown", "stringi", "tidyverse", "Seurat", "ggforce"))
Install Poetry and project dependencies
curl -sSL https://install.python-poetry.org
poetry install
Write a dev config file
cp config.py config.dev.py
Ensure etc/hosts file has the following record:
127.0.0.1 local.dsde-dev.broadinstitute.org
Once complete, copy vault read secret/dsde/firecloud/dev/common/server.crt
to /etc/ssl/certs
and
vault read secret/dsde/firecloud/dev/common/server.key
to /etc/ssl/private
.
Run a local server
DEVELOPMENT='true' SAM_ROOT='https://sam.dsde-dev.broadinstitute.org' python3 main.py
- point the calhoun URL in the terra-ui dev config to your local url http://localhost:8080
- run a local terra-ui
- Look at previews!
Run unit tests locally
./scripts/unit-test.sh
Run automation tests locally
gcloud auth login <any-terra-dev-user>
RUN_AUTHENTICATED_TEST=1 ./scripts/automation-test.sh
If you add a new test case, make sure it is imported and added to test_cases
in unit_test.py
.
Upon merging a change to dev:
- The build github workflow builds the new image
- Then it automatically updates the calhoun version in terra-helmfile