Use Github Actions to Build and Test DeepBedMap #91
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
weiji14
added
enhancement ✨
weiji14
changed the title
weiji14
force-pushed
the
master
branch
2 times, most recently
from
d86ccf8 to
f4c2c34
Compare
Trying to get Github Actions to run `docker build` on our Dockerfile!
weiji14
force-pushed
the
github-actions/init
branch
from
b1a78dd to
6a6c783
Compare
Really really fix up the Dockerfile so that we can use run and test it on more cloud providers (e.g. binder, github, etc). The key hack here is to install deepbedmap's conda dependencies into the conda 'base' environment, but name the kernel 'deepbedmap' still. This hopefully means we can directly `pipenv run python...` without having to `source activate deepbedmap` first, at least in the docker world. Changed miniconda to be installed under /home/jovyan/.conda instead of /opt/conda, and explicitly set environment variables CONDA_PREFIX and PATH so that `pipenv run` has them when it looks in our `.env` file even though we don't run `conda/source activate`.
Updated the github-actions main.workflow file to directly build from the repository's Dockerfile, and run tests on that built docker image! Technically it should be doing something like `docker build -f Dockerfile -t weiji14/deepbedmap .` and `docker run weiji14/deepbedmap pipenv run python -m pytest --verbose --disable-warnings --nbval test_ipynb.ipynb`.
Just putting the unit/integration test step directly into our Dockerfile as a multi-stage build step, and have different build targets 'app' and 'test' for the "Docker Build" and "Run Tests" github-actions steps respectively. Had problems with permissions when trying to run tests against built docker image in 06b2325. The `.local/share` folder containing the pipenv libraries wasnt't getting copied to the /github/workspace folder meaning `pipenv run` couldn't work properly. Now it does though!
Bumps [conda](https://github.com/conda/conda) from 4.6.14 to 4.7.10. Also had to change cudatoolkit to use the 'anaconda' channel instead of 'defaults' for some reason though they are aliases. - [Release notes](https://github.com/conda/conda/releases) - [Changelog](https://github.com/conda/conda/blob/master/CHANGELOG.md) - [Commits](conda/conda@4.6.14...4.7.10)
Bumps [pip](https://github.com/pypa/pip) from 18.1 to 19.2.1. Can't believe it's been so long since the last update at 56db41a! - [Release notes](https://github.com/pypa/pip/releases/) - [Changelog](https://github.com/pypa/pip/blob/master/NEWS.rst) - [Commits](pypa/pip@18.1...19.2.1)
Bumps [gmt](https://github.com/GenericMappingTools/gmt) from 6.0.0rc1 to 6.0.0rc3. See also issue documenting this release at GenericMappingTools/gmt#1176. - [Release notes](https://github.com/GenericMappingTools/gmt/releases) - [Commits](GenericMappingTools/gmt@6.0.0rc1...6.0.0rc3)
Closes #161 Conda refresh for v0.9.4.
Update badges on the main README.md file. And codefresh, thank you for the many months of sweat and tears, but it is time for you to make way for Github-Actions! Yes github-actions is slower without docker build caching enabled, but at least it can run my integration tests properly without complaining about some 1GB RAM limitation :P Also made sure that binder works properly now (with a fresh new badge!), because it is about time the world can just click and run deepbedmap from anywhere. Following best practices from https://mybinder.readthedocs.io/en/latest/tutorials/dockerfile.html. Specifically, we refactored our Dockerfile with a `.entrypoint.sh` ENTRYPOINT that basically does `conda activate && pipenv run $@` where `$@` is the `jupyter lab --ip 0.0.0.0` command we invoke at runtime (similar to what binder uses, see jupyterhub/mybinder.org-user-guide#87). This ensures that our conda and pipenv libraries will be loaded properly alongside the environment variables in our `.env` file. Made sure to tidy up the Dockerfile to remove some unnecessary bits.
weiji14
force-pushed
the
github-actions/init
branch
from
bd93889 to
1d8f278
Compare
Seems like [Pangeo](https://pangeo.io/) provided a better fit, being a "Big Data geoscience" community platform! Notably it launches a jupyter lab instance directly instead of jupyter notebook, and supports dask a bit better apparently. Actual reason to change was because the unit tests seemed to only pass on binder.pangeo.io and not mybinder.org... Also quickfix that conda installation reference, no idea why pdal.io is there!
Removing the 'test' stage build step from our Dockerfile, because we have a fancy entrypoint introduced in 1d8f278 that does `conda activate && pipenv run "$@"` which accepts pytest properly! Essentially reverting d4ca21a but we are keeping the multi-stage feature in, just to remind myself that there's some docker image size optimization that could be done still. The changes here might also mean things work properly in mybinder.org now (since there's no tests to fail/pass), but we'll stick to good ol' pangeo's binder instance.
Owner
Author
|
I've isolated why the unit tests for data_prep.ipynb failed with https://mybinder.org but not https://binder.pangeo.io. The failed test is when trying to download a csv file using FTP, something to do with networking? Oh well, we're using pangeo anyway as per c290c4a. |
weiji14
changed the title
weiji14
added a commit
that referenced
this pull request
Aug 4, 2019
Closes #91 Use Github Actions to Build and Test DeepBedMap.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Have our Continuous Integration/Continuous Deployment (CI/CD) pipeline work using Github Actions. From building the docker image to running the unit/integration tests
and deploying to some cloud registry once checks have passed.Also deprecates use of codefresh CI/CD which switched to a limiting 1GB RAM instance insufficient for our needs. For historical purposes, you should be able to track the old codefresh builds at
.
TODO:
docker buildwork to create a docker image (6a6c783)