-
Notifications
You must be signed in to change notification settings - Fork 1
CI Setup
It is cumbersome to the point of impossibility to manually run tests on every code change and commit of a project. Therefore, quite a lot of infrastructure exists to automate the process of testing new code for regressions of any kind. This is particularly easy (though rare to see) in scientific code development, as it's possible to write integration and regression tests that ensure large portions of the codebase are working (and working together) correctly, just by comparing against an analytic result, or the result of what should be an equivalent calculation.
In addition to the AFD-Illinois organization on GitHub, there is a "shadow" version on GitLab, accessible here. This organization exists to host mirrors which will automatically pull new code from the repositories here at AFD-Illinois, and in turn compile and test new code as it is committed, reporting results back to GitHub to display the test pass/fail markers you see next to pull requests and commits. A compile/testing setup is called a "Continuous Integration Pipeline" and is specified in a file .gitlab-ci.yml in the root of a repository.
For compute-heavy projects, GitLab requires the user to specify somewhere to compile and test the code -- core-hours do not come free! Usually, this is done with a container manager like Kubernetes, so that each project can specify the environment (container) it wishes to compile in, avoiding machine-specific idiosyncrasies. Several supercomputers offer Kubernetes instances for this and other uses. For our projects, the runner does not use a container but compiles the code raw inside of a user environment running a free daemon program called gitlab-runner (currently bprather on LMC). A gitlab-runner can be set up and connected to the AFD-Illinois GitLab organization by anyone with an account on BH cluster and GitLab, and GitLab itself will choose whatever runner is available from the pool when it needs to run something (particular runners may be specified with tags like gpu but this is not used yet).
Currently, only KHARMA and ipole are tested automatically. The .gitlab-ci.yml scripts for each are very simple, calling through to bash scripts which run the actual compile and testing processes.
- For
ipole, this means a single script which compiles the various problem configurations, then runs each and compares to a "gold-standard" version downloaded from a particular directory. You can replicate the CI pipeline by running the scriptsdownload_tests.shandrun_tests.shintests. Note that the test download requires SSH access to Illinois BH cluster; theanalytictest is standalone and can be used for basic validation on a new machine. - For KHARMA, this means compiling the code once, then running a battery of convergence and identity tests using the different possible configurations. These tests are (so far, and in the future nearly) all standalone. Each test can be run with
./run.shand./check.shin its folder.
CI tests in KHARMA are split by problem setup: all Bondi problem tests, all MHD modes problem tests, etc. Each folder has a ./run.sh file, which will run all tests in turn, exiting immediately if KHARMA crashes or returns a non-zero value, and a check.sh file, which will run all tests regardless of return values but but will itself return 1 if any test fails.