Add automated CI testing through cron on Orion #12
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,5 @@ | ||
| name: Run Unit Tests | ||
| on: [push, pull_request] | ||
|
|
||
| jobs: | ||
| ctests: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| #!/bin/bash --login | ||
|
|
||
| my_dir="$( cd "$( dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd )" | ||
|
|
||
| # ============================================================================== | ||
| usage() { | ||
| set +x | ||
| echo | ||
| echo "Usage: $0 -t <target> -h" | ||
| echo | ||
| echo " -t target/machine script is running on DEFAULT: $(hostname)" | ||
| echo " -h display this message and quit" | ||
| echo | ||
| exit 1 | ||
| } | ||
|
|
||
| # ============================================================================== | ||
| # First, set up runtime environment | ||
|
|
||
| export TARGET="$(hostname)" | ||
|
|
||
| while getopts "t:h" opt; do | ||
| case $opt in | ||
| t) | ||
| TARGET=$OPTARG | ||
| ;; | ||
| h|\?|:) | ||
| usage | ||
| ;; | ||
| esac | ||
| done | ||
|
|
||
| case ${TARGET} in | ||
| hera | orion) | ||
| echo "Running Automated Testing on $TARGET" | ||
| source $MODULESHOME/init/sh | ||
| source $my_dir/${TARGET}.sh | ||
| module purge | ||
| module use $GDAS_MODULE_USE | ||
| module load GDAS/$TARGET | ||
| module list | ||
| ;; | ||
| *) | ||
| echo "Unsupported platform. Exiting with error." | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
|
|
||
| # ============================================================================== | ||
| # pull on the repo and get list of open PRs | ||
| cd $GDAS_CI_ROOT/repo | ||
| CI_LABEL="${GDAS_CI_HOST}-RT" | ||
| gh pr list --label "$CI_LABEL" --state "open" | awk '{print $1;}' > $GDAS_CI_ROOT/open_pr_list | ||
| open_pr_list=$(cat $GDAS_CI_ROOT/open_pr_list) | ||
|
|
||
| # ============================================================================== | ||
| # clone, checkout, build, test, etc. | ||
| repo_url="https://github.com/NOAA-EMC/GDASApp.git" | ||
| # loop through all open PRs | ||
| for pr in $open_pr_list; do | ||
| gh pr edit $pr --remove-label $CI_LABEL --add-label ${CI_LABEL}-Running | ||
| echo "Processing Pull Request #${pr}" | ||
| mkdir -p $GDAS_CI_ROOT/PR/$pr | ||
| cd $GDAS_CI_ROOT/PR/$pr | ||
|
|
||
| # clone copy of repo | ||
| git clone $repo_url | ||
| cd GDASApp | ||
|
|
||
| # checkout pull request | ||
| git pull | ||
| gh pr checkout $pr | ||
|
|
||
| # get commit hash | ||
| commit=$(git log --pretty=format:'%h' -n 1) | ||
| if [ -f "$GDAS_CI_ROOT/PR/$pr/commit" ]; then | ||
| oldcommit=$(cat $GDAS_CI_ROOT/PR/$pr/commit) | ||
| if [ $oldcommit == $commit ]; then | ||
| # do no more for this PR, as the commit has already been tested | ||
| continue | ||
| fi | ||
| fi | ||
| echo "$commit" > $GDAS_CI_ROOT/PR/$pr/commit | ||
|
|
||
| # run build and testing command | ||
| $my_dir/run_ci.sh -d $GDAS_CI_ROOT/PR/$pr/GDASApp -o $GDAS_CI_ROOT/PR/$pr/output_${commit} | ||
| ci_status=$? | ||
| gh pr comment $pr --body-file $GDAS_CI_ROOT/PR/$pr/output_${commit} | ||
| if [ $ci_status -eq 0 ]; then | ||
| gh pr edit $pr --remove-label ${CI_LABEL}-Running --add-label ${CI_LABEL}-Passed | ||
| else | ||
| gh pr edit $pr --remove-label ${CI_LABEL}-Running --add-label ${CI_LABEL}-Failed | ||
| fi | ||
| done | ||
|
|
||
| # ============================================================================== | ||
| # scrub working directory for older files | ||
| find $GDAS_CI_ROOT/PR/* -mtime +3 -exec rm -rf {} \; | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| GDAS_CI_ROOT=/work2/noaa/stmp/cmartin/CI/GDASApp | ||
| GDAS_CI_HOST='orion' | ||
|
aerorahul marked this conversation as resolved.
|
||
| export GDAS_MODULE_USE=$GDAS_CI_ROOT/repo/modulefiles | ||
| export SLURM_ACCOUNT=da-cpu | ||
| export SALLOC_ACCOUNT=$SLURM_ACCOUNT | ||
| export SBATCH_ACCOUNT=$SLURM_ACCOUNT | ||
| export SLURM_QOS=debug | ||
| export SLURM_EXCLUSIVE=user | ||
| export OMP_NUM_THREADS=1 | ||
| ulimit -s unlimited | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| #!/bin/bash | ||
| #set -eu | ||
|
Contributor
There was a problem hiding this comment. did you intend to leave this commented out? |
||
|
|
||
| # ============================================================================== | ||
| usage() { | ||
| set +x | ||
| echo | ||
| echo "Usage: $0 -d <directory> -o <output> -h" | ||
| echo | ||
| echo " -d Run build and ctest for clone in <directory>" | ||
| echo " -o Path to output message detailing results of CI tests" | ||
| echo " -h display this message and quit" | ||
| echo | ||
| exit 1 | ||
| } | ||
|
|
||
| # ============================================================================== | ||
| while getopts "d:o:h" opt; do | ||
| case $opt in | ||
| d) | ||
| repodir=$OPTARG | ||
| ;; | ||
| o) | ||
| outfile=$OPTARG | ||
| ;; | ||
| h|\?|:) | ||
| usage | ||
| ;; | ||
| esac | ||
| done | ||
|
|
||
| # ============================================================================== | ||
| # start output file | ||
| echo "Automated Pull Request Testing Results:" > $outfile | ||
| echo "Machine: ${TARGET}" >> $outfile | ||
| echo '```' >> $outfile | ||
| echo "Start: $(date) on $(hostname)" >> $outfile | ||
| echo "---------------------------------------------------" >> $outfile | ||
| # ============================================================================== | ||
| # run build script | ||
| cd $repodir | ||
| module purge | ||
| ./build.sh -t $TARGET &>> log.build | ||
|
Contributor
There was a problem hiding this comment. don't you want to capture stderr as well?
Contributor
Author
There was a problem hiding this comment. Doesn't this do that?
Contributor
There was a problem hiding this comment. I don't think so, but I may be wrong.
Contributor
Author
There was a problem hiding this comment.
|
||
| build_status=$? | ||
| if [ $build_status -eq 0 ]; then | ||
| echo "Build: *SUCCESS*" >> $outfile | ||
| echo "Build: Completed at $(date)" >> $outfile | ||
| else | ||
| echo "Build: *FAILED*" >> $outfile | ||
| echo "Build: Failed at $(date)" >> $outfile | ||
| echo "Build: see output at $repodir/log.build" >> $outfile | ||
| echo '```' >> $outfile | ||
| exit $build_status | ||
| fi | ||
| # ============================================================================== | ||
| # run ctests | ||
| cd $repodir/build | ||
|
Contributor
There was a problem hiding this comment. @CoryMartin-NOAA , why not limit the testing to this repo only and ignore the test status of the jedi components? It might sounds silly right now, but in the next few pr's we will have specific tests for the "prep" "run" and "post" steps, which will be making use of the various jedi components. I think it's a good enough "canary".
Contributor
Author
There was a problem hiding this comment. Would you also suggest not building the JEDI apps (at least for now) also?
Contributor
There was a problem hiding this comment. Nope, we might as well do the build now. We're going to start adding tests using the jedi apps soon enough. build$ ctest -R gdasapp
Test project /home/gvernier/sandboxes/GDASApp/build
Start 1448: test_gdasapp_soca_obsdb
1/4 Test #1448: test_gdasapp_soca_obsdb .......... Passed 0.27 sec
Start 1449: test_gdasapp_soca_ana_prep
2/4 Test #1449: test_gdasapp_soca_ana_prep ....... Passed 0.01 sec
Start 1450: test_gdasapp_soca_ana_run
3/4 Test #1450: test_gdasapp_soca_ana_run ........ Passed 0.00 sec
Start 1451: test_gdasapp_soca_ana_post
4/4 Test #1451: test_gdasapp_soca_ana_post ....... Passed 0.00 sec
100% tests passed, 0 tests failed out of 4
Label Time Summary:
gdasapp = 0.29 sec*proc (4 tests)
script = 0.29 sec*proc (4 tests)
Contributor
There was a problem hiding this comment. We'll need to remember adding |
||
| module use $GDAS_MODULE_USE | ||
| module load GDAS/$TARGET | ||
| echo "---------------------------------------------------" >> $outfile | ||
| ctest -R gdasapp --output-on-failure &>> log.ctest | ||
| ctest_status=$? | ||
| npassed=$(cat log.ctest | grep "tests passed") | ||
| if [ $ctest_status -eq 0 ]; then | ||
| echo "Tests: *SUCCESS*" >> $outfile | ||
| echo "Tests: Completed at $(date)" >> $outfile | ||
| echo "Tests: $npassed" >> $outfile | ||
| else | ||
| echo "Tests: *Failed*" >> $outfile | ||
| echo "Tests: Failed at $(date)" >> $outfile | ||
| echo "Tests: $npassed" >> $outfile | ||
| echo "Tests: see output at $repodir/build/log.ctest" >> $outfile | ||
| fi | ||
| echo '```' >> $outfile | ||
|
aerorahul marked this conversation as resolved.
|
||
| exit $ctest_status | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.