Add automated CI testing through cron on Orion by CoryMartin-NOAA · Pull Request #12 · NOAA-EMC/GDASApp

CoryMartin-NOAA · 2022-03-22T20:15:01Z

Closes #9

This PR adds scripts for automated CI testing on NOAA HPC platforms. First Orion, but minor additions will be needed for Hera once the python virtual environment there is set up properly.

Every 15 mins, Orion-login-1 will run $clone/ci/driver.sh -t orion.

The script will check to see which open PR(s) in GDASApp have the label Orion-RT and for each one, checkout, build, and test and report back/change labels as appropriate through @emcbot .

Note: the modulefile changed in #6 but has not been merged yet but is also included in this PR as it is needed for the GitHub CLI.

emcbot · 2022-03-22T21:52:15Z

Automated Pull Request Testing Results:
Machine: orion

Start: Tue Mar 22 15:15:14 CDT 2022 on Orion-login-1.HPC.MsState.Edu
Build:                          *SUCCESS*
Build: Completed at Tue Mar 22 16:11:19 CDT 2022
---------------------------------------------------
Tests:                          *Failed*
Tests: Failed at Tue Mar 22 16:52:14 CDT 2022
Tests:         93% tests passed, 96 tests failed out of 1383
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/PR/12/GDASApp/build/log.ctest

guillaumevernieres · 2022-03-23T15:25:36Z

+fi
+# ==============================================================================
+# run ctests
+cd $repodir/build


@CoryMartin-NOAA , why not limit the testing to this repo only and ignore the test status of the jedi components? It might sounds silly right now, but in the next few pr's we will have specific tests for the "prep" "run" and "post" steps, which will be making use of the various jedi components. I think it's a good enough "canary".
I vote for ctest -R gdasapp or something of that effect.
Or maybe 2 reports? One for the jedi components and one for the gdasapp tests?

Would you also suggest not building the JEDI apps (at least for now) also?

Nope, we might as well do the build now. We're going to start adding tests using the jedi apps soon enough.
That's currently what I'm working on, just empty stubs right now but I'm hoping to have a basic jedi/soca var application by the end of the day:

build$ ctest -R gdasapp Test project /home/gvernier/sandboxes/GDASApp/build Start 1448: test_gdasapp_soca_obsdb 1/4 Test #1448: test_gdasapp_soca_obsdb .......... Passed 0.27 sec Start 1449: test_gdasapp_soca_ana_prep 2/4 Test #1449: test_gdasapp_soca_ana_prep ....... Passed 0.01 sec Start 1450: test_gdasapp_soca_ana_run 3/4 Test #1450: test_gdasapp_soca_ana_run ........ Passed 0.00 sec Start 1451: test_gdasapp_soca_ana_post 4/4 Test #1451: test_gdasapp_soca_ana_post ....... Passed 0.00 sec 100% tests passed, 0 tests failed out of 4 Label Time Summary: gdasapp = 0.29 sec*proc (4 tests) script = 0.29 sec*proc (4 tests)

We'll need to remember adding gdasapp (or something) to the one test we currently have. It's now called test_check_yaml_keys so something like test_gdasapp_check_yaml_keys, or whatever consensus we decide on.

aerorahul

looks good mostly, just a few minor comments and clarification.

aerorahul · 2022-03-23T15:06:21Z

@@ -0,0 +1,75 @@
+#!/bin/bash
+#set -eu


did you intend to leave this commented out?

aerorahul · 2022-03-23T15:07:31Z

+# run build script
+cd $repodir
+module purge
+./build.sh -t $TARGET &>> log.build


don't you want to capture stderr as well? 2>&1

Doesn't this do that?

I don't think so, but I may be wrong.

>> out is stdout but &>> adds the stderr I thought.

aerorahul · 2022-03-23T15:08:33Z

+module use $GDAS_MODULE_USE
+module load GDAS/$TARGET
+echo "---------------------------------------------------" >> $outfile
+ctest --output-on-failure &>> log.ctest


is this why set -e was commented up?
If so, you can wrap the ctest line around set +e and set -e.

The set -e comes from conda when you do module load, it is something in the pyproj init shell script. I could probably manually fix this in the conda env if you think that is okay?

aerorahul · 2022-03-23T15:09:54Z

+load("hpc")
+load("miniconda3")
+load("gdasapp")


Try and list the explicit versions instead of relying on defaults

Good suggestion.

aerorahul

looks good.

CoryMartin-NOAA · 2022-03-23T16:57:07Z

@guillaumevernieres none of the tests ran with this newest commit on Hera... need to figure out why.

CoryMartin-NOAA · 2022-03-23T16:58:51Z

@guillaumevernieres none of the tests ran with this newest commit on Hera... need to figure out why.

Oh duh, it ran the test with the old test names... all should be good here I think.

guillaumevernieres

👍

**What was done:** - reorganization/code tidy to facilitate the addition of a GFSv17 prototype ctest - exclude the new ctest by default, the user will have to run ```cmake -DRUNGWCI=ON .``` on an old build to configure the new ctest He're the current list: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: WCDA-3DVAR-C48mx500 Test #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200 Test #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 Test #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800 Test #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800 Test #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 Test #7: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800 Test #8: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800 Test #9: Aero-Snow-3DVAR-C96 Test #10: Aero-Snow-3DVAR-C96_gdasfcst_202112201200 Test #11: Atm-hyb-C96C48 Test #12: Atm-hyb-C96C48_gdasfcst_202402231800 Test #13: GFSv17-3DVAR-C384mx025 Test #14: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000 Test #15: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600 Test #16: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600 Test #17: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600 Test #18: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600 Test #19: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600 Test #20: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600 Test #21: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600 Test #22: GFSv17-3DVAR-C384mx025_gdasprep_202106300600 Test #23: GFSv17-3DVAR-C384mx025_gdasanal_202106300600 Total Tests: 23 ``` --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

* Start of CI cron scripts

**What was done:** - reorganization/code tidy to facilitate the addition of a GFSv17 prototype ctest - exclude the new ctest by default, the user will have to run ```cmake -DRUNGWCI=ON .``` on an old build to configure the new ctest He're the current list: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: WCDA-3DVAR-C48mx500 Test #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200 Test #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 Test #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800 Test #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800 Test #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 Test #7: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800 Test #8: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800 Test #9: Aero-Snow-3DVAR-C96 Test #10: Aero-Snow-3DVAR-C96_gdasfcst_202112201200 Test #11: Atm-hyb-C96C48 Test #12: Atm-hyb-C96C48_gdasfcst_202402231800 Test #13: GFSv17-3DVAR-C384mx025 Test #14: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000 Test #15: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600 Test #16: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600 Test #17: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600 Test #18: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600 Test #19: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600 Test #20: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600 Test #21: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600 Test #22: GFSv17-3DVAR-C384mx025_gdasprep_202106300600 Test #23: GFSv17-3DVAR-C384mx025_gdasanal_202106300600 Total Tests: 23 ``` --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

CoryMartin-NOAA added 13 commits March 21, 2022 15:27

Use new python env on orion

bd2d636

Start of CI cron scripts

b9d6e10

Add writing of comment message

14a2d1e

Make it a login shell for modules

52a0c02

Merge branch 'feature/new_orion_modules' into feature/orion_cron_rt

4a19577

Fix typo

5d9a9ef

Commit to save

d54c8c6

move files to ci/

3135026

Comment out repos for testing/debugging

3189c35

Remove CRTM

4bfc5c7

Add scrubber and change of labels

9f11f28

Changes to the CI script

39dab94

Add back all repos

86d99c2

CoryMartin-NOAA added the orion-RT Queue for automated testing on Orion label Mar 22, 2022

CoryMartin-NOAA requested review from RussTreadon-NOAA, aerorahul and guillaumevernieres March 22, 2022 20:15

CoryMartin-NOAA self-assigned this Mar 22, 2022

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Mar 22, 2022

Update comment format

78484ef

emcbot added orion-RT-Failed Automated testing failed on Orion and removed orion-RT-Running Automated testing running on Orion labels Mar 22, 2022

CoryMartin-NOAA added orion-RT Queue for automated testing on Orion and removed orion-RT-Failed Automated testing failed on Orion labels Mar 23, 2022

CoryMartin-NOAA added 2 commits March 23, 2022 15:20

Merge branch 'develop' into feature/orion_cron_rt

6d84e38

Add back unit tests on push

d78686c

guillaumevernieres reviewed Mar 23, 2022

View reviewed changes

aerorahul reviewed Mar 23, 2022

View reviewed changes

Address reviewer comments round 1

2ff4158

CoryMartin-NOAA mentioned this pull request Mar 23, 2022

Add automated CI testing on Hera #14

Merged

aerorahul reviewed Mar 23, 2022

View reviewed changes

Comment thread ci/orion.sh

aerorahul previously approved these changes Mar 23, 2022

View reviewed changes

Change test names

e1de501

CoryMartin-NOAA dismissed aerorahul’s stale review via e1de501 March 23, 2022 16:03

CoryMartin-NOAA requested review from aerorahul and guillaumevernieres March 23, 2022 17:00

guillaumevernieres approved these changes Mar 23, 2022

View reviewed changes

CoryMartin-NOAA merged commit 9c489ce into develop Mar 23, 2022

CoryMartin-NOAA deleted the feature/orion_cron_rt branch March 23, 2022 17:09

DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026

Add automated CI testing through cron on Orion (#12)

8915748

* Start of CI cron scripts

Conversation

CoryMartin-NOAA commented Mar 22, 2022

Uh oh!

emcbot commented Mar 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

CoryMartin-NOAA commented Mar 23, 2022

Uh oh!

CoryMartin-NOAA commented Mar 23, 2022

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants