Jenkins Pipeline Updates by TerrenceMcGuinness-NOAA · Pull Request #2815 · NOAA-EMC/global-workflow

TerrenceMcGuinness-NOAA · 2024-08-07T17:08:16Z

Description

Pipeline updates:

get gh location on remote machine and define global bash env variable $GH for GitHub CLI
Failed cases are now displayed accordingly in the Jenkins dashboard (see NOTE below)
Added the Build # in messaging for clarity when running from re-runs.
Replaced Matrix construct for concurrency with the parallel method that can use dynamic case lists
- With removing of the hard coded list of cases we now get list of cases dynamically from the PR case directory
- See new look of dashboard below (has more annotations and displays only used cases)

NOTE: failFast (quitting all cases on failing of one) still does not work because it isn't quitting the running remote shells. We can make this a configurable capability in feature request with some custom code. The current behavior has the remaining cases continuing to run after a FAIL label has been issued and it is incumbent of the code manager to kill the CI job in the controller before resetting another Ready label.

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

How has this been tested?

Ran in the authoritative repo with full self-tests

…ccount running agents for EMC global-workflows

…n the Jenkins Controller

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

…stablished

emcbot · 2024-08-08T21:10:30Z

Experiment C48_S2SW FAILED on Hercules in Build# 26 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_S2SW_cb158a81

emcbot · 2024-08-08T21:10:31Z

Experiment C48_ATM FAILED on Hercules in Build# 26 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_ATM_cb158a81

emcbot · 2024-08-08T21:42:37Z

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 26 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_S2SWA_gefs_cb158a81

emcbot · 2024-08-08T21:42:38Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 26 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96C48_hybatmDA_cb158a81

emcbot · 2024-08-08T21:42:43Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 26 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96_atm3DVar_cb158a81

emcbot · 2024-08-08T21:42:52Z

CI Failed on Hercules in Build# 26
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2815


Experiment C48_ATM_cb158a81 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 16:10:21 CDT 2024
Experiment C48_ATM_cb158a81 Terminated: *FAIL*
Experiment C48_S2SW_cb158a81 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 16:10:21 CDT 2024
Experiment C48_S2SW_cb158a81 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_ATM_cb158a81/logs/2021032312/gfsfcst.log
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_S2SW_cb158a81/logs/2021032312/gfsfcst.log

emcbot · 2024-08-08T21:57:33Z

Experiment C48_ATM FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_ATM_b4d5ed66

emcbot · 2024-08-08T22:09:28Z

Experiment C48_S2SW FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C48_S2SW_b4d5ed66

emcbot · 2024-08-08T22:39:53Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96C48_hybatmDA_b4d5ed66

emcbot · 2024-08-08T22:45:52Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 29 in
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/EXPDIR/C96_atm3DVar_b4d5ed66

emcbot · 2024-08-09T01:30:27Z

CI Failed on Hercules in Build# 29
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2815


Experiment C48_ATM_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 16:57:17 CDT 2024
Experiment C48_ATM_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_ATM_b4d5ed66/logs/2021032312/gfsfcst.log
Experiment C48_S2SW_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:09:21 CDT 2024
Experiment C48_S2SW_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C48_S2SW_b4d5ed66/logs/2021032312/gfsfcst.log
Experiment C96C48_hybatmDA_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:39:39 CDT 2024
Experiment C96C48_hybatmDA_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C96C48_hybatmDA_b4d5ed66/logs/2021122100/gfsfcst.log
Experiment C96_atm3DVar_b4d5ed66 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Aug  8 17:45:38 CDT 2024
Experiment C96_atm3DVar_b4d5ed66 Terminated: *FAIL*
Error logs:
/work2/noaa/stmp/CI/HERCULES/2815/RUNTESTS/COMROOT/C96_atm3DVar_b4d5ed66/logs/2021122100/gfsfcst.log
Experiment C48_S2SWA_gefs_b4d5ed66 Completed 1 Cycles: *SUCCESS* at Thu Aug  8 20:30:16 CDT 2024

emcbot · 2024-08-09T20:45:22Z

CI Passed on Hercules in Build# 30
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2815


Experiment C48_ATM_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 13:19:10 CDT 2024
Experiment C96_atm3DVar_a670daf2 Completed 3 Cycles: *SUCCESS* at Fri Aug  9 14:50:04 CDT 2024
Experiment C48_S2SW_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 15:32:21 CDT 2024
Experiment C96C48_hybatmDA_a670daf2 Completed 3 Cycles: *SUCCESS* at Fri Aug  9 15:32:28 CDT 2024
Experiment C48_S2SWA_gefs_a670daf2 Completed 1 Cycles: *SUCCESS* at Fri Aug  9 15:45:13 CDT 2024

TerrenceMcGuinness-NOAA added 3 commits August 7, 2024 12:45

add global default env variable for gh the GitHub CLI executable in a…

499401d

…ccount running agents for EMC global-workflows

add global default env variable for gh the GitHub CLI executable in a…

0ee59ea

…ccount running agents for EMC global-workflows

reverted to authrative URL after testing

2f49a04

TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Aug 7, 2024

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA August 7, 2024 17:08

TerrenceMcGuinness-NOAA self-assigned this Aug 7, 2024

added a catcherror block around running experments to capture fails o…

291d1de

…n the Jenkins Controller

TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Aug 7, 2024

WalterKolczynski-NOAA requested changes Aug 7, 2024

View reviewed changes

Comment thread ci/Jenkinsfile Outdated

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Aug 7, 2024

DavidHuber-NOAA reviewed Aug 7, 2024

View reviewed changes

Comment thread ci/Jenkinsfile Outdated

TerrenceMcGuinness-NOAA and others added 2 commits August 7, 2024 14:31

Update ci/Jenkinsfile

a12d611

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

reverted suggestion and added indent

a101111

TerrenceMcGuinness-NOAA removed the CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules label Aug 7, 2024

added build number to final message

e7b38e2

TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Aug 7, 2024

TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Aug 7, 2024

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Aug 7, 2024

make GH a global variable and assign it once the Node conncetion is e…

ffbd2be

…stablished

emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Aug 7, 2024

TerrenceMcGuinness-NOAA added 11 commits August 8, 2024 17:29

parallel section needs to be in a script under steps

20f3528

still trying to get stag steps hiarchy in parallel block

530bd19

third attept at getting stages in the right place

246bac4

fourth with aliginging brackets

7659b37

moved script inside of stages

0bfe315

removed extra stage block

b7e48be

removed extra steps block too

f154fd6

removed steps back and try to solve when again

7e40a90

removed when and used if instead sence they can not be used inside steps

a7ab366

no steps in stepsgit add Jenkinsfile

5c1c666

parallelStages is working now, rename loop contol variable for caseName

cb158a8

cleaned up code after getting parrallel working

5f5df7a

matrix stage labels must be literal

b4d5ed6

TerrenceMcGuinness-NOAA added 2 commits August 9, 2024 12:52

added dyamic case list

a670daf

removed hard coded list cases

b2f6c60

Reverse order in case label to match Create

05a83cc

WalterKolczynski-NOAA approved these changes Aug 13, 2024

View reviewed changes

Merge branch 'NOAA-EMC:develop' into jenkins_shell_env_update

beee137

Conversation

TerrenceMcGuinness-NOAA commented Aug 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How has this been tested?

Uh oh!

Uh oh!

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 8, 2024

Uh oh!

emcbot commented Aug 9, 2024

Uh oh!

emcbot commented Aug 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TerrenceMcGuinness-NOAA commented Aug 7, 2024 •

edited

Loading