Launch Multiple Platforms to Jenkins with polling by TerrenceMcGuinness-NOAA · Pull Request #2548 · NOAA-EMC/global-workflow

TerrenceMcGuinness-NOAA · 2024-04-27T00:13:46Z

Description

When launching Jenkins CI Tests all requested RDHPCS machines can now be selected via the Ready label at once.
This is all that is required sense polling also works now on the controller end.

Killing the jobs still needs to be done directly on the Jenkins Controller.
Do not try to update the CI process using Labels. It is for launching only.
Jenkins will update the labels as the states change.

NOTE: When a case fails the label will be update to a Fail for that system but will continue to report failures of subsequent cases until otherwise stopped in the Jenkins Controller directly.

Do not update any labels to Ready until a push has been made to the repo of the PR, and
The job has been completed or stopped directly on the controller first.
You can do this at anytime without having to wait until the other machines are completed.

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

This PR will be self tested by running Jenkins on it with three labels at once

emcbot · 2024-04-29T14:50:01Z

Experiment C48_S2SWA_gefs FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/COMROOT/C48_S2SWA_gefs_19605adc/logs/2021032312/wave_init.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-04-29T14:50:07Z

Experiment C48_S2SWA_gefs FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/C48_S2SWA_gefs_19605adc

emcbot · 2024-04-29T14:54:58Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Mon Apr 29 14:43:40 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 04/29/24 02:54:50 PM
Case setup: Completed for experiment C48_ATM_19605adc
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_19605adc
Case setup: Skipped for experiment C48_S2SWA_gefs_19605adc
Case setup: Completed for experiment C48_S2SW_19605adc
Case setup: Completed for experiment C96_atm3DVar_19605adc
Case setup: Skipped for experiment C96_atmaerosnowDA_19605adc
Case setup: Completed for experiment C96C48_hybatmDA_19605adc
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_19605adc

emcbot · 2024-04-29T15:36:56Z

Experiment C96C48_hybatmDA FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/COMROOT/C96C48_hybatmDA_19605adc/logs/2021122100/gdasverfozn.log
/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/COMROOT/C96C48_hybatmDA_19605adc/logs/2021122100/gdasverfrad.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-04-29T15:36:57Z

Experiment C96_atm3DVar FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/C96_atm3DVar_19605adc

emcbot · 2024-04-29T15:37:04Z

Experiment C96C48_hybatmDA FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/C96C48_hybatmDA_19605adc

emcbot · 2024-04-29T16:06:14Z

Experiment C48_ATM_19605adc SUCCESS on Wcoss2 at 04/29/24 04:06:13 PM

emcbot · 2024-04-29T16:09:18Z

Experiment C48_ATM FAILED on Orion with error logs:

/work2/noaa/stmp/CI/ORION/2548/RUNTESTS/COMROOT/C48_ATM_19605adc/logs/2021032312/gfsfcst.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-04-29T16:09:24Z

Experiment C48_ATM FAILED on Orion in
/work2/noaa/stmp/CI/ORION/2548/RUNTESTS/C48_ATM_19605adc

emcbot · 2024-04-29T16:56:22Z

Experiment C96C48_hybatmDA FAILED on Orion with error logs:

/work2/noaa/stmp/CI/ORION/2548/RUNTESTS/COMROOT/C96C48_hybatmDA_19605adc/logs/2021122100/gfsatmos_prod_f036-f042.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-04-29T16:56:29Z

Experiment C96C48_hybatmDA FAILED on Orion in
/work2/noaa/stmp/CI/ORION/2548/RUNTESTS/C96C48_hybatmDA_19605adc

emcbot · 2024-04-29T17:12:18Z

Experiment C96_atm3DVar_19605adc SUCCESS on Wcoss2 at 04/29/24 05:12:17 PM

emcbot · 2024-04-29T17:12:28Z

Experiment C96C48_hybatmDA_19605adc SUCCESS on Wcoss2 at 04/29/24 05:12:28 PM

emcbot · 2024-04-29T17:18:11Z

Experiment C48_S2SW_19605adc SUCCESS on Wcoss2 at 04/29/24 05:18:10 PM

emcbot · 2024-04-29T17:21:12Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_19605adc *** SUCCESS *** at 04/29/24 04:06:13 PM
Experiment C96_atm3DVar_19605adc *** SUCCESS *** at 04/29/24 05:12:17 PM
Experiment C96C48_hybatmDA_19605adc *** SUCCESS *** at 04/29/24 05:12:28 PM
Experiment C48_S2SW_19605adc *** SUCCESS *** at 04/29/24 05:18:10 PM

emcbot · 2024-04-29T20:18:07Z

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2548

WalterKolczynski-NOAA · 2024-04-29T20:43:31Z

Seem like this PR is successfully doing what it set out to do and any remaining issues on MSU machines are unrelated.

emcbot · 2024-04-29T23:31:21Z

Experiment C48_S2SW FAILED on Hercules with error logs:

/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/COMROOT/C48_S2SW_19605adc/logs/2021032312/gfswavepostpnt.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-04-29T23:31:35Z

Experiment C48_S2SW FAILED on Hercules in
/work2/noaa/stmp/CI/HERCULES/2548/RUNTESTS/C48_S2SW_19605adc

emcbot · 2024-04-30T03:50:47Z

Experiment C96_atm3DVar FAILED on Orion in
/work2/noaa/stmp/CI/ORION/2548/RUNTESTS/C96_atm3DVar_19605adc

TerrenceMcGuinness-NOAA added 30 commits April 26, 2024 19:57

first file update for multilable feature into main Jenkinsfile

fbbb2a7

debug out env.Node as it was not flagging the parent job

a6bb515

damn env.Node is now nulll checking for BUILD_CAUSE

b816340

echo out getCauses

abd87cc

found consice way to check for spawned jobs

8363a2d

moved scope of run_nodes

54343a7

are we failing on pullRequests again

a6f023b

adding more debug statments for nodes found and spawning

8007c97

testing each llooop

f6a73e7

just trying two nodes

2c0e9fe

added debug running common works space on machine in question

2fc9f2c

test machien assiginment

794aca1

echo start of get machine

6700292

still tryinig anything at this point

435d57f

it was the regex object I bet

8fa8887

surgecly removeing reg

4276202

surgecly removeing reg

02977fa

assigning a matcher is a mistake

5ad0c8e

release the matcher

49792a9

force match to bool and test bool

0fb30e2

force match to bool and test bool

5a6db2b

force match to bool and test bool

9beae19

null after bool on matcher

dedbadc

null after bool on matcher

d9f1ad2

null after bool on matcher

b35c940

removed matcher and used lable.matches

d449c27

removed matcher and used lable.matches

b263ebd

lable is label

16a45c2

vars did not need $ evaluations

ed21a60

finally works removin one extra debug line

7e4c6f0

emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Apr 29, 2024

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Apr 29, 2024

emcbot added CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 labels Apr 29, 2024

emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Apr 29, 2024

emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Apr 29, 2024

Conversation

TerrenceMcGuinness-NOAA commented Apr 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

WalterKolczynski-NOAA commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 29, 2024

Uh oh!

emcbot commented Apr 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TerrenceMcGuinness-NOAA commented Apr 27, 2024 •

edited

Loading