CI Refactoring and STALLED case detection

[TerrenceMcGuinness-NOAA]

Description

These updates to the CI Framework does some bash refactoring and adds python tools in order to effectively create the feature for detecting when a CI Case has an experiment that is in a state where it can not advance such as missing a requisite dependency:

• Added separate python script for checking the status of Rocoto driven cases and integrated its use into the bash CI drivers and Jenkins having state logic done in one place.

• Added log publishing python utilities into the bash CI drivers as part of refactoring and consolidations of functionalities

• Update Jenkins behavior while incorporating the above python codes for Rocoto state checking:

  • polling on PR works with one update away from including multiple labels as well
  • Label updates to FAIL as soon as first case fails and continues other cases until completes or is killed by user

  Resolves Feature BASH CI detects when a dependacy isn't being met #2008

Type of change

• New feature (adds functionality to detect stalled CI cases)
• Maintenance (code refactor, clean-up, new CI functionalities and behavior)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

BASH cases tested for stall and log failing reporting in dev bash cron
Jenkins tested in development mulit-branch project with fail tests and full-end-to end success path

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[emcbot]

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2488

[emcbot]

CI Update on Wcoss2 at 04/19/24 09:21:13 PM
============================================
Cloning and Building global-workflow PR: 2488
with PID: 70662 on host: dlogin08

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Fri Apr 19 21:25:06 UTC 2024 on dlogin08
---------------------------------------------------
Build: Completed at 04/19/24 09:36:26 PM
Case setup: Completed for experiment C48_ATM_07617928
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_07617928
Case setup: Skipped for experiment C48_S2SWA_gefs_07617928
Case setup: Completed for experiment C48_S2SW_07617928
Case setup: Completed for experiment C96_atm3DVar_07617928
Case setup: Skipped for experiment C96_atmaerosnowDA_07617928
Case setup: Completed for experiment C96C48_hybatmDA_07617928
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_07617928

[emcbot]

Experiment C48_ATM_07617928 SUCCESS on Wcoss2 at 04/19/24 10:52:10 PM

[emcbot]

Experiment C96C48_hybatmDA_07617928 SUCCESS on Wcoss2 at 04/19/24 11:52:24 PM

[emcbot]

Experiment C96_atm3DVar_07617928 SUCCESS on Wcoss2 at 04/19/24 11:56:11 PM

[emcbot]

Experiment C48_S2SW_07617928 SUCCESS on Wcoss2 at 04/20/24 12:08:14 AM

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_07617928 *** SUCCESS *** at 04/19/24 10:52:10 PM
Experiment C96C48_hybatmDA_07617928 *** SUCCESS *** at 04/19/24 11:52:24 PM
Experiment C96_atm3DVar_07617928 *** SUCCESS *** at 04/19/24 11:56:11 PM
Experiment C48_S2SW_07617928 *** SUCCESS *** at 04/20/24 12:08:14 AM