Remove RUNDIRS before running CI cases to cover re-run events by TerrenceMcGuinness-NOAA · Pull Request #3076 · NOAA-EMC/global-workflow

TerrenceMcGuinness-NOAA · 2024-11-07T17:01:33Z

Description

This Pr removes pslot dir (with hash) from local archive folder on CI case completion and also adds cleanup_experiment function in the BASH utilities to clean up after a CI case runs. This change consolidates cleanup actions and adds functionality to clean the local archive folders.

Key changes:

Fixes bug for cleaning up the RUNDIRS directory before new experiments are run in the case the pipeline is being ran for a second time
Updated $HOMEgfs/ci/scripts/check_ci.sh to use the new cleanup_experiment function under the BASH CI system.
Added cleanup_experiment function in $HOMEgfs/ci/scripts/utils/ci_utils.sh to handle various cleanups including:
- COMROOT/EXPDIR per case
- ARCDIR and ATADIR directories for archive
- STMP/RUNDIRS/${PSLOT}
Added get config var from EXPDIR utility get_config_var.py for getting config values in BASH

Resolves #2954, #3066

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

This PR still needs to be tested with the CI system which would effectively be a self-test.
# Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have documented my code, including function, input, and output descriptions
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added
I have made corresponding changes to the system documentation if necessary

emcbot · 2024-11-07T17:14:01Z

Checkout Failed on Hera in Build# 2: Error cloning remote repo 'origin'

TerrenceMcGuinness-NOAA · 2024-11-07T17:47:16Z

Checkout failed for gfs but not for gefs on a basic git command on Hera. Simply restarting CI in attempts to repeat the error.
Error was not repeatable for the gfs checkout and proceeded normally. scm also did the checkout in normal git fashion with updates only on previously gefs side as well.

emcbot · 2024-11-07T21:42:57Z

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076


Experiment C48_ATM_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 13:41:03 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 14:05:23 CST 2024
Experiment C48_S2SW_6d349326 Completed 2 Cycles: *SUCCESS* at Thu Nov  7 15:06:18 CST 2024
Experiment C96_atm3DVar_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:18:15 CST 2024
Experiment C48_S2SWA_gefs_6d349326 Completed 1 Cycles: *SUCCESS* at Thu Nov  7 15:31:08 CST 2024
Experiment C96C48_hybatmDA_6d349326 Completed 3 Cycles: *SUCCESS* at Thu Nov  7 15:42:48 CST 2024

emcbot · 2024-11-09T02:25:32Z

Experiment C96C48_hybatmDA FAILED on Hera in Build# 8 in
/scratch1/NCEPDEV/global/CI/3076/RUNTESTS/EXPDIR/C96C48_hybatmDA_d9d63af5

emcbot · 2024-11-09T02:25:33Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hera in Build# 8 in
/scratch1/NCEPDEV/global/CI/3076/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_d9d63af5

emcbot · 2024-11-09T02:26:35Z

Checkout Failed on Orion in Build# 11: Error fetching remote repo 'origin'

emcbot · 2024-11-09T02:26:36Z

Checkout Failed on Orion in Build# 11: Error fetching remote repo 'origin'

…f parse_yaml line when getting STMP from host file

emcbot · 2024-11-09T02:30:44Z

Experiment C48_ATM FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C48_ATM_45b565a6

emcbot · 2024-11-09T02:30:45Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C96_atm3DVar_45b565a6

emcbot · 2024-11-09T02:30:45Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C96C48_hybatmDA_45b565a6

emcbot · 2024-11-09T02:30:48Z

Experiment C96_S2SWA_gefs_replay_ics FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C96_S2SWA_gefs_replay_ics_45b565a6

emcbot · 2024-11-09T02:30:49Z

Experiment C48_S2SWA_gefs FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C48_S2SWA_gefs_45b565a6

emcbot · 2024-11-09T02:30:54Z

Experiment C48_S2SW FAILED on Hercules in Build# 9 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C48_S2SW_45b565a6

emcbot · 2024-11-09T02:37:10Z

Checkout Failed on Orion in Build# 14: Error fetching remote repo 'origin'

fatal: write error: Disk quota exceeded

emcbot · 2024-11-09T02:37:11Z

Checkout Failed on Orion in Build# 14: Error fetching remote repo 'origin'

emcbot · 2024-11-09T02:47:04Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 12 in
/work2/noaa/global/CI/HERCULES/3076/RUNTESTS/EXPDIR/C96C48_hybatmDA_db78344e

emcbot · 2024-11-09T06:51:08Z

CI Passed on Hercules in Build# 17
Built and ran in directory /work2/noaa/global/CI/HERCULES/3076


Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Fri Nov  8 22:31:17 CST 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Fri Nov  8 22:43:24 CST 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 00:14:57 CST 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:26:20 CST 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 00:38:24 CST 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 00:50:51 CST 2024

emcbot · 2024-11-09T12:11:12Z

CI Passed on Hera in Build# 18
Built and ran in directory /scratch1/NCEPDEV/global/CI/3076


Experiment C48_ATM_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 07:18:41 UTC 2024
Experiment C96_S2SWA_gefs_replay_ics_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 07:49:07 UTC 2024
Experiment C48mx500_3DVarAOWCDA_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 08:19:42 UTC 2024
Experiment C48_S2SWA_gefs_1e17407f Completed 1 Cycles: *SUCCESS* at Sat Nov  9 09:40:22 UTC 2024
Experiment C48_S2SW_1e17407f Completed 2 Cycles: *SUCCESS* at Sat Nov  9 09:51:05 UTC 2024
Experiment C96C48_hybatmaerosnowDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:45:46 UTC 2024
Experiment C96_atm3DVar_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 10:51:50 UTC 2024
Experiment C96C48_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 11:04:37 UTC 2024
Experiment C96C48_ufs_hybatmDA_1e17407f Completed 3 Cycles: *SUCCESS* at Sat Nov  9 12:11:02 UTC 2024

emcbot · 2024-11-09T12:57:53Z

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

emcbot · 2024-11-09T12:57:53Z

Checkout Failed on Orion in Build# 19: Error fetching remote repo 'origin'

TerrenceMcGuinness-NOAA · 2024-11-12T14:25:59Z

Massive failure of scm checkout of global-workflow on Orion.
Looking into a lead with this part of the error log.

fatal: write error: Disk quota exceeded
fatal: fetch-pack: invalid index-pack output

WalterKolczynski-NOAA · 2024-11-12T16:35:14Z

PR description needs to be updated before merging.

addeing no up for ci develop branch self test

6d34932

TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Nov 7, 2024

TerrenceMcGuinness-NOAA self-assigned this Nov 7, 2024

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA as a code owner November 7, 2024 17:01

TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Nov 7, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Nov 7, 2024

TerrenceMcGuinness-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 7, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Nov 9, 2024

TerrenceMcGuinness-NOAA removed the CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed label Nov 9, 2024

fixed cut and paste error by removing extranous grep sring in front o…

40bb91d

…f parse_yaml line when getting STMP from host file

adding :? at the end of rm paths for shellnorms

db78344

left out utils in path to pars_yaml utility

bc47272

TerrenceMcGuinness-NOAA added 2 commits November 8, 2024 21:00

OMG still had srting wrong to getting STMP with unbalanced quotes

1e17407

extra white spaces in get config python script

9727149

TerrenceMcGuinness-NOAA mentioned this pull request Nov 9, 2024

Remove pslot dir from local archive folder on CI case completion #2961

Closed

12 tasks

Merge branch 'NOAA-EMC:develop' into ci_develop_selftest

2bb7709

WalterKolczynski-NOAA approved these changes Nov 12, 2024

View reviewed changes

Conversation

TerrenceMcGuinness-NOAA commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Uh oh!

emcbot commented Nov 7, 2024

Uh oh!

TerrenceMcGuinness-NOAA commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emcbot commented Nov 7, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024 • edited by TerrenceMcGuinness-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

emcbot commented Nov 9, 2024

Uh oh!

TerrenceMcGuinness-NOAA commented Nov 12, 2024

Uh oh!

WalterKolczynski-NOAA commented Nov 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TerrenceMcGuinness-NOAA commented Nov 7, 2024 •

edited

Loading

TerrenceMcGuinness-NOAA commented Nov 7, 2024 •

edited

Loading

emcbot commented Nov 9, 2024 •

edited by TerrenceMcGuinness-NOAA

Loading