Use err_chk in GSI J-Jobs and scripts by DavidHuber-NOAA · Pull Request #3549 · NOAA-EMC/global-workflow

DavidHuber-NOAA · 2025-04-04T14:56:29Z

Description

This converts exit commands in GSI J-Jobs and ex scripts to err_chk commands to comply with EE2 standards.

Resolves #3509
Refs #294

Type of change

NCO bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

CI tests on C6. All tests passed.

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added

aerorahul

a few shellcheck corrections and a suggestion

aerorahul · 2025-04-04T15:25:27Z

+set -o pipefail
 LEVS=$(${NCDUMP} -h "${ATMGES_ENSMEAN}" | grep -i "pfull" | head -1 | awk -F" = " '{print $2}' | awk -F" " '{print $1}') # get LEVS
-# shellcheck disable=
-err=$?
-if [[ ${err} -ne 0 ]]; then
-    exit "${err}"
-fi
+export err=$?


See if you can use getncdimlen?

Below in this PR, you will see

export LEVS=$(${NCLEN} ${ATMGES} pfull) && true

aerorahul · 2025-04-04T19:03:33Z


 $NCP $ENKFEXEC $DATA
-$APRUN_ENKF ${DATA}/$(basename $ENKFEXEC) 1>stdout 2>stderr
+$APRUN_ENKF "${DATA}/$(basename $ENKFEXEC)" 1>stdout 2>stderr &&


Suggested change

$APRUN_ENKF "${DATA}/$(basename $ENKFEXEC)" 1>stdout 2>stderr &&

${APRUN_ENKF} "${DATA}/$(basename $ENKFEXEC)" 1>stdout 2>stderr &&

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

emcbot · 2025-04-07T14:46:02Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 3 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_e1a089c0

emcbot · 2025-04-07T14:46:04Z

Experiment C48mx500_hybAOWCDA FAILED on Hercules in Build# 3 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_e1a089c0

RussTreadon-NOAA · 2025-04-07T14:50:22Z

@DavidHuber-NOAA documented the reason for the gdas_prepoceanobs failure in issue #3555. PR #3556 has been opened with the fix.

emcbot · 2025-04-07T19:25:10Z

CI Failed on Hercules in Build# 3
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549


Experiment C48mx500_hybAOWCDA_e1a089c0 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Mon Apr  7 09:45:42 CDT 2025
Experiment C48mx500_hybAOWCDA_e1a089c0 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_e1a089c0/logs/2021032500/gdas_prepoceanobs.log
Experiment C48mx500_3DVarAOWCDA_e1a089c0 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Mon Apr  7 09:45:44 CDT 2025
Experiment C48mx500_3DVarAOWCDA_e1a089c0 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_e1a089c0/logs/2021032500/gdas_prepoceanobs.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_e1a089c0/logs/2021032500/gfs_prepoceanobs.log
Experiment C48_ATM_e1a089c0 Completed 1 Cycles: *SUCCESS* at Mon Apr  7 10:40:01 CDT 2025
Experiment C96mx100_S2S_e1a089c0 Completed 1 Cycles: *SUCCESS* at Mon Apr  7 11:16:19 CDT 2025
Experiment C48_S2SW_e1a089c0 Completed 1 Cycles: *SUCCESS* at Mon Apr  7 11:22:27 CDT 2025
Experiment C96C48_hybatmDA_e1a089c0 Completed 3 Cycles: *SUCCESS* at Mon Apr  7 13:29:40 CDT 2025
Experiment C96_atm3DVar_e1a089c0 Completed 3 Cycles: *SUCCESS* at Mon Apr  7 13:29:44 CDT 2025
Experiment C48_S2SWA_gefs_e1a089c0 Completed 1 Cycles: *SUCCESS* at Mon Apr  7 14:24:56 CDT 2025

emcbot · 2025-04-08T20:08:17Z

CI Passed on Hercules in Build# 4
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3549


Experiment C48_ATM_17356cb3 Completed 1 Cycles: *SUCCESS* at Tue Apr  8 12:35:50 CDT 2025
Experiment C96mx100_S2S_17356cb3 Completed 1 Cycles: *SUCCESS* at Tue Apr  8 13:24:23 CDT 2025
Experiment C48mx500_hybAOWCDA_17356cb3 Completed 2 Cycles: *SUCCESS* at Tue Apr  8 13:30:26 CDT 2025
Experiment C96_atm3DVar_17356cb3 Completed 3 Cycles: *SUCCESS* at Tue Apr  8 14:24:58 CDT 2025
Experiment C48_S2SW_17356cb3 Completed 1 Cycles: *SUCCESS* at Tue Apr  8 14:31:05 CDT 2025
Experiment C96C48_hybatmDA_17356cb3 Completed 3 Cycles: *SUCCESS* at Tue Apr  8 14:55:25 CDT 2025
Experiment C48_S2SWA_gefs_17356cb3 Completed 1 Cycles: *SUCCESS* at Tue Apr  8 15:02:36 CDT 2025
Experiment C48mx500_3DVarAOWCDA_17356cb3 Completed 2 Cycles: *SUCCESS* at Tue Apr  8 15:07:56 CDT 2025

KateFriedman-NOAA

Looks good, thanks @DavidHuber-NOAA !

aerorahul

lgtm

* develop: Update GSI hash and GSI fix version to resolve bugs (NOAA-EMC#3626) Add missing marine DA files to archiving (NOAA-EMC#3596) Add a low resolution test to mimic GFSv17 cycling as much as possible (NOAA-EMC#3617) Add the setting to use the reject list for station t/q observations in GSI based soil DA (NOAA-EMC#3599) GitLab CI Framework for schedule PR cases and ctests on multi hosts (NOAA-EMC#3603) Avoid parallel restart I/O on WCOSS2 (NOAA-EMC#3615) Enables user toggling of GDASApp g-w ctests (NOAA-EMC#3587) COM variable updates for prep and some external downstream jobs (NOAA-EMC#3608) Remove MOS from system (NOAA-EMC#3612) Updates to enable soil DA (NOAA-EMC#3452) Unexport SHELLOPTS when running htar (NOAA-EMC#3601) Fix check for netcdf wave restart (NOAA-EMC#3594) Call err_chk/err_exit for fatal errors in post JJobs/ex-scripts (NOAA-EMC#3571) Remove support for Jet and S4 (NOAA-EMC#3572) Hotfix in GitLab pipline for Nightly (env MACHINE breaks build on head node) (NOAA-EMC#3578) [hotfix] Missed a path during merging develop (NOAA-EMC#3577) Prepare for ops readiness - part 1 (NOAA-EMC#3557) Update UFS weather-model to 20250328 hash (NOAA-EMC#3528) Fix SFS fcst config (NOAA-EMC#3574) Use err_chk in GDAS j-jobs (NOAA-EMC#3570) Perform compute builds on Gaea head nodes (NOAA-EMC#3560) Add initial capability to produce JEDI-based observation space summary stat files (NOAA-EMC#3471) Spread epos over more nodes on Hera to increase allocated memory (NOAA-EMC#3567) Create separate gists when multiple files are published on GitHub (NOAA-EMC#3551) Use err_chk in GSI J-Jobs and scripts (NOAA-EMC#3549) Add unified jinja obs list to marine DA (NOAA-EMC#3530) Save snow and aerosol analysis increments (and logs and YAMLs) every cycle (NOAA-EMC#3537) Add Dependencies to SFS Cleanup Job (NOAA-EMC#3559) Updates archiving to reflect current naming of marine anl output (NOAA-EMC#3541) Temporarily disable compute builds on C6 (NOAA-EMC#3558) Update gdas.cd hash to resolve msu prod_util failure (NOAA-EMC#3556) COMIN/COMOUT updates for enkf chgres and downstream product jobs (NOAA-EMC#3518) Call err_chk in forecast scripts for fatal errors (NOAA-EMC#3515) Add Rocoto Jobs for the Missing Products of GEFS (NOAA-EMC#3466) Download subset fix data with python script (NOAA-EMC#3400) Check that partition should be set (NOAA-EMC#3543) Rename wave output and refactor some wave scripts to use MPMD, and fix some bugzillas along the way (NOAA-EMC#3517) Add support for dual batch partitions on AWS NOAA-EMC#3483 Update CI build and run directories for GitLab Nightlies on C6 and added GitLab support on Hera (NOAA-EMC#3536) Hotfix path for CI in Jenkins on Gaea C6 to it's world-share path (NOAA-EMC#3532) Create single ocean grib2 product file (NOAA-EMC#3529) Scheduled Nightly CI/CD Pipeline Script in GitLab on Gaea C6 (NOAA-EMC#3493) make sure cold starts are handled correctly when DOIAU=YES (issue NOAA-EMC#3516) (NOAA-EMC#3520) Add check for DO_AERO_FCST before copying fv_tracer files (NOAA-EMC#3485) Use jinja templates instead of `@VARNAME@` in config files (NOAA-EMC#3411) Replace "status" (or comparable) with "err" in preparation for moving to err_chk/err_exit (NOAA-EMC#3507) Error in Java launch script for CI (NOAA-EMC#3465) Delete DATAROOT when running generate_workflows.sh (NOAA-EMC#3504) Fix 3244 garbled change (NOAA-EMC#3492) Enable ensemble archiving via Globus (NOAA-EMC#3479) Update MSU FIX_DIR paths (NOAA-EMC#3488) Updates for AOWCDA and hybatmaerosnowDA cases on Gaea C6 (NOAA-EMC#3487) Update GOCART path for GDAS/GFS/GCAFS implementations (NOAA-EMC#3455) Make RUN Variables Explicit in `config.resources` (NOAA-EMC#3478) Remove unused key from enkfgdas_earc_vrfy (NOAA-EMC#3473) Bug fix to the failing early cycle marine DA ensemble re-centering (NOAA-EMC#3454) Make marine LETKF optional (NOAA-EMC#3462) When sourcing for RUN=enkf*, use CASE_ENS (NOAA-EMC#3475) Updates for Gaea: verif-global tag, tracker tag, Fit2Obs tag, and C768 analysis resources (NOAA-EMC#3463) Update gefswave glo_025 mesh file with new mask (NOAA-EMC#3457) Update MSU glopara paths to new role-global space (NOAA-EMC#3443) Enable CI testing on AWS (NOAA-EMC#3459) Enable Gaea C5 Jenkins CI (NOAA-EMC#3447) Job reference removal from WMO product names (NOAA-EMC#3460) Turn off aerosol prognostic radiative feedback for GDAS NOAA-EMC#2926 (NOAA-EMC#3445) Add DO_GEMPAK check to postsnd subtask (NOAA-EMC#3451) Add a force option to setup_xml to ignore unwritable directories (NOAA-EMC#3448) Remove the eomg job (NOAA-EMC#3331) Migration to role account for Jenkins on Orion (NOAA-EMC#3440) Eliminate `_gfs`, `_gdas`, etc, variables and add necessary if blocks (NOAA-EMC#3420) Update workflow staging for sfcanl tiles and waveinit (NOAA-EMC#3429) Improve messaging to display clear warning when missing snogrb file (NOAA-EMC#3317) JEDI-based ensemble recentering and analysis calculation (NOAA-EMC#3312) Enable HPSS archiving on C5/6 (NOAA-EMC#3437) Check if HOMEDIR STMP and PTMP are writable (NOAA-EMC#3430) Update UFS_Utils and GFS-utils hashes to update Gaea support and ocean/ice post products (NOAA-EMC#3433) Enable C1152 forecasts on gaea C6 (NOAA-EMC#3438) Migration to role account for Jenkins on Hercules (NOAA-EMC#3423) Remove Direct Linking to COM from DATA for `extractvars` Job (NOAA-EMC#3379) Enable HPSS via Globus on Hercules and Orion Remove job name from product files & update GEMPAK module. (NOAA-EMC#3415) `link` instead of `copy` in staging jobs (NOAA-EMC#3410) Migrate CI Jenkins to role account on Hera (NOAA-EMC#3414) Add rocotorc documentation when using scrontab (NOAA-EMC#3417) Update jgdas atmos verfozn and verfrad with COMIN/COMOUT prefix instead of COM (NOAA-EMC#3342) Add configuration for empirically-corrected ozone parameters (NOAA-EMC#3386) Enable global-workflow to run C768C384 GSI on Gaea-C6 (NOAA-EMC#3412) Move logical checks into if blocks (NOAA-EMC#3339) Adding Jenkins CI to GaeaC6 using role account (NOAA-EMC#3389) Enable GDASApp g-w CI cases to run on wcoss2 (NOAA-EMC#3399) CI/CD Test on Gaea C5- And update config.gaea under ci/platform (NOAA-EMC#3280) Enable cycling support for Gaea C6 (NOAA-EMC#3323) Update enkf archive jobs to use COMIN/COMOUT (NOAA-EMC#3393) Copy marine ensemble output observation diags and spread (NOAA-EMC#3407) Ci testing on aws 2 (NOAA-EMC#3408) Disable METplus jobs on Hera (NOAA-EMC#3403) Add the mean EnKF soil increment to the deterministic member (NOAA-EMC#3295) Add mpich/8.1.19 to the WCOSS2 LD_LIBRARY_PATH for GDASApp jobs (NOAA-EMC#3396) Change order of RUNs (NOAA-EMC#3335) CI testing on aws (NOAA-EMC#3391) Rename Gulf of Mexico in bufr station list in GFSv17 (NOAA-EMC#3384) Enabling AWS CI/testing (NOAA-EMC#3383) Update issue templates to use new issue type field (NOAA-EMC#3369) Replace WAVECUR_DID variable with "rtofs" (NOAA-EMC#3337) Allow for C1152 ATM-Aero cycled DA to run on WCOSS2 (NOAA-EMC#3309) Remove Direct Linking to COM from DATA for `wavepostsbs` Job (NOAA-EMC#3303) Update jgdas enkf update job with COMIN or COMOUT prefix instead of COM (NOAA-EMC#3333) Add capability to run diff resolutions for marine anl and background (NOAA-EMC#3238) Update high resolution tests and fix minor wave issues (NOAA-EMC#3289) Add sfs as valid system (NOAA-EMC#3243) Add missing arch_tars dependencies (NOAA-EMC#3319) Fix the empty aerosol DA aerostat tar file issue (NOAA-EMC#3332) Add missing file safeguard for IMS prep in snow analysis tasks (NOAA-EMC#3329) Fix memory unsetting on Gaea (NOAA-EMC#3325) Fix error log parsing in compute build CI (NOAA-EMC#3301) Remove marineanlvrfy task from global-workflow (NOAA-EMC#3314) Add `gfs_wavepostpnt` dependencies to gfs_cleanup (NOAA-EMC#3313) Increase the GDASApp build wallclock (NOAA-EMC#3298) Capture build fail in Jenkins pipeline when no error logs are produced (NOAA-EMC#3297) Add/update config files for Gaea and check existence before sourcing config files in generate_workflows.sh (NOAA-EMC#3286) Fix ocean restarts when cold starting with DOIAU=YES (NOAA-EMC#3278) Splitting up the archive task (NOAA-EMC#3242) CTests extended validation for C48_ATM and staged C48_S2SW for gfs_fcst and gfs_atmos (NOAA-EMC#3256) Add esnowanl to enkfgfs cycle (NOAA-EMC#3283) Add gfs cycles to C48mx500_3DVarAOWCDA (NOAA-EMC#3249) Add fetch job and update stage_ic to work with fetched ICs (NOAA-EMC#3141) Remove WAFS files and references from `develop` (NOAA-EMC#3263) fix intel stack version number on c5 (NOAA-EMC#3258) Update gsi_monitor and ufs_utils hashes to recent hashes for C5/C6 build and run (NOAA-EMC#3252) Enable DA cycling on gaea C5/C6 (NOAA-EMC#3255) Copy post-processed sea ice increment for diagnostics (NOAA-EMC#3235)

DavidHuber-NOAA added 3 commits April 3, 2025 14:49

Merge err_chk forecast branch

aa0a2e7

Initial stab at err_chk-ing GSI jobs

02f0115

Merge remote-tracking branch 'origin/develop' into feature/gsi_err

dba190b

DavidHuber-NOAA requested review from CatherineThomas-NOAA, CoryMartin-NOAA, EdwardSafford-NOAA, GeorgeGayno-NOAA and RussTreadon-NOAA as code owners April 4, 2025 14:56

Double quote err

170e77e

DavidHuber-NOAA requested review from KateFriedman-NOAA and aerorahul April 4, 2025 18:01

KateFriedman-NOAA reviewed Apr 4, 2025

View reviewed changes

DavidHuber-NOAA added 2 commits April 4, 2025 18:45

Add missing true

c2f3a87

Run err_chk on separate line

72018e7

aerorahul reviewed Apr 4, 2025

View reviewed changes

DavidHuber-NOAA commented Apr 4, 2025

View reviewed changes

Comment thread scripts/exgdas_enkf_post.sh Outdated

DavidHuber-NOAA commented Apr 4, 2025

View reviewed changes

Comment thread scripts/exgdas_enkf_sfc.sh Outdated

DavidHuber-NOAA and others added 2 commits April 4, 2025 16:06

Apply suggestions from code review

2fc4a88

Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

Double quote pgmout

43ef9aa

DavidHuber-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Apr 4, 2025

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Apr 4, 2025

Remove pipefail wrap that's no longer needed

e1a089c

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Apr 4, 2025

DavidHuber-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Apr 4, 2025

emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Apr 7, 2025

emcbot removed the CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress label Apr 7, 2025

Merge branch 'develop' into feature/gsi_err

5162e7f

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Apr 7, 2025

Merge remote-tracking branch 'origin/develop' into feature/gsi_err

17356cb

DavidHuber-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Apr 8, 2025

Do not log err_chk calls with error messages

f665740

DavidHuber-NOAA requested a review from WalterKolczynski-NOAA as a code owner April 8, 2025 15:03

emcbot added CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Apr 8, 2025

Merge branch 'develop' into feature/gsi_err

c30d08e

DavidHuber-NOAA requested review from KateFriedman-NOAA and aerorahul April 9, 2025 12:21

KateFriedman-NOAA approved these changes Apr 9, 2025

View reviewed changes

Merge branch 'develop' into feature/gsi_err

13d49ba

DavidHuber-NOAA merged commit 2a0318d into NOAA-EMC:develop Apr 9, 2025

aerorahul approved these changes Apr 9, 2025

View reviewed changes

DavidHuber-NOAA deleted the feature/gsi_err branch April 10, 2025 11:59

RussTreadon-NOAA mentioned this pull request Mar 10, 2026

ENKF_SELECT_OBS job silent failure #4615

Closed

	$APRUN_ENKF "${DATA}/$(basename $ENKFEXEC)" 1>stdout 2>stderr &&
	${APRUN_ENKF} "${DATA}/$(basename $ENKFEXEC)" 1>stdout 2>stderr &&

Conversation

DavidHuber-NOAA commented Apr 4, 2025

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aerorahul Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

emcbot commented Apr 7, 2025

Uh oh!

emcbot commented Apr 7, 2025

Uh oh!

RussTreadon-NOAA commented Apr 7, 2025

Uh oh!

emcbot commented Apr 7, 2025

Uh oh!

emcbot commented Apr 8, 2025

Uh oh!

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants