[dev/gfs.v17] Add informative error messages to all err_exit calls by AntonMFernando-NOAA · Pull Request #4895 · NOAA-EMC/global-workflow

AntonMFernando-NOAA · 2026-05-07T15:57:30Z

Description

This adds informative error messages to all err_exit calls. This was flagged as an issue for GCAFS and is being addressed across the board. It also removes err_exit calls from ush/ scripts in favor of exit calls.

Resolves (dev/gfs.v17) Add reason for failure into 'FATAL ERROR' message #4812
Resolves Add reason for failure into 'FATAL ERROR' message #4722

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO (If YES, please indicate to which system(s))
- GFS
- GEFS
- SFS
- GCAFS
Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

…s and echo error messages

…to err_exit calls

…ls across multiple job scripts

…ge in JGDAS_AERO_ANALYSIS_GENERATE_BMATRIX

AntonMFernando-NOAA · 2026-05-08T14:17:08Z

CI tests passed on wcoss2.
location: /lfs/h2/emc/stmp/anton.fernando/RUNTESTS/EXPDIR

  commit: 89faa496   branch: feature/gfsv17-fatal_error
  build: up to date
  submodules: up to date
Fri May  8 14:09:26 UTC 2026
******** C48_ATM_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 08 2026 03:35:12    May 08 2026 04:30:12

******** C48mx500_3DVarAOWCDA_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 08 2026 03:35:16    May 08 2026 03:50:12
202103250000        Done    May 08 2026 03:35:16    May 08 2026 05:10:13

******** C48mx500_hybAOWCDA_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 08 2026 03:38:47    May 08 2026 03:55:13
202103250000        Done    May 08 2026 03:38:47    May 08 2026 05:00:17

******** C48_S2SW_extended_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 08 2026 03:38:58    May 08 2026 06:05:10
202103231800        Done    May 08 2026 03:38:58    May 08 2026 05:45:10

******** C96_atm3DVar_extended_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 08 2026 03:39:00    May 08 2026 03:55:08
202112210000        Done    May 08 2026 03:39:00    May 08 2026 08:25:09
202112210600        Done    May 08 2026 03:39:00    May 08 2026 09:40:22
202112211200        Done    May 08 2026 04:00:53    May 08 2026 11:11:04
202112211800        Done    May 08 2026 08:30:10    May 08 2026 13:40:16

******** C96C48_hybatmDA_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 08 2026 03:38:49    May 08 2026 03:55:13
202112210000        Done    May 08 2026 03:38:49    May 08 2026 05:50:18
202112210600        Done    May 08 2026 03:38:49    May 08 2026 05:45:23

******** C96C48_hybatmsnowDA_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 08 2026 03:38:51    May 08 2026 03:55:11
202112201800        Done    May 08 2026 03:38:51    May 08 2026 05:45:11
202112210000        Done    May 08 2026 03:38:51    May 08 2026 05:50:11

******** C96C48_hybatmsoilDA_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    May 08 2026 03:38:53    May 08 2026 03:55:17
202205151200        Done    May 08 2026 03:38:53    May 08 2026 05:50:12
202205151800        Done    May 08 2026 03:38:53    May 08 2026 05:45:17

******** C96C48mx500_S2SW_cyc_gfs_t ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 08 2026 03:38:55    May 08 2026 03:55:14
202112201800        Done    May 08 2026 03:38:55    May 08 2026 06:25:25
202112210000        Done    May 08 2026 03:38:55    May 08 2026 06:55:16
202112211800        Done    May 08 2026 04:00:14    May 08 2026 07:05:17

JessicaMeixner-NOAA

I've looked at the wave updates and am good to go as long as CI passes on both gaea and wcoss2.

ChristopherHill-NOAA · 2026-05-08T16:52:34Z

For the following jobs and script:
JGDAS_ATMOS_GEMPAK
JGDAS_ATMOS_GEMPAK_META_NCDC
JGFS_ATMOS_AWIPS_20KM_1P0DEG
JGFS_ATMOS_GEMPAK_NCDC_UPAPGIF
JGFS_ATMOS_GEMPAK_PGRB2_SPEC
JGFS_ATMOS_PGRB2_SPEC_NPOESS
exgfs_atmos_nawips.sh

I recommend including the elaborated error messages that were accepted in #4793, specifically those from commit 1e7692b.

…e scripts

AntonMFernando-NOAA · 2026-05-08T18:58:41Z

For the following jobs and script: JGDAS_ATMOS_GEMPAK JGDAS_ATMOS_GEMPAK_META_NCDC JGFS_ATMOS_AWIPS_20KM_1P0DEG JGFS_ATMOS_GEMPAK_NCDC_UPAPGIF JGFS_ATMOS_GEMPAK_PGRB2_SPEC JGFS_ATMOS_PGRB2_SPEC_NPOESS exgfs_atmos_nawips.sh

I recommend including the elaborated error messages that were accepted in #4793, specifically those from commit 1e7692b.

@ChristopherHill-NOAA Thanks for pointing that out.

ChristopherHill-NOAA

Changes proposed for the atmospheric downstream product jobs are reflective of those in #4793, and are approved.

DavidNew-NOAA

Some suggestions for JEDI jobs

…ysis scripts for clarity and consistency

…ew WDQMS Python script for GSI diagnostic file processing

AntonMFernando-NOAA · 2026-05-11T18:01:45Z

@DavidNew-NOAA Made the suggested changes. Thanks!

CatherineThomas-NOAA

Thanks for incorporating my suggestions @AntonMFernando-NOAA. Approve.

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

DavidHuber-NOAA · 2026-05-18T14:46:34Z

Running tests on WCOSS2.

DavidHuber-NOAA · 2026-05-19T12:34:09Z

All tests passed on WCOSS2.

DavidHuber-NOAA and others added 8 commits May 6, 2026 11:33

Increase wallclock for gempak jobs to 10 minutes

cbcc193

Add messages to all err_exit/err_chk calls

7bdeec2

Refactor error handling in tropcy_relocate.sh to use direct exit call…

02c3182

…s and echo error messages

Enhance error handling in job scripts by adding descriptive messages …

c3e7330

…to err_exit calls

Enhance error handling by adding descriptive messages to err_exit cal…

0a6f216

…ls across multiple job scripts

Enhance error handling by adding a failure message for directory chan…

1406142

…ge in JGDAS_AERO_ANALYSIS_GENERATE_BMATRIX

Merge branch 'dev/gfs.v17' into feature/gfsv17-fatal_error

a3427ae

Merge branch 'dev/gfs.v17' into feature/gfsv17-fatal_error

89faa49

AntonMFernando-NOAA marked this pull request as ready for review May 8, 2026 14:17

JessicaMeixner-NOAA approved these changes May 8, 2026

View reviewed changes

DavidNew-NOAA approved these changes May 8, 2026

View reviewed changes

CatherineThomas-NOAA reviewed May 8, 2026

View reviewed changes

Comment thread dev/scripts/exgdas_atmos_chgres_forenkf.sh Outdated

Enhance error messages for product generation failures across multipl…

31ce868

…e scripts

ChristopherHill-NOAA approved these changes May 11, 2026

View reviewed changes

DavidNew-NOAA reviewed May 11, 2026

View reviewed changes

AntonMFernando-NOAA added 2 commits May 11, 2026 13:42

Refine error messages for aerosol, atmospheric, marine, and snow anal…

97390b7

…ysis scripts for clarity and consistency

Remove obsolete scripts for wave current and ice preprocessing; add n…

503bf04

…ew WDQMS Python script for GSI diagnostic file processing

CatherineThomas-NOAA approved these changes May 13, 2026

View reviewed changes

DavidHuber-NOAA reviewed May 14, 2026

View reviewed changes

Comment thread ush/tropcy_relocate.sh Outdated

DavidHuber-NOAA reviewed May 14, 2026

View reviewed changes

Comment thread ush/tropcy_relocate.sh Outdated

DavidHuber-NOAA added the Ready to Merge label May 14, 2026

DavidHuber-NOAA reviewed May 18, 2026

View reviewed changes

Comment thread dev/jobs/JGLOBAL_ENKF_ARCHIVE_VRFY

DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request May 18, 2026

Merge NOAA-EMC#4895

8ea4f5b

DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request May 18, 2026

Merge NOAA-EMC#4895

5edba69

AntonMFernando-NOAA and others added 3 commits May 18, 2026 09:55

Update ush/tropcy_relocate.sh

4e8f163

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/tropcy_relocate.sh

5730eb6

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update dev/jobs/JGLOBAL_ENKF_ARCHIVE_VRFY

8e5ac50

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

DavidHuber-NOAA removed the Ready to Merge label May 18, 2026

DavidHuber-NOAA added the CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress label May 18, 2026

DavidHuber-NOAA added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully Ready to Merge and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels May 19, 2026

DavidHuber-NOAA merged commit fa77a75 into NOAA-EMC:dev/gfs.v17 May 19, 2026
7 checks passed

DavidHuber-NOAA mentioned this pull request May 19, 2026

GFSv17 realtime silent failure in gfs_arch_vrfy #4890

Closed

DavidHuber-NOAA mentioned this pull request Jun 1, 2026

(dev/gfs.v17) Add reason for failure into 'FATAL ERROR' message #4812

Closed

AntonMFernando-NOAA deleted the feature/gfsv17-fatal_error branch June 3, 2026 19:27

Conversation

AntonMFernando-NOAA commented May 7, 2026 • edited by DavidHuber-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Uh oh!

AntonMFernando-NOAA commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ChristopherHill-NOAA commented May 8, 2026

Uh oh!

AntonMFernando-NOAA commented May 8, 2026

Uh oh!

ChristopherHill-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AntonMFernando-NOAA commented May 11, 2026

Uh oh!

CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented May 18, 2026

Uh oh!

DavidHuber-NOAA commented May 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

AntonMFernando-NOAA commented May 7, 2026 •

edited by DavidHuber-NOAA

Loading

AntonMFernando-NOAA commented May 8, 2026 •

edited

Loading