Update regional_workflow for use with authoritative UFS_UTILS utilities and add support for global/regional grib2 functionality in chgres_cube

[JeffBeck-NOAA]

DESCRIPTION OF CHANGES:

This PR must be tested with PR #17 from ufs-srweather-app

• Modifies j-jobs/exregional scripts to work with a single UFS_UTILS repository and the new version of chgres_cube, up-to-date with the authoritative UFS_UTILS repository.

• Adds support for Thompson climatology code within chgres_cube when using external model data that does not contain the ice- and water-friendly aerosol fields for Thompson MP. Fixed file for Thompson climatology code needs to be propagated to other platforms (currently resides only on Jet and Hera in ${FIXam}).

• Adds support for the new regional_grid without alpha and kappa parameters in the namelist (these variables are optimized in the new version of the code)

• Removes all utilities (/sorc directory) from regional_workflow

• Sources modulefiles from UFS_UTILS for build and run-time (removes "modulefiles/codes" directory and no longer uses fixed make_grid, mosaic_file, and global_equiv_resol modulefiles from "modulefiles/tasks")

• Removes all unnecessary surface-related variables and processing from exregional_make_lbcs.sh

TESTS CONDUCTED:

Tested successfully on Hera. Needs to be tested on at least Jet, WCOSS, and Cheyenne.

Note: Not all external model data and SDF combinations are supported by chgres_cube. Combining RAP or HRRR external model data with a Noah MP SDF will fail, since interpolation of 9 to 4 soil layers has not yet been implemented. Fixed in subsequent commit to Larissa's fork of UFS_UTILS. Using FV3GFS model data with an SDF containing RUC-LSM (4 to 9 soil levels) has a known bug that will be fixed shortly. - Fixed in subsequent commit to this PR.

CONTRIBUTORS (optional):

@LarissaReames-NOAA

[gsketefian]

The regional_014 test will be fixed as long as this PR makes its way into the feature/regional_release branch of Larissa's UFS_UTILS fork.

[gsketefian]

Here's a summary of the latest failed tests. I'd say start with trying to fix nco_GSD_HRRR3km_HRRRX_RAPX because that's what GSL wants to run. The experiment directories for all the WE2E tests I ran for this PR can be found under here (on hera):

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/expt_dirs/PR274_UFS_UTILS_20200902

Below, I list the test name followed by the grid, suite, and external model for ICs/LBCs that the test uses. Then the bullet points state which task fails and other info. You can look in the log files for detailed error messages.

GSD_RAP13km -- GSD_RAP13km, FV3_GSD_v0, FV3GFS/FV3GFS:

• 6 of 7 run_post tasks fail.
• Probably a bug in EMC_post. Not a priority to fix since this grid is not supported.

nco_GSD_HRRR3km_HRRRX_RAPX -- GSD_HRRR3km, FV3_GSD_SAR, HRRRX/RAPX:

• run_fcst task fails.
• The error is the same as for test regional_004, I think (error in ThompMP).

regional_003 -- GSD_HRRR25km, FV3_GSD_v0, HRRRX/RAPX:

• run_fcst task fails.

regional_004 -- GSD_HRRR25km, FV3_GSD_SAR, HRRRX/RAPX:

• run_fcst task fails.
• The error is the same as for test nco_GSD_HRRR3km_HRRRX_RAPX, I think (error in ThompMP).

regional_005 -- GSD_HRRR_AK_50km, FV3_GSD_SAR, RAPX/RAPX:

• run_fcst task fails.

regional_010 -- GSD_HRRR25km, FV3_CPT_v0, GSMGFS/GSMGFS:

• run_fcst task fails.
• Probably something is wrong with the namelist file.
• Not a priority to fix right now since this problem already exists in the develop branch.

regional_013 -- GSD_HRRR25km, FV3_GSD_SAR_v1, GSMGFS/GSMGFS:

• run_fcst task fails.
• Not a priority since the FV3_GSD_SAR_v1 suite is deprecated.

regional_014 -- GSD_HRRR25km, FV3_GFS_2017_gfdlmp, GSMGFS/GSMGFS:

• Will fail in make_ics task UNLESS PR#15 into Larissa's fork/branch is merged.

regional_016 -- GSD_HRRR25km, FV3_RRFS_v0, GSMGFS/GSMGFS:

• run_fcst task fails
• Not a priority since the FV3_RRFS_v0 suite is deprecated.

[JeffBeck-NOAA]

@gsketefian Latest commit should fix your WE2E errors.

[gsketefian]

@JeffBeck-NOAA I reran the failed tests. The NCO-mode one (nco_GSD_HRRR3km_HRRRX_RAPX) works but the others still fail because of the bugs in PR #280 that are currently in the HEAD of develop. I have a bugfix that I will push this afternoon which you'll have to merge and then we can rerun the community mode tests.

[JeffBeck-NOAA]

@gsketefian, sounds good. I will merge as soon as it's available.

[gsketefian]

@JeffBeck-NOAA I just checked this out but it doesn't have my latest PR. Can you merge the latest develop into this so I can retest? Thx.

[JeffBeck-NOAA]

Yes. Doing that now.

…

On Tue, Sep 15, 2020 at 2:21 PM gsketefian ***@***.***> wrote: @JeffBeck-NOAA <https://github.com/JeffBeck-NOAA> I just checked this out but it doesn't have my latest PR. Can you merge the latest develop into this so I can retest? Thx. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/NOAA-EMC/regional_workflow/pull/274#issuecomment-692957391>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANFE562IUA7LMRE64MTDH53SF7EDVANCNFSM4QGN2PFA> .

[gsketefian]

@JeffBeck-NOAA Can you update your PR#17 into ufs-srweather-app so that it uses the release/public-v2 branch? The current ufs-weather-model does not have the RRFS_v1beta suite defined yet (SDF is missing).

[gsketefian]

@JeffBeck-NOAA Also, can you check if the FV3_CPT_v0 suite accidentally got deleted from the make_ics.sh and make_lbcs.sh scripts where the variable mapping table is set? This is the error I'm getting from the make_ics.sh script (from test regional_010, which uses the FV3_CPT_v0 suite):

ERROR:                                                                   
  From script:  "exregional_make_ics.sh"                                 
  Full path to script:  "/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/ufs-srweather-app_Jeff_UFS_UTILS/regional_workflow/scripts/exregional_make_ics.sh"
A variable mapping table has not yet been defined for this physics suite:
  CCPP_PHYS_SUITE = "FV3_CPT_v0"                                         
Exiting with nonzero status.                                             
+ print_err_msg_exit 'Call to ex-script corresponding to J-job "JREGIONAL_MAKE_ICS" failed.'

Same error from the make_lbcs.sh script. Thanks.

[JeffBeck-NOAA]

@gsketefian I added the FV3_CPT_v0 SDF to the necessary lines in the make_ics/lbcs scripts. I just pushed the commit to this PR, so it should work now.

[gsketefian]

@JeffBeck-NOAA Ok, the FV3_CPT_v0 issue is fixed. Also, the RRFV_v1beta suite is now available in ufs-weather-model. BUT, because we're now using regional_esg_grid instead of the old regional_grid to generate grids, the orography statistics files that Mike Toy made no longer work. Most importantly, there is an unknown grid number issue in the tests that use HRRRX or RAPX for the ICs, which is probably the most important problem to fix (it wasn't there before I pulled your latest commits). Below is a summary of the latest WE2E test results on hera. They can be found in this directory:

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/expt_dirs/runs_for_PR_Jeff_UFS_UTILS_20200916

There is an earlier set of runs (from before when I pulled the latest updates to the feature/chgres_cube forks yesterday) in which the "unknown input file grid template number" error in make_ics and make_lbcs does not occur. Those runs are here:

/scratch2/BMC/det/Gerard.Ketefian/UFS_CAM/expt_dirs/runs_for_PR_Jeff_UFS_UTILS_20200915

Here's a summary of the latest runs (i.e. the ones in the first directory I listed above):

FV3_RRFS_v1beta                   FAILURE - In make_orog:  Don't have the orography statistics files for new grids generated by regional_esg_grid (e.g. C401 vs. C403).
GSD_RAP13km                       FAILURE - Many of the run_post tasks die.  They seem to hang, then time-out when the job runs out of time.
community_ensemble_008mems        SUCCESS
community_ensemble_2mems          SUCCESS
nco_GSD_HRRR3km_FV3GFS_FV3GFS     SUCCESS
nco_GSD_HRRR3km_HRRRX_RAPX        FAILURE - In make_ics and make_lbcs tasks:  Unknown input file grid template number. Must be one of: 3, 3.30, 3.32769
nco_conus                         SUCCESS
nco_conus_c96                     SUCCESS
nco_ensemble                      SUCCESS
new_ESGgrid                       SUCCESS
regional_001                      SUCCESS
regional_002                      SUCCESS
regional_003                      FAILURE - In make_ics and make_lbcs tasks:  Unknown input file grid template number. Must be one of: 3, 3.30, 3.32769
regional_004                      FAILURE - In make_ics and make_lbcs tasks:  Unknown input file grid template number. Must be one of: 3, 3.30, 3.32769
regional_005                      FAILURE - In make_ics and make_lbcs tasks:  Unknown input file grid template number. Must be one of: 3, 3.30, 3.32769
regional_006                      SUCCESS
regional_007                      SUCCESS
regional_008                      SUCCESS
regional_009                      SUCCESS
regional_010                      FAILURE - in run_fcst task (expected)
regional_011                      SUCCESS
regional_012                      SUCCESS
regional_014                      SUCCESS
regional_015                      SUCCESS
user_staged_extrn_files           SUCCESS

[JeffBeck-NOAA]

@gsketefian Recent changes in the UFS_UTILS build.hera module file have broken wgrib2 on Hera. I just left a comment for the commit in UFS_UTILS that has these changes and detailed what the problem is. This is the source of the problem with an "unknown input file grid". Chgres_cube is unable to run wgrib2 on the input file to identify what kind of grid it is.

[gsketefian]

As of 20200918, on hera all WE2E tests pass except for the following three, which are expected to fail:

FV3_RRFS_v1beta       FAILURE - In make_orog:  Don't have the orography statistics files for new grids generated by 
                                regional_esg_grid (e.g. C401 vs. C403).  These are not available because the change from 
                                the regional_grid to regional_esg_grid code slightly modifies the grid files, and the 
                                corresponding orography statistics files are not yet available.
GSD_RAP13km           FAILURE - Many of the run_post tasks die.  They seem to hang, then time-out when the job runs out 
                                of time.  This failure is either due to the new EMC_post branch being used or to the 
                                change from the regional_grid to the regional_esg_grid code.
regional_010          FAILURE - In run_fcst task (expected).  This is due to an error in the FV3 namelist file (probably 
                                an incorrect variable name).

Since this PR needs to go in urgently, I will approve it along with with PR #17 to ufs-community/ufs-srweather-app.