Merge the hafsv0.2 baseline configuration back into develop

[BinLiu-NOAA]

Merge the hafsv0.2 baseline configuration back into develop

• Turn on the scale-aware SAS convection scheme by default
• Turn on the orographic GWD but keep the convective GWD off by default
• Use halo_blend of 10 by default
• When coupling with hycom, use the CMEPS based coupling by default
• Use GFSv16 and RTOFSv2 input data for HAFS by default
• Clean up trailing whitespace for parm and rocoto files
• Clean up and unify the regression tests in rocoto/cronjob_hafs_rt.sh
• Add the hafsv0p2a baseline configuration and cronjob driver script
• Update or clean up the computation resources for the jobs/tasks on different platforms

[BinLiu-NOAA]

Appreciate it if you could help to run the workflow level regression tests as you helped previously!

• @evankalina and @mrinalbiswas (Hera and Orion)
• @JunghoonShin-NOAA (Jet, default is xjet, maybe also test kjet?)
• @LinZhu-NOAA (wcoss_dell_p3)
• @BijuThomas-NOAA (wcoss_cray)
• @ChunxiZhang-NOAA (ufs-weather-model level regression tests)

[BinLiu-NOAA]

@ghassan-alaka and @AndrewHazelton, although the hrd_graphics job is turned off by default, if you would like to, you may also help to test the hrd_graphics job on Orion/Jet. Last time, I tried on Orion, it failed.

[ChunxiZhang-NOAA]

UFS-weather-model level regression test on Orion was successful.

[JunghoonShin-NOAA]

Hello Bin, some updates about the regression tests on xJet (and kJet).
On xJet, all regression tests completed successfully.
On kJet, the forecast jobs for all regression tests were submitted successfully yesterday, but they are still in the queue states.

[AndrewHazelton]

Looks good.

[BijuThomas-NOAA]

Appreciate it if you could help to run the workflow level regression tests as you helped previously!

• @evankalina and @mrinalbiswas (Hera and Orion)
• @JunghoonShin-NOAA (Jet, default is xjet, maybe also test kjet?)
• @LinZhu-NOAA (wcoss_dell_p3)
• @BijuThomas-NOAA (wcoss_cray)
• @ChunxiZhang-NOAA (ufs-weather-model level regression tests)

Bin: Do you know the location of pio libraries on wcoss(cray)?. The forecast build complains it missing. I think the
CMEPS needs it.

[evankalina]

The build and six workflow-level regression tests were all successful on Hera. Once @mrinalbiswas provides confirmation on Orion, I will approve the PR.

[BinLiu-NOAA]

Bin: Do you know the location of pio libraries on wcoss(cray)?. The forecast build complains it missing. I think the
CMEPS needs it.

Hang Lei (EMC) installed pio/2.5.2 on wcoss_cray here:
module use /usrx/local/nceplibs/NCEPLIBS/cmake/install/NCEPLIBS-v1.3.0/modules
module load pio/2.5.2

[mrinalbiswas]

The build and six workflow-level regression tests were all successful on Orion. @evankalina You can approve the PR.

[evankalina]

Approved. The changes pass all of the regression tests on Hera and Orion.

[BinLiu-NOAA]

Thanks, @ChunxiZhang-NOAA, @evankalina, @mrinalbiswas, and @JunghoonShin-NOAA! Glad that the regression tests have passed on Orion, Hera, and Jet! And the ufs-weather-model level regression tests also went through.

I know @BijuThomas-NOAA is having some issues to build the forecast executable on wcoss_cray with some issues related to PIO (GPTL) for the CMEPS HYCOM coupling.

And @LinZhu-NOAA, were you able to build and run the regression tests on wcoss_dell_p3 with this branch?

Thanks!

Bin

[LinZhu-NOAA]

@BinLiu-NOAA Yes, the hasfv.02 baseline built successfully on WCOSS DELL. All the regression tests are waiting for the resources for forecast jobs.

[LinZhu-NOAA]

@BinLiu-NOAA regression tests on DELL completed successfully with some changes.

[BinLiu-NOAA]

Since the regression tests have passed on orion, hera, jet, and wcoss_dell_p3, I am going to merge this pull request. Thanks to all who helped conduct the regression tests!

Note: for wcoss_cray, according to @BijuThomas-NOAA, the HAFS-HYCOM coupled forecast jobs failed probably due to library/module issues. We will work to figure it out later on.

[JunghoonShin-NOAA]

Hello Bin,
Just some minor comments about kJet although the regression test on xJet (our main platform) was successful. kJet was extremely slow. I submitted the six forecast jobs for the regression test 11 days ago; two of them are still waiting in the queue, four of them completed after waiting for 4-5 days (although two post jobs failed). I believe we can run the HAFS on kJet, but it seems that waiting time is very long on this machine.
Thank you!

[BijuThomas-NOAA]

The build and workflow-level regression tests were all successful on WCOSS_CRAY after

1. Using the ESMF library that was built with YAML-CPP
2. gfs2ofs2.x binary from HWRF build
3. A change of memory value from 24G to 2G for PRODUCT task in wcoss_cray.ent