Switch cpld_bmark_p8 test to rectilinear 1/2deg grid; add Mesh creation info to wiki; Update P8 sfc ICs for FV3; Update WW3_inputdata directory; Bug fix for #1309: false reports of test failures

[DeniseWorthen]

PR Checklist

• This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.

• This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR

• An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
  are specified below.

• Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.

• New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.

Instructions: All subsequent sections of text should be filled in as appropriate.

The information provided below allows the code managers to understand the changes relevant to this PR, whether those changes are in the ufs-weather-model repository or in a subcomponent repository. Ufs-weather-model code managers will use the information provided to add any applicable labels, assign reviewers and place it in the Commit Queue. Once the PR is in the Commit Queue, it is the PR owner's responsibility to keep the PR up-to-date with the develop branch of ufs-weather-model.

Description

• Changes domain for the cpld_bmark_p8 test to use the gwes_30m domain.
• Updates WW3 input data for new mod_def files created using current intel 2022 compiler.
• A new WW3_input_data_20220624 directory has been added to hera and will need to be sync'd across all platforms. The updated mod_def files will change baselines for all tests using the wave model.
• The BM_IC-20220207 directory on hera has a new p8c directory for each date; this needs to be sync'd across all platforms.
• Adds information on mesh creation to wiki.

Issue(s) addressed

• fixes Switch bmark_p8 test to a rectilinear 1/2deg grid #1297

Testing

The full rt.conf was run on hera.intel at ba1b2a6 using a copy of the current input-data-20220414/WW3_input_data_20220418/ with new mesh.gwes_30m.nc and mod_def.gwes_30m added. All tests passed with the exception of cpld_bmark_p8, which ran to completion.

• hera.intel
• hera.gnu
• orion.intel
• cheyenne.intel
• cheyenne.gnu
• gaea.intel
• jet.intel
• wcoss2.intel
• NA wcoss_cray
• NAA wcoss_dell_p3
• opnReqTest for newly added/changed feature. oRTs for cpld_bmark_p8 (std_base,thr,rst,dcp) passed at 7e4a9ba
• CI

Dependencies

None. Script level changes only.

[jkbk2004]

@DeniseWorthen cheyenne.intel BL creation one case atmaero_control_p8_rad crashes with mpt/mpi error: /glade/scratch/jongkim/rt-1292-intel/jongkim/FV3_RT/rt_15184/atmaero_control_p8_rad. I will try to re-submit. Hera que is very slow today.

[jkbk2004]

@DeniseWorthen cheyenne.intel BL creation one case atmaero_control_p8_rad crashes with mpt/mpi error: /glade/scratch/jongkim/rt-1292-intel/jongkim/FV3_RT/rt_15184/atmaero_control_p8_rad. I will try to re-submit. Hera que is very slow today.

atmaero_control_p8_rad runs ok on Cheyenne.intel.

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: orion
Compiler: intel
Job: BL
[BL] Repo location: /work/noaa/nems/emc.nemspara/autort/pr/978684257/20220706171519/ufs-weather-model
[BL] Error: Test datm_cdeps_mx025_gefs 100 failed in run_test failed
[BL] Error: Test control_atmwav 104 failed in run_test failed
Please make changes and add the following label back: orion-intel-BL

[BrianCurtis-NOAA]

Automated RT Failure Notification
Machine: jet
Compiler: intel
Job: BL
[BL] Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/978684257/20220706230009/ufs-weather-model
Please make changes and add the following label back: jet-intel-BL

[DeniseWorthen]

Both orion failures seem to be system glitches.

The atmwav test had

108: Error in system call pthread_mutex_destroy: Device or resource busy
108:     ../../src/mpi/init/init_thread_cs.c:66
108: Abort(1090703) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:

The datm test had

147: [Orion-25-40:425553:0:425553] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
....
 26: [Orion-01-43:108691:0:108691]     address.c:1052 Assertion `*addr_version == UCP_OBJECT_VERSION_V2' failed: addr version 3

and then segfault.

[jkbk2004]

Yeah, I will work on orion. Jet has ecflow server connection issue. I am sending a help desk ticket.

[SamuelTrahanNOAA]

Are my regression test system changes working? I cannot see if your workflows are resubmitting jobs, so I don't know.

[binli2337]

@DeniseWorthen For the cpld_bmark_p8 test, do you plan to revise the TEST_DESCR parameter to indicate that the resolution of wave model is rectilinear 30m or 1/2-degree?

[DeniseWorthen]

@binli2337 I will make this change in a later PR.

[jkbk2004]

Are my regression test system changes working? I cannot see if your workflows are resubmitting jobs, so I don't know.

I think its working better on jet. It was rare case jet went thru by itself overnight. But currently ecflow issue on jet. I am using rocoto but it improves, I think.