UFSATM: Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine // removing variables from CICE NetCDF file (#2951)

[climbfuji]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

#2880

Update submodule pointer for UFSATM for the changes in NOAA-EMC/ufsatm#1004 (Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine) and ufs-community/ccpp-physics#310 (Update GFS_debug (GFS_interstitialtoscreen)).

This currently changes the answers for the RRTMGP regression test because of NOAA-EMC/ufsatm#1005 (fixed in a separate, forthcoming PR).

#2951

Removes variables for CICE history output for space reduction.

Commit Message:

* UFSWM - Update submodule pointer for UFSATM and ccpp-physics; remove vars from CICE netCDF file
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine
    * ccpp-physics - Update GFS_debug schemes: remove GFS_interstitialtoscreen_timestep_init and GFS_interstitialtoscreen_init
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * fire_behavior
  * stochastic_physics - 

Priority:

• Normal

Git Tracking

• Related to RRTMGP uses an interstitial variable that is only valid in the physics group, not in the radiation group of the suite NOAA-EMC/ufsatm#1005

Sub component Pull Requests:

• AQM:
• CDEPS:
• CICE:
• CMEPS:
• CMakeModules:
• FV3: Replace GFS_interstitial phys_reset and rad_reset routines with combined reset routine NOAA-EMC/ufsatm#1004
  • ccpp-physics: Update GFS_debug (GFS_interstitialtoscreen) ccpp-physics#310
  • atmos_cubed_sphere:
• GOCART:
• HYCOM:
• MOM6:
• NOAHMP:
• WW3:
• fire_behavior:
• stochastic_physics:
• None

UFSWM Blocking Dependencies:

• None

Documentation:

• No documentation update is required for this PR.

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.
• PR Updates/Changes Baselines: control_p8_rrtmgp intel only; needs new baseline because of RRTMGP uses an interstitial variable that is only valid in the physics group, not in the radiation group of the suite NOAA-EMC/ufsatm#1005, which will be fixed in UWM changes for CCPP framework update // Miscellaneous updates (#2974) #2881
• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[climbfuji]

@dustinswales I am getting a lot of test failures for certain classes of tests, not just RRTMGP. They all seem to be regional runs. Is there a different physics scheme that is being used in these tests? We'll need to check if these also "misuse" an interstitial variable. Should be relatively easy to verify. Alternatively, I might have missed resetting a certain interstitial variable in the combined reset call ...

[Dom.Heinzeller@uecflow01 tests]$ ls -l | grep fail_test
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 14:29 fail_test_control_p8_rrtmgp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:21 fail_test_conus13km_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 14:44 fail_test_conus13km_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:24 fail_test_conus13km_control_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 14:59 fail_test_conus13km_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        148 Sep  5 15:25 fail_test_conus13km_debug_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 15:02 fail_test_conus13km_debug_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        142 Sep  5 15:33 fail_test_conus13km_debug_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb         48 Sep  5 15:29 fail_test_conus13km_debug_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 15:31 fail_test_conus13km_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:14 fail_test_conus13km_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:33 fail_test_conus13km_debug_qr_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 15:13 fail_test_conus13km_debug_qr_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        124 Sep  5 15:24 fail_test_conus13km_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 14:57 fail_test_conus13km_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 15:32 fail_test_conus13km_radar_tten_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        160 Sep  5 15:31 fail_test_conus13km_radar_tten_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        145 Sep  5 15:13 fail_test_cpld_regional_atm_fbh_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        115 Sep  5 15:18 fail_test_hrrr_c3_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 14:43 fail_test_hrrr_c3_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        175 Sep  5 15:22 fail_test_hrrr_control_2threads_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        181 Sep  5 14:45 fail_test_hrrr_control_2threads_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        139 Sep  5 15:14 fail_test_hrrr_control_2threads_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        145 Sep  5 14:35 fail_test_hrrr_control_2threads_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        166 Sep  5 15:21 fail_test_hrrr_control_debug_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        172 Sep  5 14:50 fail_test_hrrr_control_debug_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        130 Sep  5 15:14 fail_test_hrrr_control_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        136 Sep  5 14:43 fail_test_hrrr_control_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        169 Sep  5 15:23 fail_test_hrrr_control_decomp_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        175 Sep  5 14:48 fail_test_hrrr_control_decomp_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        133 Sep  5 15:17 fail_test_hrrr_control_decomp_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        139 Sep  5 14:35 fail_test_hrrr_control_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        148 Sep  5 15:23 fail_test_hrrr_control_dyn32_phy32_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        154 Sep  5 14:46 fail_test_hrrr_control_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        112 Sep  5 15:19 fail_test_hrrr_control_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        118 Sep  5 14:34 fail_test_hrrr_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        127 Sep  5 15:16 fail_test_hrrr_control_noqr_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        115 Sep  5 15:14 fail_test_hrrr_gf_debug_gnu
-rw-r--r-- 1 Dom.Heinzeller gmtb        121 Sep  5 14:43 fail_test_hrrr_gf_debug_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb         42 Sep  5 14:55 fail_test_regional_control_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb         41 Sep  5 14:56 fail_test_regional_decomp_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        199 Sep  5 14:45 fail_test_regional_spp_sppt_shum_skeb_dyn32_phy32_intel
-rw-r--r-- 1 Dom.Heinzeller gmtb        163 Sep  5 14:33 fail_test_regional_spp_sppt_shum_skeb_intel

[dustinswales]

@climbfuji The regional physics are quite different than the global.
But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

[climbfuji]

@climbfuji The regional physics are quite different than the global. But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

@dustinswales On Friday I was able to fix all but two regression tests (the error was my fault). One of the failures is the RRTMGP test (b4b differences, as expected, thus good), the other one might have been a fluke (will rerun).

[dustinswales]

@climbfuji Found it, dustinswales/fv3atm@004fb86
Should you include it here? Or should I add it to the framework hash update branch?

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

[climbfuji]

@climbfuji The regional physics are quite different than the global. But... GP adopted some of the subgridscale cloud business from the regional applications, so maybe this helps narrow down the misused variables. I'm looking into this now.

@dustinswales On Friday I was able to fix all but two regression tests (the error was my fault). One of the failures is the RRTMGP test (b4b differences, as expected, thus good), the other one might have been a fluke (will rerun).

@dustinswales Turns out the other test that failed (timed out), regional_noquilt, went through when I reran it. I didn't rerun the whole thing to get a "clean" log, since I know that we'll need updates for the RRTMGP test, that there will be several more PRs before this PR is at the top of the queue, etc.

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

[dustinswales]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

@climbfuji Yes. Included in #2881

[climbfuji]

@climbfuji Found it, dustinswales/fv3atm@004fb86 Should you include it here? Or should I add it to the framework hash update branch?

Excellent! I think a separate PR would be great. I can pull it into my cleanup PRs for the interstitial afterwards, but at least it's recorded separately. Thanks!

@dustinswales Just checking if you created a PR for this?

@climbfuji Yes. Included in #2881

Great, thanks. I will mention this in the PR description

[climbfuji]

@jkbk2004 @dustinswales Just to avoid any confusion, this PR and its submodule components should go before #2881 (correct, @dustinswales?)

[dustinswales]

@jkbk2004 @dustinswales Just to avoid any confusion, this PR and its submodule components should go before #2881 (correct, @dustinswales?)

That is my understanding.

[gspetro-NOAA]

@climbfuji Is this PR ready to process/plan for commit queue?

[climbfuji]

@climbfuji Is this PR ready to process/plan for commit queue?

Yes, definitely. @dustinswales is waiting for this. Let me update the branches.

Thanks!

[BrianCurtis-NOAA]

Acorn gave me some troubles earlier, running again. Apologies for the delay.

[gspetro-NOAA]

Acorn gave me some troubles earlier, running again. Apologies for the delay.

No worries--I think we're also waiting on Ursa because the /scratch4 disk quota issue caused @FernandoAndrade-NOAA 's testing there to fail.

[FernandoAndrade-NOAA]

ORTs are a little troublesome today as well.

[FernandoAndrade-NOAA]

cpld_control_nowave_noaero_p8 and regional_control ORTs have passed. control_p8's dbg case is failing due to floating point exceptions, please refer to
/scratch3/NCEPDEV/stmp/Fernando.Andrade-maldonado/ORT-ptmp/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_3317430/dbg_base_dbg_base/err
on Hera.

[NickSzapiro-NOAA]

I see

147: [h34m48:2164517:0:2164517] Caught signal 8 (Floating point exception: floating-point invalid operation)
148: [h34m48:2164518:0:2164518] Caught signal 8 (Floating point exception: floating-point invalid operation)
146: [h34m48:2164516:0:2164516] Caught signal 8 (Floating point exception: floating-point invalid operation)
149: [h34m48:2164519:0:2164519] Caught signal 8 (Floating point exception: floating-point invalid operation)
145: [h34m48:2164515:0:2164515] Caught signal 8 (Floating point exception: floating-point invalid operation)
144: [h34m48:2164514:0:2164514] Caught signal 8 (Floating point exception: floating-point invalid operation)
149: ==== backtrace (tid:2164519) ====
149:  0 0x0000000000053519 ucs_debug_print_backtrace()  ???:0
149:  1 0x0000000000012990 __funlockfile()  :0
149:  2 0x000000000dca98cb grib2_module_mp_getgds_()  /scratch3/NAGAPE/epic/Fernando.Andrade-maldonado/reg-test/wm/2880/orts/control_p8/ufs-weather-model/UFSATM/upp/sorc/ncep_post.fd/grib2_module.f:1982

Then that's the same issue as NOAA-EMC/UPP#1359

Has dbg ORT passed since UPP was updated in #2932 ?

[FernandoAndrade-NOAA]

I see

147: [h34m48:2164517:0:2164517] Caught signal 8 (Floating point exception: floating-point invalid operation)
148: [h34m48:2164518:0:2164518] Caught signal 8 (Floating point exception: floating-point invalid operation)
146: [h34m48:2164516:0:2164516] Caught signal 8 (Floating point exception: floating-point invalid operation)
149: [h34m48:2164519:0:2164519] Caught signal 8 (Floating point exception: floating-point invalid operation)
145: [h34m48:2164515:0:2164515] Caught signal 8 (Floating point exception: floating-point invalid operation)
144: [h34m48:2164514:0:2164514] Caught signal 8 (Floating point exception: floating-point invalid operation)
149: ==== backtrace (tid:2164519) ====
149:  0 0x0000000000053519 ucs_debug_print_backtrace()  ???:0
149:  1 0x0000000000012990 __funlockfile()  :0
149:  2 0x000000000dca98cb grib2_module_mp_getgds_()  /scratch3/NAGAPE/epic/Fernando.Andrade-maldonado/reg-test/wm/2880/orts/control_p8/ufs-weather-model/UFSATM/upp/sorc/ncep_post.fd/grib2_module.f:1982

Then that's the same issue as NOAA-EMC/UPP#1359

Has dbg ORT passed since UPP was updated in #2932 ?

From what I can tell yes, all ORTs and cases have been successful since #2932 was merged

[NickSzapiro-NOAA]

So the known issue can be fixed locally with 1-line change

diff --git a/sorc/ncep_post.fd/CTLBLK.f b/sorc/ncep_post.fd/CTLBLK.f
index 9360f218..8b77d2e0 100644
--- a/sorc/ncep_post.fd/CTLBLK.f
+++ b/sorc/ncep_post.fd/CTLBLK.f
@@ -54,7 +54,7 @@
-  real :: earth_radius                             !< Radius of the earth (meters), as optionally specified in the namelist
+  real :: earth_radius = 0.0                       !< Radius of the earth (meters), as optionally specified in the namelist

Can you re-run failed ORT with this change locally? If that's ok, then I think we're ok to continue

[FernandoAndrade-NOAA]

So the known issue can be fixed locally with 1-line change

diff --git a/sorc/ncep_post.fd/CTLBLK.f b/sorc/ncep_post.fd/CTLBLK.f
index 9360f218..8b77d2e0 100644
--- a/sorc/ncep_post.fd/CTLBLK.f
+++ b/sorc/ncep_post.fd/CTLBLK.f
@@ -54,7 +54,7 @@
-  real :: earth_radius                             !< Radius of the earth (meters), as optionally specified in the namelist
+  real :: earth_radius = 0.0                       !< Radius of the earth (meters), as optionally specified in the namelist

Can you re-run failed ORT with this change locally? If that's ok, then I think we're ok to continue

That seems to have resolved the error, thanks for the suggestion. Logs are pushed up. @jkbk2004 @gspetro-NOAA FYI

[gspetro-NOAA]

@climbfuji You can revert .gitmodules and update the UFSATM hash to c122cfe.

[climbfuji]

Done!