[develop] Add checks for correct SPP settings for the selected CCPP suite, resolve four more issues

.github/CODEOWNERS

@@ -3,7 +3,7 @@
 
 # These owners will be the default owners for everything in the repo.
 #*       @defunkt
-*       @mkavulich @gsketefian @JeffBeck-NOAA @RatkoVasic-NOAA @BenjaminBlake-NOAA @ywangwof @chan-hoo @panll @christinaholtNOAA @christopherwharrop-noaa @danielabdi-noaa @mark-a-potts @jkbk2004 @willmayfield @dmwright526 @gspetro-NOAA @natalie-perlin @EdwardSnyder-NOAA @MichaelLueken @rickgrubin-noaa @BruceKropp-Raytheon @BenKoziol-NOAA @WeirAE
+*       @mkavulich @gsketefian @JeffBeck-NOAA @RatkoVasic-NOAA @BenjaminBlake-NOAA @ywangwof @chan-hoo @panll @christinaholtNOAA @christopherwharrop-noaa @danielabdi-noaa @mark-a-potts @jkbk2004 @willmayfield @dmwright526 @gspetro-NOAA @natalie-perlin @EdwardSnyder-NOAA @MichaelLueken @rickgrubin-noaa @BruceKropp-Raytheon @benkozi @WeirAE
 
 # Order is important. The last matching pattern has the most precedence.
 # So if a pull request only touches javascript files, only these owners

.github/workflows/python_tests.yaml

@@ -40,6 +40,7 @@ jobs:
           pylint ush/set_fv3nml*.py
           pylint ush/update_input_nml.py
           pylint ush/smoke_dust
+          pylint ush/setup.py
 
       - name: Checkout externals
         run: |

aqm_environment.yml

@@ -6,7 +6,7 @@ dependencies:
   - esmpy
   - netcdf4=1.6.*
   - numpy=1.23.*
-  - pylint=2.17*
+  - pylint=3.1.*
   - pytest=7.2*
   - scipy=1.10.*
   - uwtools=2.6*

doc/UsersGuide/BackgroundInfo/Components.rst

@@ -40,8 +40,15 @@ Model Physics
 
 The Common Community Physics Package (CCPP), described `here <https://dtcenter.org/community-code/common-community-physics-package-ccpp>`_, supports interoperable atmospheric physics and land surface model options. Atmospheric physics are a set of numerical methods describing small-scale processes such as clouds, turbulence, radiation, and their interactions. The most recent SRW App release (|latestr|) included five supported physics suites: FV3_RRFS_v1beta, FV3_GFS_v16, FV3_WoFS_v0, FV3_HRRR, and FV3_RAP. The FV3_RRFS_v1beta physics suite is being tested for use in the future operational implementation of the Rapid Refresh Forecast System (:term:`RRFS`) planned for 2023-2024, and the FV3_GFS_v16 is an updated version of the physics suite used in the operational Global Forecast System (GFS) v16. A detailed list of CCPP updates since the SRW App v2.1.0 release is available :ref:`here <CCPPUpdates>`. A full scientific description of CCPP parameterizations and suites can be found in the `CCPP Scientific Documentation <https://dtcenter.ucar.edu/GMTB/UFS_SRW_App_v2.2.0/sci_doc/index.html>`_, and CCPP technical aspects are described in the :doc:`CCPP Technical Documentation <ccpp-techdoc:index>`. The model namelist has many settings beyond the physics options that can optimize various aspects of the model for use with each of the supported suites. Additional information on Stochastic Physics options is available :doc:`here <stochphys:index>`. 
 
++                         'pbl': ['mynnedmf_wrapper'],
++                         'sfc': ['mynnsfc_wrapper'],
++                         'mp':  ['mp_thompson'],
++                         'rad': ['rrtmg_sw', 'rrtmg_lw'],
++                         'gwd': ['drag_suite'],
++                         'cu_deep': ['cu_gf_driver'],
+
 .. note::
-   SPP is currently only available for specific physics schemes used in the RAP/HRRR physics suite. Users need to be aware of which physics suite definition file (:term:`SDF`) is chosen when turning this option on. Among the supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR`` option for ``CCPP_PHYS_SUITE``.
+   SPP is currently only available for specific physics schemes: MYNN-EDMF (pbl), MYNN SFC (sfc), Thompson Microphysics (mp), RRTMG (rad), GFS gravity wave drag (gwd) and Grell-Freidas cumulus parameterization (cu_deep). Users need to be aware of which physics suite definition file (:term:`SDF`) is chosen when turning this option on. If SPP perturbations for a given parameterization are not possible with the selected suite, that SPP option will be deactivated. Among the supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR``, ``FV3_HRRR_gf``, ``FV3_RAP``, and ``RRFS_sas`` options for ``CCPP_PHYS_SUITE``.
 
 Additionally, a CCPP single-column model (`CCPP-SCM <https://github.com/NCAR/ccpp-scm>`_) option has also been developed as a child repository. Users can refer to the `CCPP Single Column Model User and Technical Guide <https://ccpp-scm.readthedocs.io/en/latest/>`_ for more details. This CCPP-SCM user guide contains a Quick Start Guide with instructions for obtaining the code, compiling, and running test cases, which include five standard test cases and two additional FV3 replay cases (refer to section 5.2 in the CCPP-SCM user guide for more details). Moreover, the CCPP-SCM supports a precompiled version in a docker container, allowing it to be easily executed on NOAA's cloud computing platforms without any issues (see section 2.5 in the CCPP-SCM user guide for more details).
 

doc/UsersGuide/CustomizingTheWorkflow/ConfigWorkflow.rst

@@ -1504,13 +1504,13 @@ Parameters for Stochastically Perturbed Parameterizations (SPP)
 SPP perturbs specific tuning parameters within a physics :term:`parameterization <parameterizations>` (unlike :ref:`SPPT <SPPT>`, which multiplies overall physics tendencies by a random perturbation field *after* the call to the physics suite). Patterns evolve and are applied at each time step. Each SPP option is an array, applicable (in order) to the :term:`RAP`/:term:`HRRR`-based parameterization listed in ``SPP_VAR_LIST``. Enter each value of the array in ``config.yaml`` as shown below without commas or single quotes (e.g., ``SPP_VAR_LIST: [ "pbl" "sfc" "mp" "rad" "gwd" ]`` ). Both commas and single quotes will be added by Jinja when creating the namelist.
 
 .. note::
-   SPP is currently only available for specific physics schemes used in the RAP/HRRR physics suite. Users need to be aware of which :term:`SDF` is chosen when turning this option on. Of the five supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR`` option for ``CCPP_PHYS_SUITE``.
+   SPP is currently only available for specific physics schemes: MYNN-EDMF (pbl), MYNN SFC (sfc), Thompson Microphysics (mp), RRTMG (rad), GFS gravity wave drag (gwd) and Grell-Freidas cumulus parameterization (cu_deep). Users need to be aware of which physics suite definition file (:term:`SDF`) is chosen when turning this option on. If SPP perturbations for a given parameterization are not possible with the selected suite, that SPP option will be deactivated. Among the supported physics suites, the full set of parameterizations can only be used with the ``FV3_HRRR``, ``FV3_HRRR_gf``, ``FV3_RAP``, and ``RRFS_sas`` options for ``CCPP_PHYS_SUITE``.
 
 ``DO_SPP``: (Default: false)
    Flag to turn SPP on or off. SPP perturbs parameters or variables with unknown or uncertain magnitudes within the physics code based on ranges provided by physics experts. Valid values: ``True`` | ``False``
 
 ``SPP_VAR_LIST``: (Default: [ "pbl", "sfc", "mp", "rad", "gwd" ] )
-   The list of parameterizations to perturb: planetary boundary layer (PBL), surface physics (SFC), microphysics (MP), radiation (RAD), gravity wave drag (GWD). Valid values: ``"pbl"`` | ``"sfc"`` | ``"rad"`` | ``"gwd"`` | ``"mp"``
+   The list of parameterizations to perturb: planetary boundary layer (PBL), surface physics (SFC), microphysics (MP), radiation (RAD), gravity wave drag (GWD). Valid values: ``"pbl"`` | ``"sfc"`` | ``"rad"`` | ``"gwd"`` | ``"mp"`` | ``"cu_deep"``
 
 ``SPP_MAG_LIST``: (Default: [ 0.2, 0.2, 0.75, 0.2, 0.2 ] ) 
    SPP perturbation magnitudes used in each parameterization. Corresponds to the variable ``spp_prt_list`` in ``input.nml``

environment.yml

@@ -3,7 +3,7 @@ channels:
   - conda-forge
   - ufs-community
 dependencies:
-  - pylint=2.17*
+  - pylint=3.1*
   - pytest=7.2*
   - uwtools=2.6*
   - sphinx=7.4.0*

graphics_environment.yml

@@ -9,7 +9,7 @@ dependencies:
   - matplotlib=3.5*
   - pillow=9.2*
   - pygrib=2.1*
-  - pylint=2.16*
+  - pylint=3.1*
   - pyproj=2.6*
   - pytest=7.1*
   - pyyaml=6.0*

parm/field_table.RRFS_sas

@@ -68,18 +68,3 @@
            "longname",     "sigma fraction"
            "units",        "fraction"
        "profile_type", "fixed", "surface_value=0.0" /
-# prognostic smoke mixing ratio tracer
-  "TRACER", "atmos_mod", "smoke"
-            "longname",     "smoke mixing ratio"
-            "units",        "ug/kg"
-       "profile_type", "fixed", "surface_value=1.e-12" /
-# prognostic dust mixing ratio tracer
-            "TRACER", "atmos_mod", "dust"
-            "longname",     "dust mixing ratio"
-            "units",        "ug/kg"
-       "profile_type", "fixed", "surface_value=1.e-12" /
-# prognostic coarsepm mixing ratio tracer
-  "TRACER", "atmos_mod", "coarsepm"
-            "longname",     "coarsepm mixing ratio"
-            "units",        "ug/kg"
-       "profile_type", "fixed", "surface_value=1.e-12" /

sd_environment.yml

@@ -8,7 +8,7 @@ dependencies:
   - pydantic=2.10.*
   - netcdf4=1.6.*=mpi_mpich*
   - numpy=1.23.*
-  - pylint=2.17.*
+  - pylint=3.1.*
   - pytest=7.2.*
   - pytest-mock=3.14.*
   - scipy=1.10.*

tests/WE2E/machine_suites/comprehensive.derecho

@@ -55,3 +55,4 @@ MET_verification_only_vx
 pregen_grid_orog_sfc_climo
 specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
 specify_template_filenames
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas

tests/WE2E/machine_suites/comprehensive.hera.gnu

@@ -61,6 +61,7 @@ MET_ensemble_verification_winter_wx
 MET_verification_only_vx
 pregen_grid_orog_sfc_climo
 smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas
 specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
 specify_template_filenames
 vx-det_long-fcst_custom-vx-config_aiml-fourcastnet

tests/WE2E/machine_suites/comprehensive.hera.intel

@@ -65,6 +65,7 @@ MET_ensemble_verification_winter_wx
 MET_verification_only_vx
 pregen_grid_orog_sfc_climo
 smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas
 specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
 specify_template_filenames
 vx-det_long-fcst_custom-vx-config_aiml-fourcastnet

tests/WE2E/machine_suites/comprehensive.noaacloud

@@ -56,3 +56,4 @@ specify_template_filenames
 get_from_AWS_ics_GEFS_lbcs_GEFS_fmt_grib2_2022040400_ensemble_2mems
 get_from_NOMADS_ics_FV3GFS_lbcs_FV3GFS
 long_fcst
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas

tests/WE2E/machine_suites/comprehensive.orion

@@ -54,5 +54,6 @@ MET_ensemble_verification_winter_wx
 MET_verification_only_vx
 pregen_grid_orog_sfc_climo
 smoke_dust_grid_RRFS_CONUS_3km_suite_HRRR_gf
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas
 specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS
 specify_template_filenames

tests/WE2E/machine_suites/coverage.hera.intel.nco

@@ -8,6 +8,7 @@ grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_RAP
 grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_HRRR
 MET_ensemble_verification_only_vx_time_lag
 pregen_grid_orog_sfc_climo
+smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas
 vx-det_long-fcst_custom-vx-config_aiml-graphcast
 vx-det_multicyc_long-fcst-overlap_nssl-mpas
 vx-det_multicyc_long-fcst-no-overlap_nssl-mpas

tests/WE2E/run_we2e_tests.py

@@ -192,7 +192,6 @@ def run_we2e_tests(homedir, args) -> None:
             },
             "workflow": {
                 "COMPILER": args.compiler,
-                "CRON_RELAUNCH_INTVL_MNTS": args.cron_relaunch_intvl_mnts,
                 "EXPT_SUBDIR": test_name,
                 "USE_CRON_TO_RELAUNCH": args.launch == "cron",
                 },
@@ -440,7 +439,8 @@ def check_task_get_extrn_bcs(
 
     # Make our lives easier by shortening some dictionary calls
     cfg_bcs = cfg[f"task_get_extrn_{ics_or_lbcs}"]
-    cfg_bcs_vars = cfg_bcs["envvars"]
+    if (cfg_bcs_vars := cfg_bcs.get("envvars")) is None:
+        raise KeyError(f"Required 'envvars' section not found in task_get_extrn_{ics_or_lbcs}")
 
     # If the task is turned off explicitly, do nothing and return
     # To turn off that task, taskgroups is included without the

tests/WE2E/test_configs/smoke_dust/config.smoke_dust_grid_RRFS_CONUS_25km_suite_RRFS_sas.yaml

@@ -0,0 +1,55 @@
+metadata:
+  description: config for Smoke and Dust, RRFS_CONUS_25km, RRFS_sas suite
+user:
+  RUN_ENVIR: community
+workflow:
+  PREDEF_GRID_NAME: RRFS_CONUS_25km
+  CCPP_PHYS_SUITE: RRFS_sas
+  DATE_FIRST_CYCL: '2019072200'
+  DATE_LAST_CYCL: '2019072206'
+  INCR_CYCL_FREQ: 6
+  FCST_LEN_HRS: 6
+  COMPILER: intel
+  DO_REAL_TIME: false
+  COLDSTART: true
+  PREEXISTING_DIR_METHOD: rename
+  taskgroups:
+    - parm/wflow/prep.yaml
+    - parm/wflow/smoke_dust.yaml
+    - parm/wflow/coldstart.yaml
+    - parm/wflow/post.yaml
+nco:
+  envir_default: we2e_smoke_dust
+  NET_default: smoke_dust
+  RUN_default: smoke_dust
+task_get_extrn_ics:
+  envvars:
+    EXTRN_MDL_NAME_ICS: RAP
+    EXTRN_MDL_ICS_OFFSET_HRS: 0
+    USE_USER_STAGED_EXTRN_FILES: true
+task_get_extrn_lbcs:
+  envvars:
+    EXTRN_MDL_NAME_LBCS: RAP
+    LBC_SPEC_INTVL_HRS: 6
+    EXTRN_MDL_LBCS_OFFSET_HRS: 0
+    USE_USER_STAGED_EXTRN_FILES: true
+task_run_fcst:
+  envvars:
+    RESTART_INTERVAL: 6 12 18 24
+    PRINT_ESMF: false
+  fv3:
+    execution:
+      batchargs:
+        walltime: 02:00:00
+task_run_post:
+  envvars:
+    POST_OUTPUT_DOMAIN_NAME: conus25km
+    USE_CUSTOM_POST_CONFIG_FILE: false
+global:
+  DO_ENSEMBLE: false
+  NUM_ENS_MEMBERS: 2
+smoke_dust_parm:
+  DO_SMOKE_DUST: true
+  EBB_DCYCLE: 1
+  SMOKE_DUST_FILE_PREFIX: "SMOKE_RRFS_data"
+

tests/WE2E/test_configs/wflow_features/config.get_from_HPSS_ics_GDAS_lbcs_GDAS_fmt_netcdf_2022040400_ensemble_2mems.yaml

@@ -1,14 +1,15 @@
 metadata:
   description: |-
     This test checks the capability of the workflow to retrieve from NOAA
-    HPSS netcdf-formatted output files generated by GDAS.
+    HPSS netcdf-formatted output files generated by GDAS. Also ensures that
+    LSM SPP settings can be activated properly.
 user:
   RUN_ENVIR: community
 platform:
   EXTRN_MDL_DATA_STORES: hpss
 workflow:
   CCPP_PHYS_SUITE: FV3_HRRR
-  PREDEF_GRID_NAME: RRFS_CONUS_3km
+  PREDEF_GRID_NAME: RRFS_CONUS_13km
   DATE_FIRST_CYCL: '2022040400'
   DATE_LAST_CYCL: '2022040400'
   FCST_LEN_HRS: 6
@@ -27,3 +28,8 @@ task_get_extrn_lbcs:
 global:
   DO_ENSEMBLE: true
   NUM_ENS_MEMBERS: 2
+  DO_LSM_SPP: true
+  LSM_SPP_VAR_LIST: [ "smc", "vgf", "emi", "zol", "stc" ]
+  LSM_SPP_TSCALE:  [ 21600, 21600, 21600, 21600, 21600 ]
+  LSM_SPP_LSCALE: [ 150000, 150000, 150000, 150000, 150000 ]
+  LSM_SPP_MAG_LIST: [ 0.017, 0.001, 0.001, 0.001, 0.2 ]

ush/generate_FV3LAM_wflow.py

@@ -302,7 +302,9 @@ def generate_FV3LAM_wflow(
         #
         # -----------------------------------------------------------------------
         #
-        setup_fv3_namelist(expt_config,debug)
+        if ( dict_find(expt_config["rocoto"]["tasks"], "task_run_fcst") or
+           dict_find(expt_config["rocoto"]["tasks"], "task_make_grid") ):
+            setup_fv3_namelist(expt_config,debug)
 
         #
         # -----------------------------------------------------------------------
@@ -319,15 +321,15 @@ def generate_FV3LAM_wflow(
         import_vars(dictionary=expt_config["global"])
         # pylint: disable=undefined-variable
         settings = {}
+        n_var_lndp = len(LSM_SPP_VAR_LIST) if DO_LSM_SPP else 0
         settings["gfs_physics_nml"] = {
             "do_shum": DO_SHUM,
             "do_sppt": DO_SPPT,
             "do_skeb": DO_SKEB,
             "do_spp": DO_SPP,
             "n_var_spp": N_VAR_SPP,
-            "n_var_lndp": N_VAR_LNDP,
+            "n_var_lndp": n_var_lndp,
             "lndp_type": LNDP_TYPE,
-            "fhcyc": FHCYC_LSM_SPP_OR_NOT,
         }
         nam_stochy_dict = {}
         if DO_SPPT: