From 207f33be729f17493ca8fb1e3fe66094ba8dd73a Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Wed, 17 Feb 2021 19:27:37 +0000 Subject: [PATCH 01/17] Update path to fix files and sample model data on AWS --- docs/UsersGuide/source/InputOutputFiles.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index d8efd7529a..92b2fa571e 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -239,9 +239,9 @@ location of the static files: ``FIXgsm``, ``TOPO_DIR``, and ``SFC_CLIMO_INPUT_DI where the static files are located. If you are on a pre-configured or configurable platform, there is no need to stage the fixed files manually because they are already available on those platforms and the paths are set in ``regional_workflow/ush/setup.sh`` for the static files. If the users platform is not defined -in that file, the static files can be pulled from the `FTP data repository -`_ or from `Amazon Web Services cloud storage -`_ +in that file, the static files can be pulled individually or as a full tar file from the `FTP data repository +`_ or from `Amazon Web Services (AWS) cloud storage +`_ and staged on your machine. The paths to the staged files must then be set in ``config.sh`` as follows: @@ -268,7 +268,8 @@ not have access to the NOAA HPSS and you need to pull and stage the data manuall set ``USE_USER_STAGED_EXTRN_FILES`` to ``TRUE`` and then set the paths to the where the IC/LBC files are located. A small sample of IC/LBCs is available at the `FTP data repository -`_. +`_ or from `AWS cloud storage +`_. Initial and Lateral Boundary Condition Organization --------------------------------------------------- From a5b5369211d71469d56a30232f749ac6ac8cef2e Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Thu, 18 Feb 2021 03:06:43 +0000 Subject: [PATCH 02/17] Updated links to several model archives --- docs/UsersGuide/source/InputOutputFiles.rst | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index 92b2fa571e..f010d2b039 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -348,14 +348,14 @@ Raw model files may be available from a number of sources. A few examples are pr NOMADS: https://nomads.ncep.noaa.gov/pub/data/nccf/com/{model}/prod, where model may be: -* gfs (grib2 or nemsio) - available for the last 10 days +* GFS (grib2 or nemsio) - available for the last 10 days https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/ -* nam - available for the last 8 days - https://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/nam.YYYYMMDD/nam.tCYz.conusnest.hiresfFH.tmCY.grib2 -* rap - available for the last 2 days - https://nomads.ncep.noaa.gov/pub/data/nccf/com/rap/prod/rap.YYYYMMDD/rap.tCYz.wrfnatfFH.grib2 -* hrrr - available for the last 2 days - https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/hrrr.YYYYMMDD/conus/hrrr.tCYz.wrfnatfFH.grib2 +* NAM - available for the last 8 days + https://nomads.ncep.noaa.gov/pub/data/nccf/com/nam/prod/ +* RAP - available for the last 2 days + https://nomads.ncep.noaa.gov/pub/data/nccf/com/rap/prod/ +* HRRR - available for the last 2 days + https://nomads.ncep.noaa.gov/pub/data/nccf/com/hrrr/prod/ NCDC archive: @@ -374,8 +374,10 @@ Google Cloud: Others: -* Univ. of Utah HRRR archive: http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/hrrr_FAQ.html -* NAM nest archive: https://www.ready.noaa.gov/archives.php +* Univ. of Utah HRRR archive: http://home.chpc.utah.edu/~u0553130/Brian_Blaylock/cgi-bin/hrrr_download.cgi +* NAM nest archive: https://www.ready.noaa.gov/archives.php +* NAM data older than 6 months can be requested through the Archive Information Request System: https://www.ncei.noaa.gov/has/HAS.FileAppRouter?datasetname=NAM218&subqueryby=STATION&applname=&outdest=FILE +* RAP isobaric data older than 6 months can be requested through the Archive Information Request System: https://www.ncei.noaa.gov/has/HAS.FileAppRouter?datasetname=RAP130&subqueryby=STATION&applname=&outdest=FILE Coexistence of Multiple Files for the Same Date ----------------------------------------------- From 72b8786718f57598e3903843b2388af5c1fe840b Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 23 Feb 2021 19:33:49 +0000 Subject: [PATCH 03/17] Update links for final versions and make a few small edits --- docs/UsersGuide/source/Introduction.rst | 33 +++++++++++---------- docs/UsersGuide/source/Quickstart.rst | 38 ++++++++++++++++++------- 2 files changed, 46 insertions(+), 25 deletions(-) diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 25fdb2af0f..24b0e77f72 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -10,11 +10,11 @@ while enabling research, development, and contribution opportunities for the bro For more information about the UFS, visit the UFS Portal at https://ufscommunity.org/. The UFS can be configured for multiple applications (see a complete list at -https://ufscommunity.org/#/science/aboutapps). The configuration described here is the UFS Short-Range +https://ufscommunity.org/science/aboutapps/). The configuration described here is the UFS Short-Range Weather (SRW) Application, which targets predictions of atmospheric behavior on a limited spatial domain -and on time scales from less than an hour out to several days. The SRW Application v1.0 includes a +and on time scales from less than an hour out to several days. The SRW Application v1.0 release includes a prognostic atmospheric model, pre- and post-processing, and a community workflow for running the system -end-to-end, which are documented within the user's guide and supported through a community forum. +end-to-end, which are documented within the User's Guide and supported through a community forum. Future work will include expanding the capabilities of the application to include data assimilation (DA) and a verification package (e.g. METplus) as part of the workflow. This documentation provides an overview of the release components, a description of the supported capabilities, a quick start guide @@ -30,11 +30,11 @@ files on that grid. There are additional utilities included to handle the correc points and topography filtering ``filter_topo``. The pre-processing software ``chgres_cube`` is used to convert the raw external model data into initial and lateral boundary condition files in netCDF format, needed as input to the FV3-LAM. Additional information about the UFS pre-processor utilities can -be found in the `UFS_UTILS User’s Guide `_. +be found in the `UFS_UTILS User’s Guide `_. The SRW Application can be initialized from a range of operational initial condition files. It is possible to initialize the model from GFS, NAM, RAP, and HRRR files in Gridded Binary v2 (GRIB2) -format for past dates. Please note, for GFS data, dates prior to 1 January 2018 may work but are +format and GFS in NEMSIO format for past dates. Please note, for GFS data, dates prior to 1 January 2018 may work but are not guaranteed. Public archives of model data can be accessed through the `National Centers for Environmental Information `_ (NCEI) or through the `NOAA Operational Model Archive and Distribution System `_ @@ -65,15 +65,15 @@ Atmospheric physics are a set of numerical methods describing small-scale proces as clouds, turbulence, radiation, and their interactions. There are two physics options supported for the release. The first is an experimental physics suite being tested for use in the future operational implementation of the Rapid Refresh Forecast System (RRFS) planned -for 2023, and the second is an updated version of the physics suite used in the operational +for 2023-2024, and the second is an updated version of the physics suite used in the operational Global Forecast System (GFS) v15. A scientific description of the CCPP parameterizations and -suites can be found in the `CCPP Scientific Documentation `_, +suites can be found in the `CCPP Scientific Documentation `_, and CCPP technical aspects are described in the `CCPP Technical Documentation `_. The model namelist has many settings beyond the physics options that can optimize various aspects of the model for use with each of the supported suites. -The SRW App supports the use of both GRIB2 and :term:`nemsio` input data. The UFS Weather Model +The SRW App supports the use of both GRIB2 and :term:`NEMSIO` input data. The UFS Weather Model ingests initial and lateral boundary condition files produced by :term:`chgres_cube` and outputs files in NetCDF format on a specific projection (e.g., Lambert Conformal) in the horizontal and model levels in the vertical. @@ -84,7 +84,7 @@ Post-processor The SRW Application is distributed with the Unified Post Processor (:term:`UPP`) included in the workflow as a way to convert the NetCDF output on the native model grid to GRIB2 format on standard isobaric vertical coordinates. UPP can also be used to compute a variety of useful -diagnostic fields, as described in the `UPP user’s guide `_. +diagnostic fields, as described in the `UPP user’s guide `_. Output from UPP can be used with visualization, plotting, and verification packages, or for further downstream post-processing, e.g. statistical post-processing techniques. @@ -100,7 +100,8 @@ that the application is producing reasonable results. The scripts are available in the `regional_workflow repository `_ -under ush/Python. Usage information and instructions are included at the top of the script. +under ush/Python. Usage information and instructions are described in Chapter 10 +(:numref:`Chapter %s `) and are also included at the top of the script. Build System and Workflow ========================= @@ -164,7 +165,9 @@ User Support, Documentation, and Contributing Development A forum-based, online `support system `_ with topical sections provides a centralized location for UFS users and developers to post questions and exchange information. The forum complements the formal, written documentation, summarized here for ease of -use. A list of available documentation is shown in :numref:`Table %s `. +use. + +A list of available documentation is shown in :numref:`Table %s `. .. _list_of_documentation: @@ -189,15 +192,15 @@ use. A list of available documentation is shown in :numref:`Table %s & build.out & + make -j 4 >& build.out & Output from the build will be in the ``ufs-srweather-app/build/build.out`` file. When the build completes, you should see the forecast model executable ``NEMS.exe`` and eleven @@ -84,7 +84,7 @@ Generating the workflow experiment requires three steps: * Set experiment parameters in config.sh * Set Python and other environment parameters -* Run the generate_FV3LAM_wflow.sh script +* Run the ``generate_FV3LAM_wflow.sh`` script The first two steps depend on the platform being used and are described here for each Level 1 platform. @@ -96,7 +96,7 @@ The workflow requires a file called ``config.sh`` to specify the values of your Two example templates are provided: ``config.community.sh`` and ``config.nco.sh`` and can be found in the ``ufs-srweather-app/regional_workflow/ush directory``. The first file is a minimal example for creating and running an experiment in the *community* mode (with ``RUN_ENVIR`` set to ``community``), -while the second is an example of creating and running an experiment in the *NCO*’ (operational) mode +while the second is an example of creating and running an experiment in the *NCO* (operational) mode (with ``RUN_ENVIR`` set to ``nco``). The *community* mode is recommended in most cases and will be fully supported for this release while the operational mode will be more exclusively used by NOAA/NCEP Central Operations (NCO) and those in the NOAA/NCEP/Environmental Modeling Center (EMC) working with @@ -109,8 +109,14 @@ Make a copy of ``config.community.sh`` to get started: cd ufs-srweather-app/regional_workflow/ush cp config.community.sh config.sh -Edit the ``config.sh`` file to use an account you can charge to ``ACCOUNT``, and the name of the -experiment ``EXPT_SUBDIR``. The following parameters should be set for the machine you are using: +Edit the ``config.sh`` file to set the machine you are running on to ``MACHINE``, use an account you can charge for +``ACCOUNT``, and set the name of the experiment using ``EXPT_SUBDIR``. If you have access to the NOAA HPSS from the +machine you are running on, those changes should be sufficient; however, if that is not the case (for example, +on Cheyenne), or if you have pre-staged the initialization data you would like to use, you will also want to set +``USE_USER_STAGED_EXTRN_FILES="TRUE"`` and set the paths to the data for ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and +``EXTRN_MDL_SOURCE_BASEDIR_LBCS``. + +Given that, at a minimum, the following parameters should be set for the machine you are using: For Cheyenne: @@ -167,7 +173,7 @@ the following way: .. code-block:: console - source env/wflow_.env + source ufs-srweather-app/env/wflow_.env Run the ``generate_FV3LAM_wflow.sh`` script ------------------------------------------- @@ -184,7 +190,16 @@ Run the Workflow Using Rocoto ============================= The information in this section assumes that Rocoto is available on the desired platform. If Rocoto is not available, it is still possible to run the workflow using stand-alone scripts -described in :numref:`Section %s `. To run the workflow with Rocoto: +described in :numref:`Section %s `. There are two ways you can run +the workflow with Rocoto using either the ``./launch_FV3LAM_wflow.sh`` or by hand. To run Rocoto +using the script: + +.. code-block:: console + + cd $EXPTDIR + ./launch_FV3LAM_wflow.sh + +Or you can manually call Rocoto: .. code-block:: console @@ -192,12 +207,15 @@ described in :numref:`Section %s `. To run the workfl rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -For automatic resubmission of the workflow (every 3 minutes), the following line can be added -to the user's crontab (use ``crontab -e`` to edit the cron table): +For automatic resubmission of the workflow (every 3 minutes), one of the following lines can be added +to the user's crontab (use ``crontab -e`` to edit the cron table) depending on your preference of how +you call Rocoto: .. code-block:: console - */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + -- OR -- + */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && ./launch_FV3LAM_wflow.sh .. note:: From d6f48a2a210c458a7feb83709f73afc951752c26 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 23 Feb 2021 20:21:55 +0000 Subject: [PATCH 04/17] Update link to Chpt 10 --- docs/UsersGuide/source/Introduction.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 24b0e77f72..1e1946ed64 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -100,8 +100,8 @@ that the application is producing reasonable results. The scripts are available in the `regional_workflow repository `_ -under ush/Python. Usage information and instructions are described in Chapter 10 -(:numref:`Chapter %s `) and are also included at the top of the script. +under ush/Python. Usage information and instructions are described in +:numref:`Chapter %s ` and are also included at the top of the script. Build System and Workflow ========================= From cd5b6c3cc806348281148028c0ca8771f304778a Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 23 Feb 2021 20:29:22 +0000 Subject: [PATCH 05/17] Final tweaks --- docs/UsersGuide/source/Quickstart.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index 1716f77d50..46ae160871 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -110,13 +110,13 @@ Make a copy of ``config.community.sh`` to get started: cp config.community.sh config.sh Edit the ``config.sh`` file to set the machine you are running on to ``MACHINE``, use an account you can charge for -``ACCOUNT``, and set the name of the experiment using ``EXPT_SUBDIR``. If you have access to the NOAA HPSS from the +``ACCOUNT``, and set the name of the experiment with ``EXPT_SUBDIR``. If you have access to the NOAA HPSS from the machine you are running on, those changes should be sufficient; however, if that is not the case (for example, on Cheyenne), or if you have pre-staged the initialization data you would like to use, you will also want to set ``USE_USER_STAGED_EXTRN_FILES="TRUE"`` and set the paths to the data for ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS``. -Given that, at a minimum, the following parameters should be set for the machine you are using: +At a minimum, the following parameters should be set for the machine you are using: For Cheyenne: @@ -191,15 +191,16 @@ Run the Workflow Using Rocoto The information in this section assumes that Rocoto is available on the desired platform. If Rocoto is not available, it is still possible to run the workflow using stand-alone scripts described in :numref:`Section %s `. There are two ways you can run -the workflow with Rocoto using either the ``./launch_FV3LAM_wflow.sh`` or by hand. To run Rocoto -using the script: +the workflow with Rocoto using either the ``./launch_FV3LAM_wflow.sh`` or by hand. + +To run Rocoto using the script: .. code-block:: console cd $EXPTDIR ./launch_FV3LAM_wflow.sh -Or you can manually call Rocoto: +Or to manually call Rocoto: .. code-block:: console From 468e093fd983f774a4935761c418b5be3d6973c9 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 23 Feb 2021 22:29:24 +0000 Subject: [PATCH 06/17] Updated link for UFS_UTILS doc --- docs/UsersGuide/source/Introduction.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 1e1946ed64..2163ec6963 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -30,7 +30,7 @@ files on that grid. There are additional utilities included to handle the correc points and topography filtering ``filter_topo``. The pre-processing software ``chgres_cube`` is used to convert the raw external model data into initial and lateral boundary condition files in netCDF format, needed as input to the FV3-LAM. Additional information about the UFS pre-processor utilities can -be found in the `UFS_UTILS User’s Guide `_. +be found in the `UFS_UTILS User’s Guide `_. The SRW Application can be initialized from a range of operational initial condition files. It is possible to initialize the model from GFS, NAM, RAP, and HRRR files in Gridded Binary v2 (GRIB2) @@ -179,7 +179,7 @@ A list of available documentation is shown in :numref:`Table %s Date: Wed, 24 Feb 2021 23:10:59 +0000 Subject: [PATCH 07/17] Update DOI publish date in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f4005131e0..60144e2bbf 100644 --- a/README.md +++ b/README.md @@ -7,5 +7,5 @@ The UFS can be configured for multiple applications (see a complete list at http For the most up-to-date instructions on how to clone the repository, build the code, and run the workflow, see: https://github.com/ufs-community/ufs-srweather-app/wiki/Getting-Started -UFS Development Team. (2021, February 25). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.4534994 +UFS Development Team. (2021, March 4). Unified Forecast System (UFS) Short-Range Weather (SRW) Application (Version v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.4534994 From bf1cbfcb1caae18316016e247ee4718377d2c6a6 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Fri, 26 Feb 2021 18:51:45 +0000 Subject: [PATCH 08/17] Update wording of citation in Introduction --- docs/UsersGuide/source/Introduction.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 2163ec6963..4e81dd2629 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -20,6 +20,12 @@ Future work will include expanding the capabilities of the application to includ overview of the release components, a description of the supported capabilities, a quick start guide for running the application, and information on where to find more information and obtain support. +The SRW App v1.0.0 citation is as follows and should be used when presenting results based on research +conducted with the App. + +UFS Development Team. (2021, March 4). Unified Forecast System (UFS) Short-Range Weather (SRW) Application +(Version v1.0.0). Zenodo. https://doi.org/10.5281/zenodo.4534994 + Pre-processor Utilities and Initial Conditions ============================================== From 532b0860eb0ec85f8940c51e6585acece9a7b13e Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Sun, 28 Feb 2021 17:02:24 +0000 Subject: [PATCH 09/17] First pass at addressing Dom's comments. More to come, especially related to platform environments. --- docs/UsersGuide/source/CodeReposAndDirs.rst | 2 +- docs/UsersGuide/source/ConfigNewPlatform.rst | 32 ++++++++------ docs/UsersGuide/source/ConfigWorkflow.rst | 10 ++--- docs/UsersGuide/source/Graphics.rst | 15 +++---- docs/UsersGuide/source/InputOutputFiles.rst | 10 ++--- docs/UsersGuide/source/Introduction.rst | 2 +- docs/UsersGuide/source/LAMGrids.rst | 14 +++--- docs/UsersGuide/source/Quickstart.rst | 41 ++++++++++++++---- docs/UsersGuide/source/SRWAppOverview.rst | 45 +++++++++++++------- 9 files changed, 107 insertions(+), 64 deletions(-) diff --git a/docs/UsersGuide/source/CodeReposAndDirs.rst b/docs/UsersGuide/source/CodeReposAndDirs.rst index edfdab16e7..c5f429b0c5 100644 --- a/docs/UsersGuide/source/CodeReposAndDirs.rst +++ b/docs/UsersGuide/source/CodeReposAndDirs.rst @@ -157,7 +157,7 @@ workflow is run, is shown in :numref:`Table %s `. +---------------------------+-------------------------------------------------------------------------------------------------------+ | data_table | Cycle-independent input file (empty) | +---------------------------+-------------------------------------------------------------------------------------------------------+ - | field_table | Scalar fields in the `forecast model | + | field_table | Tracers in the `forecast model | | | `_ | +---------------------------+-------------------------------------------------------------------------------------------------------+ | FV3LAM_wflow.xml | Rocoto XML file to run the workflow | diff --git a/docs/UsersGuide/source/ConfigNewPlatform.rst b/docs/UsersGuide/source/ConfigNewPlatform.rst index 5c39d86d1f..a7722c6d94 100644 --- a/docs/UsersGuide/source/ConfigNewPlatform.rst +++ b/docs/UsersGuide/source/ConfigNewPlatform.rst @@ -14,11 +14,11 @@ The first step to installing on a new machine is to install :term:`NCEPLIBS` (ht * C and C++ compilers compatible with the Fortran compiler - * gcc v9+, ifort v18+, and clang (MacOS) have been tested + * gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang or LLVM clang) have been tested * Python v3.6+ - * Prerequisite packages must be downloaded: jinja2, yaml and f90nml, as well as pygrib if the user would like to use the provided graphics scripts + * Prerequisite packages must be downloaded: jinja2, yaml and f90nml, as well as a number of additional Python modules (see :numref:`Section %s `) if the user would like to use the provided graphics scripts * Perl 5 @@ -28,16 +28,20 @@ The first step to installing on a new machine is to install :term:`NCEPLIBS` (ht * CMake v3.15+ is needed for building NCEPLIBS, but versions as old as 3.12 can be used to build NCEPLIBS-external, which contains a newer CMake that can be used for the rest of the build. -For Linux systems, as long as the above software is available, you can move on to the next step: installing the :term:`NCEPLIBS-external` package. - -For MacOS systems, you will also need to set the stack size to “unlimited”. +For both Linux and macOS, you will need to set the stack size to "unlimited" (if allowed) or the largest possible value. .. code-block:: console + # Linux, if allowed + ulimit -s unlimited + + # macOS, this corresponds to 65MB ulimit -S -s unlimited -Additionally, some extra software is needed: ``wget``, ``coreutils``, ``pkg-config``, and ``gnu-sed``. -It is recommended that you install this software using the Homebrew package manager for MacOS (https://brew.sh/): +For Linux systems, as long as the above software is available, you can move on to the next step: installing the :term:`NCEPLIBS-external` package. + +For macOS systems, some extra software is needed: ``wget``, ``coreutils``, ``pkg-config``, and ``gnu-sed``. +It is recommended that you install this software using the Homebrew package manager for macOS (https://brew.sh/): * brew install wget @@ -160,7 +164,7 @@ At this point there are just a few more variables that need to be set prior to b export CMAKE_CXX_COMPILER=mpicxx export CMAKE_Fortran_COMPILER=mpifort -If you are using your machine’s built-in MPI compilers, it is recommended you set the ``CMAKE_*_COMPILER`` flags to full paths to ensure that the correct MPI aliases are used. Finally, one last environment variable, ``CMAKE_Platform``, must be set. This will depend on your machine; for example, on a MacOS operating system with GNU compilers: +If you are using your machine’s built-in MPI compilers, it is recommended you set the ``CMAKE_*_COMPILER`` flags to full paths to ensure that the correct MPI aliases are used. Finally, one last environment variable, ``CMAKE_Platform``, must be set. This will depend on your machine; for example, on a macOS operating system with GNU compilers: .. code-block:: console @@ -198,7 +202,7 @@ Running the graphics scripts in ``${WORKDIR}/ufs-srweather-app/regional_workflow For the final step of creating and running an experiment, the exact methods will depend on if you are running with or without a workflow manager (Rocoto). -Running Without a Workflow Manager: Generic Linux and MacOS Platforms +Running Without a Workflow Manager: Generic Linux and macOS Platforms ===================================================================== Now that the code has been built, you can stage your data as described in :numref:`Section %s `. @@ -244,7 +248,7 @@ From here, you can run each individual task of the UFS SRW App using the provide cp ${WORKDIR}/ufs-srweather-app/regional_workflow/ush/wrappers/*sh . cp ${WORKDIR}/ufs-srweather-app/regional_workflow/ush/wrappers/README.md . -The ``README.md`` file will contain instructions on the order that each script should be run in. An example of wallclock times for each task for an example run (2017 Macbook Pro, MacOS Catalina, 25km CONUS domain, 48hr forecast) is listed in :numref:`Table %s `. +The ``README.md`` file will contain instructions on the order that each script should be run in. An example of wallclock times for each task for an example run (2017 Macbook Pro, macOS Catalina, 25km CONUS domain, 48hr forecast) is listed in :numref:`Table %s `. .. _WallClockTimes: @@ -341,7 +345,7 @@ Those requirements highlighted in **bold** are included in the NCEPLIBS-external * 4GB memory (CONUS 25km domain) -* Fortran compiler with full Fortran 2003 standard support +* Fortran compiler with full Fortran 2008 standard support * C and C++ compiler @@ -361,13 +365,13 @@ Those requirements highlighted in **bold** are included in the NCEPLIBS-external * **netCDF (C and Fortran libraries)** * **HDF5** - * **ESMF** + * **ESMF** 8.0.0 * **Jasper** * **libJPG** * **libPNG** * **zlib** -MacOS-specific prerequisites: +macOS-specific prerequisites: * brew install wget * brew install cmake @@ -381,4 +385,4 @@ Optional but recommended prerequisites: * Bash v4+ * Rocoto Workflow Management System (1.3.1) * **CMake v3.15+** -* Python package pygrib for graphics +* Python packages scipy, matplotlib, pygrib, cartopy, and pillow for graphics diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index a8bfe8e0a7..ba80dd8131 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -5,7 +5,7 @@ Configuring the Workflow: ``config.sh`` and ``config_defaults.sh`` ================================================================== To create the experiment directory and workflow when running the SRW App, the user must create an experiment configuration file named ``config.sh``. This file contains experiment-specific information, such as dates, external model data, directories, and other relevant settings. To help the user, two sample configuration files have been included in the ``regional_workflow`` repository’s ``ush`` directory: ``config.community.sh`` and ``config.nco.sh``. The first is for running experiments in community mode (``RUN_ENVIR`` set to “community”; see below), and the second is for running experiments in “nco” mode (``RUN_ENVIR`` set to “nco”). Note that for this release, only “community” mode is supported. These files can be used as the starting point from which to generate a variety of experiment configurations in which to run the SRW App. -There is an extensive list of experiment parameters that a user can set when configuring the experiment. All of these do not need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ```config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``) Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). +There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ``config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``). Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). The following is a list of the parameters in the ``config_defaults.sh`` file. For each parameter, the default value and a brief description is given. In addition, any relevant information on features and settings supported or unsupported in this release is specified. @@ -68,7 +68,7 @@ These settings control run commands for platforms without a workflow manager. V The run command for pre-processing utilities (shave, orog, sfc_climo_gen, etc.). This can be left blank for smaller domains, in which case the executables will run without MPI. ``RUN_CMD_FCST``: (Default: "mpirun -np \${PE_MEMBER01}") - The run command for the model forecast step. This will be appended to the end of the variable definitions file (var_defns.sh).. + The run command for the model forecast step. This will be appended to the end of the variable definitions file ("var_defns.sh"). ``RUN_CMD_POST``: (Default: "mpirun -np 1") The run command for post-processing (UPP). Can be left blank for smaller domains, in which case UPP will run without MPI. @@ -211,7 +211,7 @@ Initial and Lateral Boundary Condition Generation Parameters The name of the external model that will provide fields from which lateral boundary condition (LBC) files (except for the 0-th hour LBC file) will be generated for input into the forecast model. ``LBC_SPEC_INTVL_HRS``: (Default: “6”) - The interval (in integer hours) with which LBC files will be generated, referred to as the boundary specification interval. Note that the model specified in ``EXTRN_MDL_NAME_LBCS`` must have data available at a frequency greater than or equal to that implied by ``LBC_SPEC_INTVL_HRS``. For example, if ``LBC_SPEC_INTVL_HRS`` is set to 6, then the model must have data available at least every 6 hours. It is up to the user to ensure that this is the case. + The interval (in integer hours) at which LBC files will be generated, referred to as the boundary specification interval. Note that the model specified in ``EXTRN_MDL_NAME_LBCS`` must have data available at a frequency greater than or equal to that implied by ``LBC_SPEC_INTVL_HRS``. For example, if ``LBC_SPEC_INTVL_HRS`` is set to 6, then the model must have data available at least every 6 hours. It is up to the user to ensure that this is the case. ``FV3GFS_FILE_FMT_ICS``: (Default: “nemsio”) If using the FV3GFS model as the source of the ICs (i.e. if ``EXTRN_MDL_NAME_ICS`` is set to "FV3GFS"), this variable specifies the format of the model files to use when generating the ICs. @@ -225,7 +225,7 @@ User-Staged External Model Directory and File Parameters Flag that determines whether or not the workflow will look for the external model files needed for generating ICs and LBCs in user-specified directories (as opposed to fetching them from mass storage like NOAA HPSS). ``EXTRN_MDL_SOURCE_BASEDIR_ICS``: (Default: “/base/dir/containing/user/staged/extrn/mdl/files/for/ICs") - Directory in which to look for external model files for generating ICs. If ``USE_USER_STAGED_EXTRN_FILES`` is set to "TRUE", the workflow looks in this directory (specifically, in a subdirectory under this directory named "YYYYMMDDHH" consisting of the starting date and cycle hour of the forecast, where YYYY is the 4-digit year, MM the 2-digit month, DD the 2-digit day of the month, and HH the 2-digit hour of the day) for the external model files specified by the array ``EXTRN_MDL_FILES_ICS``` (these files will be used to generate the ICs on the native FV3-LAM grid). This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". + Directory in which to look for external model files for generating ICs. If ``USE_USER_STAGED_EXTRN_FILES`` is set to "TRUE", the workflow looks in this directory (specifically, in a subdirectory under this directory named "YYYYMMDDHH" consisting of the starting date and cycle hour of the forecast, where YYYY is the 4-digit year, MM the 2-digit month, DD the 2-digit day of the month, and HH the 2-digit hour of the day) for the external model files specified by the array ``EXTRN_MDL_FILES_ICS`` (these files will be used to generate the ICs on the native FV3-LAM grid). This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". ``EXTRN_MDL_FILES_ICS``: (Default: "ICS_file1” “ICS_file2” “...”) Array containing the names of the files to search for in the directory specified by ``EXTRN_MDL_SOURCE_BASEDIR_ICS``. This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". @@ -239,6 +239,6 @@ User-Staged External Model Directory and File Parameters CCPP Parameter ============== ``CCPP_PHYS_SUITE``: (Default: "FV3_GFS_v15p2") - The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “RRFS_v1alpha.” + The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “FV3_RRFS_v1alpha.” .. include:: ConfigParameters.inc diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index ac46a495cf..38c4777a8b 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -153,12 +153,13 @@ In this case, the output png files will be located in the directory ``EXPTDIR1/C If the Python scripts are being used to create plots of multiple forecast lead times and forecast -variables, then they should be submitted to the batch system using either the ``sq_job.sh`` -or ``sq_job_diff.sh`` script (for platforms such as Hera, Jet, Orion, and Gaea that use slurm as -the job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for platforms such as -Cheyenne that use PBS or PBS Pro as the job scheduler). These scripts are located under -``ufs-srweather-app/regional_workflow/ush/Python`` and must be submitted using the command appropriate -for the job scheduler used on the current platform. For example, on Hera, Jet, Orion, and Gaea, +variables, then they should be submitted to the batch system using either something like the ``sq_job.sh`` +or ``sq_job_diff.sh`` script (for a platform such as Hera that uses the slurm +job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for a platform such as +Cheyenne that uses PBS or PBS Pro as the job scheduler). Examples of these scripts are located under +``ufs-srweather-app/regional_workflow/ush/Python`` and can be used as an example to create a batch script +for your platform/job scheduler of use. The scripts must be submitted using the command appropriate +for the job scheduler used on your platform. For example, on Hera, ``sq_job.sh`` can be submitted as follows: .. code-block:: console @@ -226,5 +227,3 @@ and ending with the last forecast hour, use export FCST_START=6 export FCST_END=${FCST_LEN_HRS} export FCST_INC=6 - - diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index 3fc839a0cd..3e1e14effc 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -75,12 +75,12 @@ and are shown in :numref:`Table %s `. | | the start of each forecast. It is an empty file. No need to | | | change. | +-----------------------------+-------------------------------------------------------------+ - | data_table_[CCPP] | File specifying the output fields of the forecast model. | + | diag_table_[CCPP] | File specifying the output fields of the forecast model. | | | A different diag_table may be configured for different | | | CCPP suites. | +-----------------------------+-------------------------------------------------------------+ | field_table_[CCPP] | Cycle-independent file that the forecast model reads in at | - | | the start of each forecast. It specifies the scalars that | + | | the start of each forecast. It specifies the tracers that | | | the forecast model will advect. A different field_table | | | may be needed for different CCPP suites. | +-----------------------------+-------------------------------------------------------------+ @@ -237,8 +237,8 @@ Static Files A set of fix files are necessary to run the SRW Application. Environment variables describe the location of the static files: ``FIXgsm``, ``TOPO_DIR``, and ``SFC_CLIMO_INPUT_DIR`` are the directories where the static files are located. If you are on a pre-configured or configurable platform, there is no -need to stage the fixed files manually because they are already available on those platforms and the paths -are set in ``regional_workflow/ush/setup.sh`` for the static files. If the users platform is not defined +need to stage the fixed files manually because they have been prestaged and the paths +are set in ``regional_workflow/ush/setup.sh``. If the user's platform is not defined in that file, the static files can be pulled individually or as a full tar file from the `FTP data repository `_ or from `Amazon Web Services (AWS) cloud storage `_ @@ -406,7 +406,7 @@ are initializing from NEMSIO format FV3GFS files. Best Practices for Conserving Disk Space and Keeping Files Safe --------------------------------------------------------------- Initial and lateral boundary condition files are large and can occupy a significant amount of -disk space. If various users will employ a common files system to conduct runs, it is recommended +disk space. If various users will employ a common file system to conduct runs, it is recommended that the users share the same ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS`` directories. That way, if raw model input files are already on disk for a given date they do not need to be replicated. diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 4e81dd2629..7da355e7f6 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -145,7 +145,7 @@ have been defined for the SRW Application, including pre-configured (level 1), c (level 2), limited test platforms (level 3), and build only platforms (level 4). Each level is further described below. -For the select computational platforms that have been pre-configured (level 1), all the +For the selected computational platforms that have been pre-configured (level 1), all the required libraries for building the SRW Application are available in a central place. That means bundled libraries (NCEPLIBS) and third-party libraries (NCEPLIBS-external) have both been built. The SRW Application is expected to build and run out of the box on these diff --git a/docs/UsersGuide/source/LAMGrids.rst b/docs/UsersGuide/source/LAMGrids.rst index 650ac91bc4..b1e2b97e69 100644 --- a/docs/UsersGuide/source/LAMGrids.rst +++ b/docs/UsersGuide/source/LAMGrids.rst @@ -31,19 +31,19 @@ of the following three options: The predefined grids are named after the prototype 3-km continental United States (CONUS) grid being tested for the Rapid Refresh Forecast System (RRFS), which will be a convection-allowing, hourly-cycled, FV3-LAM-based ensemble planned for operational implementation in 2024. To allow -for use of HRRR data to initialize the SRW App, all three supported grids were created to fit completely within -the HRRR domain to allow external model data from the HRRR to be used as initial conditions for -the FV3-LAM. Three resolution options were provided for flexibility related to compute resources +for use of High Resolution Rapid Refresh (`HRRR `_) data to +initialize the SRW App, all three supported grids were created to fit completely within the HRRR domain. +Three resolution options were provided for flexibility related to compute resources and physics options. For example, a user may wish to use the 13-km or 25-km domain when running -with the ``FV3_GFS_v15p2`` suite definition file (SDF), since that SDF uses cumulus physics which is -not currently configured to run at 3-km. In addition, users will have much fewer computational +with the ``FV3_GFS_v15p2`` suite definition file (SDF), since that SDF uses cumulus physics that are +not configured to run at 3-km. In addition, users will have much fewer computational constraints when running with the 13-km and 25-km domains. The boundary of the ``RRFS_CONUS_3km`` domain is shown in :numref:`Figure %s ` (in red). Note that while it is possible to initialize the FV3-LAM with coarser external model data when using the ``RRFS_CONUS_3km`` domain, it is generally advised to use external model data that has a resolution similar to that of the native FV3-LAM (predefined) grid. In addition, this grid is ideal for running the -``FV3_RRFS_v1alpha`` SDF, since it was specifically created for convection-allowing scales, and is the +``FV3_RRFS_v1alpha`` suite definition file (SDF), since this SDF was specifically created for convection-allowing scales, and is the precursor to the operational physics suite that will be used in the RRFS. As can be seen in :numref:`Figure %s `, the boundary of the write-component grid (in blue) sits @@ -80,7 +80,7 @@ While the three predefined grids available in this release are ideal for users j out with the SRW App, more advanced users may wish to create their own grid for testing over a different region and/or with a different resolution. Creating a user-defined grid requires knowledge of how the SRW App workflow functions, in particular, understanding the set of -scripts that handle the workflow and experiment generation. It’s also important to note that +scripts that handle the workflow and experiment generation. It iss also important to note that user-defined grids are not a supported feature of the current release, however information is being provided for the benefit of the FV3-LAM community. diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index 46ae160871..991828f11b 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -28,7 +28,7 @@ The necessary source code is publicly available on GitHub. To clone the release .. code-block:: console - git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git + git clone -b ufs-v1.0.0 https://github.com/ufs-community/ufs-srweather-app.git cd ufs-srweather-app Then, check out the submodules for the SRW application: @@ -76,7 +76,7 @@ Run ``cmake`` to set up the ``Makefile``, then run ``make``: Output from the build will be in the ``ufs-srweather-app/build/build.out`` file. When the build completes, you should see the forecast model executable ``NEMS.exe`` and eleven pre- and post-processing executables in the ``ufs-srweather-app/bin`` directory which are -described in :numref:`Table %s `. +described in :numref:`Table %s `. Generate the Workflow Experiment ================================ @@ -102,11 +102,11 @@ fully supported for this release while the operational mode will be more exclusi Central Operations (NCO) and those in the NOAA/NCEP/Environmental Modeling Center (EMC) working with NCO on pre-implementation testing. Sample config.sh files are discussed in this section for Level 1 platforms. -Make a copy of ``config.community.sh`` to get started: +Make a copy of ``config.community.sh`` to get started (under /path-to-ufs-srweather-app/regional_workflow/ush): .. code-block:: console - cd ufs-srweather-app/regional_workflow/ush + cd ../regional_workflow/ush cp config.community.sh config.sh Edit the ``config.sh`` file to set the machine you are running on to ``MACHINE``, use an account you can charge for @@ -153,6 +153,14 @@ For Orion: ACCOUNT="my_account" EXPT_SUBDIR="my_expt_name" +For Gaea: + +.. code-block:: console + + MACHINE="gaea" + ACCOUNT="my_account" + EXPT_SUBDIR="my_expt_name" + For WCOSS, edit ``config.sh`` with these WCOSS-specific parameters, and use a valid WCOSS project code for the account parameter: @@ -169,11 +177,11 @@ Set up the Python and other Environment Parameters Next, it is necessary to load the appropriate Python environment for the workflow. The workflow requires Python 3, with the packages 'PyYAML', 'Jinja2', and 'f90nml' available. This Python environment has already been set up on Level 1 platforms, and can be activated in -the following way: +the following way (when in /path-to-ufs-srweather-app/regional_workflow/ush): .. code-block:: console - source ufs-srweather-app/env/wflow_.env + source ../../env/wflow_.env Run the ``generate_FV3LAM_wflow.sh`` script ------------------------------------------- @@ -183,8 +191,10 @@ For all platforms, the workflow can then be generated with the command: ./generate_FV3LAM_wflow.sh -The generated workflow will be in ``$EXPTDIR``, where ``EXPTDIR=${EXPT_BASEDIR}/${EXPT_SUBDIR}``. The -settings for these paths can be found in the output from the ``./generate_FV3LAM_wflow.sh`` script. +The generated workflow will be in ``$EXPTDIR``, where ``EXPTDIR=${EXPT_BASEDIR}/${EXPT_SUBDIR}``. A +log file called ``log.generate_FV3LAM_wflow`` is generated by this step and can also be found in +``$EXPTDIR``. The settings for these paths can be found in the output from the +``./generate_FV3LAM_wflow.sh`` script. Run the Workflow Using Rocoto ============================= @@ -193,6 +203,19 @@ If Rocoto is not available, it is still possible to run the workflow using stand described in :numref:`Section %s `. There are two ways you can run the workflow with Rocoto using either the ``./launch_FV3LAM_wflow.sh`` or by hand. +An environment variable may be set to navigate to the ``$EXPTDIR`` more easily. If the login +shell is bash, it can be set as follws: + +.. code-block:: console + + export EXPTDIR=/path-to-experiment/directory + +Or if the login shell is csh/tcsh, it can be set using: + +.. code-block:: console + + setenv EXPTDIR /path-to-experiment/directory + To run Rocoto using the script: .. code-block:: console @@ -200,6 +223,8 @@ To run Rocoto using the script: cd $EXPTDIR ./launch_FV3LAM_wflow.sh +A log file called ``log.launch_FV3LAM_wflow`` is created in the ``$EXPTDIR``. + Or to manually call Rocoto: .. code-block:: console diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index 137d040c0c..8259ea548e 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -7,7 +7,7 @@ The UFS Short-Range Weather Application (SRW App) is an umbrella repository that ``manage_externals`` to check out all of the components required for the application. Once the build process is complete, all the files and executables necessary for a regional experiment are located in the ``regional_workflow`` and ``bin`` directories, respectively, under the ``ufs-srweather-app`` directory. -Users can utilize the pre-defined domains or build their own domain (details provided in `Chapter %s `). +Users can utilize the pre-defined domains or build their own domain (details provided in :numref:`Chapter %s `). In either case, users must create/modify the case-specific (``config.sh``) and/or grid-specific configuration files (``set_predef_grid_params.sh``). The overall procedure is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. The steps are as follows: @@ -33,11 +33,11 @@ Each step will be described in detail in the following sections. Download from GitHub ==================== -Retrieve the UFS Short Range Weather Application (SRW App) repository from GitHub and checkout the ``release/public-v1`` branch: +Retrieve the UFS Short Range Weather Application (SRW App) repository from GitHub and checkout the ``ufs-v1.0.0`` tag: .. code-block:: console - git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git + git clone -b ufs-v1.0.0 https://github.com/ufs-community/ufs-srweather-app.git cd ufs-srweather-app The cloned repository contains the configuration files and sub-directories shown in @@ -52,7 +52,7 @@ The cloned repository contains the configuration files and sub-directories shown +================================+========================================================+ | CMakeLists.txt | Main cmake file for SRW App | +--------------------------------+--------------------------------------------------------+ - | Externals.cfg | Hashes of the GitHub repositories/branches for the | + | Externals.cfg | Tags of the GitHub repositories/branches for the | | | external repositories | +--------------------------------+--------------------------------------------------------+ | LICENSE.md | CC0 license information | @@ -66,7 +66,7 @@ The cloned repository contains the configuration files and sub-directories shown +--------------------------------+--------------------------------------------------------+ | env | Contains build and workflow environment files | +--------------------------------+--------------------------------------------------------+ - | docs | Contains Release notes, documentation, and Users' Guide| + | docs | Contains release notes, documentation, and Users' Guide| +--------------------------------+--------------------------------------------------------+ | manage_externals | Utility for checking out external repositories | +--------------------------------+--------------------------------------------------------+ @@ -85,7 +85,7 @@ Check out the external repositories, including regional_workflow, ufs-weather-mo ./manage_externals/checkout_externals This step will use the configuration ``Externals.cfg`` file in the ``ufs-srweather-app`` directory to -clone the specific hashes (version of codes) of the external repositories as listed in +clone the specific tags (version of codes) of the external repositories as listed in :numref:`Section %s `. .. _BuildExecutables: @@ -114,16 +114,16 @@ The following steps will build the pre-processing utilities, forecast model, and make dir cd build cmake .. -DCMAKE_INSTALL_PREFIX=.. - make -j 8 >& build.out & + make -j 4 >& build.out & where ``-DCMAKE_INSTALL_PREFIX`` specifies the location in which the ``bin``, ``include``, ``lib``, and ``share`` directories containing various components of the SRW App will be created, and its recommended value ``..`` denotes one directory up from the build directory. In the next line for -the ``make`` call, ``-j 8`` indicates the build will run in parallel with 8 threads. If this step is successful, the -executables listed in :numref:`Table %s ` will be located in the +the ``make`` call, ``-j 4`` indicates the build will run in parallel with 4 threads. If this step is successful, the +executables listed in :numref:`Table %s ` will be located in the ``ufs-srweather-app/bin`` directory. -.. _exec_description: +.. _ExecDescription: .. table:: Names and descriptions of the executables produced by the build step and used by the SRW App. @@ -146,7 +146,7 @@ executables listed in :numref:`Table %s ` will be located in t +------------------------+---------------------------------------------------------------------------------+ | orog | Generates orography, land mask, and gravity wave drag files from fixed files | +------------------------+---------------------------------------------------------------------------------+ - | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | + | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | +------------------------+---------------------------------------------------------------------------------+ | sfc_climo_gen | Creates surface climatology fields from fixed files for use in ``chgres_cube`` | +------------------------+---------------------------------------------------------------------------------+ @@ -168,7 +168,8 @@ grids as shown in :numref:`Table %s `. Their names can be found ``valid_vals_PREDEF_GRID_NAME`` in the ``valid_param_vals`` script, and their grid-specific configuration variables are specified in the ``set_predef_grid_params`` script. If users want to create a new domain, they should put its name in the ``valid_param_vals`` script and the corresponding grid-specific -parameters in the ``set_predef_grid_params`` script. +parameters in the ``set_predef_grid_params`` script. More information on the predefined and user-generated options +can be found in :numref:`Chapter %s `. .. _PredefinedGrids: @@ -193,12 +194,13 @@ Default configuration: ``config_defaults.sh`` -------------------------------------------- When generating a new experiment (described in detail in :numref:`Section %s `), the ``config_defaults.sh`` file is read first and assigns default values to the experiment -parameters. Important configuration variables in the ``config_defaults.sh`` file are shown in +parameters. Important configuration variables in the ``config_defaults.sh`` file are shown in :numref:`Table %s `, with more documentation found in the file itself, and -in `Chapter %s `. Some of these default values are intentionally invalid in order +in :numref:`Chapter %s `. Some of these default values are intentionally invalid in order to ensure that the user assigns valid values in the user-specified configuration ``config.sh`` file. Therefore, any settings provided in ``config.sh`` will override the default ``config_defaults.sh`` settings. Note that there is usually no need for a user to modify the default configuration file. + .. _ConfigVarsDefault: .. table:: Configuration variables specified in the config_defaults.sh script. @@ -522,6 +524,19 @@ There are two ways to launch the workflow using Rocoto: (1) with the ``launch_FV script, and (2) manually calling the ``rocotorun`` command. Moreover, you can run the workflow separately using stand-alone scripts. +An environment variable may be set to navigate to the ``$EXPTDIR`` more easily. If the login +shell is bash, it can be set as follws: + +.. code-block:: console + + export EXPTDIR=/path-to-experiment/directory + +Or if the login shell is csh/tcsh, it can be set using: + +.. code-block:: console + + setenv EXPTDIR /path-to-experiment/directory + Launch with the ``launch_FV3LAM_wflow.sh`` script ------------------------------------------------- To launch the ``launch_FV3LAM_wflow.sh`` script, simply call it without any arguments as follows: @@ -640,7 +655,7 @@ a wrapper script to set environment variables and run the job script. Example batch-submit scripts for Hera (Slurm) and Cheyenne (PBS) are included: ``sq_job.sh`` and ``qsub_job.sh``. These examples set the build and run environment for Hera or Cheyenne so that run-time libraries match the compiled libraries (i.e. netcdf, mpi). Users may either -modify the one submit batch script as each task is submitted, or duplicate this batch wrapper +modify the submit batch script as each task is submitted, or duplicate this batch wrapper for their system settings for each task. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). This piece will be unique to your platform. The tasks run by the regional workflow From 964a78bb9c263b6f49b458d3b9a90e8a18e2d027 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Sun, 28 Feb 2021 17:18:44 +0000 Subject: [PATCH 10/17] More small edits. More to come. --- docs/UsersGuide/source/ConfigWorkflow.rst | 4 ++-- docs/UsersGuide/source/Graphics.rst | 2 +- docs/UsersGuide/source/LAMGrids.rst | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index ba80dd8131..4c85493440 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -5,7 +5,7 @@ Configuring the Workflow: ``config.sh`` and ``config_defaults.sh`` ================================================================== To create the experiment directory and workflow when running the SRW App, the user must create an experiment configuration file named ``config.sh``. This file contains experiment-specific information, such as dates, external model data, directories, and other relevant settings. To help the user, two sample configuration files have been included in the ``regional_workflow`` repository’s ``ush`` directory: ``config.community.sh`` and ``config.nco.sh``. The first is for running experiments in community mode (``RUN_ENVIR`` set to “community”; see below), and the second is for running experiments in “nco” mode (``RUN_ENVIR`` set to “nco”). Note that for this release, only “community” mode is supported. These files can be used as the starting point from which to generate a variety of experiment configurations in which to run the SRW App. -There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ``config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``). Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). +There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ``config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``). Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). The following is a list of the parameters in the ``config_defaults.sh`` file. For each parameter, the default value and a brief description is given. In addition, any relevant information on features and settings supported or unsupported in this release is specified. @@ -239,6 +239,6 @@ User-Staged External Model Directory and File Parameters CCPP Parameter ============== ``CCPP_PHYS_SUITE``: (Default: "FV3_GFS_v15p2") - The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “FV3_RRFS_v1alpha.” + The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “FV3_RRFS_v1alpha”. .. include:: ConfigParameters.inc diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index 38c4777a8b..f5bed62e0d 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -157,7 +157,7 @@ variables, then they should be submitted to the batch system using either someth or ``sq_job_diff.sh`` script (for a platform such as Hera that uses the slurm job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for a platform such as Cheyenne that uses PBS or PBS Pro as the job scheduler). Examples of these scripts are located under -``ufs-srweather-app/regional_workflow/ush/Python`` and can be used as an example to create a batch script +``ufs-srweather-app/regional_workflow/ush/Python`` and can be used as a starting point to create a batch script for your platform/job scheduler of use. The scripts must be submitted using the command appropriate for the job scheduler used on your platform. For example, on Hera, ``sq_job.sh`` can be submitted as follows: diff --git a/docs/UsersGuide/source/LAMGrids.rst b/docs/UsersGuide/source/LAMGrids.rst index b1e2b97e69..f76f3565d8 100644 --- a/docs/UsersGuide/source/LAMGrids.rst +++ b/docs/UsersGuide/source/LAMGrids.rst @@ -80,7 +80,7 @@ While the three predefined grids available in this release are ideal for users j out with the SRW App, more advanced users may wish to create their own grid for testing over a different region and/or with a different resolution. Creating a user-defined grid requires knowledge of how the SRW App workflow functions, in particular, understanding the set of -scripts that handle the workflow and experiment generation. It iss also important to note that +scripts that handle the workflow and experiment generation. It is also important to note that user-defined grids are not a supported feature of the current release, however information is being provided for the benefit of the FV3-LAM community. From 1ac490c07d85db16bab9954de53b403e34a4d8da Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Sun, 28 Feb 2021 22:41:10 +0000 Subject: [PATCH 11/17] A few more updates. Still more to come. --- docs/UsersGuide/source/ConfigWorkflow.rst | 2 +- docs/UsersGuide/source/Graphics.rst | 10 ++-- docs/UsersGuide/source/Quickstart.rst | 61 ++++++++++++++++++++++- 3 files changed, 65 insertions(+), 8 deletions(-) diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 4c85493440..9c85d7be1d 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -84,7 +84,7 @@ Cron-Associated Parameters Directory Parameters ==================== ``EXPT_BASEDIR``: (Default: “”) - The base directory in which the experiment directory will be created. If this is not specified or if it is set to an empty string, it will default to ``${HOMErrfs}/../expt_dirs``, where ``${HOMErrfs}`` contains the full path to the ``regional_workflow`` directory. + The base directory in which the experiment directory will be created. If this is not specified or if it is set to an empty string, it will default to ``${HOMErrfs}/../../expt_dirs``, where ``${HOMErrfs}`` contains the full path to the ``regional_workflow`` directory. ``EXPT_SUBDIR``: (Default: “”) The name that the experiment directory (without the full path) will have. The full path to the experiment directory, which will be contained in the variable ``EXPTDIR``, will be: diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index f5bed62e0d..c6557c7848 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -44,7 +44,7 @@ On Hera: .. code-block:: console - /scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/NaturalEarth + /scratch2/BMC/det/UFS_SRW_app/v1p0/fix_files/NaturalEarth On Jet: @@ -56,7 +56,7 @@ On Orion: .. code-block:: console - /home/chjeon/tools/NaturalEarth + /work/noaa/global/glopara/fix/NaturalEarth On Gaea: @@ -100,9 +100,9 @@ On Gaea: .. code-block:: console - module use -a /apps/contrib/miniconda3-noaa-gsl/modulefiles - module load miniconda3 - conda activate pygraf + module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles + module load rocoto/1.3.3 + module load miniconda3/4.8.3-regional-workflow .. note:: diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index 991828f11b..3cb9e9a61a 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -227,6 +227,63 @@ A log file called ``log.launch_FV3LAM_wflow`` is created in the ``$EXPTDIR``. Or to manually call Rocoto: +First load the Rocoto module, depending on the platform used. + +For Cheyenne: + +.. code-block:: console + + module use -a /glade/p/ral/jntp/UFS_SRW_app/modules/ + module load rocoto + +For Hera or Jet: + +.. code-block:: console + + module purge + module load rocoto + +For Orion: + +.. code-block:: console + + module purge + module load contrib rocoto + +For Gaea: + +.. code-block:: console + + module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles + module load rocoto/1.3.3 + +For WCOSS_DELL_P3: + +.. code-block:: console + + module purge + module load lsf/10.1 + module use /gpfs/dell3/usrx/local/dev/emc_rocoto/modulefiles/ + module load ruby/2.5.1 rocoto/1.2.4 + +For WCOSS_DELL_P3: + +.. code-block:: console + + module purge + module load xt-lsfhpc/9.1.3 + module use -a /usrx/local/emc_rocoto/modulefiles + module load rocoto/1.2.4 + +For Gaea: + +.. code-block:: console + + module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles + module load rocoto/1.3.3 + +Then manually call ``rocotorun`` to launch the tasks and ``rocotostat`` to monitor the progress: + .. code-block:: console cd $EXPTDIR @@ -239,9 +296,9 @@ you call Rocoto: .. code-block:: console - */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 - -- OR -- */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && ./launch_FV3LAM_wflow.sh + -- OR -- + */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 .. note:: From 296e9be60657c824af4b1b814fbbf77378243f81 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Sun, 28 Feb 2021 23:46:24 +0000 Subject: [PATCH 12/17] Getting closer to addressing all of Dom's concerns. --- docs/UsersGuide/source/ConfigParameters.inc | 18 ++++++++++-------- docs/UsersGuide/source/Graphics.rst | 1 + docs/UsersGuide/source/Quickstart.rst | 17 ++++------------- docs/UsersGuide/source/SRWAppOverview.rst | 16 +++++++++------- 4 files changed, 24 insertions(+), 28 deletions(-) diff --git a/docs/UsersGuide/source/ConfigParameters.inc b/docs/UsersGuide/source/ConfigParameters.inc index c10e916cf5..b67ed0c0cc 100644 --- a/docs/UsersGuide/source/ConfigParameters.inc +++ b/docs/UsersGuide/source/ConfigParameters.inc @@ -5,7 +5,9 @@ Grid Generation Parameters ========================== ``GRID_GEN_METHOD``: (Default: “”) - This variable specifies the method to use to generate a regional grid in the horizontal. The only supported value of this parameter is “ESGgrid”, in which case the Extended Schmidt Gnomonic grid generation method developed by Jim Purser of EMC will be used. + This variable specifies the method to use to generate a regional grid in the horizontal. The only supported value of this parameter is “ESGgrid”, in which case the Extended Schmidt Gnomonic grid generation method developed by Jim Purser(1) of EMC will be used. + +(1)Purser, R. J., D. Jovic, G. Ketefian, T. Black, J. Beck, J. Dong, and J. Carley, 2020: The Extended Schmidt Gnomonic Grid for Regional Applications. Unified Forecast System (UFS) Users’ Workshop. July 27-29, 2020. .. note:: @@ -49,11 +51,11 @@ Computational Forecast Parameters ``BLOCKSIZE``: (Default: “”) The amount of data that is passed into the cache at a time. -Here, we set these parameters to null strings. This is so that, for any one of these parameters: +Here, we set these parameters to null strings. This is so that, for any one of these parameters: -#. If the experiment is using a predefined grid, then if the user sets the parameter in the user-specified experiment configuration file (``EXPT_CONFIG_FN``), that value will be used in the forecast(s). Otherwise, the default value of the parameter for that predefined grid will be used. +#. If the experiment is using a predefined grid and the user sets the parameter in the user-specified experiment configuration file (``EXPT_CONFIG_FN``), that value will be used in the forecast(s). Otherwise, the default value of the parameter for that predefined grid will be used. -#. If the experiment is not using a predefined grid (i.e. it is using a custom grid whose parameters are specified in the experiment configuration file), then the user must specify a value for the parameter in that configuration file. Otherwise, the parameter will remain set to a null string, and the experiment generation will fail because the generation scripts check to ensure that all the parameters defined in this section are set to non-empty strings before creating the experiment directory. +#. If the experiment is not using a predefined grid (i.e. it is using a custom grid whose parameters are specified in the experiment configuration file), then the user must specify a value for the parameter in that configuration file. Otherwise, the parameter will remain set to a null string, and the experiment generation will fail, because the generation scripts check to ensure that all the parameters defined in this section are set to non-empty strings before creating the experiment directory. Write-Component (Quilting) Parameters ===================================== @@ -91,7 +93,7 @@ Currently supported ``PREDEF_GRID_NAME`` options are "RRFS_CONUS_25km," "RRFS_CO Pre-existing Directory Parameter ================================ ``PREEXISTING_DIR_METHOD``: (Default: “delete”) - This variable determines the method to use to deal with pre-existing directories [e.g ones generated by previous calls to the experiment generation script using the same experiment name (``EXPT_SUBDIR``) as the current experiment]. This variable must be set to one of "delete", "rename", and "quit". The resulting behavior for each of these values is as follows: + This variable determines the method to deal with pre-existing directories [e.g ones generated by previous calls to the experiment generation script using the same experiment name (``EXPT_SUBDIR``) as the current experiment]. This variable must be set to one of "delete", "rename", and "quit". The resulting behavior for each of these values is as follows: * "delete": The preexisting directory is deleted and a new directory (having the same name as the original preexisting directory) is created. @@ -128,12 +130,12 @@ These parameters set flags (and related directories) that determine whether the Surface Climatology Parameter ============================= -``SFC_CLIMO_FIELDS``: (Default: “("facsf" "maximum_snow_albedo" "slope_type" "snowfree_albedo" "soil_type" "substrate_temperature" "vegetation_greenness" "vegetation_type")” +``SFC_CLIMO_FIELDS``: (Default: “("facsf" "maximum_snow_albedo" "slope_type" "snowfree_albedo" "soil_type" "substrate_temperature" "vegetation_greenness" "vegetation_type")”) Array containing the names of all the fields for which the ``MAKE_SFC_CLIMO_TN`` task generates files on the native FV3-LAM grid. Fixed File Parameters ===================== -Set parameters associated with the fixed (i.e. static) files. For the main NOAA HPC platforms, as well as Cheyenne, Odin, and Stampede, fixed files are prestaged in locations on each machine with paths defined in the ``setup.sh`` script. +Set parameters associated with the fixed (i.e. static) files. For the main NOAA HPC platforms, as well as Cheyenne, Odin, and Stampede, fixed files are prestaged with paths defined in the ``setup.sh`` script. ``FIXgsm``: (Default: “”) System directory in which the majority of fixed (i.e. time-independent) files that are needed to run the FV3-LAM model are located. @@ -221,7 +223,7 @@ Set parameters associated with the fixed (i.e. static) files. For the main NOAA ``CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING``: (Default: see below) .. code-block:: console - ("aerosol.dat | global_climaeropac_global.txt" \ + ("aerosol.dat | global_climaeropac_global.txt" \ "co2historicaldata_2010.txt | fix_co2_proj/global_co2historicaldata_2010.txt" \ "co2historicaldata_2011.txt | fix_co2_proj/global_co2historicaldata_2011.txt" \ "co2historicaldata_2012.txt | fix_co2_proj/global_co2historicaldata_2012.txt" \ diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index c6557c7848..b2ba1beaf0 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -78,6 +78,7 @@ On Cheyenne: .. code-block:: console + module load ncarenv ncar_pylib /glade/p/ral/jntp/UFS_SRW_app/ncar_pylib/python_graphics On Hera and Jet: diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index 3cb9e9a61a..f65fbe3572 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -275,14 +275,8 @@ For WCOSS_DELL_P3: module use -a /usrx/local/emc_rocoto/modulefiles module load rocoto/1.2.4 -For Gaea: - -.. code-block:: console - - module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles - module load rocoto/1.3.3 - -Then manually call ``rocotorun`` to launch the tasks and ``rocotostat`` to monitor the progress: +Then manually call ``rocotorun`` to launch the tasks that have all dependencies satisfied +and ``rocotostat`` to monitor the progress: .. code-block:: console @@ -290,15 +284,12 @@ Then manually call ``rocotorun`` to launch the tasks and ``rocotostat`` to monit rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -For automatic resubmission of the workflow (every 3 minutes), one of the following lines can be added -to the user's crontab (use ``crontab -e`` to edit the cron table) depending on your preference of how -you call Rocoto: +For automatic resubmission of the workflow (e.g., every 3 minutes), the following line can be added +to the user's crontab (use ``crontab -e`` to edit the cron table). .. code-block:: console */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && ./launch_FV3LAM_wflow.sh - -- OR -- - */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 .. note:: diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index 8259ea548e..14e1b9d0ac 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -401,11 +401,6 @@ On Cheyenne: module load ncarenv ncar_pylib /glade/p/ral/jntp/UFS_SRW_app/ncar_pylib/regional_workflow - -Load the Rocoto module: - -.. code-block:: console - module use -a /glade/p/ral/jntp/UFS_SRW_app/modules module load rocoto @@ -426,6 +421,13 @@ On Orion: module load miniconda3 conda activate regional_workflow +On Gaea: + +.. code-block:: console + + module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles + module load miniconda3/4.8.3-regional-workflow + module load rocoto/1.3.3 .. _GeneratingWflowExpt: @@ -494,10 +496,10 @@ delete these two *.db files and then call the launch script repeatedly for each +----------------------+------------------------------------------------------------+ | **Workflow Task** | **Task Description** | +======================+============================================================+ - | make_grid | Pre-processing task to generate regional grid files. Can | + | make_grid | Pre-processing task to generate regional grid files. Can | | | be run, at most, once per experiment. | +----------------------+------------------------------------------------------------+ - | make_orog | Pre-processing task to generate orography files. Can be | + | make_orog | Pre-processing task to generate orography files. Can be | | | run, at most, once per experiment. | +----------------------+------------------------------------------------------------+ | make_sfc_climo | Pre-processing task to generate surface climatology files. | From 4de8a7f7555c3f0873c97ba1cceb1ef8b98ff318 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Mon, 1 Mar 2021 00:11:37 +0000 Subject: [PATCH 13/17] Another small edit --- docs/UsersGuide/source/Quickstart.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index f65fbe3572..6d87dcd15a 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -223,7 +223,8 @@ To run Rocoto using the script: cd $EXPTDIR ./launch_FV3LAM_wflow.sh -A log file called ``log.launch_FV3LAM_wflow`` is created in the ``$EXPTDIR``. +Once the workflow is launched with the ``launch_FV3LAM_wflow.sh`` script, a log file named +``log.launch_FV3LAM_wflow`` will be created (or appended to it if it already exists) in ``EXPTDIR``. Or to manually call Rocoto: From 83b5417af9c51fc78683e27aa491cce420824630 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Mon, 1 Mar 2021 00:51:12 +0000 Subject: [PATCH 14/17] A few more edits to the graphics chapter to add an example command line for the diff plot --- docs/UsersGuide/source/Graphics.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index b2ba1beaf0..a5d80cd438 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -150,6 +150,13 @@ seven command line arguments: #. The top level of the first experiment directory ``EXPTDIR2`` containing the second set of post-processed data. The script will look for the data files in the directory ``EXPTDIR2/CDATE/postprd``. #. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shape files (``*.shp``) in the directory ``CARTOPY_DIR/shapefiles/natural_earth/cultural``. +An example of plotting differences from two experiments for the same date and predefined domain where one uses +the "FV3_GFS_v15p2" suite definition file (SDF) and one using the "FV3_RRFS_v1alpha" SDF is as follows: + +.. code-block:: console + + python plot_allvars_diff.py 2019061518 6 18 3 /path-to/expt_dirs1/test_CONUS_3km_GFSv15p2 /path-to/expt_dirs2/test_CONUS_3km_RRFSv1alpha /path-to/NaturalEarth + In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``. From 2181f246ec537d91dd63e729cfdaa04ca22adeff Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Mon, 1 Mar 2021 22:17:54 +0000 Subject: [PATCH 15/17] Updated some remaining concens raised by Dom. This will now be tested and the final issues will be added prior to a PR. --- docs/UsersGuide/source/Graphics.rst | 3 +- docs/UsersGuide/source/SRWAppOverview.rst | 46 +++++------------------ 2 files changed, 11 insertions(+), 38 deletions(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index a5d80cd438..ffbe904e02 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -56,7 +56,7 @@ On Orion: .. code-block:: console - /work/noaa/global/glopara/fix/NaturalEarth + /work/noaa/gsd-fv3-dev/UFS_SRW_App/v1p0/fix_files/NaturalEarth On Gaea: @@ -102,7 +102,6 @@ On Gaea: .. code-block:: console module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles - module load rocoto/1.3.3 module load miniconda3/4.8.3-regional-workflow .. note:: diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index 14e1b9d0ac..ef71c80805 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -391,43 +391,16 @@ values in ``config_default.sh`` and the values defined in ``config.community.sh` Python Environment for Workflow =============================== -It is necessary to load the appropriate Python environment for the workflow. The workflow -requires Python 3, with the packages 'PyYAML', 'Jinja2', and 'f90nml' available. This Python -environment has already been set up on Level 1 platforms, and can be activated in the following way: - -On Cheyenne: +It is necessary to load the appropriate Python environment for the workflow. +The workflow requires Python 3, with the packages 'PyYAML', 'Jinja2', and 'f90nml' available. +This Python environment has already been set up on Level 1 platforms, and can be activated in +the following way: .. code-block:: console - module load ncarenv - ncar_pylib /glade/p/ral/jntp/UFS_SRW_app/ncar_pylib/regional_workflow - module use -a /glade/p/ral/jntp/UFS_SRW_app/modules - module load rocoto - -On Hera and Jet: - -.. code-block:: console - - module use -a /contrib/miniconda3/modulefiles - module load miniconda3 - conda activate regional_workflow - module load rocoto - -On Orion: - -.. code-block:: console - - module use -a /apps/contrib/miniconda3-noaa-gsl/modulefiles - module load miniconda3 - conda activate regional_workflow - -On Gaea: - -.. code-block:: console + source ../../env/wflow_.env - module use /lustre/f2/pdata/esrl/gsd/contrib/modulefiles - module load miniconda3/4.8.3-regional-workflow - module load rocoto/1.3.3 +when in the ``ufs-srweather-app/regional_workflow/ush`` directory. .. _GeneratingWflowExpt: @@ -655,7 +628,7 @@ Rocoto software is not available on a given platform. These scripts are located a wrapper script to set environment variables and run the job script. Example batch-submit scripts for Hera (Slurm) and Cheyenne (PBS) are included: ``sq_job.sh`` -and ``qsub_job.sh``. These examples set the build and run environment for Hera or Cheyenne +and ``qsub_job.sh``, respectively. These examples set the build and run environment for Hera or Cheyenne so that run-time libraries match the compiled libraries (i.e. netcdf, mpi). Users may either modify the submit batch script as each task is submitted, or duplicate this batch wrapper for their system settings for each task. Alternatively, some batch systems allow users to @@ -668,7 +641,8 @@ be run concurrently (no dependency). .. table:: List of tasks in the regional workflow in the order that they are executed. Scripts with the same stage number may be run simultaneously. The number of - processors is typical for Cheyenne or Hera. + processors and wall clock time is a good starting point for Cheyenne or Hera + when running a 48-h forecast on the 25-km CONUS domain. +------------+------------------------+----------------+----------------------------+ | **Stage/** | **Task Run Script** | **Number of** | **Wall clock time (H:MM)** | @@ -690,7 +664,7 @@ be run concurrently (no dependency). +------------+------------------------+----------------+----------------------------+ | 4 | run_make_lbcs.sh | 48 | 0:30 | +------------+------------------------+----------------+----------------------------+ - | 5 | run_fcst.sh | 48 | 2:30 | + | 5 | run_fcst.sh | 48 | 0:30 | +------------+------------------------+----------------+----------------------------+ | 6 | run_post.sh | 48 | 0:25 (2 min per output | | | | | forecast hour) | From 2bb32e92ebdb17f187029976bad731dca6fdb8ea Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 2 Mar 2021 02:28:48 +0000 Subject: [PATCH 16/17] More updates to the graphics chapter --- docs/UsersGuide/source/Graphics.rst | 38 ++++++++++++++++++----------- 1 file changed, 24 insertions(+), 14 deletions(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index ffbe904e02..27256a611a 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -158,26 +158,22 @@ the "FV3_GFS_v15p2" suite definition file (SDF) and one using the "FV3_RRFS_v1al In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``. - If the Python scripts are being used to create plots of multiple forecast lead times and forecast -variables, then they should be submitted to the batch system using either something like the ``sq_job.sh`` -or ``sq_job_diff.sh`` script (for a platform such as Hera that uses the slurm -job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for a platform such as -Cheyenne that uses PBS or PBS Pro as the job scheduler). Examples of these scripts are located under +variables, then you may need to submit them to the batch system. Example scripts are provided called +``sq_job.sh`` and ``sq_job_diff.sh`` for use on a platform such as Hera that uses the Slurm +job scheduler or ``qsub_job.sh`` and ``qsub_job_diff.sh`` for use on a platform such as +Cheyenne that uses PBS as the job scheduler. Examples of these scripts are located under ``ufs-srweather-app/regional_workflow/ush/Python`` and can be used as a starting point to create a batch script -for your platform/job scheduler of use. The scripts must be submitted using the command appropriate -for the job scheduler used on your platform. For example, on Hera, -``sq_job.sh`` can be submitted as follows: - -.. code-block:: console +for your platform/job scheduler of use. - sbatch sq_job.sh - -On Cheyenne, ``qsub_job.sh`` can be submitted as follows: +At a minimum, the account should be set appropriately prior to job submission: .. code-block:: console - qsub qsub_job.sh + #SBATCH --account=an_account + +Depending on the platform you are running on, you may also need to adjust the settings to use +the correct Python environment and path to the shape files. When using these batch scripts, several environment variables must be set prior to submission. If plotting output from a single cycle, the variables to set are ``HOMErrfs`` and ``EXPTDIR``. @@ -234,3 +230,17 @@ and ending with the last forecast hour, use export FCST_START=6 export FCST_END=${FCST_LEN_HRS} export FCST_INC=6 + +The scripts must be submitted using the command appropriate +for the job scheduler used on your platform. For example, on Hera, +``sq_job.sh`` can be submitted as follows: + +.. code-block:: console + + sbatch sq_job.sh + +On Cheyenne, ``qsub_job.sh`` can be submitted as follows: + +.. code-block:: console + + qsub qsub_job.sh From 88043dd8bc9429942601b51a3a2dd806dc285f02 Mon Sep 17 00:00:00 2001 From: Jamie Wolff Date: Tue, 2 Mar 2021 02:40:51 +0000 Subject: [PATCH 17/17] Added subsections for easier reading --- docs/UsersGuide/source/Graphics.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index 27256a611a..ccff303413 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -109,6 +109,9 @@ On Gaea: If using one of the batch submission scripts described below, the user does not need to manually load an environment because the scripts perform this task. +Plotting output from one experiment +=================================== + Before generating plots, it is convenient to change location to the directory containing the plotting scripts: @@ -138,6 +141,9 @@ The output files (in .png format) will be located in the directory ``EXPTDIR/CDA where in this case ``EXPTDIR`` is ``/path-to/expt_dirs/test_CONUS_25km_GFSv15p2`` and ``CDATE`` is ``2019061500``. +Plotting differences from two experiments +========================================= + To generate difference plots, the ``plot_allvars_diff.py`` script must be called with the following seven command line arguments: @@ -158,6 +164,9 @@ the "FV3_GFS_v15p2" suite definition file (SDF) and one using the "FV3_RRFS_v1al In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``. +Submitting plotting scripts through a batch system +================================================== + If the Python scripts are being used to create plots of multiple forecast lead times and forecast variables, then you may need to submit them to the batch system. Example scripts are provided called ``sq_job.sh`` and ``sq_job_diff.sh`` for use on a platform such as Hera that uses the Slurm