From 5ed71b8f3ae5b3d269b6102164950533e6ee74fa Mon Sep 17 00:00:00 2001 From: Dom Heinzeller Date: Sat, 27 Feb 2021 15:11:59 -0700 Subject: [PATCH] Corrections and notes for users guide --- docs/UsersGuide/source/CodeReposAndDirs.rst | 4 ++ docs/UsersGuide/source/ConfigNewPlatform.rst | 32 +++++++------ docs/UsersGuide/source/ConfigParameters.inc | 16 +++---- docs/UsersGuide/source/ConfigWorkflow.rst | 12 +++-- docs/UsersGuide/source/Graphics.rst | 12 ++++- docs/UsersGuide/source/InputOutputFiles.rst | 11 +++-- docs/UsersGuide/source/Introduction.rst | 2 +- docs/UsersGuide/source/LAMGrids.rst | 14 +++--- docs/UsersGuide/source/Quickstart.rst | 22 ++++++++- docs/UsersGuide/source/SRWAppOverview.rst | 48 +++++++++++++------- 10 files changed, 114 insertions(+), 59 deletions(-) diff --git a/docs/UsersGuide/source/CodeReposAndDirs.rst b/docs/UsersGuide/source/CodeReposAndDirs.rst index edfdab16e7..0ce8f9047c 100644 --- a/docs/UsersGuide/source/CodeReposAndDirs.rst +++ b/docs/UsersGuide/source/CodeReposAndDirs.rst @@ -117,6 +117,8 @@ contents of these sub-directories are described in :numref:`Table %s `. +NOTE DOM 20220227 better to say "Tracers" instead of "Scalar fields in the table below", to be consistent with section 5.7 (also makes more sense) + .. _ExptDirStructure: .. table:: Files and sub-directory initially created in the experimental directory. diff --git a/docs/UsersGuide/source/ConfigNewPlatform.rst b/docs/UsersGuide/source/ConfigNewPlatform.rst index 5c39d86d1f..369e7cde3b 100644 --- a/docs/UsersGuide/source/ConfigNewPlatform.rst +++ b/docs/UsersGuide/source/ConfigNewPlatform.rst @@ -14,11 +14,11 @@ The first step to installing on a new machine is to install :term:`NCEPLIBS` (ht * C and C++ compilers compatible with the Fortran compiler - * gcc v9+, ifort v18+, and clang (MacOS) have been tested + * gcc v9+, ifort v18+, and clang v9+ (macOS, native Apple clang or LLVM clang) have been tested * Python v3.6+ - * Prerequisite packages must be downloaded: jinja2, yaml and f90nml, as well as pygrib if the user would like to use the provided graphics scripts + * Prerequisite packages must be downloaded: jinja2, yaml and f90nml, as well as a number of additional Python modules (see :numref:`Section %s ` if the user would like to use the provided graphics scripts * Perl 5 @@ -28,16 +28,20 @@ The first step to installing on a new machine is to install :term:`NCEPLIBS` (ht * CMake v3.15+ is needed for building NCEPLIBS, but versions as old as 3.12 can be used to build NCEPLIBS-external, which contains a newer CMake that can be used for the rest of the build. -For Linux systems, as long as the above software is available, you can move on to the next step: installing the :term:`NCEPLIBS-external` package. - -For MacOS systems, you will also need to set the stack size to “unlimited”. +For both Linux and macOS, you will need to set the stack size to "unlimited" (if allowed) or the largest possible value .. code-block:: console + # Linux, if allowed + ulimit -s unlimited + + # macOS, this corresponds to 65MB ulimit -S -s unlimited -Additionally, some extra software is needed: ``wget``, ``coreutils``, ``pkg-config``, and ``gnu-sed``. -It is recommended that you install this software using the Homebrew package manager for MacOS (https://brew.sh/): +For Linux systems, as long as the above software is available, you can move on to the next step: installing the :term:`NCEPLIBS-external` package. + +For macOS systems, some extra software is needed: ``wget``, ``coreutils``, ``pkg-config``, and ``gnu-sed``. +It is recommended that you install this software using the Homebrew package manager for macOS (https://brew.sh/): * brew install wget @@ -160,7 +164,7 @@ At this point there are just a few more variables that need to be set prior to b export CMAKE_CXX_COMPILER=mpicxx export CMAKE_Fortran_COMPILER=mpifort -If you are using your machine’s built-in MPI compilers, it is recommended you set the ``CMAKE_*_COMPILER`` flags to full paths to ensure that the correct MPI aliases are used. Finally, one last environment variable, ``CMAKE_Platform``, must be set. This will depend on your machine; for example, on a MacOS operating system with GNU compilers: +If you are using your machine’s built-in MPI compilers, it is recommended you set the ``CMAKE_*_COMPILER`` flags to full paths to ensure that the correct MPI aliases are used. Finally, one last environment variable, ``CMAKE_Platform``, must be set. This will depend on your machine; for example, on a macOS operating system with GNU compilers: .. code-block:: console @@ -198,7 +202,7 @@ Running the graphics scripts in ``${WORKDIR}/ufs-srweather-app/regional_workflow For the final step of creating and running an experiment, the exact methods will depend on if you are running with or without a workflow manager (Rocoto). -Running Without a Workflow Manager: Generic Linux and MacOS Platforms +Running Without a Workflow Manager: Generic Linux and macOS Platforms ===================================================================== Now that the code has been built, you can stage your data as described in :numref:`Section %s `. @@ -244,7 +248,7 @@ From here, you can run each individual task of the UFS SRW App using the provide cp ${WORKDIR}/ufs-srweather-app/regional_workflow/ush/wrappers/*sh . cp ${WORKDIR}/ufs-srweather-app/regional_workflow/ush/wrappers/README.md . -The ``README.md`` file will contain instructions on the order that each script should be run in. An example of wallclock times for each task for an example run (2017 Macbook Pro, MacOS Catalina, 25km CONUS domain, 48hr forecast) is listed in :numref:`Table %s `. +The ``README.md`` file will contain instructions on the order that each script should be run in. An example of wallclock times for each task for an example run (2017 Macbook Pro, macOS Catalina, 25km CONUS domain, 48hr forecast) is listed in :numref:`Table %s `. .. _WallClockTimes: @@ -341,7 +345,7 @@ Those requirements highlighted in **bold** are included in the NCEPLIBS-external * 4GB memory (CONUS 25km domain) -* Fortran compiler with full Fortran 2003 standard support +* Fortran compiler with full Fortran 2008 standard support * C and C++ compiler @@ -361,13 +365,13 @@ Those requirements highlighted in **bold** are included in the NCEPLIBS-external * **netCDF (C and Fortran libraries)** * **HDF5** - * **ESMF** + * **ESMF** 8.0.0 * **Jasper** * **libJPG** * **libPNG** * **zlib** -MacOS-specific prerequisites: +macOS-specific prerequisites: * brew install wget * brew install cmake @@ -381,4 +385,4 @@ Optional but recommended prerequisites: * Bash v4+ * Rocoto Workflow Management System (1.3.1) * **CMake v3.15+** -* Python package pygrib for graphics +* Python packages scipy, matplotlib, pygrib, cartopy, and pillow for graphics diff --git a/docs/UsersGuide/source/ConfigParameters.inc b/docs/UsersGuide/source/ConfigParameters.inc index c10e916cf5..682a7f094d 100644 --- a/docs/UsersGuide/source/ConfigParameters.inc +++ b/docs/UsersGuide/source/ConfigParameters.inc @@ -5,7 +5,7 @@ Grid Generation Parameters ========================== ``GRID_GEN_METHOD``: (Default: “”) - This variable specifies the method to use to generate a regional grid in the horizontal. The only supported value of this parameter is “ESGgrid”, in which case the Extended Schmidt Gnomonic grid generation method developed by Jim Purser of EMC will be used. + This variable specifies the method to use to generate a regional grid in the horizontal. The only supported value of this parameter is “ESGgrid”, in which case the Extended Schmidt Gnomonic grid generation method developed by Jim Purser of EMC will be used. NOTE DOM 20220227 insert proper reference here or remove "developed by ..." .. note:: @@ -49,11 +49,11 @@ Computational Forecast Parameters ``BLOCKSIZE``: (Default: “”) The amount of data that is passed into the cache at a time. -Here, we set these parameters to null strings. This is so that, for any one of these parameters: +Here, we set these parameters to null strings. This is so that, for any one of these parameters: -#. If the experiment is using a predefined grid, then if the user sets the parameter in the user-specified experiment configuration file (``EXPT_CONFIG_FN``), that value will be used in the forecast(s). Otherwise, the default value of the parameter for that predefined grid will be used. +#. If the experiment is using a predefined grid and the user sets the parameter in the user-specified experiment configuration file (``EXPT_CONFIG_FN``), that value will be used in the forecast(s). Otherwise, the default value of the parameter for that predefined grid will be used. -#. If the experiment is not using a predefined grid (i.e. it is using a custom grid whose parameters are specified in the experiment configuration file), then the user must specify a value for the parameter in that configuration file. Otherwise, the parameter will remain set to a null string, and the experiment generation will fail because the generation scripts check to ensure that all the parameters defined in this section are set to non-empty strings before creating the experiment directory. +#. If the experiment is not using a predefined grid (i.e. it is using a custom grid whose parameters are specified in the experiment configuration file), then the user must specify a value for the parameter in that configuration file. Otherwise, the parameter will remain set to a null string, and the experiment generation will fail, because the generation scripts check to ensure that all the parameters defined in this section are set to non-empty strings before creating the experiment directory. Write-Component (Quilting) Parameters ===================================== @@ -91,7 +91,7 @@ Currently supported ``PREDEF_GRID_NAME`` options are "RRFS_CONUS_25km," "RRFS_CO Pre-existing Directory Parameter ================================ ``PREEXISTING_DIR_METHOD``: (Default: “delete”) - This variable determines the method to use to deal with pre-existing directories [e.g ones generated by previous calls to the experiment generation script using the same experiment name (``EXPT_SUBDIR``) as the current experiment]. This variable must be set to one of "delete", "rename", and "quit". The resulting behavior for each of these values is as follows: + This variable determines the method to deal with pre-existing directories [e.g ones generated by previous calls to the experiment generation script using the same experiment name (``EXPT_SUBDIR``) as the current experiment]. This variable must be set to one of "delete", "rename", and "quit". The resulting behavior for each of these values is as follows: * "delete": The preexisting directory is deleted and a new directory (having the same name as the original preexisting directory) is created. @@ -128,12 +128,12 @@ These parameters set flags (and related directories) that determine whether the Surface Climatology Parameter ============================= -``SFC_CLIMO_FIELDS``: (Default: “("facsf" "maximum_snow_albedo" "slope_type" "snowfree_albedo" "soil_type" "substrate_temperature" "vegetation_greenness" "vegetation_type")” +``SFC_CLIMO_FIELDS``: (Default: “("facsf" "maximum_snow_albedo" "slope_type" "snowfree_albedo" "soil_type" "substrate_temperature" "vegetation_greenness" "vegetation_type")”) Array containing the names of all the fields for which the ``MAKE_SFC_CLIMO_TN`` task generates files on the native FV3-LAM grid. Fixed File Parameters ===================== -Set parameters associated with the fixed (i.e. static) files. For the main NOAA HPC platforms, as well as Cheyenne, Odin, and Stampede, fixed files are prestaged in locations on each machine with paths defined in the ``setup.sh`` script. +Set parameters associated with the fixed (i.e. static) files. For the main NOAA HPC platforms, as well as Cheyenne, Odin, and Stampede, fixed files are prestaged with paths defined in the ``setup.sh`` script. ``FIXgsm``: (Default: “”) System directory in which the majority of fixed (i.e. time-independent) files that are needed to run the FV3-LAM model are located. @@ -221,7 +221,7 @@ Set parameters associated with the fixed (i.e. static) files. For the main NOAA ``CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING``: (Default: see below) .. code-block:: console - ("aerosol.dat | global_climaeropac_global.txt" \ + ("aerosol.dat | global_climaeropac_global.txt" \ "co2historicaldata_2010.txt | fix_co2_proj/global_co2historicaldata_2010.txt" \ "co2historicaldata_2011.txt | fix_co2_proj/global_co2historicaldata_2011.txt" \ "co2historicaldata_2012.txt | fix_co2_proj/global_co2historicaldata_2012.txt" \ diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index a8bfe8e0a7..f86312c37d 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -5,7 +5,7 @@ Configuring the Workflow: ``config.sh`` and ``config_defaults.sh`` ================================================================== To create the experiment directory and workflow when running the SRW App, the user must create an experiment configuration file named ``config.sh``. This file contains experiment-specific information, such as dates, external model data, directories, and other relevant settings. To help the user, two sample configuration files have been included in the ``regional_workflow`` repository’s ``ush`` directory: ``config.community.sh`` and ``config.nco.sh``. The first is for running experiments in community mode (``RUN_ENVIR`` set to “community”; see below), and the second is for running experiments in “nco” mode (``RUN_ENVIR`` set to “nco”). Note that for this release, only “community” mode is supported. These files can be used as the starting point from which to generate a variety of experiment configurations in which to run the SRW App. -There is an extensive list of experiment parameters that a user can set when configuring the experiment. All of these do not need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ```config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``) Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). +There is an extensive list of experiment parameters that a user can set when configuring the experiment. Not all of these need to be explicitly set by the user in ``config.sh``. In the case that a user does not define an entry in the ``config.sh`` script, either its value in ``config_defaults.sh`` will be used, or it will be reset depending on other parameters, e.g. the platform on which the experiment will be run (specified by ``MACHINE``). Note that ``config_defaults.sh`` contains the full list of experiment parameters that a user may set in ``config.sh`` (i.e. the user cannot set parameters in config.sh that are not initialized in ``config_defaults.sh``). The following is a list of the parameters in the ``config_defaults.sh`` file. For each parameter, the default value and a brief description is given. In addition, any relevant information on features and settings supported or unsupported in this release is specified. @@ -68,7 +68,7 @@ These settings control run commands for platforms without a workflow manager. V The run command for pre-processing utilities (shave, orog, sfc_climo_gen, etc.). This can be left blank for smaller domains, in which case the executables will run without MPI. ``RUN_CMD_FCST``: (Default: "mpirun -np \${PE_MEMBER01}") - The run command for the model forecast step. This will be appended to the end of the variable definitions file (var_defns.sh).. + The run command for the model forecast step. This will be appended to the end of the variable definitions file ("var_defns.sh"). ``RUN_CMD_POST``: (Default: "mpirun -np 1") The run command for post-processing (UPP). Can be left blank for smaller domains, in which case UPP will run without MPI. @@ -86,6 +86,8 @@ Directory Parameters ``EXPT_BASEDIR``: (Default: “”) The base directory in which the experiment directory will be created. If this is not specified or if it is set to an empty string, it will default to ``${HOMErrfs}/../expt_dirs``, where ``${HOMErrfs}`` contains the full path to the ``regional_workflow`` directory. +NOTE DOM 20220227 need to double check, beforehamnd expt_dirs sits in ufs-srweather-app/../expt_dirs, i.e. homerrfs would be ufs-srweather-app + ``EXPT_SUBDIR``: (Default: “”) The name that the experiment directory (without the full path) will have. The full path to the experiment directory, which will be contained in the variable ``EXPTDIR``, will be: @@ -211,7 +213,7 @@ Initial and Lateral Boundary Condition Generation Parameters The name of the external model that will provide fields from which lateral boundary condition (LBC) files (except for the 0-th hour LBC file) will be generated for input into the forecast model. ``LBC_SPEC_INTVL_HRS``: (Default: “6”) - The interval (in integer hours) with which LBC files will be generated, referred to as the boundary specification interval. Note that the model specified in ``EXTRN_MDL_NAME_LBCS`` must have data available at a frequency greater than or equal to that implied by ``LBC_SPEC_INTVL_HRS``. For example, if ``LBC_SPEC_INTVL_HRS`` is set to 6, then the model must have data available at least every 6 hours. It is up to the user to ensure that this is the case. + The interval (in integer hours) at which LBC files will be generated, referred to as the boundary specification interval. Note that the model specified in ``EXTRN_MDL_NAME_LBCS`` must have data available at a frequency greater than or equal to that implied by ``LBC_SPEC_INTVL_HRS``. For example, if ``LBC_SPEC_INTVL_HRS`` is set to 6, then the model must have data available at least every 6 hours. It is up to the user to ensure that this is the case. ``FV3GFS_FILE_FMT_ICS``: (Default: “nemsio”) If using the FV3GFS model as the source of the ICs (i.e. if ``EXTRN_MDL_NAME_ICS`` is set to "FV3GFS"), this variable specifies the format of the model files to use when generating the ICs. @@ -225,7 +227,7 @@ User-Staged External Model Directory and File Parameters Flag that determines whether or not the workflow will look for the external model files needed for generating ICs and LBCs in user-specified directories (as opposed to fetching them from mass storage like NOAA HPSS). ``EXTRN_MDL_SOURCE_BASEDIR_ICS``: (Default: “/base/dir/containing/user/staged/extrn/mdl/files/for/ICs") - Directory in which to look for external model files for generating ICs. If ``USE_USER_STAGED_EXTRN_FILES`` is set to "TRUE", the workflow looks in this directory (specifically, in a subdirectory under this directory named "YYYYMMDDHH" consisting of the starting date and cycle hour of the forecast, where YYYY is the 4-digit year, MM the 2-digit month, DD the 2-digit day of the month, and HH the 2-digit hour of the day) for the external model files specified by the array ``EXTRN_MDL_FILES_ICS``` (these files will be used to generate the ICs on the native FV3-LAM grid). This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". + Directory in which to look for external model files for generating ICs. If ``USE_USER_STAGED_EXTRN_FILES`` is set to "TRUE", the workflow looks in this directory (specifically, in a subdirectory under this directory named "YYYYMMDDHH" consisting of the starting date and cycle hour of the forecast, where YYYY is the 4-digit year, MM the 2-digit month, DD the 2-digit day of the month, and HH the 2-digit hour of the day) for the external model files specified by the array ``EXTRN_MDL_FILES_ICS`` (these files will be used to generate the ICs on the native FV3-LAM grid). This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". ``EXTRN_MDL_FILES_ICS``: (Default: "ICS_file1” “ICS_file2” “...”) Array containing the names of the files to search for in the directory specified by ``EXTRN_MDL_SOURCE_BASEDIR_ICS``. This variable is not used if ``USE_USER_STAGED_EXTRN_FILES`` is set to "FALSE". @@ -239,6 +241,6 @@ User-Staged External Model Directory and File Parameters CCPP Parameter ============== ``CCPP_PHYS_SUITE``: (Default: "FV3_GFS_v15p2") - The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “RRFS_v1alpha.” + The CCPP (Common Community Physics Package) physics suite to use for the forecast(s). The choice of physics suite determines the forecast model's namelist file, the diagnostics table file, the field table file, and the XML physics suite definition file that are staged in the experiment directory or the cycle directories under it. Current supported settings for this parameter are “FV3_GFS_v15p2” and “FV3_RRFS_v1alpha”. .. include:: ConfigParameters.inc diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index ac46a495cf..80ac6c05c1 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -54,6 +54,8 @@ On Jet: On Orion: +NOTE DOM 20220227 - why is this not in a shared, user-independent directory? We have /work/noaa/nems/emc.nemspara and /work/noaa/nems/glopara, the latter already hosts the fixed files etc. A conditionn for tier-1 platforms was that no data/scripts/... resides in user directories. This disqualified stampede from being a tier-1 platform. + .. code-block:: console /home/chjeon/tools/NaturalEarth @@ -82,6 +84,8 @@ On Cheyenne: On Hera and Jet: +NOTE DOM 20220227 - This /apps directory doesn't exist on jet (haven't checked hera) + .. code-block:: console module use -a /contrib/miniconda3/modulefiles @@ -98,6 +102,10 @@ On Orion: On Gaea: +NOTE DOM 20220227 - This /apps directory doesn't exist. I added all the plotting libraries to the regional_workflow environment, +therefore one should do what is in env/wflow_gaea.env. Makes a lot of sense anyway to be able to use the same Python environment to generate +and run the workflow and to plot the results. Note that I didn't check any of the other platforms, just gaea. + .. code-block:: console module use -a /apps/contrib/miniconda3-noaa-gsl/modulefiles @@ -161,6 +169,8 @@ Cheyenne that use PBS or PBS Pro as the job scheduler). These scripts are locat for the job scheduler used on the current platform. For example, on Hera, Jet, Orion, and Gaea, ``sq_job.sh`` can be submitted as follows: +NOTE DOM 20220227 - this script will *not* work on gaea for sure. First of all because the module load commands are wrong (see above, second because gaea uses #SBATCH --clusters, third because you cannot do module purge and module load hpss). The line ". /apps/lmod/lmod/init/sh" doesn't work on gaea, it doesn't work on jet either. + .. code-block:: console sbatch sq_job.sh @@ -226,5 +236,3 @@ and ending with the last forecast hour, use export FCST_START=6 export FCST_END=${FCST_LEN_HRS} export FCST_INC=6 - - diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index 3fc839a0cd..193a7e9d16 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -75,12 +75,12 @@ and are shown in :numref:`Table %s `. | | the start of each forecast. It is an empty file. No need to | | | change. | +-----------------------------+-------------------------------------------------------------+ - | data_table_[CCPP] | File specifying the output fields of the forecast model. | + | diag_table_[CCPP] | File specifying the output fields of the forecast model. | | | A different diag_table may be configured for different | | | CCPP suites. | +-----------------------------+-------------------------------------------------------------+ | field_table_[CCPP] | Cycle-independent file that the forecast model reads in at | - | | the start of each forecast. It specifies the scalars that | + | | the start of each forecast. It specifies the tracers that | | | the forecast model will advect. A different field_table | | | may be needed for different CCPP suites. | +-----------------------------+-------------------------------------------------------------+ @@ -237,8 +237,8 @@ Static Files A set of fix files are necessary to run the SRW Application. Environment variables describe the location of the static files: ``FIXgsm``, ``TOPO_DIR``, and ``SFC_CLIMO_INPUT_DIR`` are the directories where the static files are located. If you are on a pre-configured or configurable platform, there is no -need to stage the fixed files manually because they are already available on those platforms and the paths -are set in ``regional_workflow/ush/setup.sh`` for the static files. If the users platform is not defined +need to stage the fixed files manually because they have been prestages and the paths +are set in ``regional_workflow/ush/setup.sh``. If the user's platform is not defined in that file, the static files can be pulled individually or as a full tar file from the `FTP data repository `_ or from `Amazon Web Services (AWS) cloud storage `_ @@ -405,6 +405,9 @@ are initializing from NEMSIO format FV3GFS files. Best Practices for Conserving Disk Space and Keeping Files Safe --------------------------------------------------------------- + +NOTE DOM 20220227 is this section really necessary? + Initial and lateral boundary condition files are large and can occupy a significant amount of disk space. If various users will employ a common files system to conduct runs, it is recommended that the users share the same ``EXTRN_MDL_SOURCE_BASEDIR_ICS`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS`` diff --git a/docs/UsersGuide/source/Introduction.rst b/docs/UsersGuide/source/Introduction.rst index 4e81dd2629..7da355e7f6 100644 --- a/docs/UsersGuide/source/Introduction.rst +++ b/docs/UsersGuide/source/Introduction.rst @@ -145,7 +145,7 @@ have been defined for the SRW Application, including pre-configured (level 1), c (level 2), limited test platforms (level 3), and build only platforms (level 4). Each level is further described below. -For the select computational platforms that have been pre-configured (level 1), all the +For the selected computational platforms that have been pre-configured (level 1), all the required libraries for building the SRW Application are available in a central place. That means bundled libraries (NCEPLIBS) and third-party libraries (NCEPLIBS-external) have both been built. The SRW Application is expected to build and run out of the box on these diff --git a/docs/UsersGuide/source/LAMGrids.rst b/docs/UsersGuide/source/LAMGrids.rst index 650ac91bc4..2f955b77df 100644 --- a/docs/UsersGuide/source/LAMGrids.rst +++ b/docs/UsersGuide/source/LAMGrids.rst @@ -31,19 +31,19 @@ of the following three options: The predefined grids are named after the prototype 3-km continental United States (CONUS) grid being tested for the Rapid Refresh Forecast System (RRFS), which will be a convection-allowing, hourly-cycled, FV3-LAM-based ensemble planned for operational implementation in 2024. To allow -for use of HRRR data to initialize the SRW App, all three supported grids were created to fit completely within -the HRRR domain to allow external model data from the HRRR to be used as initial conditions for -the FV3-LAM. Three resolution options were provided for flexibility related to compute resources +for use of HRRR data NOTE DOM 20220227 spell out and add link or reference +to initialize the SRW App, all three supported grids were created to fit completely within +the HRRR domain. Three resolution options were provided for flexibility related to compute resources and physics options. For example, a user may wish to use the 13-km or 25-km domain when running -with the ``FV3_GFS_v15p2`` suite definition file (SDF), since that SDF uses cumulus physics which is -not currently configured to run at 3-km. In addition, users will have much fewer computational +with the ``FV3_GFS_v15p2`` suite definition file (SDF), since that SDF uses cumulus physics that are +not configured to run at 3-km. In addition, users will have much fewer computational constraints when running with the 13-km and 25-km domains. The boundary of the ``RRFS_CONUS_3km`` domain is shown in :numref:`Figure %s ` (in red). Note that while it is possible to initialize the FV3-LAM with coarser external model data when using the ``RRFS_CONUS_3km`` domain, it is generally advised to use external model data that has a resolution similar to that of the native FV3-LAM (predefined) grid. In addition, this grid is ideal for running the -``FV3_RRFS_v1alpha`` SDF, since it was specifically created for convection-allowing scales, and is the +``FV3_RRFS_v1alpha`` suite definiton file (SDF), since this SDF was specifically created for convection-allowing scales and is the precursor to the operational physics suite that will be used in the RRFS. As can be seen in :numref:`Figure %s `, the boundary of the write-component grid (in blue) sits @@ -80,7 +80,7 @@ While the three predefined grids available in this release are ideal for users j out with the SRW App, more advanced users may wish to create their own grid for testing over a different region and/or with a different resolution. Creating a user-defined grid requires knowledge of how the SRW App workflow functions, in particular, understanding the set of -scripts that handle the workflow and experiment generation. It’s also important to note that +scripts that handle the workflow and experiment generation. It is also important to note that user-defined grids are not a supported feature of the current release, however information is being provided for the benefit of the FV3-LAM community. diff --git a/docs/UsersGuide/source/Quickstart.rst b/docs/UsersGuide/source/Quickstart.rst index 46ae160871..a5308a6e42 100644 --- a/docs/UsersGuide/source/Quickstart.rst +++ b/docs/UsersGuide/source/Quickstart.rst @@ -28,7 +28,7 @@ The necessary source code is publicly available on GitHub. To clone the release .. code-block:: console - git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git + git clone -b ufs-v1.0.0 https://github.com/ufs-community/ufs-srweather-app.git cd ufs-srweather-app Then, check out the submodules for the SRW application: @@ -76,7 +76,7 @@ Run ``cmake`` to set up the ``Makefile``, then run ``make``: Output from the build will be in the ``ufs-srweather-app/build/build.out`` file. When the build completes, you should see the forecast model executable ``NEMS.exe`` and eleven pre- and post-processing executables in the ``ufs-srweather-app/bin`` directory which are -described in :numref:`Table %s `. +described in :numref:`Table %s `. NOTE DOM 20220227 TABLE BROKEN LINK Generate the Workflow Experiment ================================ @@ -92,6 +92,7 @@ The first two steps depend on the platform being used and are described here for Set up ``config.sh`` file ------------------------- + The workflow requires a file called ``config.sh`` to specify the values of your experiment parameters. Two example templates are provided: ``config.community.sh`` and ``config.nco.sh`` and can be found in the ``ufs-srweather-app/regional_workflow/ush directory``. The first file is a minimal example for @@ -102,6 +103,8 @@ fully supported for this release while the operational mode will be more exclusi Central Operations (NCO) and those in the NOAA/NCEP/Environmental Modeling Center (EMC) working with NCO on pre-implementation testing. Sample config.sh files are discussed in this section for Level 1 platforms. +NOTE DOM 20220227 at this point the user is in directory build + Make a copy of ``config.community.sh`` to get started: .. code-block:: console @@ -162,6 +165,9 @@ project code for the account parameter: ACCOUNT="my_account" EXPT_SUBDIR="my_expt_name" + +NOTE DOM 20220227 gaea missing + .. _SetUpPythonEnv: Set up the Python and other Environment Parameters @@ -171,6 +177,8 @@ The workflow requires Python 3, with the packages 'PyYAML', 'Jinja2', and 'f90nm This Python environment has already been set up on Level 1 platforms, and can be activated in the following way: +NOTE DOM 20220227 at this point the user is in directory regional_workflow/ush (as required for the next section to make sense), better mention this explicitly and use source ../../env/wflow_jet.env ? + .. code-block:: console source ufs-srweather-app/env/wflow_.env @@ -179,6 +187,8 @@ Run the ``generate_FV3LAM_wflow.sh`` script ------------------------------------------- For all platforms, the workflow can then be generated with the command: +NOTE DOM 20220227 need to mention the logfile that the generate_fv3lam_workflow command creates (and where) + .. code-block:: console ./generate_FV3LAM_wflow.sh @@ -193,6 +203,10 @@ If Rocoto is not available, it is still possible to run the workflow using stand described in :numref:`Section %s `. There are two ways you can run the workflow with Rocoto using either the ``./launch_FV3LAM_wflow.sh`` or by hand. +NOTE DOM 20220227 the user is not told to do "export EXPTDIR ..." / "setenv EXPTDIR ..." + +NOTE DOM 20220227 need to mention the logfile that the launch_FV3LAM_wflow command creates (and where) + To run Rocoto using the script: .. code-block:: console @@ -202,6 +216,8 @@ To run Rocoto using the script: Or to manually call Rocoto: +NOTE DOM 20220227 on jet at least, rocotorun is not in the path, not even after running "source env/build_jet_intel.env" and "source ../../env/wflow_jet.env"; someone needs to check that on all platforms and insert a code block here to load the rocoto module. + .. code-block:: console cd $EXPTDIR @@ -212,6 +228,8 @@ For automatic resubmission of the workflow (every 3 minutes), one of the followi to the user's crontab (use ``crontab -e`` to edit the cron table) depending on your preference of how you call Rocoto: +NOTE DOM 20220227 rocotorun will not be in the path unless the user sets PATH=... in the crontab + .. code-block:: console */3 * * * * cd /glade/p/ral/jntp/$USER/expt_dirs/test_CONUS_25km_GFSv15p2 && /glade/p/ral/jntp/tools/rocoto/rocoto-1.3.1/bin/rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index 137d040c0c..908f2c8e2d 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -7,7 +7,7 @@ The UFS Short-Range Weather Application (SRW App) is an umbrella repository that ``manage_externals`` to check out all of the components required for the application. Once the build process is complete, all the files and executables necessary for a regional experiment are located in the ``regional_workflow`` and ``bin`` directories, respectively, under the ``ufs-srweather-app`` directory. -Users can utilize the pre-defined domains or build their own domain (details provided in `Chapter %s `). +Users can utilize the pre-defined domains or build their own domain (details provided in `Chapter %s `). NOTE DOM 20220227 link broken In either case, users must create/modify the case-specific (``config.sh``) and/or grid-specific configuration files (``set_predef_grid_params.sh``). The overall procedure is shown in :numref:`Figure %s `, with the scripts to generate and run the workflow shown in red. The steps are as follows: @@ -33,11 +33,11 @@ Each step will be described in detail in the following sections. Download from GitHub ==================== -Retrieve the UFS Short Range Weather Application (SRW App) repository from GitHub and checkout the ``release/public-v1`` branch: +Retrieve the UFS Short Range Weather Application (SRW App) repository from GitHub and checkout the ``ufs-v1.0.0`` tag: .. code-block:: console - git clone -b release/public-v1 https://github.com/ufs-community/ufs-srweather-app.git + git clone -b ufs-v1.0.0 https://github.com/ufs-community/ufs-srweather-app.git cd ufs-srweather-app The cloned repository contains the configuration files and sub-directories shown in @@ -52,7 +52,7 @@ The cloned repository contains the configuration files and sub-directories shown +================================+========================================================+ | CMakeLists.txt | Main cmake file for SRW App | +--------------------------------+--------------------------------------------------------+ - | Externals.cfg | Hashes of the GitHub repositories/branches for the | + | Externals.cfg | Tags of the GitHub repositories/branches for the | | | external repositories | +--------------------------------+--------------------------------------------------------+ | LICENSE.md | CC0 license information | @@ -66,7 +66,7 @@ The cloned repository contains the configuration files and sub-directories shown +--------------------------------+--------------------------------------------------------+ | env | Contains build and workflow environment files | +--------------------------------+--------------------------------------------------------+ - | docs | Contains Release notes, documentation, and Users' Guide| + | docs | Contains release notes, documentation, and Users' Guide| +--------------------------------+--------------------------------------------------------+ | manage_externals | Utility for checking out external repositories | +--------------------------------+--------------------------------------------------------+ @@ -85,7 +85,7 @@ Check out the external repositories, including regional_workflow, ufs-weather-mo ./manage_externals/checkout_externals This step will use the configuration ``Externals.cfg`` file in the ``ufs-srweather-app`` directory to -clone the specific hashes (version of codes) of the external repositories as listed in +clone the specific tags (version of codes) of the external repositories as listed in :numref:`Section %s `. .. _BuildExecutables: @@ -114,15 +114,17 @@ The following steps will build the pre-processing utilities, forecast model, and make dir cd build cmake .. -DCMAKE_INSTALL_PREFIX=.. - make -j 8 >& build.out & + make -j 4 >& build.out & where ``-DCMAKE_INSTALL_PREFIX`` specifies the location in which the ``bin``, ``include``, ``lib``, and ``share`` directories containing various components of the SRW App will be created, and its recommended value ``..`` denotes one directory up from the build directory. In the next line for -the ``make`` call, ``-j 8`` indicates the build will run in parallel with 8 threads. If this step is successful, the -executables listed in :numref:`Table %s ` will be located in the +the ``make`` call, ``-j 4`` indicates the build will run in parallel with 4 threads. If this step is successful, the +executables listed in :numref:`Table %s ` NOTE DOM 20220227 link broken will be located in the ``ufs-srweather-app/bin`` directory. +NOTE DOM 20220227 this table doesn't show in https://ufs-srweather-app.readthedocs.io/en/latest/SRWAppOverview.html + .. _exec_description: .. table:: Names and descriptions of the executables produced by the build step and used by the SRW App. @@ -146,7 +148,7 @@ executables listed in :numref:`Table %s ` will be located in t +------------------------+---------------------------------------------------------------------------------+ | orog | Generates orography, land mask, and gravity wave drag files from fixed files | +------------------------+---------------------------------------------------------------------------------+ - | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | + | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | +------------------------+---------------------------------------------------------------------------------+ | sfc_climo_gen | Creates surface climatology fields from fixed files for use in ``chgres_cube`` | +------------------------+---------------------------------------------------------------------------------+ @@ -184,6 +186,8 @@ parameters in the ``set_predef_grid_params`` script. | RRFS_CONUS_3km | ESG grid | lambert_conformal | +----------------------+-------------------+--------------------------------+ +NOTE DOM 20220227 would it be good to mention the number of horizontal grid points NX x NY in this table/section? Or any other information (min/max lat/lon etc)? or a plot of the three domains? Or provide a link to section 6.1? + Case-specific Configuration =========================== @@ -194,11 +198,12 @@ Default configuration: ``config_defaults.sh`` When generating a new experiment (described in detail in :numref:`Section %s `), the ``config_defaults.sh`` file is read first and assigns default values to the experiment parameters. Important configuration variables in the ``config_defaults.sh`` file are shown in -:numref:`Table %s `, with more documentation found in the file itself, and -in `Chapter %s `. Some of these default values are intentionally invalid in order +:numref:`Table %s ` NOTE DOM 20220227 broken link, with more documentation found in the file itself, and +in `Chapter %s ` NOTE DOM 20220227 broken link. Some of these default values are intentionally invalid in order to ensure that the user assigns valid values in the user-specified configuration ``config.sh`` file. Therefore, any settings provided in ``config.sh`` will override the default ``config_defaults.sh`` -settings. Note that there is usually no need for a user to modify the default configuration file. +settings. Note that there is usually no need for a user to modify the default configuration file. + .. _ConfigVarsDefault: .. table:: Configuration variables specified in the config_defaults.sh script. @@ -305,7 +310,9 @@ settings. Note that there is usually no need for a user to modify the default co +----------------------+------------------------------------------------------------+ | Compiler | COMPILER | +----------------------+------------------------------------------------------------+ - + +NOTE DOM 20220227 the COMPILER variable is mentioned here, but I don't see any place later where it is described how to switch between Intel and GNU on cheyenne, for example. I don't even see a note that both are supported on Cheyenne, but maybe that is ok as long as the user is referred to the "supported platforms" wiki page often enough. + .. _UserSpecificConfig: User-specific configuration: ``config.sh`` @@ -424,6 +431,7 @@ On Orion: module load miniconda3 conda activate regional_workflow +NOTE DOM 20220227 no load rocoto on orion? gaea missing! .. _GeneratingWflowExpt: @@ -472,7 +480,9 @@ illustrated in :numref:`Figure %s `, and each task is describe surface climatology fix files, these three tasks can be skipped by setting ``RUN_TASK_MAKE_GRID=”FALSE”``, ``RUN_TASK_MAKE_OROG=”FALSE”``, and ``RUN_TASK_MAKE_SFC_CLIMO=”FALSE”`` in the ``regional_workflow/ush/config.sh`` file before running the ``generate_FV3LAM_wflow.sh`` script. As shown in the figure, the ``FV3LAM_wflow.xml`` -file runs the specific j-job scripts in the prescribed order (``regional_workflow/jobs/JREGIONAL_[task name]``) +file runs the specific j-job scripts +NOTE DOM 20220227 j-job not familiar to all users +in the prescribed order (``regional_workflow/jobs/JREGIONAL_[task name]``) when the ``launch_FV3LAM_wflow.sh`` is submitted. Each j-job task has its own source script named ``exregional_[task name].sh`` in the ``regional_workflow/scripts`` directory. Two database files ``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are generated and updated by the Rocoto calls. @@ -485,6 +495,8 @@ delete these two *.db files and then call the launch script repeatedly for each *Flowchart of the workflow tasks* +NOTE DOM 20220227 all these "can be run, at most, ocne per experiment" - must be run exactly once per experiment ? + .. _WorkflowTasksTable: .. table:: Workflow tasks in SRW App @@ -522,6 +534,8 @@ There are two ways to launch the workflow using Rocoto: (1) with the ``launch_FV script, and (2) manually calling the ``rocotorun`` command. Moreover, you can run the workflow separately using stand-alone scripts. +NOTE DOM 20220227 need to tell the user to define EXPTDIR + Launch with the ``launch_FV3LAM_wflow.sh`` script ------------------------------------------------- To launch the ``launch_FV3LAM_wflow.sh`` script, simply call it without any arguments as follows: @@ -640,13 +654,15 @@ a wrapper script to set environment variables and run the job script. Example batch-submit scripts for Hera (Slurm) and Cheyenne (PBS) are included: ``sq_job.sh`` and ``qsub_job.sh``. These examples set the build and run environment for Hera or Cheyenne so that run-time libraries match the compiled libraries (i.e. netcdf, mpi). Users may either -modify the one submit batch script as each task is submitted, or duplicate this batch wrapper +modify the submit batch script as each task is submitted, or duplicate this batch wrapper for their system settings for each task. Alternatively, some batch systems allow users to specify most of the settings on the command line (with the ``sbatch`` or ``qsub`` command, for example). This piece will be unique to your platform. The tasks run by the regional workflow are shown in :numref:`Table %s `. Tasks with the same stage level may be run concurrently (no dependency). +NOTE DOM 20220227 need to say for which grid these timings are + .. _RegionalWflowTasks: .. table:: List of tasks in the regional workflow in the order that they are executed.