From 66e758d3728f55f2ebbc4768ae8199de3daf807e Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Wed, 24 Feb 2021 17:25:03 -0700 Subject: [PATCH 1/6] Fixes to the documentation. --- docs/UsersGuide/source/Graphics.rst | 1 + docs/UsersGuide/source/InputOutputFiles.rst | 6 +- docs/UsersGuide/source/WE2Etests.rst | 118 ++++++++++++-------- 3 files changed, 77 insertions(+), 48 deletions(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index b9e48646fc..4654b10cca 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -113,6 +113,7 @@ seven command line arguments: #. The top level of the first experiment directory ``EXPTDIR2`` containing the second set of post-processed data. The script will look for the data files in the directory ``EXPTDIR2/CDATE/postprd``. #. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shape files (``*.shp``) in the directory ``CARTOPY_DIR/shapefiles/natural_earth/cultural``. +In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``, The Cartopy shape files are available on a number of Tier 1 platforms in the following locations: diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index f8d91b6e69..3fc839a0cd 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -325,8 +325,10 @@ Default Initial and Lateral Boundary Conditions ----------------------------------------------- The default initial and lateral boundary condition files are set to be a severe weather case from 20190615 at 00 UTC. FV3GFS grib2 files are the default model and file format. A tar file -containing the model data for this case is available on the FTP data repository at: -https://ftp.emc.ncep.noaa.gov/EIB/UFS/SRW/v1p0/simple_test_case/. +(``gst_model_data.tar.gz``) containing the model data for this case is available on EMC's FTP +data repository at https://ftp.emc.ncep.noaa.gov/EIB/UFS/SRW/v1p0/simple_test_case/. It is +also available on Amazon Web Services (AWS) at +https://ufs-data.s3.amazonaws.com/public_release/ufs-srweather-app-v1.0.0/ic/gst_model_data.tar.gz. Running the App for Different Dates ----------------------------------- diff --git a/docs/UsersGuide/source/WE2Etests.rst b/docs/UsersGuide/source/WE2Etests.rst index da6a5156ab..19cc5785b1 100644 --- a/docs/UsersGuide/source/WE2Etests.rst +++ b/docs/UsersGuide/source/WE2Etests.rst @@ -3,44 +3,47 @@ ================================ Workflow End-to-End (WE2E) Tests ================================ -The SRW Application workflow contains a set of end-to-end tests that exercise various -configurations of the pre-processing, UFS Weather Model, and the UPP post-processor. The -scripts to run these tests are located in ``ufs-srweather-app/regional_workflow/tests``. -A complete list of the available tests is in ``baselines_list.txt``. This list is extensive; -it is not recommended to run all of the tests, as it is computationally expensive. The tests -that start with ``nco_`` are run in an operational mode and are used more exclusively by NOAA/NCEP -Central Operations (NCO). A subset of tests supported for this release are in the file -``testlist.release_public_v1.txt``. +The SRW Application's experiment generation system contains a set of end-to-end tests that +exercise various configurations of that system as well as those of the pre-processing, +UFS Weather Model, and UPP post-processing codes. The script to run these tests is named +``run_experiments.sh`` and is located in the directory ``ufs-srweather-app/regional_workflow/tests``. +A complete list of the available tests can be found in ``baselines_list.txt`` in that directory. +This list is extensive; it is not recommended to run all of the tests as some are computationally +expensive. A subset of the tests supported for this release of the SRW Application can be found +in the file ``testlist.release_public_v1.txt``. -The full set of the test configuration files is in the ``baseline_configs`` subdirectory. Each -file is named ``config.${config_name}.sh``, where ``${config_name}`` is the name of the corresponding -configuration. These configuration files are variations of the ``config.sh`` file used in -:numref:`Section %s ` and described in :numref:`Section %s `. -Since the purpose of these tests is to ensure that the workflow completes successfully, most of the -WE2E tests use coarse grids to minimize test duration and the use of computational resources. +The base experiment configuration file for each test is located in the ``baseline_configs`` +subdirectory. Each file is named ``config.${expt_name}.sh``, where ``${expt_name}`` is the +name of the corresponding test configuration. These base configuration files are subsets of +the full ``config.sh`` experiment configuration file used in :numref:`Section %s ` +and described in :numref:`Section %s `. For each test that the user wants +to run, the ``run_experiments.sh`` script reads in its base configuration file and generates from +it a full ``config.sh`` file (a copy of which is placed in the experiment directory for the test). Running the Tests ----------------- -The WE2E test script assumes that all of the executables have been built. A script -``ufs-srweather-app/regional_workflow/tests/run_experiments.sh`` is available to run one or more -of the WE2E tests. For each test, the script will generate an experiment directory and launch its -workflow. By default, each workflow will be resubmitted via a cron job until it either completes -successfully (i.e. all tasks are successful) or fails (i.e. at least one task fails). - -If the user does not wish to run all of the tests listed in ``baselines_list.txt``, it is recommended -to make a copy of ``my_expts.txt`` to contain a subset of the tests to be run or use the subset of -tests in ``testlist.release_public_v1.txt``. - -Since these tests run ``./generate_FV3LAM_wflow.sh``, the Python modules must be loaded before the -tests are run. See :numref:`Section %s ` for information on loading the Python -environment on supported platforms. - -Running the test script ``run_experiments.sh`` uses command line arguments shown in +Since ``run_experiments.sh`` calls ``generate_FV3LAM_wflow.sh`` for each test to be run, the +Python modules required for experiment generation must be loaded before ``run_experiments.sh`` +can be called. See :numref:`Section %s ` for information on loading the Python +environment on supported platforms. Note also that ``run_experiments.sh`` assumes that all of +the executables have been built. + +The user specifies the set of test configurations that the ``run_experiments.sh`` script will +run by creating a text file, say ``my_expts.txt``, that contains a list of tests (one per line) +and passing the name of that file to the script. For each test in the file, ``run_experiments.sh`` +will generate an experiment directory and, by default, will continuously (re)launch its workflow +by inserting a new cron job in the user's cron table. This cron job calls the workflow launch script +``launch_FV3LAM_wflow.sh`` located in the experiment directory until the workflow either +completes successfully (i.e. all tasks are successful) or fails (i.e. at least one task fails). +The cron job is then removed from the user's cron table. + + +The script ``run_experiments.sh`` accepts the command line arguments shown in :numref:`Table %s `. .. _WE2ECommandLineArgs: -.. list-table:: Command line arguments for the test script ``run_experiments.sh``. +.. list-table:: Command line arguments for the WE2E testing script ``run_experiments.sh``. :widths: 20 40 40 :header-rows: 1 @@ -48,7 +51,7 @@ Running the test script ``run_experiments.sh`` uses command line arguments shown - Description - Optional * - expts_file - - Name of the file containing the list of experiments to run. If ``expts_file`` is the absolute path + - Name of the file containing the list of tests to run. If ``expts_file`` is the absolute path to a file, it is used as is. If it is a relative path (including just a file name), it is assumed to be given relative to the path from which this script is called. - No @@ -59,30 +62,37 @@ Running the test script ``run_experiments.sh`` uses command line arguments shown - HPC account to use - No * - use_cron_to_relaunch - - Set to ``TRUE`` to use crontab to relaunch the workflow - - Yes. If not set, the value set in ``config_defaults.sh`` will be used, which has the default set to ``FALSE``. + - Flag that specifies whether or not to use a cron job to continuously relaunch the workflow + - Yes. The default value is "TRUE" set in ``run_experiments.sh``. * - cron_relaunch_intvl_mnts - - Interval (in minutes) set in crontab to relaunch the workflow - - Used only if ``use_cron_to_relaunch`` is ``TRUE``. The default value is 03 set in ``config_default.sh``. + - Frequency (in minutes) with which cron will relaunch the workflow + - Used only if ``use_cron_to_relaunch`` is ``TRUE``. The default value is "02" set in ``run_experiments.sh``. -To run the tests from the tests directory on Cheyenne: +For example, to run the tests named ``grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2`` +and ``grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha`` on Cheyenne, first create the file +``my_expts.txt`` containing the following lines: .. code-block:: console - ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account=$ACCOUNT use_cron_to_relaunch=TRUE cron_relaunch_intvl_mnts=05 - -Running this command will automatically insert a command into your crontab to regularly relaunch the workflow -tasks. The tests will be run in the ``${EXPT_BASEDIR}`` directory, with each test run in its own subdirectory -``${EXPT_SUBDIR}``. You can also turn off using cron for all the tests like this: + + grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2 + grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha + +Then, from the ``ufs-srweather-app/regional_workflow/tests`` directory, issue the following command: .. code-block:: console - ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account=$ACCOUNT use_cron_to_relaunch=FALSE + ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account="your_account" -Once ``./run_experiments.sh`` has been executed, with the cron options set, your crontab file will be -automatically modified to run the experiments to completion, and the line inserted into your crontab file -that relaunches the workflow tasks will also be removed when the experiment is complete. To see if an -experiment was successful, look at the end of the ``log.launch_FV3LAM_wflow`` file for: +where ``your_account`` should be replaced by the account to which to charge the core-hours used +by the tests. Running this command will automatically insert an entry into the user's crontab +that regularly (re)launches the workflow. The experiment directories will be created under +``ufs-srweather-app/../expt_dirs``, and the name of each experiment directory will be identical +to the name of the corresponding test. + +To see if a test completed successfully, look at the end of the ``log.launch_FV3LAM_wflow`` file (which +is the log file that ``launch_FV3LAM_wflow.sh`` appends to every time it is called) located in the +experiment directory for that test: .. code-block:: console @@ -96,3 +106,19 @@ experiment was successful, look at the end of the ``log.launch_FV3LAM_wflow`` fi End of output from script "launch_FV3LAM_wflow.sh". ======================================================================== + +You can turn off using cron for all the tests to be run by ``run_experiments.sh`` by instead issuing +the following command: + +.. code-block:: console + + ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account="my_account" use_cron_to_relaunch=FALSE + +In this case, the experiment directories for the tests will be created, but their workflows will +not be (re)launched. For each test, the user will have to go into the experiment directory and +either manually call the ``launch_FV3LAM_wflow.sh`` script or use the rocoto commands described +in :numref:`Chapter %s ` to (re)launch the workflow. Note that if using the rocoto +commands directly, the log file ``log.launch_FV3LAM_wflow`` will not be created; in this case, +the status of the workflow can be checked using the ``rocotostat`` command. + + From 55584f574aa6e6aa8244b0a6ce0ff1931296b8b5 Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Thu, 25 Feb 2021 16:46:18 -0700 Subject: [PATCH 2/6] Further edits to the graphics chapter. --- docs/UsersGuide/source/Graphics.rst | 133 +++++++++++++++------------- 1 file changed, 73 insertions(+), 60 deletions(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index 4654b10cca..5ff76e3f34 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -21,22 +21,58 @@ The Python scripts are located under ``ufs-srweather-app/regional_workflow/ush/P The script ``plot_allvars.py`` plots the output from a single cycle within an experiment, while the script ``plot_allvars_diff.py`` plots the difference between the same cycle from two different experiments (e.g. the experiments may differ in some aspect such as the physics suite used). If -you are plotting the difference, the two experiments must be on the same domain and available for +plotting the difference, the two experiments must be on the same domain and available for the same cycle starting date/time and forecast hours. The Python scripts require a cycle starting date/time in YYYYMMDDHH format, a starting forecast hour, an ending forecast hour, a forecast hour increment, the paths to one or two experiment directories, -and a path to the directory where the Natural Earth shape files are located. +and a path to the directory where the Cartopy Natural Earth shape files are located. +The full set of Cartopy shape files can be downloaded at https://www.naturalearthdata.com/downloads/. +For convenience, the small subset of files required for these Python scripts can be obtained from the +`EMC ftp data repository `_ +or from `AWS cloud storage `_. +In addition, the Cartopy shape files are available on a number of Tier 1 platforms in the following +locations: -The Cartopy shape files can be downloaded at https://www.naturalearthdata.com/downloads/. The medium scale -(1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders -on the map. Cartopy provides the ‘background_img()’ method to add background images in a convenient way. -The default scale (resolution) of background attributes in the Python scripts is 1:50m Natural Earth I -with Shaded Relief and Water, which should be sufficient for most regional applications. +On Cheyenne: -The appropriate environment must be loaded to run the scripts, which require Python 3 with -the scipy, matplotlib, pygrib, and cartopy packages. This Python environment has already -been set up on Level 1 platforms and can be activated as follows: +.. code-block:: console + + /glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth + +On Hera: + +.. code-block:: console + + /scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/NaturalEarth + +On Jet: + +.. code-block:: console + + /lfs4/BMC/wrfruc/FV3-LAM/NaturalEarth + +On Orion: + +.. code-block:: console + + /home/chjeon/tools/NaturalEarth + +On Gaea: + +.. code-block:: console + + /lustre/f2/pdata/esrl/gsd/ufs/NaturalEarth + +The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other +geopolitical borders on the map. Cartopy provides the ‘background_img()’ method to add background +images in a convenient way. The default scale (resolution) of background attributes in the Python +scripts is 1:50m Natural Earth I with Shaded Relief and Water, which should be sufficient for most +regional applications. + +The appropriate environment must be loaded to run the plotting scripts, which require Python 3 with +the scipy, matplotlib, pygrib, and cartopy packages. This Python environment has already been set +up on Tier 1 platforms and can be activated as follows: On Cheyenne: @@ -70,8 +106,8 @@ On Gaea: .. note:: - If you are using the batch submission scripts (see instructions later on), the environments are - set for you and you do not need to set them on the command line prior to running the script. + If using one of the batch submission scripts described below, the user does not need to + manually load an environment because the scripts perform this task. Before generating plots, it is convenient to change location to the directory containing the plotting scripts: @@ -113,43 +149,11 @@ seven command line arguments: #. The top level of the first experiment directory ``EXPTDIR2`` containing the second set of post-processed data. The script will look for the data files in the directory ``EXPTDIR2/CDATE/postprd``. #. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shape files (``*.shp``) in the directory ``CARTOPY_DIR/shapefiles/natural_earth/cultural``. -In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``, - -The Cartopy shape files are available on a number of Tier 1 platforms in the following locations: - -On Cheyenne: - -.. code-block:: console - - /glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth - -On Hera: - -.. code-block:: console - - /scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/NaturalEarth - -On Jet: - -.. code-block:: console - - /lfs4/BMC/wrfruc/FV3-LAM/NaturalEarth - -On Orion: - -.. code-block:: console - - /home/chjeon/tools/NaturalEarth - -On Gaea: - -.. code-block:: console - - /lustre/f2/pdata/esrl/gsd/ufs/NaturalEarth +In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``. If the Python scripts are being used to create plots of multiple forecast lead times and forecast -variables, then they should be submitted through the batch system using either the ``sq_job.sh`` +variables, then they should be submitted to the batch system using either the ``sq_job.sh`` or ``sq_job_diff.sh`` script (for platforms such as Hera, Jet, Orion, and Gaea that use slurm as the job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for platforms such as Cheyenne that use PBS or PBS Pro as the job scheduler). These scripts are located under @@ -169,46 +173,55 @@ On Cheyenne, ``qsub_job.sh`` can be submitted as follows: When using these batch scripts, several environment variables must be set prior to submission. If plotting output from a single cycle, the variables to set are ``HOMErrfs`` and ``EXPTDIR``. -In this case, if the user's login shell is csh/tcsh, these variables are set as follows: +In this case, if the user's login shell is csh/tcsh, these variables can be set as follows: .. code-block:: console setenv HOMErrfs /path-to/ufs-srweather-app/regional_workflow - setenv EXPTDIR /path-to/EXPTDIR + setenv EXPTDIR /path-to/experiment/directory -If the user's login shell is bash, these are set as follows: +If the user's login shell is bash, they can be set as follows: .. code-block:: console export HOMErrfs=/path-to/ufs-srweather-app/regional_workflow - export EXPTDIR=/path-to/EXPTDIR + export EXPTDIR=/path-to/experiment/directory If plotting the difference between the same cycle from two different experiments, the variables to set are ``HOMErrfs``, ``EXPTDIR1``. and ``EXPTDIR2``. In this case, if the user's login shell -is csh/tcsh, these variables are set as follows: +is csh/tcsh, these variables can be set as follows: .. code-block:: console setenv HOMErrfs /path-to/ufs-srweather-app/regional_workflow - setenv EXPTDIR1 /path-to/EXPTDIR1 - setenv EXPTDIR2 /path-to/EXPTDIR2 + setenv EXPTDIR1 /path-to/experiment/directory1 + setenv EXPTDIR2 /path-to/experiment/directory2 -If the user's login shell is bash, these are set as follows: +If the user's login shell is bash, they can be set as follows: .. code-block:: console export HOMErrfs=/path-to/ufs-srweather-app/regional_workflow - export EXPTDIR1=/path-to/EXPTDIR1 - export EXPTDIR2=/path-to/EXPTDIR2 + export EXPTDIR1=/path-to/experiment/directory1 + export EXPTDIR2=/path-to/experiment/directory2 -In addition, the following variables can be modified in the batch scripts depending on your -needs (for example, if you want to plot hourly forecast output, ``FCST_INC`` should be set to 1; -if you just want to plot a subset of your model output, you can set ``FCST_START``, ``FCST_END``, -and ``FCST_INC`` accordingly): +In addition, the variables ``CDATE``, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` in the batch +scripts can be modified depending on the user's needs. By default, ``CDATE`` is set as follows +in the batch scripts: .. code-block:: console export CDATE=${DATE_FIRST_CYCL}${CYCL_HRS} + +This sets ``CDATE`` to the first cycle in the set of cycles that the experiment has run. If the +experiment contains multiple cycles and the user wants to plot output from a cycle other than +the very first one, ``CDATE`` in the batch scripts will have to be set to the specific YYYYMMDDHH +value for that cycle. Also, to plot hourly forecast output, ``FCST_INC`` should be set to 1; to +plot only a subset of the output hours, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` must be +set accordingly, e.g. to generate plots only for output hours after forecast hour 6, use + +.. code-block:: console + export FCST_START=6 export FCST_END=${FCST_LEN_HRS} export FCST_INC=6 From 8e27d824068b95cab249f2c2f77e4efa93d417f8 Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Thu, 25 Feb 2021 17:14:08 -0700 Subject: [PATCH 3/6] For consistency with the RocotoInfo.rst chapter, change "rocoto" to "Rocoto" where appropriate. --- docs/UsersGuide/source/ConfigNewPlatform.rst | 2 +- docs/UsersGuide/source/ConfigParameters.inc | 2 +- docs/UsersGuide/source/ConfigWorkflow.rst | 4 ++-- docs/UsersGuide/source/SRWAppOverview.rst | 4 ++-- docs/UsersGuide/source/WE2Etests.rst | 24 +++++++++----------- 5 files changed, 17 insertions(+), 19 deletions(-) diff --git a/docs/UsersGuide/source/ConfigNewPlatform.rst b/docs/UsersGuide/source/ConfigNewPlatform.rst index c14a87dc61..9accba88ad 100644 --- a/docs/UsersGuide/source/ConfigNewPlatform.rst +++ b/docs/UsersGuide/source/ConfigNewPlatform.rst @@ -194,7 +194,7 @@ Now that the code has been built, you can stage your data as described in :numre Once the data has been staged, setting up your experiment on a platform without a workflow manager is similar to the procedure for other platforms described in earlier chapters. Enter the ``${WORKDIR}/ufs-srweather-app/regional_workflow/ush`` directory and configure the workflow by creating a ``config.sh`` file as described in :numref:`Chapter %s `. There will be a few specific settings that you may need change prior to generating the experiment compared to the instructions for pre-configured platforms: ``MACHINE="MACOS" or MACHINE="LINUX"`` - These are the two ``MACHINE`` settings for generic, non-rocoto-based platforms; you should choose the one most appropriate for your machine. ``MACOS`` has its own setting due to some differences in how command-line utilities function on Darwin-based operating systems. + These are the two ``MACHINE`` settings for generic, non-Rocoto-based platforms; you should choose the one most appropriate for your machine. ``MACOS`` has its own setting due to some differences in how command-line utilities function on Darwin-based operating systems. ``LAYOUT_X=2, LAYOUT_Y=2`` These are the settings that control the MPI decomposition when running the weather model. There are default values, but for your machine it is recommended that you specify your own layout to achieve the correct number of MPI processes for your application. In total, your machine should be able to handle ``LAYOUT_X×LAYOUT_Y+WRTCMP_write_tasks_per_group`` tasks. ``WRTCMP_write_tasks_per_group`` is the number of MPI tasks that will be set aside for writing model output, and it is a setting dependent on the domain you have selected. You can find and edit the value of this variable in the file ``regional_workflow/ush/set_predef_grid_params.sh``. diff --git a/docs/UsersGuide/source/ConfigParameters.inc b/docs/UsersGuide/source/ConfigParameters.inc index ec70ae59b5..c10e916cf5 100644 --- a/docs/UsersGuide/source/ConfigParameters.inc +++ b/docs/UsersGuide/source/ConfigParameters.inc @@ -58,7 +58,7 @@ Here, we set these parameters to null strings. This is so that, for any one of Write-Component (Quilting) Parameters ===================================== ``QUILTING``: (Default: “TRUE”) - Flag that determines whether or not to use the write-component for writing forecast output files to disk. If set to “TRUE”, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where HHH is the 3-hour output forecast hour) containing dynamics and physics fields, respectively, on the write-component grid (the regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model). If ``QUILTING`` is set to "FALSE", then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc`` and contain fields on the native grid. Note that if ``QUILTING`` is set to “FALSE”, then the ``RUN_POST_TN`` (meta)task cannot be run because the Unified Post Processor (UPP) code that this task calls cannot process fields on the native grid. In that case, the ``RUN_POST_TN`` (meta)task will be automatically removed from the rocoto workflow XML. + Flag that determines whether or not to use the write-component for writing forecast output files to disk. If set to “TRUE”, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where HHH is the 3-hour output forecast hour) containing dynamics and physics fields, respectively, on the write-component grid (the regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model). If ``QUILTING`` is set to "FALSE", then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc`` and contain fields on the native grid. Note that if ``QUILTING`` is set to “FALSE”, then the ``RUN_POST_TN`` (meta)task cannot be run because the Unified Post Processor (UPP) code that this task calls cannot process fields on the native grid. In that case, the ``RUN_POST_TN`` (meta)task will be automatically removed from the Rocoto workflow XML. ``PRINT_ESMF``: (Default: “FALSE”) Flag for whether or not to output extra (debugging) information from ESMF routines. Must be "TRUE" or "FALSE". Note that the write-component uses ESMF library routines to interpolate from the native forecast model grid to the user-specified output grid (which is defined in the model configuration file (model_configure) in the forecast run directory). diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index 8985755395..a8bfe8e0a7 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -171,7 +171,7 @@ File Name Parameters Name of the forecast model executable in the executables directory (``EXECDIR``; set during experiment generation). ``WFLOW_XML_FN``: (Default: “FV3LAM_wflow.xml”) - Name of the rocoto workflow XML file that the experiment generation script creates and that defines the workflow for the experiment. + Name of the Rocoto workflow XML file that the experiment generation script creates and that defines the workflow for the experiment. ``GLOBAL_VAR_DEFNS_FN``: (Default: “var_defns.sh”) Name of the file (a shell script) containing the definitions of the primary experiment variables (parameters) defined in this default configuration script and in config.sh as well as secondary experiment variables generated by the experiment generation script. This file is sourced by many scripts (e.g. the J-job scripts corresponding to each workflow task) in order to make all the experiment variables available in those scripts. @@ -183,7 +183,7 @@ File Name Parameters Name of the file (a shell script) containing the definitions of variables associated with the external model from which LBCs are generated. This file is created by the ``GET_EXTRN_LBCS_TN`` task because the values of the variables it contains are not known before this task runs. The file is then sourced by the ``MAKE_ICS_TN`` task. ``WFLOW_LAUNCH_SCRIPT_FN``: (Default: “launch_FV3LAM_wflow.sh") - Name of the script that can be used to (re)launch the experiment's rocoto workflow. + Name of the script that can be used to (re)launch the experiment's Rocoto workflow. ``WFLOW_LAUNCH_LOG_FN``: (Default: “log.launch_FV3LAM_wflow”) Name of the log file that contains the output from successive calls to the workflow launch script (``WFLOW_LAUNCH_SCRIPT_FN``). diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index 32ace21469..4bcb113ffd 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -413,7 +413,7 @@ On Cheyenne: module load ncarenv ncar_pylib /glade/p/ral/jntp/UFS_SRW_app/ncar_pylib/regional_workflow -Load the rocoto module: +Load the ``rocoto`` module: .. code-block:: console @@ -489,7 +489,7 @@ script before running the ``generate_FV3LAM_wflow.sh`` script. As shown in the f file runs the specific J-job scripts in the prescribed order (``regional_workflow/jobs/JREGIONAL_[task name]``) when the ``launch_FV3LAM_wflow.sh`` is submitted. Each J-job task has its own source script named ``exregional_[task name].sh`` in the ``regional_workflow/scripts`` directory. Two database files -``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are generated and updated by the rocoto calls. +``FV3LAM_wflow.db`` and ``FV3LAM_wflow_lock.db`` are generated and updated by the Rocoto calls. There is usually no need for users to modify these files. To relaunch the workflow from scratch, delete these files and then call the launch script (multiple times, as usual). diff --git a/docs/UsersGuide/source/WE2Etests.rst b/docs/UsersGuide/source/WE2Etests.rst index 19cc5785b1..7332bc4b5f 100644 --- a/docs/UsersGuide/source/WE2Etests.rst +++ b/docs/UsersGuide/source/WE2Etests.rst @@ -20,8 +20,6 @@ and described in :numref:`Section %s `. For each test that to run, the ``run_experiments.sh`` script reads in its base configuration file and generates from it a full ``config.sh`` file (a copy of which is placed in the experiment directory for the test). -Running the Tests ------------------ Since ``run_experiments.sh`` calls ``generate_FV3LAM_wflow.sh`` for each test to be run, the Python modules required for experiment generation must be loaded before ``run_experiments.sh`` can be called. See :numref:`Section %s ` for information on loading the Python @@ -29,7 +27,7 @@ environment on supported platforms. Note also that ``run_experiments.sh`` assum the executables have been built. The user specifies the set of test configurations that the ``run_experiments.sh`` script will -run by creating a text file, say ``my_expts.txt``, that contains a list of tests (one per line) +run by creating a text file, say ``expts_list.txt``, that contains a list of tests (one per line) and passing the name of that file to the script. For each test in the file, ``run_experiments.sh`` will generate an experiment directory and, by default, will continuously (re)launch its workflow by inserting a new cron job in the user's cron table. This cron job calls the workflow launch script @@ -63,14 +61,14 @@ The script ``run_experiments.sh`` accepts the command line arguments shown in - No * - use_cron_to_relaunch - Flag that specifies whether or not to use a cron job to continuously relaunch the workflow - - Yes. The default value is "TRUE" set in ``run_experiments.sh``. + - Yes. Default value is ``TRUE`` (set in ``run_experiments.sh``). * - cron_relaunch_intvl_mnts - Frequency (in minutes) with which cron will relaunch the workflow - - Used only if ``use_cron_to_relaunch`` is ``TRUE``. The default value is "02" set in ``run_experiments.sh``. + - Used only if ``use_cron_to_relaunch`` is set to ``TRUE``. Default value is "02" (set in ``run_experiments.sh``). For example, to run the tests named ``grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2`` and ``grid_RRFS_CONUS_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1alpha`` on Cheyenne, first create the file -``my_expts.txt`` containing the following lines: +``expts_list.txt`` containing the following lines: .. code-block:: console @@ -82,9 +80,9 @@ Then, from the ``ufs-srweather-app/regional_workflow/tests`` directory, issue th .. code-block:: console - ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account="your_account" + ./run_experiments.sh expts_file="expts_list.txt" machine=cheyenne account="account_name" -where ``your_account`` should be replaced by the account to which to charge the core-hours used +where ``account_name`` should be replaced by the account to which to charge the core-hours used by the tests. Running this command will automatically insert an entry into the user's crontab that regularly (re)launches the workflow. The experiment directories will be created under ``ufs-srweather-app/../expt_dirs``, and the name of each experiment directory will be identical @@ -107,18 +105,18 @@ experiment directory for that test: ======================================================================== -You can turn off using cron for all the tests to be run by ``run_experiments.sh`` by instead issuing +Use of cron for all tests to be run by ``run_experiments.sh`` can be turned off by instead issuing the following command: .. code-block:: console - ./run_experiments.sh expts_file="my_expts.txt" machine=cheyenne account="my_account" use_cron_to_relaunch=FALSE + ./run_experiments.sh expts_file="expts_list.txt" machine=cheyenne account="account_name" use_cron_to_relaunch=FALSE In this case, the experiment directories for the tests will be created, but their workflows will not be (re)launched. For each test, the user will have to go into the experiment directory and -either manually call the ``launch_FV3LAM_wflow.sh`` script or use the rocoto commands described -in :numref:`Chapter %s ` to (re)launch the workflow. Note that if using the rocoto +either manually call the ``launch_FV3LAM_wflow.sh`` script or use the Rocoto commands described +in :numref:`Chapter %s ` to (re)launch the workflow. Note that if using the Rocoto commands directly, the log file ``log.launch_FV3LAM_wflow`` will not be created; in this case, -the status of the workflow can be checked using the ``rocotostat`` command. +the status of the workflow can be checked using the ``rocotostat`` command (see :numref:`Chapter %s `). From 4adb6b5d9631891d89a73d73ec87e0a0c2b570d6 Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Thu, 25 Feb 2021 17:32:55 -0700 Subject: [PATCH 4/6] Removing accidentally introduced differences (all minor). --- docs/UsersGuide/source/SRWAppOverview.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/UsersGuide/source/SRWAppOverview.rst b/docs/UsersGuide/source/SRWAppOverview.rst index f6aef28d44..137d040c0c 100644 --- a/docs/UsersGuide/source/SRWAppOverview.rst +++ b/docs/UsersGuide/source/SRWAppOverview.rst @@ -146,7 +146,7 @@ executables listed in :numref:`Table %s ` will be located in t +------------------------+---------------------------------------------------------------------------------+ | orog | Generates orography, land mask, and gravity wave drag files from fixed files | +------------------------+---------------------------------------------------------------------------------+ - | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | + | regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist | +------------------------+---------------------------------------------------------------------------------+ | sfc_climo_gen | Creates surface climatology fields from fixed files for use in ``chgres_cube`` | +------------------------+---------------------------------------------------------------------------------+ @@ -395,7 +395,6 @@ environment has already been set up on Level 1 platforms, and can be activated i On Cheyenne: - .. code-block:: console module load ncarenv @@ -408,7 +407,6 @@ Load the Rocoto module: module use -a /glade/p/ral/jntp/UFS_SRW_app/modules module load rocoto - On Hera and Jet: .. code-block:: console From b7b07270bcd71ec9e019062f02178cb0e2e45630 Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Thu, 25 Feb 2021 18:11:16 -0700 Subject: [PATCH 5/6] Remove subsections from rocoto chapter and make them all sections (since previously there was only one section); change "Tier 1" to "Level 1". --- docs/UsersGuide/source/Graphics.rst | 4 ++-- docs/UsersGuide/source/RocotoInfo.rst | 14 ++++++-------- 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index 5ff76e3f34..c60284028b 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -31,7 +31,7 @@ The full set of Cartopy shape files can be downloaded at https://www.naturaleart For convenience, the small subset of files required for these Python scripts can be obtained from the `EMC ftp data repository `_ or from `AWS cloud storage `_. -In addition, the Cartopy shape files are available on a number of Tier 1 platforms in the following +In addition, the Cartopy shape files are available on a number of Level 1 platforms in the following locations: On Cheyenne: @@ -72,7 +72,7 @@ regional applications. The appropriate environment must be loaded to run the plotting scripts, which require Python 3 with the scipy, matplotlib, pygrib, and cartopy packages. This Python environment has already been set -up on Tier 1 platforms and can be activated as follows: +up on Level 1 platforms and can be activated as follows: On Cheyenne: diff --git a/docs/UsersGuide/source/RocotoInfo.rst b/docs/UsersGuide/source/RocotoInfo.rst index 97106b5477..03d13571cb 100644 --- a/docs/UsersGuide/source/RocotoInfo.rst +++ b/docs/UsersGuide/source/RocotoInfo.rst @@ -16,14 +16,12 @@ are filled in. The completed file contains the workflow task names, parameters n and task interdependencies. The generated XML file is then copied to the experiment directory: ``$EXPTDIR/FV3LAM_wflow.xml``. -Helpful Commands -================ There are a number of Rocoto commands available to run and monitor the workflow and can be found in the complete `Rocoto documentation `_. -Descriptions and examples of commonly used commands are discussed in this section. +Descriptions and examples of commonly used commands are discussed below. rocotorun ---------- +========= The ``rocotorun`` command is used to run the workflow by submitting tasks to the batch system. It will automatically resubmit failed tasks and can recover from system outages without user intervention. An example is: @@ -60,7 +58,7 @@ the workflow from scratch, both database files can be deleted, and the workflow or the launch script ``launch_FV3LAM_wflow.sh`` (executed multiple times as described above). rocotostat ----------- +========== ``rocotostat`` is a tool for querying the status of tasks in an active Rocoto workflow. Once the workflow has been started with the ``rocotorun`` command, Rocoto can also check the status of the workflow using the ``rocotostat`` command: @@ -131,7 +129,7 @@ messages. Optional arguments for the ``rocotostat`` command can be found at htt .. _rocotocheck: rocotocheck ------------ +=========== Sometimes, issuing a ``rocotorun`` command will not cause the next task to launch. ``rocotocheck`` is a tool that can be used to query detailed information about a task or cycle in the Rocoto workflow. To determine the cause of a particular task not being submitted, the ``rocotocheck`` command can be used @@ -203,7 +201,7 @@ have been met. If not, the dependencies section in the output of ``rocotocheck` dependency "is NOT satisfied". rocotorewind ------------- +============ ``rocotorewind`` is a tool that attempts to undo the effects of running a task and is commonly used to rerun part of a workflow that has failed. If a task fails to run (the STATE is DEAD), and needs to be restarted, the ``rocotorewind`` command will rerun tasks in the workflow. The command line options are the same as those described in the ``rocotocheck`` @@ -224,7 +222,7 @@ command to rerun the forecast task from ``$EXPTDIR`` is: rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst rocotoboot ----------- +========== ``rocotoboot`` will force a specific task of a cycle in a Rocoto workflow to run. All dependencies and throttle limits are ignored, and it is generally recommended to use ``rocotorewind`` instead. An example of how to use this command to rerun the ``make_ics`` task from ``$EXPTDIR`` is: From 01690b786a93c9161920d11c3c06cdbae6ca4bb1 Mon Sep 17 00:00:00 2001 From: gerard ketefian Date: Fri, 26 Feb 2021 02:34:40 -0700 Subject: [PATCH 6/6] Clarification/rewording in graphics chapter. --- docs/UsersGuide/source/Graphics.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/UsersGuide/source/Graphics.rst b/docs/UsersGuide/source/Graphics.rst index c60284028b..9ee4f0152a 100644 --- a/docs/UsersGuide/source/Graphics.rst +++ b/docs/UsersGuide/source/Graphics.rst @@ -218,7 +218,8 @@ experiment contains multiple cycles and the user wants to plot output from a cyc the very first one, ``CDATE`` in the batch scripts will have to be set to the specific YYYYMMDDHH value for that cycle. Also, to plot hourly forecast output, ``FCST_INC`` should be set to 1; to plot only a subset of the output hours, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` must be -set accordingly, e.g. to generate plots only for output hours after forecast hour 6, use +set accordingly, e.g. to generate plots for every 6th forecast hour starting with forecast hour 6 +and ending with the last forecast hour, use .. code-block:: console