Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/UsersGuide/source/ConfigNewPlatform.rst
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ Now that the code has been built, you can stage your data as described in :numre
Once the data has been staged, setting up your experiment on a platform without a workflow manager is similar to the procedure for other platforms described in earlier chapters. Enter the ``${WORKDIR}/ufs-srweather-app/regional_workflow/ush`` directory and configure the workflow by creating a ``config.sh`` file as described in :numref:`Chapter %s <ConfigWorkflow>`. There will be a few specific settings that you may need change prior to generating the experiment compared to the instructions for pre-configured platforms:

``MACHINE="MACOS" or MACHINE="LINUX"``
These are the two ``MACHINE`` settings for generic, non-rocoto-based platforms; you should choose the one most appropriate for your machine. ``MACOS`` has its own setting due to some differences in how command-line utilities function on Darwin-based operating systems.
These are the two ``MACHINE`` settings for generic, non-Rocoto-based platforms; you should choose the one most appropriate for your machine. ``MACOS`` has its own setting due to some differences in how command-line utilities function on Darwin-based operating systems.

``LAYOUT_X=2, LAYOUT_Y=2``
These are the settings that control the MPI decomposition when running the weather model. There are default values, but for your machine it is recommended that you specify your own layout to achieve the correct number of MPI processes for your application. In total, your machine should be able to handle ``LAYOUT_X×LAYOUT_Y+WRTCMP_write_tasks_per_group`` tasks. ``WRTCMP_write_tasks_per_group`` is the number of MPI tasks that will be set aside for writing model output, and it is a setting dependent on the domain you have selected. You can find and edit the value of this variable in the file ``regional_workflow/ush/set_predef_grid_params.sh``.
Expand Down
2 changes: 1 addition & 1 deletion docs/UsersGuide/source/ConfigParameters.inc
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Here, we set these parameters to null strings. This is so that, for any one of
Write-Component (Quilting) Parameters
=====================================
``QUILTING``: (Default: “TRUE”)
Flag that determines whether or not to use the write-component for writing forecast output files to disk. If set to “TRUE”, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where HHH is the 3-hour output forecast hour) containing dynamics and physics fields, respectively, on the write-component grid (the regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model). If ``QUILTING`` is set to "FALSE", then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc`` and contain fields on the native grid. Note that if ``QUILTING`` is set to “FALSE”, then the ``RUN_POST_TN`` (meta)task cannot be run because the Unified Post Processor (UPP) code that this task calls cannot process fields on the native grid. In that case, the ``RUN_POST_TN`` (meta)task will be automatically removed from the rocoto workflow XML.
Flag that determines whether or not to use the write-component for writing forecast output files to disk. If set to “TRUE”, the forecast model will output files named ``dynf$HHH.nc`` and ``phyf$HHH.nc`` (where HHH is the 3-hour output forecast hour) containing dynamics and physics fields, respectively, on the write-component grid (the regridding from the native FV3-LAM grid to the write-component grid is done by the forecast model). If ``QUILTING`` is set to "FALSE", then the output file names are ``fv3_history.nc`` and ``fv3_history2d.nc`` and contain fields on the native grid. Note that if ``QUILTING`` is set to “FALSE”, then the ``RUN_POST_TN`` (meta)task cannot be run because the Unified Post Processor (UPP) code that this task calls cannot process fields on the native grid. In that case, the ``RUN_POST_TN`` (meta)task will be automatically removed from the Rocoto workflow XML.

``PRINT_ESMF``: (Default: “FALSE”)
Flag for whether or not to output extra (debugging) information from ESMF routines. Must be "TRUE" or "FALSE". Note that the write-component uses ESMF library routines to interpolate from the native forecast model grid to the user-specified output grid (which is defined in the model configuration file (model_configure) in the forecast run directory).
Expand Down
4 changes: 2 additions & 2 deletions docs/UsersGuide/source/ConfigWorkflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ File Name Parameters
Name of the forecast model executable in the executables directory (``EXECDIR``; set during experiment generation).

``WFLOW_XML_FN``: (Default: “FV3LAM_wflow.xml”)
Name of the rocoto workflow XML file that the experiment generation script creates and that defines the workflow for the experiment.
Name of the Rocoto workflow XML file that the experiment generation script creates and that defines the workflow for the experiment.

``GLOBAL_VAR_DEFNS_FN``: (Default: “var_defns.sh”)
Name of the file (a shell script) containing the definitions of the primary experiment variables (parameters) defined in this default configuration script and in config.sh as well as secondary experiment variables generated by the experiment generation script. This file is sourced by many scripts (e.g. the J-job scripts corresponding to each workflow task) in order to make all the experiment variables available in those scripts.
Expand All @@ -183,7 +183,7 @@ File Name Parameters
Name of the file (a shell script) containing the definitions of variables associated with the external model from which LBCs are generated. This file is created by the ``GET_EXTRN_LBCS_TN`` task because the values of the variables it contains are not known before this task runs. The file is then sourced by the ``MAKE_ICS_TN`` task.

``WFLOW_LAUNCH_SCRIPT_FN``: (Default: “launch_FV3LAM_wflow.sh")
Name of the script that can be used to (re)launch the experiment's rocoto workflow.
Name of the script that can be used to (re)launch the experiment's Rocoto workflow.

``WFLOW_LAUNCH_LOG_FN``: (Default: “log.launch_FV3LAM_wflow”)
Name of the log file that contains the output from successive calls to the workflow launch script (``WFLOW_LAUNCH_SCRIPT_FN``).
Expand Down
133 changes: 74 additions & 59 deletions docs/UsersGuide/source/Graphics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,22 +21,58 @@ The Python scripts are located under ``ufs-srweather-app/regional_workflow/ush/P
The script ``plot_allvars.py`` plots the output from a single cycle within an experiment, while
the script ``plot_allvars_diff.py`` plots the difference between the same cycle from two different
experiments (e.g. the experiments may differ in some aspect such as the physics suite used). If
you are plotting the difference, the two experiments must be on the same domain and available for
plotting the difference, the two experiments must be on the same domain and available for
the same cycle starting date/time and forecast hours.

The Python scripts require a cycle starting date/time in YYYYMMDDHH format, a starting forecast
hour, an ending forecast hour, a forecast hour increment, the paths to one or two experiment directories,
and a path to the directory where the Natural Earth shape files are located.
and a path to the directory where the Cartopy Natural Earth shape files are located.
The full set of Cartopy shape files can be downloaded at https://www.naturalearthdata.com/downloads/.
For convenience, the small subset of files required for these Python scripts can be obtained from the
`EMC ftp data repository <https://ftp.emc.ncep.noaa.gov/EIB/UFS/SRW/v1p0/natural_earth/natural_earth_ufs-srw-release-v1.0.0.tar.gz>`_
or from `AWS cloud storage <https://ufs-data.s3.amazonaws.com/public_release/ufs-srweather-app-v1.0.0/natural_earth/natural_earth_ufs-srw-release-v1.0.0.tar.gz>`_.
In addition, the Cartopy shape files are available on a number of Level 1 platforms in the following
locations:

The Cartopy shape files can be downloaded at https://www.naturalearthdata.com/downloads/. The medium scale
(1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders
on the map. Cartopy provides the ‘background_img()’ method to add background images in a convenient way.
The default scale (resolution) of background attributes in the Python scripts is 1:50m Natural Earth I
with Shaded Relief and Water, which should be sufficient for most regional applications.
On Cheyenne:

The appropriate environment must be loaded to run the scripts, which require Python 3 with
the scipy, matplotlib, pygrib, and cartopy packages. This Python environment has already
been set up on Level 1 platforms and can be activated as follows:
.. code-block:: console

/glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth

On Hera:

.. code-block:: console

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/NaturalEarth

On Jet:

.. code-block:: console

/lfs4/BMC/wrfruc/FV3-LAM/NaturalEarth

On Orion:

.. code-block:: console

/home/chjeon/tools/NaturalEarth

On Gaea:

.. code-block:: console

/lustre/f2/pdata/esrl/gsd/ufs/NaturalEarth

The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other
geopolitical borders on the map. Cartopy provides the ‘background_img()’ method to add background
images in a convenient way. The default scale (resolution) of background attributes in the Python
scripts is 1:50m Natural Earth I with Shaded Relief and Water, which should be sufficient for most
regional applications.

The appropriate environment must be loaded to run the plotting scripts, which require Python 3 with
the scipy, matplotlib, pygrib, and cartopy packages. This Python environment has already been set
up on Level 1 platforms and can be activated as follows:

On Cheyenne:

Expand Down Expand Up @@ -70,8 +106,8 @@ On Gaea:

.. note::

If you are using the batch submission scripts (see instructions later on), the environments are
set for you and you do not need to set them on the command line prior to running the script.
If using one of the batch submission scripts described below, the user does not need to
manually load an environment because the scripts perform this task.

Before generating plots, it is convenient to change location to the directory containing the plotting
scripts:
Expand Down Expand Up @@ -113,42 +149,11 @@ seven command line arguments:
#. The top level of the first experiment directory ``EXPTDIR2`` containing the second set of post-processed data. The script will look for the data files in the directory ``EXPTDIR2/CDATE/postprd``.
#. The base directory ``CARTOPY_DIR`` of the cartopy shapefiles. The script will look for the shape files (``*.shp``) in the directory ``CARTOPY_DIR/shapefiles/natural_earth/cultural``.


The Cartopy shape files are available on a number of Tier 1 platforms in the following locations:

On Cheyenne:

.. code-block:: console

/glade/p/ral/jntp/UFS_SRW_app/tools/NaturalEarth

On Hera:

.. code-block:: console

/scratch2/NCEPDEV/fv3-cam/Chan-hoo.Jeon/tools/NaturalEarth

On Jet:

.. code-block:: console

/lfs4/BMC/wrfruc/FV3-LAM/NaturalEarth

On Orion:

.. code-block:: console

/home/chjeon/tools/NaturalEarth

On Gaea:

.. code-block:: console

/lustre/f2/pdata/esrl/gsd/ufs/NaturalEarth
In this case, the output png files will be located in the directory ``EXPTDIR1/CDATE/postprd``.


If the Python scripts are being used to create plots of multiple forecast lead times and forecast
variables, then they should be submitted through the batch system using either the ``sq_job.sh``
variables, then they should be submitted to the batch system using either the ``sq_job.sh``
or ``sq_job_diff.sh`` script (for platforms such as Hera, Jet, Orion, and Gaea that use slurm as
the job scheduler) or the ``qsub_job.sh`` or ``qsub_job_diff.sh`` script (for platforms such as
Cheyenne that use PBS or PBS Pro as the job scheduler). These scripts are located under
Expand All @@ -168,46 +173,56 @@ On Cheyenne, ``qsub_job.sh`` can be submitted as follows:

When using these batch scripts, several environment variables must be set prior to submission.
If plotting output from a single cycle, the variables to set are ``HOMErrfs`` and ``EXPTDIR``.
In this case, if the user's login shell is csh/tcsh, these variables are set as follows:
In this case, if the user's login shell is csh/tcsh, these variables can be set as follows:

.. code-block:: console

setenv HOMErrfs /path-to/ufs-srweather-app/regional_workflow
setenv EXPTDIR /path-to/EXPTDIR
setenv EXPTDIR /path-to/experiment/directory

If the user's login shell is bash, these are set as follows:
If the user's login shell is bash, they can be set as follows:

.. code-block:: console

export HOMErrfs=/path-to/ufs-srweather-app/regional_workflow
export EXPTDIR=/path-to/EXPTDIR
export EXPTDIR=/path-to/experiment/directory

If plotting the difference between the same cycle from two different experiments, the variables
to set are ``HOMErrfs``, ``EXPTDIR1``. and ``EXPTDIR2``. In this case, if the user's login shell
is csh/tcsh, these variables are set as follows:
is csh/tcsh, these variables can be set as follows:

.. code-block:: console

setenv HOMErrfs /path-to/ufs-srweather-app/regional_workflow
setenv EXPTDIR1 /path-to/EXPTDIR1
setenv EXPTDIR2 /path-to/EXPTDIR2
setenv EXPTDIR1 /path-to/experiment/directory1
setenv EXPTDIR2 /path-to/experiment/directory2

If the user's login shell is bash, these are set as follows:
If the user's login shell is bash, they can be set as follows:

.. code-block:: console

export HOMErrfs=/path-to/ufs-srweather-app/regional_workflow
export EXPTDIR1=/path-to/EXPTDIR1
export EXPTDIR2=/path-to/EXPTDIR2
export EXPTDIR1=/path-to/experiment/directory1
export EXPTDIR2=/path-to/experiment/directory2

In addition, the following variables can be modified in the batch scripts depending on your
needs (for example, if you want to plot hourly forecast output, ``FCST_INC`` should be set to 1;
if you just want to plot a subset of your model output, you can set ``FCST_START``, ``FCST_END``,
and ``FCST_INC`` accordingly):
In addition, the variables ``CDATE``, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` in the batch
scripts can be modified depending on the user's needs. By default, ``CDATE`` is set as follows
in the batch scripts:

.. code-block:: console

export CDATE=${DATE_FIRST_CYCL}${CYCL_HRS}

This sets ``CDATE`` to the first cycle in the set of cycles that the experiment has run. If the
experiment contains multiple cycles and the user wants to plot output from a cycle other than
the very first one, ``CDATE`` in the batch scripts will have to be set to the specific YYYYMMDDHH
value for that cycle. Also, to plot hourly forecast output, ``FCST_INC`` should be set to 1; to
plot only a subset of the output hours, ``FCST_START``, ``FCST_END``, and ``FCST_INC`` must be
set accordingly, e.g. to generate plots for every 6th forecast hour starting with forecast hour 6
and ending with the last forecast hour, use

.. code-block:: console

export FCST_START=6
export FCST_END=${FCST_LEN_HRS}
export FCST_INC=6
Expand Down
6 changes: 4 additions & 2 deletions docs/UsersGuide/source/InputOutputFiles.rst
Original file line number Diff line number Diff line change
Expand Up @@ -325,8 +325,10 @@ Default Initial and Lateral Boundary Conditions
-----------------------------------------------
The default initial and lateral boundary condition files are set to be a severe weather case
from 20190615 at 00 UTC. FV3GFS grib2 files are the default model and file format. A tar file
containing the model data for this case is available on the FTP data repository at:
https://ftp.emc.ncep.noaa.gov/EIB/UFS/SRW/v1p0/simple_test_case/.
(``gst_model_data.tar.gz``) containing the model data for this case is available on EMC's FTP
data repository at https://ftp.emc.ncep.noaa.gov/EIB/UFS/SRW/v1p0/simple_test_case/. It is
also available on Amazon Web Services (AWS) at
https://ufs-data.s3.amazonaws.com/public_release/ufs-srweather-app-v1.0.0/ic/gst_model_data.tar.gz.
Comment thread
gsketefian marked this conversation as resolved.

Running the App for Different Dates
-----------------------------------
Expand Down
14 changes: 6 additions & 8 deletions docs/UsersGuide/source/RocotoInfo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,12 @@ are filled in. The completed file contains the workflow task names, parameters n
and task interdependencies. The generated XML file is then copied to the experiment directory:
``$EXPTDIR/FV3LAM_wflow.xml``.

Helpful Commands
================
There are a number of Rocoto commands available to run and monitor the workflow and can be found in the
complete `Rocoto documentation <https://github.com/christopherwharrop/rocoto/wiki/documentation>`_.
Descriptions and examples of commonly used commands are discussed in this section.
Descriptions and examples of commonly used commands are discussed below.

rocotorun
---------
=========
The ``rocotorun`` command is used to run the workflow by submitting tasks to the batch system. It will
automatically resubmit failed tasks and can recover from system outages without user intervention.
An example is:
Expand Down Expand Up @@ -60,7 +58,7 @@ the workflow from scratch, both database files can be deleted, and the workflow
or the launch script ``launch_FV3LAM_wflow.sh`` (executed multiple times as described above).

rocotostat
----------
==========
``rocotostat`` is a tool for querying the status of tasks in an active Rocoto workflow. Once the
workflow has been started with the ``rocotorun`` command, Rocoto can also check the status of the
workflow using the ``rocotostat`` command:
Expand Down Expand Up @@ -131,7 +129,7 @@ messages. Optional arguments for the ``rocotostat`` command can be found at htt
.. _rocotocheck:

rocotocheck
-----------
===========
Sometimes, issuing a ``rocotorun`` command will not cause the next task to launch. ``rocotocheck`` is a
tool that can be used to query detailed information about a task or cycle in the Rocoto workflow. To
determine the cause of a particular task not being submitted, the ``rocotocheck`` command can be used
Expand Down Expand Up @@ -203,7 +201,7 @@ have been met. If not, the dependencies section in the output of ``rocotocheck`
dependency "is NOT satisfied".

rocotorewind
------------
============
``rocotorewind`` is a tool that attempts to undo the effects of running a task and is commonly used to rerun part
of a workflow that has failed. If a task fails to run (the STATE is DEAD), and needs to be restarted, the ``rocotorewind``
command will rerun tasks in the workflow. The command line options are the same as those described in the ``rocotocheck``
Expand All @@ -224,7 +222,7 @@ command to rerun the forecast task from ``$EXPTDIR`` is:
rocotorewind -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 -c 201907010000 -t run_fcst

rocotoboot
----------
==========
``rocotoboot`` will force a specific task of a cycle in a Rocoto workflow to run. All dependencies and throttle
limits are ignored, and it is generally recommended to use ``rocotorewind`` instead. An example of how to
use this command to rerun the ``make_ics`` task from ``$EXPTDIR`` is:
Expand Down
2 changes: 1 addition & 1 deletion docs/UsersGuide/source/SRWAppOverview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ executables listed in :numref:`Table %s <exec_description>` will be located in t
+------------------------+---------------------------------------------------------------------------------+
| orog | Generates orography, land mask, and gravity wave drag files from fixed files |
+------------------------+---------------------------------------------------------------------------------+
| regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist |
| regional_esg_grid | Generates an ESG regional grid based on a user-defined namelist |
+------------------------+---------------------------------------------------------------------------------+
| sfc_climo_gen | Creates surface climatology fields from fixed files for use in ``chgres_cube`` |
+------------------------+---------------------------------------------------------------------------------+
Expand Down
Loading