TechDoc changes for new metadata format (target gmtb/develop)

doc/CCPPtechnical/source/AddingNewSchemes.rst

@@ -6,9 +6,9 @@ Tips for Adding a New Scheme
 
 This chapter contains a brief description on how to add a new scheme to the *CCPP-Physics* pool.
 
-* Identify the variables required for the new scheme and check if they are already available for use in the CCPP by checking the metadata tables in ``GFS_typedefs.F90`` or by perusing file ``ccpp-framework/doc/DevelopersGuide/CCPP_VARIABLES_{FV3,SCM}.pdf`` generated by ``ccpp_prebuild.py``.
+* Identify the variables required for the new scheme and check if they are already available for use in the CCPP by checking the metadata information in ``GFS_typedefs.meta`` or by perusing file ``ccpp-framework/doc/DevelopersGuide/CCPP_VARIABLES_{FV3,SCM}.pdf`` generated by ``ccpp_prebuild.py``.
 
-    * If the variables are already available, they can be invoked in the scheme’s metadata table and one can skip the rest of this subsection. If the variable required is not available, consider if it can be calculated from the existing variables in the CCPP. If so, an interstitial scheme (such as ``scheme_pre``; see more in :numref:`Chapter %s <CompliantPhysParams>`) can be created to calculate the variable. However, the variable must be defined but not initialized in the host model as the memory for this variable must be allocated on the host model side.  Instructions for how to add variables to the host model side is described in :numref:`Chapter %s <Host-side Coding>`.
+    * If the variables are already available, they can be invoked in the scheme’s metadata file and one can skip the rest of this subsection. If the variable required is not available, consider if it can be calculated from the existing variables in the CCPP. If so, an interstitial scheme (such as ``scheme_pre``; see more in :numref:`Chapter %s <CompliantPhysParams>`) can be created to calculate the variable. However, the variable must be defined but not initialized in the host model as the memory for this variable must be allocated on the host model side.  Instructions for how to add variables to the host model side is described in :numref:`Chapter %s <Host-side Coding>`.
 
     * It is important to note that not all data types are persistent in memory. Most variables in the interstitial data type are reset (to zero or other initial values) at the beginning of a physics group and do not persist from one set to another or from one group to another. The diagnostic data type is periodically reset because it is used to accumulate variables for given time intervals.  However, there is a small subset of interstitial variables that are set at creation time and are not reset; these are typically dimensions used in other interstitial variables. 
 
@@ -65,7 +65,7 @@ This chapter contains a brief description on how to add a new scheme to the *CCP
     * Check *prebuild* script for success/failure and associated messages.
     * Compile code in DEBUG mode, run through debugger if necessary (gdb, Allinea DDT, totalview, ...).  See :numref:`Chapter %s <BuildingRunningHostModels>` for information on debugging.
     * Use memory check utilities such as valgrind.
-    * Double-check the metadata table in your scheme to make sure that the standard names correspond to the correct local variables.
+    * Double-check the metadata file associated with your scheme to make sure that the standard names correspond to the correct local variables.
 
 * Done. Note that no further modifications of the build system are required, since the *CCPP-Framework* will autogenerate the necessary makefiles that allow the host model to compile the scheme.
 

doc/CCPPtechnical/source/AutoGenPhysCaps.rst

@@ -23,10 +23,11 @@ Dynamic Build *Caps*
 ====================
 
 With the dynamic build, the CCPP-Framework and physics are dynamically linked to the executable
-to allow maximum runtime flexibility. A *cap* is generated using the metadata requested in 
-each scheme argument table and the variables provided in the host model metadata tables (see 
-left side of :numref:`Figure %s <ccpp_dynamic_build>`). The CCPP *prebuild* step for the dynamic
-build performs the following tasks:
+to allow maximum runtime flexibility. A *cap* is generated using the metadata associated with 
+the arguments of each scheme and the metadata associated with the variables provided by the 
+host model (see left side of :numref:`Figure %s <ccpp_dynamic_build>`). These metadata variables
+are described in the ``.meta`` files associated with the host model and the schemes.
+The CCPP *prebuild* step for the dynamic build performs the following tasks:
 
 * Checks requested vs provided variables by ``standard_name``.
 * Checks units, rank, type.
@@ -88,12 +89,12 @@ The variables added to ``*_fast_physics.inc`` do not use ``GFS_typedefs.F90`` or
 
 For each *cap*, ``ccpp_prebuild.py`` generates “use” statements based on the host model template. 
 Only the public *caps* (``init``, ``run`` and ``finalize``) are exposed (see code example below). 
-Each *cap* consists of a module containing three functions. For example,  ``scheme_pre_cap.F90`` 
+Each *cap* consists of a module containing three functions. For example, ``scheme_pre_cap.F90`` 
 would contain the functions ``scheme_pre_init_cap``, ``scheme_pre_run_cap`` and ``scheme_pre_finalize_cap``, which perform the functions below.
 
 * Declare data types ``cptr``, ``cdims`` and ``ckind``.
 * Create a pointer to the Fortran data type ``cdata``.
-* Call ``ccpp_field_get`` for each variable in metadata table and pulls data from the ``cdata`` structure. 
+* Call ``ccpp_field_get`` for each variable in the metadata file for the scheme and pull data from the ``cdata`` structure. 
 
 The index defined in each call speeds up memory access by avoiding a binary search, 
 since variables are no longer searched by name; the order of the data in ``cdata`` are known.
@@ -193,12 +194,13 @@ For example, the autogenerated scheme *cap* for ``rrtmg_lw_pre_cap.F90`` is show
     end function rrtmg_lw_pre_finalize_cap
    end module rrtmg_lw_pre_cap
 
-*Listing 5.1: Condensed version of the autogenerated scheme cap* ``rrtmg_lw_pre_cap.F90`` *for the dynamic build.  Note the calls to* ``ccpp_field_get`` *for each variable.*
+*Listing 5.1: Condensed version of the autogenerated scheme cap* ``rrtmg_lw_pre_cap.F90`` *for the dynamic build. Note the calls to* ``ccpp_field_get`` *for each variable.*
 
-The fields accessed from ``cdata`` are determined by the metadata in the scheme argument table(s). In this example, 
+The fields accessed from ``cdata`` are determined by the metadata for the arguments of the scheme. In this example, 
 ``rrtmg_lw_pre_init`` and ``rrtmg_lw_pre_finalize`` are empty subroutines, i.e. they have no arguments passed in or out, 
-no metadata table, and no calls to ``ccpp_field_get``. However, ``rrtmg_lw_pre_run`` has a metadata table, so ``ccpp_field_get`` 
-is called for each variable in the table and the value put into the call to ``rrtmg_lw_pre_run``.
+no metadata section in the ``.meta`` file, and no calls to ``ccpp_field_get``. However, ``rrtmg_lw_pre_run`` 
+has a metadata section in file ``rrtmg_lw_pre.meta``, so ``ccpp_field_get`` 
+is called for each variable described in the metadata section and the value put into the call to ``rrtmg_lw_pre_run``.
 
 Static Build Caps
 =================

doc/CCPPtechnical/source/CCPPPreBuild.rst

@@ -16,7 +16,7 @@ incorporated in that model’s build system. In the case of NEMSfv3gfs, ``ccpp_p
 or automatically as a step in the build system.
 
 The :term:`CCPP` *prebuild* script automates several tasks based on the information collected from the metadata
-tables on the host model side and from the individual physics schemes (see :numref:`Figure %s <ccpp_prebuild>`):
+on the host model side and from the individual physics schemes (``.meta`` files; see :numref:`Figure %s <ccpp_prebuild>`):
 
  * Compiles a list of variables required to run all schemes in the :term:`CCPP-Physics` pool.
 
@@ -123,7 +123,7 @@ Running ccpp_prebuild.py
 
 Once the configuration in ``ccpp_prebuild_config.py`` is complete, the ``ccpp_prebuild.py`` script can be run from the top level directory. For the SCM, this script must be run (in dynamic build mode only) to reconcile data provided by the SCM with data required by the physics schemes before compilation and to generate physics caps and makefile segments. For the :term:`UFS` Atmosphere host model, the ``ccpp_prebuild.py`` script is called automatically by the NEMSfv3gfs build system when the :term:`CCPP` build is requested (by running the :term:`CCPP` regression tests or by passing the option CCPP=Y and others to the ``compile.sh`` script; see the compile commands defined in the :term:`CCPP` regression test configurations for further details). 
 
-For developers adding a CCPP-compliant physics scheme, running ``ccpp_prebuild.py`` periodically is recommended to check that the metadata in the argument tables match between the host model and the physics scheme. For the :term:`UFS` Atmosphere, running ``ccpp_prebuild.py`` manually is identical to running it for the SCM (since the relative paths to their respective ``ccpp_prebuild_config.py`` files are identical), except it may be necessary to add the ``--static`` and ``--suites`` command-line arguments for the static option.
+For developers adding a CCPP-compliant physics scheme, running ``ccpp_prebuild.py`` periodically is recommended to check that the metadata provided with the physics schemes matches what the host model provided. For the :term:`UFS` Atmosphere, running ``ccpp_prebuild.py`` manually is identical to running it for the SCM (since the relative paths to their respective ``ccpp_prebuild_config.py`` files are identical), except it may be necessary to add the ``--static`` and ``--suites`` command-line arguments for the static option.
 
 As alluded to above, the ``ccpp_prebuild.py`` script has six command line options, with the path to a host-model specific configuration file (``--config``) being the only necessary input option:
 
@@ -164,7 +164,7 @@ To remove all files created by ``ccpp_prebuild.py``, for example as part of a ho
 Troubleshooting
 =============================
 
-If invoking the ``ccpp_prebuild.py`` script fails, some message other than the success message will be written to the terminal output. Specifically, the terminal output will include informational logging messages generated from the script, any error messages written to the python logging utility, and a Python traceback that pinpoints the line within the script where the error caused failure. Some common errors (minus the typical logging output and traceback output) and solutions are described below, with non-bold font used to denote aspects of the message that will differ depending on the problem encountered. This is not an exhaustive list of possible errors, however. For example, in this version of the code, there is no cross-checking that the information provided in the metadata tables corresponds to the actual Fortran code, so even though ``ccpp_prebuild.py`` may complete successfully, there may be related compilation errors later in the build process. For further help with an undescribed error, please contact gmtb-help@ucar.edu. 
+If invoking the ``ccpp_prebuild.py`` script fails, some message other than the success message will be written to the terminal output. Specifically, the terminal output will include informational logging messages generated from the script, any error messages written to the python logging utility, and a Python traceback that pinpoints the line within the script where the error caused failure. Some common errors (minus the typical logging output and traceback output) and solutions are described below, with non-bold font used to denote aspects of the message that will differ depending on the problem encountered. This is not an exhaustive list of possible errors, however. For example, in this version of the code, there is no cross-checking that the metadata information provided corresponds to the actual Fortran code, so even though ``ccpp_prebuild.py`` may complete successfully, there may be related compilation errors later in the build process. For further help with an undescribed error, please contact gmtb-help@ucar.edu. 
 
 
  #. ``ERROR: Configuration file`` erroneous/path/to/config/file ``not found``
@@ -186,11 +186,9 @@ If invoking the ``ccpp_prebuild.py`` script fails, some message other than the s
  #. ``INFO: Parsing metadata tables for variables provided by host model`` …
 
     ``IOError: [Errno 2] No such file or directory``: 'erroneous_file.f90'
-      * Check that the paths specified in the ``VARIABLE_DEFINITION_FILES`` of the supplied configuration file are valid and contain CCPP-compliant host model variable metadata tables.
- #. ``Exception: Encountered invalid line`` "some fortran" ``in argument table`` variable_metadata_table_name
-      * This is likely the result of not ending a variable metadata table with a line containing only ‘!!’. Check that the formatting of the offending variable metadata table is correct.
+      * Check that the paths specified in the ``VARIABLE_DEFINITION_FILES`` of the supplied configuration file are valid and contain CCPP-compliant host model snippets for insertion of metadata information. (see :ref:`example <SnippetMetadata>`)
  #. ``Exception: Error parsing variable entry`` "erroneous variable metadata table entry data" ``in argument table`` variable_metadata_table_name
-      * Check that the formatting of the metadata entry described in the error message is OK. The number of metadata columns must match the table header and each entry’s columns must be separated by a ‘|’ character (be sure that the ‘!’ character was not used accidentally).
+      * Check that the formatting of the metadata entry described in the error message is OK. 
  #. ``Exception: New entry for variable`` var_name ``in argument table`` variable_metadata_table_name ``is incompatible with existing entry``:
      | ``Existing: Contents of <mkcap.Var object at 0x10299a290> (* = mandatory for compatibility)``:
      |  ``standard_name`` = var_name *
@@ -248,17 +246,17 @@ If invoking the ``ccpp_prebuild.py`` script fails, some message other than the s
      |  ``target``        = None
      |  ``container``     = MODULE_X TYPE_Y
 
-     * This error is associated with physics scheme variable metadata entries that have the same standard name with different mandatory properties (either units, type, rank, or kind currently -- those attributes denoted with a ``*``). This error is distinguished from the error described in 9 above, because the error message mentions “in argument table of subroutine” instead of just “in argument table”.
+     * This error is associated with physics scheme variable metadata entries that have the same standard name with different mandatory properties (either units, type, rank, or kind currently -- those attributes denoted with a ``*``). This error is distinguished from the error described in 8 above, because the error message mentions “in argument table of subroutine” instead of just “in argument table”.
  #. ``ERROR: Check that all subroutines in module`` module_name ``have the same root name``:
      ``i.e. scheme_A_init, scheme_A_run, scheme_A_finalize``
      ``Here is a list of the subroutine names for scheme`` scheme_name: scheme_name_finalize, scheme_name_run
      * All schemes must have ``*_init``, ``*_run``, ``*_finalize`` subroutines contained within its entry/exit point module.
  #. ``ERROR: Variable`` X ``requested by MODULE_``\Y ``SCHEME_``\Z ``SUBROUTINE_``\A ``not provided by the model``
      ``Exception: Call to compare_metadata failed.``
 
-     * A variable requested by one or more physics schemes is not being provided by the host model. If the variable exists in the host model but is not being made available for the :term:`CCPP`, an entry must be added to one of the host model variable metadata tables.
+     * A variable requested by one or more physics schemes is not being provided by the host model. If the variable exists in the host model but is not being made available for the :term:`CCPP`, an entry must be added to one of the host model variable metadata sections. 
  #. ``ERROR:   error, variable`` X ``requested by MODULE_``\Y ``SCHEME_``\Z ``SUBROUTINE_``\A ``cannot be identified unambiguously. Multiple definitions in MODULE_``\Y ``TYPE_``\B
-      * A variable is defined in the host model variable metadata tables more than once (with the same standard name). Remove the offending entry or provide a different standard name for one of the duplicates.
+      * A variable is defined in the host model variable metadata more than once (with the same standard name). Remove the offending entry or provide a different standard name for one of the duplicates.
  #. ``ERROR:   incompatible entries in metadata for variable`` var_name:
      | ``provided:  Contents of <mkcap.Var object at 0x104883210> (* = mandatory for compatibility)``:
      |  ``standard_name`` = var_name *
@@ -287,6 +285,6 @@ If invoking the ``ccpp_prebuild.py`` script fails, some message other than the s
  #. ``Exception: Call to compare_metadata failed``.
       * This error indicates a mismatch between the attributes of a variable provided by the host model and what is requested by the physics. Specifically, the units, type, rank, or kind don’t match for a given variable standard name. Double-check that the attributes for the provided and requested mismatched variable are accurate. If after checking the attributes are indeed mismatched, reconcile as appropriate (by adopting the correct variable attributes either on the host or physics side).
 
-Note: One error that the ``ccpp_prebuild.py`` script will not catch is if a physics scheme lists a variable in its actual (Fortran) argument list without a corresponding entry in the subroutine’s variable metadata table. This will lead to a compilation error when the autogenerated scheme cap is compiled:
+Note: One error that the ``ccpp_prebuild.py`` script will not catch is if a physics scheme lists a variable in its actual (Fortran) argument list without a corresponding entry in the subroutine’s variable metadata. This will lead to a compilation error when the autogenerated scheme cap is compiled:
 
 ``Error: Missing actual argument for argument 'X' at (1)``