Update the CMORizer command

doc/sphinx/source/community/code_documentation.rst

@@ -260,6 +260,11 @@ However, a pull request where the CircleCI/pre-commit tests are failing cannot b
 NCL
 ~~~
 
+.. warning::
+
+  Writing new NCL code is not recommended because the
+  `NCL interpreter <https://github.com/NCAR/ncl>`__ is no longer maintained.
+
 Because there is no standard best practices document for NCL, we use
 `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for NCL code as
 well, with some minor adjustments to accommodate for differences in the

doc/sphinx/source/community/dataset.rst

@@ -43,14 +43,15 @@ and run the recipe, to make sure the CMOR checks pass without warnings or errors
 To test a pull request for a new CMORizer script:
 
 #. Download the data following the instructions included in the script and
-   place it in the ``RAWOBS`` ``rootpath`` specified in your
-   :ref:`configuration <esmvalcore:config_options>`
+   place it in the directory specified by the ``--original-data-dir`` flag
+   followed by a subdirectory ``Tier1``, ``Tier2`` or ``Tier3``, followed by
+   a subdirectory with the same name as the dataset.
 #. If available, use the downloading script by running
-   ``esmvaltool data download --config_file <config-file>  <dataset>``
-#. Run the cmorization by running ``esmvaltool data format <config-file> <dataset>``
-#. Copy the resulting data to the ``OBS`` (for CMIP5 compliant data) or ``OBS6``
-   (for CMIP6 compliant data) ``rootpath`` specified in your
-   :ref:`configuration <esmvalcore:config_options>`
+   ``esmvaltool data download --original-data-dir </path/to/save/data> <dataset>``
+#. Run the cmorization by running ``esmvaltool data format --original-data-dir </path/to/save/data> <dataset>``
+#. Copy the resulting data to the directory specified in your
+   :ref:`configuration <esmvalcore:config-data-sources>` for the ``OBS``
+   (for CMIP5 compliant data) or ``OBS6`` (for CMIP6 compliant data) project.
 #. Run ``recipes/examples/recipe_check_obs.yml`` with the new dataset to check that
    the data can be used
 

doc/sphinx/source/develop/dataset.rst

@@ -10,15 +10,14 @@ The following steps are necessary to prepare an observational
 data set for the use in ESMValTool.
 
 | `1. Check if your variable is CMOR standard`_
-| `2. Edit your configuration file`_
-| `3. Store your dataset in the right place`_
-| `3.1 Downloader script (optional)`_
-| `4. Create a cmorizer for the dataset`_
-| `4.1 Cmorizer script written in python`_
-| `4.2 Cmorizer script written in NCL`_
-| `5. Run the cmorizing script`_
-| `6. Naming convention of the observational data files`_
-| `7. Test the cmorized dataset`_
+| `2. Store your dataset in the right place`_
+| `2.1 Downloader script (optional)`_
+| `3. Create a cmorizer for the dataset`_
+| `3.1 Cmorizer script written in python`_
+| `3.2 Cmorizer script written in NCL`_
+| `4. Run the cmorizing script`_
+| `5. Naming convention of the observational data files`_
+| `6. Test the cmorized dataset`_
 
 .. note::
   **CMORization as a fix.** As of early 2020, we've started implementing cmorization as
@@ -60,17 +59,10 @@ guidelines:
 It is recommended to use an existing custom table as a template, to edit the
 content and save it as ``CMOR_<short_name>.dat``.
 
-2. Edit your configuration file
-===============================
-
-Make sure that beside the paths to the model simulations and observations, also
-the path to raw observational data to be cmorized (``RAWOBS``) is present in
-your configuration file.
-
-3. Store your dataset in the right place
+2. Store your dataset in the right place
 ========================================
 
-The folder ``RAWOBS`` needs the subdirectories ``Tier1``, ``Tier2`` and
+The folder specified by ``--original-data-dir`` needs the subdirectories ``Tier1``, ``Tier2`` and
 ``Tier3``. The different tiers describe the different levels of restrictions
 for downloading (e.g. providing contact information, licence agreements)
 and using the observations. The unformatted (raw) observations
@@ -87,14 +79,14 @@ The dataset entry should contain:
 - the ``info`` that explain how to download the data.
 
 Note that these fields should be identical to the content of the header
-of the cmorizing script (see Section `4. Create a cmorizer for the dataset`_).
+of the cmorizing script (see Section `3. Create a cmorizer for the dataset`_).
 
-3.1 Downloader script (optional)
+2.1 Downloader script (optional)
 --------------------------------
 
 A Python script can be written to download raw observations
 from source and store the data in the appropriate tier subdirectory of the
-folder ``RAWOBS`` automatically.
+folder specified by the ``--original-data-dir`` flag automatically.
 There are many downloading scripts available in
 `/esmvaltool/cmorizers/data/downloaders/datasets/
 <https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/cmorizers/data/downloaders/datasets>`_
@@ -117,15 +109,15 @@ The downloading script for the given dataset can be run with:
 
 .. code-block:: console
 
- esmvaltool data download --config_dir </path/to/config/dir/>  <dataset-name>
+ esmvaltool data download --original-data-dir </path/to/save/data>  <dataset-name>
 
 The options ``--start`` and ``--end`` can be added to the command above to
 restrict the download of raw data to a time range. They will be ignored if a specific dataset
 does not support it (i.e. because it is provided as a single file). Valid formats are
 ``YYYY``, ``YYYYMM`` and ``YYYYMMDD``. By default, already downloaded data are not overwritten
 unless the option ``--overwrite=True`` is used.
 
-4. Create a cmorizer for the dataset
+3. Create a cmorizer for the dataset
 ====================================
 
 There are many cmorizing scripts available in
@@ -145,7 +137,7 @@ the strict CMOR standard.
 In the following two subsections two cmorizing scripts, one written in Python
 and one written in NCL, are explained in more detail.
 
-4.1 Cmorizer script written in python
+3.1 Cmorizer script written in python
 -------------------------------------
 
 Find here an example of a cmorizing script, written for the ``MTE`` dataset
@@ -166,7 +158,7 @@ augmented with the custom variables described above, are available to your scrip
 The first part of this configuration file defines the filename of the raw
 observations file. The second part defines the common global attributes for
 the cmorizer output, e.g. information that is needed to piece together the
-final observations file name in the correct structure (see Section `6. Naming convention of the observational data files`_).
+final observations file name in the correct structure (see Section `5. Naming convention of the observational data files`_).
 Another global attribute is ``reference`` which includes a ``doi`` related to the dataset.
 Please see the section `adding references
 <https://docs.esmvaltool.org/en/latest/community/diagnostic.html#adding-references>`_
@@ -218,9 +210,14 @@ saves a single variable from the raw data.
 .. _utilities.py: https://github.com/ESMValGroup/ESMValTool/blob/main/esmvaltool/cmorizers/data/utilities.py
 
 
-4.2 Cmorizer script written in NCL
+3.2 Cmorizer script written in NCL
 ----------------------------------
 
+.. warning::
+
+  Writing new NCL code is not recommended because the
+  `NCL interpreter <https://github.com/NCAR/ncl>`__ is no longer maintained.
+
 Find here an example of a cmorizing script, written for the ``ESACCI XCH4``
 dataset that is available on the Copernicus Climate Data Store:
 `cds_xch4.ncl
@@ -253,7 +250,7 @@ This module contains different kinds of small fixes to the data attributes,
 coordinates, and metadata which are necessary for the data field to be
 CMOR-compliant.
 
-5. Run the cmorizing script
+4. Run the cmorizing script
 ===========================
 
 The cmorizing script for the given dataset can be run with:
@@ -269,10 +266,9 @@ does not support it (i.e. because it is provided as a single file). Valid format
 
 .. note::
 
-   The output path given in the configuration file is the path where
+   The ``output_dir`` path given in the :ref:`configuration file <esmvalcore:config>` is the path where
    your cmorized dataset will be stored. The ESMValTool will create a folder
-   with the correct tier information
-   (see Section `2. Edit your configuration file`_) if that tier folder is not
+   with the correct tier information if that tier folder is not
    already available, and then a folder named after the dataset.
    In this folder the cmorized data set will be stored as a NetCDF file.
    The cmorized dataset will be automatically moved to the correct tier
@@ -293,7 +289,7 @@ the cmorizing scripts can be run in a single command with:
 Note that options from the ```esmvaltool data download`` and
 ``esmvaltool data format`` commands can be passed to the above command.
 
-6. Naming convention of the observational data files
+5. Naming convention of the observational data files
 ====================================================
 
 For the ESMValTool to be able to read the observations from the NetCDF file,
@@ -343,8 +339,8 @@ The different parts of the name are explained in more detail here:
    specifications for the obs4MIPs data file naming convention in the
    ``config-developer.yml`` file).
 
-7. Test the cmorized dataset
-======================================
+6. Test the cmorized dataset
+============================
 
 To verify that the cmorized data file is indeed correctly formatted, you can
 run a dedicated test recipe, that does not include any diagnostic, but only

doc/sphinx/source/input.rst

@@ -171,7 +171,10 @@ Datasets for which auto-download is supported can be downloaded with:
 
 .. code-block:: bash
 
-    esmvaltool data download --config_file [CONFIG_FILE] [DATASET_LIST]
+    esmvaltool data download --original-data-dir ~/RAWOBS [DATASET_LIST]
+
+where the ``--original-data-dir`` option specifies a directory where the orignal
+data will be stored, ``~/RAWOBS`` in this example but it can be chosen freely.
 
 Note that all Tier3 and some Tier2 datasets for which auto-download is supported
 will require an authentication. In such cases enter your credentials in your
@@ -204,11 +207,12 @@ To CMORize one or more datasets, run:
 
 .. code-block:: bash
 
-    esmvaltool data format --config_file [CONFIG_FILE] [DATASET_LIST]
+    esmvaltool data format --original-data-dir ~/RAWOBS [DATASET_LIST]
+
+where the ``--original-data-dir`` option specifies a directory where the orignal
+data is stored, ``~/RAWOBS`` in this example but it can be chosen freely.
 
-The ``rootpath`` to the raw data to be CMORized must be specified in the
-:ref:`configuration <esmvalcore:config_options>` as ``RAWOBS``.
-Within this path, the data are expected to be organized in subdirectories
+Within the directory pointed to by ``--original-data-dir``, the data are expected to be organized in subdirectories
 corresponding to the data tier: Tier2 for freely-available datasets (other than
 obs4MIPs and ana4mips) and Tier3 for restricted datasets (i.e., dataset which
 requires a registration to be retrieved or provided upon request to the
@@ -219,8 +223,7 @@ The CMORization follows the `CMIP5 CMOR tables
 respectively.
 The resulting output is saved in the output_dir, again following the Tier
 structure.
-The output file names follow the definition given in :ref:`config-developer
-file <esmvalcore:config-developer>` for the ``OBS`` project:
+The output file names follow the convention:
 
 .. code-block::