Merge configuration object from multiple files (instead of one single file) (#2448)

Co-authored-by: Bouwe Andela <[email protected]>

Author: schlunma

doc/api/esmvalcore.config.rst

@@ -1,13 +1,15 @@
+.. _api_configuration:
+
 Configuration
 =============
 
 This section describes the :py:class:`~esmvalcore.config` module.
 
-Config
-******
+CFG
+***
 
-Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.config.Config` object.
-The global configuration can be imported from the :py:mod:`esmvalcore.config` module as :py:data:`~esmvalcore.config.CFG`:
+Configuration of ESMValCore/Tool is done via :py:data:`~esmvalcore.config.CFG`
+object:
 
 .. code-block:: python
 
@@ -16,7 +18,6 @@ The global configuration can be imported from the :py:mod:`esmvalcore.config` mo
     Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
             'compress_netcdf': False,
             'config_developer_file': None,
-            'config_file': PosixPath('/home/user/.esmvaltool/config-user.yml'),
             'drs': {'CMIP5': 'default', 'CMIP6': 'default'},
             'exit_on_warning': False,
             'log_level': 'info',
@@ -30,9 +31,10 @@ The global configuration can be imported from the :py:mod:`esmvalcore.config` mo
                          'default': '~/default_inputpath'},
             'save_intermediary_cubes': False)
 
-The parameters for the user configuration file are listed :ref:`here <user configuration file>`.
+All configuration parameters are listed :ref:`here <config_options>`.
 
-:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few extra functions, similar to :py:data:`matplotlib.rcParams`.
+:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few
+extra functions, similar to :py:data:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -41,8 +43,10 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
-All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
+Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path
+to an instance of :class:`pathlib.Path` and expands the home directory.
+All values entered into the config are validated to prevent mistakes, for
+example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
 
@@ -56,43 +60,53 @@ Or, if the value entered cannot be converted to the expected type:
     >>> CFG['max_parallel_tasks'] = '🐜'
     InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-:py:class:`~esmvalcore.config.Config` is also flexible, so it tries to correct the type of your input if possible:
+:py:data:`~esmvalcore.config.CFG` is also flexible, so it tries to correct the
+type of your input if possible:
 
 .. code-block:: python
 
     >>> CFG['max_parallel_tasks'] = '8'  # str
     >>> type(CFG['max_parallel_tasks'])
     int
 
-By default, the config is loaded from the default location (``/home/user/.esmvaltool/config-user.yml``).
-If it does not exist, it falls back to the default values.
-to load a different file:
+By default, the configuration is loaded from YAML files in the user's home
+directory at ``~/.config/esmvaltool``.
+If set, this can be overwritten with the ``ESMVALTOOL_CONFIG_DIR`` environment
+variable.
+Defaults for options that are not specified explicitly are listed :ref:`here
+<config_options>`.
+To reload the current configuration object according to these rules, use:
 
 .. code-block:: python
 
-    >>> CFG.load_from_file('~/my-config.yml')
+    >>> CFG.reload()
 
-Or to reload the current config:
+To load the configuration object from custom directories, use:
 
 .. code-block:: python
 
-    >>> CFG.reload()
+    >>> dirs = ['my/default/config', 'my/custom/config']
+    >>> CFG.load_from_dirs(dirs)
 
 
 Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the :py:data:`~esmvalcore.config.Session` object.
-A :py:data:`~esmvalcore.config.Session` can be initiated from the global :py:class:`~esmvalcore.config.Config`.
+This behavior can be controlled via the :py:data:`~esmvalcore.config.Session`
+object.
+A :py:data:`~esmvalcore.config.Session` must always be initiated from the
+global :py:data:`~esmvalcore.config.CFG` object:
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
 A :py:data:`~esmvalcore.config.Session` is very similar to the config.
-It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.config.Config`.
-At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.config.CFG`:
+It is also a dictionary, and copies all the keys from the
+:py:data:`~esmvalcore.config.CFG` object.
+At this moment, ``session`` is essentially a copy of
+:py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
@@ -102,7 +116,8 @@ At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.confi
     >>> print(session == CFG)  # False
     False
 
-A :py:data:`~esmvalcore.config.Session` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.config.Session` also knows about the directories where
+the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -118,7 +133,8 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.config.Config`.
+Unlike the global configuration, of which only one can exist, multiple sessions
+can be initiated from :py:data:`~esmvalcore.config.CFG`.
 
 
 API reference

doc/contributing.rst

@@ -571,7 +571,7 @@ users.
 
 When making changes, e.g. to the :ref:`recipe format <recipe_overview>`, the
 :ref:`diagnostic script interface <interfaces>`, the public
-:ref:`Python API <api>`, or the :ref:`configuration file format <config>`,
+:ref:`Python API <api>`, or the :ref:`configuration format <config>`,
 keep in mind that this may affect many users.
 To keep the tool user friendly, try to avoid making changes that are not
 backward compatible, i.e. changes that require users to change their existing

doc/develop/fixing_data.rst

@@ -329,9 +329,9 @@ severity. From highest to lowest:
 
 Users can have control about which levels of issues are interpreted as errors,
 and therefore make the checker fail or warnings or debug messages.
-For this purpose there is an optional command line option `--check-level`
-that can take a number of values, listed below from the lowest level of
-strictness to the highest:
+For this purpose there is an optional :ref:`configuration option
+<config_options>` ``check_level`` that can take a number of values, listed
+below from the lowest level of strictness to the highest:
 
 - ``ignore``: all issues, regardless of severity, will be reported as
   warnings. Checker will never fail. Use this at your own risk.
@@ -375,8 +375,8 @@ To allow ESMValCore to locate the data files, use the following steps:
 
    - If you want to use the ``native6`` project (recommended for datasets whose
      input files can be easily moved to the usual ``native6`` directory
-     structure given by the ``rootpath`` in your :ref:`user configuration
-     file`; this is usually the case for native reanalysis/observational
+     structure given by the :ref:`configuration option <config_options>`
+     ``rootpath``; this is usually the case for native reanalysis/observational
      datasets):
 
      The entry ``native6`` of ``config-developer.yml`` should be complemented
@@ -399,17 +399,17 @@ To allow ESMValCore to locate the data files, use the following steps:
 
      To find your native data (e.g., called ``MYDATA``) that is for example
      located in ``{rootpath}/MYDATA/amip/run1/42-0/atm/run1_1979.nc``
-     (``{rootpath}`` is ESMValTool's ``rootpath`` for the project ``native6``
-     defined in your :ref:`user configuration file`), use the following dataset
+     (``{rootpath}`` is ESMValTool's ``rootpath`` :ref:`configuration option
+     <config_options>` for the project ``native6``), use the following dataset
      entry in your recipe
 
      .. code-block:: yaml
 
         datasets:
           - {project: native6, dataset: MYDATA, exp: amip, simulation: run1, version: 42-0, type: atm}
 
-     and make sure to use the following DRS for the project ``native6`` in your
-     :ref:`user configuration file`:
+     and make sure to use the following :ref:`configuration option
+     <config_options>` ``drs``:
 
      .. code-block:: yaml
 
@@ -437,9 +437,8 @@ To allow ESMValCore to locate the data files, use the following steps:
 
      To find your ICON data that is for example located in files like
      ``{rootpath}/amip/amip_atm_2d_ml_20000101T000000Z.nc`` (``{rootpath}`` is
-     ESMValTool ``rootpath`` for the project ``ICON`` defined in your
-     :ref:`user configuration file`), use the following dataset entry in your
-     recipe:
+     ESMValCore's :ref:`configuration option <config_options>` ``rootpath`` for
+     the project ``ICON``), use the following dataset entry in your recipe:
 
      .. code-block:: yaml