Add esmvalcore.config module

doc/api/esmvalcore.api.config.rst → doc/api/esmvalcore.config.rst

@@ -1,19 +1,17 @@
-.. _api_config:
-
 Configuration
 =============
 
-This section describes the :py:class:`~esmvalcore.experimental.config` submodule of the API (:py:mod:`esmvalcore.experimental`).
+This section describes the :py:class:`~esmvalcore.config` module.
 
 Config
 ******
 
-Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.experimental.config.Config` object.
-The global configuration can be imported from the :py:mod:`esmvalcore.experimental` module as :py:data:`~esmvalcore.experimental.CFG`:
+Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.config.Config` object.
+The global configuration can be imported from the :py:mod:`esmvalcore.config` module as :py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
-    >>> from esmvalcore.experimental import CFG
+    >>> from esmvalcore.config import CFG
     >>> CFG
     Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'),
             'compress_netcdf': False,
@@ -34,7 +32,7 @@ The global configuration can be imported from the :py:mod:`esmvalcore.experiment
 
 The parameters for the user configuration file are listed :ref:`here <user configuration file>`.
 
-:py:data:`~esmvalcore.experimental.CFG` is essentially a python dictionary with a few extra functions, similar to :py:mod:`matplotlib.rcParams`.
+:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few extra functions, similar to :py:data:`matplotlib.rcParams`.
 This means that values can be updated like this:
 
 .. code-block:: python
@@ -43,7 +41,7 @@ This means that values can be updated like this:
     >>> CFG['output_dir']
     PosixPath('/home/user/esmvaltool_output')
 
-Notice that :py:data:`~esmvalcore.experimental.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
+Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory.
 All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key:
 
 .. code-block:: python
@@ -58,7 +56,7 @@ Or, if the value entered cannot be converted to the expected type:
     >>> CFG['max_parallel_tasks'] = '🐜'
     InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int
 
-:py:class:`~esmvalcore.experimental.config.Config` is also flexible, so it tries to correct the type of your input if possible:
+:py:class:`~esmvalcore.config.Config` is also flexible, so it tries to correct the type of your input if possible:
 
 .. code-block:: python
 
@@ -85,16 +83,16 @@ Session
 *******
 
 Recipes and diagnostics will be run in their own directories.
-This behaviour can be controlled via the :py:data:`~esmvalcore.experimental.config.Session` object.
-A :py:data:`~esmvalcore.experimental.config.Session` can be initiated from the global :py:class:`~esmvalcore.experimental.config.Config`.
+This behaviour can be controlled via the :py:data:`~esmvalcore.config.Session` object.
+A :py:data:`~esmvalcore.config.Session` can be initiated from the global :py:class:`~esmvalcore.config.Config`.
 
 .. code-block:: python
 
     >>> session = CFG.start_session(name='my_session')
 
-A :py:data:`~esmvalcore.experimental.config.Session` is very similar to the config.
-It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.experimental.config.Config`.
-At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.experimental.CFG`:
+A :py:data:`~esmvalcore.config.Session` is very similar to the config.
+It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.config.Config`.
+At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.config.CFG`:
 
 .. code-block:: python
 
@@ -104,7 +102,7 @@ At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.exper
     >>> print(session == CFG)  # False
     False
 
-A :py:data:`~esmvalcore.experimental.config.Session` also knows about the directories where the data will stored.
+A :py:data:`~esmvalcore.config.Session` also knows about the directories where the data will stored.
 The session name is used to prefix the directories.
 
 .. code-block:: python
@@ -120,12 +118,12 @@ The session name is used to prefix the directories.
     >>> session.plot_dir
     /home/user/my_output_dir/my_session_20201203_155821/plots
 
-Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.experimental.config.Config`.
+Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.config.Config`.
 
 
 API reference
 *************
 
-.. automodule:: esmvalcore.experimental.config
+.. automodule:: esmvalcore.config
     :no-inherited-members:
     :no-show-inheritance:

doc/api/esmvalcore.experimental.config.rst

@@ -0,0 +1,6 @@
+.. _api_config:
+
+Configuration
+=============
+
+The :py:mod:`~esmvalcore.experimental.config` module has been moved to :py:mod:`esmvalcore.config`.

doc/api/esmvalcore.experimental.rst

@@ -0,0 +1,17 @@
+.. _experimental_api:
+
+Experimental API
+================
+
+This page describes the new ESMValCore API.
+The experimental API module is available in the submodule ``esmvalcore.experimental``.
+The API is under development, so use at your own risk!
+
+.. toctree::
+   :maxdepth: 1
+
+   esmvalcore.experimental.config
+   esmvalcore.experimental.recipe
+   esmvalcore.experimental.recipe_output
+   esmvalcore.experimental.recipe_metadata
+   esmvalcore.experimental.utils

doc/api/esmvalcore.rst

@@ -7,10 +7,12 @@ ESMValCore is mostly used as a commandline tool. However, it is also possibly to
 library. This section documents the public API of ESMValCore.
 
 .. toctree::
+   :maxdepth: 1
 
    esmvalcore.cmor
+   esmvalcore.config
    esmvalcore.esgf
    esmvalcore.exceptions
    esmvalcore.iris_helpers
    esmvalcore.preprocessor
-   esmvalcore.api
+   esmvalcore.experimental

doc/develop/fixing_data.rst

@@ -441,7 +441,7 @@ To allow ESMValCore to locate the data files, use the following steps:
      Note that it is possible to predefine facets in an :ref:`extra facets file
      <add_new_fix_native_datasets_extra_facets>`.
      In this ICON example, the facet ``var_type`` is :download:`predefined
-     </../esmvalcore/_config/extra_facets/icon-mappings.yml>` for many
+     </../esmvalcore/config/extra_facets/icon-mappings.yml>` for many
      variables.
 
 .. _add_new_fix_native_datasets_fix_data:
@@ -472,7 +472,7 @@ e.g. with variable naming issues for finding files or additional information
 that is required for the fixes.
 See :ref:`extra_facets` and :ref:`extra-facets-fixes` for more details on this.
 An example of such a file for IPSL-CM6 is given :download:`here
-<../../esmvalcore/_config/extra_facets/ipslcm-mappings.yml>`.
+<../../esmvalcore/config/extra_facets/ipslcm-mappings.yml>`.
 
 
 .. _extra-facets-fixes:

doc/quickstart/configure.rst

@@ -754,7 +754,7 @@ For example, this is used to automatically add ``product: output1`` to any
 variable of any CMIP5 dataset that does not have a ``product`` key yet:
 
 .. code-block:: yaml
-   :caption: Extra facet example file `cmip5-product.yml <https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/_config/extra_facets/cmip5-product.yml>`_
+   :caption: Extra facet example file `cmip5-product.yml <https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/config/extra_facets/cmip5-product.yml>`_
 
    '*':
      '*':
@@ -765,15 +765,15 @@ Location of the extra facets files
 Extra facets files can be placed in several different places. When we use them
 to support a particular use-case within the ESMValTool project, they will be
 provided in the sub-folder `extra_facets` inside the package
-`esmvalcore._config`. If they are used from the user side, they can be either
+:mod:`esmvalcore.config`. If they are used from the user side, they can be either
 placed in `~/.esmvaltool/extra_facets` or in any other directory of the users
 choosing. In that case this directory must be added to the `config-user.yml`
 file under the `extra_facets_dir` setting, which can take a single directory or
 a list of directories.
 
 The order in which the directories are searched is
 
-1. The internal directory `esmvalcore._config/extra_facets`
+1. The internal directory `esmvalcore.config/extra_facets`
 2. The default user directory `~/.esmvaltool/extra_facets`
 3. The custom user directories in the order in which they are given in
    `config-user.yml`.

doc/quickstart/find_data.rst

@@ -202,7 +202,7 @@ overwritten in the recipe.
 Similar to any other fix, the CESM fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`cesm-mappings.yml
-</../esmvalcore/_config/extra_facets/cesm-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/cesm-mappings.yml>` is used for that
 purpose.
 Currently, this file only contains default facets for a single variable
 (`tas`); for other variables, these entries need to be defined in the recipe.
@@ -269,7 +269,7 @@ the recipe.
 Similar to any other fix, the EMAC fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`emac-mappings.yml
-</../esmvalcore/_config/extra_facets/emac-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/emac-mappings.yml>` is used for that
 purpose.
 For some variables, extra facets are necessary; otherwise ESMValTool cannot
 read them properly.
@@ -340,7 +340,7 @@ the recipe.
 Similar to any other fix, the ICON fix allows the use of :ref:`extra
 facets<extra_facets>`.
 By default, the file :download:`icon-mappings.yml
-</../esmvalcore/_config/extra_facets/icon-mappings.yml>` is used for that
+</../esmvalcore/config/extra_facets/icon-mappings.yml>` is used for that
 purpose.
 For some variables, extra facets are necessary; otherwise ESMValTool cannot
 read them properly.
@@ -405,7 +405,7 @@ The ``Output`` format is an example of a case where variables are grouped in
 multi-variable files, which name cannot be computed directly from datasets
 attributes alone but requires to use an extra_facets file, which principles are
 explained in :ref:`extra_facets`, and which content is :download:`available here
-</../esmvalcore/_config/extra_facets/ipslcm-mappings.yml>`. These multi-variable
+</../esmvalcore/config/extra_facets/ipslcm-mappings.yml>`. These multi-variable
 files must also undergo some data selection.
 
 

esmvalcore/_citation.py

@@ -7,7 +7,7 @@
 
 import requests
 
-from ._config import DIAGNOSTICS
+from .config._diagnostics import DIAGNOSTICS
 
 logger = logging.getLogger(__name__)
 

esmvalcore/_config/__init__.py

@@ -1,25 +1,27 @@
-"""ESMValTool configuration."""
-from ._config import (
-    get_activity,
-    get_institutes,
-    get_project_config,
-    get_extra_facets,
-    load_config_developer,
-    read_config_developer_file,
-    read_config_user_file,
-)
-from ._diagnostics import DIAGNOSTICS, TAGS
-from ._logging import configure_logging
+import warnings
 
-__all__ = (
-    'read_config_user_file',
-    'read_config_developer_file',
-    'load_config_developer',
-    'get_extra_facets',
-    'get_project_config',
-    'get_institutes',
-    'get_activity',
-    'DIAGNOSTICS',
-    'TAGS',
+from ..config import CFG
+from ..config._logging import configure_logging
+from ..exceptions import ESMValCoreDeprecationWarning
+
+__all__ = [
+    'CFG',
     'configure_logging',
+    'read_config_user_file',
+]
+
+warnings.warn(
+    "The private module `esmvalcore._config` has been deprecated in "
+    "ESMValCore version 2.8.0 and is scheduled for removal in version 2.9.0. "
+    "Please use the public module `esmvalcore.config` instead.",
+    ESMValCoreDeprecationWarning,
 )
+
+
+def read_config_user_file(config_file, folder_name, options=None):
+    """Read config user file and store settings in a dictionary."""
+    CFG.load_from_file(config_file)
+    session = CFG.start_session(folder_name)
+    session.update(options)
+    cfg = session.to_config_user()
+    return cfg