diff --git a/doc/api/esmvalcore.api.rst b/doc/api/esmvalcore.api.rst deleted file mode 100644 index 9290731e5f..0000000000 --- a/doc/api/esmvalcore.api.rst +++ /dev/null @@ -1,16 +0,0 @@ -.. _experimental_api: - -Experimental API -================ - -This page describes the new ESMValCore API. -The API module is available in the submodule ``esmvalcore.experimental``. -The API is under development, so use at your own risk! - -.. toctree:: - - esmvalcore.api.config - esmvalcore.api.recipe - esmvalcore.api.recipe_output - esmvalcore.api.recipe_metadata - esmvalcore.api.utils diff --git a/doc/api/esmvalcore.api.config.rst b/doc/api/esmvalcore.config.rst similarity index 71% rename from doc/api/esmvalcore.api.config.rst rename to doc/api/esmvalcore.config.rst index eb85547ada..659d574509 100644 --- a/doc/api/esmvalcore.api.config.rst +++ b/doc/api/esmvalcore.config.rst @@ -1,19 +1,17 @@ -.. _api_config: - Configuration ============= -This section describes the :py:class:`~esmvalcore.experimental.config` submodule of the API (:py:mod:`esmvalcore.experimental`). +This section describes the :py:class:`~esmvalcore.config` module. Config ****** -Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.experimental.config.Config` object. -The global configuration can be imported from the :py:mod:`esmvalcore.experimental` module as :py:data:`~esmvalcore.experimental.CFG`: +Configuration of ESMValCore/Tool is done via the :py:class:`~esmvalcore.config.Config` object. +The global configuration can be imported from the :py:mod:`esmvalcore.config` module as :py:data:`~esmvalcore.config.CFG`: .. code-block:: python - >>> from esmvalcore.experimental import CFG + >>> from esmvalcore.config import CFG >>> CFG Config({'auxiliary_data_dir': PosixPath('/home/user/auxiliary_data'), 'compress_netcdf': False, @@ -34,7 +32,7 @@ The global configuration can be imported from the :py:mod:`esmvalcore.experiment The parameters for the user configuration file are listed :ref:`here `. -:py:data:`~esmvalcore.experimental.CFG` is essentially a python dictionary with a few extra functions, similar to :py:mod:`matplotlib.rcParams`. +:py:data:`~esmvalcore.config.CFG` is essentially a python dictionary with a few extra functions, similar to :py:data:`matplotlib.rcParams`. This means that values can be updated like this: .. code-block:: python @@ -43,7 +41,7 @@ This means that values can be updated like this: >>> CFG['output_dir'] PosixPath('/home/user/esmvaltool_output') -Notice that :py:data:`~esmvalcore.experimental.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory. +Notice that :py:data:`~esmvalcore.config.CFG` automatically converts the path to an instance of ``pathlib.Path`` and expands the home directory. All values entered into the config are validated to prevent mistakes, for example, it will warn you if you make a typo in the key: .. code-block:: python @@ -58,7 +56,7 @@ Or, if the value entered cannot be converted to the expected type: >>> CFG['max_parallel_tasks'] = '🐜' InvalidConfigParameter: Key `max_parallel_tasks`: Could not convert '🐜' to int -:py:class:`~esmvalcore.experimental.config.Config` is also flexible, so it tries to correct the type of your input if possible: +:py:class:`~esmvalcore.config.Config` is also flexible, so it tries to correct the type of your input if possible: .. code-block:: python @@ -85,16 +83,16 @@ Session ******* Recipes and diagnostics will be run in their own directories. -This behaviour can be controlled via the :py:data:`~esmvalcore.experimental.config.Session` object. -A :py:data:`~esmvalcore.experimental.config.Session` can be initiated from the global :py:class:`~esmvalcore.experimental.config.Config`. +This behaviour can be controlled via the :py:data:`~esmvalcore.config.Session` object. +A :py:data:`~esmvalcore.config.Session` can be initiated from the global :py:class:`~esmvalcore.config.Config`. .. code-block:: python >>> session = CFG.start_session(name='my_session') -A :py:data:`~esmvalcore.experimental.config.Session` is very similar to the config. -It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.experimental.config.Config`. -At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.experimental.CFG`: +A :py:data:`~esmvalcore.config.Session` is very similar to the config. +It is also a dictionary, and copies all the keys from the :py:class:`~esmvalcore.config.Config`. +At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.config.CFG`: .. code-block:: python @@ -104,7 +102,7 @@ At this moment, ``session`` is essentially a copy of :py:data:`~esmvalcore.exper >>> print(session == CFG) # False False -A :py:data:`~esmvalcore.experimental.config.Session` also knows about the directories where the data will stored. +A :py:data:`~esmvalcore.config.Session` also knows about the directories where the data will stored. The session name is used to prefix the directories. .. code-block:: python @@ -120,12 +118,12 @@ The session name is used to prefix the directories. >>> session.plot_dir /home/user/my_output_dir/my_session_20201203_155821/plots -Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.experimental.config.Config`. +Unlike the global configuration, of which only one can exist, multiple sessions can be initiated from :py:class:`~esmvalcore.config.Config`. API reference ************* -.. automodule:: esmvalcore.experimental.config +.. automodule:: esmvalcore.config :no-inherited-members: :no-show-inheritance: diff --git a/doc/api/esmvalcore.experimental.config.rst b/doc/api/esmvalcore.experimental.config.rst new file mode 100644 index 0000000000..cc2ddfab33 --- /dev/null +++ b/doc/api/esmvalcore.experimental.config.rst @@ -0,0 +1,6 @@ +.. _api_config: + +Configuration +============= + +The :py:mod:`~esmvalcore.experimental.config` module has been moved to :py:mod:`esmvalcore.config`. diff --git a/doc/api/esmvalcore.api.recipe.rst b/doc/api/esmvalcore.experimental.recipe.rst similarity index 100% rename from doc/api/esmvalcore.api.recipe.rst rename to doc/api/esmvalcore.experimental.recipe.rst diff --git a/doc/api/esmvalcore.api.recipe_metadata.rst b/doc/api/esmvalcore.experimental.recipe_metadata.rst similarity index 100% rename from doc/api/esmvalcore.api.recipe_metadata.rst rename to doc/api/esmvalcore.experimental.recipe_metadata.rst diff --git a/doc/api/esmvalcore.api.recipe_output.rst b/doc/api/esmvalcore.experimental.recipe_output.rst similarity index 100% rename from doc/api/esmvalcore.api.recipe_output.rst rename to doc/api/esmvalcore.experimental.recipe_output.rst diff --git a/doc/api/esmvalcore.experimental.rst b/doc/api/esmvalcore.experimental.rst new file mode 100644 index 0000000000..fae566d4f4 --- /dev/null +++ b/doc/api/esmvalcore.experimental.rst @@ -0,0 +1,17 @@ +.. _experimental_api: + +Experimental API +================ + +This page describes the new ESMValCore API. +The experimental API module is available in the submodule ``esmvalcore.experimental``. +The API is under development, so use at your own risk! + +.. toctree:: + :maxdepth: 1 + + esmvalcore.experimental.config + esmvalcore.experimental.recipe + esmvalcore.experimental.recipe_output + esmvalcore.experimental.recipe_metadata + esmvalcore.experimental.utils diff --git a/doc/api/esmvalcore.api.utils.rst b/doc/api/esmvalcore.experimental.utils.rst similarity index 100% rename from doc/api/esmvalcore.api.utils.rst rename to doc/api/esmvalcore.experimental.utils.rst diff --git a/doc/api/esmvalcore.rst b/doc/api/esmvalcore.rst index 71699eb496..23787500da 100644 --- a/doc/api/esmvalcore.rst +++ b/doc/api/esmvalcore.rst @@ -7,10 +7,12 @@ ESMValCore is mostly used as a commandline tool. However, it is also possibly to library. This section documents the public API of ESMValCore. .. toctree:: + :maxdepth: 1 esmvalcore.cmor + esmvalcore.config esmvalcore.esgf esmvalcore.exceptions esmvalcore.iris_helpers esmvalcore.preprocessor - esmvalcore.api + esmvalcore.experimental diff --git a/doc/develop/fixing_data.rst b/doc/develop/fixing_data.rst index ede188291b..d097327e65 100644 --- a/doc/develop/fixing_data.rst +++ b/doc/develop/fixing_data.rst @@ -441,7 +441,7 @@ To allow ESMValCore to locate the data files, use the following steps: Note that it is possible to predefine facets in an :ref:`extra facets file `. In this ICON example, the facet ``var_type`` is :download:`predefined - ` for many + ` for many variables. .. _add_new_fix_native_datasets_fix_data: @@ -472,7 +472,7 @@ e.g. with variable naming issues for finding files or additional information that is required for the fixes. See :ref:`extra_facets` and :ref:`extra-facets-fixes` for more details on this. An example of such a file for IPSL-CM6 is given :download:`here -<../../esmvalcore/_config/extra_facets/ipslcm-mappings.yml>`. +<../../esmvalcore/config/extra_facets/ipslcm-mappings.yml>`. .. _extra-facets-fixes: diff --git a/doc/quickstart/configure.rst b/doc/quickstart/configure.rst index 691ac97dd5..916c10306a 100644 --- a/doc/quickstart/configure.rst +++ b/doc/quickstart/configure.rst @@ -754,7 +754,7 @@ For example, this is used to automatically add ``product: output1`` to any variable of any CMIP5 dataset that does not have a ``product`` key yet: .. code-block:: yaml - :caption: Extra facet example file `cmip5-product.yml `_ + :caption: Extra facet example file `cmip5-product.yml `_ '*': '*': @@ -765,7 +765,7 @@ Location of the extra facets files Extra facets files can be placed in several different places. When we use them to support a particular use-case within the ESMValTool project, they will be provided in the sub-folder `extra_facets` inside the package -`esmvalcore._config`. If they are used from the user side, they can be either +:mod:`esmvalcore.config`. If they are used from the user side, they can be either placed in `~/.esmvaltool/extra_facets` or in any other directory of the users choosing. In that case this directory must be added to the `config-user.yml` file under the `extra_facets_dir` setting, which can take a single directory or @@ -773,7 +773,7 @@ a list of directories. The order in which the directories are searched is -1. The internal directory `esmvalcore._config/extra_facets` +1. The internal directory `esmvalcore.config/extra_facets` 2. The default user directory `~/.esmvaltool/extra_facets` 3. The custom user directories in the order in which they are given in `config-user.yml`. diff --git a/doc/quickstart/find_data.rst b/doc/quickstart/find_data.rst index 8146377e74..347b5265e3 100644 --- a/doc/quickstart/find_data.rst +++ b/doc/quickstart/find_data.rst @@ -202,7 +202,7 @@ overwritten in the recipe. Similar to any other fix, the CESM fix allows the use of :ref:`extra facets`. By default, the file :download:`cesm-mappings.yml -` is used for that +` is used for that purpose. Currently, this file only contains default facets for a single variable (`tas`); for other variables, these entries need to be defined in the recipe. @@ -269,7 +269,7 @@ the recipe. Similar to any other fix, the EMAC fix allows the use of :ref:`extra facets`. By default, the file :download:`emac-mappings.yml -` is used for that +` is used for that purpose. For some variables, extra facets are necessary; otherwise ESMValTool cannot read them properly. @@ -340,7 +340,7 @@ the recipe. Similar to any other fix, the ICON fix allows the use of :ref:`extra facets`. By default, the file :download:`icon-mappings.yml -` is used for that +` is used for that purpose. For some variables, extra facets are necessary; otherwise ESMValTool cannot read them properly. @@ -405,7 +405,7 @@ The ``Output`` format is an example of a case where variables are grouped in multi-variable files, which name cannot be computed directly from datasets attributes alone but requires to use an extra_facets file, which principles are explained in :ref:`extra_facets`, and which content is :download:`available here -`. These multi-variable +`. These multi-variable files must also undergo some data selection. diff --git a/esmvalcore/_citation.py b/esmvalcore/_citation.py index c6129c21a6..96e9286f47 100644 --- a/esmvalcore/_citation.py +++ b/esmvalcore/_citation.py @@ -7,7 +7,7 @@ import requests -from ._config import DIAGNOSTICS +from .config._diagnostics import DIAGNOSTICS logger = logging.getLogger(__name__) diff --git a/esmvalcore/_config/__init__.py b/esmvalcore/_config/__init__.py index 1b7357ac1c..ca5d7d03ea 100644 --- a/esmvalcore/_config/__init__.py +++ b/esmvalcore/_config/__init__.py @@ -1,25 +1,27 @@ -"""ESMValTool configuration.""" -from ._config import ( - get_activity, - get_institutes, - get_project_config, - get_extra_facets, - load_config_developer, - read_config_developer_file, - read_config_user_file, -) -from ._diagnostics import DIAGNOSTICS, TAGS -from ._logging import configure_logging +import warnings -__all__ = ( - 'read_config_user_file', - 'read_config_developer_file', - 'load_config_developer', - 'get_extra_facets', - 'get_project_config', - 'get_institutes', - 'get_activity', - 'DIAGNOSTICS', - 'TAGS', +from ..config import CFG +from ..config._logging import configure_logging +from ..exceptions import ESMValCoreDeprecationWarning + +__all__ = [ + 'CFG', 'configure_logging', + 'read_config_user_file', +] + +warnings.warn( + "The private module `esmvalcore._config` has been deprecated in " + "ESMValCore version 2.8.0 and is scheduled for removal in version 2.9.0. " + "Please use the public module `esmvalcore.config` instead.", + ESMValCoreDeprecationWarning, ) + + +def read_config_user_file(config_file, folder_name, options=None): + """Read config user file and store settings in a dictionary.""" + CFG.load_from_file(config_file) + session = CFG.start_session(folder_name) + session.update(options) + cfg = session.to_config_user() + return cfg diff --git a/esmvalcore/_config/_config.py b/esmvalcore/_config/_config.py deleted file mode 100644 index d5c69d5885..0000000000 --- a/esmvalcore/_config/_config.py +++ /dev/null @@ -1,259 +0,0 @@ -"""Functions dealing with config-user.yml / config-developer.yml.""" -import collections.abc -import datetime -import fnmatch -import logging -import os -import sys -import warnings -from functools import lru_cache -from pathlib import Path - -import yaml - -from esmvalcore.cmor.table import CMOR_TABLES, read_cmor_tables -from esmvalcore.exceptions import RecipeError - -logger = logging.getLogger(__name__) - -CFG = {} - -if sys.version_info[:2] >= (3, 9): - # pylint: disable=no-name-in-module - from importlib.resources import files as importlib_files -else: - from importlib_resources import files as importlib_files - - -def _deep_update(dictionary, update): - for key, value in update.items(): - if isinstance(value, collections.abc.Mapping): - dictionary[key] = _deep_update(dictionary.get(key, {}), value) - else: - dictionary[key] = value - return dictionary - - -@lru_cache() -def _load_extra_facets(project, extra_facets_dir): - config = {} - config_paths = [ - importlib_files("esmvalcore._config") / "extra_facets", - Path.home() / ".esmvaltool" / "extra_facets", - ] - config_paths.extend([Path(p) for p in extra_facets_dir]) - for config_path in config_paths: - config_file_paths = config_path.glob(f"{project.lower()}-*.yml") - for config_file_path in sorted(config_file_paths): - logger.debug("Loading extra facets from %s", config_file_path) - with config_file_path.open() as config_file: - config_piece = yaml.safe_load(config_file) - if config_piece: - _deep_update(config, config_piece) - return config - - -def get_extra_facets(project, dataset, mip, short_name, extra_facets_dir): - """Read configuration files with additional variable information.""" - project_details = _load_extra_facets(project, extra_facets_dir) - - def pattern_filter(patterns, name): - """Get the subset of the list `patterns` that `name` matches. - - Parameters - ---------- - patterns : :obj:`list` of :obj:`str` - A list of strings that may contain shell-style wildcards. - name : str - A string describing the dataset, mip, or short_name. - - Returns - ------- - :obj:`list` of :obj:`str` - The subset of patterns that `name` matches. - """ - return [pat for pat in patterns if fnmatch.fnmatchcase(name, pat)] - - extra_facets = {} - for dataset_ in pattern_filter(project_details, dataset): - for mip_ in pattern_filter(project_details[dataset_], mip): - for var in pattern_filter(project_details[dataset_][mip_], - short_name): - facets = project_details[dataset_][mip_][var] - extra_facets.update(facets) - - return extra_facets - - -def read_config_user_file(config_file, folder_name, options=None): - """Read config user file and store settings in a dictionary.""" - if not config_file: - config_file = '~/.esmvaltool/config-user.yml' - config_file = _normalize_path(config_file) - # Read user config file - if not os.path.exists(config_file): - print(f"ERROR: Config file {config_file} does not exist") - - with open(config_file, 'r') as file: - cfg = yaml.safe_load(file) - - if options is None: - options = dict() - for key, value in options.items(): - cfg[key] = value - - # set defaults - defaults = { - 'auxiliary_data_dir': '~/auxiliary_data', - 'compress_netcdf': False, - 'config_developer_file': None, - 'drs': {}, - 'download_dir': '~/climate_data', - 'exit_on_warning': False, - 'extra_facets_dir': tuple(), - 'max_parallel_tasks': None, - 'offline': True, - 'output_file_type': 'png', - 'output_dir': '~/esmvaltool_output', - 'profile_diagnostic': False, - 'remove_preproc_dir': True, - 'resume_from': [], - 'run_diagnostic': True, - 'save_intermediary_cubes': False, - } - - for key in defaults: - if key not in cfg: - logger.info( - "No %s specification in config file, " - "defaulting to %s", key, defaults[key]) - cfg[key] = defaults[key] - - cfg['output_dir'] = _normalize_path(cfg['output_dir']) - cfg['download_dir'] = _normalize_path(cfg['download_dir']) - cfg['auxiliary_data_dir'] = _normalize_path(cfg['auxiliary_data_dir']) - - if isinstance(cfg['extra_facets_dir'], str): - cfg['extra_facets_dir'] = (_normalize_path(cfg['extra_facets_dir']), ) - else: - cfg['extra_facets_dir'] = tuple( - _normalize_path(p) for p in cfg['extra_facets_dir']) - - cfg['config_developer_file'] = _normalize_path( - cfg['config_developer_file']) - cfg['config_file'] = config_file - - for section in ['rootpath', 'drs']: - if 'obs4mips' in cfg[section]: - logger.warning( - "Correcting capitalization, project 'obs4mips'" - " should be written as 'obs4MIPs' in %s in %s", section, - config_file) - cfg[section]['obs4MIPs'] = cfg[section].pop('obs4mips') - - for key in cfg['rootpath']: - root = cfg['rootpath'][key] - if isinstance(root, str): - cfg['rootpath'][key] = [_normalize_path(root)] - else: - cfg['rootpath'][key] = [_normalize_path(path) for path in root] - - # insert a directory date_time_recipe_usertag in the output paths - now = datetime.datetime.utcnow().strftime("%Y%m%d_%H%M%S") - new_subdir = '_'.join((folder_name, now)) - cfg['output_dir'] = os.path.join(cfg['output_dir'], new_subdir) - - # create subdirectories - cfg['preproc_dir'] = os.path.join(cfg['output_dir'], 'preproc') - cfg['work_dir'] = os.path.join(cfg['output_dir'], 'work') - cfg['plot_dir'] = os.path.join(cfg['output_dir'], 'plots') - cfg['run_dir'] = os.path.join(cfg['output_dir'], 'run') - - # Read developer configuration file - load_config_developer(cfg['config_developer_file']) - - # Validate configuration using the experimental module to avoid a crash - # after running the recipe because the html output writer uses this. - # In the long run, we need to replace this module with the Session from - # the experimental module. - with warnings.catch_warnings(): - # ignore experimental API warning - warnings.simplefilter("ignore") - from esmvalcore.experimental.config._config_object import Session - Session.from_config_user(cfg) - - return cfg - - -def _normalize_path(path): - """Normalize paths. - - Expand ~ character and environment variables and convert path to absolute. - - Parameters - ---------- - path: str - Original path - - Returns - ------- - str: - Normalized path - """ - if path is None: - return None - return os.path.abspath(os.path.expanduser(os.path.expandvars(path))) - - -def read_config_developer_file(cfg_file=None): - """Read the developer's configuration file.""" - if cfg_file is None: - cfg_file = Path(__file__).parents[1] / 'config-developer.yml' - - with open(cfg_file, 'r') as file: - cfg = yaml.safe_load(file) - - if 'obs4mips' in cfg: - logger.warning( - "Correcting capitalization, project 'obs4mips'" - " should be written as 'obs4MIPs' in %s", cfg_file) - cfg['obs4MIPs'] = cfg.pop('obs4mips') - - return cfg - - -def load_config_developer(cfg_file=None): - """Load the config developer file and initialize CMOR tables.""" - cfg_developer = read_config_developer_file(cfg_file) - for key, value in cfg_developer.items(): - CFG[key] = value - read_cmor_tables(CFG) - - -def get_project_config(project): - """Get developer-configuration for project.""" - if project in CFG: - return CFG[project] - raise RecipeError(f"Project '{project}' not in config-developer.yml") - - -def get_institutes(variable): - """Return the institutes given the dataset name in CMIP6.""" - dataset = variable['dataset'] - project = variable['project'] - try: - return CMOR_TABLES[project].institutes[dataset] - except (KeyError, AttributeError): - return [] - - -def get_activity(variable): - """Return the activity given the experiment name in CMIP6.""" - project = variable['project'] - try: - exp = variable['exp'] - if isinstance(exp, list): - return [CMOR_TABLES[project].activities[value][0] for value in exp] - return CMOR_TABLES[project].activities[exp][0] - except (KeyError, AttributeError): - return None diff --git a/esmvalcore/_config/config-logging.yml b/esmvalcore/_config/config-logging.yml deleted file mode 100644 index b78b308ed8..0000000000 --- a/esmvalcore/_config/config-logging.yml +++ /dev/null @@ -1,33 +0,0 @@ -# Logger configuration ---- - -version: 1 -disable_existing_loggers: false -formatters: - console: - format: '%(asctime)s UTC [%(process)d] %(levelname)-7s %(message)s' - brief: - format: '%(levelname)-7s [%(process)d] %(message)s' - debug: - format: '%(asctime)s UTC [%(process)d] %(levelname)-7s %(name)s:%(lineno)s %(message)s' -handlers: - console: - class: logging.StreamHandler - level: INFO - formatter: console - stream: ext://sys.stdout - simple_log_file: - class: logging.FileHandler - level: INFO - formatter: brief - filename: main_log.txt - mode: w - debug_log_file: - class: logging.FileHandler - level: DEBUG - formatter: debug - filename: main_log_debug.txt - mode: w -root: - level: DEBUG - handlers: [console, simple_log_file, debug_log_file] diff --git a/esmvalcore/_config/config-logging.yml b/esmvalcore/_config/config-logging.yml new file mode 120000 index 0000000000..8afb2cca08 --- /dev/null +++ b/esmvalcore/_config/config-logging.yml @@ -0,0 +1 @@ +../config/config-logging.yml \ No newline at end of file diff --git a/esmvalcore/_data_finder.py b/esmvalcore/_data_finder.py index 0dfaec0c33..a2c4b90ff2 100644 --- a/esmvalcore/_data_finder.py +++ b/esmvalcore/_data_finder.py @@ -8,7 +8,7 @@ import iris import isodate -from ._config import get_project_config +from .config._config import get_project_config from .exceptions import RecipeError logger = logging.getLogger(__name__) @@ -144,7 +144,6 @@ def dates_to_timerange(start_date, end_date): ------- str ``timerange`` in the form ``'start_date/end_date'``. - """ start_date = str(start_date) end_date = str(end_date) @@ -283,7 +282,6 @@ def _truncate_dates(date, file_date): same number of digits. If this is not the case, pad the dates with leading zeros (e.g., use ``date='0100'`` and ``file_date='199901'`` for a correct comparison). - """ date = re.sub("[^0-9]", '', date) file_date = re.sub("[^0-9]", '', file_date) @@ -436,7 +434,7 @@ def get_rootpath(rootpath, project): if nonexistent and (key, nonexistent) not in ROOTPATH_WARNED: logger.warning( "'%s' rootpaths '%s' set in config-user.yml do not exist", - key, ', '.join(nonexistent)) + key, ', '.join(str(p) for p in nonexistent)) ROOTPATH_WARNED.add((key, nonexistent)) return rootpath[key] raise KeyError('default rootpath must be specified in config-user file') diff --git a/esmvalcore/_main.py b/esmvalcore/_main.py index d6b9d94cd5..0488359ffe 100755 --- a/esmvalcore/_main.py +++ b/esmvalcore/_main.py @@ -28,6 +28,7 @@ """ # noqa: line-too-long pylint: disable=line-too-long # pylint: disable=import-outside-toplevel import logging +import os from pathlib import Path import fire @@ -50,7 +51,6 @@ def parse_resume(resume, recipe): """Set `resume` to a correct value and sanity check.""" - import os if not resume: return [] if isinstance(resume, str): @@ -68,14 +68,14 @@ def parse_resume(resume, recipe): return resume -def process_recipe(recipe_file, config_user): +def process_recipe(recipe_file: Path, session): """Process recipe.""" import datetime - import os import shutil + import warnings from ._recipe import read_recipe_file - if not os.path.isfile(recipe_file): + if not recipe_file.is_file(): import errno raise OSError(errno.ENOENT, "Specified recipe file does not exist", recipe_file) @@ -89,14 +89,13 @@ def process_recipe(recipe_file, config_user): logger.info(70 * "-") logger.info("RECIPE = %s", recipe_file) - logger.info("RUNDIR = %s", config_user['run_dir']) - logger.info("WORKDIR = %s", config_user["work_dir"]) - logger.info("PREPROCDIR = %s", config_user["preproc_dir"]) - logger.info("PLOTDIR = %s", config_user["plot_dir"]) + logger.info("RUNDIR = %s", session.run_dir) + logger.info("WORKDIR = %s", session.work_dir) + logger.info("PREPROCDIR = %s", session.preproc_dir) + logger.info("PLOTDIR = %s", session.plot_dir) logger.info(70 * "-") - from multiprocessing import cpu_count - n_processes = config_user['max_parallel_tasks'] or cpu_count() + n_processes = session['max_parallel_tasks'] or os.cpu_count() logger.info("Running tasks using at most %s processes", n_processes) logger.info( @@ -105,7 +104,7 @@ def process_recipe(recipe_file, config_user): logger.info("If you experience memory problems, try reducing " "'max_parallel_tasks' in your user configuration file.") - if config_user['compress_netcdf']: + if session['compress_netcdf']: logger.warning( "You have enabled NetCDF compression. Accessing .nc files can be " "much slower than expected if your access pattern does not match " @@ -115,9 +114,13 @@ def process_recipe(recipe_file, config_user): "NetCDF compression.") # copy recipe to run_dir for future reference - shutil.copy2(recipe_file, config_user['run_dir']) + shutil.copy2(recipe_file, session.run_dir) # parse recipe + with warnings.catch_warnings(): + # ignore deprecation warning + warnings.simplefilter("ignore") + config_user = session.to_config_user() recipe = read_recipe_file(recipe_file, config_user) logger.debug("Recipe summary:\n%s", recipe) # run @@ -139,10 +142,9 @@ class Config(): @staticmethod def _copy_config_file(filename, overwrite, path): - import os import shutil - from ._config import configure_logging + from .config._logging import configure_logging configure_logging(console_log_level='info') if not path: path = os.path.join(os.path.expanduser('~/.esmvaltool'), filename) @@ -214,9 +216,8 @@ def list(): Show all installed recipes, grouped by folder. """ - import os - - from ._config import DIAGNOSTICS, configure_logging + from .config._diagnostics import DIAGNOSTICS + from .config._logging import configure_logging configure_logging(console_log_level='info') recipes_folder = DIAGNOSTICS.recipes logger.info("Showing recipes installed in %s", recipes_folder) @@ -244,7 +245,8 @@ def get(recipe): """ import shutil - from ._config import DIAGNOSTICS, configure_logging + from .config._diagnostics import DIAGNOSTICS + from .config._logging import configure_logging configure_logging(console_log_level='info') installed_recipe = DIAGNOSTICS.recipes / recipe if not installed_recipe.exists(): @@ -266,7 +268,8 @@ def show(recipe): recipe: str Name of the recipe to get, including any subdirectories. """ - from ._config import DIAGNOSTICS, configure_logging + from .config._diagnostics import DIAGNOSTICS + from .config._logging import configure_logging configure_logging(console_log_level='info') installed_recipe = DIAGNOSTICS.recipes / recipe if not installed_recipe.exists(): @@ -276,8 +279,7 @@ def show(recipe): msg = f'Recipe {recipe}' logger.info(msg) logger.info('=' * len(msg)) - with open(installed_recipe) as recipe_file: - print(recipe_file.read()) + print(installed_recipe.read_text(encoding='utf-8')) class ESMValTool(): @@ -296,8 +298,8 @@ class ESMValTool(): """ def __init__(self): - self.recipes = Recipes() self.config = Config() + self.recipes = Recipes() self._extra_packages = {} for entry_point in iter_entry_points('esmvaltool_commands'): self._extra_packages[entry_point.dist.project_name] = \ @@ -326,10 +328,10 @@ def run(self, resume_from=None, max_datasets=None, max_years=None, - skip_nonexistent=False, + skip_nonexistent=None, offline=None, diagnostics=None, - check_level='default', + check_level=None, **kwargs): """Execute an ESMValTool recipe. @@ -367,102 +369,72 @@ def run(self, default (fail if there are any errors), strict (fail if there are any warnings). """ - import os - import warnings - - from ._config import configure_logging, read_config_user_file - from ._recipe import TASKSEP - from .cmor.check import CheckLevels - from .esgf._logon import logon - - # Check validity of optional command line arguments with experimental - # API - with warnings.catch_warnings(): - # ignore experimental API warning - warnings.simplefilter("ignore") - from .experimental.config._config_object import Config as ExpConfig - explicit_optional_kwargs = { - 'config_file': config_file, - 'resume_from': resume_from, - 'max_datasets': max_datasets, - 'max_years': max_years, - 'skip_nonexistent': skip_nonexistent, - 'offline': offline, - 'diagnostics': diagnostics, - 'check_level': check_level, - } - all_optional_kwargs = dict(kwargs) - for (key, val) in explicit_optional_kwargs.items(): - if val is not None: - all_optional_kwargs[key] = val - ExpConfig(all_optional_kwargs) + from .config import CFG recipe = self._get_recipe(recipe) - cfg = read_config_user_file(config_file, recipe.stem, kwargs) + CFG.load_from_file(config_file) + session = CFG.start_session(recipe.stem) + if check_level is not None: + session['check_level'] = check_level + if diagnostics is not None: + session['diagnostics'] = diagnostics + if max_datasets is not None: + session['max_datasets'] = max_datasets + if max_years is not None: + session['max_years'] = max_years + if offline is not None: + session['offline'] = offline + if skip_nonexistent is not None: + session['skip_nonexistent'] = skip_nonexistent + session['resume_from'] = parse_resume(resume_from, recipe) + session.update(kwargs) + + self._run(recipe, session) + + def _run(self, recipe: Path, session) -> None: + """Run `recipe` using `session`.""" # Create run dir - if os.path.exists(cfg['run_dir']): - print("ERROR: run_dir {} already exists, aborting to " - "prevent data loss".format(cfg['output_dir'])) - os.makedirs(cfg['run_dir']) + if session.session_dir.exists(): + print(f"ERROR: output directory {session.session_dir} already" + " exists, aborting to prevent data loss") + session.session_dir.mkdir(parents=True) + session.run_dir.mkdir() # configure logging - log_files = configure_logging(output_dir=cfg['run_dir'], - console_log_level=cfg['log_level']) - - self._log_header(cfg['config_file'], log_files) - - cfg['resume_from'] = parse_resume(resume_from, recipe) - cfg['skip_nonexistent'] = skip_nonexistent - if isinstance(diagnostics, str): - diagnostics = diagnostics.split(' ') - cfg['diagnostics'] = { - pattern if TASKSEP in pattern else pattern + TASKSEP + '*' - for pattern in diagnostics or () - } - cfg['check_level'] = CheckLevels[check_level.upper()] - if offline is not None: - # Override config-user.yml from command line - cfg['offline'] = offline - if not cfg['offline']: - logon() - - def _check_limit(limit, value): - if value is not None and value < 1: - raise ValueError("--{} should be larger than 0.".format( - limit.replace('_', '-'))) - if value: - cfg[limit] = value + from .config._logging import configure_logging + log_files = configure_logging(output_dir=session.run_dir, + console_log_level=session['log_level']) + self._log_header(session['config_file'], log_files) - _check_limit('max_datasets', max_datasets) - _check_limit('max_years', max_years) + if not session['offline']: + from .esgf._logon import logon + logon() - resource_log = os.path.join(cfg['run_dir'], 'resource_usage.txt') + # configure resource logger and run program from ._task import resource_usage_logger + resource_log = session.run_dir / 'resource_usage.txt' with resource_usage_logger(pid=os.getpid(), filename=resource_log): - process_recipe(recipe_file=recipe, config_user=cfg) + process_recipe(recipe_file=recipe, session=session) - self._clean_preproc(cfg) + self._clean_preproc(session) logger.info("Run was successful") @staticmethod - def _clean_preproc(cfg): - import os + def _clean_preproc(session): import shutil - if os.path.exists(cfg["preproc_dir"]) and cfg["remove_preproc_dir"]: + if session["remove_preproc_dir"] and session.preproc_dir.exists(): logger.info("Removing preproc containing preprocessed data") logger.info("If this data is further needed, then") logger.info("set remove_preproc_dir to false in config-user.yml") - shutil.rmtree(cfg["preproc_dir"]) + shutil.rmtree(session.preproc_dir) @staticmethod - def _get_recipe(recipe): - import os - - from esmvalcore._config import DIAGNOSTICS + def _get_recipe(recipe) -> Path: + from esmvalcore.config._diagnostics import DIAGNOSTICS if not os.path.isfile(recipe): - installed_recipe = str(DIAGNOSTICS.recipes / recipe) + installed_recipe = DIAGNOSTICS.recipes / recipe if os.path.isfile(installed_recipe): recipe = installed_recipe recipe = Path(os.path.expandvars(recipe)).expanduser().absolute() diff --git a/esmvalcore/_recipe.py b/esmvalcore/_recipe.py index 5cb1414f7f..66c1285c99 100644 --- a/esmvalcore/_recipe.py +++ b/esmvalcore/_recipe.py @@ -16,13 +16,6 @@ from . import __version__ from . import _recipe_checks as check from . import esgf -from ._config import ( - TAGS, - get_activity, - get_extra_facets, - get_institutes, - get_project_config, -) from ._data_finder import ( _find_input_files, _get_timerange_from_years, @@ -38,6 +31,13 @@ from ._task import DiagnosticTask, ResumeTask, TaskSet from .cmor.check import CheckLevels from .cmor.table import CMOR_TABLES +from .config._config import ( + get_activity, + get_extra_facets, + get_institutes, + get_project_config, +) +from .config._diagnostics import TAGS from .exceptions import InputFilesNotFound, RecipeError from .preprocessor import ( DEFAULT_ORDER, diff --git a/esmvalcore/_task.py b/esmvalcore/_task.py index cf2ddda20b..7fc7bf893b 100644 --- a/esmvalcore/_task.py +++ b/esmvalcore/_task.py @@ -13,17 +13,15 @@ import time from copy import deepcopy from multiprocessing import Pool -from multiprocessing.pool import ApplyResult from pathlib import Path, PosixPath from shutil import which -from typing import Dict, Type import psutil import yaml from ._citation import _write_citation_files -from ._config import DIAGNOSTICS, TAGS from ._provenance import TrackedFile, get_task_provenance +from .config._diagnostics import DIAGNOSTICS, TAGS def path_representer(dumper, data): @@ -145,7 +143,7 @@ def _py2ncl(value, var_name=''): txt = var_name + ' = ' if var_name else '' if value is None: txt += '_Missing' - elif isinstance(value, str): + elif isinstance(value, (str, Path)): txt += '"{}"'.format(value) elif isinstance(value, (list, tuple)): if not value: @@ -632,7 +630,7 @@ def _collect_provenance(self): attrs[key] = self.settings[key] ancestor_products = { - p.filename: p + str(p.filename): p for a in self.ancestors for p in a.products } @@ -736,7 +734,7 @@ def _run_sequential(self) -> None: def _run_parallel(self, max_parallel_tasks=None): """Run tasks in parallel.""" scheduled = self.flatten() - running: Dict[Type[BaseTask], Type[ApplyResult]] = {} + running = {} n_tasks = n_scheduled = len(scheduled) n_running = 0 diff --git a/esmvalcore/cmor/table.py b/esmvalcore/cmor/table.py index 60a863dd1a..e6a76c0767 100644 --- a/esmvalcore/cmor/table.py +++ b/esmvalcore/cmor/table.py @@ -9,16 +9,22 @@ import json import logging import os +import tempfile +import warnings from collections import Counter -from functools import total_ordering +from functools import lru_cache, total_ordering from pathlib import Path -from typing import Dict, Type +from typing import Optional, Union import yaml +from esmvalcore.exceptions import ESMValCoreDeprecationWarning + logger = logging.getLogger(__name__) -CMOR_TABLES: Dict[str, Type['InfoBase']] = {} +CMORTable = Union['CMIP3Info', 'CMIP5Info', 'CMIP6Info', 'CustomInfo'] + +CMOR_TABLES: dict[str, CMORTable] = {} """dict of str, obj: CMOR info objects.""" @@ -37,25 +43,67 @@ def get_var_info(project, mip, short_name): return CMOR_TABLES[project].get_variable(mip, short_name) -def read_cmor_tables(cfg_developer=None): +def read_cmor_tables(cfg_developer: Optional[Path] = None) -> None: """Read cmor tables required in the configuration. Parameters ---------- - cfg_developer : dict of str - Parsed config-developer file + cfg_developer: + Path to config-developer.yml file. + + Prior to v2.8.0 `cfg_developer` was an :obj:`dict` with the contents + of config-developer.yml. This is deprecated and support will be + removed in v2.10.0. """ - if cfg_developer is None: + if isinstance(cfg_developer, dict): + warnings.warn( + "Using the `read_cmor_tables` file with a dictionary as argument " + "has been deprecated in ESMValCore version 2.8.0 and is " + "scheduled for removal in version 2.10.0. " + "Please use the path to the config-developer.yml file instead.", + ESMValCoreDeprecationWarning, + ) + with tempfile.NamedTemporaryFile( + mode='w', + encoding='utf-8', + delete=False, + ) as file: + yaml.safe_dump(cfg_developer, file) + cfg_file = Path(file.name) + else: + cfg_file = cfg_developer + if cfg_file is None: cfg_file = Path(__file__).parents[1] / 'config-developer.yml' - with cfg_file.open() as file: - cfg_developer = yaml.safe_load(file) + mtime = cfg_file.stat().st_mtime + cmor_tables = _read_cmor_tables(cfg_file, mtime) + if isinstance(cfg_developer, dict): + # clean up the temporary file + cfg_file.unlink() + CMOR_TABLES.clear() + CMOR_TABLES.update(cmor_tables) + +@lru_cache +def _read_cmor_tables(cfg_file: Path, mtime: float) -> dict[str, CMORTable]: + """Read cmor tables required in the configuration. + + Parameters + ---------- + cfg_file: pathlib.Path + Path to config-developer.yml file. + mtime: float + Modification time of config-developer.yml file. Only used by the + `lru_cache` decorator to make sure the file is read again when it + is changed. + """ + with cfg_file.open('r', encoding='utf-8') as file: + cfg_developer = yaml.safe_load(file) cwd = os.path.dirname(os.path.realpath(__file__)) var_alt_names_file = os.path.join(cwd, 'variable_alt_names.yml') with open(var_alt_names_file, 'r') as yfile: alt_names = yaml.safe_load(yfile) - CMOR_TABLES.clear() + cmor_tables: dict[str, CMORTable] = {} # Try to infer location for custom tables from config-developer.yml file, # if not possible, use default location @@ -65,14 +113,15 @@ def read_cmor_tables(cfg_developer=None): if custom_path is not None: custom_path = os.path.expandvars(os.path.expanduser(custom_path)) custom = CustomInfo(custom_path) - CMOR_TABLES['custom'] = custom + cmor_tables['custom'] = custom install_dir = os.path.dirname(os.path.realpath(__file__)) for table in cfg_developer: if table == 'custom': continue - CMOR_TABLES[table] = _read_table(cfg_developer, table, install_dir, + cmor_tables[table] = _read_table(cfg_developer, table, install_dir, custom, alt_names) + return cmor_tables def _read_table(cfg_developer, table, install_dir, custom, alt_names): @@ -919,4 +968,5 @@ def _read_table_file(self, table_file, table=None): return +# Load the default tables on initializing the module. read_cmor_tables() diff --git a/esmvalcore/config/__init__.py b/esmvalcore/config/__init__.py new file mode 100644 index 0000000000..74bcc486df --- /dev/null +++ b/esmvalcore/config/__init__.py @@ -0,0 +1,17 @@ +"""Configuration module. + +.. data:: CFG + + ESMValCore configuration. + + By default this will be loaded from the file + ``~/.esmvaltool/config-user.yml``. +""" + +from ._config_object import CFG, Config, Session + +__all__ = ( + 'CFG', + 'Config', + 'Session', +) diff --git a/esmvalcore/config/_config.py b/esmvalcore/config/_config.py new file mode 100644 index 0000000000..ae7441197d --- /dev/null +++ b/esmvalcore/config/_config.py @@ -0,0 +1,130 @@ +"""Functions dealing with config-user.yml / config-developer.yml.""" +import collections.abc +import fnmatch +import logging +import os +import sys +from functools import lru_cache +from pathlib import Path + +import yaml + +from esmvalcore.cmor.table import CMOR_TABLES, read_cmor_tables +from esmvalcore.exceptions import RecipeError + +logger = logging.getLogger(__name__) + +TASKSEP = os.sep + +CFG = {} + +if sys.version_info[:2] >= (3, 9): + # pylint: disable=no-name-in-module + from importlib.resources import files as importlib_files +else: + from importlib_resources import files as importlib_files + + +def _deep_update(dictionary, update): + for key, value in update.items(): + if isinstance(value, collections.abc.Mapping): + dictionary[key] = _deep_update(dictionary.get(key, {}), value) + else: + dictionary[key] = value + return dictionary + + +@lru_cache() +def _load_extra_facets(project, extra_facets_dir): + config = {} + config_paths = [ + importlib_files("esmvalcore.config") / "extra_facets", + Path.home() / ".esmvaltool" / "extra_facets", + ] + config_paths.extend([Path(p) for p in extra_facets_dir]) + for config_path in config_paths: + config_file_paths = config_path.glob(f"{project.lower()}-*.yml") + for config_file_path in sorted(config_file_paths): + logger.debug("Loading extra facets from %s", config_file_path) + with config_file_path.open() as config_file: + config_piece = yaml.safe_load(config_file) + if config_piece: + _deep_update(config, config_piece) + return config + + +def get_extra_facets(project, dataset, mip, short_name, extra_facets_dir): + """Read configuration files with additional variable information.""" + project_details = _load_extra_facets(project, extra_facets_dir) + + def pattern_filter(patterns, name): + """Get the subset of the list `patterns` that `name` matches. + + Parameters + ---------- + patterns : :obj:`list` of :obj:`str` + A list of strings that may contain shell-style wildcards. + name : str + A string describing the dataset, mip, or short_name. + + Returns + ------- + :obj:`list` of :obj:`str` + The subset of patterns that `name` matches. + """ + return [pat for pat in patterns if fnmatch.fnmatchcase(name, pat)] + + extra_facets = {} + for dataset_ in pattern_filter(project_details, dataset): + for mip_ in pattern_filter(project_details[dataset_], mip): + for var in pattern_filter(project_details[dataset_][mip_], + short_name): + facets = project_details[dataset_][mip_][var] + extra_facets.update(facets) + + return extra_facets + + +def load_config_developer(cfg_file): + """Read the developer's configuration file.""" + with open(cfg_file, 'r', encoding='utf-8') as file: + cfg = yaml.safe_load(file) + + if 'obs4mips' in cfg: + logger.warning( + "Correcting capitalization, project 'obs4mips'" + " should be written as 'obs4MIPs' in %s", cfg_file) + cfg['obs4MIPs'] = cfg.pop('obs4mips') + + for project, settings in cfg.items(): + CFG[project] = settings + read_cmor_tables(cfg_file) + + +def get_project_config(project): + """Get developer-configuration for project.""" + if project in CFG: + return CFG[project] + raise RecipeError(f"Project '{project}' not in config-developer.yml") + + +def get_institutes(variable): + """Return the institutes given the dataset name in CMIP6.""" + dataset = variable['dataset'] + project = variable['project'] + try: + return CMOR_TABLES[project].institutes[dataset] + except (KeyError, AttributeError): + return [] + + +def get_activity(variable): + """Return the activity given the experiment name in CMIP6.""" + project = variable['project'] + try: + exp = variable['exp'] + if isinstance(exp, list): + return [CMOR_TABLES[project].activities[value][0] for value in exp] + return CMOR_TABLES[project].activities[exp][0] + except (KeyError, AttributeError): + return None diff --git a/esmvalcore/experimental/config/_config_object.py b/esmvalcore/config/_config_object.py similarity index 82% rename from esmvalcore/experimental/config/_config_object.py rename to esmvalcore/config/_config_object.py index 627b75f636..52ff2d5019 100644 --- a/esmvalcore/experimental/config/_config_object.py +++ b/esmvalcore/config/_config_object.py @@ -1,13 +1,17 @@ """Importable config object.""" import os +import warnings from datetime import datetime from pathlib import Path -from typing import Union +from types import MappingProxyType +from typing import Optional, Union import yaml import esmvalcore +from esmvalcore.cmor.check import CheckLevels +from esmvalcore.exceptions import ESMValCoreDeprecationWarning from ._config_validators import _validators from ._validated_config import ValidatedConfig @@ -20,7 +24,7 @@ class Config(ValidatedConfig): """ESMValTool configuration object. Do not instantiate this class directly, but use - :obj:`esmvalcore.experimental.CFG` instead. + :obj:`esmvalcore.config.CFG` instead. """ _validate = _validators @@ -71,15 +75,27 @@ def _load_default_config(cls, filename: Union[os.PathLike, str]): mapping = _read_config_file(filename) # Add defaults that are not available in esmvalcore/config-user.yml + mapping['config_file'] = filename + mapping['diagnostics'] = None mapping['extra_facets_dir'] = tuple() mapping['resume_from'] = [] + mapping['check_level'] = CheckLevels.DEFAULT + mapping['max_datasets'] = None + mapping['max_years'] = None + mapping['run_diagnostic'] = True + mapping['skip_nonexistent'] = False new.update(mapping) return new - def load_from_file(self, filename: Union[os.PathLike, str]): + def load_from_file( + self, + filename: Optional[Union[os.PathLike, str]] = None, + ) -> None: """Load user configuration from the given file.""" + if filename is None: + filename = USER_CONFIG path = Path(filename).expanduser() if not path.exists(): try_path = USER_CONFIG_DIR / filename @@ -193,9 +209,17 @@ def main_log_debug(self): def to_config_user(self) -> dict: """Turn the `Session` object into a recipe-compatible dict. + Deprecated since v2.8.0, scheduled for removal in v2.9.0. + This dict is compatible with the `config-user` argument in :obj:`esmvalcore._recipe.Recipe`. """ + warnings.warn( + "The `esmvalcore.[experimental.]config.Session.to_config_user` " + "method has been deprecated in ESMValCore version 2.8.0 and is " + "scheduled for removal in version 2.9.0. ", + ESMValCoreDeprecationWarning, + ) dct = self.copy() dct['run_dir'] = self.run_dir dct['work_dir'] = self.work_dir @@ -208,8 +232,16 @@ def to_config_user(self) -> dict: def from_config_user(cls, config_user: dict) -> 'Session': """Convert `config-user` dict to API-compatible `Session` object. + Deprecated since v2.8.0, scheduled for removal in v2.9.0. + For example, `_recipe.Recipe._cfg`. """ + warnings.warn( + "The `esmvalcore.[experimental.]config.Session.from_config_user` " + "method has been deprecated in ESMValCore version 2.8.0 and is " + "scheduled for removal in version 2.9.0. ", + ESMValCoreDeprecationWarning, + ) dct = config_user.copy() dct.pop('run_dir') dct.pop('work_dir') @@ -246,5 +278,5 @@ def _read_config_file(config_file): USER_CONFIG = USER_CONFIG_DIR / 'config-user.yml' # initialize placeholders -CFG_DEFAULT = Config._load_default_config(DEFAULT_CONFIG) +CFG_DEFAULT = MappingProxyType(Config._load_default_config(DEFAULT_CONFIG)) CFG = Config._load_user_config(USER_CONFIG, raise_exception=False) diff --git a/esmvalcore/experimental/config/_config_validators.py b/esmvalcore/config/_config_validators.py similarity index 86% rename from esmvalcore/experimental/config/_config_validators.py rename to esmvalcore/config/_config_validators.py index c87411ba99..26990c49e7 100644 --- a/esmvalcore/experimental/config/_config_validators.py +++ b/esmvalcore/config/_config_validators.py @@ -1,25 +1,28 @@ """List of config validators.""" - +import logging +import os.path import warnings from collections.abc import Iterable from functools import lru_cache from pathlib import Path +from typing import Optional, Union from esmvalcore import __version__ as current_version -from esmvalcore._config import load_config_developer -from esmvalcore._recipe import TASKSEP from esmvalcore.cmor.check import CheckLevels +from esmvalcore.config._config import ( + TASKSEP, + importlib_files, + load_config_developer, +) +from esmvalcore.exceptions import ESMValCoreDeprecationWarning + +logger = logging.getLogger(__name__) class ValidationError(ValueError): """Custom validation error.""" -# Custom warning, because DeprecationWarning is hidden by default -class ESMValToolDeprecationWarning(UserWarning): - """Configuration key has been deprecated.""" - - # The code for this function was taken from matplotlib (v3.3) and modified # to fit the needs of ESMValCore. Matplotlib is licenced under the terms of # the the 'Python Software Foundation License' @@ -123,7 +126,7 @@ def validate_path(value, allow_none=False): if (value is None) and allow_none: return value try: - path = Path(value).expanduser().absolute() + path = Path(os.path.expandvars(value)).expanduser().absolute() except TypeError as err: raise ValidationError(f"Expected a path, but got {value}") from err else: @@ -173,25 +176,39 @@ def chained(value): allow_none=True) -def validate_oldstyle_rootpath(value): +def validate_rootpath(value): """Validate `rootpath` mapping.""" mapping = validate_dict(value) new_mapping = {} for key, paths in mapping.items(): + if key == 'obs4mips': + logger.warning( + "Correcting capitalization, project 'obs4mips' should be " + "written as 'obs4MIPs' in 'rootpath' in config-user.yml") + key = 'obs4MIPs' new_mapping[key] = validate_pathlist(paths) return new_mapping -def validate_oldstyle_drs(value): +def validate_drs(value): """Validate `drs` mapping.""" mapping = validate_dict(value) - return mapping + new_mapping = {} + for key, drs in mapping.items(): + if key == 'obs4mips': + logger.warning( + "Correcting capitalization, project 'obs4mips' should be " + "written as 'obs4MIPs' in 'drs' in config-user.yml") + key = 'obs4MIPs' + new_mapping[key] = validate_string(drs) + return new_mapping def validate_config_developer(value): """Validate and load config developer path.""" path = validate_path_or_none(value) - + if path is None: + path = importlib_files('esmvalcore') / 'config-developer.yml' load_config_developer(path) return path @@ -212,8 +229,12 @@ def validate_check_level(value): return value -def validate_diagnostics(diagnostics): +def validate_diagnostics( + diagnostics: Union[Iterable[str], str, None], +) -> Optional[set[str]]: """Validate diagnostic location.""" + if diagnostics is None: + return None if isinstance(diagnostics, str): diagnostics = diagnostics.strip().split(' ') return { @@ -242,10 +263,10 @@ def deprecate(func, variable, version: str = None): if current_version >= version: warnings.warn(f"`{variable}` has been removed in {version}", - ESMValToolDeprecationWarning) + ESMValCoreDeprecationWarning) else: warnings.warn(f"`{variable}` will be removed in {version}.", - ESMValToolDeprecationWarning, + ESMValCoreDeprecationWarning, stacklevel=2) return func @@ -267,13 +288,13 @@ def deprecate(func, variable, version: str = None): 'profile_diagnostic': validate_bool, 'run_diagnostic': validate_bool, 'output_file_type': validate_string, + "offline": validate_bool, # From CLI "resume_from": validate_pathlist, "skip_nonexistent": validate_bool, "diagnostics": validate_diagnostics, "check_level": validate_check_level, - "offline": validate_bool, 'max_years': validate_int_positive_or_none, 'max_datasets': validate_int_positive_or_none, @@ -281,8 +302,8 @@ def deprecate(func, variable, version: str = None): 'write_ncl_interface': validate_bool, # oldstyle - 'rootpath': validate_oldstyle_rootpath, - 'drs': validate_oldstyle_drs, + 'rootpath': validate_rootpath, + 'drs': validate_drs, # config location 'config_file': validate_path, diff --git a/esmvalcore/_config/_diagnostics.py b/esmvalcore/config/_diagnostics.py similarity index 100% rename from esmvalcore/_config/_diagnostics.py rename to esmvalcore/config/_diagnostics.py diff --git a/esmvalcore/_config/_esgf_pyclient.py b/esmvalcore/config/_esgf_pyclient.py similarity index 98% rename from esmvalcore/_config/_esgf_pyclient.py rename to esmvalcore/config/_esgf_pyclient.py index 94a1abe7bb..8f0af74235 100644 --- a/esmvalcore/_config/_esgf_pyclient.py +++ b/esmvalcore/config/_esgf_pyclient.py @@ -20,8 +20,6 @@ import yaml -from ._config import _normalize_path - keyring: Optional[ModuleType] = None try: keyring = importlib.import_module('keyring') @@ -169,7 +167,8 @@ def load_esgf_pyclient_config(): cfg[section].update(file_cfg.get(section, {})) if 'cache' in cfg['search_connection']: - cache_file = _normalize_path(cfg['search_connection']['cache']) + cache_file = Path(os.path.expandvars( + cfg['search_connection']['cache'])).expanduser().absolute() cfg['search_connection']['cache'] = cache_file Path(cache_file).parent.mkdir(parents=True, exist_ok=True) diff --git a/esmvalcore/_config/_logging.py b/esmvalcore/config/_logging.py similarity index 100% rename from esmvalcore/_config/_logging.py rename to esmvalcore/config/_logging.py diff --git a/esmvalcore/experimental/config/_validated_config.py b/esmvalcore/config/_validated_config.py similarity index 91% rename from esmvalcore/experimental/config/_validated_config.py rename to esmvalcore/config/_validated_config.py index bed3f5e3c5..3f508e7420 100644 --- a/esmvalcore/experimental/config/_validated_config.py +++ b/esmvalcore/config/_validated_config.py @@ -5,16 +5,12 @@ from collections.abc import MutableMapping from typing import Callable, Dict, Tuple -from .._exceptions import SuppressedError -from ._config_validators import ValidationError - - -class InvalidConfigParameter(SuppressedError): - """Config parameter is invalid.""" +from esmvalcore.exceptions import ( + InvalidConfigParameter, + MissingConfigParameter, +) - -class MissingConfigParameter(UserWarning): - """Config parameter is missing.""" +from ._config_validators import ValidationError # The code for this class was take from matplotlib (v3.3) and modified to @@ -89,4 +85,4 @@ def copy(self): def clear(self): """Clear Config.""" - self._mapping.clear(self) + self._mapping.clear() diff --git a/esmvalcore/config/config-logging.yml b/esmvalcore/config/config-logging.yml new file mode 100644 index 0000000000..b78b308ed8 --- /dev/null +++ b/esmvalcore/config/config-logging.yml @@ -0,0 +1,33 @@ +# Logger configuration +--- + +version: 1 +disable_existing_loggers: false +formatters: + console: + format: '%(asctime)s UTC [%(process)d] %(levelname)-7s %(message)s' + brief: + format: '%(levelname)-7s [%(process)d] %(message)s' + debug: + format: '%(asctime)s UTC [%(process)d] %(levelname)-7s %(name)s:%(lineno)s %(message)s' +handlers: + console: + class: logging.StreamHandler + level: INFO + formatter: console + stream: ext://sys.stdout + simple_log_file: + class: logging.FileHandler + level: INFO + formatter: brief + filename: main_log.txt + mode: w + debug_log_file: + class: logging.FileHandler + level: DEBUG + formatter: debug + filename: main_log_debug.txt + mode: w +root: + level: DEBUG + handlers: [console, simple_log_file, debug_log_file] diff --git a/esmvalcore/_config/extra_facets/cesm-mappings.yml b/esmvalcore/config/extra_facets/cesm-mappings.yml similarity index 100% rename from esmvalcore/_config/extra_facets/cesm-mappings.yml rename to esmvalcore/config/extra_facets/cesm-mappings.yml diff --git a/esmvalcore/_config/extra_facets/cmip3-institutes.yml b/esmvalcore/config/extra_facets/cmip3-institutes.yml similarity index 100% rename from esmvalcore/_config/extra_facets/cmip3-institutes.yml rename to esmvalcore/config/extra_facets/cmip3-institutes.yml diff --git a/esmvalcore/config/extra_facets/cmip5-fx.yml b/esmvalcore/config/extra_facets/cmip5-fx.yml new file mode 100644 index 0000000000..41d715f996 --- /dev/null +++ b/esmvalcore/config/extra_facets/cmip5-fx.yml @@ -0,0 +1,5 @@ +--- +'*': + 'fx': + '*': + ensemble: r0i0p0 diff --git a/esmvalcore/_config/extra_facets/cmip5-institutes.yml b/esmvalcore/config/extra_facets/cmip5-institutes.yml similarity index 100% rename from esmvalcore/_config/extra_facets/cmip5-institutes.yml rename to esmvalcore/config/extra_facets/cmip5-institutes.yml diff --git a/esmvalcore/_config/extra_facets/cmip5-product.yml b/esmvalcore/config/extra_facets/cmip5-product.yml similarity index 100% rename from esmvalcore/_config/extra_facets/cmip5-product.yml rename to esmvalcore/config/extra_facets/cmip5-product.yml diff --git a/esmvalcore/_config/extra_facets/emac-mappings.yml b/esmvalcore/config/extra_facets/emac-mappings.yml similarity index 100% rename from esmvalcore/_config/extra_facets/emac-mappings.yml rename to esmvalcore/config/extra_facets/emac-mappings.yml diff --git a/esmvalcore/_config/extra_facets/icon-mappings.yml b/esmvalcore/config/extra_facets/icon-mappings.yml similarity index 100% rename from esmvalcore/_config/extra_facets/icon-mappings.yml rename to esmvalcore/config/extra_facets/icon-mappings.yml diff --git a/esmvalcore/_config/extra_facets/ipslcm-mappings.yml b/esmvalcore/config/extra_facets/ipslcm-mappings.yml similarity index 100% rename from esmvalcore/_config/extra_facets/ipslcm-mappings.yml rename to esmvalcore/config/extra_facets/ipslcm-mappings.yml diff --git a/esmvalcore/esgf/_logon.py b/esmvalcore/esgf/_logon.py index 17c1d1c9d0..e9c33251c7 100644 --- a/esmvalcore/esgf/_logon.py +++ b/esmvalcore/esgf/_logon.py @@ -5,7 +5,7 @@ import pyesgf.logon import pyesgf.search -from .._config._esgf_pyclient import get_esgf_config +from ..config._esgf_pyclient import get_esgf_config logger = logging.getLogger(__name__) diff --git a/esmvalcore/esgf/_search.py b/esmvalcore/esgf/_search.py index 9dc831f992..09723ef9ee 100644 --- a/esmvalcore/esgf/_search.py +++ b/esmvalcore/esgf/_search.py @@ -6,13 +6,13 @@ import pyesgf.search import requests.exceptions -from .._config._esgf_pyclient import get_esgf_config from .._data_finder import ( _get_timerange_from_years, _parse_period, _truncate_dates, get_start_end_date, ) +from ..config._esgf_pyclient import get_esgf_config from ._download import ESGFFile from .facets import DATASET_MAP, FACETS diff --git a/esmvalcore/esgf/facets.py b/esmvalcore/esgf/facets.py index c9ee994729..9a071b6c97 100644 --- a/esmvalcore/esgf/facets.py +++ b/esmvalcore/esgf/facets.py @@ -2,7 +2,7 @@ import pyesgf.search -from .._config._esgf_pyclient import get_esgf_config +from ..config._esgf_pyclient import get_esgf_config FACETS = { 'CMIP3': { diff --git a/esmvalcore/exceptions.py b/esmvalcore/exceptions.py index 6013d29292..f7d775a032 100644 --- a/esmvalcore/exceptions.py +++ b/esmvalcore/exceptions.py @@ -1,7 +1,41 @@ """Exceptions that may be raised by ESMValCore.""" +import sys -class RecipeError(Exception): +class Error(Exception): + """Base class from which other exceptions are derived.""" + + +class SuppressedError(Exception): + """Errors subclassed from SuppressedError hide the full traceback. + + This can be used for simple user-facing errors that do not need the + full traceback. + """ + + +def _suppressed_hook(error, message, traceback): + """https://stackoverflow.com/a/27674608.""" + if issubclass(error, SuppressedError): + # Print only the message and hide the traceback + print(f'{error.__name__}: {message}', file=sys.stderr) + else: + # Print full traceback + sys.__excepthook__(error, message, traceback) + + +sys.excepthook = _suppressed_hook + + +class InvalidConfigParameter(Error, SuppressedError): + """Config parameter is invalid.""" + + +class MissingConfigParameter(UserWarning): + """Config parameter is missing.""" + + +class RecipeError(Error): """Recipe contains an error.""" def __init__(self, msg): diff --git a/esmvalcore/experimental/_exceptions.py b/esmvalcore/experimental/_exceptions.py deleted file mode 100644 index 437ed124cc..0000000000 --- a/esmvalcore/experimental/_exceptions.py +++ /dev/null @@ -1,24 +0,0 @@ -"""ESMValCore exceptions.""" - -import sys - - -class SuppressedError(Exception): - """Errors subclassed from SuppressedError hide the full traceback. - - This can be used for simple user-facing errors that do not need the - full traceback. - """ - - -def _suppressed_hook(error, message, traceback): - """https://stackoverflow.com/a/27674608.""" - if issubclass(error, SuppressedError): - # Print only the message and hide the traceback - print(f'{error.__name__}: {message}'.format(error.__name__, message)) - else: - # Print full traceback - sys.__excepthook__(error, message, traceback) - - -sys.excepthook = _suppressed_hook diff --git a/esmvalcore/experimental/config/__init__.py b/esmvalcore/experimental/config/__init__.py index 8ac5313cad..99ad18efda 100644 --- a/esmvalcore/experimental/config/__init__.py +++ b/esmvalcore/experimental/config/__init__.py @@ -1,13 +1,23 @@ -"""ESMValTool config module. +"""Configuration module. .. data:: CFG ESMValCore configuration. - By default this will loaded from the file ~/.esmvaltool/config-user.yml. + By default this will be loaded from the file + ``~/.esmvaltool/config-user.yml``. """ +import warnings -from ._config_object import CFG, Config, Session +from esmvalcore.config import CFG, Config, Session +from esmvalcore.exceptions import ESMValCoreDeprecationWarning + +warnings.warn( + "The module `esmvalcore.experimental.config` has been deprecated in " + "ESMValCore version 2.8.0 and is scheduled for removal in version 2.9.0. " + "Please use the module `esmvalcore.config` instead.", + ESMValCoreDeprecationWarning, +) __all__ = [ 'CFG', diff --git a/esmvalcore/experimental/recipe_metadata.py b/esmvalcore/experimental/recipe_metadata.py index da3a66e9c1..e8842a5fe2 100644 --- a/esmvalcore/experimental/recipe_metadata.py +++ b/esmvalcore/experimental/recipe_metadata.py @@ -3,7 +3,7 @@ import pybtex from pybtex.database.input import bibtex -from esmvalcore._config import DIAGNOSTICS, TAGS +from esmvalcore.config._diagnostics import DIAGNOSTICS, TAGS class RenderError(BaseException): diff --git a/esmvalcore/experimental/recipe_output.py b/esmvalcore/experimental/recipe_output.py index 6b4f7e3beb..7ed357595b 100644 --- a/esmvalcore/experimental/recipe_output.py +++ b/esmvalcore/experimental/recipe_output.py @@ -1,13 +1,14 @@ """API for handing recipe output.""" import base64 import logging -import os +import warnings from collections.abc import Mapping from pathlib import Path from typing import Optional, Tuple, Type import iris +from ..config._config import TASKSEP from .config import Session from .recipe_info import RecipeInfo from .recipe_metadata import Contributor, Reference @@ -15,8 +16,6 @@ logger = logging.getLogger(__name__) -TASKSEP = os.sep - class TaskOutput: """Container for task output. @@ -190,7 +189,10 @@ def from_core_recipe_output(cls, recipe_output: dict): recipe_config = recipe_output['recipe_config'] recipe_filename = recipe_output['recipe_filename'] - session = Session.from_config_user(recipe_config) + with warnings.catch_warnings(): + # ignore deprecation warning + warnings.simplefilter("ignore") + session = Session.from_config_user(recipe_config) info = RecipeInfo(recipe_data, filename=recipe_filename) info.resolve() diff --git a/esmvalcore/experimental/utils.py b/esmvalcore/experimental/utils.py index a28e163267..2926053dee 100644 --- a/esmvalcore/experimental/utils.py +++ b/esmvalcore/experimental/utils.py @@ -5,7 +5,7 @@ from pathlib import Path from typing import Pattern, Tuple, Union -from esmvalcore._config import DIAGNOSTICS +from esmvalcore.config._diagnostics import DIAGNOSTICS from .recipe import Recipe diff --git a/setup.cfg b/setup.cfg index 5550bd7fbc..668bd364e9 100644 --- a/setup.cfg +++ b/setup.cfg @@ -34,6 +34,7 @@ include_trailing_comma = true [mypy] ignore_missing_imports = True +implicit_optional = True files = esmvalcore, tests [yapf] diff --git a/tests/integration/cmor/_fixes/cesm/test_cesm2.py b/tests/integration/cmor/_fixes/cesm/test_cesm2.py index 111fd99376..cd6f5d3cd0 100644 --- a/tests/integration/cmor/_fixes/cesm/test_cesm2.py +++ b/tests/integration/cmor/_fixes/cesm/test_cesm2.py @@ -7,9 +7,9 @@ from iris.cube import Cube, CubeList import esmvalcore.cmor._fixes.cesm.cesm2 -from esmvalcore._config import get_extra_facets from esmvalcore.cmor.fix import Fix from esmvalcore.cmor.table import get_var_info +from esmvalcore.config._config import get_extra_facets # Note: test_data_path is defined in tests/integration/cmor/_fixes/conftest.py diff --git a/tests/integration/cmor/_fixes/emac/test_emac.py b/tests/integration/cmor/_fixes/emac/test_emac.py index 2a72f9c38b..201b8cd8ef 100644 --- a/tests/integration/cmor/_fixes/emac/test_emac.py +++ b/tests/integration/cmor/_fixes/emac/test_emac.py @@ -9,7 +9,6 @@ from iris.coords import AuxCoord, DimCoord from iris.cube import Cube, CubeList -from esmvalcore._config import get_extra_facets from esmvalcore.cmor._fixes.emac.emac import ( AllVars, Cl, @@ -43,6 +42,7 @@ ) from esmvalcore.cmor.fix import Fix from esmvalcore.cmor.table import get_var_info +from esmvalcore.config._config import get_extra_facets @pytest.fixture diff --git a/tests/integration/cmor/_fixes/icon/test_icon.py b/tests/integration/cmor/_fixes/icon/test_icon.py index f2ca6a6351..22699aec8f 100644 --- a/tests/integration/cmor/_fixes/icon/test_icon.py +++ b/tests/integration/cmor/_fixes/icon/test_icon.py @@ -9,10 +9,10 @@ from iris.coords import AuxCoord, DimCoord from iris.cube import Cube, CubeList -from esmvalcore._config import get_extra_facets from esmvalcore.cmor._fixes.icon.icon import AllVars, Siconc, Siconca from esmvalcore.cmor.fix import Fix from esmvalcore.cmor.table import get_var_info +from esmvalcore.config._config import get_extra_facets # Note: test_data_path is defined in tests/integration/cmor/_fixes/conftest.py diff --git a/tests/integration/cmor/test_read_cmor_tables.py b/tests/integration/cmor/test_read_cmor_tables.py index 8c4169cbc8..facb94f96b 100644 --- a/tests/integration/cmor/test_read_cmor_tables.py +++ b/tests/integration/cmor/test_read_cmor_tables.py @@ -1,6 +1,8 @@ from pathlib import Path -from esmvalcore._config import read_config_developer_file +import pytest +import yaml + from esmvalcore.cmor.table import CMOR_TABLES from esmvalcore.cmor.table import __file__ as root from esmvalcore.cmor.table import read_cmor_tables @@ -19,9 +21,6 @@ def test_read_cmor_tables(): """Test that the function `read_cmor_tables` loads the tables correctly.""" - # Read the tables - read_cmor_tables(read_config_developer_file()) - table_path = Path(root).parent / 'tables' for project in 'CMIP5', 'CMIP6': @@ -46,9 +45,17 @@ def test_read_cmor_tables(): assert table.strict is False -def test_read_custom_cmor_tables(): +@pytest.mark.parametrize('behaviour', ['current', 'deprecated']) +def test_read_custom_cmor_tables(tmp_path, behaviour): """Test reading of custom CMOR tables.""" - read_cmor_tables(CUSTOM_CFG_DEVELOPER) + cfg_file = tmp_path / 'config-developer.yml' + if behaviour == 'deprecated': + cfg_file = CUSTOM_CFG_DEVELOPER + else: + with cfg_file.open('w', encoding='utf-8') as file: + yaml.safe_dump(CUSTOM_CFG_DEVELOPER, file) + + read_cmor_tables(cfg_file) assert len(CMOR_TABLES) == 2 assert 'CMIP6' in CMOR_TABLES @@ -64,4 +71,4 @@ def test_read_custom_cmor_tables(): assert cmip6_table.default is custom_table # Restore default tables - read_cmor_tables(read_config_developer_file()) + read_cmor_tables() diff --git a/tests/integration/config/__init__.py b/tests/integration/config/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 7964cc71d8..3bdd9a9c42 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -4,6 +4,23 @@ import pytest from esmvalcore import _data_finder +from esmvalcore.config import CFG, _config +from esmvalcore.config._config_object import CFG_DEFAULT + + +@pytest.fixture +def session(tmp_path, monkeypatch): + session = CFG.start_session('recipe_test') + session.clear() + session.update(CFG_DEFAULT) + session['output_dir'] = tmp_path / 'esmvaltool_output' + + # The patched_data_finder fixture does not return the correct input + # directory structure, so make sure it is set to flat for every project + session['drs'] = {} + for project in _config.CFG: + monkeypatch.setitem(_config.CFG[project]['input_dir'], 'default', '/') + return session def create_test_file(filename, tracking_id=None): diff --git a/tests/integration/preprocessor/_regrid/test_get_cmor_levels.py b/tests/integration/preprocessor/_regrid/test_get_cmor_levels.py index c50349dffa..b4f869d171 100644 --- a/tests/integration/preprocessor/_regrid/test_get_cmor_levels.py +++ b/tests/integration/preprocessor/_regrid/test_get_cmor_levels.py @@ -7,16 +7,10 @@ import unittest -from esmvalcore._config import read_config_developer_file -from esmvalcore.cmor.table import read_cmor_tables from esmvalcore.preprocessor import _regrid class TestGetCmorLevels(unittest.TestCase): - @staticmethod - def setUpClass(): - """Read cmor tables before testing""" - read_cmor_tables(read_config_developer_file()) def test_cmip6_alt40(self): self.assertListEqual( diff --git a/tests/integration/test_data_finder.py b/tests/integration/test_data_finder.py index 966e5a268d..7e74266f45 100644 --- a/tests/integration/test_data_finder.py +++ b/tests/integration/test_data_finder.py @@ -6,19 +6,12 @@ import pytest import yaml -import esmvalcore._config +import esmvalcore.config from esmvalcore._data_finder import ( _find_input_files, get_input_filelist, get_output_file, ) -from esmvalcore.cmor.table import read_cmor_tables - -# Initialize with standard config developer file -CFG_DEVELOPER = esmvalcore._config.read_config_developer_file() -esmvalcore._config._config.CFG = CFG_DEVELOPER -# Initialize CMOR tables -read_cmor_tables(CFG_DEVELOPER) # Load test configuration with open(os.path.join(os.path.dirname(__file__), 'data_finder.yml')) as file: @@ -27,7 +20,7 @@ def _augment_with_extra_facets(variable): """Augment variable dict with extra facets.""" - extra_facets = esmvalcore._config.get_extra_facets( + extra_facets = esmvalcore.config._config.get_extra_facets( variable['project'], variable['dataset'], variable['mip'], diff --git a/tests/integration/test_deprecated_config.py b/tests/integration/test_deprecated_config.py new file mode 100644 index 0000000000..06ca360e43 --- /dev/null +++ b/tests/integration/test_deprecated_config.py @@ -0,0 +1,11 @@ +from pathlib import Path + +import esmvalcore +from esmvalcore._config import read_config_user_file + + +def test_read_config_user(): + config_file = Path(esmvalcore.__file__).parent / 'config-user.yml' + cfg = read_config_user_file(config_file, 'recipe_test', {'offline': False}) + assert len(cfg) > 1 + assert cfg['offline'] is False diff --git a/tests/integration/test_diagnostic_run.py b/tests/integration/test_diagnostic_run.py index cb90235f4e..4d95dd850f 100644 --- a/tests/integration/test_diagnostic_run.py +++ b/tests/integration/test_diagnostic_run.py @@ -8,8 +8,8 @@ import pytest import yaml -from esmvalcore._config import TAGS from esmvalcore._main import run +from esmvalcore.config._diagnostics import TAGS def write_config_user_file(dirname): diff --git a/tests/integration/test_recipe.py b/tests/integration/test_recipe.py index 67bf519dbe..231913788f 100644 --- a/tests/integration/test_recipe.py +++ b/tests/integration/test_recipe.py @@ -13,7 +13,6 @@ from PIL import Image import esmvalcore -from esmvalcore._config import TAGS from esmvalcore._recipe import ( TASKSEP, _dataset_to_file, @@ -22,11 +21,11 @@ ) from esmvalcore._task import DiagnosticTask from esmvalcore.cmor.check import CheckLevels +from esmvalcore.config._diagnostics import TAGS from esmvalcore.exceptions import InputFilesNotFound, RecipeError from esmvalcore.preprocessor import DEFAULT_ORDER, PreprocessingTask from esmvalcore.preprocessor._io import concatenate_callback -from .test_diagnostic_run import write_config_user_file from .test_provenance import check_provenance TAGS_FOR_TESTING = { @@ -101,9 +100,8 @@ @pytest.fixture -def config_user(tmp_path): - filename = write_config_user_file(tmp_path) - cfg = esmvalcore._config.read_config_user_file(filename, 'recipe_test', {}) +def config_user(session): + cfg = session.to_config_user() cfg['offline'] = True cfg['check_level'] = CheckLevels.DEFAULT cfg['diagnostics'] = set() @@ -3603,6 +3601,7 @@ def test_recipe_run(tmp_path, patched_datafinder, config_user, mocker): recipe.tasks.run = mocker.Mock() recipe.write_filled_recipe = mocker.Mock() + recipe.write_html_summary = mocker.Mock() recipe.run() esmvalcore._recipe.esgf.download.assert_called_once_with( @@ -3610,6 +3609,7 @@ def test_recipe_run(tmp_path, patched_datafinder, config_user, mocker): recipe.tasks.run.assert_called_once_with( max_parallel_tasks=config_user['max_parallel_tasks']) recipe.write_filled_recipe.assert_called_once() + recipe.write_html_summary.assert_called_once() @patch('esmvalcore._recipe.check.data_availability', autospec=True) diff --git a/tests/integration/test_task.py b/tests/integration/test_task.py index 4ceb62ac91..42b724e1c9 100644 --- a/tests/integration/test_task.py +++ b/tests/integration/test_task.py @@ -7,7 +7,6 @@ import pytest import esmvalcore -from esmvalcore._config import DIAGNOSTICS from esmvalcore._task import ( BaseTask, DiagnosticError, @@ -15,6 +14,7 @@ TaskSet, _py2ncl, ) +from esmvalcore.config._diagnostics import DIAGNOSTICS class MockBaseTask(BaseTask): diff --git a/tests/sample_data/experimental/test_run_recipe.py b/tests/sample_data/experimental/test_run_recipe.py index 3b88659600..958f4f6fb7 100644 --- a/tests/sample_data/experimental/test_run_recipe.py +++ b/tests/sample_data/experimental/test_run_recipe.py @@ -8,7 +8,7 @@ import iris import pytest -from esmvalcore._config import TAGS +from esmvalcore.config._diagnostics import TAGS from esmvalcore.exceptions import RecipeError from esmvalcore.experimental import CFG, Recipe, get_recipe from esmvalcore.experimental.recipe_output import ( diff --git a/tests/unit/config/test_config.py b/tests/unit/config/test_config.py index a223007adc..a3a1909400 100644 --- a/tests/unit/config/test_config.py +++ b/tests/unit/config/test_config.py @@ -4,8 +4,9 @@ import pytest import yaml -from esmvalcore._config import _config -from esmvalcore._config._config import ( +from esmvalcore.cmor.check import CheckLevels +from esmvalcore.config import CFG, _config +from esmvalcore.config._config import ( _deep_update, _load_extra_facets, get_extra_facets, @@ -129,19 +130,34 @@ def test_get_project_config(mocker): _config.get_project_config('non-existent-project') -def test_load_default_config(monkeypatch): +CONFIG_USER_FILE = importlib_files('esmvalcore') / 'config-user.yml' + + +@pytest.fixture +def default_config(): + # Load default configuration + CFG.load_from_file(CONFIG_USER_FILE) + # Run test + yield + # Restore default configuration + CFG.load_from_file(CONFIG_USER_FILE) + + +def test_load_default_config(monkeypatch, default_config): """Test that the default configuration can be loaded.""" project_cfg = {} monkeypatch.setattr(_config, 'CFG', project_cfg) - default_cfg_file = importlib_files('esmvalcore') / 'config-user.yml' - cfg = _config.read_config_user_file(default_cfg_file, 'recipe_example') + default_dev_file = importlib_files('esmvalcore') / 'config-developer.yml' + cfg = CFG.start_session('recipe_example') default_cfg = { - 'auxiliary_data_dir': str(Path.home() / 'auxiliary_data'), + 'auxiliary_data_dir': Path.home() / 'auxiliary_data', + 'check_level': CheckLevels.DEFAULT, 'compress_netcdf': False, - 'config_developer_file': None, - 'config_file': str(default_cfg_file), - 'download_dir': str(Path.home() / 'climate_data'), + 'config_developer_file': default_dev_file, + 'config_file': CONFIG_USER_FILE, + 'diagnostics': None, + 'download_dir': Path.home() / 'climate_data', 'drs': { 'CMIP3': 'ESGF', 'CMIP5': 'ESGF', @@ -152,75 +168,77 @@ def test_load_default_config(monkeypatch): 'exit_on_warning': False, 'extra_facets_dir': tuple(), 'log_level': 'info', + 'max_datasets': None, 'max_parallel_tasks': None, + 'max_years': None, 'offline': True, + 'output_dir': Path.home() / 'esmvaltool_output', 'output_file_type': 'png', 'profile_diagnostic': False, 'remove_preproc_dir': True, 'resume_from': [], 'rootpath': { - 'default': [str(Path.home() / 'climate_data')] + 'default': [Path.home() / 'climate_data'] }, 'run_diagnostic': True, + 'skip_nonexistent': False, 'save_intermediary_cubes': False, } - default_keys = set( - list(default_cfg) + [ - 'output_dir', - 'plot_dir', - 'preproc_dir', - 'run_dir', - 'work_dir', - ]) + directory_attrs = { + 'session_dir', + 'plot_dir', + 'preproc_dir', + 'run_dir', + 'work_dir', + } # Check that only allowed keys are in it - assert default_keys == set(cfg) + assert set(default_cfg) == set(cfg) + + # Check that all required directories are available + assert all(hasattr(cfg, attr) for attr in directory_attrs) # Check default values for key in default_cfg: assert cfg[key] == default_cfg[key] # Check output directories - assert cfg['output_dir'].startswith( + assert str(cfg.session_dir).startswith( str(Path.home() / 'esmvaltool_output' / 'recipe_example')) for path in ('preproc', 'work', 'run'): - assert cfg[path + '_dir'] == str(Path(cfg['output_dir'], path)) - assert cfg['plot_dir'] == str(Path(cfg['output_dir'], 'plots')) + assert getattr(cfg, path + '_dir') == cfg.session_dir / path + assert cfg.plot_dir == cfg.session_dir / 'plots' # Check that projects were configured assert project_cfg -def test_rootpath_obs4mips_case_correction(tmp_path, monkeypatch, mocker): +def test_rootpath_obs4mips_case_correction(default_config): """Test that the name of the obs4MIPs project is correct in rootpath.""" - monkeypatch.setattr(_config, 'CFG', {}) - mocker.patch.object(_config, 'read_cmor_tables', autospec=True) - cfg_file = tmp_path / 'config-user.yml' - cfg_user = { - 'rootpath': { - 'obs4mips': '/path/to/data', - }, - } - with cfg_file.open('w') as file: - yaml.safe_dump(cfg_user, file) + CFG['rootpath'] = {'obs4mips': '/path/to/data'} + assert 'obs4mips' not in CFG['rootpath'] + assert CFG['rootpath']['obs4MIPs'] == [Path('/path/to/data')] - cfg = _config.read_config_user_file(cfg_file, 'recipe_example') - assert 'obs4mips' not in cfg['rootpath'] - assert cfg['rootpath']['obs4MIPs'] == ['/path/to/data'] +def test_drs_obs4mips_case_correction(default_config): + """Test that the name of the obs4MIPs project is correct in rootpath.""" + CFG['drs'] = {'obs4mips': 'ESGF'} + assert 'obs4mips' not in CFG['drs'] + assert CFG['drs']['obs4MIPs'] == 'ESGF' def test_project_obs4mips_case_correction(tmp_path, monkeypatch, mocker): monkeypatch.setattr(_config, 'CFG', {}) mocker.patch.object(_config, 'read_cmor_tables', autospec=True) cfg_file = tmp_path / 'config-developer.yml' + project_cfg = {'input_dir': {'default': '/'}} cfg_dev = { - 'obs4mips': {}, + 'obs4mips': project_cfg, } with cfg_file.open('w') as file: yaml.safe_dump(cfg_dev, file) - cfg = _config.read_config_developer_file(cfg_file) + _config.load_config_developer(cfg_file) - assert 'obs4mips' not in cfg - assert cfg['obs4MIPs'] == {} + assert 'obs4mips' not in _config.CFG + assert _config.CFG['obs4MIPs'] == project_cfg diff --git a/tests/unit/config/test_config_object.py b/tests/unit/config/test_config_object.py new file mode 100644 index 0000000000..fbabf58874 --- /dev/null +++ b/tests/unit/config/test_config_object.py @@ -0,0 +1,80 @@ +from collections.abc import MutableMapping +from pathlib import Path + +import pytest + +from esmvalcore.config import Config, _config_object +from esmvalcore.exceptions import InvalidConfigParameter + + +def test_config_class(): + config = { + 'log_level': 'info', + 'exit_on_warning': False, + 'output_file_type': 'png', + 'output_dir': './esmvaltool_output', + 'auxiliary_data_dir': './auxiliary_data', + 'save_intermediary_cubes': False, + 'remove_preproc_dir': True, + 'max_parallel_tasks': None, + 'profile_diagnostic': False, + 'rootpath': { + 'CMIP6': '~/data/CMIP6' + }, + 'drs': { + 'CMIP6': 'default' + }, + } + + cfg = Config(config) + + assert isinstance(cfg['output_dir'], Path) + assert isinstance(cfg['auxiliary_data_dir'], Path) + + from esmvalcore.config._config import CFG as CFG_DEV + assert CFG_DEV + + +def test_config_update(): + config = Config({'output_dir': 'directory'}) + fail_dict = {'output_dir': 123} + + with pytest.raises(InvalidConfigParameter): + config.update(fail_dict) + + +def test_set_bad_item(): + config = Config({'output_dir': 'config'}) + with pytest.raises(InvalidConfigParameter) as err_exc: + config['bad_item'] = 47 + + assert str(err_exc.value) == '`bad_item` is not a valid config parameter.' + + +def test_config_init(): + config = Config() + assert isinstance(config, MutableMapping) + + +def test_load_from_file(monkeypatch): + default_config_user_file = Path.home() / '.esmvaltool' / 'config-user.yml' + assert _config_object.USER_CONFIG == default_config_user_file + monkeypatch.setattr( + _config_object, + 'USER_CONFIG', + _config_object.DEFAULT_CONFIG, + ) + config = Config() + assert not config + config.load_from_file() + assert config + + +def test_session(): + config = Config({'output_dir': 'config'}) + + session = config.start_session('recipe_name') + assert session == config + + session['output_dir'] = 'session' + assert session != config diff --git a/tests/unit/experimental/test_config.py b/tests/unit/config/test_config_validator.py similarity index 77% rename from tests/unit/experimental/test_config.py rename to tests/unit/config/test_config_validator.py index 549b4c78de..47da116416 100644 --- a/tests/unit/experimental/test_config.py +++ b/tests/unit/config/test_config_validator.py @@ -1,12 +1,10 @@ -from collections.abc import MutableMapping from pathlib import Path import numpy as np import pytest from esmvalcore import __version__ as current_version -from esmvalcore.experimental.config._config_object import Config -from esmvalcore.experimental.config._config_validators import ( +from esmvalcore.config._config_validators import ( _listify_validator, deprecate, validate_bool, @@ -22,8 +20,6 @@ validate_string, validate_string_or_none, ) -from esmvalcore.experimental.config._validated_config import ( - InvalidConfigParameter, ) def generate_validator_testcases(valid): @@ -199,62 +195,3 @@ def test_func(): f = deprecate(test_func, 'test_var', version) assert callable(f) - - -def test_config_class(): - config = { - 'log_level': 'info', - 'exit_on_warning': False, - 'output_file_type': 'png', - 'output_dir': './esmvaltool_output', - 'auxiliary_data_dir': './auxiliary_data', - 'save_intermediary_cubes': False, - 'remove_preproc_dir': True, - 'max_parallel_tasks': None, - 'profile_diagnostic': False, - 'rootpath': { - 'CMIP6': '~/data/CMIP6' - }, - 'drs': { - 'CMIP6': 'default' - }, - } - - cfg = Config(config) - - assert isinstance(cfg['output_dir'], Path) - assert isinstance(cfg['auxiliary_data_dir'], Path) - - from esmvalcore._config._config import CFG as CFG_DEV - assert CFG_DEV - - -def test_config_update(): - config = Config({'output_dir': 'directory'}) - fail_dict = {'output_dir': 123} - - with pytest.raises(InvalidConfigParameter): - config.update(fail_dict) - - -def test_set_bad_item(): - config = Config({'output_dir': 'config'}) - with pytest.raises(InvalidConfigParameter) as err_exc: - config['bad_item'] = 47 - - assert str(err_exc.value) == '`bad_item` is not a valid config parameter.' - - -def test_config_init(): - config = Config() - assert isinstance(config, MutableMapping) - - -def test_session(): - config = Config({'output_dir': 'config'}) - - session = config.start_session('recipe_name') - assert session == config - - session['output_dir'] = 'session' - assert session != config diff --git a/tests/integration/config/test_diagnostic.py b/tests/unit/config/test_diagnostic.py similarity index 95% rename from tests/integration/config/test_diagnostic.py rename to tests/unit/config/test_diagnostic.py index b4604a790e..e7c836d283 100644 --- a/tests/integration/config/test_diagnostic.py +++ b/tests/unit/config/test_diagnostic.py @@ -1,7 +1,7 @@ """Test Diagnostics and TagsManager.""" import pytest -from esmvalcore._config._diagnostics import Diagnostics, TagsManager +from esmvalcore.config._diagnostics import Diagnostics, TagsManager def test_diagnostics_class(): @@ -72,7 +72,7 @@ def test_tags_manager_fails(): tags.replace_tags_in_dict(dict_with_undefined_tags) -def test_load_tags_from_non_existant_file(): +def test_load_tags_from_non_existent_file(): """Test fallback if no diagnostics are installed.""" tags = TagsManager.from_file('non-existent') assert isinstance(tags, TagsManager) diff --git a/tests/unit/config/test_esgf_pyclient.py b/tests/unit/config/test_esgf_pyclient.py index a582396f13..5a481da8ed 100644 --- a/tests/unit/config/test_esgf_pyclient.py +++ b/tests/unit/config/test_esgf_pyclient.py @@ -5,7 +5,7 @@ import pytest import yaml -from esmvalcore._config import _esgf_pyclient +from esmvalcore.config import _esgf_pyclient DEFAULT_CONFIG: dict = { 'logon': { @@ -28,7 +28,7 @@ 'timeout': 120, 'cache': - str(Path.home() / '.esmvaltool' / 'cache' / 'pyesgf-search-results'), + Path.home() / '.esmvaltool' / 'cache' / 'pyesgf-search-results', 'expire_after': 86400, }, diff --git a/tests/unit/experimental/test_recipe.py b/tests/unit/experimental/test_recipe.py index ada4e792c6..32fc22f214 100644 --- a/tests/unit/experimental/test_recipe.py +++ b/tests/unit/experimental/test_recipe.py @@ -1,6 +1,6 @@ import pytest -from esmvalcore._config import DIAGNOSTICS, TAGS +from esmvalcore.config._diagnostics import DIAGNOSTICS, TAGS from esmvalcore.experimental import get_recipe pytest.importorskip( diff --git a/tests/unit/experimental/test_recipe_info.py b/tests/unit/experimental/test_recipe_info.py index c54a37536f..5b6989caa7 100644 --- a/tests/unit/experimental/test_recipe_info.py +++ b/tests/unit/experimental/test_recipe_info.py @@ -2,8 +2,7 @@ from pathlib import Path import esmvalcore -from esmvalcore._config import TAGS -from esmvalcore._config._diagnostics import Diagnostics +from esmvalcore.config._diagnostics import TAGS, Diagnostics from esmvalcore.experimental.recipe_info import ( Contributor, Project, diff --git a/tests/unit/experimental/test_utils.py b/tests/unit/experimental/test_utils.py index ab9dbed388..54e1a835ac 100644 --- a/tests/unit/experimental/test_utils.py +++ b/tests/unit/experimental/test_utils.py @@ -1,6 +1,6 @@ import pytest -from esmvalcore._config import DIAGNOSTICS, TAGS +from esmvalcore.config._diagnostics import DIAGNOSTICS, TAGS from esmvalcore.experimental.recipe import Recipe from esmvalcore.experimental.utils import ( RecipeList, diff --git a/tests/unit/main/test_esmvaltool.py b/tests/unit/main/test_esmvaltool.py index 634a8c6899..08a9b15ab4 100644 --- a/tests/unit/main/test_esmvaltool.py +++ b/tests/unit/main/test_esmvaltool.py @@ -1,65 +1,97 @@ import logging import os -import pathlib +from pathlib import Path from unittest import mock import pytest -import esmvalcore._config import esmvalcore._main import esmvalcore._task +import esmvalcore.config +import esmvalcore.config._logging import esmvalcore.esgf from esmvalcore import __version__ from esmvalcore._main import HEADER, ESMValTool -from esmvalcore.cmor.check import CheckLevels LOGGER = logging.getLogger(__name__) -@pytest.mark.parametrize('cmd_offline', [None, True, False]) -@pytest.mark.parametrize('cfg_offline', [True, False]) -def test_run(mocker, tmp_path, cmd_offline, cfg_offline): - - output_dir = tmp_path / 'output_dir' - recipe = tmp_path / 'recipe_test.yml' - recipe.touch() - offline = cmd_offline is True or (cmd_offline is None - and cfg_offline is True) - - # Minimal config-user.yml for ESMValTool run function. - cfg = { - 'config_file': tmp_path / '.esmvaltool' / 'config-user.yml', - 'log_level': 'info', - 'offline': cfg_offline, - 'preproc_dir': str(output_dir / 'preproc_dir'), - 'run_dir': str(output_dir / 'run_dir'), - } - - # Expected configuration after updating from command line. - reference = dict(cfg) - reference.update({ - 'check_level': CheckLevels.DEFAULT, - 'diagnostics': set(), - 'offline': offline, - 'resume_from': [], - 'skip_nonexistent': False, - - }) +@pytest.fixture +def cfg(mocker, tmp_path): + """Mock `esmvalcore.config.CFG`.""" + session = mocker.MagicMock() - # Patch every imported function + cfg_dict = {} + session.__getitem__.side_effect = cfg_dict.__getitem__ + session.__setitem__.side_effect = cfg_dict.__setitem__ + + output_dir = tmp_path / 'esmvaltool_output' + session.session_dir = output_dir / 'recipe_test' + session.run_dir = session.session_dir / 'run_dir' + session.preproc_dir = session.session_dir / 'preproc_dir' + + cfg = mocker.Mock() + cfg.start_session.return_value = session + + return cfg + + +@pytest.fixture +def session(cfg): + return cfg.start_session.return_value + + +@pytest.mark.parametrize('argument,value', [ + ('max_datasets', 2), + ('max_years', 2), + ('skip_nonexistent', True), + ('offline', False), + ('diagnostics', 'diagnostic_name/group_name'), + ('check_level', 'strict'), +]) +def test_run_command_line_config(mocker, cfg, argument, value): + """Check that the configuration is updated from the command line.""" mocker.patch.object( - esmvalcore._config, - 'read_config_user_file', - create_autospec=True, - return_value=cfg, + esmvalcore.config, + 'CFG', + cfg, ) + session = cfg.start_session.return_value + + program = ESMValTool() + recipe_file = '/path/to/recipe_test.yml' + config_file = '/path/to/config-user.yml' + + mocker.patch.object(program, '_get_recipe', return_value=Path(recipe_file)) + mocker.patch.object(program, '_run') + + program.run(recipe_file, config_file, **{argument: value}) + + cfg.load_from_file.assert_called_with(config_file) + cfg.start_session.assert_called_once_with(Path(recipe_file).stem) + program._get_recipe.assert_called_with(recipe_file) + program._run.assert_called_with(program._get_recipe.return_value, session) + + assert session[argument] == value + + +@pytest.mark.parametrize('offline', [True, False]) +def test_run(mocker, session, offline): + session['offline'] = offline + session['log_level'] = 'default' + session['config_file'] = '/path/to/config-user.yml' + session['remove_preproc_dir'] = True + + recipe = Path('/recipe_dir/recipe_test.yml') + + # Patch every imported function mocker.patch.object( - esmvalcore._config, + esmvalcore.config._logging, 'configure_logging', create_autospec=True, ) mocker.patch.object( - esmvalcore._config, + esmvalcore.config._diagnostics, 'DIAGNOSTICS', create_autospec=True, ) @@ -79,20 +111,12 @@ def test_run(mocker, tmp_path, cmd_offline, cfg_offline): create_autospec=True, ) - ESMValTool().run(str(recipe), offline=cmd_offline) - - # Check that configuration has been updated from the command line - assert cfg == reference + ESMValTool()._run(recipe, session=session) # Check that the correct functions have been called - esmvalcore._config.read_config_user_file.assert_called_once_with( - None, - recipe.stem, - {}, - ) - esmvalcore._config.configure_logging.assert_called_once_with( - output_dir=cfg['run_dir'], - console_log_level=cfg['log_level'], + esmvalcore.config._logging.configure_logging.assert_called_once_with( + output_dir=session.run_dir, + console_log_level=session['log_level'], ) if offline: @@ -102,14 +126,29 @@ def test_run(mocker, tmp_path, cmd_offline, cfg_offline): esmvalcore._task.resource_usage_logger.assert_called_once_with( pid=os.getpid(), - filename=os.path.join(cfg['run_dir'], 'resource_usage.txt'), + filename=session.run_dir / 'resource_usage.txt', ) esmvalcore._main.process_recipe.assert_called_once_with( recipe_file=recipe, - config_user=cfg, + session=session, ) +def test_run_fail_session_dir_exists(session): + program = ESMValTool() + session.session_dir.mkdir(parents=True) + with pytest.raises(FileExistsError): + program._run(Path('/path/to/recipe_test.yml'), session) + + +def test_clean_preproc_dir(session): + session.preproc_dir.mkdir(parents=True) + session['remove_preproc_dir'] = True + program = ESMValTool() + program._clean_preproc(session) + assert not session.preproc_dir.exists() + + @mock.patch('esmvalcore._main.iter_entry_points') def test_header(mock_entry_points, caplog): @@ -144,18 +183,18 @@ def test_get_recipe(is_file): """Test get recipe.""" is_file.return_value = True recipe = ESMValTool()._get_recipe('/recipe.yaml') - assert recipe == pathlib.Path('/recipe.yaml') + assert recipe == Path('/recipe.yaml') @mock.patch('os.path.isfile') -@mock.patch('esmvalcore._config.DIAGNOSTICS') +@mock.patch('esmvalcore.config._diagnostics.DIAGNOSTICS') def test_get_installed_recipe(diagnostics, is_file): def encountered(path): - return path == '/install_folder/recipe.yaml' + return Path(path) == Path('/install_folder/recipe.yaml') is_file.side_effect = encountered - diagnostics.recipes = pathlib.Path('/install_folder') + diagnostics.recipes = Path('/install_folder') recipe = ESMValTool()._get_recipe('recipe.yaml') - assert recipe == pathlib.Path('/install_folder/recipe.yaml') + assert recipe == Path('/install_folder/recipe.yaml') @mock.patch('os.path.isfile') @@ -163,4 +202,4 @@ def test_get_recipe_not_found(is_file): """Test get recipe.""" is_file.return_value = False recipe = ESMValTool()._get_recipe('/recipe.yaml') - assert recipe == pathlib.Path('/recipe.yaml') + assert recipe == Path('/recipe.yaml') diff --git a/tests/unit/test_main.py b/tests/unit/main/test_main.py similarity index 100% rename from tests/unit/test_main.py rename to tests/unit/main/test_main.py diff --git a/tests/unit/main/test_recipes.py b/tests/unit/main/test_recipes.py new file mode 100644 index 0000000000..6b95561c9a --- /dev/null +++ b/tests/unit/main/test_recipes.py @@ -0,0 +1,56 @@ +"""Test the `Recipe` class implementing the `esmvaltool recipes` command.""" +import textwrap + +import esmvalcore.config._diagnostics +from esmvalcore._main import Recipes + + +def test_list(mocker, tmp_path, capsys): + """Test the command `esmvaltool recipes list`.""" + recipe_dir = tmp_path + recipe1 = recipe_dir / 'recipe_test1.yml' + recipe2 = recipe_dir / 'subdir' / 'recipe_test2.yml' + recipe1.touch() + recipe2.parent.mkdir() + recipe2.touch() + + diagnostics = mocker.patch.object( + esmvalcore.config._diagnostics, + 'DIAGNOSTICS', + create_autospec=True, + ) + diagnostics.recipes = recipe_dir + + Recipes().list() + + msg = capsys.readouterr().out + expected = textwrap.dedent(f""" + # Installed recipes + {recipe1.relative_to(recipe_dir)} + + # Subdir + {recipe2.relative_to(recipe_dir)} + """) + print(msg) + assert msg.endswith(expected) + + +def test_show(mocker, tmp_path, capsys): + """Test the command `esmvaltool recipes list`.""" + recipe_dir = tmp_path + recipe = recipe_dir / 'recipe_test.yml' + recipe.write_text("example") + + diagnostics = mocker.patch.object( + esmvalcore.config._diagnostics, + 'DIAGNOSTICS', + create_autospec=True, + ) + diagnostics.recipes = recipe_dir + + Recipes().show(recipe.name) + + msg = capsys.readouterr().out + print(msg) + assert f"Recipe {recipe.name}" in msg + assert "example" in msg diff --git a/tests/unit/task/test_diagnostic_task.py b/tests/unit/task/test_diagnostic_task.py index d80cab9d68..85db69f534 100644 --- a/tests/unit/task/test_diagnostic_task.py +++ b/tests/unit/task/test_diagnostic_task.py @@ -7,8 +7,8 @@ import yaml import esmvalcore._task -from esmvalcore._config._diagnostics import TagsManager from esmvalcore._task import DiagnosticError, write_ncl_settings +from esmvalcore.config._diagnostics import TagsManager def test_write_ncl_settings(tmp_path): @@ -45,7 +45,7 @@ def test_initialize_env(ext, tmp_path, monkeypatch): lambda self: None) esmvaltool_path = tmp_path / 'esmvaltool' - monkeypatch.setattr(esmvalcore._config.DIAGNOSTICS, 'path', + monkeypatch.setattr(esmvalcore.config._diagnostics.DIAGNOSTICS, 'path', esmvaltool_path) diagnostics_path = esmvaltool_path / 'diag_scripts' diff --git a/tests/unit/test_exceptions.py b/tests/unit/test_exceptions.py new file mode 100644 index 0000000000..1d73412cd3 --- /dev/null +++ b/tests/unit/test_exceptions.py @@ -0,0 +1,23 @@ +import sys + +import pytest + +from esmvalcore.exceptions import SuppressedError + + +@pytest.mark.parametrize('exception', [SuppressedError, ValueError]) +def test_suppressedhook(capsys, exception): + try: + raise exception('error') + except exception: + args = sys.exc_info() + sys.excepthook(*args) + msg = capsys.readouterr().err + if issubclass(exception, SuppressedError): + assert msg == "SuppressedError: error\n" + else: + ending = "ValueError: error\n" + assert msg.endswith(ending) + # because `msg` also contains the traceback, it should be + # longer than `ending` + assert len(msg) > len(ending) diff --git a/tests/unit/test_logging.py b/tests/unit/test_logging.py index e9b51d42ab..74383918c6 100644 --- a/tests/unit/test_logging.py +++ b/tests/unit/test_logging.py @@ -5,7 +5,7 @@ import pytest -from esmvalcore._config._logging import configure_logging +from esmvalcore.config._logging import configure_logging @pytest.mark.parametrize('level', (None, 'INFO', 'DEBUG'))