From 08ec133f0bf99a5e6e9213ebbe3bb00b09980b06 Mon Sep 17 00:00:00 2001 From: sloosvel Date: Thu, 27 Feb 2025 12:18:51 +0100 Subject: [PATCH 1/6] Update release version --- CITATION.cff | 4 ++-- doc/changelog.rst | 1 + 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/CITATION.cff b/CITATION.cff index a81b785705..744b54cc20 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -225,11 +225,11 @@ authors: orcid: "https://orcid.org/0009-0004-2333-3358" cff-version: 1.2.0 -date-released: 2025-02-21 +date-released: 2025-02-27 doi: "10.5281/zenodo.3387139" license: "Apache-2.0" message: "If you use this software, please cite it using these metadata." repository-code: "https://github.com/ESMValGroup/ESMValCore/" title: ESMValCore -version: "v2.12.0rc2" +version: "v2.12.0" ... diff --git a/doc/changelog.rst b/doc/changelog.rst index f83bead538..644cba25dc 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -10,6 +10,7 @@ v2.12.0 ------- Highlights +~~~~~~~~~~ Preprocessor :func:`esmvalcore.preprocessor.extract_time` now allows to extract time blocks in each year by making parameters ``start_year`` and From 3e297629433654502cb17c281473b397374c26a8 Mon Sep 17 00:00:00 2001 From: sloosvel Date: Thu, 27 Feb 2025 13:14:56 +0100 Subject: [PATCH 2/6] Update highlights --- doc/changelog.rst | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index 644cba25dc..c37c7d80ba 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -12,9 +12,22 @@ v2.12.0 Highlights ~~~~~~~~~~ -Preprocessor :func:`esmvalcore.preprocessor.extract_time` now allows to -extract time blocks in each year by making parameters ``start_year`` and -``end_year`` optional. +- Preprocessor :func:`esmvalcore.preprocessor.extract_time` now allows to + extract time blocks in each year by making parameters ``start_year`` and + ``end_year`` optional. +- A new way of configuring the tool has been developed. +- Performance improvements: + + - An iris-esmf-regrid scheme has been added to + preprocessor :func:`esmvalcore.preprocessor.regrid`, which improves + the regridding of 2D grids and adds the capability to regrid UGRID meshes + out of the box. + - Data is now saved from one preprocessing task at the time when using the + distributed scheduler, in order to avoid running out of memory. + - A better default ``num_workers`` has been set when using more than one + ``max_parallel_tasks`` with an unconfigured threaded scheduler. + +- An on-the-fly cmorizer for ACCESS native data is now available. This release includes From b4f702b8c414a6ebc2c2a61a6428e48816c5316a Mon Sep 17 00:00:00 2001 From: sloosvel Date: Thu, 27 Feb 2025 13:19:12 +0100 Subject: [PATCH 3/6] Move from backwards incompatible change to deprecation --- doc/changelog.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index c37c7d80ba..fc1fa1897a 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -35,7 +35,6 @@ Backwards incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Make derivation of total column ozone (`toz`) more flexible and add derivation of stratospheric and tropospheric column ozone (:pull:`2509`) by :user:`schlunma` -- Merge configuration object from multiple files (instead of one single file) (:pull:`2448`) by :user:`schlunma` - Remove deprecated CMOR fix/check code (:pull:`2552`) by :user:`schlunma` - Remove deprecated statistical operators (:pull:`2553`) by :user:`schlunma` - Save all files in a task at the same time to avoid recomputing intermediate results (:pull:`2522`) by :user:`bouweandela` @@ -43,6 +42,7 @@ Backwards incompatible changes Deprecations ~~~~~~~~~~~~ +- Merge configuration object from multiple files (instead of one single file) (:pull:`2448`) by :user:`schlunma` - Make Dask configurable in our configuration (:pull:`2616`) by :user:`schlunma` Bug fixes From b06ed2a16cb17e4817af9747b0f19d7791f8f356 Mon Sep 17 00:00:00 2001 From: sloosvel <45196700+sloosvel@users.noreply.github.com> Date: Thu, 27 Feb 2025 15:24:46 +0100 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Bouwe Andela --- doc/changelog.rst | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/changelog.rst b/doc/changelog.rst index fc1fa1897a..ccdb03cc33 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -15,7 +15,7 @@ Highlights - Preprocessor :func:`esmvalcore.preprocessor.extract_time` now allows to extract time blocks in each year by making parameters ``start_year`` and ``end_year`` optional. -- A new way of configuring the tool has been developed. +- A new way of :ref:`configuring the tool ` has been developed. - Performance improvements: - An iris-esmf-regrid scheme has been added to @@ -38,6 +38,11 @@ Backwards incompatible changes - Remove deprecated CMOR fix/check code (:pull:`2552`) by :user:`schlunma` - Remove deprecated statistical operators (:pull:`2553`) by :user:`schlunma` - Save all files in a task at the same time to avoid recomputing intermediate results (:pull:`2522`) by :user:`bouweandela` + - The signature of the preprocessor function :func:`~esmvalcore.preprocessor.save` + has changed. The function now accepts a ``compute`` argument that can be :obj:`True`, + in which case the return value will be :obj:`None` or :obj:`False`, in which case + the return value will be a :class:`~dask.delayed.Delayed` object that can be used + to compute and save the data of the cube. Deprecations ~~~~~~~~~~~~ From 17f89ab1e65c119437a6ae231c91907560041daa Mon Sep 17 00:00:00 2001 From: sloosvel Date: Thu, 27 Feb 2025 15:35:40 +0100 Subject: [PATCH 5/6] Update changelog --- doc/changelog.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index ccdb03cc33..0ba26161e2 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -35,8 +35,18 @@ Backwards incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Make derivation of total column ozone (`toz`) more flexible and add derivation of stratospheric and tropospheric column ozone (:pull:`2509`) by :user:`schlunma` + - The units of `toz` have been changed from ``DU`` to ``m`` to be consistent with the CMIP6 CMOR table. + To restore the old behaviour, preprocessor :func:`esmvalcore.preprocessor.convert_units` can be used + to set the units back to ``DU``. - Remove deprecated CMOR fix/check code (:pull:`2552`) by :user:`schlunma` + - CMOR fixes and checks have been clearly separated in v2.10.0, and the old code has now been removed. + Use functions :func:`esmvalcore.preprocessors.fix_metadata`, :func:`esmvalcore.preprocessors.fix_data`, + or :func:`esmvalcore.dataset.Dataset.load` to fix data. + Use functions :func:`esmvalcore.preprocessor.cmor_check_metadata`, :func:`esmvalcore.preprocessor.cmor_check_data`, + or :func:`esmvalcore.preprocessor.cmor_check` to check data. - Remove deprecated statistical operators (:pull:`2553`) by :user:`schlunma` + - Old statistical operators that have been deprecated in v2.10.0 have now been removed. + Please refer to :ref:`stat_preprocs` for a detailed description on how to use the operators. - Save all files in a task at the same time to avoid recomputing intermediate results (:pull:`2522`) by :user:`bouweandela` - The signature of the preprocessor function :func:`~esmvalcore.preprocessor.save` has changed. The function now accepts a ``compute`` argument that can be :obj:`True`, @@ -48,7 +58,19 @@ Deprecations ~~~~~~~~~~~~ - Merge configuration object from multiple files (instead of one single file) (:pull:`2448`) by :user:`schlunma` + - The single configuration file ``config-user.yml`` has been deprecated in favour of configuration directories. + By default, the directory ``~/.config/esmvaltool`` will be considered. + To switch to the new format run: + + .. code-block:: bash + + mkdir -p ~/.config/esmvaltool && mv ~/.esmvaltool/config-user.yml ~/.config/esmvaltool + + You can also specify the location of the configuration directory with the ``--config_dir`` flag. + Please refer to :ref:`config` for a detailed description on how to configure the tool. - Make Dask configurable in our configuration (:pull:`2616`) by :user:`schlunma` + - The old Dask configuration file that needed to be located at ``~/.esmvaltool/dask.yml`` is now deprecated. + Please refer to :ref:`config-dask` for a detailed description on how to configure Dask. Bug fixes ~~~~~~~~~ From 137d0d7ce7fb34eddc4f9539a080ab5f2b479ad1 Mon Sep 17 00:00:00 2001 From: sloosvel Date: Thu, 27 Feb 2025 15:54:07 +0100 Subject: [PATCH 6/6] Update changelog --- doc/changelog.rst | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/doc/changelog.rst b/doc/changelog.rst index 0ba26161e2..3731d9d1a7 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -35,19 +35,26 @@ Backwards incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Make derivation of total column ozone (`toz`) more flexible and add derivation of stratospheric and tropospheric column ozone (:pull:`2509`) by :user:`schlunma` + - The units of `toz` have been changed from ``DU`` to ``m`` to be consistent with the CMIP6 CMOR table. To restore the old behaviour, preprocessor :func:`esmvalcore.preprocessor.convert_units` can be used to set the units back to ``DU``. + - Remove deprecated CMOR fix/check code (:pull:`2552`) by :user:`schlunma` + - CMOR fixes and checks have been clearly separated in v2.10.0, and the old code has now been removed. Use functions :func:`esmvalcore.preprocessors.fix_metadata`, :func:`esmvalcore.preprocessors.fix_data`, or :func:`esmvalcore.dataset.Dataset.load` to fix data. Use functions :func:`esmvalcore.preprocessor.cmor_check_metadata`, :func:`esmvalcore.preprocessor.cmor_check_data`, or :func:`esmvalcore.preprocessor.cmor_check` to check data. + - Remove deprecated statistical operators (:pull:`2553`) by :user:`schlunma` + - Old statistical operators that have been deprecated in v2.10.0 have now been removed. Please refer to :ref:`stat_preprocs` for a detailed description on how to use the operators. + - Save all files in a task at the same time to avoid recomputing intermediate results (:pull:`2522`) by :user:`bouweandela` + - The signature of the preprocessor function :func:`~esmvalcore.preprocessor.save` has changed. The function now accepts a ``compute`` argument that can be :obj:`True`, in which case the return value will be :obj:`None` or :obj:`False`, in which case @@ -58,6 +65,7 @@ Deprecations ~~~~~~~~~~~~ - Merge configuration object from multiple files (instead of one single file) (:pull:`2448`) by :user:`schlunma` + - The single configuration file ``config-user.yml`` has been deprecated in favour of configuration directories. By default, the directory ``~/.config/esmvaltool`` will be considered. To switch to the new format run: @@ -68,7 +76,9 @@ Deprecations You can also specify the location of the configuration directory with the ``--config_dir`` flag. Please refer to :ref:`config` for a detailed description on how to configure the tool. + - Make Dask configurable in our configuration (:pull:`2616`) by :user:`schlunma` + - The old Dask configuration file that needed to be located at ``~/.esmvaltool/dask.yml`` is now deprecated. Please refer to :ref:`config-dask` for a detailed description on how to configure Dask.