Skip to content
Closed
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
6f977cd
created extract surface preprocessor.
ledm Mar 16, 2021
e47fc4a
added some new uses and documentatiotn.
ledm Mar 16, 2021
66147bc
Added extract_levels documentation
ledm Mar 16, 2021
22e6f4b
added unit test for surface extraction
ledm Mar 16, 2021
a68937b
Update _volume.py
ledm Mar 19, 2021
b9b5800
minor bugfix
ledm Mar 19, 2021
3c7b0c0
fixed test
ledm Mar 19, 2021
c6c8e36
added additional documentation and comments.
ledm Mar 19, 2021
a027d96
removing trailing space
ledm Mar 19, 2021
cb78b9c
added doc fix.
ledm Mar 19, 2021
0b04d4e
Update doc/recipe/preprocessor.rst
ledm Mar 24, 2021
319ba19
Update esmvalcore/preprocessor/_volume.py
ledm Mar 24, 2021
053cfc0
Update esmvalcore/preprocessor/_volume.py
ledm Mar 24, 2021
96fc7e4
Update doc/recipe/preprocessor.rst
ledm Mar 24, 2021
deb6918
Merge remote-tracking branch 'origin/master' into extract_surface
ledm Mar 29, 2021
2f729e9
make surface more generic
ledm Mar 29, 2021
d644526
Search r0i0p0 if fx data not found under original ensemble
thomascrocker Apr 26, 2021
da8cf7e
typo. Moved second search under if statement
thomascrocker Apr 26, 2021
574f9f8
fixed line length complaint
thomascrocker Apr 26, 2021
4378116
fix by using globbing
thomascrocker May 5, 2021
54a5105
Merge branch 'master' into fix_fx_ensembles
valeriupredoi May 10, 2021
0c539c3
fix line too long
valeriupredoi May 10, 2021
5f5bb10
modded tests
valeriupredoi May 10, 2021
a5a933b
Added docs to additional places where FX variables are used
thomascrocker May 11, 2021
d471de2
Merge branch 'master' into fix_fx_ensembles
thomascrocker May 14, 2021
6eb521c
Merge remote-tracking branch 'origin/master' into extract_surface
ledm May 17, 2021
c17f85b
downgrade logging message to debug
thomascrocker May 18, 2021
7df08ee
addressing review comments and maintaining backwards compatibility
thomascrocker May 18, 2021
87b5f48
reverting tests to original
thomascrocker May 18, 2021
b53a991
refactored to resolve wildcards in separate function
thomascrocker May 19, 2021
3611451
CMIP6 wildcard FX test
thomascrocker May 19, 2021
a87d290
added CORDEX wildcard fx test
thomascrocker May 19, 2021
3f1b881
minor tweak to docs
thomascrocker May 19, 2021
8d3bf2d
flake8 fixes
thomascrocker May 19, 2021
443208f
Update _data_finder.py
thomascrocker May 19, 2021
62d091a
codacy fix
thomascrocker May 19, 2021
e2c76b1
minor changes to address review comments
thomascrocker May 20, 2021
ad9b702
additional tests
thomascrocker May 20, 2021
6909061
docs update
thomascrocker May 20, 2021
c88898b
Merge branch 'fix_fx_ensembles' of https://github.com/ESMValGroup/ESM…
thomascrocker May 20, 2021
a7bccf8
pylint fix
thomascrocker May 20, 2021
439ce43
Merge remote-tracking branch 'origin/master' into fix_fx_ensembles
schlunma May 20, 2021
3961cfc
Added further check on variable's ensemble during fx file retrieval i…
schlunma May 20, 2021
893aaaf
refactor to deal with more file path cases
thomascrocker May 21, 2021
c9cdbe4
further dir finder test
thomascrocker May 21, 2021
f0637f2
tweaks for codacy
thomascrocker May 21, 2021
fc61e06
Fixed globbing for fx files when latestversion tag is present
schlunma May 21, 2021
09543a0
Fixed output path for fx files when wildcards are used
schlunma May 21, 2021
92a6207
Fixed output path for fx files when wildcards are used (again)
schlunma May 21, 2021
3676f66
Fixed output path for fx files when wildcards are used (this time for…
schlunma May 21, 2021
3b1c0a2
Removed print() statement
schlunma May 21, 2021
845cc8d
Merge remote-tracking branch 'origin/master' into fix_fx_ensembles
schlunma May 21, 2021
b562249
Fixed tests
schlunma May 21, 2021
993ceaa
added fixes to cmip6 omon.
ledm May 25, 2021
fc60589
working here.
ledm Jun 11, 2021
ce87c3c
pre-merge changes
ledm Jun 15, 2021
15807ae
Merge remote-tracking branch 'origin/main' into extract_surface
ledm Jun 15, 2021
0947529
Merge remote-tracking branch 'origin/main' into extract_surface
ledm Jun 16, 2021
ca850b1
Merge remote-tracking branch 'origin/main' into extract_surface
ledm Aug 17, 2021
d4a2552
added fixes
ledm Aug 17, 2021
2b32a17
Merge remote-tracking branch 'origin/main' into extract_surface
ledm Sep 6, 2021
b50b094
Merge remote-tracking branch 'origin/main' into fix_fx_ensembles
schlunma Sep 9, 2021
fae8f0c
added fatal error if fx missing
ledm Sep 9, 2021
f200c14
Merge remote-tracking branch 'origin/fix_fx_ensembles' into extract_s…
ledm Sep 9, 2021
2d13465
Merge remote-tracking branch 'origin/main' into extract_surface
ledm Sep 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions doc/recipe/preprocessor.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1326,6 +1326,7 @@ The ``_volume.py`` module contains the following preprocessor functions:
* ``extract_transect``: Extract data along a line of constant latitude or
longitude.
* ``extract_trajectory``: Extract data along a specified trajectory.
* ``extract_surface``: Extract the surface layer from a cube.


``extract_volume``
Expand Down Expand Up @@ -1421,6 +1422,24 @@ Note that this function uses the expensive ``interpolate`` method from
See also :func:`esmvalcore.preprocessor.extract_trajectory`.


``extract_surface``
-------------------

This function extracts the surface layer from a dataset.

The surface layer is defined as the minimum
of the absolute value of the Z-dimension.
This is typically the case for ocean models, but might not be the
case for atmospheric models.

The same functionality exists in the extract_levels preprocessor,
Comment thread
ledm marked this conversation as resolved.
Outdated
and this function should be used for more complex datasets
if extract_surface fails. However, extract_levels also
Comment thread
ledm marked this conversation as resolved.
Outdated
has a higher computational cost, and may be slower.

See also :func:`esmvalcore.preprocessor.extract_levels`.


.. _cycles:

Cycles
Expand Down
2 changes: 2 additions & 0 deletions esmvalcore/preprocessor/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
extract_trajectory,
extract_transect,
extract_volume,
extract_surface,
volume_statistics,
)
from ._weighting import weighting_landsea_fraction
Expand Down Expand Up @@ -99,6 +100,7 @@
'resample_hours',
'resample_time',
# Level extraction
'extract_surface',
'extract_levels',
# Weighting
'weighting_landsea_fraction',
Expand Down
62 changes: 62 additions & 0 deletions esmvalcore/preprocessor/_volume.py
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,68 @@ def volume_statistics(
return _create_cube_time(src_cube, result, times)


def extract_surface(cube):
"""
Extact the surface layer from a 3D cube.

Extracts the entire layer along the z axis where the z coordinate is
the minimum of the absolute value of the z-axis dimensions points.

Requires a cube with a z axis.

This assumes that the surface is the closest layer to zero.
This assumption is usually true in the ocean, but may not be the
case in all models for all z-axes.

In the case of temporally or spatially varying depth grid,
(ie the depth array is 3D or 4D), the surface layer is determined
using the mean of the depth points along the time, latitude and
longitude axes. This preprocessor may also behave strangely
if the z axis data are not monotonic, or unusual in some other way.

The preprocessor extract_layer can also do this, but it may be
much slower and more memory intensie as it regrids.
It is more robust to variability in the grid definitions.

Arguments
Comment thread
ledm marked this conversation as resolved.
Outdated
---------
Comment thread
ledm marked this conversation as resolved.
Outdated
cube: iris.cube.Cube
input cube.

Returns
-------
iris.cube.Cube
collapsed cube.

"""
# Get the z axis.
zcoord = cube.coord(axis='Z')
positive = zcoord.attributes['positive']

# Get the Z points:
if zcoord.points.ndim == 1:
points = zcoord.points
elif zcoord.points.ndim == 3:
points = zcoord.points.mean(axis=[1, 2])
elif zcoord.points.ndim == 4:
points = zcoord.points.mean(axis=[0, 2, 3])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could zcoord.points.ndim == 2?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either way, maybe add an exception if any of the dims are not the ones you analyze

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right.

The problem with that (and this method as a whole) is that it makes assumptions about the z-axis position. For instance, what if you have a cube with dimensions (time, depth, latitude) ie a meridional transect? The preprocessor would fail.

Maybe instead of that, we could replace this if-else loop with something more generic?

    zcoord_dim = cube.coord_dims(zcoord)[0]
    axes = np.arrange(zcoord.points.ndim).pop(zcoord_dim)
    points = zcoord.points.mean(axis=axes)


# Calculate the surface layer index:
surf = np.abs(points).argmin()

# Get the z axis dimension in the cude:
zcoord_dim = cube.coord_dims(zcoord)
if zcoord_dim in [0, (0,)]:
return cube[surf]

if zcoord_dim in [1, (1,)]:
return cube[:, surf]

logger.error('Condition not recognised: postive: %s , zcoord_dim: %s, '
'surface level: %s', positive, zcoord_dim, surf)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a very cryptic exception, could you reformulate for the user to understand what actually is wrong pls?

return cube


def depth_integration(cube):
"""
Determine the total sum over the vertical component.
Expand Down
15 changes: 14 additions & 1 deletion tests/unit/preprocessor/_volume/test_volume.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,9 @@
from esmvalcore.preprocessor._volume import (volume_statistics,
depth_integration,
extract_trajectory,
extract_transect, extract_volume)
extract_transect,
extract_volume,
extract_surface)


class Test(tests.Test):
Expand All @@ -19,6 +21,7 @@ class Test(tests.Test):
def setUp(self):
"""Prepare tests"""
coord_sys = iris.coord_systems.GeogCS(iris.fileformats.pp.EARTH_RADIUS)
data0 = np.ones((2, 2))
data1 = np.ones((3, 2, 2))
data2 = np.ma.ones((2, 3, 2, 2))
data3 = np.ma.ones((4, 3, 2, 2))
Expand Down Expand Up @@ -61,6 +64,9 @@ def setUp(self):
units='degrees_north',
coord_system=coord_sys)

coords_spec2 = [(lats2, 0), (lons2, 1)]
self.grid_2d = iris.cube.Cube(data0, dim_coords_and_dims=coords_spec2)

coords_spec3 = [(zcoord, 0), (lats2, 1), (lons2, 2)]
self.grid_3d = iris.cube.Cube(data1, dim_coords_and_dims=coords_spec3)

Expand Down Expand Up @@ -152,6 +158,13 @@ def test_extract_trajectory(self):
expected = np.ones((3, 2))
self.assert_array_equal(result.data, expected)

def test_extract_surface(self):
"""Tests to extract the surface from a 3D and 4D cube."""
result = extract_surface(self.grid_3d)
self.assert_array_equal(result.data, self.grid_2d.data)
result2 = extract_surface(self.grid_4d)
self.assert_array_equal(self.grid_4d[:, 0, :, :].data, result2.data)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have a case for 1D as well?



if __name__ == '__main__':
unittest.main()