Skip to content

Futures netcdf no unlimited #2838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
djkirkham merged 7 commits into from
Oct 27, 2017
Merged

Futures netcdf no unlimited #2838

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
d027557
switched boolean operator and altered docstring
corinnebosley Oct 19, 2017
6ebcfcd
raised error on future flag and tweaked functionality
corinnebosley Oct 24, 2017
1b9dae7
small change to deprecation warning message
corinnebosley Oct 24, 2017
da6e2b2
Code changes
DPeterK Oct 25, 2017
142d018
remake cdl results
DPeterK Oct 25, 2017
6683383
More CDL updates
DPeterK Oct 25, 2017
a6c3c9c
Review actions
DPeterK Oct 26, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/iris/src/whatsnew/contributions_2.0/incompatiblechange_2017-Oct-24_netcdf-no-unlim.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
* Deprecated FUTURE flag :attr:`iris.Future.netcdf_no_unlimited`.

* Removed deprecated behaviour that automatically set the length of the outer netCDF variable to 'UNLIMITED' on save.
Copy link
Contributor

@djkirkham djkirkham Oct 26, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps change 'on save' to 'when saving in netCDF format' or something similar, to give more context.

This change means that no cube dimension coordinates will be automatically saved as netCDF variables with 'UNLIMITED' length.
You can manually specify cube dimension coordinates to save with 'UNLIMITED' length.
Copy link
Contributor

@djkirkham djkirkham Oct 26, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find using the second-person a bit jarring in a technical context. Can you change this to the passive voice? Something like 'Cube dimension coordinates to save with 'UNLIMITED' length can be manually specified.'

Copy link
Contributor

@djkirkham djkirkham Oct 26, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or 'It's possible to manually specify...'


For example::
>>> iris.save(my_cube, 'my_result_file.nc', unlimited_dimensions=['latitude'])
16 changes: 11 additions & 5 deletions lib/iris/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ class Future(threading.local):
"""Run-time configuration controller."""

def __init__(self, cell_datetime_objects=True, netcdf_promote=True,
netcdf_no_unlimited=False, clip_latitudes=True):
netcdf_no_unlimited=True, clip_latitudes=True):
"""
A container for run-time options controls.

Expand Down Expand Up @@ -182,10 +182,15 @@ def __init__(self, cell_datetime_objects=True, netcdf_promote=True,
exposed variables that defined reference surfaces for
dimensionless vertical coordinates as independent Cubes.

The option `netcdf_no_unlimited`, when True, changes the
behaviour of the netCDF saver, such that no dimensions are set to
unlimited. The current default is that the leading dimension is
unlimited unless otherwise specified.
.. deprecated:: 2.0.0

The option `netcdf_no_unlimited` is deprecated and will be removed
in a future release. The deprecated code paths this option used to
toggle have been removed.

The option `netcdf_no_unlimited` changed the behaviour of the
netCDF saver regarding unlimited dimensions. The netCDF saver now
sets no dimensions to unlimited.

.. deprecated:: 2.0.0

Expand All @@ -210,6 +215,7 @@ def __repr__(self):
self.netcdf_no_unlimited, self.clip_latitudes)

deprecated_options = {'cell_datetime_objects': 'warning',
'netcdf_no_unlimited': 'error',
Copy link
Member

@rhattersley rhattersley Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "error" behaviour doesn't seem right. 😕
If I've been a Good User and taken account of the deprecation warnings then my current, v1.9 compatible code should look something like:

with iris.Future(netcdf_no_unlimited=True):
    # Do clever stuff with netCDF files

With the behaviour proposed in this PR, when I upgrade to v2.0 my code will get an error. 👎 The whole point of deprecations is to allow seemless migration.
Instead, I should get a deprecation warning about setting netcdf_no_unlimited=True. This lets my code continue to run, and I can remove the now unnecessary with block in my own good time.

If I've been a Bad User and ignored the deprecation warnings from v1.9, my code might just be:

# Do clever stuff with netCDF files that relies on the old behaviour

In which case my code is now broken. That's my own silly fault.
Or perhaps my code looks like:

with iris.Future(netcdf_no_unlimited=False):
    # Do clever stuff with netCDF files that relies on the old behaviour

In which case I should now get an error because I'm trying to set the context to an unsupported value. Again, this is my own silly fault.

In other words, setting netcdf_no_unlimited to True should provoke a warning, but setting it to False should raise an error.

Copy link
Member

@pelson pelson Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Indeed it should. I notice that is what is documented in the what's new too...

Copy link
Contributor

@djkirkham djkirkham Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same behaviour has been added in #2848

Copy link
Contributor

@djkirkham djkirkham Oct 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline with @pelson: I'll put up a separate PR to change the behaviour of the Future object so that the 'error' behaviour is to raise an exception if the flag is set to False and raise a warning when it is set to True.

'netcdf_promote': 'error',
'clip_latitudes': 'warning'}

Expand Down
78 changes: 20 additions & 58 deletions lib/iris/fileformats/netcdf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -851,13 +851,11 @@ def write(self, cube, local_keys=None, unlimited_dimensions=None,

* unlimited_dimensions (iterable of strings and/or
:class:`iris.coords.Coord` objects):
Explicit list of coordinate names (or coordinate objects)
List of coordinate names (or coordinate objects)
corresponding to coordinate dimensions of `cube` to save with the
NetCDF dimension variable length 'UNLIMITED'. By default, the
outermost (first) dimension for each cube is used. Only the
'NETCDF4' format supports multiple 'UNLIMITED' dimensions. To save
no unlimited dimensions, use `unlimited_dimensions=[]` (an empty
list).
NetCDF dimension variable length 'UNLIMITED'. By default, no
unlimited dimensions are saved. Only the 'NETCDF4' format
supports multiple 'UNLIMITED' dimensions.

* zlib (bool):
If `True`, the data will be compressed in the netCDF file using
Expand Down Expand Up @@ -942,19 +940,9 @@ def write(self, cube, local_keys=None, unlimited_dimensions=None,
`chunksizes` and `endian` keywords are silently ignored for netCDF
3 files that do not use HDF5.

.. deprecated:: 1.8.0

NetCDF default saving behaviour currently assigns the outermost
dimension as unlimited. This behaviour is to be deprecated, in
favour of no automatic assignment. To switch to the new behaviour,
set `iris.FUTURE.netcdf_no_unlimited` to True.

"""
"""
if unlimited_dimensions is None:
if iris.FUTURE.netcdf_no_unlimited:
unlimited_dimensions = []
else:
_no_unlim_dep_warning()

cf_profile_available = (iris.site_configuration.get('cf_profile') not
in [None, False])
Expand Down Expand Up @@ -1088,29 +1076,23 @@ def _create_cf_dimensions(self, cube, dimension_names,

* unlimited_dimensions (iterable of strings and/or
:class:`iris.coords.Coord` objects):
List of coordinates to make unlimited. By default, the
outermost dimension is made unlimited.
List of coordinates to make unlimited (None by default).

Returns:
None.

"""
unlimited_dim_names = []
if (unlimited_dimensions is None and
not iris.FUTURE.netcdf_no_unlimited):
if dimension_names:
unlimited_dim_names.append(dimension_names[0])
else:
for coord in unlimited_dimensions:
try:
coord = cube.coord(name_or_coord=coord, dim_coords=True)
except iris.exceptions.CoordinateNotFoundError:
# coordinate isn't used for this cube, but it might be
# used for a different one
pass
else:
dim_name = self._get_coord_variable_name(cube, coord)
unlimited_dim_names.append(dim_name)
for coord in unlimited_dimensions:
try:
coord = cube.coord(name_or_coord=coord, dim_coords=True)
except iris.exceptions.CoordinateNotFoundError:
# coordinate isn't used for this cube, but it might be
# used for a different one
pass
else:
dim_name = self._get_coord_variable_name(cube, coord)
unlimited_dim_names.append(dim_name)

for dim_name in dimension_names:
if dim_name not in self._dataset.dimensions:
Expand Down Expand Up @@ -2127,12 +2109,11 @@ def save(cube, filename, netcdf_format='NETCDF4', local_keys=None,

* unlimited_dimensions (iterable of strings and/or
:class:`iris.coords.Coord` objects):
Explicit list of coordinate names (or coordinate objects) corresponding
List of coordinate names (or coordinate objects) corresponding
to coordinate dimensions of `cube` to save with the NetCDF dimension
variable length 'UNLIMITED'. By default, the outermost (first)
dimension for each cube is used. Only the 'NETCDF4' format supports
multiple 'UNLIMITED' dimensions. To save no unlimited dimensions, use
`unlimited_dimensions=[]` (an empty list).
variable length 'UNLIMITED'. By default, no unlimited dimensions are
saved. Only the 'NETCDF4' format supports multiple 'UNLIMITED'
dimensions.

* zlib (bool):
If `True`, the data will be compressed in the netCDF file using gzip
Expand Down Expand Up @@ -2225,19 +2206,9 @@ def save(cube, filename, netcdf_format='NETCDF4', local_keys=None,

NetCDF Context manager (:class:`~Saver`).

.. deprecated:: 1.8.0

NetCDF default saving behaviour currently assigns the outermost
dimensions to unlimited. This behaviour is to be deprecated, in
favour of no automatic assignment. To switch to the new behaviour,
set `iris.FUTURE.netcdf_no_unlimited` to True.

"""
if unlimited_dimensions is None:
if iris.FUTURE.netcdf_no_unlimited:
unlimited_dimensions = []
else:
_no_unlim_dep_warning()

if isinstance(cube, iris.cube.Cube):
cubes = iris.cube.CubeList()
Expand Down Expand Up @@ -2347,12 +2318,3 @@ def is_valid_packspec(p):

# Add conventions attribute.
sman.update_global_attributes(Conventions=conventions)


def _no_unlim_dep_warning():
msg = ('NetCDF default saving behaviour currently assigns the '
'outermost dimensions to unlimited. This behaviour is to be '
'deprecated, in favour of no automatic assignment. To switch '
'to the new behaviour, set iris.FUTURE.netcdf_no_unlimited to '
'True.')
warn_deprecated(msg)
Loading