Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion docs/iris/src/userguide/saving_iris_cubes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@
Saving Iris cubes
==================

Iris supports the saving of cubes and cube lists to:
Iris supports the saving of :class:~`iris.cube.Cube` and :class:~`iris.cube.CubeList` objects to:

* CF netCDF (1.5)
* GRIB (edition 2)
* Met Office PP
* Python pickles, via pickle module


The :py:func:`iris.save` function saves one or more cubes to a file.
Expand Down Expand Up @@ -106,3 +107,38 @@ Such a custom saver will need be written to meet the needs of the file format an

Implementing a bespoke saver is out of scope for the user guide.

Caching cubes and cube lists to pickle files
--------------------------------------------

It should always be possible to create a temporary cache file containing a:class:~`iris.cube.Cube` or :class:~`iris.cube.CubeList` objects using the Python `Pickle <https://docs.python.org/2/library/pickle.html>`_ functionality. This can be useful when the cube or cube list has been lazily loaded so the pickle file itself will contain only a reference to the data in the original files. In this state writing, and subsequently reading, a pickle file is very fast.

.. warning::

Pickle files are not a portable file format, there is no guarantee that pickle files will work if there are any differences between the python/iris environment/version.

If a pickle file contains lazily loaded cubes then any changes to the files that contain the data will break the ability of the cubes to read its data.

The pickle file is especially useful for code development, where the same data files might need to be loaded repeatedly. Another common use case is parent-child processes where a single process loads data and then a number of subsequent processes can quickly access the data in order to run parallel processing tasks. Similarly the ability to pickle cubes is also vital to the using cubes in `parallel processes https://docs.python.org/2/library/multiprocessing.html`_ so any cube which cannot be pickled should be raised as an `issue on the Iris github page https://github.com/SciTools/iris/issues/new`_.

A quick example of saving and reading a pickle file is:

.. code-block:: python
import pickle

cubelist = iris.load(['a.pp', 'list.pp', 'of.pp', 'many.pp', 'files.pp'])
# save cubelist to a pickle
with open('filename.pickle', 'wb') as pfile:
pickle.dump(cubelist, pfile)

# then to load:
with open('filename.pickle', 'rb') as pfile:
loaded_cubelist = pickle.load(pfile)

.. tip::
In Python 2.7, the cPickle module is a faster implementation of the pickle functionality so this will speed up the reading/writing of pickle files:

.. code-block:: python
# for Python 2.7:
import cPickle as pickle

There is no difference in Python 3.