diff --git a/docs/iris/src/userguide/saving_iris_cubes.rst b/docs/iris/src/userguide/saving_iris_cubes.rst index d1658ef70e..839bfae9f2 100644 --- a/docs/iris/src/userguide/saving_iris_cubes.rst +++ b/docs/iris/src/userguide/saving_iris_cubes.rst @@ -4,11 +4,12 @@ Saving Iris cubes ================== -Iris supports the saving of cubes and cube lists to: +Iris supports the saving of :class:~`iris.cube.Cube` and :class:~`iris.cube.CubeList` objects to: * CF netCDF (1.5) * GRIB (edition 2) * Met Office PP +* Python pickles, via pickle module The :py:func:`iris.save` function saves one or more cubes to a file. @@ -106,3 +107,38 @@ Such a custom saver will need be written to meet the needs of the file format an Implementing a bespoke saver is out of scope for the user guide. +Caching cubes and cube lists to pickle files +-------------------------------------------- + +It should always be possible to create a temporary cache file containing a:class:~`iris.cube.Cube` or :class:~`iris.cube.CubeList` objects using the Python `Pickle `_ functionality. This can be useful when the cube or cube list has been lazily loaded so the pickle file itself will contain only a reference to the data in the original files. In this state writing, and subsequently reading, a pickle file is very fast. + +.. warning:: + + Pickle files are not a portable file format, there is no guarantee that pickle files will work if there are any differences between the python/iris environment/version. + + If a pickle file contains lazily loaded cubes then any changes to the files that contain the data will break the ability of the cubes to read its data. + +The pickle file is especially useful for code development, where the same data files might need to be loaded repeatedly. Another common use case is parent-child processes where a single process loads data and then a number of subsequent processes can quickly access the data in order to run parallel processing tasks. Similarly the ability to pickle cubes is also vital to the using cubes in `parallel processes https://docs.python.org/2/library/multiprocessing.html`_ so any cube which cannot be pickled should be raised as an `issue on the Iris github page https://github.com/SciTools/iris/issues/new`_. + +A quick example of saving and reading a pickle file is: + +.. code-block:: python + import pickle + + cubelist = iris.load(['a.pp', 'list.pp', 'of.pp', 'many.pp', 'files.pp']) + # save cubelist to a pickle + with open('filename.pickle', 'wb') as pfile: + pickle.dump(cubelist, pfile) + + # then to load: + with open('filename.pickle', 'rb') as pfile: + loaded_cubelist = pickle.load(pfile) + +.. tip:: + In Python 2.7, the cPickle module is a faster implementation of the pickle functionality so this will speed up the reading/writing of pickle files: + + .. code-block:: python + # for Python 2.7: + import cPickle as pickle + + There is no difference in Python 3.