Skip to content

[DOC] Dask processing options docs updates #2900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelson merged 1 commit into from
Nov 1, 2017
Merged

[DOC] Dask processing options docs updates #2900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 3 additions & 34 deletions docs/iris/src/userguide/real_and_lazy_data.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,41 +226,10 @@ coordinates' lazy points and bounds:
Dask processing options
-----------------------

As stated earlier in this discussion, Iris uses dask to provide
lazy data arrays for both Iris cubes and coordinates. Iris also uses dask
functionality for processing deferred operations on lazy arrays.
Iris uses dask to provide lazy data arrays for both Iris cubes and coordinates,
and for computing deferred operations on lazy arrays.

Dask provides processing options to control how deferred operations on lazy arrays
are computed. This is provided via the ``dask.set_options`` interface.
We can make use of this functionality in Iris. This means we can
control how dask arrays in Iris are processed, for example giving us power to
run Iris processing in parallel.

Iris by default applies a single dask processing option. This specifies that
all dask processing in Iris should be run in serial (that is, without any
parallel processing enabled).

The dask processing option applied by Iris can be overridden by manually setting
dask processing options for either or both of:

* the number of parallel workers to use,
* the scheduler to use.

This must be done **before** importing Iris. For example, to specify that dask
processing within Iris should use four workers in a thread pool::

>>> from multiprocessing.pool import ThreadPool
>>> import dask
>>> dask.set_options(get=dask.threaded.get, pool=ThreadPool(4))

>>> import iris
>>> # Iris processing here...

.. note::
These dask processing options will last for the lifetime of the Python session
and must be re-applied in other or subsequent sessions.

Other dask processing options are also available. See the
are computed. This is provided via the ``dask.set_options`` interface. See the
`dask documentation <http://dask.pydata.org/en/latest/scheduler-overview.html>`_
for more information on setting dask processing options.

4 changes: 1 addition & 3 deletions docs/iris/src/whatsnew/2.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ Iris 2.0.0 Features

In particular, Dask's *threaded*, *multiprocessing* or *distributed*
`schedulers`_ can be used in order to best utilise available compute and
memory resource.

For fuller details, see :doc:`Real and Lazy Data
memory resource. For further details, see :doc:`Real and Lazy Data
</userguide/real_and_lazy_data>`.

* Changes to the :class:`iris.cube.Cube`:
Expand Down