Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/source/resilience.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,12 @@ This has some fail cases.
causes a segmentation fault, then that bad function will repeatedly be
called on other workers. This function will be marked as "bad" after it
kills a fixed number of workers (defaults to three).
3. Data scattered out to the workers is not kept in the scheduler (it is
often quite large) and so the loss of this data is irreparable. You may
wish to call ``Client.replicate`` on the data with a suitable replication
factor to ensure that it remains long-lived or else back the data off of
some resilient store, like a file system.
3. Data sent out directly to the workers via a call to ``scatter()`` (instead
of being created via a call to ``persist()`` from a Dask task graph) is not

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This parenthetical is slightly troublesome because the persist function is only one of a few methods that will trigger computation (client.compute, submit, map, ...)

kept in the scheduler, as it is often quite large, and so the loss of this
data is irreparable. You may wish to call ``Client.replicate`` on the data
with a suitable replication factor to ensure that it remains long-lived or
else back the data off of some resilient store, like a file system.


Hardware Failures
Expand Down
4 changes: 2 additions & 2 deletions docs/source/scheduling-state.rst
Original file line number Diff line number Diff line change
Expand Up @@ -131,8 +131,8 @@ Conversely, "saturated" workers may see their workload lightened through
Client State
------------

Information about each individual client is kept in a :class:`ClientState`
object:
Information about each individual client of the scheduler is kept
in a :class:`ClientState` object:

.. autoclass:: ClientState

Expand Down
6 changes: 6 additions & 0 deletions docs/source/worker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,12 @@ YARN, Mesos, SGE, etc..). After termination the nanny will restart the worker
in a fresh state.


Nanny
~~~~~

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Dask workers are often launched, monitored, and managed by a small Nanny process.

Then, I would add more context into the Nanny docstring itself. In particular, we should probably add a See Also section that links to the worker, and we should mention that the parameters to Nanny are mostly the same as the parameters that are in the Worker.

.. autoclass:: distributed.worker.Nanny


API Documentation
-----------------

Expand Down