Support slowly arriving workers in SpecCluster - #2904
Merged
Conversation
Previously SpecCluster waited until all workers had checked in with the scheduler. This made sense for LocalCluster or SSHCluster because there isn't really a significant delay in starting things that we can't control. However, for other systems like dask-jobqueue or dask-kubernetes workers might not ever start, so we need a different system. Now, SpecCluster still awaits the Worker object that it is passed, but doesn't require that the worker has started in the scheduler. We now expect awaiting to mean *"We have successfully handed control of starting the worker to some other robust system"* Our job at this point is done. We hope that the worker arrives, but from our perspective this local Worker object is awaited and "running". This commit also includes a minimal example of a Worker class, `SlowWorker`, that serves as a nice minimal example for what SpecCluster expects.
This commit does two things:
1. We wait to shutdown the executor a little longer in case it is still
in use
2. The worker no longer asks the Nanny to terminate it. Instead it
asks the nanny to shutdown gracefully after it is gone, and then
continues closing itself as normal.
Member
Author
|
@jcrist would you mind reviewing the adaptive_core.py file? I think that it abstracts away all of the adaptive logic from assumptions about the cluster class. It would be both to get your perspective as a controls person, and also as someone who might want to connect this logic to another cluster class type (or two) in the future. |
Member
Author
|
Moving ahead with this. I'm more than happy to change things after merging. |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously SpecCluster waited until all workers had checked in with the
scheduler. This made sense for LocalCluster or SSHCluster because
there isn't really a significant delay in starting things that we can't
control. However, for other systems like dask-jobqueue or
dask-kubernetes workers might not ever start, so we need a different
system.
Now, SpecCluster still awaits the Worker object that it is passed, but
doesn't require that the worker has started in the scheduler. We now
expect awaiting to mean
"We have successfully handed control of starting the worker to some other robust system"
Our job at this point is done. We hope that the worker arrives, but
from our perspective this local Worker object is awaited and "running".
This commit also includes a minimal example of a Worker class,
SlowWorker, that serves as a nice minimal example for what SpecClusterexpects.
I plan to do a bit more work here and review our Adaptive policies when workers
may take a while. I hope that this allows us to replicate/replace a lot of the
fine work that @jhamman did over at dask-jobqueue.
cc @jcrist @jacobtomlinson