[tune] Cluster Fault Tolerance

[richardliaw]

What do these changes do?

A redo of #3165 with extraneous cleanup changes removed.

This currently does not use the same restoring code-path as #3238, but this can change later when component FT is implemented... (i.e., this doesn't notify components that some trials go RUNNING -> PENDING).

This adds the following functionality:

• pickleable trials and TrialRunner.
• checkpointing/restoring functionality for Trial runner
• user endpoints for experiment checkpointing

Example:

In [6]: import time
   ...: import ray
   ...: from ray import tune
   ...:
   ...: ray.init()
   ...:
   ...: kwargs = dict(
   ...:     run="__fake",
   ...:     stop=dict(training_iteration=5),
   ...:     checkpoint_freq=1,
   ...:     max_failures=1)
   ...:
   ...: # This will save the experiment state to disk on each step
   ...: tune.run_experiments(
   ...:     dict(experiment1=kwargs),
   ...:     raise_on_failed_trial=False)
   ...:

TODO:

• User endpoints implemented.
• NODE FT: Add test for scheduler notification when nodes die and trials running -> pending

NOTE: this should be a lot easier to review after #3414 is merged.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10432/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10433/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10435/
Test FAILed.

[ericl]

Works great now! I also patched rllib's train.py to only resume on --resume.

It might be a bit aggressive to enable this by default, but we can always default it to false later, and I think this will help a lot for visibility.

[ericl]

@old-bear any comments on the changes to the trial_executor interface?

[ericl]

Also nit: "This will ignore any new changes to specification" isn't grammatically correct.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10452/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10456/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10471/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10473/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10476/
Test PASSed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10479/
Test FAILed.

[richardliaw]

jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10481/
Test FAILed.

[richardliaw]

jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10487/
Test FAILed.

[richardliaw]

Merging this since:

• travis tests pass
• on jenkins, tests are hanging on SGD (which is not related to this PR it seems), but rest of the tests pass and the only tests after the SGD portion are either Ray multi-node or pytorch, so this should be unrelated

@ericl, thanks for the multiple rounds of extensive review!

[hartikainen]

Nice, this helps a lot! Thanks guys!