[ray_client]: Add more retry logic

[barakmich]

Related issue number

Closes #13446

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

Do we need three different retry loops for the different stages? Why not have just a single one that tests this last condition?

[ericl]

Comment on simplifying

[barakmich]

Because it could fail at any one of these stages.

Now, one big retry loop would look like a finite state machine of

while retrying:
   if in step A:
   if in step B:
   if in step C:

But we can't "test this last condition" alone. If this is what you'd prefer, I'll go do that.

[ericl]

Why can't we test the last condition alone? C implies {A, B} are fine right? I think a loop that just tests C would be sufficient. If it's for better error messages, we could also check {A, B} prior to C in the loop.

[ericl]

Btw, it would be good to have a test here

[barakmich]

Sure.

Removed state machine and added test.

[ericl]

//python/ray/tests:test_client                                          TIMEOUT in 3 out of 3 in 75.0s
  Stats over 3 runs: max = 75.0s, min = 75.0s, avg = 75.0s, dev = 0.0s
  /home/travis/.cache/bazel/_bazel_travis/b88c129a127452fc94033a29d9f90e20/execroot/com_github_ray_project_ray/bazel-out/k8-opt/testlogs/python/ray/tests/test_client/test.log
  /home/travis/.cache/bazel/_bazel_travis/b88c129a127452fc94033a29d9f90e20/execroot/com_github_ray_project_ray/bazel-out/k8-opt/testlogs/python/ray/tests/test_client/test_attempts/attempt_1.log
  /home/travis/.cache/bazel/_bazel_travis/b88c129a127452fc94033a29d9f90e20/execroot/com_github_ray_project_ray/bazel-out/k8-opt/testlogs/python/ray/tests/test_client/test_attempts/attempt_2.log

[ericl]

Tests / lint still failing

[barakmich]

I'm starting to run out of ideas, but we'll see if this goes

[barakmich]

I think I figured it out

[barakmich]

This time for sure -- I got the reproduction locally and tracked it down