Fix Ray client GOAWAY

[pcmoritz]

Why are these changes needed?

When Ray client is proxied over nginx (e.g. via the Kubernetes nginx ingress) and the configuration reloaded, the proxy sends a GOAWAY to the client, which resets the state of the stream to IDLE. Our code interprets this as a disconnect, but in reality the underlying streams seems to still be functional.

Related issue number

Fixes a problem with GOAWAYs that was introduced in #13386

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ckw017]

Can merge these changes to propagate the error correctly (get rid of _pickle.UnpicklingError: invalid load key, '%'): ckw017@09226a1

Other notes:

• This fix will probably just propagate the internal error seen on the server side logs. I'm guessing "INTERNAL" in this context means "Flow-control protocol violation" mentioned in here: https://grpc.github.io/grpc/core/md_doc_statuscodes.html
• This fix might be worthwhile. Should be noted that on the next RPC it will technically transition from IDLE -> CONNECTING -> READY, so CONNECTING might also be a valid "is_connected" state after the initial connection is resolved, (can add an additional flag)

[ericl]

Failing LINT and other tests (merge master?)

[ericl]

Do we still want this?

[pcmoritz]

Yeah I think it is not really needed any more. It was most likely due to a protocol error in nginx and if that error happens other things are broken too so it doesn't really help...