[autoscaler][local node provider] Make sure docker state is completely cleaned up

[DmitriGekhtman]

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS):

Some users report trouble due to incomplete docker clean-up with the local node provider.
See discussion here:
https://discuss.ray.io/t/how-do-i-troubleshoot-nodes-that-remaining-uninitialized/2991/5
Might need some LocalNodeProvider-specific termination logic to fix that. Related to #17022.

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[olly-writes-code]

This still seems to be a problem :/

[DmitriGekhtman]

I believe local node provider has been deprecated for a while, so this issue can probably be closed.

[DmitriGekhtman]

Cc @anyscalesam

[DmitriGekhtman]

Correction: local node provider is not yet officially deprecated. It just hasn't been maintained in a while. Fine to keep this issue open.

For anyone else who stumbles on this thread: For on-premises clusters, it's simpler to start Ray by manually ssh-ing ray start into each of the nodes in the cluster.

[olly-writes-code]

That defeats the point of using Ray. There are much simpler alternatives if Ray can't handle orchestration.

[DmitriGekhtman]

Cluster management is not a feature of Ray itself.
To manage infrastructure for Ray, the best options are to use a cluster manager (Kubernetes and KubeRay) or the managed solution (Anyscale).

But yeah, don't use Ray if there's a simpler solution for your application.