Private/onprem clusters always need explicit ssh_private_key in docker

[AmeerHajAli]

When starting private clusters without docker, it is not necessary to provide the ssh_private_key as the user has authorized access to the nodes. But when adding docker, the head node process inside the container does not have privileges to ssh to the other nodes and hence needs the ssh_private_key to be provided explicitly.

I think it is fine to leave it as is but just wanted to bring it up.

[ijrsvt]

If the key is available on the host at /path/to/key/key on the host a quick fix would be to add a docker file mount to docker::run_options such as

run_options:
 -  "-v /path/to/key/key:root/key"

[jmakov]

When starting private clusters without docker, it is not necessary to provide the ssh_private_key as the user has authorized access to the nodes. But when adding docker, the head node process inside the container does not have privileges to ssh to the other nodes and hence needs the ssh_private_key to be provided explicitly.

I think it is fine to leave it as is but just wanted to bring it up.

This should really be documented...

[AmeerHajAli]

@DmitriGekhtman , what do you suggest we do here?

[DmitriGekhtman]

@AmeerHajAli @ijrsvt why is this an issue for on-prem clusters but not for cloud clusters? In both cases, the head needs ssh access to workers.

[DmitriGekhtman]

Oh I see, it's a documentation problem. I think we could add an example-docker.yaml or something with this info to the local cluster examples.

Or modify example-full to use docker -- my main hesitation there is that variable size clusters don't really work right now with docker, also currently we're not careful enough to clean up docker state when we're done using Ray on an on-prem node.
#17689

[ijrsvt]

why is this an issue for on-prem clusters but not for cloud clusters?

For cloud clusters we auto-insert this field!
AWS:

ray/python/ray/autoscaler/_private/aws/config.py

Line 384 in ac0a153

config["auth"]["ssh_private_key"] = key_path

GCP:

ray/python/ray/autoscaler/_private/gcp/config.py

Line 453 in ac0a153

config["auth"]["ssh_private_key"] = private_key_path

[DmitriGekhtman]

Right, we try to auto-configure a key for cloud providers.

But for cloud users who supply a key manually, the situation should be the same as on-prem users who supply a key?