Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ray] Redis has started but no raylets have registered yet error when connecting to a ray cluster in a private kuberenetes cluster #7050

Open
jCrompton opened this issue Feb 4, 2020 · 5 comments
Labels
fix-error-msg This issue has a bad error message that should be improved. needs-repro-script Issue needs a runnable script to be reproduced question Just a question :)

Comments

@jCrompton
Copy link
Contributor

I'm trying to connect to a ray (0.8.0) cluster in my private kuberenetes cluster (using the yaml file provided in the docs) but I get a redis authentication error. I have the ray head node exposed as a NodePort, what am I missing?

Ray version 0.8.0

@jCrompton jCrompton added the question Just a question :) label Feb 4, 2020
@jCrompton jCrompton changed the title Redis authentication error when connecting to a ray cluster in a private kuberenetes cluster [ray] Redis authentication error when connecting to a ray cluster in a private kuberenetes cluster Feb 5, 2020
@jCrompton
Copy link
Contributor Author

Since I'm on 0.8.0 I had to pass in the default redis password, on 0.9.0 I was able to connect without a password but my redis cluster thinks no raylets have been registered (very similar to this issue #5437) exception:

Exception: Redis has started but no raylets have registered yet.

How do I resolve this in a kubernetes yaml file though, I need the ray-head nodes' public IP in the config but there is no environment variable for the public IP so is the only solution to start the container shell into the container and then run the ray start command with the determined public IP?

@jCrompton jCrompton changed the title [ray] Redis authentication error when connecting to a ray cluster in a private kuberenetes cluster [ray] Redis has started but no raylets have registered yet error when connecting to a ray cluster in a private kuberenetes cluster Feb 10, 2020
@jCrompton
Copy link
Contributor Author

Any updates on this, I have tried in ray v0.9.dev too with the same issue

@jCrompton
Copy link
Contributor Author

Pleasseeeeeeee

@ericl ericl added the needs-repro-script Issue needs a runnable script to be reproduced label Mar 19, 2020
@stale
Copy link

stale bot commented Nov 13, 2020

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

  • If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
  • If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

@stale stale bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Nov 13, 2020
@richardliaw richardliaw added fix-error-msg This issue has a bad error message that should be improved. P2 Important issue, but not time-critical labels Nov 15, 2020
@stale stale bot removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Nov 15, 2020
@richardliaw
Copy link
Contributor

@jCrompton did you figure this out?

@richardliaw richardliaw added this to the Better Error Messages milestone Nov 15, 2020
@rkooo567 rkooo567 removed the P2 Important issue, but not time-critical label Oct 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix-error-msg This issue has a bad error message that should be improved. needs-repro-script Issue needs a runnable script to be reproduced question Just a question :)
Projects
None yet
Development

No branches or pull requests

4 participants