Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the import thread from the workers and driver. #951

Open
robertnishihara opened this issue Sep 8, 2017 · 2 comments
Open

Remove the import thread from the workers and driver. #951

robertnishihara opened this issue Sep 8, 2017 · 2 comments
Labels
P3 Issue moderate in impact or severity

Comments

@robertnishihara
Copy link
Collaborator

Right now the import thread fetches remote function definitions, actor definitions, custom class definitions (needed for custom serialization callbacks). It should be possible to get rid of the extra thread and to simply fetch these values from Redis when they are needed (though that could add latency the first time a remote function is invoked on a worker).

This would also address an issue brought up in #824.

@imzhenyu
Copy link
Contributor

imzhenyu commented Sep 9, 2017

Thx, @robertnishihara . This actually reminds me another problem - do we have GC capability in python for remote functions? I.e., whether we support long running services for people continuously submitting new ray jobs (we did that on the java side and we want to co-deploy the python as well, as a shared ray job cluster)?

@robertnishihara
Copy link
Collaborator Author

Currently there is no GC for remote functions in Redis (or in the Python worker processes), but it would not too challenging to associate each remote function with the ID of the driver that defined it and to remove it when the driver exits.

I just created issue #958 to track this.

@edoakes edoakes added the P3 Issue moderate in impact or severity label Mar 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 Issue moderate in impact or severity
Projects
None yet
Development

No branches or pull requests

3 participants