-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
allow multiprocess dep instead of multiprocessing #94
base: master
Are you sure you want to change the base?
Conversation
I wonder why CI didn't trigger. |
Hey @lalo, sorry for being out so long. from pypeln import config
import multiprocess
config.set_multiprocessing_impl(multiprocess) |
I'll take a look |
@@ -18,13 +23,13 @@ | |||
|
|||
class IterableQueue(Queue, tp.Generic[T], tp.Iterable[T]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this inheritance of Queue might complicate things
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typehints should stay the same, the important thing would be to use a different module when initializing Queue
and Process
at runtime.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
heres a wip impl #100
multiprocess external lib has other benefits like using dill instead of pickle, allowing us more leeway on certain edge cases that are not compatible with native multiprocessing.
https://github.com/uqfoundation/multiprocess
from their readme:
Let me know your thoughts on this type of change. Happy to iterate on it.
Thanks
Related: #53