[Feature Request] implement MultiprocessingLauncher

[Jasha10]

We might consider implementing a launcher that employs Python's multiprocessing module (or some other simple mechanism from Python's standard library) for launching jobs in separate processes. This would be useful in multirun mode for the following reasons:

• concurrency: such a launcher would enable process parallelism (unlike Hydra's BasicLauncher, which runs jobs sequentially)
• isolation: Running each job in a separate process would prevent state from leaking between jobs (unlike BasicLauncher, which runs all jobs in the same process).

Concurrency and isolation are already possible via e.g. the Joblib launcher (as pointed out in discussion #2186) or the other advanced launchers (RayLauncher, SubititLauncher, RQLauncher). As I see it, the main advantages of a MultiprocessingLauncher over something like JoblibLauncher would be (1) simplicity and (2) less reliance on third-party packages (joblib).

[maxfrei750]

I'd like to add that the current workaround (see discussion #2186) does not result in a prefect run isolation, but will require jobs to be paired, due to a limitation of joblib (see joblib/joblib#1294).

[jbaczek]

Hi, what is the status of this FR?
In my case I have some objects created by PyTorch like CUDA context or caching allocator which are destroyed at the end of a process. According to this discussion it is impossible to destroy CUDA context programmatically. Caching allocator together with garbage collector are hard to control and sometimes they leave memory allocated even when torch.cuda.empty_cache() is called.
I need to run my trainings in isolation to prevent memory leaks and OOMs in subsequent trainings.
In joblib documentation there is a claim that they are able to save tenths of a second per process spawn which is irrelevant in my case.

[Jasha10]

Hi @jbaczek,

Deallocating pytorch CUDA memory is a compelling use-case, and this is an issue I've faced as well.

I'm not sure if we'll have the bandwidth to implement this feature in time for Hydra 1.3. I'll add a tentative v1.3 milestone.

If you have time to work on this, a PR would be welcome.
I think that, since multiprocessing is part of the standard library, we can include the proposed multiprocessing_launcher in the hydra/_internal/core_plugins directory instead of creating a new folder under the plugins directory.

[Zhylkaaa]

Hi @Jasha10, I was experimenting with ProcessPoolExecutor during my work on above mentioned PR. You cant have purely multiprocessing launcher without any 3rd party dependencies because you need to pickle methods defined in main and lambdas.
This can be achived by hacking pickle module of multiprocessing to use dill or cloudpickle: eg. https://stackoverflow.com/questions/19984152/what-can-multiprocessing-and-dill-do-together

EDIT: after some experiments I think the best chance of doing this is using cloudpickle (used internally by joblib) and multiprocessing. Only issue is how to implement waiting for AsyncResults, but I have few ideas. Anyway is it ok with you to make hydra depend on cloudpickle? We can probably write this plugin by Monday (including integration with proposition from #2461)

EDIT2: we are testing some solutions, are you interested? @Jasha10

[Zhylkaaa]

Hi, I was experimenting with multiprocessing and it's capabilities and limits. Also with our implementation of it and found few issue you might like to know before considering adding it to core (or even plugins).
In short:

• you need to maintain your own version of Pool that doesn't spawn daemons (which is not that bad)
• multiprocessing have known issues with handling out of ram errors (or even simple pool.apply_async(sys.exit, (1,))) which you can overcome, but at a great price.

If you want to have some more details I can offer an online meeting. (you probably have my contacts in CLA :))

[Jasha10]

Hi @Zhylkaaa,

Thanks for looking into this! I sincerely apologize for the delayed reply.

after some experiments I think the best chance of doing this is using cloudpickle (used internally by joblib)

This is consistent with the solution used by several of Hydra's other launcher plugins.

Anyway is it ok with you to make hydra depend on cloudpickle?

To guarantee long-term stability, I think it's best for hydra-core not to depend on cloudpickle. Currently we only depend on packages like antlr/pyyaml/omegaconf that are essential to Hydra's business logic.

We can create a new folder under hydra's plugins directory, however.

If you want to have some more details I can offer an online meeting. (you probably have my contacts in CLA :))

Thanks! Unfortunately I don't have access to the CLA -- I suspect I'd need to go through Meta's legal department to get that :P
Could you please send me an email at [email protected] ?