Support DDPPlugin to be used on CPU

[awaelchli]

What does this PR do?

Before 1.2, the setting accelerator="ddp_cpu" would use the DDPSpawnAccelerator and today it is the DDPSpawnPlugin.
However, after the refactor we can also support DDPPlugin (non-spawn) to be used with cpu.

This can be interpreted both as a bugfix or a added feature.

# this always worked, still works, would use spawn method
Trainer(accelerator="ddp_cpu", num_processes=2)

# this now works too, uses spawn method
Trainer(accelerator="ddp_cpu", num_processes=2, plugins=[DDPSpawnPlugin(find_unused_parameters=True)])

# this now works, uses subprocess ddp
Trainer(accelerator="ddp_cpu", num_processes=2, plugins=[DDPPlugin(find_unused_parameters=True)])

Fixes #6121
Related #7810

Introduces a new conftest fixture to all tests to address the following problem:
All ddp tests that init a process group have global side effects, since the process group is a global state in the pytorch library. It means the process group can carry over from one test to the other. This leads to problems when the world size changes or the distributed backend changes from e.g. gloo to nccl, leading to broken pipe and hangs. We address this by force killing the process group after every test. A different teardown approach will be explored in #8080.

Additionally, some test refactoring changes are necessary.
"Special tests" execute one new process per test, but this includes the parameterization. This is not good. We want that each parameterization is it's own new independent run. Otherwise we will just end up with subprocesses trying to launch new processes.

Before submitting

• Was this discussed/approved via a GitHub issue? (not for typos and docs)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you update the CHANGELOG? (not for typos, docs, test updates, or internal minor changes/refactorings)

PR review

Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read Review guidelines. In short, see the following bullet-list:

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified

Did you have fun?

No, not on this one.

[codecov]

Codecov Report

Merging #6208 (5b40e21) into master (baa7de2) will decrease coverage by 5%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #6208    +/-   ##
=======================================
- Coverage      93%     88%    -5%     
=======================================
  Files         212     212            
  Lines       13701   13697     -4     
=======================================
- Hits        12741   12054   -687     
- Misses        960    1643   +683     

[justusschock]

I like it. Can we maybe raise a Warning for

 # this always worked, still works, would use spawn method
Trainer(accelerator="ddp_cpu", num_processes=2)

That this behavior might change in the future in favor of ddp and people should use

Trainer(accelerator="ddp_cpu_spawn", num_processes=2)

instead?

Also in

# this now works too, uses spawn method
Trainer(accelerator="ddp_cpu", num_processes=2, plugins=[DDPSpawnPlugin(find_unused_parameters=True)])

# this now works, uses subprocess ddp
Trainer(accelerator="ddp_cpu", num_processes=2, plugins=[DDPPlugin(find_unused_parameters=True)])

specifying the accelerator is not necessary when specifying the plugin and vice versa, right? So when I do Trainer(num_processes=2, plugins=[DDPPlugin()]) this should work if I don't specify any GPUs?

[Borda]

@awaelchli still in progress?

[awaelchli]

I have some problems with RPC tests failing, when I am able to fix the issue, it will be ready for review.

[carmocca]

Removing "Ready to go" as tag as there are still some hangs :(

[tchaton]

LGTM !