Config instantiation is not DDP-aware

[surajpaib]

Because config is instantiated only on the main process and then DDP is initialized, objects that are initialized in the config are not DDP aware.

Maybe there is a way to fix this by instantiating trainer first and separate and then the system

[ibro45]

I think this isn't an issue anymore, right? Closing.

[surajpaib]

@ibro45 This is still an issue.

Now I needed this for Lightning-AI/pytorch-lightning#11763

But since at instantiation, objects are not DDP aware, this problem persists.

[ibro45]

Any idea on how to approach it?

[ibro45]

Wait how is this a lighter problem? I think it has to do with improper usage/behavior of CacheDataset only

[ibro45]

Would swapping the order of trainer and system help?

lighter/lighter/utils/runner.py

Line 92 in 5d60768

trainer = parser.get_parsed_content("trainer")