Monitor subprocesses to avoid zombies #18218

awaelchli · 2023-08-02T16:38:35Z

What does this PR do?

This PR adds a process observer thread that monitors the child processes launched with the SubprocessScriptLauncher. The observer periodically checks whether the processes have terminated or not. If any processes have non-zero exit code, the thread will forcefully shutdown all other processes. Note that this only quality-of-life improvement applies only to our special process launcher from Lightning for single-node use. For multi-node training, we still rely on the external process launcher (SLURM, torchelastic, etc.) to handle this.

Fixes #16410
Fixes #16518

Here is a simple example script if you want to test it yourself. It crashes on rank 1 and hangs on rank 0. On this branch, the processes will be cleaned up properly.

import torch
from torch.utils.data import DataLoader, Dataset

from lightning.pytorch import LightningModule, Trainer


class RandomDataset(Dataset):
    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def training_step(self, batch, batch_idx):
        loss = self(batch).sum()
        self.log("train_loss", loss)
        if batch_idx > 5 and self.global_rank == 3:
            raise RuntimeError("Rank 2 exit")
        return {"loss": loss}

    def configure_optimizers(self):
        return torch.optim.SGD(self.layer.parameters(), lr=0.1)


train_data = DataLoader(RandomDataset(32, 64), batch_size=2)
model = BoringModel()
trainer = Trainer(
    max_epochs=100,
    devices=4,
    accelerator="cpu",
    strategy="ddp",
)
trainer.fit(model, train_dataloaders=train_data)

cc @Borda @carmocca @justusschock @awaelchli

src/lightning/fabric/strategies/launchers/base.py

for more information, see https://pre-commit.ci

…feature/monitor-processes

for more information, see https://pre-commit.ci

…feature/monitor-processes

for more information, see https://pre-commit.ci

…feature/monitor-processes

github-actions · 2023-08-02T21:11:35Z

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow

Check ID	Status
pl-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
pl-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
pl-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
pl-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
pl-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
pl-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
pl-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
pl-cpu (macOS-11, pytorch, 3.8, 1.13)	success	✅
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13)	success	✅
pl-cpu (windows-2022, pytorch, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py.

🟢 pytorch_lightning: Azure GPU

Check ID	Status
pytorch-lightning (GPUs)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/launchers/subprocess_script.py, tests/tests_pytorch/strategies/launchers/test_subprocess_script.py, src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py.

🟢 pytorch_lightning: Benchmarks

Check ID	Status
lightning.Benchmarks	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py.

🟢 fabric: Docs

Check ID	Status
make-doctest (fabric)	success	✅
make-html (fabric)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py.

🟢 pytorch_lightning: Docs

Check ID	Status
make-doctest (pytorch)	success	✅
make-html (pytorch)	success	✅

These checks are required after the changes to src/lightning/pytorch/strategies/launchers/subprocess_script.py.

🟢 lightning_fabric: CPU workflow

Check ID	Status
fabric-cpu (macOS-11, lightning, 3.8, 1.11)	success	✅
fabric-cpu (macOS-11, lightning, 3.9, 1.12)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 1.13)	success	✅
fabric-cpu (macOS-11, lightning, 3.10, 2.0)	success	✅
fabric-cpu (macOS-11, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.9, 1.12)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.10, 2.0)	success	✅
fabric-cpu (ubuntu-20.04, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11)	success	✅
fabric-cpu (windows-2022, lightning, 3.9, 1.12)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 1.13)	success	✅
fabric-cpu (windows-2022, lightning, 3.10, 2.0)	success	✅
fabric-cpu (windows-2022, lightning, 3.8, 1.11, oldest)	success	✅
fabric-cpu (macOS-11, fabric, 3.8, 1.13)	success	✅
fabric-cpu (ubuntu-20.04, fabric, 3.8, 1.13)	success	✅
fabric-cpu (windows-2022, fabric, 3.8, 1.13)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, tests/tests_fabric/strategies/launchers/test_subprocess_script.py.

🟢 lightning_fabric: Azure GPU

Check ID	Status
lightning-fabric (GPUs)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, tests/tests_fabric/strategies/launchers/test_subprocess_script.py.

🟢 mypy

Check ID	Status
mypy	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py.

🟢 install

Check ID	Status
install-pkg (ubuntu-22.04, app, 3.8)	success	✅
install-pkg (ubuntu-22.04, app, 3.10)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.8)	success	✅
install-pkg (ubuntu-22.04, fabric, 3.10)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.8)	success	✅
install-pkg (ubuntu-22.04, pytorch, 3.10)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.8)	success	✅
install-pkg (ubuntu-22.04, lightning, 3.10)	success	✅
install-pkg (ubuntu-22.04, notset, 3.8)	success	✅
install-pkg (ubuntu-22.04, notset, 3.10)	success	✅
install-pkg (macOS-12, app, 3.8)	success	✅
install-pkg (macOS-12, app, 3.10)	success	✅
install-pkg (macOS-12, fabric, 3.8)	success	✅
install-pkg (macOS-12, fabric, 3.10)	success	✅
install-pkg (macOS-12, pytorch, 3.8)	success	✅
install-pkg (macOS-12, pytorch, 3.10)	success	✅
install-pkg (macOS-12, lightning, 3.8)	success	✅
install-pkg (macOS-12, lightning, 3.10)	success	✅
install-pkg (macOS-12, notset, 3.8)	success	✅
install-pkg (macOS-12, notset, 3.10)	success	✅
install-pkg (windows-2022, app, 3.8)	success	✅
install-pkg (windows-2022, app, 3.10)	success	✅
install-pkg (windows-2022, fabric, 3.8)	success	✅
install-pkg (windows-2022, fabric, 3.10)	success	✅
install-pkg (windows-2022, pytorch, 3.8)	success	✅
install-pkg (windows-2022, pytorch, 3.10)	success	✅
install-pkg (windows-2022, lightning, 3.8)	success	✅
install-pkg (windows-2022, lightning, 3.10)	success	✅
install-pkg (windows-2022, notset, 3.8)	success	✅
install-pkg (windows-2022, notset, 3.10)	success	✅

These checks are required after the changes to src/lightning/fabric/strategies/launchers/base.py, src/lightning/fabric/strategies/launchers/subprocess_script.py, src/lightning/pytorch/strategies/launchers/subprocess_script.py.

🟢 link-check

Check ID	Status
check-md-links / markdown-link-check	success	✅

These checks are required after the changes to src/lightning/fabric/CHANGELOG.md, src/lightning/pytorch/CHANGELOG.md.

Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

…feature/monitor-processes

carmocca

Do you plan to add this to the spawn launchers too?

I would also update the existing PyTorch logic (added in #16525) to use the same code

src/lightning/fabric/strategies/launchers/subprocess_script.py

awaelchli added 4 commits August 2, 2023 18:04

POC

e7f4b60

remove unneeded base

da94700

send signal

7af76a0

change default

bc295cc

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Aug 2, 2023

awaelchli added feature Is an improvement or enhancement strategy: ddp DistributedDataParallel performance and removed fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Aug 2, 2023

awaelchli added this to the 2.1 milestone Aug 2, 2023

awaelchli commented Aug 2, 2023

View reviewed changes

src/lightning/fabric/strategies/launchers/base.py Show resolved Hide resolved

[pre-commit.ci] auto fixes from pre-commit.com hooks

af95753

for more information, see https://pre-commit.ci

github-actions bot added fabric lightning.fabric.Fabric pl Generic label for PyTorch Lightning package labels Aug 2, 2023

awaelchli and others added 13 commits August 2, 2023 18:43

refactor

d72e7f7

Merge remote-tracking branch 'origin/feature/monitor-processes' into …

54f6be3

…feature/monitor-processes

[pre-commit.ci] auto fixes from pre-commit.com hooks

42a63cc

for more information, see https://pre-commit.ci

validation

3d5b58f

Merge remote-tracking branch 'origin/feature/monitor-processes' into …

cfadfec

…feature/monitor-processes

refactor

a03e7d4

tests

d401bbc

[pre-commit.ci] auto fixes from pre-commit.com hooks

735fae6

for more information, see https://pre-commit.ci

sigkill

db08f0f

Merge remote-tracking branch 'origin/feature/monitor-processes' into …

939cbeb

…feature/monitor-processes

fix typing

0a1ea11

changelog

8835c77

Merge branch 'master' into feature/monitor-processes

ade4951

awaelchli marked this pull request as ready for review August 2, 2023 21:11

awaelchli requested review from carmocca, justusschock, Borda and williamFalcon as code owners August 2, 2023 21:11

awaelchli added 3 commits August 2, 2023 23:14

add test

2383023

Merge remote-tracking branch 'origin/feature/monitor-processes' into …

11220f5

…feature/monitor-processes

handle windows

593a559

Borda approved these changes Aug 3, 2023

View reviewed changes

justusschock approved these changes Aug 3, 2023

View reviewed changes

mergify bot added the ready PRs ready to be merged label Aug 3, 2023

Borda merged commit 7e13eb7 into master Aug 8, 2023

Borda deleted the feature/monitor-processes branch August 8, 2023 07:25

awaelchli mentioned this pull request Aug 8, 2023

Improve info message of process observer with rank information #18257

Merged

7 tasks

carmocca reviewed Aug 8, 2023

View reviewed changes

src/lightning/fabric/strategies/launchers/subprocess_script.py Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monitor subprocesses to avoid zombies #18218

Monitor subprocesses to avoid zombies #18218

awaelchli commented Aug 2, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Aug 2, 2023 •

edited

Loading

carmocca left a comment

Monitor subprocesses to avoid zombies #18218

Monitor subprocesses to avoid zombies #18218

Conversation

awaelchli commented Aug 2, 2023 • edited by github-actions bot Loading

What does this PR do?

github-actions bot commented Aug 2, 2023 • edited Loading

⚡ Required checks status: All passing 🟢

Groups summary

carmocca left a comment

Choose a reason for hiding this comment

awaelchli commented Aug 2, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Aug 2, 2023 •

edited

Loading