Add subprocess liveness monitor to detect scheduler crashes

[Simon-Li]

Summary

Adds a subprocess liveness watchdog to detect scheduler crashes that bypass Python exception handlers (e.g., NCCL timeout causing C++ std::terminate()).

Problem: When a scheduler subprocess crashes at the C++ level, std::terminate() runs before Python exception handlers, leaving the main process unaware. The service continues accepting requests but cannot process them ("zombie service").

Solution: A daemon thread in Engine polls proc.is_alive() every 1 second. On detecting an abnormal exit (exitcode != 0), it sends SIGQUIT to trigger existing cleanup infrastructure.

Engine (Main Process)
  │
  └── SubprocessWatchdog (daemon thread)
        polls is_alive() every 1s
        on crash (exitcode != 0): os.kill(SIGQUIT)
        │
        ├── scheduler_0
        ├── scheduler_1
        └── detokenizer

Changes

File	Change
python/sglang/srt/utils/watchdog.py	+63 lines: SubprocessWatchdog class
python/sglang/srt/entrypoints/engine.py	+24 lines: create/store/stop watchdog
python/sglang/srt/entrypoints/http_server.py	+1 line: unpack return value
python/sglang/srt/ray/http_server.py	+1 line: unpack return value
test/registered/unit/utils/test_subprocess_monitor.py	+213 lines: 6 test cases

Note: Zero changes to tokenizer_manager.py.

Design Decisions

1. Placed in watchdog.py: Reuses the same daemon thread + SIGQUIT pattern as existing WatchdogRaw
2. Owned by Engine: All process handles available in _launch_subprocesses(), no cross-module passing needed
3. Direct SIGQUIT: Leverages existing signal handling infrastructure, thread-safe

Edge Cases Handled

• Normal exit (exitcode == 0) does NOT trigger SIGQUIT
• shutdown() stops watchdog before killing process tree (prevents false positives)
• Empty process list handled gracefully
• Multi-node: non-zero rank nodes get None watchdog
• Ray backend: only monitors detokenizer (schedulers are actors)

Test Plan

• test_healthy_processes_no_callback - healthy processes don't trigger
• test_crashed_process_triggers_sigquit - delayed crash detected
• test_immediate_crash_detection - immediate crash detected
• test_multiple_processes_one_crashes - one crash in multi-process triggers
• test_empty_processes_list - empty list handled
• test_normal_exit_no_sigquit - normal exit (exitcode=0) doesn't trigger

Fixes #18421

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Simon-Li]

Hi @Ying1123 @CatherineSue , this PR adds subprocess liveness monitoring to detect scheduler crashes caused by
NCCL timeout / C++ std::terminate() (issue #18421). Would appreciate your review when you have a chance — happy to adjust based on feedback. Thanks!

[Simon-Li]

/tag-run-ci-label

[alphabetc1]

/tag-and-rerun-ci

[alphabetc1]

/rerun-failed-ci

[alphabetc1]

/rerun-failed-ci 1

[Simon-Li]

Hi @hnyls2002 , I've addressed all the review comments. Could you please take another look when you have a chance? Thanks!

[hnyls2002]

/rerun-failed-ci

[alphabetc1]

/rerun-failed-ci

[Simon-Li]

Hi @hnyls2002 , I noticed the CI failures in the GPT OSS quantization tests (test_bf16_20b and test_mxfp4_20b) appear to be unrelated to this PR. Could you please take another look when you have a chance? Thanks!

[alphabetc1]

/rerun-failed-ci