Graceful shutdown with SIGTERM for child processes

[chenkaiyue]

Motivation

When SGLang receives SIGTERM (e.g. a Kubernetes pod deletion / rolling update), the current
TokenizerManager.sigterm_watchdog() calls kill_process_tree(os.getpid(), include_parent=True), which
SIGKILLs every process including itself. Two problems:

1. Exit code 137 instead of 0. SIGKILL of self surfaces to the orchestrator as exitCode 137, which
   looks like a crash/OOM in dashboards and restart policies.
2. No cleanup for child processes. Processes that hold external resources — most importantly the
   hicache Mooncake storage backend (RDMA memory regions, segment descriptors, RPC metadata) — rely on
   Python atexit handlers, __del__, and C++ destructors (PyClient::~PyClient() →
   tearDownAll_internal(), ~TransferEngine() → freeEngine()). SIGKILL bypasses all of them, leaking
   RDMA registrations and stale metadata, and can hang pod sandbox teardown.

This PR makes shutdown send SIGTERM first (so children run their cleanup), escalate to SIGKILL only for
stragglers, and lets the scheduler / data parallel controller exit normally so their cleanup runs.

Modifications

python/sglang/srt/utils/common.py

• Add graceful_kill_process_tree(): SIGTERM all children → wait up to a timeout (polling /proc via
  is_running()/status(), not psutil.wait_procs, whose pidfd_open path can raise
  OSError(EINVAL) against recursively-collected grandchildren on some kernels) → SIGKILL stragglers.
• Add get_child_process_shutdown_timeout() / SGLANG_CHILD_PROCESS_SHUTDOWN_TIMEOUT env (default 10s).

python/sglang/srt/managers/tokenizer_manager.py

• sigterm_watchdog() stops the SubprocessWatchdog before terminating children — otherwise it
  observes a child exit (e.g. the detokenizer dying on the default SIGTERM with exit code -15) and
  misclassifies the graceful shutdown as a crash, firing SIGQUIT → SIGKILL of the whole tree.
• Then calls graceful_kill_process_tree(include_parent=False, timeout=...) and os._exit(0) (a plain
  sys.exit would be swallowed by the asyncio event loop running the coroutine).
• The force paths (server_status == UnHealthy, SGL_FORCE_SHUTDOWN) keep the old immediate
  kill_process_tree(include_parent=True) (no graceful wait).

python/sglang/srt/managers/scheduler.py

• Register a SIGTERM handler in run_scheduler_process() that sys.exit(0) so atexit / __del__ /
  C++ destructors run. Exit code 0 is intentional: SubprocessWatchdog treats a non-zero / by-signal child
  exit as a crash. First line re-arms SIGTERM to SIG_IGN so a re-propagated signal cannot re-enter the
  handler and interrupt cleanup.

python/sglang/srt/managers/data_parallel_controller.py

• Register a SIGTERM handler that propagates graceful shutdown to its scheduler children via
  graceful_kill_process_tree(), then sys.exit(0). In DP mode the schedulers are children of the
  controller, which sets PR_SET_PDEATHSIG=SIGKILL; without this the controller would die immediately and
  the kernel would SIGKILL the schedulers mid-cleanup.

Validation

Validated end-to-end on a production TKE deployment (MiniMax-M2.7, sglang:v0.5.12-cu129, TP8/EP8,
HCCPNV6 8×GPU + 8×mlx5_bond RoCE), with mooncake_master (HTTP metadata server) running in-process and
the hicache Mooncake L3 backend enabled.

Graceful shutdown (SIGTERM, TP8 + Mooncake L3): process exit code 0; all 8 scheduler ranks logged
SIGTERM received in scheduler process (TPx PP0); exiting normally...; all 8 Mooncake segment descriptors
removed by the transfer-engine destructor (removeSegmentDesc ... finish) and master-mounted segments
dropped 8→0 with immediate UnmountSegment status=success; no SubprocessWatchdog false-crash.

SIGKILL straggler fallback (timeout=10): ... did not terminate within 10.0s, sending SIGKILL: pids=[...] fires and the process still exits 0 (set the timeout generously under RDMA and below
terminationGracePeriodSeconds).

Force path (SGL_FORCE_SHUTDOWN=1): Force exiting. with zero graceful-kill logs and exit code 137 —
the intentional immediate kill, a clean contrast to the graceful path's 0.

DP controller propagation (--dp-size 8 --enable-dp-attention), A/B:

	propagation log	scheduler handlers ran	result
Patched	present	8 / 8	exit code 0
Baseline (only data_parallel_controller.py reverted)	absent	0 / 8	schedulers SIGKILLed via PR_SET_PDEATHSIG within ~3s

Real Kubernetes-recorded container exit code (patched image vs stock image, server as container PID 1,
SIGTERM as kubelet delivers it, terminationGracePeriodSeconds=180):

image	K8s-recorded container exit
patched (this PR)	exitCode=0, reason=Completed
baseline (stock)	exitCode=137, reason=Error

This is complementary to kvcache-ai/Mooncake#1363 (master-side cleanup of HTTP metadata on client
heartbeat timeout), which covers the crash path; together no stale Mooncake metadata is left in any
shutdown scenario.

See Validation report here: #16484 (comment)

---

CI States

Latest PR Test (Base): ❌ Run #31763700004
Latest PR Test (Extra): ❌ Run #31763699789

[gemini-code-assist]

Summary of Changes

Hello @chenkaiyue, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the graceful shutdown mechanism within SGLang, particularly for child processes like hicache, which require time for proper resource cleanup. By introducing a phased termination approach that prioritizes SIGTERM before resorting to SIGKILL, and by refining how exit signals are handled in key managers, the system can now perform necessary cleanup operations, such as executing atexit handlers and C++ destructors, leading to more stable and reliable process termination, especially in orchestrated environments like Kubernetes.

Highlights

• Introduce graceful_kill_process_tree function: A new utility function has been added that enables a two-phase shutdown for process trees. It first sends SIGTERM to child processes, waits for a configurable timeout (defaulting to 10 seconds), and then sends SIGKILL to any processes that have not yet terminated, allowing for proper cleanup.
• Enhance sigterm_watchdog in TokenizerManager: The sigterm_watchdog now leverages the new graceful_kill_process_tree for managing child process termination. Additionally, sys.exit(0) has been replaced with os._exit(0) to prevent the SystemExit exception from interrupting FastAPI's lifespan cleanup, ensuring a smoother server shutdown.
• Implement graceful shutdown in Scheduler: A dedicated SIGTERM handler has been introduced in the scheduler process. This handler ensures that sys.exit(0) is called upon receiving SIGTERM, which allows Python's atexit handlers and C++ destructors (critical for Mooncake/hicache cleanup) to execute correctly before the process exits.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a crucial feature for graceful shutdown of child processes, which significantly improves server stability and resource management in containerized environments. The approach of sending SIGTERM before SIGKILL is well-implemented in the new graceful_kill_process_tree function. The changes in tokenizer_manager to use this new function and os._exit(0) are correct and well-reasoned. Similarly, adding a SIGTERM handler in the scheduler process is a good move for ensuring proper cleanup. My review includes a few suggestions to enhance code quality by removing redundant code and simplifying some logic.

[chenkaiyue]

Validation report — graceful shutdown verified end-to-end (with raw logs)

Supplementing the description with the actual evidence. Validated on a production TKE deployment:
MiniMax-M2.7, image sglang:v0.5.12-cu129, --tp-size 8 --ep-size 8, node HCCPNV6 (8×GPU + 8×mlx5_bond
RoCE, hostNetwork, privileged). A mooncake_master with co-located HTTP metadata server was run
in-process and the hicache Mooncake L3 backend was enabled
(--hicache-storage-backend mooncake, MOONCAKE_PROTOCOL=rdma,
MOONCAKE_TE_META_DATA_SERVER=http://127.0.0.1:8080/metadata, MOONCAKE_MASTER=127.0.0.1:50051).

Scenario A — graceful SIGTERM (TP8 + Mooncake L3, SGLANG_CHILD_PROCESS_SHUTDOWN_TIMEOUT=60)

All 8 scheduler ranks run the handler; the Mooncake transfer-engine destructor removes every segment
descriptor; master-mounted segments drop 8 → 0; process exits 0; no SubprocessWatchdog false-crash.

# scheduler handlers (8/8)
SIGTERM received in scheduler process (TP0 PP0); exiting normally to allow cleanup...
SIGTERM received in scheduler process (TP1 PP0); exiting normally to allow cleanup...
... (TP2..TP7) ...

# Mooncake transfer-engine cleanup actually ran (8/8 removeSegmentDesc ... finish)
I0618 ... transfer_metadata.cpp:472] removeSegmentDesc 10.0.0.76:15301 finish

# master-mounted segments over time (poll of /get_all_segments)
T0 segments=8 ...   t=5s segments=5   t=11s segments=1   t=13s segments=0

# process exit code
RUN_EXIT label=Aclean code=0

# only 1 process needed SIGKILL at the 60s cap (slow RDMA freeEngine reap, not a cleanup failure)
1 child process(es) did not terminate within 60.0s, sending SIGKILL: pids=[1557066]

Scenario A (timeout=10) — SIGKILL straggler-fallback branch

Sending SIGTERM to child process 1157939 (sglang::scheduler_TP0_EP0)
... (all schedulers + detokenizer) ...
Waiting up to 10.0s for 18 child process(es) to terminate gracefully...
6 child process(es) did not terminate within 10.0s, sending SIGKILL: pids=[1157937, 1157947, 1157948, 1157950, 1157953, 1157954]

Master side: gracefully-exited ranks call UnmountSegment immediately; SIGKILLed stragglers are reaped via
client_ttl:

UnmountSegment request: ... ; UnmountSegment response: status=success
client_id=..., segment_name=127.0.0.1:13398, action=unmount_expired_segment

Scenario A2 — force path (SGL_FORCE_SHUTDOWN=1)

The force path is preserved (immediate kill_process_tree, no graceful wait); exit code 137 is the
intentional immediate kill — a clean contrast with the graceful path's 0.

Signal SIGTERM received while force shutdown flag set. Force exiting.
RUN_EXIT label=A2force code=137
# zero graceful-kill / scheduler-handler logs, zero removeSegmentDesc

Scenario C — DP controller propagation (--dp-size 8 --enable-dp-attention), A/B

	propagation log	scheduler handlers ran	exit
Patched	present	8 / 8	0
Baseline (only data_parallel_controller.py reverted)	absent	0 / 8	schedulers SIGKILLed via PR_SET_PDEATHSIG

# PATCHED
SIGTERM received in data_parallel_controller; propagating graceful shutdown to scheduler children...
scheduler SIGTERM handler count = 8
RUN_EXIT label=Cpatched code=0

# BASELINE (un-patched data_parallel_controller.py): schedulers die in ~3s, before any handler runs
t=1s main_alive=yes sched_alive=8
t=2s main_alive=yes sched_alive=7
t=3s main_alive=yes sched_alive=2
t=4s main_alive=yes sched_alive=0
propagating graceful shutdown : 0 matches
SIGTERM received in scheduler process : 0 matches

Without the fix, none of the 8 schedulers run any cleanup — they are SIGKILLed before processing SIGTERM.

Scenario D — real Kubernetes-recorded container exit code (137 → 0)

A patched image (FROM sglang:v0.5.12-cu129 + the 4 files) was run as the Deployment with the sglang
server as the container PID-1 process (exec). SIGTERM was delivered as kubelet does on pod termination
(terminationGracePeriodSeconds=180), and the exit code was read from the pod containerStatuses:

image	K8s-recorded container exit
patched (this PR)	exitCode=0, reason=Completed
baseline (stock)	exitCode=137, reason=Error

On a real scale-to-0 pod delete the patched pod also terminated gracefully over ~54s (within the grace
period) instead of being force-killed.

Summary

scenario	result
A — graceful (TP8 + Mooncake L3)	exit 0; 8/8 handlers; 8/8 removeSegmentDesc; segments 8→0; no watchdog false-crash
A (timeout=10) — SIGKILL fallback	timeout→SIGKILL branch fires; still exits 0
A2 — force path	immediate kill; exit 137 (intentional)
C — DP controller propagation	patched 8/8 handlers + exit 0 vs baseline 0/8 (pdeathsig SIGKILL)
D — real K8s exit code	patched 0/Completed vs baseline 137/Error

Takeaway for operators: under RDMA, full freeEngine teardown can exceed 10s, so set
SGLANG_CHILD_PROCESS_SHUTDOWN_TIMEOUT generously (e.g. 60) and below terminationGracePeriodSeconds.
This change is complementary to kvcache-ai/Mooncake#1363 (master-side cleanup of HTTP metadata on client
heartbeat timeout, the crash path); together no stale Mooncake metadata is left in any shutdown scenario.

[chenkaiyue]

How this differs from #28779, and why it's still needed

#28779 (merged) and this PR are complementary layers; this PR is rebased on top of it.

#28779 — in-band, scheduler-only. A ShutdownReq control message (tokenizer → scheduler, broadcast
across TP ranks) breaks the scheduler loop and runs release_host_resources() before the scheduler exits.
It does not change the exit code and does not touch the non-scheduler processes.

This PR — the outer layer. After sending ShutdownReq and waiting for the schedulers, it SIGTERMs
the rest of the tree and exits the parent cleanly. It adds three things #28779 leaves open:

1. Exit code 137 → 0. After [Feature] Add graceful scheduler shutdown; free hisparse host buffer on exit #28779 the tokenizer still ended with
   kill_process_tree(include_parent=True), i.e. it SIGKILLs itself → kubelet records exitCode 137 / Error. This PR switches the last step to graceful_kill_process_tree(include_parent=False) + os._exit(0)
   → 0 / Completed.
2. The non-scheduler children ShutdownReq can't reach. ShutdownReq only talks to the scheduler. The
   hicache Mooncake L3 backend, the detokenizer, etc. are separate children; this PR SIGTERMs them
   first so their atexit / __del__ / C++ destructors run (RDMA segment unmount) instead of being
   SIGKILL'd.
3. DP-attention. See below.

Why DP-attention still needs the signal handlers (the subtle part). ShutdownReq does reach DP
schedulers — the data_parallel_controller forwards it — so they begin a graceful in-band shutdown. The
issue is the teardown window: DP schedulers are children of the controller, which sets
PR_SET_PDEATHSIG=SIGKILL on them. When the parent's graceful_kill_process_tree SIGTERMs the
controller, a controller with no handler dies immediately, and the kernel then races to SIGKILL every
scheduler that is still finishing its cleanup. The controller's handler closes this race (re-arm SIGTERM →
propagate graceful kill → wait → exit 0); the scheduler's own handler turns a direct SIGTERM into a clean
sys.exit(0) that runs destructors instead of dying by signal -15.

Validation

Prod TKE, MiniMax-M2.7, sglang:v0.5.12-cu129, TP8/EP8, 8×mlx5_bond RoCE, this PR hot-patched on top
of #28779; SGLANG_CHILD_PROCESS_SHUTDOWN_TIMEOUT set generously for RDMA/hicache teardown.

Scenario	exitCode	scheduler outcome	watchdog false-crash
non-DP (TP8, hicache=99)	0	8/8 graceful via ShutdownReq (scheduler SIGTERM-handler fired 0× — it's the safety net here)	none
DP-attention, patched (--dp-size 8 --enable-dp-attention, hicache=99)	0	8/8 clean; controller logs propagating graceful shutdown to scheduler children; fast	none
DP-attention, only the controller change reverted	0	schedulers still start their in-band shutdown via ShutdownReq, but teardown is no longer clean — the controller dies immediately so PR_SET_PDEATHSIG races the schedulers' cleanup; shutdown drags to the full graceful-kill timeout and a straggler is SIGKILL'd	none

Takeaways: the parent exits 0 in every graceful path (vs 137 on stock); ShutdownReq reaches DP
schedulers but does not by itself make their teardown clean — the controller handler keeps the
controller alive so its schedulers finish before PR_SET_PDEATHSIG fires. The controller handler only runs
under --enable-dp-attention; non-DP is fully covered by ShutdownReq + the tokenizer graceful-kill + the
scheduler handler.

[chenkaiyue]

Follow-up: deterministic evidence that the DP controller handler is load-bearing

The validation table above uses the real hicache config, where the PR_SET_PDEATHSIG race is timing-
dependent (it shows up as a drawn-out teardown). To pin the race down deterministically, I ran the same
DP-attention A/B with a temporary test instrument: a TEST_SCHED_TEARDOWN_DELAY=20s sleep injected
into the scheduler's exit path, simulating slow RDMA / hisparse / pinned-buffer teardown so the scheduler is
provably still cleaning up when the controller receives SIGTERM. (Instrument-only, removed before merge.)

Same prod box (MiniMax-M2.7, sglang:v0.5.12-cu129, TP8/EP8, --dp-size 8 --enable-dp-attention),
SGLANG_CHILD_PROCESS_SHUTDOWN_TIMEOUT=60. The distinguishing metric is how many of the 8 scheduler ranks
exit gracefully (their SIGTERM handler runs → destructors run) vs are SIGKILL'd by PR_SET_PDEATHSIG.

Run	controller handler	ranks that exited gracefully	orphaned child processes left for the parent to reap	parent exitCode
patched	present	8 / 8	0	0
baseline	reverted	5 / 8 (3 ranks PR_SET_PDEATHSIG-SIGKILL'd mid-teardown)	~275	0

Mechanism: in both runs all 8 ranks receive ShutdownReq and enter teardown (confirmed via a marker). With
the controller handler, the controller stays alive, propagates a graceful kill and waits, so all 8 finish →
8/8 graceful, no orphans. Without it, the controller dies immediately on the parent's SIGTERM and the kernel
SIGKILLs the still-cleaning-up ranks; the race is partial and non-deterministic (3/8 here), and each
ungracefully-killed rank leaves its own worker subprocesses orphaned (~275 total) for the parent to reap.

The parent process still exits 0 in both cases — so this is specifically about per-rank cleanup
reliability (RDMA registrations, pinned-host buffers, hisparse buffers), which is exactly what gets leaked
when a rank is SIGKILL'd mid-teardown.

[JustinTong0323]

/tag-and-rerun-ci

[chenkaiyue]

/rerun-failed-ci

[chenkaiyue]

CI status: only build-test (xeon-gnr, base-b-test-cpu) is red, and it's an upstream/main-side breakage — not this PR

All NVIDIA GPU gates are green on the latest head (111ed532b5): base-a, base-b, and base-c all pass. The only remaining failure is the CPU lane build-test (xeon-gnr, base-b-test-cpu).

This PR only touches process-shutdown / signal-handling code (utils/common.py, managers/tokenizer_manager.py, managers/scheduler.py, managers/data_parallel_controller.py). It does not touch MoE, kernels, enums, or any CPU path. The CPU-lane failures have been a moving target that tracks whatever just landed on main, not this branch:

when	failing test / step	root cause	origin
earlier	test_norm.py	sgl_kernel missing fused_qk_gemma_rmsnorm_cpu	main packaging (log)
earlier	test_spec_eagle_topk_cpu.py	HF 401 on gated meta-llama/Llama-2-7b-chat-hf	infra/auth, cleared on rerun (log)
earlier	run_suite.py	HWBackend has no attribute 'MLX'	version skew from #30121, fixed after I re-merged main (log)
now	test_intel_amx_attention_backend_b.py	TypeError: grouped_topk_cpu() got an unexpected keyword argument 'scoring_func'	Python↔sgl_kernel signature skew (log)

Current failure (TestIntelAMXAttnBackendQuant.test_latency_fp8_moe_model):

File ".../sglang/srt/layers/moe/topk.py", line 2024, in select_experts
    topk_weights, topk_ids = grouped_topk(...)
TypeError: grouped_topk_cpu() got an unexpected keyword argument 'scoring_func'

The Python MoE top-k path now passes scoring_func into the CPU kernel, but the grouped_topk_cpu() binding on the CPU lane does not accept it — a main-side Python/kernel mismatch that fails independently of this PR.

Whether build-test (xeon-gnr, base-b-test-cpu) is a required gate for this PR? If it's the known-broken main CPU lane, this PR is otherwise ready (GPU gates green, changes unrelated to the CPU path). Happy to re-merge main once the CPU lane is fixed upstream.

[chenkaiyue]

/rerun-failed-ci

[stmatengss]

/rerun-failed-ci

[chenkaiyue]

/rerun-failed-ci

[chenkaiyue]

/rerun-failed-ci

[chenkaiyue]

/rerun-failed-ci

[chenkaiyue]

/rerun-failed-ci

[chenkaiyue]

@hnyls2002 Could you please help review this PR?