Skip to content

[Perf] Early forkserver: fork workers from a preloaded server instead of re-importing - #1

Open
xenshinu wants to merge 156 commits into
startup/1-fork-safe-importfrom
startup/2-early-forkserver
Open

xenshinu wants to merge 156 commits into
startup/1-fork-safe-importfrom
startup/2-early-forkserver

Conversation

@xenshinu

@xenshinu xenshinu commented Sep 18, 2026 •

Copy link
Copy Markdown

Stacked PRs following sgl-project#40201

Motivation:

this PR changes how a worker process is created. Today every worker is a fresh interpreter (spawn) that re-imports the sglang/torch/kernel stack, 7-9 s per worker, and a third time on the DP path where the controller re-imports before it spawns the schedulers. Python's forkserver start method creates one small helper process and forks every child from it; if that helper has imported the worker modules once ("preload"), a forked worker inherits them and imports nothing. The launcher itself cannot be the fork parent: it has threads and, through sgl_kernel, a CUDA context.

Modifications:

SGLANG_EARLY_FORKSERVER=1 starts the multiprocessing forkserver at CLI entry with the scheduler, DP controller and detokenizer modules preloaded; workers fork from it. The child re-applies the launcher's environment and stdio, re-derives the HF offline flags, reports the launcher as its parent, keeps its sglang::* process title, and nested Process() calls reuse the same forkserver. Import-time GPU probes of third-party packages are answered from NVML inside the forkserver; an attribute NVML cannot supply raises a named error. --enable-memory-saver falls back to spawn with a warning. New Envs: SGLANG_EARLY_FORKSERVER (off), SGLANG_MP_START_METHOD (set by start_early()), internal SGLANG_FORKSERVER_ADDRESS/PID. Default behaviour unchanged (engine.py: the start method is read from Envs, default spawn).

Accuracy:

greedy output identical with the flag on and off (Qwen3-1.7B and Qwen3-30B-A3B, 128- and 64-prompt batches). Speed: single-GPU Qwen3-30B-A3B time to /health 47.3 s -> about 43 s with the forkserver alone (the rest of the gain needs PR 3). Tests: entrypoints/test_early_forkserver.py (env merge, offline flags, NVML stand-in, forked-worker title regression, switches).

@xenshinu
xenshinu added this pull request to stack #3 September 18, 2026 16:32
niehen6174 and others added 27 commits September 21, 2026 00:07
…ject#35452)

Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com>
Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Signed-off-by: JinYan Su <751080330@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
…roject#39777)

Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: mmangkad <mohammad.angkad@radixark.ai>
… global (sgl-project#38932)

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
…d implement PowerOfTwo (sgl-project#40241)

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…memory (sgl-project#40481)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
…sgl-project#39206)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
…gl-project#40487)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
Lzy17 and others added 16 commits September 22, 2026 02:15
…quant (sgl-project#40557)

Co-authored-by: siliangchen-amd <{"message":"Although you appear to have the correct authorization credentials, the `amd-eng-emu` enterprise has an IP allow list enabled, and your IP address is not permitted to access this resource.","documentation_url":"https://docs.github.com/rest/users/users#get-the-authenticated-user","status":"403"}siliangchen-amd@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
… set (sgl-project#40592)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…cope (sgl-project#40593)

Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… DeepGEMM (sgl-project#40352)

Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com>
…-Exp (sgl-project#40501)

Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
@xenshinu
xenshinu force-pushed the startup/2-early-forkserver branch 2 times, most recently from 968e09f to 123dca1 Compare September 22, 2026 16:58
@xenshinu
xenshinu force-pushed the startup/2-early-forkserver branch from 123dca1 to db58856 Compare September 22, 2026 18:37
…project#39901)

Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
@xenshinu
xenshinu force-pushed the startup/2-early-forkserver branch from db58856 to 3c2bd4d Compare September 22, 2026 19:20
…FORKSERVER)

Under spawn every worker (scheduler, DP controller, detokenizer) is a fresh
interpreter that imports sglang, torch and the kernel packages, 7-9 s each,
and the DP controller pays it again for the schedulers it starts. With
SGLANG_ENABLE_EARLY_FORKSERVER=1, `sglang serve` starts the multiprocessing
forkserver at CLI entry with those modules preloaded and forks the workers
from it. Off by default; every other entry point keeps spawn.

The forkserver is registered as its own start method, sglang_forkserver.
Its _Popen attaches the launcher's environment to the process object, and an
after-fork hook registered in the server runs in every forked child before
run(): it applies the changes the launcher made to its environment since CLI entry
(three-way merge against the environment the server started with), takes the
launcher's rlimits, restores
the real torch.cuda functions, releases the server's alive pipe and points
nested starts at the same server. Workers and the watchdog that signal their parent now
ask multiprocessing for the process that called start() (get_parent_process);
under spawn that is the OS parent, so the default path is unchanged. Because
only the launcher holds the server's alive pipe, the server exits with the
launcher and PR_SET_PDEATHSIG takes the workers down as it does under spawn.

Inside the server, this module, first in the preload list, answers the GPU
probes sgl_kernel and flashinfer make at import from NVML, so the server never initializes CUDA
and stays forkable; a child that finds itself in a bad fork fails with a
message naming the cause.

--enable-memory-saver falls back to spawn (its hook is LD_PRELOADed into an
exec'd worker) and stops the server, as does a non-LLM serve backend. With the
forkserver, NUMA binding takes the in-process path.
A module-level constant computed from an env var is evaluated when the module
is first imported. Under spawn that happens in the worker after the arguments
are resolved; a forkserver imports the preload list at CLI entry, so a worker
forked from it inherits the pre-resolution value while the variable itself is
already correct.

moe_hook sets SGLANG_MOE_NVFP4_DISPATCH and the Qwen3-VL override raises
SGLANG_MM_FEATURE_CACHE_MB during argument resolution. The MoE flag is read
at call time, resolved once when the runner is built, and weight loading
fails loudly if the answer changed in between; the multimodal
budget becomes mm_feature_cache_size(), with MM_FEATURE_CACHE_SIZE still
resolving through the compat module cuda_ipc_transport_utils.

A static test walks the preload closure and fails on any module-level read,
through Envs or raw os.environ, of a variable sglang writes at runtime.
@xenshinu
xenshinu force-pushed the startup/2-early-forkserver branch from 3c2bd4d to 7a26233 Compare September 22, 2026 19:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.