Repository navigation
Conversation
xenshinu
added this pull request to stack #3
September 18, 2026 16:32
…ject#35452) Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com> Co-authored-by: Ke Bao <ispobaoke@gmail.com>
Signed-off-by: JinYan Su <751080330@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com>
…roject#39777) Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com> Co-authored-by: hnyls2002 <lsyincs@gmail.com>
…ect#40163) Co-authored-by: Hank Han <hanhan7630@outlook.com>
Co-authored-by: mmangkad <mohammad.angkad@radixark.ai>
… global (sgl-project#38932) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
…gl-project#40272) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…d implement PowerOfTwo (sgl-project#40241) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…memory (sgl-project#40481) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
…sgl-project#39206) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
…gl-project#40487) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
…quant (sgl-project#40557) Co-authored-by: siliangchen-amd <{"message":"Although you appear to have the correct authorization credentials, the `amd-eng-emu` enterprise has an IP allow list enabled, and your IP address is not permitted to access this resource.","documentation_url":"https://docs.github.com/rest/users/users#get-the-authenticated-user","status":"403"}siliangchen-amd@users.noreply.github.com> Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
…gl-project#40175) Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
… set (sgl-project#40592) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…cope (sgl-project#40593) Co-authored-by: Mick Qian <mickqian@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… DeepGEMM (sgl-project#40352) Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com> Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com>
…-Exp (sgl-project#40501) Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
xenshinu
force-pushed
the
startup/2-early-forkserver
branch
2 times, most recently
from
September 22, 2026 16:58
968e09f to
123dca1
Compare
…t#40747) Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
…iring the parallel getters (sgl-project#40707)
…ment (sgl-project#40686) Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
xenshinu
force-pushed
the
startup/2-early-forkserver
branch
from
September 22, 2026 18:37
123dca1 to
db58856
Compare
…project#39901) Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com> Co-authored-by: jacky.cheng <yichiche@amd.com>
xenshinu
force-pushed
the
startup/2-early-forkserver
branch
from
September 22, 2026 19:20
db58856 to
3c2bd4d
Compare
…FORKSERVER) Under spawn every worker (scheduler, DP controller, detokenizer) is a fresh interpreter that imports sglang, torch and the kernel packages, 7-9 s each, and the DP controller pays it again for the schedulers it starts. With SGLANG_ENABLE_EARLY_FORKSERVER=1, `sglang serve` starts the multiprocessing forkserver at CLI entry with those modules preloaded and forks the workers from it. Off by default; every other entry point keeps spawn. The forkserver is registered as its own start method, sglang_forkserver. Its _Popen attaches the launcher's environment to the process object, and an after-fork hook registered in the server runs in every forked child before run(): it applies the changes the launcher made to its environment since CLI entry (three-way merge against the environment the server started with), takes the launcher's rlimits, restores the real torch.cuda functions, releases the server's alive pipe and points nested starts at the same server. Workers and the watchdog that signal their parent now ask multiprocessing for the process that called start() (get_parent_process); under spawn that is the OS parent, so the default path is unchanged. Because only the launcher holds the server's alive pipe, the server exits with the launcher and PR_SET_PDEATHSIG takes the workers down as it does under spawn. Inside the server, this module, first in the preload list, answers the GPU probes sgl_kernel and flashinfer make at import from NVML, so the server never initializes CUDA and stays forkable; a child that finds itself in a bad fork fails with a message naming the cause. --enable-memory-saver falls back to spawn (its hook is LD_PRELOADed into an exec'd worker) and stops the server, as does a non-LLM serve backend. With the forkserver, NUMA binding takes the in-process path.
A module-level constant computed from an env var is evaluated when the module is first imported. Under spawn that happens in the worker after the arguments are resolved; a forkserver imports the preload list at CLI entry, so a worker forked from it inherits the pre-resolution value while the variable itself is already correct. moe_hook sets SGLANG_MOE_NVFP4_DISPATCH and the Qwen3-VL override raises SGLANG_MM_FEATURE_CACHE_MB during argument resolution. The MoE flag is read at call time, resolved once when the runner is built, and weight loading fails loudly if the answer changed in between; the multimodal budget becomes mm_feature_cache_size(), with MM_FEATURE_CACHE_SIZE still resolving through the compat module cuda_ipc_transport_utils. A static test walks the preload closure and fails on any module-level read, through Envs or raw os.environ, of a variable sglang writes at runtime.
xenshinu
force-pushed
the
startup/2-early-forkserver
branch
from
September 22, 2026 19:24
3c2bd4d to
7a26233
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs following sgl-project#40201
Motivation:
this PR changes how a worker process is created. Today every worker is a fresh interpreter (
spawn) that re-imports the sglang/torch/kernel stack, 7-9 s per worker, and a third time on the DP path where the controller re-imports before it spawns the schedulers. Python'sforkserverstart method creates one small helper process and forks every child from it; if that helper has imported the worker modules once ("preload"), a forked worker inherits them and imports nothing. The launcher itself cannot be the fork parent: it has threads and, throughsgl_kernel, a CUDA context.Modifications:
SGLANG_EARLY_FORKSERVER=1starts the multiprocessing forkserver at CLI entry with the scheduler, DP controller and detokenizer modules preloaded; workers fork from it. The child re-applies the launcher's environment and stdio, re-derives the HF offline flags, reports the launcher as its parent, keeps itssglang::*process title, and nestedProcess()calls reuse the same forkserver. Import-time GPU probes of third-party packages are answered from NVML inside the forkserver; an attribute NVML cannot supply raises a named error.--enable-memory-saverfalls back tospawnwith a warning. NewEnvs:SGLANG_EARLY_FORKSERVER(off),SGLANG_MP_START_METHOD(set bystart_early()), internalSGLANG_FORKSERVER_ADDRESS/PID. Default behaviour unchanged (engine.py: the start method is read fromEnvs, defaultspawn).Accuracy:
greedy output identical with the flag on and off (Qwen3-1.7B and Qwen3-30B-A3B, 128- and 64-prompt batches). Speed: single-GPU Qwen3-30B-A3B time to
/health47.3 s -> about 43 s with the forkserver alone (the rest of the gain needs PR 3). Tests:entrypoints/test_early_forkserver.py(env merge, offline flags, NVML stand-in, forked-worker title regression, switches).