Skip to content

fix(python-backend): make JIT subprocesses work on hosts of any size - #9679

Merged
mudler merged 1 commit into
mudler:masterfrom
richiejp:fix/python-backend-jit-subprocess
May 5, 2026
Merged

fix(python-backend): make JIT subprocesses work on hosts of any size#9679
mudler merged 1 commit into
mudler:masterfrom
richiejp:fix/python-backend-jit-subprocess

Conversation

@richiejp

@richiejp richiejp commented May 5, 2026

Copy link
Copy Markdown
Collaborator

Put ninja on the PATH by putting the venv bin in the PATH correctly. Instead of trying to patch up the activate script we just do the necessary bits ourselves; it's very simple and shouldn't change.


Two related runtime fixes for Python backends that JIT-compile CUDA
kernels at first model load (FlashInfer, PyTorch inductor, triton):

  1. libbackend.sh: replace source ${EDIR}/venv/bin/activate with a
    minimal manual setup (_activateVenv: export VIRTUAL_ENV, prepend
    PATH, unset PYTHONHOME) computed from $EDIR at runtime. uv venv
    and python -m venv both bake the create-time absolute path into
    bin/activate (e.g. VIRTUAL_ENV='/vllm/venv' from the Docker build
    stage), so sourcing activate on a relocated venv — copied out of
    the build container and unpacked at an arbitrary backend dir —
    prepends a stale, non-existent path to $PATH. Pip-installed CLI
    tools (e.g. ninja, used by FlashInfer's NVFP4 GEMM JIT) are then
    never found and the load aborts with FileNotFoundError. Doing the
    env setup ourselves matches what uv run does internally and
    sidesteps the relocation problem entirely. Generic — every Python
    backend benefits.

  2. vllm/run.sh: replace ninja's default -j$(nproc)+2 with an adaptive
    MAX_JOBS = min(nproc, (MemAvailable-4)/4). Each concurrent
    nvcc/cudafe++ peaks at multiple GiB; the default OOM-kills on
    memory-tight hosts (e.g. a 16 GiB desktop loading a 27B NVFP4
    model) but underutilises 100-core / 1 TB boxes. User-set MAX_JOBS
    still wins. Also pin NVCC_THREADS=2 unless overridden.

Refs: vllm-project/vllm#20079

Assisted-by: Claude:claude-opus-4-7 [Edit] [Bash]

Two related runtime fixes for Python backends that JIT-compile CUDA
kernels at first model load (FlashInfer, PyTorch inductor, triton):

1. libbackend.sh: replace `source ${EDIR}/venv/bin/activate` with a
   minimal manual setup (_activateVenv: export VIRTUAL_ENV, prepend
   PATH, unset PYTHONHOME) computed from $EDIR at runtime. `uv venv`
   and `python -m venv` both bake the create-time absolute path into
   bin/activate (e.g. VIRTUAL_ENV='/vllm/venv' from the Docker build
   stage), so sourcing activate on a relocated venv — copied out of
   the build container and unpacked at an arbitrary backend dir —
   prepends a stale, non-existent path to $PATH. Pip-installed CLI
   tools (e.g. ninja, used by FlashInfer's NVFP4 GEMM JIT) are then
   never found and the load aborts with FileNotFoundError. Doing the
   env setup ourselves matches what `uv run` does internally and
   sidesteps the relocation problem entirely. Generic — every Python
   backend benefits.

2. vllm/run.sh: replace ninja's default -j$(nproc)+2 with an adaptive
   MAX_JOBS = min(nproc, (MemAvailable-4)/4). Each concurrent
   nvcc/cudafe++ peaks at multiple GiB; the default OOM-kills on
   memory-tight hosts (e.g. a 16 GiB desktop loading a 27B NVFP4
   model) but underutilises 100-core / 1 TB boxes. User-set MAX_JOBS
   still wins. Also pin NVCC_THREADS=2 unless overridden.

Refs: vllm-project/vllm#20079

Assisted-by: Claude:claude-opus-4-7 [Edit] [Bash]
@mudler
mudler merged commit 16b2d4c into mudler:master May 5, 2026
47 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants