server: router mode subprocess still occupies GPU after sleep-idle-seconds

[leonardcser]

Feature Description

In router mode, when --sleep-idle-seconds triggers, the child subprocess unloads the model from VRAM but the process remains alive and attached to the GPU, consuming ~600MiB per idle subprocess:

# Active
467282 dev 0 Compute  0% 10386MiB 11% 824MiB llama-server ...

# After sleep-idle triggers — process still on GPU
467282 dev 0 Compute N/A   614MiB  1% 369MiB llama-server ...

Motivation

Idle subprocesses should not remain as GPU processes when they are not needed. With multiple models in router mode, the residual ~600MiB per dormant process wastes significant VRAM.

Relation to #18189

Follow-up to #18189. PR #18228 implemented --sleep-idle-seconds, but it only unloads the model within the living process — it does not terminate the subprocess. The original request was closed as stale without this being addressed.

Possible Implementation

A new option (e.g. --stop-idle-seconds) that triggers full subprocess termination in router mode via the existing unload() path. The building blocks are already there:

• server_queue already tracks idle time
• server_models::unload() already handles graceful shutdown → force-kill
• These two just need to be wired together, with the router re-spawning the process on the next request (same as --models-max LRU eviction already does)

[IMbackK]

What kind of gpu? If its rocm, rocblas uses ~500MB of vram for itself to load its solutions, this is out of controll of the applicaiton and happens when the first rocblas call is made and is only freed when the application exits.

I could imagine cublas might have similar behavior.

Just as suggestion where the allocation here could be coming from

[leonardcser]

The card in question is NVIDIA RTX PRO with cuda13.

But even if we cannot control the memory, I feel it would make more sense to stop the processes running on the GPU.

Currently we "unload" the vram but we still have a dormant process attached.

[apunkt]

using b8215 I was able to manually unload models and no GPU processes remained
using b8261 I was able to manually unload models when sleeping, but GPU processes remain

using CUDA 13 and 12

[apunkt]

b8405 works for me

[leonardcser]

Not working on b8720.

On startup, the router reserves ~600MiB for "unloaded" models.
When model is loaded and goes back to sleep, it is still attached to the GPU processes with ~600MiB.
I was able to manually unload models and no GPU processes remained.

version: 907 (d12cc3d1c)
built with GNU 13.4.0 for Linux x86_64

    PID USER DEV     TYPE  GPU        GPU MEM    CPU  HOST MEM Command
1511854  dev   0  Compute   0%    612MiB   1%     0%    391MiB /home/dev/.local/bin/llama-server --h
1511852  dev   0  Compute   0%    612MiB   1%     0%    380MiB /home/dev/.local/bin/llama-server --h

[monksy]

I'm seeing this with docker container:
"Image": "sha256:668c6112d7b5e424e172300b7e7429c1e22f872347881a55072607458e480b38",
Not sure what version this is tagged with. But it's as of 3 days agao.

Env setting:
"LLAMA_ARG_SLEEP_IDLE_SECONDS=360",

Nvidia-smi evidence:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 Ti     Off |   00000000:01:00.0 Off |                  N/A |
|  0%   40C    P8              9W /  165W |   14888MiB /  16380MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1653      G   /usr/lib/Xorg                             4MiB |
|    0   N/A  N/A          676147      C   python                                  424MiB |
|    0   N/A  N/A          935905      C   /app/llama-server                     14434MiB |
+-----------------------------------------------------------------------------------------+

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[anagnorisis2peripeteia]

Still the case on current master: sleep-idle-seconds only destroys the model/context inside the child (handle_sleeping_state → destroy()); the child process itself stays alive, and no --stop-idle-seconds-style option exists in common/arg.cpp. So the ~600 MiB backend-context residual per sleeping child reported above remains by-design.

A use case where this hurts more than on datacenter cards: shared consumer GPUs. Our RTX 5070 (12 GB) is a gaming card that also serves a 35B MoE via router mode — a dormant sleeping child permanently holds ~600 MiB of VRAM that games would otherwise use, which on a 12 GB card is often the difference between fitting VRAM budget or not.

Workaround that works today (verified on version: 9760 (6ee0f6579)): an external watchdog that polls GET /v1/models — side-effect-free: it neither wakes a sleeping child nor resets the idle timer — and POSTs /models/unload {"model": "..."} for anything reporting sleeping longer than a threshold. The unload path fully terminates the child: status returns to unloaded, the process exits, and all VRAM and host RSS are released.

+1 for --stop-idle-seconds (sleep after N, exit after M) — this was closed by the stale bot without being implemented.