server: also free backend on sleep

[ngxson]

Overview

Call llama_backend_free() on sleep

As a side effect of this change, llama_backend_init() is not longer called on router mode (which is expected)

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: no

[DEV-DUFORD]

@ngxson Sadly does not resolve the issue I presented in #25243, on sleep I'm still seeing roughly 376MB used on each of my GPUs when in sleep mode, which is then leading to that higher power draw and increased idle temps.

[DEV-DUFORD]

I always hate doing this (because it feels like it can come off as disrespectful to be given back an AI trace), you don't need to read it, but this is quick summarization of an initial investigation against my hardware:

The tl;dr that I still need to do more digging on is that it might be driver level, as I can't even get hipDeviceReset to free these resources.

AI Investigation

 Goal: Understand why ~376MB per GPU remains resident on your gfx900 rig when  llama-server  enters sleep, including under PR #25265.

What PR #25265 changes. It wraps  llama_backend_init/free  in a scoped  server_backend  object and calls  llama_backend_free() when entering sleep. In  src/llama.cpp , that function currently does:

void llama_backend_free(void) {
    ggml_quantize_free();
}

So at present it releases quantization-related host state and does not itself tear down device contexts, memory pools, or loaded GPU kernels. The CUDA/HIP backend also has no  hipDeviceReset / cudaDeviceReset  in its teardown path.

Measurements on this rig (gfx900, ROCm 6.3), using small HIP probes and  hipMemGetInfo :

1. Bare HIP context ( hipFree(0) ): ~16MB. A 256MB alloc/free returns cleanly to 16MB.
2. Add hipBLAS + one  Sgemm : rises to ~208MB; after freeing all buffers and  hipblasDestroy , ~174MB remains resident. This residual corresponds to the rocBLAS/Tensile kernel library loaded into VRAM.
3. Following that with  hipDeviceReset()  (returned success) and re-init: still ~174MB.

Interpretation. The residual VRAM in these probes is dominated by BLAS kernel-object memory rather than model weights or ggml pool allocations. In this setup it persisted across buffer frees, handle destruction, and a device reset. This suggests the sleep-time residual you observed is largely driver/library-level memory that the current software teardown path (model unload +  llama_backend_free() ) does not target.

[ngxson]

yes, I looked into ggml_backend_unload() and turns out it only release the pointer to backend without actually destroying it

implementing real backend_free() API is necessary to make it work, I think it's will be the best method overall, rather than destroying the whole process

if you can confirm if calling hipDeviceReset() works in your case, I can push a proposal to add the backend_free() across all backends

[ngxson]

I have a better prototype that works on CUDA with cudaDeviceReset(), will push another PR that expands from this

[ngxson]

@DEV-DUFORD PTAL #25271

tested on CUDA: nvtop reports 0 memory usage on sleep