server: add --sleep-mode rst, reset the proc while keeping metrics

common/arg.cpp

@@ -3757,6 +3757,22 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.sleep_idle_seconds = value;
         }
     ).set_examples({LLAMA_EXAMPLE_SERVER}));
+    add_opt(common_arg(
+        {"--sleep-mode"}, "MODE",
+        "sleep behavior:\n"
+        "- 'free' frees context and model memory\n"
+        "- 'rst' restarts the whole process, may help reset memory to zero on certain backend (only support posix env)\n"
+        "(default: free)",
+        [](common_params & params, const std::string & value) {
+            if (value == "free") {
+                params.sleep_mode = COMMON_SLEEP_MODE_FREE;
+            } else if (value == "rst") {
+                params.sleep_mode = COMMON_SLEEP_MODE_RST;
+            } else {
+                throw std::invalid_argument("invalid value: " + value);
+            }
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}));
     add_opt(common_arg(
         {"--simple-io"},
         "use basic IO for better compatibility in subprocesses and limited consoles",

common/common.h

@@ -408,6 +408,11 @@ struct common_params_diffusion {
 
 // reasoning API response format (not to be confused as chat template's reasoning format)
 // only used by server
+enum common_sleep_mode {
+    COMMON_SLEEP_MODE_FREE, // free context and model memory
+    COMMON_SLEEP_MODE_RST, // also restart the process, releasing all backend resources
+};
+
 enum common_reasoning_format {
     COMMON_REASONING_FORMAT_NONE,
     COMMON_REASONING_FORMAT_AUTO,            // Same as deepseek, using `message.reasoning_content`
@@ -633,6 +638,7 @@ struct common_params {
     int enable_reasoning = -1; // -1 = auto, 0 = disable, 1 = enable
     bool prefill_assistant = true; // if true, any trailing assistant message will be prefilled into the response
     int sleep_idle_seconds = -1;   // if >0, server will sleep after this many seconds of idle time
+    common_sleep_mode sleep_mode = COMMON_SLEEP_MODE_FREE;
 
     std::vector<std::string> api_keys;
 

tools/server/README-dev.md

@@ -321,6 +321,25 @@ Call stack on waking up:
 
 Endpoints created with `create_response(true)` (`/health`, `/props`, `/models`, `/metrics`) skip `wait_until_no_sleep`, so they answer from the cached responses instead of waking the server.
 
+#### Process reset (`--sleep-mode rst`)
+
+As described in the section above, the sleeping function frees the `llama_context` and `llama_model` instances. However, on certain backends, the underlying driver or library still leaves behind some residual memory. See issue: [#19379](https://github.com/ggml-org/llama.cpp/issues/19379), [#25570](https://github.com/ggml-org/llama.cpp/issues/25570)
+
+Multiple PRs attempted to fix the problem by simply exiting the process ([#25243](https://github.com/ggml-org/llama.cpp/pull/25243), [#27307](https://github.com/ggml-org/llama.cpp/pull/27307)). However, the main issues with this approach are: (1) it requires router mode to handle the respawn, (2) `/props` and `/models` cannot be accessed during sleep and (3) metrics are reset.
+
+Note that [#25271](https://github.com/ggml-org/llama.cpp/pull/25271) proposed a deeper solution, adding a GGML API to reset the physical device. However, it does not work correctly on AMD GPUs due to a limitation of the AMD Tensile library, and the CUDA backend keeps static state that assumes the context is never destroyed.
+
+Therefore, `--sleep-mode rst` was added to reset the process instead (PR [#27418](https://github.com/ggml-org/llama.cpp/pull/27418)), while still allowing `/props` and `/models` to work as-is, and keeping the metrics. Only POSIX platforms are supported for now, because this relies on `exec()` to re-use the same PID. Windows is not yet supported and may require another method.
+
+The way it works: The restart is handled by `server_sleep_rst`, owned by `server_context_impl`:
+- `restart()` is called by `handle_sleeping_state()` right after `destroy()`, so the model is already unloaded
+- the state to preserve is the cached responses (`server_routes::cache_to_json`), passed to the new process via the `LLAMA_SERVER_SLEEP_STATE` env var
+- `init()` reads that env var and clears it, so that child processes do not inherit it
+- all file descriptors except stdio are marked `FD_CLOEXEC`, so `exec()` releases the listening port and the backend devices
+- `load_model()` sees the restored state, skips loading and starts the queue in sleeping state; the model is then loaded upon the first request
+
+The env var is limited to 128 kB by `exec()` on Linux (`MAX_ARG_STRLEN`). If the state does not fit, the restart is skipped and the server stays in a normal sleeping state.
+
 ### Notable Related PRs
 
 - Initial server implementation: https://github.com/ggml-org/llama.cpp/pull/1443

tools/server/README.md

@@ -196,11 +196,11 @@ For the full list of features, please refer to [server's changelog](https://gith
 | `--ui-config, --webui-config JSON` | JSON that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG) |
 | `--ui-config-file, --webui-config-file PATH` | JSON file that provides default UI settings (overrides UI defaults)<br/>(env: LLAMA_ARG_UI_CONFIG_FILE) |
 | `--ui-mcp-proxy, --webui-mcp-proxy, --no-ui-mcp-proxy, --no-webui-mcp-proxy` | experimental: whether to enable MCP CORS proxy - do not enable in untrusted environments (default: disabled)<br/>(env: LLAMA_ARG_UI_MCP_PROXY) |
-| `--tools TOOL1,TOOL2,...` | experimental: whether to enable server tools for AI agents - do not enable in untrusted environments (default: no tools)<br/>specify "all" to enable all tools<br/>available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_TOOLS) |
+| `--tools TOOL1,TOOL2,...` | experimental: whether to enable built-in tools for AI agents - do not enable in untrusted environments (default: no tools)<br/>specify "all" to enable all tools<br/>available tools: read_file, file_glob_search, grep_search, exec_shell_command, write_file, edit_file, get_info<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_TOOLS) |
 | `--tools-runtime OPTION` | experimental: run tools in a separate runtime environment (default: none, use host environment)<br/>available options:<br/>  'docker:<image>', 'podman:<image>': spin up a new container and reuse it for all invocations, clean up on server exit<br/>  'docker-container:<id>', 'podman-container:<id>': use an existing container by ID, won't stop on server exit<br/>  'ssh:<target>': run tools on a remote POSIX host over SSH, key-based auth and a trusted host key are required<br/><br/>(env: LLAMA_ARG_TOOLS_RUNTIME) |
 | `--mcp-servers-config PATH` | experimental: path to JSON file with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_CONFIG) |
 | `--mcp-servers-json JSON` | experimental: inline JSON with MCP server definitions (Cursor-compatible format) - do not enable in untrusted environments (default: none)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_MCP_SERVERS_JSON) |
-| `-ag, --agent, -no-ag, --no-agent` | whether to enable CORS proxy and all server tools - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_AGENT) |
+| `-ag, --agent, -no-ag, --no-agent` | whether to enable CORS proxy and all built-in tools - do not enable in untrusted environments (default: disabled)<br/>note: for security reasons, this will limit --cors-origins to localhost by default<br/>(env: LLAMA_ARG_AGENT) |
 | `--ui, --webui, --no-ui, --no-webui` | whether to enable the Web UI (default: enabled)<br/>(env: LLAMA_ARG_UI) |
 | `--embedding, --embeddings` | restrict to only support embedding use case; use only with dedicated embedding models (default: disabled)<br/>(env: LLAMA_ARG_EMBEDDINGS) |
 | `--rerank, --reranking` | enable reranking endpoint on server (default: disabled)<br/>(env: LLAMA_ARG_RERANKING) |
@@ -237,6 +237,7 @@ For the full list of features, please refer to [server's changelog](https://gith
 | `-sps, --slot-prompt-similarity SIMILARITY` | how much the prompt of a request must match the prompt of a slot in order to use that slot (default: 0.10, 0.0 = disabled) |
 | `--lora-init-without-apply` | load LoRA adapters without applying them (apply later via POST /lora-adapters) (default: disabled) |
 | `--sleep-idle-seconds SECONDS` | number of seconds of idleness after which the server will sleep (default: -1; -1 = disabled) |
+| `--sleep-mode MODE` | sleep behavior:<br/>- 'free' frees context and model memory<br/>- 'rst' restarts the whole process, may help reset memory to zero on certain backend (only support posix env)<br/>(default: free) |
 | `--log-prompts-dir PATH` | Log prompts to directory (auto-created if not present; only used for debugging, default: disabled) |
 | `--spec-draft-hf, -hfd, -hfrd, --hf-repo-draft <user>/<model>[:quant]` | Same as --hf-repo, but for the draft model (default: unused)<br/>(env: LLAMA_ARG_SPEC_DRAFT_HF_REPO) |
 | `--spec-draft-threads, -td, --threads-draft N` | number of threads to use during generation (default: same as --threads) |
@@ -2073,6 +2074,8 @@ Note that the following endpoints are exempt from being considered as incoming t
 - `GET /models`
 - `GET /metrics`
 
+Some backends keep memory allocated even after the model is unloaded, for example a CUDA context stays on the GPU. To also release that memory, use `--sleep-mode rst`, which restarts the server process upon sleeping. The process keeps the same PID and port, and the responses of the endpoints listed above are preserved across the restart. This mode is not supported on Windows.
+
 ## More examples
 
 ### Interactive mode

tools/server/server-common.cpp

@@ -16,6 +16,18 @@
 #include <cstring>
 #include <type_traits>
 
+#if !defined(_WIN32)
+#include <unistd.h>
+#include <limits.h>
+#include <cerrno>
+#include <fcntl.h>
+#include <sys/resource.h>
+#endif
+
+#if defined(__APPLE__)
+#include <mach-o/dyld.h>
+#endif
+
 json format_error_response(const std::string & message, const enum error_type type) {
     std::string type_str;
     int code = 500;
@@ -87,6 +99,64 @@ json server_slot_stats::to_json() const {
     return base;
 }
 
+//
+// server_metrics
+//
+
+json server_metrics::bucket::to_json() const {
+    return json {
+        {"count", count},
+        {"steps", steps},
+        {"time",  time },
+    };
+}
+
+void server_metrics::bucket::from_json(const json & data) {
+    count = data.at("count");
+    steps = data.at("steps");
+    time  = data.at("time");
+}
+
+json server_metrics::to_json() const {
+    return json {
+        {"t_start",             t_start},
+
+        {"prompt_bucket",       prompt_bucket .to_json()},
+        {"predict_bucket",      predict_bucket.to_json()},
+        {"prompt",              prompt        .to_json()},
+        {"predict",             predict       .to_json()},
+
+        {"n_prompt_cached",     n_prompt_cached},
+        {"n_tokens_max",        n_tokens_max},
+        {"n_decode",            n_decode},
+        {"n_busy_slots",        n_busy_slots},
+
+        {"n_draft_tokens",      n_draft_tokens},
+        {"n_draft_accepted",    n_draft_accepted},
+        {"n_draft_verif_steps", n_draft_verif_steps},
+        {"n_accepted_per_pos",  n_accepted_per_pos},
+    };
+}
+
+void server_metrics::from_json(const json & data) {
+    t_start = data.at("t_start");
+
+    prompt_bucket .from_json(data.at("prompt_bucket"));
+    predict_bucket.from_json(data.at("predict_bucket"));
+    prompt        .from_json(data.at("prompt"));
+    predict       .from_json(data.at("predict"));
+
+    n_prompt_cached = data.at("n_prompt_cached");
+    n_tokens_max    = data.at("n_tokens_max");
+    n_decode        = data.at("n_decode");
+    n_busy_slots    = data.at("n_busy_slots");
+
+    n_draft_tokens      = data.at("n_draft_tokens");
+    n_draft_accepted    = data.at("n_draft_accepted");
+    n_draft_verif_steps = data.at("n_draft_verif_steps");
+    n_accepted_per_pos  = data.at("n_accepted_per_pos").get<std::vector<uint64_t>>();
+}
+
 //
 // random string / id
 //
@@ -1819,3 +1889,140 @@ server_tokens format_prompt_rerank(
 
     return result;
 }
+
+
+//
+// server_sleep_rst
+//
+
+#if !defined(_WIN32)
+static std::string server_proc_exe_path(char ** argv) {
+    char buf[PATH_MAX];
+#if defined(__linux__)
+    const ssize_t len = readlink("/proc/self/exe", buf, sizeof(buf) - 1);
+    if (len > 0) {
+        buf[len] = '\0';
+        return buf;
+    }
+#elif defined(__APPLE__)
+    uint32_t size = sizeof(buf);
+    if (_NSGetExecutablePath(buf, &size) == 0) {
+        return buf;
+    }
+#endif
+    return argv[0];
+}
+
+// exec() keeps the file descriptors open, so mark them all to be closed instead
+// this releases the listening port and the backend devices, and makes child processes see EOF
+static void server_proc_close_fds_on_exec() {
+    int n_fd = 4096;
+
+    struct rlimit lim;
+    if (getrlimit(RLIMIT_NOFILE, &lim) == 0 && lim.rlim_cur != RLIM_INFINITY) {
+        n_fd = std::min<int>(lim.rlim_cur, 65536);
+    }
+
+    // skip stdin/stdout/stderr, they are used to communicate with the router
+    for (int fd = 3; fd < n_fd; fd++) {
+        const int flags = fcntl(fd, F_GETFD);
+        if (flags != -1) {
+            fcntl(fd, F_SETFD, flags | FD_CLOEXEC);
+        }
+    }
+}
+#endif
+
+static void server_proc_restart(char ** argv, const char * env_name, const std::string & env_value) {
+#if defined(_WIN32) || defined(__EMSCRIPTEN__)
+    GGML_UNUSED(argv);
+    GGML_UNUSED(env_name);
+    GGML_UNUSED(env_value);
+    SRV_ERR("%s", "restarting the process is not supported on this platform\n");
+#else
+    GGML_ASSERT(argv != nullptr);
+
+    // exec() rejects an env var larger than MAX_ARG_STRLEN (128 kB on linux)
+    if (env_value.size() > 64*1024) {
+        SRV_ERR("cannot restart the process, '%s' is too large (%zu bytes)\n", env_name, env_value.size());
+        return;
+    }
+
+    common_set_env(env_name, env_value);
+
+    const std::string exe = server_proc_exe_path(argv);
+    SRV_INF("restarting the process, exe = '%s'\n", exe.c_str());
+
+    server_proc_close_fds_on_exec();
+
+    // the log worker thread does not survive exec(), flush it while we still can
+    common_log_pause(common_log_main());
+    fflush(stdout);
+    fflush(stderr);
+
+    execv(exe.c_str(), argv);
+
+    // exec() only returns on error, the server can no longer serve requests at this point
+    GGML_ABORT("execv() failed: %s", strerror(errno));
+#endif
+}
+
+static const char * SLEEP_STATE_ENV = "LLAMA_SERVER_SLEEP_STATE";
+
+void server_sleep_rst::init(int argc, char ** argv) {
+    GGML_ASSERT(argv == nullptr || argc > 0);
+
+    this->argv = argv;
+
+    const std::string state = common_get_env(SLEEP_STATE_ENV);
+    if (state.empty()) {
+        return;
+    }
+
+    // note: the env var is kept, is_boot_to_sleep() reads it during the whole process lifetime
+    try {
+        boot_state = json::parse(state);
+    } catch (const std::exception & e) {
+        SRV_ERR("failed to read the state left by the previous process: %s\n", e.what());
+        common_set_env(SLEEP_STATE_ENV, ""); // unusable, boot normally instead
+    }
+}
+
+void server_sleep_rst::enable(common_params & params) {
+    // the state left by the previous process is unusable without the restart, drop it
+    auto disable = [this]() {
+        boot_state = json();
+        common_set_env(SLEEP_STATE_ENV, "");
+    };
+
+    if (params.sleep_mode != COMMON_SLEEP_MODE_RST) {
+        disable();
+        return;
+    }
+
+    if (argv == nullptr) {
+        // exec() can only restart a standalone process
+        SRV_WRN("%s", "--sleep-mode rst is not supported in this mode, using --sleep-mode free\n");
+        params.sleep_mode = COMMON_SLEEP_MODE_FREE;
+        disable();
+        return;
+    }
+
+    if (params.sleep_idle_seconds < 0) {
+        SRV_WRN("%s", "--sleep-mode has no effect without --sleep-idle-seconds\n");
+    }
+
+    enabled = true;
+}
+
+bool server_sleep_rst::is_boot_to_sleep() {
+    return !common_get_env(SLEEP_STATE_ENV).empty();
+}
+
+void server_sleep_rst::restart() const {
+    if (!enabled) {
+        return;
+    }
+
+    server_proc_restart(argv, SLEEP_STATE_ENV, safe_json_to_str(state_provider ? state_provider() : json()));
+}

tools/server/server-common.h

@@ -450,6 +450,9 @@ struct server_metrics {
             steps += n_steps;
             time  += t_us;
         }
+
+        json to_json() const;
+        void from_json(const json & data);
     };
 
     // these are reset by reset_bucket(), only the rate is read from them
@@ -490,6 +493,10 @@ struct server_metrics {
     void add_prompt_cached(uint64_t n_tokens) {
         n_prompt_cached += n_tokens;
     }
+
+    // used to keep the metrics across a process restart, see --sleep-mode rst
+    json to_json() const;
+    void from_json(const json & data);
 };
 
 //
@@ -604,3 +611,39 @@ struct server_pipe {
         return true;
     }
 };
+
+//
+// server_sleep_rst
+// this allow --sleep-mode rst to reset the whole process, but still preserve metrics and props data
+// see README-dev.md for more info
+//
+
+struct server_sleep_rst {
+    // true if the process was restarted by a previous instance
+    // in this case, the model is only loaded upon the first request
+    static bool is_boot_to_sleep();
+
+    // remember argv and read the state left by the previous process
+    // must be called once at startup, before spawning any thread or child process
+    void init(int argc, char ** argv);
+
+    // enable the restart upon sleeping, warns and falls back to --sleep-mode free if not possible
+    void enable(common_params & params);
+
+    // state left by the previous process, only valid if is_boot_to_sleep()
+    const json & get_boot_state() const { return boot_state; }
+
+    // set the state to be preserved across the restart
+    void set_state_provider(std::function<json()> provider) { state_provider = std::move(provider); }
+
+    // restart the process, does nothing if not enabled, does not return on success
+    void restart() const;
+
+private:
+    bool    enabled = false;
+    char ** argv    = nullptr;
+
+    json boot_state;
+
+    std::function<json()> state_provider;
+};