feat: Add multi-GPU MoE hot-cache expert lanes

README.md

@@ -15,6 +15,133 @@ Supported:
 - Qwen3Next
 - Mellum MoE models
 
+## Cached-experts configuration examples
+
+The examples below use `llama-server` and keep the normal model path on the primary CUDA device. Replace `MODEL.gguf` and `/path/to/moe-perf-data` with your model and the directory or JSON data produced for the hot-cache planner.
+
+### Default cached-experts, one CUDA card
+
+Use this when one CUDA device should hold the normal graph/KV path and as many cached experts as the automatic budget allows. Remaining experts stay on CPU via `--cpu-moe`.
+
+```sh
+LLAMA_MOE_HOT_CACHE_CPU_DECODE_ROUTING=1 \
+LLAMA_MOE_HOT_CACHE_PARALLEL=1 \
+./build/bin/llama-server \
+  --model MODEL.gguf \
+  --device CUDA0 \
+  --split-mode none \
+  --main-gpu 0 \
+  --n-gpu-layers 99 \
+  --cpu-moe \
+  --ctx-size 4096 \
+  --ubatch-size 32 \
+  --flash-attn on \
+  --cache-type-k q8_0 \
+  --cache-type-v q8_0 \
+  --moe-hot-cache /path/to/moe-perf-data \
+  --moe-hot-cache-max-mib -1 \
+  --moe-hot-cache-auto-reserve-mib 1024 \
+  --moe-hot-cache-pp-reduce-merge on
+```
+
+### Two CUDA cards
+
+Use this when `CUDA0` should remain the primary card for graph/KV/router/final merge and `CUDA1` should act as an additional expert lane. `warm` fills the primary lane first, then the second lane. For similar cards, try `hot-even`.
+
+```sh
+GGML_CUDA_P2P=1 \
+LLAMA_MOE_HOT_CACHE_CPU_DECODE_ROUTING=1 \
+LLAMA_MOE_HOT_CACHE_PARALLEL=1 \
+./build/bin/llama-server \
+  --model MODEL.gguf \
+  --device CUDA0 \
+  --split-mode none \
+  --main-gpu 0 \
+  --n-gpu-layers 99 \
+  --cpu-moe \
+  --ctx-size 4096 \
+  --ubatch-size 32 \
+  --flash-attn on \
+  --cache-type-k q8_0 \
+  --cache-type-v q8_0 \
+  --moe-hot-cache /path/to/moe-perf-data \
+  --moe-hot-cache-max-mib -1 \
+  --moe-hot-cache-auto-reserve-mib 1024 \
+  --moe-hot-cache-second-device CUDA1 \
+  --moe-hot-cache-second-max-mib -1 \
+  --moe-hot-cache-second-auto-reserve-mib 512 \
+  --moe-hot-cache-device-strategy warm \
+  --moe-hot-cache-pp-reduce-merge on
+```
+
+For a small primary GPU that should only run graph/KV/router/final merge, disable the primary expert cache and place experts on the second device:
+
+```sh
+GGML_CUDA_P2P=1 \
+LLAMA_MOE_HOT_CACHE_CPU_DECODE_ROUTING=1 \
+LLAMA_MOE_HOT_CACHE_PARALLEL=1 \
+./build/bin/llama-server \
+  --model MODEL.gguf \
+  --device CUDA0 \
+  --split-mode none \
+  --main-gpu 0 \
+  --n-gpu-layers 99 \
+  --cpu-moe \
+  --ctx-size 4096 \
+  --ubatch-size 32 \
+  --flash-attn on \
+  --cache-type-k q8_0 \
+  --cache-type-v q8_0 \
+  --moe-hot-cache /path/to/moe-perf-data \
+  --moe-hot-cache-max-mib 0 \
+  --moe-hot-cache-second-device CUDA1 \
+  --moe-hot-cache-second-max-mib -1 \
+  --moe-hot-cache-second-auto-reserve-mib 512 \
+  --moe-hot-cache-device-strategy warm \
+  --moe-hot-cache-pp-reduce-merge on
+```
+
+### Three CUDA cards
+
+Use this when `CUDA0` is the primary card and `CUDA1`/`CUDA2` are additional expert lanes. This is the intended shape for one primary GPU plus two expert-only GPUs. Keep per-device reserve high enough for temporary buffers; reduce `--ctx-size` or `--ubatch-size` first if CUDA allocation fails.
+
+```sh
+GGML_CUDA_P2P=1 \
+LLAMA_MOE_HOT_CACHE_CPU_DECODE_ROUTING=1 \
+LLAMA_MOE_HOT_CACHE_PARALLEL=1 \
+./build/bin/llama-server \
+  --model MODEL.gguf \
+  --device CUDA0 \
+  --split-mode none \
+  --main-gpu 0 \
+  --n-gpu-layers 99 \
+  --cpu-moe \
+  --ctx-size 4096 \
+  --ubatch-size 32 \
+  --flash-attn on \
+  --cache-type-k q8_0 \
+  --cache-type-v q8_0 \
+  --moe-hot-cache /path/to/moe-perf-data \
+  --moe-hot-cache-max-mib -1 \
+  --moe-hot-cache-auto-reserve-mib 1024 \
+  --moe-hot-cache-second-device CUDA1 \
+  --moe-hot-cache-second-max-mib -1 \
+  --moe-hot-cache-second-auto-reserve-mib 512 \
+  --moe-hot-cache-third-device CUDA2 \
+  --moe-hot-cache-third-max-mib -1 \
+  --moe-hot-cache-third-auto-reserve-mib 512 \
+  --moe-hot-cache-device-strategy hot-even \
+  --moe-hot-cache-pp-reduce-merge on
+```
+
+Notes:
+
+- `--moe-hot-cache-max-mib -1` auto-sizes a lane from currently free VRAM minus its reserve.
+- `--moe-hot-cache-max-mib 0` disables the primary expert lane while keeping secondary or tertiary expert lanes available.
+- `GGML_CUDA_P2P=1` enables CUDA peer-copy when the cards and driver support it; unsupported pairs fall back internally.
+- `LLAMA_MOE_HOT_CACHE_PARALLEL=force` is a debugging mode for valid parallel regions. Use `1`/`auto` for normal runs.
+- A general speedup claim for two GPUs could not be validated on the available test hardware because the cards are very asymmetric. Treat the two-GPU examples as configuration starting points, not as benchmark guidance.
+
 These changes will probably never reach upstream llama because I broke the contribution rules hardly. I am a Java developer and the last time I wrote anything in C is, I even don't remember when it was, therefore, the bit of knowlegde of C that I had is gone. Secondly, this is a tool for me, I want it to function, I want it to be easy and I used other tools to create it faster.
 And lastly, I saw some discussions in the PRs and the tone is not what I would expect. I know especially big PRs are hard to overlook, but great features often create big PRs. I also hate big PRs. But, sometimes they are necessary. Anyway, I don't want to have such discussions, it's just a waste of time.
 

common/arg.cpp

@@ -73,6 +73,10 @@ static bool llama_moe_hot_cache_weighting_valid(const std::string & value) {
            value == "flat";
 }
 
+static bool llama_moe_hot_cache_device_strategy_valid(const std::string & value) {
+    return value == "warm" || value == "hot-even";
+}
+
 static void llama_moe_hot_cache_set_weighting_env(const std::string & value) {
 #if defined(_WIN32)
     _putenv_s("LLAMA_MOE_HOT_CACHE_WEIGHTING", value.c_str());
@@ -999,17 +1003,35 @@ bool common_params_parse(int argc, char ** argv, common_params & params, llama_e
             common_params_print_completion(ctx_arg);
             exit(0);
         }
-        if (ctx_arg.params.moe_hot_cache_max_mib != 0 && ctx_arg.params.moe_hot_cache.empty()) {
+        const bool moe_hot_cache_any_lane =
+            ctx_arg.params.moe_hot_cache_max_mib != 0 ||
+            ctx_arg.params.moe_hot_cache_second_max_mib != 0 ||
+            ctx_arg.params.moe_hot_cache_third_max_mib != 0;
+        if (moe_hot_cache_any_lane && ctx_arg.params.moe_hot_cache.empty()) {
             throw std::invalid_argument("--moe-hot-cache is required when --moe-hot-cache-max-mib is not 0");
         }
         if (ctx_arg.params.moe_hot_cache_max_mib < -1) {
             throw std::invalid_argument("--moe-hot-cache-max-mib must be -1 or greater");
         }
-        if (ctx_arg.params.moe_hot_cache_max_mib == -1 && ctx_arg.params.n_ctx <= 0) {
-            throw std::invalid_argument("--moe-hot-cache-max-mib -1 requires an explicit --ctx-size");
+        if (ctx_arg.params.moe_hot_cache_second_max_mib < -1) {
+            throw std::invalid_argument("--moe-hot-cache-second-max-mib must be -1 or greater");
+        }
+        if (ctx_arg.params.moe_hot_cache_third_max_mib < -1) {
+            throw std::invalid_argument("--moe-hot-cache-third-max-mib must be -1 or greater");
+        }
+        if ((ctx_arg.params.moe_hot_cache_max_mib == -1 ||
+             ctx_arg.params.moe_hot_cache_second_max_mib == -1 ||
+             ctx_arg.params.moe_hot_cache_third_max_mib == -1) && ctx_arg.params.n_ctx <= 0) {
+            throw std::invalid_argument("--moe-hot-cache-*-max-mib -1 requires an explicit --ctx-size");
+        }
+        if (ctx_arg.params.moe_hot_cache_second_max_mib != 0 && ctx_arg.params.moe_hot_cache_second_device.empty()) {
+            throw std::invalid_argument("--moe-hot-cache-second-device is required when --moe-hot-cache-second-max-mib is not 0");
+        }
+        if (ctx_arg.params.moe_hot_cache_third_max_mib != 0 && ctx_arg.params.moe_hot_cache_third_device.empty()) {
+            throw std::invalid_argument("--moe-hot-cache-third-device is required when --moe-hot-cache-third-max-mib is not 0");
         }
-        if (ctx_arg.params.moe_hot_cache_update_rate > 0.0f && ctx_arg.params.moe_hot_cache_max_mib == 0) {
-            throw std::invalid_argument("--moe-hot-cache-update-rate requires --moe-hot-cache-max-mib");
+        if (ctx_arg.params.moe_hot_cache_update_rate > 0.0f && !moe_hot_cache_any_lane) {
+            throw std::invalid_argument("--moe-hot-cache-update-rate requires a MoE hot-cache lane budget");
         }
         if (!ctx_arg.params.moe_layer_perf_out.empty()) {
             ctx_arg.params.no_perf = false;
@@ -2417,6 +2439,13 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.moe_hot_cache_max_mib = value;
         }
     ).set_env("LLAMA_ARG_MOE_HOT_CACHE_MAX_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-device"}, "DEV",
+        "experimental: backend device for the primary MoE hot-cache expert lane (default: first model GPU/iGPU)",
+        [](common_params & params, const std::string & value) {
+            params.moe_hot_cache_device = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_DEVICE"));
     add_opt(common_arg(
         {"--moe-hot-cache-auto-reserve-mib"}, "N",
         string_format("experimental: MiB to keep free when --moe-hot-cache-max-mib -1 auto-sizes the hot cache (default: %zu)", (size_t) params.moe_hot_cache_auto_reserve_mib),
@@ -2428,6 +2457,74 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.moe_hot_cache_auto_reserve_mib = uint64_t(value);
         }
     ).set_env("LLAMA_ARG_MOE_HOT_CACHE_AUTO_RESERVE_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-second-device"}, "DEV",
+        "experimental: backend device for the optional second MoE hot-cache expert lane",
+        [](common_params & params, const std::string & value) {
+            params.moe_hot_cache_second_device = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_SECOND_DEVICE"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-second-max-mib"}, "N",
+        "experimental: max MiB for the optional second MoE hot-cache expert lane (0 = disabled, -1 = auto)",
+        [](common_params & params, const std::string & value_str) {
+            const int64_t value = std::stoll(value_str);
+            if (value < -1) {
+                throw std::invalid_argument("invalid value");
+            }
+            params.moe_hot_cache_second_max_mib = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_SECOND_MAX_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-second-auto-reserve-mib"}, "N",
+        string_format("experimental: MiB to keep free when --moe-hot-cache-second-max-mib -1 auto-sizes the second lane (default: %zu)", (size_t) params.moe_hot_cache_second_auto_reserve_mib),
+        [](common_params & params, const std::string & value_str) {
+            const int64_t value = std::stoll(value_str);
+            if (value < 0) {
+                throw std::invalid_argument("invalid value");
+            }
+            params.moe_hot_cache_second_auto_reserve_mib = uint64_t(value);
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_SECOND_AUTO_RESERVE_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-third-device"}, "DEV",
+        "experimental: backend device for the optional third MoE hot-cache expert lane",
+        [](common_params & params, const std::string & value) {
+            params.moe_hot_cache_third_device = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_THIRD_DEVICE"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-third-max-mib"}, "N",
+        "experimental: max MiB for the optional third MoE hot-cache expert lane (0 = disabled, -1 = auto)",
+        [](common_params & params, const std::string & value_str) {
+            const int64_t value = std::stoll(value_str);
+            if (value < -1) {
+                throw std::invalid_argument("invalid value");
+            }
+            params.moe_hot_cache_third_max_mib = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_THIRD_MAX_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-third-auto-reserve-mib"}, "N",
+        string_format("experimental: MiB to keep free when --moe-hot-cache-third-max-mib -1 auto-sizes the third lane (default: %zu)", (size_t) params.moe_hot_cache_third_auto_reserve_mib),
+        [](common_params & params, const std::string & value_str) {
+            const int64_t value = std::stoll(value_str);
+            if (value < 0) {
+                throw std::invalid_argument("invalid value");
+            }
+            params.moe_hot_cache_third_auto_reserve_mib = uint64_t(value);
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_THIRD_AUTO_RESERVE_MIB"));
+    add_opt(common_arg(
+        {"--moe-hot-cache-device-strategy"}, "{warm,hot-even}",
+        "experimental: distribute MoE hot-cache experts as warm lanes or per-layer hot-even lanes (default: warm)",
+        [](common_params & params, const std::string & value) {
+            if (!llama_moe_hot_cache_device_strategy_valid(value)) {
+                throw std::invalid_argument("--moe-hot-cache-device-strategy must be one of: warm, hot-even");
+            }
+            params.moe_hot_cache_device_strategy = value;
+        }
+    ).set_env("LLAMA_ARG_MOE_HOT_CACHE_DEVICE_STRATEGY"));
     add_opt(common_arg(
         {"--moe-hot-cache"}, "FNAME",
         "experimental: path to /moe-layer-perf JSON used by --moe-hot-cache-max-mib",

common/common.cpp

@@ -1534,6 +1534,14 @@ struct llama_model_params common_model_params_to_llama(common_params & params) {
     mparams.no_host         = params.no_host;
     mparams.moe_hot_cache_max_mib = params.moe_hot_cache_max_mib;
     mparams.moe_hot_cache_path    = params.moe_hot_cache.empty() ? nullptr : params.moe_hot_cache.c_str();
+    mparams.moe_hot_cache_device  = params.moe_hot_cache_device.empty() ? nullptr : params.moe_hot_cache_device.c_str();
+    mparams.moe_hot_cache_second_max_mib = params.moe_hot_cache_second_max_mib;
+    mparams.moe_hot_cache_second_device = params.moe_hot_cache_second_device.empty() ? nullptr : params.moe_hot_cache_second_device.c_str();
+    mparams.moe_hot_cache_second_auto_reserve_mib = params.moe_hot_cache_second_auto_reserve_mib;
+    mparams.moe_hot_cache_third_max_mib = params.moe_hot_cache_third_max_mib;
+    mparams.moe_hot_cache_third_device = params.moe_hot_cache_third_device.empty() ? nullptr : params.moe_hot_cache_third_device.c_str();
+    mparams.moe_hot_cache_third_auto_reserve_mib = params.moe_hot_cache_third_auto_reserve_mib;
+    mparams.moe_hot_cache_device_strategy = params.moe_hot_cache_device_strategy.empty() ? "warm" : params.moe_hot_cache_device_strategy.c_str();
     mparams.moe_hot_cache_auto_n_ctx = params.n_ctx > 0 ? uint32_t(params.n_ctx) : 0;
     mparams.moe_hot_cache_auto_n_seq_max = params.n_parallel > 0 ? uint32_t(params.n_parallel) : 1;
     mparams.moe_hot_cache_auto_n_ubatch = params.n_ubatch > 0 ? uint32_t(params.n_ubatch) : uint32_t(params.n_batch);

common/common.h

@@ -562,6 +562,14 @@ struct common_params {
 
     int64_t moe_hot_cache_max_mib = 0;   // max MiB for experimental MoE hot expert cache, 0 = disabled, -1 = auto
     uint64_t moe_hot_cache_auto_reserve_mib = 1024; // MiB kept free when auto-sizing the MoE hot expert cache
+    std::string moe_hot_cache_device;    // optional backend device for primary MoE hot-cache expert lane
+    int64_t moe_hot_cache_second_max_mib = 0; // max MiB for optional second MoE hot-cache expert lane
+    uint64_t moe_hot_cache_second_auto_reserve_mib = 512; // MiB kept free when auto-sizing the second lane
+    std::string moe_hot_cache_second_device; // backend device for optional second expert lane
+    int64_t moe_hot_cache_third_max_mib = 0; // max MiB for optional third MoE hot-cache expert lane
+    uint64_t moe_hot_cache_third_auto_reserve_mib = 512; // MiB kept free when auto-sizing the third lane
+    std::string moe_hot_cache_third_device; // backend device for optional third expert lane
+    std::string moe_hot_cache_device_strategy = "warm"; // MoE hot-cache device strategy: warm or hot-even
     std::string moe_hot_cache;           // path to /moe-layer-perf JSON
     float moe_hot_cache_update_rate = 0.0f; // fraction of hot-cache entries to update after each completed server run
     float moe_hot_cache_layer_curve = 0.5f; // MoE hot-cache layer-pressure weighting curve, 0 = flat, 1 = aggressive