[Feature] Support llama.cpp cache_prompt parameter

[Mushoz]

llama.cpp server accepts an optional parameter cache_prompt in the request to reuse the KV cache for matching prefixes, see: https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md . This can massively speed up prompt processing. Furthermore, it is a required parameter to utilize the newly introduced speculative decoding, see: ggml-org/llama.cpp#10455 (comment)

Very few clients support setting this optional parameter. That means there is no easy way to use this functionality. Therefor, I believe it might be ideal if llama-swap would be capable of adding this parameter to requests. This way, this feature can be implemented without requiring to implement it on a client-by-client basis.

[mostlygeek]

I think support for a default cache_prompt should land either in the client or in llama.cpp's server's code. My preference is that llama.cpp server support it directly with a --cache-prompt 1 flag.

Supporting it in llama-swap would be a hack to work around clients/llama.cpp not being in sync.

FWIW: I use librechat and I added it into the configuration with addParams like this:

endpoints: 
  custom: 
    - name: "lswapdev"
      apiKey: "sk-no-key-required"
      baseURL: "http://10.0.1.135:8080/v1"
      models:
        default:
          - "llama"
          - "qwen"
        fetch: true
      addParams:
        cache_prompt: true

[mostlygeek]

cache_prompt defaults to true now as of ggml-org/llama.cpp#10501.

[Mushoz]

I noticed it as well :) That makes things a lot easier. Thanks for having a look!