cuda: reserve space for quantize kv-cache at startup

[am17an]

Overview

ref #23646 (comment). Quantized kv-cache can lead to OOM even when using --fit since it does not know about these backend allocations. There are some other quantization buffers in FA and MMQ which should also be removed, but this one seems it takes the most space as it scales with ctx size.

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, codex wrote this on my direction. I tested it on a few devices

[ggerganov]

Did you test -sm tensor setups? I think it should work, but might be worth double-checking.

[JohannesGaessler]

I was going to say that -sm tensor is implicitly being tested via test-llama-archs but there only the FP16/FP16 configuration is being tested. More generally: do we already have automated tests for different KV cache types? If not it may make sense to add some in test-llama-archs.

[am17an]

I did not check yet, we can wait for #23792 to get merged and check again because currently it will throw.

[AbdulrahmanHashem]

Did you test -sm tensor setups? I think it should work, but might be worth double-checking.

i have after also merging #23792, and as far as i see there is not more memory creeping, on my system (5060ti + 2060 super)

in all the following cases there is a 125 mp allocation during first prompt processing

without kv quant
without kv quant + MTP
without kv quant + ngram-mod
with kv quant
with kv quant + MTP
with kv quant + ngram-mod

with or without kv quant + ngram-mod
there is an additional variable amount of allocation under 150mp during tg

on a different issue with ngram-mod with or without kv quant
i tried ngram-mod + MTP it causes a crash with no error during tg after thinking is done.
it lags the llama ui very hard and just stops tg with no logs before it crashes

[coder543]

A couple of other things that I believe the fit algorithm is not reserving space for: cache-ram and ctx-checkpoints. On unified memory systems like the DGX Spark, this makes it hard to rely on the fit algorithm without specifying an arbitrarily large fit-target.

[am17an]

I think this should be ok to merge.

[am17an]

@ggml-org/ggml-cuda can I get another approval?

[am17an]

cc @ggml-org/maintainers, need another approval

[TomTheWise]

Thank you! This commit solves the issue #23978

[thomasbergersen]

Hello @am17an, this change has resulted in an additional increase in GPU memory usage.

b9488 -> GPU:0 13.8/16.0 GB GPU:1 13.8/16.0 GB
b9489 -> GPU:0 14.9/16.0 GB GPU:1 14.2/16.0 GB

[JohannesGaessler]

An increase in VRAM consumption is expected since llama.cpp is now pre-allocating the VRAM as part of the compute graphs. On master the initial VRAM consumption would be lower but eventually end up higher as the context fills up because the VRAM for converting the KV cache cannot be recycled for other operations. And previously the crash from OOMing would only happen after the program has already been running for some time which is undesirable if you're not babysitting it.

In any case, the VRAM consumption for KV cache conversion can be reduced by setting -ub to a lower value than 512.

[thomasbergersen]

After the test, after the pre-allocation adjustment was made, and after conducting multiple tests with long conversations, the GPU memory usage of my computer only increased by approximately 300 MB. Then is it absolutely safe from OOM?