Fix VRAM usage increase on HIP/ROCm with RDNA3 and possibly more

[stragulus]

Overview

Fixes unbounded VRAM usage growth when using flash attention and quantized kv cache types on HIP/ROCm with RDNA3 architecture (and possibly more)

Additional details

This PR batches flash attention kernel invocation with a fixed size batch when quantized kv cache types are used. Without this change, the kernels would increasingly allocate more vram as they require the entire kv cache to be dequantized to f16 types. A compounding problem existed when using the legacy memory pool ggml_cuda_pool_leg (e.g. when using HIP with at least RDNA3). The memory pool allocates fragments of a fixed size, and allocation requests use a best-fit algorithm. Allocation requests that don't fit into any of the existing individual fragments trigger allocation of a new fragment. The pool never deallocates fragments until the pool itself is destructed. Due to the ever-growing sizes of the allocation requests, new allocations would not fit in a previously created fragment, thus another new fragment would be allocated. This rapidly consumes more and more vram, completely cancelling out any benefit to be had from using quantized kv cache types for non-synthetic workloads.

The VEC flash attention kernel solves this by dequantizing the kv cache entries one by one instead of all in advance. Simple, however, this has a very significant impact on prompt processing speed (easily ~50% reduction compared to the other kernels like TILE). While this does completely solve the growing vram usage problem, the performance impact is not acceptable.

Thus, I chose a hybrid solution where we don't fall back to slow VEC, but instead the flash attention kernel calls are batched with a maximum batch size if quantized kv types are used. The code will choose the previous unbatched path for maximum performance if batching is not required. Benchmarks show that prompt processing speeds for a 30k prompt and turbo3 k/v types are ~700 tokens per seconds, versus ~3xx tokens/sec when using the VEC kernel. VRAM usage stays stable, also tested with a 120k bench that shows 0 new allocations after prompt processing starts.

Token generation speed should be unaffected by these changes.

This SHOULD work on the other flash attention kernels as well, but I did not test this.

Passed basic coding challenge using opencode against llama-server with turbo3/turbo3 k/v quants

Tested on an arch linux system with a 7900 XTX/24GB and rocm-7.2.1

Benchmarks

Before changes:

VRAM flat at 18024
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 |  1 |         pp30000 |        697.66 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 |  1 |           tg128 |         28.81 ± 0.00 |

VRAM grows steadily during run to 17856, will keep increasing with longer prompts
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 |   q8_0 |   q8_0 |  1 |         pp30000 |        697.05 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 |   q8_0 |   q8_0 |  1 |           tg128 |         28.35 ± 0.00 |

VRAM grows steadily during run to 17222, will keep increasing with longer prompts
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 | turbo3 | turbo3 |  1 |         pp30000 |        695.24 ± 0.00 |
| qwen35 27B Q4_K - Medium       |  15.39 GiB |    26.90 B | ROCm       | 100 | turbo3 | turbo3 |  1 |           tg128 |         28.18 ± 0.00 |

After changes:

VRAM flat at 18024
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | 1 | pp30000 | 695.04 ± 0.00 |
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | 1 | tg128 | 27.92 ± 0.00 |

VRAM usage stable at 17124MB
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | q8_0 | q8_0 | 1 | pp30000 | 698.56 ± 0.00 |
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | q8_0 | q8_0 | 1 | tg128 | 28.82 ± 0.00 |

VRAM stable at 16508MB
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | turbo3 | turbo3 | 1 | pp30000 | 675.35 ± 0.00 |
| qwen35 27B Q4_K - Medium | 15.39 GiB | 26.90 B | ROCm | 100 | turbo3 | turbo3 | 1 | tg128 | 27.47 ± 0.00 |

- I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md)
- AI usage disclosure: YES, used claude to assist finding the problem, and prototype solutions

[stragulus]

I had to wipe out my fork as it somehow blocked forking llama.cpp itself - since these changes are not related to the turbo quants themselves and are a pre-existing issue in llama.cpp, I will instead add the PR there. ggml-org#19979