sycl: make --fit respect --fit-target better

[nicois]

Overview

Make --fit work properly on SYCL, so --fit-target 1 is viable

Additional information

There are two problems which make --fit not work as well as it should for SYCL devices:

• it doesn't properly account for how much extra VRAM will be needed when the KV cache is fully populated
• it is too pessimistic about how much VRAM is actually required for other reasons.

This means that SYNC users have to spend a long time adjusting ---fit-target to find a value large enough that it doesn't OOM when all the context is used.

This is a bigger problem for users who can fit all their KV cache onto their GPU, like me with a Arc b70 running unsloth's qwen 3.8. If only 12Gb of VRAM is available, most of the KV cache is in normal RAM anyway, so the under-estimate is less obvious.

With this PR applied, you should be able to run with --fit-target 1 and not have it OOM. The maximum context size might be a little lower than --fit calculated previously, but this time it's calculated correctly and won't crash mid-run.

To properly log memory allocations and deallocations I used a sibling PR, but as this is my first contribution I'll keep the other PR in draft until this is merged.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: yes: I used Claude Opus for most of the heavy lifting here, and have reviewed what it's produced to the best of my ability - but I am still learning how this all works, so this certainly needs more competent review before it can be merged. I've tried to undo most of claude's excessive inline comments but have left a few here which I found helpful.

[ggml-gh-bot]

Hi @nicois, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[NeoZhangJianyu]

It's good job!
Small comments to make the code be graceful and easy to understand.

Thank you!

[nicois]

Thanks for that feedback. I believe I've incorporated all of it.

[Titaniumtown]

@nicois thank you for your contribution! I like the changes and good thinking, I like the code in some areas too. Had some comments and attached those. Looking forward to seeing this get merged :)

[nicois]

Thanks for the further feedback. I'll definitely remove the verbosity and unslop what remains.

[Titaniumtown]

@nicois No problem. I know it is very useful, and I think I tend to overuse sometimes too, but we (as-in people in general) have to think a bit about the code and how to describe it and not just rely on the AI comments. Human written comments tend to make more sense and have more character in my opinion.

[arthw]

It's good job!

Thank you!

[Titaniumtown]

@nicois Can you show it actually working? Like setting a fit-target before, and after your change, and showing vram usage and ctx size being different?

[nicois]

With this patch, on my Arc b70 (32Gb of VRAM), this is what I see.

First scenario (omitting args which don't affect memory usage, such as temperature etc.) is

llama-server --no-mmproj-auto --spec-draft-n-max 3 --spec-type draft-mtp --batch-size 8192 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --fit on --fit-target 1 --hf-repo unsloth/Qwen3.8-27B-GGUF:Q8_0 --ubatch-size 1024

which yields

srv    load_model: initializing, n_slots = 1, n_ctx_slot = 98560, kv_unified = 'false'

Using the unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL model, I can increase --ubatch-size 2048 and still get maximal context:

srv    load_model: initializing, n_slots = 1, n_ctx_slot = 262144, kv_unified = 'false'

.. and this context is fully usable; I have filled it and not OOMed.

Using the unpatched version, with the Q8 model and 1024 ubatch size, llama-server reports

Aug 29 08:42:12 rainbow llama-b70-server[601154]: [57657] level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)Exception caught at file:/app/ggml/src/ggml-sycl/ggml-sycl.cpp, line:5428
Aug 29 08:42:12 rainbow llama-b70-server[601154]: [57657] Error OP MUL_MAT

with --fit-target 1.

With Q4 and 2048 ubatch it reports

srv    load_model: initializing, n_slots = 1, n_ctx_slot = 262144, kv_unified = 'false'

same as with the patch, but then crashes at around X tokens:

level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

[Titaniumtown]

Fantastic! Thank you <3