Context size limit with SYCL backend

[rim99]

I'm playing llama.cpp with SYCL backend using integrated graphics of Core Gen 13.

I find that when the context size goes beyond about 750 tokens, the 1.7B model I'm testing with starts to generate meaningless characters.

./build/bin/llama-server -m ~/Projects/quantized-models/qwen1_7b.q3_k.gguf --port 8080

The limit of 750-token would disappear if parameter -nkvo added in the cmdline.

Is it expected? Or it needs to be improved?

[NeoZhangJianyu]

@rim99
-nkvo will disable KV cache and give more memory for running.
It will make support bigger context.
But memory size issue will lead to crash, instead of meaning less characters.

If the beginning part is normal, but there are more meaningless chars appear in the end of output. It's another issue.
Could you share the logs?

Thank you!

[rim99]

@NeoZhangJianyu
OK, here is a file of truncated logs. The original logs file is huge (more than 30k lines)
llama.cpp.truncated.log.txt

Command I use

./build/bin/llama-server -m ~/Projects/quantized-models/qwen1_7b.q3_k.gguf --port 8080 -c 4096 -ngl 1 -v &> ~/llama.cpp.log

The screenshot of a similar case, but not exacly the case matches with these logs

Also, the verison info

➜  llama.cpp git:(master) ✗ ./build/bin/llama-cli --version                                                                                               
version: 6725 (1faa13a11)
built with Intel(R) oneAPI DPC++/C++ Compiler 2025.2.1 (2025.2.0.20250806) for x86_64-unknown-linux-gnu

Let me know if you need anything else. I'm happy to help!

[NeoZhangJianyu]

Sorry for the delayed reply!

In your case, there is only 1 layer handled by SYCL, others are handled by CPU in fact.
"offloaded 1/29 layers to GPU"

The result should be corrected.
The meaningless chars are created due to the quantized model issue: The little errors in many layers will be accumulated and become wrong.

Thank you!

[rim99]

Thanks for confirmation